2024-09-11T07:50:08.752 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-11T07:50:08.770 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-11T07:50:08.856 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/gabrioux-2024-09-11_06:53:43-orch:cephadm-wip-guits-quincy-2024-09-10-2008-quincy-distro-default-smithi/7901248 branch: wip-guits-quincy-2024-09-10-2008-quincy description: orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} email: gabrioux@ibm.com first_in_suite: false job_id: '7901248' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: gabrioux-2024-09-11_06:53:43-orch:cephadm-wip-guits-quincy-2024-09-10-2008-quincy-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-guits-quincy-2024-09-10-2008-quincy ceph: conf: client.0: debug ms: 1 debug objecter: 20 debug rados: 20 global: ms bind msgr1: false ms bind msgr2: true ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug ms: 1 debug osd: 20 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - \(POOL_APP_NOT_ENABLED\) sha1: d0e6828a2016d48cf25ad84064e50742bb1c39b9 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: root install: ceph: flavor: default sha1: d0e6828a2016d48cf25ad84064e50742bb1c39b9 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-guits-quincy-2024-09-10-2008-quincy sha1: d0e6828a2016d48cf25ad84064e50742bb1c39b9 owner: scheduled_gabrioux@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 2282 sha1: d0e6828a2016d48cf25ad84064e50742bb1c39b9 sleep_before_teardown: 0 subset: 1/8 suite: orch:cephadm suite_branch: wip-guits-quincy-2024-09-10-2008-quincy suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_d0e6828a2016d48cf25ad84064e50742bb1c39b9/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: d0e6828a2016d48cf25ad84064e50742bb1c39b9 targets: smithi150.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBc41GpyAmd0NOe1euduKXxVdCg7ekBFwlTQMPbBWZ6gGSFjpBiNo+bsJnNe0QdqR1B0X0HTlIBavE+YVYGDuPA= smithi181.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI+6kHiCuvO5RX5AmBBhr0ebmMppv989bWYZLFSTs4+0Ilu2vo1lM88v0VffgX2+z86Oq++FWPXZGMF+XCGnp+M= tasks: - pexec: all: - sudo dnf remove nvme-cli -y - sudo dnf install nvmetcli nvme-cli -y - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-11_06:53:43 tube: smithi user: gabrioux verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-11T07:50:08.856 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_d0e6828a2016d48cf25ad84064e50742bb1c39b9/qa; will attempt to use it 2024-09-11T07:50:08.857 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_d0e6828a2016d48cf25ad84064e50742bb1c39b9/qa/tasks 2024-09-11T07:50:08.857 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-11T07:50:08.859 INFO:teuthology.task.internal:Checking packages... 2024-09-11T07:50:08.880 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd0e6828a2016d48cf25ad84064e50742bb1c39b9' 2024-09-11T07:50:08.880 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-11T07:50:08.880 INFO:teuthology.packaging:ref: None 2024-09-11T07:50:08.880 INFO:teuthology.packaging:tag: None 2024-09-11T07:50:08.880 INFO:teuthology.packaging:branch: wip-guits-quincy-2024-09-10-2008-quincy 2024-09-11T07:50:08.880 INFO:teuthology.packaging:sha1: d0e6828a2016d48cf25ad84064e50742bb1c39b9 2024-09-11T07:50:08.880 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-guits-quincy-2024-09-10-2008-quincy 2024-09-11T07:50:09.029 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-1623.gd0e6828a 2024-09-11T07:50:09.031 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-11T07:50:09.037 INFO:teuthology.task.internal:no buildpackages task found 2024-09-11T07:50:09.037 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-11T07:50:09.050 INFO:teuthology.task.internal:Saving configuration 2024-09-11T07:50:09.061 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-11T07:50:09.068 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-11T07:50:09.088 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi150.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-09-11_06:53:43-orch:cephadm-wip-guits-quincy-2024-09-10-2008-quincy-distro-default-smithi/7901248', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-11 07:45:02.599298', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBc41GpyAmd0NOe1euduKXxVdCg7ekBFwlTQMPbBWZ6gGSFjpBiNo+bsJnNe0QdqR1B0X0HTlIBavE+YVYGDuPA='} 2024-09-11T07:50:09.107 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi181.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-09-11_06:53:43-orch:cephadm-wip-guits-quincy-2024-09-10-2008-quincy-distro-default-smithi/7901248', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-11 07:45:02.597300', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI+6kHiCuvO5RX5AmBBhr0ebmMppv989bWYZLFSTs4+0Ilu2vo1lM88v0VffgX2+z86Oq++FWPXZGMF+XCGnp+M='} 2024-09-11T07:50:09.107 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-11T07:50:09.113 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-09-11T07:50:09.114 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-09-11T07:50:09.114 INFO:teuthology.run_tasks:Running task console_log... 2024-09-11T07:50:09.166 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1ad33e3ac0>, signals=[15]) 2024-09-11T07:50:09.166 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-11T07:50:09.172 INFO:teuthology.task.internal:Opening connections... 2024-09-11T07:50:09.172 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi150.front.sepia.ceph.com 2024-09-11T07:50:09.173 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T07:50:09.248 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi181.front.sepia.ceph.com 2024-09-11T07:50:09.249 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T07:50:09.323 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-11T07:50:09.330 DEBUG:teuthology.orchestra.run.smithi150:> uname -m 2024-09-11T07:50:09.348 INFO:teuthology.orchestra.run.smithi150.stdout:x86_64 2024-09-11T07:50:09.349 DEBUG:teuthology.orchestra.run.smithi150:> cat /etc/os-release 2024-09-11T07:50:09.406 INFO:teuthology.orchestra.run.smithi150.stdout:NAME="CentOS Stream" 2024-09-11T07:50:09.406 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION="9" 2024-09-11T07:50:09.406 INFO:teuthology.orchestra.run.smithi150.stdout:ID="centos" 2024-09-11T07:50:09.407 INFO:teuthology.orchestra.run.smithi150.stdout:ID_LIKE="rhel fedora" 2024-09-11T07:50:09.407 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION_ID="9" 2024-09-11T07:50:09.407 INFO:teuthology.orchestra.run.smithi150.stdout:PLATFORM_ID="platform:el9" 2024-09-11T07:50:09.407 INFO:teuthology.orchestra.run.smithi150.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-11T07:50:09.407 INFO:teuthology.orchestra.run.smithi150.stdout:ANSI_COLOR="0;31" 2024-09-11T07:50:09.407 INFO:teuthology.orchestra.run.smithi150.stdout:LOGO="fedora-logo-icon" 2024-09-11T07:50:09.407 INFO:teuthology.orchestra.run.smithi150.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-11T07:50:09.407 INFO:teuthology.orchestra.run.smithi150.stdout:HOME_URL="https://centos.org/" 2024-09-11T07:50:09.407 INFO:teuthology.orchestra.run.smithi150.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-11T07:50:09.407 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-11T07:50:09.407 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-11T07:50:09.408 INFO:teuthology.lock.ops:Updating smithi150.front.sepia.ceph.com on lock server 2024-09-11T07:50:09.432 DEBUG:teuthology.orchestra.run.smithi181:> uname -m 2024-09-11T07:50:09.451 INFO:teuthology.orchestra.run.smithi181.stdout:x86_64 2024-09-11T07:50:09.451 DEBUG:teuthology.orchestra.run.smithi181:> cat /etc/os-release 2024-09-11T07:50:09.509 INFO:teuthology.orchestra.run.smithi181.stdout:NAME="CentOS Stream" 2024-09-11T07:50:09.510 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION="9" 2024-09-11T07:50:09.510 INFO:teuthology.orchestra.run.smithi181.stdout:ID="centos" 2024-09-11T07:50:09.510 INFO:teuthology.orchestra.run.smithi181.stdout:ID_LIKE="rhel fedora" 2024-09-11T07:50:09.510 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION_ID="9" 2024-09-11T07:50:09.510 INFO:teuthology.orchestra.run.smithi181.stdout:PLATFORM_ID="platform:el9" 2024-09-11T07:50:09.510 INFO:teuthology.orchestra.run.smithi181.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-11T07:50:09.510 INFO:teuthology.orchestra.run.smithi181.stdout:ANSI_COLOR="0;31" 2024-09-11T07:50:09.510 INFO:teuthology.orchestra.run.smithi181.stdout:LOGO="fedora-logo-icon" 2024-09-11T07:50:09.510 INFO:teuthology.orchestra.run.smithi181.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-11T07:50:09.510 INFO:teuthology.orchestra.run.smithi181.stdout:HOME_URL="https://centos.org/" 2024-09-11T07:50:09.510 INFO:teuthology.orchestra.run.smithi181.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-11T07:50:09.510 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-11T07:50:09.510 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-11T07:50:09.511 INFO:teuthology.lock.ops:Updating smithi181.front.sepia.ceph.com on lock server 2024-09-11T07:50:09.523 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-11T07:50:09.532 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-11T07:50:09.544 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-11T07:50:09.544 DEBUG:teuthology.orchestra.run.smithi150:> test '!' -e /home/ubuntu/cephtest 2024-09-11T07:50:09.547 DEBUG:teuthology.orchestra.run.smithi181:> test '!' -e /home/ubuntu/cephtest 2024-09-11T07:50:09.567 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-11T07:50:09.621 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-11T07:50:09.622 DEBUG:teuthology.orchestra.run.smithi150:> test -z $(ls -A /var/lib/ceph) 2024-09-11T07:50:09.624 DEBUG:teuthology.orchestra.run.smithi181:> test -z $(ls -A /var/lib/ceph) 2024-09-11T07:50:09.715 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-11T07:50:09.771 INFO:teuthology.run_tasks:Running task kernel... 2024-09-11T07:50:09.790 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-11T07:50:09.790 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-11T07:50:09.790 DEBUG:teuthology.orchestra.run.smithi150:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-11T07:50:09.791 DEBUG:teuthology.orchestra.run.smithi181:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-11T07:50:09.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T07:50:09.808 DEBUG:teuthology.orchestra.run.smithi150:> uname -r 2024-09-11T07:50:09.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T07:50:09.809 DEBUG:teuthology.orchestra.run.smithi181:> uname -r 2024-09-11T07:50:09.866 INFO:teuthology.orchestra.run.smithi150.stdout:5.14.0-503.el9.x86_64 2024-09-11T07:50:09.866 INFO:teuthology.task.kernel:Running kernel on smithi150: 5.14.0-503.el9.x86_64 2024-09-11T07:50:09.866 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum install -y kernel 2024-09-11T07:50:09.867 INFO:teuthology.orchestra.run.smithi181.stdout:5.14.0-503.el9.x86_64 2024-09-11T07:50:09.867 INFO:teuthology.task.kernel:Running kernel on smithi181: 5.14.0-503.el9.x86_64 2024-09-11T07:50:09.868 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum install -y kernel 2024-09-11T07:50:12.238 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.2 MB 00:00 2024-09-11T07:50:12.470 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.2 MB 00:00 2024-09-11T07:50:13.981 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - AppStream 34 MB/s | 20 MB 00:00 2024-09-11T07:50:19.529 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - AppStream 3.4 MB/s | 20 MB 00:05 2024-09-11T07:50:19.715 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - CRB 3.5 MB/s | 6.5 MB 00:01 2024-09-11T07:50:21.223 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - Extras packages 68 kB/s | 19 kB 00:00 2024-09-11T07:50:22.344 INFO:teuthology.orchestra.run.smithi150.stdout:Extra Packages for Enterprise Linux 23 MB/s | 23 MB 00:00 2024-09-11T07:50:25.955 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - CRB 2.5 MB/s | 6.5 MB 00:02 2024-09-11T07:50:27.759 INFO:teuthology.orchestra.run.smithi150.stdout:lab-extras 37 kB/s | 1.7 kB 00:00 2024-09-11T07:50:28.172 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - Extras packages 19 kB/s | 19 kB 00:00 2024-09-11T07:50:28.934 INFO:teuthology.orchestra.run.smithi181.stdout:Extra Packages for Enterprise Linux 36 MB/s | 23 MB 00:00 2024-09-11T07:50:29.715 INFO:teuthology.orchestra.run.smithi150.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-11T07:50:29.716 INFO:teuthology.orchestra.run.smithi150.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-11T07:50:29.804 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-09-11T07:50:29.813 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-09-11T07:50:29.813 INFO:teuthology.orchestra.run.smithi150.stdout: Package Architecture Version Repository Size 2024-09-11T07:50:29.813 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-09-11T07:50:29.813 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-09-11T07:50:29.813 INFO:teuthology.orchestra.run.smithi150.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-11T07:50:29.813 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2024-09-11T07:50:29.814 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-11T07:50:29.814 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-11T07:50:29.814 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-11T07:50:29.814 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:50:29.814 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-09-11T07:50:29.814 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-09-11T07:50:29.814 INFO:teuthology.orchestra.run.smithi150.stdout:Install 4 Packages 2024-09-11T07:50:29.814 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:50:29.814 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 79 M 2024-09-11T07:50:29.814 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 126 M 2024-09-11T07:50:29.814 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-09-11T07:50:30.018 INFO:teuthology.orchestra.run.smithi150.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 258 kB/s | 23 kB 00:00 2024-09-11T07:50:30.411 INFO:teuthology.orchestra.run.smithi150.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 33 MB/s | 16 MB 00:00 2024-09-11T07:50:30.777 INFO:teuthology.orchestra.run.smithi150.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 38 MB/s | 29 MB 00:00 2024-09-11T07:50:31.086 INFO:teuthology.orchestra.run.smithi150.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 30 MB/s | 35 MB 00:01 2024-09-11T07:50:31.086 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-09-11T07:50:31.086 INFO:teuthology.orchestra.run.smithi150.stdout:Total 62 MB/s | 79 MB 00:01 2024-09-11T07:50:31.768 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-09-11T07:50:32.127 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-09-11T07:50:32.127 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-09-11T07:50:33.076 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-09-11T07:50:33.076 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-09-11T07:50:34.382 INFO:teuthology.orchestra.run.smithi181.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-11T07:50:35.515 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-09-11T07:50:36.153 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-11T07:50:36.253 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T07:50:36.388 INFO:teuthology.orchestra.run.smithi181.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-11T07:50:36.388 INFO:teuthology.orchestra.run.smithi181.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-11T07:50:36.483 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-11T07:50:36.490 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T07:50:36.490 INFO:teuthology.orchestra.run.smithi181.stdout: Package Architecture Version Repository Size 2024-09-11T07:50:36.490 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T07:50:36.490 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-11T07:50:36.490 INFO:teuthology.orchestra.run.smithi181.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-11T07:50:36.490 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-11T07:50:36.490 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-11T07:50:36.490 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-11T07:50:36.490 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-11T07:50:36.490 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:50:36.491 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-11T07:50:36.491 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T07:50:36.491 INFO:teuthology.orchestra.run.smithi181.stdout:Install 4 Packages 2024-09-11T07:50:36.491 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:50:36.491 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 79 M 2024-09-11T07:50:36.491 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 126 M 2024-09-11T07:50:36.491 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-11T07:50:36.675 INFO:teuthology.orchestra.run.smithi181.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 274 kB/s | 23 kB 00:00 2024-09-11T07:50:37.083 INFO:teuthology.orchestra.run.smithi181.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 32 MB/s | 16 MB 00:00 2024-09-11T07:50:37.475 INFO:teuthology.orchestra.run.smithi181.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 39 MB/s | 35 MB 00:00 2024-09-11T07:50:37.867 INFO:teuthology.orchestra.run.smithi181.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 24 MB/s | 29 MB 00:01 2024-09-11T07:50:37.867 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-11T07:50:37.867 INFO:teuthology.orchestra.run.smithi181.stdout:Total 57 MB/s | 79 MB 00:01 2024-09-11T07:50:38.547 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-11T07:50:38.860 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-11T07:50:38.860 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-11T07:50:39.080 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T07:50:39.213 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T07:50:39.827 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-11T07:50:39.827 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-11T07:50:42.268 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-11T07:50:42.959 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-11T07:50:43.050 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T07:50:44.025 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T07:50:44.107 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-11T07:50:45.866 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T07:50:45.996 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T07:50:48.972 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T07:50:50.830 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T07:50:50.888 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-11T07:50:55.728 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T07:51:27.427 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T07:51:27.466 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-11T07:51:27.827 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-11T07:51:27.827 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-11T07:51:27.827 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T07:51:27.827 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T07:51:28.661 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T07:51:28.661 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:51:28.661 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-09-11T07:51:28.661 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-11T07:51:28.661 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-11T07:51:28.661 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-11T07:51:28.662 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-11T07:51:28.662 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:51:28.662 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-09-11T07:51:29.010 DEBUG:teuthology.orchestra.run.smithi150:> rpm -q kernel | sort -rV | head -n 1 2024-09-11T07:51:29.099 INFO:teuthology.orchestra.run.smithi150.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-11T07:51:29.100 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-11T07:51:29.100 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-11T07:51:29.100 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-11T07:51:29.100 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-11T07:51:29.100 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-11T07:51:29.100 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi150.front.sepia.ceph.com, path=None, version=distro) 2024-09-11T07:51:29.100 DEBUG:teuthology.orchestra.run.smithi150:> rpm -q kernel | sort -rV | head -n 1 2024-09-11T07:51:29.158 INFO:teuthology.orchestra.run.smithi150.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-11T07:51:29.159 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-11T07:51:29.159 DEBUG:teuthology.orchestra.run.smithi150:> sudo rpm -qi grub2-tools 2024-09-11T07:51:29.201 INFO:teuthology.orchestra.run.smithi150.stdout:Name : grub2-tools 2024-09-11T07:51:29.201 INFO:teuthology.orchestra.run.smithi150.stdout:Epoch : 1 2024-09-11T07:51:29.201 INFO:teuthology.orchestra.run.smithi150.stdout:Version : 2.06 2024-09-11T07:51:29.201 INFO:teuthology.orchestra.run.smithi150.stdout:Release : 82.el9 2024-09-11T07:51:29.201 INFO:teuthology.orchestra.run.smithi150.stdout:Architecture: x86_64 2024-09-11T07:51:29.202 INFO:teuthology.orchestra.run.smithi150.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-11T07:51:29.202 INFO:teuthology.orchestra.run.smithi150.stdout:Group : Unspecified 2024-09-11T07:51:29.202 INFO:teuthology.orchestra.run.smithi150.stdout:Size : 8274670 2024-09-11T07:51:29.202 INFO:teuthology.orchestra.run.smithi150.stdout:License : GPLv3+ 2024-09-11T07:51:29.202 INFO:teuthology.orchestra.run.smithi150.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-11T07:51:29.202 INFO:teuthology.orchestra.run.smithi150.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-11T07:51:29.202 INFO:teuthology.orchestra.run.smithi150.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-11T07:51:29.202 INFO:teuthology.orchestra.run.smithi150.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-11T07:51:29.202 INFO:teuthology.orchestra.run.smithi150.stdout:Packager : builder@centos.org 2024-09-11T07:51:29.202 INFO:teuthology.orchestra.run.smithi150.stdout:Vendor : CentOS 2024-09-11T07:51:29.203 INFO:teuthology.orchestra.run.smithi150.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-11T07:51:29.203 INFO:teuthology.orchestra.run.smithi150.stdout:Summary : Support tools for GRUB. 2024-09-11T07:51:29.203 INFO:teuthology.orchestra.run.smithi150.stdout:Description : 2024-09-11T07:51:29.203 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:51:29.203 INFO:teuthology.orchestra.run.smithi150.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-11T07:51:29.203 INFO:teuthology.orchestra.run.smithi150.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-11T07:51:29.203 INFO:teuthology.orchestra.run.smithi150.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-11T07:51:29.203 INFO:teuthology.orchestra.run.smithi150.stdout:hardware devices. 2024-09-11T07:51:29.203 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:51:29.203 INFO:teuthology.orchestra.run.smithi150.stdout:This subpackage provides tools for support of all platforms. 2024-09-11T07:51:29.204 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-11T07:51:29.204 INFO:teuthology.task.kernel:Updating grub on smithi150 to boot 5.14.0-505.el9.x86_64 2024-09-11T07:51:29.204 DEBUG:teuthology.orchestra.run.smithi150:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-11T07:51:29.864 INFO:teuthology.orchestra.run.smithi150.stderr:Generating grub configuration file ... 2024-09-11T07:51:31.230 INFO:teuthology.orchestra.run.smithi150.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-11T07:51:31.262 INFO:teuthology.orchestra.run.smithi150.stderr:done 2024-09-11T07:51:31.265 DEBUG:teuthology.orchestra.run.smithi150:> mktemp 2024-09-11T07:51:31.281 INFO:teuthology.orchestra.run.smithi150.stdout:/tmp/tmp.mkIVOAbwyB 2024-09-11T07:51:31.281 DEBUG:teuthology.orchestra.run.smithi150:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.mkIVOAbwyB 2024-09-11T07:51:31.349 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 0666 /tmp/tmp.mkIVOAbwyB 2024-09-11T07:51:31.485 DEBUG:teuthology.orchestra.remote:smithi150:/tmp/tmp.mkIVOAbwyB is 6KB 2024-09-11T07:51:31.536 DEBUG:teuthology.orchestra.run.smithi150:> rm -fr /tmp/tmp.mkIVOAbwyB 2024-09-11T07:51:31.552 DEBUG:teuthology.orchestra.run.smithi150:> sudo /bin/ls /boot/loader/entries || true 2024-09-11T07:51:31.615 INFO:teuthology.orchestra.run.smithi150.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-11T07:51:31.615 INFO:teuthology.orchestra.run.smithi150.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-11T07:51:31.615 INFO:teuthology.orchestra.run.smithi150.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-11T07:51:31.615 INFO:teuthology.orchestra.run.smithi150.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-11T07:51:31.616 DEBUG:teuthology.orchestra.run.smithi150:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-11T07:51:31.759 DEBUG:teuthology.orchestra.run.smithi150:> sudo shutdown -r now 2024-09-11T07:51:33.859 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T07:51:33.899 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-11T07:51:34.262 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-11T07:51:34.262 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-11T07:51:34.262 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T07:51:34.262 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T07:51:35.120 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T07:51:35.120 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:51:35.120 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-11T07:51:35.120 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-11T07:51:35.120 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-11T07:51:35.120 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-11T07:51:35.120 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-11T07:51:35.121 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:51:35.121 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-11T07:51:35.455 DEBUG:teuthology.orchestra.run.smithi181:> rpm -q kernel | sort -rV | head -n 1 2024-09-11T07:51:35.541 INFO:teuthology.orchestra.run.smithi181.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-11T07:51:35.542 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-11T07:51:35.542 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-11T07:51:35.542 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-11T07:51:35.542 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-11T07:51:35.543 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-11T07:51:35.543 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi181.front.sepia.ceph.com, path=None, version=distro) 2024-09-11T07:51:35.543 DEBUG:teuthology.orchestra.run.smithi181:> rpm -q kernel | sort -rV | head -n 1 2024-09-11T07:51:35.601 INFO:teuthology.orchestra.run.smithi181.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-11T07:51:35.602 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-11T07:51:35.602 DEBUG:teuthology.orchestra.run.smithi181:> sudo rpm -qi grub2-tools 2024-09-11T07:51:35.637 INFO:teuthology.orchestra.run.smithi181.stdout:Name : grub2-tools 2024-09-11T07:51:35.637 INFO:teuthology.orchestra.run.smithi181.stdout:Epoch : 1 2024-09-11T07:51:35.637 INFO:teuthology.orchestra.run.smithi181.stdout:Version : 2.06 2024-09-11T07:51:35.637 INFO:teuthology.orchestra.run.smithi181.stdout:Release : 82.el9 2024-09-11T07:51:35.637 INFO:teuthology.orchestra.run.smithi181.stdout:Architecture: x86_64 2024-09-11T07:51:35.637 INFO:teuthology.orchestra.run.smithi181.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-11T07:51:35.637 INFO:teuthology.orchestra.run.smithi181.stdout:Group : Unspecified 2024-09-11T07:51:35.637 INFO:teuthology.orchestra.run.smithi181.stdout:Size : 8274670 2024-09-11T07:51:35.638 INFO:teuthology.orchestra.run.smithi181.stdout:License : GPLv3+ 2024-09-11T07:51:35.638 INFO:teuthology.orchestra.run.smithi181.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-11T07:51:35.638 INFO:teuthology.orchestra.run.smithi181.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-11T07:51:35.638 INFO:teuthology.orchestra.run.smithi181.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-11T07:51:35.638 INFO:teuthology.orchestra.run.smithi181.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-11T07:51:35.638 INFO:teuthology.orchestra.run.smithi181.stdout:Packager : builder@centos.org 2024-09-11T07:51:35.638 INFO:teuthology.orchestra.run.smithi181.stdout:Vendor : CentOS 2024-09-11T07:51:35.638 INFO:teuthology.orchestra.run.smithi181.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-11T07:51:35.638 INFO:teuthology.orchestra.run.smithi181.stdout:Summary : Support tools for GRUB. 2024-09-11T07:51:35.638 INFO:teuthology.orchestra.run.smithi181.stdout:Description : 2024-09-11T07:51:35.638 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:51:35.638 INFO:teuthology.orchestra.run.smithi181.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-11T07:51:35.638 INFO:teuthology.orchestra.run.smithi181.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-11T07:51:35.638 INFO:teuthology.orchestra.run.smithi181.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-11T07:51:35.639 INFO:teuthology.orchestra.run.smithi181.stdout:hardware devices. 2024-09-11T07:51:35.639 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:51:35.639 INFO:teuthology.orchestra.run.smithi181.stdout:This subpackage provides tools for support of all platforms. 2024-09-11T07:51:35.639 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-11T07:51:35.640 INFO:teuthology.task.kernel:Updating grub on smithi181 to boot 5.14.0-505.el9.x86_64 2024-09-11T07:51:35.640 DEBUG:teuthology.orchestra.run.smithi181:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-11T07:51:36.271 INFO:teuthology.orchestra.run.smithi181.stderr:Generating grub configuration file ... 2024-09-11T07:51:37.560 INFO:teuthology.orchestra.run.smithi181.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-11T07:51:37.605 INFO:teuthology.orchestra.run.smithi181.stderr:done 2024-09-11T07:51:37.608 DEBUG:teuthology.orchestra.run.smithi181:> mktemp 2024-09-11T07:51:37.644 INFO:teuthology.orchestra.run.smithi181.stdout:/tmp/tmp.4XWdVCTJzo 2024-09-11T07:51:37.644 DEBUG:teuthology.orchestra.run.smithi181:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.4XWdVCTJzo 2024-09-11T07:51:37.711 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 0666 /tmp/tmp.4XWdVCTJzo 2024-09-11T07:51:37.849 DEBUG:teuthology.orchestra.remote:smithi181:/tmp/tmp.4XWdVCTJzo is 6KB 2024-09-11T07:51:37.898 DEBUG:teuthology.orchestra.run.smithi181:> rm -fr /tmp/tmp.4XWdVCTJzo 2024-09-11T07:51:37.913 DEBUG:teuthology.orchestra.run.smithi181:> sudo /bin/ls /boot/loader/entries || true 2024-09-11T07:51:37.975 INFO:teuthology.orchestra.run.smithi181.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-11T07:51:37.976 INFO:teuthology.orchestra.run.smithi181.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-11T07:51:37.976 INFO:teuthology.orchestra.run.smithi181.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-11T07:51:37.976 INFO:teuthology.orchestra.run.smithi181.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-11T07:51:37.977 DEBUG:teuthology.orchestra.run.smithi181:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-11T07:51:38.115 DEBUG:teuthology.orchestra.run.smithi181:> sudo shutdown -r now 2024-09-11T07:52:01.785 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-11T07:52:01.786 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-09-11T07:52:01.787 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T07:52:08.118 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-11T07:52:08.119 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-09-11T07:52:08.119 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T07:52:36.417 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-09-11T07:52:45.425 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-09-11T07:52:45.426 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T07:53:03.805 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-09-11T07:53:08.128 DEBUG:teuthology.orchestra.remote:timed out 2024-09-11T07:53:15.813 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-09-11T07:53:15.813 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T07:53:16.308 DEBUG:teuthology.orchestra.run.smithi150:> true 2024-09-11T07:53:16.847 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-09-11T07:53:16.847 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-11T07:53:16.848 DEBUG:teuthology.orchestra.run.smithi150:> uname -r 2024-09-11T07:53:16.862 INFO:teuthology.orchestra.run.smithi150.stdout:5.14.0-505.el9.x86_64 2024-09-11T07:53:16.862 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-11T07:53:16.863 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-11T07:53:16.863 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-11T07:53:17.129 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-09-11T07:53:17.129 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T07:53:17.864 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-11T07:53:17.864 DEBUG:teuthology.orchestra.run.smithi150:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-11T07:53:17.966 INFO:teuthology.orchestra.run.smithi150.stdout:ttyS1 2024-09-11T07:53:17.985 DEBUG:teuthology.parallel:result is None 2024-09-11T07:53:18.551 DEBUG:teuthology.orchestra.run.smithi181:> true 2024-09-11T07:53:19.032 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-09-11T07:53:19.032 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-505.el9.x86_64"... 2024-09-11T07:53:19.032 DEBUG:teuthology.orchestra.run.smithi181:> uname -r 2024-09-11T07:53:19.047 INFO:teuthology.orchestra.run.smithi181.stdout:5.14.0-505.el9.x86_64 2024-09-11T07:53:19.047 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-11T07:53:19.047 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-11T07:53:19.047 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-11T07:53:20.048 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-11T07:53:20.048 DEBUG:teuthology.orchestra.run.smithi181:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-11T07:53:20.133 INFO:teuthology.orchestra.run.smithi181.stdout:ttyS1 2024-09-11T07:53:20.156 DEBUG:teuthology.parallel:result is None 2024-09-11T07:53:20.157 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-11T07:53:20.164 INFO:teuthology.task.internal:Creating test directory... 2024-09-11T07:53:20.165 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-11T07:53:20.167 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-11T07:53:20.182 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-11T07:53:20.188 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-11T07:53:20.220 INFO:teuthology.task.internal:Creating archive directory... 2024-09-11T07:53:20.220 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-11T07:53:20.227 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-11T07:53:20.270 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-11T07:53:20.277 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-11T07:53:20.277 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-11T07:53:20.287 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-11T07:53:20.317 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T07:53:20.327 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T07:53:20.336 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T07:53:20.346 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T07:53:20.347 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-11T07:53:20.355 INFO:teuthology.task.internal:Configuring sudo... 2024-09-11T07:53:20.355 DEBUG:teuthology.orchestra.run.smithi150:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-11T07:53:20.374 DEBUG:teuthology.orchestra.run.smithi181:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-11T07:53:20.413 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-11T07:53:20.422 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-11T07:53:20.422 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-11T07:53:20.443 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-11T07:53:20.469 DEBUG:teuthology.orchestra.run.smithi150:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T07:53:20.541 DEBUG:teuthology.orchestra.run.smithi150:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T07:53:20.591 DEBUG:teuthology.orchestra.run.smithi150:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T07:53:20.676 DEBUG:teuthology.orchestra.run.smithi150:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T07:53:20.720 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-11T07:53:20.720 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-11T07:53:20.788 DEBUG:teuthology.orchestra.run.smithi181:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T07:53:20.832 DEBUG:teuthology.orchestra.run.smithi181:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T07:53:20.880 DEBUG:teuthology.orchestra.run.smithi181:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T07:53:20.967 DEBUG:teuthology.orchestra.run.smithi181:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T07:53:21.002 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-11T07:53:21.002 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-11T07:53:21.068 DEBUG:teuthology.orchestra.run.smithi150:> sudo service rsyslog restart 2024-09-11T07:53:21.071 DEBUG:teuthology.orchestra.run.smithi181:> sudo service rsyslog restart 2024-09-11T07:53:21.116 INFO:teuthology.orchestra.run.smithi150.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-11T07:53:21.163 INFO:teuthology.orchestra.run.smithi181.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-11T07:53:21.353 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-11T07:53:21.361 INFO:teuthology.task.internal:Starting timer... 2024-09-11T07:53:21.361 INFO:teuthology.run_tasks:Running task pcp... 2024-09-11T07:53:21.377 INFO:teuthology.run_tasks:Running task selinux... 2024-09-11T07:53:21.387 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-09-11T07:53:21.387 DEBUG:teuthology.orchestra.run.smithi150:> sudo service auditd rotate 2024-09-11T07:53:21.429 INFO:teuthology.orchestra.run.smithi150.stdout:Rotating logs: 2024-09-11T07:53:21.431 DEBUG:teuthology.orchestra.run.smithi181:> sudo service auditd rotate 2024-09-11T07:53:21.496 INFO:teuthology.orchestra.run.smithi181.stdout:Rotating logs: 2024-09-11T07:53:21.497 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-11T07:53:21.498 DEBUG:teuthology.orchestra.run.smithi150:> /usr/sbin/getenforce 2024-09-11T07:53:21.534 INFO:teuthology.orchestra.run.smithi150.stdout:Permissive 2024-09-11T07:53:21.534 DEBUG:teuthology.orchestra.run.smithi181:> /usr/sbin/getenforce 2024-09-11T07:53:21.558 INFO:teuthology.orchestra.run.smithi181.stdout:Permissive 2024-09-11T07:53:21.559 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi150.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi181.front.sepia.ceph.com': 'permissive'} 2024-09-11T07:53:21.559 DEBUG:teuthology.orchestra.run.smithi150:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-11T07:53:21.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T07:53:21.599 DEBUG:teuthology.orchestra.run.smithi181:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-11T07:53:21.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T07:53:21.624 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-11T07:53:21.624 DEBUG:teuthology.orchestra.run.smithi150:> sudo /usr/sbin/setenforce permissive 2024-09-11T07:53:21.681 DEBUG:teuthology.orchestra.run.smithi181:> sudo /usr/sbin/setenforce permissive 2024-09-11T07:53:21.732 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-11T07:53:21.741 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-11T07:53:21.754 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-11T07:53:21.755 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi150.front.sepia.ceph.com,smithi181.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-11T07:57:07.924 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi150.front.sepia.ceph.com'), Remote(name='ubuntu@smithi181.front.sepia.ceph.com')] 2024-09-11T07:57:07.926 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-09-11T07:57:07.928 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T07:57:08.015 DEBUG:teuthology.orchestra.run.smithi150:> true 2024-09-11T07:57:08.092 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-09-11T07:57:08.092 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-09-11T07:57:08.093 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T07:57:08.173 DEBUG:teuthology.orchestra.run.smithi181:> true 2024-09-11T07:57:08.244 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-09-11T07:57:08.244 INFO:teuthology.run_tasks:Running task clock... 2024-09-11T07:57:08.254 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-11T07:57:08.255 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-11T07:57:08.255 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T07:57:08.258 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-11T07:57:08.258 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T07:57:08.285 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-11T07:57:08.299 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-11T07:57:08.314 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-11T07:57:08.324 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: ntpd: command not found 2024-09-11T07:57:08.328 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-11T07:57:08.334 INFO:teuthology.orchestra.run.smithi150.stdout:506 Cannot talk to daemon 2024-09-11T07:57:08.349 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-11T07:57:08.353 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: ntpd: command not found 2024-09-11T07:57:08.363 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-11T07:57:08.364 INFO:teuthology.orchestra.run.smithi181.stdout:506 Cannot talk to daemon 2024-09-11T07:57:08.378 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-11T07:57:08.392 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-11T07:57:08.416 INFO:teuthology.orchestra.run.smithi150.stderr:bash: line 1: ntpq: command not found 2024-09-11T07:57:08.419 INFO:teuthology.orchestra.run.smithi150.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-11T07:57:08.419 INFO:teuthology.orchestra.run.smithi150.stdout:=============================================================================== 2024-09-11T07:57:08.419 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T07:57:08.419 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T07:57:08.420 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T07:57:08.420 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T07:57:08.447 INFO:teuthology.orchestra.run.smithi181.stderr:bash: line 1: ntpq: command not found 2024-09-11T07:57:08.450 INFO:teuthology.orchestra.run.smithi181.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-11T07:57:08.450 INFO:teuthology.orchestra.run.smithi181.stdout:=============================================================================== 2024-09-11T07:57:08.450 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T07:57:08.450 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T07:57:08.450 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T07:57:08.450 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T07:57:08.451 INFO:teuthology.run_tasks:Running task pexec... 2024-09-11T07:57:08.467 INFO:teuthology.task.pexec:Executing custom commands... 2024-09-11T07:57:08.468 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi150.front.sepia.ceph.com 2024-09-11T07:57:08.468 DEBUG:teuthology.orchestra.run.smithi150:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-11T07:57:08.469 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi181.front.sepia.ceph.com 2024-09-11T07:57:08.469 DEBUG:teuthology.orchestra.run.smithi181:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-11T07:57:08.821 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-09-11T07:57:08.835 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-09-11T07:57:08.835 INFO:teuthology.orchestra.run.smithi150.stdout: Package Architecture Version Repository Size 2024-09-11T07:57:08.835 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-09-11T07:57:08.835 INFO:teuthology.orchestra.run.smithi150.stdout:Removing: 2024-09-11T07:57:08.835 INFO:teuthology.orchestra.run.smithi150.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-11T07:57:08.835 INFO:teuthology.orchestra.run.smithi150.stdout:Removing unused dependencies: 2024-09-11T07:57:08.835 INFO:teuthology.orchestra.run.smithi150.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-11T07:57:08.835 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:57:08.835 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-09-11T07:57:08.835 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-09-11T07:57:08.836 INFO:teuthology.orchestra.run.smithi150.stdout:Remove 2 Packages 2024-09-11T07:57:08.836 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:57:08.836 INFO:teuthology.orchestra.run.smithi150.stdout:Freed space: 5.5 M 2024-09-11T07:57:08.836 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-09-11T07:57:08.839 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-09-11T07:57:08.839 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-09-11T07:57:08.844 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-11T07:57:08.853 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T07:57:08.853 INFO:teuthology.orchestra.run.smithi181.stdout: Package Architecture Version Repository Size 2024-09-11T07:57:08.853 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T07:57:08.853 INFO:teuthology.orchestra.run.smithi181.stdout:Removing: 2024-09-11T07:57:08.853 INFO:teuthology.orchestra.run.smithi181.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-11T07:57:08.853 INFO:teuthology.orchestra.run.smithi181.stdout:Removing unused dependencies: 2024-09-11T07:57:08.853 INFO:teuthology.orchestra.run.smithi181.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-11T07:57:08.853 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:57:08.853 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-11T07:57:08.853 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T07:57:08.853 INFO:teuthology.orchestra.run.smithi181.stdout:Remove 2 Packages 2024-09-11T07:57:08.854 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:57:08.854 INFO:teuthology.orchestra.run.smithi181.stdout:Freed space: 5.5 M 2024-09-11T07:57:08.854 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-11T07:57:08.857 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-11T07:57:08.857 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-11T07:57:08.896 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-09-11T07:57:08.896 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-09-11T07:57:08.919 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-11T07:57:08.919 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-11T07:57:09.031 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-09-11T07:57:09.094 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-11T07:57:09.118 INFO:teuthology.orchestra.run.smithi150.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-11T07:57:09.172 INFO:teuthology.orchestra.run.smithi150.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-11T07:57:09.177 INFO:teuthology.orchestra.run.smithi181.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-11T07:57:09.230 INFO:teuthology.orchestra.run.smithi181.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-11T07:57:09.677 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-11T07:57:09.677 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-11T07:57:09.763 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-11T07:57:09.763 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-11T07:57:09.983 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-11T07:57:09.983 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:57:09.983 INFO:teuthology.orchestra.run.smithi150.stdout:Removed: 2024-09-11T07:57:09.983 INFO:teuthology.orchestra.run.smithi150.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-11T07:57:09.983 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:57:09.983 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-09-11T07:57:10.065 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-11T07:57:10.065 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:57:10.065 INFO:teuthology.orchestra.run.smithi181.stdout:Removed: 2024-09-11T07:57:10.065 INFO:teuthology.orchestra.run.smithi181.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-11T07:57:10.065 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:57:10.065 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-11T07:57:10.792 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:00:51 ago on Wed 11 Sep 2024 07:56:19 AM UTC. 2024-09-11T07:57:10.868 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:51 ago on Wed 11 Sep 2024 07:56:19 AM UTC. 2024-09-11T07:57:10.924 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-09-11T07:57:10.925 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-09-11T07:57:10.925 INFO:teuthology.orchestra.run.smithi150.stdout: Package Architecture Version Repository Size 2024-09-11T07:57:10.925 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-09-11T07:57:10.925 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-09-11T07:57:10.925 INFO:teuthology.orchestra.run.smithi150.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-11T07:57:10.925 INFO:teuthology.orchestra.run.smithi150.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-11T07:57:10.926 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2024-09-11T07:57:10.926 INFO:teuthology.orchestra.run.smithi150.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-11T07:57:10.926 INFO:teuthology.orchestra.run.smithi150.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-11T07:57:10.926 INFO:teuthology.orchestra.run.smithi150.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-11T07:57:10.926 INFO:teuthology.orchestra.run.smithi150.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-11T07:57:10.926 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:57:10.926 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-09-11T07:57:10.926 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-09-11T07:57:10.926 INFO:teuthology.orchestra.run.smithi150.stdout:Install 6 Packages 2024-09-11T07:57:10.926 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:57:10.936 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 2.0 M 2024-09-11T07:57:10.936 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 9.2 M 2024-09-11T07:57:10.936 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-09-11T07:57:11.003 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-11T07:57:11.004 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T07:57:11.004 INFO:teuthology.orchestra.run.smithi181.stdout: Package Architecture Version Repository Size 2024-09-11T07:57:11.004 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T07:57:11.004 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-11T07:57:11.004 INFO:teuthology.orchestra.run.smithi181.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-11T07:57:11.004 INFO:teuthology.orchestra.run.smithi181.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-11T07:57:11.004 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-11T07:57:11.004 INFO:teuthology.orchestra.run.smithi181.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-11T07:57:11.004 INFO:teuthology.orchestra.run.smithi181.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-11T07:57:11.004 INFO:teuthology.orchestra.run.smithi181.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-11T07:57:11.004 INFO:teuthology.orchestra.run.smithi181.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-11T07:57:11.005 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:57:11.005 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-11T07:57:11.005 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T07:57:11.005 INFO:teuthology.orchestra.run.smithi181.stdout:Install 6 Packages 2024-09-11T07:57:11.005 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:57:11.014 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 2.0 M 2024-09-11T07:57:11.014 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 9.2 M 2024-09-11T07:57:11.014 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-11T07:57:11.170 INFO:teuthology.orchestra.run.smithi181.stdout:(1/6): nvmetcli-0.7-3.el9.noarch.rpm 418 kB/s | 44 kB 00:00 2024-09-11T07:57:11.195 INFO:teuthology.orchestra.run.smithi181.stdout:(2/6): libnvme-1.9-3.el9.x86_64.rpm 804 kB/s | 106 kB 00:00 2024-09-11T07:57:11.254 INFO:teuthology.orchestra.run.smithi181.stdout:(3/6): python3-configshell-1.1.30-1.el9.noarch. 854 kB/s | 72 kB 00:00 2024-09-11T07:57:11.279 INFO:teuthology.orchestra.run.smithi181.stdout:(4/6): python3-kmod-0.9-32.el9.x86_64.rpm 1.0 MB/s | 84 kB 00:00 2024-09-11T07:57:11.321 INFO:teuthology.orchestra.run.smithi181.stdout:(5/6): nvme-cli-2.9.1-6.el9.x86_64.rpm 3.4 MB/s | 906 kB 00:00 2024-09-11T07:57:11.423 INFO:teuthology.orchestra.run.smithi150.stdout:(1/6): nvmetcli-0.7-3.el9.noarch.rpm 116 kB/s | 44 kB 00:00 2024-09-11T07:57:11.438 INFO:teuthology.orchestra.run.smithi181.stdout:(6/6): python3-urwid-2.1.2-4.el9.x86_64.rpm 4.4 MB/s | 837 kB 00:00 2024-09-11T07:57:11.438 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-11T07:57:11.438 INFO:teuthology.orchestra.run.smithi181.stdout:Total 4.7 MB/s | 2.0 MB 00:00 2024-09-11T07:57:11.498 INFO:teuthology.orchestra.run.smithi150.stdout:(2/6): libnvme-1.9-3.el9.x86_64.rpm 231 kB/s | 106 kB 00:00 2024-09-11T07:57:11.511 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-11T07:57:11.525 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-11T07:57:11.525 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-11T07:57:11.715 INFO:teuthology.orchestra.run.smithi150.stdout:(3/6): python3-configshell-1.1.30-1.el9.noarch. 246 kB/s | 72 kB 00:00 2024-09-11T07:57:11.740 INFO:teuthology.orchestra.run.smithi150.stdout:(4/6): python3-kmod-0.9-32.el9.x86_64.rpm 348 kB/s | 84 kB 00:00 2024-09-11T07:57:11.782 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-11T07:57:11.783 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-11T07:57:12.116 INFO:teuthology.orchestra.run.smithi150.stdout:(5/6): python3-urwid-2.1.2-4.el9.x86_64.rpm 2.0 MB/s | 837 kB 00:00 2024-09-11T07:57:12.203 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-11T07:57:12.216 INFO:teuthology.orchestra.run.smithi150.stdout:(6/6): nvme-cli-2.9.1-6.el9.x86_64.rpm 769 kB/s | 906 kB 00:01 2024-09-11T07:57:12.216 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-09-11T07:57:12.216 INFO:teuthology.orchestra.run.smithi150.stdout:Total 1.6 MB/s | 2.0 MB 00:01 2024-09-11T07:57:12.291 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/6 2024-09-11T07:57:12.291 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-09-11T07:57:12.304 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-09-11T07:57:12.304 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-09-11T07:57:12.325 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/6 2024-09-11T07:57:12.354 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/6 2024-09-11T07:57:12.564 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/6 2024-09-11T07:57:12.579 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-09-11T07:57:12.580 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-09-11T07:57:12.593 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-11T07:57:12.999 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-09-11T07:57:13.035 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-11T07:57:13.063 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/6 2024-09-11T07:57:13.094 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/6 2024-09-11T07:57:13.136 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/6 2024-09-11T07:57:13.165 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/6 2024-09-11T07:57:13.390 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/6 2024-09-11T07:57:13.421 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-11T07:57:13.857 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-11T07:57:13.885 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/6 2024-09-11T07:57:14.082 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/6 2024-09-11T07:57:14.082 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/6 2024-09-11T07:57:14.082 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/6 2024-09-11T07:57:14.083 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/6 2024-09-11T07:57:14.083 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/6 2024-09-11T07:57:14.083 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/6 2024-09-11T07:57:14.441 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/6 2024-09-11T07:57:14.442 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:57:14.442 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-11T07:57:14.442 INFO:teuthology.orchestra.run.smithi181.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-11T07:57:14.442 INFO:teuthology.orchestra.run.smithi181.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-11T07:57:14.442 INFO:teuthology.orchestra.run.smithi181.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-11T07:57:14.442 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:57:14.442 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-11T07:57:14.784 DEBUG:teuthology.parallel:result is None 2024-09-11T07:57:14.826 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/6 2024-09-11T07:57:14.826 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/6 2024-09-11T07:57:14.826 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/6 2024-09-11T07:57:14.826 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/6 2024-09-11T07:57:14.826 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/6 2024-09-11T07:57:14.826 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/6 2024-09-11T07:57:15.237 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/6 2024-09-11T07:57:15.238 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:57:15.238 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-09-11T07:57:15.238 INFO:teuthology.orchestra.run.smithi150.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-11T07:57:15.238 INFO:teuthology.orchestra.run.smithi150.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-11T07:57:15.238 INFO:teuthology.orchestra.run.smithi150.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-11T07:57:15.238 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:57:15.238 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-09-11T07:57:15.563 DEBUG:teuthology.parallel:result is None 2024-09-11T07:57:15.563 INFO:teuthology.run_tasks:Running task install... 2024-09-11T07:57:15.572 DEBUG:teuthology.task.install:project ceph 2024-09-11T07:57:15.572 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd0e6828a2016d48cf25ad84064e50742bb1c39b9'}} 2024-09-11T07:57:15.572 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'd0e6828a2016d48cf25ad84064e50742bb1c39b9'} 2024-09-11T07:57:15.572 INFO:teuthology.task.install:Using flavor: default 2024-09-11T07:57:15.578 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-11T07:57:15.578 INFO:teuthology.task.install:extra packages: [] 2024-09-11T07:57:15.578 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd0e6828a2016d48cf25ad84064e50742bb1c39b9', 'tag': None, 'wait_for_package': False} 2024-09-11T07:57:15.579 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d0e6828a2016d48cf25ad84064e50742bb1c39b9 2024-09-11T07:57:15.581 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd0e6828a2016d48cf25ad84064e50742bb1c39b9', 'tag': None, 'wait_for_package': False} 2024-09-11T07:57:15.581 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d0e6828a2016d48cf25ad84064e50742bb1c39b9 2024-09-11T07:57:15.729 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-10-2008-quincy/d0e6828a2016d48cf25ad84064e50742bb1c39b9/centos/9/flavors/default/ 2024-09-11T07:57:15.729 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1623.gd0e6828a 2024-09-11T07:57:15.738 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-10-2008-quincy/d0e6828a2016d48cf25ad84064e50742bb1c39b9/centos/9/flavors/default/ 2024-09-11T07:57:15.738 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1623.gd0e6828a 2024-09-11T07:57:15.838 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-10-2008-quincy/d0e6828a2016d48cf25ad84064e50742bb1c39b9/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-10-2008-quincy/d0e6828a2016d48cf25ad84064e50742bb1c39b9/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-10-2008-quincy/d0e6828a2016d48cf25ad84064e50742bb1c39b9/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-11T07:57:15.838 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-11T07:57:15.838 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-11T07:57:15.847 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-10-2008-quincy/d0e6828a2016d48cf25ad84064e50742bb1c39b9/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-10-2008-quincy/d0e6828a2016d48cf25ad84064e50742bb1c39b9/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-10-2008-quincy/d0e6828a2016d48cf25ad84064e50742bb1c39b9/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-11T07:57:15.847 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-11T07:57:15.848 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-11T07:57:15.866 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-11T07:57:15.866 DEBUG:teuthology.orchestra.run.smithi150:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d0e6828a2016d48cf25ad84064e50742bb1c39b9/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-11T07:57:15.881 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-11T07:57:15.881 DEBUG:teuthology.orchestra.run.smithi181:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d0e6828a2016d48cf25ad84064e50742bb1c39b9/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-11T07:57:15.932 DEBUG:teuthology.orchestra.run.smithi150:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-11T07:57:15.952 DEBUG:teuthology.orchestra.run.smithi181:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-11T07:57:16.007 DEBUG:teuthology.orchestra.run.smithi150:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-11T07:57:16.033 DEBUG:teuthology.orchestra.run.smithi181:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-11T07:57:16.072 INFO:teuthology.orchestra.run.smithi150.stdout:check_obsoletes = 1 2024-09-11T07:57:16.074 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum clean all 2024-09-11T07:57:16.101 INFO:teuthology.orchestra.run.smithi181.stdout:check_obsoletes = 1 2024-09-11T07:57:16.103 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum clean all 2024-09-11T07:57:16.298 INFO:teuthology.orchestra.run.smithi150.stdout:45 files removed 2024-09-11T07:57:16.325 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-radosgw 2024-09-11T07:57:16.336 INFO:teuthology.orchestra.run.smithi181.stdout:45 files removed 2024-09-11T07:57:16.361 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-radosgw 2024-09-11T07:57:16.880 INFO:teuthology.orchestra.run.smithi150.stdout:ceph packages for x86_64 285 kB/s | 77 kB 00:00 2024-09-11T07:57:16.911 INFO:teuthology.orchestra.run.smithi181.stdout:ceph packages for x86_64 295 kB/s | 77 kB 00:00 2024-09-11T07:57:17.173 INFO:teuthology.orchestra.run.smithi150.stdout:ceph noarch packages 43 kB/s | 11 kB 00:00 2024-09-11T07:57:17.177 INFO:teuthology.orchestra.run.smithi181.stdout:ceph noarch packages 48 kB/s | 11 kB 00:00 2024-09-11T07:57:17.436 INFO:teuthology.orchestra.run.smithi181.stdout:ceph source packages 8.0 kB/s | 1.8 kB 00:00 2024-09-11T07:57:17.440 INFO:teuthology.orchestra.run.smithi150.stdout:ceph source packages 7.7 kB/s | 1.8 kB 00:00 2024-09-11T07:57:18.148 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.2 MB 00:00 2024-09-11T07:57:19.328 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - BaseOS 4.4 MB/s | 8.2 MB 00:01 2024-09-11T07:57:21.038 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - AppStream 34 MB/s | 20 MB 00:00 2024-09-11T07:57:22.166 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - AppStream 7.0 MB/s | 20 MB 00:02 2024-09-11T07:57:25.539 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - CRB 9.5 MB/s | 6.5 MB 00:00 2024-09-11T07:57:27.306 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 19 kB 00:00 2024-09-11T07:57:28.261 INFO:teuthology.orchestra.run.smithi181.stdout:Extra Packages for Enterprise Linux 27 MB/s | 23 MB 00:00 2024-09-11T07:57:29.000 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - CRB 2.2 MB/s | 6.5 MB 00:03 2024-09-11T07:57:30.659 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - Extras packages 54 kB/s | 19 kB 00:00 2024-09-11T07:57:31.501 INFO:teuthology.orchestra.run.smithi150.stdout:Extra Packages for Enterprise Linux 32 MB/s | 23 MB 00:00 2024-09-11T07:57:33.766 INFO:teuthology.orchestra.run.smithi181.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-11T07:57:36.139 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-11T07:57:36.140 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T07:57:36.140 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-11T07:57:36.141 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T07:57:36.141 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-11T07:57:36.141 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-radosgw x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 14 M 2024-09-11T07:57:36.141 INFO:teuthology.orchestra.run.smithi181.stdout:Upgrading: 2024-09-11T07:57:36.141 INFO:teuthology.orchestra.run.smithi181.stdout: librados2 x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 3.1 M 2024-09-11T07:57:36.141 INFO:teuthology.orchestra.run.smithi181.stdout: librbd1 x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 2.9 M 2024-09-11T07:57:36.141 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-11T07:57:36.141 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-base x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 5.0 M 2024-09-11T07:57:36.141 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-common x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 23 M 2024-09-11T07:57:36.141 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-selinux x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 25 k 2024-09-11T07:57:36.141 INFO:teuthology.orchestra.run.smithi181.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-11T07:57:36.141 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs2 x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 636 k 2024-09-11T07:57:36.142 INFO:teuthology.orchestra.run.smithi181.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-11T07:57:36.142 INFO:teuthology.orchestra.run.smithi181.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-11T07:57:36.142 INFO:teuthology.orchestra.run.smithi181.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-11T07:57:36.142 INFO:teuthology.orchestra.run.smithi181.stdout: libradosstriper1 x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 457 k 2024-09-11T07:57:36.142 INFO:teuthology.orchestra.run.smithi181.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-11T07:57:36.142 INFO:teuthology.orchestra.run.smithi181.stdout: librgw2 x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 4.9 M 2024-09-11T07:57:36.142 INFO:teuthology.orchestra.run.smithi181.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-11T07:57:36.142 INFO:teuthology.orchestra.run.smithi181.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-11T07:57:36.142 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-argparse x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 45 k 2024-09-11T07:57:36.142 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-common x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 120 k 2024-09-11T07:57:36.142 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cephfs x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 156 k 2024-09-11T07:57:36.142 INFO:teuthology.orchestra.run.smithi181.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-11T07:57:36.142 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rados x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 320 k 2024-09-11T07:57:36.143 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rbd x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 294 k 2024-09-11T07:57:36.143 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rgw x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 99 k 2024-09-11T07:57:36.143 INFO:teuthology.orchestra.run.smithi181.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-11T07:57:36.143 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:57:36.143 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-11T07:57:36.143 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T07:57:36.143 INFO:teuthology.orchestra.run.smithi181.stdout:Install 22 Packages 2024-09-11T07:57:36.143 INFO:teuthology.orchestra.run.smithi181.stdout:Upgrade 2 Packages 2024-09-11T07:57:36.143 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:57:36.143 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 58 M 2024-09-11T07:57:36.144 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-11T07:57:36.786 INFO:teuthology.orchestra.run.smithi150.stdout:lab-extras 50 kB/s | 1.7 kB 00:00 2024-09-11T07:57:37.159 INFO:teuthology.orchestra.run.smithi181.stdout:(1/24): ceph-base-17.2.7-1623.gd0e6828a.el9.x86 10 MB/s | 5.0 MB 00:00 2024-09-11T07:57:37.210 INFO:teuthology.orchestra.run.smithi181.stdout:(2/24): ceph-selinux-17.2.7-1623.gd0e6828a.el9. 505 kB/s | 25 kB 00:00 2024-09-11T07:57:37.310 INFO:teuthology.orchestra.run.smithi181.stdout:(3/24): libcephfs2-17.2.7-1623.gd0e6828a.el9.x8 6.2 MB/s | 636 kB 00:00 2024-09-11T07:57:37.410 INFO:teuthology.orchestra.run.smithi181.stdout:(4/24): libradosstriper1-17.2.7-1623.gd0e6828a. 4.5 MB/s | 457 kB 00:00 2024-09-11T07:57:37.602 INFO:teuthology.orchestra.run.smithi181.stdout:(5/24): ceph-radosgw-17.2.7-1623.gd0e6828a.el9. 15 MB/s | 14 MB 00:00 2024-09-11T07:57:37.652 INFO:teuthology.orchestra.run.smithi181.stdout:(6/24): python3-ceph-argparse-17.2.7-1623.gd0e6 907 kB/s | 45 kB 00:00 2024-09-11T07:57:37.728 INFO:teuthology.orchestra.run.smithi181.stdout:(7/24): librgw2-17.2.7-1623.gd0e6828a.el9.x86_6 16 MB/s | 4.9 MB 00:00 2024-09-11T07:57:37.753 INFO:teuthology.orchestra.run.smithi181.stdout:(8/24): python3-ceph-common-17.2.7-1623.gd0e682 1.2 MB/s | 120 kB 00:00 2024-09-11T07:57:37.778 INFO:teuthology.orchestra.run.smithi181.stdout:(9/24): python3-cephfs-17.2.7-1623.gd0e6828a.el 3.0 MB/s | 156 kB 00:00 2024-09-11T07:57:37.803 INFO:teuthology.orchestra.run.smithi181.stdout:(10/24): python3-rados-17.2.7-1623.gd0e6828a.el 6.2 MB/s | 320 kB 00:00 2024-09-11T07:57:37.829 INFO:teuthology.orchestra.run.smithi181.stdout:(11/24): python3-rbd-17.2.7-1623.gd0e6828a.el9. 5.7 MB/s | 294 kB 00:00 2024-09-11T07:57:37.854 INFO:teuthology.orchestra.run.smithi181.stdout:(12/24): python3-rgw-17.2.7-1623.gd0e6828a.el9. 1.9 MB/s | 99 kB 00:00 2024-09-11T07:57:38.071 INFO:teuthology.orchestra.run.smithi181.stdout:(13/24): ceph-common-17.2.7-1623.gd0e6828a.el9. 16 MB/s | 23 MB 00:01 2024-09-11T07:57:38.155 INFO:teuthology.orchestra.run.smithi181.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 151 kB/s | 45 kB 00:00 2024-09-11T07:57:38.180 INFO:teuthology.orchestra.run.smithi181.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 457 kB/s | 160 kB 00:00 2024-09-11T07:57:38.230 INFO:teuthology.orchestra.run.smithi181.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 845 kB/s | 42 kB 00:00 2024-09-11T07:57:38.281 INFO:teuthology.orchestra.run.smithi181.stdout:(17/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 6.0 MB/s | 309 kB 00:00 2024-09-11T07:57:38.314 INFO:teuthology.orchestra.run.smithi181.stdout:(18/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.8 MB/s | 292 kB 00:00 2024-09-11T07:57:38.356 INFO:teuthology.orchestra.run.smithi181.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 645 kB/s | 49 kB 00:00 2024-09-11T07:57:38.390 INFO:teuthology.orchestra.run.smithi181.stdout:(20/24): librdkafka-1.6.1-102.el9.x86_64.rpm 2.0 MB/s | 662 kB 00:00 2024-09-11T07:57:38.432 INFO:teuthology.orchestra.run.smithi181.stdout:(21/24): libunwind-1.6.2-1.el9.x86_64.rpm 576 kB/s | 67 kB 00:00 2024-09-11T07:57:38.482 INFO:teuthology.orchestra.run.smithi181.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-09-11T07:57:38.624 INFO:teuthology.orchestra.run.smithi181.stdout:(23/24): librados2-17.2.7-1623.gd0e6828a.el9.x8 13 MB/s | 3.1 MB 00:00 2024-09-11T07:57:38.691 INFO:teuthology.orchestra.run.smithi181.stdout:(24/24): librbd1-17.2.7-1623.gd0e6828a.el9.x86_ 11 MB/s | 2.9 MB 00:00 2024-09-11T07:57:38.696 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-11T07:57:38.696 INFO:teuthology.orchestra.run.smithi181.stdout:Total 23 MB/s | 58 MB 00:02 2024-09-11T07:57:38.762 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-11T07:57:38.796 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-11T07:57:38.796 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-11T07:57:39.144 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-09-11T07:57:39.145 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-09-11T07:57:39.145 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2024-09-11T07:57:39.146 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-09-11T07:57:39.146 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-09-11T07:57:39.146 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-radosgw x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 14 M 2024-09-11T07:57:39.146 INFO:teuthology.orchestra.run.smithi150.stdout:Upgrading: 2024-09-11T07:57:39.146 INFO:teuthology.orchestra.run.smithi150.stdout: librados2 x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 3.1 M 2024-09-11T07:57:39.146 INFO:teuthology.orchestra.run.smithi150.stdout: librbd1 x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 2.9 M 2024-09-11T07:57:39.146 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2024-09-11T07:57:39.146 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-base x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 5.0 M 2024-09-11T07:57:39.146 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-common x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 23 M 2024-09-11T07:57:39.146 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-selinux x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 25 k 2024-09-11T07:57:39.146 INFO:teuthology.orchestra.run.smithi150.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-11T07:57:39.147 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs2 x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 636 k 2024-09-11T07:57:39.147 INFO:teuthology.orchestra.run.smithi150.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-11T07:57:39.147 INFO:teuthology.orchestra.run.smithi150.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-11T07:57:39.147 INFO:teuthology.orchestra.run.smithi150.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-11T07:57:39.147 INFO:teuthology.orchestra.run.smithi150.stdout: libradosstriper1 x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 457 k 2024-09-11T07:57:39.147 INFO:teuthology.orchestra.run.smithi150.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-11T07:57:39.147 INFO:teuthology.orchestra.run.smithi150.stdout: librgw2 x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 4.9 M 2024-09-11T07:57:39.147 INFO:teuthology.orchestra.run.smithi150.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-11T07:57:39.147 INFO:teuthology.orchestra.run.smithi150.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-11T07:57:39.147 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-argparse x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 45 k 2024-09-11T07:57:39.147 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-common x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 120 k 2024-09-11T07:57:39.147 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cephfs x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 156 k 2024-09-11T07:57:39.147 INFO:teuthology.orchestra.run.smithi150.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-11T07:57:39.148 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rados x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 320 k 2024-09-11T07:57:39.148 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rbd x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 294 k 2024-09-11T07:57:39.148 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rgw x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 99 k 2024-09-11T07:57:39.148 INFO:teuthology.orchestra.run.smithi150.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-11T07:57:39.148 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:57:39.148 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-09-11T07:57:39.148 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-09-11T07:57:39.148 INFO:teuthology.orchestra.run.smithi150.stdout:Install 22 Packages 2024-09-11T07:57:39.148 INFO:teuthology.orchestra.run.smithi150.stdout:Upgrade 2 Packages 2024-09-11T07:57:39.148 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:57:39.149 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 58 M 2024-09-11T07:57:39.149 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-09-11T07:57:39.255 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-11T07:57:39.255 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-11T07:57:40.262 INFO:teuthology.orchestra.run.smithi150.stdout:(1/24): ceph-base-17.2.7-1623.gd0e6828a.el9.x86 8.0 MB/s | 5.0 MB 00:00 2024-09-11T07:57:40.312 INFO:teuthology.orchestra.run.smithi150.stdout:(2/24): ceph-selinux-17.2.7-1623.gd0e6828a.el9. 503 kB/s | 25 kB 00:00 2024-09-11T07:57:40.379 INFO:teuthology.orchestra.run.smithi150.stdout:(3/24): libcephfs2-17.2.7-1623.gd0e6828a.el9.x8 9.3 MB/s | 636 kB 00:00 2024-09-11T07:57:40.440 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-11T07:57:40.446 INFO:teuthology.orchestra.run.smithi150.stdout:(4/24): libradosstriper1-17.2.7-1623.gd0e6828a. 6.7 MB/s | 457 kB 00:00 2024-09-11T07:57:40.671 INFO:teuthology.orchestra.run.smithi150.stdout:(5/24): librgw2-17.2.7-1623.gd0e6828a.el9.x86_6 22 MB/s | 4.9 MB 00:00 2024-09-11T07:57:40.722 INFO:teuthology.orchestra.run.smithi150.stdout:(6/24): python3-ceph-argparse-17.2.7-1623.gd0e6 903 kB/s | 45 kB 00:00 2024-09-11T07:57:40.780 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-11T07:57:40.881 INFO:teuthology.orchestra.run.smithi150.stdout:(7/24): ceph-radosgw-17.2.7-1623.gd0e6828a.el9. 12 MB/s | 14 MB 00:01 2024-09-11T07:57:40.893 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : librados2-2:17.2.7-1623.gd0e6828a.el9.x86_64 2/26 2024-09-11T07:57:40.906 INFO:teuthology.orchestra.run.smithi150.stdout:(8/24): python3-ceph-common-17.2.7-1623.gd0e682 655 kB/s | 120 kB 00:00 2024-09-11T07:57:40.931 INFO:teuthology.orchestra.run.smithi150.stdout:(9/24): python3-cephfs-17.2.7-1623.gd0e6828a.el 3.0 MB/s | 156 kB 00:00 2024-09-11T07:57:40.973 INFO:teuthology.orchestra.run.smithi150.stdout:(10/24): python3-rados-17.2.7-1623.gd0e6828a.el 4.7 MB/s | 320 kB 00:00 2024-09-11T07:57:40.985 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados2-2:17.2.7-1623.gd0e6828a.el9.x86_64 2/26 2024-09-11T07:57:40.998 INFO:teuthology.orchestra.run.smithi150.stdout:(11/24): python3-rbd-17.2.7-1623.gd0e6828a.el9. 4.3 MB/s | 294 kB 00:00 2024-09-11T07:57:41.049 INFO:teuthology.orchestra.run.smithi150.stdout:(12/24): python3-rgw-17.2.7-1623.gd0e6828a.el9. 1.3 MB/s | 99 kB 00:00 2024-09-11T07:57:41.063 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rados-2:17.2.7-1623.gd0e6828a.el9.x86_64 3/26 2024-09-11T07:57:41.111 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephfs2-2:17.2.7-1623.gd0e6828a.el9.x86_64 4/26 2024-09-11T07:57:41.141 INFO:teuthology.orchestra.run.smithi150.stdout:(13/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.1 MB/s | 160 kB 00:00 2024-09-11T07:57:41.166 INFO:teuthology.orchestra.run.smithi150.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 386 kB/s | 45 kB 00:00 2024-09-11T07:57:41.182 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephfs2-2:17.2.7-1623.gd0e6828a.el9.x86_64 4/26 2024-09-11T07:57:41.225 INFO:teuthology.orchestra.run.smithi150.stdout:(15/24): librdkafka-1.6.1-102.el9.x86_64.rpm 7.7 MB/s | 662 kB 00:00 2024-09-11T07:57:41.243 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-11T07:57:41.266 INFO:teuthology.orchestra.run.smithi150.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 42 kB 00:00 2024-09-11T07:57:41.292 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-11T07:57:41.300 INFO:teuthology.orchestra.run.smithi150.stdout:(17/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.1 MB/s | 292 kB 00:00 2024-09-11T07:57:41.342 INFO:teuthology.orchestra.run.smithi150.stdout:(18/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 4.0 MB/s | 309 kB 00:00 2024-09-11T07:57:41.367 INFO:teuthology.orchestra.run.smithi150.stdout:(19/24): libunwind-1.6.2-1.el9.x86_64.rpm 2.8 MB/s | 67 kB 00:00 2024-09-11T07:57:41.393 INFO:teuthology.orchestra.run.smithi150.stdout:(20/24): liboath-2.6.7-2.el9.x86_64.rpm 537 kB/s | 49 kB 00:00 2024-09-11T07:57:41.443 INFO:teuthology.orchestra.run.smithi150.stdout:(21/24): thrift-0.15.0-2.el9.x86_64.rpm 21 MB/s | 1.6 MB 00:00 2024-09-11T07:57:41.577 INFO:teuthology.orchestra.run.smithi150.stdout:(22/24): librados2-17.2.7-1623.gd0e6828a.el9.x8 17 MB/s | 3.1 MB 00:00 2024-09-11T07:57:41.677 INFO:teuthology.orchestra.run.smithi150.stdout:(23/24): librbd1-17.2.7-1623.gd0e6828a.el9.x86_ 13 MB/s | 2.9 MB 00:00 2024-09-11T07:57:41.737 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-11T07:57:41.773 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librgw2-2:17.2.7-1623.gd0e6828a.el9.x86_64 8/26 2024-09-11T07:57:41.835 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librgw2-2:17.2.7-1623.gd0e6828a.el9.x86_64 8/26 2024-09-11T07:57:41.864 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-11T07:57:41.907 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ceph-argparse-2:17.2.7-1623.gd0e6828a.el9. 10/26 2024-09-11T07:57:41.943 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cephfs-2:17.2.7-1623.gd0e6828a.el9.x86_64 11/26 2024-09-11T07:57:42.015 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rgw-2:17.2.7-1623.gd0e6828a.el9.x86_64 12/26 2024-09-11T07:57:42.043 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libradosstriper1-2:17.2.7-1623.gd0e6828a.el9.x86_6 13/26 2024-09-11T07:57:42.101 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1623.gd0e6828a.el9.x86_6 13/26 2024-09-11T07:57:42.127 INFO:teuthology.orchestra.run.smithi150.stdout:(24/24): ceph-common-17.2.7-1623.gd0e6828a.el9. 9.1 MB/s | 23 MB 00:02 2024-09-11T07:57:42.132 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-09-11T07:57:42.133 INFO:teuthology.orchestra.run.smithi150.stdout:Total 19 MB/s | 58 MB 00:02 2024-09-11T07:57:42.145 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-11T07:57:42.177 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-09-11T07:57:42.193 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-09-11T07:57:42.209 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-11T07:57:42.228 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-09-11T07:57:42.229 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-09-11T07:57:42.503 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-11T07:57:42.547 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : librbd1-2:17.2.7-1623.gd0e6828a.el9.x86_64 18/26 2024-09-11T07:57:42.636 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librbd1-2:17.2.7-1623.gd0e6828a.el9.x86_64 18/26 2024-09-11T07:57:42.698 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-09-11T07:57:42.698 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-09-11T07:57:42.711 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rbd-2:17.2.7-1623.gd0e6828a.el9.x86_64 19/26 2024-09-11T07:57:42.741 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ceph-common-2:17.2.7-1623.gd0e6828a.el9.x8 20/26 2024-09-11T07:57:43.886 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-09-11T07:57:44.215 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-11T07:57:44.322 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : librados2-2:17.2.7-1623.gd0e6828a.el9.x86_64 2/26 2024-09-11T07:57:44.413 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librados2-2:17.2.7-1623.gd0e6828a.el9.x86_64 2/26 2024-09-11T07:57:44.499 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rados-2:17.2.7-1623.gd0e6828a.el9.x86_64 3/26 2024-09-11T07:57:44.539 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libcephfs2-2:17.2.7-1623.gd0e6828a.el9.x86_64 4/26 2024-09-11T07:57:44.627 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libcephfs2-2:17.2.7-1623.gd0e6828a.el9.x86_64 4/26 2024-09-11T07:57:44.680 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-11T07:57:44.720 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-11T07:57:45.114 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-common-2:17.2.7-1623.gd0e6828a.el9.x86_64 21/26 2024-09-11T07:57:45.146 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-common-2:17.2.7-1623.gd0e6828a.el9.x86_64 21/26 2024-09-11T07:57:45.162 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-11T07:57:45.192 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librgw2-2:17.2.7-1623.gd0e6828a.el9.x86_64 8/26 2024-09-11T07:57:45.264 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librgw2-2:17.2.7-1623.gd0e6828a.el9.x86_64 8/26 2024-09-11T07:57:45.292 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-11T07:57:45.336 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-ceph-argparse-2:17.2.7-1623.gd0e6828a.el9. 10/26 2024-09-11T07:57:45.372 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cephfs-2:17.2.7-1623.gd0e6828a.el9.x86_64 11/26 2024-09-11T07:57:45.434 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rgw-2:17.2.7-1623.gd0e6828a.el9.x86_64 12/26 2024-09-11T07:57:45.462 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libradosstriper1-2:17.2.7-1623.gd0e6828a.el9.x86_6 13/26 2024-09-11T07:57:45.511 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1623.gd0e6828a.el9.x86_6 13/26 2024-09-11T07:57:45.563 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-11T07:57:45.595 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-09-11T07:57:45.629 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-11T07:57:45.670 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-common-2:17.2.7-1623.gd0e6828a.el9.x86_64 21/26 2024-09-11T07:57:45.708 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-base-2:17.2.7-1623.gd0e6828a.el9.x86_64 22/26 2024-09-11T07:57:45.808 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-base-2:17.2.7-1623.gd0e6828a.el9.x86_64 22/26 2024-09-11T07:57:45.809 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-11T07:57:45.809 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-11T07:57:45.809 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:57:45.842 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-selinux-2:17.2.7-1623.gd0e6828a.el9.x86_64 23/26 2024-09-11T07:57:45.922 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-11T07:57:45.974 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : librbd1-2:17.2.7-1623.gd0e6828a.el9.x86_64 18/26 2024-09-11T07:57:46.064 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librbd1-2:17.2.7-1623.gd0e6828a.el9.x86_64 18/26 2024-09-11T07:57:46.137 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rbd-2:17.2.7-1623.gd0e6828a.el9.x86_64 19/26 2024-09-11T07:57:46.168 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-ceph-common-2:17.2.7-1623.gd0e6828a.el9.x8 20/26 2024-09-11T07:57:48.627 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-common-2:17.2.7-1623.gd0e6828a.el9.x86_64 21/26 2024-09-11T07:57:48.665 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-common-2:17.2.7-1623.gd0e6828a.el9.x86_64 21/26 2024-09-11T07:57:49.199 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-common-2:17.2.7-1623.gd0e6828a.el9.x86_64 21/26 2024-09-11T07:57:49.243 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-base-2:17.2.7-1623.gd0e6828a.el9.x86_64 22/26 2024-09-11T07:57:49.323 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-base-2:17.2.7-1623.gd0e6828a.el9.x86_64 22/26 2024-09-11T07:57:49.323 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-11T07:57:49.323 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-11T07:57:49.323 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:57:49.353 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-selinux-2:17.2.7-1623.gd0e6828a.el9.x86_64 23/26 2024-09-11T07:57:55.803 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1623.gd0e6828a.el9.x86_64 23/26 2024-09-11T07:57:55.803 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /sys 2024-09-11T07:57:55.803 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /proc 2024-09-11T07:57:55.803 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /mnt 2024-09-11T07:57:55.803 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /var/tmp 2024-09-11T07:57:55.803 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /home 2024-09-11T07:57:55.803 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /root 2024-09-11T07:57:55.803 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /tmp 2024-09-11T07:57:55.803 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:57:57.135 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-radosgw-2:17.2.7-1623.gd0e6828a.el9.x86_64 24/26 2024-09-11T07:57:57.197 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1623.gd0e6828a.el9.x86_64 24/26 2024-09-11T07:57:57.197 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T07:57:57.197 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-11T07:57:57.197 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-11T07:57:57.197 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-11T07:57:57.197 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:57:57.198 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-11T07:57:57.279 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-11T07:57:57.279 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-11T07:57:58.559 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-11T07:57:58.559 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-base-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/26 2024-09-11T07:57:58.559 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-common-2:17.2.7-1623.gd0e6828a.el9.x86_64 2/26 2024-09-11T07:57:58.559 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-radosgw-2:17.2.7-1623.gd0e6828a.el9.x86_64 3/26 2024-09-11T07:57:58.559 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-selinux-2:17.2.7-1623.gd0e6828a.el9.x86_64 4/26 2024-09-11T07:57:58.559 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephfs2-2:17.2.7-1623.gd0e6828a.el9.x86_64 5/26 2024-09-11T07:57:58.559 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libradosstriper1-2:17.2.7-1623.gd0e6828a.el9.x86_6 6/26 2024-09-11T07:57:58.560 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librgw2-2:17.2.7-1623.gd0e6828a.el9.x86_64 7/26 2024-09-11T07:57:58.560 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1623.gd0e6828a.el9. 8/26 2024-09-11T07:57:58.560 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ceph-common-2:17.2.7-1623.gd0e6828a.el9.x8 9/26 2024-09-11T07:57:58.560 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cephfs-2:17.2.7-1623.gd0e6828a.el9.x86_64 10/26 2024-09-11T07:57:58.560 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rados-2:17.2.7-1623.gd0e6828a.el9.x86_64 11/26 2024-09-11T07:57:58.560 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rbd-2:17.2.7-1623.gd0e6828a.el9.x86_64 12/26 2024-09-11T07:57:58.560 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rgw-2:17.2.7-1623.gd0e6828a.el9.x86_64 13/26 2024-09-11T07:57:58.560 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-11T07:57:58.560 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-11T07:57:58.560 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-11T07:57:58.560 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-11T07:57:58.560 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-11T07:57:58.560 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-09-11T07:57:58.561 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-11T07:57:58.561 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-11T07:57:58.561 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-11T07:57:58.561 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados2-2:17.2.7-1623.gd0e6828a.el9.x86_64 23/26 2024-09-11T07:57:58.561 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-11T07:57:58.561 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librbd1-2:17.2.7-1623.gd0e6828a.el9.x86_64 25/26 2024-09-11T07:57:59.339 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1623.gd0e6828a.el9.x86_64 23/26 2024-09-11T07:57:59.340 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /sys 2024-09-11T07:57:59.340 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /proc 2024-09-11T07:57:59.340 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /mnt 2024-09-11T07:57:59.340 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /var/tmp 2024-09-11T07:57:59.340 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /home 2024-09-11T07:57:59.340 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /root 2024-09-11T07:57:59.340 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /tmp 2024-09-11T07:57:59.340 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:57:59.437 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-11T07:57:59.437 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:57:59.437 INFO:teuthology.orchestra.run.smithi181.stdout:Upgraded: 2024-09-11T07:57:59.437 INFO:teuthology.orchestra.run.smithi181.stdout: librados2-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:57:59.437 INFO:teuthology.orchestra.run.smithi181.stdout: librbd1-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:57:59.437 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-11T07:57:59.438 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-base-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:57:59.438 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-common-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:57:59.438 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-radosgw-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:57:59.438 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-selinux-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:57:59.438 INFO:teuthology.orchestra.run.smithi181.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-11T07:57:59.438 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs2-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:57:59.438 INFO:teuthology.orchestra.run.smithi181.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-11T07:57:59.438 INFO:teuthology.orchestra.run.smithi181.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-11T07:57:59.438 INFO:teuthology.orchestra.run.smithi181.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-11T07:57:59.438 INFO:teuthology.orchestra.run.smithi181.stdout: libradosstriper1-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:57:59.438 INFO:teuthology.orchestra.run.smithi181.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-11T07:57:59.438 INFO:teuthology.orchestra.run.smithi181.stdout: librgw2-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:57:59.439 INFO:teuthology.orchestra.run.smithi181.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-11T07:57:59.439 INFO:teuthology.orchestra.run.smithi181.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-11T07:57:59.439 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-argparse-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:57:59.439 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-common-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:57:59.439 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cephfs-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:57:59.439 INFO:teuthology.orchestra.run.smithi181.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-11T07:57:59.439 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rados-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:57:59.439 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rbd-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:57:59.439 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rgw-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:57:59.439 INFO:teuthology.orchestra.run.smithi181.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-11T07:57:59.439 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:57:59.439 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-11T07:57:59.830 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-test 2024-09-11T07:58:00.399 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:27 ago on Wed 11 Sep 2024 07:57:33 AM UTC. 2024-09-11T07:58:00.539 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-11T07:58:00.539 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T07:58:00.539 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-11T07:58:00.540 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T07:58:00.540 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-11T07:58:00.540 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-test x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 48 M 2024-09-11T07:58:00.540 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-11T07:58:00.540 INFO:teuthology.orchestra.run.smithi181.stdout: libcephsqlite x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 163 k 2024-09-11T07:58:00.540 INFO:teuthology.orchestra.run.smithi181.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-11T07:58:00.540 INFO:teuthology.orchestra.run.smithi181.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-11T07:58:00.540 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:58:00.540 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-11T07:58:00.540 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T07:58:00.540 INFO:teuthology.orchestra.run.smithi181.stdout:Install 4 Packages 2024-09-11T07:58:00.540 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:58:00.541 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 48 M 2024-09-11T07:58:00.541 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 193 M 2024-09-11T07:58:00.541 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-11T07:58:00.634 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-radosgw-2:17.2.7-1623.gd0e6828a.el9.x86_64 24/26 2024-09-11T07:58:00.727 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1623.gd0e6828a.el9.x86_64 24/26 2024-09-11T07:58:00.727 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T07:58:00.727 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-11T07:58:00.727 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-11T07:58:00.727 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-11T07:58:00.727 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:58:00.729 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-11T07:58:00.820 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-11T07:58:00.821 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-11T07:58:00.857 INFO:teuthology.orchestra.run.smithi181.stdout:(1/4): libcephsqlite-17.2.7-1623.gd0e6828a.el9. 755 kB/s | 163 kB 00:00 2024-09-11T07:58:00.916 INFO:teuthology.orchestra.run.smithi181.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.1 MB/s | 304 kB 00:00 2024-09-11T07:58:01.016 INFO:teuthology.orchestra.run.smithi181.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 402 kB/s | 64 kB 00:00 2024-09-11T07:58:02.124 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-11T07:58:02.125 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-base-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/26 2024-09-11T07:58:02.125 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-common-2:17.2.7-1623.gd0e6828a.el9.x86_64 2/26 2024-09-11T07:58:02.125 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-radosgw-2:17.2.7-1623.gd0e6828a.el9.x86_64 3/26 2024-09-11T07:58:02.125 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-selinux-2:17.2.7-1623.gd0e6828a.el9.x86_64 4/26 2024-09-11T07:58:02.125 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libcephfs2-2:17.2.7-1623.gd0e6828a.el9.x86_64 5/26 2024-09-11T07:58:02.125 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libradosstriper1-2:17.2.7-1623.gd0e6828a.el9.x86_6 6/26 2024-09-11T07:58:02.125 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librgw2-2:17.2.7-1623.gd0e6828a.el9.x86_64 7/26 2024-09-11T07:58:02.125 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1623.gd0e6828a.el9. 8/26 2024-09-11T07:58:02.125 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-ceph-common-2:17.2.7-1623.gd0e6828a.el9.x8 9/26 2024-09-11T07:58:02.125 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cephfs-2:17.2.7-1623.gd0e6828a.el9.x86_64 10/26 2024-09-11T07:58:02.125 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rados-2:17.2.7-1623.gd0e6828a.el9.x86_64 11/26 2024-09-11T07:58:02.126 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rbd-2:17.2.7-1623.gd0e6828a.el9.x86_64 12/26 2024-09-11T07:58:02.126 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rgw-2:17.2.7-1623.gd0e6828a.el9.x86_64 13/26 2024-09-11T07:58:02.126 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-11T07:58:02.126 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-11T07:58:02.126 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-11T07:58:02.126 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-11T07:58:02.126 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-11T07:58:02.126 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-09-11T07:58:02.126 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-11T07:58:02.127 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-11T07:58:02.127 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-11T07:58:02.127 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librados2-2:17.2.7-1623.gd0e6828a.el9.x86_64 23/26 2024-09-11T07:58:02.127 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-11T07:58:02.127 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librbd1-2:17.2.7-1623.gd0e6828a.el9.x86_64 25/26 2024-09-11T07:58:02.175 INFO:teuthology.orchestra.run.smithi181.stdout:(4/4): ceph-test-17.2.7-1623.gd0e6828a.el9.x86_ 31 MB/s | 48 MB 00:01 2024-09-11T07:58:02.177 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-11T07:58:02.177 INFO:teuthology.orchestra.run.smithi181.stdout:Total 30 MB/s | 48 MB 00:01 2024-09-11T07:58:02.205 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-11T07:58:02.224 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-11T07:58:02.224 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-11T07:58:02.594 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-11T07:58:02.594 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-11T07:58:02.884 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-11T07:58:02.884 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:58:02.884 INFO:teuthology.orchestra.run.smithi150.stdout:Upgraded: 2024-09-11T07:58:02.884 INFO:teuthology.orchestra.run.smithi150.stdout: librados2-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:58:02.884 INFO:teuthology.orchestra.run.smithi150.stdout: librbd1-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:58:02.884 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-09-11T07:58:02.884 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-base-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:58:02.884 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-common-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:58:02.884 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-radosgw-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:58:02.885 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-selinux-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:58:02.885 INFO:teuthology.orchestra.run.smithi150.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-11T07:58:02.885 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs2-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:58:02.885 INFO:teuthology.orchestra.run.smithi150.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-11T07:58:02.885 INFO:teuthology.orchestra.run.smithi150.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-11T07:58:02.885 INFO:teuthology.orchestra.run.smithi150.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-11T07:58:02.885 INFO:teuthology.orchestra.run.smithi150.stdout: libradosstriper1-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:58:02.885 INFO:teuthology.orchestra.run.smithi150.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-11T07:58:02.885 INFO:teuthology.orchestra.run.smithi150.stdout: librgw2-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:58:02.885 INFO:teuthology.orchestra.run.smithi150.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-11T07:58:02.885 INFO:teuthology.orchestra.run.smithi150.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-11T07:58:02.885 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-argparse-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:58:02.885 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-common-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:58:02.885 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cephfs-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:58:02.886 INFO:teuthology.orchestra.run.smithi150.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-11T07:58:02.886 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rados-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:58:02.886 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rbd-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:58:02.886 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rgw-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:58:02.886 INFO:teuthology.orchestra.run.smithi150.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-11T07:58:02.886 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:58:02.886 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-09-11T07:58:03.115 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-11T07:58:03.202 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-11T07:58:03.240 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-11T07:58:03.246 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-test 2024-09-11T07:58:03.260 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephsqlite-2:17.2.7-1623.gd0e6828a.el9.x86_64 3/4 2024-09-11T07:58:03.819 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:00:27 ago on Wed 11 Sep 2024 07:57:36 AM UTC. 2024-09-11T07:58:03.961 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-09-11T07:58:03.963 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-09-11T07:58:03.963 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2024-09-11T07:58:03.963 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-09-11T07:58:03.963 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-09-11T07:58:03.963 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-test x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 48 M 2024-09-11T07:58:03.963 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2024-09-11T07:58:03.964 INFO:teuthology.orchestra.run.smithi150.stdout: libcephsqlite x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 163 k 2024-09-11T07:58:03.964 INFO:teuthology.orchestra.run.smithi150.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-11T07:58:03.964 INFO:teuthology.orchestra.run.smithi150.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-11T07:58:03.964 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:58:03.964 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-09-11T07:58:03.964 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-09-11T07:58:03.964 INFO:teuthology.orchestra.run.smithi150.stdout:Install 4 Packages 2024-09-11T07:58:03.964 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:58:03.964 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 48 M 2024-09-11T07:58:03.964 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 193 M 2024-09-11T07:58:03.964 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-09-11T07:58:04.293 INFO:teuthology.orchestra.run.smithi150.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.4 MB/s | 304 kB 00:00 2024-09-11T07:58:04.334 INFO:teuthology.orchestra.run.smithi150.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-11T07:58:04.493 INFO:teuthology.orchestra.run.smithi150.stdout:(3/4): libcephsqlite-17.2.7-1623.gd0e6828a.el9. 396 kB/s | 163 kB 00:00 2024-09-11T07:58:05.860 INFO:teuthology.orchestra.run.smithi150.stdout:(4/4): ceph-test-17.2.7-1623.gd0e6828a.el9.x86_ 27 MB/s | 48 MB 00:01 2024-09-11T07:58:05.862 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-09-11T07:58:05.862 INFO:teuthology.orchestra.run.smithi150.stdout:Total 26 MB/s | 48 MB 00:01 2024-09-11T07:58:05.890 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-09-11T07:58:05.910 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-09-11T07:58:05.910 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-09-11T07:58:06.277 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-09-11T07:58:06.277 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-09-11T07:58:06.793 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-09-11T07:58:06.880 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-11T07:58:06.929 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-11T07:58:06.957 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libcephsqlite-2:17.2.7-1623.gd0e6828a.el9.x86_64 3/4 2024-09-11T07:58:07.690 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1623.gd0e6828a.el9.x86_64 3/4 2024-09-11T07:58:07.778 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-test-2:17.2.7-1623.gd0e6828a.el9.x86_64 4/4 2024-09-11T07:58:10.045 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-test-2:17.2.7-1623.gd0e6828a.el9.x86_64 4/4 2024-09-11T07:58:10.045 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-test-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/4 2024-09-11T07:58:10.046 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephsqlite-2:17.2.7-1623.gd0e6828a.el9.x86_64 2/4 2024-09-11T07:58:10.046 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-11T07:58:10.390 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-11T07:58:10.390 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:58:10.390 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-11T07:58:10.390 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-test-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:58:10.390 INFO:teuthology.orchestra.run.smithi181.stdout: libcephsqlite-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:58:10.390 INFO:teuthology.orchestra.run.smithi181.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-11T07:58:10.390 INFO:teuthology.orchestra.run.smithi181.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-11T07:58:10.390 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:58:10.390 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-11T07:58:10.675 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph 2024-09-11T07:58:11.246 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:38 ago on Wed 11 Sep 2024 07:57:33 AM UTC. 2024-09-11T07:58:11.381 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-11T07:58:11.384 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T07:58:11.384 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-11T07:58:11.384 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T07:58:11.384 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-11T07:58:11.384 INFO:teuthology.orchestra.run.smithi181.stdout: ceph x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 6.5 k 2024-09-11T07:58:11.384 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-11T07:58:11.384 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mds x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 2.1 M 2024-09-11T07:58:11.384 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 1.4 M 2024-09-11T07:58:11.384 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1623.gd0e6828a.el9 ceph-noarch 240 k 2024-09-11T07:58:11.384 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mon x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 3.9 M 2024-09-11T07:58:11.385 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-osd x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 15 M 2024-09-11T07:58:11.385 INFO:teuthology.orchestra.run.smithi181.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-11T07:58:11.385 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-11T07:58:11.385 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-11T07:58:11.385 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-11T07:58:11.385 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-11T07:58:11.385 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-11T07:58:11.385 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-11T07:58:11.385 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-collections 2024-09-11T07:58:11.385 INFO:teuthology.orchestra.run.smithi181.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-11T07:58:11.385 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-11T07:58:11.385 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-11T07:58:11.385 INFO:teuthology.orchestra.run.smithi181.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-11T07:58:11.385 INFO:teuthology.orchestra.run.smithi181.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-11T07:58:11.386 INFO:teuthology.orchestra.run.smithi181.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-11T07:58:11.386 INFO:teuthology.orchestra.run.smithi181.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-11T07:58:11.386 INFO:teuthology.orchestra.run.smithi181.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-11T07:58:11.386 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-11T07:58:11.386 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-11T07:58:11.386 INFO:teuthology.orchestra.run.smithi181.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-11T07:58:11.386 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-11T07:58:11.386 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-11T07:58:11.386 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-11T07:58:11.386 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-11T07:58:11.386 INFO:teuthology.orchestra.run.smithi181.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-11T07:58:11.386 INFO:teuthology.orchestra.run.smithi181.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-11T07:58:11.386 INFO:teuthology.orchestra.run.smithi181.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-11T07:58:11.387 INFO:teuthology.orchestra.run.smithi181.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-11T07:58:11.387 INFO:teuthology.orchestra.run.smithi181.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-11T07:58:11.387 INFO:teuthology.orchestra.run.smithi181.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-11T07:58:11.387 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:58:11.387 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-11T07:58:11.387 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T07:58:11.387 INFO:teuthology.orchestra.run.smithi181.stdout:Install 34 Packages 2024-09-11T07:58:11.387 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:58:11.388 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1623.gd0e6828a.el9.x86_64 3/4 2024-09-11T07:58:11.388 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 27 M 2024-09-11T07:58:11.388 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 96 M 2024-09-11T07:58:11.388 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-11T07:58:11.463 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-test-2:17.2.7-1623.gd0e6828a.el9.x86_64 4/4 2024-09-11T07:58:11.754 INFO:teuthology.orchestra.run.smithi181.stdout:(1/34): ceph-17.2.7-1623.gd0e6828a.el9.x86_64.r 55 kB/s | 6.5 kB 00:00 2024-09-11T07:58:11.929 INFO:teuthology.orchestra.run.smithi181.stdout:(2/34): ceph-mgr-17.2.7-1623.gd0e6828a.el9.x86_ 4.8 MB/s | 1.4 MB 00:00 2024-09-11T07:58:11.996 INFO:teuthology.orchestra.run.smithi181.stdout:(3/34): ceph-mds-17.2.7-1623.gd0e6828a.el9.x86_ 5.7 MB/s | 2.1 MB 00:00 2024-09-11T07:58:12.080 INFO:teuthology.orchestra.run.smithi181.stdout:(4/34): ceph-mon-17.2.7-1623.gd0e6828a.el9.x86_ 12 MB/s | 3.9 MB 00:00 2024-09-11T07:58:12.105 INFO:teuthology.orchestra.run.smithi181.stdout:(5/34): ceph-mgr-modules-core-17.2.7-1623.gd0e6 2.2 MB/s | 240 kB 00:00 2024-09-11T07:58:12.214 INFO:teuthology.orchestra.run.smithi181.stdout:(6/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.9 MB/s | 253 kB 00:00 2024-09-11T07:58:12.256 INFO:teuthology.orchestra.run.smithi181.stdout:(7/34): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-11T07:58:12.297 INFO:teuthology.orchestra.run.smithi181.stdout:(8/34): python3-pycparser-2.20-6.el9.noarch.rpm 3.2 MB/s | 135 kB 00:00 2024-09-11T07:58:12.448 INFO:teuthology.orchestra.run.smithi181.stdout:(9/34): ceph-osd-17.2.7-1623.gd0e6828a.el9.x86_ 30 MB/s | 15 MB 00:00 2024-09-11T07:58:12.473 INFO:teuthology.orchestra.run.smithi181.stdout:(10/34): python3-requests-2.25.1-8.el9.noarch.r 713 kB/s | 125 kB 00:00 2024-09-11T07:58:12.515 INFO:teuthology.orchestra.run.smithi181.stdout:(11/34): python3-cryptography-36.0.1-4.el9.x86_ 3.0 MB/s | 1.2 MB 00:00 2024-09-11T07:58:12.592 INFO:teuthology.orchestra.run.smithi181.stdout:(12/34): python3-urllib3-1.26.5-6.el9.noarch.rp 1.5 MB/s | 215 kB 00:00 2024-09-11T07:58:12.800 INFO:teuthology.orchestra.run.smithi181.stdout:(13/34): python3-markupsafe-1.1.1-12.el9.x86_64 122 kB/s | 35 kB 00:00 2024-09-11T07:58:13.000 INFO:teuthology.orchestra.run.smithi181.stdout:(14/34): python3-pytz-2021.1-5.el9.noarch.rpm 255 kB/s | 51 kB 00:00 2024-09-11T07:58:13.051 INFO:teuthology.orchestra.run.smithi181.stdout:(15/34): python3-packaging-20.9-5.el9.noarch.rp 168 kB/s | 77 kB 00:00 2024-09-11T07:58:13.084 INFO:teuthology.orchestra.run.smithi181.stdout:(16/34): python3-mako-1.1.4-6.el9.noarch.rpm 282 kB/s | 172 kB 00:00 2024-09-11T07:58:13.126 INFO:teuthology.orchestra.run.smithi181.stdout:(17/34): python3-toml-0.10.2-6.el9.noarch.rpm 331 kB/s | 42 kB 00:00 2024-09-11T07:58:13.152 INFO:teuthology.orchestra.run.smithi181.stdout:(18/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 431 kB/s | 43 kB 00:00 2024-09-11T07:58:13.185 INFO:teuthology.orchestra.run.smithi181.stdout:(19/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 409 kB/s | 11 kB 00:00 2024-09-11T07:58:13.210 INFO:teuthology.orchestra.run.smithi181.stdout:(20/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-09-11T07:58:13.227 INFO:teuthology.orchestra.run.smithi181.stdout:(21/34): python3-jaraco-classes-3.2.1-5.el9.noa 439 kB/s | 18 kB 00:00 2024-09-11T07:58:13.252 INFO:teuthology.orchestra.run.smithi181.stdout:(22/34): python3-jaraco-collections-3.0.0-8.el9 554 kB/s | 23 kB 00:00 2024-09-11T07:58:13.269 INFO:teuthology.orchestra.run.smithi181.stdout:(23/34): python3-jaraco-functools-3.5.0-2.el9.n 465 kB/s | 19 kB 00:00 2024-09-11T07:58:13.286 INFO:teuthology.orchestra.run.smithi181.stdout:(24/34): python3-jaraco-text-3.2.0-6.el9.noarch 586 kB/s | 20 kB 00:00 2024-09-11T07:58:13.303 INFO:teuthology.orchestra.run.smithi181.stdout:(25/34): python3-logutils-0.3.5-21.el9.noarch.r 1.3 MB/s | 46 kB 00:00 2024-09-11T07:58:13.329 INFO:teuthology.orchestra.run.smithi181.stdout:(26/34): python3-more-itertools-8.12.0-2.el9.no 1.8 MB/s | 79 kB 00:00 2024-09-11T07:58:13.346 INFO:teuthology.orchestra.run.smithi181.stdout:(27/34): python3-portend-3.1.0-2.el9.noarch.rpm 993 kB/s | 16 kB 00:00 2024-09-11T07:58:13.371 INFO:teuthology.orchestra.run.smithi181.stdout:(28/34): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2024-09-11T07:58:13.396 INFO:teuthology.orchestra.run.smithi181.stdout:(29/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-09-11T07:58:13.413 INFO:teuthology.orchestra.run.smithi181.stdout:(30/34): python3-tempora-5.0.0-2.el9.noarch.rpm 854 kB/s | 36 kB 00:00 2024-09-11T07:58:13.438 INFO:teuthology.orchestra.run.smithi181.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 2.0 MB/s | 272 kB 00:00 2024-09-11T07:58:13.455 INFO:teuthology.orchestra.run.smithi181.stdout:(32/34): python3-zc-lockfile-2.0-10.el9.noarch. 1.2 MB/s | 20 kB 00:00 2024-09-11T07:58:13.481 INFO:teuthology.orchestra.run.smithi181.stdout:(33/34): python3-webob-1.8.8-2.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-09-11T07:58:13.505 INFO:teuthology.orchestra.run.smithi181.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.5 MB/s | 427 kB 00:00 2024-09-11T07:58:13.509 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-11T07:58:13.510 INFO:teuthology.orchestra.run.smithi181.stdout:Total 13 MB/s | 27 MB 00:02 2024-09-11T07:58:13.637 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-11T07:58:13.674 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-11T07:58:13.674 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-11T07:58:13.687 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-test-2:17.2.7-1623.gd0e6828a.el9.x86_64 4/4 2024-09-11T07:58:13.687 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-test-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/4 2024-09-11T07:58:13.687 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libcephsqlite-2:17.2.7-1623.gd0e6828a.el9.x86_64 2/4 2024-09-11T07:58:13.687 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-11T07:58:14.004 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-11T07:58:14.004 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:58:14.005 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-09-11T07:58:14.005 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-test-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:58:14.005 INFO:teuthology.orchestra.run.smithi150.stdout: libcephsqlite-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:58:14.005 INFO:teuthology.orchestra.run.smithi150.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-11T07:58:14.005 INFO:teuthology.orchestra.run.smithi150.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-11T07:58:14.005 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:58:14.005 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-09-11T07:58:14.041 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-11T07:58:14.041 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-11T07:58:14.321 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph 2024-09-11T07:58:14.889 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:00:38 ago on Wed 11 Sep 2024 07:57:36 AM UTC. 2024-09-11T07:58:15.031 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-09-11T07:58:15.033 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-09-11T07:58:15.033 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2024-09-11T07:58:15.034 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-09-11T07:58:15.034 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-09-11T07:58:15.034 INFO:teuthology.orchestra.run.smithi150.stdout: ceph x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 6.5 k 2024-09-11T07:58:15.034 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2024-09-11T07:58:15.034 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mds x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 2.1 M 2024-09-11T07:58:15.034 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 1.4 M 2024-09-11T07:58:15.034 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1623.gd0e6828a.el9 ceph-noarch 240 k 2024-09-11T07:58:15.034 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mon x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 3.9 M 2024-09-11T07:58:15.034 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-osd x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 15 M 2024-09-11T07:58:15.034 INFO:teuthology.orchestra.run.smithi150.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-11T07:58:15.034 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-11T07:58:15.034 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-11T07:58:15.034 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-11T07:58:15.034 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-11T07:58:15.035 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-11T07:58:15.035 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-11T07:58:15.035 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-collections 2024-09-11T07:58:15.035 INFO:teuthology.orchestra.run.smithi150.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-11T07:58:15.035 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-11T07:58:15.035 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-11T07:58:15.035 INFO:teuthology.orchestra.run.smithi150.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-11T07:58:15.035 INFO:teuthology.orchestra.run.smithi150.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-11T07:58:15.035 INFO:teuthology.orchestra.run.smithi150.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-11T07:58:15.035 INFO:teuthology.orchestra.run.smithi150.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-11T07:58:15.035 INFO:teuthology.orchestra.run.smithi150.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-11T07:58:15.035 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-11T07:58:15.035 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-11T07:58:15.036 INFO:teuthology.orchestra.run.smithi150.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-11T07:58:15.036 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-11T07:58:15.036 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-11T07:58:15.036 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-11T07:58:15.036 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-11T07:58:15.036 INFO:teuthology.orchestra.run.smithi150.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-11T07:58:15.036 INFO:teuthology.orchestra.run.smithi150.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-11T07:58:15.036 INFO:teuthology.orchestra.run.smithi150.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-11T07:58:15.036 INFO:teuthology.orchestra.run.smithi150.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-11T07:58:15.036 INFO:teuthology.orchestra.run.smithi150.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-11T07:58:15.037 INFO:teuthology.orchestra.run.smithi150.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-11T07:58:15.037 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:58:15.037 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-09-11T07:58:15.037 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-09-11T07:58:15.037 INFO:teuthology.orchestra.run.smithi150.stdout:Install 34 Packages 2024-09-11T07:58:15.037 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:58:15.038 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 27 M 2024-09-11T07:58:15.038 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 96 M 2024-09-11T07:58:15.038 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-09-11T07:58:15.320 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-11T07:58:15.392 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-11T07:58:15.425 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-11T07:58:15.457 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-11T07:58:15.549 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-11T07:58:15.623 INFO:teuthology.orchestra.run.smithi150.stdout:(1/34): ceph-17.2.7-1623.gd0e6828a.el9.x86_64.r 55 kB/s | 6.5 kB 00:00 2024-09-11T07:58:15.637 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-11T07:58:15.677 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-11T07:58:15.720 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-11T07:58:15.750 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-11T07:58:15.785 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-11T07:58:15.826 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-11T07:58:15.840 INFO:teuthology.orchestra.run.smithi150.stdout:(2/34): ceph-mds-17.2.7-1623.gd0e6828a.el9.x86_ 6.2 MB/s | 2.1 MB 00:00 2024-09-11T07:58:15.867 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-11T07:58:15.882 INFO:teuthology.orchestra.run.smithi150.stdout:(3/34): ceph-mgr-17.2.7-1623.gd0e6828a.el9.x86_ 3.7 MB/s | 1.4 MB 00:00 2024-09-11T07:58:15.902 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-11T07:58:15.936 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-11T07:58:15.966 INFO:teuthology.orchestra.run.smithi150.stdout:(4/34): ceph-mon-17.2.7-1623.gd0e6828a.el9.x86_ 11 MB/s | 3.9 MB 00:00 2024-09-11T07:58:15.984 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-11T07:58:15.991 INFO:teuthology.orchestra.run.smithi150.stdout:(5/34): ceph-mgr-modules-core-17.2.7-1623.gd0e6 2.2 MB/s | 240 kB 00:00 2024-09-11T07:58:16.016 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-11T07:58:16.090 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-11T07:58:16.108 INFO:teuthology.orchestra.run.smithi150.stdout:(6/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.7 MB/s | 253 kB 00:00 2024-09-11T07:58:16.150 INFO:teuthology.orchestra.run.smithi150.stdout:(7/34): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-11T07:58:16.192 INFO:teuthology.orchestra.run.smithi150.stdout:(8/34): python3-pycparser-2.20-6.el9.noarch.rpm 3.1 MB/s | 135 kB 00:00 2024-09-11T07:58:16.229 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-11T07:58:16.234 INFO:teuthology.orchestra.run.smithi150.stdout:(9/34): python3-cryptography-36.0.1-4.el9.x86_6 5.1 MB/s | 1.2 MB 00:00 2024-09-11T07:58:16.259 INFO:teuthology.orchestra.run.smithi150.stdout:(10/34): python3-requests-2.25.1-8.el9.noarch.r 1.8 MB/s | 125 kB 00:00 2024-09-11T07:58:16.308 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-11T07:58:16.369 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-11T07:58:16.408 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-11T07:58:16.426 INFO:teuthology.orchestra.run.smithi150.stdout:(11/34): ceph-osd-17.2.7-1623.gd0e6828a.el9.x86 26 MB/s | 15 MB 00:00 2024-09-11T07:58:16.451 INFO:teuthology.orchestra.run.smithi150.stdout:(12/34): python3-urllib3-1.26.5-6.el9.noarch.rp 988 kB/s | 215 kB 00:00 2024-09-11T07:58:16.465 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-11T07:58:16.520 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-11T07:58:16.527 INFO:teuthology.orchestra.run.smithi150.stdout:(13/34): python3-markupsafe-1.1.1-12.el9.x86_64 347 kB/s | 35 kB 00:00 2024-09-11T07:58:16.569 INFO:teuthology.orchestra.run.smithi150.stdout:(14/34): python3-mako-1.1.4-6.el9.noarch.rpm 557 kB/s | 172 kB 00:00 2024-09-11T07:58:16.594 INFO:teuthology.orchestra.run.smithi150.stdout:(15/34): python3-packaging-20.9-5.el9.noarch.rp 543 kB/s | 77 kB 00:00 2024-09-11T07:58:16.619 INFO:teuthology.orchestra.run.smithi150.stdout:(16/34): python3-pytz-2021.1-5.el9.noarch.rpm 553 kB/s | 51 kB 00:00 2024-09-11T07:58:16.645 INFO:teuthology.orchestra.run.smithi150.stdout:(17/34): python3-toml-0.10.2-6.el9.noarch.rpm 552 kB/s | 42 kB 00:00 2024-09-11T07:58:16.695 INFO:teuthology.orchestra.run.smithi150.stdout:(18/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 433 kB/s | 43 kB 00:00 2024-09-11T07:58:16.707 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-11T07:58:16.720 INFO:teuthology.orchestra.run.smithi150.stdout:(19/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 458 kB/s | 11 kB 00:00 2024-09-11T07:58:16.745 INFO:teuthology.orchestra.run.smithi150.stdout:(20/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-09-11T07:58:16.762 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-11T07:58:16.770 INFO:teuthology.orchestra.run.smithi150.stdout:(21/34): python3-jaraco-classes-3.2.1-5.el9.noa 354 kB/s | 18 kB 00:00 2024-09-11T07:58:16.796 INFO:teuthology.orchestra.run.smithi150.stdout:(22/34): python3-jaraco-functools-3.5.0-2.el9.n 781 kB/s | 19 kB 00:00 2024-09-11T07:58:16.821 INFO:teuthology.orchestra.run.smithi150.stdout:(23/34): python3-jaraco-collections-3.0.0-8.el9 462 kB/s | 23 kB 00:00 2024-09-11T07:58:16.840 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-11T07:58:16.846 INFO:teuthology.orchestra.run.smithi150.stdout:(24/34): python3-jaraco-text-3.2.0-6.el9.noarch 392 kB/s | 20 kB 00:00 2024-09-11T07:58:16.880 INFO:teuthology.orchestra.run.smithi150.stdout:(25/34): python3-cherrypy-18.6.1-2.el9.noarch.r 1.5 MB/s | 358 kB 00:00 2024-09-11T07:58:16.905 INFO:teuthology.orchestra.run.smithi150.stdout:(26/34): python3-logutils-0.3.5-21.el9.noarch.r 551 kB/s | 46 kB 00:00 2024-09-11T07:58:16.930 INFO:teuthology.orchestra.run.smithi150.stdout:(27/34): python3-more-itertools-8.12.0-2.el9.no 941 kB/s | 79 kB 00:00 2024-09-11T07:58:16.932 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-11T07:58:16.956 INFO:teuthology.orchestra.run.smithi150.stdout:(28/34): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-09-11T07:58:16.981 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-11T07:58:16.981 INFO:teuthology.orchestra.run.smithi150.stdout:(29/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-09-11T07:58:17.006 INFO:teuthology.orchestra.run.smithi150.stdout:(30/34): python3-tempora-5.0.0-2.el9.noarch.rpm 713 kB/s | 36 kB 00:00 2024-09-11T07:58:17.031 INFO:teuthology.orchestra.run.smithi150.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-11T07:58:17.057 INFO:teuthology.orchestra.run.smithi150.stdout:(32/34): python3-zc-lockfile-2.0-10.el9.noarch. 802 kB/s | 20 kB 00:00 2024-09-11T07:58:17.082 INFO:teuthology.orchestra.run.smithi150.stdout:(33/34): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-11T07:58:17.095 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-11T07:58:17.115 INFO:teuthology.orchestra.run.smithi150.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-09-11T07:58:17.119 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-09-11T07:58:17.119 INFO:teuthology.orchestra.run.smithi150.stdout:Total 13 MB/s | 27 MB 00:02 2024-09-11T07:58:17.242 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1623.gd0e6828a.el9. 29/34 2024-09-11T07:58:17.247 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-09-11T07:58:17.273 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-2:17.2.7-1623.gd0e6828a.el9.x86_64 30/34 2024-09-11T07:58:17.285 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-09-11T07:58:17.285 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-09-11T07:58:17.299 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1623.gd0e6828a.el9.x86_64 30/34 2024-09-11T07:58:17.299 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T07:58:17.299 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-11T07:58:17.299 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-11T07:58:17.300 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-11T07:58:17.300 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:58:17.666 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-09-11T07:58:17.667 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-09-11T07:58:18.675 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-osd-2:17.2.7-1623.gd0e6828a.el9.x86_64 31/34 2024-09-11T07:58:18.710 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-osd-2:17.2.7-1623.gd0e6828a.el9.x86_64 31/34 2024-09-11T07:58:18.710 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T07:58:18.710 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-11T07:58:18.710 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-11T07:58:18.710 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-11T07:58:18.710 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:58:18.906 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-09-11T07:58:18.977 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-11T07:58:19.009 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-11T07:58:19.042 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-11T07:58:19.084 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mon-2:17.2.7-1623.gd0e6828a.el9.x86_64 32/34 2024-09-11T07:58:19.108 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mon-2:17.2.7-1623.gd0e6828a.el9.x86_64 32/34 2024-09-11T07:58:19.108 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T07:58:19.109 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-11T07:58:19.109 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-11T07:58:19.109 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-11T07:58:19.109 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:58:19.133 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-11T07:58:19.232 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-11T07:58:19.277 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-11T07:58:19.309 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mds-2:17.2.7-1623.gd0e6828a.el9.x86_64 33/34 2024-09-11T07:58:19.321 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-11T07:58:19.333 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mds-2:17.2.7-1623.gd0e6828a.el9.x86_64 33/34 2024-09-11T07:58:19.333 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T07:58:19.333 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-11T07:58:19.333 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-11T07:58:19.333 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-11T07:58:19.334 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:58:19.359 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-11T07:58:19.394 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-11T07:58:19.398 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-2:17.2.7-1623.gd0e6828a.el9.x86_64 34/34 2024-09-11T07:58:19.427 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-11T07:58:19.467 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-11T07:58:19.502 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-11T07:58:19.537 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-11T07:58:19.585 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-11T07:58:19.617 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-11T07:58:19.690 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-11T07:58:19.838 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-11T07:58:19.925 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-11T07:58:19.978 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-11T07:58:20.017 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-11T07:58:20.066 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-11T07:58:20.121 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-11T07:58:20.313 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-11T07:58:20.363 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-11T07:58:20.432 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-11T07:58:20.520 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-11T07:58:20.566 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-11T07:58:20.691 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-11T07:58:20.849 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1623.gd0e6828a.el9. 29/34 2024-09-11T07:58:20.881 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-2:17.2.7-1623.gd0e6828a.el9.x86_64 30/34 2024-09-11T07:58:20.906 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1623.gd0e6828a.el9.x86_64 30/34 2024-09-11T07:58:20.907 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T07:58:20.907 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-11T07:58:20.907 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-11T07:58:20.907 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-11T07:58:20.907 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:58:21.163 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-2:17.2.7-1623.gd0e6828a.el9.x86_64 34/34 2024-09-11T07:58:21.163 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/34 2024-09-11T07:58:21.163 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mds-2:17.2.7-1623.gd0e6828a.el9.x86_64 2/34 2024-09-11T07:58:21.163 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-2:17.2.7-1623.gd0e6828a.el9.x86_64 3/34 2024-09-11T07:58:21.163 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mon-2:17.2.7-1623.gd0e6828a.el9.x86_64 4/34 2024-09-11T07:58:21.163 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-osd-2:17.2.7-1623.gd0e6828a.el9.x86_64 5/34 2024-09-11T07:58:21.163 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1623.gd0e6828a.el9. 6/34 2024-09-11T07:58:21.163 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-11T07:58:21.164 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-11T07:58:21.164 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-11T07:58:21.164 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-11T07:58:21.164 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-11T07:58:21.164 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-11T07:58:21.164 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-11T07:58:21.164 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-11T07:58:21.164 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-11T07:58:21.164 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-11T07:58:21.164 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-11T07:58:21.164 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-11T07:58:21.164 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-11T07:58:21.164 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-11T07:58:21.165 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-11T07:58:21.165 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-11T07:58:21.165 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-11T07:58:21.165 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-11T07:58:21.165 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-11T07:58:21.165 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-11T07:58:21.165 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-11T07:58:21.165 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-11T07:58:21.165 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-11T07:58:21.165 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-11T07:58:21.165 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-11T07:58:21.165 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-11T07:58:21.165 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-11T07:58:22.061 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-11T07:58:22.062 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:58:22.062 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-11T07:58:22.062 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:58:22.062 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mds-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:58:22.062 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:58:22.062 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-modules-core-2:17.2.7-1623.gd0e6828a.el9.noarch 2024-09-11T07:58:22.063 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mon-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:58:22.063 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-osd-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:58:22.063 INFO:teuthology.orchestra.run.smithi181.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-11T07:58:22.063 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-11T07:58:22.063 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-11T07:58:22.063 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-11T07:58:22.063 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-11T07:58:22.063 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-11T07:58:22.063 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-11T07:58:22.063 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-11T07:58:22.063 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-11T07:58:22.063 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-11T07:58:22.064 INFO:teuthology.orchestra.run.smithi181.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-11T07:58:22.064 INFO:teuthology.orchestra.run.smithi181.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-11T07:58:22.064 INFO:teuthology.orchestra.run.smithi181.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-11T07:58:22.064 INFO:teuthology.orchestra.run.smithi181.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-11T07:58:22.064 INFO:teuthology.orchestra.run.smithi181.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-11T07:58:22.064 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-11T07:58:22.064 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ply-3.11-14.el9.noarch 2024-09-11T07:58:22.064 INFO:teuthology.orchestra.run.smithi181.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-11T07:58:22.064 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-11T07:58:22.064 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-11T07:58:22.064 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-11T07:58:22.064 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-11T07:58:22.065 INFO:teuthology.orchestra.run.smithi181.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-11T07:58:22.065 INFO:teuthology.orchestra.run.smithi181.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-11T07:58:22.065 INFO:teuthology.orchestra.run.smithi181.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-11T07:58:22.065 INFO:teuthology.orchestra.run.smithi181.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-11T07:58:22.065 INFO:teuthology.orchestra.run.smithi181.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-11T07:58:22.065 INFO:teuthology.orchestra.run.smithi181.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-11T07:58:22.065 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:58:22.065 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-11T07:58:22.282 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-osd-2:17.2.7-1623.gd0e6828a.el9.x86_64 31/34 2024-09-11T07:58:22.312 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-osd-2:17.2.7-1623.gd0e6828a.el9.x86_64 31/34 2024-09-11T07:58:22.312 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T07:58:22.312 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-11T07:58:22.312 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-11T07:58:22.312 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-11T07:58:22.312 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:58:22.430 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-base 2024-09-11T07:58:22.708 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mon-2:17.2.7-1623.gd0e6828a.el9.x86_64 32/34 2024-09-11T07:58:22.732 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mon-2:17.2.7-1623.gd0e6828a.el9.x86_64 32/34 2024-09-11T07:58:22.732 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T07:58:22.732 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-11T07:58:22.732 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-11T07:58:22.732 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-11T07:58:22.733 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:58:22.933 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mds-2:17.2.7-1623.gd0e6828a.el9.x86_64 33/34 2024-09-11T07:58:22.957 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mds-2:17.2.7-1623.gd0e6828a.el9.x86_64 33/34 2024-09-11T07:58:22.958 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T07:58:22.958 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-11T07:58:22.958 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-11T07:58:22.958 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-11T07:58:22.958 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:58:22.997 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:49 ago on Wed 11 Sep 2024 07:57:33 AM UTC. 2024-09-11T07:58:23.022 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-2:17.2.7-1623.gd0e6828a.el9.x86_64 34/34 2024-09-11T07:58:23.089 INFO:teuthology.orchestra.run.smithi181.stdout:Package ceph-base-2:17.2.7-1623.gd0e6828a.el9.x86_64 is already installed. 2024-09-11T07:58:23.129 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-11T07:58:23.130 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-11T07:58:23.130 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-11T07:58:23.194 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install cephadm 2024-09-11T07:58:23.755 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:50 ago on Wed 11 Sep 2024 07:57:33 AM UTC. 2024-09-11T07:58:23.890 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-11T07:58:23.890 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T07:58:23.891 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-11T07:58:23.891 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T07:58:23.891 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-11T07:58:23.891 INFO:teuthology.orchestra.run.smithi181.stdout: cephadm noarch 2:17.2.7-1623.gd0e6828a.el9 ceph-noarch 83 k 2024-09-11T07:58:23.891 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:58:23.891 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-11T07:58:23.891 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T07:58:23.891 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-11T07:58:23.891 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:58:23.891 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 83 k 2024-09-11T07:58:23.891 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 375 k 2024-09-11T07:58:23.892 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-11T07:58:24.084 INFO:teuthology.orchestra.run.smithi181.stdout:cephadm-17.2.7-1623.gd0e6828a.el9.noarch.rpm 429 kB/s | 83 kB 00:00 2024-09-11T07:58:24.085 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-11T07:58:24.085 INFO:teuthology.orchestra.run.smithi181.stdout:Total 425 kB/s | 83 kB 00:00 2024-09-11T07:58:24.085 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-11T07:58:24.090 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-11T07:58:24.090 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-11T07:58:24.092 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-11T07:58:24.092 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-11T07:58:24.174 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-11T07:58:24.675 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: cephadm-2:17.2.7-1623.gd0e6828a.el9.noarch 1/1 2024-09-11T07:58:24.760 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-2:17.2.7-1623.gd0e6828a.el9.x86_64 34/34 2024-09-11T07:58:24.760 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/34 2024-09-11T07:58:24.760 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mds-2:17.2.7-1623.gd0e6828a.el9.x86_64 2/34 2024-09-11T07:58:24.760 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-2:17.2.7-1623.gd0e6828a.el9.x86_64 3/34 2024-09-11T07:58:24.760 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mon-2:17.2.7-1623.gd0e6828a.el9.x86_64 4/34 2024-09-11T07:58:24.760 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-osd-2:17.2.7-1623.gd0e6828a.el9.x86_64 5/34 2024-09-11T07:58:24.760 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1623.gd0e6828a.el9. 6/34 2024-09-11T07:58:24.760 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-11T07:58:24.760 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-11T07:58:24.760 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-11T07:58:24.760 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-11T07:58:24.761 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-11T07:58:24.761 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-11T07:58:24.764 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-11T07:58:24.764 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-11T07:58:24.764 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-11T07:58:24.764 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-11T07:58:24.764 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-11T07:58:24.764 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-11T07:58:24.764 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-11T07:58:24.764 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-11T07:58:24.765 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-11T07:58:24.765 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-11T07:58:24.765 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-11T07:58:24.765 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-11T07:58:24.765 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-11T07:58:24.765 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-11T07:58:24.765 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-11T07:58:24.766 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : cephadm-2:17.2.7-1623.gd0e6828a.el9.noarch 1/1 2024-09-11T07:58:24.766 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-11T07:58:24.766 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-11T07:58:24.766 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-11T07:58:24.766 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-11T07:58:24.767 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-11T07:58:24.767 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-11T07:58:25.106 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: cephadm-2:17.2.7-1623.gd0e6828a.el9.noarch 1/1 2024-09-11T07:58:25.354 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : cephadm-2:17.2.7-1623.gd0e6828a.el9.noarch 1/1 2024-09-11T07:58:25.354 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:58:25.354 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-11T07:58:25.354 INFO:teuthology.orchestra.run.smithi181.stdout: cephadm-2:17.2.7-1623.gd0e6828a.el9.noarch 2024-09-11T07:58:25.354 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:58:25.355 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-11T07:58:25.563 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-immutable-object-cache 2024-09-11T07:58:25.688 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-11T07:58:25.688 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:58:25.688 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-09-11T07:58:25.689 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:58:25.689 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mds-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:58:25.689 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:58:25.689 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-modules-core-2:17.2.7-1623.gd0e6828a.el9.noarch 2024-09-11T07:58:25.689 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mon-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:58:25.690 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-osd-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:58:25.690 INFO:teuthology.orchestra.run.smithi150.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-11T07:58:25.690 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-11T07:58:25.690 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-11T07:58:25.690 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-11T07:58:25.691 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-11T07:58:25.691 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-11T07:58:25.691 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-11T07:58:25.691 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-11T07:58:25.691 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-11T07:58:25.691 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-11T07:58:25.692 INFO:teuthology.orchestra.run.smithi150.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-11T07:58:25.692 INFO:teuthology.orchestra.run.smithi150.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-11T07:58:25.692 INFO:teuthology.orchestra.run.smithi150.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-11T07:58:25.692 INFO:teuthology.orchestra.run.smithi150.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-11T07:58:25.692 INFO:teuthology.orchestra.run.smithi150.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-11T07:58:25.693 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-11T07:58:25.693 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ply-3.11-14.el9.noarch 2024-09-11T07:58:25.693 INFO:teuthology.orchestra.run.smithi150.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-11T07:58:25.693 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-11T07:58:25.693 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-11T07:58:25.693 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-11T07:58:25.694 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-11T07:58:25.694 INFO:teuthology.orchestra.run.smithi150.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-11T07:58:25.694 INFO:teuthology.orchestra.run.smithi150.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-11T07:58:25.694 INFO:teuthology.orchestra.run.smithi150.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-11T07:58:25.694 INFO:teuthology.orchestra.run.smithi150.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-11T07:58:25.694 INFO:teuthology.orchestra.run.smithi150.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-11T07:58:25.695 INFO:teuthology.orchestra.run.smithi150.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-11T07:58:25.695 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:58:25.695 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-09-11T07:58:26.103 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-base 2024-09-11T07:58:26.136 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:53 ago on Wed 11 Sep 2024 07:57:33 AM UTC. 2024-09-11T07:58:26.271 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-11T07:58:26.272 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T07:58:26.272 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-09-11T07:58:26.272 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T07:58:26.272 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-11T07:58:26.272 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 137 k 2024-09-11T07:58:26.272 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:58:26.272 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-11T07:58:26.273 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T07:58:26.273 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-11T07:58:26.273 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:58:26.273 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 137 k 2024-09-11T07:58:26.273 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 379 k 2024-09-11T07:58:26.273 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-11T07:58:26.482 INFO:teuthology.orchestra.run.smithi181.stdout:ceph-immutable-object-cache-17.2.7-1623.gd0e682 657 kB/s | 137 kB 00:00 2024-09-11T07:58:26.483 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-11T07:58:26.483 INFO:teuthology.orchestra.run.smithi181.stdout:Total 652 kB/s | 137 kB 00:00 2024-09-11T07:58:26.483 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-11T07:58:26.490 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-11T07:58:26.490 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-11T07:58:26.550 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-11T07:58:26.550 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-11T07:58:26.686 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:00:50 ago on Wed 11 Sep 2024 07:57:36 AM UTC. 2024-09-11T07:58:26.691 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-11T07:58:26.775 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1623.gd0e6828a. 1/1 2024-09-11T07:58:26.778 INFO:teuthology.orchestra.run.smithi150.stdout:Package ceph-base-2:17.2.7-1623.gd0e6828a.el9.x86_64 is already installed. 2024-09-11T07:58:26.800 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1623.gd0e6828a. 1/1 2024-09-11T07:58:26.801 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T07:58:26.801 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-11T07:58:26.801 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:58:26.820 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-09-11T07:58:26.824 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2024-09-11T07:58:26.824 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-09-11T07:58:26.891 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install cephadm 2024-09-11T07:58:27.448 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:00:51 ago on Wed 11 Sep 2024 07:57:36 AM UTC. 2024-09-11T07:58:27.582 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-09-11T07:58:27.583 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-09-11T07:58:27.583 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2024-09-11T07:58:27.583 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-09-11T07:58:27.583 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-09-11T07:58:27.583 INFO:teuthology.orchestra.run.smithi150.stdout: cephadm noarch 2:17.2.7-1623.gd0e6828a.el9 ceph-noarch 83 k 2024-09-11T07:58:27.583 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:58:27.583 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-09-11T07:58:27.584 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-09-11T07:58:27.584 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2024-09-11T07:58:27.584 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:58:27.584 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 83 k 2024-09-11T07:58:27.584 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 375 k 2024-09-11T07:58:27.584 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-09-11T07:58:27.695 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1623.gd0e6828a. 1/1 2024-09-11T07:58:27.695 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:58:27.695 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-11T07:58:27.695 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-immutable-object-cache-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:58:27.695 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:58:27.696 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-11T07:58:27.776 INFO:teuthology.orchestra.run.smithi150.stdout:cephadm-17.2.7-1623.gd0e6828a.el9.noarch.rpm 430 kB/s | 83 kB 00:00 2024-09-11T07:58:27.777 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-09-11T07:58:27.777 INFO:teuthology.orchestra.run.smithi150.stdout:Total 426 kB/s | 83 kB 00:00 2024-09-11T07:58:27.777 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-09-11T07:58:27.782 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-09-11T07:58:27.782 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-09-11T07:58:27.784 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-09-11T07:58:27.785 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-09-11T07:58:27.855 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-09-11T07:58:27.912 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr 2024-09-11T07:58:28.327 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: cephadm-2:17.2.7-1623.gd0e6828a.el9.noarch 1/1 2024-09-11T07:58:28.422 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : cephadm-2:17.2.7-1623.gd0e6828a.el9.noarch 1/1 2024-09-11T07:58:28.480 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:55 ago on Wed 11 Sep 2024 07:57:33 AM UTC. 2024-09-11T07:58:28.576 INFO:teuthology.orchestra.run.smithi181.stdout:Package ceph-mgr-2:17.2.7-1623.gd0e6828a.el9.x86_64 is already installed. 2024-09-11T07:58:28.618 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-11T07:58:28.619 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-11T07:58:28.619 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-11T07:58:28.676 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-dashboard 2024-09-11T07:58:28.778 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: cephadm-2:17.2.7-1623.gd0e6828a.el9.noarch 1/1 2024-09-11T07:58:29.028 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : cephadm-2:17.2.7-1623.gd0e6828a.el9.noarch 1/1 2024-09-11T07:58:29.028 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:58:29.028 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-09-11T07:58:29.028 INFO:teuthology.orchestra.run.smithi150.stdout: cephadm-2:17.2.7-1623.gd0e6828a.el9.noarch 2024-09-11T07:58:29.028 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:58:29.029 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-09-11T07:58:29.230 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-immutable-object-cache 2024-09-11T07:58:29.245 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:56 ago on Wed 11 Sep 2024 07:57:33 AM UTC. 2024-09-11T07:58:29.378 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-11T07:58:29.379 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T07:58:29.379 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-11T07:58:29.379 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T07:58:29.379 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-11T07:58:29.379 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1623.gd0e6828a.el9 ceph-noarch 3.4 M 2024-09-11T07:58:29.379 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-11T07:58:29.380 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1623.gd0e6828a.el9 ceph-noarch 24 k 2024-09-11T07:58:29.380 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1623.gd0e6828a.el9 ceph-noarch 15 k 2024-09-11T07:58:29.380 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-11T07:58:29.380 INFO:teuthology.orchestra.run.smithi181.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-11T07:58:29.380 INFO:teuthology.orchestra.run.smithi181.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-11T07:58:29.380 INFO:teuthology.orchestra.run.smithi181.stdout:Installing weak dependencies: 2024-09-11T07:58:29.380 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-11T07:58:29.380 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:58:29.380 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-11T07:58:29.380 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T07:58:29.380 INFO:teuthology.orchestra.run.smithi181.stdout:Install 7 Packages 2024-09-11T07:58:29.380 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:58:29.380 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 3.7 M 2024-09-11T07:58:29.381 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 80 M 2024-09-11T07:58:29.381 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-11T07:58:29.803 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:00:53 ago on Wed 11 Sep 2024 07:57:36 AM UTC. 2024-09-11T07:58:29.940 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-09-11T07:58:29.941 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-09-11T07:58:29.941 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repo Size 2024-09-11T07:58:29.941 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-09-11T07:58:29.941 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-09-11T07:58:29.941 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 137 k 2024-09-11T07:58:29.941 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:58:29.941 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-09-11T07:58:29.941 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-09-11T07:58:29.941 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2024-09-11T07:58:29.941 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:58:29.941 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 137 k 2024-09-11T07:58:29.941 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 379 k 2024-09-11T07:58:29.942 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-09-11T07:58:29.945 INFO:teuthology.orchestra.run.smithi181.stdout:(1/7): ceph-grafana-dashboards-17.2.7-1623.gd0e 156 kB/s | 24 kB 00:00 2024-09-11T07:58:29.961 INFO:teuthology.orchestra.run.smithi181.stdout:(2/7): ceph-prometheus-alerts-17.2.7-1623.gd0e6 84 kB/s | 15 kB 00:00 2024-09-11T07:58:29.995 INFO:teuthology.orchestra.run.smithi181.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 181 kB/s | 9.0 kB 00:00 2024-09-11T07:58:30.012 INFO:teuthology.orchestra.run.smithi181.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 2.0 MB/s | 31 kB 00:00 2024-09-11T07:58:30.029 INFO:teuthology.orchestra.run.smithi181.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 609 kB/s | 41 kB 00:00 2024-09-11T07:58:30.054 INFO:teuthology.orchestra.run.smithi181.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 4.4 MB/s | 188 kB 00:00 2024-09-11T07:58:30.133 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-immutable-object-cache-17.2.7-1623.gd0e682 720 kB/s | 137 kB 00:00 2024-09-11T07:58:30.133 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-09-11T07:58:30.134 INFO:teuthology.orchestra.run.smithi150.stdout:Total 714 kB/s | 137 kB 00:00 2024-09-11T07:58:30.134 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-09-11T07:58:30.141 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-09-11T07:58:30.141 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-09-11T07:58:30.163 INFO:teuthology.orchestra.run.smithi181.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1623.gd0e6828a 9.0 MB/s | 3.4 MB 00:00 2024-09-11T07:58:30.166 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-11T07:58:30.166 INFO:teuthology.orchestra.run.smithi181.stdout:Total 4.6 MB/s | 3.7 MB 00:00 2024-09-11T07:58:30.167 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-11T07:58:30.175 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-11T07:58:30.175 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-11T07:58:30.199 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-09-11T07:58:30.199 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-09-11T07:58:30.252 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-11T07:58:30.253 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-11T07:58:30.361 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-09-11T07:58:30.442 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1623.gd0e6828a. 1/1 2024-09-11T07:58:30.466 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1623.gd0e6828a. 1/1 2024-09-11T07:58:30.466 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T07:58:30.466 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-11T07:58:30.466 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:58:30.552 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-11T07:58:30.630 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-11T07:58:30.664 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-11T07:58:30.712 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-11T07:58:30.743 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-11T07:58:30.772 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1623.gd0e6828a.el9.n 5/7 2024-09-11T07:58:31.378 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1623.gd0e6828a. 1/1 2024-09-11T07:58:31.378 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:58:31.378 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-09-11T07:58:31.378 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-immutable-object-cache-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:58:31.378 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:58:31.378 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-09-11T07:58:31.612 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-mgr 2024-09-11T07:58:31.756 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1623.gd0e6828a.el9. 6/7 2024-09-11T07:58:31.801 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1623.gd0e6828a.el9.noarc 7/7 2024-09-11T07:58:32.194 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:00:56 ago on Wed 11 Sep 2024 07:57:36 AM UTC. 2024-09-11T07:58:32.285 INFO:teuthology.orchestra.run.smithi150.stdout:Package ceph-mgr-2:17.2.7-1623.gd0e6828a.el9.x86_64 is already installed. 2024-09-11T07:58:32.326 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-09-11T07:58:32.327 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2024-09-11T07:58:32.327 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-09-11T07:58:32.400 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-mgr-dashboard 2024-09-11T07:58:32.912 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1623.gd0e6828a.el9.noarc 7/7 2024-09-11T07:58:32.912 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1623.gd0e6828a.el9. 1/7 2024-09-11T07:58:32.912 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1623.gd0e6828a.el9.noarc 2/7 2024-09-11T07:58:32.912 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1623.gd0e6828a.el9.n 3/7 2024-09-11T07:58:32.912 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-11T07:58:32.912 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-11T07:58:32.912 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-11T07:58:32.968 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:00:56 ago on Wed 11 Sep 2024 07:57:36 AM UTC. 2024-09-11T07:58:33.101 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-09-11T07:58:33.102 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-09-11T07:58:33.102 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2024-09-11T07:58:33.102 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-09-11T07:58:33.103 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-09-11T07:58:33.103 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1623.gd0e6828a.el9 ceph-noarch 3.4 M 2024-09-11T07:58:33.103 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2024-09-11T07:58:33.103 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1623.gd0e6828a.el9 ceph-noarch 24 k 2024-09-11T07:58:33.103 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1623.gd0e6828a.el9 ceph-noarch 15 k 2024-09-11T07:58:33.103 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-11T07:58:33.103 INFO:teuthology.orchestra.run.smithi150.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-11T07:58:33.103 INFO:teuthology.orchestra.run.smithi150.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-11T07:58:33.103 INFO:teuthology.orchestra.run.smithi150.stdout:Installing weak dependencies: 2024-09-11T07:58:33.103 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-11T07:58:33.103 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:58:33.103 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-09-11T07:58:33.104 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-09-11T07:58:33.104 INFO:teuthology.orchestra.run.smithi150.stdout:Install 7 Packages 2024-09-11T07:58:33.104 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:58:33.104 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 3.7 M 2024-09-11T07:58:33.104 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 80 M 2024-09-11T07:58:33.104 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-09-11T07:58:33.228 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-11T07:58:33.228 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:58:33.228 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-11T07:58:33.229 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-grafana-dashboards-2:17.2.7-1623.gd0e6828a.el9.noarch 2024-09-11T07:58:33.229 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-dashboard-2:17.2.7-1623.gd0e6828a.el9.noarch 2024-09-11T07:58:33.229 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-prometheus-alerts-2:17.2.7-1623.gd0e6828a.el9.noarch 2024-09-11T07:58:33.229 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-11T07:58:33.229 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-11T07:58:33.229 INFO:teuthology.orchestra.run.smithi181.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-11T07:58:33.229 INFO:teuthology.orchestra.run.smithi181.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-11T07:58:33.229 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:58:33.229 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-11T07:58:33.454 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-11T07:58:33.645 INFO:teuthology.orchestra.run.smithi150.stdout:(1/7): ceph-grafana-dashboards-17.2.7-1623.gd0e 169 kB/s | 24 kB 00:00 2024-09-11T07:58:33.662 INFO:teuthology.orchestra.run.smithi150.stdout:(2/7): ceph-prometheus-alerts-17.2.7-1623.gd0e6 91 kB/s | 15 kB 00:00 2024-09-11T07:58:33.695 INFO:teuthology.orchestra.run.smithi150.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 181 kB/s | 9.0 kB 00:00 2024-09-11T07:58:33.712 INFO:teuthology.orchestra.run.smithi150.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 2.0 MB/s | 31 kB 00:00 2024-09-11T07:58:33.729 INFO:teuthology.orchestra.run.smithi150.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 610 kB/s | 41 kB 00:00 2024-09-11T07:58:33.763 INFO:teuthology.orchestra.run.smithi150.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-09-11T07:58:33.863 INFO:teuthology.orchestra.run.smithi150.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1623.gd0e6828a 9.3 MB/s | 3.4 MB 00:00 2024-09-11T07:58:33.867 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-09-11T07:58:33.867 INFO:teuthology.orchestra.run.smithi150.stdout:Total 4.8 MB/s | 3.7 MB 00:00 2024-09-11T07:58:33.867 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-09-11T07:58:33.875 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-09-11T07:58:33.875 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-09-11T07:58:33.977 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-09-11T07:58:33.977 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-09-11T07:58:34.011 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:01 ago on Wed 11 Sep 2024 07:57:33 AM UTC. 2024-09-11T07:58:34.145 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-11T07:58:34.146 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================= 2024-09-11T07:58:34.146 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-11T07:58:34.146 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================= 2024-09-11T07:58:34.146 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-11T07:58:34.146 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1623.gd0e6828a.el9 ceph-noarch 7.4 M 2024-09-11T07:58:34.146 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-11T07:58:34.146 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-11T07:58:34.147 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-11T07:58:34.147 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-11T07:58:34.147 INFO:teuthology.orchestra.run.smithi181.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-11T07:58:34.147 INFO:teuthology.orchestra.run.smithi181.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-11T07:58:34.147 INFO:teuthology.orchestra.run.smithi181.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-11T07:58:34.147 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-11T07:58:34.147 INFO:teuthology.orchestra.run.smithi181.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-11T07:58:34.147 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-11T07:58:34.147 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-11T07:58:34.147 INFO:teuthology.orchestra.run.smithi181.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-11T07:58:34.147 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:58:34.147 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-11T07:58:34.148 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================= 2024-09-11T07:58:34.148 INFO:teuthology.orchestra.run.smithi181.stdout:Install 12 Packages 2024-09-11T07:58:34.148 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:58:34.148 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 38 M 2024-09-11T07:58:34.148 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 212 M 2024-09-11T07:58:34.148 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-11T07:58:34.276 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-09-11T07:58:34.339 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-11T07:58:34.373 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-11T07:58:34.428 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-11T07:58:34.469 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-11T07:58:34.497 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1623.gd0e6828a.el9.n 5/7 2024-09-11T07:58:34.755 INFO:teuthology.orchestra.run.smithi181.stdout:(1/12): ceph-mgr-diskprediction-local-17.2.7-16 17 MB/s | 7.4 MB 00:00 2024-09-11T07:58:34.871 INFO:teuthology.orchestra.run.smithi181.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 281 kB/s | 33 kB 00:00 2024-09-11T07:58:35.255 INFO:teuthology.orchestra.run.smithi181.stdout:(3/12): libquadmath-11.5.0-2.el9.x86_64.rpm 201 kB/s | 191 kB 00:00 2024-09-11T07:58:35.330 INFO:teuthology.orchestra.run.smithi181.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 244 kB/s | 18 kB 00:00 2024-09-11T07:58:35.422 INFO:teuthology.orchestra.run.smithi181.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 419 kB/s | 38 kB 00:00 2024-09-11T07:58:35.489 INFO:teuthology.orchestra.run.smithi181.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.8 MB/s | 3.0 MB 00:00 2024-09-11T07:58:35.493 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1623.gd0e6828a.el9. 6/7 2024-09-11T07:58:35.534 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1623.gd0e6828a.el9.noarc 7/7 2024-09-11T07:58:35.573 INFO:teuthology.orchestra.run.smithi181.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 2.9 MB/s | 245 kB 00:00 2024-09-11T07:58:35.998 INFO:teuthology.orchestra.run.smithi181.stdout:(8/12): libgfortran-11.5.0-2.el9.x86_64.rpm 474 kB/s | 801 kB 00:01 2024-09-11T07:58:36.173 INFO:teuthology.orchestra.run.smithi181.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.4 MB/s | 250 kB 00:00 2024-09-11T07:58:36.374 INFO:teuthology.orchestra.run.smithi181.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 6.4 MB/s | 5.1 MB 00:00 2024-09-11T07:58:36.474 INFO:teuthology.orchestra.run.smithi181.stdout:(11/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 4.7 MB/s | 4.9 MB 00:01 2024-09-11T07:58:36.594 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1623.gd0e6828a.el9.noarc 7/7 2024-09-11T07:58:36.594 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1623.gd0e6828a.el9. 1/7 2024-09-11T07:58:36.594 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1623.gd0e6828a.el9.noarc 2/7 2024-09-11T07:58:36.594 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1623.gd0e6828a.el9.n 3/7 2024-09-11T07:58:36.594 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-11T07:58:36.594 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-11T07:58:36.594 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-11T07:58:36.877 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-11T07:58:36.877 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:58:36.877 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-09-11T07:58:36.877 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-grafana-dashboards-2:17.2.7-1623.gd0e6828a.el9.noarch 2024-09-11T07:58:36.877 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-dashboard-2:17.2.7-1623.gd0e6828a.el9.noarch 2024-09-11T07:58:36.877 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-prometheus-alerts-2:17.2.7-1623.gd0e6828a.el9.noarch 2024-09-11T07:58:36.877 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-11T07:58:36.877 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-11T07:58:36.877 INFO:teuthology.orchestra.run.smithi150.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-11T07:58:36.877 INFO:teuthology.orchestra.run.smithi150.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-11T07:58:36.877 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:58:36.878 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-09-11T07:58:37.127 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-11T07:58:37.709 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:01 ago on Wed 11 Sep 2024 07:57:36 AM UTC. 2024-09-11T07:58:37.843 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-09-11T07:58:37.844 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================= 2024-09-11T07:58:37.844 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2024-09-11T07:58:37.844 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================= 2024-09-11T07:58:37.844 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-09-11T07:58:37.844 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1623.gd0e6828a.el9 ceph-noarch 7.4 M 2024-09-11T07:58:37.844 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2024-09-11T07:58:37.844 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-11T07:58:37.844 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-11T07:58:37.844 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-11T07:58:37.844 INFO:teuthology.orchestra.run.smithi150.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-11T07:58:37.844 INFO:teuthology.orchestra.run.smithi150.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-11T07:58:37.844 INFO:teuthology.orchestra.run.smithi150.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-11T07:58:37.844 INFO:teuthology.orchestra.run.smithi150.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-11T07:58:37.845 INFO:teuthology.orchestra.run.smithi150.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-11T07:58:37.845 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-11T07:58:37.845 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-11T07:58:37.845 INFO:teuthology.orchestra.run.smithi150.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-11T07:58:37.845 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:58:37.845 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-09-11T07:58:37.845 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================= 2024-09-11T07:58:37.845 INFO:teuthology.orchestra.run.smithi150.stdout:Install 12 Packages 2024-09-11T07:58:37.845 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:58:37.846 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 38 M 2024-09-11T07:58:37.846 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 212 M 2024-09-11T07:58:37.846 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-09-11T07:58:38.546 INFO:teuthology.orchestra.run.smithi150.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.3 MB/s | 191 kB 00:00 2024-09-11T07:58:38.596 INFO:teuthology.orchestra.run.smithi150.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 4.0 MB/s | 801 kB 00:00 2024-09-11T07:58:38.630 INFO:teuthology.orchestra.run.smithi150.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 392 kB/s | 33 kB 00:00 2024-09-11T07:58:38.672 INFO:teuthology.orchestra.run.smithi150.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 440 kB/s | 18 kB 00:00 2024-09-11T07:58:38.722 INFO:teuthology.orchestra.run.smithi150.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 769 kB/s | 38 kB 00:00 2024-09-11T07:58:38.782 INFO:teuthology.orchestra.run.smithi181.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 6.2 MB/s | 16 MB 00:02 2024-09-11T07:58:38.783 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-11T07:58:38.783 INFO:teuthology.orchestra.run.smithi181.stdout:Total 8.2 MB/s | 38 MB 00:04 2024-09-11T07:58:38.897 INFO:teuthology.orchestra.run.smithi150.stdout:(6/12): ceph-mgr-diskprediction-local-17.2.7-16 15 MB/s | 7.4 MB 00:00 2024-09-11T07:58:38.956 INFO:teuthology.orchestra.run.smithi150.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 8.3 MB/s | 3.0 MB 00:00 2024-09-11T07:58:39.065 INFO:teuthology.orchestra.run.smithi150.stdout:(8/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 14 MB/s | 4.9 MB 00:00 2024-09-11T07:58:39.140 INFO:teuthology.orchestra.run.smithi150.stdout:(9/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 28 MB/s | 5.1 MB 00:00 2024-09-11T07:58:39.143 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-11T07:58:39.165 INFO:teuthology.orchestra.run.smithi150.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 2.4 MB/s | 250 kB 00:00 2024-09-11T07:58:39.169 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-11T07:58:39.169 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-11T07:58:39.190 INFO:teuthology.orchestra.run.smithi150.stdout:(11/12): python3-devel-3.9.19-8.el9.x86_64.rpm 836 kB/s | 245 kB 00:00 2024-09-11T07:58:39.424 INFO:teuthology.orchestra.run.smithi150.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 57 MB/s | 16 MB 00:00 2024-09-11T07:58:39.425 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-09-11T07:58:39.425 INFO:teuthology.orchestra.run.smithi150.stdout:Total 24 MB/s | 38 MB 00:01 2024-09-11T07:58:39.687 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-11T07:58:39.687 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-11T07:58:39.779 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-09-11T07:58:39.805 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-09-11T07:58:39.805 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-09-11T07:58:40.328 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-09-11T07:58:40.328 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-09-11T07:58:40.445 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-11T07:58:40.527 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-11T07:58:40.563 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-11T07:58:40.689 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-11T07:58:40.731 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-11T07:58:40.925 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-11T07:58:40.955 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-11T07:58:41.038 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-11T07:58:41.109 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-09-11T07:58:41.208 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-11T07:58:41.256 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-11T07:58:41.390 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-11T07:58:41.440 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-11T07:58:41.636 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-11T07:58:41.673 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-11T07:58:41.694 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-11T07:58:41.750 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-11T07:58:41.810 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-11T07:58:42.406 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-11T07:58:42.527 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-11T07:58:43.080 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-11T07:58:43.804 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-11T07:58:44.119 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-11T07:58:44.146 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1623.gd0e68 12/12 2024-09-11T07:58:44.862 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-11T07:58:44.888 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1623.gd0e68 12/12 2024-09-11T07:58:46.798 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1623.gd0e68 12/12 2024-09-11T07:58:46.799 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1623.gd0e68 1/12 2024-09-11T07:58:46.799 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-11T07:58:46.799 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-11T07:58:46.799 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-11T07:58:46.799 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-11T07:58:46.799 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-11T07:58:46.799 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-11T07:58:46.799 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-11T07:58:46.799 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-11T07:58:46.799 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-11T07:58:46.799 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-11T07:58:47.245 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-11T07:58:47.245 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:58:47.245 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-11T07:58:47.245 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1623.gd0e6828a.el9.noarch 2024-09-11T07:58:47.245 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-11T07:58:47.245 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-11T07:58:47.245 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-11T07:58:47.245 INFO:teuthology.orchestra.run.smithi181.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-11T07:58:47.245 INFO:teuthology.orchestra.run.smithi181.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-11T07:58:47.246 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-11T07:58:47.246 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-11T07:58:47.246 INFO:teuthology.orchestra.run.smithi181.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-11T07:58:47.246 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-11T07:58:47.246 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-11T07:58:47.246 INFO:teuthology.orchestra.run.smithi181.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-11T07:58:47.246 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:58:47.246 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-11T07:58:47.442 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1623.gd0e68 12/12 2024-09-11T07:58:47.442 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1623.gd0e68 1/12 2024-09-11T07:58:47.442 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-11T07:58:47.442 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-11T07:58:47.442 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-11T07:58:47.442 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-11T07:58:47.442 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-11T07:58:47.442 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-11T07:58:47.443 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-11T07:58:47.443 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-11T07:58:47.443 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-11T07:58:47.443 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-11T07:58:47.626 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-rook 2024-09-11T07:58:47.929 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-11T07:58:47.929 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:58:47.929 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-09-11T07:58:47.929 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1623.gd0e6828a.el9.noarch 2024-09-11T07:58:47.929 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-11T07:58:47.929 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-11T07:58:47.929 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-11T07:58:47.929 INFO:teuthology.orchestra.run.smithi150.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-11T07:58:47.929 INFO:teuthology.orchestra.run.smithi150.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-11T07:58:47.930 INFO:teuthology.orchestra.run.smithi150.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-11T07:58:47.930 INFO:teuthology.orchestra.run.smithi150.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-11T07:58:47.930 INFO:teuthology.orchestra.run.smithi150.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-11T07:58:47.930 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-11T07:58:47.930 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-11T07:58:47.930 INFO:teuthology.orchestra.run.smithi150.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-11T07:58:47.930 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:58:47.930 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-09-11T07:58:48.207 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:15 ago on Wed 11 Sep 2024 07:57:33 AM UTC. 2024-09-11T07:58:48.317 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-mgr-rook 2024-09-11T07:58:48.348 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-11T07:58:48.349 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T07:58:48.349 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-11T07:58:48.349 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T07:58:48.349 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-11T07:58:48.349 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-rook noarch 2:17.2.7-1623.gd0e6828a.el9 ceph-noarch 47 k 2024-09-11T07:58:48.350 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-11T07:58:48.350 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-11T07:58:48.350 INFO:teuthology.orchestra.run.smithi181.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-11T07:58:48.350 INFO:teuthology.orchestra.run.smithi181.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-11T07:58:48.350 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-11T07:58:48.350 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-11T07:58:48.350 INFO:teuthology.orchestra.run.smithi181.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-11T07:58:48.350 INFO:teuthology.orchestra.run.smithi181.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-11T07:58:48.350 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-11T07:58:48.350 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-11T07:58:48.350 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-11T07:58:48.350 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-11T07:58:48.350 INFO:teuthology.orchestra.run.smithi181.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-11T07:58:48.350 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:58:48.351 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-11T07:58:48.351 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T07:58:48.351 INFO:teuthology.orchestra.run.smithi181.stdout:Install 13 Packages 2024-09-11T07:58:48.351 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:58:48.351 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 2.2 M 2024-09-11T07:58:48.351 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 26 M 2024-09-11T07:58:48.351 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-11T07:58:48.547 INFO:teuthology.orchestra.run.smithi181.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 223 kB/s | 26 kB 00:00 2024-09-11T07:58:48.571 INFO:teuthology.orchestra.run.smithi181.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 137 kB/s | 19 kB 00:00 2024-09-11T07:58:48.597 INFO:teuthology.orchestra.run.smithi181.stdout:(3/13): ceph-mgr-rook-17.2.7-1623.gd0e6828a.el9 287 kB/s | 47 kB 00:00 2024-09-11T07:58:48.722 INFO:teuthology.orchestra.run.smithi181.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.0 MB/s | 159 kB 00:00 2024-09-11T07:58:48.747 INFO:teuthology.orchestra.run.smithi181.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.2 MB/s | 222 kB 00:00 2024-09-11T07:58:48.781 INFO:teuthology.orchestra.run.smithi181.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 916 kB/s | 54 kB 00:00 2024-09-11T07:58:48.814 INFO:teuthology.orchestra.run.smithi181.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 480 kB/s | 32 kB 00:00 2024-09-11T07:58:48.840 INFO:teuthology.orchestra.run.smithi181.stdout:(8/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.1 MB/s | 279 kB 00:00 2024-09-11T07:58:48.857 INFO:teuthology.orchestra.run.smithi181.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 191 kB/s | 14 kB 00:00 2024-09-11T07:58:48.882 INFO:teuthology.orchestra.run.smithi181.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 3.2 MB/s | 59 kB 00:00 2024-09-11T07:58:48.889 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:12 ago on Wed 11 Sep 2024 07:57:36 AM UTC. 2024-09-11T07:58:48.907 INFO:teuthology.orchestra.run.smithi181.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 2.4 MB/s | 220 kB 00:00 2024-09-11T07:58:48.933 INFO:teuthology.orchestra.run.smithi181.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-09-11T07:58:48.974 INFO:teuthology.orchestra.run.smithi181.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.6 MB/s | 1.0 MB 00:00 2024-09-11T07:58:48.977 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-11T07:58:48.978 INFO:teuthology.orchestra.run.smithi181.stdout:Total 3.5 MB/s | 2.2 MB 00:00 2024-09-11T07:58:49.024 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-09-11T07:58:49.025 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-09-11T07:58:49.026 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2024-09-11T07:58:49.026 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-09-11T07:58:49.026 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-09-11T07:58:49.026 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-rook noarch 2:17.2.7-1623.gd0e6828a.el9 ceph-noarch 47 k 2024-09-11T07:58:49.026 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2024-09-11T07:58:49.026 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-11T07:58:49.026 INFO:teuthology.orchestra.run.smithi150.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-11T07:58:49.026 INFO:teuthology.orchestra.run.smithi150.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-11T07:58:49.026 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-11T07:58:49.026 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-11T07:58:49.026 INFO:teuthology.orchestra.run.smithi150.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-11T07:58:49.026 INFO:teuthology.orchestra.run.smithi150.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-11T07:58:49.026 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-11T07:58:49.027 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-11T07:58:49.027 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-11T07:58:49.027 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-11T07:58:49.027 INFO:teuthology.orchestra.run.smithi150.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-11T07:58:49.027 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:58:49.027 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-09-11T07:58:49.027 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-09-11T07:58:49.027 INFO:teuthology.orchestra.run.smithi150.stdout:Install 13 Packages 2024-09-11T07:58:49.027 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:58:49.028 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 2.2 M 2024-09-11T07:58:49.028 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 26 M 2024-09-11T07:58:49.028 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-09-11T07:58:49.044 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-11T07:58:49.059 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-11T07:58:49.059 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-11T07:58:49.153 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-11T07:58:49.153 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-11T07:58:49.329 INFO:teuthology.orchestra.run.smithi150.stdout:(1/13): ceph-mgr-rook-17.2.7-1623.gd0e6828a.el9 274 kB/s | 47 kB 00:00 2024-09-11T07:58:49.674 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-11T07:58:49.721 INFO:teuthology.orchestra.run.smithi150.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 34 kB/s | 19 kB 00:00 2024-09-11T07:58:49.763 INFO:teuthology.orchestra.run.smithi150.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 42 kB/s | 26 kB 00:00 2024-09-11T07:58:49.850 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-11T07:58:49.926 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-11T07:58:49.981 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-11T07:58:50.018 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-11T07:58:50.060 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-11T07:58:50.138 INFO:teuthology.orchestra.run.smithi150.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 275 kB/s | 222 kB 00:00 2024-09-11T07:58:50.162 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-11T07:58:50.272 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-11T07:58:50.320 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-11T07:58:50.422 INFO:teuthology.orchestra.run.smithi150.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 227 kB/s | 159 kB 00:00 2024-09-11T07:58:50.481 INFO:teuthology.orchestra.run.smithi150.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 555 kB/s | 32 kB 00:00 2024-09-11T07:58:50.506 INFO:teuthology.orchestra.run.smithi150.stdout:(7/13): python3-certifi-2023.05.07-4.el9.noarch 567 kB/s | 14 kB 00:00 2024-09-11T07:58:50.539 INFO:teuthology.orchestra.run.smithi150.stdout:(8/13): python3-google-auth-2.34.0-1.el9.noarch 6.5 MB/s | 220 kB 00:00 2024-09-11T07:58:50.590 INFO:teuthology.orchestra.run.smithi150.stdout:(9/13): python3-kubernetes-26.1.0-2.el9.noarch. 20 MB/s | 1.0 MB 00:00 2024-09-11T07:58:50.615 INFO:teuthology.orchestra.run.smithi150.stdout:(10/13): python3-pyasn1-modules-0.4.8-6.el9.noa 327 kB/s | 279 kB 00:00 2024-09-11T07:58:50.640 INFO:teuthology.orchestra.run.smithi150.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 1.1 MB/s | 59 kB 00:00 2024-09-11T07:58:50.682 INFO:teuthology.orchestra.run.smithi150.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.3 MB/s | 90 kB 00:00 2024-09-11T07:58:50.741 INFO:teuthology.orchestra.run.smithi150.stdout:(13/13): python3-requests-oauthlib-1.3.0-12.el9 89 kB/s | 54 kB 00:00 2024-09-11T07:58:50.744 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-09-11T07:58:50.744 INFO:teuthology.orchestra.run.smithi150.stdout:Total 1.3 MB/s | 2.2 MB 00:01 2024-09-11T07:58:50.816 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-09-11T07:58:50.832 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-09-11T07:58:50.832 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-09-11T07:58:50.922 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-09-11T07:58:50.923 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-09-11T07:58:51.042 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-11T07:58:51.093 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-11T07:58:51.126 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-11T07:58:51.180 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-11T07:58:51.213 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-rook-2:17.2.7-1623.gd0e6828a.el9.noarch 13/13 2024-09-11T07:58:51.435 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-09-11T07:58:51.598 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-11T07:58:51.659 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-11T07:58:51.715 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-11T07:58:51.751 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-11T07:58:51.793 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-11T07:58:51.895 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-11T07:58:51.997 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-11T07:58:52.027 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1623.gd0e6828a.el9.noarch 13/13 2024-09-11T07:58:52.027 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1623.gd0e6828a.el9.noarch 1/13 2024-09-11T07:58:52.027 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-11T07:58:52.027 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-11T07:58:52.027 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-11T07:58:52.027 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-11T07:58:52.027 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-11T07:58:52.027 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-11T07:58:52.027 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-11T07:58:52.027 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-11T07:58:52.027 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-11T07:58:52.027 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-11T07:58:52.028 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-11T07:58:52.045 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-11T07:58:52.368 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-11T07:58:52.368 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:58:52.368 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-11T07:58:52.368 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-rook-2:17.2.7-1623.gd0e6828a.el9.noarch 2024-09-11T07:58:52.368 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-11T07:58:52.368 INFO:teuthology.orchestra.run.smithi181.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-11T07:58:52.368 INFO:teuthology.orchestra.run.smithi181.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-11T07:58:52.368 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-11T07:58:52.368 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-11T07:58:52.368 INFO:teuthology.orchestra.run.smithi181.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-11T07:58:52.369 INFO:teuthology.orchestra.run.smithi181.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-11T07:58:52.369 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-11T07:58:52.369 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-11T07:58:52.369 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-11T07:58:52.369 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-11T07:58:52.369 INFO:teuthology.orchestra.run.smithi181.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-11T07:58:52.369 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:58:52.369 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-11T07:58:52.619 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-cephadm 2024-09-11T07:58:52.768 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-11T07:58:52.817 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-11T07:58:52.851 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-11T07:58:52.897 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-11T07:58:52.920 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-rook-2:17.2.7-1623.gd0e6828a.el9.noarch 13/13 2024-09-11T07:58:53.188 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:20 ago on Wed 11 Sep 2024 07:57:33 AM UTC. 2024-09-11T07:58:53.321 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-11T07:58:53.322 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T07:58:53.322 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-11T07:58:53.323 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T07:58:53.323 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-11T07:58:53.323 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1623.gd0e6828a.el9 ceph-noarch 124 k 2024-09-11T07:58:53.323 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-11T07:58:53.323 INFO:teuthology.orchestra.run.smithi181.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-11T07:58:53.323 INFO:teuthology.orchestra.run.smithi181.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-11T07:58:53.323 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-11T07:58:53.323 INFO:teuthology.orchestra.run.smithi181.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-11T07:58:53.323 INFO:teuthology.orchestra.run.smithi181.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-11T07:58:53.323 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:58:53.323 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-11T07:58:53.323 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T07:58:53.324 INFO:teuthology.orchestra.run.smithi181.stdout:Install 6 Packages 2024-09-11T07:58:53.324 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:58:53.324 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 7.0 M 2024-09-11T07:58:53.324 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 33 M 2024-09-11T07:58:53.324 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-11T07:58:53.510 INFO:teuthology.orchestra.run.smithi181.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.8 MB/s | 248 kB 00:00 2024-09-11T07:58:53.561 INFO:teuthology.orchestra.run.smithi181.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 11 MB/s | 548 kB 00:00 2024-09-11T07:58:53.586 INFO:teuthology.orchestra.run.smithi181.stdout:(3/6): ceph-mgr-cephadm-17.2.7-1623.gd0e6828a.e 591 kB/s | 124 kB 00:00 2024-09-11T07:58:53.611 INFO:teuthology.orchestra.run.smithi181.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-09-11T07:58:53.678 INFO:teuthology.orchestra.run.smithi181.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 812 kB/s | 74 kB 00:00 2024-09-11T07:58:53.780 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1623.gd0e6828a.el9.noarch 13/13 2024-09-11T07:58:53.780 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1623.gd0e6828a.el9.noarch 1/13 2024-09-11T07:58:53.780 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-11T07:58:53.780 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-11T07:58:53.780 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-11T07:58:53.780 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-11T07:58:53.780 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-11T07:58:53.781 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-11T07:58:53.781 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-11T07:58:53.781 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-11T07:58:53.781 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-11T07:58:53.781 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-11T07:58:53.781 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-11T07:58:53.811 INFO:teuthology.orchestra.run.smithi181.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 6.0 MB 00:00 2024-09-11T07:58:53.814 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-11T07:58:53.814 INFO:teuthology.orchestra.run.smithi181.stdout:Total 14 MB/s | 7.0 MB 00:00 2024-09-11T07:58:53.885 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-11T07:58:53.896 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-11T07:58:53.897 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-11T07:58:53.996 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-11T07:58:53.997 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-11T07:58:54.136 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-11T07:58:54.136 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:58:54.136 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-09-11T07:58:54.136 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-rook-2:17.2.7-1623.gd0e6828a.el9.noarch 2024-09-11T07:58:54.136 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-11T07:58:54.136 INFO:teuthology.orchestra.run.smithi150.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-11T07:58:54.136 INFO:teuthology.orchestra.run.smithi150.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-11T07:58:54.136 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-11T07:58:54.137 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-11T07:58:54.137 INFO:teuthology.orchestra.run.smithi150.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-11T07:58:54.137 INFO:teuthology.orchestra.run.smithi150.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-11T07:58:54.137 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-11T07:58:54.137 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-11T07:58:54.137 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-11T07:58:54.138 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-11T07:58:54.138 INFO:teuthology.orchestra.run.smithi150.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-11T07:58:54.138 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:58:54.138 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-09-11T07:58:54.299 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-11T07:58:54.417 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-mgr-cephadm 2024-09-11T07:58:54.469 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-11T07:58:54.522 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-11T07:58:54.984 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:18 ago on Wed 11 Sep 2024 07:57:36 AM UTC. 2024-09-11T07:58:55.044 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-11T07:58:55.119 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-09-11T07:58:55.119 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-09-11T07:58:55.119 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2024-09-11T07:58:55.120 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-09-11T07:58:55.120 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-09-11T07:58:55.120 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1623.gd0e6828a.el9 ceph-noarch 124 k 2024-09-11T07:58:55.120 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2024-09-11T07:58:55.120 INFO:teuthology.orchestra.run.smithi150.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-11T07:58:55.120 INFO:teuthology.orchestra.run.smithi150.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-11T07:58:55.120 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-11T07:58:55.120 INFO:teuthology.orchestra.run.smithi150.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-11T07:58:55.120 INFO:teuthology.orchestra.run.smithi150.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-11T07:58:55.120 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:58:55.120 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-09-11T07:58:55.120 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-09-11T07:58:55.121 INFO:teuthology.orchestra.run.smithi150.stdout:Install 6 Packages 2024-09-11T07:58:55.121 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:58:55.121 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 7.0 M 2024-09-11T07:58:55.121 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 33 M 2024-09-11T07:58:55.121 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-09-11T07:58:55.133 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-11T07:58:55.196 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-11T07:58:55.225 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1623.gd0e6828a.el9.noarch 6/6 2024-09-11T07:58:55.535 INFO:teuthology.orchestra.run.smithi150.stdout:(1/6): ceph-mgr-cephadm-17.2.7-1623.gd0e6828a.e 577 kB/s | 124 kB 00:00 2024-09-11T07:58:55.602 INFO:teuthology.orchestra.run.smithi150.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 10 MB/s | 548 kB 00:00 2024-09-11T07:58:55.628 INFO:teuthology.orchestra.run.smithi150.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.2 MB/s | 58 kB 00:00 2024-09-11T07:58:55.653 INFO:teuthology.orchestra.run.smithi150.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 3.1 MB/s | 74 kB 00:00 2024-09-11T07:58:55.955 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1623.gd0e6828a.el9.noarch 6/6 2024-09-11T07:58:55.955 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1623.gd0e6828a.el9.noarch 1/6 2024-09-11T07:58:55.955 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-11T07:58:55.955 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-11T07:58:55.955 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-11T07:58:55.955 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-11T07:58:56.211 INFO:teuthology.orchestra.run.smithi150.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 279 kB/s | 248 kB 00:00 2024-09-11T07:58:56.242 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-11T07:58:56.242 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:58:56.242 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-11T07:58:56.242 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-cephadm-2:17.2.7-1623.gd0e6828a.el9.noarch 2024-09-11T07:58:56.242 INFO:teuthology.orchestra.run.smithi181.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-11T07:58:56.242 INFO:teuthology.orchestra.run.smithi181.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-11T07:58:56.242 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-11T07:58:56.243 INFO:teuthology.orchestra.run.smithi181.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-11T07:58:56.243 INFO:teuthology.orchestra.run.smithi181.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-11T07:58:56.243 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:58:56.243 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-11T07:58:56.502 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-fuse 2024-09-11T07:58:56.745 INFO:teuthology.orchestra.run.smithi150.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.2 MB/s | 6.0 MB 00:01 2024-09-11T07:58:56.746 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-09-11T07:58:56.747 INFO:teuthology.orchestra.run.smithi150.stdout:Total 4.3 MB/s | 7.0 MB 00:01 2024-09-11T07:58:56.819 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-09-11T07:58:56.831 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-09-11T07:58:56.831 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-09-11T07:58:56.931 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-09-11T07:58:56.931 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-09-11T07:58:57.066 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:24 ago on Wed 11 Sep 2024 07:57:33 AM UTC. 2024-09-11T07:58:57.199 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-11T07:58:57.200 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T07:58:57.200 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-11T07:58:57.200 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T07:58:57.200 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-11T07:58:57.200 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-fuse x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 813 k 2024-09-11T07:58:57.200 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-11T07:58:57.200 INFO:teuthology.orchestra.run.smithi181.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-11T07:58:57.201 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:58:57.201 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-11T07:58:57.201 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T07:58:57.201 INFO:teuthology.orchestra.run.smithi181.stdout:Install 2 Packages 2024-09-11T07:58:57.201 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:58:57.201 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 893 k 2024-09-11T07:58:57.201 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 2.6 M 2024-09-11T07:58:57.201 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-11T07:58:57.239 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-09-11T07:58:57.371 INFO:teuthology.orchestra.run.smithi181.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 646 kB/s | 80 kB 00:00 2024-09-11T07:58:57.402 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-11T07:58:57.446 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-11T07:58:57.538 INFO:teuthology.orchestra.run.smithi181.stdout:(2/2): ceph-fuse-17.2.7-1623.gd0e6828a.el9.x86_ 2.7 MB/s | 813 kB 00:00 2024-09-11T07:58:57.539 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-11T07:58:57.539 INFO:teuthology.orchestra.run.smithi181.stdout:Total 2.6 MB/s | 893 kB 00:00 2024-09-11T07:58:57.550 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-11T07:58:57.572 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-11T07:58:57.572 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-11T07:58:57.657 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-11T07:58:57.657 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-11T07:58:57.847 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-11T07:58:57.996 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-11T07:58:58.017 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-11T07:58:58.062 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-fuse-2:17.2.7-1623.gd0e6828a.el9.x86_64 2/2 2024-09-11T07:58:58.107 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-11T07:58:58.171 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-11T07:58:58.200 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1623.gd0e6828a.el9.noarch 6/6 2024-09-11T07:58:58.736 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1623.gd0e6828a.el9.x86_64 2/2 2024-09-11T07:58:58.736 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-fuse-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/2 2024-09-11T07:58:58.977 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1623.gd0e6828a.el9.noarch 6/6 2024-09-11T07:58:58.978 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1623.gd0e6828a.el9.noarch 1/6 2024-09-11T07:58:58.978 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-11T07:58:58.978 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-11T07:58:58.978 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-11T07:58:58.978 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-11T07:58:59.016 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-11T07:58:59.016 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:58:59.016 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-11T07:58:59.016 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-fuse-2:17.2.7-1623.gd0e6828a.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-11T07:58:59.016 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:58:59.017 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-11T07:58:59.250 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-11T07:58:59.250 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:58:59.250 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-09-11T07:58:59.250 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-cephadm-2:17.2.7-1623.gd0e6828a.el9.noarch 2024-09-11T07:58:59.250 INFO:teuthology.orchestra.run.smithi150.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-11T07:58:59.250 INFO:teuthology.orchestra.run.smithi150.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-11T07:58:59.250 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-11T07:58:59.250 INFO:teuthology.orchestra.run.smithi150.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-11T07:58:59.250 INFO:teuthology.orchestra.run.smithi150.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-11T07:58:59.251 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:58:59.251 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-09-11T07:58:59.289 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-volume 2024-09-11T07:58:59.535 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-fuse 2024-09-11T07:58:59.869 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:26 ago on Wed 11 Sep 2024 07:57:33 AM UTC. 2024-09-11T07:59:00.009 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-11T07:59:00.009 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T07:59:00.009 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-11T07:59:00.009 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T07:59:00.009 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-11T07:59:00.009 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-volume noarch 2:17.2.7-1623.gd0e6828a.el9 ceph-noarch 270 k 2024-09-11T07:59:00.009 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:59:00.009 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-11T07:59:00.009 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T07:59:00.009 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-11T07:59:00.009 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:59:00.009 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 270 k 2024-09-11T07:59:00.010 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 1.3 M 2024-09-11T07:59:00.010 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-11T07:59:00.098 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:24 ago on Wed 11 Sep 2024 07:57:36 AM UTC. 2024-09-11T07:59:00.230 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-09-11T07:59:00.231 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-09-11T07:59:00.232 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2024-09-11T07:59:00.232 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-09-11T07:59:00.232 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-09-11T07:59:00.232 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-fuse x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 813 k 2024-09-11T07:59:00.232 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2024-09-11T07:59:00.232 INFO:teuthology.orchestra.run.smithi150.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-11T07:59:00.233 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:59:00.233 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-09-11T07:59:00.233 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-09-11T07:59:00.233 INFO:teuthology.orchestra.run.smithi150.stdout:Install 2 Packages 2024-09-11T07:59:00.233 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:59:00.234 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 893 k 2024-09-11T07:59:00.234 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 2.6 M 2024-09-11T07:59:00.234 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-09-11T07:59:00.236 INFO:teuthology.orchestra.run.smithi181.stdout:ceph-volume-17.2.7-1623.gd0e6828a.el9.noarch.rp 1.2 MB/s | 270 kB 00:00 2024-09-11T07:59:00.236 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-11T07:59:00.236 INFO:teuthology.orchestra.run.smithi181.stdout:Total 1.2 MB/s | 270 kB 00:00 2024-09-11T07:59:00.237 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-11T07:59:00.245 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-11T07:59:00.245 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-11T07:59:00.278 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-11T07:59:00.279 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-11T07:59:00.463 INFO:teuthology.orchestra.run.smithi150.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 437 kB/s | 80 kB 00:00 2024-09-11T07:59:00.500 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-11T07:59:00.546 INFO:teuthology.orchestra.run.smithi150.stdout:(2/2): ceph-fuse-17.2.7-1623.gd0e6828a.el9.x86_ 3.0 MB/s | 813 kB 00:00 2024-09-11T07:59:00.547 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-09-11T07:59:00.547 INFO:teuthology.orchestra.run.smithi150.stdout:Total 2.8 MB/s | 893 kB 00:00 2024-09-11T07:59:00.559 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-09-11T07:59:00.582 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-09-11T07:59:00.582 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-09-11T07:59:00.621 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-volume-2:17.2.7-1623.gd0e6828a.el9.noarch 1/1 2024-09-11T07:59:00.637 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-volume-2:17.2.7-1623.gd0e6828a.el9.noarch 1/1 2024-09-11T07:59:00.638 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T07:59:00.638 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-11T07:59:00.638 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:59:00.665 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-09-11T07:59:00.666 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-09-11T07:59:00.861 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-09-11T07:59:01.008 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-11T07:59:01.087 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-fuse-2:17.2.7-1623.gd0e6828a.el9.x86_64 2/2 2024-09-11T07:59:01.582 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-volume-2:17.2.7-1623.gd0e6828a.el9.noarch 1/1 2024-09-11T07:59:01.582 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:59:01.583 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-11T07:59:01.583 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-volume-2:17.2.7-1623.gd0e6828a.el9.noarch 2024-09-11T07:59:01.583 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:59:01.583 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-11T07:59:01.757 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1623.gd0e6828a.el9.x86_64 2/2 2024-09-11T07:59:01.757 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-fuse-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/2 2024-09-11T07:59:01.883 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install librados-devel 2024-09-11T07:59:02.042 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-11T07:59:02.042 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:59:02.042 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-09-11T07:59:02.042 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-fuse-2:17.2.7-1623.gd0e6828a.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-11T07:59:02.042 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:59:02.042 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-09-11T07:59:02.344 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-volume 2024-09-11T07:59:02.452 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:29 ago on Wed 11 Sep 2024 07:57:33 AM UTC. 2024-09-11T07:59:02.591 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-11T07:59:02.592 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T07:59:02.592 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-09-11T07:59:02.592 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T07:59:02.592 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-11T07:59:02.592 INFO:teuthology.orchestra.run.smithi181.stdout: librados-devel x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 126 k 2024-09-11T07:59:02.592 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:59:02.592 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-11T07:59:02.592 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T07:59:02.592 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-11T07:59:02.592 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:59:02.592 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 126 k 2024-09-11T07:59:02.592 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 448 k 2024-09-11T07:59:02.593 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-11T07:59:02.774 INFO:teuthology.orchestra.run.smithi181.stdout:librados-devel-17.2.7-1623.gd0e6828a.el9.x86_64 697 kB/s | 126 kB 00:00 2024-09-11T07:59:02.774 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-11T07:59:02.775 INFO:teuthology.orchestra.run.smithi181.stdout:Total 692 kB/s | 126 kB 00:00 2024-09-11T07:59:02.775 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-11T07:59:02.780 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-11T07:59:02.780 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-11T07:59:02.844 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-11T07:59:02.845 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-11T07:59:02.923 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:26 ago on Wed 11 Sep 2024 07:57:36 AM UTC. 2024-09-11T07:59:03.008 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-11T07:59:03.062 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-09-11T07:59:03.062 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-09-11T07:59:03.063 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2024-09-11T07:59:03.063 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-09-11T07:59:03.063 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-09-11T07:59:03.063 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-volume noarch 2:17.2.7-1623.gd0e6828a.el9 ceph-noarch 270 k 2024-09-11T07:59:03.063 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:59:03.063 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-09-11T07:59:03.063 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-09-11T07:59:03.063 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2024-09-11T07:59:03.063 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:59:03.063 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 270 k 2024-09-11T07:59:03.063 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 1.3 M 2024-09-11T07:59:03.063 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-09-11T07:59:03.112 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librados-devel-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T07:59:03.267 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-volume-17.2.7-1623.gd0e6828a.el9.noarch.rp 1.3 MB/s | 270 kB 00:00 2024-09-11T07:59:03.268 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-09-11T07:59:03.268 INFO:teuthology.orchestra.run.smithi150.stdout:Total 1.3 MB/s | 270 kB 00:00 2024-09-11T07:59:03.268 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-09-11T07:59:03.276 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-09-11T07:59:03.276 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-09-11T07:59:03.309 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-09-11T07:59:03.310 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-09-11T07:59:03.518 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-09-11T07:59:03.578 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-volume-2:17.2.7-1623.gd0e6828a.el9.noarch 1/1 2024-09-11T07:59:03.594 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-volume-2:17.2.7-1623.gd0e6828a.el9.noarch 1/1 2024-09-11T07:59:03.594 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T07:59:03.594 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-11T07:59:03.594 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:59:03.609 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados-devel-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T07:59:03.882 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados-devel-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T07:59:03.883 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:59:03.883 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-11T07:59:03.883 INFO:teuthology.orchestra.run.smithi181.stdout: librados-devel-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:59:03.883 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:59:03.883 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-11T07:59:04.186 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install libcephfs2 2024-09-11T07:59:04.599 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-volume-2:17.2.7-1623.gd0e6828a.el9.noarch 1/1 2024-09-11T07:59:04.599 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:59:04.599 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-09-11T07:59:04.599 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-volume-2:17.2.7-1623.gd0e6828a.el9.noarch 2024-09-11T07:59:04.599 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:59:04.599 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-09-11T07:59:04.767 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:31 ago on Wed 11 Sep 2024 07:57:33 AM UTC. 2024-09-11T07:59:04.860 INFO:teuthology.orchestra.run.smithi181.stdout:Package libcephfs2-2:17.2.7-1623.gd0e6828a.el9.x86_64 is already installed. 2024-09-11T07:59:04.903 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-11T07:59:04.905 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-11T07:59:04.905 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-11T07:59:04.908 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install librados-devel 2024-09-11T07:59:04.976 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install libcephfs-devel 2024-09-11T07:59:05.486 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:29 ago on Wed 11 Sep 2024 07:57:36 AM UTC. 2024-09-11T07:59:05.543 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:32 ago on Wed 11 Sep 2024 07:57:33 AM UTC. 2024-09-11T07:59:05.624 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-09-11T07:59:05.624 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-09-11T07:59:05.624 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repo Size 2024-09-11T07:59:05.624 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-09-11T07:59:05.624 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-09-11T07:59:05.624 INFO:teuthology.orchestra.run.smithi150.stdout: librados-devel x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 126 k 2024-09-11T07:59:05.625 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:59:05.625 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-09-11T07:59:05.625 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-09-11T07:59:05.625 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2024-09-11T07:59:05.625 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:59:05.625 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 126 k 2024-09-11T07:59:05.626 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 448 k 2024-09-11T07:59:05.626 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-09-11T07:59:05.676 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-11T07:59:05.677 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T07:59:05.677 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-09-11T07:59:05.677 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T07:59:05.677 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-11T07:59:05.677 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs-devel x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 25 k 2024-09-11T07:59:05.677 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:59:05.677 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-11T07:59:05.677 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T07:59:05.678 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-11T07:59:05.678 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:59:05.678 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 25 k 2024-09-11T07:59:05.678 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 110 k 2024-09-11T07:59:05.678 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-11T07:59:05.814 INFO:teuthology.orchestra.run.smithi150.stdout:librados-devel-17.2.7-1623.gd0e6828a.el9.x86_64 670 kB/s | 126 kB 00:00 2024-09-11T07:59:05.815 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-09-11T07:59:05.815 INFO:teuthology.orchestra.run.smithi150.stdout:Total 665 kB/s | 126 kB 00:00 2024-09-11T07:59:05.815 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-09-11T07:59:05.820 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-09-11T07:59:05.820 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-09-11T07:59:05.836 INFO:teuthology.orchestra.run.smithi181.stdout:libcephfs-devel-17.2.7-1623.gd0e6828a.el9.x86_6 157 kB/s | 25 kB 00:00 2024-09-11T07:59:05.837 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-11T07:59:05.837 INFO:teuthology.orchestra.run.smithi181.stdout:Total 156 kB/s | 25 kB 00:00 2024-09-11T07:59:05.838 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-11T07:59:05.841 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-11T07:59:05.841 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-11T07:59:05.842 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-11T07:59:05.843 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-11T07:59:05.883 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-09-11T07:59:05.883 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-09-11T07:59:05.927 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-11T07:59:06.032 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-09-11T07:59:06.041 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephfs-devel-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T07:59:06.119 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librados-devel-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T07:59:06.415 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T07:59:06.520 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librados-devel-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T07:59:06.655 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephfs-devel-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T07:59:06.655 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:59:06.655 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-11T07:59:06.656 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs-devel-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:59:06.656 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:59:06.656 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-11T07:59:06.798 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librados-devel-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T07:59:06.798 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:59:06.798 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-09-11T07:59:06.798 INFO:teuthology.orchestra.run.smithi150.stdout: librados-devel-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:59:06.798 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:59:06.798 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-09-11T07:59:06.859 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install librados2 2024-09-11T07:59:07.084 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install libcephfs2 2024-09-11T07:59:07.412 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:34 ago on Wed 11 Sep 2024 07:57:33 AM UTC. 2024-09-11T07:59:07.500 INFO:teuthology.orchestra.run.smithi181.stdout:Package librados2-2:17.2.7-1623.gd0e6828a.el9.x86_64 is already installed. 2024-09-11T07:59:07.542 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-11T07:59:07.543 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-11T07:59:07.543 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-11T07:59:07.621 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install librbd1 2024-09-11T07:59:07.665 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:31 ago on Wed 11 Sep 2024 07:57:36 AM UTC. 2024-09-11T07:59:07.757 INFO:teuthology.orchestra.run.smithi150.stdout:Package libcephfs2-2:17.2.7-1623.gd0e6828a.el9.x86_64 is already installed. 2024-09-11T07:59:07.802 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-09-11T07:59:07.803 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2024-09-11T07:59:07.803 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-09-11T07:59:07.876 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install libcephfs-devel 2024-09-11T07:59:08.182 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:35 ago on Wed 11 Sep 2024 07:57:33 AM UTC. 2024-09-11T07:59:08.272 INFO:teuthology.orchestra.run.smithi181.stdout:Package librbd1-2:17.2.7-1623.gd0e6828a.el9.x86_64 is already installed. 2024-09-11T07:59:08.316 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-11T07:59:08.317 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-11T07:59:08.317 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-11T07:59:08.391 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-rados 2024-09-11T07:59:08.477 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:32 ago on Wed 11 Sep 2024 07:57:36 AM UTC. 2024-09-11T07:59:08.612 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-09-11T07:59:08.613 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-09-11T07:59:08.613 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repo Size 2024-09-11T07:59:08.613 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-09-11T07:59:08.613 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-09-11T07:59:08.614 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs-devel x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 25 k 2024-09-11T07:59:08.614 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:59:08.614 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-09-11T07:59:08.614 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-09-11T07:59:08.614 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2024-09-11T07:59:08.614 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:59:08.614 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 25 k 2024-09-11T07:59:08.614 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 110 k 2024-09-11T07:59:08.614 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-09-11T07:59:08.767 INFO:teuthology.orchestra.run.smithi150.stdout:libcephfs-devel-17.2.7-1623.gd0e6828a.el9.x86_6 163 kB/s | 25 kB 00:00 2024-09-11T07:59:08.768 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-09-11T07:59:08.768 INFO:teuthology.orchestra.run.smithi150.stdout:Total 161 kB/s | 25 kB 00:00 2024-09-11T07:59:08.768 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-09-11T07:59:08.772 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-09-11T07:59:08.772 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-09-11T07:59:08.773 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-09-11T07:59:08.773 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-09-11T07:59:08.860 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-09-11T07:59:08.955 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:35 ago on Wed 11 Sep 2024 07:57:33 AM UTC. 2024-09-11T07:59:08.997 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libcephfs-devel-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T07:59:09.046 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-rados-2:17.2.7-1623.gd0e6828a.el9.x86_64 is already installed. 2024-09-11T07:59:09.089 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-11T07:59:09.090 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-11T07:59:09.090 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-11T07:59:09.164 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-rgw 2024-09-11T07:59:09.385 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T07:59:09.623 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libcephfs-devel-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T07:59:09.623 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:59:09.623 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-09-11T07:59:09.623 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs-devel-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:59:09.623 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:59:09.623 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-09-11T07:59:09.725 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:36 ago on Wed 11 Sep 2024 07:57:33 AM UTC. 2024-09-11T07:59:09.814 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-rgw-2:17.2.7-1623.gd0e6828a.el9.x86_64 is already installed. 2024-09-11T07:59:09.842 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install librados2 2024-09-11T07:59:09.857 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-11T07:59:09.858 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-11T07:59:09.858 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-11T07:59:09.933 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-cephfs 2024-09-11T07:59:10.403 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:34 ago on Wed 11 Sep 2024 07:57:36 AM UTC. 2024-09-11T07:59:10.492 INFO:teuthology.orchestra.run.smithi150.stdout:Package librados2-2:17.2.7-1623.gd0e6828a.el9.x86_64 is already installed. 2024-09-11T07:59:10.499 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:37 ago on Wed 11 Sep 2024 07:57:33 AM UTC. 2024-09-11T07:59:10.536 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-09-11T07:59:10.537 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2024-09-11T07:59:10.537 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-09-11T07:59:10.590 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-cephfs-2:17.2.7-1623.gd0e6828a.el9.x86_64 is already installed. 2024-09-11T07:59:10.602 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install librbd1 2024-09-11T07:59:10.632 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-11T07:59:10.634 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-11T07:59:10.634 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-11T07:59:10.707 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-rbd 2024-09-11T07:59:11.164 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:35 ago on Wed 11 Sep 2024 07:57:36 AM UTC. 2024-09-11T07:59:11.253 INFO:teuthology.orchestra.run.smithi150.stdout:Package librbd1-2:17.2.7-1623.gd0e6828a.el9.x86_64 is already installed. 2024-09-11T07:59:11.268 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:38 ago on Wed 11 Sep 2024 07:57:33 AM UTC. 2024-09-11T07:59:11.296 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-09-11T07:59:11.297 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2024-09-11T07:59:11.297 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-09-11T07:59:11.358 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-rbd-2:17.2.7-1623.gd0e6828a.el9.x86_64 is already installed. 2024-09-11T07:59:11.377 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install python3-rados 2024-09-11T07:59:11.401 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-11T07:59:11.402 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-11T07:59:11.402 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-11T07:59:11.472 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install rbd-fuse 2024-09-11T07:59:11.937 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:35 ago on Wed 11 Sep 2024 07:57:36 AM UTC. 2024-09-11T07:59:12.028 INFO:teuthology.orchestra.run.smithi150.stdout:Package python3-rados-2:17.2.7-1623.gd0e6828a.el9.x86_64 is already installed. 2024-09-11T07:59:12.037 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:39 ago on Wed 11 Sep 2024 07:57:33 AM UTC. 2024-09-11T07:59:12.072 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-09-11T07:59:12.072 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2024-09-11T07:59:12.073 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-09-11T07:59:12.147 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install python3-rgw 2024-09-11T07:59:12.168 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-11T07:59:12.169 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T07:59:12.169 INFO:teuthology.orchestra.run.smithi181.stdout: Package Architecture Version Repository Size 2024-09-11T07:59:12.169 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T07:59:12.169 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-11T07:59:12.169 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-fuse x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 82 k 2024-09-11T07:59:12.169 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:59:12.169 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-11T07:59:12.169 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T07:59:12.169 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-11T07:59:12.169 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:59:12.170 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 82 k 2024-09-11T07:59:12.170 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 222 k 2024-09-11T07:59:12.170 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-11T07:59:12.340 INFO:teuthology.orchestra.run.smithi181.stdout:rbd-fuse-17.2.7-1623.gd0e6828a.el9.x86_64.rpm 482 kB/s | 82 kB 00:00 2024-09-11T07:59:12.341 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-11T07:59:12.341 INFO:teuthology.orchestra.run.smithi181.stdout:Total 478 kB/s | 82 kB 00:00 2024-09-11T07:59:12.341 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-11T07:59:12.347 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-11T07:59:12.348 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-11T07:59:12.410 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-11T07:59:12.411 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-11T07:59:12.569 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-11T07:59:12.671 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-fuse-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T07:59:12.715 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:36 ago on Wed 11 Sep 2024 07:57:36 AM UTC. 2024-09-11T07:59:12.804 INFO:teuthology.orchestra.run.smithi150.stdout:Package python3-rgw-2:17.2.7-1623.gd0e6828a.el9.x86_64 is already installed. 2024-09-11T07:59:12.847 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-09-11T07:59:12.848 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2024-09-11T07:59:12.848 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-09-11T07:59:12.927 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install python3-cephfs 2024-09-11T07:59:13.045 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T07:59:13.279 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-fuse-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T07:59:13.279 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:59:13.279 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-11T07:59:13.279 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-fuse-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:59:13.279 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:59:13.279 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-11T07:59:13.490 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:37 ago on Wed 11 Sep 2024 07:57:36 AM UTC. 2024-09-11T07:59:13.516 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install rbd-mirror 2024-09-11T07:59:13.580 INFO:teuthology.orchestra.run.smithi150.stdout:Package python3-cephfs-2:17.2.7-1623.gd0e6828a.el9.x86_64 is already installed. 2024-09-11T07:59:13.622 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-09-11T07:59:13.624 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2024-09-11T07:59:13.624 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-09-11T07:59:13.700 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install python3-rbd 2024-09-11T07:59:14.091 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:41 ago on Wed 11 Sep 2024 07:57:33 AM UTC. 2024-09-11T07:59:14.226 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-11T07:59:14.227 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T07:59:14.227 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-09-11T07:59:14.227 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T07:59:14.227 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-11T07:59:14.227 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-mirror x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 2.9 M 2024-09-11T07:59:14.227 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:59:14.227 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-11T07:59:14.227 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T07:59:14.228 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-11T07:59:14.228 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:59:14.228 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 2.9 M 2024-09-11T07:59:14.228 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 12 M 2024-09-11T07:59:14.228 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-11T07:59:14.265 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:38 ago on Wed 11 Sep 2024 07:57:36 AM UTC. 2024-09-11T07:59:14.355 INFO:teuthology.orchestra.run.smithi150.stdout:Package python3-rbd-2:17.2.7-1623.gd0e6828a.el9.x86_64 is already installed. 2024-09-11T07:59:14.398 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-09-11T07:59:14.400 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2024-09-11T07:59:14.400 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-09-11T07:59:14.471 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install rbd-fuse 2024-09-11T07:59:14.560 INFO:teuthology.orchestra.run.smithi181.stdout:rbd-mirror-17.2.7-1623.gd0e6828a.el9.x86_64.rpm 8.8 MB/s | 2.9 MB 00:00 2024-09-11T07:59:14.561 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-11T07:59:14.561 INFO:teuthology.orchestra.run.smithi181.stdout:Total 8.8 MB/s | 2.9 MB 00:00 2024-09-11T07:59:14.561 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-11T07:59:14.570 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-11T07:59:14.570 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-11T07:59:14.650 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-11T07:59:14.650 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-11T07:59:15.032 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:39 ago on Wed 11 Sep 2024 07:57:36 AM UTC. 2024-09-11T07:59:15.058 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-11T07:59:15.127 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-mirror-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T07:59:15.153 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T07:59:15.153 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T07:59:15.153 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-11T07:59:15.153 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-11T07:59:15.153 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-11T07:59:15.153 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:59:15.165 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-09-11T07:59:15.166 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-09-11T07:59:15.166 INFO:teuthology.orchestra.run.smithi150.stdout: Package Architecture Version Repository Size 2024-09-11T07:59:15.166 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-09-11T07:59:15.166 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-09-11T07:59:15.166 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-fuse x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 82 k 2024-09-11T07:59:15.166 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:59:15.166 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-09-11T07:59:15.166 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-09-11T07:59:15.166 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2024-09-11T07:59:15.166 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:59:15.167 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 82 k 2024-09-11T07:59:15.167 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 222 k 2024-09-11T07:59:15.167 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-09-11T07:59:15.363 INFO:teuthology.orchestra.run.smithi150.stdout:rbd-fuse-17.2.7-1623.gd0e6828a.el9.x86_64.rpm 419 kB/s | 82 kB 00:00 2024-09-11T07:59:15.363 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-09-11T07:59:15.363 INFO:teuthology.orchestra.run.smithi150.stdout:Total 416 kB/s | 82 kB 00:00 2024-09-11T07:59:15.363 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-09-11T07:59:15.370 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-09-11T07:59:15.370 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-09-11T07:59:15.433 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-09-11T07:59:15.433 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-09-11T07:59:15.573 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-09-11T07:59:15.676 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : rbd-fuse-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T07:59:16.066 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T07:59:16.187 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-mirror-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T07:59:16.187 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:59:16.188 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-11T07:59:16.188 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-mirror-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:59:16.188 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:59:16.188 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-11T07:59:16.372 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : rbd-fuse-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T07:59:16.373 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:59:16.373 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-09-11T07:59:16.373 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-fuse-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:59:16.373 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:59:16.373 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-09-11T07:59:16.422 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install rbd-nbd 2024-09-11T07:59:16.639 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install rbd-mirror 2024-09-11T07:59:16.999 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:43 ago on Wed 11 Sep 2024 07:57:33 AM UTC. 2024-09-11T07:59:17.137 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-11T07:59:17.137 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T07:59:17.137 INFO:teuthology.orchestra.run.smithi181.stdout: Package Architecture Version Repository Size 2024-09-11T07:59:17.137 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T07:59:17.137 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-11T07:59:17.137 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-nbd x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 165 k 2024-09-11T07:59:17.137 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:59:17.138 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-11T07:59:17.138 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T07:59:17.138 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-11T07:59:17.138 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:59:17.138 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 165 k 2024-09-11T07:59:17.138 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 477 k 2024-09-11T07:59:17.139 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-11T07:59:17.209 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:41 ago on Wed 11 Sep 2024 07:57:36 AM UTC. 2024-09-11T07:59:17.340 INFO:teuthology.orchestra.run.smithi181.stdout:rbd-nbd-17.2.7-1623.gd0e6828a.el9.x86_64.rpm 823 kB/s | 165 kB 00:00 2024-09-11T07:59:17.340 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-11T07:59:17.340 INFO:teuthology.orchestra.run.smithi181.stdout:Total 816 kB/s | 165 kB 00:00 2024-09-11T07:59:17.340 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-11T07:59:17.342 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-09-11T07:59:17.343 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-09-11T07:59:17.343 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repo Size 2024-09-11T07:59:17.343 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-09-11T07:59:17.343 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-09-11T07:59:17.343 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-mirror x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 2.9 M 2024-09-11T07:59:17.343 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:59:17.343 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-09-11T07:59:17.343 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-09-11T07:59:17.343 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2024-09-11T07:59:17.343 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:59:17.344 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 2.9 M 2024-09-11T07:59:17.344 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 12 M 2024-09-11T07:59:17.344 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-09-11T07:59:17.348 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-11T07:59:17.348 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-11T07:59:17.411 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-11T07:59:17.412 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-11T07:59:17.567 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-11T07:59:17.658 INFO:teuthology.orchestra.run.smithi150.stdout:rbd-mirror-17.2.7-1623.gd0e6828a.el9.x86_64.rpm 9.4 MB/s | 2.9 MB 00:00 2024-09-11T07:59:17.658 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-09-11T07:59:17.658 INFO:teuthology.orchestra.run.smithi150.stdout:Total 9.3 MB/s | 2.9 MB 00:00 2024-09-11T07:59:17.659 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-09-11T07:59:17.667 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-09-11T07:59:17.667 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-09-11T07:59:17.677 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-nbd-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T07:59:17.748 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-09-11T07:59:17.748 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-09-11T07:59:18.120 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T07:59:18.168 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-09-11T07:59:18.309 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : rbd-mirror-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T07:59:18.335 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T07:59:18.335 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T07:59:18.335 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-11T07:59:18.335 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-11T07:59:18.335 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-11T07:59:18.336 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:59:18.398 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-nbd-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T07:59:18.398 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:59:18.398 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-11T07:59:18.398 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-nbd-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:59:18.398 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T07:59:18.398 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-11T07:59:18.621 DEBUG:teuthology.parallel:result is None 2024-09-11T07:59:19.338 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : rbd-mirror-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T07:59:19.339 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:59:19.339 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-09-11T07:59:19.339 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-mirror-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:59:19.339 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:59:19.339 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-09-11T07:59:19.561 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install rbd-nbd 2024-09-11T07:59:20.137 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:44 ago on Wed 11 Sep 2024 07:57:36 AM UTC. 2024-09-11T07:59:20.271 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-09-11T07:59:20.272 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-09-11T07:59:20.272 INFO:teuthology.orchestra.run.smithi150.stdout: Package Architecture Version Repository Size 2024-09-11T07:59:20.272 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-09-11T07:59:20.272 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-09-11T07:59:20.272 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-nbd x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 165 k 2024-09-11T07:59:20.272 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:59:20.272 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-09-11T07:59:20.272 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-09-11T07:59:20.272 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2024-09-11T07:59:20.272 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:59:20.273 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 165 k 2024-09-11T07:59:20.273 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 477 k 2024-09-11T07:59:20.273 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-09-11T07:59:20.487 INFO:teuthology.orchestra.run.smithi150.stdout:rbd-nbd-17.2.7-1623.gd0e6828a.el9.x86_64.rpm 774 kB/s | 165 kB 00:00 2024-09-11T07:59:20.487 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-09-11T07:59:20.487 INFO:teuthology.orchestra.run.smithi150.stdout:Total 768 kB/s | 165 kB 00:00 2024-09-11T07:59:20.487 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-09-11T07:59:20.495 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-09-11T07:59:20.495 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-09-11T07:59:20.559 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-09-11T07:59:20.559 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-09-11T07:59:20.709 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-09-11T07:59:20.808 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : rbd-nbd-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T07:59:21.253 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T07:59:21.523 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : rbd-nbd-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T07:59:21.523 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:59:21.523 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-09-11T07:59:21.523 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-nbd-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T07:59:21.523 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T07:59:21.523 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-09-11T07:59:21.820 DEBUG:teuthology.parallel:result is None 2024-09-11T07:59:21.821 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d0e6828a2016d48cf25ad84064e50742bb1c39b9 2024-09-11T07:59:21.971 DEBUG:teuthology.orchestra.run.smithi150:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-11T07:59:21.993 INFO:teuthology.orchestra.run.smithi150.stdout:17.2.7-1623.gd0e6828a.el9 2024-09-11T07:59:21.994 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1623.gd0e6828a.el9 2024-09-11T07:59:21.994 INFO:teuthology.task.install:The correct ceph version 17.2.7-1623.gd0e6828a is installed. 2024-09-11T07:59:21.996 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d0e6828a2016d48cf25ad84064e50742bb1c39b9 2024-09-11T07:59:22.141 DEBUG:teuthology.orchestra.run.smithi181:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-11T07:59:22.166 INFO:teuthology.orchestra.run.smithi181.stdout:17.2.7-1623.gd0e6828a.el9 2024-09-11T07:59:22.167 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1623.gd0e6828a.el9 2024-09-11T07:59:22.167 INFO:teuthology.task.install:The correct ceph version 17.2.7-1623.gd0e6828a is installed. 2024-09-11T07:59:22.168 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-11T07:59:22.169 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-11T07:59:22.169 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-11T07:59:22.195 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-11T07:59:22.195 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-11T07:59:22.237 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-11T07:59:22.237 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-11T07:59:22.238 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/daemon-helper 2024-09-11T07:59:22.262 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-11T07:59:22.326 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-11T07:59:22.327 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/daemon-helper 2024-09-11T07:59:22.355 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-11T07:59:22.424 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-11T07:59:22.424 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-11T07:59:22.424 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-11T07:59:22.462 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-11T07:59:22.527 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-11T07:59:22.527 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-11T07:59:22.553 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-11T07:59:22.619 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-11T07:59:22.620 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-11T07:59:22.620 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/stdin-killer 2024-09-11T07:59:22.644 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-11T07:59:22.708 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-11T07:59:22.708 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/stdin-killer 2024-09-11T07:59:22.734 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-11T07:59:22.800 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-11T07:59:22.890 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'd0e6828a2016d48cf25ad84064e50742bb1c39b9', 'cephadm_mode': 'root'} 2024-09-11T07:59:22.890 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 2024-09-11T07:59:22.891 INFO:tasks.cephadm:Cluster fsid is c1bddff4-7013-11ef-bcea-c7b262605968 2024-09-11T07:59:22.891 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-11T07:59:22.891 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.150', 'mon.c': '[v2:172.21.15.150:3301,v1:172.21.15.150:6790]', 'mon.b': '172.21.15.181'} 2024-09-11T07:59:22.891 INFO:tasks.cephadm:First mon is mon.a on smithi150 2024-09-11T07:59:22.891 INFO:tasks.cephadm:First mgr is y 2024-09-11T07:59:22.891 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-11T07:59:22.892 DEBUG:teuthology.orchestra.run.smithi150:> sudo hostname $(hostname -s) 2024-09-11T07:59:22.916 DEBUG:teuthology.orchestra.run.smithi181:> sudo hostname $(hostname -s) 2024-09-11T07:59:22.943 INFO:tasks.cephadm:Downloading cephadm (repo https://git.ceph.com/ceph-ci.git ref d0e6828a2016d48cf25ad84064e50742bb1c39b9)... 2024-09-11T07:59:22.943 DEBUG:teuthology.orchestra.run.smithi150:> git clone https://git.ceph.com/ceph-ci.git testrepo && cd testrepo && git show d0e6828a2016d48cf25ad84064e50742bb1c39b9:src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-11T07:59:22.972 INFO:teuthology.orchestra.run.smithi150.stderr:Cloning into 'testrepo'... 2024-09-11T08:00:22.133 INFO:teuthology.orchestra.run.smithi150.stdout:-rw-r--r--. 1 ubuntu ubuntu 378585 Sep 11 08:00 /home/ubuntu/cephtest/cephadm 2024-09-11T08:00:22.134 DEBUG:teuthology.orchestra.run.smithi181:> git clone https://git.ceph.com/ceph-ci.git testrepo && cd testrepo && git show d0e6828a2016d48cf25ad84064e50742bb1c39b9:src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-11T08:00:22.155 INFO:teuthology.orchestra.run.smithi181.stderr:Cloning into 'testrepo'... 2024-09-11T08:01:20.150 INFO:teuthology.orchestra.run.smithi181.stdout:-rw-r--r--. 1 ubuntu ubuntu 378585 Sep 11 08:01 /home/ubuntu/cephtest/cephadm 2024-09-11T08:01:20.151 DEBUG:teuthology.orchestra.run.smithi150:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-11T08:01:20.173 DEBUG:teuthology.orchestra.run.smithi181:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-11T08:01:20.220 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 on all hosts... 2024-09-11T08:01:20.220 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 pull 2024-09-11T08:01:20.223 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 pull 2024-09-11T08:01:21.795 INFO:teuthology.orchestra.run.smithi150.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9... 2024-09-11T08:01:23.696 INFO:teuthology.orchestra.run.smithi181.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9... 2024-09-11T08:01:51.621 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-11T08:01:51.621 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_version": "ceph version 17.2.7-1623-gd0e6828a (d0e6828a2016d48cf25ad84064e50742bb1c39b9) quincy (stable)", 2024-09-11T08:01:51.621 INFO:teuthology.orchestra.run.smithi150.stdout: "image_id": "4a918172a49766385959227c53e75e2c302e923539324792cb5bee6cad0b2f2b", 2024-09-11T08:01:51.621 INFO:teuthology.orchestra.run.smithi150.stdout: "repo_digests": [ 2024-09-11T08:01:51.621 INFO:teuthology.orchestra.run.smithi150.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40" 2024-09-11T08:01:51.622 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:01:51.622 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-11T08:02:05.750 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-11T08:02:05.751 INFO:teuthology.orchestra.run.smithi181.stdout: "ceph_version": "ceph version 17.2.7-1623-gd0e6828a (d0e6828a2016d48cf25ad84064e50742bb1c39b9) quincy (stable)", 2024-09-11T08:02:05.751 INFO:teuthology.orchestra.run.smithi181.stdout: "image_id": "4a918172a49766385959227c53e75e2c302e923539324792cb5bee6cad0b2f2b", 2024-09-11T08:02:05.751 INFO:teuthology.orchestra.run.smithi181.stdout: "repo_digests": [ 2024-09-11T08:02:05.751 INFO:teuthology.orchestra.run.smithi181.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40" 2024-09-11T08:02:05.751 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:02:05.751 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-11T08:02:05.774 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /etc/ceph 2024-09-11T08:02:05.809 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /etc/ceph 2024-09-11T08:02:05.837 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 777 /etc/ceph 2024-09-11T08:02:05.878 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 777 /etc/ceph 2024-09-11T08:02:05.903 INFO:tasks.cephadm:Writing seed config... 2024-09-11T08:02:05.904 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-11T08:02:05.904 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-11T08:02:05.904 INFO:tasks.cephadm: override: [client.0] debug ms = 1 2024-09-11T08:02:05.904 INFO:tasks.cephadm: override: [client.0] debug objecter = 20 2024-09-11T08:02:05.904 INFO:tasks.cephadm: override: [client.0] debug rados = 20 2024-09-11T08:02:05.904 INFO:tasks.cephadm: override: [global] ms bind msgr1 = False 2024-09-11T08:02:05.905 INFO:tasks.cephadm: override: [global] ms bind msgr2 = True 2024-09-11T08:02:05.905 INFO:tasks.cephadm: override: [global] ms type = async 2024-09-11T08:02:05.905 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-11T08:02:05.905 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-11T08:02:05.905 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-11T08:02:05.905 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-09-11T08:02:05.905 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-09-11T08:02:05.905 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-09-11T08:02:05.905 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-09-11T08:02:05.905 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-09-11T08:02:05.905 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-11T08:02:05.905 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-11T08:02:05.905 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-09-11T08:02:05.906 INFO:tasks.cephadm: override: [osd] osd delete sleep = 1 2024-09-11T08:02:05.906 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-09-11T08:02:05.906 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-09-11T08:02:05.906 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-09-11T08:02:05.906 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-09-11T08:02:05.906 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-09-11T08:02:05.906 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-11T08:02:05.907 DEBUG:teuthology.orchestra.run.smithi150:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-11T08:02:05.936 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = c1bddff4-7013-11ef-bcea-c7b262605968 ms bind msgr1 = False ms bind msgr2 = True ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd debug reject backfill probability = 0.3 osd delete sleep = 1 osd max backfills = 3 osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client.0] debug ms = 1 debug objecter = 20 debug rados = 20 2024-09-11T08:02:05.936 DEBUG:teuthology.orchestra.run.smithi150:mon.a> sudo journalctl -f -n 0 -u ceph-c1bddff4-7013-11ef-bcea-c7b262605968@mon.a.service 2024-09-11T08:02:05.979 DEBUG:teuthology.orchestra.run.smithi150:mgr.y> sudo journalctl -f -n 0 -u ceph-c1bddff4-7013-11ef-bcea-c7b262605968@mgr.y.service 2024-09-11T08:02:06.022 INFO:tasks.cephadm:Bootstrapping... 2024-09-11T08:02:06.022 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 -v bootstrap --fsid c1bddff4-7013-11ef-bcea-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.150 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-11T08:02:06.226 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-09-11T08:02:06.226 INFO:teuthology.orchestra.run.smithi150.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9', '-v', 'bootstrap', '--fsid', 'c1bddff4-7013-11ef-bcea-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.150', '--skip-admin-label'] 2024-09-11T08:02:06.250 INFO:teuthology.orchestra.run.smithi150.stdout:/bin/podman: stdout 5.2.2 2024-09-11T08:02:06.250 INFO:teuthology.orchestra.run.smithi150.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-11T08:02:06.251 INFO:teuthology.orchestra.run.smithi150.stdout:Verifying podman|docker is present... 2024-09-11T08:02:06.279 INFO:teuthology.orchestra.run.smithi150.stdout:/bin/podman: stdout 5.2.2 2024-09-11T08:02:06.279 INFO:teuthology.orchestra.run.smithi150.stdout:Verifying lvm2 is present... 2024-09-11T08:02:06.279 INFO:teuthology.orchestra.run.smithi150.stdout:Verifying time synchronization is in place... 2024-09-11T08:02:06.287 INFO:teuthology.orchestra.run.smithi150.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-11T08:02:06.287 INFO:teuthology.orchestra.run.smithi150.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-11T08:02:06.293 INFO:teuthology.orchestra.run.smithi150.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-11T08:02:06.293 INFO:teuthology.orchestra.run.smithi150.stdout:systemctl: stdout inactive 2024-09-11T08:02:06.301 INFO:teuthology.orchestra.run.smithi150.stdout:systemctl: stdout enabled 2024-09-11T08:02:06.307 INFO:teuthology.orchestra.run.smithi150.stdout:systemctl: stdout active 2024-09-11T08:02:06.307 INFO:teuthology.orchestra.run.smithi150.stdout:Unit chronyd.service is enabled and running 2024-09-11T08:02:06.308 INFO:teuthology.orchestra.run.smithi150.stdout:Repeating the final host check... 2024-09-11T08:02:06.330 INFO:teuthology.orchestra.run.smithi150.stdout:/bin/podman: stdout 5.2.2 2024-09-11T08:02:06.331 INFO:teuthology.orchestra.run.smithi150.stdout:podman (/bin/podman) version 5.2.2 is present 2024-09-11T08:02:06.331 INFO:teuthology.orchestra.run.smithi150.stdout:systemctl is present 2024-09-11T08:02:06.331 INFO:teuthology.orchestra.run.smithi150.stdout:lvcreate is present 2024-09-11T08:02:06.337 INFO:teuthology.orchestra.run.smithi150.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-11T08:02:06.337 INFO:teuthology.orchestra.run.smithi150.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-11T08:02:06.343 INFO:teuthology.orchestra.run.smithi150.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-11T08:02:06.344 INFO:teuthology.orchestra.run.smithi150.stdout:systemctl: stdout inactive 2024-09-11T08:02:06.351 INFO:teuthology.orchestra.run.smithi150.stdout:systemctl: stdout enabled 2024-09-11T08:02:06.359 INFO:teuthology.orchestra.run.smithi150.stdout:systemctl: stdout active 2024-09-11T08:02:06.359 INFO:teuthology.orchestra.run.smithi150.stdout:Unit chronyd.service is enabled and running 2024-09-11T08:02:06.359 INFO:teuthology.orchestra.run.smithi150.stdout:Host looks OK 2024-09-11T08:02:06.359 INFO:teuthology.orchestra.run.smithi150.stdout:Cluster fsid: c1bddff4-7013-11ef-bcea-c7b262605968 2024-09-11T08:02:06.359 INFO:teuthology.orchestra.run.smithi150.stdout:Acquiring lock 139962520126368 on /run/cephadm/c1bddff4-7013-11ef-bcea-c7b262605968.lock 2024-09-11T08:02:06.360 INFO:teuthology.orchestra.run.smithi150.stdout:Lock 139962520126368 acquired on /run/cephadm/c1bddff4-7013-11ef-bcea-c7b262605968.lock 2024-09-11T08:02:06.360 INFO:teuthology.orchestra.run.smithi150.stdout:Verifying IP 172.21.15.150 port 3300 ... 2024-09-11T08:02:06.360 INFO:teuthology.orchestra.run.smithi150.stdout:Verifying IP 172.21.15.150 port 6789 ... 2024-09-11T08:02:06.360 INFO:teuthology.orchestra.run.smithi150.stdout:Base mon IP(s) is [172.21.15.150:3300, 172.21.15.150:6789], mon addrv is [v2:172.21.15.150:3300,v1:172.21.15.150:6789] 2024-09-11T08:02:06.363 INFO:teuthology.orchestra.run.smithi150.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.150 metric 100 2024-09-11T08:02:06.363 INFO:teuthology.orchestra.run.smithi150.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.150 metric 100 2024-09-11T08:02:06.366 INFO:teuthology.orchestra.run.smithi150.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-11T08:02:06.366 INFO:teuthology.orchestra.run.smithi150.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-09-11T08:02:06.366 INFO:teuthology.orchestra.run.smithi150.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-09-11T08:02:06.369 INFO:teuthology.orchestra.run.smithi150.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-11T08:02:06.370 INFO:teuthology.orchestra.run.smithi150.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-11T08:02:06.370 INFO:teuthology.orchestra.run.smithi150.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-11T08:02:06.370 INFO:teuthology.orchestra.run.smithi150.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-09-11T08:02:06.370 INFO:teuthology.orchestra.run.smithi150.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe88:6e35/64 scope link noprefixroute 2024-09-11T08:02:06.370 INFO:teuthology.orchestra.run.smithi150.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-11T08:02:06.370 INFO:teuthology.orchestra.run.smithi150.stdout:Mon IP `172.21.15.150` is in CIDR network `172.21.0.0/20` 2024-09-11T08:02:06.371 INFO:teuthology.orchestra.run.smithi150.stdout:Mon IP `172.21.15.150` is in CIDR network `172.21.0.0/20` 2024-09-11T08:02:06.371 INFO:teuthology.orchestra.run.smithi150.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-09-11T08:02:06.371 INFO:teuthology.orchestra.run.smithi150.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-11T08:02:06.372 INFO:teuthology.orchestra.run.smithi150.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9... 2024-09-11T08:02:07.319 INFO:teuthology.orchestra.run.smithi150.stdout:/bin/podman: stdout 4a918172a49766385959227c53e75e2c302e923539324792cb5bee6cad0b2f2b 2024-09-11T08:02:07.319 INFO:teuthology.orchestra.run.smithi150.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9... 2024-09-11T08:02:07.319 INFO:teuthology.orchestra.run.smithi150.stdout:/bin/podman: stderr Getting image source signatures 2024-09-11T08:02:07.319 INFO:teuthology.orchestra.run.smithi150.stdout:/bin/podman: stderr Copying blob sha256:0d2f264d63cec2f6fb0f85eed7b2c8c5b4db3ab9edd9d71a74c04647294a553f 2024-09-11T08:02:07.319 INFO:teuthology.orchestra.run.smithi150.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-09-11T08:02:07.319 INFO:teuthology.orchestra.run.smithi150.stdout:/bin/podman: stderr Copying config sha256:4a918172a49766385959227c53e75e2c302e923539324792cb5bee6cad0b2f2b 2024-09-11T08:02:07.319 INFO:teuthology.orchestra.run.smithi150.stdout:/bin/podman: stderr Writing manifest to image destination 2024-09-11T08:02:08.190 INFO:teuthology.orchestra.run.smithi150.stdout:ceph: stdout ceph version 17.2.7-1623-gd0e6828a (d0e6828a2016d48cf25ad84064e50742bb1c39b9) quincy (stable) 2024-09-11T08:02:08.190 INFO:teuthology.orchestra.run.smithi150.stdout:Ceph version: ceph version 17.2.7-1623-gd0e6828a (d0e6828a2016d48cf25ad84064e50742bb1c39b9) quincy (stable) 2024-09-11T08:02:08.190 INFO:teuthology.orchestra.run.smithi150.stdout:Extracting ceph user uid/gid from container image... 2024-09-11T08:02:09.095 INFO:teuthology.orchestra.run.smithi150.stdout:stat: stdout 167 167 2024-09-11T08:02:09.096 INFO:teuthology.orchestra.run.smithi150.stdout:Creating initial keys... 2024-09-11T08:02:10.068 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph-authtool: stdout AQCBTuFmPnzLGhAArLwy28cvuFlnskdUpuI0Lg== 2024-09-11T08:02:11.073 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph-authtool: stdout AQCCTuFmnVqwFxAA/IqlNyOnb7Fex8idRND9Uw== 2024-09-11T08:02:12.062 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph-authtool: stdout AQCDTuFm8lsYFRAAYlbnNTmKmZuK6Ti/+Z9prw== 2024-09-11T08:02:12.063 INFO:teuthology.orchestra.run.smithi150.stdout:Creating initial monmap... 2024-09-11T08:02:12.961 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-11T08:02:12.961 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2024-09-11T08:02:12.961 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to c1bddff4-7013-11ef-bcea-c7b262605968 2024-09-11T08:02:12.961 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-11T08:02:12.961 INFO:teuthology.orchestra.run.smithi150.stdout:monmaptool for a [v2:172.21.15.150:3300,v1:172.21.15.150:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-11T08:02:12.961 INFO:teuthology.orchestra.run.smithi150.stdout:setting min_mon_release = octopus 2024-09-11T08:02:12.961 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/monmaptool: set fsid to c1bddff4-7013-11ef-bcea-c7b262605968 2024-09-11T08:02:12.961 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-11T08:02:12.961 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:02:12.962 INFO:teuthology.orchestra.run.smithi150.stdout:Creating mon... 2024-09-11T08:02:13.933 INFO:teuthology.orchestra.run.smithi150.stdout:create mon.a on 2024-09-11T08:02:14.137 INFO:teuthology.orchestra.run.smithi150.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-09-11T08:02:14.290 INFO:teuthology.orchestra.run.smithi150.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-11T08:02:14.448 INFO:teuthology.orchestra.run.smithi150.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-c1bddff4-7013-11ef-bcea-c7b262605968.target → /etc/systemd/system/ceph-c1bddff4-7013-11ef-bcea-c7b262605968.target. 2024-09-11T08:02:14.448 INFO:teuthology.orchestra.run.smithi150.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-c1bddff4-7013-11ef-bcea-c7b262605968.target → /etc/systemd/system/ceph-c1bddff4-7013-11ef-bcea-c7b262605968.target. 2024-09-11T08:02:14.623 INFO:teuthology.orchestra.run.smithi150.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-c1bddff4-7013-11ef-bcea-c7b262605968@mon.a 2024-09-11T08:02:14.623 INFO:teuthology.orchestra.run.smithi150.stdout:systemctl: stderr Failed to reset failed state of unit ceph-c1bddff4-7013-11ef-bcea-c7b262605968@mon.a.service: Unit ceph-c1bddff4-7013-11ef-bcea-c7b262605968@mon.a.service not loaded. 2024-09-11T08:02:14.809 INFO:teuthology.orchestra.run.smithi150.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-c1bddff4-7013-11ef-bcea-c7b262605968.target.wants/ceph-c1bddff4-7013-11ef-bcea-c7b262605968@mon.a.service → /etc/systemd/system/ceph-c1bddff4-7013-11ef-bcea-c7b262605968@.service. 2024-09-11T08:02:15.432 INFO:teuthology.orchestra.run.smithi150.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-11T08:02:15.432 INFO:teuthology.orchestra.run.smithi150.stdout:systemctl: stdout disabled 2024-09-11T08:02:15.440 INFO:teuthology.orchestra.run.smithi150.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-11T08:02:15.441 INFO:teuthology.orchestra.run.smithi150.stdout:systemctl: stdout inactive 2024-09-11T08:02:15.441 INFO:teuthology.orchestra.run.smithi150.stdout:firewalld.service is not enabled 2024-09-11T08:02:15.441 INFO:teuthology.orchestra.run.smithi150.stdout:Not possible to enable service . firewalld.service is not available 2024-09-11T08:02:15.441 INFO:teuthology.orchestra.run.smithi150.stdout:Waiting for mon to start... 2024-09-11T08:02:15.441 INFO:teuthology.orchestra.run.smithi150.stdout:Waiting for mon... 2024-09-11T08:02:16.479 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout cluster: 2024-09-11T08:02:16.480 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout id: c1bddff4-7013-11ef-bcea-c7b262605968 2024-09-11T08:02:16.480 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-11T08:02:16.480 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout 2024-09-11T08:02:16.480 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout services: 2024-09-11T08:02:16.480 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.581959s) 2024-09-11T08:02:16.480 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-11T08:02:16.480 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-11T08:02:16.480 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout 2024-09-11T08:02:16.480 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout data: 2024-09-11T08:02:16.481 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-11T08:02:16.481 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-11T08:02:16.481 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-11T08:02:16.481 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout pgs: 2024-09-11T08:02:16.481 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout 2024-09-11T08:02:16.481 INFO:teuthology.orchestra.run.smithi150.stdout:mon is available 2024-09-11T08:02:16.481 INFO:teuthology.orchestra.run.smithi150.stdout:Assimilating anything we can from ceph.conf... 2024-09-11T08:02:17.442 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout 2024-09-11T08:02:17.442 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout [global] 2024-09-11T08:02:17.442 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout fsid = c1bddff4-7013-11ef-bcea-c7b262605968 2024-09-11T08:02:17.442 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-11T08:02:17.442 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.150:3300,v1:172.21.15.150:6789] 2024-09-11T08:02:17.442 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-11T08:02:17.442 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-11T08:02:17.442 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-11T08:02:17.442 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-11T08:02:17.443 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout 2024-09-11T08:02:17.443 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-11T08:02:17.443 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-11T08:02:17.443 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout 2024-09-11T08:02:17.443 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout [osd] 2024-09-11T08:02:17.443 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-11T08:02:17.443 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-11T08:02:17.443 INFO:teuthology.orchestra.run.smithi150.stdout:Generating new minimal ceph.conf... 2024-09-11T08:02:18.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:18 smithi150 ceph-mon[28913]: from='client.? 172.21.15.150:0/206623510' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:02:23.683 INFO:teuthology.orchestra.run.smithi150.stdout:Restarting the monitor... 2024-09-11T08:02:23.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:23 smithi150 systemd[1]: Stopping Ceph mon.a for c1bddff4-7013-11ef-bcea-c7b262605968... 2024-09-11T08:02:25.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:24 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mon-a[28909]: 2024-09-11T08:02:24.899+0000 7fcae632c640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-11T08:02:25.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:24 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mon-a[28909]: 2024-09-11T08:02:24.899+0000 7fcae632c640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-11T08:02:25.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:24 smithi150 podman[29152]: 2024-09-11 08:02:24.930978637 +0000 UTC m=+1.075434316 container died 42b15f9930d2d9f9f21ad4390fb36de98ff5daae3ceb720492b5d90d7d294380 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-11T08:02:29.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:28 smithi150 podman[29152]: 2024-09-11 08:02:28.77862641 +0000 UTC m=+4.923082089 container cleanup 42b15f9930d2d9f9f21ad4390fb36de98ff5daae3ceb720492b5d90d7d294380 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mon-a, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-11T08:02:29.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:28 smithi150 bash[29152]: ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mon-a 2024-09-11T08:02:29.966 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:29 smithi150 podman[29163]: 2024-09-11 08:02:29.576097682 +0000 UTC m=+4.645380280 container remove 42b15f9930d2d9f9f21ad4390fb36de98ff5daae3ceb720492b5d90d7d294380 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mon-a, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_BRANCH=HEAD, ceph=True) 2024-09-11T08:02:29.966 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:29 smithi150 systemd[1]: ceph-c1bddff4-7013-11ef-bcea-c7b262605968@mon.a.service: Deactivated successfully. 2024-09-11T08:02:29.966 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:29 smithi150 systemd[1]: Stopped Ceph mon.a for c1bddff4-7013-11ef-bcea-c7b262605968. 2024-09-11T08:02:29.966 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:29 smithi150 systemd[1]: Starting Ceph mon.a for c1bddff4-7013-11ef-bcea-c7b262605968... 2024-09-11T08:02:30.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:29 smithi150 podman[29257]: 2024-09-11 08:02:29.868031085 +0000 UTC m=+0.015830144 image pull 4a918172a49766385959227c53e75e2c302e923539324792cb5bee6cad0b2f2b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 2024-09-11T08:02:30.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:29 smithi150 podman[29257]: 2024-09-11 08:02:29.968309651 +0000 UTC m=+0.116108702 container create d7334d5149102c624f81121c01d70bd96d37a60f0157b33fa338f4caf9ffa9de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mon-a, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0) 2024-09-11T08:02:30.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 podman[29257]: 2024-09-11 08:02:30.170681507 +0000 UTC m=+0.318480556 container init d7334d5149102c624f81121c01d70bd96d37a60f0157b33fa338f4caf9ffa9de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mon-a, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-11T08:02:30.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 podman[29257]: 2024-09-11 08:02:30.173751435 +0000 UTC m=+0.321550484 container start d7334d5149102c624f81121c01d70bd96d37a60f0157b33fa338f4caf9ffa9de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mon-a, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-11T08:02:30.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: set uid:gid to 167:167 (ceph:ceph) 2024-09-11T08:02:30.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: ceph version 17.2.7-1623-gd0e6828a (d0e6828a2016d48cf25ad84064e50742bb1c39b9) quincy (stable), process ceph-mon, pid 2 2024-09-11T08:02:30.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: pidfile_write: ignore empty --pid-file 2024-09-11T08:02:30.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: load: jerasure load: lrc 2024-09-11T08:02:30.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: RocksDB version: 6.15.5 2024-09-11T08:02:30.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-09-11T08:02:30.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Compile date Sep 10 2024 2024-09-11T08:02:30.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: DB SUMMARY 2024-09-11T08:02:30.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: DB Session ID: S4317ENVB8KJQNW1SQ0D 2024-09-11T08:02:30.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: CURRENT file: CURRENT 2024-09-11T08:02:30.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: IDENTITY file: IDENTITY 2024-09-11T08:02:30.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-09-11T08:02:30.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-09-11T08:02:30.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 89958 ; 2024-09-11T08:02:30.243 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.error_if_exists: 0 2024-09-11T08:02:30.243 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.create_if_missing: 0 2024-09-11T08:02:30.243 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.paranoid_checks: 1 2024-09-11T08:02:30.243 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-11T08:02:30.243 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.env: 0x5577331c0440 2024-09-11T08:02:30.243 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.fs: Posix File System 2024-09-11T08:02:30.243 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.info_log: 0x557735248ec0 2024-09-11T08:02:30.244 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.max_file_opening_threads: 16 2024-09-11T08:02:30.244 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.statistics: (nil) 2024-09-11T08:02:30.244 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.use_fsync: 0 2024-09-11T08:02:30.244 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.max_log_file_size: 0 2024-09-11T08:02:30.244 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-11T08:02:30.244 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-11T08:02:30.244 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.keep_log_file_num: 1000 2024-09-11T08:02:30.244 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.recycle_log_file_num: 0 2024-09-11T08:02:30.244 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.allow_fallocate: 1 2024-09-11T08:02:30.244 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.allow_mmap_reads: 0 2024-09-11T08:02:30.244 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.allow_mmap_writes: 0 2024-09-11T08:02:30.244 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.use_direct_reads: 0 2024-09-11T08:02:30.245 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-11T08:02:30.245 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.create_missing_column_families: 0 2024-09-11T08:02:30.245 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.db_log_dir: 2024-09-11T08:02:30.245 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-09-11T08:02:30.245 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-11T08:02:30.245 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-11T08:02:30.245 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-11T08:02:30.245 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-11T08:02:30.245 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-11T08:02:30.245 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-11T08:02:30.246 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.advise_random_on_open: 1 2024-09-11T08:02:30.246 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.db_write_buffer_size: 0 2024-09-11T08:02:30.246 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.write_buffer_manager: 0x557735306780 2024-09-11T08:02:30.246 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-11T08:02:30.246 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-09-11T08:02:30.246 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-11T08:02:30.246 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-11T08:02:30.246 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.rate_limiter: (nil) 2024-09-11T08:02:30.246 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-11T08:02:30.246 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.wal_recovery_mode: 2 2024-09-11T08:02:30.246 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.enable_thread_tracking: 0 2024-09-11T08:02:30.246 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.enable_pipelined_write: 0 2024-09-11T08:02:30.247 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.unordered_write: 0 2024-09-11T08:02:30.247 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-11T08:02:30.247 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-11T08:02:30.247 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-11T08:02:30.247 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-11T08:02:30.247 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.row_cache: None 2024-09-11T08:02:30.247 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.wal_filter: None 2024-09-11T08:02:30.248 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-11T08:02:30.248 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.allow_ingest_behind: 0 2024-09-11T08:02:30.248 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.preserve_deletes: 0 2024-09-11T08:02:30.248 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.two_write_queues: 0 2024-09-11T08:02:30.248 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.manual_wal_flush: 0 2024-09-11T08:02:30.248 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.atomic_flush: 0 2024-09-11T08:02:30.248 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-11T08:02:30.248 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-11T08:02:30.248 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-11T08:02:30.248 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.log_readahead_size: 0 2024-09-11T08:02:30.248 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-11T08:02:30.248 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.best_efforts_recovery: 0 2024-09-11T08:02:30.248 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-11T08:02:30.249 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-11T08:02:30.249 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.allow_data_in_errors: 0 2024-09-11T08:02:30.249 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.db_host_id: __hostname__ 2024-09-11T08:02:30.249 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.max_background_jobs: 2 2024-09-11T08:02:30.249 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.max_background_compactions: -1 2024-09-11T08:02:30.249 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.max_subcompactions: 1 2024-09-11T08:02:30.249 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-11T08:02:30.249 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-11T08:02:30.249 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-11T08:02:30.249 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.max_total_wal_size: 0 2024-09-11T08:02:30.249 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-11T08:02:30.249 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-11T08:02:30.250 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-11T08:02:30.250 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-11T08:02:30.250 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.max_open_files: -1 2024-09-11T08:02:30.250 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.bytes_per_sync: 0 2024-09-11T08:02:30.250 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-11T08:02:30.250 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-11T08:02:30.250 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.compaction_readahead_size: 0 2024-09-11T08:02:30.250 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.max_background_flushes: -1 2024-09-11T08:02:30.250 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Compression algorithms supported: 2024-09-11T08:02:30.250 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: kZSTD supported: 0 2024-09-11T08:02:30.250 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: kXpressCompression supported: 0 2024-09-11T08:02:30.250 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: kBZip2Compression supported: 0 2024-09-11T08:02:30.250 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-11T08:02:30.250 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: kLZ4Compression supported: 1 2024-09-11T08:02:30.251 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: kZlibCompression supported: 1 2024-09-11T08:02:30.251 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: kLZ4HCCompression supported: 1 2024-09-11T08:02:30.251 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: kSnappyCompression supported: 1 2024-09-11T08:02:30.251 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-11T08:02:30.251 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2024-09-11T08:02:30.251 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-09-11T08:02:30.251 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-11T08:02:30.251 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.merge_operator: 2024-09-11T08:02:30.251 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.compaction_filter: None 2024-09-11T08:02:30.251 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.compaction_filter_factory: None 2024-09-11T08:02:30.251 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.sst_partitioner_factory: None 2024-09-11T08:02:30.251 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-11T08:02:30.251 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-11T08:02:30.251 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5577351efa38) 2024-09-11T08:02:30.252 INFO:journalctl@ceph.mon.a.smithi150.stdout: cache_index_and_filter_blocks: 1 2024-09-11T08:02:30.252 INFO:journalctl@ceph.mon.a.smithi150.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-11T08:02:30.252 INFO:journalctl@ceph.mon.a.smithi150.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-11T08:02:30.252 INFO:journalctl@ceph.mon.a.smithi150.stdout: pin_top_level_index_and_filter: 1 2024-09-11T08:02:30.252 INFO:journalctl@ceph.mon.a.smithi150.stdout: index_type: 0 2024-09-11T08:02:30.252 INFO:journalctl@ceph.mon.a.smithi150.stdout: data_block_index_type: 0 2024-09-11T08:02:30.252 INFO:journalctl@ceph.mon.a.smithi150.stdout: index_shortening: 1 2024-09-11T08:02:30.252 INFO:journalctl@ceph.mon.a.smithi150.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-11T08:02:30.252 INFO:journalctl@ceph.mon.a.smithi150.stdout: hash_index_allow_collision: 1 2024-09-11T08:02:30.252 INFO:journalctl@ceph.mon.a.smithi150.stdout: checksum: 1 2024-09-11T08:02:30.252 INFO:journalctl@ceph.mon.a.smithi150.stdout: no_block_cache: 0 2024-09-11T08:02:30.252 INFO:journalctl@ceph.mon.a.smithi150.stdout: block_cache: 0x5577352251f0 2024-09-11T08:02:30.253 INFO:journalctl@ceph.mon.a.smithi150.stdout: block_cache_name: BinnedLRUCache 2024-09-11T08:02:30.253 INFO:journalctl@ceph.mon.a.smithi150.stdout: block_cache_options: 2024-09-11T08:02:30.253 INFO:journalctl@ceph.mon.a.smithi150.stdout: capacity : 536870912 2024-09-11T08:02:30.253 INFO:journalctl@ceph.mon.a.smithi150.stdout: num_shard_bits : 4 2024-09-11T08:02:30.253 INFO:journalctl@ceph.mon.a.smithi150.stdout: strict_capacity_limit : 0 2024-09-11T08:02:30.253 INFO:journalctl@ceph.mon.a.smithi150.stdout: high_pri_pool_ratio: 0.000 2024-09-11T08:02:30.253 INFO:journalctl@ceph.mon.a.smithi150.stdout: block_cache_compressed: (nil) 2024-09-11T08:02:30.253 INFO:journalctl@ceph.mon.a.smithi150.stdout: persistent_cache: (nil) 2024-09-11T08:02:30.253 INFO:journalctl@ceph.mon.a.smithi150.stdout: block_size: 4096 2024-09-11T08:02:30.253 INFO:journalctl@ceph.mon.a.smithi150.stdout: block_size_deviation: 10 2024-09-11T08:02:30.253 INFO:journalctl@ceph.mon.a.smithi150.stdout: block_restart_interval: 16 2024-09-11T08:02:30.253 INFO:journalctl@ceph.mon.a.smithi150.stdout: index_block_restart_interval: 1 2024-09-11T08:02:30.253 INFO:journalctl@ceph.mon.a.smithi150.stdout: metadata_block_size: 4096 2024-09-11T08:02:30.254 INFO:journalctl@ceph.mon.a.smithi150.stdout: partition_filters: 0 2024-09-11T08:02:30.254 INFO:journalctl@ceph.mon.a.smithi150.stdout: use_delta_encoding: 1 2024-09-11T08:02:30.254 INFO:journalctl@ceph.mon.a.smithi150.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-09-11T08:02:30.254 INFO:journalctl@ceph.mon.a.smithi150.stdout: whole_key_filtering: 1 2024-09-11T08:02:30.254 INFO:journalctl@ceph.mon.a.smithi150.stdout: verify_compression: 0 2024-09-11T08:02:30.254 INFO:journalctl@ceph.mon.a.smithi150.stdout: read_amp_bytes_per_bit: 0 2024-09-11T08:02:30.254 INFO:journalctl@ceph.mon.a.smithi150.stdout: format_version: 4 2024-09-11T08:02:30.254 INFO:journalctl@ceph.mon.a.smithi150.stdout: enable_index_compression: 1 2024-09-11T08:02:30.254 INFO:journalctl@ceph.mon.a.smithi150.stdout: block_align: 0 2024-09-11T08:02:30.254 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.write_buffer_size: 33554432 2024-09-11T08:02:30.254 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.max_write_buffer_number: 2 2024-09-11T08:02:30.254 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.compression: NoCompression 2024-09-11T08:02:30.254 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.bottommost_compression: Disabled 2024-09-11T08:02:30.254 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.prefix_extractor: nullptr 2024-09-11T08:02:30.255 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-11T08:02:30.255 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.num_levels: 7 2024-09-11T08:02:30.255 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-11T08:02:30.255 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-11T08:02:30.255 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-11T08:02:30.255 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-11T08:02:30.255 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-11T08:02:30.255 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-11T08:02:30.255 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-11T08:02:30.255 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-11T08:02:30.255 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-11T08:02:30.255 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-11T08:02:30.255 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-11T08:02:30.255 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.compression_opts.level: 32767 2024-09-11T08:02:30.256 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.compression_opts.strategy: 0 2024-09-11T08:02:30.256 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-11T08:02:30.256 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-11T08:02:30.256 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-11T08:02:30.256 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.compression_opts.enabled: false 2024-09-11T08:02:30.256 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-11T08:02:30.256 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-11T08:02:30.256 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-11T08:02:30.256 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.target_file_size_base: 67108864 2024-09-11T08:02:30.256 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-11T08:02:30.256 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-11T08:02:30.256 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-11T08:02:30.256 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-11T08:02:30.257 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-11T08:02:30.257 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-11T08:02:30.257 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-11T08:02:30.257 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-11T08:02:30.257 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-11T08:02:30.257 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-11T08:02:30.257 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-11T08:02:30.257 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-11T08:02:30.257 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-11T08:02:30.257 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.arena_block_size: 4194304 2024-09-11T08:02:30.257 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-11T08:02:30.257 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-11T08:02:30.258 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-09-11T08:02:30.258 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.disable_auto_compactions: 0 2024-09-11T08:02:30.258 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-11T08:02:30.258 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-11T08:02:30.258 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-11T08:02:30.258 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-11T08:02:30.258 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-11T08:02:30.258 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-11T08:02:30.258 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-11T08:02:30.258 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-11T08:02:30.258 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-11T08:02:30.258 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-11T08:02:30.258 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-11T08:02:30.259 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.inplace_update_support: 0 2024-09-11T08:02:30.259 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-11T08:02:30.259 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-11T08:02:30.259 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-11T08:02:30.259 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-11T08:02:30.259 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.bloom_locality: 0 2024-09-11T08:02:30.259 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.max_successive_merges: 0 2024-09-11T08:02:30.259 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-11T08:02:30.259 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.paranoid_file_checks: 0 2024-09-11T08:02:30.259 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.force_consistency_checks: 1 2024-09-11T08:02:30.260 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.report_bg_io_stats: 0 2024-09-11T08:02:30.260 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.ttl: 2592000 2024-09-11T08:02:30.260 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-11T08:02:30.260 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.enable_blob_files: false 2024-09-11T08:02:30.260 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.min_blob_size: 0 2024-09-11T08:02:30.260 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.blob_file_size: 268435456 2024-09-11T08:02:30.260 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-11T08:02:30.260 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-11T08:02:30.260 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-11T08:02:30.260 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-11T08:02:30.260 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2024-09-11T08:02:30.260 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: [db/version_set.cc:4082] Creating manifest 13 2024-09-11T08:02:30.260 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726041750196849, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-09-11T08:02:30.261 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2024-09-11T08:02:30.261 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-09-11T08:02:30.261 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726041750197591, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 85934, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 84026, "index_size": 242, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 11534, "raw_average_key_size": 48, "raw_value_size": 77702, "raw_average_value_size": 327, "num_data_blocks": 10, "num_entries": 237, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1726041750, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "9270f2f2-70d0-448b-b3a8-5aecde7b267f", "db_session_id": "S4317ENVB8KJQNW1SQ0D"}} 2024-09-11T08:02:30.261 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: [db/version_set.cc:4082] Creating manifest 15 2024-09-11T08:02:30.261 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726041750198079, "job": 1, "event": "recovery_finished"} 2024-09-11T08:02:30.261 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-11T08:02:30.261 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x557735258700 2024-09-11T08:02:30.261 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: DB pointer 0x557735228000 2024-09-11T08:02:30.261 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-09-11T08:02:30.261 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-09-11T08:02:30.261 INFO:journalctl@ceph.mon.a.smithi150.stdout: ** DB Stats ** 2024-09-11T08:02:30.262 INFO:journalctl@ceph.mon.a.smithi150.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-11T08:02:30.262 INFO:journalctl@ceph.mon.a.smithi150.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-11T08:02:30.262 INFO:journalctl@ceph.mon.a.smithi150.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-11T08:02:30.262 INFO:journalctl@ceph.mon.a.smithi150.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-11T08:02:30.262 INFO:journalctl@ceph.mon.a.smithi150.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-11T08:02:30.262 INFO:journalctl@ceph.mon.a.smithi150.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-09-11T08:02:30.262 INFO:journalctl@ceph.mon.a.smithi150.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-11T08:02:30.262 INFO:journalctl@ceph.mon.a.smithi150.stdout: 2024-09-11T08:02:30.262 INFO:journalctl@ceph.mon.a.smithi150.stdout: ** Compaction Stats [default] ** 2024-09-11T08:02:30.262 INFO:journalctl@ceph.mon.a.smithi150.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-11T08:02:30.262 INFO:journalctl@ceph.mon.a.smithi150.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-11T08:02:30.262 INFO:journalctl@ceph.mon.a.smithi150.stdout: L0 2/0 85.71 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 164.6 0.00 0.00 1 0.000 0 0 2024-09-11T08:02:30.263 INFO:journalctl@ceph.mon.a.smithi150.stdout: Sum 2/0 85.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 164.6 0.00 0.00 1 0.000 0 0 2024-09-11T08:02:30.263 INFO:journalctl@ceph.mon.a.smithi150.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 164.6 0.00 0.00 1 0.000 0 0 2024-09-11T08:02:30.263 INFO:journalctl@ceph.mon.a.smithi150.stdout: 2024-09-11T08:02:30.263 INFO:journalctl@ceph.mon.a.smithi150.stdout: ** Compaction Stats [default] ** 2024-09-11T08:02:30.263 INFO:journalctl@ceph.mon.a.smithi150.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-11T08:02:30.263 INFO:journalctl@ceph.mon.a.smithi150.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-11T08:02:30.263 INFO:journalctl@ceph.mon.a.smithi150.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 164.6 0.00 0.00 1 0.000 0 0 2024-09-11T08:02:30.263 INFO:journalctl@ceph.mon.a.smithi150.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-11T08:02:30.263 INFO:journalctl@ceph.mon.a.smithi150.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-11T08:02:30.263 INFO:journalctl@ceph.mon.a.smithi150.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-11T08:02:30.263 INFO:journalctl@ceph.mon.a.smithi150.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-11T08:02:30.263 INFO:journalctl@ceph.mon.a.smithi150.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-11T08:02:30.263 INFO:journalctl@ceph.mon.a.smithi150.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-11T08:02:30.263 INFO:journalctl@ceph.mon.a.smithi150.stdout: Cumulative compaction: 0.00 GB write, 25.57 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-11T08:02:30.264 INFO:journalctl@ceph.mon.a.smithi150.stdout: Interval compaction: 0.00 GB write, 25.57 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-11T08:02:30.264 INFO:journalctl@ceph.mon.a.smithi150.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-11T08:02:30.264 INFO:journalctl@ceph.mon.a.smithi150.stdout: 2024-09-11T08:02:30.264 INFO:journalctl@ceph.mon.a.smithi150.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-11T08:02:30.264 INFO:journalctl@ceph.mon.a.smithi150.stdout: 2024-09-11T08:02:30.264 INFO:journalctl@ceph.mon.a.smithi150.stdout: ** Compaction Stats [default] ** 2024-09-11T08:02:30.264 INFO:journalctl@ceph.mon.a.smithi150.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-11T08:02:30.264 INFO:journalctl@ceph.mon.a.smithi150.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-11T08:02:30.264 INFO:journalctl@ceph.mon.a.smithi150.stdout: L0 2/0 85.71 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 164.6 0.00 0.00 1 0.000 0 0 2024-09-11T08:02:30.264 INFO:journalctl@ceph.mon.a.smithi150.stdout: Sum 2/0 85.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 164.6 0.00 0.00 1 0.000 0 0 2024-09-11T08:02:30.264 INFO:journalctl@ceph.mon.a.smithi150.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-09-11T08:02:30.264 INFO:journalctl@ceph.mon.a.smithi150.stdout: 2024-09-11T08:02:30.264 INFO:journalctl@ceph.mon.a.smithi150.stdout: ** Compaction Stats [default] ** 2024-09-11T08:02:30.265 INFO:journalctl@ceph.mon.a.smithi150.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-11T08:02:30.265 INFO:journalctl@ceph.mon.a.smithi150.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-11T08:02:30.265 INFO:journalctl@ceph.mon.a.smithi150.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 164.6 0.00 0.00 1 0.000 0 0 2024-09-11T08:02:30.265 INFO:journalctl@ceph.mon.a.smithi150.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-11T08:02:30.265 INFO:journalctl@ceph.mon.a.smithi150.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-11T08:02:30.265 INFO:journalctl@ceph.mon.a.smithi150.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-11T08:02:30.265 INFO:journalctl@ceph.mon.a.smithi150.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-11T08:02:30.265 INFO:journalctl@ceph.mon.a.smithi150.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-11T08:02:30.265 INFO:journalctl@ceph.mon.a.smithi150.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-11T08:02:30.265 INFO:journalctl@ceph.mon.a.smithi150.stdout: Cumulative compaction: 0.00 GB write, 25.31 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-11T08:02:30.265 INFO:journalctl@ceph.mon.a.smithi150.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-11T08:02:30.265 INFO:journalctl@ceph.mon.a.smithi150.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-11T08:02:30.265 INFO:journalctl@ceph.mon.a.smithi150.stdout: 2024-09-11T08:02:30.266 INFO:journalctl@ceph.mon.a.smithi150.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-11T08:02:30.266 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: starting mon.a rank 0 at public addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] at bind addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid c1bddff4-7013-11ef-bcea-c7b262605968 2024-09-11T08:02:30.266 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: mon.a@-1(???) e1 preinit fsid c1bddff4-7013-11ef-bcea-c7b262605968 2024-09-11T08:02:30.266 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: mon.a@-1(???).mds e1 new map 2024-09-11T08:02:30.266 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: mon.a@-1(???).mds e1 print_map 2024-09-11T08:02:30.266 INFO:journalctl@ceph.mon.a.smithi150.stdout: e1 2024-09-11T08:02:30.266 INFO:journalctl@ceph.mon.a.smithi150.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-11T08:02:30.266 INFO:journalctl@ceph.mon.a.smithi150.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-11T08:02:30.266 INFO:journalctl@ceph.mon.a.smithi150.stdout: legacy client fscid: -1 2024-09-11T08:02:30.266 INFO:journalctl@ceph.mon.a.smithi150.stdout: 2024-09-11T08:02:30.266 INFO:journalctl@ceph.mon.a.smithi150.stdout: No filesystems configured 2024-09-11T08:02:30.266 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-11T08:02:30.266 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-11T08:02:30.266 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-11T08:02:30.267 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-11T08:02:30.267 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-11T08:02:30.267 INFO:teuthology.orchestra.run.smithi150.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-09-11T08:02:30.516 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 bash[29257]: d7334d5149102c624f81121c01d70bd96d37a60f0157b33fa338f4caf9ffa9de 2024-09-11T08:02:30.516 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 systemd[1]: Started Ceph mon.a for c1bddff4-7013-11ef-bcea-c7b262605968. 2024-09-11T08:02:30.516 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-11T08:02:30.516 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: monmap e1: 1 mons at {a=[v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-11T08:02:30.517 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: fsmap 2024-09-11T08:02:30.517 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: osdmap e1: 0 total, 0 up, 0 in 2024-09-11T08:02:30.517 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:30 smithi150 ceph-mon[29276]: mgrmap e1: no daemons active 2024-09-11T08:02:31.322 INFO:teuthology.orchestra.run.smithi150.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-11T08:02:31.323 INFO:teuthology.orchestra.run.smithi150.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-11T08:02:31.323 INFO:teuthology.orchestra.run.smithi150.stdout:Creating mgr... 2024-09-11T08:02:31.323 INFO:teuthology.orchestra.run.smithi150.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-11T08:02:31.323 INFO:teuthology.orchestra.run.smithi150.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-11T08:02:31.499 INFO:teuthology.orchestra.run.smithi150.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-c1bddff4-7013-11ef-bcea-c7b262605968@mgr.y 2024-09-11T08:02:31.500 INFO:teuthology.orchestra.run.smithi150.stdout:systemctl: stderr Failed to reset failed state of unit ceph-c1bddff4-7013-11ef-bcea-c7b262605968@mgr.y.service: Unit ceph-c1bddff4-7013-11ef-bcea-c7b262605968@mgr.y.service not loaded. 2024-09-11T08:02:31.648 INFO:teuthology.orchestra.run.smithi150.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-c1bddff4-7013-11ef-bcea-c7b262605968.target.wants/ceph-c1bddff4-7013-11ef-bcea-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-c1bddff4-7013-11ef-bcea-c7b262605968@.service. 2024-09-11T08:02:31.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:31 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2530733046' entity='client.admin' 2024-09-11T08:02:31.989 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:31 smithi150 podman[29488]: 2024-09-11 08:02:31.781385163 +0000 UTC m=+0.015902406 image pull 4a918172a49766385959227c53e75e2c302e923539324792cb5bee6cad0b2f2b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 2024-09-11T08:02:32.204 INFO:teuthology.orchestra.run.smithi150.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-11T08:02:32.204 INFO:teuthology.orchestra.run.smithi150.stdout:systemctl: stdout disabled 2024-09-11T08:02:32.212 INFO:teuthology.orchestra.run.smithi150.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-11T08:02:32.212 INFO:teuthology.orchestra.run.smithi150.stdout:systemctl: stdout inactive 2024-09-11T08:02:32.212 INFO:teuthology.orchestra.run.smithi150.stdout:firewalld.service is not enabled 2024-09-11T08:02:32.212 INFO:teuthology.orchestra.run.smithi150.stdout:Not possible to enable service . firewalld.service is not available 2024-09-11T08:02:32.222 INFO:teuthology.orchestra.run.smithi150.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-11T08:02:32.222 INFO:teuthology.orchestra.run.smithi150.stdout:systemctl: stdout disabled 2024-09-11T08:02:32.231 INFO:teuthology.orchestra.run.smithi150.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-11T08:02:32.231 INFO:teuthology.orchestra.run.smithi150.stdout:systemctl: stdout inactive 2024-09-11T08:02:32.231 INFO:teuthology.orchestra.run.smithi150.stdout:firewalld.service is not enabled 2024-09-11T08:02:32.231 INFO:teuthology.orchestra.run.smithi150.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-09-11T08:02:32.231 INFO:teuthology.orchestra.run.smithi150.stdout:Waiting for mgr to start... 2024-09-11T08:02:32.231 INFO:teuthology.orchestra.run.smithi150.stdout:Waiting for mgr... 2024-09-11T08:02:32.294 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:32 smithi150 podman[29488]: 2024-09-11 08:02:32.096911941 +0000 UTC m=+0.331429186 container init 995a40ad9a2ce775e2bdf0bf07fb082a87e9433284309f21045f29866f625a66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-09-11T08:02:32.294 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:32 smithi150 podman[29488]: 2024-09-11 08:02:32.099646153 +0000 UTC m=+0.334163398 container start 995a40ad9a2ce775e2bdf0bf07fb082a87e9433284309f21045f29866f625a66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0) 2024-09-11T08:02:32.294 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:32 smithi150 bash[29488]: 995a40ad9a2ce775e2bdf0bf07fb082a87e9433284309f21045f29866f625a66 2024-09-11T08:02:32.294 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:32 smithi150 systemd[1]: Started Ceph mgr.y for c1bddff4-7013-11ef-bcea-c7b262605968. 2024-09-11T08:02:32.294 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:32 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:02:32.217+0000 7fc17aa18ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-11T08:02:32.294 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:32 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:02:32.292+0000 7fc17aa18ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-11T08:02:32.622 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:32 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:02:32.357+0000 7fc17aa18ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-11T08:02:32.622 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:32 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:02:32.481+0000 7fc17aa18ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-11T08:02:32.623 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:32 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:02:32.621+0000 7fc17aa18ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-11T08:02:32.893 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:32 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2091047801' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-11T08:02:33.239 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:32 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:02:32.892+0000 7fc17aa18ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-11T08:02:33.239 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:32 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:02:32.953+0000 7fc17aa18ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-11T08:02:33.239 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:33 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:02:33.116+0000 7fc17aa18ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-11T08:02:33.426 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout 2024-09-11T08:02:33.426 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout { 2024-09-11T08:02:33.426 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "fsid": "c1bddff4-7013-11ef-bcea-c7b262605968", 2024-09-11T08:02:33.426 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "health": { 2024-09-11T08:02:33.426 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-11T08:02:33.427 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-11T08:02:33.427 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-11T08:02:33.427 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout }, 2024-09-11T08:02:33.427 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-11T08:02:33.427 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-11T08:02:33.427 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout 0 2024-09-11T08:02:33.427 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout ], 2024-09-11T08:02:33.427 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-11T08:02:33.427 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "a" 2024-09-11T08:02:33.427 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout ], 2024-09-11T08:02:33.427 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-09-11T08:02:33.427 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-11T08:02:33.427 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-11T08:02:33.427 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-09-11T08:02:33.428 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-11T08:02:33.428 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout }, 2024-09-11T08:02:33.428 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-11T08:02:33.428 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-11T08:02:33.428 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-11T08:02:33.428 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-11T08:02:33.428 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-11T08:02:33.428 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-11T08:02:33.428 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-11T08:02:33.428 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-11T08:02:33.431 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout }, 2024-09-11T08:02:33.431 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-11T08:02:33.431 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-11T08:02:33.431 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-11T08:02:33.431 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-11T08:02:33.431 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-11T08:02:33.431 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-11T08:02:33.432 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-11T08:02:33.432 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-11T08:02:33.432 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-11T08:02:33.432 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout }, 2024-09-11T08:02:33.432 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-11T08:02:33.432 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-11T08:02:33.432 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-11T08:02:33.432 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-11T08:02:33.432 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout }, 2024-09-11T08:02:33.432 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-11T08:02:33.432 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-11T08:02:33.432 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-11T08:02:33.432 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-11T08:02:33.432 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-11T08:02:33.433 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-11T08:02:33.433 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "restful" 2024-09-11T08:02:33.433 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout ], 2024-09-11T08:02:33.433 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-11T08:02:33.433 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout }, 2024-09-11T08:02:33.433 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-11T08:02:33.433 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-11T08:02:33.433 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "modified": "2024-09-11T08:02:15.345039+0000", 2024-09-11T08:02:33.433 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-11T08:02:33.433 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout }, 2024-09-11T08:02:33.433 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-11T08:02:33.433 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout } 2024-09-11T08:02:33.433 INFO:teuthology.orchestra.run.smithi150.stdout:mgr not available, waiting (1/15)... 2024-09-11T08:02:34.239 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:33 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:02:33.964+0000 7fc17aa18ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-11T08:02:34.239 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:34 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:02:34.027+0000 7fc17aa18ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-11T08:02:34.989 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:34 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:02:34.737+0000 7fc17aa18ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-11T08:02:34.989 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:34 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:02:34.848+0000 7fc17aa18ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-11T08:02:34.989 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:34 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:02:34.969+0000 7fc17aa18ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-11T08:02:35.413 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:35 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:02:35.028+0000 7fc17aa18ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-11T08:02:35.413 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:35 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:02:35.153+0000 7fc17aa18ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-11T08:02:35.413 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:35 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:02:35.220+0000 7fc17aa18ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-11T08:02:35.413 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:35 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:02:35.278+0000 7fc17aa18ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-11T08:02:35.413 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:35 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:02:35.411+0000 7fc17aa18ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-11T08:02:35.671 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:35 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:02:35.479+0000 7fc17aa18ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-11T08:02:35.671 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:35 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:02:35.545+0000 7fc17aa18ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-11T08:02:35.671 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:35 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:02:35.627+0000 7fc17aa18ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-11T08:02:35.929 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:35 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:02:35.687+0000 7fc17aa18ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-11T08:02:35.929 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:35 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:02:35.744+0000 7fc17aa18ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-11T08:02:36.219 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:35 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/4042969018' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-11T08:02:36.548 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout 2024-09-11T08:02:36.548 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout { 2024-09-11T08:02:36.548 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "fsid": "c1bddff4-7013-11ef-bcea-c7b262605968", 2024-09-11T08:02:36.548 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "health": { 2024-09-11T08:02:36.549 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-11T08:02:36.549 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-11T08:02:36.549 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-11T08:02:36.549 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout }, 2024-09-11T08:02:36.549 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-11T08:02:36.549 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-11T08:02:36.549 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout 0 2024-09-11T08:02:36.549 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout ], 2024-09-11T08:02:36.549 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-11T08:02:36.549 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "a" 2024-09-11T08:02:36.549 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout ], 2024-09-11T08:02:36.549 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-09-11T08:02:36.549 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-11T08:02:36.549 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-11T08:02:36.549 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-09-11T08:02:36.550 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-11T08:02:36.550 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout }, 2024-09-11T08:02:36.550 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-11T08:02:36.550 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-11T08:02:36.550 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-11T08:02:36.550 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-11T08:02:36.550 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-11T08:02:36.550 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-11T08:02:36.550 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-11T08:02:36.550 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-11T08:02:36.553 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout }, 2024-09-11T08:02:36.553 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-11T08:02:36.553 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-11T08:02:36.553 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-11T08:02:36.553 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-11T08:02:36.553 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-11T08:02:36.553 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-11T08:02:36.553 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-11T08:02:36.554 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-11T08:02:36.554 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-11T08:02:36.554 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout }, 2024-09-11T08:02:36.554 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-11T08:02:36.554 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-11T08:02:36.554 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-11T08:02:36.554 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-11T08:02:36.554 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout }, 2024-09-11T08:02:36.554 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-11T08:02:36.554 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-11T08:02:36.554 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-11T08:02:36.554 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-11T08:02:36.554 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-11T08:02:36.554 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-11T08:02:36.554 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "restful" 2024-09-11T08:02:36.555 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout ], 2024-09-11T08:02:36.555 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-11T08:02:36.555 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout }, 2024-09-11T08:02:36.555 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-11T08:02:36.555 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-11T08:02:36.555 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "modified": "2024-09-11T08:02:15.345039+0000", 2024-09-11T08:02:36.555 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-11T08:02:36.555 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout }, 2024-09-11T08:02:36.555 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-11T08:02:36.555 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout } 2024-09-11T08:02:36.555 INFO:teuthology.orchestra.run.smithi150.stdout:mgr not available, waiting (2/15)... 2024-09-11T08:02:36.980 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:36 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:02:36.506+0000 7fc17aa18ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-11T08:02:37.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:36 smithi150 ceph-mon[29276]: Activating manager daemon y 2024-09-11T08:02:37.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:36 smithi150 ceph-mon[29276]: mgrmap e2: y(active, starting, since 0.00298572s) 2024-09-11T08:02:37.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:36 smithi150 ceph-mon[29276]: from='mgr.14100 172.21.15.150:0/1379006188' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-11T08:02:37.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:36 smithi150 ceph-mon[29276]: from='mgr.14100 172.21.15.150:0/1379006188' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-11T08:02:37.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:36 smithi150 ceph-mon[29276]: from='mgr.14100 172.21.15.150:0/1379006188' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-11T08:02:37.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:36 smithi150 ceph-mon[29276]: from='mgr.14100 172.21.15.150:0/1379006188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-11T08:02:37.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:36 smithi150 ceph-mon[29276]: from='mgr.14100 172.21.15.150:0/1379006188' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-11T08:02:37.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:36 smithi150 ceph-mon[29276]: Manager daemon y is now available 2024-09-11T08:02:37.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:36 smithi150 ceph-mon[29276]: from='mgr.14100 172.21.15.150:0/1379006188' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-11T08:02:37.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:36 smithi150 ceph-mon[29276]: from='mgr.14100 172.21.15.150:0/1379006188' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-11T08:02:37.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:36 smithi150 ceph-mon[29276]: from='mgr.14100 172.21.15.150:0/1379006188' entity='mgr.y' 2024-09-11T08:02:37.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:36 smithi150 ceph-mon[29276]: from='mgr.14100 172.21.15.150:0/1379006188' entity='mgr.y' 2024-09-11T08:02:37.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:36 smithi150 ceph-mon[29276]: from='mgr.14100 172.21.15.150:0/1379006188' entity='mgr.y' 2024-09-11T08:02:38.780 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:38 smithi150 ceph-mon[29276]: mgrmap e3: y(active, since 1.00641s) 2024-09-11T08:02:39.807 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:39 smithi150 ceph-mon[29276]: mgrmap e4: y(active, since 2s) 2024-09-11T08:02:39.808 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:39 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1387432715' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-11T08:02:39.812 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout 2024-09-11T08:02:39.812 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout { 2024-09-11T08:02:39.812 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "fsid": "c1bddff4-7013-11ef-bcea-c7b262605968", 2024-09-11T08:02:39.813 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "health": { 2024-09-11T08:02:39.813 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-11T08:02:39.813 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-11T08:02:39.813 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-11T08:02:39.813 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout }, 2024-09-11T08:02:39.813 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-11T08:02:39.813 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-11T08:02:39.813 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout 0 2024-09-11T08:02:39.813 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout ], 2024-09-11T08:02:39.813 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-11T08:02:39.813 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "a" 2024-09-11T08:02:39.813 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout ], 2024-09-11T08:02:39.813 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-09-11T08:02:39.813 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-11T08:02:39.814 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-11T08:02:39.814 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-09-11T08:02:39.814 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-11T08:02:39.814 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout }, 2024-09-11T08:02:39.814 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-11T08:02:39.814 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-11T08:02:39.814 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-11T08:02:39.814 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-11T08:02:39.814 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-11T08:02:39.814 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-11T08:02:39.814 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-11T08:02:39.814 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-11T08:02:39.817 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout }, 2024-09-11T08:02:39.817 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-11T08:02:39.818 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-11T08:02:39.818 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-11T08:02:39.818 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-11T08:02:39.818 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-11T08:02:39.818 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-11T08:02:39.818 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-11T08:02:39.818 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-11T08:02:39.818 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-11T08:02:39.818 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout }, 2024-09-11T08:02:39.818 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-11T08:02:39.818 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-11T08:02:39.818 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-11T08:02:39.818 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-11T08:02:39.818 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout }, 2024-09-11T08:02:39.819 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-11T08:02:39.819 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-11T08:02:39.819 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-11T08:02:39.819 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-11T08:02:39.819 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-11T08:02:39.819 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-11T08:02:39.819 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "restful" 2024-09-11T08:02:39.819 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout ], 2024-09-11T08:02:39.819 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-11T08:02:39.819 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout }, 2024-09-11T08:02:39.819 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-11T08:02:39.819 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-11T08:02:39.819 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "modified": "2024-09-11T08:02:15.345039+0000", 2024-09-11T08:02:39.820 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-11T08:02:39.820 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout }, 2024-09-11T08:02:39.820 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-11T08:02:39.820 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout } 2024-09-11T08:02:39.820 INFO:teuthology.orchestra.run.smithi150.stdout:mgr is available 2024-09-11T08:02:40.697 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:40 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3111497389' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-11T08:02:40.976 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout 2024-09-11T08:02:40.977 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout [global] 2024-09-11T08:02:40.977 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout fsid = c1bddff4-7013-11ef-bcea-c7b262605968 2024-09-11T08:02:40.977 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-11T08:02:40.977 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-11T08:02:40.977 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-11T08:02:40.977 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-11T08:02:40.977 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-11T08:02:40.977 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout 2024-09-11T08:02:40.977 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-11T08:02:40.977 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-11T08:02:40.977 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout 2024-09-11T08:02:40.977 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout [osd] 2024-09-11T08:02:40.977 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-11T08:02:40.978 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-11T08:02:40.978 INFO:teuthology.orchestra.run.smithi150.stdout:Enabling cephadm module... 2024-09-11T08:02:41.980 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:41 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3338331532' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-09-11T08:02:41.980 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:41 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ignoring --setuser ceph since I am not root 2024-09-11T08:02:41.980 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:41 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ignoring --setgroup ceph since I am not root 2024-09-11T08:02:41.980 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:41 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:02:41.751+0000 7f9126e30ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-11T08:02:41.980 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:41 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:02:41.805+0000 7f9126e30ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-11T08:02:41.980 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:41 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:02:41.869+0000 7f9126e30ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-11T08:02:42.239 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:41 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:02:41.996+0000 7f9126e30ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-11T08:02:42.239 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:42 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:02:42.131+0000 7f9126e30ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-11T08:02:42.622 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:42 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:02:42.394+0000 7f9126e30ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-11T08:02:42.622 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:42 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:02:42.455+0000 7f9126e30ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-11T08:02:42.622 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:42 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:02:42.619+0000 7f9126e30ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-11T08:02:42.918 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:42 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3338331532' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-11T08:02:42.918 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:42 smithi150 ceph-mon[29276]: mgrmap e5: y(active, since 5s) 2024-09-11T08:02:43.488 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout { 2024-09-11T08:02:43.488 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-09-11T08:02:43.488 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-11T08:02:43.488 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-11T08:02:43.488 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-11T08:02:43.488 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout } 2024-09-11T08:02:43.488 INFO:teuthology.orchestra.run.smithi150.stdout:Waiting for the mgr to restart... 2024-09-11T08:02:43.488 INFO:teuthology.orchestra.run.smithi150.stdout:Waiting for mgr epoch 5... 2024-09-11T08:02:43.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:43 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1151902624' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-11T08:02:43.739 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:43 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:02:43.482+0000 7f9126e30ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-11T08:02:43.739 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:43 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:02:43.554+0000 7f9126e30ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-11T08:02:44.553 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:44 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:02:44.265+0000 7f9126e30ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-11T08:02:44.554 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:44 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:02:44.374+0000 7f9126e30ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-11T08:02:44.554 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:44 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:02:44.493+0000 7f9126e30ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-11T08:02:44.554 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:44 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:02:44.551+0000 7f9126e30ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-11T08:02:44.930 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:44 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:02:44.675+0000 7f9126e30ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-11T08:02:44.931 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:44 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:02:44.740+0000 7f9126e30ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-11T08:02:44.931 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:44 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:02:44.796+0000 7f9126e30ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-11T08:02:44.931 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:44 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:02:44.928+0000 7f9126e30ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-11T08:02:45.193 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:44 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:02:44.991+0000 7f9126e30ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-11T08:02:45.193 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:45 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:02:45.055+0000 7f9126e30ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-11T08:02:45.193 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:45 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:02:45.134+0000 7f9126e30ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-11T08:02:45.193 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:45 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:02:45.190+0000 7f9126e30ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-11T08:02:45.489 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:45 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:02:45.247+0000 7f9126e30ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-11T08:02:46.239 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:45 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:02:45.983+0000 7f9126e30ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-11T08:02:46.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:46 smithi150 ceph-mon[29276]: Active manager daemon y restarted 2024-09-11T08:02:46.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:46 smithi150 ceph-mon[29276]: Activating manager daemon y 2024-09-11T08:02:46.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:46 smithi150 ceph-mon[29276]: osdmap e2: 0 total, 0 up, 0 in 2024-09-11T08:02:46.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:46 smithi150 ceph-mon[29276]: mgrmap e6: y(active, starting, since 0.00366339s) 2024-09-11T08:02:46.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:46 smithi150 ceph-mon[29276]: from='mgr.14118 172.21.15.150:0/3162717040' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-11T08:02:46.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:46 smithi150 ceph-mon[29276]: from='mgr.14118 172.21.15.150:0/3162717040' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-11T08:02:46.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:46 smithi150 ceph-mon[29276]: from='mgr.14118 172.21.15.150:0/3162717040' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-11T08:02:46.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:46 smithi150 ceph-mon[29276]: from='mgr.14118 172.21.15.150:0/3162717040' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-11T08:02:46.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:46 smithi150 ceph-mon[29276]: from='mgr.14118 172.21.15.150:0/3162717040' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-11T08:02:46.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:46 smithi150 ceph-mon[29276]: Manager daemon y is now available 2024-09-11T08:02:46.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:46 smithi150 ceph-mon[29276]: from='mgr.14118 172.21.15.150:0/3162717040' entity='mgr.y' 2024-09-11T08:02:46.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:46 smithi150 ceph-mon[29276]: from='mgr.14118 172.21.15.150:0/3162717040' entity='mgr.y' 2024-09-11T08:02:46.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:46 smithi150 ceph-mon[29276]: from='mgr.14118 172.21.15.150:0/3162717040' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:02:46.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:46 smithi150 ceph-mon[29276]: from='mgr.14118 172.21.15.150:0/3162717040' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:02:46.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:46 smithi150 ceph-mon[29276]: from='mgr.14118 172.21.15.150:0/3162717040' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-11T08:02:46.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:46 smithi150 ceph-mon[29276]: from='mgr.14118 172.21.15.150:0/3162717040' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-11T08:02:47.290 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:47 smithi150 ceph-mon[29276]: Found migration_current of "None". Setting to last migration. 2024-09-11T08:02:47.290 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:47 smithi150 ceph-mon[29276]: mgrmap e7: y(active, since 1.00505s) 2024-09-11T08:02:47.560 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout { 2024-09-11T08:02:47.560 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-09-11T08:02:47.560 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-11T08:02:47.560 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout } 2024-09-11T08:02:47.560 INFO:teuthology.orchestra.run.smithi150.stdout:mgr epoch 5 is available 2024-09-11T08:02:47.560 INFO:teuthology.orchestra.run.smithi150.stdout:Setting orchestrator backend to cephadm... 2024-09-11T08:02:48.188 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:48 smithi150 ceph-mon[29276]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-11T08:02:48.188 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:48 smithi150 ceph-mon[29276]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-11T08:02:48.188 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:48 smithi150 ceph-mon[29276]: from='mgr.14118 172.21.15.150:0/3162717040' entity='mgr.y' 2024-09-11T08:02:48.188 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:48 smithi150 ceph-mon[29276]: from='mgr.14118 172.21.15.150:0/3162717040' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:02:48.188 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:47 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:02:47.819+0000 7f90d921e640 -1 log_channel(cephadm) log [ERR] : [11/Sep/2024:08:02:47] ENGINE Error in HTTPServer.serve 2024-09-11T08:02:48.188 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:47 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: Traceback (most recent call last): 2024-09-11T08:02:48.188 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:47 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-11T08:02:48.188 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:47 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: self._connections.run(self.expiration_interval) 2024-09-11T08:02:48.189 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:47 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-11T08:02:48.189 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:47 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: self._run(expiration_interval) 2024-09-11T08:02:48.189 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:47 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-11T08:02:48.189 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:47 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: new_conn = self._from_server_socket(self.server.socket) 2024-09-11T08:02:48.189 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:47 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-11T08:02:48.189 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:47 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-11T08:02:48.189 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:47 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-11T08:02:48.189 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:47 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: s = self.context.wrap_socket( 2024-09-11T08:02:48.189 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:47 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-11T08:02:48.189 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:47 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: return self.sslsocket_class._create( 2024-09-11T08:02:48.189 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:47 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-11T08:02:48.189 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:47 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: self.do_handshake() 2024-09-11T08:02:48.189 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:47 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-11T08:02:48.189 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:47 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: self._sslobj.do_handshake() 2024-09-11T08:02:48.190 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:47 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-11T08:02:48.190 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:47 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:02:49.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:49 smithi150 ceph-mon[29276]: [11/Sep/2024:08:02:47] ENGINE Bus STARTING 2024-09-11T08:02:49.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:49 smithi150 ceph-mon[29276]: [11/Sep/2024:08:02:47] ENGINE Serving on https://172.21.15.150:7150 2024-09-11T08:02:49.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:49 smithi150 ceph-mon[29276]: [11/Sep/2024:08:02:47] ENGINE Bus STARTED 2024-09-11T08:02:49.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:49 smithi150 ceph-mon[29276]: [11/Sep/2024:08:02:47] ENGINE Error in HTTPServer.serve 2024-09-11T08:02:49.489 INFO:journalctl@ceph.mon.a.smithi150.stdout: Traceback (most recent call last): 2024-09-11T08:02:49.489 INFO:journalctl@ceph.mon.a.smithi150.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-11T08:02:49.489 INFO:journalctl@ceph.mon.a.smithi150.stdout: self._connections.run(self.expiration_interval) 2024-09-11T08:02:49.489 INFO:journalctl@ceph.mon.a.smithi150.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-11T08:02:49.489 INFO:journalctl@ceph.mon.a.smithi150.stdout: self._run(expiration_interval) 2024-09-11T08:02:49.490 INFO:journalctl@ceph.mon.a.smithi150.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-11T08:02:49.490 INFO:journalctl@ceph.mon.a.smithi150.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-11T08:02:49.490 INFO:journalctl@ceph.mon.a.smithi150.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-11T08:02:49.490 INFO:journalctl@ceph.mon.a.smithi150.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-11T08:02:49.490 INFO:journalctl@ceph.mon.a.smithi150.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-11T08:02:49.490 INFO:journalctl@ceph.mon.a.smithi150.stdout: s = self.context.wrap_socket( 2024-09-11T08:02:49.490 INFO:journalctl@ceph.mon.a.smithi150.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-11T08:02:49.490 INFO:journalctl@ceph.mon.a.smithi150.stdout: return self.sslsocket_class._create( 2024-09-11T08:02:49.490 INFO:journalctl@ceph.mon.a.smithi150.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-11T08:02:49.490 INFO:journalctl@ceph.mon.a.smithi150.stdout: self.do_handshake() 2024-09-11T08:02:49.490 INFO:journalctl@ceph.mon.a.smithi150.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-11T08:02:49.490 INFO:journalctl@ceph.mon.a.smithi150.stdout: self._sslobj.do_handshake() 2024-09-11T08:02:49.490 INFO:journalctl@ceph.mon.a.smithi150.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-11T08:02:49.491 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:49 smithi150 ceph-mon[29276]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:02:49.491 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:49 smithi150 ceph-mon[29276]: from='mgr.14118 172.21.15.150:0/3162717040' entity='mgr.y' 2024-09-11T08:02:49.491 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:49 smithi150 ceph-mon[29276]: from='mgr.14118 172.21.15.150:0/3162717040' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:02:49.491 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:49 smithi150 ceph-mon[29276]: mgrmap e8: y(active, since 2s) 2024-09-11T08:02:50.097 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-11T08:02:50.097 INFO:teuthology.orchestra.run.smithi150.stdout:Generating ssh key... 2024-09-11T08:02:50.413 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:50 smithi150 ceph-mon[29276]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:02:51.067 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:50 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: Generating public/private rsa key pair. 2024-09-11T08:02:51.067 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:50 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: Your identification has been saved in /tmp/tmpuc87fo55/key 2024-09-11T08:02:51.067 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:50 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: Your public key has been saved in /tmp/tmpuc87fo55/key.pub 2024-09-11T08:02:51.067 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:50 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: The key fingerprint is: 2024-09-11T08:02:51.067 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:50 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: SHA256:alpCzjho+FnigHhP8/0YuYKgjJC1DSMCZyy4qx49boc ceph-c1bddff4-7013-11ef-bcea-c7b262605968 2024-09-11T08:02:51.067 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:50 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: The key's randomart image is: 2024-09-11T08:02:51.067 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:50 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: +---[RSA 3072]----+ 2024-09-11T08:02:51.067 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:50 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: |.. | 2024-09-11T08:02:51.067 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:50 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: |+ + | 2024-09-11T08:02:51.067 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:50 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: |.= | 2024-09-11T08:02:51.068 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:50 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: |+ + | 2024-09-11T08:02:51.068 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:50 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: |.= =. S | 2024-09-11T08:02:51.068 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:50 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: |B.+=. . . | 2024-09-11T08:02:51.068 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:50 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: |@+=+O.+ o | 2024-09-11T08:02:51.068 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:50 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: |=*oEoO.. + | 2024-09-11T08:02:51.068 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:50 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: |..=.+ ..+.. | 2024-09-11T08:02:51.068 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:02:50 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: +----[SHA256]-----+ 2024-09-11T08:02:51.392 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:51 smithi150 ceph-mon[29276]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:02:51.392 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:51 smithi150 ceph-mon[29276]: Generating ssh key... 2024-09-11T08:02:51.392 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:51 smithi150 ceph-mon[29276]: from='mgr.14118 172.21.15.150:0/3162717040' entity='mgr.y' 2024-09-11T08:02:51.392 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:51 smithi150 ceph-mon[29276]: from='mgr.14118 172.21.15.150:0/3162717040' entity='mgr.y' 2024-09-11T08:02:52.442 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYRAuDBpzOf8s8bDTdKUkjO350Fskbtdto09u0sN4UP08cs1dDJpmjqAvxbyOUdCcfQYgujeUyjMFMsN40Pl/PmZLzCxhXDIxhXjl9K0fpdAOfAajdjF1dkp+FTRprGHNI1oiGHTBsIRlJIWRPJHC06QiUlL7+Vh/VosLK5N0sgfdWirxWoL6nNB7XSFrIXis2QGx5KZkec9vvKXkwraAMpwuA2zfmETxXIPe9kMiTUo52FLFyZgbf5ByzhO/KnHDXbvBB9NEAqvTNwhTjAVNoCwlPcqbKSNX1GIpQ2VokFX3slm3zXuIYm9tONj4tRgztVxn5dC1YXT5W4ZyAVaen7LB4qZdXEIP1GiwjtTA0B8kwR/Wxkb5kz77X/sTKgU3CUON96C2Xvnoyx15GRDI9C6uT2byBFf+whB/mZtshUlwbfaGR7xa1tEghEUKtChkdOAkoZ/cuHEDC38a/JMBfN84+lDMTwwLIWXWESvlCkhjXLzY4B+p7n4Pk5wr+Gjc= ceph-c1bddff4-7013-11ef-bcea-c7b262605968 2024-09-11T08:02:52.442 INFO:teuthology.orchestra.run.smithi150.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-11T08:02:52.442 INFO:teuthology.orchestra.run.smithi150.stdout:Adding key to root@localhost authorized_keys... 2024-09-11T08:02:52.449 INFO:teuthology.orchestra.run.smithi150.stdout:Adding host smithi150... 2024-09-11T08:02:52.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:52 smithi150 ceph-mon[29276]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:02:53.926 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:53 smithi150 ceph-mon[29276]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi150", "addr": "172.21.15.150", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:02:54.858 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:54 smithi150 ceph-mon[29276]: Deploying cephadm binary to smithi150 2024-09-11T08:02:56.401 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout Added host 'smithi150' with addr '172.21.15.150' 2024-09-11T08:02:56.402 INFO:teuthology.orchestra.run.smithi150.stdout:Deploying unmanaged mon service... 2024-09-11T08:02:57.021 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:56 smithi150 ceph-mon[29276]: from='mgr.14118 172.21.15.150:0/3162717040' entity='mgr.y' 2024-09-11T08:02:57.022 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:56 smithi150 ceph-mon[29276]: Added host smithi150 2024-09-11T08:02:57.022 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:56 smithi150 ceph-mon[29276]: from='mgr.14118 172.21.15.150:0/3162717040' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:02:57.592 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-11T08:02:57.593 INFO:teuthology.orchestra.run.smithi150.stdout:Deploying unmanaged mgr service... 2024-09-11T08:02:58.187 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:58 smithi150 ceph-mon[29276]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:02:58.187 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:58 smithi150 ceph-mon[29276]: Saving service mon spec with placement count:5 2024-09-11T08:02:58.187 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:58 smithi150 ceph-mon[29276]: from='mgr.14118 172.21.15.150:0/3162717040' entity='mgr.y' 2024-09-11T08:02:59.703 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:59 smithi150 ceph-mon[29276]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:02:59.703 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:59 smithi150 ceph-mon[29276]: Saving service mgr spec with placement count:2 2024-09-11T08:02:59.704 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:59 smithi150 ceph-mon[29276]: from='mgr.14118 172.21.15.150:0/3162717040' entity='mgr.y' 2024-09-11T08:02:59.704 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:02:59 smithi150 ceph-mon[29276]: from='mgr.14118 172.21.15.150:0/3162717040' entity='mgr.y' 2024-09-11T08:02:59.705 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-11T08:03:01.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:01 smithi150 ceph-mon[29276]: from='mgr.14118 172.21.15.150:0/3162717040' entity='mgr.y' 2024-09-11T08:03:01.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:01 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/4190861486' entity='client.admin' 2024-09-11T08:03:02.818 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:02 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/229108542' entity='client.admin' 2024-09-11T08:03:02.818 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:02 smithi150 ceph-mon[29276]: from='mgr.14118 172.21.15.150:0/3162717040' entity='mgr.y' 2024-09-11T08:03:02.820 INFO:teuthology.orchestra.run.smithi150.stdout:Enabling the dashboard module... 2024-09-11T08:03:04.570 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:04 smithi150 ceph-mon[29276]: from='mgr.14118 172.21.15.150:0/3162717040' entity='mgr.y' 2024-09-11T08:03:04.570 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:04 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2437906014' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-09-11T08:03:04.570 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:03:04 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ignoring --setuser ceph since I am not root 2024-09-11T08:03:04.570 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:03:04 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ignoring --setgroup ceph since I am not root 2024-09-11T08:03:04.570 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:03:04 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:03:04.447+0000 7efce5526ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-11T08:03:04.570 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:03:04 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:03:04.502+0000 7efce5526ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-11T08:03:04.570 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:03:04 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:03:04.568+0000 7efce5526ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-11T08:03:04.831 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:03:04 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:03:04.694+0000 7efce5526ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-11T08:03:04.831 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:03:04 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:03:04.830+0000 7efce5526ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-11T08:03:05.346 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:05 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2437906014' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-11T08:03:05.346 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:05 smithi150 ceph-mon[29276]: mgrmap e9: y(active, since 18s) 2024-09-11T08:03:05.346 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:03:05 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:03:05.091+0000 7efce5526ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-11T08:03:05.346 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:03:05 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:03:05.152+0000 7efce5526ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-11T08:03:05.346 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:03:05 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:03:05.319+0000 7efce5526ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-11T08:03:06.420 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:06 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2155194388' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-11T08:03:06.420 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:03:06 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:03:06.194+0000 7efce5526ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-11T08:03:06.420 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:03:06 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:03:06.258+0000 7efce5526ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-11T08:03:06.422 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout { 2024-09-11T08:03:06.423 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-09-11T08:03:06.423 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-11T08:03:06.423 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-11T08:03:06.423 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-11T08:03:06.423 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout } 2024-09-11T08:03:06.423 INFO:teuthology.orchestra.run.smithi150.stdout:Waiting for the mgr to restart... 2024-09-11T08:03:06.424 INFO:teuthology.orchestra.run.smithi150.stdout:Waiting for mgr epoch 9... 2024-09-11T08:03:07.282 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:03:06 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:03:06.989+0000 7efce5526ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-11T08:03:07.282 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:03:07 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:03:07.101+0000 7efce5526ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-11T08:03:07.282 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:03:07 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:03:07.222+0000 7efce5526ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-11T08:03:07.283 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:03:07 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:03:07.281+0000 7efce5526ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-11T08:03:07.532 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:03:07 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:03:07.406+0000 7efce5526ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-11T08:03:07.533 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:03:07 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:03:07.472+0000 7efce5526ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-11T08:03:07.533 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:03:07 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:03:07.531+0000 7efce5526ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-11T08:03:07.796 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:03:07 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:03:07.666+0000 7efce5526ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-11T08:03:07.796 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:03:07 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:03:07.728+0000 7efce5526ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-11T08:03:07.796 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:03:07 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:03:07.794+0000 7efce5526ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-11T08:03:08.239 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:03:07 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:03:07.875+0000 7efce5526ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-11T08:03:08.239 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:03:07 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:03:07.932+0000 7efce5526ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-11T08:03:08.239 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:03:07 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:03:07.988+0000 7efce5526ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-11T08:03:09.239 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:03:08 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:03:08.738+0000 7efce5526ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-11T08:03:09.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:08 smithi150 ceph-mon[29276]: Active manager daemon y restarted 2024-09-11T08:03:09.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:08 smithi150 ceph-mon[29276]: Activating manager daemon y 2024-09-11T08:03:09.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:08 smithi150 ceph-mon[29276]: osdmap e3: 0 total, 0 up, 0 in 2024-09-11T08:03:09.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:08 smithi150 ceph-mon[29276]: mgrmap e10: y(active, starting, since 0.00563454s) 2024-09-11T08:03:09.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:08 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-11T08:03:09.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:08 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-11T08:03:09.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:08 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-11T08:03:09.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:08 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-11T08:03:09.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:08 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-11T08:03:09.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:08 smithi150 ceph-mon[29276]: Manager daemon y is now available 2024-09-11T08:03:09.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:08 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:03:09.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:08 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-11T08:03:09.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:08 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-11T08:03:10.238 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:03:09 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:03:09.972+0000 7efc96083640 -1 log_channel(cephadm) log [ERR] : [11/Sep/2024:08:03:09] ENGINE Error in HTTPServer.serve 2024-09-11T08:03:10.239 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:03:09 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: Traceback (most recent call last): 2024-09-11T08:03:10.239 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:03:09 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-11T08:03:10.239 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:03:09 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: self._connections.run(self.expiration_interval) 2024-09-11T08:03:10.239 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:03:09 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-11T08:03:10.239 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:03:09 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: self._run(expiration_interval) 2024-09-11T08:03:10.239 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:03:09 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-11T08:03:10.239 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:03:09 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: new_conn = self._from_server_socket(self.server.socket) 2024-09-11T08:03:10.239 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:03:09 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-11T08:03:10.239 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:03:09 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-11T08:03:10.239 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:03:09 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-11T08:03:10.239 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:03:09 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: s = self.context.wrap_socket( 2024-09-11T08:03:10.239 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:03:09 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-11T08:03:10.239 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:03:09 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: return self.sslsocket_class._create( 2024-09-11T08:03:10.239 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:03:09 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-11T08:03:10.240 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:03:09 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: self.do_handshake() 2024-09-11T08:03:10.240 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:03:09 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-11T08:03:10.240 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:03:09 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: self._sslobj.do_handshake() 2024-09-11T08:03:10.240 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:03:09 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-11T08:03:10.240 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:03:09 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:03:10.507 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout { 2024-09-11T08:03:10.507 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-09-11T08:03:10.507 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-11T08:03:10.507 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout } 2024-09-11T08:03:10.507 INFO:teuthology.orchestra.run.smithi150.stdout:mgr epoch 9 is available 2024-09-11T08:03:10.507 INFO:teuthology.orchestra.run.smithi150.stdout:Generating a dashboard self-signed certificate... 2024-09-11T08:03:10.857 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:10 smithi150 ceph-mon[29276]: mgrmap e11: y(active, since 1.00569s) 2024-09-11T08:03:10.857 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:10 smithi150 ceph-mon[29276]: [11/Sep/2024:08:03:09] ENGINE Bus STARTING 2024-09-11T08:03:10.857 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:10 smithi150 ceph-mon[29276]: [11/Sep/2024:08:03:09] ENGINE Serving on https://172.21.15.150:7150 2024-09-11T08:03:10.857 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:10 smithi150 ceph-mon[29276]: [11/Sep/2024:08:03:09] ENGINE Bus STARTED 2024-09-11T08:03:10.857 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:10 smithi150 ceph-mon[29276]: [11/Sep/2024:08:03:09] ENGINE Error in HTTPServer.serve 2024-09-11T08:03:10.858 INFO:journalctl@ceph.mon.a.smithi150.stdout: Traceback (most recent call last): 2024-09-11T08:03:10.858 INFO:journalctl@ceph.mon.a.smithi150.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-11T08:03:10.858 INFO:journalctl@ceph.mon.a.smithi150.stdout: self._connections.run(self.expiration_interval) 2024-09-11T08:03:10.858 INFO:journalctl@ceph.mon.a.smithi150.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-11T08:03:10.858 INFO:journalctl@ceph.mon.a.smithi150.stdout: self._run(expiration_interval) 2024-09-11T08:03:10.858 INFO:journalctl@ceph.mon.a.smithi150.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-11T08:03:10.858 INFO:journalctl@ceph.mon.a.smithi150.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-11T08:03:10.858 INFO:journalctl@ceph.mon.a.smithi150.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-11T08:03:10.858 INFO:journalctl@ceph.mon.a.smithi150.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-11T08:03:10.858 INFO:journalctl@ceph.mon.a.smithi150.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-11T08:03:10.858 INFO:journalctl@ceph.mon.a.smithi150.stdout: s = self.context.wrap_socket( 2024-09-11T08:03:10.858 INFO:journalctl@ceph.mon.a.smithi150.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-11T08:03:10.858 INFO:journalctl@ceph.mon.a.smithi150.stdout: return self.sslsocket_class._create( 2024-09-11T08:03:10.858 INFO:journalctl@ceph.mon.a.smithi150.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-11T08:03:10.858 INFO:journalctl@ceph.mon.a.smithi150.stdout: self.do_handshake() 2024-09-11T08:03:10.859 INFO:journalctl@ceph.mon.a.smithi150.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-11T08:03:10.859 INFO:journalctl@ceph.mon.a.smithi150.stdout: self._sslobj.do_handshake() 2024-09-11T08:03:10.859 INFO:journalctl@ceph.mon.a.smithi150.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-11T08:03:10.859 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:10 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:11.780 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-09-11T08:03:11.780 INFO:teuthology.orchestra.run.smithi150.stdout:Creating initial admin user... 2024-09-11T08:03:12.164 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:11 smithi150 ceph-mon[29276]: mgrmap e12: y(active, since 2s) 2024-09-11T08:03:12.164 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:11 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:12.164 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:11 smithi150 ceph-mon[29276]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:03:12.164 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:11 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:12.164 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:11 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:12.164 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:11 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:13.947 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:13 smithi150 ceph-mon[29276]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:03:13.948 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:13 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:13.950 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$0e3t6/0mbCpOw99oT03JbOU0iUVulCpLY7ePMNy1RhpGmHRh55IXa", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1726041792, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-11T08:03:13.951 INFO:teuthology.orchestra.run.smithi150.stdout:Fetching dashboard port number... 2024-09-11T08:03:14.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:14 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1025928269' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-09-11T08:03:15.065 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout 8443 2024-09-11T08:03:15.076 INFO:teuthology.orchestra.run.smithi150.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-11T08:03:15.076 INFO:teuthology.orchestra.run.smithi150.stdout:systemctl: stdout disabled 2024-09-11T08:03:15.084 INFO:teuthology.orchestra.run.smithi150.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-11T08:03:15.084 INFO:teuthology.orchestra.run.smithi150.stdout:systemctl: stdout inactive 2024-09-11T08:03:15.084 INFO:teuthology.orchestra.run.smithi150.stdout:firewalld.service is not enabled 2024-09-11T08:03:15.084 INFO:teuthology.orchestra.run.smithi150.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-11T08:03:15.086 INFO:teuthology.orchestra.run.smithi150.stdout:Ceph Dashboard is now available at: 2024-09-11T08:03:15.086 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:03:15.086 INFO:teuthology.orchestra.run.smithi150.stdout: URL: https://smithi150.front.sepia.ceph.com:8443/ 2024-09-11T08:03:15.086 INFO:teuthology.orchestra.run.smithi150.stdout: User: admin 2024-09-11T08:03:15.086 INFO:teuthology.orchestra.run.smithi150.stdout: Password: rmop5rhi50 2024-09-11T08:03:15.086 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:03:15.087 INFO:teuthology.orchestra.run.smithi150.stdout:Saving cluster configuration to /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/config directory 2024-09-11T08:03:15.087 INFO:teuthology.orchestra.run.smithi150.stdout:Enabling autotune for osd_memory_target 2024-09-11T08:03:16.961 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:16 smithi150 ceph-mon[29276]: mgrmap e13: y(active, since 6s) 2024-09-11T08:03:17.510 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-11T08:03:17.510 INFO:teuthology.orchestra.run.smithi150.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-11T08:03:17.510 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:03:17.510 INFO:teuthology.orchestra.run.smithi150.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-11T08:03:17.510 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:03:17.510 INFO:teuthology.orchestra.run.smithi150.stdout:Or, if you are only running a single cluster on this host: 2024-09-11T08:03:17.511 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:03:17.511 INFO:teuthology.orchestra.run.smithi150.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-09-11T08:03:17.511 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:03:17.511 INFO:teuthology.orchestra.run.smithi150.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-11T08:03:17.511 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:03:17.511 INFO:teuthology.orchestra.run.smithi150.stdout: ceph telemetry on 2024-09-11T08:03:17.511 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:03:17.511 INFO:teuthology.orchestra.run.smithi150.stdout:For more information see: 2024-09-11T08:03:17.511 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:03:17.511 INFO:teuthology.orchestra.run.smithi150.stdout: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-09-11T08:03:17.511 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:03:17.511 INFO:teuthology.orchestra.run.smithi150.stdout:Bootstrap complete. 2024-09-11T08:03:17.545 INFO:tasks.cephadm:Fetching config... 2024-09-11T08:03:17.546 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-11T08:03:17.546 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-11T08:03:17.561 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-11T08:03:17.561 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-11T08:03:17.561 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-11T08:03:17.617 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-11T08:03:17.617 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-11T08:03:17.618 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/keyring of=/dev/stdout 2024-09-11T08:03:17.683 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-11T08:03:17.683 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-11T08:03:17.683 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-11T08:03:17.738 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-11T08:03:17.738 DEBUG:teuthology.orchestra.run.smithi150:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYRAuDBpzOf8s8bDTdKUkjO350Fskbtdto09u0sN4UP08cs1dDJpmjqAvxbyOUdCcfQYgujeUyjMFMsN40Pl/PmZLzCxhXDIxhXjl9K0fpdAOfAajdjF1dkp+FTRprGHNI1oiGHTBsIRlJIWRPJHC06QiUlL7+Vh/VosLK5N0sgfdWirxWoL6nNB7XSFrIXis2QGx5KZkec9vvKXkwraAMpwuA2zfmETxXIPe9kMiTUo52FLFyZgbf5ByzhO/KnHDXbvBB9NEAqvTNwhTjAVNoCwlPcqbKSNX1GIpQ2VokFX3slm3zXuIYm9tONj4tRgztVxn5dC1YXT5W4ZyAVaen7LB4qZdXEIP1GiwjtTA0B8kwR/Wxkb5kz77X/sTKgU3CUON96C2Xvnoyx15GRDI9C6uT2byBFf+whB/mZtshUlwbfaGR7xa1tEghEUKtChkdOAkoZ/cuHEDC38a/JMBfN84+lDMTwwLIWXWESvlCkhjXLzY4B+p7n4Pk5wr+Gjc= ceph-c1bddff4-7013-11ef-bcea-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-11T08:03:17.811 INFO:teuthology.orchestra.run.smithi150.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYRAuDBpzOf8s8bDTdKUkjO350Fskbtdto09u0sN4UP08cs1dDJpmjqAvxbyOUdCcfQYgujeUyjMFMsN40Pl/PmZLzCxhXDIxhXjl9K0fpdAOfAajdjF1dkp+FTRprGHNI1oiGHTBsIRlJIWRPJHC06QiUlL7+Vh/VosLK5N0sgfdWirxWoL6nNB7XSFrIXis2QGx5KZkec9vvKXkwraAMpwuA2zfmETxXIPe9kMiTUo52FLFyZgbf5ByzhO/KnHDXbvBB9NEAqvTNwhTjAVNoCwlPcqbKSNX1GIpQ2VokFX3slm3zXuIYm9tONj4tRgztVxn5dC1YXT5W4ZyAVaen7LB4qZdXEIP1GiwjtTA0B8kwR/Wxkb5kz77X/sTKgU3CUON96C2Xvnoyx15GRDI9C6uT2byBFf+whB/mZtshUlwbfaGR7xa1tEghEUKtChkdOAkoZ/cuHEDC38a/JMBfN84+lDMTwwLIWXWESvlCkhjXLzY4B+p7n4Pk5wr+Gjc= ceph-c1bddff4-7013-11ef-bcea-c7b262605968 2024-09-11T08:03:17.822 DEBUG:teuthology.orchestra.run.smithi181:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYRAuDBpzOf8s8bDTdKUkjO350Fskbtdto09u0sN4UP08cs1dDJpmjqAvxbyOUdCcfQYgujeUyjMFMsN40Pl/PmZLzCxhXDIxhXjl9K0fpdAOfAajdjF1dkp+FTRprGHNI1oiGHTBsIRlJIWRPJHC06QiUlL7+Vh/VosLK5N0sgfdWirxWoL6nNB7XSFrIXis2QGx5KZkec9vvKXkwraAMpwuA2zfmETxXIPe9kMiTUo52FLFyZgbf5ByzhO/KnHDXbvBB9NEAqvTNwhTjAVNoCwlPcqbKSNX1GIpQ2VokFX3slm3zXuIYm9tONj4tRgztVxn5dC1YXT5W4ZyAVaen7LB4qZdXEIP1GiwjtTA0B8kwR/Wxkb5kz77X/sTKgU3CUON96C2Xvnoyx15GRDI9C6uT2byBFf+whB/mZtshUlwbfaGR7xa1tEghEUKtChkdOAkoZ/cuHEDC38a/JMBfN84+lDMTwwLIWXWESvlCkhjXLzY4B+p7n4Pk5wr+Gjc= ceph-c1bddff4-7013-11ef-bcea-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-11T08:03:17.961 INFO:teuthology.orchestra.run.smithi181.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYRAuDBpzOf8s8bDTdKUkjO350Fskbtdto09u0sN4UP08cs1dDJpmjqAvxbyOUdCcfQYgujeUyjMFMsN40Pl/PmZLzCxhXDIxhXjl9K0fpdAOfAajdjF1dkp+FTRprGHNI1oiGHTBsIRlJIWRPJHC06QiUlL7+Vh/VosLK5N0sgfdWirxWoL6nNB7XSFrIXis2QGx5KZkec9vvKXkwraAMpwuA2zfmETxXIPe9kMiTUo52FLFyZgbf5ByzhO/KnHDXbvBB9NEAqvTNwhTjAVNoCwlPcqbKSNX1GIpQ2VokFX3slm3zXuIYm9tONj4tRgztVxn5dC1YXT5W4ZyAVaen7LB4qZdXEIP1GiwjtTA0B8kwR/Wxkb5kz77X/sTKgU3CUON96C2Xvnoyx15GRDI9C6uT2byBFf+whB/mZtshUlwbfaGR7xa1tEghEUKtChkdOAkoZ/cuHEDC38a/JMBfN84+lDMTwwLIWXWESvlCkhjXLzY4B+p7n4Pk5wr+Gjc= ceph-c1bddff4-7013-11ef-bcea-c7b262605968 2024-09-11T08:03:17.974 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-11T08:03:18.094 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:17 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2021438967' entity='client.admin' 2024-09-11T08:03:18.223 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:03:21.139 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-11T08:03:21.139 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-11T08:03:21.370 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:21 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/577827070' entity='client.admin' 2024-09-11T08:03:21.370 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:21 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:21.370 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:21 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:21.370 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:21 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi150", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:03:21.370 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:21 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:21.371 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:21 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:03:21.371 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:21 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:21.618 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:03:24.762 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi181 2024-09-11T08:03:24.762 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-11T08:03:24.763 DEBUG:teuthology.orchestra.run.smithi181:> dd of=/etc/ceph/ceph.conf 2024-09-11T08:03:24.782 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-11T08:03:24.782 DEBUG:teuthology.orchestra.run.smithi181:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-11T08:03:24.840 INFO:tasks.cephadm:Adding host smithi181 to orchestrator... 2024-09-11T08:03:24.840 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph orch host add smithi181 2024-09-11T08:03:25.083 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:03:25.236 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:24 smithi150 ceph-mon[29276]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:03:25.237 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:24 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:25.237 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:24 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:03:25.237 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:24 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:03:25.237 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:24 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:03:25.237 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:24 smithi150 ceph-mon[29276]: Updating smithi150:/etc/ceph/ceph.conf 2024-09-11T08:03:25.237 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:24 smithi150 ceph-mon[29276]: Updating smithi150:/var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-11T08:03:25.237 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:24 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:25.237 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:24 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:25.237 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:24 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:26.199 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:25 smithi150 ceph-mon[29276]: Updating smithi150:/etc/ceph/ceph.client.admin.keyring 2024-09-11T08:03:26.199 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:25 smithi150 ceph-mon[29276]: Updating smithi150:/var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-11T08:03:27.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:27 smithi150 ceph-mon[29276]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi181", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:03:28.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:28 smithi150 ceph-mon[29276]: Deploying cephadm binary to smithi181 2024-09-11T08:03:29.659 INFO:teuthology.orchestra.run.smithi150.stdout:Added host 'smithi181' with addr '172.21.15.181' 2024-09-11T08:03:29.963 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:29 smithi150 ceph-mon[29276]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:03:29.963 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:29 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:29.963 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:29 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:03:30.247 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph orch host ls --format=json 2024-09-11T08:03:30.485 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:03:31.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:30 smithi150 ceph-mon[29276]: Added host smithi181 2024-09-11T08:03:31.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:30 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:31.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:30 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:32.175 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:31 smithi150 ceph-mon[29276]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:03:32.175 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:31 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:32.321 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:03:32.321 INFO:teuthology.orchestra.run.smithi150.stdout:[{"addr": "172.21.15.150", "hostname": "smithi150", "labels": [], "status": ""}, {"addr": "172.21.15.181", "hostname": "smithi181", "labels": [], "status": ""}] 2024-09-11T08:03:32.960 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-11T08:03:32.960 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd crush tunables default 2024-09-11T08:03:33.198 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:03:33.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:32 smithi150 ceph-mon[29276]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:03:34.371 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:33 smithi150 ceph-mon[29276]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:03:34.371 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:33 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:34.371 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:33 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:34.371 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:33 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:34.371 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:33 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:34.371 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:33 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi181", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:03:34.371 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:33 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:03:34.371 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:33 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:03:35.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:34 smithi150 ceph-mon[29276]: Updating smithi181:/etc/ceph/ceph.conf 2024-09-11T08:03:35.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:34 smithi150 ceph-mon[29276]: Updating smithi181:/var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-11T08:03:35.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:34 smithi150 ceph-mon[29276]: Updating smithi181:/etc/ceph/ceph.client.admin.keyring 2024-09-11T08:03:35.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:34 smithi150 ceph-mon[29276]: Updating smithi181:/var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-11T08:03:35.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:34 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:35.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:34 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:35.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:34 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:36.001 INFO:teuthology.orchestra.run.smithi150.stderr:adjusted tunables profile to default 2024-09-11T08:03:36.305 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:35 smithi150 ceph-mon[29276]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:03:36.305 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:35 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1589513420' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-11T08:03:36.618 INFO:tasks.cephadm:Adding mon.a on smithi150 2024-09-11T08:03:36.618 INFO:tasks.cephadm:Adding mon.c on smithi150 2024-09-11T08:03:36.618 INFO:tasks.cephadm:Adding mon.b on smithi181 2024-09-11T08:03:36.618 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph orch apply mon '3;smithi150:172.21.15.150=a;smithi150:[v2:172.21.15.150:3301,v1:172.21.15.150:6790]=c;smithi181:172.21.15.181=b' 2024-09-11T08:03:36.858 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-11T08:03:37.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:37 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1589513420' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-11T08:03:37.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:37 smithi150 ceph-mon[29276]: osdmap e4: 0 total, 0 up, 0 in 2024-09-11T08:03:38.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:38 smithi150 ceph-mon[29276]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:03:38.547 INFO:teuthology.orchestra.run.smithi181.stdout:Scheduled mon update... 2024-09-11T08:03:39.042 DEBUG:teuthology.orchestra.run.smithi150:mon.c> sudo journalctl -f -n 0 -u ceph-c1bddff4-7013-11ef-bcea-c7b262605968@mon.c.service 2024-09-11T08:03:39.045 DEBUG:teuthology.orchestra.run.smithi181:mon.b> sudo journalctl -f -n 0 -u ceph-c1bddff4-7013-11ef-bcea-c7b262605968@mon.b.service 2024-09-11T08:03:39.047 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-11T08:03:39.048 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph mon dump -f json 2024-09-11T08:03:39.773 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-11T08:03:39.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:39 smithi150 ceph-mon[29276]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi150:172.21.15.150=a;smithi150:[v2:172.21.15.150:3301,v1:172.21.15.150:6790]=c;smithi181:172.21.15.181=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:03:39.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:39 smithi150 ceph-mon[29276]: Saving service mon spec with placement smithi150:172.21.15.150=a;smithi150:[v2:172.21.15.150:3301,v1:172.21.15.150:6790]=c;smithi181:172.21.15.181=b;count:3 2024-09-11T08:03:39.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:39 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:39.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:39 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:03:39.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:39 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:03:39.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:39 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:03:39.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:39 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:39.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:39 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-11T08:03:39.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:39 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:03:39.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:39 smithi150 ceph-mon[29276]: Deploying daemon mon.b on smithi181 2024-09-11T08:03:39.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:39 smithi150 ceph-mon[29276]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:03:42.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:41 smithi150 ceph-mon[29276]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:03:42.933 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T08:03:42.933 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":1,"fsid":"c1bddff4-7013-11ef-bcea-c7b262605968","modified":"2024-09-11T08:02:12.380959Z","created":"2024-09-11T08:02:12.380959Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-11T08:03:42.934 INFO:teuthology.orchestra.run.smithi181.stderr:dumped monmap epoch 1 2024-09-11T08:03:43.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:43 smithi181 ceph-mon[32701]: mon.b@-1(???) e0 preinit fsid c1bddff4-7013-11ef-bcea-c7b262605968 2024-09-11T08:03:43.439 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:43 smithi181 ceph-mon[32701]: mon.b@-1(synchronizing).mds e1 new map 2024-09-11T08:03:43.439 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:43 smithi181 ceph-mon[32701]: mon.b@-1(synchronizing).mds e1 print_map 2024-09-11T08:03:43.439 INFO:journalctl@ceph.mon.b.smithi181.stdout: e1 2024-09-11T08:03:43.439 INFO:journalctl@ceph.mon.b.smithi181.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-11T08:03:43.440 INFO:journalctl@ceph.mon.b.smithi181.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-11T08:03:43.440 INFO:journalctl@ceph.mon.b.smithi181.stdout: legacy client fscid: -1 2024-09-11T08:03:43.440 INFO:journalctl@ceph.mon.b.smithi181.stdout: 2024-09-11T08:03:43.440 INFO:journalctl@ceph.mon.b.smithi181.stdout: No filesystems configured 2024-09-11T08:03:43.440 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:43 smithi181 ceph-mon[32701]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-11T08:03:43.440 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:43 smithi181 ceph-mon[32701]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-11T08:03:43.440 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:43 smithi181 ceph-mon[32701]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-11T08:03:43.440 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:43 smithi181 ceph-mon[32701]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-11T08:03:43.440 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:43 smithi181 ceph-mon[32701]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-11T08:03:43.440 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:43 smithi181 ceph-mon[32701]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-11T08:03:43.440 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:43 smithi181 ceph-mon[32701]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-11T08:03:43.440 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:43 smithi181 ceph-mon[32701]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-11T08:03:43.440 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:43 smithi181 ceph-mon[32701]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-11T08:03:43.441 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:43 smithi181 ceph-mon[32701]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-11T08:03:43.441 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:43 smithi181 ceph-mon[32701]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:03:43.441 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:43 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1589513420' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-11T08:03:43.441 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:43 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1589513420' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-11T08:03:43.441 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:43 smithi181 ceph-mon[32701]: osdmap e4: 0 total, 0 up, 0 in 2024-09-11T08:03:43.441 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:43 smithi181 ceph-mon[32701]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:03:43.441 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:43 smithi181 ceph-mon[32701]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi150:172.21.15.150=a;smithi150:[v2:172.21.15.150:3301,v1:172.21.15.150:6790]=c;smithi181:172.21.15.181=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:03:43.441 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:43 smithi181 ceph-mon[32701]: Saving service mon spec with placement smithi150:172.21.15.150=a;smithi150:[v2:172.21.15.150:3301,v1:172.21.15.150:6790]=c;smithi181:172.21.15.181=b;count:3 2024-09-11T08:03:43.441 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:43 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:43.441 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:43 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:03:43.441 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:43 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:03:43.441 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:43 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:03:43.441 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:43 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:43.441 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:43 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-11T08:03:43.442 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:43 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:03:43.442 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:43 smithi181 ceph-mon[32701]: Deploying daemon mon.b on smithi181 2024-09-11T08:03:43.442 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:43 smithi181 ceph-mon[32701]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:03:43.442 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:43 smithi181 ceph-mon[32701]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:03:43.442 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:43 smithi181 ceph-mon[32701]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-11T08:03:43.442 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:43 smithi181 bash[32648]: 0b11e8513d3800890de360ff760a72ab4a9683fd1b614e7a71c468ff60d510b1 2024-09-11T08:03:43.442 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:43 smithi181 systemd[1]: Started Ceph mon.b for c1bddff4-7013-11ef-bcea-c7b262605968. 2024-09-11T08:03:44.731 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-11T08:03:44.731 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph mon dump -f json 2024-09-11T08:03:44.978 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:03:48.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:48 smithi181 ceph-mon[32701]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:03:48.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:48 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-11T08:03:48.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:48 smithi181 ceph-mon[32701]: mon.a calling monitor election 2024-09-11T08:03:48.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:48 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:03:48.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:48 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:03:48.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:48 smithi181 ceph-mon[32701]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:03:48.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:48 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:03:48.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:48 smithi181 ceph-mon[32701]: mon.b calling monitor election 2024-09-11T08:03:48.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:48 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:03:48.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:48 smithi181 ceph-mon[32701]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:03:48.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:48 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:03:48.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:48 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:03:48.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:48 smithi181 ceph-mon[32701]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-11T08:03:48.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:48 smithi181 ceph-mon[32701]: monmap e2: 2 mons at {a=[v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0],b=[v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-11T08:03:48.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:48 smithi181 ceph-mon[32701]: fsmap 2024-09-11T08:03:48.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:48 smithi181 ceph-mon[32701]: osdmap e4: 0 total, 0 up, 0 in 2024-09-11T08:03:48.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:48 smithi181 ceph-mon[32701]: mgrmap e13: y(active, since 39s) 2024-09-11T08:03:48.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:48 smithi181 ceph-mon[32701]: overall HEALTH_OK 2024-09-11T08:03:48.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:48 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:48.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:48 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:48.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:48 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:48.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:48 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-11T08:03:48.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:48 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:03:48.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:48 smithi150 ceph-mon[29276]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:03:48.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:48 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-11T08:03:48.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:48 smithi150 ceph-mon[29276]: mon.a calling monitor election 2024-09-11T08:03:48.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:48 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:03:48.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:48 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:03:48.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:48 smithi150 ceph-mon[29276]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:03:48.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:48 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:03:48.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:48 smithi150 ceph-mon[29276]: mon.b calling monitor election 2024-09-11T08:03:48.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:48 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:03:48.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:48 smithi150 ceph-mon[29276]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:03:48.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:48 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:03:48.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:48 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:03:48.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:48 smithi150 ceph-mon[29276]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-11T08:03:48.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:48 smithi150 ceph-mon[29276]: monmap e2: 2 mons at {a=[v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0],b=[v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-11T08:03:48.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:48 smithi150 ceph-mon[29276]: fsmap 2024-09-11T08:03:48.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:48 smithi150 ceph-mon[29276]: osdmap e4: 0 total, 0 up, 0 in 2024-09-11T08:03:48.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:48 smithi150 ceph-mon[29276]: mgrmap e13: y(active, since 39s) 2024-09-11T08:03:48.491 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:48 smithi150 ceph-mon[29276]: overall HEALTH_OK 2024-09-11T08:03:48.491 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:48 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:48.491 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:48 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:48.491 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:48 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:48.491 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:48 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-11T08:03:48.491 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:48 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:03:49.485 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:49 smithi150 ceph-mon[29276]: Deploying daemon mon.c on smithi150 2024-09-11T08:03:49.486 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:49 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:03:49.530 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T08:03:49.530 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":2,"fsid":"c1bddff4-7013-11ef-bcea-c7b262605968","modified":"2024-09-11T08:03:43.082281Z","created":"2024-09-11T08:02:12.380959Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-11T08:03:49.530 INFO:teuthology.orchestra.run.smithi181.stderr:dumped monmap epoch 2 2024-09-11T08:03:49.530 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:49 smithi181 ceph-mon[32701]: Deploying daemon mon.c on smithi150 2024-09-11T08:03:49.530 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:49 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:03:50.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:50 smithi150 ceph-mon[29276]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:03:50.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:50 smithi150 ceph-mon[29276]: from='client.? 172.21.15.181:0/3932200566' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-11T08:03:50.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:50 smithi181 ceph-mon[32701]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:03:50.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:50 smithi181 ceph-mon[32701]: from='client.? 172.21.15.181:0/3932200566' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-11T08:03:51.051 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-11T08:03:51.052 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph mon dump -f json 2024-09-11T08:03:51.176 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:50 smithi150 systemd[1]: Starting Ceph mon.c for c1bddff4-7013-11ef-bcea-c7b262605968... 2024-09-11T08:03:51.300 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:03:51.437 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 podman[39592]: 2024-09-11 08:03:51.077494745 +0000 UTC m=+0.015406549 image pull 4a918172a49766385959227c53e75e2c302e923539324792cb5bee6cad0b2f2b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40 2024-09-11T08:03:51.437 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 podman[39592]: 2024-09-11 08:03:51.209481002 +0000 UTC m=+0.147392793 container create a2ca8ac978f75d735e866b17a98f9ac64aa00dbac8746a97bb511813766705c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mon-c, GIT_CLEAN=True, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-11T08:03:51.437 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 podman[39592]: 2024-09-11 08:03:51.436825276 +0000 UTC m=+0.374737071 container init a2ca8ac978f75d735e866b17a98f9ac64aa00dbac8746a97bb511813766705c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mon-c, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-11T08:03:51.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 podman[39592]: 2024-09-11 08:03:51.440476908 +0000 UTC m=+0.378388709 container start a2ca8ac978f75d735e866b17a98f9ac64aa00dbac8746a97bb511813766705c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mon-c, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-11T08:03:51.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: set uid:gid to 167:167 (ceph:ceph) 2024-09-11T08:03:51.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: ceph version 17.2.7-1623-gd0e6828a (d0e6828a2016d48cf25ad84064e50742bb1c39b9) quincy (stable), process ceph-mon, pid 2 2024-09-11T08:03:51.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: pidfile_write: ignore empty --pid-file 2024-09-11T08:03:51.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: load: jerasure load: lrc 2024-09-11T08:03:51.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: RocksDB version: 6.15.5 2024-09-11T08:03:51.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-09-11T08:03:51.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Compile date Sep 10 2024 2024-09-11T08:03:51.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: DB SUMMARY 2024-09-11T08:03:51.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: DB Session ID: WW6KTZ7RGXWVLDQ7JKQ2 2024-09-11T08:03:51.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: CURRENT file: CURRENT 2024-09-11T08:03:51.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: IDENTITY file: IDENTITY 2024-09-11T08:03:51.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-09-11T08:03:51.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-09-11T08:03:51.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 636 ; 2024-09-11T08:03:51.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.error_if_exists: 0 2024-09-11T08:03:51.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.create_if_missing: 0 2024-09-11T08:03:51.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.paranoid_checks: 1 2024-09-11T08:03:51.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-11T08:03:51.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.env: 0x560a6ab17440 2024-09-11T08:03:51.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.fs: Posix File System 2024-09-11T08:03:51.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.info_log: 0x560a6c23eec0 2024-09-11T08:03:51.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.max_file_opening_threads: 16 2024-09-11T08:03:51.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.statistics: (nil) 2024-09-11T08:03:51.742 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.use_fsync: 0 2024-09-11T08:03:51.742 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.max_log_file_size: 0 2024-09-11T08:03:51.742 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-11T08:03:51.742 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-11T08:03:51.742 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.keep_log_file_num: 1000 2024-09-11T08:03:51.742 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.recycle_log_file_num: 0 2024-09-11T08:03:51.742 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.allow_fallocate: 1 2024-09-11T08:03:51.742 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.allow_mmap_reads: 0 2024-09-11T08:03:51.742 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.allow_mmap_writes: 0 2024-09-11T08:03:51.743 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.use_direct_reads: 0 2024-09-11T08:03:51.743 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-11T08:03:51.743 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.create_missing_column_families: 0 2024-09-11T08:03:51.743 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.db_log_dir: 2024-09-11T08:03:51.743 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2024-09-11T08:03:51.743 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-11T08:03:51.743 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-11T08:03:51.743 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-11T08:03:51.743 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-11T08:03:51.744 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-11T08:03:51.744 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-11T08:03:51.744 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.advise_random_on_open: 1 2024-09-11T08:03:51.744 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.db_write_buffer_size: 0 2024-09-11T08:03:51.744 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.write_buffer_manager: 0x560a6c2fc7b0 2024-09-11T08:03:51.744 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-11T08:03:51.744 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-09-11T08:03:51.744 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-11T08:03:51.744 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-11T08:03:51.744 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.rate_limiter: (nil) 2024-09-11T08:03:51.744 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-11T08:03:51.744 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.wal_recovery_mode: 2 2024-09-11T08:03:51.745 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.enable_thread_tracking: 0 2024-09-11T08:03:51.745 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.enable_pipelined_write: 0 2024-09-11T08:03:51.745 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.unordered_write: 0 2024-09-11T08:03:51.745 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-11T08:03:51.745 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-11T08:03:51.745 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-11T08:03:51.745 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-11T08:03:51.745 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.row_cache: None 2024-09-11T08:03:51.745 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.wal_filter: None 2024-09-11T08:03:51.745 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-11T08:03:51.745 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.allow_ingest_behind: 0 2024-09-11T08:03:51.745 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.preserve_deletes: 0 2024-09-11T08:03:51.745 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.two_write_queues: 0 2024-09-11T08:03:51.746 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.manual_wal_flush: 0 2024-09-11T08:03:51.746 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.atomic_flush: 0 2024-09-11T08:03:51.746 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-11T08:03:51.746 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-11T08:03:51.746 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-11T08:03:51.746 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.log_readahead_size: 0 2024-09-11T08:03:51.746 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-11T08:03:51.746 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.best_efforts_recovery: 0 2024-09-11T08:03:51.746 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-11T08:03:51.746 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-11T08:03:51.746 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.allow_data_in_errors: 0 2024-09-11T08:03:51.746 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.db_host_id: __hostname__ 2024-09-11T08:03:51.747 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.max_background_jobs: 2 2024-09-11T08:03:51.747 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.max_background_compactions: -1 2024-09-11T08:03:51.747 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.max_subcompactions: 1 2024-09-11T08:03:51.747 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-11T08:03:51.747 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-11T08:03:51.747 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-11T08:03:51.747 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.max_total_wal_size: 0 2024-09-11T08:03:51.747 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-11T08:03:51.747 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-11T08:03:51.747 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-11T08:03:51.747 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-11T08:03:51.747 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.max_open_files: -1 2024-09-11T08:03:51.747 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.bytes_per_sync: 0 2024-09-11T08:03:51.747 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-11T08:03:51.747 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-11T08:03:51.747 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.compaction_readahead_size: 0 2024-09-11T08:03:51.748 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.max_background_flushes: -1 2024-09-11T08:03:51.748 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Compression algorithms supported: 2024-09-11T08:03:51.748 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: kZSTD supported: 0 2024-09-11T08:03:51.748 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: kXpressCompression supported: 0 2024-09-11T08:03:51.748 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: kBZip2Compression supported: 0 2024-09-11T08:03:51.748 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-11T08:03:51.748 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: kLZ4Compression supported: 1 2024-09-11T08:03:51.748 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: kZlibCompression supported: 1 2024-09-11T08:03:51.748 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: kLZ4HCCompression supported: 1 2024-09-11T08:03:51.748 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: kSnappyCompression supported: 1 2024-09-11T08:03:51.748 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-11T08:03:51.748 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 2024-09-11T08:03:51.748 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-09-11T08:03:51.748 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-11T08:03:51.748 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.merge_operator: 2024-09-11T08:03:51.749 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.compaction_filter: None 2024-09-11T08:03:51.749 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.compaction_filter_factory: None 2024-09-11T08:03:51.749 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.sst_partitioner_factory: None 2024-09-11T08:03:51.749 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-11T08:03:51.749 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-11T08:03:51.749 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560a6c1e5a38) 2024-09-11T08:03:51.749 INFO:journalctl@ceph.mon.c.smithi150.stdout: cache_index_and_filter_blocks: 1 2024-09-11T08:03:51.749 INFO:journalctl@ceph.mon.c.smithi150.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-11T08:03:51.749 INFO:journalctl@ceph.mon.c.smithi150.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-11T08:03:51.749 INFO:journalctl@ceph.mon.c.smithi150.stdout: pin_top_level_index_and_filter: 1 2024-09-11T08:03:51.749 INFO:journalctl@ceph.mon.c.smithi150.stdout: index_type: 0 2024-09-11T08:03:51.749 INFO:journalctl@ceph.mon.c.smithi150.stdout: data_block_index_type: 0 2024-09-11T08:03:51.749 INFO:journalctl@ceph.mon.c.smithi150.stdout: index_shortening: 1 2024-09-11T08:03:51.749 INFO:journalctl@ceph.mon.c.smithi150.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-11T08:03:51.749 INFO:journalctl@ceph.mon.c.smithi150.stdout: hash_index_allow_collision: 1 2024-09-11T08:03:51.750 INFO:journalctl@ceph.mon.c.smithi150.stdout: checksum: 1 2024-09-11T08:03:51.750 INFO:journalctl@ceph.mon.c.smithi150.stdout: no_block_cache: 0 2024-09-11T08:03:51.750 INFO:journalctl@ceph.mon.c.smithi150.stdout: block_cache: 0x560a6c21b1f0 2024-09-11T08:03:51.750 INFO:journalctl@ceph.mon.c.smithi150.stdout: block_cache_name: BinnedLRUCache 2024-09-11T08:03:51.750 INFO:journalctl@ceph.mon.c.smithi150.stdout: block_cache_options: 2024-09-11T08:03:51.750 INFO:journalctl@ceph.mon.c.smithi150.stdout: capacity : 536870912 2024-09-11T08:03:51.750 INFO:journalctl@ceph.mon.c.smithi150.stdout: num_shard_bits : 4 2024-09-11T08:03:51.750 INFO:journalctl@ceph.mon.c.smithi150.stdout: strict_capacity_limit : 0 2024-09-11T08:03:51.750 INFO:journalctl@ceph.mon.c.smithi150.stdout: high_pri_pool_ratio: 0.000 2024-09-11T08:03:51.750 INFO:journalctl@ceph.mon.c.smithi150.stdout: block_cache_compressed: (nil) 2024-09-11T08:03:51.750 INFO:journalctl@ceph.mon.c.smithi150.stdout: persistent_cache: (nil) 2024-09-11T08:03:51.750 INFO:journalctl@ceph.mon.c.smithi150.stdout: block_size: 4096 2024-09-11T08:03:51.750 INFO:journalctl@ceph.mon.c.smithi150.stdout: block_size_deviation: 10 2024-09-11T08:03:51.750 INFO:journalctl@ceph.mon.c.smithi150.stdout: block_restart_interval: 16 2024-09-11T08:03:51.750 INFO:journalctl@ceph.mon.c.smithi150.stdout: index_block_restart_interval: 1 2024-09-11T08:03:51.750 INFO:journalctl@ceph.mon.c.smithi150.stdout: metadata_block_size: 4096 2024-09-11T08:03:51.751 INFO:journalctl@ceph.mon.c.smithi150.stdout: partition_filters: 0 2024-09-11T08:03:51.751 INFO:journalctl@ceph.mon.c.smithi150.stdout: use_delta_encoding: 1 2024-09-11T08:03:51.751 INFO:journalctl@ceph.mon.c.smithi150.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-09-11T08:03:51.751 INFO:journalctl@ceph.mon.c.smithi150.stdout: whole_key_filtering: 1 2024-09-11T08:03:51.751 INFO:journalctl@ceph.mon.c.smithi150.stdout: verify_compression: 0 2024-09-11T08:03:51.751 INFO:journalctl@ceph.mon.c.smithi150.stdout: read_amp_bytes_per_bit: 0 2024-09-11T08:03:51.751 INFO:journalctl@ceph.mon.c.smithi150.stdout: format_version: 4 2024-09-11T08:03:51.751 INFO:journalctl@ceph.mon.c.smithi150.stdout: enable_index_compression: 1 2024-09-11T08:03:51.751 INFO:journalctl@ceph.mon.c.smithi150.stdout: block_align: 0 2024-09-11T08:03:51.751 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.write_buffer_size: 33554432 2024-09-11T08:03:51.751 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.max_write_buffer_number: 2 2024-09-11T08:03:51.751 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.compression: NoCompression 2024-09-11T08:03:51.751 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.bottommost_compression: Disabled 2024-09-11T08:03:51.751 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.prefix_extractor: nullptr 2024-09-11T08:03:51.751 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-11T08:03:51.751 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.num_levels: 7 2024-09-11T08:03:51.752 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-11T08:03:51.752 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-11T08:03:51.752 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-11T08:03:51.752 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-11T08:03:51.752 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-11T08:03:51.752 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-11T08:03:51.752 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-11T08:03:51.752 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-11T08:03:51.752 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-11T08:03:51.752 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-11T08:03:51.752 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-11T08:03:51.752 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.compression_opts.level: 32767 2024-09-11T08:03:51.752 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.compression_opts.strategy: 0 2024-09-11T08:03:51.752 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-11T08:03:51.752 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-11T08:03:51.753 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-11T08:03:51.753 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.compression_opts.enabled: false 2024-09-11T08:03:51.753 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-11T08:03:51.753 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-11T08:03:51.753 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-11T08:03:51.753 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.target_file_size_base: 67108864 2024-09-11T08:03:51.753 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-11T08:03:51.753 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-11T08:03:51.753 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-11T08:03:51.753 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-11T08:03:51.753 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-11T08:03:51.753 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-11T08:03:51.753 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-11T08:03:51.754 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-11T08:03:51.754 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-11T08:03:51.754 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-11T08:03:51.754 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-11T08:03:51.754 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-11T08:03:51.754 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-11T08:03:51.754 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.arena_block_size: 4194304 2024-09-11T08:03:51.754 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-11T08:03:51.754 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-11T08:03:51.754 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-09-11T08:03:51.754 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.disable_auto_compactions: 0 2024-09-11T08:03:51.754 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-11T08:03:51.754 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-11T08:03:51.754 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-11T08:03:51.754 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-11T08:03:51.755 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-11T08:03:51.755 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-11T08:03:51.755 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-11T08:03:51.755 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-11T08:03:51.756 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-11T08:03:51.756 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-11T08:03:51.756 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-11T08:03:51.756 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.inplace_update_support: 0 2024-09-11T08:03:51.756 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-11T08:03:51.756 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-11T08:03:51.756 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-11T08:03:51.756 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-11T08:03:51.756 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.bloom_locality: 0 2024-09-11T08:03:51.756 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.max_successive_merges: 0 2024-09-11T08:03:51.756 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-11T08:03:51.756 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.paranoid_file_checks: 0 2024-09-11T08:03:51.756 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.force_consistency_checks: 1 2024-09-11T08:03:51.757 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.report_bg_io_stats: 0 2024-09-11T08:03:51.757 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.ttl: 2592000 2024-09-11T08:03:51.757 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-11T08:03:51.757 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.enable_blob_files: false 2024-09-11T08:03:51.757 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.min_blob_size: 0 2024-09-11T08:03:51.757 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.blob_file_size: 268435456 2024-09-11T08:03:51.757 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-11T08:03:51.757 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-11T08:03:51.757 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-11T08:03:51.757 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-11T08:03:51.758 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 0 2024-09-11T08:03:51.758 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: [db/version_set.cc:4082] Creating manifest 7 2024-09-11T08:03:51.758 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726041831465230, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-11T08:03:51.758 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #4 mode 2 2024-09-11T08:03:51.758 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-09-11T08:03:51.758 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726041831465530, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1696, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1726041831, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "6174f732-b232-4387-a31c-66e2bad61ba7", "db_session_id": "WW6KTZ7RGXWVLDQ7JKQ2"}} 2024-09-11T08:03:51.758 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: [db/version_set.cc:4082] Creating manifest 9 2024-09-11T08:03:51.758 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726041831465853, "job": 1, "event": "recovery_finished"} 2024-09-11T08:03:51.758 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-11T08:03:51.758 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x560a6c24e700 2024-09-11T08:03:51.758 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: DB pointer 0x560a6c21e000 2024-09-11T08:03:51.758 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-09-11T08:03:51.758 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-09-11T08:03:51.759 INFO:journalctl@ceph.mon.c.smithi150.stdout: ** DB Stats ** 2024-09-11T08:03:51.759 INFO:journalctl@ceph.mon.c.smithi150.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-11T08:03:51.759 INFO:journalctl@ceph.mon.c.smithi150.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-11T08:03:51.759 INFO:journalctl@ceph.mon.c.smithi150.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-11T08:03:51.759 INFO:journalctl@ceph.mon.c.smithi150.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-11T08:03:51.759 INFO:journalctl@ceph.mon.c.smithi150.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-11T08:03:51.759 INFO:journalctl@ceph.mon.c.smithi150.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-09-11T08:03:51.759 INFO:journalctl@ceph.mon.c.smithi150.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-11T08:03:51.759 INFO:journalctl@ceph.mon.c.smithi150.stdout: 2024-09-11T08:03:51.759 INFO:journalctl@ceph.mon.c.smithi150.stdout: ** Compaction Stats [default] ** 2024-09-11T08:03:51.759 INFO:journalctl@ceph.mon.c.smithi150.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-11T08:03:51.759 INFO:journalctl@ceph.mon.c.smithi150.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-11T08:03:51.759 INFO:journalctl@ceph.mon.c.smithi150.stdout: L0 1/0 1.66 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.7 0.00 0.00 1 0.000 0 0 2024-09-11T08:03:51.759 INFO:journalctl@ceph.mon.c.smithi150.stdout: Sum 1/0 1.66 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.7 0.00 0.00 1 0.000 0 0 2024-09-11T08:03:51.759 INFO:journalctl@ceph.mon.c.smithi150.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.7 0.00 0.00 1 0.000 0 0 2024-09-11T08:03:51.759 INFO:journalctl@ceph.mon.c.smithi150.stdout: 2024-09-11T08:03:51.760 INFO:journalctl@ceph.mon.c.smithi150.stdout: ** Compaction Stats [default] ** 2024-09-11T08:03:51.760 INFO:journalctl@ceph.mon.c.smithi150.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-11T08:03:51.760 INFO:journalctl@ceph.mon.c.smithi150.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-11T08:03:51.760 INFO:journalctl@ceph.mon.c.smithi150.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.7 0.00 0.00 1 0.000 0 0 2024-09-11T08:03:51.760 INFO:journalctl@ceph.mon.c.smithi150.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-11T08:03:51.760 INFO:journalctl@ceph.mon.c.smithi150.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-11T08:03:51.760 INFO:journalctl@ceph.mon.c.smithi150.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-11T08:03:51.760 INFO:journalctl@ceph.mon.c.smithi150.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-11T08:03:51.760 INFO:journalctl@ceph.mon.c.smithi150.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-11T08:03:51.760 INFO:journalctl@ceph.mon.c.smithi150.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-11T08:03:51.760 INFO:journalctl@ceph.mon.c.smithi150.stdout: Cumulative compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-11T08:03:51.760 INFO:journalctl@ceph.mon.c.smithi150.stdout: Interval compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-11T08:03:51.760 INFO:journalctl@ceph.mon.c.smithi150.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-11T08:03:51.760 INFO:journalctl@ceph.mon.c.smithi150.stdout: 2024-09-11T08:03:51.761 INFO:journalctl@ceph.mon.c.smithi150.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-11T08:03:51.761 INFO:journalctl@ceph.mon.c.smithi150.stdout: 2024-09-11T08:03:51.761 INFO:journalctl@ceph.mon.c.smithi150.stdout: ** Compaction Stats [default] ** 2024-09-11T08:03:51.761 INFO:journalctl@ceph.mon.c.smithi150.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-11T08:03:51.761 INFO:journalctl@ceph.mon.c.smithi150.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-11T08:03:51.761 INFO:journalctl@ceph.mon.c.smithi150.stdout: L0 1/0 1.66 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.7 0.00 0.00 1 0.000 0 0 2024-09-11T08:03:51.761 INFO:journalctl@ceph.mon.c.smithi150.stdout: Sum 1/0 1.66 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.7 0.00 0.00 1 0.000 0 0 2024-09-11T08:03:51.761 INFO:journalctl@ceph.mon.c.smithi150.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-09-11T08:03:51.761 INFO:journalctl@ceph.mon.c.smithi150.stdout: 2024-09-11T08:03:51.761 INFO:journalctl@ceph.mon.c.smithi150.stdout: ** Compaction Stats [default] ** 2024-09-11T08:03:51.761 INFO:journalctl@ceph.mon.c.smithi150.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-11T08:03:51.761 INFO:journalctl@ceph.mon.c.smithi150.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-11T08:03:51.761 INFO:journalctl@ceph.mon.c.smithi150.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.7 0.00 0.00 1 0.000 0 0 2024-09-11T08:03:51.761 INFO:journalctl@ceph.mon.c.smithi150.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-11T08:03:51.761 INFO:journalctl@ceph.mon.c.smithi150.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-11T08:03:51.761 INFO:journalctl@ceph.mon.c.smithi150.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-11T08:03:51.762 INFO:journalctl@ceph.mon.c.smithi150.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-11T08:03:51.762 INFO:journalctl@ceph.mon.c.smithi150.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-11T08:03:51.762 INFO:journalctl@ceph.mon.c.smithi150.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-11T08:03:51.762 INFO:journalctl@ceph.mon.c.smithi150.stdout: Cumulative compaction: 0.00 GB write, 0.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-11T08:03:51.762 INFO:journalctl@ceph.mon.c.smithi150.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-11T08:03:51.762 INFO:journalctl@ceph.mon.c.smithi150.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-11T08:03:51.762 INFO:journalctl@ceph.mon.c.smithi150.stdout: 2024-09-11T08:03:51.762 INFO:journalctl@ceph.mon.c.smithi150.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-11T08:03:51.762 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-09-11T08:03:51.762 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: using public_addrv [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] 2024-09-11T08:03:51.762 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: starting mon.c rank -1 at public addrs [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] at bind addrs [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid c1bddff4-7013-11ef-bcea-c7b262605968 2024-09-11T08:03:51.762 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: mon.c@-1(???) e0 preinit fsid c1bddff4-7013-11ef-bcea-c7b262605968 2024-09-11T08:03:51.762 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: mon.c@-1(synchronizing).mds e1 new map 2024-09-11T08:03:51.762 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: mon.c@-1(synchronizing).mds e1 print_map 2024-09-11T08:03:51.762 INFO:journalctl@ceph.mon.c.smithi150.stdout: e1 2024-09-11T08:03:51.763 INFO:journalctl@ceph.mon.c.smithi150.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-11T08:03:51.763 INFO:journalctl@ceph.mon.c.smithi150.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-11T08:03:51.763 INFO:journalctl@ceph.mon.c.smithi150.stdout: legacy client fscid: -1 2024-09-11T08:03:51.763 INFO:journalctl@ceph.mon.c.smithi150.stdout: 2024-09-11T08:03:51.763 INFO:journalctl@ceph.mon.c.smithi150.stdout: No filesystems configured 2024-09-11T08:03:51.763 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-11T08:03:51.763 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-11T08:03:51.763 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-11T08:03:51.763 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-11T08:03:51.763 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-11T08:03:51.763 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-11T08:03:51.763 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-11T08:03:51.763 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-11T08:03:51.763 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-11T08:03:51.763 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-11T08:03:51.763 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:03:51.763 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1589513420' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-11T08:03:51.764 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1589513420' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-11T08:03:51.764 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: osdmap e4: 0 total, 0 up, 0 in 2024-09-11T08:03:51.764 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:03:51.764 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi150:172.21.15.150=a;smithi150:[v2:172.21.15.150:3301,v1:172.21.15.150:6790]=c;smithi181:172.21.15.181=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:03:51.764 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: Saving service mon spec with placement smithi150:172.21.15.150=a;smithi150:[v2:172.21.15.150:3301,v1:172.21.15.150:6790]=c;smithi181:172.21.15.181=b;count:3 2024-09-11T08:03:51.764 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:51.764 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:03:51.764 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:03:51.764 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:03:51.764 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:51.764 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-11T08:03:51.764 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:03:51.764 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: Deploying daemon mon.b on smithi181 2024-09-11T08:03:51.764 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:03:51.765 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:03:51.765 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:03:51.765 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-11T08:03:51.765 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: mon.a calling monitor election 2024-09-11T08:03:51.765 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:03:51.765 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:03:51.765 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:03:51.765 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:03:51.765 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: mon.b calling monitor election 2024-09-11T08:03:51.765 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:03:51.765 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:03:51.765 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:03:51.765 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:03:51.765 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-11T08:03:51.765 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: monmap e2: 2 mons at {a=[v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0],b=[v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-11T08:03:51.765 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: fsmap 2024-09-11T08:03:51.766 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: osdmap e4: 0 total, 0 up, 0 in 2024-09-11T08:03:51.766 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: mgrmap e13: y(active, since 39s) 2024-09-11T08:03:51.766 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: overall HEALTH_OK 2024-09-11T08:03:51.766 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:51.766 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:51.766 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:51.766 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-11T08:03:51.766 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:03:51.766 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: Deploying daemon mon.c on smithi150 2024-09-11T08:03:51.766 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:03:51.766 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:03:51.766 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: from='client.? 172.21.15.181:0/3932200566' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-11T08:03:51.766 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 ceph-mon[39612]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-11T08:03:51.766 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 bash[39592]: a2ca8ac978f75d735e866b17a98f9ac64aa00dbac8746a97bb511813766705c9 2024-09-11T08:03:51.766 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:51 smithi150 systemd[1]: Started Ceph mon.c for c1bddff4-7013-11ef-bcea-c7b262605968. 2024-09-11T08:03:56.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:56 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-11T08:03:56.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:56 smithi181 ceph-mon[32701]: mon.a calling monitor election 2024-09-11T08:03:56.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:56 smithi181 ceph-mon[32701]: mon.b calling monitor election 2024-09-11T08:03:56.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:56 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:03:56.825 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:56 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:03:56.825 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:56 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:03:56.825 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:56 smithi181 ceph-mon[32701]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:03:56.825 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:56 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:03:56.825 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:56 smithi181 ceph-mon[32701]: mon.c calling monitor election 2024-09-11T08:03:56.825 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:56 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:03:56.825 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:56 smithi181 ceph-mon[32701]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:03:56.825 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:56 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:03:56.825 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:56 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:03:56.825 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:56 smithi181 ceph-mon[32701]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-11T08:03:56.825 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:56 smithi181 ceph-mon[32701]: monmap e3: 3 mons at {a=[v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0],b=[v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0],c=[v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-11T08:03:56.825 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:56 smithi181 ceph-mon[32701]: fsmap 2024-09-11T08:03:56.825 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:56 smithi181 ceph-mon[32701]: osdmap e4: 0 total, 0 up, 0 in 2024-09-11T08:03:56.825 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:56 smithi181 ceph-mon[32701]: mgrmap e13: y(active, since 47s) 2024-09-11T08:03:56.825 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:56 smithi181 ceph-mon[32701]: overall HEALTH_OK 2024-09-11T08:03:56.826 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:56 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:56.826 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:56 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:56.826 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:56 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:56.826 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:56 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:56.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:56 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-11T08:03:56.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:56 smithi150 ceph-mon[39612]: mon.a calling monitor election 2024-09-11T08:03:56.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:56 smithi150 ceph-mon[39612]: mon.b calling monitor election 2024-09-11T08:03:56.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:56 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:03:56.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:56 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:03:56.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:56 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:03:56.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:56 smithi150 ceph-mon[39612]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:03:56.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:56 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:03:56.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:56 smithi150 ceph-mon[39612]: mon.c calling monitor election 2024-09-11T08:03:56.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:56 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:03:56.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:56 smithi150 ceph-mon[39612]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:03:56.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:56 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:03:56.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:56 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:03:56.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:56 smithi150 ceph-mon[39612]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-11T08:03:56.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:56 smithi150 ceph-mon[39612]: monmap e3: 3 mons at {a=[v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0],b=[v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0],c=[v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-11T08:03:56.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:56 smithi150 ceph-mon[39612]: fsmap 2024-09-11T08:03:56.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:56 smithi150 ceph-mon[39612]: osdmap e4: 0 total, 0 up, 0 in 2024-09-11T08:03:56.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:56 smithi150 ceph-mon[39612]: mgrmap e13: y(active, since 47s) 2024-09-11T08:03:56.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:56 smithi150 ceph-mon[39612]: overall HEALTH_OK 2024-09-11T08:03:56.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:56 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:56.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:56 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:56.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:56 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:56.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:56 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:56.991 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:56 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-11T08:03:56.991 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:56 smithi150 ceph-mon[29276]: mon.a calling monitor election 2024-09-11T08:03:56.991 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:56 smithi150 ceph-mon[29276]: mon.b calling monitor election 2024-09-11T08:03:56.991 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:56 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:03:56.991 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:56 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:03:56.991 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:56 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:03:56.991 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:56 smithi150 ceph-mon[29276]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:03:56.991 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:56 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:03:56.991 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:56 smithi150 ceph-mon[29276]: mon.c calling monitor election 2024-09-11T08:03:56.991 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:56 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:03:56.991 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:56 smithi150 ceph-mon[29276]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:03:56.991 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:56 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:03:56.992 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:56 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:03:56.992 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:56 smithi150 ceph-mon[29276]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-11T08:03:56.992 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:56 smithi150 ceph-mon[29276]: monmap e3: 3 mons at {a=[v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0],b=[v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0],c=[v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-11T08:03:56.992 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:56 smithi150 ceph-mon[29276]: fsmap 2024-09-11T08:03:56.992 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:56 smithi150 ceph-mon[29276]: osdmap e4: 0 total, 0 up, 0 in 2024-09-11T08:03:56.992 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:56 smithi150 ceph-mon[29276]: mgrmap e13: y(active, since 47s) 2024-09-11T08:03:56.992 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:56 smithi150 ceph-mon[29276]: overall HEALTH_OK 2024-09-11T08:03:56.992 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:56 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:56.992 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:56 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:56.992 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:56 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:56.992 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:56 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:57.338 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T08:03:57.338 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":3,"fsid":"c1bddff4-7013-11ef-bcea-c7b262605968","modified":"2024-09-11T08:03:51.487980Z","created":"2024-09-11T08:02:12.380959Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3301","nonce":0},{"type":"v1","addr":"172.21.15.150:6790","nonce":0}]},"addr":"172.21.15.150:6790/0","public_addr":"172.21.15.150:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-09-11T08:03:57.338 INFO:teuthology.orchestra.run.smithi181.stderr:dumped monmap epoch 3 2024-09-11T08:03:57.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:57 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:03:57.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:57 smithi150 ceph-mon[29276]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:03:57.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:57 smithi150 ceph-mon[29276]: from='client.? 172.21.15.181:0/3933051494' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-11T08:03:57.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:57 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:03:57.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:57 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:03:57.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:57 smithi150 ceph-mon[39612]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:03:57.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:57 smithi150 ceph-mon[39612]: from='client.? 172.21.15.181:0/3933051494' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-11T08:03:57.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:57 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:03:57.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:57 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:03:57.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:57 smithi181 ceph-mon[32701]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:03:57.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:57 smithi181 ceph-mon[32701]: from='client.? 172.21.15.181:0/3933051494' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-11T08:03:57.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:57 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:03:58.332 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-11T08:03:58.332 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph config generate-minimal-conf 2024-09-11T08:03:58.928 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:58 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:58.929 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:58 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:58.929 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:58 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:58.929 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:58 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:58.929 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:58 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:03:58.929 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:58 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:03:58.929 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:58 smithi150 ceph-mon[39612]: Updating smithi150:/etc/ceph/ceph.conf 2024-09-11T08:03:58.929 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:58 smithi150 ceph-mon[39612]: Updating smithi181:/etc/ceph/ceph.conf 2024-09-11T08:03:58.929 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:58 smithi150 ceph-mon[39612]: Updating smithi150:/var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-11T08:03:58.929 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:58 smithi150 ceph-mon[39612]: Updating smithi181:/var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-11T08:03:58.929 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:58 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:58.929 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:58 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:58.929 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:58 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:58.929 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:58 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:58.930 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:58 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:58.930 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:58 smithi150 ceph-mon[39612]: Reconfiguring mon.a (unknown last config time)... 2024-09-11T08:03:58.930 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:58 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-11T08:03:58.930 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:58 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-11T08:03:58.930 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:58 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:03:58.930 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:58 smithi150 ceph-mon[39612]: Reconfiguring daemon mon.a on smithi150 2024-09-11T08:03:58.930 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:58 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:58.930 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:58 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:58.930 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:58 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:58.930 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:58 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:58.930 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:58 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:03:58.931 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:58 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:03:58.931 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:58 smithi150 ceph-mon[29276]: Updating smithi150:/etc/ceph/ceph.conf 2024-09-11T08:03:58.931 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:58 smithi150 ceph-mon[29276]: Updating smithi181:/etc/ceph/ceph.conf 2024-09-11T08:03:58.931 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:58 smithi150 ceph-mon[29276]: Updating smithi150:/var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-11T08:03:58.931 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:58 smithi150 ceph-mon[29276]: Updating smithi181:/var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-11T08:03:58.931 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:58 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:58.931 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:58 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:58.931 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:58 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:58.931 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:58 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:58.931 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:58 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:58.931 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:58 smithi150 ceph-mon[29276]: Reconfiguring mon.a (unknown last config time)... 2024-09-11T08:03:58.931 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:58 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-11T08:03:58.931 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:58 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-11T08:03:58.931 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:58 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:03:58.932 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:58 smithi150 ceph-mon[29276]: Reconfiguring daemon mon.a on smithi150 2024-09-11T08:03:59.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:58 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:59.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:58 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:59.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:58 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:59.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:58 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:59.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:58 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:03:59.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:58 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:03:59.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:58 smithi181 ceph-mon[32701]: Updating smithi150:/etc/ceph/ceph.conf 2024-09-11T08:03:59.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:58 smithi181 ceph-mon[32701]: Updating smithi181:/etc/ceph/ceph.conf 2024-09-11T08:03:59.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:58 smithi181 ceph-mon[32701]: Updating smithi150:/var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-11T08:03:59.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:58 smithi181 ceph-mon[32701]: Updating smithi181:/var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-11T08:03:59.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:58 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:59.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:58 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:59.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:58 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:59.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:58 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:59.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:58 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:59.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:58 smithi181 ceph-mon[32701]: Reconfiguring mon.a (unknown last config time)... 2024-09-11T08:03:59.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:58 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-11T08:03:59.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:58 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-11T08:03:59.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:58 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:03:59.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:58 smithi181 ceph-mon[32701]: Reconfiguring daemon mon.a on smithi150 2024-09-11T08:03:59.184 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:03:59.901 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:59 smithi150 ceph-mon[29276]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:03:59.901 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:59 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:59.901 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:59 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:59.901 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:59 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:59.901 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:59 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-11T08:03:59.901 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:59 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-11T08:03:59.902 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:03:59 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:03:59.902 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:59 smithi150 ceph-mon[39612]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:03:59.902 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:59 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:59.902 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:59 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:59.902 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:59 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:03:59.902 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:59 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-11T08:03:59.902 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:59 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-11T08:03:59.902 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:03:59 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:04:00.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:59 smithi181 ceph-mon[32701]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:04:00.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:59 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:00.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:59 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:00.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:59 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:00.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:59 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-11T08:04:00.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:59 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-11T08:04:00.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:03:59 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:04:01.416 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:01 smithi150 ceph-mon[39612]: Reconfiguring mon.c (monmap changed)... 2024-09-11T08:04:01.416 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:01 smithi150 ceph-mon[39612]: Reconfiguring daemon mon.c on smithi150 2024-09-11T08:04:01.416 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:01 smithi150 ceph-mon[29276]: Reconfiguring mon.c (monmap changed)... 2024-09-11T08:04:01.416 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:01 smithi150 ceph-mon[29276]: Reconfiguring daemon mon.c on smithi150 2024-09-11T08:04:01.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:01 smithi181 ceph-mon[32701]: Reconfiguring mon.c (monmap changed)... 2024-09-11T08:04:01.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:01 smithi181 ceph-mon[32701]: Reconfiguring daemon mon.c on smithi150 2024-09-11T08:04:02.338 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:02 smithi150 ceph-mon[29276]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:04:02.338 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:02 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:02.338 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:02 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:02.338 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:02 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-11T08:04:02.338 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:02 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-11T08:04:02.338 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:02 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:04:02.338 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:02 smithi150 ceph-mon[39612]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:04:02.339 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:02 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:02.339 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:02 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:02.339 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:02 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-11T08:04:02.339 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:02 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-11T08:04:02.339 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:02 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:04:02.496 INFO:teuthology.orchestra.run.smithi150.stdout:# minimal ceph.conf for c1bddff4-7013-11ef-bcea-c7b262605968 2024-09-11T08:04:02.496 INFO:teuthology.orchestra.run.smithi150.stdout:[global] 2024-09-11T08:04:02.496 INFO:teuthology.orchestra.run.smithi150.stdout: fsid = c1bddff4-7013-11ef-bcea-c7b262605968 2024-09-11T08:04:02.496 INFO:teuthology.orchestra.run.smithi150.stdout: mon_host = [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] 2024-09-11T08:04:02.503 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:02 smithi181 ceph-mon[32701]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:04:02.503 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:02 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:02.503 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:02 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:02.503 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:02 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-11T08:04:02.503 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:02 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-11T08:04:02.503 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:02 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:04:03.216 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-11T08:04:03.216 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-11T08:04:03.216 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/ceph/ceph.conf 2024-09-11T08:04:03.244 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-11T08:04:03.244 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-11T08:04:03.309 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-11T08:04:03.310 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/ceph/ceph.conf 2024-09-11T08:04:03.340 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-11T08:04:03.340 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-11T08:04:03.406 INFO:tasks.cephadm:Adding mgr.y on smithi150 2024-09-11T08:04:03.406 INFO:tasks.cephadm:Adding mgr.x on smithi181 2024-09-11T08:04:03.406 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph orch apply mgr '2;smithi150=y;smithi181=x' 2024-09-11T08:04:03.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:03 smithi150 ceph-mon[29276]: Reconfiguring mon.b (monmap changed)... 2024-09-11T08:04:03.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:03 smithi150 ceph-mon[29276]: Reconfiguring daemon mon.b on smithi181 2024-09-11T08:04:03.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:03 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1508796310' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:04:03.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:03 smithi150 ceph-mon[39612]: Reconfiguring mon.b (monmap changed)... 2024-09-11T08:04:03.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:03 smithi150 ceph-mon[39612]: Reconfiguring daemon mon.b on smithi181 2024-09-11T08:04:03.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:03 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1508796310' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:04:03.530 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:03 smithi181 ceph-mon[32701]: Reconfiguring mon.b (monmap changed)... 2024-09-11T08:04:03.530 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:03 smithi181 ceph-mon[32701]: Reconfiguring daemon mon.b on smithi181 2024-09-11T08:04:03.530 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:03 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1508796310' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:04:03.696 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:04:04.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:04 smithi150 ceph-mon[29276]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:04:04.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:04 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:04.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:04 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:04.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:04 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:04:04.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:04 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:04:04.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:04 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:04:04.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:04 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:04.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:04 smithi150 ceph-mon[39612]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:04:04.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:04 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:04.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:04 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:04.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:04 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:04:04.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:04 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:04:04.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:04 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:04:04.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:04 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:04.531 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:04 smithi181 ceph-mon[32701]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:04:04.531 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:04 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:04.531 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:04 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:04.531 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:04 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:04:04.531 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:04 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:04:04.531 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:04 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:04:04.531 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:04 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:05.433 INFO:teuthology.orchestra.run.smithi181.stdout:Scheduled mgr update... 2024-09-11T08:04:05.959 DEBUG:teuthology.orchestra.run.smithi181:mgr.x> sudo journalctl -f -n 0 -u ceph-c1bddff4-7013-11ef-bcea-c7b262605968@mgr.x.service 2024-09-11T08:04:05.961 INFO:tasks.cephadm:Deploying OSDs... 2024-09-11T08:04:05.961 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-11T08:04:05.961 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/scratch_devs of=/dev/stdout 2024-09-11T08:04:05.981 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-11T08:04:05.981 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_1 2024-09-11T08:04:06.040 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-11T08:04:06.040 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T08:04:06.040 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 973 Links: 1 2024-09-11T08:04:06.040 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T08:04:06.040 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T08:04:06.040 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-09-11 08:03:50.908568209 +0000 2024-09-11T08:04:06.040 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-09-11 07:57:13.969220123 +0000 2024-09-11T08:04:06.040 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-09-11 07:57:13.969220123 +0000 2024-09-11T08:04:06.040 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2024-09-11 07:57:13.969220123 +0000 2024-09-11T08:04:06.041 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-11T08:04:06.107 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-09-11T08:04:06.107 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-09-11T08:04:06.107 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000241743 s, 2.1 MB/s 2024-09-11T08:04:06.109 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-11T08:04:06.167 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_2 2024-09-11T08:04:06.225 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-11T08:04:06.225 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T08:04:06.225 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 961 Links: 1 2024-09-11T08:04:06.225 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T08:04:06.225 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T08:04:06.225 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-09-11 08:03:50.908568209 +0000 2024-09-11T08:04:06.225 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-09-11 07:57:13.967220472 +0000 2024-09-11T08:04:06.225 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-09-11 07:57:13.967220472 +0000 2024-09-11T08:04:06.225 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2024-09-11 07:57:13.967220472 +0000 2024-09-11T08:04:06.225 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-11T08:04:06.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:06 smithi181 ceph-mon[32701]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:04:06.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:06 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:06.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:06 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:04:06.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:06 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:04:06.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:06 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:04:06.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:06 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:06.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:06 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-11T08:04:06.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:06 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-11T08:04:06.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:06 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-11T08:04:06.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:06 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:04:06.290 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-09-11T08:04:06.290 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-09-11T08:04:06.291 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000230798 s, 2.2 MB/s 2024-09-11T08:04:06.292 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-11T08:04:06.349 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_3 2024-09-11T08:04:06.406 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-11T08:04:06.406 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T08:04:06.406 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 984 Links: 1 2024-09-11T08:04:06.406 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T08:04:06.406 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T08:04:06.406 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-09-11 08:03:50.909568190 +0000 2024-09-11T08:04:06.406 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-09-11 07:57:13.970219948 +0000 2024-09-11T08:04:06.406 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-09-11 07:57:13.970219948 +0000 2024-09-11T08:04:06.406 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2024-09-11 07:57:13.970219948 +0000 2024-09-11T08:04:06.407 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-11T08:04:06.468 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:06 smithi150 ceph-mon[29276]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:04:06.468 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:06 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:06.468 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:06 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:04:06.468 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:06 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:04:06.468 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:06 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:04:06.468 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:06 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:06.468 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:06 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-11T08:04:06.468 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:06 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-11T08:04:06.468 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:06 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-11T08:04:06.468 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:06 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:04:06.469 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:06 smithi150 ceph-mon[39612]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:04:06.469 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:06 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:06.469 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:06 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:04:06.469 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:06 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:04:06.469 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:06 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:04:06.469 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:06 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:06.469 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:06 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-11T08:04:06.469 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:06 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-11T08:04:06.469 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:06 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-11T08:04:06.469 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:06 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:04:06.470 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-09-11T08:04:06.470 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-09-11T08:04:06.470 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000225236 s, 2.3 MB/s 2024-09-11T08:04:06.471 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-11T08:04:06.528 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_4 2024-09-11T08:04:06.584 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-11T08:04:06.584 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T08:04:06.584 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 978 Links: 1 2024-09-11T08:04:06.585 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T08:04:06.585 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T08:04:06.585 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-09-11 08:03:50.909568190 +0000 2024-09-11T08:04:06.585 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-09-11 07:57:13.969220123 +0000 2024-09-11T08:04:06.585 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-09-11 07:57:13.969220123 +0000 2024-09-11T08:04:06.585 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2024-09-11 07:57:13.969220123 +0000 2024-09-11T08:04:06.585 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-11T08:04:06.648 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-09-11T08:04:06.648 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-09-11T08:04:06.649 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000302268 s, 1.7 MB/s 2024-09-11T08:04:06.650 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-11T08:04:06.706 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-11T08:04:06.706 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/scratch_devs of=/dev/stdout 2024-09-11T08:04:06.722 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-11T08:04:06.722 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_1 2024-09-11T08:04:06.778 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-11T08:04:06.778 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T08:04:06.778 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 981 Links: 1 2024-09-11T08:04:06.778 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T08:04:06.779 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T08:04:06.779 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-09-11 08:03:42.188656633 +0000 2024-09-11T08:04:06.779 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-09-11 07:57:13.136905372 +0000 2024-09-11T08:04:06.779 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-09-11 07:57:13.136905372 +0000 2024-09-11T08:04:06.779 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-09-11 07:57:13.136905372 +0000 2024-09-11T08:04:06.779 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-11T08:04:06.842 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-09-11T08:04:06.842 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-09-11T08:04:06.842 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000248272 s, 2.1 MB/s 2024-09-11T08:04:06.844 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-11T08:04:06.900 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_2 2024-09-11T08:04:06.955 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-11T08:04:06.955 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T08:04:06.955 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 971 Links: 1 2024-09-11T08:04:06.955 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T08:04:06.955 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T08:04:06.955 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-09-11 08:03:42.188656633 +0000 2024-09-11T08:04:06.955 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-09-11 07:57:13.135905391 +0000 2024-09-11T08:04:06.955 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-09-11 07:57:13.135905391 +0000 2024-09-11T08:04:06.955 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-09-11 07:57:13.135905391 +0000 2024-09-11T08:04:06.956 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-11T08:04:07.153 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-09-11T08:04:07.153 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-09-11T08:04:07.154 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000215196 s, 2.4 MB/s 2024-09-11T08:04:07.154 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-11T08:04:07.170 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_3 2024-09-11T08:04:07.230 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-11T08:04:07.230 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T08:04:07.230 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 954 Links: 1 2024-09-11T08:04:07.230 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T08:04:07.230 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T08:04:07.230 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-09-11 08:04:07.108187588 +0000 2024-09-11T08:04:07.230 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-09-11 07:57:13.132905448 +0000 2024-09-11T08:04:07.230 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-09-11 07:57:13.132905448 +0000 2024-09-11T08:04:07.230 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-09-11 07:57:13.132905448 +0000 2024-09-11T08:04:07.231 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-11T08:04:07.333 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-09-11T08:04:07.333 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-09-11T08:04:07.333 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000197672 s, 2.6 MB/s 2024-09-11T08:04:07.334 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-11T08:04:07.349 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_4 2024-09-11T08:04:07.405 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-11T08:04:07.406 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T08:04:07.406 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 962 Links: 1 2024-09-11T08:04:07.406 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T08:04:07.406 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T08:04:07.406 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-09-11 08:04:07.288184200 +0000 2024-09-11T08:04:07.406 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-09-11 07:57:13.132905448 +0000 2024-09-11T08:04:07.406 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-09-11 07:57:13.132905448 +0000 2024-09-11T08:04:07.406 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-09-11 07:57:13.132905448 +0000 2024-09-11T08:04:07.406 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-11T08:04:07.473 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-09-11T08:04:07.473 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-09-11T08:04:07.474 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000173661 s, 2.9 MB/s 2024-09-11T08:04:07.475 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-11T08:04:07.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:07 smithi150 ceph-mon[29276]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi150=y;smithi181=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:04:07.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:07 smithi150 ceph-mon[29276]: Saving service mgr spec with placement smithi150=y;smithi181=x;count:2 2024-09-11T08:04:07.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:07 smithi150 ceph-mon[29276]: Deploying daemon mgr.x on smithi181 2024-09-11T08:04:07.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:07 smithi150 ceph-mon[39612]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi150=y;smithi181=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:04:07.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:07 smithi150 ceph-mon[39612]: Saving service mgr spec with placement smithi150=y;smithi181=x;count:2 2024-09-11T08:04:07.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:07 smithi150 ceph-mon[39612]: Deploying daemon mgr.x on smithi181 2024-09-11T08:04:07.531 INFO:tasks.cephadm:Deploying osd.0 on smithi150 with /dev/vg_nvme/lv_4... 2024-09-11T08:04:07.531 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-09-11T08:04:07.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:07 smithi181 ceph-mon[32701]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi150=y;smithi181=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:04:07.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:07 smithi181 ceph-mon[32701]: Saving service mgr spec with placement smithi150=y;smithi181=x;count:2 2024-09-11T08:04:07.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:07 smithi181 ceph-mon[32701]: Deploying daemon mgr.x on smithi181 2024-09-11T08:04:07.779 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:04:08.209 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:04:08 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: 2024-09-11T08:04:08.077+0000 7faf4282bec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-11T08:04:08.209 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:04:08 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: 2024-09-11T08:04:08.207+0000 7faf4282bec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-11T08:04:08.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:08 smithi150 ceph-mon[29276]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:04:08.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:08 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:08.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:08 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:08.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:08 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:08.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:08 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:08.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:08 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:04:08.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:08 smithi150 ceph-mon[39612]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:04:08.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:08 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:08.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:08 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:08.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:08 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:08.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:08 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:08.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:08 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:04:08.574 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:04:08 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: 2024-09-11T08:04:08.361+0000 7faf4282bec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-11T08:04:08.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:08 smithi181 ceph-mon[32701]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:04:08.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:08 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:08.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:08 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:08.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:08 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:08.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:08 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:08.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:08 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:04:08.883 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:04:08 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: 2024-09-11T08:04:08.641+0000 7faf4282bec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-11T08:04:08.883 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:04:08 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: 2024-09-11T08:04:08.704+0000 7faf4282bec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-11T08:04:08.883 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:04:08 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: 2024-09-11T08:04:08.869+0000 7faf4282bec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-11T08:04:09.591 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-09-11T08:04:09.686 INFO:teuthology.orchestra.run.smithi150.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-09-11T08:04:09.699 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 10+0 records in 2024-09-11T08:04:09.700 INFO:teuthology.orchestra.run.smithi150.stderr:10+0 records out 2024-09-11T08:04:09.700 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0116102 s, 903 MB/s 2024-09-11T08:04:09.700 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping successful for: 2024-09-11T08:04:09.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:09 smithi150 ceph-mon[29276]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:04:09.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:09 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:09.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:09 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:09.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:09 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:09.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:09 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:04:09.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:09 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:04:09.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:09 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:09.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:09 smithi150 ceph-mon[29276]: Reconfiguring mgr.y (unknown last config time)... 2024-09-11T08:04:09.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:09 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-11T08:04:09.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:09 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-11T08:04:09.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:09 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:04:09.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:09 smithi150 ceph-mon[29276]: Reconfiguring daemon mgr.y on smithi150 2024-09-11T08:04:09.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:09 smithi150 ceph-mon[39612]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:04:09.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:09 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:09.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:09 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:09.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:09 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:09.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:09 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:04:09.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:09 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:04:09.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:09 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:09.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:09 smithi150 ceph-mon[39612]: Reconfiguring mgr.y (unknown last config time)... 2024-09-11T08:04:09.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:09 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-11T08:04:09.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:09 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-11T08:04:09.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:09 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:04:09.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:09 smithi150 ceph-mon[39612]: Reconfiguring daemon mgr.y on smithi150 2024-09-11T08:04:10.074 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:04:09 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: 2024-09-11T08:04:09.716+0000 7faf4282bec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-11T08:04:10.074 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:04:09 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: 2024-09-11T08:04:09.778+0000 7faf4282bec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-11T08:04:10.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:09 smithi181 ceph-mon[32701]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:04:10.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:09 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:10.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:09 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:10.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:09 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:10.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:09 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:04:10.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:09 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:04:10.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:09 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:10.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:09 smithi181 ceph-mon[32701]: Reconfiguring mgr.y (unknown last config time)... 2024-09-11T08:04:10.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:09 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-11T08:04:10.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:09 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-11T08:04:10.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:09 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:04:10.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:09 smithi181 ceph-mon[32701]: Reconfiguring daemon mgr.y on smithi150 2024-09-11T08:04:10.762 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:04:10 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: 2024-09-11T08:04:10.478+0000 7faf4282bec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-11T08:04:10.762 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:04:10 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: 2024-09-11T08:04:10.586+0000 7faf4282bec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-11T08:04:10.762 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:04:10 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: 2024-09-11T08:04:10.704+0000 7faf4282bec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-11T08:04:11.015 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:04:10 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: 2024-09-11T08:04:10.761+0000 7faf4282bec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-11T08:04:11.015 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:04:10 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: 2024-09-11T08:04:10.888+0000 7faf4282bec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-11T08:04:11.015 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:04:10 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: 2024-09-11T08:04:10.957+0000 7faf4282bec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-11T08:04:11.015 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:04:11 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: 2024-09-11T08:04:11.014+0000 7faf4282bec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-11T08:04:11.239 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi150:vg_nvme/lv_4 2024-09-11T08:04:11.272 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:04:11 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: 2024-09-11T08:04:11.145+0000 7faf4282bec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-11T08:04:11.273 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:04:11 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: 2024-09-11T08:04:11.206+0000 7faf4282bec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-11T08:04:11.273 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:04:11 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: 2024-09-11T08:04:11.271+0000 7faf4282bec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-11T08:04:11.481 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:04:11.574 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:04:11 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: 2024-09-11T08:04:11.351+0000 7faf4282bec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-11T08:04:11.574 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:04:11 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: 2024-09-11T08:04:11.406+0000 7faf4282bec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-11T08:04:11.574 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:04:11 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: 2024-09-11T08:04:11.461+0000 7faf4282bec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-11T08:04:11.913 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:11 smithi150 ceph-mon[29276]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:04:11.913 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:11 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:11.913 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:11 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:11.913 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:11 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:04:11.914 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:11 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:04:11.914 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:11 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:04:11.914 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:11 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:11.914 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:11 smithi150 ceph-mon[39612]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:04:11.914 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:11 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:11.914 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:11 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:11.914 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:11 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:04:11.914 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:11 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:04:11.914 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:11 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:04:11.914 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:11 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:12.193 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:11 smithi181 ceph-mon[32701]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:04:12.193 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:11 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:12.193 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:11 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:12.193 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:11 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:04:12.193 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:11 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:04:12.193 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:11 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:04:12.193 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:11 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:12.574 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:04:12 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: 2024-09-11T08:04:12.191+0000 7faf4282bec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-11T08:04:13.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:12 smithi150 ceph-mon[29276]: Standby manager daemon x started 2024-09-11T08:04:13.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:12 smithi150 ceph-mon[29276]: from='mgr.? 172.21.15.181:0/2182450792' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-11T08:04:13.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:12 smithi150 ceph-mon[29276]: from='mgr.? 172.21.15.181:0/2182450792' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-11T08:04:13.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:12 smithi150 ceph-mon[29276]: from='mgr.? 172.21.15.181:0/2182450792' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-11T08:04:13.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:12 smithi150 ceph-mon[29276]: from='mgr.? 172.21.15.181:0/2182450792' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-11T08:04:13.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:12 smithi150 ceph-mon[39612]: Standby manager daemon x started 2024-09-11T08:04:13.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:12 smithi150 ceph-mon[39612]: from='mgr.? 172.21.15.181:0/2182450792' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-11T08:04:13.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:12 smithi150 ceph-mon[39612]: from='mgr.? 172.21.15.181:0/2182450792' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-11T08:04:13.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:12 smithi150 ceph-mon[39612]: from='mgr.? 172.21.15.181:0/2182450792' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-11T08:04:13.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:12 smithi150 ceph-mon[39612]: from='mgr.? 172.21.15.181:0/2182450792' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-11T08:04:13.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:12 smithi181 ceph-mon[32701]: Standby manager daemon x started 2024-09-11T08:04:13.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:12 smithi181 ceph-mon[32701]: from='mgr.? 172.21.15.181:0/2182450792' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-11T08:04:13.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:12 smithi181 ceph-mon[32701]: from='mgr.? 172.21.15.181:0/2182450792' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-11T08:04:13.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:12 smithi181 ceph-mon[32701]: from='mgr.? 172.21.15.181:0/2182450792' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-11T08:04:13.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:12 smithi181 ceph-mon[32701]: from='mgr.? 172.21.15.181:0/2182450792' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-11T08:04:13.972 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:13 smithi150 ceph-mon[29276]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:04:13.972 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:13 smithi150 ceph-mon[29276]: mgrmap e14: y(active, since 64s), standbys: x 2024-09-11T08:04:13.972 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:13 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-11T08:04:13.973 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:13 smithi150 ceph-mon[29276]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:04:13.973 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:13 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-11T08:04:13.973 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:13 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-11T08:04:13.973 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:13 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:04:13.973 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:13 smithi150 ceph-mon[39612]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:04:13.973 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:13 smithi150 ceph-mon[39612]: mgrmap e14: y(active, since 64s), standbys: x 2024-09-11T08:04:13.973 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:13 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-11T08:04:13.973 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:13 smithi150 ceph-mon[39612]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:04:13.973 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:13 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-11T08:04:13.973 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:13 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-11T08:04:13.974 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:13 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:04:14.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:13 smithi181 ceph-mon[32701]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:04:14.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:13 smithi181 ceph-mon[32701]: mgrmap e14: y(active, since 64s), standbys: x 2024-09-11T08:04:14.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:13 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-11T08:04:14.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:13 smithi181 ceph-mon[32701]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:04:14.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:13 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-11T08:04:14.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:13 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-11T08:04:14.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:13 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:04:15.952 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:15 smithi150 ceph-mon[29276]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:04:15.952 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:15 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/4071040659' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "37b63db9-ed2a-4701-b63d-03ff9885cc0e"}]: dispatch 2024-09-11T08:04:15.952 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:15 smithi150 ceph-mon[29276]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "37b63db9-ed2a-4701-b63d-03ff9885cc0e"}]: dispatch 2024-09-11T08:04:15.952 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:15 smithi150 ceph-mon[29276]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "37b63db9-ed2a-4701-b63d-03ff9885cc0e"}]': finished 2024-09-11T08:04:15.953 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:15 smithi150 ceph-mon[29276]: osdmap e5: 1 total, 0 up, 1 in 2024-09-11T08:04:15.953 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:15 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-11T08:04:15.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:15 smithi150 ceph-mon[39612]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:04:15.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:15 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/4071040659' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "37b63db9-ed2a-4701-b63d-03ff9885cc0e"}]: dispatch 2024-09-11T08:04:15.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:15 smithi150 ceph-mon[39612]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "37b63db9-ed2a-4701-b63d-03ff9885cc0e"}]: dispatch 2024-09-11T08:04:15.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:15 smithi150 ceph-mon[39612]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "37b63db9-ed2a-4701-b63d-03ff9885cc0e"}]': finished 2024-09-11T08:04:15.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:15 smithi150 ceph-mon[39612]: osdmap e5: 1 total, 0 up, 1 in 2024-09-11T08:04:15.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:15 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-11T08:04:16.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:15 smithi181 ceph-mon[32701]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:04:16.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:15 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/4071040659' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "37b63db9-ed2a-4701-b63d-03ff9885cc0e"}]: dispatch 2024-09-11T08:04:16.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:15 smithi181 ceph-mon[32701]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "37b63db9-ed2a-4701-b63d-03ff9885cc0e"}]: dispatch 2024-09-11T08:04:16.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:15 smithi181 ceph-mon[32701]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "37b63db9-ed2a-4701-b63d-03ff9885cc0e"}]': finished 2024-09-11T08:04:16.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:15 smithi181 ceph-mon[32701]: osdmap e5: 1 total, 0 up, 1 in 2024-09-11T08:04:16.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:15 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-11T08:04:17.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:16 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1677874990' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-11T08:04:17.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:16 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1677874990' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-11T08:04:17.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:16 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1677874990' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-11T08:04:18.137 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:17 smithi150 ceph-mon[29276]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:04:18.137 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:17 smithi150 ceph-mon[39612]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:04:18.225 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:17 smithi181 ceph-mon[32701]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:04:20.214 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:19 smithi150 ceph-mon[29276]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:04:20.214 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:19 smithi150 ceph-mon[39612]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:04:20.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:19 smithi181 ceph-mon[32701]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:04:22.059 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:21 smithi150 ceph-mon[29276]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:04:22.059 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:21 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-11T08:04:22.060 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:21 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:04:22.060 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:21 smithi150 ceph-mon[39612]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:04:22.060 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:21 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-11T08:04:22.060 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:21 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:04:22.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:21 smithi181 ceph-mon[32701]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:04:22.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:21 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-11T08:04:22.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:21 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:04:22.893 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:22 smithi150 ceph-mon[29276]: Deploying daemon osd.0 on smithi150 2024-09-11T08:04:22.893 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:22 smithi150 ceph-mon[39612]: Deploying daemon osd.0 on smithi150 2024-09-11T08:04:23.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:22 smithi181 ceph-mon[32701]: Deploying daemon osd.0 on smithi150 2024-09-11T08:04:24.074 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:23 smithi150 ceph-mon[29276]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:04:24.074 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:23 smithi150 ceph-mon[39612]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:04:24.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:23 smithi181 ceph-mon[32701]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:04:26.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:26 smithi150 ceph-mon[29276]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:04:26.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:26 smithi150 ceph-mon[39612]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:04:26.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:26 smithi181 ceph-mon[32701]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:04:27.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:27 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:04:27.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:27 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:27.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:27 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:27.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:27 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:04:27.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:27 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:27.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:27 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:27.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:27 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:04:27.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:27 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:27.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:27 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:28.328 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:28 smithi150 ceph-mon[29276]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:04:28.328 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:28 smithi150 ceph-mon[39612]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:04:28.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:28 smithi181 ceph-mon[32701]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:04:29.842 INFO:teuthology.orchestra.run.smithi150.stdout:Created osd(s) 0 on host 'smithi150' 2024-09-11T08:04:30.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:30 smithi150 ceph-mon[29276]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:04:30.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:30 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:30.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:30 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:30.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:30 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:04:30.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:30 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:04:30.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:30 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:30.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:30 smithi150 ceph-mon[29276]: from='osd.0 v2:172.21.15.150:6801/334255279' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-11T08:04:30.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:30 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:04:30.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:30 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:30.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:30 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:30.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:30 smithi150 ceph-mon[39612]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:04:30.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:30 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:30.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:30 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:30.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:30 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:04:30.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:30 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:04:30.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:30 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:30.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:30 smithi150 ceph-mon[39612]: from='osd.0 v2:172.21.15.150:6801/334255279' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-11T08:04:30.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:30 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:04:30.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:30 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:30.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:30 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:30.572 DEBUG:teuthology.orchestra.run.smithi150:osd.0> sudo journalctl -f -n 0 -u ceph-c1bddff4-7013-11ef-bcea-c7b262605968@osd.0.service 2024-09-11T08:04:30.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:30 smithi181 ceph-mon[32701]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:04:30.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:30 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:30.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:30 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:30.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:30 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:04:30.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:30 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:04:30.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:30 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:30.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:30 smithi181 ceph-mon[32701]: from='osd.0 v2:172.21.15.150:6801/334255279' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-11T08:04:30.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:30 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:04:30.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:30 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:30.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:30 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:30.575 INFO:tasks.cephadm:Deploying osd.1 on smithi150 with /dev/vg_nvme/lv_3... 2024-09-11T08:04:30.575 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-09-11T08:04:31.247 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:04:31.651 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:31 smithi150 ceph-mon[29276]: from='osd.0 v2:172.21.15.150:6801/334255279' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-11T08:04:31.652 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:31 smithi150 ceph-mon[29276]: osdmap e6: 1 total, 0 up, 1 in 2024-09-11T08:04:31.652 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:31 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-11T08:04:31.652 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:31 smithi150 ceph-mon[29276]: from='osd.0 v2:172.21.15.150:6801/334255279' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-09-11T08:04:31.652 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:31 smithi150 ceph-mon[29276]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:04:31.652 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:31 smithi150 ceph-mon[39612]: from='osd.0 v2:172.21.15.150:6801/334255279' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-11T08:04:31.652 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:31 smithi150 ceph-mon[39612]: osdmap e6: 1 total, 0 up, 1 in 2024-09-11T08:04:31.652 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:31 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-11T08:04:31.652 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:31 smithi150 ceph-mon[39612]: from='osd.0 v2:172.21.15.150:6801/334255279' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-09-11T08:04:31.653 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:31 smithi150 ceph-mon[39612]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:04:31.653 INFO:journalctl@ceph.osd.0.smithi150.stdout:Sep 11 08:04:31 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-0[43516]: 2024-09-11T08:04:31.520+0000 7fdc665ca640 -1 osd.0 0 waiting for initial osdmap 2024-09-11T08:04:31.653 INFO:journalctl@ceph.osd.0.smithi150.stdout:Sep 11 08:04:31 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-0[43516]: 2024-09-11T08:04:31.524+0000 7fdc5dbd8640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T08:04:31.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:31 smithi181 ceph-mon[32701]: from='osd.0 v2:172.21.15.150:6801/334255279' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-11T08:04:31.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:31 smithi181 ceph-mon[32701]: osdmap e6: 1 total, 0 up, 1 in 2024-09-11T08:04:31.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:31 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-11T08:04:31.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:31 smithi181 ceph-mon[32701]: from='osd.0 v2:172.21.15.150:6801/334255279' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-09-11T08:04:31.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:31 smithi181 ceph-mon[32701]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:04:32.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:32 smithi150 ceph-mon[39612]: from='osd.0 v2:172.21.15.150:6801/334255279' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-09-11T08:04:32.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:32 smithi150 ceph-mon[39612]: osdmap e7: 1 total, 0 up, 1 in 2024-09-11T08:04:32.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:32 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-11T08:04:32.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:32 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-11T08:04:32.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:32 smithi150 ceph-mon[29276]: from='osd.0 v2:172.21.15.150:6801/334255279' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-09-11T08:04:32.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:32 smithi150 ceph-mon[29276]: osdmap e7: 1 total, 0 up, 1 in 2024-09-11T08:04:32.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:32 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-11T08:04:32.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:32 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-11T08:04:32.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:32 smithi181 ceph-mon[32701]: from='osd.0 v2:172.21.15.150:6801/334255279' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-09-11T08:04:32.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:32 smithi181 ceph-mon[32701]: osdmap e7: 1 total, 0 up, 1 in 2024-09-11T08:04:32.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:32 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-11T08:04:32.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:32 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-11T08:04:33.705 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:33 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:04:33.705 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:33 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:04:33.705 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:33 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-11T08:04:33.705 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:33 smithi150 ceph-mon[29276]: osd.0 v2:172.21.15.150:6801/334255279 boot 2024-09-11T08:04:33.705 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:33 smithi150 ceph-mon[29276]: osdmap e8: 1 total, 1 up, 1 in 2024-09-11T08:04:33.705 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:33 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-11T08:04:33.705 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:33 smithi150 ceph-mon[29276]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:04:33.706 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:33 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:04:33.706 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:33 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:04:33.706 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:33 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-11T08:04:33.706 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:33 smithi150 ceph-mon[39612]: osd.0 v2:172.21.15.150:6801/334255279 boot 2024-09-11T08:04:33.706 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:33 smithi150 ceph-mon[39612]: osdmap e8: 1 total, 1 up, 1 in 2024-09-11T08:04:33.706 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:33 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-11T08:04:33.706 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:33 smithi150 ceph-mon[39612]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:04:33.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:33 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:04:33.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:33 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:04:33.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:33 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-11T08:04:33.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:33 smithi181 ceph-mon[32701]: osd.0 v2:172.21.15.150:6801/334255279 boot 2024-09-11T08:04:33.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:33 smithi181 ceph-mon[32701]: osdmap e8: 1 total, 1 up, 1 in 2024-09-11T08:04:33.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:33 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-11T08:04:33.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:33 smithi181 ceph-mon[32701]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:04:33.953 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-09-11T08:04:34.039 INFO:teuthology.orchestra.run.smithi150.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-09-11T08:04:34.053 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 10+0 records in 2024-09-11T08:04:34.054 INFO:teuthology.orchestra.run.smithi150.stderr:10+0 records out 2024-09-11T08:04:34.054 INFO:teuthology.orchestra.run.smithi150.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0125629 s, 835 MB/s 2024-09-11T08:04:34.054 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping successful for: 2024-09-11T08:04:34.651 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:34 smithi150 ceph-mon[39612]: osdmap e9: 1 total, 1 up, 1 in 2024-09-11T08:04:34.651 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:34 smithi150 ceph-mon[29276]: osdmap e9: 1 total, 1 up, 1 in 2024-09-11T08:04:34.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:34 smithi181 ceph-mon[32701]: osdmap e9: 1 total, 1 up, 1 in 2024-09-11T08:04:35.379 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi150:vg_nvme/lv_3 2024-09-11T08:04:35.620 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:04:36.024 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:35 smithi150 ceph-mon[39612]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:04:36.024 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:35 smithi150 ceph-mon[39612]: Detected new or changed devices on smithi150 2024-09-11T08:04:36.024 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:35 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:36.024 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:35 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:36.024 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:35 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:04:36.024 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:35 smithi150 ceph-mon[39612]: Adjusting osd_memory_target on smithi150 to 16064M 2024-09-11T08:04:36.024 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:35 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:36.024 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:35 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:04:36.024 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:35 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:04:36.024 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:35 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:36.025 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:35 smithi150 ceph-mon[29276]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:04:36.025 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:35 smithi150 ceph-mon[29276]: Detected new or changed devices on smithi150 2024-09-11T08:04:36.025 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:35 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:36.025 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:35 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:36.025 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:35 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:04:36.025 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:35 smithi150 ceph-mon[29276]: Adjusting osd_memory_target on smithi150 to 16064M 2024-09-11T08:04:36.025 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:35 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:36.025 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:35 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:04:36.025 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:35 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:04:36.025 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:35 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:36.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:35 smithi181 ceph-mon[32701]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:04:36.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:35 smithi181 ceph-mon[32701]: Detected new or changed devices on smithi150 2024-09-11T08:04:36.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:35 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:36.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:35 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:36.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:35 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:04:36.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:35 smithi181 ceph-mon[32701]: Adjusting osd_memory_target on smithi150 to 16064M 2024-09-11T08:04:36.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:35 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:36.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:35 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:04:36.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:35 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:04:36.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:35 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:38.107 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:37 smithi150 ceph-mon[29276]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:04:38.108 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:37 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-11T08:04:38.108 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:37 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-11T08:04:38.108 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:37 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:04:38.108 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:37 smithi150 ceph-mon[39612]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:04:38.108 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:37 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-11T08:04:38.108 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:37 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-11T08:04:38.108 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:37 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:04:38.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:37 smithi181 ceph-mon[32701]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:04:38.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:37 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-11T08:04:38.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:37 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-11T08:04:38.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:37 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:04:39.199 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:38 smithi150 ceph-mon[29276]: from='client.14238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:04:39.199 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:38 smithi150 ceph-mon[39612]: from='client.14238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:04:39.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:38 smithi181 ceph-mon[32701]: from='client.14238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:04:40.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:39 smithi150 ceph-mon[29276]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:04:40.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:39 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/760461058' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80378442-02bc-48aa-b8ca-6113a4092079"}]: dispatch 2024-09-11T08:04:40.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:39 smithi150 ceph-mon[29276]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80378442-02bc-48aa-b8ca-6113a4092079"}]: dispatch 2024-09-11T08:04:40.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:39 smithi150 ceph-mon[29276]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "80378442-02bc-48aa-b8ca-6113a4092079"}]': finished 2024-09-11T08:04:40.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:39 smithi150 ceph-mon[29276]: osdmap e10: 2 total, 1 up, 2 in 2024-09-11T08:04:40.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:39 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-11T08:04:40.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:39 smithi150 ceph-mon[39612]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:04:40.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:39 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/760461058' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80378442-02bc-48aa-b8ca-6113a4092079"}]: dispatch 2024-09-11T08:04:40.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:39 smithi150 ceph-mon[39612]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80378442-02bc-48aa-b8ca-6113a4092079"}]: dispatch 2024-09-11T08:04:40.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:39 smithi150 ceph-mon[39612]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "80378442-02bc-48aa-b8ca-6113a4092079"}]': finished 2024-09-11T08:04:40.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:39 smithi150 ceph-mon[39612]: osdmap e10: 2 total, 1 up, 2 in 2024-09-11T08:04:40.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:39 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-11T08:04:40.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:39 smithi181 ceph-mon[32701]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:04:40.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:39 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/760461058' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80378442-02bc-48aa-b8ca-6113a4092079"}]: dispatch 2024-09-11T08:04:40.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:39 smithi181 ceph-mon[32701]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80378442-02bc-48aa-b8ca-6113a4092079"}]: dispatch 2024-09-11T08:04:40.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:39 smithi181 ceph-mon[32701]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "80378442-02bc-48aa-b8ca-6113a4092079"}]': finished 2024-09-11T08:04:40.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:39 smithi181 ceph-mon[32701]: osdmap e10: 2 total, 1 up, 2 in 2024-09-11T08:04:40.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:39 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-11T08:04:41.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:40 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2174013562' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-11T08:04:41.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:40 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2174013562' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-11T08:04:41.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:40 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2174013562' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-11T08:04:42.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:41 smithi150 ceph-mon[29276]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:04:42.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:41 smithi150 ceph-mon[39612]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:04:42.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:41 smithi181 ceph-mon[32701]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:04:43.979 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:43 smithi150 ceph-mon[29276]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:04:43.979 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:43 smithi150 ceph-mon[39612]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:04:44.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:43 smithi181 ceph-mon[32701]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:04:46.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:46 smithi150 ceph-mon[39612]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:04:46.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:46 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-11T08:04:46.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:46 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:04:46.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:46 smithi150 ceph-mon[29276]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:04:46.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:46 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-11T08:04:46.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:46 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:04:46.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:46 smithi181 ceph-mon[32701]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:04:46.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:46 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-11T08:04:46.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:46 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:04:47.234 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:47 smithi150 ceph-mon[29276]: Deploying daemon osd.1 on smithi150 2024-09-11T08:04:47.234 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:47 smithi150 ceph-mon[39612]: Deploying daemon osd.1 on smithi150 2024-09-11T08:04:47.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:47 smithi181 ceph-mon[32701]: Deploying daemon osd.1 on smithi150 2024-09-11T08:04:48.388 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:48 smithi150 ceph-mon[29276]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:04:48.388 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:48 smithi150 ceph-mon[39612]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:04:48.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:48 smithi181 ceph-mon[32701]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:04:50.323 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:50 smithi150 ceph-mon[29276]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:04:50.324 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:50 smithi150 ceph-mon[39612]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:04:50.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:50 smithi181 ceph-mon[32701]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:04:51.334 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:51 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:04:51.335 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:51 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:51.335 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:51 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:51.335 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:51 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:04:51.335 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:51 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:51.335 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:51 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:51.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:51 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:04:51.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:51 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:51.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:51 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:52.351 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:52 smithi150 ceph-mon[29276]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:04:52.351 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:52 smithi150 ceph-mon[39612]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:04:52.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:52 smithi181 ceph-mon[32701]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:04:53.843 INFO:teuthology.orchestra.run.smithi150.stdout:Created osd(s) 1 on host 'smithi150' 2024-09-11T08:04:54.468 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:54 smithi150 ceph-mon[29276]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:04:54.468 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:54 smithi150 ceph-mon[29276]: from='osd.1 v2:172.21.15.150:6805/1129510619' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-11T08:04:54.468 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:54 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:54.468 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:54 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:54.468 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:54 smithi150 ceph-mon[39612]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:04:54.469 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:54 smithi150 ceph-mon[39612]: from='osd.1 v2:172.21.15.150:6805/1129510619' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-11T08:04:54.469 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:54 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:54.469 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:54 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:54.495 DEBUG:teuthology.orchestra.run.smithi150:osd.1> sudo journalctl -f -n 0 -u ceph-c1bddff4-7013-11ef-bcea-c7b262605968@osd.1.service 2024-09-11T08:04:54.497 INFO:tasks.cephadm:Deploying osd.2 on smithi150 with /dev/vg_nvme/lv_2... 2024-09-11T08:04:54.497 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-09-11T08:04:54.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:54 smithi181 ceph-mon[32701]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:04:54.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:54 smithi181 ceph-mon[32701]: from='osd.1 v2:172.21.15.150:6805/1129510619' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-11T08:04:54.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:54 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:54.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:54 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:54.788 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:04:55.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:55 smithi150 ceph-mon[29276]: from='osd.1 v2:172.21.15.150:6805/1129510619' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-11T08:04:55.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:55 smithi150 ceph-mon[29276]: osdmap e11: 2 total, 1 up, 2 in 2024-09-11T08:04:55.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:55 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-11T08:04:55.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:55 smithi150 ceph-mon[29276]: from='osd.1 v2:172.21.15.150:6805/1129510619' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-09-11T08:04:55.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:55 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:55.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:55 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:55.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:55 smithi150 ceph-mon[39612]: from='osd.1 v2:172.21.15.150:6805/1129510619' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-11T08:04:55.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:55 smithi150 ceph-mon[39612]: osdmap e11: 2 total, 1 up, 2 in 2024-09-11T08:04:55.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:55 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-11T08:04:55.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:55 smithi150 ceph-mon[39612]: from='osd.1 v2:172.21.15.150:6805/1129510619' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-09-11T08:04:55.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:55 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:55.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:55 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:55.490 INFO:journalctl@ceph.osd.1.smithi150.stdout:Sep 11 08:04:55 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-1[49526]: 2024-09-11T08:04:55.246+0000 7f3b88adb640 -1 osd.1 0 waiting for initial osdmap 2024-09-11T08:04:55.491 INFO:journalctl@ceph.osd.1.smithi150.stdout:Sep 11 08:04:55 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-1[49526]: 2024-09-11T08:04:55.249+0000 7f3b84914640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T08:04:55.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:55 smithi181 ceph-mon[32701]: from='osd.1 v2:172.21.15.150:6805/1129510619' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-11T08:04:55.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:55 smithi181 ceph-mon[32701]: osdmap e11: 2 total, 1 up, 2 in 2024-09-11T08:04:55.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:55 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-11T08:04:55.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:55 smithi181 ceph-mon[32701]: from='osd.1 v2:172.21.15.150:6805/1129510619' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-09-11T08:04:55.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:55 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:55.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:55 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:04:56.405 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:56 smithi150 ceph-mon[29276]: pgmap v54: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:04:56.405 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:56 smithi150 ceph-mon[29276]: from='osd.1 v2:172.21.15.150:6805/1129510619' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-09-11T08:04:56.405 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:56 smithi150 ceph-mon[29276]: osdmap e12: 2 total, 1 up, 2 in 2024-09-11T08:04:56.406 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:56 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-11T08:04:56.406 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:56 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-11T08:04:56.406 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:56 smithi150 ceph-mon[39612]: pgmap v54: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:04:56.406 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:56 smithi150 ceph-mon[39612]: from='osd.1 v2:172.21.15.150:6805/1129510619' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-09-11T08:04:56.406 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:56 smithi150 ceph-mon[39612]: osdmap e12: 2 total, 1 up, 2 in 2024-09-11T08:04:56.406 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:56 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-11T08:04:56.406 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:56 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-11T08:04:56.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:56 smithi181 ceph-mon[32701]: pgmap v54: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:04:56.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:56 smithi181 ceph-mon[32701]: from='osd.1 v2:172.21.15.150:6805/1129510619' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-09-11T08:04:56.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:56 smithi181 ceph-mon[32701]: osdmap e12: 2 total, 1 up, 2 in 2024-09-11T08:04:56.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:56 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-11T08:04:56.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:56 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-11T08:04:56.642 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-09-11T08:04:56.750 INFO:teuthology.orchestra.run.smithi150.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-09-11T08:04:56.765 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 10+0 records in 2024-09-11T08:04:56.765 INFO:teuthology.orchestra.run.smithi150.stderr:10+0 records out 2024-09-11T08:04:56.765 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0132518 s, 791 MB/s 2024-09-11T08:04:56.765 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping successful for: 2024-09-11T08:04:57.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:57 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:04:57.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:57 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:04:57.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:57 smithi181 ceph-mon[32701]: osd.1 v2:172.21.15.150:6805/1129510619 boot 2024-09-11T08:04:57.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:57 smithi181 ceph-mon[32701]: osdmap e13: 2 total, 2 up, 2 in 2024-09-11T08:04:57.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:57 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-11T08:04:57.611 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:57 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:04:57.611 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:57 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:04:57.611 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:57 smithi150 ceph-mon[29276]: osd.1 v2:172.21.15.150:6805/1129510619 boot 2024-09-11T08:04:57.611 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:57 smithi150 ceph-mon[29276]: osdmap e13: 2 total, 2 up, 2 in 2024-09-11T08:04:57.611 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:57 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-11T08:04:57.611 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:57 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:04:57.611 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:57 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:04:57.611 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:57 smithi150 ceph-mon[39612]: osd.1 v2:172.21.15.150:6805/1129510619 boot 2024-09-11T08:04:57.611 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:57 smithi150 ceph-mon[39612]: osdmap e13: 2 total, 2 up, 2 in 2024-09-11T08:04:57.612 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:57 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-11T08:04:57.979 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi150:vg_nvme/lv_2 2024-09-11T08:04:58.467 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:04:58.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:58 smithi150 ceph-mon[29276]: pgmap v57: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:04:58.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:04:58 smithi150 ceph-mon[29276]: osdmap e14: 2 total, 2 up, 2 in 2024-09-11T08:04:58.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:58 smithi150 ceph-mon[39612]: pgmap v57: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:04:58.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:04:58 smithi150 ceph-mon[39612]: osdmap e14: 2 total, 2 up, 2 in 2024-09-11T08:04:58.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:58 smithi181 ceph-mon[32701]: pgmap v57: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:04:58.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:04:58 smithi181 ceph-mon[32701]: osdmap e14: 2 total, 2 up, 2 in 2024-09-11T08:05:00.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:00 smithi150 ceph-mon[29276]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:05:00.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:00 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:00.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:00 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:00.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:00 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:05:00.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:00 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:05:00.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:00 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:00.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:00 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:05:00.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:00 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:05:00.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:00 smithi150 ceph-mon[39612]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:05:00.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:00 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:00.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:00 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:00.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:00 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:05:00.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:00 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:05:00.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:00 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:00.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:00 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:05:00.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:00 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:05:00.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:00 smithi181 ceph-mon[32701]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:05:00.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:00 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:00.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:00 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:00.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:00 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:05:00.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:00 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:05:00.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:00 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:00.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:00 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:05:00.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:00 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:05:01.516 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:01 smithi150 ceph-mon[29276]: Detected new or changed devices on smithi150 2024-09-11T08:05:01.516 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:01 smithi150 ceph-mon[29276]: Adjusting osd_memory_target on smithi150 to 8032M 2024-09-11T08:05:01.516 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:01 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:01.516 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:01 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:05:01.516 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:01 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:05:01.517 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:01 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:05:01.517 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:01 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:01.517 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:01 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-11T08:05:01.517 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:01 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-11T08:05:01.517 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:01 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:05:01.517 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:01 smithi150 ceph-mon[39612]: Detected new or changed devices on smithi150 2024-09-11T08:05:01.517 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:01 smithi150 ceph-mon[39612]: Adjusting osd_memory_target on smithi150 to 8032M 2024-09-11T08:05:01.517 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:01 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:01.517 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:01 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:05:01.517 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:01 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:05:01.517 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:01 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:05:01.517 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:01 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:01.518 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:01 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-11T08:05:01.518 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:01 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-11T08:05:01.518 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:01 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:05:01.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:01 smithi181 ceph-mon[32701]: Detected new or changed devices on smithi150 2024-09-11T08:05:01.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:01 smithi181 ceph-mon[32701]: Adjusting osd_memory_target on smithi150 to 8032M 2024-09-11T08:05:01.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:01 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:01.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:01 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:05:01.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:01 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:05:01.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:01 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:05:01.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:01 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:01.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:01 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-11T08:05:01.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:01 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-11T08:05:01.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:01 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:05:02.537 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:02 smithi150 ceph-mon[29276]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:05:02.537 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:02 smithi150 ceph-mon[29276]: from='client.24170 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:05:02.537 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:02 smithi150 ceph-mon[39612]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:05:02.537 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:02 smithi150 ceph-mon[39612]: from='client.24170 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:05:02.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:02 smithi181 ceph-mon[32701]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:05:02.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:02 smithi181 ceph-mon[32701]: from='client.24170 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:05:03.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:03 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/4133411197' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0955bb92-36a0-4e9a-98e1-b3a5af89f6d2"}]: dispatch 2024-09-11T08:05:03.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:03 smithi150 ceph-mon[39612]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0955bb92-36a0-4e9a-98e1-b3a5af89f6d2"}]: dispatch 2024-09-11T08:05:03.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:03 smithi150 ceph-mon[39612]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0955bb92-36a0-4e9a-98e1-b3a5af89f6d2"}]': finished 2024-09-11T08:05:03.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:03 smithi150 ceph-mon[39612]: osdmap e15: 3 total, 2 up, 3 in 2024-09-11T08:05:03.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:03 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-11T08:05:03.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:03 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/4133411197' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0955bb92-36a0-4e9a-98e1-b3a5af89f6d2"}]: dispatch 2024-09-11T08:05:03.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:03 smithi150 ceph-mon[29276]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0955bb92-36a0-4e9a-98e1-b3a5af89f6d2"}]: dispatch 2024-09-11T08:05:03.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:03 smithi150 ceph-mon[29276]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0955bb92-36a0-4e9a-98e1-b3a5af89f6d2"}]': finished 2024-09-11T08:05:03.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:03 smithi150 ceph-mon[29276]: osdmap e15: 3 total, 2 up, 3 in 2024-09-11T08:05:03.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:03 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-11T08:05:03.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:03 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/4133411197' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0955bb92-36a0-4e9a-98e1-b3a5af89f6d2"}]: dispatch 2024-09-11T08:05:03.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:03 smithi181 ceph-mon[32701]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0955bb92-36a0-4e9a-98e1-b3a5af89f6d2"}]: dispatch 2024-09-11T08:05:03.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:03 smithi181 ceph-mon[32701]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0955bb92-36a0-4e9a-98e1-b3a5af89f6d2"}]': finished 2024-09-11T08:05:03.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:03 smithi181 ceph-mon[32701]: osdmap e15: 3 total, 2 up, 3 in 2024-09-11T08:05:03.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:03 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-11T08:05:04.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:04 smithi150 ceph-mon[29276]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:05:04.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:04 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/420972747' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-11T08:05:04.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:04 smithi150 ceph-mon[39612]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:05:04.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:04 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/420972747' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-11T08:05:04.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:04 smithi181 ceph-mon[32701]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:05:04.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:04 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/420972747' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-11T08:05:06.612 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:06 smithi150 ceph-mon[29276]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:05:06.612 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:06 smithi150 ceph-mon[39612]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:05:06.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:06 smithi181 ceph-mon[32701]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:05:08.652 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:08 smithi150 ceph-mon[29276]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:05:08.653 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:08 smithi150 ceph-mon[39612]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:05:08.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:08 smithi181 ceph-mon[32701]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:05:09.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:09 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-11T08:05:09.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:09 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:05:09.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:09 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-11T08:05:09.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:09 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:05:09.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:09 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-11T08:05:09.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:09 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:05:10.621 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:10 smithi150 ceph-mon[29276]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:05:10.621 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:10 smithi150 ceph-mon[29276]: Deploying daemon osd.2 on smithi150 2024-09-11T08:05:10.622 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:10 smithi150 ceph-mon[39612]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:05:10.622 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:10 smithi150 ceph-mon[39612]: Deploying daemon osd.2 on smithi150 2024-09-11T08:05:10.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:10 smithi181 ceph-mon[32701]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:05:10.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:10 smithi181 ceph-mon[32701]: Deploying daemon osd.2 on smithi150 2024-09-11T08:05:12.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:12 smithi150 ceph-mon[29276]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:05:12.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:12 smithi150 ceph-mon[39612]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:05:12.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:12 smithi181 ceph-mon[32701]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:05:14.663 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:14 smithi150 ceph-mon[29276]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:05:14.663 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:14 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:05:14.663 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:14 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:14.664 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:14 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:14.664 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:14 smithi150 ceph-mon[39612]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:05:14.664 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:14 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:05:14.664 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:14 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:14.664 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:14 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:14.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:14 smithi181 ceph-mon[32701]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:05:14.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:14 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:05:14.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:14 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:14.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:14 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:16.634 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:16 smithi150 ceph-mon[29276]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:05:16.634 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:16 smithi150 ceph-mon[39612]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:05:16.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:16 smithi181 ceph-mon[32701]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:05:17.373 INFO:teuthology.orchestra.run.smithi150.stdout:Created osd(s) 2 on host 'smithi150' 2024-09-11T08:05:17.987 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:17 smithi150 ceph-mon[29276]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:05:17.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:17 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:17.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:17 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:17.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:17 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:05:17.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:17 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:05:17.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:17 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:17.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:17 smithi150 ceph-mon[29276]: from='osd.2 v2:172.21.15.150:6809/719975422' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-11T08:05:17.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:17 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:05:17.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:17 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:17.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:17 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:17.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:17 smithi150 ceph-mon[39612]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:05:17.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:17 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:17.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:17 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:17.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:17 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:05:17.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:17 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:05:17.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:17 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:17.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:17 smithi150 ceph-mon[39612]: from='osd.2 v2:172.21.15.150:6809/719975422' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-11T08:05:17.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:17 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:05:17.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:17 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:17.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:17 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:17.991 DEBUG:teuthology.orchestra.run.smithi150:osd.2> sudo journalctl -f -n 0 -u ceph-c1bddff4-7013-11ef-bcea-c7b262605968@osd.2.service 2024-09-11T08:05:17.993 INFO:tasks.cephadm:Deploying osd.3 on smithi150 with /dev/vg_nvme/lv_1... 2024-09-11T08:05:17.993 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-09-11T08:05:18.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:17 smithi181 ceph-mon[32701]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:05:18.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:17 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:18.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:17 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:18.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:17 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:05:18.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:17 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:05:18.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:17 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:18.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:17 smithi181 ceph-mon[32701]: from='osd.2 v2:172.21.15.150:6809/719975422' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-11T08:05:18.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:17 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:05:18.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:17 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:18.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:17 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:19.125 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:05:19.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:18 smithi150 ceph-mon[29276]: from='osd.2 v2:172.21.15.150:6809/719975422' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-11T08:05:19.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:18 smithi150 ceph-mon[29276]: osdmap e16: 3 total, 2 up, 3 in 2024-09-11T08:05:19.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:18 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-11T08:05:19.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:18 smithi150 ceph-mon[29276]: from='osd.2 v2:172.21.15.150:6809/719975422' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-09-11T08:05:19.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:18 smithi150 ceph-mon[39612]: from='osd.2 v2:172.21.15.150:6809/719975422' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-11T08:05:19.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:18 smithi150 ceph-mon[39612]: osdmap e16: 3 total, 2 up, 3 in 2024-09-11T08:05:19.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:18 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-11T08:05:19.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:18 smithi150 ceph-mon[39612]: from='osd.2 v2:172.21.15.150:6809/719975422' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-09-11T08:05:19.240 INFO:journalctl@ceph.osd.2.smithi150.stdout:Sep 11 08:05:18 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-2[55516]: 2024-09-11T08:05:18.876+0000 7f82ff7f5640 -1 osd.2 0 waiting for initial osdmap 2024-09-11T08:05:19.240 INFO:journalctl@ceph.osd.2.smithi150.stdout:Sep 11 08:05:18 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-2[55516]: 2024-09-11T08:05:18.879+0000 7f82fae2d640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T08:05:19.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:18 smithi181 ceph-mon[32701]: from='osd.2 v2:172.21.15.150:6809/719975422' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-11T08:05:19.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:18 smithi181 ceph-mon[32701]: osdmap e16: 3 total, 2 up, 3 in 2024-09-11T08:05:19.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:18 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-11T08:05:19.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:18 smithi181 ceph-mon[32701]: from='osd.2 v2:172.21.15.150:6809/719975422' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-09-11T08:05:20.221 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:19 smithi150 ceph-mon[29276]: pgmap v71: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:05:20.221 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:19 smithi150 ceph-mon[29276]: from='osd.2 v2:172.21.15.150:6809/719975422' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-09-11T08:05:20.221 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:19 smithi150 ceph-mon[29276]: osdmap e17: 3 total, 2 up, 3 in 2024-09-11T08:05:20.221 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:19 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-11T08:05:20.221 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:19 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-11T08:05:20.222 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:19 smithi150 ceph-mon[39612]: pgmap v71: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:05:20.222 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:19 smithi150 ceph-mon[39612]: from='osd.2 v2:172.21.15.150:6809/719975422' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-09-11T08:05:20.222 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:19 smithi150 ceph-mon[39612]: osdmap e17: 3 total, 2 up, 3 in 2024-09-11T08:05:20.222 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:19 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-11T08:05:20.222 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:19 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-11T08:05:20.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:19 smithi181 ceph-mon[32701]: pgmap v71: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:05:20.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:19 smithi181 ceph-mon[32701]: from='osd.2 v2:172.21.15.150:6809/719975422' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-09-11T08:05:20.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:19 smithi181 ceph-mon[32701]: osdmap e17: 3 total, 2 up, 3 in 2024-09-11T08:05:20.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:19 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-11T08:05:20.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:19 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-11T08:05:21.243 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-09-11T08:05:21.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:21 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:05:21.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:21 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:05:21.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:21 smithi181 ceph-mon[32701]: osd.2 v2:172.21.15.150:6809/719975422 boot 2024-09-11T08:05:21.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:21 smithi181 ceph-mon[32701]: osdmap e18: 3 total, 3 up, 3 in 2024-09-11T08:05:21.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:21 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-11T08:05:21.326 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:21 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:05:21.326 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:21 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:05:21.326 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:21 smithi150 ceph-mon[29276]: osd.2 v2:172.21.15.150:6809/719975422 boot 2024-09-11T08:05:21.326 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:21 smithi150 ceph-mon[29276]: osdmap e18: 3 total, 3 up, 3 in 2024-09-11T08:05:21.326 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:21 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-11T08:05:21.327 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:21 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:05:21.327 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:21 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:05:21.327 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:21 smithi150 ceph-mon[39612]: osd.2 v2:172.21.15.150:6809/719975422 boot 2024-09-11T08:05:21.327 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:21 smithi150 ceph-mon[39612]: osdmap e18: 3 total, 3 up, 3 in 2024-09-11T08:05:21.327 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:21 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-11T08:05:21.336 INFO:teuthology.orchestra.run.smithi150.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-09-11T08:05:21.353 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 10+0 records in 2024-09-11T08:05:21.353 INFO:teuthology.orchestra.run.smithi150.stderr:10+0 records out 2024-09-11T08:05:21.353 INFO:teuthology.orchestra.run.smithi150.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0138605 s, 757 MB/s 2024-09-11T08:05:21.353 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping successful for: 2024-09-11T08:05:22.200 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:22 smithi150 ceph-mon[29276]: pgmap v74: 0 pgs: ; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:05:22.200 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:22 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-09-11T08:05:22.200 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:22 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-09-11T08:05:22.200 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:22 smithi150 ceph-mon[29276]: osdmap e19: 3 total, 3 up, 3 in 2024-09-11T08:05:22.200 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:22 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-11T08:05:22.200 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:22 smithi150 ceph-mon[39612]: pgmap v74: 0 pgs: ; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:05:22.200 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:22 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-09-11T08:05:22.201 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:22 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-09-11T08:05:22.201 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:22 smithi150 ceph-mon[39612]: osdmap e19: 3 total, 3 up, 3 in 2024-09-11T08:05:22.201 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:22 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-11T08:05:22.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:22 smithi181 ceph-mon[32701]: pgmap v74: 0 pgs: ; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:05:22.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:22 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-09-11T08:05:22.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:22 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-09-11T08:05:22.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:22 smithi181 ceph-mon[32701]: osdmap e19: 3 total, 3 up, 3 in 2024-09-11T08:05:22.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:22 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-11T08:05:22.700 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi150:vg_nvme/lv_1 2024-09-11T08:05:22.951 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:05:23.192 INFO:journalctl@ceph.osd.0.smithi150.stdout:Sep 11 08:05:22 smithi150 sudo[59157]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-11T08:05:23.192 INFO:journalctl@ceph.osd.0.smithi150.stdout:Sep 11 08:05:22 smithi150 sudo[59157]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-11T08:05:23.192 INFO:journalctl@ceph.osd.0.smithi150.stdout:Sep 11 08:05:22 smithi150 sudo[59157]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-11T08:05:23.192 INFO:journalctl@ceph.osd.0.smithi150.stdout:Sep 11 08:05:22 smithi150 sudo[59157]: pam_unix(sudo:session): session closed for user root 2024-09-11T08:05:23.464 INFO:journalctl@ceph.osd.0.smithi150.stdout:Sep 11 08:05:23 smithi150 sudo[59192]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-11T08:05:23.464 INFO:journalctl@ceph.osd.0.smithi150.stdout:Sep 11 08:05:23 smithi150 sudo[59192]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-11T08:05:23.464 INFO:journalctl@ceph.osd.0.smithi150.stdout:Sep 11 08:05:23 smithi150 sudo[59192]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-11T08:05:23.464 INFO:journalctl@ceph.osd.0.smithi150.stdout:Sep 11 08:05:23 smithi150 sudo[59192]: pam_unix(sudo:session): session closed for user root 2024-09-11T08:05:23.465 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:23 smithi150 ceph-mon[29276]: Detected new or changed devices on smithi150 2024-09-11T08:05:23.465 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:23 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:23.465 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:23 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:23.465 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:23 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:05:23.465 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:23 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:05:23.465 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:23 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:05:23.465 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:23 smithi150 ceph-mon[29276]: Adjusting osd_memory_target on smithi150 to 5354M 2024-09-11T08:05:23.465 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:23 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:23.465 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:23 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:05:23.465 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:23 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-11T08:05:23.465 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:23 smithi150 ceph-mon[29276]: osdmap e20: 3 total, 3 up, 3 in 2024-09-11T08:05:23.466 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:23 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:05:23.466 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:23 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:23.466 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:23 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:23.466 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:23 smithi150 ceph-mon[39612]: Detected new or changed devices on smithi150 2024-09-11T08:05:23.466 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:23 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:23.466 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:23 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:23.466 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:23 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:05:23.466 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:23 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:05:23.466 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:23 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:05:23.466 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:23 smithi150 ceph-mon[39612]: Adjusting osd_memory_target on smithi150 to 5354M 2024-09-11T08:05:23.466 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:23 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:23.466 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:23 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:05:23.467 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:23 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-11T08:05:23.467 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:23 smithi150 ceph-mon[39612]: osdmap e20: 3 total, 3 up, 3 in 2024-09-11T08:05:23.467 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:23 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:05:23.467 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:23 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:23.467 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:23 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:23.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:23 smithi181 ceph-mon[32701]: Detected new or changed devices on smithi150 2024-09-11T08:05:23.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:23 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:23.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:23 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:23.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:23 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:05:23.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:23 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:05:23.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:23 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:05:23.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:23 smithi181 ceph-mon[32701]: Adjusting osd_memory_target on smithi150 to 5354M 2024-09-11T08:05:23.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:23 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:23.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:23 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:05:23.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:23 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-11T08:05:23.575 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:23 smithi181 ceph-mon[32701]: osdmap e20: 3 total, 3 up, 3 in 2024-09-11T08:05:23.575 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:23 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:05:23.575 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:23 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:23.575 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:23 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:23.726 INFO:journalctl@ceph.osd.1.smithi150.stdout:Sep 11 08:05:23 smithi150 sudo[59208]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-11T08:05:23.726 INFO:journalctl@ceph.osd.1.smithi150.stdout:Sep 11 08:05:23 smithi150 sudo[59208]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-11T08:05:23.726 INFO:journalctl@ceph.osd.1.smithi150.stdout:Sep 11 08:05:23 smithi150 sudo[59208]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-11T08:05:23.727 INFO:journalctl@ceph.osd.1.smithi150.stdout:Sep 11 08:05:23 smithi150 sudo[59208]: pam_unix(sudo:session): session closed for user root 2024-09-11T08:05:23.977 INFO:journalctl@ceph.osd.1.smithi150.stdout:Sep 11 08:05:23 smithi150 sudo[59212]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-11T08:05:23.977 INFO:journalctl@ceph.osd.1.smithi150.stdout:Sep 11 08:05:23 smithi150 sudo[59212]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-11T08:05:23.977 INFO:journalctl@ceph.osd.1.smithi150.stdout:Sep 11 08:05:23 smithi150 sudo[59212]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-11T08:05:23.977 INFO:journalctl@ceph.osd.1.smithi150.stdout:Sep 11 08:05:23 smithi150 sudo[59212]: pam_unix(sudo:session): session closed for user root 2024-09-11T08:05:24.238 INFO:journalctl@ceph.osd.2.smithi150.stdout:Sep 11 08:05:23 smithi150 sudo[59235]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-11T08:05:24.238 INFO:journalctl@ceph.osd.2.smithi150.stdout:Sep 11 08:05:23 smithi150 sudo[59235]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-11T08:05:24.238 INFO:journalctl@ceph.osd.2.smithi150.stdout:Sep 11 08:05:23 smithi150 sudo[59235]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-11T08:05:24.238 INFO:journalctl@ceph.osd.2.smithi150.stdout:Sep 11 08:05:23 smithi150 sudo[59235]: pam_unix(sudo:session): session closed for user root 2024-09-11T08:05:24.238 INFO:journalctl@ceph.osd.2.smithi150.stdout:Sep 11 08:05:24 smithi150 sudo[59246]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-11T08:05:24.489 INFO:journalctl@ceph.osd.2.smithi150.stdout:Sep 11 08:05:24 smithi150 sudo[59246]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-11T08:05:24.489 INFO:journalctl@ceph.osd.2.smithi150.stdout:Sep 11 08:05:24 smithi150 sudo[59246]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-11T08:05:24.489 INFO:journalctl@ceph.osd.2.smithi150.stdout:Sep 11 08:05:24 smithi150 sudo[59246]: pam_unix(sudo:session): session closed for user root 2024-09-11T08:05:24.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:24 smithi150 sudo[59270]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-11T08:05:24.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:24 smithi150 sudo[59270]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-11T08:05:24.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:24 smithi150 sudo[59270]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-11T08:05:24.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:24 smithi150 sudo[59270]: pam_unix(sudo:session): session closed for user root 2024-09-11T08:05:25.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:24 smithi181 ceph-mon[32701]: pgmap v77: 1 pgs: 1 creating+activating; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:05:25.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:24 smithi181 ceph-mon[32701]: osdmap e21: 3 total, 3 up, 3 in 2024-09-11T08:05:25.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:24 smithi181 ceph-mon[32701]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-11T08:05:25.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:24 smithi181 ceph-mon[32701]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-11T08:05:25.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:24 smithi181 sudo[35205]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-11T08:05:25.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:24 smithi181 sudo[35205]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-11T08:05:25.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:24 smithi181 sudo[35205]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-11T08:05:25.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:25 smithi181 sudo[35205]: pam_unix(sudo:session): session closed for user root 2024-09-11T08:05:25.153 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:24 smithi150 ceph-mon[39612]: pgmap v77: 1 pgs: 1 creating+activating; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:05:25.153 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:24 smithi150 ceph-mon[39612]: osdmap e21: 3 total, 3 up, 3 in 2024-09-11T08:05:25.153 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:24 smithi150 ceph-mon[39612]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-11T08:05:25.153 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:24 smithi150 ceph-mon[39612]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-11T08:05:25.153 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:24 smithi150 sudo[59274]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-11T08:05:25.154 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:24 smithi150 sudo[59274]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-11T08:05:25.154 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:24 smithi150 sudo[59274]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-11T08:05:25.154 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:24 smithi150 sudo[59274]: pam_unix(sudo:session): session closed for user root 2024-09-11T08:05:25.154 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:24 smithi150 ceph-mon[29276]: pgmap v77: 1 pgs: 1 creating+activating; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:05:25.154 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:24 smithi150 ceph-mon[29276]: osdmap e21: 3 total, 3 up, 3 in 2024-09-11T08:05:25.154 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:24 smithi150 ceph-mon[29276]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-11T08:05:25.154 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:24 smithi150 ceph-mon[29276]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-11T08:05:25.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:25 smithi181 sudo[35208]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-11T08:05:25.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:25 smithi181 sudo[35208]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-11T08:05:25.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:25 smithi181 sudo[35208]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-11T08:05:25.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:25 smithi181 sudo[35208]: pam_unix(sudo:session): session closed for user root 2024-09-11T08:05:25.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:25 smithi150 sudo[59414]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-11T08:05:25.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:25 smithi150 sudo[59414]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-11T08:05:25.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:25 smithi150 sudo[59414]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-11T08:05:25.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:25 smithi150 sudo[59414]: pam_unix(sudo:session): session closed for user root 2024-09-11T08:05:26.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:25 smithi181 ceph-mon[32701]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:05:26.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:25 smithi181 ceph-mon[32701]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-11T08:05:26.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:25 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-11T08:05:26.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:25 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:05:26.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:25 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:05:26.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:25 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-11T08:05:26.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:25 smithi181 ceph-mon[32701]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-11T08:05:26.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:25 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-11T08:05:26.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:25 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:05:26.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:25 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:05:26.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:25 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-11T08:05:26.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:25 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:05:26.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:25 smithi181 ceph-mon[32701]: pgmap v79: 1 pgs: 1 creating+activating; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:05:26.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:25 smithi181 ceph-mon[32701]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-11T08:05:26.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:25 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:26.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:25 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-11T08:05:26.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:25 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:05:26.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:25 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:05:26.136 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:25 smithi150 ceph-mon[29276]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:05:26.136 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:25 smithi150 ceph-mon[29276]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-11T08:05:26.136 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:25 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-11T08:05:26.136 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:25 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:05:26.136 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:25 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:05:26.136 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:25 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-11T08:05:26.136 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:25 smithi150 ceph-mon[29276]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-11T08:05:26.136 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:25 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-11T08:05:26.136 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:25 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:05:26.136 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:25 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:05:26.137 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:25 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-11T08:05:26.137 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:25 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:05:26.137 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:25 smithi150 ceph-mon[29276]: pgmap v79: 1 pgs: 1 creating+activating; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:05:26.137 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:25 smithi150 ceph-mon[29276]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-11T08:05:26.137 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:25 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:26.137 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:25 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-11T08:05:26.137 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:25 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:05:26.137 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:25 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:05:26.137 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:25 smithi150 sudo[59418]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-11T08:05:26.137 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:25 smithi150 sudo[59418]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-11T08:05:26.137 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:25 smithi150 sudo[59418]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-11T08:05:26.138 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:25 smithi150 sudo[59418]: pam_unix(sudo:session): session closed for user root 2024-09-11T08:05:26.138 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:25 smithi150 ceph-mon[39612]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:05:26.138 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:25 smithi150 ceph-mon[39612]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-11T08:05:26.138 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:25 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-11T08:05:26.138 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:25 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:05:26.138 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:25 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:05:26.138 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:25 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-11T08:05:26.138 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:25 smithi150 ceph-mon[39612]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-11T08:05:26.138 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:25 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-11T08:05:26.138 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:25 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:05:26.138 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:25 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:05:26.138 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:25 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-11T08:05:26.138 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:25 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:05:26.138 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:25 smithi150 ceph-mon[39612]: pgmap v79: 1 pgs: 1 creating+activating; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:05:26.139 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:25 smithi150 ceph-mon[39612]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-11T08:05:26.139 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:25 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:26.139 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:25 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-11T08:05:26.139 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:25 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:05:26.139 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:25 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:05:27.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:26 smithi181 ceph-mon[32701]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-11T08:05:27.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:26 smithi181 ceph-mon[32701]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-11T08:05:27.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:26 smithi150 ceph-mon[29276]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-11T08:05:27.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:26 smithi150 ceph-mon[29276]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-11T08:05:27.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:26 smithi150 ceph-mon[39612]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-11T08:05:27.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:26 smithi150 ceph-mon[39612]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-11T08:05:27.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:27 smithi150 ceph-mon[29276]: pgmap v80: 1 pgs: 1 creating+activating; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:05:27.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:27 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1915042962' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1d04d4d3-62e8-4460-8a07-77f7e5c62929"}]: dispatch 2024-09-11T08:05:27.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:27 smithi150 ceph-mon[29276]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1d04d4d3-62e8-4460-8a07-77f7e5c62929"}]: dispatch 2024-09-11T08:05:27.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:27 smithi150 ceph-mon[29276]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1d04d4d3-62e8-4460-8a07-77f7e5c62929"}]': finished 2024-09-11T08:05:27.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:27 smithi150 ceph-mon[29276]: osdmap e22: 4 total, 3 up, 4 in 2024-09-11T08:05:27.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:27 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-11T08:05:27.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:27 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3835076407' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-11T08:05:27.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:27 smithi150 ceph-mon[39612]: pgmap v80: 1 pgs: 1 creating+activating; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:05:27.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:27 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1915042962' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1d04d4d3-62e8-4460-8a07-77f7e5c62929"}]: dispatch 2024-09-11T08:05:27.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:27 smithi150 ceph-mon[39612]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1d04d4d3-62e8-4460-8a07-77f7e5c62929"}]: dispatch 2024-09-11T08:05:27.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:27 smithi150 ceph-mon[39612]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1d04d4d3-62e8-4460-8a07-77f7e5c62929"}]': finished 2024-09-11T08:05:27.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:27 smithi150 ceph-mon[39612]: osdmap e22: 4 total, 3 up, 4 in 2024-09-11T08:05:27.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:27 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-11T08:05:27.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:27 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3835076407' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-11T08:05:28.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:27 smithi181 ceph-mon[32701]: pgmap v80: 1 pgs: 1 creating+activating; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:05:28.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:27 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1915042962' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1d04d4d3-62e8-4460-8a07-77f7e5c62929"}]: dispatch 2024-09-11T08:05:28.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:27 smithi181 ceph-mon[32701]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1d04d4d3-62e8-4460-8a07-77f7e5c62929"}]: dispatch 2024-09-11T08:05:28.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:27 smithi181 ceph-mon[32701]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1d04d4d3-62e8-4460-8a07-77f7e5c62929"}]': finished 2024-09-11T08:05:28.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:27 smithi181 ceph-mon[32701]: osdmap e22: 4 total, 3 up, 4 in 2024-09-11T08:05:28.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:27 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-11T08:05:28.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:27 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3835076407' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-11T08:05:30.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:29 smithi181 ceph-mon[32701]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:05:30.236 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:29 smithi150 ceph-mon[29276]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:05:30.236 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:29 smithi150 ceph-mon[39612]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:05:32.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:32 smithi150 ceph-mon[29276]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:05:32.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:32 smithi150 ceph-mon[39612]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:05:32.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:32 smithi181 ceph-mon[32701]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:05:34.388 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:34 smithi150 ceph-mon[29276]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:05:34.388 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:34 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-11T08:05:34.389 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:34 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:05:34.389 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:34 smithi150 ceph-mon[29276]: Deploying daemon osd.3 on smithi150 2024-09-11T08:05:34.389 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:34 smithi150 ceph-mon[39612]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:05:34.389 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:34 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-11T08:05:34.389 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:34 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:05:34.389 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:34 smithi150 ceph-mon[39612]: Deploying daemon osd.3 on smithi150 2024-09-11T08:05:34.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:34 smithi181 ceph-mon[32701]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:05:34.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:34 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-11T08:05:34.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:34 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:05:34.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:34 smithi181 ceph-mon[32701]: Deploying daemon osd.3 on smithi150 2024-09-11T08:05:36.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:36 smithi181 ceph-mon[32701]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:05:36.637 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:36 smithi150 ceph-mon[29276]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:05:36.637 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:36 smithi150 ceph-mon[39612]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:05:38.341 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:38 smithi150 ceph-mon[29276]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:05:38.342 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:38 smithi150 ceph-mon[39612]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:05:38.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:38 smithi181 ceph-mon[32701]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:05:39.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:39 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:05:39.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:39 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:39.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:39 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:39.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:39 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:05:39.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:39 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:39.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:39 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:39.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:39 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:05:39.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:39 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:39.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:39 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:40.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:40 smithi150 ceph-mon[29276]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:05:40.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:40 smithi150 ceph-mon[39612]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:05:40.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:40 smithi181 ceph-mon[32701]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:05:42.005 INFO:teuthology.orchestra.run.smithi150.stdout:Created osd(s) 3 on host 'smithi150' 2024-09-11T08:05:42.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:42 smithi181 ceph-mon[32701]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:05:42.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:42 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:42.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:42 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:42.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:42 smithi181 ceph-mon[32701]: from='osd.3 v2:172.21.15.150:6813/3101032500' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-11T08:05:42.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:42 smithi181 ceph-mon[32701]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-11T08:05:42.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:42 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:05:42.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:42 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:05:42.575 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:42 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:42.575 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:42 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:05:42.575 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:42 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:42.575 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:42 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:42.611 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:42 smithi150 ceph-mon[29276]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:05:42.612 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:42 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:42.612 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:42 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:42.612 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:42 smithi150 ceph-mon[29276]: from='osd.3 v2:172.21.15.150:6813/3101032500' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-11T08:05:42.612 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:42 smithi150 ceph-mon[29276]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-11T08:05:42.612 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:42 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:05:42.612 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:42 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:05:42.612 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:42 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:42.612 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:42 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:05:42.612 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:42 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:42.612 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:42 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:42.612 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:42 smithi150 ceph-mon[39612]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:05:42.613 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:42 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:42.613 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:42 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:42.613 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:42 smithi150 ceph-mon[39612]: from='osd.3 v2:172.21.15.150:6813/3101032500' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-11T08:05:42.613 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:42 smithi150 ceph-mon[39612]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-11T08:05:42.613 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:42 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:05:42.613 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:42 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:05:42.613 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:42 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:42.613 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:42 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:05:42.613 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:42 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:42.613 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:42 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:42.636 DEBUG:teuthology.orchestra.run.smithi150:osd.3> sudo journalctl -f -n 0 -u ceph-c1bddff4-7013-11ef-bcea-c7b262605968@osd.3.service 2024-09-11T08:05:42.638 INFO:tasks.cephadm:Deploying osd.4 on smithi181 with /dev/vg_nvme/lv_4... 2024-09-11T08:05:42.638 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-09-11T08:05:42.886 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:05:43.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:43 smithi181 ceph-mon[32701]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-11T08:05:43.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:43 smithi181 ceph-mon[32701]: osdmap e23: 4 total, 3 up, 4 in 2024-09-11T08:05:43.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:43 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-11T08:05:43.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:43 smithi181 ceph-mon[32701]: from='osd.3 v2:172.21.15.150:6813/3101032500' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-09-11T08:05:43.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:43 smithi181 ceph-mon[32701]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-09-11T08:05:43.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:43 smithi181 ceph-mon[32701]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:05:43.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:43 smithi150 ceph-mon[39612]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-11T08:05:43.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:43 smithi150 ceph-mon[39612]: osdmap e23: 4 total, 3 up, 4 in 2024-09-11T08:05:43.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:43 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-11T08:05:43.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:43 smithi150 ceph-mon[39612]: from='osd.3 v2:172.21.15.150:6813/3101032500' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-09-11T08:05:43.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:43 smithi150 ceph-mon[39612]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-09-11T08:05:43.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:43 smithi150 ceph-mon[39612]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:05:43.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:43 smithi150 ceph-mon[29276]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-11T08:05:43.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:43 smithi150 ceph-mon[29276]: osdmap e23: 4 total, 3 up, 4 in 2024-09-11T08:05:43.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:43 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-11T08:05:43.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:43 smithi150 ceph-mon[29276]: from='osd.3 v2:172.21.15.150:6813/3101032500' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-09-11T08:05:43.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:43 smithi150 ceph-mon[29276]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-09-11T08:05:43.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:43 smithi150 ceph-mon[29276]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:05:43.741 INFO:journalctl@ceph.osd.3.smithi150.stdout:Sep 11 08:05:43 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-3[61128]: 2024-09-11T08:05:43.405+0000 7f7c2a5ee640 -1 osd.3 0 waiting for initial osdmap 2024-09-11T08:05:43.741 INFO:journalctl@ceph.osd.3.smithi150.stdout:Sep 11 08:05:43 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-3[61128]: 2024-09-11T08:05:43.408+0000 7f7c24401640 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T08:05:44.539 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-09-11T08:05:44.539 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:44 smithi181 ceph-mon[32701]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-09-11T08:05:44.539 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:44 smithi181 ceph-mon[32701]: osdmap e24: 4 total, 3 up, 4 in 2024-09-11T08:05:44.540 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:44 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-11T08:05:44.540 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:44 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-11T08:05:44.624 INFO:teuthology.orchestra.run.smithi181.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-09-11T08:05:44.639 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10+0 records in 2024-09-11T08:05:44.639 INFO:teuthology.orchestra.run.smithi181.stderr:10+0 records out 2024-09-11T08:05:44.639 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0124615 s, 841 MB/s 2024-09-11T08:05:44.639 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping successful for: 2024-09-11T08:05:44.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:44 smithi150 ceph-mon[39612]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-09-11T08:05:44.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:44 smithi150 ceph-mon[39612]: osdmap e24: 4 total, 3 up, 4 in 2024-09-11T08:05:44.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:44 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-11T08:05:44.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:44 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-11T08:05:44.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:44 smithi150 ceph-mon[29276]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-09-11T08:05:44.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:44 smithi150 ceph-mon[29276]: osdmap e24: 4 total, 3 up, 4 in 2024-09-11T08:05:44.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:44 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-11T08:05:44.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:44 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-11T08:05:45.265 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi181:vg_nvme/lv_4 2024-09-11T08:05:45.520 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:05:45.552 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:45 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:05:45.552 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:45 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:05:45.553 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:45 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-11T08:05:45.553 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:45 smithi181 ceph-mon[32701]: osd.3 v2:172.21.15.150:6813/3101032500 boot 2024-09-11T08:05:45.553 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:45 smithi181 ceph-mon[32701]: osdmap e25: 4 total, 4 up, 4 in 2024-09-11T08:05:45.553 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:45 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-11T08:05:45.553 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:45 smithi181 ceph-mon[32701]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:05:45.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:45 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:05:45.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:45 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:05:45.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:45 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-11T08:05:45.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:45 smithi150 ceph-mon[29276]: osd.3 v2:172.21.15.150:6813/3101032500 boot 2024-09-11T08:05:45.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:45 smithi150 ceph-mon[29276]: osdmap e25: 4 total, 4 up, 4 in 2024-09-11T08:05:45.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:45 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-11T08:05:45.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:45 smithi150 ceph-mon[29276]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:05:45.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:45 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:05:45.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:45 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:05:45.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:45 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-11T08:05:45.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:45 smithi150 ceph-mon[39612]: osd.3 v2:172.21.15.150:6813/3101032500 boot 2024-09-11T08:05:45.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:45 smithi150 ceph-mon[39612]: osdmap e25: 4 total, 4 up, 4 in 2024-09-11T08:05:45.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:45 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-11T08:05:45.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:45 smithi150 ceph-mon[39612]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:05:46.526 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:46 smithi181 ceph-mon[32701]: osdmap e26: 4 total, 4 up, 4 in 2024-09-11T08:05:46.526 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:46 smithi181 ceph-mon[32701]: Detected new or changed devices on smithi150 2024-09-11T08:05:46.526 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:46 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:46.526 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:46 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:46.526 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:46 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:05:46.526 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:46 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:05:46.527 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:46 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:05:46.527 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:46 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:05:46.527 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:46 smithi181 ceph-mon[32701]: Adjusting osd_memory_target on smithi150 to 4016M 2024-09-11T08:05:46.527 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:46 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:46.527 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:46 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:05:46.527 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:46 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:05:46.527 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:46 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:46.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:46 smithi150 ceph-mon[29276]: osdmap e26: 4 total, 4 up, 4 in 2024-09-11T08:05:46.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:46 smithi150 ceph-mon[29276]: Detected new or changed devices on smithi150 2024-09-11T08:05:46.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:46 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:46.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:46 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:46.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:46 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:05:46.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:46 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:05:46.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:46 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:05:46.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:46 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:05:46.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:46 smithi150 ceph-mon[29276]: Adjusting osd_memory_target on smithi150 to 4016M 2024-09-11T08:05:46.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:46 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:46.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:46 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:05:46.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:46 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:05:46.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:46 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:46.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:46 smithi150 ceph-mon[39612]: osdmap e26: 4 total, 4 up, 4 in 2024-09-11T08:05:46.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:46 smithi150 ceph-mon[39612]: Detected new or changed devices on smithi150 2024-09-11T08:05:46.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:46 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:46.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:46 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:46.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:46 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:05:46.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:46 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:05:46.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:46 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:05:46.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:46 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:05:46.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:46 smithi150 ceph-mon[39612]: Adjusting osd_memory_target on smithi150 to 4016M 2024-09-11T08:05:46.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:46 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:46.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:46 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:05:46.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:46 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:05:46.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:46 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:05:47.696 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:47 smithi181 ceph-mon[32701]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:05:47.696 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:47 smithi181 ceph-mon[32701]: from='client.24202 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:05:47.696 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:47 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-11T08:05:47.696 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:47 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-11T08:05:47.696 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:47 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:05:47.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:47 smithi150 ceph-mon[29276]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:05:47.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:47 smithi150 ceph-mon[29276]: from='client.24202 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:05:47.741 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:47 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-11T08:05:47.741 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:47 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-11T08:05:47.741 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:47 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:05:47.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:47 smithi150 ceph-mon[39612]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:05:47.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:47 smithi150 ceph-mon[39612]: from='client.24202 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:05:47.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:47 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-11T08:05:47.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:47 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-11T08:05:47.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:47 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:05:50.164 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:49 smithi181 ceph-mon[32701]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:05:50.165 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:49 smithi181 ceph-mon[32701]: from='client.? 172.21.15.181:0/3277264490' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5321a3c-d2f0-4440-8475-2c111d261139"}]: dispatch 2024-09-11T08:05:50.165 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:49 smithi181 ceph-mon[32701]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5321a3c-d2f0-4440-8475-2c111d261139"}]: dispatch 2024-09-11T08:05:50.165 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:49 smithi181 ceph-mon[32701]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c5321a3c-d2f0-4440-8475-2c111d261139"}]': finished 2024-09-11T08:05:50.165 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:49 smithi181 ceph-mon[32701]: osdmap e27: 5 total, 4 up, 5 in 2024-09-11T08:05:50.165 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:49 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-11T08:05:50.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:49 smithi150 ceph-mon[29276]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:05:50.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:49 smithi150 ceph-mon[29276]: from='client.? 172.21.15.181:0/3277264490' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5321a3c-d2f0-4440-8475-2c111d261139"}]: dispatch 2024-09-11T08:05:50.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:49 smithi150 ceph-mon[29276]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5321a3c-d2f0-4440-8475-2c111d261139"}]: dispatch 2024-09-11T08:05:50.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:49 smithi150 ceph-mon[29276]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c5321a3c-d2f0-4440-8475-2c111d261139"}]': finished 2024-09-11T08:05:50.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:49 smithi150 ceph-mon[29276]: osdmap e27: 5 total, 4 up, 5 in 2024-09-11T08:05:50.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:49 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-11T08:05:50.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:49 smithi150 ceph-mon[39612]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:05:50.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:49 smithi150 ceph-mon[39612]: from='client.? 172.21.15.181:0/3277264490' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5321a3c-d2f0-4440-8475-2c111d261139"}]: dispatch 2024-09-11T08:05:50.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:49 smithi150 ceph-mon[39612]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5321a3c-d2f0-4440-8475-2c111d261139"}]: dispatch 2024-09-11T08:05:50.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:49 smithi150 ceph-mon[39612]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c5321a3c-d2f0-4440-8475-2c111d261139"}]': finished 2024-09-11T08:05:50.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:49 smithi150 ceph-mon[39612]: osdmap e27: 5 total, 4 up, 5 in 2024-09-11T08:05:50.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:49 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-11T08:05:51.141 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:50 smithi150 ceph-mon[29276]: from='client.? 172.21.15.181:0/3142351361' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-11T08:05:51.142 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:50 smithi150 ceph-mon[39612]: from='client.? 172.21.15.181:0/3142351361' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-11T08:05:51.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:50 smithi181 ceph-mon[32701]: from='client.? 172.21.15.181:0/3142351361' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-11T08:05:52.412 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:52 smithi150 ceph-mon[29276]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:05:52.413 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:52 smithi150 ceph-mon[39612]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:05:52.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:52 smithi181 ceph-mon[32701]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:05:54.479 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:54 smithi181 ceph-mon[32701]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:05:54.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:54 smithi150 ceph-mon[29276]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:05:54.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:54 smithi150 ceph-mon[39612]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:05:55.350 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:55 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:05:55.350 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:55 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:05:55.350 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:55 smithi181 ceph-mon[32701]: osdmap e28: 5 total, 4 up, 5 in 2024-09-11T08:05:55.351 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:55 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-11T08:05:55.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:55 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:05:55.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:55 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:05:55.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:55 smithi150 ceph-mon[29276]: osdmap e28: 5 total, 4 up, 5 in 2024-09-11T08:05:55.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:55 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-11T08:05:55.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:55 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:05:55.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:55 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:05:55.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:55 smithi150 ceph-mon[39612]: osdmap e28: 5 total, 4 up, 5 in 2024-09-11T08:05:55.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:55 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-11T08:05:56.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:56 smithi181 ceph-mon[32701]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:05:56.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:56 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-11T08:05:56.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:56 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:05:56.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:56 smithi181 ceph-mon[32701]: Deploying daemon osd.4 on smithi181 2024-09-11T08:05:56.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:56 smithi150 ceph-mon[29276]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:05:56.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:56 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-11T08:05:56.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:56 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:05:56.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:56 smithi150 ceph-mon[29276]: Deploying daemon osd.4 on smithi181 2024-09-11T08:05:56.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:56 smithi150 ceph-mon[39612]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:05:56.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:56 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-11T08:05:56.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:56 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:05:56.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:56 smithi150 ceph-mon[39612]: Deploying daemon osd.4 on smithi181 2024-09-11T08:05:58.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:05:58 smithi181 ceph-mon[32701]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:05:58.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:05:58 smithi150 ceph-mon[29276]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:05:58.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:05:58 smithi150 ceph-mon[39612]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:06:00.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:00 smithi181 ceph-mon[32701]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:06:00.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:00 smithi150 ceph-mon[29276]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:06:00.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:00 smithi150 ceph-mon[39612]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:06:01.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:01 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:06:01.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:01 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:01.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:01 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:01.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:01 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:06:01.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:01 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:01.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:01 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:01.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:01 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:06:01.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:01 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:01.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:01 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:02.527 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:02 smithi181 ceph-mon[32701]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:06:02.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:02 smithi150 ceph-mon[29276]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:06:02.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:02 smithi150 ceph-mon[39612]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:06:03.473 INFO:teuthology.orchestra.run.smithi181.stdout:Created osd(s) 4 on host 'smithi181' 2024-09-11T08:06:04.085 DEBUG:teuthology.orchestra.run.smithi181:osd.4> sudo journalctl -f -n 0 -u ceph-c1bddff4-7013-11ef-bcea-c7b262605968@osd.4.service 2024-09-11T08:06:04.088 INFO:tasks.cephadm:Deploying osd.5 on smithi181 with /dev/vg_nvme/lv_3... 2024-09-11T08:06:04.088 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-09-11T08:06:04.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:04 smithi181 ceph-mon[32701]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:06:04.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:04 smithi181 ceph-mon[32701]: from='osd.4 v2:172.21.15.181:6800/2567452825' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-11T08:06:04.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:04 smithi181 ceph-mon[32701]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-11T08:06:04.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:04 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:04.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:04 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:04.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:04 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:04.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:04 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:04.378 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:06:04.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:04 smithi150 ceph-mon[39612]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:06:04.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:04 smithi150 ceph-mon[39612]: from='osd.4 v2:172.21.15.181:6800/2567452825' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-11T08:06:04.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:04 smithi150 ceph-mon[39612]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-11T08:06:04.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:04 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:04.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:04 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:04.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:04 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:04.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:04 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:04.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:04 smithi150 ceph-mon[29276]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:06:04.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:04 smithi150 ceph-mon[29276]: from='osd.4 v2:172.21.15.181:6800/2567452825' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-11T08:06:04.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:04 smithi150 ceph-mon[29276]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-11T08:06:04.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:04 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:04.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:04 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:04.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:04 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:04.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:04 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:05.576 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:05 smithi181 ceph-mon[32701]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-11T08:06:05.576 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:05 smithi181 ceph-mon[32701]: osdmap e29: 5 total, 4 up, 5 in 2024-09-11T08:06:05.576 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:05 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-11T08:06:05.576 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:05 smithi181 ceph-mon[32701]: from='osd.4 v2:172.21.15.181:6800/2567452825' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-09-11T08:06:05.577 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:05 smithi181 ceph-mon[32701]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-09-11T08:06:05.577 INFO:journalctl@ceph.osd.4.smithi181.stdout:Sep 11 08:06:05 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-4[37497]: 2024-09-11T08:06:05.306+0000 7f1044285640 -1 osd.4 0 waiting for initial osdmap 2024-09-11T08:06:05.577 INFO:journalctl@ceph.osd.4.smithi181.stdout:Sep 11 08:06:05 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-4[37497]: 2024-09-11T08:06:05.310+0000 7f103c094640 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T08:06:05.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:05 smithi150 ceph-mon[29276]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-11T08:06:05.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:05 smithi150 ceph-mon[29276]: osdmap e29: 5 total, 4 up, 5 in 2024-09-11T08:06:05.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:05 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-11T08:06:05.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:05 smithi150 ceph-mon[29276]: from='osd.4 v2:172.21.15.181:6800/2567452825' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-09-11T08:06:05.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:05 smithi150 ceph-mon[29276]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-09-11T08:06:05.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:05 smithi150 ceph-mon[39612]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-11T08:06:05.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:05 smithi150 ceph-mon[39612]: osdmap e29: 5 total, 4 up, 5 in 2024-09-11T08:06:05.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:05 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-11T08:06:05.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:05 smithi150 ceph-mon[39612]: from='osd.4 v2:172.21.15.181:6800/2567452825' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-09-11T08:06:05.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:05 smithi150 ceph-mon[39612]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-09-11T08:06:06.481 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:06 smithi181 ceph-mon[32701]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:06:06.481 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:06 smithi181 ceph-mon[32701]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-09-11T08:06:06.481 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:06 smithi181 ceph-mon[32701]: osdmap e30: 5 total, 4 up, 5 in 2024-09-11T08:06:06.481 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:06 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-11T08:06:06.481 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:06 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-11T08:06:06.708 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-09-11T08:06:06.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:06 smithi150 ceph-mon[29276]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:06:06.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:06 smithi150 ceph-mon[29276]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-09-11T08:06:06.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:06 smithi150 ceph-mon[29276]: osdmap e30: 5 total, 4 up, 5 in 2024-09-11T08:06:06.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:06 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-11T08:06:06.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:06 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-11T08:06:06.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:06 smithi150 ceph-mon[39612]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:06:06.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:06 smithi150 ceph-mon[39612]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-09-11T08:06:06.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:06 smithi150 ceph-mon[39612]: osdmap e30: 5 total, 4 up, 5 in 2024-09-11T08:06:06.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:06 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-11T08:06:06.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:06 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-11T08:06:06.791 INFO:teuthology.orchestra.run.smithi181.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-09-11T08:06:06.805 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10+0 records in 2024-09-11T08:06:06.806 INFO:teuthology.orchestra.run.smithi181.stderr:10+0 records out 2024-09-11T08:06:06.806 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.01304 s, 804 MB/s 2024-09-11T08:06:06.806 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping successful for: 2024-09-11T08:06:07.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:07 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:06:07.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:07 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:06:07.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:07 smithi181 ceph-mon[32701]: osd.4 v2:172.21.15.181:6800/2567452825 boot 2024-09-11T08:06:07.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:07 smithi181 ceph-mon[32701]: osdmap e31: 5 total, 5 up, 5 in 2024-09-11T08:06:07.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:07 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-11T08:06:07.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:07 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:06:07.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:07 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:06:07.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:07 smithi150 ceph-mon[29276]: osd.4 v2:172.21.15.181:6800/2567452825 boot 2024-09-11T08:06:07.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:07 smithi150 ceph-mon[29276]: osdmap e31: 5 total, 5 up, 5 in 2024-09-11T08:06:07.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:07 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-11T08:06:07.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:07 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:06:07.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:07 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:06:07.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:07 smithi150 ceph-mon[39612]: osd.4 v2:172.21.15.181:6800/2567452825 boot 2024-09-11T08:06:07.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:07 smithi150 ceph-mon[39612]: osdmap e31: 5 total, 5 up, 5 in 2024-09-11T08:06:07.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:07 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-11T08:06:07.872 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi181:vg_nvme/lv_3 2024-09-11T08:06:08.343 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:06:08.464 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:08 smithi181 ceph-mon[32701]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-11T08:06:08.464 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:08 smithi181 ceph-mon[32701]: osdmap e32: 5 total, 5 up, 5 in 2024-09-11T08:06:08.464 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:08 smithi181 ceph-mon[32701]: osdmap e33: 5 total, 5 up, 5 in 2024-09-11T08:06:08.622 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:08 smithi150 ceph-mon[29276]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-11T08:06:08.623 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:08 smithi150 ceph-mon[29276]: osdmap e32: 5 total, 5 up, 5 in 2024-09-11T08:06:08.623 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:08 smithi150 ceph-mon[29276]: osdmap e33: 5 total, 5 up, 5 in 2024-09-11T08:06:08.623 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:08 smithi150 ceph-mon[39612]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-11T08:06:08.623 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:08 smithi150 ceph-mon[39612]: osdmap e32: 5 total, 5 up, 5 in 2024-09-11T08:06:08.623 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:08 smithi150 ceph-mon[39612]: osdmap e33: 5 total, 5 up, 5 in 2024-09-11T08:06:10.489 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:10 smithi181 ceph-mon[32701]: pgmap v113: 1 pgs: 1 remapped+peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-11T08:06:10.489 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:10 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:10.489 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:10 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:10.489 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:10 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:06:10.489 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:10 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:10.489 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:10 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:06:10.489 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:10 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:06:10.489 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:10 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:10.489 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:10 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:06:10.489 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:10 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:06:10.489 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:10 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:06:10.490 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:10 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:10.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:10 smithi150 ceph-mon[39612]: pgmap v113: 1 pgs: 1 remapped+peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-11T08:06:10.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:10 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:10.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:10 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:10.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:10 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:06:10.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:10 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:10.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:10 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:06:10.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:10 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:06:10.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:10 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:10.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:10 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:06:10.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:10 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:06:10.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:10 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:06:10.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:10 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:10.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:10 smithi150 ceph-mon[29276]: pgmap v113: 1 pgs: 1 remapped+peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-11T08:06:10.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:10 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:10.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:10 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:10.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:10 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:06:10.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:10 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:10.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:10 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:06:10.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:10 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:06:10.741 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:10 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:10.741 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:10 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:06:10.741 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:10 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:06:10.741 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:10 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:06:10.741 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:10 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:11.408 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:11 smithi181 ceph-mon[32701]: Detected new or changed devices on smithi181 2024-09-11T08:06:11.408 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:11 smithi181 ceph-mon[32701]: Adjusting osd_memory_target on smithi181 to 17087M 2024-09-11T08:06:11.408 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:11 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-11T08:06:11.409 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:11 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-11T08:06:11.409 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:11 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:06:11.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:11 smithi150 ceph-mon[29276]: Detected new or changed devices on smithi181 2024-09-11T08:06:11.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:11 smithi150 ceph-mon[29276]: Adjusting osd_memory_target on smithi181 to 17087M 2024-09-11T08:06:11.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:11 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-11T08:06:11.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:11 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-11T08:06:11.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:11 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:06:11.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:11 smithi150 ceph-mon[39612]: Detected new or changed devices on smithi181 2024-09-11T08:06:11.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:11 smithi150 ceph-mon[39612]: Adjusting osd_memory_target on smithi181 to 17087M 2024-09-11T08:06:11.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:11 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-11T08:06:11.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:11 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-11T08:06:11.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:11 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:06:12.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:12 smithi181 ceph-mon[32701]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:06:12.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:12 smithi181 ceph-mon[32701]: pgmap v114: 1 pgs: 1 remapped+peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-11T08:06:12.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:12 smithi150 ceph-mon[29276]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:06:12.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:12 smithi150 ceph-mon[29276]: pgmap v114: 1 pgs: 1 remapped+peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-11T08:06:12.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:12 smithi150 ceph-mon[39612]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:06:12.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:12 smithi150 ceph-mon[39612]: pgmap v114: 1 pgs: 1 remapped+peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-11T08:06:13.505 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:13 smithi181 ceph-mon[32701]: from='client.? 172.21.15.181:0/3334274823' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1262b2fe-f253-4a26-bfd4-c9dffb697c77"}]: dispatch 2024-09-11T08:06:13.505 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:13 smithi181 ceph-mon[32701]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1262b2fe-f253-4a26-bfd4-c9dffb697c77"}]: dispatch 2024-09-11T08:06:13.505 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:13 smithi181 ceph-mon[32701]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1262b2fe-f253-4a26-bfd4-c9dffb697c77"}]': finished 2024-09-11T08:06:13.505 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:13 smithi181 ceph-mon[32701]: osdmap e34: 6 total, 5 up, 6 in 2024-09-11T08:06:13.506 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:13 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-11T08:06:13.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:13 smithi150 ceph-mon[29276]: from='client.? 172.21.15.181:0/3334274823' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1262b2fe-f253-4a26-bfd4-c9dffb697c77"}]: dispatch 2024-09-11T08:06:13.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:13 smithi150 ceph-mon[29276]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1262b2fe-f253-4a26-bfd4-c9dffb697c77"}]: dispatch 2024-09-11T08:06:13.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:13 smithi150 ceph-mon[29276]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1262b2fe-f253-4a26-bfd4-c9dffb697c77"}]': finished 2024-09-11T08:06:13.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:13 smithi150 ceph-mon[29276]: osdmap e34: 6 total, 5 up, 6 in 2024-09-11T08:06:13.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:13 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-11T08:06:13.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:13 smithi150 ceph-mon[39612]: from='client.? 172.21.15.181:0/3334274823' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1262b2fe-f253-4a26-bfd4-c9dffb697c77"}]: dispatch 2024-09-11T08:06:13.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:13 smithi150 ceph-mon[39612]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1262b2fe-f253-4a26-bfd4-c9dffb697c77"}]: dispatch 2024-09-11T08:06:13.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:13 smithi150 ceph-mon[39612]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1262b2fe-f253-4a26-bfd4-c9dffb697c77"}]': finished 2024-09-11T08:06:13.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:13 smithi150 ceph-mon[39612]: osdmap e34: 6 total, 5 up, 6 in 2024-09-11T08:06:13.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:13 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-11T08:06:14.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:14 smithi150 ceph-mon[29276]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-09-11T08:06:14.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:14 smithi150 ceph-mon[29276]: from='client.? 172.21.15.181:0/3378081031' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-11T08:06:14.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:14 smithi150 ceph-mon[39612]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-09-11T08:06:14.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:14 smithi150 ceph-mon[39612]: from='client.? 172.21.15.181:0/3378081031' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-11T08:06:14.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:14 smithi181 ceph-mon[32701]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-09-11T08:06:14.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:14 smithi181 ceph-mon[32701]: from='client.? 172.21.15.181:0/3378081031' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-11T08:06:16.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:16 smithi181 ceph-mon[32701]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-09-11T08:06:16.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:16 smithi150 ceph-mon[29276]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-09-11T08:06:16.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:16 smithi150 ceph-mon[39612]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-09-11T08:06:18.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:18 smithi181 ceph-mon[32701]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-09-11T08:06:18.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:18 smithi150 ceph-mon[29276]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-09-11T08:06:18.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:18 smithi150 ceph-mon[39612]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-09-11T08:06:19.535 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:19 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:06:19.535 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:19 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:06:19.535 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:19 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-11T08:06:19.535 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:19 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:06:19.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:19 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:06:19.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:19 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:06:19.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:19 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-11T08:06:19.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:19 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:06:19.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:19 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:06:19.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:19 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:06:19.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:19 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-11T08:06:19.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:19 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:06:20.629 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:20 smithi181 ceph-mon[32701]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-11T08:06:20.629 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:20 smithi181 ceph-mon[32701]: Deploying daemon osd.5 on smithi181 2024-09-11T08:06:20.629 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:20 smithi181 ceph-mon[32701]: osdmap e35: 6 total, 5 up, 6 in 2024-09-11T08:06:20.629 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:20 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-11T08:06:20.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:20 smithi150 ceph-mon[29276]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-11T08:06:20.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:20 smithi150 ceph-mon[29276]: Deploying daemon osd.5 on smithi181 2024-09-11T08:06:20.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:20 smithi150 ceph-mon[29276]: osdmap e35: 6 total, 5 up, 6 in 2024-09-11T08:06:20.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:20 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-11T08:06:20.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:20 smithi150 ceph-mon[39612]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-11T08:06:20.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:20 smithi150 ceph-mon[39612]: Deploying daemon osd.5 on smithi181 2024-09-11T08:06:20.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:20 smithi150 ceph-mon[39612]: osdmap e35: 6 total, 5 up, 6 in 2024-09-11T08:06:20.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:20 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-11T08:06:22.476 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:22 smithi181 ceph-mon[32701]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-11T08:06:22.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:22 smithi150 ceph-mon[29276]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-11T08:06:22.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:22 smithi150 ceph-mon[39612]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-11T08:06:24.620 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:24 smithi181 ceph-mon[32701]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-11T08:06:24.620 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:24 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:06:24.620 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:24 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:24.620 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:24 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:24.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:24 smithi150 ceph-mon[29276]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-11T08:06:24.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:24 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:06:24.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:24 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:24.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:24 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:24.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:24 smithi150 ceph-mon[39612]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-11T08:06:24.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:24 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:06:24.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:24 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:24.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:24 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:26.446 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:26 smithi181 ceph-mon[32701]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-11T08:06:26.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:26 smithi150 ceph-mon[29276]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-11T08:06:26.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:26 smithi150 ceph-mon[39612]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-11T08:06:27.304 INFO:teuthology.orchestra.run.smithi181.stdout:Created osd(s) 5 on host 'smithi181' 2024-09-11T08:06:27.919 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:27 smithi181 ceph-mon[32701]: 1.0 scrub starts 2024-09-11T08:06:27.920 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:27 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:27.920 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:27 smithi181 ceph-mon[32701]: 1.0 scrub ok 2024-09-11T08:06:27.920 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:27 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:27.920 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:27 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:06:27.920 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:27 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:06:27.920 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:27 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:27.920 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:27 smithi181 ceph-mon[32701]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-11T08:06:27.920 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:27 smithi181 ceph-mon[32701]: from='osd.5 v2:172.21.15.181:6804/2043435508' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-11T08:06:27.920 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:27 smithi181 ceph-mon[32701]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-11T08:06:27.920 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:27 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:06:27.920 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:27 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:27.920 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:27 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:27.943 DEBUG:teuthology.orchestra.run.smithi181:osd.5> sudo journalctl -f -n 0 -u ceph-c1bddff4-7013-11ef-bcea-c7b262605968@osd.5.service 2024-09-11T08:06:27.945 INFO:tasks.cephadm:Deploying osd.6 on smithi181 with /dev/vg_nvme/lv_2... 2024-09-11T08:06:27.945 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-09-11T08:06:27.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:27 smithi150 ceph-mon[29276]: 1.0 scrub starts 2024-09-11T08:06:27.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:27 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:27.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:27 smithi150 ceph-mon[29276]: 1.0 scrub ok 2024-09-11T08:06:27.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:27 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:27.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:27 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:06:27.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:27 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:06:27.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:27 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:27.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:27 smithi150 ceph-mon[29276]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-11T08:06:27.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:27 smithi150 ceph-mon[29276]: from='osd.5 v2:172.21.15.181:6804/2043435508' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-11T08:06:27.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:27 smithi150 ceph-mon[29276]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-11T08:06:27.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:27 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:06:27.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:27 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:27.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:27 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:27.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:27 smithi150 ceph-mon[39612]: 1.0 scrub starts 2024-09-11T08:06:27.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:27 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:27.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:27 smithi150 ceph-mon[39612]: 1.0 scrub ok 2024-09-11T08:06:27.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:27 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:27.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:27 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:06:27.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:27 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:06:27.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:27 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:27.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:27 smithi150 ceph-mon[39612]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-11T08:06:27.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:27 smithi150 ceph-mon[39612]: from='osd.5 v2:172.21.15.181:6804/2043435508' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-11T08:06:27.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:27 smithi150 ceph-mon[39612]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-11T08:06:27.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:27 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:06:27.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:27 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:27.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:27 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:28.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:28 smithi181 ceph-mon[32701]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-11T08:06:28.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:28 smithi181 ceph-mon[32701]: osdmap e36: 6 total, 5 up, 6 in 2024-09-11T08:06:28.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:28 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-11T08:06:28.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:28 smithi181 ceph-mon[32701]: from='osd.5 v2:172.21.15.181:6804/2043435508' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-09-11T08:06:28.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:28 smithi181 ceph-mon[32701]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-09-11T08:06:28.824 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:06:28 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5[43199]: 2024-09-11T08:06:28.649+0000 7fbb08f85640 -1 osd.5 0 waiting for initial osdmap 2024-09-11T08:06:28.824 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:06:28 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5[43199]: 2024-09-11T08:06:28.654+0000 7fbb055bf640 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T08:06:28.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:28 smithi150 ceph-mon[29276]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-11T08:06:28.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:28 smithi150 ceph-mon[29276]: osdmap e36: 6 total, 5 up, 6 in 2024-09-11T08:06:28.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:28 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-11T08:06:28.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:28 smithi150 ceph-mon[29276]: from='osd.5 v2:172.21.15.181:6804/2043435508' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-09-11T08:06:28.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:28 smithi150 ceph-mon[29276]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-09-11T08:06:28.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:28 smithi150 ceph-mon[39612]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-11T08:06:28.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:28 smithi150 ceph-mon[39612]: osdmap e36: 6 total, 5 up, 6 in 2024-09-11T08:06:28.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:28 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-11T08:06:28.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:28 smithi150 ceph-mon[39612]: from='osd.5 v2:172.21.15.181:6804/2043435508' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-09-11T08:06:28.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:28 smithi150 ceph-mon[39612]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-09-11T08:06:29.273 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:06:29.777 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:29 smithi181 ceph-mon[32701]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-09-11T08:06:29.778 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:29 smithi181 ceph-mon[32701]: osdmap e37: 6 total, 5 up, 6 in 2024-09-11T08:06:29.778 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:29 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-11T08:06:29.778 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:29 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-11T08:06:29.778 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:29 smithi181 ceph-mon[32701]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-11T08:06:29.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:29 smithi150 ceph-mon[29276]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-09-11T08:06:29.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:29 smithi150 ceph-mon[29276]: osdmap e37: 6 total, 5 up, 6 in 2024-09-11T08:06:29.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:29 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-11T08:06:29.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:29 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-11T08:06:29.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:29 smithi150 ceph-mon[29276]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-11T08:06:29.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:29 smithi150 ceph-mon[39612]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-09-11T08:06:29.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:29 smithi150 ceph-mon[39612]: osdmap e37: 6 total, 5 up, 6 in 2024-09-11T08:06:29.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:29 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-11T08:06:29.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:29 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-11T08:06:29.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:29 smithi150 ceph-mon[39612]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-11T08:06:30.686 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:30 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:06:30.686 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:30 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:06:30.686 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:30 smithi181 ceph-mon[32701]: osd.5 v2:172.21.15.181:6804/2043435508 boot 2024-09-11T08:06:30.686 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:30 smithi181 ceph-mon[32701]: osdmap e38: 6 total, 6 up, 6 in 2024-09-11T08:06:30.686 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:30 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-11T08:06:30.686 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:30 smithi181 ceph-mon[32701]: osdmap e39: 6 total, 6 up, 6 in 2024-09-11T08:06:30.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:30 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:06:30.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:30 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:06:30.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:30 smithi150 ceph-mon[29276]: osd.5 v2:172.21.15.181:6804/2043435508 boot 2024-09-11T08:06:30.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:30 smithi150 ceph-mon[29276]: osdmap e38: 6 total, 6 up, 6 in 2024-09-11T08:06:30.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:30 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-11T08:06:30.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:30 smithi150 ceph-mon[29276]: osdmap e39: 6 total, 6 up, 6 in 2024-09-11T08:06:30.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:30 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:06:30.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:30 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:06:30.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:30 smithi150 ceph-mon[39612]: osd.5 v2:172.21.15.181:6804/2043435508 boot 2024-09-11T08:06:30.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:30 smithi150 ceph-mon[39612]: osdmap e38: 6 total, 6 up, 6 in 2024-09-11T08:06:30.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:30 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-11T08:06:30.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:30 smithi150 ceph-mon[39612]: osdmap e39: 6 total, 6 up, 6 in 2024-09-11T08:06:31.387 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-09-11T08:06:31.473 INFO:teuthology.orchestra.run.smithi181.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-09-11T08:06:31.487 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10+0 records in 2024-09-11T08:06:31.487 INFO:teuthology.orchestra.run.smithi181.stderr:10+0 records out 2024-09-11T08:06:31.487 INFO:teuthology.orchestra.run.smithi181.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0125344 s, 837 MB/s 2024-09-11T08:06:31.487 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping successful for: 2024-09-11T08:06:32.497 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi181:vg_nvme/lv_2 2024-09-11T08:06:32.519 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:32 smithi181 ceph-mon[32701]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:06:32.520 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:32 smithi181 ceph-mon[32701]: osdmap e40: 6 total, 6 up, 6 in 2024-09-11T08:06:32.520 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:32 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:32.520 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:32 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:32.520 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:32 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:06:32.520 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:32 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:06:32.520 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:32 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:32.520 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:32 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:06:32.520 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:32 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:06:32.520 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:32 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:32.735 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:06:32.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:32 smithi150 ceph-mon[39612]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:06:32.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:32 smithi150 ceph-mon[39612]: osdmap e40: 6 total, 6 up, 6 in 2024-09-11T08:06:32.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:32 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:32.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:32 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:32.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:32 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:06:32.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:32 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:06:32.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:32 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:32.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:32 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:06:32.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:32 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:06:32.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:32 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:32.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:32 smithi150 ceph-mon[29276]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:06:32.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:32 smithi150 ceph-mon[29276]: osdmap e40: 6 total, 6 up, 6 in 2024-09-11T08:06:32.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:32 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:32.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:32 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:32.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:32 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:06:32.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:32 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:06:32.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:32 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:32.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:32 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:06:32.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:32 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:06:32.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:32 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:33.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:33 smithi181 ceph-mon[32701]: Detected new or changed devices on smithi181 2024-09-11T08:06:33.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:33 smithi181 ceph-mon[32701]: Adjusting osd_memory_target on smithi181 to 8543M 2024-09-11T08:06:33.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:33 smithi150 ceph-mon[29276]: Detected new or changed devices on smithi181 2024-09-11T08:06:33.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:33 smithi150 ceph-mon[29276]: Adjusting osd_memory_target on smithi181 to 8543M 2024-09-11T08:06:33.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:33 smithi150 ceph-mon[39612]: Detected new or changed devices on smithi181 2024-09-11T08:06:33.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:33 smithi150 ceph-mon[39612]: Adjusting osd_memory_target on smithi181 to 8543M 2024-09-11T08:06:34.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:34 smithi181 ceph-mon[32701]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:06:34.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:34 smithi150 ceph-mon[29276]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:06:34.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:34 smithi150 ceph-mon[39612]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:06:35.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:35 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-11T08:06:35.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:35 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-11T08:06:35.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:35 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:06:35.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:35 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-11T08:06:35.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:35 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-11T08:06:35.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:35 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:06:35.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:35 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-11T08:06:35.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:35 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-11T08:06:35.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:35 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:06:36.441 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:36 smithi181 ceph-mon[32701]: from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:06:36.441 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:36 smithi181 ceph-mon[32701]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:06:36.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:36 smithi150 ceph-mon[29276]: from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:06:36.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:36 smithi150 ceph-mon[29276]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:06:36.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:36 smithi150 ceph-mon[39612]: from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:06:36.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:36 smithi150 ceph-mon[39612]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:06:37.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:37 smithi181 ceph-mon[32701]: from='client.? 172.21.15.181:0/4185536042' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7facca22-b3f1-4c20-833e-e3db798a08c4"}]: dispatch 2024-09-11T08:06:37.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:37 smithi181 ceph-mon[32701]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7facca22-b3f1-4c20-833e-e3db798a08c4"}]: dispatch 2024-09-11T08:06:37.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:37 smithi181 ceph-mon[32701]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7facca22-b3f1-4c20-833e-e3db798a08c4"}]': finished 2024-09-11T08:06:37.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:37 smithi181 ceph-mon[32701]: osdmap e41: 7 total, 6 up, 7 in 2024-09-11T08:06:37.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:37 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-11T08:06:37.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:37 smithi150 ceph-mon[29276]: from='client.? 172.21.15.181:0/4185536042' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7facca22-b3f1-4c20-833e-e3db798a08c4"}]: dispatch 2024-09-11T08:06:37.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:37 smithi150 ceph-mon[29276]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7facca22-b3f1-4c20-833e-e3db798a08c4"}]: dispatch 2024-09-11T08:06:37.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:37 smithi150 ceph-mon[29276]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7facca22-b3f1-4c20-833e-e3db798a08c4"}]': finished 2024-09-11T08:06:37.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:37 smithi150 ceph-mon[29276]: osdmap e41: 7 total, 6 up, 7 in 2024-09-11T08:06:37.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:37 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-11T08:06:37.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:37 smithi150 ceph-mon[39612]: from='client.? 172.21.15.181:0/4185536042' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7facca22-b3f1-4c20-833e-e3db798a08c4"}]: dispatch 2024-09-11T08:06:37.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:37 smithi150 ceph-mon[39612]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7facca22-b3f1-4c20-833e-e3db798a08c4"}]: dispatch 2024-09-11T08:06:37.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:37 smithi150 ceph-mon[39612]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7facca22-b3f1-4c20-833e-e3db798a08c4"}]': finished 2024-09-11T08:06:37.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:37 smithi150 ceph-mon[39612]: osdmap e41: 7 total, 6 up, 7 in 2024-09-11T08:06:37.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:37 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-11T08:06:38.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:38 smithi181 ceph-mon[32701]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:06:38.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:38 smithi181 ceph-mon[32701]: from='client.? 172.21.15.181:0/3583483178' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-11T08:06:38.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:38 smithi150 ceph-mon[29276]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:06:38.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:38 smithi150 ceph-mon[29276]: from='client.? 172.21.15.181:0/3583483178' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-11T08:06:38.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:38 smithi150 ceph-mon[39612]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:06:38.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:38 smithi150 ceph-mon[39612]: from='client.? 172.21.15.181:0/3583483178' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-11T08:06:40.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:40 smithi181 ceph-mon[32701]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:06:40.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:40 smithi150 ceph-mon[29276]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:06:40.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:40 smithi150 ceph-mon[39612]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:06:42.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:42 smithi181 ceph-mon[32701]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:06:42.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:42 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:06:42.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:42 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:06:42.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:42 smithi150 ceph-mon[29276]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:06:42.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:42 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:06:42.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:42 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:06:42.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:42 smithi150 ceph-mon[39612]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:06:42.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:42 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:06:42.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:42 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:06:43.454 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:43 smithi181 ceph-mon[32701]: osdmap e42: 7 total, 6 up, 7 in 2024-09-11T08:06:43.454 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:43 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-11T08:06:43.454 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:43 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-11T08:06:43.454 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:43 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:06:43.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:43 smithi150 ceph-mon[29276]: osdmap e42: 7 total, 6 up, 7 in 2024-09-11T08:06:43.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:43 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-11T08:06:43.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:43 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-11T08:06:43.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:43 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:06:43.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:43 smithi150 ceph-mon[39612]: osdmap e42: 7 total, 6 up, 7 in 2024-09-11T08:06:43.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:43 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-11T08:06:43.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:43 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-11T08:06:43.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:43 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:06:44.400 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:44 smithi181 ceph-mon[32701]: Deploying daemon osd.6 on smithi181 2024-09-11T08:06:44.400 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:44 smithi181 ceph-mon[32701]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:06:44.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:44 smithi150 ceph-mon[29276]: Deploying daemon osd.6 on smithi181 2024-09-11T08:06:44.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:44 smithi150 ceph-mon[29276]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:06:44.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:44 smithi150 ceph-mon[39612]: Deploying daemon osd.6 on smithi181 2024-09-11T08:06:44.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:44 smithi150 ceph-mon[39612]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:06:46.455 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:46 smithi181 ceph-mon[32701]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:06:46.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:46 smithi150 ceph-mon[29276]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:06:46.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:46 smithi150 ceph-mon[39612]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:06:48.570 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:48 smithi181 ceph-mon[32701]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:06:48.570 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:48 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:06:48.570 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:48 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:48.570 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:48 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:48.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:48 smithi150 ceph-mon[29276]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:06:48.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:48 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:06:48.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:48 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:48.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:48 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:48.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:48 smithi150 ceph-mon[39612]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:06:48.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:48 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:06:48.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:48 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:48.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:48 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:50.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:50 smithi181 ceph-mon[32701]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:06:50.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:50 smithi150 ceph-mon[29276]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:06:50.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:50 smithi150 ceph-mon[39612]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:06:50.803 INFO:teuthology.orchestra.run.smithi181.stdout:Created osd(s) 6 on host 'smithi181' 2024-09-11T08:06:51.404 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:51 smithi181 ceph-mon[32701]: from='osd.6 v2:172.21.15.181:6808/1789880331' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-11T08:06:51.405 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:51 smithi181 ceph-mon[32701]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-11T08:06:51.405 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:51 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:51.405 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:51 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:51.431 DEBUG:teuthology.orchestra.run.smithi181:osd.6> sudo journalctl -f -n 0 -u ceph-c1bddff4-7013-11ef-bcea-c7b262605968@osd.6.service 2024-09-11T08:06:51.476 INFO:tasks.cephadm:Deploying osd.7 on smithi181 with /dev/vg_nvme/lv_1... 2024-09-11T08:06:51.476 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-09-11T08:06:51.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:51 smithi150 ceph-mon[29276]: from='osd.6 v2:172.21.15.181:6808/1789880331' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-11T08:06:51.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:51 smithi150 ceph-mon[29276]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-11T08:06:51.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:51 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:51.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:51 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:51.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:51 smithi150 ceph-mon[39612]: from='osd.6 v2:172.21.15.181:6808/1789880331' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-11T08:06:51.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:51 smithi150 ceph-mon[39612]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-11T08:06:51.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:51 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:51.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:51 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:51.769 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:06:52.570 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:52 smithi181 ceph-mon[32701]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:06:52.570 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:52 smithi181 ceph-mon[32701]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-11T08:06:52.571 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:52 smithi181 ceph-mon[32701]: osdmap e43: 7 total, 6 up, 7 in 2024-09-11T08:06:52.571 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:52 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-11T08:06:52.571 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:52 smithi181 ceph-mon[32701]: from='osd.6 v2:172.21.15.181:6808/1789880331' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-09-11T08:06:52.571 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:52 smithi181 ceph-mon[32701]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-09-11T08:06:52.571 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:52 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:52.571 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:52 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:52.571 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:06:52 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:06:52.306+0000 7fa512867640 -1 osd.6 0 waiting for initial osdmap 2024-09-11T08:06:52.571 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:06:52 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:06:52.314+0000 7fa50f6a2640 -1 osd.6 44 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T08:06:52.640 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:52 smithi150 ceph-mon[39612]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:06:52.640 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:52 smithi150 ceph-mon[39612]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-11T08:06:52.641 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:52 smithi150 ceph-mon[39612]: osdmap e43: 7 total, 6 up, 7 in 2024-09-11T08:06:52.641 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:52 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-11T08:06:52.641 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:52 smithi150 ceph-mon[39612]: from='osd.6 v2:172.21.15.181:6808/1789880331' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-09-11T08:06:52.641 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:52 smithi150 ceph-mon[39612]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-09-11T08:06:52.641 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:52 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:52.641 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:52 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:52.641 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:52 smithi150 ceph-mon[29276]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:06:52.641 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:52 smithi150 ceph-mon[29276]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-11T08:06:52.641 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:52 smithi150 ceph-mon[29276]: osdmap e43: 7 total, 6 up, 7 in 2024-09-11T08:06:52.641 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:52 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-11T08:06:52.641 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:52 smithi150 ceph-mon[29276]: from='osd.6 v2:172.21.15.181:6808/1789880331' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-09-11T08:06:52.641 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:52 smithi150 ceph-mon[29276]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-09-11T08:06:52.642 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:52 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:52.642 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:52 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:53.506 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:53 smithi181 ceph-mon[32701]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-09-11T08:06:53.507 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:53 smithi181 ceph-mon[32701]: osdmap e44: 7 total, 6 up, 7 in 2024-09-11T08:06:53.507 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:53 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-11T08:06:53.507 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:53 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-11T08:06:53.507 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:53 smithi181 ceph-mon[32701]: osd.6 v2:172.21.15.181:6808/1789880331 boot 2024-09-11T08:06:53.507 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:53 smithi181 ceph-mon[32701]: osdmap e45: 7 total, 7 up, 7 in 2024-09-11T08:06:53.507 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:53 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-11T08:06:53.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:53 smithi150 ceph-mon[29276]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-09-11T08:06:53.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:53 smithi150 ceph-mon[29276]: osdmap e44: 7 total, 6 up, 7 in 2024-09-11T08:06:53.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:53 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-11T08:06:53.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:53 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-11T08:06:53.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:53 smithi150 ceph-mon[29276]: osd.6 v2:172.21.15.181:6808/1789880331 boot 2024-09-11T08:06:53.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:53 smithi150 ceph-mon[29276]: osdmap e45: 7 total, 7 up, 7 in 2024-09-11T08:06:53.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:53 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-11T08:06:53.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:53 smithi150 ceph-mon[39612]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-09-11T08:06:53.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:53 smithi150 ceph-mon[39612]: osdmap e44: 7 total, 6 up, 7 in 2024-09-11T08:06:53.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:53 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-11T08:06:53.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:53 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-11T08:06:53.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:53 smithi150 ceph-mon[39612]: osd.6 v2:172.21.15.181:6808/1789880331 boot 2024-09-11T08:06:53.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:53 smithi150 ceph-mon[39612]: osdmap e45: 7 total, 7 up, 7 in 2024-09-11T08:06:53.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:53 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-11T08:06:54.571 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:54 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:06:54.571 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:54 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:06:54.571 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:54 smithi181 ceph-mon[32701]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:06:54.571 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:54 smithi181 ceph-mon[32701]: osdmap e46: 7 total, 7 up, 7 in 2024-09-11T08:06:54.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:54 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:06:54.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:54 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:06:54.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:54 smithi150 ceph-mon[29276]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:06:54.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:54 smithi150 ceph-mon[29276]: osdmap e46: 7 total, 7 up, 7 in 2024-09-11T08:06:54.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:54 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:06:54.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:54 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:06:54.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:54 smithi150 ceph-mon[39612]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:06:54.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:54 smithi150 ceph-mon[39612]: osdmap e46: 7 total, 7 up, 7 in 2024-09-11T08:06:55.239 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-09-11T08:06:55.330 INFO:teuthology.orchestra.run.smithi181.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-09-11T08:06:55.343 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10+0 records in 2024-09-11T08:06:55.343 INFO:teuthology.orchestra.run.smithi181.stderr:10+0 records out 2024-09-11T08:06:55.343 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113888 s, 921 MB/s 2024-09-11T08:06:55.343 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping successful for: 2024-09-11T08:06:56.415 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi181:vg_nvme/lv_1 2024-09-11T08:06:56.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:56 smithi181 ceph-mon[32701]: pgmap v149: 1 pgs: 1 remapped+peering; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:06:56.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:56 smithi181 ceph-mon[32701]: osdmap e47: 7 total, 7 up, 7 in 2024-09-11T08:06:56.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:56 smithi181 ceph-mon[32701]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-11T08:06:56.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:56 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:56.575 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:56 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:56.575 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:56 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:06:56.575 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:56 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:06:56.575 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:56 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:06:56.575 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:56 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:56.575 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:56 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:06:56.576 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:56 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:06:56.576 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:56 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:56.576 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:56 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:06:56.576 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:56 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:06:56.576 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:56 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:06:56.576 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:56 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:56.655 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:06:56.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:56 smithi150 ceph-mon[29276]: pgmap v149: 1 pgs: 1 remapped+peering; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:06:56.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:56 smithi150 ceph-mon[29276]: osdmap e47: 7 total, 7 up, 7 in 2024-09-11T08:06:56.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:56 smithi150 ceph-mon[29276]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-11T08:06:56.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:56 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:56.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:56 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:56.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:56 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:06:56.741 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:56 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:06:56.741 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:56 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:06:56.741 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:56 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:56.741 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:56 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:06:56.741 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:56 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:06:56.742 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:56 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:56.742 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:56 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:06:56.742 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:56 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:06:56.742 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:56 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:06:56.742 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:56 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:56.743 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:56 smithi150 ceph-mon[39612]: pgmap v149: 1 pgs: 1 remapped+peering; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:06:56.743 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:56 smithi150 ceph-mon[39612]: osdmap e47: 7 total, 7 up, 7 in 2024-09-11T08:06:56.743 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:56 smithi150 ceph-mon[39612]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-11T08:06:56.743 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:56 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:56.743 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:56 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:56.744 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:56 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:06:56.744 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:56 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:06:56.744 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:56 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:06:56.744 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:56 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:56.744 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:56 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:06:56.745 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:56 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:06:56.745 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:56 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:56.745 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:56 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:06:56.745 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:56 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:06:56.745 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:56 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:06:56.745 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:56 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:06:57.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:57 smithi181 ceph-mon[32701]: Detected new or changed devices on smithi181 2024-09-11T08:06:57.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:57 smithi181 ceph-mon[32701]: Adjusting osd_memory_target on smithi181 to 5695M 2024-09-11T08:06:57.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:57 smithi150 ceph-mon[29276]: Detected new or changed devices on smithi181 2024-09-11T08:06:57.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:57 smithi150 ceph-mon[29276]: Adjusting osd_memory_target on smithi181 to 5695M 2024-09-11T08:06:57.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:57 smithi150 ceph-mon[39612]: Detected new or changed devices on smithi181 2024-09-11T08:06:57.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:57 smithi150 ceph-mon[39612]: Adjusting osd_memory_target on smithi181 to 5695M 2024-09-11T08:06:58.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:58 smithi181 ceph-mon[32701]: pgmap v151: 1 pgs: 1 remapped+peering; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:06:58.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:58 smithi150 ceph-mon[39612]: pgmap v151: 1 pgs: 1 remapped+peering; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:06:58.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:58 smithi150 ceph-mon[29276]: pgmap v151: 1 pgs: 1 remapped+peering; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:06:59.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:59 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-11T08:06:59.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:59 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-11T08:06:59.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:06:59 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:06:59.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:59 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-11T08:06:59.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:59 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-11T08:06:59.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:06:59 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:06:59.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:59 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-11T08:06:59.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:59 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-11T08:06:59.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:06:59 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:07:00.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:00 smithi181 ceph-mon[32701]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:07:00.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:00 smithi181 ceph-mon[32701]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:07:00.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:00 smithi181 ceph-mon[32701]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-11T08:07:00.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:00 smithi181 ceph-mon[32701]: Cluster is now healthy 2024-09-11T08:07:00.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:00 smithi150 ceph-mon[29276]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:07:00.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:00 smithi150 ceph-mon[29276]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:07:00.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:00 smithi150 ceph-mon[29276]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-11T08:07:00.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:00 smithi150 ceph-mon[29276]: Cluster is now healthy 2024-09-11T08:07:00.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:00 smithi150 ceph-mon[39612]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:07:00.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:00 smithi150 ceph-mon[39612]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:07:00.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:00 smithi150 ceph-mon[39612]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-11T08:07:00.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:00 smithi150 ceph-mon[39612]: Cluster is now healthy 2024-09-11T08:07:01.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:01 smithi150 ceph-mon[29276]: from='client.? 172.21.15.181:0/2954967002' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6ac1b06a-79bd-43ca-b91e-fcb0f4033a34"}]: dispatch 2024-09-11T08:07:01.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:01 smithi150 ceph-mon[29276]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6ac1b06a-79bd-43ca-b91e-fcb0f4033a34"}]: dispatch 2024-09-11T08:07:01.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:01 smithi150 ceph-mon[29276]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6ac1b06a-79bd-43ca-b91e-fcb0f4033a34"}]': finished 2024-09-11T08:07:01.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:01 smithi150 ceph-mon[29276]: osdmap e48: 8 total, 7 up, 8 in 2024-09-11T08:07:01.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:01 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-11T08:07:01.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:01 smithi150 ceph-mon[39612]: from='client.? 172.21.15.181:0/2954967002' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6ac1b06a-79bd-43ca-b91e-fcb0f4033a34"}]: dispatch 2024-09-11T08:07:01.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:01 smithi150 ceph-mon[39612]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6ac1b06a-79bd-43ca-b91e-fcb0f4033a34"}]: dispatch 2024-09-11T08:07:01.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:01 smithi150 ceph-mon[39612]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6ac1b06a-79bd-43ca-b91e-fcb0f4033a34"}]': finished 2024-09-11T08:07:01.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:01 smithi150 ceph-mon[39612]: osdmap e48: 8 total, 7 up, 8 in 2024-09-11T08:07:01.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:01 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-11T08:07:01.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:01 smithi181 ceph-mon[32701]: from='client.? 172.21.15.181:0/2954967002' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6ac1b06a-79bd-43ca-b91e-fcb0f4033a34"}]: dispatch 2024-09-11T08:07:01.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:01 smithi181 ceph-mon[32701]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6ac1b06a-79bd-43ca-b91e-fcb0f4033a34"}]: dispatch 2024-09-11T08:07:01.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:01 smithi181 ceph-mon[32701]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6ac1b06a-79bd-43ca-b91e-fcb0f4033a34"}]': finished 2024-09-11T08:07:01.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:01 smithi181 ceph-mon[32701]: osdmap e48: 8 total, 7 up, 8 in 2024-09-11T08:07:01.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:01 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-11T08:07:02.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:02 smithi150 ceph-mon[29276]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:07:02.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:02 smithi150 ceph-mon[29276]: from='client.? 172.21.15.181:0/1677567152' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-11T08:07:02.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:02 smithi150 ceph-mon[39612]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:07:02.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:02 smithi150 ceph-mon[39612]: from='client.? 172.21.15.181:0/1677567152' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-11T08:07:02.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:02 smithi181 ceph-mon[32701]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:07:02.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:02 smithi181 ceph-mon[32701]: from='client.? 172.21.15.181:0/1677567152' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-11T08:07:04.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:04 smithi181 ceph-mon[32701]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-11T08:07:04.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:04 smithi150 ceph-mon[29276]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-11T08:07:04.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:04 smithi150 ceph-mon[39612]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-11T08:07:06.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:06 smithi181 ceph-mon[32701]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-09-11T08:07:06.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:06 smithi150 ceph-mon[29276]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-09-11T08:07:06.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:06 smithi150 ceph-mon[39612]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-09-11T08:07:07.605 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:07 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-11T08:07:07.605 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:07 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:07:07.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:07 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-11T08:07:07.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:07 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:07:07.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:07 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-11T08:07:07.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:07 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:07:08.667 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:08 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:07:08.668 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:08 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:07:08.668 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:08 smithi181 ceph-mon[32701]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-11T08:07:08.668 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:08 smithi181 ceph-mon[32701]: Deploying daemon osd.7 on smithi181 2024-09-11T08:07:08.668 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:08 smithi181 ceph-mon[32701]: osdmap e49: 8 total, 7 up, 8 in 2024-09-11T08:07:08.668 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:08 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-11T08:07:08.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:08 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:07:08.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:08 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:07:08.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:08 smithi150 ceph-mon[29276]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-11T08:07:08.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:08 smithi150 ceph-mon[29276]: Deploying daemon osd.7 on smithi181 2024-09-11T08:07:08.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:08 smithi150 ceph-mon[29276]: osdmap e49: 8 total, 7 up, 8 in 2024-09-11T08:07:08.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:08 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-11T08:07:08.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:08 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:07:08.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:08 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:07:08.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:08 smithi150 ceph-mon[39612]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-11T08:07:08.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:08 smithi150 ceph-mon[39612]: Deploying daemon osd.7 on smithi181 2024-09-11T08:07:08.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:08 smithi150 ceph-mon[39612]: osdmap e49: 8 total, 7 up, 8 in 2024-09-11T08:07:08.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:08 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-11T08:07:10.380 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:10 smithi181 ceph-mon[32701]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:07:10.380 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:10 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:07:10.380 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:10 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:07:10.380 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:10 smithi181 ceph-mon[32701]: osdmap e50: 8 total, 7 up, 8 in 2024-09-11T08:07:10.380 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:10 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-11T08:07:10.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:10 smithi150 ceph-mon[29276]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:07:10.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:10 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:07:10.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:10 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:07:10.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:10 smithi150 ceph-mon[29276]: osdmap e50: 8 total, 7 up, 8 in 2024-09-11T08:07:10.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:10 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-11T08:07:10.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:10 smithi150 ceph-mon[39612]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:07:10.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:10 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:07:10.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:10 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:07:10.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:10 smithi150 ceph-mon[39612]: osdmap e50: 8 total, 7 up, 8 in 2024-09-11T08:07:10.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:10 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-11T08:07:12.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:12 smithi181 ceph-mon[32701]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:07:12.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:12 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:07:12.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:12 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:12.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:12 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:12.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:12 smithi150 ceph-mon[29276]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:07:12.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:12 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:07:12.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:12 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:12.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:12 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:12.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:12 smithi150 ceph-mon[39612]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:07:12.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:12 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:07:12.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:12 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:12.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:12 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:14.432 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:14 smithi181 ceph-mon[32701]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:07:14.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:14 smithi150 ceph-mon[29276]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:07:14.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:14 smithi150 ceph-mon[39612]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:07:15.138 INFO:teuthology.orchestra.run.smithi181.stdout:Created osd(s) 7 on host 'smithi181' 2024-09-11T08:07:15.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:15 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:15.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:15 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:15.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:15 smithi181 ceph-mon[32701]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:07:15.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:15 smithi181 ceph-mon[32701]: from='osd.7 v2:172.21.15.181:6812/3218153424' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-11T08:07:15.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:15 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:15.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:15 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:15.783 DEBUG:teuthology.orchestra.run.smithi181:osd.7> sudo journalctl -f -n 0 -u ceph-c1bddff4-7013-11ef-bcea-c7b262605968@osd.7.service 2024-09-11T08:07:15.785 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-09-11T08:07:15.786 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd stat -f json 2024-09-11T08:07:15.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:15 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:15.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:15 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:15.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:15 smithi150 ceph-mon[39612]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:07:15.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:15 smithi150 ceph-mon[39612]: from='osd.7 v2:172.21.15.181:6812/3218153424' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-11T08:07:15.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:15 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:15.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:15 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:15.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:15 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:15.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:15 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:15.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:15 smithi150 ceph-mon[29276]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:07:15.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:15 smithi150 ceph-mon[29276]: from='osd.7 v2:172.21.15.181:6812/3218153424' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-11T08:07:15.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:15 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:15.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:15 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:16.036 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:07:16.900 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:16 smithi181 ceph-mon[32701]: from='osd.7 v2:172.21.15.181:6812/3218153424' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-11T08:07:16.900 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:16 smithi181 ceph-mon[32701]: osdmap e51: 8 total, 7 up, 8 in 2024-09-11T08:07:16.900 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:16 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-11T08:07:16.900 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:16 smithi181 ceph-mon[32701]: from='osd.7 v2:172.21.15.181:6812/3218153424' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-09-11T08:07:16.900 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:07:16 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[55141]: 2024-09-11T08:07:16.722+0000 7f7b776ab640 -1 osd.7 0 waiting for initial osdmap 2024-09-11T08:07:16.901 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:07:16 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[55141]: 2024-09-11T08:07:16.733+0000 7f7b71ce1640 -1 osd.7 52 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T08:07:16.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:16 smithi150 ceph-mon[29276]: from='osd.7 v2:172.21.15.181:6812/3218153424' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-11T08:07:16.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:16 smithi150 ceph-mon[29276]: osdmap e51: 8 total, 7 up, 8 in 2024-09-11T08:07:16.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:16 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-11T08:07:16.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:16 smithi150 ceph-mon[29276]: from='osd.7 v2:172.21.15.181:6812/3218153424' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-09-11T08:07:16.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:16 smithi150 ceph-mon[39612]: from='osd.7 v2:172.21.15.181:6812/3218153424' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-11T08:07:16.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:16 smithi150 ceph-mon[39612]: osdmap e51: 8 total, 7 up, 8 in 2024-09-11T08:07:16.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:16 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-11T08:07:16.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:16 smithi150 ceph-mon[39612]: from='osd.7 v2:172.21.15.181:6812/3218153424' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-09-11T08:07:17.822 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:07:17.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:17 smithi150 ceph-mon[39612]: from='osd.7 v2:172.21.15.181:6812/3218153424' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-09-11T08:07:17.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:17 smithi150 ceph-mon[39612]: osdmap e52: 8 total, 7 up, 8 in 2024-09-11T08:07:17.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:17 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-11T08:07:17.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:17 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-11T08:07:17.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:17 smithi150 ceph-mon[39612]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:07:17.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:17 smithi150 ceph-mon[39612]: osd.7 v2:172.21.15.181:6812/3218153424 boot 2024-09-11T08:07:17.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:17 smithi150 ceph-mon[39612]: osdmap e53: 8 total, 8 up, 8 in 2024-09-11T08:07:17.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:17 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-11T08:07:17.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:17 smithi150 ceph-mon[29276]: from='osd.7 v2:172.21.15.181:6812/3218153424' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-09-11T08:07:17.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:17 smithi150 ceph-mon[29276]: osdmap e52: 8 total, 7 up, 8 in 2024-09-11T08:07:17.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:17 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-11T08:07:17.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:17 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-11T08:07:17.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:17 smithi150 ceph-mon[29276]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:07:17.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:17 smithi150 ceph-mon[29276]: osd.7 v2:172.21.15.181:6812/3218153424 boot 2024-09-11T08:07:17.991 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:17 smithi150 ceph-mon[29276]: osdmap e53: 8 total, 8 up, 8 in 2024-09-11T08:07:17.991 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:17 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-11T08:07:18.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:17 smithi181 ceph-mon[32701]: from='osd.7 v2:172.21.15.181:6812/3218153424' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-09-11T08:07:18.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:17 smithi181 ceph-mon[32701]: osdmap e52: 8 total, 7 up, 8 in 2024-09-11T08:07:18.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:17 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-11T08:07:18.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:17 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-11T08:07:18.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:17 smithi181 ceph-mon[32701]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:07:18.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:17 smithi181 ceph-mon[32701]: osd.7 v2:172.21.15.181:6812/3218153424 boot 2024-09-11T08:07:18.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:17 smithi181 ceph-mon[32701]: osdmap e53: 8 total, 8 up, 8 in 2024-09-11T08:07:18.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:17 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-11T08:07:18.613 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":53,"num_osds":8,"num_up_osds":8,"osd_up_since":1726042037,"num_in_osds":8,"osd_in_since":1726042020,"num_remapped_pgs":1} 2024-09-11T08:07:18.613 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd dump --format=json 2024-09-11T08:07:18.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:18 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:07:18.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:18 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:07:18.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:18 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/404361861' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-11T08:07:18.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:18 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:18.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:18 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:18.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:18 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:07:18.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:18 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:07:18.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:18 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:07:18.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:18 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:07:18.825 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:18 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:18.825 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:18 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:07:18.825 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:18 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:07:18.825 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:18 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:18.825 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:18 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:07:18.825 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:18 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:07:18.825 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:18 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:07:18.825 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:18 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:18.825 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:18 smithi181 ceph-mon[32701]: osdmap e54: 8 total, 8 up, 8 in 2024-09-11T08:07:18.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:18 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:07:18.843 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:18 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:07:18.843 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:18 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/404361861' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-11T08:07:18.843 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:18 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:18.843 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:18 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:18.843 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:18 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:07:18.843 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:18 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:07:18.843 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:18 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:07:18.843 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:18 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:07:18.843 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:18 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:18.843 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:18 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:07:18.844 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:18 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:07:18.844 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:18 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:18.844 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:18 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:07:18.844 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:18 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:07:18.844 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:18 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:07:18.844 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:18 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:18.844 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:18 smithi150 ceph-mon[29276]: osdmap e54: 8 total, 8 up, 8 in 2024-09-11T08:07:18.845 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:18 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:07:18.845 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:18 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:07:18.845 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:18 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/404361861' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-11T08:07:18.845 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:18 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:18.845 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:18 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:18.845 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:18 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:07:18.845 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:18 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:07:18.845 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:18 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:07:18.845 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:18 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:07:18.845 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:18 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:18.845 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:18 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:07:18.845 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:18 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:07:18.846 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:18 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:18.846 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:18 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:07:18.846 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:18 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:07:18.846 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:18 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:07:18.846 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:18 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:18.846 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:18 smithi150 ceph-mon[39612]: osdmap e54: 8 total, 8 up, 8 in 2024-09-11T08:07:18.875 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:07:19.973 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:19 smithi181 ceph-mon[32701]: Detected new or changed devices on smithi181 2024-09-11T08:07:19.974 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:19 smithi181 ceph-mon[32701]: Adjusting osd_memory_target on smithi181 to 4271M 2024-09-11T08:07:19.974 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:19 smithi181 ceph-mon[32701]: pgmap v169: 1 pgs: 1 remapped+peering; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:07:19.974 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:19 smithi181 ceph-mon[32701]: osdmap e55: 8 total, 8 up, 8 in 2024-09-11T08:07:19.983 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:19 smithi150 ceph-mon[29276]: Detected new or changed devices on smithi181 2024-09-11T08:07:19.983 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:19 smithi150 ceph-mon[29276]: Adjusting osd_memory_target on smithi181 to 4271M 2024-09-11T08:07:19.983 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:19 smithi150 ceph-mon[29276]: pgmap v169: 1 pgs: 1 remapped+peering; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:07:19.983 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:19 smithi150 ceph-mon[29276]: osdmap e55: 8 total, 8 up, 8 in 2024-09-11T08:07:19.983 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:19 smithi150 ceph-mon[39612]: Detected new or changed devices on smithi181 2024-09-11T08:07:19.983 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:19 smithi150 ceph-mon[39612]: Adjusting osd_memory_target on smithi181 to 4271M 2024-09-11T08:07:19.983 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:19 smithi150 ceph-mon[39612]: pgmap v169: 1 pgs: 1 remapped+peering; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:07:19.984 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:19 smithi150 ceph-mon[39612]: osdmap e55: 8 total, 8 up, 8 in 2024-09-11T08:07:20.652 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:07:20.652 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":55,"fsid":"c1bddff4-7013-11ef-bcea-c7b262605968","created":"2024-09-11T08:02:15.344618+0000","modified":"2024-09-11T08:07:19.722078+0000","last_up_change":"2024-09-11T08:07:17.717541+0000","last_in_change":"2024-09-11T08:07:00.805384+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T08:05:21.044729+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"37b63db9-ed2a-4701-b63d-03ff9885cc0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6801","nonce":334255279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":334255279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":334255279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6803","nonce":334255279}]},"public_addr":"172.21.15.150:6801/334255279","cluster_addr":"172.21.15.150:6802/334255279","heartbeat_back_addr":"172.21.15.150:6804/334255279","heartbeat_front_addr":"172.21.15.150:6803/334255279","state":["exists","up"]},{"osd":1,"uuid":"80378442-02bc-48aa-b8ca-6113a4092079","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6805","nonce":1129510619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1129510619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1129510619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6807","nonce":1129510619}]},"public_addr":"172.21.15.150:6805/1129510619","cluster_addr":"172.21.15.150:6806/1129510619","heartbeat_back_addr":"172.21.15.150:6808/1129510619","heartbeat_front_addr":"172.21.15.150:6807/1129510619","state":["exists","up"]},{"osd":2,"uuid":"0955bb92-36a0-4e9a-98e1-b3a5af89f6d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6809","nonce":719975422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":719975422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":719975422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6811","nonce":719975422}]},"public_addr":"172.21.15.150:6809/719975422","cluster_addr":"172.21.15.150:6810/719975422","heartbeat_back_addr":"172.21.15.150:6812/719975422","heartbeat_front_addr":"172.21.15.150:6811/719975422","state":["exists","up"]},{"osd":3,"uuid":"1d04d4d3-62e8-4460-8a07-77f7e5c62929","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6813","nonce":3101032500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3101032500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3101032500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6815","nonce":3101032500}]},"public_addr":"172.21.15.150:6813/3101032500","cluster_addr":"172.21.15.150:6814/3101032500","heartbeat_back_addr":"172.21.15.150:6816/3101032500","heartbeat_front_addr":"172.21.15.150:6815/3101032500","state":["exists","up"]},{"osd":4,"uuid":"c5321a3c-d2f0-4440-8475-2c111d261139","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":2567452825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6801","nonce":2567452825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6803","nonce":2567452825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2567452825}]},"public_addr":"172.21.15.181:6800/2567452825","cluster_addr":"172.21.15.181:6801/2567452825","heartbeat_back_addr":"172.21.15.181:6803/2567452825","heartbeat_front_addr":"172.21.15.181:6802/2567452825","state":["exists","up"]},{"osd":5,"uuid":"1262b2fe-f253-4a26-bfd4-c9dffb697c77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2043435508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6805","nonce":2043435508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6807","nonce":2043435508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2043435508}]},"public_addr":"172.21.15.181:6804/2043435508","cluster_addr":"172.21.15.181:6805/2043435508","heartbeat_back_addr":"172.21.15.181:6807/2043435508","heartbeat_front_addr":"172.21.15.181:6806/2043435508","state":["exists","up"]},{"osd":6,"uuid":"7facca22-b3f1-4c20-833e-e3db798a08c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1789880331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6809","nonce":1789880331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6811","nonce":1789880331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1789880331}]},"public_addr":"172.21.15.181:6808/1789880331","cluster_addr":"172.21.15.181:6809/1789880331","heartbeat_back_addr":"172.21.15.181:6811/1789880331","heartbeat_front_addr":"172.21.15.181:6810/1789880331","state":["exists","up"]},{"osd":7,"uuid":"6ac1b06a-79bd-43ca-b91e-fcb0f4033a34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":3218153424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6813","nonce":3218153424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6815","nonce":3218153424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":3218153424}]},"public_addr":"172.21.15.181:6812/3218153424","cluster_addr":"172.21.15.181:6813/3218153424","heartbeat_back_addr":"172.21.15.181:6815/3218153424","heartbeat_front_addr":"172.21.15.181:6814/3218153424","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:05:53.736951+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:06:18.538693+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:06:41.282222+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:07:09.210029+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:07:06.499614+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:06:28.071419+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:06:51.748571+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:07:15.966133+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.150:0/3271479010":"2024-09-12T08:03:08.740828+0000","172.21.15.150:0/3666733994":"2024-09-12T08:03:08.740828+0000","172.21.15.150:6800/556498589":"2024-09-12T08:03:08.740828+0000","172.21.15.150:0/3673218166":"2024-09-12T08:03:08.740828+0000","172.21.15.150:0/1614413548":"2024-09-12T08:02:45.986151+0000","172.21.15.150:0/2740222408":"2024-09-12T08:02:45.986151+0000","172.21.15.150:6800/603451071":"2024-09-12T08:02:45.986151+0000","172.21.15.150:0/1073581409":"2024-09-12T08:02:45.986151+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T08:07:20.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:20 smithi150 ceph-mon[39612]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-11T08:07:20.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:20 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2029547714' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-11T08:07:20.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:20 smithi150 ceph-mon[29276]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-11T08:07:20.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:20 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2029547714' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-11T08:07:21.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:20 smithi181 ceph-mon[32701]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-11T08:07:21.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:20 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2029547714' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-11T08:07:21.281 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-11T08:05:21.044729+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-09-11T08:07:21.281 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-09-11T08:07:21.524 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:07:21.942 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:21 smithi150 ceph-mon[29276]: pgmap v171: 1 pgs: 1 remapped+peering; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:07:21.943 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:21 smithi150 ceph-mon[39612]: pgmap v171: 1 pgs: 1 remapped+peering; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:07:22.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:21 smithi181 ceph-mon[32701]: pgmap v171: 1 pgs: 1 remapped+peering; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:07:22.782 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:22 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:07:22.782 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:22 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:07:22.782 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:22 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:07:22.782 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:22 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:07:23.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:22 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:07:23.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:22 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:07:23.146 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-09-11T08:07:23.763 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi150 2024-09-11T08:07:23.764 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph orch apply rgw foo.a --placement '1;smithi150=foo.a' 2024-09-11T08:07:23.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:23 smithi150 ceph-mon[29276]: osdmap e56: 8 total, 8 up, 8 in 2024-09-11T08:07:23.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:23 smithi150 ceph-mon[29276]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:07:23.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:23 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1783554831' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-11T08:07:23.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:23 smithi150 ceph-mon[39612]: osdmap e56: 8 total, 8 up, 8 in 2024-09-11T08:07:23.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:23 smithi150 ceph-mon[39612]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:07:23.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:23 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1783554831' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-11T08:07:24.018 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:07:24.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:23 smithi181 ceph-mon[32701]: osdmap e56: 8 total, 8 up, 8 in 2024-09-11T08:07:24.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:23 smithi181 ceph-mon[32701]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:07:24.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:23 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1783554831' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-11T08:07:25.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:24 smithi181 ceph-mon[32701]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-11T08:07:25.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:24 smithi181 ceph-mon[32701]: Cluster is now healthy 2024-09-11T08:07:25.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:24 smithi150 ceph-mon[29276]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-11T08:07:25.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:24 smithi150 ceph-mon[29276]: Cluster is now healthy 2024-09-11T08:07:25.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:24 smithi150 ceph-mon[39612]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-11T08:07:25.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:24 smithi150 ceph-mon[39612]: Cluster is now healthy 2024-09-11T08:07:25.860 INFO:teuthology.orchestra.run.smithi181.stdout:Scheduled rgw.foo.a update... 2024-09-11T08:07:26.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:25 smithi181 ceph-mon[32701]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-09-11T08:07:26.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:25 smithi150 ceph-mon[29276]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-09-11T08:07:26.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:25 smithi150 ceph-mon[39612]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-09-11T08:07:26.537 DEBUG:teuthology.orchestra.run.smithi150:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-c1bddff4-7013-11ef-bcea-c7b262605968@rgw.foo.a.service 2024-09-11T08:07:26.540 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi181 2024-09-11T08:07:26.540 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd pool create datapool 3 3 replicated 2024-09-11T08:07:26.780 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:07:27.134 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:26 smithi181 ceph-mon[32701]: from='client.24328 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi150=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:07:27.134 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:26 smithi181 ceph-mon[32701]: Saving service rgw.foo.a spec with placement smithi150=foo.a;count:1 2024-09-11T08:07:27.134 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:26 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:27.134 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:26 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:07:27.134 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:26 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:07:27.134 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:26 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:07:27.134 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:26 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:27.134 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:26 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-11T08:07:27.134 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:26 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-11T08:07:27.135 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:26 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:27.135 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:26 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:07:27.135 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:26 smithi181 ceph-mon[32701]: Deploying daemon rgw.foo.a on smithi150 2024-09-11T08:07:27.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:26 smithi150 ceph-mon[39612]: from='client.24328 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi150=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:07:27.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:26 smithi150 ceph-mon[39612]: Saving service rgw.foo.a spec with placement smithi150=foo.a;count:1 2024-09-11T08:07:27.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:26 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:27.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:26 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:07:27.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:26 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:07:27.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:26 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:07:27.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:26 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:27.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:26 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-11T08:07:27.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:26 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-11T08:07:27.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:26 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:27.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:26 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:07:27.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:26 smithi150 ceph-mon[39612]: Deploying daemon rgw.foo.a on smithi150 2024-09-11T08:07:27.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:26 smithi150 ceph-mon[29276]: from='client.24328 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi150=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:07:27.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:26 smithi150 ceph-mon[29276]: Saving service rgw.foo.a spec with placement smithi150=foo.a;count:1 2024-09-11T08:07:27.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:26 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:27.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:26 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:07:27.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:26 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:07:27.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:26 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:07:27.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:26 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:27.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:26 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-11T08:07:27.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:26 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-11T08:07:27.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:26 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:27.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:26 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:07:27.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:26 smithi150 ceph-mon[29276]: Deploying daemon rgw.foo.a on smithi150 2024-09-11T08:07:28.181 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:27 smithi181 ceph-mon[32701]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-11T08:07:28.181 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:27 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:07:28.181 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:27 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:07:28.181 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:27 smithi181 ceph-mon[32701]: osdmap e57: 8 total, 8 up, 8 in 2024-09-11T08:07:28.201 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:27 smithi150 ceph-mon[29276]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-11T08:07:28.201 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:27 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:07:28.201 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:27 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:07:28.201 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:27 smithi150 ceph-mon[29276]: osdmap e57: 8 total, 8 up, 8 in 2024-09-11T08:07:28.201 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:27 smithi150 ceph-mon[39612]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-11T08:07:28.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:27 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:07:28.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:27 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:07:28.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:27 smithi150 ceph-mon[39612]: osdmap e57: 8 total, 8 up, 8 in 2024-09-11T08:07:28.202 INFO:journalctl@ceph.rgw.foo.a.smithi150.stdout:Sep 11 08:07:27 smithi150 systemd[1]: Starting Ceph rgw.foo.a for c1bddff4-7013-11ef-bcea-c7b262605968... 2024-09-11T08:07:28.489 INFO:journalctl@ceph.rgw.foo.a.smithi150.stdout:Sep 11 08:07:28 smithi150 podman[65744]: 2024-09-11 08:07:28.102764327 +0000 UTC m=+0.016005466 image pull 4a918172a49766385959227c53e75e2c302e923539324792cb5bee6cad0b2f2b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40 2024-09-11T08:07:28.489 INFO:journalctl@ceph.rgw.foo.a.smithi150.stdout:Sep 11 08:07:28 smithi150 podman[65744]: 2024-09-11 08:07:28.235221606 +0000 UTC m=+0.148462733 container create cee87a4b3901c6799e0a4c06b06fe4118900b377c6babeb96466d1b890c3d417 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-rgw-foo-a, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828) 2024-09-11T08:07:28.989 INFO:journalctl@ceph.rgw.foo.a.smithi150.stdout:Sep 11 08:07:28 smithi150 podman[65744]: 2024-09-11 08:07:28.508395161 +0000 UTC m=+0.421636292 container init cee87a4b3901c6799e0a4c06b06fe4118900b377c6babeb96466d1b890c3d417 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-rgw-foo-a, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-11T08:07:28.989 INFO:journalctl@ceph.rgw.foo.a.smithi150.stdout:Sep 11 08:07:28 smithi150 podman[65744]: 2024-09-11 08:07:28.51141336 +0000 UTC m=+0.424654495 container start cee87a4b3901c6799e0a4c06b06fe4118900b377c6babeb96466d1b890c3d417 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-rgw-foo-a, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-11T08:07:28.989 INFO:journalctl@ceph.rgw.foo.a.smithi150.stdout:Sep 11 08:07:28 smithi150 bash[65744]: cee87a4b3901c6799e0a4c06b06fe4118900b377c6babeb96466d1b890c3d417 2024-09-11T08:07:28.989 INFO:journalctl@ceph.rgw.foo.a.smithi150.stdout:Sep 11 08:07:28 smithi150 systemd[1]: Started Ceph rgw.foo.a for c1bddff4-7013-11ef-bcea-c7b262605968. 2024-09-11T08:07:29.034 INFO:teuthology.orchestra.run.smithi181.stderr:pool 'datapool' created 2024-09-11T08:07:29.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:29 smithi181 ceph-mon[32701]: from='client.? 172.21.15.181:0/1185209271' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-11T08:07:29.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:29 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-11T08:07:29.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:29 smithi150 ceph-mon[29276]: from='client.? 172.21.15.181:0/1185209271' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-11T08:07:29.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:29 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-11T08:07:29.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:29 smithi150 ceph-mon[39612]: from='client.? 172.21.15.181:0/1185209271' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-11T08:07:29.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:29 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-11T08:07:29.696 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- rbd pool init datapool 2024-09-11T08:07:29.937 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:07:30.043 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:30 smithi181 ceph-mon[32701]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-11T08:07:30.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:30 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-11T08:07:30.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:30 smithi181 ceph-mon[32701]: osdmap e58: 8 total, 8 up, 8 in 2024-09-11T08:07:30.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:30 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:30.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:30 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1503848706' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-11T08:07:30.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:30 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:30.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:30 smithi181 ceph-mon[32701]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-11T08:07:30.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:30 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:30.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:30 smithi181 ceph-mon[32701]: Saving service rgw.foo.a spec with placement smithi150=foo.a;count:1 2024-09-11T08:07:30.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:30 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:30.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:30 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:30.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:30 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:07:30.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:30 smithi150 ceph-mon[39612]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-11T08:07:30.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:30 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-11T08:07:30.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:30 smithi150 ceph-mon[39612]: osdmap e58: 8 total, 8 up, 8 in 2024-09-11T08:07:30.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:30 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:30.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:30 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1503848706' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-11T08:07:30.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:30 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:30.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:30 smithi150 ceph-mon[39612]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-11T08:07:30.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:30 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:30.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:30 smithi150 ceph-mon[39612]: Saving service rgw.foo.a spec with placement smithi150=foo.a;count:1 2024-09-11T08:07:30.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:30 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:30.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:30 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:30.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:30 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:07:30.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:30 smithi150 ceph-mon[29276]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-11T08:07:30.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:30 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-11T08:07:30.491 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:30 smithi150 ceph-mon[29276]: osdmap e58: 8 total, 8 up, 8 in 2024-09-11T08:07:30.491 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:30 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:30.491 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:30 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1503848706' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-11T08:07:30.491 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:30 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:30.491 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:30 smithi150 ceph-mon[29276]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-11T08:07:30.491 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:30 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:30.491 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:30 smithi150 ceph-mon[29276]: Saving service rgw.foo.a spec with placement smithi150=foo.a;count:1 2024-09-11T08:07:30.491 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:30 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:30.491 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:30 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:30.491 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:30 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:07:31.286 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:31 smithi181 ceph-mon[32701]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-11T08:07:31.286 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:31 smithi181 ceph-mon[32701]: osdmap e59: 8 total, 8 up, 8 in 2024-09-11T08:07:31.286 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:31 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:31.286 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:31 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:31.286 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:31 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:07:31.286 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:31 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:07:31.286 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:31 smithi181 ceph-mon[32701]: osdmap e60: 8 total, 8 up, 8 in 2024-09-11T08:07:31.287 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:31 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1503848706' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-11T08:07:31.287 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:31 smithi181 ceph-mon[32701]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-11T08:07:31.287 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:31 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1444277864' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-11T08:07:31.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:31 smithi150 ceph-mon[29276]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-11T08:07:31.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:31 smithi150 ceph-mon[29276]: osdmap e59: 8 total, 8 up, 8 in 2024-09-11T08:07:31.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:31 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:31.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:31 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:31.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:31 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:07:31.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:31 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:07:31.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:31 smithi150 ceph-mon[29276]: osdmap e60: 8 total, 8 up, 8 in 2024-09-11T08:07:31.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:31 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1503848706' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-11T08:07:31.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:31 smithi150 ceph-mon[29276]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-11T08:07:31.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:31 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1444277864' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-11T08:07:31.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:31 smithi150 ceph-mon[39612]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-11T08:07:31.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:31 smithi150 ceph-mon[39612]: osdmap e59: 8 total, 8 up, 8 in 2024-09-11T08:07:31.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:31 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:31.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:31 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:31.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:31 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:07:31.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:31 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:07:31.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:31 smithi150 ceph-mon[39612]: osdmap e60: 8 total, 8 up, 8 in 2024-09-11T08:07:31.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:31 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1503848706' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-11T08:07:31.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:31 smithi150 ceph-mon[39612]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-11T08:07:31.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:31 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1444277864' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-11T08:07:32.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:32 smithi181 ceph-mon[32701]: Checking dashboard <-> RGW credentials 2024-09-11T08:07:32.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:32 smithi181 ceph-mon[32701]: pgmap v180: 36 pgs: 7 creating+peering, 22 unknown, 7 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:07:32.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:32 smithi181 ceph-mon[32701]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-11T08:07:32.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:32 smithi181 ceph-mon[32701]: from='client.? 172.21.15.181:0/1659971528' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-11T08:07:32.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:32 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-11T08:07:32.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:32 smithi181 ceph-mon[32701]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-11T08:07:32.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:32 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1444277864' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-11T08:07:32.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:32 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-11T08:07:32.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:32 smithi181 ceph-mon[32701]: osdmap e61: 8 total, 8 up, 8 in 2024-09-11T08:07:32.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:32 smithi150 ceph-mon[29276]: Checking dashboard <-> RGW credentials 2024-09-11T08:07:32.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:32 smithi150 ceph-mon[29276]: pgmap v180: 36 pgs: 7 creating+peering, 22 unknown, 7 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:07:32.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:32 smithi150 ceph-mon[29276]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-11T08:07:32.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:32 smithi150 ceph-mon[29276]: from='client.? 172.21.15.181:0/1659971528' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-11T08:07:32.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:32 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-11T08:07:32.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:32 smithi150 ceph-mon[29276]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-11T08:07:32.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:32 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1444277864' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-11T08:07:32.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:32 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-11T08:07:32.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:32 smithi150 ceph-mon[29276]: osdmap e61: 8 total, 8 up, 8 in 2024-09-11T08:07:32.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:32 smithi150 ceph-mon[39612]: Checking dashboard <-> RGW credentials 2024-09-11T08:07:32.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:32 smithi150 ceph-mon[39612]: pgmap v180: 36 pgs: 7 creating+peering, 22 unknown, 7 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:07:32.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:32 smithi150 ceph-mon[39612]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-11T08:07:32.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:32 smithi150 ceph-mon[39612]: from='client.? 172.21.15.181:0/1659971528' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-11T08:07:32.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:32 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-11T08:07:32.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:32 smithi150 ceph-mon[39612]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-11T08:07:32.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:32 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1444277864' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-11T08:07:32.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:32 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-11T08:07:32.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:32 smithi150 ceph-mon[39612]: osdmap e61: 8 total, 8 up, 8 in 2024-09-11T08:07:34.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:34 smithi181 ceph-mon[32701]: pgmap v183: 68 pgs: 16 creating+peering, 26 unknown, 26 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 767 B/s wr, 4 op/s 2024-09-11T08:07:34.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:34 smithi181 ceph-mon[32701]: osdmap e62: 8 total, 8 up, 8 in 2024-09-11T08:07:34.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:34 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1503848706' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-11T08:07:34.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:34 smithi181 ceph-mon[32701]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-11T08:07:34.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:34 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1444277864' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-11T08:07:34.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:34 smithi181 ceph-mon[32701]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-11T08:07:34.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:34 smithi181 ceph-mon[32701]: Cluster is now healthy 2024-09-11T08:07:34.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:34 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:34.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:34 smithi150 ceph-mon[29276]: pgmap v183: 68 pgs: 16 creating+peering, 26 unknown, 26 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 767 B/s wr, 4 op/s 2024-09-11T08:07:34.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:34 smithi150 ceph-mon[29276]: osdmap e62: 8 total, 8 up, 8 in 2024-09-11T08:07:34.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:34 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1503848706' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-11T08:07:34.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:34 smithi150 ceph-mon[29276]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-11T08:07:34.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:34 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1444277864' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-11T08:07:34.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:34 smithi150 ceph-mon[29276]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-11T08:07:34.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:34 smithi150 ceph-mon[29276]: Cluster is now healthy 2024-09-11T08:07:34.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:34 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:34.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:34 smithi150 ceph-mon[39612]: pgmap v183: 68 pgs: 16 creating+peering, 26 unknown, 26 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 767 B/s wr, 4 op/s 2024-09-11T08:07:34.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:34 smithi150 ceph-mon[39612]: osdmap e62: 8 total, 8 up, 8 in 2024-09-11T08:07:34.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:34 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1503848706' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-11T08:07:34.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:34 smithi150 ceph-mon[39612]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-11T08:07:34.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:34 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1444277864' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-11T08:07:34.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:34 smithi150 ceph-mon[39612]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-11T08:07:34.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:34 smithi150 ceph-mon[39612]: Cluster is now healthy 2024-09-11T08:07:34.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:34 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:34.840 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.181 --placement '1;smithi181=iscsi.a' 2024-09-11T08:07:35.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:35 smithi181 ceph-mon[32701]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-11T08:07:35.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:35 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1444277864' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-11T08:07:35.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:35 smithi181 ceph-mon[32701]: osdmap e63: 8 total, 8 up, 8 in 2024-09-11T08:07:35.087 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:07:35.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:35 smithi150 ceph-mon[29276]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-11T08:07:35.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:35 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1444277864' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-11T08:07:35.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:35 smithi150 ceph-mon[29276]: osdmap e63: 8 total, 8 up, 8 in 2024-09-11T08:07:35.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:35 smithi150 ceph-mon[39612]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-11T08:07:35.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:35 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1444277864' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-11T08:07:35.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:35 smithi150 ceph-mon[39612]: osdmap e63: 8 total, 8 up, 8 in 2024-09-11T08:07:36.217 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:36 smithi181 ceph-mon[32701]: pgmap v186: 100 pgs: 16 creating+peering, 29 unknown, 55 active+clean; 578 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-09-11T08:07:36.217 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:36 smithi181 ceph-mon[32701]: osdmap e64: 8 total, 8 up, 8 in 2024-09-11T08:07:36.217 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:36 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1330059493' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-11T08:07:36.217 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:36 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3411048405' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-11T08:07:36.217 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:36 smithi181 ceph-mon[32701]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-11T08:07:36.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:36 smithi150 ceph-mon[29276]: pgmap v186: 100 pgs: 16 creating+peering, 29 unknown, 55 active+clean; 578 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-09-11T08:07:36.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:36 smithi150 ceph-mon[29276]: osdmap e64: 8 total, 8 up, 8 in 2024-09-11T08:07:36.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:36 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1330059493' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-11T08:07:36.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:36 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3411048405' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-11T08:07:36.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:36 smithi150 ceph-mon[29276]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-11T08:07:36.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:36 smithi150 ceph-mon[39612]: pgmap v186: 100 pgs: 16 creating+peering, 29 unknown, 55 active+clean; 578 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-09-11T08:07:36.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:36 smithi150 ceph-mon[39612]: osdmap e64: 8 total, 8 up, 8 in 2024-09-11T08:07:36.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:36 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1330059493' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-11T08:07:36.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:36 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3411048405' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-11T08:07:36.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:36 smithi150 ceph-mon[39612]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-11T08:07:36.891 INFO:teuthology.orchestra.run.smithi181.stdout:Scheduled iscsi.datapool update... 2024-09-11T08:07:37.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:37 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:07:37.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:37 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:07:37.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:37 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3411048405' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-11T08:07:37.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:37 smithi181 ceph-mon[32701]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-11T08:07:37.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:37 smithi181 ceph-mon[32701]: osdmap e65: 8 total, 8 up, 8 in 2024-09-11T08:07:37.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:37 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1330059493' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-11T08:07:37.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:37 smithi181 ceph-mon[32701]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-11T08:07:37.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:37 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3411048405' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-11T08:07:37.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:37 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:37.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:37 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:07:37.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:37 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:07:37.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:37 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3411048405' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-11T08:07:37.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:37 smithi150 ceph-mon[29276]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-11T08:07:37.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:37 smithi150 ceph-mon[29276]: osdmap e65: 8 total, 8 up, 8 in 2024-09-11T08:07:37.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:37 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1330059493' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-11T08:07:37.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:37 smithi150 ceph-mon[29276]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-11T08:07:37.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:37 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3411048405' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-11T08:07:37.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:37 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:37.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:37 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:07:37.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:37 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:07:37.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:37 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3411048405' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-11T08:07:37.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:37 smithi150 ceph-mon[39612]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-11T08:07:37.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:37 smithi150 ceph-mon[39612]: osdmap e65: 8 total, 8 up, 8 in 2024-09-11T08:07:37.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:37 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1330059493' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-11T08:07:37.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:37 smithi150 ceph-mon[39612]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-11T08:07:37.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:37 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3411048405' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-11T08:07:37.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:37 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:37.543 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-09-11T08:07:37.543 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-11T08:07:37.543 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-09-11T08:07:37.571 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-11T08:07:37.571 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-09-11T08:07:37.611 DEBUG:teuthology.orchestra.run.smithi181:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-c1bddff4-7013-11ef-bcea-c7b262605968@iscsi.iscsi.a.service 2024-09-11T08:07:37.654 INFO:tasks.cephadm:Adding prometheus.a on smithi181 2024-09-11T08:07:37.655 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph orch apply prometheus '1;smithi181=a' 2024-09-11T08:07:38.264 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:07:38.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:38 smithi181 ceph-mon[32701]: pgmap v189: 132 pgs: 1 creating+activating, 13 creating+peering, 28 unknown, 90 active+clean; 578 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 767 B/s wr, 1 op/s 2024-09-11T08:07:38.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:38 smithi181 ceph-mon[32701]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.181", "placement": "1;smithi181=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:07:38.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:38 smithi181 ceph-mon[32701]: Saving service iscsi.datapool spec with placement smithi181=iscsi.a;count:1 2024-09-11T08:07:38.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:38 smithi181 ceph-mon[32701]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-11T08:07:38.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:38 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3411048405' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-11T08:07:38.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:38 smithi181 ceph-mon[32701]: osdmap e66: 8 total, 8 up, 8 in 2024-09-11T08:07:38.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:38 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:38.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:38 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:38.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:38 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:38.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:38 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-11T08:07:38.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:38 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-11T08:07:38.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:38 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:07:38.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:38 smithi181 ceph-mon[32701]: Deploying daemon iscsi.iscsi.a on smithi181 2024-09-11T08:07:38.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:38 smithi150 ceph-mon[29276]: pgmap v189: 132 pgs: 1 creating+activating, 13 creating+peering, 28 unknown, 90 active+clean; 578 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 767 B/s wr, 1 op/s 2024-09-11T08:07:38.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:38 smithi150 ceph-mon[29276]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.181", "placement": "1;smithi181=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:07:38.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:38 smithi150 ceph-mon[29276]: Saving service iscsi.datapool spec with placement smithi181=iscsi.a;count:1 2024-09-11T08:07:38.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:38 smithi150 ceph-mon[29276]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-11T08:07:38.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:38 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3411048405' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-11T08:07:38.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:38 smithi150 ceph-mon[29276]: osdmap e66: 8 total, 8 up, 8 in 2024-09-11T08:07:38.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:38 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:38.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:38 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:38.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:38 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:38.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:38 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-11T08:07:38.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:38 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-11T08:07:38.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:38 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:07:38.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:38 smithi150 ceph-mon[29276]: Deploying daemon iscsi.iscsi.a on smithi181 2024-09-11T08:07:38.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:38 smithi150 ceph-mon[39612]: pgmap v189: 132 pgs: 1 creating+activating, 13 creating+peering, 28 unknown, 90 active+clean; 578 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 767 B/s wr, 1 op/s 2024-09-11T08:07:38.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:38 smithi150 ceph-mon[39612]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.181", "placement": "1;smithi181=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:07:38.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:38 smithi150 ceph-mon[39612]: Saving service iscsi.datapool spec with placement smithi181=iscsi.a;count:1 2024-09-11T08:07:38.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:38 smithi150 ceph-mon[39612]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-11T08:07:38.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:38 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3411048405' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-11T08:07:38.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:38 smithi150 ceph-mon[39612]: osdmap e66: 8 total, 8 up, 8 in 2024-09-11T08:07:38.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:38 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:38.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:38 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:38.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:38 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:38.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:38 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-11T08:07:38.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:38 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-11T08:07:38.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:38 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:07:38.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:38 smithi150 ceph-mon[39612]: Deploying daemon iscsi.iscsi.a on smithi181 2024-09-11T08:07:40.434 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:40 smithi181 ceph-mon[32701]: pgmap v191: 132 pgs: 1 creating+activating, 7 creating+peering, 6 unknown, 118 active+clean; 580 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 39 KiB/s rd, 4.4 KiB/s wr, 94 op/s 2024-09-11T08:07:40.486 INFO:teuthology.orchestra.run.smithi181.stdout:Scheduled prometheus update... 2024-09-11T08:07:40.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:40 smithi150 ceph-mon[29276]: pgmap v191: 132 pgs: 1 creating+activating, 7 creating+peering, 6 unknown, 118 active+clean; 580 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 39 KiB/s rd, 4.4 KiB/s wr, 94 op/s 2024-09-11T08:07:40.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:40 smithi150 ceph-mon[39612]: pgmap v191: 132 pgs: 1 creating+activating, 7 creating+peering, 6 unknown, 118 active+clean; 580 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 39 KiB/s rd, 4.4 KiB/s wr, 94 op/s 2024-09-11T08:07:41.551 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:41 smithi181 ceph-mon[32701]: osdmap e67: 8 total, 8 up, 8 in 2024-09-11T08:07:41.551 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:41 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:41.575 DEBUG:teuthology.orchestra.run.smithi181:prometheus.a> sudo journalctl -f -n 0 -u ceph-c1bddff4-7013-11ef-bcea-c7b262605968@prometheus.a.service 2024-09-11T08:07:41.578 INFO:tasks.cephadm:Adding node-exporter.a on smithi150 2024-09-11T08:07:41.578 INFO:tasks.cephadm:Adding node-exporter.b on smithi181 2024-09-11T08:07:41.578 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph orch apply node-exporter '2;smithi150=a;smithi181=b' 2024-09-11T08:07:41.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:41 smithi150 ceph-mon[29276]: osdmap e67: 8 total, 8 up, 8 in 2024-09-11T08:07:41.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:41 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:41.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:41 smithi150 ceph-mon[39612]: osdmap e67: 8 total, 8 up, 8 in 2024-09-11T08:07:41.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:41 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:41.819 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:07:41 smithi181 podman[59933]: 2024-09-11 08:07:41.611249238 +0000 UTC m=+0.016860356 image pull 4a918172a49766385959227c53e75e2c302e923539324792cb5bee6cad0b2f2b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40 2024-09-11T08:07:41.820 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:07:41 smithi181 podman[59933]: 2024-09-11 08:07:41.819189859 +0000 UTC m=+0.224800976 container create ef5f265239d83a21f073a7de0ee3d874eac3c3b9f2eb471ed20ae09b8d09f5da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-11T08:07:42.074 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:07:42 smithi181 podman[59933]: 2024-09-11 08:07:42.012983062 +0000 UTC m=+0.418594180 container init ef5f265239d83a21f073a7de0ee3d874eac3c3b9f2eb471ed20ae09b8d09f5da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828) 2024-09-11T08:07:42.074 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:07:42 smithi181 podman[59933]: 2024-09-11 08:07:42.016177684 +0000 UTC m=+0.421788802 container start ef5f265239d83a21f073a7de0ee3d874eac3c3b9f2eb471ed20ae09b8d09f5da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True) 2024-09-11T08:07:42.146 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:07:42.397 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:42 smithi181 ceph-mon[32701]: from='client.24400 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi181=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:07:42.397 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:42 smithi181 ceph-mon[32701]: Saving service prometheus spec with placement smithi181=a;count:1 2024-09-11T08:07:42.397 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:42 smithi181 ceph-mon[32701]: pgmap v193: 132 pgs: 1 creating+activating, 131 active+clean; 581 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 56 KiB/s rd, 4.9 KiB/s wr, 134 op/s 2024-09-11T08:07:42.397 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:42 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:42.397 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:42 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:42.397 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:42 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:42.397 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:42 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:42.397 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:42 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:07:42.398 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:07:42 smithi181 bash[59933]: ef5f265239d83a21f073a7de0ee3d874eac3c3b9f2eb471ed20ae09b8d09f5da 2024-09-11T08:07:42.398 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:07:42 smithi181 systemd[1]: Started Ceph iscsi.iscsi.a for c1bddff4-7013-11ef-bcea-c7b262605968. 2024-09-11T08:07:42.398 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:07:42 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug Started the configuration object watcher 2024-09-11T08:07:42.398 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:07:42 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug Checking for config object changes every 1s 2024-09-11T08:07:42.398 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:07:42 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug Processing osd blocklist entries for this node 2024-09-11T08:07:42.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:42 smithi150 ceph-mon[39612]: from='client.24400 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi181=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:07:42.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:42 smithi150 ceph-mon[39612]: Saving service prometheus spec with placement smithi181=a;count:1 2024-09-11T08:07:42.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:42 smithi150 ceph-mon[39612]: pgmap v193: 132 pgs: 1 creating+activating, 131 active+clean; 581 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 56 KiB/s rd, 4.9 KiB/s wr, 134 op/s 2024-09-11T08:07:42.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:42 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:42.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:42 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:42.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:42 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:42.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:42 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:42.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:42 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:07:42.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:42 smithi150 ceph-mon[29276]: from='client.24400 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi181=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:07:42.741 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:42 smithi150 ceph-mon[29276]: Saving service prometheus spec with placement smithi181=a;count:1 2024-09-11T08:07:42.741 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:42 smithi150 ceph-mon[29276]: pgmap v193: 132 pgs: 1 creating+activating, 131 active+clean; 581 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 56 KiB/s rd, 4.9 KiB/s wr, 134 op/s 2024-09-11T08:07:42.741 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:42 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:42.741 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:42 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:42.741 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:42 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:42.741 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:42 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:42.741 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:42 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:07:42.824 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:07:42 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug Reading the configuration object to update local LIO configuration 2024-09-11T08:07:42.824 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:07:42 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug Configuration does not have an entry for this host(smithi181.front.sepia.ceph.com) - nothing to define to LIO 2024-09-11T08:07:42.824 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:07:42 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: * Serving Flask app 'rbd-target-api' (lazy loading) 2024-09-11T08:07:42.824 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:07:42 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: * Environment: production 2024-09-11T08:07:42.824 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:07:42 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: WARNING: This is a development server. Do not use it in a production deployment. 2024-09-11T08:07:42.824 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:07:42 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: Use a production WSGI server instead. 2024-09-11T08:07:42.824 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:07:42 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: * Debug mode: off 2024-09-11T08:07:42.825 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:07:42 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug * Running on all addresses. 2024-09-11T08:07:42.825 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:07:42 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: WARNING: This is a development server. Do not use it in a production deployment. 2024-09-11T08:07:42.825 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:07:42 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: * Running on all addresses. 2024-09-11T08:07:42.825 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:07:42 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: WARNING: This is a development server. Do not use it in a production deployment. 2024-09-11T08:07:42.825 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:07:42 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug * Running on http://[fe80::ec4:7aff:fe88:7b9d]:5000/ (Press CTRL+C to quit) 2024-09-11T08:07:42.825 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:07:42 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: * Running on http://[fe80::ec4:7aff:fe88:7b9d]:5000/ (Press CTRL+C to quit) 2024-09-11T08:07:43.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:43 smithi181 ceph-mon[32701]: Checking pool "datapool" exists for service iscsi.datapool 2024-09-11T08:07:43.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:43 smithi181 ceph-mon[32701]: from='client.? 172.21.15.181:0/1971122842' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-11T08:07:43.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:43 smithi150 ceph-mon[39612]: Checking pool "datapool" exists for service iscsi.datapool 2024-09-11T08:07:43.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:43 smithi150 ceph-mon[39612]: from='client.? 172.21.15.181:0/1971122842' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-11T08:07:43.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:43 smithi150 ceph-mon[29276]: Checking pool "datapool" exists for service iscsi.datapool 2024-09-11T08:07:43.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:43 smithi150 ceph-mon[29276]: from='client.? 172.21.15.181:0/1971122842' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-11T08:07:43.935 INFO:teuthology.orchestra.run.smithi181.stdout:Scheduled node-exporter update... 2024-09-11T08:07:44.602 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:44 smithi181 ceph-mon[32701]: pgmap v194: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 69 KiB/s rd, 5.5 KiB/s wr, 168 op/s 2024-09-11T08:07:44.602 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:44 smithi181 ceph-mon[32701]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-11T08:07:44.603 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:44 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:44.603 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:44 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:44.604 DEBUG:teuthology.orchestra.run.smithi150:node-exporter.a> sudo journalctl -f -n 0 -u ceph-c1bddff4-7013-11ef-bcea-c7b262605968@node-exporter.a.service 2024-09-11T08:07:44.608 DEBUG:teuthology.orchestra.run.smithi181:node-exporter.b> sudo journalctl -f -n 0 -u ceph-c1bddff4-7013-11ef-bcea-c7b262605968@node-exporter.b.service 2024-09-11T08:07:44.611 INFO:tasks.cephadm:Adding alertmanager.a on smithi150 2024-09-11T08:07:44.611 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph orch apply alertmanager '1;smithi150=a' 2024-09-11T08:07:44.631 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:44 smithi150 ceph-mon[29276]: pgmap v194: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 69 KiB/s rd, 5.5 KiB/s wr, 168 op/s 2024-09-11T08:07:44.631 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:44 smithi150 ceph-mon[29276]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-11T08:07:44.631 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:44 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:44.631 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:44 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:44.632 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:44 smithi150 ceph-mon[39612]: pgmap v194: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 69 KiB/s rd, 5.5 KiB/s wr, 168 op/s 2024-09-11T08:07:44.632 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:44 smithi150 ceph-mon[39612]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-11T08:07:44.632 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:44 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:44.632 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:44 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:45.072 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:07:45.535 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:45 smithi181 ceph-mon[32701]: from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi150=a;smithi181=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:07:45.535 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:45 smithi181 ceph-mon[32701]: Saving service node-exporter spec with placement smithi150=a;smithi181=b;count:2 2024-09-11T08:07:45.536 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:45 smithi181 ceph-mon[32701]: 1.0 scrub starts 2024-09-11T08:07:45.536 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:45 smithi181 ceph-mon[32701]: 1.0 scrub ok 2024-09-11T08:07:45.536 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:45 smithi181 ceph-mon[32701]: osdmap e68: 8 total, 8 up, 8 in 2024-09-11T08:07:45.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:45 smithi150 ceph-mon[29276]: from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi150=a;smithi181=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:07:45.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:45 smithi150 ceph-mon[29276]: Saving service node-exporter spec with placement smithi150=a;smithi181=b;count:2 2024-09-11T08:07:45.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:45 smithi150 ceph-mon[29276]: 1.0 scrub starts 2024-09-11T08:07:45.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:45 smithi150 ceph-mon[29276]: 1.0 scrub ok 2024-09-11T08:07:45.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:45 smithi150 ceph-mon[29276]: osdmap e68: 8 total, 8 up, 8 in 2024-09-11T08:07:45.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:45 smithi150 ceph-mon[39612]: from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi150=a;smithi181=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:07:45.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:45 smithi150 ceph-mon[39612]: Saving service node-exporter spec with placement smithi150=a;smithi181=b;count:2 2024-09-11T08:07:45.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:45 smithi150 ceph-mon[39612]: 1.0 scrub starts 2024-09-11T08:07:45.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:45 smithi150 ceph-mon[39612]: 1.0 scrub ok 2024-09-11T08:07:45.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:45 smithi150 ceph-mon[39612]: osdmap e68: 8 total, 8 up, 8 in 2024-09-11T08:07:46.437 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:46 smithi181 ceph-mon[32701]: pgmap v195: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 58 KiB/s rd, 4.6 KiB/s wr, 142 op/s 2024-09-11T08:07:46.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:46 smithi150 ceph-mon[29276]: pgmap v195: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 58 KiB/s rd, 4.6 KiB/s wr, 142 op/s 2024-09-11T08:07:46.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:46 smithi150 ceph-mon[39612]: pgmap v195: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 58 KiB/s rd, 4.6 KiB/s wr, 142 op/s 2024-09-11T08:07:46.843 INFO:teuthology.orchestra.run.smithi181.stdout:Scheduled alertmanager update... 2024-09-11T08:07:48.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:47 smithi181 ceph-mon[32701]: pgmap v197: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 35 KiB/s rd, 2.2 KiB/s wr, 86 op/s 2024-09-11T08:07:48.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:47 smithi181 ceph-mon[32701]: from='client.24442 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi150=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:07:48.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:47 smithi181 ceph-mon[32701]: Saving service alertmanager spec with placement smithi150=a;count:1 2024-09-11T08:07:48.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:47 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:48.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:47 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:48.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:47 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:48.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:47 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:07:48.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:47 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:07:48.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:47 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:48.135 DEBUG:teuthology.orchestra.run.smithi150:alertmanager.a> sudo journalctl -f -n 0 -u ceph-c1bddff4-7013-11ef-bcea-c7b262605968@alertmanager.a.service 2024-09-11T08:07:48.138 INFO:tasks.cephadm:Adding grafana.a on smithi181 2024-09-11T08:07:48.138 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph orch apply grafana '1;smithi181=a' 2024-09-11T08:07:48.162 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:47 smithi150 ceph-mon[29276]: pgmap v197: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 35 KiB/s rd, 2.2 KiB/s wr, 86 op/s 2024-09-11T08:07:48.162 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:47 smithi150 ceph-mon[29276]: from='client.24442 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi150=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:07:48.162 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:47 smithi150 ceph-mon[29276]: Saving service alertmanager spec with placement smithi150=a;count:1 2024-09-11T08:07:48.162 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:47 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:48.162 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:47 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:48.162 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:47 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:48.162 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:47 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:07:48.162 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:47 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:07:48.163 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:47 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:48.163 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:47 smithi150 ceph-mon[39612]: pgmap v197: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 35 KiB/s rd, 2.2 KiB/s wr, 86 op/s 2024-09-11T08:07:48.163 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:47 smithi150 ceph-mon[39612]: from='client.24442 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi150=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:07:48.163 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:47 smithi150 ceph-mon[39612]: Saving service alertmanager spec with placement smithi150=a;count:1 2024-09-11T08:07:48.163 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:47 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:48.163 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:47 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:48.163 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:47 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:48.163 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:47 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:07:48.163 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:47 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:07:48.163 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:47 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:48.425 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:07:49.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:48 smithi181 ceph-mon[32701]: Deploying daemon prometheus.a on smithi181 2024-09-11T08:07:49.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:48 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:49.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:48 smithi150 ceph-mon[29276]: Deploying daemon prometheus.a on smithi181 2024-09-11T08:07:49.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:48 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:49.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:48 smithi150 ceph-mon[39612]: Deploying daemon prometheus.a on smithi181 2024-09-11T08:07:49.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:48 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:50.166 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:49 smithi181 ceph-mon[32701]: pgmap v198: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 1.8 KiB/s wr, 69 op/s 2024-09-11T08:07:50.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:49 smithi150 ceph-mon[29276]: pgmap v198: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 1.8 KiB/s wr, 69 op/s 2024-09-11T08:07:50.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:49 smithi150 ceph-mon[39612]: pgmap v198: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 1.8 KiB/s wr, 69 op/s 2024-09-11T08:07:50.855 INFO:teuthology.orchestra.run.smithi181.stdout:Scheduled grafana update... 2024-09-11T08:07:51.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:51 smithi150 ceph-mon[39612]: osdmap e69: 8 total, 8 up, 8 in 2024-09-11T08:07:51.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:51 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:51.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:51 smithi150 ceph-mon[29276]: osdmap e69: 8 total, 8 up, 8 in 2024-09-11T08:07:51.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:51 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:51.753 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:51 smithi181 ceph-mon[32701]: osdmap e69: 8 total, 8 up, 8 in 2024-09-11T08:07:51.753 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:51 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:07:51.777 DEBUG:teuthology.orchestra.run.smithi181:grafana.a> sudo journalctl -f -n 0 -u ceph-c1bddff4-7013-11ef-bcea-c7b262605968@grafana.a.service 2024-09-11T08:07:51.779 INFO:tasks.cephadm:Setting up client nodes... 2024-09-11T08:07:51.780 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-11T08:07:52.033 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:07:52.428 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:52 smithi150 ceph-mon[29276]: pgmap v200: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-11T08:07:52.428 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:52 smithi150 ceph-mon[29276]: from='client.24448 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi181=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:07:52.428 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:52 smithi150 ceph-mon[29276]: Saving service grafana spec with placement smithi181=a;count:1 2024-09-11T08:07:52.428 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:52 smithi150 ceph-mon[39612]: pgmap v200: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-11T08:07:52.428 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:52 smithi150 ceph-mon[39612]: from='client.24448 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi181=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:07:52.428 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:52 smithi150 ceph-mon[39612]: Saving service grafana spec with placement smithi181=a;count:1 2024-09-11T08:07:52.823 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:07:52 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:07:52.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:52 smithi181 ceph-mon[32701]: pgmap v200: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-11T08:07:52.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:52 smithi181 ceph-mon[32701]: from='client.24448 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi181=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:07:52.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:52 smithi181 ceph-mon[32701]: Saving service grafana spec with placement smithi181=a;count:1 2024-09-11T08:07:53.436 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:53 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:07:53.437 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:53 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:07:53.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:53 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:07:54.005 INFO:teuthology.orchestra.run.smithi150.stdout:[client.0] 2024-09-11T08:07:54.005 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQDaT+FmW78LABAAs13C9Mn94x8NOuHt2MGRrw== 2024-09-11T08:07:54.614 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:54 smithi150 ceph-mon[39612]: pgmap v201: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:07:54.614 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:54 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2929535206' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-11T08:07:54.614 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:54 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-11T08:07:54.614 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:54 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-11T08:07:54.614 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:54 smithi150 ceph-mon[29276]: pgmap v201: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:07:54.614 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:54 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2929535206' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-11T08:07:54.614 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:54 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-11T08:07:54.615 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:54 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-11T08:07:54.637 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-11T08:07:54.637 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-11T08:07:54.637 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-11T08:07:54.675 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-11T08:07:54.698 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:54 smithi181 ceph-mon[32701]: pgmap v201: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:07:54.698 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:54 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2929535206' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-11T08:07:54.698 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:54 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-11T08:07:54.699 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:54 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-11T08:07:55.219 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:07:56.700 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:56 smithi181 ceph-mon[32701]: pgmap v202: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-11T08:07:56.700 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:56 smithi181 ceph-mon[32701]: osdmap e70: 8 total, 8 up, 8 in 2024-09-11T08:07:56.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:56 smithi150 ceph-mon[29276]: pgmap v202: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-11T08:07:56.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:56 smithi150 ceph-mon[29276]: osdmap e70: 8 total, 8 up, 8 in 2024-09-11T08:07:56.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:56 smithi150 ceph-mon[39612]: pgmap v202: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-11T08:07:56.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:56 smithi150 ceph-mon[39612]: osdmap e70: 8 total, 8 up, 8 in 2024-09-11T08:07:58.681 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:07:58 smithi181 ceph-mon[32701]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-11T08:07:58.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:07:58 smithi150 ceph-mon[29276]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-11T08:07:58.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:07:58 smithi150 ceph-mon[39612]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-11T08:07:59.497 INFO:teuthology.orchestra.run.smithi181.stdout:[client.1] 2024-09-11T08:07:59.497 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQDfT+FmAy5oHRAALOm0FzKsJLtLqjtaGxtVTA== 2024-09-11T08:08:00.288 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:08:00 smithi181 systemd[1]: Starting Ceph prometheus.a for c1bddff4-7013-11ef-bcea-c7b262605968... 2024-09-11T08:08:00.554 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:00 smithi181 ceph-mon[32701]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:08:00.554 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:00 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:08:00.554 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:00 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:08:00.555 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:00 smithi181 ceph-mon[32701]: osdmap e71: 8 total, 8 up, 8 in 2024-09-11T08:08:00.555 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:00 smithi181 ceph-mon[32701]: from='client.? 172.21.15.181:0/3657554763' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-11T08:08:00.555 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:00 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-11T08:08:00.555 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:00 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-11T08:08:00.555 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:00 smithi181 ceph-mon[32701]: osdmap e72: 8 total, 8 up, 8 in 2024-09-11T08:08:00.581 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-11T08:08:00.581 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-09-11T08:08:00.582 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-11T08:08:00.665 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-11T08:08:00.666 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-11T08:08:00.666 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph mgr dump --format=json 2024-09-11T08:08:00.693 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:00 smithi150 ceph-mon[39612]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:08:00.693 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:00 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:08:00.693 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:00 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:08:00.693 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:00 smithi150 ceph-mon[39612]: osdmap e71: 8 total, 8 up, 8 in 2024-09-11T08:08:00.693 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:00 smithi150 ceph-mon[39612]: from='client.? 172.21.15.181:0/3657554763' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-11T08:08:00.693 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:00 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-11T08:08:00.694 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:00 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-11T08:08:00.694 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:00 smithi150 ceph-mon[39612]: osdmap e72: 8 total, 8 up, 8 in 2024-09-11T08:08:00.694 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:00 smithi150 ceph-mon[29276]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:08:00.694 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:00 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:08:00.694 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:00 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:08:00.694 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:00 smithi150 ceph-mon[29276]: osdmap e71: 8 total, 8 up, 8 in 2024-09-11T08:08:00.694 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:00 smithi150 ceph-mon[29276]: from='client.? 172.21.15.181:0/3657554763' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-11T08:08:00.694 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:00 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-11T08:08:00.695 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:00 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-11T08:08:00.695 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:00 smithi150 ceph-mon[29276]: osdmap e72: 8 total, 8 up, 8 in 2024-09-11T08:08:00.823 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:08:00 smithi181 podman[61593]: 2024-09-11 08:08:00.662816944 +0000 UTC m=+0.018830070 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-09-11T08:08:00.824 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:08:00 smithi181 podman[61593]: 2024-09-11 08:08:00.780015921 +0000 UTC m=+0.136029030 container create d7433e687125daf607eb61db3622623931a796d2032a82b54ae522b4c1670e7e (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-11T08:08:00.925 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:08:01.324 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:08:01 smithi181 podman[61593]: 2024-09-11 08:08:01.024015798 +0000 UTC m=+0.380028906 container init d7433e687125daf607eb61db3622623931a796d2032a82b54ae522b4c1670e7e (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-11T08:08:01.324 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:08:01 smithi181 podman[61593]: 2024-09-11 08:08:01.027105569 +0000 UTC m=+0.383118699 container start d7433e687125daf607eb61db3622623931a796d2032a82b54ae522b4c1670e7e (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-11T08:08:01.324 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:08:01 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[61611]: ts=2024-09-11T08:08:01.050Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-11T08:08:01.324 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:08:01 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[61611]: ts=2024-09-11T08:08:01.050Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-11T08:08:01.325 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:08:01 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[61611]: ts=2024-09-11T08:08:01.050Z caller=main.go:570 level=info host_details="(Linux 5.14.0-505.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024 x86_64 smithi181 (none))" 2024-09-11T08:08:01.325 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:08:01 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[61611]: ts=2024-09-11T08:08:01.050Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-11T08:08:01.325 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:08:01 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[61611]: ts=2024-09-11T08:08:01.050Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-11T08:08:01.325 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:08:01 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[61611]: ts=2024-09-11T08:08:01.051Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-11T08:08:01.325 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:08:01 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[61611]: ts=2024-09-11T08:08:01.052Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-11T08:08:01.325 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:08:01 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[61611]: ts=2024-09-11T08:08:01.053Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-11T08:08:01.325 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:08:01 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[61611]: ts=2024-09-11T08:08:01.053Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-11T08:08:01.325 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:08:01 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[61611]: ts=2024-09-11T08:08:01.055Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-11T08:08:01.325 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:08:01 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[61611]: ts=2024-09-11T08:08:01.055Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.457µs 2024-09-11T08:08:01.325 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:08:01 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[61611]: ts=2024-09-11T08:08:01.055Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-11T08:08:01.325 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:08:01 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[61611]: ts=2024-09-11T08:08:01.055Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-09-11T08:08:01.326 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:08:01 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[61611]: ts=2024-09-11T08:08:01.055Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=16.899µs wal_replay_duration=202.92µs wbl_replay_duration=89ns total_replay_duration=236.778µs 2024-09-11T08:08:01.326 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:08:01 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[61611]: ts=2024-09-11T08:08:01.056Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-11T08:08:01.326 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:08:01 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[61611]: ts=2024-09-11T08:08:01.056Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-11T08:08:01.326 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:08:01 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[61611]: ts=2024-09-11T08:08:01.056Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-11T08:08:01.326 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:08:01 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[61611]: ts=2024-09-11T08:08:01.071Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=14.618571ms db_storage=756ns remote_storage=1.319µs web_handler=281ns query_engine=553ns scrape=399.083µs scrape_sd=16.916µs notify=789ns notify_sd=1.523µs rules=13.978384ms tracing=6.441µs 2024-09-11T08:08:01.326 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:08:01 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[61611]: ts=2024-09-11T08:08:01.071Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-11T08:08:01.326 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:08:01 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[61611]: ts=2024-09-11T08:08:01.071Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-11T08:08:01.326 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:08:01 smithi181 bash[61593]: d7433e687125daf607eb61db3622623931a796d2032a82b54ae522b4c1670e7e 2024-09-11T08:08:01.326 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:08:01 smithi181 systemd[1]: Started Ceph prometheus.a for c1bddff4-7013-11ef-bcea-c7b262605968. 2024-09-11T08:08:02.360 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:02 smithi150 ceph-mon[29276]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:08:02.360 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:02 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:08:02.360 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:02 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:08:02.360 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:02 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:08:02.360 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:02 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-11T08:08:02.361 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:02 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ignoring --setuser ceph since I am not root 2024-09-11T08:08:02.361 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:02 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ignoring --setgroup ceph since I am not root 2024-09-11T08:08:02.361 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:02 smithi150 ceph-mgr[29507]: -- 172.21.15.150:0/682622579 <== mon.2 v2:172.21.15.150:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f680554340 con 0x55f680560800 2024-09-11T08:08:02.361 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:02 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:08:02.304+0000 7fb0de1d2ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-11T08:08:02.361 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:02 smithi150 ceph-mon[39612]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:08:02.361 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:02 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:08:02.361 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:02 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:08:02.361 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:02 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:08:02.361 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:02 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-11T08:08:02.426 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:02 smithi181 ceph-mon[32701]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:08:02.427 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:02 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:08:02.427 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:02 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:08:02.427 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:02 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' 2024-09-11T08:08:02.427 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:02 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-11T08:08:02.427 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:08:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ignoring --setuser ceph since I am not root 2024-09-11T08:08:02.427 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:08:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ignoring --setgroup ceph since I am not root 2024-09-11T08:08:02.427 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:08:02 smithi181 ceph-mgr[34564]: -- 172.21.15.181:0/3186561394 <== mon.1 v2:172.21.15.181:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e61f850340 con 0x55e61f85d000 2024-09-11T08:08:02.427 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:08:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: 2024-09-11T08:08:02.304+0000 7ff68fbe9ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-11T08:08:02.427 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:08:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: 2024-09-11T08:08:02.359+0000 7ff68fbe9ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-11T08:08:02.427 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:08:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: 2024-09-11T08:08:02.424+0000 7ff68fbe9ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-11T08:08:02.682 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:02 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:08:02.359+0000 7fb0de1d2ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-11T08:08:02.682 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:02 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:08:02.428+0000 7fb0de1d2ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-11T08:08:02.683 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:02 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:08:02.555+0000 7fb0de1d2ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-11T08:08:02.683 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:08:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: 2024-09-11T08:08:02.547+0000 7ff68fbe9ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-11T08:08:02.892 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:08:02.937 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:08:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: 2024-09-11T08:08:02.681+0000 7ff68fbe9ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-11T08:08:02.962 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:02 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:08:02.696+0000 7fb0de1d2ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-11T08:08:02.962 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:02 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:08:02.961+0000 7fb0de1d2ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-11T08:08:03.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:03 smithi150 ceph-mon[29276]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-11T08:08:03.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:03 smithi150 ceph-mon[29276]: mgrmap e17: y(active, since 4m), standbys: x 2024-09-11T08:08:03.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:03 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3587559547' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-11T08:08:03.240 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:03 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:08:03.021+0000 7fb0de1d2ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-11T08:08:03.240 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:03 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:08:03.187+0000 7fb0de1d2ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-11T08:08:03.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:03 smithi150 ceph-mon[39612]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-11T08:08:03.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:03 smithi150 ceph-mon[39612]: mgrmap e17: y(active, since 4m), standbys: x 2024-09-11T08:08:03.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:03 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3587559547' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-11T08:08:03.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:03 smithi181 ceph-mon[32701]: from='mgr.14150 172.21.15.150:0/1396000645' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-11T08:08:03.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:03 smithi181 ceph-mon[32701]: mgrmap e17: y(active, since 4m), standbys: x 2024-09-11T08:08:03.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:03 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3587559547' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-11T08:08:03.324 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:08:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: 2024-09-11T08:08:02.935+0000 7ff68fbe9ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-11T08:08:03.324 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:08:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: 2024-09-11T08:08:02.995+0000 7ff68fbe9ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-11T08:08:03.324 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:08:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: 2024-09-11T08:08:03.155+0000 7ff68fbe9ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-11T08:08:03.510 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":17,"flags":0,"active_gid":14150,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":127856313}]},"active_addr":"172.21.15.150:6800/127856313","active_change":"2024-09-11T08:03:08.740961+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":24115,"name":"x","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.150:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.150:0","nonce":2539412006}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.150:0","nonce":3185731090}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.150:0","nonce":2620931070}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.150:0","nonce":2363649697}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.150:0","nonce":2375121603}]}]} 2024-09-11T08:08:03.515 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-11T08:08:03.515 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-11T08:08:03.515 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd dump --format=json 2024-09-11T08:08:03.808 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:08:04.181 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:04 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:08:04.070+0000 7fb0de1d2ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-11T08:08:04.181 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:04 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:08:04.141+0000 7fb0de1d2ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-11T08:08:04.323 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:08:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: 2024-09-11T08:08:03.996+0000 7ff68fbe9ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-11T08:08:04.324 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:08:04 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: 2024-09-11T08:08:04.062+0000 7ff68fbe9ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-11T08:08:05.047 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:08:04 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: 2024-09-11T08:08:04.760+0000 7ff68fbe9ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-11T08:08:05.048 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:08:04 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: 2024-09-11T08:08:04.870+0000 7ff68fbe9ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-11T08:08:05.048 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:08:04 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: 2024-09-11T08:08:04.988+0000 7ff68fbe9ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-11T08:08:05.151 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:04 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:08:04.852+0000 7fb0de1d2ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-11T08:08:05.151 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:04 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:08:04.965+0000 7fb0de1d2ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-11T08:08:05.152 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:05 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:08:05.089+0000 7fb0de1d2ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-11T08:08:05.152 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:05 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:08:05.150+0000 7fb0de1d2ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-11T08:08:05.323 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:08:05 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: 2024-09-11T08:08:05.045+0000 7ff68fbe9ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-11T08:08:05.324 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:08:05 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: 2024-09-11T08:08:05.170+0000 7ff68fbe9ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-11T08:08:05.324 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:08:05 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: 2024-09-11T08:08:05.237+0000 7ff68fbe9ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-11T08:08:05.324 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:08:05 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: 2024-09-11T08:08:05.294+0000 7ff68fbe9ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-11T08:08:05.409 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:05 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:08:05.278+0000 7fb0de1d2ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-11T08:08:05.410 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:05 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:08:05.349+0000 7fb0de1d2ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-11T08:08:05.410 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:05 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:08:05.408+0000 7fb0de1d2ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-11T08:08:05.681 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:05 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:08:05.548+0000 7fb0de1d2ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-11T08:08:05.681 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:05 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:08:05.614+0000 7fb0de1d2ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-11T08:08:05.681 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:05 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:08:05.679+0000 7fb0de1d2ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-11T08:08:05.693 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:08:05 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: 2024-09-11T08:08:05.428+0000 7ff68fbe9ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-11T08:08:05.694 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:08:05 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: 2024-09-11T08:08:05.490+0000 7ff68fbe9ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-11T08:08:05.694 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:08:05 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: 2024-09-11T08:08:05.555+0000 7ff68fbe9ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-11T08:08:05.694 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:08:05 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: 2024-09-11T08:08:05.635+0000 7ff68fbe9ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-11T08:08:05.694 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:08:05 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: 2024-09-11T08:08:05.692+0000 7ff68fbe9ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-11T08:08:05.717 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:08:05.717 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":72,"fsid":"c1bddff4-7013-11ef-bcea-c7b262605968","created":"2024-09-11T08:02:15.344618+0000","modified":"2024-09-11T08:08:00.342990+0000","last_up_change":"2024-09-11T08:07:17.717541+0000","last_in_change":"2024-09-11T08:07:00.805384+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T08:05:21.044729+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"datapool","create_time":"2024-09-11T08:07:28.786572+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":63,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-09-11T08:07:29.021286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-11T08:07:30.334181+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-11T08:07:32.289626+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-11T08:07:34.351861+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"37b63db9-ed2a-4701-b63d-03ff9885cc0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6801","nonce":334255279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":334255279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":334255279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6803","nonce":334255279}]},"public_addr":"172.21.15.150:6801/334255279","cluster_addr":"172.21.15.150:6802/334255279","heartbeat_back_addr":"172.21.15.150:6804/334255279","heartbeat_front_addr":"172.21.15.150:6803/334255279","state":["exists","up"]},{"osd":1,"uuid":"80378442-02bc-48aa-b8ca-6113a4092079","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6805","nonce":1129510619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1129510619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1129510619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6807","nonce":1129510619}]},"public_addr":"172.21.15.150:6805/1129510619","cluster_addr":"172.21.15.150:6806/1129510619","heartbeat_back_addr":"172.21.15.150:6808/1129510619","heartbeat_front_addr":"172.21.15.150:6807/1129510619","state":["exists","up"]},{"osd":2,"uuid":"0955bb92-36a0-4e9a-98e1-b3a5af89f6d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6809","nonce":719975422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":719975422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":719975422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6811","nonce":719975422}]},"public_addr":"172.21.15.150:6809/719975422","cluster_addr":"172.21.15.150:6810/719975422","heartbeat_back_addr":"172.21.15.150:6812/719975422","heartbeat_front_addr":"172.21.15.150:6811/719975422","state":["exists","up"]},{"osd":3,"uuid":"1d04d4d3-62e8-4460-8a07-77f7e5c62929","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6813","nonce":3101032500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3101032500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3101032500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6815","nonce":3101032500}]},"public_addr":"172.21.15.150:6813/3101032500","cluster_addr":"172.21.15.150:6814/3101032500","heartbeat_back_addr":"172.21.15.150:6816/3101032500","heartbeat_front_addr":"172.21.15.150:6815/3101032500","state":["exists","up"]},{"osd":4,"uuid":"c5321a3c-d2f0-4440-8475-2c111d261139","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":2567452825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6801","nonce":2567452825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6803","nonce":2567452825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2567452825}]},"public_addr":"172.21.15.181:6800/2567452825","cluster_addr":"172.21.15.181:6801/2567452825","heartbeat_back_addr":"172.21.15.181:6803/2567452825","heartbeat_front_addr":"172.21.15.181:6802/2567452825","state":["exists","up"]},{"osd":5,"uuid":"1262b2fe-f253-4a26-bfd4-c9dffb697c77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2043435508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6805","nonce":2043435508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6807","nonce":2043435508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2043435508}]},"public_addr":"172.21.15.181:6804/2043435508","cluster_addr":"172.21.15.181:6805/2043435508","heartbeat_back_addr":"172.21.15.181:6807/2043435508","heartbeat_front_addr":"172.21.15.181:6806/2043435508","state":["exists","up"]},{"osd":6,"uuid":"7facca22-b3f1-4c20-833e-e3db798a08c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1789880331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6809","nonce":1789880331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6811","nonce":1789880331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1789880331}]},"public_addr":"172.21.15.181:6808/1789880331","cluster_addr":"172.21.15.181:6809/1789880331","heartbeat_back_addr":"172.21.15.181:6811/1789880331","heartbeat_front_addr":"172.21.15.181:6810/1789880331","state":["exists","up"]},{"osd":7,"uuid":"6ac1b06a-79bd-43ca-b91e-fcb0f4033a34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":3218153424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6813","nonce":3218153424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6815","nonce":3218153424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":3218153424}]},"public_addr":"172.21.15.181:6812/3218153424","cluster_addr":"172.21.15.181:6813/3218153424","heartbeat_back_addr":"172.21.15.181:6815/3218153424","heartbeat_front_addr":"172.21.15.181:6814/3218153424","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:07:22.028944+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:07:26.819072+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:07:59.344221+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:07:09.210029+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:07:06.499614+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:07:35.633359+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:06:51.748571+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:07:15.966133+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.150:0/3271479010":"2024-09-12T08:03:08.740828+0000","172.21.15.150:0/3666733994":"2024-09-12T08:03:08.740828+0000","172.21.15.150:6800/556498589":"2024-09-12T08:03:08.740828+0000","172.21.15.150:0/3673218166":"2024-09-12T08:03:08.740828+0000","172.21.15.150:0/1614413548":"2024-09-12T08:02:45.986151+0000","172.21.15.150:0/2740222408":"2024-09-12T08:02:45.986151+0000","172.21.15.150:6800/603451071":"2024-09-12T08:02:45.986151+0000","172.21.15.150:0/1073581409":"2024-09-12T08:02:45.986151+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T08:08:05.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:05 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3981271065' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-11T08:08:05.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:05 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3981271065' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-11T08:08:05.989 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:05 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:08:05.772+0000 7fb0de1d2ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-11T08:08:05.989 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:05 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:08:05.830+0000 7fb0de1d2ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-11T08:08:05.989 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:05 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:08:05.887+0000 7fb0de1d2ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-11T08:08:06.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:05 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3981271065' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-11T08:08:06.074 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:08:05 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: 2024-09-11T08:08:05.748+0000 7ff68fbe9ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-11T08:08:06.492 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-11T08:08:06.492 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd dump --format=json 2024-09-11T08:08:06.719 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:08:06.647+0000 7fb0de1d2ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-11T08:08:06.719 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: [11/Sep/2024:08:08:06] ENGINE Bus STARTING 2024-09-11T08:08:06.770 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:08:06 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: 2024-09-11T08:08:06.493+0000 7ff68fbe9ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-11T08:08:06.770 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:08:06 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: [11/Sep/2024:08:08:06] ENGINE Bus STARTING 2024-09-11T08:08:06.770 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:08:06 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: CherryPy Checker: 2024-09-11T08:08:06.770 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:08:06 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: The Application mounted at '' has an empty config. 2024-09-11T08:08:06.770 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:08:06 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: 2024-09-11T08:08:06.771 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:08:06 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: [11/Sep/2024:08:08:06] ENGINE Serving on http://:::9283 2024-09-11T08:08:06.771 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:08:06 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: [11/Sep/2024:08:08:06] ENGINE Bus STARTED 2024-09-11T08:08:06.774 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:08:06.989 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: CherryPy Checker: 2024-09-11T08:08:06.989 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: The Application mounted at '' has an empty config. 2024-09-11T08:08:06.989 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:08:06.989 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: [11/Sep/2024:08:08:06] ENGINE Serving on http://:::9283 2024-09-11T08:08:06.989 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: [11/Sep/2024:08:08:06] ENGINE Bus STARTED 2024-09-11T08:08:06.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[39612]: Standby manager daemon x restarted 2024-09-11T08:08:06.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[39612]: Standby manager daemon x started 2024-09-11T08:08:06.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[39612]: from='mgr.? 172.21.15.181:0/1016928843' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-11T08:08:06.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[39612]: from='mgr.? 172.21.15.181:0/1016928843' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-11T08:08:06.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[39612]: from='mgr.? 172.21.15.181:0/1016928843' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-11T08:08:06.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[39612]: from='mgr.? 172.21.15.181:0/1016928843' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-11T08:08:06.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[39612]: Active manager daemon y restarted 2024-09-11T08:08:06.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[39612]: Activating manager daemon y 2024-09-11T08:08:06.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[39612]: osdmap e73: 8 total, 8 up, 8 in 2024-09-11T08:08:06.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[39612]: mgrmap e18: y(active, starting, since 0.0101297s), standbys: x 2024-09-11T08:08:06.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-11T08:08:06.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:08:06.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:08:06.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-11T08:08:06.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-11T08:08:06.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-11T08:08:06.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-11T08:08:06.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-11T08:08:06.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-11T08:08:06.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-11T08:08:06.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-11T08:08:06.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-11T08:08:06.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-11T08:08:06.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-11T08:08:06.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-11T08:08:06.992 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-11T08:08:06.992 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[39612]: Manager daemon y is now available 2024-09-11T08:08:06.992 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:08:06.992 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:06.992 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:08:06.992 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-11T08:08:06.992 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-11T08:08:06.992 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[29276]: Standby manager daemon x restarted 2024-09-11T08:08:06.992 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[29276]: Standby manager daemon x started 2024-09-11T08:08:06.993 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[29276]: from='mgr.? 172.21.15.181:0/1016928843' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-11T08:08:06.993 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[29276]: from='mgr.? 172.21.15.181:0/1016928843' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-11T08:08:06.993 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[29276]: from='mgr.? 172.21.15.181:0/1016928843' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-11T08:08:06.993 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[29276]: from='mgr.? 172.21.15.181:0/1016928843' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-11T08:08:06.993 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[29276]: Active manager daemon y restarted 2024-09-11T08:08:06.993 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[29276]: Activating manager daemon y 2024-09-11T08:08:06.993 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[29276]: osdmap e73: 8 total, 8 up, 8 in 2024-09-11T08:08:06.993 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[29276]: mgrmap e18: y(active, starting, since 0.0101297s), standbys: x 2024-09-11T08:08:06.993 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-11T08:08:06.993 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:08:06.993 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:08:06.993 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-11T08:08:06.993 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-11T08:08:06.993 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-11T08:08:06.994 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-11T08:08:06.994 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-11T08:08:06.994 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-11T08:08:06.994 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-11T08:08:06.994 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-11T08:08:06.994 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-11T08:08:06.994 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-11T08:08:06.994 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-11T08:08:06.994 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-11T08:08:06.994 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-11T08:08:06.994 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[29276]: Manager daemon y is now available 2024-09-11T08:08:06.994 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:08:06.994 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:06.995 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:08:06.995 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-11T08:08:06.995 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:06 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-11T08:08:07.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:06 smithi181 ceph-mon[32701]: Standby manager daemon x restarted 2024-09-11T08:08:07.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:06 smithi181 ceph-mon[32701]: Standby manager daemon x started 2024-09-11T08:08:07.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:06 smithi181 ceph-mon[32701]: from='mgr.? 172.21.15.181:0/1016928843' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-11T08:08:07.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:06 smithi181 ceph-mon[32701]: from='mgr.? 172.21.15.181:0/1016928843' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-11T08:08:07.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:06 smithi181 ceph-mon[32701]: from='mgr.? 172.21.15.181:0/1016928843' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-11T08:08:07.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:06 smithi181 ceph-mon[32701]: from='mgr.? 172.21.15.181:0/1016928843' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-11T08:08:07.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:06 smithi181 ceph-mon[32701]: Active manager daemon y restarted 2024-09-11T08:08:07.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:06 smithi181 ceph-mon[32701]: Activating manager daemon y 2024-09-11T08:08:07.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:06 smithi181 ceph-mon[32701]: osdmap e73: 8 total, 8 up, 8 in 2024-09-11T08:08:07.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:06 smithi181 ceph-mon[32701]: mgrmap e18: y(active, starting, since 0.0101297s), standbys: x 2024-09-11T08:08:07.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:06 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-11T08:08:07.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:06 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:08:07.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:06 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:08:07.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:06 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-11T08:08:07.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:06 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-11T08:08:07.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:06 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-11T08:08:07.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:06 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-11T08:08:07.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:06 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-11T08:08:07.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:06 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-11T08:08:07.076 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:06 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-11T08:08:07.076 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:06 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-11T08:08:07.076 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:06 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-11T08:08:07.076 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:06 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-11T08:08:07.076 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:06 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-11T08:08:07.076 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:06 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-11T08:08:07.076 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:06 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-11T08:08:07.076 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:06 smithi181 ceph-mon[32701]: Manager daemon y is now available 2024-09-11T08:08:07.076 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:06 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:08:07.076 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:06 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:07.076 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:06 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:08:07.076 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:06 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-11T08:08:07.076 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:06 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-11T08:08:08.073 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:08:07 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:08:08.536 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:08 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:08:08.273+0000 7fb080f54640 -1 log_channel(cephadm) log [ERR] : [11/Sep/2024:08:08:08] ENGINE Error in HTTPServer.serve 2024-09-11T08:08:08.536 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:08 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: Traceback (most recent call last): 2024-09-11T08:08:08.536 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:08 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-11T08:08:08.536 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:08 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: self._connections.run(self.expiration_interval) 2024-09-11T08:08:08.536 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:08 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-11T08:08:08.536 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:08 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: self._run(expiration_interval) 2024-09-11T08:08:08.536 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:08 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-11T08:08:08.536 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:08 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: new_conn = self._from_server_socket(self.server.socket) 2024-09-11T08:08:08.536 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:08 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-11T08:08:08.536 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:08 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-11T08:08:08.537 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:08 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-11T08:08:08.537 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:08 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: s = self.context.wrap_socket( 2024-09-11T08:08:08.537 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:08 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-11T08:08:08.537 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:08 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: return self.sslsocket_class._create( 2024-09-11T08:08:08.537 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:08 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-11T08:08:08.537 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:08 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: self.do_handshake() 2024-09-11T08:08:08.537 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:08 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-11T08:08:08.537 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:08 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: self._sslobj.do_handshake() 2024-09-11T08:08:08.537 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:08 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-11T08:08:08.537 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:08 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:08:08.692 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:08:08.693 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":73,"fsid":"c1bddff4-7013-11ef-bcea-c7b262605968","created":"2024-09-11T08:02:15.344618+0000","modified":"2024-09-11T08:08:06.649729+0000","last_up_change":"2024-09-11T08:07:17.717541+0000","last_in_change":"2024-09-11T08:07:00.805384+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T08:05:21.044729+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"datapool","create_time":"2024-09-11T08:07:28.786572+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":63,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-09-11T08:07:29.021286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-11T08:07:30.334181+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-11T08:07:32.289626+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-11T08:07:34.351861+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"37b63db9-ed2a-4701-b63d-03ff9885cc0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6801","nonce":334255279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":334255279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":334255279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6803","nonce":334255279}]},"public_addr":"172.21.15.150:6801/334255279","cluster_addr":"172.21.15.150:6802/334255279","heartbeat_back_addr":"172.21.15.150:6804/334255279","heartbeat_front_addr":"172.21.15.150:6803/334255279","state":["exists","up"]},{"osd":1,"uuid":"80378442-02bc-48aa-b8ca-6113a4092079","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6805","nonce":1129510619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1129510619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1129510619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6807","nonce":1129510619}]},"public_addr":"172.21.15.150:6805/1129510619","cluster_addr":"172.21.15.150:6806/1129510619","heartbeat_back_addr":"172.21.15.150:6808/1129510619","heartbeat_front_addr":"172.21.15.150:6807/1129510619","state":["exists","up"]},{"osd":2,"uuid":"0955bb92-36a0-4e9a-98e1-b3a5af89f6d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6809","nonce":719975422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":719975422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":719975422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6811","nonce":719975422}]},"public_addr":"172.21.15.150:6809/719975422","cluster_addr":"172.21.15.150:6810/719975422","heartbeat_back_addr":"172.21.15.150:6812/719975422","heartbeat_front_addr":"172.21.15.150:6811/719975422","state":["exists","up"]},{"osd":3,"uuid":"1d04d4d3-62e8-4460-8a07-77f7e5c62929","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6813","nonce":3101032500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3101032500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3101032500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6815","nonce":3101032500}]},"public_addr":"172.21.15.150:6813/3101032500","cluster_addr":"172.21.15.150:6814/3101032500","heartbeat_back_addr":"172.21.15.150:6816/3101032500","heartbeat_front_addr":"172.21.15.150:6815/3101032500","state":["exists","up"]},{"osd":4,"uuid":"c5321a3c-d2f0-4440-8475-2c111d261139","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":2567452825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6801","nonce":2567452825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6803","nonce":2567452825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2567452825}]},"public_addr":"172.21.15.181:6800/2567452825","cluster_addr":"172.21.15.181:6801/2567452825","heartbeat_back_addr":"172.21.15.181:6803/2567452825","heartbeat_front_addr":"172.21.15.181:6802/2567452825","state":["exists","up"]},{"osd":5,"uuid":"1262b2fe-f253-4a26-bfd4-c9dffb697c77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2043435508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6805","nonce":2043435508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6807","nonce":2043435508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2043435508}]},"public_addr":"172.21.15.181:6804/2043435508","cluster_addr":"172.21.15.181:6805/2043435508","heartbeat_back_addr":"172.21.15.181:6807/2043435508","heartbeat_front_addr":"172.21.15.181:6806/2043435508","state":["exists","up"]},{"osd":6,"uuid":"7facca22-b3f1-4c20-833e-e3db798a08c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1789880331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6809","nonce":1789880331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6811","nonce":1789880331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1789880331}]},"public_addr":"172.21.15.181:6808/1789880331","cluster_addr":"172.21.15.181:6809/1789880331","heartbeat_back_addr":"172.21.15.181:6811/1789880331","heartbeat_front_addr":"172.21.15.181:6810/1789880331","state":["exists","up"]},{"osd":7,"uuid":"6ac1b06a-79bd-43ca-b91e-fcb0f4033a34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":3218153424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6813","nonce":3218153424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6815","nonce":3218153424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":3218153424}]},"public_addr":"172.21.15.181:6812/3218153424","cluster_addr":"172.21.15.181:6813/3218153424","heartbeat_back_addr":"172.21.15.181:6815/3218153424","heartbeat_front_addr":"172.21.15.181:6814/3218153424","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:07:22.028944+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:07:26.819072+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:07:59.344221+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:07:09.210029+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:07:06.499614+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:07:35.633359+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:06:51.748571+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:07:15.966133+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.150:0/2620931070":"2024-09-12T08:08:06.649692+0000","172.21.15.150:0/1073581409":"2024-09-12T08:02:45.986151+0000","172.21.15.150:0/2363649697":"2024-09-12T08:08:06.649692+0000","172.21.15.150:6800/603451071":"2024-09-12T08:02:45.986151+0000","172.21.15.150:0/1614413548":"2024-09-12T08:02:45.986151+0000","172.21.15.150:0/3673218166":"2024-09-12T08:03:08.740828+0000","172.21.15.150:6800/556498589":"2024-09-12T08:03:08.740828+0000","172.21.15.150:0/3666733994":"2024-09-12T08:03:08.740828+0000","172.21.15.150:0/3271479010":"2024-09-12T08:03:08.740828+0000","172.21.15.150:0/3185731090":"2024-09-12T08:08:06.649692+0000","172.21.15.150:0/2539412006":"2024-09-12T08:08:06.649692+0000","172.21.15.150:0/2740222408":"2024-09-12T08:02:45.986151+0000","172.21.15.150:6800/127856313":"2024-09-12T08:08:06.649692+0000","172.21.15.150:0/2375121603":"2024-09-12T08:08:06.649692+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T08:08:08.733 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:08 smithi181 ceph-mon[32701]: mgrmap e19: y(active, since 1.01184s), standbys: x 2024-09-11T08:08:08.733 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:08 smithi181 ceph-mon[32701]: [11/Sep/2024:08:08:08] ENGINE Bus STARTING 2024-09-11T08:08:08.733 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:08 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:08.914 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:08 smithi150 ceph-mon[29276]: mgrmap e19: y(active, since 1.01184s), standbys: x 2024-09-11T08:08:08.915 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:08 smithi150 ceph-mon[29276]: [11/Sep/2024:08:08:08] ENGINE Bus STARTING 2024-09-11T08:08:08.915 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:08 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:08.915 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:08 smithi150 ceph-mon[39612]: mgrmap e19: y(active, since 1.01184s), standbys: x 2024-09-11T08:08:08.915 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:08 smithi150 ceph-mon[39612]: [11/Sep/2024:08:08:08] ENGINE Bus STARTING 2024-09-11T08:08:08.915 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:08 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:09.657 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-11T08:08:09.658 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-11T08:08:09.658 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-11T08:08:09.659 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-11T08:08:09.659 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-11T08:08:09.659 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-11T08:08:09.660 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-11T08:08:09.660 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-11T08:08:09.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:09 smithi150 ceph-mon[29276]: [11/Sep/2024:08:08:08] ENGINE Serving on https://172.21.15.150:7150 2024-09-11T08:08:09.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:09 smithi150 ceph-mon[29276]: [11/Sep/2024:08:08:08] ENGINE Bus STARTED 2024-09-11T08:08:09.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:09 smithi150 ceph-mon[29276]: [11/Sep/2024:08:08:08] ENGINE Error in HTTPServer.serve 2024-09-11T08:08:09.989 INFO:journalctl@ceph.mon.a.smithi150.stdout: Traceback (most recent call last): 2024-09-11T08:08:09.989 INFO:journalctl@ceph.mon.a.smithi150.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-11T08:08:09.989 INFO:journalctl@ceph.mon.a.smithi150.stdout: self._connections.run(self.expiration_interval) 2024-09-11T08:08:09.989 INFO:journalctl@ceph.mon.a.smithi150.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-11T08:08:09.990 INFO:journalctl@ceph.mon.a.smithi150.stdout: self._run(expiration_interval) 2024-09-11T08:08:09.990 INFO:journalctl@ceph.mon.a.smithi150.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-11T08:08:09.990 INFO:journalctl@ceph.mon.a.smithi150.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-11T08:08:09.990 INFO:journalctl@ceph.mon.a.smithi150.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-11T08:08:09.990 INFO:journalctl@ceph.mon.a.smithi150.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-11T08:08:09.990 INFO:journalctl@ceph.mon.a.smithi150.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-11T08:08:09.990 INFO:journalctl@ceph.mon.a.smithi150.stdout: s = self.context.wrap_socket( 2024-09-11T08:08:09.990 INFO:journalctl@ceph.mon.a.smithi150.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-11T08:08:09.990 INFO:journalctl@ceph.mon.a.smithi150.stdout: return self.sslsocket_class._create( 2024-09-11T08:08:09.990 INFO:journalctl@ceph.mon.a.smithi150.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-11T08:08:09.990 INFO:journalctl@ceph.mon.a.smithi150.stdout: self.do_handshake() 2024-09-11T08:08:09.990 INFO:journalctl@ceph.mon.a.smithi150.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-11T08:08:09.990 INFO:journalctl@ceph.mon.a.smithi150.stdout: self._sslobj.do_handshake() 2024-09-11T08:08:09.991 INFO:journalctl@ceph.mon.a.smithi150.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-11T08:08:09.991 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:09 smithi150 ceph-mon[29276]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:08:09.991 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:09 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/4196440930' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-11T08:08:09.991 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:09 smithi150 ceph-mon[29276]: mgrmap e20: y(active, since 2s), standbys: x 2024-09-11T08:08:09.991 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:09 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:09.991 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:09 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:09.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:09 smithi150 ceph-mon[39612]: [11/Sep/2024:08:08:08] ENGINE Serving on https://172.21.15.150:7150 2024-09-11T08:08:09.992 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:09 smithi150 ceph-mon[39612]: [11/Sep/2024:08:08:08] ENGINE Bus STARTED 2024-09-11T08:08:09.992 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:09 smithi150 ceph-mon[39612]: [11/Sep/2024:08:08:08] ENGINE Error in HTTPServer.serve 2024-09-11T08:08:09.992 INFO:journalctl@ceph.mon.c.smithi150.stdout: Traceback (most recent call last): 2024-09-11T08:08:09.992 INFO:journalctl@ceph.mon.c.smithi150.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-11T08:08:09.992 INFO:journalctl@ceph.mon.c.smithi150.stdout: self._connections.run(self.expiration_interval) 2024-09-11T08:08:09.992 INFO:journalctl@ceph.mon.c.smithi150.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-11T08:08:09.992 INFO:journalctl@ceph.mon.c.smithi150.stdout: self._run(expiration_interval) 2024-09-11T08:08:09.992 INFO:journalctl@ceph.mon.c.smithi150.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-11T08:08:09.992 INFO:journalctl@ceph.mon.c.smithi150.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-11T08:08:09.992 INFO:journalctl@ceph.mon.c.smithi150.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-11T08:08:09.992 INFO:journalctl@ceph.mon.c.smithi150.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-11T08:08:09.993 INFO:journalctl@ceph.mon.c.smithi150.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-11T08:08:09.993 INFO:journalctl@ceph.mon.c.smithi150.stdout: s = self.context.wrap_socket( 2024-09-11T08:08:09.993 INFO:journalctl@ceph.mon.c.smithi150.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-11T08:08:09.993 INFO:journalctl@ceph.mon.c.smithi150.stdout: return self.sslsocket_class._create( 2024-09-11T08:08:09.993 INFO:journalctl@ceph.mon.c.smithi150.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-11T08:08:09.993 INFO:journalctl@ceph.mon.c.smithi150.stdout: self.do_handshake() 2024-09-11T08:08:09.993 INFO:journalctl@ceph.mon.c.smithi150.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-11T08:08:09.993 INFO:journalctl@ceph.mon.c.smithi150.stdout: self._sslobj.do_handshake() 2024-09-11T08:08:09.993 INFO:journalctl@ceph.mon.c.smithi150.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-11T08:08:09.993 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:09 smithi150 ceph-mon[39612]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:08:09.994 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:09 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/4196440930' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-11T08:08:09.994 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:09 smithi150 ceph-mon[39612]: mgrmap e20: y(active, since 2s), standbys: x 2024-09-11T08:08:09.994 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:09 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:09.994 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:09 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:10.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:09 smithi181 ceph-mon[32701]: [11/Sep/2024:08:08:08] ENGINE Serving on https://172.21.15.150:7150 2024-09-11T08:08:10.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:09 smithi181 ceph-mon[32701]: [11/Sep/2024:08:08:08] ENGINE Bus STARTED 2024-09-11T08:08:10.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:09 smithi181 ceph-mon[32701]: [11/Sep/2024:08:08:08] ENGINE Error in HTTPServer.serve 2024-09-11T08:08:10.074 INFO:journalctl@ceph.mon.b.smithi181.stdout: Traceback (most recent call last): 2024-09-11T08:08:10.074 INFO:journalctl@ceph.mon.b.smithi181.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-11T08:08:10.074 INFO:journalctl@ceph.mon.b.smithi181.stdout: self._connections.run(self.expiration_interval) 2024-09-11T08:08:10.074 INFO:journalctl@ceph.mon.b.smithi181.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-11T08:08:10.074 INFO:journalctl@ceph.mon.b.smithi181.stdout: self._run(expiration_interval) 2024-09-11T08:08:10.074 INFO:journalctl@ceph.mon.b.smithi181.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-11T08:08:10.074 INFO:journalctl@ceph.mon.b.smithi181.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-11T08:08:10.074 INFO:journalctl@ceph.mon.b.smithi181.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-11T08:08:10.075 INFO:journalctl@ceph.mon.b.smithi181.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-11T08:08:10.075 INFO:journalctl@ceph.mon.b.smithi181.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-11T08:08:10.075 INFO:journalctl@ceph.mon.b.smithi181.stdout: s = self.context.wrap_socket( 2024-09-11T08:08:10.075 INFO:journalctl@ceph.mon.b.smithi181.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-11T08:08:10.075 INFO:journalctl@ceph.mon.b.smithi181.stdout: return self.sslsocket_class._create( 2024-09-11T08:08:10.075 INFO:journalctl@ceph.mon.b.smithi181.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-11T08:08:10.075 INFO:journalctl@ceph.mon.b.smithi181.stdout: self.do_handshake() 2024-09-11T08:08:10.075 INFO:journalctl@ceph.mon.b.smithi181.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-11T08:08:10.075 INFO:journalctl@ceph.mon.b.smithi181.stdout: self._sslobj.do_handshake() 2024-09-11T08:08:10.075 INFO:journalctl@ceph.mon.b.smithi181.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-11T08:08:10.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:09 smithi181 ceph-mon[32701]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:08:10.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:09 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/4196440930' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-11T08:08:10.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:09 smithi181 ceph-mon[32701]: mgrmap e20: y(active, since 2s), standbys: x 2024-09-11T08:08:10.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:09 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:10.076 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:09 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:10.257 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:08:10.259 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:08:10.262 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:08:10.263 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:08:10.265 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:08:10.265 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:08:10.269 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:08:10.269 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:08:11.333 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:11 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:11.334 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:11 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:11.334 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:11 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:08:11.334 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:11 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:08:11.334 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:11 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:08:11.334 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:11 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:08:11.334 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:11 smithi150 ceph-mon[39612]: Adjusting osd_memory_target on smithi181 to 3759M 2024-09-11T08:08:11.334 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:11 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:11.334 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:11 smithi150 ceph-mon[39612]: osdmap e74: 8 total, 8 up, 8 in 2024-09-11T08:08:11.335 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:11 smithi150 ceph-mon[39612]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:08:11.335 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:11 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:11.335 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:11 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:11.335 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:11 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:08:11.335 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:11 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:08:11.336 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:11 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:08:11.336 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:11 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:08:11.336 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:11 smithi150 ceph-mon[29276]: Adjusting osd_memory_target on smithi181 to 3759M 2024-09-11T08:08:11.336 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:11 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:11.336 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:11 smithi150 ceph-mon[29276]: osdmap e74: 8 total, 8 up, 8 in 2024-09-11T08:08:11.336 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:11 smithi150 ceph-mon[29276]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:08:11.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:11 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:11.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:11 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:11.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:11 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:08:11.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:11 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:08:11.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:11 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:08:11.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:11 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:08:11.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:11 smithi181 ceph-mon[32701]: Adjusting osd_memory_target on smithi181 to 3759M 2024-09-11T08:08:11.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:11 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:11.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:11 smithi181 ceph-mon[32701]: osdmap e74: 8 total, 8 up, 8 in 2024-09-11T08:08:11.575 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:11 smithi181 ceph-mon[32701]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:08:12.659 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:12 smithi150 ceph-mon[29276]: mgrmap e21: y(active, since 4s), standbys: x 2024-09-11T08:08:12.659 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:12 smithi150 ceph-mon[39612]: mgrmap e21: y(active, since 4s), standbys: x 2024-09-11T08:08:12.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:12 smithi181 ceph-mon[32701]: mgrmap e21: y(active, since 4s), standbys: x 2024-09-11T08:08:13.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:13 smithi150 ceph-mon[29276]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:08:13.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:13 smithi150 ceph-mon[39612]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:08:13.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:13 smithi181 ceph-mon[32701]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:08:15.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:08:15 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:08:15] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:08:15.902 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:15 smithi150 ceph-mon[29276]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-11T08:08:15.902 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:15 smithi150 ceph-mon[39612]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-11T08:08:16.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:15 smithi181 ceph-mon[32701]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-11T08:08:17.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:17 smithi150 ceph-mon[29276]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-11T08:08:17.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:17 smithi150 ceph-mon[39612]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-11T08:08:18.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:17 smithi181 ceph-mon[32701]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-11T08:08:18.074 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:08:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:08:18.954 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:18 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:08:18.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:18 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:08:19.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:18 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:08:19.846 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:19 smithi150 ceph-mon[39612]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-11T08:08:19.846 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:19 smithi150 ceph-mon[29276]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-11T08:08:19.870 INFO:teuthology.orchestra.run.smithi150.stdout:77309411365 2024-09-11T08:08:19.870 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-11T08:08:19.992 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:19 smithi181 ceph-mon[32701]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-11T08:08:20.889 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:20 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:08:20.889 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:20 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:08:20.889 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:20 smithi150 ceph-mon[29276]: osdmap e75: 8 total, 8 up, 8 in 2024-09-11T08:08:20.889 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:20 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:20.889 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:20 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:20.889 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:20 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:08:20.889 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:20 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:08:20.889 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:20 smithi150 ceph-mon[29276]: osdmap e76: 8 total, 8 up, 8 in 2024-09-11T08:08:20.890 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:20 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:08:20.890 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:20 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:08:20.890 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:20 smithi150 ceph-mon[39612]: osdmap e75: 8 total, 8 up, 8 in 2024-09-11T08:08:20.890 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:20 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:20.890 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:20 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:20.890 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:20 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:08:20.890 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:20 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:08:20.890 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:20 smithi150 ceph-mon[39612]: osdmap e76: 8 total, 8 up, 8 in 2024-09-11T08:08:21.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:20 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:08:21.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:20 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:08:21.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:20 smithi181 ceph-mon[32701]: osdmap e75: 8 total, 8 up, 8 in 2024-09-11T08:08:21.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:20 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:21.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:20 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:21.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:20 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:08:21.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:20 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:08:21.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:20 smithi181 ceph-mon[32701]: osdmap e76: 8 total, 8 up, 8 in 2024-09-11T08:08:21.911 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:21 smithi150 ceph-mon[29276]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-11T08:08:21.912 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:21 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:08:21.912 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:21 smithi150 ceph-mon[39612]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-11T08:08:21.912 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:21 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:08:22.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:21 smithi181 ceph-mon[32701]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-11T08:08:22.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:21 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:08:23.498 INFO:teuthology.orchestra.run.smithi150.stdout:133143986205 2024-09-11T08:08:23.498 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-11T08:08:23.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:23 smithi150 ceph-mon[29276]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-11T08:08:23.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:23 smithi150 ceph-mon[39612]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-11T08:08:24.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:23 smithi181 ceph-mon[32701]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-11T08:08:24.453 INFO:teuthology.orchestra.run.smithi150.stdout:55834574891 2024-09-11T08:08:24.453 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-11T08:08:24.852 INFO:teuthology.orchestra.run.smithi150.stdout:34359738415 2024-09-11T08:08:25.063 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-11T08:08:25.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:08:25 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:08:25] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:08:25.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:25 smithi150 ceph-mon[29276]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-11T08:08:25.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:25 smithi150 ceph-mon[39612]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-11T08:08:26.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:25 smithi181 ceph-mon[32701]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-11T08:08:26.173 INFO:teuthology.orchestra.run.smithi150.stdout:163208757273 2024-09-11T08:08:26.173 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-11T08:08:26.804 INFO:teuthology.orchestra.run.smithi150.stdout:227633266703 2024-09-11T08:08:26.804 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-11T08:08:27.733 INFO:teuthology.orchestra.run.smithi150.stdout:193273528340 2024-09-11T08:08:27.733 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-11T08:08:27.801 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:08:27.802 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:08:27.802 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:08:27.803 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:08:27.804 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:08:27.863 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:08:27.939 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:27 smithi150 ceph-mon[39612]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-11T08:08:27.940 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:27 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:27.940 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:27 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:27.940 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:27 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:08:27.940 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:27 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:08:27.940 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:27 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:08:27.940 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:27 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:08:27.940 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:27 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:27.940 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:27 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:08:27.940 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:27 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:08:27.941 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:27 smithi150 ceph-mon[29276]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-11T08:08:27.941 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:27 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:27.941 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:27 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:27.941 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:27 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:08:27.941 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:27 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:08:27.941 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:27 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:08:27.941 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:27 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:08:27.941 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:27 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:27.941 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:27 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:08:27.941 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:27 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:08:28.073 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:08:27 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:08:28.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:27 smithi181 ceph-mon[32701]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-11T08:08:28.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:27 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:28.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:27 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:28.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:27 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:08:28.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:27 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:08:28.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:27 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:08:28.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:27 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:08:28.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:27 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:28.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:27 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:08:28.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:27 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:08:28.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:28 smithi150 ceph-mon[29276]: Adjusting osd_memory_target on smithi150 to 3760M 2024-09-11T08:08:28.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:28 smithi150 ceph-mon[29276]: Updating smithi150:/etc/ceph/ceph.conf 2024-09-11T08:08:28.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:28 smithi150 ceph-mon[29276]: Updating smithi181:/etc/ceph/ceph.conf 2024-09-11T08:08:28.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:28 smithi150 ceph-mon[29276]: Updating smithi150:/var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-11T08:08:28.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:28 smithi150 ceph-mon[29276]: Updating smithi181:/var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-11T08:08:28.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:28 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:08:28.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:28 smithi150 ceph-mon[29276]: Updating smithi181:/etc/ceph/ceph.client.admin.keyring 2024-09-11T08:08:28.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:28 smithi150 ceph-mon[29276]: Updating smithi150:/etc/ceph/ceph.client.admin.keyring 2024-09-11T08:08:28.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:28 smithi150 ceph-mon[29276]: Updating smithi181:/var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-11T08:08:28.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:28 smithi150 ceph-mon[29276]: Updating smithi150:/var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-11T08:08:28.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:28 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:28.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:28 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:28.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:28 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:28.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:28 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:28.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:28 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:28.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:28 smithi150 ceph-mon[39612]: Adjusting osd_memory_target on smithi150 to 3760M 2024-09-11T08:08:28.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:28 smithi150 ceph-mon[39612]: Updating smithi150:/etc/ceph/ceph.conf 2024-09-11T08:08:28.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:28 smithi150 ceph-mon[39612]: Updating smithi181:/etc/ceph/ceph.conf 2024-09-11T08:08:28.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:28 smithi150 ceph-mon[39612]: Updating smithi150:/var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-11T08:08:28.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:28 smithi150 ceph-mon[39612]: Updating smithi181:/var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-11T08:08:28.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:28 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:08:28.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:28 smithi150 ceph-mon[39612]: Updating smithi181:/etc/ceph/ceph.client.admin.keyring 2024-09-11T08:08:28.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:28 smithi150 ceph-mon[39612]: Updating smithi150:/etc/ceph/ceph.client.admin.keyring 2024-09-11T08:08:28.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:28 smithi150 ceph-mon[39612]: Updating smithi181:/var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-11T08:08:28.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:28 smithi150 ceph-mon[39612]: Updating smithi150:/var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-11T08:08:28.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:28 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:28.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:28 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:28.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:28 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:28.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:28 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:28.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:28 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:29.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:28 smithi181 ceph-mon[32701]: Adjusting osd_memory_target on smithi150 to 3760M 2024-09-11T08:08:29.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:28 smithi181 ceph-mon[32701]: Updating smithi150:/etc/ceph/ceph.conf 2024-09-11T08:08:29.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:28 smithi181 ceph-mon[32701]: Updating smithi181:/etc/ceph/ceph.conf 2024-09-11T08:08:29.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:28 smithi181 ceph-mon[32701]: Updating smithi150:/var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-11T08:08:29.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:28 smithi181 ceph-mon[32701]: Updating smithi181:/var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-11T08:08:29.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:28 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:08:29.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:28 smithi181 ceph-mon[32701]: Updating smithi181:/etc/ceph/ceph.client.admin.keyring 2024-09-11T08:08:29.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:28 smithi181 ceph-mon[32701]: Updating smithi150:/etc/ceph/ceph.client.admin.keyring 2024-09-11T08:08:29.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:28 smithi181 ceph-mon[32701]: Updating smithi181:/var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-11T08:08:29.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:28 smithi181 ceph-mon[32701]: Updating smithi150:/var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-11T08:08:29.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:28 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:29.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:28 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:29.076 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:28 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:29.076 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:28 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:29.076 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:28 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:29.976 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:29 smithi150 ceph-mon[29276]: Deploying daemon alertmanager.a on smithi150 2024-09-11T08:08:29.976 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:29 smithi150 ceph-mon[29276]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-11T08:08:29.977 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:29 smithi150 ceph-mon[39612]: Deploying daemon alertmanager.a on smithi150 2024-09-11T08:08:29.977 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:29 smithi150 ceph-mon[39612]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-11T08:08:30.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:29 smithi181 ceph-mon[32701]: Deploying daemon alertmanager.a on smithi150 2024-09-11T08:08:30.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:29 smithi181 ceph-mon[32701]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-11T08:08:31.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:31 smithi150 ceph-mon[29276]: 3.b scrub starts 2024-09-11T08:08:31.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:31 smithi150 ceph-mon[29276]: 3.b scrub ok 2024-09-11T08:08:31.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:31 smithi150 ceph-mon[29276]: 3.11 deep-scrub starts 2024-09-11T08:08:31.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:31 smithi150 ceph-mon[29276]: 3.11 deep-scrub ok 2024-09-11T08:08:31.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:31 smithi150 ceph-mon[29276]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 992 B/s rd, 0 op/s 2024-09-11T08:08:31.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:31 smithi150 ceph-mon[39612]: 3.b scrub starts 2024-09-11T08:08:31.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:31 smithi150 ceph-mon[39612]: 3.b scrub ok 2024-09-11T08:08:31.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:31 smithi150 ceph-mon[39612]: 3.11 deep-scrub starts 2024-09-11T08:08:31.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:31 smithi150 ceph-mon[39612]: 3.11 deep-scrub ok 2024-09-11T08:08:31.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:31 smithi150 ceph-mon[39612]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 992 B/s rd, 0 op/s 2024-09-11T08:08:31.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:31 smithi181 ceph-mon[32701]: 3.b scrub starts 2024-09-11T08:08:31.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:31 smithi181 ceph-mon[32701]: 3.b scrub ok 2024-09-11T08:08:31.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:31 smithi181 ceph-mon[32701]: 3.11 deep-scrub starts 2024-09-11T08:08:31.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:31 smithi181 ceph-mon[32701]: 3.11 deep-scrub ok 2024-09-11T08:08:31.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:31 smithi181 ceph-mon[32701]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 992 B/s rd, 0 op/s 2024-09-11T08:08:32.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:32 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:08:32.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:32 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:08:32.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:32 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:08:32.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:32 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:08:32.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:32 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:08:32.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:32 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:08:33.105 INFO:teuthology.orchestra.run.smithi150.stdout:34359738417 2024-09-11T08:08:33.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:33 smithi150 ceph-mon[39612]: osdmap e77: 8 total, 8 up, 8 in 2024-09-11T08:08:33.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:33 smithi150 ceph-mon[39612]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-11T08:08:33.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:33 smithi150 ceph-mon[39612]: 2.1 scrub starts 2024-09-11T08:08:33.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:33 smithi150 ceph-mon[39612]: 2.1 scrub ok 2024-09-11T08:08:33.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:33 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/4154884724' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-11T08:08:33.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:33 smithi150 ceph-mon[29276]: osdmap e77: 8 total, 8 up, 8 in 2024-09-11T08:08:33.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:33 smithi150 ceph-mon[29276]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-11T08:08:33.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:33 smithi150 ceph-mon[29276]: 2.1 scrub starts 2024-09-11T08:08:33.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:33 smithi150 ceph-mon[29276]: 2.1 scrub ok 2024-09-11T08:08:33.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:33 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/4154884724' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-11T08:08:33.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:33 smithi181 ceph-mon[32701]: osdmap e77: 8 total, 8 up, 8 in 2024-09-11T08:08:33.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:33 smithi181 ceph-mon[32701]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-11T08:08:33.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:33 smithi181 ceph-mon[32701]: 2.1 scrub starts 2024-09-11T08:08:33.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:33 smithi181 ceph-mon[32701]: 2.1 scrub ok 2024-09-11T08:08:33.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:33 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/4154884724' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-11T08:08:34.367 INFO:teuthology.orchestra.run.smithi150.stdout:107374182435 2024-09-11T08:08:34.367 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-11T08:08:34.702 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:34 smithi150 ceph-mon[29276]: 4.d deep-scrub starts 2024-09-11T08:08:34.703 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:34 smithi150 ceph-mon[29276]: 4.d deep-scrub ok 2024-09-11T08:08:34.703 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:34 smithi150 ceph-mon[29276]: 3.0 scrub starts 2024-09-11T08:08:34.703 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:34 smithi150 ceph-mon[29276]: 3.0 scrub ok 2024-09-11T08:08:34.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:34 smithi150 ceph-mon[39612]: 4.d deep-scrub starts 2024-09-11T08:08:34.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:34 smithi150 ceph-mon[39612]: 4.d deep-scrub ok 2024-09-11T08:08:34.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:34 smithi150 ceph-mon[39612]: 3.0 scrub starts 2024-09-11T08:08:34.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:34 smithi150 ceph-mon[39612]: 3.0 scrub ok 2024-09-11T08:08:34.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:34 smithi181 ceph-mon[32701]: 4.d deep-scrub starts 2024-09-11T08:08:34.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:34 smithi181 ceph-mon[32701]: 4.d deep-scrub ok 2024-09-11T08:08:34.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:34 smithi181 ceph-mon[32701]: 3.0 scrub starts 2024-09-11T08:08:34.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:34 smithi181 ceph-mon[32701]: 3.0 scrub ok 2024-09-11T08:08:35.535 INFO:teuthology.orchestra.run.smithi150.stdout:77309411368 2024-09-11T08:08:35.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:35 smithi181 ceph-mon[32701]: 5.11 scrub starts 2024-09-11T08:08:35.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:35 smithi181 ceph-mon[32701]: 5.11 scrub ok 2024-09-11T08:08:35.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:35 smithi181 ceph-mon[32701]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-11T08:08:35.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:35 smithi181 ceph-mon[32701]: 3.c scrub starts 2024-09-11T08:08:35.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:35 smithi181 ceph-mon[32701]: 3.c scrub ok 2024-09-11T08:08:35.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:35 smithi181 ceph-mon[32701]: osdmap e78: 8 total, 8 up, 8 in 2024-09-11T08:08:35.574 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:08:35 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:08:35] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:08:35.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:35 smithi150 ceph-mon[29276]: 5.11 scrub starts 2024-09-11T08:08:35.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:35 smithi150 ceph-mon[29276]: 5.11 scrub ok 2024-09-11T08:08:35.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:35 smithi150 ceph-mon[29276]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-11T08:08:35.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:35 smithi150 ceph-mon[29276]: 3.c scrub starts 2024-09-11T08:08:35.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:35 smithi150 ceph-mon[29276]: 3.c scrub ok 2024-09-11T08:08:35.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:35 smithi150 ceph-mon[29276]: osdmap e78: 8 total, 8 up, 8 in 2024-09-11T08:08:35.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:35 smithi150 ceph-mon[39612]: 5.11 scrub starts 2024-09-11T08:08:35.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:35 smithi150 ceph-mon[39612]: 5.11 scrub ok 2024-09-11T08:08:35.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:35 smithi150 ceph-mon[39612]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-11T08:08:35.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:35 smithi150 ceph-mon[39612]: 3.c scrub starts 2024-09-11T08:08:35.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:35 smithi150 ceph-mon[39612]: 3.c scrub ok 2024-09-11T08:08:35.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:35 smithi150 ceph-mon[39612]: osdmap e78: 8 total, 8 up, 8 in 2024-09-11T08:08:35.784 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:08:36.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:36 smithi150 ceph-mon[39612]: 3.3 scrub starts 2024-09-11T08:08:36.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:36 smithi150 ceph-mon[39612]: 3.3 scrub ok 2024-09-11T08:08:36.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:36 smithi150 ceph-mon[39612]: 4.7 scrub starts 2024-09-11T08:08:36.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:36 smithi150 ceph-mon[39612]: 4.7 scrub ok 2024-09-11T08:08:36.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:36 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1390048377' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-11T08:08:36.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:36 smithi150 ceph-mon[29276]: 3.3 scrub starts 2024-09-11T08:08:36.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:36 smithi150 ceph-mon[29276]: 3.3 scrub ok 2024-09-11T08:08:36.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:36 smithi150 ceph-mon[29276]: 4.7 scrub starts 2024-09-11T08:08:36.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:36 smithi150 ceph-mon[29276]: 4.7 scrub ok 2024-09-11T08:08:36.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:36 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1390048377' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-11T08:08:36.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:36 smithi181 ceph-mon[32701]: 3.3 scrub starts 2024-09-11T08:08:36.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:36 smithi181 ceph-mon[32701]: 3.3 scrub ok 2024-09-11T08:08:36.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:36 smithi181 ceph-mon[32701]: 4.7 scrub starts 2024-09-11T08:08:36.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:36 smithi181 ceph-mon[32701]: 4.7 scrub ok 2024-09-11T08:08:36.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:36 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1390048377' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-11T08:08:37.538 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:37 smithi150 ceph-mon[29276]: 4.e scrub starts 2024-09-11T08:08:37.538 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:37 smithi150 ceph-mon[29276]: 4.e scrub ok 2024-09-11T08:08:37.538 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:37 smithi150 ceph-mon[29276]: 3.1f scrub starts 2024-09-11T08:08:37.538 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:37 smithi150 ceph-mon[29276]: 3.1f scrub ok 2024-09-11T08:08:37.538 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:37 smithi150 ceph-mon[29276]: 6.8 scrub starts 2024-09-11T08:08:37.538 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:37 smithi150 ceph-mon[29276]: 6.8 scrub ok 2024-09-11T08:08:37.538 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:37 smithi150 ceph-mon[29276]: 4.14 scrub starts 2024-09-11T08:08:37.538 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:37 smithi150 ceph-mon[29276]: 4.14 scrub ok 2024-09-11T08:08:37.538 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:37 smithi150 ceph-mon[29276]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-11T08:08:37.538 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:37 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:08:37.538 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:37 smithi150 ceph-mon[29276]: 3.1c scrub starts 2024-09-11T08:08:37.539 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:37 smithi150 ceph-mon[29276]: 3.1c scrub ok 2024-09-11T08:08:37.539 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:37 smithi150 ceph-mon[39612]: 4.e scrub starts 2024-09-11T08:08:37.539 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:37 smithi150 ceph-mon[39612]: 4.e scrub ok 2024-09-11T08:08:37.539 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:37 smithi150 ceph-mon[39612]: 3.1f scrub starts 2024-09-11T08:08:37.539 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:37 smithi150 ceph-mon[39612]: 3.1f scrub ok 2024-09-11T08:08:37.539 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:37 smithi150 ceph-mon[39612]: 6.8 scrub starts 2024-09-11T08:08:37.539 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:37 smithi150 ceph-mon[39612]: 6.8 scrub ok 2024-09-11T08:08:37.539 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:37 smithi150 ceph-mon[39612]: 4.14 scrub starts 2024-09-11T08:08:37.539 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:37 smithi150 ceph-mon[39612]: 4.14 scrub ok 2024-09-11T08:08:37.539 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:37 smithi150 ceph-mon[39612]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-11T08:08:37.540 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:37 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:08:37.540 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:37 smithi150 ceph-mon[39612]: 3.1c scrub starts 2024-09-11T08:08:37.540 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:37 smithi150 ceph-mon[39612]: 3.1c scrub ok 2024-09-11T08:08:37.699 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:37 smithi181 ceph-mon[32701]: 4.e scrub starts 2024-09-11T08:08:37.699 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:37 smithi181 ceph-mon[32701]: 4.e scrub ok 2024-09-11T08:08:37.700 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:37 smithi181 ceph-mon[32701]: 3.1f scrub starts 2024-09-11T08:08:37.700 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:37 smithi181 ceph-mon[32701]: 3.1f scrub ok 2024-09-11T08:08:37.700 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:37 smithi181 ceph-mon[32701]: 6.8 scrub starts 2024-09-11T08:08:37.700 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:37 smithi181 ceph-mon[32701]: 6.8 scrub ok 2024-09-11T08:08:37.700 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:37 smithi181 ceph-mon[32701]: 4.14 scrub starts 2024-09-11T08:08:37.700 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:37 smithi181 ceph-mon[32701]: 4.14 scrub ok 2024-09-11T08:08:37.700 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:37 smithi181 ceph-mon[32701]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-11T08:08:37.700 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:37 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:08:37.700 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:37 smithi181 ceph-mon[32701]: 3.1c scrub starts 2024-09-11T08:08:37.701 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:37 smithi181 ceph-mon[32701]: 3.1c scrub ok 2024-09-11T08:08:37.725 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738415 got 34359738417 for osd.0 2024-09-11T08:08:37.725 DEBUG:teuthology.parallel:result is None 2024-09-11T08:08:37.960 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411365 got 77309411368 for osd.2 2024-09-11T08:08:37.961 DEBUG:teuthology.parallel:result is None 2024-09-11T08:08:38.073 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:08:37 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:08:38.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:38 smithi150 ceph-mon[29276]: 3.9 deep-scrub starts 2024-09-11T08:08:38.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:38 smithi150 ceph-mon[29276]: 3.9 deep-scrub ok 2024-09-11T08:08:38.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:38 smithi150 ceph-mon[29276]: 4.6 scrub starts 2024-09-11T08:08:38.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:38 smithi150 ceph-mon[29276]: 4.6 scrub ok 2024-09-11T08:08:38.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:38 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:08:38.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:38 smithi150 ceph-mon[29276]: 2.2 scrub starts 2024-09-11T08:08:38.491 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:38 smithi150 ceph-mon[29276]: 2.2 scrub ok 2024-09-11T08:08:38.491 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:38 smithi150 ceph-mon[29276]: 5.a scrub starts 2024-09-11T08:08:38.491 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:38 smithi150 ceph-mon[29276]: 5.a scrub ok 2024-09-11T08:08:38.491 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:38 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:08:38.491 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:38 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:08:38.492 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:38 smithi150 ceph-mon[39612]: 3.9 deep-scrub starts 2024-09-11T08:08:38.492 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:38 smithi150 ceph-mon[39612]: 3.9 deep-scrub ok 2024-09-11T08:08:38.492 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:38 smithi150 ceph-mon[39612]: 4.6 scrub starts 2024-09-11T08:08:38.492 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:38 smithi150 ceph-mon[39612]: 4.6 scrub ok 2024-09-11T08:08:38.492 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:38 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:08:38.493 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:38 smithi150 ceph-mon[39612]: 2.2 scrub starts 2024-09-11T08:08:38.493 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:38 smithi150 ceph-mon[39612]: 2.2 scrub ok 2024-09-11T08:08:38.493 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:38 smithi150 ceph-mon[39612]: 5.a scrub starts 2024-09-11T08:08:38.493 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:38 smithi150 ceph-mon[39612]: 5.a scrub ok 2024-09-11T08:08:38.493 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:38 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:08:38.493 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:38 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:08:38.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:38 smithi181 ceph-mon[32701]: 3.9 deep-scrub starts 2024-09-11T08:08:38.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:38 smithi181 ceph-mon[32701]: 3.9 deep-scrub ok 2024-09-11T08:08:38.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:38 smithi181 ceph-mon[32701]: 4.6 scrub starts 2024-09-11T08:08:38.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:38 smithi181 ceph-mon[32701]: 4.6 scrub ok 2024-09-11T08:08:38.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:38 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:08:38.825 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:38 smithi181 ceph-mon[32701]: 2.2 scrub starts 2024-09-11T08:08:38.825 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:38 smithi181 ceph-mon[32701]: 2.2 scrub ok 2024-09-11T08:08:38.825 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:38 smithi181 ceph-mon[32701]: 5.a scrub starts 2024-09-11T08:08:38.825 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:38 smithi181 ceph-mon[32701]: 5.a scrub ok 2024-09-11T08:08:38.825 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:38 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:08:38.825 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:38 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:08:39.032 INFO:teuthology.orchestra.run.smithi150.stdout:163208757275 2024-09-11T08:08:39.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:39 smithi150 ceph-mon[29276]: 6.15 scrub starts 2024-09-11T08:08:39.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:39 smithi150 ceph-mon[29276]: 6.15 scrub ok 2024-09-11T08:08:39.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:39 smithi150 ceph-mon[29276]: osdmap e79: 8 total, 8 up, 8 in 2024-09-11T08:08:39.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:39 smithi150 ceph-mon[29276]: 5.19 scrub starts 2024-09-11T08:08:39.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:39 smithi150 ceph-mon[29276]: 5.19 scrub ok 2024-09-11T08:08:39.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:39 smithi150 ceph-mon[29276]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-11T08:08:39.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:39 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2994762454' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-11T08:08:39.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:39 smithi150 ceph-mon[39612]: 6.15 scrub starts 2024-09-11T08:08:39.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:39 smithi150 ceph-mon[39612]: 6.15 scrub ok 2024-09-11T08:08:39.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:39 smithi150 ceph-mon[39612]: osdmap e79: 8 total, 8 up, 8 in 2024-09-11T08:08:39.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:39 smithi150 ceph-mon[39612]: 5.19 scrub starts 2024-09-11T08:08:39.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:39 smithi150 ceph-mon[39612]: 5.19 scrub ok 2024-09-11T08:08:39.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:39 smithi150 ceph-mon[39612]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-11T08:08:39.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:39 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2994762454' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-11T08:08:39.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:39 smithi181 ceph-mon[32701]: 6.15 scrub starts 2024-09-11T08:08:39.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:39 smithi181 ceph-mon[32701]: 6.15 scrub ok 2024-09-11T08:08:39.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:39 smithi181 ceph-mon[32701]: osdmap e79: 8 total, 8 up, 8 in 2024-09-11T08:08:39.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:39 smithi181 ceph-mon[32701]: 5.19 scrub starts 2024-09-11T08:08:39.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:39 smithi181 ceph-mon[32701]: 5.19 scrub ok 2024-09-11T08:08:39.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:39 smithi181 ceph-mon[32701]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-11T08:08:39.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:39 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2994762454' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-11T08:08:40.261 INFO:teuthology.orchestra.run.smithi150.stdout:55834574895 2024-09-11T08:08:40.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:40 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3595764823' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-11T08:08:40.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:40 smithi150 ceph-mon[29276]: osdmap e80: 8 total, 8 up, 8 in 2024-09-11T08:08:40.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:40 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3595764823' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-11T08:08:40.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:40 smithi150 ceph-mon[39612]: osdmap e80: 8 total, 8 up, 8 in 2024-09-11T08:08:40.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:40 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3595764823' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-11T08:08:40.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:40 smithi181 ceph-mon[32701]: osdmap e80: 8 total, 8 up, 8 in 2024-09-11T08:08:41.045 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:08:41.459 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757273 got 163208757275 for osd.5 2024-09-11T08:08:41.459 DEBUG:teuthology.parallel:result is None 2024-09-11T08:08:41.705 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:41 smithi150 ceph-mon[29276]: 6.18 scrub starts 2024-09-11T08:08:41.705 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:41 smithi150 ceph-mon[29276]: 6.18 scrub ok 2024-09-11T08:08:41.705 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:41 smithi150 ceph-mon[29276]: 5.1c scrub starts 2024-09-11T08:08:41.705 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:41 smithi150 ceph-mon[29276]: 5.1c scrub ok 2024-09-11T08:08:41.705 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:41 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:08:41.705 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:41 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:08:41.705 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:41 smithi150 ceph-mon[29276]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:08:41.706 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:41 smithi150 ceph-mon[39612]: 6.18 scrub starts 2024-09-11T08:08:41.706 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:41 smithi150 ceph-mon[39612]: 6.18 scrub ok 2024-09-11T08:08:41.706 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:41 smithi150 ceph-mon[39612]: 5.1c scrub starts 2024-09-11T08:08:41.706 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:41 smithi150 ceph-mon[39612]: 5.1c scrub ok 2024-09-11T08:08:41.706 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:41 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:08:41.706 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:41 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:08:41.706 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:41 smithi150 ceph-mon[39612]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:08:41.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:41 smithi181 ceph-mon[32701]: 6.18 scrub starts 2024-09-11T08:08:41.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:41 smithi181 ceph-mon[32701]: 6.18 scrub ok 2024-09-11T08:08:41.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:41 smithi181 ceph-mon[32701]: 5.1c scrub starts 2024-09-11T08:08:41.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:41 smithi181 ceph-mon[32701]: 5.1c scrub ok 2024-09-11T08:08:41.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:41 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:08:41.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:41 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:08:41.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:41 smithi181 ceph-mon[32701]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:08:42.326 INFO:teuthology.orchestra.run.smithi150.stdout:133143986209 2024-09-11T08:08:42.596 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:42 smithi150 ceph-mon[29276]: 3.1 scrub starts 2024-09-11T08:08:42.597 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:42 smithi150 ceph-mon[29276]: 3.1 scrub ok 2024-09-11T08:08:42.597 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:42 smithi150 ceph-mon[29276]: osdmap e81: 8 total, 8 up, 8 in 2024-09-11T08:08:42.597 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:42 smithi150 ceph-mon[29276]: 5.c scrub starts 2024-09-11T08:08:42.597 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:42 smithi150 ceph-mon[29276]: 5.c scrub ok 2024-09-11T08:08:42.597 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:42 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3246626285' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-11T08:08:42.597 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:42 smithi150 ceph-mon[39612]: 3.1 scrub starts 2024-09-11T08:08:42.597 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:42 smithi150 ceph-mon[39612]: 3.1 scrub ok 2024-09-11T08:08:42.597 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:42 smithi150 ceph-mon[39612]: osdmap e81: 8 total, 8 up, 8 in 2024-09-11T08:08:42.597 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:42 smithi150 ceph-mon[39612]: 5.c scrub starts 2024-09-11T08:08:42.598 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:42 smithi150 ceph-mon[39612]: 5.c scrub ok 2024-09-11T08:08:42.598 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:42 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3246626285' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-11T08:08:42.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:42 smithi181 ceph-mon[32701]: 3.1 scrub starts 2024-09-11T08:08:42.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:42 smithi181 ceph-mon[32701]: 3.1 scrub ok 2024-09-11T08:08:42.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:42 smithi181 ceph-mon[32701]: osdmap e81: 8 total, 8 up, 8 in 2024-09-11T08:08:42.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:42 smithi181 ceph-mon[32701]: 5.c scrub starts 2024-09-11T08:08:42.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:42 smithi181 ceph-mon[32701]: 5.c scrub ok 2024-09-11T08:08:42.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:42 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3246626285' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-11T08:08:43.021 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574891 got 55834574895 for osd.1 2024-09-11T08:08:43.022 DEBUG:teuthology.parallel:result is None 2024-09-11T08:08:43.357 INFO:teuthology.orchestra.run.smithi150.stdout:227633266706 2024-09-11T08:08:43.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:43 smithi150 ceph-mon[39612]: 3.17 scrub starts 2024-09-11T08:08:43.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:43 smithi150 ceph-mon[39612]: 3.17 scrub ok 2024-09-11T08:08:43.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:43 smithi150 ceph-mon[39612]: 4.c scrub starts 2024-09-11T08:08:43.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:43 smithi150 ceph-mon[39612]: 4.c scrub ok 2024-09-11T08:08:43.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:43 smithi150 ceph-mon[39612]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-11T08:08:43.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:43 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2396291460' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-11T08:08:43.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:43 smithi150 ceph-mon[29276]: 3.17 scrub starts 2024-09-11T08:08:43.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:43 smithi150 ceph-mon[29276]: 3.17 scrub ok 2024-09-11T08:08:43.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:43 smithi150 ceph-mon[29276]: 4.c scrub starts 2024-09-11T08:08:43.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:43 smithi150 ceph-mon[29276]: 4.c scrub ok 2024-09-11T08:08:43.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:43 smithi150 ceph-mon[29276]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-11T08:08:43.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:43 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2396291460' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-11T08:08:43.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:43 smithi181 ceph-mon[32701]: 3.17 scrub starts 2024-09-11T08:08:43.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:43 smithi181 ceph-mon[32701]: 3.17 scrub ok 2024-09-11T08:08:43.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:43 smithi181 ceph-mon[32701]: 4.c scrub starts 2024-09-11T08:08:43.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:43 smithi181 ceph-mon[32701]: 4.c scrub ok 2024-09-11T08:08:43.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:43 smithi181 ceph-mon[32701]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-11T08:08:43.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:43 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2396291460' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-11T08:08:44.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:44 smithi150 ceph-mon[29276]: 4.0 scrub starts 2024-09-11T08:08:44.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:44 smithi150 ceph-mon[29276]: 4.0 scrub ok 2024-09-11T08:08:44.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:44 smithi150 ceph-mon[39612]: 4.0 scrub starts 2024-09-11T08:08:44.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:44 smithi150 ceph-mon[39612]: 4.0 scrub ok 2024-09-11T08:08:44.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:44 smithi181 ceph-mon[32701]: 4.0 scrub starts 2024-09-11T08:08:44.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:44 smithi181 ceph-mon[32701]: 4.0 scrub ok 2024-09-11T08:08:45.438 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:08:45 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:08:45] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:08:45.442 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:45 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:08:45.442 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:45 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:08:45.442 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:45 smithi150 ceph-mon[29276]: 3.d deep-scrub starts 2024-09-11T08:08:45.442 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:45 smithi150 ceph-mon[29276]: 3.d deep-scrub ok 2024-09-11T08:08:45.442 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:45 smithi150 ceph-mon[29276]: osdmap e82: 8 total, 8 up, 8 in 2024-09-11T08:08:45.443 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:45 smithi150 ceph-mon[29276]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:08:45.443 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:45 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:08:45.443 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:45 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:08:45.443 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:45 smithi150 ceph-mon[39612]: 3.d deep-scrub starts 2024-09-11T08:08:45.443 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:45 smithi150 ceph-mon[39612]: 3.d deep-scrub ok 2024-09-11T08:08:45.443 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:45 smithi150 ceph-mon[39612]: osdmap e82: 8 total, 8 up, 8 in 2024-09-11T08:08:45.443 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:45 smithi150 ceph-mon[39612]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:08:45.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:45 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:08:45.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:45 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:08:45.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:45 smithi181 ceph-mon[32701]: 3.d deep-scrub starts 2024-09-11T08:08:45.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:45 smithi181 ceph-mon[32701]: 3.d deep-scrub ok 2024-09-11T08:08:45.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:45 smithi181 ceph-mon[32701]: osdmap e82: 8 total, 8 up, 8 in 2024-09-11T08:08:45.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:45 smithi181 ceph-mon[32701]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:08:46.050 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986205 got 133143986209 for osd.4 2024-09-11T08:08:46.051 DEBUG:teuthology.parallel:result is None 2024-09-11T08:08:46.692 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:46 smithi150 ceph-mon[39612]: 4.f scrub starts 2024-09-11T08:08:46.693 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:46 smithi150 ceph-mon[39612]: 4.f scrub ok 2024-09-11T08:08:46.693 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:46 smithi150 ceph-mon[39612]: 3.15 scrub starts 2024-09-11T08:08:46.693 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:46 smithi150 ceph-mon[39612]: 3.15 scrub ok 2024-09-11T08:08:46.693 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:46 smithi150 ceph-mon[29276]: 4.f scrub starts 2024-09-11T08:08:46.693 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:46 smithi150 ceph-mon[29276]: 4.f scrub ok 2024-09-11T08:08:46.693 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:46 smithi150 ceph-mon[29276]: 3.15 scrub starts 2024-09-11T08:08:46.693 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:46 smithi150 ceph-mon[29276]: 3.15 scrub ok 2024-09-11T08:08:46.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:46 smithi181 ceph-mon[32701]: 4.f scrub starts 2024-09-11T08:08:46.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:46 smithi181 ceph-mon[32701]: 4.f scrub ok 2024-09-11T08:08:46.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:46 smithi181 ceph-mon[32701]: 3.15 scrub starts 2024-09-11T08:08:46.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:46 smithi181 ceph-mon[32701]: 3.15 scrub ok 2024-09-11T08:08:47.222 INFO:tasks.cephadm.ceph_manager.ceph:need seq 227633266703 got 227633266706 for osd.7 2024-09-11T08:08:47.222 DEBUG:teuthology.parallel:result is None 2024-09-11T08:08:47.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:47 smithi150 ceph-mon[29276]: 6.5 scrub starts 2024-09-11T08:08:47.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:47 smithi150 ceph-mon[29276]: 6.5 scrub ok 2024-09-11T08:08:47.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:47 smithi150 ceph-mon[29276]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 810 B/s rd, 0 op/s 2024-09-11T08:08:47.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:47 smithi150 ceph-mon[29276]: 5.8 scrub starts 2024-09-11T08:08:47.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:47 smithi150 ceph-mon[29276]: 5.8 scrub ok 2024-09-11T08:08:47.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:47 smithi150 ceph-mon[39612]: 6.5 scrub starts 2024-09-11T08:08:47.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:47 smithi150 ceph-mon[39612]: 6.5 scrub ok 2024-09-11T08:08:47.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:47 smithi150 ceph-mon[39612]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 810 B/s rd, 0 op/s 2024-09-11T08:08:47.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:47 smithi150 ceph-mon[39612]: 5.8 scrub starts 2024-09-11T08:08:47.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:47 smithi150 ceph-mon[39612]: 5.8 scrub ok 2024-09-11T08:08:47.710 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:47 smithi181 ceph-mon[32701]: 6.5 scrub starts 2024-09-11T08:08:47.710 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:47 smithi181 ceph-mon[32701]: 6.5 scrub ok 2024-09-11T08:08:47.710 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:47 smithi181 ceph-mon[32701]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 810 B/s rd, 0 op/s 2024-09-11T08:08:47.710 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:47 smithi181 ceph-mon[32701]: 5.8 scrub starts 2024-09-11T08:08:47.710 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:47 smithi181 ceph-mon[32701]: 5.8 scrub ok 2024-09-11T08:08:48.073 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:08:47 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:08:48.622 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:48 smithi150 ceph-mon[29276]: 5.5 scrub starts 2024-09-11T08:08:48.622 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:48 smithi150 ceph-mon[29276]: 5.5 scrub ok 2024-09-11T08:08:48.622 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:48 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:08:48.622 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:48 smithi150 ceph-mon[29276]: 5.15 scrub starts 2024-09-11T08:08:48.622 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:48 smithi150 ceph-mon[29276]: 5.15 scrub ok 2024-09-11T08:08:48.623 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:48 smithi150 ceph-mon[29276]: 3.14 scrub starts 2024-09-11T08:08:48.623 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:48 smithi150 ceph-mon[29276]: 3.14 scrub ok 2024-09-11T08:08:48.623 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:48 smithi150 ceph-mon[39612]: 5.5 scrub starts 2024-09-11T08:08:48.623 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:48 smithi150 ceph-mon[39612]: 5.5 scrub ok 2024-09-11T08:08:48.623 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:48 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:08:48.623 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:48 smithi150 ceph-mon[39612]: 5.15 scrub starts 2024-09-11T08:08:48.623 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:48 smithi150 ceph-mon[39612]: 5.15 scrub ok 2024-09-11T08:08:48.623 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:48 smithi150 ceph-mon[39612]: 3.14 scrub starts 2024-09-11T08:08:48.623 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:48 smithi150 ceph-mon[39612]: 3.14 scrub ok 2024-09-11T08:08:48.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:48 smithi181 ceph-mon[32701]: 5.5 scrub starts 2024-09-11T08:08:48.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:48 smithi181 ceph-mon[32701]: 5.5 scrub ok 2024-09-11T08:08:48.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:48 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:08:48.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:48 smithi181 ceph-mon[32701]: 5.15 scrub starts 2024-09-11T08:08:48.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:48 smithi181 ceph-mon[32701]: 5.15 scrub ok 2024-09-11T08:08:48.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:48 smithi181 ceph-mon[32701]: 3.14 scrub starts 2024-09-11T08:08:48.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:48 smithi181 ceph-mon[32701]: 3.14 scrub ok 2024-09-11T08:08:49.672 INFO:teuthology.orchestra.run.smithi150.stdout:193273528345 2024-09-11T08:08:49.673 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:49 smithi150 ceph-mon[29276]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:08:49.673 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:49 smithi150 ceph-mon[29276]: 5.d scrub starts 2024-09-11T08:08:49.673 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:49 smithi150 ceph-mon[29276]: 5.d scrub ok 2024-09-11T08:08:49.674 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:49 smithi150 ceph-mon[39612]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:08:49.674 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:49 smithi150 ceph-mon[39612]: 5.d scrub starts 2024-09-11T08:08:49.674 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:49 smithi150 ceph-mon[39612]: 5.d scrub ok 2024-09-11T08:08:49.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:49 smithi181 ceph-mon[32701]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:08:49.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:49 smithi181 ceph-mon[32701]: 5.d scrub starts 2024-09-11T08:08:49.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:49 smithi181 ceph-mon[32701]: 5.d scrub ok 2024-09-11T08:08:49.882 INFO:teuthology.orchestra.run.smithi150.stdout:107374182440 2024-09-11T08:08:50.638 INFO:tasks.cephadm.ceph_manager.ceph:need seq 193273528340 got 193273528345 for osd.6 2024-09-11T08:08:50.639 DEBUG:teuthology.parallel:result is None 2024-09-11T08:08:50.731 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:50 smithi150 ceph-mon[29276]: 3.12 deep-scrub starts 2024-09-11T08:08:50.731 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:50 smithi150 ceph-mon[29276]: 3.12 deep-scrub ok 2024-09-11T08:08:50.731 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:50 smithi150 ceph-mon[29276]: 3.4 scrub starts 2024-09-11T08:08:50.731 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:50 smithi150 ceph-mon[29276]: 3.4 scrub ok 2024-09-11T08:08:50.731 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:50 smithi150 ceph-mon[29276]: 3.18 scrub starts 2024-09-11T08:08:50.731 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:50 smithi150 ceph-mon[29276]: 3.18 scrub ok 2024-09-11T08:08:50.731 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:50 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/38638227' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-11T08:08:50.731 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:50 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/4174888071' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-11T08:08:50.731 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:50 smithi150 ceph-mon[29276]: 4.15 scrub starts 2024-09-11T08:08:50.731 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:50 smithi150 ceph-mon[29276]: 4.15 scrub ok 2024-09-11T08:08:50.732 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:50 smithi150 ceph-mon[39612]: 3.12 deep-scrub starts 2024-09-11T08:08:50.732 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:50 smithi150 ceph-mon[39612]: 3.12 deep-scrub ok 2024-09-11T08:08:50.732 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:50 smithi150 ceph-mon[39612]: 3.4 scrub starts 2024-09-11T08:08:50.732 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:50 smithi150 ceph-mon[39612]: 3.4 scrub ok 2024-09-11T08:08:50.732 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:50 smithi150 ceph-mon[39612]: 3.18 scrub starts 2024-09-11T08:08:50.732 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:50 smithi150 ceph-mon[39612]: 3.18 scrub ok 2024-09-11T08:08:50.732 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:50 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/38638227' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-11T08:08:50.732 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:50 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/4174888071' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-11T08:08:50.732 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:50 smithi150 ceph-mon[39612]: 4.15 scrub starts 2024-09-11T08:08:50.732 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:50 smithi150 ceph-mon[39612]: 4.15 scrub ok 2024-09-11T08:08:50.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:50 smithi181 ceph-mon[32701]: 3.12 deep-scrub starts 2024-09-11T08:08:50.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:50 smithi181 ceph-mon[32701]: 3.12 deep-scrub ok 2024-09-11T08:08:50.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:50 smithi181 ceph-mon[32701]: 3.4 scrub starts 2024-09-11T08:08:50.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:50 smithi181 ceph-mon[32701]: 3.4 scrub ok 2024-09-11T08:08:50.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:50 smithi181 ceph-mon[32701]: 3.18 scrub starts 2024-09-11T08:08:50.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:50 smithi181 ceph-mon[32701]: 3.18 scrub ok 2024-09-11T08:08:50.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:50 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/38638227' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-11T08:08:50.825 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:50 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/4174888071' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-11T08:08:50.825 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:50 smithi181 ceph-mon[32701]: 4.15 scrub starts 2024-09-11T08:08:50.825 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:50 smithi181 ceph-mon[32701]: 4.15 scrub ok 2024-09-11T08:08:51.042 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182435 got 107374182440 for osd.3 2024-09-11T08:08:51.042 DEBUG:teuthology.parallel:result is None 2024-09-11T08:08:51.042 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-11T08:08:51.042 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T08:08:51.358 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:08:51 smithi150 podman[72074]: 2024-09-11 08:08:51.248629879 +0000 UTC m=+0.119483949 volume create 7ccd31928cc244156dab9e44a5267acbdea4b7d35440473dce2520795f0a2eab 2024-09-11T08:08:51.358 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:08:51 smithi150 podman[72074]: 2024-09-11 08:08:51.357832937 +0000 UTC m=+0.228687008 container create 6af74a554f48afed6100272abc1c40bfd7c00cbf6577f093ddd620284b580f96 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-11T08:08:51.719 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:51 smithi150 ceph-mon[29276]: 6.4 scrub starts 2024-09-11T08:08:51.720 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:51 smithi150 ceph-mon[29276]: 6.4 scrub ok 2024-09-11T08:08:51.720 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:51 smithi150 ceph-mon[29276]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-11T08:08:51.720 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:51 smithi150 ceph-mon[29276]: 3.1a scrub starts 2024-09-11T08:08:51.720 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:51 smithi150 ceph-mon[29276]: 3.1a scrub ok 2024-09-11T08:08:51.720 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:51 smithi150 ceph-mon[39612]: 6.4 scrub starts 2024-09-11T08:08:51.720 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:51 smithi150 ceph-mon[39612]: 6.4 scrub ok 2024-09-11T08:08:51.720 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:51 smithi150 ceph-mon[39612]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-11T08:08:51.720 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:51 smithi150 ceph-mon[39612]: 3.1a scrub starts 2024-09-11T08:08:51.720 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:51 smithi150 ceph-mon[39612]: 3.1a scrub ok 2024-09-11T08:08:51.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:51 smithi181 ceph-mon[32701]: 6.4 scrub starts 2024-09-11T08:08:51.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:51 smithi181 ceph-mon[32701]: 6.4 scrub ok 2024-09-11T08:08:51.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:51 smithi181 ceph-mon[32701]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-11T08:08:51.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:51 smithi181 ceph-mon[32701]: 3.1a scrub starts 2024-09-11T08:08:51.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:51 smithi181 ceph-mon[32701]: 3.1a scrub ok 2024-09-11T08:08:51.952 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:08:51.989 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:08:51 smithi150 podman[72074]: 2024-09-11 08:08:51.719258562 +0000 UTC m=+0.590112632 container init 6af74a554f48afed6100272abc1c40bfd7c00cbf6577f093ddd620284b580f96 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-11T08:08:51.989 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:08:51 smithi150 podman[72074]: 2024-09-11 08:08:51.722487548 +0000 UTC m=+0.593341629 container start 6af74a554f48afed6100272abc1c40bfd7c00cbf6577f093ddd620284b580f96 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-11T08:08:51.989 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:08:51 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-alertmanager-a[72111]: ts=2024-09-11T08:08:51.734Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-09-11T08:08:51.989 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:08:51 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-alertmanager-a[72111]: ts=2024-09-11T08:08:51.734Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-09-11T08:08:51.989 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:08:51 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-alertmanager-a[72111]: ts=2024-09-11T08:08:51.735Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.150 port=9094 2024-09-11T08:08:51.989 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:08:51 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-alertmanager-a[72111]: ts=2024-09-11T08:08:51.736Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-11T08:08:51.990 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:08:51 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-alertmanager-a[72111]: ts=2024-09-11T08:08:51.759Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-11T08:08:51.990 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:08:51 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-alertmanager-a[72111]: ts=2024-09-11T08:08:51.760Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-11T08:08:51.990 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:08:51 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-alertmanager-a[72111]: ts=2024-09-11T08:08:51.763Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-09-11T08:08:51.990 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:08:51 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-alertmanager-a[72111]: ts=2024-09-11T08:08:51.763Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-11T08:08:51.990 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:08:51 smithi150 bash[72074]: 6af74a554f48afed6100272abc1c40bfd7c00cbf6577f093ddd620284b580f96 2024-09-11T08:08:51.990 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:08:51 smithi150 systemd[1]: Started Ceph alertmanager.a for c1bddff4-7013-11ef-bcea-c7b262605968. 2024-09-11T08:08:52.315 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:51 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: [11/Sep/2024:08:08:51] ENGINE Bus STOPPING 2024-09-11T08:08:52.590 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:52 smithi150 ceph-mon[39612]: 5.4 deep-scrub starts 2024-09-11T08:08:52.590 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:52 smithi150 ceph-mon[39612]: 5.4 deep-scrub ok 2024-09-11T08:08:52.590 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:52 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:08:52.590 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:52 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:52.590 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:52 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:52.590 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:52 smithi150 ceph-mon[39612]: 6.d scrub starts 2024-09-11T08:08:52.590 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:52 smithi150 ceph-mon[39612]: 6.d scrub ok 2024-09-11T08:08:52.590 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:52 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:52.591 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:52 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:52.591 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:52 smithi150 ceph-mon[39612]: 5.b scrub starts 2024-09-11T08:08:52.591 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:52 smithi150 ceph-mon[39612]: 5.b scrub ok 2024-09-11T08:08:52.591 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:52 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:52.591 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:52 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:52.591 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:52 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-11T08:08:52.591 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:52 smithi150 ceph-mon[39612]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-11T08:08:52.591 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:52 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:52.591 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:52 smithi150 ceph-mon[39612]: Deploying daemon grafana.a on smithi181 2024-09-11T08:08:52.591 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:52 smithi150 ceph-mon[29276]: 5.4 deep-scrub starts 2024-09-11T08:08:52.591 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:52 smithi150 ceph-mon[29276]: 5.4 deep-scrub ok 2024-09-11T08:08:52.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:52 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:08:52.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:52 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:52.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:52 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:52.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:52 smithi150 ceph-mon[29276]: 6.d scrub starts 2024-09-11T08:08:52.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:52 smithi150 ceph-mon[29276]: 6.d scrub ok 2024-09-11T08:08:52.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:52 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:52.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:52 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:52.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:52 smithi150 ceph-mon[29276]: 5.b scrub starts 2024-09-11T08:08:52.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:52 smithi150 ceph-mon[29276]: 5.b scrub ok 2024-09-11T08:08:52.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:52 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:52.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:52 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:52.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:52 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-11T08:08:52.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:52 smithi150 ceph-mon[29276]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-11T08:08:52.593 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:52 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:52.593 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:52 smithi150 ceph-mon[29276]: Deploying daemon grafana.a on smithi181 2024-09-11T08:08:52.593 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:52 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: [11/Sep/2024:08:08:52] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-11T08:08:52.593 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:52 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: [11/Sep/2024:08:08:52] ENGINE Bus STOPPED 2024-09-11T08:08:52.593 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:52 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: [11/Sep/2024:08:08:52] ENGINE Bus STARTING 2024-09-11T08:08:52.593 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:52 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: [11/Sep/2024:08:08:52] ENGINE Serving on http://:::9283 2024-09-11T08:08:52.593 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:08:52 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: [11/Sep/2024:08:08:52] ENGINE Bus STARTED 2024-09-11T08:08:52.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:52 smithi181 ceph-mon[32701]: 5.4 deep-scrub starts 2024-09-11T08:08:52.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:52 smithi181 ceph-mon[32701]: 5.4 deep-scrub ok 2024-09-11T08:08:52.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:52 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:08:52.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:52 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:52.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:52 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:52.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:52 smithi181 ceph-mon[32701]: 6.d scrub starts 2024-09-11T08:08:52.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:52 smithi181 ceph-mon[32701]: 6.d scrub ok 2024-09-11T08:08:52.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:52 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:52.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:52 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:52.825 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:52 smithi181 ceph-mon[32701]: 5.b scrub starts 2024-09-11T08:08:52.825 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:52 smithi181 ceph-mon[32701]: 5.b scrub ok 2024-09-11T08:08:52.825 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:52 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:52.825 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:52 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:52.825 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:52 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-11T08:08:52.825 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:52 smithi181 ceph-mon[32701]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-11T08:08:52.825 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:52 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:52.825 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:52 smithi181 ceph-mon[32701]: Deploying daemon grafana.a on smithi181 2024-09-11T08:08:53.640 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:53 smithi150 ceph-mon[29276]: 5.14 scrub starts 2024-09-11T08:08:53.640 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:53 smithi150 ceph-mon[29276]: 5.14 scrub ok 2024-09-11T08:08:53.641 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:53 smithi150 ceph-mon[29276]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s 2024-09-11T08:08:53.641 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:53 smithi150 ceph-mon[39612]: 5.14 scrub starts 2024-09-11T08:08:53.641 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:53 smithi150 ceph-mon[39612]: 5.14 scrub ok 2024-09-11T08:08:53.641 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:53 smithi150 ceph-mon[39612]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s 2024-09-11T08:08:53.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:53 smithi181 ceph-mon[32701]: 5.14 scrub starts 2024-09-11T08:08:53.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:53 smithi181 ceph-mon[32701]: 5.14 scrub ok 2024-09-11T08:08:53.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:53 smithi181 ceph-mon[32701]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s 2024-09-11T08:08:53.892 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:08:53.893 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:08:53 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-alertmanager-a[72111]: ts=2024-09-11T08:08:53.736Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000135121s 2024-09-11T08:08:53.896 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-09-11T08:08:54.502 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2024-09-11T08:08:52.689329+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596188,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":929,"num_read_kb":786,"num_write":543,"num_write_kb":1671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":556,"ondisk_log_size":556,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2387160,"kb_used_data":8340,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":747574056,"statfs":{"total":767960285184,"available":765515833344,"internally_reserved":0,"allocated":8540160,"data_stored":4556839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.018201"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.435521+0000","last_change":"2024-09-11T08:08:36.105232+0000","last_active":"2024-09-11T08:08:41.435521+0000","last_peered":"2024-09-11T08:08:41.435521+0000","last_clean":"2024-09-11T08:08:41.435521+0000","last_became_active":"2024-09-11T08:07:30.044976+0000","last_became_peered":"2024-09-11T08:07:30.044976+0000","last_unstale":"2024-09-11T08:08:41.435521+0000","last_undegraded":"2024-09-11T08:08:41.435521+0000","last_fullsized":"2024-09-11T08:08:41.435521+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:36.105175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:36.105175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:56.596929+0000","scrub_duration":0.0016664970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":46,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.413093+0000","last_change":"2024-09-11T08:07:32.048578+0000","last_active":"2024-09-11T08:08:41.413093+0000","last_peered":"2024-09-11T08:08:41.413093+0000","last_clean":"2024-09-11T08:08:41.413093+0000","last_became_active":"2024-09-11T08:07:32.048048+0000","last_became_peered":"2024-09-11T08:07:32.048048+0000","last_unstale":"2024-09-11T08:08:41.413093+0000","last_undegraded":"2024-09-11T08:08:41.413093+0000","last_fullsized":"2024-09-11T08:08:41.413093+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:58.436753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":38,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.415054+0000","last_change":"2024-09-11T08:08:38.486126+0000","last_active":"2024-09-11T08:08:41.415054+0000","last_peered":"2024-09-11T08:08:41.415054+0000","last_clean":"2024-09-11T08:08:41.415054+0000","last_became_active":"2024-09-11T08:07:34.058828+0000","last_became_peered":"2024-09-11T08:07:34.058828+0000","last_unstale":"2024-09-11T08:08:41.415054+0000","last_undegraded":"2024-09-11T08:08:41.415054+0000","last_fullsized":"2024-09-11T08:08:41.415054+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:38.486057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:38.486057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:44.914804+0000","scrub_duration":0.002230531,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.413045+0000","last_change":"2024-09-11T08:07:36.075118+0000","last_active":"2024-09-11T08:08:41.413045+0000","last_peered":"2024-09-11T08:08:41.413045+0000","last_clean":"2024-09-11T08:08:41.413045+0000","last_became_active":"2024-09-11T08:07:36.074789+0000","last_became_peered":"2024-09-11T08:07:36.074789+0000","last_unstale":"2024-09-11T08:08:41.413045+0000","last_undegraded":"2024-09-11T08:08:41.413045+0000","last_fullsized":"2024-09-11T08:08:41.413045+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.414207+0000","last_change":"2024-09-11T08:07:36.633716+0000","last_active":"2024-09-11T08:08:41.414207+0000","last_peered":"2024-09-11T08:08:41.414207+0000","last_clean":"2024-09-11T08:08:41.414207+0000","last_became_active":"2024-09-11T08:07:36.633423+0000","last_became_peered":"2024-09-11T08:07:36.633423+0000","last_unstale":"2024-09-11T08:08:41.414207+0000","last_undegraded":"2024-09-11T08:08:41.414207+0000","last_fullsized":"2024-09-11T08:08:41.414207+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.414800+0000","last_change":"2024-09-11T08:07:30.045845+0000","last_active":"2024-09-11T08:08:41.414800+0000","last_peered":"2024-09-11T08:08:41.414800+0000","last_clean":"2024-09-11T08:08:41.414800+0000","last_became_active":"2024-09-11T08:07:30.045579+0000","last_became_peered":"2024-09-11T08:07:30.045579+0000","last_unstale":"2024-09-11T08:08:41.414800+0000","last_undegraded":"2024-09-11T08:08:41.414800+0000","last_fullsized":"2024-09-11T08:08:41.414800+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:51.248162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":55,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.414362+0000","last_change":"2024-09-11T08:07:32.047593+0000","last_active":"2024-09-11T08:08:41.414362+0000","last_peered":"2024-09-11T08:08:41.414362+0000","last_clean":"2024-09-11T08:08:41.414362+0000","last_became_active":"2024-09-11T08:07:32.045848+0000","last_became_peered":"2024-09-11T08:07:32.045848+0000","last_unstale":"2024-09-11T08:08:41.414362+0000","last_undegraded":"2024-09-11T08:08:41.414362+0000","last_fullsized":"2024-09-11T08:08:41.414362+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.412943+0000","last_change":"2024-09-11T08:07:34.053483+0000","last_active":"2024-09-11T08:08:41.412943+0000","last_peered":"2024-09-11T08:08:41.412943+0000","last_clean":"2024-09-11T08:08:41.412943+0000","last_became_active":"2024-09-11T08:07:34.052243+0000","last_became_peered":"2024-09-11T08:07:34.052243+0000","last_unstale":"2024-09-11T08:08:41.412943+0000","last_undegraded":"2024-09-11T08:08:41.412943+0000","last_fullsized":"2024-09-11T08:08:41.412943+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:00.987472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.055146+0000","last_change":"2024-09-11T08:07:30.044353+0000","last_active":"2024-09-11T08:08:42.055146+0000","last_peered":"2024-09-11T08:08:42.055146+0000","last_clean":"2024-09-11T08:08:42.055146+0000","last_became_active":"2024-09-11T08:07:30.043950+0000","last_became_peered":"2024-09-11T08:07:30.043950+0000","last_unstale":"2024-09-11T08:08:42.055146+0000","last_undegraded":"2024-09-11T08:08:42.055146+0000","last_fullsized":"2024-09-11T08:08:42.055146+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:45.265565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":46,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.413406+0000","last_change":"2024-09-11T08:07:32.044928+0000","last_active":"2024-09-11T08:08:41.413406+0000","last_peered":"2024-09-11T08:08:41.413406+0000","last_clean":"2024-09-11T08:08:41.413406+0000","last_became_active":"2024-09-11T08:07:32.044626+0000","last_became_peered":"2024-09-11T08:07:32.044626+0000","last_unstale":"2024-09-11T08:08:41.413406+0000","last_undegraded":"2024-09-11T08:08:41.413406+0000","last_fullsized":"2024-09-11T08:08:41.413406+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.055594+0000","last_change":"2024-09-11T08:07:34.061363+0000","last_active":"2024-09-11T08:08:42.055594+0000","last_peered":"2024-09-11T08:08:42.055594+0000","last_clean":"2024-09-11T08:08:42.055594+0000","last_became_active":"2024-09-11T08:07:34.058689+0000","last_became_peered":"2024-09-11T08:07:34.058689+0000","last_unstale":"2024-09-11T08:08:42.055594+0000","last_undegraded":"2024-09-11T08:08:42.055594+0000","last_fullsized":"2024-09-11T08:08:42.055594+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:01.558153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.435255+0000","last_change":"2024-09-11T08:08:40.071203+0000","last_active":"2024-09-11T08:08:41.435255+0000","last_peered":"2024-09-11T08:08:41.435255+0000","last_clean":"2024-09-11T08:08:41.435255+0000","last_became_active":"2024-09-11T08:07:36.076824+0000","last_became_peered":"2024-09-11T08:07:36.076824+0000","last_unstale":"2024-09-11T08:08:41.435255+0000","last_undegraded":"2024-09-11T08:08:41.435255+0000","last_fullsized":"2024-09-11T08:08:41.435255+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:40.071106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:08:40.071106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:54.333357+0000","scrub_duration":0.0019674499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.054968+0000","last_change":"2024-09-11T08:08:36.825857+0000","last_active":"2024-09-11T08:08:42.054968+0000","last_peered":"2024-09-11T08:08:42.054968+0000","last_clean":"2024-09-11T08:08:42.054968+0000","last_became_active":"2024-09-11T08:07:30.043277+0000","last_became_peered":"2024-09-11T08:07:30.043277+0000","last_unstale":"2024-09-11T08:08:42.054968+0000","last_undegraded":"2024-09-11T08:08:42.054968+0000","last_fullsized":"2024-09-11T08:08:42.054968+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:36.825794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:36.825794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:47.493434+0000","scrub_duration":0.0016805920000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":40,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.413068+0000","last_change":"2024-09-11T08:07:32.048969+0000","last_active":"2024-09-11T08:08:41.413068+0000","last_peered":"2024-09-11T08:08:41.413068+0000","last_clean":"2024-09-11T08:08:41.413068+0000","last_became_active":"2024-09-11T08:07:32.048661+0000","last_became_peered":"2024-09-11T08:07:32.048661+0000","last_unstale":"2024-09-11T08:08:41.413068+0000","last_undegraded":"2024-09-11T08:08:41.413068+0000","last_fullsized":"2024-09-11T08:08:41.413068+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:44.853813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.377991+0000","last_change":"2024-09-11T08:07:34.061194+0000","last_active":"2024-09-11T08:08:42.377991+0000","last_peered":"2024-09-11T08:08:42.377991+0000","last_clean":"2024-09-11T08:08:42.377991+0000","last_became_active":"2024-09-11T08:07:34.060871+0000","last_became_peered":"2024-09-11T08:07:34.060871+0000","last_unstale":"2024-09-11T08:08:42.377991+0000","last_undegraded":"2024-09-11T08:08:42.377991+0000","last_fullsized":"2024-09-11T08:08:42.377991+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:50.814439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.055345+0000","last_change":"2024-09-11T08:07:36.077890+0000","last_active":"2024-09-11T08:08:42.055345+0000","last_peered":"2024-09-11T08:08:42.055345+0000","last_clean":"2024-09-11T08:08:42.055345+0000","last_became_active":"2024-09-11T08:07:36.077602+0000","last_became_peered":"2024-09-11T08:07:36.077602+0000","last_unstale":"2024-09-11T08:08:42.055345+0000","last_undegraded":"2024-09-11T08:08:42.055345+0000","last_fullsized":"2024-09-11T08:08:42.055345+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:56.644431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.413616+0000","last_change":"2024-09-11T08:07:36.633951+0000","last_active":"2024-09-11T08:08:41.413616+0000","last_peered":"2024-09-11T08:08:41.413616+0000","last_clean":"2024-09-11T08:08:41.413616+0000","last_became_active":"2024-09-11T08:07:36.633338+0000","last_became_peered":"2024-09-11T08:07:36.633338+0000","last_unstale":"2024-09-11T08:08:41.413616+0000","last_undegraded":"2024-09-11T08:08:41.413616+0000","last_fullsized":"2024-09-11T08:08:41.413616+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:56.348854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.435745+0000","last_change":"2024-09-11T08:07:30.046004+0000","last_active":"2024-09-11T08:08:41.435745+0000","last_peered":"2024-09-11T08:08:41.435745+0000","last_clean":"2024-09-11T08:08:41.435745+0000","last_became_active":"2024-09-11T08:07:30.045739+0000","last_became_peered":"2024-09-11T08:07:30.045739+0000","last_unstale":"2024-09-11T08:08:41.435745+0000","last_undegraded":"2024-09-11T08:08:41.435745+0000","last_fullsized":"2024-09-11T08:08:41.435745+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:51.187269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"66'15","reported_seq":54,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.433511+0000","last_change":"2024-09-11T08:07:32.046658+0000","last_active":"2024-09-11T08:08:41.433511+0000","last_peered":"2024-09-11T08:08:41.433511+0000","last_clean":"2024-09-11T08:08:41.433511+0000","last_became_active":"2024-09-11T08:07:32.046363+0000","last_became_peered":"2024-09-11T08:07:32.046363+0000","last_unstale":"2024-09-11T08:08:41.433511+0000","last_undegraded":"2024-09-11T08:08:41.433511+0000","last_fullsized":"2024-09-11T08:08:41.433511+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:48.322062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.415156+0000","last_change":"2024-09-11T08:07:34.056221+0000","last_active":"2024-09-11T08:08:41.415156+0000","last_peered":"2024-09-11T08:08:41.415156+0000","last_clean":"2024-09-11T08:08:41.415156+0000","last_became_active":"2024-09-11T08:07:34.055410+0000","last_became_peered":"2024-09-11T08:07:34.055410+0000","last_unstale":"2024-09-11T08:08:41.415156+0000","last_undegraded":"2024-09-11T08:08:41.415156+0000","last_fullsized":"2024-09-11T08:08:41.415156+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:49.556611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.415115+0000","last_change":"2024-09-11T08:07:36.633626+0000","last_active":"2024-09-11T08:08:41.415115+0000","last_peered":"2024-09-11T08:08:41.415115+0000","last_clean":"2024-09-11T08:08:41.415115+0000","last_became_active":"2024-09-11T08:07:36.633258+0000","last_became_peered":"2024-09-11T08:07:36.633258+0000","last_unstale":"2024-09-11T08:08:41.415115+0000","last_undegraded":"2024-09-11T08:08:41.415115+0000","last_fullsized":"2024-09-11T08:08:41.415115+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:01.540476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":42,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-11T08:08:51.401988+0000","last_change":"2024-09-11T08:08:51.401929+0000","last_active":"2024-09-11T08:08:51.401988+0000","last_peered":"2024-09-11T08:08:51.401988+0000","last_clean":"2024-09-11T08:08:51.401988+0000","last_became_active":"2024-09-11T08:07:30.046284+0000","last_became_peered":"2024-09-11T08:07:30.046284+0000","last_unstale":"2024-09-11T08:08:51.401988+0000","last_undegraded":"2024-09-11T08:08:51.401988+0000","last_fullsized":"2024-09-11T08:08:51.401988+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:51.401864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:51.401864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:18.688155+0000","scrub_duration":0.001460034,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":53,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.415141+0000","last_change":"2024-09-11T08:07:32.047173+0000","last_active":"2024-09-11T08:08:41.415141+0000","last_peered":"2024-09-11T08:08:41.415141+0000","last_clean":"2024-09-11T08:08:41.415141+0000","last_became_active":"2024-09-11T08:07:32.046737+0000","last_became_peered":"2024-09-11T08:07:32.046737+0000","last_unstale":"2024-09-11T08:08:41.415141+0000","last_undegraded":"2024-09-11T08:08:41.415141+0000","last_fullsized":"2024-09-11T08:08:41.415141+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:52.570790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":38,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.413364+0000","last_change":"2024-09-11T08:08:40.419900+0000","last_active":"2024-09-11T08:08:41.413364+0000","last_peered":"2024-09-11T08:08:41.413364+0000","last_clean":"2024-09-11T08:08:41.413364+0000","last_became_active":"2024-09-11T08:07:34.056528+0000","last_became_peered":"2024-09-11T08:07:34.056528+0000","last_unstale":"2024-09-11T08:08:41.413364+0000","last_undegraded":"2024-09-11T08:08:41.413364+0000","last_fullsized":"2024-09-11T08:08:41.413364+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:40.419830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:40.419830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:09.111095+0000","scrub_duration":0.0029308149999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.378307+0000","last_change":"2024-09-11T08:07:36.072658+0000","last_active":"2024-09-11T08:08:42.378307+0000","last_peered":"2024-09-11T08:08:42.378307+0000","last_clean":"2024-09-11T08:08:42.378307+0000","last_became_active":"2024-09-11T08:07:36.072347+0000","last_became_peered":"2024-09-11T08:07:36.072347+0000","last_unstale":"2024-09-11T08:08:42.378307+0000","last_undegraded":"2024-09-11T08:08:42.378307+0000","last_fullsized":"2024-09-11T08:08:42.378307+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:57.979955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.415125+0000","last_change":"2024-09-11T08:07:30.046498+0000","last_active":"2024-09-11T08:08:41.415125+0000","last_peered":"2024-09-11T08:08:41.415125+0000","last_clean":"2024-09-11T08:08:41.415125+0000","last_became_active":"2024-09-11T08:07:30.045932+0000","last_became_peered":"2024-09-11T08:07:30.045932+0000","last_unstale":"2024-09-11T08:08:41.415125+0000","last_undegraded":"2024-09-11T08:08:41.415125+0000","last_fullsized":"2024-09-11T08:08:41.415125+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:56.901581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":45,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.435563+0000","last_change":"2024-09-11T08:07:32.048372+0000","last_active":"2024-09-11T08:08:41.435563+0000","last_peered":"2024-09-11T08:08:41.435563+0000","last_clean":"2024-09-11T08:08:41.435563+0000","last_became_active":"2024-09-11T08:07:32.048022+0000","last_became_peered":"2024-09-11T08:07:32.048022+0000","last_unstale":"2024-09-11T08:08:41.435563+0000","last_undegraded":"2024-09-11T08:08:41.435563+0000","last_fullsized":"2024-09-11T08:08:41.435563+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:52.163263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":54,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-11T08:08:45.102996+0000","last_change":"2024-09-11T08:07:34.055014+0000","last_active":"2024-09-11T08:08:45.102996+0000","last_peered":"2024-09-11T08:08:45.102996+0000","last_clean":"2024-09-11T08:08:45.102996+0000","last_became_active":"2024-09-11T08:07:34.054014+0000","last_became_peered":"2024-09-11T08:07:34.054014+0000","last_unstale":"2024-09-11T08:08:45.102996+0000","last_undegraded":"2024-09-11T08:08:45.102996+0000","last_fullsized":"2024-09-11T08:08:45.102996+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":65,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-11T08:08:45.358132+0000","last_change":"2024-09-11T08:08:45.358105+0000","last_active":"2024-09-11T08:08:45.358132+0000","last_peered":"2024-09-11T08:08:45.358132+0000","last_clean":"2024-09-11T08:08:45.358132+0000","last_became_active":"2024-09-11T08:07:32.054317+0000","last_became_peered":"2024-09-11T08:07:32.054317+0000","last_unstale":"2024-09-11T08:08:45.358132+0000","last_undegraded":"2024-09-11T08:08:45.358132+0000","last_fullsized":"2024-09-11T08:08:45.358132+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-11T08:08:45.358041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:45.358041+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:50.858524+0000","scrub_duration":0.0021022879999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.414788+0000","last_change":"2024-09-11T08:07:30.044933+0000","last_active":"2024-09-11T08:08:41.414788+0000","last_peered":"2024-09-11T08:08:41.414788+0000","last_clean":"2024-09-11T08:08:41.414788+0000","last_became_active":"2024-09-11T08:07:30.044624+0000","last_became_peered":"2024-09-11T08:07:30.044624+0000","last_unstale":"2024-09-11T08:08:41.414788+0000","last_undegraded":"2024-09-11T08:08:41.414788+0000","last_fullsized":"2024-09-11T08:08:41.414788+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:58.050628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":52,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-11T08:08:44.433741+0000","last_change":"2024-09-11T08:07:34.053292+0000","last_active":"2024-09-11T08:08:44.433741+0000","last_peered":"2024-09-11T08:08:44.433741+0000","last_clean":"2024-09-11T08:08:44.433741+0000","last_became_active":"2024-09-11T08:07:34.052981+0000","last_became_peered":"2024-09-11T08:07:34.052981+0000","last_unstale":"2024-09-11T08:08:44.433741+0000","last_undegraded":"2024-09-11T08:08:44.433741+0000","last_fullsized":"2024-09-11T08:08:44.433741+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-11T08:08:51.893893+0000","last_change":"2024-09-11T08:08:51.893846+0000","last_active":"2024-09-11T08:08:51.893893+0000","last_peered":"2024-09-11T08:08:51.893893+0000","last_clean":"2024-09-11T08:08:51.893893+0000","last_became_active":"2024-09-11T08:07:36.073969+0000","last_became_peered":"2024-09-11T08:07:36.073969+0000","last_unstale":"2024-09-11T08:08:51.893893+0000","last_undegraded":"2024-09-11T08:08:51.893893+0000","last_fullsized":"2024-09-11T08:08:51.893893+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:51.893774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:08:51.893774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:00.876260+0000","scrub_duration":0.001532858,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":72,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:43.441659+0000","last_change":"2024-09-11T08:08:43.441600+0000","last_active":"2024-09-11T08:08:43.441659+0000","last_peered":"2024-09-11T08:08:43.441659+0000","last_clean":"2024-09-11T08:08:43.441659+0000","last_became_active":"2024-09-11T08:07:32.045332+0000","last_became_peered":"2024-09-11T08:07:32.045332+0000","last_unstale":"2024-09-11T08:08:43.441659+0000","last_undegraded":"2024-09-11T08:08:43.441659+0000","last_fullsized":"2024-09-11T08:08:43.441659+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-11T08:08:43.441544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:43.441544+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:55.649394+0000","scrub_duration":0.002168648,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.415443+0000","last_change":"2024-09-11T08:07:30.047446+0000","last_active":"2024-09-11T08:08:41.415443+0000","last_peered":"2024-09-11T08:08:41.415443+0000","last_clean":"2024-09-11T08:08:41.415443+0000","last_became_active":"2024-09-11T08:07:30.047176+0000","last_became_peered":"2024-09-11T08:07:30.047176+0000","last_unstale":"2024-09-11T08:08:41.415443+0000","last_undegraded":"2024-09-11T08:08:41.415443+0000","last_fullsized":"2024-09-11T08:08:41.415443+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:58.438972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.412919+0000","last_change":"2024-09-11T08:07:34.055816+0000","last_active":"2024-09-11T08:08:41.412919+0000","last_peered":"2024-09-11T08:08:41.412919+0000","last_clean":"2024-09-11T08:08:41.412919+0000","last_became_active":"2024-09-11T08:07:34.055523+0000","last_became_peered":"2024-09-11T08:07:34.055523+0000","last_unstale":"2024-09-11T08:08:41.412919+0000","last_undegraded":"2024-09-11T08:08:41.412919+0000","last_fullsized":"2024-09-11T08:08:41.412919+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:57.216258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.413201+0000","last_change":"2024-09-11T08:07:36.069535+0000","last_active":"2024-09-11T08:08:41.413201+0000","last_peered":"2024-09-11T08:08:41.413201+0000","last_clean":"2024-09-11T08:08:41.413201+0000","last_became_active":"2024-09-11T08:07:36.069225+0000","last_became_peered":"2024-09-11T08:07:36.069225+0000","last_unstale":"2024-09-11T08:08:41.413201+0000","last_undegraded":"2024-09-11T08:08:41.413201+0000","last_fullsized":"2024-09-11T08:08:41.413201+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:57.222586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":51,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.413212+0000","last_change":"2024-09-11T08:07:32.040334+0000","last_active":"2024-09-11T08:08:41.413212+0000","last_peered":"2024-09-11T08:08:41.413212+0000","last_clean":"2024-09-11T08:08:41.413212+0000","last_became_active":"2024-09-11T08:07:32.040069+0000","last_became_peered":"2024-09-11T08:07:32.040069+0000","last_unstale":"2024-09-11T08:08:41.413212+0000","last_undegraded":"2024-09-11T08:08:41.413212+0000","last_fullsized":"2024-09-11T08:08:41.413212+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:48.042800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.435679+0000","last_change":"2024-09-11T08:07:30.050806+0000","last_active":"2024-09-11T08:08:41.435679+0000","last_peered":"2024-09-11T08:08:41.435679+0000","last_clean":"2024-09-11T08:08:41.435679+0000","last_became_active":"2024-09-11T08:07:30.050484+0000","last_became_peered":"2024-09-11T08:07:30.050484+0000","last_unstale":"2024-09-11T08:08:41.435679+0000","last_undegraded":"2024-09-11T08:08:41.435679+0000","last_fullsized":"2024-09-11T08:08:41.435679+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:47.129727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"66'8","reported_seq":55,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-11T08:08:44.434674+0000","last_change":"2024-09-11T08:07:34.056427+0000","last_active":"2024-09-11T08:08:44.434674+0000","last_peered":"2024-09-11T08:08:44.434674+0000","last_clean":"2024-09-11T08:08:44.434674+0000","last_became_active":"2024-09-11T08:07:34.056143+0000","last_became_peered":"2024-09-11T08:07:34.056143+0000","last_unstale":"2024-09-11T08:08:44.434674+0000","last_undegraded":"2024-09-11T08:08:44.434674+0000","last_fullsized":"2024-09-11T08:08:44.434674+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.378087+0000","last_change":"2024-09-11T08:07:36.633730+0000","last_active":"2024-09-11T08:08:42.378087+0000","last_peered":"2024-09-11T08:08:42.378087+0000","last_clean":"2024-09-11T08:08:42.378087+0000","last_became_active":"2024-09-11T08:07:36.633432+0000","last_became_peered":"2024-09-11T08:07:36.633432+0000","last_unstale":"2024-09-11T08:08:42.378087+0000","last_undegraded":"2024-09-11T08:08:42.378087+0000","last_fullsized":"2024-09-11T08:08:42.378087+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":45,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.415406+0000","last_change":"2024-09-11T08:07:32.054861+0000","last_active":"2024-09-11T08:08:41.415406+0000","last_peered":"2024-09-11T08:08:41.415406+0000","last_clean":"2024-09-11T08:08:41.415406+0000","last_became_active":"2024-09-11T08:07:32.054294+0000","last_became_peered":"2024-09-11T08:07:32.054294+0000","last_unstale":"2024-09-11T08:08:41.415406+0000","last_undegraded":"2024-09-11T08:08:41.415406+0000","last_fullsized":"2024-09-11T08:08:41.415406+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:00.767425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.054950+0000","last_change":"2024-09-11T08:07:30.047746+0000","last_active":"2024-09-11T08:08:42.054950+0000","last_peered":"2024-09-11T08:08:42.054950+0000","last_clean":"2024-09-11T08:08:42.054950+0000","last_became_active":"2024-09-11T08:07:30.047497+0000","last_became_peered":"2024-09-11T08:07:30.047497+0000","last_unstale":"2024-09-11T08:08:42.054950+0000","last_undegraded":"2024-09-11T08:08:42.054950+0000","last_fullsized":"2024-09-11T08:08:42.054950+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:54.652323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":52,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-11T08:08:45.103228+0000","last_change":"2024-09-11T08:07:34.060999+0000","last_active":"2024-09-11T08:08:45.103228+0000","last_peered":"2024-09-11T08:08:45.103228+0000","last_clean":"2024-09-11T08:08:45.103228+0000","last_became_active":"2024-09-11T08:07:34.060660+0000","last_became_peered":"2024-09-11T08:07:34.060660+0000","last_unstale":"2024-09-11T08:08:45.103228+0000","last_undegraded":"2024-09-11T08:08:45.103228+0000","last_fullsized":"2024-09-11T08:08:45.103228+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:54.412248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"66'1","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.435044+0000","last_change":"2024-09-11T08:07:36.072890+0000","last_active":"2024-09-11T08:08:41.435044+0000","last_peered":"2024-09-11T08:08:41.435044+0000","last_clean":"2024-09-11T08:08:41.435044+0000","last_became_active":"2024-09-11T08:07:36.072593+0000","last_became_peered":"2024-09-11T08:07:36.072593+0000","last_unstale":"2024-09-11T08:08:41.435044+0000","last_undegraded":"2024-09-11T08:08:41.435044+0000","last_fullsized":"2024-09-11T08:08:41.435044+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:58.785592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":66,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.435613+0000","last_change":"2024-09-11T08:07:32.044836+0000","last_active":"2024-09-11T08:08:41.435613+0000","last_peered":"2024-09-11T08:08:41.435613+0000","last_clean":"2024-09-11T08:08:41.435613+0000","last_became_active":"2024-09-11T08:07:32.044376+0000","last_became_peered":"2024-09-11T08:07:32.044376+0000","last_unstale":"2024-09-11T08:08:41.435613+0000","last_undegraded":"2024-09-11T08:08:41.435613+0000","last_fullsized":"2024-09-11T08:08:41.435613+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":42,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-11T08:08:49.384449+0000","last_change":"2024-09-11T08:08:49.384423+0000","last_active":"2024-09-11T08:08:49.384449+0000","last_peered":"2024-09-11T08:08:49.384449+0000","last_clean":"2024-09-11T08:08:49.384449+0000","last_became_active":"2024-09-11T08:07:30.045380+0000","last_became_peered":"2024-09-11T08:07:30.045380+0000","last_unstale":"2024-09-11T08:08:49.384449+0000","last_undegraded":"2024-09-11T08:08:49.384449+0000","last_fullsized":"2024-09-11T08:08:49.384449+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:49.384357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:49.384357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:50.840709+0000","scrub_duration":0.0012073310000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.445741+0000","last_change":"2024-09-11T08:07:34.051286+0000","last_active":"2024-09-11T08:08:42.445741+0000","last_peered":"2024-09-11T08:08:42.445741+0000","last_clean":"2024-09-11T08:08:42.445741+0000","last_became_active":"2024-09-11T08:07:34.051004+0000","last_became_peered":"2024-09-11T08:07:34.051004+0000","last_unstale":"2024-09-11T08:08:42.445741+0000","last_undegraded":"2024-09-11T08:08:42.445741+0000","last_fullsized":"2024-09-11T08:08:42.445741+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:56.890501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.415001+0000","last_change":"2024-09-11T08:07:36.632119+0000","last_active":"2024-09-11T08:08:41.415001+0000","last_peered":"2024-09-11T08:08:41.415001+0000","last_clean":"2024-09-11T08:08:41.415001+0000","last_became_active":"2024-09-11T08:07:36.631668+0000","last_became_peered":"2024-09-11T08:07:36.631668+0000","last_unstale":"2024-09-11T08:08:41.415001+0000","last_undegraded":"2024-09-11T08:08:41.415001+0000","last_fullsized":"2024-09-11T08:08:41.415001+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:03.103464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"66'28","reported_seq":94,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-11T08:08:45.104014+0000","last_change":"2024-09-11T08:07:32.053328+0000","last_active":"2024-09-11T08:08:45.104014+0000","last_peered":"2024-09-11T08:08:45.104014+0000","last_clean":"2024-09-11T08:08:45.104014+0000","last_became_active":"2024-09-11T08:07:32.052956+0000","last_became_peered":"2024-09-11T08:07:32.052956+0000","last_unstale":"2024-09-11T08:08:45.104014+0000","last_undegraded":"2024-09-11T08:08:45.104014+0000","last_fullsized":"2024-09-11T08:08:45.104014+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:54.911563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"59'1","reported_seq":51,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.413328+0000","last_change":"2024-09-11T08:08:34.423550+0000","last_active":"2024-09-11T08:08:41.413328+0000","last_peered":"2024-09-11T08:08:41.413328+0000","last_clean":"2024-09-11T08:08:41.413328+0000","last_became_active":"2024-09-11T08:07:30.047513+0000","last_became_peered":"2024-09-11T08:07:30.047513+0000","last_unstale":"2024-09-11T08:08:41.413328+0000","last_undegraded":"2024-09-11T08:08:41.413328+0000","last_fullsized":"2024-09-11T08:08:41.413328+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-11T08:08:34.423474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:34.423474+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:57.535838+0000","scrub_duration":0.0016749149999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":48,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.054945+0000","last_change":"2024-09-11T08:08:37.859401+0000","last_active":"2024-09-11T08:08:42.054945+0000","last_peered":"2024-09-11T08:08:42.054945+0000","last_clean":"2024-09-11T08:08:42.054945+0000","last_became_active":"2024-09-11T08:07:30.043130+0000","last_became_peered":"2024-09-11T08:07:30.043130+0000","last_unstale":"2024-09-11T08:08:42.054945+0000","last_undegraded":"2024-09-11T08:08:42.054945+0000","last_fullsized":"2024-09-11T08:08:42.054945+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-11T08:08:37.859328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:37.859328+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:48.485842+0000","scrub_duration":0.001679816,"objects_trimmed":0,"snaptrim_duration":0.0026120850000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-11T08:08:46.959610+0000","last_change":"2024-09-11T08:08:46.959561+0000","last_active":"2024-09-11T08:08:46.959610+0000","last_peered":"2024-09-11T08:08:46.959610+0000","last_clean":"2024-09-11T08:08:46.959610+0000","last_became_active":"2024-09-11T08:07:34.057154+0000","last_became_peered":"2024-09-11T08:07:34.057154+0000","last_unstale":"2024-09-11T08:08:46.959610+0000","last_undegraded":"2024-09-11T08:08:46.959610+0000","last_fullsized":"2024-09-11T08:08:46.959610+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:46.959500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:46.959500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:05.161397+0000","scrub_duration":0.001608806,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.414698+0000","last_change":"2024-09-11T08:07:36.076211+0000","last_active":"2024-09-11T08:08:41.414698+0000","last_peered":"2024-09-11T08:08:41.414698+0000","last_clean":"2024-09-11T08:08:41.414698+0000","last_became_active":"2024-09-11T08:07:36.075690+0000","last_became_peered":"2024-09-11T08:07:36.075690+0000","last_unstale":"2024-09-11T08:08:41.414698+0000","last_undegraded":"2024-09-11T08:08:41.414698+0000","last_fullsized":"2024-09-11T08:08:41.414698+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:46.558963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":67,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.415366+0000","last_change":"2024-09-11T08:08:35.441946+0000","last_active":"2024-09-11T08:08:41.415366+0000","last_peered":"2024-09-11T08:08:41.415366+0000","last_clean":"2024-09-11T08:08:41.415366+0000","last_became_active":"2024-09-11T08:07:32.050673+0000","last_became_peered":"2024-09-11T08:07:32.050673+0000","last_unstale":"2024-09-11T08:08:41.415366+0000","last_undegraded":"2024-09-11T08:08:41.415366+0000","last_fullsized":"2024-09-11T08:08:41.415366+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'13","last_scrub_stamp":"2024-09-11T08:08:35.441894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:35.441894+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:02.914423+0000","scrub_duration":0.001888965,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.414479+0000","last_change":"2024-09-11T08:08:33.492709+0000","last_active":"2024-09-11T08:08:41.414479+0000","last_peered":"2024-09-11T08:08:41.414479+0000","last_clean":"2024-09-11T08:08:41.414479+0000","last_became_active":"2024-09-11T08:07:30.047014+0000","last_became_peered":"2024-09-11T08:07:30.047014+0000","last_unstale":"2024-09-11T08:08:41.414479+0000","last_undegraded":"2024-09-11T08:08:41.414479+0000","last_fullsized":"2024-09-11T08:08:41.414479+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:33.492652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:33.492652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:54.786224+0000","scrub_duration":0.00156573,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":47,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.433167+0000","last_change":"2024-09-11T08:08:32.909292+0000","last_active":"2024-09-11T08:08:41.433167+0000","last_peered":"2024-09-11T08:08:41.433167+0000","last_clean":"2024-09-11T08:08:41.433167+0000","last_became_active":"2024-09-11T08:07:30.045009+0000","last_became_peered":"2024-09-11T08:07:30.045009+0000","last_unstale":"2024-09-11T08:08:41.433167+0000","last_undegraded":"2024-09-11T08:08:41.433167+0000","last_fullsized":"2024-09-11T08:08:41.433167+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-11T08:08:32.909183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:32.909183+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:02.458449+0000","scrub_duration":0.001625769,"objects_trimmed":0,"snaptrim_duration":0.001859533,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.433460+0000","last_change":"2024-09-11T08:07:34.064252+0000","last_active":"2024-09-11T08:08:41.433460+0000","last_peered":"2024-09-11T08:08:41.433460+0000","last_clean":"2024-09-11T08:08:41.433460+0000","last_became_active":"2024-09-11T08:07:34.063906+0000","last_became_peered":"2024-09-11T08:07:34.063906+0000","last_unstale":"2024-09-11T08:08:41.433460+0000","last_undegraded":"2024-09-11T08:08:41.433460+0000","last_fullsized":"2024-09-11T08:08:41.433460+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:54.283587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-11T08:08:46.366623+0000","last_change":"2024-09-11T08:08:46.366578+0000","last_active":"2024-09-11T08:08:46.366623+0000","last_peered":"2024-09-11T08:08:46.366623+0000","last_clean":"2024-09-11T08:08:46.366623+0000","last_became_active":"2024-09-11T08:07:36.631480+0000","last_became_peered":"2024-09-11T08:07:36.631480+0000","last_unstale":"2024-09-11T08:08:46.366623+0000","last_undegraded":"2024-09-11T08:08:46.366623+0000","last_fullsized":"2024-09-11T08:08:46.366623+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:46.366508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:08:46.366508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:52.248237+0000","scrub_duration":0.0015317200000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":58,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.435575+0000","last_change":"2024-09-11T08:08:37.115649+0000","last_active":"2024-09-11T08:08:41.435575+0000","last_peered":"2024-09-11T08:08:41.435575+0000","last_clean":"2024-09-11T08:08:41.435575+0000","last_became_active":"2024-09-11T08:07:32.055407+0000","last_became_peered":"2024-09-11T08:07:32.055407+0000","last_unstale":"2024-09-11T08:08:41.435575+0000","last_undegraded":"2024-09-11T08:08:41.435575+0000","last_fullsized":"2024-09-11T08:08:41.435575+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-11T08:08:37.115582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:37.115582+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:57.869478+0000","scrub_duration":0.0020267729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"59'1","reported_seq":49,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.435295+0000","last_change":"2024-09-11T08:08:41.094879+0000","last_active":"2024-09-11T08:08:41.435295+0000","last_peered":"2024-09-11T08:08:41.435295+0000","last_clean":"2024-09-11T08:08:41.435295+0000","last_became_active":"2024-09-11T08:07:30.050504+0000","last_became_peered":"2024-09-11T08:07:30.050504+0000","last_unstale":"2024-09-11T08:08:41.435295+0000","last_undegraded":"2024-09-11T08:08:41.435295+0000","last_fullsized":"2024-09-11T08:08:41.435295+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-11T08:08:41.094821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:41.094821+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:00.561470+0000","scrub_duration":0.0017674310000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":116,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-11T08:08:51.426931+0000","last_change":"2024-09-11T08:07:34.046252+0000","last_active":"2024-09-11T08:08:51.426931+0000","last_peered":"2024-09-11T08:08:51.426931+0000","last_clean":"2024-09-11T08:08:51.426931+0000","last_became_active":"2024-09-11T08:07:30.045051+0000","last_became_peered":"2024-09-11T08:07:30.045051+0000","last_unstale":"2024-09-11T08:08:51.426931+0000","last_undegraded":"2024-09-11T08:08:51.426931+0000","last_fullsized":"2024-09-11T08:08:51.426931+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000318821,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":76,"num_read_kb":71,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.056081+0000","last_change":"2024-09-11T08:07:34.059262+0000","last_active":"2024-09-11T08:08:42.056081+0000","last_peered":"2024-09-11T08:08:42.056081+0000","last_clean":"2024-09-11T08:08:42.056081+0000","last_became_active":"2024-09-11T08:07:34.058962+0000","last_became_peered":"2024-09-11T08:07:34.058962+0000","last_unstale":"2024-09-11T08:08:42.056081+0000","last_undegraded":"2024-09-11T08:08:42.056081+0000","last_fullsized":"2024-09-11T08:08:42.056081+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:57.093439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-11T08:08:50.346156+0000","last_change":"2024-09-11T08:08:50.346126+0000","last_active":"2024-09-11T08:08:50.346156+0000","last_peered":"2024-09-11T08:08:50.346156+0000","last_clean":"2024-09-11T08:08:50.346156+0000","last_became_active":"2024-09-11T08:07:36.077597+0000","last_became_peered":"2024-09-11T08:07:36.077597+0000","last_unstale":"2024-09-11T08:08:50.346156+0000","last_undegraded":"2024-09-11T08:08:50.346156+0000","last_fullsized":"2024-09-11T08:08:50.346156+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:50.346069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:08:50.346069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:15.900195+0000","scrub_duration":0.001294906,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"66'14","reported_seq":51,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.446202+0000","last_change":"2024-09-11T08:07:32.042626+0000","last_active":"2024-09-11T08:08:42.446202+0000","last_peered":"2024-09-11T08:08:42.446202+0000","last_clean":"2024-09-11T08:08:42.446202+0000","last_became_active":"2024-09-11T08:07:32.042339+0000","last_became_peered":"2024-09-11T08:07:32.042339+0000","last_unstale":"2024-09-11T08:08:42.446202+0000","last_undegraded":"2024-09-11T08:08:42.446202+0000","last_fullsized":"2024-09-11T08:08:42.446202+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.415648+0000","last_change":"2024-09-11T08:07:30.047535+0000","last_active":"2024-09-11T08:08:41.415648+0000","last_peered":"2024-09-11T08:08:41.415648+0000","last_clean":"2024-09-11T08:08:41.415648+0000","last_became_active":"2024-09-11T08:07:30.047246+0000","last_became_peered":"2024-09-11T08:07:30.047246+0000","last_unstale":"2024-09-11T08:08:41.415648+0000","last_undegraded":"2024-09-11T08:08:41.415648+0000","last_fullsized":"2024-09-11T08:08:41.415648+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:51.674705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":78,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.378273+0000","last_change":"2024-09-11T08:07:44.054743+0000","last_active":"2024-09-11T08:08:42.378273+0000","last_peered":"2024-09-11T08:08:42.378273+0000","last_clean":"2024-09-11T08:08:42.378273+0000","last_became_active":"2024-09-11T08:07:19.726378+0000","last_became_peered":"2024-09-11T08:07:19.726378+0000","last_unstale":"2024-09-11T08:08:42.378273+0000","last_undegraded":"2024-09-11T08:08:42.378273+0000","last_fullsized":"2024-09-11T08:08:42.378273+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'72","last_scrub_stamp":"2024-09-11T08:07:44.054686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:05:21.231856+0000","last_clean_scrub_stamp":"2024-09-11T08:07:44.054686+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:58.084790+0000","scrub_duration":0.001872931,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-11T08:08:51.289982+0000","last_change":"2024-09-11T08:08:51.289948+0000","last_active":"2024-09-11T08:08:51.289982+0000","last_peered":"2024-09-11T08:08:51.289982+0000","last_clean":"2024-09-11T08:08:51.289982+0000","last_became_active":"2024-09-11T08:07:34.058414+0000","last_became_peered":"2024-09-11T08:07:34.058414+0000","last_unstale":"2024-09-11T08:08:51.289982+0000","last_undegraded":"2024-09-11T08:08:51.289982+0000","last_fullsized":"2024-09-11T08:08:51.289982+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:51.289872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:51.289872+0000","last_clean_scrub_stamp":"2024-09-11T08:08:51.289872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:08.912691+0000","scrub_duration":0.001988812,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.056024+0000","last_change":"2024-09-11T08:07:36.073155+0000","last_active":"2024-09-11T08:08:42.056024+0000","last_peered":"2024-09-11T08:08:42.056024+0000","last_clean":"2024-09-11T08:08:42.056024+0000","last_became_active":"2024-09-11T08:07:36.072508+0000","last_became_peered":"2024-09-11T08:07:36.072508+0000","last_unstale":"2024-09-11T08:08:42.056024+0000","last_undegraded":"2024-09-11T08:08:42.056024+0000","last_fullsized":"2024-09-11T08:08:42.056024+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:56.201478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"66'11","reported_seq":59,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.413748+0000","last_change":"2024-09-11T08:08:35.418440+0000","last_active":"2024-09-11T08:08:41.413748+0000","last_peered":"2024-09-11T08:08:41.413748+0000","last_clean":"2024-09-11T08:08:41.413748+0000","last_became_active":"2024-09-11T08:07:32.047971+0000","last_became_peered":"2024-09-11T08:07:32.047971+0000","last_unstale":"2024-09-11T08:08:41.413748+0000","last_undegraded":"2024-09-11T08:08:41.413748+0000","last_fullsized":"2024-09-11T08:08:41.413748+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-11T08:08:35.418378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:35.418378+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:38.165860+0000","scrub_duration":0.001791207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.413322+0000","last_change":"2024-09-11T08:08:36.417975+0000","last_active":"2024-09-11T08:08:41.413322+0000","last_peered":"2024-09-11T08:08:41.413322+0000","last_clean":"2024-09-11T08:08:41.413322+0000","last_became_active":"2024-09-11T08:07:30.039884+0000","last_became_peered":"2024-09-11T08:07:30.039884+0000","last_unstale":"2024-09-11T08:08:41.413322+0000","last_undegraded":"2024-09-11T08:08:41.413322+0000","last_fullsized":"2024-09-11T08:08:41.413322+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:36.417911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:36.417911+0000","last_clean_scrub_stamp":"2024-09-11T08:08:36.417911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:59.734565+0000","scrub_duration":0.0014387580000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.055965+0000","last_change":"2024-09-11T08:07:34.053720+0000","last_active":"2024-09-11T08:08:42.055965+0000","last_peered":"2024-09-11T08:08:42.055965+0000","last_clean":"2024-09-11T08:08:42.055965+0000","last_became_active":"2024-09-11T08:07:34.053347+0000","last_became_peered":"2024-09-11T08:07:34.053347+0000","last_unstale":"2024-09-11T08:08:42.055965+0000","last_undegraded":"2024-09-11T08:08:42.055965+0000","last_fullsized":"2024-09-11T08:08:42.055965+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:49.867695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.414124+0000","last_change":"2024-09-11T08:07:36.632329+0000","last_active":"2024-09-11T08:08:41.414124+0000","last_peered":"2024-09-11T08:08:41.414124+0000","last_clean":"2024-09-11T08:08:41.414124+0000","last_became_active":"2024-09-11T08:07:36.631687+0000","last_became_peered":"2024-09-11T08:07:36.631687+0000","last_unstale":"2024-09-11T08:08:41.414124+0000","last_undegraded":"2024-09-11T08:08:41.414124+0000","last_fullsized":"2024-09-11T08:08:41.414124+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"77'25","reported_seq":76,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.413800+0000","last_change":"2024-09-11T08:08:32.458272+0000","last_active":"2024-09-11T08:08:41.413800+0000","last_peered":"2024-09-11T08:08:41.413800+0000","last_clean":"2024-09-11T08:08:41.413800+0000","last_became_active":"2024-09-11T08:07:32.050815+0000","last_became_peered":"2024-09-11T08:07:32.050815+0000","last_unstale":"2024-09-11T08:08:41.413800+0000","last_undegraded":"2024-09-11T08:08:41.413800+0000","last_fullsized":"2024-09-11T08:08:41.413800+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'25","last_scrub_stamp":"2024-09-11T08:08:32.458210+0000","last_deep_scrub":"77'25","last_deep_scrub_stamp":"2024-09-11T08:08:32.458210+0000","last_clean_scrub_stamp":"2024-09-11T08:08:32.458210+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:00.578835+0000","scrub_duration":0.0060523319999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.445299+0000","last_change":"2024-09-11T08:07:30.042789+0000","last_active":"2024-09-11T08:08:42.445299+0000","last_peered":"2024-09-11T08:08:42.445299+0000","last_clean":"2024-09-11T08:08:42.445299+0000","last_became_active":"2024-09-11T08:07:30.042555+0000","last_became_peered":"2024-09-11T08:07:30.042555+0000","last_unstale":"2024-09-11T08:08:42.445299+0000","last_undegraded":"2024-09-11T08:08:42.445299+0000","last_fullsized":"2024-09-11T08:08:42.445299+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:57.728934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":38,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.437134+0000","last_change":"2024-09-11T08:08:41.437100+0000","last_active":"2024-09-11T08:08:41.437134+0000","last_peered":"2024-09-11T08:08:41.437134+0000","last_clean":"2024-09-11T08:08:41.437134+0000","last_became_active":"2024-09-11T08:07:34.054961+0000","last_became_peered":"2024-09-11T08:07:34.054961+0000","last_unstale":"2024-09-11T08:08:41.437134+0000","last_undegraded":"2024-09-11T08:08:41.437134+0000","last_fullsized":"2024-09-11T08:08:41.437134+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:41.437045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:41.437045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:10.512019+0000","scrub_duration":0.0029926649999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.433534+0000","last_change":"2024-09-11T08:07:36.071503+0000","last_active":"2024-09-11T08:08:41.433534+0000","last_peered":"2024-09-11T08:08:41.433534+0000","last_clean":"2024-09-11T08:08:41.433534+0000","last_became_active":"2024-09-11T08:07:36.071194+0000","last_became_peered":"2024-09-11T08:07:36.071194+0000","last_unstale":"2024-09-11T08:08:41.433534+0000","last_undegraded":"2024-09-11T08:08:41.433534+0000","last_fullsized":"2024-09-11T08:08:41.433534+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:58.811284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":55,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.448040+0000","last_change":"2024-09-11T08:08:42.447971+0000","last_active":"2024-09-11T08:08:42.448040+0000","last_peered":"2024-09-11T08:08:42.448040+0000","last_clean":"2024-09-11T08:08:42.448040+0000","last_became_active":"2024-09-11T08:07:32.043314+0000","last_became_peered":"2024-09-11T08:07:32.043314+0000","last_unstale":"2024-09-11T08:08:42.448040+0000","last_undegraded":"2024-09-11T08:08:42.448040+0000","last_fullsized":"2024-09-11T08:08:42.448040+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-11T08:08:42.447892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:42.447892+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:03.778832+0000","scrub_duration":0.0039015859999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.414548+0000","last_change":"2024-09-11T08:08:30.327580+0000","last_active":"2024-09-11T08:08:41.414548+0000","last_peered":"2024-09-11T08:08:41.414548+0000","last_clean":"2024-09-11T08:08:41.414548+0000","last_became_active":"2024-09-11T08:07:30.046684+0000","last_became_peered":"2024-09-11T08:07:30.046684+0000","last_unstale":"2024-09-11T08:08:41.414548+0000","last_undegraded":"2024-09-11T08:08:41.414548+0000","last_fullsized":"2024-09-11T08:08:41.414548+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:30.327525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:30.327525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:57.278810+0000","scrub_duration":0.001475589,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"66'8","reported_seq":61,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-11T08:08:49.026900+0000","last_change":"2024-09-11T08:08:49.026859+0000","last_active":"2024-09-11T08:08:49.026900+0000","last_peered":"2024-09-11T08:08:49.026900+0000","last_clean":"2024-09-11T08:08:49.026900+0000","last_became_active":"2024-09-11T08:07:34.062469+0000","last_became_peered":"2024-09-11T08:07:34.062469+0000","last_unstale":"2024-09-11T08:08:49.026900+0000","last_undegraded":"2024-09-11T08:08:49.026900+0000","last_fullsized":"2024-09-11T08:08:49.026900+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:08:49.026755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:49.026755+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:59.460486+0000","scrub_duration":0.0018077970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.413612+0000","last_change":"2024-09-11T08:07:36.074961+0000","last_active":"2024-09-11T08:08:41.413612+0000","last_peered":"2024-09-11T08:08:41.413612+0000","last_clean":"2024-09-11T08:08:41.413612+0000","last_became_active":"2024-09-11T08:07:36.074684+0000","last_became_peered":"2024-09-11T08:07:36.074684+0000","last_unstale":"2024-09-11T08:08:41.413612+0000","last_undegraded":"2024-09-11T08:08:41.413612+0000","last_fullsized":"2024-09-11T08:08:41.413612+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:48.050543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":46,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.435246+0000","last_change":"2024-09-11T08:07:32.055237+0000","last_active":"2024-09-11T08:08:41.435246+0000","last_peered":"2024-09-11T08:08:41.435246+0000","last_clean":"2024-09-11T08:08:41.435246+0000","last_became_active":"2024-09-11T08:07:32.054885+0000","last_became_peered":"2024-09-11T08:07:32.054885+0000","last_unstale":"2024-09-11T08:08:41.435246+0000","last_undegraded":"2024-09-11T08:08:41.435246+0000","last_fullsized":"2024-09-11T08:08:41.435246+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:55.102195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.055047+0000","last_change":"2024-09-11T08:08:34.828231+0000","last_active":"2024-09-11T08:08:42.055047+0000","last_peered":"2024-09-11T08:08:42.055047+0000","last_clean":"2024-09-11T08:08:42.055047+0000","last_became_active":"2024-09-11T08:07:30.045613+0000","last_became_peered":"2024-09-11T08:07:30.045613+0000","last_unstale":"2024-09-11T08:08:42.055047+0000","last_undegraded":"2024-09-11T08:08:42.055047+0000","last_fullsized":"2024-09-11T08:08:42.055047+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:34.828129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:34.828129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:58.397313+0000","scrub_duration":0.0013782530000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.433227+0000","last_change":"2024-09-11T08:08:37.871017+0000","last_active":"2024-09-11T08:08:41.433227+0000","last_peered":"2024-09-11T08:08:41.433227+0000","last_clean":"2024-09-11T08:08:41.433227+0000","last_became_active":"2024-09-11T08:07:34.056606+0000","last_became_peered":"2024-09-11T08:07:34.056606+0000","last_unstale":"2024-09-11T08:08:41.433227+0000","last_undegraded":"2024-09-11T08:08:41.433227+0000","last_fullsized":"2024-09-11T08:08:41.433227+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:37.870955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:37.870955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:58.505876+0000","scrub_duration":0.0016274460000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.435194+0000","last_change":"2024-09-11T08:07:36.072775+0000","last_active":"2024-09-11T08:08:41.435194+0000","last_peered":"2024-09-11T08:08:41.435194+0000","last_clean":"2024-09-11T08:08:41.435194+0000","last_became_active":"2024-09-11T08:07:36.072452+0000","last_became_peered":"2024-09-11T08:07:36.072452+0000","last_unstale":"2024-09-11T08:08:41.435194+0000","last_undegraded":"2024-09-11T08:08:41.435194+0000","last_fullsized":"2024-09-11T08:08:41.435194+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":60,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.445966+0000","last_change":"2024-09-11T08:07:32.051152+0000","last_active":"2024-09-11T08:08:42.445966+0000","last_peered":"2024-09-11T08:08:42.445966+0000","last_clean":"2024-09-11T08:08:42.445966+0000","last_became_active":"2024-09-11T08:07:32.050880+0000","last_became_peered":"2024-09-11T08:07:32.050880+0000","last_unstale":"2024-09-11T08:08:42.445966+0000","last_undegraded":"2024-09-11T08:08:42.445966+0000","last_fullsized":"2024-09-11T08:08:42.445966+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:44.385502+0000","last_change":"2024-09-11T08:08:44.385440+0000","last_active":"2024-09-11T08:08:44.385502+0000","last_peered":"2024-09-11T08:08:44.385502+0000","last_clean":"2024-09-11T08:08:44.385502+0000","last_became_active":"2024-09-11T08:07:30.043294+0000","last_became_peered":"2024-09-11T08:07:30.043294+0000","last_unstale":"2024-09-11T08:08:44.385502+0000","last_undegraded":"2024-09-11T08:08:44.385502+0000","last_fullsized":"2024-09-11T08:08:44.385502+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:44.385372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:44.385372+0000","last_clean_scrub_stamp":"2024-09-11T08:08:44.385372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:46.537624+0000","scrub_duration":0.001441007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-11T08:08:51.921519+0000","last_change":"2024-09-11T08:08:51.921479+0000","last_active":"2024-09-11T08:08:51.921519+0000","last_peered":"2024-09-11T08:08:51.921519+0000","last_clean":"2024-09-11T08:08:51.921519+0000","last_became_active":"2024-09-11T08:07:34.063589+0000","last_became_peered":"2024-09-11T08:07:34.063589+0000","last_unstale":"2024-09-11T08:08:51.921519+0000","last_undegraded":"2024-09-11T08:08:51.921519+0000","last_fullsized":"2024-09-11T08:08:51.921519+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:51.921389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:51.921389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:01.266430+0000","scrub_duration":0.001807091,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.378268+0000","last_change":"2024-09-11T08:08:36.367954+0000","last_active":"2024-09-11T08:08:42.378268+0000","last_peered":"2024-09-11T08:08:42.378268+0000","last_clean":"2024-09-11T08:08:42.378268+0000","last_became_active":"2024-09-11T08:07:36.077523+0000","last_became_peered":"2024-09-11T08:07:36.077523+0000","last_unstale":"2024-09-11T08:08:42.378268+0000","last_undegraded":"2024-09-11T08:08:42.378268+0000","last_fullsized":"2024-09-11T08:08:42.378268+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:36.367890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:08:36.367890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:40.081903+0000","scrub_duration":0.0015112560000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":53,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.413156+0000","last_change":"2024-09-11T08:07:32.048615+0000","last_active":"2024-09-11T08:08:41.413156+0000","last_peered":"2024-09-11T08:08:41.413156+0000","last_clean":"2024-09-11T08:08:41.413156+0000","last_became_active":"2024-09-11T08:07:32.048164+0000","last_became_peered":"2024-09-11T08:07:32.048164+0000","last_unstale":"2024-09-11T08:08:41.413156+0000","last_undegraded":"2024-09-11T08:08:41.413156+0000","last_fullsized":"2024-09-11T08:08:41.413156+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.377595+0000","last_change":"2024-09-11T08:07:30.046025+0000","last_active":"2024-09-11T08:08:42.377595+0000","last_peered":"2024-09-11T08:08:42.377595+0000","last_clean":"2024-09-11T08:08:42.377595+0000","last_became_active":"2024-09-11T08:07:30.041219+0000","last_became_peered":"2024-09-11T08:07:30.041219+0000","last_unstale":"2024-09-11T08:08:42.377595+0000","last_undegraded":"2024-09-11T08:08:42.377595+0000","last_fullsized":"2024-09-11T08:08:42.377595+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-11T08:08:46.970619+0000","last_change":"2024-09-11T08:08:46.970593+0000","last_active":"2024-09-11T08:08:46.970619+0000","last_peered":"2024-09-11T08:08:46.970619+0000","last_clean":"2024-09-11T08:08:46.970619+0000","last_became_active":"2024-09-11T08:07:34.057479+0000","last_became_peered":"2024-09-11T08:07:34.057479+0000","last_unstale":"2024-09-11T08:08:46.970619+0000","last_undegraded":"2024-09-11T08:08:46.970619+0000","last_fullsized":"2024-09-11T08:08:46.970619+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:46.970539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:46.970539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:09.551366+0000","scrub_duration":0.001167143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.414282+0000","last_change":"2024-09-11T08:07:36.078707+0000","last_active":"2024-09-11T08:08:41.414282+0000","last_peered":"2024-09-11T08:08:41.414282+0000","last_clean":"2024-09-11T08:08:41.414282+0000","last_became_active":"2024-09-11T08:07:36.078075+0000","last_became_peered":"2024-09-11T08:07:36.078075+0000","last_unstale":"2024-09-11T08:08:41.414282+0000","last_undegraded":"2024-09-11T08:08:41.414282+0000","last_fullsized":"2024-09-11T08:08:41.414282+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":54,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.055202+0000","last_change":"2024-09-11T08:07:32.039555+0000","last_active":"2024-09-11T08:08:42.055202+0000","last_peered":"2024-09-11T08:08:42.055202+0000","last_clean":"2024-09-11T08:08:42.055202+0000","last_became_active":"2024-09-11T08:07:32.039275+0000","last_became_peered":"2024-09-11T08:07:32.039275+0000","last_unstale":"2024-09-11T08:08:42.055202+0000","last_undegraded":"2024-09-11T08:08:42.055202+0000","last_fullsized":"2024-09-11T08:08:42.055202+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:00.903547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":49,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.377640+0000","last_change":"2024-09-11T08:07:30.046228+0000","last_active":"2024-09-11T08:08:42.377640+0000","last_peered":"2024-09-11T08:08:42.377640+0000","last_clean":"2024-09-11T08:08:42.377640+0000","last_became_active":"2024-09-11T08:07:30.045967+0000","last_became_peered":"2024-09-11T08:07:30.045967+0000","last_unstale":"2024-09-11T08:08:42.377640+0000","last_undegraded":"2024-09-11T08:08:42.377640+0000","last_fullsized":"2024-09-11T08:08:42.377640+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:58.653748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"66'8","reported_seq":52,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-11T08:08:44.434281+0000","last_change":"2024-09-11T08:07:34.058424+0000","last_active":"2024-09-11T08:08:44.434281+0000","last_peered":"2024-09-11T08:08:44.434281+0000","last_clean":"2024-09-11T08:08:44.434281+0000","last_became_active":"2024-09-11T08:07:34.058015+0000","last_became_peered":"2024-09-11T08:07:34.058015+0000","last_unstale":"2024-09-11T08:08:44.434281+0000","last_undegraded":"2024-09-11T08:08:44.434281+0000","last_fullsized":"2024-09-11T08:08:44.434281+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.055356+0000","last_change":"2024-09-11T08:07:36.632018+0000","last_active":"2024-09-11T08:08:42.055356+0000","last_peered":"2024-09-11T08:08:42.055356+0000","last_clean":"2024-09-11T08:08:42.055356+0000","last_became_active":"2024-09-11T08:07:36.631755+0000","last_became_peered":"2024-09-11T08:07:36.631755+0000","last_unstale":"2024-09-11T08:08:42.055356+0000","last_undegraded":"2024-09-11T08:08:42.055356+0000","last_fullsized":"2024-09-11T08:08:42.055356+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:58.191062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.445216+0000","last_change":"2024-09-11T08:07:30.046145+0000","last_active":"2024-09-11T08:08:42.445216+0000","last_peered":"2024-09-11T08:08:42.445216+0000","last_clean":"2024-09-11T08:08:42.445216+0000","last_became_active":"2024-09-11T08:07:30.045910+0000","last_became_peered":"2024-09-11T08:07:30.045910+0000","last_unstale":"2024-09-11T08:08:42.445216+0000","last_undegraded":"2024-09-11T08:08:42.445216+0000","last_fullsized":"2024-09-11T08:08:42.445216+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:56.568613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.415323+0000","last_change":"2024-09-11T08:07:32.047333+0000","last_active":"2024-09-11T08:08:41.415323+0000","last_peered":"2024-09-11T08:08:41.415323+0000","last_clean":"2024-09-11T08:08:41.415323+0000","last_became_active":"2024-09-11T08:07:32.046891+0000","last_became_peered":"2024-09-11T08:07:32.046891+0000","last_unstale":"2024-09-11T08:08:41.415323+0000","last_undegraded":"2024-09-11T08:08:41.415323+0000","last_fullsized":"2024-09-11T08:08:41.415323+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.055933+0000","last_change":"2024-09-11T08:07:34.058334+0000","last_active":"2024-09-11T08:08:42.055933+0000","last_peered":"2024-09-11T08:08:42.055933+0000","last_clean":"2024-09-11T08:08:42.055933+0000","last_became_active":"2024-09-11T08:07:34.057774+0000","last_became_peered":"2024-09-11T08:07:34.057774+0000","last_unstale":"2024-09-11T08:08:42.055933+0000","last_undegraded":"2024-09-11T08:08:42.055933+0000","last_fullsized":"2024-09-11T08:08:42.055933+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:57.690064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.378351+0000","last_change":"2024-09-11T08:08:38.326345+0000","last_active":"2024-09-11T08:08:42.378351+0000","last_peered":"2024-09-11T08:08:42.378351+0000","last_clean":"2024-09-11T08:08:42.378351+0000","last_became_active":"2024-09-11T08:07:36.633244+0000","last_became_peered":"2024-09-11T08:07:36.633244+0000","last_unstale":"2024-09-11T08:08:42.378351+0000","last_undegraded":"2024-09-11T08:08:42.378351+0000","last_fullsized":"2024-09-11T08:08:42.378351+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:38.326265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:08:38.326265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:06.322672+0000","scrub_duration":0.001417361,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":46,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.435685+0000","last_change":"2024-09-11T08:07:32.049425+0000","last_active":"2024-09-11T08:08:41.435685+0000","last_peered":"2024-09-11T08:08:41.435685+0000","last_clean":"2024-09-11T08:08:41.435685+0000","last_became_active":"2024-09-11T08:07:32.048350+0000","last_became_peered":"2024-09-11T08:07:32.048350+0000","last_unstale":"2024-09-11T08:08:41.435685+0000","last_undegraded":"2024-09-11T08:08:41.435685+0000","last_fullsized":"2024-09-11T08:08:41.435685+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.377377+0000","last_change":"2024-09-11T08:08:30.471690+0000","last_active":"2024-09-11T08:08:42.377377+0000","last_peered":"2024-09-11T08:08:42.377377+0000","last_clean":"2024-09-11T08:08:42.377377+0000","last_became_active":"2024-09-11T08:07:30.044055+0000","last_became_peered":"2024-09-11T08:07:30.044055+0000","last_unstale":"2024-09-11T08:08:42.377377+0000","last_undegraded":"2024-09-11T08:08:42.377377+0000","last_fullsized":"2024-09-11T08:08:42.377377+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:30.471620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:30.471620+0000","last_clean_scrub_stamp":"2024-09-11T08:08:30.471620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:37.891558+0000","scrub_duration":0.0016049980000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.415189+0000","last_change":"2024-09-11T08:07:34.057387+0000","last_active":"2024-09-11T08:08:41.415189+0000","last_peered":"2024-09-11T08:08:41.415189+0000","last_clean":"2024-09-11T08:08:41.415189+0000","last_became_active":"2024-09-11T08:07:34.057127+0000","last_became_peered":"2024-09-11T08:07:34.057127+0000","last_unstale":"2024-09-11T08:08:41.415189+0000","last_undegraded":"2024-09-11T08:08:41.415189+0000","last_fullsized":"2024-09-11T08:08:41.415189+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.433263+0000","last_change":"2024-09-11T08:07:36.071851+0000","last_active":"2024-09-11T08:08:41.433263+0000","last_peered":"2024-09-11T08:08:41.433263+0000","last_clean":"2024-09-11T08:08:41.433263+0000","last_became_active":"2024-09-11T08:07:36.071513+0000","last_became_peered":"2024-09-11T08:07:36.071513+0000","last_unstale":"2024-09-11T08:08:41.433263+0000","last_undegraded":"2024-09-11T08:08:41.433263+0000","last_fullsized":"2024-09-11T08:08:41.433263+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:56.903122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":55,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-11T08:08:49.951430+0000","last_change":"2024-09-11T08:08:49.951403+0000","last_active":"2024-09-11T08:08:49.951430+0000","last_peered":"2024-09-11T08:08:49.951430+0000","last_clean":"2024-09-11T08:08:49.951430+0000","last_became_active":"2024-09-11T08:07:32.041761+0000","last_became_peered":"2024-09-11T08:07:32.041761+0000","last_unstale":"2024-09-11T08:08:49.951430+0000","last_undegraded":"2024-09-11T08:08:49.951430+0000","last_fullsized":"2024-09-11T08:08:49.951430+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:08:49.951345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:49.951345+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:51.233235+0000","scrub_duration":0.002079914,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":42,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-11T08:08:48.935480+0000","last_change":"2024-09-11T08:08:48.935452+0000","last_active":"2024-09-11T08:08:48.935480+0000","last_peered":"2024-09-11T08:08:48.935480+0000","last_clean":"2024-09-11T08:08:48.935480+0000","last_became_active":"2024-09-11T08:07:30.045380+0000","last_became_peered":"2024-09-11T08:07:30.045380+0000","last_unstale":"2024-09-11T08:08:48.935480+0000","last_undegraded":"2024-09-11T08:08:48.935480+0000","last_fullsized":"2024-09-11T08:08:48.935480+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:48.935387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:48.935387+0000","last_clean_scrub_stamp":"2024-09-11T08:08:48.935387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:05.810526+0000","scrub_duration":0.001422134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"66'8","reported_seq":62,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-11T08:08:52.374617+0000","last_change":"2024-09-11T08:08:52.374593+0000","last_active":"2024-09-11T08:08:52.374617+0000","last_peered":"2024-09-11T08:08:52.374617+0000","last_clean":"2024-09-11T08:08:52.374617+0000","last_became_active":"2024-09-11T08:07:34.057937+0000","last_became_peered":"2024-09-11T08:07:34.057937+0000","last_unstale":"2024-09-11T08:08:52.374617+0000","last_undegraded":"2024-09-11T08:08:52.374617+0000","last_fullsized":"2024-09-11T08:08:52.374617+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:08:52.374539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:52.374539+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:12.088877+0000","scrub_duration":0.00146886,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.413837+0000","last_change":"2024-09-11T08:07:36.068798+0000","last_active":"2024-09-11T08:08:41.413837+0000","last_peered":"2024-09-11T08:08:41.413837+0000","last_clean":"2024-09-11T08:08:41.413837+0000","last_became_active":"2024-09-11T08:07:36.068493+0000","last_became_peered":"2024-09-11T08:07:36.068493+0000","last_unstale":"2024-09-11T08:08:41.413837+0000","last_undegraded":"2024-09-11T08:08:41.413837+0000","last_fullsized":"2024-09-11T08:08:41.413837+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:54.914337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":55,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.414505+0000","last_change":"2024-09-11T08:08:36.400676+0000","last_active":"2024-09-11T08:08:41.414505+0000","last_peered":"2024-09-11T08:08:41.414505+0000","last_clean":"2024-09-11T08:08:41.414505+0000","last_became_active":"2024-09-11T08:07:32.047347+0000","last_became_peered":"2024-09-11T08:07:32.047347+0000","last_unstale":"2024-09-11T08:08:41.414505+0000","last_undegraded":"2024-09-11T08:08:41.414505+0000","last_fullsized":"2024-09-11T08:08:41.414505+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-11T08:08:36.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:36.400586+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:59.480993+0000","scrub_duration":0.002193113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.377395+0000","last_change":"2024-09-11T08:07:30.039991+0000","last_active":"2024-09-11T08:08:42.377395+0000","last_peered":"2024-09-11T08:08:42.377395+0000","last_clean":"2024-09-11T08:08:42.377395+0000","last_became_active":"2024-09-11T08:07:30.039673+0000","last_became_peered":"2024-09-11T08:07:30.039673+0000","last_unstale":"2024-09-11T08:08:42.377395+0000","last_undegraded":"2024-09-11T08:08:42.377395+0000","last_fullsized":"2024-09-11T08:08:42.377395+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:48.464728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":61,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-11T08:08:47.966853+0000","last_change":"2024-09-11T08:08:47.966763+0000","last_active":"2024-09-11T08:08:47.966853+0000","last_peered":"2024-09-11T08:08:47.966853+0000","last_clean":"2024-09-11T08:08:47.966853+0000","last_became_active":"2024-09-11T08:07:34.060736+0000","last_became_peered":"2024-09-11T08:07:34.060736+0000","last_unstale":"2024-09-11T08:08:47.966853+0000","last_undegraded":"2024-09-11T08:08:47.966853+0000","last_fullsized":"2024-09-11T08:08:47.966853+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:08:47.966690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:47.966690+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:12.893275+0000","scrub_duration":0.0019777940000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.435361+0000","last_change":"2024-09-11T08:07:36.072648+0000","last_active":"2024-09-11T08:08:41.435361+0000","last_peered":"2024-09-11T08:08:41.435361+0000","last_clean":"2024-09-11T08:08:41.435361+0000","last_became_active":"2024-09-11T08:07:36.072359+0000","last_became_peered":"2024-09-11T08:07:36.072359+0000","last_unstale":"2024-09-11T08:08:41.435361+0000","last_undegraded":"2024-09-11T08:08:41.435361+0000","last_fullsized":"2024-09-11T08:08:41.435361+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:00.144060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":49,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.413129+0000","last_change":"2024-09-11T08:07:32.048316+0000","last_active":"2024-09-11T08:08:41.413129+0000","last_peered":"2024-09-11T08:08:41.413129+0000","last_clean":"2024-09-11T08:08:41.413129+0000","last_became_active":"2024-09-11T08:07:32.047981+0000","last_became_peered":"2024-09-11T08:07:32.047981+0000","last_unstale":"2024-09-11T08:08:41.413129+0000","last_undegraded":"2024-09-11T08:08:41.413129+0000","last_fullsized":"2024-09-11T08:08:41.413129+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":42,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-11T08:08:48.330387+0000","last_change":"2024-09-11T08:08:48.330351+0000","last_active":"2024-09-11T08:08:48.330387+0000","last_peered":"2024-09-11T08:08:48.330387+0000","last_clean":"2024-09-11T08:08:48.330387+0000","last_became_active":"2024-09-11T08:07:30.043691+0000","last_became_peered":"2024-09-11T08:07:30.043691+0000","last_unstale":"2024-09-11T08:08:48.330387+0000","last_undegraded":"2024-09-11T08:08:48.330387+0000","last_fullsized":"2024-09-11T08:08:48.330387+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:48.330285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:48.330285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:05.763303+0000","scrub_duration":0.001379929,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.415286+0000","last_change":"2024-09-11T08:07:34.053427+0000","last_active":"2024-09-11T08:08:41.415286+0000","last_peered":"2024-09-11T08:08:41.415286+0000","last_clean":"2024-09-11T08:08:41.415286+0000","last_became_active":"2024-09-11T08:07:34.053137+0000","last_became_peered":"2024-09-11T08:07:34.053137+0000","last_unstale":"2024-09-11T08:08:41.415286+0000","last_undegraded":"2024-09-11T08:08:41.415286+0000","last_fullsized":"2024-09-11T08:08:41.415286+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:00.814813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.413962+0000","last_change":"2024-09-11T08:07:36.076837+0000","last_active":"2024-09-11T08:08:41.413962+0000","last_peered":"2024-09-11T08:08:41.413962+0000","last_clean":"2024-09-11T08:08:41.413962+0000","last_became_active":"2024-09-11T08:07:36.076132+0000","last_became_peered":"2024-09-11T08:07:36.076132+0000","last_unstale":"2024-09-11T08:08:41.413962+0000","last_undegraded":"2024-09-11T08:08:41.413962+0000","last_fullsized":"2024-09-11T08:08:41.413962+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:01.236619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":46,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.415518+0000","last_change":"2024-09-11T08:07:32.049611+0000","last_active":"2024-09-11T08:08:41.415518+0000","last_peered":"2024-09-11T08:08:41.415518+0000","last_clean":"2024-09-11T08:08:41.415518+0000","last_became_active":"2024-09-11T08:07:32.049368+0000","last_became_peered":"2024-09-11T08:07:32.049368+0000","last_unstale":"2024-09-11T08:08:41.415518+0000","last_undegraded":"2024-09-11T08:08:41.415518+0000","last_fullsized":"2024-09-11T08:08:41.415518+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:49.402283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":42,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-11T08:08:45.388123+0000","last_change":"2024-09-11T08:08:45.388090+0000","last_active":"2024-09-11T08:08:45.388123+0000","last_peered":"2024-09-11T08:08:45.388123+0000","last_clean":"2024-09-11T08:08:45.388123+0000","last_became_active":"2024-09-11T08:07:30.037692+0000","last_became_peered":"2024-09-11T08:07:30.037692+0000","last_unstale":"2024-09-11T08:08:45.388123+0000","last_undegraded":"2024-09-11T08:08:45.388123+0000","last_fullsized":"2024-09-11T08:08:45.388123+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:45.388024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:45.388024+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:59.280827+0000","scrub_duration":0.0014875719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.414349+0000","last_change":"2024-09-11T08:07:34.057358+0000","last_active":"2024-09-11T08:08:41.414349+0000","last_peered":"2024-09-11T08:08:41.414349+0000","last_clean":"2024-09-11T08:08:41.414349+0000","last_became_active":"2024-09-11T08:07:34.057064+0000","last_became_peered":"2024-09-11T08:07:34.057064+0000","last_unstale":"2024-09-11T08:08:41.414349+0000","last_undegraded":"2024-09-11T08:08:41.414349+0000","last_fullsized":"2024-09-11T08:08:41.414349+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.434869+0000","last_change":"2024-09-11T08:07:36.077697+0000","last_active":"2024-09-11T08:08:41.434869+0000","last_peered":"2024-09-11T08:08:41.434869+0000","last_clean":"2024-09-11T08:08:41.434869+0000","last_became_active":"2024-09-11T08:07:36.077406+0000","last_became_peered":"2024-09-11T08:07:36.077406+0000","last_unstale":"2024-09-11T08:08:41.434869+0000","last_undegraded":"2024-09-11T08:08:41.434869+0000","last_fullsized":"2024-09-11T08:08:41.434869+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:48.444976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":49,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.414098+0000","last_change":"2024-09-11T08:07:32.043568+0000","last_active":"2024-09-11T08:08:41.414098+0000","last_peered":"2024-09-11T08:08:41.414098+0000","last_clean":"2024-09-11T08:08:41.414098+0000","last_became_active":"2024-09-11T08:07:32.043276+0000","last_became_peered":"2024-09-11T08:07:32.043276+0000","last_unstale":"2024-09-11T08:08:41.414098+0000","last_undegraded":"2024-09-11T08:08:41.414098+0000","last_fullsized":"2024-09-11T08:08:41.414098+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:47.264071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.054987+0000","last_change":"2024-09-11T08:07:30.049461+0000","last_active":"2024-09-11T08:08:42.054987+0000","last_peered":"2024-09-11T08:08:42.054987+0000","last_clean":"2024-09-11T08:08:42.054987+0000","last_became_active":"2024-09-11T08:07:30.049152+0000","last_became_peered":"2024-09-11T08:07:30.049152+0000","last_unstale":"2024-09-11T08:08:42.054987+0000","last_undegraded":"2024-09-11T08:08:42.054987+0000","last_fullsized":"2024-09-11T08:08:42.054987+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:48.303459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.378163+0000","last_change":"2024-09-11T08:07:34.057322+0000","last_active":"2024-09-11T08:08:42.378163+0000","last_peered":"2024-09-11T08:08:42.378163+0000","last_clean":"2024-09-11T08:08:42.378163+0000","last_became_active":"2024-09-11T08:07:34.056935+0000","last_became_peered":"2024-09-11T08:07:34.056935+0000","last_unstale":"2024-09-11T08:08:42.378163+0000","last_undegraded":"2024-09-11T08:08:42.378163+0000","last_fullsized":"2024-09-11T08:08:42.378163+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:57.406825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.414480+0000","last_change":"2024-09-11T08:07:36.633645+0000","last_active":"2024-09-11T08:08:41.414480+0000","last_peered":"2024-09-11T08:08:41.414480+0000","last_clean":"2024-09-11T08:08:41.414480+0000","last_became_active":"2024-09-11T08:07:36.633368+0000","last_became_peered":"2024-09-11T08:07:36.633368+0000","last_unstale":"2024-09-11T08:08:41.414480+0000","last_undegraded":"2024-09-11T08:08:41.414480+0000","last_fullsized":"2024-09-11T08:08:41.414480+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:48.747284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.414046+0000","last_change":"2024-09-11T08:07:32.046050+0000","last_active":"2024-09-11T08:08:41.414046+0000","last_peered":"2024-09-11T08:08:41.414046+0000","last_clean":"2024-09-11T08:08:41.414046+0000","last_became_active":"2024-09-11T08:07:32.045728+0000","last_became_peered":"2024-09-11T08:07:32.045728+0000","last_unstale":"2024-09-11T08:08:41.414046+0000","last_undegraded":"2024-09-11T08:08:41.414046+0000","last_fullsized":"2024-09-11T08:08:41.414046+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:54.611022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.056694+0000","last_change":"2024-09-11T08:08:42.056647+0000","last_active":"2024-09-11T08:08:42.056694+0000","last_peered":"2024-09-11T08:08:42.056694+0000","last_clean":"2024-09-11T08:08:42.056694+0000","last_became_active":"2024-09-11T08:07:30.049409+0000","last_became_peered":"2024-09-11T08:07:30.049409+0000","last_unstale":"2024-09-11T08:08:42.056694+0000","last_undegraded":"2024-09-11T08:08:42.056694+0000","last_fullsized":"2024-09-11T08:08:42.056694+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:42.056591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:42.056591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:56.872271+0000","scrub_duration":0.003068809,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.445735+0000","last_change":"2024-09-11T08:08:34.506991+0000","last_active":"2024-09-11T08:08:42.445735+0000","last_peered":"2024-09-11T08:08:42.445735+0000","last_clean":"2024-09-11T08:08:42.445735+0000","last_became_active":"2024-09-11T08:07:34.055082+0000","last_became_peered":"2024-09-11T08:07:34.055082+0000","last_unstale":"2024-09-11T08:08:42.445735+0000","last_undegraded":"2024-09-11T08:08:42.445735+0000","last_fullsized":"2024-09-11T08:08:42.445735+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:34.506927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:34.506927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:46.584592+0000","scrub_duration":0.0015499070000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.377962+0000","last_change":"2024-09-11T08:07:36.072035+0000","last_active":"2024-09-11T08:08:42.377962+0000","last_peered":"2024-09-11T08:08:42.377962+0000","last_clean":"2024-09-11T08:08:42.377962+0000","last_became_active":"2024-09-11T08:07:36.071669+0000","last_became_peered":"2024-09-11T08:07:36.071669+0000","last_unstale":"2024-09-11T08:08:42.377962+0000","last_undegraded":"2024-09-11T08:08:42.377962+0000","last_fullsized":"2024-09-11T08:08:42.377962+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:03.407787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.414019+0000","last_change":"2024-09-11T08:07:36.071729+0000","last_active":"2024-09-11T08:08:41.414019+0000","last_peered":"2024-09-11T08:08:41.414019+0000","last_clean":"2024-09-11T08:08:41.414019+0000","last_became_active":"2024-09-11T08:07:36.071468+0000","last_became_peered":"2024-09-11T08:07:36.071468+0000","last_unstale":"2024-09-11T08:08:41.414019+0000","last_undegraded":"2024-09-11T08:08:41.414019+0000","last_fullsized":"2024-09-11T08:08:41.414019+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:54.816006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":42,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-11T08:08:49.428072+0000","last_change":"2024-09-11T08:08:49.428032+0000","last_active":"2024-09-11T08:08:49.428072+0000","last_peered":"2024-09-11T08:08:49.428072+0000","last_clean":"2024-09-11T08:08:49.428072+0000","last_became_active":"2024-09-11T08:07:30.046702+0000","last_became_peered":"2024-09-11T08:07:30.046702+0000","last_unstale":"2024-09-11T08:08:49.428072+0000","last_undegraded":"2024-09-11T08:08:49.428072+0000","last_fullsized":"2024-09-11T08:08:49.428072+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:49.427944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:49.427944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:11.537893+0000","scrub_duration":0.001383034,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":48,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.446325+0000","last_change":"2024-09-11T08:07:32.054959+0000","last_active":"2024-09-11T08:08:42.446325+0000","last_peered":"2024-09-11T08:08:42.446325+0000","last_clean":"2024-09-11T08:08:42.446325+0000","last_became_active":"2024-09-11T08:07:32.054656+0000","last_became_peered":"2024-09-11T08:07:32.054656+0000","last_unstale":"2024-09-11T08:08:42.446325+0000","last_undegraded":"2024-09-11T08:08:42.446325+0000","last_fullsized":"2024-09-11T08:08:42.446325+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:47.428697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.435434+0000","last_change":"2024-09-11T08:07:34.061746+0000","last_active":"2024-09-11T08:08:41.435434+0000","last_peered":"2024-09-11T08:08:41.435434+0000","last_clean":"2024-09-11T08:08:41.435434+0000","last_became_active":"2024-09-11T08:07:34.061453+0000","last_became_peered":"2024-09-11T08:07:34.061453+0000","last_unstale":"2024-09-11T08:08:41.435434+0000","last_undegraded":"2024-09-11T08:08:41.435434+0000","last_fullsized":"2024-09-11T08:08:41.435434+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":399,"ondisk_log_size":399,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":29,"num_read_kb":29,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":76,"num_read_kb":71,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266709,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298756,"kb_used_data":1404,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446396,"statfs":{"total":95995035648,"available":95689109504,"internally_reserved":0,"allocated":1437696,"data_stored":938655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.63300000000000001}]},{"osd":1,"last update":"Wed Sep 11 08:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.621},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.53500000000000003}]},{"osd":2,"last update":"Wed Sep 11 08:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.56200000000000006}]},{"osd":3,"last update":"Wed Sep 11 08:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":7.9210000000000003,"5min":7.9210000000000003,"15min":7.9210000000000003},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.59199999999999997}]},{"osd":4,"last update":"Wed Sep 11 08:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.379}]},{"osd":5,"last update":"Wed Sep 11 08:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.443}]},{"osd":6,"last update":"Wed Sep 11 08:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.434}]}]},{"osd":6,"up_from":45,"seq":193273528346,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298744,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446408,"statfs":{"total":95995035648,"available":95689121792,"internally_reserved":0,"allocated":1421312,"data_stored":938478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:07:53 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.45200000000000001}]},{"osd":1,"last update":"Wed Sep 11 08:07:53 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.46899999999999997}]},{"osd":2,"last update":"Wed Sep 11 08:07:53 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.504},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.432}]},{"osd":3,"last update":"Wed Sep 11 08:07:53 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.61099999999999999}]},{"osd":4,"last update":"Wed Sep 11 08:07:53 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.26800000000000002}]},{"osd":5,"last update":"Wed Sep 11 08:07:53 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.317},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.254}]},{"osd":7,"last update":"Wed Sep 11 08:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.29999999999999999}]}]},{"osd":5,"up_from":38,"seq":163208757279,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298156,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446996,"statfs":{"total":95995035648,"available":95689723904,"internally_reserved":0,"allocated":823296,"data_stored":347317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.51400000000000001,"15min":0.52200000000000002},"min":{"1min":0.318,"5min":0.31,"15min":0.31},"max":{"1min":0.63500000000000001,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.503,"15min":0.501},"min":{"1min":0.41399999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.64000000000000001}]},{"osd":1,"last update":"Wed Sep 11 08:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":1.1910000000000001,"15min":1.3029999999999999},"min":{"1min":0.33000000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.82499999999999996,"5min":16.004999999999999,"15min":16.004999999999999},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.36199999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.61299999999999999}]},{"osd":2,"last update":"Wed Sep 11 08:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":1.206,"15min":1.323},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.754,"5min":16.018000000000001,"15min":16.018000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.65600000000000003}]},{"osd":3,"last update":"Wed Sep 11 08:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.53100000000000003,"15min":0.52900000000000003},"min":{"1min":0.44900000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.72299999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.505,"5min":0.51600000000000001,"15min":0.51800000000000002},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.63200000000000001}]},{"osd":4,"last update":"Wed Sep 11 08:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.39700000000000002,"15min":0.39300000000000002},"min":{"1min":0.27500000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.40000000000000002,"15min":0.38800000000000001},"min":{"1min":0.36499999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.50800000000000001}]},{"osd":6,"last update":"Wed Sep 11 08:07:53 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.57299999999999995}]},{"osd":7,"last update":"Wed Sep 11 08:08:22 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.54700000000000004}]}]},{"osd":4,"up_from":31,"seq":133143986212,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298180,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446972,"statfs":{"total":95995035648,"available":95689699328,"internally_reserved":0,"allocated":847872,"data_stored":348670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.499,"15min":0.50900000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.59699999999999998,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.52400000000000002,"15min":0.53500000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.58899999999999997,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.436}]},{"osd":1,"last update":"Wed Sep 11 08:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.505,"15min":0.51300000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.66800000000000004,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.52300000000000002,"15min":0.52900000000000003},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.68899999999999995,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.66900000000000004}]},{"osd":2,"last update":"Wed Sep 11 08:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.52000000000000002,"15min":0.52800000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.58799999999999997,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.51600000000000001,"15min":0.52500000000000002},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.623,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.58399999999999996}]},{"osd":3,"last update":"Wed Sep 11 08:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.50900000000000001,"15min":0.51500000000000001},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.64300000000000002,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.51800000000000002,"15min":0.52600000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.61499999999999999,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.50700000000000001}]},{"osd":5,"last update":"Wed Sep 11 08:08:31 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.36899999999999999,"15min":0.36699999999999999},"min":{"1min":0.27800000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.36899999999999999,"15min":0.36699999999999999},"min":{"1min":0.27900000000000003,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.48399999999999999}]},{"osd":6,"last update":"Wed Sep 11 08:07:53 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.38,"15min":0.38},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.628}]},{"osd":7,"last update":"Wed Sep 11 08:08:20 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.755},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.61699999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182441,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298196,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446956,"statfs":{"total":95995035648,"available":95689682944,"internally_reserved":0,"allocated":864256,"data_stored":348456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:07:52 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.373,"15min":0.377},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.51200000000000001,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.376,"5min":0.41599999999999998,"15min":0.42199999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.53700000000000003,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.35999999999999999}]},{"osd":1,"last update":"Wed Sep 11 08:07:52 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.435,"15min":0.443},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.57499999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.317},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.43099999999999999,"15min":0.435},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.30199999999999999}]},{"osd":2,"last update":"Wed Sep 11 08:07:52 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.45000000000000001,"15min":0.45400000000000001},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.53700000000000003,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.41699999999999998,"15min":0.42299999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.27400000000000002}]},{"osd":4,"last update":"Wed Sep 11 08:08:12 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54800000000000004,"15min":0.54900000000000004},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.53500000000000003,"15min":0.53700000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.66500000000000004}]},{"osd":5,"last update":"Wed Sep 11 08:08:33 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.54200000000000004,"15min":0.53800000000000003},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.40699999999999997,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.55700000000000005}]},{"osd":6,"last update":"Wed Sep 11 08:07:55 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.46899999999999997}]},{"osd":7,"last update":"Wed Sep 11 08:08:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.64700000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411372,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298152,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447000,"statfs":{"total":95995035648,"available":95689728000,"internally_reserved":0,"allocated":819200,"data_stored":347048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:08:24 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.38400000000000001,"15min":0.38100000000000001},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.58099999999999996,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.37,"15min":0.35999999999999999},"min":{"1min":0.25,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.55600000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.441}]},{"osd":1,"last update":"Wed Sep 11 08:08:24 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.39500000000000002,"15min":0.38},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.60899999999999999,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.433},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.39800000000000002,"15min":0.38600000000000001},"min":{"1min":0.28699999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.41699999999999998}]},{"osd":3,"last update":"Wed Sep 11 08:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.45100000000000001,"15min":0.46100000000000002},"min":{"1min":0.27300000000000002,"5min":0.218,"15min":0.218},"max":{"1min":0.629,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.45100000000000001,"15min":0.46500000000000002},"min":{"1min":0.318,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.36199999999999999}]},{"osd":4,"last update":"Wed Sep 11 08:08:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.58299999999999996,"15min":0.58699999999999997},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.71199999999999997,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.59799999999999998}]},{"osd":5,"last update":"Wed Sep 11 08:08:35 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.57699999999999996,"15min":0.58299999999999996},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.751,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.59199999999999997,"15min":0.59999999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.65300000000000002,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.64300000000000002}]},{"osd":6,"last update":"Wed Sep 11 08:07:57 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.55600000000000005}]},{"osd":7,"last update":"Wed Sep 11 08:08:22 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.51000000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298208,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446944,"statfs":{"total":95995035648,"available":95689670656,"internally_reserved":0,"allocated":876544,"data_stored":348614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.32000000000000001,"15min":0.29799999999999999},"min":{"1min":0.22600000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.33200000000000002,"15min":0.312},"min":{"1min":0.249,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"last":0.29299999999999998}]},{"osd":2,"last update":"Wed Sep 11 08:08:28 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.36199999999999999,"15min":0.34799999999999998},"min":{"1min":0.27700000000000002,"5min":0.224,"15min":0.224},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.21299999999999999},{"interface":"front","average":{"1min":0.37,"5min":0.35399999999999998,"15min":0.34200000000000003},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.307}]},{"osd":3,"last update":"Wed Sep 11 08:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.41399999999999998,"15min":0.42199999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.439,"15min":0.437},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.198,"5min":1.198,"15min":1.198},"last":0.32300000000000001}]},{"osd":4,"last update":"Wed Sep 11 08:08:08 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54300000000000004,"15min":0.54200000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.55800000000000005,"15min":0.55600000000000005},"min":{"1min":0.41899999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.73499999999999999,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.60499999999999998}]},{"osd":5,"last update":"Wed Sep 11 08:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.53700000000000003,"15min":0.53000000000000003},"min":{"1min":0.40699999999999997,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.77600000000000002,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.53100000000000003,"15min":0.52700000000000002},"min":{"1min":0.38400000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.58799999999999997}]},{"osd":6,"last update":"Wed Sep 11 08:07:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.55600000000000005}]},{"osd":7,"last update":"Wed Sep 11 08:08:17 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.495}]}]},{"osd":0,"up_from":8,"seq":34359738422,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298768,"kb_used_data":1416,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446384,"statfs":{"total":95995035648,"available":95689097216,"internally_reserved":0,"allocated":1449984,"data_stored":939601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 11 08:08:01 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.34100000000000003,"15min":0.315},"min":{"1min":0.25800000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.53900000000000003,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.35599999999999998,"15min":0.33700000000000002},"min":{"1min":0.23999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.47999999999999998}]},{"osd":2,"last update":"Wed Sep 11 08:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.34699999999999998},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.48799999999999999,"5min":0.53000000000000003,"15min":0.53000000000000003},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.378,"5min":0.378,"15min":0.36899999999999999},"min":{"1min":0.26600000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.56299999999999994,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.375}]},{"osd":3,"last update":"Wed Sep 11 08:08:51 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.44,"15min":0.44},"min":{"1min":0.27600000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.52300000000000002,"5min":0.62,"15min":0.62},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.42499999999999999,"15min":0.42599999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.52800000000000002,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.35699999999999998}]},{"osd":4,"last update":"Wed Sep 11 08:08:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.55000000000000004,"15min":0.54600000000000004},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.629}]},{"osd":5,"last update":"Wed Sep 11 08:08:36 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.58499999999999996,"15min":0.58299999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.748,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.55800000000000005,"15min":0.55300000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.66900000000000004}]},{"osd":6,"last update":"Wed Sep 11 08:07:56 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.65500000000000003}]},{"osd":7,"last update":"Wed Sep 11 08:08:19 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.69299999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T08:08:54.512 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T08:08:54.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:54 smithi150 ceph-mon[29276]: 4.3 scrub starts 2024-09-11T08:08:54.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:54 smithi150 ceph-mon[29276]: 4.3 scrub ok 2024-09-11T08:08:54.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:54 smithi150 ceph-mon[29276]: 3.e deep-scrub starts 2024-09-11T08:08:54.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:54 smithi150 ceph-mon[29276]: 3.e deep-scrub ok 2024-09-11T08:08:54.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:54 smithi150 ceph-mon[29276]: 5.0 deep-scrub starts 2024-09-11T08:08:54.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:54 smithi150 ceph-mon[29276]: 5.0 deep-scrub ok 2024-09-11T08:08:54.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:54 smithi150 ceph-mon[29276]: 4.a scrub starts 2024-09-11T08:08:54.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:54 smithi150 ceph-mon[29276]: 4.a scrub ok 2024-09-11T08:08:54.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:54 smithi150 ceph-mon[29276]: from='client.24587 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:08:54.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:54 smithi150 ceph-mon[39612]: 4.3 scrub starts 2024-09-11T08:08:54.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:54 smithi150 ceph-mon[39612]: 4.3 scrub ok 2024-09-11T08:08:54.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:54 smithi150 ceph-mon[39612]: 3.e deep-scrub starts 2024-09-11T08:08:54.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:54 smithi150 ceph-mon[39612]: 3.e deep-scrub ok 2024-09-11T08:08:54.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:54 smithi150 ceph-mon[39612]: 5.0 deep-scrub starts 2024-09-11T08:08:54.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:54 smithi150 ceph-mon[39612]: 5.0 deep-scrub ok 2024-09-11T08:08:54.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:54 smithi150 ceph-mon[39612]: 4.a scrub starts 2024-09-11T08:08:54.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:54 smithi150 ceph-mon[39612]: 4.a scrub ok 2024-09-11T08:08:54.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:54 smithi150 ceph-mon[39612]: from='client.24587 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:08:54.759 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:08:54.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:54 smithi181 ceph-mon[32701]: 4.3 scrub starts 2024-09-11T08:08:54.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:54 smithi181 ceph-mon[32701]: 4.3 scrub ok 2024-09-11T08:08:54.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:54 smithi181 ceph-mon[32701]: 3.e deep-scrub starts 2024-09-11T08:08:54.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:54 smithi181 ceph-mon[32701]: 3.e deep-scrub ok 2024-09-11T08:08:54.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:54 smithi181 ceph-mon[32701]: 5.0 deep-scrub starts 2024-09-11T08:08:54.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:54 smithi181 ceph-mon[32701]: 5.0 deep-scrub ok 2024-09-11T08:08:54.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:54 smithi181 ceph-mon[32701]: 4.a scrub starts 2024-09-11T08:08:54.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:54 smithi181 ceph-mon[32701]: 4.a scrub ok 2024-09-11T08:08:54.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:54 smithi181 ceph-mon[32701]: from='client.24587 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:08:55.478 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:08:55 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:08:55] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:08:55.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:55 smithi150 ceph-mon[29276]: 5.1e scrub starts 2024-09-11T08:08:55.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:55 smithi150 ceph-mon[29276]: 5.1e scrub ok 2024-09-11T08:08:55.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:55 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:08:55.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:55 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:08:55.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:55 smithi150 ceph-mon[29276]: osdmap e83: 8 total, 8 up, 8 in 2024-09-11T08:08:55.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:55 smithi150 ceph-mon[29276]: 5.1f scrub starts 2024-09-11T08:08:55.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:55 smithi150 ceph-mon[29276]: 5.1f scrub ok 2024-09-11T08:08:55.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:55 smithi150 ceph-mon[29276]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s 2024-09-11T08:08:55.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:55 smithi150 ceph-mon[29276]: 3.1d deep-scrub starts 2024-09-11T08:08:55.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:55 smithi150 ceph-mon[29276]: 3.1d deep-scrub ok 2024-09-11T08:08:55.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:55 smithi150 ceph-mon[29276]: osdmap e84: 8 total, 8 up, 8 in 2024-09-11T08:08:55.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:55 smithi150 ceph-mon[39612]: 5.1e scrub starts 2024-09-11T08:08:55.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:55 smithi150 ceph-mon[39612]: 5.1e scrub ok 2024-09-11T08:08:55.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:55 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:08:55.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:55 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:08:55.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:55 smithi150 ceph-mon[39612]: osdmap e83: 8 total, 8 up, 8 in 2024-09-11T08:08:55.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:55 smithi150 ceph-mon[39612]: 5.1f scrub starts 2024-09-11T08:08:55.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:55 smithi150 ceph-mon[39612]: 5.1f scrub ok 2024-09-11T08:08:55.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:55 smithi150 ceph-mon[39612]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s 2024-09-11T08:08:55.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:55 smithi150 ceph-mon[39612]: 3.1d deep-scrub starts 2024-09-11T08:08:55.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:55 smithi150 ceph-mon[39612]: 3.1d deep-scrub ok 2024-09-11T08:08:55.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:55 smithi150 ceph-mon[39612]: osdmap e84: 8 total, 8 up, 8 in 2024-09-11T08:08:55.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:55 smithi181 ceph-mon[32701]: 5.1e scrub starts 2024-09-11T08:08:55.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:55 smithi181 ceph-mon[32701]: 5.1e scrub ok 2024-09-11T08:08:55.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:55 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:08:55.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:55 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:08:55.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:55 smithi181 ceph-mon[32701]: osdmap e83: 8 total, 8 up, 8 in 2024-09-11T08:08:55.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:55 smithi181 ceph-mon[32701]: 5.1f scrub starts 2024-09-11T08:08:55.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:55 smithi181 ceph-mon[32701]: 5.1f scrub ok 2024-09-11T08:08:55.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:55 smithi181 ceph-mon[32701]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s 2024-09-11T08:08:55.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:55 smithi181 ceph-mon[32701]: 3.1d deep-scrub starts 2024-09-11T08:08:55.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:55 smithi181 ceph-mon[32701]: 3.1d deep-scrub ok 2024-09-11T08:08:55.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:55 smithi181 ceph-mon[32701]: osdmap e84: 8 total, 8 up, 8 in 2024-09-11T08:08:56.568 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:08:56.570 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-09-11T08:08:56.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:56 smithi150 ceph-mon[29276]: 6.3 scrub starts 2024-09-11T08:08:56.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:56 smithi150 ceph-mon[29276]: 6.3 scrub ok 2024-09-11T08:08:56.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:56 smithi150 ceph-mon[29276]: 4.12 scrub starts 2024-09-11T08:08:56.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:56 smithi150 ceph-mon[29276]: 4.12 scrub ok 2024-09-11T08:08:56.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:56 smithi150 ceph-mon[29276]: 3.16 scrub starts 2024-09-11T08:08:56.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:56 smithi150 ceph-mon[29276]: 3.16 scrub ok 2024-09-11T08:08:56.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:56 smithi150 ceph-mon[39612]: 6.3 scrub starts 2024-09-11T08:08:56.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:56 smithi150 ceph-mon[39612]: 6.3 scrub ok 2024-09-11T08:08:56.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:56 smithi150 ceph-mon[39612]: 4.12 scrub starts 2024-09-11T08:08:56.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:56 smithi150 ceph-mon[39612]: 4.12 scrub ok 2024-09-11T08:08:56.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:56 smithi150 ceph-mon[39612]: 3.16 scrub starts 2024-09-11T08:08:56.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:56 smithi150 ceph-mon[39612]: 3.16 scrub ok 2024-09-11T08:08:56.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:56 smithi181 ceph-mon[32701]: 6.3 scrub starts 2024-09-11T08:08:56.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:56 smithi181 ceph-mon[32701]: 6.3 scrub ok 2024-09-11T08:08:56.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:56 smithi181 ceph-mon[32701]: 4.12 scrub starts 2024-09-11T08:08:56.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:56 smithi181 ceph-mon[32701]: 4.12 scrub ok 2024-09-11T08:08:56.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:56 smithi181 ceph-mon[32701]: 3.16 scrub starts 2024-09-11T08:08:56.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:56 smithi181 ceph-mon[32701]: 3.16 scrub ok 2024-09-11T08:08:57.234 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2024-09-11T08:08:55.358067+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596188,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":929,"num_read_kb":786,"num_write":543,"num_write_kb":1671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":556,"ondisk_log_size":556,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2387160,"kb_used_data":8340,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":747574056,"statfs":{"total":767960285184,"available":765515833344,"internally_reserved":0,"allocated":8540160,"data_stored":4556839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.685987"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.435521+0000","last_change":"2024-09-11T08:08:36.105232+0000","last_active":"2024-09-11T08:08:41.435521+0000","last_peered":"2024-09-11T08:08:41.435521+0000","last_clean":"2024-09-11T08:08:41.435521+0000","last_became_active":"2024-09-11T08:07:30.044976+0000","last_became_peered":"2024-09-11T08:07:30.044976+0000","last_unstale":"2024-09-11T08:08:41.435521+0000","last_undegraded":"2024-09-11T08:08:41.435521+0000","last_fullsized":"2024-09-11T08:08:41.435521+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:36.105175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:36.105175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:56.596929+0000","scrub_duration":0.0016664970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":46,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.413093+0000","last_change":"2024-09-11T08:07:32.048578+0000","last_active":"2024-09-11T08:08:41.413093+0000","last_peered":"2024-09-11T08:08:41.413093+0000","last_clean":"2024-09-11T08:08:41.413093+0000","last_became_active":"2024-09-11T08:07:32.048048+0000","last_became_peered":"2024-09-11T08:07:32.048048+0000","last_unstale":"2024-09-11T08:08:41.413093+0000","last_undegraded":"2024-09-11T08:08:41.413093+0000","last_fullsized":"2024-09-11T08:08:41.413093+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:58.436753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":38,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.415054+0000","last_change":"2024-09-11T08:08:38.486126+0000","last_active":"2024-09-11T08:08:41.415054+0000","last_peered":"2024-09-11T08:08:41.415054+0000","last_clean":"2024-09-11T08:08:41.415054+0000","last_became_active":"2024-09-11T08:07:34.058828+0000","last_became_peered":"2024-09-11T08:07:34.058828+0000","last_unstale":"2024-09-11T08:08:41.415054+0000","last_undegraded":"2024-09-11T08:08:41.415054+0000","last_fullsized":"2024-09-11T08:08:41.415054+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:38.486057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:38.486057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:44.914804+0000","scrub_duration":0.002230531,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.413045+0000","last_change":"2024-09-11T08:07:36.075118+0000","last_active":"2024-09-11T08:08:41.413045+0000","last_peered":"2024-09-11T08:08:41.413045+0000","last_clean":"2024-09-11T08:08:41.413045+0000","last_became_active":"2024-09-11T08:07:36.074789+0000","last_became_peered":"2024-09-11T08:07:36.074789+0000","last_unstale":"2024-09-11T08:08:41.413045+0000","last_undegraded":"2024-09-11T08:08:41.413045+0000","last_fullsized":"2024-09-11T08:08:41.413045+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.414207+0000","last_change":"2024-09-11T08:07:36.633716+0000","last_active":"2024-09-11T08:08:41.414207+0000","last_peered":"2024-09-11T08:08:41.414207+0000","last_clean":"2024-09-11T08:08:41.414207+0000","last_became_active":"2024-09-11T08:07:36.633423+0000","last_became_peered":"2024-09-11T08:07:36.633423+0000","last_unstale":"2024-09-11T08:08:41.414207+0000","last_undegraded":"2024-09-11T08:08:41.414207+0000","last_fullsized":"2024-09-11T08:08:41.414207+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.414800+0000","last_change":"2024-09-11T08:07:30.045845+0000","last_active":"2024-09-11T08:08:41.414800+0000","last_peered":"2024-09-11T08:08:41.414800+0000","last_clean":"2024-09-11T08:08:41.414800+0000","last_became_active":"2024-09-11T08:07:30.045579+0000","last_became_peered":"2024-09-11T08:07:30.045579+0000","last_unstale":"2024-09-11T08:08:41.414800+0000","last_undegraded":"2024-09-11T08:08:41.414800+0000","last_fullsized":"2024-09-11T08:08:41.414800+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:51.248162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":55,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.414362+0000","last_change":"2024-09-11T08:07:32.047593+0000","last_active":"2024-09-11T08:08:41.414362+0000","last_peered":"2024-09-11T08:08:41.414362+0000","last_clean":"2024-09-11T08:08:41.414362+0000","last_became_active":"2024-09-11T08:07:32.045848+0000","last_became_peered":"2024-09-11T08:07:32.045848+0000","last_unstale":"2024-09-11T08:08:41.414362+0000","last_undegraded":"2024-09-11T08:08:41.414362+0000","last_fullsized":"2024-09-11T08:08:41.414362+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.412943+0000","last_change":"2024-09-11T08:07:34.053483+0000","last_active":"2024-09-11T08:08:41.412943+0000","last_peered":"2024-09-11T08:08:41.412943+0000","last_clean":"2024-09-11T08:08:41.412943+0000","last_became_active":"2024-09-11T08:07:34.052243+0000","last_became_peered":"2024-09-11T08:07:34.052243+0000","last_unstale":"2024-09-11T08:08:41.412943+0000","last_undegraded":"2024-09-11T08:08:41.412943+0000","last_fullsized":"2024-09-11T08:08:41.412943+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:00.987472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.055146+0000","last_change":"2024-09-11T08:07:30.044353+0000","last_active":"2024-09-11T08:08:42.055146+0000","last_peered":"2024-09-11T08:08:42.055146+0000","last_clean":"2024-09-11T08:08:42.055146+0000","last_became_active":"2024-09-11T08:07:30.043950+0000","last_became_peered":"2024-09-11T08:07:30.043950+0000","last_unstale":"2024-09-11T08:08:42.055146+0000","last_undegraded":"2024-09-11T08:08:42.055146+0000","last_fullsized":"2024-09-11T08:08:42.055146+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:45.265565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":46,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.413406+0000","last_change":"2024-09-11T08:07:32.044928+0000","last_active":"2024-09-11T08:08:41.413406+0000","last_peered":"2024-09-11T08:08:41.413406+0000","last_clean":"2024-09-11T08:08:41.413406+0000","last_became_active":"2024-09-11T08:07:32.044626+0000","last_became_peered":"2024-09-11T08:07:32.044626+0000","last_unstale":"2024-09-11T08:08:41.413406+0000","last_undegraded":"2024-09-11T08:08:41.413406+0000","last_fullsized":"2024-09-11T08:08:41.413406+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.055594+0000","last_change":"2024-09-11T08:07:34.061363+0000","last_active":"2024-09-11T08:08:42.055594+0000","last_peered":"2024-09-11T08:08:42.055594+0000","last_clean":"2024-09-11T08:08:42.055594+0000","last_became_active":"2024-09-11T08:07:34.058689+0000","last_became_peered":"2024-09-11T08:07:34.058689+0000","last_unstale":"2024-09-11T08:08:42.055594+0000","last_undegraded":"2024-09-11T08:08:42.055594+0000","last_fullsized":"2024-09-11T08:08:42.055594+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:01.558153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.435255+0000","last_change":"2024-09-11T08:08:40.071203+0000","last_active":"2024-09-11T08:08:41.435255+0000","last_peered":"2024-09-11T08:08:41.435255+0000","last_clean":"2024-09-11T08:08:41.435255+0000","last_became_active":"2024-09-11T08:07:36.076824+0000","last_became_peered":"2024-09-11T08:07:36.076824+0000","last_unstale":"2024-09-11T08:08:41.435255+0000","last_undegraded":"2024-09-11T08:08:41.435255+0000","last_fullsized":"2024-09-11T08:08:41.435255+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:40.071106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:08:40.071106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:54.333357+0000","scrub_duration":0.0019674499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.054968+0000","last_change":"2024-09-11T08:08:36.825857+0000","last_active":"2024-09-11T08:08:42.054968+0000","last_peered":"2024-09-11T08:08:42.054968+0000","last_clean":"2024-09-11T08:08:42.054968+0000","last_became_active":"2024-09-11T08:07:30.043277+0000","last_became_peered":"2024-09-11T08:07:30.043277+0000","last_unstale":"2024-09-11T08:08:42.054968+0000","last_undegraded":"2024-09-11T08:08:42.054968+0000","last_fullsized":"2024-09-11T08:08:42.054968+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:36.825794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:36.825794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:47.493434+0000","scrub_duration":0.0016805920000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":40,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.413068+0000","last_change":"2024-09-11T08:07:32.048969+0000","last_active":"2024-09-11T08:08:41.413068+0000","last_peered":"2024-09-11T08:08:41.413068+0000","last_clean":"2024-09-11T08:08:41.413068+0000","last_became_active":"2024-09-11T08:07:32.048661+0000","last_became_peered":"2024-09-11T08:07:32.048661+0000","last_unstale":"2024-09-11T08:08:41.413068+0000","last_undegraded":"2024-09-11T08:08:41.413068+0000","last_fullsized":"2024-09-11T08:08:41.413068+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:44.853813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.377991+0000","last_change":"2024-09-11T08:07:34.061194+0000","last_active":"2024-09-11T08:08:42.377991+0000","last_peered":"2024-09-11T08:08:42.377991+0000","last_clean":"2024-09-11T08:08:42.377991+0000","last_became_active":"2024-09-11T08:07:34.060871+0000","last_became_peered":"2024-09-11T08:07:34.060871+0000","last_unstale":"2024-09-11T08:08:42.377991+0000","last_undegraded":"2024-09-11T08:08:42.377991+0000","last_fullsized":"2024-09-11T08:08:42.377991+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:50.814439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.055345+0000","last_change":"2024-09-11T08:07:36.077890+0000","last_active":"2024-09-11T08:08:42.055345+0000","last_peered":"2024-09-11T08:08:42.055345+0000","last_clean":"2024-09-11T08:08:42.055345+0000","last_became_active":"2024-09-11T08:07:36.077602+0000","last_became_peered":"2024-09-11T08:07:36.077602+0000","last_unstale":"2024-09-11T08:08:42.055345+0000","last_undegraded":"2024-09-11T08:08:42.055345+0000","last_fullsized":"2024-09-11T08:08:42.055345+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:56.644431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.413616+0000","last_change":"2024-09-11T08:07:36.633951+0000","last_active":"2024-09-11T08:08:41.413616+0000","last_peered":"2024-09-11T08:08:41.413616+0000","last_clean":"2024-09-11T08:08:41.413616+0000","last_became_active":"2024-09-11T08:07:36.633338+0000","last_became_peered":"2024-09-11T08:07:36.633338+0000","last_unstale":"2024-09-11T08:08:41.413616+0000","last_undegraded":"2024-09-11T08:08:41.413616+0000","last_fullsized":"2024-09-11T08:08:41.413616+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:56.348854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.435745+0000","last_change":"2024-09-11T08:07:30.046004+0000","last_active":"2024-09-11T08:08:41.435745+0000","last_peered":"2024-09-11T08:08:41.435745+0000","last_clean":"2024-09-11T08:08:41.435745+0000","last_became_active":"2024-09-11T08:07:30.045739+0000","last_became_peered":"2024-09-11T08:07:30.045739+0000","last_unstale":"2024-09-11T08:08:41.435745+0000","last_undegraded":"2024-09-11T08:08:41.435745+0000","last_fullsized":"2024-09-11T08:08:41.435745+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:51.187269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"66'15","reported_seq":54,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.433511+0000","last_change":"2024-09-11T08:07:32.046658+0000","last_active":"2024-09-11T08:08:41.433511+0000","last_peered":"2024-09-11T08:08:41.433511+0000","last_clean":"2024-09-11T08:08:41.433511+0000","last_became_active":"2024-09-11T08:07:32.046363+0000","last_became_peered":"2024-09-11T08:07:32.046363+0000","last_unstale":"2024-09-11T08:08:41.433511+0000","last_undegraded":"2024-09-11T08:08:41.433511+0000","last_fullsized":"2024-09-11T08:08:41.433511+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:48.322062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.415156+0000","last_change":"2024-09-11T08:07:34.056221+0000","last_active":"2024-09-11T08:08:41.415156+0000","last_peered":"2024-09-11T08:08:41.415156+0000","last_clean":"2024-09-11T08:08:41.415156+0000","last_became_active":"2024-09-11T08:07:34.055410+0000","last_became_peered":"2024-09-11T08:07:34.055410+0000","last_unstale":"2024-09-11T08:08:41.415156+0000","last_undegraded":"2024-09-11T08:08:41.415156+0000","last_fullsized":"2024-09-11T08:08:41.415156+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:49.556611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.415115+0000","last_change":"2024-09-11T08:07:36.633626+0000","last_active":"2024-09-11T08:08:41.415115+0000","last_peered":"2024-09-11T08:08:41.415115+0000","last_clean":"2024-09-11T08:08:41.415115+0000","last_became_active":"2024-09-11T08:07:36.633258+0000","last_became_peered":"2024-09-11T08:07:36.633258+0000","last_unstale":"2024-09-11T08:08:41.415115+0000","last_undegraded":"2024-09-11T08:08:41.415115+0000","last_fullsized":"2024-09-11T08:08:41.415115+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:01.540476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":42,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-11T08:08:51.401988+0000","last_change":"2024-09-11T08:08:51.401929+0000","last_active":"2024-09-11T08:08:51.401988+0000","last_peered":"2024-09-11T08:08:51.401988+0000","last_clean":"2024-09-11T08:08:51.401988+0000","last_became_active":"2024-09-11T08:07:30.046284+0000","last_became_peered":"2024-09-11T08:07:30.046284+0000","last_unstale":"2024-09-11T08:08:51.401988+0000","last_undegraded":"2024-09-11T08:08:51.401988+0000","last_fullsized":"2024-09-11T08:08:51.401988+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:51.401864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:51.401864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:18.688155+0000","scrub_duration":0.001460034,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":53,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.415141+0000","last_change":"2024-09-11T08:07:32.047173+0000","last_active":"2024-09-11T08:08:41.415141+0000","last_peered":"2024-09-11T08:08:41.415141+0000","last_clean":"2024-09-11T08:08:41.415141+0000","last_became_active":"2024-09-11T08:07:32.046737+0000","last_became_peered":"2024-09-11T08:07:32.046737+0000","last_unstale":"2024-09-11T08:08:41.415141+0000","last_undegraded":"2024-09-11T08:08:41.415141+0000","last_fullsized":"2024-09-11T08:08:41.415141+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:52.570790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":38,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.413364+0000","last_change":"2024-09-11T08:08:40.419900+0000","last_active":"2024-09-11T08:08:41.413364+0000","last_peered":"2024-09-11T08:08:41.413364+0000","last_clean":"2024-09-11T08:08:41.413364+0000","last_became_active":"2024-09-11T08:07:34.056528+0000","last_became_peered":"2024-09-11T08:07:34.056528+0000","last_unstale":"2024-09-11T08:08:41.413364+0000","last_undegraded":"2024-09-11T08:08:41.413364+0000","last_fullsized":"2024-09-11T08:08:41.413364+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:40.419830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:40.419830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:09.111095+0000","scrub_duration":0.0029308149999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.378307+0000","last_change":"2024-09-11T08:07:36.072658+0000","last_active":"2024-09-11T08:08:42.378307+0000","last_peered":"2024-09-11T08:08:42.378307+0000","last_clean":"2024-09-11T08:08:42.378307+0000","last_became_active":"2024-09-11T08:07:36.072347+0000","last_became_peered":"2024-09-11T08:07:36.072347+0000","last_unstale":"2024-09-11T08:08:42.378307+0000","last_undegraded":"2024-09-11T08:08:42.378307+0000","last_fullsized":"2024-09-11T08:08:42.378307+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:57.979955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.415125+0000","last_change":"2024-09-11T08:07:30.046498+0000","last_active":"2024-09-11T08:08:41.415125+0000","last_peered":"2024-09-11T08:08:41.415125+0000","last_clean":"2024-09-11T08:08:41.415125+0000","last_became_active":"2024-09-11T08:07:30.045932+0000","last_became_peered":"2024-09-11T08:07:30.045932+0000","last_unstale":"2024-09-11T08:08:41.415125+0000","last_undegraded":"2024-09-11T08:08:41.415125+0000","last_fullsized":"2024-09-11T08:08:41.415125+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:56.901581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":45,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.435563+0000","last_change":"2024-09-11T08:07:32.048372+0000","last_active":"2024-09-11T08:08:41.435563+0000","last_peered":"2024-09-11T08:08:41.435563+0000","last_clean":"2024-09-11T08:08:41.435563+0000","last_became_active":"2024-09-11T08:07:32.048022+0000","last_became_peered":"2024-09-11T08:07:32.048022+0000","last_unstale":"2024-09-11T08:08:41.435563+0000","last_undegraded":"2024-09-11T08:08:41.435563+0000","last_fullsized":"2024-09-11T08:08:41.435563+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:52.163263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":54,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-11T08:08:45.102996+0000","last_change":"2024-09-11T08:07:34.055014+0000","last_active":"2024-09-11T08:08:45.102996+0000","last_peered":"2024-09-11T08:08:45.102996+0000","last_clean":"2024-09-11T08:08:45.102996+0000","last_became_active":"2024-09-11T08:07:34.054014+0000","last_became_peered":"2024-09-11T08:07:34.054014+0000","last_unstale":"2024-09-11T08:08:45.102996+0000","last_undegraded":"2024-09-11T08:08:45.102996+0000","last_fullsized":"2024-09-11T08:08:45.102996+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":65,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-11T08:08:45.358132+0000","last_change":"2024-09-11T08:08:45.358105+0000","last_active":"2024-09-11T08:08:45.358132+0000","last_peered":"2024-09-11T08:08:45.358132+0000","last_clean":"2024-09-11T08:08:45.358132+0000","last_became_active":"2024-09-11T08:07:32.054317+0000","last_became_peered":"2024-09-11T08:07:32.054317+0000","last_unstale":"2024-09-11T08:08:45.358132+0000","last_undegraded":"2024-09-11T08:08:45.358132+0000","last_fullsized":"2024-09-11T08:08:45.358132+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-11T08:08:45.358041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:45.358041+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:50.858524+0000","scrub_duration":0.0021022879999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.414788+0000","last_change":"2024-09-11T08:07:30.044933+0000","last_active":"2024-09-11T08:08:41.414788+0000","last_peered":"2024-09-11T08:08:41.414788+0000","last_clean":"2024-09-11T08:08:41.414788+0000","last_became_active":"2024-09-11T08:07:30.044624+0000","last_became_peered":"2024-09-11T08:07:30.044624+0000","last_unstale":"2024-09-11T08:08:41.414788+0000","last_undegraded":"2024-09-11T08:08:41.414788+0000","last_fullsized":"2024-09-11T08:08:41.414788+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:58.050628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":52,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-11T08:08:44.433741+0000","last_change":"2024-09-11T08:07:34.053292+0000","last_active":"2024-09-11T08:08:44.433741+0000","last_peered":"2024-09-11T08:08:44.433741+0000","last_clean":"2024-09-11T08:08:44.433741+0000","last_became_active":"2024-09-11T08:07:34.052981+0000","last_became_peered":"2024-09-11T08:07:34.052981+0000","last_unstale":"2024-09-11T08:08:44.433741+0000","last_undegraded":"2024-09-11T08:08:44.433741+0000","last_fullsized":"2024-09-11T08:08:44.433741+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-11T08:08:51.893893+0000","last_change":"2024-09-11T08:08:51.893846+0000","last_active":"2024-09-11T08:08:51.893893+0000","last_peered":"2024-09-11T08:08:51.893893+0000","last_clean":"2024-09-11T08:08:51.893893+0000","last_became_active":"2024-09-11T08:07:36.073969+0000","last_became_peered":"2024-09-11T08:07:36.073969+0000","last_unstale":"2024-09-11T08:08:51.893893+0000","last_undegraded":"2024-09-11T08:08:51.893893+0000","last_fullsized":"2024-09-11T08:08:51.893893+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:51.893774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:08:51.893774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:00.876260+0000","scrub_duration":0.001532858,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":72,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:43.441659+0000","last_change":"2024-09-11T08:08:43.441600+0000","last_active":"2024-09-11T08:08:43.441659+0000","last_peered":"2024-09-11T08:08:43.441659+0000","last_clean":"2024-09-11T08:08:43.441659+0000","last_became_active":"2024-09-11T08:07:32.045332+0000","last_became_peered":"2024-09-11T08:07:32.045332+0000","last_unstale":"2024-09-11T08:08:43.441659+0000","last_undegraded":"2024-09-11T08:08:43.441659+0000","last_fullsized":"2024-09-11T08:08:43.441659+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-11T08:08:43.441544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:43.441544+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:55.649394+0000","scrub_duration":0.002168648,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.415443+0000","last_change":"2024-09-11T08:07:30.047446+0000","last_active":"2024-09-11T08:08:41.415443+0000","last_peered":"2024-09-11T08:08:41.415443+0000","last_clean":"2024-09-11T08:08:41.415443+0000","last_became_active":"2024-09-11T08:07:30.047176+0000","last_became_peered":"2024-09-11T08:07:30.047176+0000","last_unstale":"2024-09-11T08:08:41.415443+0000","last_undegraded":"2024-09-11T08:08:41.415443+0000","last_fullsized":"2024-09-11T08:08:41.415443+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:58.438972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.412919+0000","last_change":"2024-09-11T08:07:34.055816+0000","last_active":"2024-09-11T08:08:41.412919+0000","last_peered":"2024-09-11T08:08:41.412919+0000","last_clean":"2024-09-11T08:08:41.412919+0000","last_became_active":"2024-09-11T08:07:34.055523+0000","last_became_peered":"2024-09-11T08:07:34.055523+0000","last_unstale":"2024-09-11T08:08:41.412919+0000","last_undegraded":"2024-09-11T08:08:41.412919+0000","last_fullsized":"2024-09-11T08:08:41.412919+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:57.216258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.413201+0000","last_change":"2024-09-11T08:07:36.069535+0000","last_active":"2024-09-11T08:08:41.413201+0000","last_peered":"2024-09-11T08:08:41.413201+0000","last_clean":"2024-09-11T08:08:41.413201+0000","last_became_active":"2024-09-11T08:07:36.069225+0000","last_became_peered":"2024-09-11T08:07:36.069225+0000","last_unstale":"2024-09-11T08:08:41.413201+0000","last_undegraded":"2024-09-11T08:08:41.413201+0000","last_fullsized":"2024-09-11T08:08:41.413201+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:57.222586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":51,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.413212+0000","last_change":"2024-09-11T08:07:32.040334+0000","last_active":"2024-09-11T08:08:41.413212+0000","last_peered":"2024-09-11T08:08:41.413212+0000","last_clean":"2024-09-11T08:08:41.413212+0000","last_became_active":"2024-09-11T08:07:32.040069+0000","last_became_peered":"2024-09-11T08:07:32.040069+0000","last_unstale":"2024-09-11T08:08:41.413212+0000","last_undegraded":"2024-09-11T08:08:41.413212+0000","last_fullsized":"2024-09-11T08:08:41.413212+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:48.042800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.435679+0000","last_change":"2024-09-11T08:07:30.050806+0000","last_active":"2024-09-11T08:08:41.435679+0000","last_peered":"2024-09-11T08:08:41.435679+0000","last_clean":"2024-09-11T08:08:41.435679+0000","last_became_active":"2024-09-11T08:07:30.050484+0000","last_became_peered":"2024-09-11T08:07:30.050484+0000","last_unstale":"2024-09-11T08:08:41.435679+0000","last_undegraded":"2024-09-11T08:08:41.435679+0000","last_fullsized":"2024-09-11T08:08:41.435679+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:47.129727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"66'8","reported_seq":55,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-11T08:08:44.434674+0000","last_change":"2024-09-11T08:07:34.056427+0000","last_active":"2024-09-11T08:08:44.434674+0000","last_peered":"2024-09-11T08:08:44.434674+0000","last_clean":"2024-09-11T08:08:44.434674+0000","last_became_active":"2024-09-11T08:07:34.056143+0000","last_became_peered":"2024-09-11T08:07:34.056143+0000","last_unstale":"2024-09-11T08:08:44.434674+0000","last_undegraded":"2024-09-11T08:08:44.434674+0000","last_fullsized":"2024-09-11T08:08:44.434674+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.378087+0000","last_change":"2024-09-11T08:07:36.633730+0000","last_active":"2024-09-11T08:08:42.378087+0000","last_peered":"2024-09-11T08:08:42.378087+0000","last_clean":"2024-09-11T08:08:42.378087+0000","last_became_active":"2024-09-11T08:07:36.633432+0000","last_became_peered":"2024-09-11T08:07:36.633432+0000","last_unstale":"2024-09-11T08:08:42.378087+0000","last_undegraded":"2024-09-11T08:08:42.378087+0000","last_fullsized":"2024-09-11T08:08:42.378087+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":45,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.415406+0000","last_change":"2024-09-11T08:07:32.054861+0000","last_active":"2024-09-11T08:08:41.415406+0000","last_peered":"2024-09-11T08:08:41.415406+0000","last_clean":"2024-09-11T08:08:41.415406+0000","last_became_active":"2024-09-11T08:07:32.054294+0000","last_became_peered":"2024-09-11T08:07:32.054294+0000","last_unstale":"2024-09-11T08:08:41.415406+0000","last_undegraded":"2024-09-11T08:08:41.415406+0000","last_fullsized":"2024-09-11T08:08:41.415406+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:00.767425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.054950+0000","last_change":"2024-09-11T08:07:30.047746+0000","last_active":"2024-09-11T08:08:42.054950+0000","last_peered":"2024-09-11T08:08:42.054950+0000","last_clean":"2024-09-11T08:08:42.054950+0000","last_became_active":"2024-09-11T08:07:30.047497+0000","last_became_peered":"2024-09-11T08:07:30.047497+0000","last_unstale":"2024-09-11T08:08:42.054950+0000","last_undegraded":"2024-09-11T08:08:42.054950+0000","last_fullsized":"2024-09-11T08:08:42.054950+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:54.652323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":52,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-11T08:08:45.103228+0000","last_change":"2024-09-11T08:07:34.060999+0000","last_active":"2024-09-11T08:08:45.103228+0000","last_peered":"2024-09-11T08:08:45.103228+0000","last_clean":"2024-09-11T08:08:45.103228+0000","last_became_active":"2024-09-11T08:07:34.060660+0000","last_became_peered":"2024-09-11T08:07:34.060660+0000","last_unstale":"2024-09-11T08:08:45.103228+0000","last_undegraded":"2024-09-11T08:08:45.103228+0000","last_fullsized":"2024-09-11T08:08:45.103228+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:54.412248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"66'1","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.435044+0000","last_change":"2024-09-11T08:07:36.072890+0000","last_active":"2024-09-11T08:08:41.435044+0000","last_peered":"2024-09-11T08:08:41.435044+0000","last_clean":"2024-09-11T08:08:41.435044+0000","last_became_active":"2024-09-11T08:07:36.072593+0000","last_became_peered":"2024-09-11T08:07:36.072593+0000","last_unstale":"2024-09-11T08:08:41.435044+0000","last_undegraded":"2024-09-11T08:08:41.435044+0000","last_fullsized":"2024-09-11T08:08:41.435044+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:58.785592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":66,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.435613+0000","last_change":"2024-09-11T08:07:32.044836+0000","last_active":"2024-09-11T08:08:41.435613+0000","last_peered":"2024-09-11T08:08:41.435613+0000","last_clean":"2024-09-11T08:08:41.435613+0000","last_became_active":"2024-09-11T08:07:32.044376+0000","last_became_peered":"2024-09-11T08:07:32.044376+0000","last_unstale":"2024-09-11T08:08:41.435613+0000","last_undegraded":"2024-09-11T08:08:41.435613+0000","last_fullsized":"2024-09-11T08:08:41.435613+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":42,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-11T08:08:49.384449+0000","last_change":"2024-09-11T08:08:49.384423+0000","last_active":"2024-09-11T08:08:49.384449+0000","last_peered":"2024-09-11T08:08:49.384449+0000","last_clean":"2024-09-11T08:08:49.384449+0000","last_became_active":"2024-09-11T08:07:30.045380+0000","last_became_peered":"2024-09-11T08:07:30.045380+0000","last_unstale":"2024-09-11T08:08:49.384449+0000","last_undegraded":"2024-09-11T08:08:49.384449+0000","last_fullsized":"2024-09-11T08:08:49.384449+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:49.384357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:49.384357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:50.840709+0000","scrub_duration":0.0012073310000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.445741+0000","last_change":"2024-09-11T08:07:34.051286+0000","last_active":"2024-09-11T08:08:42.445741+0000","last_peered":"2024-09-11T08:08:42.445741+0000","last_clean":"2024-09-11T08:08:42.445741+0000","last_became_active":"2024-09-11T08:07:34.051004+0000","last_became_peered":"2024-09-11T08:07:34.051004+0000","last_unstale":"2024-09-11T08:08:42.445741+0000","last_undegraded":"2024-09-11T08:08:42.445741+0000","last_fullsized":"2024-09-11T08:08:42.445741+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:56.890501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.415001+0000","last_change":"2024-09-11T08:07:36.632119+0000","last_active":"2024-09-11T08:08:41.415001+0000","last_peered":"2024-09-11T08:08:41.415001+0000","last_clean":"2024-09-11T08:08:41.415001+0000","last_became_active":"2024-09-11T08:07:36.631668+0000","last_became_peered":"2024-09-11T08:07:36.631668+0000","last_unstale":"2024-09-11T08:08:41.415001+0000","last_undegraded":"2024-09-11T08:08:41.415001+0000","last_fullsized":"2024-09-11T08:08:41.415001+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:03.103464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"66'28","reported_seq":94,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-11T08:08:45.104014+0000","last_change":"2024-09-11T08:07:32.053328+0000","last_active":"2024-09-11T08:08:45.104014+0000","last_peered":"2024-09-11T08:08:45.104014+0000","last_clean":"2024-09-11T08:08:45.104014+0000","last_became_active":"2024-09-11T08:07:32.052956+0000","last_became_peered":"2024-09-11T08:07:32.052956+0000","last_unstale":"2024-09-11T08:08:45.104014+0000","last_undegraded":"2024-09-11T08:08:45.104014+0000","last_fullsized":"2024-09-11T08:08:45.104014+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:54.911563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"59'1","reported_seq":51,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.413328+0000","last_change":"2024-09-11T08:08:34.423550+0000","last_active":"2024-09-11T08:08:41.413328+0000","last_peered":"2024-09-11T08:08:41.413328+0000","last_clean":"2024-09-11T08:08:41.413328+0000","last_became_active":"2024-09-11T08:07:30.047513+0000","last_became_peered":"2024-09-11T08:07:30.047513+0000","last_unstale":"2024-09-11T08:08:41.413328+0000","last_undegraded":"2024-09-11T08:08:41.413328+0000","last_fullsized":"2024-09-11T08:08:41.413328+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-11T08:08:34.423474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:34.423474+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:57.535838+0000","scrub_duration":0.0016749149999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":48,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.054945+0000","last_change":"2024-09-11T08:08:37.859401+0000","last_active":"2024-09-11T08:08:42.054945+0000","last_peered":"2024-09-11T08:08:42.054945+0000","last_clean":"2024-09-11T08:08:42.054945+0000","last_became_active":"2024-09-11T08:07:30.043130+0000","last_became_peered":"2024-09-11T08:07:30.043130+0000","last_unstale":"2024-09-11T08:08:42.054945+0000","last_undegraded":"2024-09-11T08:08:42.054945+0000","last_fullsized":"2024-09-11T08:08:42.054945+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-11T08:08:37.859328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:37.859328+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:48.485842+0000","scrub_duration":0.001679816,"objects_trimmed":0,"snaptrim_duration":0.0026120850000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-11T08:08:46.959610+0000","last_change":"2024-09-11T08:08:46.959561+0000","last_active":"2024-09-11T08:08:46.959610+0000","last_peered":"2024-09-11T08:08:46.959610+0000","last_clean":"2024-09-11T08:08:46.959610+0000","last_became_active":"2024-09-11T08:07:34.057154+0000","last_became_peered":"2024-09-11T08:07:34.057154+0000","last_unstale":"2024-09-11T08:08:46.959610+0000","last_undegraded":"2024-09-11T08:08:46.959610+0000","last_fullsized":"2024-09-11T08:08:46.959610+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:46.959500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:46.959500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:05.161397+0000","scrub_duration":0.001608806,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.414698+0000","last_change":"2024-09-11T08:07:36.076211+0000","last_active":"2024-09-11T08:08:41.414698+0000","last_peered":"2024-09-11T08:08:41.414698+0000","last_clean":"2024-09-11T08:08:41.414698+0000","last_became_active":"2024-09-11T08:07:36.075690+0000","last_became_peered":"2024-09-11T08:07:36.075690+0000","last_unstale":"2024-09-11T08:08:41.414698+0000","last_undegraded":"2024-09-11T08:08:41.414698+0000","last_fullsized":"2024-09-11T08:08:41.414698+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:46.558963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":67,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.415366+0000","last_change":"2024-09-11T08:08:35.441946+0000","last_active":"2024-09-11T08:08:41.415366+0000","last_peered":"2024-09-11T08:08:41.415366+0000","last_clean":"2024-09-11T08:08:41.415366+0000","last_became_active":"2024-09-11T08:07:32.050673+0000","last_became_peered":"2024-09-11T08:07:32.050673+0000","last_unstale":"2024-09-11T08:08:41.415366+0000","last_undegraded":"2024-09-11T08:08:41.415366+0000","last_fullsized":"2024-09-11T08:08:41.415366+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'13","last_scrub_stamp":"2024-09-11T08:08:35.441894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:35.441894+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:02.914423+0000","scrub_duration":0.001888965,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.414479+0000","last_change":"2024-09-11T08:08:33.492709+0000","last_active":"2024-09-11T08:08:41.414479+0000","last_peered":"2024-09-11T08:08:41.414479+0000","last_clean":"2024-09-11T08:08:41.414479+0000","last_became_active":"2024-09-11T08:07:30.047014+0000","last_became_peered":"2024-09-11T08:07:30.047014+0000","last_unstale":"2024-09-11T08:08:41.414479+0000","last_undegraded":"2024-09-11T08:08:41.414479+0000","last_fullsized":"2024-09-11T08:08:41.414479+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:33.492652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:33.492652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:54.786224+0000","scrub_duration":0.00156573,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":47,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.433167+0000","last_change":"2024-09-11T08:08:32.909292+0000","last_active":"2024-09-11T08:08:41.433167+0000","last_peered":"2024-09-11T08:08:41.433167+0000","last_clean":"2024-09-11T08:08:41.433167+0000","last_became_active":"2024-09-11T08:07:30.045009+0000","last_became_peered":"2024-09-11T08:07:30.045009+0000","last_unstale":"2024-09-11T08:08:41.433167+0000","last_undegraded":"2024-09-11T08:08:41.433167+0000","last_fullsized":"2024-09-11T08:08:41.433167+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-11T08:08:32.909183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:32.909183+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:02.458449+0000","scrub_duration":0.001625769,"objects_trimmed":0,"snaptrim_duration":0.001859533,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.433460+0000","last_change":"2024-09-11T08:07:34.064252+0000","last_active":"2024-09-11T08:08:41.433460+0000","last_peered":"2024-09-11T08:08:41.433460+0000","last_clean":"2024-09-11T08:08:41.433460+0000","last_became_active":"2024-09-11T08:07:34.063906+0000","last_became_peered":"2024-09-11T08:07:34.063906+0000","last_unstale":"2024-09-11T08:08:41.433460+0000","last_undegraded":"2024-09-11T08:08:41.433460+0000","last_fullsized":"2024-09-11T08:08:41.433460+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:54.283587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-11T08:08:46.366623+0000","last_change":"2024-09-11T08:08:46.366578+0000","last_active":"2024-09-11T08:08:46.366623+0000","last_peered":"2024-09-11T08:08:46.366623+0000","last_clean":"2024-09-11T08:08:46.366623+0000","last_became_active":"2024-09-11T08:07:36.631480+0000","last_became_peered":"2024-09-11T08:07:36.631480+0000","last_unstale":"2024-09-11T08:08:46.366623+0000","last_undegraded":"2024-09-11T08:08:46.366623+0000","last_fullsized":"2024-09-11T08:08:46.366623+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:46.366508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:08:46.366508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:52.248237+0000","scrub_duration":0.0015317200000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":58,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.435575+0000","last_change":"2024-09-11T08:08:37.115649+0000","last_active":"2024-09-11T08:08:41.435575+0000","last_peered":"2024-09-11T08:08:41.435575+0000","last_clean":"2024-09-11T08:08:41.435575+0000","last_became_active":"2024-09-11T08:07:32.055407+0000","last_became_peered":"2024-09-11T08:07:32.055407+0000","last_unstale":"2024-09-11T08:08:41.435575+0000","last_undegraded":"2024-09-11T08:08:41.435575+0000","last_fullsized":"2024-09-11T08:08:41.435575+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-11T08:08:37.115582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:37.115582+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:57.869478+0000","scrub_duration":0.0020267729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"59'1","reported_seq":49,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.435295+0000","last_change":"2024-09-11T08:08:41.094879+0000","last_active":"2024-09-11T08:08:41.435295+0000","last_peered":"2024-09-11T08:08:41.435295+0000","last_clean":"2024-09-11T08:08:41.435295+0000","last_became_active":"2024-09-11T08:07:30.050504+0000","last_became_peered":"2024-09-11T08:07:30.050504+0000","last_unstale":"2024-09-11T08:08:41.435295+0000","last_undegraded":"2024-09-11T08:08:41.435295+0000","last_fullsized":"2024-09-11T08:08:41.435295+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-11T08:08:41.094821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:41.094821+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:00.561470+0000","scrub_duration":0.0017674310000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":116,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-11T08:08:51.426931+0000","last_change":"2024-09-11T08:07:34.046252+0000","last_active":"2024-09-11T08:08:51.426931+0000","last_peered":"2024-09-11T08:08:51.426931+0000","last_clean":"2024-09-11T08:08:51.426931+0000","last_became_active":"2024-09-11T08:07:30.045051+0000","last_became_peered":"2024-09-11T08:07:30.045051+0000","last_unstale":"2024-09-11T08:08:51.426931+0000","last_undegraded":"2024-09-11T08:08:51.426931+0000","last_fullsized":"2024-09-11T08:08:51.426931+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000318821,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":76,"num_read_kb":71,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.056081+0000","last_change":"2024-09-11T08:07:34.059262+0000","last_active":"2024-09-11T08:08:42.056081+0000","last_peered":"2024-09-11T08:08:42.056081+0000","last_clean":"2024-09-11T08:08:42.056081+0000","last_became_active":"2024-09-11T08:07:34.058962+0000","last_became_peered":"2024-09-11T08:07:34.058962+0000","last_unstale":"2024-09-11T08:08:42.056081+0000","last_undegraded":"2024-09-11T08:08:42.056081+0000","last_fullsized":"2024-09-11T08:08:42.056081+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:57.093439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-11T08:08:50.346156+0000","last_change":"2024-09-11T08:08:50.346126+0000","last_active":"2024-09-11T08:08:50.346156+0000","last_peered":"2024-09-11T08:08:50.346156+0000","last_clean":"2024-09-11T08:08:50.346156+0000","last_became_active":"2024-09-11T08:07:36.077597+0000","last_became_peered":"2024-09-11T08:07:36.077597+0000","last_unstale":"2024-09-11T08:08:50.346156+0000","last_undegraded":"2024-09-11T08:08:50.346156+0000","last_fullsized":"2024-09-11T08:08:50.346156+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:50.346069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:08:50.346069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:15.900195+0000","scrub_duration":0.001294906,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"66'14","reported_seq":51,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.446202+0000","last_change":"2024-09-11T08:07:32.042626+0000","last_active":"2024-09-11T08:08:42.446202+0000","last_peered":"2024-09-11T08:08:42.446202+0000","last_clean":"2024-09-11T08:08:42.446202+0000","last_became_active":"2024-09-11T08:07:32.042339+0000","last_became_peered":"2024-09-11T08:07:32.042339+0000","last_unstale":"2024-09-11T08:08:42.446202+0000","last_undegraded":"2024-09-11T08:08:42.446202+0000","last_fullsized":"2024-09-11T08:08:42.446202+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.415648+0000","last_change":"2024-09-11T08:07:30.047535+0000","last_active":"2024-09-11T08:08:41.415648+0000","last_peered":"2024-09-11T08:08:41.415648+0000","last_clean":"2024-09-11T08:08:41.415648+0000","last_became_active":"2024-09-11T08:07:30.047246+0000","last_became_peered":"2024-09-11T08:07:30.047246+0000","last_unstale":"2024-09-11T08:08:41.415648+0000","last_undegraded":"2024-09-11T08:08:41.415648+0000","last_fullsized":"2024-09-11T08:08:41.415648+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:51.674705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":78,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.378273+0000","last_change":"2024-09-11T08:07:44.054743+0000","last_active":"2024-09-11T08:08:42.378273+0000","last_peered":"2024-09-11T08:08:42.378273+0000","last_clean":"2024-09-11T08:08:42.378273+0000","last_became_active":"2024-09-11T08:07:19.726378+0000","last_became_peered":"2024-09-11T08:07:19.726378+0000","last_unstale":"2024-09-11T08:08:42.378273+0000","last_undegraded":"2024-09-11T08:08:42.378273+0000","last_fullsized":"2024-09-11T08:08:42.378273+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'72","last_scrub_stamp":"2024-09-11T08:07:44.054686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:05:21.231856+0000","last_clean_scrub_stamp":"2024-09-11T08:07:44.054686+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:58.084790+0000","scrub_duration":0.001872931,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-11T08:08:51.289982+0000","last_change":"2024-09-11T08:08:51.289948+0000","last_active":"2024-09-11T08:08:51.289982+0000","last_peered":"2024-09-11T08:08:51.289982+0000","last_clean":"2024-09-11T08:08:51.289982+0000","last_became_active":"2024-09-11T08:07:34.058414+0000","last_became_peered":"2024-09-11T08:07:34.058414+0000","last_unstale":"2024-09-11T08:08:51.289982+0000","last_undegraded":"2024-09-11T08:08:51.289982+0000","last_fullsized":"2024-09-11T08:08:51.289982+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:51.289872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:51.289872+0000","last_clean_scrub_stamp":"2024-09-11T08:08:51.289872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:08.912691+0000","scrub_duration":0.001988812,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.056024+0000","last_change":"2024-09-11T08:07:36.073155+0000","last_active":"2024-09-11T08:08:42.056024+0000","last_peered":"2024-09-11T08:08:42.056024+0000","last_clean":"2024-09-11T08:08:42.056024+0000","last_became_active":"2024-09-11T08:07:36.072508+0000","last_became_peered":"2024-09-11T08:07:36.072508+0000","last_unstale":"2024-09-11T08:08:42.056024+0000","last_undegraded":"2024-09-11T08:08:42.056024+0000","last_fullsized":"2024-09-11T08:08:42.056024+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:56.201478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"66'11","reported_seq":59,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.413748+0000","last_change":"2024-09-11T08:08:35.418440+0000","last_active":"2024-09-11T08:08:41.413748+0000","last_peered":"2024-09-11T08:08:41.413748+0000","last_clean":"2024-09-11T08:08:41.413748+0000","last_became_active":"2024-09-11T08:07:32.047971+0000","last_became_peered":"2024-09-11T08:07:32.047971+0000","last_unstale":"2024-09-11T08:08:41.413748+0000","last_undegraded":"2024-09-11T08:08:41.413748+0000","last_fullsized":"2024-09-11T08:08:41.413748+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-11T08:08:35.418378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:35.418378+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:38.165860+0000","scrub_duration":0.001791207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.413322+0000","last_change":"2024-09-11T08:08:36.417975+0000","last_active":"2024-09-11T08:08:41.413322+0000","last_peered":"2024-09-11T08:08:41.413322+0000","last_clean":"2024-09-11T08:08:41.413322+0000","last_became_active":"2024-09-11T08:07:30.039884+0000","last_became_peered":"2024-09-11T08:07:30.039884+0000","last_unstale":"2024-09-11T08:08:41.413322+0000","last_undegraded":"2024-09-11T08:08:41.413322+0000","last_fullsized":"2024-09-11T08:08:41.413322+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:36.417911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:36.417911+0000","last_clean_scrub_stamp":"2024-09-11T08:08:36.417911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:59.734565+0000","scrub_duration":0.0014387580000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.055965+0000","last_change":"2024-09-11T08:07:34.053720+0000","last_active":"2024-09-11T08:08:42.055965+0000","last_peered":"2024-09-11T08:08:42.055965+0000","last_clean":"2024-09-11T08:08:42.055965+0000","last_became_active":"2024-09-11T08:07:34.053347+0000","last_became_peered":"2024-09-11T08:07:34.053347+0000","last_unstale":"2024-09-11T08:08:42.055965+0000","last_undegraded":"2024-09-11T08:08:42.055965+0000","last_fullsized":"2024-09-11T08:08:42.055965+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:49.867695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.414124+0000","last_change":"2024-09-11T08:07:36.632329+0000","last_active":"2024-09-11T08:08:41.414124+0000","last_peered":"2024-09-11T08:08:41.414124+0000","last_clean":"2024-09-11T08:08:41.414124+0000","last_became_active":"2024-09-11T08:07:36.631687+0000","last_became_peered":"2024-09-11T08:07:36.631687+0000","last_unstale":"2024-09-11T08:08:41.414124+0000","last_undegraded":"2024-09-11T08:08:41.414124+0000","last_fullsized":"2024-09-11T08:08:41.414124+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"77'25","reported_seq":76,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.413800+0000","last_change":"2024-09-11T08:08:32.458272+0000","last_active":"2024-09-11T08:08:41.413800+0000","last_peered":"2024-09-11T08:08:41.413800+0000","last_clean":"2024-09-11T08:08:41.413800+0000","last_became_active":"2024-09-11T08:07:32.050815+0000","last_became_peered":"2024-09-11T08:07:32.050815+0000","last_unstale":"2024-09-11T08:08:41.413800+0000","last_undegraded":"2024-09-11T08:08:41.413800+0000","last_fullsized":"2024-09-11T08:08:41.413800+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'25","last_scrub_stamp":"2024-09-11T08:08:32.458210+0000","last_deep_scrub":"77'25","last_deep_scrub_stamp":"2024-09-11T08:08:32.458210+0000","last_clean_scrub_stamp":"2024-09-11T08:08:32.458210+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:00.578835+0000","scrub_duration":0.0060523319999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.445299+0000","last_change":"2024-09-11T08:07:30.042789+0000","last_active":"2024-09-11T08:08:42.445299+0000","last_peered":"2024-09-11T08:08:42.445299+0000","last_clean":"2024-09-11T08:08:42.445299+0000","last_became_active":"2024-09-11T08:07:30.042555+0000","last_became_peered":"2024-09-11T08:07:30.042555+0000","last_unstale":"2024-09-11T08:08:42.445299+0000","last_undegraded":"2024-09-11T08:08:42.445299+0000","last_fullsized":"2024-09-11T08:08:42.445299+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:57.728934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":38,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.437134+0000","last_change":"2024-09-11T08:08:41.437100+0000","last_active":"2024-09-11T08:08:41.437134+0000","last_peered":"2024-09-11T08:08:41.437134+0000","last_clean":"2024-09-11T08:08:41.437134+0000","last_became_active":"2024-09-11T08:07:34.054961+0000","last_became_peered":"2024-09-11T08:07:34.054961+0000","last_unstale":"2024-09-11T08:08:41.437134+0000","last_undegraded":"2024-09-11T08:08:41.437134+0000","last_fullsized":"2024-09-11T08:08:41.437134+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:41.437045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:41.437045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:10.512019+0000","scrub_duration":0.0029926649999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.433534+0000","last_change":"2024-09-11T08:07:36.071503+0000","last_active":"2024-09-11T08:08:41.433534+0000","last_peered":"2024-09-11T08:08:41.433534+0000","last_clean":"2024-09-11T08:08:41.433534+0000","last_became_active":"2024-09-11T08:07:36.071194+0000","last_became_peered":"2024-09-11T08:07:36.071194+0000","last_unstale":"2024-09-11T08:08:41.433534+0000","last_undegraded":"2024-09-11T08:08:41.433534+0000","last_fullsized":"2024-09-11T08:08:41.433534+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:58.811284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":55,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.448040+0000","last_change":"2024-09-11T08:08:42.447971+0000","last_active":"2024-09-11T08:08:42.448040+0000","last_peered":"2024-09-11T08:08:42.448040+0000","last_clean":"2024-09-11T08:08:42.448040+0000","last_became_active":"2024-09-11T08:07:32.043314+0000","last_became_peered":"2024-09-11T08:07:32.043314+0000","last_unstale":"2024-09-11T08:08:42.448040+0000","last_undegraded":"2024-09-11T08:08:42.448040+0000","last_fullsized":"2024-09-11T08:08:42.448040+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-11T08:08:42.447892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:42.447892+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:03.778832+0000","scrub_duration":0.0039015859999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.414548+0000","last_change":"2024-09-11T08:08:30.327580+0000","last_active":"2024-09-11T08:08:41.414548+0000","last_peered":"2024-09-11T08:08:41.414548+0000","last_clean":"2024-09-11T08:08:41.414548+0000","last_became_active":"2024-09-11T08:07:30.046684+0000","last_became_peered":"2024-09-11T08:07:30.046684+0000","last_unstale":"2024-09-11T08:08:41.414548+0000","last_undegraded":"2024-09-11T08:08:41.414548+0000","last_fullsized":"2024-09-11T08:08:41.414548+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:30.327525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:30.327525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:57.278810+0000","scrub_duration":0.001475589,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"66'8","reported_seq":61,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-11T08:08:49.026900+0000","last_change":"2024-09-11T08:08:49.026859+0000","last_active":"2024-09-11T08:08:49.026900+0000","last_peered":"2024-09-11T08:08:49.026900+0000","last_clean":"2024-09-11T08:08:49.026900+0000","last_became_active":"2024-09-11T08:07:34.062469+0000","last_became_peered":"2024-09-11T08:07:34.062469+0000","last_unstale":"2024-09-11T08:08:49.026900+0000","last_undegraded":"2024-09-11T08:08:49.026900+0000","last_fullsized":"2024-09-11T08:08:49.026900+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:08:49.026755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:49.026755+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:59.460486+0000","scrub_duration":0.0018077970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.413612+0000","last_change":"2024-09-11T08:07:36.074961+0000","last_active":"2024-09-11T08:08:41.413612+0000","last_peered":"2024-09-11T08:08:41.413612+0000","last_clean":"2024-09-11T08:08:41.413612+0000","last_became_active":"2024-09-11T08:07:36.074684+0000","last_became_peered":"2024-09-11T08:07:36.074684+0000","last_unstale":"2024-09-11T08:08:41.413612+0000","last_undegraded":"2024-09-11T08:08:41.413612+0000","last_fullsized":"2024-09-11T08:08:41.413612+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:48.050543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":46,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.435246+0000","last_change":"2024-09-11T08:07:32.055237+0000","last_active":"2024-09-11T08:08:41.435246+0000","last_peered":"2024-09-11T08:08:41.435246+0000","last_clean":"2024-09-11T08:08:41.435246+0000","last_became_active":"2024-09-11T08:07:32.054885+0000","last_became_peered":"2024-09-11T08:07:32.054885+0000","last_unstale":"2024-09-11T08:08:41.435246+0000","last_undegraded":"2024-09-11T08:08:41.435246+0000","last_fullsized":"2024-09-11T08:08:41.435246+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:55.102195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.055047+0000","last_change":"2024-09-11T08:08:34.828231+0000","last_active":"2024-09-11T08:08:42.055047+0000","last_peered":"2024-09-11T08:08:42.055047+0000","last_clean":"2024-09-11T08:08:42.055047+0000","last_became_active":"2024-09-11T08:07:30.045613+0000","last_became_peered":"2024-09-11T08:07:30.045613+0000","last_unstale":"2024-09-11T08:08:42.055047+0000","last_undegraded":"2024-09-11T08:08:42.055047+0000","last_fullsized":"2024-09-11T08:08:42.055047+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:34.828129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:34.828129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:58.397313+0000","scrub_duration":0.0013782530000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.433227+0000","last_change":"2024-09-11T08:08:37.871017+0000","last_active":"2024-09-11T08:08:41.433227+0000","last_peered":"2024-09-11T08:08:41.433227+0000","last_clean":"2024-09-11T08:08:41.433227+0000","last_became_active":"2024-09-11T08:07:34.056606+0000","last_became_peered":"2024-09-11T08:07:34.056606+0000","last_unstale":"2024-09-11T08:08:41.433227+0000","last_undegraded":"2024-09-11T08:08:41.433227+0000","last_fullsized":"2024-09-11T08:08:41.433227+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:37.870955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:37.870955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:58.505876+0000","scrub_duration":0.0016274460000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.435194+0000","last_change":"2024-09-11T08:07:36.072775+0000","last_active":"2024-09-11T08:08:41.435194+0000","last_peered":"2024-09-11T08:08:41.435194+0000","last_clean":"2024-09-11T08:08:41.435194+0000","last_became_active":"2024-09-11T08:07:36.072452+0000","last_became_peered":"2024-09-11T08:07:36.072452+0000","last_unstale":"2024-09-11T08:08:41.435194+0000","last_undegraded":"2024-09-11T08:08:41.435194+0000","last_fullsized":"2024-09-11T08:08:41.435194+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":60,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.445966+0000","last_change":"2024-09-11T08:07:32.051152+0000","last_active":"2024-09-11T08:08:42.445966+0000","last_peered":"2024-09-11T08:08:42.445966+0000","last_clean":"2024-09-11T08:08:42.445966+0000","last_became_active":"2024-09-11T08:07:32.050880+0000","last_became_peered":"2024-09-11T08:07:32.050880+0000","last_unstale":"2024-09-11T08:08:42.445966+0000","last_undegraded":"2024-09-11T08:08:42.445966+0000","last_fullsized":"2024-09-11T08:08:42.445966+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:44.385502+0000","last_change":"2024-09-11T08:08:44.385440+0000","last_active":"2024-09-11T08:08:44.385502+0000","last_peered":"2024-09-11T08:08:44.385502+0000","last_clean":"2024-09-11T08:08:44.385502+0000","last_became_active":"2024-09-11T08:07:30.043294+0000","last_became_peered":"2024-09-11T08:07:30.043294+0000","last_unstale":"2024-09-11T08:08:44.385502+0000","last_undegraded":"2024-09-11T08:08:44.385502+0000","last_fullsized":"2024-09-11T08:08:44.385502+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:44.385372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:44.385372+0000","last_clean_scrub_stamp":"2024-09-11T08:08:44.385372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:46.537624+0000","scrub_duration":0.001441007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-11T08:08:51.921519+0000","last_change":"2024-09-11T08:08:51.921479+0000","last_active":"2024-09-11T08:08:51.921519+0000","last_peered":"2024-09-11T08:08:51.921519+0000","last_clean":"2024-09-11T08:08:51.921519+0000","last_became_active":"2024-09-11T08:07:34.063589+0000","last_became_peered":"2024-09-11T08:07:34.063589+0000","last_unstale":"2024-09-11T08:08:51.921519+0000","last_undegraded":"2024-09-11T08:08:51.921519+0000","last_fullsized":"2024-09-11T08:08:51.921519+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:51.921389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:51.921389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:01.266430+0000","scrub_duration":0.001807091,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.378268+0000","last_change":"2024-09-11T08:08:36.367954+0000","last_active":"2024-09-11T08:08:42.378268+0000","last_peered":"2024-09-11T08:08:42.378268+0000","last_clean":"2024-09-11T08:08:42.378268+0000","last_became_active":"2024-09-11T08:07:36.077523+0000","last_became_peered":"2024-09-11T08:07:36.077523+0000","last_unstale":"2024-09-11T08:08:42.378268+0000","last_undegraded":"2024-09-11T08:08:42.378268+0000","last_fullsized":"2024-09-11T08:08:42.378268+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:36.367890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:08:36.367890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:40.081903+0000","scrub_duration":0.0015112560000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":53,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.413156+0000","last_change":"2024-09-11T08:07:32.048615+0000","last_active":"2024-09-11T08:08:41.413156+0000","last_peered":"2024-09-11T08:08:41.413156+0000","last_clean":"2024-09-11T08:08:41.413156+0000","last_became_active":"2024-09-11T08:07:32.048164+0000","last_became_peered":"2024-09-11T08:07:32.048164+0000","last_unstale":"2024-09-11T08:08:41.413156+0000","last_undegraded":"2024-09-11T08:08:41.413156+0000","last_fullsized":"2024-09-11T08:08:41.413156+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.377595+0000","last_change":"2024-09-11T08:07:30.046025+0000","last_active":"2024-09-11T08:08:42.377595+0000","last_peered":"2024-09-11T08:08:42.377595+0000","last_clean":"2024-09-11T08:08:42.377595+0000","last_became_active":"2024-09-11T08:07:30.041219+0000","last_became_peered":"2024-09-11T08:07:30.041219+0000","last_unstale":"2024-09-11T08:08:42.377595+0000","last_undegraded":"2024-09-11T08:08:42.377595+0000","last_fullsized":"2024-09-11T08:08:42.377595+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-11T08:08:46.970619+0000","last_change":"2024-09-11T08:08:46.970593+0000","last_active":"2024-09-11T08:08:46.970619+0000","last_peered":"2024-09-11T08:08:46.970619+0000","last_clean":"2024-09-11T08:08:46.970619+0000","last_became_active":"2024-09-11T08:07:34.057479+0000","last_became_peered":"2024-09-11T08:07:34.057479+0000","last_unstale":"2024-09-11T08:08:46.970619+0000","last_undegraded":"2024-09-11T08:08:46.970619+0000","last_fullsized":"2024-09-11T08:08:46.970619+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:46.970539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:46.970539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:09.551366+0000","scrub_duration":0.001167143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.414282+0000","last_change":"2024-09-11T08:07:36.078707+0000","last_active":"2024-09-11T08:08:41.414282+0000","last_peered":"2024-09-11T08:08:41.414282+0000","last_clean":"2024-09-11T08:08:41.414282+0000","last_became_active":"2024-09-11T08:07:36.078075+0000","last_became_peered":"2024-09-11T08:07:36.078075+0000","last_unstale":"2024-09-11T08:08:41.414282+0000","last_undegraded":"2024-09-11T08:08:41.414282+0000","last_fullsized":"2024-09-11T08:08:41.414282+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":54,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.055202+0000","last_change":"2024-09-11T08:07:32.039555+0000","last_active":"2024-09-11T08:08:42.055202+0000","last_peered":"2024-09-11T08:08:42.055202+0000","last_clean":"2024-09-11T08:08:42.055202+0000","last_became_active":"2024-09-11T08:07:32.039275+0000","last_became_peered":"2024-09-11T08:07:32.039275+0000","last_unstale":"2024-09-11T08:08:42.055202+0000","last_undegraded":"2024-09-11T08:08:42.055202+0000","last_fullsized":"2024-09-11T08:08:42.055202+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:00.903547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":49,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.377640+0000","last_change":"2024-09-11T08:07:30.046228+0000","last_active":"2024-09-11T08:08:42.377640+0000","last_peered":"2024-09-11T08:08:42.377640+0000","last_clean":"2024-09-11T08:08:42.377640+0000","last_became_active":"2024-09-11T08:07:30.045967+0000","last_became_peered":"2024-09-11T08:07:30.045967+0000","last_unstale":"2024-09-11T08:08:42.377640+0000","last_undegraded":"2024-09-11T08:08:42.377640+0000","last_fullsized":"2024-09-11T08:08:42.377640+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:58.653748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"66'8","reported_seq":52,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-11T08:08:44.434281+0000","last_change":"2024-09-11T08:07:34.058424+0000","last_active":"2024-09-11T08:08:44.434281+0000","last_peered":"2024-09-11T08:08:44.434281+0000","last_clean":"2024-09-11T08:08:44.434281+0000","last_became_active":"2024-09-11T08:07:34.058015+0000","last_became_peered":"2024-09-11T08:07:34.058015+0000","last_unstale":"2024-09-11T08:08:44.434281+0000","last_undegraded":"2024-09-11T08:08:44.434281+0000","last_fullsized":"2024-09-11T08:08:44.434281+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.055356+0000","last_change":"2024-09-11T08:07:36.632018+0000","last_active":"2024-09-11T08:08:42.055356+0000","last_peered":"2024-09-11T08:08:42.055356+0000","last_clean":"2024-09-11T08:08:42.055356+0000","last_became_active":"2024-09-11T08:07:36.631755+0000","last_became_peered":"2024-09-11T08:07:36.631755+0000","last_unstale":"2024-09-11T08:08:42.055356+0000","last_undegraded":"2024-09-11T08:08:42.055356+0000","last_fullsized":"2024-09-11T08:08:42.055356+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:58.191062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.445216+0000","last_change":"2024-09-11T08:07:30.046145+0000","last_active":"2024-09-11T08:08:42.445216+0000","last_peered":"2024-09-11T08:08:42.445216+0000","last_clean":"2024-09-11T08:08:42.445216+0000","last_became_active":"2024-09-11T08:07:30.045910+0000","last_became_peered":"2024-09-11T08:07:30.045910+0000","last_unstale":"2024-09-11T08:08:42.445216+0000","last_undegraded":"2024-09-11T08:08:42.445216+0000","last_fullsized":"2024-09-11T08:08:42.445216+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:56.568613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.415323+0000","last_change":"2024-09-11T08:07:32.047333+0000","last_active":"2024-09-11T08:08:41.415323+0000","last_peered":"2024-09-11T08:08:41.415323+0000","last_clean":"2024-09-11T08:08:41.415323+0000","last_became_active":"2024-09-11T08:07:32.046891+0000","last_became_peered":"2024-09-11T08:07:32.046891+0000","last_unstale":"2024-09-11T08:08:41.415323+0000","last_undegraded":"2024-09-11T08:08:41.415323+0000","last_fullsized":"2024-09-11T08:08:41.415323+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.055933+0000","last_change":"2024-09-11T08:07:34.058334+0000","last_active":"2024-09-11T08:08:42.055933+0000","last_peered":"2024-09-11T08:08:42.055933+0000","last_clean":"2024-09-11T08:08:42.055933+0000","last_became_active":"2024-09-11T08:07:34.057774+0000","last_became_peered":"2024-09-11T08:07:34.057774+0000","last_unstale":"2024-09-11T08:08:42.055933+0000","last_undegraded":"2024-09-11T08:08:42.055933+0000","last_fullsized":"2024-09-11T08:08:42.055933+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:57.690064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.378351+0000","last_change":"2024-09-11T08:08:38.326345+0000","last_active":"2024-09-11T08:08:42.378351+0000","last_peered":"2024-09-11T08:08:42.378351+0000","last_clean":"2024-09-11T08:08:42.378351+0000","last_became_active":"2024-09-11T08:07:36.633244+0000","last_became_peered":"2024-09-11T08:07:36.633244+0000","last_unstale":"2024-09-11T08:08:42.378351+0000","last_undegraded":"2024-09-11T08:08:42.378351+0000","last_fullsized":"2024-09-11T08:08:42.378351+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:38.326265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:08:38.326265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:06.322672+0000","scrub_duration":0.001417361,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":46,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.435685+0000","last_change":"2024-09-11T08:07:32.049425+0000","last_active":"2024-09-11T08:08:41.435685+0000","last_peered":"2024-09-11T08:08:41.435685+0000","last_clean":"2024-09-11T08:08:41.435685+0000","last_became_active":"2024-09-11T08:07:32.048350+0000","last_became_peered":"2024-09-11T08:07:32.048350+0000","last_unstale":"2024-09-11T08:08:41.435685+0000","last_undegraded":"2024-09-11T08:08:41.435685+0000","last_fullsized":"2024-09-11T08:08:41.435685+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.377377+0000","last_change":"2024-09-11T08:08:30.471690+0000","last_active":"2024-09-11T08:08:42.377377+0000","last_peered":"2024-09-11T08:08:42.377377+0000","last_clean":"2024-09-11T08:08:42.377377+0000","last_became_active":"2024-09-11T08:07:30.044055+0000","last_became_peered":"2024-09-11T08:07:30.044055+0000","last_unstale":"2024-09-11T08:08:42.377377+0000","last_undegraded":"2024-09-11T08:08:42.377377+0000","last_fullsized":"2024-09-11T08:08:42.377377+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:30.471620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:30.471620+0000","last_clean_scrub_stamp":"2024-09-11T08:08:30.471620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:37.891558+0000","scrub_duration":0.0016049980000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.415189+0000","last_change":"2024-09-11T08:07:34.057387+0000","last_active":"2024-09-11T08:08:41.415189+0000","last_peered":"2024-09-11T08:08:41.415189+0000","last_clean":"2024-09-11T08:08:41.415189+0000","last_became_active":"2024-09-11T08:07:34.057127+0000","last_became_peered":"2024-09-11T08:07:34.057127+0000","last_unstale":"2024-09-11T08:08:41.415189+0000","last_undegraded":"2024-09-11T08:08:41.415189+0000","last_fullsized":"2024-09-11T08:08:41.415189+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.433263+0000","last_change":"2024-09-11T08:07:36.071851+0000","last_active":"2024-09-11T08:08:41.433263+0000","last_peered":"2024-09-11T08:08:41.433263+0000","last_clean":"2024-09-11T08:08:41.433263+0000","last_became_active":"2024-09-11T08:07:36.071513+0000","last_became_peered":"2024-09-11T08:07:36.071513+0000","last_unstale":"2024-09-11T08:08:41.433263+0000","last_undegraded":"2024-09-11T08:08:41.433263+0000","last_fullsized":"2024-09-11T08:08:41.433263+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:56.903122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":55,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-11T08:08:49.951430+0000","last_change":"2024-09-11T08:08:49.951403+0000","last_active":"2024-09-11T08:08:49.951430+0000","last_peered":"2024-09-11T08:08:49.951430+0000","last_clean":"2024-09-11T08:08:49.951430+0000","last_became_active":"2024-09-11T08:07:32.041761+0000","last_became_peered":"2024-09-11T08:07:32.041761+0000","last_unstale":"2024-09-11T08:08:49.951430+0000","last_undegraded":"2024-09-11T08:08:49.951430+0000","last_fullsized":"2024-09-11T08:08:49.951430+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:08:49.951345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:49.951345+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:51.233235+0000","scrub_duration":0.002079914,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":42,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-11T08:08:48.935480+0000","last_change":"2024-09-11T08:08:48.935452+0000","last_active":"2024-09-11T08:08:48.935480+0000","last_peered":"2024-09-11T08:08:48.935480+0000","last_clean":"2024-09-11T08:08:48.935480+0000","last_became_active":"2024-09-11T08:07:30.045380+0000","last_became_peered":"2024-09-11T08:07:30.045380+0000","last_unstale":"2024-09-11T08:08:48.935480+0000","last_undegraded":"2024-09-11T08:08:48.935480+0000","last_fullsized":"2024-09-11T08:08:48.935480+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:48.935387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:48.935387+0000","last_clean_scrub_stamp":"2024-09-11T08:08:48.935387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:05.810526+0000","scrub_duration":0.001422134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"66'8","reported_seq":62,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-11T08:08:52.374617+0000","last_change":"2024-09-11T08:08:52.374593+0000","last_active":"2024-09-11T08:08:52.374617+0000","last_peered":"2024-09-11T08:08:52.374617+0000","last_clean":"2024-09-11T08:08:52.374617+0000","last_became_active":"2024-09-11T08:07:34.057937+0000","last_became_peered":"2024-09-11T08:07:34.057937+0000","last_unstale":"2024-09-11T08:08:52.374617+0000","last_undegraded":"2024-09-11T08:08:52.374617+0000","last_fullsized":"2024-09-11T08:08:52.374617+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:08:52.374539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:52.374539+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:12.088877+0000","scrub_duration":0.00146886,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.413837+0000","last_change":"2024-09-11T08:07:36.068798+0000","last_active":"2024-09-11T08:08:41.413837+0000","last_peered":"2024-09-11T08:08:41.413837+0000","last_clean":"2024-09-11T08:08:41.413837+0000","last_became_active":"2024-09-11T08:07:36.068493+0000","last_became_peered":"2024-09-11T08:07:36.068493+0000","last_unstale":"2024-09-11T08:08:41.413837+0000","last_undegraded":"2024-09-11T08:08:41.413837+0000","last_fullsized":"2024-09-11T08:08:41.413837+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:54.914337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":55,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.414505+0000","last_change":"2024-09-11T08:08:36.400676+0000","last_active":"2024-09-11T08:08:41.414505+0000","last_peered":"2024-09-11T08:08:41.414505+0000","last_clean":"2024-09-11T08:08:41.414505+0000","last_became_active":"2024-09-11T08:07:32.047347+0000","last_became_peered":"2024-09-11T08:07:32.047347+0000","last_unstale":"2024-09-11T08:08:41.414505+0000","last_undegraded":"2024-09-11T08:08:41.414505+0000","last_fullsized":"2024-09-11T08:08:41.414505+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-11T08:08:36.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:36.400586+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:59.480993+0000","scrub_duration":0.002193113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.377395+0000","last_change":"2024-09-11T08:07:30.039991+0000","last_active":"2024-09-11T08:08:42.377395+0000","last_peered":"2024-09-11T08:08:42.377395+0000","last_clean":"2024-09-11T08:08:42.377395+0000","last_became_active":"2024-09-11T08:07:30.039673+0000","last_became_peered":"2024-09-11T08:07:30.039673+0000","last_unstale":"2024-09-11T08:08:42.377395+0000","last_undegraded":"2024-09-11T08:08:42.377395+0000","last_fullsized":"2024-09-11T08:08:42.377395+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:48.464728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":61,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-11T08:08:47.966853+0000","last_change":"2024-09-11T08:08:47.966763+0000","last_active":"2024-09-11T08:08:47.966853+0000","last_peered":"2024-09-11T08:08:47.966853+0000","last_clean":"2024-09-11T08:08:47.966853+0000","last_became_active":"2024-09-11T08:07:34.060736+0000","last_became_peered":"2024-09-11T08:07:34.060736+0000","last_unstale":"2024-09-11T08:08:47.966853+0000","last_undegraded":"2024-09-11T08:08:47.966853+0000","last_fullsized":"2024-09-11T08:08:47.966853+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:08:47.966690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:47.966690+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:12.893275+0000","scrub_duration":0.0019777940000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.435361+0000","last_change":"2024-09-11T08:07:36.072648+0000","last_active":"2024-09-11T08:08:41.435361+0000","last_peered":"2024-09-11T08:08:41.435361+0000","last_clean":"2024-09-11T08:08:41.435361+0000","last_became_active":"2024-09-11T08:07:36.072359+0000","last_became_peered":"2024-09-11T08:07:36.072359+0000","last_unstale":"2024-09-11T08:08:41.435361+0000","last_undegraded":"2024-09-11T08:08:41.435361+0000","last_fullsized":"2024-09-11T08:08:41.435361+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:00.144060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":49,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.413129+0000","last_change":"2024-09-11T08:07:32.048316+0000","last_active":"2024-09-11T08:08:41.413129+0000","last_peered":"2024-09-11T08:08:41.413129+0000","last_clean":"2024-09-11T08:08:41.413129+0000","last_became_active":"2024-09-11T08:07:32.047981+0000","last_became_peered":"2024-09-11T08:07:32.047981+0000","last_unstale":"2024-09-11T08:08:41.413129+0000","last_undegraded":"2024-09-11T08:08:41.413129+0000","last_fullsized":"2024-09-11T08:08:41.413129+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":42,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-11T08:08:48.330387+0000","last_change":"2024-09-11T08:08:48.330351+0000","last_active":"2024-09-11T08:08:48.330387+0000","last_peered":"2024-09-11T08:08:48.330387+0000","last_clean":"2024-09-11T08:08:48.330387+0000","last_became_active":"2024-09-11T08:07:30.043691+0000","last_became_peered":"2024-09-11T08:07:30.043691+0000","last_unstale":"2024-09-11T08:08:48.330387+0000","last_undegraded":"2024-09-11T08:08:48.330387+0000","last_fullsized":"2024-09-11T08:08:48.330387+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:48.330285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:48.330285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:05.763303+0000","scrub_duration":0.001379929,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.415286+0000","last_change":"2024-09-11T08:07:34.053427+0000","last_active":"2024-09-11T08:08:41.415286+0000","last_peered":"2024-09-11T08:08:41.415286+0000","last_clean":"2024-09-11T08:08:41.415286+0000","last_became_active":"2024-09-11T08:07:34.053137+0000","last_became_peered":"2024-09-11T08:07:34.053137+0000","last_unstale":"2024-09-11T08:08:41.415286+0000","last_undegraded":"2024-09-11T08:08:41.415286+0000","last_fullsized":"2024-09-11T08:08:41.415286+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:00.814813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.413962+0000","last_change":"2024-09-11T08:07:36.076837+0000","last_active":"2024-09-11T08:08:41.413962+0000","last_peered":"2024-09-11T08:08:41.413962+0000","last_clean":"2024-09-11T08:08:41.413962+0000","last_became_active":"2024-09-11T08:07:36.076132+0000","last_became_peered":"2024-09-11T08:07:36.076132+0000","last_unstale":"2024-09-11T08:08:41.413962+0000","last_undegraded":"2024-09-11T08:08:41.413962+0000","last_fullsized":"2024-09-11T08:08:41.413962+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:01.236619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":46,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.415518+0000","last_change":"2024-09-11T08:07:32.049611+0000","last_active":"2024-09-11T08:08:41.415518+0000","last_peered":"2024-09-11T08:08:41.415518+0000","last_clean":"2024-09-11T08:08:41.415518+0000","last_became_active":"2024-09-11T08:07:32.049368+0000","last_became_peered":"2024-09-11T08:07:32.049368+0000","last_unstale":"2024-09-11T08:08:41.415518+0000","last_undegraded":"2024-09-11T08:08:41.415518+0000","last_fullsized":"2024-09-11T08:08:41.415518+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:49.402283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":42,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-11T08:08:45.388123+0000","last_change":"2024-09-11T08:08:45.388090+0000","last_active":"2024-09-11T08:08:45.388123+0000","last_peered":"2024-09-11T08:08:45.388123+0000","last_clean":"2024-09-11T08:08:45.388123+0000","last_became_active":"2024-09-11T08:07:30.037692+0000","last_became_peered":"2024-09-11T08:07:30.037692+0000","last_unstale":"2024-09-11T08:08:45.388123+0000","last_undegraded":"2024-09-11T08:08:45.388123+0000","last_fullsized":"2024-09-11T08:08:45.388123+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:45.388024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:45.388024+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:59.280827+0000","scrub_duration":0.0014875719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.414349+0000","last_change":"2024-09-11T08:07:34.057358+0000","last_active":"2024-09-11T08:08:41.414349+0000","last_peered":"2024-09-11T08:08:41.414349+0000","last_clean":"2024-09-11T08:08:41.414349+0000","last_became_active":"2024-09-11T08:07:34.057064+0000","last_became_peered":"2024-09-11T08:07:34.057064+0000","last_unstale":"2024-09-11T08:08:41.414349+0000","last_undegraded":"2024-09-11T08:08:41.414349+0000","last_fullsized":"2024-09-11T08:08:41.414349+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.434869+0000","last_change":"2024-09-11T08:07:36.077697+0000","last_active":"2024-09-11T08:08:41.434869+0000","last_peered":"2024-09-11T08:08:41.434869+0000","last_clean":"2024-09-11T08:08:41.434869+0000","last_became_active":"2024-09-11T08:07:36.077406+0000","last_became_peered":"2024-09-11T08:07:36.077406+0000","last_unstale":"2024-09-11T08:08:41.434869+0000","last_undegraded":"2024-09-11T08:08:41.434869+0000","last_fullsized":"2024-09-11T08:08:41.434869+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:48.444976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":49,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.414098+0000","last_change":"2024-09-11T08:07:32.043568+0000","last_active":"2024-09-11T08:08:41.414098+0000","last_peered":"2024-09-11T08:08:41.414098+0000","last_clean":"2024-09-11T08:08:41.414098+0000","last_became_active":"2024-09-11T08:07:32.043276+0000","last_became_peered":"2024-09-11T08:07:32.043276+0000","last_unstale":"2024-09-11T08:08:41.414098+0000","last_undegraded":"2024-09-11T08:08:41.414098+0000","last_fullsized":"2024-09-11T08:08:41.414098+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:47.264071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.054987+0000","last_change":"2024-09-11T08:07:30.049461+0000","last_active":"2024-09-11T08:08:42.054987+0000","last_peered":"2024-09-11T08:08:42.054987+0000","last_clean":"2024-09-11T08:08:42.054987+0000","last_became_active":"2024-09-11T08:07:30.049152+0000","last_became_peered":"2024-09-11T08:07:30.049152+0000","last_unstale":"2024-09-11T08:08:42.054987+0000","last_undegraded":"2024-09-11T08:08:42.054987+0000","last_fullsized":"2024-09-11T08:08:42.054987+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:48.303459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.378163+0000","last_change":"2024-09-11T08:07:34.057322+0000","last_active":"2024-09-11T08:08:42.378163+0000","last_peered":"2024-09-11T08:08:42.378163+0000","last_clean":"2024-09-11T08:08:42.378163+0000","last_became_active":"2024-09-11T08:07:34.056935+0000","last_became_peered":"2024-09-11T08:07:34.056935+0000","last_unstale":"2024-09-11T08:08:42.378163+0000","last_undegraded":"2024-09-11T08:08:42.378163+0000","last_fullsized":"2024-09-11T08:08:42.378163+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:57.406825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.414480+0000","last_change":"2024-09-11T08:07:36.633645+0000","last_active":"2024-09-11T08:08:41.414480+0000","last_peered":"2024-09-11T08:08:41.414480+0000","last_clean":"2024-09-11T08:08:41.414480+0000","last_became_active":"2024-09-11T08:07:36.633368+0000","last_became_peered":"2024-09-11T08:07:36.633368+0000","last_unstale":"2024-09-11T08:08:41.414480+0000","last_undegraded":"2024-09-11T08:08:41.414480+0000","last_fullsized":"2024-09-11T08:08:41.414480+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:48.747284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.414046+0000","last_change":"2024-09-11T08:07:32.046050+0000","last_active":"2024-09-11T08:08:41.414046+0000","last_peered":"2024-09-11T08:08:41.414046+0000","last_clean":"2024-09-11T08:08:41.414046+0000","last_became_active":"2024-09-11T08:07:32.045728+0000","last_became_peered":"2024-09-11T08:07:32.045728+0000","last_unstale":"2024-09-11T08:08:41.414046+0000","last_undegraded":"2024-09-11T08:08:41.414046+0000","last_fullsized":"2024-09-11T08:08:41.414046+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:54.611022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.056694+0000","last_change":"2024-09-11T08:08:42.056647+0000","last_active":"2024-09-11T08:08:42.056694+0000","last_peered":"2024-09-11T08:08:42.056694+0000","last_clean":"2024-09-11T08:08:42.056694+0000","last_became_active":"2024-09-11T08:07:30.049409+0000","last_became_peered":"2024-09-11T08:07:30.049409+0000","last_unstale":"2024-09-11T08:08:42.056694+0000","last_undegraded":"2024-09-11T08:08:42.056694+0000","last_fullsized":"2024-09-11T08:08:42.056694+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:42.056591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:42.056591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:56.872271+0000","scrub_duration":0.003068809,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.445735+0000","last_change":"2024-09-11T08:08:34.506991+0000","last_active":"2024-09-11T08:08:42.445735+0000","last_peered":"2024-09-11T08:08:42.445735+0000","last_clean":"2024-09-11T08:08:42.445735+0000","last_became_active":"2024-09-11T08:07:34.055082+0000","last_became_peered":"2024-09-11T08:07:34.055082+0000","last_unstale":"2024-09-11T08:08:42.445735+0000","last_undegraded":"2024-09-11T08:08:42.445735+0000","last_fullsized":"2024-09-11T08:08:42.445735+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:34.506927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:34.506927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:46.584592+0000","scrub_duration":0.0015499070000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.377962+0000","last_change":"2024-09-11T08:07:36.072035+0000","last_active":"2024-09-11T08:08:42.377962+0000","last_peered":"2024-09-11T08:08:42.377962+0000","last_clean":"2024-09-11T08:08:42.377962+0000","last_became_active":"2024-09-11T08:07:36.071669+0000","last_became_peered":"2024-09-11T08:07:36.071669+0000","last_unstale":"2024-09-11T08:08:42.377962+0000","last_undegraded":"2024-09-11T08:08:42.377962+0000","last_fullsized":"2024-09-11T08:08:42.377962+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:03.407787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.414019+0000","last_change":"2024-09-11T08:07:36.071729+0000","last_active":"2024-09-11T08:08:41.414019+0000","last_peered":"2024-09-11T08:08:41.414019+0000","last_clean":"2024-09-11T08:08:41.414019+0000","last_became_active":"2024-09-11T08:07:36.071468+0000","last_became_peered":"2024-09-11T08:07:36.071468+0000","last_unstale":"2024-09-11T08:08:41.414019+0000","last_undegraded":"2024-09-11T08:08:41.414019+0000","last_fullsized":"2024-09-11T08:08:41.414019+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:54.816006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":42,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-11T08:08:49.428072+0000","last_change":"2024-09-11T08:08:49.428032+0000","last_active":"2024-09-11T08:08:49.428072+0000","last_peered":"2024-09-11T08:08:49.428072+0000","last_clean":"2024-09-11T08:08:49.428072+0000","last_became_active":"2024-09-11T08:07:30.046702+0000","last_became_peered":"2024-09-11T08:07:30.046702+0000","last_unstale":"2024-09-11T08:08:49.428072+0000","last_undegraded":"2024-09-11T08:08:49.428072+0000","last_fullsized":"2024-09-11T08:08:49.428072+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:49.427944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:49.427944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:11.537893+0000","scrub_duration":0.001383034,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":48,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:42.446325+0000","last_change":"2024-09-11T08:07:32.054959+0000","last_active":"2024-09-11T08:08:42.446325+0000","last_peered":"2024-09-11T08:08:42.446325+0000","last_clean":"2024-09-11T08:08:42.446325+0000","last_became_active":"2024-09-11T08:07:32.054656+0000","last_became_peered":"2024-09-11T08:07:32.054656+0000","last_unstale":"2024-09-11T08:08:42.446325+0000","last_undegraded":"2024-09-11T08:08:42.446325+0000","last_fullsized":"2024-09-11T08:08:42.446325+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:47.428697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T08:08:41.435434+0000","last_change":"2024-09-11T08:07:34.061746+0000","last_active":"2024-09-11T08:08:41.435434+0000","last_peered":"2024-09-11T08:08:41.435434+0000","last_clean":"2024-09-11T08:08:41.435434+0000","last_became_active":"2024-09-11T08:07:34.061453+0000","last_became_peered":"2024-09-11T08:07:34.061453+0000","last_unstale":"2024-09-11T08:08:41.435434+0000","last_undegraded":"2024-09-11T08:08:41.435434+0000","last_fullsized":"2024-09-11T08:08:41.435434+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":399,"ondisk_log_size":399,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":29,"num_read_kb":29,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":76,"num_read_kb":71,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266709,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298756,"kb_used_data":1404,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446396,"statfs":{"total":95995035648,"available":95689109504,"internally_reserved":0,"allocated":1437696,"data_stored":938655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.63300000000000001}]},{"osd":1,"last update":"Wed Sep 11 08:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.621},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.53500000000000003}]},{"osd":2,"last update":"Wed Sep 11 08:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.56200000000000006}]},{"osd":3,"last update":"Wed Sep 11 08:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":7.9210000000000003,"5min":7.9210000000000003,"15min":7.9210000000000003},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.59199999999999997}]},{"osd":4,"last update":"Wed Sep 11 08:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.379}]},{"osd":5,"last update":"Wed Sep 11 08:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.443}]},{"osd":6,"last update":"Wed Sep 11 08:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.434}]}]},{"osd":6,"up_from":45,"seq":193273528346,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298744,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446408,"statfs":{"total":95995035648,"available":95689121792,"internally_reserved":0,"allocated":1421312,"data_stored":938478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:07:53 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.45200000000000001}]},{"osd":1,"last update":"Wed Sep 11 08:07:53 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.46899999999999997}]},{"osd":2,"last update":"Wed Sep 11 08:07:53 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.504},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.432}]},{"osd":3,"last update":"Wed Sep 11 08:07:53 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.61099999999999999}]},{"osd":4,"last update":"Wed Sep 11 08:07:53 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.26800000000000002}]},{"osd":5,"last update":"Wed Sep 11 08:07:53 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.317},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.254}]},{"osd":7,"last update":"Wed Sep 11 08:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.29999999999999999}]}]},{"osd":5,"up_from":38,"seq":163208757279,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298156,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446996,"statfs":{"total":95995035648,"available":95689723904,"internally_reserved":0,"allocated":823296,"data_stored":347317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.51400000000000001,"15min":0.52200000000000002},"min":{"1min":0.318,"5min":0.31,"15min":0.31},"max":{"1min":0.63500000000000001,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.503,"15min":0.501},"min":{"1min":0.41399999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.64000000000000001}]},{"osd":1,"last update":"Wed Sep 11 08:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":1.1910000000000001,"15min":1.3029999999999999},"min":{"1min":0.33000000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.82499999999999996,"5min":16.004999999999999,"15min":16.004999999999999},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.36199999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.61299999999999999}]},{"osd":2,"last update":"Wed Sep 11 08:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":1.206,"15min":1.323},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.754,"5min":16.018000000000001,"15min":16.018000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.65600000000000003}]},{"osd":3,"last update":"Wed Sep 11 08:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.53100000000000003,"15min":0.52900000000000003},"min":{"1min":0.44900000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.72299999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.505,"5min":0.51600000000000001,"15min":0.51800000000000002},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.63200000000000001}]},{"osd":4,"last update":"Wed Sep 11 08:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.39700000000000002,"15min":0.39300000000000002},"min":{"1min":0.27500000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.40000000000000002,"15min":0.38800000000000001},"min":{"1min":0.36499999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.50800000000000001}]},{"osd":6,"last update":"Wed Sep 11 08:07:53 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.57299999999999995}]},{"osd":7,"last update":"Wed Sep 11 08:08:22 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.54700000000000004}]}]},{"osd":4,"up_from":31,"seq":133143986212,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298180,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446972,"statfs":{"total":95995035648,"available":95689699328,"internally_reserved":0,"allocated":847872,"data_stored":348670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.499,"15min":0.50900000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.59699999999999998,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.52400000000000002,"15min":0.53500000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.58899999999999997,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.436}]},{"osd":1,"last update":"Wed Sep 11 08:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.505,"15min":0.51300000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.66800000000000004,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.52300000000000002,"15min":0.52900000000000003},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.68899999999999995,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.66900000000000004}]},{"osd":2,"last update":"Wed Sep 11 08:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.52000000000000002,"15min":0.52800000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.58799999999999997,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.51600000000000001,"15min":0.52500000000000002},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.623,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.58399999999999996}]},{"osd":3,"last update":"Wed Sep 11 08:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.50900000000000001,"15min":0.51500000000000001},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.64300000000000002,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.51800000000000002,"15min":0.52600000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.61499999999999999,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.50700000000000001}]},{"osd":5,"last update":"Wed Sep 11 08:08:31 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.36899999999999999,"15min":0.36699999999999999},"min":{"1min":0.27800000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.36899999999999999,"15min":0.36699999999999999},"min":{"1min":0.27900000000000003,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.48399999999999999}]},{"osd":6,"last update":"Wed Sep 11 08:07:53 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.38,"15min":0.38},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.628}]},{"osd":7,"last update":"Wed Sep 11 08:08:20 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.755},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.61699999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182441,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298196,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446956,"statfs":{"total":95995035648,"available":95689682944,"internally_reserved":0,"allocated":864256,"data_stored":348456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:07:52 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.373,"15min":0.377},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.51200000000000001,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.376,"5min":0.41599999999999998,"15min":0.42199999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.53700000000000003,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.35999999999999999}]},{"osd":1,"last update":"Wed Sep 11 08:07:52 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.435,"15min":0.443},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.57499999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.317},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.43099999999999999,"15min":0.435},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.30199999999999999}]},{"osd":2,"last update":"Wed Sep 11 08:07:52 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.45000000000000001,"15min":0.45400000000000001},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.53700000000000003,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.41699999999999998,"15min":0.42299999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.27400000000000002}]},{"osd":4,"last update":"Wed Sep 11 08:08:12 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54800000000000004,"15min":0.54900000000000004},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.53500000000000003,"15min":0.53700000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.66500000000000004}]},{"osd":5,"last update":"Wed Sep 11 08:08:33 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.54200000000000004,"15min":0.53800000000000003},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.40699999999999997,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.55700000000000005}]},{"osd":6,"last update":"Wed Sep 11 08:07:55 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.46899999999999997}]},{"osd":7,"last update":"Wed Sep 11 08:08:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.64700000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411372,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298152,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447000,"statfs":{"total":95995035648,"available":95689728000,"internally_reserved":0,"allocated":819200,"data_stored":347048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:08:24 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.38400000000000001,"15min":0.38100000000000001},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.58099999999999996,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.37,"15min":0.35999999999999999},"min":{"1min":0.25,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.55600000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.441}]},{"osd":1,"last update":"Wed Sep 11 08:08:24 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.39500000000000002,"15min":0.38},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.60899999999999999,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.433},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.39800000000000002,"15min":0.38600000000000001},"min":{"1min":0.28699999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.41699999999999998}]},{"osd":3,"last update":"Wed Sep 11 08:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.45100000000000001,"15min":0.46100000000000002},"min":{"1min":0.27300000000000002,"5min":0.218,"15min":0.218},"max":{"1min":0.629,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.45100000000000001,"15min":0.46500000000000002},"min":{"1min":0.318,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.36199999999999999}]},{"osd":4,"last update":"Wed Sep 11 08:08:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.58299999999999996,"15min":0.58699999999999997},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.71199999999999997,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.59799999999999998}]},{"osd":5,"last update":"Wed Sep 11 08:08:35 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.57699999999999996,"15min":0.58299999999999996},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.751,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.59199999999999997,"15min":0.59999999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.65300000000000002,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.64300000000000002}]},{"osd":6,"last update":"Wed Sep 11 08:07:57 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.55600000000000005}]},{"osd":7,"last update":"Wed Sep 11 08:08:22 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.51000000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298208,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446944,"statfs":{"total":95995035648,"available":95689670656,"internally_reserved":0,"allocated":876544,"data_stored":348614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.32000000000000001,"15min":0.29799999999999999},"min":{"1min":0.22600000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.33200000000000002,"15min":0.312},"min":{"1min":0.249,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"last":0.29299999999999998}]},{"osd":2,"last update":"Wed Sep 11 08:08:28 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.36199999999999999,"15min":0.34799999999999998},"min":{"1min":0.27700000000000002,"5min":0.224,"15min":0.224},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.21299999999999999},{"interface":"front","average":{"1min":0.37,"5min":0.35399999999999998,"15min":0.34200000000000003},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.307}]},{"osd":3,"last update":"Wed Sep 11 08:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.41399999999999998,"15min":0.42199999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.439,"15min":0.437},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.198,"5min":1.198,"15min":1.198},"last":0.32300000000000001}]},{"osd":4,"last update":"Wed Sep 11 08:08:08 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54300000000000004,"15min":0.54200000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.55800000000000005,"15min":0.55600000000000005},"min":{"1min":0.41899999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.73499999999999999,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.60499999999999998}]},{"osd":5,"last update":"Wed Sep 11 08:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.53700000000000003,"15min":0.53000000000000003},"min":{"1min":0.40699999999999997,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.77600000000000002,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.53100000000000003,"15min":0.52700000000000002},"min":{"1min":0.38400000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.58799999999999997}]},{"osd":6,"last update":"Wed Sep 11 08:07:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.55600000000000005}]},{"osd":7,"last update":"Wed Sep 11 08:08:17 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.495}]}]},{"osd":0,"up_from":8,"seq":34359738422,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298768,"kb_used_data":1416,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446384,"statfs":{"total":95995035648,"available":95689097216,"internally_reserved":0,"allocated":1449984,"data_stored":939601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 11 08:08:01 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.34100000000000003,"15min":0.315},"min":{"1min":0.25800000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.53900000000000003,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.35599999999999998,"15min":0.33700000000000002},"min":{"1min":0.23999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.47999999999999998}]},{"osd":2,"last update":"Wed Sep 11 08:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.34699999999999998},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.48799999999999999,"5min":0.53000000000000003,"15min":0.53000000000000003},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.378,"5min":0.378,"15min":0.36899999999999999},"min":{"1min":0.26600000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.56299999999999994,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.375}]},{"osd":3,"last update":"Wed Sep 11 08:08:51 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.44,"15min":0.44},"min":{"1min":0.27600000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.52300000000000002,"5min":0.62,"15min":0.62},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.42499999999999999,"15min":0.42599999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.52800000000000002,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.35699999999999998}]},{"osd":4,"last update":"Wed Sep 11 08:08:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.55000000000000004,"15min":0.54600000000000004},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.629}]},{"osd":5,"last update":"Wed Sep 11 08:08:36 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.58499999999999996,"15min":0.58299999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.748,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.55800000000000005,"15min":0.55300000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.66900000000000004}]},{"osd":6,"last update":"Wed Sep 11 08:07:56 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.65500000000000003}]},{"osd":7,"last update":"Wed Sep 11 08:08:19 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.69299999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T08:08:57.241 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-11T08:08:57.241 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-11T08:08:57.242 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-11T08:08:57.242 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph health --format=json 2024-09-11T08:08:57.488 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:08:57.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:57 smithi150 ceph-mon[29276]: 4.19 scrub starts 2024-09-11T08:08:57.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:57 smithi150 ceph-mon[29276]: 4.19 scrub ok 2024-09-11T08:08:57.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:57 smithi150 ceph-mon[29276]: from='client.14685 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:08:57.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:57 smithi150 ceph-mon[29276]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-11T08:08:57.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:57 smithi150 ceph-mon[29276]: 4.5 deep-scrub starts 2024-09-11T08:08:57.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:57 smithi150 ceph-mon[29276]: 4.5 deep-scrub ok 2024-09-11T08:08:57.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:57 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:57.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:57 smithi150 ceph-mon[29276]: 4.1c scrub starts 2024-09-11T08:08:57.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:57 smithi150 ceph-mon[29276]: 4.1c scrub ok 2024-09-11T08:08:57.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:57 smithi150 ceph-mon[39612]: 4.19 scrub starts 2024-09-11T08:08:57.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:57 smithi150 ceph-mon[39612]: 4.19 scrub ok 2024-09-11T08:08:57.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:57 smithi150 ceph-mon[39612]: from='client.14685 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:08:57.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:57 smithi150 ceph-mon[39612]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-11T08:08:57.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:57 smithi150 ceph-mon[39612]: 4.5 deep-scrub starts 2024-09-11T08:08:57.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:57 smithi150 ceph-mon[39612]: 4.5 deep-scrub ok 2024-09-11T08:08:57.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:57 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:57.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:57 smithi150 ceph-mon[39612]: 4.1c scrub starts 2024-09-11T08:08:57.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:57 smithi150 ceph-mon[39612]: 4.1c scrub ok 2024-09-11T08:08:57.823 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:08:57 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:08:57.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:57 smithi181 ceph-mon[32701]: 4.19 scrub starts 2024-09-11T08:08:57.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:57 smithi181 ceph-mon[32701]: 4.19 scrub ok 2024-09-11T08:08:57.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:57 smithi181 ceph-mon[32701]: from='client.14685 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:08:57.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:57 smithi181 ceph-mon[32701]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-11T08:08:57.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:57 smithi181 ceph-mon[32701]: 4.5 deep-scrub starts 2024-09-11T08:08:57.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:57 smithi181 ceph-mon[32701]: 4.5 deep-scrub ok 2024-09-11T08:08:57.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:57 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:08:57.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:57 smithi181 ceph-mon[32701]: 4.1c scrub starts 2024-09-11T08:08:57.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:57 smithi181 ceph-mon[32701]: 4.1c scrub ok 2024-09-11T08:08:58.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:58 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:08:58.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:58 smithi150 ceph-mon[29276]: 5.6 scrub starts 2024-09-11T08:08:58.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:58 smithi150 ceph-mon[29276]: 5.6 scrub ok 2024-09-11T08:08:58.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:58 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:08:58.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:58 smithi150 ceph-mon[39612]: 5.6 scrub starts 2024-09-11T08:08:58.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:58 smithi150 ceph-mon[39612]: 5.6 scrub ok 2024-09-11T08:08:58.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:58 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:08:58.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:58 smithi181 ceph-mon[32701]: 5.6 scrub starts 2024-09-11T08:08:58.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:58 smithi181 ceph-mon[32701]: 5.6 scrub ok 2024-09-11T08:08:59.375 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:08:59.375 INFO:teuthology.orchestra.run.smithi150.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-11T08:08:59.813 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:59 smithi181 ceph-mon[32701]: 4.16 deep-scrub starts 2024-09-11T08:08:59.814 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:59 smithi181 ceph-mon[32701]: 4.16 deep-scrub ok 2024-09-11T08:08:59.814 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:59 smithi181 ceph-mon[32701]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:08:59.814 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:59 smithi181 ceph-mon[32701]: 5.f scrub starts 2024-09-11T08:08:59.814 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:59 smithi181 ceph-mon[32701]: 5.f scrub ok 2024-09-11T08:08:59.814 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:08:59 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3411305067' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-11T08:08:59.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:59 smithi150 ceph-mon[29276]: 4.16 deep-scrub starts 2024-09-11T08:08:59.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:59 smithi150 ceph-mon[29276]: 4.16 deep-scrub ok 2024-09-11T08:08:59.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:59 smithi150 ceph-mon[29276]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:08:59.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:59 smithi150 ceph-mon[29276]: 5.f scrub starts 2024-09-11T08:08:59.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:59 smithi150 ceph-mon[29276]: 5.f scrub ok 2024-09-11T08:08:59.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:08:59 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3411305067' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-11T08:08:59.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:59 smithi150 ceph-mon[39612]: 4.16 deep-scrub starts 2024-09-11T08:08:59.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:59 smithi150 ceph-mon[39612]: 4.16 deep-scrub ok 2024-09-11T08:08:59.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:59 smithi150 ceph-mon[39612]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:08:59.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:59 smithi150 ceph-mon[39612]: 5.f scrub starts 2024-09-11T08:08:59.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:59 smithi150 ceph-mon[39612]: 5.f scrub ok 2024-09-11T08:08:59.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:08:59 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3411305067' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-11T08:09:00.051 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-11T08:09:00.051 INFO:tasks.cephadm:Setup complete, yielding 2024-09-11T08:09:00.051 INFO:teuthology.run_tasks:Running task thrashosds... 2024-09-11T08:09:00.063 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-11T08:09:00.063 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-guits-quincy-2024-09-10-2008-quincy'}, 'ceph': {'conf': {'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'd0e6828a2016d48cf25ad84064e50742bb1c39b9'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'cephadm': {'cephadm_mode': 'root'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd0e6828a2016d48cf25ad84064e50742bb1c39b9'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-guits-quincy-2024-09-10-2008-quincy', 'sha1': 'd0e6828a2016d48cf25ad84064e50742bb1c39b9'}} 2024-09-11T08:09:00.063 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-11T08:09:00.063 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-09-11T08:09:00.307 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:09:00.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:00 smithi150 ceph-mon[29276]: 5.9 scrub starts 2024-09-11T08:09:00.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:00 smithi150 ceph-mon[29276]: 5.9 scrub ok 2024-09-11T08:09:00.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:00 smithi150 ceph-mon[29276]: 4.9 scrub starts 2024-09-11T08:09:00.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:00 smithi150 ceph-mon[29276]: 4.9 scrub ok 2024-09-11T08:09:00.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:00 smithi150 ceph-mon[39612]: 5.9 scrub starts 2024-09-11T08:09:00.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:00 smithi150 ceph-mon[39612]: 5.9 scrub ok 2024-09-11T08:09:00.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:00 smithi150 ceph-mon[39612]: 4.9 scrub starts 2024-09-11T08:09:00.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:00 smithi150 ceph-mon[39612]: 4.9 scrub ok 2024-09-11T08:09:00.768 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:00 smithi181 ceph-mon[32701]: 5.9 scrub starts 2024-09-11T08:09:00.768 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:00 smithi181 ceph-mon[32701]: 5.9 scrub ok 2024-09-11T08:09:00.768 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:00 smithi181 ceph-mon[32701]: 4.9 scrub starts 2024-09-11T08:09:00.769 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:00 smithi181 ceph-mon[32701]: 4.9 scrub ok 2024-09-11T08:09:01.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:01 smithi150 ceph-mon[29276]: 6.b scrub starts 2024-09-11T08:09:01.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:01 smithi150 ceph-mon[29276]: 6.b scrub ok 2024-09-11T08:09:01.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:01 smithi150 ceph-mon[29276]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-11T08:09:01.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:01 smithi150 ceph-mon[29276]: 3.5 scrub starts 2024-09-11T08:09:01.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:01 smithi150 ceph-mon[29276]: 3.5 scrub ok 2024-09-11T08:09:01.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:01 smithi150 ceph-mon[39612]: 6.b scrub starts 2024-09-11T08:09:01.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:01 smithi150 ceph-mon[39612]: 6.b scrub ok 2024-09-11T08:09:01.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:01 smithi150 ceph-mon[39612]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-11T08:09:01.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:01 smithi150 ceph-mon[39612]: 3.5 scrub starts 2024-09-11T08:09:01.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:01 smithi150 ceph-mon[39612]: 3.5 scrub ok 2024-09-11T08:09:01.808 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:01 smithi181 ceph-mon[32701]: 6.b scrub starts 2024-09-11T08:09:01.808 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:01 smithi181 ceph-mon[32701]: 6.b scrub ok 2024-09-11T08:09:01.808 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:01 smithi181 ceph-mon[32701]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-11T08:09:01.808 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:01 smithi181 ceph-mon[32701]: 3.5 scrub starts 2024-09-11T08:09:01.808 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:01 smithi181 ceph-mon[32701]: 3.5 scrub ok 2024-09-11T08:09:02.157 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:09:01 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-alertmanager-a[72111]: ts=2024-09-11T08:09:01.738Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002125494s 2024-09-11T08:09:02.489 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:09:02 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: [11/Sep/2024:08:09:02] ENGINE Bus STOPPING 2024-09-11T08:09:02.497 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 podman[63437]: 2024-09-11 08:09:02.450478691 +0000 UTC m=+0.142410237 container create d20c42749742dd0da7ff419a232b487ec4285591c8d96fc3fd943b9351f7ba41 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a, io.k8s.display-name=Red Hat Universal Base Image 8, io.buildah.version=1.32.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, maintainer=Paul Cuzner , vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2022-04-29T11:48:11.055897, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, description=Ceph Grafana Container, com.redhat.component=ubi8-container, release=239.1651231664, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, io.openshift.tags=base rhel8, name=ubi8, vcs-type=git, distribution-scope=public, io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com) 2024-09-11T08:09:02.778 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-mon[32701]: 2.0 scrub starts 2024-09-11T08:09:02.778 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-mon[32701]: 2.0 scrub ok 2024-09-11T08:09:02.778 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-mon[32701]: 5.1d scrub starts 2024-09-11T08:09:02.778 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-mon[32701]: 5.1d scrub ok 2024-09-11T08:09:02.778 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-mon[32701]: 6.7 deep-scrub starts 2024-09-11T08:09:02.778 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-mon[32701]: 6.7 deep-scrub ok 2024-09-11T08:09:02.778 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-mon[32701]: 6.14 scrub starts 2024-09-11T08:09:02.779 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-mon[32701]: 6.14 scrub ok 2024-09-11T08:09:02.779 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' 2024-09-11T08:09:02.779 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 podman[63437]: 2024-09-11 08:09:02.694552596 +0000 UTC m=+0.386484152 container init d20c42749742dd0da7ff419a232b487ec4285591c8d96fc3fd943b9351f7ba41 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a, vcs-type=git, name=ubi8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, io.buildah.version=1.32.0, distribution-scope=public, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-04-29T11:48:11.055897, io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64, io.openshift.tags=base rhel8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.expose-services=, version=8.5, maintainer=Paul Cuzner , release=239.1651231664) 2024-09-11T08:09:02.779 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 podman[63437]: 2024-09-11 08:09:02.697391702 +0000 UTC m=+0.389323249 container start d20c42749742dd0da7ff419a232b487ec4285591c8d96fc3fd943b9351f7ba41 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, io.buildah.version=1.32.0, vendor=Red Hat, Inc., io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.openshift.tags=base rhel8, distribution-scope=public, build-date=2022-04-29T11:48:11.055897, version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.component=ubi8-container, release=239.1651231664) 2024-09-11T08:09:02.865 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:09:02 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: [11/Sep/2024:08:09:02] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-11T08:09:02.865 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:09:02 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: [11/Sep/2024:08:09:02] ENGINE Bus STOPPED 2024-09-11T08:09:02.865 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:09:02 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: [11/Sep/2024:08:09:02] ENGINE Bus STARTING 2024-09-11T08:09:02.865 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:09:02 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: [11/Sep/2024:08:09:02] ENGINE Serving on http://:::9283 2024-09-11T08:09:02.865 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:09:02 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: [11/Sep/2024:08:09:02] ENGINE Bus STARTED 2024-09-11T08:09:02.866 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:02 smithi150 ceph-mon[29276]: 2.0 scrub starts 2024-09-11T08:09:02.866 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:02 smithi150 ceph-mon[29276]: 2.0 scrub ok 2024-09-11T08:09:02.866 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:02 smithi150 ceph-mon[29276]: 5.1d scrub starts 2024-09-11T08:09:02.866 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:02 smithi150 ceph-mon[29276]: 5.1d scrub ok 2024-09-11T08:09:02.866 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:02 smithi150 ceph-mon[29276]: 6.7 deep-scrub starts 2024-09-11T08:09:02.866 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:02 smithi150 ceph-mon[29276]: 6.7 deep-scrub ok 2024-09-11T08:09:02.866 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:02 smithi150 ceph-mon[29276]: 6.14 scrub starts 2024-09-11T08:09:02.866 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:02 smithi150 ceph-mon[29276]: 6.14 scrub ok 2024-09-11T08:09:02.866 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:02 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' 2024-09-11T08:09:02.866 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:02 smithi150 ceph-mon[39612]: 2.0 scrub starts 2024-09-11T08:09:02.866 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:02 smithi150 ceph-mon[39612]: 2.0 scrub ok 2024-09-11T08:09:02.867 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:02 smithi150 ceph-mon[39612]: 5.1d scrub starts 2024-09-11T08:09:02.867 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:02 smithi150 ceph-mon[39612]: 5.1d scrub ok 2024-09-11T08:09:02.867 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:02 smithi150 ceph-mon[39612]: 6.7 deep-scrub starts 2024-09-11T08:09:02.867 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:02 smithi150 ceph-mon[39612]: 6.7 deep-scrub ok 2024-09-11T08:09:02.867 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:02 smithi150 ceph-mon[39612]: 6.14 scrub starts 2024-09-11T08:09:02.867 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:02 smithi150 ceph-mon[39612]: 6.14 scrub ok 2024-09-11T08:09:02.867 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:02 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' 2024-09-11T08:09:02.890 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-11T08:09:02.890 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-11T08:09:02.891 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T08:09:03.029 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 bash[63437]: d20c42749742dd0da7ff419a232b487ec4285591c8d96fc3fd943b9351f7ba41 2024-09-11T08:09:03.029 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 systemd[1]: Started Ceph grafana.a for c1bddff4-7013-11ef-bcea-c7b262605968. 2024-09-11T08:09:03.029 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=settings t=2024-09-11T08:09:02.851035896Z level=info msg="Starting Grafana" version=9.4.7 commit=4add91f03d branch=HEAD compiled=2023-03-16T23:56:52Z 2024-09-11T08:09:03.029 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=settings t=2024-09-11T08:09:02.851242386Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-09-11T08:09:03.029 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=settings t=2024-09-11T08:09:02.851268883Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-09-11T08:09:03.030 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=settings t=2024-09-11T08:09:02.851285399Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-09-11T08:09:03.030 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=settings t=2024-09-11T08:09:02.851295639Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-09-11T08:09:03.030 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=settings t=2024-09-11T08:09:02.851306563Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-09-11T08:09:03.030 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=settings t=2024-09-11T08:09:02.851316715Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-09-11T08:09:03.030 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=settings t=2024-09-11T08:09:02.851327715Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-09-11T08:09:03.030 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=settings t=2024-09-11T08:09:02.85133915Z level=info msg="Path Home" path=/usr/share/grafana 2024-09-11T08:09:03.030 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=settings t=2024-09-11T08:09:02.851350173Z level=info msg="Path Data" path=/var/lib/grafana 2024-09-11T08:09:03.030 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=settings t=2024-09-11T08:09:02.851360421Z level=info msg="Path Logs" path=/var/log/grafana 2024-09-11T08:09:03.030 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=settings t=2024-09-11T08:09:02.85137019Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-09-11T08:09:03.030 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=settings t=2024-09-11T08:09:02.851382211Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-09-11T08:09:03.030 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=settings t=2024-09-11T08:09:02.851393189Z level=info msg="App mode production" 2024-09-11T08:09:03.030 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=sqlstore t=2024-09-11T08:09:02.851467797Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-09-11T08:09:03.030 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=sqlstore t=2024-09-11T08:09:02.851495708Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-09-11T08:09:03.031 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.852643029Z level=info msg="Starting DB migrations" 2024-09-11T08:09:03.031 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.852741728Z level=info msg="Executing migration" id="create migration_log table" 2024-09-11T08:09:03.031 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.855671962Z level=info msg="Executing migration" id="create user table" 2024-09-11T08:09:03.031 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.856189923Z level=info msg="Executing migration" id="add unique index user.login" 2024-09-11T08:09:03.031 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.856684976Z level=info msg="Executing migration" id="add unique index user.email" 2024-09-11T08:09:03.031 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.857131958Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-09-11T08:09:03.031 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.857741678Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-09-11T08:09:03.031 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.858143197Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-09-11T08:09:03.031 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.858910376Z level=info msg="Executing migration" id="create user table v2" 2024-09-11T08:09:03.031 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.859309524Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-09-11T08:09:03.031 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.859726695Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-09-11T08:09:03.031 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.860161629Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-09-11T08:09:03.031 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.860489023Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-09-11T08:09:03.031 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.860851557Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-09-11T08:09:03.032 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.861241533Z level=info msg="Executing migration" id="Update user table charset" 2024-09-11T08:09:03.032 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.861519295Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-09-11T08:09:03.032 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.861937878Z level=info msg="Executing migration" id="Add missing user data" 2024-09-11T08:09:03.032 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.862266675Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-09-11T08:09:03.032 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.864683841Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-09-11T08:09:03.032 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.865100512Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-09-11T08:09:03.032 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.865501842Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-09-11T08:09:03.033 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.866610217Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-09-11T08:09:03.033 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.867055844Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-09-11T08:09:03.033 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.86752625Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-09-11T08:09:03.033 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.867945741Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-09-11T08:09:03.033 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.868361468Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-09-11T08:09:03.033 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.868773881Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-09-11T08:09:03.033 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.869088136Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-09-11T08:09:03.034 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.8694713Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-09-11T08:09:03.034 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.869883486Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-09-11T08:09:03.034 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.870256327Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-09-11T08:09:03.034 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.870671268Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-09-11T08:09:03.034 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.873645862Z level=info msg="Executing migration" id="create temp_user v2" 2024-09-11T08:09:03.034 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.87426518Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-09-11T08:09:03.034 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.874703207Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-09-11T08:09:03.034 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.875124904Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-09-11T08:09:03.034 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.875509543Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-09-11T08:09:03.034 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.877969807Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-09-11T08:09:03.034 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.878358567Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-09-11T08:09:03.034 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.878734121Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-09-11T08:09:03.034 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.879091075Z level=info msg="Executing migration" id="create star table" 2024-09-11T08:09:03.034 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.879491317Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-09-11T08:09:03.035 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.879909212Z level=info msg="Executing migration" id="create org table v1" 2024-09-11T08:09:03.035 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.880304784Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-09-11T08:09:03.035 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.880725646Z level=info msg="Executing migration" id="create org_user table v1" 2024-09-11T08:09:03.035 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.881114682Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-09-11T08:09:03.035 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.88153492Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-09-11T08:09:03.035 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.88195455Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-09-11T08:09:03.035 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.882399215Z level=info msg="Executing migration" id="Update org table charset" 2024-09-11T08:09:03.035 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.88527411Z level=info msg="Executing migration" id="Update org_user table charset" 2024-09-11T08:09:03.035 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.888069209Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-09-11T08:09:03.035 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.890649721Z level=info msg="Executing migration" id="create dashboard table" 2024-09-11T08:09:03.035 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.893281728Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-09-11T08:09:03.035 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.896162138Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-09-11T08:09:03.035 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.898886865Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-09-11T08:09:03.035 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.901418223Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-09-11T08:09:03.036 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.903980816Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-09-11T08:09:03.036 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.906710246Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-09-11T08:09:03.036 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.909424914Z level=info msg="Executing migration" id="create dashboard v2" 2024-09-11T08:09:03.036 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.912129668Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-09-11T08:09:03.036 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.912604874Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-09-11T08:09:03.036 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.913507866Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-09-11T08:09:03.037 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.914364969Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-09-11T08:09:03.037 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.915353444Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-09-11T08:09:03.037 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.918106366Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-09-11T08:09:03.037 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.921014082Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-09-11T08:09:03.037 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.923878657Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-09-11T08:09:03.037 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.926595581Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-09-11T08:09:03.037 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.929616822Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-09-11T08:09:03.037 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.932279623Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-09-11T08:09:03.037 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.935128608Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-09-11T08:09:03.037 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.938081998Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-09-11T08:09:03.037 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.940814465Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-09-11T08:09:03.037 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.943733261Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-09-11T08:09:03.037 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.946568881Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-09-11T08:09:03.037 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.949402985Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-09-11T08:09:03.038 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.952094403Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-09-11T08:09:03.038 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.954927933Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-09-11T08:09:03.038 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.957516574Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-09-11T08:09:03.038 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.960658397Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-09-11T08:09:03.038 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.963481015Z level=info msg="Executing migration" id="Update dashboard title length" 2024-09-11T08:09:03.038 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.966164752Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-09-11T08:09:03.038 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.968892232Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-09-11T08:09:03.038 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.971668392Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-09-11T08:09:03.038 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.974341339Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-09-11T08:09:03.038 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.977074754Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-09-11T08:09:03.038 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.979742229Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-09-11T08:09:03.038 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.982294472Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-09-11T08:09:03.038 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.984890878Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-09-11T08:09:03.038 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.987644395Z level=info msg="Executing migration" id="Add check_sum column" 2024-09-11T08:09:03.039 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.99026766Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-09-11T08:09:03.039 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.992659539Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-09-11T08:09:03.039 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.995352988Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-09-11T08:09:03.039 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:02 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:02.998249982Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-09-11T08:09:03.039 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.000965359Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-09-11T08:09:03.039 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.003581941Z level=info msg="Executing migration" id="create data_source table" 2024-09-11T08:09:03.039 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.00629065Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-09-11T08:09:03.039 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.00910894Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-09-11T08:09:03.039 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.01164246Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-09-11T08:09:03.039 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.012499165Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-09-11T08:09:03.039 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.013239873Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-09-11T08:09:03.039 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.014291792Z level=info msg="Executing migration" id="create data_source table v2" 2024-09-11T08:09:03.039 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.015250895Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-09-11T08:09:03.039 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.015958263Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-09-11T08:09:03.040 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.016792262Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-09-11T08:09:03.040 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.017754533Z level=info msg="Executing migration" id="Add column with_credentials" 2024-09-11T08:09:03.040 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.018537745Z level=info msg="Executing migration" id="Add secure json data column" 2024-09-11T08:09:03.040 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.019385179Z level=info msg="Executing migration" id="Update data_source table charset" 2024-09-11T08:09:03.040 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.020228145Z level=info msg="Executing migration" id="Update initial version to 1" 2024-09-11T08:09:03.040 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.021060977Z level=info msg="Executing migration" id="Add read_only data column" 2024-09-11T08:09:03.040 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.021629459Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-09-11T08:09:03.040 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.021942445Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-09-11T08:09:03.040 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.022854503Z level=info msg="Executing migration" id="Add uid column" 2024-09-11T08:09:03.040 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.02354985Z level=info msg="Executing migration" id="Update uid value" 2024-09-11T08:09:03.040 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.023864489Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-09-11T08:09:03.040 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.024700372Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-09-11T08:09:03.040 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.025132505Z level=info msg="Executing migration" id="create api_key table" 2024-09-11T08:09:03.040 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.025588556Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-09-11T08:09:03.041 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.026471503Z level=info msg="Executing migration" id="add index api_key.key" 2024-09-11T08:09:03.041 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.027320846Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-09-11T08:09:03.041 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.028217728Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-09-11T08:09:03.041 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.029107376Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-09-11T08:09:03.041 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.029557495Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-09-11T08:09:03.041 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.030454076Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-09-11T08:09:03.174 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:09:03.279 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.031779644Z level=info msg="Executing migration" id="create api_key table v2" 2024-09-11T08:09:03.279 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.032606748Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-09-11T08:09:03.279 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.033064458Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-09-11T08:09:03.279 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.033928414Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-09-11T08:09:03.279 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.034756868Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-09-11T08:09:03.279 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.035575354Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-09-11T08:09:03.279 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.035987677Z level=info msg="Executing migration" id="Update api_key table charset" 2024-09-11T08:09:03.279 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.036834393Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-09-11T08:09:03.279 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.03748008Z level=info msg="Executing migration" id="Add service account foreign key" 2024-09-11T08:09:03.279 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.038120472Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-09-11T08:09:03.279 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.04095142Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-09-11T08:09:03.279 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.041639585Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-09-11T08:09:03.280 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.042190945Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-09-11T08:09:03.280 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.044936835Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-09-11T08:09:03.280 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.045316314Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-09-11T08:09:03.280 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.048049647Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-09-11T08:09:03.280 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.050751096Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-09-11T08:09:03.280 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.051247409Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-09-11T08:09:03.280 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.054075588Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-09-11T08:09:03.281 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.056754416Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-09-11T08:09:03.281 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.059508336Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-09-11T08:09:03.281 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.060130131Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-09-11T08:09:03.281 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.063182125Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-09-11T08:09:03.281 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.065816391Z level=info msg="Executing migration" id="create quota table v1" 2024-09-11T08:09:03.281 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.068732444Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-09-11T08:09:03.281 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.069320678Z level=info msg="Executing migration" id="Update quota table charset" 2024-09-11T08:09:03.282 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.072191187Z level=info msg="Executing migration" id="create plugin_setting table" 2024-09-11T08:09:03.282 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.074755683Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-09-11T08:09:03.282 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.077480454Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-09-11T08:09:03.282 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.078046552Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-09-11T08:09:03.282 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.080546292Z level=info msg="Executing migration" id="create session table" 2024-09-11T08:09:03.282 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.081019587Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-09-11T08:09:03.282 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.08322729Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-09-11T08:09:03.282 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.083622422Z level=info msg="Executing migration" id="create playlist table v2" 2024-09-11T08:09:03.282 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.085885638Z level=info msg="Executing migration" id="create playlist item table v2" 2024-09-11T08:09:03.282 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.08629185Z level=info msg="Executing migration" id="Update playlist table charset" 2024-09-11T08:09:03.282 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.088831274Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-09-11T08:09:03.282 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.091639352Z level=info msg="Executing migration" id="drop preferences table v2" 2024-09-11T08:09:03.282 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.094306135Z level=info msg="Executing migration" id="drop preferences table v3" 2024-09-11T08:09:03.283 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.094754334Z level=info msg="Executing migration" id="create preferences table v3" 2024-09-11T08:09:03.283 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.097622021Z level=info msg="Executing migration" id="Update preferences table charset" 2024-09-11T08:09:03.283 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.100138171Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-09-11T08:09:03.283 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.102860672Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-09-11T08:09:03.283 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.105465877Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-09-11T08:09:03.283 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.108109746Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-09-11T08:09:03.283 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.108933097Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-09-11T08:09:03.283 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.109956986Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-09-11T08:09:03.283 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.110662047Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-09-11T08:09:03.283 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.111712024Z level=info msg="Executing migration" id="create alert table v1" 2024-09-11T08:09:03.284 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.112360815Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-09-11T08:09:03.284 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.113367864Z level=info msg="Executing migration" id="add index alert state" 2024-09-11T08:09:03.284 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.114061949Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-09-11T08:09:03.284 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.115032252Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-09-11T08:09:03.284 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.11581133Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-09-11T08:09:03.284 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.116535123Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-09-11T08:09:03.284 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.11738505Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-09-11T08:09:03.284 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.118777298Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-09-11T08:09:03.284 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.119733868Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-09-11T08:09:03.285 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.120486795Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-09-11T08:09:03.285 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.121304051Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-09-11T08:09:03.285 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.122206482Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-09-11T08:09:03.285 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.123162829Z level=info msg="Executing migration" id="Add column is_default" 2024-09-11T08:09:03.285 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.123932084Z level=info msg="Executing migration" id="Add column frequency" 2024-09-11T08:09:03.285 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.124879191Z level=info msg="Executing migration" id="Add column send_reminder" 2024-09-11T08:09:03.285 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.125649923Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-09-11T08:09:03.286 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.126522335Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-09-11T08:09:03.286 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.127397839Z level=info msg="Executing migration" id="Update alert table charset" 2024-09-11T08:09:03.286 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.128360014Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-09-11T08:09:03.286 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.129084095Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-09-11T08:09:03.286 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.129956757Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-09-11T08:09:03.286 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.130907388Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-09-11T08:09:03.286 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.131452258Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-09-11T08:09:03.286 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.134271834Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-09-11T08:09:03.286 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.137063529Z level=info msg="Executing migration" id="Add for to alert table" 2024-09-11T08:09:03.286 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.139681001Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-09-11T08:09:03.286 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.142286433Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-09-11T08:09:03.286 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.144834082Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-09-11T08:09:03.286 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.147640223Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-09-11T08:09:03.286 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.150328263Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-09-11T08:09:03.287 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.153003386Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-09-11T08:09:03.287 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.15554152Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-09-11T08:09:03.287 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.158258079Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-09-11T08:09:03.287 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.16072462Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-09-11T08:09:03.287 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.16351257Z level=info msg="Executing migration" id="create annotation table v5" 2024-09-11T08:09:03.287 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.166023948Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-09-11T08:09:03.287 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.168645098Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-09-11T08:09:03.287 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.171374933Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-09-11T08:09:03.287 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.174045573Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-09-11T08:09:03.287 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.174838177Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-09-11T08:09:03.287 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.175519197Z level=info msg="Executing migration" id="Update annotation table charset" 2024-09-11T08:09:03.287 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.176310751Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-09-11T08:09:03.287 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.177079104Z level=info msg="Executing migration" id="Drop category_id index" 2024-09-11T08:09:03.287 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.177865853Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-09-11T08:09:03.288 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.17865628Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-09-11T08:09:03.288 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.179535144Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-09-11T08:09:03.288 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.180417072Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-09-11T08:09:03.288 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.181089381Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-09-11T08:09:03.288 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.182640756Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-09-11T08:09:03.288 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.183471176Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-09-11T08:09:03.288 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.184303994Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-09-11T08:09:03.288 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.185067701Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-09-11T08:09:03.288 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.185971401Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-09-11T08:09:03.288 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.186632929Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-09-11T08:09:03.288 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.18742184Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-09-11T08:09:03.288 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.188214887Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-09-11T08:09:03.288 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.189017279Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-09-11T08:09:03.288 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.189855955Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-09-11T08:09:03.289 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.190800915Z level=info msg="Executing migration" id="Add epoch_end column" 2024-09-11T08:09:03.289 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.19157297Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-09-11T08:09:03.289 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.192563248Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-09-11T08:09:03.289 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.193270442Z level=info msg="Executing migration" id="Move region to single row" 2024-09-11T08:09:03.289 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.194172905Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-09-11T08:09:03.289 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.194918256Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-09-11T08:09:03.289 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.195862121Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-09-11T08:09:03.289 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.196599473Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-09-11T08:09:03.289 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.197489845Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-09-11T08:09:03.289 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.198417481Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-09-11T08:09:03.289 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.199186527Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-09-11T08:09:03.289 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.199995946Z level=info msg="Executing migration" id="create test_data table" 2024-09-11T08:09:03.289 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.20040994Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-09-11T08:09:03.290 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.200869833Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-09-11T08:09:03.290 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.201720701Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-09-11T08:09:03.290 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.202562705Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-09-11T08:09:03.290 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.203389399Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-09-11T08:09:03.290 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.206148287Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-09-11T08:09:03.290 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.208805452Z level=info msg="Executing migration" id="create team table" 2024-09-11T08:09:03.290 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.211363628Z level=info msg="Executing migration" id="add index team.org_id" 2024-09-11T08:09:03.290 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.214196762Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-09-11T08:09:03.290 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.217003047Z level=info msg="Executing migration" id="create team member table" 2024-09-11T08:09:03.290 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.219702623Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-09-11T08:09:03.290 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.222318765Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-09-11T08:09:03.290 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.225000324Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-09-11T08:09:03.290 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.227770301Z level=info msg="Executing migration" id="Add column email to team table" 2024-09-11T08:09:03.291 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.230466722Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-09-11T08:09:03.291 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.233350344Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-09-11T08:09:03.291 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.236100792Z level=info msg="Executing migration" id="create dashboard acl table" 2024-09-11T08:09:03.291 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.238846101Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-09-11T08:09:03.291 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.241498249Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-09-11T08:09:03.292 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.244131247Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-09-11T08:09:03.292 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.24458787Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-09-11T08:09:03.292 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.247331904Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-09-11T08:09:03.292 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.249961589Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-09-11T08:09:03.292 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.250404218Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-09-11T08:09:03.292 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.250825853Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-09-11T08:09:03.292 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.251282706Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-09-11T08:09:03.292 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.251608407Z level=info msg="Executing migration" id="create tag table" 2024-09-11T08:09:03.292 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.251996569Z level=info msg="Executing migration" id="add index tag.key_value" 2024-09-11T08:09:03.292 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.252421936Z level=info msg="Executing migration" id="create login attempt table" 2024-09-11T08:09:03.292 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.252819416Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-09-11T08:09:03.292 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.253247042Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-09-11T08:09:03.293 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.253663717Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-09-11T08:09:03.293 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.255529806Z level=info msg="Executing migration" id="create login_attempt v2" 2024-09-11T08:09:03.293 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.255945069Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-09-11T08:09:03.293 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.256365074Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-09-11T08:09:03.293 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.256680247Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-09-11T08:09:03.293 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.257033958Z level=info msg="Executing migration" id="create user auth table" 2024-09-11T08:09:03.293 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.257378236Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-09-11T08:09:03.293 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.257789193Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-09-11T08:09:03.293 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.258072744Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-09-11T08:09:03.293 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.25881947Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-09-11T08:09:03.293 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.259557357Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-09-11T08:09:03.293 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.260296373Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-09-11T08:09:03.293 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.261055173Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-09-11T08:09:03.293 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.261470361Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-09-11T08:09:03.294 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.262214053Z level=info msg="Executing migration" id="create server_lock table" 2024-09-11T08:09:03.294 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.262601846Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-09-11T08:09:03.294 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.263019679Z level=info msg="Executing migration" id="create user auth token table" 2024-09-11T08:09:03.294 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.263435404Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-09-11T08:09:03.294 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.263885092Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-09-11T08:09:03.294 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.26428733Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-09-11T08:09:03.294 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.264700143Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-09-11T08:09:03.294 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.265437229Z level=info msg="Executing migration" id="create cache_data table" 2024-09-11T08:09:03.294 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.265845799Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-09-11T08:09:03.294 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.266255361Z level=info msg="Executing migration" id="create short_url table v1" 2024-09-11T08:09:03.294 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.26667871Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-09-11T08:09:03.294 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.269301042Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-09-11T08:09:03.294 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.269643001Z level=info msg="Executing migration" id="delete alert_definition table" 2024-09-11T08:09:03.294 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.269940731Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-09-11T08:09:03.295 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.27036554Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-09-11T08:09:03.295 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.273252517Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-09-11T08:09:03.295 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.273834583Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-09-11T08:09:03.295 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.274124484Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-09-11T08:09:03.295 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.274501728Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-09-11T08:09:03.295 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.274905167Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-09-11T08:09:03.295 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.275295678Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-09-11T08:09:03.295 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.275709202Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-09-11T08:09:03.295 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.276492692Z level=info msg="Executing migration" id="drop alert_definition table" 2024-09-11T08:09:03.295 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.276916024Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-09-11T08:09:03.295 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.277178321Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-09-11T08:09:03.295 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.277546626Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-09-11T08:09:03.296 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.277963251Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-09-11T08:09:03.296 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.278344609Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-09-11T08:09:03.296 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.278670622Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-09-11T08:09:03.296 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.279127383Z level=info msg="Executing migration" id="create alert_instance table" 2024-09-11T08:09:03.296 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.279578739Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-09-11T08:09:03.296 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.2800555Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-09-11T08:09:03.297 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.280513071Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-09-11T08:09:03.297 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.281373074Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-09-11T08:09:03.297 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.28184318Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-09-11T08:09:03.297 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.282306481Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-09-11T08:09:03.532 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.286730196Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-09-11T08:09:03.532 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.290719692Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-09-11T08:09:03.532 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.29115385Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-09-11T08:09:03.532 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.291537408Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-09-11T08:09:03.532 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.292312996Z level=info msg="Executing migration" id="create alert_rule table" 2024-09-11T08:09:03.532 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.292759578Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-09-11T08:09:03.532 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.295546404Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-09-11T08:09:03.532 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.298172901Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-09-11T08:09:03.532 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.298684847Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-09-11T08:09:03.532 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.299532407Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-09-11T08:09:03.532 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.30037995Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-09-11T08:09:03.533 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.301230723Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-09-11T08:09:03.533 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.302118904Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-09-11T08:09:03.533 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.303008941Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-09-11T08:09:03.533 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.30385098Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-09-11T08:09:03.533 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.304730053Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-09-11T08:09:03.533 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.305577684Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-09-11T08:09:03.533 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.306476459Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-09-11T08:09:03.533 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.307307003Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-09-11T08:09:03.533 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.308154241Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-09-11T08:09:03.533 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.31097454Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-09-11T08:09:03.533 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.313898134Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-09-11T08:09:03.533 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.316700379Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-09-11T08:09:03.533 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.31948418Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-09-11T08:09:03.534 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.32250285Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-09-11T08:09:03.534 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.325433307Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-09-11T08:09:03.534 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.328116771Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-09-11T08:09:03.534 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.331079617Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-09-11T08:09:03.534 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.333658036Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-09-11T08:09:03.534 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.336700121Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-09-11T08:09:03.534 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.339598298Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-09-11T08:09:03.534 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.342414627Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-09-11T08:09:03.534 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.34524654Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-09-11T08:09:03.534 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.347948948Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-09-11T08:09:03.534 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.350890817Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-09-11T08:09:03.534 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.353657807Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-09-11T08:09:03.534 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.35640753Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-09-11T08:09:03.534 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.35924844Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-09-11T08:09:03.535 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.361891692Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-09-11T08:09:03.535 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.364512708Z level=info msg="Executing migration" id="create provenance_type table" 2024-09-11T08:09:03.535 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.367219849Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-09-11T08:09:03.535 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.369931706Z level=info msg="Executing migration" id="create alert_image table" 2024-09-11T08:09:03.535 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.372466434Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-09-11T08:09:03.535 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.375070248Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-09-11T08:09:03.535 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.377882291Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-09-11T08:09:03.535 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.38039307Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-09-11T08:09:03.535 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.383078253Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-09-11T08:09:03.535 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.38318634Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-09-11T08:09:03.535 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.385724235Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-09-11T08:09:03.535 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.388307518Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-09-11T08:09:03.535 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.391042738Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-09-11T08:09:03.536 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.391275172Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-09-11T08:09:03.536 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.391411269Z level=info msg="alerts found to migrate" alerts=0 2024-09-11T08:09:03.536 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.39382662Z level=info msg="Executing migration" id="create library_element table v1" 2024-09-11T08:09:03.536 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.396496858Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-09-11T08:09:03.537 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.399017342Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-09-11T08:09:03.537 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.401727935Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-09-11T08:09:03.537 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.404208829Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-09-11T08:09:03.537 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.405083051Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-09-11T08:09:03.537 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.406055934Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-09-11T08:09:03.537 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.406805589Z level=info msg="Executing migration" id="create data_keys table" 2024-09-11T08:09:03.537 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.407660647Z level=info msg="Executing migration" id="create secrets table" 2024-09-11T08:09:03.537 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.408502861Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-09-11T08:09:03.537 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.412627433Z level=info msg="Executing migration" id="add name column into data_keys" 2024-09-11T08:09:03.537 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.41354638Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-09-11T08:09:03.537 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.414555403Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-09-11T08:09:03.537 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.419237563Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-09-11T08:09:03.538 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.423470508Z level=info msg="Executing migration" id="create kv_store table v1" 2024-09-11T08:09:03.538 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.424346937Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-09-11T08:09:03.538 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.425159187Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-09-11T08:09:03.538 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.425996133Z level=info msg="Executing migration" id="create permission table" 2024-09-11T08:09:03.538 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.426946353Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-09-11T08:09:03.538 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.427834324Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-09-11T08:09:03.538 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.42862638Z level=info msg="Executing migration" id="create role table" 2024-09-11T08:09:03.538 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.4294904Z level=info msg="Executing migration" id="add column display_name" 2024-09-11T08:09:03.538 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.430464716Z level=info msg="Executing migration" id="add column group_name" 2024-09-11T08:09:03.538 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.431385034Z level=info msg="Executing migration" id="add index role.org_id" 2024-09-11T08:09:03.538 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.432245349Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-09-11T08:09:03.539 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.433270738Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-09-11T08:09:03.539 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.433994696Z level=info msg="Executing migration" id="create team role table" 2024-09-11T08:09:03.539 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.435019187Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-09-11T08:09:03.539 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.435744068Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-09-11T08:09:03.539 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.436558979Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-09-11T08:09:03.539 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.437560677Z level=info msg="Executing migration" id="create user role table" 2024-09-11T08:09:03.539 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.438408898Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-09-11T08:09:03.539 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.439173505Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-09-11T08:09:03.539 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.440173386Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-09-11T08:09:03.540 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.440956657Z level=info msg="Executing migration" id="create builtin role table" 2024-09-11T08:09:03.540 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.441842043Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-09-11T08:09:03.540 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.442803665Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-09-11T08:09:03.540 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.443529858Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-09-11T08:09:03.540 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.446564974Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-09-11T08:09:03.540 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.449335419Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-09-11T08:09:03.540 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.451978996Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-09-11T08:09:03.540 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.45472461Z level=info msg="Executing migration" id="add unique index role.uid" 2024-09-11T08:09:03.540 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.457740101Z level=info msg="Executing migration" id="create seed assignment table" 2024-09-11T08:09:03.540 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.460504737Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-09-11T08:09:03.540 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.463333348Z level=info msg="Executing migration" id="add column hidden to role table" 2024-09-11T08:09:03.540 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.466167516Z level=info msg="Executing migration" id="create query_history table v1" 2024-09-11T08:09:03.540 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.468794027Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-09-11T08:09:03.541 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.471653485Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-09-11T08:09:03.541 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.474586929Z level=info msg="Executing migration" id="teams permissions migration" 2024-09-11T08:09:03.541 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.477464416Z level=info msg="Executing migration" id="dashboard permissions" 2024-09-11T08:09:03.541 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.480393042Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-09-11T08:09:03.541 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.483241417Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-09-11T08:09:03.541 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.486025211Z level=info msg="Executing migration" id="alerting notification permissions" 2024-09-11T08:09:03.541 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.488879587Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-09-11T08:09:03.541 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.491769426Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-09-11T08:09:03.541 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.494555862Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-09-11T08:09:03.541 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.496942192Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-09-11T08:09:03.542 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.499590498Z level=info msg="Executing migration" id="create correlation table v1" 2024-09-11T08:09:03.542 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.502273083Z level=info msg="Executing migration" id="add index correlations.uid" 2024-09-11T08:09:03.542 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.505073873Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-09-11T08:09:03.542 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.507893502Z level=info msg="Executing migration" id="add correlation config column" 2024-09-11T08:09:03.542 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.510583846Z level=info msg="Executing migration" id="create entity_events table" 2024-09-11T08:09:03.542 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.513192403Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-09-11T08:09:03.542 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.515907962Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-11T08:09:03.542 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.516031079Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-11T08:09:03.542 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.516344522Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-11T08:09:03.543 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.516422336Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-11T08:09:03.543 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.519028548Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-09-11T08:09:03.543 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.521718549Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-09-11T08:09:03.543 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.524394468Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-09-11T08:09:03.543 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.525001565Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-11T08:09:03.543 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.527778327Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-09-11T08:09:03.543 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.530560236Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-11T08:09:03.543 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.533195271Z level=info msg="Executing migration" id="Drop public config table" 2024-09-11T08:09:03.824 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.533655223Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-09-11T08:09:03.824 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.534526155Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-09-11T08:09:03.825 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.53538356Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-11T08:09:03.825 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.536198926Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-09-11T08:09:03.825 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.536649408Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-09-11T08:09:03.825 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.539648428Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-09-11T08:09:03.825 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.540646581Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-09-11T08:09:03.825 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.54163641Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-09-11T08:09:03.825 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.542585667Z level=info msg="Executing migration" id="add share column" 2024-09-11T08:09:03.825 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.543714793Z level=info msg="Executing migration" id="create default alerting folders" 2024-09-11T08:09:03.825 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.544598711Z level=info msg="Executing migration" id="create file table" 2024-09-11T08:09:03.825 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.54552429Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-09-11T08:09:03.825 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.546003837Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-09-11T08:09:03.825 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.546814169Z level=info msg="Executing migration" id="create file_meta table" 2024-09-11T08:09:03.825 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.547223733Z level=info msg="Executing migration" id="file table idx: path key" 2024-09-11T08:09:03.826 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.548097663Z level=info msg="Executing migration" id="set path collation in file table" 2024-09-11T08:09:03.826 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.548386752Z level=info msg="Executing migration" id="managed permissions migration" 2024-09-11T08:09:03.826 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.548823204Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-09-11T08:09:03.826 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.549223519Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-09-11T08:09:03.826 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.54967687Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-09-11T08:09:03.826 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.550714256Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-09-11T08:09:03.826 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.551533655Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-09-11T08:09:03.826 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.552010674Z level=info msg="Executing migration" id="update group index for alert rules" 2024-09-11T08:09:03.827 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.552818658Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-09-11T08:09:03.827 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.553182117Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-09-11T08:09:03.827 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.553605522Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-09-11T08:09:03.827 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.554638999Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-09-11T08:09:03.827 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.555663847Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-09-11T08:09:03.827 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.556094124Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-09-11T08:09:03.827 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.566375676Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-09-11T08:09:03.827 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.567212774Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-09-11T08:09:03.827 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.570074148Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-09-11T08:09:03.827 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.573580159Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-09-11T08:09:03.827 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.576177284Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-09-11T08:09:03.827 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.578982963Z level=info msg="Executing migration" id="create folder table" 2024-09-11T08:09:03.827 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.581593686Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-09-11T08:09:03.828 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.584204975Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-09-11T08:09:03.828 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.586782685Z level=info msg="Executing migration" id="Update folder title length" 2024-09-11T08:09:03.828 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.589378555Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-09-11T08:09:03.828 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=migrator t=2024-09-11T08:09:03.592123967Z level=info msg="migrations completed" performed=485 skipped=0 duration=739.415101ms 2024-09-11T08:09:03.828 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=sqlstore t=2024-09-11T08:09:03.592578356Z level=info msg="Created default organization" 2024-09-11T08:09:03.828 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=plugin.loader t=2024-09-11T08:09:03.64895237Z level=info msg="Plugin registered" pluginID=input 2024-09-11T08:09:03.828 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=plugin.loader t=2024-09-11T08:09:03.676125074Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-09-11T08:09:03.828 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=plugin.loader t=2024-09-11T08:09:03.676145575Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-09-11T08:09:03.828 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=secrets t=2024-09-11T08:09:03.676178593Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-09-11T08:09:03.828 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=query_data t=2024-09-11T08:09:03.677904999Z level=info msg="Query Service initialization" 2024-09-11T08:09:03.828 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=live.push_http t=2024-09-11T08:09:03.680226987Z level=info msg="Live Push Gateway initialization" 2024-09-11T08:09:04.173 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:03 smithi150 ceph-mon[29276]: 6.1a deep-scrub starts 2024-09-11T08:09:04.173 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:03 smithi150 ceph-mon[29276]: 6.1a deep-scrub ok 2024-09-11T08:09:04.173 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:03 smithi150 ceph-mon[29276]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:09:04.173 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:03 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:04.174 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:03 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:04.174 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:03 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:04.174 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:03 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:04.174 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:03 smithi150 ceph-mon[29276]: Deploying daemon node-exporter.a on smithi150 2024-09-11T08:09:04.174 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:03 smithi150 ceph-mon[29276]: 4.17 deep-scrub starts 2024-09-11T08:09:04.174 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:03 smithi150 ceph-mon[29276]: 4.17 deep-scrub ok 2024-09-11T08:09:04.174 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:03 smithi150 ceph-mon[39612]: 6.1a deep-scrub starts 2024-09-11T08:09:04.174 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:03 smithi150 ceph-mon[39612]: 6.1a deep-scrub ok 2024-09-11T08:09:04.174 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:03 smithi150 ceph-mon[39612]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:09:04.174 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:03 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:04.175 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:03 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:04.175 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:03 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:04.175 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:03 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:04.175 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:03 smithi150 ceph-mon[39612]: Deploying daemon node-exporter.a on smithi150 2024-09-11T08:09:04.175 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:03 smithi150 ceph-mon[39612]: 4.17 deep-scrub starts 2024-09-11T08:09:04.175 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:03 smithi150 ceph-mon[39612]: 4.17 deep-scrub ok 2024-09-11T08:09:04.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-mon[32701]: 6.1a deep-scrub starts 2024-09-11T08:09:04.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-mon[32701]: 6.1a deep-scrub ok 2024-09-11T08:09:04.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-mon[32701]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:09:04.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:04.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:04.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:04.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:04.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-mon[32701]: Deploying daemon node-exporter.a on smithi150 2024-09-11T08:09:04.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-mon[32701]: 4.17 deep-scrub starts 2024-09-11T08:09:04.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:03 smithi181 ceph-mon[32701]: 4.17 deep-scrub ok 2024-09-11T08:09:04.824 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:04 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=infra.usagestats.collector t=2024-09-11T08:09:04.396740099Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-09-11T08:09:04.824 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:04 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=provisioning.datasources t=2024-09-11T08:09:04.397083935Z level=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-09-11T08:09:04.824 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:04 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=provisioning.datasources t=2024-09-11T08:09:04.429622444Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-09-11T08:09:04.824 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:04 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=provisioning.datasources t=2024-09-11T08:09:04.46105578Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-09-11T08:09:04.824 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:04 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=provisioning.alerting t=2024-09-11T08:09:04.482426212Z level=info msg="starting to provision alerting" 2024-09-11T08:09:04.824 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:04 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=provisioning.alerting t=2024-09-11T08:09:04.482453825Z level=info msg="finished to provision alerting" 2024-09-11T08:09:04.824 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:04 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=grafanaStorageLogger t=2024-09-11T08:09:04.482581766Z level=info msg="storage starting" 2024-09-11T08:09:04.825 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:04 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=ngalert.state.manager t=2024-09-11T08:09:04.482703278Z level=info msg="Warming state cache for startup" 2024-09-11T08:09:04.825 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:04 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=http.server t=2024-09-11T08:09:04.48869555Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-09-11T08:09:04.825 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:04 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=ngalert.state.manager t=2024-09-11T08:09:04.493375325Z level=info msg="State cache has been initialized" states=0 duration=10.672169ms 2024-09-11T08:09:04.825 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:04 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=ngalert.multiorg.alertmanager t=2024-09-11T08:09:04.493468089Z level=info msg="starting MultiOrg Alertmanager" 2024-09-11T08:09:04.825 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:04 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=ticker t=2024-09-11T08:09:04.493472902Z level=info msg=starting first_tick=2024-09-11T08:09:10Z 2024-09-11T08:09:04.825 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:04 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=sqlstore.transactions t=2024-09-11T08:09:04.493755064Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-11T08:09:04.825 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:04 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=sqlstore.transactions t=2024-09-11T08:09:04.528730879Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-11T08:09:04.825 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:04 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=sqlstore.transactions t=2024-09-11T08:09:04.569707088Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-11T08:09:04.825 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:09:04 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=sqlstore.transactions t=2024-09-11T08:09:04.58020855Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-09-11T08:09:05.144 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:04 smithi181 ceph-mon[32701]: 3.13 scrub starts 2024-09-11T08:09:05.144 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:04 smithi181 ceph-mon[32701]: 3.13 scrub ok 2024-09-11T08:09:05.144 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:04 smithi181 ceph-mon[32701]: 4.13 scrub starts 2024-09-11T08:09:05.144 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:04 smithi181 ceph-mon[32701]: 4.13 scrub ok 2024-09-11T08:09:05.144 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:04 smithi181 ceph-mon[32701]: 4.1f scrub starts 2024-09-11T08:09:05.144 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:04 smithi181 ceph-mon[32701]: 4.1f scrub ok 2024-09-11T08:09:05.144 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:04 smithi181 ceph-mon[32701]: 6.f scrub starts 2024-09-11T08:09:05.145 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:04 smithi181 ceph-mon[32701]: 6.f scrub ok 2024-09-11T08:09:05.145 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:04 smithi181 ceph-mon[32701]: 5.17 deep-scrub starts 2024-09-11T08:09:05.145 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:04 smithi181 ceph-mon[32701]: 5.17 deep-scrub ok 2024-09-11T08:09:05.183 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:09:05.183 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:04 smithi150 ceph-mon[29276]: 3.13 scrub starts 2024-09-11T08:09:05.183 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:04 smithi150 ceph-mon[29276]: 3.13 scrub ok 2024-09-11T08:09:05.183 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:04 smithi150 ceph-mon[29276]: 4.13 scrub starts 2024-09-11T08:09:05.183 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:04 smithi150 ceph-mon[29276]: 4.13 scrub ok 2024-09-11T08:09:05.184 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:04 smithi150 ceph-mon[29276]: 4.1f scrub starts 2024-09-11T08:09:05.184 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:04 smithi150 ceph-mon[29276]: 4.1f scrub ok 2024-09-11T08:09:05.184 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:04 smithi150 ceph-mon[29276]: 6.f scrub starts 2024-09-11T08:09:05.184 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:04 smithi150 ceph-mon[29276]: 6.f scrub ok 2024-09-11T08:09:05.184 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:04 smithi150 ceph-mon[29276]: 5.17 deep-scrub starts 2024-09-11T08:09:05.184 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:04 smithi150 ceph-mon[29276]: 5.17 deep-scrub ok 2024-09-11T08:09:05.184 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:04 smithi150 ceph-mon[39612]: 3.13 scrub starts 2024-09-11T08:09:05.184 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:04 smithi150 ceph-mon[39612]: 3.13 scrub ok 2024-09-11T08:09:05.184 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:04 smithi150 ceph-mon[39612]: 4.13 scrub starts 2024-09-11T08:09:05.184 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:04 smithi150 ceph-mon[39612]: 4.13 scrub ok 2024-09-11T08:09:05.185 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:04 smithi150 ceph-mon[39612]: 4.1f scrub starts 2024-09-11T08:09:05.185 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:04 smithi150 ceph-mon[39612]: 4.1f scrub ok 2024-09-11T08:09:05.185 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:04 smithi150 ceph-mon[39612]: 6.f scrub starts 2024-09-11T08:09:05.185 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:04 smithi150 ceph-mon[39612]: 6.f scrub ok 2024-09-11T08:09:05.185 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:04 smithi150 ceph-mon[39612]: 5.17 deep-scrub starts 2024-09-11T08:09:05.185 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:04 smithi150 ceph-mon[39612]: 5.17 deep-scrub ok 2024-09-11T08:09:05.190 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-09-11T08:09:05.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:09:05 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:09:05] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:09:05.847 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":43,"stamp":"2024-09-11T08:09:04.692273+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596188,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":939,"num_read_kb":796,"num_write":543,"num_write_kb":1671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":568,"ondisk_log_size":568,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2387348,"kb_used_data":8532,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":747573868,"statfs":{"total":767960285184,"available":765515640832,"internally_reserved":0,"allocated":8736768,"data_stored":4658999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002428"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":44,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.003854+0000","last_change":"2024-09-11T08:08:36.105232+0000","last_active":"2024-09-11T08:08:55.003854+0000","last_peered":"2024-09-11T08:08:55.003854+0000","last_clean":"2024-09-11T08:08:55.003854+0000","last_became_active":"2024-09-11T08:07:30.044976+0000","last_became_peered":"2024-09-11T08:07:30.044976+0000","last_unstale":"2024-09-11T08:08:55.003854+0000","last_undegraded":"2024-09-11T08:08:55.003854+0000","last_fullsized":"2024-09-11T08:08:55.003854+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:36.105175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:36.105175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:56.596929+0000","scrub_duration":0.0016664970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":48,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.473193+0000","last_change":"2024-09-11T08:07:32.048578+0000","last_active":"2024-09-11T08:08:54.473193+0000","last_peered":"2024-09-11T08:08:54.473193+0000","last_clean":"2024-09-11T08:08:54.473193+0000","last_became_active":"2024-09-11T08:07:32.048048+0000","last_became_peered":"2024-09-11T08:07:32.048048+0000","last_unstale":"2024-09-11T08:08:54.473193+0000","last_undegraded":"2024-09-11T08:08:54.473193+0000","last_fullsized":"2024-09-11T08:08:54.473193+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:58.436753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.003838+0000","last_change":"2024-09-11T08:08:38.486126+0000","last_active":"2024-09-11T08:08:55.003838+0000","last_peered":"2024-09-11T08:08:55.003838+0000","last_clean":"2024-09-11T08:08:55.003838+0000","last_became_active":"2024-09-11T08:07:34.058828+0000","last_became_peered":"2024-09-11T08:07:34.058828+0000","last_unstale":"2024-09-11T08:08:55.003838+0000","last_undegraded":"2024-09-11T08:08:55.003838+0000","last_fullsized":"2024-09-11T08:08:55.003838+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:38.486057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:38.486057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:44.914804+0000","scrub_duration":0.002230531,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":38,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:02.470781+0000","last_change":"2024-09-11T08:09:02.470722+0000","last_active":"2024-09-11T08:09:02.470781+0000","last_peered":"2024-09-11T08:09:02.470781+0000","last_clean":"2024-09-11T08:09:02.470781+0000","last_became_active":"2024-09-11T08:07:36.074789+0000","last_became_peered":"2024-09-11T08:07:36.074789+0000","last_unstale":"2024-09-11T08:09:02.470781+0000","last_undegraded":"2024-09-11T08:09:02.470781+0000","last_fullsized":"2024-09-11T08:09:02.470781+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:02.470649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:02.470649+0000","last_clean_scrub_stamp":"2024-09-11T08:09:02.470649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:18.421855+0000","scrub_duration":0.00141182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":28,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.473672+0000","last_change":"2024-09-11T08:07:36.633716+0000","last_active":"2024-09-11T08:08:54.473672+0000","last_peered":"2024-09-11T08:08:54.473672+0000","last_clean":"2024-09-11T08:08:54.473672+0000","last_became_active":"2024-09-11T08:07:36.633423+0000","last_became_peered":"2024-09-11T08:07:36.633423+0000","last_unstale":"2024-09-11T08:08:54.473672+0000","last_undegraded":"2024-09-11T08:08:54.473672+0000","last_fullsized":"2024-09-11T08:08:54.473672+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.474226+0000","last_change":"2024-09-11T08:07:30.045845+0000","last_active":"2024-09-11T08:08:54.474226+0000","last_peered":"2024-09-11T08:08:54.474226+0000","last_clean":"2024-09-11T08:08:54.474226+0000","last_became_active":"2024-09-11T08:07:30.045579+0000","last_became_peered":"2024-09-11T08:07:30.045579+0000","last_unstale":"2024-09-11T08:08:54.474226+0000","last_undegraded":"2024-09-11T08:08:54.474226+0000","last_fullsized":"2024-09-11T08:08:54.474226+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":67,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.408354+0000","last_change":"2024-09-11T08:08:56.408326+0000","last_active":"2024-09-11T08:08:56.408354+0000","last_peered":"2024-09-11T08:08:56.408354+0000","last_clean":"2024-09-11T08:08:56.408354+0000","last_became_active":"2024-09-11T08:07:32.045848+0000","last_became_peered":"2024-09-11T08:07:32.045848+0000","last_unstale":"2024-09-11T08:08:56.408354+0000","last_undegraded":"2024-09-11T08:08:56.408354+0000","last_fullsized":"2024-09-11T08:08:56.408354+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-11T08:08:56.408272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:56.408272+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:09.630406+0000","scrub_duration":0.001771557,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.473106+0000","last_change":"2024-09-11T08:07:34.053483+0000","last_active":"2024-09-11T08:08:54.473106+0000","last_peered":"2024-09-11T08:08:54.473106+0000","last_clean":"2024-09-11T08:08:54.473106+0000","last_became_active":"2024-09-11T08:07:34.052243+0000","last_became_peered":"2024-09-11T08:07:34.052243+0000","last_unstale":"2024-09-11T08:08:54.473106+0000","last_undegraded":"2024-09-11T08:08:54.473106+0000","last_fullsized":"2024-09-11T08:08:54.473106+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:00.987472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.879266+0000","last_change":"2024-09-11T08:08:54.878501+0000","last_active":"2024-09-11T08:08:54.879266+0000","last_peered":"2024-09-11T08:08:54.879266+0000","last_clean":"2024-09-11T08:08:54.879266+0000","last_became_active":"2024-09-11T08:07:30.043950+0000","last_became_peered":"2024-09-11T08:07:30.043950+0000","last_unstale":"2024-09-11T08:08:54.879266+0000","last_undegraded":"2024-09-11T08:08:54.879266+0000","last_fullsized":"2024-09-11T08:08:54.879266+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:54.878409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:54.878409+0000","last_clean_scrub_stamp":"2024-09-11T08:08:54.878409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:07.483340+0000","scrub_duration":0.0015972390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":48,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.474085+0000","last_change":"2024-09-11T08:07:32.044928+0000","last_active":"2024-09-11T08:08:54.474085+0000","last_peered":"2024-09-11T08:08:54.474085+0000","last_clean":"2024-09-11T08:08:54.474085+0000","last_became_active":"2024-09-11T08:07:32.044626+0000","last_became_peered":"2024-09-11T08:07:32.044626+0000","last_unstale":"2024-09-11T08:08:54.474085+0000","last_undegraded":"2024-09-11T08:08:54.474085+0000","last_fullsized":"2024-09-11T08:08:54.474085+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":29,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.879024+0000","last_change":"2024-09-11T08:07:34.061363+0000","last_active":"2024-09-11T08:08:54.879024+0000","last_peered":"2024-09-11T08:08:54.879024+0000","last_clean":"2024-09-11T08:08:54.879024+0000","last_became_active":"2024-09-11T08:07:34.058689+0000","last_became_peered":"2024-09-11T08:07:34.058689+0000","last_unstale":"2024-09-11T08:08:54.879024+0000","last_undegraded":"2024-09-11T08:08:54.879024+0000","last_fullsized":"2024-09-11T08:08:54.879024+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:01.558153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.002887+0000","last_change":"2024-09-11T08:08:40.071203+0000","last_active":"2024-09-11T08:08:55.002887+0000","last_peered":"2024-09-11T08:08:55.002887+0000","last_clean":"2024-09-11T08:08:55.002887+0000","last_became_active":"2024-09-11T08:07:36.076824+0000","last_became_peered":"2024-09-11T08:07:36.076824+0000","last_unstale":"2024-09-11T08:08:55.002887+0000","last_undegraded":"2024-09-11T08:08:55.002887+0000","last_fullsized":"2024-09-11T08:08:55.002887+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:40.071106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:08:40.071106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:54.333357+0000","scrub_duration":0.0019674499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.878844+0000","last_change":"2024-09-11T08:08:36.825857+0000","last_active":"2024-09-11T08:08:54.878844+0000","last_peered":"2024-09-11T08:08:54.878844+0000","last_clean":"2024-09-11T08:08:54.878844+0000","last_became_active":"2024-09-11T08:07:30.043277+0000","last_became_peered":"2024-09-11T08:07:30.043277+0000","last_unstale":"2024-09-11T08:08:54.878844+0000","last_undegraded":"2024-09-11T08:08:54.878844+0000","last_fullsized":"2024-09-11T08:08:54.878844+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:36.825794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:36.825794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:47.493434+0000","scrub_duration":0.0016805920000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":42,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.474398+0000","last_change":"2024-09-11T08:07:32.048969+0000","last_active":"2024-09-11T08:08:54.474398+0000","last_peered":"2024-09-11T08:08:54.474398+0000","last_clean":"2024-09-11T08:08:54.474398+0000","last_became_active":"2024-09-11T08:07:32.048661+0000","last_became_peered":"2024-09-11T08:07:32.048661+0000","last_unstale":"2024-09-11T08:08:54.474398+0000","last_undegraded":"2024-09-11T08:08:54.474398+0000","last_fullsized":"2024-09-11T08:08:54.474398+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.474806+0000","last_change":"2024-09-11T08:07:34.061194+0000","last_active":"2024-09-11T08:08:54.474806+0000","last_peered":"2024-09-11T08:08:54.474806+0000","last_clean":"2024-09-11T08:08:54.474806+0000","last_became_active":"2024-09-11T08:07:34.060871+0000","last_became_peered":"2024-09-11T08:07:34.060871+0000","last_unstale":"2024-09-11T08:08:54.474806+0000","last_undegraded":"2024-09-11T08:08:54.474806+0000","last_fullsized":"2024-09-11T08:08:54.474806+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.878494+0000","last_change":"2024-09-11T08:07:36.077890+0000","last_active":"2024-09-11T08:08:54.878494+0000","last_peered":"2024-09-11T08:08:54.878494+0000","last_clean":"2024-09-11T08:08:54.878494+0000","last_became_active":"2024-09-11T08:07:36.077602+0000","last_became_peered":"2024-09-11T08:07:36.077602+0000","last_unstale":"2024-09-11T08:08:54.878494+0000","last_undegraded":"2024-09-11T08:08:54.878494+0000","last_fullsized":"2024-09-11T08:08:54.878494+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:56.644431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":28,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.473912+0000","last_change":"2024-09-11T08:07:36.633951+0000","last_active":"2024-09-11T08:08:54.473912+0000","last_peered":"2024-09-11T08:08:54.473912+0000","last_clean":"2024-09-11T08:08:54.473912+0000","last_became_active":"2024-09-11T08:07:36.633338+0000","last_became_peered":"2024-09-11T08:07:36.633338+0000","last_unstale":"2024-09-11T08:08:54.473912+0000","last_undegraded":"2024-09-11T08:08:54.473912+0000","last_fullsized":"2024-09-11T08:08:54.473912+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:56.348854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.003819+0000","last_change":"2024-09-11T08:07:30.046004+0000","last_active":"2024-09-11T08:08:55.003819+0000","last_peered":"2024-09-11T08:08:55.003819+0000","last_clean":"2024-09-11T08:08:55.003819+0000","last_became_active":"2024-09-11T08:07:30.045739+0000","last_became_peered":"2024-09-11T08:07:30.045739+0000","last_unstale":"2024-09-11T08:08:55.003819+0000","last_undegraded":"2024-09-11T08:08:55.003819+0000","last_fullsized":"2024-09-11T08:08:55.003819+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"66'15","reported_seq":66,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.975422+0000","last_change":"2024-09-11T08:08:56.975395+0000","last_active":"2024-09-11T08:08:56.975422+0000","last_peered":"2024-09-11T08:08:56.975422+0000","last_clean":"2024-09-11T08:08:56.975422+0000","last_became_active":"2024-09-11T08:07:32.046363+0000","last_became_peered":"2024-09-11T08:07:32.046363+0000","last_unstale":"2024-09-11T08:08:56.975422+0000","last_undegraded":"2024-09-11T08:08:56.975422+0000","last_fullsized":"2024-09-11T08:08:56.975422+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-11T08:08:56.975338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:56.975338+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:19.207433+0000","scrub_duration":0.0017787899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:01.315972+0000","last_change":"2024-09-11T08:09:01.315948+0000","last_active":"2024-09-11T08:09:01.315972+0000","last_peered":"2024-09-11T08:09:01.315972+0000","last_clean":"2024-09-11T08:09:01.315972+0000","last_became_active":"2024-09-11T08:07:34.055410+0000","last_became_peered":"2024-09-11T08:07:34.055410+0000","last_unstale":"2024-09-11T08:09:01.315972+0000","last_undegraded":"2024-09-11T08:09:01.315972+0000","last_fullsized":"2024-09-11T08:09:01.315972+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:01.315897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:09:01.315897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:14.655816+0000","scrub_duration":0.001561163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":28,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.474299+0000","last_change":"2024-09-11T08:07:36.633626+0000","last_active":"2024-09-11T08:08:54.474299+0000","last_peered":"2024-09-11T08:08:54.474299+0000","last_clean":"2024-09-11T08:08:54.474299+0000","last_became_active":"2024-09-11T08:07:36.633258+0000","last_became_peered":"2024-09-11T08:07:36.633258+0000","last_unstale":"2024-09-11T08:08:54.474299+0000","last_undegraded":"2024-09-11T08:08:54.474299+0000","last_fullsized":"2024-09-11T08:08:54.474299+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:01.540476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":44,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.474205+0000","last_change":"2024-09-11T08:08:51.401929+0000","last_active":"2024-09-11T08:08:54.474205+0000","last_peered":"2024-09-11T08:08:54.474205+0000","last_clean":"2024-09-11T08:08:54.474205+0000","last_became_active":"2024-09-11T08:07:30.046284+0000","last_became_peered":"2024-09-11T08:07:30.046284+0000","last_unstale":"2024-09-11T08:08:54.474205+0000","last_undegraded":"2024-09-11T08:08:54.474205+0000","last_fullsized":"2024-09-11T08:08:54.474205+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:51.401864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:51.401864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:18.688155+0000","scrub_duration":0.001460034,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":55,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.473939+0000","last_change":"2024-09-11T08:07:32.047173+0000","last_active":"2024-09-11T08:08:54.473939+0000","last_peered":"2024-09-11T08:08:54.473939+0000","last_clean":"2024-09-11T08:08:54.473939+0000","last_became_active":"2024-09-11T08:07:32.046737+0000","last_became_peered":"2024-09-11T08:07:32.046737+0000","last_unstale":"2024-09-11T08:08:54.473939+0000","last_undegraded":"2024-09-11T08:08:54.473939+0000","last_fullsized":"2024-09-11T08:08:54.473939+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.473728+0000","last_change":"2024-09-11T08:08:40.419900+0000","last_active":"2024-09-11T08:08:54.473728+0000","last_peered":"2024-09-11T08:08:54.473728+0000","last_clean":"2024-09-11T08:08:54.473728+0000","last_became_active":"2024-09-11T08:07:34.056528+0000","last_became_peered":"2024-09-11T08:07:34.056528+0000","last_unstale":"2024-09-11T08:08:54.473728+0000","last_undegraded":"2024-09-11T08:08:54.473728+0000","last_fullsized":"2024-09-11T08:08:54.473728+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:40.419830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:40.419830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:09.111095+0000","scrub_duration":0.0029308149999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":29,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.475006+0000","last_change":"2024-09-11T08:07:36.072658+0000","last_active":"2024-09-11T08:08:54.475006+0000","last_peered":"2024-09-11T08:08:54.475006+0000","last_clean":"2024-09-11T08:08:54.475006+0000","last_became_active":"2024-09-11T08:07:36.072347+0000","last_became_peered":"2024-09-11T08:07:36.072347+0000","last_unstale":"2024-09-11T08:08:54.475006+0000","last_undegraded":"2024-09-11T08:08:54.475006+0000","last_fullsized":"2024-09-11T08:08:54.475006+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:57.979955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.003942+0000","last_change":"2024-09-11T08:07:30.046498+0000","last_active":"2024-09-11T08:08:55.003942+0000","last_peered":"2024-09-11T08:08:55.003942+0000","last_clean":"2024-09-11T08:08:55.003942+0000","last_became_active":"2024-09-11T08:07:30.045932+0000","last_became_peered":"2024-09-11T08:07:30.045932+0000","last_unstale":"2024-09-11T08:08:55.003942+0000","last_undegraded":"2024-09-11T08:08:55.003942+0000","last_fullsized":"2024-09-11T08:08:55.003942+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:56.901581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":47,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.004091+0000","last_change":"2024-09-11T08:07:32.048372+0000","last_active":"2024-09-11T08:08:55.004091+0000","last_peered":"2024-09-11T08:08:55.004091+0000","last_clean":"2024-09-11T08:08:55.004091+0000","last_became_active":"2024-09-11T08:07:32.048022+0000","last_became_peered":"2024-09-11T08:07:32.048022+0000","last_unstale":"2024-09-11T08:08:55.004091+0000","last_undegraded":"2024-09-11T08:08:55.004091+0000","last_fullsized":"2024-09-11T08:08:55.004091+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":68,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.402673+0000","last_change":"2024-09-11T08:08:54.652958+0000","last_active":"2024-09-11T08:08:56.402673+0000","last_peered":"2024-09-11T08:08:56.402673+0000","last_clean":"2024-09-11T08:08:56.402673+0000","last_became_active":"2024-09-11T08:07:34.054014+0000","last_became_peered":"2024-09-11T08:07:34.054014+0000","last_unstale":"2024-09-11T08:08:56.402673+0000","last_undegraded":"2024-09-11T08:08:56.402673+0000","last_fullsized":"2024-09-11T08:08:56.402673+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:08:54.652867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:54.652867+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:09.166003+0000","scrub_duration":0.0022231320000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":67,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.004584+0000","last_change":"2024-09-11T08:08:45.358105+0000","last_active":"2024-09-11T08:08:55.004584+0000","last_peered":"2024-09-11T08:08:55.004584+0000","last_clean":"2024-09-11T08:08:55.004584+0000","last_became_active":"2024-09-11T08:07:32.054317+0000","last_became_peered":"2024-09-11T08:07:32.054317+0000","last_unstale":"2024-09-11T08:08:55.004584+0000","last_undegraded":"2024-09-11T08:08:55.004584+0000","last_fullsized":"2024-09-11T08:08:55.004584+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-11T08:08:45.358041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:45.358041+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:50.858524+0000","scrub_duration":0.0021022879999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.473663+0000","last_change":"2024-09-11T08:07:30.044933+0000","last_active":"2024-09-11T08:08:54.473663+0000","last_peered":"2024-09-11T08:08:54.473663+0000","last_clean":"2024-09-11T08:08:54.473663+0000","last_became_active":"2024-09-11T08:07:30.044624+0000","last_became_peered":"2024-09-11T08:07:30.044624+0000","last_unstale":"2024-09-11T08:08:54.473663+0000","last_undegraded":"2024-09-11T08:08:54.473663+0000","last_fullsized":"2024-09-11T08:08:54.473663+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:58.050628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":56,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.359231+0000","last_change":"2024-09-11T08:07:34.053292+0000","last_active":"2024-09-11T08:08:55.359231+0000","last_peered":"2024-09-11T08:08:55.359231+0000","last_clean":"2024-09-11T08:08:55.359231+0000","last_became_active":"2024-09-11T08:07:34.052981+0000","last_became_peered":"2024-09-11T08:07:34.052981+0000","last_unstale":"2024-09-11T08:08:55.359231+0000","last_undegraded":"2024-09-11T08:08:55.359231+0000","last_fullsized":"2024-09-11T08:08:55.359231+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.878579+0000","last_change":"2024-09-11T08:08:51.893846+0000","last_active":"2024-09-11T08:08:54.878579+0000","last_peered":"2024-09-11T08:08:54.878579+0000","last_clean":"2024-09-11T08:08:54.878579+0000","last_became_active":"2024-09-11T08:07:36.073969+0000","last_became_peered":"2024-09-11T08:07:36.073969+0000","last_unstale":"2024-09-11T08:08:54.878579+0000","last_undegraded":"2024-09-11T08:08:54.878579+0000","last_fullsized":"2024-09-11T08:08:54.878579+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:51.893774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:08:51.893774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:00.876260+0000","scrub_duration":0.001532858,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":74,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.473463+0000","last_change":"2024-09-11T08:08:43.441600+0000","last_active":"2024-09-11T08:08:54.473463+0000","last_peered":"2024-09-11T08:08:54.473463+0000","last_clean":"2024-09-11T08:08:54.473463+0000","last_became_active":"2024-09-11T08:07:32.045332+0000","last_became_peered":"2024-09-11T08:07:32.045332+0000","last_unstale":"2024-09-11T08:08:54.473463+0000","last_undegraded":"2024-09-11T08:08:54.473463+0000","last_fullsized":"2024-09-11T08:08:54.473463+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-11T08:08:43.441544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:43.441544+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:55.649394+0000","scrub_duration":0.002168648,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.474422+0000","last_change":"2024-09-11T08:07:30.047446+0000","last_active":"2024-09-11T08:08:54.474422+0000","last_peered":"2024-09-11T08:08:54.474422+0000","last_clean":"2024-09-11T08:08:54.474422+0000","last_became_active":"2024-09-11T08:07:30.047176+0000","last_became_peered":"2024-09-11T08:07:30.047176+0000","last_unstale":"2024-09-11T08:08:54.474422+0000","last_undegraded":"2024-09-11T08:08:54.474422+0000","last_fullsized":"2024-09-11T08:08:54.474422+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:58.438972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.473138+0000","last_change":"2024-09-11T08:07:34.055816+0000","last_active":"2024-09-11T08:08:54.473138+0000","last_peered":"2024-09-11T08:08:54.473138+0000","last_clean":"2024-09-11T08:08:54.473138+0000","last_became_active":"2024-09-11T08:07:34.055523+0000","last_became_peered":"2024-09-11T08:07:34.055523+0000","last_unstale":"2024-09-11T08:08:54.473138+0000","last_undegraded":"2024-09-11T08:08:54.473138+0000","last_fullsized":"2024-09-11T08:08:54.473138+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:57.216258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":28,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.474007+0000","last_change":"2024-09-11T08:07:36.069535+0000","last_active":"2024-09-11T08:08:54.474007+0000","last_peered":"2024-09-11T08:08:54.474007+0000","last_clean":"2024-09-11T08:08:54.474007+0000","last_became_active":"2024-09-11T08:07:36.069225+0000","last_became_peered":"2024-09-11T08:07:36.069225+0000","last_unstale":"2024-09-11T08:08:54.474007+0000","last_undegraded":"2024-09-11T08:08:54.474007+0000","last_fullsized":"2024-09-11T08:08:54.474007+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:57.222586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":53,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.473388+0000","last_change":"2024-09-11T08:07:32.040334+0000","last_active":"2024-09-11T08:08:54.473388+0000","last_peered":"2024-09-11T08:08:54.473388+0000","last_clean":"2024-09-11T08:08:54.473388+0000","last_became_active":"2024-09-11T08:07:32.040069+0000","last_became_peered":"2024-09-11T08:07:32.040069+0000","last_unstale":"2024-09-11T08:08:54.473388+0000","last_undegraded":"2024-09-11T08:08:54.473388+0000","last_fullsized":"2024-09-11T08:08:54.473388+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.004218+0000","last_change":"2024-09-11T08:07:30.050806+0000","last_active":"2024-09-11T08:08:55.004218+0000","last_peered":"2024-09-11T08:08:55.004218+0000","last_clean":"2024-09-11T08:08:55.004218+0000","last_became_active":"2024-09-11T08:07:30.050484+0000","last_became_peered":"2024-09-11T08:07:30.050484+0000","last_unstale":"2024-09-11T08:08:55.004218+0000","last_undegraded":"2024-09-11T08:08:55.004218+0000","last_fullsized":"2024-09-11T08:08:55.004218+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"82'9","reported_seq":70,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.359676+0000","last_change":"2024-09-11T08:08:53.421634+0000","last_active":"2024-09-11T08:08:55.359676+0000","last_peered":"2024-09-11T08:08:55.359676+0000","last_clean":"2024-09-11T08:08:55.359676+0000","last_became_active":"2024-09-11T08:07:34.056143+0000","last_became_peered":"2024-09-11T08:07:34.056143+0000","last_unstale":"2024-09-11T08:08:55.359676+0000","last_undegraded":"2024-09-11T08:08:55.359676+0000","last_fullsized":"2024-09-11T08:08:55.359676+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-11T08:08:53.421587+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-09-11T08:08:53.421587+0000","last_clean_scrub_stamp":"2024-09-11T08:08:53.421587+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:11.501044+0000","scrub_duration":0.0021116799999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.249252+0000","last_change":"2024-09-11T08:08:55.249190+0000","last_active":"2024-09-11T08:08:55.249252+0000","last_peered":"2024-09-11T08:08:55.249252+0000","last_clean":"2024-09-11T08:08:55.249252+0000","last_became_active":"2024-09-11T08:07:36.633432+0000","last_became_peered":"2024-09-11T08:07:36.633432+0000","last_unstale":"2024-09-11T08:08:55.249252+0000","last_undegraded":"2024-09-11T08:08:55.249252+0000","last_fullsized":"2024-09-11T08:08:55.249252+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:55.249081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:08:55.249081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:00.965734+0000","scrub_duration":0.001581839,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":47,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.003945+0000","last_change":"2024-09-11T08:07:32.054861+0000","last_active":"2024-09-11T08:08:55.003945+0000","last_peered":"2024-09-11T08:08:55.003945+0000","last_clean":"2024-09-11T08:08:55.003945+0000","last_became_active":"2024-09-11T08:07:32.054294+0000","last_became_peered":"2024-09-11T08:07:32.054294+0000","last_unstale":"2024-09-11T08:08:55.003945+0000","last_undegraded":"2024-09-11T08:08:55.003945+0000","last_fullsized":"2024-09-11T08:08:55.003945+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:00.767425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:00.819967+0000","last_change":"2024-09-11T08:09:00.819941+0000","last_active":"2024-09-11T08:09:00.819967+0000","last_peered":"2024-09-11T08:09:00.819967+0000","last_clean":"2024-09-11T08:09:00.819967+0000","last_became_active":"2024-09-11T08:07:30.047497+0000","last_became_peered":"2024-09-11T08:07:30.047497+0000","last_unstale":"2024-09-11T08:09:00.819967+0000","last_undegraded":"2024-09-11T08:09:00.819967+0000","last_fullsized":"2024-09-11T08:09:00.819967+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:00.819884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:09:00.819884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:12.227536+0000","scrub_duration":0.0014673640000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":56,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.402923+0000","last_change":"2024-09-11T08:07:34.060999+0000","last_active":"2024-09-11T08:08:56.402923+0000","last_peered":"2024-09-11T08:08:56.402923+0000","last_clean":"2024-09-11T08:08:56.402923+0000","last_became_active":"2024-09-11T08:07:34.060660+0000","last_became_peered":"2024-09-11T08:07:34.060660+0000","last_unstale":"2024-09-11T08:08:56.402923+0000","last_undegraded":"2024-09-11T08:08:56.402923+0000","last_fullsized":"2024-09-11T08:08:56.402923+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"66'1","reported_seq":29,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.002976+0000","last_change":"2024-09-11T08:07:36.072890+0000","last_active":"2024-09-11T08:08:55.002976+0000","last_peered":"2024-09-11T08:08:55.002976+0000","last_clean":"2024-09-11T08:08:55.002976+0000","last_became_active":"2024-09-11T08:07:36.072593+0000","last_became_peered":"2024-09-11T08:07:36.072593+0000","last_unstale":"2024-09-11T08:08:55.002976+0000","last_undegraded":"2024-09-11T08:08:55.002976+0000","last_fullsized":"2024-09-11T08:08:55.002976+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:58.785592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":78,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.003607+0000","last_change":"2024-09-11T08:08:52.842005+0000","last_active":"2024-09-11T08:08:55.003607+0000","last_peered":"2024-09-11T08:08:55.003607+0000","last_clean":"2024-09-11T08:08:55.003607+0000","last_became_active":"2024-09-11T08:07:32.044376+0000","last_became_peered":"2024-09-11T08:07:32.044376+0000","last_unstale":"2024-09-11T08:08:55.003607+0000","last_undegraded":"2024-09-11T08:08:55.003607+0000","last_fullsized":"2024-09-11T08:08:55.003607+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-11T08:08:52.841903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:52.841903+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:08.737667+0000","scrub_duration":0.0025248150000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":44,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.004495+0000","last_change":"2024-09-11T08:08:49.384423+0000","last_active":"2024-09-11T08:08:55.004495+0000","last_peered":"2024-09-11T08:08:55.004495+0000","last_clean":"2024-09-11T08:08:55.004495+0000","last_became_active":"2024-09-11T08:07:30.045380+0000","last_became_peered":"2024-09-11T08:07:30.045380+0000","last_unstale":"2024-09-11T08:08:55.004495+0000","last_undegraded":"2024-09-11T08:08:55.004495+0000","last_fullsized":"2024-09-11T08:08:55.004495+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:49.384357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:49.384357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:50.840709+0000","scrub_duration":0.0012073310000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":29,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.652166+0000","last_change":"2024-09-11T08:07:34.051286+0000","last_active":"2024-09-11T08:08:54.652166+0000","last_peered":"2024-09-11T08:08:54.652166+0000","last_clean":"2024-09-11T08:08:54.652166+0000","last_became_active":"2024-09-11T08:07:34.051004+0000","last_became_peered":"2024-09-11T08:07:34.051004+0000","last_unstale":"2024-09-11T08:08:54.652166+0000","last_undegraded":"2024-09-11T08:08:54.652166+0000","last_fullsized":"2024-09-11T08:08:54.652166+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:56.890501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":28,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.003753+0000","last_change":"2024-09-11T08:07:36.632119+0000","last_active":"2024-09-11T08:08:55.003753+0000","last_peered":"2024-09-11T08:08:55.003753+0000","last_clean":"2024-09-11T08:08:55.003753+0000","last_became_active":"2024-09-11T08:07:36.631668+0000","last_became_peered":"2024-09-11T08:07:36.631668+0000","last_unstale":"2024-09-11T08:08:55.003753+0000","last_undegraded":"2024-09-11T08:08:55.003753+0000","last_fullsized":"2024-09-11T08:08:55.003753+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:03.103464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"66'28","reported_seq":98,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.011365+0000","last_change":"2024-09-11T08:07:32.053328+0000","last_active":"2024-09-11T08:08:56.011365+0000","last_peered":"2024-09-11T08:08:56.011365+0000","last_clean":"2024-09-11T08:08:56.011365+0000","last_became_active":"2024-09-11T08:07:32.052956+0000","last_became_peered":"2024-09-11T08:07:32.052956+0000","last_unstale":"2024-09-11T08:08:56.011365+0000","last_undegraded":"2024-09-11T08:08:56.011365+0000","last_fullsized":"2024-09-11T08:08:56.011365+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"59'1","reported_seq":53,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.474555+0000","last_change":"2024-09-11T08:08:34.423550+0000","last_active":"2024-09-11T08:08:54.474555+0000","last_peered":"2024-09-11T08:08:54.474555+0000","last_clean":"2024-09-11T08:08:54.474555+0000","last_became_active":"2024-09-11T08:07:30.047513+0000","last_became_peered":"2024-09-11T08:07:30.047513+0000","last_unstale":"2024-09-11T08:08:54.474555+0000","last_undegraded":"2024-09-11T08:08:54.474555+0000","last_fullsized":"2024-09-11T08:08:54.474555+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-11T08:08:34.423474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:34.423474+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:57.535838+0000","scrub_duration":0.0016749149999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":50,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.878783+0000","last_change":"2024-09-11T08:08:37.859401+0000","last_active":"2024-09-11T08:08:54.878783+0000","last_peered":"2024-09-11T08:08:54.878783+0000","last_clean":"2024-09-11T08:08:54.878783+0000","last_became_active":"2024-09-11T08:07:30.043130+0000","last_became_peered":"2024-09-11T08:07:30.043130+0000","last_unstale":"2024-09-11T08:08:54.878783+0000","last_undegraded":"2024-09-11T08:08:54.878783+0000","last_fullsized":"2024-09-11T08:08:54.878783+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-11T08:08:37.859328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:37.859328+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:48.485842+0000","scrub_duration":0.001679816,"objects_trimmed":0,"snaptrim_duration":0.0026120850000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.003582+0000","last_change":"2024-09-11T08:08:46.959561+0000","last_active":"2024-09-11T08:08:55.003582+0000","last_peered":"2024-09-11T08:08:55.003582+0000","last_clean":"2024-09-11T08:08:55.003582+0000","last_became_active":"2024-09-11T08:07:34.057154+0000","last_became_peered":"2024-09-11T08:07:34.057154+0000","last_unstale":"2024-09-11T08:08:55.003582+0000","last_undegraded":"2024-09-11T08:08:55.003582+0000","last_fullsized":"2024-09-11T08:08:55.003582+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:46.959500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:46.959500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:05.161397+0000","scrub_duration":0.001608806,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":28,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.474122+0000","last_change":"2024-09-11T08:07:36.076211+0000","last_active":"2024-09-11T08:08:54.474122+0000","last_peered":"2024-09-11T08:08:54.474122+0000","last_clean":"2024-09-11T08:08:54.474122+0000","last_became_active":"2024-09-11T08:07:36.075690+0000","last_became_peered":"2024-09-11T08:07:36.075690+0000","last_unstale":"2024-09-11T08:08:54.474122+0000","last_undegraded":"2024-09-11T08:08:54.474122+0000","last_fullsized":"2024-09-11T08:08:54.474122+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":69,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.004625+0000","last_change":"2024-09-11T08:08:35.441946+0000","last_active":"2024-09-11T08:08:55.004625+0000","last_peered":"2024-09-11T08:08:55.004625+0000","last_clean":"2024-09-11T08:08:55.004625+0000","last_became_active":"2024-09-11T08:07:32.050673+0000","last_became_peered":"2024-09-11T08:07:32.050673+0000","last_unstale":"2024-09-11T08:08:55.004625+0000","last_undegraded":"2024-09-11T08:08:55.004625+0000","last_fullsized":"2024-09-11T08:08:55.004625+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'13","last_scrub_stamp":"2024-09-11T08:08:35.441894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:35.441894+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:02.914423+0000","scrub_duration":0.001888965,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":44,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.004388+0000","last_change":"2024-09-11T08:08:33.492709+0000","last_active":"2024-09-11T08:08:55.004388+0000","last_peered":"2024-09-11T08:08:55.004388+0000","last_clean":"2024-09-11T08:08:55.004388+0000","last_became_active":"2024-09-11T08:07:30.047014+0000","last_became_peered":"2024-09-11T08:07:30.047014+0000","last_unstale":"2024-09-11T08:08:55.004388+0000","last_undegraded":"2024-09-11T08:08:55.004388+0000","last_fullsized":"2024-09-11T08:08:55.004388+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:33.492652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:33.492652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:54.786224+0000","scrub_duration":0.00156573,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":49,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.003191+0000","last_change":"2024-09-11T08:08:32.909292+0000","last_active":"2024-09-11T08:08:55.003191+0000","last_peered":"2024-09-11T08:08:55.003191+0000","last_clean":"2024-09-11T08:08:55.003191+0000","last_became_active":"2024-09-11T08:07:30.045009+0000","last_became_peered":"2024-09-11T08:07:30.045009+0000","last_unstale":"2024-09-11T08:08:55.003191+0000","last_undegraded":"2024-09-11T08:08:55.003191+0000","last_fullsized":"2024-09-11T08:08:55.003191+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-11T08:08:32.909183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:32.909183+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:02.458449+0000","scrub_duration":0.001625769,"objects_trimmed":0,"snaptrim_duration":0.001859533,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:58.000032+0000","last_change":"2024-09-11T08:08:58.000006+0000","last_active":"2024-09-11T08:08:58.000032+0000","last_peered":"2024-09-11T08:08:58.000032+0000","last_clean":"2024-09-11T08:08:58.000032+0000","last_became_active":"2024-09-11T08:07:34.063906+0000","last_became_peered":"2024-09-11T08:07:34.063906+0000","last_unstale":"2024-09-11T08:08:58.000032+0000","last_undegraded":"2024-09-11T08:08:58.000032+0000","last_fullsized":"2024-09-11T08:08:58.000032+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:57.999952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:57.999952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:01.053836+0000","scrub_duration":0.0014942130000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.473882+0000","last_change":"2024-09-11T08:08:46.366578+0000","last_active":"2024-09-11T08:08:54.473882+0000","last_peered":"2024-09-11T08:08:54.473882+0000","last_clean":"2024-09-11T08:08:54.473882+0000","last_became_active":"2024-09-11T08:07:36.631480+0000","last_became_peered":"2024-09-11T08:07:36.631480+0000","last_unstale":"2024-09-11T08:08:54.473882+0000","last_undegraded":"2024-09-11T08:08:54.473882+0000","last_fullsized":"2024-09-11T08:08:54.473882+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:46.366508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:08:46.366508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:52.248237+0000","scrub_duration":0.0015317200000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":60,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.004143+0000","last_change":"2024-09-11T08:08:37.115649+0000","last_active":"2024-09-11T08:08:55.004143+0000","last_peered":"2024-09-11T08:08:55.004143+0000","last_clean":"2024-09-11T08:08:55.004143+0000","last_became_active":"2024-09-11T08:07:32.055407+0000","last_became_peered":"2024-09-11T08:07:32.055407+0000","last_unstale":"2024-09-11T08:08:55.004143+0000","last_undegraded":"2024-09-11T08:08:55.004143+0000","last_fullsized":"2024-09-11T08:08:55.004143+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-11T08:08:37.115582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:37.115582+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:57.869478+0000","scrub_duration":0.0020267729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"59'1","reported_seq":51,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.003231+0000","last_change":"2024-09-11T08:08:41.094879+0000","last_active":"2024-09-11T08:08:55.003231+0000","last_peered":"2024-09-11T08:08:55.003231+0000","last_clean":"2024-09-11T08:08:55.003231+0000","last_became_active":"2024-09-11T08:07:30.050504+0000","last_became_peered":"2024-09-11T08:07:30.050504+0000","last_unstale":"2024-09-11T08:08:55.003231+0000","last_undegraded":"2024-09-11T08:08:55.003231+0000","last_fullsized":"2024-09-11T08:08:55.003231+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-11T08:08:41.094821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:41.094821+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:00.561470+0000","scrub_duration":0.0017674310000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":138,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:01.438288+0000","last_change":"2024-09-11T08:09:01.175592+0000","last_active":"2024-09-11T08:09:01.438288+0000","last_peered":"2024-09-11T08:09:01.438288+0000","last_clean":"2024-09-11T08:09:01.438288+0000","last_became_active":"2024-09-11T08:07:30.045051+0000","last_became_peered":"2024-09-11T08:07:30.045051+0000","last_unstale":"2024-09-11T08:09:01.438288+0000","last_undegraded":"2024-09-11T08:09:01.438288+0000","last_fullsized":"2024-09-11T08:09:01.438288+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-11T08:09:01.175530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:09:01.175530+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:30.186314+0000","scrub_duration":0.001752651,"objects_trimmed":0,"snaptrim_duration":0.000318821,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":86,"num_read_kb":81,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":29,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.879606+0000","last_change":"2024-09-11T08:07:34.059262+0000","last_active":"2024-09-11T08:08:54.879606+0000","last_peered":"2024-09-11T08:08:54.879606+0000","last_clean":"2024-09-11T08:08:54.879606+0000","last_became_active":"2024-09-11T08:07:34.058962+0000","last_became_peered":"2024-09-11T08:07:34.058962+0000","last_unstale":"2024-09-11T08:08:54.879606+0000","last_undegraded":"2024-09-11T08:08:54.879606+0000","last_fullsized":"2024-09-11T08:08:54.879606+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:57.093439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.004134+0000","last_change":"2024-09-11T08:08:50.346126+0000","last_active":"2024-09-11T08:08:55.004134+0000","last_peered":"2024-09-11T08:08:55.004134+0000","last_clean":"2024-09-11T08:08:55.004134+0000","last_became_active":"2024-09-11T08:07:36.077597+0000","last_became_peered":"2024-09-11T08:07:36.077597+0000","last_unstale":"2024-09-11T08:08:55.004134+0000","last_undegraded":"2024-09-11T08:08:55.004134+0000","last_fullsized":"2024-09-11T08:08:55.004134+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:50.346069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:08:50.346069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:15.900195+0000","scrub_duration":0.001294906,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"84'21","reported_seq":70,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.705399+0000","last_change":"2024-09-11T08:08:56.705362+0000","last_active":"2024-09-11T08:08:56.705399+0000","last_peered":"2024-09-11T08:08:56.705399+0000","last_clean":"2024-09-11T08:08:56.705399+0000","last_became_active":"2024-09-11T08:07:32.042339+0000","last_became_peered":"2024-09-11T08:07:32.042339+0000","last_unstale":"2024-09-11T08:08:56.705399+0000","last_undegraded":"2024-09-11T08:08:56.705399+0000","last_fullsized":"2024-09-11T08:08:56.705399+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'21","last_scrub_stamp":"2024-09-11T08:08:56.705306+0000","last_deep_scrub":"84'21","last_deep_scrub_stamp":"2024-09-11T08:08:56.705306+0000","last_clean_scrub_stamp":"2024-09-11T08:08:56.705306+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:14.758337+0000","scrub_duration":0.0050944290000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.473685+0000","last_change":"2024-09-11T08:07:30.047535+0000","last_active":"2024-09-11T08:08:54.473685+0000","last_peered":"2024-09-11T08:08:54.473685+0000","last_clean":"2024-09-11T08:08:54.473685+0000","last_became_active":"2024-09-11T08:07:30.047246+0000","last_became_peered":"2024-09-11T08:07:30.047246+0000","last_unstale":"2024-09-11T08:08:54.473685+0000","last_undegraded":"2024-09-11T08:08:54.473685+0000","last_fullsized":"2024-09-11T08:08:54.473685+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":80,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.473537+0000","last_change":"2024-09-11T08:07:44.054743+0000","last_active":"2024-09-11T08:08:54.473537+0000","last_peered":"2024-09-11T08:08:54.473537+0000","last_clean":"2024-09-11T08:08:54.473537+0000","last_became_active":"2024-09-11T08:07:19.726378+0000","last_became_peered":"2024-09-11T08:07:19.726378+0000","last_unstale":"2024-09-11T08:08:54.473537+0000","last_undegraded":"2024-09-11T08:08:54.473537+0000","last_fullsized":"2024-09-11T08:08:54.473537+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'72","last_scrub_stamp":"2024-09-11T08:07:44.054686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:05:21.231856+0000","last_clean_scrub_stamp":"2024-09-11T08:07:44.054686+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:58.084790+0000","scrub_duration":0.001872931,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.475088+0000","last_change":"2024-09-11T08:08:51.289948+0000","last_active":"2024-09-11T08:08:54.475088+0000","last_peered":"2024-09-11T08:08:54.475088+0000","last_clean":"2024-09-11T08:08:54.475088+0000","last_became_active":"2024-09-11T08:07:34.058414+0000","last_became_peered":"2024-09-11T08:07:34.058414+0000","last_unstale":"2024-09-11T08:08:54.475088+0000","last_undegraded":"2024-09-11T08:08:54.475088+0000","last_fullsized":"2024-09-11T08:08:54.475088+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:51.289872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:51.289872+0000","last_clean_scrub_stamp":"2024-09-11T08:08:51.289872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:08.912691+0000","scrub_duration":0.001988812,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":37,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:01.777573+0000","last_change":"2024-09-11T08:09:01.777522+0000","last_active":"2024-09-11T08:09:01.777573+0000","last_peered":"2024-09-11T08:09:01.777573+0000","last_clean":"2024-09-11T08:09:01.777573+0000","last_became_active":"2024-09-11T08:07:36.072508+0000","last_became_peered":"2024-09-11T08:07:36.072508+0000","last_unstale":"2024-09-11T08:09:01.777573+0000","last_undegraded":"2024-09-11T08:09:01.777573+0000","last_fullsized":"2024-09-11T08:09:01.777573+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:01.777429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:01.777429+0000","last_clean_scrub_stamp":"2024-09-11T08:09:01.777429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:07.815841+0000","scrub_duration":0.001444961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"66'11","reported_seq":61,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.474810+0000","last_change":"2024-09-11T08:08:35.418440+0000","last_active":"2024-09-11T08:08:54.474810+0000","last_peered":"2024-09-11T08:08:54.474810+0000","last_clean":"2024-09-11T08:08:54.474810+0000","last_became_active":"2024-09-11T08:07:32.047971+0000","last_became_peered":"2024-09-11T08:07:32.047971+0000","last_unstale":"2024-09-11T08:08:54.474810+0000","last_undegraded":"2024-09-11T08:08:54.474810+0000","last_fullsized":"2024-09-11T08:08:54.474810+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-11T08:08:35.418378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:35.418378+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:38.165860+0000","scrub_duration":0.001791207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":44,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.473512+0000","last_change":"2024-09-11T08:08:36.417975+0000","last_active":"2024-09-11T08:08:54.473512+0000","last_peered":"2024-09-11T08:08:54.473512+0000","last_clean":"2024-09-11T08:08:54.473512+0000","last_became_active":"2024-09-11T08:07:30.039884+0000","last_became_peered":"2024-09-11T08:07:30.039884+0000","last_unstale":"2024-09-11T08:08:54.473512+0000","last_undegraded":"2024-09-11T08:08:54.473512+0000","last_fullsized":"2024-09-11T08:08:54.473512+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:36.417911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:36.417911+0000","last_clean_scrub_stamp":"2024-09-11T08:08:36.417911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:59.734565+0000","scrub_duration":0.0014387580000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:58.845172+0000","last_change":"2024-09-11T08:08:58.845142+0000","last_active":"2024-09-11T08:08:58.845172+0000","last_peered":"2024-09-11T08:08:58.845172+0000","last_clean":"2024-09-11T08:08:58.845172+0000","last_became_active":"2024-09-11T08:07:34.053347+0000","last_became_peered":"2024-09-11T08:07:34.053347+0000","last_unstale":"2024-09-11T08:08:58.845172+0000","last_undegraded":"2024-09-11T08:08:58.845172+0000","last_fullsized":"2024-09-11T08:08:58.845172+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:58.845085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:58.845085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:08.089077+0000","scrub_duration":0.001233404,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":28,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.473751+0000","last_change":"2024-09-11T08:07:36.632329+0000","last_active":"2024-09-11T08:08:54.473751+0000","last_peered":"2024-09-11T08:08:54.473751+0000","last_clean":"2024-09-11T08:08:54.473751+0000","last_became_active":"2024-09-11T08:07:36.631687+0000","last_became_peered":"2024-09-11T08:07:36.631687+0000","last_unstale":"2024-09-11T08:08:54.473751+0000","last_undegraded":"2024-09-11T08:08:54.473751+0000","last_fullsized":"2024-09-11T08:08:54.473751+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"77'25","reported_seq":78,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.474131+0000","last_change":"2024-09-11T08:08:32.458272+0000","last_active":"2024-09-11T08:08:54.474131+0000","last_peered":"2024-09-11T08:08:54.474131+0000","last_clean":"2024-09-11T08:08:54.474131+0000","last_became_active":"2024-09-11T08:07:32.050815+0000","last_became_peered":"2024-09-11T08:07:32.050815+0000","last_unstale":"2024-09-11T08:08:54.474131+0000","last_undegraded":"2024-09-11T08:08:54.474131+0000","last_fullsized":"2024-09-11T08:08:54.474131+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'25","last_scrub_stamp":"2024-09-11T08:08:32.458210+0000","last_deep_scrub":"77'25","last_deep_scrub_stamp":"2024-09-11T08:08:32.458210+0000","last_clean_scrub_stamp":"2024-09-11T08:08:32.458210+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:00.578835+0000","scrub_duration":0.0060523319999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.652343+0000","last_change":"2024-09-11T08:07:30.042789+0000","last_active":"2024-09-11T08:08:54.652343+0000","last_peered":"2024-09-11T08:08:54.652343+0000","last_clean":"2024-09-11T08:08:54.652343+0000","last_became_active":"2024-09-11T08:07:30.042555+0000","last_became_peered":"2024-09-11T08:07:30.042555+0000","last_unstale":"2024-09-11T08:08:54.652343+0000","last_undegraded":"2024-09-11T08:08:54.652343+0000","last_fullsized":"2024-09-11T08:08:54.652343+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:57.728934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.004205+0000","last_change":"2024-09-11T08:08:41.437100+0000","last_active":"2024-09-11T08:08:55.004205+0000","last_peered":"2024-09-11T08:08:55.004205+0000","last_clean":"2024-09-11T08:08:55.004205+0000","last_became_active":"2024-09-11T08:07:34.054961+0000","last_became_peered":"2024-09-11T08:07:34.054961+0000","last_unstale":"2024-09-11T08:08:55.004205+0000","last_undegraded":"2024-09-11T08:08:55.004205+0000","last_fullsized":"2024-09-11T08:08:55.004205+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:41.437045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:41.437045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:10.512019+0000","scrub_duration":0.0029926649999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.003327+0000","last_change":"2024-09-11T08:07:36.071503+0000","last_active":"2024-09-11T08:08:55.003327+0000","last_peered":"2024-09-11T08:08:55.003327+0000","last_clean":"2024-09-11T08:08:55.003327+0000","last_became_active":"2024-09-11T08:07:36.071194+0000","last_became_peered":"2024-09-11T08:07:36.071194+0000","last_unstale":"2024-09-11T08:08:55.003327+0000","last_undegraded":"2024-09-11T08:08:55.003327+0000","last_fullsized":"2024-09-11T08:08:55.003327+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:58.811284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":57,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.473809+0000","last_change":"2024-09-11T08:08:42.447971+0000","last_active":"2024-09-11T08:08:54.473809+0000","last_peered":"2024-09-11T08:08:54.473809+0000","last_clean":"2024-09-11T08:08:54.473809+0000","last_became_active":"2024-09-11T08:07:32.043314+0000","last_became_peered":"2024-09-11T08:07:32.043314+0000","last_unstale":"2024-09-11T08:08:54.473809+0000","last_undegraded":"2024-09-11T08:08:54.473809+0000","last_fullsized":"2024-09-11T08:08:54.473809+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-11T08:08:42.447892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:42.447892+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:03.778832+0000","scrub_duration":0.0039015859999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":44,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.473922+0000","last_change":"2024-09-11T08:08:30.327580+0000","last_active":"2024-09-11T08:08:54.473922+0000","last_peered":"2024-09-11T08:08:54.473922+0000","last_clean":"2024-09-11T08:08:54.473922+0000","last_became_active":"2024-09-11T08:07:30.046684+0000","last_became_peered":"2024-09-11T08:07:30.046684+0000","last_unstale":"2024-09-11T08:08:54.473922+0000","last_undegraded":"2024-09-11T08:08:54.473922+0000","last_fullsized":"2024-09-11T08:08:54.473922+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:30.327525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:30.327525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:57.278810+0000","scrub_duration":0.001475589,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"66'8","reported_seq":65,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.359346+0000","last_change":"2024-09-11T08:08:49.026859+0000","last_active":"2024-09-11T08:08:55.359346+0000","last_peered":"2024-09-11T08:08:55.359346+0000","last_clean":"2024-09-11T08:08:55.359346+0000","last_became_active":"2024-09-11T08:07:34.062469+0000","last_became_peered":"2024-09-11T08:07:34.062469+0000","last_unstale":"2024-09-11T08:08:55.359346+0000","last_undegraded":"2024-09-11T08:08:55.359346+0000","last_fullsized":"2024-09-11T08:08:55.359346+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:08:49.026755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:49.026755+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:59.460486+0000","scrub_duration":0.0018077970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":28,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.473818+0000","last_change":"2024-09-11T08:07:36.074961+0000","last_active":"2024-09-11T08:08:54.473818+0000","last_peered":"2024-09-11T08:08:54.473818+0000","last_clean":"2024-09-11T08:08:54.473818+0000","last_became_active":"2024-09-11T08:07:36.074684+0000","last_became_peered":"2024-09-11T08:07:36.074684+0000","last_unstale":"2024-09-11T08:08:54.473818+0000","last_undegraded":"2024-09-11T08:08:54.473818+0000","last_fullsized":"2024-09-11T08:08:54.473818+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":48,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.003537+0000","last_change":"2024-09-11T08:07:32.055237+0000","last_active":"2024-09-11T08:08:55.003537+0000","last_peered":"2024-09-11T08:08:55.003537+0000","last_clean":"2024-09-11T08:08:55.003537+0000","last_became_active":"2024-09-11T08:07:32.054885+0000","last_became_peered":"2024-09-11T08:07:32.054885+0000","last_unstale":"2024-09-11T08:08:55.003537+0000","last_undegraded":"2024-09-11T08:08:55.003537+0000","last_fullsized":"2024-09-11T08:08:55.003537+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.878942+0000","last_change":"2024-09-11T08:08:34.828231+0000","last_active":"2024-09-11T08:08:54.878942+0000","last_peered":"2024-09-11T08:08:54.878942+0000","last_clean":"2024-09-11T08:08:54.878942+0000","last_became_active":"2024-09-11T08:07:30.045613+0000","last_became_peered":"2024-09-11T08:07:30.045613+0000","last_unstale":"2024-09-11T08:08:54.878942+0000","last_undegraded":"2024-09-11T08:08:54.878942+0000","last_fullsized":"2024-09-11T08:08:54.878942+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:34.828129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:34.828129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:58.397313+0000","scrub_duration":0.0013782530000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":39,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.003326+0000","last_change":"2024-09-11T08:08:37.871017+0000","last_active":"2024-09-11T08:08:55.003326+0000","last_peered":"2024-09-11T08:08:55.003326+0000","last_clean":"2024-09-11T08:08:55.003326+0000","last_became_active":"2024-09-11T08:07:34.056606+0000","last_became_peered":"2024-09-11T08:07:34.056606+0000","last_unstale":"2024-09-11T08:08:55.003326+0000","last_undegraded":"2024-09-11T08:08:55.003326+0000","last_fullsized":"2024-09-11T08:08:55.003326+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:37.870955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:37.870955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:58.505876+0000","scrub_duration":0.0016274460000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":28,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.003160+0000","last_change":"2024-09-11T08:07:36.072775+0000","last_active":"2024-09-11T08:08:55.003160+0000","last_peered":"2024-09-11T08:08:55.003160+0000","last_clean":"2024-09-11T08:08:55.003160+0000","last_became_active":"2024-09-11T08:07:36.072452+0000","last_became_peered":"2024-09-11T08:07:36.072452+0000","last_unstale":"2024-09-11T08:08:55.003160+0000","last_undegraded":"2024-09-11T08:08:55.003160+0000","last_fullsized":"2024-09-11T08:08:55.003160+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":72,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.652425+0000","last_change":"2024-09-11T08:08:53.616543+0000","last_active":"2024-09-11T08:08:54.652425+0000","last_peered":"2024-09-11T08:08:54.652425+0000","last_clean":"2024-09-11T08:08:54.652425+0000","last_became_active":"2024-09-11T08:07:32.050880+0000","last_became_peered":"2024-09-11T08:07:32.050880+0000","last_unstale":"2024-09-11T08:08:54.652425+0000","last_undegraded":"2024-09-11T08:08:54.652425+0000","last_fullsized":"2024-09-11T08:08:54.652425+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-11T08:08:53.616450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:53.616450+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:18.217121+0000","scrub_duration":0.0024100559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":44,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.474182+0000","last_change":"2024-09-11T08:08:44.385440+0000","last_active":"2024-09-11T08:08:54.474182+0000","last_peered":"2024-09-11T08:08:54.474182+0000","last_clean":"2024-09-11T08:08:54.474182+0000","last_became_active":"2024-09-11T08:07:30.043294+0000","last_became_peered":"2024-09-11T08:07:30.043294+0000","last_unstale":"2024-09-11T08:08:54.474182+0000","last_undegraded":"2024-09-11T08:08:54.474182+0000","last_fullsized":"2024-09-11T08:08:54.474182+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:44.385372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:44.385372+0000","last_clean_scrub_stamp":"2024-09-11T08:08:44.385372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:46.537624+0000","scrub_duration":0.001441007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":39,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.003529+0000","last_change":"2024-09-11T08:08:51.921479+0000","last_active":"2024-09-11T08:08:55.003529+0000","last_peered":"2024-09-11T08:08:55.003529+0000","last_clean":"2024-09-11T08:08:55.003529+0000","last_became_active":"2024-09-11T08:07:34.063589+0000","last_became_peered":"2024-09-11T08:07:34.063589+0000","last_unstale":"2024-09-11T08:08:55.003529+0000","last_undegraded":"2024-09-11T08:08:55.003529+0000","last_fullsized":"2024-09-11T08:08:55.003529+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:51.921389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:51.921389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:01.266430+0000","scrub_duration":0.001807091,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.473452+0000","last_change":"2024-09-11T08:08:36.367954+0000","last_active":"2024-09-11T08:08:54.473452+0000","last_peered":"2024-09-11T08:08:54.473452+0000","last_clean":"2024-09-11T08:08:54.473452+0000","last_became_active":"2024-09-11T08:07:36.077523+0000","last_became_peered":"2024-09-11T08:07:36.077523+0000","last_unstale":"2024-09-11T08:08:54.473452+0000","last_undegraded":"2024-09-11T08:08:54.473452+0000","last_fullsized":"2024-09-11T08:08:54.473452+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:36.367890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:08:36.367890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:40.081903+0000","scrub_duration":0.0015112560000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":65,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:59.494084+0000","last_change":"2024-09-11T08:08:59.494039+0000","last_active":"2024-09-11T08:08:59.494084+0000","last_peered":"2024-09-11T08:08:59.494084+0000","last_clean":"2024-09-11T08:08:59.494084+0000","last_became_active":"2024-09-11T08:07:32.048164+0000","last_became_peered":"2024-09-11T08:07:32.048164+0000","last_unstale":"2024-09-11T08:08:59.494084+0000","last_undegraded":"2024-09-11T08:08:59.494084+0000","last_fullsized":"2024-09-11T08:08:59.494084+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-11T08:08:59.493975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:59.493975+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:17.528763+0000","scrub_duration":0.0019438280000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":44,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.474638+0000","last_change":"2024-09-11T08:08:53.247349+0000","last_active":"2024-09-11T08:08:54.474638+0000","last_peered":"2024-09-11T08:08:54.474638+0000","last_clean":"2024-09-11T08:08:54.474638+0000","last_became_active":"2024-09-11T08:07:30.041219+0000","last_became_peered":"2024-09-11T08:07:30.041219+0000","last_unstale":"2024-09-11T08:08:54.474638+0000","last_undegraded":"2024-09-11T08:08:54.474638+0000","last_fullsized":"2024-09-11T08:08:54.474638+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:53.247275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:53.247275+0000","last_clean_scrub_stamp":"2024-09-11T08:08:53.247275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:00.288102+0000","scrub_duration":0.0012568169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":39,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.003005+0000","last_change":"2024-09-11T08:08:46.970593+0000","last_active":"2024-09-11T08:08:55.003005+0000","last_peered":"2024-09-11T08:08:55.003005+0000","last_clean":"2024-09-11T08:08:55.003005+0000","last_became_active":"2024-09-11T08:07:34.057479+0000","last_became_peered":"2024-09-11T08:07:34.057479+0000","last_unstale":"2024-09-11T08:08:55.003005+0000","last_undegraded":"2024-09-11T08:08:55.003005+0000","last_fullsized":"2024-09-11T08:08:55.003005+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:46.970539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:46.970539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:09.551366+0000","scrub_duration":0.001167143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":38,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:00.395391+0000","last_change":"2024-09-11T08:09:00.395353+0000","last_active":"2024-09-11T08:09:00.395391+0000","last_peered":"2024-09-11T08:09:00.395391+0000","last_clean":"2024-09-11T08:09:00.395391+0000","last_became_active":"2024-09-11T08:07:36.078075+0000","last_became_peered":"2024-09-11T08:07:36.078075+0000","last_unstale":"2024-09-11T08:09:00.395391+0000","last_undegraded":"2024-09-11T08:09:00.395391+0000","last_fullsized":"2024-09-11T08:09:00.395391+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:00.395264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:09:00.395264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:07.333876+0000","scrub_duration":0.001470147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":56,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.878297+0000","last_change":"2024-09-11T08:07:32.039555+0000","last_active":"2024-09-11T08:08:54.878297+0000","last_peered":"2024-09-11T08:08:54.878297+0000","last_clean":"2024-09-11T08:08:54.878297+0000","last_became_active":"2024-09-11T08:07:32.039275+0000","last_became_peered":"2024-09-11T08:07:32.039275+0000","last_unstale":"2024-09-11T08:08:54.878297+0000","last_undegraded":"2024-09-11T08:08:54.878297+0000","last_fullsized":"2024-09-11T08:08:54.878297+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:00.903547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":51,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.474935+0000","last_change":"2024-09-11T08:07:30.046228+0000","last_active":"2024-09-11T08:08:54.474935+0000","last_peered":"2024-09-11T08:08:54.474935+0000","last_clean":"2024-09-11T08:08:54.474935+0000","last_became_active":"2024-09-11T08:07:30.045967+0000","last_became_peered":"2024-09-11T08:07:30.045967+0000","last_unstale":"2024-09-11T08:08:54.474935+0000","last_undegraded":"2024-09-11T08:08:54.474935+0000","last_fullsized":"2024-09-11T08:08:54.474935+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:58.653748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"66'8","reported_seq":66,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:59.171003+0000","last_change":"2024-09-11T08:08:59.170974+0000","last_active":"2024-09-11T08:08:59.171003+0000","last_peered":"2024-09-11T08:08:59.171003+0000","last_clean":"2024-09-11T08:08:59.171003+0000","last_became_active":"2024-09-11T08:07:34.058015+0000","last_became_peered":"2024-09-11T08:07:34.058015+0000","last_unstale":"2024-09-11T08:08:59.171003+0000","last_undegraded":"2024-09-11T08:08:59.171003+0000","last_fullsized":"2024-09-11T08:08:59.171003+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:08:59.170918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:59.170918+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:12.511014+0000","scrub_duration":0.001468128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.878577+0000","last_change":"2024-09-11T08:07:36.632018+0000","last_active":"2024-09-11T08:08:54.878577+0000","last_peered":"2024-09-11T08:08:54.878577+0000","last_clean":"2024-09-11T08:08:54.878577+0000","last_became_active":"2024-09-11T08:07:36.631755+0000","last_became_peered":"2024-09-11T08:07:36.631755+0000","last_unstale":"2024-09-11T08:08:54.878577+0000","last_undegraded":"2024-09-11T08:08:54.878577+0000","last_fullsized":"2024-09-11T08:08:54.878577+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:58.191062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.652205+0000","last_change":"2024-09-11T08:07:30.046145+0000","last_active":"2024-09-11T08:08:54.652205+0000","last_peered":"2024-09-11T08:08:54.652205+0000","last_clean":"2024-09-11T08:08:54.652205+0000","last_became_active":"2024-09-11T08:07:30.045910+0000","last_became_peered":"2024-09-11T08:07:30.045910+0000","last_unstale":"2024-09-11T08:08:54.652205+0000","last_undegraded":"2024-09-11T08:08:54.652205+0000","last_fullsized":"2024-09-11T08:08:54.652205+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:56.568613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":41,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.474384+0000","last_change":"2024-09-11T08:07:32.047333+0000","last_active":"2024-09-11T08:08:54.474384+0000","last_peered":"2024-09-11T08:08:54.474384+0000","last_clean":"2024-09-11T08:08:54.474384+0000","last_became_active":"2024-09-11T08:07:32.046891+0000","last_became_peered":"2024-09-11T08:07:32.046891+0000","last_unstale":"2024-09-11T08:08:54.474384+0000","last_undegraded":"2024-09-11T08:08:54.474384+0000","last_fullsized":"2024-09-11T08:08:54.474384+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":29,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.879323+0000","last_change":"2024-09-11T08:07:34.058334+0000","last_active":"2024-09-11T08:08:54.879323+0000","last_peered":"2024-09-11T08:08:54.879323+0000","last_clean":"2024-09-11T08:08:54.879323+0000","last_became_active":"2024-09-11T08:07:34.057774+0000","last_became_peered":"2024-09-11T08:07:34.057774+0000","last_unstale":"2024-09-11T08:08:54.879323+0000","last_undegraded":"2024-09-11T08:08:54.879323+0000","last_fullsized":"2024-09-11T08:08:54.879323+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:57.690064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.473812+0000","last_change":"2024-09-11T08:08:38.326345+0000","last_active":"2024-09-11T08:08:54.473812+0000","last_peered":"2024-09-11T08:08:54.473812+0000","last_clean":"2024-09-11T08:08:54.473812+0000","last_became_active":"2024-09-11T08:07:36.633244+0000","last_became_peered":"2024-09-11T08:07:36.633244+0000","last_unstale":"2024-09-11T08:08:54.473812+0000","last_undegraded":"2024-09-11T08:08:54.473812+0000","last_fullsized":"2024-09-11T08:08:54.473812+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:38.326265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:08:38.326265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:06.322672+0000","scrub_duration":0.001417361,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"84'13","reported_seq":62,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:57.835716+0000","last_change":"2024-09-11T08:08:57.835686+0000","last_active":"2024-09-11T08:08:57.835716+0000","last_peered":"2024-09-11T08:08:57.835716+0000","last_clean":"2024-09-11T08:08:57.835716+0000","last_became_active":"2024-09-11T08:07:32.048350+0000","last_became_peered":"2024-09-11T08:07:32.048350+0000","last_unstale":"2024-09-11T08:08:57.835716+0000","last_undegraded":"2024-09-11T08:08:57.835716+0000","last_fullsized":"2024-09-11T08:08:57.835716+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'13","last_scrub_stamp":"2024-09-11T08:08:57.835634+0000","last_deep_scrub":"84'13","last_deep_scrub_stamp":"2024-09-11T08:08:57.835634+0000","last_clean_scrub_stamp":"2024-09-11T08:08:57.835634+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:18.729258+0000","scrub_duration":0.0033293229999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":44,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.474787+0000","last_change":"2024-09-11T08:08:30.471690+0000","last_active":"2024-09-11T08:08:54.474787+0000","last_peered":"2024-09-11T08:08:54.474787+0000","last_clean":"2024-09-11T08:08:54.474787+0000","last_became_active":"2024-09-11T08:07:30.044055+0000","last_became_peered":"2024-09-11T08:07:30.044055+0000","last_unstale":"2024-09-11T08:08:54.474787+0000","last_undegraded":"2024-09-11T08:08:54.474787+0000","last_fullsized":"2024-09-11T08:08:54.474787+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:30.471620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:30.471620+0000","last_clean_scrub_stamp":"2024-09-11T08:08:30.471620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:37.891558+0000","scrub_duration":0.0016049980000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.474250+0000","last_change":"2024-09-11T08:07:34.057387+0000","last_active":"2024-09-11T08:08:54.474250+0000","last_peered":"2024-09-11T08:08:54.474250+0000","last_clean":"2024-09-11T08:08:54.474250+0000","last_became_active":"2024-09-11T08:07:34.057127+0000","last_became_peered":"2024-09-11T08:07:34.057127+0000","last_unstale":"2024-09-11T08:08:54.474250+0000","last_undegraded":"2024-09-11T08:08:54.474250+0000","last_fullsized":"2024-09-11T08:08:54.474250+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":37,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:01.995140+0000","last_change":"2024-09-11T08:09:01.995115+0000","last_active":"2024-09-11T08:09:01.995140+0000","last_peered":"2024-09-11T08:09:01.995140+0000","last_clean":"2024-09-11T08:09:01.995140+0000","last_became_active":"2024-09-11T08:07:36.071513+0000","last_became_peered":"2024-09-11T08:07:36.071513+0000","last_unstale":"2024-09-11T08:09:01.995140+0000","last_undegraded":"2024-09-11T08:09:01.995140+0000","last_fullsized":"2024-09-11T08:09:01.995140+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:01.995061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:09:01.995061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:18.017918+0000","scrub_duration":0.00115722,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":57,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.879197+0000","last_change":"2024-09-11T08:08:49.951403+0000","last_active":"2024-09-11T08:08:54.879197+0000","last_peered":"2024-09-11T08:08:54.879197+0000","last_clean":"2024-09-11T08:08:54.879197+0000","last_became_active":"2024-09-11T08:07:32.041761+0000","last_became_peered":"2024-09-11T08:07:32.041761+0000","last_unstale":"2024-09-11T08:08:54.879197+0000","last_undegraded":"2024-09-11T08:08:54.879197+0000","last_fullsized":"2024-09-11T08:08:54.879197+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:08:49.951345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:49.951345+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:51.233235+0000","scrub_duration":0.002079914,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":44,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.003302+0000","last_change":"2024-09-11T08:08:48.935452+0000","last_active":"2024-09-11T08:08:55.003302+0000","last_peered":"2024-09-11T08:08:55.003302+0000","last_clean":"2024-09-11T08:08:55.003302+0000","last_became_active":"2024-09-11T08:07:30.045380+0000","last_became_peered":"2024-09-11T08:07:30.045380+0000","last_unstale":"2024-09-11T08:08:55.003302+0000","last_undegraded":"2024-09-11T08:08:55.003302+0000","last_fullsized":"2024-09-11T08:08:55.003302+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:48.935387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:48.935387+0000","last_clean_scrub_stamp":"2024-09-11T08:08:48.935387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:05.810526+0000","scrub_duration":0.001422134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"66'8","reported_seq":66,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.360500+0000","last_change":"2024-09-11T08:08:52.374593+0000","last_active":"2024-09-11T08:08:55.360500+0000","last_peered":"2024-09-11T08:08:55.360500+0000","last_clean":"2024-09-11T08:08:55.360500+0000","last_became_active":"2024-09-11T08:07:34.057937+0000","last_became_peered":"2024-09-11T08:07:34.057937+0000","last_unstale":"2024-09-11T08:08:55.360500+0000","last_undegraded":"2024-09-11T08:08:55.360500+0000","last_fullsized":"2024-09-11T08:08:55.360500+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:08:52.374539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:52.374539+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:12.088877+0000","scrub_duration":0.00146886,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":28,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.474433+0000","last_change":"2024-09-11T08:07:36.068798+0000","last_active":"2024-09-11T08:08:54.474433+0000","last_peered":"2024-09-11T08:08:54.474433+0000","last_clean":"2024-09-11T08:08:54.474433+0000","last_became_active":"2024-09-11T08:07:36.068493+0000","last_became_peered":"2024-09-11T08:07:36.068493+0000","last_unstale":"2024-09-11T08:08:54.474433+0000","last_undegraded":"2024-09-11T08:08:54.474433+0000","last_fullsized":"2024-09-11T08:08:54.474433+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":57,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.474066+0000","last_change":"2024-09-11T08:08:36.400676+0000","last_active":"2024-09-11T08:08:54.474066+0000","last_peered":"2024-09-11T08:08:54.474066+0000","last_clean":"2024-09-11T08:08:54.474066+0000","last_became_active":"2024-09-11T08:07:32.047347+0000","last_became_peered":"2024-09-11T08:07:32.047347+0000","last_unstale":"2024-09-11T08:08:54.474066+0000","last_undegraded":"2024-09-11T08:08:54.474066+0000","last_fullsized":"2024-09-11T08:08:54.474066+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-11T08:08:36.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:36.400586+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:59.480993+0000","scrub_duration":0.002193113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.473723+0000","last_change":"2024-09-11T08:07:30.039991+0000","last_active":"2024-09-11T08:08:54.473723+0000","last_peered":"2024-09-11T08:08:54.473723+0000","last_clean":"2024-09-11T08:08:54.473723+0000","last_became_active":"2024-09-11T08:07:30.039673+0000","last_became_peered":"2024-09-11T08:07:30.039673+0000","last_unstale":"2024-09-11T08:08:54.473723+0000","last_undegraded":"2024-09-11T08:08:54.473723+0000","last_fullsized":"2024-09-11T08:08:54.473723+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":65,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.295204+0000","last_change":"2024-09-11T08:08:47.966763+0000","last_active":"2024-09-11T08:08:56.295204+0000","last_peered":"2024-09-11T08:08:56.295204+0000","last_clean":"2024-09-11T08:08:56.295204+0000","last_became_active":"2024-09-11T08:07:34.060736+0000","last_became_peered":"2024-09-11T08:07:34.060736+0000","last_unstale":"2024-09-11T08:08:56.295204+0000","last_undegraded":"2024-09-11T08:08:56.295204+0000","last_fullsized":"2024-09-11T08:08:56.295204+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:08:47.966690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:47.966690+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:12.893275+0000","scrub_duration":0.0019777940000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":28,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.003951+0000","last_change":"2024-09-11T08:07:36.072648+0000","last_active":"2024-09-11T08:08:55.003951+0000","last_peered":"2024-09-11T08:08:55.003951+0000","last_clean":"2024-09-11T08:08:55.003951+0000","last_became_active":"2024-09-11T08:07:36.072359+0000","last_became_peered":"2024-09-11T08:07:36.072359+0000","last_unstale":"2024-09-11T08:08:55.003951+0000","last_undegraded":"2024-09-11T08:08:55.003951+0000","last_fullsized":"2024-09-11T08:08:55.003951+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:00.144060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":51,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.474467+0000","last_change":"2024-09-11T08:07:32.048316+0000","last_active":"2024-09-11T08:08:54.474467+0000","last_peered":"2024-09-11T08:08:54.474467+0000","last_clean":"2024-09-11T08:08:54.474467+0000","last_became_active":"2024-09-11T08:07:32.047981+0000","last_became_peered":"2024-09-11T08:07:32.047981+0000","last_unstale":"2024-09-11T08:08:54.474467+0000","last_undegraded":"2024-09-11T08:08:54.474467+0000","last_fullsized":"2024-09-11T08:08:54.474467+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":44,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.473994+0000","last_change":"2024-09-11T08:08:48.330351+0000","last_active":"2024-09-11T08:08:54.473994+0000","last_peered":"2024-09-11T08:08:54.473994+0000","last_clean":"2024-09-11T08:08:54.473994+0000","last_became_active":"2024-09-11T08:07:30.043691+0000","last_became_peered":"2024-09-11T08:07:30.043691+0000","last_unstale":"2024-09-11T08:08:54.473994+0000","last_undegraded":"2024-09-11T08:08:54.473994+0000","last_fullsized":"2024-09-11T08:08:54.473994+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:48.330285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:48.330285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:05.763303+0000","scrub_duration":0.001379929,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.003808+0000","last_change":"2024-09-11T08:07:34.053427+0000","last_active":"2024-09-11T08:08:55.003808+0000","last_peered":"2024-09-11T08:08:55.003808+0000","last_clean":"2024-09-11T08:08:55.003808+0000","last_became_active":"2024-09-11T08:07:34.053137+0000","last_became_peered":"2024-09-11T08:07:34.053137+0000","last_unstale":"2024-09-11T08:08:55.003808+0000","last_undegraded":"2024-09-11T08:08:55.003808+0000","last_fullsized":"2024-09-11T08:08:55.003808+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:00.814813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":28,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.473388+0000","last_change":"2024-09-11T08:07:36.076837+0000","last_active":"2024-09-11T08:08:54.473388+0000","last_peered":"2024-09-11T08:08:54.473388+0000","last_clean":"2024-09-11T08:08:54.473388+0000","last_became_active":"2024-09-11T08:07:36.076132+0000","last_became_peered":"2024-09-11T08:07:36.076132+0000","last_unstale":"2024-09-11T08:08:54.473388+0000","last_undegraded":"2024-09-11T08:08:54.473388+0000","last_fullsized":"2024-09-11T08:08:54.473388+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:01.236619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":58,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.332181+0000","last_change":"2024-09-11T08:08:55.332126+0000","last_active":"2024-09-11T08:08:55.332181+0000","last_peered":"2024-09-11T08:08:55.332181+0000","last_clean":"2024-09-11T08:08:55.332181+0000","last_became_active":"2024-09-11T08:07:32.049368+0000","last_became_peered":"2024-09-11T08:07:32.049368+0000","last_unstale":"2024-09-11T08:08:55.332181+0000","last_undegraded":"2024-09-11T08:08:55.332181+0000","last_fullsized":"2024-09-11T08:08:55.332181+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:08:55.332065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:55.332065+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:15.584089+0000","scrub_duration":0.001632218,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":44,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.474056+0000","last_change":"2024-09-11T08:08:45.388090+0000","last_active":"2024-09-11T08:08:54.474056+0000","last_peered":"2024-09-11T08:08:54.474056+0000","last_clean":"2024-09-11T08:08:54.474056+0000","last_became_active":"2024-09-11T08:07:30.037692+0000","last_became_peered":"2024-09-11T08:07:30.037692+0000","last_unstale":"2024-09-11T08:08:54.474056+0000","last_undegraded":"2024-09-11T08:08:54.474056+0000","last_fullsized":"2024-09-11T08:08:54.474056+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:45.388024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:45.388024+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:59.280827+0000","scrub_duration":0.0014875719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.473753+0000","last_change":"2024-09-11T08:07:34.057358+0000","last_active":"2024-09-11T08:08:54.473753+0000","last_peered":"2024-09-11T08:08:54.473753+0000","last_clean":"2024-09-11T08:08:54.473753+0000","last_became_active":"2024-09-11T08:07:34.057064+0000","last_became_peered":"2024-09-11T08:07:34.057064+0000","last_unstale":"2024-09-11T08:08:54.473753+0000","last_undegraded":"2024-09-11T08:08:54.473753+0000","last_fullsized":"2024-09-11T08:08:54.473753+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":28,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.002821+0000","last_change":"2024-09-11T08:07:36.077697+0000","last_active":"2024-09-11T08:08:55.002821+0000","last_peered":"2024-09-11T08:08:55.002821+0000","last_clean":"2024-09-11T08:08:55.002821+0000","last_became_active":"2024-09-11T08:07:36.077406+0000","last_became_peered":"2024-09-11T08:07:36.077406+0000","last_unstale":"2024-09-11T08:08:55.002821+0000","last_undegraded":"2024-09-11T08:08:55.002821+0000","last_fullsized":"2024-09-11T08:08:55.002821+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":51,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.473491+0000","last_change":"2024-09-11T08:07:32.043568+0000","last_active":"2024-09-11T08:08:54.473491+0000","last_peered":"2024-09-11T08:08:54.473491+0000","last_clean":"2024-09-11T08:08:54.473491+0000","last_became_active":"2024-09-11T08:07:32.043276+0000","last_became_peered":"2024-09-11T08:07:32.043276+0000","last_unstale":"2024-09-11T08:08:54.473491+0000","last_undegraded":"2024-09-11T08:08:54.473491+0000","last_fullsized":"2024-09-11T08:08:54.473491+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.853222+0000","last_change":"2024-09-11T08:08:55.853168+0000","last_active":"2024-09-11T08:08:55.853222+0000","last_peered":"2024-09-11T08:08:55.853222+0000","last_clean":"2024-09-11T08:08:55.853222+0000","last_became_active":"2024-09-11T08:07:30.049152+0000","last_became_peered":"2024-09-11T08:07:30.049152+0000","last_unstale":"2024-09-11T08:08:55.853222+0000","last_undegraded":"2024-09-11T08:08:55.853222+0000","last_fullsized":"2024-09-11T08:08:55.853222+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:55.853076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:55.853076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:03.761618+0000","scrub_duration":0.0018073309999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.473057+0000","last_change":"2024-09-11T08:07:34.057322+0000","last_active":"2024-09-11T08:08:54.473057+0000","last_peered":"2024-09-11T08:08:54.473057+0000","last_clean":"2024-09-11T08:08:54.473057+0000","last_became_active":"2024-09-11T08:07:34.056935+0000","last_became_peered":"2024-09-11T08:07:34.056935+0000","last_unstale":"2024-09-11T08:08:54.473057+0000","last_undegraded":"2024-09-11T08:08:54.473057+0000","last_fullsized":"2024-09-11T08:08:54.473057+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:08:57.406825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":28,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.473850+0000","last_change":"2024-09-11T08:07:36.633645+0000","last_active":"2024-09-11T08:08:54.473850+0000","last_peered":"2024-09-11T08:08:54.473850+0000","last_clean":"2024-09-11T08:08:54.473850+0000","last_became_active":"2024-09-11T08:07:36.633368+0000","last_became_peered":"2024-09-11T08:07:36.633368+0000","last_unstale":"2024-09-11T08:08:54.473850+0000","last_undegraded":"2024-09-11T08:08:54.473850+0000","last_fullsized":"2024-09-11T08:08:54.473850+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.473381+0000","last_change":"2024-09-11T08:07:32.046050+0000","last_active":"2024-09-11T08:08:54.473381+0000","last_peered":"2024-09-11T08:08:54.473381+0000","last_clean":"2024-09-11T08:08:54.473381+0000","last_became_active":"2024-09-11T08:07:32.045728+0000","last_became_peered":"2024-09-11T08:07:32.045728+0000","last_unstale":"2024-09-11T08:08:54.473381+0000","last_undegraded":"2024-09-11T08:08:54.473381+0000","last_fullsized":"2024-09-11T08:08:54.473381+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":44,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.003920+0000","last_change":"2024-09-11T08:08:42.056647+0000","last_active":"2024-09-11T08:08:55.003920+0000","last_peered":"2024-09-11T08:08:55.003920+0000","last_clean":"2024-09-11T08:08:55.003920+0000","last_became_active":"2024-09-11T08:07:30.049409+0000","last_became_peered":"2024-09-11T08:07:30.049409+0000","last_unstale":"2024-09-11T08:08:55.003920+0000","last_undegraded":"2024-09-11T08:08:55.003920+0000","last_fullsized":"2024-09-11T08:08:55.003920+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:42.056591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:42.056591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:56.872271+0000","scrub_duration":0.003068809,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":39,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.652154+0000","last_change":"2024-09-11T08:08:34.506991+0000","last_active":"2024-09-11T08:08:54.652154+0000","last_peered":"2024-09-11T08:08:54.652154+0000","last_clean":"2024-09-11T08:08:54.652154+0000","last_became_active":"2024-09-11T08:07:34.055082+0000","last_became_peered":"2024-09-11T08:07:34.055082+0000","last_unstale":"2024-09-11T08:08:54.652154+0000","last_undegraded":"2024-09-11T08:08:54.652154+0000","last_fullsized":"2024-09-11T08:08:54.652154+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:34.506927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:34.506927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:46.584592+0000","scrub_duration":0.0015499070000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":28,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.474742+0000","last_change":"2024-09-11T08:07:36.072035+0000","last_active":"2024-09-11T08:08:54.474742+0000","last_peered":"2024-09-11T08:08:54.474742+0000","last_clean":"2024-09-11T08:08:54.474742+0000","last_became_active":"2024-09-11T08:07:36.071669+0000","last_became_peered":"2024-09-11T08:07:36.071669+0000","last_unstale":"2024-09-11T08:08:54.474742+0000","last_undegraded":"2024-09-11T08:08:54.474742+0000","last_fullsized":"2024-09-11T08:08:54.474742+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:03.407787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":28,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.004080+0000","last_change":"2024-09-11T08:07:36.071729+0000","last_active":"2024-09-11T08:08:55.004080+0000","last_peered":"2024-09-11T08:08:55.004080+0000","last_clean":"2024-09-11T08:08:55.004080+0000","last_became_active":"2024-09-11T08:07:36.071468+0000","last_became_peered":"2024-09-11T08:07:36.071468+0000","last_unstale":"2024-09-11T08:08:55.004080+0000","last_undegraded":"2024-09-11T08:08:55.004080+0000","last_fullsized":"2024-09-11T08:08:55.004080+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":44,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.473590+0000","last_change":"2024-09-11T08:08:49.428032+0000","last_active":"2024-09-11T08:08:54.473590+0000","last_peered":"2024-09-11T08:08:54.473590+0000","last_clean":"2024-09-11T08:08:54.473590+0000","last_became_active":"2024-09-11T08:07:30.046702+0000","last_became_peered":"2024-09-11T08:07:30.046702+0000","last_unstale":"2024-09-11T08:08:54.473590+0000","last_undegraded":"2024-09-11T08:08:54.473590+0000","last_fullsized":"2024-09-11T08:08:54.473590+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:49.427944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:49.427944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:11.537893+0000","scrub_duration":0.001383034,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":50,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:54.652733+0000","last_change":"2024-09-11T08:07:32.054959+0000","last_active":"2024-09-11T08:08:54.652733+0000","last_peered":"2024-09-11T08:08:54.652733+0000","last_clean":"2024-09-11T08:08:54.652733+0000","last_became_active":"2024-09-11T08:07:32.054656+0000","last_became_peered":"2024-09-11T08:07:32.054656+0000","last_unstale":"2024-09-11T08:08:54.652733+0000","last_undegraded":"2024-09-11T08:08:54.652733+0000","last_fullsized":"2024-09-11T08:08:54.652733+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.004016+0000","last_change":"2024-09-11T08:08:53.869870+0000","last_active":"2024-09-11T08:08:55.004016+0000","last_peered":"2024-09-11T08:08:55.004016+0000","last_clean":"2024-09-11T08:08:55.004016+0000","last_became_active":"2024-09-11T08:07:34.061453+0000","last_became_peered":"2024-09-11T08:07:34.061453+0000","last_unstale":"2024-09-11T08:08:55.004016+0000","last_undegraded":"2024-09-11T08:08:55.004016+0000","last_fullsized":"2024-09-11T08:08:55.004016+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:53.869813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:53.869813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:59.468130+0000","scrub_duration":0.0012524350000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":410,"ondisk_log_size":410,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":29,"num_read_kb":29,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":86,"num_read_kb":81,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266711,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298780,"kb_used_data":1428,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446372,"statfs":{"total":95995035648,"available":95689084928,"internally_reserved":0,"allocated":1462272,"data_stored":951425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.45300000000000001}]},{"osd":1,"last update":"Wed Sep 11 08:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.46899999999999997}]},{"osd":2,"last update":"Wed Sep 11 08:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.439},{"interface":"front","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.39100000000000001}]},{"osd":3,"last update":"Wed Sep 11 08:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":7.9210000000000003,"5min":7.9210000000000003,"15min":7.9210000000000003},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.41699999999999998}]},{"osd":4,"last update":"Wed Sep 11 08:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.32100000000000001}]},{"osd":5,"last update":"Wed Sep 11 08:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.37}]},{"osd":6,"last update":"Wed Sep 11 08:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.433},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.49099999999999999}]}]},{"osd":6,"up_from":45,"seq":193273528348,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298764,"kb_used_data":1412,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446388,"statfs":{"total":95995035648,"available":95689101312,"internally_reserved":0,"allocated":1445888,"data_stored":951248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.49299999999999999,"15min":0.496},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.59099999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.50700000000000001,"15min":0.50600000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.72099999999999997,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.42099999999999999}]},{"osd":1,"last update":"Wed Sep 11 08:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.47999999999999998,"15min":0.47799999999999998},"min":{"1min":0.32700000000000001,"5min":0.318,"15min":0.318},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.497,"5min":0.505,"15min":0.50600000000000001},"min":{"1min":0.33500000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.65900000000000003,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.378}]},{"osd":2,"last update":"Wed Sep 11 08:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.5,"15min":0.502},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.67200000000000004,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.499,"5min":0.48499999999999999,"15min":0.48299999999999998},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.72399999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.34799999999999998}]},{"osd":3,"last update":"Wed Sep 11 08:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.52400000000000002,"15min":0.52600000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.53900000000000003,"15min":0.53800000000000003},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.73499999999999999,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.46000000000000002}]},{"osd":4,"last update":"Wed Sep 11 08:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.39000000000000001,"15min":0.38700000000000001},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.59399999999999997,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.41999999999999998,"15min":0.41199999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.318}]},{"osd":5,"last update":"Wed Sep 11 08:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.48899999999999999,"15min":0.47999999999999998},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.434,"5min":0.41399999999999998,"15min":0.41099999999999998},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.62,"5min":0.629,"15min":0.629},"last":0.40000000000000002}]},{"osd":7,"last update":"Wed Sep 11 08:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.44}]}]},{"osd":5,"up_from":38,"seq":163208757281,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298180,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446972,"statfs":{"total":95995035648,"available":95689699328,"internally_reserved":0,"allocated":847872,"data_stored":360087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.51400000000000001,"15min":0.52200000000000002},"min":{"1min":0.318,"5min":0.31,"15min":0.31},"max":{"1min":0.63500000000000001,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.503,"15min":0.501},"min":{"1min":0.41399999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.56999999999999995}]},{"osd":1,"last update":"Wed Sep 11 08:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":1.1910000000000001,"15min":1.3029999999999999},"min":{"1min":0.33000000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.82499999999999996,"5min":16.004999999999999,"15min":16.004999999999999},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.36199999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.52000000000000002}]},{"osd":2,"last update":"Wed Sep 11 08:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":1.206,"15min":1.323},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.754,"5min":16.018000000000001,"15min":16.018000000000001},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.57799999999999996}]},{"osd":3,"last update":"Wed Sep 11 08:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.53100000000000003,"15min":0.52900000000000003},"min":{"1min":0.44900000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.72299999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.505,"5min":0.51600000000000001,"15min":0.51800000000000002},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.623}]},{"osd":4,"last update":"Wed Sep 11 08:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.39700000000000002,"15min":0.39300000000000002},"min":{"1min":0.27500000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.40000000000000002,"15min":0.38800000000000001},"min":{"1min":0.36499999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.38400000000000001}]},{"osd":6,"last update":"Wed Sep 11 08:08:56 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.57699999999999996,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.495,"5min":0.45200000000000001,"15min":0.44500000000000001},"min":{"1min":0.316,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.67100000000000004,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.46200000000000002}]},{"osd":7,"last update":"Wed Sep 11 08:08:22 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.495},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.443}]}]},{"osd":4,"up_from":31,"seq":133143986214,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298204,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446948,"statfs":{"total":95995035648,"available":95689674752,"internally_reserved":0,"allocated":872448,"data_stored":361440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.499,"15min":0.50900000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.59699999999999998,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.52400000000000002,"15min":0.53500000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.58899999999999997,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.438}]},{"osd":1,"last update":"Wed Sep 11 08:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.505,"15min":0.51300000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.66800000000000004,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.52300000000000002,"15min":0.52900000000000003},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.68899999999999995,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.498}]},{"osd":2,"last update":"Wed Sep 11 08:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.52000000000000002,"15min":0.52800000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.58799999999999997,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.51600000000000001,"15min":0.52500000000000002},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.623,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.39900000000000002}]},{"osd":3,"last update":"Wed Sep 11 08:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.50900000000000001,"15min":0.51500000000000001},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.64300000000000002,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.51800000000000002,"15min":0.52600000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.61499999999999999,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.40799999999999997}]},{"osd":5,"last update":"Wed Sep 11 08:08:31 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.36899999999999999,"15min":0.36699999999999999},"min":{"1min":0.27800000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.36899999999999999,"15min":0.36699999999999999},"min":{"1min":0.27900000000000003,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.33500000000000002}]},{"osd":6,"last update":"Wed Sep 11 08:08:56 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.39100000000000001,"15min":0.38300000000000001},"min":{"1min":0.30399999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.314},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.44,"15min":0.432},"min":{"1min":0.32200000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.38500000000000001}]},{"osd":7,"last update":"Wed Sep 11 08:08:20 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.377}]}]},{"osd":3,"up_from":25,"seq":107374182443,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298220,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446932,"statfs":{"total":95995035648,"available":95689658368,"internally_reserved":0,"allocated":888832,"data_stored":361226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:08:53 2024","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.36199999999999999,"15min":0.374},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":0.55100000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.40300000000000002,"15min":0.41799999999999998},"min":{"1min":0.26900000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.55100000000000005,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.23899999999999999}]},{"osd":1,"last update":"Wed Sep 11 08:08:53 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.41599999999999998,"15min":0.437},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.497,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.41399999999999998,"15min":0.42899999999999999},"min":{"1min":0.24399999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.53800000000000003,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.35699999999999998}]},{"osd":2,"last update":"Wed Sep 11 08:08:53 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.434,"15min":0.44900000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.47999999999999998,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.371,"5min":0.40600000000000003,"15min":0.41899999999999998},"min":{"1min":0.26700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.53100000000000003,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.377}]},{"osd":4,"last update":"Wed Sep 11 08:08:12 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54800000000000004,"15min":0.54900000000000004},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.53500000000000003,"15min":0.53700000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.42999999999999999}]},{"osd":5,"last update":"Wed Sep 11 08:08:33 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.54200000000000004,"15min":0.53800000000000003},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.501},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.40699999999999997,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":16.757000000000001}]},{"osd":6,"last update":"Wed Sep 11 08:08:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.53400000000000003,"15min":0.53000000000000003},"min":{"1min":0.35899999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":16.789000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.55400000000000005,"15min":0.55800000000000005},"min":{"1min":0.39500000000000002,"5min":0.373,"15min":0.373},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":16.748000000000001}]},{"osd":7,"last update":"Wed Sep 11 08:08:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.442}]}]},{"osd":2,"up_from":18,"seq":77309411374,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298176,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446976,"statfs":{"total":95995035648,"available":95689703424,"internally_reserved":0,"allocated":843776,"data_stored":359818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:08:24 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.38400000000000001,"15min":0.38100000000000001},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.58099999999999996,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.254},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.37,"15min":0.35999999999999999},"min":{"1min":0.25,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.55600000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.32600000000000001}]},{"osd":1,"last update":"Wed Sep 11 08:08:24 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.39500000000000002,"15min":0.38},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.60899999999999999,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.39800000000000002,"15min":0.38600000000000001},"min":{"1min":0.28699999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.26900000000000002}]},{"osd":3,"last update":"Wed Sep 11 08:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.45100000000000001,"15min":0.46100000000000002},"min":{"1min":0.27300000000000002,"5min":0.218,"15min":0.218},"max":{"1min":0.629,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.309},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.45100000000000001,"15min":0.46500000000000002},"min":{"1min":0.318,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.29999999999999999}]},{"osd":4,"last update":"Wed Sep 11 08:08:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.58299999999999996,"15min":0.58699999999999997},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.71199999999999997,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.42599999999999999}]},{"osd":5,"last update":"Wed Sep 11 08:08:35 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.57699999999999996,"15min":0.58299999999999996},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.751,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.59199999999999997,"15min":0.59999999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.65300000000000002,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.51800000000000002}]},{"osd":6,"last update":"Wed Sep 11 08:08:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.58499999999999996,"15min":0.59399999999999997},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.59099999999999997,"15min":0.59699999999999998},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.70299999999999996,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.40300000000000002}]},{"osd":7,"last update":"Wed Sep 11 08:08:22 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.501}]}]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":901120,"data_stored":361384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.32000000000000001,"15min":0.29799999999999999},"min":{"1min":0.22600000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.33200000000000002,"15min":0.312},"min":{"1min":0.249,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"last":0.224}]},{"osd":2,"last update":"Wed Sep 11 08:08:28 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.36199999999999999,"15min":0.34799999999999998},"min":{"1min":0.27700000000000002,"5min":0.224,"15min":0.224},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.37,"5min":0.35399999999999998,"15min":0.34200000000000003},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.31900000000000001}]},{"osd":3,"last update":"Wed Sep 11 08:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.41399999999999998,"15min":0.42199999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.246},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.439,"15min":0.437},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.198,"5min":1.198,"15min":1.198},"last":0.28199999999999997}]},{"osd":4,"last update":"Wed Sep 11 08:08:08 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54300000000000004,"15min":0.54200000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.55800000000000005,"15min":0.55600000000000005},"min":{"1min":0.41899999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.73499999999999999,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.46500000000000002}]},{"osd":5,"last update":"Wed Sep 11 08:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.53700000000000003,"15min":0.53000000000000003},"min":{"1min":0.40699999999999997,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.77600000000000002,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.53100000000000003,"15min":0.52700000000000002},"min":{"1min":0.38400000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.52300000000000002}]},{"osd":6,"last update":"Wed Sep 11 08:08:56 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.55500000000000005,"15min":0.55900000000000005},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.59899999999999998,"15min":0.60099999999999998},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.434}]},{"osd":7,"last update":"Wed Sep 11 08:08:17 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.45900000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738424,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298792,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446360,"statfs":{"total":95995035648,"available":95689072640,"internally_reserved":0,"allocated":1474560,"data_stored":952371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 11 08:08:01 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.34100000000000003,"15min":0.315},"min":{"1min":0.25800000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.53900000000000003,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.35599999999999998,"15min":0.33700000000000002},"min":{"1min":0.23999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.313}]},{"osd":2,"last update":"Wed Sep 11 08:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.34699999999999998},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.48799999999999999,"5min":0.53000000000000003,"15min":0.53000000000000003},"last":0.215},{"interface":"front","average":{"1min":0.378,"5min":0.378,"15min":0.36899999999999999},"min":{"1min":0.26600000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.56299999999999994,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.32300000000000001}]},{"osd":3,"last update":"Wed Sep 11 08:08:51 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.44,"15min":0.44},"min":{"1min":0.27600000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.52300000000000002,"5min":0.62,"15min":0.62},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.42499999999999999,"15min":0.42599999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.52800000000000002,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.27900000000000003}]},{"osd":4,"last update":"Wed Sep 11 08:08:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.55000000000000004,"15min":0.54600000000000004},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.43099999999999999}]},{"osd":5,"last update":"Wed Sep 11 08:08:36 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.58499999999999996,"15min":0.58299999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.748,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.55800000000000005,"15min":0.55300000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.379}]},{"osd":6,"last update":"Wed Sep 11 08:08:59 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.60899999999999999,"15min":0.61699999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.75600000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.60299999999999998,"15min":0.57099999999999995},"min":{"1min":0.433,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":4.7069999999999999,"5min":4.7069999999999999,"15min":4.7069999999999999},"last":0.439}]},{"osd":7,"last update":"Wed Sep 11 08:08:19 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.39100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T08:09:05.854 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T08:09:05.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:05 smithi150 ceph-mon[29276]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-11T08:09:05.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:05 smithi150 ceph-mon[29276]: 6.19 deep-scrub starts 2024-09-11T08:09:05.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:05 smithi150 ceph-mon[29276]: 6.19 deep-scrub ok 2024-09-11T08:09:05.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:05 smithi150 ceph-mon[29276]: from='client.24611 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:09:05.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:05 smithi150 ceph-mon[39612]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-11T08:09:05.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:05 smithi150 ceph-mon[39612]: 6.19 deep-scrub starts 2024-09-11T08:09:05.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:05 smithi150 ceph-mon[39612]: 6.19 deep-scrub ok 2024-09-11T08:09:05.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:05 smithi150 ceph-mon[39612]: from='client.24611 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:09:06.137 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:09:06.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:05 smithi181 ceph-mon[32701]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-11T08:09:06.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:05 smithi181 ceph-mon[32701]: 6.19 deep-scrub starts 2024-09-11T08:09:06.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:05 smithi181 ceph-mon[32701]: 6.19 deep-scrub ok 2024-09-11T08:09:06.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:05 smithi181 ceph-mon[32701]: from='client.24611 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:09:07.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:06 smithi150 ceph-mon[29276]: 5.7 scrub starts 2024-09-11T08:09:07.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:06 smithi150 ceph-mon[29276]: 5.7 scrub ok 2024-09-11T08:09:07.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:06 smithi150 ceph-mon[29276]: 6.9 scrub starts 2024-09-11T08:09:07.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:06 smithi150 ceph-mon[29276]: 6.9 scrub ok 2024-09-11T08:09:07.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:06 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-09-11T08:09:07.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:06 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-09-11T08:09:07.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:06 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-09-11T08:09:07.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:06 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:09:07.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:06 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:07.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:06 smithi150 ceph-mon[39612]: 5.7 scrub starts 2024-09-11T08:09:07.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:06 smithi150 ceph-mon[39612]: 5.7 scrub ok 2024-09-11T08:09:07.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:06 smithi150 ceph-mon[39612]: 6.9 scrub starts 2024-09-11T08:09:07.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:06 smithi150 ceph-mon[39612]: 6.9 scrub ok 2024-09-11T08:09:07.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:06 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-09-11T08:09:07.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:06 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-09-11T08:09:07.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:06 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-09-11T08:09:07.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:06 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:09:07.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:06 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:07.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:06 smithi181 ceph-mon[32701]: 5.7 scrub starts 2024-09-11T08:09:07.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:06 smithi181 ceph-mon[32701]: 5.7 scrub ok 2024-09-11T08:09:07.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:06 smithi181 ceph-mon[32701]: 6.9 scrub starts 2024-09-11T08:09:07.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:06 smithi181 ceph-mon[32701]: 6.9 scrub ok 2024-09-11T08:09:07.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:06 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-09-11T08:09:07.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:06 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-09-11T08:09:07.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:06 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-09-11T08:09:07.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:06 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:09:07.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:06 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:07.978 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:07 smithi150 ceph-mon[29276]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 903 B/s rd, 0 op/s 2024-09-11T08:09:07.979 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:07 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-09-11T08:09:07.979 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:07 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-09-11T08:09:07.979 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:07 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-09-11T08:09:07.979 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:07 smithi150 ceph-mon[29276]: osdmap e85: 8 total, 8 up, 8 in 2024-09-11T08:09:07.979 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:07 smithi150 ceph-mon[29276]: 3.6 scrub starts 2024-09-11T08:09:07.979 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:07 smithi150 ceph-mon[29276]: 3.6 scrub ok 2024-09-11T08:09:07.979 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:07 smithi150 ceph-mon[39612]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 903 B/s rd, 0 op/s 2024-09-11T08:09:07.979 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:07 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-09-11T08:09:07.979 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:07 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-09-11T08:09:07.979 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:07 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-09-11T08:09:07.979 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:07 smithi150 ceph-mon[39612]: osdmap e85: 8 total, 8 up, 8 in 2024-09-11T08:09:07.979 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:07 smithi150 ceph-mon[39612]: 3.6 scrub starts 2024-09-11T08:09:07.980 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:07 smithi150 ceph-mon[39612]: 3.6 scrub ok 2024-09-11T08:09:08.073 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:09:07 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:09:08.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:07 smithi181 ceph-mon[32701]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 903 B/s rd, 0 op/s 2024-09-11T08:09:08.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:07 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-09-11T08:09:08.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:07 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-09-11T08:09:08.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:07 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-09-11T08:09:08.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:07 smithi181 ceph-mon[32701]: osdmap e85: 8 total, 8 up, 8 in 2024-09-11T08:09:08.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:07 smithi181 ceph-mon[32701]: 3.6 scrub starts 2024-09-11T08:09:08.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:07 smithi181 ceph-mon[32701]: 3.6 scrub ok 2024-09-11T08:09:09.176 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:09:09.176 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:08 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:09:09.177 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:08 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:09:09.177 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:08 smithi150 ceph-mon[29276]: 4.1a scrub starts 2024-09-11T08:09:09.177 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:08 smithi150 ceph-mon[29276]: 4.1a scrub ok 2024-09-11T08:09:09.177 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:08 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:09:09.177 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:08 smithi150 ceph-mon[29276]: 5.16 scrub starts 2024-09-11T08:09:09.177 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:08 smithi150 ceph-mon[29276]: 5.16 scrub ok 2024-09-11T08:09:09.177 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:08 smithi150 ceph-mon[29276]: 3.10 scrub starts 2024-09-11T08:09:09.177 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:08 smithi150 ceph-mon[29276]: 3.10 scrub ok 2024-09-11T08:09:09.177 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:08 smithi150 ceph-mon[29276]: osdmap e86: 8 total, 8 up, 8 in 2024-09-11T08:09:09.177 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:08 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:09.177 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:08 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:09.177 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:08 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:09.178 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:08 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:09:09.178 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:08 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:09:09.178 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:08 smithi150 ceph-mon[39612]: 4.1a scrub starts 2024-09-11T08:09:09.178 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:08 smithi150 ceph-mon[39612]: 4.1a scrub ok 2024-09-11T08:09:09.178 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:08 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:09:09.180 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:08 smithi150 ceph-mon[39612]: 5.16 scrub starts 2024-09-11T08:09:09.180 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:08 smithi150 ceph-mon[39612]: 5.16 scrub ok 2024-09-11T08:09:09.180 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:08 smithi150 ceph-mon[39612]: 3.10 scrub starts 2024-09-11T08:09:09.180 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:08 smithi150 ceph-mon[39612]: 3.10 scrub ok 2024-09-11T08:09:09.180 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:08 smithi150 ceph-mon[39612]: osdmap e86: 8 total, 8 up, 8 in 2024-09-11T08:09:09.180 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:08 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:09.180 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:08 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:09.180 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:08 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:09.184 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-09-11T08:09:09.267 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:09 smithi181 systemd[1]: Starting Ceph node-exporter.b for c1bddff4-7013-11ef-bcea-c7b262605968... 2024-09-11T08:09:09.267 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:08 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:09:09.267 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:08 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:09:09.267 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:08 smithi181 ceph-mon[32701]: 4.1a scrub starts 2024-09-11T08:09:09.267 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:08 smithi181 ceph-mon[32701]: 4.1a scrub ok 2024-09-11T08:09:09.267 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:08 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:09:09.268 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:08 smithi181 ceph-mon[32701]: 5.16 scrub starts 2024-09-11T08:09:09.268 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:08 smithi181 ceph-mon[32701]: 5.16 scrub ok 2024-09-11T08:09:09.268 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:08 smithi181 ceph-mon[32701]: 3.10 scrub starts 2024-09-11T08:09:09.268 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:08 smithi181 ceph-mon[32701]: 3.10 scrub ok 2024-09-11T08:09:09.268 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:08 smithi181 ceph-mon[32701]: osdmap e86: 8 total, 8 up, 8 in 2024-09-11T08:09:09.268 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:08 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:09.268 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:08 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:09.268 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:08 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:09.573 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:09 smithi181 bash[63714]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-09-11T08:09:09.921 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":47,"stamp":"2024-09-11T08:09:08.693032+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596188,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":944,"num_read_kb":801,"num_write":543,"num_write_kb":1671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":571,"ondisk_log_size":571,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2387444,"kb_used_data":7164,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":747573772,"statfs":{"total":767960285184,"available":765515542528,"internally_reserved":0,"allocated":7335936,"data_stored":4241015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001710"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.403338+0000","last_change":"2024-09-11T08:08:36.105232+0000","last_active":"2024-09-11T08:08:56.403338+0000","last_peered":"2024-09-11T08:08:56.403338+0000","last_clean":"2024-09-11T08:08:56.403338+0000","last_became_active":"2024-09-11T08:07:30.044976+0000","last_became_peered":"2024-09-11T08:07:30.044976+0000","last_unstale":"2024-09-11T08:08:56.403338+0000","last_undegraded":"2024-09-11T08:08:56.403338+0000","last_fullsized":"2024-09-11T08:08:56.403338+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:36.105175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:36.105175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:56.596929+0000","scrub_duration":0.0016664970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":49,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.358857+0000","last_change":"2024-09-11T08:07:32.048578+0000","last_active":"2024-09-11T08:08:55.358857+0000","last_peered":"2024-09-11T08:08:55.358857+0000","last_clean":"2024-09-11T08:08:55.358857+0000","last_became_active":"2024-09-11T08:07:32.048048+0000","last_became_peered":"2024-09-11T08:07:32.048048+0000","last_unstale":"2024-09-11T08:08:55.358857+0000","last_undegraded":"2024-09-11T08:08:55.358857+0000","last_fullsized":"2024-09-11T08:08:55.358857+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.011005+0000","last_change":"2024-09-11T08:08:38.486126+0000","last_active":"2024-09-11T08:08:56.011005+0000","last_peered":"2024-09-11T08:08:56.011005+0000","last_clean":"2024-09-11T08:08:56.011005+0000","last_became_active":"2024-09-11T08:07:34.058828+0000","last_became_peered":"2024-09-11T08:07:34.058828+0000","last_unstale":"2024-09-11T08:08:56.011005+0000","last_undegraded":"2024-09-11T08:08:56.011005+0000","last_fullsized":"2024-09-11T08:08:56.011005+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:38.486057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:38.486057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:44.914804+0000","scrub_duration":0.002230531,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:02.470792+0000","last_change":"2024-09-11T08:09:02.470722+0000","last_active":"2024-09-11T08:09:02.470792+0000","last_peered":"2024-09-11T08:09:02.470792+0000","last_clean":"2024-09-11T08:09:02.470792+0000","last_became_active":"2024-09-11T08:07:36.074789+0000","last_became_peered":"2024-09-11T08:07:36.074789+0000","last_unstale":"2024-09-11T08:09:02.470792+0000","last_undegraded":"2024-09-11T08:09:02.470792+0000","last_fullsized":"2024-09-11T08:09:02.470792+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:02.470649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:02.470649+0000","last_clean_scrub_stamp":"2024-09-11T08:09:02.470649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:18.421855+0000","scrub_duration":0.00141182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":29,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.359766+0000","last_change":"2024-09-11T08:07:36.633716+0000","last_active":"2024-09-11T08:08:55.359766+0000","last_peered":"2024-09-11T08:08:55.359766+0000","last_clean":"2024-09-11T08:08:55.359766+0000","last_became_active":"2024-09-11T08:07:36.633423+0000","last_became_peered":"2024-09-11T08:07:36.633423+0000","last_unstale":"2024-09-11T08:08:55.359766+0000","last_undegraded":"2024-09-11T08:08:55.359766+0000","last_fullsized":"2024-09-11T08:08:55.359766+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.360053+0000","last_change":"2024-09-11T08:07:30.045845+0000","last_active":"2024-09-11T08:08:55.360053+0000","last_peered":"2024-09-11T08:08:55.360053+0000","last_clean":"2024-09-11T08:08:55.360053+0000","last_became_active":"2024-09-11T08:07:30.045579+0000","last_became_peered":"2024-09-11T08:07:30.045579+0000","last_unstale":"2024-09-11T08:08:55.360053+0000","last_undegraded":"2024-09-11T08:08:55.360053+0000","last_fullsized":"2024-09-11T08:08:55.360053+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":68,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.408379+0000","last_change":"2024-09-11T08:08:56.408326+0000","last_active":"2024-09-11T08:08:56.408379+0000","last_peered":"2024-09-11T08:08:56.408379+0000","last_clean":"2024-09-11T08:08:56.408379+0000","last_became_active":"2024-09-11T08:07:32.045848+0000","last_became_peered":"2024-09-11T08:07:32.045848+0000","last_unstale":"2024-09-11T08:08:56.408379+0000","last_undegraded":"2024-09-11T08:08:56.408379+0000","last_fullsized":"2024-09-11T08:08:56.408379+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-11T08:08:56.408272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:56.408272+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:09.630406+0000","scrub_duration":0.001771557,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":31,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.358773+0000","last_change":"2024-09-11T08:07:34.053483+0000","last_active":"2024-09-11T08:08:55.358773+0000","last_peered":"2024-09-11T08:08:55.358773+0000","last_clean":"2024-09-11T08:08:55.358773+0000","last_became_active":"2024-09-11T08:07:34.052243+0000","last_became_peered":"2024-09-11T08:07:34.052243+0000","last_unstale":"2024-09-11T08:08:55.358773+0000","last_undegraded":"2024-09-11T08:08:55.358773+0000","last_fullsized":"2024-09-11T08:08:55.358773+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.295460+0000","last_change":"2024-09-11T08:08:54.878501+0000","last_active":"2024-09-11T08:08:56.295460+0000","last_peered":"2024-09-11T08:08:56.295460+0000","last_clean":"2024-09-11T08:08:56.295460+0000","last_became_active":"2024-09-11T08:07:30.043950+0000","last_became_peered":"2024-09-11T08:07:30.043950+0000","last_unstale":"2024-09-11T08:08:56.295460+0000","last_undegraded":"2024-09-11T08:08:56.295460+0000","last_fullsized":"2024-09-11T08:08:56.295460+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:54.878409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:54.878409+0000","last_clean_scrub_stamp":"2024-09-11T08:08:54.878409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:07.483340+0000","scrub_duration":0.0015972390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:07.628517+0000","last_change":"2024-09-11T08:09:07.628460+0000","last_active":"2024-09-11T08:09:07.628517+0000","last_peered":"2024-09-11T08:09:07.628517+0000","last_clean":"2024-09-11T08:09:07.628517+0000","last_became_active":"2024-09-11T08:07:32.044626+0000","last_became_peered":"2024-09-11T08:07:32.044626+0000","last_unstale":"2024-09-11T08:09:07.628517+0000","last_undegraded":"2024-09-11T08:09:07.628517+0000","last_fullsized":"2024-09-11T08:09:07.628517+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:09:07.628369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:09:07.628369+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:10.676317+0000","scrub_duration":0.0023051830000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.294789+0000","last_change":"2024-09-11T08:07:34.061363+0000","last_active":"2024-09-11T08:08:56.294789+0000","last_peered":"2024-09-11T08:08:56.294789+0000","last_clean":"2024-09-11T08:08:56.294789+0000","last_became_active":"2024-09-11T08:07:34.058689+0000","last_became_peered":"2024-09-11T08:07:34.058689+0000","last_unstale":"2024-09-11T08:08:56.294789+0000","last_undegraded":"2024-09-11T08:08:56.294789+0000","last_fullsized":"2024-09-11T08:08:56.294789+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.402638+0000","last_change":"2024-09-11T08:08:40.071203+0000","last_active":"2024-09-11T08:08:56.402638+0000","last_peered":"2024-09-11T08:08:56.402638+0000","last_clean":"2024-09-11T08:08:56.402638+0000","last_became_active":"2024-09-11T08:07:36.076824+0000","last_became_peered":"2024-09-11T08:07:36.076824+0000","last_unstale":"2024-09-11T08:08:56.402638+0000","last_undegraded":"2024-09-11T08:08:56.402638+0000","last_fullsized":"2024-09-11T08:08:56.402638+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:40.071106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:08:40.071106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:54.333357+0000","scrub_duration":0.0019674499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.295183+0000","last_change":"2024-09-11T08:08:36.825857+0000","last_active":"2024-09-11T08:08:56.295183+0000","last_peered":"2024-09-11T08:08:56.295183+0000","last_clean":"2024-09-11T08:08:56.295183+0000","last_became_active":"2024-09-11T08:07:30.043277+0000","last_became_peered":"2024-09-11T08:07:30.043277+0000","last_unstale":"2024-09-11T08:08:56.295183+0000","last_undegraded":"2024-09-11T08:08:56.295183+0000","last_fullsized":"2024-09-11T08:08:56.295183+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:36.825794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:36.825794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:47.493434+0000","scrub_duration":0.0016805920000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.359266+0000","last_change":"2024-09-11T08:07:32.048969+0000","last_active":"2024-09-11T08:08:55.359266+0000","last_peered":"2024-09-11T08:08:55.359266+0000","last_clean":"2024-09-11T08:08:55.359266+0000","last_became_active":"2024-09-11T08:07:32.048661+0000","last_became_peered":"2024-09-11T08:07:32.048661+0000","last_unstale":"2024-09-11T08:08:55.359266+0000","last_undegraded":"2024-09-11T08:08:55.359266+0000","last_fullsized":"2024-09-11T08:08:55.359266+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":31,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.174047+0000","last_change":"2024-09-11T08:07:34.061194+0000","last_active":"2024-09-11T08:08:56.174047+0000","last_peered":"2024-09-11T08:08:56.174047+0000","last_clean":"2024-09-11T08:08:56.174047+0000","last_became_active":"2024-09-11T08:07:34.060871+0000","last_became_peered":"2024-09-11T08:07:34.060871+0000","last_unstale":"2024-09-11T08:08:56.174047+0000","last_undegraded":"2024-09-11T08:08:56.174047+0000","last_fullsized":"2024-09-11T08:08:56.174047+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":38,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:04.782671+0000","last_change":"2024-09-11T08:09:04.782636+0000","last_active":"2024-09-11T08:09:04.782671+0000","last_peered":"2024-09-11T08:09:04.782671+0000","last_clean":"2024-09-11T08:09:04.782671+0000","last_became_active":"2024-09-11T08:07:36.077602+0000","last_became_peered":"2024-09-11T08:07:36.077602+0000","last_unstale":"2024-09-11T08:09:04.782671+0000","last_undegraded":"2024-09-11T08:09:04.782671+0000","last_fullsized":"2024-09-11T08:09:04.782671+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:04.782577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:04.782577+0000","last_clean_scrub_stamp":"2024-09-11T08:09:04.782577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:23.367449+0000","scrub_duration":0.0014982330000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":29,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.359276+0000","last_change":"2024-09-11T08:07:36.633951+0000","last_active":"2024-09-11T08:08:55.359276+0000","last_peered":"2024-09-11T08:08:55.359276+0000","last_clean":"2024-09-11T08:08:55.359276+0000","last_became_active":"2024-09-11T08:07:36.633338+0000","last_became_peered":"2024-09-11T08:07:36.633338+0000","last_unstale":"2024-09-11T08:08:55.359276+0000","last_undegraded":"2024-09-11T08:08:55.359276+0000","last_fullsized":"2024-09-11T08:08:55.359276+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.403206+0000","last_change":"2024-09-11T08:07:30.046004+0000","last_active":"2024-09-11T08:08:56.403206+0000","last_peered":"2024-09-11T08:08:56.403206+0000","last_clean":"2024-09-11T08:08:56.403206+0000","last_became_active":"2024-09-11T08:07:30.045739+0000","last_became_peered":"2024-09-11T08:07:30.045739+0000","last_unstale":"2024-09-11T08:08:56.403206+0000","last_undegraded":"2024-09-11T08:08:56.403206+0000","last_fullsized":"2024-09-11T08:08:56.403206+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"66'15","reported_seq":67,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.975447+0000","last_change":"2024-09-11T08:08:56.975395+0000","last_active":"2024-09-11T08:08:56.975447+0000","last_peered":"2024-09-11T08:08:56.975447+0000","last_clean":"2024-09-11T08:08:56.975447+0000","last_became_active":"2024-09-11T08:07:32.046363+0000","last_became_peered":"2024-09-11T08:07:32.046363+0000","last_unstale":"2024-09-11T08:08:56.975447+0000","last_undegraded":"2024-09-11T08:08:56.975447+0000","last_fullsized":"2024-09-11T08:08:56.975447+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-11T08:08:56.975338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:56.975338+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:19.207433+0000","scrub_duration":0.0017787899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:01.315982+0000","last_change":"2024-09-11T08:09:01.315948+0000","last_active":"2024-09-11T08:09:01.315982+0000","last_peered":"2024-09-11T08:09:01.315982+0000","last_clean":"2024-09-11T08:09:01.315982+0000","last_became_active":"2024-09-11T08:07:34.055410+0000","last_became_peered":"2024-09-11T08:07:34.055410+0000","last_unstale":"2024-09-11T08:09:01.315982+0000","last_undegraded":"2024-09-11T08:09:01.315982+0000","last_fullsized":"2024-09-11T08:09:01.315982+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:01.315897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:09:01.315897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:14.655816+0000","scrub_duration":0.001561163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":29,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.360143+0000","last_change":"2024-09-11T08:07:36.633626+0000","last_active":"2024-09-11T08:08:55.360143+0000","last_peered":"2024-09-11T08:08:55.360143+0000","last_clean":"2024-09-11T08:08:55.360143+0000","last_became_active":"2024-09-11T08:07:36.633258+0000","last_became_peered":"2024-09-11T08:07:36.633258+0000","last_unstale":"2024-09-11T08:08:55.360143+0000","last_undegraded":"2024-09-11T08:08:55.360143+0000","last_fullsized":"2024-09-11T08:08:55.360143+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.359567+0000","last_change":"2024-09-11T08:08:51.401929+0000","last_active":"2024-09-11T08:08:55.359567+0000","last_peered":"2024-09-11T08:08:55.359567+0000","last_clean":"2024-09-11T08:08:55.359567+0000","last_became_active":"2024-09-11T08:07:30.046284+0000","last_became_peered":"2024-09-11T08:07:30.046284+0000","last_unstale":"2024-09-11T08:08:55.359567+0000","last_undegraded":"2024-09-11T08:08:55.359567+0000","last_fullsized":"2024-09-11T08:08:55.359567+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:51.401864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:51.401864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:18.688155+0000","scrub_duration":0.001460034,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":58,"reported_epoch":85,"state":"peering","last_fresh":"2024-09-11T08:09:06.874089+0000","last_change":"2024-09-11T08:09:06.874325+0000","last_active":"2024-09-11T08:09:06.874061+0000","last_peered":"2024-09-11T08:08:55.359627+0000","last_clean":"2024-09-11T08:08:55.359627+0000","last_became_active":"2024-09-11T08:07:32.046737+0000","last_became_peered":"2024-09-11T08:07:32.046737+0000","last_unstale":"2024-09-11T08:09:06.874089+0000","last_undegraded":"2024-09-11T08:09:06.874089+0000","last_fullsized":"2024-09-11T08:09:06.874089+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,4],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.358990+0000","last_change":"2024-09-11T08:08:40.419900+0000","last_active":"2024-09-11T08:08:55.358990+0000","last_peered":"2024-09-11T08:08:55.358990+0000","last_clean":"2024-09-11T08:08:55.358990+0000","last_became_active":"2024-09-11T08:07:34.056528+0000","last_became_peered":"2024-09-11T08:07:34.056528+0000","last_unstale":"2024-09-11T08:08:55.358990+0000","last_undegraded":"2024-09-11T08:08:55.358990+0000","last_fullsized":"2024-09-11T08:08:55.358990+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:40.419830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:40.419830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:09.111095+0000","scrub_duration":0.0029308149999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.173624+0000","last_change":"2024-09-11T08:07:36.072658+0000","last_active":"2024-09-11T08:08:56.173624+0000","last_peered":"2024-09-11T08:08:56.173624+0000","last_clean":"2024-09-11T08:08:56.173624+0000","last_became_active":"2024-09-11T08:07:36.072347+0000","last_became_peered":"2024-09-11T08:07:36.072347+0000","last_unstale":"2024-09-11T08:08:56.173624+0000","last_undegraded":"2024-09-11T08:08:56.173624+0000","last_fullsized":"2024-09-11T08:08:56.173624+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.011099+0000","last_change":"2024-09-11T08:07:30.046498+0000","last_active":"2024-09-11T08:08:56.011099+0000","last_peered":"2024-09-11T08:08:56.011099+0000","last_clean":"2024-09-11T08:08:56.011099+0000","last_became_active":"2024-09-11T08:07:30.045932+0000","last_became_peered":"2024-09-11T08:07:30.045932+0000","last_unstale":"2024-09-11T08:08:56.011099+0000","last_undegraded":"2024-09-11T08:08:56.011099+0000","last_fullsized":"2024-09-11T08:08:56.011099+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":48,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.403204+0000","last_change":"2024-09-11T08:07:32.048372+0000","last_active":"2024-09-11T08:08:56.403204+0000","last_peered":"2024-09-11T08:08:56.403204+0000","last_clean":"2024-09-11T08:08:56.403204+0000","last_became_active":"2024-09-11T08:07:32.048022+0000","last_became_peered":"2024-09-11T08:07:32.048022+0000","last_unstale":"2024-09-11T08:08:56.403204+0000","last_undegraded":"2024-09-11T08:08:56.403204+0000","last_fullsized":"2024-09-11T08:08:56.403204+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":70,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:03.560173+0000","last_change":"2024-09-11T08:08:54.652958+0000","last_active":"2024-09-11T08:09:03.560173+0000","last_peered":"2024-09-11T08:09:03.560173+0000","last_clean":"2024-09-11T08:09:03.560173+0000","last_became_active":"2024-09-11T08:07:34.054014+0000","last_became_peered":"2024-09-11T08:07:34.054014+0000","last_unstale":"2024-09-11T08:09:03.560173+0000","last_undegraded":"2024-09-11T08:09:03.560173+0000","last_fullsized":"2024-09-11T08:09:03.560173+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:08:54.652867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:54.652867+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:09.166003+0000","scrub_duration":0.0022231320000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":68,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.011633+0000","last_change":"2024-09-11T08:08:45.358105+0000","last_active":"2024-09-11T08:08:56.011633+0000","last_peered":"2024-09-11T08:08:56.011633+0000","last_clean":"2024-09-11T08:08:56.011633+0000","last_became_active":"2024-09-11T08:07:32.054317+0000","last_became_peered":"2024-09-11T08:07:32.054317+0000","last_unstale":"2024-09-11T08:08:56.011633+0000","last_undegraded":"2024-09-11T08:08:56.011633+0000","last_fullsized":"2024-09-11T08:08:56.011633+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-11T08:08:45.358041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:45.358041+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:50.858524+0000","scrub_duration":0.0021022879999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.360164+0000","last_change":"2024-09-11T08:07:30.044933+0000","last_active":"2024-09-11T08:08:55.360164+0000","last_peered":"2024-09-11T08:08:55.360164+0000","last_clean":"2024-09-11T08:08:55.360164+0000","last_became_active":"2024-09-11T08:07:30.044624+0000","last_became_peered":"2024-09-11T08:07:30.044624+0000","last_unstale":"2024-09-11T08:08:55.360164+0000","last_undegraded":"2024-09-11T08:08:55.360164+0000","last_fullsized":"2024-09-11T08:08:55.360164+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":58,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:03.560184+0000","last_change":"2024-09-11T08:07:34.053292+0000","last_active":"2024-09-11T08:09:03.560184+0000","last_peered":"2024-09-11T08:09:03.560184+0000","last_clean":"2024-09-11T08:09:03.560184+0000","last_became_active":"2024-09-11T08:07:34.052981+0000","last_became_peered":"2024-09-11T08:07:34.052981+0000","last_unstale":"2024-09-11T08:09:03.560184+0000","last_undegraded":"2024-09-11T08:09:03.560184+0000","last_fullsized":"2024-09-11T08:09:03.560184+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":38,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.294967+0000","last_change":"2024-09-11T08:08:51.893846+0000","last_active":"2024-09-11T08:08:56.294967+0000","last_peered":"2024-09-11T08:08:56.294967+0000","last_clean":"2024-09-11T08:08:56.294967+0000","last_became_active":"2024-09-11T08:07:36.073969+0000","last_became_peered":"2024-09-11T08:07:36.073969+0000","last_unstale":"2024-09-11T08:08:56.294967+0000","last_undegraded":"2024-09-11T08:08:56.294967+0000","last_fullsized":"2024-09-11T08:08:56.294967+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:51.893774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:08:51.893774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:00.876260+0000","scrub_duration":0.001532858,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":75,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.359977+0000","last_change":"2024-09-11T08:08:43.441600+0000","last_active":"2024-09-11T08:08:55.359977+0000","last_peered":"2024-09-11T08:08:55.359977+0000","last_clean":"2024-09-11T08:08:55.359977+0000","last_became_active":"2024-09-11T08:07:32.045332+0000","last_became_peered":"2024-09-11T08:07:32.045332+0000","last_unstale":"2024-09-11T08:08:55.359977+0000","last_undegraded":"2024-09-11T08:08:55.359977+0000","last_fullsized":"2024-09-11T08:08:55.359977+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-11T08:08:43.441544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:43.441544+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:55.649394+0000","scrub_duration":0.002168648,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.360438+0000","last_change":"2024-09-11T08:07:30.047446+0000","last_active":"2024-09-11T08:08:55.360438+0000","last_peered":"2024-09-11T08:08:55.360438+0000","last_clean":"2024-09-11T08:08:55.360438+0000","last_became_active":"2024-09-11T08:07:30.047176+0000","last_became_peered":"2024-09-11T08:07:30.047176+0000","last_unstale":"2024-09-11T08:08:55.360438+0000","last_undegraded":"2024-09-11T08:08:55.360438+0000","last_fullsized":"2024-09-11T08:08:55.360438+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":31,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.358854+0000","last_change":"2024-09-11T08:07:34.055816+0000","last_active":"2024-09-11T08:08:55.358854+0000","last_peered":"2024-09-11T08:08:55.358854+0000","last_clean":"2024-09-11T08:08:55.358854+0000","last_became_active":"2024-09-11T08:07:34.055523+0000","last_became_peered":"2024-09-11T08:07:34.055523+0000","last_unstale":"2024-09-11T08:08:55.358854+0000","last_undegraded":"2024-09-11T08:08:55.358854+0000","last_fullsized":"2024-09-11T08:08:55.358854+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":29,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.359373+0000","last_change":"2024-09-11T08:07:36.069535+0000","last_active":"2024-09-11T08:08:55.359373+0000","last_peered":"2024-09-11T08:08:55.359373+0000","last_clean":"2024-09-11T08:08:55.359373+0000","last_became_active":"2024-09-11T08:07:36.069225+0000","last_became_peered":"2024-09-11T08:07:36.069225+0000","last_unstale":"2024-09-11T08:08:55.359373+0000","last_undegraded":"2024-09-11T08:08:55.359373+0000","last_fullsized":"2024-09-11T08:08:55.359373+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":54,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.358983+0000","last_change":"2024-09-11T08:07:32.040334+0000","last_active":"2024-09-11T08:08:55.358983+0000","last_peered":"2024-09-11T08:08:55.358983+0000","last_clean":"2024-09-11T08:08:55.358983+0000","last_became_active":"2024-09-11T08:07:32.040069+0000","last_became_peered":"2024-09-11T08:07:32.040069+0000","last_unstale":"2024-09-11T08:08:55.358983+0000","last_undegraded":"2024-09-11T08:08:55.358983+0000","last_fullsized":"2024-09-11T08:08:55.358983+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.960723+0000","last_change":"2024-09-11T08:09:06.960696+0000","last_active":"2024-09-11T08:09:06.960723+0000","last_peered":"2024-09-11T08:09:06.960723+0000","last_clean":"2024-09-11T08:09:06.960723+0000","last_became_active":"2024-09-11T08:07:30.050484+0000","last_became_peered":"2024-09-11T08:07:30.050484+0000","last_unstale":"2024-09-11T08:09:06.960723+0000","last_undegraded":"2024-09-11T08:09:06.960723+0000","last_fullsized":"2024-09-11T08:09:06.960723+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:06.960639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:09:06.960639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:19.133864+0000","scrub_duration":0.001342766,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"82'9","reported_seq":72,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:03.560188+0000","last_change":"2024-09-11T08:08:53.421634+0000","last_active":"2024-09-11T08:09:03.560188+0000","last_peered":"2024-09-11T08:09:03.560188+0000","last_clean":"2024-09-11T08:09:03.560188+0000","last_became_active":"2024-09-11T08:07:34.056143+0000","last_became_peered":"2024-09-11T08:07:34.056143+0000","last_unstale":"2024-09-11T08:09:03.560188+0000","last_undegraded":"2024-09-11T08:09:03.560188+0000","last_fullsized":"2024-09-11T08:09:03.560188+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-11T08:08:53.421587+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-09-11T08:08:53.421587+0000","last_clean_scrub_stamp":"2024-09-11T08:08:53.421587+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:11.501044+0000","scrub_duration":0.0021116799999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.174126+0000","last_change":"2024-09-11T08:08:55.249190+0000","last_active":"2024-09-11T08:08:56.174126+0000","last_peered":"2024-09-11T08:08:56.174126+0000","last_clean":"2024-09-11T08:08:56.174126+0000","last_became_active":"2024-09-11T08:07:36.633432+0000","last_became_peered":"2024-09-11T08:07:36.633432+0000","last_unstale":"2024-09-11T08:08:56.174126+0000","last_undegraded":"2024-09-11T08:08:56.174126+0000","last_fullsized":"2024-09-11T08:08:56.174126+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:55.249081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:08:55.249081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:00.965734+0000","scrub_duration":0.001581839,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":48,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.011183+0000","last_change":"2024-09-11T08:07:32.054861+0000","last_active":"2024-09-11T08:08:56.011183+0000","last_peered":"2024-09-11T08:08:56.011183+0000","last_clean":"2024-09-11T08:08:56.011183+0000","last_became_active":"2024-09-11T08:07:32.054294+0000","last_became_peered":"2024-09-11T08:07:32.054294+0000","last_unstale":"2024-09-11T08:08:56.011183+0000","last_undegraded":"2024-09-11T08:08:56.011183+0000","last_fullsized":"2024-09-11T08:08:56.011183+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:00.819977+0000","last_change":"2024-09-11T08:09:00.819941+0000","last_active":"2024-09-11T08:09:00.819977+0000","last_peered":"2024-09-11T08:09:00.819977+0000","last_clean":"2024-09-11T08:09:00.819977+0000","last_became_active":"2024-09-11T08:07:30.047497+0000","last_became_peered":"2024-09-11T08:07:30.047497+0000","last_unstale":"2024-09-11T08:09:00.819977+0000","last_undegraded":"2024-09-11T08:09:00.819977+0000","last_fullsized":"2024-09-11T08:09:00.819977+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:00.819884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:09:00.819884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:12.227536+0000","scrub_duration":0.0014673640000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":58,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:03.560153+0000","last_change":"2024-09-11T08:07:34.060999+0000","last_active":"2024-09-11T08:09:03.560153+0000","last_peered":"2024-09-11T08:09:03.560153+0000","last_clean":"2024-09-11T08:09:03.560153+0000","last_became_active":"2024-09-11T08:07:34.060660+0000","last_became_peered":"2024-09-11T08:07:34.060660+0000","last_unstale":"2024-09-11T08:09:03.560153+0000","last_undegraded":"2024-09-11T08:09:03.560153+0000","last_fullsized":"2024-09-11T08:09:03.560153+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"66'1","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.402521+0000","last_change":"2024-09-11T08:07:36.072890+0000","last_active":"2024-09-11T08:08:56.402521+0000","last_peered":"2024-09-11T08:08:56.402521+0000","last_clean":"2024-09-11T08:08:56.402521+0000","last_became_active":"2024-09-11T08:07:36.072593+0000","last_became_peered":"2024-09-11T08:07:36.072593+0000","last_unstale":"2024-09-11T08:08:56.402521+0000","last_undegraded":"2024-09-11T08:08:56.402521+0000","last_fullsized":"2024-09-11T08:08:56.402521+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":79,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.402986+0000","last_change":"2024-09-11T08:08:52.842005+0000","last_active":"2024-09-11T08:08:56.402986+0000","last_peered":"2024-09-11T08:08:56.402986+0000","last_clean":"2024-09-11T08:08:56.402986+0000","last_became_active":"2024-09-11T08:07:32.044376+0000","last_became_peered":"2024-09-11T08:07:32.044376+0000","last_unstale":"2024-09-11T08:08:56.402986+0000","last_undegraded":"2024-09-11T08:08:56.402986+0000","last_fullsized":"2024-09-11T08:08:56.402986+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-11T08:08:52.841903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:52.841903+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:08.737667+0000","scrub_duration":0.0025248150000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.011437+0000","last_change":"2024-09-11T08:08:49.384423+0000","last_active":"2024-09-11T08:08:56.011437+0000","last_peered":"2024-09-11T08:08:56.011437+0000","last_clean":"2024-09-11T08:08:56.011437+0000","last_became_active":"2024-09-11T08:07:30.045380+0000","last_became_peered":"2024-09-11T08:07:30.045380+0000","last_unstale":"2024-09-11T08:08:56.011437+0000","last_undegraded":"2024-09-11T08:08:56.011437+0000","last_fullsized":"2024-09-11T08:08:56.011437+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:49.384357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:49.384357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:50.840709+0000","scrub_duration":0.0012073310000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.402455+0000","last_change":"2024-09-11T08:07:34.051286+0000","last_active":"2024-09-11T08:08:56.402455+0000","last_peered":"2024-09-11T08:08:56.402455+0000","last_clean":"2024-09-11T08:08:56.402455+0000","last_became_active":"2024-09-11T08:07:34.051004+0000","last_became_peered":"2024-09-11T08:07:34.051004+0000","last_unstale":"2024-09-11T08:08:56.402455+0000","last_undegraded":"2024-09-11T08:08:56.402455+0000","last_fullsized":"2024-09-11T08:08:56.402455+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":29,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.010952+0000","last_change":"2024-09-11T08:07:36.632119+0000","last_active":"2024-09-11T08:08:56.010952+0000","last_peered":"2024-09-11T08:08:56.010952+0000","last_clean":"2024-09-11T08:08:56.010952+0000","last_became_active":"2024-09-11T08:07:36.631668+0000","last_became_peered":"2024-09-11T08:07:36.631668+0000","last_unstale":"2024-09-11T08:08:56.010952+0000","last_undegraded":"2024-09-11T08:08:56.010952+0000","last_fullsized":"2024-09-11T08:08:56.010952+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"66'28","reported_seq":100,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:03.560160+0000","last_change":"2024-09-11T08:07:32.053328+0000","last_active":"2024-09-11T08:09:03.560160+0000","last_peered":"2024-09-11T08:09:03.560160+0000","last_clean":"2024-09-11T08:09:03.560160+0000","last_became_active":"2024-09-11T08:07:32.052956+0000","last_became_peered":"2024-09-11T08:07:32.052956+0000","last_unstale":"2024-09-11T08:09:03.560160+0000","last_undegraded":"2024-09-11T08:09:03.560160+0000","last_fullsized":"2024-09-11T08:09:03.560160+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"59'1","reported_seq":54,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.359638+0000","last_change":"2024-09-11T08:08:34.423550+0000","last_active":"2024-09-11T08:08:55.359638+0000","last_peered":"2024-09-11T08:08:55.359638+0000","last_clean":"2024-09-11T08:08:55.359638+0000","last_became_active":"2024-09-11T08:07:30.047513+0000","last_became_peered":"2024-09-11T08:07:30.047513+0000","last_unstale":"2024-09-11T08:08:55.359638+0000","last_undegraded":"2024-09-11T08:08:55.359638+0000","last_fullsized":"2024-09-11T08:08:55.359638+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-11T08:08:34.423474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:34.423474+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:57.535838+0000","scrub_duration":0.0016749149999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":51,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.294761+0000","last_change":"2024-09-11T08:08:37.859401+0000","last_active":"2024-09-11T08:08:56.294761+0000","last_peered":"2024-09-11T08:08:56.294761+0000","last_clean":"2024-09-11T08:08:56.294761+0000","last_became_active":"2024-09-11T08:07:30.043130+0000","last_became_peered":"2024-09-11T08:07:30.043130+0000","last_unstale":"2024-09-11T08:08:56.294761+0000","last_undegraded":"2024-09-11T08:08:56.294761+0000","last_fullsized":"2024-09-11T08:08:56.294761+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-11T08:08:37.859328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:37.859328+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:48.485842+0000","scrub_duration":0.001679816,"objects_trimmed":0,"snaptrim_duration":0.0026120850000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.403121+0000","last_change":"2024-09-11T08:08:46.959561+0000","last_active":"2024-09-11T08:08:56.403121+0000","last_peered":"2024-09-11T08:08:56.403121+0000","last_clean":"2024-09-11T08:08:56.403121+0000","last_became_active":"2024-09-11T08:07:34.057154+0000","last_became_peered":"2024-09-11T08:07:34.057154+0000","last_unstale":"2024-09-11T08:08:56.403121+0000","last_undegraded":"2024-09-11T08:08:56.403121+0000","last_fullsized":"2024-09-11T08:08:56.403121+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:46.959500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:46.959500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:05.161397+0000","scrub_duration":0.001608806,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":29,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.359854+0000","last_change":"2024-09-11T08:07:36.076211+0000","last_active":"2024-09-11T08:08:55.359854+0000","last_peered":"2024-09-11T08:08:55.359854+0000","last_clean":"2024-09-11T08:08:55.359854+0000","last_became_active":"2024-09-11T08:07:36.075690+0000","last_became_peered":"2024-09-11T08:07:36.075690+0000","last_unstale":"2024-09-11T08:08:55.359854+0000","last_undegraded":"2024-09-11T08:08:55.359854+0000","last_fullsized":"2024-09-11T08:08:55.359854+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":70,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.011646+0000","last_change":"2024-09-11T08:08:35.441946+0000","last_active":"2024-09-11T08:08:56.011646+0000","last_peered":"2024-09-11T08:08:56.011646+0000","last_clean":"2024-09-11T08:08:56.011646+0000","last_became_active":"2024-09-11T08:07:32.050673+0000","last_became_peered":"2024-09-11T08:07:32.050673+0000","last_unstale":"2024-09-11T08:08:56.011646+0000","last_undegraded":"2024-09-11T08:08:56.011646+0000","last_fullsized":"2024-09-11T08:08:56.011646+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'13","last_scrub_stamp":"2024-09-11T08:08:35.441894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:35.441894+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:02.914423+0000","scrub_duration":0.001888965,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.011256+0000","last_change":"2024-09-11T08:08:33.492709+0000","last_active":"2024-09-11T08:08:56.011256+0000","last_peered":"2024-09-11T08:08:56.011256+0000","last_clean":"2024-09-11T08:08:56.011256+0000","last_became_active":"2024-09-11T08:07:30.047014+0000","last_became_peered":"2024-09-11T08:07:30.047014+0000","last_unstale":"2024-09-11T08:08:56.011256+0000","last_undegraded":"2024-09-11T08:08:56.011256+0000","last_fullsized":"2024-09-11T08:08:56.011256+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:33.492652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:33.492652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:54.786224+0000","scrub_duration":0.00156573,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":50,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.359069+0000","last_change":"2024-09-11T08:08:32.909292+0000","last_active":"2024-09-11T08:08:55.359069+0000","last_peered":"2024-09-11T08:08:55.359069+0000","last_clean":"2024-09-11T08:08:55.359069+0000","last_became_active":"2024-09-11T08:07:30.045009+0000","last_became_peered":"2024-09-11T08:07:30.045009+0000","last_unstale":"2024-09-11T08:08:55.359069+0000","last_undegraded":"2024-09-11T08:08:55.359069+0000","last_fullsized":"2024-09-11T08:08:55.359069+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-11T08:08:32.909183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:32.909183+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:02.458449+0000","scrub_duration":0.001625769,"objects_trimmed":0,"snaptrim_duration":0.001859533,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:58.000043+0000","last_change":"2024-09-11T08:08:58.000006+0000","last_active":"2024-09-11T08:08:58.000043+0000","last_peered":"2024-09-11T08:08:58.000043+0000","last_clean":"2024-09-11T08:08:58.000043+0000","last_became_active":"2024-09-11T08:07:34.063906+0000","last_became_peered":"2024-09-11T08:07:34.063906+0000","last_unstale":"2024-09-11T08:08:58.000043+0000","last_undegraded":"2024-09-11T08:08:58.000043+0000","last_fullsized":"2024-09-11T08:08:58.000043+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:57.999952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:57.999952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:01.053836+0000","scrub_duration":0.0014942130000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.174225+0000","last_change":"2024-09-11T08:08:46.366578+0000","last_active":"2024-09-11T08:08:56.174225+0000","last_peered":"2024-09-11T08:08:56.174225+0000","last_clean":"2024-09-11T08:08:56.174225+0000","last_became_active":"2024-09-11T08:07:36.631480+0000","last_became_peered":"2024-09-11T08:07:36.631480+0000","last_unstale":"2024-09-11T08:08:56.174225+0000","last_undegraded":"2024-09-11T08:08:56.174225+0000","last_fullsized":"2024-09-11T08:08:56.174225+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:46.366508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:08:46.366508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:52.248237+0000","scrub_duration":0.0015317200000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":61,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.403259+0000","last_change":"2024-09-11T08:08:37.115649+0000","last_active":"2024-09-11T08:08:56.403259+0000","last_peered":"2024-09-11T08:08:56.403259+0000","last_clean":"2024-09-11T08:08:56.403259+0000","last_became_active":"2024-09-11T08:07:32.055407+0000","last_became_peered":"2024-09-11T08:07:32.055407+0000","last_unstale":"2024-09-11T08:08:56.403259+0000","last_undegraded":"2024-09-11T08:08:56.403259+0000","last_fullsized":"2024-09-11T08:08:56.403259+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-11T08:08:37.115582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:37.115582+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:57.869478+0000","scrub_duration":0.0020267729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"59'1","reported_seq":52,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.402768+0000","last_change":"2024-09-11T08:08:41.094879+0000","last_active":"2024-09-11T08:08:56.402768+0000","last_peered":"2024-09-11T08:08:56.402768+0000","last_clean":"2024-09-11T08:08:56.402768+0000","last_became_active":"2024-09-11T08:07:30.050504+0000","last_became_peered":"2024-09-11T08:07:30.050504+0000","last_unstale":"2024-09-11T08:08:56.402768+0000","last_undegraded":"2024-09-11T08:08:56.402768+0000","last_fullsized":"2024-09-11T08:08:56.402768+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-11T08:08:41.094821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:41.094821+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:00.561470+0000","scrub_duration":0.0017674310000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":144,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.870958+0000","last_change":"2024-09-11T08:09:01.175592+0000","last_active":"2024-09-11T08:09:06.870958+0000","last_peered":"2024-09-11T08:09:06.870958+0000","last_clean":"2024-09-11T08:09:06.870958+0000","last_became_active":"2024-09-11T08:07:30.045051+0000","last_became_peered":"2024-09-11T08:07:30.045051+0000","last_unstale":"2024-09-11T08:09:06.870958+0000","last_undegraded":"2024-09-11T08:09:06.870958+0000","last_fullsized":"2024-09-11T08:09:06.870958+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-11T08:09:01.175530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:09:01.175530+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:30.186314+0000","scrub_duration":0.001752651,"objects_trimmed":0,"snaptrim_duration":0.000318821,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":91,"num_read_kb":86,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:05.758231+0000","last_change":"2024-09-11T08:09:05.758196+0000","last_active":"2024-09-11T08:09:05.758231+0000","last_peered":"2024-09-11T08:09:05.758231+0000","last_clean":"2024-09-11T08:09:05.758231+0000","last_became_active":"2024-09-11T08:07:34.058962+0000","last_became_peered":"2024-09-11T08:07:34.058962+0000","last_unstale":"2024-09-11T08:09:05.758231+0000","last_undegraded":"2024-09-11T08:09:05.758231+0000","last_fullsized":"2024-09-11T08:09:05.758231+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:05.758141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:09:05.758141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:09.559173+0000","scrub_duration":0.0012345450000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.011166+0000","last_change":"2024-09-11T08:08:50.346126+0000","last_active":"2024-09-11T08:08:56.011166+0000","last_peered":"2024-09-11T08:08:56.011166+0000","last_clean":"2024-09-11T08:08:56.011166+0000","last_became_active":"2024-09-11T08:07:36.077597+0000","last_became_peered":"2024-09-11T08:07:36.077597+0000","last_unstale":"2024-09-11T08:08:56.011166+0000","last_undegraded":"2024-09-11T08:08:56.011166+0000","last_fullsized":"2024-09-11T08:08:56.011166+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:50.346069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:08:50.346069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:15.900195+0000","scrub_duration":0.001294906,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"84'21","reported_seq":71,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.705429+0000","last_change":"2024-09-11T08:08:56.705362+0000","last_active":"2024-09-11T08:08:56.705429+0000","last_peered":"2024-09-11T08:08:56.705429+0000","last_clean":"2024-09-11T08:08:56.705429+0000","last_became_active":"2024-09-11T08:07:32.042339+0000","last_became_peered":"2024-09-11T08:07:32.042339+0000","last_unstale":"2024-09-11T08:08:56.705429+0000","last_undegraded":"2024-09-11T08:08:56.705429+0000","last_fullsized":"2024-09-11T08:08:56.705429+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'21","last_scrub_stamp":"2024-09-11T08:08:56.705306+0000","last_deep_scrub":"84'21","last_deep_scrub_stamp":"2024-09-11T08:08:56.705306+0000","last_clean_scrub_stamp":"2024-09-11T08:08:56.705306+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:14.758337+0000","scrub_duration":0.0050944290000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.360355+0000","last_change":"2024-09-11T08:07:30.047535+0000","last_active":"2024-09-11T08:08:55.360355+0000","last_peered":"2024-09-11T08:08:55.360355+0000","last_clean":"2024-09-11T08:08:55.360355+0000","last_became_active":"2024-09-11T08:07:30.047246+0000","last_became_peered":"2024-09-11T08:07:30.047246+0000","last_unstale":"2024-09-11T08:08:55.360355+0000","last_undegraded":"2024-09-11T08:08:55.360355+0000","last_fullsized":"2024-09-11T08:08:55.360355+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":81,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.174050+0000","last_change":"2024-09-11T08:07:44.054743+0000","last_active":"2024-09-11T08:08:56.174050+0000","last_peered":"2024-09-11T08:08:56.174050+0000","last_clean":"2024-09-11T08:08:56.174050+0000","last_became_active":"2024-09-11T08:07:19.726378+0000","last_became_peered":"2024-09-11T08:07:19.726378+0000","last_unstale":"2024-09-11T08:08:56.174050+0000","last_undegraded":"2024-09-11T08:08:56.174050+0000","last_fullsized":"2024-09-11T08:08:56.174050+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'72","last_scrub_stamp":"2024-09-11T08:07:44.054686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:05:21.231856+0000","last_clean_scrub_stamp":"2024-09-11T08:07:44.054686+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001872931,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.174294+0000","last_change":"2024-09-11T08:08:51.289948+0000","last_active":"2024-09-11T08:08:56.174294+0000","last_peered":"2024-09-11T08:08:56.174294+0000","last_clean":"2024-09-11T08:08:56.174294+0000","last_became_active":"2024-09-11T08:07:34.058414+0000","last_became_peered":"2024-09-11T08:07:34.058414+0000","last_unstale":"2024-09-11T08:08:56.174294+0000","last_undegraded":"2024-09-11T08:08:56.174294+0000","last_fullsized":"2024-09-11T08:08:56.174294+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:51.289872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:51.289872+0000","last_clean_scrub_stamp":"2024-09-11T08:08:51.289872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:08.912691+0000","scrub_duration":0.001988812,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":38,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:01.777595+0000","last_change":"2024-09-11T08:09:01.777522+0000","last_active":"2024-09-11T08:09:01.777595+0000","last_peered":"2024-09-11T08:09:01.777595+0000","last_clean":"2024-09-11T08:09:01.777595+0000","last_became_active":"2024-09-11T08:07:36.072508+0000","last_became_peered":"2024-09-11T08:07:36.072508+0000","last_unstale":"2024-09-11T08:09:01.777595+0000","last_undegraded":"2024-09-11T08:09:01.777595+0000","last_fullsized":"2024-09-11T08:09:01.777595+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:01.777429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:01.777429+0000","last_clean_scrub_stamp":"2024-09-11T08:09:01.777429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:07.815841+0000","scrub_duration":0.001444961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"66'11","reported_seq":62,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.359357+0000","last_change":"2024-09-11T08:08:35.418440+0000","last_active":"2024-09-11T08:08:55.359357+0000","last_peered":"2024-09-11T08:08:55.359357+0000","last_clean":"2024-09-11T08:08:55.359357+0000","last_became_active":"2024-09-11T08:07:32.047971+0000","last_became_peered":"2024-09-11T08:07:32.047971+0000","last_unstale":"2024-09-11T08:08:55.359357+0000","last_undegraded":"2024-09-11T08:08:55.359357+0000","last_fullsized":"2024-09-11T08:08:55.359357+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-11T08:08:35.418378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:35.418378+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:38.165860+0000","scrub_duration":0.001791207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.359103+0000","last_change":"2024-09-11T08:08:36.417975+0000","last_active":"2024-09-11T08:08:55.359103+0000","last_peered":"2024-09-11T08:08:55.359103+0000","last_clean":"2024-09-11T08:08:55.359103+0000","last_became_active":"2024-09-11T08:07:30.039884+0000","last_became_peered":"2024-09-11T08:07:30.039884+0000","last_unstale":"2024-09-11T08:08:55.359103+0000","last_undegraded":"2024-09-11T08:08:55.359103+0000","last_fullsized":"2024-09-11T08:08:55.359103+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:36.417911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:36.417911+0000","last_clean_scrub_stamp":"2024-09-11T08:08:36.417911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:59.734565+0000","scrub_duration":0.0014387580000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:58.845189+0000","last_change":"2024-09-11T08:08:58.845142+0000","last_active":"2024-09-11T08:08:58.845189+0000","last_peered":"2024-09-11T08:08:58.845189+0000","last_clean":"2024-09-11T08:08:58.845189+0000","last_became_active":"2024-09-11T08:07:34.053347+0000","last_became_peered":"2024-09-11T08:07:34.053347+0000","last_unstale":"2024-09-11T08:08:58.845189+0000","last_undegraded":"2024-09-11T08:08:58.845189+0000","last_fullsized":"2024-09-11T08:08:58.845189+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:58.845085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:58.845085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:08.089077+0000","scrub_duration":0.001233404,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":29,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.360446+0000","last_change":"2024-09-11T08:07:36.632329+0000","last_active":"2024-09-11T08:08:55.360446+0000","last_peered":"2024-09-11T08:08:55.360446+0000","last_clean":"2024-09-11T08:08:55.360446+0000","last_became_active":"2024-09-11T08:07:36.631687+0000","last_became_peered":"2024-09-11T08:07:36.631687+0000","last_unstale":"2024-09-11T08:08:55.360446+0000","last_undegraded":"2024-09-11T08:08:55.360446+0000","last_fullsized":"2024-09-11T08:08:55.360446+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"77'25","reported_seq":81,"reported_epoch":85,"state":"peering","last_fresh":"2024-09-11T08:09:06.872554+0000","last_change":"2024-09-11T08:09:06.872760+0000","last_active":"2024-09-11T08:09:06.872524+0000","last_peered":"2024-09-11T08:08:55.359257+0000","last_clean":"2024-09-11T08:08:55.359257+0000","last_became_active":"2024-09-11T08:07:32.050815+0000","last_became_peered":"2024-09-11T08:07:32.050815+0000","last_unstale":"2024-09-11T08:09:06.872554+0000","last_undegraded":"2024-09-11T08:09:06.872554+0000","last_fullsized":"2024-09-11T08:09:06.872554+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'25","last_scrub_stamp":"2024-09-11T08:08:32.458210+0000","last_deep_scrub":"77'25","last_deep_scrub_stamp":"2024-09-11T08:08:32.458210+0000","last_clean_scrub_stamp":"2024-09-11T08:08:32.458210+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:53.685411+0000","scrub_duration":0.0060523319999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,5],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":34,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.402849+0000","last_change":"2024-09-11T08:07:30.042789+0000","last_active":"2024-09-11T08:08:56.402849+0000","last_peered":"2024-09-11T08:08:56.402849+0000","last_clean":"2024-09-11T08:08:56.402849+0000","last_became_active":"2024-09-11T08:07:30.042555+0000","last_became_peered":"2024-09-11T08:07:30.042555+0000","last_unstale":"2024-09-11T08:08:56.402849+0000","last_undegraded":"2024-09-11T08:08:56.402849+0000","last_fullsized":"2024-09-11T08:08:56.402849+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.011250+0000","last_change":"2024-09-11T08:08:41.437100+0000","last_active":"2024-09-11T08:08:56.011250+0000","last_peered":"2024-09-11T08:08:56.011250+0000","last_clean":"2024-09-11T08:08:56.011250+0000","last_became_active":"2024-09-11T08:07:34.054961+0000","last_became_peered":"2024-09-11T08:07:34.054961+0000","last_unstale":"2024-09-11T08:08:56.011250+0000","last_undegraded":"2024-09-11T08:08:56.011250+0000","last_fullsized":"2024-09-11T08:08:56.011250+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:41.437045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:41.437045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:10.512019+0000","scrub_duration":0.0029926649999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":38,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:04.031993+0000","last_change":"2024-09-11T08:09:04.031954+0000","last_active":"2024-09-11T08:09:04.031993+0000","last_peered":"2024-09-11T08:09:04.031993+0000","last_clean":"2024-09-11T08:09:04.031993+0000","last_became_active":"2024-09-11T08:07:36.071194+0000","last_became_peered":"2024-09-11T08:07:36.071194+0000","last_unstale":"2024-09-11T08:09:04.031993+0000","last_undegraded":"2024-09-11T08:09:04.031993+0000","last_fullsized":"2024-09-11T08:09:04.031993+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:04.031894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:09:04.031894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:28.971103+0000","scrub_duration":0.0013516579999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":58,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.359082+0000","last_change":"2024-09-11T08:08:42.447971+0000","last_active":"2024-09-11T08:08:55.359082+0000","last_peered":"2024-09-11T08:08:55.359082+0000","last_clean":"2024-09-11T08:08:55.359082+0000","last_became_active":"2024-09-11T08:07:32.043314+0000","last_became_peered":"2024-09-11T08:07:32.043314+0000","last_unstale":"2024-09-11T08:08:55.359082+0000","last_undegraded":"2024-09-11T08:08:55.359082+0000","last_fullsized":"2024-09-11T08:08:55.359082+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-11T08:08:42.447892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:42.447892+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:03.778832+0000","scrub_duration":0.0039015859999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.359949+0000","last_change":"2024-09-11T08:08:30.327580+0000","last_active":"2024-09-11T08:08:55.359949+0000","last_peered":"2024-09-11T08:08:55.359949+0000","last_clean":"2024-09-11T08:08:55.359949+0000","last_became_active":"2024-09-11T08:07:30.046684+0000","last_became_peered":"2024-09-11T08:07:30.046684+0000","last_unstale":"2024-09-11T08:08:55.359949+0000","last_undegraded":"2024-09-11T08:08:55.359949+0000","last_fullsized":"2024-09-11T08:08:55.359949+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:30.327525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:30.327525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:57.278810+0000","scrub_duration":0.001475589,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"66'8","reported_seq":67,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:03.560198+0000","last_change":"2024-09-11T08:08:49.026859+0000","last_active":"2024-09-11T08:09:03.560198+0000","last_peered":"2024-09-11T08:09:03.560198+0000","last_clean":"2024-09-11T08:09:03.560198+0000","last_became_active":"2024-09-11T08:07:34.062469+0000","last_became_peered":"2024-09-11T08:07:34.062469+0000","last_unstale":"2024-09-11T08:09:03.560198+0000","last_undegraded":"2024-09-11T08:09:03.560198+0000","last_fullsized":"2024-09-11T08:09:03.560198+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:08:49.026755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:49.026755+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:59.460486+0000","scrub_duration":0.0018077970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":29,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.359109+0000","last_change":"2024-09-11T08:07:36.074961+0000","last_active":"2024-09-11T08:08:55.359109+0000","last_peered":"2024-09-11T08:08:55.359109+0000","last_clean":"2024-09-11T08:08:55.359109+0000","last_became_active":"2024-09-11T08:07:36.074684+0000","last_became_peered":"2024-09-11T08:07:36.074684+0000","last_unstale":"2024-09-11T08:08:55.359109+0000","last_undegraded":"2024-09-11T08:08:55.359109+0000","last_fullsized":"2024-09-11T08:08:55.359109+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":49,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.403006+0000","last_change":"2024-09-11T08:07:32.055237+0000","last_active":"2024-09-11T08:08:56.403006+0000","last_peered":"2024-09-11T08:08:56.403006+0000","last_clean":"2024-09-11T08:08:56.403006+0000","last_became_active":"2024-09-11T08:07:32.054885+0000","last_became_peered":"2024-09-11T08:07:32.054885+0000","last_unstale":"2024-09-11T08:08:56.403006+0000","last_undegraded":"2024-09-11T08:08:56.403006+0000","last_fullsized":"2024-09-11T08:08:56.403006+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.295274+0000","last_change":"2024-09-11T08:08:34.828231+0000","last_active":"2024-09-11T08:08:56.295274+0000","last_peered":"2024-09-11T08:08:56.295274+0000","last_clean":"2024-09-11T08:08:56.295274+0000","last_became_active":"2024-09-11T08:07:30.045613+0000","last_became_peered":"2024-09-11T08:07:30.045613+0000","last_unstale":"2024-09-11T08:08:56.295274+0000","last_undegraded":"2024-09-11T08:08:56.295274+0000","last_fullsized":"2024-09-11T08:08:56.295274+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:34.828129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:34.828129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:58.397313+0000","scrub_duration":0.0013782530000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.359164+0000","last_change":"2024-09-11T08:08:37.871017+0000","last_active":"2024-09-11T08:08:55.359164+0000","last_peered":"2024-09-11T08:08:55.359164+0000","last_clean":"2024-09-11T08:08:55.359164+0000","last_became_active":"2024-09-11T08:07:34.056606+0000","last_became_peered":"2024-09-11T08:07:34.056606+0000","last_unstale":"2024-09-11T08:08:55.359164+0000","last_undegraded":"2024-09-11T08:08:55.359164+0000","last_fullsized":"2024-09-11T08:08:55.359164+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:37.870955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:37.870955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:58.505876+0000","scrub_duration":0.0016274460000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:05.932313+0000","last_change":"2024-09-11T08:09:05.932261+0000","last_active":"2024-09-11T08:09:05.932313+0000","last_peered":"2024-09-11T08:09:05.932313+0000","last_clean":"2024-09-11T08:09:05.932313+0000","last_became_active":"2024-09-11T08:07:36.072452+0000","last_became_peered":"2024-09-11T08:07:36.072452+0000","last_unstale":"2024-09-11T08:09:05.932313+0000","last_undegraded":"2024-09-11T08:09:05.932313+0000","last_fullsized":"2024-09-11T08:09:05.932313+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:05.932180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:09:05.932180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:07.863587+0000","scrub_duration":0.0013481960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":73,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.402907+0000","last_change":"2024-09-11T08:08:53.616543+0000","last_active":"2024-09-11T08:08:56.402907+0000","last_peered":"2024-09-11T08:08:56.402907+0000","last_clean":"2024-09-11T08:08:56.402907+0000","last_became_active":"2024-09-11T08:07:32.050880+0000","last_became_peered":"2024-09-11T08:07:32.050880+0000","last_unstale":"2024-09-11T08:08:56.402907+0000","last_undegraded":"2024-09-11T08:08:56.402907+0000","last_fullsized":"2024-09-11T08:08:56.402907+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-11T08:08:53.616450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:53.616450+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:18.217121+0000","scrub_duration":0.0024100559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.174487+0000","last_change":"2024-09-11T08:08:44.385440+0000","last_active":"2024-09-11T08:08:56.174487+0000","last_peered":"2024-09-11T08:08:56.174487+0000","last_clean":"2024-09-11T08:08:56.174487+0000","last_became_active":"2024-09-11T08:07:30.043294+0000","last_became_peered":"2024-09-11T08:07:30.043294+0000","last_unstale":"2024-09-11T08:08:56.174487+0000","last_undegraded":"2024-09-11T08:08:56.174487+0000","last_fullsized":"2024-09-11T08:08:56.174487+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:44.385372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:44.385372+0000","last_clean_scrub_stamp":"2024-09-11T08:08:44.385372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:46.537624+0000","scrub_duration":0.001441007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.359174+0000","last_change":"2024-09-11T08:08:51.921479+0000","last_active":"2024-09-11T08:08:55.359174+0000","last_peered":"2024-09-11T08:08:55.359174+0000","last_clean":"2024-09-11T08:08:55.359174+0000","last_became_active":"2024-09-11T08:07:34.063589+0000","last_became_peered":"2024-09-11T08:07:34.063589+0000","last_unstale":"2024-09-11T08:08:55.359174+0000","last_undegraded":"2024-09-11T08:08:55.359174+0000","last_fullsized":"2024-09-11T08:08:55.359174+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:51.921389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:51.921389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:01.266430+0000","scrub_duration":0.001807091,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.173568+0000","last_change":"2024-09-11T08:08:36.367954+0000","last_active":"2024-09-11T08:08:56.173568+0000","last_peered":"2024-09-11T08:08:56.173568+0000","last_clean":"2024-09-11T08:08:56.173568+0000","last_became_active":"2024-09-11T08:07:36.077523+0000","last_became_peered":"2024-09-11T08:07:36.077523+0000","last_unstale":"2024-09-11T08:08:56.173568+0000","last_undegraded":"2024-09-11T08:08:56.173568+0000","last_fullsized":"2024-09-11T08:08:56.173568+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:36.367890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:08:36.367890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:40.081903+0000","scrub_duration":0.0015112560000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":66,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:59.494109+0000","last_change":"2024-09-11T08:08:59.494039+0000","last_active":"2024-09-11T08:08:59.494109+0000","last_peered":"2024-09-11T08:08:59.494109+0000","last_clean":"2024-09-11T08:08:59.494109+0000","last_became_active":"2024-09-11T08:07:32.048164+0000","last_became_peered":"2024-09-11T08:07:32.048164+0000","last_unstale":"2024-09-11T08:08:59.494109+0000","last_undegraded":"2024-09-11T08:08:59.494109+0000","last_fullsized":"2024-09-11T08:08:59.494109+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-11T08:08:59.493975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:59.493975+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:17.528763+0000","scrub_duration":0.0019438280000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.174662+0000","last_change":"2024-09-11T08:08:53.247349+0000","last_active":"2024-09-11T08:08:56.174662+0000","last_peered":"2024-09-11T08:08:56.174662+0000","last_clean":"2024-09-11T08:08:56.174662+0000","last_became_active":"2024-09-11T08:07:30.041219+0000","last_became_peered":"2024-09-11T08:07:30.041219+0000","last_unstale":"2024-09-11T08:08:56.174662+0000","last_undegraded":"2024-09-11T08:08:56.174662+0000","last_fullsized":"2024-09-11T08:08:56.174662+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:53.247275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:53.247275+0000","last_clean_scrub_stamp":"2024-09-11T08:08:53.247275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:00.288102+0000","scrub_duration":0.0012568169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.359011+0000","last_change":"2024-09-11T08:08:46.970593+0000","last_active":"2024-09-11T08:08:55.359011+0000","last_peered":"2024-09-11T08:08:55.359011+0000","last_clean":"2024-09-11T08:08:55.359011+0000","last_became_active":"2024-09-11T08:07:34.057479+0000","last_became_peered":"2024-09-11T08:07:34.057479+0000","last_unstale":"2024-09-11T08:08:55.359011+0000","last_undegraded":"2024-09-11T08:08:55.359011+0000","last_fullsized":"2024-09-11T08:08:55.359011+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:46.970539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:46.970539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:09.551366+0000","scrub_duration":0.001167143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:00.395407+0000","last_change":"2024-09-11T08:09:00.395353+0000","last_active":"2024-09-11T08:09:00.395407+0000","last_peered":"2024-09-11T08:09:00.395407+0000","last_clean":"2024-09-11T08:09:00.395407+0000","last_became_active":"2024-09-11T08:07:36.078075+0000","last_became_peered":"2024-09-11T08:07:36.078075+0000","last_unstale":"2024-09-11T08:09:00.395407+0000","last_undegraded":"2024-09-11T08:09:00.395407+0000","last_fullsized":"2024-09-11T08:09:00.395407+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:00.395264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:09:00.395264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:07.333876+0000","scrub_duration":0.001470147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":57,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.294658+0000","last_change":"2024-09-11T08:07:32.039555+0000","last_active":"2024-09-11T08:08:56.294658+0000","last_peered":"2024-09-11T08:08:56.294658+0000","last_clean":"2024-09-11T08:08:56.294658+0000","last_became_active":"2024-09-11T08:07:32.039275+0000","last_became_peered":"2024-09-11T08:07:32.039275+0000","last_unstale":"2024-09-11T08:08:56.294658+0000","last_undegraded":"2024-09-11T08:08:56.294658+0000","last_fullsized":"2024-09-11T08:08:56.294658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":52,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.174704+0000","last_change":"2024-09-11T08:07:30.046228+0000","last_active":"2024-09-11T08:08:56.174704+0000","last_peered":"2024-09-11T08:08:56.174704+0000","last_clean":"2024-09-11T08:08:56.174704+0000","last_became_active":"2024-09-11T08:07:30.045967+0000","last_became_peered":"2024-09-11T08:07:30.045967+0000","last_unstale":"2024-09-11T08:08:56.174704+0000","last_undegraded":"2024-09-11T08:08:56.174704+0000","last_fullsized":"2024-09-11T08:08:56.174704+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"66'8","reported_seq":68,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:03.560270+0000","last_change":"2024-09-11T08:08:59.170974+0000","last_active":"2024-09-11T08:09:03.560270+0000","last_peered":"2024-09-11T08:09:03.560270+0000","last_clean":"2024-09-11T08:09:03.560270+0000","last_became_active":"2024-09-11T08:07:34.058015+0000","last_became_peered":"2024-09-11T08:07:34.058015+0000","last_unstale":"2024-09-11T08:09:03.560270+0000","last_undegraded":"2024-09-11T08:09:03.560270+0000","last_fullsized":"2024-09-11T08:09:03.560270+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:08:59.170918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:59.170918+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:12.511014+0000","scrub_duration":0.001468128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":28,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.294597+0000","last_change":"2024-09-11T08:07:36.632018+0000","last_active":"2024-09-11T08:08:56.294597+0000","last_peered":"2024-09-11T08:08:56.294597+0000","last_clean":"2024-09-11T08:08:56.294597+0000","last_became_active":"2024-09-11T08:07:36.631755+0000","last_became_peered":"2024-09-11T08:07:36.631755+0000","last_unstale":"2024-09-11T08:08:56.294597+0000","last_undegraded":"2024-09-11T08:08:56.294597+0000","last_fullsized":"2024-09-11T08:08:56.294597+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":34,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.402423+0000","last_change":"2024-09-11T08:07:30.046145+0000","last_active":"2024-09-11T08:08:56.402423+0000","last_peered":"2024-09-11T08:08:56.402423+0000","last_clean":"2024-09-11T08:08:56.402423+0000","last_became_active":"2024-09-11T08:07:30.045910+0000","last_became_peered":"2024-09-11T08:07:30.045910+0000","last_unstale":"2024-09-11T08:08:56.402423+0000","last_undegraded":"2024-09-11T08:08:56.402423+0000","last_fullsized":"2024-09-11T08:08:56.402423+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"84'9","reported_seq":55,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:03.418572+0000","last_change":"2024-09-11T08:09:03.418515+0000","last_active":"2024-09-11T08:09:03.418572+0000","last_peered":"2024-09-11T08:09:03.418572+0000","last_clean":"2024-09-11T08:09:03.418572+0000","last_became_active":"2024-09-11T08:07:32.046891+0000","last_became_peered":"2024-09-11T08:07:32.046891+0000","last_unstale":"2024-09-11T08:09:03.418572+0000","last_undegraded":"2024-09-11T08:09:03.418572+0000","last_fullsized":"2024-09-11T08:09:03.418572+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-09-11T08:09:03.418457+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-09-11T08:09:03.418457+0000","last_clean_scrub_stamp":"2024-09-11T08:09:03.418457+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:10.688358+0000","scrub_duration":0.003076058,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.295114+0000","last_change":"2024-09-11T08:07:34.058334+0000","last_active":"2024-09-11T08:08:56.295114+0000","last_peered":"2024-09-11T08:08:56.295114+0000","last_clean":"2024-09-11T08:08:56.295114+0000","last_became_active":"2024-09-11T08:07:34.057774+0000","last_became_peered":"2024-09-11T08:07:34.057774+0000","last_unstale":"2024-09-11T08:08:56.295114+0000","last_undegraded":"2024-09-11T08:08:56.295114+0000","last_fullsized":"2024-09-11T08:08:56.295114+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.174229+0000","last_change":"2024-09-11T08:08:38.326345+0000","last_active":"2024-09-11T08:08:56.174229+0000","last_peered":"2024-09-11T08:08:56.174229+0000","last_clean":"2024-09-11T08:08:56.174229+0000","last_became_active":"2024-09-11T08:07:36.633244+0000","last_became_peered":"2024-09-11T08:07:36.633244+0000","last_unstale":"2024-09-11T08:08:56.174229+0000","last_undegraded":"2024-09-11T08:08:56.174229+0000","last_fullsized":"2024-09-11T08:08:56.174229+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:38.326265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:08:38.326265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:06.322672+0000","scrub_duration":0.001417361,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"84'13","reported_seq":63,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:57.835740+0000","last_change":"2024-09-11T08:08:57.835686+0000","last_active":"2024-09-11T08:08:57.835740+0000","last_peered":"2024-09-11T08:08:57.835740+0000","last_clean":"2024-09-11T08:08:57.835740+0000","last_became_active":"2024-09-11T08:07:32.048350+0000","last_became_peered":"2024-09-11T08:07:32.048350+0000","last_unstale":"2024-09-11T08:08:57.835740+0000","last_undegraded":"2024-09-11T08:08:57.835740+0000","last_fullsized":"2024-09-11T08:08:57.835740+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'13","last_scrub_stamp":"2024-09-11T08:08:57.835634+0000","last_deep_scrub":"84'13","last_deep_scrub_stamp":"2024-09-11T08:08:57.835634+0000","last_clean_scrub_stamp":"2024-09-11T08:08:57.835634+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:18.729258+0000","scrub_duration":0.0033293229999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.174130+0000","last_change":"2024-09-11T08:08:30.471690+0000","last_active":"2024-09-11T08:08:56.174130+0000","last_peered":"2024-09-11T08:08:56.174130+0000","last_clean":"2024-09-11T08:08:56.174130+0000","last_became_active":"2024-09-11T08:07:30.044055+0000","last_became_peered":"2024-09-11T08:07:30.044055+0000","last_unstale":"2024-09-11T08:08:56.174130+0000","last_undegraded":"2024-09-11T08:08:56.174130+0000","last_fullsized":"2024-09-11T08:08:56.174130+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:30.471620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:30.471620+0000","last_clean_scrub_stamp":"2024-09-11T08:08:30.471620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:37.891558+0000","scrub_duration":0.0016049980000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:04.370859+0000","last_change":"2024-09-11T08:09:04.370800+0000","last_active":"2024-09-11T08:09:04.370859+0000","last_peered":"2024-09-11T08:09:04.370859+0000","last_clean":"2024-09-11T08:09:04.370859+0000","last_became_active":"2024-09-11T08:07:34.057127+0000","last_became_peered":"2024-09-11T08:07:34.057127+0000","last_unstale":"2024-09-11T08:09:04.370859+0000","last_undegraded":"2024-09-11T08:09:04.370859+0000","last_fullsized":"2024-09-11T08:09:04.370859+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:04.370734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:04.370734+0000","last_clean_scrub_stamp":"2024-09-11T08:09:04.370734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:09.164201+0000","scrub_duration":0.001224483,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":38,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:01.995150+0000","last_change":"2024-09-11T08:09:01.995115+0000","last_active":"2024-09-11T08:09:01.995150+0000","last_peered":"2024-09-11T08:09:01.995150+0000","last_clean":"2024-09-11T08:09:01.995150+0000","last_became_active":"2024-09-11T08:07:36.071513+0000","last_became_peered":"2024-09-11T08:07:36.071513+0000","last_unstale":"2024-09-11T08:09:01.995150+0000","last_undegraded":"2024-09-11T08:09:01.995150+0000","last_fullsized":"2024-09-11T08:09:01.995150+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:01.995061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:09:01.995061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:18.017918+0000","scrub_duration":0.00115722,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":58,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.295415+0000","last_change":"2024-09-11T08:08:49.951403+0000","last_active":"2024-09-11T08:08:56.295415+0000","last_peered":"2024-09-11T08:08:56.295415+0000","last_clean":"2024-09-11T08:08:56.295415+0000","last_became_active":"2024-09-11T08:07:32.041761+0000","last_became_peered":"2024-09-11T08:07:32.041761+0000","last_unstale":"2024-09-11T08:08:56.295415+0000","last_undegraded":"2024-09-11T08:08:56.295415+0000","last_fullsized":"2024-09-11T08:08:56.295415+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:08:49.951345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:49.951345+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:51.233235+0000","scrub_duration":0.002079914,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.402831+0000","last_change":"2024-09-11T08:08:48.935452+0000","last_active":"2024-09-11T08:08:56.402831+0000","last_peered":"2024-09-11T08:08:56.402831+0000","last_clean":"2024-09-11T08:08:56.402831+0000","last_became_active":"2024-09-11T08:07:30.045380+0000","last_became_peered":"2024-09-11T08:07:30.045380+0000","last_unstale":"2024-09-11T08:08:56.402831+0000","last_undegraded":"2024-09-11T08:08:56.402831+0000","last_fullsized":"2024-09-11T08:08:56.402831+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:48.935387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:48.935387+0000","last_clean_scrub_stamp":"2024-09-11T08:08:48.935387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:05.810526+0000","scrub_duration":0.001422134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"66'8","reported_seq":68,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:03.560375+0000","last_change":"2024-09-11T08:08:52.374593+0000","last_active":"2024-09-11T08:09:03.560375+0000","last_peered":"2024-09-11T08:09:03.560375+0000","last_clean":"2024-09-11T08:09:03.560375+0000","last_became_active":"2024-09-11T08:07:34.057937+0000","last_became_peered":"2024-09-11T08:07:34.057937+0000","last_unstale":"2024-09-11T08:09:03.560375+0000","last_undegraded":"2024-09-11T08:09:03.560375+0000","last_fullsized":"2024-09-11T08:09:03.560375+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:08:52.374539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:52.374539+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:12.088877+0000","scrub_duration":0.00146886,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":29,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.359685+0000","last_change":"2024-09-11T08:07:36.068798+0000","last_active":"2024-09-11T08:08:55.359685+0000","last_peered":"2024-09-11T08:08:55.359685+0000","last_clean":"2024-09-11T08:08:55.359685+0000","last_became_active":"2024-09-11T08:07:36.068493+0000","last_became_peered":"2024-09-11T08:07:36.068493+0000","last_unstale":"2024-09-11T08:08:55.359685+0000","last_undegraded":"2024-09-11T08:08:55.359685+0000","last_fullsized":"2024-09-11T08:08:55.359685+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":60,"reported_epoch":85,"state":"peering","last_fresh":"2024-09-11T08:09:06.873141+0000","last_change":"2024-09-11T08:09:06.873379+0000","last_active":"2024-09-11T08:09:06.873107+0000","last_peered":"2024-09-11T08:08:55.360586+0000","last_clean":"2024-09-11T08:08:55.360586+0000","last_became_active":"2024-09-11T08:07:32.047347+0000","last_became_peered":"2024-09-11T08:07:32.047347+0000","last_unstale":"2024-09-11T08:09:06.873141+0000","last_undegraded":"2024-09-11T08:09:06.873141+0000","last_fullsized":"2024-09-11T08:09:06.873141+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-11T08:08:36.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:36.400586+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:50.012415+0000","scrub_duration":0.002193113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:03.210255+0000","last_change":"2024-09-11T08:09:03.210218+0000","last_active":"2024-09-11T08:09:03.210255+0000","last_peered":"2024-09-11T08:09:03.210255+0000","last_clean":"2024-09-11T08:09:03.210255+0000","last_became_active":"2024-09-11T08:07:30.039673+0000","last_became_peered":"2024-09-11T08:07:30.039673+0000","last_unstale":"2024-09-11T08:09:03.210255+0000","last_undegraded":"2024-09-11T08:09:03.210255+0000","last_fullsized":"2024-09-11T08:09:03.210255+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:03.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:09:03.210166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:05.614628+0000","scrub_duration":0.0011643720000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":67,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:03.560191+0000","last_change":"2024-09-11T08:08:47.966763+0000","last_active":"2024-09-11T08:09:03.560191+0000","last_peered":"2024-09-11T08:09:03.560191+0000","last_clean":"2024-09-11T08:09:03.560191+0000","last_became_active":"2024-09-11T08:07:34.060736+0000","last_became_peered":"2024-09-11T08:07:34.060736+0000","last_unstale":"2024-09-11T08:09:03.560191+0000","last_undegraded":"2024-09-11T08:09:03.560191+0000","last_fullsized":"2024-09-11T08:09:03.560191+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:08:47.966690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:47.966690+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:12.893275+0000","scrub_duration":0.0019777940000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":29,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.402525+0000","last_change":"2024-09-11T08:07:36.072648+0000","last_active":"2024-09-11T08:08:56.402525+0000","last_peered":"2024-09-11T08:08:56.402525+0000","last_clean":"2024-09-11T08:08:56.402525+0000","last_became_active":"2024-09-11T08:07:36.072359+0000","last_became_peered":"2024-09-11T08:07:36.072359+0000","last_unstale":"2024-09-11T08:08:56.402525+0000","last_undegraded":"2024-09-11T08:08:56.402525+0000","last_fullsized":"2024-09-11T08:08:56.402525+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":62,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:03.516765+0000","last_change":"2024-09-11T08:09:03.516720+0000","last_active":"2024-09-11T08:09:03.516765+0000","last_peered":"2024-09-11T08:09:03.516765+0000","last_clean":"2024-09-11T08:09:03.516765+0000","last_became_active":"2024-09-11T08:07:32.047981+0000","last_became_peered":"2024-09-11T08:07:32.047981+0000","last_unstale":"2024-09-11T08:09:03.516765+0000","last_undegraded":"2024-09-11T08:09:03.516765+0000","last_fullsized":"2024-09-11T08:09:03.516765+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-11T08:09:03.516664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:09:03.516664+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:23.127237+0000","scrub_duration":0.001710917,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.359557+0000","last_change":"2024-09-11T08:08:48.330351+0000","last_active":"2024-09-11T08:08:55.359557+0000","last_peered":"2024-09-11T08:08:55.359557+0000","last_clean":"2024-09-11T08:08:55.359557+0000","last_became_active":"2024-09-11T08:07:30.043691+0000","last_became_peered":"2024-09-11T08:07:30.043691+0000","last_unstale":"2024-09-11T08:08:55.359557+0000","last_undegraded":"2024-09-11T08:08:55.359557+0000","last_fullsized":"2024-09-11T08:08:55.359557+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:48.330285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:48.330285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:05.763303+0000","scrub_duration":0.001379929,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":31,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.011091+0000","last_change":"2024-09-11T08:07:34.053427+0000","last_active":"2024-09-11T08:08:56.011091+0000","last_peered":"2024-09-11T08:08:56.011091+0000","last_clean":"2024-09-11T08:08:56.011091+0000","last_became_active":"2024-09-11T08:07:34.053137+0000","last_became_peered":"2024-09-11T08:07:34.053137+0000","last_unstale":"2024-09-11T08:08:56.011091+0000","last_undegraded":"2024-09-11T08:08:56.011091+0000","last_fullsized":"2024-09-11T08:08:56.011091+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":29,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.360100+0000","last_change":"2024-09-11T08:07:36.076837+0000","last_active":"2024-09-11T08:08:55.360100+0000","last_peered":"2024-09-11T08:08:55.360100+0000","last_clean":"2024-09-11T08:08:55.360100+0000","last_became_active":"2024-09-11T08:07:36.076132+0000","last_became_peered":"2024-09-11T08:07:36.076132+0000","last_unstale":"2024-09-11T08:08:55.360100+0000","last_undegraded":"2024-09-11T08:08:55.360100+0000","last_fullsized":"2024-09-11T08:08:55.360100+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":59,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.011296+0000","last_change":"2024-09-11T08:08:55.332126+0000","last_active":"2024-09-11T08:08:56.011296+0000","last_peered":"2024-09-11T08:08:56.011296+0000","last_clean":"2024-09-11T08:08:56.011296+0000","last_became_active":"2024-09-11T08:07:32.049368+0000","last_became_peered":"2024-09-11T08:07:32.049368+0000","last_unstale":"2024-09-11T08:08:56.011296+0000","last_undegraded":"2024-09-11T08:08:56.011296+0000","last_fullsized":"2024-09-11T08:08:56.011296+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:08:55.332065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:55.332065+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:15.584089+0000","scrub_duration":0.001632218,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.174392+0000","last_change":"2024-09-11T08:08:45.388090+0000","last_active":"2024-09-11T08:08:56.174392+0000","last_peered":"2024-09-11T08:08:56.174392+0000","last_clean":"2024-09-11T08:08:56.174392+0000","last_became_active":"2024-09-11T08:07:30.037692+0000","last_became_peered":"2024-09-11T08:07:30.037692+0000","last_unstale":"2024-09-11T08:08:56.174392+0000","last_undegraded":"2024-09-11T08:08:56.174392+0000","last_fullsized":"2024-09-11T08:08:56.174392+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:45.388024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:45.388024+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:59.280827+0000","scrub_duration":0.0014875719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":31,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.359847+0000","last_change":"2024-09-11T08:07:34.057358+0000","last_active":"2024-09-11T08:08:55.359847+0000","last_peered":"2024-09-11T08:08:55.359847+0000","last_clean":"2024-09-11T08:08:55.359847+0000","last_became_active":"2024-09-11T08:07:34.057064+0000","last_became_peered":"2024-09-11T08:07:34.057064+0000","last_unstale":"2024-09-11T08:08:55.359847+0000","last_undegraded":"2024-09-11T08:08:55.359847+0000","last_fullsized":"2024-09-11T08:08:55.359847+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":29,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.402664+0000","last_change":"2024-09-11T08:07:36.077697+0000","last_active":"2024-09-11T08:08:56.402664+0000","last_peered":"2024-09-11T08:08:56.402664+0000","last_clean":"2024-09-11T08:08:56.402664+0000","last_became_active":"2024-09-11T08:07:36.077406+0000","last_became_peered":"2024-09-11T08:07:36.077406+0000","last_unstale":"2024-09-11T08:08:56.402664+0000","last_undegraded":"2024-09-11T08:08:56.402664+0000","last_fullsized":"2024-09-11T08:08:56.402664+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":52,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.360345+0000","last_change":"2024-09-11T08:07:32.043568+0000","last_active":"2024-09-11T08:08:55.360345+0000","last_peered":"2024-09-11T08:08:55.360345+0000","last_clean":"2024-09-11T08:08:55.360345+0000","last_became_active":"2024-09-11T08:07:32.043276+0000","last_became_peered":"2024-09-11T08:07:32.043276+0000","last_unstale":"2024-09-11T08:08:55.360345+0000","last_undegraded":"2024-09-11T08:08:55.360345+0000","last_fullsized":"2024-09-11T08:08:55.360345+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.295188+0000","last_change":"2024-09-11T08:08:55.853168+0000","last_active":"2024-09-11T08:08:56.295188+0000","last_peered":"2024-09-11T08:08:56.295188+0000","last_clean":"2024-09-11T08:08:56.295188+0000","last_became_active":"2024-09-11T08:07:30.049152+0000","last_became_peered":"2024-09-11T08:07:30.049152+0000","last_unstale":"2024-09-11T08:08:56.295188+0000","last_undegraded":"2024-09-11T08:08:56.295188+0000","last_fullsized":"2024-09-11T08:08:56.295188+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:55.853076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:55.853076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:03.761618+0000","scrub_duration":0.0018073309999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":31,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.173826+0000","last_change":"2024-09-11T08:07:34.057322+0000","last_active":"2024-09-11T08:08:56.173826+0000","last_peered":"2024-09-11T08:08:56.173826+0000","last_clean":"2024-09-11T08:08:56.173826+0000","last_became_active":"2024-09-11T08:07:34.056935+0000","last_became_peered":"2024-09-11T08:07:34.056935+0000","last_unstale":"2024-09-11T08:08:56.173826+0000","last_undegraded":"2024-09-11T08:08:56.173826+0000","last_fullsized":"2024-09-11T08:08:56.173826+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":29,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.359925+0000","last_change":"2024-09-11T08:07:36.633645+0000","last_active":"2024-09-11T08:08:55.359925+0000","last_peered":"2024-09-11T08:08:55.359925+0000","last_clean":"2024-09-11T08:08:55.359925+0000","last_became_active":"2024-09-11T08:07:36.633368+0000","last_became_peered":"2024-09-11T08:07:36.633368+0000","last_unstale":"2024-09-11T08:08:55.359925+0000","last_undegraded":"2024-09-11T08:08:55.359925+0000","last_fullsized":"2024-09-11T08:08:55.359925+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.359749+0000","last_change":"2024-09-11T08:07:32.046050+0000","last_active":"2024-09-11T08:08:55.359749+0000","last_peered":"2024-09-11T08:08:55.359749+0000","last_clean":"2024-09-11T08:08:55.359749+0000","last_became_active":"2024-09-11T08:07:32.045728+0000","last_became_peered":"2024-09-11T08:07:32.045728+0000","last_unstale":"2024-09-11T08:08:55.359749+0000","last_undegraded":"2024-09-11T08:08:55.359749+0000","last_fullsized":"2024-09-11T08:08:55.359749+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.403337+0000","last_change":"2024-09-11T08:08:42.056647+0000","last_active":"2024-09-11T08:08:56.403337+0000","last_peered":"2024-09-11T08:08:56.403337+0000","last_clean":"2024-09-11T08:08:56.403337+0000","last_became_active":"2024-09-11T08:07:30.049409+0000","last_became_peered":"2024-09-11T08:07:30.049409+0000","last_unstale":"2024-09-11T08:08:56.403337+0000","last_undegraded":"2024-09-11T08:08:56.403337+0000","last_fullsized":"2024-09-11T08:08:56.403337+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:42.056591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:42.056591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:56.872271+0000","scrub_duration":0.003068809,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.402392+0000","last_change":"2024-09-11T08:08:34.506991+0000","last_active":"2024-09-11T08:08:56.402392+0000","last_peered":"2024-09-11T08:08:56.402392+0000","last_clean":"2024-09-11T08:08:56.402392+0000","last_became_active":"2024-09-11T08:07:34.055082+0000","last_became_peered":"2024-09-11T08:07:34.055082+0000","last_unstale":"2024-09-11T08:08:56.402392+0000","last_undegraded":"2024-09-11T08:08:56.402392+0000","last_fullsized":"2024-09-11T08:08:56.402392+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:34.506927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:34.506927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:46.584592+0000","scrub_duration":0.0015499070000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":29,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.173566+0000","last_change":"2024-09-11T08:07:36.072035+0000","last_active":"2024-09-11T08:08:56.173566+0000","last_peered":"2024-09-11T08:08:56.173566+0000","last_clean":"2024-09-11T08:08:56.173566+0000","last_became_active":"2024-09-11T08:07:36.071669+0000","last_became_peered":"2024-09-11T08:07:36.071669+0000","last_unstale":"2024-09-11T08:08:56.173566+0000","last_undegraded":"2024-09-11T08:08:56.173566+0000","last_fullsized":"2024-09-11T08:08:56.173566+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":29,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.011339+0000","last_change":"2024-09-11T08:07:36.071729+0000","last_active":"2024-09-11T08:08:56.011339+0000","last_peered":"2024-09-11T08:08:56.011339+0000","last_clean":"2024-09-11T08:08:56.011339+0000","last_became_active":"2024-09-11T08:07:36.071468+0000","last_became_peered":"2024-09-11T08:07:36.071468+0000","last_unstale":"2024-09-11T08:08:56.011339+0000","last_undegraded":"2024-09-11T08:08:56.011339+0000","last_fullsized":"2024-09-11T08:08:56.011339+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.360095+0000","last_change":"2024-09-11T08:08:49.428032+0000","last_active":"2024-09-11T08:08:55.360095+0000","last_peered":"2024-09-11T08:08:55.360095+0000","last_clean":"2024-09-11T08:08:55.360095+0000","last_became_active":"2024-09-11T08:07:30.046702+0000","last_became_peered":"2024-09-11T08:07:30.046702+0000","last_unstale":"2024-09-11T08:08:55.360095+0000","last_undegraded":"2024-09-11T08:08:55.360095+0000","last_fullsized":"2024-09-11T08:08:55.360095+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:49.427944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:49.427944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:11.537893+0000","scrub_duration":0.001383034,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":61,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:03.841823+0000","last_change":"2024-09-11T08:09:03.841778+0000","last_active":"2024-09-11T08:09:03.841823+0000","last_peered":"2024-09-11T08:09:03.841823+0000","last_clean":"2024-09-11T08:09:03.841823+0000","last_became_active":"2024-09-11T08:07:32.054656+0000","last_became_peered":"2024-09-11T08:07:32.054656+0000","last_unstale":"2024-09-11T08:09:03.841823+0000","last_undegraded":"2024-09-11T08:09:03.841823+0000","last_fullsized":"2024-09-11T08:09:03.841823+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-11T08:09:03.841721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:09:03.841721+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:20.583780+0000","scrub_duration":0.00194933,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.403143+0000","last_change":"2024-09-11T08:08:53.869870+0000","last_active":"2024-09-11T08:08:56.403143+0000","last_peered":"2024-09-11T08:08:56.403143+0000","last_clean":"2024-09-11T08:08:56.403143+0000","last_became_active":"2024-09-11T08:07:34.061453+0000","last_became_peered":"2024-09-11T08:07:34.061453+0000","last_unstale":"2024-09-11T08:08:56.403143+0000","last_undegraded":"2024-09-11T08:08:56.403143+0000","last_fullsized":"2024-09-11T08:08:56.403143+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:53.869813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:53.869813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:59.468130+0000","scrub_duration":0.0012524350000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":413,"ondisk_log_size":413,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":29,"num_read_kb":29,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":91,"num_read_kb":86,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266712,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298796,"kb_used_data":1200,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446356,"statfs":{"total":95995035648,"available":95689068544,"internally_reserved":0,"allocated":1228800,"data_stored":881761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.47199999999999998}]},{"osd":1,"last update":"Wed Sep 11 08:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.437},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.42799999999999999}]},{"osd":2,"last update":"Wed Sep 11 08:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.41699999999999998}]},{"osd":3,"last update":"Wed Sep 11 08:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":7.9210000000000003,"5min":7.9210000000000003,"15min":7.9210000000000003},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.374}]},{"osd":4,"last update":"Wed Sep 11 08:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.314}]},{"osd":5,"last update":"Wed Sep 11 08:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.39000000000000001}]},{"osd":6,"last update":"Wed Sep 11 08:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.34399999999999997}]}]},{"osd":6,"up_from":45,"seq":193273528349,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298764,"kb_used_data":1412,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446388,"statfs":{"total":95995035648,"available":95689101312,"internally_reserved":0,"allocated":1445888,"data_stored":951248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.49299999999999999,"15min":0.496},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.59099999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.50700000000000001,"15min":0.50600000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.72099999999999997,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.54000000000000004}]},{"osd":1,"last update":"Wed Sep 11 08:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.47999999999999998,"15min":0.47799999999999998},"min":{"1min":0.32700000000000001,"5min":0.318,"15min":0.318},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.505,"15min":0.50600000000000001},"min":{"1min":0.33500000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.65900000000000003,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.443}]},{"osd":2,"last update":"Wed Sep 11 08:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.5,"15min":0.502},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.67200000000000004,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.499,"5min":0.48499999999999999,"15min":0.48299999999999998},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.72399999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.51000000000000001}]},{"osd":3,"last update":"Wed Sep 11 08:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.52400000000000002,"15min":0.52600000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.5},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.53900000000000003,"15min":0.53800000000000003},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.73499999999999999,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.52500000000000002}]},{"osd":4,"last update":"Wed Sep 11 08:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.39000000000000001,"15min":0.38700000000000001},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.59399999999999997,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.41999999999999998,"15min":0.41199999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.38}]},{"osd":5,"last update":"Wed Sep 11 08:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.48899999999999999,"15min":0.47999999999999998},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.434,"5min":0.41399999999999998,"15min":0.41099999999999998},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.62,"5min":0.629,"15min":0.629},"last":0.41199999999999998}]},{"osd":7,"last update":"Wed Sep 11 08:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.39900000000000002}]}]},{"osd":5,"up_from":38,"seq":163208757282,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298180,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446972,"statfs":{"total":95995035648,"available":95689699328,"internally_reserved":0,"allocated":847872,"data_stored":360087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.51400000000000001,"15min":0.52200000000000002},"min":{"1min":0.318,"5min":0.31,"15min":0.31},"max":{"1min":0.63500000000000001,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.503,"15min":0.501},"min":{"1min":0.41399999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.54000000000000004}]},{"osd":1,"last update":"Wed Sep 11 08:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":1.1910000000000001,"15min":1.3029999999999999},"min":{"1min":0.33000000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.82499999999999996,"5min":16.004999999999999,"15min":16.004999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.36199999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.57999999999999996}]},{"osd":2,"last update":"Wed Sep 11 08:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":1.206,"15min":1.323},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.754,"5min":16.018000000000001,"15min":16.018000000000001},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.55900000000000005}]},{"osd":3,"last update":"Wed Sep 11 08:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.53100000000000003,"15min":0.52900000000000003},"min":{"1min":0.44900000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.72299999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.505,"5min":0.51600000000000001,"15min":0.51800000000000002},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.45400000000000001}]},{"osd":4,"last update":"Wed Sep 11 08:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.39700000000000002,"15min":0.39300000000000002},"min":{"1min":0.27500000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.434},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.40000000000000002,"15min":0.38800000000000001},"min":{"1min":0.36499999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.46899999999999997}]},{"osd":6,"last update":"Wed Sep 11 08:08:56 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.57699999999999996,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.372},{"interface":"front","average":{"1min":0.495,"5min":0.45200000000000001,"15min":0.44500000000000001},"min":{"1min":0.316,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.67100000000000004,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.496}]},{"osd":7,"last update":"Wed Sep 11 08:08:22 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.39800000000000002}]}]},{"osd":4,"up_from":31,"seq":133143986215,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298220,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446932,"statfs":{"total":95995035648,"available":95689658368,"internally_reserved":0,"allocated":638976,"data_stored":291776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.499,"15min":0.50900000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.59699999999999998,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.52400000000000002,"15min":0.53500000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.58899999999999997,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.438}]},{"osd":1,"last update":"Wed Sep 11 08:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.505,"15min":0.51300000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.66800000000000004,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.52300000000000002,"15min":0.52900000000000003},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.68899999999999995,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.498}]},{"osd":2,"last update":"Wed Sep 11 08:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.52000000000000002,"15min":0.52800000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.58799999999999997,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.51600000000000001,"15min":0.52500000000000002},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.623,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.39900000000000002}]},{"osd":3,"last update":"Wed Sep 11 08:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.50900000000000001,"15min":0.51500000000000001},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.64300000000000002,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.51800000000000002,"15min":0.52600000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.61499999999999999,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.40799999999999997}]},{"osd":5,"last update":"Wed Sep 11 08:08:31 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.36899999999999999,"15min":0.36699999999999999},"min":{"1min":0.27800000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.36899999999999999,"15min":0.36699999999999999},"min":{"1min":0.27900000000000003,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.33500000000000002}]},{"osd":6,"last update":"Wed Sep 11 08:08:56 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.39100000000000001,"15min":0.38300000000000001},"min":{"1min":0.30399999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.314},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.44,"15min":0.432},"min":{"1min":0.32200000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.38500000000000001}]},{"osd":7,"last update":"Wed Sep 11 08:08:20 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.377}]}]},{"osd":3,"up_from":25,"seq":107374182444,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298236,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446916,"statfs":{"total":95995035648,"available":95689641984,"internally_reserved":0,"allocated":655360,"data_stored":291562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:08:53 2024","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.36199999999999999,"15min":0.374},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":0.55100000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.315},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.40300000000000002,"15min":0.41799999999999998},"min":{"1min":0.26900000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.55100000000000005,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.26800000000000002}]},{"osd":1,"last update":"Wed Sep 11 08:08:53 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.41599999999999998,"15min":0.437},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.497,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.371},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.41399999999999998,"15min":0.42899999999999999},"min":{"1min":0.24399999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.53800000000000003,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.29199999999999998}]},{"osd":2,"last update":"Wed Sep 11 08:08:53 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.434,"15min":0.44900000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.47999999999999998,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.371,"5min":0.40600000000000003,"15min":0.41899999999999998},"min":{"1min":0.26700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.53100000000000003,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.47199999999999998}]},{"osd":4,"last update":"Wed Sep 11 08:08:12 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54800000000000004,"15min":0.54900000000000004},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.5},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.53500000000000003,"15min":0.53700000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.51900000000000002}]},{"osd":5,"last update":"Wed Sep 11 08:08:33 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.54200000000000004,"15min":0.53800000000000003},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.40699999999999997,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.40500000000000003}]},{"osd":6,"last update":"Wed Sep 11 08:08:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.53400000000000003,"15min":0.53000000000000003},"min":{"1min":0.35899999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.55400000000000005,"15min":0.55800000000000005},"min":{"1min":0.39500000000000002,"5min":0.373,"15min":0.373},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.53300000000000003}]},{"osd":7,"last update":"Wed Sep 11 08:08:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.434}]}]},{"osd":2,"up_from":18,"seq":77309411375,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298192,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446960,"statfs":{"total":95995035648,"available":95689687040,"internally_reserved":0,"allocated":610304,"data_stored":290154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:08:24 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.38400000000000001,"15min":0.38100000000000001},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.58099999999999996,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.37,"15min":0.35999999999999999},"min":{"1min":0.25,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.55600000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.52100000000000002}]},{"osd":1,"last update":"Wed Sep 11 08:08:24 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.39500000000000002,"15min":0.38},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.60899999999999999,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.504},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.39800000000000002,"15min":0.38600000000000001},"min":{"1min":0.28699999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.45300000000000001}]},{"osd":3,"last update":"Wed Sep 11 08:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.45100000000000001,"15min":0.46100000000000002},"min":{"1min":0.27300000000000002,"5min":0.218,"15min":0.218},"max":{"1min":0.629,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.45100000000000001,"15min":0.46500000000000002},"min":{"1min":0.318,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.48899999999999999}]},{"osd":4,"last update":"Wed Sep 11 08:08:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.58299999999999996,"15min":0.58699999999999997},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.71199999999999997,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.71599999999999997}]},{"osd":5,"last update":"Wed Sep 11 08:08:35 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.57699999999999996,"15min":0.58299999999999996},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.751,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.59199999999999997,"15min":0.59999999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.65300000000000002,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.73199999999999998}]},{"osd":6,"last update":"Wed Sep 11 08:08:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.58499999999999996,"15min":0.59399999999999997},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.59099999999999997,"15min":0.59699999999999998},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.70299999999999996,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.74299999999999999}]},{"osd":7,"last update":"Wed Sep 11 08:08:22 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.70299999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574901,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298248,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446904,"statfs":{"total":95995035648,"available":95689629696,"internally_reserved":0,"allocated":667648,"data_stored":291720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:09:04 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.33700000000000002,"15min":0.30399999999999999},"min":{"1min":0.23599999999999999,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.33900000000000002,"15min":0.315},"min":{"1min":0.224,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.22900000000000001}]},{"osd":2,"last update":"Wed Sep 11 08:08:28 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.36199999999999999,"15min":0.34799999999999998},"min":{"1min":0.27700000000000002,"5min":0.224,"15min":0.224},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.37,"5min":0.35399999999999998,"15min":0.34200000000000003},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.247}]},{"osd":3,"last update":"Wed Sep 11 08:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.41399999999999998,"15min":0.42199999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.439,"15min":0.437},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.198,"5min":1.198,"15min":1.198},"last":0.35999999999999999}]},{"osd":4,"last update":"Wed Sep 11 08:08:08 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54300000000000004,"15min":0.54200000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.55800000000000005,"15min":0.55600000000000005},"min":{"1min":0.41899999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.73499999999999999,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.33100000000000002}]},{"osd":5,"last update":"Wed Sep 11 08:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.53700000000000003,"15min":0.53000000000000003},"min":{"1min":0.40699999999999997,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.77600000000000002,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.312},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.53100000000000003,"15min":0.52700000000000002},"min":{"1min":0.38400000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.40600000000000003}]},{"osd":6,"last update":"Wed Sep 11 08:08:56 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.55500000000000005,"15min":0.55900000000000005},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.59899999999999998,"15min":0.60099999999999998},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.433}]},{"osd":7,"last update":"Wed Sep 11 08:08:17 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.38500000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738425,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298808,"kb_used_data":1212,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446344,"statfs":{"total":95995035648,"available":95689056256,"internally_reserved":0,"allocated":1241088,"data_stored":882707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 11 08:09:04 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.34799999999999998,"15min":0.317},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":0.53400000000000003,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.35899999999999999,"15min":0.33800000000000002},"min":{"1min":0.219,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.47999999999999998,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.254}]},{"osd":2,"last update":"Wed Sep 11 08:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.34699999999999998},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.48799999999999999,"5min":0.53000000000000003,"15min":0.53000000000000003},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.378,"5min":0.378,"15min":0.36899999999999999},"min":{"1min":0.26600000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.56299999999999994,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.32200000000000001}]},{"osd":3,"last update":"Wed Sep 11 08:08:51 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.44,"15min":0.44},"min":{"1min":0.27600000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.52300000000000002,"5min":0.62,"15min":0.62},"last":0.379},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.42499999999999999,"15min":0.42599999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.52800000000000002,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.30399999999999999}]},{"osd":4,"last update":"Wed Sep 11 08:08:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.55000000000000004,"15min":0.54600000000000004},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.47699999999999998}]},{"osd":5,"last update":"Wed Sep 11 08:08:36 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.58499999999999996,"15min":0.58299999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.748,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.495},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.55800000000000005,"15min":0.55300000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.37}]},{"osd":6,"last update":"Wed Sep 11 08:08:59 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.60899999999999999,"15min":0.61699999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.75600000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.60299999999999998,"15min":0.57099999999999995},"min":{"1min":0.433,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":4.7069999999999999,"5min":4.7069999999999999,"15min":4.7069999999999999},"last":0.39200000000000002}]},{"osd":7,"last update":"Wed Sep 11 08:08:19 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.435},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.33900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T08:09:09.928 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T08:09:10.073 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:09 smithi181 bash[63714]: Getting image source signatures 2024-09-11T08:09:10.074 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:09 smithi181 bash[63714]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-09-11T08:09:10.074 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:09 smithi181 bash[63714]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-09-11T08:09:10.074 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:09 smithi181 bash[63714]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-09-11T08:09:10.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:09 smithi181 ceph-mon[32701]: 5.1a scrub starts 2024-09-11T08:09:10.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:09 smithi181 ceph-mon[32701]: 5.1a scrub ok 2024-09-11T08:09:10.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:09 smithi181 ceph-mon[32701]: Deploying daemon node-exporter.b on smithi181 2024-09-11T08:09:10.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:09 smithi181 ceph-mon[32701]: 4.1b scrub starts 2024-09-11T08:09:10.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:09 smithi181 ceph-mon[32701]: 4.1b scrub ok 2024-09-11T08:09:10.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:09 smithi181 ceph-mon[32701]: pgmap v47: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:09:10.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:09 smithi181 ceph-mon[32701]: from='client.14709 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:09:10.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:09 smithi181 ceph-mon[32701]: 6.1d scrub starts 2024-09-11T08:09:10.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:09 smithi181 ceph-mon[32701]: 6.1d scrub ok 2024-09-11T08:09:10.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:09 smithi181 ceph-mon[32701]: 4.1d scrub starts 2024-09-11T08:09:10.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:09 smithi181 ceph-mon[32701]: 4.1d scrub ok 2024-09-11T08:09:10.179 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:09:10.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:09 smithi150 ceph-mon[39612]: 5.1a scrub starts 2024-09-11T08:09:10.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:09 smithi150 ceph-mon[39612]: 5.1a scrub ok 2024-09-11T08:09:10.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:09 smithi150 ceph-mon[39612]: Deploying daemon node-exporter.b on smithi181 2024-09-11T08:09:10.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:09 smithi150 ceph-mon[39612]: 4.1b scrub starts 2024-09-11T08:09:10.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:09 smithi150 ceph-mon[39612]: 4.1b scrub ok 2024-09-11T08:09:10.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:09 smithi150 ceph-mon[39612]: pgmap v47: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:09:10.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:09 smithi150 ceph-mon[39612]: from='client.14709 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:09:10.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:09 smithi150 ceph-mon[39612]: 6.1d scrub starts 2024-09-11T08:09:10.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:09 smithi150 ceph-mon[39612]: 6.1d scrub ok 2024-09-11T08:09:10.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:09 smithi150 ceph-mon[39612]: 4.1d scrub starts 2024-09-11T08:09:10.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:09 smithi150 ceph-mon[39612]: 4.1d scrub ok 2024-09-11T08:09:10.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:09 smithi150 ceph-mon[29276]: 5.1a scrub starts 2024-09-11T08:09:10.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:09 smithi150 ceph-mon[29276]: 5.1a scrub ok 2024-09-11T08:09:10.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:09 smithi150 ceph-mon[29276]: Deploying daemon node-exporter.b on smithi181 2024-09-11T08:09:10.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:09 smithi150 ceph-mon[29276]: 4.1b scrub starts 2024-09-11T08:09:10.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:09 smithi150 ceph-mon[29276]: 4.1b scrub ok 2024-09-11T08:09:10.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:09 smithi150 ceph-mon[29276]: pgmap v47: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:09:10.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:09 smithi150 ceph-mon[29276]: from='client.14709 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:09:10.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:09 smithi150 ceph-mon[29276]: 6.1d scrub starts 2024-09-11T08:09:10.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:09 smithi150 ceph-mon[29276]: 6.1d scrub ok 2024-09-11T08:09:10.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:09 smithi150 ceph-mon[29276]: 4.1d scrub starts 2024-09-11T08:09:10.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:09 smithi150 ceph-mon[29276]: 4.1d scrub ok 2024-09-11T08:09:11.193 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:10 smithi181 ceph-mon[32701]: 6.a scrub starts 2024-09-11T08:09:11.193 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:10 smithi181 ceph-mon[32701]: 6.a scrub ok 2024-09-11T08:09:11.193 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:10 smithi181 ceph-mon[32701]: 6.10 deep-scrub starts 2024-09-11T08:09:11.193 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:10 smithi181 ceph-mon[32701]: 6.10 deep-scrub ok 2024-09-11T08:09:11.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:10 smithi150 ceph-mon[29276]: 6.a scrub starts 2024-09-11T08:09:11.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:10 smithi150 ceph-mon[29276]: 6.a scrub ok 2024-09-11T08:09:11.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:10 smithi150 ceph-mon[29276]: 6.10 deep-scrub starts 2024-09-11T08:09:11.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:10 smithi150 ceph-mon[29276]: 6.10 deep-scrub ok 2024-09-11T08:09:11.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:10 smithi150 ceph-mon[39612]: 6.a scrub starts 2024-09-11T08:09:11.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:10 smithi150 ceph-mon[39612]: 6.a scrub ok 2024-09-11T08:09:11.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:10 smithi150 ceph-mon[39612]: 6.10 deep-scrub starts 2024-09-11T08:09:11.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:10 smithi150 ceph-mon[39612]: 6.10 deep-scrub ok 2024-09-11T08:09:11.573 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:11 smithi181 bash[63714]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-09-11T08:09:11.974 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:11 smithi150 ceph-mon[29276]: 5.10 scrub starts 2024-09-11T08:09:11.974 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:11 smithi150 ceph-mon[29276]: 5.10 scrub ok 2024-09-11T08:09:11.974 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:11 smithi150 ceph-mon[29276]: 5.e scrub starts 2024-09-11T08:09:11.974 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:11 smithi150 ceph-mon[29276]: 5.e scrub ok 2024-09-11T08:09:11.975 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:11 smithi150 ceph-mon[29276]: pgmap v48: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-11T08:09:11.975 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:11 smithi150 ceph-mon[29276]: 4.8 scrub starts 2024-09-11T08:09:11.975 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:11 smithi150 ceph-mon[29276]: 4.8 scrub ok 2024-09-11T08:09:11.975 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:11 smithi150 ceph-mon[29276]: 4.4 deep-scrub starts 2024-09-11T08:09:11.975 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:11 smithi150 ceph-mon[29276]: 4.4 deep-scrub ok 2024-09-11T08:09:11.975 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:11 smithi150 ceph-mon[39612]: 5.10 scrub starts 2024-09-11T08:09:11.975 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:11 smithi150 ceph-mon[39612]: 5.10 scrub ok 2024-09-11T08:09:11.975 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:11 smithi150 ceph-mon[39612]: 5.e scrub starts 2024-09-11T08:09:11.975 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:11 smithi150 ceph-mon[39612]: 5.e scrub ok 2024-09-11T08:09:11.975 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:11 smithi150 ceph-mon[39612]: pgmap v48: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-11T08:09:11.975 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:11 smithi150 ceph-mon[39612]: 4.8 scrub starts 2024-09-11T08:09:11.975 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:11 smithi150 ceph-mon[39612]: 4.8 scrub ok 2024-09-11T08:09:11.976 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:11 smithi150 ceph-mon[39612]: 4.4 deep-scrub starts 2024-09-11T08:09:11.976 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:11 smithi150 ceph-mon[39612]: 4.4 deep-scrub ok 2024-09-11T08:09:12.119 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:09:12.121 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-09-11T08:09:12.315 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:11 smithi181 ceph-mon[32701]: 5.10 scrub starts 2024-09-11T08:09:12.316 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:11 smithi181 ceph-mon[32701]: 5.10 scrub ok 2024-09-11T08:09:12.316 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:11 smithi181 ceph-mon[32701]: 5.e scrub starts 2024-09-11T08:09:12.316 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:11 smithi181 ceph-mon[32701]: 5.e scrub ok 2024-09-11T08:09:12.316 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:11 smithi181 ceph-mon[32701]: pgmap v48: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-11T08:09:12.316 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:11 smithi181 ceph-mon[32701]: 4.8 scrub starts 2024-09-11T08:09:12.316 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:11 smithi181 ceph-mon[32701]: 4.8 scrub ok 2024-09-11T08:09:12.316 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:11 smithi181 ceph-mon[32701]: 4.4 deep-scrub starts 2024-09-11T08:09:12.316 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:11 smithi181 ceph-mon[32701]: 4.4 deep-scrub ok 2024-09-11T08:09:12.573 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:12 smithi181 bash[63714]: Writing manifest to image destination 2024-09-11T08:09:12.823 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":48,"stamp":"2024-09-11T08:09:10.693442+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596188,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":944,"num_read_kb":801,"num_write":543,"num_write_kb":1671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":571,"ondisk_log_size":571,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2387444,"kb_used_data":7164,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":747573772,"statfs":{"total":767960285184,"available":765515542528,"internally_reserved":0,"allocated":7335936,"data_stored":4241015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001791"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.403338+0000","last_change":"2024-09-11T08:08:36.105232+0000","last_active":"2024-09-11T08:08:56.403338+0000","last_peered":"2024-09-11T08:08:56.403338+0000","last_clean":"2024-09-11T08:08:56.403338+0000","last_became_active":"2024-09-11T08:07:30.044976+0000","last_became_peered":"2024-09-11T08:07:30.044976+0000","last_unstale":"2024-09-11T08:08:56.403338+0000","last_undegraded":"2024-09-11T08:08:56.403338+0000","last_fullsized":"2024-09-11T08:08:56.403338+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:36.105175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:36.105175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:56.596929+0000","scrub_duration":0.0016664970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":49,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.358857+0000","last_change":"2024-09-11T08:07:32.048578+0000","last_active":"2024-09-11T08:08:55.358857+0000","last_peered":"2024-09-11T08:08:55.358857+0000","last_clean":"2024-09-11T08:08:55.358857+0000","last_became_active":"2024-09-11T08:07:32.048048+0000","last_became_peered":"2024-09-11T08:07:32.048048+0000","last_unstale":"2024-09-11T08:08:55.358857+0000","last_undegraded":"2024-09-11T08:08:55.358857+0000","last_fullsized":"2024-09-11T08:08:55.358857+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.011005+0000","last_change":"2024-09-11T08:08:38.486126+0000","last_active":"2024-09-11T08:08:56.011005+0000","last_peered":"2024-09-11T08:08:56.011005+0000","last_clean":"2024-09-11T08:08:56.011005+0000","last_became_active":"2024-09-11T08:07:34.058828+0000","last_became_peered":"2024-09-11T08:07:34.058828+0000","last_unstale":"2024-09-11T08:08:56.011005+0000","last_undegraded":"2024-09-11T08:08:56.011005+0000","last_fullsized":"2024-09-11T08:08:56.011005+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:38.486057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:38.486057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:44.914804+0000","scrub_duration":0.002230531,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:02.470792+0000","last_change":"2024-09-11T08:09:02.470722+0000","last_active":"2024-09-11T08:09:02.470792+0000","last_peered":"2024-09-11T08:09:02.470792+0000","last_clean":"2024-09-11T08:09:02.470792+0000","last_became_active":"2024-09-11T08:07:36.074789+0000","last_became_peered":"2024-09-11T08:07:36.074789+0000","last_unstale":"2024-09-11T08:09:02.470792+0000","last_undegraded":"2024-09-11T08:09:02.470792+0000","last_fullsized":"2024-09-11T08:09:02.470792+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:02.470649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:02.470649+0000","last_clean_scrub_stamp":"2024-09-11T08:09:02.470649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:18.421855+0000","scrub_duration":0.00141182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":29,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.359766+0000","last_change":"2024-09-11T08:07:36.633716+0000","last_active":"2024-09-11T08:08:55.359766+0000","last_peered":"2024-09-11T08:08:55.359766+0000","last_clean":"2024-09-11T08:08:55.359766+0000","last_became_active":"2024-09-11T08:07:36.633423+0000","last_became_peered":"2024-09-11T08:07:36.633423+0000","last_unstale":"2024-09-11T08:08:55.359766+0000","last_undegraded":"2024-09-11T08:08:55.359766+0000","last_fullsized":"2024-09-11T08:08:55.359766+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.360053+0000","last_change":"2024-09-11T08:07:30.045845+0000","last_active":"2024-09-11T08:08:55.360053+0000","last_peered":"2024-09-11T08:08:55.360053+0000","last_clean":"2024-09-11T08:08:55.360053+0000","last_became_active":"2024-09-11T08:07:30.045579+0000","last_became_peered":"2024-09-11T08:07:30.045579+0000","last_unstale":"2024-09-11T08:08:55.360053+0000","last_undegraded":"2024-09-11T08:08:55.360053+0000","last_fullsized":"2024-09-11T08:08:55.360053+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":68,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.408379+0000","last_change":"2024-09-11T08:08:56.408326+0000","last_active":"2024-09-11T08:08:56.408379+0000","last_peered":"2024-09-11T08:08:56.408379+0000","last_clean":"2024-09-11T08:08:56.408379+0000","last_became_active":"2024-09-11T08:07:32.045848+0000","last_became_peered":"2024-09-11T08:07:32.045848+0000","last_unstale":"2024-09-11T08:08:56.408379+0000","last_undegraded":"2024-09-11T08:08:56.408379+0000","last_fullsized":"2024-09-11T08:08:56.408379+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-11T08:08:56.408272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:56.408272+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:09.630406+0000","scrub_duration":0.001771557,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":31,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.358773+0000","last_change":"2024-09-11T08:07:34.053483+0000","last_active":"2024-09-11T08:08:55.358773+0000","last_peered":"2024-09-11T08:08:55.358773+0000","last_clean":"2024-09-11T08:08:55.358773+0000","last_became_active":"2024-09-11T08:07:34.052243+0000","last_became_peered":"2024-09-11T08:07:34.052243+0000","last_unstale":"2024-09-11T08:08:55.358773+0000","last_undegraded":"2024-09-11T08:08:55.358773+0000","last_fullsized":"2024-09-11T08:08:55.358773+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.295460+0000","last_change":"2024-09-11T08:08:54.878501+0000","last_active":"2024-09-11T08:08:56.295460+0000","last_peered":"2024-09-11T08:08:56.295460+0000","last_clean":"2024-09-11T08:08:56.295460+0000","last_became_active":"2024-09-11T08:07:30.043950+0000","last_became_peered":"2024-09-11T08:07:30.043950+0000","last_unstale":"2024-09-11T08:08:56.295460+0000","last_undegraded":"2024-09-11T08:08:56.295460+0000","last_fullsized":"2024-09-11T08:08:56.295460+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:54.878409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:54.878409+0000","last_clean_scrub_stamp":"2024-09-11T08:08:54.878409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:07.483340+0000","scrub_duration":0.0015972390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:07.628517+0000","last_change":"2024-09-11T08:09:07.628460+0000","last_active":"2024-09-11T08:09:07.628517+0000","last_peered":"2024-09-11T08:09:07.628517+0000","last_clean":"2024-09-11T08:09:07.628517+0000","last_became_active":"2024-09-11T08:07:32.044626+0000","last_became_peered":"2024-09-11T08:07:32.044626+0000","last_unstale":"2024-09-11T08:09:07.628517+0000","last_undegraded":"2024-09-11T08:09:07.628517+0000","last_fullsized":"2024-09-11T08:09:07.628517+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:09:07.628369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:09:07.628369+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:10.676317+0000","scrub_duration":0.0023051830000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.294789+0000","last_change":"2024-09-11T08:07:34.061363+0000","last_active":"2024-09-11T08:08:56.294789+0000","last_peered":"2024-09-11T08:08:56.294789+0000","last_clean":"2024-09-11T08:08:56.294789+0000","last_became_active":"2024-09-11T08:07:34.058689+0000","last_became_peered":"2024-09-11T08:07:34.058689+0000","last_unstale":"2024-09-11T08:08:56.294789+0000","last_undegraded":"2024-09-11T08:08:56.294789+0000","last_fullsized":"2024-09-11T08:08:56.294789+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.402638+0000","last_change":"2024-09-11T08:08:40.071203+0000","last_active":"2024-09-11T08:08:56.402638+0000","last_peered":"2024-09-11T08:08:56.402638+0000","last_clean":"2024-09-11T08:08:56.402638+0000","last_became_active":"2024-09-11T08:07:36.076824+0000","last_became_peered":"2024-09-11T08:07:36.076824+0000","last_unstale":"2024-09-11T08:08:56.402638+0000","last_undegraded":"2024-09-11T08:08:56.402638+0000","last_fullsized":"2024-09-11T08:08:56.402638+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:40.071106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:08:40.071106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:54.333357+0000","scrub_duration":0.0019674499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.295183+0000","last_change":"2024-09-11T08:08:36.825857+0000","last_active":"2024-09-11T08:08:56.295183+0000","last_peered":"2024-09-11T08:08:56.295183+0000","last_clean":"2024-09-11T08:08:56.295183+0000","last_became_active":"2024-09-11T08:07:30.043277+0000","last_became_peered":"2024-09-11T08:07:30.043277+0000","last_unstale":"2024-09-11T08:08:56.295183+0000","last_undegraded":"2024-09-11T08:08:56.295183+0000","last_fullsized":"2024-09-11T08:08:56.295183+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:36.825794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:36.825794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:47.493434+0000","scrub_duration":0.0016805920000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.359266+0000","last_change":"2024-09-11T08:07:32.048969+0000","last_active":"2024-09-11T08:08:55.359266+0000","last_peered":"2024-09-11T08:08:55.359266+0000","last_clean":"2024-09-11T08:08:55.359266+0000","last_became_active":"2024-09-11T08:07:32.048661+0000","last_became_peered":"2024-09-11T08:07:32.048661+0000","last_unstale":"2024-09-11T08:08:55.359266+0000","last_undegraded":"2024-09-11T08:08:55.359266+0000","last_fullsized":"2024-09-11T08:08:55.359266+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":31,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.174047+0000","last_change":"2024-09-11T08:07:34.061194+0000","last_active":"2024-09-11T08:08:56.174047+0000","last_peered":"2024-09-11T08:08:56.174047+0000","last_clean":"2024-09-11T08:08:56.174047+0000","last_became_active":"2024-09-11T08:07:34.060871+0000","last_became_peered":"2024-09-11T08:07:34.060871+0000","last_unstale":"2024-09-11T08:08:56.174047+0000","last_undegraded":"2024-09-11T08:08:56.174047+0000","last_fullsized":"2024-09-11T08:08:56.174047+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":38,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:04.782671+0000","last_change":"2024-09-11T08:09:04.782636+0000","last_active":"2024-09-11T08:09:04.782671+0000","last_peered":"2024-09-11T08:09:04.782671+0000","last_clean":"2024-09-11T08:09:04.782671+0000","last_became_active":"2024-09-11T08:07:36.077602+0000","last_became_peered":"2024-09-11T08:07:36.077602+0000","last_unstale":"2024-09-11T08:09:04.782671+0000","last_undegraded":"2024-09-11T08:09:04.782671+0000","last_fullsized":"2024-09-11T08:09:04.782671+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:04.782577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:04.782577+0000","last_clean_scrub_stamp":"2024-09-11T08:09:04.782577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:23.367449+0000","scrub_duration":0.0014982330000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":29,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.359276+0000","last_change":"2024-09-11T08:07:36.633951+0000","last_active":"2024-09-11T08:08:55.359276+0000","last_peered":"2024-09-11T08:08:55.359276+0000","last_clean":"2024-09-11T08:08:55.359276+0000","last_became_active":"2024-09-11T08:07:36.633338+0000","last_became_peered":"2024-09-11T08:07:36.633338+0000","last_unstale":"2024-09-11T08:08:55.359276+0000","last_undegraded":"2024-09-11T08:08:55.359276+0000","last_fullsized":"2024-09-11T08:08:55.359276+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.403206+0000","last_change":"2024-09-11T08:07:30.046004+0000","last_active":"2024-09-11T08:08:56.403206+0000","last_peered":"2024-09-11T08:08:56.403206+0000","last_clean":"2024-09-11T08:08:56.403206+0000","last_became_active":"2024-09-11T08:07:30.045739+0000","last_became_peered":"2024-09-11T08:07:30.045739+0000","last_unstale":"2024-09-11T08:08:56.403206+0000","last_undegraded":"2024-09-11T08:08:56.403206+0000","last_fullsized":"2024-09-11T08:08:56.403206+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"66'15","reported_seq":67,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.975447+0000","last_change":"2024-09-11T08:08:56.975395+0000","last_active":"2024-09-11T08:08:56.975447+0000","last_peered":"2024-09-11T08:08:56.975447+0000","last_clean":"2024-09-11T08:08:56.975447+0000","last_became_active":"2024-09-11T08:07:32.046363+0000","last_became_peered":"2024-09-11T08:07:32.046363+0000","last_unstale":"2024-09-11T08:08:56.975447+0000","last_undegraded":"2024-09-11T08:08:56.975447+0000","last_fullsized":"2024-09-11T08:08:56.975447+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-11T08:08:56.975338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:56.975338+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:19.207433+0000","scrub_duration":0.0017787899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:01.315982+0000","last_change":"2024-09-11T08:09:01.315948+0000","last_active":"2024-09-11T08:09:01.315982+0000","last_peered":"2024-09-11T08:09:01.315982+0000","last_clean":"2024-09-11T08:09:01.315982+0000","last_became_active":"2024-09-11T08:07:34.055410+0000","last_became_peered":"2024-09-11T08:07:34.055410+0000","last_unstale":"2024-09-11T08:09:01.315982+0000","last_undegraded":"2024-09-11T08:09:01.315982+0000","last_fullsized":"2024-09-11T08:09:01.315982+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:01.315897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:09:01.315897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:14.655816+0000","scrub_duration":0.001561163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":29,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.360143+0000","last_change":"2024-09-11T08:07:36.633626+0000","last_active":"2024-09-11T08:08:55.360143+0000","last_peered":"2024-09-11T08:08:55.360143+0000","last_clean":"2024-09-11T08:08:55.360143+0000","last_became_active":"2024-09-11T08:07:36.633258+0000","last_became_peered":"2024-09-11T08:07:36.633258+0000","last_unstale":"2024-09-11T08:08:55.360143+0000","last_undegraded":"2024-09-11T08:08:55.360143+0000","last_fullsized":"2024-09-11T08:08:55.360143+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.359567+0000","last_change":"2024-09-11T08:08:51.401929+0000","last_active":"2024-09-11T08:08:55.359567+0000","last_peered":"2024-09-11T08:08:55.359567+0000","last_clean":"2024-09-11T08:08:55.359567+0000","last_became_active":"2024-09-11T08:07:30.046284+0000","last_became_peered":"2024-09-11T08:07:30.046284+0000","last_unstale":"2024-09-11T08:08:55.359567+0000","last_undegraded":"2024-09-11T08:08:55.359567+0000","last_fullsized":"2024-09-11T08:08:55.359567+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:51.401864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:51.401864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:18.688155+0000","scrub_duration":0.001460034,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":58,"reported_epoch":85,"state":"peering","last_fresh":"2024-09-11T08:09:06.874089+0000","last_change":"2024-09-11T08:09:06.874325+0000","last_active":"2024-09-11T08:09:06.874061+0000","last_peered":"2024-09-11T08:08:55.359627+0000","last_clean":"2024-09-11T08:08:55.359627+0000","last_became_active":"2024-09-11T08:07:32.046737+0000","last_became_peered":"2024-09-11T08:07:32.046737+0000","last_unstale":"2024-09-11T08:09:06.874089+0000","last_undegraded":"2024-09-11T08:09:06.874089+0000","last_fullsized":"2024-09-11T08:09:06.874089+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,4],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.358990+0000","last_change":"2024-09-11T08:08:40.419900+0000","last_active":"2024-09-11T08:08:55.358990+0000","last_peered":"2024-09-11T08:08:55.358990+0000","last_clean":"2024-09-11T08:08:55.358990+0000","last_became_active":"2024-09-11T08:07:34.056528+0000","last_became_peered":"2024-09-11T08:07:34.056528+0000","last_unstale":"2024-09-11T08:08:55.358990+0000","last_undegraded":"2024-09-11T08:08:55.358990+0000","last_fullsized":"2024-09-11T08:08:55.358990+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:40.419830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:40.419830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:09.111095+0000","scrub_duration":0.0029308149999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.173624+0000","last_change":"2024-09-11T08:07:36.072658+0000","last_active":"2024-09-11T08:08:56.173624+0000","last_peered":"2024-09-11T08:08:56.173624+0000","last_clean":"2024-09-11T08:08:56.173624+0000","last_became_active":"2024-09-11T08:07:36.072347+0000","last_became_peered":"2024-09-11T08:07:36.072347+0000","last_unstale":"2024-09-11T08:08:56.173624+0000","last_undegraded":"2024-09-11T08:08:56.173624+0000","last_fullsized":"2024-09-11T08:08:56.173624+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.011099+0000","last_change":"2024-09-11T08:07:30.046498+0000","last_active":"2024-09-11T08:08:56.011099+0000","last_peered":"2024-09-11T08:08:56.011099+0000","last_clean":"2024-09-11T08:08:56.011099+0000","last_became_active":"2024-09-11T08:07:30.045932+0000","last_became_peered":"2024-09-11T08:07:30.045932+0000","last_unstale":"2024-09-11T08:08:56.011099+0000","last_undegraded":"2024-09-11T08:08:56.011099+0000","last_fullsized":"2024-09-11T08:08:56.011099+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":48,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.403204+0000","last_change":"2024-09-11T08:07:32.048372+0000","last_active":"2024-09-11T08:08:56.403204+0000","last_peered":"2024-09-11T08:08:56.403204+0000","last_clean":"2024-09-11T08:08:56.403204+0000","last_became_active":"2024-09-11T08:07:32.048022+0000","last_became_peered":"2024-09-11T08:07:32.048022+0000","last_unstale":"2024-09-11T08:08:56.403204+0000","last_undegraded":"2024-09-11T08:08:56.403204+0000","last_fullsized":"2024-09-11T08:08:56.403204+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":70,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:03.560173+0000","last_change":"2024-09-11T08:08:54.652958+0000","last_active":"2024-09-11T08:09:03.560173+0000","last_peered":"2024-09-11T08:09:03.560173+0000","last_clean":"2024-09-11T08:09:03.560173+0000","last_became_active":"2024-09-11T08:07:34.054014+0000","last_became_peered":"2024-09-11T08:07:34.054014+0000","last_unstale":"2024-09-11T08:09:03.560173+0000","last_undegraded":"2024-09-11T08:09:03.560173+0000","last_fullsized":"2024-09-11T08:09:03.560173+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:08:54.652867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:54.652867+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:09.166003+0000","scrub_duration":0.0022231320000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":68,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.011633+0000","last_change":"2024-09-11T08:08:45.358105+0000","last_active":"2024-09-11T08:08:56.011633+0000","last_peered":"2024-09-11T08:08:56.011633+0000","last_clean":"2024-09-11T08:08:56.011633+0000","last_became_active":"2024-09-11T08:07:32.054317+0000","last_became_peered":"2024-09-11T08:07:32.054317+0000","last_unstale":"2024-09-11T08:08:56.011633+0000","last_undegraded":"2024-09-11T08:08:56.011633+0000","last_fullsized":"2024-09-11T08:08:56.011633+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-11T08:08:45.358041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:45.358041+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:50.858524+0000","scrub_duration":0.0021022879999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.360164+0000","last_change":"2024-09-11T08:07:30.044933+0000","last_active":"2024-09-11T08:08:55.360164+0000","last_peered":"2024-09-11T08:08:55.360164+0000","last_clean":"2024-09-11T08:08:55.360164+0000","last_became_active":"2024-09-11T08:07:30.044624+0000","last_became_peered":"2024-09-11T08:07:30.044624+0000","last_unstale":"2024-09-11T08:08:55.360164+0000","last_undegraded":"2024-09-11T08:08:55.360164+0000","last_fullsized":"2024-09-11T08:08:55.360164+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":58,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:03.560184+0000","last_change":"2024-09-11T08:07:34.053292+0000","last_active":"2024-09-11T08:09:03.560184+0000","last_peered":"2024-09-11T08:09:03.560184+0000","last_clean":"2024-09-11T08:09:03.560184+0000","last_became_active":"2024-09-11T08:07:34.052981+0000","last_became_peered":"2024-09-11T08:07:34.052981+0000","last_unstale":"2024-09-11T08:09:03.560184+0000","last_undegraded":"2024-09-11T08:09:03.560184+0000","last_fullsized":"2024-09-11T08:09:03.560184+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":38,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.294967+0000","last_change":"2024-09-11T08:08:51.893846+0000","last_active":"2024-09-11T08:08:56.294967+0000","last_peered":"2024-09-11T08:08:56.294967+0000","last_clean":"2024-09-11T08:08:56.294967+0000","last_became_active":"2024-09-11T08:07:36.073969+0000","last_became_peered":"2024-09-11T08:07:36.073969+0000","last_unstale":"2024-09-11T08:08:56.294967+0000","last_undegraded":"2024-09-11T08:08:56.294967+0000","last_fullsized":"2024-09-11T08:08:56.294967+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:51.893774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:08:51.893774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:00.876260+0000","scrub_duration":0.001532858,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":75,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.359977+0000","last_change":"2024-09-11T08:08:43.441600+0000","last_active":"2024-09-11T08:08:55.359977+0000","last_peered":"2024-09-11T08:08:55.359977+0000","last_clean":"2024-09-11T08:08:55.359977+0000","last_became_active":"2024-09-11T08:07:32.045332+0000","last_became_peered":"2024-09-11T08:07:32.045332+0000","last_unstale":"2024-09-11T08:08:55.359977+0000","last_undegraded":"2024-09-11T08:08:55.359977+0000","last_fullsized":"2024-09-11T08:08:55.359977+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-11T08:08:43.441544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:43.441544+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:55.649394+0000","scrub_duration":0.002168648,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.360438+0000","last_change":"2024-09-11T08:07:30.047446+0000","last_active":"2024-09-11T08:08:55.360438+0000","last_peered":"2024-09-11T08:08:55.360438+0000","last_clean":"2024-09-11T08:08:55.360438+0000","last_became_active":"2024-09-11T08:07:30.047176+0000","last_became_peered":"2024-09-11T08:07:30.047176+0000","last_unstale":"2024-09-11T08:08:55.360438+0000","last_undegraded":"2024-09-11T08:08:55.360438+0000","last_fullsized":"2024-09-11T08:08:55.360438+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":31,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.358854+0000","last_change":"2024-09-11T08:07:34.055816+0000","last_active":"2024-09-11T08:08:55.358854+0000","last_peered":"2024-09-11T08:08:55.358854+0000","last_clean":"2024-09-11T08:08:55.358854+0000","last_became_active":"2024-09-11T08:07:34.055523+0000","last_became_peered":"2024-09-11T08:07:34.055523+0000","last_unstale":"2024-09-11T08:08:55.358854+0000","last_undegraded":"2024-09-11T08:08:55.358854+0000","last_fullsized":"2024-09-11T08:08:55.358854+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":29,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.359373+0000","last_change":"2024-09-11T08:07:36.069535+0000","last_active":"2024-09-11T08:08:55.359373+0000","last_peered":"2024-09-11T08:08:55.359373+0000","last_clean":"2024-09-11T08:08:55.359373+0000","last_became_active":"2024-09-11T08:07:36.069225+0000","last_became_peered":"2024-09-11T08:07:36.069225+0000","last_unstale":"2024-09-11T08:08:55.359373+0000","last_undegraded":"2024-09-11T08:08:55.359373+0000","last_fullsized":"2024-09-11T08:08:55.359373+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":54,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.358983+0000","last_change":"2024-09-11T08:07:32.040334+0000","last_active":"2024-09-11T08:08:55.358983+0000","last_peered":"2024-09-11T08:08:55.358983+0000","last_clean":"2024-09-11T08:08:55.358983+0000","last_became_active":"2024-09-11T08:07:32.040069+0000","last_became_peered":"2024-09-11T08:07:32.040069+0000","last_unstale":"2024-09-11T08:08:55.358983+0000","last_undegraded":"2024-09-11T08:08:55.358983+0000","last_fullsized":"2024-09-11T08:08:55.358983+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.960723+0000","last_change":"2024-09-11T08:09:06.960696+0000","last_active":"2024-09-11T08:09:06.960723+0000","last_peered":"2024-09-11T08:09:06.960723+0000","last_clean":"2024-09-11T08:09:06.960723+0000","last_became_active":"2024-09-11T08:07:30.050484+0000","last_became_peered":"2024-09-11T08:07:30.050484+0000","last_unstale":"2024-09-11T08:09:06.960723+0000","last_undegraded":"2024-09-11T08:09:06.960723+0000","last_fullsized":"2024-09-11T08:09:06.960723+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:06.960639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:09:06.960639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:19.133864+0000","scrub_duration":0.001342766,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"82'9","reported_seq":72,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:03.560188+0000","last_change":"2024-09-11T08:08:53.421634+0000","last_active":"2024-09-11T08:09:03.560188+0000","last_peered":"2024-09-11T08:09:03.560188+0000","last_clean":"2024-09-11T08:09:03.560188+0000","last_became_active":"2024-09-11T08:07:34.056143+0000","last_became_peered":"2024-09-11T08:07:34.056143+0000","last_unstale":"2024-09-11T08:09:03.560188+0000","last_undegraded":"2024-09-11T08:09:03.560188+0000","last_fullsized":"2024-09-11T08:09:03.560188+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-11T08:08:53.421587+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-09-11T08:08:53.421587+0000","last_clean_scrub_stamp":"2024-09-11T08:08:53.421587+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:11.501044+0000","scrub_duration":0.0021116799999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.174126+0000","last_change":"2024-09-11T08:08:55.249190+0000","last_active":"2024-09-11T08:08:56.174126+0000","last_peered":"2024-09-11T08:08:56.174126+0000","last_clean":"2024-09-11T08:08:56.174126+0000","last_became_active":"2024-09-11T08:07:36.633432+0000","last_became_peered":"2024-09-11T08:07:36.633432+0000","last_unstale":"2024-09-11T08:08:56.174126+0000","last_undegraded":"2024-09-11T08:08:56.174126+0000","last_fullsized":"2024-09-11T08:08:56.174126+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:55.249081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:08:55.249081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:00.965734+0000","scrub_duration":0.001581839,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":48,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.011183+0000","last_change":"2024-09-11T08:07:32.054861+0000","last_active":"2024-09-11T08:08:56.011183+0000","last_peered":"2024-09-11T08:08:56.011183+0000","last_clean":"2024-09-11T08:08:56.011183+0000","last_became_active":"2024-09-11T08:07:32.054294+0000","last_became_peered":"2024-09-11T08:07:32.054294+0000","last_unstale":"2024-09-11T08:08:56.011183+0000","last_undegraded":"2024-09-11T08:08:56.011183+0000","last_fullsized":"2024-09-11T08:08:56.011183+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:00.819977+0000","last_change":"2024-09-11T08:09:00.819941+0000","last_active":"2024-09-11T08:09:00.819977+0000","last_peered":"2024-09-11T08:09:00.819977+0000","last_clean":"2024-09-11T08:09:00.819977+0000","last_became_active":"2024-09-11T08:07:30.047497+0000","last_became_peered":"2024-09-11T08:07:30.047497+0000","last_unstale":"2024-09-11T08:09:00.819977+0000","last_undegraded":"2024-09-11T08:09:00.819977+0000","last_fullsized":"2024-09-11T08:09:00.819977+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:00.819884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:09:00.819884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:12.227536+0000","scrub_duration":0.0014673640000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":58,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:03.560153+0000","last_change":"2024-09-11T08:07:34.060999+0000","last_active":"2024-09-11T08:09:03.560153+0000","last_peered":"2024-09-11T08:09:03.560153+0000","last_clean":"2024-09-11T08:09:03.560153+0000","last_became_active":"2024-09-11T08:07:34.060660+0000","last_became_peered":"2024-09-11T08:07:34.060660+0000","last_unstale":"2024-09-11T08:09:03.560153+0000","last_undegraded":"2024-09-11T08:09:03.560153+0000","last_fullsized":"2024-09-11T08:09:03.560153+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"66'1","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.402521+0000","last_change":"2024-09-11T08:07:36.072890+0000","last_active":"2024-09-11T08:08:56.402521+0000","last_peered":"2024-09-11T08:08:56.402521+0000","last_clean":"2024-09-11T08:08:56.402521+0000","last_became_active":"2024-09-11T08:07:36.072593+0000","last_became_peered":"2024-09-11T08:07:36.072593+0000","last_unstale":"2024-09-11T08:08:56.402521+0000","last_undegraded":"2024-09-11T08:08:56.402521+0000","last_fullsized":"2024-09-11T08:08:56.402521+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":79,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.402986+0000","last_change":"2024-09-11T08:08:52.842005+0000","last_active":"2024-09-11T08:08:56.402986+0000","last_peered":"2024-09-11T08:08:56.402986+0000","last_clean":"2024-09-11T08:08:56.402986+0000","last_became_active":"2024-09-11T08:07:32.044376+0000","last_became_peered":"2024-09-11T08:07:32.044376+0000","last_unstale":"2024-09-11T08:08:56.402986+0000","last_undegraded":"2024-09-11T08:08:56.402986+0000","last_fullsized":"2024-09-11T08:08:56.402986+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-11T08:08:52.841903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:52.841903+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:08.737667+0000","scrub_duration":0.0025248150000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.011437+0000","last_change":"2024-09-11T08:08:49.384423+0000","last_active":"2024-09-11T08:08:56.011437+0000","last_peered":"2024-09-11T08:08:56.011437+0000","last_clean":"2024-09-11T08:08:56.011437+0000","last_became_active":"2024-09-11T08:07:30.045380+0000","last_became_peered":"2024-09-11T08:07:30.045380+0000","last_unstale":"2024-09-11T08:08:56.011437+0000","last_undegraded":"2024-09-11T08:08:56.011437+0000","last_fullsized":"2024-09-11T08:08:56.011437+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:49.384357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:49.384357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:50.840709+0000","scrub_duration":0.0012073310000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.402455+0000","last_change":"2024-09-11T08:07:34.051286+0000","last_active":"2024-09-11T08:08:56.402455+0000","last_peered":"2024-09-11T08:08:56.402455+0000","last_clean":"2024-09-11T08:08:56.402455+0000","last_became_active":"2024-09-11T08:07:34.051004+0000","last_became_peered":"2024-09-11T08:07:34.051004+0000","last_unstale":"2024-09-11T08:08:56.402455+0000","last_undegraded":"2024-09-11T08:08:56.402455+0000","last_fullsized":"2024-09-11T08:08:56.402455+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":29,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.010952+0000","last_change":"2024-09-11T08:07:36.632119+0000","last_active":"2024-09-11T08:08:56.010952+0000","last_peered":"2024-09-11T08:08:56.010952+0000","last_clean":"2024-09-11T08:08:56.010952+0000","last_became_active":"2024-09-11T08:07:36.631668+0000","last_became_peered":"2024-09-11T08:07:36.631668+0000","last_unstale":"2024-09-11T08:08:56.010952+0000","last_undegraded":"2024-09-11T08:08:56.010952+0000","last_fullsized":"2024-09-11T08:08:56.010952+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"66'28","reported_seq":100,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:03.560160+0000","last_change":"2024-09-11T08:07:32.053328+0000","last_active":"2024-09-11T08:09:03.560160+0000","last_peered":"2024-09-11T08:09:03.560160+0000","last_clean":"2024-09-11T08:09:03.560160+0000","last_became_active":"2024-09-11T08:07:32.052956+0000","last_became_peered":"2024-09-11T08:07:32.052956+0000","last_unstale":"2024-09-11T08:09:03.560160+0000","last_undegraded":"2024-09-11T08:09:03.560160+0000","last_fullsized":"2024-09-11T08:09:03.560160+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"59'1","reported_seq":54,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.359638+0000","last_change":"2024-09-11T08:08:34.423550+0000","last_active":"2024-09-11T08:08:55.359638+0000","last_peered":"2024-09-11T08:08:55.359638+0000","last_clean":"2024-09-11T08:08:55.359638+0000","last_became_active":"2024-09-11T08:07:30.047513+0000","last_became_peered":"2024-09-11T08:07:30.047513+0000","last_unstale":"2024-09-11T08:08:55.359638+0000","last_undegraded":"2024-09-11T08:08:55.359638+0000","last_fullsized":"2024-09-11T08:08:55.359638+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-11T08:08:34.423474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:34.423474+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:57.535838+0000","scrub_duration":0.0016749149999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":51,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.294761+0000","last_change":"2024-09-11T08:08:37.859401+0000","last_active":"2024-09-11T08:08:56.294761+0000","last_peered":"2024-09-11T08:08:56.294761+0000","last_clean":"2024-09-11T08:08:56.294761+0000","last_became_active":"2024-09-11T08:07:30.043130+0000","last_became_peered":"2024-09-11T08:07:30.043130+0000","last_unstale":"2024-09-11T08:08:56.294761+0000","last_undegraded":"2024-09-11T08:08:56.294761+0000","last_fullsized":"2024-09-11T08:08:56.294761+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-11T08:08:37.859328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:37.859328+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:48.485842+0000","scrub_duration":0.001679816,"objects_trimmed":0,"snaptrim_duration":0.0026120850000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.403121+0000","last_change":"2024-09-11T08:08:46.959561+0000","last_active":"2024-09-11T08:08:56.403121+0000","last_peered":"2024-09-11T08:08:56.403121+0000","last_clean":"2024-09-11T08:08:56.403121+0000","last_became_active":"2024-09-11T08:07:34.057154+0000","last_became_peered":"2024-09-11T08:07:34.057154+0000","last_unstale":"2024-09-11T08:08:56.403121+0000","last_undegraded":"2024-09-11T08:08:56.403121+0000","last_fullsized":"2024-09-11T08:08:56.403121+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:46.959500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:46.959500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:05.161397+0000","scrub_duration":0.001608806,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":29,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.359854+0000","last_change":"2024-09-11T08:07:36.076211+0000","last_active":"2024-09-11T08:08:55.359854+0000","last_peered":"2024-09-11T08:08:55.359854+0000","last_clean":"2024-09-11T08:08:55.359854+0000","last_became_active":"2024-09-11T08:07:36.075690+0000","last_became_peered":"2024-09-11T08:07:36.075690+0000","last_unstale":"2024-09-11T08:08:55.359854+0000","last_undegraded":"2024-09-11T08:08:55.359854+0000","last_fullsized":"2024-09-11T08:08:55.359854+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":70,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.011646+0000","last_change":"2024-09-11T08:08:35.441946+0000","last_active":"2024-09-11T08:08:56.011646+0000","last_peered":"2024-09-11T08:08:56.011646+0000","last_clean":"2024-09-11T08:08:56.011646+0000","last_became_active":"2024-09-11T08:07:32.050673+0000","last_became_peered":"2024-09-11T08:07:32.050673+0000","last_unstale":"2024-09-11T08:08:56.011646+0000","last_undegraded":"2024-09-11T08:08:56.011646+0000","last_fullsized":"2024-09-11T08:08:56.011646+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'13","last_scrub_stamp":"2024-09-11T08:08:35.441894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:35.441894+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:02.914423+0000","scrub_duration":0.001888965,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.011256+0000","last_change":"2024-09-11T08:08:33.492709+0000","last_active":"2024-09-11T08:08:56.011256+0000","last_peered":"2024-09-11T08:08:56.011256+0000","last_clean":"2024-09-11T08:08:56.011256+0000","last_became_active":"2024-09-11T08:07:30.047014+0000","last_became_peered":"2024-09-11T08:07:30.047014+0000","last_unstale":"2024-09-11T08:08:56.011256+0000","last_undegraded":"2024-09-11T08:08:56.011256+0000","last_fullsized":"2024-09-11T08:08:56.011256+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:33.492652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:33.492652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:54.786224+0000","scrub_duration":0.00156573,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":50,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.359069+0000","last_change":"2024-09-11T08:08:32.909292+0000","last_active":"2024-09-11T08:08:55.359069+0000","last_peered":"2024-09-11T08:08:55.359069+0000","last_clean":"2024-09-11T08:08:55.359069+0000","last_became_active":"2024-09-11T08:07:30.045009+0000","last_became_peered":"2024-09-11T08:07:30.045009+0000","last_unstale":"2024-09-11T08:08:55.359069+0000","last_undegraded":"2024-09-11T08:08:55.359069+0000","last_fullsized":"2024-09-11T08:08:55.359069+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-11T08:08:32.909183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:32.909183+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:02.458449+0000","scrub_duration":0.001625769,"objects_trimmed":0,"snaptrim_duration":0.001859533,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:58.000043+0000","last_change":"2024-09-11T08:08:58.000006+0000","last_active":"2024-09-11T08:08:58.000043+0000","last_peered":"2024-09-11T08:08:58.000043+0000","last_clean":"2024-09-11T08:08:58.000043+0000","last_became_active":"2024-09-11T08:07:34.063906+0000","last_became_peered":"2024-09-11T08:07:34.063906+0000","last_unstale":"2024-09-11T08:08:58.000043+0000","last_undegraded":"2024-09-11T08:08:58.000043+0000","last_fullsized":"2024-09-11T08:08:58.000043+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:57.999952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:57.999952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:01.053836+0000","scrub_duration":0.0014942130000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.174225+0000","last_change":"2024-09-11T08:08:46.366578+0000","last_active":"2024-09-11T08:08:56.174225+0000","last_peered":"2024-09-11T08:08:56.174225+0000","last_clean":"2024-09-11T08:08:56.174225+0000","last_became_active":"2024-09-11T08:07:36.631480+0000","last_became_peered":"2024-09-11T08:07:36.631480+0000","last_unstale":"2024-09-11T08:08:56.174225+0000","last_undegraded":"2024-09-11T08:08:56.174225+0000","last_fullsized":"2024-09-11T08:08:56.174225+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:46.366508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:08:46.366508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:52.248237+0000","scrub_duration":0.0015317200000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":61,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.403259+0000","last_change":"2024-09-11T08:08:37.115649+0000","last_active":"2024-09-11T08:08:56.403259+0000","last_peered":"2024-09-11T08:08:56.403259+0000","last_clean":"2024-09-11T08:08:56.403259+0000","last_became_active":"2024-09-11T08:07:32.055407+0000","last_became_peered":"2024-09-11T08:07:32.055407+0000","last_unstale":"2024-09-11T08:08:56.403259+0000","last_undegraded":"2024-09-11T08:08:56.403259+0000","last_fullsized":"2024-09-11T08:08:56.403259+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-11T08:08:37.115582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:37.115582+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:57.869478+0000","scrub_duration":0.0020267729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"59'1","reported_seq":52,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.402768+0000","last_change":"2024-09-11T08:08:41.094879+0000","last_active":"2024-09-11T08:08:56.402768+0000","last_peered":"2024-09-11T08:08:56.402768+0000","last_clean":"2024-09-11T08:08:56.402768+0000","last_became_active":"2024-09-11T08:07:30.050504+0000","last_became_peered":"2024-09-11T08:07:30.050504+0000","last_unstale":"2024-09-11T08:08:56.402768+0000","last_undegraded":"2024-09-11T08:08:56.402768+0000","last_fullsized":"2024-09-11T08:08:56.402768+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-11T08:08:41.094821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:41.094821+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:00.561470+0000","scrub_duration":0.0017674310000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":144,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.870958+0000","last_change":"2024-09-11T08:09:01.175592+0000","last_active":"2024-09-11T08:09:06.870958+0000","last_peered":"2024-09-11T08:09:06.870958+0000","last_clean":"2024-09-11T08:09:06.870958+0000","last_became_active":"2024-09-11T08:07:30.045051+0000","last_became_peered":"2024-09-11T08:07:30.045051+0000","last_unstale":"2024-09-11T08:09:06.870958+0000","last_undegraded":"2024-09-11T08:09:06.870958+0000","last_fullsized":"2024-09-11T08:09:06.870958+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-11T08:09:01.175530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:09:01.175530+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:30.186314+0000","scrub_duration":0.001752651,"objects_trimmed":0,"snaptrim_duration":0.000318821,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":91,"num_read_kb":86,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:05.758231+0000","last_change":"2024-09-11T08:09:05.758196+0000","last_active":"2024-09-11T08:09:05.758231+0000","last_peered":"2024-09-11T08:09:05.758231+0000","last_clean":"2024-09-11T08:09:05.758231+0000","last_became_active":"2024-09-11T08:07:34.058962+0000","last_became_peered":"2024-09-11T08:07:34.058962+0000","last_unstale":"2024-09-11T08:09:05.758231+0000","last_undegraded":"2024-09-11T08:09:05.758231+0000","last_fullsized":"2024-09-11T08:09:05.758231+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:05.758141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:09:05.758141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:09.559173+0000","scrub_duration":0.0012345450000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.011166+0000","last_change":"2024-09-11T08:08:50.346126+0000","last_active":"2024-09-11T08:08:56.011166+0000","last_peered":"2024-09-11T08:08:56.011166+0000","last_clean":"2024-09-11T08:08:56.011166+0000","last_became_active":"2024-09-11T08:07:36.077597+0000","last_became_peered":"2024-09-11T08:07:36.077597+0000","last_unstale":"2024-09-11T08:08:56.011166+0000","last_undegraded":"2024-09-11T08:08:56.011166+0000","last_fullsized":"2024-09-11T08:08:56.011166+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:50.346069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:08:50.346069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:15.900195+0000","scrub_duration":0.001294906,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"84'21","reported_seq":71,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.705429+0000","last_change":"2024-09-11T08:08:56.705362+0000","last_active":"2024-09-11T08:08:56.705429+0000","last_peered":"2024-09-11T08:08:56.705429+0000","last_clean":"2024-09-11T08:08:56.705429+0000","last_became_active":"2024-09-11T08:07:32.042339+0000","last_became_peered":"2024-09-11T08:07:32.042339+0000","last_unstale":"2024-09-11T08:08:56.705429+0000","last_undegraded":"2024-09-11T08:08:56.705429+0000","last_fullsized":"2024-09-11T08:08:56.705429+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'21","last_scrub_stamp":"2024-09-11T08:08:56.705306+0000","last_deep_scrub":"84'21","last_deep_scrub_stamp":"2024-09-11T08:08:56.705306+0000","last_clean_scrub_stamp":"2024-09-11T08:08:56.705306+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:14.758337+0000","scrub_duration":0.0050944290000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.360355+0000","last_change":"2024-09-11T08:07:30.047535+0000","last_active":"2024-09-11T08:08:55.360355+0000","last_peered":"2024-09-11T08:08:55.360355+0000","last_clean":"2024-09-11T08:08:55.360355+0000","last_became_active":"2024-09-11T08:07:30.047246+0000","last_became_peered":"2024-09-11T08:07:30.047246+0000","last_unstale":"2024-09-11T08:08:55.360355+0000","last_undegraded":"2024-09-11T08:08:55.360355+0000","last_fullsized":"2024-09-11T08:08:55.360355+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":81,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.174050+0000","last_change":"2024-09-11T08:07:44.054743+0000","last_active":"2024-09-11T08:08:56.174050+0000","last_peered":"2024-09-11T08:08:56.174050+0000","last_clean":"2024-09-11T08:08:56.174050+0000","last_became_active":"2024-09-11T08:07:19.726378+0000","last_became_peered":"2024-09-11T08:07:19.726378+0000","last_unstale":"2024-09-11T08:08:56.174050+0000","last_undegraded":"2024-09-11T08:08:56.174050+0000","last_fullsized":"2024-09-11T08:08:56.174050+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'72","last_scrub_stamp":"2024-09-11T08:07:44.054686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:05:21.231856+0000","last_clean_scrub_stamp":"2024-09-11T08:07:44.054686+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001872931,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.174294+0000","last_change":"2024-09-11T08:08:51.289948+0000","last_active":"2024-09-11T08:08:56.174294+0000","last_peered":"2024-09-11T08:08:56.174294+0000","last_clean":"2024-09-11T08:08:56.174294+0000","last_became_active":"2024-09-11T08:07:34.058414+0000","last_became_peered":"2024-09-11T08:07:34.058414+0000","last_unstale":"2024-09-11T08:08:56.174294+0000","last_undegraded":"2024-09-11T08:08:56.174294+0000","last_fullsized":"2024-09-11T08:08:56.174294+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:51.289872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:51.289872+0000","last_clean_scrub_stamp":"2024-09-11T08:08:51.289872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:08.912691+0000","scrub_duration":0.001988812,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":38,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:01.777595+0000","last_change":"2024-09-11T08:09:01.777522+0000","last_active":"2024-09-11T08:09:01.777595+0000","last_peered":"2024-09-11T08:09:01.777595+0000","last_clean":"2024-09-11T08:09:01.777595+0000","last_became_active":"2024-09-11T08:07:36.072508+0000","last_became_peered":"2024-09-11T08:07:36.072508+0000","last_unstale":"2024-09-11T08:09:01.777595+0000","last_undegraded":"2024-09-11T08:09:01.777595+0000","last_fullsized":"2024-09-11T08:09:01.777595+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:01.777429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:01.777429+0000","last_clean_scrub_stamp":"2024-09-11T08:09:01.777429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:07.815841+0000","scrub_duration":0.001444961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"66'11","reported_seq":62,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.359357+0000","last_change":"2024-09-11T08:08:35.418440+0000","last_active":"2024-09-11T08:08:55.359357+0000","last_peered":"2024-09-11T08:08:55.359357+0000","last_clean":"2024-09-11T08:08:55.359357+0000","last_became_active":"2024-09-11T08:07:32.047971+0000","last_became_peered":"2024-09-11T08:07:32.047971+0000","last_unstale":"2024-09-11T08:08:55.359357+0000","last_undegraded":"2024-09-11T08:08:55.359357+0000","last_fullsized":"2024-09-11T08:08:55.359357+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-11T08:08:35.418378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:35.418378+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:38.165860+0000","scrub_duration":0.001791207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.359103+0000","last_change":"2024-09-11T08:08:36.417975+0000","last_active":"2024-09-11T08:08:55.359103+0000","last_peered":"2024-09-11T08:08:55.359103+0000","last_clean":"2024-09-11T08:08:55.359103+0000","last_became_active":"2024-09-11T08:07:30.039884+0000","last_became_peered":"2024-09-11T08:07:30.039884+0000","last_unstale":"2024-09-11T08:08:55.359103+0000","last_undegraded":"2024-09-11T08:08:55.359103+0000","last_fullsized":"2024-09-11T08:08:55.359103+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:36.417911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:36.417911+0000","last_clean_scrub_stamp":"2024-09-11T08:08:36.417911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:59.734565+0000","scrub_duration":0.0014387580000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:58.845189+0000","last_change":"2024-09-11T08:08:58.845142+0000","last_active":"2024-09-11T08:08:58.845189+0000","last_peered":"2024-09-11T08:08:58.845189+0000","last_clean":"2024-09-11T08:08:58.845189+0000","last_became_active":"2024-09-11T08:07:34.053347+0000","last_became_peered":"2024-09-11T08:07:34.053347+0000","last_unstale":"2024-09-11T08:08:58.845189+0000","last_undegraded":"2024-09-11T08:08:58.845189+0000","last_fullsized":"2024-09-11T08:08:58.845189+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:58.845085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:58.845085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:08.089077+0000","scrub_duration":0.001233404,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":29,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.360446+0000","last_change":"2024-09-11T08:07:36.632329+0000","last_active":"2024-09-11T08:08:55.360446+0000","last_peered":"2024-09-11T08:08:55.360446+0000","last_clean":"2024-09-11T08:08:55.360446+0000","last_became_active":"2024-09-11T08:07:36.631687+0000","last_became_peered":"2024-09-11T08:07:36.631687+0000","last_unstale":"2024-09-11T08:08:55.360446+0000","last_undegraded":"2024-09-11T08:08:55.360446+0000","last_fullsized":"2024-09-11T08:08:55.360446+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"77'25","reported_seq":81,"reported_epoch":85,"state":"peering","last_fresh":"2024-09-11T08:09:06.872554+0000","last_change":"2024-09-11T08:09:06.872760+0000","last_active":"2024-09-11T08:09:06.872524+0000","last_peered":"2024-09-11T08:08:55.359257+0000","last_clean":"2024-09-11T08:08:55.359257+0000","last_became_active":"2024-09-11T08:07:32.050815+0000","last_became_peered":"2024-09-11T08:07:32.050815+0000","last_unstale":"2024-09-11T08:09:06.872554+0000","last_undegraded":"2024-09-11T08:09:06.872554+0000","last_fullsized":"2024-09-11T08:09:06.872554+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'25","last_scrub_stamp":"2024-09-11T08:08:32.458210+0000","last_deep_scrub":"77'25","last_deep_scrub_stamp":"2024-09-11T08:08:32.458210+0000","last_clean_scrub_stamp":"2024-09-11T08:08:32.458210+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:53.685411+0000","scrub_duration":0.0060523319999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,5],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":34,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.402849+0000","last_change":"2024-09-11T08:07:30.042789+0000","last_active":"2024-09-11T08:08:56.402849+0000","last_peered":"2024-09-11T08:08:56.402849+0000","last_clean":"2024-09-11T08:08:56.402849+0000","last_became_active":"2024-09-11T08:07:30.042555+0000","last_became_peered":"2024-09-11T08:07:30.042555+0000","last_unstale":"2024-09-11T08:08:56.402849+0000","last_undegraded":"2024-09-11T08:08:56.402849+0000","last_fullsized":"2024-09-11T08:08:56.402849+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.011250+0000","last_change":"2024-09-11T08:08:41.437100+0000","last_active":"2024-09-11T08:08:56.011250+0000","last_peered":"2024-09-11T08:08:56.011250+0000","last_clean":"2024-09-11T08:08:56.011250+0000","last_became_active":"2024-09-11T08:07:34.054961+0000","last_became_peered":"2024-09-11T08:07:34.054961+0000","last_unstale":"2024-09-11T08:08:56.011250+0000","last_undegraded":"2024-09-11T08:08:56.011250+0000","last_fullsized":"2024-09-11T08:08:56.011250+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:41.437045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:41.437045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:10.512019+0000","scrub_duration":0.0029926649999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":38,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:04.031993+0000","last_change":"2024-09-11T08:09:04.031954+0000","last_active":"2024-09-11T08:09:04.031993+0000","last_peered":"2024-09-11T08:09:04.031993+0000","last_clean":"2024-09-11T08:09:04.031993+0000","last_became_active":"2024-09-11T08:07:36.071194+0000","last_became_peered":"2024-09-11T08:07:36.071194+0000","last_unstale":"2024-09-11T08:09:04.031993+0000","last_undegraded":"2024-09-11T08:09:04.031993+0000","last_fullsized":"2024-09-11T08:09:04.031993+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:04.031894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:09:04.031894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:28.971103+0000","scrub_duration":0.0013516579999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":58,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.359082+0000","last_change":"2024-09-11T08:08:42.447971+0000","last_active":"2024-09-11T08:08:55.359082+0000","last_peered":"2024-09-11T08:08:55.359082+0000","last_clean":"2024-09-11T08:08:55.359082+0000","last_became_active":"2024-09-11T08:07:32.043314+0000","last_became_peered":"2024-09-11T08:07:32.043314+0000","last_unstale":"2024-09-11T08:08:55.359082+0000","last_undegraded":"2024-09-11T08:08:55.359082+0000","last_fullsized":"2024-09-11T08:08:55.359082+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-11T08:08:42.447892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:42.447892+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:03.778832+0000","scrub_duration":0.0039015859999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.359949+0000","last_change":"2024-09-11T08:08:30.327580+0000","last_active":"2024-09-11T08:08:55.359949+0000","last_peered":"2024-09-11T08:08:55.359949+0000","last_clean":"2024-09-11T08:08:55.359949+0000","last_became_active":"2024-09-11T08:07:30.046684+0000","last_became_peered":"2024-09-11T08:07:30.046684+0000","last_unstale":"2024-09-11T08:08:55.359949+0000","last_undegraded":"2024-09-11T08:08:55.359949+0000","last_fullsized":"2024-09-11T08:08:55.359949+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:30.327525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:30.327525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:57.278810+0000","scrub_duration":0.001475589,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"66'8","reported_seq":67,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:03.560198+0000","last_change":"2024-09-11T08:08:49.026859+0000","last_active":"2024-09-11T08:09:03.560198+0000","last_peered":"2024-09-11T08:09:03.560198+0000","last_clean":"2024-09-11T08:09:03.560198+0000","last_became_active":"2024-09-11T08:07:34.062469+0000","last_became_peered":"2024-09-11T08:07:34.062469+0000","last_unstale":"2024-09-11T08:09:03.560198+0000","last_undegraded":"2024-09-11T08:09:03.560198+0000","last_fullsized":"2024-09-11T08:09:03.560198+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:08:49.026755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:49.026755+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:59.460486+0000","scrub_duration":0.0018077970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":29,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.359109+0000","last_change":"2024-09-11T08:07:36.074961+0000","last_active":"2024-09-11T08:08:55.359109+0000","last_peered":"2024-09-11T08:08:55.359109+0000","last_clean":"2024-09-11T08:08:55.359109+0000","last_became_active":"2024-09-11T08:07:36.074684+0000","last_became_peered":"2024-09-11T08:07:36.074684+0000","last_unstale":"2024-09-11T08:08:55.359109+0000","last_undegraded":"2024-09-11T08:08:55.359109+0000","last_fullsized":"2024-09-11T08:08:55.359109+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":49,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.403006+0000","last_change":"2024-09-11T08:07:32.055237+0000","last_active":"2024-09-11T08:08:56.403006+0000","last_peered":"2024-09-11T08:08:56.403006+0000","last_clean":"2024-09-11T08:08:56.403006+0000","last_became_active":"2024-09-11T08:07:32.054885+0000","last_became_peered":"2024-09-11T08:07:32.054885+0000","last_unstale":"2024-09-11T08:08:56.403006+0000","last_undegraded":"2024-09-11T08:08:56.403006+0000","last_fullsized":"2024-09-11T08:08:56.403006+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.295274+0000","last_change":"2024-09-11T08:08:34.828231+0000","last_active":"2024-09-11T08:08:56.295274+0000","last_peered":"2024-09-11T08:08:56.295274+0000","last_clean":"2024-09-11T08:08:56.295274+0000","last_became_active":"2024-09-11T08:07:30.045613+0000","last_became_peered":"2024-09-11T08:07:30.045613+0000","last_unstale":"2024-09-11T08:08:56.295274+0000","last_undegraded":"2024-09-11T08:08:56.295274+0000","last_fullsized":"2024-09-11T08:08:56.295274+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:34.828129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:34.828129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:58.397313+0000","scrub_duration":0.0013782530000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.359164+0000","last_change":"2024-09-11T08:08:37.871017+0000","last_active":"2024-09-11T08:08:55.359164+0000","last_peered":"2024-09-11T08:08:55.359164+0000","last_clean":"2024-09-11T08:08:55.359164+0000","last_became_active":"2024-09-11T08:07:34.056606+0000","last_became_peered":"2024-09-11T08:07:34.056606+0000","last_unstale":"2024-09-11T08:08:55.359164+0000","last_undegraded":"2024-09-11T08:08:55.359164+0000","last_fullsized":"2024-09-11T08:08:55.359164+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:37.870955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:37.870955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:58.505876+0000","scrub_duration":0.0016274460000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:05.932313+0000","last_change":"2024-09-11T08:09:05.932261+0000","last_active":"2024-09-11T08:09:05.932313+0000","last_peered":"2024-09-11T08:09:05.932313+0000","last_clean":"2024-09-11T08:09:05.932313+0000","last_became_active":"2024-09-11T08:07:36.072452+0000","last_became_peered":"2024-09-11T08:07:36.072452+0000","last_unstale":"2024-09-11T08:09:05.932313+0000","last_undegraded":"2024-09-11T08:09:05.932313+0000","last_fullsized":"2024-09-11T08:09:05.932313+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:05.932180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:09:05.932180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:07.863587+0000","scrub_duration":0.0013481960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":73,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.402907+0000","last_change":"2024-09-11T08:08:53.616543+0000","last_active":"2024-09-11T08:08:56.402907+0000","last_peered":"2024-09-11T08:08:56.402907+0000","last_clean":"2024-09-11T08:08:56.402907+0000","last_became_active":"2024-09-11T08:07:32.050880+0000","last_became_peered":"2024-09-11T08:07:32.050880+0000","last_unstale":"2024-09-11T08:08:56.402907+0000","last_undegraded":"2024-09-11T08:08:56.402907+0000","last_fullsized":"2024-09-11T08:08:56.402907+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-11T08:08:53.616450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:53.616450+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:18.217121+0000","scrub_duration":0.0024100559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.174487+0000","last_change":"2024-09-11T08:08:44.385440+0000","last_active":"2024-09-11T08:08:56.174487+0000","last_peered":"2024-09-11T08:08:56.174487+0000","last_clean":"2024-09-11T08:08:56.174487+0000","last_became_active":"2024-09-11T08:07:30.043294+0000","last_became_peered":"2024-09-11T08:07:30.043294+0000","last_unstale":"2024-09-11T08:08:56.174487+0000","last_undegraded":"2024-09-11T08:08:56.174487+0000","last_fullsized":"2024-09-11T08:08:56.174487+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:44.385372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:44.385372+0000","last_clean_scrub_stamp":"2024-09-11T08:08:44.385372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:46.537624+0000","scrub_duration":0.001441007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.359174+0000","last_change":"2024-09-11T08:08:51.921479+0000","last_active":"2024-09-11T08:08:55.359174+0000","last_peered":"2024-09-11T08:08:55.359174+0000","last_clean":"2024-09-11T08:08:55.359174+0000","last_became_active":"2024-09-11T08:07:34.063589+0000","last_became_peered":"2024-09-11T08:07:34.063589+0000","last_unstale":"2024-09-11T08:08:55.359174+0000","last_undegraded":"2024-09-11T08:08:55.359174+0000","last_fullsized":"2024-09-11T08:08:55.359174+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:51.921389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:51.921389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:01.266430+0000","scrub_duration":0.001807091,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.173568+0000","last_change":"2024-09-11T08:08:36.367954+0000","last_active":"2024-09-11T08:08:56.173568+0000","last_peered":"2024-09-11T08:08:56.173568+0000","last_clean":"2024-09-11T08:08:56.173568+0000","last_became_active":"2024-09-11T08:07:36.077523+0000","last_became_peered":"2024-09-11T08:07:36.077523+0000","last_unstale":"2024-09-11T08:08:56.173568+0000","last_undegraded":"2024-09-11T08:08:56.173568+0000","last_fullsized":"2024-09-11T08:08:56.173568+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:36.367890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:08:36.367890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:40.081903+0000","scrub_duration":0.0015112560000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":66,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:59.494109+0000","last_change":"2024-09-11T08:08:59.494039+0000","last_active":"2024-09-11T08:08:59.494109+0000","last_peered":"2024-09-11T08:08:59.494109+0000","last_clean":"2024-09-11T08:08:59.494109+0000","last_became_active":"2024-09-11T08:07:32.048164+0000","last_became_peered":"2024-09-11T08:07:32.048164+0000","last_unstale":"2024-09-11T08:08:59.494109+0000","last_undegraded":"2024-09-11T08:08:59.494109+0000","last_fullsized":"2024-09-11T08:08:59.494109+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-11T08:08:59.493975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:59.493975+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:17.528763+0000","scrub_duration":0.0019438280000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.174662+0000","last_change":"2024-09-11T08:08:53.247349+0000","last_active":"2024-09-11T08:08:56.174662+0000","last_peered":"2024-09-11T08:08:56.174662+0000","last_clean":"2024-09-11T08:08:56.174662+0000","last_became_active":"2024-09-11T08:07:30.041219+0000","last_became_peered":"2024-09-11T08:07:30.041219+0000","last_unstale":"2024-09-11T08:08:56.174662+0000","last_undegraded":"2024-09-11T08:08:56.174662+0000","last_fullsized":"2024-09-11T08:08:56.174662+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:53.247275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:53.247275+0000","last_clean_scrub_stamp":"2024-09-11T08:08:53.247275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:00.288102+0000","scrub_duration":0.0012568169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.359011+0000","last_change":"2024-09-11T08:08:46.970593+0000","last_active":"2024-09-11T08:08:55.359011+0000","last_peered":"2024-09-11T08:08:55.359011+0000","last_clean":"2024-09-11T08:08:55.359011+0000","last_became_active":"2024-09-11T08:07:34.057479+0000","last_became_peered":"2024-09-11T08:07:34.057479+0000","last_unstale":"2024-09-11T08:08:55.359011+0000","last_undegraded":"2024-09-11T08:08:55.359011+0000","last_fullsized":"2024-09-11T08:08:55.359011+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:46.970539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:46.970539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:09.551366+0000","scrub_duration":0.001167143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:00.395407+0000","last_change":"2024-09-11T08:09:00.395353+0000","last_active":"2024-09-11T08:09:00.395407+0000","last_peered":"2024-09-11T08:09:00.395407+0000","last_clean":"2024-09-11T08:09:00.395407+0000","last_became_active":"2024-09-11T08:07:36.078075+0000","last_became_peered":"2024-09-11T08:07:36.078075+0000","last_unstale":"2024-09-11T08:09:00.395407+0000","last_undegraded":"2024-09-11T08:09:00.395407+0000","last_fullsized":"2024-09-11T08:09:00.395407+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:00.395264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:09:00.395264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:07.333876+0000","scrub_duration":0.001470147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":57,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.294658+0000","last_change":"2024-09-11T08:07:32.039555+0000","last_active":"2024-09-11T08:08:56.294658+0000","last_peered":"2024-09-11T08:08:56.294658+0000","last_clean":"2024-09-11T08:08:56.294658+0000","last_became_active":"2024-09-11T08:07:32.039275+0000","last_became_peered":"2024-09-11T08:07:32.039275+0000","last_unstale":"2024-09-11T08:08:56.294658+0000","last_undegraded":"2024-09-11T08:08:56.294658+0000","last_fullsized":"2024-09-11T08:08:56.294658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":52,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.174704+0000","last_change":"2024-09-11T08:07:30.046228+0000","last_active":"2024-09-11T08:08:56.174704+0000","last_peered":"2024-09-11T08:08:56.174704+0000","last_clean":"2024-09-11T08:08:56.174704+0000","last_became_active":"2024-09-11T08:07:30.045967+0000","last_became_peered":"2024-09-11T08:07:30.045967+0000","last_unstale":"2024-09-11T08:08:56.174704+0000","last_undegraded":"2024-09-11T08:08:56.174704+0000","last_fullsized":"2024-09-11T08:08:56.174704+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"66'8","reported_seq":68,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:03.560270+0000","last_change":"2024-09-11T08:08:59.170974+0000","last_active":"2024-09-11T08:09:03.560270+0000","last_peered":"2024-09-11T08:09:03.560270+0000","last_clean":"2024-09-11T08:09:03.560270+0000","last_became_active":"2024-09-11T08:07:34.058015+0000","last_became_peered":"2024-09-11T08:07:34.058015+0000","last_unstale":"2024-09-11T08:09:03.560270+0000","last_undegraded":"2024-09-11T08:09:03.560270+0000","last_fullsized":"2024-09-11T08:09:03.560270+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:08:59.170918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:59.170918+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:12.511014+0000","scrub_duration":0.001468128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":28,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.294597+0000","last_change":"2024-09-11T08:07:36.632018+0000","last_active":"2024-09-11T08:08:56.294597+0000","last_peered":"2024-09-11T08:08:56.294597+0000","last_clean":"2024-09-11T08:08:56.294597+0000","last_became_active":"2024-09-11T08:07:36.631755+0000","last_became_peered":"2024-09-11T08:07:36.631755+0000","last_unstale":"2024-09-11T08:08:56.294597+0000","last_undegraded":"2024-09-11T08:08:56.294597+0000","last_fullsized":"2024-09-11T08:08:56.294597+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":34,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.402423+0000","last_change":"2024-09-11T08:07:30.046145+0000","last_active":"2024-09-11T08:08:56.402423+0000","last_peered":"2024-09-11T08:08:56.402423+0000","last_clean":"2024-09-11T08:08:56.402423+0000","last_became_active":"2024-09-11T08:07:30.045910+0000","last_became_peered":"2024-09-11T08:07:30.045910+0000","last_unstale":"2024-09-11T08:08:56.402423+0000","last_undegraded":"2024-09-11T08:08:56.402423+0000","last_fullsized":"2024-09-11T08:08:56.402423+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"84'9","reported_seq":55,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:03.418572+0000","last_change":"2024-09-11T08:09:03.418515+0000","last_active":"2024-09-11T08:09:03.418572+0000","last_peered":"2024-09-11T08:09:03.418572+0000","last_clean":"2024-09-11T08:09:03.418572+0000","last_became_active":"2024-09-11T08:07:32.046891+0000","last_became_peered":"2024-09-11T08:07:32.046891+0000","last_unstale":"2024-09-11T08:09:03.418572+0000","last_undegraded":"2024-09-11T08:09:03.418572+0000","last_fullsized":"2024-09-11T08:09:03.418572+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-09-11T08:09:03.418457+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-09-11T08:09:03.418457+0000","last_clean_scrub_stamp":"2024-09-11T08:09:03.418457+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:10.688358+0000","scrub_duration":0.003076058,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.295114+0000","last_change":"2024-09-11T08:07:34.058334+0000","last_active":"2024-09-11T08:08:56.295114+0000","last_peered":"2024-09-11T08:08:56.295114+0000","last_clean":"2024-09-11T08:08:56.295114+0000","last_became_active":"2024-09-11T08:07:34.057774+0000","last_became_peered":"2024-09-11T08:07:34.057774+0000","last_unstale":"2024-09-11T08:08:56.295114+0000","last_undegraded":"2024-09-11T08:08:56.295114+0000","last_fullsized":"2024-09-11T08:08:56.295114+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.174229+0000","last_change":"2024-09-11T08:08:38.326345+0000","last_active":"2024-09-11T08:08:56.174229+0000","last_peered":"2024-09-11T08:08:56.174229+0000","last_clean":"2024-09-11T08:08:56.174229+0000","last_became_active":"2024-09-11T08:07:36.633244+0000","last_became_peered":"2024-09-11T08:07:36.633244+0000","last_unstale":"2024-09-11T08:08:56.174229+0000","last_undegraded":"2024-09-11T08:08:56.174229+0000","last_fullsized":"2024-09-11T08:08:56.174229+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:38.326265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:08:38.326265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:06.322672+0000","scrub_duration":0.001417361,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"84'13","reported_seq":63,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:57.835740+0000","last_change":"2024-09-11T08:08:57.835686+0000","last_active":"2024-09-11T08:08:57.835740+0000","last_peered":"2024-09-11T08:08:57.835740+0000","last_clean":"2024-09-11T08:08:57.835740+0000","last_became_active":"2024-09-11T08:07:32.048350+0000","last_became_peered":"2024-09-11T08:07:32.048350+0000","last_unstale":"2024-09-11T08:08:57.835740+0000","last_undegraded":"2024-09-11T08:08:57.835740+0000","last_fullsized":"2024-09-11T08:08:57.835740+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'13","last_scrub_stamp":"2024-09-11T08:08:57.835634+0000","last_deep_scrub":"84'13","last_deep_scrub_stamp":"2024-09-11T08:08:57.835634+0000","last_clean_scrub_stamp":"2024-09-11T08:08:57.835634+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:18.729258+0000","scrub_duration":0.0033293229999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.174130+0000","last_change":"2024-09-11T08:08:30.471690+0000","last_active":"2024-09-11T08:08:56.174130+0000","last_peered":"2024-09-11T08:08:56.174130+0000","last_clean":"2024-09-11T08:08:56.174130+0000","last_became_active":"2024-09-11T08:07:30.044055+0000","last_became_peered":"2024-09-11T08:07:30.044055+0000","last_unstale":"2024-09-11T08:08:56.174130+0000","last_undegraded":"2024-09-11T08:08:56.174130+0000","last_fullsized":"2024-09-11T08:08:56.174130+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:30.471620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:30.471620+0000","last_clean_scrub_stamp":"2024-09-11T08:08:30.471620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:37.891558+0000","scrub_duration":0.0016049980000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:04.370859+0000","last_change":"2024-09-11T08:09:04.370800+0000","last_active":"2024-09-11T08:09:04.370859+0000","last_peered":"2024-09-11T08:09:04.370859+0000","last_clean":"2024-09-11T08:09:04.370859+0000","last_became_active":"2024-09-11T08:07:34.057127+0000","last_became_peered":"2024-09-11T08:07:34.057127+0000","last_unstale":"2024-09-11T08:09:04.370859+0000","last_undegraded":"2024-09-11T08:09:04.370859+0000","last_fullsized":"2024-09-11T08:09:04.370859+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:04.370734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:04.370734+0000","last_clean_scrub_stamp":"2024-09-11T08:09:04.370734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:09.164201+0000","scrub_duration":0.001224483,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":38,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:01.995150+0000","last_change":"2024-09-11T08:09:01.995115+0000","last_active":"2024-09-11T08:09:01.995150+0000","last_peered":"2024-09-11T08:09:01.995150+0000","last_clean":"2024-09-11T08:09:01.995150+0000","last_became_active":"2024-09-11T08:07:36.071513+0000","last_became_peered":"2024-09-11T08:07:36.071513+0000","last_unstale":"2024-09-11T08:09:01.995150+0000","last_undegraded":"2024-09-11T08:09:01.995150+0000","last_fullsized":"2024-09-11T08:09:01.995150+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:01.995061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:09:01.995061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:18.017918+0000","scrub_duration":0.00115722,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":58,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.295415+0000","last_change":"2024-09-11T08:08:49.951403+0000","last_active":"2024-09-11T08:08:56.295415+0000","last_peered":"2024-09-11T08:08:56.295415+0000","last_clean":"2024-09-11T08:08:56.295415+0000","last_became_active":"2024-09-11T08:07:32.041761+0000","last_became_peered":"2024-09-11T08:07:32.041761+0000","last_unstale":"2024-09-11T08:08:56.295415+0000","last_undegraded":"2024-09-11T08:08:56.295415+0000","last_fullsized":"2024-09-11T08:08:56.295415+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:08:49.951345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:49.951345+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:51.233235+0000","scrub_duration":0.002079914,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.402831+0000","last_change":"2024-09-11T08:08:48.935452+0000","last_active":"2024-09-11T08:08:56.402831+0000","last_peered":"2024-09-11T08:08:56.402831+0000","last_clean":"2024-09-11T08:08:56.402831+0000","last_became_active":"2024-09-11T08:07:30.045380+0000","last_became_peered":"2024-09-11T08:07:30.045380+0000","last_unstale":"2024-09-11T08:08:56.402831+0000","last_undegraded":"2024-09-11T08:08:56.402831+0000","last_fullsized":"2024-09-11T08:08:56.402831+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:48.935387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:48.935387+0000","last_clean_scrub_stamp":"2024-09-11T08:08:48.935387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:05.810526+0000","scrub_duration":0.001422134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"66'8","reported_seq":68,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:03.560375+0000","last_change":"2024-09-11T08:08:52.374593+0000","last_active":"2024-09-11T08:09:03.560375+0000","last_peered":"2024-09-11T08:09:03.560375+0000","last_clean":"2024-09-11T08:09:03.560375+0000","last_became_active":"2024-09-11T08:07:34.057937+0000","last_became_peered":"2024-09-11T08:07:34.057937+0000","last_unstale":"2024-09-11T08:09:03.560375+0000","last_undegraded":"2024-09-11T08:09:03.560375+0000","last_fullsized":"2024-09-11T08:09:03.560375+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:08:52.374539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:52.374539+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:12.088877+0000","scrub_duration":0.00146886,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":29,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.359685+0000","last_change":"2024-09-11T08:07:36.068798+0000","last_active":"2024-09-11T08:08:55.359685+0000","last_peered":"2024-09-11T08:08:55.359685+0000","last_clean":"2024-09-11T08:08:55.359685+0000","last_became_active":"2024-09-11T08:07:36.068493+0000","last_became_peered":"2024-09-11T08:07:36.068493+0000","last_unstale":"2024-09-11T08:08:55.359685+0000","last_undegraded":"2024-09-11T08:08:55.359685+0000","last_fullsized":"2024-09-11T08:08:55.359685+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":60,"reported_epoch":85,"state":"peering","last_fresh":"2024-09-11T08:09:06.873141+0000","last_change":"2024-09-11T08:09:06.873379+0000","last_active":"2024-09-11T08:09:06.873107+0000","last_peered":"2024-09-11T08:08:55.360586+0000","last_clean":"2024-09-11T08:08:55.360586+0000","last_became_active":"2024-09-11T08:07:32.047347+0000","last_became_peered":"2024-09-11T08:07:32.047347+0000","last_unstale":"2024-09-11T08:09:06.873141+0000","last_undegraded":"2024-09-11T08:09:06.873141+0000","last_fullsized":"2024-09-11T08:09:06.873141+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-11T08:08:36.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:36.400586+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:50.012415+0000","scrub_duration":0.002193113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:03.210255+0000","last_change":"2024-09-11T08:09:03.210218+0000","last_active":"2024-09-11T08:09:03.210255+0000","last_peered":"2024-09-11T08:09:03.210255+0000","last_clean":"2024-09-11T08:09:03.210255+0000","last_became_active":"2024-09-11T08:07:30.039673+0000","last_became_peered":"2024-09-11T08:07:30.039673+0000","last_unstale":"2024-09-11T08:09:03.210255+0000","last_undegraded":"2024-09-11T08:09:03.210255+0000","last_fullsized":"2024-09-11T08:09:03.210255+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:03.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:09:03.210166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:05.614628+0000","scrub_duration":0.0011643720000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":67,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:03.560191+0000","last_change":"2024-09-11T08:08:47.966763+0000","last_active":"2024-09-11T08:09:03.560191+0000","last_peered":"2024-09-11T08:09:03.560191+0000","last_clean":"2024-09-11T08:09:03.560191+0000","last_became_active":"2024-09-11T08:07:34.060736+0000","last_became_peered":"2024-09-11T08:07:34.060736+0000","last_unstale":"2024-09-11T08:09:03.560191+0000","last_undegraded":"2024-09-11T08:09:03.560191+0000","last_fullsized":"2024-09-11T08:09:03.560191+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:08:47.966690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:47.966690+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:12.893275+0000","scrub_duration":0.0019777940000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":29,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.402525+0000","last_change":"2024-09-11T08:07:36.072648+0000","last_active":"2024-09-11T08:08:56.402525+0000","last_peered":"2024-09-11T08:08:56.402525+0000","last_clean":"2024-09-11T08:08:56.402525+0000","last_became_active":"2024-09-11T08:07:36.072359+0000","last_became_peered":"2024-09-11T08:07:36.072359+0000","last_unstale":"2024-09-11T08:08:56.402525+0000","last_undegraded":"2024-09-11T08:08:56.402525+0000","last_fullsized":"2024-09-11T08:08:56.402525+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":62,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:03.516765+0000","last_change":"2024-09-11T08:09:03.516720+0000","last_active":"2024-09-11T08:09:03.516765+0000","last_peered":"2024-09-11T08:09:03.516765+0000","last_clean":"2024-09-11T08:09:03.516765+0000","last_became_active":"2024-09-11T08:07:32.047981+0000","last_became_peered":"2024-09-11T08:07:32.047981+0000","last_unstale":"2024-09-11T08:09:03.516765+0000","last_undegraded":"2024-09-11T08:09:03.516765+0000","last_fullsized":"2024-09-11T08:09:03.516765+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-11T08:09:03.516664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:09:03.516664+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:23.127237+0000","scrub_duration":0.001710917,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.359557+0000","last_change":"2024-09-11T08:08:48.330351+0000","last_active":"2024-09-11T08:08:55.359557+0000","last_peered":"2024-09-11T08:08:55.359557+0000","last_clean":"2024-09-11T08:08:55.359557+0000","last_became_active":"2024-09-11T08:07:30.043691+0000","last_became_peered":"2024-09-11T08:07:30.043691+0000","last_unstale":"2024-09-11T08:08:55.359557+0000","last_undegraded":"2024-09-11T08:08:55.359557+0000","last_fullsized":"2024-09-11T08:08:55.359557+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:48.330285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:48.330285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:05.763303+0000","scrub_duration":0.001379929,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":31,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.011091+0000","last_change":"2024-09-11T08:07:34.053427+0000","last_active":"2024-09-11T08:08:56.011091+0000","last_peered":"2024-09-11T08:08:56.011091+0000","last_clean":"2024-09-11T08:08:56.011091+0000","last_became_active":"2024-09-11T08:07:34.053137+0000","last_became_peered":"2024-09-11T08:07:34.053137+0000","last_unstale":"2024-09-11T08:08:56.011091+0000","last_undegraded":"2024-09-11T08:08:56.011091+0000","last_fullsized":"2024-09-11T08:08:56.011091+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":29,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.360100+0000","last_change":"2024-09-11T08:07:36.076837+0000","last_active":"2024-09-11T08:08:55.360100+0000","last_peered":"2024-09-11T08:08:55.360100+0000","last_clean":"2024-09-11T08:08:55.360100+0000","last_became_active":"2024-09-11T08:07:36.076132+0000","last_became_peered":"2024-09-11T08:07:36.076132+0000","last_unstale":"2024-09-11T08:08:55.360100+0000","last_undegraded":"2024-09-11T08:08:55.360100+0000","last_fullsized":"2024-09-11T08:08:55.360100+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":59,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.011296+0000","last_change":"2024-09-11T08:08:55.332126+0000","last_active":"2024-09-11T08:08:56.011296+0000","last_peered":"2024-09-11T08:08:56.011296+0000","last_clean":"2024-09-11T08:08:56.011296+0000","last_became_active":"2024-09-11T08:07:32.049368+0000","last_became_peered":"2024-09-11T08:07:32.049368+0000","last_unstale":"2024-09-11T08:08:56.011296+0000","last_undegraded":"2024-09-11T08:08:56.011296+0000","last_fullsized":"2024-09-11T08:08:56.011296+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:08:55.332065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:55.332065+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:15.584089+0000","scrub_duration":0.001632218,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.174392+0000","last_change":"2024-09-11T08:08:45.388090+0000","last_active":"2024-09-11T08:08:56.174392+0000","last_peered":"2024-09-11T08:08:56.174392+0000","last_clean":"2024-09-11T08:08:56.174392+0000","last_became_active":"2024-09-11T08:07:30.037692+0000","last_became_peered":"2024-09-11T08:07:30.037692+0000","last_unstale":"2024-09-11T08:08:56.174392+0000","last_undegraded":"2024-09-11T08:08:56.174392+0000","last_fullsized":"2024-09-11T08:08:56.174392+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:45.388024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:45.388024+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:59.280827+0000","scrub_duration":0.0014875719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":31,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.359847+0000","last_change":"2024-09-11T08:07:34.057358+0000","last_active":"2024-09-11T08:08:55.359847+0000","last_peered":"2024-09-11T08:08:55.359847+0000","last_clean":"2024-09-11T08:08:55.359847+0000","last_became_active":"2024-09-11T08:07:34.057064+0000","last_became_peered":"2024-09-11T08:07:34.057064+0000","last_unstale":"2024-09-11T08:08:55.359847+0000","last_undegraded":"2024-09-11T08:08:55.359847+0000","last_fullsized":"2024-09-11T08:08:55.359847+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":29,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.402664+0000","last_change":"2024-09-11T08:07:36.077697+0000","last_active":"2024-09-11T08:08:56.402664+0000","last_peered":"2024-09-11T08:08:56.402664+0000","last_clean":"2024-09-11T08:08:56.402664+0000","last_became_active":"2024-09-11T08:07:36.077406+0000","last_became_peered":"2024-09-11T08:07:36.077406+0000","last_unstale":"2024-09-11T08:08:56.402664+0000","last_undegraded":"2024-09-11T08:08:56.402664+0000","last_fullsized":"2024-09-11T08:08:56.402664+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":52,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.360345+0000","last_change":"2024-09-11T08:07:32.043568+0000","last_active":"2024-09-11T08:08:55.360345+0000","last_peered":"2024-09-11T08:08:55.360345+0000","last_clean":"2024-09-11T08:08:55.360345+0000","last_became_active":"2024-09-11T08:07:32.043276+0000","last_became_peered":"2024-09-11T08:07:32.043276+0000","last_unstale":"2024-09-11T08:08:55.360345+0000","last_undegraded":"2024-09-11T08:08:55.360345+0000","last_fullsized":"2024-09-11T08:08:55.360345+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.295188+0000","last_change":"2024-09-11T08:08:55.853168+0000","last_active":"2024-09-11T08:08:56.295188+0000","last_peered":"2024-09-11T08:08:56.295188+0000","last_clean":"2024-09-11T08:08:56.295188+0000","last_became_active":"2024-09-11T08:07:30.049152+0000","last_became_peered":"2024-09-11T08:07:30.049152+0000","last_unstale":"2024-09-11T08:08:56.295188+0000","last_undegraded":"2024-09-11T08:08:56.295188+0000","last_fullsized":"2024-09-11T08:08:56.295188+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:55.853076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:55.853076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:03.761618+0000","scrub_duration":0.0018073309999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":31,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.173826+0000","last_change":"2024-09-11T08:07:34.057322+0000","last_active":"2024-09-11T08:08:56.173826+0000","last_peered":"2024-09-11T08:08:56.173826+0000","last_clean":"2024-09-11T08:08:56.173826+0000","last_became_active":"2024-09-11T08:07:34.056935+0000","last_became_peered":"2024-09-11T08:07:34.056935+0000","last_unstale":"2024-09-11T08:08:56.173826+0000","last_undegraded":"2024-09-11T08:08:56.173826+0000","last_fullsized":"2024-09-11T08:08:56.173826+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":29,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.359925+0000","last_change":"2024-09-11T08:07:36.633645+0000","last_active":"2024-09-11T08:08:55.359925+0000","last_peered":"2024-09-11T08:08:55.359925+0000","last_clean":"2024-09-11T08:08:55.359925+0000","last_became_active":"2024-09-11T08:07:36.633368+0000","last_became_peered":"2024-09-11T08:07:36.633368+0000","last_unstale":"2024-09-11T08:08:55.359925+0000","last_undegraded":"2024-09-11T08:08:55.359925+0000","last_fullsized":"2024-09-11T08:08:55.359925+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.359749+0000","last_change":"2024-09-11T08:07:32.046050+0000","last_active":"2024-09-11T08:08:55.359749+0000","last_peered":"2024-09-11T08:08:55.359749+0000","last_clean":"2024-09-11T08:08:55.359749+0000","last_became_active":"2024-09-11T08:07:32.045728+0000","last_became_peered":"2024-09-11T08:07:32.045728+0000","last_unstale":"2024-09-11T08:08:55.359749+0000","last_undegraded":"2024-09-11T08:08:55.359749+0000","last_fullsized":"2024-09-11T08:08:55.359749+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.403337+0000","last_change":"2024-09-11T08:08:42.056647+0000","last_active":"2024-09-11T08:08:56.403337+0000","last_peered":"2024-09-11T08:08:56.403337+0000","last_clean":"2024-09-11T08:08:56.403337+0000","last_became_active":"2024-09-11T08:07:30.049409+0000","last_became_peered":"2024-09-11T08:07:30.049409+0000","last_unstale":"2024-09-11T08:08:56.403337+0000","last_undegraded":"2024-09-11T08:08:56.403337+0000","last_fullsized":"2024-09-11T08:08:56.403337+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:42.056591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:42.056591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:56.872271+0000","scrub_duration":0.003068809,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.402392+0000","last_change":"2024-09-11T08:08:34.506991+0000","last_active":"2024-09-11T08:08:56.402392+0000","last_peered":"2024-09-11T08:08:56.402392+0000","last_clean":"2024-09-11T08:08:56.402392+0000","last_became_active":"2024-09-11T08:07:34.055082+0000","last_became_peered":"2024-09-11T08:07:34.055082+0000","last_unstale":"2024-09-11T08:08:56.402392+0000","last_undegraded":"2024-09-11T08:08:56.402392+0000","last_fullsized":"2024-09-11T08:08:56.402392+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:34.506927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:34.506927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:46.584592+0000","scrub_duration":0.0015499070000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":29,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.173566+0000","last_change":"2024-09-11T08:07:36.072035+0000","last_active":"2024-09-11T08:08:56.173566+0000","last_peered":"2024-09-11T08:08:56.173566+0000","last_clean":"2024-09-11T08:08:56.173566+0000","last_became_active":"2024-09-11T08:07:36.071669+0000","last_became_peered":"2024-09-11T08:07:36.071669+0000","last_unstale":"2024-09-11T08:08:56.173566+0000","last_undegraded":"2024-09-11T08:08:56.173566+0000","last_fullsized":"2024-09-11T08:08:56.173566+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":29,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.011339+0000","last_change":"2024-09-11T08:07:36.071729+0000","last_active":"2024-09-11T08:08:56.011339+0000","last_peered":"2024-09-11T08:08:56.011339+0000","last_clean":"2024-09-11T08:08:56.011339+0000","last_became_active":"2024-09-11T08:07:36.071468+0000","last_became_peered":"2024-09-11T08:07:36.071468+0000","last_unstale":"2024-09-11T08:08:56.011339+0000","last_undegraded":"2024-09-11T08:08:56.011339+0000","last_fullsized":"2024-09-11T08:08:56.011339+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:55.360095+0000","last_change":"2024-09-11T08:08:49.428032+0000","last_active":"2024-09-11T08:08:55.360095+0000","last_peered":"2024-09-11T08:08:55.360095+0000","last_clean":"2024-09-11T08:08:55.360095+0000","last_became_active":"2024-09-11T08:07:30.046702+0000","last_became_peered":"2024-09-11T08:07:30.046702+0000","last_unstale":"2024-09-11T08:08:55.360095+0000","last_undegraded":"2024-09-11T08:08:55.360095+0000","last_fullsized":"2024-09-11T08:08:55.360095+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:49.427944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:49.427944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:11.537893+0000","scrub_duration":0.001383034,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":61,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:09:03.841823+0000","last_change":"2024-09-11T08:09:03.841778+0000","last_active":"2024-09-11T08:09:03.841823+0000","last_peered":"2024-09-11T08:09:03.841823+0000","last_clean":"2024-09-11T08:09:03.841823+0000","last_became_active":"2024-09-11T08:07:32.054656+0000","last_became_peered":"2024-09-11T08:07:32.054656+0000","last_unstale":"2024-09-11T08:09:03.841823+0000","last_undegraded":"2024-09-11T08:09:03.841823+0000","last_fullsized":"2024-09-11T08:09:03.841823+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-11T08:09:03.841721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:09:03.841721+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:20.583780+0000","scrub_duration":0.00194933,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T08:08:56.403143+0000","last_change":"2024-09-11T08:08:53.869870+0000","last_active":"2024-09-11T08:08:56.403143+0000","last_peered":"2024-09-11T08:08:56.403143+0000","last_clean":"2024-09-11T08:08:56.403143+0000","last_became_active":"2024-09-11T08:07:34.061453+0000","last_became_peered":"2024-09-11T08:07:34.061453+0000","last_unstale":"2024-09-11T08:08:56.403143+0000","last_undegraded":"2024-09-11T08:08:56.403143+0000","last_fullsized":"2024-09-11T08:08:56.403143+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:53.869813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:53.869813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:59.468130+0000","scrub_duration":0.0012524350000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":413,"ondisk_log_size":413,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":29,"num_read_kb":29,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":91,"num_read_kb":86,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266712,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298796,"kb_used_data":1200,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446356,"statfs":{"total":95995035648,"available":95689068544,"internally_reserved":0,"allocated":1228800,"data_stored":881761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.47199999999999998}]},{"osd":1,"last update":"Wed Sep 11 08:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.437},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.42799999999999999}]},{"osd":2,"last update":"Wed Sep 11 08:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.41699999999999998}]},{"osd":3,"last update":"Wed Sep 11 08:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":7.9210000000000003,"5min":7.9210000000000003,"15min":7.9210000000000003},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.374}]},{"osd":4,"last update":"Wed Sep 11 08:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.314}]},{"osd":5,"last update":"Wed Sep 11 08:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.39000000000000001}]},{"osd":6,"last update":"Wed Sep 11 08:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.34399999999999997}]}]},{"osd":6,"up_from":45,"seq":193273528349,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298764,"kb_used_data":1412,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446388,"statfs":{"total":95995035648,"available":95689101312,"internally_reserved":0,"allocated":1445888,"data_stored":951248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.49299999999999999,"15min":0.496},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.59099999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.50700000000000001,"15min":0.50600000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.72099999999999997,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.54000000000000004}]},{"osd":1,"last update":"Wed Sep 11 08:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.47999999999999998,"15min":0.47799999999999998},"min":{"1min":0.32700000000000001,"5min":0.318,"15min":0.318},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.505,"15min":0.50600000000000001},"min":{"1min":0.33500000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.65900000000000003,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.443}]},{"osd":2,"last update":"Wed Sep 11 08:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.5,"15min":0.502},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.67200000000000004,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.499,"5min":0.48499999999999999,"15min":0.48299999999999998},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.72399999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.51000000000000001}]},{"osd":3,"last update":"Wed Sep 11 08:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.52400000000000002,"15min":0.52600000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.5},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.53900000000000003,"15min":0.53800000000000003},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.73499999999999999,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.52500000000000002}]},{"osd":4,"last update":"Wed Sep 11 08:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.39000000000000001,"15min":0.38700000000000001},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.59399999999999997,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.41999999999999998,"15min":0.41199999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.38}]},{"osd":5,"last update":"Wed Sep 11 08:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.48899999999999999,"15min":0.47999999999999998},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.434,"5min":0.41399999999999998,"15min":0.41099999999999998},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.62,"5min":0.629,"15min":0.629},"last":0.41199999999999998}]},{"osd":7,"last update":"Wed Sep 11 08:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.39900000000000002}]}]},{"osd":5,"up_from":38,"seq":163208757282,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298180,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446972,"statfs":{"total":95995035648,"available":95689699328,"internally_reserved":0,"allocated":847872,"data_stored":360087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.51400000000000001,"15min":0.52200000000000002},"min":{"1min":0.318,"5min":0.31,"15min":0.31},"max":{"1min":0.63500000000000001,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.503,"15min":0.501},"min":{"1min":0.41399999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.54000000000000004}]},{"osd":1,"last update":"Wed Sep 11 08:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":1.1910000000000001,"15min":1.3029999999999999},"min":{"1min":0.33000000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.82499999999999996,"5min":16.004999999999999,"15min":16.004999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.36199999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.57999999999999996}]},{"osd":2,"last update":"Wed Sep 11 08:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":1.206,"15min":1.323},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.754,"5min":16.018000000000001,"15min":16.018000000000001},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.55900000000000005}]},{"osd":3,"last update":"Wed Sep 11 08:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.53100000000000003,"15min":0.52900000000000003},"min":{"1min":0.44900000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.72299999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.505,"5min":0.51600000000000001,"15min":0.51800000000000002},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.45400000000000001}]},{"osd":4,"last update":"Wed Sep 11 08:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.39700000000000002,"15min":0.39300000000000002},"min":{"1min":0.27500000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.434},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.40000000000000002,"15min":0.38800000000000001},"min":{"1min":0.36499999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.46899999999999997}]},{"osd":6,"last update":"Wed Sep 11 08:08:56 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.57699999999999996,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.372},{"interface":"front","average":{"1min":0.495,"5min":0.45200000000000001,"15min":0.44500000000000001},"min":{"1min":0.316,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.67100000000000004,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.496}]},{"osd":7,"last update":"Wed Sep 11 08:08:22 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.39800000000000002}]}]},{"osd":4,"up_from":31,"seq":133143986215,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298220,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446932,"statfs":{"total":95995035648,"available":95689658368,"internally_reserved":0,"allocated":638976,"data_stored":291776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.499,"15min":0.50900000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.59699999999999998,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.52400000000000002,"15min":0.53500000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.58899999999999997,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.438}]},{"osd":1,"last update":"Wed Sep 11 08:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.505,"15min":0.51300000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.66800000000000004,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.52300000000000002,"15min":0.52900000000000003},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.68899999999999995,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.498}]},{"osd":2,"last update":"Wed Sep 11 08:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.52000000000000002,"15min":0.52800000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.58799999999999997,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.51600000000000001,"15min":0.52500000000000002},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.623,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.39900000000000002}]},{"osd":3,"last update":"Wed Sep 11 08:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.50900000000000001,"15min":0.51500000000000001},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.64300000000000002,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.51800000000000002,"15min":0.52600000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.61499999999999999,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.40799999999999997}]},{"osd":5,"last update":"Wed Sep 11 08:08:31 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.36899999999999999,"15min":0.36699999999999999},"min":{"1min":0.27800000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.36899999999999999,"15min":0.36699999999999999},"min":{"1min":0.27900000000000003,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.33500000000000002}]},{"osd":6,"last update":"Wed Sep 11 08:08:56 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.39100000000000001,"15min":0.38300000000000001},"min":{"1min":0.30399999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.314},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.44,"15min":0.432},"min":{"1min":0.32200000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.38500000000000001}]},{"osd":7,"last update":"Wed Sep 11 08:08:20 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.377}]}]},{"osd":3,"up_from":25,"seq":107374182444,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298236,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446916,"statfs":{"total":95995035648,"available":95689641984,"internally_reserved":0,"allocated":655360,"data_stored":291562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:08:53 2024","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.36199999999999999,"15min":0.374},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":0.55100000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.315},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.40300000000000002,"15min":0.41799999999999998},"min":{"1min":0.26900000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.55100000000000005,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.26800000000000002}]},{"osd":1,"last update":"Wed Sep 11 08:08:53 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.41599999999999998,"15min":0.437},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.497,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.371},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.41399999999999998,"15min":0.42899999999999999},"min":{"1min":0.24399999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.53800000000000003,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.29199999999999998}]},{"osd":2,"last update":"Wed Sep 11 08:08:53 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.434,"15min":0.44900000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.47999999999999998,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.371,"5min":0.40600000000000003,"15min":0.41899999999999998},"min":{"1min":0.26700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.53100000000000003,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.47199999999999998}]},{"osd":4,"last update":"Wed Sep 11 08:08:12 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54800000000000004,"15min":0.54900000000000004},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.5},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.53500000000000003,"15min":0.53700000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.51900000000000002}]},{"osd":5,"last update":"Wed Sep 11 08:08:33 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.54200000000000004,"15min":0.53800000000000003},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.40699999999999997,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.40500000000000003}]},{"osd":6,"last update":"Wed Sep 11 08:08:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.53400000000000003,"15min":0.53000000000000003},"min":{"1min":0.35899999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.55400000000000005,"15min":0.55800000000000005},"min":{"1min":0.39500000000000002,"5min":0.373,"15min":0.373},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.53300000000000003}]},{"osd":7,"last update":"Wed Sep 11 08:08:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.434}]}]},{"osd":2,"up_from":18,"seq":77309411375,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298192,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446960,"statfs":{"total":95995035648,"available":95689687040,"internally_reserved":0,"allocated":610304,"data_stored":290154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:08:24 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.38400000000000001,"15min":0.38100000000000001},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.58099999999999996,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.37,"15min":0.35999999999999999},"min":{"1min":0.25,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.55600000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.52100000000000002}]},{"osd":1,"last update":"Wed Sep 11 08:08:24 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.39500000000000002,"15min":0.38},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.60899999999999999,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.504},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.39800000000000002,"15min":0.38600000000000001},"min":{"1min":0.28699999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.45300000000000001}]},{"osd":3,"last update":"Wed Sep 11 08:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.45100000000000001,"15min":0.46100000000000002},"min":{"1min":0.27300000000000002,"5min":0.218,"15min":0.218},"max":{"1min":0.629,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.45100000000000001,"15min":0.46500000000000002},"min":{"1min":0.318,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.48899999999999999}]},{"osd":4,"last update":"Wed Sep 11 08:08:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.58299999999999996,"15min":0.58699999999999997},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.71199999999999997,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.71599999999999997}]},{"osd":5,"last update":"Wed Sep 11 08:08:35 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.57699999999999996,"15min":0.58299999999999996},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.751,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.59199999999999997,"15min":0.59999999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.65300000000000002,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.73199999999999998}]},{"osd":6,"last update":"Wed Sep 11 08:08:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.58499999999999996,"15min":0.59399999999999997},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.59099999999999997,"15min":0.59699999999999998},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.70299999999999996,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.74299999999999999}]},{"osd":7,"last update":"Wed Sep 11 08:08:22 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.70299999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574901,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298248,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446904,"statfs":{"total":95995035648,"available":95689629696,"internally_reserved":0,"allocated":667648,"data_stored":291720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:09:04 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.33700000000000002,"15min":0.30399999999999999},"min":{"1min":0.23599999999999999,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.33900000000000002,"15min":0.315},"min":{"1min":0.224,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.22900000000000001}]},{"osd":2,"last update":"Wed Sep 11 08:08:28 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.36199999999999999,"15min":0.34799999999999998},"min":{"1min":0.27700000000000002,"5min":0.224,"15min":0.224},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.37,"5min":0.35399999999999998,"15min":0.34200000000000003},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.247}]},{"osd":3,"last update":"Wed Sep 11 08:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.41399999999999998,"15min":0.42199999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.439,"15min":0.437},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.198,"5min":1.198,"15min":1.198},"last":0.35999999999999999}]},{"osd":4,"last update":"Wed Sep 11 08:08:08 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54300000000000004,"15min":0.54200000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.55800000000000005,"15min":0.55600000000000005},"min":{"1min":0.41899999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.73499999999999999,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.33100000000000002}]},{"osd":5,"last update":"Wed Sep 11 08:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.53700000000000003,"15min":0.53000000000000003},"min":{"1min":0.40699999999999997,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.77600000000000002,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.312},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.53100000000000003,"15min":0.52700000000000002},"min":{"1min":0.38400000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.40600000000000003}]},{"osd":6,"last update":"Wed Sep 11 08:08:56 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.55500000000000005,"15min":0.55900000000000005},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.59899999999999998,"15min":0.60099999999999998},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.433}]},{"osd":7,"last update":"Wed Sep 11 08:08:17 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.38500000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738425,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298808,"kb_used_data":1212,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446344,"statfs":{"total":95995035648,"available":95689056256,"internally_reserved":0,"allocated":1241088,"data_stored":882707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 11 08:09:04 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.34799999999999998,"15min":0.317},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":0.53400000000000003,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.35899999999999999,"15min":0.33800000000000002},"min":{"1min":0.219,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.47999999999999998,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.254}]},{"osd":2,"last update":"Wed Sep 11 08:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.34699999999999998},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.48799999999999999,"5min":0.53000000000000003,"15min":0.53000000000000003},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.378,"5min":0.378,"15min":0.36899999999999999},"min":{"1min":0.26600000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.56299999999999994,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.32200000000000001}]},{"osd":3,"last update":"Wed Sep 11 08:08:51 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.44,"15min":0.44},"min":{"1min":0.27600000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.52300000000000002,"5min":0.62,"15min":0.62},"last":0.379},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.42499999999999999,"15min":0.42599999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.52800000000000002,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.30399999999999999}]},{"osd":4,"last update":"Wed Sep 11 08:08:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.55000000000000004,"15min":0.54600000000000004},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.47699999999999998}]},{"osd":5,"last update":"Wed Sep 11 08:08:36 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.58499999999999996,"15min":0.58299999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.748,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.495},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.55800000000000005,"15min":0.55300000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.37}]},{"osd":6,"last update":"Wed Sep 11 08:08:59 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.60899999999999999,"15min":0.61699999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.75600000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.60299999999999998,"15min":0.57099999999999995},"min":{"1min":0.433,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":4.7069999999999999,"5min":4.7069999999999999,"15min":4.7069999999999999},"last":0.39200000000000002}]},{"osd":7,"last update":"Wed Sep 11 08:08:19 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.435},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.33900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T08:09:12.830 INFO:tasks.cephadm.ceph_manager.ceph:PG 4.1d is not active+clean 2024-09-11T08:09:12.830 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '4.1d', 'version': "66'12", 'reported_seq': 58, 'reported_epoch': 85, 'state': 'peering', 'last_fresh': '2024-09-11T08:09:06.874089+0000', 'last_change': '2024-09-11T08:09:06.874325+0000', 'last_active': '2024-09-11T08:09:06.874061+0000', 'last_peered': '2024-09-11T08:08:55.359627+0000', 'last_clean': '2024-09-11T08:08:55.359627+0000', 'last_became_active': '2024-09-11T08:07:32.046737+0000', 'last_became_peered': '2024-09-11T08:07:32.046737+0000', 'last_unstale': '2024-09-11T08:09:06.874089+0000', 'last_undegraded': '2024-09-11T08:09:06.874089+0000', 'last_fullsized': '2024-09-11T08:09:06.874089+0000', 'mapping_epoch': 85, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 60, 'last_epoch_clean': 61, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T08:07:31.028627+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T08:07:31.028627+0000', 'last_clean_scrub_stamp': '2024-09-11T08:07:31.028627+0000', 'objects_scrubbed': 0, 'log_size': 12, 'log_dups_size': 0, 'ondisk_log_size': 12, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 220, 'num_objects': 5, 'num_object_clones': 0, 'num_object_copies': 15, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5, 'num_whiteouts': 0, 'num_read': 25, 'num_read_kb': 16, 'num_write': 14, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2, 4], 'acting': [3, 2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1, 2, 4], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T08:09:12.831 INFO:tasks.cephadm.ceph_manager.ceph:PG 4.d is not active+clean 2024-09-11T08:09:12.831 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '4.d', 'version': "77'25", 'reported_seq': 81, 'reported_epoch': 85, 'state': 'peering', 'last_fresh': '2024-09-11T08:09:06.872554+0000', 'last_change': '2024-09-11T08:09:06.872760+0000', 'last_active': '2024-09-11T08:09:06.872524+0000', 'last_peered': '2024-09-11T08:08:55.359257+0000', 'last_clean': '2024-09-11T08:08:55.359257+0000', 'last_became_active': '2024-09-11T08:07:32.050815+0000', 'last_became_peered': '2024-09-11T08:07:32.050815+0000', 'last_unstale': '2024-09-11T08:09:06.872554+0000', 'last_undegraded': '2024-09-11T08:09:06.872554+0000', 'last_fullsized': '2024-09-11T08:09:06.872554+0000', 'mapping_epoch': 85, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 60, 'last_epoch_clean': 61, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "77'25", 'last_scrub_stamp': '2024-09-11T08:08:32.458210+0000', 'last_deep_scrub': "77'25", 'last_deep_scrub_stamp': '2024-09-11T08:08:32.458210+0000', 'last_clean_scrub_stamp': '2024-09-11T08:08:32.458210+0000', 'objects_scrubbed': 8, 'log_size': 25, 'log_dups_size': 0, 'ondisk_log_size': 25, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-11T08:09:53.685411+0000', 'scrub_duration': 0.006052332, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 110, 'num_objects': 8, 'num_object_clones': 0, 'num_object_copies': 24, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8, 'num_whiteouts': 0, 'num_read': 29, 'num_read_kb': 19, 'num_write': 18, 'num_write_kb': 1, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2, 5], 'acting': [4, 2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1, 2, 5], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-11T08:09:12.831 INFO:tasks.cephadm.ceph_manager.ceph:PG 4.14 is not active+clean 2024-09-11T08:09:12.831 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '4.14', 'version': "66'10", 'reported_seq': 60, 'reported_epoch': 85, 'state': 'peering', 'last_fresh': '2024-09-11T08:09:06.873141+0000', 'last_change': '2024-09-11T08:09:06.873379+0000', 'last_active': '2024-09-11T08:09:06.873107+0000', 'last_peered': '2024-09-11T08:08:55.360586+0000', 'last_clean': '2024-09-11T08:08:55.360586+0000', 'last_became_active': '2024-09-11T08:07:32.047347+0000', 'last_became_peered': '2024-09-11T08:07:32.047347+0000', 'last_unstale': '2024-09-11T08:09:06.873141+0000', 'last_undegraded': '2024-09-11T08:09:06.873141+0000', 'last_fullsized': '2024-09-11T08:09:06.873141+0000', 'mapping_epoch': 85, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 60, 'last_epoch_clean': 61, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "66'10", 'last_scrub_stamp': '2024-09-11T08:08:36.400586+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T08:07:31.028627+0000', 'last_clean_scrub_stamp': '2024-09-11T08:08:36.400586+0000', 'objects_scrubbed': 5, 'log_size': 10, 'log_dups_size': 0, 'ondisk_log_size': 10, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-11T08:09:50.012415+0000', 'scrub_duration': 0.002193113, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 5, 'num_object_clones': 0, 'num_object_copies': 15, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5, 'num_whiteouts': 0, 'num_read': 15, 'num_read_kb': 10, 'num_write': 10, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2, 7], 'acting': [3, 2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1, 2, 7], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T08:09:12.832 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph status --format=json 2024-09-11T08:09:13.079 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:09:13.200 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:12 smithi150 ceph-mon[29276]: 4.1 scrub starts 2024-09-11T08:09:13.201 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:12 smithi150 ceph-mon[29276]: 4.1 scrub ok 2024-09-11T08:09:13.201 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:12 smithi150 ceph-mon[29276]: 5.1b scrub starts 2024-09-11T08:09:13.201 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:12 smithi150 ceph-mon[29276]: 5.1b scrub ok 2024-09-11T08:09:13.201 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:12 smithi150 ceph-mon[29276]: 3.1b scrub starts 2024-09-11T08:09:13.201 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:12 smithi150 ceph-mon[29276]: 3.1b scrub ok 2024-09-11T08:09:13.201 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:12 smithi150 ceph-mon[29276]: from='client.14715 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:09:13.201 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:12 smithi150 ceph-mon[39612]: 4.1 scrub starts 2024-09-11T08:09:13.201 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:12 smithi150 ceph-mon[39612]: 4.1 scrub ok 2024-09-11T08:09:13.201 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:12 smithi150 ceph-mon[39612]: 5.1b scrub starts 2024-09-11T08:09:13.201 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:12 smithi150 ceph-mon[39612]: 5.1b scrub ok 2024-09-11T08:09:13.201 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:12 smithi150 ceph-mon[39612]: 3.1b scrub starts 2024-09-11T08:09:13.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:12 smithi150 ceph-mon[39612]: 3.1b scrub ok 2024-09-11T08:09:13.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:12 smithi150 ceph-mon[39612]: from='client.14715 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:09:13.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:12 smithi181 ceph-mon[32701]: 4.1 scrub starts 2024-09-11T08:09:13.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:12 smithi181 ceph-mon[32701]: 4.1 scrub ok 2024-09-11T08:09:13.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:12 smithi181 ceph-mon[32701]: 5.1b scrub starts 2024-09-11T08:09:13.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:12 smithi181 ceph-mon[32701]: 5.1b scrub ok 2024-09-11T08:09:13.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:12 smithi181 ceph-mon[32701]: 3.1b scrub starts 2024-09-11T08:09:13.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:12 smithi181 ceph-mon[32701]: 3.1b scrub ok 2024-09-11T08:09:13.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:12 smithi181 ceph-mon[32701]: from='client.14715 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:09:14.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:13 smithi150 ceph-mon[29276]: 6.1c scrub starts 2024-09-11T08:09:14.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:13 smithi150 ceph-mon[29276]: 6.1c scrub ok 2024-09-11T08:09:14.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:13 smithi150 ceph-mon[29276]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 41 B/s, 2 objects/s recovering 2024-09-11T08:09:14.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:13 smithi150 ceph-mon[29276]: 5.2 scrub starts 2024-09-11T08:09:14.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:13 smithi150 ceph-mon[29276]: 5.2 scrub ok 2024-09-11T08:09:14.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:13 smithi150 ceph-mon[29276]: 4.1e scrub starts 2024-09-11T08:09:14.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:13 smithi150 ceph-mon[29276]: 4.1e scrub ok 2024-09-11T08:09:14.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:13 smithi150 ceph-mon[39612]: 6.1c scrub starts 2024-09-11T08:09:14.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:13 smithi150 ceph-mon[39612]: 6.1c scrub ok 2024-09-11T08:09:14.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:13 smithi150 ceph-mon[39612]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 41 B/s, 2 objects/s recovering 2024-09-11T08:09:14.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:13 smithi150 ceph-mon[39612]: 5.2 scrub starts 2024-09-11T08:09:14.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:13 smithi150 ceph-mon[39612]: 5.2 scrub ok 2024-09-11T08:09:14.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:13 smithi150 ceph-mon[39612]: 4.1e scrub starts 2024-09-11T08:09:14.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:13 smithi150 ceph-mon[39612]: 4.1e scrub ok 2024-09-11T08:09:14.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:13 smithi181 ceph-mon[32701]: 6.1c scrub starts 2024-09-11T08:09:14.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:13 smithi181 ceph-mon[32701]: 6.1c scrub ok 2024-09-11T08:09:14.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:13 smithi181 ceph-mon[32701]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 41 B/s, 2 objects/s recovering 2024-09-11T08:09:14.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:13 smithi181 ceph-mon[32701]: 5.2 scrub starts 2024-09-11T08:09:14.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:13 smithi181 ceph-mon[32701]: 5.2 scrub ok 2024-09-11T08:09:14.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:13 smithi181 ceph-mon[32701]: 4.1e scrub starts 2024-09-11T08:09:14.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:13 smithi181 ceph-mon[32701]: 4.1e scrub ok 2024-09-11T08:09:15.063 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:09:15.064 INFO:teuthology.orchestra.run.smithi150.stdout:{"fsid":"c1bddff4-7013-11ef-bcea-c7b262605968","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":318,"monmap":{"epoch":3,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":86,"num_osds":8,"num_up_osds":8,"osd_up_since":1726042037,"num_in_osds":8,"osd_in_since":1726042020,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":132}],"num_pgs":132,"num_pools":6,"num_objects":196,"data_bytes":596188,"bytes_used":2442678272,"bytes_avail":765517606912,"bytes_total":767960285184,"recovering_objects_per_sec":2,"recovering_bytes_per_sec":41,"recovering_keys_per_sec":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"read_bytes_sec":1279,"read_op_per_sec":1,"write_op_per_sec":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.150:8443/","prometheus":"http://172.21.15.150:9283/"}},"servicemap":{"epoch":14,"modified":"2024-09-11T08:09:08.693126+0000","services":{"rgw":{"daemons":{"summary":"","14451":{"start_epoch":12,"start_stamp":"2024-09-11T08:07:37.191516+0000","gid":14451,"addr":"172.21.15.150:0/3411048405","metadata":{"arch":"x86_64","ceph_release":"quincy","ceph_version":"ceph version 17.2.7-1623-gd0e6828a (d0e6828a2016d48cf25ad84064e50742bb1c39b9) quincy (stable)","ceph_version_short":"17.2.7-1623-gd0e6828a","container_hostname":"smithi150","container_image":"quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40","cpu":"Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz","distro":"centos","distro_description":"CentOS Stream 9","distro_version":"9","frontend_config#0":"beast port=80","frontend_type#0":"beast","hostname":"smithi150","id":"foo.a","kernel_description":"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024","kernel_version":"5.14.0-505.el9.x86_64","mem_swap_kb":"0","mem_total_kb":"32487300","num_handles":"1","os":"Linux","pid":"2","realm_id":"","realm_name":"","zone_id":"faa33b33-895b-44d2-8ef5-97d186d0231e","zone_name":"default","zonegroup_id":"247f1a8f-1665-4228-87e1-08d663c7ff54","zonegroup_name":"default"},"task_status":{}}}}}},"progress_events":{"be2a0a0d-be29-4745-aa79-0743b101d3b2":{"message":"Updating node-exporter deployment (+2 -> 2) (5s)\n [==============..............] (remaining: 5s)","progress":0.5,"add_to_ceph_s":true},"e2800faf-1d6e-44af-9b1c-9bdbbb84a689":{"message":"Global Recovery Event (0s)\n [............................] ","progress":0,"add_to_ceph_s":true}}} 2024-09-11T08:09:15.073 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:14 smithi181 podman[63714]: 2024-09-11 08:09:14.645174174 +0000 UTC m=+5.284756678 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-09-11T08:09:15.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:09:15 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:09:15] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:09:15.850 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-11T08:09:15.850 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T08:09:15.892 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:15 smithi181 podman[63714]: 2024-09-11 08:09:15.582121941 +0000 UTC m=+6.221704434 container create b9166bc40a5326f874f68362d7558e38183100b6e2db11c98759f0b27273aa7f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-11T08:09:16.099 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:09:16.213 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:15 smithi181 ceph-mon[32701]: 6.e deep-scrub starts 2024-09-11T08:09:16.214 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:15 smithi181 ceph-mon[32701]: 6.e deep-scrub ok 2024-09-11T08:09:16.214 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:15 smithi181 ceph-mon[32701]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 41 B/s, 2 objects/s recovering 2024-09-11T08:09:16.214 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:15 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2461801247' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-09-11T08:09:16.214 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:15 smithi181 ceph-mon[32701]: 3.a scrub starts 2024-09-11T08:09:16.214 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:15 smithi181 ceph-mon[32701]: 3.a scrub ok 2024-09-11T08:09:16.220 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:15 smithi150 ceph-mon[29276]: 6.e deep-scrub starts 2024-09-11T08:09:16.220 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:15 smithi150 ceph-mon[29276]: 6.e deep-scrub ok 2024-09-11T08:09:16.220 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:15 smithi150 ceph-mon[29276]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 41 B/s, 2 objects/s recovering 2024-09-11T08:09:16.220 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:15 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2461801247' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-09-11T08:09:16.220 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:15 smithi150 ceph-mon[29276]: 3.a scrub starts 2024-09-11T08:09:16.220 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:15 smithi150 ceph-mon[29276]: 3.a scrub ok 2024-09-11T08:09:16.220 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:15 smithi150 ceph-mon[39612]: 6.e deep-scrub starts 2024-09-11T08:09:16.221 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:15 smithi150 ceph-mon[39612]: 6.e deep-scrub ok 2024-09-11T08:09:16.221 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:15 smithi150 ceph-mon[39612]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 41 B/s, 2 objects/s recovering 2024-09-11T08:09:16.221 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:15 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2461801247' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-09-11T08:09:16.221 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:15 smithi150 ceph-mon[39612]: 3.a scrub starts 2024-09-11T08:09:16.221 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:15 smithi150 ceph-mon[39612]: 3.a scrub ok 2024-09-11T08:09:17.191 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:16 smithi150 ceph-mon[29276]: 1.0 scrub starts 2024-09-11T08:09:17.191 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:16 smithi150 ceph-mon[29276]: 1.0 scrub ok 2024-09-11T08:09:17.191 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:16 smithi150 ceph-mon[29276]: 6.17 scrub starts 2024-09-11T08:09:17.191 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:16 smithi150 ceph-mon[29276]: 6.17 scrub ok 2024-09-11T08:09:17.191 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:16 smithi150 ceph-mon[29276]: 5.3 scrub starts 2024-09-11T08:09:17.191 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:16 smithi150 ceph-mon[29276]: 5.3 scrub ok 2024-09-11T08:09:17.191 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:16 smithi150 ceph-mon[39612]: 1.0 scrub starts 2024-09-11T08:09:17.191 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:16 smithi150 ceph-mon[39612]: 1.0 scrub ok 2024-09-11T08:09:17.191 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:16 smithi150 ceph-mon[39612]: 6.17 scrub starts 2024-09-11T08:09:17.192 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:16 smithi150 ceph-mon[39612]: 6.17 scrub ok 2024-09-11T08:09:17.192 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:16 smithi150 ceph-mon[39612]: 5.3 scrub starts 2024-09-11T08:09:17.192 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:16 smithi150 ceph-mon[39612]: 5.3 scrub ok 2024-09-11T08:09:17.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:16 smithi181 ceph-mon[32701]: 1.0 scrub starts 2024-09-11T08:09:17.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:16 smithi181 ceph-mon[32701]: 1.0 scrub ok 2024-09-11T08:09:17.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:16 smithi181 ceph-mon[32701]: 6.17 scrub starts 2024-09-11T08:09:17.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:16 smithi181 ceph-mon[32701]: 6.17 scrub ok 2024-09-11T08:09:17.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:16 smithi181 ceph-mon[32701]: 5.3 scrub starts 2024-09-11T08:09:17.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:16 smithi181 ceph-mon[32701]: 5.3 scrub ok 2024-09-11T08:09:17.737 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 podman[63714]: 2024-09-11 08:09:17.467895332 +0000 UTC m=+8.107477825 container init b9166bc40a5326f874f68362d7558e38183100b6e2db11c98759f0b27273aa7f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-11T08:09:17.737 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 podman[63714]: 2024-09-11 08:09:17.471421541 +0000 UTC m=+8.111004035 container start b9166bc40a5326f874f68362d7558e38183100b6e2db11c98759f0b27273aa7f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-11T08:09:17.737 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b[63791]: ts=2024-09-11T08:09:17.476Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-11T08:09:17.737 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b[63791]: ts=2024-09-11T08:09:17.476Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-11T08:09:17.737 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b[63791]: ts=2024-09-11T08:09:17.476Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-11T08:09:17.737 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b[63791]: ts=2024-09-11T08:09:17.477Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-11T08:09:17.738 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b[63791]: ts=2024-09-11T08:09:17.477Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-11T08:09:17.738 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b[63791]: ts=2024-09-11T08:09:17.477Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-11T08:09:17.738 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b[63791]: ts=2024-09-11T08:09:17.478Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-11T08:09:17.738 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b[63791]: ts=2024-09-11T08:09:17.478Z caller=node_exporter.go:117 level=info collector=arp 2024-09-11T08:09:17.738 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b[63791]: ts=2024-09-11T08:09:17.478Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-11T08:09:17.738 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b[63791]: ts=2024-09-11T08:09:17.478Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-11T08:09:17.738 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b[63791]: ts=2024-09-11T08:09:17.478Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-11T08:09:17.738 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b[63791]: ts=2024-09-11T08:09:17.478Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-11T08:09:17.738 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b[63791]: ts=2024-09-11T08:09:17.478Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-11T08:09:17.738 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b[63791]: ts=2024-09-11T08:09:17.478Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-11T08:09:17.739 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b[63791]: ts=2024-09-11T08:09:17.478Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-11T08:09:17.739 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b[63791]: ts=2024-09-11T08:09:17.478Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-11T08:09:17.739 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b[63791]: ts=2024-09-11T08:09:17.478Z caller=node_exporter.go:117 level=info collector=edac 2024-09-11T08:09:17.739 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b[63791]: ts=2024-09-11T08:09:17.478Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-11T08:09:17.739 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b[63791]: ts=2024-09-11T08:09:17.478Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-11T08:09:17.739 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b[63791]: ts=2024-09-11T08:09:17.478Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-11T08:09:17.739 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b[63791]: ts=2024-09-11T08:09:17.478Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-11T08:09:17.739 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b[63791]: ts=2024-09-11T08:09:17.478Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-11T08:09:17.740 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b[63791]: ts=2024-09-11T08:09:17.478Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-11T08:09:17.740 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b[63791]: ts=2024-09-11T08:09:17.478Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-11T08:09:17.740 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b[63791]: ts=2024-09-11T08:09:17.478Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-11T08:09:17.740 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b[63791]: ts=2024-09-11T08:09:17.478Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-11T08:09:17.740 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b[63791]: ts=2024-09-11T08:09:17.478Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-11T08:09:17.740 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b[63791]: ts=2024-09-11T08:09:17.478Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-11T08:09:17.740 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b[63791]: ts=2024-09-11T08:09:17.478Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-11T08:09:17.740 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b[63791]: ts=2024-09-11T08:09:17.478Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-11T08:09:17.740 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b[63791]: ts=2024-09-11T08:09:17.478Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-11T08:09:17.740 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b[63791]: ts=2024-09-11T08:09:17.478Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-11T08:09:17.741 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b[63791]: ts=2024-09-11T08:09:17.478Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-11T08:09:17.741 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b[63791]: ts=2024-09-11T08:09:17.478Z caller=node_exporter.go:117 level=info collector=os 2024-09-11T08:09:17.741 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b[63791]: ts=2024-09-11T08:09:17.478Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-11T08:09:17.741 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b[63791]: ts=2024-09-11T08:09:17.478Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-11T08:09:17.741 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b[63791]: ts=2024-09-11T08:09:17.478Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-11T08:09:17.741 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b[63791]: ts=2024-09-11T08:09:17.478Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-11T08:09:17.741 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b[63791]: ts=2024-09-11T08:09:17.478Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-11T08:09:17.741 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b[63791]: ts=2024-09-11T08:09:17.478Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-11T08:09:17.741 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b[63791]: ts=2024-09-11T08:09:17.478Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-11T08:09:17.741 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b[63791]: ts=2024-09-11T08:09:17.478Z caller=node_exporter.go:117 level=info collector=stat 2024-09-11T08:09:17.741 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b[63791]: ts=2024-09-11T08:09:17.478Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-11T08:09:17.741 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b[63791]: ts=2024-09-11T08:09:17.478Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-11T08:09:17.741 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b[63791]: ts=2024-09-11T08:09:17.478Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-11T08:09:17.741 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b[63791]: ts=2024-09-11T08:09:17.478Z caller=node_exporter.go:117 level=info collector=time 2024-09-11T08:09:17.741 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b[63791]: ts=2024-09-11T08:09:17.478Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-11T08:09:17.742 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b[63791]: ts=2024-09-11T08:09:17.478Z caller=node_exporter.go:117 level=info collector=uname 2024-09-11T08:09:17.742 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b[63791]: ts=2024-09-11T08:09:17.478Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-11T08:09:17.742 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b[63791]: ts=2024-09-11T08:09:17.478Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-11T08:09:17.742 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b[63791]: ts=2024-09-11T08:09:17.478Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-11T08:09:17.742 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b[63791]: ts=2024-09-11T08:09:17.478Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-11T08:09:17.742 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b[63791]: ts=2024-09-11T08:09:17.478Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-11T08:09:17.911 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:09:17.913 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-09-11T08:09:18.073 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:09:18.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-mon[32701]: 6.1e scrub starts 2024-09-11T08:09:18.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-mon[32701]: 6.1e scrub ok 2024-09-11T08:09:18.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-mon[32701]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 33 B/s, 1 objects/s recovering 2024-09-11T08:09:18.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-mon[32701]: 3.19 scrub starts 2024-09-11T08:09:18.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:17 smithi181 ceph-mon[32701]: 3.19 scrub ok 2024-09-11T08:09:18.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:17 smithi150 ceph-mon[39612]: 6.1e scrub starts 2024-09-11T08:09:18.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:17 smithi150 ceph-mon[39612]: 6.1e scrub ok 2024-09-11T08:09:18.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:17 smithi150 ceph-mon[39612]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 33 B/s, 1 objects/s recovering 2024-09-11T08:09:18.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:17 smithi150 ceph-mon[39612]: 3.19 scrub starts 2024-09-11T08:09:18.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:17 smithi150 ceph-mon[39612]: 3.19 scrub ok 2024-09-11T08:09:18.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:17 smithi150 ceph-mon[29276]: 6.1e scrub starts 2024-09-11T08:09:18.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:17 smithi150 ceph-mon[29276]: 6.1e scrub ok 2024-09-11T08:09:18.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:17 smithi150 ceph-mon[29276]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 33 B/s, 1 objects/s recovering 2024-09-11T08:09:18.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:17 smithi150 ceph-mon[29276]: 3.19 scrub starts 2024-09-11T08:09:18.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:17 smithi150 ceph-mon[29276]: 3.19 scrub ok 2024-09-11T08:09:18.689 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":51,"stamp":"2024-09-11T08:09:16.694681+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596188,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":949,"num_read_kb":806,"num_write":543,"num_write_kb":1671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":590698,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":581,"ondisk_log_size":581,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":395,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2385428,"kb_used_data":6612,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":747575788,"statfs":{"total":767960285184,"available":765517606912,"internally_reserved":0,"allocated":6770688,"data_stored":4069199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.825617"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.937842+0000","last_change":"2024-09-11T08:08:36.105232+0000","last_active":"2024-09-11T08:09:06.937842+0000","last_peered":"2024-09-11T08:09:06.937842+0000","last_clean":"2024-09-11T08:09:06.937842+0000","last_became_active":"2024-09-11T08:07:30.044976+0000","last_became_peered":"2024-09-11T08:07:30.044976+0000","last_unstale":"2024-09-11T08:09:06.937842+0000","last_undegraded":"2024-09-11T08:09:06.937842+0000","last_fullsized":"2024-09-11T08:09:06.937842+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:36.105175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:36.105175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:56.596929+0000","scrub_duration":0.0016664970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":50,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.871277+0000","last_change":"2024-09-11T08:07:32.048578+0000","last_active":"2024-09-11T08:09:06.871277+0000","last_peered":"2024-09-11T08:09:06.871277+0000","last_clean":"2024-09-11T08:09:06.871277+0000","last_became_active":"2024-09-11T08:07:32.048048+0000","last_became_peered":"2024-09-11T08:07:32.048048+0000","last_unstale":"2024-09-11T08:09:06.871277+0000","last_undegraded":"2024-09-11T08:09:06.871277+0000","last_fullsized":"2024-09-11T08:09:06.871277+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.875109+0000","last_change":"2024-09-11T08:08:38.486126+0000","last_active":"2024-09-11T08:09:06.875109+0000","last_peered":"2024-09-11T08:09:06.875109+0000","last_clean":"2024-09-11T08:09:06.875109+0000","last_became_active":"2024-09-11T08:07:34.058828+0000","last_became_peered":"2024-09-11T08:07:34.058828+0000","last_unstale":"2024-09-11T08:09:06.875109+0000","last_undegraded":"2024-09-11T08:09:06.875109+0000","last_fullsized":"2024-09-11T08:09:06.875109+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:38.486057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:38.486057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:44.914804+0000","scrub_duration":0.002230531,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.871537+0000","last_change":"2024-09-11T08:09:02.470722+0000","last_active":"2024-09-11T08:09:06.871537+0000","last_peered":"2024-09-11T08:09:06.871537+0000","last_clean":"2024-09-11T08:09:06.871537+0000","last_became_active":"2024-09-11T08:07:36.074789+0000","last_became_peered":"2024-09-11T08:07:36.074789+0000","last_unstale":"2024-09-11T08:09:06.871537+0000","last_undegraded":"2024-09-11T08:09:06.871537+0000","last_fullsized":"2024-09-11T08:09:06.871537+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:02.470649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:02.470649+0000","last_clean_scrub_stamp":"2024-09-11T08:09:02.470649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:18.421855+0000","scrub_duration":0.00141182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.872003+0000","last_change":"2024-09-11T08:07:36.633716+0000","last_active":"2024-09-11T08:09:06.872003+0000","last_peered":"2024-09-11T08:09:06.872003+0000","last_clean":"2024-09-11T08:09:06.872003+0000","last_became_active":"2024-09-11T08:07:36.633423+0000","last_became_peered":"2024-09-11T08:07:36.633423+0000","last_unstale":"2024-09-11T08:09:06.872003+0000","last_undegraded":"2024-09-11T08:09:06.872003+0000","last_fullsized":"2024-09-11T08:09:06.872003+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.874288+0000","last_change":"2024-09-11T08:07:30.045845+0000","last_active":"2024-09-11T08:09:06.874288+0000","last_peered":"2024-09-11T08:09:06.874288+0000","last_clean":"2024-09-11T08:09:06.874288+0000","last_became_active":"2024-09-11T08:07:30.045579+0000","last_became_peered":"2024-09-11T08:07:30.045579+0000","last_unstale":"2024-09-11T08:09:06.874288+0000","last_undegraded":"2024-09-11T08:09:06.874288+0000","last_fullsized":"2024-09-11T08:09:06.874288+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":69,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.872152+0000","last_change":"2024-09-11T08:08:56.408326+0000","last_active":"2024-09-11T08:09:06.872152+0000","last_peered":"2024-09-11T08:09:06.872152+0000","last_clean":"2024-09-11T08:09:06.872152+0000","last_became_active":"2024-09-11T08:07:32.045848+0000","last_became_peered":"2024-09-11T08:07:32.045848+0000","last_unstale":"2024-09-11T08:09:06.872152+0000","last_undegraded":"2024-09-11T08:09:06.872152+0000","last_fullsized":"2024-09-11T08:09:06.872152+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-11T08:08:56.408272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:56.408272+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:09.630406+0000","scrub_duration":0.001771557,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":32,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.871201+0000","last_change":"2024-09-11T08:07:34.053483+0000","last_active":"2024-09-11T08:09:06.871201+0000","last_peered":"2024-09-11T08:09:06.871201+0000","last_clean":"2024-09-11T08:09:06.871201+0000","last_became_active":"2024-09-11T08:07:34.052243+0000","last_became_peered":"2024-09-11T08:07:34.052243+0000","last_unstale":"2024-09-11T08:09:06.871201+0000","last_undegraded":"2024-09-11T08:09:06.871201+0000","last_fullsized":"2024-09-11T08:09:06.871201+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.875077+0000","last_change":"2024-09-11T08:08:54.878501+0000","last_active":"2024-09-11T08:09:06.875077+0000","last_peered":"2024-09-11T08:09:06.875077+0000","last_clean":"2024-09-11T08:09:06.875077+0000","last_became_active":"2024-09-11T08:07:30.043950+0000","last_became_peered":"2024-09-11T08:07:30.043950+0000","last_unstale":"2024-09-11T08:09:06.875077+0000","last_undegraded":"2024-09-11T08:09:06.875077+0000","last_fullsized":"2024-09-11T08:09:06.875077+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:54.878409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:54.878409+0000","last_clean_scrub_stamp":"2024-09-11T08:08:54.878409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:07.483340+0000","scrub_duration":0.0015972390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":60,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:07.628550+0000","last_change":"2024-09-11T08:09:07.628460+0000","last_active":"2024-09-11T08:09:07.628550+0000","last_peered":"2024-09-11T08:09:07.628550+0000","last_clean":"2024-09-11T08:09:07.628550+0000","last_became_active":"2024-09-11T08:07:32.044626+0000","last_became_peered":"2024-09-11T08:07:32.044626+0000","last_unstale":"2024-09-11T08:09:07.628550+0000","last_undegraded":"2024-09-11T08:09:07.628550+0000","last_fullsized":"2024-09-11T08:09:07.628550+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:09:07.628369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:09:07.628369+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:10.676317+0000","scrub_duration":0.0023051830000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:11.803410+0000","last_change":"2024-09-11T08:09:11.803380+0000","last_active":"2024-09-11T08:09:11.803410+0000","last_peered":"2024-09-11T08:09:11.803410+0000","last_clean":"2024-09-11T08:09:11.803410+0000","last_became_active":"2024-09-11T08:07:34.058689+0000","last_became_peered":"2024-09-11T08:07:34.058689+0000","last_unstale":"2024-09-11T08:09:11.803410+0000","last_undegraded":"2024-09-11T08:09:11.803410+0000","last_fullsized":"2024-09-11T08:09:11.803410+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:11.803311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:09:11.803311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:35.953085+0000","scrub_duration":0.0015061359999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.937357+0000","last_change":"2024-09-11T08:08:40.071203+0000","last_active":"2024-09-11T08:09:06.937357+0000","last_peered":"2024-09-11T08:09:06.937357+0000","last_clean":"2024-09-11T08:09:06.937357+0000","last_became_active":"2024-09-11T08:07:36.076824+0000","last_became_peered":"2024-09-11T08:07:36.076824+0000","last_unstale":"2024-09-11T08:09:06.937357+0000","last_undegraded":"2024-09-11T08:09:06.937357+0000","last_fullsized":"2024-09-11T08:09:06.937357+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:40.071106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:08:40.071106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:54.333357+0000","scrub_duration":0.0019674499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.874855+0000","last_change":"2024-09-11T08:08:36.825857+0000","last_active":"2024-09-11T08:09:06.874855+0000","last_peered":"2024-09-11T08:09:06.874855+0000","last_clean":"2024-09-11T08:09:06.874855+0000","last_became_active":"2024-09-11T08:07:30.043277+0000","last_became_peered":"2024-09-11T08:07:30.043277+0000","last_unstale":"2024-09-11T08:09:06.874855+0000","last_undegraded":"2024-09-11T08:09:06.874855+0000","last_fullsized":"2024-09-11T08:09:06.874855+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:36.825794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:36.825794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:47.493434+0000","scrub_duration":0.0016805920000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":54,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:08.669016+0000","last_change":"2024-09-11T08:09:08.668983+0000","last_active":"2024-09-11T08:09:08.669016+0000","last_peered":"2024-09-11T08:09:08.669016+0000","last_clean":"2024-09-11T08:09:08.669016+0000","last_became_active":"2024-09-11T08:07:32.048661+0000","last_became_peered":"2024-09-11T08:07:32.048661+0000","last_unstale":"2024-09-11T08:09:08.669016+0000","last_undegraded":"2024-09-11T08:09:08.669016+0000","last_fullsized":"2024-09-11T08:09:08.669016+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-11T08:09:08.668925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:09:08.668925+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:30.120080+0000","scrub_duration":0.001704496,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:08.208415+0000","last_change":"2024-09-11T08:09:08.208383+0000","last_active":"2024-09-11T08:09:08.208415+0000","last_peered":"2024-09-11T08:09:08.208415+0000","last_clean":"2024-09-11T08:09:08.208415+0000","last_became_active":"2024-09-11T08:07:34.060871+0000","last_became_peered":"2024-09-11T08:07:34.060871+0000","last_unstale":"2024-09-11T08:09:08.208415+0000","last_undegraded":"2024-09-11T08:09:08.208415+0000","last_fullsized":"2024-09-11T08:09:08.208415+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:08.208319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:09:08.208319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:22.301114+0000","scrub_duration":0.0015808269999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.874542+0000","last_change":"2024-09-11T08:09:04.782636+0000","last_active":"2024-09-11T08:09:06.874542+0000","last_peered":"2024-09-11T08:09:06.874542+0000","last_clean":"2024-09-11T08:09:06.874542+0000","last_became_active":"2024-09-11T08:07:36.077602+0000","last_became_peered":"2024-09-11T08:07:36.077602+0000","last_unstale":"2024-09-11T08:09:06.874542+0000","last_undegraded":"2024-09-11T08:09:06.874542+0000","last_fullsized":"2024-09-11T08:09:06.874542+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:04.782577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:04.782577+0000","last_clean_scrub_stamp":"2024-09-11T08:09:04.782577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:23.367449+0000","scrub_duration":0.0014982330000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.872346+0000","last_change":"2024-09-11T08:07:36.633951+0000","last_active":"2024-09-11T08:09:06.872346+0000","last_peered":"2024-09-11T08:09:06.872346+0000","last_clean":"2024-09-11T08:09:06.872346+0000","last_became_active":"2024-09-11T08:07:36.633338+0000","last_became_peered":"2024-09-11T08:07:36.633338+0000","last_unstale":"2024-09-11T08:09:06.872346+0000","last_undegraded":"2024-09-11T08:09:06.872346+0000","last_fullsized":"2024-09-11T08:09:06.872346+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":46,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:11.966762+0000","last_change":"2024-09-11T08:09:11.966738+0000","last_active":"2024-09-11T08:09:11.966762+0000","last_peered":"2024-09-11T08:09:11.966762+0000","last_clean":"2024-09-11T08:09:11.966762+0000","last_became_active":"2024-09-11T08:07:30.045739+0000","last_became_peered":"2024-09-11T08:07:30.045739+0000","last_unstale":"2024-09-11T08:09:11.966762+0000","last_undegraded":"2024-09-11T08:09:11.966762+0000","last_fullsized":"2024-09-11T08:09:11.966762+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:11.966683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:09:11.966683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:32.828034+0000","scrub_duration":0.0015807659999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"66'15","reported_seq":68,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.877186+0000","last_change":"2024-09-11T08:08:56.975395+0000","last_active":"2024-09-11T08:09:06.877186+0000","last_peered":"2024-09-11T08:09:06.877186+0000","last_clean":"2024-09-11T08:09:06.877186+0000","last_became_active":"2024-09-11T08:07:32.046363+0000","last_became_peered":"2024-09-11T08:07:32.046363+0000","last_unstale":"2024-09-11T08:09:06.877186+0000","last_undegraded":"2024-09-11T08:09:06.877186+0000","last_fullsized":"2024-09-11T08:09:06.877186+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-11T08:08:56.975338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:56.975338+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:19.207433+0000","scrub_duration":0.0017787899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.876129+0000","last_change":"2024-09-11T08:09:01.315948+0000","last_active":"2024-09-11T08:09:06.876129+0000","last_peered":"2024-09-11T08:09:06.876129+0000","last_clean":"2024-09-11T08:09:06.876129+0000","last_became_active":"2024-09-11T08:07:34.055410+0000","last_became_peered":"2024-09-11T08:07:34.055410+0000","last_unstale":"2024-09-11T08:09:06.876129+0000","last_undegraded":"2024-09-11T08:09:06.876129+0000","last_fullsized":"2024-09-11T08:09:06.876129+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:01.315897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:09:01.315897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:14.655816+0000","scrub_duration":0.001561163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.871890+0000","last_change":"2024-09-11T08:07:36.633626+0000","last_active":"2024-09-11T08:09:06.871890+0000","last_peered":"2024-09-11T08:09:06.871890+0000","last_clean":"2024-09-11T08:09:06.871890+0000","last_became_active":"2024-09-11T08:07:36.633258+0000","last_became_peered":"2024-09-11T08:07:36.633258+0000","last_unstale":"2024-09-11T08:09:06.871890+0000","last_undegraded":"2024-09-11T08:09:06.871890+0000","last_fullsized":"2024-09-11T08:09:06.871890+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.872735+0000","last_change":"2024-09-11T08:08:51.401929+0000","last_active":"2024-09-11T08:09:06.872735+0000","last_peered":"2024-09-11T08:09:06.872735+0000","last_clean":"2024-09-11T08:09:06.872735+0000","last_became_active":"2024-09-11T08:07:30.046284+0000","last_became_peered":"2024-09-11T08:07:30.046284+0000","last_unstale":"2024-09-11T08:09:06.872735+0000","last_undegraded":"2024-09-11T08:09:06.872735+0000","last_fullsized":"2024-09-11T08:09:06.872735+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:51.401864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:51.401864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:18.688155+0000","scrub_duration":0.001460034,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":89,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:09.426269+0000","last_change":"2024-09-11T08:09:09.426244+0000","last_active":"2024-09-11T08:09:09.426269+0000","last_peered":"2024-09-11T08:09:09.426269+0000","last_clean":"2024-09-11T08:09:09.426269+0000","last_became_active":"2024-09-11T08:09:07.878432+0000","last_became_peered":"2024-09-11T08:09:07.878432+0000","last_unstale":"2024-09-11T08:09:09.426269+0000","last_undegraded":"2024-09-11T08:09:09.426269+0000","last_fullsized":"2024-09-11T08:09:09.426269+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-11T08:09:09.426191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:09:09.426191+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:25.720395+0000","scrub_duration":0.0016259,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.871365+0000","last_change":"2024-09-11T08:08:40.419900+0000","last_active":"2024-09-11T08:09:06.871365+0000","last_peered":"2024-09-11T08:09:06.871365+0000","last_clean":"2024-09-11T08:09:06.871365+0000","last_became_active":"2024-09-11T08:07:34.056528+0000","last_became_peered":"2024-09-11T08:07:34.056528+0000","last_unstale":"2024-09-11T08:09:06.871365+0000","last_undegraded":"2024-09-11T08:09:06.871365+0000","last_fullsized":"2024-09-11T08:09:06.871365+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:40.419830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:40.419830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:09.111095+0000","scrub_duration":0.0029308149999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:12.217943+0000","last_change":"2024-09-11T08:09:12.217904+0000","last_active":"2024-09-11T08:09:12.217943+0000","last_peered":"2024-09-11T08:09:12.217943+0000","last_clean":"2024-09-11T08:09:12.217943+0000","last_became_active":"2024-09-11T08:07:36.072347+0000","last_became_peered":"2024-09-11T08:07:36.072347+0000","last_unstale":"2024-09-11T08:09:12.217943+0000","last_undegraded":"2024-09-11T08:09:12.217943+0000","last_fullsized":"2024-09-11T08:09:12.217943+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-11T08:09:12.217832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:09:12.217832+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:17.655732+0000","scrub_duration":0.0020299329999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.875416+0000","last_change":"2024-09-11T08:07:30.046498+0000","last_active":"2024-09-11T08:09:06.875416+0000","last_peered":"2024-09-11T08:09:06.875416+0000","last_clean":"2024-09-11T08:09:06.875416+0000","last_became_active":"2024-09-11T08:07:30.045932+0000","last_became_peered":"2024-09-11T08:07:30.045932+0000","last_unstale":"2024-09-11T08:09:06.875416+0000","last_undegraded":"2024-09-11T08:09:06.875416+0000","last_fullsized":"2024-09-11T08:09:06.875416+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":49,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.937744+0000","last_change":"2024-09-11T08:07:32.048372+0000","last_active":"2024-09-11T08:09:06.937744+0000","last_peered":"2024-09-11T08:09:06.937744+0000","last_clean":"2024-09-11T08:09:06.937744+0000","last_became_active":"2024-09-11T08:07:32.048022+0000","last_became_peered":"2024-09-11T08:07:32.048022+0000","last_unstale":"2024-09-11T08:09:06.937744+0000","last_undegraded":"2024-09-11T08:09:06.937744+0000","last_fullsized":"2024-09-11T08:09:06.937744+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":72,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:08.106120+0000","last_change":"2024-09-11T08:08:54.652958+0000","last_active":"2024-09-11T08:09:08.106120+0000","last_peered":"2024-09-11T08:09:08.106120+0000","last_clean":"2024-09-11T08:09:08.106120+0000","last_became_active":"2024-09-11T08:07:34.054014+0000","last_became_peered":"2024-09-11T08:07:34.054014+0000","last_unstale":"2024-09-11T08:09:08.106120+0000","last_undegraded":"2024-09-11T08:09:08.106120+0000","last_fullsized":"2024-09-11T08:09:08.106120+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:08:54.652867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:54.652867+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:09.166003+0000","scrub_duration":0.0022231320000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":69,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.875705+0000","last_change":"2024-09-11T08:08:45.358105+0000","last_active":"2024-09-11T08:09:06.875705+0000","last_peered":"2024-09-11T08:09:06.875705+0000","last_clean":"2024-09-11T08:09:06.875705+0000","last_became_active":"2024-09-11T08:07:32.054317+0000","last_became_peered":"2024-09-11T08:07:32.054317+0000","last_unstale":"2024-09-11T08:09:06.875705+0000","last_undegraded":"2024-09-11T08:09:06.875705+0000","last_fullsized":"2024-09-11T08:09:06.875705+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-11T08:08:45.358041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:45.358041+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:50.858524+0000","scrub_duration":0.0021022879999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.871677+0000","last_change":"2024-09-11T08:07:30.044933+0000","last_active":"2024-09-11T08:09:06.871677+0000","last_peered":"2024-09-11T08:09:06.871677+0000","last_clean":"2024-09-11T08:09:06.871677+0000","last_became_active":"2024-09-11T08:07:30.044624+0000","last_became_peered":"2024-09-11T08:07:30.044624+0000","last_unstale":"2024-09-11T08:09:06.871677+0000","last_undegraded":"2024-09-11T08:09:06.871677+0000","last_fullsized":"2024-09-11T08:09:06.871677+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":70,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:10.668105+0000","last_change":"2024-09-11T08:09:10.668081+0000","last_active":"2024-09-11T08:09:10.668105+0000","last_peered":"2024-09-11T08:09:10.668105+0000","last_clean":"2024-09-11T08:09:10.668105+0000","last_became_active":"2024-09-11T08:07:34.052981+0000","last_became_peered":"2024-09-11T08:07:34.052981+0000","last_unstale":"2024-09-11T08:09:10.668105+0000","last_undegraded":"2024-09-11T08:09:10.668105+0000","last_fullsized":"2024-09-11T08:09:10.668105+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:09:10.668024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:09:10.668024+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:19.607766+0000","scrub_duration":0.0015739580000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.874836+0000","last_change":"2024-09-11T08:08:51.893846+0000","last_active":"2024-09-11T08:09:06.874836+0000","last_peered":"2024-09-11T08:09:06.874836+0000","last_clean":"2024-09-11T08:09:06.874836+0000","last_became_active":"2024-09-11T08:07:36.073969+0000","last_became_peered":"2024-09-11T08:07:36.073969+0000","last_unstale":"2024-09-11T08:09:06.874836+0000","last_undegraded":"2024-09-11T08:09:06.874836+0000","last_fullsized":"2024-09-11T08:09:06.874836+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:51.893774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:08:51.893774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:00.876260+0000","scrub_duration":0.001532858,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":76,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.871575+0000","last_change":"2024-09-11T08:08:43.441600+0000","last_active":"2024-09-11T08:09:06.871575+0000","last_peered":"2024-09-11T08:09:06.871575+0000","last_clean":"2024-09-11T08:09:06.871575+0000","last_became_active":"2024-09-11T08:07:32.045332+0000","last_became_peered":"2024-09-11T08:07:32.045332+0000","last_unstale":"2024-09-11T08:09:06.871575+0000","last_undegraded":"2024-09-11T08:09:06.871575+0000","last_fullsized":"2024-09-11T08:09:06.871575+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-11T08:08:43.441544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:43.441544+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:55.649394+0000","scrub_duration":0.002168648,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.872253+0000","last_change":"2024-09-11T08:07:30.047446+0000","last_active":"2024-09-11T08:09:06.872253+0000","last_peered":"2024-09-11T08:09:06.872253+0000","last_clean":"2024-09-11T08:09:06.872253+0000","last_became_active":"2024-09-11T08:07:30.047176+0000","last_became_peered":"2024-09-11T08:07:30.047176+0000","last_unstale":"2024-09-11T08:09:06.872253+0000","last_undegraded":"2024-09-11T08:09:06.872253+0000","last_fullsized":"2024-09-11T08:09:06.872253+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":32,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.872674+0000","last_change":"2024-09-11T08:07:34.055816+0000","last_active":"2024-09-11T08:09:06.872674+0000","last_peered":"2024-09-11T08:09:06.872674+0000","last_clean":"2024-09-11T08:09:06.872674+0000","last_became_active":"2024-09-11T08:07:34.055523+0000","last_became_peered":"2024-09-11T08:07:34.055523+0000","last_unstale":"2024-09-11T08:09:06.872674+0000","last_undegraded":"2024-09-11T08:09:06.872674+0000","last_fullsized":"2024-09-11T08:09:06.872674+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.871712+0000","last_change":"2024-09-11T08:07:36.069535+0000","last_active":"2024-09-11T08:09:06.871712+0000","last_peered":"2024-09-11T08:09:06.871712+0000","last_clean":"2024-09-11T08:09:06.871712+0000","last_became_active":"2024-09-11T08:07:36.069225+0000","last_became_peered":"2024-09-11T08:07:36.069225+0000","last_unstale":"2024-09-11T08:09:06.871712+0000","last_undegraded":"2024-09-11T08:09:06.871712+0000","last_fullsized":"2024-09-11T08:09:06.871712+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":65,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:11.688244+0000","last_change":"2024-09-11T08:09:11.688215+0000","last_active":"2024-09-11T08:09:11.688244+0000","last_peered":"2024-09-11T08:09:11.688244+0000","last_clean":"2024-09-11T08:09:11.688244+0000","last_became_active":"2024-09-11T08:07:32.040069+0000","last_became_peered":"2024-09-11T08:07:32.040069+0000","last_unstale":"2024-09-11T08:09:11.688244+0000","last_undegraded":"2024-09-11T08:09:11.688244+0000","last_fullsized":"2024-09-11T08:09:11.688244+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-11T08:09:11.688151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:09:11.688151+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:27.200867+0000","scrub_duration":0.0019655670000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.960732+0000","last_change":"2024-09-11T08:09:06.960696+0000","last_active":"2024-09-11T08:09:06.960732+0000","last_peered":"2024-09-11T08:09:06.960732+0000","last_clean":"2024-09-11T08:09:06.960732+0000","last_became_active":"2024-09-11T08:07:30.050484+0000","last_became_peered":"2024-09-11T08:07:30.050484+0000","last_unstale":"2024-09-11T08:09:06.960732+0000","last_undegraded":"2024-09-11T08:09:06.960732+0000","last_fullsized":"2024-09-11T08:09:06.960732+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:06.960639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:09:06.960639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:19.133864+0000","scrub_duration":0.001342766,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"82'9","reported_seq":74,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:07.871568+0000","last_change":"2024-09-11T08:08:53.421634+0000","last_active":"2024-09-11T08:09:07.871568+0000","last_peered":"2024-09-11T08:09:07.871568+0000","last_clean":"2024-09-11T08:09:07.871568+0000","last_became_active":"2024-09-11T08:07:34.056143+0000","last_became_peered":"2024-09-11T08:07:34.056143+0000","last_unstale":"2024-09-11T08:09:07.871568+0000","last_undegraded":"2024-09-11T08:09:07.871568+0000","last_fullsized":"2024-09-11T08:09:07.871568+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-11T08:08:53.421587+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-09-11T08:08:53.421587+0000","last_clean_scrub_stamp":"2024-09-11T08:08:53.421587+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:11.501044+0000","scrub_duration":0.0021116799999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.870847+0000","last_change":"2024-09-11T08:08:55.249190+0000","last_active":"2024-09-11T08:09:06.870847+0000","last_peered":"2024-09-11T08:09:06.870847+0000","last_clean":"2024-09-11T08:09:06.870847+0000","last_became_active":"2024-09-11T08:07:36.633432+0000","last_became_peered":"2024-09-11T08:07:36.633432+0000","last_unstale":"2024-09-11T08:09:06.870847+0000","last_undegraded":"2024-09-11T08:09:06.870847+0000","last_fullsized":"2024-09-11T08:09:06.870847+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:55.249081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:08:55.249081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:00.965734+0000","scrub_duration":0.001581839,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":49,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.876500+0000","last_change":"2024-09-11T08:07:32.054861+0000","last_active":"2024-09-11T08:09:06.876500+0000","last_peered":"2024-09-11T08:09:06.876500+0000","last_clean":"2024-09-11T08:09:06.876500+0000","last_became_active":"2024-09-11T08:07:32.054294+0000","last_became_peered":"2024-09-11T08:07:32.054294+0000","last_unstale":"2024-09-11T08:09:06.876500+0000","last_undegraded":"2024-09-11T08:09:06.876500+0000","last_fullsized":"2024-09-11T08:09:06.876500+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.875129+0000","last_change":"2024-09-11T08:09:00.819941+0000","last_active":"2024-09-11T08:09:06.875129+0000","last_peered":"2024-09-11T08:09:06.875129+0000","last_clean":"2024-09-11T08:09:06.875129+0000","last_became_active":"2024-09-11T08:07:30.047497+0000","last_became_peered":"2024-09-11T08:07:30.047497+0000","last_unstale":"2024-09-11T08:09:06.875129+0000","last_undegraded":"2024-09-11T08:09:06.875129+0000","last_fullsized":"2024-09-11T08:09:06.875129+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:00.819884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:09:00.819884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:12.227536+0000","scrub_duration":0.0014673640000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":60,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:08.105061+0000","last_change":"2024-09-11T08:07:34.060999+0000","last_active":"2024-09-11T08:09:08.105061+0000","last_peered":"2024-09-11T08:09:08.105061+0000","last_clean":"2024-09-11T08:09:08.105061+0000","last_became_active":"2024-09-11T08:07:34.060660+0000","last_became_peered":"2024-09-11T08:07:34.060660+0000","last_unstale":"2024-09-11T08:09:08.105061+0000","last_undegraded":"2024-09-11T08:09:08.105061+0000","last_fullsized":"2024-09-11T08:09:08.105061+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"66'1","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.937427+0000","last_change":"2024-09-11T08:07:36.072890+0000","last_active":"2024-09-11T08:09:06.937427+0000","last_peered":"2024-09-11T08:09:06.937427+0000","last_clean":"2024-09-11T08:09:06.937427+0000","last_became_active":"2024-09-11T08:07:36.072593+0000","last_became_peered":"2024-09-11T08:07:36.072593+0000","last_unstale":"2024-09-11T08:09:06.937427+0000","last_undegraded":"2024-09-11T08:09:06.937427+0000","last_fullsized":"2024-09-11T08:09:06.937427+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":80,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.937777+0000","last_change":"2024-09-11T08:08:52.842005+0000","last_active":"2024-09-11T08:09:06.937777+0000","last_peered":"2024-09-11T08:09:06.937777+0000","last_clean":"2024-09-11T08:09:06.937777+0000","last_became_active":"2024-09-11T08:07:32.044376+0000","last_became_peered":"2024-09-11T08:07:32.044376+0000","last_unstale":"2024-09-11T08:09:06.937777+0000","last_undegraded":"2024-09-11T08:09:06.937777+0000","last_fullsized":"2024-09-11T08:09:06.937777+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-11T08:08:52.841903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:52.841903+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:08.737667+0000","scrub_duration":0.0025248150000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.877070+0000","last_change":"2024-09-11T08:08:49.384423+0000","last_active":"2024-09-11T08:09:06.877070+0000","last_peered":"2024-09-11T08:09:06.877070+0000","last_clean":"2024-09-11T08:09:06.877070+0000","last_became_active":"2024-09-11T08:07:30.045380+0000","last_became_peered":"2024-09-11T08:07:30.045380+0000","last_unstale":"2024-09-11T08:09:06.877070+0000","last_undegraded":"2024-09-11T08:09:06.877070+0000","last_fullsized":"2024-09-11T08:09:06.877070+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:49.384357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:49.384357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:50.840709+0000","scrub_duration":0.0012073310000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.937668+0000","last_change":"2024-09-11T08:07:34.051286+0000","last_active":"2024-09-11T08:09:06.937668+0000","last_peered":"2024-09-11T08:09:06.937668+0000","last_clean":"2024-09-11T08:09:06.937668+0000","last_became_active":"2024-09-11T08:07:34.051004+0000","last_became_peered":"2024-09-11T08:07:34.051004+0000","last_unstale":"2024-09-11T08:09:06.937668+0000","last_undegraded":"2024-09-11T08:09:06.937668+0000","last_fullsized":"2024-09-11T08:09:06.937668+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.875037+0000","last_change":"2024-09-11T08:07:36.632119+0000","last_active":"2024-09-11T08:09:06.875037+0000","last_peered":"2024-09-11T08:09:06.875037+0000","last_clean":"2024-09-11T08:09:06.875037+0000","last_became_active":"2024-09-11T08:07:36.631668+0000","last_became_peered":"2024-09-11T08:07:36.631668+0000","last_unstale":"2024-09-11T08:09:06.875037+0000","last_undegraded":"2024-09-11T08:09:06.875037+0000","last_fullsized":"2024-09-11T08:09:06.875037+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"86'38","reported_seq":122,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:11.362444+0000","last_change":"2024-09-11T08:09:11.362407+0000","last_active":"2024-09-11T08:09:11.362444+0000","last_peered":"2024-09-11T08:09:11.362444+0000","last_clean":"2024-09-11T08:09:11.362444+0000","last_became_active":"2024-09-11T08:07:32.052956+0000","last_became_peered":"2024-09-11T08:07:32.052956+0000","last_unstale":"2024-09-11T08:09:11.362444+0000","last_undegraded":"2024-09-11T08:09:11.362444+0000","last_fullsized":"2024-09-11T08:09:11.362444+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'38","last_scrub_stamp":"2024-09-11T08:09:11.362339+0000","last_deep_scrub":"86'38","last_deep_scrub_stamp":"2024-09-11T08:09:11.362339+0000","last_clean_scrub_stamp":"2024-09-11T08:09:11.362339+0000","objects_scrubbed":10,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:36.998367+0000","scrub_duration":0.0075376050000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"59'1","reported_seq":55,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.872582+0000","last_change":"2024-09-11T08:08:34.423550+0000","last_active":"2024-09-11T08:09:06.872582+0000","last_peered":"2024-09-11T08:09:06.872582+0000","last_clean":"2024-09-11T08:09:06.872582+0000","last_became_active":"2024-09-11T08:07:30.047513+0000","last_became_peered":"2024-09-11T08:07:30.047513+0000","last_unstale":"2024-09-11T08:09:06.872582+0000","last_undegraded":"2024-09-11T08:09:06.872582+0000","last_fullsized":"2024-09-11T08:09:06.872582+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-11T08:08:34.423474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:34.423474+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:57.535838+0000","scrub_duration":0.0016749149999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":52,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.874774+0000","last_change":"2024-09-11T08:08:37.859401+0000","last_active":"2024-09-11T08:09:06.874774+0000","last_peered":"2024-09-11T08:09:06.874774+0000","last_clean":"2024-09-11T08:09:06.874774+0000","last_became_active":"2024-09-11T08:07:30.043130+0000","last_became_peered":"2024-09-11T08:07:30.043130+0000","last_unstale":"2024-09-11T08:09:06.874774+0000","last_undegraded":"2024-09-11T08:09:06.874774+0000","last_fullsized":"2024-09-11T08:09:06.874774+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-11T08:08:37.859328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:37.859328+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:48.485842+0000","scrub_duration":0.001679816,"objects_trimmed":0,"snaptrim_duration":0.0026120850000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.937675+0000","last_change":"2024-09-11T08:08:46.959561+0000","last_active":"2024-09-11T08:09:06.937675+0000","last_peered":"2024-09-11T08:09:06.937675+0000","last_clean":"2024-09-11T08:09:06.937675+0000","last_became_active":"2024-09-11T08:07:34.057154+0000","last_became_peered":"2024-09-11T08:07:34.057154+0000","last_unstale":"2024-09-11T08:09:06.937675+0000","last_undegraded":"2024-09-11T08:09:06.937675+0000","last_fullsized":"2024-09-11T08:09:06.937675+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:46.959500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:46.959500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:05.161397+0000","scrub_duration":0.001608806,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.874109+0000","last_change":"2024-09-11T08:07:36.076211+0000","last_active":"2024-09-11T08:09:06.874109+0000","last_peered":"2024-09-11T08:09:06.874109+0000","last_clean":"2024-09-11T08:09:06.874109+0000","last_became_active":"2024-09-11T08:07:36.075690+0000","last_became_peered":"2024-09-11T08:07:36.075690+0000","last_unstale":"2024-09-11T08:09:06.874109+0000","last_undegraded":"2024-09-11T08:09:06.874109+0000","last_fullsized":"2024-09-11T08:09:06.874109+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":71,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.875789+0000","last_change":"2024-09-11T08:08:35.441946+0000","last_active":"2024-09-11T08:09:06.875789+0000","last_peered":"2024-09-11T08:09:06.875789+0000","last_clean":"2024-09-11T08:09:06.875789+0000","last_became_active":"2024-09-11T08:07:32.050673+0000","last_became_peered":"2024-09-11T08:07:32.050673+0000","last_unstale":"2024-09-11T08:09:06.875789+0000","last_undegraded":"2024-09-11T08:09:06.875789+0000","last_fullsized":"2024-09-11T08:09:06.875789+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'13","last_scrub_stamp":"2024-09-11T08:08:35.441894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:35.441894+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:02.914423+0000","scrub_duration":0.001888965,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.876600+0000","last_change":"2024-09-11T08:08:33.492709+0000","last_active":"2024-09-11T08:09:06.876600+0000","last_peered":"2024-09-11T08:09:06.876600+0000","last_clean":"2024-09-11T08:09:06.876600+0000","last_became_active":"2024-09-11T08:07:30.047014+0000","last_became_peered":"2024-09-11T08:07:30.047014+0000","last_unstale":"2024-09-11T08:09:06.876600+0000","last_undegraded":"2024-09-11T08:09:06.876600+0000","last_fullsized":"2024-09-11T08:09:06.876600+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:33.492652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:33.492652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:54.786224+0000","scrub_duration":0.00156573,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.877615+0000","last_change":"2024-09-11T08:08:32.909292+0000","last_active":"2024-09-11T08:09:06.877615+0000","last_peered":"2024-09-11T08:09:06.877615+0000","last_clean":"2024-09-11T08:09:06.877615+0000","last_became_active":"2024-09-11T08:07:30.045009+0000","last_became_peered":"2024-09-11T08:07:30.045009+0000","last_unstale":"2024-09-11T08:09:06.877615+0000","last_undegraded":"2024-09-11T08:09:06.877615+0000","last_fullsized":"2024-09-11T08:09:06.877615+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-11T08:08:32.909183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:32.909183+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:02.458449+0000","scrub_duration":0.001625769,"objects_trimmed":0,"snaptrim_duration":0.001859533,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.876433+0000","last_change":"2024-09-11T08:08:58.000006+0000","last_active":"2024-09-11T08:09:06.876433+0000","last_peered":"2024-09-11T08:09:06.876433+0000","last_clean":"2024-09-11T08:09:06.876433+0000","last_became_active":"2024-09-11T08:07:34.063906+0000","last_became_peered":"2024-09-11T08:07:34.063906+0000","last_unstale":"2024-09-11T08:09:06.876433+0000","last_undegraded":"2024-09-11T08:09:06.876433+0000","last_fullsized":"2024-09-11T08:09:06.876433+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:57.999952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:57.999952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:01.053836+0000","scrub_duration":0.0014942130000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.871283+0000","last_change":"2024-09-11T08:08:46.366578+0000","last_active":"2024-09-11T08:09:06.871283+0000","last_peered":"2024-09-11T08:09:06.871283+0000","last_clean":"2024-09-11T08:09:06.871283+0000","last_became_active":"2024-09-11T08:07:36.631480+0000","last_became_peered":"2024-09-11T08:07:36.631480+0000","last_unstale":"2024-09-11T08:09:06.871283+0000","last_undegraded":"2024-09-11T08:09:06.871283+0000","last_fullsized":"2024-09-11T08:09:06.871283+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:46.366508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:08:46.366508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:52.248237+0000","scrub_duration":0.0015317200000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":62,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.937761+0000","last_change":"2024-09-11T08:08:37.115649+0000","last_active":"2024-09-11T08:09:06.937761+0000","last_peered":"2024-09-11T08:09:06.937761+0000","last_clean":"2024-09-11T08:09:06.937761+0000","last_became_active":"2024-09-11T08:07:32.055407+0000","last_became_peered":"2024-09-11T08:07:32.055407+0000","last_unstale":"2024-09-11T08:09:06.937761+0000","last_undegraded":"2024-09-11T08:09:06.937761+0000","last_fullsized":"2024-09-11T08:09:06.937761+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-11T08:08:37.115582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:37.115582+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:57.869478+0000","scrub_duration":0.0020267729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"59'1","reported_seq":53,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.937503+0000","last_change":"2024-09-11T08:08:41.094879+0000","last_active":"2024-09-11T08:09:06.937503+0000","last_peered":"2024-09-11T08:09:06.937503+0000","last_clean":"2024-09-11T08:09:06.937503+0000","last_became_active":"2024-09-11T08:07:30.050504+0000","last_became_peered":"2024-09-11T08:07:30.050504+0000","last_unstale":"2024-09-11T08:09:06.937503+0000","last_undegraded":"2024-09-11T08:09:06.937503+0000","last_fullsized":"2024-09-11T08:09:06.937503+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-11T08:08:41.094821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:41.094821+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:00.561470+0000","scrub_duration":0.0017674310000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":150,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:11.451689+0000","last_change":"2024-09-11T08:09:01.175592+0000","last_active":"2024-09-11T08:09:11.451689+0000","last_peered":"2024-09-11T08:09:11.451689+0000","last_clean":"2024-09-11T08:09:11.451689+0000","last_became_active":"2024-09-11T08:07:30.045051+0000","last_became_peered":"2024-09-11T08:07:30.045051+0000","last_unstale":"2024-09-11T08:09:11.451689+0000","last_undegraded":"2024-09-11T08:09:11.451689+0000","last_fullsized":"2024-09-11T08:09:11.451689+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-11T08:09:01.175530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:09:01.175530+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:30.186314+0000","scrub_duration":0.001752651,"objects_trimmed":0,"snaptrim_duration":0.000318821,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":96,"num_read_kb":91,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.875254+0000","last_change":"2024-09-11T08:09:05.758196+0000","last_active":"2024-09-11T08:09:06.875254+0000","last_peered":"2024-09-11T08:09:06.875254+0000","last_clean":"2024-09-11T08:09:06.875254+0000","last_became_active":"2024-09-11T08:07:34.058962+0000","last_became_peered":"2024-09-11T08:07:34.058962+0000","last_unstale":"2024-09-11T08:09:06.875254+0000","last_undegraded":"2024-09-11T08:09:06.875254+0000","last_fullsized":"2024-09-11T08:09:06.875254+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:05.758141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:09:05.758141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:09.559173+0000","scrub_duration":0.0012345450000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.875199+0000","last_change":"2024-09-11T08:08:50.346126+0000","last_active":"2024-09-11T08:09:06.875199+0000","last_peered":"2024-09-11T08:09:06.875199+0000","last_clean":"2024-09-11T08:09:06.875199+0000","last_became_active":"2024-09-11T08:07:36.077597+0000","last_became_peered":"2024-09-11T08:07:36.077597+0000","last_unstale":"2024-09-11T08:09:06.875199+0000","last_undegraded":"2024-09-11T08:09:06.875199+0000","last_fullsized":"2024-09-11T08:09:06.875199+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:50.346069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:08:50.346069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:15.900195+0000","scrub_duration":0.001294906,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"84'21","reported_seq":72,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.938014+0000","last_change":"2024-09-11T08:08:56.705362+0000","last_active":"2024-09-11T08:09:06.938014+0000","last_peered":"2024-09-11T08:09:06.938014+0000","last_clean":"2024-09-11T08:09:06.938014+0000","last_became_active":"2024-09-11T08:07:32.042339+0000","last_became_peered":"2024-09-11T08:07:32.042339+0000","last_unstale":"2024-09-11T08:09:06.938014+0000","last_undegraded":"2024-09-11T08:09:06.938014+0000","last_fullsized":"2024-09-11T08:09:06.938014+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'21","last_scrub_stamp":"2024-09-11T08:08:56.705306+0000","last_deep_scrub":"84'21","last_deep_scrub_stamp":"2024-09-11T08:08:56.705306+0000","last_clean_scrub_stamp":"2024-09-11T08:08:56.705306+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:14.758337+0000","scrub_duration":0.0050944290000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.874353+0000","last_change":"2024-09-11T08:07:30.047535+0000","last_active":"2024-09-11T08:09:06.874353+0000","last_peered":"2024-09-11T08:09:06.874353+0000","last_clean":"2024-09-11T08:09:06.874353+0000","last_became_active":"2024-09-11T08:07:30.047246+0000","last_became_peered":"2024-09-11T08:07:30.047246+0000","last_unstale":"2024-09-11T08:09:06.874353+0000","last_undegraded":"2024-09-11T08:09:06.874353+0000","last_fullsized":"2024-09-11T08:09:06.874353+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":82,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.871167+0000","last_change":"2024-09-11T08:07:44.054743+0000","last_active":"2024-09-11T08:09:06.871167+0000","last_peered":"2024-09-11T08:09:06.871167+0000","last_clean":"2024-09-11T08:09:06.871167+0000","last_became_active":"2024-09-11T08:07:19.726378+0000","last_became_peered":"2024-09-11T08:07:19.726378+0000","last_unstale":"2024-09-11T08:09:06.871167+0000","last_undegraded":"2024-09-11T08:09:06.871167+0000","last_fullsized":"2024-09-11T08:09:06.871167+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'72","last_scrub_stamp":"2024-09-11T08:07:44.054686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:05:21.231856+0000","last_clean_scrub_stamp":"2024-09-11T08:07:44.054686+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001872931,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.871174+0000","last_change":"2024-09-11T08:08:51.289948+0000","last_active":"2024-09-11T08:09:06.871174+0000","last_peered":"2024-09-11T08:09:06.871174+0000","last_clean":"2024-09-11T08:09:06.871174+0000","last_became_active":"2024-09-11T08:07:34.058414+0000","last_became_peered":"2024-09-11T08:07:34.058414+0000","last_unstale":"2024-09-11T08:09:06.871174+0000","last_undegraded":"2024-09-11T08:09:06.871174+0000","last_fullsized":"2024-09-11T08:09:06.871174+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:51.289872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:51.289872+0000","last_clean_scrub_stamp":"2024-09-11T08:08:51.289872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:08.912691+0000","scrub_duration":0.001988812,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.875166+0000","last_change":"2024-09-11T08:09:01.777522+0000","last_active":"2024-09-11T08:09:06.875166+0000","last_peered":"2024-09-11T08:09:06.875166+0000","last_clean":"2024-09-11T08:09:06.875166+0000","last_became_active":"2024-09-11T08:07:36.072508+0000","last_became_peered":"2024-09-11T08:07:36.072508+0000","last_unstale":"2024-09-11T08:09:06.875166+0000","last_undegraded":"2024-09-11T08:09:06.875166+0000","last_fullsized":"2024-09-11T08:09:06.875166+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:01.777429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:01.777429+0000","last_clean_scrub_stamp":"2024-09-11T08:09:01.777429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:07.815841+0000","scrub_duration":0.001444961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"66'11","reported_seq":63,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.872152+0000","last_change":"2024-09-11T08:08:35.418440+0000","last_active":"2024-09-11T08:09:06.872152+0000","last_peered":"2024-09-11T08:09:06.872152+0000","last_clean":"2024-09-11T08:09:06.872152+0000","last_became_active":"2024-09-11T08:07:32.047971+0000","last_became_peered":"2024-09-11T08:07:32.047971+0000","last_unstale":"2024-09-11T08:09:06.872152+0000","last_undegraded":"2024-09-11T08:09:06.872152+0000","last_fullsized":"2024-09-11T08:09:06.872152+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-11T08:08:35.418378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:35.418378+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:38.165860+0000","scrub_duration":0.001791207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.872840+0000","last_change":"2024-09-11T08:08:36.417975+0000","last_active":"2024-09-11T08:09:06.872840+0000","last_peered":"2024-09-11T08:09:06.872840+0000","last_clean":"2024-09-11T08:09:06.872840+0000","last_became_active":"2024-09-11T08:07:30.039884+0000","last_became_peered":"2024-09-11T08:07:30.039884+0000","last_unstale":"2024-09-11T08:09:06.872840+0000","last_undegraded":"2024-09-11T08:09:06.872840+0000","last_fullsized":"2024-09-11T08:09:06.872840+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:36.417911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:36.417911+0000","last_clean_scrub_stamp":"2024-09-11T08:08:36.417911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:59.734565+0000","scrub_duration":0.0014387580000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.875075+0000","last_change":"2024-09-11T08:08:58.845142+0000","last_active":"2024-09-11T08:09:06.875075+0000","last_peered":"2024-09-11T08:09:06.875075+0000","last_clean":"2024-09-11T08:09:06.875075+0000","last_became_active":"2024-09-11T08:07:34.053347+0000","last_became_peered":"2024-09-11T08:07:34.053347+0000","last_unstale":"2024-09-11T08:09:06.875075+0000","last_undegraded":"2024-09-11T08:09:06.875075+0000","last_fullsized":"2024-09-11T08:09:06.875075+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:58.845085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:58.845085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:08.089077+0000","scrub_duration":0.001233404,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.872446+0000","last_change":"2024-09-11T08:07:36.632329+0000","last_active":"2024-09-11T08:09:06.872446+0000","last_peered":"2024-09-11T08:09:06.872446+0000","last_clean":"2024-09-11T08:09:06.872446+0000","last_became_active":"2024-09-11T08:07:36.631687+0000","last_became_peered":"2024-09-11T08:07:36.631687+0000","last_unstale":"2024-09-11T08:09:06.872446+0000","last_undegraded":"2024-09-11T08:09:06.872446+0000","last_fullsized":"2024-09-11T08:09:06.872446+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"77'25","reported_seq":108,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:08.115802+0000","last_change":"2024-09-11T08:09:08.115802+0000","last_active":"2024-09-11T08:09:08.115802+0000","last_peered":"2024-09-11T08:09:08.115802+0000","last_clean":"2024-09-11T08:09:08.115802+0000","last_became_active":"2024-09-11T08:09:08.106771+0000","last_became_peered":"2024-09-11T08:09:08.106771+0000","last_unstale":"2024-09-11T08:09:08.115802+0000","last_undegraded":"2024-09-11T08:09:08.115802+0000","last_fullsized":"2024-09-11T08:09:08.115802+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'25","last_scrub_stamp":"2024-09-11T08:08:32.458210+0000","last_deep_scrub":"77'25","last_deep_scrub_stamp":"2024-09-11T08:08:32.458210+0000","last_clean_scrub_stamp":"2024-09-11T08:08:32.458210+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:00.312847+0000","scrub_duration":0.0060523319999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.937874+0000","last_change":"2024-09-11T08:07:30.042789+0000","last_active":"2024-09-11T08:09:06.937874+0000","last_peered":"2024-09-11T08:09:06.937874+0000","last_clean":"2024-09-11T08:09:06.937874+0000","last_became_active":"2024-09-11T08:07:30.042555+0000","last_became_peered":"2024-09-11T08:07:30.042555+0000","last_unstale":"2024-09-11T08:09:06.937874+0000","last_undegraded":"2024-09-11T08:09:06.937874+0000","last_fullsized":"2024-09-11T08:09:06.937874+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.875266+0000","last_change":"2024-09-11T08:08:41.437100+0000","last_active":"2024-09-11T08:09:06.875266+0000","last_peered":"2024-09-11T08:09:06.875266+0000","last_clean":"2024-09-11T08:09:06.875266+0000","last_became_active":"2024-09-11T08:07:34.054961+0000","last_became_peered":"2024-09-11T08:07:34.054961+0000","last_unstale":"2024-09-11T08:09:06.875266+0000","last_undegraded":"2024-09-11T08:09:06.875266+0000","last_fullsized":"2024-09-11T08:09:06.875266+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:41.437045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:41.437045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:10.512019+0000","scrub_duration":0.0029926649999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.876629+0000","last_change":"2024-09-11T08:09:04.031954+0000","last_active":"2024-09-11T08:09:06.876629+0000","last_peered":"2024-09-11T08:09:06.876629+0000","last_clean":"2024-09-11T08:09:06.876629+0000","last_became_active":"2024-09-11T08:07:36.071194+0000","last_became_peered":"2024-09-11T08:07:36.071194+0000","last_unstale":"2024-09-11T08:09:06.876629+0000","last_undegraded":"2024-09-11T08:09:06.876629+0000","last_fullsized":"2024-09-11T08:09:06.876629+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:04.031894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:09:04.031894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:28.971103+0000","scrub_duration":0.0013516579999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.871456+0000","last_change":"2024-09-11T08:08:42.447971+0000","last_active":"2024-09-11T08:09:06.871456+0000","last_peered":"2024-09-11T08:09:06.871456+0000","last_clean":"2024-09-11T08:09:06.871456+0000","last_became_active":"2024-09-11T08:07:32.043314+0000","last_became_peered":"2024-09-11T08:07:32.043314+0000","last_unstale":"2024-09-11T08:09:06.871456+0000","last_undegraded":"2024-09-11T08:09:06.871456+0000","last_fullsized":"2024-09-11T08:09:06.871456+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-11T08:08:42.447892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:42.447892+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:03.778832+0000","scrub_duration":0.0039015859999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.872368+0000","last_change":"2024-09-11T08:08:30.327580+0000","last_active":"2024-09-11T08:09:06.872368+0000","last_peered":"2024-09-11T08:09:06.872368+0000","last_clean":"2024-09-11T08:09:06.872368+0000","last_became_active":"2024-09-11T08:07:30.046684+0000","last_became_peered":"2024-09-11T08:07:30.046684+0000","last_unstale":"2024-09-11T08:09:06.872368+0000","last_undegraded":"2024-09-11T08:09:06.872368+0000","last_fullsized":"2024-09-11T08:09:06.872368+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:30.327525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:30.327525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:57.278810+0000","scrub_duration":0.001475589,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"66'8","reported_seq":69,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:07.877337+0000","last_change":"2024-09-11T08:08:49.026859+0000","last_active":"2024-09-11T08:09:07.877337+0000","last_peered":"2024-09-11T08:09:07.877337+0000","last_clean":"2024-09-11T08:09:07.877337+0000","last_became_active":"2024-09-11T08:07:34.062469+0000","last_became_peered":"2024-09-11T08:07:34.062469+0000","last_unstale":"2024-09-11T08:09:07.877337+0000","last_undegraded":"2024-09-11T08:09:07.877337+0000","last_fullsized":"2024-09-11T08:09:07.877337+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:08:49.026755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:49.026755+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:59.460486+0000","scrub_duration":0.0018077970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.871170+0000","last_change":"2024-09-11T08:07:36.074961+0000","last_active":"2024-09-11T08:09:06.871170+0000","last_peered":"2024-09-11T08:09:06.871170+0000","last_clean":"2024-09-11T08:09:06.871170+0000","last_became_active":"2024-09-11T08:07:36.074684+0000","last_became_peered":"2024-09-11T08:07:36.074684+0000","last_unstale":"2024-09-11T08:09:06.871170+0000","last_undegraded":"2024-09-11T08:09:06.871170+0000","last_fullsized":"2024-09-11T08:09:06.871170+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":50,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.937776+0000","last_change":"2024-09-11T08:07:32.055237+0000","last_active":"2024-09-11T08:09:06.937776+0000","last_peered":"2024-09-11T08:09:06.937776+0000","last_clean":"2024-09-11T08:09:06.937776+0000","last_became_active":"2024-09-11T08:07:32.054885+0000","last_became_peered":"2024-09-11T08:07:32.054885+0000","last_unstale":"2024-09-11T08:09:06.937776+0000","last_undegraded":"2024-09-11T08:09:06.937776+0000","last_fullsized":"2024-09-11T08:09:06.937776+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.874962+0000","last_change":"2024-09-11T08:08:34.828231+0000","last_active":"2024-09-11T08:09:06.874962+0000","last_peered":"2024-09-11T08:09:06.874962+0000","last_clean":"2024-09-11T08:09:06.874962+0000","last_became_active":"2024-09-11T08:07:30.045613+0000","last_became_peered":"2024-09-11T08:07:30.045613+0000","last_unstale":"2024-09-11T08:09:06.874962+0000","last_undegraded":"2024-09-11T08:09:06.874962+0000","last_fullsized":"2024-09-11T08:09:06.874962+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:34.828129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:34.828129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:58.397313+0000","scrub_duration":0.0013782530000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.875873+0000","last_change":"2024-09-11T08:08:37.871017+0000","last_active":"2024-09-11T08:09:06.875873+0000","last_peered":"2024-09-11T08:09:06.875873+0000","last_clean":"2024-09-11T08:09:06.875873+0000","last_became_active":"2024-09-11T08:07:34.056606+0000","last_became_peered":"2024-09-11T08:07:34.056606+0000","last_unstale":"2024-09-11T08:09:06.875873+0000","last_undegraded":"2024-09-11T08:09:06.875873+0000","last_fullsized":"2024-09-11T08:09:06.875873+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:37.870955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:37.870955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:58.505876+0000","scrub_duration":0.0016274460000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.937446+0000","last_change":"2024-09-11T08:09:05.932261+0000","last_active":"2024-09-11T08:09:06.937446+0000","last_peered":"2024-09-11T08:09:06.937446+0000","last_clean":"2024-09-11T08:09:06.937446+0000","last_became_active":"2024-09-11T08:07:36.072452+0000","last_became_peered":"2024-09-11T08:07:36.072452+0000","last_unstale":"2024-09-11T08:09:06.937446+0000","last_undegraded":"2024-09-11T08:09:06.937446+0000","last_fullsized":"2024-09-11T08:09:06.937446+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:05.932180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:09:05.932180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:07.863587+0000","scrub_duration":0.0013481960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":74,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.937899+0000","last_change":"2024-09-11T08:08:53.616543+0000","last_active":"2024-09-11T08:09:06.937899+0000","last_peered":"2024-09-11T08:09:06.937899+0000","last_clean":"2024-09-11T08:09:06.937899+0000","last_became_active":"2024-09-11T08:07:32.050880+0000","last_became_peered":"2024-09-11T08:07:32.050880+0000","last_unstale":"2024-09-11T08:09:06.937899+0000","last_undegraded":"2024-09-11T08:09:06.937899+0000","last_fullsized":"2024-09-11T08:09:06.937899+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-11T08:08:53.616450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:53.616450+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:18.217121+0000","scrub_duration":0.0024100559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.871804+0000","last_change":"2024-09-11T08:08:44.385440+0000","last_active":"2024-09-11T08:09:06.871804+0000","last_peered":"2024-09-11T08:09:06.871804+0000","last_clean":"2024-09-11T08:09:06.871804+0000","last_became_active":"2024-09-11T08:07:30.043294+0000","last_became_peered":"2024-09-11T08:07:30.043294+0000","last_unstale":"2024-09-11T08:09:06.871804+0000","last_undegraded":"2024-09-11T08:09:06.871804+0000","last_fullsized":"2024-09-11T08:09:06.871804+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:44.385372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:44.385372+0000","last_clean_scrub_stamp":"2024-09-11T08:08:44.385372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:46.537624+0000","scrub_duration":0.001441007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.876135+0000","last_change":"2024-09-11T08:08:51.921479+0000","last_active":"2024-09-11T08:09:06.876135+0000","last_peered":"2024-09-11T08:09:06.876135+0000","last_clean":"2024-09-11T08:09:06.876135+0000","last_became_active":"2024-09-11T08:07:34.063589+0000","last_became_peered":"2024-09-11T08:07:34.063589+0000","last_unstale":"2024-09-11T08:09:06.876135+0000","last_undegraded":"2024-09-11T08:09:06.876135+0000","last_fullsized":"2024-09-11T08:09:06.876135+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:51.921389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:51.921389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:01.266430+0000","scrub_duration":0.001807091,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.871054+0000","last_change":"2024-09-11T08:08:36.367954+0000","last_active":"2024-09-11T08:09:06.871054+0000","last_peered":"2024-09-11T08:09:06.871054+0000","last_clean":"2024-09-11T08:09:06.871054+0000","last_became_active":"2024-09-11T08:07:36.077523+0000","last_became_peered":"2024-09-11T08:07:36.077523+0000","last_unstale":"2024-09-11T08:09:06.871054+0000","last_undegraded":"2024-09-11T08:09:06.871054+0000","last_fullsized":"2024-09-11T08:09:06.871054+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:36.367890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:08:36.367890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:40.081903+0000","scrub_duration":0.0015112560000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":67,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.872732+0000","last_change":"2024-09-11T08:08:59.494039+0000","last_active":"2024-09-11T08:09:06.872732+0000","last_peered":"2024-09-11T08:09:06.872732+0000","last_clean":"2024-09-11T08:09:06.872732+0000","last_became_active":"2024-09-11T08:07:32.048164+0000","last_became_peered":"2024-09-11T08:07:32.048164+0000","last_unstale":"2024-09-11T08:09:06.872732+0000","last_undegraded":"2024-09-11T08:09:06.872732+0000","last_fullsized":"2024-09-11T08:09:06.872732+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-11T08:08:59.493975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:59.493975+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:17.528763+0000","scrub_duration":0.0019438280000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.872188+0000","last_change":"2024-09-11T08:08:53.247349+0000","last_active":"2024-09-11T08:09:06.872188+0000","last_peered":"2024-09-11T08:09:06.872188+0000","last_clean":"2024-09-11T08:09:06.872188+0000","last_became_active":"2024-09-11T08:07:30.041219+0000","last_became_peered":"2024-09-11T08:07:30.041219+0000","last_unstale":"2024-09-11T08:09:06.872188+0000","last_undegraded":"2024-09-11T08:09:06.872188+0000","last_fullsized":"2024-09-11T08:09:06.872188+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:53.247275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:53.247275+0000","last_clean_scrub_stamp":"2024-09-11T08:08:53.247275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:00.288102+0000","scrub_duration":0.0012568169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.874982+0000","last_change":"2024-09-11T08:08:46.970593+0000","last_active":"2024-09-11T08:09:06.874982+0000","last_peered":"2024-09-11T08:09:06.874982+0000","last_clean":"2024-09-11T08:09:06.874982+0000","last_became_active":"2024-09-11T08:07:34.057479+0000","last_became_peered":"2024-09-11T08:07:34.057479+0000","last_unstale":"2024-09-11T08:09:06.874982+0000","last_undegraded":"2024-09-11T08:09:06.874982+0000","last_fullsized":"2024-09-11T08:09:06.874982+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:46.970539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:46.970539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:09.551366+0000","scrub_duration":0.001167143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.872073+0000","last_change":"2024-09-11T08:09:00.395353+0000","last_active":"2024-09-11T08:09:06.872073+0000","last_peered":"2024-09-11T08:09:06.872073+0000","last_clean":"2024-09-11T08:09:06.872073+0000","last_became_active":"2024-09-11T08:07:36.078075+0000","last_became_peered":"2024-09-11T08:07:36.078075+0000","last_unstale":"2024-09-11T08:09:06.872073+0000","last_undegraded":"2024-09-11T08:09:06.872073+0000","last_fullsized":"2024-09-11T08:09:06.872073+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:00.395264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:09:00.395264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:07.333876+0000","scrub_duration":0.001470147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":68,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:10.812355+0000","last_change":"2024-09-11T08:09:10.812331+0000","last_active":"2024-09-11T08:09:10.812355+0000","last_peered":"2024-09-11T08:09:10.812355+0000","last_clean":"2024-09-11T08:09:10.812355+0000","last_became_active":"2024-09-11T08:07:32.039275+0000","last_became_peered":"2024-09-11T08:07:32.039275+0000","last_unstale":"2024-09-11T08:09:10.812355+0000","last_undegraded":"2024-09-11T08:09:10.812355+0000","last_fullsized":"2024-09-11T08:09:10.812355+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-11T08:09:10.812274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:09:10.812274+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:23.639951+0000","scrub_duration":0.0016606539999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":53,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.872576+0000","last_change":"2024-09-11T08:07:30.046228+0000","last_active":"2024-09-11T08:09:06.872576+0000","last_peered":"2024-09-11T08:09:06.872576+0000","last_clean":"2024-09-11T08:09:06.872576+0000","last_became_active":"2024-09-11T08:07:30.045967+0000","last_became_peered":"2024-09-11T08:07:30.045967+0000","last_unstale":"2024-09-11T08:09:06.872576+0000","last_undegraded":"2024-09-11T08:09:06.872576+0000","last_fullsized":"2024-09-11T08:09:06.872576+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"66'8","reported_seq":70,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:07.872017+0000","last_change":"2024-09-11T08:08:59.170974+0000","last_active":"2024-09-11T08:09:07.872017+0000","last_peered":"2024-09-11T08:09:07.872017+0000","last_clean":"2024-09-11T08:09:07.872017+0000","last_became_active":"2024-09-11T08:07:34.058015+0000","last_became_peered":"2024-09-11T08:07:34.058015+0000","last_unstale":"2024-09-11T08:09:07.872017+0000","last_undegraded":"2024-09-11T08:09:07.872017+0000","last_fullsized":"2024-09-11T08:09:07.872017+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:08:59.170918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:59.170918+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:12.511014+0000","scrub_duration":0.001468128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":39,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:09.802062+0000","last_change":"2024-09-11T08:09:09.802025+0000","last_active":"2024-09-11T08:09:09.802062+0000","last_peered":"2024-09-11T08:09:09.802062+0000","last_clean":"2024-09-11T08:09:09.802062+0000","last_became_active":"2024-09-11T08:07:36.631755+0000","last_became_peered":"2024-09-11T08:07:36.631755+0000","last_unstale":"2024-09-11T08:09:09.802062+0000","last_undegraded":"2024-09-11T08:09:09.802062+0000","last_fullsized":"2024-09-11T08:09:09.802062+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:09.801945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:09:09.801945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:36.844477+0000","scrub_duration":0.0013599739999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:07.818093+0000","last_change":"2024-09-11T08:09:07.818014+0000","last_active":"2024-09-11T08:09:07.818093+0000","last_peered":"2024-09-11T08:09:07.818093+0000","last_clean":"2024-09-11T08:09:07.818093+0000","last_became_active":"2024-09-11T08:07:30.045910+0000","last_became_peered":"2024-09-11T08:07:30.045910+0000","last_unstale":"2024-09-11T08:09:07.818093+0000","last_undegraded":"2024-09-11T08:09:07.818093+0000","last_fullsized":"2024-09-11T08:09:07.818093+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:07.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:09:07.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:31.673092+0000","scrub_duration":0.001347413,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"84'9","reported_seq":56,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.872128+0000","last_change":"2024-09-11T08:09:03.418515+0000","last_active":"2024-09-11T08:09:06.872128+0000","last_peered":"2024-09-11T08:09:06.872128+0000","last_clean":"2024-09-11T08:09:06.872128+0000","last_became_active":"2024-09-11T08:07:32.046891+0000","last_became_peered":"2024-09-11T08:07:32.046891+0000","last_unstale":"2024-09-11T08:09:06.872128+0000","last_undegraded":"2024-09-11T08:09:06.872128+0000","last_fullsized":"2024-09-11T08:09:06.872128+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-09-11T08:09:03.418457+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-09-11T08:09:03.418457+0000","last_clean_scrub_stamp":"2024-09-11T08:09:03.418457+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:10.688358+0000","scrub_duration":0.003076058,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:07.779603+0000","last_change":"2024-09-11T08:09:07.779562+0000","last_active":"2024-09-11T08:09:07.779603+0000","last_peered":"2024-09-11T08:09:07.779603+0000","last_clean":"2024-09-11T08:09:07.779603+0000","last_became_active":"2024-09-11T08:07:34.057774+0000","last_became_peered":"2024-09-11T08:07:34.057774+0000","last_unstale":"2024-09-11T08:09:07.779603+0000","last_undegraded":"2024-09-11T08:09:07.779603+0000","last_fullsized":"2024-09-11T08:09:07.779603+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:07.779494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:09:07.779494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:34.783182+0000","scrub_duration":0.0018758360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.871267+0000","last_change":"2024-09-11T08:08:38.326345+0000","last_active":"2024-09-11T08:09:06.871267+0000","last_peered":"2024-09-11T08:09:06.871267+0000","last_clean":"2024-09-11T08:09:06.871267+0000","last_became_active":"2024-09-11T08:07:36.633244+0000","last_became_peered":"2024-09-11T08:07:36.633244+0000","last_unstale":"2024-09-11T08:09:06.871267+0000","last_undegraded":"2024-09-11T08:09:06.871267+0000","last_fullsized":"2024-09-11T08:09:06.871267+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:38.326265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:08:38.326265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:06.322672+0000","scrub_duration":0.001417361,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"84'13","reported_seq":64,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.937847+0000","last_change":"2024-09-11T08:08:57.835686+0000","last_active":"2024-09-11T08:09:06.937847+0000","last_peered":"2024-09-11T08:09:06.937847+0000","last_clean":"2024-09-11T08:09:06.937847+0000","last_became_active":"2024-09-11T08:07:32.048350+0000","last_became_peered":"2024-09-11T08:07:32.048350+0000","last_unstale":"2024-09-11T08:09:06.937847+0000","last_undegraded":"2024-09-11T08:09:06.937847+0000","last_fullsized":"2024-09-11T08:09:06.937847+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'13","last_scrub_stamp":"2024-09-11T08:08:57.835634+0000","last_deep_scrub":"84'13","last_deep_scrub_stamp":"2024-09-11T08:08:57.835634+0000","last_clean_scrub_stamp":"2024-09-11T08:08:57.835634+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:18.729258+0000","scrub_duration":0.0033293229999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.871043+0000","last_change":"2024-09-11T08:08:30.471690+0000","last_active":"2024-09-11T08:09:06.871043+0000","last_peered":"2024-09-11T08:09:06.871043+0000","last_clean":"2024-09-11T08:09:06.871043+0000","last_became_active":"2024-09-11T08:07:30.044055+0000","last_became_peered":"2024-09-11T08:07:30.044055+0000","last_unstale":"2024-09-11T08:09:06.871043+0000","last_undegraded":"2024-09-11T08:09:06.871043+0000","last_fullsized":"2024-09-11T08:09:06.871043+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:30.471620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:30.471620+0000","last_clean_scrub_stamp":"2024-09-11T08:08:30.471620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:37.891558+0000","scrub_duration":0.0016049980000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.871964+0000","last_change":"2024-09-11T08:09:04.370800+0000","last_active":"2024-09-11T08:09:06.871964+0000","last_peered":"2024-09-11T08:09:06.871964+0000","last_clean":"2024-09-11T08:09:06.871964+0000","last_became_active":"2024-09-11T08:07:34.057127+0000","last_became_peered":"2024-09-11T08:07:34.057127+0000","last_unstale":"2024-09-11T08:09:06.871964+0000","last_undegraded":"2024-09-11T08:09:06.871964+0000","last_fullsized":"2024-09-11T08:09:06.871964+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:04.370734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:04.370734+0000","last_clean_scrub_stamp":"2024-09-11T08:09:04.370734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:09.164201+0000","scrub_duration":0.001224483,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.876838+0000","last_change":"2024-09-11T08:09:01.995115+0000","last_active":"2024-09-11T08:09:06.876838+0000","last_peered":"2024-09-11T08:09:06.876838+0000","last_clean":"2024-09-11T08:09:06.876838+0000","last_became_active":"2024-09-11T08:07:36.071513+0000","last_became_peered":"2024-09-11T08:07:36.071513+0000","last_unstale":"2024-09-11T08:09:06.876838+0000","last_undegraded":"2024-09-11T08:09:06.876838+0000","last_fullsized":"2024-09-11T08:09:06.876838+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:01.995061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:09:01.995061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:18.017918+0000","scrub_duration":0.00115722,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.875054+0000","last_change":"2024-09-11T08:08:49.951403+0000","last_active":"2024-09-11T08:09:06.875054+0000","last_peered":"2024-09-11T08:09:06.875054+0000","last_clean":"2024-09-11T08:09:06.875054+0000","last_became_active":"2024-09-11T08:07:32.041761+0000","last_became_peered":"2024-09-11T08:07:32.041761+0000","last_unstale":"2024-09-11T08:09:06.875054+0000","last_undegraded":"2024-09-11T08:09:06.875054+0000","last_fullsized":"2024-09-11T08:09:06.875054+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:08:49.951345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:49.951345+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:51.233235+0000","scrub_duration":0.002079914,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.937500+0000","last_change":"2024-09-11T08:08:48.935452+0000","last_active":"2024-09-11T08:09:06.937500+0000","last_peered":"2024-09-11T08:09:06.937500+0000","last_clean":"2024-09-11T08:09:06.937500+0000","last_became_active":"2024-09-11T08:07:30.045380+0000","last_became_peered":"2024-09-11T08:07:30.045380+0000","last_unstale":"2024-09-11T08:09:06.937500+0000","last_undegraded":"2024-09-11T08:09:06.937500+0000","last_fullsized":"2024-09-11T08:09:06.937500+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:48.935387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:48.935387+0000","last_clean_scrub_stamp":"2024-09-11T08:08:48.935387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:05.810526+0000","scrub_duration":0.001422134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"66'8","reported_seq":70,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:07.872094+0000","last_change":"2024-09-11T08:08:52.374593+0000","last_active":"2024-09-11T08:09:07.872094+0000","last_peered":"2024-09-11T08:09:07.872094+0000","last_clean":"2024-09-11T08:09:07.872094+0000","last_became_active":"2024-09-11T08:07:34.057937+0000","last_became_peered":"2024-09-11T08:07:34.057937+0000","last_unstale":"2024-09-11T08:09:07.872094+0000","last_undegraded":"2024-09-11T08:09:07.872094+0000","last_fullsized":"2024-09-11T08:09:07.872094+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:08:52.374539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:52.374539+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:12.088877+0000","scrub_duration":0.00146886,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.871724+0000","last_change":"2024-09-11T08:07:36.068798+0000","last_active":"2024-09-11T08:09:06.871724+0000","last_peered":"2024-09-11T08:09:06.871724+0000","last_clean":"2024-09-11T08:09:06.871724+0000","last_became_active":"2024-09-11T08:07:36.068493+0000","last_became_peered":"2024-09-11T08:07:36.068493+0000","last_unstale":"2024-09-11T08:09:06.871724+0000","last_undegraded":"2024-09-11T08:09:06.871724+0000","last_fullsized":"2024-09-11T08:09:06.871724+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":81,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:07.884651+0000","last_change":"2024-09-11T08:09:07.884651+0000","last_active":"2024-09-11T08:09:07.884651+0000","last_peered":"2024-09-11T08:09:07.884651+0000","last_clean":"2024-09-11T08:09:07.884651+0000","last_became_active":"2024-09-11T08:09:07.878358+0000","last_became_peered":"2024-09-11T08:09:07.878358+0000","last_unstale":"2024-09-11T08:09:07.884651+0000","last_undegraded":"2024-09-11T08:09:07.884651+0000","last_fullsized":"2024-09-11T08:09:07.884651+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-11T08:08:36.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:36.400586+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:57.137379+0000","scrub_duration":0.002193113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.871165+0000","last_change":"2024-09-11T08:09:03.210218+0000","last_active":"2024-09-11T08:09:06.871165+0000","last_peered":"2024-09-11T08:09:06.871165+0000","last_clean":"2024-09-11T08:09:06.871165+0000","last_became_active":"2024-09-11T08:07:30.039673+0000","last_became_peered":"2024-09-11T08:07:30.039673+0000","last_unstale":"2024-09-11T08:09:06.871165+0000","last_undegraded":"2024-09-11T08:09:06.871165+0000","last_fullsized":"2024-09-11T08:09:06.871165+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:03.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:09:03.210166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:05.614628+0000","scrub_duration":0.0011643720000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":69,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:08.105575+0000","last_change":"2024-09-11T08:08:47.966763+0000","last_active":"2024-09-11T08:09:08.105575+0000","last_peered":"2024-09-11T08:09:08.105575+0000","last_clean":"2024-09-11T08:09:08.105575+0000","last_became_active":"2024-09-11T08:07:34.060736+0000","last_became_peered":"2024-09-11T08:07:34.060736+0000","last_unstale":"2024-09-11T08:09:08.105575+0000","last_undegraded":"2024-09-11T08:09:08.105575+0000","last_fullsized":"2024-09-11T08:09:08.105575+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:08:47.966690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:47.966690+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:12.893275+0000","scrub_duration":0.0019777940000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.937623+0000","last_change":"2024-09-11T08:07:36.072648+0000","last_active":"2024-09-11T08:09:06.937623+0000","last_peered":"2024-09-11T08:09:06.937623+0000","last_clean":"2024-09-11T08:09:06.937623+0000","last_became_active":"2024-09-11T08:07:36.072359+0000","last_became_peered":"2024-09-11T08:07:36.072359+0000","last_unstale":"2024-09-11T08:09:06.937623+0000","last_undegraded":"2024-09-11T08:09:06.937623+0000","last_fullsized":"2024-09-11T08:09:06.937623+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":63,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.872546+0000","last_change":"2024-09-11T08:09:03.516720+0000","last_active":"2024-09-11T08:09:06.872546+0000","last_peered":"2024-09-11T08:09:06.872546+0000","last_clean":"2024-09-11T08:09:06.872546+0000","last_became_active":"2024-09-11T08:07:32.047981+0000","last_became_peered":"2024-09-11T08:07:32.047981+0000","last_unstale":"2024-09-11T08:09:06.872546+0000","last_undegraded":"2024-09-11T08:09:06.872546+0000","last_fullsized":"2024-09-11T08:09:06.872546+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-11T08:09:03.516664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:09:03.516664+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:23.127237+0000","scrub_duration":0.001710917,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.871934+0000","last_change":"2024-09-11T08:08:48.330351+0000","last_active":"2024-09-11T08:09:06.871934+0000","last_peered":"2024-09-11T08:09:06.871934+0000","last_clean":"2024-09-11T08:09:06.871934+0000","last_became_active":"2024-09-11T08:07:30.043691+0000","last_became_peered":"2024-09-11T08:07:30.043691+0000","last_unstale":"2024-09-11T08:09:06.871934+0000","last_undegraded":"2024-09-11T08:09:06.871934+0000","last_fullsized":"2024-09-11T08:09:06.871934+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:48.330285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:48.330285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:05.763303+0000","scrub_duration":0.001379929,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":32,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.876371+0000","last_change":"2024-09-11T08:07:34.053427+0000","last_active":"2024-09-11T08:09:06.876371+0000","last_peered":"2024-09-11T08:09:06.876371+0000","last_clean":"2024-09-11T08:09:06.876371+0000","last_became_active":"2024-09-11T08:07:34.053137+0000","last_became_peered":"2024-09-11T08:07:34.053137+0000","last_unstale":"2024-09-11T08:09:06.876371+0000","last_undegraded":"2024-09-11T08:09:06.876371+0000","last_fullsized":"2024-09-11T08:09:06.876371+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.871896+0000","last_change":"2024-09-11T08:07:36.076837+0000","last_active":"2024-09-11T08:09:06.871896+0000","last_peered":"2024-09-11T08:09:06.871896+0000","last_clean":"2024-09-11T08:09:06.871896+0000","last_became_active":"2024-09-11T08:07:36.076132+0000","last_became_peered":"2024-09-11T08:07:36.076132+0000","last_unstale":"2024-09-11T08:09:06.871896+0000","last_undegraded":"2024-09-11T08:09:06.871896+0000","last_fullsized":"2024-09-11T08:09:06.871896+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":60,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.876551+0000","last_change":"2024-09-11T08:08:55.332126+0000","last_active":"2024-09-11T08:09:06.876551+0000","last_peered":"2024-09-11T08:09:06.876551+0000","last_clean":"2024-09-11T08:09:06.876551+0000","last_became_active":"2024-09-11T08:07:32.049368+0000","last_became_peered":"2024-09-11T08:07:32.049368+0000","last_unstale":"2024-09-11T08:09:06.876551+0000","last_undegraded":"2024-09-11T08:09:06.876551+0000","last_fullsized":"2024-09-11T08:09:06.876551+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:08:55.332065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:55.332065+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:15.584089+0000","scrub_duration":0.001632218,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.871401+0000","last_change":"2024-09-11T08:08:45.388090+0000","last_active":"2024-09-11T08:09:06.871401+0000","last_peered":"2024-09-11T08:09:06.871401+0000","last_clean":"2024-09-11T08:09:06.871401+0000","last_became_active":"2024-09-11T08:07:30.037692+0000","last_became_peered":"2024-09-11T08:07:30.037692+0000","last_unstale":"2024-09-11T08:09:06.871401+0000","last_undegraded":"2024-09-11T08:09:06.871401+0000","last_fullsized":"2024-09-11T08:09:06.871401+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:45.388024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:45.388024+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:59.280827+0000","scrub_duration":0.0014875719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":32,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.872132+0000","last_change":"2024-09-11T08:07:34.057358+0000","last_active":"2024-09-11T08:09:06.872132+0000","last_peered":"2024-09-11T08:09:06.872132+0000","last_clean":"2024-09-11T08:09:06.872132+0000","last_became_active":"2024-09-11T08:07:34.057064+0000","last_became_peered":"2024-09-11T08:07:34.057064+0000","last_unstale":"2024-09-11T08:09:06.872132+0000","last_undegraded":"2024-09-11T08:09:06.872132+0000","last_fullsized":"2024-09-11T08:09:06.872132+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:09.974085+0000","last_change":"2024-09-11T08:09:09.974044+0000","last_active":"2024-09-11T08:09:09.974085+0000","last_peered":"2024-09-11T08:09:09.974085+0000","last_clean":"2024-09-11T08:09:09.974085+0000","last_became_active":"2024-09-11T08:07:36.077406+0000","last_became_peered":"2024-09-11T08:07:36.077406+0000","last_unstale":"2024-09-11T08:09:09.974085+0000","last_undegraded":"2024-09-11T08:09:09.974085+0000","last_fullsized":"2024-09-11T08:09:09.974085+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:09.973943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:09.973943+0000","last_clean_scrub_stamp":"2024-09-11T08:09:09.973943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:12.082942+0000","scrub_duration":0.001430668,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":53,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.871972+0000","last_change":"2024-09-11T08:07:32.043568+0000","last_active":"2024-09-11T08:09:06.871972+0000","last_peered":"2024-09-11T08:09:06.871972+0000","last_clean":"2024-09-11T08:09:06.871972+0000","last_became_active":"2024-09-11T08:07:32.043276+0000","last_became_peered":"2024-09-11T08:07:32.043276+0000","last_unstale":"2024-09-11T08:09:06.871972+0000","last_undegraded":"2024-09-11T08:09:06.871972+0000","last_fullsized":"2024-09-11T08:09:06.871972+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.875072+0000","last_change":"2024-09-11T08:08:55.853168+0000","last_active":"2024-09-11T08:09:06.875072+0000","last_peered":"2024-09-11T08:09:06.875072+0000","last_clean":"2024-09-11T08:09:06.875072+0000","last_became_active":"2024-09-11T08:07:30.049152+0000","last_became_peered":"2024-09-11T08:07:30.049152+0000","last_unstale":"2024-09-11T08:09:06.875072+0000","last_undegraded":"2024-09-11T08:09:06.875072+0000","last_fullsized":"2024-09-11T08:09:06.875072+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:55.853076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:55.853076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:03.761618+0000","scrub_duration":0.0018073309999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:10.182603+0000","last_change":"2024-09-11T08:09:10.182577+0000","last_active":"2024-09-11T08:09:10.182603+0000","last_peered":"2024-09-11T08:09:10.182603+0000","last_clean":"2024-09-11T08:09:10.182603+0000","last_became_active":"2024-09-11T08:07:34.056935+0000","last_became_peered":"2024-09-11T08:07:34.056935+0000","last_unstale":"2024-09-11T08:09:10.182603+0000","last_undegraded":"2024-09-11T08:09:10.182603+0000","last_fullsized":"2024-09-11T08:09:10.182603+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:10.182522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:09:10.182522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:38.697741+0000","scrub_duration":0.001386111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.872294+0000","last_change":"2024-09-11T08:07:36.633645+0000","last_active":"2024-09-11T08:09:06.872294+0000","last_peered":"2024-09-11T08:09:06.872294+0000","last_clean":"2024-09-11T08:09:06.872294+0000","last_became_active":"2024-09-11T08:07:36.633368+0000","last_became_peered":"2024-09-11T08:07:36.633368+0000","last_unstale":"2024-09-11T08:09:06.872294+0000","last_undegraded":"2024-09-11T08:09:06.872294+0000","last_fullsized":"2024-09-11T08:09:06.872294+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.871541+0000","last_change":"2024-09-11T08:07:32.046050+0000","last_active":"2024-09-11T08:09:06.871541+0000","last_peered":"2024-09-11T08:09:06.871541+0000","last_clean":"2024-09-11T08:09:06.871541+0000","last_became_active":"2024-09-11T08:07:32.045728+0000","last_became_peered":"2024-09-11T08:07:32.045728+0000","last_unstale":"2024-09-11T08:09:06.871541+0000","last_undegraded":"2024-09-11T08:09:06.871541+0000","last_fullsized":"2024-09-11T08:09:06.871541+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.937832+0000","last_change":"2024-09-11T08:08:42.056647+0000","last_active":"2024-09-11T08:09:06.937832+0000","last_peered":"2024-09-11T08:09:06.937832+0000","last_clean":"2024-09-11T08:09:06.937832+0000","last_became_active":"2024-09-11T08:07:30.049409+0000","last_became_peered":"2024-09-11T08:07:30.049409+0000","last_unstale":"2024-09-11T08:09:06.937832+0000","last_undegraded":"2024-09-11T08:09:06.937832+0000","last_fullsized":"2024-09-11T08:09:06.937832+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:42.056591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:42.056591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:56.872271+0000","scrub_duration":0.003068809,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.937670+0000","last_change":"2024-09-11T08:08:34.506991+0000","last_active":"2024-09-11T08:09:06.937670+0000","last_peered":"2024-09-11T08:09:06.937670+0000","last_clean":"2024-09-11T08:09:06.937670+0000","last_became_active":"2024-09-11T08:07:34.055082+0000","last_became_peered":"2024-09-11T08:07:34.055082+0000","last_unstale":"2024-09-11T08:09:06.937670+0000","last_undegraded":"2024-09-11T08:09:06.937670+0000","last_fullsized":"2024-09-11T08:09:06.937670+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:34.506927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:34.506927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:46.584592+0000","scrub_duration":0.0015499070000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.870837+0000","last_change":"2024-09-11T08:07:36.072035+0000","last_active":"2024-09-11T08:09:06.870837+0000","last_peered":"2024-09-11T08:09:06.870837+0000","last_clean":"2024-09-11T08:09:06.870837+0000","last_became_active":"2024-09-11T08:07:36.071669+0000","last_became_peered":"2024-09-11T08:07:36.071669+0000","last_unstale":"2024-09-11T08:09:06.870837+0000","last_undegraded":"2024-09-11T08:09:06.870837+0000","last_fullsized":"2024-09-11T08:09:06.870837+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:09.297179+0000","last_change":"2024-09-11T08:09:09.297155+0000","last_active":"2024-09-11T08:09:09.297179+0000","last_peered":"2024-09-11T08:09:09.297179+0000","last_clean":"2024-09-11T08:09:09.297179+0000","last_became_active":"2024-09-11T08:07:36.071468+0000","last_became_peered":"2024-09-11T08:07:36.071468+0000","last_unstale":"2024-09-11T08:09:09.297179+0000","last_undegraded":"2024-09-11T08:09:09.297179+0000","last_fullsized":"2024-09-11T08:09:09.297179+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:09.297101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:09:09.297101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:12.099110+0000","scrub_duration":0.001287886,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.871675+0000","last_change":"2024-09-11T08:08:49.428032+0000","last_active":"2024-09-11T08:09:06.871675+0000","last_peered":"2024-09-11T08:09:06.871675+0000","last_clean":"2024-09-11T08:09:06.871675+0000","last_became_active":"2024-09-11T08:07:30.046702+0000","last_became_peered":"2024-09-11T08:07:30.046702+0000","last_unstale":"2024-09-11T08:09:06.871675+0000","last_undegraded":"2024-09-11T08:09:06.871675+0000","last_fullsized":"2024-09-11T08:09:06.871675+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:49.427944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:49.427944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:11.537893+0000","scrub_duration":0.001383034,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":62,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.938140+0000","last_change":"2024-09-11T08:09:03.841778+0000","last_active":"2024-09-11T08:09:06.938140+0000","last_peered":"2024-09-11T08:09:06.938140+0000","last_clean":"2024-09-11T08:09:06.938140+0000","last_became_active":"2024-09-11T08:07:32.054656+0000","last_became_peered":"2024-09-11T08:07:32.054656+0000","last_unstale":"2024-09-11T08:09:06.938140+0000","last_undegraded":"2024-09-11T08:09:06.938140+0000","last_fullsized":"2024-09-11T08:09:06.938140+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-11T08:09:03.841721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:09:03.841721+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:20.583780+0000","scrub_duration":0.00194933,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.937664+0000","last_change":"2024-09-11T08:08:53.869870+0000","last_active":"2024-09-11T08:09:06.937664+0000","last_peered":"2024-09-11T08:09:06.937664+0000","last_clean":"2024-09-11T08:09:06.937664+0000","last_became_active":"2024-09-11T08:07:34.061453+0000","last_became_peered":"2024-09-11T08:07:34.061453+0000","last_unstale":"2024-09-11T08:09:06.937664+0000","last_undegraded":"2024-09-11T08:09:06.937664+0000","last_fullsized":"2024-09-11T08:09:06.937664+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:53.869813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:53.869813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:59.468130+0000","scrub_duration":0.0012524350000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":423,"ondisk_log_size":423,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":29,"num_read_kb":29,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":96,"num_read_kb":91,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266713,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298540,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446612,"statfs":{"total":95995035648,"available":95689330688,"internally_reserved":0,"allocated":1216512,"data_stored":877700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.29499999999999998}]},{"osd":1,"last update":"Wed Sep 11 08:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.33800000000000002}]},{"osd":2,"last update":"Wed Sep 11 08:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.28299999999999997}]},{"osd":3,"last update":"Wed Sep 11 08:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":7.9210000000000003,"5min":7.9210000000000003,"15min":7.9210000000000003},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.309}]},{"osd":4,"last update":"Wed Sep 11 08:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.25}]},{"osd":5,"last update":"Wed Sep 11 08:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.317}]},{"osd":6,"last update":"Wed Sep 11 08:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.48599999999999999}]}]},{"osd":6,"up_from":45,"seq":193273528350,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298524,"kb_used_data":1172,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446628,"statfs":{"total":95995035648,"available":95689347072,"internally_reserved":0,"allocated":1200128,"data_stored":877523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.49299999999999999,"15min":0.496},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.59099999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.502},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.50700000000000001,"15min":0.50600000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.72099999999999997,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.56200000000000006}]},{"osd":1,"last update":"Wed Sep 11 08:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.47999999999999998,"15min":0.47799999999999998},"min":{"1min":0.32700000000000001,"5min":0.318,"15min":0.318},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.497,"5min":0.505,"15min":0.50600000000000001},"min":{"1min":0.33500000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.65900000000000003,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.49099999999999999}]},{"osd":2,"last update":"Wed Sep 11 08:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.5,"15min":0.502},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.67200000000000004,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.499,"5min":0.48499999999999999,"15min":0.48299999999999998},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.72399999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.48399999999999999}]},{"osd":3,"last update":"Wed Sep 11 08:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.52400000000000002,"15min":0.52600000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.53900000000000003,"15min":0.53800000000000003},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.73499999999999999,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.55500000000000005}]},{"osd":4,"last update":"Wed Sep 11 08:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.39000000000000001,"15min":0.38700000000000001},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.59399999999999997,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.41999999999999998,"15min":0.41199999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.312}]},{"osd":5,"last update":"Wed Sep 11 08:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.48899999999999999,"15min":0.47999999999999998},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.434,"5min":0.41399999999999998,"15min":0.41099999999999998},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.62,"5min":0.629,"15min":0.629},"last":0.29499999999999998}]},{"osd":7,"last update":"Wed Sep 11 08:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.32600000000000001}]}]},{"osd":5,"up_from":38,"seq":163208757283,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297944,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447208,"statfs":{"total":95995035648,"available":95689940992,"internally_reserved":0,"allocated":606208,"data_stored":286472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.51400000000000001,"15min":0.52200000000000002},"min":{"1min":0.318,"5min":0.31,"15min":0.31},"max":{"1min":0.63500000000000001,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.503,"15min":0.501},"min":{"1min":0.41399999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.54000000000000004}]},{"osd":1,"last update":"Wed Sep 11 08:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":1.1910000000000001,"15min":1.3029999999999999},"min":{"1min":0.33000000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.82499999999999996,"5min":16.004999999999999,"15min":16.004999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.36199999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.57999999999999996}]},{"osd":2,"last update":"Wed Sep 11 08:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":1.206,"15min":1.323},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.754,"5min":16.018000000000001,"15min":16.018000000000001},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.55900000000000005}]},{"osd":3,"last update":"Wed Sep 11 08:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.53100000000000003,"15min":0.52900000000000003},"min":{"1min":0.44900000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.72299999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.505,"5min":0.51600000000000001,"15min":0.51800000000000002},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.45400000000000001}]},{"osd":4,"last update":"Wed Sep 11 08:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.39700000000000002,"15min":0.39300000000000002},"min":{"1min":0.27500000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.434},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.40000000000000002,"15min":0.38800000000000001},"min":{"1min":0.36499999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.46899999999999997}]},{"osd":6,"last update":"Wed Sep 11 08:08:56 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.57699999999999996,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.372},{"interface":"front","average":{"1min":0.495,"5min":0.45200000000000001,"15min":0.44500000000000001},"min":{"1min":0.316,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.67100000000000004,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.496}]},{"osd":7,"last update":"Wed Sep 11 08:08:22 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.39800000000000002}]}]},{"osd":4,"up_from":31,"seq":133143986216,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297964,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447188,"statfs":{"total":95995035648,"available":95689920512,"internally_reserved":0,"allocated":626688,"data_stored":287715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:09:10 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.495,"15min":0.50700000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.51100000000000001,"15min":0.53100000000000003},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.624,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.40600000000000003}]},{"osd":1,"last update":"Wed Sep 11 08:09:10 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.496,"15min":0.51000000000000001},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.65600000000000003,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52200000000000002,"15min":0.52900000000000003},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.66900000000000004,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.38}]},{"osd":2,"last update":"Wed Sep 11 08:09:10 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.51200000000000001,"15min":0.52500000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.60999999999999999,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.51800000000000002,"15min":0.52600000000000002},"min":{"1min":0.39900000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.70799999999999996,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.46700000000000003}]},{"osd":3,"last update":"Wed Sep 11 08:09:10 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.50600000000000001,"15min":0.51400000000000001},"min":{"1min":0.36899999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.57899999999999996,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.50800000000000001,"15min":0.52200000000000002},"min":{"1min":0.40799999999999997,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.59999999999999998,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.42799999999999999}]},{"osd":5,"last update":"Wed Sep 11 08:08:31 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.36899999999999999,"15min":0.36699999999999999},"min":{"1min":0.27800000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.36899999999999999,"15min":0.36699999999999999},"min":{"1min":0.27900000000000003,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.32000000000000001}]},{"osd":6,"last update":"Wed Sep 11 08:08:56 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.39100000000000001,"15min":0.38300000000000001},"min":{"1min":0.30399999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.44,"15min":0.432},"min":{"1min":0.32200000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.44900000000000001}]},{"osd":7,"last update":"Wed Sep 11 08:08:20 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.44}]}]},{"osd":3,"up_from":25,"seq":107374182445,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297980,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447172,"statfs":{"total":95995035648,"available":95689904128,"internally_reserved":0,"allocated":643072,"data_stored":287501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:08:53 2024","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.36199999999999999,"15min":0.374},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":0.55100000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.40300000000000002,"15min":0.41799999999999998},"min":{"1min":0.26900000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.55100000000000005,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.28599999999999998}]},{"osd":1,"last update":"Wed Sep 11 08:08:53 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.41599999999999998,"15min":0.437},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.497,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.252},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.41399999999999998,"15min":0.42899999999999999},"min":{"1min":0.24399999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.53800000000000003,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.30099999999999999}]},{"osd":2,"last update":"Wed Sep 11 08:08:53 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.434,"15min":0.44900000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.47999999999999998,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.371,"5min":0.40600000000000003,"15min":0.41899999999999998},"min":{"1min":0.26700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.53100000000000003,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.314}]},{"osd":4,"last update":"Wed Sep 11 08:09:12 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.53900000000000003,"15min":0.54600000000000004},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.52800000000000002,"15min":0.53500000000000003},"min":{"1min":0.34799999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.745,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.67600000000000005}]},{"osd":5,"last update":"Wed Sep 11 08:08:33 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.54200000000000004,"15min":0.53800000000000003},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.40699999999999997,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.58999999999999997}]},{"osd":6,"last update":"Wed Sep 11 08:08:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.53400000000000003,"15min":0.53000000000000003},"min":{"1min":0.35899999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.629},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.55400000000000005,"15min":0.55800000000000005},"min":{"1min":0.39500000000000002,"5min":0.373,"15min":0.373},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.55400000000000005}]},{"osd":7,"last update":"Wed Sep 11 08:08:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.54400000000000004}]}]},{"osd":2,"up_from":18,"seq":77309411376,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297944,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447208,"statfs":{"total":95995035648,"available":95689940992,"internally_reserved":0,"allocated":606208,"data_stored":286313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:08:24 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.38400000000000001,"15min":0.38100000000000001},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.58099999999999996,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.37,"15min":0.35999999999999999},"min":{"1min":0.25,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.55600000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.308}]},{"osd":1,"last update":"Wed Sep 11 08:08:24 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.39500000000000002,"15min":0.38},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.60899999999999999,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.39800000000000002,"15min":0.38600000000000001},"min":{"1min":0.28699999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.27500000000000002}]},{"osd":3,"last update":"Wed Sep 11 08:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.45100000000000001,"15min":0.46100000000000002},"min":{"1min":0.27300000000000002,"5min":0.218,"15min":0.218},"max":{"1min":0.629,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.45100000000000001,"15min":0.46500000000000002},"min":{"1min":0.318,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.34899999999999998}]},{"osd":4,"last update":"Wed Sep 11 08:09:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53100000000000003},"min":{"1min":0.41099999999999998,"5min":0.315,"15min":0.315},"max":{"1min":0.77100000000000002,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.48999999999999999},{"interface":"front","average":{"1min":1.2010000000000001,"5min":0.70499999999999996,"15min":0.627},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":12.129,"5min":12.129,"15min":12.129},"last":0.379}]},{"osd":5,"last update":"Wed Sep 11 08:08:35 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.57699999999999996,"15min":0.58299999999999996},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.751,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.59199999999999997,"15min":0.59999999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.65300000000000002,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.55100000000000005}]},{"osd":6,"last update":"Wed Sep 11 08:08:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.58499999999999996,"15min":0.59399999999999997},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.59099999999999997,"15min":0.59699999999999998},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.70299999999999996,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.45000000000000001}]},{"osd":7,"last update":"Wed Sep 11 08:08:22 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.57699999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574902,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297980,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447172,"statfs":{"total":95995035648,"available":95689904128,"internally_reserved":0,"allocated":643072,"data_stored":287329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:09:04 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.33700000000000002,"15min":0.30399999999999999},"min":{"1min":0.23599999999999999,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.312},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.33900000000000002,"15min":0.315},"min":{"1min":0.224,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.34699999999999998}]},{"osd":2,"last update":"Wed Sep 11 08:08:28 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.36199999999999999,"15min":0.34799999999999998},"min":{"1min":0.27700000000000002,"5min":0.224,"15min":0.224},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.37,"5min":0.35399999999999998,"15min":0.34200000000000003},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.33000000000000002}]},{"osd":3,"last update":"Wed Sep 11 08:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.41399999999999998,"15min":0.42199999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.439,"15min":0.437},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.198,"5min":1.198,"15min":1.198},"last":0.44800000000000001}]},{"osd":4,"last update":"Wed Sep 11 08:09:08 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.79800000000000004,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56100000000000005,"15min":0.55700000000000005},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.68700000000000006}]},{"osd":5,"last update":"Wed Sep 11 08:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.53700000000000003,"15min":0.53000000000000003},"min":{"1min":0.40699999999999997,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.77600000000000002,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.53100000000000003,"15min":0.52700000000000002},"min":{"1min":0.38400000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.51900000000000002}]},{"osd":6,"last update":"Wed Sep 11 08:08:56 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.55500000000000005,"15min":0.55900000000000005},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.59899999999999998,"15min":0.60099999999999998},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.71999999999999997}]},{"osd":7,"last update":"Wed Sep 11 08:08:17 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.73199999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738426,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298552,"kb_used_data":1200,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446600,"statfs":{"total":95995035648,"available":95689318400,"internally_reserved":0,"allocated":1228800,"data_stored":878646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 11 08:09:04 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.34799999999999998,"15min":0.317},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":0.53400000000000003,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.35899999999999999,"15min":0.33800000000000002},"min":{"1min":0.219,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.47999999999999998,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.28899999999999998}]},{"osd":2,"last update":"Wed Sep 11 08:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.34699999999999998},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.48799999999999999,"5min":0.53000000000000003,"15min":0.53000000000000003},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.378,"5min":0.378,"15min":0.36899999999999999},"min":{"1min":0.26600000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.56299999999999994,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.307}]},{"osd":3,"last update":"Wed Sep 11 08:08:51 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.44,"15min":0.44},"min":{"1min":0.27600000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.52300000000000002,"5min":0.62,"15min":0.62},"last":0.373},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.42499999999999999,"15min":0.42599999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.52800000000000002,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.32800000000000001}]},{"osd":4,"last update":"Wed Sep 11 08:09:09 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55100000000000005,"15min":0.54700000000000004},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.44},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.56799999999999995,"15min":0.57399999999999995},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.71699999999999997,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.45000000000000001}]},{"osd":5,"last update":"Wed Sep 11 08:08:36 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.58499999999999996,"15min":0.58299999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.748,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.55800000000000005,"15min":0.55300000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.50700000000000001}]},{"osd":6,"last update":"Wed Sep 11 08:08:59 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.60899999999999999,"15min":0.61699999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.75600000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.60299999999999998,"15min":0.57099999999999995},"min":{"1min":0.433,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":4.7069999999999999,"5min":4.7069999999999999,"15min":4.7069999999999999},"last":0.496}]},{"osd":7,"last update":"Wed Sep 11 08:08:19 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.42999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T08:09:18.750 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:18 smithi181 bash[63714]: b9166bc40a5326f874f68362d7558e38183100b6e2db11c98759f0b27273aa7f 2024-09-11T08:09:18.750 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:09:18 smithi181 systemd[1]: Started Ceph node-exporter.b for c1bddff4-7013-11ef-bcea-c7b262605968. 2024-09-11T08:09:18.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:18 smithi150 ceph-mon[29276]: 3.f scrub starts 2024-09-11T08:09:18.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:18 smithi150 ceph-mon[29276]: 3.f scrub ok 2024-09-11T08:09:18.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:18 smithi150 ceph-mon[29276]: 5.1 scrub starts 2024-09-11T08:09:18.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:18 smithi150 ceph-mon[29276]: 5.1 scrub ok 2024-09-11T08:09:18.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:18 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:09:18.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:18 smithi150 ceph-mon[29276]: from='client.24635 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:09:18.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:18 smithi150 ceph-mon[29276]: 4.2 scrub starts 2024-09-11T08:09:18.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:18 smithi150 ceph-mon[29276]: 4.2 scrub ok 2024-09-11T08:09:18.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:18 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:18.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:18 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:18.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:18 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:18.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:18 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:18.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:18 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:18.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:18 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:09:18.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:18 smithi150 ceph-mon[39612]: 3.f scrub starts 2024-09-11T08:09:18.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:18 smithi150 ceph-mon[39612]: 3.f scrub ok 2024-09-11T08:09:18.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:18 smithi150 ceph-mon[39612]: 5.1 scrub starts 2024-09-11T08:09:18.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:18 smithi150 ceph-mon[39612]: 5.1 scrub ok 2024-09-11T08:09:18.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:18 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:09:18.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:18 smithi150 ceph-mon[39612]: from='client.24635 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:09:18.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:18 smithi150 ceph-mon[39612]: 4.2 scrub starts 2024-09-11T08:09:18.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:18 smithi150 ceph-mon[39612]: 4.2 scrub ok 2024-09-11T08:09:18.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:18 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:18.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:18 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:18.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:18 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:18.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:18 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:18.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:18 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:18.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:18 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:09:19.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:18 smithi181 ceph-mon[32701]: 3.f scrub starts 2024-09-11T08:09:19.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:18 smithi181 ceph-mon[32701]: 3.f scrub ok 2024-09-11T08:09:19.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:18 smithi181 ceph-mon[32701]: 5.1 scrub starts 2024-09-11T08:09:19.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:18 smithi181 ceph-mon[32701]: 5.1 scrub ok 2024-09-11T08:09:19.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:18 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:09:19.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:18 smithi181 ceph-mon[32701]: from='client.24635 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:09:19.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:18 smithi181 ceph-mon[32701]: 4.2 scrub starts 2024-09-11T08:09:19.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:18 smithi181 ceph-mon[32701]: 4.2 scrub ok 2024-09-11T08:09:19.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:18 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:19.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:18 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:19.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:18 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:19.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:18 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:19.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:18 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:19.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:18 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:09:20.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:19 smithi181 ceph-mon[32701]: 6.12 scrub starts 2024-09-11T08:09:20.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:19 smithi181 ceph-mon[32701]: 6.12 scrub ok 2024-09-11T08:09:20.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:19 smithi181 ceph-mon[32701]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 945 B/s rd, 0 op/s; 30 B/s, 1 objects/s recovering 2024-09-11T08:09:20.184 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:19 smithi150 ceph-mon[29276]: 6.12 scrub starts 2024-09-11T08:09:20.184 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:19 smithi150 ceph-mon[29276]: 6.12 scrub ok 2024-09-11T08:09:20.184 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:19 smithi150 ceph-mon[29276]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 945 B/s rd, 0 op/s; 30 B/s, 1 objects/s recovering 2024-09-11T08:09:20.184 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:19 smithi150 ceph-mon[39612]: 6.12 scrub starts 2024-09-11T08:09:20.184 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:19 smithi150 ceph-mon[39612]: 6.12 scrub ok 2024-09-11T08:09:20.185 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:19 smithi150 ceph-mon[39612]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 945 B/s rd, 0 op/s; 30 B/s, 1 objects/s recovering 2024-09-11T08:09:21.697 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T08:09:21.721 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:21 smithi150 ceph-mon[29276]: 5.13 scrub starts 2024-09-11T08:09:21.721 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:21 smithi150 ceph-mon[29276]: 5.13 scrub ok 2024-09-11T08:09:21.721 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:21 smithi150 ceph-mon[29276]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 27 B/s, 1 objects/s recovering 2024-09-11T08:09:21.721 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:21 smithi150 ceph-mon[39612]: 5.13 scrub starts 2024-09-11T08:09:21.721 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:21 smithi150 ceph-mon[39612]: 5.13 scrub ok 2024-09-11T08:09:21.721 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:21 smithi150 ceph-mon[39612]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 27 B/s, 1 objects/s recovering 2024-09-11T08:09:21.734 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:21 smithi181 ceph-mon[32701]: 5.13 scrub starts 2024-09-11T08:09:21.734 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:21 smithi181 ceph-mon[32701]: 5.13 scrub ok 2024-09-11T08:09:21.734 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:21 smithi181 ceph-mon[32701]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 27 B/s, 1 objects/s recovering 2024-09-11T08:09:21.944 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:09:22.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:22 smithi150 ceph-mon[39612]: 6.1b scrub starts 2024-09-11T08:09:22.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:22 smithi150 ceph-mon[39612]: 6.1b scrub ok 2024-09-11T08:09:22.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:22 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:22.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:22 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:22.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:22 smithi150 ceph-mon[39612]: 6.c scrub starts 2024-09-11T08:09:22.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:22 smithi150 ceph-mon[39612]: 6.c scrub ok 2024-09-11T08:09:22.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:22 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:09:22.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:22 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:22.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:22 smithi150 ceph-mon[29276]: 6.1b scrub starts 2024-09-11T08:09:22.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:22 smithi150 ceph-mon[29276]: 6.1b scrub ok 2024-09-11T08:09:22.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:22 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:22.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:22 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:22.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:22 smithi150 ceph-mon[29276]: 6.c scrub starts 2024-09-11T08:09:22.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:22 smithi150 ceph-mon[29276]: 6.c scrub ok 2024-09-11T08:09:22.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:22 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:09:22.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:22 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:22.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:22 smithi181 ceph-mon[32701]: 6.1b scrub starts 2024-09-11T08:09:22.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:22 smithi181 ceph-mon[32701]: 6.1b scrub ok 2024-09-11T08:09:22.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:22 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:22.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:22 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:22.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:22 smithi181 ceph-mon[32701]: 6.c scrub starts 2024-09-11T08:09:22.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:22 smithi181 ceph-mon[32701]: 6.c scrub ok 2024-09-11T08:09:22.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:22 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:09:22.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:22 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:23.678 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:23 smithi150 ceph-mon[29276]: 6.2 scrub starts 2024-09-11T08:09:23.678 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:23 smithi150 ceph-mon[29276]: 6.2 scrub ok 2024-09-11T08:09:23.678 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:23 smithi150 ceph-mon[29276]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-09-11T08:09:23.678 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:23 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:23.678 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:23 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:23.678 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:23 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:09:23.678 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:23 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:09:23.678 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:23 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:23.679 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:23 smithi150 ceph-mon[39612]: 6.2 scrub starts 2024-09-11T08:09:23.679 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:23 smithi150 ceph-mon[39612]: 6.2 scrub ok 2024-09-11T08:09:23.679 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:23 smithi150 ceph-mon[39612]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-09-11T08:09:23.679 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:23 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:23.679 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:23 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:23.679 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:23 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:09:23.679 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:23 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:09:23.679 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:23 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:23.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:23 smithi181 ceph-mon[32701]: 6.2 scrub starts 2024-09-11T08:09:23.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:23 smithi181 ceph-mon[32701]: 6.2 scrub ok 2024-09-11T08:09:23.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:23 smithi181 ceph-mon[32701]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-09-11T08:09:23.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:23 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:23.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:23 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:23.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:23 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:09:23.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:23 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:09:23.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:23 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:23.931 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:09:23.933 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-09-11T08:09:24.634 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:24 smithi150 ceph-mon[29276]: 4.b scrub starts 2024-09-11T08:09:24.634 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:24 smithi150 ceph-mon[29276]: 4.b scrub ok 2024-09-11T08:09:24.634 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:24 smithi150 ceph-mon[29276]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-11T08:09:24.634 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:24 smithi150 ceph-mon[29276]: Reconfiguring daemon alertmanager.a on smithi150 2024-09-11T08:09:24.634 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:24 smithi150 ceph-mon[29276]: 5.12 scrub starts 2024-09-11T08:09:24.634 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:24 smithi150 ceph-mon[29276]: 5.12 scrub ok 2024-09-11T08:09:24.634 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:24 smithi150 ceph-mon[29276]: from='client.24641 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:09:24.635 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:24 smithi150 ceph-mon[39612]: 4.b scrub starts 2024-09-11T08:09:24.635 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:24 smithi150 ceph-mon[39612]: 4.b scrub ok 2024-09-11T08:09:24.635 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:24 smithi150 ceph-mon[39612]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-11T08:09:24.635 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:24 smithi150 ceph-mon[39612]: Reconfiguring daemon alertmanager.a on smithi150 2024-09-11T08:09:24.635 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:24 smithi150 ceph-mon[39612]: 5.12 scrub starts 2024-09-11T08:09:24.635 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:24 smithi150 ceph-mon[39612]: 5.12 scrub ok 2024-09-11T08:09:24.635 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:24 smithi150 ceph-mon[39612]: from='client.24641 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:09:24.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:24 smithi181 ceph-mon[32701]: 4.b scrub starts 2024-09-11T08:09:24.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:24 smithi181 ceph-mon[32701]: 4.b scrub ok 2024-09-11T08:09:24.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:24 smithi181 ceph-mon[32701]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-11T08:09:24.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:24 smithi181 ceph-mon[32701]: Reconfiguring daemon alertmanager.a on smithi150 2024-09-11T08:09:24.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:24 smithi181 ceph-mon[32701]: 5.12 scrub starts 2024-09-11T08:09:24.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:24 smithi181 ceph-mon[32701]: 5.12 scrub ok 2024-09-11T08:09:24.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:24 smithi181 ceph-mon[32701]: from='client.24641 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:09:25.411 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:09:25 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:09:25] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:09:25.691 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:25 smithi150 ceph-mon[29276]: 4.18 scrub starts 2024-09-11T08:09:25.691 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:25 smithi150 ceph-mon[29276]: 4.18 scrub ok 2024-09-11T08:09:25.691 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:25 smithi150 ceph-mon[29276]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:09:25.691 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:25 smithi150 ceph-mon[39612]: 4.18 scrub starts 2024-09-11T08:09:25.691 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:25 smithi150 ceph-mon[39612]: 4.18 scrub ok 2024-09-11T08:09:25.691 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:25 smithi150 ceph-mon[39612]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:09:25.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:25 smithi181 ceph-mon[32701]: 4.18 scrub starts 2024-09-11T08:09:25.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:25 smithi181 ceph-mon[32701]: 4.18 scrub ok 2024-09-11T08:09:25.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:25 smithi181 ceph-mon[32701]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:09:26.098 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":54,"stamp":"2024-09-11T08:09:22.695991+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596188,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":959,"num_read_kb":816,"num_write":543,"num_write_kb":1671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":590698,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":581,"ondisk_log_size":581,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2385428,"kb_used_data":6612,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":747575788,"statfs":{"total":767960285184,"available":765517606912,"internally_reserved":0,"allocated":6770688,"data_stored":4069199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002549"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.937842+0000","last_change":"2024-09-11T08:08:36.105232+0000","last_active":"2024-09-11T08:09:06.937842+0000","last_peered":"2024-09-11T08:09:06.937842+0000","last_clean":"2024-09-11T08:09:06.937842+0000","last_became_active":"2024-09-11T08:07:30.044976+0000","last_became_peered":"2024-09-11T08:07:30.044976+0000","last_unstale":"2024-09-11T08:09:06.937842+0000","last_undegraded":"2024-09-11T08:09:06.937842+0000","last_fullsized":"2024-09-11T08:09:06.937842+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:36.105175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:36.105175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:56.596929+0000","scrub_duration":0.0016664970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":50,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.871277+0000","last_change":"2024-09-11T08:07:32.048578+0000","last_active":"2024-09-11T08:09:06.871277+0000","last_peered":"2024-09-11T08:09:06.871277+0000","last_clean":"2024-09-11T08:09:06.871277+0000","last_became_active":"2024-09-11T08:07:32.048048+0000","last_became_peered":"2024-09-11T08:07:32.048048+0000","last_unstale":"2024-09-11T08:09:06.871277+0000","last_undegraded":"2024-09-11T08:09:06.871277+0000","last_fullsized":"2024-09-11T08:09:06.871277+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.875109+0000","last_change":"2024-09-11T08:08:38.486126+0000","last_active":"2024-09-11T08:09:06.875109+0000","last_peered":"2024-09-11T08:09:06.875109+0000","last_clean":"2024-09-11T08:09:06.875109+0000","last_became_active":"2024-09-11T08:07:34.058828+0000","last_became_peered":"2024-09-11T08:07:34.058828+0000","last_unstale":"2024-09-11T08:09:06.875109+0000","last_undegraded":"2024-09-11T08:09:06.875109+0000","last_fullsized":"2024-09-11T08:09:06.875109+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:38.486057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:38.486057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:44.914804+0000","scrub_duration":0.002230531,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.871537+0000","last_change":"2024-09-11T08:09:02.470722+0000","last_active":"2024-09-11T08:09:06.871537+0000","last_peered":"2024-09-11T08:09:06.871537+0000","last_clean":"2024-09-11T08:09:06.871537+0000","last_became_active":"2024-09-11T08:07:36.074789+0000","last_became_peered":"2024-09-11T08:07:36.074789+0000","last_unstale":"2024-09-11T08:09:06.871537+0000","last_undegraded":"2024-09-11T08:09:06.871537+0000","last_fullsized":"2024-09-11T08:09:06.871537+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:02.470649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:02.470649+0000","last_clean_scrub_stamp":"2024-09-11T08:09:02.470649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:18.421855+0000","scrub_duration":0.00141182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:20.526512+0000","last_change":"2024-09-11T08:09:20.526483+0000","last_active":"2024-09-11T08:09:20.526512+0000","last_peered":"2024-09-11T08:09:20.526512+0000","last_clean":"2024-09-11T08:09:20.526512+0000","last_became_active":"2024-09-11T08:07:36.633423+0000","last_became_peered":"2024-09-11T08:07:36.633423+0000","last_unstale":"2024-09-11T08:09:20.526512+0000","last_undegraded":"2024-09-11T08:09:20.526512+0000","last_fullsized":"2024-09-11T08:09:20.526512+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:20.526415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:09:20.526415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:37.185298+0000","scrub_duration":0.001350539,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.874288+0000","last_change":"2024-09-11T08:07:30.045845+0000","last_active":"2024-09-11T08:09:06.874288+0000","last_peered":"2024-09-11T08:09:06.874288+0000","last_clean":"2024-09-11T08:09:06.874288+0000","last_became_active":"2024-09-11T08:07:30.045579+0000","last_became_peered":"2024-09-11T08:07:30.045579+0000","last_unstale":"2024-09-11T08:09:06.874288+0000","last_undegraded":"2024-09-11T08:09:06.874288+0000","last_fullsized":"2024-09-11T08:09:06.874288+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":69,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.872152+0000","last_change":"2024-09-11T08:08:56.408326+0000","last_active":"2024-09-11T08:09:06.872152+0000","last_peered":"2024-09-11T08:09:06.872152+0000","last_clean":"2024-09-11T08:09:06.872152+0000","last_became_active":"2024-09-11T08:07:32.045848+0000","last_became_peered":"2024-09-11T08:07:32.045848+0000","last_unstale":"2024-09-11T08:09:06.872152+0000","last_undegraded":"2024-09-11T08:09:06.872152+0000","last_fullsized":"2024-09-11T08:09:06.872152+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-11T08:08:56.408272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:56.408272+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:09.630406+0000","scrub_duration":0.001771557,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":32,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.871201+0000","last_change":"2024-09-11T08:07:34.053483+0000","last_active":"2024-09-11T08:09:06.871201+0000","last_peered":"2024-09-11T08:09:06.871201+0000","last_clean":"2024-09-11T08:09:06.871201+0000","last_became_active":"2024-09-11T08:07:34.052243+0000","last_became_peered":"2024-09-11T08:07:34.052243+0000","last_unstale":"2024-09-11T08:09:06.871201+0000","last_undegraded":"2024-09-11T08:09:06.871201+0000","last_fullsized":"2024-09-11T08:09:06.871201+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.875077+0000","last_change":"2024-09-11T08:08:54.878501+0000","last_active":"2024-09-11T08:09:06.875077+0000","last_peered":"2024-09-11T08:09:06.875077+0000","last_clean":"2024-09-11T08:09:06.875077+0000","last_became_active":"2024-09-11T08:07:30.043950+0000","last_became_peered":"2024-09-11T08:07:30.043950+0000","last_unstale":"2024-09-11T08:09:06.875077+0000","last_undegraded":"2024-09-11T08:09:06.875077+0000","last_fullsized":"2024-09-11T08:09:06.875077+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:54.878409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:54.878409+0000","last_clean_scrub_stamp":"2024-09-11T08:08:54.878409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:07.483340+0000","scrub_duration":0.0015972390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":60,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:07.628550+0000","last_change":"2024-09-11T08:09:07.628460+0000","last_active":"2024-09-11T08:09:07.628550+0000","last_peered":"2024-09-11T08:09:07.628550+0000","last_clean":"2024-09-11T08:09:07.628550+0000","last_became_active":"2024-09-11T08:07:32.044626+0000","last_became_peered":"2024-09-11T08:07:32.044626+0000","last_unstale":"2024-09-11T08:09:07.628550+0000","last_undegraded":"2024-09-11T08:09:07.628550+0000","last_fullsized":"2024-09-11T08:09:07.628550+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:09:07.628369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:09:07.628369+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:10.676317+0000","scrub_duration":0.0023051830000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:11.803410+0000","last_change":"2024-09-11T08:09:11.803380+0000","last_active":"2024-09-11T08:09:11.803410+0000","last_peered":"2024-09-11T08:09:11.803410+0000","last_clean":"2024-09-11T08:09:11.803410+0000","last_became_active":"2024-09-11T08:07:34.058689+0000","last_became_peered":"2024-09-11T08:07:34.058689+0000","last_unstale":"2024-09-11T08:09:11.803410+0000","last_undegraded":"2024-09-11T08:09:11.803410+0000","last_fullsized":"2024-09-11T08:09:11.803410+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:11.803311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:09:11.803311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:35.953085+0000","scrub_duration":0.0015061359999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.937357+0000","last_change":"2024-09-11T08:08:40.071203+0000","last_active":"2024-09-11T08:09:06.937357+0000","last_peered":"2024-09-11T08:09:06.937357+0000","last_clean":"2024-09-11T08:09:06.937357+0000","last_became_active":"2024-09-11T08:07:36.076824+0000","last_became_peered":"2024-09-11T08:07:36.076824+0000","last_unstale":"2024-09-11T08:09:06.937357+0000","last_undegraded":"2024-09-11T08:09:06.937357+0000","last_fullsized":"2024-09-11T08:09:06.937357+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:40.071106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:08:40.071106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:54.333357+0000","scrub_duration":0.0019674499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.874855+0000","last_change":"2024-09-11T08:08:36.825857+0000","last_active":"2024-09-11T08:09:06.874855+0000","last_peered":"2024-09-11T08:09:06.874855+0000","last_clean":"2024-09-11T08:09:06.874855+0000","last_became_active":"2024-09-11T08:07:30.043277+0000","last_became_peered":"2024-09-11T08:07:30.043277+0000","last_unstale":"2024-09-11T08:09:06.874855+0000","last_undegraded":"2024-09-11T08:09:06.874855+0000","last_fullsized":"2024-09-11T08:09:06.874855+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:36.825794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:36.825794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:47.493434+0000","scrub_duration":0.0016805920000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":54,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:08.669016+0000","last_change":"2024-09-11T08:09:08.668983+0000","last_active":"2024-09-11T08:09:08.669016+0000","last_peered":"2024-09-11T08:09:08.669016+0000","last_clean":"2024-09-11T08:09:08.669016+0000","last_became_active":"2024-09-11T08:07:32.048661+0000","last_became_peered":"2024-09-11T08:07:32.048661+0000","last_unstale":"2024-09-11T08:09:08.669016+0000","last_undegraded":"2024-09-11T08:09:08.669016+0000","last_fullsized":"2024-09-11T08:09:08.669016+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-11T08:09:08.668925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:09:08.668925+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:30.120080+0000","scrub_duration":0.001704496,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:08.208415+0000","last_change":"2024-09-11T08:09:08.208383+0000","last_active":"2024-09-11T08:09:08.208415+0000","last_peered":"2024-09-11T08:09:08.208415+0000","last_clean":"2024-09-11T08:09:08.208415+0000","last_became_active":"2024-09-11T08:07:34.060871+0000","last_became_peered":"2024-09-11T08:07:34.060871+0000","last_unstale":"2024-09-11T08:09:08.208415+0000","last_undegraded":"2024-09-11T08:09:08.208415+0000","last_fullsized":"2024-09-11T08:09:08.208415+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:08.208319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:09:08.208319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:22.301114+0000","scrub_duration":0.0015808269999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.874542+0000","last_change":"2024-09-11T08:09:04.782636+0000","last_active":"2024-09-11T08:09:06.874542+0000","last_peered":"2024-09-11T08:09:06.874542+0000","last_clean":"2024-09-11T08:09:06.874542+0000","last_became_active":"2024-09-11T08:07:36.077602+0000","last_became_peered":"2024-09-11T08:07:36.077602+0000","last_unstale":"2024-09-11T08:09:06.874542+0000","last_undegraded":"2024-09-11T08:09:06.874542+0000","last_fullsized":"2024-09-11T08:09:06.874542+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:04.782577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:04.782577+0000","last_clean_scrub_stamp":"2024-09-11T08:09:04.782577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:23.367449+0000","scrub_duration":0.0014982330000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:16.643972+0000","last_change":"2024-09-11T08:09:16.643916+0000","last_active":"2024-09-11T08:09:16.643972+0000","last_peered":"2024-09-11T08:09:16.643972+0000","last_clean":"2024-09-11T08:09:16.643972+0000","last_became_active":"2024-09-11T08:07:36.633338+0000","last_became_peered":"2024-09-11T08:07:36.633338+0000","last_unstale":"2024-09-11T08:09:16.643972+0000","last_undegraded":"2024-09-11T08:09:16.643972+0000","last_fullsized":"2024-09-11T08:09:16.643972+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:16.643842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:09:16.643842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:21.390379+0000","scrub_duration":0.0016132099999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":46,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:11.966762+0000","last_change":"2024-09-11T08:09:11.966738+0000","last_active":"2024-09-11T08:09:11.966762+0000","last_peered":"2024-09-11T08:09:11.966762+0000","last_clean":"2024-09-11T08:09:11.966762+0000","last_became_active":"2024-09-11T08:07:30.045739+0000","last_became_peered":"2024-09-11T08:07:30.045739+0000","last_unstale":"2024-09-11T08:09:11.966762+0000","last_undegraded":"2024-09-11T08:09:11.966762+0000","last_fullsized":"2024-09-11T08:09:11.966762+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:11.966683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:09:11.966683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:32.828034+0000","scrub_duration":0.0015807659999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"66'15","reported_seq":68,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.877186+0000","last_change":"2024-09-11T08:08:56.975395+0000","last_active":"2024-09-11T08:09:06.877186+0000","last_peered":"2024-09-11T08:09:06.877186+0000","last_clean":"2024-09-11T08:09:06.877186+0000","last_became_active":"2024-09-11T08:07:32.046363+0000","last_became_peered":"2024-09-11T08:07:32.046363+0000","last_unstale":"2024-09-11T08:09:06.877186+0000","last_undegraded":"2024-09-11T08:09:06.877186+0000","last_fullsized":"2024-09-11T08:09:06.877186+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-11T08:08:56.975338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:56.975338+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:19.207433+0000","scrub_duration":0.0017787899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.876129+0000","last_change":"2024-09-11T08:09:01.315948+0000","last_active":"2024-09-11T08:09:06.876129+0000","last_peered":"2024-09-11T08:09:06.876129+0000","last_clean":"2024-09-11T08:09:06.876129+0000","last_became_active":"2024-09-11T08:07:34.055410+0000","last_became_peered":"2024-09-11T08:07:34.055410+0000","last_unstale":"2024-09-11T08:09:06.876129+0000","last_undegraded":"2024-09-11T08:09:06.876129+0000","last_fullsized":"2024-09-11T08:09:06.876129+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:01.315897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:09:01.315897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:14.655816+0000","scrub_duration":0.001561163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.871890+0000","last_change":"2024-09-11T08:07:36.633626+0000","last_active":"2024-09-11T08:09:06.871890+0000","last_peered":"2024-09-11T08:09:06.871890+0000","last_clean":"2024-09-11T08:09:06.871890+0000","last_became_active":"2024-09-11T08:07:36.633258+0000","last_became_peered":"2024-09-11T08:07:36.633258+0000","last_unstale":"2024-09-11T08:09:06.871890+0000","last_undegraded":"2024-09-11T08:09:06.871890+0000","last_fullsized":"2024-09-11T08:09:06.871890+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.872735+0000","last_change":"2024-09-11T08:08:51.401929+0000","last_active":"2024-09-11T08:09:06.872735+0000","last_peered":"2024-09-11T08:09:06.872735+0000","last_clean":"2024-09-11T08:09:06.872735+0000","last_became_active":"2024-09-11T08:07:30.046284+0000","last_became_peered":"2024-09-11T08:07:30.046284+0000","last_unstale":"2024-09-11T08:09:06.872735+0000","last_undegraded":"2024-09-11T08:09:06.872735+0000","last_fullsized":"2024-09-11T08:09:06.872735+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:51.401864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:51.401864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:18.688155+0000","scrub_duration":0.001460034,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":89,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:09.426269+0000","last_change":"2024-09-11T08:09:09.426244+0000","last_active":"2024-09-11T08:09:09.426269+0000","last_peered":"2024-09-11T08:09:09.426269+0000","last_clean":"2024-09-11T08:09:09.426269+0000","last_became_active":"2024-09-11T08:09:07.878432+0000","last_became_peered":"2024-09-11T08:09:07.878432+0000","last_unstale":"2024-09-11T08:09:09.426269+0000","last_undegraded":"2024-09-11T08:09:09.426269+0000","last_fullsized":"2024-09-11T08:09:09.426269+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-11T08:09:09.426191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:09:09.426191+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:25.720395+0000","scrub_duration":0.0016259,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.871365+0000","last_change":"2024-09-11T08:08:40.419900+0000","last_active":"2024-09-11T08:09:06.871365+0000","last_peered":"2024-09-11T08:09:06.871365+0000","last_clean":"2024-09-11T08:09:06.871365+0000","last_became_active":"2024-09-11T08:07:34.056528+0000","last_became_peered":"2024-09-11T08:07:34.056528+0000","last_unstale":"2024-09-11T08:09:06.871365+0000","last_undegraded":"2024-09-11T08:09:06.871365+0000","last_fullsized":"2024-09-11T08:09:06.871365+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:40.419830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:40.419830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:09.111095+0000","scrub_duration":0.0029308149999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:12.217943+0000","last_change":"2024-09-11T08:09:12.217904+0000","last_active":"2024-09-11T08:09:12.217943+0000","last_peered":"2024-09-11T08:09:12.217943+0000","last_clean":"2024-09-11T08:09:12.217943+0000","last_became_active":"2024-09-11T08:07:36.072347+0000","last_became_peered":"2024-09-11T08:07:36.072347+0000","last_unstale":"2024-09-11T08:09:12.217943+0000","last_undegraded":"2024-09-11T08:09:12.217943+0000","last_fullsized":"2024-09-11T08:09:12.217943+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-11T08:09:12.217832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:09:12.217832+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:17.655732+0000","scrub_duration":0.0020299329999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":46,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:17.461676+0000","last_change":"2024-09-11T08:09:17.461649+0000","last_active":"2024-09-11T08:09:17.461676+0000","last_peered":"2024-09-11T08:09:17.461676+0000","last_clean":"2024-09-11T08:09:17.461676+0000","last_became_active":"2024-09-11T08:07:30.045932+0000","last_became_peered":"2024-09-11T08:07:30.045932+0000","last_unstale":"2024-09-11T08:09:17.461676+0000","last_undegraded":"2024-09-11T08:09:17.461676+0000","last_fullsized":"2024-09-11T08:09:17.461676+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:17.461591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:09:17.461591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:27.173867+0000","scrub_duration":0.0018513749999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":59,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:12.939270+0000","last_change":"2024-09-11T08:09:12.939246+0000","last_active":"2024-09-11T08:09:12.939270+0000","last_peered":"2024-09-11T08:09:12.939270+0000","last_clean":"2024-09-11T08:09:12.939270+0000","last_became_active":"2024-09-11T08:07:32.048022+0000","last_became_peered":"2024-09-11T08:07:32.048022+0000","last_unstale":"2024-09-11T08:09:12.939270+0000","last_undegraded":"2024-09-11T08:09:12.939270+0000","last_fullsized":"2024-09-11T08:09:12.939270+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-11T08:09:12.939190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:09:12.939190+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:15.289227+0000","scrub_duration":0.0020690029999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":74,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:13.562552+0000","last_change":"2024-09-11T08:08:54.652958+0000","last_active":"2024-09-11T08:09:13.562552+0000","last_peered":"2024-09-11T08:09:13.562552+0000","last_clean":"2024-09-11T08:09:13.562552+0000","last_became_active":"2024-09-11T08:07:34.054014+0000","last_became_peered":"2024-09-11T08:07:34.054014+0000","last_unstale":"2024-09-11T08:09:13.562552+0000","last_undegraded":"2024-09-11T08:09:13.562552+0000","last_fullsized":"2024-09-11T08:09:13.562552+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:08:54.652867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:54.652867+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:09.166003+0000","scrub_duration":0.0022231320000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":69,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.875705+0000","last_change":"2024-09-11T08:08:45.358105+0000","last_active":"2024-09-11T08:09:06.875705+0000","last_peered":"2024-09-11T08:09:06.875705+0000","last_clean":"2024-09-11T08:09:06.875705+0000","last_became_active":"2024-09-11T08:07:32.054317+0000","last_became_peered":"2024-09-11T08:07:32.054317+0000","last_unstale":"2024-09-11T08:09:06.875705+0000","last_undegraded":"2024-09-11T08:09:06.875705+0000","last_fullsized":"2024-09-11T08:09:06.875705+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-11T08:08:45.358041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:45.358041+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:50.858524+0000","scrub_duration":0.0021022879999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.871677+0000","last_change":"2024-09-11T08:07:30.044933+0000","last_active":"2024-09-11T08:09:06.871677+0000","last_peered":"2024-09-11T08:09:06.871677+0000","last_clean":"2024-09-11T08:09:06.871677+0000","last_became_active":"2024-09-11T08:07:30.044624+0000","last_became_peered":"2024-09-11T08:07:30.044624+0000","last_unstale":"2024-09-11T08:09:06.871677+0000","last_undegraded":"2024-09-11T08:09:06.871677+0000","last_fullsized":"2024-09-11T08:09:06.871677+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":72,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:13.562537+0000","last_change":"2024-09-11T08:09:10.668081+0000","last_active":"2024-09-11T08:09:13.562537+0000","last_peered":"2024-09-11T08:09:13.562537+0000","last_clean":"2024-09-11T08:09:13.562537+0000","last_became_active":"2024-09-11T08:07:34.052981+0000","last_became_peered":"2024-09-11T08:07:34.052981+0000","last_unstale":"2024-09-11T08:09:13.562537+0000","last_undegraded":"2024-09-11T08:09:13.562537+0000","last_fullsized":"2024-09-11T08:09:13.562537+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:09:10.668024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:09:10.668024+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:19.607766+0000","scrub_duration":0.0015739580000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.874836+0000","last_change":"2024-09-11T08:08:51.893846+0000","last_active":"2024-09-11T08:09:06.874836+0000","last_peered":"2024-09-11T08:09:06.874836+0000","last_clean":"2024-09-11T08:09:06.874836+0000","last_became_active":"2024-09-11T08:07:36.073969+0000","last_became_peered":"2024-09-11T08:07:36.073969+0000","last_unstale":"2024-09-11T08:09:06.874836+0000","last_undegraded":"2024-09-11T08:09:06.874836+0000","last_fullsized":"2024-09-11T08:09:06.874836+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:51.893774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:08:51.893774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:00.876260+0000","scrub_duration":0.001532858,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":76,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.871575+0000","last_change":"2024-09-11T08:08:43.441600+0000","last_active":"2024-09-11T08:09:06.871575+0000","last_peered":"2024-09-11T08:09:06.871575+0000","last_clean":"2024-09-11T08:09:06.871575+0000","last_became_active":"2024-09-11T08:07:32.045332+0000","last_became_peered":"2024-09-11T08:07:32.045332+0000","last_unstale":"2024-09-11T08:09:06.871575+0000","last_undegraded":"2024-09-11T08:09:06.871575+0000","last_fullsized":"2024-09-11T08:09:06.871575+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-11T08:08:43.441544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:43.441544+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:55.649394+0000","scrub_duration":0.002168648,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.872253+0000","last_change":"2024-09-11T08:07:30.047446+0000","last_active":"2024-09-11T08:09:06.872253+0000","last_peered":"2024-09-11T08:09:06.872253+0000","last_clean":"2024-09-11T08:09:06.872253+0000","last_became_active":"2024-09-11T08:07:30.047176+0000","last_became_peered":"2024-09-11T08:07:30.047176+0000","last_unstale":"2024-09-11T08:09:06.872253+0000","last_undegraded":"2024-09-11T08:09:06.872253+0000","last_fullsized":"2024-09-11T08:09:06.872253+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:17.611511+0000","last_change":"2024-09-11T08:09:17.611439+0000","last_active":"2024-09-11T08:09:17.611511+0000","last_peered":"2024-09-11T08:09:17.611511+0000","last_clean":"2024-09-11T08:09:17.611511+0000","last_became_active":"2024-09-11T08:07:34.055523+0000","last_became_peered":"2024-09-11T08:07:34.055523+0000","last_unstale":"2024-09-11T08:09:17.611511+0000","last_undegraded":"2024-09-11T08:09:17.611511+0000","last_fullsized":"2024-09-11T08:09:17.611511+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:17.611370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:09:17.611370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:19.065362+0000","scrub_duration":0.001678803,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:21.618853+0000","last_change":"2024-09-11T08:09:21.618824+0000","last_active":"2024-09-11T08:09:21.618853+0000","last_peered":"2024-09-11T08:09:21.618853+0000","last_clean":"2024-09-11T08:09:21.618853+0000","last_became_active":"2024-09-11T08:07:36.069225+0000","last_became_peered":"2024-09-11T08:07:36.069225+0000","last_unstale":"2024-09-11T08:09:21.618853+0000","last_undegraded":"2024-09-11T08:09:21.618853+0000","last_fullsized":"2024-09-11T08:09:21.618853+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:21.618769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:09:21.618769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:36.342241+0000","scrub_duration":0.001637814,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":65,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:11.688244+0000","last_change":"2024-09-11T08:09:11.688215+0000","last_active":"2024-09-11T08:09:11.688244+0000","last_peered":"2024-09-11T08:09:11.688244+0000","last_clean":"2024-09-11T08:09:11.688244+0000","last_became_active":"2024-09-11T08:07:32.040069+0000","last_became_peered":"2024-09-11T08:07:32.040069+0000","last_unstale":"2024-09-11T08:09:11.688244+0000","last_undegraded":"2024-09-11T08:09:11.688244+0000","last_fullsized":"2024-09-11T08:09:11.688244+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-11T08:09:11.688151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:09:11.688151+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:27.200867+0000","scrub_duration":0.0019655670000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.960732+0000","last_change":"2024-09-11T08:09:06.960696+0000","last_active":"2024-09-11T08:09:06.960732+0000","last_peered":"2024-09-11T08:09:06.960732+0000","last_clean":"2024-09-11T08:09:06.960732+0000","last_became_active":"2024-09-11T08:07:30.050484+0000","last_became_peered":"2024-09-11T08:07:30.050484+0000","last_unstale":"2024-09-11T08:09:06.960732+0000","last_undegraded":"2024-09-11T08:09:06.960732+0000","last_fullsized":"2024-09-11T08:09:06.960732+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:06.960639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:09:06.960639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:19.133864+0000","scrub_duration":0.001342766,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"82'9","reported_seq":76,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:13.562286+0000","last_change":"2024-09-11T08:08:53.421634+0000","last_active":"2024-09-11T08:09:13.562286+0000","last_peered":"2024-09-11T08:09:13.562286+0000","last_clean":"2024-09-11T08:09:13.562286+0000","last_became_active":"2024-09-11T08:07:34.056143+0000","last_became_peered":"2024-09-11T08:07:34.056143+0000","last_unstale":"2024-09-11T08:09:13.562286+0000","last_undegraded":"2024-09-11T08:09:13.562286+0000","last_fullsized":"2024-09-11T08:09:13.562286+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-11T08:08:53.421587+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-09-11T08:08:53.421587+0000","last_clean_scrub_stamp":"2024-09-11T08:08:53.421587+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:11.501044+0000","scrub_duration":0.0021116799999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.870847+0000","last_change":"2024-09-11T08:08:55.249190+0000","last_active":"2024-09-11T08:09:06.870847+0000","last_peered":"2024-09-11T08:09:06.870847+0000","last_clean":"2024-09-11T08:09:06.870847+0000","last_became_active":"2024-09-11T08:07:36.633432+0000","last_became_peered":"2024-09-11T08:07:36.633432+0000","last_unstale":"2024-09-11T08:09:06.870847+0000","last_undegraded":"2024-09-11T08:09:06.870847+0000","last_fullsized":"2024-09-11T08:09:06.870847+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:55.249081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:08:55.249081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:00.965734+0000","scrub_duration":0.001581839,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":59,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:18.424583+0000","last_change":"2024-09-11T08:09:18.424558+0000","last_active":"2024-09-11T08:09:18.424583+0000","last_peered":"2024-09-11T08:09:18.424583+0000","last_clean":"2024-09-11T08:09:18.424583+0000","last_became_active":"2024-09-11T08:07:32.054294+0000","last_became_peered":"2024-09-11T08:07:32.054294+0000","last_unstale":"2024-09-11T08:09:18.424583+0000","last_undegraded":"2024-09-11T08:09:18.424583+0000","last_fullsized":"2024-09-11T08:09:18.424583+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-11T08:09:18.424501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:09:18.424501+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:38.421434+0000","scrub_duration":0.0019538310000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.875129+0000","last_change":"2024-09-11T08:09:00.819941+0000","last_active":"2024-09-11T08:09:06.875129+0000","last_peered":"2024-09-11T08:09:06.875129+0000","last_clean":"2024-09-11T08:09:06.875129+0000","last_became_active":"2024-09-11T08:07:30.047497+0000","last_became_peered":"2024-09-11T08:07:30.047497+0000","last_unstale":"2024-09-11T08:09:06.875129+0000","last_undegraded":"2024-09-11T08:09:06.875129+0000","last_fullsized":"2024-09-11T08:09:06.875129+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:00.819884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:09:00.819884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:12.227536+0000","scrub_duration":0.0014673640000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":72,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:15.936837+0000","last_change":"2024-09-11T08:09:15.936774+0000","last_active":"2024-09-11T08:09:15.936837+0000","last_peered":"2024-09-11T08:09:15.936837+0000","last_clean":"2024-09-11T08:09:15.936837+0000","last_became_active":"2024-09-11T08:07:34.060660+0000","last_became_peered":"2024-09-11T08:07:34.060660+0000","last_unstale":"2024-09-11T08:09:15.936837+0000","last_undegraded":"2024-09-11T08:09:15.936837+0000","last_fullsized":"2024-09-11T08:09:15.936837+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:09:15.936712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:09:15.936712+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:30.355035+0000","scrub_duration":0.0016362620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"66'1","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.937427+0000","last_change":"2024-09-11T08:07:36.072890+0000","last_active":"2024-09-11T08:09:06.937427+0000","last_peered":"2024-09-11T08:09:06.937427+0000","last_clean":"2024-09-11T08:09:06.937427+0000","last_became_active":"2024-09-11T08:07:36.072593+0000","last_became_peered":"2024-09-11T08:07:36.072593+0000","last_unstale":"2024-09-11T08:09:06.937427+0000","last_undegraded":"2024-09-11T08:09:06.937427+0000","last_fullsized":"2024-09-11T08:09:06.937427+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":80,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.937777+0000","last_change":"2024-09-11T08:08:52.842005+0000","last_active":"2024-09-11T08:09:06.937777+0000","last_peered":"2024-09-11T08:09:06.937777+0000","last_clean":"2024-09-11T08:09:06.937777+0000","last_became_active":"2024-09-11T08:07:32.044376+0000","last_became_peered":"2024-09-11T08:07:32.044376+0000","last_unstale":"2024-09-11T08:09:06.937777+0000","last_undegraded":"2024-09-11T08:09:06.937777+0000","last_fullsized":"2024-09-11T08:09:06.937777+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-11T08:08:52.841903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:52.841903+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:08.737667+0000","scrub_duration":0.0025248150000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.877070+0000","last_change":"2024-09-11T08:08:49.384423+0000","last_active":"2024-09-11T08:09:06.877070+0000","last_peered":"2024-09-11T08:09:06.877070+0000","last_clean":"2024-09-11T08:09:06.877070+0000","last_became_active":"2024-09-11T08:07:30.045380+0000","last_became_peered":"2024-09-11T08:07:30.045380+0000","last_unstale":"2024-09-11T08:09:06.877070+0000","last_undegraded":"2024-09-11T08:09:06.877070+0000","last_fullsized":"2024-09-11T08:09:06.877070+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:49.384357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:49.384357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:50.840709+0000","scrub_duration":0.0012073310000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:12.819936+0000","last_change":"2024-09-11T08:09:12.819864+0000","last_active":"2024-09-11T08:09:12.819936+0000","last_peered":"2024-09-11T08:09:12.819936+0000","last_clean":"2024-09-11T08:09:12.819936+0000","last_became_active":"2024-09-11T08:07:34.051004+0000","last_became_peered":"2024-09-11T08:07:34.051004+0000","last_unstale":"2024-09-11T08:09:12.819936+0000","last_undegraded":"2024-09-11T08:09:12.819936+0000","last_fullsized":"2024-09-11T08:09:12.819936+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:12.819778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:09:12.819778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:22.930277+0000","scrub_duration":0.001593955,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.875037+0000","last_change":"2024-09-11T08:07:36.632119+0000","last_active":"2024-09-11T08:09:06.875037+0000","last_peered":"2024-09-11T08:09:06.875037+0000","last_clean":"2024-09-11T08:09:06.875037+0000","last_became_active":"2024-09-11T08:07:36.631668+0000","last_became_peered":"2024-09-11T08:07:36.631668+0000","last_unstale":"2024-09-11T08:09:06.875037+0000","last_undegraded":"2024-09-11T08:09:06.875037+0000","last_fullsized":"2024-09-11T08:09:06.875037+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"86'38","reported_seq":124,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:13.562304+0000","last_change":"2024-09-11T08:09:11.362407+0000","last_active":"2024-09-11T08:09:13.562304+0000","last_peered":"2024-09-11T08:09:13.562304+0000","last_clean":"2024-09-11T08:09:13.562304+0000","last_became_active":"2024-09-11T08:07:32.052956+0000","last_became_peered":"2024-09-11T08:07:32.052956+0000","last_unstale":"2024-09-11T08:09:13.562304+0000","last_undegraded":"2024-09-11T08:09:13.562304+0000","last_fullsized":"2024-09-11T08:09:13.562304+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'38","last_scrub_stamp":"2024-09-11T08:09:11.362339+0000","last_deep_scrub":"86'38","last_deep_scrub_stamp":"2024-09-11T08:09:11.362339+0000","last_clean_scrub_stamp":"2024-09-11T08:09:11.362339+0000","objects_scrubbed":10,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:36.998367+0000","scrub_duration":0.0075376050000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"59'1","reported_seq":55,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.872582+0000","last_change":"2024-09-11T08:08:34.423550+0000","last_active":"2024-09-11T08:09:06.872582+0000","last_peered":"2024-09-11T08:09:06.872582+0000","last_clean":"2024-09-11T08:09:06.872582+0000","last_became_active":"2024-09-11T08:07:30.047513+0000","last_became_peered":"2024-09-11T08:07:30.047513+0000","last_unstale":"2024-09-11T08:09:06.872582+0000","last_undegraded":"2024-09-11T08:09:06.872582+0000","last_fullsized":"2024-09-11T08:09:06.872582+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-11T08:08:34.423474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:34.423474+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:57.535838+0000","scrub_duration":0.0016749149999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":52,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.874774+0000","last_change":"2024-09-11T08:08:37.859401+0000","last_active":"2024-09-11T08:09:06.874774+0000","last_peered":"2024-09-11T08:09:06.874774+0000","last_clean":"2024-09-11T08:09:06.874774+0000","last_became_active":"2024-09-11T08:07:30.043130+0000","last_became_peered":"2024-09-11T08:07:30.043130+0000","last_unstale":"2024-09-11T08:09:06.874774+0000","last_undegraded":"2024-09-11T08:09:06.874774+0000","last_fullsized":"2024-09-11T08:09:06.874774+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-11T08:08:37.859328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:37.859328+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:48.485842+0000","scrub_duration":0.001679816,"objects_trimmed":0,"snaptrim_duration":0.0026120850000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.937675+0000","last_change":"2024-09-11T08:08:46.959561+0000","last_active":"2024-09-11T08:09:06.937675+0000","last_peered":"2024-09-11T08:09:06.937675+0000","last_clean":"2024-09-11T08:09:06.937675+0000","last_became_active":"2024-09-11T08:07:34.057154+0000","last_became_peered":"2024-09-11T08:07:34.057154+0000","last_unstale":"2024-09-11T08:09:06.937675+0000","last_undegraded":"2024-09-11T08:09:06.937675+0000","last_fullsized":"2024-09-11T08:09:06.937675+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:46.959500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:46.959500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:05.161397+0000","scrub_duration":0.001608806,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.874109+0000","last_change":"2024-09-11T08:07:36.076211+0000","last_active":"2024-09-11T08:09:06.874109+0000","last_peered":"2024-09-11T08:09:06.874109+0000","last_clean":"2024-09-11T08:09:06.874109+0000","last_became_active":"2024-09-11T08:07:36.075690+0000","last_became_peered":"2024-09-11T08:07:36.075690+0000","last_unstale":"2024-09-11T08:09:06.874109+0000","last_undegraded":"2024-09-11T08:09:06.874109+0000","last_fullsized":"2024-09-11T08:09:06.874109+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":71,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.875789+0000","last_change":"2024-09-11T08:08:35.441946+0000","last_active":"2024-09-11T08:09:06.875789+0000","last_peered":"2024-09-11T08:09:06.875789+0000","last_clean":"2024-09-11T08:09:06.875789+0000","last_became_active":"2024-09-11T08:07:32.050673+0000","last_became_peered":"2024-09-11T08:07:32.050673+0000","last_unstale":"2024-09-11T08:09:06.875789+0000","last_undegraded":"2024-09-11T08:09:06.875789+0000","last_fullsized":"2024-09-11T08:09:06.875789+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'13","last_scrub_stamp":"2024-09-11T08:08:35.441894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:35.441894+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:02.914423+0000","scrub_duration":0.001888965,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.876600+0000","last_change":"2024-09-11T08:08:33.492709+0000","last_active":"2024-09-11T08:09:06.876600+0000","last_peered":"2024-09-11T08:09:06.876600+0000","last_clean":"2024-09-11T08:09:06.876600+0000","last_became_active":"2024-09-11T08:07:30.047014+0000","last_became_peered":"2024-09-11T08:07:30.047014+0000","last_unstale":"2024-09-11T08:09:06.876600+0000","last_undegraded":"2024-09-11T08:09:06.876600+0000","last_fullsized":"2024-09-11T08:09:06.876600+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:33.492652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:33.492652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:54.786224+0000","scrub_duration":0.00156573,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.877615+0000","last_change":"2024-09-11T08:08:32.909292+0000","last_active":"2024-09-11T08:09:06.877615+0000","last_peered":"2024-09-11T08:09:06.877615+0000","last_clean":"2024-09-11T08:09:06.877615+0000","last_became_active":"2024-09-11T08:07:30.045009+0000","last_became_peered":"2024-09-11T08:07:30.045009+0000","last_unstale":"2024-09-11T08:09:06.877615+0000","last_undegraded":"2024-09-11T08:09:06.877615+0000","last_fullsized":"2024-09-11T08:09:06.877615+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-11T08:08:32.909183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:32.909183+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:02.458449+0000","scrub_duration":0.001625769,"objects_trimmed":0,"snaptrim_duration":0.001859533,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.876433+0000","last_change":"2024-09-11T08:08:58.000006+0000","last_active":"2024-09-11T08:09:06.876433+0000","last_peered":"2024-09-11T08:09:06.876433+0000","last_clean":"2024-09-11T08:09:06.876433+0000","last_became_active":"2024-09-11T08:07:34.063906+0000","last_became_peered":"2024-09-11T08:07:34.063906+0000","last_unstale":"2024-09-11T08:09:06.876433+0000","last_undegraded":"2024-09-11T08:09:06.876433+0000","last_fullsized":"2024-09-11T08:09:06.876433+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:57.999952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:57.999952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:01.053836+0000","scrub_duration":0.0014942130000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.871283+0000","last_change":"2024-09-11T08:08:46.366578+0000","last_active":"2024-09-11T08:09:06.871283+0000","last_peered":"2024-09-11T08:09:06.871283+0000","last_clean":"2024-09-11T08:09:06.871283+0000","last_became_active":"2024-09-11T08:07:36.631480+0000","last_became_peered":"2024-09-11T08:07:36.631480+0000","last_unstale":"2024-09-11T08:09:06.871283+0000","last_undegraded":"2024-09-11T08:09:06.871283+0000","last_fullsized":"2024-09-11T08:09:06.871283+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:46.366508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:08:46.366508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:52.248237+0000","scrub_duration":0.0015317200000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":62,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.937761+0000","last_change":"2024-09-11T08:08:37.115649+0000","last_active":"2024-09-11T08:09:06.937761+0000","last_peered":"2024-09-11T08:09:06.937761+0000","last_clean":"2024-09-11T08:09:06.937761+0000","last_became_active":"2024-09-11T08:07:32.055407+0000","last_became_peered":"2024-09-11T08:07:32.055407+0000","last_unstale":"2024-09-11T08:09:06.937761+0000","last_undegraded":"2024-09-11T08:09:06.937761+0000","last_fullsized":"2024-09-11T08:09:06.937761+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-11T08:08:37.115582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:37.115582+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:57.869478+0000","scrub_duration":0.0020267729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"59'1","reported_seq":53,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.937503+0000","last_change":"2024-09-11T08:08:41.094879+0000","last_active":"2024-09-11T08:09:06.937503+0000","last_peered":"2024-09-11T08:09:06.937503+0000","last_clean":"2024-09-11T08:09:06.937503+0000","last_became_active":"2024-09-11T08:07:30.050504+0000","last_became_peered":"2024-09-11T08:07:30.050504+0000","last_unstale":"2024-09-11T08:09:06.937503+0000","last_undegraded":"2024-09-11T08:09:06.937503+0000","last_fullsized":"2024-09-11T08:09:06.937503+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-11T08:08:41.094821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:41.094821+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:00.561470+0000","scrub_duration":0.0017674310000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":160,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:21.465853+0000","last_change":"2024-09-11T08:09:01.175592+0000","last_active":"2024-09-11T08:09:21.465853+0000","last_peered":"2024-09-11T08:09:21.465853+0000","last_clean":"2024-09-11T08:09:21.465853+0000","last_became_active":"2024-09-11T08:07:30.045051+0000","last_became_peered":"2024-09-11T08:07:30.045051+0000","last_unstale":"2024-09-11T08:09:21.465853+0000","last_undegraded":"2024-09-11T08:09:21.465853+0000","last_fullsized":"2024-09-11T08:09:21.465853+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-11T08:09:01.175530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:09:01.175530+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:30.186314+0000","scrub_duration":0.001752651,"objects_trimmed":0,"snaptrim_duration":0.000318821,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":106,"num_read_kb":101,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.875254+0000","last_change":"2024-09-11T08:09:05.758196+0000","last_active":"2024-09-11T08:09:06.875254+0000","last_peered":"2024-09-11T08:09:06.875254+0000","last_clean":"2024-09-11T08:09:06.875254+0000","last_became_active":"2024-09-11T08:07:34.058962+0000","last_became_peered":"2024-09-11T08:07:34.058962+0000","last_unstale":"2024-09-11T08:09:06.875254+0000","last_undegraded":"2024-09-11T08:09:06.875254+0000","last_fullsized":"2024-09-11T08:09:06.875254+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:05.758141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:09:05.758141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:09.559173+0000","scrub_duration":0.0012345450000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.875199+0000","last_change":"2024-09-11T08:08:50.346126+0000","last_active":"2024-09-11T08:09:06.875199+0000","last_peered":"2024-09-11T08:09:06.875199+0000","last_clean":"2024-09-11T08:09:06.875199+0000","last_became_active":"2024-09-11T08:07:36.077597+0000","last_became_peered":"2024-09-11T08:07:36.077597+0000","last_unstale":"2024-09-11T08:09:06.875199+0000","last_undegraded":"2024-09-11T08:09:06.875199+0000","last_fullsized":"2024-09-11T08:09:06.875199+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:50.346069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:08:50.346069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:15.900195+0000","scrub_duration":0.001294906,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"84'21","reported_seq":72,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.938014+0000","last_change":"2024-09-11T08:08:56.705362+0000","last_active":"2024-09-11T08:09:06.938014+0000","last_peered":"2024-09-11T08:09:06.938014+0000","last_clean":"2024-09-11T08:09:06.938014+0000","last_became_active":"2024-09-11T08:07:32.042339+0000","last_became_peered":"2024-09-11T08:07:32.042339+0000","last_unstale":"2024-09-11T08:09:06.938014+0000","last_undegraded":"2024-09-11T08:09:06.938014+0000","last_fullsized":"2024-09-11T08:09:06.938014+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'21","last_scrub_stamp":"2024-09-11T08:08:56.705306+0000","last_deep_scrub":"84'21","last_deep_scrub_stamp":"2024-09-11T08:08:56.705306+0000","last_clean_scrub_stamp":"2024-09-11T08:08:56.705306+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:14.758337+0000","scrub_duration":0.0050944290000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.874353+0000","last_change":"2024-09-11T08:07:30.047535+0000","last_active":"2024-09-11T08:09:06.874353+0000","last_peered":"2024-09-11T08:09:06.874353+0000","last_clean":"2024-09-11T08:09:06.874353+0000","last_became_active":"2024-09-11T08:07:30.047246+0000","last_became_peered":"2024-09-11T08:07:30.047246+0000","last_unstale":"2024-09-11T08:09:06.874353+0000","last_undegraded":"2024-09-11T08:09:06.874353+0000","last_fullsized":"2024-09-11T08:09:06.874353+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:07:29.021655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":92,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:15.273857+0000","last_change":"2024-09-11T08:09:15.273825+0000","last_active":"2024-09-11T08:09:15.273857+0000","last_peered":"2024-09-11T08:09:15.273857+0000","last_clean":"2024-09-11T08:09:15.273857+0000","last_became_active":"2024-09-11T08:07:19.726378+0000","last_became_peered":"2024-09-11T08:07:19.726378+0000","last_unstale":"2024-09-11T08:09:15.273857+0000","last_undegraded":"2024-09-11T08:09:15.273857+0000","last_fullsized":"2024-09-11T08:09:15.273857+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-11T08:09:15.273758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:05:21.231856+0000","last_clean_scrub_stamp":"2024-09-11T08:09:15.273758+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:20.475275+0000","scrub_duration":0.0021802700000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.871174+0000","last_change":"2024-09-11T08:08:51.289948+0000","last_active":"2024-09-11T08:09:06.871174+0000","last_peered":"2024-09-11T08:09:06.871174+0000","last_clean":"2024-09-11T08:09:06.871174+0000","last_became_active":"2024-09-11T08:07:34.058414+0000","last_became_peered":"2024-09-11T08:07:34.058414+0000","last_unstale":"2024-09-11T08:09:06.871174+0000","last_undegraded":"2024-09-11T08:09:06.871174+0000","last_fullsized":"2024-09-11T08:09:06.871174+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:51.289872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:51.289872+0000","last_clean_scrub_stamp":"2024-09-11T08:08:51.289872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:08.912691+0000","scrub_duration":0.001988812,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.875166+0000","last_change":"2024-09-11T08:09:01.777522+0000","last_active":"2024-09-11T08:09:06.875166+0000","last_peered":"2024-09-11T08:09:06.875166+0000","last_clean":"2024-09-11T08:09:06.875166+0000","last_became_active":"2024-09-11T08:07:36.072508+0000","last_became_peered":"2024-09-11T08:07:36.072508+0000","last_unstale":"2024-09-11T08:09:06.875166+0000","last_undegraded":"2024-09-11T08:09:06.875166+0000","last_fullsized":"2024-09-11T08:09:06.875166+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:01.777429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:01.777429+0000","last_clean_scrub_stamp":"2024-09-11T08:09:01.777429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:07.815841+0000","scrub_duration":0.001444961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"66'11","reported_seq":63,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.872152+0000","last_change":"2024-09-11T08:08:35.418440+0000","last_active":"2024-09-11T08:09:06.872152+0000","last_peered":"2024-09-11T08:09:06.872152+0000","last_clean":"2024-09-11T08:09:06.872152+0000","last_became_active":"2024-09-11T08:07:32.047971+0000","last_became_peered":"2024-09-11T08:07:32.047971+0000","last_unstale":"2024-09-11T08:09:06.872152+0000","last_undegraded":"2024-09-11T08:09:06.872152+0000","last_fullsized":"2024-09-11T08:09:06.872152+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-11T08:08:35.418378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:35.418378+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:38.165860+0000","scrub_duration":0.001791207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.872840+0000","last_change":"2024-09-11T08:08:36.417975+0000","last_active":"2024-09-11T08:09:06.872840+0000","last_peered":"2024-09-11T08:09:06.872840+0000","last_clean":"2024-09-11T08:09:06.872840+0000","last_became_active":"2024-09-11T08:07:30.039884+0000","last_became_peered":"2024-09-11T08:07:30.039884+0000","last_unstale":"2024-09-11T08:09:06.872840+0000","last_undegraded":"2024-09-11T08:09:06.872840+0000","last_fullsized":"2024-09-11T08:09:06.872840+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:36.417911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:36.417911+0000","last_clean_scrub_stamp":"2024-09-11T08:08:36.417911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:59.734565+0000","scrub_duration":0.0014387580000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.875075+0000","last_change":"2024-09-11T08:08:58.845142+0000","last_active":"2024-09-11T08:09:06.875075+0000","last_peered":"2024-09-11T08:09:06.875075+0000","last_clean":"2024-09-11T08:09:06.875075+0000","last_became_active":"2024-09-11T08:07:34.053347+0000","last_became_peered":"2024-09-11T08:07:34.053347+0000","last_unstale":"2024-09-11T08:09:06.875075+0000","last_undegraded":"2024-09-11T08:09:06.875075+0000","last_fullsized":"2024-09-11T08:09:06.875075+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:58.845085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:58.845085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:08.089077+0000","scrub_duration":0.001233404,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:21.498346+0000","last_change":"2024-09-11T08:09:21.498306+0000","last_active":"2024-09-11T08:09:21.498346+0000","last_peered":"2024-09-11T08:09:21.498346+0000","last_clean":"2024-09-11T08:09:21.498346+0000","last_became_active":"2024-09-11T08:07:36.631687+0000","last_became_peered":"2024-09-11T08:07:36.631687+0000","last_unstale":"2024-09-11T08:09:21.498346+0000","last_undegraded":"2024-09-11T08:09:21.498346+0000","last_fullsized":"2024-09-11T08:09:21.498346+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:21.498248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:09:21.498248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:43.565905+0000","scrub_duration":0.001184149,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"77'25","reported_seq":108,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:08.115802+0000","last_change":"2024-09-11T08:09:08.115802+0000","last_active":"2024-09-11T08:09:08.115802+0000","last_peered":"2024-09-11T08:09:08.115802+0000","last_clean":"2024-09-11T08:09:08.115802+0000","last_became_active":"2024-09-11T08:09:08.106771+0000","last_became_peered":"2024-09-11T08:09:08.106771+0000","last_unstale":"2024-09-11T08:09:08.115802+0000","last_undegraded":"2024-09-11T08:09:08.115802+0000","last_fullsized":"2024-09-11T08:09:08.115802+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'25","last_scrub_stamp":"2024-09-11T08:08:32.458210+0000","last_deep_scrub":"77'25","last_deep_scrub_stamp":"2024-09-11T08:08:32.458210+0000","last_clean_scrub_stamp":"2024-09-11T08:08:32.458210+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:00.312847+0000","scrub_duration":0.0060523319999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":45,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:15.752153+0000","last_change":"2024-09-11T08:09:15.752118+0000","last_active":"2024-09-11T08:09:15.752153+0000","last_peered":"2024-09-11T08:09:15.752153+0000","last_clean":"2024-09-11T08:09:15.752153+0000","last_became_active":"2024-09-11T08:07:30.042555+0000","last_became_peered":"2024-09-11T08:07:30.042555+0000","last_unstale":"2024-09-11T08:09:15.752153+0000","last_undegraded":"2024-09-11T08:09:15.752153+0000","last_fullsized":"2024-09-11T08:09:15.752153+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:15.752032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:09:15.752032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:45.186609+0000","scrub_duration":0.0014555519999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.875266+0000","last_change":"2024-09-11T08:08:41.437100+0000","last_active":"2024-09-11T08:09:06.875266+0000","last_peered":"2024-09-11T08:09:06.875266+0000","last_clean":"2024-09-11T08:09:06.875266+0000","last_became_active":"2024-09-11T08:07:34.054961+0000","last_became_peered":"2024-09-11T08:07:34.054961+0000","last_unstale":"2024-09-11T08:09:06.875266+0000","last_undegraded":"2024-09-11T08:09:06.875266+0000","last_fullsized":"2024-09-11T08:09:06.875266+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:41.437045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:41.437045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:10.512019+0000","scrub_duration":0.0029926649999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.876629+0000","last_change":"2024-09-11T08:09:04.031954+0000","last_active":"2024-09-11T08:09:06.876629+0000","last_peered":"2024-09-11T08:09:06.876629+0000","last_clean":"2024-09-11T08:09:06.876629+0000","last_became_active":"2024-09-11T08:07:36.071194+0000","last_became_peered":"2024-09-11T08:07:36.071194+0000","last_unstale":"2024-09-11T08:09:06.876629+0000","last_undegraded":"2024-09-11T08:09:06.876629+0000","last_fullsized":"2024-09-11T08:09:06.876629+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:04.031894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:09:04.031894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:28.971103+0000","scrub_duration":0.0013516579999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.871456+0000","last_change":"2024-09-11T08:08:42.447971+0000","last_active":"2024-09-11T08:09:06.871456+0000","last_peered":"2024-09-11T08:09:06.871456+0000","last_clean":"2024-09-11T08:09:06.871456+0000","last_became_active":"2024-09-11T08:07:32.043314+0000","last_became_peered":"2024-09-11T08:07:32.043314+0000","last_unstale":"2024-09-11T08:09:06.871456+0000","last_undegraded":"2024-09-11T08:09:06.871456+0000","last_fullsized":"2024-09-11T08:09:06.871456+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-11T08:08:42.447892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:42.447892+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:03.778832+0000","scrub_duration":0.0039015859999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.872368+0000","last_change":"2024-09-11T08:08:30.327580+0000","last_active":"2024-09-11T08:09:06.872368+0000","last_peered":"2024-09-11T08:09:06.872368+0000","last_clean":"2024-09-11T08:09:06.872368+0000","last_became_active":"2024-09-11T08:07:30.046684+0000","last_became_peered":"2024-09-11T08:07:30.046684+0000","last_unstale":"2024-09-11T08:09:06.872368+0000","last_undegraded":"2024-09-11T08:09:06.872368+0000","last_fullsized":"2024-09-11T08:09:06.872368+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:30.327525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:30.327525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:57.278810+0000","scrub_duration":0.001475589,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"66'8","reported_seq":71,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:13.562291+0000","last_change":"2024-09-11T08:08:49.026859+0000","last_active":"2024-09-11T08:09:13.562291+0000","last_peered":"2024-09-11T08:09:13.562291+0000","last_clean":"2024-09-11T08:09:13.562291+0000","last_became_active":"2024-09-11T08:07:34.062469+0000","last_became_peered":"2024-09-11T08:07:34.062469+0000","last_unstale":"2024-09-11T08:09:13.562291+0000","last_undegraded":"2024-09-11T08:09:13.562291+0000","last_fullsized":"2024-09-11T08:09:13.562291+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:08:49.026755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:49.026755+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:59.460486+0000","scrub_duration":0.0018077970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:14.613631+0000","last_change":"2024-09-11T08:09:14.613599+0000","last_active":"2024-09-11T08:09:14.613631+0000","last_peered":"2024-09-11T08:09:14.613631+0000","last_clean":"2024-09-11T08:09:14.613631+0000","last_became_active":"2024-09-11T08:07:36.074684+0000","last_became_peered":"2024-09-11T08:07:36.074684+0000","last_unstale":"2024-09-11T08:09:14.613631+0000","last_undegraded":"2024-09-11T08:09:14.613631+0000","last_fullsized":"2024-09-11T08:09:14.613631+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:14.613532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:14.613532+0000","last_clean_scrub_stamp":"2024-09-11T08:09:14.613532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:29.864488+0000","scrub_duration":0.001503648,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":50,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.937776+0000","last_change":"2024-09-11T08:07:32.055237+0000","last_active":"2024-09-11T08:09:06.937776+0000","last_peered":"2024-09-11T08:09:06.937776+0000","last_clean":"2024-09-11T08:09:06.937776+0000","last_became_active":"2024-09-11T08:07:32.054885+0000","last_became_peered":"2024-09-11T08:07:32.054885+0000","last_unstale":"2024-09-11T08:09:06.937776+0000","last_undegraded":"2024-09-11T08:09:06.937776+0000","last_fullsized":"2024-09-11T08:09:06.937776+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.874962+0000","last_change":"2024-09-11T08:08:34.828231+0000","last_active":"2024-09-11T08:09:06.874962+0000","last_peered":"2024-09-11T08:09:06.874962+0000","last_clean":"2024-09-11T08:09:06.874962+0000","last_became_active":"2024-09-11T08:07:30.045613+0000","last_became_peered":"2024-09-11T08:07:30.045613+0000","last_unstale":"2024-09-11T08:09:06.874962+0000","last_undegraded":"2024-09-11T08:09:06.874962+0000","last_fullsized":"2024-09-11T08:09:06.874962+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:34.828129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:34.828129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:58.397313+0000","scrub_duration":0.0013782530000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.875873+0000","last_change":"2024-09-11T08:08:37.871017+0000","last_active":"2024-09-11T08:09:06.875873+0000","last_peered":"2024-09-11T08:09:06.875873+0000","last_clean":"2024-09-11T08:09:06.875873+0000","last_became_active":"2024-09-11T08:07:34.056606+0000","last_became_peered":"2024-09-11T08:07:34.056606+0000","last_unstale":"2024-09-11T08:09:06.875873+0000","last_undegraded":"2024-09-11T08:09:06.875873+0000","last_fullsized":"2024-09-11T08:09:06.875873+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:37.870955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:37.870955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:58.505876+0000","scrub_duration":0.0016274460000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.937446+0000","last_change":"2024-09-11T08:09:05.932261+0000","last_active":"2024-09-11T08:09:06.937446+0000","last_peered":"2024-09-11T08:09:06.937446+0000","last_clean":"2024-09-11T08:09:06.937446+0000","last_became_active":"2024-09-11T08:07:36.072452+0000","last_became_peered":"2024-09-11T08:07:36.072452+0000","last_unstale":"2024-09-11T08:09:06.937446+0000","last_undegraded":"2024-09-11T08:09:06.937446+0000","last_fullsized":"2024-09-11T08:09:06.937446+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:05.932180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:09:05.932180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:07.863587+0000","scrub_duration":0.0013481960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":74,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.937899+0000","last_change":"2024-09-11T08:08:53.616543+0000","last_active":"2024-09-11T08:09:06.937899+0000","last_peered":"2024-09-11T08:09:06.937899+0000","last_clean":"2024-09-11T08:09:06.937899+0000","last_became_active":"2024-09-11T08:07:32.050880+0000","last_became_peered":"2024-09-11T08:07:32.050880+0000","last_unstale":"2024-09-11T08:09:06.937899+0000","last_undegraded":"2024-09-11T08:09:06.937899+0000","last_fullsized":"2024-09-11T08:09:06.937899+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-11T08:08:53.616450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:53.616450+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:18.217121+0000","scrub_duration":0.0024100559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.871804+0000","last_change":"2024-09-11T08:08:44.385440+0000","last_active":"2024-09-11T08:09:06.871804+0000","last_peered":"2024-09-11T08:09:06.871804+0000","last_clean":"2024-09-11T08:09:06.871804+0000","last_became_active":"2024-09-11T08:07:30.043294+0000","last_became_peered":"2024-09-11T08:07:30.043294+0000","last_unstale":"2024-09-11T08:09:06.871804+0000","last_undegraded":"2024-09-11T08:09:06.871804+0000","last_fullsized":"2024-09-11T08:09:06.871804+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:44.385372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:44.385372+0000","last_clean_scrub_stamp":"2024-09-11T08:08:44.385372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:46.537624+0000","scrub_duration":0.001441007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.876135+0000","last_change":"2024-09-11T08:08:51.921479+0000","last_active":"2024-09-11T08:09:06.876135+0000","last_peered":"2024-09-11T08:09:06.876135+0000","last_clean":"2024-09-11T08:09:06.876135+0000","last_became_active":"2024-09-11T08:07:34.063589+0000","last_became_peered":"2024-09-11T08:07:34.063589+0000","last_unstale":"2024-09-11T08:09:06.876135+0000","last_undegraded":"2024-09-11T08:09:06.876135+0000","last_fullsized":"2024-09-11T08:09:06.876135+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:51.921389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:51.921389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:01.266430+0000","scrub_duration":0.001807091,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.871054+0000","last_change":"2024-09-11T08:08:36.367954+0000","last_active":"2024-09-11T08:09:06.871054+0000","last_peered":"2024-09-11T08:09:06.871054+0000","last_clean":"2024-09-11T08:09:06.871054+0000","last_became_active":"2024-09-11T08:07:36.077523+0000","last_became_peered":"2024-09-11T08:07:36.077523+0000","last_unstale":"2024-09-11T08:09:06.871054+0000","last_undegraded":"2024-09-11T08:09:06.871054+0000","last_fullsized":"2024-09-11T08:09:06.871054+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:36.367890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:08:36.367890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:40.081903+0000","scrub_duration":0.0015112560000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":67,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.872732+0000","last_change":"2024-09-11T08:08:59.494039+0000","last_active":"2024-09-11T08:09:06.872732+0000","last_peered":"2024-09-11T08:09:06.872732+0000","last_clean":"2024-09-11T08:09:06.872732+0000","last_became_active":"2024-09-11T08:07:32.048164+0000","last_became_peered":"2024-09-11T08:07:32.048164+0000","last_unstale":"2024-09-11T08:09:06.872732+0000","last_undegraded":"2024-09-11T08:09:06.872732+0000","last_fullsized":"2024-09-11T08:09:06.872732+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-11T08:08:59.493975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:59.493975+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:17.528763+0000","scrub_duration":0.0019438280000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.872188+0000","last_change":"2024-09-11T08:08:53.247349+0000","last_active":"2024-09-11T08:09:06.872188+0000","last_peered":"2024-09-11T08:09:06.872188+0000","last_clean":"2024-09-11T08:09:06.872188+0000","last_became_active":"2024-09-11T08:07:30.041219+0000","last_became_peered":"2024-09-11T08:07:30.041219+0000","last_unstale":"2024-09-11T08:09:06.872188+0000","last_undegraded":"2024-09-11T08:09:06.872188+0000","last_fullsized":"2024-09-11T08:09:06.872188+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:53.247275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:53.247275+0000","last_clean_scrub_stamp":"2024-09-11T08:08:53.247275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:00.288102+0000","scrub_duration":0.0012568169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.874982+0000","last_change":"2024-09-11T08:08:46.970593+0000","last_active":"2024-09-11T08:09:06.874982+0000","last_peered":"2024-09-11T08:09:06.874982+0000","last_clean":"2024-09-11T08:09:06.874982+0000","last_became_active":"2024-09-11T08:07:34.057479+0000","last_became_peered":"2024-09-11T08:07:34.057479+0000","last_unstale":"2024-09-11T08:09:06.874982+0000","last_undegraded":"2024-09-11T08:09:06.874982+0000","last_fullsized":"2024-09-11T08:09:06.874982+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:46.970539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:46.970539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:09.551366+0000","scrub_duration":0.001167143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.872073+0000","last_change":"2024-09-11T08:09:00.395353+0000","last_active":"2024-09-11T08:09:06.872073+0000","last_peered":"2024-09-11T08:09:06.872073+0000","last_clean":"2024-09-11T08:09:06.872073+0000","last_became_active":"2024-09-11T08:07:36.078075+0000","last_became_peered":"2024-09-11T08:07:36.078075+0000","last_unstale":"2024-09-11T08:09:06.872073+0000","last_undegraded":"2024-09-11T08:09:06.872073+0000","last_fullsized":"2024-09-11T08:09:06.872073+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:00.395264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:09:00.395264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:07.333876+0000","scrub_duration":0.001470147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":68,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:10.812355+0000","last_change":"2024-09-11T08:09:10.812331+0000","last_active":"2024-09-11T08:09:10.812355+0000","last_peered":"2024-09-11T08:09:10.812355+0000","last_clean":"2024-09-11T08:09:10.812355+0000","last_became_active":"2024-09-11T08:07:32.039275+0000","last_became_peered":"2024-09-11T08:07:32.039275+0000","last_unstale":"2024-09-11T08:09:10.812355+0000","last_undegraded":"2024-09-11T08:09:10.812355+0000","last_fullsized":"2024-09-11T08:09:10.812355+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-11T08:09:10.812274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:09:10.812274+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:23.639951+0000","scrub_duration":0.0016606539999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":63,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:17.263222+0000","last_change":"2024-09-11T08:09:17.263189+0000","last_active":"2024-09-11T08:09:17.263222+0000","last_peered":"2024-09-11T08:09:17.263222+0000","last_clean":"2024-09-11T08:09:17.263222+0000","last_became_active":"2024-09-11T08:07:30.045967+0000","last_became_peered":"2024-09-11T08:07:30.045967+0000","last_unstale":"2024-09-11T08:09:17.263222+0000","last_undegraded":"2024-09-11T08:09:17.263222+0000","last_fullsized":"2024-09-11T08:09:17.263222+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-11T08:09:17.263115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:09:17.263115+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:32.872969+0000","scrub_duration":0.001972723,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"66'8","reported_seq":72,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:13.562496+0000","last_change":"2024-09-11T08:08:59.170974+0000","last_active":"2024-09-11T08:09:13.562496+0000","last_peered":"2024-09-11T08:09:13.562496+0000","last_clean":"2024-09-11T08:09:13.562496+0000","last_became_active":"2024-09-11T08:07:34.058015+0000","last_became_peered":"2024-09-11T08:07:34.058015+0000","last_unstale":"2024-09-11T08:09:13.562496+0000","last_undegraded":"2024-09-11T08:09:13.562496+0000","last_fullsized":"2024-09-11T08:09:13.562496+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:08:59.170918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:59.170918+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:12.511014+0000","scrub_duration":0.001468128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":39,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:09.802062+0000","last_change":"2024-09-11T08:09:09.802025+0000","last_active":"2024-09-11T08:09:09.802062+0000","last_peered":"2024-09-11T08:09:09.802062+0000","last_clean":"2024-09-11T08:09:09.802062+0000","last_became_active":"2024-09-11T08:07:36.631755+0000","last_became_peered":"2024-09-11T08:07:36.631755+0000","last_unstale":"2024-09-11T08:09:09.802062+0000","last_undegraded":"2024-09-11T08:09:09.802062+0000","last_fullsized":"2024-09-11T08:09:09.802062+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:09.801945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:09:09.801945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:36.844477+0000","scrub_duration":0.0013599739999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:07.818093+0000","last_change":"2024-09-11T08:09:07.818014+0000","last_active":"2024-09-11T08:09:07.818093+0000","last_peered":"2024-09-11T08:09:07.818093+0000","last_clean":"2024-09-11T08:09:07.818093+0000","last_became_active":"2024-09-11T08:07:30.045910+0000","last_became_peered":"2024-09-11T08:07:30.045910+0000","last_unstale":"2024-09-11T08:09:07.818093+0000","last_undegraded":"2024-09-11T08:09:07.818093+0000","last_fullsized":"2024-09-11T08:09:07.818093+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:07.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:09:07.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:31.673092+0000","scrub_duration":0.001347413,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"84'9","reported_seq":56,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.872128+0000","last_change":"2024-09-11T08:09:03.418515+0000","last_active":"2024-09-11T08:09:06.872128+0000","last_peered":"2024-09-11T08:09:06.872128+0000","last_clean":"2024-09-11T08:09:06.872128+0000","last_became_active":"2024-09-11T08:07:32.046891+0000","last_became_peered":"2024-09-11T08:07:32.046891+0000","last_unstale":"2024-09-11T08:09:06.872128+0000","last_undegraded":"2024-09-11T08:09:06.872128+0000","last_fullsized":"2024-09-11T08:09:06.872128+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-09-11T08:09:03.418457+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-09-11T08:09:03.418457+0000","last_clean_scrub_stamp":"2024-09-11T08:09:03.418457+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:10.688358+0000","scrub_duration":0.003076058,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:07.779603+0000","last_change":"2024-09-11T08:09:07.779562+0000","last_active":"2024-09-11T08:09:07.779603+0000","last_peered":"2024-09-11T08:09:07.779603+0000","last_clean":"2024-09-11T08:09:07.779603+0000","last_became_active":"2024-09-11T08:07:34.057774+0000","last_became_peered":"2024-09-11T08:07:34.057774+0000","last_unstale":"2024-09-11T08:09:07.779603+0000","last_undegraded":"2024-09-11T08:09:07.779603+0000","last_fullsized":"2024-09-11T08:09:07.779603+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:07.779494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:09:07.779494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:34.783182+0000","scrub_duration":0.0018758360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.871267+0000","last_change":"2024-09-11T08:08:38.326345+0000","last_active":"2024-09-11T08:09:06.871267+0000","last_peered":"2024-09-11T08:09:06.871267+0000","last_clean":"2024-09-11T08:09:06.871267+0000","last_became_active":"2024-09-11T08:07:36.633244+0000","last_became_peered":"2024-09-11T08:07:36.633244+0000","last_unstale":"2024-09-11T08:09:06.871267+0000","last_undegraded":"2024-09-11T08:09:06.871267+0000","last_fullsized":"2024-09-11T08:09:06.871267+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:38.326265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:08:38.326265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:06.322672+0000","scrub_duration":0.001417361,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"84'13","reported_seq":64,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.937847+0000","last_change":"2024-09-11T08:08:57.835686+0000","last_active":"2024-09-11T08:09:06.937847+0000","last_peered":"2024-09-11T08:09:06.937847+0000","last_clean":"2024-09-11T08:09:06.937847+0000","last_became_active":"2024-09-11T08:07:32.048350+0000","last_became_peered":"2024-09-11T08:07:32.048350+0000","last_unstale":"2024-09-11T08:09:06.937847+0000","last_undegraded":"2024-09-11T08:09:06.937847+0000","last_fullsized":"2024-09-11T08:09:06.937847+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'13","last_scrub_stamp":"2024-09-11T08:08:57.835634+0000","last_deep_scrub":"84'13","last_deep_scrub_stamp":"2024-09-11T08:08:57.835634+0000","last_clean_scrub_stamp":"2024-09-11T08:08:57.835634+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:18.729258+0000","scrub_duration":0.0033293229999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.871043+0000","last_change":"2024-09-11T08:08:30.471690+0000","last_active":"2024-09-11T08:09:06.871043+0000","last_peered":"2024-09-11T08:09:06.871043+0000","last_clean":"2024-09-11T08:09:06.871043+0000","last_became_active":"2024-09-11T08:07:30.044055+0000","last_became_peered":"2024-09-11T08:07:30.044055+0000","last_unstale":"2024-09-11T08:09:06.871043+0000","last_undegraded":"2024-09-11T08:09:06.871043+0000","last_fullsized":"2024-09-11T08:09:06.871043+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:30.471620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:30.471620+0000","last_clean_scrub_stamp":"2024-09-11T08:08:30.471620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:37.891558+0000","scrub_duration":0.0016049980000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.871964+0000","last_change":"2024-09-11T08:09:04.370800+0000","last_active":"2024-09-11T08:09:06.871964+0000","last_peered":"2024-09-11T08:09:06.871964+0000","last_clean":"2024-09-11T08:09:06.871964+0000","last_became_active":"2024-09-11T08:07:34.057127+0000","last_became_peered":"2024-09-11T08:07:34.057127+0000","last_unstale":"2024-09-11T08:09:06.871964+0000","last_undegraded":"2024-09-11T08:09:06.871964+0000","last_fullsized":"2024-09-11T08:09:06.871964+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:04.370734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:04.370734+0000","last_clean_scrub_stamp":"2024-09-11T08:09:04.370734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:09.164201+0000","scrub_duration":0.001224483,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.876838+0000","last_change":"2024-09-11T08:09:01.995115+0000","last_active":"2024-09-11T08:09:06.876838+0000","last_peered":"2024-09-11T08:09:06.876838+0000","last_clean":"2024-09-11T08:09:06.876838+0000","last_became_active":"2024-09-11T08:07:36.071513+0000","last_became_peered":"2024-09-11T08:07:36.071513+0000","last_unstale":"2024-09-11T08:09:06.876838+0000","last_undegraded":"2024-09-11T08:09:06.876838+0000","last_fullsized":"2024-09-11T08:09:06.876838+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:01.995061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:09:01.995061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:18.017918+0000","scrub_duration":0.00115722,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.875054+0000","last_change":"2024-09-11T08:08:49.951403+0000","last_active":"2024-09-11T08:09:06.875054+0000","last_peered":"2024-09-11T08:09:06.875054+0000","last_clean":"2024-09-11T08:09:06.875054+0000","last_became_active":"2024-09-11T08:07:32.041761+0000","last_became_peered":"2024-09-11T08:07:32.041761+0000","last_unstale":"2024-09-11T08:09:06.875054+0000","last_undegraded":"2024-09-11T08:09:06.875054+0000","last_fullsized":"2024-09-11T08:09:06.875054+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:08:49.951345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:49.951345+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:51.233235+0000","scrub_duration":0.002079914,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.937500+0000","last_change":"2024-09-11T08:08:48.935452+0000","last_active":"2024-09-11T08:09:06.937500+0000","last_peered":"2024-09-11T08:09:06.937500+0000","last_clean":"2024-09-11T08:09:06.937500+0000","last_became_active":"2024-09-11T08:07:30.045380+0000","last_became_peered":"2024-09-11T08:07:30.045380+0000","last_unstale":"2024-09-11T08:09:06.937500+0000","last_undegraded":"2024-09-11T08:09:06.937500+0000","last_fullsized":"2024-09-11T08:09:06.937500+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:48.935387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:48.935387+0000","last_clean_scrub_stamp":"2024-09-11T08:08:48.935387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:05.810526+0000","scrub_duration":0.001422134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"66'8","reported_seq":72,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:13.562484+0000","last_change":"2024-09-11T08:08:52.374593+0000","last_active":"2024-09-11T08:09:13.562484+0000","last_peered":"2024-09-11T08:09:13.562484+0000","last_clean":"2024-09-11T08:09:13.562484+0000","last_became_active":"2024-09-11T08:07:34.057937+0000","last_became_peered":"2024-09-11T08:07:34.057937+0000","last_unstale":"2024-09-11T08:09:13.562484+0000","last_undegraded":"2024-09-11T08:09:13.562484+0000","last_fullsized":"2024-09-11T08:09:13.562484+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:08:52.374539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:52.374539+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:12.088877+0000","scrub_duration":0.00146886,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:15.617459+0000","last_change":"2024-09-11T08:09:15.617419+0000","last_active":"2024-09-11T08:09:15.617459+0000","last_peered":"2024-09-11T08:09:15.617459+0000","last_clean":"2024-09-11T08:09:15.617459+0000","last_became_active":"2024-09-11T08:07:36.068493+0000","last_became_peered":"2024-09-11T08:07:36.068493+0000","last_unstale":"2024-09-11T08:09:15.617459+0000","last_undegraded":"2024-09-11T08:09:15.617459+0000","last_fullsized":"2024-09-11T08:09:15.617459+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:15.617360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:09:15.617360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:37.620427+0000","scrub_duration":0.0014734049999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":81,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:07.884651+0000","last_change":"2024-09-11T08:09:07.884651+0000","last_active":"2024-09-11T08:09:07.884651+0000","last_peered":"2024-09-11T08:09:07.884651+0000","last_clean":"2024-09-11T08:09:07.884651+0000","last_became_active":"2024-09-11T08:09:07.878358+0000","last_became_peered":"2024-09-11T08:09:07.878358+0000","last_unstale":"2024-09-11T08:09:07.884651+0000","last_undegraded":"2024-09-11T08:09:07.884651+0000","last_fullsized":"2024-09-11T08:09:07.884651+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-11T08:08:36.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:36.400586+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:57.137379+0000","scrub_duration":0.002193113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.871165+0000","last_change":"2024-09-11T08:09:03.210218+0000","last_active":"2024-09-11T08:09:06.871165+0000","last_peered":"2024-09-11T08:09:06.871165+0000","last_clean":"2024-09-11T08:09:06.871165+0000","last_became_active":"2024-09-11T08:07:30.039673+0000","last_became_peered":"2024-09-11T08:07:30.039673+0000","last_unstale":"2024-09-11T08:09:06.871165+0000","last_undegraded":"2024-09-11T08:09:06.871165+0000","last_fullsized":"2024-09-11T08:09:06.871165+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:03.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:09:03.210166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:05.614628+0000","scrub_duration":0.0011643720000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":71,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:13.562533+0000","last_change":"2024-09-11T08:08:47.966763+0000","last_active":"2024-09-11T08:09:13.562533+0000","last_peered":"2024-09-11T08:09:13.562533+0000","last_clean":"2024-09-11T08:09:13.562533+0000","last_became_active":"2024-09-11T08:07:34.060736+0000","last_became_peered":"2024-09-11T08:07:34.060736+0000","last_unstale":"2024-09-11T08:09:13.562533+0000","last_undegraded":"2024-09-11T08:09:13.562533+0000","last_fullsized":"2024-09-11T08:09:13.562533+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:08:47.966690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:47.966690+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:12.893275+0000","scrub_duration":0.0019777940000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.937623+0000","last_change":"2024-09-11T08:07:36.072648+0000","last_active":"2024-09-11T08:09:06.937623+0000","last_peered":"2024-09-11T08:09:06.937623+0000","last_clean":"2024-09-11T08:09:06.937623+0000","last_became_active":"2024-09-11T08:07:36.072359+0000","last_became_peered":"2024-09-11T08:07:36.072359+0000","last_unstale":"2024-09-11T08:09:06.937623+0000","last_undegraded":"2024-09-11T08:09:06.937623+0000","last_fullsized":"2024-09-11T08:09:06.937623+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":63,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.872546+0000","last_change":"2024-09-11T08:09:03.516720+0000","last_active":"2024-09-11T08:09:06.872546+0000","last_peered":"2024-09-11T08:09:06.872546+0000","last_clean":"2024-09-11T08:09:06.872546+0000","last_became_active":"2024-09-11T08:07:32.047981+0000","last_became_peered":"2024-09-11T08:07:32.047981+0000","last_unstale":"2024-09-11T08:09:06.872546+0000","last_undegraded":"2024-09-11T08:09:06.872546+0000","last_fullsized":"2024-09-11T08:09:06.872546+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-11T08:09:03.516664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:09:03.516664+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:23.127237+0000","scrub_duration":0.001710917,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.871934+0000","last_change":"2024-09-11T08:08:48.330351+0000","last_active":"2024-09-11T08:09:06.871934+0000","last_peered":"2024-09-11T08:09:06.871934+0000","last_clean":"2024-09-11T08:09:06.871934+0000","last_became_active":"2024-09-11T08:07:30.043691+0000","last_became_peered":"2024-09-11T08:07:30.043691+0000","last_unstale":"2024-09-11T08:09:06.871934+0000","last_undegraded":"2024-09-11T08:09:06.871934+0000","last_fullsized":"2024-09-11T08:09:06.871934+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:48.330285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:48.330285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:05.763303+0000","scrub_duration":0.001379929,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":32,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.876371+0000","last_change":"2024-09-11T08:07:34.053427+0000","last_active":"2024-09-11T08:09:06.876371+0000","last_peered":"2024-09-11T08:09:06.876371+0000","last_clean":"2024-09-11T08:09:06.876371+0000","last_became_active":"2024-09-11T08:07:34.053137+0000","last_became_peered":"2024-09-11T08:07:34.053137+0000","last_unstale":"2024-09-11T08:09:06.876371+0000","last_undegraded":"2024-09-11T08:09:06.876371+0000","last_fullsized":"2024-09-11T08:09:06.876371+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:07:33.032025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.871896+0000","last_change":"2024-09-11T08:07:36.076837+0000","last_active":"2024-09-11T08:09:06.871896+0000","last_peered":"2024-09-11T08:09:06.871896+0000","last_clean":"2024-09-11T08:09:06.871896+0000","last_became_active":"2024-09-11T08:07:36.076132+0000","last_became_peered":"2024-09-11T08:07:36.076132+0000","last_unstale":"2024-09-11T08:09:06.871896+0000","last_undegraded":"2024-09-11T08:09:06.871896+0000","last_fullsized":"2024-09-11T08:09:06.871896+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":60,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.876551+0000","last_change":"2024-09-11T08:08:55.332126+0000","last_active":"2024-09-11T08:09:06.876551+0000","last_peered":"2024-09-11T08:09:06.876551+0000","last_clean":"2024-09-11T08:09:06.876551+0000","last_became_active":"2024-09-11T08:07:32.049368+0000","last_became_peered":"2024-09-11T08:07:32.049368+0000","last_unstale":"2024-09-11T08:09:06.876551+0000","last_undegraded":"2024-09-11T08:09:06.876551+0000","last_fullsized":"2024-09-11T08:09:06.876551+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:08:55.332065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:08:55.332065+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:15.584089+0000","scrub_duration":0.001632218,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.871401+0000","last_change":"2024-09-11T08:08:45.388090+0000","last_active":"2024-09-11T08:09:06.871401+0000","last_peered":"2024-09-11T08:09:06.871401+0000","last_clean":"2024-09-11T08:09:06.871401+0000","last_became_active":"2024-09-11T08:07:30.037692+0000","last_became_peered":"2024-09-11T08:07:30.037692+0000","last_unstale":"2024-09-11T08:09:06.871401+0000","last_undegraded":"2024-09-11T08:09:06.871401+0000","last_fullsized":"2024-09-11T08:09:06.871401+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:45.388024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:45.388024+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:59.280827+0000","scrub_duration":0.0014875719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:19.535727+0000","last_change":"2024-09-11T08:09:19.535702+0000","last_active":"2024-09-11T08:09:19.535727+0000","last_peered":"2024-09-11T08:09:19.535727+0000","last_clean":"2024-09-11T08:09:19.535727+0000","last_became_active":"2024-09-11T08:07:34.057064+0000","last_became_peered":"2024-09-11T08:07:34.057064+0000","last_unstale":"2024-09-11T08:09:19.535727+0000","last_undegraded":"2024-09-11T08:09:19.535727+0000","last_fullsized":"2024-09-11T08:09:19.535727+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:19.535648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:09:19.535648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:42.152022+0000","scrub_duration":0.001265045,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:09.974085+0000","last_change":"2024-09-11T08:09:09.974044+0000","last_active":"2024-09-11T08:09:09.974085+0000","last_peered":"2024-09-11T08:09:09.974085+0000","last_clean":"2024-09-11T08:09:09.974085+0000","last_became_active":"2024-09-11T08:07:36.077406+0000","last_became_peered":"2024-09-11T08:07:36.077406+0000","last_unstale":"2024-09-11T08:09:09.974085+0000","last_undegraded":"2024-09-11T08:09:09.974085+0000","last_fullsized":"2024-09-11T08:09:09.974085+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:09.973943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:09.973943+0000","last_clean_scrub_stamp":"2024-09-11T08:09:09.973943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:12.082942+0000","scrub_duration":0.001430668,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":53,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.871972+0000","last_change":"2024-09-11T08:07:32.043568+0000","last_active":"2024-09-11T08:09:06.871972+0000","last_peered":"2024-09-11T08:09:06.871972+0000","last_clean":"2024-09-11T08:09:06.871972+0000","last_became_active":"2024-09-11T08:07:32.043276+0000","last_became_peered":"2024-09-11T08:07:32.043276+0000","last_unstale":"2024-09-11T08:09:06.871972+0000","last_undegraded":"2024-09-11T08:09:06.871972+0000","last_fullsized":"2024-09-11T08:09:06.871972+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.875072+0000","last_change":"2024-09-11T08:08:55.853168+0000","last_active":"2024-09-11T08:09:06.875072+0000","last_peered":"2024-09-11T08:09:06.875072+0000","last_clean":"2024-09-11T08:09:06.875072+0000","last_became_active":"2024-09-11T08:07:30.049152+0000","last_became_peered":"2024-09-11T08:07:30.049152+0000","last_unstale":"2024-09-11T08:09:06.875072+0000","last_undegraded":"2024-09-11T08:09:06.875072+0000","last_fullsized":"2024-09-11T08:09:06.875072+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:55.853076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:55.853076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:03.761618+0000","scrub_duration":0.0018073309999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:10.182603+0000","last_change":"2024-09-11T08:09:10.182577+0000","last_active":"2024-09-11T08:09:10.182603+0000","last_peered":"2024-09-11T08:09:10.182603+0000","last_clean":"2024-09-11T08:09:10.182603+0000","last_became_active":"2024-09-11T08:07:34.056935+0000","last_became_peered":"2024-09-11T08:07:34.056935+0000","last_unstale":"2024-09-11T08:09:10.182603+0000","last_undegraded":"2024-09-11T08:09:10.182603+0000","last_fullsized":"2024-09-11T08:09:10.182603+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:10.182522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:09:10.182522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:38.697741+0000","scrub_duration":0.001386111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.872294+0000","last_change":"2024-09-11T08:07:36.633645+0000","last_active":"2024-09-11T08:09:06.872294+0000","last_peered":"2024-09-11T08:09:06.872294+0000","last_clean":"2024-09-11T08:09:06.872294+0000","last_became_active":"2024-09-11T08:07:36.633368+0000","last_became_peered":"2024-09-11T08:07:36.633368+0000","last_unstale":"2024-09-11T08:09:06.872294+0000","last_undegraded":"2024-09-11T08:09:06.872294+0000","last_fullsized":"2024-09-11T08:09:06.872294+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:07:35.048712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.871541+0000","last_change":"2024-09-11T08:07:32.046050+0000","last_active":"2024-09-11T08:09:06.871541+0000","last_peered":"2024-09-11T08:09:06.871541+0000","last_clean":"2024-09-11T08:09:06.871541+0000","last_became_active":"2024-09-11T08:07:32.045728+0000","last_became_peered":"2024-09-11T08:07:32.045728+0000","last_unstale":"2024-09-11T08:09:06.871541+0000","last_undegraded":"2024-09-11T08:09:06.871541+0000","last_fullsized":"2024-09-11T08:09:06.871541+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:07:31.028627+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.937832+0000","last_change":"2024-09-11T08:08:42.056647+0000","last_active":"2024-09-11T08:09:06.937832+0000","last_peered":"2024-09-11T08:09:06.937832+0000","last_clean":"2024-09-11T08:09:06.937832+0000","last_became_active":"2024-09-11T08:07:30.049409+0000","last_became_peered":"2024-09-11T08:07:30.049409+0000","last_unstale":"2024-09-11T08:09:06.937832+0000","last_undegraded":"2024-09-11T08:09:06.937832+0000","last_fullsized":"2024-09-11T08:09:06.937832+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:42.056591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:42.056591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:56.872271+0000","scrub_duration":0.003068809,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.937670+0000","last_change":"2024-09-11T08:08:34.506991+0000","last_active":"2024-09-11T08:09:06.937670+0000","last_peered":"2024-09-11T08:09:06.937670+0000","last_clean":"2024-09-11T08:09:06.937670+0000","last_became_active":"2024-09-11T08:07:34.055082+0000","last_became_peered":"2024-09-11T08:07:34.055082+0000","last_unstale":"2024-09-11T08:09:06.937670+0000","last_undegraded":"2024-09-11T08:09:06.937670+0000","last_fullsized":"2024-09-11T08:09:06.937670+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:34.506927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:34.506927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:46.584592+0000","scrub_duration":0.0015499070000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:18.304281+0000","last_change":"2024-09-11T08:09:18.304241+0000","last_active":"2024-09-11T08:09:18.304281+0000","last_peered":"2024-09-11T08:09:18.304281+0000","last_clean":"2024-09-11T08:09:18.304281+0000","last_became_active":"2024-09-11T08:07:36.071669+0000","last_became_peered":"2024-09-11T08:07:36.071669+0000","last_unstale":"2024-09-11T08:09:18.304281+0000","last_undegraded":"2024-09-11T08:09:18.304281+0000","last_fullsized":"2024-09-11T08:09:18.304281+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:18.304148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:09:18.304148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:20.420545+0000","scrub_duration":0.0015506610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T08:09:09.297179+0000","last_change":"2024-09-11T08:09:09.297155+0000","last_active":"2024-09-11T08:09:09.297179+0000","last_peered":"2024-09-11T08:09:09.297179+0000","last_clean":"2024-09-11T08:09:09.297179+0000","last_became_active":"2024-09-11T08:07:36.071468+0000","last_became_peered":"2024-09-11T08:07:36.071468+0000","last_unstale":"2024-09-11T08:09:09.297179+0000","last_undegraded":"2024-09-11T08:09:09.297179+0000","last_fullsized":"2024-09-11T08:09:09.297179+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:09.297101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:09:09.297101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:12.099110+0000","scrub_duration":0.001287886,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.871675+0000","last_change":"2024-09-11T08:08:49.428032+0000","last_active":"2024-09-11T08:09:06.871675+0000","last_peered":"2024-09-11T08:09:06.871675+0000","last_clean":"2024-09-11T08:09:06.871675+0000","last_became_active":"2024-09-11T08:07:30.046702+0000","last_became_peered":"2024-09-11T08:07:30.046702+0000","last_unstale":"2024-09-11T08:09:06.871675+0000","last_undegraded":"2024-09-11T08:09:06.871675+0000","last_fullsized":"2024-09-11T08:09:06.871675+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:49.427944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:08:49.427944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:11.537893+0000","scrub_duration":0.001383034,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":62,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.938140+0000","last_change":"2024-09-11T08:09:03.841778+0000","last_active":"2024-09-11T08:09:06.938140+0000","last_peered":"2024-09-11T08:09:06.938140+0000","last_clean":"2024-09-11T08:09:06.938140+0000","last_became_active":"2024-09-11T08:07:32.054656+0000","last_became_peered":"2024-09-11T08:07:32.054656+0000","last_unstale":"2024-09-11T08:09:06.938140+0000","last_undegraded":"2024-09-11T08:09:06.938140+0000","last_fullsized":"2024-09-11T08:09:06.938140+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-11T08:09:03.841721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:09:03.841721+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:10:20.583780+0000","scrub_duration":0.00194933,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-11T08:09:06.937664+0000","last_change":"2024-09-11T08:08:53.869870+0000","last_active":"2024-09-11T08:09:06.937664+0000","last_peered":"2024-09-11T08:09:06.937664+0000","last_clean":"2024-09-11T08:09:06.937664+0000","last_became_active":"2024-09-11T08:07:34.061453+0000","last_became_peered":"2024-09-11T08:07:34.061453+0000","last_unstale":"2024-09-11T08:09:06.937664+0000","last_undegraded":"2024-09-11T08:09:06.937664+0000","last_fullsized":"2024-09-11T08:09:06.937664+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:08:53.869813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:08:53.869813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:09:59.468130+0000","scrub_duration":0.0012524350000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":423,"ondisk_log_size":423,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":29,"num_read_kb":29,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":106,"num_read_kb":101,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266715,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298540,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446612,"statfs":{"total":95995035648,"available":95689330688,"internally_reserved":0,"allocated":1216512,"data_stored":877700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.47799999999999998,"15min":0.48099999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.60599999999999998,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.28899999999999998,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.65400000000000003,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.496}]},{"osd":1,"last update":"Wed Sep 11 08:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.495,"15min":0.49299999999999999},"min":{"1min":0.26400000000000001,"5min":0.216,"15min":0.216},"max":{"1min":0.65200000000000002,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.501,"5min":0.52300000000000002,"15min":0.52700000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.66500000000000004}]},{"osd":2,"last update":"Wed Sep 11 08:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.503,"15min":0.48899999999999999},"min":{"1min":0.34399999999999997,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":2.948,"5min":2.948,"15min":2.948},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.498,"15min":0.5},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.54400000000000004}]},{"osd":3,"last update":"Wed Sep 11 08:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.81999999999999995,"15min":0.878},"min":{"1min":0.39700000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.67500000000000004,"5min":7.9210000000000003,"15min":7.9210000000000003},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.48499999999999999,"15min":0.48299999999999998},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.67300000000000004}]},{"osd":4,"last update":"Wed Sep 11 08:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.373},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.56000000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.40200000000000002}]},{"osd":5,"last update":"Wed Sep 11 08:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.42199999999999999,"15min":0.42299999999999999},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.46000000000000002,"15min":0.45700000000000002},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.34300000000000003}]},{"osd":6,"last update":"Wed Sep 11 08:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.47099999999999997,"15min":0.46800000000000003},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.38}]}]},{"osd":6,"up_from":45,"seq":193273528352,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298524,"kb_used_data":1172,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446628,"statfs":{"total":95995035648,"available":95689347072,"internally_reserved":0,"allocated":1200128,"data_stored":877523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.49299999999999999,"15min":0.496},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.59099999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.50700000000000001,"15min":0.50600000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.72099999999999997,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.61399999999999999}]},{"osd":1,"last update":"Wed Sep 11 08:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.47999999999999998,"15min":0.47799999999999998},"min":{"1min":0.32700000000000001,"5min":0.318,"15min":0.318},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.497,"5min":0.505,"15min":0.50600000000000001},"min":{"1min":0.33500000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.65900000000000003,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.68600000000000005}]},{"osd":2,"last update":"Wed Sep 11 08:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.5,"15min":0.502},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.67200000000000004,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.499,"5min":0.48499999999999999,"15min":0.48299999999999998},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.72399999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.54000000000000004}]},{"osd":3,"last update":"Wed Sep 11 08:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.52400000000000002,"15min":0.52600000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.53900000000000003,"15min":0.53800000000000003},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.73499999999999999,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.755}]},{"osd":4,"last update":"Wed Sep 11 08:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.39000000000000001,"15min":0.38700000000000001},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.59399999999999997,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.41999999999999998,"15min":0.41199999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.66900000000000004}]},{"osd":5,"last update":"Wed Sep 11 08:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.48899999999999999,"15min":0.47999999999999998},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.434,"5min":0.41399999999999998,"15min":0.41099999999999998},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.62,"5min":0.629,"15min":0.629},"last":0.71999999999999997}]},{"osd":7,"last update":"Wed Sep 11 08:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.64300000000000002}]}]},{"osd":5,"up_from":38,"seq":163208757285,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297944,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447208,"statfs":{"total":95995035648,"available":95689940992,"internally_reserved":0,"allocated":606208,"data_stored":286472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.51400000000000001,"15min":0.52200000000000002},"min":{"1min":0.318,"5min":0.31,"15min":0.31},"max":{"1min":0.63500000000000001,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.503,"15min":0.501},"min":{"1min":0.41399999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.66000000000000003}]},{"osd":1,"last update":"Wed Sep 11 08:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":1.1910000000000001,"15min":1.3029999999999999},"min":{"1min":0.33000000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.82499999999999996,"5min":16.004999999999999,"15min":16.004999999999999},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.36199999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.499}]},{"osd":2,"last update":"Wed Sep 11 08:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":1.206,"15min":1.323},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.754,"5min":16.018000000000001,"15min":16.018000000000001},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.54800000000000004}]},{"osd":3,"last update":"Wed Sep 11 08:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.53100000000000003,"15min":0.52900000000000003},"min":{"1min":0.44900000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.72299999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.505,"5min":0.51600000000000001,"15min":0.51800000000000002},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.45200000000000001}]},{"osd":4,"last update":"Wed Sep 11 08:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.39700000000000002,"15min":0.39300000000000002},"min":{"1min":0.27500000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.40000000000000002,"15min":0.38800000000000001},"min":{"1min":0.36499999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.58399999999999996}]},{"osd":6,"last update":"Wed Sep 11 08:08:56 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.57699999999999996,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.495,"5min":0.45200000000000001,"15min":0.44500000000000001},"min":{"1min":0.316,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.67100000000000004,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.73599999999999999}]},{"osd":7,"last update":"Wed Sep 11 08:08:22 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.63}]}]},{"osd":4,"up_from":31,"seq":133143986218,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297964,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447188,"statfs":{"total":95995035648,"available":95689920512,"internally_reserved":0,"allocated":626688,"data_stored":287715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:09:10 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.495,"15min":0.50700000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.51100000000000001,"15min":0.53100000000000003},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.624,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.39300000000000002}]},{"osd":1,"last update":"Wed Sep 11 08:09:10 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.496,"15min":0.51000000000000001},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.65600000000000003,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52200000000000002,"15min":0.52900000000000003},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.66900000000000004,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.55400000000000005}]},{"osd":2,"last update":"Wed Sep 11 08:09:10 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.51200000000000001,"15min":0.52500000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.60999999999999999,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.498},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.51800000000000002,"15min":0.52600000000000002},"min":{"1min":0.39900000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.70799999999999996,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.41299999999999998}]},{"osd":3,"last update":"Wed Sep 11 08:09:10 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.50600000000000001,"15min":0.51400000000000001},"min":{"1min":0.36899999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.57899999999999996,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.436},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.50800000000000001,"15min":0.52200000000000002},"min":{"1min":0.40799999999999997,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.59999999999999998,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.45700000000000002}]},{"osd":5,"last update":"Wed Sep 11 08:08:31 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.36899999999999999,"15min":0.36699999999999999},"min":{"1min":0.27800000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.36899999999999999,"15min":0.36699999999999999},"min":{"1min":0.27900000000000003,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.28699999999999998}]},{"osd":6,"last update":"Wed Sep 11 08:08:56 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.39100000000000001,"15min":0.38300000000000001},"min":{"1min":0.30399999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.44,"15min":0.432},"min":{"1min":0.32200000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.35799999999999998}]},{"osd":7,"last update":"Wed Sep 11 08:09:21 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.46800000000000003,"15min":0.45500000000000002},"min":{"1min":0.35699999999999998,"5min":0.316,"15min":0.316},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.46200000000000002,"15min":0.45400000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.34300000000000003}]}]},{"osd":3,"up_from":25,"seq":107374182447,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297980,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447172,"statfs":{"total":95995035648,"available":95689904128,"internally_reserved":0,"allocated":643072,"data_stored":287501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:08:53 2024","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.36199999999999999,"15min":0.374},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":0.55100000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.40300000000000002,"15min":0.41799999999999998},"min":{"1min":0.26900000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.55100000000000005,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Sep 11 08:08:53 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.41599999999999998,"15min":0.437},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.497,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.316},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.41399999999999998,"15min":0.42899999999999999},"min":{"1min":0.24399999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.53800000000000003,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.379}]},{"osd":2,"last update":"Wed Sep 11 08:08:53 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.434,"15min":0.44900000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.47999999999999998,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.371,"5min":0.40600000000000003,"15min":0.41899999999999998},"min":{"1min":0.26700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.53100000000000003,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.35599999999999998}]},{"osd":4,"last update":"Wed Sep 11 08:09:12 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.53900000000000003,"15min":0.54600000000000004},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.52800000000000002,"15min":0.53500000000000003},"min":{"1min":0.34799999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.745,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.51100000000000001}]},{"osd":5,"last update":"Wed Sep 11 08:08:33 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.54200000000000004,"15min":0.53800000000000003},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.443},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.40699999999999997,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.5}]},{"osd":6,"last update":"Wed Sep 11 08:08:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.53400000000000003,"15min":0.53000000000000003},"min":{"1min":0.35899999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.55400000000000005,"15min":0.55800000000000005},"min":{"1min":0.39500000000000002,"5min":0.373,"15min":0.373},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.46100000000000002}]},{"osd":7,"last update":"Wed Sep 11 08:08:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.42799999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411378,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297944,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447208,"statfs":{"total":95995035648,"available":95689940992,"internally_reserved":0,"allocated":606208,"data_stored":286313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:08:24 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.38400000000000001,"15min":0.38100000000000001},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.58099999999999996,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.307},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.37,"15min":0.35999999999999999},"min":{"1min":0.25,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.55600000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.254}]},{"osd":1,"last update":"Wed Sep 11 08:08:24 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.39500000000000002,"15min":0.38},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.60899999999999999,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.39800000000000002,"15min":0.38600000000000001},"min":{"1min":0.28699999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.29199999999999998}]},{"osd":3,"last update":"Wed Sep 11 08:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.45100000000000001,"15min":0.46100000000000002},"min":{"1min":0.27300000000000002,"5min":0.218,"15min":0.218},"max":{"1min":0.629,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.45100000000000001,"15min":0.46500000000000002},"min":{"1min":0.318,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.32600000000000001}]},{"osd":4,"last update":"Wed Sep 11 08:09:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53100000000000003},"min":{"1min":0.41099999999999998,"5min":0.315,"15min":0.315},"max":{"1min":0.77100000000000002,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.45100000000000001},{"interface":"front","average":{"1min":1.2010000000000001,"5min":0.70499999999999996,"15min":0.627},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":12.129,"5min":12.129,"15min":12.129},"last":0.52000000000000002}]},{"osd":5,"last update":"Wed Sep 11 08:08:35 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.57699999999999996,"15min":0.58299999999999996},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.751,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.59199999999999997,"15min":0.59999999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.65300000000000002,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.49299999999999999}]},{"osd":6,"last update":"Wed Sep 11 08:08:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.58499999999999996,"15min":0.59399999999999997},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.59099999999999997,"15min":0.59699999999999998},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.70299999999999996,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.47699999999999998}]},{"osd":7,"last update":"Wed Sep 11 08:09:22 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.56999999999999995,"15min":0.57199999999999995},"min":{"1min":0.39100000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.79700000000000004,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.505},{"interface":"front","average":{"1min":1.1220000000000001,"5min":0.70499999999999996,"15min":0.63500000000000001},"min":{"1min":0.42099999999999999,"5min":0.371,"15min":0.371},"max":{"1min":12.115,"5min":12.115,"15min":12.115},"last":0.42099999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574904,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297980,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447172,"statfs":{"total":95995035648,"available":95689904128,"internally_reserved":0,"allocated":643072,"data_stored":287329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:09:04 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.33700000000000002,"15min":0.30399999999999999},"min":{"1min":0.23599999999999999,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.33900000000000002,"15min":0.315},"min":{"1min":0.224,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.42199999999999999}]},{"osd":2,"last update":"Wed Sep 11 08:08:28 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.36199999999999999,"15min":0.34799999999999998},"min":{"1min":0.27700000000000002,"5min":0.224,"15min":0.224},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.442},{"interface":"front","average":{"1min":0.37,"5min":0.35399999999999998,"15min":0.34200000000000003},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.40799999999999997}]},{"osd":3,"last update":"Wed Sep 11 08:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.41399999999999998,"15min":0.42199999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.377},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.439,"15min":0.437},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.198,"5min":1.198,"15min":1.198},"last":0.48499999999999999}]},{"osd":4,"last update":"Wed Sep 11 08:09:08 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.79800000000000004,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56100000000000005,"15min":0.55700000000000005},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.56299999999999994}]},{"osd":5,"last update":"Wed Sep 11 08:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.53700000000000003,"15min":0.53000000000000003},"min":{"1min":0.40699999999999997,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.77600000000000002,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.53100000000000003,"15min":0.52700000000000002},"min":{"1min":0.38400000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.59399999999999997}]},{"osd":6,"last update":"Wed Sep 11 08:08:56 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.55500000000000005,"15min":0.55900000000000005},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.629},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.59899999999999998,"15min":0.60099999999999998},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.58799999999999997}]},{"osd":7,"last update":"Wed Sep 11 08:09:21 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61599999999999999,"15min":0.62},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.78800000000000003,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.56399999999999995,"15min":0.56499999999999995},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.54700000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738428,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298552,"kb_used_data":1200,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446600,"statfs":{"total":95995035648,"available":95689318400,"internally_reserved":0,"allocated":1228800,"data_stored":878646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 11 08:09:04 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.34799999999999998,"15min":0.317},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":0.53400000000000003,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.35899999999999999,"15min":0.33800000000000002},"min":{"1min":0.219,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.47999999999999998,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.25600000000000001}]},{"osd":2,"last update":"Wed Sep 11 08:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.34699999999999998},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.48799999999999999,"5min":0.53000000000000003,"15min":0.53000000000000003},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.378,"5min":0.378,"15min":0.36899999999999999},"min":{"1min":0.26600000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.56299999999999994,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.32100000000000001}]},{"osd":3,"last update":"Wed Sep 11 08:08:51 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.44,"15min":0.44},"min":{"1min":0.27600000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.52300000000000002,"5min":0.62,"15min":0.62},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.42499999999999999,"15min":0.42599999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.52800000000000002,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.41399999999999998}]},{"osd":4,"last update":"Wed Sep 11 08:09:09 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55100000000000005,"15min":0.54700000000000004},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.56799999999999995,"15min":0.57399999999999995},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.71699999999999997,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.40400000000000003}]},{"osd":5,"last update":"Wed Sep 11 08:08:36 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.58499999999999996,"15min":0.58299999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.748,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.55800000000000005,"15min":0.55300000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.36799999999999999}]},{"osd":6,"last update":"Wed Sep 11 08:08:59 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.60899999999999999,"15min":0.61699999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.75600000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.60299999999999998,"15min":0.57099999999999995},"min":{"1min":0.433,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":4.7069999999999999,"5min":4.7069999999999999,"15min":4.7069999999999999},"last":0.46300000000000002}]},{"osd":7,"last update":"Wed Sep 11 08:09:21 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.59799999999999998,"15min":0.60999999999999999},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.68200000000000005,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.56200000000000006,"15min":0.57099999999999995},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.69399999999999995,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.434}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T08:09:26.109 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-11T08:09:26.110 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd dump 2024-09-11T08:09:26.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:26 smithi150 ceph-mon[29276]: 6.0 scrub starts 2024-09-11T08:09:26.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:26 smithi150 ceph-mon[29276]: 6.0 scrub ok 2024-09-11T08:09:26.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:26 smithi150 ceph-mon[39612]: 6.0 scrub starts 2024-09-11T08:09:26.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:26 smithi150 ceph-mon[39612]: 6.0 scrub ok 2024-09-11T08:09:27.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:26 smithi181 ceph-mon[32701]: 6.0 scrub starts 2024-09-11T08:09:27.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:26 smithi181 ceph-mon[32701]: 6.0 scrub ok 2024-09-11T08:09:27.572 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:09:27.852 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:27 smithi150 ceph-mon[29276]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:09:27.852 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:09:27 smithi150 systemd[1]: Stopping Ceph alertmanager.a for c1bddff4-7013-11ef-bcea-c7b262605968... 2024-09-11T08:09:27.853 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:27 smithi150 ceph-mon[39612]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:09:28.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:27 smithi181 ceph-mon[32701]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:09:28.073 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:09:27 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:09:28.239 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:09:28 smithi150 podman[74735]: 2024-09-11 08:09:28.029661519 +0000 UTC m=+0.453388667 container died 5acf50c31d731d66da9c3c5ca65359a4b3c01b335247c6beeb5ce8ec8a9b0ce1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9, name=sad_snyder, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, ceph=True, org.label-schema.build-date=20240731) 2024-09-11T08:09:28.971 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:09:28 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-alertmanager-a[72111]: ts=2024-09-11T08:09:28.846Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-11T08:09:28.971 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:28 smithi150 ceph-mon[29276]: 6.6 scrub starts 2024-09-11T08:09:28.971 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:28 smithi150 ceph-mon[29276]: 6.6 scrub ok 2024-09-11T08:09:28.971 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:28 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:09:28.971 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:28 smithi150 ceph-mon[29276]: 6.16 scrub starts 2024-09-11T08:09:28.971 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:28 smithi150 ceph-mon[29276]: 6.16 scrub ok 2024-09-11T08:09:28.972 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:28 smithi150 ceph-mon[39612]: 6.6 scrub starts 2024-09-11T08:09:28.972 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:28 smithi150 ceph-mon[39612]: 6.6 scrub ok 2024-09-11T08:09:28.972 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:28 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:09:28.972 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:28 smithi150 ceph-mon[39612]: 6.16 scrub starts 2024-09-11T08:09:28.972 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:28 smithi150 ceph-mon[39612]: 6.16 scrub ok 2024-09-11T08:09:29.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:28 smithi181 ceph-mon[32701]: 6.6 scrub starts 2024-09-11T08:09:29.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:28 smithi181 ceph-mon[32701]: 6.6 scrub ok 2024-09-11T08:09:29.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:28 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:09:29.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:28 smithi181 ceph-mon[32701]: 6.16 scrub starts 2024-09-11T08:09:29.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:28 smithi181 ceph-mon[32701]: 6.16 scrub ok 2024-09-11T08:09:29.238 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:09:29 smithi150 podman[74794]: 2024-09-11 08:09:29.095654366 +0000 UTC m=+0.897153300 container died 6af74a554f48afed6100272abc1c40bfd7c00cbf6577f093ddd620284b580f96 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-11T08:09:29.666 INFO:teuthology.orchestra.run.smithi150.stdout:epoch 86 2024-09-11T08:09:29.666 INFO:teuthology.orchestra.run.smithi150.stdout:fsid c1bddff4-7013-11ef-bcea-c7b262605968 2024-09-11T08:09:29.667 INFO:teuthology.orchestra.run.smithi150.stdout:created 2024-09-11T08:02:15.344618+0000 2024-09-11T08:09:29.667 INFO:teuthology.orchestra.run.smithi150.stdout:modified 2024-09-11T08:09:07.868684+0000 2024-09-11T08:09:29.667 INFO:teuthology.orchestra.run.smithi150.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-11T08:09:29.667 INFO:teuthology.orchestra.run.smithi150.stdout:crush_version 18 2024-09-11T08:09:29.667 INFO:teuthology.orchestra.run.smithi150.stdout:full_ratio 0.95 2024-09-11T08:09:29.667 INFO:teuthology.orchestra.run.smithi150.stdout:backfillfull_ratio 0.9 2024-09-11T08:09:29.667 INFO:teuthology.orchestra.run.smithi150.stdout:nearfull_ratio 0.85 2024-09-11T08:09:29.667 INFO:teuthology.orchestra.run.smithi150.stdout:require_min_compat_client luminous 2024-09-11T08:09:29.667 INFO:teuthology.orchestra.run.smithi150.stdout:min_compat_client luminous 2024-09-11T08:09:29.667 INFO:teuthology.orchestra.run.smithi150.stdout:require_osd_release quincy 2024-09-11T08:09:29.667 INFO:teuthology.orchestra.run.smithi150.stdout:stretch_mode_enabled false 2024-09-11T08:09:29.668 INFO:teuthology.orchestra.run.smithi150.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr 2024-09-11T08:09:29.668 INFO:teuthology.orchestra.run.smithi150.stdout:pool 2 'datapool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 3 pgp_num 3 autoscale_mode off last_change 63 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd 2024-09-11T08:09:29.668 INFO:teuthology.orchestra.run.smithi150.stdout:pool 3 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 60 flags hashpspool stripe_width 0 application rgw 2024-09-11T08:09:29.668 INFO:teuthology.orchestra.run.smithi150.stdout:pool 4 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 62 flags hashpspool stripe_width 0 application rgw 2024-09-11T08:09:29.668 INFO:teuthology.orchestra.run.smithi150.stdout:pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 64 flags hashpspool stripe_width 0 application rgw 2024-09-11T08:09:29.668 INFO:teuthology.orchestra.run.smithi150.stdout:pool 6 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 66 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw 2024-09-11T08:09:29.668 INFO:teuthology.orchestra.run.smithi150.stdout:max_osd 8 2024-09-11T08:09:29.668 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0 up in weight 1 up_from 8 up_thru 64 down_at 0 last_clean_interval [0,0) v2:172.21.15.150:6801/334255279 v2:172.21.15.150:6802/334255279 exists,up 37b63db9-ed2a-4701-b63d-03ff9885cc0e 2024-09-11T08:09:29.669 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1 up in weight 1 up_from 13 up_thru 64 down_at 0 last_clean_interval [0,0) v2:172.21.15.150:6805/1129510619 v2:172.21.15.150:6806/1129510619 exists,up 80378442-02bc-48aa-b8ca-6113a4092079 2024-09-11T08:09:29.669 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2 up in weight 1 up_from 18 up_thru 64 down_at 0 last_clean_interval [0,0) v2:172.21.15.150:6809/719975422 v2:172.21.15.150:6810/719975422 exists,up 0955bb92-36a0-4e9a-98e1-b3a5af89f6d2 2024-09-11T08:09:29.669 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3 up in weight 1 up_from 25 up_thru 85 down_at 0 last_clean_interval [0,0) v2:172.21.15.150:6813/3101032500 v2:172.21.15.150:6814/3101032500 exists,up 1d04d4d3-62e8-4460-8a07-77f7e5c62929 2024-09-11T08:09:29.669 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4 up in weight 1 up_from 31 up_thru 85 down_at 0 last_clean_interval [0,0) v2:172.21.15.181:6800/2567452825 v2:172.21.15.181:6801/2567452825 exists,up c5321a3c-d2f0-4440-8475-2c111d261139 2024-09-11T08:09:29.669 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5 up in weight 1 up_from 38 up_thru 64 down_at 0 last_clean_interval [0,0) v2:172.21.15.181:6804/2043435508 v2:172.21.15.181:6805/2043435508 exists,up 1262b2fe-f253-4a26-bfd4-c9dffb697c77 2024-09-11T08:09:29.669 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6 up in weight 1 up_from 45 up_thru 62 down_at 0 last_clean_interval [0,0) v2:172.21.15.181:6808/1789880331 v2:172.21.15.181:6809/1789880331 exists,up 7facca22-b3f1-4c20-833e-e3db798a08c4 2024-09-11T08:09:29.669 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7 up in weight 1 up_from 53 up_thru 64 down_at 0 last_clean_interval [0,0) v2:172.21.15.181:6812/3218153424 v2:172.21.15.181:6813/3218153424 exists,up 6ac1b06a-79bd-43ca-b91e-fcb0f4033a34 2024-09-11T08:09:29.669 INFO:teuthology.orchestra.run.smithi150.stdout:pg_upmap_items 4.d [1,5] 2024-09-11T08:09:29.669 INFO:teuthology.orchestra.run.smithi150.stdout:pg_upmap_items 4.14 [1,2] 2024-09-11T08:09:29.669 INFO:teuthology.orchestra.run.smithi150.stdout:pg_upmap_items 4.1d [1,2] 2024-09-11T08:09:29.669 INFO:teuthology.orchestra.run.smithi150.stdout:blocklist 172.21.15.150:0/2620931070 expires 2024-09-12T08:08:06.649692+0000 2024-09-11T08:09:29.670 INFO:teuthology.orchestra.run.smithi150.stdout:blocklist 172.21.15.150:0/1073581409 expires 2024-09-12T08:02:45.986151+0000 2024-09-11T08:09:29.670 INFO:teuthology.orchestra.run.smithi150.stdout:blocklist 172.21.15.150:0/2363649697 expires 2024-09-12T08:08:06.649692+0000 2024-09-11T08:09:29.670 INFO:teuthology.orchestra.run.smithi150.stdout:blocklist 172.21.15.150:6800/603451071 expires 2024-09-12T08:02:45.986151+0000 2024-09-11T08:09:29.670 INFO:teuthology.orchestra.run.smithi150.stdout:blocklist 172.21.15.150:0/1614413548 expires 2024-09-12T08:02:45.986151+0000 2024-09-11T08:09:29.670 INFO:teuthology.orchestra.run.smithi150.stdout:blocklist 172.21.15.150:0/3673218166 expires 2024-09-12T08:03:08.740828+0000 2024-09-11T08:09:29.670 INFO:teuthology.orchestra.run.smithi150.stdout:blocklist 172.21.15.150:6800/556498589 expires 2024-09-12T08:03:08.740828+0000 2024-09-11T08:09:29.670 INFO:teuthology.orchestra.run.smithi150.stdout:blocklist 172.21.15.150:0/3666733994 expires 2024-09-12T08:03:08.740828+0000 2024-09-11T08:09:29.670 INFO:teuthology.orchestra.run.smithi150.stdout:blocklist 172.21.15.150:0/3271479010 expires 2024-09-12T08:03:08.740828+0000 2024-09-11T08:09:29.670 INFO:teuthology.orchestra.run.smithi150.stdout:blocklist 172.21.15.150:0/3185731090 expires 2024-09-12T08:08:06.649692+0000 2024-09-11T08:09:29.671 INFO:teuthology.orchestra.run.smithi150.stdout:blocklist 172.21.15.150:0/2539412006 expires 2024-09-12T08:08:06.649692+0000 2024-09-11T08:09:29.671 INFO:teuthology.orchestra.run.smithi150.stdout:blocklist 172.21.15.150:0/2740222408 expires 2024-09-12T08:02:45.986151+0000 2024-09-11T08:09:29.671 INFO:teuthology.orchestra.run.smithi150.stdout:blocklist 172.21.15.150:6800/127856313 expires 2024-09-12T08:08:06.649692+0000 2024-09-11T08:09:29.671 INFO:teuthology.orchestra.run.smithi150.stdout:blocklist 172.21.15.150:0/2375121603 expires 2024-09-12T08:08:06.649692+0000 2024-09-11T08:09:29.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:29 smithi150 ceph-mon[29276]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:09:29.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:29 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/500102935' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-11T08:09:29.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:29 smithi150 ceph-mon[39612]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:09:29.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:29 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/500102935' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-11T08:09:29.990 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:09:29 smithi150 podman[74794]: 2024-09-11 08:09:29.842589743 +0000 UTC m=+1.644088631 container cleanup 6af74a554f48afed6100272abc1c40bfd7c00cbf6577f093ddd620284b580f96 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-11T08:09:29.990 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:09:29 smithi150 bash[74794]: ceph-c1bddff4-7013-11ef-bcea-c7b262605968-alertmanager-a 2024-09-11T08:09:30.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:29 smithi181 ceph-mon[32701]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:09:30.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:29 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/500102935' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-11T08:09:30.539 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:09:30 smithi150 podman[74830]: 2024-09-11 08:09:30.278302094 +0000 UTC m=+1.411599453 container remove 6af74a554f48afed6100272abc1c40bfd7c00cbf6577f093ddd620284b580f96 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-11T08:09:30.540 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:09:30 smithi150 podman[74830]: 2024-09-11 08:09:30.387608984 +0000 UTC m=+1.520906339 volume remove 7ccd31928cc244156dab9e44a5267acbdea4b7d35440473dce2520795f0a2eab 2024-09-11T08:09:30.849 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 64 down_at 0 last_clean_interval [0,0) v2:172.21.15.150:6801/334255279 v2:172.21.15.150:6802/334255279 exists,up 37b63db9-ed2a-4701-b63d-03ff9885cc0e', 'osd.1 up in weight 1 up_from 13 up_thru 64 down_at 0 last_clean_interval [0,0) v2:172.21.15.150:6805/1129510619 v2:172.21.15.150:6806/1129510619 exists,up 80378442-02bc-48aa-b8ca-6113a4092079', 'osd.2 up in weight 1 up_from 18 up_thru 64 down_at 0 last_clean_interval [0,0) v2:172.21.15.150:6809/719975422 v2:172.21.15.150:6810/719975422 exists,up 0955bb92-36a0-4e9a-98e1-b3a5af89f6d2', 'osd.3 up in weight 1 up_from 25 up_thru 85 down_at 0 last_clean_interval [0,0) v2:172.21.15.150:6813/3101032500 v2:172.21.15.150:6814/3101032500 exists,up 1d04d4d3-62e8-4460-8a07-77f7e5c62929', 'osd.4 up in weight 1 up_from 31 up_thru 85 down_at 0 last_clean_interval [0,0) v2:172.21.15.181:6800/2567452825 v2:172.21.15.181:6801/2567452825 exists,up c5321a3c-d2f0-4440-8475-2c111d261139', 'osd.5 up in weight 1 up_from 38 up_thru 64 down_at 0 last_clean_interval [0,0) v2:172.21.15.181:6804/2043435508 v2:172.21.15.181:6805/2043435508 exists,up 1262b2fe-f253-4a26-bfd4-c9dffb697c77', 'osd.6 up in weight 1 up_from 45 up_thru 62 down_at 0 last_clean_interval [0,0) v2:172.21.15.181:6808/1789880331 v2:172.21.15.181:6809/1789880331 exists,up 7facca22-b3f1-4c20-833e-e3db798a08c4', 'osd.7 up in weight 1 up_from 53 up_thru 64 down_at 0 last_clean_interval [0,0) v2:172.21.15.181:6812/3218153424 v2:172.21.15.181:6813/3218153424 exists,up 6ac1b06a-79bd-43ca-b91e-fcb0f4033a34'] 2024-09-11T08:09:30.849 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon mon.a config show 2024-09-11T08:09:31.143 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:09:31.233 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:09:31 smithi150 systemd[1]: ceph-c1bddff4-7013-11ef-bcea-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-11T08:09:31.233 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:09:31 smithi150 systemd[1]: Stopped Ceph alertmanager.a for c1bddff4-7013-11ef-bcea-c7b262605968. 2024-09-11T08:09:31.233 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:09:31 smithi150 systemd[1]: Starting Ceph alertmanager.a for c1bddff4-7013-11ef-bcea-c7b262605968... 2024-09-11T08:09:31.485 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:09:31 smithi150 podman[75022]: 2024-09-11 08:09:31.232853344 +0000 UTC m=+0.101082553 volume create e4279716c2bf91d2fae6197b032a9659fd7be03dbc35fc9d6c845bd64c2d4ee3 2024-09-11T08:09:31.485 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:09:31 smithi150 podman[75022]: 2024-09-11 08:09:31.149597237 +0000 UTC m=+0.017826447 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-09-11T08:09:31.485 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:09:31 smithi150 podman[75022]: 2024-09-11 08:09:31.308372589 +0000 UTC m=+0.176601839 container create 908683d933675cb66b516cec0428b4954ea70ed995d24014a63e0fab412ad320 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-11T08:09:31.838 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:31 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:09:31.839 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:31 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:09:31.839 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:31 smithi150 ceph-mon[29276]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:09:31.839 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:31 smithi150 ceph-mon[29276]: osdmap e87: 8 total, 8 up, 8 in 2024-09-11T08:09:31.839 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:31 smithi150 ceph-mon[29276]: 4.11 scrub starts 2024-09-11T08:09:31.839 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:31 smithi150 ceph-mon[29276]: 4.11 scrub ok 2024-09-11T08:09:31.839 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:31 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:09:31.839 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:31 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:09:31.839 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:31 smithi150 ceph-mon[39612]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:09:31.839 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:31 smithi150 ceph-mon[39612]: osdmap e87: 8 total, 8 up, 8 in 2024-09-11T08:09:31.839 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:31 smithi150 ceph-mon[39612]: 4.11 scrub starts 2024-09-11T08:09:31.840 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:31 smithi150 ceph-mon[39612]: 4.11 scrub ok 2024-09-11T08:09:32.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:31 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:09:32.092 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:31 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:09:32.093 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:31 smithi181 ceph-mon[32701]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:09:32.093 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:31 smithi181 ceph-mon[32701]: osdmap e87: 8 total, 8 up, 8 in 2024-09-11T08:09:32.093 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:31 smithi181 ceph-mon[32701]: 4.11 scrub starts 2024-09-11T08:09:32.093 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:31 smithi181 ceph-mon[32701]: 4.11 scrub ok 2024-09-11T08:09:32.098 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:09:31 smithi150 podman[75022]: 2024-09-11 08:09:31.838312648 +0000 UTC m=+0.706541863 container init 908683d933675cb66b516cec0428b4954ea70ed995d24014a63e0fab412ad320 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-11T08:09:32.099 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:09:31 smithi150 podman[75022]: 2024-09-11 08:09:31.841392097 +0000 UTC m=+0.709621306 container start 908683d933675cb66b516cec0428b4954ea70ed995d24014a63e0fab412ad320 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-11T08:09:32.099 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:09:31 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-alertmanager-a[75052]: ts=2024-09-11T08:09:31.854Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-09-11T08:09:32.099 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:09:31 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-alertmanager-a[75052]: ts=2024-09-11T08:09:31.854Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-09-11T08:09:32.099 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:09:31 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-alertmanager-a[75052]: ts=2024-09-11T08:09:31.855Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.150 port=9094 2024-09-11T08:09:32.099 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:09:31 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-alertmanager-a[75052]: ts=2024-09-11T08:09:31.856Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-11T08:09:32.099 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:09:31 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-alertmanager-a[75052]: ts=2024-09-11T08:09:31.878Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-11T08:09:32.099 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:09:31 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-alertmanager-a[75052]: ts=2024-09-11T08:09:31.879Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-11T08:09:32.099 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:09:31 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-alertmanager-a[75052]: ts=2024-09-11T08:09:31.881Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-09-11T08:09:32.099 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:09:31 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-alertmanager-a[75052]: ts=2024-09-11T08:09:31.881Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-11T08:09:32.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:32 smithi150 ceph-mon[29276]: 6.1 scrub starts 2024-09-11T08:09:32.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:32 smithi150 ceph-mon[29276]: 6.1 scrub ok 2024-09-11T08:09:32.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:32 smithi150 ceph-mon[29276]: 5.18 scrub starts 2024-09-11T08:09:32.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:32 smithi150 ceph-mon[29276]: 5.18 scrub ok 2024-09-11T08:09:32.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:32 smithi150 ceph-mon[39612]: 6.1 scrub starts 2024-09-11T08:09:32.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:32 smithi150 ceph-mon[39612]: 6.1 scrub ok 2024-09-11T08:09:32.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:32 smithi150 ceph-mon[39612]: 5.18 scrub starts 2024-09-11T08:09:32.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:32 smithi150 ceph-mon[39612]: 5.18 scrub ok 2024-09-11T08:09:33.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:32 smithi181 ceph-mon[32701]: 6.1 scrub starts 2024-09-11T08:09:33.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:32 smithi181 ceph-mon[32701]: 6.1 scrub ok 2024-09-11T08:09:33.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:32 smithi181 ceph-mon[32701]: 5.18 scrub starts 2024-09-11T08:09:33.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:32 smithi181 ceph-mon[32701]: 5.18 scrub ok 2024-09-11T08:09:33.372 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:09:33 smithi150 bash[75022]: 908683d933675cb66b516cec0428b4954ea70ed995d24014a63e0fab412ad320 2024-09-11T08:09:33.372 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:09:33 smithi150 systemd[1]: Started Ceph alertmanager.a for c1bddff4-7013-11ef-bcea-c7b262605968. 2024-09-11T08:09:33.402 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-11T08:09:33.402 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "mon.a", 2024-09-11T08:09:33.402 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster": "ceph", 2024-09-11T08:09:33.402 INFO:teuthology.orchestra.run.smithi150.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-11T08:09:33.402 INFO:teuthology.orchestra.run.smithi150.stdout: "admin_socket_mode": "", 2024-09-11T08:09:33.403 INFO:teuthology.orchestra.run.smithi150.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-09-11T08:09:33.403 INFO:teuthology.orchestra.run.smithi150.stdout: "auth_client_required": "cephx, none", 2024-09-11T08:09:33.403 INFO:teuthology.orchestra.run.smithi150.stdout: "auth_cluster_required": "cephx", 2024-09-11T08:09:33.403 INFO:teuthology.orchestra.run.smithi150.stdout: "auth_debug": "true", 2024-09-11T08:09:33.403 INFO:teuthology.orchestra.run.smithi150.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-11T08:09:33.403 INFO:teuthology.orchestra.run.smithi150.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-11T08:09:33.403 INFO:teuthology.orchestra.run.smithi150.stdout: "auth_service_required": "cephx", 2024-09-11T08:09:33.403 INFO:teuthology.orchestra.run.smithi150.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-11T08:09:33.403 INFO:teuthology.orchestra.run.smithi150.stdout: "auth_supported": "", 2024-09-11T08:09:33.403 INFO:teuthology.orchestra.run.smithi150.stdout: "bdev_aio": "true", 2024-09-11T08:09:33.404 INFO:teuthology.orchestra.run.smithi150.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-11T08:09:33.404 INFO:teuthology.orchestra.run.smithi150.stdout: "bdev_aio_poll_ms": "250", 2024-09-11T08:09:33.404 INFO:teuthology.orchestra.run.smithi150.stdout: "bdev_aio_reap_max": "16", 2024-09-11T08:09:33.404 INFO:teuthology.orchestra.run.smithi150.stdout: "bdev_async_discard": "false", 2024-09-11T08:09:33.404 INFO:teuthology.orchestra.run.smithi150.stdout: "bdev_block_size": "4096", 2024-09-11T08:09:33.404 INFO:teuthology.orchestra.run.smithi150.stdout: "bdev_debug_aio": "false", 2024-09-11T08:09:33.404 INFO:teuthology.orchestra.run.smithi150.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-11T08:09:33.404 INFO:teuthology.orchestra.run.smithi150.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-11T08:09:33.404 INFO:teuthology.orchestra.run.smithi150.stdout: "bdev_debug_inflight_ios": "false", 2024-09-11T08:09:33.404 INFO:teuthology.orchestra.run.smithi150.stdout: "bdev_enable_discard": "false", 2024-09-11T08:09:33.404 INFO:teuthology.orchestra.run.smithi150.stdout: "bdev_flock_retry": "3", 2024-09-11T08:09:33.404 INFO:teuthology.orchestra.run.smithi150.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-11T08:09:33.404 INFO:teuthology.orchestra.run.smithi150.stdout: "bdev_inject_crash": "0", 2024-09-11T08:09:33.405 INFO:teuthology.orchestra.run.smithi150.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-11T08:09:33.405 INFO:teuthology.orchestra.run.smithi150.stdout: "bdev_ioring": "false", 2024-09-11T08:09:33.405 INFO:teuthology.orchestra.run.smithi150.stdout: "bdev_ioring_hipri": "false", 2024-09-11T08:09:33.405 INFO:teuthology.orchestra.run.smithi150.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-11T08:09:33.405 INFO:teuthology.orchestra.run.smithi150.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-11T08:09:33.405 INFO:teuthology.orchestra.run.smithi150.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-11T08:09:33.405 INFO:teuthology.orchestra.run.smithi150.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-11T08:09:33.405 INFO:teuthology.orchestra.run.smithi150.stdout: "bdev_type": "", 2024-09-11T08:09:33.405 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_alloc_size": "1048576", 2024-09-11T08:09:33.405 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_allocator": "hybrid", 2024-09-11T08:09:33.405 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_buffered_io": "true", 2024-09-11T08:09:33.406 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_check_for_zeros": "false", 2024-09-11T08:09:33.406 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-11T08:09:33.406 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-11T08:09:33.406 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_compact_log_sync": "false", 2024-09-11T08:09:33.406 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-11T08:09:33.406 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-11T08:09:33.406 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-11T08:09:33.406 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-11T08:09:33.406 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_max_log_runway": "4194304", 2024-09-11T08:09:33.406 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_max_prefetch": "1048576", 2024-09-11T08:09:33.406 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_min_flush_size": "524288", 2024-09-11T08:09:33.406 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_min_log_runway": "1048576", 2024-09-11T08:09:33.406 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_replay_recovery": "false", 2024-09-11T08:09:33.406 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-11T08:09:33.407 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-11T08:09:33.407 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_sync_write": "false", 2024-09-11T08:09:33.407 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-11T08:09:33.407 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-11T08:09:33.407 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-11T08:09:33.407 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_allocation_from_file": "true", 2024-09-11T08:09:33.407 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_allocator": "hybrid", 2024-09-11T08:09:33.407 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_async_db_compaction": "true", 2024-09-11T08:09:33.407 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-11T08:09:33.407 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-11T08:09:33.407 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-11T08:09:33.407 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-11T08:09:33.407 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-11T08:09:33.408 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-11T08:09:33.408 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-11T08:09:33.408 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_block_create": "true", 2024-09-11T08:09:33.408 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_block_db_create": "false", 2024-09-11T08:09:33.408 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_block_db_path": "", 2024-09-11T08:09:33.408 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_block_db_size": "0", 2024-09-11T08:09:33.408 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_block_path": "", 2024-09-11T08:09:33.408 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_block_preallocate_file": "false", 2024-09-11T08:09:33.408 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_block_size": "107374182400", 2024-09-11T08:09:33.408 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_block_wal_create": "false", 2024-09-11T08:09:33.408 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_block_wal_path": "", 2024-09-11T08:09:33.408 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_block_wal_size": "100663296", 2024-09-11T08:09:33.409 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_bluefs": "true", 2024-09-11T08:09:33.409 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-11T08:09:33.409 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-11T08:09:33.409 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-11T08:09:33.409 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-11T08:09:33.409 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-11T08:09:33.409 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-11T08:09:33.409 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-11T08:09:33.409 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-11T08:09:33.409 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_autotune": "true", 2024-09-11T08:09:33.409 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-11T08:09:33.409 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-11T08:09:33.409 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-11T08:09:33.410 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-11T08:09:33.410 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_size": "0", 2024-09-11T08:09:33.410 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-11T08:09:33.410 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-11T08:09:33.410 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-11T08:09:33.410 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-11T08:09:33.410 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_type": "2q", 2024-09-11T08:09:33.410 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-11T08:09:33.410 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_clone_cow": "true", 2024-09-11T08:09:33.410 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-11T08:09:33.410 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-11T08:09:33.410 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-11T08:09:33.410 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-11T08:09:33.410 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-11T08:09:33.411 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-11T08:09:33.411 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-11T08:09:33.411 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_compression_mode": "none", 2024-09-11T08:09:33.411 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-11T08:09:33.411 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_csum_type": "crc32c", 2024-09-11T08:09:33.411 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-11T08:09:33.411 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_debug_freelist": "false", 2024-09-11T08:09:33.411 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-11T08:09:33.411 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-11T08:09:33.412 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_debug_inject_bug21040": "false", 2024-09-11T08:09:33.412 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-11T08:09:33.412 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-11T08:09:33.412 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-11T08:09:33.412 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-11T08:09:33.412 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-11T08:09:33.412 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-11T08:09:33.412 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-11T08:09:33.412 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_debug_prefill": "0.000000", 2024-09-11T08:09:33.412 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-11T08:09:33.412 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-11T08:09:33.412 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-11T08:09:33.412 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_debug_small_allocations": "0", 2024-09-11T08:09:33.412 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-11T08:09:33.412 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_default_buffered_read": "true", 2024-09-11T08:09:33.413 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_default_buffered_write": "false", 2024-09-11T08:09:33.413 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-11T08:09:33.413 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-11T08:09:33.413 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-11T08:09:33.413 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-11T08:09:33.413 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-11T08:09:33.413 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-11T08:09:33.413 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-11T08:09:33.413 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-11T08:09:33.413 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_fail_eio": "false", 2024-09-11T08:09:33.413 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-11T08:09:33.413 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-11T08:09:33.413 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-11T08:09:33.413 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-11T08:09:33.414 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-11T08:09:33.414 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-11T08:09:33.414 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_fsck_on_mount": "false", 2024-09-11T08:09:33.414 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-11T08:09:33.414 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_fsck_on_umount": "false", 2024-09-11T08:09:33.414 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-11T08:09:33.414 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-11T08:09:33.414 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-11T08:09:33.414 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-11T08:09:33.415 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-11T08:09:33.415 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-11T08:09:33.415 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-11T08:09:33.415 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-11T08:09:33.415 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_ignore_data_csum": "false", 2024-09-11T08:09:33.415 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-11T08:09:33.415 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-11T08:09:33.415 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-11T08:09:33.415 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-11T08:09:33.415 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_log_op_age": "5.000000", 2024-09-11T08:09:33.415 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_max_alloc_size": "0", 2024-09-11T08:09:33.415 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_max_blob_size": "0", 2024-09-11T08:09:33.415 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-11T08:09:33.415 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-11T08:09:33.415 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-11T08:09:33.416 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_max_deferred_txc": "32", 2024-09-11T08:09:33.416 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_min_alloc_size": "0", 2024-09-11T08:09:33.416 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-11T08:09:33.416 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-11T08:09:33.416 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_nid_prealloc": "1024", 2024-09-11T08:09:33.416 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-11T08:09:33.416 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-11T08:09:33.416 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-11T08:09:33.416 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-11T08:09:33.416 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_retry_disk_reads": "3", 2024-09-11T08:09:33.416 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_rocksdb_cf": "true", 2024-09-11T08:09:33.416 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L P", 2024-09-11T08:09:33.418 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_rocksdb_options": "compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824", 2024-09-11T08:09:33.418 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-11T08:09:33.418 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-11T08:09:33.418 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-11T08:09:33.418 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-11T08:09:33.418 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_spdk_mem": "512", 2024-09-11T08:09:33.418 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-11T08:09:33.418 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-11T08:09:33.418 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-11T08:09:33.418 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-11T08:09:33.418 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-11T08:09:33.419 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-11T08:09:33.419 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-11T08:09:33.419 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_tracing": "false", 2024-09-11T08:09:33.419 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-11T08:09:33.419 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-11T08:09:33.419 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-11T08:09:33.419 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-11T08:09:33.419 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-11T08:09:33.419 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-11T08:09:33.419 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-11T08:09:33.419 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-11T08:09:33.419 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-11T08:09:33.419 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_zero_block_detection": "false", 2024-09-11T08:09:33.420 INFO:teuthology.orchestra.run.smithi150.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-11T08:09:33.420 INFO:teuthology.orchestra.run.smithi150.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-11T08:09:33.420 INFO:teuthology.orchestra.run.smithi150.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-11T08:09:33.420 INFO:teuthology.orchestra.run.smithi150.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-11T08:09:33.420 INFO:teuthology.orchestra.run.smithi150.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-11T08:09:33.420 INFO:teuthology.orchestra.run.smithi150.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-11T08:09:33.420 INFO:teuthology.orchestra.run.smithi150.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-11T08:09:33.420 INFO:teuthology.orchestra.run.smithi150.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-11T08:09:33.420 INFO:teuthology.orchestra.run.smithi150.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-11T08:09:33.420 INFO:teuthology.orchestra.run.smithi150.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-11T08:09:33.420 INFO:teuthology.orchestra.run.smithi150.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-11T08:09:33.420 INFO:teuthology.orchestra.run.smithi150.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-11T08:09:33.420 INFO:teuthology.orchestra.run.smithi150.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-11T08:09:33.420 INFO:teuthology.orchestra.run.smithi150.stdout: "cephx_cluster_require_signatures": "false", 2024-09-11T08:09:33.421 INFO:teuthology.orchestra.run.smithi150.stdout: "cephx_cluster_require_version": "2", 2024-09-11T08:09:33.421 INFO:teuthology.orchestra.run.smithi150.stdout: "cephx_require_signatures": "false", 2024-09-11T08:09:33.421 INFO:teuthology.orchestra.run.smithi150.stdout: "cephx_require_version": "2", 2024-09-11T08:09:33.421 INFO:teuthology.orchestra.run.smithi150.stdout: "cephx_service_require_signatures": "false", 2024-09-11T08:09:33.421 INFO:teuthology.orchestra.run.smithi150.stdout: "cephx_service_require_version": "2", 2024-09-11T08:09:33.421 INFO:teuthology.orchestra.run.smithi150.stdout: "cephx_sign_messages": "true", 2024-09-11T08:09:33.421 INFO:teuthology.orchestra.run.smithi150.stdout: "chdir": "", 2024-09-11T08:09:33.421 INFO:teuthology.orchestra.run.smithi150.stdout: "client_acl_type": "", 2024-09-11T08:09:33.421 INFO:teuthology.orchestra.run.smithi150.stdout: "client_asio_thread_count": "2", 2024-09-11T08:09:33.422 INFO:teuthology.orchestra.run.smithi150.stdout: "client_cache_mid": "0.750000", 2024-09-11T08:09:33.422 INFO:teuthology.orchestra.run.smithi150.stdout: "client_cache_size": "16384", 2024-09-11T08:09:33.422 INFO:teuthology.orchestra.run.smithi150.stdout: "client_caps_release_delay": "5", 2024-09-11T08:09:33.422 INFO:teuthology.orchestra.run.smithi150.stdout: "client_check_pool_perm": "true", 2024-09-11T08:09:33.422 INFO:teuthology.orchestra.run.smithi150.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-11T08:09:33.422 INFO:teuthology.orchestra.run.smithi150.stdout: "client_debug_force_sync_read": "false", 2024-09-11T08:09:33.422 INFO:teuthology.orchestra.run.smithi150.stdout: "client_debug_getattr_caps": "false", 2024-09-11T08:09:33.422 INFO:teuthology.orchestra.run.smithi150.stdout: "client_debug_inject_features": "", 2024-09-11T08:09:33.422 INFO:teuthology.orchestra.run.smithi150.stdout: "client_debug_inject_tick_delay": "0", 2024-09-11T08:09:33.422 INFO:teuthology.orchestra.run.smithi150.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-11T08:09:33.422 INFO:teuthology.orchestra.run.smithi150.stdout: "client_die_on_failed_remount": "false", 2024-09-11T08:09:33.422 INFO:teuthology.orchestra.run.smithi150.stdout: "client_dirsize_rbytes": "true", 2024-09-11T08:09:33.422 INFO:teuthology.orchestra.run.smithi150.stdout: "client_force_lazyio": "false", 2024-09-11T08:09:33.422 INFO:teuthology.orchestra.run.smithi150.stdout: "client_fs": "", 2024-09-11T08:09:33.423 INFO:teuthology.orchestra.run.smithi150.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-11T08:09:33.423 INFO:teuthology.orchestra.run.smithi150.stdout: "client_inject_release_failure": "false", 2024-09-11T08:09:33.423 INFO:teuthology.orchestra.run.smithi150.stdout: "client_max_inline_size": "4096", 2024-09-11T08:09:33.423 INFO:teuthology.orchestra.run.smithi150.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-11T08:09:33.423 INFO:teuthology.orchestra.run.smithi150.stdout: "client_mds_namespace": "", 2024-09-11T08:09:33.423 INFO:teuthology.orchestra.run.smithi150.stdout: "client_metadata": "", 2024-09-11T08:09:33.423 INFO:teuthology.orchestra.run.smithi150.stdout: "client_mount_gid": "-1", 2024-09-11T08:09:33.423 INFO:teuthology.orchestra.run.smithi150.stdout: "client_mount_timeout": "300", 2024-09-11T08:09:33.423 INFO:teuthology.orchestra.run.smithi150.stdout: "client_mount_uid": "-1", 2024-09-11T08:09:33.423 INFO:teuthology.orchestra.run.smithi150.stdout: "client_mountpoint": "/", 2024-09-11T08:09:33.423 INFO:teuthology.orchestra.run.smithi150.stdout: "client_notify_timeout": "10", 2024-09-11T08:09:33.423 INFO:teuthology.orchestra.run.smithi150.stdout: "client_oc": "true", 2024-09-11T08:09:33.424 INFO:teuthology.orchestra.run.smithi150.stdout: "client_oc_max_dirty": "104857600", 2024-09-11T08:09:33.424 INFO:teuthology.orchestra.run.smithi150.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-11T08:09:33.424 INFO:teuthology.orchestra.run.smithi150.stdout: "client_oc_max_objects": "1000", 2024-09-11T08:09:33.424 INFO:teuthology.orchestra.run.smithi150.stdout: "client_oc_size": "209715200", 2024-09-11T08:09:33.424 INFO:teuthology.orchestra.run.smithi150.stdout: "client_oc_target_dirty": "8388608", 2024-09-11T08:09:33.424 INFO:teuthology.orchestra.run.smithi150.stdout: "client_permissions": "true", 2024-09-11T08:09:33.424 INFO:teuthology.orchestra.run.smithi150.stdout: "client_quota": "true", 2024-09-11T08:09:33.425 INFO:teuthology.orchestra.run.smithi150.stdout: "client_quota_df": "true", 2024-09-11T08:09:33.425 INFO:teuthology.orchestra.run.smithi150.stdout: "client_readahead_max_bytes": "0", 2024-09-11T08:09:33.425 INFO:teuthology.orchestra.run.smithi150.stdout: "client_readahead_max_periods": "4", 2024-09-11T08:09:33.425 INFO:teuthology.orchestra.run.smithi150.stdout: "client_readahead_min": "131072", 2024-09-11T08:09:33.425 INFO:teuthology.orchestra.run.smithi150.stdout: "client_reconnect_stale": "false", 2024-09-11T08:09:33.425 INFO:teuthology.orchestra.run.smithi150.stdout: "client_shutdown_timeout": "30", 2024-09-11T08:09:33.425 INFO:teuthology.orchestra.run.smithi150.stdout: "client_snapdir": ".snap", 2024-09-11T08:09:33.425 INFO:teuthology.orchestra.run.smithi150.stdout: "client_tick_interval": "1", 2024-09-11T08:09:33.425 INFO:teuthology.orchestra.run.smithi150.stdout: "client_trace": "", 2024-09-11T08:09:33.425 INFO:teuthology.orchestra.run.smithi150.stdout: "client_try_dentry_invalidate": "false", 2024-09-11T08:09:33.425 INFO:teuthology.orchestra.run.smithi150.stdout: "client_use_faked_inos": "false", 2024-09-11T08:09:33.426 INFO:teuthology.orchestra.run.smithi150.stdout: "client_use_random_mds": "false", 2024-09-11T08:09:33.426 INFO:teuthology.orchestra.run.smithi150.stdout: "clog_to_graylog": "false", 2024-09-11T08:09:33.426 INFO:teuthology.orchestra.run.smithi150.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-11T08:09:33.426 INFO:teuthology.orchestra.run.smithi150.stdout: "clog_to_graylog_port": "12201", 2024-09-11T08:09:33.426 INFO:teuthology.orchestra.run.smithi150.stdout: "clog_to_monitors": "default=true", 2024-09-11T08:09:33.426 INFO:teuthology.orchestra.run.smithi150.stdout: "clog_to_syslog": "false", 2024-09-11T08:09:33.426 INFO:teuthology.orchestra.run.smithi150.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-11T08:09:33.426 INFO:teuthology.orchestra.run.smithi150.stdout: "clog_to_syslog_level": "info", 2024-09-11T08:09:33.426 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addr": "-", 2024-09-11T08:09:33.426 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_network": "", 2024-09-11T08:09:33.426 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_network_interface": "", 2024-09-11T08:09:33.426 INFO:teuthology.orchestra.run.smithi150.stdout: "compressor_zlib_isal": "false", 2024-09-11T08:09:33.426 INFO:teuthology.orchestra.run.smithi150.stdout: "compressor_zlib_level": "5", 2024-09-11T08:09:33.426 INFO:teuthology.orchestra.run.smithi150.stdout: "compressor_zlib_winsize": "-15", 2024-09-11T08:09:33.427 INFO:teuthology.orchestra.run.smithi150.stdout: "compressor_zstd_level": "1", 2024-09-11T08:09:33.427 INFO:teuthology.orchestra.run.smithi150.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40", 2024-09-11T08:09:33.427 INFO:teuthology.orchestra.run.smithi150.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-11T08:09:33.427 INFO:teuthology.orchestra.run.smithi150.stdout: "crimson_alien_op_num_threads": "6", 2024-09-11T08:09:33.427 INFO:teuthology.orchestra.run.smithi150.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-11T08:09:33.427 INFO:teuthology.orchestra.run.smithi150.stdout: "crimson_osd_obc_lru_size": "10", 2024-09-11T08:09:33.427 INFO:teuthology.orchestra.run.smithi150.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-11T08:09:33.427 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_location": "", 2024-09-11T08:09:33.427 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_location_hook": "", 2024-09-11T08:09:33.427 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_location_hook_timeout": "10", 2024-09-11T08:09:33.427 INFO:teuthology.orchestra.run.smithi150.stdout: "daemonize": "false", 2024-09-11T08:09:33.427 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_alienstore": "0/5", 2024-09-11T08:09:33.427 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_allow_any_pool_priority": "false", 2024-09-11T08:09:33.427 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_asok": "1/5", 2024-09-11T08:09:33.428 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_asok_assert_abort": "false", 2024-09-11T08:09:33.428 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_asserts_on_shutdown": "true", 2024-09-11T08:09:33.428 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_auth": "1/5", 2024-09-11T08:09:33.428 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_bdev": "1/3", 2024-09-11T08:09:33.428 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_bluefs": "1/5", 2024-09-11T08:09:33.428 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_bluestore": "1/5", 2024-09-11T08:09:33.428 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_buffer": "0/1", 2024-09-11T08:09:33.428 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_ceph_exporter": "1/5", 2024-09-11T08:09:33.428 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_cephfs_mirror": "0/5", 2024-09-11T08:09:33.428 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_cephsqlite": "0/5", 2024-09-11T08:09:33.428 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_civetweb": "1/10", 2024-09-11T08:09:33.428 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_client": "0/5", 2024-09-11T08:09:33.428 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_compressor": "1/5", 2024-09-11T08:09:33.429 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_context": "0/1", 2024-09-11T08:09:33.429 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_crush": "1/1", 2024-09-11T08:09:33.429 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_crypto": "1/5", 2024-09-11T08:09:33.429 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_deliberately_leak_memory": "false", 2024-09-11T08:09:33.429 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_disable_randomized_ping": "false", 2024-09-11T08:09:33.429 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_dpdk": "1/5", 2024-09-11T08:09:33.429 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_eventtrace": "1/5", 2024-09-11T08:09:33.429 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_filer": "0/1", 2024-09-11T08:09:33.429 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_filestore": "1/3", 2024-09-11T08:09:33.429 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_finisher": "1/1", 2024-09-11T08:09:33.429 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_fuse": "1/5", 2024-09-11T08:09:33.429 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_heartbeat_testing_span": "0", 2024-09-11T08:09:33.429 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_heartbeatmap": "1/5", 2024-09-11T08:09:33.429 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-11T08:09:33.430 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_javaclient": "1/5", 2024-09-11T08:09:33.430 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_journal": "1/3", 2024-09-11T08:09:33.430 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_journaler": "0/5", 2024-09-11T08:09:33.430 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_kstore": "1/5", 2024-09-11T08:09:33.430 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_leveldb": "4/5", 2024-09-11T08:09:33.430 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_lockdep": "0/1", 2024-09-11T08:09:33.430 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_mclock": "1/5", 2024-09-11T08:09:33.430 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_mds": "1/5", 2024-09-11T08:09:33.430 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_mds_balancer": "1/5", 2024-09-11T08:09:33.430 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_mds_locker": "1/5", 2024-09-11T08:09:33.430 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_mds_log": "1/5", 2024-09-11T08:09:33.430 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_mds_log_expire": "1/5", 2024-09-11T08:09:33.430 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_mds_migrator": "1/5", 2024-09-11T08:09:33.430 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_memdb": "4/5", 2024-09-11T08:09:33.431 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_mgr": "2/5", 2024-09-11T08:09:33.431 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_mgrc": "1/5", 2024-09-11T08:09:33.431 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_mon": "20/20", 2024-09-11T08:09:33.431 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_monc": "0/10", 2024-09-11T08:09:33.431 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_ms": "1/1", 2024-09-11T08:09:33.431 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_none": "0/5", 2024-09-11T08:09:33.431 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_objclass": "0/5", 2024-09-11T08:09:33.431 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_objectcacher": "0/5", 2024-09-11T08:09:33.431 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_objecter": "0/1", 2024-09-11T08:09:33.431 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_optracker": "0/5", 2024-09-11T08:09:33.431 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_osd": "1/5", 2024-09-11T08:09:33.431 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_paxos": "20/20", 2024-09-11T08:09:33.431 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_perfcounter": "1/5", 2024-09-11T08:09:33.431 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_prioritycache": "1/5", 2024-09-11T08:09:33.432 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_rados": "0/5", 2024-09-11T08:09:33.432 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_rbd": "0/5", 2024-09-11T08:09:33.432 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_rbd_mirror": "0/5", 2024-09-11T08:09:33.432 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_rbd_pwl": "0/5", 2024-09-11T08:09:33.432 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_rbd_replay": "0/5", 2024-09-11T08:09:33.432 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_refs": "0/0", 2024-09-11T08:09:33.432 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_reserver": "1/1", 2024-09-11T08:09:33.432 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_rgw": "1/5", 2024-09-11T08:09:33.432 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_rgw_datacache": "1/5", 2024-09-11T08:09:33.432 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_rgw_sync": "1/5", 2024-09-11T08:09:33.432 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_rocksdb": "4/5", 2024-09-11T08:09:33.432 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_seastore": "0/5", 2024-09-11T08:09:33.432 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_seastore_cache": "0/5", 2024-09-11T08:09:33.432 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_seastore_cleaner": "0/5", 2024-09-11T08:09:33.433 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_seastore_device": "0/5", 2024-09-11T08:09:33.433 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_seastore_journal": "0/5", 2024-09-11T08:09:33.433 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_seastore_lba": "0/5", 2024-09-11T08:09:33.433 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_seastore_odata": "0/5", 2024-09-11T08:09:33.433 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_seastore_omap": "0/5", 2024-09-11T08:09:33.433 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_seastore_onode": "0/5", 2024-09-11T08:09:33.433 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_seastore_tm": "0/5", 2024-09-11T08:09:33.433 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_striper": "0/1", 2024-09-11T08:09:33.433 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_test": "0/5", 2024-09-11T08:09:33.433 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_throttle": "1/1", 2024-09-11T08:09:33.433 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_timer": "0/1", 2024-09-11T08:09:33.433 INFO:teuthology.orchestra.run.smithi150.stdout: "debug_tp": "0/5", 2024-09-11T08:09:33.433 INFO:teuthology.orchestra.run.smithi150.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-11T08:09:33.433 INFO:teuthology.orchestra.run.smithi150.stdout: "device_failure_prediction_mode": "none", 2024-09-11T08:09:33.434 INFO:teuthology.orchestra.run.smithi150.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-11T08:09:33.434 INFO:teuthology.orchestra.run.smithi150.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-09-11T08:09:33.434 INFO:teuthology.orchestra.run.smithi150.stdout: "err_to_graylog": "false", 2024-09-11T08:09:33.434 INFO:teuthology.orchestra.run.smithi150.stdout: "err_to_journald": "false", 2024-09-11T08:09:33.434 INFO:teuthology.orchestra.run.smithi150.stdout: "err_to_stderr": "true", 2024-09-11T08:09:33.434 INFO:teuthology.orchestra.run.smithi150.stdout: "err_to_syslog": "false", 2024-09-11T08:09:33.434 INFO:teuthology.orchestra.run.smithi150.stdout: "event_tracing": "false", 2024-09-11T08:09:33.434 INFO:teuthology.orchestra.run.smithi150.stdout: "exporter_addr": "0.0.0.0", 2024-09-11T08:09:33.434 INFO:teuthology.orchestra.run.smithi150.stdout: "exporter_http_port": "9926", 2024-09-11T08:09:33.434 INFO:teuthology.orchestra.run.smithi150.stdout: "exporter_prio_limit": "5", 2024-09-11T08:09:33.434 INFO:teuthology.orchestra.run.smithi150.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-11T08:09:33.434 INFO:teuthology.orchestra.run.smithi150.stdout: "exporter_sort_metrics": "true", 2024-09-11T08:09:33.434 INFO:teuthology.orchestra.run.smithi150.stdout: "exporter_stats_period": "5", 2024-09-11T08:09:33.434 INFO:teuthology.orchestra.run.smithi150.stdout: "fake_statfs_for_testing": "0", 2024-09-11T08:09:33.435 INFO:teuthology.orchestra.run.smithi150.stdout: "fatal_signal_handlers": "true", 2024-09-11T08:09:33.435 INFO:teuthology.orchestra.run.smithi150.stdout: "filer_max_purge_ops": "10", 2024-09-11T08:09:33.435 INFO:teuthology.orchestra.run.smithi150.stdout: "filer_max_truncate_ops": "128", 2024-09-11T08:09:33.435 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_apply_finisher_threads": "1", 2024-09-11T08:09:33.435 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_blackhole": "false", 2024-09-11T08:09:33.435 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_btrfs_clone_range": "true", 2024-09-11T08:09:33.435 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_btrfs_snap": "true", 2024-09-11T08:09:33.435 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_caller_concurrency": "10", 2024-09-11T08:09:33.435 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_collect_device_partition_information": "true", 2024-09-11T08:09:33.435 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_commit_timeout": "600.000000", 2024-09-11T08:09:33.435 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_debug_inject_read_err": "false", 2024-09-11T08:09:33.435 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_debug_omap_check": "false", 2024-09-11T08:09:33.435 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_debug_random_read_err": "0.000000", 2024-09-11T08:09:33.436 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_debug_verify_split": "false", 2024-09-11T08:09:33.436 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_dump_file": "", 2024-09-11T08:09:33.436 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-11T08:09:33.436 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-11T08:09:33.436 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_fadvise": "true", 2024-09-11T08:09:33.436 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_fail_eio": "true", 2024-09-11T08:09:33.436 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_fd_cache_shards": "16", 2024-09-11T08:09:33.436 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_fd_cache_size": "128", 2024-09-11T08:09:33.436 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_fiemap": "false", 2024-09-11T08:09:33.436 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_fiemap_threshold": "4096", 2024-09-11T08:09:33.436 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-11T08:09:33.436 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-11T08:09:33.436 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_inject_stall": "0", 2024-09-11T08:09:33.437 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_journal_parallel": "false", 2024-09-11T08:09:33.437 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_journal_trailing": "false", 2024-09-11T08:09:33.437 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_journal_writeahead": "false", 2024-09-11T08:09:33.437 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_kill_at": "0", 2024-09-11T08:09:33.437 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-11T08:09:33.437 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-11T08:09:33.437 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-11T08:09:33.437 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-11T08:09:33.437 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-11T08:09:33.437 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_max_inline_xattrs": "0", 2024-09-11T08:09:33.437 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-11T08:09:33.437 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-11T08:09:33.437 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-11T08:09:33.437 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-11T08:09:33.438 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_max_xattr_value_size": "0", 2024-09-11T08:09:33.438 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-11T08:09:33.438 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-11T08:09:33.438 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-11T08:09:33.438 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_merge_threshold": "-10", 2024-09-11T08:09:33.438 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-11T08:09:33.438 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_odsync_write": "false", 2024-09-11T08:09:33.438 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_omap_backend": "rocksdb", 2024-09-11T08:09:33.438 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_omap_backend_path": "", 2024-09-11T08:09:33.438 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-11T08:09:33.438 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-11T08:09:33.438 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-11T08:09:33.438 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_op_thread_timeout": "60", 2024-09-11T08:09:33.438 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_op_threads": "2", 2024-09-11T08:09:33.439 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_punch_hole": "false", 2024-09-11T08:09:33.439 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-11T08:09:33.439 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-11T08:09:33.439 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-11T08:09:33.439 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-11T08:09:33.439 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-11T08:09:33.439 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-11T08:09:33.439 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-11T08:09:33.439 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-11T08:09:33.439 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-11T08:09:33.439 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_queue_max_ops": "50", 2024-09-11T08:09:33.439 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-11T08:09:33.439 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_seek_data_hole": "false", 2024-09-11T08:09:33.440 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_sloppy_crc": "false", 2024-09-11T08:09:33.440 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-11T08:09:33.440 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_splice": "false", 2024-09-11T08:09:33.440 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_split_multiple": "2", 2024-09-11T08:09:33.440 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_split_rand_factor": "20", 2024-09-11T08:09:33.440 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_update_to": "1000", 2024-09-11T08:09:33.440 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-11T08:09:33.440 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-11T08:09:33.440 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-11T08:09:33.440 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-11T08:09:33.440 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-11T08:09:33.440 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-11T08:09:33.440 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_wbthrottle_enable": "true", 2024-09-11T08:09:33.440 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-11T08:09:33.441 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-11T08:09:33.441 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-11T08:09:33.441 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-11T08:09:33.441 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-11T08:09:33.441 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-11T08:09:33.441 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_xfs_extsize": "false", 2024-09-11T08:09:33.441 INFO:teuthology.orchestra.run.smithi150.stdout: "filestore_zfs_snap": "false", 2024-09-11T08:09:33.441 INFO:teuthology.orchestra.run.smithi150.stdout: "fio_dir": "/tmp/fio", 2024-09-11T08:09:33.441 INFO:teuthology.orchestra.run.smithi150.stdout: "fsid": "c1bddff4-7013-11ef-bcea-c7b262605968", 2024-09-11T08:09:33.441 INFO:teuthology.orchestra.run.smithi150.stdout: "fuse_allow_other": "true", 2024-09-11T08:09:33.441 INFO:teuthology.orchestra.run.smithi150.stdout: "fuse_atomic_o_trunc": "true", 2024-09-11T08:09:33.441 INFO:teuthology.orchestra.run.smithi150.stdout: "fuse_big_writes": "true", 2024-09-11T08:09:33.441 INFO:teuthology.orchestra.run.smithi150.stdout: "fuse_debug": "false", 2024-09-11T08:09:33.442 INFO:teuthology.orchestra.run.smithi150.stdout: "fuse_default_permissions": "false", 2024-09-11T08:09:33.442 INFO:teuthology.orchestra.run.smithi150.stdout: "fuse_disable_pagecache": "false", 2024-09-11T08:09:33.442 INFO:teuthology.orchestra.run.smithi150.stdout: "fuse_max_write": "0", 2024-09-11T08:09:33.442 INFO:teuthology.orchestra.run.smithi150.stdout: "fuse_multithreaded": "true", 2024-09-11T08:09:33.442 INFO:teuthology.orchestra.run.smithi150.stdout: "fuse_require_active_mds": "true", 2024-09-11T08:09:33.442 INFO:teuthology.orchestra.run.smithi150.stdout: "fuse_set_user_groups": "true", 2024-09-11T08:09:33.442 INFO:teuthology.orchestra.run.smithi150.stdout: "fuse_splice_move": "true", 2024-09-11T08:09:33.442 INFO:teuthology.orchestra.run.smithi150.stdout: "fuse_splice_read": "true", 2024-09-11T08:09:33.442 INFO:teuthology.orchestra.run.smithi150.stdout: "fuse_splice_write": "true", 2024-09-11T08:09:33.442 INFO:teuthology.orchestra.run.smithi150.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-11T08:09:33.442 INFO:teuthology.orchestra.run.smithi150.stdout: "fuse_use_invalidate_cb": "true", 2024-09-11T08:09:33.442 INFO:teuthology.orchestra.run.smithi150.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-11T08:09:33.442 INFO:teuthology.orchestra.run.smithi150.stdout: "gss_target_name": "ceph", 2024-09-11T08:09:33.442 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_file": "", 2024-09-11T08:09:33.443 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_inject_failure": "0", 2024-09-11T08:09:33.443 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_interval": "5", 2024-09-11T08:09:33.443 INFO:teuthology.orchestra.run.smithi150.stdout: "host": "smithi150", 2024-09-11T08:09:33.443 INFO:teuthology.orchestra.run.smithi150.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-11T08:09:33.443 INFO:teuthology.orchestra.run.smithi150.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-11T08:09:33.443 INFO:teuthology.orchestra.run.smithi150.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-11T08:09:33.443 INFO:teuthology.orchestra.run.smithi150.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-11T08:09:33.443 INFO:teuthology.orchestra.run.smithi150.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-11T08:09:33.443 INFO:teuthology.orchestra.run.smithi150.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-11T08:09:33.443 INFO:teuthology.orchestra.run.smithi150.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-11T08:09:33.443 INFO:teuthology.orchestra.run.smithi150.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-11T08:09:33.443 INFO:teuthology.orchestra.run.smithi150.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-11T08:09:33.443 INFO:teuthology.orchestra.run.smithi150.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-11T08:09:33.443 INFO:teuthology.orchestra.run.smithi150.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-11T08:09:33.444 INFO:teuthology.orchestra.run.smithi150.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-11T08:09:33.444 INFO:teuthology.orchestra.run.smithi150.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-11T08:09:33.444 INFO:teuthology.orchestra.run.smithi150.stdout: "inject_early_sigterm": "false", 2024-09-11T08:09:33.444 INFO:teuthology.orchestra.run.smithi150.stdout: "jaeger_tracing_enable": "false", 2024-09-11T08:09:33.444 INFO:teuthology.orchestra.run.smithi150.stdout: "journal_aio": "true", 2024-09-11T08:09:33.444 INFO:teuthology.orchestra.run.smithi150.stdout: "journal_align_min_size": "65536", 2024-09-11T08:09:33.444 INFO:teuthology.orchestra.run.smithi150.stdout: "journal_block_align": "true", 2024-09-11T08:09:33.444 INFO:teuthology.orchestra.run.smithi150.stdout: "journal_block_size": "4096", 2024-09-11T08:09:33.444 INFO:teuthology.orchestra.run.smithi150.stdout: "journal_dio": "true", 2024-09-11T08:09:33.444 INFO:teuthology.orchestra.run.smithi150.stdout: "journal_discard": "false", 2024-09-11T08:09:33.445 INFO:teuthology.orchestra.run.smithi150.stdout: "journal_force_aio": "false", 2024-09-11T08:09:33.445 INFO:teuthology.orchestra.run.smithi150.stdout: "journal_ignore_corruption": "false", 2024-09-11T08:09:33.445 INFO:teuthology.orchestra.run.smithi150.stdout: "journal_max_write_bytes": "10485760", 2024-09-11T08:09:33.445 INFO:teuthology.orchestra.run.smithi150.stdout: "journal_max_write_entries": "100", 2024-09-11T08:09:33.445 INFO:teuthology.orchestra.run.smithi150.stdout: "journal_replay_from": "0", 2024-09-11T08:09:33.445 INFO:teuthology.orchestra.run.smithi150.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-11T08:09:33.445 INFO:teuthology.orchestra.run.smithi150.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-11T08:09:33.445 INFO:teuthology.orchestra.run.smithi150.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-11T08:09:33.445 INFO:teuthology.orchestra.run.smithi150.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-11T08:09:33.445 INFO:teuthology.orchestra.run.smithi150.stdout: "journal_write_header_frequency": "0", 2024-09-11T08:09:33.445 INFO:teuthology.orchestra.run.smithi150.stdout: "journal_zero_on_create": "false", 2024-09-11T08:09:33.445 INFO:teuthology.orchestra.run.smithi150.stdout: "journaler_prefetch_periods": "10", 2024-09-11T08:09:33.445 INFO:teuthology.orchestra.run.smithi150.stdout: "journaler_prezero_periods": "5", 2024-09-11T08:09:33.446 INFO:teuthology.orchestra.run.smithi150.stdout: "journaler_write_head_interval": "15", 2024-09-11T08:09:33.446 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "", 2024-09-11T08:09:33.446 INFO:teuthology.orchestra.run.smithi150.stdout: "keyfile": "", 2024-09-11T08:09:33.446 INFO:teuthology.orchestra.run.smithi150.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-11T08:09:33.446 INFO:teuthology.orchestra.run.smithi150.stdout: "kstore_backend": "rocksdb", 2024-09-11T08:09:33.446 INFO:teuthology.orchestra.run.smithi150.stdout: "kstore_default_stripe_size": "65536", 2024-09-11T08:09:33.446 INFO:teuthology.orchestra.run.smithi150.stdout: "kstore_fsck_on_mount": "false", 2024-09-11T08:09:33.446 INFO:teuthology.orchestra.run.smithi150.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-11T08:09:33.446 INFO:teuthology.orchestra.run.smithi150.stdout: "kstore_max_bytes": "67108864", 2024-09-11T08:09:33.446 INFO:teuthology.orchestra.run.smithi150.stdout: "kstore_max_ops": "512", 2024-09-11T08:09:33.446 INFO:teuthology.orchestra.run.smithi150.stdout: "kstore_nid_prealloc": "1024", 2024-09-11T08:09:33.446 INFO:teuthology.orchestra.run.smithi150.stdout: "kstore_onode_map_size": "1024", 2024-09-11T08:09:33.446 INFO:teuthology.orchestra.run.smithi150.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-11T08:09:33.446 INFO:teuthology.orchestra.run.smithi150.stdout: "kstore_sync_submit_transaction": "false", 2024-09-11T08:09:33.447 INFO:teuthology.orchestra.run.smithi150.stdout: "kstore_sync_transaction": "false", 2024-09-11T08:09:33.447 INFO:teuthology.orchestra.run.smithi150.stdout: "leveldb_block_size": "65536", 2024-09-11T08:09:33.447 INFO:teuthology.orchestra.run.smithi150.stdout: "leveldb_bloom_size": "0", 2024-09-11T08:09:33.447 INFO:teuthology.orchestra.run.smithi150.stdout: "leveldb_cache_size": "536870912", 2024-09-11T08:09:33.447 INFO:teuthology.orchestra.run.smithi150.stdout: "leveldb_compact_on_mount": "false", 2024-09-11T08:09:33.447 INFO:teuthology.orchestra.run.smithi150.stdout: "leveldb_compression": "false", 2024-09-11T08:09:33.447 INFO:teuthology.orchestra.run.smithi150.stdout: "leveldb_log": "", 2024-09-11T08:09:33.447 INFO:teuthology.orchestra.run.smithi150.stdout: "leveldb_log_to_ceph_log": "true", 2024-09-11T08:09:33.447 INFO:teuthology.orchestra.run.smithi150.stdout: "leveldb_max_open_files": "0", 2024-09-11T08:09:33.447 INFO:teuthology.orchestra.run.smithi150.stdout: "leveldb_paranoid": "false", 2024-09-11T08:09:33.447 INFO:teuthology.orchestra.run.smithi150.stdout: "leveldb_write_buffer_size": "33554432", 2024-09-11T08:09:33.447 INFO:teuthology.orchestra.run.smithi150.stdout: "librados_thread_count": "2", 2024-09-11T08:09:33.447 INFO:teuthology.orchestra.run.smithi150.stdout: "lockdep": "false", 2024-09-11T08:09:33.447 INFO:teuthology.orchestra.run.smithi150.stdout: "lockdep_force_backtrace": "false", 2024-09-11T08:09:33.447 INFO:teuthology.orchestra.run.smithi150.stdout: "log_coarse_timestamps": "true", 2024-09-11T08:09:33.448 INFO:teuthology.orchestra.run.smithi150.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-11T08:09:33.448 INFO:teuthology.orchestra.run.smithi150.stdout: "log_flush_on_exit": "false", 2024-09-11T08:09:33.448 INFO:teuthology.orchestra.run.smithi150.stdout: "log_graylog_host": "127.0.0.1", 2024-09-11T08:09:33.448 INFO:teuthology.orchestra.run.smithi150.stdout: "log_graylog_port": "12201", 2024-09-11T08:09:33.448 INFO:teuthology.orchestra.run.smithi150.stdout: "log_max_new": "1000", 2024-09-11T08:09:33.448 INFO:teuthology.orchestra.run.smithi150.stdout: "log_max_recent": "10000", 2024-09-11T08:09:33.448 INFO:teuthology.orchestra.run.smithi150.stdout: "log_stderr_prefix": "", 2024-09-11T08:09:33.448 INFO:teuthology.orchestra.run.smithi150.stdout: "log_stop_at_utilization": "0.970000", 2024-09-11T08:09:33.448 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": "true", 2024-09-11T08:09:33.448 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_graylog": "false", 2024-09-11T08:09:33.448 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_journald": "false", 2024-09-11T08:09:33.448 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_stderr": "false", 2024-09-11T08:09:33.448 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_syslog": "false", 2024-09-11T08:09:33.449 INFO:teuthology.orchestra.run.smithi150.stdout: "max_rotating_auth_attempts": "10", 2024-09-11T08:09:33.449 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-11T08:09:33.449 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_action_on_write_error": "1", 2024-09-11T08:09:33.449 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_alternate_name_max": "8192", 2024-09-11T08:09:33.449 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_asio_thread_count": "2", 2024-09-11T08:09:33.449 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_bal_export_pin": "true", 2024-09-11T08:09:33.449 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_bal_fragment_dirs": "true", 2024-09-11T08:09:33.449 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-11T08:09:33.449 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_bal_fragment_interval": "5", 2024-09-11T08:09:33.449 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-11T08:09:33.449 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-11T08:09:33.449 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_bal_interval": "10", 2024-09-11T08:09:33.449 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_bal_max": "-1", 2024-09-11T08:09:33.449 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_bal_max_until": "-1", 2024-09-11T08:09:33.450 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_bal_merge_size": "50", 2024-09-11T08:09:33.450 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_bal_midchunk": "0.300000", 2024-09-11T08:09:33.450 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-11T08:09:33.450 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_bal_min_start": "0.200000", 2024-09-11T08:09:33.450 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_bal_minchunk": "0.001000", 2024-09-11T08:09:33.450 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_bal_mode": "0", 2024-09-11T08:09:33.450 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_bal_need_max": "1.200000", 2024-09-11T08:09:33.450 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_bal_need_min": "0.800000", 2024-09-11T08:09:33.450 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-11T08:09:33.450 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-11T08:09:33.450 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_bal_split_bits": "3", 2024-09-11T08:09:33.450 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-11T08:09:33.450 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_bal_split_size": "10000", 2024-09-11T08:09:33.450 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-11T08:09:33.451 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_bal_target_decay": "10.000000", 2024-09-11T08:09:33.451 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-11T08:09:33.451 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_beacon_grace": "15.000000", 2024-09-11T08:09:33.451 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_beacon_interval": "4.000000", 2024-09-11T08:09:33.451 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-11T08:09:33.451 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-11T08:09:33.451 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_cache_mid": "0.700000", 2024-09-11T08:09:33.451 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_cache_release_free_interval": "10", 2024-09-11T08:09:33.451 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_cache_reservation": "0.050000", 2024-09-11T08:09:33.451 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-11T08:09:33.451 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_cache_trim_interval": "1", 2024-09-11T08:09:33.451 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_cache_trim_threshold": "262144", 2024-09-11T08:09:33.451 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-11T08:09:33.451 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-11T08:09:33.452 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-11T08:09:33.452 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_client_prealloc_inos": "1000", 2024-09-11T08:09:33.452 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-11T08:09:33.452 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_connect_bootstrapping": "false", 2024-09-11T08:09:33.452 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_damage_table_max_entries": "10000", 2024-09-11T08:09:33.452 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-11T08:09:33.452 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_debug_auth_pins": "false", 2024-09-11T08:09:33.452 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_debug_frag": "false", 2024-09-11T08:09:33.452 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_debug_scatterstat": "false", 2024-09-11T08:09:33.452 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_debug_subtrees": "false", 2024-09-11T08:09:33.452 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_decay_halflife": "5.000000", 2024-09-11T08:09:33.452 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_default_dir_hash": "2", 2024-09-11T08:09:33.452 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_defer_session_stale": "true", 2024-09-11T08:09:33.453 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_deny_all_reconnect": "false", 2024-09-11T08:09:33.453 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_dir_keys_per_op": "16384", 2024-09-11T08:09:33.453 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_dir_max_commit_size": "10", 2024-09-11T08:09:33.453 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_dir_max_entries": "0", 2024-09-11T08:09:33.453 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-11T08:09:33.453 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-11T08:09:33.453 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_dump_cache_on_map": "false", 2024-09-11T08:09:33.453 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-11T08:09:33.453 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-11T08:09:33.453 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_early_reply": "true", 2024-09-11T08:09:33.453 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_enable_op_tracker": "true", 2024-09-11T08:09:33.453 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_enforce_unique_name": "true", 2024-09-11T08:09:33.453 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-11T08:09:33.453 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-11T08:09:33.454 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_export_ephemeral_random": "true", 2024-09-11T08:09:33.454 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-11T08:09:33.454 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-11T08:09:33.454 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-11T08:09:33.454 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-11T08:09:33.454 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-11T08:09:33.454 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-11T08:09:33.454 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_health_summarize_threshold": "10", 2024-09-11T08:09:33.454 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-11T08:09:33.454 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-11T08:09:33.454 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_inject_health_dummy": "false", 2024-09-11T08:09:33.454 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-11T08:09:33.454 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_inject_migrator_session_race": "false", 2024-09-11T08:09:33.454 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-11T08:09:33.455 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-11T08:09:33.455 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-11T08:09:33.455 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_join_fs": "", 2024-09-11T08:09:33.455 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_journal_format": "1", 2024-09-11T08:09:33.455 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_kill_create_at": "0", 2024-09-11T08:09:33.455 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_kill_export_at": "0", 2024-09-11T08:09:33.455 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_kill_import_at": "0", 2024-09-11T08:09:33.455 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_kill_journal_at": "0", 2024-09-11T08:09:33.455 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_kill_journal_expire_at": "0", 2024-09-11T08:09:33.455 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_kill_journal_replay_at": "0", 2024-09-11T08:09:33.456 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_kill_link_at": "0", 2024-09-11T08:09:33.456 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_kill_mdstable_at": "0", 2024-09-11T08:09:33.456 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_kill_openc_at": "0", 2024-09-11T08:09:33.456 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_kill_rename_at": "0", 2024-09-11T08:09:33.456 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-09-11T08:09:33.456 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_log_events_per_segment": "1024", 2024-09-11T08:09:33.456 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_log_max_events": "-1", 2024-09-11T08:09:33.456 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_log_max_segments": "128", 2024-09-11T08:09:33.456 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_log_pause": "false", 2024-09-11T08:09:33.456 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_log_segment_size": "0", 2024-09-11T08:09:33.457 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-11T08:09:33.457 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_log_warn_factor": "2.000000", 2024-09-11T08:09:33.457 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_max_caps_per_client": "1048576", 2024-09-11T08:09:33.457 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_max_completed_flushes": "100000", 2024-09-11T08:09:33.457 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_max_completed_requests": "100000", 2024-09-11T08:09:33.457 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_max_export_size": "20971520", 2024-09-11T08:09:33.457 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_max_file_recover": "32", 2024-09-11T08:09:33.457 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_max_purge_files": "64", 2024-09-11T08:09:33.457 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_max_purge_ops": "8192", 2024-09-11T08:09:33.458 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-11T08:09:33.458 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-11T08:09:33.458 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_max_snaps_per_dir": "100", 2024-09-11T08:09:33.458 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_max_xattr_pairs_size": "65536", 2024-09-11T08:09:33.458 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_metrics_update_interval": "2", 2024-09-11T08:09:33.458 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_min_caps_per_client": "100", 2024-09-11T08:09:33.458 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_min_caps_working_set": "10000", 2024-09-11T08:09:33.458 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-11T08:09:33.458 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_numa_node": "-1", 2024-09-11T08:09:33.458 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-11T08:09:33.458 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_op_complaint_time": "30.000000", 2024-09-11T08:09:33.458 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_op_history_duration": "600", 2024-09-11T08:09:33.458 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_op_history_size": "20", 2024-09-11T08:09:33.458 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_op_history_slow_op_size": "20", 2024-09-11T08:09:33.459 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-11T08:09:33.459 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_op_log_threshold": "5", 2024-09-11T08:09:33.459 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_ping_grace": "15", 2024-09-11T08:09:33.459 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_ping_interval": "5", 2024-09-11T08:09:33.459 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-11T08:09:33.459 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-11T08:09:33.459 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_recall_max_caps": "30000", 2024-09-11T08:09:33.459 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-11T08:09:33.459 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-11T08:09:33.459 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-11T08:09:33.459 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_recall_warning_threshold": "262144", 2024-09-11T08:09:33.459 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-11T08:09:33.459 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_replay_interval": "1.000000", 2024-09-11T08:09:33.459 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-11T08:09:33.460 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-11T08:09:33.460 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_root_ino_gid": "0", 2024-09-11T08:09:33.460 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_root_ino_uid": "0", 2024-09-11T08:09:33.460 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-11T08:09:33.460 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-11T08:09:33.460 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-11T08:09:33.460 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-11T08:09:33.460 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-11T08:09:33.460 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-11T08:09:33.460 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-11T08:09:33.460 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-11T08:09:33.461 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-11T08:09:33.461 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-11T08:09:33.461 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_shutdown_check": "0", 2024-09-11T08:09:33.461 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_skip_ino": "0", 2024-09-11T08:09:33.461 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-11T08:09:33.461 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_snap_max_uid": "4294967294", 2024-09-11T08:09:33.461 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_snap_min_uid": "0", 2024-09-11T08:09:33.461 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_snap_rstat": "false", 2024-09-11T08:09:33.461 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_standby_replay_damaged": "false", 2024-09-11T08:09:33.461 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_symlink_recovery": "true", 2024-09-11T08:09:33.461 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-11T08:09:33.461 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_thrash_exports": "0", 2024-09-11T08:09:33.461 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_thrash_fragments": "0", 2024-09-11T08:09:33.461 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_tick_interval": "5.000000", 2024-09-11T08:09:33.462 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_valgrind_exit": "false", 2024-09-11T08:09:33.462 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_verify_backtrace": "1", 2024-09-11T08:09:33.462 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_verify_scatter": "false", 2024-09-11T08:09:33.462 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-11T08:09:33.462 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_wipe_sessions": "false", 2024-09-11T08:09:33.462 INFO:teuthology.orchestra.run.smithi150.stdout: "mempool_debug": "false", 2024-09-11T08:09:33.462 INFO:teuthology.orchestra.run.smithi150.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-11T08:09:33.462 INFO:teuthology.orchestra.run.smithi150.stdout: "memstore_device_bytes": "1073741824", 2024-09-11T08:09:33.462 INFO:teuthology.orchestra.run.smithi150.stdout: "memstore_page_set": "false", 2024-09-11T08:09:33.462 INFO:teuthology.orchestra.run.smithi150.stdout: "memstore_page_size": "65536", 2024-09-11T08:09:33.462 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr_client_bytes": "134217728", 2024-09-11T08:09:33.462 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr_client_messages": "512", 2024-09-11T08:09:33.462 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-11T08:09:33.462 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-11T08:09:33.463 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-11T08:09:33.463 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-11T08:09:33.463 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr_disabled_modules": "diskprediction_local", 2024-09-11T08:09:33.463 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-11T08:09:33.463 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr_max_pg_num_change": "128", 2024-09-11T08:09:33.463 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr_mds_bytes": "134217728", 2024-09-11T08:09:33.463 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr_mds_messages": "128", 2024-09-11T08:09:33.463 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-11T08:09:33.463 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr_mon_bytes": "134217728", 2024-09-11T08:09:33.463 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr_mon_messages": "128", 2024-09-11T08:09:33.463 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr_osd_bytes": "536870912", 2024-09-11T08:09:33.463 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr_osd_messages": "8192", 2024-09-11T08:09:33.463 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr_pool": "true", 2024-09-11T08:09:33.463 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-11T08:09:33.464 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr_standby_modules": "true", 2024-09-11T08:09:33.464 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr_stats_period": "5", 2024-09-11T08:09:33.464 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr_stats_threshold": "5", 2024-09-11T08:09:33.464 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr_tick_period": "2", 2024-09-11T08:09:33.464 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-11T08:09:33.464 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-11T08:09:33.464 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_allow_pool_delete": "true", 2024-09-11T08:09:33.464 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_allow_pool_size_one": "false", 2024-09-11T08:09:33.464 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_auth_validate_all_caps": "true", 2024-09-11T08:09:33.464 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-11T08:09:33.464 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-11T08:09:33.464 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_client_bytes": "104857600", 2024-09-11T08:09:33.464 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_client_directed_command_retry": "2", 2024-09-11T08:09:33.464 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-11T08:09:33.465 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-11T08:09:33.465 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-11T08:09:33.465 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-11T08:09:33.465 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_client_hunt_parallel": "3", 2024-09-11T08:09:33.465 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_client_log_interval": "1.000000", 2024-09-11T08:09:33.465 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-11T08:09:33.465 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_client_ping_interval": "10.000000", 2024-09-11T08:09:33.465 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-11T08:09:33.465 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-11T08:09:33.465 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-11T08:09:33.465 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-11T08:09:33.465 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_cluster_log_level": "debug", 2024-09-11T08:09:33.465 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_cluster_log_to_file": "false", 2024-09-11T08:09:33.465 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-11T08:09:33.466 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-11T08:09:33.466 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-11T08:09:33.466 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_cluster_log_to_journald": "true", 2024-09-11T08:09:33.466 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-11T08:09:33.466 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-11T08:09:33.466 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-11T08:09:33.466 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_compact_on_bootstrap": "false", 2024-09-11T08:09:33.466 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_compact_on_start": "false", 2024-09-11T08:09:33.466 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_compact_on_trim": "true", 2024-09-11T08:09:33.466 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-11T08:09:33.466 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-11T08:09:33.466 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-11T08:09:33.466 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_cpu_threads": "4", 2024-09-11T08:09:33.466 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_crush_min_required_version": "hammer", 2024-09-11T08:09:33.467 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_daemon_bytes": "419430400", 2024-09-11T08:09:33.467 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-11T08:09:33.467 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_data_avail_crit": "5", 2024-09-11T08:09:33.467 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_data_avail_warn": "5", 2024-09-11T08:09:33.467 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_data_size_warn": "16106127360", 2024-09-11T08:09:33.467 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-11T08:09:33.467 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-11T08:09:33.467 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_debug_dump_json": "false", 2024-09-11T08:09:33.467 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-11T08:09:33.467 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_debug_dump_transactions": "false", 2024-09-11T08:09:33.467 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_debug_extra_checks": "false", 2024-09-11T08:09:33.467 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-11T08:09:33.467 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-11T08:09:33.467 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_debug_no_require_pacific": "false", 2024-09-11T08:09:33.468 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_debug_no_require_quincy": "false", 2024-09-11T08:09:33.468 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-11T08:09:33.468 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-11T08:09:33.468 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-11T08:09:33.468 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_down_mkfs_grace": "60", 2024-09-11T08:09:33.468 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_election_default_strategy": "1", 2024-09-11T08:09:33.468 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_election_timeout": "5.000000", 2024-09-11T08:09:33.468 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-11T08:09:33.468 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_elector_ping_divisor": "2", 2024-09-11T08:09:33.468 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-11T08:09:33.468 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_enable_op_tracker": "true", 2024-09-11T08:09:33.468 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_fake_pool_delete": "false", 2024-09-11T08:09:33.468 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_force_quorum_join": "false", 2024-09-11T08:09:33.469 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_globalid_prealloc": "10000", 2024-09-11T08:09:33.469 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_health_detail_to_clog": "true", 2024-09-11T08:09:33.469 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_health_log_update_period": "5", 2024-09-11T08:09:33.469 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_health_max_detail": "50", 2024-09-11T08:09:33.469 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_health_to_clog": "true", 2024-09-11T08:09:33.469 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_health_to_clog_interval": "600", 2024-09-11T08:09:33.469 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-11T08:09:33.469 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_host": "[v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0]", 2024-09-11T08:09:33.469 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_host_override": "", 2024-09-11T08:09:33.469 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_initial_members": "", 2024-09-11T08:09:33.469 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-11T08:09:33.470 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-11T08:09:33.470 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-11T08:09:33.470 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-11T08:09:33.470 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-11T08:09:33.470 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_lease": "5.000000", 2024-09-11T08:09:33.470 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-11T08:09:33.470 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-11T08:09:33.470 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_log_full_interval": "50", 2024-09-11T08:09:33.470 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_log_max": "10000", 2024-09-11T08:09:33.470 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_log_max_summary": "50", 2024-09-11T08:09:33.470 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-11T08:09:33.470 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_max_log_epochs": "500", 2024-09-11T08:09:33.470 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-11T08:09:33.470 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-11T08:09:33.471 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_max_osd": "10000", 2024-09-11T08:09:33.471 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_max_pg_per_osd": "10000", 2024-09-11T08:09:33.471 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_max_pool_pg_num": "65536", 2024-09-11T08:09:33.471 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-11T08:09:33.471 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-11T08:09:33.471 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_mds_force_trim_to": "0", 2024-09-11T08:09:33.471 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_mds_skip_sanity": "false", 2024-09-11T08:09:33.471 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_memory_autotune": "true", 2024-09-11T08:09:33.471 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_memory_target": "2147483648", 2024-09-11T08:09:33.471 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_mgr_beacon_grace": "30", 2024-09-11T08:09:33.471 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-11T08:09:33.471 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_mgr_digest_period": "5", 2024-09-11T08:09:33.471 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_mgr_inactive_grace": "60", 2024-09-11T08:09:33.471 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-11T08:09:33.472 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-11T08:09:33.472 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_min_osdmap_epochs": "50", 2024-09-11T08:09:33.472 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_op_complaint_time": "30", 2024-09-11T08:09:33.472 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_op_history_duration": "600", 2024-09-11T08:09:33.472 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_op_history_size": "20", 2024-09-11T08:09:33.472 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_op_history_slow_op_size": "20", 2024-09-11T08:09:33.472 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-11T08:09:33.472 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_op_log_threshold": "5", 2024-09-11T08:09:33.472 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-11T08:09:33.472 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-11T08:09:33.472 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-11T08:09:33.472 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_osd_auto_mark_in": "false", 2024-09-11T08:09:33.472 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-11T08:09:33.472 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-11T08:09:33.473 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-11T08:09:33.473 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_osd_cache_size": "500", 2024-09-11T08:09:33.473 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-11T08:09:33.473 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-11T08:09:33.473 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-11T08:09:33.473 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_osd_down_out_interval": "600", 2024-09-11T08:09:33.473 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-11T08:09:33.473 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-11T08:09:33.473 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_osd_force_trim_to": "0", 2024-09-11T08:09:33.473 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-11T08:09:33.473 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-11T08:09:33.473 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-11T08:09:33.473 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-11T08:09:33.473 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-11T08:09:33.474 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-11T08:09:33.474 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_osd_max_creating_pgs": "1024", 2024-09-11T08:09:33.474 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-11T08:09:33.474 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_osd_min_down_reporters": "2", 2024-09-11T08:09:33.474 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-11T08:09:33.474 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-11T08:09:33.474 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-11T08:09:33.474 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-11T08:09:33.474 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-11T08:09:33.474 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-11T08:09:33.474 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_osd_report_timeout": "900", 2024-09-11T08:09:33.474 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-11T08:09:33.474 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-11T08:09:33.475 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-11T08:09:33.475 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-11T08:09:33.475 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-11T08:09:33.475 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_osdmap_full_prune_interval": "2", 2024-09-11T08:09:33.475 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_osdmap_full_prune_min": "15", 2024-09-11T08:09:33.475 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-09-11T08:09:33.475 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-11T08:09:33.475 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_pg_stuck_threshold": "60", 2024-09-11T08:09:33.475 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-11T08:09:33.475 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-11T08:09:33.475 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-11T08:09:33.475 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-11T08:09:33.475 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-11T08:09:33.475 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-11T08:09:33.476 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_probe_timeout": "2.000000", 2024-09-11T08:09:33.476 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_reweight_max_change": "0.050000", 2024-09-11T08:09:33.476 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_reweight_max_osds": "4", 2024-09-11T08:09:33.476 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-11T08:09:33.476 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-11T08:09:33.476 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-11T08:09:33.476 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-11T08:09:33.476 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-11T08:09:33.476 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_scrub_interval": "86400", 2024-09-11T08:09:33.476 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_scrub_max_keys": "100", 2024-09-11T08:09:33.476 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_scrub_timeout": "300", 2024-09-11T08:09:33.476 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_session_timeout": "300", 2024-09-11T08:09:33.476 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_smart_report_timeout": "5", 2024-09-11T08:09:33.476 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_stat_smooth_intervals": "6", 2024-09-11T08:09:33.477 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-11T08:09:33.477 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_stretch_pool_min_size": "2", 2024-09-11T08:09:33.477 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_stretch_pool_size": "4", 2024-09-11T08:09:33.477 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-11T08:09:33.477 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-11T08:09:33.477 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_sync_debug": "false", 2024-09-11T08:09:33.477 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-11T08:09:33.477 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-11T08:09:33.477 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_sync_provider_kill_at": "0", 2024-09-11T08:09:33.477 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_sync_requester_kill_at": "0", 2024-09-11T08:09:33.477 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_sync_timeout": "60.000000", 2024-09-11T08:09:33.477 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_target_pg_per_osd": "100", 2024-09-11T08:09:33.477 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_tick_interval": "5", 2024-09-11T08:09:33.477 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_timecheck_interval": "300.000000", 2024-09-11T08:09:33.478 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-11T08:09:33.478 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_warn_older_version_delay": "604800", 2024-09-11T08:09:33.478 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-11T08:09:33.478 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-11T08:09:33.478 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-11T08:09:33.478 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_warn_on_filestore_osds": "true", 2024-09-11T08:09:33.478 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-11T08:09:33.478 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-11T08:09:33.478 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-11T08:09:33.478 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_warn_on_misplaced": "false", 2024-09-11T08:09:33.478 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-11T08:09:33.478 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_warn_on_older_version": "true", 2024-09-11T08:09:33.478 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-11T08:09:33.479 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_warn_on_pool_no_app": "true", 2024-09-11T08:09:33.480 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-09-11T08:09:33.480 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-11T08:09:33.480 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-11T08:09:33.480 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-11T08:09:33.480 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-11T08:09:33.480 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-11T08:09:33.480 INFO:teuthology.orchestra.run.smithi150.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-11T08:09:33.480 INFO:teuthology.orchestra.run.smithi150.stdout: "monmap": "", 2024-09-11T08:09:33.480 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_async_op_threads": "3", 2024-09-11T08:09:33.480 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-11T08:09:33.480 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_async_rdma_cm": "false", 2024-09-11T08:09:33.480 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_async_rdma_device_name": "", 2024-09-11T08:09:33.480 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_async_rdma_dscp": "96", 2024-09-11T08:09:33.481 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-11T08:09:33.481 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-11T08:09:33.481 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_async_rdma_local_gid": "", 2024-09-11T08:09:33.481 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-11T08:09:33.481 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_async_rdma_port_num": "1", 2024-09-11T08:09:33.481 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-11T08:09:33.481 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-11T08:09:33.481 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-11T08:09:33.481 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-11T08:09:33.481 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_async_rdma_sl": "3", 2024-09-11T08:09:33.481 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_async_rdma_support_srq": "true", 2024-09-11T08:09:33.481 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_async_rdma_type": "ib", 2024-09-11T08:09:33.481 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_async_reap_threshold": "5", 2024-09-11T08:09:33.482 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_bind_before_connect": "false", 2024-09-11T08:09:33.482 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_bind_ipv4": "true", 2024-09-11T08:09:33.482 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_bind_ipv6": "false", 2024-09-11T08:09:33.482 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_bind_msgr1": "false", 2024-09-11T08:09:33.482 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_bind_msgr2": "true", 2024-09-11T08:09:33.482 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_bind_port_max": "7568", 2024-09-11T08:09:33.482 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_bind_port_min": "6800", 2024-09-11T08:09:33.482 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-11T08:09:33.482 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_bind_retry_count": "3", 2024-09-11T08:09:33.482 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_bind_retry_delay": "5", 2024-09-11T08:09:33.482 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_blackhole_client": "false", 2024-09-11T08:09:33.482 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_blackhole_mds": "false", 2024-09-11T08:09:33.482 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_blackhole_mgr": "false", 2024-09-11T08:09:33.482 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_blackhole_mon": "false", 2024-09-11T08:09:33.483 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_blackhole_osd": "false", 2024-09-11T08:09:33.483 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_client_mode": "crc secure", 2024-09-11T08:09:33.483 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_cluster_mode": "crc secure", 2024-09-11T08:09:33.483 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_cluster_type": "", 2024-09-11T08:09:33.483 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_compress_secure": "false", 2024-09-11T08:09:33.483 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_connection_idle_timeout": "900", 2024-09-11T08:09:33.483 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_connection_ready_timeout": "10", 2024-09-11T08:09:33.483 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_crc_data": "true", 2024-09-11T08:09:33.483 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_crc_header": "true", 2024-09-11T08:09:33.483 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_die_on_bad_msg": "false", 2024-09-11T08:09:33.483 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_die_on_bug": "true", 2024-09-11T08:09:33.483 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_die_on_old_message": "true", 2024-09-11T08:09:33.483 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_die_on_skipped_message": "false", 2024-09-11T08:09:33.483 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-11T08:09:33.484 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-11T08:09:33.484 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_dpdk_coremask": "0xF", 2024-09-11T08:09:33.484 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-11T08:09:33.484 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-11T08:09:33.484 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_dpdk_enable_tso": "true", 2024-09-11T08:09:33.484 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-11T08:09:33.484 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-11T08:09:33.484 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_dpdk_hugepages": "", 2024-09-11T08:09:33.484 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-11T08:09:33.484 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-11T08:09:33.484 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_dpdk_lro": "true", 2024-09-11T08:09:33.484 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_dpdk_memory_channel": "4", 2024-09-11T08:09:33.484 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-11T08:09:33.484 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_dpdk_pmd": "", 2024-09-11T08:09:33.485 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_dpdk_port_id": "0", 2024-09-11T08:09:33.485 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-11T08:09:33.485 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-11T08:09:33.485 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_dump_on_send": "false", 2024-09-11T08:09:33.485 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_initial_backoff": "0.200000", 2024-09-11T08:09:33.485 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_inject_delay_max": "1.000000", 2024-09-11T08:09:33.485 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_inject_delay_probability": "0.000000", 2024-09-11T08:09:33.485 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_inject_delay_type": "", 2024-09-11T08:09:33.485 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_inject_internal_delays": "0.000000", 2024-09-11T08:09:33.485 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_inject_socket_failures": "0", 2024-09-11T08:09:33.485 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_learn_addr_from_peer": "true", 2024-09-11T08:09:33.485 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_max_accept_failures": "4", 2024-09-11T08:09:33.485 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_max_backoff": "15.000000", 2024-09-11T08:09:33.486 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_mon_client_mode": "secure crc", 2024-09-11T08:09:33.486 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-11T08:09:33.486 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_mon_service_mode": "secure crc", 2024-09-11T08:09:33.486 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_osd_compress_min_size": "1024", 2024-09-11T08:09:33.486 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_osd_compress_mode": "none", 2024-09-11T08:09:33.486 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_osd_compression_algorithm": "snappy zlib zstd lz4", 2024-09-11T08:09:33.486 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-11T08:09:33.486 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_pq_min_cost": "65536", 2024-09-11T08:09:33.486 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_public_type": "", 2024-09-11T08:09:33.486 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_service_mode": "crc secure", 2024-09-11T08:09:33.486 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_tcp_listen_backlog": "512", 2024-09-11T08:09:33.486 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_tcp_nodelay": "true", 2024-09-11T08:09:33.486 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-11T08:09:33.486 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_tcp_rcvbuf": "0", 2024-09-11T08:09:33.487 INFO:teuthology.orchestra.run.smithi150.stdout: "ms_type": "async+posix", 2024-09-11T08:09:33.487 INFO:teuthology.orchestra.run.smithi150.stdout: "no_config_file": "false", 2024-09-11T08:09:33.487 INFO:teuthology.orchestra.run.smithi150.stdout: "objecter_completion_locks_per_session": "32", 2024-09-11T08:09:33.487 INFO:teuthology.orchestra.run.smithi150.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-11T08:09:33.487 INFO:teuthology.orchestra.run.smithi150.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-11T08:09:33.487 INFO:teuthology.orchestra.run.smithi150.stdout: "objecter_inflight_ops": "1024", 2024-09-11T08:09:33.487 INFO:teuthology.orchestra.run.smithi150.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-11T08:09:33.487 INFO:teuthology.orchestra.run.smithi150.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-11T08:09:33.487 INFO:teuthology.orchestra.run.smithi150.stdout: "objecter_tick_interval": "5.000000", 2024-09-11T08:09:33.487 INFO:teuthology.orchestra.run.smithi150.stdout: "objecter_timeout": "10.000000", 2024-09-11T08:09:33.487 INFO:teuthology.orchestra.run.smithi150.stdout: "objectstore_blackhole": "false", 2024-09-11T08:09:33.487 INFO:teuthology.orchestra.run.smithi150.stdout: "openssl_engine_opts": "", 2024-09-11T08:09:33.487 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_agent_delay_time": "5.000000", 2024-09-11T08:09:33.487 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_agent_hist_halflife": "1000", 2024-09-11T08:09:33.488 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_agent_max_low_ops": "2", 2024-09-11T08:09:33.488 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_agent_max_ops": "4", 2024-09-11T08:09:33.488 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-11T08:09:33.488 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-11T08:09:33.488 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_agent_slop": "0.020000", 2024-09-11T08:09:33.488 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-11T08:09:33.488 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-11T08:09:33.488 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_asio_thread_count": "2", 2024-09-11T08:09:33.488 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_async_recovery_min_cost": "100", 2024-09-11T08:09:33.488 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-11T08:09:33.488 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-11T08:09:33.489 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_backfill_scan_max": "512", 2024-09-11T08:09:33.489 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_backfill_scan_min": "64", 2024-09-11T08:09:33.489 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_backoff_on_degraded": "false", 2024-09-11T08:09:33.489 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_backoff_on_peering": "false", 2024-09-11T08:09:33.489 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_backoff_on_unfound": "true", 2024-09-11T08:09:33.489 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_beacon_report_interval": "300", 2024-09-11T08:09:33.489 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_bench_duration": "30", 2024-09-11T08:09:33.489 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-11T08:09:33.489 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_bench_max_block_size": "67108864", 2024-09-11T08:09:33.489 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-11T08:09:33.489 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_blkin_trace_all": "false", 2024-09-11T08:09:33.489 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-11T08:09:33.489 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-11T08:09:33.489 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_check_for_log_corruption": "false", 2024-09-11T08:09:33.490 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-11T08:09:33.490 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-11T08:09:33.490 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-09-11T08:09:33.490 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-11T08:09:33.490 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_class_update_on_start": "true", 2024-09-11T08:09:33.490 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_client_message_cap": "256", 2024-09-11T08:09:33.490 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_client_message_size_cap": "524288000", 2024-09-11T08:09:33.490 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_client_op_priority": "63", 2024-09-11T08:09:33.490 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_client_watch_timeout": "30", 2024-09-11T08:09:33.490 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_command_max_records": "256", 2024-09-11T08:09:33.490 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_compact_on_start": "false", 2024-09-11T08:09:33.490 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-11T08:09:33.490 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_crush_chooseleaf_type": "1", 2024-09-11T08:09:33.490 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-11T08:09:33.491 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_crush_update_on_start": "true", 2024-09-11T08:09:33.491 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_crush_update_weight_set": "true", 2024-09-11T08:09:33.491 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-11T08:09:33.491 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-11T08:09:33.491 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-11T08:09:33.491 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-11T08:09:33.491 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-11T08:09:33.491 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_debug_feed_pullee": "-1", 2024-09-11T08:09:33.491 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-11T08:09:33.491 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-11T08:09:33.491 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-11T08:09:33.491 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_debug_misdirected_ops": "false", 2024-09-11T08:09:33.491 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_debug_no_acting_change": "false", 2024-09-11T08:09:33.491 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_debug_no_purge_strays": "false", 2024-09-11T08:09:33.492 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_debug_op_order": "false", 2024-09-11T08:09:33.492 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-11T08:09:33.492 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-11T08:09:33.492 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-11T08:09:33.492 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-11T08:09:33.492 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_debug_shutdown": "false", 2024-09-11T08:09:33.492 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-11T08:09:33.492 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-11T08:09:33.492 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-11T08:09:33.492 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-11T08:09:33.492 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_debug_verify_snaps": "false", 2024-09-11T08:09:33.492 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-11T08:09:33.492 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-11T08:09:33.492 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_deep_scrub_keys": "1024", 2024-09-11T08:09:33.493 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-11T08:09:33.493 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-11T08:09:33.493 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-11T08:09:33.493 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_deep_scrub_stride": "524288", 2024-09-11T08:09:33.493 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-11T08:09:33.493 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_default_data_pool_replay_window": "45", 2024-09-11T08:09:33.493 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_default_notify_timeout": "30", 2024-09-11T08:09:33.493 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_delete_sleep": "0.000000", 2024-09-11T08:09:33.493 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-11T08:09:33.493 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-11T08:09:33.493 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-11T08:09:33.493 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_discard_disconnected_ops": "true", 2024-09-11T08:09:33.493 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_enable_op_tracker": "true", 2024-09-11T08:09:33.493 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-11T08:09:33.494 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-11T08:09:33.494 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-11T08:09:33.494 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_fast_info": "true", 2024-09-11T08:09:33.494 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_fast_shutdown": "true", 2024-09-11T08:09:33.494 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-11T08:09:33.494 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-11T08:09:33.494 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-11T08:09:33.494 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-11T08:09:33.494 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-11T08:09:33.494 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_function_tracing": "false", 2024-09-11T08:09:33.494 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_heartbeat_grace": "20", 2024-09-11T08:09:33.494 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_heartbeat_interval": "6", 2024-09-11T08:09:33.494 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-11T08:09:33.495 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_heartbeat_min_peers": "10", 2024-09-11T08:09:33.495 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_heartbeat_min_size": "2000", 2024-09-11T08:09:33.495 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_heartbeat_stale": "600", 2024-09-11T08:09:33.495 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-11T08:09:33.495 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_hit_set_max_size": "100000", 2024-09-11T08:09:33.495 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_hit_set_min_size": "1000", 2024-09-11T08:09:33.495 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-11T08:09:33.495 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-11T08:09:33.495 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-11T08:09:33.495 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-11T08:09:33.495 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-11T08:09:33.495 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-11T08:09:33.495 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_journal_size": "5120", 2024-09-11T08:09:33.495 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_kill_backfill_at": "0", 2024-09-11T08:09:33.496 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-11T08:09:33.496 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_map_cache_size": "50", 2024-09-11T08:09:33.496 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_map_dedup": "true", 2024-09-11T08:09:33.496 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_map_message_max": "40", 2024-09-11T08:09:33.496 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-11T08:09:33.496 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_map_share_max_epochs": "40", 2024-09-11T08:09:33.496 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_max_attr_name_len": "100", 2024-09-11T08:09:33.496 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_max_attr_size": "0", 2024-09-11T08:09:33.496 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_max_backfills": "1", 2024-09-11T08:09:33.496 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_max_markdown_count": "5", 2024-09-11T08:09:33.496 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_max_markdown_period": "600", 2024-09-11T08:09:33.496 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_max_object_name_len": "2048", 2024-09-11T08:09:33.496 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_max_object_namespace_len": "256", 2024-09-11T08:09:33.497 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_max_object_size": "134217728", 2024-09-11T08:09:33.497 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-11T08:09:33.497 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-11T08:09:33.497 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_max_pg_blocked_by": "16", 2024-09-11T08:09:33.497 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_max_pg_log_entries": "10000", 2024-09-11T08:09:33.497 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-11T08:09:33.497 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_max_pgls": "1024", 2024-09-11T08:09:33.497 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_max_push_cost": "8388608", 2024-09-11T08:09:33.497 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_max_push_objects": "10", 2024-09-11T08:09:33.497 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_max_scrubs": "3", 2024-09-11T08:09:33.497 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-11T08:09:33.497 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_max_trimming_pgs": "2", 2024-09-11T08:09:33.497 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_max_write_op_reply_len": "64", 2024-09-11T08:09:33.498 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_max_write_size": "90", 2024-09-11T08:09:33.498 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-11T08:09:33.498 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-11T08:09:33.498 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-11T08:09:33.498 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-11T08:09:33.498 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-11T08:09:33.498 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-11T08:09:33.498 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-11T08:09:33.498 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-11T08:09:33.498 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_mclock_profile": "balanced", 2024-09-11T08:09:33.498 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-11T08:09:33.498 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-11T08:09:33.498 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-11T08:09:33.498 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-11T08:09:33.499 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-11T08:09:33.499 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-11T08:09:33.499 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-11T08:09:33.499 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-11T08:09:33.499 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-11T08:09:33.499 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-11T08:09:33.499 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-11T08:09:33.499 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_memory_base": "805306368", 2024-09-11T08:09:33.499 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_memory_cache_min": "134217728", 2024-09-11T08:09:33.499 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-11T08:09:33.499 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-11T08:09:33.499 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_memory_target": "4294967296", 2024-09-11T08:09:33.499 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_memory_target_autotune": "false", 2024-09-11T08:09:33.500 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-11T08:09:33.500 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_min_pg_log_entries": "250", 2024-09-11T08:09:33.500 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_min_recovery_priority": "0", 2024-09-11T08:09:33.500 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-11T08:09:33.500 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-11T08:09:33.500 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_mon_report_interval": "5", 2024-09-11T08:09:33.500 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-11T08:09:33.500 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-11T08:09:33.500 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_num_cache_shards": "32", 2024-09-11T08:09:33.500 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_num_op_tracker_shard": "32", 2024-09-11T08:09:33.500 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_numa_auto_affinity": "true", 2024-09-11T08:09:33.500 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_numa_node": "-1", 2024-09-11T08:09:33.500 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_numa_prefer_iface": "true", 2024-09-11T08:09:33.500 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-09-11T08:09:33.501 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_objecter_finishers": "1", 2024-09-11T08:09:33.501 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_objectstore": "bluestore", 2024-09-11T08:09:33.501 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_objectstore_fuse": "false", 2024-09-11T08:09:33.501 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_objectstore_tracing": "false", 2024-09-11T08:09:33.501 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_op_complaint_time": "30.000000", 2024-09-11T08:09:33.501 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_op_history_duration": "600", 2024-09-11T08:09:33.501 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_op_history_size": "20", 2024-09-11T08:09:33.501 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_op_history_slow_op_size": "20", 2024-09-11T08:09:33.501 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-11T08:09:33.501 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_op_log_threshold": "5", 2024-09-11T08:09:33.501 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_op_num_shards": "0", 2024-09-11T08:09:33.501 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_op_num_shards_hdd": "5", 2024-09-11T08:09:33.501 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_op_num_shards_ssd": "8", 2024-09-11T08:09:33.502 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-11T08:09:33.502 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-11T08:09:33.502 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-11T08:09:33.502 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-11T08:09:33.502 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_op_pq_min_cost": "65536", 2024-09-11T08:09:33.502 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-11T08:09:33.502 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_op_queue_cut_off": "high", 2024-09-11T08:09:33.502 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-11T08:09:33.502 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_op_thread_timeout": "15", 2024-09-11T08:09:33.502 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_open_classes_on_start": "true", 2024-09-11T08:09:33.502 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_os_flags": "0", 2024-09-11T08:09:33.502 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_peering_op_priority": "255", 2024-09-11T08:09:33.502 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pg_delete_cost": "1048576", 2024-09-11T08:09:33.502 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pg_delete_priority": "5", 2024-09-11T08:09:33.503 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-11T08:09:33.503 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-11T08:09:33.503 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-11T08:09:33.503 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pg_log_trim_max": "10000", 2024-09-11T08:09:33.503 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pg_log_trim_min": "100", 2024-09-11T08:09:33.503 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-11T08:09:33.503 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-11T08:09:33.503 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pg_stat_report_interval_max": "500", 2024-09-11T08:09:33.503 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-11T08:09:33.503 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-11T08:09:33.503 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-11T08:09:33.503 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-11T08:09:33.503 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-11T08:09:33.503 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-11T08:09:33.504 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-11T08:09:33.504 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-11T08:09:33.504 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-11T08:09:33.504 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-11T08:09:33.504 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-11T08:09:33.504 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-11T08:09:33.504 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-11T08:09:33.504 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-11T08:09:33.504 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pool_default_flags": "0", 2024-09-11T08:09:33.504 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-11T08:09:33.504 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pool_default_min_size": "0", 2024-09-11T08:09:33.504 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-11T08:09:33.504 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pool_default_pg_num": "32", 2024-09-11T08:09:33.504 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pool_default_pgp_num": "0", 2024-09-11T08:09:33.505 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-11T08:09:33.505 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pool_default_size": "3", 2024-09-11T08:09:33.505 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pool_default_type": "replicated", 2024-09-11T08:09:33.505 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-11T08:09:33.505 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-11T08:09:33.505 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_push_per_object_cost": "1000", 2024-09-11T08:09:33.505 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-11T08:09:33.505 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_recover_clone_overlap": "true", 2024-09-11T08:09:33.505 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-11T08:09:33.505 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_recovery_cost": "20971520", 2024-09-11T08:09:33.505 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-11T08:09:33.505 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_recovery_max_active": "0", 2024-09-11T08:09:33.505 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-11T08:09:33.505 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-11T08:09:33.506 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-11T08:09:33.506 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-11T08:09:33.506 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_recovery_max_single_start": "1", 2024-09-11T08:09:33.506 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_recovery_op_priority": "3", 2024-09-11T08:09:33.506 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-11T08:09:33.506 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_recovery_priority": "5", 2024-09-11T08:09:33.506 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-11T08:09:33.506 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_recovery_sleep": "0.000000", 2024-09-11T08:09:33.506 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-11T08:09:33.506 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-11T08:09:33.506 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-11T08:09:33.506 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_repair_during_recovery": "false", 2024-09-11T08:09:33.506 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_requested_scrub_priority": "120", 2024-09-11T08:09:33.506 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-11T08:09:33.507 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-11T08:09:33.507 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_scrub_auto_repair": "false", 2024-09-11T08:09:33.507 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-11T08:09:33.507 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-11T08:09:33.507 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_scrub_begin_hour": "0", 2024-09-11T08:09:33.507 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_scrub_begin_week_day": "0", 2024-09-11T08:09:33.507 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_scrub_chunk_max": "25", 2024-09-11T08:09:33.507 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_scrub_chunk_min": "5", 2024-09-11T08:09:33.507 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_scrub_cost": "52428800", 2024-09-11T08:09:33.507 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_scrub_during_recovery": "false", 2024-09-11T08:09:33.507 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_scrub_end_hour": "0", 2024-09-11T08:09:33.507 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_scrub_end_week_day": "0", 2024-09-11T08:09:33.507 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_scrub_event_cost": "4096", 2024-09-11T08:09:33.507 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-11T08:09:33.508 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-11T08:09:33.508 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_scrub_invalid_stats": "true", 2024-09-11T08:09:33.508 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-11T08:09:33.508 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-11T08:09:33.508 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_scrub_max_preemptions": "5", 2024-09-11T08:09:33.508 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-11T08:09:33.508 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_scrub_priority": "5", 2024-09-11T08:09:33.508 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_scrub_sleep": "0.000000", 2024-09-11T08:09:33.508 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-11T08:09:33.508 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_skip_data_digest": "false", 2024-09-11T08:09:33.508 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_smart_report_timeout": "5", 2024-09-11T08:09:33.508 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_snap_trim_cost": "1048576", 2024-09-11T08:09:33.508 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_snap_trim_priority": "5", 2024-09-11T08:09:33.509 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-11T08:09:33.509 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-11T08:09:33.509 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-11T08:09:33.509 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-11T08:09:33.509 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-11T08:09:33.509 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_target_transaction_size": "30", 2024-09-11T08:09:33.509 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-11T08:09:33.509 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-11T08:09:33.509 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-11T08:09:33.509 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-11T08:09:33.509 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-11T08:09:33.509 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-11T08:09:33.509 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-11T08:09:33.510 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-11T08:09:33.510 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-11T08:09:33.510 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-11T08:09:33.510 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_tracing": "false", 2024-09-11T08:09:33.510 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_use_stale_snap": "false", 2024-09-11T08:09:33.510 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-11T08:09:33.510 INFO:teuthology.orchestra.run.smithi150.stdout: "osdc_blkin_trace_all": "false", 2024-09-11T08:09:33.510 INFO:teuthology.orchestra.run.smithi150.stdout: "paxos_kill_at": "0", 2024-09-11T08:09:33.510 INFO:teuthology.orchestra.run.smithi150.stdout: "paxos_max_join_drift": "10", 2024-09-11T08:09:33.510 INFO:teuthology.orchestra.run.smithi150.stdout: "paxos_min": "500", 2024-09-11T08:09:33.510 INFO:teuthology.orchestra.run.smithi150.stdout: "paxos_min_wait": "0.050000", 2024-09-11T08:09:33.510 INFO:teuthology.orchestra.run.smithi150.stdout: "paxos_propose_interval": "1.000000", 2024-09-11T08:09:33.510 INFO:teuthology.orchestra.run.smithi150.stdout: "paxos_service_trim_max": "500", 2024-09-11T08:09:33.510 INFO:teuthology.orchestra.run.smithi150.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-11T08:09:33.511 INFO:teuthology.orchestra.run.smithi150.stdout: "paxos_service_trim_min": "10", 2024-09-11T08:09:33.511 INFO:teuthology.orchestra.run.smithi150.stdout: "paxos_stash_full_interval": "25", 2024-09-11T08:09:33.511 INFO:teuthology.orchestra.run.smithi150.stdout: "paxos_trim_max": "500", 2024-09-11T08:09:33.511 INFO:teuthology.orchestra.run.smithi150.stdout: "paxos_trim_min": "250", 2024-09-11T08:09:33.511 INFO:teuthology.orchestra.run.smithi150.stdout: "perf": "true", 2024-09-11T08:09:33.511 INFO:teuthology.orchestra.run.smithi150.stdout: "pid_file": "", 2024-09-11T08:09:33.511 INFO:teuthology.orchestra.run.smithi150.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-11T08:09:33.511 INFO:teuthology.orchestra.run.smithi150.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-09-11T08:09:33.511 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addr": "-", 2024-09-11T08:09:33.511 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addrv": "", 2024-09-11T08:09:33.511 INFO:teuthology.orchestra.run.smithi150.stdout: "public_bind_addr": "-", 2024-09-11T08:09:33.511 INFO:teuthology.orchestra.run.smithi150.stdout: "public_network": "", 2024-09-11T08:09:33.511 INFO:teuthology.orchestra.run.smithi150.stdout: "public_network_interface": "", 2024-09-11T08:09:33.511 INFO:teuthology.orchestra.run.smithi150.stdout: "qat_compressor_enabled": "false", 2024-09-11T08:09:33.512 INFO:teuthology.orchestra.run.smithi150.stdout: "rados_mon_op_timeout": "0", 2024-09-11T08:09:33.512 INFO:teuthology.orchestra.run.smithi150.stdout: "rados_osd_op_timeout": "0", 2024-09-11T08:09:33.512 INFO:teuthology.orchestra.run.smithi150.stdout: "rados_tracing": "false", 2024-09-11T08:09:33.512 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_atime_update_interval": "60", 2024-09-11T08:09:33.512 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-11T08:09:33.512 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_balance_parent_reads": "false", 2024-09-11T08:09:33.512 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_balance_snap_reads": "false", 2024-09-11T08:09:33.512 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_blkin_trace_all": "false", 2024-09-11T08:09:33.512 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-11T08:09:33.512 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-11T08:09:33.512 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_cache": "true", 2024-09-11T08:09:33.512 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-11T08:09:33.512 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-11T08:09:33.512 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-11T08:09:33.513 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-11T08:09:33.513 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_cache_policy": "writearound", 2024-09-11T08:09:33.513 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_cache_size": "33554432", 2024-09-11T08:09:33.513 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-11T08:09:33.513 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-11T08:09:33.513 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_clone_copy_on_read": "false", 2024-09-11T08:09:33.513 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_compression_hint": "none", 2024-09-11T08:09:33.513 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_concurrent_management_ops": "10", 2024-09-11T08:09:33.513 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-11T08:09:33.513 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_default_clone_format": "auto", 2024-09-11T08:09:33.513 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_default_data_pool": "", 2024-09-11T08:09:33.513 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_default_features": "61", 2024-09-11T08:09:33.513 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_default_format": "2", 2024-09-11T08:09:33.513 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_default_map_options": "", 2024-09-11T08:09:33.514 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_default_order": "22", 2024-09-11T08:09:33.514 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_default_pool": "rbd", 2024-09-11T08:09:33.514 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-11T08:09:33.514 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_default_stripe_count": "0", 2024-09-11T08:09:33.514 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_default_stripe_unit": "0", 2024-09-11T08:09:33.514 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-11T08:09:33.514 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-11T08:09:33.514 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-11T08:09:33.514 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_enable_alloc_hint": "true", 2024-09-11T08:09:33.514 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-11T08:09:33.514 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_io_scheduler": "simple", 2024-09-11T08:09:33.514 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-11T08:09:33.514 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-11T08:09:33.515 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-11T08:09:33.515 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-11T08:09:33.515 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-11T08:09:33.515 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-11T08:09:33.515 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-11T08:09:33.515 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-11T08:09:33.515 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-11T08:09:33.515 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_journal_order": "24", 2024-09-11T08:09:33.515 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_journal_pool": "", 2024-09-11T08:09:33.515 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_journal_splay_width": "4", 2024-09-11T08:09:33.515 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_localize_parent_reads": "false", 2024-09-11T08:09:33.515 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_localize_snap_reads": "false", 2024-09-11T08:09:33.515 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-11T08:09:33.515 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-11T08:09:33.515 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-11T08:09:33.516 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-11T08:09:33.516 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-11T08:09:33.516 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-11T08:09:33.516 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-11T08:09:33.516 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-11T08:09:33.516 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-11T08:09:33.516 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-11T08:09:33.516 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-11T08:09:33.516 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-11T08:09:33.516 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-11T08:09:33.516 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-11T08:09:33.516 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-11T08:09:33.516 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-11T08:09:33.517 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-11T08:09:33.517 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-11T08:09:33.517 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-11T08:09:33.517 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-11T08:09:33.517 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-11T08:09:33.517 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-11T08:09:33.517 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-11T08:09:33.517 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-11T08:09:33.517 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-11T08:09:33.517 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-11T08:09:33.517 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-11T08:09:33.517 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-11T08:09:33.517 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-11T08:09:33.518 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-11T08:09:33.518 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-11T08:09:33.518 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_mtime_update_interval": "60", 2024-09-11T08:09:33.518 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_non_blocking_aio": "true", 2024-09-11T08:09:33.518 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_op_thread_timeout": "60", 2024-09-11T08:09:33.518 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_op_threads": "1", 2024-09-11T08:09:33.518 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_parent_cache_enabled": "false", 2024-09-11T08:09:33.518 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-11T08:09:33.518 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-11T08:09:33.518 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-11T08:09:33.518 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_plugins": "", 2024-09-11T08:09:33.518 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_qos_bps_burst": "0", 2024-09-11T08:09:33.518 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-11T08:09:33.518 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_qos_bps_limit": "0", 2024-09-11T08:09:33.518 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_qos_exclude_ops": "0", 2024-09-11T08:09:33.519 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_qos_iops_burst": "0", 2024-09-11T08:09:33.519 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-11T08:09:33.519 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_qos_iops_limit": "0", 2024-09-11T08:09:33.519 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-11T08:09:33.519 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-11T08:09:33.519 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-11T08:09:33.519 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-11T08:09:33.519 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-11T08:09:33.519 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-11T08:09:33.519 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-11T08:09:33.519 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-11T08:09:33.519 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-11T08:09:33.519 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-11T08:09:33.519 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-11T08:09:33.520 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-11T08:09:33.520 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-11T08:09:33.520 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-11T08:09:33.520 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_read_from_replica_policy": "default", 2024-09-11T08:09:33.520 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-11T08:09:33.520 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-11T08:09:33.520 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-11T08:09:33.520 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-11T08:09:33.520 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_skip_partial_discard": "true", 2024-09-11T08:09:33.520 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-11T08:09:33.520 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_tracing": "false", 2024-09-11T08:09:33.520 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_validate_names": "true", 2024-09-11T08:09:33.520 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_validate_pool": "true", 2024-09-11T08:09:33.521 INFO:teuthology.orchestra.run.smithi150.stdout: "restapi_base_url": "", 2024-09-11T08:09:33.521 INFO:teuthology.orchestra.run.smithi150.stdout: "restapi_log_level": "", 2024-09-11T08:09:33.521 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_acl_grants_max_num": "100", 2024-09-11T08:09:33.521 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_admin_entry": "admin", 2024-09-11T08:09:33.521 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_backend_store": "rados", 2024-09-11T08:09:33.521 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_barbican_url": "", 2024-09-11T08:09:33.521 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_beast_enable_async": "true", 2024-09-11T08:09:33.521 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-11T08:09:33.521 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-11T08:09:33.521 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-11T08:09:33.521 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-11T08:09:33.521 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-11T08:09:33.521 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_cache_enabled": "true", 2024-09-11T08:09:33.521 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_cache_expiry_interval": "900", 2024-09-11T08:09:33.521 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_cache_lru_size": "10000", 2024-09-11T08:09:33.522 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_content_length_compat": "false", 2024-09-11T08:09:33.522 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_copy_obj_progress": "true", 2024-09-11T08:09:33.522 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-11T08:09:33.522 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_cors_rules_max_num": "100", 2024-09-11T08:09:33.522 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_cross_domain_policy": "", 2024-09-11T08:09:33.522 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-11T08:09:33.522 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_crypt_kmip_addr": "", 2024-09-11T08:09:33.522 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-11T08:09:33.522 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-11T08:09:33.522 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-11T08:09:33.522 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-11T08:09:33.522 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_crypt_kmip_password": "", 2024-09-11T08:09:33.522 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-11T08:09:33.523 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_crypt_kmip_username": "", 2024-09-11T08:09:33.523 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_crypt_require_ssl": "true", 2024-09-11T08:09:33.523 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-11T08:09:33.523 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-11T08:09:33.523 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-11T08:09:33.523 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-11T08:09:33.523 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-11T08:09:33.523 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-11T08:09:33.523 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-11T08:09:33.523 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-11T08:09:33.523 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-11T08:09:33.523 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-11T08:09:33.523 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-11T08:09:33.523 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-11T08:09:33.524 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-11T08:09:33.524 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-11T08:09:33.524 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-11T08:09:33.524 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_crypt_vault_addr": "", 2024-09-11T08:09:33.524 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_crypt_vault_auth": "token", 2024-09-11T08:09:33.524 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_crypt_vault_namespace": "", 2024-09-11T08:09:33.524 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_crypt_vault_prefix": "", 2024-09-11T08:09:33.524 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-11T08:09:33.524 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-11T08:09:33.524 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-11T08:09:33.524 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-11T08:09:33.524 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_crypt_vault_token_file": "", 2024-09-11T08:09:33.524 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-11T08:09:33.525 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_curl_buffersize": "524288", 2024-09-11T08:09:33.525 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-11T08:09:33.525 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_curl_low_speed_time": "300", 2024-09-11T08:09:33.525 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-11T08:09:33.525 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-11T08:09:33.525 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-11T08:09:33.525 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-11T08:09:33.525 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-11T08:09:33.525 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-11T08:09:33.525 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-11T08:09:33.525 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-11T08:09:33.525 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-11T08:09:33.525 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-11T08:09:33.525 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_data_log_changes_size": "1000", 2024-09-11T08:09:33.526 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_data_log_num_shards": "128", 2024-09-11T08:09:33.526 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-11T08:09:33.526 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_data_log_window": "30", 2024-09-11T08:09:33.526 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_data_notify_interval_msec": "200", 2024-09-11T08:09:33.526 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-11T08:09:33.526 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-11T08:09:33.526 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-11T08:09:33.526 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-11T08:09:33.526 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-11T08:09:33.526 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-11T08:09:33.526 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-11T08:09:33.526 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-11T08:09:33.526 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-11T08:09:33.526 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-11T08:09:33.527 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-11T08:09:33.527 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-11T08:09:33.527 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-11T08:09:33.527 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-11T08:09:33.527 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-11T08:09:33.527 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-11T08:09:33.527 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-11T08:09:33.527 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-11T08:09:33.527 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-11T08:09:33.527 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-11T08:09:33.527 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-11T08:09:33.527 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-11T08:09:33.527 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_dns_name": "", 2024-09-11T08:09:33.527 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_dns_s3website_name": "", 2024-09-11T08:09:33.528 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_dynamic_resharding": "true", 2024-09-11T08:09:33.528 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-11T08:09:33.528 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_enable_gc_threads": "true", 2024-09-11T08:09:33.528 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_enable_lc_threads": "true", 2024-09-11T08:09:33.528 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_enable_ops_log": "false", 2024-09-11T08:09:33.528 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_enable_quota_threads": "true", 2024-09-11T08:09:33.528 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_enable_static_website": "false", 2024-09-11T08:09:33.528 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_enable_usage_log": "false", 2024-09-11T08:09:33.528 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_enforce_swift_acls": "true", 2024-09-11T08:09:33.528 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_exit_timeout_secs": "120", 2024-09-11T08:09:33.528 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_expose_bucket": "false", 2024-09-11T08:09:33.528 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_extended_http_attrs": "", 2024-09-11T08:09:33.528 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-11T08:09:33.528 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_frontends": "beast port=7480", 2024-09-11T08:09:33.529 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-11T08:09:33.529 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_gc_max_deferred": "50", 2024-09-11T08:09:33.529 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-11T08:09:33.529 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_gc_max_objs": "32", 2024-09-11T08:09:33.529 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-11T08:09:33.529 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-11T08:09:33.529 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-11T08:09:33.529 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-11T08:09:33.529 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_gc_processor_period": "3600", 2024-09-11T08:09:33.529 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-11T08:09:33.529 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-11T08:09:33.529 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-11T08:09:33.529 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-11T08:09:33.529 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_init_timeout": "300", 2024-09-11T08:09:33.530 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-11T08:09:33.530 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_kafka_connection_idle": "300", 2024-09-11T08:09:33.530 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-11T08:09:33.530 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-11T08:09:33.530 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-11T08:09:33.530 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_keystone_admin_domain": "", 2024-09-11T08:09:33.530 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_keystone_admin_password": "", 2024-09-11T08:09:33.530 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_keystone_admin_password_path": "", 2024-09-11T08:09:33.530 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_keystone_admin_project": "", 2024-09-11T08:09:33.530 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_keystone_admin_tenant": "", 2024-09-11T08:09:33.530 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_keystone_admin_token": "", 2024-09-11T08:09:33.530 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_keystone_admin_token_path": "", 2024-09-11T08:09:33.530 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_keystone_admin_user": "", 2024-09-11T08:09:33.530 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_keystone_api_version": "2", 2024-09-11T08:09:33.531 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_keystone_barbican_domain": "", 2024-09-11T08:09:33.531 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_keystone_barbican_password": "", 2024-09-11T08:09:33.531 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_keystone_barbican_project": "", 2024-09-11T08:09:33.531 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-11T08:09:33.531 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_keystone_barbican_user": "", 2024-09-11T08:09:33.531 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-11T08:09:33.531 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-11T08:09:33.531 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-11T08:09:33.531 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-11T08:09:33.531 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-11T08:09:33.531 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_keystone_url": "", 2024-09-11T08:09:33.531 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-11T08:09:33.531 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_lc_debug_interval": "-1", 2024-09-11T08:09:33.531 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_lc_lock_max_time": "90", 2024-09-11T08:09:33.532 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_lc_max_objs": "32", 2024-09-11T08:09:33.532 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_lc_max_rules": "1000", 2024-09-11T08:09:33.532 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_lc_max_worker": "3", 2024-09-11T08:09:33.532 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-11T08:09:33.532 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_lc_thread_delay": "0", 2024-09-11T08:09:33.532 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-11T08:09:33.532 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_ldap_dnattr": "uid", 2024-09-11T08:09:33.532 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-11T08:09:33.532 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_ldap_searchfilter": "", 2024-09-11T08:09:33.532 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-11T08:09:33.532 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_ldap_uri": "", 2024-09-11T08:09:33.532 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-11T08:09:33.532 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-11T08:09:33.532 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-11T08:09:33.533 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_log_http_headers": "", 2024-09-11T08:09:33.533 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-11T08:09:33.533 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-11T08:09:33.533 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_log_object_name_utc": "false", 2024-09-11T08:09:33.533 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_luarocks_location": "/tmp/luarocks", 2024-09-11T08:09:33.533 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_max_attr_name_len": "0", 2024-09-11T08:09:33.533 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_max_attr_size": "0", 2024-09-11T08:09:33.533 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-11T08:09:33.533 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_max_chunk_size": "4194304", 2024-09-11T08:09:33.533 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-11T08:09:33.533 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-11T08:09:33.533 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_max_listing_results": "1000", 2024-09-11T08:09:33.533 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_max_notify_retries": "10", 2024-09-11T08:09:33.533 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-11T08:09:33.534 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_max_put_param_size": "1048576", 2024-09-11T08:09:33.534 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_max_put_size": "5368709120", 2024-09-11T08:09:33.534 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_max_slo_entries": "1000", 2024-09-11T08:09:33.534 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_md_log_max_shards": "64", 2024-09-11T08:09:33.534 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-11T08:09:33.534 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-11T08:09:33.534 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_mp_lock_max_time": "600", 2024-09-11T08:09:33.534 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-11T08:09:33.534 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-11T08:09:33.534 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-11T08:09:33.534 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-11T08:09:33.534 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-11T08:09:33.534 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-11T08:09:33.534 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-11T08:09:33.535 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_nfs_max_gc": "300", 2024-09-11T08:09:33.535 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-11T08:09:33.535 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-11T08:09:33.535 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-11T08:09:33.535 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-11T08:09:33.535 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-11T08:09:33.535 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-11T08:09:33.535 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-11T08:09:33.535 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_num_async_rados_threads": "32", 2024-09-11T08:09:33.535 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_num_control_oids": "8", 2024-09-11T08:09:33.535 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_numa_node": "-1", 2024-09-11T08:09:33.535 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-11T08:09:33.535 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-11T08:09:33.535 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_objexp_chunk_size": "100", 2024-09-11T08:09:33.535 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_objexp_gc_interval": "600", 2024-09-11T08:09:33.536 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-11T08:09:33.536 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-11T08:09:33.536 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-11T08:09:33.536 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_op_thread_timeout": "600", 2024-09-11T08:09:33.536 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_op_tracing": "false", 2024-09-11T08:09:33.536 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_opa_token": "", 2024-09-11T08:09:33.536 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_opa_url": "", 2024-09-11T08:09:33.536 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_opa_verify_ssl": "true", 2024-09-11T08:09:33.536 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-11T08:09:33.536 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_ops_log_file_path": "", 2024-09-11T08:09:33.536 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_ops_log_rados": "true", 2024-09-11T08:09:33.536 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_ops_log_socket_path": "", 2024-09-11T08:09:33.536 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-11T08:09:33.536 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-11T08:09:33.537 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_period_push_interval": "2.000000", 2024-09-11T08:09:33.537 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-11T08:09:33.537 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-11T08:09:33.537 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_print_continue": "true", 2024-09-11T08:09:33.537 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-11T08:09:33.537 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-11T08:09:33.537 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-11T08:09:33.537 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-11T08:09:33.537 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-11T08:09:33.537 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_rados_tracing": "false", 2024-09-11T08:09:33.537 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_realm": "", 2024-09-11T08:09:33.537 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_realm_id": "", 2024-09-11T08:09:33.537 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-11T08:09:33.537 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_region": "", 2024-09-11T08:09:33.538 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-11T08:09:33.538 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-11T08:09:33.538 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-11T08:09:33.538 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-11T08:09:33.538 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_request_uri": "", 2024-09-11T08:09:33.538 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_reshard_batch_size": "64", 2024-09-11T08:09:33.538 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-11T08:09:33.538 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_reshard_max_aio": "128", 2024-09-11T08:09:33.538 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_reshard_num_logs": "16", 2024-09-11T08:09:33.538 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_reshard_thread_interval": "600", 2024-09-11T08:09:33.538 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_resolve_cname": "false", 2024-09-11T08:09:33.538 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-11T08:09:33.538 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_run_sync_thread": "true", 2024-09-11T08:09:33.539 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-11T08:09:33.539 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-11T08:09:33.539 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-11T08:09:33.539 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-11T08:09:33.539 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-11T08:09:33.539 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-11T08:09:33.539 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-11T08:09:33.539 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-11T08:09:33.539 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-11T08:09:33.539 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_scheduler_type": "throttler", 2024-09-11T08:09:33.539 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_script_uri": "", 2024-09-11T08:09:33.539 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_service_provider_name": "", 2024-09-11T08:09:33.539 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-11T08:09:33.539 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_sts_client_id": "", 2024-09-11T08:09:33.540 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_sts_client_secret": "", 2024-09-11T08:09:33.540 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_sts_entry": "sts", 2024-09-11T08:09:33.540 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_sts_key": "sts", 2024-09-11T08:09:33.540 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-11T08:09:33.540 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_sts_min_session_duration": "900", 2024-09-11T08:09:33.540 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_sts_token_introspection_url": "", 2024-09-11T08:09:33.540 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_swift_account_in_url": "false", 2024-09-11T08:09:33.540 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_swift_auth_entry": "auth", 2024-09-11T08:09:33.540 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_swift_auth_url": "", 2024-09-11T08:09:33.540 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_swift_custom_header": "", 2024-09-11T08:09:33.540 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-11T08:09:33.540 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_swift_need_stats": "true", 2024-09-11T08:09:33.540 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_swift_tenant_name": "", 2024-09-11T08:09:33.540 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_swift_token_expiration": "86400", 2024-09-11T08:09:33.541 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_swift_url": "", 2024-09-11T08:09:33.541 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_swift_url_prefix": "swift", 2024-09-11T08:09:33.541 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-11T08:09:33.541 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-11T08:09:33.541 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_sync_lease_period": "120", 2024-09-11T08:09:33.541 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-11T08:09:33.541 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-11T08:09:33.541 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-11T08:09:33.541 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-11T08:09:33.541 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-11T08:09:33.541 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-11T08:09:33.541 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-11T08:09:33.541 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-11T08:09:33.541 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-11T08:09:33.542 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_thread_pool_size": "512", 2024-09-11T08:09:33.542 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_torrent_comment": "", 2024-09-11T08:09:33.542 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_torrent_createby": "", 2024-09-11T08:09:33.542 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_torrent_encoding": "", 2024-09-11T08:09:33.542 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_torrent_flag": "false", 2024-09-11T08:09:33.542 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_torrent_origin": "", 2024-09-11T08:09:33.542 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-11T08:09:33.542 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_torrent_tracker": "", 2024-09-11T08:09:33.542 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_trust_forwarded_https": "false", 2024-09-11T08:09:33.542 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-11T08:09:33.542 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-11T08:09:33.542 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_usage_max_shards": "32", 2024-09-11T08:09:33.542 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_usage_max_user_shards": "1", 2024-09-11T08:09:33.542 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_use_opa_authz": "false", 2024-09-11T08:09:33.543 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-11T08:09:33.543 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-11T08:09:33.543 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_user_max_buckets": "1000", 2024-09-11T08:09:33.543 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-11T08:09:33.543 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-11T08:09:33.543 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-11T08:09:33.543 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-11T08:09:33.543 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_user_unique_email": "true", 2024-09-11T08:09:33.543 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_verify_ssl": "true", 2024-09-11T08:09:33.543 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-11T08:09:33.543 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_zone": "", 2024-09-11T08:09:33.543 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_zone_id": "", 2024-09-11T08:09:33.543 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-11T08:09:33.543 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_zonegroup": "", 2024-09-11T08:09:33.544 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_zonegroup_id": "", 2024-09-11T08:09:33.544 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-11T08:09:33.544 INFO:teuthology.orchestra.run.smithi150.stdout: "rocksdb_block_size": "4096", 2024-09-11T08:09:33.544 INFO:teuthology.orchestra.run.smithi150.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-11T08:09:33.544 INFO:teuthology.orchestra.run.smithi150.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-11T08:09:33.544 INFO:teuthology.orchestra.run.smithi150.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-11T08:09:33.544 INFO:teuthology.orchestra.run.smithi150.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-11T08:09:33.544 INFO:teuthology.orchestra.run.smithi150.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-11T08:09:33.544 INFO:teuthology.orchestra.run.smithi150.stdout: "rocksdb_cache_size": "536870912", 2024-09-11T08:09:33.544 INFO:teuthology.orchestra.run.smithi150.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-11T08:09:33.544 INFO:teuthology.orchestra.run.smithi150.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-11T08:09:33.544 INFO:teuthology.orchestra.run.smithi150.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-11T08:09:33.544 INFO:teuthology.orchestra.run.smithi150.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-11T08:09:33.544 INFO:teuthology.orchestra.run.smithi150.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-11T08:09:33.545 INFO:teuthology.orchestra.run.smithi150.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-11T08:09:33.545 INFO:teuthology.orchestra.run.smithi150.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-11T08:09:33.545 INFO:teuthology.orchestra.run.smithi150.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-11T08:09:33.545 INFO:teuthology.orchestra.run.smithi150.stdout: "rocksdb_index_type": "binary_search", 2024-09-11T08:09:33.545 INFO:teuthology.orchestra.run.smithi150.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-11T08:09:33.545 INFO:teuthology.orchestra.run.smithi150.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-11T08:09:33.545 INFO:teuthology.orchestra.run.smithi150.stdout: "rocksdb_partition_filters": "false", 2024-09-11T08:09:33.545 INFO:teuthology.orchestra.run.smithi150.stdout: "rocksdb_perf": "false", 2024-09-11T08:09:33.545 INFO:teuthology.orchestra.run.smithi150.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-11T08:09:33.545 INFO:teuthology.orchestra.run.smithi150.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-11T08:09:33.545 INFO:teuthology.orchestra.run.smithi150.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-11T08:09:33.545 INFO:teuthology.orchestra.run.smithi150.stdout: "run_dir": "/var/run/ceph", 2024-09-11T08:09:33.545 INFO:teuthology.orchestra.run.smithi150.stdout: "seastore_block_create": "true", 2024-09-11T08:09:33.546 INFO:teuthology.orchestra.run.smithi150.stdout: "seastore_cache_lru_size": "67108864", 2024-09-11T08:09:33.546 INFO:teuthology.orchestra.run.smithi150.stdout: "seastore_default_max_object_size": "16777216", 2024-09-11T08:09:33.546 INFO:teuthology.orchestra.run.smithi150.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-11T08:09:33.546 INFO:teuthology.orchestra.run.smithi150.stdout: "seastore_device_size": "107374182400", 2024-09-11T08:09:33.546 INFO:teuthology.orchestra.run.smithi150.stdout: "seastore_init_rewrite_segments_num_per_device": "6", 2024-09-11T08:09:33.546 INFO:teuthology.orchestra.run.smithi150.stdout: "seastore_journal_batch_capacity": "16", 2024-09-11T08:09:33.546 INFO:teuthology.orchestra.run.smithi150.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-11T08:09:33.546 INFO:teuthology.orchestra.run.smithi150.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-11T08:09:33.546 INFO:teuthology.orchestra.run.smithi150.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-11T08:09:33.546 INFO:teuthology.orchestra.run.smithi150.stdout: "seastore_segment_size": "67108864", 2024-09-11T08:09:33.546 INFO:teuthology.orchestra.run.smithi150.stdout: "setgroup": "ceph", 2024-09-11T08:09:33.546 INFO:teuthology.orchestra.run.smithi150.stdout: "setuser": "ceph", 2024-09-11T08:09:33.546 INFO:teuthology.orchestra.run.smithi150.stdout: "setuser_match_path": "", 2024-09-11T08:09:33.546 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-11T08:09:33.547 INFO:teuthology.orchestra.run.smithi150.stdout: "thp": "false", 2024-09-11T08:09:33.547 INFO:teuthology.orchestra.run.smithi150.stdout: "threadpool_default_timeout": "60", 2024-09-11T08:09:33.547 INFO:teuthology.orchestra.run.smithi150.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-11T08:09:33.547 INFO:teuthology.orchestra.run.smithi150.stdout: "throttler_perf_counter": "true" 2024-09-11T08:09:33.547 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-11T08:09:34.208 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:09:33 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-alertmanager-a[75052]: ts=2024-09-11T08:09:33.856Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000508289s 2024-09-11T08:09:34.322 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-11T08:09:34.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:34 smithi150 ceph-mon[29276]: pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-11T08:09:34.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:34 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:34.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:34 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:34.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:34 smithi150 ceph-mon[29276]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-11T08:09:34.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:34 smithi150 ceph-mon[29276]: Reconfiguring daemon prometheus.a on smithi181 2024-09-11T08:09:34.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:34 smithi150 ceph-mon[39612]: pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-11T08:09:34.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:34 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:34.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:34 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:34.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:34 smithi150 ceph-mon[39612]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-11T08:09:34.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:34 smithi150 ceph-mon[39612]: Reconfiguring daemon prometheus.a on smithi181 2024-09-11T08:09:34.569 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:09:34.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:34 smithi181 ceph-mon[32701]: pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-11T08:09:34.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:34 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:34.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:34 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:34.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:34 smithi181 ceph-mon[32701]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-11T08:09:34.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:34 smithi181 ceph-mon[32701]: Reconfiguring daemon prometheus.a on smithi181 2024-09-11T08:09:35.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:35 smithi181 ceph-mon[32701]: 6.13 deep-scrub starts 2024-09-11T08:09:35.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:35 smithi181 ceph-mon[32701]: 6.13 deep-scrub ok 2024-09-11T08:09:35.324 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:09:35 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:09:35] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:09:35.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:35 smithi150 ceph-mon[29276]: 6.13 deep-scrub starts 2024-09-11T08:09:35.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:35 smithi150 ceph-mon[29276]: 6.13 deep-scrub ok 2024-09-11T08:09:35.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:35 smithi150 ceph-mon[39612]: 6.13 deep-scrub starts 2024-09-11T08:09:35.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:35 smithi150 ceph-mon[39612]: 6.13 deep-scrub ok 2024-09-11T08:09:36.323 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:09:36 smithi181 systemd[1]: Stopping Ceph prometheus.a for c1bddff4-7013-11ef-bcea-c7b262605968... 2024-09-11T08:09:36.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:36 smithi181 ceph-mon[32701]: pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-11T08:09:36.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:36 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:09:36.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:36 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:09:36.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:36 smithi181 ceph-mon[32701]: 3.1e scrub starts 2024-09-11T08:09:36.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:36 smithi181 ceph-mon[32701]: 3.1e scrub ok 2024-09-11T08:09:36.435 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:36 smithi150 ceph-mon[29276]: pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-11T08:09:36.435 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:36 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:09:36.435 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:36 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:09:36.436 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:36 smithi150 ceph-mon[29276]: 3.1e scrub starts 2024-09-11T08:09:36.436 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:36 smithi150 ceph-mon[29276]: 3.1e scrub ok 2024-09-11T08:09:36.436 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:36 smithi150 ceph-mon[39612]: pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-11T08:09:36.436 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:36 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:09:36.436 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:36 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:09:36.436 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:36 smithi150 ceph-mon[39612]: 3.1e scrub starts 2024-09-11T08:09:36.436 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:36 smithi150 ceph-mon[39612]: 3.1e scrub ok 2024-09-11T08:09:36.463 INFO:teuthology.orchestra.run.smithi150.stdout:mon.a: {} 2024-09-11T08:09:36.463 INFO:teuthology.orchestra.run.smithi150.stdout:mon.b: {} 2024-09-11T08:09:36.463 INFO:teuthology.orchestra.run.smithi150.stdout:mon.c: {} 2024-09-11T08:09:36.464 INFO:teuthology.orchestra.run.smithi150.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-11T08:09:36.464 INFO:teuthology.orchestra.run.smithi150.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-11T08:09:36.464 INFO:teuthology.orchestra.run.smithi150.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-11T08:09:36.814 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:09:36 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[61611]: ts=2024-09-11T08:09:36.521Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-11T08:09:36.814 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:09:36 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[61611]: ts=2024-09-11T08:09:36.521Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-11T08:09:36.814 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:09:36 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[61611]: ts=2024-09-11T08:09:36.521Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-11T08:09:36.814 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:09:36 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[61611]: ts=2024-09-11T08:09:36.521Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-11T08:09:36.814 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:09:36 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[61611]: ts=2024-09-11T08:09:36.521Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-11T08:09:36.814 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:09:36 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[61611]: ts=2024-09-11T08:09:36.521Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-11T08:09:36.814 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:09:36 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[61611]: ts=2024-09-11T08:09:36.521Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-11T08:09:36.815 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:09:36 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[61611]: ts=2024-09-11T08:09:36.521Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-11T08:09:36.815 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:09:36 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[61611]: ts=2024-09-11T08:09:36.521Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-11T08:09:36.815 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:09:36 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[61611]: ts=2024-09-11T08:09:36.521Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-11T08:09:36.815 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:09:36 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[61611]: ts=2024-09-11T08:09:36.521Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-11T08:09:36.815 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:09:36 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[61611]: ts=2024-09-11T08:09:36.521Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-11T08:09:36.815 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:09:36 smithi181 podman[64704]: 2024-09-11 08:09:36.531536247 +0000 UTC m=+0.141593439 container died d7433e687125daf607eb61db3622623931a796d2032a82b54ae522b4c1670e7e (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-11T08:09:37.073 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:09:36 smithi181 podman[64704]: 2024-09-11 08:09:36.937589844 +0000 UTC m=+0.547647029 container cleanup d7433e687125daf607eb61db3622623931a796d2032a82b54ae522b4c1670e7e (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-11T08:09:37.073 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:09:36 smithi181 bash[64704]: ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a 2024-09-11T08:09:37.177 INFO:teuthology.run_tasks:Running task full_sequential... 2024-09-11T08:09:37.206 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-11T08:09:37.211 INFO:tasks.radosbench:Beginning radosbench... 2024-09-11T08:09:37.211 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-09-11T08:09:37.212 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd pool create unique_pool_0 16 2024-09-11T08:09:37.212 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-11T08:09:37.212 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-11T08:09:37.212 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T08:09:37.213 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-09-11T08:09:37.213 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-11T08:09:37.213 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-11T08:09:37.213 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-09-11T08:09:37.213 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-11T08:09:37.214 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:09:37.214 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-11T08:09:37.214 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T08:09:37.215 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-11T08:09:37.215 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-11T08:09:37.215 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-11T08:09:37.216 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T08:09:37.245 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:37.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:37.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:09:37.347 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:09:37.380 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:37.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:37.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:09:37.482 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:09:37.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:37 smithi150 ceph-mon[29276]: osdmap e88: 8 total, 8 up, 8 in 2024-09-11T08:09:37.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:37 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3191832344' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-11T08:09:37.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:37 smithi150 ceph-mon[29276]: pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-11T08:09:37.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:37 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:09:37.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:37 smithi150 ceph-mon[39612]: osdmap e88: 8 total, 8 up, 8 in 2024-09-11T08:09:37.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:37 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3191832344' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-11T08:09:37.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:37 smithi150 ceph-mon[39612]: pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-11T08:09:37.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:37 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:09:37.505 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:37.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:37.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:09:37.524 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:37 smithi181 ceph-mon[32701]: osdmap e88: 8 total, 8 up, 8 in 2024-09-11T08:09:37.524 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:37 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3191832344' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-11T08:09:37.525 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:37 smithi181 ceph-mon[32701]: pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-11T08:09:37.525 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:37 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:09:37.525 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:09:37 smithi181 podman[64717]: 2024-09-11 08:09:37.180586973 +0000 UTC m=+0.650936748 container remove d7433e687125daf607eb61db3622623931a796d2032a82b54ae522b4c1670e7e (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-11T08:09:37.525 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:09:37 smithi181 systemd[1]: ceph-c1bddff4-7013-11ef-bcea-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-09-11T08:09:37.525 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:09:37 smithi181 systemd[1]: Stopped Ceph prometheus.a for c1bddff4-7013-11ef-bcea-c7b262605968. 2024-09-11T08:09:37.525 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:09:37 smithi181 systemd[1]: ceph-c1bddff4-7013-11ef-bcea-c7b262605968@prometheus.a.service: Consumed 1.045s CPU time. 2024-09-11T08:09:37.525 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:09:37 smithi181 systemd[1]: Starting Ceph prometheus.a for c1bddff4-7013-11ef-bcea-c7b262605968... 2024-09-11T08:09:37.552 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:09:37.552 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:09:37.553 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:09:37.606 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:09:37.607 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:09:37.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:37.631 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:37.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:09:37.732 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:09:37.754 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:37.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:37.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:09:37.786 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:09:37 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:09:37.786 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:09:37 smithi181 podman[64809]: 2024-09-11 08:09:37.428168694 +0000 UTC m=+0.017067011 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-09-11T08:09:37.786 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:09:37 smithi181 podman[64809]: 2024-09-11 08:09:37.54838885 +0000 UTC m=+0.137287154 container create 42045a4ed6d004ff9ac97c8f5f13818b06698ca3a11119dc4a9a1247e57b734b (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-11T08:09:37.786 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:09:37 smithi181 podman[64809]: 2024-09-11 08:09:37.758963232 +0000 UTC m=+0.347861536 container init 42045a4ed6d004ff9ac97c8f5f13818b06698ca3a11119dc4a9a1247e57b734b (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-11T08:09:37.786 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:09:37 smithi181 podman[64809]: 2024-09-11 08:09:37.761736985 +0000 UTC m=+0.350635292 container start 42045a4ed6d004ff9ac97c8f5f13818b06698ca3a11119dc4a9a1247e57b734b (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-11T08:09:37.856 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:09:37.877 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:37.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:37.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:09:37.947 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:09:37 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: [11/Sep/2024:08:09:37] ENGINE Bus STOPPING 2024-09-11T08:09:37.979 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:09:38.000 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:38.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:38.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:09:38.074 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:09:37 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[64841]: ts=2024-09-11T08:09:37.785Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-11T08:09:38.074 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:09:37 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[64841]: ts=2024-09-11T08:09:37.785Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-11T08:09:38.075 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:09:37 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[64841]: ts=2024-09-11T08:09:37.785Z caller=main.go:570 level=info host_details="(Linux 5.14.0-505.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024 x86_64 smithi181 (none))" 2024-09-11T08:09:38.075 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:09:37 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[64841]: ts=2024-09-11T08:09:37.785Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-11T08:09:38.075 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:09:37 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[64841]: ts=2024-09-11T08:09:37.785Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-11T08:09:38.075 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:09:37 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[64841]: ts=2024-09-11T08:09:37.787Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-11T08:09:38.075 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:09:37 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[64841]: ts=2024-09-11T08:09:37.787Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-11T08:09:38.075 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:09:37 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[64841]: ts=2024-09-11T08:09:37.788Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-11T08:09:38.076 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:09:37 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[64841]: ts=2024-09-11T08:09:37.788Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-11T08:09:38.076 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:09:37 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[64841]: ts=2024-09-11T08:09:37.790Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-11T08:09:38.076 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:09:37 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[64841]: ts=2024-09-11T08:09:37.790Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.66µs 2024-09-11T08:09:38.076 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:09:37 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[64841]: ts=2024-09-11T08:09:37.790Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-11T08:09:38.076 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:09:37 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[64841]: ts=2024-09-11T08:09:37.791Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-09-11T08:09:38.076 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:09:37 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[64841]: ts=2024-09-11T08:09:37.791Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-09-11T08:09:38.077 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:09:37 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[64841]: ts=2024-09-11T08:09:37.791Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=18.49µs wal_replay_duration=496.907µs wbl_replay_duration=108ns total_replay_duration=532.551µs 2024-09-11T08:09:38.077 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:09:37 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[64841]: ts=2024-09-11T08:09:37.792Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-11T08:09:38.077 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:09:37 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[64841]: ts=2024-09-11T08:09:37.792Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-11T08:09:38.077 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:09:37 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[64841]: ts=2024-09-11T08:09:37.792Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-11T08:09:38.077 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:09:37 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[64841]: ts=2024-09-11T08:09:37.806Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=14.547783ms db_storage=671ns remote_storage=1.118µs web_handler=354ns query_engine=575ns scrape=382.533µs scrape_sd=25.348µs notify=18.782µs notify_sd=6.632µs rules=13.887504ms tracing=5.889µs 2024-09-11T08:09:38.078 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:09:37 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[64841]: ts=2024-09-11T08:09:37.806Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-11T08:09:38.078 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:09:37 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[64841]: ts=2024-09-11T08:09:37.806Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-11T08:09:38.078 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:09:37 smithi181 bash[64809]: 42045a4ed6d004ff9ac97c8f5f13818b06698ca3a11119dc4a9a1247e57b734b 2024-09-11T08:09:38.078 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:09:37 smithi181 systemd[1]: Started Ceph prometheus.a for c1bddff4-7013-11ef-bcea-c7b262605968. 2024-09-11T08:09:38.102 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:09:38.123 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:38.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:38.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:09:38.226 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:09:38.238 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:09:38 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: [11/Sep/2024:08:09:38] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-11T08:09:38.238 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:09:38 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: [11/Sep/2024:08:09:38] ENGINE Bus STOPPED 2024-09-11T08:09:38.239 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:09:38 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: [11/Sep/2024:08:09:38] ENGINE Bus STARTING 2024-09-11T08:09:38.248 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:38.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:38.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:09:38.351 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:09:38.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:38.374 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:38.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:09:38.475 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:09:38.498 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:38.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:38.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:09:38.575 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:09:38.584 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:09:38 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: [11/Sep/2024:08:09:38] ENGINE Serving on http://:::9283 2024-09-11T08:09:38.584 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:09:38 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: [11/Sep/2024:08:09:38] ENGINE Bus STARTED 2024-09-11T08:09:38.584 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:09:38 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: [11/Sep/2024:08:09:38] ENGINE Bus STOPPING 2024-09-11T08:09:38.600 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:09:38.621 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:38.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:38.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:09:38.723 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:09:38.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:38.746 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:38.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:09:38.847 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:09:38.862 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:09:38 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: [11/Sep/2024:08:09:38] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-11T08:09:38.862 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:09:38 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: [11/Sep/2024:08:09:38] ENGINE Bus STOPPED 2024-09-11T08:09:38.862 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:09:38 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: [11/Sep/2024:08:09:38] ENGINE Bus STARTING 2024-09-11T08:09:38.871 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:38.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:38.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:09:38.974 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:09:38.999 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:38.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:39.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:09:39.101 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:09:39.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:39.124 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:39.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:09:39.172 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:09:38 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: [11/Sep/2024:08:09:38] ENGINE Serving on http://:::9283 2024-09-11T08:09:39.172 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:09:38 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: [11/Sep/2024:08:09:38] ENGINE Bus STARTED 2024-09-11T08:09:39.172 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:09:38 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: [11/Sep/2024:08:09:38] ENGINE Bus STOPPING 2024-09-11T08:09:39.173 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:38 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:09:39.173 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:38 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:39.173 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:38 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:39.173 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:38 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-11T08:09:39.173 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:38 smithi150 ceph-mon[29276]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-11T08:09:39.173 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:38 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi150.front.sepia.ceph.com:9093"}]: dispatch 2024-09-11T08:09:39.173 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:38 smithi150 ceph-mon[29276]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi150.front.sepia.ceph.com:9093"}]: dispatch 2024-09-11T08:09:39.173 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:38 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:39.173 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:38 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-11T08:09:39.173 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:38 smithi150 ceph-mon[29276]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-11T08:09:39.174 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:38 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi181.front.sepia.ceph.com:9095"}]: dispatch 2024-09-11T08:09:39.174 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:38 smithi150 ceph-mon[29276]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi181.front.sepia.ceph.com:9095"}]: dispatch 2024-09-11T08:09:39.174 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:38 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:39.174 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:38 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-11T08:09:39.174 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:38 smithi150 ceph-mon[29276]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-11T08:09:39.174 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:38 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi181.front.sepia.ceph.com:3000"}]: dispatch 2024-09-11T08:09:39.174 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:38 smithi150 ceph-mon[29276]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi181.front.sepia.ceph.com:3000"}]: dispatch 2024-09-11T08:09:39.174 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:38 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:39.174 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:38 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:09:39.175 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:38 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:09:39.175 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:38 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:39.175 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:38 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:39.175 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:38 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-11T08:09:39.175 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:38 smithi150 ceph-mon[39612]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-11T08:09:39.175 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:38 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi150.front.sepia.ceph.com:9093"}]: dispatch 2024-09-11T08:09:39.175 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:38 smithi150 ceph-mon[39612]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi150.front.sepia.ceph.com:9093"}]: dispatch 2024-09-11T08:09:39.175 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:38 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:39.175 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:38 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-11T08:09:39.175 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:38 smithi150 ceph-mon[39612]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-11T08:09:39.175 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:38 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi181.front.sepia.ceph.com:9095"}]: dispatch 2024-09-11T08:09:39.175 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:38 smithi150 ceph-mon[39612]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi181.front.sepia.ceph.com:9095"}]: dispatch 2024-09-11T08:09:39.175 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:38 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:39.176 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:38 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-11T08:09:39.176 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:38 smithi150 ceph-mon[39612]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-11T08:09:39.176 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:38 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi181.front.sepia.ceph.com:3000"}]: dispatch 2024-09-11T08:09:39.176 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:38 smithi150 ceph-mon[39612]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi181.front.sepia.ceph.com:3000"}]: dispatch 2024-09-11T08:09:39.176 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:38 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:39.176 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:38 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:09:39.225 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:09:39.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:39.248 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:39.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:09:39.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:38 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:09:39.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:38 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:39.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:38 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:39.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:38 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-11T08:09:39.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:38 smithi181 ceph-mon[32701]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-11T08:09:39.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:38 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi150.front.sepia.ceph.com:9093"}]: dispatch 2024-09-11T08:09:39.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:38 smithi181 ceph-mon[32701]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi150.front.sepia.ceph.com:9093"}]: dispatch 2024-09-11T08:09:39.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:38 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:39.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:38 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-11T08:09:39.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:38 smithi181 ceph-mon[32701]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-11T08:09:39.326 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:38 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi181.front.sepia.ceph.com:9095"}]: dispatch 2024-09-11T08:09:39.326 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:38 smithi181 ceph-mon[32701]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi181.front.sepia.ceph.com:9095"}]: dispatch 2024-09-11T08:09:39.326 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:38 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:39.326 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:38 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-11T08:09:39.326 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:38 smithi181 ceph-mon[32701]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-11T08:09:39.326 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:38 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi181.front.sepia.ceph.com:3000"}]: dispatch 2024-09-11T08:09:39.326 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:38 smithi181 ceph-mon[32701]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi181.front.sepia.ceph.com:3000"}]: dispatch 2024-09-11T08:09:39.326 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:38 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:39.326 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:38 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:09:39.349 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:09:39.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:39.370 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:39.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:09:39.472 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:09:39.488 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:09:39 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: [11/Sep/2024:08:09:39] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-11T08:09:39.489 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:09:39 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: [11/Sep/2024:08:09:39] ENGINE Bus STOPPED 2024-09-11T08:09:39.489 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:09:39 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: [11/Sep/2024:08:09:39] ENGINE Bus STARTING 2024-09-11T08:09:39.494 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:39.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:39.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:09:39.596 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:09:39.617 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:39.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:39.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:09:39.719 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:09:39.741 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:39.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:39.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:09:39.768 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:09:39 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: [11/Sep/2024:08:09:39] ENGINE Serving on http://:::9283 2024-09-11T08:09:39.769 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:09:39 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: [11/Sep/2024:08:09:39] ENGINE Bus STARTED 2024-09-11T08:09:39.843 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:09:39.866 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:39.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:39.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:09:39.968 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:09:39.990 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:39.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:39.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:09:40.089 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:39 smithi150 ceph-mon[29276]: pgmap v64: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:09:40.089 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:39 smithi150 ceph-mon[29276]: 3.2 scrub starts 2024-09-11T08:09:40.089 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:39 smithi150 ceph-mon[29276]: 3.2 scrub ok 2024-09-11T08:09:40.090 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:39 smithi150 ceph-mon[39612]: pgmap v64: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:09:40.090 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:39 smithi150 ceph-mon[39612]: 3.2 scrub starts 2024-09-11T08:09:40.090 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:39 smithi150 ceph-mon[39612]: 3.2 scrub ok 2024-09-11T08:09:40.092 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:09:40.113 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:40.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:40.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:09:40.215 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:09:40.236 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:40.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:40.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:09:40.302 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:39 smithi181 ceph-mon[32701]: pgmap v64: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:09:40.303 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:39 smithi181 ceph-mon[32701]: 3.2 scrub starts 2024-09-11T08:09:40.303 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:39 smithi181 ceph-mon[32701]: 3.2 scrub ok 2024-09-11T08:09:40.339 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:09:40.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:40.361 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:40.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:09:40.462 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:09:40.485 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:40.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:40.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:09:40.586 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:09:40.609 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:40.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:40.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:09:40.710 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:09:40.732 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:40.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:40.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:09:40.834 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:09:40.855 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:40.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:40.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:09:40.957 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:09:40.979 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:40.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:40.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:09:41.081 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:09:41.102 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:41.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:41.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:09:41.204 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:09:41.225 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:41.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:41.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:09:41.327 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:09:41.348 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:41.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:41.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:09:41.450 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:09:41.472 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:41.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:41.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:09:41.574 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:09:41.588 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:41 smithi150 ceph-mon[29276]: 4.e deep-scrub starts 2024-09-11T08:09:41.588 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:41 smithi150 ceph-mon[29276]: 4.e deep-scrub ok 2024-09-11T08:09:41.589 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:41 smithi150 ceph-mon[29276]: 4.10 scrub starts 2024-09-11T08:09:41.589 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:41 smithi150 ceph-mon[29276]: 4.10 scrub ok 2024-09-11T08:09:41.589 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:41 smithi150 ceph-mon[29276]: pgmap v65: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-11T08:09:41.589 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:41 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:09:41.589 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:41 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:09:41.589 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:41 smithi150 ceph-mon[39612]: 4.e deep-scrub starts 2024-09-11T08:09:41.589 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:41 smithi150 ceph-mon[39612]: 4.e deep-scrub ok 2024-09-11T08:09:41.589 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:41 smithi150 ceph-mon[39612]: 4.10 scrub starts 2024-09-11T08:09:41.589 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:41 smithi150 ceph-mon[39612]: 4.10 scrub ok 2024-09-11T08:09:41.589 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:41 smithi150 ceph-mon[39612]: pgmap v65: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-11T08:09:41.589 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:41 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:09:41.589 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:41 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:09:41.598 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:41.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:41.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:09:41.699 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:09:41.709 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:41 smithi181 ceph-mon[32701]: 4.e deep-scrub starts 2024-09-11T08:09:41.709 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:41 smithi181 ceph-mon[32701]: 4.e deep-scrub ok 2024-09-11T08:09:41.709 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:41 smithi181 ceph-mon[32701]: 4.10 scrub starts 2024-09-11T08:09:41.709 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:41 smithi181 ceph-mon[32701]: 4.10 scrub ok 2024-09-11T08:09:41.709 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:41 smithi181 ceph-mon[32701]: pgmap v65: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-11T08:09:41.709 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:41 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:09:41.709 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:41 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:09:41.722 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:41.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:41.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:09:41.824 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:09:41.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:41.846 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:41.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:09:41.947 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:09:41.969 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:41.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:41.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:09:42.071 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:09:42.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:42.093 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:42.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:09:42.194 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:09:42.216 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:42.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:42.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:09:42.238 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:09:41 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-alertmanager-a[75052]: ts=2024-09-11T08:09:41.858Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002001398s 2024-09-11T08:09:42.318 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:09:42.340 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:42.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:42.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:09:42.441 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:09:42.464 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:42.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:42.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:09:42.565 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:09:42.588 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:42.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:42.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:09:42.693 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:09:42.714 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:42.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:42.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:09:42.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:42 smithi150 ceph-mon[29276]: osdmap e89: 8 total, 8 up, 8 in 2024-09-11T08:09:42.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:42 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:42.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:42 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:42.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:42 smithi150 ceph-mon[39612]: osdmap e89: 8 total, 8 up, 8 in 2024-09-11T08:09:42.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:42 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:42.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:42 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:42.815 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:09:42.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:42 smithi181 ceph-mon[32701]: osdmap e89: 8 total, 8 up, 8 in 2024-09-11T08:09:42.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:42 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:42.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:42 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:42.837 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:42.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:42.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:09:42.939 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:09:42.961 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:42.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:42.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:09:43.062 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:09:43.087 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:43.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:43.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:09:43.189 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:09:43.212 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:43.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:43.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:09:43.314 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:09:43.338 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:43.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:43.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:09:43.395 INFO:teuthology.orchestra.run.smithi150.stderr:noscrub is set 2024-09-11T08:09:43.439 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:09:43.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:43.464 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:43.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:09:43.565 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:09:43.586 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:43.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:43.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:09:43.688 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:09:43.713 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:43.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:43.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:09:43.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:43 smithi150 ceph-mon[29276]: pgmap v67: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:09:43.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:43 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/4267341131' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:09:43.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:43 smithi150 ceph-mon[39612]: pgmap v67: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:09:43.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:43 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/4267341131' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:09:43.815 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:09:43.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:43 smithi181 ceph-mon[32701]: pgmap v67: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:09:43.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:43 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/4267341131' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:09:43.837 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:43.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:43.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:09:43.939 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:09:43.960 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:43.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:43.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:09:44.062 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:09:44.088 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:44.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:44.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:09:44.190 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:09:44.216 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:44.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:44.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:09:44.318 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:09:44.339 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:44.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:44.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:09:44.399 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'unique_pool_0' created 2024-09-11T08:09:44.401 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:44 smithi150 ceph-mon[29276]: 3.11 scrub starts 2024-09-11T08:09:44.401 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:44 smithi150 ceph-mon[29276]: 3.11 scrub ok 2024-09-11T08:09:44.401 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:44 smithi150 ceph-mon[29276]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:09:44.401 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:44 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/4267341131' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:09:44.401 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:44 smithi150 ceph-mon[29276]: osdmap e90: 8 total, 8 up, 8 in 2024-09-11T08:09:44.401 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:44 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/672022819' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-09-11T08:09:44.402 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:44 smithi150 ceph-mon[39612]: 3.11 scrub starts 2024-09-11T08:09:44.402 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:44 smithi150 ceph-mon[39612]: 3.11 scrub ok 2024-09-11T08:09:44.402 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:44 smithi150 ceph-mon[39612]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:09:44.402 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:44 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/4267341131' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:09:44.402 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:44 smithi150 ceph-mon[39612]: osdmap e90: 8 total, 8 up, 8 in 2024-09-11T08:09:44.402 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:44 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/672022819' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-09-11T08:09:44.441 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:09:44.466 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:44.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:44.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:09:44.568 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:09:44.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:44.594 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:44.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:09:44.695 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:09:44.721 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:44.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:44.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:09:44.822 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:09:44.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:44 smithi181 ceph-mon[32701]: 3.11 scrub starts 2024-09-11T08:09:44.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:44 smithi181 ceph-mon[32701]: 3.11 scrub ok 2024-09-11T08:09:44.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:44 smithi181 ceph-mon[32701]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:09:44.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:44 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/4267341131' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:09:44.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:44 smithi181 ceph-mon[32701]: osdmap e90: 8 total, 8 up, 8 in 2024-09-11T08:09:44.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:44 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/672022819' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-09-11T08:09:44.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:44.849 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:44.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:09:44.950 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:09:44.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:44.973 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:44.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:09:45.074 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:09:45.096 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:45.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:45.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:09:45.197 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:09:45.219 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:45.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:45.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:09:45.321 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:09:45.347 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:45.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:45.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:09:45.398 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:09:45 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:09:45] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:09:45.449 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:09:45.471 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:45.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:45.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:09:45.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:45 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/672022819' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-11T08:09:45.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:45 smithi150 ceph-mon[29276]: osdmap e91: 8 total, 8 up, 8 in 2024-09-11T08:09:45.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:45 smithi150 ceph-mon[29276]: pgmap v70: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:09:45.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:45 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/672022819' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-11T08:09:45.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:45 smithi150 ceph-mon[39612]: osdmap e91: 8 total, 8 up, 8 in 2024-09-11T08:09:45.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:45 smithi150 ceph-mon[39612]: pgmap v70: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:09:45.573 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:09:45.595 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:45.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:45.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:09:45.697 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:09:45.719 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:45.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:45.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:09:45.734 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-11T08:09:45.822 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:09:45.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:45 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/672022819' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-11T08:09:45.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:45 smithi181 ceph-mon[32701]: osdmap e91: 8 total, 8 up, 8 in 2024-09-11T08:09:45.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:45 smithi181 ceph-mon[32701]: pgmap v70: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:09:45.843 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:45.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:45.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:09:45.944 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:09:45.966 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:45.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:45.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:09:46.044 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:09:46.044 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:09:46.044 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:09:46.044 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:09:46.044 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:09:46.044 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:09:46.045 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:09:46.045 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:09:46.045 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:09:46.045 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:09:46.045 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:09:46.045 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:09:46.045 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:09:46.046 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:09:46.046 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:09:46.046 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:09:46.068 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:09:46.090 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:46.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:46.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:09:46.191 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:09:46.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:46.214 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:46.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:09:46.315 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:09:46.336 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:46.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:46.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:09:46.438 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:09:46.460 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:46.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:46.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:09:46.562 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:09:46.588 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:46.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:46.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:09:46.689 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:09:46.701 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:46 smithi150 ceph-mon[29276]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-11T08:09:46.701 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:46 smithi150 ceph-mon[29276]: osdmap e92: 8 total, 8 up, 8 in 2024-09-11T08:09:46.701 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:46 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2358030948' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:09:46.701 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:46 smithi150 ceph-mon[39612]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-11T08:09:46.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:46 smithi150 ceph-mon[39612]: osdmap e92: 8 total, 8 up, 8 in 2024-09-11T08:09:46.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:46 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2358030948' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:09:46.711 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:46.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:46.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:09:46.807 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-09-11T08:09:46.807 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:09:46.813 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:09:46.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:46 smithi181 ceph-mon[32701]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-11T08:09:46.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:46 smithi181 ceph-mon[32701]: osdmap e92: 8 total, 8 up, 8 in 2024-09-11T08:09:46.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:46 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2358030948' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:09:46.839 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:46.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:46.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:09:46.941 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:09:46.967 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:46.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:46.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:09:47.069 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:09:47.094 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:47.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:47.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:09:47.196 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:09:47.222 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:47.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:47.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:09:47.324 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:09:47.350 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:47.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:47.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:09:47.452 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:09:47.477 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:47.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:47.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:09:47.494 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-11T08:09:47.578 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:09:47.600 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:47.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:47.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:09:47.702 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:09:47.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:47.730 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:47.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:09:47.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:47 smithi150 ceph-mon[29276]: osdmap e93: 8 total, 8 up, 8 in 2024-09-11T08:09:47.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:47 smithi150 ceph-mon[29276]: pgmap v73: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:09:47.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:47 smithi150 ceph-mon[39612]: osdmap e93: 8 total, 8 up, 8 in 2024-09-11T08:09:47.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:47 smithi150 ceph-mon[39612]: pgmap v73: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:09:47.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:47 smithi181 ceph-mon[32701]: osdmap e93: 8 total, 8 up, 8 in 2024-09-11T08:09:47.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:47 smithi181 ceph-mon[32701]: pgmap v73: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:09:47.831 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:09:47.856 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:47.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:47.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:09:47.958 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:09:47.985 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:47.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:47.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:09:48.073 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:09:47 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:09:48.087 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:09:48.111 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:48.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:48.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:09:48.213 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:09:48.236 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:48.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:48.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:09:48.338 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:09:48.362 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:48.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:48.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:09:48.463 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:09:48.485 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:48.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:48.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:09:48.587 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:09:48.592 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:09:48 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:09:48] "GET /metrics HTTP/1.1" 200 220034 "" "Prometheus/2.43.0" 2024-09-11T08:09:48.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:48 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:09:48.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:48 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:09:48.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:48 smithi150 ceph-mon[29276]: osdmap e94: 8 total, 8 up, 8 in 2024-09-11T08:09:48.593 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:48 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:09:48.593 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:48 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:09:48.593 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:48 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:09:48.593 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:48 smithi150 ceph-mon[39612]: osdmap e94: 8 total, 8 up, 8 in 2024-09-11T08:09:48.593 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:48 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:09:48.609 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:48.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:48.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:09:48.710 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:09:48.733 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:48.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:48.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:09:48.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:48 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:09:48.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:48 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:09:48.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:48 smithi181 ceph-mon[32701]: osdmap e94: 8 total, 8 up, 8 in 2024-09-11T08:09:48.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:48 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:09:48.837 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:09:48.862 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:48.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:48.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:09:48.965 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:09:48.969 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-11T08:09:48.969 INFO:teuthology.orchestra.run.smithi150.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-11T08:09:48.969 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-11T08:09:48.989 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:48.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:48.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:09:49.091 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:09:49.114 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:49.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:49.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:09:49.165 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:09:49.216 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:09:49.240 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:49.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:49.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:09:49.342 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:09:49.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:49.367 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:49.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:09:49.467 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:09:49.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:49.494 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:49.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:09:49.595 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:09:49.599 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:49 smithi150 ceph-mon[29276]: pgmap v75: 148 pgs: 148 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:09:49.599 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:49 smithi150 ceph-mon[39612]: pgmap v75: 148 pgs: 148 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:09:49.619 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:09:49.620 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:49.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:49.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:09:49.722 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:09:49.747 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:49.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:49.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:09:49.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:49 smithi181 ceph-mon[32701]: pgmap v75: 148 pgs: 148 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:09:49.849 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:09:49.875 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:49.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:49.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:09:49.934 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T08:09:49.976 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:09:50.000 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:50.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:50.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:09:50.102 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:09:50.124 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:50.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:50.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:09:50.226 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:09:50.252 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:50.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:50.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:09:50.349 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T08:09:50.354 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:09:50.379 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:50.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:50.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:09:50.481 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:09:50.504 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:50.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:50.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:09:50.606 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:09:50.632 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:50.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:50.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:09:50.734 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:09:50.756 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:50.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:50.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:09:50.858 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:09:50.884 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:50.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:50.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:09:50.986 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:09:51.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:51.014 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:51.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:09:51.115 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:09:51.137 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:51.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:51.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:09:51.239 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:09:51.260 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:51.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:51.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:09:51.362 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:09:51.388 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:51.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:51.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:09:51.490 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:09:51.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:51.517 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:51.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:09:51.618 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:09:51.645 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:51.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:51.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:09:51.747 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:09:51.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:51.770 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:51.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:09:51.871 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:09:51.889 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:09:51.890 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:09:51.898 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:51.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:51.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:09:52.000 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:09:52.013 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:51 smithi150 ceph-mon[29276]: pgmap v76: 148 pgs: 148 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:09:52.013 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:51 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:52.013 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:51 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:09:52.014 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:51 smithi150 ceph-mon[39612]: pgmap v76: 148 pgs: 148 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:09:52.014 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:51 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:52.014 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:51 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:09:52.022 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:52.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:52.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:09:52.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:51 smithi181 ceph-mon[32701]: pgmap v76: 148 pgs: 148 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:09:52.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:51 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:09:52.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:51 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:09:52.124 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:09:52.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:52.151 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:52.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:09:52.252 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:09:52.279 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:52.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:52.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:09:52.381 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:09:52.403 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:52.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:52.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:09:52.505 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:09:52.532 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:52.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:52.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:09:52.634 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:09:52.660 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:52.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:52.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:09:52.761 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:09:52.786 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:52.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:52.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:09:52.888 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:09:52.911 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:52.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:52.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:09:53.013 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:09:53.037 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:53.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:53.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:09:53.139 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:09:53.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:53.164 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:53.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:09:53.258 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-09-11T08:09:53.258 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:09:53.265 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:09:53.286 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:53.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:53.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:09:53.388 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:09:53.414 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:53.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:53.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:09:53.516 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:09:53.537 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:53.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:53.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:09:53.639 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:09:53.665 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:53.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:53.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:09:53.771 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:09:53.797 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:53.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:53.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:09:53.899 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:09:53.921 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:53.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:53.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:09:53.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:53 smithi150 ceph-mon[29276]: pgmap v77: 148 pgs: 148 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-11T08:09:53.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:53 smithi150 ceph-mon[39612]: pgmap v77: 148 pgs: 148 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-11T08:09:54.024 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:09:54.047 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:54.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:54.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:09:54.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:53 smithi181 ceph-mon[32701]: pgmap v77: 148 pgs: 148 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-11T08:09:54.150 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:09:54.153 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-11T08:09:54.172 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:54.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:54.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:09:54.274 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:09:54.295 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:54.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:54.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:09:54.397 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:09:54.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:54.421 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:54.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:09:54.522 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:09:54.543 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:54.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:54.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:09:54.634 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-11T08:09:54.634 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T08:09:54.634 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-09-11T08:09:54.634 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-11T08:09:54.635 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-11T08:09:54.635 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-09-11T08:09:54.645 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:09:54.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:54.667 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:54.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:09:54.768 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:09:54.770 INFO:teuthology.orchestra.run.smithi150.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-11T08:09:54.791 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:54.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:54.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:09:54.893 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:09:54.896 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:09:54.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:54.916 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:54.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:09:54.925 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:54 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3325164473' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-11T08:09:54.925 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:54 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-11T08:09:54.925 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:54 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3325164473' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-11T08:09:54.925 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:54 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-11T08:09:55.017 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:09:55.018 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:54 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3325164473' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-11T08:09:55.018 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:54 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-11T08:09:55.039 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:55.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:55.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:09:55.140 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:09:55.163 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:55.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:55.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:09:55.265 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:09:55.273 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:09:55 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:09:55] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:09:55.288 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:55.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:55.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:09:55.389 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:09:55.411 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:55.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:55.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:09:55.513 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:09:55.535 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:55.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:55.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:09:55.637 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:09:55.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:55.659 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:55.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:09:55.760 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:09:55.782 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:55.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:55.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:09:55.884 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:09:55.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:55.906 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:55.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:09:55.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:55 smithi150 ceph-mon[29276]: pgmap v78: 148 pgs: 148 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:09:55.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:55 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-11T08:09:55.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:55 smithi150 ceph-mon[29276]: osdmap e95: 8 total, 8 up, 8 in 2024-09-11T08:09:55.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:55 smithi150 ceph-mon[39612]: pgmap v78: 148 pgs: 148 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:09:55.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:55 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-11T08:09:55.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:55 smithi150 ceph-mon[39612]: osdmap e95: 8 total, 8 up, 8 in 2024-09-11T08:09:56.007 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:09:56.028 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:56.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:56.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:09:56.062 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:55 smithi181 ceph-mon[32701]: pgmap v78: 148 pgs: 148 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:09:56.062 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:55 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-11T08:09:56.063 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:55 smithi181 ceph-mon[32701]: osdmap e95: 8 total, 8 up, 8 in 2024-09-11T08:09:56.130 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:09:56.153 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:56.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:56.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:09:56.255 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:09:56.276 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:56.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:56.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:09:56.378 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:09:56.399 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:56.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:56.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:09:56.432 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-11T08:09:56.433 INFO:teuthology.orchestra.run.smithi181.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-11T08:09:56.433 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-11T08:09:56.501 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:09:56.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:56.524 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:56.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:09:56.625 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:09:56.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:56.649 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:56.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:09:56.750 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:09:56.772 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:56.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:56.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:09:56.874 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:09:56.896 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:56.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:56.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:09:56.951 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:56 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:09:56.951 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:56 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:09:56.951 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:56 smithi150 ceph-mon[29276]: osdmap e96: 8 total, 8 up, 8 in 2024-09-11T08:09:56.951 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:56 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1024954469' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:09:56.951 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:56 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:09:56.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:56 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:09:56.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:56 smithi150 ceph-mon[39612]: osdmap e96: 8 total, 8 up, 8 in 2024-09-11T08:09:56.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:56 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1024954469' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:09:56.998 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:09:57.021 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:57.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:57.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:09:57.065 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:56 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:09:57.065 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:56 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:09:57.065 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:56 smithi181 ceph-mon[32701]: osdmap e96: 8 total, 8 up, 8 in 2024-09-11T08:09:57.065 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:56 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1024954469' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:09:57.123 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:09:57.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:57.147 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:57.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:09:57.248 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:09:57.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:57.271 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:57.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:09:57.372 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:09:57.380 INFO:teuthology.orchestra.run.smithi150.stderr:nodeep-scrub is set 2024-09-11T08:09:57.395 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:57.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:57.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:09:57.497 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:09:57.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:57.519 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:57.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:09:57.620 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:09:57.641 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:57.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:57.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:09:57.743 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:09:57.764 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:57.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:57.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:09:57.866 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:09:57.893 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:57.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:57.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:09:57.965 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:57 smithi150 ceph-mon[39612]: pgmap v81: 148 pgs: 148 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-11T08:09:57.966 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:57 smithi150 ceph-mon[39612]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-11T08:09:57.966 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:57 smithi150 ceph-mon[39612]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:09:57.966 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:57 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1024954469' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:09:57.966 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:57 smithi150 ceph-mon[39612]: osdmap e97: 8 total, 8 up, 8 in 2024-09-11T08:09:57.966 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:57 smithi150 ceph-mon[29276]: pgmap v81: 148 pgs: 148 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-11T08:09:57.966 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:57 smithi150 ceph-mon[29276]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-11T08:09:57.966 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:57 smithi150 ceph-mon[29276]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:09:57.966 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:57 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1024954469' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:09:57.966 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:57 smithi150 ceph-mon[29276]: osdmap e97: 8 total, 8 up, 8 in 2024-09-11T08:09:57.981 DEBUG:teuthology.orchestra.run.smithi150:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-09-11T08:09:57.994 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:09:58.012 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-11T08:09:58.012 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-11T08:09:58.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:58.034 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:58.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:09:58.072 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:hints = 1 2024-09-11T08:09:58.072 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-11T08:09:58.072 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Object prefix: benchmark_data_smithi150_78508 2024-09-11T08:09:58.072 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-11T08:09:58.072 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 0 0 0 0 0 0 - 0 2024-09-11T08:09:58.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:57 smithi181 ceph-mon[32701]: pgmap v81: 148 pgs: 148 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-11T08:09:58.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:57 smithi181 ceph-mon[32701]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-11T08:09:58.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:57 smithi181 ceph-mon[32701]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:09:58.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:57 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1024954469' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:09:58.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:57 smithi181 ceph-mon[32701]: osdmap e97: 8 total, 8 up, 8 in 2024-09-11T08:09:58.074 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:09:57 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:09:58.135 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:09:58.170 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:58.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:58.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:09:58.272 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:09:58.313 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:58.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:58.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:09:58.414 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:09:58.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:58.455 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:58.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:09:58.556 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:09:58.589 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:58.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:58.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:09:58.691 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:09:58.731 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:58.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:58.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:09:58.739 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:09:58 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:09:58] "GET /metrics HTTP/1.1" 200 222536 "" "Prometheus/2.43.0" 2024-09-11T08:09:58.833 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:09:58.872 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:58.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:58.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:09:58.974 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:09:59.015 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:59.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:59.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:09:59.039 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:58 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:09:59.040 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:58 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:09:59.069 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 1 16 3557 3541 221.25 221.312 0.00589881 0.00450979 2024-09-11T08:09:59.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:58 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:09:59.117 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:09:59.159 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:59.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:59.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:09:59.223 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:09:59.223 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:09:59.258 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:09:59.259 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:09:59.259 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:09:59.261 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:09:59.287 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:59.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:59.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:09:59.292 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:09:59.292 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:09:59.329 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:09:59.329 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:09:59.362 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:09:59.362 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:09:59.389 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:09:59.396 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:09:59.396 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:09:59.418 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:59.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:59.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:09:59.429 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:09:59.429 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:09:59.464 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:09:59.465 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:09:59.520 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:09:59.550 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:59.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:59.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:09:59.652 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:09:59.679 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:59.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:59.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:09:59.781 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:09:59.811 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:09:59.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:59.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:09:59.913 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:09:59.952 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:09:59.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:09:59.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:09:59.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:59 smithi150 ceph-mon[29276]: pgmap v83: 148 pgs: 148 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:09:59.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:09:59 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3541577163' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:09:59.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:59 smithi150 ceph-mon[39612]: pgmap v83: 148 pgs: 148 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:09:59.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:09:59 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3541577163' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:10:00.054 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:10:00.069 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 2 16 6771 6755 211.055 200.875 0.00766786 0.00472556 2024-09-11T08:10:00.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:59 smithi181 ceph-mon[32701]: pgmap v83: 148 pgs: 148 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:10:00.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:09:59 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3541577163' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:10:00.084 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:00.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:00.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:10:00.186 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:10:00.223 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:00.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:00.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:10:00.325 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:00.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:00.355 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:00.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:00.456 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:10:00.487 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:00.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:00.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:10:00.589 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:10:00.626 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:00.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:00.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:10:00.728 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:10:00.755 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:00.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:00.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:10:00.856 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:10:00.895 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:00.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:00.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:10:00.938 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T08:10:00.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:00 smithi150 ceph-mon[29276]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-09-11T08:10:00.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:00 smithi150 ceph-mon[29276]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-09-11T08:10:00.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:00 smithi150 ceph-mon[39612]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-09-11T08:10:00.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:00 smithi150 ceph-mon[39612]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-09-11T08:10:00.996 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:10:01.026 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:01.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:01.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:10:01.069 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 3 15 9808 9793 203.99 189.875 0.00359553 0.00489245 2024-09-11T08:10:01.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:00 smithi181 ceph-mon[32701]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-09-11T08:10:01.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:00 smithi181 ceph-mon[32701]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-09-11T08:10:01.128 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:10:01.156 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:01.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:01.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:10:01.258 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:01.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:01.314 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:01.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:01.415 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:10:01.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:01.459 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:01.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:10:01.560 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:01.588 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:01.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:01.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:01.690 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:10:01.722 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:01.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:01.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:10:01.824 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:01.852 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:01.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:01.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:01.954 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:01.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:01 smithi150 ceph-mon[29276]: pgmap v84: 148 pgs: 148 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:10:01.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:01 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:10:01.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:01 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:10:01.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:01 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:10:01.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:01 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:10:01.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:01 smithi150 ceph-mon[39612]: pgmap v84: 148 pgs: 148 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:10:01.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:01 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:10:01.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:01 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:10:01.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:01 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:10:01.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:01 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:10:02.001 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:02.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:02.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:02.069 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 4 15 12501 12486 195.059 168.312 0.00636505 0.00511779 2024-09-11T08:10:02.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:01 smithi181 ceph-mon[32701]: pgmap v84: 148 pgs: 148 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:10:02.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:01 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:10:02.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:01 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:10:02.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:01 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:10:02.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:01 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:10:02.104 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:02.110 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-11T08:10:02.110 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd reweight 0 1 2024-09-11T08:10:02.132 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:02.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:02.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:02.234 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:02.261 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:02.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:02.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:02.363 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:10:02.394 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:02.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:02.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:10:02.496 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:10:02.526 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:02.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:02.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:10:02.618 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:10:02.628 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:10:02.658 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:02.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:02.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:10:02.759 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:02.782 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T08:10:02.790 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:02.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:02.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:02.878 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:10:02.892 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:10:02.917 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:02.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:02.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:10:03.016 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:02 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:10:03.017 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:02 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:10:03.017 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:02 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:10:03.017 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:02 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:10:03.017 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:02 smithi181 ceph-mon[32701]: osdmap e98: 8 total, 8 up, 8 in 2024-09-11T08:10:03.019 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:10:03.021 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:02 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:10:03.021 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:02 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:10:03.021 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:02 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:10:03.021 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:02 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:10:03.021 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:02 smithi150 ceph-mon[29276]: osdmap e98: 8 total, 8 up, 8 in 2024-09-11T08:10:03.021 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:02 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:10:03.021 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:02 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:10:03.022 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:02 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:10:03.022 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:02 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:10:03.022 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:02 smithi150 ceph-mon[39612]: osdmap e98: 8 total, 8 up, 8 in 2024-09-11T08:10:03.069 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:03.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:03.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:10:03.070 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 5 16 14767 14751 184.346 141.562 0.00364738 0.00541444 2024-09-11T08:10:03.171 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:03.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:03.240 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:03.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:03.341 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:03.370 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:03.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:03.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:03.472 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:10:03.501 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:03.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:03.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:10:03.604 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:03.635 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:03.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:03.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:03.724 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-11T08:10:03.724 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 20, 2024-09-11T08:10:03.724 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 600, 2024-09-11T08:10:03.724 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [ 2024-09-11T08:10:03.725 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.725 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:5355 7.f 7:f769a44c:::benchmark_data_smithi150_78508_object5354:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-11T08:10:03.725 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:09:59.641254+0000", 2024-09-11T08:10:03.725 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 4.0305880600000004, 2024-09-11T08:10:03.725 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.019137723999999998, 2024-09-11T08:10:03.725 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:10:03.725 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:10:03.725 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:10:03.725 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:10:03.725 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:10:03.725 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 5355 2024-09-11T08:10:03.725 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.725 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:10:03.726 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.726 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:10:03.726 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.641254+0000", 2024-09-11T08:10:03.726 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:03.726 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.726 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.726 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:10:03.726 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.641254+0000", 2024-09-11T08:10:03.726 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:03.726 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.726 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.726 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:10:03.726 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.641253+0000", 2024-09-11T08:10:03.727 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:10:03.727 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.727 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.727 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:10:03.727 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.641274+0000", 2024-09-11T08:10:03.727 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.1467e-05 2024-09-11T08:10:03.727 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.727 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.727 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:10:03.727 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.641286+0000", 2024-09-11T08:10:03.727 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1386000000000001e-05 2024-09-11T08:10:03.727 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.727 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.727 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:10:03.728 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.641291+0000", 2024-09-11T08:10:03.728 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.0069999999999997e-06 2024-09-11T08:10:03.728 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.728 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.728 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:10:03.728 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.641331+0000", 2024-09-11T08:10:03.728 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.0602999999999997e-05 2024-09-11T08:10:03.728 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.728 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.729 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:10:03.729 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.641430+0000", 2024-09-11T08:10:03.729 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.8653000000000006e-05 2024-09-11T08:10:03.729 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.729 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.729 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 3,7", 2024-09-11T08:10:03.729 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.641571+0000", 2024-09-11T08:10:03.729 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00014058899999999999 2024-09-11T08:10:03.729 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.729 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.729 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:03.729 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.642632+0000", 2024-09-11T08:10:03.729 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00106131 2024-09-11T08:10:03.730 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.730 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.730 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:10:03.730 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.642925+0000", 2024-09-11T08:10:03.730 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00029330399999999998 2024-09-11T08:10:03.730 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.730 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.730 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:03.730 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.660317+0000", 2024-09-11T08:10:03.730 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.017391924 2024-09-11T08:10:03.730 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.730 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.730 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:10:03.731 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.660357+0000", 2024-09-11T08:10:03.731 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.0059999999999999e-05 2024-09-11T08:10:03.731 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.731 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.731 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:10:03.731 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.660392+0000", 2024-09-11T08:10:03.731 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.4558999999999999e-05 2024-09-11T08:10:03.731 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:03.731 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:10:03.731 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:03.731 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.731 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.731 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:5358 7.9 7:948f8a27:::benchmark_data_smithi150_78508_object5357:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-11T08:10:03.732 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:09:59.641734+0000", 2024-09-11T08:10:03.732 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 4.0301080450000004, 2024-09-11T08:10:03.732 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.018536614, 2024-09-11T08:10:03.732 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:10:03.732 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:10:03.732 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:10:03.732 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:10:03.732 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:10:03.732 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 5358 2024-09-11T08:10:03.732 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.732 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:10:03.732 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.732 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:10:03.733 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.641734+0000", 2024-09-11T08:10:03.733 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:03.733 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.733 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.733 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:10:03.733 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.641734+0000", 2024-09-11T08:10:03.733 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:03.733 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.733 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.733 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:10:03.733 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.641733+0000", 2024-09-11T08:10:03.733 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:10:03.733 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.734 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.734 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:10:03.734 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.641809+0000", 2024-09-11T08:10:03.734 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.5295000000000003e-05 2024-09-11T08:10:03.734 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.734 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.734 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:10:03.734 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.641817+0000", 2024-09-11T08:10:03.734 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.5830000000000007e-06 2024-09-11T08:10:03.734 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.734 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.734 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:10:03.734 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.641835+0000", 2024-09-11T08:10:03.734 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.7492000000000001e-05 2024-09-11T08:10:03.735 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.735 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.735 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:10:03.735 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.641905+0000", 2024-09-11T08:10:03.735 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.0727000000000002e-05 2024-09-11T08:10:03.735 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.735 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.735 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:10:03.735 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.642023+0000", 2024-09-11T08:10:03.735 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000117608 2024-09-11T08:10:03.735 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.735 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.735 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 1,4", 2024-09-11T08:10:03.735 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.642231+0000", 2024-09-11T08:10:03.735 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00020822599999999999 2024-09-11T08:10:03.736 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.736 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.736 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:10:03.736 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.659455+0000", 2024-09-11T08:10:03.736 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.017223955999999999 2024-09-11T08:10:03.736 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.736 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.736 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:03.736 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.659720+0000", 2024-09-11T08:10:03.736 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00026520500000000002 2024-09-11T08:10:03.736 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.736 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.736 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:03.736 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.660126+0000", 2024-09-11T08:10:03.737 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000405762 2024-09-11T08:10:03.737 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.737 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.737 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:10:03.737 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.660232+0000", 2024-09-11T08:10:03.737 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000105779 2024-09-11T08:10:03.737 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.737 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.737 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:10:03.737 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.660271+0000", 2024-09-11T08:10:03.737 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.8751000000000002e-05 2024-09-11T08:10:03.737 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:03.737 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:10:03.737 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:03.738 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.738 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.738 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:5359 7.9 7:9db368ad:::benchmark_data_smithi150_78508_object5358:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-11T08:10:03.738 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:09:59.642105+0000", 2024-09-11T08:10:03.738 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 4.0297367609999997, 2024-09-11T08:10:03.738 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.018281348999999999, 2024-09-11T08:10:03.738 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:10:03.738 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:10:03.738 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:10:03.738 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:10:03.738 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:10:03.738 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 5359 2024-09-11T08:10:03.738 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.739 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:10:03.739 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.739 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:10:03.739 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.642105+0000", 2024-09-11T08:10:03.739 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:03.739 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.739 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.739 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:10:03.739 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.642105+0000", 2024-09-11T08:10:03.739 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:03.739 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.739 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.739 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:10:03.739 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.642105+0000", 2024-09-11T08:10:03.740 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:10:03.740 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.740 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.740 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:10:03.740 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.642147+0000", 2024-09-11T08:10:03.740 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1860000000000002e-05 2024-09-11T08:10:03.740 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.740 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.740 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:10:03.740 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.642155+0000", 2024-09-11T08:10:03.740 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.8310000000000007e-06 2024-09-11T08:10:03.740 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.740 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.741 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:10:03.741 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.642163+0000", 2024-09-11T08:10:03.741 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.0790000000000001e-06 2024-09-11T08:10:03.741 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.741 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.741 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:10:03.741 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.642860+0000", 2024-09-11T08:10:03.741 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00069648400000000004 2024-09-11T08:10:03.741 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.741 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.741 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:10:03.741 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.642972+0000", 2024-09-11T08:10:03.741 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000112304 2024-09-11T08:10:03.741 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.742 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.742 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 1,4", 2024-09-11T08:10:03.742 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.643173+0000", 2024-09-11T08:10:03.742 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00020116900000000001 2024-09-11T08:10:03.742 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.742 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.742 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:03.742 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.659747+0000", 2024-09-11T08:10:03.742 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.016574078999999999 2024-09-11T08:10:03.742 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.742 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.742 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:10:03.742 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.660097+0000", 2024-09-11T08:10:03.742 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00034913499999999998 2024-09-11T08:10:03.743 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.743 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.743 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:03.743 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.660314+0000", 2024-09-11T08:10:03.743 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00021707900000000001 2024-09-11T08:10:03.743 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.743 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.743 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:10:03.743 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.660356+0000", 2024-09-11T08:10:03.743 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.2685999999999998e-05 2024-09-11T08:10:03.743 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.743 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.743 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:10:03.743 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.660387+0000", 2024-09-11T08:10:03.744 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.0334000000000001e-05 2024-09-11T08:10:03.744 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:03.744 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:10:03.744 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:03.744 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.744 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.744 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.14790.0:5362 7.5 e97/91 7:abd95bf1:::benchmark_data_smithi150_78508_object5361:head v 97'328, mlcod=97'325)", 2024-09-11T08:10:03.744 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:09:59.643066+0000", 2024-09-11T08:10:03.744 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 4.0287760879999999, 2024-09-11T08:10:03.744 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.017009203000000001, 2024-09-11T08:10:03.744 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:10:03.744 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:10:03.744 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:10:03.744 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.745 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:10:03.745 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.643066+0000", 2024-09-11T08:10:03.745 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:03.745 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.745 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.745 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:10:03.745 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.643066+0000", 2024-09-11T08:10:03.745 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:03.745 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.745 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.745 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:10:03.745 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.643065+0000", 2024-09-11T08:10:03.745 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:10:03.746 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.746 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.746 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:10:03.746 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.643165+0000", 2024-09-11T08:10:03.746 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.9197000000000006e-05 2024-09-11T08:10:03.746 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.746 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.746 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:10:03.746 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.643175+0000", 2024-09-11T08:10:03.746 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.8700000000000004e-06 2024-09-11T08:10:03.746 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.746 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.746 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:10:03.746 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.643179+0000", 2024-09-11T08:10:03.746 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.9479999999999999e-06 2024-09-11T08:10:03.747 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.747 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.747 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:10:03.747 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.658967+0000", 2024-09-11T08:10:03.747 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.015787026999999999 2024-09-11T08:10:03.747 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.747 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.747 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:10:03.747 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.659022+0000", 2024-09-11T08:10:03.747 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.5455999999999999e-05 2024-09-11T08:10:03.747 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.747 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.747 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:10:03.747 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.660047+0000", 2024-09-11T08:10:03.748 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00102461 2024-09-11T08:10:03.748 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.748 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.748 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:10:03.748 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.660075+0000", 2024-09-11T08:10:03.748 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.8657e-05 2024-09-11T08:10:03.748 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:03.748 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:10:03.748 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:03.748 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.748 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.748 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop_reply(client.14790.0:5358 7.9 e97/91 ondisk, result = 0)", 2024-09-11T08:10:03.748 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:09:59.643272+0000", 2024-09-11T08:10:03.749 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 4.0285698219999997, 2024-09-11T08:10:03.749 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.017007020000000001, 2024-09-11T08:10:03.749 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:10:03.749 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "started", 2024-09-11T08:10:03.749 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:10:03.749 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.749 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:10:03.749 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.643272+0000", 2024-09-11T08:10:03.749 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:03.749 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.749 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.749 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:10:03.749 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.643272+0000", 2024-09-11T08:10:03.750 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:03.750 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.750 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.750 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:10:03.750 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.643272+0000", 2024-09-11T08:10:03.750 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:10:03.750 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.750 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.750 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:10:03.750 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.643276+0000", 2024-09-11T08:10:03.750 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1640000000000003e-06 2024-09-11T08:10:03.750 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.750 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.750 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:10:03.751 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.660081+0000", 2024-09-11T08:10:03.751 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.016804611000000001 2024-09-11T08:10:03.751 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.751 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.751 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:10:03.751 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.660090+0000", 2024-09-11T08:10:03.751 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.1689999999999998e-06 2024-09-11T08:10:03.751 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.751 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.751 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:10:03.751 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.660115+0000", 2024-09-11T08:10:03.751 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.5211e-05 2024-09-11T08:10:03.751 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.751 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.752 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:10:03.752 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.660121+0000", 2024-09-11T08:10:03.752 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.1319999999999999e-06 2024-09-11T08:10:03.752 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.752 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.752 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:10:03.752 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.660279+0000", 2024-09-11T08:10:03.752 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000158286 2024-09-11T08:10:03.752 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:03.752 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:10:03.752 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:03.752 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.752 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.753 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop_reply(client.14790.0:5358 7.9 e97/91 ondisk, result = 0)", 2024-09-11T08:10:03.753 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:09:59.643276+0000", 2024-09-11T08:10:03.753 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 4.0285661199999998, 2024-09-11T08:10:03.753 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.016446378000000001, 2024-09-11T08:10:03.753 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:10:03.753 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "started", 2024-09-11T08:10:03.753 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:10:03.753 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.753 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:10:03.753 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.643276+0000", 2024-09-11T08:10:03.753 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:03.753 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.753 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.753 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:10:03.754 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.643276+0000", 2024-09-11T08:10:03.754 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:03.754 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.754 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.754 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:10:03.754 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.643275+0000", 2024-09-11T08:10:03.754 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:10:03.754 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.754 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.754 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:10:03.754 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.643280+0000", 2024-09-11T08:10:03.754 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.262e-06 2024-09-11T08:10:03.754 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.754 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.755 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:10:03.755 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.658891+0000", 2024-09-11T08:10:03.755 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.015611913999999999 2024-09-11T08:10:03.755 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.755 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.755 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:10:03.755 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.659494+0000", 2024-09-11T08:10:03.755 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00060225299999999997 2024-09-11T08:10:03.755 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.755 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.755 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:10:03.755 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.659707+0000", 2024-09-11T08:10:03.755 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00021374400000000001 2024-09-11T08:10:03.755 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.756 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.756 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:10:03.756 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.659714+0000", 2024-09-11T08:10:03.756 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.7989999999999996e-06 2024-09-11T08:10:03.756 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.756 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.756 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:10:03.756 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.659722+0000", 2024-09-11T08:10:03.756 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.1019999999999997e-06 2024-09-11T08:10:03.756 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:03.756 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:10:03.756 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:03.756 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.756 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.757 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.14790.0:5364 7.d e97/91 7:baca6501:::benchmark_data_smithi150_78508_object5363:head v 97'362, mlcod=97'360)", 2024-09-11T08:10:03.757 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:09:59.643353+0000", 2024-09-11T08:10:03.757 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 4.0284891570000001, 2024-09-11T08:10:03.757 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.018940286000000001, 2024-09-11T08:10:03.757 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:10:03.757 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:10:03.757 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:10:03.757 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.757 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:10:03.757 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.643353+0000", 2024-09-11T08:10:03.757 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:03.757 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.757 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.757 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:10:03.758 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.643353+0000", 2024-09-11T08:10:03.758 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:03.758 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.758 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.758 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:10:03.758 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.643352+0000", 2024-09-11T08:10:03.758 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:10:03.758 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.758 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.758 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:10:03.759 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.643436+0000", 2024-09-11T08:10:03.759 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.3770000000000006e-05 2024-09-11T08:10:03.759 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.759 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.759 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:10:03.759 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.660718+0000", 2024-09-11T08:10:03.759 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.017282071999999999 2024-09-11T08:10:03.759 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.759 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.759 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:10:03.759 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.660726+0000", 2024-09-11T08:10:03.759 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.1759999999999993e-06 2024-09-11T08:10:03.759 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.760 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.760 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:10:03.760 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.660767+0000", 2024-09-11T08:10:03.760 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.0948999999999997e-05 2024-09-11T08:10:03.760 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.760 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.760 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:10:03.760 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.660791+0000", 2024-09-11T08:10:03.760 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.3394999999999999e-05 2024-09-11T08:10:03.760 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.760 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.760 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:10:03.760 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.662247+0000", 2024-09-11T08:10:03.761 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0014559760000000001 2024-09-11T08:10:03.761 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.761 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.761 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:10:03.761 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:09:59.662293+0000", 2024-09-11T08:10:03.761 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.6647000000000002e-05 2024-09-11T08:10:03.761 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:03.761 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:10:03.761 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:03.761 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.761 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.761 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:7674 7.6 7:6b2556a6:::benchmark_data_smithi150_78508_object7673:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-11T08:10:03.761 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:10:00.386103+0000", 2024-09-11T08:10:03.762 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 3.2857387230000001, 2024-09-11T08:10:03.762 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.018115935, 2024-09-11T08:10:03.762 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:10:03.762 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:10:03.762 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:10:03.762 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:10:03.762 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:10:03.762 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 7674 2024-09-11T08:10:03.762 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.762 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:10:03.762 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.762 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:10:03.762 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.386103+0000", 2024-09-11T08:10:03.763 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:03.763 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.763 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.763 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:10:03.763 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.386103+0000", 2024-09-11T08:10:03.763 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:03.763 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.763 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.763 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:10:03.763 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.386102+0000", 2024-09-11T08:10:03.763 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:10:03.763 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.763 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.763 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:10:03.764 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.386125+0000", 2024-09-11T08:10:03.764 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.2379e-05 2024-09-11T08:10:03.764 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.764 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.764 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:10:03.764 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.386140+0000", 2024-09-11T08:10:03.764 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.507e-05 2024-09-11T08:10:03.764 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.764 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.764 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:10:03.764 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.386147+0000", 2024-09-11T08:10:03.764 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.8090000000000004e-06 2024-09-11T08:10:03.764 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.765 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.765 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:10:03.765 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.387654+0000", 2024-09-11T08:10:03.765 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0015070260000000001 2024-09-11T08:10:03.765 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.765 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.765 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:10:03.765 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.387776+0000", 2024-09-11T08:10:03.765 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012223299999999999 2024-09-11T08:10:03.765 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.765 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.765 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 3,7", 2024-09-11T08:10:03.765 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.387931+0000", 2024-09-11T08:10:03.766 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00015489099999999999 2024-09-11T08:10:03.766 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.766 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.766 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:10:03.766 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.388587+0000", 2024-09-11T08:10:03.766 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00065664800000000002 2024-09-11T08:10:03.766 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.766 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.766 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:03.766 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.393659+0000", 2024-09-11T08:10:03.766 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0050713449999999997 2024-09-11T08:10:03.766 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.766 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.766 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:03.767 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.404140+0000", 2024-09-11T08:10:03.767 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.010481796999999999 2024-09-11T08:10:03.767 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.767 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.767 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:10:03.767 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.404185+0000", 2024-09-11T08:10:03.767 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.4301999999999997e-05 2024-09-11T08:10:03.767 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.767 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.767 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:10:03.767 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.404219+0000", 2024-09-11T08:10:03.767 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.4581e-05 2024-09-11T08:10:03.767 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:03.767 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:10:03.768 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:03.768 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.768 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.768 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.14790.0:7684 7.7 e97/91 7:e9a4899b:::benchmark_data_smithi150_78508_object7683:head v 97'486, mlcod=97'484)", 2024-09-11T08:10:03.768 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:10:00.391402+0000", 2024-09-11T08:10:03.768 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 3.2804400879999998, 2024-09-11T08:10:03.768 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.013614405, 2024-09-11T08:10:03.768 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:10:03.768 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:10:03.768 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:10:03.768 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.768 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:10:03.768 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.391402+0000", 2024-09-11T08:10:03.769 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:03.769 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.769 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.769 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:10:03.769 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.391402+0000", 2024-09-11T08:10:03.769 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:03.769 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.769 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.769 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:10:03.769 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.391401+0000", 2024-09-11T08:10:03.769 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:10:03.769 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.769 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.769 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:10:03.770 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.391436+0000", 2024-09-11T08:10:03.770 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.5024000000000002e-05 2024-09-11T08:10:03.770 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.770 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.770 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:10:03.770 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.391457+0000", 2024-09-11T08:10:03.770 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.126e-05 2024-09-11T08:10:03.770 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.770 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.770 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:10:03.770 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.391463+0000", 2024-09-11T08:10:03.770 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.3839999999999996e-06 2024-09-11T08:10:03.770 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.770 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.770 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:10:03.771 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.391504+0000", 2024-09-11T08:10:03.771 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1831e-05 2024-09-11T08:10:03.771 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.771 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.771 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:10:03.771 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.391523+0000", 2024-09-11T08:10:03.771 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.8488e-05 2024-09-11T08:10:03.771 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.771 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.771 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:10:03.771 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.404983+0000", 2024-09-11T08:10:03.771 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.013459861 2024-09-11T08:10:03.771 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.771 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.772 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:10:03.772 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.405016+0000", 2024-09-11T08:10:03.772 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.3643000000000003e-05 2024-09-11T08:10:03.772 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:03.772 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:10:03.772 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:03.772 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.772 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.772 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:10157 7.9 7:9d79f210:::benchmark_data_smithi150_78508_object10156:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-11T08:10:03.772 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:10:01.184846+0000", 2024-09-11T08:10:03.772 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 2.4869960870000001, 2024-09-11T08:10:03.772 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.017284391999999999, 2024-09-11T08:10:03.772 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:10:03.773 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:10:03.773 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:10:03.773 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:10:03.773 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:10:03.773 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 10157 2024-09-11T08:10:03.773 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.773 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:10:03.773 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.773 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:10:03.773 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:01.184846+0000", 2024-09-11T08:10:03.773 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:03.773 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.773 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.773 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:10:03.774 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:01.184846+0000", 2024-09-11T08:10:03.774 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:03.774 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.774 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.774 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:10:03.774 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:01.184845+0000", 2024-09-11T08:10:03.774 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:10:03.774 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.774 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.774 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:10:03.774 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:01.184866+0000", 2024-09-11T08:10:03.774 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.1350999999999999e-05 2024-09-11T08:10:03.774 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.775 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.775 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:10:03.775 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:01.184878+0000", 2024-09-11T08:10:03.775 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1518999999999999e-05 2024-09-11T08:10:03.775 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.775 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.775 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:10:03.775 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:01.184882+0000", 2024-09-11T08:10:03.775 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1899999999999997e-06 2024-09-11T08:10:03.775 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.775 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.775 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:10:03.775 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:01.185265+0000", 2024-09-11T08:10:03.775 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00038340500000000002 2024-09-11T08:10:03.776 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.776 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.776 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:10:03.776 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:01.185598+0000", 2024-09-11T08:10:03.776 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00033252500000000002 2024-09-11T08:10:03.776 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.776 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.776 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 1,4", 2024-09-11T08:10:03.776 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:01.185728+0000", 2024-09-11T08:10:03.776 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000129894 2024-09-11T08:10:03.776 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.776 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.776 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:03.776 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:01.187376+0000", 2024-09-11T08:10:03.777 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0016484819999999999 2024-09-11T08:10:03.777 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.777 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.777 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:03.777 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:01.193491+0000", 2024-09-11T08:10:03.777 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0061146710000000003 2024-09-11T08:10:03.777 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.777 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.777 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:10:03.777 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:01.202034+0000", 2024-09-11T08:10:03.777 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0085436439999999995 2024-09-11T08:10:03.777 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.777 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.778 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:10:03.778 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:01.202097+0000", 2024-09-11T08:10:03.778 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.2947999999999994e-05 2024-09-11T08:10:03.778 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.778 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.778 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:10:03.778 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:01.202130+0000", 2024-09-11T08:10:03.778 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.3120999999999997e-05 2024-09-11T08:10:03.778 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:03.778 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:10:03.778 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:03.778 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.778 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.778 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:10158 7.9 7:9da7afe4:::benchmark_data_smithi150_78508_object10157:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-11T08:10:03.779 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:10:01.185346+0000", 2024-09-11T08:10:03.779 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 2.4864965240000001, 2024-09-11T08:10:03.779 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.016852771999999999, 2024-09-11T08:10:03.779 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:10:03.779 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:10:03.779 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:10:03.779 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:10:03.779 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:10:03.779 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 10158 2024-09-11T08:10:03.779 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.779 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:10:03.779 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.779 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:10:03.779 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:01.185346+0000", 2024-09-11T08:10:03.780 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:03.780 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.780 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.780 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:10:03.780 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:01.185346+0000", 2024-09-11T08:10:03.780 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:03.780 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.780 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.780 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:10:03.780 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:01.185345+0000", 2024-09-11T08:10:03.780 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:10:03.780 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.780 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.780 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:10:03.781 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:01.185364+0000", 2024-09-11T08:10:03.781 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.9202e-05 2024-09-11T08:10:03.781 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.781 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.781 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:10:03.781 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:01.185373+0000", 2024-09-11T08:10:03.781 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.935e-06 2024-09-11T08:10:03.781 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.781 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.781 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:10:03.781 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:01.185377+0000", 2024-09-11T08:10:03.781 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1479999999999999e-06 2024-09-11T08:10:03.782 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.782 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.782 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:10:03.782 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:01.187400+0000", 2024-09-11T08:10:03.782 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0020224969999999998 2024-09-11T08:10:03.782 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.782 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.782 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:10:03.782 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:01.187512+0000", 2024-09-11T08:10:03.782 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000112176 2024-09-11T08:10:03.782 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.782 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.782 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 1,4", 2024-09-11T08:10:03.782 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:01.187644+0000", 2024-09-11T08:10:03.783 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000132717 2024-09-11T08:10:03.783 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.783 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.783 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:03.783 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:01.195287+0000", 2024-09-11T08:10:03.783 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0076425169999999997 2024-09-11T08:10:03.783 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.783 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.783 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:03.783 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:01.196956+0000", 2024-09-11T08:10:03.783 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0016693050000000001 2024-09-11T08:10:03.783 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.783 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.784 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:10:03.784 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:01.202140+0000", 2024-09-11T08:10:03.784 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0051838939999999997 2024-09-11T08:10:03.784 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.784 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.784 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:10:03.784 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:01.202170+0000", 2024-09-11T08:10:03.784 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.0213e-05 2024-09-11T08:10:03.784 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.784 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.784 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:10:03.784 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:01.202198+0000", 2024-09-11T08:10:03.784 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.8036999999999999e-05 2024-09-11T08:10:03.784 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:03.785 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:10:03.785 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:03.785 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.785 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.785 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:13278 7.f 7:faba7cec:::benchmark_data_smithi150_78508_object13277:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-11T08:10:03.785 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:10:02.385430+0000", 2024-09-11T08:10:03.785 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 1.2864125040000001, 2024-09-11T08:10:03.785 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.018983546, 2024-09-11T08:10:03.785 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:10:03.785 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:10:03.785 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:10:03.785 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:10:03.785 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:10:03.786 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 13278 2024-09-11T08:10:03.786 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.786 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:10:03.786 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.786 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:10:03.786 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.385430+0000", 2024-09-11T08:10:03.786 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:03.786 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.786 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.786 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:10:03.786 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.385430+0000", 2024-09-11T08:10:03.786 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:03.786 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.786 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.787 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:10:03.787 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.385428+0000", 2024-09-11T08:10:03.787 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:10:03.787 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.787 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.787 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:10:03.787 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.385450+0000", 2024-09-11T08:10:03.787 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.1628999999999999e-05 2024-09-11T08:10:03.787 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.787 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.787 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:10:03.787 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.385460+0000", 2024-09-11T08:10:03.787 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.9820000000000003e-06 2024-09-11T08:10:03.788 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.788 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.788 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:10:03.788 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.385466+0000", 2024-09-11T08:10:03.788 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.767e-06 2024-09-11T08:10:03.788 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.788 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.788 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:10:03.788 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.386065+0000", 2024-09-11T08:10:03.788 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00059903799999999998 2024-09-11T08:10:03.788 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.788 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.788 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:10:03.788 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.386169+0000", 2024-09-11T08:10:03.789 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000104078 2024-09-11T08:10:03.789 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.789 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.789 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 3,7", 2024-09-11T08:10:03.789 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.386284+0000", 2024-09-11T08:10:03.789 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000115388 2024-09-11T08:10:03.789 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.789 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.789 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:03.789 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.390887+0000", 2024-09-11T08:10:03.789 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.004602475 2024-09-11T08:10:03.789 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.789 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.789 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:10:03.790 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.390900+0000", 2024-09-11T08:10:03.790 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3399e-05 2024-09-11T08:10:03.790 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.790 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.790 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:03.790 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.404254+0000", 2024-09-11T08:10:03.790 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.013353359 2024-09-11T08:10:03.790 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.790 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.790 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:10:03.790 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.404302+0000", 2024-09-11T08:10:03.790 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.8812e-05 2024-09-11T08:10:03.790 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.791 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.791 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:10:03.791 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.404413+0000", 2024-09-11T08:10:03.791 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00011076799999999999 2024-09-11T08:10:03.791 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:03.791 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:10:03.791 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:03.791 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.791 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.791 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:13282 7.9 7:988ec4d5:::benchmark_data_smithi150_78508_object13281:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-11T08:10:03.791 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:10:02.387448+0000", 2024-09-11T08:10:03.791 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 1.2843940039999999, 2024-09-11T08:10:03.791 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.018798539999999999, 2024-09-11T08:10:03.791 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:10:03.791 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:10:03.792 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:10:03.792 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:10:03.792 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:10:03.792 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 13282 2024-09-11T08:10:03.792 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.792 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:10:03.792 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.792 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:10:03.792 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.387448+0000", 2024-09-11T08:10:03.792 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:03.792 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.792 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.792 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:10:03.793 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.387448+0000", 2024-09-11T08:10:03.793 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:03.793 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.793 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.793 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:10:03.793 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.387447+0000", 2024-09-11T08:10:03.793 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:10:03.793 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.793 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.793 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:10:03.793 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.387474+0000", 2024-09-11T08:10:03.793 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.7016e-05 2024-09-11T08:10:03.793 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.793 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.794 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:10:03.794 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.387485+0000", 2024-09-11T08:10:03.794 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1695000000000001e-05 2024-09-11T08:10:03.794 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.794 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.794 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:10:03.794 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.387492+0000", 2024-09-11T08:10:03.794 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.4840000000000001e-06 2024-09-11T08:10:03.794 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.794 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.794 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:10:03.794 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.388280+0000", 2024-09-11T08:10:03.794 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00078770000000000001 2024-09-11T08:10:03.795 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.795 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.795 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:10:03.795 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.390161+0000", 2024-09-11T08:10:03.795 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.001881478 2024-09-11T08:10:03.795 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.795 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.795 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 1,4", 2024-09-11T08:10:03.795 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.390336+0000", 2024-09-11T08:10:03.795 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00017520900000000001 2024-09-11T08:10:03.795 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.795 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.795 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:03.795 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.393742+0000", 2024-09-11T08:10:03.796 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0034058080000000002 2024-09-11T08:10:03.796 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.796 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.796 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:03.796 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.402765+0000", 2024-09-11T08:10:03.796 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0090233029999999999 2024-09-11T08:10:03.796 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.796 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.796 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:10:03.796 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.406148+0000", 2024-09-11T08:10:03.796 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0033829289999999998 2024-09-11T08:10:03.796 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.796 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.797 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:10:03.797 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.406194+0000", 2024-09-11T08:10:03.797 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.5459e-05 2024-09-11T08:10:03.797 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.797 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.797 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:10:03.797 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.406247+0000", 2024-09-11T08:10:03.797 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.2939999999999998e-05 2024-09-11T08:10:03.797 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:03.797 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:10:03.797 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:03.797 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.797 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.798 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:13285 7.f 7:f0cf2305:::benchmark_data_smithi150_78508_object13284:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-11T08:10:03.798 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:10:02.387680+0000", 2024-09-11T08:10:03.798 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 1.2841624030000001, 2024-09-11T08:10:03.798 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.016840770000000001, 2024-09-11T08:10:03.798 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:10:03.798 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:10:03.798 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:10:03.798 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:10:03.798 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:10:03.798 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 13285 2024-09-11T08:10:03.798 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.798 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:10:03.798 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.799 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:10:03.799 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.387680+0000", 2024-09-11T08:10:03.799 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:03.799 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.799 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.799 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:10:03.799 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.387680+0000", 2024-09-11T08:10:03.799 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:03.799 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.799 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.799 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:10:03.799 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.387679+0000", 2024-09-11T08:10:03.799 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:10:03.799 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.799 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.800 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:10:03.800 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.387701+0000", 2024-09-11T08:10:03.800 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.1753999999999999e-05 2024-09-11T08:10:03.800 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.800 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.800 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:10:03.800 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.387711+0000", 2024-09-11T08:10:03.800 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.9250000000000007e-06 2024-09-11T08:10:03.800 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.800 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.800 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:10:03.800 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.387716+0000", 2024-09-11T08:10:03.800 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.6350000000000003e-06 2024-09-11T08:10:03.801 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.801 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.801 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:10:03.801 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.387801+0000", 2024-09-11T08:10:03.801 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.4615000000000002e-05 2024-09-11T08:10:03.801 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.801 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.801 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:10:03.801 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.387922+0000", 2024-09-11T08:10:03.801 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012125000000000001 2024-09-11T08:10:03.801 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.801 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.801 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 3,7", 2024-09-11T08:10:03.802 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.388051+0000", 2024-09-11T08:10:03.802 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000128515 2024-09-11T08:10:03.802 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.802 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.802 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:03.802 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.392236+0000", 2024-09-11T08:10:03.802 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.004185154 2024-09-11T08:10:03.802 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.802 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.802 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:10:03.802 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.393448+0000", 2024-09-11T08:10:03.802 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.001212077 2024-09-11T08:10:03.802 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.802 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.803 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:03.803 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.404447+0000", 2024-09-11T08:10:03.803 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.010999460000000001 2024-09-11T08:10:03.803 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.803 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.803 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:10:03.803 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.404490+0000", 2024-09-11T08:10:03.803 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.2920999999999998e-05 2024-09-11T08:10:03.803 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.803 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.803 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:10:03.803 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.404520+0000", 2024-09-11T08:10:03.804 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.0260000000000002e-05 2024-09-11T08:10:03.804 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:03.804 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:10:03.804 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:03.804 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.804 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.804 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:13287 7.f 7:ff9c8f62:::benchmark_data_smithi150_78508_object13286:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-11T08:10:03.804 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:10:02.388638+0000", 2024-09-11T08:10:03.804 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 1.2832044069999999, 2024-09-11T08:10:03.804 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.015975251999999999, 2024-09-11T08:10:03.804 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:10:03.804 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:10:03.804 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:10:03.804 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:10:03.805 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:10:03.805 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 13287 2024-09-11T08:10:03.805 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.805 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:10:03.805 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.805 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:10:03.805 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.388638+0000", 2024-09-11T08:10:03.805 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:03.805 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.805 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.805 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:10:03.805 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.388638+0000", 2024-09-11T08:10:03.805 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:03.805 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.806 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.806 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:10:03.806 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.388628+0000", 2024-09-11T08:10:03.806 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.99999 2024-09-11T08:10:03.806 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.806 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.806 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:10:03.806 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.388679+0000", 2024-09-11T08:10:03.806 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.1501e-05 2024-09-11T08:10:03.806 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.806 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.806 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:10:03.806 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.388751+0000", 2024-09-11T08:10:03.806 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.1928999999999995e-05 2024-09-11T08:10:03.807 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.807 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.807 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:10:03.807 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.388758+0000", 2024-09-11T08:10:03.807 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.4779999999999996e-06 2024-09-11T08:10:03.807 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.807 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.807 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:10:03.807 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.390983+0000", 2024-09-11T08:10:03.807 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00222551 2024-09-11T08:10:03.807 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.807 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.807 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:10:03.807 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.391107+0000", 2024-09-11T08:10:03.808 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012411099999999999 2024-09-11T08:10:03.808 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.808 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.808 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 3,7", 2024-09-11T08:10:03.808 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.391244+0000", 2024-09-11T08:10:03.808 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000136267 2024-09-11T08:10:03.808 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.808 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.808 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:03.808 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.393047+0000", 2024-09-11T08:10:03.808 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.001803607 2024-09-11T08:10:03.808 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.808 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.808 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:10:03.809 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.393453+0000", 2024-09-11T08:10:03.809 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000405432 2024-09-11T08:10:03.809 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.809 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:10:03.810 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.810 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:03.810 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.404550+0000", 2024-09-11T08:10:03.810 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.011097325 2024-09-11T08:10:03.810 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.810 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.811 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:10:03.811 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.404584+0000", 2024-09-11T08:10:03.811 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.4294000000000001e-05 2024-09-11T08:10:03.811 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.811 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.811 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:10:03.811 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.404613+0000", 2024-09-11T08:10:03.811 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.8745999999999999e-05 2024-09-11T08:10:03.811 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:03.811 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:10:03.811 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:03.811 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.811 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.812 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:13294 7.9 7:974a7073:::benchmark_data_smithi150_78508_object13293:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-11T08:10:03.812 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:10:02.397064+0000", 2024-09-11T08:10:03.812 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 1.274778046, 2024-09-11T08:10:03.812 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.01468153, 2024-09-11T08:10:03.812 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:10:03.812 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:10:03.812 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:10:03.812 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:10:03.812 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:10:03.812 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 13294 2024-09-11T08:10:03.812 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.812 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:10:03.812 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.813 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:10:03.813 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.397064+0000", 2024-09-11T08:10:03.813 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:03.813 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.813 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.813 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:10:03.813 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.397064+0000", 2024-09-11T08:10:03.813 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:03.813 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.813 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.813 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:10:03.813 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.397062+0000", 2024-09-11T08:10:03.813 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:10:03.813 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.814 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.814 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:10:03.814 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.397093+0000", 2024-09-11T08:10:03.814 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.0584999999999999e-05 2024-09-11T08:10:03.814 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.814 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.814 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:10:03.814 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.398250+0000", 2024-09-11T08:10:03.814 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.001156963 2024-09-11T08:10:03.814 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.814 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.814 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:10:03.814 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.398257+0000", 2024-09-11T08:10:03.814 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.7710000000000006e-06 2024-09-11T08:10:03.815 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.815 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.815 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:10:03.815 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.403244+0000", 2024-09-11T08:10:03.815 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0049868660000000004 2024-09-11T08:10:03.815 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.815 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.815 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:10:03.815 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.403348+0000", 2024-09-11T08:10:03.815 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000104152 2024-09-11T08:10:03.815 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.815 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.815 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 1,4", 2024-09-11T08:10:03.815 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.403463+0000", 2024-09-11T08:10:03.816 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00011456899999999999 2024-09-11T08:10:03.816 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.816 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.816 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:10:03.816 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.406363+0000", 2024-09-11T08:10:03.816 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0028998209999999999 2024-09-11T08:10:03.816 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.816 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.816 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:03.816 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.406396+0000", 2024-09-11T08:10:03.816 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.2780999999999997e-05 2024-09-11T08:10:03.816 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.816 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.816 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:03.817 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.411663+0000", 2024-09-11T08:10:03.817 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0052674489999999996 2024-09-11T08:10:03.817 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.817 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.817 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:10:03.817 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.411707+0000", 2024-09-11T08:10:03.817 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.4505000000000002e-05 2024-09-11T08:10:03.817 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.817 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.817 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:10:03.817 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.411746+0000", 2024-09-11T08:10:03.817 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.8118000000000003e-05 2024-09-11T08:10:03.817 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:03.817 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:10:03.818 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:03.818 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.818 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.818 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:13586 7.9 7:992b6db5:::benchmark_data_smithi150_78508_object13585:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-11T08:10:03.818 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:10:02.519233+0000", 2024-09-11T08:10:03.818 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 1.152608936, 2024-09-11T08:10:03.818 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.013322931999999999, 2024-09-11T08:10:03.818 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:10:03.818 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:10:03.818 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:10:03.818 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:10:03.818 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:10:03.818 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 13586 2024-09-11T08:10:03.818 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.818 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:10:03.819 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.819 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:10:03.819 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.519233+0000", 2024-09-11T08:10:03.819 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:03.819 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.819 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.819 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:10:03.819 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.519233+0000", 2024-09-11T08:10:03.819 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:03.819 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.819 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.819 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:10:03.819 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.519233+0000", 2024-09-11T08:10:03.819 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:10:03.820 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.820 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.820 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:10:03.820 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.519251+0000", 2024-09-11T08:10:03.820 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.8068e-05 2024-09-11T08:10:03.820 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.820 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.820 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:10:03.820 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.519256+0000", 2024-09-11T08:10:03.820 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.6550000000000001e-06 2024-09-11T08:10:03.820 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.820 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.820 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:10:03.820 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.519260+0000", 2024-09-11T08:10:03.821 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.9419999999999997e-06 2024-09-11T08:10:03.821 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.821 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.821 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:10:03.821 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.524367+0000", 2024-09-11T08:10:03.821 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0051070559999999996 2024-09-11T08:10:03.821 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.821 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.821 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:10:03.821 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.524500+0000", 2024-09-11T08:10:03.821 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00013258899999999999 2024-09-11T08:10:03.821 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.821 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.821 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 1,4", 2024-09-11T08:10:03.822 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.524639+0000", 2024-09-11T08:10:03.822 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00013864699999999999 2024-09-11T08:10:03.822 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.822 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.822 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:10:03.822 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.526259+0000", 2024-09-11T08:10:03.822 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.001619849 2024-09-11T08:10:03.822 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.822 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.822 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:03.822 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.526545+0000", 2024-09-11T08:10:03.822 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00028660600000000001 2024-09-11T08:10:03.822 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.823 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.823 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:03.823 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.532476+0000", 2024-09-11T08:10:03.823 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0059310430000000004 2024-09-11T08:10:03.823 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.823 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.823 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:10:03.823 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.532521+0000", 2024-09-11T08:10:03.823 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.4883999999999999e-05 2024-09-11T08:10:03.823 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.823 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.824 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:10:03.824 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.532556+0000", 2024-09-11T08:10:03.824 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.5012000000000003e-05 2024-09-11T08:10:03.824 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:03.824 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:10:03.824 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:03.824 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.824 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.824 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:13753 7.f 7:f79082be:::benchmark_data_smithi150_78508_object13752:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-11T08:10:03.824 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:10:02.608254+0000", 2024-09-11T08:10:03.824 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 1.063587769, 2024-09-11T08:10:03.824 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.013804733, 2024-09-11T08:10:03.824 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:10:03.825 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:10:03.825 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:10:03.825 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:10:03.825 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:10:03.825 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 13753 2024-09-11T08:10:03.825 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.825 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:10:03.825 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.825 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:10:03.825 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.608254+0000", 2024-09-11T08:10:03.825 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:03.825 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.825 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.825 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:10:03.826 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.608254+0000", 2024-09-11T08:10:03.826 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:03.826 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.826 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.826 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:10:03.826 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.608253+0000", 2024-09-11T08:10:03.826 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:10:03.826 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.826 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.826 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:10:03.826 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.608275+0000", 2024-09-11T08:10:03.826 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.2708e-05 2024-09-11T08:10:03.826 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.826 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.826 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:10:03.827 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.608291+0000", 2024-09-11T08:10:03.827 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.5517999999999999e-05 2024-09-11T08:10:03.827 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.827 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.827 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:10:03.827 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.608297+0000", 2024-09-11T08:10:03.827 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.5860000000000004e-06 2024-09-11T08:10:03.827 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.827 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.827 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:10:03.827 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.608727+0000", 2024-09-11T08:10:03.827 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00043078700000000002 2024-09-11T08:10:03.827 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.827 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.828 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:10:03.828 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.608845+0000", 2024-09-11T08:10:03.828 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000117562 2024-09-11T08:10:03.828 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.828 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.828 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 3,7", 2024-09-11T08:10:03.828 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.608972+0000", 2024-09-11T08:10:03.828 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012712699999999999 2024-09-11T08:10:03.828 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.828 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.828 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:10:03.828 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.610688+0000", 2024-09-11T08:10:03.828 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.001715921 2024-09-11T08:10:03.829 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.829 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.829 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:03.829 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.610757+0000", 2024-09-11T08:10:03.829 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.9392000000000002e-05 2024-09-11T08:10:03.829 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.829 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.829 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:03.829 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.621973+0000", 2024-09-11T08:10:03.829 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.011215707 2024-09-11T08:10:03.829 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.829 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.829 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:10:03.829 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.622025+0000", 2024-09-11T08:10:03.829 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.2244000000000003e-05 2024-09-11T08:10:03.830 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.830 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.830 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:10:03.830 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.622059+0000", 2024-09-11T08:10:03.830 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.3828999999999997e-05 2024-09-11T08:10:03.830 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:03.830 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:10:03.830 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:03.830 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.830 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.830 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:14097 7.6 7:6567fbb5:::benchmark_data_smithi150_78508_object14096:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-11T08:10:03.830 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:10:02.744552+0000", 2024-09-11T08:10:03.830 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.92728998699999998, 2024-09-11T08:10:03.831 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.013408990000000001, 2024-09-11T08:10:03.831 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:10:03.831 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:10:03.831 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:10:03.831 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:10:03.831 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:10:03.831 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 14097 2024-09-11T08:10:03.831 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.831 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:10:03.831 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.831 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:10:03.831 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.744552+0000", 2024-09-11T08:10:03.831 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:03.831 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.832 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.832 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:10:03.832 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.744552+0000", 2024-09-11T08:10:03.832 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:03.832 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.832 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.832 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:10:03.832 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.744551+0000", 2024-09-11T08:10:03.832 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:10:03.832 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.832 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.832 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:10:03.832 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.744573+0000", 2024-09-11T08:10:03.832 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.1679999999999999e-05 2024-09-11T08:10:03.833 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.833 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.833 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:10:03.833 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.744582+0000", 2024-09-11T08:10:03.833 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.2189999999999993e-06 2024-09-11T08:10:03.833 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.833 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.833 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:10:03.833 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.744586+0000", 2024-09-11T08:10:03.833 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1869999999999999e-06 2024-09-11T08:10:03.833 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.833 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.833 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:10:03.833 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.744922+0000", 2024-09-11T08:10:03.834 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00033586400000000001 2024-09-11T08:10:03.834 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.834 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.834 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:10:03.834 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.745033+0000", 2024-09-11T08:10:03.834 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000111056 2024-09-11T08:10:03.834 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.834 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.834 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 3,7", 2024-09-11T08:10:03.834 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.745156+0000", 2024-09-11T08:10:03.834 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000123144 2024-09-11T08:10:03.834 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.834 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.834 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:10:03.835 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.746231+0000", 2024-09-11T08:10:03.835 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0010747490000000001 2024-09-11T08:10:03.835 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.835 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.835 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:03.835 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.746901+0000", 2024-09-11T08:10:03.835 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00067012899999999999 2024-09-11T08:10:03.835 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.835 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.835 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:03.835 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.757887+0000", 2024-09-11T08:10:03.835 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.01098559 2024-09-11T08:10:03.835 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.835 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.836 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:10:03.836 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.757927+0000", 2024-09-11T08:10:03.836 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.0679000000000001e-05 2024-09-11T08:10:03.836 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.836 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.836 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:10:03.836 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.757961+0000", 2024-09-11T08:10:03.836 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.3899000000000002e-05 2024-09-11T08:10:03.836 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:03.836 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:10:03.836 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:03.836 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.836 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.837 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:14538 7.f 7:fb62e4a0:::benchmark_data_smithi150_78508_object14537:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-11T08:10:03.837 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:10:02.949541+0000", 2024-09-11T08:10:03.837 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.72230089399999997, 2024-09-11T08:10:03.837 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.019123996000000001, 2024-09-11T08:10:03.837 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:10:03.837 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:10:03.837 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:10:03.837 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:10:03.837 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:10:03.837 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 14538 2024-09-11T08:10:03.837 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.837 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:10:03.837 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.837 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:10:03.838 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.949541+0000", 2024-09-11T08:10:03.838 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:03.838 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.838 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.838 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:10:03.838 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.949541+0000", 2024-09-11T08:10:03.838 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:03.838 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.838 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.838 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:10:03.838 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.949539+0000", 2024-09-11T08:10:03.838 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:10:03.838 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.838 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.838 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:10:03.839 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.949564+0000", 2024-09-11T08:10:03.839 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.4820999999999998e-05 2024-09-11T08:10:03.839 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.839 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.839 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:10:03.839 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.949582+0000", 2024-09-11T08:10:03.839 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.8153999999999999e-05 2024-09-11T08:10:03.839 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.839 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.839 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:10:03.839 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.949589+0000", 2024-09-11T08:10:03.839 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.1949999999999997e-06 2024-09-11T08:10:03.839 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.839 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.840 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:10:03.840 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.949644+0000", 2024-09-11T08:10:03.840 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.4332000000000001e-05 2024-09-11T08:10:03.840 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.840 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.840 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:10:03.840 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.949794+0000", 2024-09-11T08:10:03.840 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00015047699999999999 2024-09-11T08:10:03.840 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.840 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.840 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 3,7", 2024-09-11T08:10:03.840 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.949940+0000", 2024-09-11T08:10:03.840 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00014628699999999999 2024-09-11T08:10:03.840 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.841 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.841 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:10:03.841 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.953398+0000", 2024-09-11T08:10:03.841 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.003458006 2024-09-11T08:10:03.841 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.841 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.841 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:03.841 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.953482+0000", 2024-09-11T08:10:03.841 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.3559000000000004e-05 2024-09-11T08:10:03.841 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.841 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.841 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:03.841 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.968572+0000", 2024-09-11T08:10:03.841 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.015090053000000001 2024-09-11T08:10:03.842 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.842 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.842 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:10:03.842 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.968630+0000", 2024-09-11T08:10:03.842 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.8202999999999998e-05 2024-09-11T08:10:03.842 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:03.842 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:03.842 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:10:03.842 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:02.968665+0000", 2024-09-11T08:10:03.842 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.4910999999999998e-05 2024-09-11T08:10:03.842 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:03.842 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:10:03.842 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:03.842 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:03.842 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:10:03.843 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-11T08:10:03.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:03.877 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:03.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:10:03.978 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:10:03.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:03 smithi150 ceph-mon[29276]: pgmap v86: 148 pgs: 148 active+clean; 868 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.6 KiB/s rd, 137 MiB/s wr, 2.19k op/s 2024-09-11T08:10:03.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:03 smithi150 ceph-mon[39612]: pgmap v86: 148 pgs: 148 active+clean; 868 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.6 KiB/s rd, 137 MiB/s wr, 2.19k op/s 2024-09-11T08:10:04.018 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:04.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:04.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:10:04.070 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 6 16 17384 17368 180.88 163.562 0.00559432 0.00552298 2024-09-11T08:10:04.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:03 smithi181 ceph-mon[32701]: pgmap v86: 148 pgs: 148 active+clean; 868 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.6 KiB/s rd, 137 MiB/s wr, 2.19k op/s 2024-09-11T08:10:04.121 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:10:04.160 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:04.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:04.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:10:04.263 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:10:04.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:04.300 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:04.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:10:04.401 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:04.430 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:04.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:04.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:04.532 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:10:04.571 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:04.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:04.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:10:04.673 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:10:04.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:04.710 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:04.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:10:04.811 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:04.842 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:04.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:04.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:04.943 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:04.978 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:04.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:04.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:05.070 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 7 16 20037 20021 178.726 165.812 0.00594085 0.00558887 2024-09-11T08:10:05.080 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:10:05.117 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:05.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:05.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:10:05.174 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:10:05 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:10:05] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:10:05.219 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:10:05.248 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:05.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:05.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:10:05.350 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:05.391 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:05.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:05.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:05.493 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:05.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:05.521 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:05.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:05.622 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:10:05.652 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:05.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:05.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:10:05.754 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:10:05.792 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:05.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:05.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:10:05.894 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:10:05.931 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:05.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:05.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:10:06.033 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:10:06.053 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:06.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:06.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:10:06.070 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 8 16 22318 22302 174.204 142.562 0.00141318 0.00570201 2024-09-11T08:10:06.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:05 smithi181 ceph-mon[32701]: pgmap v87: 148 pgs: 148 active+clean; 868 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 108 MiB/s wr, 1.73k op/s 2024-09-11T08:10:06.156 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:10:06.195 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:06.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:06.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:10:06.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:05 smithi150 ceph-mon[29276]: pgmap v87: 148 pgs: 148 active+clean; 868 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 108 MiB/s wr, 1.73k op/s 2024-09-11T08:10:06.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:05 smithi150 ceph-mon[39612]: pgmap v87: 148 pgs: 148 active+clean; 868 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 108 MiB/s wr, 1.73k op/s 2024-09-11T08:10:06.298 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:10:06.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:06.329 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:06.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:10:06.430 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:06.462 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:06.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:06.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:06.564 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:10:06.594 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:06.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:06.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:10:06.697 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:10:06.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:06.728 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:06.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:10:06.829 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:06.860 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:06.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:06.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:06.962 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:10:06.987 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:06.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:06.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:10:07.070 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 9 16 24593 24577 170.645 142.188 0.0058173 0.00585515 2024-09-11T08:10:07.090 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:10:07.120 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:07.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:07.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:10:07.222 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:07.248 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:07.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:07.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:07.350 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:10:07.379 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:07.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:07.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:10:07.481 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:10:07.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:07.515 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:07.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:10:07.617 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:10:07.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:07.653 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:07.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:10:07.754 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:10:07.796 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:07.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:07.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:10:07.898 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:10:07.933 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:07.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:07.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:10:07.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:07 smithi150 ceph-mon[29276]: pgmap v88: 148 pgs: 148 active+clean; 868 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-09-11T08:10:07.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:07 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:10:07.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:07 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:10:07.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:07 smithi150 ceph-mon[39612]: pgmap v88: 148 pgs: 148 active+clean; 868 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-09-11T08:10:07.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:07 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:10:07.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:07 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:10:08.035 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:10:08.069 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:08.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:08.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:10:08.070 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 10 16 26852 26836 167.699 141.188 0.00647935 0.00595791 2024-09-11T08:10:08.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:07 smithi181 ceph-mon[32701]: pgmap v88: 148 pgs: 148 active+clean; 868 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-09-11T08:10:08.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:07 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:10:08.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:07 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:10:08.074 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:10:07 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:10:08.171 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:10:08.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:08.201 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:08.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:10:08.302 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:10:08.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:08.353 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:08.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:10:08.454 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:08.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:08.495 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:08.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:08.596 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:08.619 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:08.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:08.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:08.722 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:10:08.759 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:08.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:08.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:10:08.861 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:10:08.869 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:08 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:10:08.870 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:10:08 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:10:08] "GET /metrics HTTP/1.1" 200 223167 "" "Prometheus/2.43.0" 2024-09-11T08:10:08.870 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:08 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:10:08.889 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:08.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:08.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:10:08.991 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:10:09.020 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:09.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:09.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:10:09.070 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 11 16 28880 28864 163.975 126.75 0.00914374 0.00609263 2024-09-11T08:10:09.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:08 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:10:09.123 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:09.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:09.161 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:09.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:09.262 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:10:09.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:09.300 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:09.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:10:09.401 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:09.437 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:09.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:09.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:09.539 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:10:09.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:09.579 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:09.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:10:09.680 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:10:09.711 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:09.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:09.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:10:09.813 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:10:09.855 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:09.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:09.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:10:09.957 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:10:09.994 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:09.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:09.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:10:10.070 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 12 16 30777 30761 160.19 118.562 0.00775125 0.00623665 2024-09-11T08:10:10.097 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:10:10.127 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:10.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:10.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:10:10.167 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:09 smithi150 ceph-mon[29276]: pgmap v89: 148 pgs: 148 active+clean; 1.6 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 162 MiB/s wr, 2.59k op/s 2024-09-11T08:10:10.167 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:09 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:10:10.167 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:09 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:10:10.167 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:09 smithi150 ceph-mon[39612]: pgmap v89: 148 pgs: 148 active+clean; 1.6 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 162 MiB/s wr, 2.59k op/s 2024-09-11T08:10:10.167 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:09 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:10:10.167 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:09 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:10:10.229 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:10:10.257 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:10.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:10.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:10:10.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:09 smithi181 ceph-mon[32701]: pgmap v89: 148 pgs: 148 active+clean; 1.6 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 162 MiB/s wr, 2.59k op/s 2024-09-11T08:10:10.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:09 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:10:10.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:09 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:10:10.359 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:10.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:10.400 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:10.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:10.501 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:10.529 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:10.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:10.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:10.631 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:10.660 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:10.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:10.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:10.762 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:10:10.790 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:10.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:10.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:10:10.891 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:10:10.929 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:10.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:10.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:10:11.031 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:10:11.060 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:11.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:11.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:10:11.070 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 13 16 32653 32637 156.886 117.25 0.00892011 0.00636917 2024-09-11T08:10:11.162 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:11.190 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:11.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:11.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:11.292 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:10:11.328 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:11.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:11.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:10:11.430 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:10:11.468 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:11.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:11.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:10:11.570 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:11.599 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:11.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:11.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:11.702 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:11.737 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:11.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:11.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:11.839 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:11.864 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:11.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:11.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:11.966 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:11.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:11.994 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:11.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:12.070 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 14 15 34327 34312 153.157 104.688 0.00796431 0.00652395 2024-09-11T08:10:12.095 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:12.125 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:12.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:12.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:12.147 INFO:teuthology.orchestra.run.smithi150.stderr:reweighted osd.0 to 1 (10000) 2024-09-11T08:10:12.148 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:11 smithi150 ceph-mon[29276]: pgmap v90: 148 pgs: 148 active+clean; 1.6 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 162 MiB/s wr, 2.59k op/s 2024-09-11T08:10:12.148 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:11 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3533680146' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-11T08:10:12.148 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:11 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-11T08:10:12.148 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:11 smithi150 ceph-mon[39612]: pgmap v90: 148 pgs: 148 active+clean; 1.6 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 162 MiB/s wr, 2.59k op/s 2024-09-11T08:10:12.148 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:11 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3533680146' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-11T08:10:12.148 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:11 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-11T08:10:12.227 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:10:12.262 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:12.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:12.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:10:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:11 smithi181 ceph-mon[32701]: pgmap v90: 148 pgs: 148 active+clean; 1.6 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 162 MiB/s wr, 2.59k op/s 2024-09-11T08:10:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:11 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3533680146' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-11T08:10:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:11 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-11T08:10:12.364 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:10:12.392 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:12.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:12.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:10:12.494 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:12.518 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:12.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:12.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:12.620 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:10:12.662 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:12.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:12.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:10:12.764 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:10:12.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:12.792 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:12.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:10:12.893 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:10:12.932 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:12.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:12.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:10:13.034 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:13.063 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:13.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:13.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:13.070 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 15 16 36241 36225 150.918 119.562 0.00306046 0.00662055 2024-09-11T08:10:13.165 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:10:13.202 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:13.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:13.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:10:13.209 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:12 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-11T08:10:13.209 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:12 smithi150 ceph-mon[39612]: osdmap e99: 8 total, 8 up, 8 in 2024-09-11T08:10:13.209 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:12 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-11T08:10:13.209 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:12 smithi150 ceph-mon[29276]: osdmap e99: 8 total, 8 up, 8 in 2024-09-11T08:10:13.268 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-11T08:10:13.304 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:10:13.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:12 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-11T08:10:13.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:12 smithi181 ceph-mon[32701]: osdmap e99: 8 total, 8 up, 8 in 2024-09-11T08:10:13.369 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:13.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:13.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:10:13.472 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:13.498 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:13.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:13.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:13.600 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:13.628 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:13.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:13.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:13.731 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:10:13.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:13.761 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:13.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:10:13.863 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:13.910 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:13.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:13.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:14.012 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:10:14.042 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:14.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:14.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:10:14.071 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 16 16 37949 37933 148.157 106.75 0.00625052 0.00674351 2024-09-11T08:10:14.144 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:14.181 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:14.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:14.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:14.218 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:13 smithi150 ceph-mon[29276]: pgmap v92: 148 pgs: 148 active+clean; 2.2 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 135 MiB/s wr, 2.16k op/s 2024-09-11T08:10:14.218 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:13 smithi150 ceph-mon[39612]: pgmap v92: 148 pgs: 148 active+clean; 2.2 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 135 MiB/s wr, 2.16k op/s 2024-09-11T08:10:14.265 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd reweight 1 1 2024-09-11T08:10:14.284 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:10:14.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:13 smithi181 ceph-mon[32701]: pgmap v92: 148 pgs: 148 active+clean; 2.2 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 135 MiB/s wr, 2.16k op/s 2024-09-11T08:10:14.348 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:14.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:14.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:10:14.409 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:10:14.450 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:10:14.475 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:14.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:14.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:10:14.479 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:10:14.577 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:14.624 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:14.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:14.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:14.727 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:14.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:14.778 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:14.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:14.875 INFO:teuthology.orchestra.run.smithi150.stderr:noscrub is unset 2024-09-11T08:10:14.879 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:14.905 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:14.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:14.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:15.007 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:10:15.048 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:15.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:15.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:10:15.071 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 17 16 39626 39610 145.606 104.812 0.00366386 0.00686245 2024-09-11T08:10:15.144 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:14 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/540639326' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:10:15.144 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:14 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:10:15.150 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:10:15.176 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:15.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:15.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:10:15.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:14 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/540639326' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:10:15.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:14 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:10:15.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:14 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/540639326' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:10:15.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:14 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:10:15.278 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:10:15.307 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:15.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:15.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:10:15.409 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:10:15.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:15.452 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:15.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:10:15.553 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:15.576 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:10:15 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:10:15] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:10:15.581 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:15.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:15.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:15.684 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:15.724 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:15.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:15.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:15.825 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:10:15.855 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:15.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:15.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:10:15.957 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:15.986 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:15.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:15.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:15.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:15 smithi150 ceph-mon[29276]: pgmap v93: 148 pgs: 148 active+clean; 2.2 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 135 MiB/s wr, 2.16k op/s 2024-09-11T08:10:15.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:15 smithi150 ceph-mon[29276]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:10:15.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:15 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:10:15.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:15 smithi150 ceph-mon[29276]: osdmap e100: 8 total, 8 up, 8 in 2024-09-11T08:10:15.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:15 smithi150 ceph-mon[29276]: 5.a scrub starts 2024-09-11T08:10:15.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:15 smithi150 ceph-mon[29276]: 5.a scrub ok 2024-09-11T08:10:15.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:15 smithi150 ceph-mon[29276]: osdmap e101: 8 total, 8 up, 8 in 2024-09-11T08:10:15.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:15 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:10:15.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:15 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:10:15.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:15 smithi150 ceph-mon[29276]: 5.11 scrub starts 2024-09-11T08:10:15.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:15 smithi150 ceph-mon[29276]: 5.11 scrub ok 2024-09-11T08:10:15.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:15 smithi150 ceph-mon[39612]: pgmap v93: 148 pgs: 148 active+clean; 2.2 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 135 MiB/s wr, 2.16k op/s 2024-09-11T08:10:15.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:15 smithi150 ceph-mon[39612]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:10:15.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:15 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:10:15.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:15 smithi150 ceph-mon[39612]: osdmap e100: 8 total, 8 up, 8 in 2024-09-11T08:10:15.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:15 smithi150 ceph-mon[39612]: 5.a scrub starts 2024-09-11T08:10:15.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:15 smithi150 ceph-mon[39612]: 5.a scrub ok 2024-09-11T08:10:15.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:15 smithi150 ceph-mon[39612]: osdmap e101: 8 total, 8 up, 8 in 2024-09-11T08:10:15.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:15 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:10:15.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:15 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:10:15.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:15 smithi150 ceph-mon[39612]: 5.11 scrub starts 2024-09-11T08:10:15.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:15 smithi150 ceph-mon[39612]: 5.11 scrub ok 2024-09-11T08:10:16.071 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 18 16 41538 41522 144.155 119.5 0.00582483 0.00693246 2024-09-11T08:10:16.088 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:10:16.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:16.127 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:16.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:10:16.229 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:16.267 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:16.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:16.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:16.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:15 smithi181 ceph-mon[32701]: pgmap v93: 148 pgs: 148 active+clean; 2.2 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 135 MiB/s wr, 2.16k op/s 2024-09-11T08:10:16.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:15 smithi181 ceph-mon[32701]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:10:16.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:15 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:10:16.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:15 smithi181 ceph-mon[32701]: osdmap e100: 8 total, 8 up, 8 in 2024-09-11T08:10:16.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:15 smithi181 ceph-mon[32701]: 5.a scrub starts 2024-09-11T08:10:16.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:15 smithi181 ceph-mon[32701]: 5.a scrub ok 2024-09-11T08:10:16.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:15 smithi181 ceph-mon[32701]: osdmap e101: 8 total, 8 up, 8 in 2024-09-11T08:10:16.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:15 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:10:16.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:15 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:10:16.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:15 smithi181 ceph-mon[32701]: 5.11 scrub starts 2024-09-11T08:10:16.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:15 smithi181 ceph-mon[32701]: 5.11 scrub ok 2024-09-11T08:10:16.370 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:10:16.411 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:16.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:16.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:10:16.512 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:10:16.539 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:16.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:16.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:10:16.641 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:10:16.680 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:16.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:16.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:10:16.782 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:10:16.811 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:16.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:16.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:10:16.913 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:16.954 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:16.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:16.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:17.057 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:17.071 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 19 16 43338 43322 142.487 112.5 0.0187808 0.00701263 2024-09-11T08:10:17.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:17.097 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:17.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:17.198 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:17.228 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:17.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:17.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:17.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:16 smithi150 ceph-mon[29276]: 6.8 scrub starts 2024-09-11T08:10:17.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:16 smithi150 ceph-mon[29276]: 6.8 scrub ok 2024-09-11T08:10:17.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:16 smithi150 ceph-mon[29276]: 3.1c scrub starts 2024-09-11T08:10:17.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:16 smithi150 ceph-mon[29276]: 3.1c scrub ok 2024-09-11T08:10:17.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:16 smithi150 ceph-mon[29276]: 6.18 scrub starts 2024-09-11T08:10:17.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:16 smithi150 ceph-mon[29276]: 6.18 scrub ok 2024-09-11T08:10:17.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:16 smithi150 ceph-mon[29276]: 3.8 scrub starts 2024-09-11T08:10:17.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:16 smithi150 ceph-mon[29276]: 3.8 scrub ok 2024-09-11T08:10:17.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:16 smithi150 ceph-mon[29276]: osdmap e102: 8 total, 8 up, 8 in 2024-09-11T08:10:17.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:16 smithi150 ceph-mon[29276]: 5.19 scrub starts 2024-09-11T08:10:17.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:16 smithi150 ceph-mon[29276]: 5.19 scrub ok 2024-09-11T08:10:17.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:16 smithi150 ceph-mon[29276]: 5.1f scrub starts 2024-09-11T08:10:17.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:16 smithi150 ceph-mon[29276]: 5.1f scrub ok 2024-09-11T08:10:17.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:16 smithi150 ceph-mon[39612]: 6.8 scrub starts 2024-09-11T08:10:17.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:16 smithi150 ceph-mon[39612]: 6.8 scrub ok 2024-09-11T08:10:17.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:16 smithi150 ceph-mon[39612]: 3.1c scrub starts 2024-09-11T08:10:17.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:16 smithi150 ceph-mon[39612]: 3.1c scrub ok 2024-09-11T08:10:17.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:16 smithi150 ceph-mon[39612]: 6.18 scrub starts 2024-09-11T08:10:17.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:16 smithi150 ceph-mon[39612]: 6.18 scrub ok 2024-09-11T08:10:17.242 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:16 smithi150 ceph-mon[39612]: 3.8 scrub starts 2024-09-11T08:10:17.242 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:16 smithi150 ceph-mon[39612]: 3.8 scrub ok 2024-09-11T08:10:17.242 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:16 smithi150 ceph-mon[39612]: osdmap e102: 8 total, 8 up, 8 in 2024-09-11T08:10:17.242 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:16 smithi150 ceph-mon[39612]: 5.19 scrub starts 2024-09-11T08:10:17.242 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:16 smithi150 ceph-mon[39612]: 5.19 scrub ok 2024-09-11T08:10:17.242 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:16 smithi150 ceph-mon[39612]: 5.1f scrub starts 2024-09-11T08:10:17.242 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:16 smithi150 ceph-mon[39612]: 5.1f scrub ok 2024-09-11T08:10:17.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:16 smithi181 ceph-mon[32701]: 6.8 scrub starts 2024-09-11T08:10:17.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:16 smithi181 ceph-mon[32701]: 6.8 scrub ok 2024-09-11T08:10:17.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:16 smithi181 ceph-mon[32701]: 3.1c scrub starts 2024-09-11T08:10:17.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:16 smithi181 ceph-mon[32701]: 3.1c scrub ok 2024-09-11T08:10:17.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:16 smithi181 ceph-mon[32701]: 6.18 scrub starts 2024-09-11T08:10:17.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:16 smithi181 ceph-mon[32701]: 6.18 scrub ok 2024-09-11T08:10:17.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:16 smithi181 ceph-mon[32701]: 3.8 scrub starts 2024-09-11T08:10:17.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:16 smithi181 ceph-mon[32701]: 3.8 scrub ok 2024-09-11T08:10:17.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:16 smithi181 ceph-mon[32701]: osdmap e102: 8 total, 8 up, 8 in 2024-09-11T08:10:17.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:16 smithi181 ceph-mon[32701]: 5.19 scrub starts 2024-09-11T08:10:17.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:16 smithi181 ceph-mon[32701]: 5.19 scrub ok 2024-09-11T08:10:17.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:16 smithi181 ceph-mon[32701]: 5.1f scrub starts 2024-09-11T08:10:17.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:16 smithi181 ceph-mon[32701]: 5.1f scrub ok 2024-09-11T08:10:17.331 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:10:17.359 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:17.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:17.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:10:17.461 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:10:17.497 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:17.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:17.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:10:17.600 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:10:17.648 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:17.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:17.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:10:17.750 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:10:17.778 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:17.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:17.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:10:17.879 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:17.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:17.907 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:17.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:18.008 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:18.036 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:18.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:18.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:18.072 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:2024-09-11T08:10:18.071988+0000 min lat: 0.00117796 max lat: 0.0864319 avg lat: 0.00708782 2024-09-11T08:10:18.072 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-11T08:10:18.072 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 20 15 45135 45120 140.98 112.375 0.011765 0.00708782 2024-09-11T08:10:18.073 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:10:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:10:18.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:17 smithi181 ceph-mon[32701]: pgmap v97: 148 pgs: 148 active+clean; 2.2 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 122 MiB/s wr, 1.95k op/s 2024-09-11T08:10:18.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:17 smithi181 ceph-mon[32701]: 2.2 scrub starts 2024-09-11T08:10:18.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:17 smithi181 ceph-mon[32701]: 2.2 scrub ok 2024-09-11T08:10:18.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:17 smithi181 ceph-mon[32701]: 5.d scrub starts 2024-09-11T08:10:18.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:17 smithi181 ceph-mon[32701]: 5.d scrub ok 2024-09-11T08:10:18.138 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:10:18.153 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:17 smithi150 ceph-mon[29276]: pgmap v97: 148 pgs: 148 active+clean; 2.2 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 122 MiB/s wr, 1.95k op/s 2024-09-11T08:10:18.153 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:17 smithi150 ceph-mon[29276]: 2.2 scrub starts 2024-09-11T08:10:18.153 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:17 smithi150 ceph-mon[29276]: 2.2 scrub ok 2024-09-11T08:10:18.153 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:17 smithi150 ceph-mon[29276]: 5.d scrub starts 2024-09-11T08:10:18.153 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:17 smithi150 ceph-mon[29276]: 5.d scrub ok 2024-09-11T08:10:18.154 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:17 smithi150 ceph-mon[39612]: pgmap v97: 148 pgs: 148 active+clean; 2.2 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 122 MiB/s wr, 1.95k op/s 2024-09-11T08:10:18.154 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:17 smithi150 ceph-mon[39612]: 2.2 scrub starts 2024-09-11T08:10:18.154 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:17 smithi150 ceph-mon[39612]: 2.2 scrub ok 2024-09-11T08:10:18.154 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:17 smithi150 ceph-mon[39612]: 5.d scrub starts 2024-09-11T08:10:18.154 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:17 smithi150 ceph-mon[39612]: 5.d scrub ok 2024-09-11T08:10:18.177 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:18.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:18.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:10:18.321 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:10:18.361 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:18.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:18.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:10:18.442 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T08:10:18.463 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:10:18.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:18.494 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:18.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:10:18.595 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:10:18.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:18.640 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:18.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:10:18.739 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:10:18 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:10:18] "GET /metrics HTTP/1.1" 200 223167 "" "Prometheus/2.43.0" 2024-09-11T08:10:18.741 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:10:18.773 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:18.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:18.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:10:18.876 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:10:18.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:18.918 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:18.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:10:18.951 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:10:19.019 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:10:19.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:19.052 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:19.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:10:19.072 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 21 16 46803 46787 139.223 104.188 0.00455423 0.0071773 2024-09-11T08:10:19.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:18 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:10:19.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:18 smithi181 ceph-mon[32701]: 3.3 scrub starts 2024-09-11T08:10:19.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:18 smithi181 ceph-mon[32701]: 3.3 scrub ok 2024-09-11T08:10:19.154 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:10:19.187 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:19.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:19.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:10:19.229 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:18 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:10:19.229 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:18 smithi150 ceph-mon[29276]: 3.3 scrub starts 2024-09-11T08:10:19.230 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:18 smithi150 ceph-mon[29276]: 3.3 scrub ok 2024-09-11T08:10:19.230 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:18 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:10:19.230 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:18 smithi150 ceph-mon[39612]: 3.3 scrub starts 2024-09-11T08:10:19.230 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:18 smithi150 ceph-mon[39612]: 3.3 scrub ok 2024-09-11T08:10:19.288 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:10:19.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:19.316 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:19.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:10:19.417 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:10:19.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:19.459 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:19.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:10:19.560 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:10:19.607 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:19.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:19.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:10:19.709 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:19.739 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:19.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:19.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:19.840 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:10:19.870 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:19.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:19.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:10:19.972 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:10:20.006 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:20.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:20.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:10:20.072 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 22 16 48413 48397 137.468 100.625 0.00564287 0.00726918 2024-09-11T08:10:20.108 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:10:20.145 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:20.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:20.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:10:20.171 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:19 smithi181 ceph-mon[32701]: pgmap v98: 148 pgs: 148 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 94 MiB/s wr, 1.50k op/s 2024-09-11T08:10:20.171 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:19 smithi181 ceph-mon[32701]: 4.15 scrub starts 2024-09-11T08:10:20.171 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:19 smithi181 ceph-mon[32701]: 4.15 scrub ok 2024-09-11T08:10:20.171 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:19 smithi181 ceph-mon[32701]: 3.1f scrub starts 2024-09-11T08:10:20.171 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:19 smithi181 ceph-mon[32701]: 3.1f scrub ok 2024-09-11T08:10:20.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:19 smithi150 ceph-mon[39612]: pgmap v98: 148 pgs: 148 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 94 MiB/s wr, 1.50k op/s 2024-09-11T08:10:20.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:19 smithi150 ceph-mon[39612]: 4.15 scrub starts 2024-09-11T08:10:20.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:19 smithi150 ceph-mon[39612]: 4.15 scrub ok 2024-09-11T08:10:20.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:19 smithi150 ceph-mon[39612]: 3.1f scrub starts 2024-09-11T08:10:20.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:19 smithi150 ceph-mon[39612]: 3.1f scrub ok 2024-09-11T08:10:20.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:19 smithi150 ceph-mon[29276]: pgmap v98: 148 pgs: 148 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 94 MiB/s wr, 1.50k op/s 2024-09-11T08:10:20.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:19 smithi150 ceph-mon[29276]: 4.15 scrub starts 2024-09-11T08:10:20.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:19 smithi150 ceph-mon[29276]: 4.15 scrub ok 2024-09-11T08:10:20.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:19 smithi150 ceph-mon[29276]: 3.1f scrub starts 2024-09-11T08:10:20.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:19 smithi150 ceph-mon[29276]: 3.1f scrub ok 2024-09-11T08:10:20.247 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:10:20.249 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-11T08:10:20.249 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [ 2024-09-11T08:10:20.249 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:20.249 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:47884 7.b 7:d8b88b20:::benchmark_data_smithi150_78508_object47883:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-11T08:10:20.249 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:10:19.767844+0000", 2024-09-11T08:10:20.249 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.0041880770000000001, 2024-09-11T08:10:20.250 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0042210019999999997, 2024-09-11T08:10:20.250 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:10:20.250 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "waiting for sub ops", 2024-09-11T08:10:20.250 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:10:20.250 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:10:20.250 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:10:20.250 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 47884 2024-09-11T08:10:20.250 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:20.250 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:10:20.250 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:20.250 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:10:20.250 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.767844+0000", 2024-09-11T08:10:20.250 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:20.251 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:20.251 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:20.251 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:10:20.251 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.767844+0000", 2024-09-11T08:10:20.251 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:20.251 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:20.251 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:20.251 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:10:20.251 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.767828+0000", 2024-09-11T08:10:20.251 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999847 2024-09-11T08:10:20.251 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:20.251 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:20.251 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:10:20.252 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.767943+0000", 2024-09-11T08:10:20.252 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000114527 2024-09-11T08:10:20.252 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:20.252 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:20.252 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:10:20.252 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.768008+0000", 2024-09-11T08:10:20.252 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.5383999999999996e-05 2024-09-11T08:10:20.252 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:20.252 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:20.252 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:10:20.252 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.768060+0000", 2024-09-11T08:10:20.252 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.1629999999999999e-05 2024-09-11T08:10:20.253 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:20.253 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:20.253 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:10:20.253 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.768319+0000", 2024-09-11T08:10:20.253 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00025934900000000003 2024-09-11T08:10:20.253 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:20.253 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:20.253 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:10:20.253 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.768440+0000", 2024-09-11T08:10:20.253 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000121116 2024-09-11T08:10:20.253 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:20.253 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:20.253 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,6", 2024-09-11T08:10:20.253 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.768575+0000", 2024-09-11T08:10:20.254 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000135127 2024-09-11T08:10:20.254 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:20.254 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:20.254 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:20.254 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.771007+0000", 2024-09-11T08:10:20.254 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0024312880000000002 2024-09-11T08:10:20.254 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:20.254 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:10:20.254 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:20.254 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:20.254 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:20.254 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop_reply(client.14790.0:47884 7.b e102/91)", 2024-09-11T08:10:20.254 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:10:19.771408+0000", 2024-09-11T08:10:20.254 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.00062402300000000003, 2024-09-11T08:10:20.255 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00073454, 2024-09-11T08:10:20.255 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:10:20.255 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "queued for pg", 2024-09-11T08:10:20.255 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:10:20.255 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:20.255 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:10:20.255 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.771408+0000", 2024-09-11T08:10:20.255 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:20.255 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:20.255 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:20.256 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:10:20.256 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.771408+0000", 2024-09-11T08:10:20.256 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:20.256 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:20.256 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:20.256 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:10:20.256 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.771407+0000", 2024-09-11T08:10:20.256 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:10:20.256 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:20.256 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:20.256 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:10:20.256 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.771413+0000", 2024-09-11T08:10:20.256 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.6170000000000003e-06 2024-09-11T08:10:20.256 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:20.257 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:20.257 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:10:20.257 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.771424+0000", 2024-09-11T08:10:20.257 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.025e-05 2024-09-11T08:10:20.257 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:20.257 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:20.257 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:10:20.257 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.771429+0000", 2024-09-11T08:10:20.257 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.3709999999999999e-06 2024-09-11T08:10:20.257 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:20.257 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:10:20.257 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:20.257 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:20.257 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:20.258 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.14790.0:47885 7.9 e102/91 7:969e5b1c:::benchmark_data_smithi150_78508_object47884:head v 102'3052, mlcod=102'3050)", 2024-09-11T08:10:20.258 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:10:19.771809+0000", 2024-09-11T08:10:20.258 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.00022253900000000001, 2024-09-11T08:10:20.258 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000386654, 2024-09-11T08:10:20.258 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:10:20.258 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "started", 2024-09-11T08:10:20.258 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:10:20.258 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:20.258 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:10:20.258 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.771809+0000", 2024-09-11T08:10:20.258 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:20.258 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:20.258 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:20.258 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:10:20.258 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.771809+0000", 2024-09-11T08:10:20.259 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:20.259 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:20.259 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:20.259 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:10:20.259 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.771808+0000", 2024-09-11T08:10:20.259 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:10:20.259 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:20.259 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:20.259 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:10:20.259 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.771834+0000", 2024-09-11T08:10:20.259 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.6010999999999999e-05 2024-09-11T08:10:20.259 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:20.259 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:20.259 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:10:20.260 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.771844+0000", 2024-09-11T08:10:20.260 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.4609999999999997e-06 2024-09-11T08:10:20.260 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:20.260 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:20.260 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:10:20.260 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.771849+0000", 2024-09-11T08:10:20.260 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.8269999999999997e-06 2024-09-11T08:10:20.260 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:20.260 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:20.260 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:10:20.260 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.772020+0000", 2024-09-11T08:10:20.260 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00017078800000000001 2024-09-11T08:10:20.260 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:20.260 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:20.261 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:10:20.261 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.772037+0000", 2024-09-11T08:10:20.261 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.7757999999999998e-05 2024-09-11T08:10:20.261 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:20.261 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:10:20.261 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:20.261 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:20.261 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-09-11T08:10:20.261 INFO:teuthology.orchestra.run.smithi150.stdout: "num_ops": 3 2024-09-11T08:10:20.261 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-11T08:10:20.273 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:20.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:20.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:10:20.375 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:20.404 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:20.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:20.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:20.506 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:10:20.536 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:20.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:20.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:10:20.638 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:10:20.677 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:20.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:20.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:10:20.779 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:20.804 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:20.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:20.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:20.906 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:10:20.944 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:20.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:20.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:10:21.046 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:21.073 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 23 16 50069 50053 135.991 103.5 0.00821729 0.00734775 2024-09-11T08:10:21.073 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:21.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:21.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:21.161 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:20 smithi150 ceph-mon[29276]: 3.d scrub starts 2024-09-11T08:10:21.161 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:20 smithi150 ceph-mon[29276]: 3.d scrub ok 2024-09-11T08:10:21.161 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:20 smithi150 ceph-mon[29276]: 5.6 scrub starts 2024-09-11T08:10:21.161 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:20 smithi150 ceph-mon[29276]: 5.6 scrub ok 2024-09-11T08:10:21.162 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:20 smithi150 ceph-mon[29276]: 3.c scrub starts 2024-09-11T08:10:21.162 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:20 smithi150 ceph-mon[29276]: 3.c scrub ok 2024-09-11T08:10:21.162 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:20 smithi150 ceph-mon[29276]: osdmap e103: 8 total, 8 up, 8 in 2024-09-11T08:10:21.162 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:20 smithi150 ceph-mon[29276]: 4.5 scrub starts 2024-09-11T08:10:21.162 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:20 smithi150 ceph-mon[29276]: 4.5 scrub ok 2024-09-11T08:10:21.163 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:20 smithi150 ceph-mon[39612]: 3.d scrub starts 2024-09-11T08:10:21.163 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:20 smithi150 ceph-mon[39612]: 3.d scrub ok 2024-09-11T08:10:21.163 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:20 smithi150 ceph-mon[39612]: 5.6 scrub starts 2024-09-11T08:10:21.163 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:20 smithi150 ceph-mon[39612]: 5.6 scrub ok 2024-09-11T08:10:21.163 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:20 smithi150 ceph-mon[39612]: 3.c scrub starts 2024-09-11T08:10:21.163 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:20 smithi150 ceph-mon[39612]: 3.c scrub ok 2024-09-11T08:10:21.163 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:20 smithi150 ceph-mon[39612]: osdmap e103: 8 total, 8 up, 8 in 2024-09-11T08:10:21.163 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:20 smithi150 ceph-mon[39612]: 4.5 scrub starts 2024-09-11T08:10:21.163 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:20 smithi150 ceph-mon[39612]: 4.5 scrub ok 2024-09-11T08:10:21.164 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-11T08:10:21.175 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:21.202 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:21.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:21.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:21.304 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:21.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:20 smithi181 ceph-mon[32701]: 3.d scrub starts 2024-09-11T08:10:21.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:20 smithi181 ceph-mon[32701]: 3.d scrub ok 2024-09-11T08:10:21.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:20 smithi181 ceph-mon[32701]: 5.6 scrub starts 2024-09-11T08:10:21.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:20 smithi181 ceph-mon[32701]: 5.6 scrub ok 2024-09-11T08:10:21.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:20 smithi181 ceph-mon[32701]: 3.c scrub starts 2024-09-11T08:10:21.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:20 smithi181 ceph-mon[32701]: 3.c scrub ok 2024-09-11T08:10:21.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:20 smithi181 ceph-mon[32701]: osdmap e103: 8 total, 8 up, 8 in 2024-09-11T08:10:21.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:20 smithi181 ceph-mon[32701]: 4.5 scrub starts 2024-09-11T08:10:21.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:20 smithi181 ceph-mon[32701]: 4.5 scrub ok 2024-09-11T08:10:21.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:21.331 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:21.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:21.432 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:21.500 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:21.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:21.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:21.602 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:10:21.669 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:10:21.669 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:10:21.688 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:21.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:21.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:10:21.703 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:10:21.704 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:10:21.739 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:10:21.739 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:10:21.783 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:10:21.783 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:10:21.790 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:21.813 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:10:21.813 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:10:21.818 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:21.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:21.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:21.845 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:10:21.846 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:10:21.875 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:10:21.875 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:10:21.908 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:10:21.908 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:10:21.921 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:21.960 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:21.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:21.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:21.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:21 smithi150 ceph-mon[29276]: pgmap v100: 148 pgs: 148 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 875 B/s rd, 96 MiB/s wr, 1.54k op/s 2024-09-11T08:10:21.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:21 smithi150 ceph-mon[29276]: 6.d scrub starts 2024-09-11T08:10:21.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:21 smithi150 ceph-mon[29276]: 6.d scrub ok 2024-09-11T08:10:21.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:21 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1090569269' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:10:21.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:21 smithi150 ceph-mon[29276]: 4.a scrub starts 2024-09-11T08:10:21.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:21 smithi150 ceph-mon[29276]: 4.a scrub ok 2024-09-11T08:10:21.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:21 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:10:21.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:21 smithi150 ceph-mon[39612]: pgmap v100: 148 pgs: 148 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 875 B/s rd, 96 MiB/s wr, 1.54k op/s 2024-09-11T08:10:21.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:21 smithi150 ceph-mon[39612]: 6.d scrub starts 2024-09-11T08:10:21.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:21 smithi150 ceph-mon[39612]: 6.d scrub ok 2024-09-11T08:10:21.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:21 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1090569269' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:10:21.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:21 smithi150 ceph-mon[39612]: 4.a scrub starts 2024-09-11T08:10:21.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:21 smithi150 ceph-mon[39612]: 4.a scrub ok 2024-09-11T08:10:21.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:21 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:10:22.062 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:10:22.073 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 24 16 51465 51449 133.957 87.25 0.008754 0.00745992 2024-09-11T08:10:22.100 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:22.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:22.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:10:22.201 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:22.238 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:22.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:22.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:22.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:21 smithi181 ceph-mon[32701]: pgmap v100: 148 pgs: 148 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 875 B/s rd, 96 MiB/s wr, 1.54k op/s 2024-09-11T08:10:22.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:21 smithi181 ceph-mon[32701]: 6.d scrub starts 2024-09-11T08:10:22.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:21 smithi181 ceph-mon[32701]: 6.d scrub ok 2024-09-11T08:10:22.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:21 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1090569269' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:10:22.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:21 smithi181 ceph-mon[32701]: 4.a scrub starts 2024-09-11T08:10:22.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:21 smithi181 ceph-mon[32701]: 4.a scrub ok 2024-09-11T08:10:22.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:21 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:10:22.339 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:10:22.382 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:22.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:22.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:10:22.484 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:10:22.517 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:10:22.545 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:22.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:22.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:10:22.650 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:10:22.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:22.682 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:22.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:10:22.783 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:10:22.822 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:22.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:22.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:10:22.924 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:10:22.951 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:22.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:22.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:10:23.053 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:10:23.074 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 25 16 52961 52945 132.332 93.5 0.0172347 0.00755158 2024-09-11T08:10:23.080 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:23.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:23.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:10:23.134 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:22 smithi150 ceph-mon[29276]: 3.4 scrub starts 2024-09-11T08:10:23.134 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:22 smithi150 ceph-mon[29276]: 3.4 scrub ok 2024-09-11T08:10:23.134 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:22 smithi150 ceph-mon[39612]: 3.4 scrub starts 2024-09-11T08:10:23.134 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:22 smithi150 ceph-mon[39612]: 3.4 scrub ok 2024-09-11T08:10:23.182 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:23.207 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:23.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:23.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:23.309 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:10:23.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:22 smithi181 ceph-mon[32701]: 3.4 scrub starts 2024-09-11T08:10:23.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:22 smithi181 ceph-mon[32701]: 3.4 scrub ok 2024-09-11T08:10:23.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:23.356 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:23.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:10:23.458 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:23.499 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:23.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:23.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:23.601 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:10:23.641 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:23.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:23.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:10:23.743 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:10:23.778 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:23.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:23.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:10:23.879 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:10:23.916 INFO:teuthology.orchestra.run.smithi150.stderr:reweighted osd.1 to 1 (10000) 2024-09-11T08:10:23.923 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:23.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:23.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:10:24.025 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:24.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:24.068 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:24.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:24.074 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 26 15 54212 54197 130.252 78.25 0.0148861 0.00767176 2024-09-11T08:10:24.169 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:10:24.207 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:24.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:24.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:10:24.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:23 smithi150 ceph-mon[29276]: 6.5 scrub starts 2024-09-11T08:10:24.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:23 smithi150 ceph-mon[29276]: 6.5 scrub ok 2024-09-11T08:10:24.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:23 smithi150 ceph-mon[29276]: pgmap v101: 148 pgs: 148 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.4 KiB/s rd, 127 MiB/s wr, 2.04k op/s 2024-09-11T08:10:24.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:23 smithi150 ceph-mon[29276]: 5.b scrub starts 2024-09-11T08:10:24.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:23 smithi150 ceph-mon[29276]: 5.b scrub ok 2024-09-11T08:10:24.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:23 smithi150 ceph-mon[29276]: 4.f scrub starts 2024-09-11T08:10:24.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:23 smithi150 ceph-mon[29276]: 4.f scrub ok 2024-09-11T08:10:24.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:23 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/4253168959' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-11T08:10:24.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:23 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-11T08:10:24.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:23 smithi150 ceph-mon[39612]: 6.5 scrub starts 2024-09-11T08:10:24.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:23 smithi150 ceph-mon[39612]: 6.5 scrub ok 2024-09-11T08:10:24.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:23 smithi150 ceph-mon[39612]: pgmap v101: 148 pgs: 148 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.4 KiB/s rd, 127 MiB/s wr, 2.04k op/s 2024-09-11T08:10:24.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:23 smithi150 ceph-mon[39612]: 5.b scrub starts 2024-09-11T08:10:24.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:23 smithi150 ceph-mon[39612]: 5.b scrub ok 2024-09-11T08:10:24.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:23 smithi150 ceph-mon[39612]: 4.f scrub starts 2024-09-11T08:10:24.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:23 smithi150 ceph-mon[39612]: 4.f scrub ok 2024-09-11T08:10:24.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:23 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/4253168959' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-11T08:10:24.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:23 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-11T08:10:24.310 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:24.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:23 smithi181 ceph-mon[32701]: 6.5 scrub starts 2024-09-11T08:10:24.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:23 smithi181 ceph-mon[32701]: 6.5 scrub ok 2024-09-11T08:10:24.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:23 smithi181 ceph-mon[32701]: pgmap v101: 148 pgs: 148 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.4 KiB/s rd, 127 MiB/s wr, 2.04k op/s 2024-09-11T08:10:24.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:23 smithi181 ceph-mon[32701]: 5.b scrub starts 2024-09-11T08:10:24.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:23 smithi181 ceph-mon[32701]: 5.b scrub ok 2024-09-11T08:10:24.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:23 smithi181 ceph-mon[32701]: 4.f scrub starts 2024-09-11T08:10:24.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:23 smithi181 ceph-mon[32701]: 4.f scrub ok 2024-09-11T08:10:24.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:23 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/4253168959' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-11T08:10:24.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:23 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-11T08:10:24.335 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:24.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:24.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:24.437 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:24.462 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:24.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:24.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:24.563 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:10:24.589 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:24.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:24.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:10:24.692 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:10:24.717 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:24.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:24.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:10:24.819 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:10:24.855 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:24.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:24.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:10:24.958 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:10:24.984 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:24.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:24.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:10:25.075 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 27 16 55566 55550 128.559 84.5625 0.0132627 0.0077726 2024-09-11T08:10:25.078 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:24 smithi150 ceph-mon[29276]: 3.15 scrub starts 2024-09-11T08:10:25.078 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:24 smithi150 ceph-mon[29276]: 3.15 scrub ok 2024-09-11T08:10:25.078 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:24 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-11T08:10:25.078 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:24 smithi150 ceph-mon[29276]: osdmap e104: 8 total, 8 up, 8 in 2024-09-11T08:10:25.078 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:24 smithi150 ceph-mon[29276]: 2.1 scrub starts 2024-09-11T08:10:25.078 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:24 smithi150 ceph-mon[29276]: 3.16 scrub starts 2024-09-11T08:10:25.078 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:24 smithi150 ceph-mon[29276]: 2.1 scrub ok 2024-09-11T08:10:25.079 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:24 smithi150 ceph-mon[29276]: 3.16 scrub ok 2024-09-11T08:10:25.079 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:24 smithi150 ceph-mon[29276]: 3.7 scrub starts 2024-09-11T08:10:25.079 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:24 smithi150 ceph-mon[29276]: 3.7 scrub ok 2024-09-11T08:10:25.079 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:24 smithi150 ceph-mon[29276]: 4.1f scrub starts 2024-09-11T08:10:25.079 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:24 smithi150 ceph-mon[29276]: 4.1f scrub ok 2024-09-11T08:10:25.079 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:24 smithi150 ceph-mon[39612]: 3.15 scrub starts 2024-09-11T08:10:25.079 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:24 smithi150 ceph-mon[39612]: 3.15 scrub ok 2024-09-11T08:10:25.079 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:24 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-11T08:10:25.079 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:24 smithi150 ceph-mon[39612]: osdmap e104: 8 total, 8 up, 8 in 2024-09-11T08:10:25.079 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:24 smithi150 ceph-mon[39612]: 2.1 scrub starts 2024-09-11T08:10:25.079 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:24 smithi150 ceph-mon[39612]: 3.16 scrub starts 2024-09-11T08:10:25.079 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:24 smithi150 ceph-mon[39612]: 2.1 scrub ok 2024-09-11T08:10:25.079 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:24 smithi150 ceph-mon[39612]: 3.16 scrub ok 2024-09-11T08:10:25.080 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:24 smithi150 ceph-mon[39612]: 3.7 scrub starts 2024-09-11T08:10:25.080 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:24 smithi150 ceph-mon[39612]: 3.7 scrub ok 2024-09-11T08:10:25.080 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:24 smithi150 ceph-mon[39612]: 4.1f scrub starts 2024-09-11T08:10:25.080 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:24 smithi150 ceph-mon[39612]: 4.1f scrub ok 2024-09-11T08:10:25.086 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:25.112 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:25.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:25.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:25.116 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd reweight 2 1 2024-09-11T08:10:25.214 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:25.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:25.250 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:25.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:25.323 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:10:25 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:10:25] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:10:25.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:24 smithi181 ceph-mon[32701]: 3.15 scrub starts 2024-09-11T08:10:25.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:24 smithi181 ceph-mon[32701]: 3.15 scrub ok 2024-09-11T08:10:25.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:24 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-11T08:10:25.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:24 smithi181 ceph-mon[32701]: osdmap e104: 8 total, 8 up, 8 in 2024-09-11T08:10:25.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:24 smithi181 ceph-mon[32701]: 2.1 scrub starts 2024-09-11T08:10:25.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:24 smithi181 ceph-mon[32701]: 3.16 scrub starts 2024-09-11T08:10:25.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:24 smithi181 ceph-mon[32701]: 2.1 scrub ok 2024-09-11T08:10:25.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:24 smithi181 ceph-mon[32701]: 3.16 scrub ok 2024-09-11T08:10:25.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:24 smithi181 ceph-mon[32701]: 3.7 scrub starts 2024-09-11T08:10:25.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:24 smithi181 ceph-mon[32701]: 3.7 scrub ok 2024-09-11T08:10:25.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:24 smithi181 ceph-mon[32701]: 4.1f scrub starts 2024-09-11T08:10:25.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:24 smithi181 ceph-mon[32701]: 4.1f scrub ok 2024-09-11T08:10:25.351 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:10:25.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:25.396 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:25.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:10:25.497 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:10:25.526 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:25.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:25.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:10:25.628 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:25.653 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:25.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:25.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:25.754 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:25.783 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:25.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:25.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:25.885 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:25.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:25.929 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:25.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:26.030 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:10:26.064 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:26.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:26.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:10:26.075 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 28 16 56956 56940 127.07 86.875 0.00408516 0.00786422 2024-09-11T08:10:26.166 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:10:26.192 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:26.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:26.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:10:26.219 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:25 smithi150 ceph-mon[29276]: pgmap v103: 148 pgs: 148 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 132 MiB/s wr, 2.12k op/s 2024-09-11T08:10:26.219 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:25 smithi150 ceph-mon[29276]: osdmap e105: 8 total, 8 up, 8 in 2024-09-11T08:10:26.219 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:25 smithi150 ceph-mon[29276]: 3.0 scrub starts 2024-09-11T08:10:26.219 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:25 smithi150 ceph-mon[29276]: 3.0 scrub ok 2024-09-11T08:10:26.219 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:25 smithi150 ceph-mon[39612]: pgmap v103: 148 pgs: 148 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 132 MiB/s wr, 2.12k op/s 2024-09-11T08:10:26.220 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:25 smithi150 ceph-mon[39612]: osdmap e105: 8 total, 8 up, 8 in 2024-09-11T08:10:26.220 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:25 smithi150 ceph-mon[39612]: 3.0 scrub starts 2024-09-11T08:10:26.220 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:25 smithi150 ceph-mon[39612]: 3.0 scrub ok 2024-09-11T08:10:26.294 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:26.320 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:26.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:26.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:26.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:25 smithi181 ceph-mon[32701]: pgmap v103: 148 pgs: 148 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 132 MiB/s wr, 2.12k op/s 2024-09-11T08:10:26.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:25 smithi181 ceph-mon[32701]: osdmap e105: 8 total, 8 up, 8 in 2024-09-11T08:10:26.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:25 smithi181 ceph-mon[32701]: 3.0 scrub starts 2024-09-11T08:10:26.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:25 smithi181 ceph-mon[32701]: 3.0 scrub ok 2024-09-11T08:10:26.422 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:26.447 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:26.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:26.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:26.549 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:26.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:26.597 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:26.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:26.698 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:10:26.725 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:26.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:26.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:10:26.827 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:10:26.840 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:10:26.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:26.894 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:26.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:10:26.961 INFO:teuthology.orchestra.run.smithi150.stderr:nodeep-scrub is unset 2024-09-11T08:10:26.996 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:10:27.007 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:10:27.019 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:27.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:27.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:10:27.077 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 29 16 58264 58248 125.5 81.75 0.0111767 0.00796289 2024-09-11T08:10:27.121 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:10:27.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:27.161 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:27.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:10:27.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:26 smithi150 ceph-mon[29276]: 3.1d scrub starts 2024-09-11T08:10:27.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:26 smithi150 ceph-mon[29276]: 3.1d scrub ok 2024-09-11T08:10:27.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:26 smithi150 ceph-mon[29276]: 3.9 scrub starts 2024-09-11T08:10:27.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:26 smithi150 ceph-mon[29276]: 3.9 scrub ok 2024-09-11T08:10:27.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:26 smithi150 ceph-mon[29276]: 4.7 scrub starts 2024-09-11T08:10:27.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:26 smithi150 ceph-mon[29276]: 4.7 scrub ok 2024-09-11T08:10:27.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:26 smithi150 ceph-mon[29276]: 5.2 scrub starts 2024-09-11T08:10:27.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:26 smithi150 ceph-mon[29276]: 5.2 scrub ok 2024-09-11T08:10:27.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:26 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2534064155' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:10:27.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:26 smithi150 ceph-mon[39612]: 3.1d scrub starts 2024-09-11T08:10:27.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:26 smithi150 ceph-mon[39612]: 3.1d scrub ok 2024-09-11T08:10:27.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:26 smithi150 ceph-mon[39612]: 3.9 scrub starts 2024-09-11T08:10:27.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:26 smithi150 ceph-mon[39612]: 3.9 scrub ok 2024-09-11T08:10:27.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:26 smithi150 ceph-mon[39612]: 4.7 scrub starts 2024-09-11T08:10:27.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:26 smithi150 ceph-mon[39612]: 4.7 scrub ok 2024-09-11T08:10:27.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:26 smithi150 ceph-mon[39612]: 5.2 scrub starts 2024-09-11T08:10:27.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:26 smithi150 ceph-mon[39612]: 5.2 scrub ok 2024-09-11T08:10:27.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:26 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2534064155' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:10:27.262 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:10:27.302 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:27.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:27.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:10:27.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:26 smithi181 ceph-mon[32701]: 3.1d scrub starts 2024-09-11T08:10:27.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:26 smithi181 ceph-mon[32701]: 3.1d scrub ok 2024-09-11T08:10:27.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:26 smithi181 ceph-mon[32701]: 3.9 scrub starts 2024-09-11T08:10:27.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:26 smithi181 ceph-mon[32701]: 3.9 scrub ok 2024-09-11T08:10:27.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:26 smithi181 ceph-mon[32701]: 4.7 scrub starts 2024-09-11T08:10:27.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:26 smithi181 ceph-mon[32701]: 4.7 scrub ok 2024-09-11T08:10:27.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:26 smithi181 ceph-mon[32701]: 5.2 scrub starts 2024-09-11T08:10:27.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:26 smithi181 ceph-mon[32701]: 5.2 scrub ok 2024-09-11T08:10:27.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:26 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2534064155' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:10:27.404 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:27.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:27.440 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:27.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:27.541 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:10:27.570 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:27.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:27.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:10:27.668 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T08:10:27.672 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:10:27.699 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:27.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:27.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:10:27.801 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:10:27.828 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:27.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:27.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:10:27.930 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:10:27.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:27.964 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:27.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:10:27.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:27 smithi150 ceph-mon[29276]: pgmap v105: 148 pgs: 148 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 808 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-09-11T08:10:27.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:27 smithi150 ceph-mon[29276]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:10:27.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:27 smithi150 ceph-mon[29276]: Cluster is now healthy 2024-09-11T08:10:27.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:27 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2534064155' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:10:27.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:27 smithi150 ceph-mon[29276]: osdmap e106: 8 total, 8 up, 8 in 2024-09-11T08:10:27.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:27 smithi150 ceph-mon[29276]: 5.8 scrub starts 2024-09-11T08:10:27.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:27 smithi150 ceph-mon[29276]: 5.8 scrub ok 2024-09-11T08:10:27.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:27 smithi150 ceph-mon[29276]: 5.c scrub starts 2024-09-11T08:10:27.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:27 smithi150 ceph-mon[29276]: 5.c scrub ok 2024-09-11T08:10:27.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:27 smithi150 ceph-mon[39612]: pgmap v105: 148 pgs: 148 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 808 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-09-11T08:10:27.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:27 smithi150 ceph-mon[39612]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:10:27.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:27 smithi150 ceph-mon[39612]: Cluster is now healthy 2024-09-11T08:10:27.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:27 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2534064155' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:10:27.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:27 smithi150 ceph-mon[39612]: osdmap e106: 8 total, 8 up, 8 in 2024-09-11T08:10:27.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:27 smithi150 ceph-mon[39612]: 5.8 scrub starts 2024-09-11T08:10:27.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:27 smithi150 ceph-mon[39612]: 5.8 scrub ok 2024-09-11T08:10:27.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:27 smithi150 ceph-mon[39612]: 5.c scrub starts 2024-09-11T08:10:27.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:27 smithi150 ceph-mon[39612]: 5.c scrub ok 2024-09-11T08:10:28.065 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:10:28.073 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:10:27 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:10:28.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:27 smithi181 ceph-mon[32701]: pgmap v105: 148 pgs: 148 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 808 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-09-11T08:10:28.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:27 smithi181 ceph-mon[32701]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:10:28.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:27 smithi181 ceph-mon[32701]: Cluster is now healthy 2024-09-11T08:10:28.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:27 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2534064155' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:10:28.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:27 smithi181 ceph-mon[32701]: osdmap e106: 8 total, 8 up, 8 in 2024-09-11T08:10:28.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:27 smithi181 ceph-mon[32701]: 5.8 scrub starts 2024-09-11T08:10:28.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:27 smithi181 ceph-mon[32701]: 5.8 scrub ok 2024-09-11T08:10:28.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:27 smithi181 ceph-mon[32701]: 5.c scrub starts 2024-09-11T08:10:28.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:27 smithi181 ceph-mon[32701]: 5.c scrub ok 2024-09-11T08:10:28.078 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 30 16 59649 59633 124.195 86.5625 0.00957929 0.00804697 2024-09-11T08:10:28.093 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:28.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:28.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:10:28.194 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:10:28.215 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:28.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:28.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:10:28.317 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:28.346 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:28.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:28.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:28.448 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:28.477 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:28.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:28.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:28.579 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:10:28.601 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:28.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:28.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:10:28.703 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:28.730 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:28.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:28.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:28.832 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:28.853 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:28.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:28.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:28.949 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:10:28 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:10:28] "GET /metrics HTTP/1.1" 200 223503 "" "Prometheus/2.43.0" 2024-09-11T08:10:28.955 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:10:28.995 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:28.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:28.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:10:29.078 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 31 16 60763 60747 122.435 69.625 0.00965405 0.00816249 2024-09-11T08:10:29.098 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:29.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:29.127 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:29.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:29.228 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:10:29.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:28 smithi150 ceph-mon[29276]: 3.e scrub starts 2024-09-11T08:10:29.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:28 smithi150 ceph-mon[29276]: 3.e scrub ok 2024-09-11T08:10:29.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:28 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:10:29.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:28 smithi150 ceph-mon[29276]: 6.7 scrub starts 2024-09-11T08:10:29.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:28 smithi150 ceph-mon[29276]: 6.7 scrub ok 2024-09-11T08:10:29.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:28 smithi150 ceph-mon[29276]: 3.17 scrub starts 2024-09-11T08:10:29.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:28 smithi150 ceph-mon[29276]: 3.17 scrub ok 2024-09-11T08:10:29.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:28 smithi150 ceph-mon[29276]: 6.1d scrub starts 2024-09-11T08:10:29.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:28 smithi150 ceph-mon[29276]: 6.1d scrub ok 2024-09-11T08:10:29.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:28 smithi150 ceph-mon[39612]: 3.e scrub starts 2024-09-11T08:10:29.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:28 smithi150 ceph-mon[39612]: 3.e scrub ok 2024-09-11T08:10:29.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:28 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:10:29.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:28 smithi150 ceph-mon[39612]: 6.7 scrub starts 2024-09-11T08:10:29.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:28 smithi150 ceph-mon[39612]: 6.7 scrub ok 2024-09-11T08:10:29.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:28 smithi150 ceph-mon[39612]: 3.17 scrub starts 2024-09-11T08:10:29.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:28 smithi150 ceph-mon[39612]: 3.17 scrub ok 2024-09-11T08:10:29.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:28 smithi150 ceph-mon[39612]: 6.1d scrub starts 2024-09-11T08:10:29.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:28 smithi150 ceph-mon[39612]: 6.1d scrub ok 2024-09-11T08:10:29.256 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:29.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:29.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:10:29.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:28 smithi181 ceph-mon[32701]: 3.e scrub starts 2024-09-11T08:10:29.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:28 smithi181 ceph-mon[32701]: 3.e scrub ok 2024-09-11T08:10:29.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:28 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:10:29.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:28 smithi181 ceph-mon[32701]: 6.7 scrub starts 2024-09-11T08:10:29.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:28 smithi181 ceph-mon[32701]: 6.7 scrub ok 2024-09-11T08:10:29.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:28 smithi181 ceph-mon[32701]: 3.17 scrub starts 2024-09-11T08:10:29.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:28 smithi181 ceph-mon[32701]: 3.17 scrub ok 2024-09-11T08:10:29.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:28 smithi181 ceph-mon[32701]: 6.1d scrub starts 2024-09-11T08:10:29.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:28 smithi181 ceph-mon[32701]: 6.1d scrub ok 2024-09-11T08:10:29.358 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:10:29.395 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:29.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:29.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:10:29.497 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:10:29.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:29.525 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:29.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:10:29.626 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:10:29.665 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:29.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:29.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:10:29.767 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:10:29.788 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:29.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:29.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:10:29.890 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:10:29.926 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:29.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:29.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:10:30.028 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:10:30.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:30.058 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:30.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:10:30.078 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 32 16 62043 62027 121.109 80 0.00398194 0.00824878 2024-09-11T08:10:30.159 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:30.195 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:30.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:30.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:30.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:29 smithi150 ceph-mon[29276]: pgmap v107: 148 pgs: 148 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-09-11T08:10:30.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:29 smithi150 ceph-mon[29276]: 4.6 scrub starts 2024-09-11T08:10:30.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:29 smithi150 ceph-mon[29276]: 4.6 scrub ok 2024-09-11T08:10:30.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:29 smithi150 ceph-mon[39612]: pgmap v107: 148 pgs: 148 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-09-11T08:10:30.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:29 smithi150 ceph-mon[39612]: 4.6 scrub starts 2024-09-11T08:10:30.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:29 smithi150 ceph-mon[39612]: 4.6 scrub ok 2024-09-11T08:10:30.297 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:29 smithi181 ceph-mon[32701]: pgmap v107: 148 pgs: 148 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-09-11T08:10:30.297 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:29 smithi181 ceph-mon[32701]: 4.6 scrub starts 2024-09-11T08:10:30.297 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:29 smithi181 ceph-mon[32701]: 4.6 scrub ok 2024-09-11T08:10:30.297 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:10:30.324 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:30.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:30.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:10:30.426 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:30.468 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:30.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:30.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:30.570 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:30.608 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:30.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:30.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:30.710 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:30.738 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:30.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:30.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:30.840 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:10:30.879 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:30.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:30.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:10:30.982 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:10:31.008 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:31.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:31.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:10:31.078 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 33 15 63450 63435 120.106 88 0.00774713 0.00832105 2024-09-11T08:10:31.110 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:31.137 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:31.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:31.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:31.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:30 smithi150 ceph-mon[39612]: 6.3 scrub starts 2024-09-11T08:10:31.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:30 smithi150 ceph-mon[39612]: 6.3 scrub ok 2024-09-11T08:10:31.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:30 smithi150 ceph-mon[39612]: 6.14 scrub starts 2024-09-11T08:10:31.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:30 smithi150 ceph-mon[39612]: 6.14 scrub ok 2024-09-11T08:10:31.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:30 smithi150 ceph-mon[39612]: 6.11 scrub starts 2024-09-11T08:10:31.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:30 smithi150 ceph-mon[39612]: 6.11 scrub ok 2024-09-11T08:10:31.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:30 smithi150 ceph-mon[39612]: osdmap e107: 8 total, 8 up, 8 in 2024-09-11T08:10:31.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:30 smithi150 ceph-mon[29276]: 6.3 scrub starts 2024-09-11T08:10:31.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:30 smithi150 ceph-mon[29276]: 6.3 scrub ok 2024-09-11T08:10:31.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:30 smithi150 ceph-mon[29276]: 6.14 scrub starts 2024-09-11T08:10:31.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:30 smithi150 ceph-mon[29276]: 6.14 scrub ok 2024-09-11T08:10:31.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:30 smithi150 ceph-mon[29276]: 6.11 scrub starts 2024-09-11T08:10:31.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:30 smithi150 ceph-mon[29276]: 6.11 scrub ok 2024-09-11T08:10:31.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:30 smithi150 ceph-mon[29276]: osdmap e107: 8 total, 8 up, 8 in 2024-09-11T08:10:31.240 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:31.267 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:31.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:31.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:31.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:30 smithi181 ceph-mon[32701]: 6.3 scrub starts 2024-09-11T08:10:31.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:30 smithi181 ceph-mon[32701]: 6.3 scrub ok 2024-09-11T08:10:31.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:30 smithi181 ceph-mon[32701]: 6.14 scrub starts 2024-09-11T08:10:31.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:30 smithi181 ceph-mon[32701]: 6.14 scrub ok 2024-09-11T08:10:31.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:30 smithi181 ceph-mon[32701]: 6.11 scrub starts 2024-09-11T08:10:31.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:30 smithi181 ceph-mon[32701]: 6.11 scrub ok 2024-09-11T08:10:31.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:30 smithi181 ceph-mon[32701]: osdmap e107: 8 total, 8 up, 8 in 2024-09-11T08:10:31.370 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:10:31.411 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:31.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:31.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:10:31.514 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:10:31.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:31.559 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:31.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:10:31.660 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:10:31.703 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:31.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:31.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:10:31.805 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:10:31.830 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:31.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:31.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:10:31.932 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:31.962 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:31.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:31.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:32.064 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:10:32.079 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 34 16 64864 64848 119.171 88.3125 0.0110025 0.00838677 2024-09-11T08:10:32.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:32.106 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:32.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:10:32.206 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:10:32.227 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:31 smithi150 ceph-mon[29276]: pgmap v109: 148 pgs: 148 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 70 MiB/s wr, 1.13k op/s 2024-09-11T08:10:32.227 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:31 smithi150 ceph-mon[29276]: 4.1c deep-scrub starts 2024-09-11T08:10:32.227 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:31 smithi150 ceph-mon[29276]: 4.1c deep-scrub ok 2024-09-11T08:10:32.227 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:31 smithi150 ceph-mon[29276]: 4.d scrub starts 2024-09-11T08:10:32.227 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:31 smithi150 ceph-mon[29276]: 4.d scrub ok 2024-09-11T08:10:32.227 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:31 smithi150 ceph-mon[29276]: 6.1f scrub starts 2024-09-11T08:10:32.227 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:31 smithi150 ceph-mon[29276]: 6.1f scrub ok 2024-09-11T08:10:32.228 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:31 smithi150 ceph-mon[29276]: 5.1d deep-scrub starts 2024-09-11T08:10:32.228 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:31 smithi150 ceph-mon[29276]: 5.1d deep-scrub ok 2024-09-11T08:10:32.228 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:31 smithi150 ceph-mon[39612]: pgmap v109: 148 pgs: 148 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 70 MiB/s wr, 1.13k op/s 2024-09-11T08:10:32.228 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:31 smithi150 ceph-mon[39612]: 4.1c deep-scrub starts 2024-09-11T08:10:32.228 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:31 smithi150 ceph-mon[39612]: 4.1c deep-scrub ok 2024-09-11T08:10:32.228 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:31 smithi150 ceph-mon[39612]: 4.d scrub starts 2024-09-11T08:10:32.228 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:31 smithi150 ceph-mon[39612]: 4.d scrub ok 2024-09-11T08:10:32.228 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:31 smithi150 ceph-mon[39612]: 6.1f scrub starts 2024-09-11T08:10:32.228 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:31 smithi150 ceph-mon[39612]: 6.1f scrub ok 2024-09-11T08:10:32.228 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:31 smithi150 ceph-mon[39612]: 5.1d deep-scrub starts 2024-09-11T08:10:32.228 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:31 smithi150 ceph-mon[39612]: 5.1d deep-scrub ok 2024-09-11T08:10:32.237 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:32.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:32.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:10:32.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:31 smithi181 ceph-mon[32701]: pgmap v109: 148 pgs: 148 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 70 MiB/s wr, 1.13k op/s 2024-09-11T08:10:32.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:31 smithi181 ceph-mon[32701]: 4.1c deep-scrub starts 2024-09-11T08:10:32.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:31 smithi181 ceph-mon[32701]: 4.1c deep-scrub ok 2024-09-11T08:10:32.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:31 smithi181 ceph-mon[32701]: 4.d scrub starts 2024-09-11T08:10:32.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:31 smithi181 ceph-mon[32701]: 4.d scrub ok 2024-09-11T08:10:32.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:31 smithi181 ceph-mon[32701]: 6.1f scrub starts 2024-09-11T08:10:32.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:31 smithi181 ceph-mon[32701]: 6.1f scrub ok 2024-09-11T08:10:32.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:31 smithi181 ceph-mon[32701]: 5.1d deep-scrub starts 2024-09-11T08:10:32.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:31 smithi181 ceph-mon[32701]: 5.1d deep-scrub ok 2024-09-11T08:10:32.339 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:32.365 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:32.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:32.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:32.467 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:10:32.507 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:32.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:32.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:10:32.609 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:32.641 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:32.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:32.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:32.742 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:10:32.782 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:32.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:32.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:10:32.884 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:32.912 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:32.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:32.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:33.014 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:10:33.031 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T08:10:33.038 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:33.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:33.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:10:33.079 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 35 16 66243 66227 118.228 86.1875 0.00778619 0.0084527 2024-09-11T08:10:33.139 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:10:33.166 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:33.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:33.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:10:33.269 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:10:33.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:33.323 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:33.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:10:33.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:33 smithi181 ceph-mon[32701]: 5.f scrub starts 2024-09-11T08:10:33.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:33 smithi181 ceph-mon[32701]: 5.f scrub ok 2024-09-11T08:10:33.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:33 smithi181 ceph-mon[32701]: 5.1e scrub starts 2024-09-11T08:10:33.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:33 smithi181 ceph-mon[32701]: 5.1e scrub ok 2024-09-11T08:10:33.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:33 smithi181 ceph-mon[32701]: 4.12 scrub starts 2024-09-11T08:10:33.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:33 smithi181 ceph-mon[32701]: 4.12 scrub ok 2024-09-11T08:10:33.423 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:33.450 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:33.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:33.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:33.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:33 smithi150 ceph-mon[29276]: 5.f scrub starts 2024-09-11T08:10:33.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:33 smithi150 ceph-mon[29276]: 5.f scrub ok 2024-09-11T08:10:33.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:33 smithi150 ceph-mon[29276]: 5.1e scrub starts 2024-09-11T08:10:33.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:33 smithi150 ceph-mon[29276]: 5.1e scrub ok 2024-09-11T08:10:33.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:33 smithi150 ceph-mon[29276]: 4.12 scrub starts 2024-09-11T08:10:33.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:33 smithi150 ceph-mon[29276]: 4.12 scrub ok 2024-09-11T08:10:33.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:33 smithi150 ceph-mon[39612]: 5.f scrub starts 2024-09-11T08:10:33.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:33 smithi150 ceph-mon[39612]: 5.f scrub ok 2024-09-11T08:10:33.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:33 smithi150 ceph-mon[39612]: 5.1e scrub starts 2024-09-11T08:10:33.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:33 smithi150 ceph-mon[39612]: 5.1e scrub ok 2024-09-11T08:10:33.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:33 smithi150 ceph-mon[39612]: 4.12 scrub starts 2024-09-11T08:10:33.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:33 smithi150 ceph-mon[39612]: 4.12 scrub ok 2024-09-11T08:10:33.552 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:33.580 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:33.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:33.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:33.683 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:33.706 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:33.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:33.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:33.808 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:10:33.837 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:33.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:33.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:10:33.939 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:10:33.984 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:33.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:33.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:10:34.079 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 36 16 67537 67521 117.191 80.875 0.0031827 0.00852346 2024-09-11T08:10:34.085 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:34.109 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:34.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:34.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:34.211 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:10:34.249 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:34.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:34.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:10:34.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:34 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:10:34.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:34 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:10:34.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:34 smithi181 ceph-mon[32701]: pgmap v110: 148 pgs: 148 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.4 KiB/s rd, 99 MiB/s wr, 1.59k op/s 2024-09-11T08:10:34.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:34 smithi181 ceph-mon[32701]: 6.f scrub starts 2024-09-11T08:10:34.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:34 smithi181 ceph-mon[32701]: 6.f scrub ok 2024-09-11T08:10:34.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:34 smithi181 ceph-mon[32701]: osdmap e108: 8 total, 8 up, 8 in 2024-09-11T08:10:34.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:34 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:10:34.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:34 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:10:34.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:34 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:10:34.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:34 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:10:34.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:34 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:10:34.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:34 smithi181 ceph-mon[32701]: 3.13 scrub starts 2024-09-11T08:10:34.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:34 smithi181 ceph-mon[32701]: 3.13 scrub ok 2024-09-11T08:10:34.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:34 smithi181 ceph-mon[32701]: 6.4 scrub starts 2024-09-11T08:10:34.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:34 smithi181 ceph-mon[32701]: 6.4 scrub ok 2024-09-11T08:10:34.351 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:10:34.396 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:34.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:34.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:10:34.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:34 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:10:34.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:34 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:10:34.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:34 smithi150 ceph-mon[29276]: pgmap v110: 148 pgs: 148 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.4 KiB/s rd, 99 MiB/s wr, 1.59k op/s 2024-09-11T08:10:34.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:34 smithi150 ceph-mon[29276]: 6.f scrub starts 2024-09-11T08:10:34.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:34 smithi150 ceph-mon[29276]: 6.f scrub ok 2024-09-11T08:10:34.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:34 smithi150 ceph-mon[29276]: osdmap e108: 8 total, 8 up, 8 in 2024-09-11T08:10:34.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:34 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:10:34.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:34 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:10:34.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:34 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:10:34.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:34 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:10:34.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:34 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:10:34.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:34 smithi150 ceph-mon[29276]: 3.13 scrub starts 2024-09-11T08:10:34.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:34 smithi150 ceph-mon[29276]: 3.13 scrub ok 2024-09-11T08:10:34.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:34 smithi150 ceph-mon[29276]: 6.4 scrub starts 2024-09-11T08:10:34.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:34 smithi150 ceph-mon[29276]: 6.4 scrub ok 2024-09-11T08:10:34.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:34 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:10:34.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:34 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:10:34.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:34 smithi150 ceph-mon[39612]: pgmap v110: 148 pgs: 148 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.4 KiB/s rd, 99 MiB/s wr, 1.59k op/s 2024-09-11T08:10:34.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:34 smithi150 ceph-mon[39612]: 6.f scrub starts 2024-09-11T08:10:34.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:34 smithi150 ceph-mon[39612]: 6.f scrub ok 2024-09-11T08:10:34.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:34 smithi150 ceph-mon[39612]: osdmap e108: 8 total, 8 up, 8 in 2024-09-11T08:10:34.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:34 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:10:34.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:34 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:10:34.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:34 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:10:34.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:34 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:10:34.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:34 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:10:34.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:34 smithi150 ceph-mon[39612]: 3.13 scrub starts 2024-09-11T08:10:34.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:34 smithi150 ceph-mon[39612]: 3.13 scrub ok 2024-09-11T08:10:34.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:34 smithi150 ceph-mon[39612]: 6.4 scrub starts 2024-09-11T08:10:34.492 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:34 smithi150 ceph-mon[39612]: 6.4 scrub ok 2024-09-11T08:10:34.498 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:10:34.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:34.527 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:34.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:10:34.628 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:10:34.655 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:34.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:34.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:10:34.757 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:34.785 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:34.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:34.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:34.887 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:34.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:34.914 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:34.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:35.016 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:10:35.044 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:35.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:35.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:10:35.079 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 37 16 68795 68779 116.148 78.625 0.00477423 0.00859745 2024-09-11T08:10:35.146 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:10:35.167 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:35.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:35.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:10:35.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:35 smithi150 ceph-mon[29276]: 5.7 scrub starts 2024-09-11T08:10:35.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:35 smithi150 ceph-mon[29276]: 5.7 scrub ok 2024-09-11T08:10:35.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:35 smithi150 ceph-mon[39612]: 5.7 scrub starts 2024-09-11T08:10:35.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:35 smithi150 ceph-mon[39612]: 5.7 scrub ok 2024-09-11T08:10:35.269 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:35.297 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:35.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:35.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:35.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:35 smithi181 ceph-mon[32701]: 5.7 scrub starts 2024-09-11T08:10:35.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:35 smithi181 ceph-mon[32701]: 5.7 scrub ok 2024-09-11T08:10:35.324 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:10:35 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:10:35] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:10:35.398 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:10:35.420 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:35.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:35.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:10:35.521 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:10:35.548 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:35.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:35.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:10:35.650 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:10:35.691 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:35.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:35.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:10:35.793 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:35.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:35.842 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:35.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:35.893 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:10:35.943 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:10:36.001 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:36.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:36.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:10:36.080 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 38 16 69939 69923 114.97 71.5 0.0139387 0.00869298 2024-09-11T08:10:36.103 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:36.131 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:36.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:36.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:36.233 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:10:36.268 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:36.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:36.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:10:36.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:36 smithi181 ceph-mon[32701]: pgmap v112: 148 pgs: 148 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.3 KiB/s rd, 107 MiB/s wr, 1.72k op/s 2024-09-11T08:10:36.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:36 smithi181 ceph-mon[32701]: 3.1 scrub starts 2024-09-11T08:10:36.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:36 smithi181 ceph-mon[32701]: 3.1 scrub ok 2024-09-11T08:10:36.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:36 smithi181 ceph-mon[32701]: 4.c scrub starts 2024-09-11T08:10:36.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:36 smithi181 ceph-mon[32701]: 4.0 scrub starts 2024-09-11T08:10:36.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:36 smithi181 ceph-mon[32701]: 4.c scrub ok 2024-09-11T08:10:36.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:36 smithi181 ceph-mon[32701]: 4.0 scrub ok 2024-09-11T08:10:36.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:36 smithi181 ceph-mon[32701]: 3.19 scrub starts 2024-09-11T08:10:36.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:36 smithi181 ceph-mon[32701]: 3.19 scrub ok 2024-09-11T08:10:36.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:36 smithi181 ceph-mon[32701]: osdmap e109: 8 total, 8 up, 8 in 2024-09-11T08:10:36.369 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:10:36.407 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:36.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:36.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:10:36.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:36 smithi150 ceph-mon[29276]: pgmap v112: 148 pgs: 148 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.3 KiB/s rd, 107 MiB/s wr, 1.72k op/s 2024-09-11T08:10:36.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:36 smithi150 ceph-mon[29276]: 3.1 scrub starts 2024-09-11T08:10:36.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:36 smithi150 ceph-mon[29276]: 3.1 scrub ok 2024-09-11T08:10:36.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:36 smithi150 ceph-mon[29276]: 4.c scrub starts 2024-09-11T08:10:36.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:36 smithi150 ceph-mon[29276]: 4.0 scrub starts 2024-09-11T08:10:36.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:36 smithi150 ceph-mon[29276]: 4.c scrub ok 2024-09-11T08:10:36.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:36 smithi150 ceph-mon[29276]: 4.0 scrub ok 2024-09-11T08:10:36.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:36 smithi150 ceph-mon[29276]: 3.19 scrub starts 2024-09-11T08:10:36.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:36 smithi150 ceph-mon[29276]: 3.19 scrub ok 2024-09-11T08:10:36.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:36 smithi150 ceph-mon[29276]: osdmap e109: 8 total, 8 up, 8 in 2024-09-11T08:10:36.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:36 smithi150 ceph-mon[39612]: pgmap v112: 148 pgs: 148 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.3 KiB/s rd, 107 MiB/s wr, 1.72k op/s 2024-09-11T08:10:36.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:36 smithi150 ceph-mon[39612]: 3.1 scrub starts 2024-09-11T08:10:36.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:36 smithi150 ceph-mon[39612]: 3.1 scrub ok 2024-09-11T08:10:36.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:36 smithi150 ceph-mon[39612]: 4.c scrub starts 2024-09-11T08:10:36.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:36 smithi150 ceph-mon[39612]: 4.0 scrub starts 2024-09-11T08:10:36.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:36 smithi150 ceph-mon[39612]: 4.c scrub ok 2024-09-11T08:10:36.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:36 smithi150 ceph-mon[39612]: 4.0 scrub ok 2024-09-11T08:10:36.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:36 smithi150 ceph-mon[39612]: 3.19 scrub starts 2024-09-11T08:10:36.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:36 smithi150 ceph-mon[39612]: 3.19 scrub ok 2024-09-11T08:10:36.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:36 smithi150 ceph-mon[39612]: osdmap e109: 8 total, 8 up, 8 in 2024-09-11T08:10:36.509 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:10:36.539 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:36.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:36.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:10:36.640 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:36.678 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:36.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:36.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:36.780 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:36.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:36.808 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:36.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:36.909 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:36.910 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:10:36.938 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:36.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:36.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:37.039 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:37.066 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:37.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:37.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:37.080 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 39 16 71168 71152 113.991 76.8125 0.0202051 0.00876752 2024-09-11T08:10:37.143 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:37 smithi150 ceph-mon[29276]: 3.5 scrub starts 2024-09-11T08:10:37.143 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:37 smithi150 ceph-mon[29276]: 3.5 scrub ok 2024-09-11T08:10:37.143 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:37 smithi150 ceph-mon[29276]: 3.14 scrub starts 2024-09-11T08:10:37.143 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:37 smithi150 ceph-mon[29276]: 3.14 scrub ok 2024-09-11T08:10:37.143 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:37 smithi150 ceph-mon[29276]: 3.10 scrub starts 2024-09-11T08:10:37.143 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:37 smithi150 ceph-mon[29276]: 3.10 scrub ok 2024-09-11T08:10:37.143 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:37 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:10:37.143 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:37 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:10:37.143 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:37 smithi150 ceph-mon[39612]: 3.5 scrub starts 2024-09-11T08:10:37.144 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:37 smithi150 ceph-mon[39612]: 3.5 scrub ok 2024-09-11T08:10:37.144 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:37 smithi150 ceph-mon[39612]: 3.14 scrub starts 2024-09-11T08:10:37.144 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:37 smithi150 ceph-mon[39612]: 3.14 scrub ok 2024-09-11T08:10:37.144 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:37 smithi150 ceph-mon[39612]: 3.10 scrub starts 2024-09-11T08:10:37.144 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:37 smithi150 ceph-mon[39612]: 3.10 scrub ok 2024-09-11T08:10:37.144 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:37 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:10:37.144 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:37 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:10:37.168 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:10:37.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:37.203 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:37.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:10:37.304 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:37.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:37 smithi181 ceph-mon[32701]: 3.5 scrub starts 2024-09-11T08:10:37.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:37 smithi181 ceph-mon[32701]: 3.5 scrub ok 2024-09-11T08:10:37.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:37 smithi181 ceph-mon[32701]: 3.14 scrub starts 2024-09-11T08:10:37.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:37 smithi181 ceph-mon[32701]: 3.14 scrub ok 2024-09-11T08:10:37.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:37 smithi181 ceph-mon[32701]: 3.10 scrub starts 2024-09-11T08:10:37.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:37 smithi181 ceph-mon[32701]: 3.10 scrub ok 2024-09-11T08:10:37.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:37 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:10:37.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:37 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:10:37.330 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:37.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:37.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:37.433 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:37.475 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:37.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:37.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:37.577 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:37.604 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:37.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:37.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:37.622 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-09-11T08:10:37.622 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:10:37.707 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:37.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:37.735 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:37.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:37.836 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:10:37.881 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:37.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:37.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:10:37.983 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:10:38.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:38.010 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:38.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:10:38.035 INFO:teuthology.orchestra.run.smithi150.stderr:reweighted osd.2 to 1 (10000) 2024-09-11T08:10:38.073 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:10:37 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:10:38.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:38 smithi181 ceph-mon[32701]: pgmap v114: 148 pgs: 148 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 808 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-09-11T08:10:38.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:38 smithi181 ceph-mon[32701]: 5.15 scrub starts 2024-09-11T08:10:38.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:38 smithi181 ceph-mon[32701]: 5.15 scrub ok 2024-09-11T08:10:38.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:38 smithi181 ceph-mon[32701]: 4.14 scrub starts 2024-09-11T08:10:38.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:38 smithi181 ceph-mon[32701]: 4.14 scrub ok 2024-09-11T08:10:38.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:38 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3782353763' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-11T08:10:38.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:38 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-11T08:10:38.080 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:2024-09-11T08:10:38.080972+0000 min lat: 0.00117796 max lat: 0.181517 avg lat: 0.0088452 2024-09-11T08:10:38.080 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-11T08:10:38.081 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 40 16 72354 72338 112.995 74.125 0.00772232 0.0088452 2024-09-11T08:10:38.111 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:10:38.138 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:38.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:38.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:10:38.239 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:38.282 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:38.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:38.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:38.383 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:10:38.414 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:38.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:38.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:10:38.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:38 smithi150 ceph-mon[29276]: pgmap v114: 148 pgs: 148 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 808 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-09-11T08:10:38.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:38 smithi150 ceph-mon[29276]: 5.15 scrub starts 2024-09-11T08:10:38.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:38 smithi150 ceph-mon[29276]: 5.15 scrub ok 2024-09-11T08:10:38.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:38 smithi150 ceph-mon[29276]: 4.14 scrub starts 2024-09-11T08:10:38.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:38 smithi150 ceph-mon[29276]: 4.14 scrub ok 2024-09-11T08:10:38.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:38 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3782353763' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-11T08:10:38.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:38 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-11T08:10:38.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:38 smithi150 ceph-mon[39612]: pgmap v114: 148 pgs: 148 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 808 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-09-11T08:10:38.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:38 smithi150 ceph-mon[39612]: 5.15 scrub starts 2024-09-11T08:10:38.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:38 smithi150 ceph-mon[39612]: 5.15 scrub ok 2024-09-11T08:10:38.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:38 smithi150 ceph-mon[39612]: 4.14 scrub starts 2024-09-11T08:10:38.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:38 smithi150 ceph-mon[39612]: 4.14 scrub ok 2024-09-11T08:10:38.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:38 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3782353763' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-11T08:10:38.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:38 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-11T08:10:38.515 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:10:38.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:38.546 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:38.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:10:38.647 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:38.679 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:38.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:38.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:38.781 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:10:38.819 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:38.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:38.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:10:38.921 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:38.948 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:38.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:38.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:38.961 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:10:38 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:10:38] "GET /metrics HTTP/1.1" 200 223580 "" "Prometheus/2.43.0" 2024-09-11T08:10:39.004 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd reweight 3 1 2024-09-11T08:10:39.049 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:10:39.076 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:39.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:39.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:10:39.081 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 41 16 73629 73613 112.18 79.6875 0.0301404 0.00890825 2024-09-11T08:10:39.178 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:39.217 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:39 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:10:39.218 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:39 smithi150 ceph-mon[29276]: 6.19 scrub starts 2024-09-11T08:10:39.218 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:39 smithi150 ceph-mon[29276]: 6.19 scrub ok 2024-09-11T08:10:39.218 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:39 smithi150 ceph-mon[29276]: 5.5 deep-scrub starts 2024-09-11T08:10:39.218 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:39 smithi150 ceph-mon[29276]: 5.5 deep-scrub ok 2024-09-11T08:10:39.218 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:39 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-11T08:10:39.218 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:39 smithi150 ceph-mon[29276]: osdmap e110: 8 total, 8 up, 8 in 2024-09-11T08:10:39.218 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:39 smithi150 ceph-mon[29276]: 3.b scrub starts 2024-09-11T08:10:39.218 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:39 smithi150 ceph-mon[29276]: 3.b scrub ok 2024-09-11T08:10:39.218 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:39 smithi150 ceph-mon[29276]: 6.15 scrub starts 2024-09-11T08:10:39.218 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:39 smithi150 ceph-mon[29276]: 6.15 scrub ok 2024-09-11T08:10:39.218 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:39 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:10:39.219 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:39 smithi150 ceph-mon[39612]: 6.19 scrub starts 2024-09-11T08:10:39.219 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:39 smithi150 ceph-mon[39612]: 6.19 scrub ok 2024-09-11T08:10:39.219 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:39 smithi150 ceph-mon[39612]: 5.5 deep-scrub starts 2024-09-11T08:10:39.219 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:39 smithi150 ceph-mon[39612]: 5.5 deep-scrub ok 2024-09-11T08:10:39.219 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:39 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-11T08:10:39.219 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:39 smithi150 ceph-mon[39612]: osdmap e110: 8 total, 8 up, 8 in 2024-09-11T08:10:39.219 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:39 smithi150 ceph-mon[39612]: 3.b scrub starts 2024-09-11T08:10:39.219 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:39 smithi150 ceph-mon[39612]: 3.b scrub ok 2024-09-11T08:10:39.219 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:39 smithi150 ceph-mon[39612]: 6.15 scrub starts 2024-09-11T08:10:39.219 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:39 smithi150 ceph-mon[39612]: 6.15 scrub ok 2024-09-11T08:10:39.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:39.243 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:39.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:39.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:39 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:10:39.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:39 smithi181 ceph-mon[32701]: 6.19 scrub starts 2024-09-11T08:10:39.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:39 smithi181 ceph-mon[32701]: 6.19 scrub ok 2024-09-11T08:10:39.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:39 smithi181 ceph-mon[32701]: 5.5 deep-scrub starts 2024-09-11T08:10:39.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:39 smithi181 ceph-mon[32701]: 5.5 deep-scrub ok 2024-09-11T08:10:39.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:39 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-11T08:10:39.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:39 smithi181 ceph-mon[32701]: osdmap e110: 8 total, 8 up, 8 in 2024-09-11T08:10:39.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:39 smithi181 ceph-mon[32701]: 3.b scrub starts 2024-09-11T08:10:39.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:39 smithi181 ceph-mon[32701]: 3.b scrub ok 2024-09-11T08:10:39.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:39 smithi181 ceph-mon[32701]: 6.15 scrub starts 2024-09-11T08:10:39.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:39 smithi181 ceph-mon[32701]: 6.15 scrub ok 2024-09-11T08:10:39.345 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:39.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:39.372 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:39.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:39.473 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:39.503 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:39.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:39.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:39.605 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:10:39.640 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:39.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:39.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:10:39.742 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:39.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:39.787 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:39.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:39.888 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:10:39.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:39.930 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:39.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:10:40.031 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:40.061 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:40.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:40.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:40.081 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 42 15 74869 74854 111.356 77.5625 0.0138931 0.00897542 2024-09-11T08:10:40.163 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:40.191 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:40.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:40.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:40.293 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:40.319 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:40.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:40.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:40.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:40 smithi181 ceph-mon[32701]: pgmap v116: 148 pgs: 148 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-09-11T08:10:40.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:40 smithi181 ceph-mon[32701]: 3.12 scrub starts 2024-09-11T08:10:40.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:40 smithi181 ceph-mon[32701]: 3.12 scrub ok 2024-09-11T08:10:40.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:40 smithi181 ceph-mon[32701]: 5.4 scrub starts 2024-09-11T08:10:40.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:40 smithi181 ceph-mon[32701]: 5.4 scrub ok 2024-09-11T08:10:40.341 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:40 smithi150 ceph-mon[29276]: pgmap v116: 148 pgs: 148 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-09-11T08:10:40.341 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:40 smithi150 ceph-mon[29276]: 3.12 scrub starts 2024-09-11T08:10:40.341 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:40 smithi150 ceph-mon[29276]: 3.12 scrub ok 2024-09-11T08:10:40.341 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:40 smithi150 ceph-mon[29276]: 5.4 scrub starts 2024-09-11T08:10:40.341 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:40 smithi150 ceph-mon[29276]: 5.4 scrub ok 2024-09-11T08:10:40.341 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:40 smithi150 ceph-mon[39612]: pgmap v116: 148 pgs: 148 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-09-11T08:10:40.341 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:40 smithi150 ceph-mon[39612]: 3.12 scrub starts 2024-09-11T08:10:40.341 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:40 smithi150 ceph-mon[39612]: 3.12 scrub ok 2024-09-11T08:10:40.341 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:40 smithi150 ceph-mon[39612]: 5.4 scrub starts 2024-09-11T08:10:40.341 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:40 smithi150 ceph-mon[39612]: 5.4 scrub ok 2024-09-11T08:10:40.382 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-11T08:10:40.421 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:10:40.467 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:40.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:40.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:10:40.569 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:40.595 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:40.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:40.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:40.697 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:40.722 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:40.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:40.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:40.824 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:10:40.867 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:40.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:40.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:10:40.969 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:10:40.996 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:40.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:40.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:10:41.081 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 43 16 76072 76056 110.514 75.125 0.0197007 0.00904378 2024-09-11T08:10:41.097 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:41.113 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:41 smithi150 ceph-mon[29276]: 4.8 scrub starts 2024-09-11T08:10:41.113 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:41 smithi150 ceph-mon[29276]: 4.8 scrub ok 2024-09-11T08:10:41.113 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:41 smithi150 ceph-mon[29276]: 5.1c deep-scrub starts 2024-09-11T08:10:41.113 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:41 smithi150 ceph-mon[29276]: 5.1c deep-scrub ok 2024-09-11T08:10:41.113 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:41 smithi150 ceph-mon[29276]: 5.9 scrub starts 2024-09-11T08:10:41.113 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:41 smithi150 ceph-mon[29276]: 5.9 scrub ok 2024-09-11T08:10:41.113 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:41 smithi150 ceph-mon[29276]: osdmap e111: 8 total, 8 up, 8 in 2024-09-11T08:10:41.113 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:41 smithi150 ceph-mon[39612]: 4.8 scrub starts 2024-09-11T08:10:41.113 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:41 smithi150 ceph-mon[39612]: 4.8 scrub ok 2024-09-11T08:10:41.114 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:41 smithi150 ceph-mon[39612]: 5.1c deep-scrub starts 2024-09-11T08:10:41.114 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:41 smithi150 ceph-mon[39612]: 5.1c deep-scrub ok 2024-09-11T08:10:41.114 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:41 smithi150 ceph-mon[39612]: 5.9 scrub starts 2024-09-11T08:10:41.114 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:41 smithi150 ceph-mon[39612]: 5.9 scrub ok 2024-09-11T08:10:41.114 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:41 smithi150 ceph-mon[39612]: osdmap e111: 8 total, 8 up, 8 in 2024-09-11T08:10:41.124 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:41.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:41.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:41.181 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:10:41.226 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:10:41.280 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:41.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:41.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:10:41.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:41 smithi181 ceph-mon[32701]: 4.8 scrub starts 2024-09-11T08:10:41.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:41 smithi181 ceph-mon[32701]: 4.8 scrub ok 2024-09-11T08:10:41.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:41 smithi181 ceph-mon[32701]: 5.1c deep-scrub starts 2024-09-11T08:10:41.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:41 smithi181 ceph-mon[32701]: 5.1c deep-scrub ok 2024-09-11T08:10:41.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:41 smithi181 ceph-mon[32701]: 5.9 scrub starts 2024-09-11T08:10:41.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:41 smithi181 ceph-mon[32701]: 5.9 scrub ok 2024-09-11T08:10:41.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:41 smithi181 ceph-mon[32701]: osdmap e111: 8 total, 8 up, 8 in 2024-09-11T08:10:41.382 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:10:41.404 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:41.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:41.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:10:41.506 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:10:41.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:41.543 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:41.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:10:41.644 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:10:41.683 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:41.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:41.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:10:41.785 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:41.814 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:41.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:41.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:41.916 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:10:41.943 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:41.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:41.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:10:42.045 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:10:42.082 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:42.082 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 44 16 77239 77223 109.66 72.9375 0.0191726 0.00911348 2024-09-11T08:10:42.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:42.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:10:42.184 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:10:42.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:42.224 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:42.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:10:42.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:42 smithi181 ceph-mon[32701]: pgmap v118: 148 pgs: 148 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-09-11T08:10:42.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:42 smithi181 ceph-mon[32701]: 6.1c scrub starts 2024-09-11T08:10:42.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:42 smithi181 ceph-mon[32701]: 6.1c scrub ok 2024-09-11T08:10:42.326 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:10:42.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:42.362 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:42.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:10:42.463 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:42.484 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:42 smithi150 ceph-mon[29276]: pgmap v118: 148 pgs: 148 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-09-11T08:10:42.484 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:42 smithi150 ceph-mon[29276]: 6.1c scrub starts 2024-09-11T08:10:42.485 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:42 smithi150 ceph-mon[29276]: 6.1c scrub ok 2024-09-11T08:10:42.485 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:42 smithi150 ceph-mon[39612]: pgmap v118: 148 pgs: 148 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-09-11T08:10:42.485 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:42 smithi150 ceph-mon[39612]: 6.1c scrub starts 2024-09-11T08:10:42.485 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:42 smithi150 ceph-mon[39612]: 6.1c scrub ok 2024-09-11T08:10:42.491 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:42.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:42.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:42.593 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:42.648 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:42.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:42.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:42.751 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:10:42.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:42.781 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:42.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:10:42.882 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:10:42.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:42.931 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:42.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:10:43.032 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:10:43.059 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:43.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:43.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:10:43.079 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:10:43.082 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 45 16 78448 78432 108.9 75.5625 0.0140199 0.00917644 2024-09-11T08:10:43.161 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:10:43.188 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:43.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:43.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:10:43.290 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:10:43.338 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:43.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:43.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:10:43.439 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:10:43.480 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:43.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:43.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:10:43.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:43 smithi150 ceph-mon[29276]: 6.b scrub starts 2024-09-11T08:10:43.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:43 smithi150 ceph-mon[29276]: 6.b scrub ok 2024-09-11T08:10:43.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:43 smithi150 ceph-mon[29276]: 4.1a scrub starts 2024-09-11T08:10:43.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:43 smithi150 ceph-mon[29276]: 4.1a scrub ok 2024-09-11T08:10:43.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:43 smithi150 ceph-mon[29276]: 6.12 scrub starts 2024-09-11T08:10:43.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:43 smithi150 ceph-mon[29276]: 6.12 scrub ok 2024-09-11T08:10:43.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:43 smithi150 ceph-mon[39612]: 6.b scrub starts 2024-09-11T08:10:43.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:43 smithi150 ceph-mon[39612]: 6.b scrub ok 2024-09-11T08:10:43.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:43 smithi150 ceph-mon[39612]: 4.1a scrub starts 2024-09-11T08:10:43.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:43 smithi150 ceph-mon[39612]: 4.1a scrub ok 2024-09-11T08:10:43.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:43 smithi150 ceph-mon[39612]: 6.12 scrub starts 2024-09-11T08:10:43.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:43 smithi150 ceph-mon[39612]: 6.12 scrub ok 2024-09-11T08:10:43.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:43 smithi181 ceph-mon[32701]: 6.b scrub starts 2024-09-11T08:10:43.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:43 smithi181 ceph-mon[32701]: 6.b scrub ok 2024-09-11T08:10:43.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:43 smithi181 ceph-mon[32701]: 4.1a scrub starts 2024-09-11T08:10:43.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:43 smithi181 ceph-mon[32701]: 4.1a scrub ok 2024-09-11T08:10:43.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:43 smithi181 ceph-mon[32701]: 6.12 scrub starts 2024-09-11T08:10:43.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:43 smithi181 ceph-mon[32701]: 6.12 scrub ok 2024-09-11T08:10:43.583 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:43.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:43.608 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:43.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:43.709 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:10:43.733 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:43.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:43.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:10:43.835 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:10:43.901 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:43.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:43.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:10:44.005 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:44.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:44.046 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:44.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:44.082 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 46 15 79566 79551 108.053 69.9375 0.0436232 0.00924939 2024-09-11T08:10:44.145 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:10:44.146 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:10:44.146 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:44 smithi150 ceph-mon[29276]: pgmap v119: 148 pgs: 148 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.4 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-09-11T08:10:44.146 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:44 smithi150 ceph-mon[39612]: pgmap v119: 148 pgs: 148 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.4 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-09-11T08:10:44.147 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:44.171 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:44.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:44.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:44.182 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:10:44.183 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:10:44.217 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:10:44.217 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:10:44.261 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:10:44.262 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:10:44.273 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:44.293 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:10:44.293 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:10:44.298 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:44.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:44.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:44.324 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:10:44.324 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:10:44.360 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:10:44.360 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:10:44.390 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:10:44.391 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:10:44.400 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:44.426 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:44.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:44.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:44.528 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:44.569 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:44.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:44.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:44.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:44 smithi181 ceph-mon[32701]: pgmap v119: 148 pgs: 148 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.4 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-09-11T08:10:44.671 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:44.699 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:44.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:44.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:44.801 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:10:44.830 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:44.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:44.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:10:44.932 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:44.962 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:44.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:44.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:45.064 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:45.082 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 47 16 80755 80739 107.334 74.25 0.00917049 0.00931111 2024-09-11T08:10:45.090 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:45.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:45.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:45.103 INFO:teuthology.orchestra.run.smithi150.stderr:noscrub is set 2024-09-11T08:10:45.193 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:10:45.233 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:45.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:45.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:10:45.335 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:45.364 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:45.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:45.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:45.469 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:45.474 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:45 smithi150 ceph-mon[29276]: 5.16 scrub starts 2024-09-11T08:10:45.474 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:45 smithi150 ceph-mon[29276]: 5.16 scrub ok 2024-09-11T08:10:45.475 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:45 smithi150 ceph-mon[29276]: 4.9 scrub starts 2024-09-11T08:10:45.475 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:45 smithi150 ceph-mon[29276]: 4.9 scrub ok 2024-09-11T08:10:45.475 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:45 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/708764322' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:10:45.475 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:45 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2538907314' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:10:45.475 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:45 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:10:45.475 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:45 smithi150 ceph-mon[39612]: 5.16 scrub starts 2024-09-11T08:10:45.475 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:45 smithi150 ceph-mon[39612]: 5.16 scrub ok 2024-09-11T08:10:45.476 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:45 smithi150 ceph-mon[39612]: 4.9 scrub starts 2024-09-11T08:10:45.476 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:45 smithi150 ceph-mon[39612]: 4.9 scrub ok 2024-09-11T08:10:45.476 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:45 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/708764322' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:10:45.476 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:45 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2538907314' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:10:45.476 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:45 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:10:45.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:45.513 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:45.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:45.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:10:45 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:10:45] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:10:45.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:45 smithi181 ceph-mon[32701]: 5.16 scrub starts 2024-09-11T08:10:45.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:45 smithi181 ceph-mon[32701]: 5.16 scrub ok 2024-09-11T08:10:45.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:45 smithi181 ceph-mon[32701]: 4.9 scrub starts 2024-09-11T08:10:45.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:45 smithi181 ceph-mon[32701]: 4.9 scrub ok 2024-09-11T08:10:45.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:45 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/708764322' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:10:45.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:45 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2538907314' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:10:45.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:45 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:10:45.614 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:45.653 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:45.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:45.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:45.755 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:10:45.791 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:45.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:45.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:10:45.893 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:45.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:45.933 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:45.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:46.034 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:10:46.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:46.082 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:46.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:10:46.083 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 48 16 81917 81901 106.611 72.625 0.007966 0.00937465 2024-09-11T08:10:46.184 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:46.211 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:46.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:46.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:46.313 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:10:46.346 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:46.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:46.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:10:46.413 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:46 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:10:46.413 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:46 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:10:46.413 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:46 smithi150 ceph-mon[29276]: pgmap v120: 148 pgs: 148 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.53k op/s 2024-09-11T08:10:46.413 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:46 smithi150 ceph-mon[29276]: 5.1b scrub starts 2024-09-11T08:10:46.413 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:46 smithi150 ceph-mon[29276]: 5.1b scrub ok 2024-09-11T08:10:46.413 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:46 smithi150 ceph-mon[29276]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:10:46.413 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:46 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:10:46.414 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:46 smithi150 ceph-mon[29276]: osdmap e112: 8 total, 8 up, 8 in 2024-09-11T08:10:46.414 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:46 smithi150 ceph-mon[29276]: osdmap e113: 8 total, 8 up, 8 in 2024-09-11T08:10:46.414 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:46 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:10:46.414 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:46 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:10:46.414 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:46 smithi150 ceph-mon[39612]: pgmap v120: 148 pgs: 148 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.53k op/s 2024-09-11T08:10:46.414 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:46 smithi150 ceph-mon[39612]: 5.1b scrub starts 2024-09-11T08:10:46.414 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:46 smithi150 ceph-mon[39612]: 5.1b scrub ok 2024-09-11T08:10:46.414 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:46 smithi150 ceph-mon[39612]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:10:46.414 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:46 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:10:46.414 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:46 smithi150 ceph-mon[39612]: osdmap e112: 8 total, 8 up, 8 in 2024-09-11T08:10:46.414 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:46 smithi150 ceph-mon[39612]: osdmap e113: 8 total, 8 up, 8 in 2024-09-11T08:10:46.448 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:46.474 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:46.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:46.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:46.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:46 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:10:46.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:46 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:10:46.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:46 smithi181 ceph-mon[32701]: pgmap v120: 148 pgs: 148 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.53k op/s 2024-09-11T08:10:46.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:46 smithi181 ceph-mon[32701]: 5.1b scrub starts 2024-09-11T08:10:46.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:46 smithi181 ceph-mon[32701]: 5.1b scrub ok 2024-09-11T08:10:46.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:46 smithi181 ceph-mon[32701]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:10:46.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:46 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:10:46.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:46 smithi181 ceph-mon[32701]: osdmap e112: 8 total, 8 up, 8 in 2024-09-11T08:10:46.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:46 smithi181 ceph-mon[32701]: osdmap e113: 8 total, 8 up, 8 in 2024-09-11T08:10:46.577 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:10:46.603 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:46.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:46.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:10:46.705 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:10:46.732 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:46.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:46.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:10:46.835 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:10:46.860 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:46.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:46.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:10:46.962 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:10:46.987 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:46.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:46.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:10:47.083 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 49 16 83138 83122 105.993 76.3125 0.0113151 0.0094298 2024-09-11T08:10:47.089 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:10:47.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:47.128 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:47.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:10:47.229 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:47.256 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:47.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:47.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:47.358 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:10:47.394 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:47.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:47.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:10:47.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:47 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:10:47.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:47 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:10:47.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:47 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:10:47.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:47 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:10:47.496 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:47.513 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T08:10:47.528 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:47.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:47.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:47.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:47 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:10:47.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:47 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:10:47.631 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:47.655 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:47.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:47.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:47.757 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:47.785 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:47.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:47.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:47.886 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:47.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:47.930 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:47.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:48.031 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:10:48.070 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:48.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:48.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:10:48.083 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 50 16 84218 84202 105.223 67.5 0.0146768 0.00949895 2024-09-11T08:10:48.121 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:10:47 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:10:48.172 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:10:48.195 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:48.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:48.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:10:48.297 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:10:48.345 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:48.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:48.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:10:48.447 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:10:48.487 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:48.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:48.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:10:48.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:48 smithi150 ceph-mon[29276]: pgmap v123: 148 pgs: 148 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 808 B/s rd, 61 MiB/s wr, 969 op/s 2024-09-11T08:10:48.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:48 smithi150 ceph-mon[29276]: osdmap e114: 8 total, 8 up, 8 in 2024-09-11T08:10:48.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:48 smithi150 ceph-mon[39612]: pgmap v123: 148 pgs: 148 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 808 B/s rd, 61 MiB/s wr, 969 op/s 2024-09-11T08:10:48.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:48 smithi150 ceph-mon[39612]: osdmap e114: 8 total, 8 up, 8 in 2024-09-11T08:10:48.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:48 smithi181 ceph-mon[32701]: pgmap v123: 148 pgs: 148 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 808 B/s rd, 61 MiB/s wr, 969 op/s 2024-09-11T08:10:48.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:48 smithi181 ceph-mon[32701]: osdmap e114: 8 total, 8 up, 8 in 2024-09-11T08:10:48.590 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:48.618 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:48.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:48.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:48.720 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:48.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:48.749 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:48.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:48.850 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:10:48.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:48.894 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:48.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:10:48.898 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:10:48 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:10:48] "GET /metrics HTTP/1.1" 200 223580 "" "Prometheus/2.43.0" 2024-09-11T08:10:48.995 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:49.021 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:49.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:49.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:49.083 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 51 16 85322 85306 104.512 69 0.0086247 0.00956337 2024-09-11T08:10:49.124 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:49.154 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:49.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:49.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:49.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:49 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:10:49.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:49 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:10:49.256 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:10:49.294 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:49.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:49.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:10:49.396 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:49.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:49.439 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:49.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:49.540 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:49.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:49 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:10:49.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:49.579 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:49.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:49.679 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:10:49.723 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:49.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:49.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:10:49.825 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:49.850 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:49.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:49.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:49.952 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:50.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:50.004 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:50.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:50.083 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 52 16 86427 86411 103.831 69.0625 0.00868075 0.00962653 2024-09-11T08:10:50.105 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:10:50.131 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:50.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:50.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:10:50.233 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:50.260 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:50.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:50.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:50.314 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:50 smithi150 ceph-mon[29276]: pgmap v125: 148 pgs: 148 active+clean; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 89 MiB/s wr, 1.42k op/s 2024-09-11T08:10:50.314 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:50 smithi150 ceph-mon[39612]: pgmap v125: 148 pgs: 148 active+clean; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 89 MiB/s wr, 1.42k op/s 2024-09-11T08:10:50.362 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:10:50.385 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:50.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:50.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:10:50.486 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:10:50.527 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:50.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:50.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:10:50.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:50 smithi181 ceph-mon[32701]: pgmap v125: 148 pgs: 148 active+clean; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 89 MiB/s wr, 1.42k op/s 2024-09-11T08:10:50.629 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:50.653 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:50.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:50.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:50.755 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:10:50.781 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:50.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:50.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:10:50.822 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-11T08:10:50.823 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 20, 2024-09-11T08:10:50.823 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 600, 2024-09-11T08:10:50.823 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [ 2024-09-11T08:10:50.823 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.823 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14156.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-11T08:10:50.823 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:05:22.248788+0000", 2024-09-11T08:10:50.823 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 328.56849106300001, 2024-09-11T08:10:50.823 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.39763285199999998, 2024-09-11T08:10:50.823 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:10:50.823 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:10:50.823 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:10:50.823 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14156", 2024-09-11T08:10:50.824 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2620931070", 2024-09-11T08:10:50.824 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 1 2024-09-11T08:10:50.824 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.824 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:10:50.824 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.824 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:10:50.824 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:05:22.248788+0000", 2024-09-11T08:10:50.824 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:50.824 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.824 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.824 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:10:50.824 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:05:22.248788+0000", 2024-09-11T08:10:50.824 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:50.825 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.825 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.825 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:10:50.825 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:05:22.248787+0000", 2024-09-11T08:10:50.825 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:10:50.825 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.825 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.825 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:10:50.825 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:05:22.248799+0000", 2024-09-11T08:10:50.825 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2448e-05 2024-09-11T08:10:50.825 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.825 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.825 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:10:50.825 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:05:22.248811+0000", 2024-09-11T08:10:50.826 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1834000000000001e-05 2024-09-11T08:10:50.826 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.826 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.826 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:10:50.826 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:05:22.248818+0000", 2024-09-11T08:10:50.826 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.7070000000000003e-06 2024-09-11T08:10:50.826 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.826 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.826 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:10:50.826 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:05:22.248854+0000", 2024-09-11T08:10:50.826 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.5624000000000003e-05 2024-09-11T08:10:50.826 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.826 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.826 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:10:50.827 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:05:22.248859+0000", 2024-09-11T08:10:50.827 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.485e-06 2024-09-11T08:10:50.827 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.827 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.827 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:10:50.827 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:05:22.645367+0000", 2024-09-11T08:10:50.827 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.39650756399999998 2024-09-11T08:10:50.827 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.827 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.827 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:10:50.827 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:05:22.645500+0000", 2024-09-11T08:10:50.827 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00013353 2024-09-11T08:10:50.827 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.827 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.828 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 0,2", 2024-09-11T08:10:50.828 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:05:22.645636+0000", 2024-09-11T08:10:50.828 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00013545500000000001 2024-09-11T08:10:50.828 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.828 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.828 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:50.828 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:05:22.646292+0000", 2024-09-11T08:10:50.828 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000655917 2024-09-11T08:10:50.828 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.828 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.828 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:50.828 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:05:22.646313+0000", 2024-09-11T08:10:50.828 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.1684000000000001e-05 2024-09-11T08:10:50.829 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.829 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.829 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:10:50.829 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:05:22.646326+0000", 2024-09-11T08:10:50.829 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3011e-05 2024-09-11T08:10:50.829 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.829 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.829 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:10:50.829 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:05:22.646382+0000", 2024-09-11T08:10:50.829 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.5962999999999999e-05 2024-09-11T08:10:50.829 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.829 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.829 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:10:50.829 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:05:22.646421+0000", 2024-09-11T08:10:50.829 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.8961000000000002e-05 2024-09-11T08:10:50.830 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:50.830 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:10:50.830 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:50.830 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.830 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.830 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:6861 7.b 7:d357cde3:::benchmark_data_smithi150_78508_object6860:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-11T08:10:50.830 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:10:00.100561+0000", 2024-09-11T08:10:50.830 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 50.716718315000001, 2024-09-11T08:10:50.830 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.049186573999999997, 2024-09-11T08:10:50.830 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:10:50.831 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:10:50.831 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:10:50.831 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:10:50.831 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:10:50.831 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 6861 2024-09-11T08:10:50.831 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.831 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:10:50.831 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.831 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:10:50.831 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.100561+0000", 2024-09-11T08:10:50.831 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:50.831 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.831 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.831 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:10:50.831 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.100561+0000", 2024-09-11T08:10:50.832 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:50.832 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.832 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.832 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:10:50.832 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.100560+0000", 2024-09-11T08:10:50.832 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:10:50.832 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.832 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.832 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:10:50.832 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.100578+0000", 2024-09-11T08:10:50.833 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.7901999999999999e-05 2024-09-11T08:10:50.833 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.833 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.833 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:10:50.833 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.100600+0000", 2024-09-11T08:10:50.833 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.2864999999999999e-05 2024-09-11T08:10:50.833 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.833 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.833 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:10:50.833 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.100606+0000", 2024-09-11T08:10:50.833 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.9649999999999996e-06 2024-09-11T08:10:50.833 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.833 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.833 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:10:50.834 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.100640+0000", 2024-09-11T08:10:50.834 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.3504999999999999e-05 2024-09-11T08:10:50.834 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.834 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.834 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:10:50.834 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.100734+0000", 2024-09-11T08:10:50.834 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.3889999999999997e-05 2024-09-11T08:10:50.834 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.834 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.834 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,6", 2024-09-11T08:10:50.834 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.100877+0000", 2024-09-11T08:10:50.834 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00014291999999999999 2024-09-11T08:10:50.834 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.834 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.834 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:10:50.835 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.101682+0000", 2024-09-11T08:10:50.835 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00080484399999999998 2024-09-11T08:10:50.835 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.835 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.835 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:50.835 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.101976+0000", 2024-09-11T08:10:50.835 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00029404600000000001 2024-09-11T08:10:50.835 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.835 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.835 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:50.835 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.149661+0000", 2024-09-11T08:10:50.835 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.047685720000000001 2024-09-11T08:10:50.835 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.835 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.836 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:10:50.836 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.149709+0000", 2024-09-11T08:10:50.836 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.7147000000000001e-05 2024-09-11T08:10:50.836 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.836 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.836 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:10:50.836 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.149748+0000", 2024-09-11T08:10:50.836 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.9103000000000002e-05 2024-09-11T08:10:50.836 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:50.836 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:10:50.836 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:50.836 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.836 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.836 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:6889 7.b 7:d6179f28:::benchmark_data_smithi150_78508_object6888:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-11T08:10:50.837 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:10:00.109267+0000", 2024-09-11T08:10:50.837 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 50.708011927000001, 2024-09-11T08:10:50.837 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.041420910999999998, 2024-09-11T08:10:50.837 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:10:50.837 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:10:50.837 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:10:50.837 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:10:50.837 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:10:50.837 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 6889 2024-09-11T08:10:50.837 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.837 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:10:50.837 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.837 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:10:50.837 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.109267+0000", 2024-09-11T08:10:50.837 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:50.838 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.838 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.838 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:10:50.838 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.109267+0000", 2024-09-11T08:10:50.838 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:50.838 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.838 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.838 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:10:50.838 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.109266+0000", 2024-09-11T08:10:50.838 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:10:50.838 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.838 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.838 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:10:50.838 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.109293+0000", 2024-09-11T08:10:50.839 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.7432000000000001e-05 2024-09-11T08:10:50.839 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.839 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.839 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:10:50.839 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.109308+0000", 2024-09-11T08:10:50.839 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.5027e-05 2024-09-11T08:10:50.839 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.839 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.839 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:10:50.839 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.109315+0000", 2024-09-11T08:10:50.839 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.6290000000000004e-06 2024-09-11T08:10:50.839 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.839 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.839 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:10:50.840 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.109370+0000", 2024-09-11T08:10:50.840 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.5068999999999997e-05 2024-09-11T08:10:50.840 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.840 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.840 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:10:50.840 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.109493+0000", 2024-09-11T08:10:50.840 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012345600000000001 2024-09-11T08:10:50.840 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.840 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.840 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,6", 2024-09-11T08:10:50.840 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.109624+0000", 2024-09-11T08:10:50.840 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00013038000000000001 2024-09-11T08:10:50.840 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.840 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.841 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:10:50.841 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.110288+0000", 2024-09-11T08:10:50.841 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00066436800000000003 2024-09-11T08:10:50.841 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.841 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.841 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:50.841 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.111171+0000", 2024-09-11T08:10:50.841 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00088280499999999996 2024-09-11T08:10:50.841 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.841 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.841 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:50.841 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.150095+0000", 2024-09-11T08:10:50.841 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.038924733000000003 2024-09-11T08:10:50.841 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.841 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.842 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:10:50.842 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.150636+0000", 2024-09-11T08:10:50.842 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00054021699999999995 2024-09-11T08:10:50.842 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.842 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.842 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:10:50.842 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.150688+0000", 2024-09-11T08:10:50.842 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.2713000000000002e-05 2024-09-11T08:10:50.842 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:50.842 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:10:50.842 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:50.842 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.842 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.842 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:6890 7.b 7:d6fb44d5:::benchmark_data_smithi150_78508_object6889:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-11T08:10:50.843 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:10:00.110438+0000", 2024-09-11T08:10:50.843 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 50.706840894999999, 2024-09-11T08:10:50.843 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.040482401000000001, 2024-09-11T08:10:50.843 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:10:50.843 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:10:50.843 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:10:50.843 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:10:50.843 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:10:50.843 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 6890 2024-09-11T08:10:50.843 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.843 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:10:50.843 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.843 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:10:50.843 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.110438+0000", 2024-09-11T08:10:50.843 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:50.844 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.844 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.844 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:10:50.844 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.110438+0000", 2024-09-11T08:10:50.844 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:50.844 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.844 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.844 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:10:50.844 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.110437+0000", 2024-09-11T08:10:50.844 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:10:50.844 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.844 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.844 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:10:50.844 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.110459+0000", 2024-09-11T08:10:50.845 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.2138e-05 2024-09-11T08:10:50.845 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.845 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.845 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:10:50.845 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.110475+0000", 2024-09-11T08:10:50.845 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.5381999999999999e-05 2024-09-11T08:10:50.845 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.845 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.845 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:10:50.845 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.110481+0000", 2024-09-11T08:10:50.845 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.0379999999999996e-06 2024-09-11T08:10:50.845 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.845 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.845 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:10:50.845 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.110514+0000", 2024-09-11T08:10:50.846 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.3390999999999999e-05 2024-09-11T08:10:50.846 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.846 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.846 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:10:50.846 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.110622+0000", 2024-09-11T08:10:50.846 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000107834 2024-09-11T08:10:50.846 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.846 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.846 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,6", 2024-09-11T08:10:50.846 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.110760+0000", 2024-09-11T08:10:50.846 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00013754899999999999 2024-09-11T08:10:50.846 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.846 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.846 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:10:50.847 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.111224+0000", 2024-09-11T08:10:50.847 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00046417000000000002 2024-09-11T08:10:50.847 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.847 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.847 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:50.847 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.111667+0000", 2024-09-11T08:10:50.847 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00044293300000000001 2024-09-11T08:10:50.847 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.847 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.847 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:50.847 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.150726+0000", 2024-09-11T08:10:50.847 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.039059207999999998 2024-09-11T08:10:50.847 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.847 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.847 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:10:50.848 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.150887+0000", 2024-09-11T08:10:50.848 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00016132599999999999 2024-09-11T08:10:50.848 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.848 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.848 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:10:50.848 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:00.150921+0000", 2024-09-11T08:10:50.848 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.3599999999999997e-05 2024-09-11T08:10:50.848 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:50.848 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:10:50.848 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:50.848 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.848 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.848 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.14790.0:22282 7.9 e98/91 7:99349d2a:::benchmark_data_smithi150_78508_object22281:head v 98'1434, mlcod=98'1432)", 2024-09-11T08:10:50.848 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:10:06.016773+0000", 2024-09-11T08:10:50.849 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 44.800506339999998, 2024-09-11T08:10:50.849 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.053576353, 2024-09-11T08:10:50.849 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:10:50.849 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:10:50.849 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:10:50.849 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.849 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:10:50.849 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.016773+0000", 2024-09-11T08:10:50.850 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:50.850 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.850 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.850 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:10:50.850 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.016773+0000", 2024-09-11T08:10:50.850 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:50.850 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.850 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.850 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:10:50.850 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.016767+0000", 2024-09-11T08:10:50.850 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999938 2024-09-11T08:10:50.850 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.851 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.851 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:10:50.851 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.016991+0000", 2024-09-11T08:10:50.851 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000224897 2024-09-11T08:10:50.851 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.851 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.851 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:10:50.851 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.017002+0000", 2024-09-11T08:10:50.851 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0190000000000001e-05 2024-09-11T08:10:50.851 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.851 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.851 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:10:50.851 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.017008+0000", 2024-09-11T08:10:50.851 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.9370000000000001e-06 2024-09-11T08:10:50.851 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.852 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.852 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:10:50.852 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.017070+0000", 2024-09-11T08:10:50.852 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.2786000000000005e-05 2024-09-11T08:10:50.852 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.852 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.852 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:10:50.852 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.017092+0000", 2024-09-11T08:10:50.852 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.1554000000000001e-05 2024-09-11T08:10:50.852 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.852 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.852 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:10:50.852 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.070311+0000", 2024-09-11T08:10:50.852 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.053219482999999998 2024-09-11T08:10:50.853 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.853 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.853 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:10:50.853 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.070349+0000", 2024-09-11T08:10:50.853 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.7941000000000001e-05 2024-09-11T08:10:50.853 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:50.853 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:10:50.853 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:50.853 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.853 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.853 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.14790.0:22285 7.9 e98/91 7:99a34847:::benchmark_data_smithi150_78508_object22284:head v 98'1435, mlcod=98'1432)", 2024-09-11T08:10:50.853 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:10:06.017869+0000", 2024-09-11T08:10:50.854 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 44.799410672, 2024-09-11T08:10:50.854 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.053246899, 2024-09-11T08:10:50.854 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:10:50.854 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:10:50.854 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:10:50.854 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.854 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:10:50.854 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.017869+0000", 2024-09-11T08:10:50.854 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:50.854 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.854 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.854 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:10:50.854 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.017869+0000", 2024-09-11T08:10:50.854 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:50.855 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.855 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.855 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:10:50.855 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.017868+0000", 2024-09-11T08:10:50.855 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:10:50.855 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.855 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.855 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:10:50.855 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.017889+0000", 2024-09-11T08:10:50.855 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.0848000000000001e-05 2024-09-11T08:10:50.855 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.855 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.855 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:10:50.855 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.017898+0000", 2024-09-11T08:10:50.855 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.5009999999999993e-06 2024-09-11T08:10:50.856 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.856 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.856 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:10:50.856 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.017904+0000", 2024-09-11T08:10:50.856 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.0950000000000003e-06 2024-09-11T08:10:50.856 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.856 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.856 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:10:50.856 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.018140+0000", 2024-09-11T08:10:50.856 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000236541 2024-09-11T08:10:50.856 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.856 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.856 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:10:50.857 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.018160+0000", 2024-09-11T08:10:50.857 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.9984e-05 2024-09-11T08:10:50.857 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.857 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.857 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:10:50.857 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.071075+0000", 2024-09-11T08:10:50.857 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.052914899000000001 2024-09-11T08:10:50.857 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.857 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.857 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:10:50.857 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.071116+0000", 2024-09-11T08:10:50.857 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.0649000000000003e-05 2024-09-11T08:10:50.857 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:50.857 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:10:50.857 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:50.858 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.858 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.858 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.14790.0:22287 7.2 e98/91 7:4792e18f:::benchmark_data_smithi150_78508_object22286:head v 98'1407, mlcod=98'1404)", 2024-09-11T08:10:50.858 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:10:06.018959+0000", 2024-09-11T08:10:50.858 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 44.798320095000001, 2024-09-11T08:10:50.858 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.055898588999999999, 2024-09-11T08:10:50.858 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:10:50.858 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:10:50.858 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:10:50.858 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.858 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:10:50.858 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.018959+0000", 2024-09-11T08:10:50.858 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:50.858 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.859 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.859 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:10:50.859 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.018959+0000", 2024-09-11T08:10:50.859 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:50.859 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.859 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.859 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:10:50.859 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.018959+0000", 2024-09-11T08:10:50.859 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:10:50.859 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.859 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.859 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:10:50.859 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.018981+0000", 2024-09-11T08:10:50.859 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.2402999999999999e-05 2024-09-11T08:10:50.859 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.860 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.860 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:10:50.860 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.069255+0000", 2024-09-11T08:10:50.860 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.050273937999999997 2024-09-11T08:10:50.860 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.860 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.860 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:10:50.860 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.069271+0000", 2024-09-11T08:10:50.860 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.5973000000000001e-05 2024-09-11T08:10:50.860 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.860 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.860 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:10:50.860 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.070385+0000", 2024-09-11T08:10:50.860 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0011138330000000001 2024-09-11T08:10:50.861 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.861 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.861 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:10:50.861 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.070402+0000", 2024-09-11T08:10:50.861 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.7127999999999999e-05 2024-09-11T08:10:50.861 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.861 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.861 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:10:50.861 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.074749+0000", 2024-09-11T08:10:50.861 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0043471539999999998 2024-09-11T08:10:50.861 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.861 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.861 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:10:50.861 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.074858+0000", 2024-09-11T08:10:50.861 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00010889 2024-09-11T08:10:50.862 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:50.862 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:10:50.862 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:50.862 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.862 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.862 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.14790.0:22293 7.d e98/91 7:b0f73b14:::benchmark_data_smithi150_78508_object22292:head v 98'1417, mlcod=98'1415)", 2024-09-11T08:10:50.862 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:10:06.019913+0000", 2024-09-11T08:10:50.862 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 44.797366171999997, 2024-09-11T08:10:50.862 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.054161040000000001, 2024-09-11T08:10:50.862 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:10:50.862 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:10:50.862 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:10:50.862 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.862 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:10:50.862 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.019913+0000", 2024-09-11T08:10:50.863 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:50.863 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.863 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.863 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:10:50.863 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.019913+0000", 2024-09-11T08:10:50.863 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:50.863 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.863 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.863 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:10:50.863 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.019912+0000", 2024-09-11T08:10:50.863 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:10:50.863 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.863 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.863 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:10:50.864 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.019969+0000", 2024-09-11T08:10:50.864 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.6821999999999999e-05 2024-09-11T08:10:50.864 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.864 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.864 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:10:50.864 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.070145+0000", 2024-09-11T08:10:50.864 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.050175377 2024-09-11T08:10:50.864 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.864 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.864 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:10:50.864 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.070152+0000", 2024-09-11T08:10:50.864 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.1609999999999997e-06 2024-09-11T08:10:50.864 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.864 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.865 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:10:50.865 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.071373+0000", 2024-09-11T08:10:50.865 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0012211139999999999 2024-09-11T08:10:50.865 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.865 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.865 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:10:50.865 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.071391+0000", 2024-09-11T08:10:50.865 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.7961e-05 2024-09-11T08:10:50.865 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.865 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.865 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:10:50.865 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.074044+0000", 2024-09-11T08:10:50.865 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0026532130000000002 2024-09-11T08:10:50.865 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.866 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.866 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:10:50.866 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.074074+0000", 2024-09-11T08:10:50.866 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.0111999999999999e-05 2024-09-11T08:10:50.866 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:50.866 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:10:50.866 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:50.866 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.866 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.866 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:22309 7.b 7:d5e71d28:::benchmark_data_smithi150_78508_object22308:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-11T08:10:50.866 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:10:06.024599+0000", 2024-09-11T08:10:50.866 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 44.792680046000001, 2024-09-11T08:10:50.866 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.049224376, 2024-09-11T08:10:50.866 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:10:50.866 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:10:50.867 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:10:50.867 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:10:50.867 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:10:50.867 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 22309 2024-09-11T08:10:50.867 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.867 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:10:50.867 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.867 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:10:50.867 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.024599+0000", 2024-09-11T08:10:50.867 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:50.867 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.867 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.867 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:10:50.867 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.024599+0000", 2024-09-11T08:10:50.868 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:50.868 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.868 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.868 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:10:50.868 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.024597+0000", 2024-09-11T08:10:50.868 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:10:50.868 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.868 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.868 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:10:50.868 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.024643+0000", 2024-09-11T08:10:50.868 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.6060000000000003e-05 2024-09-11T08:10:50.868 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.868 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.868 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:10:50.869 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.069619+0000", 2024-09-11T08:10:50.869 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.044975589000000003 2024-09-11T08:10:50.869 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.869 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.869 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:10:50.869 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.069627+0000", 2024-09-11T08:10:50.869 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.2360000000000004e-06 2024-09-11T08:10:50.869 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.869 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.869 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:10:50.869 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.069735+0000", 2024-09-11T08:10:50.869 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000108232 2024-09-11T08:10:50.869 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.869 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.870 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:10:50.870 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.069893+0000", 2024-09-11T08:10:50.870 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00015809 2024-09-11T08:10:50.870 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.870 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.870 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,6", 2024-09-11T08:10:50.870 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.070053+0000", 2024-09-11T08:10:50.870 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00015950099999999999 2024-09-11T08:10:50.870 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.870 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.870 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:50.870 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.072304+0000", 2024-09-11T08:10:50.870 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0022513149999999998 2024-09-11T08:10:50.870 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.870 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.871 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:50.871 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.073731+0000", 2024-09-11T08:10:50.871 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0014264919999999999 2024-09-11T08:10:50.871 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.871 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.871 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:10:50.871 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.073737+0000", 2024-09-11T08:10:50.871 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.1890000000000004e-06 2024-09-11T08:10:50.871 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.871 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.871 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:10:50.871 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.073789+0000", 2024-09-11T08:10:50.871 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.1860000000000002e-05 2024-09-11T08:10:50.871 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.872 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.872 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:10:50.872 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.073824+0000", 2024-09-11T08:10:50.872 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.4931999999999997e-05 2024-09-11T08:10:50.872 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:50.872 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:10:50.872 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:50.872 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.872 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.872 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:22972 7.b 7:d49936e6:::benchmark_data_smithi150_78508_object22971:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-11T08:10:50.872 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:10:06.322710+0000", 2024-09-11T08:10:50.872 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 44.494569763999998, 2024-09-11T08:10:50.872 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.066878614000000003, 2024-09-11T08:10:50.872 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:10:50.872 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:10:50.873 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:10:50.873 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:10:50.873 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:10:50.873 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 22972 2024-09-11T08:10:50.873 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.873 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:10:50.873 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.873 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:10:50.873 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.322710+0000", 2024-09-11T08:10:50.873 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:50.873 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.873 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.873 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:10:50.873 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.322710+0000", 2024-09-11T08:10:50.873 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:50.874 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.874 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.874 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:10:50.874 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.322707+0000", 2024-09-11T08:10:50.874 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999971 2024-09-11T08:10:50.874 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.874 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.874 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:10:50.874 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.322740+0000", 2024-09-11T08:10:50.874 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.3139999999999998e-05 2024-09-11T08:10:50.874 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.874 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.874 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:10:50.874 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.322754+0000", 2024-09-11T08:10:50.875 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3516000000000001e-05 2024-09-11T08:10:50.875 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.875 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.875 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:10:50.875 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.322760+0000", 2024-09-11T08:10:50.875 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.1879999999999999e-06 2024-09-11T08:10:50.875 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.875 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.875 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:10:50.875 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.322821+0000", 2024-09-11T08:10:50.875 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.1629000000000002e-05 2024-09-11T08:10:50.875 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.875 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.875 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:10:50.875 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.322952+0000", 2024-09-11T08:10:50.876 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00013084300000000001 2024-09-11T08:10:50.876 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.876 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.876 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,6", 2024-09-11T08:10:50.876 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.323070+0000", 2024-09-11T08:10:50.876 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00011744900000000001 2024-09-11T08:10:50.876 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.876 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.876 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:50.876 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.324601+0000", 2024-09-11T08:10:50.876 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.001531218 2024-09-11T08:10:50.876 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.876 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.877 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:10:50.877 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.325295+0000", 2024-09-11T08:10:50.877 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000694251 2024-09-11T08:10:50.877 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.877 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.878 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:50.878 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.389504+0000", 2024-09-11T08:10:50.878 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.064209341000000003 2024-09-11T08:10:50.878 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.878 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.878 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:10:50.878 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.389554+0000", 2024-09-11T08:10:50.878 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.9577000000000003e-05 2024-09-11T08:10:50.878 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.878 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.878 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:10:50.878 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.389588+0000", 2024-09-11T08:10:50.878 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.4171999999999997e-05 2024-09-11T08:10:50.878 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:50.879 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:10:50.879 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:50.879 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.879 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.879 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:22988 7.b 7:d8373df3:::benchmark_data_smithi150_78508_object22987:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-11T08:10:50.879 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:10:06.329999+0000", 2024-09-11T08:10:50.879 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 44.487280458000001, 2024-09-11T08:10:50.879 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.060045689999999999, 2024-09-11T08:10:50.879 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:10:50.879 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:10:50.879 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:10:50.879 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:10:50.879 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:10:50.879 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 22988 2024-09-11T08:10:50.880 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.880 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:10:50.880 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.880 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:10:50.880 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.329999+0000", 2024-09-11T08:10:50.880 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:50.880 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.880 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.880 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:10:50.880 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.329999+0000", 2024-09-11T08:10:50.880 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:50.880 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.880 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.880 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:10:50.881 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.329997+0000", 2024-09-11T08:10:50.881 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:10:50.881 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.881 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.881 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:10:50.881 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.330027+0000", 2024-09-11T08:10:50.881 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.0493000000000001e-05 2024-09-11T08:10:50.881 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.881 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.881 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:10:50.881 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.330049+0000", 2024-09-11T08:10:50.881 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.1691e-05 2024-09-11T08:10:50.881 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.881 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.881 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:10:50.882 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.330056+0000", 2024-09-11T08:10:50.882 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.4130000000000003e-06 2024-09-11T08:10:50.882 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.882 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.882 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:10:50.882 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.330133+0000", 2024-09-11T08:10:50.882 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.7595000000000005e-05 2024-09-11T08:10:50.882 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.882 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.882 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:10:50.882 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.330267+0000", 2024-09-11T08:10:50.882 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00013357399999999999 2024-09-11T08:10:50.882 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.882 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.883 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,6", 2024-09-11T08:10:50.883 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.330392+0000", 2024-09-11T08:10:50.883 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012552700000000001 2024-09-11T08:10:50.883 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.883 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.883 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:50.883 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.331420+0000", 2024-09-11T08:10:50.883 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.001027465 2024-09-11T08:10:50.883 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.883 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.883 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:10:50.883 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.331440+0000", 2024-09-11T08:10:50.883 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.0236000000000001e-05 2024-09-11T08:10:50.883 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.883 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.884 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:50.884 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.389859+0000", 2024-09-11T08:10:50.884 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.058418741000000003 2024-09-11T08:10:50.884 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.884 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.884 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:10:50.884 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.389961+0000", 2024-09-11T08:10:50.884 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000102744 2024-09-11T08:10:50.884 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.884 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.884 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:10:50.884 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.390045+0000", 2024-09-11T08:10:50.884 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.3183000000000006e-05 2024-09-11T08:10:50.884 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:50.885 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:10:50.885 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:50.885 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.885 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.885 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:24351 7.b 7:d669e38d:::benchmark_data_smithi150_78508_object24350:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-11T08:10:50.885 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:10:06.949813+0000", 2024-09-11T08:10:50.885 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 43.867466229000001, 2024-09-11T08:10:50.885 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.034682921999999998, 2024-09-11T08:10:50.885 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:10:50.885 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:10:50.885 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:10:50.885 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:10:50.885 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:10:50.885 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 24351 2024-09-11T08:10:50.885 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.886 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:10:50.886 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.886 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:10:50.886 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.949813+0000", 2024-09-11T08:10:50.886 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:50.886 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.886 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.886 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:10:50.886 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.949813+0000", 2024-09-11T08:10:50.886 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:50.886 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.886 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.886 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:10:50.886 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.949811+0000", 2024-09-11T08:10:50.887 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:10:50.887 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.887 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.887 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:10:50.887 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.949837+0000", 2024-09-11T08:10:50.887 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.6392000000000001e-05 2024-09-11T08:10:50.887 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.887 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.887 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:10:50.887 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.949852+0000", 2024-09-11T08:10:50.887 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4518000000000001e-05 2024-09-11T08:10:50.887 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.887 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.887 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:10:50.887 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.949858+0000", 2024-09-11T08:10:50.888 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.9009999999999999e-06 2024-09-11T08:10:50.888 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.888 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.888 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:10:50.888 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.949976+0000", 2024-09-11T08:10:50.888 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00011793700000000001 2024-09-11T08:10:50.888 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.888 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.888 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:10:50.888 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.950222+0000", 2024-09-11T08:10:50.888 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000245858 2024-09-11T08:10:50.888 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.888 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.888 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,6", 2024-09-11T08:10:50.888 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.950353+0000", 2024-09-11T08:10:50.889 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00013145499999999999 2024-09-11T08:10:50.889 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.889 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.889 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:50.889 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.951428+0000", 2024-09-11T08:10:50.889 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.001074953 2024-09-11T08:10:50.889 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.889 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.889 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:10:50.889 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.951881+0000", 2024-09-11T08:10:50.889 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000452783 2024-09-11T08:10:50.889 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.889 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.889 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:50.890 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.984418+0000", 2024-09-11T08:10:50.890 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.032537375 2024-09-11T08:10:50.890 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.890 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.890 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:10:50.890 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.984462+0000", 2024-09-11T08:10:50.890 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.3983000000000002e-05 2024-09-11T08:10:50.890 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.890 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.890 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:10:50.890 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:06.984496+0000", 2024-09-11T08:10:50.890 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.3819000000000003e-05 2024-09-11T08:10:50.890 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:50.890 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:10:50.891 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:50.891 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.891 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.891 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:27942 7.b 7:d839a40a:::benchmark_data_smithi150_78508_object27941:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-11T08:10:50.891 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:10:08.592519+0000", 2024-09-11T08:10:50.891 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 42.224760127000003, 2024-09-11T08:10:50.891 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.048476287999999999, 2024-09-11T08:10:50.891 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:10:50.891 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:10:50.891 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:10:50.891 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:10:50.891 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:10:50.891 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 27942 2024-09-11T08:10:50.891 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.892 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:10:50.892 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.892 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:10:50.892 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:08.592519+0000", 2024-09-11T08:10:50.892 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:50.892 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.892 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.892 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:10:50.892 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:08.592519+0000", 2024-09-11T08:10:50.892 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:50.892 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.892 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.892 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:10:50.892 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:08.592517+0000", 2024-09-11T08:10:50.893 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:10:50.893 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.893 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.893 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:10:50.893 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:08.592543+0000", 2024-09-11T08:10:50.893 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.5786e-05 2024-09-11T08:10:50.893 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.893 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.893 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:10:50.893 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:08.592560+0000", 2024-09-11T08:10:50.893 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.6946999999999999e-05 2024-09-11T08:10:50.893 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.893 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.893 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:10:50.893 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:08.592567+0000", 2024-09-11T08:10:50.894 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.9e-06 2024-09-11T08:10:50.894 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.894 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.894 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:10:50.894 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:08.592618+0000", 2024-09-11T08:10:50.894 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.0918000000000003e-05 2024-09-11T08:10:50.894 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.894 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.894 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:10:50.894 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:08.592716+0000", 2024-09-11T08:10:50.894 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.8338999999999995e-05 2024-09-11T08:10:50.894 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.894 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.895 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,6", 2024-09-11T08:10:50.895 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:08.592848+0000", 2024-09-11T08:10:50.895 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00013174000000000001 2024-09-11T08:10:50.895 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.895 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.895 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:10:50.895 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:08.594761+0000", 2024-09-11T08:10:50.895 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0019134099999999999 2024-09-11T08:10:50.895 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.895 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.895 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:50.895 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:08.623969+0000", 2024-09-11T08:10:50.895 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.029207559000000001 2024-09-11T08:10:50.895 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.896 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.896 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:50.896 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:08.640926+0000", 2024-09-11T08:10:50.896 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.016956835999999999 2024-09-11T08:10:50.896 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.896 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.896 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:10:50.896 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:08.640961+0000", 2024-09-11T08:10:50.896 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.4904999999999999e-05 2024-09-11T08:10:50.896 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.896 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.896 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:10:50.896 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:08.640996+0000", 2024-09-11T08:10:50.896 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.4813e-05 2024-09-11T08:10:50.897 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:50.897 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:10:50.897 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:50.897 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.897 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.897 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:27950 7.b 7:d300b977:::benchmark_data_smithi150_78508_object27949:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-11T08:10:50.897 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:10:08.598073+0000", 2024-09-11T08:10:50.897 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 42.219206671999999, 2024-09-11T08:10:50.897 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.043041278000000002, 2024-09-11T08:10:50.897 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:10:50.897 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:10:50.897 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:10:50.897 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:10:50.897 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:10:50.897 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 27950 2024-09-11T08:10:50.898 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.898 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:10:50.898 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.898 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:10:50.898 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:08.598073+0000", 2024-09-11T08:10:50.898 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:50.898 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.898 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.898 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:10:50.898 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:08.598073+0000", 2024-09-11T08:10:50.898 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:50.898 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.898 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.898 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:10:50.898 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:08.598071+0000", 2024-09-11T08:10:50.899 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:10:50.899 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.899 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.899 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:10:50.899 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:08.598096+0000", 2024-09-11T08:10:50.899 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.5530000000000001e-05 2024-09-11T08:10:50.899 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.899 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.899 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:10:50.899 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:08.598111+0000", 2024-09-11T08:10:50.899 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.5033e-05 2024-09-11T08:10:50.899 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.899 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.899 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:10:50.900 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:08.598119+0000", 2024-09-11T08:10:50.900 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.2910000000000002e-06 2024-09-11T08:10:50.900 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.900 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.900 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:10:50.900 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:08.598168+0000", 2024-09-11T08:10:50.900 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.9465000000000002e-05 2024-09-11T08:10:50.900 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.900 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.900 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:10:50.900 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:08.598309+0000", 2024-09-11T08:10:50.900 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00014071099999999999 2024-09-11T08:10:50.900 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.900 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.901 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,6", 2024-09-11T08:10:50.901 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:08.598451+0000", 2024-09-11T08:10:50.901 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00014179199999999999 2024-09-11T08:10:50.901 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.901 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.901 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:10:50.901 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:08.599451+0000", 2024-09-11T08:10:50.901 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00099999700000000008 2024-09-11T08:10:50.901 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.901 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.901 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:50.901 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:08.624099+0000", 2024-09-11T08:10:50.901 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.024648059999999999 2024-09-11T08:10:50.901 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.901 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.902 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:50.902 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:08.641041+0000", 2024-09-11T08:10:50.902 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.016942763 2024-09-11T08:10:50.902 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.902 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.902 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:10:50.902 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:08.641080+0000", 2024-09-11T08:10:50.902 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.8374999999999998e-05 2024-09-11T08:10:50.902 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.902 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.902 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:10:50.902 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:08.641114+0000", 2024-09-11T08:10:50.902 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.4190000000000003e-05 2024-09-11T08:10:50.902 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:50.902 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:10:50.903 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:50.903 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.903 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.903 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:37095 7.b 7:d9f0128d:::benchmark_data_smithi150_78508_object37094:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-11T08:10:50.903 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:10:13.562560+0000", 2024-09-11T08:10:50.903 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 37.254719346999998, 2024-09-11T08:10:50.903 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.035428498000000003, 2024-09-11T08:10:50.903 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:10:50.903 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:10:50.903 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:10:50.903 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:10:50.903 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:10:50.903 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 37095 2024-09-11T08:10:50.903 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.904 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:10:50.904 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.904 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:10:50.904 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:13.562560+0000", 2024-09-11T08:10:50.904 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:50.904 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.904 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.904 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:10:50.904 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:13.562560+0000", 2024-09-11T08:10:50.904 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:50.904 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.904 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.904 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:10:50.904 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:13.562557+0000", 2024-09-11T08:10:50.905 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999971 2024-09-11T08:10:50.905 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.905 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.905 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:10:50.905 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:13.562587+0000", 2024-09-11T08:10:50.905 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.9848999999999998e-05 2024-09-11T08:10:50.905 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.905 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.905 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:10:50.905 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:13.562606+0000", 2024-09-11T08:10:50.905 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.859e-05 2024-09-11T08:10:50.905 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.905 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.905 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:10:50.906 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:13.562613+0000", 2024-09-11T08:10:50.906 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.3719999999999999e-06 2024-09-11T08:10:50.906 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.906 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.906 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:10:50.906 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:13.562664+0000", 2024-09-11T08:10:50.906 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.1326000000000003e-05 2024-09-11T08:10:50.906 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.906 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.906 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:10:50.906 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:13.563561+0000", 2024-09-11T08:10:50.906 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00089632799999999995 2024-09-11T08:10:50.906 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.906 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.906 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,6", 2024-09-11T08:10:50.907 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:13.563700+0000", 2024-09-11T08:10:50.907 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00013926300000000001 2024-09-11T08:10:50.907 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.907 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.907 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:10:50.907 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:13.567391+0000", 2024-09-11T08:10:50.907 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0036911050000000001 2024-09-11T08:10:50.907 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.907 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.907 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:50.907 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:13.580622+0000", 2024-09-11T08:10:50.907 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.013230555 2024-09-11T08:10:50.907 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.907 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.907 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:50.908 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:13.597661+0000", 2024-09-11T08:10:50.908 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.017039312000000001 2024-09-11T08:10:50.908 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.908 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.908 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:10:50.908 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:13.597940+0000", 2024-09-11T08:10:50.908 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00027953 2024-09-11T08:10:50.908 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.908 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.908 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:10:50.908 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:13.597989+0000", 2024-09-11T08:10:50.908 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.8052e-05 2024-09-11T08:10:50.908 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:50.908 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:10:50.909 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:50.909 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.909 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.909 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:47114 7.b 7:d8905973:::benchmark_data_smithi150_78508_object47113:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-11T08:10:50.909 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:10:19.254729+0000", 2024-09-11T08:10:50.909 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 31.5625499, 2024-09-11T08:10:50.909 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.037475213, 2024-09-11T08:10:50.909 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:10:50.909 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:10:50.909 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:10:50.909 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:10:50.909 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:10:50.909 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 47114 2024-09-11T08:10:50.909 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.909 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:10:50.910 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.910 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:10:50.910 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.254729+0000", 2024-09-11T08:10:50.910 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:50.910 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.910 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.910 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:10:50.910 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.254729+0000", 2024-09-11T08:10:50.910 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:50.910 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.910 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.910 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:10:50.910 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.254728+0000", 2024-09-11T08:10:50.910 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:10:50.911 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.911 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.911 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:10:50.911 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.254753+0000", 2024-09-11T08:10:50.911 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.5443e-05 2024-09-11T08:10:50.911 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.911 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.911 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:10:50.911 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.254766+0000", 2024-09-11T08:10:50.911 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3047999999999999e-05 2024-09-11T08:10:50.911 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.911 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.911 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:10:50.911 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.254772+0000", 2024-09-11T08:10:50.911 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.8329999999999999e-06 2024-09-11T08:10:50.912 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.912 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.912 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:10:50.912 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.256582+0000", 2024-09-11T08:10:50.912 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.001809871 2024-09-11T08:10:50.912 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.912 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.912 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:10:50.912 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.256699+0000", 2024-09-11T08:10:50.912 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000116914 2024-09-11T08:10:50.912 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.912 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.912 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,6", 2024-09-11T08:10:50.912 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.256838+0000", 2024-09-11T08:10:50.912 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000138923 2024-09-11T08:10:50.913 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.913 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.913 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:10:50.913 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.258888+0000", 2024-09-11T08:10:50.913 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.002049787 2024-09-11T08:10:50.913 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.913 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.913 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:50.913 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.260771+0000", 2024-09-11T08:10:50.913 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0018829140000000001 2024-09-11T08:10:50.913 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.913 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.913 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:50.913 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.292113+0000", 2024-09-11T08:10:50.914 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.031342722000000003 2024-09-11T08:10:50.914 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.914 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.914 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:10:50.914 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.292169+0000", 2024-09-11T08:10:50.914 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.5588999999999999e-05 2024-09-11T08:10:50.914 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.914 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.914 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:10:50.914 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.292205+0000", 2024-09-11T08:10:50.914 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.5868000000000003e-05 2024-09-11T08:10:50.914 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:50.914 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:10:50.914 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:50.914 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.915 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.915 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:47128 7.b 7:dfb80826:::benchmark_data_smithi150_78508_object47127:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-11T08:10:50.915 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:10:19.267802+0000", 2024-09-11T08:10:50.915 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 31.549477107000001, 2024-09-11T08:10:50.915 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.043745712999999999, 2024-09-11T08:10:50.915 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:10:50.915 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:10:50.915 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:10:50.915 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:10:50.915 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:10:50.915 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 47128 2024-09-11T08:10:50.915 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.915 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:10:50.915 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.916 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:10:50.916 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.267802+0000", 2024-09-11T08:10:50.916 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:50.916 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.916 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.916 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:10:50.916 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.267802+0000", 2024-09-11T08:10:50.916 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:50.916 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.916 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.916 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:10:50.916 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:10:50.917 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.267801+0000", 2024-09-11T08:10:50.918 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:10:50.918 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.918 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.918 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:10:50.918 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.267823+0000", 2024-09-11T08:10:50.918 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.1906e-05 2024-09-11T08:10:50.918 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.918 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.918 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:10:50.918 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.267834+0000", 2024-09-11T08:10:50.919 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0869999999999999e-05 2024-09-11T08:10:50.919 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.919 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.919 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:10:50.919 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.267840+0000", 2024-09-11T08:10:50.919 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.5539999999999997e-06 2024-09-11T08:10:50.919 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.919 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.919 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:10:50.920 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.269364+0000", 2024-09-11T08:10:50.920 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0015244220000000001 2024-09-11T08:10:50.920 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.920 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.920 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:10:50.920 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.269490+0000", 2024-09-11T08:10:50.920 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012545300000000001 2024-09-11T08:10:50.920 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.920 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.920 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,6", 2024-09-11T08:10:50.920 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.269635+0000", 2024-09-11T08:10:50.920 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000145191 2024-09-11T08:10:50.920 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.920 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.921 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:50.921 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.270984+0000", 2024-09-11T08:10:50.921 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.001349643 2024-09-11T08:10:50.921 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.921 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.921 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:10:50.921 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.271087+0000", 2024-09-11T08:10:50.921 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00010217600000000001 2024-09-11T08:10:50.921 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.921 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.921 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:50.921 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.311459+0000", 2024-09-11T08:10:50.921 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.040372475999999997 2024-09-11T08:10:50.921 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.922 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.922 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:10:50.922 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.311511+0000", 2024-09-11T08:10:50.922 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.1892999999999999e-05 2024-09-11T08:10:50.922 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.922 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.922 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:10:50.922 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.311548+0000", 2024-09-11T08:10:50.922 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.7092999999999999e-05 2024-09-11T08:10:50.922 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:50.922 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:10:50.922 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:50.922 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.922 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.922 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:47134 7.b 7:df57b2b3:::benchmark_data_smithi150_78508_object47133:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-11T08:10:50.923 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:10:19.273324+0000", 2024-09-11T08:10:50.923 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 31.543955715999999, 2024-09-11T08:10:50.923 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.038614585, 2024-09-11T08:10:50.923 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:10:50.923 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:10:50.923 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:10:50.923 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:10:50.923 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:10:50.923 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 47134 2024-09-11T08:10:50.923 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.923 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:10:50.923 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.923 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:10:50.923 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.273324+0000", 2024-09-11T08:10:50.924 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:50.924 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.924 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.924 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:10:50.924 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.273324+0000", 2024-09-11T08:10:50.924 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:50.924 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.924 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.924 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:10:50.924 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.273322+0000", 2024-09-11T08:10:50.924 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:10:50.924 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.924 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.924 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:10:50.925 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.273346+0000", 2024-09-11T08:10:50.925 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.4340000000000001e-05 2024-09-11T08:10:50.925 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.925 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.925 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:10:50.925 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.273360+0000", 2024-09-11T08:10:50.925 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3593e-05 2024-09-11T08:10:50.925 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.925 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.925 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:10:50.925 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.273366+0000", 2024-09-11T08:10:50.925 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.6290000000000004e-06 2024-09-11T08:10:50.925 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.925 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.925 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:10:50.926 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.273413+0000", 2024-09-11T08:10:50.926 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.6659000000000002e-05 2024-09-11T08:10:50.926 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.926 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.926 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:10:50.926 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.273534+0000", 2024-09-11T08:10:50.926 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000121248 2024-09-11T08:10:50.926 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.926 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.926 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,6", 2024-09-11T08:10:50.926 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.273675+0000", 2024-09-11T08:10:50.926 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000140739 2024-09-11T08:10:50.926 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.926 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.926 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:10:50.927 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.278825+0000", 2024-09-11T08:10:50.927 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0051504419999999999 2024-09-11T08:10:50.927 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.927 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.927 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:50.927 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.279800+0000", 2024-09-11T08:10:50.927 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00097416600000000005 2024-09-11T08:10:50.927 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.927 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.927 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:50.927 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.311670+0000", 2024-09-11T08:10:50.927 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.031869925 2024-09-11T08:10:50.927 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.927 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.928 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:10:50.928 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.311841+0000", 2024-09-11T08:10:50.928 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000171695 2024-09-11T08:10:50.928 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.928 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.928 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:10:50.928 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:19.311938+0000", 2024-09-11T08:10:50.928 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.7039000000000004e-05 2024-09-11T08:10:50.928 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:50.928 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:10:50.928 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:50.928 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.928 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.928 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:79889 7.b 7:d283088c:::benchmark_data_smithi150_78508_object79888:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-09-11T08:10:50.928 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:10:44.375347+0000", 2024-09-11T08:10:50.929 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 6.4419325799999996, 2024-09-11T08:10:50.929 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.034787594999999998, 2024-09-11T08:10:50.929 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:10:50.929 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:10:50.929 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:10:50.929 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:10:50.929 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:10:50.929 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 79889 2024-09-11T08:10:50.929 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.929 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:10:50.929 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.929 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:10:50.929 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:44.375347+0000", 2024-09-11T08:10:50.929 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:50.929 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.930 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.930 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:10:50.930 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:44.375347+0000", 2024-09-11T08:10:50.930 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:50.930 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.930 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.930 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:10:50.930 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:44.375346+0000", 2024-09-11T08:10:50.930 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:10:50.930 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.930 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.930 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:10:50.930 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:44.375366+0000", 2024-09-11T08:10:50.930 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.9621999999999999e-05 2024-09-11T08:10:50.930 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.931 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.931 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:10:50.931 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:44.375372+0000", 2024-09-11T08:10:50.931 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.857e-06 2024-09-11T08:10:50.931 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.931 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.931 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:10:50.931 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:44.375376+0000", 2024-09-11T08:10:50.931 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.16e-06 2024-09-11T08:10:50.931 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.931 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.931 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:10:50.931 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:44.386308+0000", 2024-09-11T08:10:50.931 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.010932614 2024-09-11T08:10:50.932 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.932 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.932 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:10:50.932 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:44.386421+0000", 2024-09-11T08:10:50.932 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000112656 2024-09-11T08:10:50.932 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.932 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.932 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,6", 2024-09-11T08:10:50.932 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:44.386550+0000", 2024-09-11T08:10:50.932 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012867299999999999 2024-09-11T08:10:50.932 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.932 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.932 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:50.932 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:44.390219+0000", 2024-09-11T08:10:50.933 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0036695130000000001 2024-09-11T08:10:50.933 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.933 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.933 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:50.933 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:44.390255+0000", 2024-09-11T08:10:50.933 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.5238999999999999e-05 2024-09-11T08:10:50.933 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.933 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.933 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:10:50.933 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:44.409687+0000", 2024-09-11T08:10:50.933 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.01943247 2024-09-11T08:10:50.933 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.933 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.933 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:10:50.933 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:44.410083+0000", 2024-09-11T08:10:50.934 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00039619200000000002 2024-09-11T08:10:50.934 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.934 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.934 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:10:50.934 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:44.410134+0000", 2024-09-11T08:10:50.934 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.1146999999999997e-05 2024-09-11T08:10:50.934 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:50.934 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:10:50.934 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:50.934 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.934 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.934 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:86729 7.b 7:d503d2b1:::benchmark_data_smithi150_78508_object86728:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-11T08:10:50.934 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:10:50.347627+0000", 2024-09-11T08:10:50.934 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.46965233299999998, 2024-09-11T08:10:50.934 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.034130619000000001, 2024-09-11T08:10:50.935 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:10:50.935 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:10:50.935 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:10:50.935 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:10:50.935 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:10:50.935 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 86729 2024-09-11T08:10:50.935 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.935 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:10:50.935 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.935 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:10:50.935 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:50.347627+0000", 2024-09-11T08:10:50.935 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:50.935 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.935 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.935 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:10:50.936 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:50.347627+0000", 2024-09-11T08:10:50.936 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:10:50.936 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.936 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.936 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:10:50.936 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:50.347625+0000", 2024-09-11T08:10:50.936 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:10:50.936 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.936 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.936 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:10:50.936 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:50.347653+0000", 2024-09-11T08:10:50.936 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.7662999999999999e-05 2024-09-11T08:10:50.936 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.936 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.937 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:10:50.937 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:50.347673+0000", 2024-09-11T08:10:50.937 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.9919999999999999e-05 2024-09-11T08:10:50.937 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.937 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.937 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:10:50.937 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:50.347680+0000", 2024-09-11T08:10:50.937 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.0670000000000003e-06 2024-09-11T08:10:50.937 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.937 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.937 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:10:50.937 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:50.347732+0000", 2024-09-11T08:10:50.937 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.2179000000000003e-05 2024-09-11T08:10:50.937 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.937 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.938 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:10:50.938 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:50.347876+0000", 2024-09-11T08:10:50.938 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000144289 2024-09-11T08:10:50.938 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.938 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.938 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,6", 2024-09-11T08:10:50.938 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:50.348003+0000", 2024-09-11T08:10:50.938 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000126544 2024-09-11T08:10:50.938 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.938 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.938 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:10:50.938 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:50.350855+0000", 2024-09-11T08:10:50.938 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.002851976 2024-09-11T08:10:50.938 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.938 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.939 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:50.939 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:50.381297+0000", 2024-09-11T08:10:50.939 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.030442673999999999 2024-09-11T08:10:50.939 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.939 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.939 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:10:50.939 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:50.381683+0000", 2024-09-11T08:10:50.939 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00038547199999999998 2024-09-11T08:10:50.939 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.939 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.939 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:10:50.939 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:50.381732+0000", 2024-09-11T08:10:50.939 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.9518000000000003e-05 2024-09-11T08:10:50.939 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:10:50.939 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:10:50.940 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:10:50.940 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:50.381758+0000", 2024-09-11T08:10:50.940 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.531e-05 2024-09-11T08:10:50.940 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:50.940 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:10:50.940 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:50.940 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:10:50.940 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:10:50.940 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-11T08:10:50.986 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:50.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:50.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:10:51.083 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 53 16 87472 87456 103.104 65.3125 0.0104652 0.00969397 2024-09-11T08:10:51.089 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:10:51.131 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:51.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:51.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:10:51.234 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:10:51.259 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:51.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:51.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:10:51.361 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:51.386 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:51.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:51.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:51.488 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:51.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:51 smithi150 ceph-mon[29276]: osdmap e115: 8 total, 8 up, 8 in 2024-09-11T08:10:51.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:51 smithi150 ceph-mon[29276]: pgmap v127: 148 pgs: 148 active+clean; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 909 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-09-11T08:10:51.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:51 smithi150 ceph-mon[39612]: osdmap e115: 8 total, 8 up, 8 in 2024-09-11T08:10:51.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:51 smithi150 ceph-mon[39612]: pgmap v127: 148 pgs: 148 active+clean; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 909 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-09-11T08:10:51.515 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:51.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:51.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:51.617 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:51.652 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:51.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:51.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:51.754 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:10:51.800 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:51.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:51.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:10:51.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:51 smithi181 ceph-mon[32701]: osdmap e115: 8 total, 8 up, 8 in 2024-09-11T08:10:51.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:51 smithi181 ceph-mon[32701]: pgmap v127: 148 pgs: 148 active+clean; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 909 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-09-11T08:10:51.902 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:51.939 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:51.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:51.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:51.994 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T08:10:52.052 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:10:52.079 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-11T08:10:52.085 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 54 16 88541 88525 102.43 66.8125 0.0359091 0.00975631 2024-09-11T08:10:52.116 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:10:52.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:52.288 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:52.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:10:52.389 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:52.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:52.442 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:52.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:52.456 INFO:teuthology.orchestra.run.smithi150.stderr:reweighted osd.3 to 1 (10000) 2024-09-11T08:10:52.544 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:10:52.569 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:52.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:52.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:10:52.671 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:10:52.696 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:52.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:52.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:10:52.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:52 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/674526460' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-11T08:10:52.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:52 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-11T08:10:52.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:52 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:10:52.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:52 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:10:52.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:52 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/674526460' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-11T08:10:52.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:52 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-11T08:10:52.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:52 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:10:52.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:52 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:10:52.797 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:10:52.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:52 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/674526460' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-11T08:10:52.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:52 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-11T08:10:52.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:52 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:10:52.825 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:52 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:10:52.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:52.825 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:52.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:10:52.927 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:10:52.967 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:52.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:52.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:10:53.069 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:10:53.086 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 55 16 89420 89404 101.566 54.9375 0.0130209 0.00984083 2024-09-11T08:10:53.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:53.111 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:53.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:10:53.212 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:10:53.225 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd reweight 4 1 2024-09-11T08:10:53.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:53.256 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:53.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:10:53.357 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:10:53.380 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:53.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:53.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:10:53.482 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:10:53.506 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:53.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:53.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:10:53.608 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:53.655 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:53.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:53.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:53.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:53 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-11T08:10:53.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:53 smithi150 ceph-mon[29276]: osdmap e116: 8 total, 8 up, 8 in 2024-09-11T08:10:53.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:53 smithi150 ceph-mon[29276]: pgmap v129: 148 pgs: 148 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 85 MiB/s wr, 1.35k op/s 2024-09-11T08:10:53.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:53 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-11T08:10:53.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:53 smithi150 ceph-mon[39612]: osdmap e116: 8 total, 8 up, 8 in 2024-09-11T08:10:53.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:53 smithi150 ceph-mon[39612]: pgmap v129: 148 pgs: 148 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 85 MiB/s wr, 1.35k op/s 2024-09-11T08:10:53.757 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:10:53.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:53.793 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:53.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:10:53.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:53 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-11T08:10:53.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:53 smithi181 ceph-mon[32701]: osdmap e116: 8 total, 8 up, 8 in 2024-09-11T08:10:53.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:53 smithi181 ceph-mon[32701]: pgmap v129: 148 pgs: 148 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 85 MiB/s wr, 1.35k op/s 2024-09-11T08:10:53.894 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:53.919 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:53.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:53.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:54.021 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:10:54.045 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:54.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:54.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:10:54.085 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 56 16 90290 90274 100.723 54.375 0.00707177 0.00992343 2024-09-11T08:10:54.148 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:10:54.174 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:54.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:54.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:10:54.276 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:10:54.318 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:54.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:54.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:10:54.419 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:10:54.463 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:54.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:54.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:10:54.565 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:54.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:54.651 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:54.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:54.656 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:10:54.752 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:54.805 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:54.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:54.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:54.907 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:10:54.954 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:54.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:54.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:10:55.056 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:10:55.085 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 57 16 91356 91340 100.125 66.625 0.0168917 0.00998274 2024-09-11T08:10:55.093 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:55.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:55.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:10:55.196 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:10:55.244 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:55.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:55.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:10:55.346 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:10:55.365 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:10:55.366 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:10:55.386 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:55.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:55.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:10:55.399 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:10:55.400 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:10:55.441 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:10:55.442 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:10:55.442 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:10:55.442 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:10:55.477 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:10:55.478 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:10:55.488 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:55.511 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:10:55.511 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:10:55.533 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:55.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:55.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:55.544 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:10:55.545 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:10:55.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:10:55 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:10:55] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:10:55.578 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:10:55.578 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:10:55.613 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:10:55.614 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:10:55.635 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:10:55.675 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:55.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:55.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:10:55.777 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:10:55.819 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:55.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:55.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:10:55.922 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:55.956 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:55 smithi150 ceph-mon[29276]: pgmap v130: 148 pgs: 148 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.3 KiB/s rd, 91 MiB/s wr, 1.45k op/s 2024-09-11T08:10:55.956 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:55 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1791156556' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:10:55.956 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:55 smithi150 ceph-mon[29276]: osdmap e117: 8 total, 8 up, 8 in 2024-09-11T08:10:55.956 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:55 smithi150 ceph-mon[39612]: pgmap v130: 148 pgs: 148 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.3 KiB/s rd, 91 MiB/s wr, 1.45k op/s 2024-09-11T08:10:55.956 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:55 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1791156556' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:10:55.956 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:55 smithi150 ceph-mon[39612]: osdmap e117: 8 total, 8 up, 8 in 2024-09-11T08:10:55.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:55.958 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:55.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:56.059 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:10:56.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:55 smithi181 ceph-mon[32701]: pgmap v130: 148 pgs: 148 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.3 KiB/s rd, 91 MiB/s wr, 1.45k op/s 2024-09-11T08:10:56.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:55 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1791156556' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:10:56.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:55 smithi181 ceph-mon[32701]: osdmap e117: 8 total, 8 up, 8 in 2024-09-11T08:10:56.086 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 58 16 92417 92401 99.5396 66.3125 0.0284819 0.0100413 2024-09-11T08:10:56.097 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:56.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:56.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:10:56.199 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:56.236 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:56.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:56.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:56.338 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:56.366 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:56.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:56.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:56.467 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:10:56.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:56.512 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:56.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:10:56.613 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:10:56.640 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:56.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:56.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:10:56.742 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:56.768 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:56.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:56.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:56.870 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:56.898 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:56.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:56.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:57.001 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:57.027 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:57.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:57.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:57.086 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 59 16 93563 93547 99.0665 71.625 0.0224953 0.0100888 2024-09-11T08:10:57.129 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:57.155 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:57.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:57.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:57.257 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:10:57.299 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:57.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:57.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:10:57.401 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:10:57.447 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:57.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:57.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:10:57.549 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:10:57.589 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:57.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:57.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:10:57.691 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:57.739 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:57.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:57.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:57.840 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:57.888 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:57.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:57.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:57.990 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:10:58.031 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:58.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:58.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:10:58.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:57 smithi181 ceph-mon[32701]: pgmap v132: 148 pgs: 148 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 807 B/s rd, 51 MiB/s wr, 823 op/s 2024-09-11T08:10:58.074 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:10:57 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:10:58.087 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:2024-09-11T08:10:58.086965+0000 min lat: 0.00117796 max lat: 0.181517 avg lat: 0.0101364 2024-09-11T08:10:58.087 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-11T08:10:58.087 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 60 16 94712 94696 98.6123 71.8125 0.00800418 0.0101364 2024-09-11T08:10:58.133 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:10:58.160 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:58.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:58.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:10:58.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:57 smithi150 ceph-mon[29276]: pgmap v132: 148 pgs: 148 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 807 B/s rd, 51 MiB/s wr, 823 op/s 2024-09-11T08:10:58.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:57 smithi150 ceph-mon[39612]: pgmap v132: 148 pgs: 148 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 807 B/s rd, 51 MiB/s wr, 823 op/s 2024-09-11T08:10:58.262 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:10:58.288 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:58.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:58.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:10:58.390 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:10:58.437 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:58.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:58.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:10:58.539 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:58.563 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:58.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:58.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:58.665 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:10:58.680 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:10:58 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:10:58] "GET /metrics HTTP/1.1" 200 223617 "" "Prometheus/2.43.0" 2024-09-11T08:10:58.707 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:58.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:58.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:10:58.809 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:58.837 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:58.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:58.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:58.923 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T08:10:58.939 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:10:58.965 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:58 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:10:58.966 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:58 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:10:59.006 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:59.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:59.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:10:59.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:58 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:10:59.086 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 61 16 95838 95822 98.1493 70.375 0.0325702 0.0101834 2024-09-11T08:10:59.112 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:10:59.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:59.183 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:59.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:10:59.285 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:10:59.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:59.330 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:59.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:10:59.431 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:10:59.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:59.458 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:59.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:10:59.560 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:59.590 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:59.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:59.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:59.692 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:10:59.715 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:59.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:59.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:10:59.817 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:10:59.845 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:10:59.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:59.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:10:59.946 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:10:59.986 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:10:59.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:59.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:00.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:59 smithi181 ceph-mon[32701]: pgmap v133: 148 pgs: 148 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-09-11T08:11:00.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:59 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:11:00.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:59 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:11:00.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:59 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:11:00.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:10:59 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:11:00.087 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 62 16 96833 96817 97.5693 62.1875 0.00777469 0.0102437 2024-09-11T08:11:00.087 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:59 smithi150 ceph-mon[29276]: pgmap v133: 148 pgs: 148 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-09-11T08:11:00.087 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:59 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:11:00.087 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:59 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:11:00.087 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:59 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:11:00.087 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:10:59 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:11:00.088 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:59 smithi150 ceph-mon[39612]: pgmap v133: 148 pgs: 148 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-09-11T08:11:00.088 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:59 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:11:00.088 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:59 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:11:00.088 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:59 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:11:00.088 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:10:59 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:11:00.088 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:00.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:00.130 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:00.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:00.231 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:11:00.254 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:00.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:00.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:11:00.356 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:11:00.381 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:00.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:00.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:11:00.426 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-09-11T08:11:00.426 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:11:00.484 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:11:00.512 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:00.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:00.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:11:00.613 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:11:00.640 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:00.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:00.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:11:00.742 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:11:00.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:00.783 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:00.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:11:00.884 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:00.923 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:00.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:00.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:01.024 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:11:01.051 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:01.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:01.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:11:01.087 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 63 15 97863 97848 97.0434 64.4375 0.0125346 0.0103 2024-09-11T08:11:01.100 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:00 smithi150 ceph-mon[29276]: osdmap e118: 8 total, 8 up, 8 in 2024-09-11T08:11:01.100 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:00 smithi150 ceph-mon[29276]: osdmap e119: 8 total, 8 up, 8 in 2024-09-11T08:11:01.100 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:00 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/606935537' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:11:01.100 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:00 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:11:01.100 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:00 smithi150 ceph-mon[39612]: osdmap e118: 8 total, 8 up, 8 in 2024-09-11T08:11:01.100 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:00 smithi150 ceph-mon[39612]: osdmap e119: 8 total, 8 up, 8 in 2024-09-11T08:11:01.101 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:00 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/606935537' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:11:01.101 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:00 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:11:01.147 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-11T08:11:01.154 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:01.243 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:01.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:01.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:01.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:00 smithi181 ceph-mon[32701]: osdmap e118: 8 total, 8 up, 8 in 2024-09-11T08:11:01.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:00 smithi181 ceph-mon[32701]: osdmap e119: 8 total, 8 up, 8 in 2024-09-11T08:11:01.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:00 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/606935537' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:11:01.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:00 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:11:01.345 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:11:01.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:01.394 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:01.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:11:01.431 INFO:teuthology.orchestra.run.smithi150.stderr:nodeep-scrub is set 2024-09-11T08:11:01.495 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:11:01.522 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:01.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:01.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:11:01.623 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:01.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:01.661 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:01.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:01.762 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:01.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:01.806 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:01.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:01.907 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:11:01.953 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:01.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:01.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:11:01.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:01 smithi150 ceph-mon[29276]: pgmap v136: 148 pgs: 148 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 55 MiB/s wr, 874 op/s 2024-09-11T08:11:01.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:01 smithi150 ceph-mon[29276]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:11:01.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:01 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:11:01.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:01 smithi150 ceph-mon[29276]: osdmap e120: 8 total, 8 up, 8 in 2024-09-11T08:11:01.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:01 smithi150 ceph-mon[39612]: pgmap v136: 148 pgs: 148 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 55 MiB/s wr, 874 op/s 2024-09-11T08:11:01.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:01 smithi150 ceph-mon[39612]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:11:01.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:01 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:11:01.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:01 smithi150 ceph-mon[39612]: osdmap e120: 8 total, 8 up, 8 in 2024-09-11T08:11:02.055 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:02.087 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 64 16 98889 98873 96.5281 64.0625 0.00771847 0.0103548 2024-09-11T08:11:02.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:02.098 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:02.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:02.126 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:11:02.200 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:02.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:02.248 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:02.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:02.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:01 smithi181 ceph-mon[32701]: pgmap v136: 148 pgs: 148 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 55 MiB/s wr, 874 op/s 2024-09-11T08:11:02.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:01 smithi181 ceph-mon[32701]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:11:02.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:01 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:11:02.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:01 smithi181 ceph-mon[32701]: osdmap e120: 8 total, 8 up, 8 in 2024-09-11T08:11:02.349 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:11:02.377 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:02.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:02.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:11:02.479 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:02.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:02.506 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:02.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:02.606 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:02.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:02.633 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:02.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:02.734 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:02.764 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:02.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:02.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:02.865 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:11:02.890 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:02.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:02.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:11:02.991 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:11:03.017 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:03.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:03.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:11:03.087 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 65 16 99993 99977 96.1046 69 0.00454508 0.0104002 2024-09-11T08:11:03.118 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:03.152 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:03.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:03.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:03.254 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:03.259 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:11:03.318 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:03.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:03.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:03.420 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:11:03.446 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:03.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:03.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:11:03.548 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:11:03.582 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:03.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:03.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:11:03.683 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:03.719 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:03.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:03.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:03.820 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:11:03.853 INFO:teuthology.orchestra.run.smithi150.stderr:reweighted osd.4 to 1 (10000) 2024-09-11T08:11:03.862 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:03.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:03.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:11:03.964 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:11:03.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:03 smithi150 ceph-mon[29276]: pgmap v138: 148 pgs: 148 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.7 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-09-11T08:11:03.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:03 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2763413155' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-11T08:11:03.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:03 smithi150 ceph-mon[39612]: pgmap v138: 148 pgs: 148 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.7 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-09-11T08:11:03.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:03 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2763413155' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-11T08:11:04.005 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:04.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:04.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:11:04.087 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 66 16 101012 100996 95.6134 63.6875 0.0143415 0.0104535 2024-09-11T08:11:04.106 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:04.143 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:04.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:04.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:04.245 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:04.287 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:04.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:04.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:04.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:03 smithi181 ceph-mon[32701]: pgmap v138: 148 pgs: 148 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.7 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-09-11T08:11:04.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:03 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2763413155' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-11T08:11:04.390 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:11:04.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:04.430 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:04.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:11:04.531 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:04.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:04.576 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:04.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:04.598 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T08:11:04.677 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:04.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:04.702 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:04.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:04.803 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:04.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:04.851 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:04.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:04.952 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:11:05.001 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:05.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:05.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:11:05.087 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 67 16 101968 101952 95.0782 59.75 0.00411322 0.0105127 2024-09-11T08:11:05.103 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:05.143 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:04 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2763413155' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-11T08:11:05.143 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:04 smithi181 ceph-mon[32701]: osdmap e121: 8 total, 8 up, 8 in 2024-09-11T08:11:05.146 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:05.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:05.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:05.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:04 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2763413155' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-11T08:11:05.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:04 smithi150 ceph-mon[29276]: osdmap e121: 8 total, 8 up, 8 in 2024-09-11T08:11:05.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:04 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2763413155' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-11T08:11:05.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:04 smithi150 ceph-mon[39612]: osdmap e121: 8 total, 8 up, 8 in 2024-09-11T08:11:05.248 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:11:05.274 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:05.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:05.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:11:05.376 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:05.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:05.410 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:05.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:05.511 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:11:05.537 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:05.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:05.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:11:05.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:11:05 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:11:05] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:11:05.606 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd reweight 5 1 2024-09-11T08:11:05.640 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:05.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:05.666 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:05.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:05.767 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:11:05.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:05.798 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:05.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:11:05.899 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:05.924 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:05.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:05.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:06.026 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:06.072 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:06.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:06.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:06.087 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 68 16 103039 103023 94.6643 66.9375 0.0157705 0.0105589 2024-09-11T08:11:06.156 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:05 smithi150 ceph-mon[29276]: pgmap v140: 148 pgs: 148 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 68 MiB/s wr, 1.08k op/s 2024-09-11T08:11:06.156 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:05 smithi150 ceph-mon[39612]: pgmap v140: 148 pgs: 148 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 68 MiB/s wr, 1.08k op/s 2024-09-11T08:11:06.174 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:11:06.199 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:06.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:06.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:11:06.300 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:11:06.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:05 smithi181 ceph-mon[32701]: pgmap v140: 148 pgs: 148 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 68 MiB/s wr, 1.08k op/s 2024-09-11T08:11:06.327 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:06.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:06.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:11:06.429 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:11:06.456 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:06.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:06.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:11:06.558 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:06.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:06.598 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:06.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:06.699 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:06.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:06.756 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:06.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:06.857 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:11:06.896 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:06.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:06.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:11:06.997 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:11:07.038 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:07.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:07.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:11:07.088 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 69 16 104132 104116 94.281 68.3125 0.0113318 0.0106015 2024-09-11T08:11:07.140 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:11:07.164 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:07.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:07.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:11:07.172 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:11:07.182 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:11:07.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:06 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:11:07.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:06 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:11:07.266 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:07.312 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:07.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:07.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:07.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:06 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:11:07.415 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:07.452 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:07.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:07.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:07.554 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:11:07.575 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:07.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:07.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:11:07.677 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:11:07.707 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:07.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:07.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:11:07.809 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:07.846 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:07.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:07.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:07.948 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:11:07.958 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:07 smithi150 ceph-mon[29276]: pgmap v141: 148 pgs: 148 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 808 B/s rd, 52 MiB/s wr, 839 op/s 2024-09-11T08:11:07.958 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:07 smithi150 ceph-mon[39612]: pgmap v141: 148 pgs: 148 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 808 B/s rd, 52 MiB/s wr, 839 op/s 2024-09-11T08:11:07.975 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:07.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:07.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:11:08.078 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:08.088 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 70 16 105051 105035 93.7547 57.4375 0.0201121 0.0106608 2024-09-11T08:11:08.105 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:08.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:08.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:08.207 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:11:08.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:08.234 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:08.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:11:08.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:07 smithi181 ceph-mon[32701]: pgmap v141: 148 pgs: 148 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 808 B/s rd, 52 MiB/s wr, 839 op/s 2024-09-11T08:11:08.324 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:11:07 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:11:08.335 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:08.369 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:08.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:08.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:08.471 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:08.508 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:08.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:08.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:08.610 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:11:08.639 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:08.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:08.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:11:08.741 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:08.777 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:08.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:08.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:08.793 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:11:08.793 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:11:08.793 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:11:08.794 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:11:08.794 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:11:08.794 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:11:08.794 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:11:08.794 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:11:08.794 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:11:08.794 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:11:08.794 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:11:08.794 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:11:08.794 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:11:08.795 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:11:08.795 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:11:08.795 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:11:08.795 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:11:08 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:11:08] "GET /metrics HTTP/1.1" 200 223649 "" "Prometheus/2.43.0" 2024-09-11T08:11:08.879 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:11:08.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:08.907 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:08.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:11:09.009 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:11:09.035 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:09.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:09.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:11:09.088 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 71 16 106116 106100 93.3717 66.5625 0.00917202 0.0107044 2024-09-11T08:11:09.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:08 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:11:09.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:08 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:11:09.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:08 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:11:09.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:08 smithi150 ceph-mon[29276]: osdmap e122: 8 total, 8 up, 8 in 2024-09-11T08:11:09.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:08 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1799417795' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:11:09.093 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:08 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:11:09.093 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:08 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:11:09.093 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:08 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:11:09.093 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:08 smithi150 ceph-mon[39612]: osdmap e122: 8 total, 8 up, 8 in 2024-09-11T08:11:09.093 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:08 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1799417795' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:11:09.137 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:11:09.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:09.181 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:09.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:11:09.282 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:11:09.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:08 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:11:09.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:08 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:11:09.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:08 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:11:09.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:08 smithi181 ceph-mon[32701]: osdmap e122: 8 total, 8 up, 8 in 2024-09-11T08:11:09.326 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:08 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1799417795' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:11:09.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:09.327 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:09.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:11:09.428 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:09.454 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:09.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:09.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:09.556 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:11:09.584 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:09.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:09.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:11:09.686 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:09.727 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:09.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:09.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:09.829 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:09.878 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:09.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:09.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:09.977 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:09 smithi150 ceph-mon[29276]: pgmap v143: 148 pgs: 148 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 89 MiB/s wr, 1.42k op/s 2024-09-11T08:11:09.977 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:09 smithi150 ceph-mon[39612]: pgmap v143: 148 pgs: 148 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 89 MiB/s wr, 1.42k op/s 2024-09-11T08:11:09.984 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:10.010 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:10.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:10.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:10.089 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 72 15 107209 107194 93.0245 68.375 0.0190525 0.0107453 2024-09-11T08:11:10.112 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:10.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:10.150 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:10.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:10.251 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:11:10.294 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:10.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:10.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:11:10.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:09 smithi181 ceph-mon[32701]: pgmap v143: 148 pgs: 148 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 89 MiB/s wr, 1.42k op/s 2024-09-11T08:11:10.396 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:11:10.440 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:10.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:10.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:11:10.541 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:10.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:10.585 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:10.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:10.686 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:10.708 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:10.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:10.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:10.809 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:10.849 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:10.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:10.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:10.951 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:11:10.972 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:10.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:10.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:11:11.074 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:11.088 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 73 16 108065 108049 92.4822 53.4375 0.00903133 0.0108081 2024-09-11T08:11:11.115 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:11.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:11.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:11.217 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:11:11.245 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:11.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:11.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:11:11.346 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:11:11.380 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:11.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:11.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:11:11.482 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:11:11.519 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:11.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:11.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:11:11.620 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:11:11.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:11.650 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:11.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:11:11.750 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:11:11.776 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:11.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:11.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:11:11.878 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:11:11.920 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:11.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:11.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:11:12.022 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:11:12.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:12.054 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:12.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:11:12.089 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 74 16 108956 108940 91.985 55.6875 0.0132121 0.010865 2024-09-11T08:11:12.155 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:11:12.182 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:12.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:12.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:11:12.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:11 smithi150 ceph-mon[29276]: pgmap v144: 148 pgs: 148 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-09-11T08:11:12.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:11 smithi150 ceph-mon[39612]: pgmap v144: 148 pgs: 148 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-09-11T08:11:12.284 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:12.309 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:12.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:12.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:11 smithi181 ceph-mon[32701]: pgmap v144: 148 pgs: 148 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-09-11T08:11:12.411 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:11:12.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:12.450 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:12.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:11:12.551 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:12.589 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:12.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:12.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:12.691 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:11:12.719 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:12.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:12.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:11:12.821 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:11:12.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:12.858 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:12.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:11:12.959 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:12.986 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:12.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:12.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:13.087 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:11:13.089 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 75 16 110069 110053 91.6861 69.5625 0.00622889 0.0109023 2024-09-11T08:11:13.121 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:13.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:13.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:11:13.223 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:11:13.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:13.282 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:13.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:11:13.383 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:11:13.411 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:13.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:13.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:11:13.513 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:13.554 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:13.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:13.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:13.656 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:11:13.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:13.683 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:13.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:11:13.744 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-11T08:11:13.744 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [], 2024-09-11T08:11:13.744 INFO:teuthology.orchestra.run.smithi150.stdout: "complaint_time": 30, 2024-09-11T08:11:13.744 INFO:teuthology.orchestra.run.smithi150.stdout: "num_blocked_ops": 0 2024-09-11T08:11:13.745 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-11T08:11:13.784 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:13.820 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:13.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:13.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:13.923 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:11:13.949 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:13.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:13.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:11:14.050 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:14.089 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 76 16 111122 111106 91.3456 65.8125 0.0105431 0.0109426 2024-09-11T08:11:14.107 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:14.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:14.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:14.209 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:11:14.236 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:14.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:14.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:11:14.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:13 smithi150 ceph-mon[29276]: pgmap v145: 148 pgs: 148 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 51 MiB/s wr, 814 op/s 2024-09-11T08:11:14.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:13 smithi150 ceph-mon[39612]: pgmap v145: 148 pgs: 148 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 51 MiB/s wr, 814 op/s 2024-09-11T08:11:14.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:13 smithi181 ceph-mon[32701]: pgmap v145: 148 pgs: 148 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 51 MiB/s wr, 814 op/s 2024-09-11T08:11:14.338 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:11:14.366 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:14.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:14.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:11:14.468 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:14.508 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:14.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:14.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:14.610 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:14.636 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:14.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:14.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:14.738 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:11:14.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:14.768 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:14.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:11:14.869 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:11:14.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:14.906 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:14.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:11:14.999 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T08:11:15.008 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:11:15.033 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:15.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:15.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:11:15.090 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 77 16 112179 112163 91.0168 66.0625 0.00837418 0.0109815 2024-09-11T08:11:15.135 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:15.183 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:15.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:15.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:15.285 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:15.327 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:15.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:15.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:15.429 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:15.456 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:15.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:15.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:15.558 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:11:15.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:11:15 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:11:15] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:11:15.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:15.596 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:15.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:11:15.697 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:11:15.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:15.734 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:15.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:11:15.835 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:15.873 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:15.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:15.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:15.945 INFO:teuthology.orchestra.run.smithi150.stderr:noscrub is unset 2024-09-11T08:11:15.957 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-11T08:11:15.975 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:16.000 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:16.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:16.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:16.089 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 78 16 113118 113102 90.6023 58.6875 0.00594995 0.0110318 2024-09-11T08:11:16.102 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:16.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:16.193 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:16.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:16.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:15 smithi150 ceph-mon[29276]: pgmap v146: 148 pgs: 148 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-09-11T08:11:16.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:15 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3577958782' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:11:16.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:15 smithi150 ceph-mon[39612]: pgmap v146: 148 pgs: 148 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-09-11T08:11:16.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:15 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3577958782' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:11:16.293 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:16.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:15 smithi181 ceph-mon[32701]: pgmap v146: 148 pgs: 148 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-09-11T08:11:16.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:15 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3577958782' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:11:16.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:16.354 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:16.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:16.455 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:16.499 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:16.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:16.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:16.602 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:11:16.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:16.636 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:16.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:11:16.737 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:11:16.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:16.764 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:16.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:11:16.865 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:11:16.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:16.925 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:16.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:11:16.980 INFO:teuthology.orchestra.run.smithi150.stderr:reweighted osd.5 to 1 (10000) 2024-09-11T08:11:16.986 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:11:17.026 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:11:17.075 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:17.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:17.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:11:17.091 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 79 16 114078 114062 90.2134 60 0.009036 0.0110798 2024-09-11T08:11:17.178 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:17.206 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:17.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:17.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:17.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:16 smithi150 ceph-mon[29276]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:11:17.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:16 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3577958782' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:11:17.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:16 smithi150 ceph-mon[29276]: osdmap e123: 8 total, 8 up, 8 in 2024-09-11T08:11:17.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:16 smithi150 ceph-mon[29276]: 6.1a scrub starts 2024-09-11T08:11:17.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:16 smithi150 ceph-mon[29276]: 6.1a scrub ok 2024-09-11T08:11:17.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:16 smithi150 ceph-mon[29276]: 5.17 scrub starts 2024-09-11T08:11:17.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:16 smithi150 ceph-mon[29276]: 5.17 scrub ok 2024-09-11T08:11:17.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:16 smithi150 ceph-mon[29276]: 3.a scrub starts 2024-09-11T08:11:17.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:16 smithi150 ceph-mon[29276]: 3.a scrub ok 2024-09-11T08:11:17.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:16 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/816902366' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-11T08:11:17.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:16 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-11T08:11:17.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:16 smithi150 ceph-mon[29276]: 4.4 scrub starts 2024-09-11T08:11:17.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:16 smithi150 ceph-mon[29276]: 4.4 scrub ok 2024-09-11T08:11:17.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:16 smithi150 ceph-mon[39612]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:11:17.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:16 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3577958782' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:11:17.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:16 smithi150 ceph-mon[39612]: osdmap e123: 8 total, 8 up, 8 in 2024-09-11T08:11:17.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:16 smithi150 ceph-mon[39612]: 6.1a scrub starts 2024-09-11T08:11:17.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:16 smithi150 ceph-mon[39612]: 6.1a scrub ok 2024-09-11T08:11:17.242 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:16 smithi150 ceph-mon[39612]: 5.17 scrub starts 2024-09-11T08:11:17.242 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:16 smithi150 ceph-mon[39612]: 5.17 scrub ok 2024-09-11T08:11:17.242 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:16 smithi150 ceph-mon[39612]: 3.a scrub starts 2024-09-11T08:11:17.242 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:16 smithi150 ceph-mon[39612]: 3.a scrub ok 2024-09-11T08:11:17.242 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:16 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/816902366' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-11T08:11:17.242 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:16 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-11T08:11:17.242 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:16 smithi150 ceph-mon[39612]: 4.4 scrub starts 2024-09-11T08:11:17.242 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:16 smithi150 ceph-mon[39612]: 4.4 scrub ok 2024-09-11T08:11:17.309 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:11:17.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:16 smithi181 ceph-mon[32701]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:11:17.326 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:16 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3577958782' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:11:17.326 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:16 smithi181 ceph-mon[32701]: osdmap e123: 8 total, 8 up, 8 in 2024-09-11T08:11:17.326 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:16 smithi181 ceph-mon[32701]: 6.1a scrub starts 2024-09-11T08:11:17.326 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:16 smithi181 ceph-mon[32701]: 6.1a scrub ok 2024-09-11T08:11:17.326 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:16 smithi181 ceph-mon[32701]: 5.17 scrub starts 2024-09-11T08:11:17.326 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:16 smithi181 ceph-mon[32701]: 5.17 scrub ok 2024-09-11T08:11:17.326 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:16 smithi181 ceph-mon[32701]: 3.a scrub starts 2024-09-11T08:11:17.326 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:16 smithi181 ceph-mon[32701]: 3.a scrub ok 2024-09-11T08:11:17.326 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:16 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/816902366' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-11T08:11:17.326 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:16 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-11T08:11:17.326 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:16 smithi181 ceph-mon[32701]: 4.4 scrub starts 2024-09-11T08:11:17.326 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:16 smithi181 ceph-mon[32701]: 4.4 scrub ok 2024-09-11T08:11:17.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:17.338 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:17.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:11:17.439 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:11:17.467 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:17.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:17.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:11:17.569 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:11:17.595 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:17.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:17.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:11:17.697 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:17.740 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:17.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:17.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:17.842 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:11:17.865 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:17.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:17.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:11:17.967 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:17.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:17.997 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:17.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:18.091 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:2024-09-11T08:11:18.091562+0000 min lat: 0.00117796 max lat: 0.181517 avg lat: 0.0111162 2024-09-11T08:11:18.091 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-11T08:11:18.091 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 80 16 115149 115133 89.9224 66.9375 0.0214641 0.0111162 2024-09-11T08:11:18.098 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:11:18.126 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:18.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:18.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:11:18.181 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:17 smithi150 ceph-mon[29276]: pgmap v148: 148 pgs: 148 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 579 B/s rd, 35 MiB/s wr, 562 op/s 2024-09-11T08:11:18.181 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:17 smithi150 ceph-mon[29276]: 6.a scrub starts 2024-09-11T08:11:18.181 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:17 smithi150 ceph-mon[29276]: 6.a scrub ok 2024-09-11T08:11:18.181 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:17 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-11T08:11:18.182 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:17 smithi150 ceph-mon[29276]: osdmap e124: 8 total, 8 up, 8 in 2024-09-11T08:11:18.182 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:17 smithi150 ceph-mon[29276]: 3.1a scrub starts 2024-09-11T08:11:18.182 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:17 smithi150 ceph-mon[29276]: 3.1a scrub ok 2024-09-11T08:11:18.182 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:17 smithi150 ceph-mon[39612]: pgmap v148: 148 pgs: 148 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 579 B/s rd, 35 MiB/s wr, 562 op/s 2024-09-11T08:11:18.182 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:17 smithi150 ceph-mon[39612]: 6.a scrub starts 2024-09-11T08:11:18.182 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:17 smithi150 ceph-mon[39612]: 6.a scrub ok 2024-09-11T08:11:18.182 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:17 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-11T08:11:18.182 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:17 smithi150 ceph-mon[39612]: osdmap e124: 8 total, 8 up, 8 in 2024-09-11T08:11:18.182 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:17 smithi150 ceph-mon[39612]: 3.1a scrub starts 2024-09-11T08:11:18.182 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:17 smithi150 ceph-mon[39612]: 3.1a scrub ok 2024-09-11T08:11:18.225 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd reweight 6 1 2024-09-11T08:11:18.226 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:17 smithi181 ceph-mon[32701]: pgmap v148: 148 pgs: 148 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 579 B/s rd, 35 MiB/s wr, 562 op/s 2024-09-11T08:11:18.226 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:17 smithi181 ceph-mon[32701]: 6.a scrub starts 2024-09-11T08:11:18.227 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:17 smithi181 ceph-mon[32701]: 6.a scrub ok 2024-09-11T08:11:18.227 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:17 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-11T08:11:18.227 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:17 smithi181 ceph-mon[32701]: osdmap e124: 8 total, 8 up, 8 in 2024-09-11T08:11:18.227 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:17 smithi181 ceph-mon[32701]: 3.1a scrub starts 2024-09-11T08:11:18.227 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:17 smithi181 ceph-mon[32701]: 3.1a scrub ok 2024-09-11T08:11:18.228 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:11:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:11:18.228 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:11:18.281 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:18.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:18.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:11:18.383 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:11:18.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:18.424 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:18.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:11:18.525 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:18.549 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:18.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:18.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:18.651 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:11:18.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:18.695 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:18.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:11:18.796 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:18.813 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:11:18.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:18.849 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:18.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:18.950 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:11:18.955 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T08:11:18.957 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:11:18 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:11:18] "GET /metrics HTTP/1.1" 200 223649 "" "Prometheus/2.43.0" 2024-09-11T08:11:18.976 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:18.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:18.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:11:19.078 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:19.091 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 81 16 116130 116114 89.569 61.3125 0.00927107 0.0111597 2024-09-11T08:11:19.105 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:19.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:19.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:19.209 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:11:19.234 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:19.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:19.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:11:19.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:18 smithi150 ceph-mon[29276]: 1.0 scrub starts 2024-09-11T08:11:19.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:18 smithi150 ceph-mon[29276]: 1.0 scrub ok 2024-09-11T08:11:19.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:18 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:11:19.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:18 smithi150 ceph-mon[29276]: 7.e scrub starts 2024-09-11T08:11:19.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:18 smithi150 ceph-mon[39612]: 1.0 scrub starts 2024-09-11T08:11:19.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:18 smithi150 ceph-mon[39612]: 1.0 scrub ok 2024-09-11T08:11:19.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:18 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:11:19.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:18 smithi150 ceph-mon[39612]: 7.e scrub starts 2024-09-11T08:11:19.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:18 smithi181 ceph-mon[32701]: 1.0 scrub starts 2024-09-11T08:11:19.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:18 smithi181 ceph-mon[32701]: 1.0 scrub ok 2024-09-11T08:11:19.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:18 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:11:19.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:18 smithi181 ceph-mon[32701]: 7.e scrub starts 2024-09-11T08:11:19.336 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:11:19.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:19.386 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:19.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:11:19.487 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:11:19.515 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:19.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:19.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:11:19.620 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:11:19.646 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:19.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:19.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:11:19.748 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:11:19.803 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:19.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:19.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:11:19.905 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:19.946 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:19.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:19.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:20.048 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:20.076 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:20.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:20.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:20.094 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 82 16 117112 117096 89.2252 61.375 0.0136528 0.0112027 2024-09-11T08:11:20.178 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:11:20.210 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:20.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:20.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:11:20.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:19 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:11:20.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:19 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:11:20.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:19 smithi150 ceph-mon[29276]: pgmap v150: 148 pgs: 148 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-09-11T08:11:20.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:19 smithi150 ceph-mon[29276]: 6.9 scrub starts 2024-09-11T08:11:20.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:19 smithi150 ceph-mon[29276]: 6.9 scrub ok 2024-09-11T08:11:20.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:19 smithi150 ceph-mon[29276]: osdmap e125: 8 total, 8 up, 8 in 2024-09-11T08:11:20.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:19 smithi150 ceph-mon[29276]: 7.8 scrub starts 2024-09-11T08:11:20.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:19 smithi150 ceph-mon[29276]: 4.19 scrub starts 2024-09-11T08:11:20.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:19 smithi150 ceph-mon[29276]: 4.19 scrub ok 2024-09-11T08:11:20.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:19 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:11:20.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:19 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:11:20.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:19 smithi150 ceph-mon[39612]: pgmap v150: 148 pgs: 148 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-09-11T08:11:20.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:19 smithi150 ceph-mon[39612]: 6.9 scrub starts 2024-09-11T08:11:20.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:19 smithi150 ceph-mon[39612]: 6.9 scrub ok 2024-09-11T08:11:20.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:19 smithi150 ceph-mon[39612]: osdmap e125: 8 total, 8 up, 8 in 2024-09-11T08:11:20.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:19 smithi150 ceph-mon[39612]: 7.8 scrub starts 2024-09-11T08:11:20.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:19 smithi150 ceph-mon[39612]: 4.19 scrub starts 2024-09-11T08:11:20.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:19 smithi150 ceph-mon[39612]: 4.19 scrub ok 2024-09-11T08:11:20.312 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:20.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:19 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:11:20.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:19 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:11:20.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:19 smithi181 ceph-mon[32701]: pgmap v150: 148 pgs: 148 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-09-11T08:11:20.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:19 smithi181 ceph-mon[32701]: 6.9 scrub starts 2024-09-11T08:11:20.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:19 smithi181 ceph-mon[32701]: 6.9 scrub ok 2024-09-11T08:11:20.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:19 smithi181 ceph-mon[32701]: osdmap e125: 8 total, 8 up, 8 in 2024-09-11T08:11:20.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:19 smithi181 ceph-mon[32701]: 7.8 scrub starts 2024-09-11T08:11:20.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:19 smithi181 ceph-mon[32701]: 4.19 scrub starts 2024-09-11T08:11:20.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:19 smithi181 ceph-mon[32701]: 4.19 scrub ok 2024-09-11T08:11:20.357 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:20.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:20.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:20.459 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:20.488 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:20.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:20.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:20.589 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:20.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:20.668 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:20.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:20.769 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:20.798 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:20.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:20.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:20.900 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:11:20.926 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:20.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:20.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:11:21.025 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:20 smithi150 ceph-mon[29276]: 7.0 scrub starts 2024-09-11T08:11:21.025 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:20 smithi150 ceph-mon[29276]: 4.3 scrub starts 2024-09-11T08:11:21.025 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:20 smithi150 ceph-mon[29276]: 4.3 scrub ok 2024-09-11T08:11:21.026 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:20 smithi150 ceph-mon[39612]: 7.0 scrub starts 2024-09-11T08:11:21.026 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:20 smithi150 ceph-mon[39612]: 4.3 scrub starts 2024-09-11T08:11:21.026 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:20 smithi150 ceph-mon[39612]: 4.3 scrub ok 2024-09-11T08:11:21.027 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:21.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:21.076 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:21.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:21.091 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 83 16 118001 117985 88.8196 55.5625 0.00641621 0.0112535 2024-09-11T08:11:21.177 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:21.233 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:21.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:21.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:21.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:20 smithi181 ceph-mon[32701]: 7.0 scrub starts 2024-09-11T08:11:21.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:20 smithi181 ceph-mon[32701]: 4.3 scrub starts 2024-09-11T08:11:21.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:20 smithi181 ceph-mon[32701]: 4.3 scrub ok 2024-09-11T08:11:21.335 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:11:21.363 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:21.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:21.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:11:21.468 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:21.490 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:11:21.493 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:11:21.514 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:21.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:21.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:21.555 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:11:21.556 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:11:21.600 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:11:21.600 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:11:21.616 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:21.656 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:11:21.656 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:11:21.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:21.674 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:21.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:21.716 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:11:21.717 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:11:21.752 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:11:21.753 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:11:21.775 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:21.786 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-09-11T08:11:21.787 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:11:21.796 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:11:21.796 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:11:21.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:21.819 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:21.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:21.830 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:11:21.830 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:11:21.870 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:11:21.871 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:11:21.921 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:21.966 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:21.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:21.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:22.068 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:11:22.092 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 84 16 118808 118792 88.3627 50.4375 0.014015 0.0113122 2024-09-11T08:11:22.097 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:22.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:22.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:11:22.199 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:11:22.222 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:21 smithi150 ceph-mon[29276]: 5.1a scrub starts 2024-09-11T08:11:22.222 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:21 smithi150 ceph-mon[29276]: 5.1a scrub ok 2024-09-11T08:11:22.222 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:21 smithi150 ceph-mon[29276]: pgmap v152: 148 pgs: 148 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 53 MiB/s wr, 856 op/s 2024-09-11T08:11:22.223 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:21 smithi150 ceph-mon[29276]: 4.17 scrub starts 2024-09-11T08:11:22.223 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:21 smithi150 ceph-mon[29276]: 4.17 scrub ok 2024-09-11T08:11:22.223 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:21 smithi150 ceph-mon[29276]: 7.d scrub starts 2024-09-11T08:11:22.223 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:21 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2590960990' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:11:22.223 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:21 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:11:22.223 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:21 smithi150 ceph-mon[39612]: 5.1a scrub starts 2024-09-11T08:11:22.223 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:21 smithi150 ceph-mon[39612]: 5.1a scrub ok 2024-09-11T08:11:22.223 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:21 smithi150 ceph-mon[39612]: pgmap v152: 148 pgs: 148 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 53 MiB/s wr, 856 op/s 2024-09-11T08:11:22.223 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:21 smithi150 ceph-mon[39612]: 4.17 scrub starts 2024-09-11T08:11:22.223 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:21 smithi150 ceph-mon[39612]: 4.17 scrub ok 2024-09-11T08:11:22.224 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:21 smithi150 ceph-mon[39612]: 7.d scrub starts 2024-09-11T08:11:22.224 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:21 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2590960990' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:11:22.224 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:21 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:11:22.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:22.241 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:22.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:11:22.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:21 smithi181 ceph-mon[32701]: 5.1a scrub starts 2024-09-11T08:11:22.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:21 smithi181 ceph-mon[32701]: 5.1a scrub ok 2024-09-11T08:11:22.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:21 smithi181 ceph-mon[32701]: pgmap v152: 148 pgs: 148 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 53 MiB/s wr, 856 op/s 2024-09-11T08:11:22.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:21 smithi181 ceph-mon[32701]: 4.17 scrub starts 2024-09-11T08:11:22.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:21 smithi181 ceph-mon[32701]: 4.17 scrub ok 2024-09-11T08:11:22.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:21 smithi181 ceph-mon[32701]: 7.d scrub starts 2024-09-11T08:11:22.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:21 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2590960990' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:11:22.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:21 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:11:22.342 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:22.390 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:22.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:22.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:22.492 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:22.523 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:22.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:22.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:22.624 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:11:22.656 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:22.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:22.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:11:22.758 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:22.785 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:22.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:22.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:22.887 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:22.936 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:22.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:22.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:23.038 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:11:23.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:23.084 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:23.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:11:23.091 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 85 16 119834 119818 88.0776 64.125 0.018329 0.0113486 2024-09-11T08:11:23.186 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:11:23.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:23.216 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:23.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:11:23.317 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:11:23.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:23 smithi181 ceph-mon[32701]: 6.10 scrub starts 2024-09-11T08:11:23.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:23 smithi181 ceph-mon[32701]: 6.10 scrub ok 2024-09-11T08:11:23.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:23 smithi181 ceph-mon[32701]: 7.8 scrub ok 2024-09-11T08:11:23.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:23 smithi181 ceph-mon[32701]: 2.0 scrub starts 2024-09-11T08:11:23.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:23 smithi181 ceph-mon[32701]: 2.0 scrub ok 2024-09-11T08:11:23.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:23 smithi181 ceph-mon[32701]: 4.2 scrub starts 2024-09-11T08:11:23.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:23 smithi181 ceph-mon[32701]: 4.2 scrub ok 2024-09-11T08:11:23.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:23.381 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:23.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:11:23.467 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:23 smithi150 ceph-mon[29276]: 6.10 scrub starts 2024-09-11T08:11:23.524 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:23 smithi150 ceph-mon[29276]: 6.10 scrub ok 2024-09-11T08:11:23.524 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:23 smithi150 ceph-mon[29276]: 7.8 scrub ok 2024-09-11T08:11:23.524 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:23 smithi150 ceph-mon[29276]: 2.0 scrub starts 2024-09-11T08:11:23.524 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:23 smithi150 ceph-mon[29276]: 2.0 scrub ok 2024-09-11T08:11:23.524 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:23 smithi150 ceph-mon[29276]: 4.2 scrub starts 2024-09-11T08:11:23.524 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:23 smithi150 ceph-mon[29276]: 4.2 scrub ok 2024-09-11T08:11:23.525 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:23.526 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:23 smithi150 ceph-mon[39612]: 6.10 scrub starts 2024-09-11T08:11:23.526 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:23 smithi150 ceph-mon[39612]: 6.10 scrub ok 2024-09-11T08:11:23.526 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:23 smithi150 ceph-mon[39612]: 7.8 scrub ok 2024-09-11T08:11:23.526 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:23 smithi150 ceph-mon[39612]: 2.0 scrub starts 2024-09-11T08:11:23.526 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:23 smithi150 ceph-mon[39612]: 2.0 scrub ok 2024-09-11T08:11:23.526 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:23 smithi150 ceph-mon[39612]: 4.2 scrub starts 2024-09-11T08:11:23.526 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:23 smithi150 ceph-mon[39612]: 4.2 scrub ok 2024-09-11T08:11:23.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:23.564 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:23.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:23.665 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:23.686 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:23.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:23.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:23.788 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:23.832 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:23.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:23.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:23.934 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:11:23.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:23.982 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:23.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:11:24.083 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:24.092 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 86 16 120638 120622 87.6375 50.25 0.0100109 0.0114056 2024-09-11T08:11:24.118 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:24.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:24.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:24.220 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:11:24.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:24 smithi150 ceph-mon[29276]: 7.e scrub ok 2024-09-11T08:11:24.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:24 smithi150 ceph-mon[29276]: 7.0 scrub ok 2024-09-11T08:11:24.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:24 smithi150 ceph-mon[29276]: pgmap v153: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.5 KiB/s rd, 63 MiB/s wr, 1.01k op/s 2024-09-11T08:11:24.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:24 smithi150 ceph-mon[29276]: 7.7 scrub starts 2024-09-11T08:11:24.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:24 smithi150 ceph-mon[39612]: 7.e scrub ok 2024-09-11T08:11:24.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:24 smithi150 ceph-mon[39612]: 7.0 scrub ok 2024-09-11T08:11:24.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:24 smithi150 ceph-mon[39612]: pgmap v153: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.5 KiB/s rd, 63 MiB/s wr, 1.01k op/s 2024-09-11T08:11:24.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:24 smithi150 ceph-mon[39612]: 7.7 scrub starts 2024-09-11T08:11:24.258 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:24.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:24.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:11:24.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:24 smithi181 ceph-mon[32701]: 7.e scrub ok 2024-09-11T08:11:24.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:24 smithi181 ceph-mon[32701]: 7.0 scrub ok 2024-09-11T08:11:24.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:24 smithi181 ceph-mon[32701]: pgmap v153: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.5 KiB/s rd, 63 MiB/s wr, 1.01k op/s 2024-09-11T08:11:24.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:24 smithi181 ceph-mon[32701]: 7.7 scrub starts 2024-09-11T08:11:24.360 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:24.395 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:24.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:24.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:24.497 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:24.543 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:24.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:24.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:24.645 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:11:24.675 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:24.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:24.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:11:24.777 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:24.812 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:24.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:24.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:24.914 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:24.949 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:24.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:24.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:25.052 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:25.092 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 87 16 121548 121532 87.2839 56.875 0.0362769 0.0114511 2024-09-11T08:11:25.094 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:25.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:25.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:25.197 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:11:25.224 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:25.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:25.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:11:25.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:25 smithi150 ceph-mon[29276]: 5.d scrub starts 2024-09-11T08:11:25.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:25 smithi150 ceph-mon[29276]: 5.d scrub ok 2024-09-11T08:11:25.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:25 smithi150 ceph-mon[29276]: 7.d scrub ok 2024-09-11T08:11:25.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:25 smithi150 ceph-mon[39612]: 5.d scrub starts 2024-09-11T08:11:25.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:25 smithi150 ceph-mon[39612]: 5.d scrub ok 2024-09-11T08:11:25.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:25 smithi150 ceph-mon[39612]: 7.d scrub ok 2024-09-11T08:11:25.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:25 smithi181 ceph-mon[32701]: 5.d scrub starts 2024-09-11T08:11:25.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:25 smithi181 ceph-mon[32701]: 5.d scrub ok 2024-09-11T08:11:25.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:25 smithi181 ceph-mon[32701]: 7.d scrub ok 2024-09-11T08:11:25.324 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:11:25 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:11:25] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:11:25.326 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:25.366 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:25.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:25.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:25.468 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:25.510 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:25.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:25.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:25.612 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:11:25.654 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:25.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:25.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:11:25.756 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:25.796 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:25.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:25.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:25.898 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:11:25.934 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:25.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:25.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:11:26.036 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:11:26.064 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:26.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:26.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:11:26.092 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 88 16 122625 122609 87.057 67.3125 0.0143727 0.0114811 2024-09-11T08:11:26.111 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-11T08:11:26.166 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:26.215 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:26.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:26.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:26.317 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:26.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:26 smithi181 ceph-mon[32701]: pgmap v154: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-09-11T08:11:26.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:26 smithi181 ceph-mon[32701]: 5.b scrub starts 2024-09-11T08:11:26.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:26 smithi181 ceph-mon[32701]: 5.b scrub ok 2024-09-11T08:11:26.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:26 smithi181 ceph-mon[32701]: 5.0 scrub starts 2024-09-11T08:11:26.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:26 smithi181 ceph-mon[32701]: 5.0 scrub ok 2024-09-11T08:11:26.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:26 smithi181 ceph-mon[32701]: 5.12 scrub starts 2024-09-11T08:11:26.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:26 smithi181 ceph-mon[32701]: 5.12 scrub ok 2024-09-11T08:11:26.371 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:26.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:26.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:26.473 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:11:26.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:26 smithi150 ceph-mon[29276]: pgmap v154: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-09-11T08:11:26.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:26 smithi150 ceph-mon[29276]: 5.b scrub starts 2024-09-11T08:11:26.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:26 smithi150 ceph-mon[29276]: 5.b scrub ok 2024-09-11T08:11:26.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:26 smithi150 ceph-mon[29276]: 5.0 scrub starts 2024-09-11T08:11:26.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:26 smithi150 ceph-mon[29276]: 5.0 scrub ok 2024-09-11T08:11:26.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:26 smithi150 ceph-mon[29276]: 5.12 scrub starts 2024-09-11T08:11:26.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:26 smithi150 ceph-mon[29276]: 5.12 scrub ok 2024-09-11T08:11:26.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:26 smithi150 ceph-mon[39612]: pgmap v154: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-09-11T08:11:26.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:26 smithi150 ceph-mon[39612]: 5.b scrub starts 2024-09-11T08:11:26.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:26 smithi150 ceph-mon[39612]: 5.b scrub ok 2024-09-11T08:11:26.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:26 smithi150 ceph-mon[39612]: 5.0 scrub starts 2024-09-11T08:11:26.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:26 smithi150 ceph-mon[39612]: 5.0 scrub ok 2024-09-11T08:11:26.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:26 smithi150 ceph-mon[39612]: 5.12 scrub starts 2024-09-11T08:11:26.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:26 smithi150 ceph-mon[39612]: 5.12 scrub ok 2024-09-11T08:11:26.501 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:26.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:26.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:11:26.603 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:11:26.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:26.633 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:26.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:11:26.734 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:11:26.785 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:26.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:26.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:11:26.887 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:11:26.916 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:26.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:26.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:11:26.996 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T08:11:27.018 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:11:27.092 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 89 16 123583 123567 86.7515 59.875 0.0177498 0.0115215 2024-09-11T08:11:27.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:27.102 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:27.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:11:27.203 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:11:27.228 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:27.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:27.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:11:27.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:27 smithi150 ceph-mon[29276]: 5.1 scrub starts 2024-09-11T08:11:27.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:27 smithi150 ceph-mon[29276]: 5.1 scrub ok 2024-09-11T08:11:27.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:27 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:11:27.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:27 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:11:27.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:27 smithi150 ceph-mon[29276]: 3.18 scrub starts 2024-09-11T08:11:27.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:27 smithi150 ceph-mon[29276]: 3.18 scrub ok 2024-09-11T08:11:27.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:27 smithi150 ceph-mon[29276]: 7.c scrub starts 2024-09-11T08:11:27.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:27 smithi150 ceph-mon[39612]: 5.1 scrub starts 2024-09-11T08:11:27.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:27 smithi150 ceph-mon[39612]: 5.1 scrub ok 2024-09-11T08:11:27.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:27 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:11:27.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:27 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:11:27.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:27 smithi150 ceph-mon[39612]: 3.18 scrub starts 2024-09-11T08:11:27.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:27 smithi150 ceph-mon[39612]: 3.18 scrub ok 2024-09-11T08:11:27.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:27 smithi150 ceph-mon[39612]: 7.c scrub starts 2024-09-11T08:11:27.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:27 smithi181 ceph-mon[32701]: 5.1 scrub starts 2024-09-11T08:11:27.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:27 smithi181 ceph-mon[32701]: 5.1 scrub ok 2024-09-11T08:11:27.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:27 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:11:27.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:27 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:11:27.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:27 smithi181 ceph-mon[32701]: 3.18 scrub starts 2024-09-11T08:11:27.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:27 smithi181 ceph-mon[32701]: 3.18 scrub ok 2024-09-11T08:11:27.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:27 smithi181 ceph-mon[32701]: 7.c scrub starts 2024-09-11T08:11:27.330 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:27.359 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:27.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:27.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:27.461 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:27.489 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:27.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:27.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:27.591 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:27.617 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:27.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:27.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:27.720 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:11:27.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:27.763 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:27.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:11:27.864 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:11:27.903 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:27.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:27.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:11:28.005 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:11:28.028 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:11:28.057 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:28 smithi150 ceph-mon[29276]: pgmap v155: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.0 KiB/s rd, 63 MiB/s wr, 1.01k op/s 2024-09-11T08:11:28.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:28.085 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:28.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:11:28.092 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Total time run: 90.0198 2024-09-11T08:11:28.092 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Total writes made: 124438 2024-09-11T08:11:28.092 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Write size: 65536 2024-09-11T08:11:28.092 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Object size: 65536 2024-09-11T08:11:28.092 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Bandwidth (MB/sec): 86.3962 2024-09-11T08:11:28.093 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Stddev Bandwidth: 34.9538 2024-09-11T08:11:28.093 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Max bandwidth (MB/sec): 221.312 2024-09-11T08:11:28.093 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Min bandwidth (MB/sec): 50.25 2024-09-11T08:11:28.093 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Average IOPS: 1382 2024-09-11T08:11:28.093 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Stddev IOPS: 559.26 2024-09-11T08:11:28.093 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Max IOPS: 3541 2024-09-11T08:11:28.093 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Min IOPS: 804 2024-09-11T08:11:28.093 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Average Latency(s): 0.0115707 2024-09-11T08:11:28.093 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Stddev Latency(s): 0.00838095 2024-09-11T08:11:28.093 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Max latency(s): 0.181517 2024-09-11T08:11:28.093 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Min latency(s): 0.00117796 2024-09-11T08:11:28.100 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Cleaning up (deleting benchmark objects) 2024-09-11T08:11:28.186 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:11:28.231 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:28.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:28.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:11:28.323 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:11:27 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:11:28.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:28 smithi181 ceph-mon[32701]: pgmap v155: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.0 KiB/s rd, 63 MiB/s wr, 1.01k op/s 2024-09-11T08:11:28.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:28 smithi181 ceph-mon[32701]: osdmap e126: 8 total, 8 up, 8 in 2024-09-11T08:11:28.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:28 smithi181 ceph-mon[32701]: 5.e scrub starts 2024-09-11T08:11:28.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:28 smithi181 ceph-mon[32701]: 5.e scrub ok 2024-09-11T08:11:28.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:28 smithi181 ceph-mon[32701]: 3.f scrub starts 2024-09-11T08:11:28.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:28 smithi181 ceph-mon[32701]: 3.f scrub ok 2024-09-11T08:11:28.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:28 smithi181 ceph-mon[32701]: 7.7 scrub ok 2024-09-11T08:11:28.333 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:28.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:28.370 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:28.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:28.471 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:11:28.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:28 smithi150 ceph-mon[29276]: osdmap e126: 8 total, 8 up, 8 in 2024-09-11T08:11:28.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:28 smithi150 ceph-mon[29276]: 5.e scrub starts 2024-09-11T08:11:28.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:28 smithi150 ceph-mon[29276]: 5.e scrub ok 2024-09-11T08:11:28.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:28 smithi150 ceph-mon[29276]: 3.f scrub starts 2024-09-11T08:11:28.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:28 smithi150 ceph-mon[29276]: 3.f scrub ok 2024-09-11T08:11:28.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:28 smithi150 ceph-mon[29276]: 7.7 scrub ok 2024-09-11T08:11:28.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:28 smithi150 ceph-mon[39612]: pgmap v155: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.0 KiB/s rd, 63 MiB/s wr, 1.01k op/s 2024-09-11T08:11:28.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:28 smithi150 ceph-mon[39612]: osdmap e126: 8 total, 8 up, 8 in 2024-09-11T08:11:28.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:28 smithi150 ceph-mon[39612]: 5.e scrub starts 2024-09-11T08:11:28.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:28 smithi150 ceph-mon[39612]: 5.e scrub ok 2024-09-11T08:11:28.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:28 smithi150 ceph-mon[39612]: 3.f scrub starts 2024-09-11T08:11:28.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:28 smithi150 ceph-mon[39612]: 3.f scrub ok 2024-09-11T08:11:28.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:28 smithi150 ceph-mon[39612]: 7.7 scrub ok 2024-09-11T08:11:28.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:28.500 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:28.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:11:28.602 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:11:28.643 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:28.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:28.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:11:28.745 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:11:28.760 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:11:28 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:11:28] "GET /metrics HTTP/1.1" 200 223631 "" "Prometheus/2.43.0" 2024-09-11T08:11:28.775 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:28.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:28.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:11:28.877 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:11:28.920 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:28.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:28.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:11:28.978 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:11:29.022 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:29.069 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:29.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:29.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:29.086 INFO:teuthology.orchestra.run.smithi150.stderr:reweighted osd.6 to 1 (10000) 2024-09-11T08:11:29.086 INFO:teuthology.orchestra.run.smithi150.stderr:nodeep-scrub is unset 2024-09-11T08:11:29.172 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:11:29.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:29.202 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:29.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:11:29.303 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:11:29.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:29 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:11:29.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:29 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/570304574' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:11:29.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:29 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:11:29.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:29 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3413261580' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-11T08:11:29.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:29 smithi181 ceph-mon[32701]: 5.1f scrub starts 2024-09-11T08:11:29.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:29 smithi181 ceph-mon[32701]: 5.1f scrub ok 2024-09-11T08:11:29.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:29 smithi181 ceph-mon[32701]: 7.c scrub ok 2024-09-11T08:11:29.329 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:29.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:29.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:11:29.431 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:29.459 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:29.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:29.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:29.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:29 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:11:29.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:29 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/570304574' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:11:29.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:29 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:11:29.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:29 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3413261580' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-11T08:11:29.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:29 smithi150 ceph-mon[29276]: 5.1f scrub starts 2024-09-11T08:11:29.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:29 smithi150 ceph-mon[29276]: 5.1f scrub ok 2024-09-11T08:11:29.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:29 smithi150 ceph-mon[29276]: 7.c scrub ok 2024-09-11T08:11:29.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:29 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:11:29.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:29 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/570304574' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:11:29.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:29 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:11:29.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:29 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3413261580' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-11T08:11:29.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:29 smithi150 ceph-mon[39612]: 5.1f scrub starts 2024-09-11T08:11:29.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:29 smithi150 ceph-mon[39612]: 5.1f scrub ok 2024-09-11T08:11:29.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:29 smithi150 ceph-mon[39612]: 7.c scrub ok 2024-09-11T08:11:29.561 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:29.588 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:29.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:29.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:29.690 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:11:29.714 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:29.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:29.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:11:29.815 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:29.854 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:29.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:29.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:29.957 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:29.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:29.998 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:29.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:30.099 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:30.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:30.139 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:30.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:30.240 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:30.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:30.275 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:30.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:30.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:30 smithi181 ceph-mon[32701]: pgmap v157: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.0 KiB/s rd, 60 MiB/s wr, 963 op/s 2024-09-11T08:11:30.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:30 smithi181 ceph-mon[32701]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:11:30.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:30 smithi181 ceph-mon[32701]: Cluster is now healthy 2024-09-11T08:11:30.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:30 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:11:30.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:30 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3413261580' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-11T08:11:30.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:30 smithi181 ceph-mon[32701]: osdmap e127: 8 total, 8 up, 8 in 2024-09-11T08:11:30.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:30 smithi181 ceph-mon[32701]: 5.14 scrub starts 2024-09-11T08:11:30.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:30 smithi181 ceph-mon[32701]: 5.14 scrub ok 2024-09-11T08:11:30.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:30 smithi181 ceph-mon[32701]: 4.a scrub starts 2024-09-11T08:11:30.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:30 smithi181 ceph-mon[32701]: 4.a scrub ok 2024-09-11T08:11:30.377 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:11:30.406 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:30.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:30.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:11:30.411 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:30 smithi150 ceph-mon[29276]: pgmap v157: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.0 KiB/s rd, 60 MiB/s wr, 963 op/s 2024-09-11T08:11:30.412 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:30 smithi150 ceph-mon[29276]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:11:30.412 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:30 smithi150 ceph-mon[29276]: Cluster is now healthy 2024-09-11T08:11:30.412 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:30 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:11:30.412 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:30 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3413261580' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-11T08:11:30.412 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:30 smithi150 ceph-mon[29276]: osdmap e127: 8 total, 8 up, 8 in 2024-09-11T08:11:30.412 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:30 smithi150 ceph-mon[29276]: 5.14 scrub starts 2024-09-11T08:11:30.412 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:30 smithi150 ceph-mon[29276]: 5.14 scrub ok 2024-09-11T08:11:30.412 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:30 smithi150 ceph-mon[29276]: 4.a scrub starts 2024-09-11T08:11:30.412 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:30 smithi150 ceph-mon[29276]: 4.a scrub ok 2024-09-11T08:11:30.413 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:30 smithi150 ceph-mon[39612]: pgmap v157: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.0 KiB/s rd, 60 MiB/s wr, 963 op/s 2024-09-11T08:11:30.413 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:30 smithi150 ceph-mon[39612]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:11:30.413 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:30 smithi150 ceph-mon[39612]: Cluster is now healthy 2024-09-11T08:11:30.413 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:30 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:11:30.413 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:30 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3413261580' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-11T08:11:30.413 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:30 smithi150 ceph-mon[39612]: osdmap e127: 8 total, 8 up, 8 in 2024-09-11T08:11:30.413 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:30 smithi150 ceph-mon[39612]: 5.14 scrub starts 2024-09-11T08:11:30.413 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:30 smithi150 ceph-mon[39612]: 5.14 scrub ok 2024-09-11T08:11:30.413 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:30 smithi150 ceph-mon[39612]: 4.a scrub starts 2024-09-11T08:11:30.413 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:30 smithi150 ceph-mon[39612]: 4.a scrub ok 2024-09-11T08:11:30.508 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:11:30.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:30.548 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:30.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:11:30.649 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:11:30.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:30.676 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:30.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:11:30.777 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:30.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:30.814 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:30.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:30.915 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:30.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:30.953 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:30.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:31.054 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:11:31.102 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:31.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:31.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:11:31.204 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:11:31.229 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:31.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:31.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:11:31.331 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:11:31.372 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:31.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:31.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:11:31.474 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:11:31.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:31.516 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:31.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:11:31.617 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:11:31.646 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:31.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:31.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:11:31.747 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:11:31.775 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:31.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:31.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:11:31.876 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:11:31.912 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:31.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:31.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:11:32.014 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:11:32.038 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:32.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:32.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:11:32.139 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:11:32.144 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:11:32.144 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:11:32.166 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:32.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:32.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:11:32.182 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:11:32.182 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:11:32.213 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:11:32.213 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:11:32.244 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:11:32.244 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:11:32.267 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:32.273 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:11:32.273 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:11:32.291 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:32.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:32.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:32.308 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:11:32.308 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:11:32.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:32 smithi181 ceph-mon[32701]: pgmap v159: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 60 MiB/s wr, 958 op/s 2024-09-11T08:11:32.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:32 smithi181 ceph-mon[32701]: 4.1e scrub starts 2024-09-11T08:11:32.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:32 smithi181 ceph-mon[32701]: 4.1e scrub ok 2024-09-11T08:11:32.340 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:11:32.340 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:11:32.340 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:32 smithi150 ceph-mon[29276]: pgmap v159: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 60 MiB/s wr, 958 op/s 2024-09-11T08:11:32.341 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:32 smithi150 ceph-mon[29276]: 4.1e scrub starts 2024-09-11T08:11:32.341 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:32 smithi150 ceph-mon[29276]: 4.1e scrub ok 2024-09-11T08:11:32.341 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:32 smithi150 ceph-mon[39612]: pgmap v159: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 60 MiB/s wr, 958 op/s 2024-09-11T08:11:32.341 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:32 smithi150 ceph-mon[39612]: 4.1e scrub starts 2024-09-11T08:11:32.341 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:32 smithi150 ceph-mon[39612]: 4.1e scrub ok 2024-09-11T08:11:32.372 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:11:32.372 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:11:32.394 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:11:32.436 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:32.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:32.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:11:32.538 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:11:32.564 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:32.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:32.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:11:32.666 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:11:32.710 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:32.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:32.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:11:32.812 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:11:32.846 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:32.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:32.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:11:32.948 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:11:32.974 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:32.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:32.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:11:33.076 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:11:33.114 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:33.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:33.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:11:33.170 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:33 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1768172461' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:11:33.170 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:33 smithi150 ceph-mon[29276]: 2.1 deep-scrub starts 2024-09-11T08:11:33.170 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:33 smithi150 ceph-mon[29276]: 2.1 deep-scrub ok 2024-09-11T08:11:33.170 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:33 smithi150 ceph-mon[29276]: 4.1d scrub starts 2024-09-11T08:11:33.170 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:33 smithi150 ceph-mon[29276]: 4.1d scrub ok 2024-09-11T08:11:33.170 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:33 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1768172461' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:11:33.170 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:33 smithi150 ceph-mon[39612]: 2.1 deep-scrub starts 2024-09-11T08:11:33.171 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:33 smithi150 ceph-mon[39612]: 2.1 deep-scrub ok 2024-09-11T08:11:33.171 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:33 smithi150 ceph-mon[39612]: 4.1d scrub starts 2024-09-11T08:11:33.171 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:33 smithi150 ceph-mon[39612]: 4.1d scrub ok 2024-09-11T08:11:33.216 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:11:33.245 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:33.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:33.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:11:33.347 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:11:33.378 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:33.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:33.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:11:33.480 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:11:33.530 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:33.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:33.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:11:33.559 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T08:11:33.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:33 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1768172461' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:11:33.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:33 smithi181 ceph-mon[32701]: 2.1 deep-scrub starts 2024-09-11T08:11:33.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:33 smithi181 ceph-mon[32701]: 2.1 deep-scrub ok 2024-09-11T08:11:33.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:33 smithi181 ceph-mon[32701]: 4.1d scrub starts 2024-09-11T08:11:33.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:33 smithi181 ceph-mon[32701]: 4.1d scrub ok 2024-09-11T08:11:33.631 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:11:33.660 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:33.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:33.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:11:33.763 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:33.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:33.786 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:33.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:33.887 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:33.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:33.914 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:33.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:34.015 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:34.055 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:34.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:34.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:34.157 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:11:34.183 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:34.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:34.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:11:34.285 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:11:34.311 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:34.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:34.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:11:34.413 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:34.438 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:34.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:34.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:34.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:34 smithi150 ceph-mon[29276]: pgmap v160: 148 pgs: 148 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.4 KiB/s rd, 38 MiB/s wr, 848 op/s 2024-09-11T08:11:34.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:34 smithi150 ceph-mon[29276]: 4.16 scrub starts 2024-09-11T08:11:34.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:34 smithi150 ceph-mon[29276]: 4.16 scrub ok 2024-09-11T08:11:34.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:34 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:11:34.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:34 smithi150 ceph-mon[29276]: 5.a scrub starts 2024-09-11T08:11:34.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:34 smithi150 ceph-mon[29276]: 5.a scrub ok 2024-09-11T08:11:34.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:34 smithi150 ceph-mon[29276]: 4.11 scrub starts 2024-09-11T08:11:34.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:34 smithi150 ceph-mon[29276]: 4.11 scrub ok 2024-09-11T08:11:34.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:34 smithi150 ceph-mon[39612]: pgmap v160: 148 pgs: 148 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.4 KiB/s rd, 38 MiB/s wr, 848 op/s 2024-09-11T08:11:34.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:34 smithi150 ceph-mon[39612]: 4.16 scrub starts 2024-09-11T08:11:34.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:34 smithi150 ceph-mon[39612]: 4.16 scrub ok 2024-09-11T08:11:34.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:34 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:11:34.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:34 smithi150 ceph-mon[39612]: 5.a scrub starts 2024-09-11T08:11:34.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:34 smithi150 ceph-mon[39612]: 5.a scrub ok 2024-09-11T08:11:34.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:34 smithi150 ceph-mon[39612]: 4.11 scrub starts 2024-09-11T08:11:34.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:34 smithi150 ceph-mon[39612]: 4.11 scrub ok 2024-09-11T08:11:34.540 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:11:34.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:34 smithi181 ceph-mon[32701]: pgmap v160: 148 pgs: 148 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.4 KiB/s rd, 38 MiB/s wr, 848 op/s 2024-09-11T08:11:34.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:34 smithi181 ceph-mon[32701]: 4.16 scrub starts 2024-09-11T08:11:34.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:34 smithi181 ceph-mon[32701]: 4.16 scrub ok 2024-09-11T08:11:34.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:34 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:11:34.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:34 smithi181 ceph-mon[32701]: 5.a scrub starts 2024-09-11T08:11:34.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:34 smithi181 ceph-mon[32701]: 5.a scrub ok 2024-09-11T08:11:34.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:34 smithi181 ceph-mon[32701]: 4.11 scrub starts 2024-09-11T08:11:34.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:34 smithi181 ceph-mon[32701]: 4.11 scrub ok 2024-09-11T08:11:34.583 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:34.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:34.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:11:34.685 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:11:34.719 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:34.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:34.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:11:34.821 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:34.849 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:34.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:34.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:34.883 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd reweight 7 1 2024-09-11T08:11:34.951 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:35.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:35.018 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:35.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:35.099 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T08:11:35.119 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:11:35.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:35.232 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:35.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:11:35.332 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:11:35.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:35.389 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:35.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:11:35.460 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:35 smithi150 ceph-mon[29276]: 3.c scrub starts 2024-09-11T08:11:35.460 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:35 smithi150 ceph-mon[29276]: 3.c scrub ok 2024-09-11T08:11:35.460 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:35 smithi150 ceph-mon[29276]: 3.6 scrub starts 2024-09-11T08:11:35.460 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:35 smithi150 ceph-mon[29276]: 3.6 scrub ok 2024-09-11T08:11:35.460 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:35 smithi150 ceph-mon[29276]: 5.10 deep-scrub starts 2024-09-11T08:11:35.460 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:35 smithi150 ceph-mon[29276]: 5.10 deep-scrub ok 2024-09-11T08:11:35.460 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:35 smithi150 ceph-mon[29276]: 6.1b scrub starts 2024-09-11T08:11:35.460 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:35 smithi150 ceph-mon[29276]: 6.1b scrub ok 2024-09-11T08:11:35.461 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:35 smithi150 ceph-mon[39612]: 3.c scrub starts 2024-09-11T08:11:35.461 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:35 smithi150 ceph-mon[39612]: 3.c scrub ok 2024-09-11T08:11:35.461 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:35 smithi150 ceph-mon[39612]: 3.6 scrub starts 2024-09-11T08:11:35.461 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:35 smithi150 ceph-mon[39612]: 3.6 scrub ok 2024-09-11T08:11:35.461 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:35 smithi150 ceph-mon[39612]: 5.10 deep-scrub starts 2024-09-11T08:11:35.461 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:35 smithi150 ceph-mon[39612]: 5.10 deep-scrub ok 2024-09-11T08:11:35.461 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:35 smithi150 ceph-mon[39612]: 6.1b scrub starts 2024-09-11T08:11:35.461 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:35 smithi150 ceph-mon[39612]: 6.1b scrub ok 2024-09-11T08:11:35.491 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:11:35.513 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:35.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:35.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:11:35.536 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:11:35.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:11:35 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:11:35] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:11:35.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:35 smithi181 ceph-mon[32701]: 3.c scrub starts 2024-09-11T08:11:35.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:35 smithi181 ceph-mon[32701]: 3.c scrub ok 2024-09-11T08:11:35.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:35 smithi181 ceph-mon[32701]: 3.6 scrub starts 2024-09-11T08:11:35.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:35 smithi181 ceph-mon[32701]: 3.6 scrub ok 2024-09-11T08:11:35.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:35 smithi181 ceph-mon[32701]: 5.10 deep-scrub starts 2024-09-11T08:11:35.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:35 smithi181 ceph-mon[32701]: 5.10 deep-scrub ok 2024-09-11T08:11:35.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:35 smithi181 ceph-mon[32701]: 6.1b scrub starts 2024-09-11T08:11:35.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:35 smithi181 ceph-mon[32701]: 6.1b scrub ok 2024-09-11T08:11:35.611 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:11:35.615 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:11:35.645 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:11:35.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:35.684 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:35.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:11:35.786 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:11:35.814 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:35.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:35.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:11:35.916 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:11:35.940 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:35.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:35.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:11:36.043 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:36.069 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:36.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:36.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:36.172 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:36.198 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:36.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:36.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:36.300 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:36.328 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:36.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:36.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:36.430 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:11:36.459 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:36.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:36.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:11:36.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:36 smithi150 ceph-mon[29276]: pgmap v161: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 39 MiB/s wr, 1.27k op/s 2024-09-11T08:11:36.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:36 smithi150 ceph-mon[29276]: 3.11 scrub starts 2024-09-11T08:11:36.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:36 smithi150 ceph-mon[29276]: 3.11 scrub ok 2024-09-11T08:11:36.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:36 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:11:36.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:36 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:11:36.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:36 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:11:36.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:36 smithi150 ceph-mon[29276]: 5.13 scrub starts 2024-09-11T08:11:36.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:36 smithi150 ceph-mon[29276]: 5.13 scrub ok 2024-09-11T08:11:36.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:36 smithi150 ceph-mon[39612]: pgmap v161: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 39 MiB/s wr, 1.27k op/s 2024-09-11T08:11:36.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:36 smithi150 ceph-mon[39612]: 3.11 scrub starts 2024-09-11T08:11:36.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:36 smithi150 ceph-mon[39612]: 3.11 scrub ok 2024-09-11T08:11:36.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:36 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:11:36.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:36 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:11:36.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:36 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:11:36.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:36 smithi150 ceph-mon[39612]: 5.13 scrub starts 2024-09-11T08:11:36.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:36 smithi150 ceph-mon[39612]: 5.13 scrub ok 2024-09-11T08:11:36.560 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:36.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:36 smithi181 ceph-mon[32701]: pgmap v161: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 39 MiB/s wr, 1.27k op/s 2024-09-11T08:11:36.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:36 smithi181 ceph-mon[32701]: 3.11 scrub starts 2024-09-11T08:11:36.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:36 smithi181 ceph-mon[32701]: 3.11 scrub ok 2024-09-11T08:11:36.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:36 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:11:36.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:36 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:11:36.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:36 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:11:36.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:36 smithi181 ceph-mon[32701]: 5.13 scrub starts 2024-09-11T08:11:36.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:36 smithi181 ceph-mon[32701]: 5.13 scrub ok 2024-09-11T08:11:36.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:36.598 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:36.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:36.699 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:36.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:36.724 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:36.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:36.825 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:11:36.854 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:36.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:36.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:11:36.957 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:11:36.990 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:36.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:36.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:11:37.092 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:37.126 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:37.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:37.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:37.227 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:11:37.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:37.269 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:37.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:11:37.370 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:37.398 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:37.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:37.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:37.501 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:11:37.525 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:37.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:37.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:11:37.626 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:37.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:37.671 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:37.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:37.772 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:37.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:37.813 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:37.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:37.915 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:37.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:37.950 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:37.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:37.981 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:37 smithi150 ceph-mon[29276]: 7.5 scrub starts 2024-09-11T08:11:37.981 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:37 smithi150 ceph-mon[29276]: 6.1 scrub starts 2024-09-11T08:11:37.981 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:37 smithi150 ceph-mon[29276]: 6.1 scrub ok 2024-09-11T08:11:37.981 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:37 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:11:37.981 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:37 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:11:37.981 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:37 smithi150 ceph-mon[29276]: pgmap v162: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 32 MiB/s wr, 1.05k op/s 2024-09-11T08:11:37.982 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:37 smithi150 ceph-mon[39612]: 7.5 scrub starts 2024-09-11T08:11:37.982 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:37 smithi150 ceph-mon[39612]: 6.1 scrub starts 2024-09-11T08:11:37.982 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:37 smithi150 ceph-mon[39612]: 6.1 scrub ok 2024-09-11T08:11:37.982 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:37 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:11:37.982 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:37 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:11:37.982 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:37 smithi150 ceph-mon[39612]: pgmap v162: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 32 MiB/s wr, 1.05k op/s 2024-09-11T08:11:38.051 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:11:38.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:37 smithi181 ceph-mon[32701]: 7.5 scrub starts 2024-09-11T08:11:38.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:37 smithi181 ceph-mon[32701]: 6.1 scrub starts 2024-09-11T08:11:38.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:37 smithi181 ceph-mon[32701]: 6.1 scrub ok 2024-09-11T08:11:38.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:37 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:11:38.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:37 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:11:38.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:37 smithi181 ceph-mon[32701]: pgmap v162: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 32 MiB/s wr, 1.05k op/s 2024-09-11T08:11:38.074 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:11:37 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:11:38.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:38.103 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:38.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:11:38.205 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:38.250 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:38.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:38.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:38.352 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:38.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:38.398 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:38.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:38.499 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:38.524 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:38.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:38.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:38.626 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:38.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:38.666 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:38.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:38.768 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:11:38.794 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:38.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:38.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:11:38.798 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-11T08:11:38.799 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [ 2024-09-11T08:11:38.799 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:11:38.799 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.14790.0:136139 7.f e127/91 7:f997c907:::benchmark_data_smithi150_78508_object11698:head v 127'8440, mlcod=127'8437)", 2024-09-11T08:11:38.799 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:38.775535+0000", 2024-09-11T08:11:38.799 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.016229170000000001, 2024-09-11T08:11:38.799 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.016267057000000001, 2024-09-11T08:11:38.799 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:11:38.799 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "started", 2024-09-11T08:11:38.799 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:11:38.799 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:11:38.800 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:11:38.800 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:38.775535+0000", 2024-09-11T08:11:38.800 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:11:38.800 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:11:38.800 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:11:38.800 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:11:38.800 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:38.775535+0000", 2024-09-11T08:11:38.800 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:11:38.800 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:11:38.801 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:11:38.801 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:11:38.801 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:38.775534+0000", 2024-09-11T08:11:38.801 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:11:38.801 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:11:38.801 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:11:38.801 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:11:38.801 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:38.775540+0000", 2024-09-11T08:11:38.802 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.4860000000000002e-06 2024-09-11T08:11:38.802 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:11:38.802 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:11:38.802 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:11:38.802 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:38.775548+0000", 2024-09-11T08:11:38.802 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.6750000000000002e-06 2024-09-11T08:11:38.802 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:11:38.803 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:11:38.803 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:11:38.803 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:38.775554+0000", 2024-09-11T08:11:38.803 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.3650000000000003e-06 2024-09-11T08:11:38.803 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:11:38.803 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:11:38.803 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:11:38.803 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:38.777754+0000", 2024-09-11T08:11:38.803 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0022007099999999998 2024-09-11T08:11:38.804 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:11:38.804 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:11:38.804 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:11:38.804 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:38.777771+0000", 2024-09-11T08:11:38.804 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.6303999999999998e-05 2024-09-11T08:11:38.804 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:11:38.804 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:11:38.804 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:11:38.805 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:11:38.805 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:11:38.805 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.14790.0:136143 7.f e127/91 7:f6c96d24:::benchmark_data_smithi150_78508_object11702:head v 127'8441, mlcod=127'8438)", 2024-09-11T08:11:38.805 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:38.783466+0000", 2024-09-11T08:11:38.805 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.008298171, 2024-09-11T08:11:38.805 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0084064770000000007, 2024-09-11T08:11:38.805 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:11:38.805 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "started", 2024-09-11T08:11:38.805 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:11:38.805 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:11:38.805 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:11:38.805 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:38.783466+0000", 2024-09-11T08:11:38.805 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:11:38.806 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:11:38.806 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:11:38.806 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:11:38.806 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:38.783466+0000", 2024-09-11T08:11:38.806 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:11:38.806 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:11:38.806 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:11:38.806 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:11:38.806 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:38.783465+0000", 2024-09-11T08:11:38.806 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:11:38.806 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:11:38.806 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:11:38.806 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:11:38.807 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:38.783470+0000", 2024-09-11T08:11:38.807 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.2249999999999999e-06 2024-09-11T08:11:38.807 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:11:38.807 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:11:38.807 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:11:38.807 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:38.783480+0000", 2024-09-11T08:11:38.807 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.6560000000000004e-06 2024-09-11T08:11:38.807 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:11:38.807 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:11:38.807 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:11:38.807 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:38.783485+0000", 2024-09-11T08:11:38.808 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.5480000000000001e-06 2024-09-11T08:11:38.808 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:11:38.808 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:11:38.808 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:11:38.808 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:38.784447+0000", 2024-09-11T08:11:38.808 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00096160600000000005 2024-09-11T08:11:38.808 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:11:38.808 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:11:38.808 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:11:38.809 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:38.784465+0000", 2024-09-11T08:11:38.809 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.7910999999999998e-05 2024-09-11T08:11:38.809 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:11:38.809 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:11:38.809 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:11:38.810 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:11:38.810 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:11:38.811 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.14790.0:136145 7.c e127/91 7:30e3b232:::benchmark_data_smithi150_78508_object11704:head v 127'8406, mlcod=127'8403)", 2024-09-11T08:11:38.811 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:38.784667+0000", 2024-09-11T08:11:38.811 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.0070964460000000002, 2024-09-11T08:11:38.811 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.007267178, 2024-09-11T08:11:38.811 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:11:38.811 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "started", 2024-09-11T08:11:38.811 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:11:38.811 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:11:38.811 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:11:38.811 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:38.784667+0000", 2024-09-11T08:11:38.811 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:11:38.812 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:11:38.812 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:11:38.812 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:11:38.812 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:38.784667+0000", 2024-09-11T08:11:38.812 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:11:38.812 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:11:38.812 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:11:38.812 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:11:38.812 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:38.784667+0000", 2024-09-11T08:11:38.812 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:11:38.812 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:11:38.812 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:11:38.813 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:11:38.813 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:38.784674+0000", 2024-09-11T08:11:38.813 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.1010000000000003e-06 2024-09-11T08:11:38.813 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:11:38.813 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:11:38.813 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:11:38.813 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:38.784683+0000", 2024-09-11T08:11:38.813 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.7410000000000004e-06 2024-09-11T08:11:38.813 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:11:38.813 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:11:38.813 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:11:38.814 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:38.784689+0000", 2024-09-11T08:11:38.814 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.3190000000000002e-06 2024-09-11T08:11:38.814 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:11:38.814 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:11:38.814 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:11:38.814 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:38.784837+0000", 2024-09-11T08:11:38.814 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000148135 2024-09-11T08:11:38.814 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:11:38.814 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:11:38.814 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:11:38.814 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:38.784854+0000", 2024-09-11T08:11:38.815 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.6801999999999999e-05 2024-09-11T08:11:38.815 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:11:38.815 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:11:38.815 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:11:38.815 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:11:38.815 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:11:38.815 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.14790.0:136146 7.7 e127/91 7:e416a02a:::benchmark_data_smithi150_78508_object11705:head v 127'8571, mlcod=127'8567)", 2024-09-11T08:11:38.815 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:38.784702+0000", 2024-09-11T08:11:38.815 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.0070614709999999997, 2024-09-11T08:11:38.815 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0072932730000000003, 2024-09-11T08:11:38.815 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:11:38.815 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "started", 2024-09-11T08:11:38.815 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:11:38.815 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:11:38.816 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:11:38.816 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:38.784702+0000", 2024-09-11T08:11:38.816 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:11:38.816 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:11:38.816 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:11:38.816 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:11:38.816 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:38.784702+0000", 2024-09-11T08:11:38.816 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:11:38.816 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:11:38.816 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:11:38.816 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:11:38.816 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:38.784702+0000", 2024-09-11T08:11:38.816 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:11:38.816 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:11:38.817 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:11:38.817 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:11:38.817 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:38.784706+0000", 2024-09-11T08:11:38.817 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.6669999999999998e-06 2024-09-11T08:11:38.817 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:11:38.817 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:11:38.817 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:11:38.817 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:38.784710+0000", 2024-09-11T08:11:38.817 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.6469999999999997e-06 2024-09-11T08:11:38.817 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:11:38.817 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:11:38.817 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:11:38.817 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:38.784713+0000", 2024-09-11T08:11:38.817 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.1499999999999999e-06 2024-09-11T08:11:38.817 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:11:38.818 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:11:38.818 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:11:38.818 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:38.787561+0000", 2024-09-11T08:11:38.818 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0028473140000000001 2024-09-11T08:11:38.818 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:11:38.818 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:11:38.818 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:11:38.818 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:38.787578+0000", 2024-09-11T08:11:38.818 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.7226000000000001e-05 2024-09-11T08:11:38.818 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:11:38.818 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:11:38.818 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:11:38.818 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:11:38.818 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:11:38.819 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.14790.0:136151 7.c e127/91)", 2024-09-11T08:11:38.819 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:38.785720+0000", 2024-09-11T08:11:38.819 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.0060435209999999996, 2024-09-11T08:11:38.819 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0063341969999999997, 2024-09-11T08:11:38.819 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:11:38.819 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "queued for pg", 2024-09-11T08:11:38.819 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:11:38.819 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:11:38.819 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:11:38.819 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:38.785720+0000", 2024-09-11T08:11:38.819 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:11:38.819 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:11:38.819 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:11:38.819 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:11:38.820 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:38.785720+0000", 2024-09-11T08:11:38.820 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:11:38.820 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:11:38.820 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:11:38.820 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:11:38.820 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:38.785720+0000", 2024-09-11T08:11:38.820 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:11:38.820 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:11:38.820 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:11:38.820 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:11:38.820 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:38.785725+0000", 2024-09-11T08:11:38.820 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.3990000000000003e-06 2024-09-11T08:11:38.820 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:11:38.820 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:11:38.821 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:11:38.821 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:38.785733+0000", 2024-09-11T08:11:38.821 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.0180000000000001e-06 2024-09-11T08:11:38.821 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:11:38.821 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:11:38.821 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:11:38.821 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:38.785738+0000", 2024-09-11T08:11:38.821 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.639e-06 2024-09-11T08:11:38.821 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:11:38.821 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:11:38.821 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:11:38.821 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:11:38.821 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-09-11T08:11:38.821 INFO:teuthology.orchestra.run.smithi150.stdout: "num_ops": 5 2024-09-11T08:11:38.821 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-11T08:11:38.890 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:11:38 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:11:38] "GET /metrics HTTP/1.1" 200 223714 "" "Prometheus/2.43.0" 2024-09-11T08:11:38.890 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:38 smithi150 ceph-mon[39612]: 6.c scrub starts 2024-09-11T08:11:38.890 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:38 smithi150 ceph-mon[39612]: 6.c scrub ok 2024-09-11T08:11:38.891 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:38 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:11:38.891 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:38 smithi150 ceph-mon[39612]: 6.1e scrub starts 2024-09-11T08:11:38.891 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:38 smithi150 ceph-mon[39612]: 6.1e scrub ok 2024-09-11T08:11:38.891 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:38 smithi150 ceph-mon[29276]: 6.c scrub starts 2024-09-11T08:11:38.891 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:38 smithi150 ceph-mon[29276]: 6.c scrub ok 2024-09-11T08:11:38.891 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:38 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:11:38.891 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:38 smithi150 ceph-mon[29276]: 6.1e scrub starts 2024-09-11T08:11:38.891 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:38 smithi150 ceph-mon[29276]: 6.1e scrub ok 2024-09-11T08:11:38.896 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:38.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:38.944 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:38.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:39.045 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:11:39.071 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:39.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:39.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:11:39.078 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:38 smithi181 ceph-mon[32701]: 6.c scrub starts 2024-09-11T08:11:39.078 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:38 smithi181 ceph-mon[32701]: 6.c scrub ok 2024-09-11T08:11:39.078 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:38 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:11:39.078 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:38 smithi181 ceph-mon[32701]: 6.1e scrub starts 2024-09-11T08:11:39.078 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:38 smithi181 ceph-mon[32701]: 6.1e scrub ok 2024-09-11T08:11:39.173 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:11:39.214 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:39.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:39.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:11:39.317 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:11:39.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:39.357 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:39.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:11:39.458 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:11:39.497 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:39.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:39.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:11:39.598 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:11:39.627 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:39.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:39.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:11:39.729 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:11:39.756 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:39.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:39.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:11:39.773 INFO:teuthology.orchestra.run.smithi150.stderr:reweighted osd.7 to 1 (10000) 2024-09-11T08:11:39.858 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:11:39.887 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:39.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:39.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:11:39.989 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:40.005 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:39 smithi150 ceph-mon[29276]: 6.6 scrub starts 2024-09-11T08:11:40.006 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:39 smithi150 ceph-mon[29276]: 6.6 scrub ok 2024-09-11T08:11:40.006 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:39 smithi150 ceph-mon[29276]: 5.11 scrub starts 2024-09-11T08:11:40.006 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:39 smithi150 ceph-mon[29276]: 5.11 scrub ok 2024-09-11T08:11:40.006 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:39 smithi150 ceph-mon[29276]: pgmap v163: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.4 MiB/s wr, 1.09k op/s 2024-09-11T08:11:40.006 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:39 smithi150 ceph-mon[29276]: 7.5 scrub ok 2024-09-11T08:11:40.006 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:39 smithi150 ceph-mon[29276]: 4.13 scrub starts 2024-09-11T08:11:40.006 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:39 smithi150 ceph-mon[29276]: 4.13 scrub ok 2024-09-11T08:11:40.006 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:39 smithi150 ceph-mon[29276]: 6.f scrub starts 2024-09-11T08:11:40.006 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:39 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/512874739' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-11T08:11:40.006 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:39 smithi150 ceph-mon[29276]: 6.f scrub ok 2024-09-11T08:11:40.006 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:39 smithi150 ceph-mon[39612]: 6.6 scrub starts 2024-09-11T08:11:40.007 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:39 smithi150 ceph-mon[39612]: 6.6 scrub ok 2024-09-11T08:11:40.007 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:39 smithi150 ceph-mon[39612]: 5.11 scrub starts 2024-09-11T08:11:40.007 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:39 smithi150 ceph-mon[39612]: 5.11 scrub ok 2024-09-11T08:11:40.007 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:39 smithi150 ceph-mon[39612]: pgmap v163: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.4 MiB/s wr, 1.09k op/s 2024-09-11T08:11:40.007 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:39 smithi150 ceph-mon[39612]: 7.5 scrub ok 2024-09-11T08:11:40.007 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:39 smithi150 ceph-mon[39612]: 4.13 scrub starts 2024-09-11T08:11:40.007 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:39 smithi150 ceph-mon[39612]: 4.13 scrub ok 2024-09-11T08:11:40.007 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:39 smithi150 ceph-mon[39612]: 6.f scrub starts 2024-09-11T08:11:40.007 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:39 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/512874739' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-11T08:11:40.007 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:39 smithi150 ceph-mon[39612]: 6.f scrub ok 2024-09-11T08:11:40.018 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:40.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:40.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:40.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:39 smithi181 ceph-mon[32701]: 6.6 scrub starts 2024-09-11T08:11:40.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:39 smithi181 ceph-mon[32701]: 6.6 scrub ok 2024-09-11T08:11:40.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:39 smithi181 ceph-mon[32701]: 5.11 scrub starts 2024-09-11T08:11:40.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:39 smithi181 ceph-mon[32701]: 5.11 scrub ok 2024-09-11T08:11:40.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:39 smithi181 ceph-mon[32701]: pgmap v163: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.4 MiB/s wr, 1.09k op/s 2024-09-11T08:11:40.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:39 smithi181 ceph-mon[32701]: 7.5 scrub ok 2024-09-11T08:11:40.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:39 smithi181 ceph-mon[32701]: 4.13 scrub starts 2024-09-11T08:11:40.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:39 smithi181 ceph-mon[32701]: 4.13 scrub ok 2024-09-11T08:11:40.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:39 smithi181 ceph-mon[32701]: 6.f scrub starts 2024-09-11T08:11:40.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:39 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/512874739' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-11T08:11:40.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:39 smithi181 ceph-mon[32701]: 6.f scrub ok 2024-09-11T08:11:40.120 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:11:40.150 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:40.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:40.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:11:40.252 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:11:40.278 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:40.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:40.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:11:40.380 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:40.407 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:40.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:40.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:40.510 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:11:40.530 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-11T08:11:40.530 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T08:11:40.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:40.559 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:40.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:11:40.660 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:11:40.687 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:40.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:40.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:11:40.789 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:11:40.815 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:40.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:40.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:11:40.917 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:40.956 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:40.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:40.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:41.038 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:40 smithi150 ceph-mon[29276]: 5.3 scrub starts 2024-09-11T08:11:41.038 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:40 smithi150 ceph-mon[29276]: 5.3 scrub ok 2024-09-11T08:11:41.039 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:40 smithi150 ceph-mon[29276]: 3.15 scrub starts 2024-09-11T08:11:41.039 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:40 smithi150 ceph-mon[29276]: 3.15 scrub ok 2024-09-11T08:11:41.039 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:40 smithi150 ceph-mon[29276]: 7.b deep-scrub starts 2024-09-11T08:11:41.039 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:40 smithi150 ceph-mon[29276]: 6.d scrub starts 2024-09-11T08:11:41.039 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:40 smithi150 ceph-mon[29276]: 6.d scrub ok 2024-09-11T08:11:41.039 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:40 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/512874739' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-11T08:11:41.039 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:40 smithi150 ceph-mon[29276]: osdmap e128: 8 total, 8 up, 8 in 2024-09-11T08:11:41.039 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:40 smithi150 ceph-mon[39612]: 5.3 scrub starts 2024-09-11T08:11:41.039 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:40 smithi150 ceph-mon[39612]: 5.3 scrub ok 2024-09-11T08:11:41.039 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:40 smithi150 ceph-mon[39612]: 3.15 scrub starts 2024-09-11T08:11:41.040 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:40 smithi150 ceph-mon[39612]: 3.15 scrub ok 2024-09-11T08:11:41.040 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:40 smithi150 ceph-mon[39612]: 7.b deep-scrub starts 2024-09-11T08:11:41.040 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:40 smithi150 ceph-mon[39612]: 6.d scrub starts 2024-09-11T08:11:41.040 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:40 smithi150 ceph-mon[39612]: 6.d scrub ok 2024-09-11T08:11:41.040 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:40 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/512874739' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-11T08:11:41.040 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:40 smithi150 ceph-mon[39612]: osdmap e128: 8 total, 8 up, 8 in 2024-09-11T08:11:41.058 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:11:41.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:40 smithi181 ceph-mon[32701]: 5.3 scrub starts 2024-09-11T08:11:41.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:40 smithi181 ceph-mon[32701]: 5.3 scrub ok 2024-09-11T08:11:41.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:40 smithi181 ceph-mon[32701]: 3.15 scrub starts 2024-09-11T08:11:41.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:40 smithi181 ceph-mon[32701]: 3.15 scrub ok 2024-09-11T08:11:41.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:40 smithi181 ceph-mon[32701]: 7.b deep-scrub starts 2024-09-11T08:11:41.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:40 smithi181 ceph-mon[32701]: 6.d scrub starts 2024-09-11T08:11:41.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:40 smithi181 ceph-mon[32701]: 6.d scrub ok 2024-09-11T08:11:41.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:40 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/512874739' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-11T08:11:41.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:40 smithi181 ceph-mon[32701]: osdmap e128: 8 total, 8 up, 8 in 2024-09-11T08:11:41.085 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:41.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:41.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:11:41.187 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:11:41.225 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:41.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:41.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:11:41.327 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:11:41.354 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:41.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:41.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:11:41.455 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:41.494 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:41.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:41.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:41.596 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:41.632 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:41.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:41.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:41.734 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:41.776 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:41.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:41.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:41.836 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-11T08:11:41.878 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:11:41.903 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:41.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:41.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:11:42.005 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:42.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:41 smithi181 ceph-mon[32701]: 4.b scrub starts 2024-09-11T08:11:42.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:41 smithi181 ceph-mon[32701]: 4.b scrub ok 2024-09-11T08:11:42.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:41 smithi181 ceph-mon[32701]: 5.19 scrub starts 2024-09-11T08:11:42.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:41 smithi181 ceph-mon[32701]: 5.19 scrub ok 2024-09-11T08:11:42.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:41 smithi181 ceph-mon[32701]: pgmap v165: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.4 MiB/s wr, 1.09k op/s 2024-09-11T08:11:42.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:41 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1429507997' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:11:42.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:41 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:11:42.080 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:41 smithi150 ceph-mon[29276]: 4.b scrub starts 2024-09-11T08:11:42.080 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:41 smithi150 ceph-mon[29276]: 4.b scrub ok 2024-09-11T08:11:42.081 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:41 smithi150 ceph-mon[29276]: 5.19 scrub starts 2024-09-11T08:11:42.081 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:41 smithi150 ceph-mon[29276]: 5.19 scrub ok 2024-09-11T08:11:42.081 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:41 smithi150 ceph-mon[29276]: pgmap v165: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.4 MiB/s wr, 1.09k op/s 2024-09-11T08:11:42.081 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:41 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1429507997' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:11:42.081 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:41 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:11:42.081 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:41 smithi150 ceph-mon[39612]: 4.b scrub starts 2024-09-11T08:11:42.081 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:41 smithi150 ceph-mon[39612]: 4.b scrub ok 2024-09-11T08:11:42.081 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:41 smithi150 ceph-mon[39612]: 5.19 scrub starts 2024-09-11T08:11:42.081 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:41 smithi150 ceph-mon[39612]: 5.19 scrub ok 2024-09-11T08:11:42.081 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:41 smithi150 ceph-mon[39612]: pgmap v165: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.4 MiB/s wr, 1.09k op/s 2024-09-11T08:11:42.081 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:41 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1429507997' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:11:42.082 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:41 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:11:42.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:42.082 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:42.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:42.183 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:11:42.210 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:42.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:42.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:11:42.312 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:11:42.354 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:42.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:42.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:11:42.434 INFO:teuthology.orchestra.run.smithi150.stderr:noscrub is set 2024-09-11T08:11:42.456 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:42.508 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:42.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:42.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:42.610 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:11:42.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:42.640 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:42.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:11:42.741 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:11:42.771 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:42.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:42.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:11:42.873 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:11:42.912 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:42.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:42.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:11:43.014 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:11:43.041 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:43.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:43.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:11:43.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:42 smithi181 ceph-mon[32701]: 7.2 scrub starts 2024-09-11T08:11:43.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:42 smithi181 ceph-mon[32701]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:11:43.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:42 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:11:43.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:42 smithi181 ceph-mon[32701]: osdmap e129: 8 total, 8 up, 8 in 2024-09-11T08:11:43.105 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:42 smithi150 ceph-mon[29276]: 7.2 scrub starts 2024-09-11T08:11:43.105 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:42 smithi150 ceph-mon[29276]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:11:43.105 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:42 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:11:43.105 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:42 smithi150 ceph-mon[29276]: osdmap e129: 8 total, 8 up, 8 in 2024-09-11T08:11:43.106 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:42 smithi150 ceph-mon[39612]: 7.2 scrub starts 2024-09-11T08:11:43.107 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:42 smithi150 ceph-mon[39612]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:11:43.107 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:42 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:11:43.107 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:42 smithi150 ceph-mon[39612]: osdmap e129: 8 total, 8 up, 8 in 2024-09-11T08:11:43.143 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:11:43.183 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:43.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:43.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:11:43.285 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:11:43.324 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:43.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:43.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:11:43.426 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:43.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:43.461 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:43.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:43.549 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:11:43.562 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:43.620 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:11:43.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:43.637 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:43.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:43.737 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:11:43.763 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:43.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:43.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:11:43.854 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:43 smithi150 ceph-mon[39612]: 7.3 scrub starts 2024-09-11T08:11:43.854 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:43 smithi150 ceph-mon[39612]: pgmap v167: 148 pgs: 148 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 919 op/s 2024-09-11T08:11:43.855 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:43 smithi150 ceph-mon[29276]: 7.3 scrub starts 2024-09-11T08:11:43.855 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:43 smithi150 ceph-mon[29276]: pgmap v167: 148 pgs: 148 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 919 op/s 2024-09-11T08:11:43.865 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:43.907 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:43.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:43.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:44.009 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:11:44.038 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:44.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:44.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:11:44.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:43 smithi181 ceph-mon[32701]: 7.3 scrub starts 2024-09-11T08:11:44.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:43 smithi181 ceph-mon[32701]: pgmap v167: 148 pgs: 148 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 919 op/s 2024-09-11T08:11:44.140 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:44.176 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:44.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:44.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:44.278 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:11:44.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:44.311 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:44.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:11:44.412 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:11:44.442 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:44.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:44.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:11:44.544 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:44.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:44.573 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:44.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:44.675 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:11:44.703 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:44.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:44.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:11:44.806 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:44.843 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:44.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:44.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:44.945 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:11:44.978 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:44.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:44.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:11:45.080 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:11:45.122 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:45.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:45.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:11:45.146 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T08:11:45.224 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:11:45.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:45.247 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:45.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:11:45.348 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:45.374 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:45.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:45.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:45.476 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:11:45.498 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:45.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:45.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:11:45.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:11:45 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:11:45] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:11:45.600 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:11:45.626 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:45.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:45.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:11:45.729 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:11:45.760 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:45.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:45.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:11:45.862 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:45.894 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:45.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:45.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:45.996 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:46.034 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:46.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:46.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:46.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:45 smithi181 ceph-mon[32701]: pgmap v168: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s 2024-09-11T08:11:46.136 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:46.170 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:46.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:46.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:46.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:45 smithi150 ceph-mon[29276]: pgmap v168: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s 2024-09-11T08:11:46.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:45 smithi150 ceph-mon[39612]: pgmap v168: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s 2024-09-11T08:11:46.272 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:46.314 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:46.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:46.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:46.416 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:11:46.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:46.524 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:46.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:11:46.625 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:11:46.652 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:46.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:46.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:11:46.754 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:11:46.781 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:46.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:46.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:11:46.883 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:46.909 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:46.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:46.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:47.011 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:47.055 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:47.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:47.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:47.157 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:47.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:47.188 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:47.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:47.289 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:11:47.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:47.334 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:47.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:11:47.435 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:11:47.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:47.470 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:47.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:11:47.571 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:11:47.594 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:47.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:47.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:11:47.696 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:47.736 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:47.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:47.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:47.838 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:11:47.866 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:47.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:47.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:11:47.889 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:11:47.898 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-09-11T08:11:47.969 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:11:47.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:47 smithi150 ceph-mon[29276]: pgmap v169: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 621 op/s 2024-09-11T08:11:47.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:47 smithi150 ceph-mon[39612]: pgmap v169: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 621 op/s 2024-09-11T08:11:48.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:48.004 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:48.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:11:48.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:47 smithi181 ceph-mon[32701]: pgmap v169: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 621 op/s 2024-09-11T08:11:48.074 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:11:47 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:11:48.105 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:48.151 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:48.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:48.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:48.253 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:48.282 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:48.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:48.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:48.383 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:48.420 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:48.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:48.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:48.523 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:48.551 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:48.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:48.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:48.653 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:11:48.687 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:48.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:48.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:11:48.733 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:11:48 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:11:48] "GET /metrics HTTP/1.1" 200 223714 "" "Prometheus/2.43.0" 2024-09-11T08:11:48.770 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":170,"stamp":"2024-09-11T08:11:47.795333+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6970022132,"num_objects":106542,"num_object_clones":0,"num_object_copies":319626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":106542,"num_whiteouts":0,"num_read":1105,"num_read_kb":962,"num_write":143075,"num_write_kb":7965704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":590698,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90556,"ondisk_log_size":90556,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":22882412,"kb_used_data":20461988,"kb_used_omap":13,"kb_used_meta":2420338,"kb_avail":727078804,"statfs":{"total":767960285184,"available":744528695296,"internally_reserved":0,"allocated":20953075712,"data_stored":20949864464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2478426816},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,15,7,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-488112128,"num_objects":-7448,"num_object_clones":0,"num_object_copies":-22344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7448,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":7448,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.036535"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":99,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.776115+0000","last_change":"2024-09-11T08:10:19.111028+0000","last_active":"2024-09-11T08:11:39.776115+0000","last_peered":"2024-09-11T08:11:39.776115+0000","last_clean":"2024-09-11T08:11:39.776115+0000","last_became_active":"2024-09-11T08:07:30.044976+0000","last_became_peered":"2024-09-11T08:07:30.044976+0000","last_unstale":"2024-09-11T08:11:39.776115+0000","last_undegraded":"2024-09-11T08:11:39.776115+0000","last_fullsized":"2024-09-11T08:11:39.776115+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:19.110647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:10:19.110647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.017276071,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":103,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.760819+0000","last_change":"2024-09-11T08:09:24.630803+0000","last_active":"2024-09-11T08:11:39.760819+0000","last_peered":"2024-09-11T08:11:39.760819+0000","last_clean":"2024-09-11T08:11:39.760819+0000","last_became_active":"2024-09-11T08:07:32.048048+0000","last_became_peered":"2024-09-11T08:07:32.048048+0000","last_unstale":"2024-09-11T08:11:39.760819+0000","last_undegraded":"2024-09-11T08:11:39.760819+0000","last_fullsized":"2024-09-11T08:11:39.760819+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:09:24.630735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:09:24.630735+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020206859999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":105,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:40.568631+0000","last_change":"2024-09-11T08:11:40.568270+0000","last_active":"2024-09-11T08:11:40.568631+0000","last_peered":"2024-09-11T08:11:40.568631+0000","last_clean":"2024-09-11T08:11:40.568631+0000","last_became_active":"2024-09-11T08:07:34.058828+0000","last_became_peered":"2024-09-11T08:07:34.058828+0000","last_unstale":"2024-09-11T08:11:40.568631+0000","last_undegraded":"2024-09-11T08:11:40.568631+0000","last_fullsized":"2024-09-11T08:11:40.568631+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:40.567720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:11:40.567720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:13:05.932274+0000","scrub_duration":0.013465969,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":93,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.760810+0000","last_change":"2024-09-11T08:11:16.061718+0000","last_active":"2024-09-11T08:11:39.760810+0000","last_peered":"2024-09-11T08:11:39.760810+0000","last_clean":"2024-09-11T08:11:39.760810+0000","last_became_active":"2024-09-11T08:07:36.074789+0000","last_became_peered":"2024-09-11T08:07:36.074789+0000","last_unstale":"2024-09-11T08:11:39.760810+0000","last_undegraded":"2024-09-11T08:11:39.760810+0000","last_fullsized":"2024-09-11T08:11:39.760810+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:16.061669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:02.470649+0000","last_clean_scrub_stamp":"2024-09-11T08:11:16.061669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:25.199054+0000","scrub_duration":0.0051287930000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":93,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.763995+0000","last_change":"2024-09-11T08:11:34.567724+0000","last_active":"2024-09-11T08:11:39.763995+0000","last_peered":"2024-09-11T08:11:39.763995+0000","last_clean":"2024-09-11T08:11:39.763995+0000","last_became_active":"2024-09-11T08:07:36.633423+0000","last_became_peered":"2024-09-11T08:07:36.633423+0000","last_unstale":"2024-09-11T08:11:39.763995+0000","last_undegraded":"2024-09-11T08:11:39.763995+0000","last_fullsized":"2024-09-11T08:11:39.763995+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:34.567654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:11:34.567654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:13:02.653394+0000","scrub_duration":0.0064880140000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":89,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.763705+0000","last_change":"2024-09-11T08:09:35.348975+0000","last_active":"2024-09-11T08:11:39.763705+0000","last_peered":"2024-09-11T08:11:39.763705+0000","last_clean":"2024-09-11T08:11:39.763705+0000","last_became_active":"2024-09-11T08:07:30.045579+0000","last_became_peered":"2024-09-11T08:07:30.045579+0000","last_unstale":"2024-09-11T08:11:39.763705+0000","last_undegraded":"2024-09-11T08:11:39.763705+0000","last_fullsized":"2024-09-11T08:11:39.763705+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:35.348912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:09:35.348912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001468128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":122,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.763819+0000","last_change":"2024-09-11T08:11:19.427044+0000","last_active":"2024-09-11T08:11:39.763819+0000","last_peered":"2024-09-11T08:11:39.763819+0000","last_clean":"2024-09-11T08:11:39.763819+0000","last_became_active":"2024-09-11T08:07:32.045848+0000","last_became_peered":"2024-09-11T08:07:32.045848+0000","last_unstale":"2024-09-11T08:11:39.763819+0000","last_undegraded":"2024-09-11T08:11:39.763819+0000","last_fullsized":"2024-09-11T08:11:39.763819+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-11T08:11:19.426563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:11:19.426563+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:34.923254+0000","scrub_duration":0.021858643000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":85,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.760644+0000","last_change":"2024-09-11T08:09:31.638930+0000","last_active":"2024-09-11T08:11:39.760644+0000","last_peered":"2024-09-11T08:11:39.760644+0000","last_clean":"2024-09-11T08:11:39.760644+0000","last_became_active":"2024-09-11T08:07:34.052243+0000","last_became_peered":"2024-09-11T08:07:34.052243+0000","last_unstale":"2024-09-11T08:11:39.760644+0000","last_undegraded":"2024-09-11T08:11:39.760644+0000","last_fullsized":"2024-09-11T08:11:39.760644+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:31.638867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:09:31.638867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014270719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":98,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.774506+0000","last_change":"2024-09-11T08:10:26.020726+0000","last_active":"2024-09-11T08:11:39.774506+0000","last_peered":"2024-09-11T08:11:39.774506+0000","last_clean":"2024-09-11T08:11:39.774506+0000","last_became_active":"2024-09-11T08:07:30.043950+0000","last_became_peered":"2024-09-11T08:07:30.043950+0000","last_unstale":"2024-09-11T08:11:39.774506+0000","last_undegraded":"2024-09-11T08:11:39.774506+0000","last_fullsized":"2024-09-11T08:11:39.774506+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:26.020674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:54.878409+0000","last_clean_scrub_stamp":"2024-09-11T08:10:26.020674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:44.331704+0000","scrub_duration":0.001074111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":113,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.760995+0000","last_change":"2024-09-11T08:10:42.141214+0000","last_active":"2024-09-11T08:11:39.760995+0000","last_peered":"2024-09-11T08:11:39.760995+0000","last_clean":"2024-09-11T08:11:39.760995+0000","last_became_active":"2024-09-11T08:07:32.044626+0000","last_became_peered":"2024-09-11T08:07:32.044626+0000","last_unstale":"2024-09-11T08:11:39.760995+0000","last_undegraded":"2024-09-11T08:11:39.760995+0000","last_fullsized":"2024-09-11T08:11:39.760995+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:10:42.141115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:10:42.141115+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:52.487697+0000","scrub_duration":0.0076042940000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":94,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.774921+0000","last_change":"2024-09-11T08:10:44.820612+0000","last_active":"2024-09-11T08:11:39.774921+0000","last_peered":"2024-09-11T08:11:39.774921+0000","last_clean":"2024-09-11T08:11:39.774921+0000","last_became_active":"2024-09-11T08:07:34.058689+0000","last_became_peered":"2024-09-11T08:07:34.058689+0000","last_unstale":"2024-09-11T08:11:39.774921+0000","last_undegraded":"2024-09-11T08:11:39.774921+0000","last_fullsized":"2024-09-11T08:11:39.774921+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:44.820382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:10:44.820382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:09.539844+0000","scrub_duration":0.0067487449999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":93,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.768638+0000","last_change":"2024-09-11T08:10:16.101381+0000","last_active":"2024-09-11T08:11:39.768638+0000","last_peered":"2024-09-11T08:11:39.768638+0000","last_clean":"2024-09-11T08:11:39.768638+0000","last_became_active":"2024-09-11T08:07:36.076824+0000","last_became_peered":"2024-09-11T08:07:36.076824+0000","last_unstale":"2024-09-11T08:11:39.768638+0000","last_undegraded":"2024-09-11T08:11:39.768638+0000","last_fullsized":"2024-09-11T08:11:39.768638+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:16.101299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:10:16.101299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0062695850000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":98,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.774167+0000","last_change":"2024-09-11T08:10:15.984728+0000","last_active":"2024-09-11T08:11:39.774167+0000","last_peered":"2024-09-11T08:11:39.774167+0000","last_clean":"2024-09-11T08:11:39.774167+0000","last_became_active":"2024-09-11T08:07:30.043277+0000","last_became_peered":"2024-09-11T08:07:30.043277+0000","last_unstale":"2024-09-11T08:11:39.774167+0000","last_undegraded":"2024-09-11T08:11:39.774167+0000","last_fullsized":"2024-09-11T08:11:39.774167+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:15.984636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:10:15.984636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:44.117790+0000","scrub_duration":0.0055326200000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":97,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.760951+0000","last_change":"2024-09-11T08:09:08.668983+0000","last_active":"2024-09-11T08:11:39.760951+0000","last_peered":"2024-09-11T08:11:39.760951+0000","last_clean":"2024-09-11T08:11:39.760951+0000","last_became_active":"2024-09-11T08:07:32.048661+0000","last_became_peered":"2024-09-11T08:07:32.048661+0000","last_unstale":"2024-09-11T08:11:39.760951+0000","last_undegraded":"2024-09-11T08:11:39.760951+0000","last_fullsized":"2024-09-11T08:11:39.760951+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-11T08:09:08.668925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:09:08.668925+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001704496,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":95,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.767960+0000","last_change":"2024-09-11T08:11:20.459210+0000","last_active":"2024-09-11T08:11:39.767960+0000","last_peered":"2024-09-11T08:11:39.767960+0000","last_clean":"2024-09-11T08:11:39.767960+0000","last_became_active":"2024-09-11T08:07:34.060871+0000","last_became_peered":"2024-09-11T08:07:34.060871+0000","last_unstale":"2024-09-11T08:11:39.767960+0000","last_undegraded":"2024-09-11T08:11:39.767960+0000","last_fullsized":"2024-09-11T08:11:39.767960+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:20.459128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:11:20.459128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:44.755722+0000","scrub_duration":0.0035582639999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":92,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.773936+0000","last_change":"2024-09-11T08:10:37.891841+0000","last_active":"2024-09-11T08:11:39.773936+0000","last_peered":"2024-09-11T08:11:39.773936+0000","last_clean":"2024-09-11T08:11:39.773936+0000","last_became_active":"2024-09-11T08:07:36.077602+0000","last_became_peered":"2024-09-11T08:07:36.077602+0000","last_unstale":"2024-09-11T08:11:39.773936+0000","last_undegraded":"2024-09-11T08:11:39.773936+0000","last_fullsized":"2024-09-11T08:11:39.773936+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:37.891755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:04.782577+0000","last_clean_scrub_stamp":"2024-09-11T08:10:37.891755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:43.342024+0000","scrub_duration":0.0069411969999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":93,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.761328+0000","last_change":"2024-09-11T08:11:38.063359+0000","last_active":"2024-09-11T08:11:39.761328+0000","last_peered":"2024-09-11T08:11:39.761328+0000","last_clean":"2024-09-11T08:11:39.761328+0000","last_became_active":"2024-09-11T08:07:36.633338+0000","last_became_peered":"2024-09-11T08:07:36.633338+0000","last_unstale":"2024-09-11T08:11:39.761328+0000","last_undegraded":"2024-09-11T08:11:39.761328+0000","last_fullsized":"2024-09-11T08:11:39.761328+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:38.063310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:11:38.063310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:44.770226+0000","scrub_duration":0.0013449110000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":89,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.774102+0000","last_change":"2024-09-11T08:09:11.966738+0000","last_active":"2024-09-11T08:11:39.774102+0000","last_peered":"2024-09-11T08:11:39.774102+0000","last_clean":"2024-09-11T08:11:39.774102+0000","last_became_active":"2024-09-11T08:07:30.045739+0000","last_became_peered":"2024-09-11T08:07:30.045739+0000","last_unstale":"2024-09-11T08:11:39.774102+0000","last_undegraded":"2024-09-11T08:11:39.774102+0000","last_fullsized":"2024-09-11T08:11:39.774102+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:11.966683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:09:11.966683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015807659999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"107'22","reported_seq":128,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.767078+0000","last_change":"2024-09-11T08:10:30.941599+0000","last_active":"2024-09-11T08:11:39.767078+0000","last_peered":"2024-09-11T08:11:39.767078+0000","last_clean":"2024-09-11T08:11:39.767078+0000","last_became_active":"2024-09-11T08:07:32.046363+0000","last_became_peered":"2024-09-11T08:07:32.046363+0000","last_unstale":"2024-09-11T08:11:39.767078+0000","last_undegraded":"2024-09-11T08:11:39.767078+0000","last_fullsized":"2024-09-11T08:11:39.767078+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'22","last_scrub_stamp":"2024-09-11T08:10:30.941505+0000","last_deep_scrub":"107'22","last_deep_scrub_stamp":"2024-09-11T08:10:30.941505+0000","last_clean_scrub_stamp":"2024-09-11T08:10:30.941505+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:53.254453+0000","scrub_duration":0.02240839,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":95,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.775325+0000","last_change":"2024-09-11T08:10:31.418953+0000","last_active":"2024-09-11T08:11:39.775325+0000","last_peered":"2024-09-11T08:11:39.775325+0000","last_clean":"2024-09-11T08:11:39.775325+0000","last_became_active":"2024-09-11T08:07:34.055410+0000","last_became_peered":"2024-09-11T08:07:34.055410+0000","last_unstale":"2024-09-11T08:11:39.775325+0000","last_undegraded":"2024-09-11T08:11:39.775325+0000","last_fullsized":"2024-09-11T08:11:39.775325+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:31.418873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:10:31.418873+0000","last_clean_scrub_stamp":"2024-09-11T08:10:31.418873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011564522000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":83,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.769600+0000","last_change":"2024-09-11T08:10:31.188608+0000","last_active":"2024-09-11T08:11:39.769600+0000","last_peered":"2024-09-11T08:11:39.769600+0000","last_clean":"2024-09-11T08:11:39.769600+0000","last_became_active":"2024-09-11T08:07:36.633258+0000","last_became_peered":"2024-09-11T08:07:36.633258+0000","last_unstale":"2024-09-11T08:11:39.769600+0000","last_undegraded":"2024-09-11T08:11:39.769600+0000","last_fullsized":"2024-09-11T08:11:39.769600+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:31.188518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:10:31.188518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:00.813319+0000","scrub_duration":0.013170371,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":99,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.761031+0000","last_change":"2024-09-11T08:11:17.089124+0000","last_active":"2024-09-11T08:11:39.761031+0000","last_peered":"2024-09-11T08:11:39.761031+0000","last_clean":"2024-09-11T08:11:39.761031+0000","last_became_active":"2024-09-11T08:07:30.046284+0000","last_became_peered":"2024-09-11T08:07:30.046284+0000","last_unstale":"2024-09-11T08:11:39.761031+0000","last_undegraded":"2024-09-11T08:11:39.761031+0000","last_fullsized":"2024-09-11T08:11:39.761031+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:17.089046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:11:17.089046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:21.125842+0000","scrub_duration":0.0042934460000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":142,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.763088+0000","last_change":"2024-09-11T08:11:32.559837+0000","last_active":"2024-09-11T08:11:39.763088+0000","last_peered":"2024-09-11T08:11:39.763088+0000","last_clean":"2024-09-11T08:11:39.763088+0000","last_became_active":"2024-09-11T08:09:07.878432+0000","last_became_peered":"2024-09-11T08:09:07.878432+0000","last_unstale":"2024-09-11T08:11:39.763088+0000","last_undegraded":"2024-09-11T08:11:39.763088+0000","last_fullsized":"2024-09-11T08:11:39.763088+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-11T08:11:32.559751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:11:32.559751+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:48.508018+0000","scrub_duration":0.0059216119999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":95,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.761170+0000","last_change":"2024-09-11T08:10:40.110540+0000","last_active":"2024-09-11T08:11:39.761170+0000","last_peered":"2024-09-11T08:11:39.761170+0000","last_clean":"2024-09-11T08:11:39.761170+0000","last_became_active":"2024-09-11T08:07:34.056528+0000","last_became_peered":"2024-09-11T08:07:34.056528+0000","last_unstale":"2024-09-11T08:11:39.761170+0000","last_undegraded":"2024-09-11T08:11:39.761170+0000","last_fullsized":"2024-09-11T08:11:39.761170+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:40.110490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:10:40.110490+0000","last_clean_scrub_stamp":"2024-09-11T08:10:40.110490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:53.563251+0000","scrub_duration":0.00164006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":94,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.769364+0000","last_change":"2024-09-11T08:10:41.305921+0000","last_active":"2024-09-11T08:11:39.769364+0000","last_peered":"2024-09-11T08:11:39.769364+0000","last_clean":"2024-09-11T08:11:39.769364+0000","last_became_active":"2024-09-11T08:07:36.072347+0000","last_became_peered":"2024-09-11T08:07:36.072347+0000","last_unstale":"2024-09-11T08:11:39.769364+0000","last_undegraded":"2024-09-11T08:11:39.769364+0000","last_fullsized":"2024-09-11T08:11:39.769364+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-11T08:10:41.305866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:10:41.305866+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:57.336810+0000","scrub_duration":0.0045172119999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":99,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.778827+0000","last_change":"2024-09-11T08:10:35.345635+0000","last_active":"2024-09-11T08:11:39.778827+0000","last_peered":"2024-09-11T08:11:39.778827+0000","last_clean":"2024-09-11T08:11:39.778827+0000","last_became_active":"2024-09-11T08:07:30.045932+0000","last_became_peered":"2024-09-11T08:07:30.045932+0000","last_unstale":"2024-09-11T08:11:39.778827+0000","last_undegraded":"2024-09-11T08:11:39.778827+0000","last_fullsized":"2024-09-11T08:11:39.778827+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:35.345484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:10:35.345484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:59.513234+0000","scrub_duration":0.010438901,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":112,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.776803+0000","last_change":"2024-09-11T08:11:30.913167+0000","last_active":"2024-09-11T08:11:39.776803+0000","last_peered":"2024-09-11T08:11:39.776803+0000","last_clean":"2024-09-11T08:11:39.776803+0000","last_became_active":"2024-09-11T08:07:32.048022+0000","last_became_peered":"2024-09-11T08:07:32.048022+0000","last_unstale":"2024-09-11T08:11:39.776803+0000","last_undegraded":"2024-09-11T08:11:39.776803+0000","last_fullsized":"2024-09-11T08:11:39.776803+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-11T08:11:30.913083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:11:30.913083+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:56.240332+0000","scrub_duration":0.0071376130000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":166,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.424127+0000","last_change":"2024-09-11T08:11:28.666314+0000","last_active":"2024-09-11T08:11:42.424127+0000","last_peered":"2024-09-11T08:11:42.424127+0000","last_clean":"2024-09-11T08:11:42.424127+0000","last_became_active":"2024-09-11T08:07:34.054014+0000","last_became_peered":"2024-09-11T08:07:34.054014+0000","last_unstale":"2024-09-11T08:11:42.424127+0000","last_undegraded":"2024-09-11T08:11:42.424127+0000","last_fullsized":"2024-09-11T08:11:42.424127+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:11:28.666260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:11:28.666260+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:35.716682+0000","scrub_duration":0.001258885,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"129'8971","reported_seq":9333,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:47.668942+0000","last_change":"2024-09-11T08:11:28.767321+0000","last_active":"2024-09-11T08:11:47.668942+0000","last_peered":"2024-09-11T08:11:47.668942+0000","last_clean":"2024-09-11T08:11:47.668942+0000","last_became_active":"2024-09-11T08:09:45.414744+0000","last_became_peered":"2024-09-11T08:09:45.414744+0000","last_unstale":"2024-09-11T08:11:47.668942+0000","last_undegraded":"2024-09-11T08:11:47.668942+0000","last_fullsized":"2024-09-11T08:11:47.668942+0000","mapping_epoch":91,"log_start":"102'2700","ondisk_log_start":"102'2700","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'7721","last_scrub_stamp":"2024-09-11T08:11:28.767232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_clean_scrub_stamp":"2024-09-11T08:11:28.767232+0000","objects_scrubbed":7645,"log_size":6271,"log_dups_size":0,"ondisk_log_size":6271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:48.988510+0000","scrub_duration":2.190530367,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":418971648,"num_objects":6393,"num_object_clones":0,"num_object_copies":19179,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8971,"num_write_kb":491648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":122,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.773194+0000","last_change":"2024-09-11T08:10:23.428879+0000","last_active":"2024-09-11T08:11:39.773194+0000","last_peered":"2024-09-11T08:11:39.773194+0000","last_clean":"2024-09-11T08:11:39.773194+0000","last_became_active":"2024-09-11T08:07:32.054317+0000","last_became_peered":"2024-09-11T08:07:32.054317+0000","last_unstale":"2024-09-11T08:11:39.773194+0000","last_undegraded":"2024-09-11T08:11:39.773194+0000","last_fullsized":"2024-09-11T08:11:39.773194+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-11T08:10:23.428796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:10:23.428796+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014719733,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":89,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.768046+0000","last_change":"2024-09-11T08:10:16.250853+0000","last_active":"2024-09-11T08:11:39.768046+0000","last_peered":"2024-09-11T08:11:39.768046+0000","last_clean":"2024-09-11T08:11:39.768046+0000","last_became_active":"2024-09-11T08:07:30.044624+0000","last_became_peered":"2024-09-11T08:07:30.044624+0000","last_unstale":"2024-09-11T08:11:39.768046+0000","last_undegraded":"2024-09-11T08:11:39.768046+0000","last_fullsized":"2024-09-11T08:11:39.768046+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:16.247162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:10:16.247162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065916309999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":154,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.420251+0000","last_change":"2024-09-11T08:11:27.139877+0000","last_active":"2024-09-11T08:11:42.420251+0000","last_peered":"2024-09-11T08:11:42.420251+0000","last_clean":"2024-09-11T08:11:42.420251+0000","last_became_active":"2024-09-11T08:07:34.052981+0000","last_became_peered":"2024-09-11T08:07:34.052981+0000","last_unstale":"2024-09-11T08:11:42.420251+0000","last_undegraded":"2024-09-11T08:11:42.420251+0000","last_fullsized":"2024-09-11T08:11:42.420251+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:11:27.139796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:11:27.139796+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:48.327854+0000","scrub_duration":0.0080431200000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":102,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.774229+0000","last_change":"2024-09-11T08:11:39.747151+0000","last_active":"2024-09-11T08:11:39.774229+0000","last_peered":"2024-09-11T08:11:39.774229+0000","last_clean":"2024-09-11T08:11:39.774229+0000","last_became_active":"2024-09-11T08:07:36.073969+0000","last_became_peered":"2024-09-11T08:07:36.073969+0000","last_unstale":"2024-09-11T08:11:39.774229+0000","last_undegraded":"2024-09-11T08:11:39.774229+0000","last_fullsized":"2024-09-11T08:11:39.774229+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:39.747073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:11:39.747073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:42.671932+0000","scrub_duration":0.0080670670000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"129'8780","reported_seq":8836,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.738162+0000","last_change":"2024-09-11T08:11:42.436912+0000","last_active":"2024-09-11T08:11:42.738162+0000","last_peered":"2024-09-11T08:11:42.738162+0000","last_clean":"2024-09-11T08:11:42.738162+0000","last_became_active":"2024-09-11T08:09:45.412326+0000","last_became_peered":"2024-09-11T08:09:45.412326+0000","last_unstale":"2024-09-11T08:11:42.738162+0000","last_undegraded":"2024-09-11T08:11:42.738162+0000","last_fullsized":"2024-09-11T08:11:42.738162+0000","mapping_epoch":91,"log_start":"107'4100","ondisk_log_start":"107'4100","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_clean_scrub_stamp":"2024-09-11T08:09:44.385805+0000","objects_scrubbed":99,"log_size":4680,"log_dups_size":0,"ondisk_log_size":4680,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":450363392,"num_objects":6872,"num_object_clones":0,"num_object_copies":20616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8780,"num_write_kb":500864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":129,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.767851+0000","last_change":"2024-09-11T08:10:35.187425+0000","last_active":"2024-09-11T08:11:39.767851+0000","last_peered":"2024-09-11T08:11:39.767851+0000","last_clean":"2024-09-11T08:11:39.767851+0000","last_became_active":"2024-09-11T08:07:32.045332+0000","last_became_peered":"2024-09-11T08:07:32.045332+0000","last_unstale":"2024-09-11T08:11:39.767851+0000","last_undegraded":"2024-09-11T08:11:39.767851+0000","last_fullsized":"2024-09-11T08:11:39.767851+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-11T08:10:35.187322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:10:35.187322+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:02.558401+0000","scrub_duration":0.016336040999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":89,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.770600+0000","last_change":"2024-09-11T08:10:24.286689+0000","last_active":"2024-09-11T08:11:39.770600+0000","last_peered":"2024-09-11T08:11:39.770600+0000","last_clean":"2024-09-11T08:11:39.770600+0000","last_became_active":"2024-09-11T08:07:30.047176+0000","last_became_peered":"2024-09-11T08:07:30.047176+0000","last_unstale":"2024-09-11T08:11:39.770600+0000","last_undegraded":"2024-09-11T08:11:39.770600+0000","last_fullsized":"2024-09-11T08:11:39.770600+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:24.286622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:10:24.286622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.022756537,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":95,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.763780+0000","last_change":"2024-09-11T08:11:26.181015+0000","last_active":"2024-09-11T08:11:39.763780+0000","last_peered":"2024-09-11T08:11:39.763780+0000","last_clean":"2024-09-11T08:11:39.763780+0000","last_became_active":"2024-09-11T08:07:34.055523+0000","last_became_peered":"2024-09-11T08:07:34.055523+0000","last_unstale":"2024-09-11T08:11:39.763780+0000","last_undegraded":"2024-09-11T08:11:39.763780+0000","last_fullsized":"2024-09-11T08:11:39.763780+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:26.180933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:11:26.180933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:31.314789+0000","scrub_duration":0.0022304540000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":83,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.760899+0000","last_change":"2024-09-11T08:09:21.618824+0000","last_active":"2024-09-11T08:11:39.760899+0000","last_peered":"2024-09-11T08:11:39.760899+0000","last_clean":"2024-09-11T08:11:39.760899+0000","last_became_active":"2024-09-11T08:07:36.069225+0000","last_became_peered":"2024-09-11T08:07:36.069225+0000","last_unstale":"2024-09-11T08:11:39.760899+0000","last_undegraded":"2024-09-11T08:11:39.760899+0000","last_fullsized":"2024-09-11T08:11:39.760899+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:21.618769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:09:21.618769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001637814,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"129'8640","reported_seq":8758,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.732258+0000","last_change":"2024-09-11T08:11:42.431777+0000","last_active":"2024-09-11T08:11:42.732258+0000","last_peered":"2024-09-11T08:11:42.732258+0000","last_clean":"2024-09-11T08:11:42.732258+0000","last_became_active":"2024-09-11T08:09:45.414656+0000","last_became_peered":"2024-09-11T08:09:45.414656+0000","last_unstale":"2024-09-11T08:11:42.732258+0000","last_undegraded":"2024-09-11T08:11:42.732258+0000","last_fullsized":"2024-09-11T08:11:42.732258+0000","mapping_epoch":91,"log_start":"107'4000","ondisk_log_start":"107'4000","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_clean_scrub_stamp":"2024-09-11T08:09:44.385805+0000","objects_scrubbed":1649,"log_size":4640,"log_dups_size":0,"ondisk_log_size":4640,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":438697984,"num_objects":6694,"num_object_clones":0,"num_object_copies":20082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8640,"num_write_kb":490688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":108,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.764028+0000","last_change":"2024-09-11T08:09:11.688215+0000","last_active":"2024-09-11T08:11:39.764028+0000","last_peered":"2024-09-11T08:11:39.764028+0000","last_clean":"2024-09-11T08:11:39.764028+0000","last_became_active":"2024-09-11T08:07:32.040069+0000","last_became_peered":"2024-09-11T08:07:32.040069+0000","last_unstale":"2024-09-11T08:11:39.764028+0000","last_undegraded":"2024-09-11T08:11:39.764028+0000","last_fullsized":"2024-09-11T08:11:39.764028+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-11T08:09:11.688151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:09:11.688151+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019655670000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":99,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.777004+0000","last_change":"2024-09-11T08:11:33.964196+0000","last_active":"2024-09-11T08:11:39.777004+0000","last_peered":"2024-09-11T08:11:39.777004+0000","last_clean":"2024-09-11T08:11:39.777004+0000","last_became_active":"2024-09-11T08:07:30.050484+0000","last_became_peered":"2024-09-11T08:07:30.050484+0000","last_unstale":"2024-09-11T08:11:39.777004+0000","last_undegraded":"2024-09-11T08:11:39.777004+0000","last_fullsized":"2024-09-11T08:11:39.777004+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:33.964053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:11:33.964053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:40.373468+0000","scrub_duration":0.003641151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"82'9","reported_seq":157,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.762475+0000","last_change":"2024-09-11T08:11:25.434358+0000","last_active":"2024-09-11T08:11:39.762475+0000","last_peered":"2024-09-11T08:11:39.762475+0000","last_clean":"2024-09-11T08:11:39.762475+0000","last_became_active":"2024-09-11T08:07:34.056143+0000","last_became_peered":"2024-09-11T08:07:34.056143+0000","last_unstale":"2024-09-11T08:11:39.762475+0000","last_undegraded":"2024-09-11T08:11:39.762475+0000","last_fullsized":"2024-09-11T08:11:39.762475+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-11T08:11:25.434210+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-09-11T08:08:53.421587+0000","last_clean_scrub_stamp":"2024-09-11T08:11:25.434210+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:46.551085+0000","scrub_duration":0.0043836509999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":93,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.767530+0000","last_change":"2024-09-11T08:10:29.251333+0000","last_active":"2024-09-11T08:11:39.767530+0000","last_peered":"2024-09-11T08:11:39.767530+0000","last_clean":"2024-09-11T08:11:39.767530+0000","last_became_active":"2024-09-11T08:07:36.633432+0000","last_became_peered":"2024-09-11T08:07:36.633432+0000","last_unstale":"2024-09-11T08:11:39.767530+0000","last_undegraded":"2024-09-11T08:11:39.767530+0000","last_fullsized":"2024-09-11T08:11:39.767530+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:29.251257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:10:29.251257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023785080000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"129'9132","reported_seq":9178,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:47.697353+0000","last_change":"2024-09-11T08:09:45.410430+0000","last_active":"2024-09-11T08:11:47.697353+0000","last_peered":"2024-09-11T08:11:47.697353+0000","last_clean":"2024-09-11T08:11:47.697353+0000","last_became_active":"2024-09-11T08:09:45.410129+0000","last_became_peered":"2024-09-11T08:09:45.410129+0000","last_unstale":"2024-09-11T08:11:47.697353+0000","last_undegraded":"2024-09-11T08:11:47.697353+0000","last_fullsized":"2024-09-11T08:11:47.697353+0000","mapping_epoch":91,"log_start":"108'4300","ondisk_log_start":"108'4300","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_clean_scrub_stamp":"2024-09-11T08:09:44.385805+0000","objects_scrubbed":0,"log_size":4832,"log_dups_size":0,"ondisk_log_size":4832,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":427425792,"num_objects":6522,"num_object_clones":0,"num_object_copies":19566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9132,"num_write_kb":500928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":112,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.773054+0000","last_change":"2024-09-11T08:11:22.581262+0000","last_active":"2024-09-11T08:11:39.773054+0000","last_peered":"2024-09-11T08:11:39.773054+0000","last_clean":"2024-09-11T08:11:39.773054+0000","last_became_active":"2024-09-11T08:07:32.054294+0000","last_became_peered":"2024-09-11T08:07:32.054294+0000","last_unstale":"2024-09-11T08:11:39.773054+0000","last_undegraded":"2024-09-11T08:11:39.773054+0000","last_fullsized":"2024-09-11T08:11:39.773054+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-11T08:11:22.581035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:11:22.581035+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:43.316970+0000","scrub_duration":0.0076992600000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":98,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.774536+0000","last_change":"2024-09-11T08:10:35.913323+0000","last_active":"2024-09-11T08:11:39.774536+0000","last_peered":"2024-09-11T08:11:39.774536+0000","last_clean":"2024-09-11T08:11:39.774536+0000","last_became_active":"2024-09-11T08:07:30.047497+0000","last_became_peered":"2024-09-11T08:07:30.047497+0000","last_unstale":"2024-09-11T08:11:39.774536+0000","last_undegraded":"2024-09-11T08:11:39.774536+0000","last_fullsized":"2024-09-11T08:11:39.774536+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:35.913239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:10:35.913239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:00.244681+0000","scrub_duration":0.007393714,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":153,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.773820+0000","last_change":"2024-09-11T08:11:38.925253+0000","last_active":"2024-09-11T08:11:39.773820+0000","last_peered":"2024-09-11T08:11:39.773820+0000","last_clean":"2024-09-11T08:11:39.773820+0000","last_became_active":"2024-09-11T08:07:34.060660+0000","last_became_peered":"2024-09-11T08:07:34.060660+0000","last_unstale":"2024-09-11T08:11:39.773820+0000","last_undegraded":"2024-09-11T08:11:39.773820+0000","last_fullsized":"2024-09-11T08:11:39.773820+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:11:38.925167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:11:38.925167+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:44.287481+0000","scrub_duration":0.0080128060000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"66'1","reported_seq":84,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.768770+0000","last_change":"2024-09-11T08:09:25.767659+0000","last_active":"2024-09-11T08:11:39.768770+0000","last_peered":"2024-09-11T08:11:39.768770+0000","last_clean":"2024-09-11T08:11:39.768770+0000","last_became_active":"2024-09-11T08:07:36.072593+0000","last_became_peered":"2024-09-11T08:07:36.072593+0000","last_unstale":"2024-09-11T08:11:39.768770+0000","last_undegraded":"2024-09-11T08:11:39.768770+0000","last_fullsized":"2024-09-11T08:11:39.768770+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-11T08:09:25.767603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:09:25.767603+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001492487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"129'8874","reported_seq":9218,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:47.685658+0000","last_change":"2024-09-11T08:11:22.612314+0000","last_active":"2024-09-11T08:11:47.685658+0000","last_peered":"2024-09-11T08:11:47.685658+0000","last_clean":"2024-09-11T08:11:47.685658+0000","last_became_active":"2024-09-11T08:09:45.415564+0000","last_became_peered":"2024-09-11T08:09:45.415564+0000","last_unstale":"2024-09-11T08:11:47.685658+0000","last_undegraded":"2024-09-11T08:11:47.685658+0000","last_fullsized":"2024-09-11T08:11:47.685658+0000","mapping_epoch":91,"log_start":"103'3100","ondisk_log_start":"103'3100","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'7301","last_scrub_stamp":"2024-09-11T08:11:22.612230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_clean_scrub_stamp":"2024-09-11T08:11:22.612230+0000","objects_scrubbed":7217,"log_size":5774,"log_dups_size":0,"ondisk_log_size":5774,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:51.523019+0000","scrub_duration":2.8618551820000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":419823616,"num_objects":6406,"num_object_clones":0,"num_object_copies":19218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8874,"num_write_kb":488960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":133,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.773965+0000","last_change":"2024-09-11T08:11:19.982267+0000","last_active":"2024-09-11T08:11:39.773965+0000","last_peered":"2024-09-11T08:11:39.773965+0000","last_clean":"2024-09-11T08:11:39.773965+0000","last_became_active":"2024-09-11T08:07:32.044376+0000","last_became_peered":"2024-09-11T08:07:32.044376+0000","last_unstale":"2024-09-11T08:11:39.773965+0000","last_undegraded":"2024-09-11T08:11:39.773965+0000","last_fullsized":"2024-09-11T08:11:39.773965+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-11T08:11:19.982185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:11:19.982185+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:38.948134+0000","scrub_duration":0.0066179189999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":99,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.775896+0000","last_change":"2024-09-11T08:10:22.436344+0000","last_active":"2024-09-11T08:11:39.775896+0000","last_peered":"2024-09-11T08:11:39.775896+0000","last_clean":"2024-09-11T08:11:39.775896+0000","last_became_active":"2024-09-11T08:07:30.045380+0000","last_became_peered":"2024-09-11T08:07:30.045380+0000","last_unstale":"2024-09-11T08:11:39.775896+0000","last_undegraded":"2024-09-11T08:11:39.775896+0000","last_fullsized":"2024-09-11T08:11:39.775896+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:22.436250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:10:22.436250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0076618179999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":94,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.763541+0000","last_change":"2024-09-11T08:10:26.621521+0000","last_active":"2024-09-11T08:11:39.763541+0000","last_peered":"2024-09-11T08:11:39.763541+0000","last_clean":"2024-09-11T08:11:39.763541+0000","last_became_active":"2024-09-11T08:07:34.051004+0000","last_became_peered":"2024-09-11T08:07:34.051004+0000","last_unstale":"2024-09-11T08:11:39.763541+0000","last_undegraded":"2024-09-11T08:11:39.763541+0000","last_fullsized":"2024-09-11T08:11:39.763541+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:26.621460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:10:26.621460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016671699999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":93,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.778635+0000","last_change":"2024-09-11T08:11:36.477823+0000","last_active":"2024-09-11T08:11:39.778635+0000","last_peered":"2024-09-11T08:11:39.778635+0000","last_clean":"2024-09-11T08:11:39.778635+0000","last_became_active":"2024-09-11T08:07:36.631668+0000","last_became_peered":"2024-09-11T08:07:36.631668+0000","last_unstale":"2024-09-11T08:11:39.778635+0000","last_undegraded":"2024-09-11T08:11:39.778635+0000","last_fullsized":"2024-09-11T08:11:39.778635+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:36.477734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:11:36.477734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:13:01.973259+0000","scrub_duration":0.0067568719999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"129'9148","reported_seq":9511,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:47.679652+0000","last_change":"2024-09-11T08:11:27.447825+0000","last_active":"2024-09-11T08:11:47.679652+0000","last_peered":"2024-09-11T08:11:47.679652+0000","last_clean":"2024-09-11T08:11:47.679652+0000","last_became_active":"2024-09-11T08:09:45.414847+0000","last_became_peered":"2024-09-11T08:09:45.414847+0000","last_unstale":"2024-09-11T08:11:47.679652+0000","last_undegraded":"2024-09-11T08:11:47.679652+0000","last_fullsized":"2024-09-11T08:11:47.679652+0000","mapping_epoch":91,"log_start":"102'2800","ondisk_log_start":"102'2800","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'7788","last_scrub_stamp":"2024-09-11T08:11:27.447746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_clean_scrub_stamp":"2024-09-11T08:11:27.447746+0000","objects_scrubbed":7674,"log_size":6348,"log_dups_size":0,"ondisk_log_size":6348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:47.727446+0000","scrub_duration":3.9345051240000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":425197568,"num_objects":6488,"num_object_clones":0,"num_object_copies":19464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9148,"num_write_kb":500352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"86'38","reported_seq":205,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.775766+0000","last_change":"2024-09-11T08:11:16.652708+0000","last_active":"2024-09-11T08:11:39.775766+0000","last_peered":"2024-09-11T08:11:39.775766+0000","last_clean":"2024-09-11T08:11:39.775766+0000","last_became_active":"2024-09-11T08:07:32.052956+0000","last_became_peered":"2024-09-11T08:07:32.052956+0000","last_unstale":"2024-09-11T08:11:39.775766+0000","last_undegraded":"2024-09-11T08:11:39.775766+0000","last_fullsized":"2024-09-11T08:11:39.775766+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'38","last_scrub_stamp":"2024-09-11T08:11:16.652273+0000","last_deep_scrub":"86'38","last_deep_scrub_stamp":"2024-09-11T08:09:11.362339+0000","last_clean_scrub_stamp":"2024-09-11T08:11:16.652273+0000","objects_scrubbed":10,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:46.242558+0000","scrub_duration":0.011886363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"59'1","reported_seq":108,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.761145+0000","last_change":"2024-09-11T08:10:18.051791+0000","last_active":"2024-09-11T08:11:39.761145+0000","last_peered":"2024-09-11T08:11:39.761145+0000","last_clean":"2024-09-11T08:11:39.761145+0000","last_became_active":"2024-09-11T08:07:30.047513+0000","last_became_peered":"2024-09-11T08:07:30.047513+0000","last_unstale":"2024-09-11T08:11:39.761145+0000","last_undegraded":"2024-09-11T08:11:39.761145+0000","last_fullsized":"2024-09-11T08:11:39.761145+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-11T08:10:18.051701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:10:18.051701+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028800570000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":105,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.774303+0000","last_change":"2024-09-11T08:10:16.989515+0000","last_active":"2024-09-11T08:11:39.774303+0000","last_peered":"2024-09-11T08:11:39.774303+0000","last_clean":"2024-09-11T08:11:39.774303+0000","last_became_active":"2024-09-11T08:07:30.043130+0000","last_became_peered":"2024-09-11T08:07:30.043130+0000","last_unstale":"2024-09-11T08:11:39.774303+0000","last_undegraded":"2024-09-11T08:11:39.774303+0000","last_fullsized":"2024-09-11T08:11:39.774303+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-11T08:10:16.989458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:10:16.989458+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:46.618957+0000","scrub_duration":0.0033825330000000001,"objects_trimmed":0,"snaptrim_duration":0.0026120850000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":95,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.777506+0000","last_change":"2024-09-11T08:10:37.957177+0000","last_active":"2024-09-11T08:11:39.777506+0000","last_peered":"2024-09-11T08:11:39.777506+0000","last_clean":"2024-09-11T08:11:39.777506+0000","last_became_active":"2024-09-11T08:07:34.057154+0000","last_became_peered":"2024-09-11T08:07:34.057154+0000","last_unstale":"2024-09-11T08:11:39.777506+0000","last_undegraded":"2024-09-11T08:11:39.777506+0000","last_fullsized":"2024-09-11T08:11:39.777506+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:37.957096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:10:37.957096+0000","last_clean_scrub_stamp":"2024-09-11T08:10:37.957096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:47.854954+0000","scrub_duration":0.011314437,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":93,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.763558+0000","last_change":"2024-09-11T08:11:38.502257+0000","last_active":"2024-09-11T08:11:39.763558+0000","last_peered":"2024-09-11T08:11:39.763558+0000","last_clean":"2024-09-11T08:11:39.763558+0000","last_became_active":"2024-09-11T08:07:36.075690+0000","last_became_peered":"2024-09-11T08:07:36.075690+0000","last_unstale":"2024-09-11T08:11:39.763558+0000","last_undegraded":"2024-09-11T08:11:39.763558+0000","last_fullsized":"2024-09-11T08:11:39.763558+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:38.502182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:11:38.502182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:55.269821+0000","scrub_duration":0.0039179619999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"129'8654","reported_seq":8700,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.857232+0000","last_change":"2024-09-11T08:09:45.414559+0000","last_active":"2024-09-11T08:11:42.857232+0000","last_peered":"2024-09-11T08:11:42.857232+0000","last_clean":"2024-09-11T08:11:42.857232+0000","last_became_active":"2024-09-11T08:09:45.414278+0000","last_became_peered":"2024-09-11T08:09:45.414278+0000","last_unstale":"2024-09-11T08:11:42.857232+0000","last_undegraded":"2024-09-11T08:11:42.857232+0000","last_fullsized":"2024-09-11T08:11:42.857232+0000","mapping_epoch":91,"log_start":"102'3000","ondisk_log_start":"102'3000","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_clean_scrub_stamp":"2024-09-11T08:09:44.385805+0000","objects_scrubbed":0,"log_size":5654,"log_dups_size":0,"ondisk_log_size":5654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":443023360,"num_objects":6760,"num_object_clones":0,"num_object_copies":20280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8654,"num_write_kb":493248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":124,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.773276+0000","last_change":"2024-09-11T08:10:26.486264+0000","last_active":"2024-09-11T08:11:39.773276+0000","last_peered":"2024-09-11T08:11:39.773276+0000","last_clean":"2024-09-11T08:11:39.773276+0000","last_became_active":"2024-09-11T08:07:32.050673+0000","last_became_peered":"2024-09-11T08:07:32.050673+0000","last_unstale":"2024-09-11T08:11:39.773276+0000","last_undegraded":"2024-09-11T08:11:39.773276+0000","last_fullsized":"2024-09-11T08:11:39.773276+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'13","last_scrub_stamp":"2024-09-11T08:10:26.486178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:10:26.486178+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:42.768627+0000","scrub_duration":0.012477777000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":99,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.773550+0000","last_change":"2024-09-11T08:10:25.425751+0000","last_active":"2024-09-11T08:11:39.773550+0000","last_peered":"2024-09-11T08:11:39.773550+0000","last_clean":"2024-09-11T08:11:39.773550+0000","last_became_active":"2024-09-11T08:07:30.047014+0000","last_became_peered":"2024-09-11T08:07:30.047014+0000","last_unstale":"2024-09-11T08:11:39.773550+0000","last_undegraded":"2024-09-11T08:11:39.773550+0000","last_fullsized":"2024-09-11T08:11:39.773550+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:25.425399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:10:25.425399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:54.995561+0000","scrub_duration":0.0032663789999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"127'2","reported_seq":115,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.766620+0000","last_change":"2024-09-11T08:11:32.348728+0000","last_active":"2024-09-11T08:11:39.766620+0000","last_peered":"2024-09-11T08:11:39.766620+0000","last_clean":"2024-09-11T08:11:39.766620+0000","last_became_active":"2024-09-11T08:07:30.045009+0000","last_became_peered":"2024-09-11T08:07:30.045009+0000","last_unstale":"2024-09-11T08:11:39.766620+0000","last_undegraded":"2024-09-11T08:11:39.766620+0000","last_fullsized":"2024-09-11T08:11:39.766620+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'2","last_scrub_stamp":"2024-09-11T08:11:32.348650+0000","last_deep_scrub":"127'2","last_deep_scrub_stamp":"2024-09-11T08:11:32.348650+0000","last_clean_scrub_stamp":"2024-09-11T08:11:32.348650+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:13:00.297188+0000","scrub_duration":0.013905997999999999,"objects_trimmed":0,"snaptrim_duration":0.001859533,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":94,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.767518+0000","last_change":"2024-09-11T08:10:19.998750+0000","last_active":"2024-09-11T08:11:39.767518+0000","last_peered":"2024-09-11T08:11:39.767518+0000","last_clean":"2024-09-11T08:11:39.767518+0000","last_became_active":"2024-09-11T08:07:34.063906+0000","last_became_peered":"2024-09-11T08:07:34.063906+0000","last_unstale":"2024-09-11T08:11:39.767518+0000","last_undegraded":"2024-09-11T08:11:39.767518+0000","last_fullsized":"2024-09-11T08:11:39.767518+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:19.998666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:10:19.998666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:43.208659+0000","scrub_duration":0.020599591,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":93,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.768412+0000","last_change":"2024-09-11T08:10:22.212876+0000","last_active":"2024-09-11T08:11:39.768412+0000","last_peered":"2024-09-11T08:11:39.768412+0000","last_clean":"2024-09-11T08:11:39.768412+0000","last_became_active":"2024-09-11T08:07:36.631480+0000","last_became_peered":"2024-09-11T08:07:36.631480+0000","last_unstale":"2024-09-11T08:11:39.768412+0000","last_undegraded":"2024-09-11T08:11:39.768412+0000","last_fullsized":"2024-09-11T08:11:39.768412+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:22.212811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:10:22.212811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:51.323655+0000","scrub_duration":0.0032158299999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"129'9116","reported_seq":9458,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:47.674480+0000","last_change":"2024-09-11T08:11:38.973718+0000","last_active":"2024-09-11T08:11:47.674480+0000","last_peered":"2024-09-11T08:11:47.674480+0000","last_clean":"2024-09-11T08:11:47.674480+0000","last_became_active":"2024-09-11T08:09:45.411757+0000","last_became_peered":"2024-09-11T08:09:45.411757+0000","last_unstale":"2024-09-11T08:11:47.674480+0000","last_undegraded":"2024-09-11T08:11:47.674480+0000","last_fullsized":"2024-09-11T08:11:47.674480+0000","mapping_epoch":91,"log_start":"106'3900","ondisk_log_start":"106'3900","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'8561","last_scrub_stamp":"2024-09-11T08:11:38.973659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_clean_scrub_stamp":"2024-09-11T08:11:38.973659+0000","objects_scrubbed":7171,"log_size":5216,"log_dups_size":0,"ondisk_log_size":5216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:13:06.422033+0000","scrub_duration":2.7886773580000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":426639360,"num_objects":6510,"num_object_clones":0,"num_object_copies":19530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9116,"num_write_kb":500032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":115,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.776873+0000","last_change":"2024-09-11T08:10:29.105854+0000","last_active":"2024-09-11T08:11:39.776873+0000","last_peered":"2024-09-11T08:11:39.776873+0000","last_clean":"2024-09-11T08:11:39.776873+0000","last_became_active":"2024-09-11T08:07:32.055407+0000","last_became_peered":"2024-09-11T08:07:32.055407+0000","last_unstale":"2024-09-11T08:11:39.776873+0000","last_undegraded":"2024-09-11T08:11:39.776873+0000","last_fullsized":"2024-09-11T08:11:39.776873+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-11T08:10:29.105566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:10:29.105566+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:43.561951+0000","scrub_duration":0.016339228000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"59'1","reported_seq":106,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.773717+0000","last_change":"2024-09-11T08:10:34.991296+0000","last_active":"2024-09-11T08:11:39.773717+0000","last_peered":"2024-09-11T08:11:39.773717+0000","last_clean":"2024-09-11T08:11:39.773717+0000","last_became_active":"2024-09-11T08:07:30.050504+0000","last_became_peered":"2024-09-11T08:07:30.050504+0000","last_unstale":"2024-09-11T08:11:39.773717+0000","last_undegraded":"2024-09-11T08:11:39.773717+0000","last_fullsized":"2024-09-11T08:11:39.773717+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-11T08:10:34.991211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:10:34.991211+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:54.573722+0000","scrub_duration":0.0092067399999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":358,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:46.635482+0000","last_change":"2024-09-11T08:11:22.408697+0000","last_active":"2024-09-11T08:11:46.635482+0000","last_peered":"2024-09-11T08:11:46.635482+0000","last_clean":"2024-09-11T08:11:46.635482+0000","last_became_active":"2024-09-11T08:07:30.045051+0000","last_became_peered":"2024-09-11T08:07:30.045051+0000","last_unstale":"2024-09-11T08:11:46.635482+0000","last_undegraded":"2024-09-11T08:11:46.635482+0000","last_fullsized":"2024-09-11T08:11:46.635482+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-11T08:11:22.408607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:11:22.408607+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:38.586477+0000","scrub_duration":0.0068790099999999996,"objects_trimmed":0,"snaptrim_duration":0.000318821,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":251,"num_read_kb":246,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":94,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.774775+0000","last_change":"2024-09-11T08:10:33.984097+0000","last_active":"2024-09-11T08:11:39.774775+0000","last_peered":"2024-09-11T08:11:39.774775+0000","last_clean":"2024-09-11T08:11:39.774775+0000","last_became_active":"2024-09-11T08:07:34.058962+0000","last_became_peered":"2024-09-11T08:07:34.058962+0000","last_unstale":"2024-09-11T08:11:39.774775+0000","last_undegraded":"2024-09-11T08:11:39.774775+0000","last_fullsized":"2024-09-11T08:11:39.774775+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:33.984021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:10:33.984021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0059758429999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":93,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.775588+0000","last_change":"2024-09-11T08:10:33.381711+0000","last_active":"2024-09-11T08:11:39.775588+0000","last_peered":"2024-09-11T08:11:39.775588+0000","last_clean":"2024-09-11T08:11:39.775588+0000","last_became_active":"2024-09-11T08:07:36.077597+0000","last_became_peered":"2024-09-11T08:07:36.077597+0000","last_unstale":"2024-09-11T08:11:39.775588+0000","last_undegraded":"2024-09-11T08:11:39.775588+0000","last_fullsized":"2024-09-11T08:11:39.775588+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:33.381554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:10:33.381554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:46.117984+0000","scrub_duration":0.014471942999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"129'8817","reported_seq":8863,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.918604+0000","last_change":"2024-09-11T08:09:45.415108+0000","last_active":"2024-09-11T08:11:42.918604+0000","last_peered":"2024-09-11T08:11:42.918604+0000","last_clean":"2024-09-11T08:11:42.918604+0000","last_became_active":"2024-09-11T08:09:45.414851+0000","last_became_peered":"2024-09-11T08:09:45.414851+0000","last_unstale":"2024-09-11T08:11:42.918604+0000","last_undegraded":"2024-09-11T08:11:42.918604+0000","last_fullsized":"2024-09-11T08:11:42.918604+0000","mapping_epoch":91,"log_start":"103'3200","ondisk_log_start":"103'3200","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_clean_scrub_stamp":"2024-09-11T08:09:44.385805+0000","objects_scrubbed":0,"log_size":5617,"log_dups_size":0,"ondisk_log_size":5617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":448069632,"num_objects":6837,"num_object_clones":0,"num_object_copies":20511,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8817,"num_write_kb":500928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"84'21","reported_seq":125,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.764253+0000","last_change":"2024-09-11T08:10:20.637001+0000","last_active":"2024-09-11T08:11:39.764253+0000","last_peered":"2024-09-11T08:11:39.764253+0000","last_clean":"2024-09-11T08:11:39.764253+0000","last_became_active":"2024-09-11T08:07:32.042339+0000","last_became_peered":"2024-09-11T08:07:32.042339+0000","last_unstale":"2024-09-11T08:11:39.764253+0000","last_undegraded":"2024-09-11T08:11:39.764253+0000","last_fullsized":"2024-09-11T08:11:39.764253+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'21","last_scrub_stamp":"2024-09-11T08:10:20.636918+0000","last_deep_scrub":"84'21","last_deep_scrub_stamp":"2024-09-11T08:08:56.705306+0000","last_clean_scrub_stamp":"2024-09-11T08:10:20.636918+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:45.324279+0000","scrub_duration":0.004681826,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":89,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.770859+0000","last_change":"2024-09-11T08:09:39.390240+0000","last_active":"2024-09-11T08:11:39.770859+0000","last_peered":"2024-09-11T08:11:39.770859+0000","last_clean":"2024-09-11T08:11:39.770859+0000","last_became_active":"2024-09-11T08:07:30.047246+0000","last_became_peered":"2024-09-11T08:07:30.047246+0000","last_unstale":"2024-09-11T08:11:39.770859+0000","last_undegraded":"2024-09-11T08:11:39.770859+0000","last_fullsized":"2024-09-11T08:11:39.770859+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:39.390158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:09:39.390158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013263249999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":145,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.767890+0000","last_change":"2024-09-11T08:11:17.419193+0000","last_active":"2024-09-11T08:11:39.767890+0000","last_peered":"2024-09-11T08:11:39.767890+0000","last_clean":"2024-09-11T08:11:39.767890+0000","last_became_active":"2024-09-11T08:07:19.726378+0000","last_became_peered":"2024-09-11T08:07:19.726378+0000","last_unstale":"2024-09-11T08:11:39.767890+0000","last_undegraded":"2024-09-11T08:11:39.767890+0000","last_fullsized":"2024-09-11T08:11:39.767890+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-11T08:11:17.419133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:05:21.231856+0000","last_clean_scrub_stamp":"2024-09-11T08:11:17.419133+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:46.438913+0000","scrub_duration":0.002818194,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":95,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.769469+0000","last_change":"2024-09-11T08:10:39.300321+0000","last_active":"2024-09-11T08:11:39.769469+0000","last_peered":"2024-09-11T08:11:39.769469+0000","last_clean":"2024-09-11T08:11:39.769469+0000","last_became_active":"2024-09-11T08:07:34.058414+0000","last_became_peered":"2024-09-11T08:07:34.058414+0000","last_unstale":"2024-09-11T08:11:39.769469+0000","last_undegraded":"2024-09-11T08:11:39.769469+0000","last_fullsized":"2024-09-11T08:11:39.769469+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:39.300265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:51.289872+0000","last_clean_scrub_stamp":"2024-09-11T08:10:39.300265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:45.021730+0000","scrub_duration":0.001629222,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":92,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.774704+0000","last_change":"2024-09-11T08:10:27.995889+0000","last_active":"2024-09-11T08:11:39.774704+0000","last_peered":"2024-09-11T08:11:39.774704+0000","last_clean":"2024-09-11T08:11:39.774704+0000","last_became_active":"2024-09-11T08:07:36.072508+0000","last_became_peered":"2024-09-11T08:07:36.072508+0000","last_unstale":"2024-09-11T08:11:39.774704+0000","last_undegraded":"2024-09-11T08:11:39.774704+0000","last_fullsized":"2024-09-11T08:11:39.774704+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:27.995806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:01.777429+0000","last_clean_scrub_stamp":"2024-09-11T08:10:27.995806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001407127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"129'9066","reported_seq":9421,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:47.655422+0000","last_change":"2024-09-11T08:11:24.558667+0000","last_active":"2024-09-11T08:11:47.655422+0000","last_peered":"2024-09-11T08:11:47.655422+0000","last_clean":"2024-09-11T08:11:47.655422+0000","last_became_active":"2024-09-11T08:09:45.412638+0000","last_became_peered":"2024-09-11T08:09:45.412638+0000","last_unstale":"2024-09-11T08:11:47.655422+0000","last_undegraded":"2024-09-11T08:11:47.655422+0000","last_fullsized":"2024-09-11T08:11:47.655422+0000","mapping_epoch":91,"log_start":"102'2800","ondisk_log_start":"102'2800","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'7543","last_scrub_stamp":"2024-09-11T08:11:24.558579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_clean_scrub_stamp":"2024-09-11T08:11:24.558579+0000","objects_scrubbed":7445,"log_size":6266,"log_dups_size":0,"ondisk_log_size":6266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:27.715583+0000","scrub_duration":3.1005590700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420806680,"num_objects":6422,"num_object_clones":0,"num_object_copies":19266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6422,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9066,"num_write_kb":495553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"88'16","reported_seq":121,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.761831+0000","last_change":"2024-09-11T08:09:39.636044+0000","last_active":"2024-09-11T08:11:39.761831+0000","last_peered":"2024-09-11T08:11:39.761831+0000","last_clean":"2024-09-11T08:11:39.761831+0000","last_became_active":"2024-09-11T08:07:32.047971+0000","last_became_peered":"2024-09-11T08:07:32.047971+0000","last_unstale":"2024-09-11T08:11:39.761831+0000","last_undegraded":"2024-09-11T08:11:39.761831+0000","last_fullsized":"2024-09-11T08:11:39.761831+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'16","last_scrub_stamp":"2024-09-11T08:09:39.635984+0000","last_deep_scrub":"88'16","last_deep_scrub_stamp":"2024-09-11T08:09:39.635984+0000","last_clean_scrub_stamp":"2024-09-11T08:09:39.635984+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038085599999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":99,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.764140+0000","last_change":"2024-09-11T08:10:26.069693+0000","last_active":"2024-09-11T08:11:39.764140+0000","last_peered":"2024-09-11T08:11:39.764140+0000","last_clean":"2024-09-11T08:11:39.764140+0000","last_became_active":"2024-09-11T08:07:30.039884+0000","last_became_peered":"2024-09-11T08:07:30.039884+0000","last_unstale":"2024-09-11T08:11:39.764140+0000","last_undegraded":"2024-09-11T08:11:39.764140+0000","last_fullsized":"2024-09-11T08:11:39.764140+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:26.069640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:36.417911+0000","last_clean_scrub_stamp":"2024-09-11T08:10:26.069640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001453558,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":94,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.774635+0000","last_change":"2024-09-11T08:10:31.929612+0000","last_active":"2024-09-11T08:11:39.774635+0000","last_peered":"2024-09-11T08:11:39.774635+0000","last_clean":"2024-09-11T08:11:39.774635+0000","last_became_active":"2024-09-11T08:07:34.053347+0000","last_became_peered":"2024-09-11T08:07:34.053347+0000","last_unstale":"2024-09-11T08:11:39.774635+0000","last_undegraded":"2024-09-11T08:11:39.774635+0000","last_fullsized":"2024-09-11T08:11:39.774635+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:31.929528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:10:31.929528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:53.171499+0000","scrub_duration":0.0015531010000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":93,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.769940+0000","last_change":"2024-09-11T08:11:37.480260+0000","last_active":"2024-09-11T08:11:39.769940+0000","last_peered":"2024-09-11T08:11:39.769940+0000","last_clean":"2024-09-11T08:11:39.769940+0000","last_became_active":"2024-09-11T08:07:36.631687+0000","last_became_peered":"2024-09-11T08:07:36.631687+0000","last_unstale":"2024-09-11T08:11:39.769940+0000","last_undegraded":"2024-09-11T08:11:39.769940+0000","last_fullsized":"2024-09-11T08:11:39.769940+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:37.480123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:11:37.480123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:48.786633+0000","scrub_duration":0.0064260089999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"129'9016","reported_seq":9366,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:47.711033+0000","last_change":"2024-09-11T08:11:22.474726+0000","last_active":"2024-09-11T08:11:47.711033+0000","last_peered":"2024-09-11T08:11:47.711033+0000","last_clean":"2024-09-11T08:11:47.711033+0000","last_became_active":"2024-09-11T08:09:45.413207+0000","last_became_peered":"2024-09-11T08:09:45.413207+0000","last_unstale":"2024-09-11T08:11:47.711033+0000","last_undegraded":"2024-09-11T08:11:47.711033+0000","last_fullsized":"2024-09-11T08:11:47.711033+0000","mapping_epoch":91,"log_start":"98'2100","ondisk_log_start":"98'2100","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'7432","last_scrub_stamp":"2024-09-11T08:11:22.474630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_clean_scrub_stamp":"2024-09-11T08:11:22.474630+0000","objects_scrubbed":7334,"log_size":6916,"log_dups_size":0,"ondisk_log_size":6916,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:23.920712+0000","scrub_duration":4.0826116800000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":426246144,"num_objects":6504,"num_object_clones":0,"num_object_copies":19512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9016,"num_write_kb":496640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"77'25","reported_seq":161,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.761557+0000","last_change":"2024-09-11T08:10:31.126024+0000","last_active":"2024-09-11T08:11:39.761557+0000","last_peered":"2024-09-11T08:11:39.761557+0000","last_clean":"2024-09-11T08:11:39.761557+0000","last_became_active":"2024-09-11T08:09:08.106771+0000","last_became_peered":"2024-09-11T08:09:08.106771+0000","last_unstale":"2024-09-11T08:11:39.761557+0000","last_undegraded":"2024-09-11T08:11:39.761557+0000","last_fullsized":"2024-09-11T08:11:39.761557+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'25","last_scrub_stamp":"2024-09-11T08:10:31.125934+0000","last_deep_scrub":"77'25","last_deep_scrub_stamp":"2024-09-11T08:08:32.458210+0000","last_clean_scrub_stamp":"2024-09-11T08:10:31.125934+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0096263719999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":98,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.763807+0000","last_change":"2024-09-11T08:11:16.505483+0000","last_active":"2024-09-11T08:11:39.763807+0000","last_peered":"2024-09-11T08:11:39.763807+0000","last_clean":"2024-09-11T08:11:39.763807+0000","last_became_active":"2024-09-11T08:07:30.042555+0000","last_became_peered":"2024-09-11T08:07:30.042555+0000","last_unstale":"2024-09-11T08:11:39.763807+0000","last_undegraded":"2024-09-11T08:11:39.763807+0000","last_fullsized":"2024-09-11T08:11:39.763807+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:16.505384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:11:16.505384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:26.959383+0000","scrub_duration":0.010166646999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":95,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.775650+0000","last_change":"2024-09-11T08:10:27.459645+0000","last_active":"2024-09-11T08:11:39.775650+0000","last_peered":"2024-09-11T08:11:39.775650+0000","last_clean":"2024-09-11T08:11:39.775650+0000","last_became_active":"2024-09-11T08:07:34.054961+0000","last_became_peered":"2024-09-11T08:07:34.054961+0000","last_unstale":"2024-09-11T08:11:39.775650+0000","last_undegraded":"2024-09-11T08:11:39.775650+0000","last_fullsized":"2024-09-11T08:11:39.775650+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:27.459436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:10:27.459436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0053113359999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":102,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.767224+0000","last_change":"2024-09-11T08:11:39.370275+0000","last_active":"2024-09-11T08:11:39.767224+0000","last_peered":"2024-09-11T08:11:39.767224+0000","last_clean":"2024-09-11T08:11:39.767224+0000","last_became_active":"2024-09-11T08:07:36.071194+0000","last_became_peered":"2024-09-11T08:07:36.071194+0000","last_unstale":"2024-09-11T08:11:39.767224+0000","last_undegraded":"2024-09-11T08:11:39.767224+0000","last_fullsized":"2024-09-11T08:11:39.767224+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:39.370193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:11:39.370193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:40.590716+0000","scrub_duration":0.014190103000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"129'8696","reported_seq":8741,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.922867+0000","last_change":"2024-09-11T08:09:45.416120+0000","last_active":"2024-09-11T08:11:42.922867+0000","last_peered":"2024-09-11T08:11:42.922867+0000","last_clean":"2024-09-11T08:11:42.922867+0000","last_became_active":"2024-09-11T08:09:45.415738+0000","last_became_peered":"2024-09-11T08:09:45.415738+0000","last_unstale":"2024-09-11T08:11:42.922867+0000","last_undegraded":"2024-09-11T08:11:42.922867+0000","last_fullsized":"2024-09-11T08:11:42.922867+0000","mapping_epoch":91,"log_start":"103'3100","ondisk_log_start":"103'3100","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_clean_scrub_stamp":"2024-09-11T08:09:44.385805+0000","objects_scrubbed":0,"log_size":5596,"log_dups_size":0,"ondisk_log_size":5596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":443285504,"num_objects":6764,"num_object_clones":0,"num_object_copies":20292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8696,"num_write_kb":494720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":112,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.761250+0000","last_change":"2024-09-11T08:10:35.175848+0000","last_active":"2024-09-11T08:11:39.761250+0000","last_peered":"2024-09-11T08:11:39.761250+0000","last_clean":"2024-09-11T08:11:39.761250+0000","last_became_active":"2024-09-11T08:07:32.043314+0000","last_became_peered":"2024-09-11T08:07:32.043314+0000","last_unstale":"2024-09-11T08:11:39.761250+0000","last_undegraded":"2024-09-11T08:11:39.761250+0000","last_fullsized":"2024-09-11T08:11:39.761250+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-11T08:10:35.175750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:10:35.175750+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:00.379714+0000","scrub_duration":0.088039645,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":99,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.770326+0000","last_change":"2024-09-11T08:10:38.126730+0000","last_active":"2024-09-11T08:11:39.770326+0000","last_peered":"2024-09-11T08:11:39.770326+0000","last_clean":"2024-09-11T08:11:39.770326+0000","last_became_active":"2024-09-11T08:07:30.046684+0000","last_became_peered":"2024-09-11T08:07:30.046684+0000","last_unstale":"2024-09-11T08:11:39.770326+0000","last_undegraded":"2024-09-11T08:11:39.770326+0000","last_fullsized":"2024-09-11T08:11:39.770326+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:38.126638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:10:38.126638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:46.945569+0000","scrub_duration":0.0084326729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"66'8","reported_seq":163,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.438703+0000","last_change":"2024-09-11T08:11:24.359888+0000","last_active":"2024-09-11T08:11:42.438703+0000","last_peered":"2024-09-11T08:11:42.438703+0000","last_clean":"2024-09-11T08:11:42.438703+0000","last_became_active":"2024-09-11T08:07:34.062469+0000","last_became_peered":"2024-09-11T08:07:34.062469+0000","last_unstale":"2024-09-11T08:11:42.438703+0000","last_undegraded":"2024-09-11T08:11:42.438703+0000","last_fullsized":"2024-09-11T08:11:42.438703+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:11:24.359780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:11:24.359780+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:31.728208+0000","scrub_duration":0.0054480609999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":83,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.761790+0000","last_change":"2024-09-11T08:09:14.613599+0000","last_active":"2024-09-11T08:11:39.761790+0000","last_peered":"2024-09-11T08:11:39.761790+0000","last_clean":"2024-09-11T08:11:39.761790+0000","last_became_active":"2024-09-11T08:07:36.074684+0000","last_became_peered":"2024-09-11T08:07:36.074684+0000","last_unstale":"2024-09-11T08:11:39.761790+0000","last_undegraded":"2024-09-11T08:11:39.761790+0000","last_fullsized":"2024-09-11T08:11:39.761790+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:14.613532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:14.613532+0000","last_clean_scrub_stamp":"2024-09-11T08:09:14.613532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001503648,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"129'9034","reported_seq":9383,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:47.759786+0000","last_change":"2024-09-11T08:11:22.178487+0000","last_active":"2024-09-11T08:11:47.759786+0000","last_peered":"2024-09-11T08:11:47.759786+0000","last_clean":"2024-09-11T08:11:47.759786+0000","last_became_active":"2024-09-11T08:09:45.411521+0000","last_became_peered":"2024-09-11T08:09:45.411521+0000","last_unstale":"2024-09-11T08:11:47.759786+0000","last_undegraded":"2024-09-11T08:11:47.759786+0000","last_fullsized":"2024-09-11T08:11:47.759786+0000","mapping_epoch":91,"log_start":"99'2200","ondisk_log_start":"99'2200","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'7415","last_scrub_stamp":"2024-09-11T08:11:22.178363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_clean_scrub_stamp":"2024-09-11T08:11:22.178363+0000","objects_scrubbed":7336,"log_size":6834,"log_dups_size":0,"ondisk_log_size":6834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:46.113024+0000","scrub_duration":2.7897035839999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":428867584,"num_objects":6544,"num_object_clones":0,"num_object_copies":19632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9034,"num_write_kb":498496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":113,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:40.062067+0000","last_change":"2024-09-11T08:11:40.062007+0000","last_active":"2024-09-11T08:11:40.062067+0000","last_peered":"2024-09-11T08:11:40.062067+0000","last_clean":"2024-09-11T08:11:40.062067+0000","last_became_active":"2024-09-11T08:07:32.054885+0000","last_became_peered":"2024-09-11T08:07:32.054885+0000","last_unstale":"2024-09-11T08:11:40.062067+0000","last_undegraded":"2024-09-11T08:11:40.062067+0000","last_fullsized":"2024-09-11T08:11:40.062067+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:11:40.061926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:11:40.061926+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:40.906878+0000","scrub_duration":0.106648482,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":108,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.774246+0000","last_change":"2024-09-11T08:11:33.819647+0000","last_active":"2024-09-11T08:11:39.774246+0000","last_peered":"2024-09-11T08:11:39.774246+0000","last_clean":"2024-09-11T08:11:39.774246+0000","last_became_active":"2024-09-11T08:07:30.045613+0000","last_became_peered":"2024-09-11T08:07:30.045613+0000","last_unstale":"2024-09-11T08:11:39.774246+0000","last_undegraded":"2024-09-11T08:11:39.774246+0000","last_fullsized":"2024-09-11T08:11:39.774246+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:33.819593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:11:33.819593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:13:03.296968+0000","scrub_duration":0.0034651399999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":104,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.774432+0000","last_change":"2024-09-11T08:11:33.364733+0000","last_active":"2024-09-11T08:11:39.774432+0000","last_peered":"2024-09-11T08:11:39.774432+0000","last_clean":"2024-09-11T08:11:39.774432+0000","last_became_active":"2024-09-11T08:07:34.056606+0000","last_became_peered":"2024-09-11T08:07:34.056606+0000","last_unstale":"2024-09-11T08:11:39.774432+0000","last_undegraded":"2024-09-11T08:11:39.774432+0000","last_fullsized":"2024-09-11T08:11:39.774432+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:33.364578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:11:33.364578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:34.370854+0000","scrub_duration":0.0099402859999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":93,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.773637+0000","last_change":"2024-09-11T08:11:18.967755+0000","last_active":"2024-09-11T08:11:39.773637+0000","last_peered":"2024-09-11T08:11:39.773637+0000","last_clean":"2024-09-11T08:11:39.773637+0000","last_became_active":"2024-09-11T08:07:36.072452+0000","last_became_peered":"2024-09-11T08:07:36.072452+0000","last_unstale":"2024-09-11T08:11:39.773637+0000","last_undegraded":"2024-09-11T08:11:39.773637+0000","last_fullsized":"2024-09-11T08:11:39.773637+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:18.967665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:11:18.967665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:48.119844+0000","scrub_duration":0.013929113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"129'8891","reported_seq":8937,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.893228+0000","last_change":"2024-09-11T08:09:45.414158+0000","last_active":"2024-09-11T08:11:42.893228+0000","last_peered":"2024-09-11T08:11:42.893228+0000","last_clean":"2024-09-11T08:11:42.893228+0000","last_became_active":"2024-09-11T08:09:45.413883+0000","last_became_peered":"2024-09-11T08:09:45.413883+0000","last_unstale":"2024-09-11T08:11:42.893228+0000","last_undegraded":"2024-09-11T08:11:42.893228+0000","last_fullsized":"2024-09-11T08:11:42.893228+0000","mapping_epoch":91,"log_start":"103'3300","ondisk_log_start":"103'3300","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_clean_scrub_stamp":"2024-09-11T08:09:44.385805+0000","objects_scrubbed":0,"log_size":5591,"log_dups_size":0,"ondisk_log_size":5591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":448987136,"num_objects":6851,"num_object_clones":0,"num_object_copies":20553,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8891,"num_write_kb":503744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":137,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.763879+0000","last_change":"2024-09-11T08:11:29.618794+0000","last_active":"2024-09-11T08:11:39.763879+0000","last_peered":"2024-09-11T08:11:39.763879+0000","last_clean":"2024-09-11T08:11:39.763879+0000","last_became_active":"2024-09-11T08:07:32.050880+0000","last_became_peered":"2024-09-11T08:07:32.050880+0000","last_unstale":"2024-09-11T08:11:39.763879+0000","last_undegraded":"2024-09-11T08:11:39.763879+0000","last_fullsized":"2024-09-11T08:11:39.763879+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-11T08:11:29.618722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:11:29.618722+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:35.510180+0000","scrub_duration":0.0019851790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":99,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.768695+0000","last_change":"2024-09-11T08:10:19.201593+0000","last_active":"2024-09-11T08:11:39.768695+0000","last_peered":"2024-09-11T08:11:39.768695+0000","last_clean":"2024-09-11T08:11:39.768695+0000","last_became_active":"2024-09-11T08:07:30.043294+0000","last_became_peered":"2024-09-11T08:07:30.043294+0000","last_unstale":"2024-09-11T08:11:39.768695+0000","last_undegraded":"2024-09-11T08:11:39.768695+0000","last_fullsized":"2024-09-11T08:11:39.768695+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:19.201434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:44.385372+0000","last_clean_scrub_stamp":"2024-09-11T08:10:19.201434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023250469999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":104,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.766451+0000","last_change":"2024-09-11T08:11:25.397839+0000","last_active":"2024-09-11T08:11:39.766451+0000","last_peered":"2024-09-11T08:11:39.766451+0000","last_clean":"2024-09-11T08:11:39.766451+0000","last_became_active":"2024-09-11T08:07:34.063589+0000","last_became_peered":"2024-09-11T08:07:34.063589+0000","last_unstale":"2024-09-11T08:11:39.766451+0000","last_undegraded":"2024-09-11T08:11:39.766451+0000","last_fullsized":"2024-09-11T08:11:39.766451+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:25.397757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:11:25.397757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:27.261790+0000","scrub_duration":0.00152206,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":93,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.767820+0000","last_change":"2024-09-11T08:10:15.257677+0000","last_active":"2024-09-11T08:11:39.767820+0000","last_peered":"2024-09-11T08:11:39.767820+0000","last_clean":"2024-09-11T08:11:39.767820+0000","last_became_active":"2024-09-11T08:07:36.077523+0000","last_became_peered":"2024-09-11T08:07:36.077523+0000","last_unstale":"2024-09-11T08:11:39.767820+0000","last_undegraded":"2024-09-11T08:11:39.767820+0000","last_fullsized":"2024-09-11T08:11:39.767820+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:15.257622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:10:15.257622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0063147990000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"129'8902","reported_seq":8948,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.794843+0000","last_change":"2024-09-11T08:09:45.411095+0000","last_active":"2024-09-11T08:11:42.794843+0000","last_peered":"2024-09-11T08:11:42.794843+0000","last_clean":"2024-09-11T08:11:42.794843+0000","last_became_active":"2024-09-11T08:09:45.410825+0000","last_became_peered":"2024-09-11T08:09:45.410825+0000","last_unstale":"2024-09-11T08:11:42.794843+0000","last_undegraded":"2024-09-11T08:11:42.794843+0000","last_fullsized":"2024-09-11T08:11:42.794843+0000","mapping_epoch":91,"log_start":"107'4200","ondisk_log_start":"107'4200","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_clean_scrub_stamp":"2024-09-11T08:09:44.385805+0000","objects_scrubbed":0,"log_size":4702,"log_dups_size":0,"ondisk_log_size":4702,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":454033408,"num_objects":6928,"num_object_clones":0,"num_object_copies":20784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8902,"num_write_kb":506560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":120,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.763912+0000","last_change":"2024-09-11T08:10:44.085605+0000","last_active":"2024-09-11T08:11:39.763912+0000","last_peered":"2024-09-11T08:11:39.763912+0000","last_clean":"2024-09-11T08:11:39.763912+0000","last_became_active":"2024-09-11T08:07:32.048164+0000","last_became_peered":"2024-09-11T08:07:32.048164+0000","last_unstale":"2024-09-11T08:11:39.763912+0000","last_undegraded":"2024-09-11T08:11:39.763912+0000","last_fullsized":"2024-09-11T08:11:39.763912+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-11T08:10:44.085517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:10:44.085517+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:13.712893+0000","scrub_duration":0.0074629060000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":99,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.768877+0000","last_change":"2024-09-11T08:10:27.250069+0000","last_active":"2024-09-11T08:11:39.768877+0000","last_peered":"2024-09-11T08:11:39.768877+0000","last_clean":"2024-09-11T08:11:39.768877+0000","last_became_active":"2024-09-11T08:07:30.041219+0000","last_became_peered":"2024-09-11T08:07:30.041219+0000","last_unstale":"2024-09-11T08:11:39.768877+0000","last_undegraded":"2024-09-11T08:11:39.768877+0000","last_fullsized":"2024-09-11T08:11:39.768877+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:27.250021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:53.247275+0000","last_clean_scrub_stamp":"2024-09-11T08:10:27.250021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011297309999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":94,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.766934+0000","last_change":"2024-09-11T08:10:27.002636+0000","last_active":"2024-09-11T08:11:39.766934+0000","last_peered":"2024-09-11T08:11:39.766934+0000","last_clean":"2024-09-11T08:11:39.766934+0000","last_became_active":"2024-09-11T08:07:34.057479+0000","last_became_peered":"2024-09-11T08:07:34.057479+0000","last_unstale":"2024-09-11T08:11:39.766934+0000","last_undegraded":"2024-09-11T08:11:39.766934+0000","last_fullsized":"2024-09-11T08:11:39.766934+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:27.002455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:10:27.002455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:53.225621+0000","scrub_duration":0.0044259809999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":93,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.769949+0000","last_change":"2024-09-11T08:10:42.097297+0000","last_active":"2024-09-11T08:11:39.769949+0000","last_peered":"2024-09-11T08:11:39.769949+0000","last_clean":"2024-09-11T08:11:39.769949+0000","last_became_active":"2024-09-11T08:07:36.078075+0000","last_became_peered":"2024-09-11T08:07:36.078075+0000","last_unstale":"2024-09-11T08:11:39.769949+0000","last_undegraded":"2024-09-11T08:11:39.769949+0000","last_fullsized":"2024-09-11T08:11:39.769949+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:42.097207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:10:42.097207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:51.535466+0000","scrub_duration":0.0056763179999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"129'9326","reported_seq":9385,"reported_epoch":129,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-11T08:11:42.771806+0000","last_change":"2024-09-11T08:11:39.467736+0000","last_active":"2024-09-11T08:11:42.771806+0000","last_peered":"2024-09-11T08:11:42.771806+0000","last_clean":"2024-09-11T08:11:42.771806+0000","last_became_active":"2024-09-11T08:09:45.412566+0000","last_became_peered":"2024-09-11T08:09:45.412566+0000","last_unstale":"2024-09-11T08:11:42.771806+0000","last_undegraded":"2024-09-11T08:11:42.771806+0000","last_fullsized":"2024-09-11T08:11:42.771806+0000","mapping_epoch":91,"log_start":"108'4301","ondisk_log_start":"108'4301","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_clean_scrub_stamp":"2024-09-11T08:09:44.385805+0000","objects_scrubbed":624,"log_size":5025,"log_dups_size":0,"ondisk_log_size":5025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 3s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":448987136,"num_objects":6851,"num_object_clones":0,"num_object_copies":20553,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8795,"num_write_kb":500672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":121,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.773893+0000","last_change":"2024-09-11T08:10:39.911118+0000","last_active":"2024-09-11T08:11:39.773893+0000","last_peered":"2024-09-11T08:11:39.773893+0000","last_clean":"2024-09-11T08:11:39.773893+0000","last_became_active":"2024-09-11T08:07:32.039275+0000","last_became_peered":"2024-09-11T08:07:32.039275+0000","last_unstale":"2024-09-11T08:11:39.773893+0000","last_undegraded":"2024-09-11T08:11:39.773893+0000","last_fullsized":"2024-09-11T08:11:39.773893+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-11T08:10:39.911043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:10:39.911043+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:03.238724+0000","scrub_duration":0.0020973419999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":116,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.768966+0000","last_change":"2024-09-11T08:11:27.256947+0000","last_active":"2024-09-11T08:11:39.768966+0000","last_peered":"2024-09-11T08:11:39.768966+0000","last_clean":"2024-09-11T08:11:39.768966+0000","last_became_active":"2024-09-11T08:07:30.045967+0000","last_became_peered":"2024-09-11T08:07:30.045967+0000","last_unstale":"2024-09-11T08:11:39.768966+0000","last_undegraded":"2024-09-11T08:11:39.768966+0000","last_fullsized":"2024-09-11T08:11:39.768966+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-11T08:11:27.256886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:11:27.256886+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:47.796310+0000","scrub_duration":0.010702672999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"66'8","reported_seq":154,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.430880+0000","last_change":"2024-09-11T08:10:40.276739+0000","last_active":"2024-09-11T08:11:42.430880+0000","last_peered":"2024-09-11T08:11:42.430880+0000","last_clean":"2024-09-11T08:11:42.430880+0000","last_became_active":"2024-09-11T08:07:34.058015+0000","last_became_peered":"2024-09-11T08:07:34.058015+0000","last_unstale":"2024-09-11T08:11:42.430880+0000","last_undegraded":"2024-09-11T08:11:42.430880+0000","last_fullsized":"2024-09-11T08:11:42.430880+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:10:40.276651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:10:40.276651+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:58.174703+0000","scrub_duration":0.0027620549999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":92,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.774076+0000","last_change":"2024-09-11T08:11:16.768991+0000","last_active":"2024-09-11T08:11:39.774076+0000","last_peered":"2024-09-11T08:11:39.774076+0000","last_clean":"2024-09-11T08:11:39.774076+0000","last_became_active":"2024-09-11T08:07:36.631755+0000","last_became_peered":"2024-09-11T08:07:36.631755+0000","last_unstale":"2024-09-11T08:11:39.774076+0000","last_undegraded":"2024-09-11T08:11:39.774076+0000","last_fullsized":"2024-09-11T08:11:39.774076+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:16.768914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:11:16.768914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:32.682600+0000","scrub_duration":0.0045193669999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":98,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.765018+0000","last_change":"2024-09-11T08:10:36.436042+0000","last_active":"2024-09-11T08:11:39.765018+0000","last_peered":"2024-09-11T08:11:39.765018+0000","last_clean":"2024-09-11T08:11:39.765018+0000","last_became_active":"2024-09-11T08:07:30.045910+0000","last_became_peered":"2024-09-11T08:07:30.045910+0000","last_unstale":"2024-09-11T08:11:39.765018+0000","last_undegraded":"2024-09-11T08:11:39.765018+0000","last_fullsized":"2024-09-11T08:11:39.765018+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:36.435971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:10:36.435971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:00.767133+0000","scrub_duration":0.0023002999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"84'9","reported_seq":109,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.770485+0000","last_change":"2024-09-11T08:11:21.397319+0000","last_active":"2024-09-11T08:11:39.770485+0000","last_peered":"2024-09-11T08:11:39.770485+0000","last_clean":"2024-09-11T08:11:39.770485+0000","last_became_active":"2024-09-11T08:07:32.046891+0000","last_became_peered":"2024-09-11T08:07:32.046891+0000","last_unstale":"2024-09-11T08:11:39.770485+0000","last_undegraded":"2024-09-11T08:11:39.770485+0000","last_fullsized":"2024-09-11T08:11:39.770485+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-09-11T08:11:21.397179+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-09-11T08:09:03.418457+0000","last_clean_scrub_stamp":"2024-09-11T08:11:21.397179+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:48.076959+0000","scrub_duration":0.0062300630000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":94,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.774535+0000","last_change":"2024-09-11T08:10:43.836622+0000","last_active":"2024-09-11T08:11:39.774535+0000","last_peered":"2024-09-11T08:11:39.774535+0000","last_clean":"2024-09-11T08:11:39.774535+0000","last_became_active":"2024-09-11T08:07:34.057774+0000","last_became_peered":"2024-09-11T08:07:34.057774+0000","last_unstale":"2024-09-11T08:11:39.774535+0000","last_undegraded":"2024-09-11T08:11:39.774535+0000","last_fullsized":"2024-09-11T08:11:39.774535+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:43.836566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:10:43.836566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:47.332606+0000","scrub_duration":0.0087284040000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":93,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.768343+0000","last_change":"2024-09-11T08:10:38.297462+0000","last_active":"2024-09-11T08:11:39.768343+0000","last_peered":"2024-09-11T08:11:39.768343+0000","last_clean":"2024-09-11T08:11:39.768343+0000","last_became_active":"2024-09-11T08:07:36.633244+0000","last_became_peered":"2024-09-11T08:07:36.633244+0000","last_unstale":"2024-09-11T08:11:39.768343+0000","last_undegraded":"2024-09-11T08:11:39.768343+0000","last_fullsized":"2024-09-11T08:11:39.768343+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:38.297367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:10:38.297367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:51.071012+0000","scrub_duration":0.00156018,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"84'13","reported_seq":117,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.776941+0000","last_change":"2024-09-11T08:11:32.926108+0000","last_active":"2024-09-11T08:11:39.776941+0000","last_peered":"2024-09-11T08:11:39.776941+0000","last_clean":"2024-09-11T08:11:39.776941+0000","last_became_active":"2024-09-11T08:07:32.048350+0000","last_became_peered":"2024-09-11T08:07:32.048350+0000","last_unstale":"2024-09-11T08:11:39.776941+0000","last_undegraded":"2024-09-11T08:11:39.776941+0000","last_fullsized":"2024-09-11T08:11:39.776941+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'13","last_scrub_stamp":"2024-09-11T08:11:32.925950+0000","last_deep_scrub":"84'13","last_deep_scrub_stamp":"2024-09-11T08:08:57.835634+0000","last_clean_scrub_stamp":"2024-09-11T08:11:32.925950+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:58.685138+0000","scrub_duration":0.013245129,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":109,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.767819+0000","last_change":"2024-09-11T08:11:35.201048+0000","last_active":"2024-09-11T08:11:39.767819+0000","last_peered":"2024-09-11T08:11:39.767819+0000","last_clean":"2024-09-11T08:11:39.767819+0000","last_became_active":"2024-09-11T08:07:30.044055+0000","last_became_peered":"2024-09-11T08:07:30.044055+0000","last_unstale":"2024-09-11T08:11:39.767819+0000","last_undegraded":"2024-09-11T08:11:39.767819+0000","last_fullsized":"2024-09-11T08:11:39.767819+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:35.200970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:30.471620+0000","last_clean_scrub_stamp":"2024-09-11T08:11:35.200970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:36.402648+0000","scrub_duration":0.0012204990000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":95,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.769668+0000","last_change":"2024-09-11T08:11:16.386090+0000","last_active":"2024-09-11T08:11:39.769668+0000","last_peered":"2024-09-11T08:11:39.769668+0000","last_clean":"2024-09-11T08:11:39.769668+0000","last_became_active":"2024-09-11T08:07:34.057127+0000","last_became_peered":"2024-09-11T08:07:34.057127+0000","last_unstale":"2024-09-11T08:11:39.769668+0000","last_undegraded":"2024-09-11T08:11:39.769668+0000","last_fullsized":"2024-09-11T08:11:39.769668+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:16.385713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:04.370734+0000","last_clean_scrub_stamp":"2024-09-11T08:11:16.385713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:38.946720+0000","scrub_duration":0.013591255999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":92,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.766451+0000","last_change":"2024-09-11T08:10:29.963987+0000","last_active":"2024-09-11T08:11:39.766451+0000","last_peered":"2024-09-11T08:11:39.766451+0000","last_clean":"2024-09-11T08:11:39.766451+0000","last_became_active":"2024-09-11T08:07:36.071513+0000","last_became_peered":"2024-09-11T08:07:36.071513+0000","last_unstale":"2024-09-11T08:11:39.766451+0000","last_undegraded":"2024-09-11T08:11:39.766451+0000","last_fullsized":"2024-09-11T08:11:39.766451+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:29.963899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:10:29.963899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:56.542813+0000","scrub_duration":0.0022607389999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":112,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.774459+0000","last_change":"2024-09-11T08:10:18.962167+0000","last_active":"2024-09-11T08:11:39.774459+0000","last_peered":"2024-09-11T08:11:39.774459+0000","last_clean":"2024-09-11T08:11:39.774459+0000","last_became_active":"2024-09-11T08:07:32.041761+0000","last_became_peered":"2024-09-11T08:07:32.041761+0000","last_unstale":"2024-09-11T08:11:39.774459+0000","last_undegraded":"2024-09-11T08:11:39.774459+0000","last_fullsized":"2024-09-11T08:11:39.774459+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:10:18.962088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:10:18.962088+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035069179999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":99,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.773901+0000","last_change":"2024-09-11T08:10:38.937153+0000","last_active":"2024-09-11T08:11:39.773901+0000","last_peered":"2024-09-11T08:11:39.773901+0000","last_clean":"2024-09-11T08:11:39.773901+0000","last_became_active":"2024-09-11T08:07:30.045380+0000","last_became_peered":"2024-09-11T08:07:30.045380+0000","last_unstale":"2024-09-11T08:11:39.773901+0000","last_undegraded":"2024-09-11T08:11:39.773901+0000","last_fullsized":"2024-09-11T08:11:39.773901+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:38.937071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:48.935387+0000","last_clean_scrub_stamp":"2024-09-11T08:10:38.937071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:57.024718+0000","scrub_duration":0.015757854000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"66'8","reported_seq":153,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.770061+0000","last_change":"2024-09-11T08:11:29.518706+0000","last_active":"2024-09-11T08:11:39.770061+0000","last_peered":"2024-09-11T08:11:39.770061+0000","last_clean":"2024-09-11T08:11:39.770061+0000","last_became_active":"2024-09-11T08:07:34.057937+0000","last_became_peered":"2024-09-11T08:07:34.057937+0000","last_unstale":"2024-09-11T08:11:39.770061+0000","last_undegraded":"2024-09-11T08:11:39.770061+0000","last_fullsized":"2024-09-11T08:11:39.770061+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:11:29.518632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:11:29.518632+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:30.606859+0000","scrub_duration":0.015872065000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":83,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.761652+0000","last_change":"2024-09-11T08:09:15.617419+0000","last_active":"2024-09-11T08:11:39.761652+0000","last_peered":"2024-09-11T08:11:39.761652+0000","last_clean":"2024-09-11T08:11:39.761652+0000","last_became_active":"2024-09-11T08:07:36.068493+0000","last_became_peered":"2024-09-11T08:07:36.068493+0000","last_unstale":"2024-09-11T08:11:39.761652+0000","last_undegraded":"2024-09-11T08:11:39.761652+0000","last_fullsized":"2024-09-11T08:11:39.761652+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:15.617360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:09:15.617360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014734049999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":134,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.770343+0000","last_change":"2024-09-11T08:10:37.122397+0000","last_active":"2024-09-11T08:11:39.770343+0000","last_peered":"2024-09-11T08:11:39.770343+0000","last_clean":"2024-09-11T08:11:39.770343+0000","last_became_active":"2024-09-11T08:09:07.878358+0000","last_became_peered":"2024-09-11T08:09:07.878358+0000","last_unstale":"2024-09-11T08:11:39.770343+0000","last_undegraded":"2024-09-11T08:11:39.770343+0000","last_fullsized":"2024-09-11T08:11:39.770343+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-11T08:10:37.122322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:10:37.122322+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:03.673311+0000","scrub_duration":0.0096164340000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":99,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.768454+0000","last_change":"2024-09-11T08:10:33.261203+0000","last_active":"2024-09-11T08:11:39.768454+0000","last_peered":"2024-09-11T08:11:39.768454+0000","last_clean":"2024-09-11T08:11:39.768454+0000","last_became_active":"2024-09-11T08:07:30.039673+0000","last_became_peered":"2024-09-11T08:07:30.039673+0000","last_unstale":"2024-09-11T08:11:39.768454+0000","last_undegraded":"2024-09-11T08:11:39.768454+0000","last_fullsized":"2024-09-11T08:11:39.768454+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:33.261145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:10:33.261145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:49.943383+0000","scrub_duration":0.004998476,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":153,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.428462+0000","last_change":"2024-09-11T08:10:36.942784+0000","last_active":"2024-09-11T08:11:42.428462+0000","last_peered":"2024-09-11T08:11:42.428462+0000","last_clean":"2024-09-11T08:11:42.428462+0000","last_became_active":"2024-09-11T08:07:34.060736+0000","last_became_peered":"2024-09-11T08:07:34.060736+0000","last_unstale":"2024-09-11T08:11:42.428462+0000","last_undegraded":"2024-09-11T08:11:42.428462+0000","last_fullsized":"2024-09-11T08:11:42.428462+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:10:36.942703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:10:36.942703+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:51.228104+0000","scrub_duration":0.017554209000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":83,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.776574+0000","last_change":"2024-09-11T08:09:27.844582+0000","last_active":"2024-09-11T08:11:39.776574+0000","last_peered":"2024-09-11T08:11:39.776574+0000","last_clean":"2024-09-11T08:11:39.776574+0000","last_became_active":"2024-09-11T08:07:36.072359+0000","last_became_peered":"2024-09-11T08:07:36.072359+0000","last_unstale":"2024-09-11T08:11:39.776574+0000","last_undegraded":"2024-09-11T08:11:39.776574+0000","last_fullsized":"2024-09-11T08:11:39.776574+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:27.844523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:09:27.844523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014038360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":116,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.761025+0000","last_change":"2024-09-11T08:11:39.092534+0000","last_active":"2024-09-11T08:11:39.761025+0000","last_peered":"2024-09-11T08:11:39.761025+0000","last_clean":"2024-09-11T08:11:39.761025+0000","last_became_active":"2024-09-11T08:07:32.047981+0000","last_became_peered":"2024-09-11T08:07:32.047981+0000","last_unstale":"2024-09-11T08:11:39.761025+0000","last_undegraded":"2024-09-11T08:11:39.761025+0000","last_fullsized":"2024-09-11T08:11:39.761025+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-11T08:11:39.092431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:11:39.092431+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:51.100772+0000","scrub_duration":0.0069898629999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":99,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.761424+0000","last_change":"2024-09-11T08:10:36.134384+0000","last_active":"2024-09-11T08:11:39.761424+0000","last_peered":"2024-09-11T08:11:39.761424+0000","last_clean":"2024-09-11T08:11:39.761424+0000","last_became_active":"2024-09-11T08:07:30.043691+0000","last_became_peered":"2024-09-11T08:07:30.043691+0000","last_unstale":"2024-09-11T08:11:39.761424+0000","last_undegraded":"2024-09-11T08:11:39.761424+0000","last_fullsized":"2024-09-11T08:11:39.761424+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:36.134300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:10:36.134300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:01.485017+0000","scrub_duration":0.0020719419999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":95,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.772917+0000","last_change":"2024-09-11T08:11:25.585872+0000","last_active":"2024-09-11T08:11:39.772917+0000","last_peered":"2024-09-11T08:11:39.772917+0000","last_clean":"2024-09-11T08:11:39.772917+0000","last_became_active":"2024-09-11T08:07:34.053137+0000","last_became_peered":"2024-09-11T08:07:34.053137+0000","last_unstale":"2024-09-11T08:11:39.772917+0000","last_undegraded":"2024-09-11T08:11:39.772917+0000","last_fullsized":"2024-09-11T08:11:39.772917+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:25.585537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:11:25.585537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:51.823113+0000","scrub_duration":0.015833313000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":83,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.763119+0000","last_change":"2024-09-11T08:10:30.195470+0000","last_active":"2024-09-11T08:11:39.763119+0000","last_peered":"2024-09-11T08:11:39.763119+0000","last_clean":"2024-09-11T08:11:39.763119+0000","last_became_active":"2024-09-11T08:07:36.076132+0000","last_became_peered":"2024-09-11T08:07:36.076132+0000","last_unstale":"2024-09-11T08:11:39.763119+0000","last_undegraded":"2024-09-11T08:11:39.763119+0000","last_fullsized":"2024-09-11T08:11:39.763119+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:30.195389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:10:30.195389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:58.081266+0000","scrub_duration":0.013816314999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":113,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.773200+0000","last_change":"2024-09-11T08:10:32.411685+0000","last_active":"2024-09-11T08:11:39.773200+0000","last_peered":"2024-09-11T08:11:39.773200+0000","last_clean":"2024-09-11T08:11:39.773200+0000","last_became_active":"2024-09-11T08:07:32.049368+0000","last_became_peered":"2024-09-11T08:07:32.049368+0000","last_unstale":"2024-09-11T08:11:39.773200+0000","last_undegraded":"2024-09-11T08:11:39.773200+0000","last_fullsized":"2024-09-11T08:11:39.773200+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:10:32.411521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:10:32.411521+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:44.101292+0000","scrub_duration":0.0076800619999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":109,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.768609+0000","last_change":"2024-09-11T08:11:39.203061+0000","last_active":"2024-09-11T08:11:39.768609+0000","last_peered":"2024-09-11T08:11:39.768609+0000","last_clean":"2024-09-11T08:11:39.768609+0000","last_became_active":"2024-09-11T08:07:30.037692+0000","last_became_peered":"2024-09-11T08:07:30.037692+0000","last_unstale":"2024-09-11T08:11:39.768609+0000","last_undegraded":"2024-09-11T08:11:39.768609+0000","last_fullsized":"2024-09-11T08:11:39.768609+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:39.202984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:11:39.202984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:51.576908+0000","scrub_duration":0.001252852,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":95,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.770392+0000","last_change":"2024-09-11T08:11:35.568457+0000","last_active":"2024-09-11T08:11:39.770392+0000","last_peered":"2024-09-11T08:11:39.770392+0000","last_clean":"2024-09-11T08:11:39.770392+0000","last_became_active":"2024-09-11T08:07:34.057064+0000","last_became_peered":"2024-09-11T08:07:34.057064+0000","last_unstale":"2024-09-11T08:11:39.770392+0000","last_undegraded":"2024-09-11T08:11:39.770392+0000","last_fullsized":"2024-09-11T08:11:39.770392+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:35.567888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:11:35.567888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:13:05.264013+0000","scrub_duration":0.029638675999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":93,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.768494+0000","last_change":"2024-09-11T08:11:22.043711+0000","last_active":"2024-09-11T08:11:39.768494+0000","last_peered":"2024-09-11T08:11:39.768494+0000","last_clean":"2024-09-11T08:11:39.768494+0000","last_became_active":"2024-09-11T08:07:36.077406+0000","last_became_peered":"2024-09-11T08:07:36.077406+0000","last_unstale":"2024-09-11T08:11:39.768494+0000","last_undegraded":"2024-09-11T08:11:39.768494+0000","last_fullsized":"2024-09-11T08:11:39.768494+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:22.043633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:09.973943+0000","last_clean_scrub_stamp":"2024-09-11T08:11:22.043633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:30.337221+0000","scrub_duration":0.0064821080000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":116,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.763476+0000","last_change":"2024-09-11T08:11:33.518272+0000","last_active":"2024-09-11T08:11:39.763476+0000","last_peered":"2024-09-11T08:11:39.763476+0000","last_clean":"2024-09-11T08:11:39.763476+0000","last_became_active":"2024-09-11T08:07:32.043276+0000","last_became_peered":"2024-09-11T08:07:32.043276+0000","last_unstale":"2024-09-11T08:11:39.763476+0000","last_undegraded":"2024-09-11T08:11:39.763476+0000","last_fullsized":"2024-09-11T08:11:39.763476+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-11T08:11:33.518136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:11:33.518136+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:43.052462+0000","scrub_duration":0.0058292969999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":98,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.774557+0000","last_change":"2024-09-11T08:10:23.978440+0000","last_active":"2024-09-11T08:11:39.774557+0000","last_peered":"2024-09-11T08:11:39.774557+0000","last_clean":"2024-09-11T08:11:39.774557+0000","last_became_active":"2024-09-11T08:07:30.049152+0000","last_became_peered":"2024-09-11T08:07:30.049152+0000","last_unstale":"2024-09-11T08:11:39.774557+0000","last_undegraded":"2024-09-11T08:11:39.774557+0000","last_fullsized":"2024-09-11T08:11:39.774557+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:23.978385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:10:23.978385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:48.317452+0000","scrub_duration":0.0025160999999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":95,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.767420+0000","last_change":"2024-09-11T08:11:34.159199+0000","last_active":"2024-09-11T08:11:39.767420+0000","last_peered":"2024-09-11T08:11:39.767420+0000","last_clean":"2024-09-11T08:11:39.767420+0000","last_became_active":"2024-09-11T08:07:34.056935+0000","last_became_peered":"2024-09-11T08:07:34.056935+0000","last_unstale":"2024-09-11T08:11:39.767420+0000","last_undegraded":"2024-09-11T08:11:39.767420+0000","last_fullsized":"2024-09-11T08:11:39.767420+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:34.159053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:11:34.159053+0000","last_clean_scrub_stamp":"2024-09-11T08:11:34.159053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:35.241223+0000","scrub_duration":0.00130974,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":83,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.770227+0000","last_change":"2024-09-11T08:09:34.378812+0000","last_active":"2024-09-11T08:11:39.770227+0000","last_peered":"2024-09-11T08:11:39.770227+0000","last_clean":"2024-09-11T08:11:39.770227+0000","last_became_active":"2024-09-11T08:07:36.633368+0000","last_became_peered":"2024-09-11T08:07:36.633368+0000","last_unstale":"2024-09-11T08:11:39.770227+0000","last_undegraded":"2024-09-11T08:11:39.770227+0000","last_fullsized":"2024-09-11T08:11:39.770227+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:34.378734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:34.378734+0000","last_clean_scrub_stamp":"2024-09-11T08:09:34.378734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001593148,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":93,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.767749+0000","last_change":"2024-09-11T08:09:40.373717+0000","last_active":"2024-09-11T08:11:39.767749+0000","last_peered":"2024-09-11T08:11:39.767749+0000","last_clean":"2024-09-11T08:11:39.767749+0000","last_became_active":"2024-09-11T08:07:32.045728+0000","last_became_peered":"2024-09-11T08:07:32.045728+0000","last_unstale":"2024-09-11T08:11:39.767749+0000","last_undegraded":"2024-09-11T08:11:39.767749+0000","last_fullsized":"2024-09-11T08:11:39.767749+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'4","last_scrub_stamp":"2024-09-11T08:09:40.373649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:09:40.373649+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001798315,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":99,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.776201+0000","last_change":"2024-09-11T08:10:28.129119+0000","last_active":"2024-09-11T08:11:39.776201+0000","last_peered":"2024-09-11T08:11:39.776201+0000","last_clean":"2024-09-11T08:11:39.776201+0000","last_became_active":"2024-09-11T08:07:30.049409+0000","last_became_peered":"2024-09-11T08:07:30.049409+0000","last_unstale":"2024-09-11T08:11:39.776201+0000","last_undegraded":"2024-09-11T08:11:39.776201+0000","last_fullsized":"2024-09-11T08:11:39.776201+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:28.129035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:10:28.129035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.019365614999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":104,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.765232+0000","last_change":"2024-09-11T08:11:38.545774+0000","last_active":"2024-09-11T08:11:39.765232+0000","last_peered":"2024-09-11T08:11:39.765232+0000","last_clean":"2024-09-11T08:11:39.765232+0000","last_became_active":"2024-09-11T08:07:34.055082+0000","last_became_peered":"2024-09-11T08:07:34.055082+0000","last_unstale":"2024-09-11T08:11:39.765232+0000","last_undegraded":"2024-09-11T08:11:39.765232+0000","last_fullsized":"2024-09-11T08:11:39.765232+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:38.545717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:11:38.545717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:43.268646+0000","scrub_duration":0.0054392340000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":93,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.767899+0000","last_change":"2024-09-11T08:10:42.297248+0000","last_active":"2024-09-11T08:11:39.767899+0000","last_peered":"2024-09-11T08:11:39.767899+0000","last_clean":"2024-09-11T08:11:39.767899+0000","last_became_active":"2024-09-11T08:07:36.071669+0000","last_became_peered":"2024-09-11T08:07:36.071669+0000","last_unstale":"2024-09-11T08:11:39.767899+0000","last_undegraded":"2024-09-11T08:11:39.767899+0000","last_fullsized":"2024-09-11T08:11:39.767899+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:42.297166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:10:42.297166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:07.034868+0000","scrub_duration":0.0077635450000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":93,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.775102+0000","last_change":"2024-09-11T08:10:28.457397+0000","last_active":"2024-09-11T08:11:39.775102+0000","last_peered":"2024-09-11T08:11:39.775102+0000","last_clean":"2024-09-11T08:11:39.775102+0000","last_became_active":"2024-09-11T08:07:36.071468+0000","last_became_peered":"2024-09-11T08:07:36.071468+0000","last_unstale":"2024-09-11T08:11:39.775102+0000","last_undegraded":"2024-09-11T08:11:39.775102+0000","last_fullsized":"2024-09-11T08:11:39.775102+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:28.457304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:10:28.457304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:46.021922+0000","scrub_duration":0.0063908749999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":99,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.767981+0000","last_change":"2024-09-11T08:11:26.404087+0000","last_active":"2024-09-11T08:11:39.767981+0000","last_peered":"2024-09-11T08:11:39.767981+0000","last_clean":"2024-09-11T08:11:39.767981+0000","last_became_active":"2024-09-11T08:07:30.046702+0000","last_became_peered":"2024-09-11T08:07:30.046702+0000","last_unstale":"2024-09-11T08:11:39.767981+0000","last_undegraded":"2024-09-11T08:11:39.767981+0000","last_fullsized":"2024-09-11T08:11:39.767981+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:26.403944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:11:26.403944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:29.575666+0000","scrub_duration":0.0056165199999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":115,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.764284+0000","last_change":"2024-09-11T08:10:24.612372+0000","last_active":"2024-09-11T08:11:39.764284+0000","last_peered":"2024-09-11T08:11:39.764284+0000","last_clean":"2024-09-11T08:11:39.764284+0000","last_became_active":"2024-09-11T08:07:32.054656+0000","last_became_peered":"2024-09-11T08:07:32.054656+0000","last_unstale":"2024-09-11T08:11:39.764284+0000","last_undegraded":"2024-09-11T08:11:39.764284+0000","last_fullsized":"2024-09-11T08:11:39.764284+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-11T08:10:24.612282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:10:24.612282+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:52.941292+0000","scrub_duration":0.0020463640000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":95,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-11T08:11:39.776674+0000","last_change":"2024-09-11T08:10:32.091677+0000","last_active":"2024-09-11T08:11:39.776674+0000","last_peered":"2024-09-11T08:11:39.776674+0000","last_clean":"2024-09-11T08:11:39.776674+0000","last_became_active":"2024-09-11T08:07:34.061453+0000","last_became_peered":"2024-09-11T08:07:34.061453+0000","last_unstale":"2024-09-11T08:11:39.776674+0000","last_undegraded":"2024-09-11T08:11:39.776674+0000","last_fullsized":"2024-09-11T08:11:39.776674+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:32.091605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:10:32.091605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:52.537395+0000","scrub_duration":0.0071709479999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":6969425944,"num_objects":106346,"num_object_clones":0,"num_object_copies":319038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":106346,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142532,"num_write_kb":7964033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20876242944,"data_stored":20876230728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89962,"ondisk_log_size":89962,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":435,"ondisk_log_size":435,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":29,"num_read_kb":29,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":251,"num_read_kb":246,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266744,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2382804,"kb_used_data":2085452,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":91362348,"statfs":{"total":95995035648,"available":93555044352,"internally_reserved":0,"allocated":2135502848,"data_stored":2135103058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:11:23 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.52300000000000002,"15min":0.496},"min":{"1min":0.20300000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":3.4119999999999999,"5min":3.4119999999999999,"15min":3.4119999999999999},"last":0.216},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.58499999999999996,"15min":0.51700000000000002},"min":{"1min":0.251,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.3279999999999998,"5min":2.3279999999999998,"15min":2.3279999999999998},"last":0.29799999999999999}]},{"osd":1,"last update":"Wed Sep 11 08:11:23 2024","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.59499999999999997,"15min":0.52700000000000002},"min":{"1min":0.23000000000000001,"5min":0.216,"15min":0.216},"max":{"1min":3.423,"5min":3.423,"15min":3.423},"last":0.25},{"interface":"front","average":{"1min":0.997,"5min":0.63300000000000001,"15min":0.56299999999999994},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":3.4409999999999998,"5min":3.4409999999999998,"15min":3.4409999999999998},"last":0.33800000000000002}]},{"osd":2,"last update":"Wed Sep 11 08:11:23 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":0.67600000000000005,"15min":0.54600000000000004},"min":{"1min":0.26500000000000001,"5min":0.254,"15min":0.254},"max":{"1min":4.1589999999999998,"5min":5.5919999999999996,"15min":5.5919999999999996},"last":1.694},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.52100000000000002,"15min":0.50700000000000001},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":1.302,"5min":2.677,"15min":2.677},"last":2.3279999999999998}]},{"osd":3,"last update":"Wed Sep 11 08:11:23 2024","interfaces":[{"interface":"back","average":{"1min":2.278,"5min":1.03,"15min":0.94799999999999995},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":10.818,"5min":10.818,"15min":10.818},"last":0.317},{"interface":"front","average":{"1min":1.329,"5min":0.66900000000000004,"15min":0.54400000000000004},"min":{"1min":0.29999999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":5.7080000000000002,"5min":5.7080000000000002,"15min":5.7080000000000002},"last":2.1829999999999998}]},{"osd":4,"last update":"Wed Sep 11 08:11:23 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.41699999999999998,"15min":0.40699999999999997},"min":{"1min":0.215,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":1.0580000000000001,"5min":1.7050000000000001,"15min":1.7050000000000001},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.42499999999999999,"15min":0.41899999999999998},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.92800000000000005,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":0.34799999999999998}]},{"osd":5,"last update":"Wed Sep 11 08:11:23 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.40600000000000003,"15min":0.41799999999999998},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.60799999999999998,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.503,"5min":0.47399999999999998,"15min":0.46200000000000002},"min":{"1min":0.26700000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.1000000000000001,"5min":1.6799999999999999,"15min":1.6799999999999999},"last":0.38400000000000001}]},{"osd":6,"last update":"Wed Sep 11 08:11:23 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50700000000000001,"15min":0.499},"min":{"1min":0.29099999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.014,"5min":1.819,"15min":1.819},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.497,"5min":0.48299999999999998,"15min":0.47199999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.84699999999999998,"5min":1.492,"15min":1.492},"last":0.47099999999999997}]}]},{"osd":6,"up_from":45,"seq":193273528381,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2377032,"kb_used_data":2075520,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":91368120,"statfs":{"total":95995035648,"available":93560954880,"internally_reserved":0,"allocated":2125332480,"data_stored":2124944825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,4],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:11:02 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.54600000000000004,"15min":0.51400000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":2.5870000000000002,"5min":2.5870000000000002,"15min":2.5870000000000002},"last":2.4660000000000002},{"interface":"front","average":{"1min":1.1140000000000001,"5min":0.623,"15min":0.54500000000000004},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":5.9320000000000004,"5min":5.9320000000000004,"15min":5.9320000000000004},"last":1.361}]},{"osd":1,"last update":"Wed Sep 11 08:11:02 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.50700000000000001,"15min":0.48699999999999999},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":4.8700000000000001,"5min":4.8700000000000001,"15min":4.8700000000000001},"last":2.4390000000000001},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.54100000000000004,"15min":0.51800000000000002},"min":{"1min":0.27600000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":2.8199999999999998,"5min":2.8199999999999998,"15min":2.8199999999999998},"last":0.31}]},{"osd":2,"last update":"Wed Sep 11 08:11:02 2024","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.58099999999999996,"15min":0.52900000000000003},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":2.6840000000000002,"5min":2.6840000000000002,"15min":2.6840000000000002},"last":0.40600000000000003},{"interface":"front","average":{"1min":1.0389999999999999,"5min":0.59399999999999997,"15min":0.51900000000000002},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":2.5489999999999999,"5min":2.5489999999999999,"15min":2.5489999999999999},"last":0.23300000000000001}]},{"osd":3,"last update":"Wed Sep 11 08:11:02 2024","interfaces":[{"interface":"back","average":{"1min":1.5029999999999999,"5min":0.70999999999999996,"15min":0.58799999999999997},"min":{"1min":0.34799999999999998,"5min":0.308,"15min":0.308},"max":{"1min":4.343,"5min":4.343,"15min":4.343},"last":2.8170000000000002},{"interface":"front","average":{"1min":1.2609999999999999,"5min":0.67600000000000005,"15min":0.58399999999999996},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":4.3540000000000001,"5min":4.3540000000000001,"15min":4.3540000000000001},"last":2.9180000000000001}]},{"osd":4,"last update":"Wed Sep 11 08:11:02 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.40699999999999997,"15min":0.39300000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":1.425},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.441,"15min":0.41899999999999998},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":1.407}]},{"osd":5,"last update":"Wed Sep 11 08:11:02 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.501,"15min":0.48399999999999999},"min":{"1min":0.29399999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.22,"5min":1.22,"15min":1.22},"last":1.3859999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.44500000000000001,"15min":0.42099999999999999},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.653,"5min":2.653,"15min":2.653},"last":2.8450000000000002}]},{"osd":7,"last update":"Wed Sep 11 08:11:27 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.49299999999999999,"15min":0.47299999999999998},"min":{"1min":0.31900000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.359,"5min":1.359,"15min":1.359},"last":1.4490000000000001},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.46899999999999997,"15min":0.44500000000000001},"min":{"1min":0.28199999999999997,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":2.9399999999999999}]}]},{"osd":5,"up_from":38,"seq":163208757314,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1981828,"kb_used_data":1684476,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":91763324,"statfs":{"total":95995035648,"available":93965643776,"internally_reserved":0,"allocated":1724903424,"data_stored":1724518858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:11:40 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":0.73299999999999998,"15min":0.59499999999999997},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.4299999999999997,"5min":6.343,"15min":6.343},"last":2.694},{"interface":"front","average":{"1min":1.0269999999999999,"5min":0.71299999999999997,"15min":0.57099999999999995},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.5150000000000001,"5min":4.6319999999999997,"15min":4.6319999999999997},"last":0.46700000000000003}]},{"osd":1,"last update":"Wed Sep 11 08:11:40 2024","interfaces":[{"interface":"back","average":{"1min":1.694,"5min":0.98299999999999998,"15min":1.2330000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":12.847,"5min":16.004999999999999,"15min":16.004999999999999},"last":0.252},{"interface":"front","average":{"1min":1.758,"5min":0.76300000000000001,"15min":0.59299999999999997},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":13.215999999999999,"5min":13.215999999999999,"15min":13.215999999999999},"last":0.34100000000000003}]},{"osd":2,"last update":"Wed Sep 11 08:11:40 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":0.83399999999999996,"15min":1.1990000000000001},"min":{"1min":0.308,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":5.1669999999999998,"5min":16.018000000000001,"15min":16.018000000000001},"last":1.974},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.63500000000000001,"15min":0.55900000000000005},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":4.7469999999999999,"5min":4.7469999999999999,"15min":4.7469999999999999},"last":0.36899999999999999}]},{"osd":3,"last update":"Wed Sep 11 08:11:40 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":0.78000000000000003,"15min":0.61199999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":4.7300000000000004,"5min":6.3230000000000004,"15min":6.3230000000000004},"last":0.38900000000000001},{"interface":"front","average":{"1min":1.242,"5min":0.75,"15min":0.59599999999999997},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":7.3159999999999998,"5min":7.3159999999999998,"15min":7.3159999999999998},"last":4.6920000000000002}]},{"osd":4,"last update":"Wed Sep 11 08:11:40 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.46300000000000002,"15min":0.41499999999999998},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":2.0779999999999998,"5min":2.0779999999999998,"15min":2.0779999999999998},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.48299999999999998,"15min":0.41499999999999998},"min":{"1min":0.26700000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":2.3090000000000002,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":0.315}]},{"osd":6,"last update":"Wed Sep 11 08:11:00 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.45800000000000002,"15min":0.432},"min":{"1min":0.254,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":1.3080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.505,"15min":0.46300000000000002},"min":{"1min":0.30399999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.319,"5min":1.319,"15min":1.319},"last":0.49099999999999999}]},{"osd":7,"last update":"Wed Sep 11 08:11:28 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.52700000000000002,"15min":0.50900000000000001},"min":{"1min":0.32000000000000001,"5min":0.307,"15min":0.307},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.443},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.55200000000000005,"15min":0.501},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":2.448,"5min":2.448,"15min":2.448},"last":0.64100000000000001}]}]},{"osd":4,"up_from":31,"seq":133143986247,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2397400,"kb_used_data":2100048,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":91347752,"statfs":{"total":95995035648,"available":93540098048,"internally_reserved":0,"allocated":2150449152,"data_stored":2150045420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:11:13 2024","interfaces":[{"interface":"back","average":{"1min":3.1589999999999998,"5min":1.6259999999999999,"15min":0.88400000000000001},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":26.754999999999999,"5min":48.728999999999999,"15min":48.728999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":2.4289999999999998,"5min":1.4079999999999999,"15min":0.82999999999999996},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":27.055,"5min":49.168999999999997,"15min":49.168999999999997},"last":0.26400000000000001}]},{"osd":1,"last update":"Wed Sep 11 08:11:13 2024","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":0.73899999999999999,"15min":0.59099999999999997},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":2.6669999999999998,"5min":13.207000000000001,"15min":13.207000000000001},"last":1.304},{"interface":"front","average":{"1min":1.163,"5min":0.64700000000000002,"15min":0.57099999999999995},"min":{"1min":0.25600000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":3.2679999999999998,"5min":3.2679999999999998,"15min":3.2679999999999998},"last":0.58699999999999997}]},{"osd":2,"last update":"Wed Sep 11 08:11:13 2024","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":0.64000000000000001,"15min":0.56799999999999995},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":3.9750000000000001,"5min":3.9750000000000001,"15min":3.9750000000000001},"last":0.91800000000000004},{"interface":"front","average":{"1min":1.6679999999999999,"5min":0.74399999999999999,"15min":0.60099999999999998},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":5.5670000000000002,"5min":5.5670000000000002,"15min":5.5670000000000002},"last":1.135}]},{"osd":3,"last update":"Wed Sep 11 08:11:13 2024","interfaces":[{"interface":"back","average":{"1min":2.7599999999999998,"5min":1.274,"15min":0.77000000000000002},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":26.884,"5min":26.884,"15min":26.884},"last":1.8819999999999999},{"interface":"front","average":{"1min":2.9710000000000001,"5min":1.1819999999999999,"15min":0.747},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":26.783000000000001,"5min":26.783000000000001,"15min":26.783000000000001},"last":0.59499999999999997}]},{"osd":5,"last update":"Wed Sep 11 08:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.40999999999999998,"15min":0.38},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":1.151,"5min":1.151,"15min":1.151},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.439,"15min":0.39000000000000001},"min":{"1min":0.28199999999999997,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":1.9630000000000001,"5min":1.9630000000000001,"15min":1.9630000000000001},"last":0.29499999999999998}]},{"osd":6,"last update":"Wed Sep 11 08:11:00 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.40500000000000003,"15min":0.38800000000000001},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.61399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.48499999999999999,"15min":0.44700000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.28399999999999997}]},{"osd":7,"last update":"Wed Sep 11 08:11:28 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.49099999999999999,"15min":0.46300000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.097,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.627,"5min":0.498,"15min":0.46600000000000003},"min":{"1min":0.32700000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.931,"5min":1.931,"15min":1.931},"last":0.438}]}]},{"osd":3,"up_from":25,"seq":107374182475,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4234392,"kb_used_data":3928720,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":89510760,"statfs":{"total":95995035648,"available":91659018240,"internally_reserved":0,"allocated":4023009280,"data_stored":4022598207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:10:59 2024","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":0.55600000000000005,"15min":0.438},"min":{"1min":0.27700000000000002,"5min":0.191,"15min":0.191},"max":{"1min":4.2779999999999996,"5min":4.2779999999999996,"15min":4.2779999999999996},"last":0.88700000000000001},{"interface":"front","average":{"1min":1.333,"5min":0.55900000000000005,"15min":0.46999999999999997},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":3.1960000000000002,"5min":3.1960000000000002,"15min":3.1960000000000002},"last":1.095}]},{"osd":1,"last update":"Wed Sep 11 08:10:59 2024","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":0.54100000000000004,"15min":0.47899999999999998},"min":{"1min":0.28699999999999998,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":3.5910000000000002,"5min":3.5910000000000002,"15min":3.5910000000000002},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.9850000000000001,"5min":0.70299999999999996,"15min":0.52500000000000002},"min":{"1min":0.252,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":8.5,"5min":8.5,"15min":8.5},"last":0.73999999999999999}]},{"osd":2,"last update":"Wed Sep 11 08:10:59 2024","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":0.59399999999999997,"15min":0.502},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":5.835,"5min":5.835,"15min":5.835},"last":0.82199999999999995},{"interface":"front","average":{"1min":1.556,"5min":0.61299999999999999,"15min":0.48799999999999999},"min":{"1min":0.22500000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":8.5950000000000006,"5min":8.5950000000000006,"15min":8.5950000000000006},"last":0.95699999999999996}]},{"osd":4,"last update":"Wed Sep 11 08:11:18 2024","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":0.73699999999999999,"15min":0.61199999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":5.157,"5min":5.157,"15min":5.157},"last":0.79700000000000004},{"interface":"front","average":{"1min":1.4530000000000001,"5min":0.70399999999999996,"15min":0.59399999999999997},"min":{"1min":0.312,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":10.563000000000001,"5min":10.563000000000001,"15min":10.563000000000001},"last":1}]},{"osd":5,"last update":"Wed Sep 11 08:11:39 2024","interfaces":[{"interface":"back","average":{"1min":1.899,"5min":0.85199999999999998,"15min":0.64200000000000002},"min":{"1min":0.26600000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":10.496,"5min":10.496,"15min":10.496},"last":0.98199999999999998},{"interface":"front","average":{"1min":1.331,"5min":0.96499999999999997,"15min":0.68600000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":6.4279999999999999,"5min":16.757000000000001,"15min":16.757000000000001},"last":1.1499999999999999}]},{"osd":6,"last update":"Wed Sep 11 08:11:03 2024","interfaces":[{"interface":"back","average":{"1min":1.329,"5min":0.89700000000000002,"15min":0.65100000000000002},"min":{"1min":0.33400000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":6.4489999999999998,"5min":16.789000000000001,"15min":16.789000000000001},"last":1.1899999999999999},{"interface":"front","average":{"1min":2.0289999999999999,"5min":1.0369999999999999,"15min":0.71899999999999997},"min":{"1min":0.40899999999999997,"5min":0.373,"15min":0.373},"max":{"1min":11.122999999999999,"5min":16.748000000000001,"15min":16.748000000000001},"last":0.77200000000000002}]},{"osd":7,"last update":"Wed Sep 11 08:11:29 2024","interfaces":[{"interface":"back","average":{"1min":1.784,"5min":0.82099999999999995,"15min":0.64100000000000001},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":10.538,"5min":10.538,"15min":10.538},"last":1.0229999999999999},{"interface":"front","average":{"1min":1.6919999999999999,"5min":0.84799999999999998,"15min":0.67200000000000004},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":11.098000000000001,"5min":11.098000000000001,"15min":11.098000000000001},"last":1.1259999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411407,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2816260,"kb_used_data":2506428,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":90928892,"statfs":{"total":95995035648,"available":93111185408,"internally_reserved":0,"allocated":2566582272,"data_stored":2566197618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:11:33 2024","interfaces":[{"interface":"back","average":{"1min":2.157,"5min":0.86199999999999999,"15min":0.54000000000000004},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":6.468,"5min":6.468,"15min":6.468},"last":0.93000000000000005},{"interface":"front","average":{"1min":2.2360000000000002,"5min":0.85199999999999998,"15min":0.52100000000000002},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":6.9080000000000004,"5min":6.9080000000000004,"15min":6.9080000000000004},"last":0.67600000000000005}]},{"osd":1,"last update":"Wed Sep 11 08:11:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7709999999999999,"5min":0.83499999999999996,"15min":0.52700000000000002},"min":{"1min":0.26100000000000001,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":6.3499999999999996,"5min":6.3499999999999996,"15min":6.3499999999999996},"last":0.81999999999999995},{"interface":"front","average":{"1min":1.7350000000000001,"5min":0.81000000000000005,"15min":0.52400000000000002},"min":{"1min":0.23200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":5.6970000000000001,"5min":5.6970000000000001,"15min":5.6970000000000001},"last":0.54200000000000004}]},{"osd":3,"last update":"Wed Sep 11 08:10:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":0.55600000000000005,"15min":0.496},"min":{"1min":0.32400000000000001,"5min":0.218,"15min":0.218},"max":{"1min":3.089,"5min":3.089,"15min":3.089},"last":0.95999999999999996},{"interface":"front","average":{"1min":1.1419999999999999,"5min":0.56699999999999995,"15min":0.503},"min":{"1min":0.35399999999999998,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":3.6440000000000001,"5min":3.6440000000000001,"15min":3.6440000000000001},"last":0.57199999999999995}]},{"osd":4,"last update":"Wed Sep 11 08:11:16 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":0.74399999999999999,"15min":0.60199999999999998},"min":{"1min":0.311,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":7.806,"5min":7.806,"15min":7.806},"last":1.7689999999999999},{"interface":"front","average":{"1min":1.4510000000000001,"5min":0.872,"15min":0.68300000000000005},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":6.8200000000000003,"5min":12.129,"15min":12.129},"last":0.59599999999999997}]},{"osd":5,"last update":"Wed Sep 11 08:11:45 2024","interfaces":[{"interface":"back","average":{"1min":1.7749999999999999,"5min":0.89700000000000002,"15min":0.68899999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":6.7709999999999999,"5min":6.7709999999999999,"15min":6.7709999999999999},"last":0.62},{"interface":"front","average":{"1min":1.329,"5min":0.76200000000000001,"15min":0.65700000000000003},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":6.4050000000000002,"5min":6.4050000000000002,"15min":6.4050000000000002},"last":0.88}]},{"osd":6,"last update":"Wed Sep 11 08:10:58 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.64100000000000001,"15min":0.61299999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":4.2350000000000003,"5min":4.2350000000000003,"15min":4.2350000000000003},"last":0.90500000000000003},{"interface":"front","average":{"1min":1.21,"5min":0.70599999999999996,"15min":0.63600000000000001},"min":{"1min":0.376,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":2.9249999999999998,"5min":2.9249999999999998,"15min":2.9249999999999998},"last":0.64400000000000002}]},{"osd":7,"last update":"Wed Sep 11 08:11:27 2024","interfaces":[{"interface":"back","average":{"1min":1.45,"5min":0.81399999999999995,"15min":0.65400000000000003},"min":{"1min":0.34799999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":7.7560000000000002,"5min":7.7560000000000002,"15min":7.7560000000000002},"last":1.74},{"interface":"front","average":{"1min":2.0409999999999999,"5min":1.0309999999999999,"15min":0.74399999999999999},"min":{"1min":0.40100000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":6.4260000000000002,"5min":12.115,"15min":12.115},"last":0.85299999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574932,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2905712,"kb_used_data":2604196,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":90839440,"statfs":{"total":95995035648,"available":93019586560,"internally_reserved":0,"allocated":2666696704,"data_stored":2666272130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,0,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:11:10 2024","interfaces":[{"interface":"back","average":{"1min":3.6480000000000001,"5min":1.0580000000000001,"15min":0.54400000000000004},"min":{"1min":0.245,"5min":0.22600000000000001,"15min":0.16300000000000001},"max":{"1min":32.478000000000002,"5min":32.478000000000002,"15min":32.478000000000002},"last":2.9289999999999998},{"interface":"front","average":{"1min":4.6200000000000001,"5min":1.2849999999999999,"15min":0.63},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20300000000000001},"max":{"1min":35.459000000000003,"5min":35.459000000000003,"15min":35.459000000000003},"last":4.0419999999999998}]},{"osd":2,"last update":"Wed Sep 11 08:11:39 2024","interfaces":[{"interface":"back","average":{"1min":2.4300000000000002,"5min":0.88500000000000001,"15min":0.52300000000000002},"min":{"1min":0.38200000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":8.3819999999999997,"5min":8.3819999999999997,"15min":8.3819999999999997},"last":2.8100000000000001},{"interface":"front","average":{"1min":3.4700000000000002,"5min":1.175,"15min":0.61499999999999999},"min":{"1min":0.30499999999999999,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":10.276,"5min":10.276,"15min":10.276},"last":8.702}]},{"osd":3,"last update":"Wed Sep 11 08:10:51 2024","interfaces":[{"interface":"back","average":{"1min":4.6459999999999999,"5min":1.244,"15min":0.69799999999999995},"min":{"1min":0.316,"5min":0.246,"15min":0.246},"max":{"1min":32.759,"5min":32.759,"15min":32.759},"last":4.0670000000000002},{"interface":"front","average":{"1min":3.6760000000000002,"5min":1.0800000000000001,"15min":0.65000000000000002},"min":{"1min":0.34599999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":32.444000000000003,"5min":32.444000000000003,"15min":32.444000000000003},"last":3.6949999999999998}]},{"osd":4,"last update":"Wed Sep 11 08:11:17 2024","interfaces":[{"interface":"back","average":{"1min":2.1640000000000001,"5min":0.91500000000000004,"15min":0.66600000000000004},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":9.0139999999999993,"5min":9.0139999999999993,"15min":9.0139999999999993},"last":8.5640000000000001},{"interface":"front","average":{"1min":1.873,"5min":0.83699999999999997,"15min":0.64900000000000002},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":8.2360000000000007,"5min":8.2360000000000007,"15min":8.2360000000000007},"last":2.847}]},{"osd":5,"last update":"Wed Sep 11 08:10:40 2024","interfaces":[{"interface":"back","average":{"1min":1.657,"5min":0.76100000000000001,"15min":0.60499999999999998},"min":{"1min":0.35099999999999998,"5min":0.312,"15min":0.312},"max":{"1min":9.0470000000000006,"5min":9.0470000000000006,"15min":9.0470000000000006},"last":8.6769999999999996},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.69599999999999995,"15min":0.58199999999999996},"min":{"1min":0.38,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":2.1850000000000001,"5min":11.967000000000001,"15min":11.967000000000001},"last":3.9540000000000002}]},{"osd":6,"last update":"Wed Sep 11 08:10:57 2024","interfaces":[{"interface":"back","average":{"1min":1.3080000000000001,"5min":0.83699999999999997,"15min":0.65300000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":5.968,"5min":11.935,"15min":11.935},"last":3.992},{"interface":"front","average":{"1min":1.385,"5min":0.74299999999999999,"15min":0.64900000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":4.46,"5min":4.46,"15min":4.46},"last":2.8759999999999999}]},{"osd":7,"last update":"Wed Sep 11 08:11:22 2024","interfaces":[{"interface":"back","average":{"1min":2.3690000000000002,"5min":0.96599999999999997,"15min":0.73699999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":8.2929999999999993,"5min":8.2929999999999993,"15min":8.2929999999999993},"last":2.9009999999999998},{"interface":"front","average":{"1min":2.4079999999999999,"5min":0.97999999999999998,"15min":0.70399999999999996},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":9.0749999999999993,"5min":9.0749999999999993,"15min":9.0749999999999993},"last":8.6020000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738456,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3786984,"kb_used_data":3477148,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":89958168,"statfs":{"total":95995035648,"available":92117164032,"internally_reserved":0,"allocated":3560599552,"data_stored":3560184348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 11 08:11:11 2024","interfaces":[{"interface":"back","average":{"1min":2.0939999999999999,"5min":0.72599999999999998,"15min":0.443},"min":{"1min":0.189,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":6.0830000000000002,"5min":6.0830000000000002,"15min":6.0830000000000002},"last":0.79700000000000004},{"interface":"front","average":{"1min":1.8500000000000001,"5min":0.67200000000000004,"15min":0.442},"min":{"1min":0.254,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.0469999999999997,"5min":6.0469999999999997,"15min":6.0469999999999997},"last":0.58099999999999996}]},{"osd":2,"last update":"Wed Sep 11 08:11:36 2024","interfaces":[{"interface":"back","average":{"1min":2.161,"5min":0.85699999999999998,"15min":0.51500000000000001},"min":{"1min":0.375,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.1029999999999998,"5min":6.1029999999999998,"15min":6.1029999999999998},"last":0.63900000000000001},{"interface":"front","average":{"1min":2.5470000000000002,"5min":0.94699999999999995,"15min":0.55900000000000005},"min":{"1min":0.34300000000000003,"5min":0.217,"15min":0.217},"max":{"1min":6.3719999999999999,"5min":6.3719999999999999,"15min":6.3719999999999999},"last":0.81799999999999995}]},{"osd":3,"last update":"Wed Sep 11 08:10:56 2024","interfaces":[{"interface":"back","average":{"1min":3.0339999999999998,"5min":0.94599999999999995,"15min":0.60799999999999998},"min":{"1min":0.38,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":7.9699999999999998,"5min":7.9699999999999998,"15min":7.9699999999999998},"last":1.242},{"interface":"front","average":{"1min":1.889,"5min":0.69799999999999995,"15min":0.51700000000000002},"min":{"1min":0.312,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":6.923,"5min":6.923,"15min":6.923},"last":0.60799999999999998}]},{"osd":4,"last update":"Wed Sep 11 08:11:17 2024","interfaces":[{"interface":"back","average":{"1min":1.7150000000000001,"5min":0.78700000000000003,"15min":0.625},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":6.726,"5min":6.726,"15min":6.726},"last":0.70499999999999996},{"interface":"front","average":{"1min":1.76,"5min":0.78800000000000003,"15min":0.64800000000000002},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":5.6379999999999999,"5min":5.6379999999999999,"15min":5.6379999999999999},"last":1.1890000000000001}]},{"osd":5,"last update":"Wed Sep 11 08:10:41 2024","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":0.68899999999999995,"15min":0.61799999999999999},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":4.5490000000000004,"5min":4.5490000000000004,"15min":4.5490000000000004},"last":1.2190000000000001},{"interface":"front","average":{"1min":1.488,"5min":0.73599999999999999,"15min":0.61199999999999999},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":6.7939999999999996,"5min":6.7939999999999996,"15min":6.7939999999999996},"last":0.76400000000000001}]},{"osd":6,"last update":"Wed Sep 11 08:11:01 2024","interfaces":[{"interface":"back","average":{"1min":1.732,"5min":0.80200000000000005,"15min":0.68200000000000005},"min":{"1min":0.35799999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":6.8330000000000002,"5min":6.8330000000000002,"15min":6.8330000000000002},"last":0.72299999999999998},{"interface":"front","average":{"1min":2.0289999999999999,"5min":0.89400000000000002,"15min":0.66800000000000004},"min":{"1min":0.46700000000000003,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":6.1520000000000001,"5min":6.1520000000000001,"15min":6.1520000000000001},"last":0.65900000000000003}]},{"osd":7,"last update":"Wed Sep 11 08:11:24 2024","interfaces":[{"interface":"back","average":{"1min":1.5960000000000001,"5min":0.89000000000000001,"15min":0.70699999999999996},"min":{"1min":0.42199999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":6.2839999999999998,"5min":6.8650000000000002,"15min":6.8650000000000002},"last":0.745},{"interface":"front","average":{"1min":1.8240000000000001,"5min":0.93000000000000005,"15min":0.69399999999999995},"min":{"1min":0.43099999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":6.1719999999999997,"5min":6.1719999999999997,"15min":6.1719999999999997},"last":0.67700000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3558412288,"data_stored":3558408216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2660110336,"data_stored":2660106264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2558590976,"data_stored":2558590976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4006936576,"data_stored":4006936576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2137063424,"data_stored":2137063424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1708392448,"data_stored":1708392448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2121273344,"data_stored":2121269272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2125463552,"data_stored":2125463552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T08:11:48.778 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T08:11:48.788 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:48.854 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:48.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:48.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:48.918 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:48 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:11:48.919 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:48 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:11:48.919 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:48 smithi181 ceph-mon[32701]: osdmap e130: 8 total, 8 up, 8 in 2024-09-11T08:11:48.919 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:48 smithi181 ceph-mon[32701]: from='client.14910 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:11:48.919 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:48 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:11:48.956 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:48.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:48 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:11:48.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:48 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:11:48.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:48 smithi150 ceph-mon[29276]: osdmap e130: 8 total, 8 up, 8 in 2024-09-11T08:11:48.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:48 smithi150 ceph-mon[29276]: from='client.14910 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:11:48.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:48 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:11:48.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:48 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:11:48.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:48 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:11:48.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:48 smithi150 ceph-mon[39612]: osdmap e130: 8 total, 8 up, 8 in 2024-09-11T08:11:48.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:48 smithi150 ceph-mon[39612]: from='client.14910 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:11:48.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:48 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:11:49.020 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:49.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:49.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:49.122 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:11:49.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:49.175 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:49.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:11:49.276 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:11:49.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:49.306 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:49.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:11:49.408 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:49.414 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-11T08:11:49.415 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [], 2024-09-11T08:11:49.415 INFO:teuthology.orchestra.run.smithi150.stdout: "complaint_time": 30, 2024-09-11T08:11:49.415 INFO:teuthology.orchestra.run.smithi150.stdout: "num_blocked_ops": 0 2024-09-11T08:11:49.415 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-11T08:11:49.456 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:49.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:49.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:49.559 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:11:49.590 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:49.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:49.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:11:49.669 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:11:49.669 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:11:49.692 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:49.712 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:11:49.713 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:11:49.729 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:49.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:49.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:49.743 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:11:49.743 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:11:49.828 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:11:49.829 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:11:49.831 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:49.860 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:49.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:49.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:49.862 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:11:49.862 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:11:49.863 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:49 smithi150 ceph-mon[29276]: pgmap v171: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s 2024-09-11T08:11:49.863 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:49 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2194572562' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:11:49.863 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:49 smithi150 ceph-mon[39612]: pgmap v171: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s 2024-09-11T08:11:49.863 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:49 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2194572562' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:11:49.888 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:11:49.888 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:11:49.920 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:11:49.920 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:11:49.951 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:11:49.952 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:11:49.962 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:11:49.986 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:49.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:49.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:11:50.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:49 smithi181 ceph-mon[32701]: pgmap v171: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s 2024-09-11T08:11:50.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:49 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2194572562' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:11:50.089 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:11:50.120 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:50.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:50.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:11:50.222 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:50.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:50.290 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:50.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:50.391 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:50.432 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:50.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:50.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:50.534 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:50.571 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:50.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:50.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:50.673 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:11:50.680 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-11T08:11:50.700 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:50.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:50.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:11:50.802 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:11:50.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:50.829 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:50.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:11:50.930 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:50.969 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:50.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:50.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:51.071 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:51.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:51.122 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:51.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:51.223 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:11:51.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:51.260 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:51.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:11:51.361 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:51.387 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:51.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:51.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:51.489 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:51.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:51.517 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:51.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:51.618 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:51.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:51.648 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:51.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:51.750 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:51.777 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:51.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:51.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:51.879 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:11:51.908 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:51.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:51.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:11:52.010 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:11:52.038 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:52.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:52.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:11:52.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:51 smithi181 ceph-mon[32701]: pgmap v172: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-09-11T08:11:52.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:51 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:11:52.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:51 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:11:52.140 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:52.176 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:52.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:52.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:52.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:51 smithi150 ceph-mon[29276]: pgmap v172: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-09-11T08:11:52.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:51 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:11:52.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:51 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:11:52.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:51 smithi150 ceph-mon[39612]: pgmap v172: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-09-11T08:11:52.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:51 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:11:52.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:51 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:11:52.278 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:11:52.305 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:52.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:52.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:11:52.381 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:11:52.407 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:11:52.442 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:11:52.457 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:11:52.474 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:52.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:52.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:11:52.576 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:52.600 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:52.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:52.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:52.702 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:11:52.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:52.752 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:52.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:11:52.853 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:52.877 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:52.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:52.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:52.979 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:11:53.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:53.020 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:53.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:11:53.121 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:11:53.156 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:53.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:53.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:11:53.258 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:11:53.289 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:53.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:53.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:11:53.391 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:11:53.416 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:53.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:53.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:11:53.518 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:11:53.563 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:53.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:53.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:11:53.665 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:53.713 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:53.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:53.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:53.815 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:53.840 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:53.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:53.841 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:53.942 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:53.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:53.969 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:53.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:54.070 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:54.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:53 smithi181 ceph-mon[32701]: pgmap v173: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 993 op/s 2024-09-11T08:11:54.089 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:53 smithi150 ceph-mon[29276]: pgmap v173: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 993 op/s 2024-09-11T08:11:54.089 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:53 smithi150 ceph-mon[39612]: pgmap v173: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 993 op/s 2024-09-11T08:11:54.116 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:54.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:54.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:54.218 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:11:54.247 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:54.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:54.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:11:54.349 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:54.354 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T08:11:54.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:54.386 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:54.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:54.487 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:11:54.510 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:54.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:54.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:11:54.612 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:11:54.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:54.666 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:54.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:11:54.767 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:54.811 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:54.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:54.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:54.914 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:11:54.956 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:54.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:54.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:11:55.058 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:55.091 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:55.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:55.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:55.193 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:55.230 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:55.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:55.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:55.332 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:11:55.360 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:55.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:55.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:11:55.464 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:11:55.508 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:55.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:55.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:11:55.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:11:55 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:11:55] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:11:55.611 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:11:55.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:55.638 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:55.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:11:55.739 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:11:55.766 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:55.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:55.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:11:55.868 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:55.897 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:55.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:55.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:55.999 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:56.031 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:56.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:56.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:56.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:55 smithi181 ceph-mon[32701]: pgmap v174: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.00k op/s 2024-09-11T08:11:56.135 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:56.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:56.166 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:56.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:56.167 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:55 smithi150 ceph-mon[29276]: pgmap v174: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.00k op/s 2024-09-11T08:11:56.167 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:55 smithi150 ceph-mon[39612]: pgmap v174: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.00k op/s 2024-09-11T08:11:56.267 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:56.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:56.319 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:56.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:56.420 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:56.448 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:56.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:56.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:56.550 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:11:56.585 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:56.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:56.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:11:56.687 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:56.719 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:56.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:56.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:56.821 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:11:56.850 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:56.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:56.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:11:56.952 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:56.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:56.980 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:56.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:57.018 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:11:57.081 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:11:57.119 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:57.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:57.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:11:57.221 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:57.244 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:57.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:57.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:57.346 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:11:57.387 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:57.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:57.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:11:57.489 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:11:57.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:57.528 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:57.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:11:57.629 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:11:57.664 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:57.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:57.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:11:57.766 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:57.802 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:57.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:57.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:57.904 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:11:57.908 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:57 smithi150 ceph-mon[39612]: pgmap v175: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.00k op/s 2024-09-11T08:11:57.908 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:57 smithi150 ceph-mon[29276]: pgmap v175: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.00k op/s 2024-09-11T08:11:57.929 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:57.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:57.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:11:58.031 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:58.070 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:58.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:58.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:58.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:57 smithi181 ceph-mon[32701]: pgmap v175: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.00k op/s 2024-09-11T08:11:58.073 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:11:57 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:11:58.172 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:58.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:58.210 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:58.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:58.311 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:11:58.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:58.338 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:58.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:11:58.439 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:11:58.464 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:58.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:58.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:11:58.566 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:11:58.591 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:58.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:58.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:11:58.621 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:11:58.623 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-09-11T08:11:58.693 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:58.733 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:58.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:58.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:58.830 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:11:58 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:11:58] "GET /metrics HTTP/1.1" 200 223764 "" "Prometheus/2.43.0" 2024-09-11T08:11:58.835 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:58.874 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:58.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:58.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:58.976 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:59.025 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:59.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:59.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:59.126 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:59.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:59.159 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:59.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:59.197 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:58 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:11:59.197 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:58 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:11:59.260 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:59.278 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:58 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:11:59.292 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:59.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:59.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:59.394 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:11:59.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:59.426 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:59.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:11:59.528 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:11:59.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:59.566 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:59.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:11:59.667 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:11:59.691 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:59.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:59.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:11:59.793 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:11:59.835 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:11:59.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:59.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:11:59.847 INFO:teuthology.orchestra.run.smithi150.stderr:nodeep-scrub is set 2024-09-11T08:11:59.852 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":175,"stamp":"2024-09-11T08:11:56.756343+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6476077300,"num_objects":99005,"num_object_clones":0,"num_object_copies":297015,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99005,"num_whiteouts":0,"num_read":1110,"num_read_kb":967,"num_write":150612,"num_write_kb":7965704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":590698,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90499,"ondisk_log_size":90499,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":21515216,"kb_used_data":19069328,"kb_used_omap":13,"kb_used_meta":2445298,"kb_avail":728446000,"statfs":{"total":767960285184,"available":745928704000,"internally_reserved":0,"allocated":19526991872,"data_stored":19523731740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2503985856},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3],"upper_bound":8},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-656277504,"num_objects":-10014,"num_object_clones":0,"num_object_copies":-30042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-10014,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10014,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002908"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":100,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.430376+0000","last_change":"2024-09-11T08:10:19.111028+0000","last_active":"2024-09-11T08:11:42.430376+0000","last_peered":"2024-09-11T08:11:42.430376+0000","last_clean":"2024-09-11T08:11:42.430376+0000","last_became_active":"2024-09-11T08:07:30.044976+0000","last_became_peered":"2024-09-11T08:07:30.044976+0000","last_unstale":"2024-09-11T08:11:42.430376+0000","last_undegraded":"2024-09-11T08:11:42.430376+0000","last_fullsized":"2024-09-11T08:11:42.430376+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:19.110647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:10:19.110647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.017276071,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":104,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.419948+0000","last_change":"2024-09-11T08:09:24.630803+0000","last_active":"2024-09-11T08:11:42.419948+0000","last_peered":"2024-09-11T08:11:42.419948+0000","last_clean":"2024-09-11T08:11:42.419948+0000","last_became_active":"2024-09-11T08:07:32.048048+0000","last_became_peered":"2024-09-11T08:07:32.048048+0000","last_unstale":"2024-09-11T08:11:42.419948+0000","last_undegraded":"2024-09-11T08:11:42.419948+0000","last_fullsized":"2024-09-11T08:11:42.419948+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:09:24.630735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:09:24.630735+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020206859999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":106,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.424075+0000","last_change":"2024-09-11T08:11:40.568270+0000","last_active":"2024-09-11T08:11:42.424075+0000","last_peered":"2024-09-11T08:11:42.424075+0000","last_clean":"2024-09-11T08:11:42.424075+0000","last_became_active":"2024-09-11T08:07:34.058828+0000","last_became_peered":"2024-09-11T08:07:34.058828+0000","last_unstale":"2024-09-11T08:11:42.424075+0000","last_undegraded":"2024-09-11T08:11:42.424075+0000","last_fullsized":"2024-09-11T08:11:42.424075+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:40.567720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:11:40.567720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:13:05.932274+0000","scrub_duration":0.013465969,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":94,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.420206+0000","last_change":"2024-09-11T08:11:16.061718+0000","last_active":"2024-09-11T08:11:42.420206+0000","last_peered":"2024-09-11T08:11:42.420206+0000","last_clean":"2024-09-11T08:11:42.420206+0000","last_became_active":"2024-09-11T08:07:36.074789+0000","last_became_peered":"2024-09-11T08:07:36.074789+0000","last_unstale":"2024-09-11T08:11:42.420206+0000","last_undegraded":"2024-09-11T08:11:42.420206+0000","last_fullsized":"2024-09-11T08:11:42.420206+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:16.061669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:02.470649+0000","last_clean_scrub_stamp":"2024-09-11T08:11:16.061669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:25.199054+0000","scrub_duration":0.0051287930000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":94,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.430920+0000","last_change":"2024-09-11T08:11:34.567724+0000","last_active":"2024-09-11T08:11:42.430920+0000","last_peered":"2024-09-11T08:11:42.430920+0000","last_clean":"2024-09-11T08:11:42.430920+0000","last_became_active":"2024-09-11T08:07:36.633423+0000","last_became_peered":"2024-09-11T08:07:36.633423+0000","last_unstale":"2024-09-11T08:11:42.430920+0000","last_undegraded":"2024-09-11T08:11:42.430920+0000","last_fullsized":"2024-09-11T08:11:42.430920+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:34.567654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:11:34.567654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:13:02.653394+0000","scrub_duration":0.0064880140000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":90,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.431171+0000","last_change":"2024-09-11T08:09:35.348975+0000","last_active":"2024-09-11T08:11:42.431171+0000","last_peered":"2024-09-11T08:11:42.431171+0000","last_clean":"2024-09-11T08:11:42.431171+0000","last_became_active":"2024-09-11T08:07:30.045579+0000","last_became_peered":"2024-09-11T08:07:30.045579+0000","last_unstale":"2024-09-11T08:11:42.431171+0000","last_undegraded":"2024-09-11T08:11:42.431171+0000","last_fullsized":"2024-09-11T08:11:42.431171+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:35.348912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:09:35.348912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001468128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":123,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.430263+0000","last_change":"2024-09-11T08:11:19.427044+0000","last_active":"2024-09-11T08:11:42.430263+0000","last_peered":"2024-09-11T08:11:42.430263+0000","last_clean":"2024-09-11T08:11:42.430263+0000","last_became_active":"2024-09-11T08:07:32.045848+0000","last_became_peered":"2024-09-11T08:07:32.045848+0000","last_unstale":"2024-09-11T08:11:42.430263+0000","last_undegraded":"2024-09-11T08:11:42.430263+0000","last_fullsized":"2024-09-11T08:11:42.430263+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-11T08:11:19.426563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:11:19.426563+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:34.923254+0000","scrub_duration":0.021858643000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":86,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.419899+0000","last_change":"2024-09-11T08:09:31.638930+0000","last_active":"2024-09-11T08:11:42.419899+0000","last_peered":"2024-09-11T08:11:42.419899+0000","last_clean":"2024-09-11T08:11:42.419899+0000","last_became_active":"2024-09-11T08:07:34.052243+0000","last_became_peered":"2024-09-11T08:07:34.052243+0000","last_unstale":"2024-09-11T08:11:42.419899+0000","last_undegraded":"2024-09-11T08:11:42.419899+0000","last_fullsized":"2024-09-11T08:11:42.419899+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:31.638867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:09:31.638867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014270719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":99,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.428611+0000","last_change":"2024-09-11T08:10:26.020726+0000","last_active":"2024-09-11T08:11:42.428611+0000","last_peered":"2024-09-11T08:11:42.428611+0000","last_clean":"2024-09-11T08:11:42.428611+0000","last_became_active":"2024-09-11T08:07:30.043950+0000","last_became_peered":"2024-09-11T08:07:30.043950+0000","last_unstale":"2024-09-11T08:11:42.428611+0000","last_undegraded":"2024-09-11T08:11:42.428611+0000","last_fullsized":"2024-09-11T08:11:42.428611+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:26.020674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:54.878409+0000","last_clean_scrub_stamp":"2024-09-11T08:10:26.020674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001074111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":114,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.420472+0000","last_change":"2024-09-11T08:10:42.141214+0000","last_active":"2024-09-11T08:11:42.420472+0000","last_peered":"2024-09-11T08:11:42.420472+0000","last_clean":"2024-09-11T08:11:42.420472+0000","last_became_active":"2024-09-11T08:07:32.044626+0000","last_became_peered":"2024-09-11T08:07:32.044626+0000","last_unstale":"2024-09-11T08:11:42.420472+0000","last_undegraded":"2024-09-11T08:11:42.420472+0000","last_fullsized":"2024-09-11T08:11:42.420472+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:10:42.141115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:10:42.141115+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:52.487697+0000","scrub_duration":0.0076042940000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":95,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.428170+0000","last_change":"2024-09-11T08:10:44.820612+0000","last_active":"2024-09-11T08:11:42.428170+0000","last_peered":"2024-09-11T08:11:42.428170+0000","last_clean":"2024-09-11T08:11:42.428170+0000","last_became_active":"2024-09-11T08:07:34.058689+0000","last_became_peered":"2024-09-11T08:07:34.058689+0000","last_unstale":"2024-09-11T08:11:42.428170+0000","last_undegraded":"2024-09-11T08:11:42.428170+0000","last_fullsized":"2024-09-11T08:11:42.428170+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:44.820382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:10:44.820382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:09.539844+0000","scrub_duration":0.0067487449999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":94,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.425388+0000","last_change":"2024-09-11T08:10:16.101381+0000","last_active":"2024-09-11T08:11:42.425388+0000","last_peered":"2024-09-11T08:11:42.425388+0000","last_clean":"2024-09-11T08:11:42.425388+0000","last_became_active":"2024-09-11T08:07:36.076824+0000","last_became_peered":"2024-09-11T08:07:36.076824+0000","last_unstale":"2024-09-11T08:11:42.425388+0000","last_undegraded":"2024-09-11T08:11:42.425388+0000","last_fullsized":"2024-09-11T08:11:42.425388+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:16.101299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:10:16.101299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0062695850000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":99,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.428299+0000","last_change":"2024-09-11T08:10:15.984728+0000","last_active":"2024-09-11T08:11:42.428299+0000","last_peered":"2024-09-11T08:11:42.428299+0000","last_clean":"2024-09-11T08:11:42.428299+0000","last_became_active":"2024-09-11T08:07:30.043277+0000","last_became_peered":"2024-09-11T08:07:30.043277+0000","last_unstale":"2024-09-11T08:11:42.428299+0000","last_undegraded":"2024-09-11T08:11:42.428299+0000","last_fullsized":"2024-09-11T08:11:42.428299+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:15.984636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:10:15.984636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0055326200000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":98,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.420129+0000","last_change":"2024-09-11T08:09:08.668983+0000","last_active":"2024-09-11T08:11:42.420129+0000","last_peered":"2024-09-11T08:11:42.420129+0000","last_clean":"2024-09-11T08:11:42.420129+0000","last_became_active":"2024-09-11T08:07:32.048661+0000","last_became_peered":"2024-09-11T08:07:32.048661+0000","last_unstale":"2024-09-11T08:11:42.420129+0000","last_undegraded":"2024-09-11T08:11:42.420129+0000","last_fullsized":"2024-09-11T08:11:42.420129+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-11T08:09:08.668925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:09:08.668925+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001704496,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":96,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.430884+0000","last_change":"2024-09-11T08:11:20.459210+0000","last_active":"2024-09-11T08:11:42.430884+0000","last_peered":"2024-09-11T08:11:42.430884+0000","last_clean":"2024-09-11T08:11:42.430884+0000","last_became_active":"2024-09-11T08:07:34.060871+0000","last_became_peered":"2024-09-11T08:07:34.060871+0000","last_unstale":"2024-09-11T08:11:42.430884+0000","last_undegraded":"2024-09-11T08:11:42.430884+0000","last_fullsized":"2024-09-11T08:11:42.430884+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:20.459128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:11:20.459128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:44.755722+0000","scrub_duration":0.0035582639999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":93,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.428145+0000","last_change":"2024-09-11T08:10:37.891841+0000","last_active":"2024-09-11T08:11:42.428145+0000","last_peered":"2024-09-11T08:11:42.428145+0000","last_clean":"2024-09-11T08:11:42.428145+0000","last_became_active":"2024-09-11T08:07:36.077602+0000","last_became_peered":"2024-09-11T08:07:36.077602+0000","last_unstale":"2024-09-11T08:11:42.428145+0000","last_undegraded":"2024-09-11T08:11:42.428145+0000","last_fullsized":"2024-09-11T08:11:42.428145+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:37.891755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:04.782577+0000","last_clean_scrub_stamp":"2024-09-11T08:10:37.891755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0069411969999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":94,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.420185+0000","last_change":"2024-09-11T08:11:38.063359+0000","last_active":"2024-09-11T08:11:42.420185+0000","last_peered":"2024-09-11T08:11:42.420185+0000","last_clean":"2024-09-11T08:11:42.420185+0000","last_became_active":"2024-09-11T08:07:36.633338+0000","last_became_peered":"2024-09-11T08:07:36.633338+0000","last_unstale":"2024-09-11T08:11:42.420185+0000","last_undegraded":"2024-09-11T08:11:42.420185+0000","last_fullsized":"2024-09-11T08:11:42.420185+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:38.063310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:11:38.063310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:44.770226+0000","scrub_duration":0.0013449110000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":90,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.426577+0000","last_change":"2024-09-11T08:09:11.966738+0000","last_active":"2024-09-11T08:11:42.426577+0000","last_peered":"2024-09-11T08:11:42.426577+0000","last_clean":"2024-09-11T08:11:42.426577+0000","last_became_active":"2024-09-11T08:07:30.045739+0000","last_became_peered":"2024-09-11T08:07:30.045739+0000","last_unstale":"2024-09-11T08:11:42.426577+0000","last_undegraded":"2024-09-11T08:11:42.426577+0000","last_fullsized":"2024-09-11T08:11:42.426577+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:11.966683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:09:11.966683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015807659999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"107'22","reported_seq":129,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.438483+0000","last_change":"2024-09-11T08:10:30.941599+0000","last_active":"2024-09-11T08:11:42.438483+0000","last_peered":"2024-09-11T08:11:42.438483+0000","last_clean":"2024-09-11T08:11:42.438483+0000","last_became_active":"2024-09-11T08:07:32.046363+0000","last_became_peered":"2024-09-11T08:07:32.046363+0000","last_unstale":"2024-09-11T08:11:42.438483+0000","last_undegraded":"2024-09-11T08:11:42.438483+0000","last_fullsized":"2024-09-11T08:11:42.438483+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'22","last_scrub_stamp":"2024-09-11T08:10:30.941505+0000","last_deep_scrub":"107'22","last_deep_scrub_stamp":"2024-09-11T08:10:30.941505+0000","last_clean_scrub_stamp":"2024-09-11T08:10:30.941505+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:53.254453+0000","scrub_duration":0.02240839,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":96,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.429012+0000","last_change":"2024-09-11T08:10:31.418953+0000","last_active":"2024-09-11T08:11:42.429012+0000","last_peered":"2024-09-11T08:11:42.429012+0000","last_clean":"2024-09-11T08:11:42.429012+0000","last_became_active":"2024-09-11T08:07:34.055410+0000","last_became_peered":"2024-09-11T08:07:34.055410+0000","last_unstale":"2024-09-11T08:11:42.429012+0000","last_undegraded":"2024-09-11T08:11:42.429012+0000","last_fullsized":"2024-09-11T08:11:42.429012+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:31.418873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:10:31.418873+0000","last_clean_scrub_stamp":"2024-09-11T08:10:31.418873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011564522000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":84,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.430596+0000","last_change":"2024-09-11T08:10:31.188608+0000","last_active":"2024-09-11T08:11:42.430596+0000","last_peered":"2024-09-11T08:11:42.430596+0000","last_clean":"2024-09-11T08:11:42.430596+0000","last_became_active":"2024-09-11T08:07:36.633258+0000","last_became_peered":"2024-09-11T08:07:36.633258+0000","last_unstale":"2024-09-11T08:11:42.430596+0000","last_undegraded":"2024-09-11T08:11:42.430596+0000","last_fullsized":"2024-09-11T08:11:42.430596+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:31.188518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:10:31.188518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:00.813319+0000","scrub_duration":0.013170371,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":100,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.420620+0000","last_change":"2024-09-11T08:11:17.089124+0000","last_active":"2024-09-11T08:11:42.420620+0000","last_peered":"2024-09-11T08:11:42.420620+0000","last_clean":"2024-09-11T08:11:42.420620+0000","last_became_active":"2024-09-11T08:07:30.046284+0000","last_became_peered":"2024-09-11T08:07:30.046284+0000","last_unstale":"2024-09-11T08:11:42.420620+0000","last_undegraded":"2024-09-11T08:11:42.420620+0000","last_fullsized":"2024-09-11T08:11:42.420620+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:17.089046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:11:17.089046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:21.125842+0000","scrub_duration":0.0042934460000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":143,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.429732+0000","last_change":"2024-09-11T08:11:32.559837+0000","last_active":"2024-09-11T08:11:42.429732+0000","last_peered":"2024-09-11T08:11:42.429732+0000","last_clean":"2024-09-11T08:11:42.429732+0000","last_became_active":"2024-09-11T08:09:07.878432+0000","last_became_peered":"2024-09-11T08:09:07.878432+0000","last_unstale":"2024-09-11T08:11:42.429732+0000","last_undegraded":"2024-09-11T08:11:42.429732+0000","last_fullsized":"2024-09-11T08:11:42.429732+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-11T08:11:32.559751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:11:32.559751+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:48.508018+0000","scrub_duration":0.0059216119999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":96,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.420621+0000","last_change":"2024-09-11T08:10:40.110540+0000","last_active":"2024-09-11T08:11:42.420621+0000","last_peered":"2024-09-11T08:11:42.420621+0000","last_clean":"2024-09-11T08:11:42.420621+0000","last_became_active":"2024-09-11T08:07:34.056528+0000","last_became_peered":"2024-09-11T08:07:34.056528+0000","last_unstale":"2024-09-11T08:11:42.420621+0000","last_undegraded":"2024-09-11T08:11:42.420621+0000","last_fullsized":"2024-09-11T08:11:42.420621+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:40.110490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:10:40.110490+0000","last_clean_scrub_stamp":"2024-09-11T08:10:40.110490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:53.563251+0000","scrub_duration":0.00164006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":95,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.432008+0000","last_change":"2024-09-11T08:10:41.305921+0000","last_active":"2024-09-11T08:11:42.432008+0000","last_peered":"2024-09-11T08:11:42.432008+0000","last_clean":"2024-09-11T08:11:42.432008+0000","last_became_active":"2024-09-11T08:07:36.072347+0000","last_became_peered":"2024-09-11T08:07:36.072347+0000","last_unstale":"2024-09-11T08:11:42.432008+0000","last_undegraded":"2024-09-11T08:11:42.432008+0000","last_fullsized":"2024-09-11T08:11:42.432008+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-11T08:10:41.305866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:10:41.305866+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:57.336810+0000","scrub_duration":0.0045172119999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":100,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.424183+0000","last_change":"2024-09-11T08:10:35.345635+0000","last_active":"2024-09-11T08:11:42.424183+0000","last_peered":"2024-09-11T08:11:42.424183+0000","last_clean":"2024-09-11T08:11:42.424183+0000","last_became_active":"2024-09-11T08:07:30.045932+0000","last_became_peered":"2024-09-11T08:07:30.045932+0000","last_unstale":"2024-09-11T08:11:42.424183+0000","last_undegraded":"2024-09-11T08:11:42.424183+0000","last_fullsized":"2024-09-11T08:11:42.424183+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:35.345484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:10:35.345484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:59.513234+0000","scrub_duration":0.010438901,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":113,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.425582+0000","last_change":"2024-09-11T08:11:30.913167+0000","last_active":"2024-09-11T08:11:42.425582+0000","last_peered":"2024-09-11T08:11:42.425582+0000","last_clean":"2024-09-11T08:11:42.425582+0000","last_became_active":"2024-09-11T08:07:32.048022+0000","last_became_peered":"2024-09-11T08:07:32.048022+0000","last_unstale":"2024-09-11T08:11:42.425582+0000","last_undegraded":"2024-09-11T08:11:42.425582+0000","last_fullsized":"2024-09-11T08:11:42.425582+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-11T08:11:30.913083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:11:30.913083+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:56.240332+0000","scrub_duration":0.0071376130000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":168,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-11T08:11:47.800632+0000","last_change":"2024-09-11T08:11:28.666314+0000","last_active":"2024-09-11T08:11:47.800632+0000","last_peered":"2024-09-11T08:11:47.800632+0000","last_clean":"2024-09-11T08:11:47.800632+0000","last_became_active":"2024-09-11T08:07:34.054014+0000","last_became_peered":"2024-09-11T08:07:34.054014+0000","last_unstale":"2024-09-11T08:11:47.800632+0000","last_undegraded":"2024-09-11T08:11:47.800632+0000","last_fullsized":"2024-09-11T08:11:47.800632+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:11:28.666260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:11:28.666260+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:35.716682+0000","scrub_duration":0.001258885,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"130'9260","reported_seq":9623,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-11T08:11:52.681939+0000","last_change":"2024-09-11T08:11:28.767321+0000","last_active":"2024-09-11T08:11:52.681939+0000","last_peered":"2024-09-11T08:11:52.681939+0000","last_clean":"2024-09-11T08:11:52.681939+0000","last_became_active":"2024-09-11T08:09:45.414744+0000","last_became_peered":"2024-09-11T08:09:45.414744+0000","last_unstale":"2024-09-11T08:11:52.681939+0000","last_undegraded":"2024-09-11T08:11:52.681939+0000","last_fullsized":"2024-09-11T08:11:52.681939+0000","mapping_epoch":91,"log_start":"102'3000","ondisk_log_start":"102'3000","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'7721","last_scrub_stamp":"2024-09-11T08:11:28.767232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_clean_scrub_stamp":"2024-09-11T08:11:28.767232+0000","objects_scrubbed":7645,"log_size":6260,"log_dups_size":0,"ondisk_log_size":6260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:48.988510+0000","scrub_duration":2.190530367,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400031744,"num_objects":6104,"num_object_clones":0,"num_object_copies":18312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9260,"num_write_kb":491648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":123,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.428016+0000","last_change":"2024-09-11T08:10:23.428879+0000","last_active":"2024-09-11T08:11:42.428016+0000","last_peered":"2024-09-11T08:11:42.428016+0000","last_clean":"2024-09-11T08:11:42.428016+0000","last_became_active":"2024-09-11T08:07:32.054317+0000","last_became_peered":"2024-09-11T08:07:32.054317+0000","last_unstale":"2024-09-11T08:11:42.428016+0000","last_undegraded":"2024-09-11T08:11:42.428016+0000","last_fullsized":"2024-09-11T08:11:42.428016+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-11T08:10:23.428796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:10:23.428796+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014719733,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":90,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.436808+0000","last_change":"2024-09-11T08:10:16.250853+0000","last_active":"2024-09-11T08:11:42.436808+0000","last_peered":"2024-09-11T08:11:42.436808+0000","last_clean":"2024-09-11T08:11:42.436808+0000","last_became_active":"2024-09-11T08:07:30.044624+0000","last_became_peered":"2024-09-11T08:07:30.044624+0000","last_unstale":"2024-09-11T08:11:42.436808+0000","last_undegraded":"2024-09-11T08:11:42.436808+0000","last_fullsized":"2024-09-11T08:11:42.436808+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:16.247162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:10:16.247162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065916309999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":156,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-11T08:11:47.793288+0000","last_change":"2024-09-11T08:11:27.139877+0000","last_active":"2024-09-11T08:11:47.793288+0000","last_peered":"2024-09-11T08:11:47.793288+0000","last_clean":"2024-09-11T08:11:47.793288+0000","last_became_active":"2024-09-11T08:07:34.052981+0000","last_became_peered":"2024-09-11T08:07:34.052981+0000","last_unstale":"2024-09-11T08:11:47.793288+0000","last_undegraded":"2024-09-11T08:11:47.793288+0000","last_fullsized":"2024-09-11T08:11:47.793288+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:11:27.139796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:11:27.139796+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:48.327854+0000","scrub_duration":0.0080431200000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":103,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.428368+0000","last_change":"2024-09-11T08:11:39.747151+0000","last_active":"2024-09-11T08:11:42.428368+0000","last_peered":"2024-09-11T08:11:42.428368+0000","last_clean":"2024-09-11T08:11:42.428368+0000","last_became_active":"2024-09-11T08:07:36.073969+0000","last_became_peered":"2024-09-11T08:07:36.073969+0000","last_unstale":"2024-09-11T08:11:42.428368+0000","last_undegraded":"2024-09-11T08:11:42.428368+0000","last_fullsized":"2024-09-11T08:11:42.428368+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:39.747073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:11:39.747073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:42.671932+0000","scrub_duration":0.0080670670000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"130'9432","reported_seq":9489,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-11T08:11:52.770007+0000","last_change":"2024-09-11T08:11:42.436912+0000","last_active":"2024-09-11T08:11:52.770007+0000","last_peered":"2024-09-11T08:11:52.770007+0000","last_clean":"2024-09-11T08:11:52.770007+0000","last_became_active":"2024-09-11T08:09:45.412326+0000","last_became_peered":"2024-09-11T08:09:45.412326+0000","last_unstale":"2024-09-11T08:11:52.770007+0000","last_undegraded":"2024-09-11T08:11:52.770007+0000","last_fullsized":"2024-09-11T08:11:52.770007+0000","mapping_epoch":91,"log_start":"111'4800","ondisk_log_start":"111'4800","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_clean_scrub_stamp":"2024-09-11T08:09:44.385805+0000","objects_scrubbed":99,"log_size":4632,"log_dups_size":0,"ondisk_log_size":4632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":407633920,"num_objects":6220,"num_object_clones":0,"num_object_copies":18660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9432,"num_write_kb":500864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":130,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.436586+0000","last_change":"2024-09-11T08:10:35.187425+0000","last_active":"2024-09-11T08:11:42.436586+0000","last_peered":"2024-09-11T08:11:42.436586+0000","last_clean":"2024-09-11T08:11:42.436586+0000","last_became_active":"2024-09-11T08:07:32.045332+0000","last_became_peered":"2024-09-11T08:07:32.045332+0000","last_unstale":"2024-09-11T08:11:42.436586+0000","last_undegraded":"2024-09-11T08:11:42.436586+0000","last_fullsized":"2024-09-11T08:11:42.436586+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-11T08:10:35.187322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:10:35.187322+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:02.558401+0000","scrub_duration":0.016336040999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":90,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.430949+0000","last_change":"2024-09-11T08:10:24.286689+0000","last_active":"2024-09-11T08:11:42.430949+0000","last_peered":"2024-09-11T08:11:42.430949+0000","last_clean":"2024-09-11T08:11:42.430949+0000","last_became_active":"2024-09-11T08:07:30.047176+0000","last_became_peered":"2024-09-11T08:07:30.047176+0000","last_unstale":"2024-09-11T08:11:42.430949+0000","last_undegraded":"2024-09-11T08:11:42.430949+0000","last_fullsized":"2024-09-11T08:11:42.430949+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:24.286622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:10:24.286622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.022756537,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":96,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.424762+0000","last_change":"2024-09-11T08:11:26.181015+0000","last_active":"2024-09-11T08:11:42.424762+0000","last_peered":"2024-09-11T08:11:42.424762+0000","last_clean":"2024-09-11T08:11:42.424762+0000","last_became_active":"2024-09-11T08:07:34.055523+0000","last_became_peered":"2024-09-11T08:07:34.055523+0000","last_unstale":"2024-09-11T08:11:42.424762+0000","last_undegraded":"2024-09-11T08:11:42.424762+0000","last_fullsized":"2024-09-11T08:11:42.424762+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:26.180933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:11:26.180933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:31.314789+0000","scrub_duration":0.0022304540000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":84,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.420388+0000","last_change":"2024-09-11T08:09:21.618824+0000","last_active":"2024-09-11T08:11:42.420388+0000","last_peered":"2024-09-11T08:11:42.420388+0000","last_clean":"2024-09-11T08:11:42.420388+0000","last_became_active":"2024-09-11T08:07:36.069225+0000","last_became_peered":"2024-09-11T08:07:36.069225+0000","last_unstale":"2024-09-11T08:11:42.420388+0000","last_undegraded":"2024-09-11T08:11:42.420388+0000","last_fullsized":"2024-09-11T08:11:42.420388+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:21.618769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:09:21.618769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001637814,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"130'9312","reported_seq":9431,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-11T08:11:52.824189+0000","last_change":"2024-09-11T08:11:42.431777+0000","last_active":"2024-09-11T08:11:52.824189+0000","last_peered":"2024-09-11T08:11:52.824189+0000","last_clean":"2024-09-11T08:11:52.824189+0000","last_became_active":"2024-09-11T08:09:45.414656+0000","last_became_peered":"2024-09-11T08:09:45.414656+0000","last_unstale":"2024-09-11T08:11:52.824189+0000","last_undegraded":"2024-09-11T08:11:52.824189+0000","last_fullsized":"2024-09-11T08:11:52.824189+0000","mapping_epoch":91,"log_start":"110'4600","ondisk_log_start":"110'4600","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_clean_scrub_stamp":"2024-09-11T08:09:44.385805+0000","objects_scrubbed":1649,"log_size":4712,"log_dups_size":0,"ondisk_log_size":4712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394657792,"num_objects":6022,"num_object_clones":0,"num_object_copies":18066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9312,"num_write_kb":490688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":109,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.424902+0000","last_change":"2024-09-11T08:09:11.688215+0000","last_active":"2024-09-11T08:11:42.424902+0000","last_peered":"2024-09-11T08:11:42.424902+0000","last_clean":"2024-09-11T08:11:42.424902+0000","last_became_active":"2024-09-11T08:07:32.040069+0000","last_became_peered":"2024-09-11T08:07:32.040069+0000","last_unstale":"2024-09-11T08:11:42.424902+0000","last_undegraded":"2024-09-11T08:11:42.424902+0000","last_fullsized":"2024-09-11T08:11:42.424902+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-11T08:09:11.688151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:09:11.688151+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019655670000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":100,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.425796+0000","last_change":"2024-09-11T08:11:33.964196+0000","last_active":"2024-09-11T08:11:42.425796+0000","last_peered":"2024-09-11T08:11:42.425796+0000","last_clean":"2024-09-11T08:11:42.425796+0000","last_became_active":"2024-09-11T08:07:30.050484+0000","last_became_peered":"2024-09-11T08:07:30.050484+0000","last_unstale":"2024-09-11T08:11:42.425796+0000","last_undegraded":"2024-09-11T08:11:42.425796+0000","last_fullsized":"2024-09-11T08:11:42.425796+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:33.964053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:11:33.964053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:40.373468+0000","scrub_duration":0.003641151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"82'9","reported_seq":160,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-11T08:11:47.801500+0000","last_change":"2024-09-11T08:11:25.434358+0000","last_active":"2024-09-11T08:11:47.801500+0000","last_peered":"2024-09-11T08:11:47.801500+0000","last_clean":"2024-09-11T08:11:47.801500+0000","last_became_active":"2024-09-11T08:07:34.056143+0000","last_became_peered":"2024-09-11T08:07:34.056143+0000","last_unstale":"2024-09-11T08:11:47.801500+0000","last_undegraded":"2024-09-11T08:11:47.801500+0000","last_fullsized":"2024-09-11T08:11:47.801500+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-11T08:11:25.434210+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-09-11T08:08:53.421587+0000","last_clean_scrub_stamp":"2024-09-11T08:11:25.434210+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:46.551085+0000","scrub_duration":0.0043836509999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":94,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.430910+0000","last_change":"2024-09-11T08:10:29.251333+0000","last_active":"2024-09-11T08:11:42.430910+0000","last_peered":"2024-09-11T08:11:42.430910+0000","last_clean":"2024-09-11T08:11:42.430910+0000","last_became_active":"2024-09-11T08:07:36.633432+0000","last_became_peered":"2024-09-11T08:07:36.633432+0000","last_unstale":"2024-09-11T08:11:42.430910+0000","last_undegraded":"2024-09-11T08:11:42.430910+0000","last_fullsized":"2024-09-11T08:11:42.430910+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:29.251257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:10:29.251257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023785080000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"130'9433","reported_seq":9480,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-11T08:11:52.693047+0000","last_change":"2024-09-11T08:09:45.410430+0000","last_active":"2024-09-11T08:11:52.693047+0000","last_peered":"2024-09-11T08:11:52.693047+0000","last_clean":"2024-09-11T08:11:52.693047+0000","last_became_active":"2024-09-11T08:09:45.410129+0000","last_became_peered":"2024-09-11T08:09:45.410129+0000","last_unstale":"2024-09-11T08:11:52.693047+0000","last_undegraded":"2024-09-11T08:11:52.693047+0000","last_fullsized":"2024-09-11T08:11:52.693047+0000","mapping_epoch":91,"log_start":"110'4600","ondisk_log_start":"110'4600","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_clean_scrub_stamp":"2024-09-11T08:09:44.385805+0000","objects_scrubbed":0,"log_size":4833,"log_dups_size":0,"ondisk_log_size":4833,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":407699456,"num_objects":6221,"num_object_clones":0,"num_object_copies":18663,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9433,"num_write_kb":500928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":113,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.426595+0000","last_change":"2024-09-11T08:11:22.581262+0000","last_active":"2024-09-11T08:11:42.426595+0000","last_peered":"2024-09-11T08:11:42.426595+0000","last_clean":"2024-09-11T08:11:42.426595+0000","last_became_active":"2024-09-11T08:07:32.054294+0000","last_became_peered":"2024-09-11T08:07:32.054294+0000","last_unstale":"2024-09-11T08:11:42.426595+0000","last_undegraded":"2024-09-11T08:11:42.426595+0000","last_fullsized":"2024-09-11T08:11:42.426595+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-11T08:11:22.581035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:11:22.581035+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:43.316970+0000","scrub_duration":0.0076992600000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":99,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.428738+0000","last_change":"2024-09-11T08:10:35.913323+0000","last_active":"2024-09-11T08:11:42.428738+0000","last_peered":"2024-09-11T08:11:42.428738+0000","last_clean":"2024-09-11T08:11:42.428738+0000","last_became_active":"2024-09-11T08:07:30.047497+0000","last_became_peered":"2024-09-11T08:07:30.047497+0000","last_unstale":"2024-09-11T08:11:42.428738+0000","last_undegraded":"2024-09-11T08:11:42.428738+0000","last_fullsized":"2024-09-11T08:11:42.428738+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:35.913239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:10:35.913239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:00.244681+0000","scrub_duration":0.007393714,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":156,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-11T08:11:47.804003+0000","last_change":"2024-09-11T08:11:38.925253+0000","last_active":"2024-09-11T08:11:47.804003+0000","last_peered":"2024-09-11T08:11:47.804003+0000","last_clean":"2024-09-11T08:11:47.804003+0000","last_became_active":"2024-09-11T08:07:34.060660+0000","last_became_peered":"2024-09-11T08:07:34.060660+0000","last_unstale":"2024-09-11T08:11:47.804003+0000","last_undegraded":"2024-09-11T08:11:47.804003+0000","last_fullsized":"2024-09-11T08:11:47.804003+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:11:38.925167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:11:38.925167+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:44.287481+0000","scrub_duration":0.0080128060000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"66'1","reported_seq":85,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.425531+0000","last_change":"2024-09-11T08:09:25.767659+0000","last_active":"2024-09-11T08:11:42.425531+0000","last_peered":"2024-09-11T08:11:42.425531+0000","last_clean":"2024-09-11T08:11:42.425531+0000","last_became_active":"2024-09-11T08:07:36.072593+0000","last_became_peered":"2024-09-11T08:07:36.072593+0000","last_unstale":"2024-09-11T08:11:42.425531+0000","last_undegraded":"2024-09-11T08:11:42.425531+0000","last_fullsized":"2024-09-11T08:11:42.425531+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-11T08:09:25.767603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:09:25.767603+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001492487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"130'9142","reported_seq":9487,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-11T08:11:52.617520+0000","last_change":"2024-09-11T08:11:22.612314+0000","last_active":"2024-09-11T08:11:52.617520+0000","last_peered":"2024-09-11T08:11:52.617520+0000","last_clean":"2024-09-11T08:11:52.617520+0000","last_became_active":"2024-09-11T08:09:45.415564+0000","last_became_peered":"2024-09-11T08:09:45.415564+0000","last_unstale":"2024-09-11T08:11:52.617520+0000","last_undegraded":"2024-09-11T08:11:52.617520+0000","last_fullsized":"2024-09-11T08:11:52.617520+0000","mapping_epoch":91,"log_start":"105'3300","ondisk_log_start":"105'3300","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'7301","last_scrub_stamp":"2024-09-11T08:11:22.612230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_clean_scrub_stamp":"2024-09-11T08:11:22.612230+0000","objects_scrubbed":7217,"log_size":5842,"log_dups_size":0,"ondisk_log_size":5842,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:51.523019+0000","scrub_duration":2.8618551820000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402259968,"num_objects":6138,"num_object_clones":0,"num_object_copies":18414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9142,"num_write_kb":488960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":134,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.426366+0000","last_change":"2024-09-11T08:11:19.982267+0000","last_active":"2024-09-11T08:11:42.426366+0000","last_peered":"2024-09-11T08:11:42.426366+0000","last_clean":"2024-09-11T08:11:42.426366+0000","last_became_active":"2024-09-11T08:07:32.044376+0000","last_became_peered":"2024-09-11T08:07:32.044376+0000","last_unstale":"2024-09-11T08:11:42.426366+0000","last_undegraded":"2024-09-11T08:11:42.426366+0000","last_fullsized":"2024-09-11T08:11:42.426366+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-11T08:11:19.982185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:11:19.982185+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:38.948134+0000","scrub_duration":0.0066179189999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":100,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.427147+0000","last_change":"2024-09-11T08:10:22.436344+0000","last_active":"2024-09-11T08:11:42.427147+0000","last_peered":"2024-09-11T08:11:42.427147+0000","last_clean":"2024-09-11T08:11:42.427147+0000","last_became_active":"2024-09-11T08:07:30.045380+0000","last_became_peered":"2024-09-11T08:07:30.045380+0000","last_unstale":"2024-09-11T08:11:42.427147+0000","last_undegraded":"2024-09-11T08:11:42.427147+0000","last_fullsized":"2024-09-11T08:11:42.427147+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:22.436250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:10:22.436250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0076618179999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":95,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.423517+0000","last_change":"2024-09-11T08:10:26.621521+0000","last_active":"2024-09-11T08:11:42.423517+0000","last_peered":"2024-09-11T08:11:42.423517+0000","last_clean":"2024-09-11T08:11:42.423517+0000","last_became_active":"2024-09-11T08:07:34.051004+0000","last_became_peered":"2024-09-11T08:07:34.051004+0000","last_unstale":"2024-09-11T08:11:42.423517+0000","last_undegraded":"2024-09-11T08:11:42.423517+0000","last_fullsized":"2024-09-11T08:11:42.423517+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:26.621460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:10:26.621460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016671699999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":94,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.424013+0000","last_change":"2024-09-11T08:11:36.477823+0000","last_active":"2024-09-11T08:11:42.424013+0000","last_peered":"2024-09-11T08:11:42.424013+0000","last_clean":"2024-09-11T08:11:42.424013+0000","last_became_active":"2024-09-11T08:07:36.631668+0000","last_became_peered":"2024-09-11T08:07:36.631668+0000","last_unstale":"2024-09-11T08:11:42.424013+0000","last_undegraded":"2024-09-11T08:11:42.424013+0000","last_fullsized":"2024-09-11T08:11:42.424013+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:36.477734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:11:36.477734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:13:01.973259+0000","scrub_duration":0.0067568719999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"130'9460","reported_seq":9824,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-11T08:11:52.682032+0000","last_change":"2024-09-11T08:11:27.447825+0000","last_active":"2024-09-11T08:11:52.682032+0000","last_peered":"2024-09-11T08:11:52.682032+0000","last_clean":"2024-09-11T08:11:52.682032+0000","last_became_active":"2024-09-11T08:09:45.414847+0000","last_became_peered":"2024-09-11T08:09:45.414847+0000","last_unstale":"2024-09-11T08:11:52.682032+0000","last_undegraded":"2024-09-11T08:11:52.682032+0000","last_fullsized":"2024-09-11T08:11:52.682032+0000","mapping_epoch":91,"log_start":"103'3200","ondisk_log_start":"103'3200","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'7788","last_scrub_stamp":"2024-09-11T08:11:27.447746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_clean_scrub_stamp":"2024-09-11T08:11:27.447746+0000","objects_scrubbed":7674,"log_size":6260,"log_dups_size":0,"ondisk_log_size":6260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:47.727446+0000","scrub_duration":3.9345051240000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404750336,"num_objects":6176,"num_object_clones":0,"num_object_copies":18528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9460,"num_write_kb":500352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"86'38","reported_seq":208,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-11T08:11:47.815876+0000","last_change":"2024-09-11T08:11:16.652708+0000","last_active":"2024-09-11T08:11:47.815876+0000","last_peered":"2024-09-11T08:11:47.815876+0000","last_clean":"2024-09-11T08:11:47.815876+0000","last_became_active":"2024-09-11T08:07:32.052956+0000","last_became_peered":"2024-09-11T08:07:32.052956+0000","last_unstale":"2024-09-11T08:11:47.815876+0000","last_undegraded":"2024-09-11T08:11:47.815876+0000","last_fullsized":"2024-09-11T08:11:47.815876+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'38","last_scrub_stamp":"2024-09-11T08:11:16.652273+0000","last_deep_scrub":"86'38","last_deep_scrub_stamp":"2024-09-11T08:09:11.362339+0000","last_clean_scrub_stamp":"2024-09-11T08:11:16.652273+0000","objects_scrubbed":10,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:46.242558+0000","scrub_duration":0.011886363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"59'1","reported_seq":109,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.420430+0000","last_change":"2024-09-11T08:10:18.051791+0000","last_active":"2024-09-11T08:11:42.420430+0000","last_peered":"2024-09-11T08:11:42.420430+0000","last_clean":"2024-09-11T08:11:42.420430+0000","last_became_active":"2024-09-11T08:07:30.047513+0000","last_became_peered":"2024-09-11T08:07:30.047513+0000","last_unstale":"2024-09-11T08:11:42.420430+0000","last_undegraded":"2024-09-11T08:11:42.420430+0000","last_fullsized":"2024-09-11T08:11:42.420430+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-11T08:10:18.051701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:10:18.051701+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028800570000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":106,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.428189+0000","last_change":"2024-09-11T08:10:16.989515+0000","last_active":"2024-09-11T08:11:42.428189+0000","last_peered":"2024-09-11T08:11:42.428189+0000","last_clean":"2024-09-11T08:11:42.428189+0000","last_became_active":"2024-09-11T08:07:30.043130+0000","last_became_peered":"2024-09-11T08:07:30.043130+0000","last_unstale":"2024-09-11T08:11:42.428189+0000","last_undegraded":"2024-09-11T08:11:42.428189+0000","last_fullsized":"2024-09-11T08:11:42.428189+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-11T08:10:16.989458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:10:16.989458+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033825330000000001,"objects_trimmed":0,"snaptrim_duration":0.0026120850000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":96,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.426726+0000","last_change":"2024-09-11T08:10:37.957177+0000","last_active":"2024-09-11T08:11:42.426726+0000","last_peered":"2024-09-11T08:11:42.426726+0000","last_clean":"2024-09-11T08:11:42.426726+0000","last_became_active":"2024-09-11T08:07:34.057154+0000","last_became_peered":"2024-09-11T08:07:34.057154+0000","last_unstale":"2024-09-11T08:11:42.426726+0000","last_undegraded":"2024-09-11T08:11:42.426726+0000","last_fullsized":"2024-09-11T08:11:42.426726+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:37.957096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:10:37.957096+0000","last_clean_scrub_stamp":"2024-09-11T08:10:37.957096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:47.854954+0000","scrub_duration":0.011314437,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":94,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.430867+0000","last_change":"2024-09-11T08:11:38.502257+0000","last_active":"2024-09-11T08:11:42.430867+0000","last_peered":"2024-09-11T08:11:42.430867+0000","last_clean":"2024-09-11T08:11:42.430867+0000","last_became_active":"2024-09-11T08:07:36.075690+0000","last_became_peered":"2024-09-11T08:07:36.075690+0000","last_unstale":"2024-09-11T08:11:42.430867+0000","last_undegraded":"2024-09-11T08:11:42.430867+0000","last_fullsized":"2024-09-11T08:11:42.430867+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:38.502182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:11:38.502182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:55.269821+0000","scrub_duration":0.0039179619999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"130'9285","reported_seq":9332,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-11T08:11:52.962042+0000","last_change":"2024-09-11T08:09:45.414559+0000","last_active":"2024-09-11T08:11:52.962042+0000","last_peered":"2024-09-11T08:11:52.962042+0000","last_clean":"2024-09-11T08:11:52.962042+0000","last_became_active":"2024-09-11T08:09:45.414278+0000","last_became_peered":"2024-09-11T08:09:45.414278+0000","last_unstale":"2024-09-11T08:11:52.962042+0000","last_undegraded":"2024-09-11T08:11:52.962042+0000","last_fullsized":"2024-09-11T08:11:52.962042+0000","mapping_epoch":91,"log_start":"106'3700","ondisk_log_start":"106'3700","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_clean_scrub_stamp":"2024-09-11T08:09:44.385805+0000","objects_scrubbed":0,"log_size":5585,"log_dups_size":0,"ondisk_log_size":5585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":401670144,"num_objects":6129,"num_object_clones":0,"num_object_copies":18387,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9285,"num_write_kb":493248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":125,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.428084+0000","last_change":"2024-09-11T08:10:26.486264+0000","last_active":"2024-09-11T08:11:42.428084+0000","last_peered":"2024-09-11T08:11:42.428084+0000","last_clean":"2024-09-11T08:11:42.428084+0000","last_became_active":"2024-09-11T08:07:32.050673+0000","last_became_peered":"2024-09-11T08:07:32.050673+0000","last_unstale":"2024-09-11T08:11:42.428084+0000","last_undegraded":"2024-09-11T08:11:42.428084+0000","last_fullsized":"2024-09-11T08:11:42.428084+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'13","last_scrub_stamp":"2024-09-11T08:10:26.486178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:10:26.486178+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012477777000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":100,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.423741+0000","last_change":"2024-09-11T08:10:25.425751+0000","last_active":"2024-09-11T08:11:42.423741+0000","last_peered":"2024-09-11T08:11:42.423741+0000","last_clean":"2024-09-11T08:11:42.423741+0000","last_became_active":"2024-09-11T08:07:30.047014+0000","last_became_peered":"2024-09-11T08:07:30.047014+0000","last_unstale":"2024-09-11T08:11:42.423741+0000","last_undegraded":"2024-09-11T08:11:42.423741+0000","last_fullsized":"2024-09-11T08:11:42.423741+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:25.425399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:10:25.425399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:54.995561+0000","scrub_duration":0.0032663789999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"127'2","reported_seq":116,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.437952+0000","last_change":"2024-09-11T08:11:32.348728+0000","last_active":"2024-09-11T08:11:42.437952+0000","last_peered":"2024-09-11T08:11:42.437952+0000","last_clean":"2024-09-11T08:11:42.437952+0000","last_became_active":"2024-09-11T08:07:30.045009+0000","last_became_peered":"2024-09-11T08:07:30.045009+0000","last_unstale":"2024-09-11T08:11:42.437952+0000","last_undegraded":"2024-09-11T08:11:42.437952+0000","last_fullsized":"2024-09-11T08:11:42.437952+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'2","last_scrub_stamp":"2024-09-11T08:11:32.348650+0000","last_deep_scrub":"127'2","last_deep_scrub_stamp":"2024-09-11T08:11:32.348650+0000","last_clean_scrub_stamp":"2024-09-11T08:11:32.348650+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:13:00.297188+0000","scrub_duration":0.013905997999999999,"objects_trimmed":0,"snaptrim_duration":0.001859533,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":95,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.439161+0000","last_change":"2024-09-11T08:10:19.998750+0000","last_active":"2024-09-11T08:11:42.439161+0000","last_peered":"2024-09-11T08:11:42.439161+0000","last_clean":"2024-09-11T08:11:42.439161+0000","last_became_active":"2024-09-11T08:07:34.063906+0000","last_became_peered":"2024-09-11T08:07:34.063906+0000","last_unstale":"2024-09-11T08:11:42.439161+0000","last_undegraded":"2024-09-11T08:11:42.439161+0000","last_fullsized":"2024-09-11T08:11:42.439161+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:19.998666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:10:19.998666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.020599591,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":94,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.431348+0000","last_change":"2024-09-11T08:10:22.212876+0000","last_active":"2024-09-11T08:11:42.431348+0000","last_peered":"2024-09-11T08:11:42.431348+0000","last_clean":"2024-09-11T08:11:42.431348+0000","last_became_active":"2024-09-11T08:07:36.631480+0000","last_became_peered":"2024-09-11T08:07:36.631480+0000","last_unstale":"2024-09-11T08:11:42.431348+0000","last_undegraded":"2024-09-11T08:11:42.431348+0000","last_fullsized":"2024-09-11T08:11:42.431348+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:22.212811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:10:22.212811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:51.323655+0000","scrub_duration":0.0032158299999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"130'9431","reported_seq":9774,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-11T08:11:52.660352+0000","last_change":"2024-09-11T08:11:38.973718+0000","last_active":"2024-09-11T08:11:52.660352+0000","last_peered":"2024-09-11T08:11:52.660352+0000","last_clean":"2024-09-11T08:11:52.660352+0000","last_became_active":"2024-09-11T08:09:45.411757+0000","last_became_peered":"2024-09-11T08:09:45.411757+0000","last_unstale":"2024-09-11T08:11:52.660352+0000","last_undegraded":"2024-09-11T08:11:52.660352+0000","last_fullsized":"2024-09-11T08:11:52.660352+0000","mapping_epoch":91,"log_start":"108'4200","ondisk_log_start":"108'4200","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'8561","last_scrub_stamp":"2024-09-11T08:11:38.973659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_clean_scrub_stamp":"2024-09-11T08:11:38.973659+0000","objects_scrubbed":7171,"log_size":5231,"log_dups_size":0,"ondisk_log_size":5231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:13:06.422033+0000","scrub_duration":2.7886773580000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":405995520,"num_objects":6195,"num_object_clones":0,"num_object_copies":18585,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9431,"num_write_kb":500032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":116,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.425651+0000","last_change":"2024-09-11T08:10:29.105854+0000","last_active":"2024-09-11T08:11:42.425651+0000","last_peered":"2024-09-11T08:11:42.425651+0000","last_clean":"2024-09-11T08:11:42.425651+0000","last_became_active":"2024-09-11T08:07:32.055407+0000","last_became_peered":"2024-09-11T08:07:32.055407+0000","last_unstale":"2024-09-11T08:11:42.425651+0000","last_undegraded":"2024-09-11T08:11:42.425651+0000","last_fullsized":"2024-09-11T08:11:42.425651+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-11T08:10:29.105566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:10:29.105566+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.016339228000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"59'1","reported_seq":107,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.427957+0000","last_change":"2024-09-11T08:10:34.991296+0000","last_active":"2024-09-11T08:11:42.427957+0000","last_peered":"2024-09-11T08:11:42.427957+0000","last_clean":"2024-09-11T08:11:42.427957+0000","last_became_active":"2024-09-11T08:07:30.050504+0000","last_became_peered":"2024-09-11T08:07:30.050504+0000","last_unstale":"2024-09-11T08:11:42.427957+0000","last_undegraded":"2024-09-11T08:11:42.427957+0000","last_fullsized":"2024-09-11T08:11:42.427957+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-11T08:10:34.991211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:10:34.991211+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:54.573722+0000","scrub_duration":0.0092067399999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":364,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-11T08:11:51.641765+0000","last_change":"2024-09-11T08:11:22.408697+0000","last_active":"2024-09-11T08:11:51.641765+0000","last_peered":"2024-09-11T08:11:51.641765+0000","last_clean":"2024-09-11T08:11:51.641765+0000","last_became_active":"2024-09-11T08:07:30.045051+0000","last_became_peered":"2024-09-11T08:07:30.045051+0000","last_unstale":"2024-09-11T08:11:51.641765+0000","last_undegraded":"2024-09-11T08:11:51.641765+0000","last_fullsized":"2024-09-11T08:11:51.641765+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-11T08:11:22.408607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:11:22.408607+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:38.586477+0000","scrub_duration":0.0068790099999999996,"objects_trimmed":0,"snaptrim_duration":0.000318821,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":256,"num_read_kb":251,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":95,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.428801+0000","last_change":"2024-09-11T08:10:33.984097+0000","last_active":"2024-09-11T08:11:42.428801+0000","last_peered":"2024-09-11T08:11:42.428801+0000","last_clean":"2024-09-11T08:11:42.428801+0000","last_became_active":"2024-09-11T08:07:34.058962+0000","last_became_peered":"2024-09-11T08:07:34.058962+0000","last_unstale":"2024-09-11T08:11:42.428801+0000","last_undegraded":"2024-09-11T08:11:42.428801+0000","last_fullsized":"2024-09-11T08:11:42.428801+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:33.984021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:10:33.984021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0059758429999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":94,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.426884+0000","last_change":"2024-09-11T08:10:33.381711+0000","last_active":"2024-09-11T08:11:42.426884+0000","last_peered":"2024-09-11T08:11:42.426884+0000","last_clean":"2024-09-11T08:11:42.426884+0000","last_became_active":"2024-09-11T08:07:36.077597+0000","last_became_peered":"2024-09-11T08:07:36.077597+0000","last_unstale":"2024-09-11T08:11:42.426884+0000","last_undegraded":"2024-09-11T08:11:42.426884+0000","last_fullsized":"2024-09-11T08:11:42.426884+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:33.381554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:10:33.381554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014471942999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"130'9425","reported_seq":9472,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-11T08:11:52.938326+0000","last_change":"2024-09-11T08:09:45.415108+0000","last_active":"2024-09-11T08:11:52.938326+0000","last_peered":"2024-09-11T08:11:52.938326+0000","last_clean":"2024-09-11T08:11:52.938326+0000","last_became_active":"2024-09-11T08:09:45.414851+0000","last_became_peered":"2024-09-11T08:09:45.414851+0000","last_unstale":"2024-09-11T08:11:52.938326+0000","last_undegraded":"2024-09-11T08:11:52.938326+0000","last_fullsized":"2024-09-11T08:11:52.938326+0000","mapping_epoch":91,"log_start":"106'3800","ondisk_log_start":"106'3800","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_clean_scrub_stamp":"2024-09-11T08:09:44.385805+0000","objects_scrubbed":0,"log_size":5625,"log_dups_size":0,"ondisk_log_size":5625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":408223744,"num_objects":6229,"num_object_clones":0,"num_object_copies":18687,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9425,"num_write_kb":500928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"84'21","reported_seq":126,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.424326+0000","last_change":"2024-09-11T08:10:20.637001+0000","last_active":"2024-09-11T08:11:42.424326+0000","last_peered":"2024-09-11T08:11:42.424326+0000","last_clean":"2024-09-11T08:11:42.424326+0000","last_became_active":"2024-09-11T08:07:32.042339+0000","last_became_peered":"2024-09-11T08:07:32.042339+0000","last_unstale":"2024-09-11T08:11:42.424326+0000","last_undegraded":"2024-09-11T08:11:42.424326+0000","last_fullsized":"2024-09-11T08:11:42.424326+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'21","last_scrub_stamp":"2024-09-11T08:10:20.636918+0000","last_deep_scrub":"84'21","last_deep_scrub_stamp":"2024-09-11T08:08:56.705306+0000","last_clean_scrub_stamp":"2024-09-11T08:10:20.636918+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004681826,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":90,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.430535+0000","last_change":"2024-09-11T08:09:39.390240+0000","last_active":"2024-09-11T08:11:42.430535+0000","last_peered":"2024-09-11T08:11:42.430535+0000","last_clean":"2024-09-11T08:11:42.430535+0000","last_became_active":"2024-09-11T08:07:30.047246+0000","last_became_peered":"2024-09-11T08:07:30.047246+0000","last_unstale":"2024-09-11T08:11:42.430535+0000","last_undegraded":"2024-09-11T08:11:42.430535+0000","last_fullsized":"2024-09-11T08:11:42.430535+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:39.390158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:09:39.390158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013263249999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":146,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.431134+0000","last_change":"2024-09-11T08:11:17.419193+0000","last_active":"2024-09-11T08:11:42.431134+0000","last_peered":"2024-09-11T08:11:42.431134+0000","last_clean":"2024-09-11T08:11:42.431134+0000","last_became_active":"2024-09-11T08:07:19.726378+0000","last_became_peered":"2024-09-11T08:07:19.726378+0000","last_unstale":"2024-09-11T08:11:42.431134+0000","last_undegraded":"2024-09-11T08:11:42.431134+0000","last_fullsized":"2024-09-11T08:11:42.431134+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-11T08:11:17.419133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:05:21.231856+0000","last_clean_scrub_stamp":"2024-09-11T08:11:17.419133+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:46.438913+0000","scrub_duration":0.002818194,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":96,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.431319+0000","last_change":"2024-09-11T08:10:39.300321+0000","last_active":"2024-09-11T08:11:42.431319+0000","last_peered":"2024-09-11T08:11:42.431319+0000","last_clean":"2024-09-11T08:11:42.431319+0000","last_became_active":"2024-09-11T08:07:34.058414+0000","last_became_peered":"2024-09-11T08:07:34.058414+0000","last_unstale":"2024-09-11T08:11:42.431319+0000","last_undegraded":"2024-09-11T08:11:42.431319+0000","last_fullsized":"2024-09-11T08:11:42.431319+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:39.300265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:51.289872+0000","last_clean_scrub_stamp":"2024-09-11T08:10:39.300265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001629222,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":93,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.428725+0000","last_change":"2024-09-11T08:10:27.995889+0000","last_active":"2024-09-11T08:11:42.428725+0000","last_peered":"2024-09-11T08:11:42.428725+0000","last_clean":"2024-09-11T08:11:42.428725+0000","last_became_active":"2024-09-11T08:07:36.072508+0000","last_became_peered":"2024-09-11T08:07:36.072508+0000","last_unstale":"2024-09-11T08:11:42.428725+0000","last_undegraded":"2024-09-11T08:11:42.428725+0000","last_fullsized":"2024-09-11T08:11:42.428725+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:27.995806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:01.777429+0000","last_clean_scrub_stamp":"2024-09-11T08:10:27.995806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001407127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"130'9382","reported_seq":9738,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-11T08:11:52.651062+0000","last_change":"2024-09-11T08:11:24.558667+0000","last_active":"2024-09-11T08:11:52.651062+0000","last_peered":"2024-09-11T08:11:52.651062+0000","last_clean":"2024-09-11T08:11:52.651062+0000","last_became_active":"2024-09-11T08:09:45.412638+0000","last_became_peered":"2024-09-11T08:09:45.412638+0000","last_unstale":"2024-09-11T08:11:52.651062+0000","last_undegraded":"2024-09-11T08:11:52.651062+0000","last_fullsized":"2024-09-11T08:11:52.651062+0000","mapping_epoch":91,"log_start":"103'3100","ondisk_log_start":"103'3100","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'7543","last_scrub_stamp":"2024-09-11T08:11:24.558579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_clean_scrub_stamp":"2024-09-11T08:11:24.558579+0000","objects_scrubbed":7445,"log_size":6282,"log_dups_size":0,"ondisk_log_size":6282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:27.715583+0000","scrub_duration":3.1005590700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400097304,"num_objects":6106,"num_object_clones":0,"num_object_copies":18318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6106,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9382,"num_write_kb":495553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"88'16","reported_seq":122,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.420343+0000","last_change":"2024-09-11T08:09:39.636044+0000","last_active":"2024-09-11T08:11:42.420343+0000","last_peered":"2024-09-11T08:11:42.420343+0000","last_clean":"2024-09-11T08:11:42.420343+0000","last_became_active":"2024-09-11T08:07:32.047971+0000","last_became_peered":"2024-09-11T08:07:32.047971+0000","last_unstale":"2024-09-11T08:11:42.420343+0000","last_undegraded":"2024-09-11T08:11:42.420343+0000","last_fullsized":"2024-09-11T08:11:42.420343+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'16","last_scrub_stamp":"2024-09-11T08:09:39.635984+0000","last_deep_scrub":"88'16","last_deep_scrub_stamp":"2024-09-11T08:09:39.635984+0000","last_clean_scrub_stamp":"2024-09-11T08:09:39.635984+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038085599999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":100,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.424952+0000","last_change":"2024-09-11T08:10:26.069693+0000","last_active":"2024-09-11T08:11:42.424952+0000","last_peered":"2024-09-11T08:11:42.424952+0000","last_clean":"2024-09-11T08:11:42.424952+0000","last_became_active":"2024-09-11T08:07:30.039884+0000","last_became_peered":"2024-09-11T08:07:30.039884+0000","last_unstale":"2024-09-11T08:11:42.424952+0000","last_undegraded":"2024-09-11T08:11:42.424952+0000","last_fullsized":"2024-09-11T08:11:42.424952+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:26.069640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:36.417911+0000","last_clean_scrub_stamp":"2024-09-11T08:10:26.069640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001453558,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":95,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.428580+0000","last_change":"2024-09-11T08:10:31.929612+0000","last_active":"2024-09-11T08:11:42.428580+0000","last_peered":"2024-09-11T08:11:42.428580+0000","last_clean":"2024-09-11T08:11:42.428580+0000","last_became_active":"2024-09-11T08:07:34.053347+0000","last_became_peered":"2024-09-11T08:07:34.053347+0000","last_unstale":"2024-09-11T08:11:42.428580+0000","last_undegraded":"2024-09-11T08:11:42.428580+0000","last_fullsized":"2024-09-11T08:11:42.428580+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:31.929528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:10:31.929528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:53.171499+0000","scrub_duration":0.0015531010000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":94,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.429882+0000","last_change":"2024-09-11T08:11:37.480260+0000","last_active":"2024-09-11T08:11:42.429882+0000","last_peered":"2024-09-11T08:11:42.429882+0000","last_clean":"2024-09-11T08:11:42.429882+0000","last_became_active":"2024-09-11T08:07:36.631687+0000","last_became_peered":"2024-09-11T08:07:36.631687+0000","last_unstale":"2024-09-11T08:11:42.429882+0000","last_undegraded":"2024-09-11T08:11:42.429882+0000","last_fullsized":"2024-09-11T08:11:42.429882+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:37.480123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:11:37.480123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:48.786633+0000","scrub_duration":0.0064260089999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"130'9364","reported_seq":9715,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-11T08:11:52.762243+0000","last_change":"2024-09-11T08:11:22.474726+0000","last_active":"2024-09-11T08:11:52.762243+0000","last_peered":"2024-09-11T08:11:52.762243+0000","last_clean":"2024-09-11T08:11:52.762243+0000","last_became_active":"2024-09-11T08:09:45.413207+0000","last_became_peered":"2024-09-11T08:09:45.413207+0000","last_unstale":"2024-09-11T08:11:52.762243+0000","last_undegraded":"2024-09-11T08:11:52.762243+0000","last_fullsized":"2024-09-11T08:11:52.762243+0000","mapping_epoch":91,"log_start":"100'2500","ondisk_log_start":"100'2500","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'7432","last_scrub_stamp":"2024-09-11T08:11:22.474630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_clean_scrub_stamp":"2024-09-11T08:11:22.474630+0000","objects_scrubbed":7334,"log_size":6864,"log_dups_size":0,"ondisk_log_size":6864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:23.920712+0000","scrub_duration":4.0826116800000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403439616,"num_objects":6156,"num_object_clones":0,"num_object_copies":18468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9364,"num_write_kb":496640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"77'25","reported_seq":162,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.420947+0000","last_change":"2024-09-11T08:10:31.126024+0000","last_active":"2024-09-11T08:11:42.420947+0000","last_peered":"2024-09-11T08:11:42.420947+0000","last_clean":"2024-09-11T08:11:42.420947+0000","last_became_active":"2024-09-11T08:09:08.106771+0000","last_became_peered":"2024-09-11T08:09:08.106771+0000","last_unstale":"2024-09-11T08:11:42.420947+0000","last_undegraded":"2024-09-11T08:11:42.420947+0000","last_fullsized":"2024-09-11T08:11:42.420947+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'25","last_scrub_stamp":"2024-09-11T08:10:31.125934+0000","last_deep_scrub":"77'25","last_deep_scrub_stamp":"2024-09-11T08:08:32.458210+0000","last_clean_scrub_stamp":"2024-09-11T08:10:31.125934+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0096263719999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":99,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.423683+0000","last_change":"2024-09-11T08:11:16.505483+0000","last_active":"2024-09-11T08:11:42.423683+0000","last_peered":"2024-09-11T08:11:42.423683+0000","last_clean":"2024-09-11T08:11:42.423683+0000","last_became_active":"2024-09-11T08:07:30.042555+0000","last_became_peered":"2024-09-11T08:07:30.042555+0000","last_unstale":"2024-09-11T08:11:42.423683+0000","last_undegraded":"2024-09-11T08:11:42.423683+0000","last_fullsized":"2024-09-11T08:11:42.423683+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:16.505384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:11:16.505384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:26.959383+0000","scrub_duration":0.010166646999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":96,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.426954+0000","last_change":"2024-09-11T08:10:27.459645+0000","last_active":"2024-09-11T08:11:42.426954+0000","last_peered":"2024-09-11T08:11:42.426954+0000","last_clean":"2024-09-11T08:11:42.426954+0000","last_became_active":"2024-09-11T08:07:34.054961+0000","last_became_peered":"2024-09-11T08:07:34.054961+0000","last_unstale":"2024-09-11T08:11:42.426954+0000","last_undegraded":"2024-09-11T08:11:42.426954+0000","last_fullsized":"2024-09-11T08:11:42.426954+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:27.459436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:10:27.459436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0053113359999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":103,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.440085+0000","last_change":"2024-09-11T08:11:39.370275+0000","last_active":"2024-09-11T08:11:42.440085+0000","last_peered":"2024-09-11T08:11:42.440085+0000","last_clean":"2024-09-11T08:11:42.440085+0000","last_became_active":"2024-09-11T08:07:36.071194+0000","last_became_peered":"2024-09-11T08:07:36.071194+0000","last_unstale":"2024-09-11T08:11:42.440085+0000","last_undegraded":"2024-09-11T08:11:42.440085+0000","last_fullsized":"2024-09-11T08:11:42.440085+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:39.370193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:11:39.370193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:40.590716+0000","scrub_duration":0.014190103000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"130'9286","reported_seq":9333,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-11T08:11:52.931633+0000","last_change":"2024-09-11T08:09:45.416120+0000","last_active":"2024-09-11T08:11:52.931633+0000","last_peered":"2024-09-11T08:11:52.931633+0000","last_clean":"2024-09-11T08:11:52.931633+0000","last_became_active":"2024-09-11T08:09:45.415738+0000","last_became_peered":"2024-09-11T08:09:45.415738+0000","last_unstale":"2024-09-11T08:11:52.931633+0000","last_undegraded":"2024-09-11T08:11:52.931633+0000","last_fullsized":"2024-09-11T08:11:52.931633+0000","mapping_epoch":91,"log_start":"106'3700","ondisk_log_start":"106'3700","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_clean_scrub_stamp":"2024-09-11T08:09:44.385805+0000","objects_scrubbed":0,"log_size":5586,"log_dups_size":0,"ondisk_log_size":5586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404619264,"num_objects":6174,"num_object_clones":0,"num_object_copies":18522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9286,"num_write_kb":494720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":113,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.420695+0000","last_change":"2024-09-11T08:10:35.175848+0000","last_active":"2024-09-11T08:11:42.420695+0000","last_peered":"2024-09-11T08:11:42.420695+0000","last_clean":"2024-09-11T08:11:42.420695+0000","last_became_active":"2024-09-11T08:07:32.043314+0000","last_became_peered":"2024-09-11T08:07:32.043314+0000","last_unstale":"2024-09-11T08:11:42.420695+0000","last_undegraded":"2024-09-11T08:11:42.420695+0000","last_fullsized":"2024-09-11T08:11:42.420695+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-11T08:10:35.175750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:10:35.175750+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:00.379714+0000","scrub_duration":0.088039645,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":100,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.431237+0000","last_change":"2024-09-11T08:10:38.126730+0000","last_active":"2024-09-11T08:11:42.431237+0000","last_peered":"2024-09-11T08:11:42.431237+0000","last_clean":"2024-09-11T08:11:42.431237+0000","last_became_active":"2024-09-11T08:07:30.046684+0000","last_became_peered":"2024-09-11T08:07:30.046684+0000","last_unstale":"2024-09-11T08:11:42.431237+0000","last_undegraded":"2024-09-11T08:11:42.431237+0000","last_fullsized":"2024-09-11T08:11:42.431237+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:38.126638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:10:38.126638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0084326729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"66'8","reported_seq":165,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-11T08:11:47.806862+0000","last_change":"2024-09-11T08:11:24.359888+0000","last_active":"2024-09-11T08:11:47.806862+0000","last_peered":"2024-09-11T08:11:47.806862+0000","last_clean":"2024-09-11T08:11:47.806862+0000","last_became_active":"2024-09-11T08:07:34.062469+0000","last_became_peered":"2024-09-11T08:07:34.062469+0000","last_unstale":"2024-09-11T08:11:47.806862+0000","last_undegraded":"2024-09-11T08:11:47.806862+0000","last_fullsized":"2024-09-11T08:11:47.806862+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:11:24.359780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:11:24.359780+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:31.728208+0000","scrub_duration":0.0054480609999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":84,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.420101+0000","last_change":"2024-09-11T08:09:14.613599+0000","last_active":"2024-09-11T08:11:42.420101+0000","last_peered":"2024-09-11T08:11:42.420101+0000","last_clean":"2024-09-11T08:11:42.420101+0000","last_became_active":"2024-09-11T08:07:36.074684+0000","last_became_peered":"2024-09-11T08:07:36.074684+0000","last_unstale":"2024-09-11T08:11:42.420101+0000","last_undegraded":"2024-09-11T08:11:42.420101+0000","last_fullsized":"2024-09-11T08:11:42.420101+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:14.613532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:14.613532+0000","last_clean_scrub_stamp":"2024-09-11T08:09:14.613532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001503648,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"130'9366","reported_seq":9716,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-11T08:11:52.752963+0000","last_change":"2024-09-11T08:11:22.178487+0000","last_active":"2024-09-11T08:11:52.752963+0000","last_peered":"2024-09-11T08:11:52.752963+0000","last_clean":"2024-09-11T08:11:52.752963+0000","last_became_active":"2024-09-11T08:09:45.411521+0000","last_became_peered":"2024-09-11T08:09:45.411521+0000","last_unstale":"2024-09-11T08:11:52.752963+0000","last_undegraded":"2024-09-11T08:11:52.752963+0000","last_fullsized":"2024-09-11T08:11:52.752963+0000","mapping_epoch":91,"log_start":"101'2500","ondisk_log_start":"101'2500","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'7415","last_scrub_stamp":"2024-09-11T08:11:22.178363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_clean_scrub_stamp":"2024-09-11T08:11:22.178363+0000","objects_scrubbed":7336,"log_size":6866,"log_dups_size":0,"ondisk_log_size":6866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:46.113024+0000","scrub_duration":2.7897035839999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":407109632,"num_objects":6212,"num_object_clones":0,"num_object_copies":18636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9366,"num_write_kb":498496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":114,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.426292+0000","last_change":"2024-09-11T08:11:40.062007+0000","last_active":"2024-09-11T08:11:42.426292+0000","last_peered":"2024-09-11T08:11:42.426292+0000","last_clean":"2024-09-11T08:11:42.426292+0000","last_became_active":"2024-09-11T08:07:32.054885+0000","last_became_peered":"2024-09-11T08:07:32.054885+0000","last_unstale":"2024-09-11T08:11:42.426292+0000","last_undegraded":"2024-09-11T08:11:42.426292+0000","last_fullsized":"2024-09-11T08:11:42.426292+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:11:40.061926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:11:40.061926+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:40.906878+0000","scrub_duration":0.106648482,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":109,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.428372+0000","last_change":"2024-09-11T08:11:33.819647+0000","last_active":"2024-09-11T08:11:42.428372+0000","last_peered":"2024-09-11T08:11:42.428372+0000","last_clean":"2024-09-11T08:11:42.428372+0000","last_became_active":"2024-09-11T08:07:30.045613+0000","last_became_peered":"2024-09-11T08:07:30.045613+0000","last_unstale":"2024-09-11T08:11:42.428372+0000","last_undegraded":"2024-09-11T08:11:42.428372+0000","last_fullsized":"2024-09-11T08:11:42.428372+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:33.819593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:11:33.819593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:13:03.296968+0000","scrub_duration":0.0034651399999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":105,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.438367+0000","last_change":"2024-09-11T08:11:33.364733+0000","last_active":"2024-09-11T08:11:42.438367+0000","last_peered":"2024-09-11T08:11:42.438367+0000","last_clean":"2024-09-11T08:11:42.438367+0000","last_became_active":"2024-09-11T08:07:34.056606+0000","last_became_peered":"2024-09-11T08:07:34.056606+0000","last_unstale":"2024-09-11T08:11:42.438367+0000","last_undegraded":"2024-09-11T08:11:42.438367+0000","last_fullsized":"2024-09-11T08:11:42.438367+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:33.364578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:11:33.364578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:34.370854+0000","scrub_duration":0.0099402859999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":94,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.427892+0000","last_change":"2024-09-11T08:11:18.967755+0000","last_active":"2024-09-11T08:11:42.427892+0000","last_peered":"2024-09-11T08:11:42.427892+0000","last_clean":"2024-09-11T08:11:42.427892+0000","last_became_active":"2024-09-11T08:07:36.072452+0000","last_became_peered":"2024-09-11T08:07:36.072452+0000","last_unstale":"2024-09-11T08:11:42.427892+0000","last_undegraded":"2024-09-11T08:11:42.427892+0000","last_fullsized":"2024-09-11T08:11:42.427892+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:18.967665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:11:18.967665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:48.119844+0000","scrub_duration":0.013929113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"130'9530","reported_seq":9577,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-11T08:11:52.891791+0000","last_change":"2024-09-11T08:09:45.414158+0000","last_active":"2024-09-11T08:11:52.891791+0000","last_peered":"2024-09-11T08:11:52.891791+0000","last_clean":"2024-09-11T08:11:52.891791+0000","last_became_active":"2024-09-11T08:09:45.413883+0000","last_became_peered":"2024-09-11T08:09:45.413883+0000","last_unstale":"2024-09-11T08:11:52.891791+0000","last_undegraded":"2024-09-11T08:11:52.891791+0000","last_fullsized":"2024-09-11T08:11:52.891791+0000","mapping_epoch":91,"log_start":"106'3900","ondisk_log_start":"106'3900","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_clean_scrub_stamp":"2024-09-11T08:09:44.385805+0000","objects_scrubbed":0,"log_size":5630,"log_dups_size":0,"ondisk_log_size":5630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":407109632,"num_objects":6212,"num_object_clones":0,"num_object_copies":18636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9530,"num_write_kb":503744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":138,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.423752+0000","last_change":"2024-09-11T08:11:29.618794+0000","last_active":"2024-09-11T08:11:42.423752+0000","last_peered":"2024-09-11T08:11:42.423752+0000","last_clean":"2024-09-11T08:11:42.423752+0000","last_became_active":"2024-09-11T08:07:32.050880+0000","last_became_peered":"2024-09-11T08:07:32.050880+0000","last_unstale":"2024-09-11T08:11:42.423752+0000","last_undegraded":"2024-09-11T08:11:42.423752+0000","last_fullsized":"2024-09-11T08:11:42.423752+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-11T08:11:29.618722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:11:29.618722+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:35.510180+0000","scrub_duration":0.0019851790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":100,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.431670+0000","last_change":"2024-09-11T08:10:19.201593+0000","last_active":"2024-09-11T08:11:42.431670+0000","last_peered":"2024-09-11T08:11:42.431670+0000","last_clean":"2024-09-11T08:11:42.431670+0000","last_became_active":"2024-09-11T08:07:30.043294+0000","last_became_peered":"2024-09-11T08:07:30.043294+0000","last_unstale":"2024-09-11T08:11:42.431670+0000","last_undegraded":"2024-09-11T08:11:42.431670+0000","last_fullsized":"2024-09-11T08:11:42.431670+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:19.201434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:44.385372+0000","last_clean_scrub_stamp":"2024-09-11T08:10:19.201434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023250469999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":105,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.437992+0000","last_change":"2024-09-11T08:11:25.397839+0000","last_active":"2024-09-11T08:11:42.437992+0000","last_peered":"2024-09-11T08:11:42.437992+0000","last_clean":"2024-09-11T08:11:42.437992+0000","last_became_active":"2024-09-11T08:07:34.063589+0000","last_became_peered":"2024-09-11T08:07:34.063589+0000","last_unstale":"2024-09-11T08:11:42.437992+0000","last_undegraded":"2024-09-11T08:11:42.437992+0000","last_fullsized":"2024-09-11T08:11:42.437992+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:25.397757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:11:25.397757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:27.261790+0000","scrub_duration":0.00152206,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":94,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.431051+0000","last_change":"2024-09-11T08:10:15.257677+0000","last_active":"2024-09-11T08:11:42.431051+0000","last_peered":"2024-09-11T08:11:42.431051+0000","last_clean":"2024-09-11T08:11:42.431051+0000","last_became_active":"2024-09-11T08:07:36.077523+0000","last_became_peered":"2024-09-11T08:07:36.077523+0000","last_unstale":"2024-09-11T08:11:42.431051+0000","last_undegraded":"2024-09-11T08:11:42.431051+0000","last_fullsized":"2024-09-11T08:11:42.431051+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:15.257622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:10:15.257622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0063147990000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"130'9543","reported_seq":9590,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-11T08:11:52.776266+0000","last_change":"2024-09-11T08:09:45.411095+0000","last_active":"2024-09-11T08:11:52.776266+0000","last_peered":"2024-09-11T08:11:52.776266+0000","last_clean":"2024-09-11T08:11:52.776266+0000","last_became_active":"2024-09-11T08:09:45.410825+0000","last_became_peered":"2024-09-11T08:09:45.410825+0000","last_unstale":"2024-09-11T08:11:52.776266+0000","last_undegraded":"2024-09-11T08:11:52.776266+0000","last_fullsized":"2024-09-11T08:11:52.776266+0000","mapping_epoch":91,"log_start":"111'4900","ondisk_log_start":"111'4900","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_clean_scrub_stamp":"2024-09-11T08:09:44.385805+0000","objects_scrubbed":0,"log_size":4643,"log_dups_size":0,"ondisk_log_size":4643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":412024832,"num_objects":6287,"num_object_clones":0,"num_object_copies":18861,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9543,"num_write_kb":506560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":121,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.424820+0000","last_change":"2024-09-11T08:10:44.085605+0000","last_active":"2024-09-11T08:11:42.424820+0000","last_peered":"2024-09-11T08:11:42.424820+0000","last_clean":"2024-09-11T08:11:42.424820+0000","last_became_active":"2024-09-11T08:07:32.048164+0000","last_became_peered":"2024-09-11T08:07:32.048164+0000","last_unstale":"2024-09-11T08:11:42.424820+0000","last_undegraded":"2024-09-11T08:11:42.424820+0000","last_fullsized":"2024-09-11T08:11:42.424820+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-11T08:10:44.085517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:10:44.085517+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:13.712893+0000","scrub_duration":0.0074629060000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":100,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.431573+0000","last_change":"2024-09-11T08:10:27.250069+0000","last_active":"2024-09-11T08:11:42.431573+0000","last_peered":"2024-09-11T08:11:42.431573+0000","last_clean":"2024-09-11T08:11:42.431573+0000","last_became_active":"2024-09-11T08:07:30.041219+0000","last_became_peered":"2024-09-11T08:07:30.041219+0000","last_unstale":"2024-09-11T08:11:42.431573+0000","last_undegraded":"2024-09-11T08:11:42.431573+0000","last_fullsized":"2024-09-11T08:11:42.431573+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:27.250021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:53.247275+0000","last_clean_scrub_stamp":"2024-09-11T08:10:27.250021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011297309999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":95,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.437770+0000","last_change":"2024-09-11T08:10:27.002636+0000","last_active":"2024-09-11T08:11:42.437770+0000","last_peered":"2024-09-11T08:11:42.437770+0000","last_clean":"2024-09-11T08:11:42.437770+0000","last_became_active":"2024-09-11T08:07:34.057479+0000","last_became_peered":"2024-09-11T08:07:34.057479+0000","last_unstale":"2024-09-11T08:11:42.437770+0000","last_undegraded":"2024-09-11T08:11:42.437770+0000","last_fullsized":"2024-09-11T08:11:42.437770+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:27.002455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:10:27.002455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:53.225621+0000","scrub_duration":0.0044259809999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":94,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.430990+0000","last_change":"2024-09-11T08:10:42.097297+0000","last_active":"2024-09-11T08:11:42.430990+0000","last_peered":"2024-09-11T08:11:42.430990+0000","last_clean":"2024-09-11T08:11:42.430990+0000","last_became_active":"2024-09-11T08:07:36.078075+0000","last_became_peered":"2024-09-11T08:07:36.078075+0000","last_unstale":"2024-09-11T08:11:42.430990+0000","last_undegraded":"2024-09-11T08:11:42.430990+0000","last_fullsized":"2024-09-11T08:11:42.430990+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:42.097207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:10:42.097207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:51.535466+0000","scrub_duration":0.0056763179999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"130'11555","reported_seq":11702,"reported_epoch":130,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-11T08:11:52.817111+0000","last_change":"2024-09-11T08:11:39.467736+0000","last_active":"2024-09-11T08:11:52.817111+0000","last_peered":"2024-09-11T08:11:52.817111+0000","last_clean":"2024-09-11T08:11:52.817111+0000","last_became_active":"2024-09-11T08:09:45.412566+0000","last_became_peered":"2024-09-11T08:09:45.412566+0000","last_unstale":"2024-09-11T08:11:52.817111+0000","last_undegraded":"2024-09-11T08:11:52.817111+0000","last_fullsized":"2024-09-11T08:11:52.817111+0000","mapping_epoch":91,"log_start":"121'6501","ondisk_log_start":"121'6501","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_clean_scrub_stamp":"2024-09-11T08:09:44.385805+0000","objects_scrubbed":2399,"log_size":5054,"log_dups_size":0,"ondisk_log_size":5054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 13s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":408158208,"num_objects":6228,"num_object_clones":0,"num_object_copies":18684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9418,"num_write_kb":500672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":122,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.427950+0000","last_change":"2024-09-11T08:10:39.911118+0000","last_active":"2024-09-11T08:11:42.427950+0000","last_peered":"2024-09-11T08:11:42.427950+0000","last_clean":"2024-09-11T08:11:42.427950+0000","last_became_active":"2024-09-11T08:07:32.039275+0000","last_became_peered":"2024-09-11T08:07:32.039275+0000","last_unstale":"2024-09-11T08:11:42.427950+0000","last_undegraded":"2024-09-11T08:11:42.427950+0000","last_fullsized":"2024-09-11T08:11:42.427950+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-11T08:10:39.911043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:10:39.911043+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:03.238724+0000","scrub_duration":0.0020973419999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":117,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.431685+0000","last_change":"2024-09-11T08:11:27.256947+0000","last_active":"2024-09-11T08:11:42.431685+0000","last_peered":"2024-09-11T08:11:42.431685+0000","last_clean":"2024-09-11T08:11:42.431685+0000","last_became_active":"2024-09-11T08:07:30.045967+0000","last_became_peered":"2024-09-11T08:07:30.045967+0000","last_unstale":"2024-09-11T08:11:42.431685+0000","last_undegraded":"2024-09-11T08:11:42.431685+0000","last_fullsized":"2024-09-11T08:11:42.431685+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-11T08:11:27.256886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:11:27.256886+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:47.796310+0000","scrub_duration":0.010702672999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"66'8","reported_seq":156,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-11T08:11:47.794968+0000","last_change":"2024-09-11T08:10:40.276739+0000","last_active":"2024-09-11T08:11:47.794968+0000","last_peered":"2024-09-11T08:11:47.794968+0000","last_clean":"2024-09-11T08:11:47.794968+0000","last_became_active":"2024-09-11T08:07:34.058015+0000","last_became_peered":"2024-09-11T08:07:34.058015+0000","last_unstale":"2024-09-11T08:11:47.794968+0000","last_undegraded":"2024-09-11T08:11:47.794968+0000","last_fullsized":"2024-09-11T08:11:47.794968+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:10:40.276651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:10:40.276651+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:58.174703+0000","scrub_duration":0.0027620549999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":93,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.427994+0000","last_change":"2024-09-11T08:11:16.768991+0000","last_active":"2024-09-11T08:11:42.427994+0000","last_peered":"2024-09-11T08:11:42.427994+0000","last_clean":"2024-09-11T08:11:42.427994+0000","last_became_active":"2024-09-11T08:07:36.631755+0000","last_became_peered":"2024-09-11T08:07:36.631755+0000","last_unstale":"2024-09-11T08:11:42.427994+0000","last_undegraded":"2024-09-11T08:11:42.427994+0000","last_fullsized":"2024-09-11T08:11:42.427994+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:16.768914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:11:16.768914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:32.682600+0000","scrub_duration":0.0045193669999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":99,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.423480+0000","last_change":"2024-09-11T08:10:36.436042+0000","last_active":"2024-09-11T08:11:42.423480+0000","last_peered":"2024-09-11T08:11:42.423480+0000","last_clean":"2024-09-11T08:11:42.423480+0000","last_became_active":"2024-09-11T08:07:30.045910+0000","last_became_peered":"2024-09-11T08:07:30.045910+0000","last_unstale":"2024-09-11T08:11:42.423480+0000","last_undegraded":"2024-09-11T08:11:42.423480+0000","last_fullsized":"2024-09-11T08:11:42.423480+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:36.435971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:10:36.435971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:00.767133+0000","scrub_duration":0.0023002999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"84'9","reported_seq":110,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.430812+0000","last_change":"2024-09-11T08:11:21.397319+0000","last_active":"2024-09-11T08:11:42.430812+0000","last_peered":"2024-09-11T08:11:42.430812+0000","last_clean":"2024-09-11T08:11:42.430812+0000","last_became_active":"2024-09-11T08:07:32.046891+0000","last_became_peered":"2024-09-11T08:07:32.046891+0000","last_unstale":"2024-09-11T08:11:42.430812+0000","last_undegraded":"2024-09-11T08:11:42.430812+0000","last_fullsized":"2024-09-11T08:11:42.430812+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-09-11T08:11:21.397179+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-09-11T08:09:03.418457+0000","last_clean_scrub_stamp":"2024-09-11T08:11:21.397179+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:48.076959+0000","scrub_duration":0.0062300630000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":95,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.428587+0000","last_change":"2024-09-11T08:10:43.836622+0000","last_active":"2024-09-11T08:11:42.428587+0000","last_peered":"2024-09-11T08:11:42.428587+0000","last_clean":"2024-09-11T08:11:42.428587+0000","last_became_active":"2024-09-11T08:07:34.057774+0000","last_became_peered":"2024-09-11T08:07:34.057774+0000","last_unstale":"2024-09-11T08:11:42.428587+0000","last_undegraded":"2024-09-11T08:11:42.428587+0000","last_fullsized":"2024-09-11T08:11:42.428587+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:43.836566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:10:43.836566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0087284040000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":94,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.431264+0000","last_change":"2024-09-11T08:10:38.297462+0000","last_active":"2024-09-11T08:11:42.431264+0000","last_peered":"2024-09-11T08:11:42.431264+0000","last_clean":"2024-09-11T08:11:42.431264+0000","last_became_active":"2024-09-11T08:07:36.633244+0000","last_became_peered":"2024-09-11T08:07:36.633244+0000","last_unstale":"2024-09-11T08:11:42.431264+0000","last_undegraded":"2024-09-11T08:11:42.431264+0000","last_fullsized":"2024-09-11T08:11:42.431264+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:38.297367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:10:38.297367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:51.071012+0000","scrub_duration":0.00156018,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"84'13","reported_seq":118,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.425721+0000","last_change":"2024-09-11T08:11:32.926108+0000","last_active":"2024-09-11T08:11:42.425721+0000","last_peered":"2024-09-11T08:11:42.425721+0000","last_clean":"2024-09-11T08:11:42.425721+0000","last_became_active":"2024-09-11T08:07:32.048350+0000","last_became_peered":"2024-09-11T08:07:32.048350+0000","last_unstale":"2024-09-11T08:11:42.425721+0000","last_undegraded":"2024-09-11T08:11:42.425721+0000","last_fullsized":"2024-09-11T08:11:42.425721+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'13","last_scrub_stamp":"2024-09-11T08:11:32.925950+0000","last_deep_scrub":"84'13","last_deep_scrub_stamp":"2024-09-11T08:08:57.835634+0000","last_clean_scrub_stamp":"2024-09-11T08:11:32.925950+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:58.685138+0000","scrub_duration":0.013245129,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":110,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.431022+0000","last_change":"2024-09-11T08:11:35.201048+0000","last_active":"2024-09-11T08:11:42.431022+0000","last_peered":"2024-09-11T08:11:42.431022+0000","last_clean":"2024-09-11T08:11:42.431022+0000","last_became_active":"2024-09-11T08:07:30.044055+0000","last_became_peered":"2024-09-11T08:07:30.044055+0000","last_unstale":"2024-09-11T08:11:42.431022+0000","last_undegraded":"2024-09-11T08:11:42.431022+0000","last_fullsized":"2024-09-11T08:11:42.431022+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:35.200970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:30.471620+0000","last_clean_scrub_stamp":"2024-09-11T08:11:35.200970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:36.402648+0000","scrub_duration":0.0012204990000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":96,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.430666+0000","last_change":"2024-09-11T08:11:16.386090+0000","last_active":"2024-09-11T08:11:42.430666+0000","last_peered":"2024-09-11T08:11:42.430666+0000","last_clean":"2024-09-11T08:11:42.430666+0000","last_became_active":"2024-09-11T08:07:34.057127+0000","last_became_peered":"2024-09-11T08:07:34.057127+0000","last_unstale":"2024-09-11T08:11:42.430666+0000","last_undegraded":"2024-09-11T08:11:42.430666+0000","last_fullsized":"2024-09-11T08:11:42.430666+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:16.385713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:04.370734+0000","last_clean_scrub_stamp":"2024-09-11T08:11:16.385713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:38.946720+0000","scrub_duration":0.013591255999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":93,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.437911+0000","last_change":"2024-09-11T08:10:29.963987+0000","last_active":"2024-09-11T08:11:42.437911+0000","last_peered":"2024-09-11T08:11:42.437911+0000","last_clean":"2024-09-11T08:11:42.437911+0000","last_became_active":"2024-09-11T08:07:36.071513+0000","last_became_peered":"2024-09-11T08:07:36.071513+0000","last_unstale":"2024-09-11T08:11:42.437911+0000","last_undegraded":"2024-09-11T08:11:42.437911+0000","last_fullsized":"2024-09-11T08:11:42.437911+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:29.963899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:10:29.963899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:56.542813+0000","scrub_duration":0.0022607389999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":113,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.428551+0000","last_change":"2024-09-11T08:10:18.962167+0000","last_active":"2024-09-11T08:11:42.428551+0000","last_peered":"2024-09-11T08:11:42.428551+0000","last_clean":"2024-09-11T08:11:42.428551+0000","last_became_active":"2024-09-11T08:07:32.041761+0000","last_became_peered":"2024-09-11T08:07:32.041761+0000","last_unstale":"2024-09-11T08:11:42.428551+0000","last_undegraded":"2024-09-11T08:11:42.428551+0000","last_fullsized":"2024-09-11T08:11:42.428551+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:10:18.962088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:10:18.962088+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035069179999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":100,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.425974+0000","last_change":"2024-09-11T08:10:38.937153+0000","last_active":"2024-09-11T08:11:42.425974+0000","last_peered":"2024-09-11T08:11:42.425974+0000","last_clean":"2024-09-11T08:11:42.425974+0000","last_became_active":"2024-09-11T08:07:30.045380+0000","last_became_peered":"2024-09-11T08:07:30.045380+0000","last_unstale":"2024-09-11T08:11:42.425974+0000","last_undegraded":"2024-09-11T08:11:42.425974+0000","last_fullsized":"2024-09-11T08:11:42.425974+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:38.937071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:48.935387+0000","last_clean_scrub_stamp":"2024-09-11T08:10:38.937071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:57.024718+0000","scrub_duration":0.015757854000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"66'8","reported_seq":156,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-11T08:11:47.801999+0000","last_change":"2024-09-11T08:11:29.518706+0000","last_active":"2024-09-11T08:11:47.801999+0000","last_peered":"2024-09-11T08:11:47.801999+0000","last_clean":"2024-09-11T08:11:47.801999+0000","last_became_active":"2024-09-11T08:07:34.057937+0000","last_became_peered":"2024-09-11T08:07:34.057937+0000","last_unstale":"2024-09-11T08:11:47.801999+0000","last_undegraded":"2024-09-11T08:11:47.801999+0000","last_fullsized":"2024-09-11T08:11:47.801999+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:11:29.518632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:11:29.518632+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:30.606859+0000","scrub_duration":0.015872065000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":84,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.420457+0000","last_change":"2024-09-11T08:09:15.617419+0000","last_active":"2024-09-11T08:11:42.420457+0000","last_peered":"2024-09-11T08:11:42.420457+0000","last_clean":"2024-09-11T08:11:42.420457+0000","last_became_active":"2024-09-11T08:07:36.068493+0000","last_became_peered":"2024-09-11T08:07:36.068493+0000","last_unstale":"2024-09-11T08:11:42.420457+0000","last_undegraded":"2024-09-11T08:11:42.420457+0000","last_fullsized":"2024-09-11T08:11:42.420457+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:15.617360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:09:15.617360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014734049999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":135,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.430231+0000","last_change":"2024-09-11T08:10:37.122397+0000","last_active":"2024-09-11T08:11:42.430231+0000","last_peered":"2024-09-11T08:11:42.430231+0000","last_clean":"2024-09-11T08:11:42.430231+0000","last_became_active":"2024-09-11T08:09:07.878358+0000","last_became_peered":"2024-09-11T08:09:07.878358+0000","last_unstale":"2024-09-11T08:11:42.430231+0000","last_undegraded":"2024-09-11T08:11:42.430231+0000","last_fullsized":"2024-09-11T08:11:42.430231+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-11T08:10:37.122322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:10:37.122322+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:03.673311+0000","scrub_duration":0.0096164340000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":100,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.431209+0000","last_change":"2024-09-11T08:10:33.261203+0000","last_active":"2024-09-11T08:11:42.431209+0000","last_peered":"2024-09-11T08:11:42.431209+0000","last_clean":"2024-09-11T08:11:42.431209+0000","last_became_active":"2024-09-11T08:07:30.039673+0000","last_became_peered":"2024-09-11T08:07:30.039673+0000","last_unstale":"2024-09-11T08:11:42.431209+0000","last_undegraded":"2024-09-11T08:11:42.431209+0000","last_fullsized":"2024-09-11T08:11:42.431209+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:33.261145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:10:33.261145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:49.943383+0000","scrub_duration":0.004998476,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":155,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-11T08:11:47.806612+0000","last_change":"2024-09-11T08:10:36.942784+0000","last_active":"2024-09-11T08:11:47.806612+0000","last_peered":"2024-09-11T08:11:47.806612+0000","last_clean":"2024-09-11T08:11:47.806612+0000","last_became_active":"2024-09-11T08:07:34.060736+0000","last_became_peered":"2024-09-11T08:07:34.060736+0000","last_unstale":"2024-09-11T08:11:47.806612+0000","last_undegraded":"2024-09-11T08:11:47.806612+0000","last_fullsized":"2024-09-11T08:11:47.806612+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:10:36.942703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:10:36.942703+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:51.228104+0000","scrub_duration":0.017554209000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":84,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.425293+0000","last_change":"2024-09-11T08:09:27.844582+0000","last_active":"2024-09-11T08:11:42.425293+0000","last_peered":"2024-09-11T08:11:42.425293+0000","last_clean":"2024-09-11T08:11:42.425293+0000","last_became_active":"2024-09-11T08:07:36.072359+0000","last_became_peered":"2024-09-11T08:07:36.072359+0000","last_unstale":"2024-09-11T08:11:42.425293+0000","last_undegraded":"2024-09-11T08:11:42.425293+0000","last_fullsized":"2024-09-11T08:11:42.425293+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:27.844523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:09:27.844523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014038360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":117,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.420205+0000","last_change":"2024-09-11T08:11:39.092534+0000","last_active":"2024-09-11T08:11:42.420205+0000","last_peered":"2024-09-11T08:11:42.420205+0000","last_clean":"2024-09-11T08:11:42.420205+0000","last_became_active":"2024-09-11T08:07:32.047981+0000","last_became_peered":"2024-09-11T08:07:32.047981+0000","last_unstale":"2024-09-11T08:11:42.420205+0000","last_undegraded":"2024-09-11T08:11:42.420205+0000","last_fullsized":"2024-09-11T08:11:42.420205+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-11T08:11:39.092431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:11:39.092431+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:51.100772+0000","scrub_duration":0.0069898629999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":100,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.420855+0000","last_change":"2024-09-11T08:10:36.134384+0000","last_active":"2024-09-11T08:11:42.420855+0000","last_peered":"2024-09-11T08:11:42.420855+0000","last_clean":"2024-09-11T08:11:42.420855+0000","last_became_active":"2024-09-11T08:07:30.043691+0000","last_became_peered":"2024-09-11T08:07:30.043691+0000","last_unstale":"2024-09-11T08:11:42.420855+0000","last_undegraded":"2024-09-11T08:11:42.420855+0000","last_fullsized":"2024-09-11T08:11:42.420855+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:36.134300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:10:36.134300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:01.485017+0000","scrub_duration":0.0020719419999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":96,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.426462+0000","last_change":"2024-09-11T08:11:25.585872+0000","last_active":"2024-09-11T08:11:42.426462+0000","last_peered":"2024-09-11T08:11:42.426462+0000","last_clean":"2024-09-11T08:11:42.426462+0000","last_became_active":"2024-09-11T08:07:34.053137+0000","last_became_peered":"2024-09-11T08:07:34.053137+0000","last_unstale":"2024-09-11T08:11:42.426462+0000","last_undegraded":"2024-09-11T08:11:42.426462+0000","last_fullsized":"2024-09-11T08:11:42.426462+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:25.585537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:11:25.585537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:51.823113+0000","scrub_duration":0.015833313000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":84,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.429903+0000","last_change":"2024-09-11T08:10:30.195470+0000","last_active":"2024-09-11T08:11:42.429903+0000","last_peered":"2024-09-11T08:11:42.429903+0000","last_clean":"2024-09-11T08:11:42.429903+0000","last_became_active":"2024-09-11T08:07:36.076132+0000","last_became_peered":"2024-09-11T08:07:36.076132+0000","last_unstale":"2024-09-11T08:11:42.429903+0000","last_undegraded":"2024-09-11T08:11:42.429903+0000","last_fullsized":"2024-09-11T08:11:42.429903+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:30.195389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:10:30.195389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:58.081266+0000","scrub_duration":0.013816314999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":114,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.426730+0000","last_change":"2024-09-11T08:10:32.411685+0000","last_active":"2024-09-11T08:11:42.426730+0000","last_peered":"2024-09-11T08:11:42.426730+0000","last_clean":"2024-09-11T08:11:42.426730+0000","last_became_active":"2024-09-11T08:07:32.049368+0000","last_became_peered":"2024-09-11T08:07:32.049368+0000","last_unstale":"2024-09-11T08:11:42.426730+0000","last_undegraded":"2024-09-11T08:11:42.426730+0000","last_fullsized":"2024-09-11T08:11:42.426730+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:10:32.411521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:10:32.411521+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0076800619999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":110,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.431550+0000","last_change":"2024-09-11T08:11:39.203061+0000","last_active":"2024-09-11T08:11:42.431550+0000","last_peered":"2024-09-11T08:11:42.431550+0000","last_clean":"2024-09-11T08:11:42.431550+0000","last_became_active":"2024-09-11T08:07:30.037692+0000","last_became_peered":"2024-09-11T08:07:30.037692+0000","last_unstale":"2024-09-11T08:11:42.431550+0000","last_undegraded":"2024-09-11T08:11:42.431550+0000","last_fullsized":"2024-09-11T08:11:42.431550+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:39.202984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:11:39.202984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:51.576908+0000","scrub_duration":0.001252852,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":96,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.431051+0000","last_change":"2024-09-11T08:11:35.568457+0000","last_active":"2024-09-11T08:11:42.431051+0000","last_peered":"2024-09-11T08:11:42.431051+0000","last_clean":"2024-09-11T08:11:42.431051+0000","last_became_active":"2024-09-11T08:07:34.057064+0000","last_became_peered":"2024-09-11T08:07:34.057064+0000","last_unstale":"2024-09-11T08:11:42.431051+0000","last_undegraded":"2024-09-11T08:11:42.431051+0000","last_fullsized":"2024-09-11T08:11:42.431051+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:35.567888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:11:35.567888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:13:05.264013+0000","scrub_duration":0.029638675999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":94,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.425254+0000","last_change":"2024-09-11T08:11:22.043711+0000","last_active":"2024-09-11T08:11:42.425254+0000","last_peered":"2024-09-11T08:11:42.425254+0000","last_clean":"2024-09-11T08:11:42.425254+0000","last_became_active":"2024-09-11T08:07:36.077406+0000","last_became_peered":"2024-09-11T08:07:36.077406+0000","last_unstale":"2024-09-11T08:11:42.425254+0000","last_undegraded":"2024-09-11T08:11:42.425254+0000","last_fullsized":"2024-09-11T08:11:42.425254+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:22.043633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:09.973943+0000","last_clean_scrub_stamp":"2024-09-11T08:11:22.043633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:30.337221+0000","scrub_duration":0.0064821080000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":117,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.430075+0000","last_change":"2024-09-11T08:11:33.518272+0000","last_active":"2024-09-11T08:11:42.430075+0000","last_peered":"2024-09-11T08:11:42.430075+0000","last_clean":"2024-09-11T08:11:42.430075+0000","last_became_active":"2024-09-11T08:07:32.043276+0000","last_became_peered":"2024-09-11T08:07:32.043276+0000","last_unstale":"2024-09-11T08:11:42.430075+0000","last_undegraded":"2024-09-11T08:11:42.430075+0000","last_fullsized":"2024-09-11T08:11:42.430075+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-11T08:11:33.518136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:11:33.518136+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:43.052462+0000","scrub_duration":0.0058292969999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":99,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.428652+0000","last_change":"2024-09-11T08:10:23.978440+0000","last_active":"2024-09-11T08:11:42.428652+0000","last_peered":"2024-09-11T08:11:42.428652+0000","last_clean":"2024-09-11T08:11:42.428652+0000","last_became_active":"2024-09-11T08:07:30.049152+0000","last_became_peered":"2024-09-11T08:07:30.049152+0000","last_unstale":"2024-09-11T08:11:42.428652+0000","last_undegraded":"2024-09-11T08:11:42.428652+0000","last_fullsized":"2024-09-11T08:11:42.428652+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:23.978385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:10:23.978385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:48.317452+0000","scrub_duration":0.0025160999999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":96,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.430737+0000","last_change":"2024-09-11T08:11:34.159199+0000","last_active":"2024-09-11T08:11:42.430737+0000","last_peered":"2024-09-11T08:11:42.430737+0000","last_clean":"2024-09-11T08:11:42.430737+0000","last_became_active":"2024-09-11T08:07:34.056935+0000","last_became_peered":"2024-09-11T08:07:34.056935+0000","last_unstale":"2024-09-11T08:11:42.430737+0000","last_undegraded":"2024-09-11T08:11:42.430737+0000","last_fullsized":"2024-09-11T08:11:42.430737+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:34.159053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:11:34.159053+0000","last_clean_scrub_stamp":"2024-09-11T08:11:34.159053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:35.241223+0000","scrub_duration":0.00130974,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":84,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.431175+0000","last_change":"2024-09-11T08:09:34.378812+0000","last_active":"2024-09-11T08:11:42.431175+0000","last_peered":"2024-09-11T08:11:42.431175+0000","last_clean":"2024-09-11T08:11:42.431175+0000","last_became_active":"2024-09-11T08:07:36.633368+0000","last_became_peered":"2024-09-11T08:07:36.633368+0000","last_unstale":"2024-09-11T08:11:42.431175+0000","last_undegraded":"2024-09-11T08:11:42.431175+0000","last_fullsized":"2024-09-11T08:11:42.431175+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:34.378734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:34.378734+0000","last_clean_scrub_stamp":"2024-09-11T08:09:34.378734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001593148,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":94,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.436472+0000","last_change":"2024-09-11T08:09:40.373717+0000","last_active":"2024-09-11T08:11:42.436472+0000","last_peered":"2024-09-11T08:11:42.436472+0000","last_clean":"2024-09-11T08:11:42.436472+0000","last_became_active":"2024-09-11T08:07:32.045728+0000","last_became_peered":"2024-09-11T08:07:32.045728+0000","last_unstale":"2024-09-11T08:11:42.436472+0000","last_undegraded":"2024-09-11T08:11:42.436472+0000","last_fullsized":"2024-09-11T08:11:42.436472+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'4","last_scrub_stamp":"2024-09-11T08:09:40.373649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:09:40.373649+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001798315,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":100,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.430434+0000","last_change":"2024-09-11T08:10:28.129119+0000","last_active":"2024-09-11T08:11:42.430434+0000","last_peered":"2024-09-11T08:11:42.430434+0000","last_clean":"2024-09-11T08:11:42.430434+0000","last_became_active":"2024-09-11T08:07:30.049409+0000","last_became_peered":"2024-09-11T08:07:30.049409+0000","last_unstale":"2024-09-11T08:11:42.430434+0000","last_undegraded":"2024-09-11T08:11:42.430434+0000","last_fullsized":"2024-09-11T08:11:42.430434+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:28.129035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:10:28.129035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.019365614999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":105,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.424435+0000","last_change":"2024-09-11T08:11:38.545774+0000","last_active":"2024-09-11T08:11:42.424435+0000","last_peered":"2024-09-11T08:11:42.424435+0000","last_clean":"2024-09-11T08:11:42.424435+0000","last_became_active":"2024-09-11T08:07:34.055082+0000","last_became_peered":"2024-09-11T08:07:34.055082+0000","last_unstale":"2024-09-11T08:11:42.424435+0000","last_undegraded":"2024-09-11T08:11:42.424435+0000","last_fullsized":"2024-09-11T08:11:42.424435+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:38.545717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:11:38.545717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:43.268646+0000","scrub_duration":0.0054392340000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":94,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.430838+0000","last_change":"2024-09-11T08:10:42.297248+0000","last_active":"2024-09-11T08:11:42.430838+0000","last_peered":"2024-09-11T08:11:42.430838+0000","last_clean":"2024-09-11T08:11:42.430838+0000","last_became_active":"2024-09-11T08:07:36.071669+0000","last_became_peered":"2024-09-11T08:07:36.071669+0000","last_unstale":"2024-09-11T08:11:42.430838+0000","last_undegraded":"2024-09-11T08:11:42.430838+0000","last_fullsized":"2024-09-11T08:11:42.430838+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:42.297166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:10:42.297166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:07.034868+0000","scrub_duration":0.0077635450000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":94,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.428808+0000","last_change":"2024-09-11T08:10:28.457397+0000","last_active":"2024-09-11T08:11:42.428808+0000","last_peered":"2024-09-11T08:11:42.428808+0000","last_clean":"2024-09-11T08:11:42.428808+0000","last_became_active":"2024-09-11T08:07:36.071468+0000","last_became_peered":"2024-09-11T08:07:36.071468+0000","last_unstale":"2024-09-11T08:11:42.428808+0000","last_undegraded":"2024-09-11T08:11:42.428808+0000","last_fullsized":"2024-09-11T08:11:42.428808+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:28.457304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:10:28.457304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0063908749999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":100,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.436734+0000","last_change":"2024-09-11T08:11:26.404087+0000","last_active":"2024-09-11T08:11:42.436734+0000","last_peered":"2024-09-11T08:11:42.436734+0000","last_clean":"2024-09-11T08:11:42.436734+0000","last_became_active":"2024-09-11T08:07:30.046702+0000","last_became_peered":"2024-09-11T08:07:30.046702+0000","last_unstale":"2024-09-11T08:11:42.436734+0000","last_undegraded":"2024-09-11T08:11:42.436734+0000","last_fullsized":"2024-09-11T08:11:42.436734+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:26.403944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:11:26.403944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:29.575666+0000","scrub_duration":0.0056165199999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":116,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.424187+0000","last_change":"2024-09-11T08:10:24.612372+0000","last_active":"2024-09-11T08:11:42.424187+0000","last_peered":"2024-09-11T08:11:42.424187+0000","last_clean":"2024-09-11T08:11:42.424187+0000","last_became_active":"2024-09-11T08:07:32.054656+0000","last_became_peered":"2024-09-11T08:07:32.054656+0000","last_unstale":"2024-09-11T08:11:42.424187+0000","last_undegraded":"2024-09-11T08:11:42.424187+0000","last_fullsized":"2024-09-11T08:11:42.424187+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-11T08:10:24.612282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:10:24.612282+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:52.941292+0000","scrub_duration":0.0020463640000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":96,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-11T08:11:42.425462+0000","last_change":"2024-09-11T08:10:32.091677+0000","last_active":"2024-09-11T08:11:42.425462+0000","last_peered":"2024-09-11T08:11:42.425462+0000","last_clean":"2024-09-11T08:11:42.425462+0000","last_became_active":"2024-09-11T08:07:34.061453+0000","last_became_peered":"2024-09-11T08:07:34.061453+0000","last_unstale":"2024-09-11T08:11:42.425462+0000","last_undegraded":"2024-09-11T08:11:42.425462+0000","last_fullsized":"2024-09-11T08:11:42.425462+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:32.091605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:10:32.091605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:11:52.537395+0000","scrub_duration":0.0071709479999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":6475481112,"num_objects":98809,"num_object_clones":0,"num_object_copies":296427,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98809,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150069,"num_write_kb":7964033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19424882688,"data_stored":19424870472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89905,"ondisk_log_size":89905,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":435,"ondisk_log_size":435,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":29,"num_read_kb":29,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":256,"num_read_kb":251,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266745,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2290080,"kb_used_data":1988568,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":91455072,"statfs":{"total":95995035648,"available":93649993728,"internally_reserved":0,"allocated":2036293632,"data_stored":2035888416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:11:23 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.52300000000000002,"15min":0.496},"min":{"1min":0.20300000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":3.4119999999999999,"5min":3.4119999999999999,"15min":3.4119999999999999},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.58499999999999996,"15min":0.51700000000000002},"min":{"1min":0.251,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.3279999999999998,"5min":2.3279999999999998,"15min":2.3279999999999998},"last":0.81000000000000005}]},{"osd":1,"last update":"Wed Sep 11 08:11:23 2024","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.59499999999999997,"15min":0.52700000000000002},"min":{"1min":0.23000000000000001,"5min":0.216,"15min":0.216},"max":{"1min":3.423,"5min":3.423,"15min":3.423},"last":1.0820000000000001},{"interface":"front","average":{"1min":0.997,"5min":0.63300000000000001,"15min":0.56299999999999994},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":3.4409999999999998,"5min":3.4409999999999998,"15min":3.4409999999999998},"last":1.097}]},{"osd":2,"last update":"Wed Sep 11 08:11:23 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":0.67600000000000005,"15min":0.54600000000000004},"min":{"1min":0.26500000000000001,"5min":0.254,"15min":0.254},"max":{"1min":4.1589999999999998,"5min":5.5919999999999996,"15min":5.5919999999999996},"last":0.91400000000000003},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.52100000000000002,"15min":0.50700000000000001},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":1.302,"5min":2.677,"15min":2.677},"last":0.30099999999999999}]},{"osd":3,"last update":"Wed Sep 11 08:11:23 2024","interfaces":[{"interface":"back","average":{"1min":2.278,"5min":1.03,"15min":0.94799999999999995},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":10.818,"5min":10.818,"15min":10.818},"last":0.73199999999999998},{"interface":"front","average":{"1min":1.329,"5min":0.66900000000000004,"15min":0.54400000000000004},"min":{"1min":0.29999999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":5.7080000000000002,"5min":5.7080000000000002,"15min":5.7080000000000002},"last":0.27600000000000002}]},{"osd":4,"last update":"Wed Sep 11 08:11:23 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.41699999999999998,"15min":0.40699999999999997},"min":{"1min":0.215,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":1.0580000000000001,"5min":1.7050000000000001,"15min":1.7050000000000001},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.42499999999999999,"15min":0.41899999999999998},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.92800000000000005,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":0.35999999999999999}]},{"osd":5,"last update":"Wed Sep 11 08:11:23 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.40600000000000003,"15min":0.41799999999999998},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.60799999999999998,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.503,"5min":0.47399999999999998,"15min":0.46200000000000002},"min":{"1min":0.26700000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.1000000000000001,"5min":1.6799999999999999,"15min":1.6799999999999999},"last":0.34300000000000003}]},{"osd":6,"last update":"Wed Sep 11 08:11:23 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50700000000000001,"15min":0.499},"min":{"1min":0.29099999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.014,"5min":1.819,"15min":1.819},"last":0.311},{"interface":"front","average":{"1min":0.497,"5min":0.48299999999999998,"15min":0.47199999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.84699999999999998,"5min":1.492,"15min":1.492},"last":0.45700000000000002}]}]},{"osd":6,"up_from":45,"seq":193273528382,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2287956,"kb_used_data":1978124,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":91457196,"statfs":{"total":95995035648,"available":93652168704,"internally_reserved":0,"allocated":2025598976,"data_stored":2025205895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:11:02 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.54600000000000004,"15min":0.51400000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":2.5870000000000002,"5min":2.5870000000000002,"15min":2.5870000000000002},"last":2.4020000000000001},{"interface":"front","average":{"1min":1.1140000000000001,"5min":0.623,"15min":0.54500000000000004},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":5.9320000000000004,"5min":5.9320000000000004,"15min":5.9320000000000004},"last":0.68300000000000005}]},{"osd":1,"last update":"Wed Sep 11 08:11:02 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.50700000000000001,"15min":0.48699999999999999},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":4.8700000000000001,"5min":4.8700000000000001,"15min":4.8700000000000001},"last":2.4319999999999999},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.54100000000000004,"15min":0.51800000000000002},"min":{"1min":0.27600000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":2.8199999999999998,"5min":2.8199999999999998,"15min":2.8199999999999998},"last":2.6520000000000001}]},{"osd":2,"last update":"Wed Sep 11 08:11:02 2024","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.58099999999999996,"15min":0.52900000000000003},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":2.6840000000000002,"5min":2.6840000000000002,"15min":2.6840000000000002},"last":2.681},{"interface":"front","average":{"1min":1.0389999999999999,"5min":0.59399999999999997,"15min":0.51900000000000002},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":2.5489999999999999,"5min":2.5489999999999999,"15min":2.5489999999999999},"last":1.4570000000000001}]},{"osd":3,"last update":"Wed Sep 11 08:11:02 2024","interfaces":[{"interface":"back","average":{"1min":1.5029999999999999,"5min":0.70999999999999996,"15min":0.58799999999999997},"min":{"1min":0.34799999999999998,"5min":0.308,"15min":0.308},"max":{"1min":4.343,"5min":4.343,"15min":4.343},"last":3.7450000000000001},{"interface":"front","average":{"1min":1.2609999999999999,"5min":0.67600000000000005,"15min":0.58399999999999996},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":4.3540000000000001,"5min":4.3540000000000001,"15min":4.3540000000000001},"last":3.141}]},{"osd":4,"last update":"Wed Sep 11 08:11:02 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.40699999999999997,"15min":0.39300000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.441,"15min":0.41899999999999998},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.70199999999999996}]},{"osd":5,"last update":"Wed Sep 11 08:11:02 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.501,"15min":0.48399999999999999},"min":{"1min":0.29399999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.22,"5min":1.22,"15min":1.22},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.44500000000000001,"15min":0.42099999999999999},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.653,"5min":2.653,"15min":2.653},"last":2.3719999999999999}]},{"osd":7,"last update":"Wed Sep 11 08:11:27 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.49299999999999999,"15min":0.47299999999999998},"min":{"1min":0.31900000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.359,"5min":1.359,"15min":1.359},"last":2.4500000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.46899999999999997,"15min":0.44500000000000001},"min":{"1min":0.28199999999999997,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":2.702}]}]},{"osd":5,"up_from":38,"seq":163208757315,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1911824,"kb_used_data":1606152,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":91833328,"statfs":{"total":95995035648,"available":94037327872,"internally_reserved":0,"allocated":1644699648,"data_stored":1644309656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:11:40 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":0.73299999999999998,"15min":0.59499999999999997},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.4299999999999997,"5min":6.343,"15min":6.343},"last":0.308},{"interface":"front","average":{"1min":1.0269999999999999,"5min":0.71299999999999997,"15min":0.57099999999999995},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.5150000000000001,"5min":4.6319999999999997,"15min":4.6319999999999997},"last":0.253}]},{"osd":1,"last update":"Wed Sep 11 08:11:40 2024","interfaces":[{"interface":"back","average":{"1min":1.694,"5min":0.98299999999999998,"15min":1.2330000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":12.847,"5min":16.004999999999999,"15min":16.004999999999999},"last":0.54700000000000004},{"interface":"front","average":{"1min":1.758,"5min":0.76300000000000001,"15min":0.59299999999999997},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":13.215999999999999,"5min":13.215999999999999,"15min":13.215999999999999},"last":0.53700000000000003}]},{"osd":2,"last update":"Wed Sep 11 08:11:40 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":0.83399999999999996,"15min":1.1990000000000001},"min":{"1min":0.308,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":5.1669999999999998,"5min":16.018000000000001,"15min":16.018000000000001},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.63500000000000001,"15min":0.55900000000000005},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":4.7469999999999999,"5min":4.7469999999999999,"15min":4.7469999999999999},"last":0.32300000000000001}]},{"osd":3,"last update":"Wed Sep 11 08:11:40 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":0.78000000000000003,"15min":0.61199999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":4.7300000000000004,"5min":6.3230000000000004,"15min":6.3230000000000004},"last":2.7989999999999999},{"interface":"front","average":{"1min":1.242,"5min":0.75,"15min":0.59599999999999997},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":7.3159999999999998,"5min":7.3159999999999998,"15min":7.3159999999999998},"last":0.90800000000000003}]},{"osd":4,"last update":"Wed Sep 11 08:11:40 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.46300000000000002,"15min":0.41499999999999998},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":2.0779999999999998,"5min":2.0779999999999998,"15min":2.0779999999999998},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.48299999999999998,"15min":0.41499999999999998},"min":{"1min":0.26700000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":2.3090000000000002,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":0.498}]},{"osd":6,"last update":"Wed Sep 11 08:11:00 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.45800000000000002,"15min":0.432},"min":{"1min":0.254,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":1.3080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.505,"15min":0.46300000000000002},"min":{"1min":0.30399999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.319,"5min":1.319,"15min":1.319},"last":0.51400000000000001}]},{"osd":7,"last update":"Wed Sep 11 08:11:28 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.52700000000000002,"15min":0.50900000000000001},"min":{"1min":0.32000000000000001,"5min":0.307,"15min":0.307},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.37},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.55200000000000005,"15min":0.501},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":2.448,"5min":2.448,"15min":2.448},"last":0.55800000000000005}]}]},{"osd":4,"up_from":31,"seq":133143986248,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2304932,"kb_used_data":1999260,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":91440220,"statfs":{"total":95995035648,"available":93634785280,"internally_reserved":0,"allocated":2047242240,"data_stored":2046833082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:11:13 2024","interfaces":[{"interface":"back","average":{"1min":3.1589999999999998,"5min":1.6259999999999999,"15min":0.88400000000000001},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":26.754999999999999,"5min":48.728999999999999,"15min":48.728999999999999},"last":2.6739999999999999},{"interface":"front","average":{"1min":2.4289999999999998,"5min":1.4079999999999999,"15min":0.82999999999999996},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":27.055,"5min":49.168999999999997,"15min":49.168999999999997},"last":0.27200000000000002}]},{"osd":1,"last update":"Wed Sep 11 08:11:13 2024","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":0.73899999999999999,"15min":0.59099999999999997},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":2.6669999999999998,"5min":13.207000000000001,"15min":13.207000000000001},"last":2.98},{"interface":"front","average":{"1min":1.163,"5min":0.64700000000000002,"15min":0.57099999999999995},"min":{"1min":0.25600000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":3.2679999999999998,"5min":3.2679999999999998,"15min":3.2679999999999998},"last":2.3879999999999999}]},{"osd":2,"last update":"Wed Sep 11 08:11:13 2024","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":0.64000000000000001,"15min":0.56799999999999995},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":3.9750000000000001,"5min":3.9750000000000001,"15min":3.9750000000000001},"last":1.466},{"interface":"front","average":{"1min":1.6679999999999999,"5min":0.74399999999999999,"15min":0.60099999999999998},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":5.5670000000000002,"5min":5.5670000000000002,"15min":5.5670000000000002},"last":1.5149999999999999}]},{"osd":3,"last update":"Wed Sep 11 08:11:13 2024","interfaces":[{"interface":"back","average":{"1min":2.7599999999999998,"5min":1.274,"15min":0.77000000000000002},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":26.884,"5min":26.884,"15min":26.884},"last":1.349},{"interface":"front","average":{"1min":2.9710000000000001,"5min":1.1819999999999999,"15min":0.747},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":26.783000000000001,"5min":26.783000000000001,"15min":26.783000000000001},"last":3.778}]},{"osd":5,"last update":"Wed Sep 11 08:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.40999999999999998,"15min":0.38},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":1.151,"5min":1.151,"15min":1.151},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.439,"15min":0.39000000000000001},"min":{"1min":0.28199999999999997,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":1.9630000000000001,"5min":1.9630000000000001,"15min":1.9630000000000001},"last":0.54900000000000004}]},{"osd":6,"last update":"Wed Sep 11 08:11:00 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.40500000000000003,"15min":0.38800000000000001},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.61399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.438},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.48499999999999999,"15min":0.44700000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.70199999999999996}]},{"osd":7,"last update":"Wed Sep 11 08:11:28 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.49099999999999999,"15min":0.46300000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.097,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.627,"5min":0.498,"15min":0.46600000000000003},"min":{"1min":0.32700000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.931,"5min":1.931,"15min":1.931},"last":0.58199999999999996}]}]},{"osd":3,"up_from":25,"seq":107374182477,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3880176,"kb_used_data":3570152,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":89864976,"statfs":{"total":95995035648,"available":92021735424,"internally_reserved":0,"allocated":3655835648,"data_stored":3655413673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:10:59 2024","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":0.55600000000000005,"15min":0.438},"min":{"1min":0.27700000000000002,"5min":0.191,"15min":0.191},"max":{"1min":4.2779999999999996,"5min":4.2779999999999996,"15min":4.2779999999999996},"last":2.6499999999999999},{"interface":"front","average":{"1min":1.333,"5min":0.55900000000000005,"15min":0.46999999999999997},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":3.1960000000000002,"5min":3.1960000000000002,"15min":3.1960000000000002},"last":8.0790000000000006}]},{"osd":1,"last update":"Wed Sep 11 08:10:59 2024","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":0.54100000000000004,"15min":0.47899999999999998},"min":{"1min":0.28699999999999998,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":3.5910000000000002,"5min":3.5910000000000002,"15min":3.5910000000000002},"last":1.8140000000000001},{"interface":"front","average":{"1min":1.9850000000000001,"5min":0.70299999999999996,"15min":0.52500000000000002},"min":{"1min":0.252,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":8.5,"5min":8.5,"15min":8.5},"last":3.8889999999999998}]},{"osd":2,"last update":"Wed Sep 11 08:10:59 2024","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":0.59399999999999997,"15min":0.502},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":5.835,"5min":5.835,"15min":5.835},"last":3.9359999999999999},{"interface":"front","average":{"1min":1.556,"5min":0.61299999999999999,"15min":0.48799999999999999},"min":{"1min":0.22500000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":8.5950000000000006,"5min":8.5950000000000006,"15min":8.5950000000000006},"last":2.6219999999999999}]},{"osd":4,"last update":"Wed Sep 11 08:11:18 2024","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":0.73699999999999999,"15min":0.61199999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":5.157,"5min":5.157,"15min":5.157},"last":3.8519999999999999},{"interface":"front","average":{"1min":1.4530000000000001,"5min":0.70399999999999996,"15min":0.59399999999999997},"min":{"1min":0.312,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":10.563000000000001,"5min":10.563000000000001,"15min":10.563000000000001},"last":1.8420000000000001}]},{"osd":5,"last update":"Wed Sep 11 08:11:39 2024","interfaces":[{"interface":"back","average":{"1min":1.899,"5min":0.85199999999999998,"15min":0.64200000000000002},"min":{"1min":0.26600000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":10.496,"5min":10.496,"15min":10.496},"last":1.901},{"interface":"front","average":{"1min":1.331,"5min":0.96499999999999997,"15min":0.68600000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":6.4279999999999999,"5min":16.757000000000001,"15min":16.757000000000001},"last":1.7430000000000001}]},{"osd":6,"last update":"Wed Sep 11 08:11:03 2024","interfaces":[{"interface":"back","average":{"1min":1.329,"5min":0.89700000000000002,"15min":0.65100000000000002},"min":{"1min":0.33400000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":6.4489999999999998,"5min":16.789000000000001,"15min":16.789000000000001},"last":1.77},{"interface":"front","average":{"1min":2.0289999999999999,"5min":1.0369999999999999,"15min":0.71899999999999997},"min":{"1min":0.40899999999999997,"5min":0.373,"15min":0.373},"max":{"1min":11.122999999999999,"5min":16.748000000000001,"15min":16.748000000000001},"last":3.915}]},{"osd":7,"last update":"Wed Sep 11 08:11:29 2024","interfaces":[{"interface":"back","average":{"1min":1.784,"5min":0.82099999999999995,"15min":0.64100000000000001},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":10.538,"5min":10.538,"15min":10.538},"last":1.8660000000000001},{"interface":"front","average":{"1min":1.6919999999999999,"5min":0.84799999999999998,"15min":0.67200000000000004},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":11.098000000000001,"5min":11.098000000000001,"15min":11.098000000000001},"last":1.79}]}]},{"osd":2,"up_from":18,"seq":77309411408,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2678992,"kb_used_data":2381640,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":91066160,"statfs":{"total":95995035648,"available":93251747840,"internally_reserved":0,"allocated":2438799360,"data_stored":2438409280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:11:33 2024","interfaces":[{"interface":"back","average":{"1min":2.157,"5min":0.86199999999999999,"15min":0.54000000000000004},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":6.468,"5min":6.468,"15min":6.468},"last":0.40699999999999997},{"interface":"front","average":{"1min":2.2360000000000002,"5min":0.85199999999999998,"15min":0.52100000000000002},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":6.9080000000000004,"5min":6.9080000000000004,"15min":6.9080000000000004},"last":0.58599999999999997}]},{"osd":1,"last update":"Wed Sep 11 08:11:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7709999999999999,"5min":0.83499999999999996,"15min":0.52700000000000002},"min":{"1min":0.26100000000000001,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":6.3499999999999996,"5min":6.3499999999999996,"15min":6.3499999999999996},"last":1.0660000000000001},{"interface":"front","average":{"1min":1.7350000000000001,"5min":0.81000000000000005,"15min":0.52400000000000002},"min":{"1min":0.23200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":5.6970000000000001,"5min":5.6970000000000001,"15min":5.6970000000000001},"last":4.3099999999999996}]},{"osd":3,"last update":"Wed Sep 11 08:10:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":0.55600000000000005,"15min":0.496},"min":{"1min":0.32400000000000001,"5min":0.218,"15min":0.218},"max":{"1min":3.089,"5min":3.089,"15min":3.089},"last":1.0209999999999999},{"interface":"front","average":{"1min":1.1419999999999999,"5min":0.56699999999999995,"15min":0.503},"min":{"1min":0.35399999999999998,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":3.6440000000000001,"5min":3.6440000000000001,"15min":3.6440000000000001},"last":0.69599999999999995}]},{"osd":4,"last update":"Wed Sep 11 08:11:16 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":0.74399999999999999,"15min":0.60199999999999998},"min":{"1min":0.311,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":7.806,"5min":7.806,"15min":7.806},"last":0.46899999999999997},{"interface":"front","average":{"1min":1.4510000000000001,"5min":0.872,"15min":0.68300000000000005},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":6.8200000000000003,"5min":12.129,"15min":12.129},"last":0.28999999999999998}]},{"osd":5,"last update":"Wed Sep 11 08:11:45 2024","interfaces":[{"interface":"back","average":{"1min":1.7749999999999999,"5min":0.89700000000000002,"15min":0.68899999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":6.7709999999999999,"5min":6.7709999999999999,"15min":6.7709999999999999},"last":0.31900000000000001},{"interface":"front","average":{"1min":1.329,"5min":0.76200000000000001,"15min":0.65700000000000003},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":6.4050000000000002,"5min":6.4050000000000002,"15min":6.4050000000000002},"last":0.54700000000000004}]},{"osd":6,"last update":"Wed Sep 11 08:10:58 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.64100000000000001,"15min":0.61299999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":4.2350000000000003,"5min":4.2350000000000003,"15min":4.2350000000000003},"last":0.34999999999999998},{"interface":"front","average":{"1min":1.21,"5min":0.70599999999999996,"15min":0.63600000000000001},"min":{"1min":0.376,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":2.9249999999999998,"5min":2.9249999999999998,"15min":2.9249999999999998},"last":1.0980000000000001}]},{"osd":7,"last update":"Wed Sep 11 08:11:27 2024","interfaces":[{"interface":"back","average":{"1min":1.45,"5min":0.81399999999999995,"15min":0.65400000000000003},"min":{"1min":0.34799999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":7.7560000000000002,"5min":7.7560000000000002,"15min":7.7560000000000002},"last":0.51500000000000001},{"interface":"front","average":{"1min":2.0409999999999999,"5min":1.0309999999999999,"15min":0.74399999999999999},"min":{"1min":0.40100000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":6.4260000000000002,"5min":12.115,"15min":12.115},"last":0.436}]}]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2667832,"kb_used_data":2362160,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":91077320,"statfs":{"total":95995035648,"available":93263175680,"internally_reserved":0,"allocated":2418851840,"data_stored":2418421840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:11:10 2024","interfaces":[{"interface":"back","average":{"1min":3.6480000000000001,"5min":1.0580000000000001,"15min":0.54400000000000004},"min":{"1min":0.245,"5min":0.22600000000000001,"15min":0.16300000000000001},"max":{"1min":32.478000000000002,"5min":32.478000000000002,"15min":32.478000000000002},"last":4.4119999999999999},{"interface":"front","average":{"1min":4.6200000000000001,"5min":1.2849999999999999,"15min":0.63},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20300000000000001},"max":{"1min":35.459000000000003,"5min":35.459000000000003,"15min":35.459000000000003},"last":1.0649999999999999}]},{"osd":2,"last update":"Wed Sep 11 08:11:39 2024","interfaces":[{"interface":"back","average":{"1min":2.4300000000000002,"5min":0.88500000000000001,"15min":0.52300000000000002},"min":{"1min":0.38200000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":8.3819999999999997,"5min":8.3819999999999997,"15min":8.3819999999999997},"last":4.351},{"interface":"front","average":{"1min":3.4700000000000002,"5min":1.175,"15min":0.61499999999999999},"min":{"1min":0.30499999999999999,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":10.276,"5min":10.276,"15min":10.276},"last":6.6349999999999998}]},{"osd":3,"last update":"Wed Sep 11 08:11:52 2024","interfaces":[{"interface":"back","average":{"1min":3.2719999999999998,"5min":1.8140000000000001,"15min":0.88800000000000001},"min":{"1min":0.254,"5min":0.246,"15min":0.246},"max":{"1min":10.102,"5min":32.759,"15min":32.759},"last":7.1100000000000003},{"interface":"front","average":{"1min":2.6699999999999999,"5min":1.5269999999999999,"15min":0.79900000000000004},"min":{"1min":0.29199999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":8.4030000000000005,"5min":32.444000000000003,"15min":32.444000000000003},"last":4.3849999999999998}]},{"osd":4,"last update":"Wed Sep 11 08:11:17 2024","interfaces":[{"interface":"back","average":{"1min":2.1640000000000001,"5min":0.91500000000000004,"15min":0.66600000000000004},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":9.0139999999999993,"5min":9.0139999999999993,"15min":9.0139999999999993},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.873,"5min":0.83699999999999997,"15min":0.64900000000000002},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":8.2360000000000007,"5min":8.2360000000000007,"15min":8.2360000000000007},"last":0.44800000000000001}]},{"osd":5,"last update":"Wed Sep 11 08:11:43 2024","interfaces":[{"interface":"back","average":{"1min":2.2389999999999999,"5min":1.103,"15min":0.71899999999999997},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":8.6769999999999996,"5min":9.0470000000000006,"15min":9.0470000000000006},"last":1.2649999999999999},{"interface":"front","average":{"1min":2.5640000000000001,"5min":1.1040000000000001,"15min":0.71799999999999997},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":8.9860000000000007,"5min":11.967000000000001,"15min":11.967000000000001},"last":0.505}]},{"osd":6,"last update":"Wed Sep 11 08:10:57 2024","interfaces":[{"interface":"back","average":{"1min":1.3080000000000001,"5min":0.83699999999999997,"15min":0.65300000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":5.968,"5min":11.935,"15min":11.935},"last":0.52800000000000002},{"interface":"front","average":{"1min":1.385,"5min":0.74299999999999999,"15min":0.64900000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":4.46,"5min":4.46,"15min":4.46},"last":0.38900000000000001}]},{"osd":7,"last update":"Wed Sep 11 08:11:22 2024","interfaces":[{"interface":"back","average":{"1min":2.3690000000000002,"5min":0.96599999999999997,"15min":0.73699999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":8.2929999999999993,"5min":8.2929999999999993,"15min":8.2929999999999993},"last":0.42599999999999999},{"interface":"front","average":{"1min":2.4079999999999999,"5min":0.97999999999999998,"15min":0.70399999999999996},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":9.0749999999999993,"5min":9.0749999999999993,"15min":9.0749999999999993},"last":1.294}]}]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3493424,"kb_used_data":3183272,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":90251728,"statfs":{"total":95995035648,"available":92417769472,"internally_reserved":0,"allocated":3259670528,"data_stored":3259249898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 11 08:11:11 2024","interfaces":[{"interface":"back","average":{"1min":2.0939999999999999,"5min":0.72599999999999998,"15min":0.443},"min":{"1min":0.189,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":6.0830000000000002,"5min":6.0830000000000002,"15min":6.0830000000000002},"last":3.4180000000000001},{"interface":"front","average":{"1min":1.8500000000000001,"5min":0.67200000000000004,"15min":0.442},"min":{"1min":0.254,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.0469999999999997,"5min":6.0469999999999997,"15min":6.0469999999999997},"last":2.9870000000000001}]},{"osd":2,"last update":"Wed Sep 11 08:11:36 2024","interfaces":[{"interface":"back","average":{"1min":2.161,"5min":0.85699999999999998,"15min":0.51500000000000001},"min":{"1min":0.375,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.1029999999999998,"5min":6.1029999999999998,"15min":6.1029999999999998},"last":1.778},{"interface":"front","average":{"1min":2.5470000000000002,"5min":0.94699999999999995,"15min":0.55900000000000005},"min":{"1min":0.34300000000000003,"5min":0.217,"15min":0.217},"max":{"1min":6.3719999999999999,"5min":6.3719999999999999,"15min":6.3719999999999999},"last":3.399}]},{"osd":3,"last update":"Wed Sep 11 08:10:56 2024","interfaces":[{"interface":"back","average":{"1min":3.0339999999999998,"5min":0.94599999999999995,"15min":0.60799999999999998},"min":{"1min":0.38,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":7.9699999999999998,"5min":7.9699999999999998,"15min":7.9699999999999998},"last":3.379},{"interface":"front","average":{"1min":1.889,"5min":0.69799999999999995,"15min":0.51700000000000002},"min":{"1min":0.312,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":6.923,"5min":6.923,"15min":6.923},"last":2.8599999999999999}]},{"osd":4,"last update":"Wed Sep 11 08:11:17 2024","interfaces":[{"interface":"back","average":{"1min":1.7150000000000001,"5min":0.78700000000000003,"15min":0.625},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":6.726,"5min":6.726,"15min":6.726},"last":2.891},{"interface":"front","average":{"1min":1.76,"5min":0.78800000000000003,"15min":0.64800000000000002},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":5.6379999999999999,"5min":5.6379999999999999,"15min":5.6379999999999999},"last":3.3559999999999999}]},{"osd":5,"last update":"Wed Sep 11 08:11:45 2024","interfaces":[{"interface":"back","average":{"1min":1.841,"5min":0.93999999999999995,"15min":0.70199999999999996},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":6.1799999999999997,"5min":6.1799999999999997,"15min":6.1799999999999997},"last":3.3290000000000002},{"interface":"front","average":{"1min":1.4850000000000001,"5min":0.92300000000000004,"15min":0.67500000000000004},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":6.3949999999999996,"5min":6.7939999999999996,"15min":6.7939999999999996},"last":2.9380000000000002}]},{"osd":6,"last update":"Wed Sep 11 08:11:01 2024","interfaces":[{"interface":"back","average":{"1min":1.732,"5min":0.80200000000000005,"15min":0.68200000000000005},"min":{"1min":0.35799999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":6.8330000000000002,"5min":6.8330000000000002,"15min":6.8330000000000002},"last":2.9119999999999999},{"interface":"front","average":{"1min":2.0289999999999999,"5min":0.89400000000000002,"15min":0.66800000000000004},"min":{"1min":0.46700000000000003,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":6.1520000000000001,"5min":6.1520000000000001,"15min":6.1520000000000001},"last":1.7310000000000001}]},{"osd":7,"last update":"Wed Sep 11 08:11:24 2024","interfaces":[{"interface":"back","average":{"1min":1.5960000000000001,"5min":0.89000000000000001,"15min":0.70699999999999996},"min":{"1min":0.42199999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":6.2839999999999998,"5min":6.8650000000000002,"15min":6.8650000000000002},"last":3.9969999999999999},{"interface":"front","average":{"1min":1.8240000000000001,"5min":0.93000000000000005,"15min":0.69399999999999995},"min":{"1min":0.43099999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":6.1719999999999997,"5min":6.1719999999999997,"15min":6.1719999999999997},"last":1.758}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3231846400,"data_stored":3231842328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2412253184,"data_stored":2412249112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2430468096,"data_stored":2430468096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3640983552,"data_stored":3640983552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2033516544,"data_stored":2033516544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1626800128,"data_stored":1626800128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2020806656,"data_stored":2020802584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2028208128,"data_stored":2028208128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T08:11:59.859 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-11T08:11:59.859 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T08:11:59.860 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_0 2024-09-11T08:11:59.860 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T08:11:59.920 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:59 smithi150 ceph-mon[39612]: from='client.24775 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:11:59.920 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:59 smithi150 ceph-mon[39612]: pgmap v176: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 934 B/s rd, 0 B/s wr, 1.13k op/s 2024-09-11T08:11:59.920 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:59 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:11:59.920 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:59 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:11:59.920 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:59 smithi150 ceph-mon[39612]: osdmap e131: 8 total, 8 up, 8 in 2024-09-11T08:11:59.920 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:59 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/282106735' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:11:59.921 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:11:59 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:11:59.921 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:59 smithi150 ceph-mon[29276]: from='client.24775 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:11:59.921 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:59 smithi150 ceph-mon[29276]: pgmap v176: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 934 B/s rd, 0 B/s wr, 1.13k op/s 2024-09-11T08:11:59.921 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:59 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:11:59.921 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:59 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:11:59.921 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:59 smithi150 ceph-mon[29276]: osdmap e131: 8 total, 8 up, 8 in 2024-09-11T08:11:59.921 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:59 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/282106735' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:11:59.921 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:11:59 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:11:59.937 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:11:59.961 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:11:59.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:11:59.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:12:00.064 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:12:00.089 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-09-11T08:12:00.089 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:12:00.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:00.117 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:00.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:12:00.219 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:00.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:00.264 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:00.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:00.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:59 smithi181 ceph-mon[32701]: from='client.24775 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:12:00.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:59 smithi181 ceph-mon[32701]: pgmap v176: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 934 B/s rd, 0 B/s wr, 1.13k op/s 2024-09-11T08:12:00.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:59 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:12:00.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:59 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:12:00.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:59 smithi181 ceph-mon[32701]: osdmap e131: 8 total, 8 up, 8 in 2024-09-11T08:12:00.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:59 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/282106735' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:12:00.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:11:59 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:12:00.364 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:12:00.391 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:00.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:00.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:12:00.493 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:00.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:00.531 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:00.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:00.632 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:00.668 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:00.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:00.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:00.769 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:00.794 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:00.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:00.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:00.896 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:00.923 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:00.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:00.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:00.955 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:00 smithi150 ceph-mon[29276]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:12:00.955 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:00 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:12:00.955 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:00 smithi150 ceph-mon[29276]: osdmap e132: 8 total, 8 up, 8 in 2024-09-11T08:12:00.956 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:00 smithi150 ceph-mon[39612]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:12:00.957 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:00 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:12:00.957 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:00 smithi150 ceph-mon[39612]: osdmap e132: 8 total, 8 up, 8 in 2024-09-11T08:12:00.993 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-11T08:12:01.026 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:01.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:01.062 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:01.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:01.163 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:01.198 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:01.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:01.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:01.299 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:12:01.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:00 smithi181 ceph-mon[32701]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:12:01.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:00 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:12:01.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:00 smithi181 ceph-mon[32701]: osdmap e132: 8 total, 8 up, 8 in 2024-09-11T08:12:01.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:01.333 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:01.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:12:01.336 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:12:01.434 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:01.476 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:01.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:01.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:01.578 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:01.612 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:01.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:01.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:01.714 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:01.754 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:01.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:01.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:01.856 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:01.885 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:01.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:01.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:01.987 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:12:02.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:02.014 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:02.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:12:02.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:01 smithi181 ceph-mon[32701]: pgmap v179: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 994 op/s 2024-09-11T08:12:02.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:01 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:12:02.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:01 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:12:02.115 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:02.148 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:02.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:02.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:02.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:01 smithi150 ceph-mon[29276]: pgmap v179: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 994 op/s 2024-09-11T08:12:02.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:01 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:12:02.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:01 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:12:02.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:01 smithi150 ceph-mon[39612]: pgmap v179: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 994 op/s 2024-09-11T08:12:02.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:01 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:12:02.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:01 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:12:02.250 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:12:02.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:02.294 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:02.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:12:02.395 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:12:02.423 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:02.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:02.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:12:02.525 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:02.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:02.555 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:02.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:02.656 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:12:02.690 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:02.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:02.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:12:02.792 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:02.817 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:02.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:02.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:02.918 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:02.944 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:02.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:02.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:03.045 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:03.084 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:03.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:03.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:03.186 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:12:03.213 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:03.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:03.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:12:03.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:02 smithi150 ceph-mon[29276]: osdmap e133: 8 total, 8 up, 8 in 2024-09-11T08:12:03.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:02 smithi150 ceph-mon[39612]: osdmap e133: 8 total, 8 up, 8 in 2024-09-11T08:12:03.315 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:12:03.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:02 smithi181 ceph-mon[32701]: osdmap e133: 8 total, 8 up, 8 in 2024-09-11T08:12:03.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:03.343 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:03.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:12:03.444 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:12:03.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:03.478 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:03.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:12:03.563 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:12:03.574 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T08:12:03.579 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:03.636 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:03.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:03.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:03.738 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:12:03.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:03.769 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:03.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:12:03.870 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:03.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:03.938 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:03.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:03.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:03 smithi150 ceph-mon[29276]: pgmap v181: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.13k op/s 2024-09-11T08:12:03.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:03 smithi150 ceph-mon[39612]: pgmap v181: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.13k op/s 2024-09-11T08:12:04.039 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:12:04.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:03 smithi181 ceph-mon[32701]: pgmap v181: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.13k op/s 2024-09-11T08:12:04.045 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-11T08:12:04.045 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:12:04.083 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:04.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:04.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:12:04.185 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:12:04.214 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:04.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:04.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:12:04.291 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:12:04.291 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:12:04.316 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:12:04.327 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:12:04.327 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:12:04.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:04.352 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:04.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:12:04.367 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:12:04.367 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:12:04.401 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:12:04.402 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:12:04.437 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:12:04.437 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:12:04.454 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:12:04.473 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:12:04.473 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:12:04.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:04.492 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:04.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:12:04.502 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:12:04.502 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:12:04.543 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:12:04.543 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:12:04.593 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:04.621 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:04.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:04.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:04.723 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:04.759 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:04.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:04.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:04.862 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:04.889 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:04.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:04.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:04.971 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:04 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2014511078' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:12:04.991 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:12:05.008 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-11T08:12:05.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:05.028 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:05.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:12:05.130 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:05.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:05.170 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:05.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:05.229 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:04 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2014511078' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:12:05.229 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:04 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2014511078' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:12:05.271 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:05.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:05.315 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:05.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:05.324 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:12:05 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:12:05] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:12:05.397 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:12:05.416 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:05.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:05.466 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:05.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:05.568 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:05.601 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:05.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:05.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:05.702 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:05.741 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:05.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:05.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:05.843 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:12:05.872 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:05.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:05.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:12:05.974 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:05.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:05 smithi150 ceph-mon[29276]: pgmap v182: 148 pgs: 148 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 878 op/s 2024-09-11T08:12:05.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:05 smithi150 ceph-mon[39612]: pgmap v182: 148 pgs: 148 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 878 op/s 2024-09-11T08:12:06.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:06.006 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:06.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:06.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:05 smithi181 ceph-mon[32701]: pgmap v182: 148 pgs: 148 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 878 op/s 2024-09-11T08:12:06.107 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:06.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:06.170 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:06.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:06.271 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:12:06.297 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:06.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:06.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:12:06.399 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:12:06.447 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:06.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:06.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:12:06.549 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:12:06.577 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:06.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:06.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:12:06.679 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:06.712 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:12:06.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:06.722 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:06.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:06.823 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:06.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:06.879 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:06.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:06.980 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:06.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:06 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:12:06.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:06 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:12:06.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:06 smithi150 ceph-mon[29276]: osdmap e134: 8 total, 8 up, 8 in 2024-09-11T08:12:06.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:06 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:12:06.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:06 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:12:06.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:06 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:12:06.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:06 smithi150 ceph-mon[39612]: osdmap e134: 8 total, 8 up, 8 in 2024-09-11T08:12:06.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:06 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:12:07.022 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:07.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:07.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:07.055 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:06 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:12:07.055 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:06 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:12:07.055 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:06 smithi181 ceph-mon[32701]: osdmap e134: 8 total, 8 up, 8 in 2024-09-11T08:12:07.055 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:06 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:12:07.062 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:12:07.065 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-09-11T08:12:07.127 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:07.161 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:07.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:07.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:07.263 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:07.296 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:07.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:07.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:07.398 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:07.429 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:07.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:07.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:07.504 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-11T08:12:07.504 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-11T08:12:07.504 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-09-11T08:12:07.504 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-09-11T08:12:07.504 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-11T08:12:07.531 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:12:07.564 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:07.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:07.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:12:07.670 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:12:07.709 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:07.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:07.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:12:07.811 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:07.849 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:07.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:07.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:07.951 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:12:07.978 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:07.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:07.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:12:08.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:07 smithi181 ceph-mon[32701]: pgmap v184: 148 pgs: 148 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 739 B/s rd, 0 B/s wr, 761 op/s 2024-09-11T08:12:08.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:07 smithi181 ceph-mon[32701]: from='client.14928 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:12:08.074 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:12:07 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:12:08.080 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:08.116 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:08.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:08.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:08.151 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T08:12:08.204 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:07 smithi150 ceph-mon[29276]: pgmap v184: 148 pgs: 148 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 739 B/s rd, 0 B/s wr, 761 op/s 2024-09-11T08:12:08.205 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:07 smithi150 ceph-mon[29276]: from='client.14928 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:12:08.205 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:07 smithi150 ceph-mon[39612]: pgmap v184: 148 pgs: 148 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 739 B/s rd, 0 B/s wr, 761 op/s 2024-09-11T08:12:08.205 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:07 smithi150 ceph-mon[39612]: from='client.14928 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:12:08.218 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:12:08.245 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:08.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:08.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:12:08.327 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-11T08:12:08.347 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:08.399 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:08.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:08.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:08.501 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:08.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:08.528 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:08.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:08.629 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:08.666 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:08.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:08.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:08.680 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:12:08.739 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:12:08 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:12:08] "GET /metrics HTTP/1.1" 200 223810 "" "Prometheus/2.43.0" 2024-09-11T08:12:08.768 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:12:08.796 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:08.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:08.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:12:08.898 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:08.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:08.930 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:08.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:09.031 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:09.048 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":184,"stamp":"2024-09-11T08:12:06.759879+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5814884596,"num_objects":88916,"num_object_clones":0,"num_object_copies":266748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88916,"num_whiteouts":0,"num_read":1120,"num_read_kb":977,"num_write":160701,"num_write_kb":7965704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":590698,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90549,"ondisk_log_size":90549,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":19577812,"kb_used_data":17139216,"kb_used_omap":13,"kb_used_meta":2436978,"kb_avail":730383404,"statfs":{"total":767960285184,"available":747912605696,"internally_reserved":0,"allocated":17550557184,"data_stored":17547687756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2495466176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-345047040,"num_objects":-5265,"num_object_clones":0,"num_object_copies":-15795,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5265,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":5265,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.922677"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":103,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.851874+0000","last_change":"2024-09-11T08:10:19.111028+0000","last_active":"2024-09-11T08:11:59.851874+0000","last_peered":"2024-09-11T08:11:59.851874+0000","last_clean":"2024-09-11T08:11:59.851874+0000","last_became_active":"2024-09-11T08:07:30.044976+0000","last_became_peered":"2024-09-11T08:07:30.044976+0000","last_unstale":"2024-09-11T08:11:59.851874+0000","last_undegraded":"2024-09-11T08:11:59.851874+0000","last_fullsized":"2024-09-11T08:11:59.851874+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:19.110647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:10:19.110647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.017276071,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":107,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.836291+0000","last_change":"2024-09-11T08:09:24.630803+0000","last_active":"2024-09-11T08:11:59.836291+0000","last_peered":"2024-09-11T08:11:59.836291+0000","last_clean":"2024-09-11T08:11:59.836291+0000","last_became_active":"2024-09-11T08:07:32.048048+0000","last_became_peered":"2024-09-11T08:07:32.048048+0000","last_unstale":"2024-09-11T08:11:59.836291+0000","last_undegraded":"2024-09-11T08:11:59.836291+0000","last_fullsized":"2024-09-11T08:11:59.836291+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:09:24.630735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:09:24.630735+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020206859999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":109,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.841971+0000","last_change":"2024-09-11T08:11:40.568270+0000","last_active":"2024-09-11T08:11:59.841971+0000","last_peered":"2024-09-11T08:11:59.841971+0000","last_clean":"2024-09-11T08:11:59.841971+0000","last_became_active":"2024-09-11T08:07:34.058828+0000","last_became_peered":"2024-09-11T08:07:34.058828+0000","last_unstale":"2024-09-11T08:11:59.841971+0000","last_undegraded":"2024-09-11T08:11:59.841971+0000","last_fullsized":"2024-09-11T08:11:59.841971+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:40.567720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:11:40.567720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:13:05.932274+0000","scrub_duration":0.013465969,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":97,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.836350+0000","last_change":"2024-09-11T08:11:16.061718+0000","last_active":"2024-09-11T08:11:59.836350+0000","last_peered":"2024-09-11T08:11:59.836350+0000","last_clean":"2024-09-11T08:11:59.836350+0000","last_became_active":"2024-09-11T08:07:36.074789+0000","last_became_peered":"2024-09-11T08:07:36.074789+0000","last_unstale":"2024-09-11T08:11:59.836350+0000","last_undegraded":"2024-09-11T08:11:59.836350+0000","last_fullsized":"2024-09-11T08:11:59.836350+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:16.061669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:02.470649+0000","last_clean_scrub_stamp":"2024-09-11T08:11:16.061669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:25.199054+0000","scrub_duration":0.0051287930000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":97,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.853755+0000","last_change":"2024-09-11T08:11:34.567724+0000","last_active":"2024-09-11T08:11:59.853755+0000","last_peered":"2024-09-11T08:11:59.853755+0000","last_clean":"2024-09-11T08:11:59.853755+0000","last_became_active":"2024-09-11T08:07:36.633423+0000","last_became_peered":"2024-09-11T08:07:36.633423+0000","last_unstale":"2024-09-11T08:11:59.853755+0000","last_undegraded":"2024-09-11T08:11:59.853755+0000","last_fullsized":"2024-09-11T08:11:59.853755+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:34.567654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:11:34.567654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:13:02.653394+0000","scrub_duration":0.0064880140000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":93,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.853497+0000","last_change":"2024-09-11T08:09:35.348975+0000","last_active":"2024-09-11T08:11:59.853497+0000","last_peered":"2024-09-11T08:11:59.853497+0000","last_clean":"2024-09-11T08:11:59.853497+0000","last_became_active":"2024-09-11T08:07:30.045579+0000","last_became_peered":"2024-09-11T08:07:30.045579+0000","last_unstale":"2024-09-11T08:11:59.853497+0000","last_undegraded":"2024-09-11T08:11:59.853497+0000","last_fullsized":"2024-09-11T08:11:59.853497+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:35.348912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:09:35.348912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001468128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":126,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.845566+0000","last_change":"2024-09-11T08:11:19.427044+0000","last_active":"2024-09-11T08:11:59.845566+0000","last_peered":"2024-09-11T08:11:59.845566+0000","last_clean":"2024-09-11T08:11:59.845566+0000","last_became_active":"2024-09-11T08:07:32.045848+0000","last_became_peered":"2024-09-11T08:07:32.045848+0000","last_unstale":"2024-09-11T08:11:59.845566+0000","last_undegraded":"2024-09-11T08:11:59.845566+0000","last_fullsized":"2024-09-11T08:11:59.845566+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-11T08:11:19.426563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:11:19.426563+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:34.923254+0000","scrub_duration":0.021858643000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":89,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.836199+0000","last_change":"2024-09-11T08:09:31.638930+0000","last_active":"2024-09-11T08:11:59.836199+0000","last_peered":"2024-09-11T08:11:59.836199+0000","last_clean":"2024-09-11T08:11:59.836199+0000","last_became_active":"2024-09-11T08:07:34.052243+0000","last_became_peered":"2024-09-11T08:07:34.052243+0000","last_unstale":"2024-09-11T08:11:59.836199+0000","last_undegraded":"2024-09-11T08:11:59.836199+0000","last_fullsized":"2024-09-11T08:11:59.836199+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:31.638867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:09:31.638867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014270719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":102,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.849520+0000","last_change":"2024-09-11T08:10:26.020726+0000","last_active":"2024-09-11T08:11:59.849520+0000","last_peered":"2024-09-11T08:11:59.849520+0000","last_clean":"2024-09-11T08:11:59.849520+0000","last_became_active":"2024-09-11T08:07:30.043950+0000","last_became_peered":"2024-09-11T08:07:30.043950+0000","last_unstale":"2024-09-11T08:11:59.849520+0000","last_undegraded":"2024-09-11T08:11:59.849520+0000","last_fullsized":"2024-09-11T08:11:59.849520+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:26.020674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:54.878409+0000","last_clean_scrub_stamp":"2024-09-11T08:10:26.020674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001074111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":117,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.836502+0000","last_change":"2024-09-11T08:10:42.141214+0000","last_active":"2024-09-11T08:11:59.836502+0000","last_peered":"2024-09-11T08:11:59.836502+0000","last_clean":"2024-09-11T08:11:59.836502+0000","last_became_active":"2024-09-11T08:07:32.044626+0000","last_became_peered":"2024-09-11T08:07:32.044626+0000","last_unstale":"2024-09-11T08:11:59.836502+0000","last_undegraded":"2024-09-11T08:11:59.836502+0000","last_fullsized":"2024-09-11T08:11:59.836502+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:10:42.141115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:10:42.141115+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0076042940000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":98,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.852803+0000","last_change":"2024-09-11T08:10:44.820612+0000","last_active":"2024-09-11T08:11:59.852803+0000","last_peered":"2024-09-11T08:11:59.852803+0000","last_clean":"2024-09-11T08:11:59.852803+0000","last_became_active":"2024-09-11T08:07:34.058689+0000","last_became_peered":"2024-09-11T08:07:34.058689+0000","last_unstale":"2024-09-11T08:11:59.852803+0000","last_undegraded":"2024-09-11T08:11:59.852803+0000","last_fullsized":"2024-09-11T08:11:59.852803+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:44.820382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:10:44.820382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:09.539844+0000","scrub_duration":0.0067487449999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":97,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.846335+0000","last_change":"2024-09-11T08:10:16.101381+0000","last_active":"2024-09-11T08:11:59.846335+0000","last_peered":"2024-09-11T08:11:59.846335+0000","last_clean":"2024-09-11T08:11:59.846335+0000","last_became_active":"2024-09-11T08:07:36.076824+0000","last_became_peered":"2024-09-11T08:07:36.076824+0000","last_unstale":"2024-09-11T08:11:59.846335+0000","last_undegraded":"2024-09-11T08:11:59.846335+0000","last_fullsized":"2024-09-11T08:11:59.846335+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:16.101299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:10:16.101299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0062695850000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":102,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.849217+0000","last_change":"2024-09-11T08:10:15.984728+0000","last_active":"2024-09-11T08:11:59.849217+0000","last_peered":"2024-09-11T08:11:59.849217+0000","last_clean":"2024-09-11T08:11:59.849217+0000","last_became_active":"2024-09-11T08:07:30.043277+0000","last_became_peered":"2024-09-11T08:07:30.043277+0000","last_unstale":"2024-09-11T08:11:59.849217+0000","last_undegraded":"2024-09-11T08:11:59.849217+0000","last_fullsized":"2024-09-11T08:11:59.849217+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:15.984636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:10:15.984636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0055326200000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":101,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.836351+0000","last_change":"2024-09-11T08:09:08.668983+0000","last_active":"2024-09-11T08:11:59.836351+0000","last_peered":"2024-09-11T08:11:59.836351+0000","last_clean":"2024-09-11T08:11:59.836351+0000","last_became_active":"2024-09-11T08:07:32.048661+0000","last_became_peered":"2024-09-11T08:07:32.048661+0000","last_unstale":"2024-09-11T08:11:59.836351+0000","last_undegraded":"2024-09-11T08:11:59.836351+0000","last_fullsized":"2024-09-11T08:11:59.836351+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-11T08:09:08.668925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:09:08.668925+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001704496,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":99,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.850899+0000","last_change":"2024-09-11T08:11:20.459210+0000","last_active":"2024-09-11T08:11:59.850899+0000","last_peered":"2024-09-11T08:11:59.850899+0000","last_clean":"2024-09-11T08:11:59.850899+0000","last_became_active":"2024-09-11T08:07:34.060871+0000","last_became_peered":"2024-09-11T08:07:34.060871+0000","last_unstale":"2024-09-11T08:11:59.850899+0000","last_undegraded":"2024-09-11T08:11:59.850899+0000","last_fullsized":"2024-09-11T08:11:59.850899+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:20.459128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:11:20.459128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:44.755722+0000","scrub_duration":0.0035582639999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":96,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.849052+0000","last_change":"2024-09-11T08:10:37.891841+0000","last_active":"2024-09-11T08:11:59.849052+0000","last_peered":"2024-09-11T08:11:59.849052+0000","last_clean":"2024-09-11T08:11:59.849052+0000","last_became_active":"2024-09-11T08:07:36.077602+0000","last_became_peered":"2024-09-11T08:07:36.077602+0000","last_unstale":"2024-09-11T08:11:59.849052+0000","last_undegraded":"2024-09-11T08:11:59.849052+0000","last_fullsized":"2024-09-11T08:11:59.849052+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:37.891755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:04.782577+0000","last_clean_scrub_stamp":"2024-09-11T08:10:37.891755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0069411969999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":97,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.837224+0000","last_change":"2024-09-11T08:11:38.063359+0000","last_active":"2024-09-11T08:11:59.837224+0000","last_peered":"2024-09-11T08:11:59.837224+0000","last_clean":"2024-09-11T08:11:59.837224+0000","last_became_active":"2024-09-11T08:07:36.633338+0000","last_became_peered":"2024-09-11T08:07:36.633338+0000","last_unstale":"2024-09-11T08:11:59.837224+0000","last_undegraded":"2024-09-11T08:11:59.837224+0000","last_fullsized":"2024-09-11T08:11:59.837224+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:38.063310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:11:38.063310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:44.770226+0000","scrub_duration":0.0013449110000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":93,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.847393+0000","last_change":"2024-09-11T08:09:11.966738+0000","last_active":"2024-09-11T08:11:59.847393+0000","last_peered":"2024-09-11T08:11:59.847393+0000","last_clean":"2024-09-11T08:11:59.847393+0000","last_became_active":"2024-09-11T08:07:30.045739+0000","last_became_peered":"2024-09-11T08:07:30.045739+0000","last_unstale":"2024-09-11T08:11:59.847393+0000","last_undegraded":"2024-09-11T08:11:59.847393+0000","last_fullsized":"2024-09-11T08:11:59.847393+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:11.966683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:09:11.966683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015807659999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"107'22","reported_seq":132,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.845737+0000","last_change":"2024-09-11T08:10:30.941599+0000","last_active":"2024-09-11T08:11:59.845737+0000","last_peered":"2024-09-11T08:11:59.845737+0000","last_clean":"2024-09-11T08:11:59.845737+0000","last_became_active":"2024-09-11T08:07:32.046363+0000","last_became_peered":"2024-09-11T08:07:32.046363+0000","last_unstale":"2024-09-11T08:11:59.845737+0000","last_undegraded":"2024-09-11T08:11:59.845737+0000","last_fullsized":"2024-09-11T08:11:59.845737+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'22","last_scrub_stamp":"2024-09-11T08:10:30.941505+0000","last_deep_scrub":"107'22","last_deep_scrub_stamp":"2024-09-11T08:10:30.941505+0000","last_clean_scrub_stamp":"2024-09-11T08:10:30.941505+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.02240839,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":99,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.839630+0000","last_change":"2024-09-11T08:10:31.418953+0000","last_active":"2024-09-11T08:11:59.839630+0000","last_peered":"2024-09-11T08:11:59.839630+0000","last_clean":"2024-09-11T08:11:59.839630+0000","last_became_active":"2024-09-11T08:07:34.055410+0000","last_became_peered":"2024-09-11T08:07:34.055410+0000","last_unstale":"2024-09-11T08:11:59.839630+0000","last_undegraded":"2024-09-11T08:11:59.839630+0000","last_fullsized":"2024-09-11T08:11:59.839630+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:31.418873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:10:31.418873+0000","last_clean_scrub_stamp":"2024-09-11T08:10:31.418873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011564522000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":87,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.852293+0000","last_change":"2024-09-11T08:10:31.188608+0000","last_active":"2024-09-11T08:11:59.852293+0000","last_peered":"2024-09-11T08:11:59.852293+0000","last_clean":"2024-09-11T08:11:59.852293+0000","last_became_active":"2024-09-11T08:07:36.633258+0000","last_became_peered":"2024-09-11T08:07:36.633258+0000","last_unstale":"2024-09-11T08:11:59.852293+0000","last_undegraded":"2024-09-11T08:11:59.852293+0000","last_fullsized":"2024-09-11T08:11:59.852293+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:31.188518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:10:31.188518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013170371,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":103,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.836665+0000","last_change":"2024-09-11T08:11:17.089124+0000","last_active":"2024-09-11T08:11:59.836665+0000","last_peered":"2024-09-11T08:11:59.836665+0000","last_clean":"2024-09-11T08:11:59.836665+0000","last_became_active":"2024-09-11T08:07:30.046284+0000","last_became_peered":"2024-09-11T08:07:30.046284+0000","last_unstale":"2024-09-11T08:11:59.836665+0000","last_undegraded":"2024-09-11T08:11:59.836665+0000","last_fullsized":"2024-09-11T08:11:59.836665+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:17.089046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:11:17.089046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:21.125842+0000","scrub_duration":0.0042934460000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":146,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.849825+0000","last_change":"2024-09-11T08:11:32.559837+0000","last_active":"2024-09-11T08:11:59.849825+0000","last_peered":"2024-09-11T08:11:59.849825+0000","last_clean":"2024-09-11T08:11:59.849825+0000","last_became_active":"2024-09-11T08:09:07.878432+0000","last_became_peered":"2024-09-11T08:09:07.878432+0000","last_unstale":"2024-09-11T08:11:59.849825+0000","last_undegraded":"2024-09-11T08:11:59.849825+0000","last_fullsized":"2024-09-11T08:11:59.849825+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-11T08:11:32.559751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:11:32.559751+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:48.508018+0000","scrub_duration":0.0059216119999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":99,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.836900+0000","last_change":"2024-09-11T08:10:40.110540+0000","last_active":"2024-09-11T08:11:59.836900+0000","last_peered":"2024-09-11T08:11:59.836900+0000","last_clean":"2024-09-11T08:11:59.836900+0000","last_became_active":"2024-09-11T08:07:34.056528+0000","last_became_peered":"2024-09-11T08:07:34.056528+0000","last_unstale":"2024-09-11T08:11:59.836900+0000","last_undegraded":"2024-09-11T08:11:59.836900+0000","last_fullsized":"2024-09-11T08:11:59.836900+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:40.110490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:10:40.110490+0000","last_clean_scrub_stamp":"2024-09-11T08:10:40.110490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00164006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":98,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.851181+0000","last_change":"2024-09-11T08:10:41.305921+0000","last_active":"2024-09-11T08:11:59.851181+0000","last_peered":"2024-09-11T08:11:59.851181+0000","last_clean":"2024-09-11T08:11:59.851181+0000","last_became_active":"2024-09-11T08:07:36.072347+0000","last_became_peered":"2024-09-11T08:07:36.072347+0000","last_unstale":"2024-09-11T08:11:59.851181+0000","last_undegraded":"2024-09-11T08:11:59.851181+0000","last_fullsized":"2024-09-11T08:11:59.851181+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-11T08:10:41.305866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:10:41.305866+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045172119999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":103,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.842078+0000","last_change":"2024-09-11T08:10:35.345635+0000","last_active":"2024-09-11T08:11:59.842078+0000","last_peered":"2024-09-11T08:11:59.842078+0000","last_clean":"2024-09-11T08:11:59.842078+0000","last_became_active":"2024-09-11T08:07:30.045932+0000","last_became_peered":"2024-09-11T08:07:30.045932+0000","last_unstale":"2024-09-11T08:11:59.842078+0000","last_undegraded":"2024-09-11T08:11:59.842078+0000","last_fullsized":"2024-09-11T08:11:59.842078+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:35.345484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:10:35.345484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010438901,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":116,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.857077+0000","last_change":"2024-09-11T08:11:30.913167+0000","last_active":"2024-09-11T08:11:59.857077+0000","last_peered":"2024-09-11T08:11:59.857077+0000","last_clean":"2024-09-11T08:11:59.857077+0000","last_became_active":"2024-09-11T08:07:32.048022+0000","last_became_peered":"2024-09-11T08:07:32.048022+0000","last_unstale":"2024-09-11T08:11:59.857077+0000","last_undegraded":"2024-09-11T08:11:59.857077+0000","last_fullsized":"2024-09-11T08:11:59.857077+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-11T08:11:30.913083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:11:30.913083+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:56.240332+0000","scrub_duration":0.0071376130000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":173,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.841913+0000","last_change":"2024-09-11T08:11:28.666314+0000","last_active":"2024-09-11T08:11:59.841913+0000","last_peered":"2024-09-11T08:11:59.841913+0000","last_clean":"2024-09-11T08:11:59.841913+0000","last_became_active":"2024-09-11T08:07:34.054014+0000","last_became_peered":"2024-09-11T08:07:34.054014+0000","last_unstale":"2024-09-11T08:11:59.841913+0000","last_undegraded":"2024-09-11T08:11:59.841913+0000","last_fullsized":"2024-09-11T08:11:59.841913+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:11:28.666260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:11:28.666260+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:35.716682+0000","scrub_duration":0.001258885,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"133'9879","reported_seq":10245,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-11T08:12:02.681450+0000","last_change":"2024-09-11T08:11:28.767321+0000","last_active":"2024-09-11T08:12:02.681450+0000","last_peered":"2024-09-11T08:12:02.681450+0000","last_clean":"2024-09-11T08:12:02.681450+0000","last_became_active":"2024-09-11T08:09:45.414744+0000","last_became_peered":"2024-09-11T08:09:45.414744+0000","last_unstale":"2024-09-11T08:12:02.681450+0000","last_undegraded":"2024-09-11T08:12:02.681450+0000","last_fullsized":"2024-09-11T08:12:02.681450+0000","mapping_epoch":91,"log_start":"106'3600","ondisk_log_start":"106'3600","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'7721","last_scrub_stamp":"2024-09-11T08:11:28.767232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_clean_scrub_stamp":"2024-09-11T08:11:28.767232+0000","objects_scrubbed":7645,"log_size":6279,"log_dups_size":0,"ondisk_log_size":6279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:48.988510+0000","scrub_duration":2.190530367,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":359464960,"num_objects":5485,"num_object_clones":0,"num_object_copies":16455,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9879,"num_write_kb":491648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":126,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.840048+0000","last_change":"2024-09-11T08:10:23.428879+0000","last_active":"2024-09-11T08:11:59.840048+0000","last_peered":"2024-09-11T08:11:59.840048+0000","last_clean":"2024-09-11T08:11:59.840048+0000","last_became_active":"2024-09-11T08:07:32.054317+0000","last_became_peered":"2024-09-11T08:07:32.054317+0000","last_unstale":"2024-09-11T08:11:59.840048+0000","last_undegraded":"2024-09-11T08:11:59.840048+0000","last_fullsized":"2024-09-11T08:11:59.840048+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-11T08:10:23.428796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:10:23.428796+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014719733,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":93,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.852060+0000","last_change":"2024-09-11T08:10:16.250853+0000","last_active":"2024-09-11T08:11:59.852060+0000","last_peered":"2024-09-11T08:11:59.852060+0000","last_clean":"2024-09-11T08:11:59.852060+0000","last_became_active":"2024-09-11T08:07:30.044624+0000","last_became_peered":"2024-09-11T08:07:30.044624+0000","last_unstale":"2024-09-11T08:11:59.852060+0000","last_undegraded":"2024-09-11T08:11:59.852060+0000","last_fullsized":"2024-09-11T08:11:59.852060+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:16.247162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:10:16.247162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065916309999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":161,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.837301+0000","last_change":"2024-09-11T08:11:27.139877+0000","last_active":"2024-09-11T08:11:59.837301+0000","last_peered":"2024-09-11T08:11:59.837301+0000","last_clean":"2024-09-11T08:11:59.837301+0000","last_became_active":"2024-09-11T08:07:34.052981+0000","last_became_peered":"2024-09-11T08:07:34.052981+0000","last_unstale":"2024-09-11T08:11:59.837301+0000","last_undegraded":"2024-09-11T08:11:59.837301+0000","last_fullsized":"2024-09-11T08:11:59.837301+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:11:27.139796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:11:27.139796+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:48.327854+0000","scrub_duration":0.0080431200000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":106,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.849266+0000","last_change":"2024-09-11T08:11:39.747151+0000","last_active":"2024-09-11T08:11:59.849266+0000","last_peered":"2024-09-11T08:11:59.849266+0000","last_clean":"2024-09-11T08:11:59.849266+0000","last_became_active":"2024-09-11T08:07:36.073969+0000","last_became_peered":"2024-09-11T08:07:36.073969+0000","last_unstale":"2024-09-11T08:11:59.849266+0000","last_undegraded":"2024-09-11T08:11:59.849266+0000","last_fullsized":"2024-09-11T08:11:59.849266+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:39.747073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:11:39.747073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:42.671932+0000","scrub_duration":0.0080670670000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"133'10095","reported_seq":10155,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-11T08:12:02.840961+0000","last_change":"2024-09-11T08:11:42.436912+0000","last_active":"2024-09-11T08:12:02.840961+0000","last_peered":"2024-09-11T08:12:02.840961+0000","last_clean":"2024-09-11T08:12:02.840961+0000","last_became_active":"2024-09-11T08:09:45.412326+0000","last_became_peered":"2024-09-11T08:09:45.412326+0000","last_unstale":"2024-09-11T08:12:02.840961+0000","last_undegraded":"2024-09-11T08:12:02.840961+0000","last_fullsized":"2024-09-11T08:12:02.840961+0000","mapping_epoch":91,"log_start":"114'5400","ondisk_log_start":"114'5400","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_clean_scrub_stamp":"2024-09-11T08:09:44.385805+0000","objects_scrubbed":99,"log_size":4695,"log_dups_size":0,"ondisk_log_size":4695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364183552,"num_objects":5557,"num_object_clones":0,"num_object_copies":16671,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10095,"num_write_kb":500864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":133,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.851962+0000","last_change":"2024-09-11T08:10:35.187425+0000","last_active":"2024-09-11T08:11:59.851962+0000","last_peered":"2024-09-11T08:11:59.851962+0000","last_clean":"2024-09-11T08:11:59.851962+0000","last_became_active":"2024-09-11T08:07:32.045332+0000","last_became_peered":"2024-09-11T08:07:32.045332+0000","last_unstale":"2024-09-11T08:11:59.851962+0000","last_undegraded":"2024-09-11T08:11:59.851962+0000","last_fullsized":"2024-09-11T08:11:59.851962+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-11T08:10:35.187322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:10:35.187322+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:02.558401+0000","scrub_duration":0.016336040999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":93,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.853188+0000","last_change":"2024-09-11T08:10:24.286689+0000","last_active":"2024-09-11T08:11:59.853188+0000","last_peered":"2024-09-11T08:11:59.853188+0000","last_clean":"2024-09-11T08:11:59.853188+0000","last_became_active":"2024-09-11T08:07:30.047176+0000","last_became_peered":"2024-09-11T08:07:30.047176+0000","last_unstale":"2024-09-11T08:11:59.853188+0000","last_undegraded":"2024-09-11T08:11:59.853188+0000","last_fullsized":"2024-09-11T08:11:59.853188+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:24.286622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:10:24.286622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.022756537,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":99,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.836453+0000","last_change":"2024-09-11T08:11:26.181015+0000","last_active":"2024-09-11T08:11:59.836453+0000","last_peered":"2024-09-11T08:11:59.836453+0000","last_clean":"2024-09-11T08:11:59.836453+0000","last_became_active":"2024-09-11T08:07:34.055523+0000","last_became_peered":"2024-09-11T08:07:34.055523+0000","last_unstale":"2024-09-11T08:11:59.836453+0000","last_undegraded":"2024-09-11T08:11:59.836453+0000","last_fullsized":"2024-09-11T08:11:59.836453+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:26.180933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:11:26.180933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:31.314789+0000","scrub_duration":0.0022304540000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":87,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.836458+0000","last_change":"2024-09-11T08:09:21.618824+0000","last_active":"2024-09-11T08:11:59.836458+0000","last_peered":"2024-09-11T08:11:59.836458+0000","last_clean":"2024-09-11T08:11:59.836458+0000","last_became_active":"2024-09-11T08:07:36.069225+0000","last_became_peered":"2024-09-11T08:07:36.069225+0000","last_unstale":"2024-09-11T08:11:59.836458+0000","last_undegraded":"2024-09-11T08:11:59.836458+0000","last_fullsized":"2024-09-11T08:11:59.836458+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:21.618769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:09:21.618769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001637814,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"133'9928","reported_seq":10050,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-11T08:12:02.835024+0000","last_change":"2024-09-11T08:11:42.431777+0000","last_active":"2024-09-11T08:12:02.835024+0000","last_peered":"2024-09-11T08:12:02.835024+0000","last_clean":"2024-09-11T08:12:02.835024+0000","last_became_active":"2024-09-11T08:09:45.414656+0000","last_became_peered":"2024-09-11T08:09:45.414656+0000","last_unstale":"2024-09-11T08:12:02.835024+0000","last_undegraded":"2024-09-11T08:12:02.835024+0000","last_fullsized":"2024-09-11T08:12:02.835024+0000","mapping_epoch":91,"log_start":"114'5300","ondisk_log_start":"114'5300","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_clean_scrub_stamp":"2024-09-11T08:09:44.385805+0000","objects_scrubbed":1649,"log_size":4628,"log_dups_size":0,"ondisk_log_size":4628,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":354287616,"num_objects":5406,"num_object_clones":0,"num_object_copies":16218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9928,"num_write_kb":490688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":112,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.836642+0000","last_change":"2024-09-11T08:09:11.688215+0000","last_active":"2024-09-11T08:11:59.836642+0000","last_peered":"2024-09-11T08:11:59.836642+0000","last_clean":"2024-09-11T08:11:59.836642+0000","last_became_active":"2024-09-11T08:07:32.040069+0000","last_became_peered":"2024-09-11T08:07:32.040069+0000","last_unstale":"2024-09-11T08:11:59.836642+0000","last_undegraded":"2024-09-11T08:11:59.836642+0000","last_fullsized":"2024-09-11T08:11:59.836642+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-11T08:09:11.688151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:09:11.688151+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019655670000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":103,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.857274+0000","last_change":"2024-09-11T08:11:33.964196+0000","last_active":"2024-09-11T08:11:59.857274+0000","last_peered":"2024-09-11T08:11:59.857274+0000","last_clean":"2024-09-11T08:11:59.857274+0000","last_became_active":"2024-09-11T08:07:30.050484+0000","last_became_peered":"2024-09-11T08:07:30.050484+0000","last_unstale":"2024-09-11T08:11:59.857274+0000","last_undegraded":"2024-09-11T08:11:59.857274+0000","last_fullsized":"2024-09-11T08:11:59.857274+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:33.964053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:11:33.964053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:40.373468+0000","scrub_duration":0.003641151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"82'9","reported_seq":165,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.851826+0000","last_change":"2024-09-11T08:11:25.434358+0000","last_active":"2024-09-11T08:11:59.851826+0000","last_peered":"2024-09-11T08:11:59.851826+0000","last_clean":"2024-09-11T08:11:59.851826+0000","last_became_active":"2024-09-11T08:07:34.056143+0000","last_became_peered":"2024-09-11T08:07:34.056143+0000","last_unstale":"2024-09-11T08:11:59.851826+0000","last_undegraded":"2024-09-11T08:11:59.851826+0000","last_fullsized":"2024-09-11T08:11:59.851826+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-11T08:11:25.434210+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-09-11T08:08:53.421587+0000","last_clean_scrub_stamp":"2024-09-11T08:11:25.434210+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:46.551085+0000","scrub_duration":0.0043836509999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":97,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.851032+0000","last_change":"2024-09-11T08:10:29.251333+0000","last_active":"2024-09-11T08:11:59.851032+0000","last_peered":"2024-09-11T08:11:59.851032+0000","last_clean":"2024-09-11T08:11:59.851032+0000","last_became_active":"2024-09-11T08:07:36.633432+0000","last_became_peered":"2024-09-11T08:07:36.633432+0000","last_unstale":"2024-09-11T08:11:59.851032+0000","last_undegraded":"2024-09-11T08:11:59.851032+0000","last_fullsized":"2024-09-11T08:11:59.851032+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:29.251257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:10:29.251257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023785080000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"133'10085","reported_seq":10135,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-11T08:12:02.691732+0000","last_change":"2024-09-11T08:09:45.410430+0000","last_active":"2024-09-11T08:12:02.691732+0000","last_peered":"2024-09-11T08:12:02.691732+0000","last_clean":"2024-09-11T08:12:02.691732+0000","last_became_active":"2024-09-11T08:09:45.410129+0000","last_became_peered":"2024-09-11T08:09:45.410129+0000","last_unstale":"2024-09-11T08:12:02.691732+0000","last_undegraded":"2024-09-11T08:12:02.691732+0000","last_fullsized":"2024-09-11T08:12:02.691732+0000","mapping_epoch":91,"log_start":"114'5300","ondisk_log_start":"114'5300","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_clean_scrub_stamp":"2024-09-11T08:09:44.385805+0000","objects_scrubbed":0,"log_size":4785,"log_dups_size":0,"ondisk_log_size":4785,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364969984,"num_objects":5569,"num_object_clones":0,"num_object_copies":16707,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10085,"num_write_kb":500928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":116,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.842579+0000","last_change":"2024-09-11T08:11:22.581262+0000","last_active":"2024-09-11T08:11:59.842579+0000","last_peered":"2024-09-11T08:11:59.842579+0000","last_clean":"2024-09-11T08:11:59.842579+0000","last_became_active":"2024-09-11T08:07:32.054294+0000","last_became_peered":"2024-09-11T08:07:32.054294+0000","last_unstale":"2024-09-11T08:11:59.842579+0000","last_undegraded":"2024-09-11T08:11:59.842579+0000","last_fullsized":"2024-09-11T08:11:59.842579+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-11T08:11:22.581035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:11:22.581035+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:43.316970+0000","scrub_duration":0.0076992600000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":102,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.849617+0000","last_change":"2024-09-11T08:10:35.913323+0000","last_active":"2024-09-11T08:11:59.849617+0000","last_peered":"2024-09-11T08:11:59.849617+0000","last_clean":"2024-09-11T08:11:59.849617+0000","last_became_active":"2024-09-11T08:07:30.047497+0000","last_became_peered":"2024-09-11T08:07:30.047497+0000","last_unstale":"2024-09-11T08:11:59.849617+0000","last_undegraded":"2024-09-11T08:11:59.849617+0000","last_fullsized":"2024-09-11T08:11:59.849617+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:35.913239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:10:35.913239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.007393714,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":161,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.847043+0000","last_change":"2024-09-11T08:11:38.925253+0000","last_active":"2024-09-11T08:11:59.847043+0000","last_peered":"2024-09-11T08:11:59.847043+0000","last_clean":"2024-09-11T08:11:59.847043+0000","last_became_active":"2024-09-11T08:07:34.060660+0000","last_became_peered":"2024-09-11T08:07:34.060660+0000","last_unstale":"2024-09-11T08:11:59.847043+0000","last_undegraded":"2024-09-11T08:11:59.847043+0000","last_fullsized":"2024-09-11T08:11:59.847043+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:11:38.925167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:11:38.925167+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:44.287481+0000","scrub_duration":0.0080128060000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"66'1","reported_seq":88,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.846456+0000","last_change":"2024-09-11T08:09:25.767659+0000","last_active":"2024-09-11T08:11:59.846456+0000","last_peered":"2024-09-11T08:11:59.846456+0000","last_clean":"2024-09-11T08:11:59.846456+0000","last_became_active":"2024-09-11T08:07:36.072593+0000","last_became_peered":"2024-09-11T08:07:36.072593+0000","last_unstale":"2024-09-11T08:11:59.846456+0000","last_undegraded":"2024-09-11T08:11:59.846456+0000","last_fullsized":"2024-09-11T08:11:59.846456+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-11T08:09:25.767603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:09:25.767603+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001492487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"133'9719","reported_seq":10067,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-11T08:12:02.702912+0000","last_change":"2024-09-11T08:11:22.612314+0000","last_active":"2024-09-11T08:12:02.702912+0000","last_peered":"2024-09-11T08:12:02.702912+0000","last_clean":"2024-09-11T08:12:02.702912+0000","last_became_active":"2024-09-11T08:09:45.415564+0000","last_became_peered":"2024-09-11T08:09:45.415564+0000","last_unstale":"2024-09-11T08:12:02.702912+0000","last_undegraded":"2024-09-11T08:12:02.702912+0000","last_fullsized":"2024-09-11T08:12:02.702912+0000","mapping_epoch":91,"log_start":"107'3900","ondisk_log_start":"107'3900","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'7301","last_scrub_stamp":"2024-09-11T08:11:22.612230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_clean_scrub_stamp":"2024-09-11T08:11:22.612230+0000","objects_scrubbed":7217,"log_size":5819,"log_dups_size":0,"ondisk_log_size":5819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:51.523019+0000","scrub_duration":2.8618551820000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364445696,"num_objects":5561,"num_object_clones":0,"num_object_copies":16683,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9719,"num_write_kb":488960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":137,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.847192+0000","last_change":"2024-09-11T08:11:19.982267+0000","last_active":"2024-09-11T08:11:59.847192+0000","last_peered":"2024-09-11T08:11:59.847192+0000","last_clean":"2024-09-11T08:11:59.847192+0000","last_became_active":"2024-09-11T08:07:32.044376+0000","last_became_peered":"2024-09-11T08:07:32.044376+0000","last_unstale":"2024-09-11T08:11:59.847192+0000","last_undegraded":"2024-09-11T08:11:59.847192+0000","last_fullsized":"2024-09-11T08:11:59.847192+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-11T08:11:19.982185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:11:19.982185+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:38.948134+0000","scrub_duration":0.0066179189999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":103,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.843501+0000","last_change":"2024-09-11T08:10:22.436344+0000","last_active":"2024-09-11T08:11:59.843501+0000","last_peered":"2024-09-11T08:11:59.843501+0000","last_clean":"2024-09-11T08:11:59.843501+0000","last_became_active":"2024-09-11T08:07:30.045380+0000","last_became_peered":"2024-09-11T08:07:30.045380+0000","last_unstale":"2024-09-11T08:11:59.843501+0000","last_undegraded":"2024-09-11T08:11:59.843501+0000","last_fullsized":"2024-09-11T08:11:59.843501+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:22.436250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:10:22.436250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0076618179999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":98,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.841490+0000","last_change":"2024-09-11T08:10:26.621521+0000","last_active":"2024-09-11T08:11:59.841490+0000","last_peered":"2024-09-11T08:11:59.841490+0000","last_clean":"2024-09-11T08:11:59.841490+0000","last_became_active":"2024-09-11T08:07:34.051004+0000","last_became_peered":"2024-09-11T08:07:34.051004+0000","last_unstale":"2024-09-11T08:11:59.841490+0000","last_undegraded":"2024-09-11T08:11:59.841490+0000","last_fullsized":"2024-09-11T08:11:59.841490+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:26.621460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:10:26.621460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016671699999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":97,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.841906+0000","last_change":"2024-09-11T08:11:36.477823+0000","last_active":"2024-09-11T08:11:59.841906+0000","last_peered":"2024-09-11T08:11:59.841906+0000","last_clean":"2024-09-11T08:11:59.841906+0000","last_became_active":"2024-09-11T08:07:36.631668+0000","last_became_peered":"2024-09-11T08:07:36.631668+0000","last_unstale":"2024-09-11T08:11:59.841906+0000","last_undegraded":"2024-09-11T08:11:59.841906+0000","last_fullsized":"2024-09-11T08:11:59.841906+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:36.477734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:11:36.477734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:13:01.973259+0000","scrub_duration":0.0067568719999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"133'10083","reported_seq":10450,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-11T08:12:02.681456+0000","last_change":"2024-09-11T08:11:27.447825+0000","last_active":"2024-09-11T08:12:02.681456+0000","last_peered":"2024-09-11T08:12:02.681456+0000","last_clean":"2024-09-11T08:12:02.681456+0000","last_became_active":"2024-09-11T08:09:45.414847+0000","last_became_peered":"2024-09-11T08:09:45.414847+0000","last_unstale":"2024-09-11T08:12:02.681456+0000","last_undegraded":"2024-09-11T08:12:02.681456+0000","last_fullsized":"2024-09-11T08:12:02.681456+0000","mapping_epoch":91,"log_start":"106'3800","ondisk_log_start":"106'3800","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'7788","last_scrub_stamp":"2024-09-11T08:11:27.447746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_clean_scrub_stamp":"2024-09-11T08:11:27.447746+0000","objects_scrubbed":7674,"log_size":6283,"log_dups_size":0,"ondisk_log_size":6283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:47.727446+0000","scrub_duration":3.9345051240000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":363921408,"num_objects":5553,"num_object_clones":0,"num_object_copies":16659,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10083,"num_write_kb":500352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"86'38","reported_seq":213,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.843400+0000","last_change":"2024-09-11T08:11:16.652708+0000","last_active":"2024-09-11T08:11:59.843400+0000","last_peered":"2024-09-11T08:11:59.843400+0000","last_clean":"2024-09-11T08:11:59.843400+0000","last_became_active":"2024-09-11T08:07:32.052956+0000","last_became_peered":"2024-09-11T08:07:32.052956+0000","last_unstale":"2024-09-11T08:11:59.843400+0000","last_undegraded":"2024-09-11T08:11:59.843400+0000","last_fullsized":"2024-09-11T08:11:59.843400+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'38","last_scrub_stamp":"2024-09-11T08:11:16.652273+0000","last_deep_scrub":"86'38","last_deep_scrub_stamp":"2024-09-11T08:09:11.362339+0000","last_clean_scrub_stamp":"2024-09-11T08:11:16.652273+0000","objects_scrubbed":10,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:46.242558+0000","scrub_duration":0.011886363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"59'1","reported_seq":112,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.836742+0000","last_change":"2024-09-11T08:10:18.051791+0000","last_active":"2024-09-11T08:11:59.836742+0000","last_peered":"2024-09-11T08:11:59.836742+0000","last_clean":"2024-09-11T08:11:59.836742+0000","last_became_active":"2024-09-11T08:07:30.047513+0000","last_became_peered":"2024-09-11T08:07:30.047513+0000","last_unstale":"2024-09-11T08:11:59.836742+0000","last_undegraded":"2024-09-11T08:11:59.836742+0000","last_fullsized":"2024-09-11T08:11:59.836742+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-11T08:10:18.051701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:10:18.051701+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028800570000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":109,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.849828+0000","last_change":"2024-09-11T08:10:16.989515+0000","last_active":"2024-09-11T08:11:59.849828+0000","last_peered":"2024-09-11T08:11:59.849828+0000","last_clean":"2024-09-11T08:11:59.849828+0000","last_became_active":"2024-09-11T08:07:30.043130+0000","last_became_peered":"2024-09-11T08:07:30.043130+0000","last_unstale":"2024-09-11T08:11:59.849828+0000","last_undegraded":"2024-09-11T08:11:59.849828+0000","last_fullsized":"2024-09-11T08:11:59.849828+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-11T08:10:16.989458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:10:16.989458+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033825330000000001,"objects_trimmed":0,"snaptrim_duration":0.0026120850000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":99,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.846866+0000","last_change":"2024-09-11T08:10:37.957177+0000","last_active":"2024-09-11T08:11:59.846866+0000","last_peered":"2024-09-11T08:11:59.846866+0000","last_clean":"2024-09-11T08:11:59.846866+0000","last_became_active":"2024-09-11T08:07:34.057154+0000","last_became_peered":"2024-09-11T08:07:34.057154+0000","last_unstale":"2024-09-11T08:11:59.846866+0000","last_undegraded":"2024-09-11T08:11:59.846866+0000","last_fullsized":"2024-09-11T08:11:59.846866+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:37.957096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:10:37.957096+0000","last_clean_scrub_stamp":"2024-09-11T08:10:37.957096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011314437,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":97,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.853312+0000","last_change":"2024-09-11T08:11:38.502257+0000","last_active":"2024-09-11T08:11:59.853312+0000","last_peered":"2024-09-11T08:11:59.853312+0000","last_clean":"2024-09-11T08:11:59.853312+0000","last_became_active":"2024-09-11T08:07:36.075690+0000","last_became_peered":"2024-09-11T08:07:36.075690+0000","last_unstale":"2024-09-11T08:11:59.853312+0000","last_undegraded":"2024-09-11T08:11:59.853312+0000","last_fullsized":"2024-09-11T08:11:59.853312+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:38.502182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:11:38.502182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:55.269821+0000","scrub_duration":0.0039179619999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"133'9927","reported_seq":9976,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-11T08:12:02.959640+0000","last_change":"2024-09-11T08:09:45.414559+0000","last_active":"2024-09-11T08:12:02.959640+0000","last_peered":"2024-09-11T08:12:02.959640+0000","last_clean":"2024-09-11T08:12:02.959640+0000","last_became_active":"2024-09-11T08:09:45.414278+0000","last_became_peered":"2024-09-11T08:09:45.414278+0000","last_unstale":"2024-09-11T08:12:02.959640+0000","last_undegraded":"2024-09-11T08:12:02.959640+0000","last_fullsized":"2024-09-11T08:12:02.959640+0000","mapping_epoch":91,"log_start":"109'4300","ondisk_log_start":"109'4300","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_clean_scrub_stamp":"2024-09-11T08:09:44.385805+0000","objects_scrubbed":0,"log_size":5627,"log_dups_size":0,"ondisk_log_size":5627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":359596032,"num_objects":5487,"num_object_clones":0,"num_object_copies":16461,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9927,"num_write_kb":493248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":128,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.840127+0000","last_change":"2024-09-11T08:10:26.486264+0000","last_active":"2024-09-11T08:11:59.840127+0000","last_peered":"2024-09-11T08:11:59.840127+0000","last_clean":"2024-09-11T08:11:59.840127+0000","last_became_active":"2024-09-11T08:07:32.050673+0000","last_became_peered":"2024-09-11T08:07:32.050673+0000","last_unstale":"2024-09-11T08:11:59.840127+0000","last_undegraded":"2024-09-11T08:11:59.840127+0000","last_fullsized":"2024-09-11T08:11:59.840127+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'13","last_scrub_stamp":"2024-09-11T08:10:26.486178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:10:26.486178+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012477777000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":103,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.840261+0000","last_change":"2024-09-11T08:10:25.425751+0000","last_active":"2024-09-11T08:11:59.840261+0000","last_peered":"2024-09-11T08:11:59.840261+0000","last_clean":"2024-09-11T08:11:59.840261+0000","last_became_active":"2024-09-11T08:07:30.047014+0000","last_became_peered":"2024-09-11T08:07:30.047014+0000","last_unstale":"2024-09-11T08:11:59.840261+0000","last_undegraded":"2024-09-11T08:11:59.840261+0000","last_fullsized":"2024-09-11T08:11:59.840261+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:25.425399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:10:25.425399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032663789999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"127'2","reported_seq":119,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.845537+0000","last_change":"2024-09-11T08:11:32.348728+0000","last_active":"2024-09-11T08:11:59.845537+0000","last_peered":"2024-09-11T08:11:59.845537+0000","last_clean":"2024-09-11T08:11:59.845537+0000","last_became_active":"2024-09-11T08:07:30.045009+0000","last_became_peered":"2024-09-11T08:07:30.045009+0000","last_unstale":"2024-09-11T08:11:59.845537+0000","last_undegraded":"2024-09-11T08:11:59.845537+0000","last_fullsized":"2024-09-11T08:11:59.845537+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'2","last_scrub_stamp":"2024-09-11T08:11:32.348650+0000","last_deep_scrub":"127'2","last_deep_scrub_stamp":"2024-09-11T08:11:32.348650+0000","last_clean_scrub_stamp":"2024-09-11T08:11:32.348650+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:13:00.297188+0000","scrub_duration":0.013905997999999999,"objects_trimmed":0,"snaptrim_duration":0.001859533,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":98,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.851296+0000","last_change":"2024-09-11T08:10:19.998750+0000","last_active":"2024-09-11T08:11:59.851296+0000","last_peered":"2024-09-11T08:11:59.851296+0000","last_clean":"2024-09-11T08:11:59.851296+0000","last_became_active":"2024-09-11T08:07:34.063906+0000","last_became_peered":"2024-09-11T08:07:34.063906+0000","last_unstale":"2024-09-11T08:11:59.851296+0000","last_undegraded":"2024-09-11T08:11:59.851296+0000","last_fullsized":"2024-09-11T08:11:59.851296+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:19.998666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:10:19.998666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.020599591,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":97,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.851392+0000","last_change":"2024-09-11T08:10:22.212876+0000","last_active":"2024-09-11T08:11:59.851392+0000","last_peered":"2024-09-11T08:11:59.851392+0000","last_clean":"2024-09-11T08:11:59.851392+0000","last_became_active":"2024-09-11T08:07:36.631480+0000","last_became_peered":"2024-09-11T08:07:36.631480+0000","last_unstale":"2024-09-11T08:11:59.851392+0000","last_undegraded":"2024-09-11T08:11:59.851392+0000","last_fullsized":"2024-09-11T08:11:59.851392+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:22.212811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:10:22.212811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032158299999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"133'10074","reported_seq":10420,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-11T08:12:02.697911+0000","last_change":"2024-09-11T08:11:38.973718+0000","last_active":"2024-09-11T08:12:02.697911+0000","last_peered":"2024-09-11T08:12:02.697911+0000","last_clean":"2024-09-11T08:12:02.697911+0000","last_became_active":"2024-09-11T08:09:45.411757+0000","last_became_peered":"2024-09-11T08:09:45.411757+0000","last_unstale":"2024-09-11T08:12:02.697911+0000","last_undegraded":"2024-09-11T08:12:02.697911+0000","last_fullsized":"2024-09-11T08:12:02.697911+0000","mapping_epoch":91,"log_start":"111'4900","ondisk_log_start":"111'4900","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'8561","last_scrub_stamp":"2024-09-11T08:11:38.973659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_clean_scrub_stamp":"2024-09-11T08:11:38.973659+0000","objects_scrubbed":7171,"log_size":5174,"log_dups_size":0,"ondisk_log_size":5174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:13:06.422033+0000","scrub_duration":2.7886773580000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":363855872,"num_objects":5552,"num_object_clones":0,"num_object_copies":16656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10074,"num_write_kb":500032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":119,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.857145+0000","last_change":"2024-09-11T08:10:29.105854+0000","last_active":"2024-09-11T08:11:59.857145+0000","last_peered":"2024-09-11T08:11:59.857145+0000","last_clean":"2024-09-11T08:11:59.857145+0000","last_became_active":"2024-09-11T08:07:32.055407+0000","last_became_peered":"2024-09-11T08:07:32.055407+0000","last_unstale":"2024-09-11T08:11:59.857145+0000","last_undegraded":"2024-09-11T08:11:59.857145+0000","last_fullsized":"2024-09-11T08:11:59.857145+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-11T08:10:29.105566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:10:29.105566+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.016339228000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"59'1","reported_seq":110,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.852366+0000","last_change":"2024-09-11T08:10:34.991296+0000","last_active":"2024-09-11T08:11:59.852366+0000","last_peered":"2024-09-11T08:11:59.852366+0000","last_clean":"2024-09-11T08:11:59.852366+0000","last_became_active":"2024-09-11T08:07:30.050504+0000","last_became_peered":"2024-09-11T08:07:30.050504+0000","last_unstale":"2024-09-11T08:11:59.852366+0000","last_undegraded":"2024-09-11T08:11:59.852366+0000","last_fullsized":"2024-09-11T08:11:59.852366+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-11T08:10:34.991211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:10:34.991211+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0092067399999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":377,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-11T08:12:01.856851+0000","last_change":"2024-09-11T08:11:22.408697+0000","last_active":"2024-09-11T08:12:01.856851+0000","last_peered":"2024-09-11T08:12:01.856851+0000","last_clean":"2024-09-11T08:12:01.856851+0000","last_became_active":"2024-09-11T08:07:30.045051+0000","last_became_peered":"2024-09-11T08:07:30.045051+0000","last_unstale":"2024-09-11T08:12:01.856851+0000","last_undegraded":"2024-09-11T08:12:01.856851+0000","last_fullsized":"2024-09-11T08:12:01.856851+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-11T08:11:22.408607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:11:22.408607+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:38.586477+0000","scrub_duration":0.0068790099999999996,"objects_trimmed":0,"snaptrim_duration":0.000318821,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":266,"num_read_kb":261,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":98,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.849940+0000","last_change":"2024-09-11T08:10:33.984097+0000","last_active":"2024-09-11T08:11:59.849940+0000","last_peered":"2024-09-11T08:11:59.849940+0000","last_clean":"2024-09-11T08:11:59.849940+0000","last_became_active":"2024-09-11T08:07:34.058962+0000","last_became_peered":"2024-09-11T08:07:34.058962+0000","last_unstale":"2024-09-11T08:11:59.849940+0000","last_undegraded":"2024-09-11T08:11:59.849940+0000","last_fullsized":"2024-09-11T08:11:59.849940+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:33.984021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:10:33.984021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0059758429999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":97,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.843225+0000","last_change":"2024-09-11T08:10:33.381711+0000","last_active":"2024-09-11T08:11:59.843225+0000","last_peered":"2024-09-11T08:11:59.843225+0000","last_clean":"2024-09-11T08:11:59.843225+0000","last_became_active":"2024-09-11T08:07:36.077597+0000","last_became_peered":"2024-09-11T08:07:36.077597+0000","last_unstale":"2024-09-11T08:11:59.843225+0000","last_undegraded":"2024-09-11T08:11:59.843225+0000","last_fullsized":"2024-09-11T08:11:59.843225+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:33.381554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:10:33.381554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014471942999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"133'10056","reported_seq":10105,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-11T08:12:02.945189+0000","last_change":"2024-09-11T08:09:45.415108+0000","last_active":"2024-09-11T08:12:02.945189+0000","last_peered":"2024-09-11T08:12:02.945189+0000","last_clean":"2024-09-11T08:12:02.945189+0000","last_became_active":"2024-09-11T08:09:45.414851+0000","last_became_peered":"2024-09-11T08:09:45.414851+0000","last_unstale":"2024-09-11T08:12:02.945189+0000","last_undegraded":"2024-09-11T08:12:02.945189+0000","last_fullsized":"2024-09-11T08:12:02.945189+0000","mapping_epoch":91,"log_start":"109'4500","ondisk_log_start":"109'4500","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_clean_scrub_stamp":"2024-09-11T08:09:44.385805+0000","objects_scrubbed":0,"log_size":5556,"log_dups_size":0,"ondisk_log_size":5556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":366870528,"num_objects":5598,"num_object_clones":0,"num_object_copies":16794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10056,"num_write_kb":500928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"84'21","reported_seq":129,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.841791+0000","last_change":"2024-09-11T08:10:20.637001+0000","last_active":"2024-09-11T08:11:59.841791+0000","last_peered":"2024-09-11T08:11:59.841791+0000","last_clean":"2024-09-11T08:11:59.841791+0000","last_became_active":"2024-09-11T08:07:32.042339+0000","last_became_peered":"2024-09-11T08:07:32.042339+0000","last_unstale":"2024-09-11T08:11:59.841791+0000","last_undegraded":"2024-09-11T08:11:59.841791+0000","last_fullsized":"2024-09-11T08:11:59.841791+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'21","last_scrub_stamp":"2024-09-11T08:10:20.636918+0000","last_deep_scrub":"84'21","last_deep_scrub_stamp":"2024-09-11T08:08:56.705306+0000","last_clean_scrub_stamp":"2024-09-11T08:10:20.636918+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004681826,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":93,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.846638+0000","last_change":"2024-09-11T08:09:39.390240+0000","last_active":"2024-09-11T08:11:59.846638+0000","last_peered":"2024-09-11T08:11:59.846638+0000","last_clean":"2024-09-11T08:11:59.846638+0000","last_became_active":"2024-09-11T08:07:30.047246+0000","last_became_peered":"2024-09-11T08:07:30.047246+0000","last_unstale":"2024-09-11T08:11:59.846638+0000","last_undegraded":"2024-09-11T08:11:59.846638+0000","last_fullsized":"2024-09-11T08:11:59.846638+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:39.390158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:09:39.390158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013263249999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":149,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.852606+0000","last_change":"2024-09-11T08:11:17.419193+0000","last_active":"2024-09-11T08:11:59.852606+0000","last_peered":"2024-09-11T08:11:59.852606+0000","last_clean":"2024-09-11T08:11:59.852606+0000","last_became_active":"2024-09-11T08:07:19.726378+0000","last_became_peered":"2024-09-11T08:07:19.726378+0000","last_unstale":"2024-09-11T08:11:59.852606+0000","last_undegraded":"2024-09-11T08:11:59.852606+0000","last_fullsized":"2024-09-11T08:11:59.852606+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-11T08:11:17.419133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:05:21.231856+0000","last_clean_scrub_stamp":"2024-09-11T08:11:17.419133+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:46.438913+0000","scrub_duration":0.002818194,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":99,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.851259+0000","last_change":"2024-09-11T08:10:39.300321+0000","last_active":"2024-09-11T08:11:59.851259+0000","last_peered":"2024-09-11T08:11:59.851259+0000","last_clean":"2024-09-11T08:11:59.851259+0000","last_became_active":"2024-09-11T08:07:34.058414+0000","last_became_peered":"2024-09-11T08:07:34.058414+0000","last_unstale":"2024-09-11T08:11:59.851259+0000","last_undegraded":"2024-09-11T08:11:59.851259+0000","last_fullsized":"2024-09-11T08:11:59.851259+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:39.300265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:51.289872+0000","last_clean_scrub_stamp":"2024-09-11T08:10:39.300265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001629222,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":96,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.849790+0000","last_change":"2024-09-11T08:10:27.995889+0000","last_active":"2024-09-11T08:11:59.849790+0000","last_peered":"2024-09-11T08:11:59.849790+0000","last_clean":"2024-09-11T08:11:59.849790+0000","last_became_active":"2024-09-11T08:07:36.072508+0000","last_became_peered":"2024-09-11T08:07:36.072508+0000","last_unstale":"2024-09-11T08:11:59.849790+0000","last_undegraded":"2024-09-11T08:11:59.849790+0000","last_fullsized":"2024-09-11T08:11:59.849790+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:27.995806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:01.777429+0000","last_clean_scrub_stamp":"2024-09-11T08:10:27.995806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001407127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"133'10012","reported_seq":10371,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-11T08:12:02.675626+0000","last_change":"2024-09-11T08:11:24.558667+0000","last_active":"2024-09-11T08:12:02.675626+0000","last_peered":"2024-09-11T08:12:02.675626+0000","last_clean":"2024-09-11T08:12:02.675626+0000","last_became_active":"2024-09-11T08:09:45.412638+0000","last_became_peered":"2024-09-11T08:09:45.412638+0000","last_unstale":"2024-09-11T08:12:02.675626+0000","last_undegraded":"2024-09-11T08:12:02.675626+0000","last_fullsized":"2024-09-11T08:12:02.675626+0000","mapping_epoch":91,"log_start":"106'3700","ondisk_log_start":"106'3700","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'7543","last_scrub_stamp":"2024-09-11T08:11:24.558579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_clean_scrub_stamp":"2024-09-11T08:11:24.558579+0000","objects_scrubbed":7445,"log_size":6312,"log_dups_size":0,"ondisk_log_size":6312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:27.715583+0000","scrub_duration":3.1005590700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358809624,"num_objects":5476,"num_object_clones":0,"num_object_copies":16428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5476,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10012,"num_write_kb":495553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"88'16","reported_seq":125,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.837371+0000","last_change":"2024-09-11T08:09:39.636044+0000","last_active":"2024-09-11T08:11:59.837371+0000","last_peered":"2024-09-11T08:11:59.837371+0000","last_clean":"2024-09-11T08:11:59.837371+0000","last_became_active":"2024-09-11T08:07:32.047971+0000","last_became_peered":"2024-09-11T08:07:32.047971+0000","last_unstale":"2024-09-11T08:11:59.837371+0000","last_undegraded":"2024-09-11T08:11:59.837371+0000","last_fullsized":"2024-09-11T08:11:59.837371+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'16","last_scrub_stamp":"2024-09-11T08:09:39.635984+0000","last_deep_scrub":"88'16","last_deep_scrub_stamp":"2024-09-11T08:09:39.635984+0000","last_clean_scrub_stamp":"2024-09-11T08:09:39.635984+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038085599999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":103,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.836669+0000","last_change":"2024-09-11T08:10:26.069693+0000","last_active":"2024-09-11T08:11:59.836669+0000","last_peered":"2024-09-11T08:11:59.836669+0000","last_clean":"2024-09-11T08:11:59.836669+0000","last_became_active":"2024-09-11T08:07:30.039884+0000","last_became_peered":"2024-09-11T08:07:30.039884+0000","last_unstale":"2024-09-11T08:11:59.836669+0000","last_undegraded":"2024-09-11T08:11:59.836669+0000","last_fullsized":"2024-09-11T08:11:59.836669+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:26.069640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:36.417911+0000","last_clean_scrub_stamp":"2024-09-11T08:10:26.069640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001453558,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":98,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.849640+0000","last_change":"2024-09-11T08:10:31.929612+0000","last_active":"2024-09-11T08:11:59.849640+0000","last_peered":"2024-09-11T08:11:59.849640+0000","last_clean":"2024-09-11T08:11:59.849640+0000","last_became_active":"2024-09-11T08:07:34.053347+0000","last_became_peered":"2024-09-11T08:07:34.053347+0000","last_unstale":"2024-09-11T08:11:59.849640+0000","last_undegraded":"2024-09-11T08:11:59.849640+0000","last_fullsized":"2024-09-11T08:11:59.849640+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:31.929528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:10:31.929528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015531010000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":97,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.845194+0000","last_change":"2024-09-11T08:11:37.480260+0000","last_active":"2024-09-11T08:11:59.845194+0000","last_peered":"2024-09-11T08:11:59.845194+0000","last_clean":"2024-09-11T08:11:59.845194+0000","last_became_active":"2024-09-11T08:07:36.631687+0000","last_became_peered":"2024-09-11T08:07:36.631687+0000","last_unstale":"2024-09-11T08:11:59.845194+0000","last_undegraded":"2024-09-11T08:11:59.845194+0000","last_fullsized":"2024-09-11T08:11:59.845194+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:37.480123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:11:37.480123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:48.786633+0000","scrub_duration":0.0064260089999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"133'10009","reported_seq":10363,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-11T08:12:02.760369+0000","last_change":"2024-09-11T08:11:22.474726+0000","last_active":"2024-09-11T08:12:02.760369+0000","last_peered":"2024-09-11T08:12:02.760369+0000","last_clean":"2024-09-11T08:12:02.760369+0000","last_became_active":"2024-09-11T08:09:45.413207+0000","last_became_peered":"2024-09-11T08:09:45.413207+0000","last_unstale":"2024-09-11T08:12:02.760369+0000","last_undegraded":"2024-09-11T08:12:02.760369+0000","last_fullsized":"2024-09-11T08:12:02.760369+0000","mapping_epoch":91,"log_start":"103'3100","ondisk_log_start":"103'3100","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'7432","last_scrub_stamp":"2024-09-11T08:11:22.474630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_clean_scrub_stamp":"2024-09-11T08:11:22.474630+0000","objects_scrubbed":7334,"log_size":6909,"log_dups_size":0,"ondisk_log_size":6909,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:23.920712+0000","scrub_duration":4.0826116800000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":361168896,"num_objects":5511,"num_object_clones":0,"num_object_copies":16533,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10009,"num_write_kb":496640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"77'25","reported_seq":165,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.837141+0000","last_change":"2024-09-11T08:10:31.126024+0000","last_active":"2024-09-11T08:11:59.837141+0000","last_peered":"2024-09-11T08:11:59.837141+0000","last_clean":"2024-09-11T08:11:59.837141+0000","last_became_active":"2024-09-11T08:09:08.106771+0000","last_became_peered":"2024-09-11T08:09:08.106771+0000","last_unstale":"2024-09-11T08:11:59.837141+0000","last_undegraded":"2024-09-11T08:11:59.837141+0000","last_fullsized":"2024-09-11T08:11:59.837141+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'25","last_scrub_stamp":"2024-09-11T08:10:31.125934+0000","last_deep_scrub":"77'25","last_deep_scrub_stamp":"2024-09-11T08:08:32.458210+0000","last_clean_scrub_stamp":"2024-09-11T08:10:31.125934+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0096263719999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":102,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.841906+0000","last_change":"2024-09-11T08:11:16.505483+0000","last_active":"2024-09-11T08:11:59.841906+0000","last_peered":"2024-09-11T08:11:59.841906+0000","last_clean":"2024-09-11T08:11:59.841906+0000","last_became_active":"2024-09-11T08:07:30.042555+0000","last_became_peered":"2024-09-11T08:07:30.042555+0000","last_unstale":"2024-09-11T08:11:59.841906+0000","last_undegraded":"2024-09-11T08:11:59.841906+0000","last_fullsized":"2024-09-11T08:11:59.841906+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:16.505384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:11:16.505384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:26.959383+0000","scrub_duration":0.010166646999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":99,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.843288+0000","last_change":"2024-09-11T08:10:27.459645+0000","last_active":"2024-09-11T08:11:59.843288+0000","last_peered":"2024-09-11T08:11:59.843288+0000","last_clean":"2024-09-11T08:11:59.843288+0000","last_became_active":"2024-09-11T08:07:34.054961+0000","last_became_peered":"2024-09-11T08:07:34.054961+0000","last_unstale":"2024-09-11T08:11:59.843288+0000","last_undegraded":"2024-09-11T08:11:59.843288+0000","last_fullsized":"2024-09-11T08:11:59.843288+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:27.459436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:10:27.459436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0053113359999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":106,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.845300+0000","last_change":"2024-09-11T08:11:39.370275+0000","last_active":"2024-09-11T08:11:59.845300+0000","last_peered":"2024-09-11T08:11:59.845300+0000","last_clean":"2024-09-11T08:11:59.845300+0000","last_became_active":"2024-09-11T08:07:36.071194+0000","last_became_peered":"2024-09-11T08:07:36.071194+0000","last_unstale":"2024-09-11T08:11:59.845300+0000","last_undegraded":"2024-09-11T08:11:59.845300+0000","last_fullsized":"2024-09-11T08:11:59.845300+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:39.370193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:11:39.370193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:40.590716+0000","scrub_duration":0.014190103000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"133'9918","reported_seq":9968,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-11T08:12:02.894198+0000","last_change":"2024-09-11T08:09:45.416120+0000","last_active":"2024-09-11T08:12:02.894198+0000","last_peered":"2024-09-11T08:12:02.894198+0000","last_clean":"2024-09-11T08:12:02.894198+0000","last_became_active":"2024-09-11T08:09:45.415738+0000","last_became_peered":"2024-09-11T08:09:45.415738+0000","last_unstale":"2024-09-11T08:12:02.894198+0000","last_undegraded":"2024-09-11T08:12:02.894198+0000","last_fullsized":"2024-09-11T08:12:02.894198+0000","mapping_epoch":91,"log_start":"109'4300","ondisk_log_start":"109'4300","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_clean_scrub_stamp":"2024-09-11T08:09:44.385805+0000","objects_scrubbed":0,"log_size":5618,"log_dups_size":0,"ondisk_log_size":5618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":363200512,"num_objects":5542,"num_object_clones":0,"num_object_copies":16626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9918,"num_write_kb":494720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":116,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.836867+0000","last_change":"2024-09-11T08:10:35.175848+0000","last_active":"2024-09-11T08:11:59.836867+0000","last_peered":"2024-09-11T08:11:59.836867+0000","last_clean":"2024-09-11T08:11:59.836867+0000","last_became_active":"2024-09-11T08:07:32.043314+0000","last_became_peered":"2024-09-11T08:07:32.043314+0000","last_unstale":"2024-09-11T08:11:59.836867+0000","last_undegraded":"2024-09-11T08:11:59.836867+0000","last_fullsized":"2024-09-11T08:11:59.836867+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-11T08:10:35.175750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:10:35.175750+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.088039645,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":103,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.854135+0000","last_change":"2024-09-11T08:10:38.126730+0000","last_active":"2024-09-11T08:11:59.854135+0000","last_peered":"2024-09-11T08:11:59.854135+0000","last_clean":"2024-09-11T08:11:59.854135+0000","last_became_active":"2024-09-11T08:07:30.046684+0000","last_became_peered":"2024-09-11T08:07:30.046684+0000","last_unstale":"2024-09-11T08:11:59.854135+0000","last_undegraded":"2024-09-11T08:11:59.854135+0000","last_fullsized":"2024-09-11T08:11:59.854135+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:38.126638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:10:38.126638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0084326729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"66'8","reported_seq":170,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.844984+0000","last_change":"2024-09-11T08:11:24.359888+0000","last_active":"2024-09-11T08:11:59.844984+0000","last_peered":"2024-09-11T08:11:59.844984+0000","last_clean":"2024-09-11T08:11:59.844984+0000","last_became_active":"2024-09-11T08:07:34.062469+0000","last_became_peered":"2024-09-11T08:07:34.062469+0000","last_unstale":"2024-09-11T08:11:59.844984+0000","last_undegraded":"2024-09-11T08:11:59.844984+0000","last_fullsized":"2024-09-11T08:11:59.844984+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:11:24.359780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:11:24.359780+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:31.728208+0000","scrub_duration":0.0054480609999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":87,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.837133+0000","last_change":"2024-09-11T08:09:14.613599+0000","last_active":"2024-09-11T08:11:59.837133+0000","last_peered":"2024-09-11T08:11:59.837133+0000","last_clean":"2024-09-11T08:11:59.837133+0000","last_became_active":"2024-09-11T08:07:36.074684+0000","last_became_peered":"2024-09-11T08:07:36.074684+0000","last_unstale":"2024-09-11T08:11:59.837133+0000","last_undegraded":"2024-09-11T08:11:59.837133+0000","last_fullsized":"2024-09-11T08:11:59.837133+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:14.613532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:14.613532+0000","last_clean_scrub_stamp":"2024-09-11T08:09:14.613532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001503648,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"133'9979","reported_seq":10332,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-11T08:12:02.792610+0000","last_change":"2024-09-11T08:11:22.178487+0000","last_active":"2024-09-11T08:12:02.792610+0000","last_peered":"2024-09-11T08:12:02.792610+0000","last_clean":"2024-09-11T08:12:02.792610+0000","last_became_active":"2024-09-11T08:09:45.411521+0000","last_became_peered":"2024-09-11T08:09:45.411521+0000","last_unstale":"2024-09-11T08:12:02.792610+0000","last_undegraded":"2024-09-11T08:12:02.792610+0000","last_fullsized":"2024-09-11T08:12:02.792610+0000","mapping_epoch":91,"log_start":"103'3100","ondisk_log_start":"103'3100","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'7415","last_scrub_stamp":"2024-09-11T08:11:22.178363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_clean_scrub_stamp":"2024-09-11T08:11:22.178363+0000","objects_scrubbed":7336,"log_size":6879,"log_dups_size":0,"ondisk_log_size":6879,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:46.113024+0000","scrub_duration":2.7897035839999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":366936064,"num_objects":5599,"num_object_clones":0,"num_object_copies":16797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9979,"num_write_kb":498496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":117,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.847112+0000","last_change":"2024-09-11T08:11:40.062007+0000","last_active":"2024-09-11T08:11:59.847112+0000","last_peered":"2024-09-11T08:11:59.847112+0000","last_clean":"2024-09-11T08:11:59.847112+0000","last_became_active":"2024-09-11T08:07:32.054885+0000","last_became_peered":"2024-09-11T08:07:32.054885+0000","last_unstale":"2024-09-11T08:11:59.847112+0000","last_undegraded":"2024-09-11T08:11:59.847112+0000","last_fullsized":"2024-09-11T08:11:59.847112+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:11:40.061926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:11:40.061926+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:40.906878+0000","scrub_duration":0.106648482,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":112,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.849315+0000","last_change":"2024-09-11T08:11:33.819647+0000","last_active":"2024-09-11T08:11:59.849315+0000","last_peered":"2024-09-11T08:11:59.849315+0000","last_clean":"2024-09-11T08:11:59.849315+0000","last_became_active":"2024-09-11T08:07:30.045613+0000","last_became_peered":"2024-09-11T08:07:30.045613+0000","last_unstale":"2024-09-11T08:11:59.849315+0000","last_undegraded":"2024-09-11T08:11:59.849315+0000","last_fullsized":"2024-09-11T08:11:59.849315+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:33.819593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:11:33.819593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:13:03.296968+0000","scrub_duration":0.0034651399999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":108,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.852326+0000","last_change":"2024-09-11T08:11:33.364733+0000","last_active":"2024-09-11T08:11:59.852326+0000","last_peered":"2024-09-11T08:11:59.852326+0000","last_clean":"2024-09-11T08:11:59.852326+0000","last_became_active":"2024-09-11T08:07:34.056606+0000","last_became_peered":"2024-09-11T08:07:34.056606+0000","last_unstale":"2024-09-11T08:11:59.852326+0000","last_undegraded":"2024-09-11T08:11:59.852326+0000","last_fullsized":"2024-09-11T08:11:59.852326+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:33.364578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:11:33.364578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:34.370854+0000","scrub_duration":0.0099402859999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":97,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.852340+0000","last_change":"2024-09-11T08:11:18.967755+0000","last_active":"2024-09-11T08:11:59.852340+0000","last_peered":"2024-09-11T08:11:59.852340+0000","last_clean":"2024-09-11T08:11:59.852340+0000","last_became_active":"2024-09-11T08:07:36.072452+0000","last_became_peered":"2024-09-11T08:07:36.072452+0000","last_unstale":"2024-09-11T08:11:59.852340+0000","last_undegraded":"2024-09-11T08:11:59.852340+0000","last_fullsized":"2024-09-11T08:11:59.852340+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:18.967665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:11:18.967665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:48.119844+0000","scrub_duration":0.013929113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"133'10145","reported_seq":10195,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-11T08:12:02.928357+0000","last_change":"2024-09-11T08:09:45.414158+0000","last_active":"2024-09-11T08:12:02.928357+0000","last_peered":"2024-09-11T08:12:02.928357+0000","last_clean":"2024-09-11T08:12:02.928357+0000","last_became_active":"2024-09-11T08:09:45.413883+0000","last_became_peered":"2024-09-11T08:09:45.413883+0000","last_unstale":"2024-09-11T08:12:02.928357+0000","last_undegraded":"2024-09-11T08:12:02.928357+0000","last_fullsized":"2024-09-11T08:12:02.928357+0000","mapping_epoch":91,"log_start":"109'4500","ondisk_log_start":"109'4500","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_clean_scrub_stamp":"2024-09-11T08:09:44.385805+0000","objects_scrubbed":0,"log_size":5645,"log_dups_size":0,"ondisk_log_size":5645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":366804992,"num_objects":5597,"num_object_clones":0,"num_object_copies":16791,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10145,"num_write_kb":503744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":141,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.842004+0000","last_change":"2024-09-11T08:11:29.618794+0000","last_active":"2024-09-11T08:11:59.842004+0000","last_peered":"2024-09-11T08:11:59.842004+0000","last_clean":"2024-09-11T08:11:59.842004+0000","last_became_active":"2024-09-11T08:07:32.050880+0000","last_became_peered":"2024-09-11T08:07:32.050880+0000","last_unstale":"2024-09-11T08:11:59.842004+0000","last_undegraded":"2024-09-11T08:11:59.842004+0000","last_fullsized":"2024-09-11T08:11:59.842004+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-11T08:11:29.618722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:11:29.618722+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:35.510180+0000","scrub_duration":0.0019851790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":103,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.851600+0000","last_change":"2024-09-11T08:10:19.201593+0000","last_active":"2024-09-11T08:11:59.851600+0000","last_peered":"2024-09-11T08:11:59.851600+0000","last_clean":"2024-09-11T08:11:59.851600+0000","last_became_active":"2024-09-11T08:07:30.043294+0000","last_became_peered":"2024-09-11T08:07:30.043294+0000","last_unstale":"2024-09-11T08:11:59.851600+0000","last_undegraded":"2024-09-11T08:11:59.851600+0000","last_fullsized":"2024-09-11T08:11:59.851600+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:19.201434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:44.385372+0000","last_clean_scrub_stamp":"2024-09-11T08:10:19.201434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023250469999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":108,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.845994+0000","last_change":"2024-09-11T08:11:25.397839+0000","last_active":"2024-09-11T08:11:59.845994+0000","last_peered":"2024-09-11T08:11:59.845994+0000","last_clean":"2024-09-11T08:11:59.845994+0000","last_became_active":"2024-09-11T08:07:34.063589+0000","last_became_peered":"2024-09-11T08:07:34.063589+0000","last_unstale":"2024-09-11T08:11:59.845994+0000","last_undegraded":"2024-09-11T08:11:59.845994+0000","last_fullsized":"2024-09-11T08:11:59.845994+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:25.397757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:11:25.397757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:27.261790+0000","scrub_duration":0.00152206,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":97,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.852585+0000","last_change":"2024-09-11T08:10:15.257677+0000","last_active":"2024-09-11T08:11:59.852585+0000","last_peered":"2024-09-11T08:11:59.852585+0000","last_clean":"2024-09-11T08:11:59.852585+0000","last_became_active":"2024-09-11T08:07:36.077523+0000","last_became_peered":"2024-09-11T08:07:36.077523+0000","last_unstale":"2024-09-11T08:11:59.852585+0000","last_undegraded":"2024-09-11T08:11:59.852585+0000","last_fullsized":"2024-09-11T08:11:59.852585+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:15.257622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:10:15.257622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0063147990000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"133'10206","reported_seq":10256,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-11T08:12:02.830311+0000","last_change":"2024-09-11T08:09:45.411095+0000","last_active":"2024-09-11T08:12:02.830311+0000","last_peered":"2024-09-11T08:12:02.830311+0000","last_clean":"2024-09-11T08:12:02.830311+0000","last_became_active":"2024-09-11T08:09:45.410825+0000","last_became_peered":"2024-09-11T08:09:45.410825+0000","last_unstale":"2024-09-11T08:12:02.830311+0000","last_undegraded":"2024-09-11T08:12:02.830311+0000","last_fullsized":"2024-09-11T08:12:02.830311+0000","mapping_epoch":91,"log_start":"114'5500","ondisk_log_start":"114'5500","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_clean_scrub_stamp":"2024-09-11T08:09:44.385805+0000","objects_scrubbed":0,"log_size":4706,"log_dups_size":0,"ondisk_log_size":4706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":368574464,"num_objects":5624,"num_object_clones":0,"num_object_copies":16872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10206,"num_write_kb":506560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":124,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.836531+0000","last_change":"2024-09-11T08:10:44.085605+0000","last_active":"2024-09-11T08:11:59.836531+0000","last_peered":"2024-09-11T08:11:59.836531+0000","last_clean":"2024-09-11T08:11:59.836531+0000","last_became_active":"2024-09-11T08:07:32.048164+0000","last_became_peered":"2024-09-11T08:07:32.048164+0000","last_unstale":"2024-09-11T08:11:59.836531+0000","last_undegraded":"2024-09-11T08:11:59.836531+0000","last_fullsized":"2024-09-11T08:11:59.836531+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-11T08:10:44.085517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:10:44.085517+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:13.712893+0000","scrub_duration":0.0074629060000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":103,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.851822+0000","last_change":"2024-09-11T08:10:27.250069+0000","last_active":"2024-09-11T08:11:59.851822+0000","last_peered":"2024-09-11T08:11:59.851822+0000","last_clean":"2024-09-11T08:11:59.851822+0000","last_became_active":"2024-09-11T08:07:30.041219+0000","last_became_peered":"2024-09-11T08:07:30.041219+0000","last_unstale":"2024-09-11T08:11:59.851822+0000","last_undegraded":"2024-09-11T08:11:59.851822+0000","last_fullsized":"2024-09-11T08:11:59.851822+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:27.250021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:53.247275+0000","last_clean_scrub_stamp":"2024-09-11T08:10:27.250021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011297309999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":98,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.854877+0000","last_change":"2024-09-11T08:10:27.002636+0000","last_active":"2024-09-11T08:11:59.854877+0000","last_peered":"2024-09-11T08:11:59.854877+0000","last_clean":"2024-09-11T08:11:59.854877+0000","last_became_active":"2024-09-11T08:07:34.057479+0000","last_became_peered":"2024-09-11T08:07:34.057479+0000","last_unstale":"2024-09-11T08:11:59.854877+0000","last_undegraded":"2024-09-11T08:11:59.854877+0000","last_fullsized":"2024-09-11T08:11:59.854877+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:27.002455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:10:27.002455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044259809999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":97,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.853852+0000","last_change":"2024-09-11T08:10:42.097297+0000","last_active":"2024-09-11T08:11:59.853852+0000","last_peered":"2024-09-11T08:11:59.853852+0000","last_clean":"2024-09-11T08:11:59.853852+0000","last_became_active":"2024-09-11T08:07:36.078075+0000","last_became_peered":"2024-09-11T08:07:36.078075+0000","last_unstale":"2024-09-11T08:11:59.853852+0000","last_undegraded":"2024-09-11T08:11:59.853852+0000","last_fullsized":"2024-09-11T08:11:59.853852+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:42.097207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:10:42.097207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0056763179999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"133'13342","reported_seq":13542,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-11T08:12:02.879870+0000","last_change":"2024-09-11T08:11:59.906894+0000","last_active":"2024-09-11T08:12:02.879870+0000","last_peered":"2024-09-11T08:12:02.879870+0000","last_clean":"2024-09-11T08:12:02.879870+0000","last_became_active":"2024-09-11T08:09:45.412566+0000","last_became_peered":"2024-09-11T08:09:45.412566+0000","last_unstale":"2024-09-11T08:12:02.879870+0000","last_undegraded":"2024-09-11T08:12:02.879870+0000","last_fullsized":"2024-09-11T08:12:02.879870+0000","mapping_epoch":91,"log_start":"127'8302","ondisk_log_start":"127'8302","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:44.385805+0000","last_clean_scrub_stamp":"2024-09-11T08:09:44.385805+0000","objects_scrubbed":3586,"log_size":5040,"log_dups_size":0,"ondisk_log_size":5040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":367198208,"num_objects":5603,"num_object_clones":0,"num_object_copies":16809,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10043,"num_write_kb":500672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":125,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.848995+0000","last_change":"2024-09-11T08:10:39.911118+0000","last_active":"2024-09-11T08:11:59.848995+0000","last_peered":"2024-09-11T08:11:59.848995+0000","last_clean":"2024-09-11T08:11:59.848995+0000","last_became_active":"2024-09-11T08:07:32.039275+0000","last_became_peered":"2024-09-11T08:07:32.039275+0000","last_unstale":"2024-09-11T08:11:59.848995+0000","last_undegraded":"2024-09-11T08:11:59.848995+0000","last_fullsized":"2024-09-11T08:11:59.848995+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-11T08:10:39.911043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:10:39.911043+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:03.238724+0000","scrub_duration":0.0020973419999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":120,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.851739+0000","last_change":"2024-09-11T08:11:27.256947+0000","last_active":"2024-09-11T08:11:59.851739+0000","last_peered":"2024-09-11T08:11:59.851739+0000","last_clean":"2024-09-11T08:11:59.851739+0000","last_became_active":"2024-09-11T08:07:30.045967+0000","last_became_peered":"2024-09-11T08:07:30.045967+0000","last_unstale":"2024-09-11T08:11:59.851739+0000","last_undegraded":"2024-09-11T08:11:59.851739+0000","last_fullsized":"2024-09-11T08:11:59.851739+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-11T08:11:27.256886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:11:27.256886+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:47.796310+0000","scrub_duration":0.010702672999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"66'8","reported_seq":161,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.850822+0000","last_change":"2024-09-11T08:10:40.276739+0000","last_active":"2024-09-11T08:11:59.850822+0000","last_peered":"2024-09-11T08:11:59.850822+0000","last_clean":"2024-09-11T08:11:59.850822+0000","last_became_active":"2024-09-11T08:07:34.058015+0000","last_became_peered":"2024-09-11T08:07:34.058015+0000","last_unstale":"2024-09-11T08:11:59.850822+0000","last_undegraded":"2024-09-11T08:11:59.850822+0000","last_fullsized":"2024-09-11T08:11:59.850822+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:10:40.276651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:10:40.276651+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027620549999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":96,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.849120+0000","last_change":"2024-09-11T08:11:16.768991+0000","last_active":"2024-09-11T08:11:59.849120+0000","last_peered":"2024-09-11T08:11:59.849120+0000","last_clean":"2024-09-11T08:11:59.849120+0000","last_became_active":"2024-09-11T08:07:36.631755+0000","last_became_peered":"2024-09-11T08:07:36.631755+0000","last_unstale":"2024-09-11T08:11:59.849120+0000","last_undegraded":"2024-09-11T08:11:59.849120+0000","last_fullsized":"2024-09-11T08:11:59.849120+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:16.768914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:11:16.768914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:32.682600+0000","scrub_duration":0.0045193669999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":102,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.841482+0000","last_change":"2024-09-11T08:10:36.436042+0000","last_active":"2024-09-11T08:11:59.841482+0000","last_peered":"2024-09-11T08:11:59.841482+0000","last_clean":"2024-09-11T08:11:59.841482+0000","last_became_active":"2024-09-11T08:07:30.045910+0000","last_became_peered":"2024-09-11T08:07:30.045910+0000","last_unstale":"2024-09-11T08:11:59.841482+0000","last_undegraded":"2024-09-11T08:11:59.841482+0000","last_fullsized":"2024-09-11T08:11:59.841482+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:36.435971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:10:36.435971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023002999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"84'9","reported_seq":113,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.852494+0000","last_change":"2024-09-11T08:11:21.397319+0000","last_active":"2024-09-11T08:11:59.852494+0000","last_peered":"2024-09-11T08:11:59.852494+0000","last_clean":"2024-09-11T08:11:59.852494+0000","last_became_active":"2024-09-11T08:07:32.046891+0000","last_became_peered":"2024-09-11T08:07:32.046891+0000","last_unstale":"2024-09-11T08:11:59.852494+0000","last_undegraded":"2024-09-11T08:11:59.852494+0000","last_fullsized":"2024-09-11T08:11:59.852494+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-09-11T08:11:21.397179+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-09-11T08:09:03.418457+0000","last_clean_scrub_stamp":"2024-09-11T08:11:21.397179+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:48.076959+0000","scrub_duration":0.0062300630000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":98,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.849680+0000","last_change":"2024-09-11T08:10:43.836622+0000","last_active":"2024-09-11T08:11:59.849680+0000","last_peered":"2024-09-11T08:11:59.849680+0000","last_clean":"2024-09-11T08:11:59.849680+0000","last_became_active":"2024-09-11T08:07:34.057774+0000","last_became_peered":"2024-09-11T08:07:34.057774+0000","last_unstale":"2024-09-11T08:11:59.849680+0000","last_undegraded":"2024-09-11T08:11:59.849680+0000","last_fullsized":"2024-09-11T08:11:59.849680+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:43.836566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:10:43.836566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0087284040000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":97,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.850911+0000","last_change":"2024-09-11T08:10:38.297462+0000","last_active":"2024-09-11T08:11:59.850911+0000","last_peered":"2024-09-11T08:11:59.850911+0000","last_clean":"2024-09-11T08:11:59.850911+0000","last_became_active":"2024-09-11T08:07:36.633244+0000","last_became_peered":"2024-09-11T08:07:36.633244+0000","last_unstale":"2024-09-11T08:11:59.850911+0000","last_undegraded":"2024-09-11T08:11:59.850911+0000","last_fullsized":"2024-09-11T08:11:59.850911+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:38.297367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:10:38.297367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00156018,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"84'13","reported_seq":121,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.857212+0000","last_change":"2024-09-11T08:11:32.926108+0000","last_active":"2024-09-11T08:11:59.857212+0000","last_peered":"2024-09-11T08:11:59.857212+0000","last_clean":"2024-09-11T08:11:59.857212+0000","last_became_active":"2024-09-11T08:07:32.048350+0000","last_became_peered":"2024-09-11T08:07:32.048350+0000","last_unstale":"2024-09-11T08:11:59.857212+0000","last_undegraded":"2024-09-11T08:11:59.857212+0000","last_fullsized":"2024-09-11T08:11:59.857212+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'13","last_scrub_stamp":"2024-09-11T08:11:32.925950+0000","last_deep_scrub":"84'13","last_deep_scrub_stamp":"2024-09-11T08:08:57.835634+0000","last_clean_scrub_stamp":"2024-09-11T08:11:32.925950+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:58.685138+0000","scrub_duration":0.013245129,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":113,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.851030+0000","last_change":"2024-09-11T08:11:35.201048+0000","last_active":"2024-09-11T08:11:59.851030+0000","last_peered":"2024-09-11T08:11:59.851030+0000","last_clean":"2024-09-11T08:11:59.851030+0000","last_became_active":"2024-09-11T08:07:30.044055+0000","last_became_peered":"2024-09-11T08:07:30.044055+0000","last_unstale":"2024-09-11T08:11:59.851030+0000","last_undegraded":"2024-09-11T08:11:59.851030+0000","last_fullsized":"2024-09-11T08:11:59.851030+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:35.200970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:30.471620+0000","last_clean_scrub_stamp":"2024-09-11T08:11:35.200970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:36.402648+0000","scrub_duration":0.0012204990000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":99,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.852363+0000","last_change":"2024-09-11T08:11:16.386090+0000","last_active":"2024-09-11T08:11:59.852363+0000","last_peered":"2024-09-11T08:11:59.852363+0000","last_clean":"2024-09-11T08:11:59.852363+0000","last_became_active":"2024-09-11T08:07:34.057127+0000","last_became_peered":"2024-09-11T08:07:34.057127+0000","last_unstale":"2024-09-11T08:11:59.852363+0000","last_undegraded":"2024-09-11T08:11:59.852363+0000","last_fullsized":"2024-09-11T08:11:59.852363+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:16.385713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:04.370734+0000","last_clean_scrub_stamp":"2024-09-11T08:11:16.385713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:38.946720+0000","scrub_duration":0.013591255999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":96,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.844877+0000","last_change":"2024-09-11T08:10:29.963987+0000","last_active":"2024-09-11T08:11:59.844877+0000","last_peered":"2024-09-11T08:11:59.844877+0000","last_clean":"2024-09-11T08:11:59.844877+0000","last_became_active":"2024-09-11T08:07:36.071513+0000","last_became_peered":"2024-09-11T08:07:36.071513+0000","last_unstale":"2024-09-11T08:11:59.844877+0000","last_undegraded":"2024-09-11T08:11:59.844877+0000","last_fullsized":"2024-09-11T08:11:59.844877+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:29.963899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:10:29.963899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022607389999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":116,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.849506+0000","last_change":"2024-09-11T08:10:18.962167+0000","last_active":"2024-09-11T08:11:59.849506+0000","last_peered":"2024-09-11T08:11:59.849506+0000","last_clean":"2024-09-11T08:11:59.849506+0000","last_became_active":"2024-09-11T08:07:32.041761+0000","last_became_peered":"2024-09-11T08:07:32.041761+0000","last_unstale":"2024-09-11T08:11:59.849506+0000","last_undegraded":"2024-09-11T08:11:59.849506+0000","last_fullsized":"2024-09-11T08:11:59.849506+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:10:18.962088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:10:18.962088+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035069179999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":103,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.846846+0000","last_change":"2024-09-11T08:10:38.937153+0000","last_active":"2024-09-11T08:11:59.846846+0000","last_peered":"2024-09-11T08:11:59.846846+0000","last_clean":"2024-09-11T08:11:59.846846+0000","last_became_active":"2024-09-11T08:07:30.045380+0000","last_became_peered":"2024-09-11T08:07:30.045380+0000","last_unstale":"2024-09-11T08:11:59.846846+0000","last_undegraded":"2024-09-11T08:11:59.846846+0000","last_fullsized":"2024-09-11T08:11:59.846846+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:38.937071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:48.935387+0000","last_clean_scrub_stamp":"2024-09-11T08:10:38.937071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015757854000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"66'8","reported_seq":161,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.850030+0000","last_change":"2024-09-11T08:11:29.518706+0000","last_active":"2024-09-11T08:11:59.850030+0000","last_peered":"2024-09-11T08:11:59.850030+0000","last_clean":"2024-09-11T08:11:59.850030+0000","last_became_active":"2024-09-11T08:07:34.057937+0000","last_became_peered":"2024-09-11T08:07:34.057937+0000","last_unstale":"2024-09-11T08:11:59.850030+0000","last_undegraded":"2024-09-11T08:11:59.850030+0000","last_fullsized":"2024-09-11T08:11:59.850030+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:11:29.518632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:11:29.518632+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:30.606859+0000","scrub_duration":0.015872065000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":87,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.837262+0000","last_change":"2024-09-11T08:09:15.617419+0000","last_active":"2024-09-11T08:11:59.837262+0000","last_peered":"2024-09-11T08:11:59.837262+0000","last_clean":"2024-09-11T08:11:59.837262+0000","last_became_active":"2024-09-11T08:07:36.068493+0000","last_became_peered":"2024-09-11T08:07:36.068493+0000","last_unstale":"2024-09-11T08:11:59.837262+0000","last_undegraded":"2024-09-11T08:11:59.837262+0000","last_fullsized":"2024-09-11T08:11:59.837262+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:15.617360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:09:15.617360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014734049999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":138,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.851114+0000","last_change":"2024-09-11T08:10:37.122397+0000","last_active":"2024-09-11T08:11:59.851114+0000","last_peered":"2024-09-11T08:11:59.851114+0000","last_clean":"2024-09-11T08:11:59.851114+0000","last_became_active":"2024-09-11T08:09:07.878358+0000","last_became_peered":"2024-09-11T08:09:07.878358+0000","last_unstale":"2024-09-11T08:11:59.851114+0000","last_undegraded":"2024-09-11T08:11:59.851114+0000","last_fullsized":"2024-09-11T08:11:59.851114+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-11T08:10:37.122322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:10:37.122322+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:03.673311+0000","scrub_duration":0.0096164340000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":103,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.851159+0000","last_change":"2024-09-11T08:10:33.261203+0000","last_active":"2024-09-11T08:11:59.851159+0000","last_peered":"2024-09-11T08:11:59.851159+0000","last_clean":"2024-09-11T08:11:59.851159+0000","last_became_active":"2024-09-11T08:07:30.039673+0000","last_became_peered":"2024-09-11T08:07:30.039673+0000","last_unstale":"2024-09-11T08:11:59.851159+0000","last_undegraded":"2024-09-11T08:11:59.851159+0000","last_fullsized":"2024-09-11T08:11:59.851159+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:33.261145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:10:33.261145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004998476,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":160,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.849401+0000","last_change":"2024-09-11T08:10:36.942784+0000","last_active":"2024-09-11T08:11:59.849401+0000","last_peered":"2024-09-11T08:11:59.849401+0000","last_clean":"2024-09-11T08:11:59.849401+0000","last_became_active":"2024-09-11T08:07:34.060736+0000","last_became_peered":"2024-09-11T08:07:34.060736+0000","last_unstale":"2024-09-11T08:11:59.849401+0000","last_undegraded":"2024-09-11T08:11:59.849401+0000","last_fullsized":"2024-09-11T08:11:59.849401+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:10:36.942703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:10:36.942703+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.017554209000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":87,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.856810+0000","last_change":"2024-09-11T08:09:27.844582+0000","last_active":"2024-09-11T08:11:59.856810+0000","last_peered":"2024-09-11T08:11:59.856810+0000","last_clean":"2024-09-11T08:11:59.856810+0000","last_became_active":"2024-09-11T08:07:36.072359+0000","last_became_peered":"2024-09-11T08:07:36.072359+0000","last_unstale":"2024-09-11T08:11:59.856810+0000","last_undegraded":"2024-09-11T08:11:59.856810+0000","last_fullsized":"2024-09-11T08:11:59.856810+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:27.844523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:09:27.844523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014038360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":120,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.836396+0000","last_change":"2024-09-11T08:11:39.092534+0000","last_active":"2024-09-11T08:11:59.836396+0000","last_peered":"2024-09-11T08:11:59.836396+0000","last_clean":"2024-09-11T08:11:59.836396+0000","last_became_active":"2024-09-11T08:07:32.047981+0000","last_became_peered":"2024-09-11T08:07:32.047981+0000","last_unstale":"2024-09-11T08:11:59.836396+0000","last_undegraded":"2024-09-11T08:11:59.836396+0000","last_fullsized":"2024-09-11T08:11:59.836396+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-11T08:11:39.092431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:11:39.092431+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:51.100772+0000","scrub_duration":0.0069898629999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":103,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.836988+0000","last_change":"2024-09-11T08:10:36.134384+0000","last_active":"2024-09-11T08:11:59.836988+0000","last_peered":"2024-09-11T08:11:59.836988+0000","last_clean":"2024-09-11T08:11:59.836988+0000","last_became_active":"2024-09-11T08:07:30.043691+0000","last_became_peered":"2024-09-11T08:07:30.043691+0000","last_unstale":"2024-09-11T08:11:59.836988+0000","last_undegraded":"2024-09-11T08:11:59.836988+0000","last_fullsized":"2024-09-11T08:11:59.836988+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:36.134300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:10:36.134300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020719419999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":99,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.842464+0000","last_change":"2024-09-11T08:11:25.585872+0000","last_active":"2024-09-11T08:11:59.842464+0000","last_peered":"2024-09-11T08:11:59.842464+0000","last_clean":"2024-09-11T08:11:59.842464+0000","last_became_active":"2024-09-11T08:07:34.053137+0000","last_became_peered":"2024-09-11T08:07:34.053137+0000","last_unstale":"2024-09-11T08:11:59.842464+0000","last_undegraded":"2024-09-11T08:11:59.842464+0000","last_fullsized":"2024-09-11T08:11:59.842464+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:25.585537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:11:25.585537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:51.823113+0000","scrub_duration":0.015833313000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":87,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.845253+0000","last_change":"2024-09-11T08:10:30.195470+0000","last_active":"2024-09-11T08:11:59.845253+0000","last_peered":"2024-09-11T08:11:59.845253+0000","last_clean":"2024-09-11T08:11:59.845253+0000","last_became_active":"2024-09-11T08:07:36.076132+0000","last_became_peered":"2024-09-11T08:07:36.076132+0000","last_unstale":"2024-09-11T08:11:59.845253+0000","last_undegraded":"2024-09-11T08:11:59.845253+0000","last_fullsized":"2024-09-11T08:11:59.845253+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:30.195389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:10:30.195389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013816314999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":117,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.842693+0000","last_change":"2024-09-11T08:10:32.411685+0000","last_active":"2024-09-11T08:11:59.842693+0000","last_peered":"2024-09-11T08:11:59.842693+0000","last_clean":"2024-09-11T08:11:59.842693+0000","last_became_active":"2024-09-11T08:07:32.049368+0000","last_became_peered":"2024-09-11T08:07:32.049368+0000","last_unstale":"2024-09-11T08:11:59.842693+0000","last_undegraded":"2024-09-11T08:11:59.842693+0000","last_fullsized":"2024-09-11T08:11:59.842693+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:10:32.411521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:10:32.411521+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0076800619999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":113,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.851481+0000","last_change":"2024-09-11T08:11:39.203061+0000","last_active":"2024-09-11T08:11:59.851481+0000","last_peered":"2024-09-11T08:11:59.851481+0000","last_clean":"2024-09-11T08:11:59.851481+0000","last_became_active":"2024-09-11T08:07:30.037692+0000","last_became_peered":"2024-09-11T08:07:30.037692+0000","last_unstale":"2024-09-11T08:11:59.851481+0000","last_undegraded":"2024-09-11T08:11:59.851481+0000","last_fullsized":"2024-09-11T08:11:59.851481+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:39.202984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:11:39.202984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:51.576908+0000","scrub_duration":0.001252852,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":99,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.853925+0000","last_change":"2024-09-11T08:11:35.568457+0000","last_active":"2024-09-11T08:11:59.853925+0000","last_peered":"2024-09-11T08:11:59.853925+0000","last_clean":"2024-09-11T08:11:59.853925+0000","last_became_active":"2024-09-11T08:07:34.057064+0000","last_became_peered":"2024-09-11T08:07:34.057064+0000","last_unstale":"2024-09-11T08:11:59.853925+0000","last_undegraded":"2024-09-11T08:11:59.853925+0000","last_fullsized":"2024-09-11T08:11:59.853925+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:35.567888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:11:35.567888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:13:05.264013+0000","scrub_duration":0.029638675999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":97,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.846211+0000","last_change":"2024-09-11T08:11:22.043711+0000","last_active":"2024-09-11T08:11:59.846211+0000","last_peered":"2024-09-11T08:11:59.846211+0000","last_clean":"2024-09-11T08:11:59.846211+0000","last_became_active":"2024-09-11T08:07:36.077406+0000","last_became_peered":"2024-09-11T08:07:36.077406+0000","last_unstale":"2024-09-11T08:11:59.846211+0000","last_undegraded":"2024-09-11T08:11:59.846211+0000","last_fullsized":"2024-09-11T08:11:59.846211+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:22.043633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:09.973943+0000","last_clean_scrub_stamp":"2024-09-11T08:11:22.043633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:30.337221+0000","scrub_duration":0.0064821080000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":120,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.845362+0000","last_change":"2024-09-11T08:11:33.518272+0000","last_active":"2024-09-11T08:11:59.845362+0000","last_peered":"2024-09-11T08:11:59.845362+0000","last_clean":"2024-09-11T08:11:59.845362+0000","last_became_active":"2024-09-11T08:07:32.043276+0000","last_became_peered":"2024-09-11T08:07:32.043276+0000","last_unstale":"2024-09-11T08:11:59.845362+0000","last_undegraded":"2024-09-11T08:11:59.845362+0000","last_fullsized":"2024-09-11T08:11:59.845362+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-11T08:11:33.518136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:11:33.518136+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:43.052462+0000","scrub_duration":0.0058292969999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":102,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.849725+0000","last_change":"2024-09-11T08:10:23.978440+0000","last_active":"2024-09-11T08:11:59.849725+0000","last_peered":"2024-09-11T08:11:59.849725+0000","last_clean":"2024-09-11T08:11:59.849725+0000","last_became_active":"2024-09-11T08:07:30.049152+0000","last_became_peered":"2024-09-11T08:07:30.049152+0000","last_unstale":"2024-09-11T08:11:59.849725+0000","last_undegraded":"2024-09-11T08:11:59.849725+0000","last_fullsized":"2024-09-11T08:11:59.849725+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:23.978385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:10:23.978385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025160999999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":99,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.850622+0000","last_change":"2024-09-11T08:11:34.159199+0000","last_active":"2024-09-11T08:11:59.850622+0000","last_peered":"2024-09-11T08:11:59.850622+0000","last_clean":"2024-09-11T08:11:59.850622+0000","last_became_active":"2024-09-11T08:07:34.056935+0000","last_became_peered":"2024-09-11T08:07:34.056935+0000","last_unstale":"2024-09-11T08:11:59.850622+0000","last_undegraded":"2024-09-11T08:11:59.850622+0000","last_fullsized":"2024-09-11T08:11:59.850622+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:34.159053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:11:34.159053+0000","last_clean_scrub_stamp":"2024-09-11T08:11:34.159053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:35.241223+0000","scrub_duration":0.00130974,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":87,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.854061+0000","last_change":"2024-09-11T08:09:34.378812+0000","last_active":"2024-09-11T08:11:59.854061+0000","last_peered":"2024-09-11T08:11:59.854061+0000","last_clean":"2024-09-11T08:11:59.854061+0000","last_became_active":"2024-09-11T08:07:36.633368+0000","last_became_peered":"2024-09-11T08:07:36.633368+0000","last_unstale":"2024-09-11T08:11:59.854061+0000","last_undegraded":"2024-09-11T08:11:59.854061+0000","last_fullsized":"2024-09-11T08:11:59.854061+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:09:34.378734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:34.378734+0000","last_clean_scrub_stamp":"2024-09-11T08:09:34.378734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001593148,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":97,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.851940+0000","last_change":"2024-09-11T08:09:40.373717+0000","last_active":"2024-09-11T08:11:59.851940+0000","last_peered":"2024-09-11T08:11:59.851940+0000","last_clean":"2024-09-11T08:11:59.851940+0000","last_became_active":"2024-09-11T08:07:32.045728+0000","last_became_peered":"2024-09-11T08:07:32.045728+0000","last_unstale":"2024-09-11T08:11:59.851940+0000","last_undegraded":"2024-09-11T08:11:59.851940+0000","last_fullsized":"2024-09-11T08:11:59.851940+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'4","last_scrub_stamp":"2024-09-11T08:09:40.373649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:09:40.373649+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001798315,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":103,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.851950+0000","last_change":"2024-09-11T08:10:28.129119+0000","last_active":"2024-09-11T08:11:59.851950+0000","last_peered":"2024-09-11T08:11:59.851950+0000","last_clean":"2024-09-11T08:11:59.851950+0000","last_became_active":"2024-09-11T08:07:30.049409+0000","last_became_peered":"2024-09-11T08:07:30.049409+0000","last_unstale":"2024-09-11T08:11:59.851950+0000","last_undegraded":"2024-09-11T08:11:59.851950+0000","last_fullsized":"2024-09-11T08:11:59.851950+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:28.129035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:10:28.129035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.019365614999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":108,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.841403+0000","last_change":"2024-09-11T08:11:38.545774+0000","last_active":"2024-09-11T08:11:59.841403+0000","last_peered":"2024-09-11T08:11:59.841403+0000","last_clean":"2024-09-11T08:11:59.841403+0000","last_became_active":"2024-09-11T08:07:34.055082+0000","last_became_peered":"2024-09-11T08:07:34.055082+0000","last_unstale":"2024-09-11T08:11:59.841403+0000","last_undegraded":"2024-09-11T08:11:59.841403+0000","last_fullsized":"2024-09-11T08:11:59.841403+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:38.545717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:11:38.545717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:43.268646+0000","scrub_duration":0.0054392340000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":97,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.850826+0000","last_change":"2024-09-11T08:10:42.297248+0000","last_active":"2024-09-11T08:11:59.850826+0000","last_peered":"2024-09-11T08:11:59.850826+0000","last_clean":"2024-09-11T08:11:59.850826+0000","last_became_active":"2024-09-11T08:07:36.071669+0000","last_became_peered":"2024-09-11T08:07:36.071669+0000","last_unstale":"2024-09-11T08:11:59.850826+0000","last_undegraded":"2024-09-11T08:11:59.850826+0000","last_fullsized":"2024-09-11T08:11:59.850826+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:42.297166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:10:42.297166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:07.034868+0000","scrub_duration":0.0077635450000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":97,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.839422+0000","last_change":"2024-09-11T08:10:28.457397+0000","last_active":"2024-09-11T08:11:59.839422+0000","last_peered":"2024-09-11T08:11:59.839422+0000","last_clean":"2024-09-11T08:11:59.839422+0000","last_became_active":"2024-09-11T08:07:36.071468+0000","last_became_peered":"2024-09-11T08:07:36.071468+0000","last_unstale":"2024-09-11T08:11:59.839422+0000","last_undegraded":"2024-09-11T08:11:59.839422+0000","last_fullsized":"2024-09-11T08:11:59.839422+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:28.457304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:10:28.457304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0063908749999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":103,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.852027+0000","last_change":"2024-09-11T08:11:26.404087+0000","last_active":"2024-09-11T08:11:59.852027+0000","last_peered":"2024-09-11T08:11:59.852027+0000","last_clean":"2024-09-11T08:11:59.852027+0000","last_became_active":"2024-09-11T08:07:30.046702+0000","last_became_peered":"2024-09-11T08:07:30.046702+0000","last_unstale":"2024-09-11T08:11:59.852027+0000","last_undegraded":"2024-09-11T08:11:59.852027+0000","last_fullsized":"2024-09-11T08:11:59.852027+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:11:26.403944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:11:26.403944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:12:29.575666+0000","scrub_duration":0.0056165199999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":119,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.841959+0000","last_change":"2024-09-11T08:10:24.612372+0000","last_active":"2024-09-11T08:11:59.841959+0000","last_peered":"2024-09-11T08:11:59.841959+0000","last_clean":"2024-09-11T08:11:59.841959+0000","last_became_active":"2024-09-11T08:07:32.054656+0000","last_became_peered":"2024-09-11T08:07:32.054656+0000","last_unstale":"2024-09-11T08:11:59.841959+0000","last_undegraded":"2024-09-11T08:11:59.841959+0000","last_fullsized":"2024-09-11T08:11:59.841959+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-11T08:10:24.612282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:10:24.612282+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020463640000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":99,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-11T08:11:59.856957+0000","last_change":"2024-09-11T08:10:32.091677+0000","last_active":"2024-09-11T08:11:59.856957+0000","last_peered":"2024-09-11T08:11:59.856957+0000","last_clean":"2024-09-11T08:11:59.856957+0000","last_became_active":"2024-09-11T08:07:34.061453+0000","last_became_peered":"2024-09-11T08:07:34.061453+0000","last_unstale":"2024-09-11T08:11:59.856957+0000","last_undegraded":"2024-09-11T08:11:59.856957+0000","last_fullsized":"2024-09-11T08:11:59.856957+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:10:32.091605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:10:32.091605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0071709479999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":5814288408,"num_objects":88720,"num_object_clones":0,"num_object_copies":266160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88720,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160158,"num_write_kb":7964033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17443270656,"data_stored":17443258440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89955,"ondisk_log_size":89955,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":435,"ondisk_log_size":435,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":29,"num_read_kb":29,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":266,"num_read_kb":261,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266747,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2092872,"kb_used_data":1795384,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":91652280,"statfs":{"total":95995035648,"available":93851934720,"internally_reserved":0,"allocated":1838473216,"data_stored":1838116838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:11:23 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.52300000000000002,"15min":0.496},"min":{"1min":0.20300000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":3.4119999999999999,"5min":3.4119999999999999,"15min":3.4119999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.58499999999999996,"15min":0.51700000000000002},"min":{"1min":0.251,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.3279999999999998,"5min":2.3279999999999998,"15min":2.3279999999999998},"last":2.4009999999999998}]},{"osd":1,"last update":"Wed Sep 11 08:11:23 2024","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.59499999999999997,"15min":0.52700000000000002},"min":{"1min":0.23000000000000001,"5min":0.216,"15min":0.216},"max":{"1min":3.423,"5min":3.423,"15min":3.423},"last":1.9219999999999999},{"interface":"front","average":{"1min":0.997,"5min":0.63300000000000001,"15min":0.56299999999999994},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":3.4409999999999998,"5min":3.4409999999999998,"15min":3.4409999999999998},"last":1.956}]},{"osd":2,"last update":"Wed Sep 11 08:11:23 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":0.67600000000000005,"15min":0.54600000000000004},"min":{"1min":0.26500000000000001,"5min":0.254,"15min":0.254},"max":{"1min":4.1589999999999998,"5min":5.5919999999999996,"15min":5.5919999999999996},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.52100000000000002,"15min":0.50700000000000001},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":1.302,"5min":2.677,"15min":2.677},"last":0.42899999999999999}]},{"osd":3,"last update":"Wed Sep 11 08:11:23 2024","interfaces":[{"interface":"back","average":{"1min":2.278,"5min":1.03,"15min":0.94799999999999995},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":10.818,"5min":10.818,"15min":10.818},"last":0.48399999999999999},{"interface":"front","average":{"1min":1.329,"5min":0.66900000000000004,"15min":0.54400000000000004},"min":{"1min":0.29999999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":5.7080000000000002,"5min":5.7080000000000002,"15min":5.7080000000000002},"last":0.52200000000000002}]},{"osd":4,"last update":"Wed Sep 11 08:11:23 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.41699999999999998,"15min":0.40699999999999997},"min":{"1min":0.215,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":1.0580000000000001,"5min":1.7050000000000001,"15min":1.7050000000000001},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.42499999999999999,"15min":0.41899999999999998},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.92800000000000005,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":0.50700000000000001}]},{"osd":5,"last update":"Wed Sep 11 08:11:23 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.40600000000000003,"15min":0.41799999999999998},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.60799999999999998,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.503,"5min":0.47399999999999998,"15min":0.46200000000000002},"min":{"1min":0.26700000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.1000000000000001,"5min":1.6799999999999999,"15min":1.6799999999999999},"last":0.316}]},{"osd":6,"last update":"Wed Sep 11 08:11:23 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50700000000000001,"15min":0.499},"min":{"1min":0.29099999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.014,"5min":1.819,"15min":1.819},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.497,"5min":0.48299999999999998,"15min":0.47199999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.84699999999999998,"5min":1.492,"15min":1.492},"last":0.79600000000000004}]}]},{"osd":6,"up_from":45,"seq":193273528384,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2086780,"kb_used_data":1776812,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":91658372,"statfs":{"total":95995035648,"available":93858172928,"internally_reserved":0,"allocated":1819455488,"data_stored":1819111245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:12:02 2024","interfaces":[{"interface":"back","average":{"1min":4.085,"5min":1.264,"15min":0.753},"min":{"1min":0.22900000000000001,"5min":0.185,"15min":0.185},"max":{"1min":22.079999999999998,"5min":22.079999999999998,"15min":22.079999999999998},"last":0.54100000000000004},{"interface":"front","average":{"1min":2.6699999999999999,"5min":1.056,"15min":0.68899999999999995},"min":{"1min":0.309,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":22.030000000000001,"5min":22.030000000000001,"15min":22.030000000000001},"last":3.2509999999999999}]},{"osd":1,"last update":"Wed Sep 11 08:12:02 2024","interfaces":[{"interface":"back","average":{"1min":1.522,"5min":0.71499999999999997,"15min":0.55700000000000005},"min":{"1min":0.254,"5min":0.217,"15min":0.217},"max":{"1min":6.0110000000000001,"5min":6.0110000000000001,"15min":6.0110000000000001},"last":1.542},{"interface":"front","average":{"1min":1.5129999999999999,"5min":0.74199999999999999,"15min":0.58499999999999996},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":6.0389999999999997,"5min":6.0389999999999997,"15min":6.0389999999999997},"last":0.248}]},{"osd":2,"last update":"Wed Sep 11 08:12:02 2024","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":0.69999999999999996,"15min":0.56899999999999995},"min":{"1min":0.26200000000000001,"5min":0.25,"15min":0.25},"max":{"1min":3.7330000000000001,"5min":3.7330000000000001,"15min":3.7330000000000001},"last":0.27800000000000002},{"interface":"front","average":{"1min":2.0990000000000002,"5min":0.91800000000000004,"15min":0.627},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":13.346,"5min":13.346,"15min":13.346},"last":2.105}]},{"osd":3,"last update":"Wed Sep 11 08:12:02 2024","interfaces":[{"interface":"back","average":{"1min":3.5179999999999998,"5min":1.3080000000000001,"15min":0.78700000000000003},"min":{"1min":0.38100000000000001,"5min":0.308,"15min":0.308},"max":{"1min":36.061,"5min":36.061,"15min":36.061},"last":1.748},{"interface":"front","average":{"1min":3.3780000000000001,"5min":1.244,"15min":0.77300000000000002},"min":{"1min":0.40799999999999997,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":36.093000000000004,"5min":36.093000000000004,"15min":36.093000000000004},"last":1.73}]},{"osd":4,"last update":"Wed Sep 11 08:12:02 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.42599999999999999,"15min":0.39900000000000002},"min":{"1min":0.219,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":1.425,"5min":1.425,"15min":1.425},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.46300000000000002,"15min":0.42699999999999999},"min":{"1min":0.251,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.407,"5min":1.407,"15min":1.407},"last":0.55300000000000005}]},{"osd":5,"last update":"Wed Sep 11 08:12:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.51400000000000001,"15min":0.48799999999999999},"min":{"1min":0.32900000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.3859999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.503,"15min":0.441},"min":{"1min":0.23000000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.8450000000000002,"5min":2.8450000000000002,"15min":2.8450000000000002},"last":0.51000000000000001}]},{"osd":7,"last update":"Wed Sep 11 08:11:27 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.49299999999999999,"15min":0.47299999999999998},"min":{"1min":0.31900000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.359,"5min":1.359,"15min":1.359},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.46899999999999997,"15min":0.44500000000000001},"min":{"1min":0.28199999999999997,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":0.41499999999999998}]}]},{"osd":5,"up_from":38,"seq":163208757317,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1736312,"kb_used_data":1430504,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":92008840,"statfs":{"total":95995035648,"available":94217052160,"internally_reserved":0,"allocated":1464836096,"data_stored":1464494942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:11:40 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":0.73299999999999998,"15min":0.59499999999999997},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.4299999999999997,"5min":6.343,"15min":6.343},"last":0.53400000000000003},{"interface":"front","average":{"1min":1.0269999999999999,"5min":0.71299999999999997,"15min":0.57099999999999995},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.5150000000000001,"5min":4.6319999999999997,"15min":4.6319999999999997},"last":0.84499999999999997}]},{"osd":1,"last update":"Wed Sep 11 08:11:40 2024","interfaces":[{"interface":"back","average":{"1min":1.694,"5min":0.98299999999999998,"15min":1.2330000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":12.847,"5min":16.004999999999999,"15min":16.004999999999999},"last":0.26400000000000001},{"interface":"front","average":{"1min":1.758,"5min":0.76300000000000001,"15min":0.59299999999999997},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":13.215999999999999,"5min":13.215999999999999,"15min":13.215999999999999},"last":0.82799999999999996}]},{"osd":2,"last update":"Wed Sep 11 08:11:40 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":0.83399999999999996,"15min":1.1990000000000001},"min":{"1min":0.308,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":5.1669999999999998,"5min":16.018000000000001,"15min":16.018000000000001},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.63500000000000001,"15min":0.55900000000000005},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":4.7469999999999999,"5min":4.7469999999999999,"15min":4.7469999999999999},"last":0.72099999999999997}]},{"osd":3,"last update":"Wed Sep 11 08:11:40 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":0.78000000000000003,"15min":0.61199999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":4.7300000000000004,"5min":6.3230000000000004,"15min":6.3230000000000004},"last":0.69899999999999995},{"interface":"front","average":{"1min":1.242,"5min":0.75,"15min":0.59599999999999997},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":7.3159999999999998,"5min":7.3159999999999998,"15min":7.3159999999999998},"last":0.51300000000000001}]},{"osd":4,"last update":"Wed Sep 11 08:11:40 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.46300000000000002,"15min":0.41499999999999998},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":2.0779999999999998,"5min":2.0779999999999998,"15min":2.0779999999999998},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.48299999999999998,"15min":0.41499999999999998},"min":{"1min":0.26700000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":2.3090000000000002,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":0.65500000000000003}]},{"osd":6,"last update":"Wed Sep 11 08:12:02 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.46200000000000002,"15min":0.433},"min":{"1min":0.26200000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.52400000000000002,"15min":0.46899999999999997},"min":{"1min":0.30299999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":2.02,"5min":2.02,"15min":2.02},"last":0.308}]},{"osd":7,"last update":"Wed Sep 11 08:11:28 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.52700000000000002,"15min":0.50900000000000001},"min":{"1min":0.32000000000000001,"5min":0.307,"15min":0.307},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.55200000000000005,"15min":0.501},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":2.448,"5min":2.448,"15min":2.448},"last":0.74099999999999999}]}]},{"osd":4,"up_from":31,"seq":133143986250,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2103180,"kb_used_data":1801532,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":91641972,"statfs":{"total":95995035648,"available":93841379328,"internally_reserved":0,"allocated":1844768768,"data_stored":1844408448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:11:13 2024","interfaces":[{"interface":"back","average":{"1min":3.1589999999999998,"5min":1.6259999999999999,"15min":0.88400000000000001},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":26.754999999999999,"5min":48.728999999999999,"15min":48.728999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":2.4289999999999998,"5min":1.4079999999999999,"15min":0.82999999999999996},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":27.055,"5min":49.168999999999997,"15min":49.168999999999997},"last":2.2989999999999999}]},{"osd":1,"last update":"Wed Sep 11 08:11:13 2024","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":0.73899999999999999,"15min":0.59099999999999997},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":2.6669999999999998,"5min":13.207000000000001,"15min":13.207000000000001},"last":1.726},{"interface":"front","average":{"1min":1.163,"5min":0.64700000000000002,"15min":0.57099999999999995},"min":{"1min":0.25600000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":3.2679999999999998,"5min":3.2679999999999998,"15min":3.2679999999999998},"last":0.95499999999999996}]},{"osd":2,"last update":"Wed Sep 11 08:11:13 2024","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":0.64000000000000001,"15min":0.56799999999999995},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":3.9750000000000001,"5min":3.9750000000000001,"15min":3.9750000000000001},"last":0.48599999999999999},{"interface":"front","average":{"1min":1.6679999999999999,"5min":0.74399999999999999,"15min":0.60099999999999998},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":5.5670000000000002,"5min":5.5670000000000002,"15min":5.5670000000000002},"last":0.28399999999999997}]},{"osd":3,"last update":"Wed Sep 11 08:11:13 2024","interfaces":[{"interface":"back","average":{"1min":2.7599999999999998,"5min":1.274,"15min":0.77000000000000002},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":26.884,"5min":26.884,"15min":26.884},"last":0.36499999999999999},{"interface":"front","average":{"1min":2.9710000000000001,"5min":1.1819999999999999,"15min":0.747},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":26.783000000000001,"5min":26.783000000000001,"15min":26.783000000000001},"last":0.65500000000000003}]},{"osd":5,"last update":"Wed Sep 11 08:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.40999999999999998,"15min":0.38},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":1.151,"5min":1.151,"15min":1.151},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.439,"15min":0.39000000000000001},"min":{"1min":0.28199999999999997,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":1.9630000000000001,"5min":1.9630000000000001,"15min":1.9630000000000001},"last":0.33500000000000002}]},{"osd":6,"last update":"Wed Sep 11 08:11:00 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.40500000000000003,"15min":0.38800000000000001},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.61399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.48499999999999999,"15min":0.44700000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.42899999999999999}]},{"osd":7,"last update":"Wed Sep 11 08:11:28 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.49099999999999999,"15min":0.46300000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.097,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.314},{"interface":"front","average":{"1min":0.627,"5min":0.498,"15min":0.46600000000000003},"min":{"1min":0.32700000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.931,"5min":1.931,"15min":1.931},"last":0.55600000000000005}]}]},{"osd":3,"up_from":25,"seq":107374182479,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3523676,"kb_used_data":3213512,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":90221476,"statfs":{"total":95995035648,"available":92386791424,"internally_reserved":0,"allocated":3290636288,"data_stored":3290263151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:12:01 2024","interfaces":[{"interface":"back","average":{"1min":2.4529999999999998,"5min":0.96999999999999997,"15min":0.57599999999999996},"min":{"1min":0.438,"5min":0.191,"15min":0.191},"max":{"1min":10.651,"5min":10.651,"15min":10.651},"last":0.998},{"interface":"front","average":{"1min":2.9470000000000001,"5min":1.0629999999999999,"15min":0.63800000000000001},"min":{"1min":0.59799999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":9.5510000000000002,"5min":9.5510000000000002,"15min":9.5510000000000002},"last":4.4329999999999998}]},{"osd":1,"last update":"Wed Sep 11 08:12:01 2024","interfaces":[{"interface":"back","average":{"1min":1.746,"5min":0.80000000000000004,"15min":0.56499999999999995},"min":{"1min":0.308,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":6.6189999999999998,"5min":6.6189999999999998,"15min":6.6189999999999998},"last":1.792},{"interface":"front","average":{"1min":2.6869999999999998,"5min":1.153,"15min":0.67500000000000004},"min":{"1min":0.29399999999999998,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":11.917999999999999,"5min":11.917999999999999,"15min":11.917999999999999},"last":2.7909999999999999}]},{"osd":2,"last update":"Wed Sep 11 08:12:01 2024","interfaces":[{"interface":"back","average":{"1min":2.6030000000000002,"5min":1.0229999999999999,"15min":0.64500000000000002},"min":{"1min":0.23699999999999999,"5min":0.218,"15min":0.218},"max":{"1min":11.882999999999999,"5min":11.882999999999999,"15min":11.882999999999999},"last":1.202},{"interface":"front","average":{"1min":2.4710000000000001,"5min":1.022,"15min":0.624},"min":{"1min":0.29599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":10.625999999999999,"5min":10.625999999999999,"15min":10.625999999999999},"last":1.2130000000000001}]},{"osd":4,"last update":"Wed Sep 11 08:11:18 2024","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":0.73699999999999999,"15min":0.61199999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":5.157,"5min":5.157,"15min":5.157},"last":1.226},{"interface":"front","average":{"1min":1.4530000000000001,"5min":0.70399999999999996,"15min":0.59399999999999997},"min":{"1min":0.312,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":10.563000000000001,"5min":10.563000000000001,"15min":10.563000000000001},"last":1.1819999999999999}]},{"osd":5,"last update":"Wed Sep 11 08:11:39 2024","interfaces":[{"interface":"back","average":{"1min":1.899,"5min":0.85199999999999998,"15min":0.64200000000000002},"min":{"1min":0.26600000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":10.496,"5min":10.496,"15min":10.496},"last":1.242},{"interface":"front","average":{"1min":1.331,"5min":0.96499999999999997,"15min":0.68600000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":6.4279999999999999,"5min":16.757000000000001,"15min":16.757000000000001},"last":1.823}]},{"osd":6,"last update":"Wed Sep 11 08:11:03 2024","interfaces":[{"interface":"back","average":{"1min":1.329,"5min":0.89700000000000002,"15min":0.65100000000000002},"min":{"1min":0.33400000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":6.4489999999999998,"5min":16.789000000000001,"15min":16.789000000000001},"last":1.8899999999999999},{"interface":"front","average":{"1min":2.0289999999999999,"5min":1.0369999999999999,"15min":0.71899999999999997},"min":{"1min":0.40899999999999997,"5min":0.373,"15min":0.373},"max":{"1min":11.122999999999999,"5min":16.748000000000001,"15min":16.748000000000001},"last":1.3080000000000001}]},{"osd":7,"last update":"Wed Sep 11 08:11:29 2024","interfaces":[{"interface":"back","average":{"1min":1.784,"5min":0.82099999999999995,"15min":0.64100000000000001},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":10.538,"5min":10.538,"15min":10.538},"last":1.385},{"interface":"front","average":{"1min":1.6919999999999999,"5min":0.84799999999999998,"15min":0.67200000000000004},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":11.098000000000001,"5min":11.098000000000001,"15min":11.098000000000001},"last":1.8600000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411410,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2435256,"kb_used_data":2137768,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":91309896,"statfs":{"total":95995035648,"available":93501333504,"internally_reserved":0,"allocated":2189074432,"data_stored":2188733190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:11:33 2024","interfaces":[{"interface":"back","average":{"1min":2.157,"5min":0.86199999999999999,"15min":0.54000000000000004},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":6.468,"5min":6.468,"15min":6.468},"last":2.7629999999999999},{"interface":"front","average":{"1min":2.2360000000000002,"5min":0.85199999999999998,"15min":0.52100000000000002},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":6.9080000000000004,"5min":6.9080000000000004,"15min":6.9080000000000004},"last":1.004}]},{"osd":1,"last update":"Wed Sep 11 08:11:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7709999999999999,"5min":0.83499999999999996,"15min":0.52700000000000002},"min":{"1min":0.26100000000000001,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":6.3499999999999996,"5min":6.3499999999999996,"15min":6.3499999999999996},"last":2.968},{"interface":"front","average":{"1min":1.7350000000000001,"5min":0.81000000000000005,"15min":0.52400000000000002},"min":{"1min":0.23200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":5.6970000000000001,"5min":5.6970000000000001,"15min":5.6970000000000001},"last":4.2439999999999998}]},{"osd":3,"last update":"Wed Sep 11 08:11:55 2024","interfaces":[{"interface":"back","average":{"1min":2.1549999999999998,"5min":0.89400000000000002,"15min":0.60899999999999999},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":6.3819999999999997,"5min":6.3819999999999997,"15min":6.3819999999999997},"last":2.9220000000000002},{"interface":"front","average":{"1min":2.266,"5min":0.92600000000000005,"15min":0.623},"min":{"1min":0.28100000000000003,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":8.3219999999999992,"5min":8.3219999999999992,"15min":8.3219999999999992},"last":0.40000000000000002}]},{"osd":4,"last update":"Wed Sep 11 08:11:16 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":0.74399999999999999,"15min":0.60199999999999998},"min":{"1min":0.311,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":7.806,"5min":7.806,"15min":7.806},"last":3.96},{"interface":"front","average":{"1min":1.4510000000000001,"5min":0.872,"15min":0.68300000000000005},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":6.8200000000000003,"5min":12.129,"15min":12.129},"last":0.435}]},{"osd":5,"last update":"Wed Sep 11 08:11:45 2024","interfaces":[{"interface":"back","average":{"1min":1.7749999999999999,"5min":0.89700000000000002,"15min":0.68899999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":6.7709999999999999,"5min":6.7709999999999999,"15min":6.7709999999999999},"last":0.45600000000000002},{"interface":"front","average":{"1min":1.329,"5min":0.76200000000000001,"15min":0.65700000000000003},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":6.4050000000000002,"5min":6.4050000000000002,"15min":6.4050000000000002},"last":2.8359999999999999}]},{"osd":6,"last update":"Wed Sep 11 08:12:00 2024","interfaces":[{"interface":"back","average":{"1min":1.651,"5min":0.85099999999999998,"15min":0.68300000000000005},"min":{"1min":0.32600000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":6.4480000000000004,"5min":6.4480000000000004,"15min":6.4480000000000004},"last":2.8079999999999998},{"interface":"front","average":{"1min":2.1179999999999999,"5min":1.0089999999999999,"15min":0.73699999999999999},"min":{"1min":0.34200000000000003,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":6.71,"5min":6.71,"15min":6.71},"last":0.47499999999999998}]},{"osd":7,"last update":"Wed Sep 11 08:11:27 2024","interfaces":[{"interface":"back","average":{"1min":1.45,"5min":0.81399999999999995,"15min":0.65400000000000003},"min":{"1min":0.34799999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":7.7560000000000002,"5min":7.7560000000000002,"15min":7.7560000000000002},"last":3.9049999999999998},{"interface":"front","average":{"1min":2.0409999999999999,"5min":1.0309999999999999,"15min":0.74399999999999999},"min":{"1min":0.40100000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":6.4260000000000002,"5min":12.115,"15min":12.115},"last":2.8580000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574936,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2429984,"kb_used_data":2123984,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":91315168,"statfs":{"total":95995035648,"available":93506732032,"internally_reserved":0,"allocated":2174959616,"data_stored":2174578454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:11:10 2024","interfaces":[{"interface":"back","average":{"1min":3.6480000000000001,"5min":1.0580000000000001,"15min":0.54400000000000004},"min":{"1min":0.245,"5min":0.22600000000000001,"15min":0.16300000000000001},"max":{"1min":32.478000000000002,"5min":32.478000000000002,"15min":32.478000000000002},"last":1.6160000000000001},{"interface":"front","average":{"1min":4.6200000000000001,"5min":1.2849999999999999,"15min":0.63},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20300000000000001},"max":{"1min":35.459000000000003,"5min":35.459000000000003,"15min":35.459000000000003},"last":0.40500000000000003}]},{"osd":2,"last update":"Wed Sep 11 08:11:39 2024","interfaces":[{"interface":"back","average":{"1min":2.4300000000000002,"5min":0.88500000000000001,"15min":0.52300000000000002},"min":{"1min":0.38200000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":8.3819999999999997,"5min":8.3819999999999997,"15min":8.3819999999999997},"last":1.4850000000000001},{"interface":"front","average":{"1min":3.4700000000000002,"5min":1.175,"15min":0.61499999999999999},"min":{"1min":0.30499999999999999,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":10.276,"5min":10.276,"15min":10.276},"last":0.36599999999999999}]},{"osd":3,"last update":"Wed Sep 11 08:11:52 2024","interfaces":[{"interface":"back","average":{"1min":3.2719999999999998,"5min":1.8140000000000001,"15min":0.88800000000000001},"min":{"1min":0.254,"5min":0.246,"15min":0.246},"max":{"1min":10.102,"5min":32.759,"15min":32.759},"last":1.9039999999999999},{"interface":"front","average":{"1min":2.6699999999999999,"5min":1.5269999999999999,"15min":0.79900000000000004},"min":{"1min":0.29199999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":8.4030000000000005,"5min":32.444000000000003,"15min":32.444000000000003},"last":1.526}]},{"osd":4,"last update":"Wed Sep 11 08:11:17 2024","interfaces":[{"interface":"back","average":{"1min":2.1640000000000001,"5min":0.91500000000000004,"15min":0.66600000000000004},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":9.0139999999999993,"5min":9.0139999999999993,"15min":9.0139999999999993},"last":0.62},{"interface":"front","average":{"1min":1.873,"5min":0.83699999999999997,"15min":0.64900000000000002},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":8.2360000000000007,"5min":8.2360000000000007,"15min":8.2360000000000007},"last":1.552}]},{"osd":5,"last update":"Wed Sep 11 08:11:43 2024","interfaces":[{"interface":"back","average":{"1min":2.2389999999999999,"5min":1.103,"15min":0.71899999999999997},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":8.6769999999999996,"5min":9.0470000000000006,"15min":9.0470000000000006},"last":0.59099999999999997},{"interface":"front","average":{"1min":2.5640000000000001,"5min":1.1040000000000001,"15min":0.71799999999999997},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":8.9860000000000007,"5min":11.967000000000001,"15min":11.967000000000001},"last":0.45500000000000002}]},{"osd":6,"last update":"Wed Sep 11 08:11:57 2024","interfaces":[{"interface":"back","average":{"1min":2.1190000000000002,"5min":1.149,"15min":0.75700000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":7.3230000000000004,"5min":11.935,"15min":11.935},"last":0.48499999999999999},{"interface":"front","average":{"1min":2.1200000000000001,"5min":1.046,"15min":0.75},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":8.3339999999999996,"5min":8.3339999999999996,"15min":8.3339999999999996},"last":1.575}]},{"osd":7,"last update":"Wed Sep 11 08:11:22 2024","interfaces":[{"interface":"back","average":{"1min":2.3690000000000002,"5min":0.96599999999999997,"15min":0.73699999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":8.2929999999999993,"5min":8.2929999999999993,"15min":8.2929999999999993},"last":1.5960000000000001},{"interface":"front","average":{"1min":2.4079999999999999,"5min":0.97999999999999998,"15min":0.70399999999999996},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":9.0749999999999993,"5min":9.0749999999999993,"15min":9.0749999999999993},"last":0.64200000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738460,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3169752,"kb_used_data":2859720,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":90575400,"statfs":{"total":95995035648,"available":92749209600,"internally_reserved":0,"allocated":2928353280,"data_stored":2927981488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 11 08:11:11 2024","interfaces":[{"interface":"back","average":{"1min":2.0939999999999999,"5min":0.72599999999999998,"15min":0.443},"min":{"1min":0.189,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":6.0830000000000002,"5min":6.0830000000000002,"15min":6.0830000000000002},"last":1.377},{"interface":"front","average":{"1min":1.8500000000000001,"5min":0.67200000000000004,"15min":0.442},"min":{"1min":0.254,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.0469999999999997,"5min":6.0469999999999997,"15min":6.0469999999999997},"last":0.50700000000000001}]},{"osd":2,"last update":"Wed Sep 11 08:11:36 2024","interfaces":[{"interface":"back","average":{"1min":2.161,"5min":0.85699999999999998,"15min":0.51500000000000001},"min":{"1min":0.375,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.1029999999999998,"5min":6.1029999999999998,"15min":6.1029999999999998},"last":2.3140000000000001},{"interface":"front","average":{"1min":2.5470000000000002,"5min":0.94699999999999995,"15min":0.55900000000000005},"min":{"1min":0.34300000000000003,"5min":0.217,"15min":0.217},"max":{"1min":6.3719999999999999,"5min":6.3719999999999999,"15min":6.3719999999999999},"last":1.5409999999999999}]},{"osd":3,"last update":"Wed Sep 11 08:11:57 2024","interfaces":[{"interface":"back","average":{"1min":2.5059999999999998,"5min":1.359,"15min":0.746},"min":{"1min":0.314,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":6.3970000000000002,"5min":7.9699999999999998,"15min":7.9699999999999998},"last":2.7839999999999998},{"interface":"front","average":{"1min":2.2130000000000001,"5min":1.0549999999999999,"15min":0.63600000000000001},"min":{"1min":0.25800000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":6.4829999999999997,"5min":6.923,"15min":6.923},"last":0.56000000000000005}]},{"osd":4,"last update":"Wed Sep 11 08:11:17 2024","interfaces":[{"interface":"back","average":{"1min":1.7150000000000001,"5min":0.78700000000000003,"15min":0.625},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":6.726,"5min":6.726,"15min":6.726},"last":1.228},{"interface":"front","average":{"1min":1.76,"5min":0.78800000000000003,"15min":0.64800000000000002},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":5.6379999999999999,"5min":5.6379999999999999,"15min":5.6379999999999999},"last":1.3460000000000001}]},{"osd":5,"last update":"Wed Sep 11 08:11:45 2024","interfaces":[{"interface":"back","average":{"1min":1.841,"5min":0.93999999999999995,"15min":0.70199999999999996},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":6.1799999999999997,"5min":6.1799999999999997,"15min":6.1799999999999997},"last":1.3999999999999999},{"interface":"front","average":{"1min":1.4850000000000001,"5min":0.92300000000000004,"15min":0.67500000000000004},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":6.3949999999999996,"5min":6.7939999999999996,"15min":6.7939999999999996},"last":0.66700000000000004}]},{"osd":6,"last update":"Wed Sep 11 08:11:01 2024","interfaces":[{"interface":"back","average":{"1min":1.732,"5min":0.80200000000000005,"15min":0.68200000000000005},"min":{"1min":0.35799999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":6.8330000000000002,"5min":6.8330000000000002,"15min":6.8330000000000002},"last":1.4390000000000001},{"interface":"front","average":{"1min":2.0289999999999999,"5min":0.89400000000000002,"15min":0.66800000000000004},"min":{"1min":0.46700000000000003,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":6.1520000000000001,"5min":6.1520000000000001,"15min":6.1520000000000001},"last":1.0009999999999999}]},{"osd":7,"last update":"Wed Sep 11 08:11:24 2024","interfaces":[{"interface":"back","average":{"1min":1.5960000000000001,"5min":0.89000000000000001,"15min":0.70699999999999996},"min":{"1min":0.42199999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":6.2839999999999998,"5min":6.8650000000000002,"15min":6.8650000000000002},"last":0.53700000000000003},{"interface":"front","average":{"1min":1.8240000000000001,"5min":0.93000000000000005,"15min":0.69399999999999995},"min":{"1min":0.43099999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":6.1719999999999997,"5min":6.1719999999999997,"15min":6.1719999999999997},"last":0.89900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2899382272,"data_stored":2899378200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2169901056,"data_stored":2169896984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2179072000,"data_stored":2179072000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3268214784,"data_stored":3268214784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1825767424,"data_stored":1825767424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1462960128,"data_stored":1462960128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1814630400,"data_stored":1814626328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1823342592,"data_stored":1823342592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T08:12:09.056 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd pool set unique_pool_0 pgp_num 16 2024-09-11T08:12:09.068 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:08 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:12:09.078 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:09.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:09.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:09.174 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:12:09.180 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:12:09.207 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:09.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:09.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:12:09.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:08 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:12:09.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:08 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:12:09.309 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:09.368 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:09.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:09.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:09.470 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:12:09.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:09.512 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:09.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:12:09.613 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:09.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:09.647 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:09.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:09.748 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:12:09.788 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:09.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:09.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:12:09.890 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:09.925 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:09.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:09.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:10.028 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:12:10.054 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:10.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:10.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:12:10.156 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:10.182 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:10.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:10.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:10.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:09 smithi150 ceph-mon[29276]: pgmap v185: 148 pgs: 148 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s 2024-09-11T08:12:10.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:09 smithi150 ceph-mon[39612]: pgmap v185: 148 pgs: 148 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s 2024-09-11T08:12:10.284 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:10.323 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:10.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:09 smithi181 ceph-mon[32701]: pgmap v185: 148 pgs: 148 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s 2024-09-11T08:12:10.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:10.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:10.425 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:10.464 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:10.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:10.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:10.566 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:12:10.606 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:10.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:10.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:12:10.708 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:10.744 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:10.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:10.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:10.848 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:12:10.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:10.888 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:10.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:12:10.960 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-11T08:12:10.961 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-09-11T08:12:10.961 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-09-11T08:12:10.961 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-09-11T08:12:10.961 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.961 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14790.0:6869 7.1 7:86fe17fb:::benchmark_data_smithi150_78508_object6868:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-11T08:12:10.961 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:10:00.103168+0000", 2024-09-11T08:12:10.961 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 130.85391781300001, 2024-09-11T08:12:10.961 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.047916186, 2024-09-11T08:12:10.961 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:12:10.961 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:12:10.961 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:12:10.961 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14790", 2024-09-11T08:12:10.962 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:12:10.962 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 6869 2024-09-11T08:12:10.962 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.962 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:12:10.962 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.962 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:12:10.962 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:00.103168+0000", 2024-09-11T08:12:10.962 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:12:10.962 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.962 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.962 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:12:10.962 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:00.103168+0000", 2024-09-11T08:12:10.962 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:12:10.962 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.963 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.963 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:12:10.963 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:00.103167+0000", 2024-09-11T08:12:10.963 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:12:10.963 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.963 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.963 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:12:10.963 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:00.103226+0000", 2024-09-11T08:12:10.963 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8870999999999999e-05 2024-09-11T08:12:10.963 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.963 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.963 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:12:10.963 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:00.103234+0000", 2024-09-11T08:12:10.964 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9480000000000008e-06 2024-09-11T08:12:10.964 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.964 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.964 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:12:10.964 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:00.103239+0000", 2024-09-11T08:12:10.964 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1749999999999996e-06 2024-09-11T08:12:10.964 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.964 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.964 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:12:10.964 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:00.103270+0000", 2024-09-11T08:12:10.964 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1371999999999997e-05 2024-09-11T08:12:10.964 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.965 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.965 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:12:10.965 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:00.103339+0000", 2024-09-11T08:12:10.965 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9079999999999996e-05 2024-09-11T08:12:10.965 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.965 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.965 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3,6", 2024-09-11T08:12:10.965 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:00.103414+0000", 2024-09-11T08:12:10.965 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5063999999999998e-05 2024-09-11T08:12:10.965 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.965 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.965 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:12:10.965 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:00.103806+0000", 2024-09-11T08:12:10.965 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00039166100000000002 2024-09-11T08:12:10.966 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.966 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.966 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:12:10.966 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:00.104856+0000", 2024-09-11T08:12:10.966 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001050187 2024-09-11T08:12:10.966 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.966 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.966 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:12:10.966 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:00.150997+0000", 2024-09-11T08:12:10.966 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.046140558999999998 2024-09-11T08:12:10.966 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.966 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.966 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:12:10.967 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:00.151047+0000", 2024-09-11T08:12:10.967 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0133999999999998e-05 2024-09-11T08:12:10.967 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.967 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.967 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:12:10.967 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:00.151084+0000", 2024-09-11T08:12:10.967 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6927000000000001e-05 2024-09-11T08:12:10.967 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:12:10.967 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:12:10.967 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:12:10.967 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.967 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.967 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14790.0:6875 7.1 7:8f2c0a0a:::benchmark_data_smithi150_78508_object6874:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-11T08:12:10.967 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:10:00.105117+0000", 2024-09-11T08:12:10.968 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 130.85196862000001, 2024-09-11T08:12:10.968 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.046097549000000002, 2024-09-11T08:12:10.968 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:12:10.968 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:12:10.968 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:12:10.968 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14790", 2024-09-11T08:12:10.968 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:12:10.968 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 6875 2024-09-11T08:12:10.968 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.968 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:12:10.968 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.968 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:12:10.968 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:00.105117+0000", 2024-09-11T08:12:10.969 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:12:10.969 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.969 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.969 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:12:10.969 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:00.105117+0000", 2024-09-11T08:12:10.969 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:12:10.969 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.969 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.969 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:12:10.969 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:00.105116+0000", 2024-09-11T08:12:10.969 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-09-11T08:12:10.969 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.969 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.970 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:12:10.970 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:00.105172+0000", 2024-09-11T08:12:10.970 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6227999999999998e-05 2024-09-11T08:12:10.970 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.970 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.970 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:12:10.970 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:00.105180+0000", 2024-09-11T08:12:10.970 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2660000000000004e-06 2024-09-11T08:12:10.970 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.970 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.970 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:12:10.970 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:00.105185+0000", 2024-09-11T08:12:10.970 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0830000000000003e-06 2024-09-11T08:12:10.970 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.970 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.971 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:12:10.971 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:00.105211+0000", 2024-09-11T08:12:10.971 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5774000000000001e-05 2024-09-11T08:12:10.971 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.971 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.971 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:12:10.971 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:00.105293+0000", 2024-09-11T08:12:10.971 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2349999999999996e-05 2024-09-11T08:12:10.971 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.971 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.971 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3,6", 2024-09-11T08:12:10.971 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:00.105387+0000", 2024-09-11T08:12:10.971 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.3806000000000003e-05 2024-09-11T08:12:10.972 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.972 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.972 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:12:10.972 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:00.105826+0000", 2024-09-11T08:12:10.972 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00043896199999999999 2024-09-11T08:12:10.972 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.972 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.972 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:12:10.972 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:00.106972+0000", 2024-09-11T08:12:10.972 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001145828 2024-09-11T08:12:10.972 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.972 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.972 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:12:10.972 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:00.151130+0000", 2024-09-11T08:12:10.973 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.044158449000000002 2024-09-11T08:12:10.973 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.973 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.973 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:12:10.973 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:00.151177+0000", 2024-09-11T08:12:10.973 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7091e-05 2024-09-11T08:12:10.973 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.973 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.973 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:12:10.973 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:00.151214+0000", 2024-09-11T08:12:10.973 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7292000000000001e-05 2024-09-11T08:12:10.973 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:12:10.973 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:12:10.973 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:12:10.974 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.974 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.974 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14790.0:22976 7.1 7:8c5dba64:::benchmark_data_smithi150_78508_object22975:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-11T08:12:10.974 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:10:06.327412+0000", 2024-09-11T08:12:10.974 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 124.629673552, 2024-09-11T08:12:10.974 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.061830546, 2024-09-11T08:12:10.974 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:12:10.974 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:12:10.974 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:12:10.974 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14790", 2024-09-11T08:12:10.975 INFO:teuthology.orchestra.run.smithi150.stderr:noscrub is unset 2024-09-11T08:12:10.975 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:12:10.975 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 22976 2024-09-11T08:12:10.975 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.976 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:12:10.976 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.976 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:12:10.976 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:06.327412+0000", 2024-09-11T08:12:10.976 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:12:10.976 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.976 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.976 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:12:10.976 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:06.327412+0000", 2024-09-11T08:12:10.976 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:12:10.976 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.976 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.976 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:12:10.976 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:06.327410+0000", 2024-09-11T08:12:10.977 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:12:10.977 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.977 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.977 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:12:10.977 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:06.327476+0000", 2024-09-11T08:12:10.977 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5897999999999998e-05 2024-09-11T08:12:10.977 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.977 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.977 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:12:10.977 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:06.327486+0000", 2024-09-11T08:12:10.977 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4159999999999993e-06 2024-09-11T08:12:10.977 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.977 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.977 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:12:10.978 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:06.327491+0000", 2024-09-11T08:12:10.978 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.553e-06 2024-09-11T08:12:10.978 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.978 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.978 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:12:10.978 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:06.327529+0000", 2024-09-11T08:12:10.978 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7246999999999998e-05 2024-09-11T08:12:10.978 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.978 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.978 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:12:10.978 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:06.327611+0000", 2024-09-11T08:12:10.978 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2547999999999996e-05 2024-09-11T08:12:10.978 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.978 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.979 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3,6", 2024-09-11T08:12:10.979 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:06.327696+0000", 2024-09-11T08:12:10.979 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5110000000000003e-05 2024-09-11T08:12:10.979 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.979 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.979 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:12:10.979 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:06.329018+0000", 2024-09-11T08:12:10.979 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0013216739999999999 2024-09-11T08:12:10.979 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.979 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.979 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:12:10.979 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:06.329362+0000", 2024-09-11T08:12:10.979 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00034435599999999998 2024-09-11T08:12:10.979 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.980 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.980 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:12:10.980 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:06.389148+0000", 2024-09-11T08:12:10.980 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.059785333000000003 2024-09-11T08:12:10.980 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.980 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.980 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:12:10.980 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:06.389196+0000", 2024-09-11T08:12:10.980 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8164000000000002e-05 2024-09-11T08:12:10.980 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.980 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.980 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:12:10.980 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:06.389242+0000", 2024-09-11T08:12:10.981 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6755999999999998e-05 2024-09-11T08:12:10.981 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:12:10.981 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:12:10.981 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:12:10.981 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.981 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.981 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.14790.0:22997 7.4 e98/91 7:23a25ccc:::benchmark_data_smithi150_78508_object22996:head v 98'1401, mlcod=98'1398)", 2024-09-11T08:12:10.981 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:10:06.335332+0000", 2024-09-11T08:12:10.981 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 124.621753148, 2024-09-11T08:12:10.981 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.049469155000000001, 2024-09-11T08:12:10.981 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:12:10.981 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:12:10.981 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:12:10.981 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.982 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:12:10.982 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:06.335332+0000", 2024-09-11T08:12:10.982 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:12:10.982 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.982 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.982 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:12:10.982 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:06.335332+0000", 2024-09-11T08:12:10.982 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:12:10.982 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.982 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.982 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:12:10.982 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:06.335332+0000", 2024-09-11T08:12:10.982 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-09-11T08:12:10.982 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.983 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.983 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:12:10.983 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:06.335390+0000", 2024-09-11T08:12:10.983 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7927000000000003e-05 2024-09-11T08:12:10.983 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.983 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.983 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:12:10.983 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:06.335395+0000", 2024-09-11T08:12:10.983 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3090000000000003e-06 2024-09-11T08:12:10.983 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.983 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.983 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:12:10.983 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:06.335401+0000", 2024-09-11T08:12:10.983 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3809999999999998e-06 2024-09-11T08:12:10.983 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.984 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.984 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:12:10.984 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:06.335430+0000", 2024-09-11T08:12:10.984 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9039000000000001e-05 2024-09-11T08:12:10.984 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.984 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.984 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:12:10.984 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:06.335440+0000", 2024-09-11T08:12:10.984 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0475000000000001e-05 2024-09-11T08:12:10.984 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.984 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.984 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:12:10.984 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:06.335881+0000", 2024-09-11T08:12:10.985 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00044119999999999999 2024-09-11T08:12:10.985 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.985 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.985 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:12:10.985 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:06.384801+0000", 2024-09-11T08:12:10.985 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.048920224999999998 2024-09-11T08:12:10.985 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:12:10.985 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:12:10.985 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:12:10.985 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.985 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.985 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.14790.0:22999 7.9 e98/91 7:9c40682d:::benchmark_data_smithi150_78508_object22998:head v 98'1490, mlcod=98'1488)", 2024-09-11T08:12:10.985 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:10:06.336972+0000", 2024-09-11T08:12:10.985 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 124.620113464, 2024-09-11T08:12:10.985 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.048548395000000001, 2024-09-11T08:12:10.986 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:12:10.986 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:12:10.986 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:12:10.986 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.986 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:12:10.986 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:06.336972+0000", 2024-09-11T08:12:10.986 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:12:10.986 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.986 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.986 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:12:10.986 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:06.336972+0000", 2024-09-11T08:12:10.986 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:12:10.986 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.986 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.987 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:12:10.987 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:06.336971+0000", 2024-09-11T08:12:10.987 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:12:10.987 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.987 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.987 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:12:10.987 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:06.337027+0000", 2024-09-11T08:12:10.987 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5376e-05 2024-09-11T08:12:10.987 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.987 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.987 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:12:10.987 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:06.384677+0000", 2024-09-11T08:12:10.987 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.047650463999999997 2024-09-11T08:12:10.987 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.987 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.988 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:12:10.988 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:06.384685+0000", 2024-09-11T08:12:10.988 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9920000000000007e-06 2024-09-11T08:12:10.988 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.988 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.988 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:12:10.988 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:06.384770+0000", 2024-09-11T08:12:10.988 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4618999999999997e-05 2024-09-11T08:12:10.988 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.988 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.988 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:12:10.988 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:06.384807+0000", 2024-09-11T08:12:10.988 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7358999999999999e-05 2024-09-11T08:12:10.988 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.989 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.989 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:12:10.989 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:06.385497+0000", 2024-09-11T08:12:10.989 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00069030700000000003 2024-09-11T08:12:10.989 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.989 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.989 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:12:10.989 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:06.385520+0000", 2024-09-11T08:12:10.989 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3014999999999999e-05 2024-09-11T08:12:10.989 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:12:10.989 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:12:10.989 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:12:10.989 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.989 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.989 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14790.0:29143 7.1 7:8f2ccb7e:::benchmark_data_smithi150_78508_object29142:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-11T08:12:10.990 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:10:09.182744+0000", 2024-09-11T08:12:10.990 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 121.77434132, 2024-09-11T08:12:10.990 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.062484357999999997, 2024-09-11T08:12:10.990 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:12:10.990 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:12:10.990 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:12:10.990 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14790", 2024-09-11T08:12:10.990 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:12:10.990 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 29143 2024-09-11T08:12:10.990 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.990 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:12:10.990 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.990 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:12:10.990 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:09.182744+0000", 2024-09-11T08:12:10.991 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:12:10.991 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.991 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.991 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:12:10.991 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:09.182744+0000", 2024-09-11T08:12:10.991 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:12:10.991 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.991 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.991 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:12:10.991 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:09.182743+0000", 2024-09-11T08:12:10.991 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:12:10.991 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.991 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.991 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:12:10.991 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:09.182794+0000", 2024-09-11T08:12:10.992 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1057999999999998e-05 2024-09-11T08:12:10.992 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.992 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.992 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:12:10.992 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:09.182804+0000", 2024-09-11T08:12:10.992 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.482e-06 2024-09-11T08:12:10.992 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.992 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.992 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:12:10.992 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:09.182808+0000", 2024-09-11T08:12:10.992 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2209999999999999e-06 2024-09-11T08:12:10.992 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.992 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.992 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:12:10.993 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:09.182835+0000", 2024-09-11T08:12:10.993 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6971999999999999e-05 2024-09-11T08:12:10.993 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.993 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.993 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:12:10.993 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:09.182914+0000", 2024-09-11T08:12:10.993 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9505999999999994e-05 2024-09-11T08:12:10.993 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.993 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.993 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3,6", 2024-09-11T08:12:10.993 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:09.182997+0000", 2024-09-11T08:12:10.993 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2447000000000005e-05 2024-09-11T08:12:10.993 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.994 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.994 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:12:10.994 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:09.183646+0000", 2024-09-11T08:12:10.994 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00064867199999999999 2024-09-11T08:12:10.994 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.994 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.994 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:12:10.994 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:09.183981+0000", 2024-09-11T08:12:10.994 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00033580200000000001 2024-09-11T08:12:10.994 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.994 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.994 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:12:10.994 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:09.245174+0000", 2024-09-11T08:12:10.994 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.061192853999999998 2024-09-11T08:12:10.995 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.995 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.995 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:12:10.995 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:09.245206+0000", 2024-09-11T08:12:10.995 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1958999999999997e-05 2024-09-11T08:12:10.995 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.995 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.995 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:12:10.995 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:09.245228+0000", 2024-09-11T08:12:10.995 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2314999999999999e-05 2024-09-11T08:12:10.995 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:12:10.995 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:12:10.995 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:12:10.995 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.996 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.996 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14790.0:84319 7.1 7:84bf4182:::benchmark_data_smithi150_78508_object84318:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-11T08:12:10.996 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:10:48.166616+0000", 2024-09-11T08:12:10.996 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 82.790469881000007, 2024-09-11T08:12:10.996 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.071601043000000003, 2024-09-11T08:12:10.996 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:12:10.996 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:12:10.996 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:12:10.996 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14790", 2024-09-11T08:12:10.996 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:12:10.996 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 84319 2024-09-11T08:12:10.996 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.996 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:12:10.997 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.997 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:12:10.997 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.166616+0000", 2024-09-11T08:12:10.997 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:12:10.997 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.997 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.997 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:12:10.997 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.166616+0000", 2024-09-11T08:12:10.997 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:12:10.997 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.997 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.997 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:12:10.997 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.166615+0000", 2024-09-11T08:12:10.997 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:12:10.998 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.998 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.998 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:12:10.998 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.166665+0000", 2024-09-11T08:12:10.998 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0025999999999998e-05 2024-09-11T08:12:10.998 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.998 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.998 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:12:10.998 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.166677+0000", 2024-09-11T08:12:10.998 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2554999999999999e-05 2024-09-11T08:12:10.998 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.998 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.998 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:12:10.998 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.166682+0000", 2024-09-11T08:12:10.999 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9490000000000004e-06 2024-09-11T08:12:10.999 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.999 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.999 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:12:10.999 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.167476+0000", 2024-09-11T08:12:10.999 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00079362300000000003 2024-09-11T08:12:10.999 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.999 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.999 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:12:10.999 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.167593+0000", 2024-09-11T08:12:10.999 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000116928 2024-09-11T08:12:10.999 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:10.999 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:10.999 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3,6", 2024-09-11T08:12:11.000 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.167738+0000", 2024-09-11T08:12:11.000 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014501999999999999 2024-09-11T08:12:11.000 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.000 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.000 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:12:11.000 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.171613+0000", 2024-09-11T08:12:11.000 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003875426 2024-09-11T08:12:11.000 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.000 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.000 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:12:11.000 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.171658+0000", 2024-09-11T08:12:11.000 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4452000000000001e-05 2024-09-11T08:12:11.000 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.000 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.001 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:12:11.001 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.238163+0000", 2024-09-11T08:12:11.001 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.066504838999999996 2024-09-11T08:12:11.001 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.001 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.001 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:12:11.001 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.238194+0000", 2024-09-11T08:12:11.001 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1869000000000003e-05 2024-09-11T08:12:11.001 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.001 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.001 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:12:11.001 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.238217+0000", 2024-09-11T08:12:11.001 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2178e-05 2024-09-11T08:12:11.001 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:12:11.001 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:12:11.002 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:12:11.002 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.002 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.002 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14790.0:84320 7.1 7:8ba3522b:::benchmark_data_smithi150_78508_object84319:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-11T08:12:11.002 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:10:48.166817+0000", 2024-09-11T08:12:11.002 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 82.790268526000006, 2024-09-11T08:12:11.002 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.071472761999999995, 2024-09-11T08:12:11.002 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:12:11.002 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:12:11.002 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:12:11.002 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14790", 2024-09-11T08:12:11.002 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:12:11.002 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 84320 2024-09-11T08:12:11.002 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.003 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:12:11.003 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.003 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:12:11.003 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.166817+0000", 2024-09-11T08:12:11.003 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:12:11.003 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.003 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.003 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:12:11.003 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.166817+0000", 2024-09-11T08:12:11.003 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:12:11.003 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.003 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.003 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:12:11.003 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.166816+0000", 2024-09-11T08:12:11.004 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-09-11T08:12:11.004 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.004 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.004 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:12:11.004 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.166882+0000", 2024-09-11T08:12:11.004 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5207000000000001e-05 2024-09-11T08:12:11.004 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.004 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.004 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:12:11.004 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.166888+0000", 2024-09-11T08:12:11.004 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9789999999999998e-06 2024-09-11T08:12:11.004 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.004 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.004 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:12:11.005 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.166891+0000", 2024-09-11T08:12:11.005 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2160000000000002e-06 2024-09-11T08:12:11.005 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.005 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.005 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:12:11.005 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.169259+0000", 2024-09-11T08:12:11.005 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0023677730000000001 2024-09-11T08:12:11.005 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.005 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.005 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:12:11.005 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.169378+0000", 2024-09-11T08:12:11.005 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000118915 2024-09-11T08:12:11.005 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.005 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.006 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3,6", 2024-09-11T08:12:11.006 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.169523+0000", 2024-09-11T08:12:11.006 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000145053 2024-09-11T08:12:11.006 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.006 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.006 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:12:11.006 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.171644+0000", 2024-09-11T08:12:11.006 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002121749 2024-09-11T08:12:11.006 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.006 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.006 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:12:11.006 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.171664+0000", 2024-09-11T08:12:11.006 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9283000000000002e-05 2024-09-11T08:12:11.006 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.006 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.007 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:12:11.007 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.238244+0000", 2024-09-11T08:12:11.007 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.066579977999999998 2024-09-11T08:12:11.007 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.007 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.007 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:12:11.007 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.238270+0000", 2024-09-11T08:12:11.007 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6548999999999999e-05 2024-09-11T08:12:11.007 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.007 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.007 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:12:11.007 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.238290+0000", 2024-09-11T08:12:11.007 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9573e-05 2024-09-11T08:12:11.007 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:12:11.007 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:12:11.008 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:12:11.008 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.008 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.008 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14790.0:84333 7.1 7:8deb33ce:::benchmark_data_smithi150_78508_object84332:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-11T08:12:11.008 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:10:48.175810+0000", 2024-09-11T08:12:11.008 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 82.781275786999998, 2024-09-11T08:12:11.008 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.069434545, 2024-09-11T08:12:11.008 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:12:11.008 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:12:11.008 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:12:11.008 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14790", 2024-09-11T08:12:11.008 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:12:11.008 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 84333 2024-09-11T08:12:11.009 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.009 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:12:11.009 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.009 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:12:11.009 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.175810+0000", 2024-09-11T08:12:11.009 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:12:11.009 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.009 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.009 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:12:11.009 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.175810+0000", 2024-09-11T08:12:11.009 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:12:11.009 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.009 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.009 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:12:11.009 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.175809+0000", 2024-09-11T08:12:11.010 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:12:11.010 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.010 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.010 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:12:11.010 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.175864+0000", 2024-09-11T08:12:11.010 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5463000000000001e-05 2024-09-11T08:12:11.010 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.010 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.010 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:12:11.010 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.175875+0000", 2024-09-11T08:12:11.010 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0445000000000001e-05 2024-09-11T08:12:11.010 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.010 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.010 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:12:11.010 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.175879+0000", 2024-09-11T08:12:11.011 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3919999999999996e-06 2024-09-11T08:12:11.011 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.011 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.011 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:12:11.011 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.175920+0000", 2024-09-11T08:12:11.011 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0846000000000002e-05 2024-09-11T08:12:11.011 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.011 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.011 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:12:11.011 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.176042+0000", 2024-09-11T08:12:11.011 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012224999999999999 2024-09-11T08:12:11.011 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.011 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.011 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3,6", 2024-09-11T08:12:11.012 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.176199+0000", 2024-09-11T08:12:11.012 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015681399999999999 2024-09-11T08:12:11.012 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.012 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.012 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:12:11.012 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.178188+0000", 2024-09-11T08:12:11.012 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0019889560000000001 2024-09-11T08:12:11.012 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.012 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.012 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:12:11.012 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.180669+0000", 2024-09-11T08:12:11.012 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0024812990000000002 2024-09-11T08:12:11.012 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.012 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.013 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:12:11.013 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.245174+0000", 2024-09-11T08:12:11.013 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.064504484000000001 2024-09-11T08:12:11.013 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.013 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.013 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:12:11.013 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.245221+0000", 2024-09-11T08:12:11.013 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7614999999999997e-05 2024-09-11T08:12:11.013 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.013 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.013 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:12:11.013 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.245244+0000", 2024-09-11T08:12:11.013 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2949e-05 2024-09-11T08:12:11.013 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:12:11.014 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:12:11.014 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:12:11.014 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.014 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.014 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14790.0:84336 7.1 7:8f60031d:::benchmark_data_smithi150_78508_object84335:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-11T08:12:11.014 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:10:48.178146+0000", 2024-09-11T08:12:11.014 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 82.778939624000003, 2024-09-11T08:12:11.014 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.067232426999999997, 2024-09-11T08:12:11.014 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:12:11.014 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:12:11.014 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:12:11.014 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14790", 2024-09-11T08:12:11.014 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:12:11.014 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 84336 2024-09-11T08:12:11.014 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.015 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:12:11.015 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.015 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:12:11.015 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.178146+0000", 2024-09-11T08:12:11.015 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:12:11.015 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.015 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.015 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:12:11.015 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.178146+0000", 2024-09-11T08:12:11.015 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:12:11.015 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.015 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.016 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:11.017 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:12:11.017 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.178145+0000", 2024-09-11T08:12:11.017 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:12:11.017 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.017 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.017 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:12:11.017 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.178205+0000", 2024-09-11T08:12:11.017 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0825000000000001e-05 2024-09-11T08:12:11.018 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.018 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.018 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:12:11.018 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.178215+0000", 2024-09-11T08:12:11.018 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.5429999999999999e-06 2024-09-11T08:12:11.018 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.018 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.018 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:12:11.018 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.178221+0000", 2024-09-11T08:12:11.018 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8189999999999997e-06 2024-09-11T08:12:11.018 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.018 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.018 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:12:11.018 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.178258+0000", 2024-09-11T08:12:11.019 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7546000000000003e-05 2024-09-11T08:12:11.019 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.019 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.019 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:12:11.019 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.178361+0000", 2024-09-11T08:12:11.019 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000102198 2024-09-11T08:12:11.019 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.019 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.019 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3,6", 2024-09-11T08:12:11.019 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.178456+0000", 2024-09-11T08:12:11.019 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.5179999999999993e-05 2024-09-11T08:12:11.019 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.019 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.019 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:12:11.019 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.180764+0000", 2024-09-11T08:12:11.020 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002307969 2024-09-11T08:12:11.020 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.020 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.020 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:12:11.020 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.181071+0000", 2024-09-11T08:12:11.020 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00030707800000000002 2024-09-11T08:12:11.020 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.020 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.020 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:12:11.020 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.245293+0000", 2024-09-11T08:12:11.020 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.064222103000000003 2024-09-11T08:12:11.020 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.020 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.021 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:12:11.021 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.245342+0000", 2024-09-11T08:12:11.021 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8931000000000003e-05 2024-09-11T08:12:11.021 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.021 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.021 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:12:11.021 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.245378+0000", 2024-09-11T08:12:11.021 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6396000000000002e-05 2024-09-11T08:12:11.021 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:12:11.021 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:12:11.021 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:12:11.021 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.021 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.021 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14790.0:84340 7.1 7:899372ea:::benchmark_data_smithi150_78508_object84339:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-11T08:12:11.022 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:10:48.187296+0000", 2024-09-11T08:12:11.022 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 82.769789188999994, 2024-09-11T08:12:11.022 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.058192016999999999, 2024-09-11T08:12:11.022 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:12:11.022 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:12:11.022 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:12:11.022 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14790", 2024-09-11T08:12:11.022 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:12:11.022 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 84340 2024-09-11T08:12:11.022 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.022 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:12:11.022 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.022 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:12:11.022 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.187296+0000", 2024-09-11T08:12:11.023 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:12:11.023 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.023 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.023 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:12:11.023 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.187296+0000", 2024-09-11T08:12:11.023 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:12:11.023 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.023 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.023 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:12:11.023 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.187294+0000", 2024-09-11T08:12:11.023 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999976 2024-09-11T08:12:11.023 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.023 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.023 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:12:11.023 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.187347+0000", 2024-09-11T08:12:11.024 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2583000000000001e-05 2024-09-11T08:12:11.024 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.024 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.024 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:12:11.024 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.187373+0000", 2024-09-11T08:12:11.024 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6415999999999999e-05 2024-09-11T08:12:11.024 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.024 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.024 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:12:11.024 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.187384+0000", 2024-09-11T08:12:11.024 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1446e-05 2024-09-11T08:12:11.025 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.025 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.025 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:12:11.025 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.187441+0000", 2024-09-11T08:12:11.025 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7003000000000003e-05 2024-09-11T08:12:11.025 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.025 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.025 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:12:11.025 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.187548+0000", 2024-09-11T08:12:11.025 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000107044 2024-09-11T08:12:11.025 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.025 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.025 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3,6", 2024-09-11T08:12:11.025 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.187645+0000", 2024-09-11T08:12:11.026 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.6273000000000005e-05 2024-09-11T08:12:11.026 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.026 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.026 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:12:11.026 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.189610+0000", 2024-09-11T08:12:11.026 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0019657630000000001 2024-09-11T08:12:11.026 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.026 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.026 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:12:11.026 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.189744+0000", 2024-09-11T08:12:11.026 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013343400000000001 2024-09-11T08:12:11.026 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.026 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.026 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:12:11.026 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.245415+0000", 2024-09-11T08:12:11.027 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.055671222999999999 2024-09-11T08:12:11.027 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.027 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.027 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:12:11.027 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.245455+0000", 2024-09-11T08:12:11.027 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9508000000000002e-05 2024-09-11T08:12:11.027 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.027 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.027 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:12:11.027 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.245488+0000", 2024-09-11T08:12:11.027 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3633999999999997e-05 2024-09-11T08:12:11.027 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:12:11.027 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:12:11.027 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:12:11.028 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.028 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.028 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14790.0:84629 7.1 7:82c1df46:::benchmark_data_smithi150_78508_object84628:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-11T08:12:11.028 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:10:48.490234+0000", 2024-09-11T08:12:11.028 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 82.466851198000001, 2024-09-11T08:12:11.028 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.049897018000000001, 2024-09-11T08:12:11.028 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:12:11.028 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:12:11.028 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:12:11.029 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14790", 2024-09-11T08:12:11.029 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:12:11.029 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 84629 2024-09-11T08:12:11.029 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.029 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:12:11.029 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.029 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:12:11.029 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.490234+0000", 2024-09-11T08:12:11.029 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:12:11.029 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.029 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.029 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:12:11.029 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.490234+0000", 2024-09-11T08:12:11.029 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:12:11.029 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.030 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.030 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:12:11.030 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.490233+0000", 2024-09-11T08:12:11.030 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:12:11.030 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.030 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.030 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:12:11.030 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.490344+0000", 2024-09-11T08:12:11.030 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000111199 2024-09-11T08:12:11.030 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.030 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.030 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:12:11.030 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.490360+0000", 2024-09-11T08:12:11.030 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5925000000000001e-05 2024-09-11T08:12:11.030 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.031 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.031 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:12:11.031 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.490366+0000", 2024-09-11T08:12:11.031 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9939999999999997e-06 2024-09-11T08:12:11.031 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.031 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.031 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:12:11.031 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.490464+0000", 2024-09-11T08:12:11.031 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.8059000000000004e-05 2024-09-11T08:12:11.031 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.031 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.031 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:12:11.031 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.490587+0000", 2024-09-11T08:12:11.031 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012267300000000001 2024-09-11T08:12:11.032 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.032 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.032 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3,6", 2024-09-11T08:12:11.032 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.490721+0000", 2024-09-11T08:12:11.032 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013472899999999999 2024-09-11T08:12:11.032 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.032 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.032 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:12:11.032 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.493729+0000", 2024-09-11T08:12:11.032 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0030073769999999999 2024-09-11T08:12:11.032 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.032 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.032 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:12:11.033 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.494524+0000", 2024-09-11T08:12:11.033 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00079562700000000003 2024-09-11T08:12:11.033 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.033 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.033 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:12:11.033 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.540037+0000", 2024-09-11T08:12:11.033 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.045512398000000003 2024-09-11T08:12:11.033 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.033 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.033 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:12:11.033 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.540088+0000", 2024-09-11T08:12:11.033 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1535e-05 2024-09-11T08:12:11.033 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.034 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.034 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:12:11.034 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:48.540131+0000", 2024-09-11T08:12:11.034 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2894e-05 2024-09-11T08:12:11.034 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:12:11.034 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:12:11.034 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:12:11.034 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.034 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.034 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14790.0:88562 7.1 7:88de227f:::benchmark_data_smithi150_78508_object88561:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-11T08:12:11.034 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:10:52.113326+0000", 2024-09-11T08:12:11.035 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 78.843759464000001, 2024-09-11T08:12:11.035 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.045356691999999997, 2024-09-11T08:12:11.035 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:12:11.035 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:12:11.035 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:12:11.035 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14790", 2024-09-11T08:12:11.035 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:12:11.035 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 88562 2024-09-11T08:12:11.035 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.035 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:12:11.035 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.035 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:12:11.035 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:52.113326+0000", 2024-09-11T08:12:11.036 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:12:11.036 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.036 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.036 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:12:11.036 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:52.113326+0000", 2024-09-11T08:12:11.036 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:12:11.036 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.036 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.036 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:12:11.036 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:52.113325+0000", 2024-09-11T08:12:11.036 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:12:11.036 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.036 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.036 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:12:11.037 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:52.113397+0000", 2024-09-11T08:12:11.037 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2551000000000004e-05 2024-09-11T08:12:11.037 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.037 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.037 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:12:11.037 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:52.113408+0000", 2024-09-11T08:12:11.037 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0618999999999999e-05 2024-09-11T08:12:11.037 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.037 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.037 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:12:11.037 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:52.113413+0000", 2024-09-11T08:12:11.037 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5739999999999997e-06 2024-09-11T08:12:11.037 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.037 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.038 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:12:11.038 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:52.113463+0000", 2024-09-11T08:12:11.038 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0547999999999998e-05 2024-09-11T08:12:11.038 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.038 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.038 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:12:11.038 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:52.113582+0000", 2024-09-11T08:12:11.038 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011871600000000001 2024-09-11T08:12:11.038 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.038 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.038 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3,6", 2024-09-11T08:12:11.038 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:52.113685+0000", 2024-09-11T08:12:11.038 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010268900000000001 2024-09-11T08:12:11.038 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.039 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.039 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:12:11.039 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:52.115653+0000", 2024-09-11T08:12:11.039 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0019681590000000001 2024-09-11T08:12:11.039 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.039 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.039 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:12:11.039 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:52.115868+0000", 2024-09-11T08:12:11.039 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000215782 2024-09-11T08:12:11.039 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.039 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.039 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:12:11.039 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:52.158620+0000", 2024-09-11T08:12:11.039 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.042751387000000002 2024-09-11T08:12:11.040 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.040 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.040 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:12:11.040 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:52.158657+0000", 2024-09-11T08:12:11.040 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6832999999999998e-05 2024-09-11T08:12:11.040 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.040 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.040 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:12:11.040 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:10:52.158683+0000", 2024-09-11T08:12:11.040 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5989000000000001e-05 2024-09-11T08:12:11.040 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:12:11.040 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:12:11.040 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:12:11.040 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.041 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.041 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14790.0:107802 7.1 7:83f1eed6:::benchmark_data_smithi150_78508_object107801:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-11T08:12:11.041 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:11:10.663541+0000", 2024-09-11T08:12:11.041 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 60.293544642999997, 2024-09-11T08:12:11.041 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.095429493000000004, 2024-09-11T08:12:11.041 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:12:11.041 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:12:11.041 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:12:11.041 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14790", 2024-09-11T08:12:11.041 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:12:11.041 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 107802 2024-09-11T08:12:11.041 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.041 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:12:11.041 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.042 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:12:11.042 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.663541+0000", 2024-09-11T08:12:11.042 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:12:11.042 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.042 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.042 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:12:11.042 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.663541+0000", 2024-09-11T08:12:11.042 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:12:11.042 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.042 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.042 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:12:11.042 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.663540+0000", 2024-09-11T08:12:11.042 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:12:11.042 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.043 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.043 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:12:11.043 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.663588+0000", 2024-09-11T08:12:11.043 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8643999999999997e-05 2024-09-11T08:12:11.043 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.043 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.043 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:12:11.043 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.663600+0000", 2024-09-11T08:12:11.043 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.149e-05 2024-09-11T08:12:11.043 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.043 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.043 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:12:11.043 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.663604+0000", 2024-09-11T08:12:11.043 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0860000000000004e-06 2024-09-11T08:12:11.044 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.044 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.044 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:12:11.044 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.663648+0000", 2024-09-11T08:12:11.044 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3914e-05 2024-09-11T08:12:11.044 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.044 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.044 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:12:11.044 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.663781+0000", 2024-09-11T08:12:11.044 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013298100000000001 2024-09-11T08:12:11.044 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.044 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.044 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3,6", 2024-09-11T08:12:11.045 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.663936+0000", 2024-09-11T08:12:11.045 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015576000000000001 2024-09-11T08:12:11.045 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.045 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.045 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:12:11.045 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.666044+0000", 2024-09-11T08:12:11.045 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002107799 2024-09-11T08:12:11.045 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.045 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.045 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:12:11.045 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.666336+0000", 2024-09-11T08:12:11.045 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000291945 2024-09-11T08:12:11.045 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.045 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.046 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:12:11.046 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.758837+0000", 2024-09-11T08:12:11.046 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.092500685999999999 2024-09-11T08:12:11.046 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.046 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.046 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:12:11.046 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.758926+0000", 2024-09-11T08:12:11.046 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.9215000000000006e-05 2024-09-11T08:12:11.046 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.046 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.046 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:12:11.046 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.758970+0000", 2024-09-11T08:12:11.046 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4233000000000002e-05 2024-09-11T08:12:11.047 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:12:11.047 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:12:11.047 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:12:11.047 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.047 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.047 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14790.0:107995 7.1 7:86913d1e:::benchmark_data_smithi150_78508_object107994:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-11T08:12:11.047 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:11:10.923701+0000", 2024-09-11T08:12:11.047 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 60.033384939999998, 2024-09-11T08:12:11.047 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.113043591, 2024-09-11T08:12:11.047 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:12:11.047 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:12:11.047 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:12:11.047 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14790", 2024-09-11T08:12:11.047 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:12:11.048 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 107995 2024-09-11T08:12:11.048 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.048 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:12:11.048 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.048 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:12:11.048 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.923701+0000", 2024-09-11T08:12:11.048 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:12:11.048 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.048 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.048 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:12:11.048 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.923701+0000", 2024-09-11T08:12:11.048 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:12:11.048 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.048 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.049 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:12:11.049 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.923699+0000", 2024-09-11T08:12:11.049 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:12:11.049 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.049 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.053 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:12:11.053 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.923729+0000", 2024-09-11T08:12:11.053 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0352e-05 2024-09-11T08:12:11.053 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.053 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.053 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:12:11.053 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.923741+0000", 2024-09-11T08:12:11.053 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2047e-05 2024-09-11T08:12:11.053 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.053 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.053 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:12:11.053 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.923748+0000", 2024-09-11T08:12:11.053 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3860000000000004e-06 2024-09-11T08:12:11.054 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.054 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.054 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:12:11.054 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.923797+0000", 2024-09-11T08:12:11.054 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9351999999999998e-05 2024-09-11T08:12:11.054 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.054 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.054 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:12:11.054 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.923918+0000", 2024-09-11T08:12:11.054 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000120975 2024-09-11T08:12:11.054 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.054 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.054 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3,6", 2024-09-11T08:12:11.054 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.924047+0000", 2024-09-11T08:12:11.054 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012928700000000001 2024-09-11T08:12:11.055 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.055 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.055 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:12:11.055 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.926368+0000", 2024-09-11T08:12:11.055 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0023208619999999999 2024-09-11T08:12:11.055 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.055 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.055 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:12:11.055 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.926543+0000", 2024-09-11T08:12:11.055 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017447799999999999 2024-09-11T08:12:11.055 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.055 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.056 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:12:11.056 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:11.036619+0000", 2024-09-11T08:12:11.056 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.110076354 2024-09-11T08:12:11.056 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.056 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.056 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:12:11.056 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:11.036705+0000", 2024-09-11T08:12:11.056 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5871999999999994e-05 2024-09-11T08:12:11.056 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.056 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.056 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:12:11.057 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:11.036744+0000", 2024-09-11T08:12:11.057 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9286000000000003e-05 2024-09-11T08:12:11.057 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:12:11.057 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:12:11.057 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:12:11.057 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.057 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.057 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14790.0:110283 7.1 7:817caa10:::benchmark_data_smithi150_78508_object110282:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-11T08:12:11.057 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:11:13.289874+0000", 2024-09-11T08:12:11.057 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 57.667211596000001, 2024-09-11T08:12:11.057 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.063032227999999996, 2024-09-11T08:12:11.057 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:12:11.057 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:12:11.057 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:12:11.058 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14790", 2024-09-11T08:12:11.058 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:12:11.058 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 110283 2024-09-11T08:12:11.058 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.058 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:12:11.058 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.058 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:12:11.058 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:13.289874+0000", 2024-09-11T08:12:11.058 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:12:11.058 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.058 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.058 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:12:11.058 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:13.289874+0000", 2024-09-11T08:12:11.058 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:12:11.059 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.059 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.059 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:12:11.059 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:13.289873+0000", 2024-09-11T08:12:11.059 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:12:11.059 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.059 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.059 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:12:11.059 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:13.346945+0000", 2024-09-11T08:12:11.059 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.057072388000000002 2024-09-11T08:12:11.059 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.059 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.059 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:12:11.059 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:13.346959+0000", 2024-09-11T08:12:11.059 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4022999999999999e-05 2024-09-11T08:12:11.060 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.060 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.060 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:12:11.060 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:13.346965+0000", 2024-09-11T08:12:11.060 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1759999999999998e-06 2024-09-11T08:12:11.060 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.060 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.060 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:12:11.060 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:13.347057+0000", 2024-09-11T08:12:11.060 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.1558999999999995e-05 2024-09-11T08:12:11.060 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.060 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.060 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:12:11.060 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:13.347215+0000", 2024-09-11T08:12:11.061 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015862 2024-09-11T08:12:11.061 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.061 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.061 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3,6", 2024-09-11T08:12:11.061 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:13.347361+0000", 2024-09-11T08:12:11.061 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000145492 2024-09-11T08:12:11.061 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.061 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.061 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:12:11.061 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:13.352753+0000", 2024-09-11T08:12:11.061 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0053924039999999999 2024-09-11T08:12:11.061 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.061 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.061 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:12:11.061 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:13.352759+0000", 2024-09-11T08:12:11.062 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9569999999999999e-06 2024-09-11T08:12:11.062 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.062 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.062 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:12:11.062 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:13.352789+0000", 2024-09-11T08:12:11.062 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9696000000000002e-05 2024-09-11T08:12:11.062 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.062 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.062 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:12:11.062 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:13.352867+0000", 2024-09-11T08:12:11.062 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8268000000000003e-05 2024-09-11T08:12:11.062 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.062 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.063 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:12:11.063 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:13.352906+0000", 2024-09-11T08:12:11.063 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8897999999999999e-05 2024-09-11T08:12:11.063 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:12:11.063 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:12:11.063 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:12:11.063 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.063 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.063 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14790.0:130741 7.1 7:8a0a8031:::benchmark_data_smithi150_78508_object6300:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-11T08:12:11.063 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:11:33.747128+0000", 2024-09-11T08:12:11.063 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 37.209957365000001, 2024-09-11T08:12:11.063 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.050709987999999998, 2024-09-11T08:12:11.063 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:12:11.063 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:12:11.064 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:12:11.064 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14790", 2024-09-11T08:12:11.064 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:12:11.064 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 130741 2024-09-11T08:12:11.064 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.064 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:12:11.064 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.064 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:12:11.064 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:33.747128+0000", 2024-09-11T08:12:11.064 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:12:11.064 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.064 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.064 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:12:11.064 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:33.747128+0000", 2024-09-11T08:12:11.064 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:12:11.065 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.065 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.065 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:12:11.065 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:33.747127+0000", 2024-09-11T08:12:11.065 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:12:11.065 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.065 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.065 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:12:11.065 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:33.747132+0000", 2024-09-11T08:12:11.065 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2650000000000004e-06 2024-09-11T08:12:11.065 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.065 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.065 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:12:11.065 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:33.747146+0000", 2024-09-11T08:12:11.066 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4153e-05 2024-09-11T08:12:11.066 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.066 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.066 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:12:11.066 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:33.747152+0000", 2024-09-11T08:12:11.066 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8880000000000002e-06 2024-09-11T08:12:11.066 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.066 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.066 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:12:11.066 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:33.747212+0000", 2024-09-11T08:12:11.066 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9685999999999997e-05 2024-09-11T08:12:11.066 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.066 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.066 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:12:11.067 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:33.747319+0000", 2024-09-11T08:12:11.067 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000107358 2024-09-11T08:12:11.067 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.067 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.067 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3,6", 2024-09-11T08:12:11.067 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:33.747431+0000", 2024-09-11T08:12:11.067 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000112001 2024-09-11T08:12:11.067 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.067 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.067 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:12:11.067 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:33.749787+0000", 2024-09-11T08:12:11.067 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0023554660000000001 2024-09-11T08:12:11.067 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.067 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.068 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:12:11.068 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:33.749793+0000", 2024-09-11T08:12:11.068 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.161e-06 2024-09-11T08:12:11.068 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.068 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.068 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:12:11.068 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:33.797778+0000", 2024-09-11T08:12:11.068 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.047985397999999999 2024-09-11T08:12:11.068 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.068 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.068 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:12:11.068 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:33.797810+0000", 2024-09-11T08:12:11.068 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2017999999999997e-05 2024-09-11T08:12:11.068 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.069 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.069 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:12:11.069 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:33.797838+0000", 2024-09-11T08:12:11.069 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7891000000000001e-05 2024-09-11T08:12:11.069 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:12:11.069 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:12:11.069 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:12:11.069 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.069 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.069 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14790.0:135924 7.1 7:89cf533d:::benchmark_data_smithi150_78508_object11483:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-11T08:12:11.069 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:11:38.566855+0000", 2024-09-11T08:12:11.069 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 32.390230379999998, 2024-09-11T08:12:11.069 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.042825861999999999, 2024-09-11T08:12:11.069 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:12:11.069 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:12:11.070 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:12:11.070 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14790", 2024-09-11T08:12:11.070 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:12:11.070 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 135924 2024-09-11T08:12:11.070 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.070 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:12:11.070 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.070 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:12:11.070 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:38.566855+0000", 2024-09-11T08:12:11.070 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:12:11.070 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.070 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.070 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:12:11.070 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:38.566855+0000", 2024-09-11T08:12:11.071 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:12:11.071 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.071 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.071 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:12:11.071 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:38.566853+0000", 2024-09-11T08:12:11.071 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999981 2024-09-11T08:12:11.071 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.071 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.071 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:12:11.071 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:38.566861+0000", 2024-09-11T08:12:11.071 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4549999999999998e-06 2024-09-11T08:12:11.071 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.071 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.071 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:12:11.071 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:38.566877+0000", 2024-09-11T08:12:11.072 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6697e-05 2024-09-11T08:12:11.072 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.072 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.072 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:12:11.072 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:38.566885+0000", 2024-09-11T08:12:11.072 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0899999999999999e-06 2024-09-11T08:12:11.072 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.072 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.072 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:12:11.072 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:38.566924+0000", 2024-09-11T08:12:11.072 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9588000000000001e-05 2024-09-11T08:12:11.072 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.072 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.072 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:12:11.073 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:38.567039+0000", 2024-09-11T08:12:11.073 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000114738 2024-09-11T08:12:11.073 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.073 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.073 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3,6", 2024-09-11T08:12:11.073 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:38.567151+0000", 2024-09-11T08:12:11.073 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000112221 2024-09-11T08:12:11.073 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.073 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.073 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:12:11.073 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:38.571473+0000", 2024-09-11T08:12:11.073 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0043223589999999996 2024-09-11T08:12:11.073 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.073 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.074 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:12:11.074 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:38.572115+0000", 2024-09-11T08:12:11.074 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00064114 2024-09-11T08:12:11.074 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.074 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.074 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:12:11.074 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:38.609621+0000", 2024-09-11T08:12:11.074 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.037506149000000003 2024-09-11T08:12:11.074 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.074 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.074 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:12:11.074 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:38.609653+0000", 2024-09-11T08:12:11.074 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2597e-05 2024-09-11T08:12:11.074 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.074 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.075 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:12:11.075 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:38.609681+0000", 2024-09-11T08:12:11.075 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7580999999999999e-05 2024-09-11T08:12:11.075 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:12:11.075 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:12:11.075 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:12:11.075 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.075 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.075 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14790.0:142826 7.1 7:8328360b:::benchmark_data_smithi150_78508_object18385:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-11T08:12:11.075 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:11:45.459807+0000", 2024-09-11T08:12:11.075 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 25.497278182999999, 2024-09-11T08:12:11.075 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10172049499999999, 2024-09-11T08:12:11.075 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:12:11.075 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:12:11.076 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:12:11.076 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14790", 2024-09-11T08:12:11.076 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:12:11.076 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 142826 2024-09-11T08:12:11.076 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.076 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:12:11.076 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.076 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:12:11.076 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.459807+0000", 2024-09-11T08:12:11.076 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:12:11.076 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.076 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.076 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:12:11.077 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.459807+0000", 2024-09-11T08:12:11.077 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:12:11.077 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.077 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.077 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:12:11.077 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.459806+0000", 2024-09-11T08:12:11.077 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999981 2024-09-11T08:12:11.077 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.077 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.077 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:12:11.077 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.459813+0000", 2024-09-11T08:12:11.077 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2359999999999998e-06 2024-09-11T08:12:11.077 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.077 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.078 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:12:11.078 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.459828+0000", 2024-09-11T08:12:11.078 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5217e-05 2024-09-11T08:12:11.078 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.078 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.078 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:12:11.078 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.459835+0000", 2024-09-11T08:12:11.078 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6490000000000002e-06 2024-09-11T08:12:11.078 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.078 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.078 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:12:11.078 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.459902+0000", 2024-09-11T08:12:11.078 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7329999999999994e-05 2024-09-11T08:12:11.078 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.079 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.079 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:12:11.079 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.460011+0000", 2024-09-11T08:12:11.079 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000109405 2024-09-11T08:12:11.079 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.079 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.079 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3,6", 2024-09-11T08:12:11.079 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.460118+0000", 2024-09-11T08:12:11.079 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000106189 2024-09-11T08:12:11.079 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.079 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.079 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:12:11.079 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.462711+0000", 2024-09-11T08:12:11.079 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0025929640000000001 2024-09-11T08:12:11.080 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.080 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.080 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:12:11.080 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.464303+0000", 2024-09-11T08:12:11.080 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0015924909999999999 2024-09-11T08:12:11.080 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.080 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.080 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:12:11.080 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.561429+0000", 2024-09-11T08:12:11.080 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.097125843000000003 2024-09-11T08:12:11.080 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.080 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.080 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:12:11.080 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.561490+0000", 2024-09-11T08:12:11.081 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1189e-05 2024-09-11T08:12:11.081 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.081 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.081 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:12:11.081 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.561528+0000", 2024-09-11T08:12:11.081 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7676000000000003e-05 2024-09-11T08:12:11.081 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:12:11.081 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:12:11.081 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:12:11.081 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.081 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.081 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14790.0:147051 7.1 7:8b62d6b4:::benchmark_data_smithi150_78508_object22610:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-11T08:12:11.081 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:11:49.744064+0000", 2024-09-11T08:12:11.082 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 21.213021235999999, 2024-09-11T08:12:11.082 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.052095651, 2024-09-11T08:12:11.082 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:12:11.082 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:12:11.082 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:12:11.082 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14790", 2024-09-11T08:12:11.082 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:12:11.082 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 147051 2024-09-11T08:12:11.082 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.082 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:12:11.082 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.082 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:12:11.082 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:49.744064+0000", 2024-09-11T08:12:11.082 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:12:11.082 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.083 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.083 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:12:11.083 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:49.744064+0000", 2024-09-11T08:12:11.083 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:12:11.083 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.083 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.083 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:12:11.083 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:49.744063+0000", 2024-09-11T08:12:11.083 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:12:11.083 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.083 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.083 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:12:11.083 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:49.744069+0000", 2024-09-11T08:12:11.083 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5819999999999998e-06 2024-09-11T08:12:11.084 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.084 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.084 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:12:11.084 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:49.744084+0000", 2024-09-11T08:12:11.084 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4839e-05 2024-09-11T08:12:11.084 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.084 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.084 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:12:11.084 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:49.744091+0000", 2024-09-11T08:12:11.084 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6200000000000001e-06 2024-09-11T08:12:11.084 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.084 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.084 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:12:11.084 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:49.744160+0000", 2024-09-11T08:12:11.085 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9596000000000002e-05 2024-09-11T08:12:11.085 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.085 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.085 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:12:11.085 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:49.744274+0000", 2024-09-11T08:12:11.085 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011326 2024-09-11T08:12:11.085 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.085 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.085 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3,6", 2024-09-11T08:12:11.085 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:49.744369+0000", 2024-09-11T08:12:11.085 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.5354999999999998e-05 2024-09-11T08:12:11.085 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.085 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.085 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:12:11.086 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:49.747385+0000", 2024-09-11T08:12:11.086 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0030159010000000001 2024-09-11T08:12:11.086 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.086 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.086 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:12:11.086 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:49.747804+0000", 2024-09-11T08:12:11.086 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00041893900000000001 2024-09-11T08:12:11.086 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.086 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.086 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:12:11.086 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:49.796069+0000", 2024-09-11T08:12:11.086 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.048264814000000003 2024-09-11T08:12:11.086 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.086 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.087 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:12:11.087 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:49.796119+0000", 2024-09-11T08:12:11.087 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9974e-05 2024-09-11T08:12:11.087 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:11.087 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:11.087 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:12:11.087 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:49.796160+0000", 2024-09-11T08:12:11.087 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1183000000000002e-05 2024-09-11T08:12:11.087 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:12:11.087 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:12:11.087 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:12:11.087 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:12:11.087 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:12:11.087 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-11T08:12:11.088 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:10 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1833605939' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:12:11.088 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:10 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:12:11.089 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:10 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1833605939' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:12:11.089 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:10 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:12:11.089 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:10 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1833605939' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:12:11.089 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:10 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:12:11.129 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:11.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:11.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:11.231 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:12:11.258 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:11.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:11.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:12:11.330 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:12:11.360 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:11.390 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:11.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:11.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:11.492 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:12:11.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:11.527 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:11.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:12:11.628 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:12:11.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:11.675 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:11.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:12:11.776 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:11.800 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:11.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:11.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:11.902 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:12:11.929 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:11.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:11.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:12:12.032 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:12.064 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:11 smithi150 ceph-mon[29276]: pgmap v186: 148 pgs: 148 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.17k op/s 2024-09-11T08:12:12.064 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:11 smithi150 ceph-mon[29276]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:12:12.064 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:11 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:12:12.064 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:11 smithi150 ceph-mon[29276]: osdmap e135: 8 total, 8 up, 8 in 2024-09-11T08:12:12.064 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:11 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:12:12.065 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:11 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:12:12.065 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:11 smithi150 ceph-mon[29276]: 5.2 scrub starts 2024-09-11T08:12:12.065 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:11 smithi150 ceph-mon[29276]: 5.2 scrub ok 2024-09-11T08:12:12.065 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:11 smithi150 ceph-mon[39612]: pgmap v186: 148 pgs: 148 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.17k op/s 2024-09-11T08:12:12.065 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:11 smithi150 ceph-mon[39612]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:12:12.065 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:11 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:12:12.065 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:11 smithi150 ceph-mon[39612]: osdmap e135: 8 total, 8 up, 8 in 2024-09-11T08:12:12.065 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:11 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:12:12.065 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:11 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:12:12.066 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:11 smithi150 ceph-mon[39612]: 5.2 scrub starts 2024-09-11T08:12:12.066 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:11 smithi150 ceph-mon[39612]: 5.2 scrub ok 2024-09-11T08:12:12.074 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:12.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:12.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:12.176 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:12.202 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:12.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:12.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:12.304 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:11 smithi181 ceph-mon[32701]: pgmap v186: 148 pgs: 148 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.17k op/s 2024-09-11T08:12:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:11 smithi181 ceph-mon[32701]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:12:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:11 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:12:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:11 smithi181 ceph-mon[32701]: osdmap e135: 8 total, 8 up, 8 in 2024-09-11T08:12:12.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:11 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:12:12.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:11 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:12:12.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:11 smithi181 ceph-mon[32701]: 5.2 scrub starts 2024-09-11T08:12:12.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:11 smithi181 ceph-mon[32701]: 5.2 scrub ok 2024-09-11T08:12:12.333 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:12.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:12.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:12.435 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:12.443 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:12:12.443 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:12:12.459 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:12.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:12.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:12.480 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:12:12.481 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:12:12.517 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:12:12.518 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:12:12.561 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:12:12.562 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:12:12.563 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:12:12.586 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:12.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:12.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:12:12.600 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:12:12.600 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:12:12.631 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:12:12.631 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:12:12.643 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-11T08:12:12.680 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:12:12.680 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:12:12.688 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:12.717 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:12:12.718 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:12:12.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:12.721 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:12.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:12.822 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:12:12.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:12.863 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:12.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:12:12.964 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:12:12.969 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:12:12.999 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:13.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:13.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:12:13.101 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:13.128 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:13.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:13.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:13.230 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:13.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:12 smithi150 ceph-mon[29276]: 6.8 scrub starts 2024-09-11T08:12:13.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:12 smithi150 ceph-mon[29276]: 6.8 scrub ok 2024-09-11T08:12:13.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:12 smithi150 ceph-mon[29276]: osdmap e136: 8 total, 8 up, 8 in 2024-09-11T08:12:13.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:12 smithi150 ceph-mon[29276]: 3.1b scrub starts 2024-09-11T08:12:13.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:12 smithi150 ceph-mon[29276]: 3.1b scrub ok 2024-09-11T08:12:13.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:12 smithi150 ceph-mon[29276]: 6.13 scrub starts 2024-09-11T08:12:13.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:12 smithi150 ceph-mon[29276]: 6.13 scrub ok 2024-09-11T08:12:13.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:12 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3286920193' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:12:13.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:12 smithi150 ceph-mon[39612]: 6.8 scrub starts 2024-09-11T08:12:13.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:12 smithi150 ceph-mon[39612]: 6.8 scrub ok 2024-09-11T08:12:13.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:12 smithi150 ceph-mon[39612]: osdmap e136: 8 total, 8 up, 8 in 2024-09-11T08:12:13.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:12 smithi150 ceph-mon[39612]: 3.1b scrub starts 2024-09-11T08:12:13.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:12 smithi150 ceph-mon[39612]: 3.1b scrub ok 2024-09-11T08:12:13.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:12 smithi150 ceph-mon[39612]: 6.13 scrub starts 2024-09-11T08:12:13.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:12 smithi150 ceph-mon[39612]: 6.13 scrub ok 2024-09-11T08:12:13.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:12 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3286920193' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:12:13.268 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:13.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:13.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:13.269 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:12 smithi181 ceph-mon[32701]: 6.8 scrub starts 2024-09-11T08:12:13.269 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:12 smithi181 ceph-mon[32701]: 6.8 scrub ok 2024-09-11T08:12:13.270 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:12 smithi181 ceph-mon[32701]: osdmap e136: 8 total, 8 up, 8 in 2024-09-11T08:12:13.270 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:12 smithi181 ceph-mon[32701]: 3.1b scrub starts 2024-09-11T08:12:13.270 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:12 smithi181 ceph-mon[32701]: 3.1b scrub ok 2024-09-11T08:12:13.270 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:12 smithi181 ceph-mon[32701]: 6.13 scrub starts 2024-09-11T08:12:13.270 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:12 smithi181 ceph-mon[32701]: 6.13 scrub ok 2024-09-11T08:12:13.270 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:12 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3286920193' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:12:13.369 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:13.401 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:13.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:13.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:13.503 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:13.541 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:13.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:13.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:13.643 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:13.680 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:13.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:13.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:13.782 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:13.817 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T08:12:13.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:13.826 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:13.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:13.927 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:13.948 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:13.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:13.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:14.051 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:12:14.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:14.099 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:14.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:12:14.200 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:14.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:14.222 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:14.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:14.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:13 smithi150 ceph-mon[29276]: pgmap v189: 148 pgs: 148 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-09-11T08:12:14.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:13 smithi150 ceph-mon[29276]: 7.a scrub starts 2024-09-11T08:12:14.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:13 smithi150 ceph-mon[39612]: pgmap v189: 148 pgs: 148 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-09-11T08:12:14.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:13 smithi150 ceph-mon[39612]: 7.a scrub starts 2024-09-11T08:12:14.254 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:13 smithi181 ceph-mon[32701]: pgmap v189: 148 pgs: 148 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-09-11T08:12:14.254 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:13 smithi181 ceph-mon[32701]: 7.a scrub starts 2024-09-11T08:12:14.323 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:14.352 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:14.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:14.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:14.454 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:14.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:14.475 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:14.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:14.577 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:14.602 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:14.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:14.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:14.704 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:14.745 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:14.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:14.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:14.847 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:14.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:14.879 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:14.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:14.980 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:14.992 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:14 smithi181 ceph-mon[32701]: 7.b scrub starts 2024-09-11T08:12:14.992 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:14 smithi181 ceph-mon[32701]: 6.16 scrub starts 2024-09-11T08:12:14.992 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:14 smithi181 ceph-mon[32701]: 6.16 scrub ok 2024-09-11T08:12:14.992 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:14 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:12:14.992 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:14 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:12:15.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:15.025 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:15.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:15.126 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:15.151 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:15.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:15.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:15.253 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:15.260 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:12:15 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:12:15] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:12:15.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:15.281 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:15.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:15.288 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-11T08:12:15.288 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:12:15.341 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:14 smithi150 ceph-mon[29276]: 7.b scrub starts 2024-09-11T08:12:15.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:14 smithi150 ceph-mon[29276]: 6.16 scrub starts 2024-09-11T08:12:15.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:14 smithi150 ceph-mon[29276]: 6.16 scrub ok 2024-09-11T08:12:15.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:14 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:12:15.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:14 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:12:15.342 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:14 smithi150 ceph-mon[39612]: 7.b scrub starts 2024-09-11T08:12:15.343 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:14 smithi150 ceph-mon[39612]: 6.16 scrub starts 2024-09-11T08:12:15.343 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:14 smithi150 ceph-mon[39612]: 6.16 scrub ok 2024-09-11T08:12:15.343 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:14 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:12:15.343 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:14 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:12:15.382 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:15.423 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:15.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:15.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:15.525 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:12:15.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:15.623 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:15.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:12:15.724 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:15.780 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:15.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:15.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:15.882 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:12:15.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:15.910 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:15.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:12:16.011 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:12:16.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:16.047 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:16.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:12:16.061 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-11T08:12:16.148 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:16.197 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:16.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:16.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:16.230 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:12:16.300 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:16.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:15 smithi181 ceph-mon[32701]: 4.15 scrub starts 2024-09-11T08:12:16.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:15 smithi181 ceph-mon[32701]: 4.15 scrub ok 2024-09-11T08:12:16.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:15 smithi181 ceph-mon[32701]: 4.f scrub starts 2024-09-11T08:12:16.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:15 smithi181 ceph-mon[32701]: 4.f scrub ok 2024-09-11T08:12:16.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:15 smithi181 ceph-mon[32701]: pgmap v190: 148 pgs: 148 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-09-11T08:12:16.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:15 smithi181 ceph-mon[32701]: osdmap e137: 8 total, 8 up, 8 in 2024-09-11T08:12:16.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:15 smithi181 ceph-mon[32701]: 4.5 scrub starts 2024-09-11T08:12:16.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:15 smithi181 ceph-mon[32701]: 4.5 scrub ok 2024-09-11T08:12:16.328 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:16.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:16.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:16.381 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:15 smithi150 ceph-mon[29276]: 4.15 scrub starts 2024-09-11T08:12:16.381 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:15 smithi150 ceph-mon[29276]: 4.15 scrub ok 2024-09-11T08:12:16.381 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:15 smithi150 ceph-mon[29276]: 4.f scrub starts 2024-09-11T08:12:16.381 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:15 smithi150 ceph-mon[29276]: 4.f scrub ok 2024-09-11T08:12:16.382 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:15 smithi150 ceph-mon[29276]: pgmap v190: 148 pgs: 148 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-09-11T08:12:16.382 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:15 smithi150 ceph-mon[29276]: osdmap e137: 8 total, 8 up, 8 in 2024-09-11T08:12:16.382 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:15 smithi150 ceph-mon[29276]: 4.5 scrub starts 2024-09-11T08:12:16.382 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:15 smithi150 ceph-mon[29276]: 4.5 scrub ok 2024-09-11T08:12:16.382 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:15 smithi150 ceph-mon[39612]: 4.15 scrub starts 2024-09-11T08:12:16.382 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:15 smithi150 ceph-mon[39612]: 4.15 scrub ok 2024-09-11T08:12:16.382 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:15 smithi150 ceph-mon[39612]: 4.f scrub starts 2024-09-11T08:12:16.382 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:15 smithi150 ceph-mon[39612]: 4.f scrub ok 2024-09-11T08:12:16.382 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:15 smithi150 ceph-mon[39612]: pgmap v190: 148 pgs: 148 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-09-11T08:12:16.383 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:15 smithi150 ceph-mon[39612]: osdmap e137: 8 total, 8 up, 8 in 2024-09-11T08:12:16.383 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:15 smithi150 ceph-mon[39612]: 4.5 scrub starts 2024-09-11T08:12:16.383 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:15 smithi150 ceph-mon[39612]: 4.5 scrub ok 2024-09-11T08:12:16.430 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:16.441 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:12:16.474 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:16.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:16.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:16.576 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:16.611 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:16.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:16.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:16.713 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:16.738 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:16.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:16.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:16.840 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:16.875 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:16.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:16.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:16.977 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:16.999 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:16.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:17.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:17.100 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:12:17.149 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:17.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:17.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:12:17.250 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:17.254 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:16 smithi150 ceph-mon[29276]: 6.3 scrub starts 2024-09-11T08:12:17.254 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:16 smithi150 ceph-mon[29276]: 6.3 scrub ok 2024-09-11T08:12:17.254 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:16 smithi150 ceph-mon[29276]: 5.7 scrub starts 2024-09-11T08:12:17.254 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:16 smithi150 ceph-mon[29276]: 5.7 scrub ok 2024-09-11T08:12:17.254 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:16 smithi150 ceph-mon[29276]: 7.b scrub ok 2024-09-11T08:12:17.254 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:16 smithi150 ceph-mon[29276]: 7.a scrub ok 2024-09-11T08:12:17.254 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:16 smithi150 ceph-mon[29276]: 4.10 scrub starts 2024-09-11T08:12:17.254 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:16 smithi150 ceph-mon[29276]: 4.10 scrub ok 2024-09-11T08:12:17.254 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:16 smithi150 ceph-mon[39612]: 6.3 scrub starts 2024-09-11T08:12:17.254 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:16 smithi150 ceph-mon[39612]: 6.3 scrub ok 2024-09-11T08:12:17.255 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:16 smithi150 ceph-mon[39612]: 5.7 scrub starts 2024-09-11T08:12:17.255 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:16 smithi150 ceph-mon[39612]: 5.7 scrub ok 2024-09-11T08:12:17.255 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:16 smithi150 ceph-mon[39612]: 7.b scrub ok 2024-09-11T08:12:17.255 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:16 smithi150 ceph-mon[39612]: 7.a scrub ok 2024-09-11T08:12:17.255 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:16 smithi150 ceph-mon[39612]: 4.10 scrub starts 2024-09-11T08:12:17.255 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:16 smithi150 ceph-mon[39612]: 4.10 scrub ok 2024-09-11T08:12:17.275 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:17.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:17.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:17.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:16 smithi181 ceph-mon[32701]: 6.3 scrub starts 2024-09-11T08:12:17.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:16 smithi181 ceph-mon[32701]: 6.3 scrub ok 2024-09-11T08:12:17.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:16 smithi181 ceph-mon[32701]: 5.7 scrub starts 2024-09-11T08:12:17.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:16 smithi181 ceph-mon[32701]: 5.7 scrub ok 2024-09-11T08:12:17.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:16 smithi181 ceph-mon[32701]: 7.b scrub ok 2024-09-11T08:12:17.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:16 smithi181 ceph-mon[32701]: 7.a scrub ok 2024-09-11T08:12:17.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:16 smithi181 ceph-mon[32701]: 4.10 scrub starts 2024-09-11T08:12:17.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:16 smithi181 ceph-mon[32701]: 4.10 scrub ok 2024-09-11T08:12:17.378 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:12:17.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:17.422 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:17.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:12:17.523 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:12:17.547 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:17.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:17.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:12:17.649 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:17.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:17.687 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:17.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:17.702 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T08:12:17.788 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:17.991 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:12:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:12:17.998 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:17.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:17.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:18.100 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:12:18.158 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:18.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:18.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:12:18.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:18 smithi150 ceph-mon[29276]: 3.d scrub starts 2024-09-11T08:12:18.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:18 smithi150 ceph-mon[29276]: 3.d scrub ok 2024-09-11T08:12:18.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:18 smithi150 ceph-mon[29276]: 6.7 scrub starts 2024-09-11T08:12:18.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:18 smithi150 ceph-mon[29276]: pgmap v192: 148 pgs: 148 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 909 op/s 2024-09-11T08:12:18.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:18 smithi150 ceph-mon[29276]: 6.7 scrub ok 2024-09-11T08:12:18.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:18 smithi150 ceph-mon[29276]: 6.0 scrub starts 2024-09-11T08:12:18.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:18 smithi150 ceph-mon[29276]: 6.0 scrub ok 2024-09-11T08:12:18.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:18 smithi150 ceph-mon[29276]: 7.2 scrub starts 2024-09-11T08:12:18.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:18 smithi150 ceph-mon[39612]: 3.d scrub starts 2024-09-11T08:12:18.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:18 smithi150 ceph-mon[39612]: 3.d scrub ok 2024-09-11T08:12:18.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:18 smithi150 ceph-mon[39612]: 6.7 scrub starts 2024-09-11T08:12:18.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:18 smithi150 ceph-mon[39612]: pgmap v192: 148 pgs: 148 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 909 op/s 2024-09-11T08:12:18.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:18 smithi150 ceph-mon[39612]: 6.7 scrub ok 2024-09-11T08:12:18.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:18 smithi150 ceph-mon[39612]: 6.0 scrub starts 2024-09-11T08:12:18.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:18 smithi150 ceph-mon[39612]: 6.0 scrub ok 2024-09-11T08:12:18.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:18 smithi150 ceph-mon[39612]: 7.2 scrub starts 2024-09-11T08:12:18.261 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:18.295 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:18.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:18.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:18.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:17 smithi181 ceph-mon[32701]: 3.d scrub starts 2024-09-11T08:12:18.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:17 smithi181 ceph-mon[32701]: 3.d scrub ok 2024-09-11T08:12:18.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:17 smithi181 ceph-mon[32701]: 6.7 scrub starts 2024-09-11T08:12:18.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:17 smithi181 ceph-mon[32701]: pgmap v192: 148 pgs: 148 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 909 op/s 2024-09-11T08:12:18.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:18 smithi181 ceph-mon[32701]: 6.7 scrub ok 2024-09-11T08:12:18.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:18 smithi181 ceph-mon[32701]: 6.0 scrub starts 2024-09-11T08:12:18.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:18 smithi181 ceph-mon[32701]: 6.0 scrub ok 2024-09-11T08:12:18.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:18 smithi181 ceph-mon[32701]: 7.2 scrub starts 2024-09-11T08:12:18.397 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:18.454 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:18.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:18.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:18.556 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:18.607 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:12:18.618 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:18.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:18.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:18.677 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-11T08:12:18.677 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:12:18.720 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:18.749 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:18.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:18.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:18.852 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:12:18.884 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:18.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:18.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:12:18.986 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:12:18.989 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:12:18 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:12:18] "GET /metrics HTTP/1.1" 200 223810 "" "Prometheus/2.43.0" 2024-09-11T08:12:19.021 INFO:teuthology.orchestra.run.smithi150.stderr:set pool 7 pgp_num to 16 2024-09-11T08:12:19.022 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:19.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:19.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:12:19.123 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:19.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:19.168 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:19.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:19.269 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:19.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:19.306 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:19.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:19.309 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:19 smithi181 ceph-mon[32701]: 5.1d scrub starts 2024-09-11T08:12:19.310 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:19 smithi181 ceph-mon[32701]: 5.1d scrub ok 2024-09-11T08:12:19.310 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:19 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:12:19.310 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:19 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2243056451' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]: dispatch 2024-09-11T08:12:19.310 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:19 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]: dispatch 2024-09-11T08:12:19.407 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:12:19.437 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:19.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:19.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:12:19.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:19 smithi150 ceph-mon[29276]: 5.1d scrub starts 2024-09-11T08:12:19.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:19 smithi150 ceph-mon[29276]: 5.1d scrub ok 2024-09-11T08:12:19.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:19 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:12:19.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:19 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2243056451' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]: dispatch 2024-09-11T08:12:19.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:19 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]: dispatch 2024-09-11T08:12:19.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:19 smithi150 ceph-mon[39612]: 5.1d scrub starts 2024-09-11T08:12:19.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:19 smithi150 ceph-mon[39612]: 5.1d scrub ok 2024-09-11T08:12:19.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:19 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:12:19.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:19 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2243056451' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]: dispatch 2024-09-11T08:12:19.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:19 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]: dispatch 2024-09-11T08:12:19.538 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:12:19.545 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-11T08:12:19.569 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:19.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:19.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:12:19.673 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:12:19.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:19.809 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:19.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:12:19.910 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:19.940 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:19.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:19.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:19.952 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:12:20.042 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:20.082 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:20.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:20.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:20.183 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:20.211 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:20.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:20.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:20.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:20 smithi150 ceph-mon[29276]: pgmap v193: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.35k op/s 2024-09-11T08:12:20.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:20 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]': finished 2024-09-11T08:12:20.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:20 smithi150 ceph-mon[29276]: osdmap e138: 8 total, 8 up, 8 in 2024-09-11T08:12:20.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:20 smithi150 ceph-mon[29276]: 4.1f scrub starts 2024-09-11T08:12:20.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:20 smithi150 ceph-mon[29276]: 4.1f scrub ok 2024-09-11T08:12:20.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:20 smithi150 ceph-mon[39612]: pgmap v193: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.35k op/s 2024-09-11T08:12:20.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:20 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]': finished 2024-09-11T08:12:20.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:20 smithi150 ceph-mon[39612]: osdmap e138: 8 total, 8 up, 8 in 2024-09-11T08:12:20.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:20 smithi150 ceph-mon[39612]: 4.1f scrub starts 2024-09-11T08:12:20.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:20 smithi150 ceph-mon[39612]: 4.1f scrub ok 2024-09-11T08:12:20.313 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:12:20.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:20 smithi181 ceph-mon[32701]: pgmap v193: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.35k op/s 2024-09-11T08:12:20.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:20 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]': finished 2024-09-11T08:12:20.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:20 smithi181 ceph-mon[32701]: osdmap e138: 8 total, 8 up, 8 in 2024-09-11T08:12:20.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:20 smithi181 ceph-mon[32701]: 4.1f scrub starts 2024-09-11T08:12:20.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:20 smithi181 ceph-mon[32701]: 4.1f scrub ok 2024-09-11T08:12:20.344 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:20.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:20.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:12:20.446 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:20.487 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:20.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:20.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:20.588 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:12:20.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:20.621 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:20.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:12:20.722 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:20.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:20.753 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:20.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:20.854 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:12:20.880 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:20.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:20.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:12:20.981 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:21.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:21.021 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:21.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:21.122 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:12:21.149 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:21.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:21.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:12:21.251 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:21.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:21.282 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:21.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:21.383 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:21.419 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:21.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:21.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:21.439 INFO:teuthology.orchestra.run.smithi150.stderr:nodeep-scrub is unset 2024-09-11T08:12:21.521 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:21.545 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:21 smithi150 ceph-mon[39612]: 4.1 scrub starts 2024-09-11T08:12:21.545 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:21 smithi150 ceph-mon[39612]: 4.1 scrub ok 2024-09-11T08:12:21.545 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:21 smithi150 ceph-mon[39612]: 7.2 scrub ok 2024-09-11T08:12:21.545 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:21 smithi150 ceph-mon[39612]: 5.6 scrub starts 2024-09-11T08:12:21.545 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:21 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1112044765' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:12:21.546 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:21 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:12:21.546 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:21 smithi150 ceph-mon[39612]: 3.10 scrub starts 2024-09-11T08:12:21.546 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:21 smithi150 ceph-mon[39612]: 3.10 scrub ok 2024-09-11T08:12:21.546 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:21 smithi150 ceph-mon[39612]: pgmap v195: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.06k op/s 2024-09-11T08:12:21.546 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:21 smithi150 ceph-mon[29276]: 4.1 scrub starts 2024-09-11T08:12:21.546 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:21 smithi150 ceph-mon[29276]: 4.1 scrub ok 2024-09-11T08:12:21.546 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:21 smithi150 ceph-mon[29276]: 7.2 scrub ok 2024-09-11T08:12:21.547 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:21 smithi150 ceph-mon[29276]: 5.6 scrub starts 2024-09-11T08:12:21.547 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:21 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1112044765' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:12:21.547 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:21 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:12:21.547 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:21 smithi150 ceph-mon[29276]: 3.10 scrub starts 2024-09-11T08:12:21.547 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:21 smithi150 ceph-mon[29276]: 3.10 scrub ok 2024-09-11T08:12:21.547 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:21 smithi150 ceph-mon[29276]: pgmap v195: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.06k op/s 2024-09-11T08:12:21.553 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:21.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:21.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:21.655 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:21.694 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:21.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:21.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:21.739 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:21 smithi181 ceph-mon[32701]: 4.1 scrub starts 2024-09-11T08:12:21.739 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:21 smithi181 ceph-mon[32701]: 4.1 scrub ok 2024-09-11T08:12:21.739 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:21 smithi181 ceph-mon[32701]: 7.2 scrub ok 2024-09-11T08:12:21.739 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:21 smithi181 ceph-mon[32701]: 5.6 scrub starts 2024-09-11T08:12:21.739 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:21 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1112044765' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:12:21.739 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:21 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:12:21.739 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:21 smithi181 ceph-mon[32701]: 3.10 scrub starts 2024-09-11T08:12:21.739 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:21 smithi181 ceph-mon[32701]: 3.10 scrub ok 2024-09-11T08:12:21.740 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:21 smithi181 ceph-mon[32701]: pgmap v195: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.06k op/s 2024-09-11T08:12:21.796 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:12:21.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:21.821 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:21.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:12:21.922 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:21.956 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:21.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:21.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:22.058 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:22.098 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:22.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:22.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:22.115 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-11T08:12:22.115 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:12:22.200 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:22.243 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:22.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:22.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:22.344 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:12:22.377 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:22.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:22.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:12:22.479 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:12:22.503 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:22.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:22.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:12:22.605 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:12:22.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:22.627 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:22.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:12:22.728 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:12:22.741 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:22 smithi150 ceph-mon[29276]: 5.6 scrub ok 2024-09-11T08:12:22.741 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:22 smithi150 ceph-mon[29276]: 7.6 scrub starts 2024-09-11T08:12:22.741 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:22 smithi150 ceph-mon[29276]: 3.e scrub starts 2024-09-11T08:12:22.741 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:22 smithi150 ceph-mon[29276]: 3.e scrub ok 2024-09-11T08:12:22.741 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:22 smithi150 ceph-mon[29276]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:12:22.741 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:22 smithi150 ceph-mon[29276]: Cluster is now healthy 2024-09-11T08:12:22.741 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:22 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:12:22.741 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:22 smithi150 ceph-mon[29276]: osdmap e139: 8 total, 8 up, 8 in 2024-09-11T08:12:22.742 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:22 smithi150 ceph-mon[29276]: 5.8 scrub starts 2024-09-11T08:12:22.742 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:22 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:12:22.742 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:22 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:12:22.742 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:22 smithi150 ceph-mon[29276]: 3.4 scrub starts 2024-09-11T08:12:22.742 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:22 smithi150 ceph-mon[29276]: 3.4 scrub ok 2024-09-11T08:12:22.743 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:22 smithi150 ceph-mon[39612]: 5.6 scrub ok 2024-09-11T08:12:22.743 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:22 smithi150 ceph-mon[39612]: 7.6 scrub starts 2024-09-11T08:12:22.743 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:22 smithi150 ceph-mon[39612]: 3.e scrub starts 2024-09-11T08:12:22.743 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:22 smithi150 ceph-mon[39612]: 3.e scrub ok 2024-09-11T08:12:22.743 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:22 smithi150 ceph-mon[39612]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:12:22.743 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:22 smithi150 ceph-mon[39612]: Cluster is now healthy 2024-09-11T08:12:22.743 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:22 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:12:22.743 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:22 smithi150 ceph-mon[39612]: osdmap e139: 8 total, 8 up, 8 in 2024-09-11T08:12:22.744 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:22 smithi150 ceph-mon[39612]: 5.8 scrub starts 2024-09-11T08:12:22.744 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:22 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:12:22.744 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:22 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:12:22.744 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:22 smithi150 ceph-mon[39612]: 3.4 scrub starts 2024-09-11T08:12:22.744 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:22 smithi150 ceph-mon[39612]: 3.4 scrub ok 2024-09-11T08:12:22.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:22.756 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:22.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:12:22.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:22 smithi181 ceph-mon[32701]: 5.6 scrub ok 2024-09-11T08:12:22.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:22 smithi181 ceph-mon[32701]: 7.6 scrub starts 2024-09-11T08:12:22.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:22 smithi181 ceph-mon[32701]: 3.e scrub starts 2024-09-11T08:12:22.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:22 smithi181 ceph-mon[32701]: 3.e scrub ok 2024-09-11T08:12:22.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:22 smithi181 ceph-mon[32701]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:12:22.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:22 smithi181 ceph-mon[32701]: Cluster is now healthy 2024-09-11T08:12:22.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:22 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:12:22.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:22 smithi181 ceph-mon[32701]: osdmap e139: 8 total, 8 up, 8 in 2024-09-11T08:12:22.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:22 smithi181 ceph-mon[32701]: 5.8 scrub starts 2024-09-11T08:12:22.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:22 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:12:22.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:22 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:12:22.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:22 smithi181 ceph-mon[32701]: 3.4 scrub starts 2024-09-11T08:12:22.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:22 smithi181 ceph-mon[32701]: 3.4 scrub ok 2024-09-11T08:12:22.793 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-11T08:12:22.857 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:22.892 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:22.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:22.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:22.999 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:23.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:23.039 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:23.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:23.140 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:12:23.172 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:12:23.172 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:23.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:23.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:12:23.274 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:23.318 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:23.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:23.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:23.420 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:23.442 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:23.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:23.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:23.544 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:12:23.561 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:23 smithi181 ceph-mon[32701]: 5.8 scrub ok 2024-09-11T08:12:23.561 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:23 smithi181 ceph-mon[32701]: 3.2 scrub starts 2024-09-11T08:12:23.561 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:23 smithi181 ceph-mon[32701]: 3.2 scrub ok 2024-09-11T08:12:23.562 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:23 smithi181 ceph-mon[32701]: 6.19 scrub starts 2024-09-11T08:12:23.562 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:23 smithi181 ceph-mon[32701]: 6.19 scrub ok 2024-09-11T08:12:23.562 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:23 smithi181 ceph-mon[32701]: pgmap v197: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 897 op/s 2024-09-11T08:12:23.562 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:23 smithi181 ceph-mon[32701]: 5.c scrub starts 2024-09-11T08:12:23.562 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:23 smithi181 ceph-mon[32701]: 5.c scrub ok 2024-09-11T08:12:23.562 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:23 smithi181 ceph-mon[32701]: 6.e scrub starts 2024-09-11T08:12:23.562 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:23 smithi181 ceph-mon[32701]: 6.e scrub ok 2024-09-11T08:12:23.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:23.591 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:23.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:12:23.620 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:23 smithi150 ceph-mon[29276]: 5.8 scrub ok 2024-09-11T08:12:23.620 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:23 smithi150 ceph-mon[29276]: 3.2 scrub starts 2024-09-11T08:12:23.620 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:23 smithi150 ceph-mon[29276]: 3.2 scrub ok 2024-09-11T08:12:23.621 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:23 smithi150 ceph-mon[29276]: 6.19 scrub starts 2024-09-11T08:12:23.621 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:23 smithi150 ceph-mon[29276]: 6.19 scrub ok 2024-09-11T08:12:23.621 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:23 smithi150 ceph-mon[29276]: pgmap v197: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 897 op/s 2024-09-11T08:12:23.621 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:23 smithi150 ceph-mon[29276]: 5.c scrub starts 2024-09-11T08:12:23.621 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:23 smithi150 ceph-mon[29276]: 5.c scrub ok 2024-09-11T08:12:23.621 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:23 smithi150 ceph-mon[29276]: 6.e scrub starts 2024-09-11T08:12:23.621 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:23 smithi150 ceph-mon[29276]: 6.e scrub ok 2024-09-11T08:12:23.622 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:23 smithi150 ceph-mon[39612]: 5.8 scrub ok 2024-09-11T08:12:23.622 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:23 smithi150 ceph-mon[39612]: 3.2 scrub starts 2024-09-11T08:12:23.622 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:23 smithi150 ceph-mon[39612]: 3.2 scrub ok 2024-09-11T08:12:23.622 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:23 smithi150 ceph-mon[39612]: 6.19 scrub starts 2024-09-11T08:12:23.622 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:23 smithi150 ceph-mon[39612]: 6.19 scrub ok 2024-09-11T08:12:23.623 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:23 smithi150 ceph-mon[39612]: pgmap v197: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 897 op/s 2024-09-11T08:12:23.623 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:23 smithi150 ceph-mon[39612]: 5.c scrub starts 2024-09-11T08:12:23.623 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:23 smithi150 ceph-mon[39612]: 5.c scrub ok 2024-09-11T08:12:23.623 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:23 smithi150 ceph-mon[39612]: 6.e scrub starts 2024-09-11T08:12:23.623 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:23 smithi150 ceph-mon[39612]: 6.e scrub ok 2024-09-11T08:12:23.693 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:23.731 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:23.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:23.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:23.833 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:23.871 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:23.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:23.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:23.973 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:24.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:24.010 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:24.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:24.111 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:24.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:24.162 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:24.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:24.263 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:24.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:24.299 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:24.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:24.400 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:12:24.426 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:24.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:24.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:12:24.528 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:24.564 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:24.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:24.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:24.666 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:12:24.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:24.706 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:24.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:12:24.723 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:24 smithi150 ceph-mon[29276]: 5.4 scrub starts 2024-09-11T08:12:24.723 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:24 smithi150 ceph-mon[29276]: 5.4 scrub ok 2024-09-11T08:12:24.723 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:24 smithi150 ceph-mon[29276]: 7.6 scrub ok 2024-09-11T08:12:24.723 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:24 smithi150 ceph-mon[29276]: 4.7 scrub starts 2024-09-11T08:12:24.723 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:24 smithi150 ceph-mon[29276]: 4.7 scrub ok 2024-09-11T08:12:24.723 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:24 smithi150 ceph-mon[39612]: 5.4 scrub starts 2024-09-11T08:12:24.723 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:24 smithi150 ceph-mon[39612]: 5.4 scrub ok 2024-09-11T08:12:24.723 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:24 smithi150 ceph-mon[39612]: 7.6 scrub ok 2024-09-11T08:12:24.723 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:24 smithi150 ceph-mon[39612]: 4.7 scrub starts 2024-09-11T08:12:24.724 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:24 smithi150 ceph-mon[39612]: 4.7 scrub ok 2024-09-11T08:12:24.725 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:24 smithi181 ceph-mon[32701]: 5.4 scrub starts 2024-09-11T08:12:24.725 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:24 smithi181 ceph-mon[32701]: 5.4 scrub ok 2024-09-11T08:12:24.725 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:24 smithi181 ceph-mon[32701]: 7.6 scrub ok 2024-09-11T08:12:24.726 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:24 smithi181 ceph-mon[32701]: 4.7 scrub starts 2024-09-11T08:12:24.726 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:24 smithi181 ceph-mon[32701]: 4.7 scrub ok 2024-09-11T08:12:24.799 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:12:24.799 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:12:24.807 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:24.830 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:24.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:24.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:24.841 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:12:24.841 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:12:24.875 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:12:24.875 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:12:24.916 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:12:24.916 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:12:24.932 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:24.949 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:12:24.949 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:12:24.969 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:24.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:24.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:24.975 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:12:24.976 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:12:25.011 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:12:25.011 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:12:25.040 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:12:25.041 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:12:25.072 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:25.119 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:25.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:25.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:25.220 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:12:25.261 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:25.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:25.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:12:25.336 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T08:12:25.363 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:12:25.389 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:25.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:25.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:12:25.463 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:12:25 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:12:25] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:12:25.490 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:25.531 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:25.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:25.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:25.633 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:12:25.660 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:25.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:25.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:12:25.723 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:25 smithi181 ceph-mon[32701]: 3.1e scrub starts 2024-09-11T08:12:25.724 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:25 smithi181 ceph-mon[32701]: 3.1e scrub ok 2024-09-11T08:12:25.724 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:25 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1683167262' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:12:25.724 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:25 smithi181 ceph-mon[32701]: pgmap v198: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-09-11T08:12:25.724 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:25 smithi181 ceph-mon[32701]: 4.12 scrub starts 2024-09-11T08:12:25.724 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:25 smithi181 ceph-mon[32701]: 4.12 scrub ok 2024-09-11T08:12:25.724 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:25 smithi181 ceph-mon[32701]: 4.1b deep-scrub starts 2024-09-11T08:12:25.724 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:25 smithi181 ceph-mon[32701]: 4.1b deep-scrub ok 2024-09-11T08:12:25.725 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-11T08:12:25.725 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-09-11T08:12:25.725 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:25.725 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14790.0:183905 7.7 7:e0b9cc20:::benchmark_data_smithi150_78508_object59464:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-11T08:12:25.726 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:12:25.707141+0000", 2024-09-11T08:12:25.726 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.015414746999999999, 2024-09-11T08:12:25.726 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.015444756, 2024-09-11T08:12:25.726 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:12:25.726 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "waiting for sub ops", 2024-09-11T08:12:25.726 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:12:25.727 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14790", 2024-09-11T08:12:25.727 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:12:25.727 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 183905 2024-09-11T08:12:25.727 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:25.728 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:12:25.728 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:25.728 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:12:25.728 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:25.707141+0000", 2024-09-11T08:12:25.728 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:12:25.728 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:25.729 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:25.729 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:12:25.729 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:25.707141+0000", 2024-09-11T08:12:25.729 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:12:25.729 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:25.729 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:25.729 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:12:25.729 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:25.707140+0000", 2024-09-11T08:12:25.729 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999981 2024-09-11T08:12:25.729 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:25.729 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:25.729 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:12:25.730 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:25.707147+0000", 2024-09-11T08:12:25.730 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0700000000000001e-06 2024-09-11T08:12:25.730 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:25.730 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:25.730 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:12:25.730 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:25.707157+0000", 2024-09-11T08:12:25.730 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0575e-05 2024-09-11T08:12:25.730 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:25.730 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:25.730 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:12:25.730 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:25.707163+0000", 2024-09-11T08:12:25.730 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2609999999999999e-06 2024-09-11T08:12:25.730 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:25.731 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:25.731 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:12:25.731 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:25.707423+0000", 2024-09-11T08:12:25.731 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000259268 2024-09-11T08:12:25.731 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:25.731 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:25.731 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:12:25.731 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:25.707529+0000", 2024-09-11T08:12:25.731 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010616300000000001 2024-09-11T08:12:25.731 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:25.731 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:25.731 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,3", 2024-09-11T08:12:25.731 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:25.707599+0000", 2024-09-11T08:12:25.732 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0271999999999994e-05 2024-09-11T08:12:25.732 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:25.732 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:25.732 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:12:25.732 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:25.714214+0000", 2024-09-11T08:12:25.732 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.006614778 2024-09-11T08:12:25.732 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:12:25.732 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:12:25.732 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:12:25.732 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:25.732 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:25.732 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14790.0:183908 7.7 7:eb077a85:::benchmark_data_smithi150_78508_object59467:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-11T08:12:25.732 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:12:25.707177+0000", 2024-09-11T08:12:25.733 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.015379406999999999, 2024-09-11T08:12:25.733 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.015505918, 2024-09-11T08:12:25.733 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:12:25.733 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "waiting for sub ops", 2024-09-11T08:12:25.733 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:12:25.733 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14790", 2024-09-11T08:12:25.733 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:12:25.733 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 183908 2024-09-11T08:12:25.733 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:25.733 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:12:25.733 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:25.733 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:12:25.734 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:25.707177+0000", 2024-09-11T08:12:25.734 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:12:25.734 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:25.734 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:25.734 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:12:25.734 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:25.707177+0000", 2024-09-11T08:12:25.734 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:12:25.734 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:25.734 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:25.734 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:12:25.734 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:25.707176+0000", 2024-09-11T08:12:25.734 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-09-11T08:12:25.734 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:25.734 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:25.735 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:12:25.735 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:25.707179+0000", 2024-09-11T08:12:25.735 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.689e-06 2024-09-11T08:12:25.735 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:25.735 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:25.735 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:12:25.735 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:25.707184+0000", 2024-09-11T08:12:25.735 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5099999999999998e-06 2024-09-11T08:12:25.735 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:25.735 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:25.735 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:12:25.735 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:25.707190+0000", 2024-09-11T08:12:25.735 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2700000000000004e-06 2024-09-11T08:12:25.735 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:25.736 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:25.736 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:12:25.736 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:25.710533+0000", 2024-09-11T08:12:25.736 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0033431860000000002 2024-09-11T08:12:25.736 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:25.736 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:25.736 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:12:25.736 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:25.710712+0000", 2024-09-11T08:12:25.736 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000178629 2024-09-11T08:12:25.736 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:25.736 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:25.736 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,3", 2024-09-11T08:12:25.736 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:25.711181+0000", 2024-09-11T08:12:25.737 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00046994099999999998 2024-09-11T08:12:25.737 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:25.737 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:25.737 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:12:25.737 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:25.714528+0000", 2024-09-11T08:12:25.737 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0033469179999999999 2024-09-11T08:12:25.737 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:12:25.737 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:12:25.737 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:12:25.737 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:25.737 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:25.737 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14790.0:183918 7.7 7:e239d101:::benchmark_data_smithi150_78508_object59477:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-11T08:12:25.737 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:12:25.718046+0000", 2024-09-11T08:12:25.737 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.0045101639999999997, 2024-09-11T08:12:25.738 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0047095729999999999, 2024-09-11T08:12:25.738 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:12:25.738 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "waiting for sub ops", 2024-09-11T08:12:25.738 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:12:25.738 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14790", 2024-09-11T08:12:25.738 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:12:25.738 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 183918 2024-09-11T08:12:25.738 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:25.738 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:12:25.738 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:25.738 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:12:25.738 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:25.718046+0000", 2024-09-11T08:12:25.738 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:12:25.738 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:25.739 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:25.739 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:12:25.739 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:25.718046+0000", 2024-09-11T08:12:25.739 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:12:25.739 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:25.739 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:25.739 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:12:25.739 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:25.718045+0000", 2024-09-11T08:12:25.739 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:12:25.739 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:25.739 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:25.739 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:12:25.739 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:25.718053+0000", 2024-09-11T08:12:25.739 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3990000000000004e-06 2024-09-11T08:12:25.740 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:25.740 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:25.740 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:12:25.740 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:25.718064+0000", 2024-09-11T08:12:25.740 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0618999999999999e-05 2024-09-11T08:12:25.740 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:25.740 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:25.740 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:12:25.740 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:25.718068+0000", 2024-09-11T08:12:25.740 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.313e-06 2024-09-11T08:12:25.740 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:25.740 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:25.740 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:12:25.740 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:25.718105+0000", 2024-09-11T08:12:25.741 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6859000000000001e-05 2024-09-11T08:12:25.741 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:25.741 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:25.741 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:12:25.741 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:25.718225+0000", 2024-09-11T08:12:25.741 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011966 2024-09-11T08:12:25.741 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:25.741 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:25.741 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,3", 2024-09-11T08:12:25.741 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:25.718336+0000", 2024-09-11T08:12:25.741 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000111292 2024-09-11T08:12:25.741 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:12:25.741 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:12:25.742 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:12:25.742 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:25.742 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:25.742 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14790.0:183920 7.c 7:3686da7f:::benchmark_data_smithi150_78508_object59479:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-11T08:12:25.742 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:12:25.720053+0000", 2024-09-11T08:12:25.742 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.002503112, 2024-09-11T08:12:25.742 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0027684770000000001, 2024-09-11T08:12:25.742 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:12:25.742 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "waiting for sub ops", 2024-09-11T08:12:25.742 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:12:25.742 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14790", 2024-09-11T08:12:25.742 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:12:25.742 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 183920 2024-09-11T08:12:25.742 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:25.743 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:12:25.743 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:25.743 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:12:25.743 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:25.720053+0000", 2024-09-11T08:12:25.743 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:12:25.743 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:25.743 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:25.743 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:12:25.743 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:25.720053+0000", 2024-09-11T08:12:25.743 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:12:25.743 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:25.743 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:25.743 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:12:25.743 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:25.720052+0000", 2024-09-11T08:12:25.744 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:12:25.744 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:25.744 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:25.744 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:12:25.744 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:25.720058+0000", 2024-09-11T08:12:25.744 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1070000000000001e-06 2024-09-11T08:12:25.744 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:25.744 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:25.744 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:12:25.744 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:25.720072+0000", 2024-09-11T08:12:25.744 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3774000000000001e-05 2024-09-11T08:12:25.744 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:25.744 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:25.744 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:12:25.745 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:25.720078+0000", 2024-09-11T08:12:25.745 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3690000000000003e-06 2024-09-11T08:12:25.745 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:25.745 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:25.745 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:12:25.745 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:25.720117+0000", 2024-09-11T08:12:25.745 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.892e-05 2024-09-11T08:12:25.745 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:25.745 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:25.745 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:12:25.745 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:25.720226+0000", 2024-09-11T08:12:25.745 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000108656 2024-09-11T08:12:25.745 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:25.746 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:25.746 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3,7", 2024-09-11T08:12:25.746 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:25.720336+0000", 2024-09-11T08:12:25.746 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011020599999999999 2024-09-11T08:12:25.746 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:12:25.746 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:12:25.746 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:12:25.746 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:25.746 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:25.746 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14790.0:183922 7.7 7:e8ae4dfa:::benchmark_data_smithi150_78508_object59481:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-11T08:12:25.746 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:12:25.721232+0000", 2024-09-11T08:12:25.746 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.0013236260000000001, 2024-09-11T08:12:25.746 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0016530819999999999, 2024-09-11T08:12:25.746 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:12:25.747 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "waiting for sub ops", 2024-09-11T08:12:25.747 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:12:25.747 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14790", 2024-09-11T08:12:25.747 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:12:25.747 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 183922 2024-09-11T08:12:25.747 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:25.747 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:12:25.747 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:25.747 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:12:25.747 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:25.721232+0000", 2024-09-11T08:12:25.747 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:12:25.747 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:25.747 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:25.747 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:12:25.748 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:25.721232+0000", 2024-09-11T08:12:25.748 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:12:25.748 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:25.748 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:25.748 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:12:25.748 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:25.721232+0000", 2024-09-11T08:12:25.748 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:12:25.748 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:25.748 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:25.748 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:12:25.748 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:25.721237+0000", 2024-09-11T08:12:25.748 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3789999999999997e-06 2024-09-11T08:12:25.748 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:25.748 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:25.749 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:12:25.749 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:25.721246+0000", 2024-09-11T08:12:25.749 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.3319999999999998e-06 2024-09-11T08:12:25.749 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:25.749 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:25.749 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:12:25.749 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:25.721252+0000", 2024-09-11T08:12:25.749 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.733e-06 2024-09-11T08:12:25.749 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:25.749 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:25.749 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:12:25.749 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:25.721308+0000", 2024-09-11T08:12:25.749 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5773000000000003e-05 2024-09-11T08:12:25.749 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:25.750 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:25.750 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:12:25.750 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:25.721436+0000", 2024-09-11T08:12:25.750 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012799099999999999 2024-09-11T08:12:25.750 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:12:25.750 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:12:25.750 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,3", 2024-09-11T08:12:25.750 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:25.721578+0000", 2024-09-11T08:12:25.750 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014219499999999999 2024-09-11T08:12:25.750 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:12:25.750 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:12:25.750 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:12:25.750 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:12:25.750 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-09-11T08:12:25.751 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 5 2024-09-11T08:12:25.751 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-11T08:12:25.751 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:25 smithi150 ceph-mon[29276]: 3.1e scrub starts 2024-09-11T08:12:25.751 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:25 smithi150 ceph-mon[29276]: 3.1e scrub ok 2024-09-11T08:12:25.751 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:25 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1683167262' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:12:25.751 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:25 smithi150 ceph-mon[29276]: pgmap v198: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-09-11T08:12:25.751 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:25 smithi150 ceph-mon[29276]: 4.12 scrub starts 2024-09-11T08:12:25.752 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:25 smithi150 ceph-mon[29276]: 4.12 scrub ok 2024-09-11T08:12:25.752 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:25 smithi150 ceph-mon[29276]: 4.1b deep-scrub starts 2024-09-11T08:12:25.752 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:25 smithi150 ceph-mon[29276]: 4.1b deep-scrub ok 2024-09-11T08:12:25.752 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:25 smithi150 ceph-mon[39612]: 3.1e scrub starts 2024-09-11T08:12:25.752 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:25 smithi150 ceph-mon[39612]: 3.1e scrub ok 2024-09-11T08:12:25.752 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:25 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1683167262' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:12:25.752 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:25 smithi150 ceph-mon[39612]: pgmap v198: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-09-11T08:12:25.752 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:25 smithi150 ceph-mon[39612]: 4.12 scrub starts 2024-09-11T08:12:25.752 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:25 smithi150 ceph-mon[39612]: 4.12 scrub ok 2024-09-11T08:12:25.752 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:25 smithi150 ceph-mon[39612]: 4.1b deep-scrub starts 2024-09-11T08:12:25.752 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:25 smithi150 ceph-mon[39612]: 4.1b deep-scrub ok 2024-09-11T08:12:25.762 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:25.792 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:25.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:25.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:25.880 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-11T08:12:25.880 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T08:12:25.880 INFO:tasks.thrashosds.thrasher:Killing osd 7, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-11T08:12:25.881 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-11T08:12:25.881 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-c1bddff4-7013-11ef-bcea-c7b262605968@osd.7 2024-09-11T08:12:25.893 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:25.936 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:12:25.956 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:25.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:25.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:26.058 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:26.073 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:12:25 smithi181 systemd[1]: Stopping Ceph osd.7 for c1bddff4-7013-11ef-bcea-c7b262605968... 2024-09-11T08:12:26.103 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:26.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:26.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:26.206 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:26.241 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:26.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:26.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:26.342 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:26.383 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:26.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:26.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:26.485 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:26.522 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:26.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:26.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:26.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:26 smithi181 ceph-mon[32701]: 3.1c scrub starts 2024-09-11T08:12:26.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:26 smithi181 ceph-mon[32701]: 3.1c scrub ok 2024-09-11T08:12:26.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:26 smithi181 ceph-mon[32701]: 6.1d deep-scrub starts 2024-09-11T08:12:26.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:26 smithi181 ceph-mon[32701]: 6.1d deep-scrub ok 2024-09-11T08:12:26.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:26 smithi181 ceph-mon[32701]: 6.2 deep-scrub starts 2024-09-11T08:12:26.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:26 smithi181 ceph-mon[32701]: 6.2 deep-scrub ok 2024-09-11T08:12:26.624 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:26.657 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:26.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:26.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:26.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:26 smithi150 ceph-mon[29276]: 3.1c scrub starts 2024-09-11T08:12:26.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:26 smithi150 ceph-mon[29276]: 3.1c scrub ok 2024-09-11T08:12:26.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:26 smithi150 ceph-mon[29276]: 6.1d deep-scrub starts 2024-09-11T08:12:26.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:26 smithi150 ceph-mon[29276]: 6.1d deep-scrub ok 2024-09-11T08:12:26.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:26 smithi150 ceph-mon[29276]: 6.2 deep-scrub starts 2024-09-11T08:12:26.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:26 smithi150 ceph-mon[29276]: 6.2 deep-scrub ok 2024-09-11T08:12:26.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:26 smithi150 ceph-mon[39612]: 3.1c scrub starts 2024-09-11T08:12:26.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:26 smithi150 ceph-mon[39612]: 3.1c scrub ok 2024-09-11T08:12:26.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:26 smithi150 ceph-mon[39612]: 6.1d deep-scrub starts 2024-09-11T08:12:26.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:26 smithi150 ceph-mon[39612]: 6.1d deep-scrub ok 2024-09-11T08:12:26.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:26 smithi150 ceph-mon[39612]: 6.2 deep-scrub starts 2024-09-11T08:12:26.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:26 smithi150 ceph-mon[39612]: 6.2 deep-scrub ok 2024-09-11T08:12:26.759 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:26.764 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-11T08:12:26.799 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:26.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:26.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:26.901 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:26.936 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:26.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:26.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:27.038 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:27.063 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:12:27 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[55141]: 2024-09-11T08:12:27.060+0000 7f7b786cf640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-11T08:12:27.064 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:12:27 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[55141]: 2024-09-11T08:12:27.060+0000 7f7b786cf640 -1 osd.7 139 *** Got signal Terminated *** 2024-09-11T08:12:27.064 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:12:27 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[55141]: 2024-09-11T08:12:27.060+0000 7f7b786cf640 -1 osd.7 139 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-11T08:12:27.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:27.067 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:27.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:27.153 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:12:27.168 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:27.205 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:27.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:27.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:27.307 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:12:27.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:27.360 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:27.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:12:27.461 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:27.503 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:27.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:27.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:27.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:27 smithi181 ceph-mon[32701]: 3.13 scrub starts 2024-09-11T08:12:27.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:27 smithi181 ceph-mon[32701]: 3.13 scrub ok 2024-09-11T08:12:27.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:27 smithi181 ceph-mon[32701]: 3.1d scrub starts 2024-09-11T08:12:27.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:27 smithi181 ceph-mon[32701]: 3.1d scrub ok 2024-09-11T08:12:27.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:27 smithi181 ceph-mon[32701]: pgmap v199: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 659 op/s 2024-09-11T08:12:27.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:27 smithi181 ceph-mon[32701]: 6.17 scrub starts 2024-09-11T08:12:27.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:27 smithi181 ceph-mon[32701]: 6.17 scrub ok 2024-09-11T08:12:27.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:27 smithi181 ceph-mon[32701]: osd.7 marked itself down and dead 2024-09-11T08:12:27.604 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:27.627 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:27.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:27.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:27.729 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:27.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:27 smithi150 ceph-mon[29276]: 3.13 scrub starts 2024-09-11T08:12:27.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:27 smithi150 ceph-mon[29276]: 3.13 scrub ok 2024-09-11T08:12:27.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:27 smithi150 ceph-mon[29276]: 3.1d scrub starts 2024-09-11T08:12:27.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:27 smithi150 ceph-mon[29276]: 3.1d scrub ok 2024-09-11T08:12:27.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:27 smithi150 ceph-mon[29276]: pgmap v199: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 659 op/s 2024-09-11T08:12:27.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:27 smithi150 ceph-mon[29276]: 6.17 scrub starts 2024-09-11T08:12:27.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:27 smithi150 ceph-mon[29276]: 6.17 scrub ok 2024-09-11T08:12:27.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:27 smithi150 ceph-mon[29276]: osd.7 marked itself down and dead 2024-09-11T08:12:27.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:27 smithi150 ceph-mon[39612]: 3.13 scrub starts 2024-09-11T08:12:27.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:27 smithi150 ceph-mon[39612]: 3.13 scrub ok 2024-09-11T08:12:27.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:27 smithi150 ceph-mon[39612]: 3.1d scrub starts 2024-09-11T08:12:27.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:27 smithi150 ceph-mon[39612]: 3.1d scrub ok 2024-09-11T08:12:27.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:27 smithi150 ceph-mon[39612]: pgmap v199: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 659 op/s 2024-09-11T08:12:27.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:27 smithi150 ceph-mon[39612]: 6.17 scrub starts 2024-09-11T08:12:27.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:27 smithi150 ceph-mon[39612]: 6.17 scrub ok 2024-09-11T08:12:27.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:27 smithi150 ceph-mon[39612]: osd.7 marked itself down and dead 2024-09-11T08:12:27.751 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:27.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:27.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:27.825 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T08:12:27.853 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:27.889 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:27.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:27.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:27.991 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:28.013 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:28.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:28.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:28.077 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:12:28.116 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:12:28.139 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:28.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:28.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:12:28.241 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:12:28.262 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:28.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:28.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:12:28.323 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:12:27 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:12:28.324 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:12:28 smithi181 podman[77435]: 2024-09-11 08:12:28.043201308 +0000 UTC m=+1.172400756 container died 87dec57510908b0270f45ebcd460003b1c3270350cdc98afa34564dde89c365b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, io.buildah.version=1.37.2, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-11T08:12:28.364 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:12:28.386 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:28.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:28.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:12:28.487 INFO:teuthology.orchestra.run.smithi150.stderr:noscrub is set 2024-09-11T08:12:28.488 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:28.516 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:28.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:28.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:28.618 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:28.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:28.654 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:28.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:28.710 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:28 smithi181 ceph-mon[32701]: 7.f scrub starts 2024-09-11T08:12:28.710 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:28 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:12:28.710 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:28 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:12:28.710 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:28 smithi181 ceph-mon[32701]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-11T08:12:28.710 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:28 smithi181 ceph-mon[32701]: osdmap e140: 8 total, 7 up, 8 in 2024-09-11T08:12:28.710 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:28 smithi181 ceph-mon[32701]: 6.4 scrub starts 2024-09-11T08:12:28.710 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:28 smithi181 ceph-mon[32701]: 6.4 scrub ok 2024-09-11T08:12:28.710 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:28 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:12:28.710 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:28 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1807491949' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:12:28.711 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:12:28 smithi181 podman[77435]: 2024-09-11 08:12:28.498849347 +0000 UTC m=+1.628048756 container cleanup 87dec57510908b0270f45ebcd460003b1c3270350cdc98afa34564dde89c365b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD) 2024-09-11T08:12:28.711 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:12:28 smithi181 bash[77435]: ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7 2024-09-11T08:12:28.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:28 smithi150 ceph-mon[29276]: 7.f scrub starts 2024-09-11T08:12:28.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:28 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:12:28.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:28 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:12:28.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:28 smithi150 ceph-mon[29276]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-11T08:12:28.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:28 smithi150 ceph-mon[29276]: osdmap e140: 8 total, 7 up, 8 in 2024-09-11T08:12:28.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:28 smithi150 ceph-mon[29276]: 6.4 scrub starts 2024-09-11T08:12:28.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:28 smithi150 ceph-mon[29276]: 6.4 scrub ok 2024-09-11T08:12:28.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:28 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:12:28.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:28 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1807491949' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:12:28.740 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:12:28 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:12:28] "GET /metrics HTTP/1.1" 200 223854 "" "Prometheus/2.43.0" 2024-09-11T08:12:28.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:28 smithi150 ceph-mon[39612]: 7.f scrub starts 2024-09-11T08:12:28.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:28 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:12:28.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:28 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:12:28.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:28 smithi150 ceph-mon[39612]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-11T08:12:28.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:28 smithi150 ceph-mon[39612]: osdmap e140: 8 total, 7 up, 8 in 2024-09-11T08:12:28.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:28 smithi150 ceph-mon[39612]: 6.4 scrub starts 2024-09-11T08:12:28.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:28 smithi150 ceph-mon[39612]: 6.4 scrub ok 2024-09-11T08:12:28.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:28 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:12:28.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:28 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1807491949' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:12:28.756 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:28.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:28.795 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:28.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:28.896 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:28.940 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:28.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:28.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:29.042 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:29.068 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:29.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:29.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:29.073 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:12:28 smithi181 podman[77552]: 2024-09-11 08:12:28.709558158 +0000 UTC m=+0.764367002 container remove 87dec57510908b0270f45ebcd460003b1c3270350cdc98afa34564dde89c365b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-11T08:12:29.170 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:29.193 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:29.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:29.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:29.297 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:29.319 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:29.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:29.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:29.421 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:29.453 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:29.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:29.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:29.555 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:29.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:29.596 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:29.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:29.682 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:29 smithi181 ceph-mon[32701]: 4.1c deep-scrub starts 2024-09-11T08:12:29.682 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:29 smithi181 ceph-mon[32701]: 4.1c deep-scrub ok 2024-09-11T08:12:29.682 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:29 smithi181 ceph-mon[32701]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:12:29.682 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:29 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1807491949' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:12:29.682 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:29 smithi181 ceph-mon[32701]: osdmap e141: 8 total, 7 up, 8 in 2024-09-11T08:12:29.682 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:29 smithi181 ceph-mon[32701]: pgmap v202: 148 pgs: 58 peering, 90 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.37k op/s 2024-09-11T08:12:29.697 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:29.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:29.727 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:29.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:29.828 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:29.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:29.863 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:29.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:29.938 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-11T08:12:29.939 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-11T08:12:29.939 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-09-11T08:12:29.939 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-09-11T08:12:29.939 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-11T08:12:29.964 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:29.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:29 smithi150 ceph-mon[29276]: 4.1c deep-scrub starts 2024-09-11T08:12:29.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:29 smithi150 ceph-mon[29276]: 4.1c deep-scrub ok 2024-09-11T08:12:29.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:29 smithi150 ceph-mon[29276]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:12:29.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:29 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1807491949' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:12:29.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:29 smithi150 ceph-mon[29276]: osdmap e141: 8 total, 7 up, 8 in 2024-09-11T08:12:29.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:29 smithi150 ceph-mon[29276]: pgmap v202: 148 pgs: 58 peering, 90 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.37k op/s 2024-09-11T08:12:29.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:29 smithi150 ceph-mon[39612]: 4.1c deep-scrub starts 2024-09-11T08:12:29.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:29 smithi150 ceph-mon[39612]: 4.1c deep-scrub ok 2024-09-11T08:12:29.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:29 smithi150 ceph-mon[39612]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:12:29.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:29 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1807491949' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:12:29.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:29 smithi150 ceph-mon[39612]: osdmap e141: 8 total, 7 up, 8 in 2024-09-11T08:12:29.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:29 smithi150 ceph-mon[39612]: pgmap v202: 148 pgs: 58 peering, 90 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.37k op/s 2024-09-11T08:12:30.009 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:30.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:30.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:30.111 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:12:30.161 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:30.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:30.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:12:30.266 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:30.302 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:30.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:30.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:30.404 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:30.437 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:30.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:30.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:30.538 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:30.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:30.574 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:30.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:30.575 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:30 smithi150 ceph-mon[29276]: Health check failed: Reduced data availability: 13 pgs inactive, 18 pgs peering (PG_AVAILABILITY) 2024-09-11T08:12:30.575 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:30 smithi150 ceph-mon[39612]: Health check failed: Reduced data availability: 13 pgs inactive, 18 pgs peering (PG_AVAILABILITY) 2024-09-11T08:12:30.675 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:30.705 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-11T08:12:30.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:30.712 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:30.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:30.734 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:30 smithi181 ceph-mon[32701]: Health check failed: Reduced data availability: 13 pgs inactive, 18 pgs peering (PG_AVAILABILITY) 2024-09-11T08:12:30.772 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:12:30.772 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:12:30.804 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:12:30.804 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:12:30.813 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:30.836 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:12:30.837 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:12:30.851 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:30.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:30.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:30.873 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:12:30.874 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:12:30.911 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:12:30.911 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:12:30.942 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:12:30.942 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:12:30.953 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:30.973 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:12:30.973 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:12:30.978 INFO:teuthology.orchestra.run.smithi150.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-11T08:12:30.992 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:30.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:30.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:31.031 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:12:30 smithi181 podman[77831]: 2024-09-11 08:12:30.773696571 +0000 UTC m=+0.024072422 image pull 4a918172a49766385959227c53e75e2c302e923539324792cb5bee6cad0b2f2b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40 2024-09-11T08:12:31.031 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:12:30 smithi181 podman[77831]: 2024-09-11 08:12:30.901299786 +0000 UTC m=+0.151675628 container create 37fe150ea3518f3322437a16c36ac54bdba08d33c54335ab0257bfea6d783b33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, ceph=True, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-09-11T08:12:31.094 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:31.121 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:31.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:31.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:31.201 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T08:12:31.223 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:31.284 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:31.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:31.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:31.324 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:12:31 smithi181 podman[77831]: 2024-09-11 08:12:31.146009547 +0000 UTC m=+0.396385387 container init 37fe150ea3518f3322437a16c36ac54bdba08d33c54335ab0257bfea6d783b33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-11T08:12:31.324 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:12:31 smithi181 podman[77831]: 2024-09-11 08:12:31.149664171 +0000 UTC m=+0.400040008 container start 37fe150ea3518f3322437a16c36ac54bdba08d33c54335ab0257bfea6d783b33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-11T08:12:31.324 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:12:31 smithi181 podman[77831]: 2024-09-11 08:12:31.237820739 +0000 UTC m=+0.488196595 container attach 37fe150ea3518f3322437a16c36ac54bdba08d33c54335ab0257bfea6d783b33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-11T08:12:31.334 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:12:31.386 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:12:31.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:31.436 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:31.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:12:31.537 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:31.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:31.591 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:31.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:31.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:31 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/581247415' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:12:31.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:31 smithi181 ceph-mon[32701]: pgmap v203: 148 pgs: 58 peering, 90 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s 2024-09-11T08:12:31.689 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-11T08:12:31.690 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-11T08:12:31.692 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:31.718 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:31.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:31.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:31.746 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:12:31.820 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:12:31.846 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:31.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:31.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:12:31.878 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:31 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/581247415' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:12:31.879 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:31 smithi150 ceph-mon[29276]: pgmap v203: 148 pgs: 58 peering, 90 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s 2024-09-11T08:12:31.879 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:31 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/581247415' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:12:31.879 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:31 smithi150 ceph-mon[39612]: pgmap v203: 148 pgs: 58 peering, 90 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s 2024-09-11T08:12:31.938 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:12:31 smithi181 podman[77831]: 2024-09-11 08:12:31.618581293 +0000 UTC m=+0.868957127 container died 37fe150ea3518f3322437a16c36ac54bdba08d33c54335ab0257bfea6d783b33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-11T08:12:31.947 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:12:31.984 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:31.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:31.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:12:32.086 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:32.120 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:32.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:32.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:32.222 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:32.246 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:32.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:32.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:32.315 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c1bddff4-7013-11ef-bcea-c7b262605968@osd.7.service' 2024-09-11T08:12:32.348 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:32.370 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-11T08:12:32.371 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-11T08:12:32.371 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.7 2024-09-11T08:12:32.371 INFO:tasks.thrashosds.thrasher:remote for osd 7 is ubuntu@smithi181.front.sepia.ceph.com 2024-09-11T08:12:32.371 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -n osd.7 --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --op list-pgs 2024-09-11T08:12:32.392 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:32.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:32.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:32.494 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:32.533 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:32.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:32.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:32.635 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:32.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:32.672 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:32.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:32.773 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:32.789 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/osd.7/config 2024-09-11T08:12:32.809 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:32.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:32.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:32.911 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:32.952 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:32.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:32.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:32.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:32 smithi150 ceph-mon[29276]: Health check failed: Degraded data redundancy: 3593/185049 objects degraded (1.942%), 9 pgs degraded (PG_DEGRADED) 2024-09-11T08:12:32.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:32 smithi150 ceph-mon[39612]: Health check failed: Degraded data redundancy: 3593/185049 objects degraded (1.942%), 9 pgs degraded (PG_DEGRADED) 2024-09-11T08:12:33.055 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:33.108 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:33.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:33.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:33.209 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:12:33.233 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:33.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:33.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:12:33.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:32 smithi181 ceph-mon[32701]: Health check failed: Degraded data redundancy: 3593/185049 objects degraded (1.942%), 9 pgs degraded (PG_DEGRADED) 2024-09-11T08:12:33.335 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:12:33.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:33.384 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:33.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:12:33.485 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:33.524 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:33.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:33.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:33.626 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:12:33.650 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:33.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:33.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:12:33.691 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T08:12:33.752 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:12:33.776 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:33.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:33.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:12:33.878 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:33.900 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:33.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:33.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:33.941 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:33 smithi181 ceph-mon[32701]: pgmap v204: 148 pgs: 15 active+undersized, 34 peering, 9 active+undersized+degraded, 90 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 782 op/s; 3593/185049 objects degraded (1.942%) 2024-09-11T08:12:33.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:33 smithi150 ceph-mon[29276]: pgmap v204: 148 pgs: 15 active+undersized, 34 peering, 9 active+undersized+degraded, 90 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 782 op/s; 3593/185049 objects degraded (1.942%) 2024-09-11T08:12:33.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:33 smithi150 ceph-mon[39612]: pgmap v204: 148 pgs: 15 active+undersized, 34 peering, 9 active+undersized+degraded, 90 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 782 op/s; 3593/185049 objects degraded (1.942%) 2024-09-11T08:12:34.002 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:34.026 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:34.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:34.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:34.128 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:12:34.152 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:34.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:34.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:12:34.163 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:12:34.253 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:34.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:34.298 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:34.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:34.400 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:34.436 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:34.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:34.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:34.538 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:34.562 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:34.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:34.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:34.664 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:12:34.689 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:34.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:34.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:12:34.791 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:12:34.814 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:34.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:34.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:12:34.887 INFO:teuthology.orchestra.run.smithi150.stderr:nodeep-scrub is set 2024-09-11T08:12:34.915 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:34.938 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:34.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:34.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:35.040 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:35.069 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:35.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:35.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:35.145 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:34 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/4069091825' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:12:35.145 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:34 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:12:35.171 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:35.195 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:35.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:35.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:35.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:34 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/4069091825' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:12:35.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:34 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:12:35.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:34 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/4069091825' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:12:35.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:34 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:12:35.297 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:35.337 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:35.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:35.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:35.399 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:12:35 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:12:35] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:12:35.439 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:35.466 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:35.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:35.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:35.568 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:35.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:35.601 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:35.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:35.702 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:35.731 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:35.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:35.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:35.776 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-11T08:12:35.776 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:12:35.833 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:35.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:35.864 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:35.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:35.917 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:35 smithi181 ceph-mon[32701]: pgmap v205: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.20k op/s; 18188/175071 objects degraded (10.389%); 0 B/s, 0 objects/s recovering 2024-09-11T08:12:35.917 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:35 smithi181 ceph-mon[32701]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:12:35.917 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:35 smithi181 ceph-mon[32701]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 6 pgs peering) 2024-09-11T08:12:35.917 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:35 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:12:35.918 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:35 smithi181 ceph-mon[32701]: osdmap e142: 8 total, 7 up, 8 in 2024-09-11T08:12:35.918 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:35 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:12:35.965 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:36.008 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:36.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:36.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:36.110 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:36.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:36.143 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:36.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:36.211 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:35 smithi150 ceph-mon[29276]: pgmap v205: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.20k op/s; 18188/175071 objects degraded (10.389%); 0 B/s, 0 objects/s recovering 2024-09-11T08:12:36.211 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:35 smithi150 ceph-mon[29276]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:12:36.211 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:35 smithi150 ceph-mon[29276]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 6 pgs peering) 2024-09-11T08:12:36.211 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:35 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:12:36.212 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:35 smithi150 ceph-mon[29276]: osdmap e142: 8 total, 7 up, 8 in 2024-09-11T08:12:36.212 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:35 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:12:36.212 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:35 smithi150 ceph-mon[39612]: pgmap v205: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.20k op/s; 18188/175071 objects degraded (10.389%); 0 B/s, 0 objects/s recovering 2024-09-11T08:12:36.212 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:35 smithi150 ceph-mon[39612]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:12:36.212 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:35 smithi150 ceph-mon[39612]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 6 pgs peering) 2024-09-11T08:12:36.212 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:35 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:12:36.212 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:35 smithi150 ceph-mon[39612]: osdmap e142: 8 total, 7 up, 8 in 2024-09-11T08:12:36.213 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:35 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:12:36.244 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:36.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:36.299 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:36.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:36.400 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:36.425 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:36.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:36.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:36.526 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:36.561 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:36.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:36.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:36.663 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:36.692 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:36.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:36.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:36.794 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:36.820 INFO:teuthology.orchestra.run.smithi181.stdout:7.5 2024-09-11T08:12:36.821 INFO:teuthology.orchestra.run.smithi181.stdout:6.b 2024-09-11T08:12:36.821 INFO:teuthology.orchestra.run.smithi181.stdout:6.9 2024-09-11T08:12:36.821 INFO:teuthology.orchestra.run.smithi181.stdout:6.8 2024-09-11T08:12:36.821 INFO:teuthology.orchestra.run.smithi181.stdout:6.5 2024-09-11T08:12:36.821 INFO:teuthology.orchestra.run.smithi181.stdout:6.1c 2024-09-11T08:12:36.821 INFO:teuthology.orchestra.run.smithi181.stdout:6.1b 2024-09-11T08:12:36.821 INFO:teuthology.orchestra.run.smithi181.stdout:6.18 2024-09-11T08:12:36.821 INFO:teuthology.orchestra.run.smithi181.stdout:6.16 2024-09-11T08:12:36.821 INFO:teuthology.orchestra.run.smithi181.stdout:6.6 2024-09-11T08:12:36.821 INFO:teuthology.orchestra.run.smithi181.stdout:6.14 2024-09-11T08:12:36.821 INFO:teuthology.orchestra.run.smithi181.stdout:6.12 2024-09-11T08:12:36.821 INFO:teuthology.orchestra.run.smithi181.stdout:5.d 2024-09-11T08:12:36.822 INFO:teuthology.orchestra.run.smithi181.stdout:5.9 2024-09-11T08:12:36.822 INFO:teuthology.orchestra.run.smithi181.stdout:5.4 2024-09-11T08:12:36.822 INFO:teuthology.orchestra.run.smithi181.stdout:5.1b 2024-09-11T08:12:36.822 INFO:teuthology.orchestra.run.smithi181.stdout:5.1 2024-09-11T08:12:36.822 INFO:teuthology.orchestra.run.smithi181.stdout:5.19 2024-09-11T08:12:36.822 INFO:teuthology.orchestra.run.smithi181.stdout:5.17 2024-09-11T08:12:36.822 INFO:teuthology.orchestra.run.smithi181.stdout:6.15 2024-09-11T08:12:36.822 INFO:teuthology.orchestra.run.smithi181.stdout:3.8 2024-09-11T08:12:36.822 INFO:teuthology.orchestra.run.smithi181.stdout:3.16 2024-09-11T08:12:36.822 INFO:teuthology.orchestra.run.smithi181.stdout:3.15 2024-09-11T08:12:36.822 INFO:teuthology.orchestra.run.smithi181.stdout:5.1f 2024-09-11T08:12:36.822 INFO:teuthology.orchestra.run.smithi181.stdout:4.3 2024-09-11T08:12:36.822 INFO:teuthology.orchestra.run.smithi181.stdout:3.7 2024-09-11T08:12:36.824 INFO:teuthology.orchestra.run.smithi181.stdout:3.13 2024-09-11T08:12:36.824 INFO:teuthology.orchestra.run.smithi181.stdout:3.12 2024-09-11T08:12:36.825 INFO:teuthology.orchestra.run.smithi181.stdout:4.0 2024-09-11T08:12:36.825 INFO:teuthology.orchestra.run.smithi181.stdout:5.1e 2024-09-11T08:12:36.825 INFO:teuthology.orchestra.run.smithi181.stdout:5.11 2024-09-11T08:12:36.825 INFO:teuthology.orchestra.run.smithi181.stdout:5.1a 2024-09-11T08:12:36.825 INFO:teuthology.orchestra.run.smithi181.stdout:3.d 2024-09-11T08:12:36.825 INFO:teuthology.orchestra.run.smithi181.stdout:3.11 2024-09-11T08:12:36.825 INFO:teuthology.orchestra.run.smithi181.stdout:3.1b 2024-09-11T08:12:36.825 INFO:teuthology.orchestra.run.smithi181.stdout:3.14 2024-09-11T08:12:36.825 INFO:teuthology.orchestra.run.smithi181.stdout:2.0 2024-09-11T08:12:36.825 INFO:teuthology.orchestra.run.smithi181.stdout:4.14 2024-09-11T08:12:36.825 INFO:teuthology.orchestra.run.smithi181.stdout:1.0 2024-09-11T08:12:36.825 INFO:teuthology.orchestra.run.smithi181.stdout:3.9 2024-09-11T08:12:36.825 INFO:teuthology.orchestra.run.smithi181.stdout:4.17 2024-09-11T08:12:36.826 INFO:teuthology.orchestra.run.smithi181.stdout:7.8 2024-09-11T08:12:36.826 INFO:teuthology.orchestra.run.smithi181.stdout:3.e 2024-09-11T08:12:36.826 INFO:teuthology.orchestra.run.smithi181.stdout:5.14 2024-09-11T08:12:36.826 INFO:teuthology.orchestra.run.smithi181.stdout:7.6 2024-09-11T08:12:36.826 INFO:teuthology.orchestra.run.smithi181.stdout:6.3 2024-09-11T08:12:36.826 INFO:teuthology.orchestra.run.smithi181.stdout:3.f 2024-09-11T08:12:36.826 INFO:teuthology.orchestra.run.smithi181.stdout:7.f 2024-09-11T08:12:36.826 INFO:teuthology.orchestra.run.smithi181.stdout:5.6 2024-09-11T08:12:36.826 INFO:teuthology.orchestra.run.smithi181.stdout:4.16 2024-09-11T08:12:36.826 INFO:teuthology.orchestra.run.smithi181.stdout:4.15 2024-09-11T08:12:36.826 INFO:teuthology.orchestra.run.smithi181.stdout:4.1e 2024-09-11T08:12:36.826 INFO:teuthology.orchestra.run.smithi181.stdout:4.11 2024-09-11T08:12:36.826 INFO:teuthology.orchestra.run.smithi181.stdout:4.5 2024-09-11T08:12:36.826 INFO:teuthology.orchestra.run.smithi181.stdout:4.8 2024-09-11T08:12:36.826 INFO:teuthology.orchestra.run.smithi181.stdout:7.c 2024-09-11T08:12:36.827 INFO:teuthology.orchestra.run.smithi181.stdout:4.a 2024-09-11T08:12:36.827 INFO:teuthology.orchestra.run.smithi181.stdout:5.10 2024-09-11T08:12:36.828 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:36.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:36.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:36.842 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-11T08:12:36.930 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:36.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:36.970 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:36.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:37.071 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:37.122 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:37.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:37.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:37.224 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:37.263 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:37.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:37.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:37.365 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:37.389 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:37.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:37.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:37.491 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:37.515 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:37.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:37.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:37.617 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:37.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:37.641 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:37.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:37.742 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:37.765 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:37.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:37.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:37.867 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:37.891 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:37.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:37.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:37.944 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:37 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:12:37.944 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:37 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:12:37.944 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:37 smithi181 ceph-mon[32701]: pgmap v207: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 617 B/s rd, 0 B/s wr, 580 op/s; 18188/175071 objects degraded (10.389%); 0 B/s, 0 objects/s recovering 2024-09-11T08:12:37.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:37 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:12:37.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:37 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:12:37.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:37 smithi150 ceph-mon[29276]: pgmap v207: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 617 B/s rd, 0 B/s wr, 580 op/s; 18188/175071 objects degraded (10.389%); 0 B/s, 0 objects/s recovering 2024-09-11T08:12:37.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:37 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:12:37.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:37 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:12:37.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:37 smithi150 ceph-mon[39612]: pgmap v207: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 617 B/s rd, 0 B/s wr, 580 op/s; 18188/175071 objects degraded (10.389%); 0 B/s, 0 objects/s recovering 2024-09-11T08:12:37.993 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:12:37.998 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -n osd.7 --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --op export --pgid 4.1e --file /var/log/ceph/exp.4.1e.7 2024-09-11T08:12:38.099 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:38.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:38.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:12:38.200 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:38.228 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:38.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:38.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:38.230 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/osd.7/config 2024-09-11T08:12:38.323 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:12:37 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:12:38.331 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:38.357 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:38.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:38.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:38.459 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:38.495 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:38.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:38.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:38.597 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:12:38.625 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:38.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:38.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:12:38.638 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:12:38 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:12:38] "GET /metrics HTTP/1.1" 200 223996 "" "Prometheus/2.43.0" 2024-09-11T08:12:38.704 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:12:38.704 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:12:38.726 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:12:38.739 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:12:38.739 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:12:38.749 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:12:38.750 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:38.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:38.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:12:38.777 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:12:38.777 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:12:38.809 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:12:38.809 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:12:38.842 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:12:38.842 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:12:38.852 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:38.875 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:38.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:38.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:38.877 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:12:38.877 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:12:38.903 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:38 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:12:38.903 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:38 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:12:38.904 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:38 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:12:38.904 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:38 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:12:38.904 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:38 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1997154241' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:12:38.904 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:38 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:12:38.904 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:38 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:12:38.904 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:38 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:12:38.904 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:38 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:12:38.904 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:38 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1997154241' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:12:38.905 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:12:38.905 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:12:38.906 INFO:teuthology.orchestra.run.smithi150.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-11T08:12:38.977 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:39.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:39.019 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:39.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:39.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:38 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:12:39.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:38 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:12:39.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:38 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:12:39.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:38 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:12:39.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:38 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1997154241' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:12:39.120 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:12:39.157 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:39.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:39.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:12:39.259 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:12:39.295 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:39.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:39.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:12:39.397 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:12:39.419 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:39.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:39.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:12:39.521 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:39.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:39.548 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:39.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:39.649 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:12:39.685 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:39.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:39.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:12:39.786 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:39.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:39 smithi181 ceph-mon[32701]: pgmap v208: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 992 op/s; 16616/159735 objects degraded (10.402%); 0 B/s, 0 objects/s recovering 2024-09-11T08:12:39.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:39.826 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:39.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:39.927 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:39.968 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:39.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:39.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:40.002 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T08:12:40.045 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:39 smithi150 ceph-mon[29276]: pgmap v208: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 992 op/s; 16616/159735 objects degraded (10.402%); 0 B/s, 0 objects/s recovering 2024-09-11T08:12:40.046 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:39 smithi150 ceph-mon[39612]: pgmap v208: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 992 op/s; 16616/159735 objects degraded (10.402%); 0 B/s, 0 objects/s recovering 2024-09-11T08:12:40.070 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:40.094 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:40.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:40.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:40.196 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:12:40.220 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:40.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:40.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:12:40.322 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:12:40.327 INFO:teuthology.orchestra.run.smithi181.stderr:Exporting 4.1e info 4.1e( v 66'10 (0'0,66'10] local-lis/les=60/61 n=5 ec=60/60 lis/c=60/60 les/c/f=61/61/0 sis=60) 2024-09-11T08:12:40.329 INFO:teuthology.orchestra.run.smithi181.stderr:Read #4:7a480f67:::obj_delete_at_hint.0000000079:head# 2024-09-11T08:12:40.329 INFO:teuthology.orchestra.run.smithi181.stderr:Read #4:7a5c7c5f:::obj_delete_at_hint.0000000049:head# 2024-09-11T08:12:40.329 INFO:teuthology.orchestra.run.smithi181.stderr:Read #4:7adbb485:::obj_delete_at_hint.0000000117:head# 2024-09-11T08:12:40.329 INFO:teuthology.orchestra.run.smithi181.stderr:Read #4:7af5a05b:::obj_delete_at_hint.0000000044:head# 2024-09-11T08:12:40.329 INFO:teuthology.orchestra.run.smithi181.stderr:Read #4:7db93595:::obj_delete_at_hint.0000000066:head# 2024-09-11T08:12:40.329 INFO:teuthology.orchestra.run.smithi181.stderr:Export successful 2024-09-11T08:12:40.347 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:40.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:40.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:12:40.449 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:40.472 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:40.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:40.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:40.574 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:40.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:40.615 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:40.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:40.716 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:40.739 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:40.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:40.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:40.841 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:40.869 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:40.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:40.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:40.971 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:41.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:41.008 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:41.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:41.070 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:40 smithi150 ceph-mon[29276]: Health check update: Degraded data redundancy: 16616/159735 objects degraded (10.402%), 24 pgs degraded (PG_DEGRADED) 2024-09-11T08:12:41.070 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:40 smithi150 ceph-mon[39612]: Health check update: Degraded data redundancy: 16616/159735 objects degraded (10.402%), 24 pgs degraded (PG_DEGRADED) 2024-09-11T08:12:41.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:40 smithi181 ceph-mon[32701]: Health check update: Degraded data redundancy: 16616/159735 objects degraded (10.402%), 24 pgs degraded (PG_DEGRADED) 2024-09-11T08:12:41.109 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:41.146 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:41.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:41.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:41.248 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:41.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:41.291 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:41.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:41.392 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:41.402 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-11T08:12:41.402 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-11T08:12:41.415 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:41.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:41.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:41.517 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:41.559 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:41.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:41.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:41.661 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:12:41.683 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:41.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:41.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:12:41.784 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:41.807 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:41.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:41.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:41.860 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -n osd.7 --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --force --op remove --pgid 4.1e 2024-09-11T08:12:41.909 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:41.916 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:41 smithi150 ceph-mon[29276]: pgmap v209: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 992 op/s; 16616/159735 objects degraded (10.402%); 0 B/s, 0 objects/s recovering 2024-09-11T08:12:41.916 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:41 smithi150 ceph-mon[39612]: pgmap v209: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 992 op/s; 16616/159735 objects degraded (10.402%); 0 B/s, 0 objects/s recovering 2024-09-11T08:12:41.934 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:41.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:41.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:41.963 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-11T08:12:41.963 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [ 2024-09-11T08:12:41.963 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.963 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:200226 7.6 7:6ae5cf26:::benchmark_data_smithi150_78508_object75785:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-09-11T08:12:41.963 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:12:41.947706+0000", 2024-09-11T08:12:41.963 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.010311376000000001, 2024-09-11T08:12:41.963 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.010342474000000001, 2024-09-11T08:12:41.963 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:12:41.964 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "waiting for sub ops", 2024-09-11T08:12:41.964 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:12:41.964 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:12:41.964 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:12:41.964 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 200226 2024-09-11T08:12:41.964 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.964 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:12:41.964 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.964 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:12:41.964 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.947706+0000", 2024-09-11T08:12:41.964 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:12:41.964 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.964 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.965 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:12:41.965 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.947706+0000", 2024-09-11T08:12:41.965 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:12:41.965 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.965 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.965 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:12:41.965 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.947704+0000", 2024-09-11T08:12:41.965 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:12:41.965 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.965 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.965 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:12:41.965 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.947712+0000", 2024-09-11T08:12:41.965 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.9230000000000002e-06 2024-09-11T08:12:41.965 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.966 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.966 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:12:41.966 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.947728+0000", 2024-09-11T08:12:41.966 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.6705e-05 2024-09-11T08:12:41.966 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.966 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.966 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:12:41.966 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.947735+0000", 2024-09-11T08:12:41.966 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.8630000000000002e-06 2024-09-11T08:12:41.966 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.966 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.966 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:12:41.966 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.948168+0000", 2024-09-11T08:12:41.967 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00043263299999999998 2024-09-11T08:12:41.967 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.967 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.967 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:12:41.967 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.948304+0000", 2024-09-11T08:12:41.967 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00013595799999999999 2024-09-11T08:12:41.967 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.967 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.967 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 3", 2024-09-11T08:12:41.967 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.948428+0000", 2024-09-11T08:12:41.967 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000124368 2024-09-11T08:12:41.967 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:12:41.967 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:12:41.968 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:12:41.968 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.968 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.968 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:200227 7.4 7:29b70c73:::benchmark_data_smithi150_78508_object75786:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-09-11T08:12:41.968 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:12:41.947770+0000", 2024-09-11T08:12:41.968 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.010247615, 2024-09-11T08:12:41.968 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.010353543, 2024-09-11T08:12:41.968 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:12:41.968 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "waiting for sub ops", 2024-09-11T08:12:41.968 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:12:41.968 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:12:41.968 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:12:41.968 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 200227 2024-09-11T08:12:41.968 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.969 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:12:41.969 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.969 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:12:41.969 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.947770+0000", 2024-09-11T08:12:41.969 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:12:41.969 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.969 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.969 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:12:41.969 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.947770+0000", 2024-09-11T08:12:41.969 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:12:41.969 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.969 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.969 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:12:41.970 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.947769+0000", 2024-09-11T08:12:41.970 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:12:41.970 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.970 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.970 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:12:41.970 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.947773+0000", 2024-09-11T08:12:41.970 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.7359999999999999e-06 2024-09-11T08:12:41.970 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.970 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.970 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:12:41.970 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.947787+0000", 2024-09-11T08:12:41.970 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.456e-05 2024-09-11T08:12:41.970 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.970 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.971 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:12:41.971 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.947792+0000", 2024-09-11T08:12:41.971 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.2100000000000001e-06 2024-09-11T08:12:41.971 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.971 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.971 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:12:41.971 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.947851+0000", 2024-09-11T08:12:41.971 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.8869999999999997e-05 2024-09-11T08:12:41.971 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.971 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.971 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:12:41.971 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.948748+0000", 2024-09-11T08:12:41.971 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00089646099999999998 2024-09-11T08:12:41.971 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.972 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.972 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 1,4", 2024-09-11T08:12:41.972 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.948853+0000", 2024-09-11T08:12:41.972 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00010582 2024-09-11T08:12:41.972 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:12:41.972 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:12:41.972 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:12:41.972 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.972 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.972 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:200232 7.4 7:2543d2f4:::benchmark_data_smithi150_78508_object75791:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-09-11T08:12:41.972 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:12:41.948072+0000", 2024-09-11T08:12:41.972 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.0099452199999999994, 2024-09-11T08:12:41.972 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.010119305, 2024-09-11T08:12:41.973 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:12:41.973 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "waiting for sub ops", 2024-09-11T08:12:41.973 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:12:41.973 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:12:41.973 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:12:41.973 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 200232 2024-09-11T08:12:41.973 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.973 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:12:41.973 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.973 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:12:41.973 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.948072+0000", 2024-09-11T08:12:41.973 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:12:41.973 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.973 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.974 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:12:41.974 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.948072+0000", 2024-09-11T08:12:41.974 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:12:41.974 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.974 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.974 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:12:41.974 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.948071+0000", 2024-09-11T08:12:41.974 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:12:41.974 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.974 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.974 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:12:41.974 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.948077+0000", 2024-09-11T08:12:41.974 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.8610000000000003e-06 2024-09-11T08:12:41.974 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.975 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.975 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:12:41.975 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.948088+0000", 2024-09-11T08:12:41.975 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1317e-05 2024-09-11T08:12:41.975 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.975 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.975 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:12:41.975 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.948094+0000", 2024-09-11T08:12:41.975 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.3969999999999999e-06 2024-09-11T08:12:41.975 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.975 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.975 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:12:41.976 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.951799+0000", 2024-09-11T08:12:41.976 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0037048099999999998 2024-09-11T08:12:41.976 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.976 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.976 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:12:41.976 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.951977+0000", 2024-09-11T08:12:41.976 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00017733899999999999 2024-09-11T08:12:41.976 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.976 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.976 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 1,4", 2024-09-11T08:12:41.976 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.952083+0000", 2024-09-11T08:12:41.976 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000106622 2024-09-11T08:12:41.977 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:12:41.977 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:12:41.977 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:12:41.977 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.977 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.977 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:200234 7.4 7.9f9708d4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-09-11T08:12:41.977 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:12:41.948102+0000", 2024-09-11T08:12:41.977 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.0099148900000000009, 2024-09-11T08:12:41.977 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.010150757, 2024-09-11T08:12:41.977 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:12:41.977 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "queued for pg", 2024-09-11T08:12:41.977 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:12:41.977 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:12:41.978 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:12:41.978 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 200234 2024-09-11T08:12:41.978 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.978 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:12:41.978 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.978 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:12:41.978 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.948102+0000", 2024-09-11T08:12:41.978 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:12:41.978 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.978 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.978 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:12:41.978 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.948102+0000", 2024-09-11T08:12:41.978 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:12:41.978 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.979 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.979 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:12:41.979 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.948102+0000", 2024-09-11T08:12:41.979 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:12:41.979 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.979 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.979 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:12:41.979 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.948105+0000", 2024-09-11T08:12:41.979 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.444e-06 2024-09-11T08:12:41.979 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.979 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.979 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:12:41.979 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.948111+0000", 2024-09-11T08:12:41.979 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.9989999999999997e-06 2024-09-11T08:12:41.979 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.980 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.980 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:12:41.980 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.948115+0000", 2024-09-11T08:12:41.980 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.3549999999999998e-06 2024-09-11T08:12:41.980 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:12:41.980 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:12:41.980 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:12:41.980 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.980 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.980 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.14790.0:200225 7.7 e142/91 7:e1b1d7e9:::benchmark_data_smithi150_78508_object75784:head v 142'12545, mlcod=142'12543)", 2024-09-11T08:12:41.980 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:12:41.950912+0000", 2024-09-11T08:12:41.981 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.007105002, 2024-09-11T08:12:41.981 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0073916859999999997, 2024-09-11T08:12:41.981 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:12:41.981 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "started", 2024-09-11T08:12:41.981 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:12:41.981 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.981 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:12:41.981 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.950912+0000", 2024-09-11T08:12:41.981 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:12:41.981 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.981 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.981 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:12:41.982 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.950912+0000", 2024-09-11T08:12:41.982 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:12:41.982 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.982 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.982 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:12:41.982 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.950911+0000", 2024-09-11T08:12:41.982 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:12:41.982 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.982 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.982 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:12:41.982 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.950922+0000", 2024-09-11T08:12:41.983 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.165e-05 2024-09-11T08:12:41.983 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.983 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.983 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:12:41.983 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.950938+0000", 2024-09-11T08:12:41.983 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.5766000000000001e-05 2024-09-11T08:12:41.983 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.983 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.983 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:12:41.983 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.950944+0000", 2024-09-11T08:12:41.983 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.3049999999999998e-06 2024-09-11T08:12:41.983 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.983 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.984 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:12:41.984 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.951673+0000", 2024-09-11T08:12:41.984 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00072815 2024-09-11T08:12:41.984 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.984 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.984 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:12:41.984 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.951694+0000", 2024-09-11T08:12:41.984 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.1052000000000001e-05 2024-09-11T08:12:41.984 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:12:41.984 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:12:41.984 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:12:41.984 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.984 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.984 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.14790.0:200235 7.7 e142/91)", 2024-09-11T08:12:41.985 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:12:41.951494+0000", 2024-09-11T08:12:41.985 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.0065229060000000002, 2024-09-11T08:12:41.985 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0068586990000000002, 2024-09-11T08:12:41.985 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:12:41.985 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "queued for pg", 2024-09-11T08:12:41.985 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:12:41.985 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.985 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:12:41.985 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.951494+0000", 2024-09-11T08:12:41.985 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:12:41.985 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.985 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.985 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:12:41.986 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.951494+0000", 2024-09-11T08:12:41.986 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:12:41.986 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.986 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.986 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:12:41.986 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.951493+0000", 2024-09-11T08:12:41.986 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:12:41.986 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.986 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.986 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:12:41.986 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.951500+0000", 2024-09-11T08:12:41.986 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.6089999999999997e-06 2024-09-11T08:12:41.986 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.986 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.987 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:12:41.987 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.951514+0000", 2024-09-11T08:12:41.987 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4328000000000001e-05 2024-09-11T08:12:41.987 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.987 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.987 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:12:41.987 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.951521+0000", 2024-09-11T08:12:41.987 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.3709999999999996e-06 2024-09-11T08:12:41.987 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:12:41.987 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:12:41.987 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:12:41.987 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.987 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.987 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.14790.0:200223 7.e e142/91 7:73aed66a:::benchmark_data_smithi150_78508_object75782:head v 142'12522, mlcod=142'12519)", 2024-09-11T08:12:41.988 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:12:41.951824+0000", 2024-09-11T08:12:41.988 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.0061935499999999999, 2024-09-11T08:12:41.988 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.006572413, 2024-09-11T08:12:41.988 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:12:41.988 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "started", 2024-09-11T08:12:41.988 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:12:41.988 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.988 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:12:41.988 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.951824+0000", 2024-09-11T08:12:41.988 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:12:41.988 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.988 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.988 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:12:41.988 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.951824+0000", 2024-09-11T08:12:41.989 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:12:41.989 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.989 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.989 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:12:41.989 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.951823+0000", 2024-09-11T08:12:41.989 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:12:41.989 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.989 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.989 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:12:41.989 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.951828+0000", 2024-09-11T08:12:41.989 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.4260000000000002e-06 2024-09-11T08:12:41.989 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.990 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.990 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:12:41.990 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.951842+0000", 2024-09-11T08:12:41.990 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4161999999999999e-05 2024-09-11T08:12:41.990 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.990 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.990 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:12:41.990 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.951848+0000", 2024-09-11T08:12:41.990 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.7300000000000002e-06 2024-09-11T08:12:41.990 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.990 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.990 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:12:41.990 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.951915+0000", 2024-09-11T08:12:41.991 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.6524000000000002e-05 2024-09-11T08:12:41.991 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.991 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.991 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:12:41.991 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.951938+0000", 2024-09-11T08:12:41.991 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.3699000000000001e-05 2024-09-11T08:12:41.991 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:12:41.991 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:12:41.991 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:12:41.991 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.991 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.992 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop_reply(client.14790.0:200222 7.4 e142/91)", 2024-09-11T08:12:41.992 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:12:41.951875+0000", 2024-09-11T08:12:41.992 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.00614242, 2024-09-11T08:12:41.992 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.006570719, 2024-09-11T08:12:41.992 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:12:41.992 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "queued for pg", 2024-09-11T08:12:41.992 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:12:41.992 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.992 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:12:41.992 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.951875+0000", 2024-09-11T08:12:41.992 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:12:41.992 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.992 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.992 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:12:41.993 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.951875+0000", 2024-09-11T08:12:41.993 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:12:41.993 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.993 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.993 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:12:41.994 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:12:41.994 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.951874+0000", 2024-09-11T08:12:41.994 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:12:41.994 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.994 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.994 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:12:41.994 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.951878+0000", 2024-09-11T08:12:41.994 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.4699999999999998e-06 2024-09-11T08:12:41.994 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.994 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.994 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:12:41.995 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.951885+0000", 2024-09-11T08:12:41.995 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.9269999999999999e-06 2024-09-11T08:12:41.995 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.995 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.995 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:12:41.995 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.951889+0000", 2024-09-11T08:12:41.995 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.4100000000000001e-06 2024-09-11T08:12:41.995 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:12:41.995 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:12:41.995 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:12:41.995 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.995 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.996 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.14790.0:200229 7.e e142/91 7:7a1eff43:::benchmark_data_smithi150_78508_object75788:head v 142'12523, mlcod=142'12520)", 2024-09-11T08:12:41.996 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:12:41.957843+0000", 2024-09-11T08:12:41.996 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.000173876, 2024-09-11T08:12:41.996 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00064724800000000001, 2024-09-11T08:12:41.996 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:12:41.996 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "started", 2024-09-11T08:12:41.996 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:12:41.996 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.996 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:12:41.996 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.957843+0000", 2024-09-11T08:12:41.996 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:12:41.996 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.996 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.996 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:12:41.997 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.957843+0000", 2024-09-11T08:12:41.997 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:12:41.997 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.997 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.997 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:12:41.997 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.957842+0000", 2024-09-11T08:12:41.997 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:12:41.997 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.997 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.997 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:12:41.997 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.957850+0000", 2024-09-11T08:12:41.997 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.2719999999999997e-06 2024-09-11T08:12:41.997 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.998 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.998 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:12:41.998 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.957867+0000", 2024-09-11T08:12:41.998 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.662e-05 2024-09-11T08:12:41.998 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.998 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.998 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:12:41.998 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.957874+0000", 2024-09-11T08:12:41.998 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.2389999999999996e-06 2024-09-11T08:12:41.998 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.998 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.998 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:12:41.998 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.958295+0000", 2024-09-11T08:12:41.999 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00042103099999999999 2024-09-11T08:12:41.999 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.999 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.999 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:12:41.999 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.958315+0000", 2024-09-11T08:12:41.999 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.0211e-05 2024-09-11T08:12:41.999 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:12:41.999 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:12:41.999 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:12:41.999 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:41.999 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:41.999 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.14790.0:200233 7.e e142/91)", 2024-09-11T08:12:41.999 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:12:41.957886+0000", 2024-09-11T08:12:42.000 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.00013067100000000001, 2024-09-11T08:12:42.000 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00065244000000000001, 2024-09-11T08:12:42.000 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:12:42.000 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "queued for pg", 2024-09-11T08:12:42.000 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:12:42.000 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:42.000 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:12:42.000 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.957886+0000", 2024-09-11T08:12:42.000 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:12:42.000 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:42.000 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:42.000 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:12:42.001 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.957886+0000", 2024-09-11T08:12:42.001 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:12:42.001 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:42.001 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:42.001 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:12:42.001 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.957886+0000", 2024-09-11T08:12:42.001 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:12:42.001 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:42.001 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:42.001 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:12:42.001 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.957890+0000", 2024-09-11T08:12:42.001 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.2370000000000003e-06 2024-09-11T08:12:42.002 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:42.002 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:42.002 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:12:42.002 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.957896+0000", 2024-09-11T08:12:42.002 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.6319999999999996e-06 2024-09-11T08:12:42.002 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:42.002 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:42.002 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:12:42.002 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.957900+0000", 2024-09-11T08:12:42.002 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.6160000000000002e-06 2024-09-11T08:12:42.002 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:12:42.002 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:12:42.002 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:12:42.002 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:42.003 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:42.003 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop_reply(client.14790.0:200227 7.4 e142/91)", 2024-09-11T08:12:42.003 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:12:41.957915+0000", 2024-09-11T08:12:42.003 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.000102172, 2024-09-11T08:12:42.003 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00066304899999999997, 2024-09-11T08:12:42.003 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:12:42.003 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "queued for pg", 2024-09-11T08:12:42.003 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:12:42.003 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:42.003 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:12:42.003 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.957915+0000", 2024-09-11T08:12:42.003 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:12:42.003 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:42.003 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:42.004 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:12:42.004 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.957915+0000", 2024-09-11T08:12:42.004 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:12:42.004 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:42.004 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:42.004 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:12:42.004 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.957915+0000", 2024-09-11T08:12:42.004 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:12:42.004 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:42.004 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:42.004 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:12:42.004 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.957918+0000", 2024-09-11T08:12:42.004 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.2329999999999998e-06 2024-09-11T08:12:42.005 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:42.005 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:42.005 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:12:42.005 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.957924+0000", 2024-09-11T08:12:42.005 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.0390000000000001e-06 2024-09-11T08:12:42.005 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:42.005 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:42.005 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:12:42.005 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.957928+0000", 2024-09-11T08:12:42.005 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.6789999999999998e-06 2024-09-11T08:12:42.005 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:12:42.005 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:12:42.005 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:12:42.005 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:42.006 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:42.006 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop_reply(client.14790.0:200232 7.4 e142/91)", 2024-09-11T08:12:42.006 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:12:41.957933+0000", 2024-09-11T08:12:42.006 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 8.4311e-05, 2024-09-11T08:12:42.006 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00068420000000000004, 2024-09-11T08:12:42.006 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:12:42.006 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "queued for pg", 2024-09-11T08:12:42.006 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:12:42.006 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:42.006 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:12:42.006 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.957933+0000", 2024-09-11T08:12:42.006 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:12:42.006 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:42.006 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:42.007 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:12:42.007 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.957933+0000", 2024-09-11T08:12:42.007 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:12:42.007 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:42.007 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:42.007 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:12:42.007 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.957933+0000", 2024-09-11T08:12:42.007 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:12:42.007 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:42.007 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:42.007 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:12:42.007 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.957935+0000", 2024-09-11T08:12:42.007 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.5110000000000002e-06 2024-09-11T08:12:42.007 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:42.007 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:42.008 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:12:42.008 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.957940+0000", 2024-09-11T08:12:42.008 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.152e-06 2024-09-11T08:12:42.008 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:42.008 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:42.008 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:12:42.008 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.957944+0000", 2024-09-11T08:12:42.008 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.3589999999999999e-06 2024-09-11T08:12:42.008 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:12:42.008 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:12:42.008 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:12:42.008 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:42.008 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:42.008 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:200222 7.4 7:22ddca1c:::benchmark_data_smithi150_78508_object75781:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-09-11T08:12:42.009 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:12:41.940945+0000", 2024-09-11T08:12:42.009 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.017071688000000002, 2024-09-11T08:12:42.009 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.017717086999999999, 2024-09-11T08:12:42.009 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:12:42.009 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "waiting for sub ops", 2024-09-11T08:12:42.009 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:12:42.009 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:12:42.009 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:12:42.009 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 200222 2024-09-11T08:12:42.009 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:42.009 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:12:42.009 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:42.010 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:12:42.010 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.940945+0000", 2024-09-11T08:12:42.010 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:12:42.010 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:42.010 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:42.010 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:12:42.010 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.940945+0000", 2024-09-11T08:12:42.010 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:12:42.010 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:42.010 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:42.010 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:12:42.010 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.940944+0000", 2024-09-11T08:12:42.010 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:12:42.010 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:42.011 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:42.011 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:12:42.011 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.940949+0000", 2024-09-11T08:12:42.011 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.3990000000000003e-06 2024-09-11T08:12:42.011 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:42.011 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:42.011 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:12:42.011 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.943057+0000", 2024-09-11T08:12:42.011 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.002107739 2024-09-11T08:12:42.011 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:42.011 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:42.011 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:12:42.011 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.943067+0000", 2024-09-11T08:12:42.012 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.7079999999999992e-06 2024-09-11T08:12:42.012 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:42.012 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:42.012 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:12:42.012 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.943127+0000", 2024-09-11T08:12:42.012 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.0310999999999999e-05 2024-09-11T08:12:42.012 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:42.012 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:42.012 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:12:42.012 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.943412+0000", 2024-09-11T08:12:42.012 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00028452399999999997 2024-09-11T08:12:42.012 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:42.013 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:42.013 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 1,4", 2024-09-11T08:12:42.013 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.943613+0000", 2024-09-11T08:12:42.013 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000200853 2024-09-11T08:12:42.013 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:12:42.013 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:12:42.013 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:12:42.013 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:41.946600+0000", 2024-09-11T08:12:42.013 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.002987726 2024-09-11T08:12:42.013 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:12:42.013 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:12:42.013 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:12:42.013 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:12:42.014 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-09-11T08:12:42.014 INFO:teuthology.orchestra.run.smithi150.stdout: "num_ops": 13 2024-09-11T08:12:42.014 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-11T08:12:42.036 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:12:42.071 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:42.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:42.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:12:42.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:41 smithi181 ceph-mon[32701]: pgmap v209: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 992 op/s; 16616/159735 objects degraded (10.402%); 0 B/s, 0 objects/s recovering 2024-09-11T08:12:42.094 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/osd.7/config 2024-09-11T08:12:42.173 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:42.218 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:42.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:42.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:42.320 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:42.355 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:42.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:42.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:42.457 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:42.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:42.486 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:42.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:42.587 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:42.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:42.628 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:42.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:42.729 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:42.754 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:42.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:42.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:42.856 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:42.882 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-11T08:12:42.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:42.914 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:42.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:43.015 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:43.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:43.064 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:43.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:43.165 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:43.188 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:43.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:43.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:43.290 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:43.315 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:43.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:43.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:43.403 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T08:12:43.417 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:43.507 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:43.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:43.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:43.609 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:12:43.633 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:43.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:43.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:12:43.734 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:43.787 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:43.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:43.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:43.873 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:43 smithi181 ceph-mon[32701]: pgmap v210: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s; 16294/154722 objects degraded (10.531%); 0 B/s, 0 objects/s recovering 2024-09-11T08:12:43.889 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:43.932 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:43.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:43.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:44.034 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:12:44.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:44.071 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:44.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:12:44.172 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:12:44.177 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:43 smithi150 ceph-mon[29276]: pgmap v210: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s; 16294/154722 objects degraded (10.531%); 0 B/s, 0 objects/s recovering 2024-09-11T08:12:44.177 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:43 smithi150 ceph-mon[39612]: pgmap v210: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s; 16294/154722 objects degraded (10.531%); 0 B/s, 0 objects/s recovering 2024-09-11T08:12:44.188 INFO:teuthology.orchestra.run.smithi181.stdout: marking collection for removal 2024-09-11T08:12:44.190 INFO:teuthology.orchestra.run.smithi181.stdout:setting '_remove' omap key 2024-09-11T08:12:44.190 INFO:teuthology.orchestra.run.smithi181.stdout:finish_remove_pgs 4.1e_head removing 4.1e 2024-09-11T08:12:44.191 INFO:teuthology.orchestra.run.smithi181.stdout:Remove successful 2024-09-11T08:12:44.221 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:44.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:44.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:12:44.323 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:12:44.348 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:12:44.352 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:12:44.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:44.386 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:44.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:12:44.487 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:12:44.534 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:44.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:44.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:12:44.639 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:12:44.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:44.680 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:44.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:12:44.781 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:44.807 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:44.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:44.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:44.909 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:44.939 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:44.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:44.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:45.041 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:45.056 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:44 smithi181 ceph-mon[32701]: Health check update: Degraded data redundancy: 14914/143517 objects degraded (10.392%), 24 pgs degraded (PG_DEGRADED) 2024-09-11T08:12:45.056 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:44 smithi181 ceph-mon[32701]: osdmap e143: 8 total, 7 up, 8 in 2024-09-11T08:12:45.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:45.079 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:45.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:45.179 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:12:45.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:45.221 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:45.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:12:45.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:44 smithi150 ceph-mon[29276]: Health check update: Degraded data redundancy: 14914/143517 objects degraded (10.392%), 24 pgs degraded (PG_DEGRADED) 2024-09-11T08:12:45.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:44 smithi150 ceph-mon[29276]: osdmap e143: 8 total, 7 up, 8 in 2024-09-11T08:12:45.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:44 smithi150 ceph-mon[39612]: Health check update: Degraded data redundancy: 14914/143517 objects degraded (10.392%), 24 pgs degraded (PG_DEGRADED) 2024-09-11T08:12:45.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:44 smithi150 ceph-mon[39612]: osdmap e143: 8 total, 7 up, 8 in 2024-09-11T08:12:45.322 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:12:45.323 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:12:45 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:12:45] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:12:45.348 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:45.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:45.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:12:45.450 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:45.474 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:45.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:45.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:45.577 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:45.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:45.611 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:45.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:45.712 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:45.751 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:45.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:45.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:45.805 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -n osd.7 --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.4.1e.7 2024-09-11T08:12:45.854 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:45.900 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:45.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:45.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:45.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:45 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:12:45.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:45 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:12:45.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:45 smithi150 ceph-mon[29276]: pgmap v211: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 2.9 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s; 14914/143517 objects degraded (10.392%) 2024-09-11T08:12:45.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:45 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:12:45.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:45 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:12:45.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:45 smithi150 ceph-mon[39612]: pgmap v211: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 2.9 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s; 14914/143517 objects degraded (10.392%) 2024-09-11T08:12:46.002 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:12:46.028 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:46.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:46.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:12:46.032 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/osd.7/config 2024-09-11T08:12:46.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:45 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:12:46.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:45 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:12:46.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:45 smithi181 ceph-mon[32701]: pgmap v211: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 2.9 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s; 14914/143517 objects degraded (10.392%) 2024-09-11T08:12:46.130 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:46.177 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:46.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:46.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:46.279 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:46.314 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:46.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:46.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:46.416 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:46.450 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:46.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:46.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:46.552 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:46.598 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:46.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:46.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:46.700 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:46.742 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:46.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:46.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:46.844 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:46.873 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:46.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:46.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:46.911 INFO:teuthology.orchestra.run.smithi150.stderr:noscrub is unset 2024-09-11T08:12:46.976 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:47.001 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:47.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:47.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:47.102 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:47.137 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:47.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:47.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:47.234 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:46 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3017785728' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:12:47.234 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:46 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:12:47.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:46 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3017785728' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:12:47.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:46 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:12:47.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:46 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3017785728' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:12:47.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:46 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:12:47.239 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:12:47.276 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:47.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:47.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:12:47.378 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:47.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:47.418 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:47.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:47.519 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:47.555 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:47.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:47.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:47.657 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:12:47.693 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:47.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:47.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:12:47.795 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:12:47.821 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:47.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:47.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:12:47.923 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:47.952 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:47 smithi181 ceph-mon[32701]: pgmap v213: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 2.9 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s; 14914/143517 objects degraded (10.392%) 2024-09-11T08:12:47.952 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:47 smithi181 ceph-mon[32701]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:12:47.952 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:47 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:12:47.952 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:47 smithi181 ceph-mon[32701]: osdmap e144: 8 total, 7 up, 8 in 2024-09-11T08:12:47.959 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:47.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:47.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:47.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:47 smithi150 ceph-mon[29276]: pgmap v213: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 2.9 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s; 14914/143517 objects degraded (10.392%) 2024-09-11T08:12:47.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:47 smithi150 ceph-mon[29276]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:12:47.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:47 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:12:47.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:47 smithi150 ceph-mon[29276]: osdmap e144: 8 total, 7 up, 8 in 2024-09-11T08:12:47.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:47 smithi150 ceph-mon[39612]: pgmap v213: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 2.9 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s; 14914/143517 objects degraded (10.392%) 2024-09-11T08:12:47.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:47 smithi150 ceph-mon[39612]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:12:47.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:47 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:12:47.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:47 smithi150 ceph-mon[39612]: osdmap e144: 8 total, 7 up, 8 in 2024-09-11T08:12:48.060 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:48.085 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:48.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:48.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:48.187 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:48.227 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:48.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:48.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:48.261 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:12:47 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:12:48.328 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:48.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:48.362 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:48.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:48.463 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:48.481 INFO:teuthology.orchestra.run.smithi181.stdout:get_pg_num_history pg_num_history pg_num_history(e139 pg_nums {1={19=1},2={58=3},3={58=32},4={60=32},5={62=32},6={64=32},7={91=16}} deleted_pools ) 2024-09-11T08:12:48.481 INFO:teuthology.orchestra.run.smithi181.stdout:Importing pgid 4.1e 2024-09-11T08:12:48.481 INFO:teuthology.orchestra.run.smithi181.stdout:Write #4:7a480f67:::obj_delete_at_hint.0000000079:head# 2024-09-11T08:12:48.481 INFO:teuthology.orchestra.run.smithi181.stdout:snapset 0=[]:{} 2024-09-11T08:12:48.481 INFO:teuthology.orchestra.run.smithi181.stdout:Write #4:7a5c7c5f:::obj_delete_at_hint.0000000049:head# 2024-09-11T08:12:48.481 INFO:teuthology.orchestra.run.smithi181.stdout:snapset 0=[]:{} 2024-09-11T08:12:48.482 INFO:teuthology.orchestra.run.smithi181.stdout:Write #4:7adbb485:::obj_delete_at_hint.0000000117:head# 2024-09-11T08:12:48.482 INFO:teuthology.orchestra.run.smithi181.stdout:snapset 0=[]:{} 2024-09-11T08:12:48.482 INFO:teuthology.orchestra.run.smithi181.stdout:Write #4:7af5a05b:::obj_delete_at_hint.0000000044:head# 2024-09-11T08:12:48.482 INFO:teuthology.orchestra.run.smithi181.stdout:snapset 0=[]:{} 2024-09-11T08:12:48.483 INFO:teuthology.orchestra.run.smithi181.stdout:Write #4:7db93595:::obj_delete_at_hint.0000000066:head# 2024-09-11T08:12:48.483 INFO:teuthology.orchestra.run.smithi181.stdout:snapset 0=[]:{} 2024-09-11T08:12:48.483 INFO:teuthology.orchestra.run.smithi181.stdout:write_pg epoch 127 info 4.1e( v 66'10 (0'0,66'10] local-lis/les=60/61 n=5 ec=60/60 lis/c=60/60 les/c/f=61/61/0 sis=60) 2024-09-11T08:12:48.483 INFO:teuthology.orchestra.run.smithi181.stdout:Import successful 2024-09-11T08:12:48.502 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:48.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:48.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:48.604 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:48.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:48.634 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:48.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:48.735 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:48.781 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:48.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:48.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:48.883 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:48.908 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:12:48 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:12:48] "GET /metrics HTTP/1.1" 200 223996 "" "Prometheus/2.43.0" 2024-09-11T08:12:48.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:48.911 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:48.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:49.013 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:49.042 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:49.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:49.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:49.144 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:49.170 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:49.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:49.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:49.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:48 smithi150 ceph-mon[29276]: 2.2 scrub starts 2024-09-11T08:12:49.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:48 smithi150 ceph-mon[29276]: 2.2 scrub ok 2024-09-11T08:12:49.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:48 smithi150 ceph-mon[29276]: 3.a scrub starts 2024-09-11T08:12:49.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:48 smithi150 ceph-mon[29276]: 3.a scrub ok 2024-09-11T08:12:49.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:48 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:12:49.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:48 smithi150 ceph-mon[39612]: 2.2 scrub starts 2024-09-11T08:12:49.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:48 smithi150 ceph-mon[39612]: 2.2 scrub ok 2024-09-11T08:12:49.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:48 smithi150 ceph-mon[39612]: 3.a scrub starts 2024-09-11T08:12:49.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:48 smithi150 ceph-mon[39612]: 3.a scrub ok 2024-09-11T08:12:49.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:48 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:12:49.272 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:49.318 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:49.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:49.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:49.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:48 smithi181 ceph-mon[32701]: 2.2 scrub starts 2024-09-11T08:12:49.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:48 smithi181 ceph-mon[32701]: 2.2 scrub ok 2024-09-11T08:12:49.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:48 smithi181 ceph-mon[32701]: 3.a scrub starts 2024-09-11T08:12:49.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:48 smithi181 ceph-mon[32701]: 3.a scrub ok 2024-09-11T08:12:49.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:48 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:12:49.420 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:49.456 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:49.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:49.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:49.557 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:49.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:49.583 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:49.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:49.685 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:49.725 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:49.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:49.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:49.778 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f /var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/exp.4.1e.7 2024-09-11T08:12:49.826 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:49.862 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:49.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:49.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:49.964 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:49.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:49 smithi150 ceph-mon[29276]: pgmap v215: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 2.6 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.34k op/s; 13187/127611 objects degraded (10.334%) 2024-09-11T08:12:49.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:49 smithi150 ceph-mon[29276]: 7.d scrub starts 2024-09-11T08:12:49.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:49 smithi150 ceph-mon[29276]: 7.9 scrub starts 2024-09-11T08:12:49.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:49 smithi150 ceph-mon[29276]: 4.18 scrub starts 2024-09-11T08:12:49.991 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:49 smithi150 ceph-mon[29276]: 4.18 scrub ok 2024-09-11T08:12:49.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:49 smithi150 ceph-mon[39612]: pgmap v215: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 2.6 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.34k op/s; 13187/127611 objects degraded (10.334%) 2024-09-11T08:12:49.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:49 smithi150 ceph-mon[39612]: 7.d scrub starts 2024-09-11T08:12:49.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:49 smithi150 ceph-mon[39612]: 7.9 scrub starts 2024-09-11T08:12:49.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:49 smithi150 ceph-mon[39612]: 4.18 scrub starts 2024-09-11T08:12:49.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:49 smithi150 ceph-mon[39612]: 4.18 scrub ok 2024-09-11T08:12:50.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:50.003 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:50.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:50.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:49 smithi181 ceph-mon[32701]: pgmap v215: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 2.6 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.34k op/s; 13187/127611 objects degraded (10.334%) 2024-09-11T08:12:50.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:49 smithi181 ceph-mon[32701]: 7.d scrub starts 2024-09-11T08:12:50.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:49 smithi181 ceph-mon[32701]: 7.9 scrub starts 2024-09-11T08:12:50.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:49 smithi181 ceph-mon[32701]: 4.18 scrub starts 2024-09-11T08:12:50.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:49 smithi181 ceph-mon[32701]: 4.18 scrub ok 2024-09-11T08:12:50.104 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:12:50.141 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:50.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:50.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:12:50.243 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:50.283 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:50.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:50.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:50.385 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:50.411 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:50.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:50.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:50.513 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:50.551 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:50.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:50.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:50.653 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:50.677 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:50.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:50.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:50.779 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:50.800 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:50.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:50.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:50.902 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:50.950 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:50.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:50.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:51.052 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:12:51.076 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:51.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:51.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:12:51.178 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:51.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:51.222 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:51.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:51.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:50 smithi150 ceph-mon[29276]: 7.d scrub ok 2024-09-11T08:12:51.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:50 smithi150 ceph-mon[29276]: 7.9 scrub ok 2024-09-11T08:12:51.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:50 smithi150 ceph-mon[29276]: 7.3 scrub starts 2024-09-11T08:12:51.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:50 smithi150 ceph-mon[29276]: Health check update: Degraded data redundancy: 13187/127611 objects degraded (10.334%), 24 pgs degraded (PG_DEGRADED) 2024-09-11T08:12:51.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:50 smithi150 ceph-mon[39612]: 7.d scrub ok 2024-09-11T08:12:51.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:50 smithi150 ceph-mon[39612]: 7.9 scrub ok 2024-09-11T08:12:51.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:50 smithi150 ceph-mon[39612]: 7.3 scrub starts 2024-09-11T08:12:51.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:50 smithi150 ceph-mon[39612]: Health check update: Degraded data redundancy: 13187/127611 objects degraded (10.334%), 24 pgs degraded (PG_DEGRADED) 2024-09-11T08:12:51.323 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:51.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:50 smithi181 ceph-mon[32701]: 7.d scrub ok 2024-09-11T08:12:51.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:50 smithi181 ceph-mon[32701]: 7.9 scrub ok 2024-09-11T08:12:51.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:50 smithi181 ceph-mon[32701]: 7.3 scrub starts 2024-09-11T08:12:51.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:50 smithi181 ceph-mon[32701]: Health check update: Degraded data redundancy: 13187/127611 objects degraded (10.334%), 24 pgs degraded (PG_DEGRADED) 2024-09-11T08:12:51.346 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:51.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:51.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:51.448 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:51.470 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:51.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:51.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:51.572 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:12:51.612 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:51.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:51.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:12:51.713 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:12:51.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:51.736 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:51.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:12:51.837 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:51.872 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:51.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:51.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:51.964 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:51 smithi150 ceph-mon[29276]: pgmap v216: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 2.6 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.13k op/s; 13187/127611 objects degraded (10.334%) 2024-09-11T08:12:51.964 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:51 smithi150 ceph-mon[29276]: 7.4 scrub starts 2024-09-11T08:12:51.964 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:51 smithi150 ceph-mon[29276]: 7.3 scrub ok 2024-09-11T08:12:51.964 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:51 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:12:51.964 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:51 smithi150 ceph-mon[39612]: pgmap v216: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 2.6 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.13k op/s; 13187/127611 objects degraded (10.334%) 2024-09-11T08:12:51.965 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:51 smithi150 ceph-mon[39612]: 7.4 scrub starts 2024-09-11T08:12:51.965 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:51 smithi150 ceph-mon[39612]: 7.3 scrub ok 2024-09-11T08:12:51.965 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:51 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:12:51.974 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:12:51.997 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:51.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:51.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:12:52.099 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:52.121 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:52.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:52.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:52.222 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:12:52.245 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:52.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:52.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:12:52.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:51 smithi181 ceph-mon[32701]: pgmap v216: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 2.6 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.13k op/s; 13187/127611 objects degraded (10.334%) 2024-09-11T08:12:52.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:51 smithi181 ceph-mon[32701]: 7.4 scrub starts 2024-09-11T08:12:52.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:51 smithi181 ceph-mon[32701]: 7.3 scrub ok 2024-09-11T08:12:52.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:51 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:12:52.328 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:12:52.329 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:12:52.347 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:52.357 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:12:52.358 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:12:52.370 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:52.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:52.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:52.412 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:12:52.412 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:12:52.413 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-09-11T08:12:52.413 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:12:52.445 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:12:52.445 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:12:52.471 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:12:52.478 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:12:52.478 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:12:52.492 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T08:12:52.494 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:52.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:52.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:12:52.511 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:12:52.511 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:12:52.541 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:12:52.542 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:12:52.548 INFO:teuthology.orchestra.run.smithi150.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-11T08:12:52.596 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:52.618 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:52.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:52.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:52.720 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:52.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:52.765 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:52.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:52.866 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:52.932 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:52.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:52.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:53.033 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:53.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:53.081 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:53.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:53.153 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:52 smithi150 ceph-mon[29276]: 7.e scrub starts 2024-09-11T08:12:53.153 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:52 smithi150 ceph-mon[29276]: 3.1f scrub starts 2024-09-11T08:12:53.153 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:52 smithi150 ceph-mon[29276]: 3.1f scrub ok 2024-09-11T08:12:53.153 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:52 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/808071464' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:12:53.153 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:52 smithi150 ceph-mon[29276]: 7.4 scrub ok 2024-09-11T08:12:53.153 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:52 smithi150 ceph-mon[39612]: 7.e scrub starts 2024-09-11T08:12:53.153 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:52 smithi150 ceph-mon[39612]: 3.1f scrub starts 2024-09-11T08:12:53.153 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:52 smithi150 ceph-mon[39612]: 3.1f scrub ok 2024-09-11T08:12:53.154 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:52 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/808071464' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:12:53.154 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:52 smithi150 ceph-mon[39612]: 7.4 scrub ok 2024-09-11T08:12:53.182 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:12:53.218 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:53.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:53.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:12:53.320 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:53.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:52 smithi181 ceph-mon[32701]: 7.e scrub starts 2024-09-11T08:12:53.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:52 smithi181 ceph-mon[32701]: 3.1f scrub starts 2024-09-11T08:12:53.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:52 smithi181 ceph-mon[32701]: 3.1f scrub ok 2024-09-11T08:12:53.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:52 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/808071464' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:12:53.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:52 smithi181 ceph-mon[32701]: 7.4 scrub ok 2024-09-11T08:12:53.344 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:53.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:53.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:53.445 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:53.522 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-11T08:12:53.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:53.567 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:53.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:53.668 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:12:53.724 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:53.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:53.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:12:53.827 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:53.849 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:53.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:53.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:53.951 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:12:53.973 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:53.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:53.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:12:54.006 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-11T08:12:54.006 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-11T08:12:54.033 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:12:54.074 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:12:54.075 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:54.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:54.133 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:54.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:54.234 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:54.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:53 smithi150 ceph-mon[29276]: pgmap v217: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 861 op/s; 12874/122856 objects degraded (10.479%) 2024-09-11T08:12:54.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:53 smithi150 ceph-mon[29276]: 7.e scrub ok 2024-09-11T08:12:54.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:53 smithi150 ceph-mon[39612]: pgmap v217: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 861 op/s; 12874/122856 objects degraded (10.479%) 2024-09-11T08:12:54.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:53 smithi150 ceph-mon[39612]: 7.e scrub ok 2024-09-11T08:12:54.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:54.277 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:54.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:54.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:53 smithi181 ceph-mon[32701]: pgmap v217: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 861 op/s; 12874/122856 objects degraded (10.479%) 2024-09-11T08:12:54.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:53 smithi181 ceph-mon[32701]: 7.e scrub ok 2024-09-11T08:12:54.378 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:54.402 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:54.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:54.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:54.505 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:54.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:54.554 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:54.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:54.655 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:54.680 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:54.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:54.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:54.782 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:54.804 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:54.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:54.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:54.813 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [7] live_osds: [0, 1, 2, 3, 4, 5, 6] 2024-09-11T08:12:54.814 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T08:12:54.814 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-09-11T08:12:54.814 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-11T08:12:54.814 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-11T08:12:54.814 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-09-11T08:12:54.906 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:54.929 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:54.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:54.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:55.031 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:12:55.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:55.083 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:55.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:12:55.142 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:54 smithi150 ceph-mon[29276]: 5.18 scrub starts 2024-09-11T08:12:55.142 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:54 smithi150 ceph-mon[29276]: 5.18 scrub ok 2024-09-11T08:12:55.142 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:54 smithi150 ceph-mon[29276]: 3.17 scrub starts 2024-09-11T08:12:55.142 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:54 smithi150 ceph-mon[29276]: 3.17 scrub ok 2024-09-11T08:12:55.143 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:54 smithi150 ceph-mon[39612]: 5.18 scrub starts 2024-09-11T08:12:55.143 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:54 smithi150 ceph-mon[39612]: 5.18 scrub ok 2024-09-11T08:12:55.143 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:54 smithi150 ceph-mon[39612]: 3.17 scrub starts 2024-09-11T08:12:55.143 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:54 smithi150 ceph-mon[39612]: 3.17 scrub ok 2024-09-11T08:12:55.184 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:12:55.236 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:55.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:55.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:12:55.323 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:12:55 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:12:55] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:12:55.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:54 smithi181 ceph-mon[32701]: 5.18 scrub starts 2024-09-11T08:12:55.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:54 smithi181 ceph-mon[32701]: 5.18 scrub ok 2024-09-11T08:12:55.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:54 smithi181 ceph-mon[32701]: 3.17 scrub starts 2024-09-11T08:12:55.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:54 smithi181 ceph-mon[32701]: 3.17 scrub ok 2024-09-11T08:12:55.338 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:55.376 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:55.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:55.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:55.478 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:55.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:55.520 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:55.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:55.621 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:12:55.646 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:55.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:55.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:12:55.748 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:55.771 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:55.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:55.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:55.873 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:12:55.911 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:55.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:55.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:12:56.008 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T08:12:56.013 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:56.037 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:56.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:56.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:56.138 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:56.164 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:56.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:56.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:56.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:55 smithi150 ceph-mon[39612]: pgmap v218: 148 pgs: 1 active+clean+scrubbing, 34 active+undersized, 24 active+undersized+degraded, 89 active+clean; 2.3 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.04k op/s; 11638/112527 objects degraded (10.342%) 2024-09-11T08:12:56.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:55 smithi150 ceph-mon[39612]: 4.e scrub starts 2024-09-11T08:12:56.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:55 smithi150 ceph-mon[39612]: 4.e scrub ok 2024-09-11T08:12:56.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:55 smithi150 ceph-mon[39612]: 4.6 scrub starts 2024-09-11T08:12:56.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:55 smithi150 ceph-mon[39612]: 4.6 scrub ok 2024-09-11T08:12:56.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:55 smithi150 ceph-mon[39612]: 3.b scrub starts 2024-09-11T08:12:56.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:55 smithi150 ceph-mon[39612]: 3.b scrub ok 2024-09-11T08:12:56.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:55 smithi150 ceph-mon[39612]: Health check update: Degraded data redundancy: 11638/112527 objects degraded (10.342%), 24 pgs degraded (PG_DEGRADED) 2024-09-11T08:12:56.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:55 smithi150 ceph-mon[29276]: pgmap v218: 148 pgs: 1 active+clean+scrubbing, 34 active+undersized, 24 active+undersized+degraded, 89 active+clean; 2.3 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.04k op/s; 11638/112527 objects degraded (10.342%) 2024-09-11T08:12:56.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:55 smithi150 ceph-mon[29276]: 4.e scrub starts 2024-09-11T08:12:56.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:55 smithi150 ceph-mon[29276]: 4.e scrub ok 2024-09-11T08:12:56.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:55 smithi150 ceph-mon[29276]: 4.6 scrub starts 2024-09-11T08:12:56.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:55 smithi150 ceph-mon[29276]: 4.6 scrub ok 2024-09-11T08:12:56.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:55 smithi150 ceph-mon[29276]: 3.b scrub starts 2024-09-11T08:12:56.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:55 smithi150 ceph-mon[29276]: 3.b scrub ok 2024-09-11T08:12:56.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:55 smithi150 ceph-mon[29276]: Health check update: Degraded data redundancy: 11638/112527 objects degraded (10.342%), 24 pgs degraded (PG_DEGRADED) 2024-09-11T08:12:56.266 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:56.303 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:56.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:56.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:56.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:55 smithi181 ceph-mon[32701]: pgmap v218: 148 pgs: 1 active+clean+scrubbing, 34 active+undersized, 24 active+undersized+degraded, 89 active+clean; 2.3 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.04k op/s; 11638/112527 objects degraded (10.342%) 2024-09-11T08:12:56.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:55 smithi181 ceph-mon[32701]: 4.e scrub starts 2024-09-11T08:12:56.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:55 smithi181 ceph-mon[32701]: 4.e scrub ok 2024-09-11T08:12:56.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:55 smithi181 ceph-mon[32701]: 4.6 scrub starts 2024-09-11T08:12:56.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:55 smithi181 ceph-mon[32701]: 4.6 scrub ok 2024-09-11T08:12:56.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:55 smithi181 ceph-mon[32701]: 3.b scrub starts 2024-09-11T08:12:56.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:55 smithi181 ceph-mon[32701]: 3.b scrub ok 2024-09-11T08:12:56.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:55 smithi181 ceph-mon[32701]: Health check update: Degraded data redundancy: 11638/112527 objects degraded (10.342%), 24 pgs degraded (PG_DEGRADED) 2024-09-11T08:12:56.404 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:12:56.429 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:56.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:56.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:12:56.531 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:56.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:56.558 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:56.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:56.659 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:56.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:56.683 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:56.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:56.784 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:12:56.823 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:56.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:56.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:12:56.925 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:56.982 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:56.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:56.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:57.084 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:57.108 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:57.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:57.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:57.209 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:57.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:57.255 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:57.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:57.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:56 smithi181 ceph-mon[32701]: 5.b scrub starts 2024-09-11T08:12:57.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:56 smithi181 ceph-mon[32701]: 5.b scrub ok 2024-09-11T08:12:57.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:56 smithi181 ceph-mon[32701]: 3.0 scrub starts 2024-09-11T08:12:57.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:56 smithi181 ceph-mon[32701]: 3.0 scrub ok 2024-09-11T08:12:57.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:56 smithi181 ceph-mon[32701]: 7.1 scrub starts 2024-09-11T08:12:57.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:56 smithi181 ceph-mon[32701]: 6.11 scrub starts 2024-09-11T08:12:57.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:56 smithi181 ceph-mon[32701]: 6.11 scrub ok 2024-09-11T08:12:57.341 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:56 smithi150 ceph-mon[29276]: 5.b scrub starts 2024-09-11T08:12:57.341 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:56 smithi150 ceph-mon[29276]: 5.b scrub ok 2024-09-11T08:12:57.341 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:56 smithi150 ceph-mon[29276]: 3.0 scrub starts 2024-09-11T08:12:57.341 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:57 smithi150 ceph-mon[29276]: 3.0 scrub ok 2024-09-11T08:12:57.341 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:57 smithi150 ceph-mon[29276]: 7.1 scrub starts 2024-09-11T08:12:57.341 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:57 smithi150 ceph-mon[29276]: 6.11 scrub starts 2024-09-11T08:12:57.341 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:57 smithi150 ceph-mon[29276]: 6.11 scrub ok 2024-09-11T08:12:57.341 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:57 smithi150 ceph-mon[39612]: 5.b scrub starts 2024-09-11T08:12:57.341 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:57 smithi150 ceph-mon[39612]: 5.b scrub ok 2024-09-11T08:12:57.342 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:57 smithi150 ceph-mon[39612]: 3.0 scrub starts 2024-09-11T08:12:57.342 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:57 smithi150 ceph-mon[39612]: 3.0 scrub ok 2024-09-11T08:12:57.342 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:57 smithi150 ceph-mon[39612]: 7.1 scrub starts 2024-09-11T08:12:57.342 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:57 smithi150 ceph-mon[39612]: 6.11 scrub starts 2024-09-11T08:12:57.342 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:57 smithi150 ceph-mon[39612]: 6.11 scrub ok 2024-09-11T08:12:57.356 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:12:57.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:57.423 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:57.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:12:57.524 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:57.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:57.584 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:57.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:57.685 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:57.709 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:57.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:57.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:57.811 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:12:57.836 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:57.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:57.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:12:57.937 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:12:57.978 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:57.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:57.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:12:58.080 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:58.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:58.122 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:58.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:58.223 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:12:58.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:58 smithi150 ceph-mon[29276]: 7.1 scrub ok 2024-09-11T08:12:58.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:58 smithi150 ceph-mon[29276]: 3.19 scrub starts 2024-09-11T08:12:58.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:58 smithi150 ceph-mon[29276]: 3.19 scrub ok 2024-09-11T08:12:58.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:58 smithi150 ceph-mon[29276]: pgmap v219: 148 pgs: 1 active+clean+scrubbing, 34 active+undersized, 24 active+undersized+degraded, 89 active+clean; 2.3 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s; 11638/112527 objects degraded (10.342%) 2024-09-11T08:12:58.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:58 smithi150 ceph-mon[29276]: 3.3 scrub starts 2024-09-11T08:12:58.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:58 smithi150 ceph-mon[29276]: 3.3 scrub ok 2024-09-11T08:12:58.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:58 smithi150 ceph-mon[39612]: 7.1 scrub ok 2024-09-11T08:12:58.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:58 smithi150 ceph-mon[39612]: 3.19 scrub starts 2024-09-11T08:12:58.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:58 smithi150 ceph-mon[39612]: 3.19 scrub ok 2024-09-11T08:12:58.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:58 smithi150 ceph-mon[39612]: pgmap v219: 148 pgs: 1 active+clean+scrubbing, 34 active+undersized, 24 active+undersized+degraded, 89 active+clean; 2.3 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s; 11638/112527 objects degraded (10.342%) 2024-09-11T08:12:58.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:58 smithi150 ceph-mon[39612]: 3.3 scrub starts 2024-09-11T08:12:58.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:58 smithi150 ceph-mon[39612]: 3.3 scrub ok 2024-09-11T08:12:58.247 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:58.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:58.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:12:58.270 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:12:58.270 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:12:58.323 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:12:57 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:12:58.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:58 smithi181 ceph-mon[32701]: 7.1 scrub ok 2024-09-11T08:12:58.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:58 smithi181 ceph-mon[32701]: 3.19 scrub starts 2024-09-11T08:12:58.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:58 smithi181 ceph-mon[32701]: 3.19 scrub ok 2024-09-11T08:12:58.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:58 smithi181 ceph-mon[32701]: pgmap v219: 148 pgs: 1 active+clean+scrubbing, 34 active+undersized, 24 active+undersized+degraded, 89 active+clean; 2.3 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s; 11638/112527 objects degraded (10.342%) 2024-09-11T08:12:58.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:58 smithi181 ceph-mon[32701]: 3.3 scrub starts 2024-09-11T08:12:58.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:58 smithi181 ceph-mon[32701]: 3.3 scrub ok 2024-09-11T08:12:58.348 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:58.372 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:58.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:58.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:58.409 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-09-11T08:12:58.410 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:12:58.474 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:12:58.510 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:58.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:58.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:12:58.612 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:58.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:58.637 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:58.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:58.738 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:12:58.761 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:58.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:58.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:12:58.863 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:58.886 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:58.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:58.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:58.902 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:12:58 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:12:58] "GET /metrics HTTP/1.1" 200 224013 "" "Prometheus/2.43.0" 2024-09-11T08:12:58.987 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:59.010 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:59.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:59.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:59.041 INFO:teuthology.orchestra.run.smithi150.stderr:nodeep-scrub is unset 2024-09-11T08:12:59.112 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:59.154 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:59.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:59.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:59.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:59 smithi150 ceph-mon[39612]: 5.a scrub starts 2024-09-11T08:12:59.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:59 smithi150 ceph-mon[39612]: 5.a scrub ok 2024-09-11T08:12:59.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:59 smithi150 ceph-mon[39612]: 4.2 scrub starts 2024-09-11T08:12:59.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:59 smithi150 ceph-mon[39612]: 5.16 scrub starts 2024-09-11T08:12:59.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:59 smithi150 ceph-mon[39612]: 5.16 scrub ok 2024-09-11T08:12:59.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:59 smithi150 ceph-mon[39612]: 4.2 scrub ok 2024-09-11T08:12:59.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:59 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:12:59.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:59 smithi150 ceph-mon[39612]: 4.d scrub starts 2024-09-11T08:12:59.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:59 smithi150 ceph-mon[39612]: 4.d scrub ok 2024-09-11T08:12:59.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:59 smithi150 ceph-mon[39612]: 6.1f scrub starts 2024-09-11T08:12:59.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:59 smithi150 ceph-mon[39612]: 6.1f scrub ok 2024-09-11T08:12:59.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:59 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1395060505' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:12:59.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:59 smithi150 ceph-mon[39612]: 4.4 scrub starts 2024-09-11T08:12:59.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:12:59 smithi150 ceph-mon[39612]: 4.4 scrub ok 2024-09-11T08:12:59.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:59 smithi150 ceph-mon[29276]: 5.a scrub starts 2024-09-11T08:12:59.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:59 smithi150 ceph-mon[29276]: 5.a scrub ok 2024-09-11T08:12:59.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:59 smithi150 ceph-mon[29276]: 4.2 scrub starts 2024-09-11T08:12:59.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:59 smithi150 ceph-mon[29276]: 5.16 scrub starts 2024-09-11T08:12:59.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:59 smithi150 ceph-mon[29276]: 5.16 scrub ok 2024-09-11T08:12:59.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:59 smithi150 ceph-mon[29276]: 4.2 scrub ok 2024-09-11T08:12:59.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:59 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:12:59.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:59 smithi150 ceph-mon[29276]: 4.d scrub starts 2024-09-11T08:12:59.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:59 smithi150 ceph-mon[29276]: 4.d scrub ok 2024-09-11T08:12:59.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:59 smithi150 ceph-mon[29276]: 6.1f scrub starts 2024-09-11T08:12:59.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:59 smithi150 ceph-mon[29276]: 6.1f scrub ok 2024-09-11T08:12:59.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:59 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1395060505' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:12:59.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:59 smithi150 ceph-mon[29276]: 4.4 scrub starts 2024-09-11T08:12:59.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:12:59 smithi150 ceph-mon[29276]: 4.4 scrub ok 2024-09-11T08:12:59.257 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:12:59.281 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:59.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:59.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:12:59.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:59 smithi181 ceph-mon[32701]: 5.a scrub starts 2024-09-11T08:12:59.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:59 smithi181 ceph-mon[32701]: 5.a scrub ok 2024-09-11T08:12:59.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:59 smithi181 ceph-mon[32701]: 4.2 scrub starts 2024-09-11T08:12:59.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:59 smithi181 ceph-mon[32701]: 5.16 scrub starts 2024-09-11T08:12:59.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:59 smithi181 ceph-mon[32701]: 5.16 scrub ok 2024-09-11T08:12:59.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:59 smithi181 ceph-mon[32701]: 4.2 scrub ok 2024-09-11T08:12:59.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:59 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:12:59.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:59 smithi181 ceph-mon[32701]: 4.d scrub starts 2024-09-11T08:12:59.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:59 smithi181 ceph-mon[32701]: 4.d scrub ok 2024-09-11T08:12:59.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:59 smithi181 ceph-mon[32701]: 6.1f scrub starts 2024-09-11T08:12:59.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:59 smithi181 ceph-mon[32701]: 6.1f scrub ok 2024-09-11T08:12:59.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:59 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1395060505' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:12:59.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:59 smithi181 ceph-mon[32701]: 4.4 scrub starts 2024-09-11T08:12:59.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:12:59 smithi181 ceph-mon[32701]: 4.4 scrub ok 2024-09-11T08:12:59.383 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:12:59.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:59.410 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:59.411 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-11T08:12:59.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:12:59.512 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:12:59.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:59.556 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:59.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:12:59.657 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:12:59.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:59.683 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:59.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:12:59.784 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:12:59.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:59.811 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:12:59.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:12:59.912 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:12:59.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:12:59.957 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:12:59.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:00.058 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:00.099 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:00.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:00.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:00.201 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:00.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:00 smithi150 ceph-mon[29276]: pgmap v220: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 2.0 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.30k op/s; 10131/97338 objects degraded (10.408%) 2024-09-11T08:13:00.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:00 smithi150 ceph-mon[29276]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:13:00.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:00 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1395060505' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:13:00.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:00 smithi150 ceph-mon[29276]: osdmap e145: 8 total, 7 up, 8 in 2024-09-11T08:13:00.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:00 smithi150 ceph-mon[39612]: pgmap v220: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 2.0 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.30k op/s; 10131/97338 objects degraded (10.408%) 2024-09-11T08:13:00.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:00 smithi150 ceph-mon[39612]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:13:00.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:00 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1395060505' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:13:00.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:00 smithi150 ceph-mon[39612]: osdmap e145: 8 total, 7 up, 8 in 2024-09-11T08:13:00.242 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:00.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:00.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:00.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:00 smithi181 ceph-mon[32701]: pgmap v220: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 2.0 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.30k op/s; 10131/97338 objects degraded (10.408%) 2024-09-11T08:13:00.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:00 smithi181 ceph-mon[32701]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:13:00.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:00 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1395060505' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:13:00.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:00 smithi181 ceph-mon[32701]: osdmap e145: 8 total, 7 up, 8 in 2024-09-11T08:13:00.344 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:00.383 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:00.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:00.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:00.485 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:00.511 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:00.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:00.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:00.613 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:00.649 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:00.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:00.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:00.751 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:00.775 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:00.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:00.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:00.877 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:00.901 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:00.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:00.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:01.002 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:01.047 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:01.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:01.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:01.149 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:01.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:01.186 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:01.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:01.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:01 smithi150 ceph-mon[29276]: 6.f scrub starts 2024-09-11T08:13:01.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:01 smithi150 ceph-mon[29276]: 6.f scrub ok 2024-09-11T08:13:01.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:01 smithi150 ceph-mon[29276]: Health check update: Degraded data redundancy: 10131/97338 objects degraded (10.408%), 24 pgs degraded (PG_DEGRADED) 2024-09-11T08:13:01.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:01 smithi150 ceph-mon[29276]: 5.12 scrub starts 2024-09-11T08:13:01.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:01 smithi150 ceph-mon[29276]: 5.12 scrub ok 2024-09-11T08:13:01.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:01 smithi150 ceph-mon[39612]: 6.f scrub starts 2024-09-11T08:13:01.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:01 smithi150 ceph-mon[39612]: 6.f scrub ok 2024-09-11T08:13:01.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:01 smithi150 ceph-mon[39612]: Health check update: Degraded data redundancy: 10131/97338 objects degraded (10.408%), 24 pgs degraded (PG_DEGRADED) 2024-09-11T08:13:01.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:01 smithi150 ceph-mon[39612]: 5.12 scrub starts 2024-09-11T08:13:01.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:01 smithi150 ceph-mon[39612]: 5.12 scrub ok 2024-09-11T08:13:01.287 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:01.323 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:01.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:01 smithi181 ceph-mon[32701]: 6.f scrub starts 2024-09-11T08:13:01.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:01 smithi181 ceph-mon[32701]: 6.f scrub ok 2024-09-11T08:13:01.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:01 smithi181 ceph-mon[32701]: Health check update: Degraded data redundancy: 10131/97338 objects degraded (10.408%), 24 pgs degraded (PG_DEGRADED) 2024-09-11T08:13:01.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:01 smithi181 ceph-mon[32701]: 5.12 scrub starts 2024-09-11T08:13:01.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:01 smithi181 ceph-mon[32701]: 5.12 scrub ok 2024-09-11T08:13:01.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:01.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:01.426 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:01.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:01.466 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:01.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:01.567 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:01.611 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:01.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:01.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:01.712 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:01.735 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:01.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:01.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:01.837 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:01.869 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:01.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:01.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:01.971 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:01.993 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:01.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:01.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:02.096 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:02.121 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:02.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:02.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:02.223 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:02.248 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:02.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:02.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:02.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:02 smithi181 ceph-mon[32701]: 5.15 scrub starts 2024-09-11T08:13:02.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:02 smithi181 ceph-mon[32701]: 5.15 scrub ok 2024-09-11T08:13:02.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:02 smithi181 ceph-mon[32701]: pgmap v222: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 2.0 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s; 10131/97338 objects degraded (10.408%) 2024-09-11T08:13:02.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:02 smithi181 ceph-mon[32701]: 5.5 scrub starts 2024-09-11T08:13:02.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:02 smithi181 ceph-mon[32701]: 5.5 scrub ok 2024-09-11T08:13:02.351 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:02.399 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:02.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:02.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:02.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:02 smithi150 ceph-mon[29276]: 5.15 scrub starts 2024-09-11T08:13:02.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:02 smithi150 ceph-mon[29276]: 5.15 scrub ok 2024-09-11T08:13:02.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:02 smithi150 ceph-mon[29276]: pgmap v222: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 2.0 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s; 10131/97338 objects degraded (10.408%) 2024-09-11T08:13:02.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:02 smithi150 ceph-mon[29276]: 5.5 scrub starts 2024-09-11T08:13:02.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:02 smithi150 ceph-mon[29276]: 5.5 scrub ok 2024-09-11T08:13:02.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:02 smithi150 ceph-mon[39612]: 5.15 scrub starts 2024-09-11T08:13:02.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:02 smithi150 ceph-mon[39612]: 5.15 scrub ok 2024-09-11T08:13:02.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:02 smithi150 ceph-mon[39612]: pgmap v222: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 2.0 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s; 10131/97338 objects degraded (10.408%) 2024-09-11T08:13:02.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:02 smithi150 ceph-mon[39612]: 5.5 scrub starts 2024-09-11T08:13:02.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:02 smithi150 ceph-mon[39612]: 5.5 scrub ok 2024-09-11T08:13:02.503 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:02.527 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:02.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:02.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:02.629 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:02.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:02.652 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:02.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:02.753 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:02.776 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:02.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:02.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:02.806 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:13:02.879 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:02.904 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:02.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:02.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:02.985 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T08:13:03.006 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:03.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:03.081 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:03.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:03.182 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:03.233 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:03.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:03.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:03.323 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:03 smithi150 ceph-mon[29276]: 5.f deep-scrub starts 2024-09-11T08:13:03.323 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:03 smithi150 ceph-mon[29276]: 5.f deep-scrub ok 2024-09-11T08:13:03.323 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:03 smithi150 ceph-mon[29276]: 3.18 scrub starts 2024-09-11T08:13:03.323 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:03 smithi150 ceph-mon[29276]: 3.18 scrub ok 2024-09-11T08:13:03.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:03 smithi181 ceph-mon[32701]: 5.f deep-scrub starts 2024-09-11T08:13:03.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:03 smithi181 ceph-mon[32701]: 5.f deep-scrub ok 2024-09-11T08:13:03.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:03 smithi181 ceph-mon[32701]: 3.18 scrub starts 2024-09-11T08:13:03.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:03 smithi181 ceph-mon[32701]: 3.18 scrub ok 2024-09-11T08:13:03.325 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:03 smithi150 ceph-mon[39612]: 5.f deep-scrub starts 2024-09-11T08:13:03.325 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:03 smithi150 ceph-mon[39612]: 5.f deep-scrub ok 2024-09-11T08:13:03.326 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:03 smithi150 ceph-mon[39612]: 3.18 scrub starts 2024-09-11T08:13:03.326 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:03 smithi150 ceph-mon[39612]: 3.18 scrub ok 2024-09-11T08:13:03.335 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:03.357 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:03.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:03.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:03.459 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:03.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:03.504 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:03.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:03.606 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:03.630 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:03.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:03.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:03.732 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:03.756 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:03.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:03.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:03.857 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:03.895 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:03.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:03.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:03.997 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:04.025 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:04.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:04.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:04.085 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-11T08:13:04.085 INFO:teuthology.orchestra.run.smithi150.stdout: "success": "" 2024-09-11T08:13:04.085 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-11T08:13:04.128 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:04.174 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:04.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:04.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:04.251 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:04 smithi150 ceph-mon[29276]: 3.5 scrub starts 2024-09-11T08:13:04.251 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:04 smithi150 ceph-mon[29276]: 3.5 scrub ok 2024-09-11T08:13:04.252 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:04 smithi150 ceph-mon[29276]: pgmap v223: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 1.9 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s; 9778/92319 objects degraded (10.592%) 2024-09-11T08:13:04.252 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:04 smithi150 ceph-mon[29276]: 7.7 scrub starts 2024-09-11T08:13:04.252 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:04 smithi150 ceph-mon[29276]: 7.7 scrub ok 2024-09-11T08:13:04.252 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:04 smithi150 ceph-mon[39612]: 3.5 scrub starts 2024-09-11T08:13:04.252 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:04 smithi150 ceph-mon[39612]: 3.5 scrub ok 2024-09-11T08:13:04.252 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:04 smithi150 ceph-mon[39612]: pgmap v223: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 1.9 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s; 9778/92319 objects degraded (10.592%) 2024-09-11T08:13:04.252 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:04 smithi150 ceph-mon[39612]: 7.7 scrub starts 2024-09-11T08:13:04.252 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:04 smithi150 ceph-mon[39612]: 7.7 scrub ok 2024-09-11T08:13:04.277 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:04.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:04.321 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:04.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:04.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:04 smithi181 ceph-mon[32701]: 3.5 scrub starts 2024-09-11T08:13:04.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:04 smithi181 ceph-mon[32701]: 3.5 scrub ok 2024-09-11T08:13:04.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:04 smithi181 ceph-mon[32701]: pgmap v223: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 1.9 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s; 9778/92319 objects degraded (10.592%) 2024-09-11T08:13:04.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:04 smithi181 ceph-mon[32701]: 7.7 scrub starts 2024-09-11T08:13:04.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:04 smithi181 ceph-mon[32701]: 7.7 scrub ok 2024-09-11T08:13:04.422 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:04.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:04.464 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:04.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:04.565 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:04.588 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:04.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:04.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:04.690 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:04.729 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:04.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:04.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:04.831 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:04.855 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:04.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:04.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:04.957 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:04.993 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:04.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:04.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:05.095 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:05.118 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:05.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:05.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:05.220 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:05.247 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:05.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:05.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:05.349 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:05.364 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:13:05.364 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:13:05.365 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:05 smithi150 ceph-mon[29276]: 6.a scrub starts 2024-09-11T08:13:05.365 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:05 smithi150 ceph-mon[29276]: 6.a scrub ok 2024-09-11T08:13:05.365 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:05 smithi150 ceph-mon[29276]: 4.1a scrub starts 2024-09-11T08:13:05.365 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:05 smithi150 ceph-mon[29276]: 4.1a scrub ok 2024-09-11T08:13:05.365 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:05 smithi150 ceph-mon[29276]: 6.1 scrub starts 2024-09-11T08:13:05.365 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:05 smithi150 ceph-mon[29276]: 6.1 scrub ok 2024-09-11T08:13:05.365 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:05 smithi150 ceph-mon[39612]: 6.a scrub starts 2024-09-11T08:13:05.365 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:05 smithi150 ceph-mon[39612]: 6.a scrub ok 2024-09-11T08:13:05.365 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:05 smithi150 ceph-mon[39612]: 4.1a scrub starts 2024-09-11T08:13:05.366 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:05 smithi150 ceph-mon[39612]: 4.1a scrub ok 2024-09-11T08:13:05.366 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:05 smithi150 ceph-mon[39612]: 6.1 scrub starts 2024-09-11T08:13:05.366 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:05 smithi150 ceph-mon[39612]: 6.1 scrub ok 2024-09-11T08:13:05.371 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:05.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:05.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:05.404 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:13:05.404 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:13:05.432 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:13:05.432 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:13:05.473 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:05.473 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:13:05.474 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:13:05.508 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:05.508 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:13:05.509 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:13:05.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:05.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:05.536 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:13:05.536 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:13:05.568 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:13:05.569 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:13:05.569 INFO:teuthology.orchestra.run.smithi150.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-11T08:13:05.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:13:05 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:13:05] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:13:05.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:05 smithi181 ceph-mon[32701]: 6.a scrub starts 2024-09-11T08:13:05.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:05 smithi181 ceph-mon[32701]: 6.a scrub ok 2024-09-11T08:13:05.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:05 smithi181 ceph-mon[32701]: 4.1a scrub starts 2024-09-11T08:13:05.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:05 smithi181 ceph-mon[32701]: 4.1a scrub ok 2024-09-11T08:13:05.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:05 smithi181 ceph-mon[32701]: 6.1 scrub starts 2024-09-11T08:13:05.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:05 smithi181 ceph-mon[32701]: 6.1 scrub ok 2024-09-11T08:13:05.610 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:05.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:05.645 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:05.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:05.746 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:05.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:05.795 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:05.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:05.896 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:05.943 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:05.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:05.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:06.045 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:06.070 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:06.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:06.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:06.172 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:06.194 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:06.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:06.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:06.217 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:06 smithi150 ceph-mon[29276]: pgmap v224: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 1.6 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s; 8460/81309 objects degraded (10.405%) 2024-09-11T08:13:06.217 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:06 smithi150 ceph-mon[29276]: 3.1 scrub starts 2024-09-11T08:13:06.217 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:06 smithi150 ceph-mon[29276]: 3.1 scrub ok 2024-09-11T08:13:06.217 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:06 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/849448376' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:13:06.217 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:06 smithi150 ceph-mon[29276]: Health check update: Degraded data redundancy: 8460/81309 objects degraded (10.405%), 24 pgs degraded (PG_DEGRADED) 2024-09-11T08:13:06.218 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:06 smithi150 ceph-mon[39612]: pgmap v224: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 1.6 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s; 8460/81309 objects degraded (10.405%) 2024-09-11T08:13:06.218 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:06 smithi150 ceph-mon[39612]: 3.1 scrub starts 2024-09-11T08:13:06.218 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:06 smithi150 ceph-mon[39612]: 3.1 scrub ok 2024-09-11T08:13:06.218 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:06 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/849448376' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:13:06.218 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:06 smithi150 ceph-mon[39612]: Health check update: Degraded data redundancy: 8460/81309 objects degraded (10.405%), 24 pgs degraded (PG_DEGRADED) 2024-09-11T08:13:06.296 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:06.320 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:06.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:06.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:06.422 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:06.445 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:06.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:06.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:06.547 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:06.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:06 smithi181 ceph-mon[32701]: pgmap v224: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 1.6 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s; 8460/81309 objects degraded (10.405%) 2024-09-11T08:13:06.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:06 smithi181 ceph-mon[32701]: 3.1 scrub starts 2024-09-11T08:13:06.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:06 smithi181 ceph-mon[32701]: 3.1 scrub ok 2024-09-11T08:13:06.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:06 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/849448376' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:13:06.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:06 smithi181 ceph-mon[32701]: Health check update: Degraded data redundancy: 8460/81309 objects degraded (10.405%), 24 pgs degraded (PG_DEGRADED) 2024-09-11T08:13:06.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:06.588 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:06.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:06.689 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:06.728 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:06.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:06.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:06.830 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:06.893 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:06.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:06.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:06.995 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:07.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:07.036 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:07.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:07.137 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:07.161 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:07.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:07.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:07.263 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:07.304 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:07.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:07.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:07.405 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:07.441 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:07.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:07.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:07.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:07 smithi150 ceph-mon[29276]: 4.19 deep-scrub starts 2024-09-11T08:13:07.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:07 smithi150 ceph-mon[29276]: 4.19 deep-scrub ok 2024-09-11T08:13:07.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:07 smithi150 ceph-mon[29276]: 5.1c scrub starts 2024-09-11T08:13:07.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:07 smithi150 ceph-mon[29276]: 5.1c scrub ok 2024-09-11T08:13:07.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:07 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:13:07.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:07 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:13:07.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:07 smithi150 ceph-mon[39612]: 4.19 deep-scrub starts 2024-09-11T08:13:07.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:07 smithi150 ceph-mon[39612]: 4.19 deep-scrub ok 2024-09-11T08:13:07.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:07 smithi150 ceph-mon[39612]: 5.1c scrub starts 2024-09-11T08:13:07.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:07 smithi150 ceph-mon[39612]: 5.1c scrub ok 2024-09-11T08:13:07.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:07 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:13:07.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:07 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:13:07.544 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:07.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:07 smithi181 ceph-mon[32701]: 4.19 deep-scrub starts 2024-09-11T08:13:07.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:07 smithi181 ceph-mon[32701]: 4.19 deep-scrub ok 2024-09-11T08:13:07.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:07 smithi181 ceph-mon[32701]: 5.1c scrub starts 2024-09-11T08:13:07.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:07 smithi181 ceph-mon[32701]: 5.1c scrub ok 2024-09-11T08:13:07.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:07 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:13:07.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:07 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:13:07.580 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:07.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:07.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:07.629 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-11T08:13:07.630 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-11T08:13:07.683 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:07.706 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-11T08:13:07.707 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [ 2024-09-11T08:13:07.707 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:13:07.707 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:226366 7.b 7:df569640:::benchmark_data_smithi150_78508_object101925:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-11T08:13:07.707 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:13:07.123697+0000", 2024-09-11T08:13:07.707 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.0044265839999999999, 2024-09-11T08:13:07.707 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.004456181, 2024-09-11T08:13:07.707 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:13:07.707 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "waiting for sub ops", 2024-09-11T08:13:07.707 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:13:07.707 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:13:07.707 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:13:07.708 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 226366 2024-09-11T08:13:07.708 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:13:07.708 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:13:07.708 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:13:07.708 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:13:07.708 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:07.123697+0000", 2024-09-11T08:13:07.708 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:13:07.708 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:13:07.708 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:13:07.708 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:13:07.708 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:07.123697+0000", 2024-09-11T08:13:07.709 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:13:07.709 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:13:07.709 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:13:07.709 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:13:07.709 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:07.123696+0000", 2024-09-11T08:13:07.709 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:13:07.709 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:13:07.709 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:13:07.709 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:13:07.709 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:07.123702+0000", 2024-09-11T08:13:07.709 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.2149999999999998e-06 2024-09-11T08:13:07.709 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:13:07.710 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:13:07.710 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:13:07.710 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:07.123713+0000", 2024-09-11T08:13:07.710 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0431999999999999e-05 2024-09-11T08:13:07.710 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:13:07.710 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:13:07.710 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:13:07.710 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:07.123718+0000", 2024-09-11T08:13:07.710 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.614e-06 2024-09-11T08:13:07.711 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:13:07.711 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:13:07.711 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:13:07.711 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:07.123758+0000", 2024-09-11T08:13:07.711 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.0000000000000003e-05 2024-09-11T08:13:07.711 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:13:07.711 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:13:07.711 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:13:07.711 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:07.124020+0000", 2024-09-11T08:13:07.711 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00026197900000000002 2024-09-11T08:13:07.712 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:13:07.712 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:13:07.712 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,6", 2024-09-11T08:13:07.712 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:07.124414+0000", 2024-09-11T08:13:07.712 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00039367999999999998 2024-09-11T08:13:07.712 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:13:07.712 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:13:07.712 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:13:07.712 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:07.127582+0000", 2024-09-11T08:13:07.713 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0031682260000000001 2024-09-11T08:13:07.713 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:13:07.713 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:13:07.713 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:13:07.713 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:13:07.713 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:13:07.713 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop_reply(client.14790.0:226366 7.b e145/91)", 2024-09-11T08:13:07.713 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:13:07.127928+0000", 2024-09-11T08:13:07.713 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.000196193, 2024-09-11T08:13:07.713 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00030043499999999999, 2024-09-11T08:13:07.713 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:13:07.713 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "queued for pg", 2024-09-11T08:13:07.713 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:13:07.713 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:13:07.714 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:13:07.714 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:07.127928+0000", 2024-09-11T08:13:07.714 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:13:07.714 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:13:07.714 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:13:07.714 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:13:07.714 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:07.127928+0000", 2024-09-11T08:13:07.714 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:13:07.714 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:13:07.714 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:13:07.714 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:13:07.714 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:07.127927+0000", 2024-09-11T08:13:07.714 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:13:07.714 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:13:07.715 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:13:07.715 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:13:07.715 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:07.127933+0000", 2024-09-11T08:13:07.715 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.9989999999999997e-06 2024-09-11T08:13:07.715 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:13:07.715 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:13:07.715 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:13:07.715 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:07.127947+0000", 2024-09-11T08:13:07.715 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4166e-05 2024-09-11T08:13:07.715 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:13:07.715 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:13:07.715 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:13:07.715 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:07.127953+0000", 2024-09-11T08:13:07.716 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.8649999999999998e-06 2024-09-11T08:13:07.716 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:13:07.716 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:13:07.716 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:13:07.716 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:13:07.716 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-09-11T08:13:07.716 INFO:teuthology.orchestra.run.smithi150.stdout: "num_ops": 2 2024-09-11T08:13:07.716 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-11T08:13:07.726 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:07.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:07.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:07.828 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:07.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:07.866 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:07.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:07.967 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:08.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:08.020 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:08.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:08.121 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:08.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:08.143 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:08.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:08.244 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:08.277 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:08.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:08.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:08.323 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:13:07 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:13:08.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:08 smithi181 ceph-mon[32701]: 6.d deep-scrub starts 2024-09-11T08:13:08.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:08 smithi181 ceph-mon[32701]: 6.d deep-scrub ok 2024-09-11T08:13:08.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:08 smithi181 ceph-mon[32701]: pgmap v225: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 1.6 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s; 8460/81309 objects degraded (10.405%) 2024-09-11T08:13:08.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:08 smithi181 ceph-mon[32701]: 4.c scrub starts 2024-09-11T08:13:08.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:08 smithi181 ceph-mon[32701]: 4.c scrub ok 2024-09-11T08:13:08.379 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:08.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:08.415 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:08.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:08.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:08 smithi150 ceph-mon[29276]: 6.d deep-scrub starts 2024-09-11T08:13:08.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:08 smithi150 ceph-mon[29276]: 6.d deep-scrub ok 2024-09-11T08:13:08.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:08 smithi150 ceph-mon[29276]: pgmap v225: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 1.6 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s; 8460/81309 objects degraded (10.405%) 2024-09-11T08:13:08.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:08 smithi150 ceph-mon[29276]: 4.c scrub starts 2024-09-11T08:13:08.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:08 smithi150 ceph-mon[29276]: 4.c scrub ok 2024-09-11T08:13:08.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:08 smithi150 ceph-mon[39612]: 6.d deep-scrub starts 2024-09-11T08:13:08.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:08 smithi150 ceph-mon[39612]: 6.d deep-scrub ok 2024-09-11T08:13:08.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:08 smithi150 ceph-mon[39612]: pgmap v225: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 1.6 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s; 8460/81309 objects degraded (10.405%) 2024-09-11T08:13:08.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:08 smithi150 ceph-mon[39612]: 4.c scrub starts 2024-09-11T08:13:08.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:08 smithi150 ceph-mon[39612]: 4.c scrub ok 2024-09-11T08:13:08.517 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:08.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:08.553 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:08.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:08.654 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:08.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:08.697 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:08.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:08.798 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:08.834 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:08.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:08.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:08.879 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:13:08 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:13:08] "GET /metrics HTTP/1.1" 200 224001 "" "Prometheus/2.43.0" 2024-09-11T08:13:08.924 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-11T08:13:08.936 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:08.945 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:13:08.960 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:08.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:08.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:09.061 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:09.122 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:09.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:09.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:09.224 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:09.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:09 smithi150 ceph-mon[29276]: 7.0 scrub starts 2024-09-11T08:13:09.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:09 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:13:09.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:09 smithi150 ceph-mon[29276]: 4.9 deep-scrub starts 2024-09-11T08:13:09.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:09 smithi150 ceph-mon[29276]: 6.10 scrub starts 2024-09-11T08:13:09.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:09 smithi150 ceph-mon[29276]: 4.9 deep-scrub ok 2024-09-11T08:13:09.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:09 smithi150 ceph-mon[29276]: 6.10 scrub ok 2024-09-11T08:13:09.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:09 smithi150 ceph-mon[39612]: 7.0 scrub starts 2024-09-11T08:13:09.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:09 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:13:09.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:09 smithi150 ceph-mon[39612]: 4.9 deep-scrub starts 2024-09-11T08:13:09.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:09 smithi150 ceph-mon[39612]: 6.10 scrub starts 2024-09-11T08:13:09.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:09 smithi150 ceph-mon[39612]: 4.9 deep-scrub ok 2024-09-11T08:13:09.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:09 smithi150 ceph-mon[39612]: 6.10 scrub ok 2024-09-11T08:13:09.248 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:09.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:09.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:09.350 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:09.372 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:09.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:09.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:09.474 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:09.512 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:09.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:09.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:09.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:09 smithi181 ceph-mon[32701]: 7.0 scrub starts 2024-09-11T08:13:09.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:09 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:13:09.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:09 smithi181 ceph-mon[32701]: 4.9 deep-scrub starts 2024-09-11T08:13:09.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:09 smithi181 ceph-mon[32701]: 6.10 scrub starts 2024-09-11T08:13:09.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:09 smithi181 ceph-mon[32701]: 4.9 deep-scrub ok 2024-09-11T08:13:09.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:09 smithi181 ceph-mon[32701]: 6.10 scrub ok 2024-09-11T08:13:09.614 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:09.631 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T08:13:09.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:09.655 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:09.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:09.756 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:09.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:09.799 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:09.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:09.900 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:09.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:09.954 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:09.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:10.055 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:10.081 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:10.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:10.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:10.183 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:10.214 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:10.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:10.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:10.316 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:10.340 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:10.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:10.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:10.442 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:10.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:10.480 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:10.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:10.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:10 smithi150 ceph-mon[29276]: 7.0 scrub ok 2024-09-11T08:13:10.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:10 smithi150 ceph-mon[29276]: pgmap v226: 148 pgs: 1 active+clean+scrubbing, 34 active+undersized, 24 active+undersized+degraded, 89 active+clean; 1.3 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s; 6822/65652 objects degraded (10.391%) 2024-09-11T08:13:10.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:10 smithi150 ceph-mon[29276]: 3.1a scrub starts 2024-09-11T08:13:10.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:10 smithi150 ceph-mon[29276]: 3.1a scrub ok 2024-09-11T08:13:10.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:10 smithi150 ceph-mon[29276]: 3.6 scrub starts 2024-09-11T08:13:10.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:10 smithi150 ceph-mon[29276]: 3.6 scrub ok 2024-09-11T08:13:10.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:10 smithi150 ceph-mon[39612]: 7.0 scrub ok 2024-09-11T08:13:10.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:10 smithi150 ceph-mon[39612]: pgmap v226: 148 pgs: 1 active+clean+scrubbing, 34 active+undersized, 24 active+undersized+degraded, 89 active+clean; 1.3 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s; 6822/65652 objects degraded (10.391%) 2024-09-11T08:13:10.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:10 smithi150 ceph-mon[39612]: 3.1a scrub starts 2024-09-11T08:13:10.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:10 smithi150 ceph-mon[39612]: 3.1a scrub ok 2024-09-11T08:13:10.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:10 smithi150 ceph-mon[39612]: 3.6 scrub starts 2024-09-11T08:13:10.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:10 smithi150 ceph-mon[39612]: 3.6 scrub ok 2024-09-11T08:13:10.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:10 smithi181 ceph-mon[32701]: 7.0 scrub ok 2024-09-11T08:13:10.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:10 smithi181 ceph-mon[32701]: pgmap v226: 148 pgs: 1 active+clean+scrubbing, 34 active+undersized, 24 active+undersized+degraded, 89 active+clean; 1.3 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s; 6822/65652 objects degraded (10.391%) 2024-09-11T08:13:10.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:10 smithi181 ceph-mon[32701]: 3.1a scrub starts 2024-09-11T08:13:10.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:10 smithi181 ceph-mon[32701]: 3.1a scrub ok 2024-09-11T08:13:10.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:10 smithi181 ceph-mon[32701]: 3.6 scrub starts 2024-09-11T08:13:10.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:10 smithi181 ceph-mon[32701]: 3.6 scrub ok 2024-09-11T08:13:10.581 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:10.604 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:10.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:10.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:10.706 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:10.728 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:10.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:10.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:10.830 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:10.856 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:10.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:10.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:10.957 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:10.979 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:10.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:10.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:11.080 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:11.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:11.105 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:11.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:11.206 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:11.230 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:11.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:11.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:11.332 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:11.368 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:11.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:11.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:11.469 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:11.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:11 smithi150 ceph-mon[29276]: 6.1a scrub starts 2024-09-11T08:13:11.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:11 smithi150 ceph-mon[29276]: 6.1a scrub ok 2024-09-11T08:13:11.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:11 smithi150 ceph-mon[29276]: 5.0 deep-scrub starts 2024-09-11T08:13:11.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:11 smithi150 ceph-mon[29276]: 5.0 deep-scrub ok 2024-09-11T08:13:11.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:11 smithi150 ceph-mon[29276]: Health check update: Degraded data redundancy: 6822/65652 objects degraded (10.391%), 24 pgs degraded (PG_DEGRADED) 2024-09-11T08:13:11.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:11 smithi150 ceph-mon[39612]: 6.1a scrub starts 2024-09-11T08:13:11.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:11 smithi150 ceph-mon[39612]: 6.1a scrub ok 2024-09-11T08:13:11.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:11 smithi150 ceph-mon[39612]: 5.0 deep-scrub starts 2024-09-11T08:13:11.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:11 smithi150 ceph-mon[39612]: 5.0 deep-scrub ok 2024-09-11T08:13:11.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:11 smithi150 ceph-mon[39612]: Health check update: Degraded data redundancy: 6822/65652 objects degraded (10.391%), 24 pgs degraded (PG_DEGRADED) 2024-09-11T08:13:11.493 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:11.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:11.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:11.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:11 smithi181 ceph-mon[32701]: 6.1a scrub starts 2024-09-11T08:13:11.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:11 smithi181 ceph-mon[32701]: 6.1a scrub ok 2024-09-11T08:13:11.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:11 smithi181 ceph-mon[32701]: 5.0 deep-scrub starts 2024-09-11T08:13:11.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:11 smithi181 ceph-mon[32701]: 5.0 deep-scrub ok 2024-09-11T08:13:11.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:11 smithi181 ceph-mon[32701]: Health check update: Degraded data redundancy: 6822/65652 objects degraded (10.391%), 24 pgs degraded (PG_DEGRADED) 2024-09-11T08:13:11.595 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:11.628 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:11.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:11.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:11.730 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:11.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:11.771 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:11.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:11.872 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:11.876 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [7] live_osds: [0, 1, 2, 3, 4, 5, 6] 2024-09-11T08:13:11.876 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T08:13:11.876 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-09-11T08:13:11.876 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd test-reweight-by-utilization 110 0.05 2024-09-11T08:13:11.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:11.912 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:11.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:12.013 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:12.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:12.056 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:12.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:12.157 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:12.190 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:12.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:12.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:12.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:12 smithi150 ceph-mon[39612]: 3.c scrub starts 2024-09-11T08:13:12.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:12 smithi150 ceph-mon[39612]: 3.c scrub ok 2024-09-11T08:13:12.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:12 smithi150 ceph-mon[39612]: pgmap v227: 148 pgs: 1 active+clean+scrubbing, 34 active+undersized, 24 active+undersized+degraded, 89 active+clean; 1.3 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 871 B/s rd, 0 B/s wr, 899 op/s; 6822/65652 objects degraded (10.391%) 2024-09-11T08:13:12.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:12 smithi150 ceph-mon[39612]: 6.1e scrub starts 2024-09-11T08:13:12.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:12 smithi150 ceph-mon[39612]: 6.1e scrub ok 2024-09-11T08:13:12.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:12 smithi150 ceph-mon[39612]: 4.1d scrub starts 2024-09-11T08:13:12.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:12 smithi150 ceph-mon[39612]: 4.1d scrub ok 2024-09-11T08:13:12.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:12 smithi150 ceph-mon[39612]: 2.1 scrub starts 2024-09-11T08:13:12.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:12 smithi150 ceph-mon[39612]: 2.1 scrub ok 2024-09-11T08:13:12.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:12 smithi150 ceph-mon[29276]: 3.c scrub starts 2024-09-11T08:13:12.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:12 smithi150 ceph-mon[29276]: 3.c scrub ok 2024-09-11T08:13:12.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:12 smithi150 ceph-mon[29276]: pgmap v227: 148 pgs: 1 active+clean+scrubbing, 34 active+undersized, 24 active+undersized+degraded, 89 active+clean; 1.3 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 871 B/s rd, 0 B/s wr, 899 op/s; 6822/65652 objects degraded (10.391%) 2024-09-11T08:13:12.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:12 smithi150 ceph-mon[29276]: 6.1e scrub starts 2024-09-11T08:13:12.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:12 smithi150 ceph-mon[29276]: 6.1e scrub ok 2024-09-11T08:13:12.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:12 smithi150 ceph-mon[29276]: 4.1d scrub starts 2024-09-11T08:13:12.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:12 smithi150 ceph-mon[29276]: 4.1d scrub ok 2024-09-11T08:13:12.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:12 smithi150 ceph-mon[29276]: 2.1 scrub starts 2024-09-11T08:13:12.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:12 smithi150 ceph-mon[29276]: 2.1 scrub ok 2024-09-11T08:13:12.292 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:12.316 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:12.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:12.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:12.418 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:12.440 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:12.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:12.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:12.542 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:12.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:12 smithi181 ceph-mon[32701]: 3.c scrub starts 2024-09-11T08:13:12.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:12 smithi181 ceph-mon[32701]: 3.c scrub ok 2024-09-11T08:13:12.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:12 smithi181 ceph-mon[32701]: pgmap v227: 148 pgs: 1 active+clean+scrubbing, 34 active+undersized, 24 active+undersized+degraded, 89 active+clean; 1.3 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 871 B/s rd, 0 B/s wr, 899 op/s; 6822/65652 objects degraded (10.391%) 2024-09-11T08:13:12.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:12 smithi181 ceph-mon[32701]: 6.1e scrub starts 2024-09-11T08:13:12.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:12 smithi181 ceph-mon[32701]: 6.1e scrub ok 2024-09-11T08:13:12.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:12 smithi181 ceph-mon[32701]: 4.1d scrub starts 2024-09-11T08:13:12.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:12 smithi181 ceph-mon[32701]: 4.1d scrub ok 2024-09-11T08:13:12.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:12 smithi181 ceph-mon[32701]: 2.1 scrub starts 2024-09-11T08:13:12.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:12 smithi181 ceph-mon[32701]: 2.1 scrub ok 2024-09-11T08:13:12.579 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:12.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:12.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:12.681 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:12.704 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:12.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:12.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:12.806 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:12.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:12.879 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:12.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:12.980 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:13.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:13.022 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:13.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:13.123 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:13.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:13.164 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:13.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:13.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:13 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:13:13.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:13 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:13:13.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:13 smithi150 ceph-mon[29276]: 5.e scrub starts 2024-09-11T08:13:13.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:13 smithi150 ceph-mon[29276]: 5.e scrub ok 2024-09-11T08:13:13.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:13 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:13:13.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:13 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:13:13.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:13 smithi150 ceph-mon[29276]: osdmap e146: 8 total, 7 up, 8 in 2024-09-11T08:13:13.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:13 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:13:13.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:13 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:13:13.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:13 smithi150 ceph-mon[39612]: 5.e scrub starts 2024-09-11T08:13:13.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:13 smithi150 ceph-mon[39612]: 5.e scrub ok 2024-09-11T08:13:13.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:13 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:13:13.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:13 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:13:13.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:13 smithi150 ceph-mon[39612]: osdmap e146: 8 total, 7 up, 8 in 2024-09-11T08:13:13.266 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:13.308 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:13.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:13.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:13.410 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:13.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:13.467 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:13.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:13.502 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:13:13.526 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:13:13.569 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:13.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:13 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:13:13.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:13 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:13:13.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:13 smithi181 ceph-mon[32701]: 5.e scrub starts 2024-09-11T08:13:13.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:13 smithi181 ceph-mon[32701]: 5.e scrub ok 2024-09-11T08:13:13.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:13 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:13:13.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:13 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:13:13.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:13 smithi181 ceph-mon[32701]: osdmap e146: 8 total, 7 up, 8 in 2024-09-11T08:13:13.576 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:13:13.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:13.633 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:13.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:13.734 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:13.757 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:13.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:13.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:13.859 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:13.881 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:13.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:13.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:13.983 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:14.021 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:14.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:14.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:14.123 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:14.160 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:14.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:14.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:14.217 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:14 smithi150 ceph-mon[29276]: pgmap v229: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 1.2 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s; 6517/60990 objects degraded (10.685%) 2024-09-11T08:13:14.217 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:14 smithi150 ceph-mon[29276]: 4.13 scrub starts 2024-09-11T08:13:14.217 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:14 smithi150 ceph-mon[29276]: 4.13 scrub ok 2024-09-11T08:13:14.218 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:14 smithi150 ceph-mon[39612]: pgmap v229: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 1.2 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s; 6517/60990 objects degraded (10.685%) 2024-09-11T08:13:14.218 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:14 smithi150 ceph-mon[39612]: 4.13 scrub starts 2024-09-11T08:13:14.218 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:14 smithi150 ceph-mon[39612]: 4.13 scrub ok 2024-09-11T08:13:14.262 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:14.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:14.306 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:14.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:14.407 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:14.429 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:14.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:14.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:14.531 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:14.565 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:14.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:14.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:14.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:14 smithi181 ceph-mon[32701]: pgmap v229: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 1.2 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s; 6517/60990 objects degraded (10.685%) 2024-09-11T08:13:14.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:14 smithi181 ceph-mon[32701]: 4.13 scrub starts 2024-09-11T08:13:14.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:14 smithi181 ceph-mon[32701]: 4.13 scrub ok 2024-09-11T08:13:14.667 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:14.708 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:14.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:14.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:14.810 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:14.845 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:14.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:14.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:14.946 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:14.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:14.971 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:14.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:15.072 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:15.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:15.108 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:15.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:15.209 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:15.258 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:15.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:15.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:15.360 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:15.419 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:15.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:15.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:15.449 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:15 smithi150 ceph-mon[29276]: 4.b scrub starts 2024-09-11T08:13:15.449 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:15 smithi150 ceph-mon[29276]: 4.b scrub ok 2024-09-11T08:13:15.449 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:15 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/805505267' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:13:15.449 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:15 smithi150 ceph-mon[39612]: 4.b scrub starts 2024-09-11T08:13:15.449 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:15 smithi150 ceph-mon[39612]: 4.b scrub ok 2024-09-11T08:13:15.449 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:15 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/805505267' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:13:15.521 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:15.569 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:15.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:15.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:15.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:15 smithi181 ceph-mon[32701]: 4.b scrub starts 2024-09-11T08:13:15.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:15 smithi181 ceph-mon[32701]: 4.b scrub ok 2024-09-11T08:13:15.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:15 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/805505267' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:13:15.574 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:13:15 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:13:15] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:13:15.672 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:15.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:15.713 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:15.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:15.814 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:15.863 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:15.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:15.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:15.965 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:16.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:16.005 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:16.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:16.106 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:16.153 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:16.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:16.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:16.255 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:16.279 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:16.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:16.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:16.380 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:16.405 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:16.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:16.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:16.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:16 smithi150 ceph-mon[29276]: pgmap v230: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 1.0 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s; 5218/50196 objects degraded (10.395%) 2024-09-11T08:13:16.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:16 smithi150 ceph-mon[29276]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:13:16.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:16 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/805505267' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:13:16.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:16 smithi150 ceph-mon[29276]: osdmap e147: 8 total, 7 up, 8 in 2024-09-11T08:13:16.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:16 smithi150 ceph-mon[29276]: Health check update: Degraded data redundancy: 5218/50196 objects degraded (10.395%), 24 pgs degraded (PG_DEGRADED) 2024-09-11T08:13:16.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:16 smithi150 ceph-mon[39612]: pgmap v230: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 1.0 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s; 5218/50196 objects degraded (10.395%) 2024-09-11T08:13:16.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:16 smithi150 ceph-mon[39612]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:13:16.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:16 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/805505267' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:13:16.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:16 smithi150 ceph-mon[39612]: osdmap e147: 8 total, 7 up, 8 in 2024-09-11T08:13:16.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:16 smithi150 ceph-mon[39612]: Health check update: Degraded data redundancy: 5218/50196 objects degraded (10.395%), 24 pgs degraded (PG_DEGRADED) 2024-09-11T08:13:16.507 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:16.543 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:16.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:16.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:16.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:16 smithi181 ceph-mon[32701]: pgmap v230: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 1.0 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s; 5218/50196 objects degraded (10.395%) 2024-09-11T08:13:16.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:16 smithi181 ceph-mon[32701]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:13:16.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:16 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/805505267' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:13:16.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:16 smithi181 ceph-mon[32701]: osdmap e147: 8 total, 7 up, 8 in 2024-09-11T08:13:16.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:16 smithi181 ceph-mon[32701]: Health check update: Degraded data redundancy: 5218/50196 objects degraded (10.395%), 24 pgs degraded (PG_DEGRADED) 2024-09-11T08:13:16.645 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:16.669 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:16.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:16.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:16.770 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:16.792 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:16.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:16.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:16.894 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:16.939 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:16.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:16.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:17.041 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:17.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:17.083 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:17.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:17.184 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:17.206 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:17.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:17.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:17.252 INFO:teuthology.orchestra.run.smithi150.stderr:noscrub is set 2024-09-11T08:13:17.308 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:17.330 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:17.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:17.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:17.432 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:17.458 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:17.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:17.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:17.560 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:17.598 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:17.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:17.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:17.700 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:17.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:17.740 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:17.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:17.841 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:17.879 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:17.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:17.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:17.981 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:18.004 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:18.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:18.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:18.106 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:18.139 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:18.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:18.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:18.241 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:18.287 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:18.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:18.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:18.323 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:13:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:13:18.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:18 smithi181 ceph-mon[32701]: pgmap v232: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 1.0 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 644 op/s; 5218/50196 objects degraded (10.395%) 2024-09-11T08:13:18.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:18 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:13:18.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:18 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:13:18.389 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:18.431 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:18.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:18.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:18.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:18 smithi150 ceph-mon[29276]: pgmap v232: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 1.0 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 644 op/s; 5218/50196 objects degraded (10.395%) 2024-09-11T08:13:18.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:18 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:13:18.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:18 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:13:18.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:18 smithi150 ceph-mon[39612]: pgmap v232: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 1.0 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 644 op/s; 5218/50196 objects degraded (10.395%) 2024-09-11T08:13:18.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:18 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:13:18.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:18 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:13:18.533 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:18.559 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:18.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:18.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:18.661 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:18.686 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:18.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:18.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:18.788 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:18.826 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:18.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:18.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:18.928 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:18.959 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:18.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:18.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:18.988 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:13:18 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:13:18] "GET /metrics HTTP/1.1" 200 224001 "" "Prometheus/2.43.0" 2024-09-11T08:13:19.061 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:19.097 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:19.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:19.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:19.198 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:19.223 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:19.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:19.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:19.325 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:19.350 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:19.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:19.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:19.453 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:19.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:19.489 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:19.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:19.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:19 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:13:19.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:19 smithi150 ceph-mon[29276]: osdmap e148: 8 total, 7 up, 8 in 2024-09-11T08:13:19.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:19 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:13:19.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:19 smithi150 ceph-mon[39612]: osdmap e148: 8 total, 7 up, 8 in 2024-09-11T08:13:19.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:19 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:13:19.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:19 smithi181 ceph-mon[32701]: osdmap e148: 8 total, 7 up, 8 in 2024-09-11T08:13:19.591 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:19.623 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:19.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:19.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:19.725 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:19.763 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:19.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:19.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:19.865 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:19.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:19.915 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:19.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:20.017 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:20.054 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:20.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:20.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:20.066 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-11T08:13:20.067 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [], 2024-09-11T08:13:20.067 INFO:teuthology.orchestra.run.smithi150.stdout: "complaint_time": 30, 2024-09-11T08:13:20.067 INFO:teuthology.orchestra.run.smithi150.stdout: "num_blocked_ops": 0 2024-09-11T08:13:20.067 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-11T08:13:20.157 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:20.199 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:20.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:20.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:20.301 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:20.337 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:20.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:20.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:20.439 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:20.464 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:20.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:20.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:20.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:20 smithi150 ceph-mon[29276]: pgmap v234: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 702 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.65k op/s; 3589/34239 objects degraded (10.482%) 2024-09-11T08:13:20.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:20 smithi150 ceph-mon[39612]: pgmap v234: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 702 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.65k op/s; 3589/34239 objects degraded (10.482%) 2024-09-11T08:13:20.566 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:20.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:20 smithi181 ceph-mon[32701]: pgmap v234: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 702 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.65k op/s; 3589/34239 objects degraded (10.482%) 2024-09-11T08:13:20.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:20.600 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:20.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:20.701 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:20.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:20.741 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:20.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:20.842 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:20.879 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:20.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:20.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:20.981 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:21.016 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:21.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:21.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:21.119 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:21.152 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:21.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:21.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:21.254 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:21.278 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:21.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:21.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:21.381 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:21.404 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:21.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:21.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:21.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:21 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:13:21.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:21 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:13:21.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:21 smithi150 ceph-mon[29276]: osdmap e149: 8 total, 7 up, 8 in 2024-09-11T08:13:21.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:21 smithi150 ceph-mon[29276]: Health check update: Degraded data redundancy: 3589/34239 objects degraded (10.482%), 24 pgs degraded (PG_DEGRADED) 2024-09-11T08:13:21.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:21 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:13:21.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:21 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:13:21.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:21 smithi150 ceph-mon[39612]: osdmap e149: 8 total, 7 up, 8 in 2024-09-11T08:13:21.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:21 smithi150 ceph-mon[39612]: Health check update: Degraded data redundancy: 3589/34239 objects degraded (10.482%), 24 pgs degraded (PG_DEGRADED) 2024-09-11T08:13:21.506 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:21.540 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:21.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:21.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:21.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:21 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:13:21.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:21 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:13:21.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:21 smithi181 ceph-mon[32701]: osdmap e149: 8 total, 7 up, 8 in 2024-09-11T08:13:21.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:21 smithi181 ceph-mon[32701]: Health check update: Degraded data redundancy: 3589/34239 objects degraded (10.482%), 24 pgs degraded (PG_DEGRADED) 2024-09-11T08:13:21.642 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:21.667 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:21.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:21.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:21.769 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:21.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:21.793 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:21.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:21.854 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-11T08:13:21.894 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:21.942 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:21.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:21.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:22.044 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:22.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:22.101 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:22.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:22.202 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:22.214 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:22 smithi150 ceph-mon[29276]: pgmap v236: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 702 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 886 op/s; 3589/34239 objects degraded (10.482%) 2024-09-11T08:13:22.214 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:22 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:13:22.214 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:22 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:13:22.214 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:22 smithi150 ceph-mon[39612]: pgmap v236: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 702 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 886 op/s; 3589/34239 objects degraded (10.482%) 2024-09-11T08:13:22.214 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:22 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:13:22.214 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:22 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:13:22.268 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:22.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:22.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:22.370 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:22.433 INFO:teuthology.orchestra.run.smithi150.stdout:moved 0 / 444 (0%) 2024-09-11T08:13:22.433 INFO:teuthology.orchestra.run.smithi150.stdout:avg 63.4286 2024-09-11T08:13:22.433 INFO:teuthology.orchestra.run.smithi150.stdout:stddev 11.0158 -> 11.0158 (expected baseline 7.37342) 2024-09-11T08:13:22.433 INFO:teuthology.orchestra.run.smithi150.stdout:min osd.2 with 44 -> 44 pgs (0.693694 -> 0.693694 * mean) 2024-09-11T08:13:22.433 INFO:teuthology.orchestra.run.smithi150.stdout:max osd.3 with 65 -> 65 pgs (1.02477 -> 1.02477 * mean) 2024-09-11T08:13:22.433 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:13:22.433 INFO:teuthology.orchestra.run.smithi150.stdout:oload 110 2024-09-11T08:13:22.433 INFO:teuthology.orchestra.run.smithi150.stdout:max_change 0.05 2024-09-11T08:13:22.434 INFO:teuthology.orchestra.run.smithi150.stdout:max_change_osds 4 2024-09-11T08:13:22.434 INFO:teuthology.orchestra.run.smithi150.stdout:average_utilization 0.0078 2024-09-11T08:13:22.434 INFO:teuthology.orchestra.run.smithi150.stdout:overload_utilization 0.0086 2024-09-11T08:13:22.434 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7 weight 1.0000 -> 0.9500 2024-09-11T08:13:22.434 INFO:teuthology.orchestra.run.smithi150.stderr:no change 2024-09-11T08:13:22.461 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:13:22 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:13:22.428+0000 7fb0c2764640 -1 mgr.server reply reply (1) Operation not permitted no change 2024-09-11T08:13:22.463 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:22.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:22.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:22.565 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:22.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:22 smithi181 ceph-mon[32701]: pgmap v236: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 702 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 886 op/s; 3589/34239 objects degraded (10.482%) 2024-09-11T08:13:22.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:22 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:13:22.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:22 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:13:22.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:22.602 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:22.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:22.703 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:22.726 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:22.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:22.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:22.734 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:13:22.734 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:13:22.770 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:13:22.771 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:13:22.810 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:13:22.811 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:13:22.827 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:22.846 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:13:22.846 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:13:22.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:22.868 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:22.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:22.877 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:13:22.877 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:13:22.908 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:13:22.908 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:13:22.940 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:13:22.940 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:13:22.946 INFO:teuthology.orchestra.run.smithi150.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-11T08:13:22.969 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:22.993 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:22.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:22.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:23.095 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:23.131 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:23.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:23.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:23.233 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:23.277 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:23.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:23.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:23.379 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:23.420 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:23.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:23.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:23.462 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:23 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2179560653' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:13:23.462 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:23 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2179560653' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:13:23.503 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-11T08:13:23.503 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd reweight-by-utilization 110 3.0 2024-09-11T08:13:23.523 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:23.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:23 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2179560653' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:13:23.586 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:23.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:23.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:23.688 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:23.711 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:23.712 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T08:13:23.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:23.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:23.813 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:23.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:23.837 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:23.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:23.938 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:23.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:23.966 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:23.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:24.067 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:24.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:24.115 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:24.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:24.211 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-11T08:13:24.211 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-11T08:13:24.216 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:24.245 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:13:24.275 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:24.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:24.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:24.310 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:13:24.377 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:24.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:24.423 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:24.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:24.428 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:24 smithi150 ceph-mon[29276]: from='client.15000 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:13:24.428 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:24 smithi150 ceph-mon[29276]: pgmap v237: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 605 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 667 B/s rd, 0 B/s wr, 895 op/s; 3295/29604 objects degraded (11.130%) 2024-09-11T08:13:24.429 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:24 smithi150 ceph-mon[39612]: from='client.15000 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:13:24.429 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:24 smithi150 ceph-mon[39612]: pgmap v237: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 605 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 667 B/s rd, 0 B/s wr, 895 op/s; 3295/29604 objects degraded (11.130%) 2024-09-11T08:13:24.523 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:24.546 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:24.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:24.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:24.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:24 smithi181 ceph-mon[32701]: from='client.15000 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:13:24.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:24 smithi181 ceph-mon[32701]: pgmap v237: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 605 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 667 B/s rd, 0 B/s wr, 895 op/s; 3295/29604 objects degraded (11.130%) 2024-09-11T08:13:24.648 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:24.688 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:24.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:24.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:24.790 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:24.836 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:24.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:24.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:24.938 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:24.960 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:24.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:24.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:25.062 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:25.086 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:25.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:25.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:25.188 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:25.230 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:25.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:25.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:25.331 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:25.355 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:25.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:25.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:25.458 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:25.482 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:25.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:25.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:25.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:13:25 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:13:25] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:13:25.584 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:25.609 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:25.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:25.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:25.695 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:13:25.711 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:25.763 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:25.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:25.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:25.865 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:25.888 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:25.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:25.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:25.989 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:26.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:26.030 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:26.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:26.131 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:26.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:26.154 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:26.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:26.212 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T08:13:26.255 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:26.278 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:26.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:26.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:26.379 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:26.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:26.437 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:26.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:26.471 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:26 smithi150 ceph-mon[29276]: pgmap v238: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 399 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.27k op/s; 2103/19701 objects degraded (10.675%) 2024-09-11T08:13:26.471 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:26 smithi150 ceph-mon[29276]: Health check update: Degraded data redundancy: 2103/19701 objects degraded (10.675%), 24 pgs degraded (PG_DEGRADED) 2024-09-11T08:13:26.471 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:26 smithi150 ceph-mon[39612]: pgmap v238: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 399 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.27k op/s; 2103/19701 objects degraded (10.675%) 2024-09-11T08:13:26.471 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:26 smithi150 ceph-mon[39612]: Health check update: Degraded data redundancy: 2103/19701 objects degraded (10.675%), 24 pgs degraded (PG_DEGRADED) 2024-09-11T08:13:26.538 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:26.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:26 smithi181 ceph-mon[32701]: pgmap v238: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 399 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.27k op/s; 2103/19701 objects degraded (10.675%) 2024-09-11T08:13:26.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:26 smithi181 ceph-mon[32701]: Health check update: Degraded data redundancy: 2103/19701 objects degraded (10.675%), 24 pgs degraded (PG_DEGRADED) 2024-09-11T08:13:26.578 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:26.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:26.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:26.680 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:26.704 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:26.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:26.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:26.806 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:26.843 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:26.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:26.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:26.945 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:26.968 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:26.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:26.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:27.071 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:27.121 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:27.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:27.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:27.223 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:27.246 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:27.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:27.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:27.348 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:27.389 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:27.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:27.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:27.490 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:27.540 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:27.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:27.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:27.642 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:27.664 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:27.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:27.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:27.766 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:27.823 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:27.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:27.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:27.925 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:27.948 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:27.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:27.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:28.050 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:28.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:28.090 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:28.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:28.191 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:28.202 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-09-11T08:13:28.202 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:13:28.213 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:28.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:28.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:28.315 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:28.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:28 smithi181 ceph-mon[32701]: pgmap v239: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 399 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 592 B/s rd, 0 B/s wr, 560 op/s; 2103/19701 objects degraded (10.675%) 2024-09-11T08:13:28.323 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:13:27 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:13:28.352 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:28.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:28.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:28.453 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:28.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:28 smithi150 ceph-mon[29276]: pgmap v239: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 399 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 592 B/s rd, 0 B/s wr, 560 op/s; 2103/19701 objects degraded (10.675%) 2024-09-11T08:13:28.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:28 smithi150 ceph-mon[39612]: pgmap v239: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 399 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 592 B/s rd, 0 B/s wr, 560 op/s; 2103/19701 objects degraded (10.675%) 2024-09-11T08:13:28.491 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:28.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:28.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:28.593 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:28.635 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:28.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:28.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:28.737 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:28.762 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:28.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:28.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:28.864 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:28.898 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:28.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:28.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:28.989 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:13:28 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:13:28] "GET /metrics HTTP/1.1" 200 223999 "" "Prometheus/2.43.0" 2024-09-11T08:13:29.000 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:29.024 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:29.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:29.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:29.091 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-11T08:13:29.126 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:29.140 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Removed 124438 objects 2024-09-11T08:13:29.140 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Clean up completed and total clean up time :121.04 2024-09-11T08:13:29.155 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:29.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:29.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:29.220 INFO:teuthology.orchestra.run.smithi150.stderr:nodeep-scrub is set 2024-09-11T08:13:29.257 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:29.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:29.281 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:29.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:29.382 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:29.403 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:29.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:29.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:29.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:29 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:13:29.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:29 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/869184505' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:13:29.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:29 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:13:29.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:29 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:13:29.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:29 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/869184505' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:13:29.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:29 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:13:29.505 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:29.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:29.528 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:29.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:29.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:29 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:13:29.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:29 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/869184505' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:13:29.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:29 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:13:29.629 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:29.651 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:29.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:29.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:29.753 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:29.774 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:29.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:29.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:29.876 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:29.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:29.897 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:29.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:29.998 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:30.020 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:30.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:30.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:30.121 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:30.143 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:30.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:30.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:30.169 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:13:30.244 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:30.267 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:30.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:30.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:30.369 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:30.391 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:30.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:30.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:30.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:30 smithi150 ceph-mon[29276]: pgmap v240: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 81 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 991 op/s; 466/4467 objects degraded (10.432%) 2024-09-11T08:13:30.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:30 smithi150 ceph-mon[29276]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:13:30.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:30 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:13:30.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:30 smithi150 ceph-mon[29276]: osdmap e150: 8 total, 7 up, 8 in 2024-09-11T08:13:30.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:30 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:13:30.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:30 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:13:30.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:30 smithi150 ceph-mon[39612]: pgmap v240: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 81 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 991 op/s; 466/4467 objects degraded (10.432%) 2024-09-11T08:13:30.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:30 smithi150 ceph-mon[39612]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:13:30.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:30 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:13:30.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:30 smithi150 ceph-mon[39612]: osdmap e150: 8 total, 7 up, 8 in 2024-09-11T08:13:30.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:30 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:13:30.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:30 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:13:30.493 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:30.519 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:30.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:30.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:30.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:30 smithi181 ceph-mon[32701]: pgmap v240: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 81 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 991 op/s; 466/4467 objects degraded (10.432%) 2024-09-11T08:13:30.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:30 smithi181 ceph-mon[32701]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:13:30.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:30 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:13:30.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:30 smithi181 ceph-mon[32701]: osdmap e150: 8 total, 7 up, 8 in 2024-09-11T08:13:30.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:30 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:13:30.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:30 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:13:30.620 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:30.642 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:30.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:30.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:30.744 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:30.770 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:30.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:30.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:30.871 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:30.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:30.895 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:30.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:30.996 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:31.021 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:31.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:31.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:31.123 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:31.147 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:31.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:31.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:31.249 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:31.271 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:31.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:31.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:31.373 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:31.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:31.400 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:31.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:31.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:31 smithi150 ceph-mon[29276]: osdmap e151: 8 total, 7 up, 8 in 2024-09-11T08:13:31.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:31 smithi150 ceph-mon[29276]: Health check update: Degraded data redundancy: 466/4467 objects degraded (10.432%), 24 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-09-11T08:13:31.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:31 smithi150 ceph-mon[39612]: osdmap e151: 8 total, 7 up, 8 in 2024-09-11T08:13:31.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:31 smithi150 ceph-mon[39612]: Health check update: Degraded data redundancy: 466/4467 objects degraded (10.432%), 24 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-09-11T08:13:31.500 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:31.522 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:31.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:31.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:31.556 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-09-11T08:13:31.556 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-09-11T08:13:31.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:31 smithi181 ceph-mon[32701]: osdmap e151: 8 total, 7 up, 8 in 2024-09-11T08:13:31.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:31 smithi181 ceph-mon[32701]: Health check update: Degraded data redundancy: 466/4467 objects degraded (10.432%), 24 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-09-11T08:13:31.625 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:31.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:31.647 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:31.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:31.748 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:31.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:31.771 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:31.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:31.873 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:31.896 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:31.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:31.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:31.999 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:32.021 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:32.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:32.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:32.123 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:32.144 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:32.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:32.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:32.246 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:32.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:32.271 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:32.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:32.373 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:32.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:32.396 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:32.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:32.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:32 smithi150 ceph-mon[29276]: pgmap v243: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 81 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.05k op/s; 466/4467 objects degraded (10.432%) 2024-09-11T08:13:32.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:32 smithi150 ceph-mon[39612]: pgmap v243: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 81 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.05k op/s; 466/4467 objects degraded (10.432%) 2024-09-11T08:13:32.497 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:32.520 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:32.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:32.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:32.544 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:13:32.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:32 smithi181 ceph-mon[32701]: pgmap v243: 148 pgs: 34 active+undersized, 24 active+undersized+degraded, 90 active+clean; 81 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.05k op/s; 466/4467 objects degraded (10.432%) 2024-09-11T08:13:32.622 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:32.645 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:32.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:32.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:32.707 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T08:13:32.747 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:32.771 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:32.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:32.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:32.873 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:32.899 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:32.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:32.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:33.000 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:33.022 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:33.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:33.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:33.124 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:33.150 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:33.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:33.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:33.252 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:33.273 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:33.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:33.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:33.376 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:33.402 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:33.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:33.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:33.505 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:33.526 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:33.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:33.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:33.627 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:33.649 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:33.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:33.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:33.751 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:33.773 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:33.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:33.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:33.875 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:33.896 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:33.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:33.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:33.997 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:34.018 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:34.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:34.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:34.120 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:34.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:34.142 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:34.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:34.243 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:34.269 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:34.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:34.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:34.371 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:34.393 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:34.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:34.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:34.436 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:34 smithi150 ceph-mon[29276]: pgmap v244: 148 pgs: 35 active+undersized, 23 active+undersized+degraded, 90 active+clean; 52 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 693 op/s; 364/3057 objects degraded (11.907%) 2024-09-11T08:13:34.436 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:34 smithi150 ceph-mon[39612]: pgmap v244: 148 pgs: 35 active+undersized, 23 active+undersized+degraded, 90 active+clean; 52 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 693 op/s; 364/3057 objects degraded (11.907%) 2024-09-11T08:13:34.495 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:34.521 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:34.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:34.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:34.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:34 smithi181 ceph-mon[32701]: pgmap v244: 148 pgs: 35 active+undersized, 23 active+undersized+degraded, 90 active+clean; 52 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 693 op/s; 364/3057 objects degraded (11.907%) 2024-09-11T08:13:34.623 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:34.648 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:34.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:34.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:34.750 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:34.774 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:34.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:34.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:34.876 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:34.899 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:34.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:34.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:35.001 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:35.025 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:35.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:35.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:35.127 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:35.154 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:35.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:35.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:35.244 INFO:teuthology.orchestra.run.smithi150.stdout:moved 0 / 444 (0%) 2024-09-11T08:13:35.244 INFO:teuthology.orchestra.run.smithi150.stdout:avg 63.4286 2024-09-11T08:13:35.245 INFO:teuthology.orchestra.run.smithi150.stdout:stddev 11.0158 -> 8.22391 (expected baseline 7.37342) 2024-09-11T08:13:35.245 INFO:teuthology.orchestra.run.smithi150.stdout:min osd.2 with 44 -> 49 pgs (0.693694 -> 0.772523 * mean) 2024-09-11T08:13:35.245 INFO:teuthology.orchestra.run.smithi150.stdout:max osd.3 with 65 -> 69 pgs (1.02477 -> 1.08784 * mean) 2024-09-11T08:13:35.245 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:13:35.245 INFO:teuthology.orchestra.run.smithi150.stdout:oload 110 2024-09-11T08:13:35.245 INFO:teuthology.orchestra.run.smithi150.stdout:max_change 3 2024-09-11T08:13:35.245 INFO:teuthology.orchestra.run.smithi150.stdout:max_change_osds 4 2024-09-11T08:13:35.245 INFO:teuthology.orchestra.run.smithi150.stdout:average_utilization 0.0053 2024-09-11T08:13:35.245 INFO:teuthology.orchestra.run.smithi150.stdout:overload_utilization 0.0059 2024-09-11T08:13:35.245 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7 weight 1.0000 -> 0.2975 2024-09-11T08:13:35.257 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:35.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:35.281 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:35.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:35.383 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:35.405 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:35.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:35.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:35.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:35 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1031011817' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:13:35.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:35 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'7':'19500'}"}]: dispatch 2024-09-11T08:13:35.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:35 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1031011817' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:13:35.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:35 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'7':'19500'}"}]: dispatch 2024-09-11T08:13:35.506 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:35.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:35.530 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:35.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:35.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:35 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1031011817' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:13:35.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:35 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'7':'19500'}"}]: dispatch 2024-09-11T08:13:35.574 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:13:35 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:13:35] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:13:35.632 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:35.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:35.655 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:35.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:35.756 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:35.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:35.779 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:35.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:35.880 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:35.902 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:35.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:35.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:36.003 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:36.025 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:36.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:36.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:36.040 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-11T08:13:36.040 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd reweight-by-utilization 1000 0.05 2024-09-11T08:13:36.127 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:36.150 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:13:36.150 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:13:36.150 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:13:36.150 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:13:36.150 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:13:36.150 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:13:36.151 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:13:36.151 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:13:36.151 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:13:36.151 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:13:36.151 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:13:36.151 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:13:36.151 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:13:36.152 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:13:36.152 INFO:teuthology.orchestra.run.smithi150.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-11T08:13:36.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:36.153 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:36.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:36.254 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:36.275 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:36.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:36.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:36.341 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:13:36.377 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:36.398 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:36.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:36.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:36.430 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:36 smithi150 ceph-mon[29276]: from='client.15018 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:13:36.431 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:36 smithi150 ceph-mon[29276]: pgmap v245: 148 pgs: 39 active+undersized, 19 active+undersized+degraded, 90 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 797 op/s; 75/588 objects degraded (12.755%) 2024-09-11T08:13:36.431 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:36 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'7':'19500'}"}]': finished 2024-09-11T08:13:36.431 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:36 smithi150 ceph-mon[29276]: osdmap e152: 8 total, 7 up, 8 in 2024-09-11T08:13:36.431 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:36 smithi150 ceph-mon[29276]: Health check update: Degraded data redundancy: 75/588 objects degraded (12.755%), 19 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-09-11T08:13:36.431 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:36 smithi150 ceph-mon[29276]: osdmap e153: 8 total, 7 up, 8 in 2024-09-11T08:13:36.432 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:36 smithi150 ceph-mon[39612]: from='client.15018 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:13:36.432 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:36 smithi150 ceph-mon[39612]: pgmap v245: 148 pgs: 39 active+undersized, 19 active+undersized+degraded, 90 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 797 op/s; 75/588 objects degraded (12.755%) 2024-09-11T08:13:36.432 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:36 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'7':'19500'}"}]': finished 2024-09-11T08:13:36.432 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:36 smithi150 ceph-mon[39612]: osdmap e152: 8 total, 7 up, 8 in 2024-09-11T08:13:36.432 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:36 smithi150 ceph-mon[39612]: Health check update: Degraded data redundancy: 75/588 objects degraded (12.755%), 19 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-09-11T08:13:36.432 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:36 smithi150 ceph-mon[39612]: osdmap e153: 8 total, 7 up, 8 in 2024-09-11T08:13:36.500 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:36.527 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:36.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:36.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:36.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:36 smithi181 ceph-mon[32701]: from='client.15018 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:13:36.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:36 smithi181 ceph-mon[32701]: pgmap v245: 148 pgs: 39 active+undersized, 19 active+undersized+degraded, 90 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 797 op/s; 75/588 objects degraded (12.755%) 2024-09-11T08:13:36.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:36 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'7':'19500'}"}]': finished 2024-09-11T08:13:36.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:36 smithi181 ceph-mon[32701]: osdmap e152: 8 total, 7 up, 8 in 2024-09-11T08:13:36.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:36 smithi181 ceph-mon[32701]: Health check update: Degraded data redundancy: 75/588 objects degraded (12.755%), 19 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-09-11T08:13:36.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:36 smithi181 ceph-mon[32701]: osdmap e153: 8 total, 7 up, 8 in 2024-09-11T08:13:36.629 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:36.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:36.653 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:36.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:36.754 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:36.776 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-11T08:13:36.776 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-11T08:13:36.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:36.777 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:36.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:36.878 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:36.900 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:36.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:36.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:37.002 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:37.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:37.024 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:37.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:37.125 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:37.147 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:37.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:37.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:37.249 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:37.271 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:37.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:37.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:37.312 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-11T08:13:37.312 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [], 2024-09-11T08:13:37.313 INFO:teuthology.orchestra.run.smithi150.stdout: "num_ops": 0 2024-09-11T08:13:37.313 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-11T08:13:37.373 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:37.395 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:37.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:37.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:37.497 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:37.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:37.520 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:37.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:37.621 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:37.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:37.645 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:37.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:37.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:37 smithi150 ceph-mon[29276]: osdmap e154: 8 total, 7 up, 8 in 2024-09-11T08:13:37.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:37 smithi150 ceph-mon[29276]: 7.6 starting backfill to osd.5 from (0'0,0'0] MAX to 149'15654 2024-09-11T08:13:37.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:37 smithi150 ceph-mon[29276]: 7.5 starting backfill to osd.1 from (0'0,0'0] MAX to 149'15626 2024-09-11T08:13:37.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:37 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:13:37.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:37 smithi150 ceph-mon[29276]: pgmap v249: 148 pgs: 39 active+undersized, 19 active+undersized+degraded, 90 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 216 op/s; 75/588 objects degraded (12.755%) 2024-09-11T08:13:37.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:37 smithi150 ceph-mon[39612]: osdmap e154: 8 total, 7 up, 8 in 2024-09-11T08:13:37.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:37 smithi150 ceph-mon[39612]: 7.6 starting backfill to osd.5 from (0'0,0'0] MAX to 149'15654 2024-09-11T08:13:37.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:37 smithi150 ceph-mon[39612]: 7.5 starting backfill to osd.1 from (0'0,0'0] MAX to 149'15626 2024-09-11T08:13:37.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:37 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:13:37.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:37 smithi150 ceph-mon[39612]: pgmap v249: 148 pgs: 39 active+undersized, 19 active+undersized+degraded, 90 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 216 op/s; 75/588 objects degraded (12.755%) 2024-09-11T08:13:37.746 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:37.769 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:37.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:37.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:37.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:37 smithi181 ceph-mon[32701]: osdmap e154: 8 total, 7 up, 8 in 2024-09-11T08:13:37.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:37 smithi181 ceph-mon[32701]: 7.6 starting backfill to osd.5 from (0'0,0'0] MAX to 149'15654 2024-09-11T08:13:37.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:37 smithi181 ceph-mon[32701]: 7.5 starting backfill to osd.1 from (0'0,0'0] MAX to 149'15626 2024-09-11T08:13:37.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:37 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:13:37.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:37 smithi181 ceph-mon[32701]: pgmap v249: 148 pgs: 39 active+undersized, 19 active+undersized+degraded, 90 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 216 op/s; 75/588 objects degraded (12.755%) 2024-09-11T08:13:37.870 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:37.893 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:37.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:37.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:37.995 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:38.016 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:38.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:38.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:38.118 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:38.143 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:38.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:38.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:38.245 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:38.270 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:38.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:38.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:38.323 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:13:38 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:13:38.373 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:38.395 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:38.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:38.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:38.497 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:38.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:38.519 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:38.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:38.620 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:38.642 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:38.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:38.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:38.739 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:13:38 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:13:38] "GET /metrics HTTP/1.1" 200 223966 "" "Prometheus/2.43.0" 2024-09-11T08:13:38.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:38 smithi150 ceph-mon[39612]: osdmap e155: 8 total, 7 up, 8 in 2024-09-11T08:13:38.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:38 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:13:38.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:38 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:13:38.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:38 smithi150 ceph-mon[29276]: osdmap e155: 8 total, 7 up, 8 in 2024-09-11T08:13:38.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:38 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:13:38.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:38 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:13:38.744 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:38.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:38.766 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:38.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:38.777 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T08:13:38.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:38 smithi181 ceph-mon[32701]: osdmap e155: 8 total, 7 up, 8 in 2024-09-11T08:13:38.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:38 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:13:38.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:38 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:13:38.867 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:38.889 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:38.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:38.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:38.991 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:39.012 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:39.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:39.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:39.116 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:39.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:39.138 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:39.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:39.239 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:39.260 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:39.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:39.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:39.362 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:39.384 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:39.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:39.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:39.486 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:39.508 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:39.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:39.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:39.513 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-11T08:13:39.609 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:39.631 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:39.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:39.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:39.733 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:39.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:39 smithi150 ceph-mon[29276]: osdmap e156: 8 total, 7 up, 8 in 2024-09-11T08:13:39.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:39 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:13:39.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:39 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:13:39.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:39 smithi150 ceph-mon[29276]: pgmap v252: 148 pgs: 1 active+undersized+remapped+backfill_toofull, 1 peering, 11 active+undersized, 5 active+undersized+degraded, 130 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 15/588 objects degraded (2.551%); 161 KiB/s, 16 objects/s recovering 2024-09-11T08:13:39.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:39 smithi150 ceph-mon[39612]: osdmap e156: 8 total, 7 up, 8 in 2024-09-11T08:13:39.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:39 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:13:39.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:39 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:13:39.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:39 smithi150 ceph-mon[39612]: pgmap v252: 148 pgs: 1 active+undersized+remapped+backfill_toofull, 1 peering, 11 active+undersized, 5 active+undersized+degraded, 130 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 15/588 objects degraded (2.551%); 161 KiB/s, 16 objects/s recovering 2024-09-11T08:13:39.754 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:39.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:39.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:39.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:39 smithi181 ceph-mon[32701]: osdmap e156: 8 total, 7 up, 8 in 2024-09-11T08:13:39.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:39 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:13:39.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:39 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:13:39.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:39 smithi181 ceph-mon[32701]: pgmap v252: 148 pgs: 1 active+undersized+remapped+backfill_toofull, 1 peering, 11 active+undersized, 5 active+undersized+degraded, 130 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 15/588 objects degraded (2.551%); 161 KiB/s, 16 objects/s recovering 2024-09-11T08:13:39.856 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:39.878 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:39.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:39.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:39.937 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:13:39.980 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:40.007 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:40.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:40.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:40.109 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:40.131 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:40.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:40.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:40.233 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:40.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:40.257 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:40.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:40.358 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:40.381 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:40.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:40.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:40.483 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:40.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:40.506 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:40.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:40.586 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:40 smithi150 ceph-mon[29276]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-11T08:13:40.586 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:40 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:13:40.586 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:40 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:13:40.586 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:40 smithi150 ceph-mon[29276]: Health check update: Degraded data redundancy: 15/588 objects degraded (2.551%), 5 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-09-11T08:13:40.586 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:40 smithi150 ceph-mon[39612]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-11T08:13:40.586 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:40 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:13:40.586 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:40 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:13:40.587 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:40 smithi150 ceph-mon[39612]: Health check update: Degraded data redundancy: 15/588 objects degraded (2.551%), 5 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-09-11T08:13:40.607 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:40.628 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:40.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:40.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:40.729 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:40.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:40.753 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:40.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:40.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:40 smithi181 ceph-mon[32701]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-11T08:13:40.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:40 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:13:40.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:40 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:13:40.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:40 smithi181 ceph-mon[32701]: Health check update: Degraded data redundancy: 15/588 objects degraded (2.551%), 5 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-09-11T08:13:40.854 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:40.876 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:40.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:40.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:40.978 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:41.000 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:41.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:41.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:41.006 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:13:41.007 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:13:41.007 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:13:41.102 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:41.123 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:41.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:41.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:41.225 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:41.246 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:41.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:41.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:41.348 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:41.370 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:41.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:41.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:41.472 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:41.494 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:41.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:41.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:41.596 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:41.617 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:41.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:41.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:41.719 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:41.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:41 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:13:41.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:41 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:13:41.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:41 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:13:41.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:41 smithi150 ceph-mon[29276]: pgmap v253: 148 pgs: 1 active+undersized+remapped+backfill_toofull, 1 peering, 11 active+undersized, 5 active+undersized+degraded, 130 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 953 B/s rd, 0 op/s; 15/588 objects degraded (2.551%); 108 KiB/s, 11 objects/s recovering 2024-09-11T08:13:41.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:41 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:13:41.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:41 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:13:41.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:41 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:13:41.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:41 smithi150 ceph-mon[39612]: pgmap v253: 148 pgs: 1 active+undersized+remapped+backfill_toofull, 1 peering, 11 active+undersized, 5 active+undersized+degraded, 130 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 953 B/s rd, 0 op/s; 15/588 objects degraded (2.551%); 108 KiB/s, 11 objects/s recovering 2024-09-11T08:13:41.741 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:41.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:41.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:41.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:41 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:13:41.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:41 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:13:41.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:41 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:13:41.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:41 smithi181 ceph-mon[32701]: pgmap v253: 148 pgs: 1 active+undersized+remapped+backfill_toofull, 1 peering, 11 active+undersized, 5 active+undersized+degraded, 130 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 953 B/s rd, 0 op/s; 15/588 objects degraded (2.551%); 108 KiB/s, 11 objects/s recovering 2024-09-11T08:13:41.842 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:41.868 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:41.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:41.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:41.970 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:41.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:41.992 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:41.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:42.093 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:42.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:42.115 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:42.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:42.217 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:42.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:42.243 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:42.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:42.344 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:42.366 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:42.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:42.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:42.468 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:42.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:42.491 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:42.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:42.592 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:42.618 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:42.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:42.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:42.720 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:42.743 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:42.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:42.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:42.752 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'unique_pool_0' removed 2024-09-11T08:13:42.846 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:42.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:42.870 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:42.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:42.971 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:42.995 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:42.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:42.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:43.049 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:42 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:13:43.049 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:42 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:13:43.050 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:42 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3202603863' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-11T08:13:43.050 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:42 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:13:43.050 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:42 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:13:43.050 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:42 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3202603863' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-11T08:13:43.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:42 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:13:43.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:42 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:13:43.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:42 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3202603863' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-11T08:13:43.097 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:43.121 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:43.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:43.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:43.223 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:43.248 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:43.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:43.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:43.350 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:43.374 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:43.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:43.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:43.476 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:43.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:43.501 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:43.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:43.602 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:43.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:43.629 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:43.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:43.730 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:43.752 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:43.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:43.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:43.854 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:43.875 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:43.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:43.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:43.906 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:43 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3202603863' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-11T08:13:43.906 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:43 smithi150 ceph-mon[29276]: osdmap e157: 8 total, 7 up, 8 in 2024-09-11T08:13:43.906 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:43 smithi150 ceph-mon[29276]: pgmap v255: 132 pgs: 8 active+undersized, 5 active+undersized+degraded, 119 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 15/588 objects degraded (2.551%); 96 KiB/s, 9 objects/s recovering 2024-09-11T08:13:43.907 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:43 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3202603863' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-11T08:13:43.907 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:43 smithi150 ceph-mon[39612]: osdmap e157: 8 total, 7 up, 8 in 2024-09-11T08:13:43.907 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:43 smithi150 ceph-mon[39612]: pgmap v255: 132 pgs: 8 active+undersized, 5 active+undersized+degraded, 119 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 15/588 objects degraded (2.551%); 96 KiB/s, 9 objects/s recovering 2024-09-11T08:13:43.977 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:44.000 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:44.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:44.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:44.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:43 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3202603863' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-11T08:13:44.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:43 smithi181 ceph-mon[32701]: osdmap e157: 8 total, 7 up, 8 in 2024-09-11T08:13:44.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:43 smithi181 ceph-mon[32701]: pgmap v255: 132 pgs: 8 active+undersized, 5 active+undersized+degraded, 119 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 15/588 objects degraded (2.551%); 96 KiB/s, 9 objects/s recovering 2024-09-11T08:13:44.102 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:44.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:44.124 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:44.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:44.225 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:44.246 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:44.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:44.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:44.348 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:44.370 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:44.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:44.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:44.471 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:44.493 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:44.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:44.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:44.595 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:44.616 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:44.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:44.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:44.718 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:44.740 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:44.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:44.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:44.842 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:44.863 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:44.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:44.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:44.965 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:44.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:44.987 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:44.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:44.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:44 smithi150 ceph-mon[29276]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-11T08:13:44.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:44 smithi150 ceph-mon[39612]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-11T08:13:45.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:44 smithi181 ceph-mon[32701]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-11T08:13:45.088 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:45.111 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:45.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:45.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:45.212 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:45.234 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:45.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:45.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:45.336 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:45.357 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:45.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:45.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:45.459 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:45.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:45.482 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:45.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:45.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:13:45 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:13:45] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:13:45.583 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:45.604 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:45.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:45.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:45.625 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-11T08:13:45.626 INFO:tasks.radosbench:Beginning radosbench... 2024-09-11T08:13:45.626 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-09-11T08:13:45.627 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd pool create unique_pool_1 16 2024-09-11T08:13:45.706 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:45.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:45.732 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:45.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:45.833 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:45.855 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:45.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:45.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:45.957 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:45.978 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:45.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:45.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:45.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:45 smithi150 ceph-mon[29276]: pgmap v256: 132 pgs: 8 active+undersized, 5 active+undersized+degraded, 119 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 15/588 objects degraded (2.551%); 78 KiB/s, 7 objects/s recovering 2024-09-11T08:13:45.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:45 smithi150 ceph-mon[29276]: from='client.15030 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:13:45.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:45 smithi150 ceph-mon[29276]: Health check update: Degraded data redundancy: 15/588 objects degraded (2.551%), 5 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-09-11T08:13:45.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:45 smithi150 ceph-mon[39612]: pgmap v256: 132 pgs: 8 active+undersized, 5 active+undersized+degraded, 119 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 15/588 objects degraded (2.551%); 78 KiB/s, 7 objects/s recovering 2024-09-11T08:13:45.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:45 smithi150 ceph-mon[39612]: from='client.15030 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:13:45.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:45 smithi150 ceph-mon[39612]: Health check update: Degraded data redundancy: 15/588 objects degraded (2.551%), 5 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-09-11T08:13:46.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:45 smithi181 ceph-mon[32701]: pgmap v256: 132 pgs: 8 active+undersized, 5 active+undersized+degraded, 119 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 15/588 objects degraded (2.551%); 78 KiB/s, 7 objects/s recovering 2024-09-11T08:13:46.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:45 smithi181 ceph-mon[32701]: from='client.15030 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:13:46.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:45 smithi181 ceph-mon[32701]: Health check update: Degraded data redundancy: 15/588 objects degraded (2.551%), 5 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-09-11T08:13:46.080 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:46.104 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:46.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:46.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:46.205 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:46.226 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:46.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:46.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:46.328 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:46.349 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:46.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:46.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:46.451 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:46.477 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:46.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:46.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:46.549 INFO:teuthology.orchestra.run.smithi150.stdout:moved 0 / 396 (0%) 2024-09-11T08:13:46.549 INFO:teuthology.orchestra.run.smithi150.stdout:avg 56.5714 2024-09-11T08:13:46.549 INFO:teuthology.orchestra.run.smithi150.stdout:stddev 7.44092 -> 7.44092 (expected baseline 6.96346) 2024-09-11T08:13:46.549 INFO:teuthology.orchestra.run.smithi150.stdout:min osd.2 with 43 -> 43 pgs (0.760101 -> 0.760101 * mean) 2024-09-11T08:13:46.549 INFO:teuthology.orchestra.run.smithi150.stdout:max osd.1 with 64 -> 64 pgs (1.13131 -> 1.13131 * mean) 2024-09-11T08:13:46.549 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:13:46.549 INFO:teuthology.orchestra.run.smithi150.stdout:oload 1000 2024-09-11T08:13:46.550 INFO:teuthology.orchestra.run.smithi150.stdout:max_change 0.05 2024-09-11T08:13:46.550 INFO:teuthology.orchestra.run.smithi150.stdout:max_change_osds 4 2024-09-11T08:13:46.550 INFO:teuthology.orchestra.run.smithi150.stdout:average_utilization 0.0051 2024-09-11T08:13:46.550 INFO:teuthology.orchestra.run.smithi150.stdout:overload_utilization 0.0507 2024-09-11T08:13:46.550 INFO:teuthology.orchestra.run.smithi150.stderr:no change 2024-09-11T08:13:46.578 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:46.601 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:46.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:46.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:46.702 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:46.723 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:46.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:46.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:46.773 INFO:teuthology.orchestra.run.smithi150.stderr:noscrub is unset 2024-09-11T08:13:46.826 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:46.847 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:46.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:46.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:46.949 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:46.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:46.971 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:46.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:47.072 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:47.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:46 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1924003864' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:13:47.099 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:47.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:47.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:47.145 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:46 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1924003864' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:13:47.146 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:46 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1924003864' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:13:47.169 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-11T08:13:47.169 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd reweight-by-utilization 1000 1.0 2024-09-11T08:13:47.200 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:47.225 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:47.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:47.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:47.327 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:47.349 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:47.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:47.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:47.451 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:47.477 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:47.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:47.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:47.579 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:47.605 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:47.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:47.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:47.706 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:47.728 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:47.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:47.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:47.830 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:47.856 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:47.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:47.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:47.958 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:47.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:47.981 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:47.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:47.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:47 smithi150 ceph-mon[29276]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:13:47.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:47 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1924003864' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:13:47.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:47 smithi150 ceph-mon[29276]: osdmap e158: 8 total, 7 up, 8 in 2024-09-11T08:13:47.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:47 smithi150 ceph-mon[29276]: pgmap v258: 132 pgs: 8 active+undersized, 5 active+undersized+degraded, 119 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 15/588 objects degraded (2.551%) 2024-09-11T08:13:47.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:47 smithi150 ceph-mon[29276]: 3.7 scrub starts 2024-09-11T08:13:47.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:47 smithi150 ceph-mon[29276]: 3.7 scrub ok 2024-09-11T08:13:47.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:47 smithi150 ceph-mon[29276]: 5.19 scrub starts 2024-09-11T08:13:47.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:47 smithi150 ceph-mon[29276]: 5.19 scrub ok 2024-09-11T08:13:47.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:47 smithi150 ceph-mon[39612]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:13:47.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:47 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1924003864' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:13:47.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:47 smithi150 ceph-mon[39612]: osdmap e158: 8 total, 7 up, 8 in 2024-09-11T08:13:47.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:47 smithi150 ceph-mon[39612]: pgmap v258: 132 pgs: 8 active+undersized, 5 active+undersized+degraded, 119 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 15/588 objects degraded (2.551%) 2024-09-11T08:13:47.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:47 smithi150 ceph-mon[39612]: 3.7 scrub starts 2024-09-11T08:13:47.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:47 smithi150 ceph-mon[39612]: 3.7 scrub ok 2024-09-11T08:13:47.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:47 smithi150 ceph-mon[39612]: 5.19 scrub starts 2024-09-11T08:13:47.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:47 smithi150 ceph-mon[39612]: 5.19 scrub ok 2024-09-11T08:13:48.014 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:47 smithi181 ceph-mon[32701]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:13:48.014 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:47 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1924003864' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:13:48.014 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:47 smithi181 ceph-mon[32701]: osdmap e158: 8 total, 7 up, 8 in 2024-09-11T08:13:48.014 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:47 smithi181 ceph-mon[32701]: pgmap v258: 132 pgs: 8 active+undersized, 5 active+undersized+degraded, 119 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 15/588 objects degraded (2.551%) 2024-09-11T08:13:48.014 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:47 smithi181 ceph-mon[32701]: 3.7 scrub starts 2024-09-11T08:13:48.014 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:47 smithi181 ceph-mon[32701]: 3.7 scrub ok 2024-09-11T08:13:48.014 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:47 smithi181 ceph-mon[32701]: 5.19 scrub starts 2024-09-11T08:13:48.014 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:47 smithi181 ceph-mon[32701]: 5.19 scrub ok 2024-09-11T08:13:48.082 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:48.103 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:48.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:48.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:48.205 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:48.231 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:48.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:48.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:48.323 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:13:48 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:13:48.333 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:48.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:48.355 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:48.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:48.456 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:48.482 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:48.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:48.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:48.584 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:48.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:48.606 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:48.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:48.708 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:48.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:48.730 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:48.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:48.831 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:48.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:48.854 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:48.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:48.876 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:13:48 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:13:48] "GET /metrics HTTP/1.1" 200 223966 "" "Prometheus/2.43.0" 2024-09-11T08:13:48.876 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:48 smithi150 ceph-mon[29276]: 5.9 scrub starts 2024-09-11T08:13:48.876 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:48 smithi150 ceph-mon[29276]: 5.9 scrub ok 2024-09-11T08:13:48.876 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:48 smithi150 ceph-mon[29276]: 6.12 scrub starts 2024-09-11T08:13:48.876 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:48 smithi150 ceph-mon[29276]: 6.12 scrub ok 2024-09-11T08:13:48.876 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:48 smithi150 ceph-mon[29276]: 5.1d scrub starts 2024-09-11T08:13:48.876 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:48 smithi150 ceph-mon[29276]: 5.1d scrub ok 2024-09-11T08:13:48.877 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:48 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:13:48.877 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:48 smithi150 ceph-mon[39612]: 5.9 scrub starts 2024-09-11T08:13:48.877 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:48 smithi150 ceph-mon[39612]: 5.9 scrub ok 2024-09-11T08:13:48.877 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:48 smithi150 ceph-mon[39612]: 6.12 scrub starts 2024-09-11T08:13:48.877 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:48 smithi150 ceph-mon[39612]: 6.12 scrub ok 2024-09-11T08:13:48.877 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:48 smithi150 ceph-mon[39612]: 5.1d scrub starts 2024-09-11T08:13:48.877 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:48 smithi150 ceph-mon[39612]: 5.1d scrub ok 2024-09-11T08:13:48.877 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:48 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:13:48.901 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:13:48.955 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:48.982 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:48.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:48.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:49.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:48 smithi181 ceph-mon[32701]: 5.9 scrub starts 2024-09-11T08:13:49.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:48 smithi181 ceph-mon[32701]: 5.9 scrub ok 2024-09-11T08:13:49.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:48 smithi181 ceph-mon[32701]: 6.12 scrub starts 2024-09-11T08:13:49.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:48 smithi181 ceph-mon[32701]: 6.12 scrub ok 2024-09-11T08:13:49.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:48 smithi181 ceph-mon[32701]: 5.1d scrub starts 2024-09-11T08:13:49.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:48 smithi181 ceph-mon[32701]: 5.1d scrub ok 2024-09-11T08:13:49.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:48 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:13:49.084 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:49.106 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:49.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:49.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:49.208 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:49.230 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:49.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:49.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:49.332 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:49.354 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:49.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:49.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:49.456 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:49.482 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:49.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:49.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:49.584 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:49.609 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:49.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:49.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:49.644 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-11T08:13:49.644 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [], 2024-09-11T08:13:49.645 INFO:teuthology.orchestra.run.smithi150.stdout: "complaint_time": 30, 2024-09-11T08:13:49.645 INFO:teuthology.orchestra.run.smithi150.stdout: "num_blocked_ops": 0 2024-09-11T08:13:49.645 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-11T08:13:49.711 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:49.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:49.733 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:49.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:49.834 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:49.856 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:49.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:49.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:49.955 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:49 smithi150 ceph-mon[29276]: 3.12 scrub starts 2024-09-11T08:13:49.955 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:49 smithi150 ceph-mon[29276]: 3.12 scrub ok 2024-09-11T08:13:49.955 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:49 smithi150 ceph-mon[29276]: pgmap v259: 132 pgs: 8 active+undersized, 5 active+undersized+degraded, 119 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 15/588 objects degraded (2.551%) 2024-09-11T08:13:49.956 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:49 smithi150 ceph-mon[39612]: 3.12 scrub starts 2024-09-11T08:13:49.956 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:49 smithi150 ceph-mon[39612]: 3.12 scrub ok 2024-09-11T08:13:49.956 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:49 smithi150 ceph-mon[39612]: pgmap v259: 132 pgs: 8 active+undersized, 5 active+undersized+degraded, 119 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 15/588 objects degraded (2.551%) 2024-09-11T08:13:49.958 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:49.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:49.981 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:49.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:50.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:49 smithi181 ceph-mon[32701]: 3.12 scrub starts 2024-09-11T08:13:50.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:49 smithi181 ceph-mon[32701]: 3.12 scrub ok 2024-09-11T08:13:50.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:49 smithi181 ceph-mon[32701]: pgmap v259: 132 pgs: 8 active+undersized, 5 active+undersized+degraded, 119 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 15/588 objects degraded (2.551%) 2024-09-11T08:13:50.081 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:50.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:50.110 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:50.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:50.212 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:50.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:50.238 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:50.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:50.339 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:50.361 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:50.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:50.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:50.463 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:50.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:50.486 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:50.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:50.559 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-11T08:13:50.589 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:50.615 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:50.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:50.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:50.653 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T08:13:50.716 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:50.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:50.743 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:50.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:50.836 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:50 smithi150 ceph-mon[29276]: 5.1f scrub starts 2024-09-11T08:13:50.836 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:50 smithi150 ceph-mon[29276]: 5.1f scrub ok 2024-09-11T08:13:50.836 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:50 smithi150 ceph-mon[29276]: 3.16 scrub starts 2024-09-11T08:13:50.837 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:50 smithi150 ceph-mon[29276]: 3.16 scrub ok 2024-09-11T08:13:50.837 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:50 smithi150 ceph-mon[29276]: 6.14 scrub starts 2024-09-11T08:13:50.837 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:50 smithi150 ceph-mon[29276]: 6.14 scrub ok 2024-09-11T08:13:50.837 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:50 smithi150 ceph-mon[39612]: 5.1f scrub starts 2024-09-11T08:13:50.837 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:50 smithi150 ceph-mon[39612]: 5.1f scrub ok 2024-09-11T08:13:50.837 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:50 smithi150 ceph-mon[39612]: 3.16 scrub starts 2024-09-11T08:13:50.837 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:50 smithi150 ceph-mon[39612]: 3.16 scrub ok 2024-09-11T08:13:50.837 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:50 smithi150 ceph-mon[39612]: 6.14 scrub starts 2024-09-11T08:13:50.837 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:50 smithi150 ceph-mon[39612]: 6.14 scrub ok 2024-09-11T08:13:50.844 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:50.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:50.866 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:50.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:50.967 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:50.989 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:50.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:50.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:51.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:50 smithi181 ceph-mon[32701]: 5.1f scrub starts 2024-09-11T08:13:51.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:50 smithi181 ceph-mon[32701]: 5.1f scrub ok 2024-09-11T08:13:51.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:50 smithi181 ceph-mon[32701]: 3.16 scrub starts 2024-09-11T08:13:51.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:50 smithi181 ceph-mon[32701]: 3.16 scrub ok 2024-09-11T08:13:51.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:50 smithi181 ceph-mon[32701]: 6.14 scrub starts 2024-09-11T08:13:51.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:50 smithi181 ceph-mon[32701]: 6.14 scrub ok 2024-09-11T08:13:51.091 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:51.113 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:51.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:51.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:51.215 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:51.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:51.238 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:51.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:51.339 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:51.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:51.365 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:51.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:51.467 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:51.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:51.490 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:51.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:51.591 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:51.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:51.620 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:51.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:51.721 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:51.742 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:51.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:51.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:51.844 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:51.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:51.871 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:51.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:51.896 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:51 smithi150 ceph-mon[29276]: 4.a scrub starts 2024-09-11T08:13:51.896 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:51 smithi150 ceph-mon[29276]: 4.a scrub ok 2024-09-11T08:13:51.896 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:51 smithi150 ceph-mon[29276]: 5.4 scrub starts 2024-09-11T08:13:51.896 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:51 smithi150 ceph-mon[29276]: 5.4 scrub ok 2024-09-11T08:13:51.897 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:51 smithi150 ceph-mon[29276]: pgmap v260: 132 pgs: 8 active+undersized, 5 active+undersized+degraded, 119 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 15/588 objects degraded (2.551%) 2024-09-11T08:13:51.897 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:51 smithi150 ceph-mon[29276]: 4.f scrub starts 2024-09-11T08:13:51.897 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:51 smithi150 ceph-mon[29276]: 4.f scrub ok 2024-09-11T08:13:51.897 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:51 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:13:51.897 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:51 smithi150 ceph-mon[39612]: 4.a scrub starts 2024-09-11T08:13:51.897 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:51 smithi150 ceph-mon[39612]: 4.a scrub ok 2024-09-11T08:13:51.897 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:51 smithi150 ceph-mon[39612]: 5.4 scrub starts 2024-09-11T08:13:51.897 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:51 smithi150 ceph-mon[39612]: 5.4 scrub ok 2024-09-11T08:13:51.897 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:51 smithi150 ceph-mon[39612]: pgmap v260: 132 pgs: 8 active+undersized, 5 active+undersized+degraded, 119 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 15/588 objects degraded (2.551%) 2024-09-11T08:13:51.897 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:51 smithi150 ceph-mon[39612]: 4.f scrub starts 2024-09-11T08:13:51.897 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:51 smithi150 ceph-mon[39612]: 4.f scrub ok 2024-09-11T08:13:51.898 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:51 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:13:51.902 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:13:51.902 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:13:51.908 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:13:51.913 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:13:51.913 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:13:51.932 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:13:51.932 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:13:51.952 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:13:51.953 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:13:51.969 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:13:51.969 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:13:51.972 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:51.985 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:13:51.985 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:13:51.993 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:51.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:51.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:52.001 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:13:52.001 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:13:52.001 INFO:teuthology.orchestra.run.smithi150.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-11T08:13:52.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:51 smithi181 ceph-mon[32701]: 4.a scrub starts 2024-09-11T08:13:52.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:51 smithi181 ceph-mon[32701]: 4.a scrub ok 2024-09-11T08:13:52.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:51 smithi181 ceph-mon[32701]: 5.4 scrub starts 2024-09-11T08:13:52.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:51 smithi181 ceph-mon[32701]: 5.4 scrub ok 2024-09-11T08:13:52.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:51 smithi181 ceph-mon[32701]: pgmap v260: 132 pgs: 8 active+undersized, 5 active+undersized+degraded, 119 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 15/588 objects degraded (2.551%) 2024-09-11T08:13:52.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:51 smithi181 ceph-mon[32701]: 4.f scrub starts 2024-09-11T08:13:52.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:51 smithi181 ceph-mon[32701]: 4.f scrub ok 2024-09-11T08:13:52.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:51 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:13:52.095 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:52.117 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:52.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:52.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:52.219 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:52.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:52.241 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:52.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:52.342 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:52.367 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:52.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:52.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:52.469 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:52.495 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:52.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:52.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:52.597 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:52.623 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:52.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:52.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:52.725 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:52.746 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:52.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:52.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:52.848 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:52.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:52.870 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:52.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:52.941 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-11T08:13:52.942 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-11T08:13:52.972 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:52.980 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:13:52.994 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:52.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:52.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:53.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:52 smithi181 ceph-mon[32701]: 3.8 scrub starts 2024-09-11T08:13:53.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:52 smithi181 ceph-mon[32701]: 3.8 scrub ok 2024-09-11T08:13:53.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:52 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3246932073' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:13:53.077 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:52 smithi150 ceph-mon[29276]: 3.8 scrub starts 2024-09-11T08:13:53.078 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:52 smithi150 ceph-mon[29276]: 3.8 scrub ok 2024-09-11T08:13:53.078 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:52 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3246932073' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:13:53.078 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:52 smithi150 ceph-mon[39612]: 3.8 scrub starts 2024-09-11T08:13:53.078 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:52 smithi150 ceph-mon[39612]: 3.8 scrub ok 2024-09-11T08:13:53.078 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:52 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3246932073' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:13:53.095 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:53.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:53.121 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:53.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:53.222 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:53.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:53.245 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:53.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:53.346 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:53.367 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:53.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:53.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:53.469 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:53.495 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:53.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:53.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:53.597 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:53.619 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:53.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:53.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:53.721 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:53.743 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:53.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:53.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:53.845 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:53.874 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:53.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:53.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:53.878 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'unique_pool_1' created 2024-09-11T08:13:53.976 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:53.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:53 smithi150 ceph-mon[29276]: 5.11 scrub starts 2024-09-11T08:13:53.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:53 smithi150 ceph-mon[29276]: 5.11 scrub ok 2024-09-11T08:13:53.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:53 smithi150 ceph-mon[29276]: 4.8 scrub starts 2024-09-11T08:13:53.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:53 smithi150 ceph-mon[29276]: 4.8 scrub ok 2024-09-11T08:13:53.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:53 smithi150 ceph-mon[29276]: pgmap v261: 132 pgs: 8 active+undersized, 5 active+undersized+degraded, 119 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 15/588 objects degraded (2.551%) 2024-09-11T08:13:53.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:53 smithi150 ceph-mon[29276]: 4.7 scrub starts 2024-09-11T08:13:53.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:53 smithi150 ceph-mon[29276]: 4.7 scrub ok 2024-09-11T08:13:53.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:53 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3783975317' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-09-11T08:13:53.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:53 smithi150 ceph-mon[39612]: 5.11 scrub starts 2024-09-11T08:13:53.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:53 smithi150 ceph-mon[39612]: 5.11 scrub ok 2024-09-11T08:13:53.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:53 smithi150 ceph-mon[39612]: 4.8 scrub starts 2024-09-11T08:13:53.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:53 smithi150 ceph-mon[39612]: 4.8 scrub ok 2024-09-11T08:13:53.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:53 smithi150 ceph-mon[39612]: pgmap v261: 132 pgs: 8 active+undersized, 5 active+undersized+degraded, 119 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 15/588 objects degraded (2.551%) 2024-09-11T08:13:53.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:53 smithi150 ceph-mon[39612]: 4.7 scrub starts 2024-09-11T08:13:53.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:53 smithi150 ceph-mon[39612]: 4.7 scrub ok 2024-09-11T08:13:53.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:53 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3783975317' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-09-11T08:13:54.001 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:54.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:54.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:54.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:53 smithi181 ceph-mon[32701]: 5.11 scrub starts 2024-09-11T08:13:54.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:53 smithi181 ceph-mon[32701]: 5.11 scrub ok 2024-09-11T08:13:54.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:53 smithi181 ceph-mon[32701]: 4.8 scrub starts 2024-09-11T08:13:54.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:53 smithi181 ceph-mon[32701]: 4.8 scrub ok 2024-09-11T08:13:54.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:53 smithi181 ceph-mon[32701]: pgmap v261: 132 pgs: 8 active+undersized, 5 active+undersized+degraded, 119 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 15/588 objects degraded (2.551%) 2024-09-11T08:13:54.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:53 smithi181 ceph-mon[32701]: 4.7 scrub starts 2024-09-11T08:13:54.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:53 smithi181 ceph-mon[32701]: 4.7 scrub ok 2024-09-11T08:13:54.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:53 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3783975317' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-09-11T08:13:54.103 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:54.126 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:54.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:54.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:54.228 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:54.249 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:54.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:54.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:54.351 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:54.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:54.374 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:54.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:54.475 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:54.502 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:54.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:54.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:54.603 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:54.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:54.626 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:54.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:54.727 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:54.748 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:54.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:54.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:54.850 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:54.873 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:54.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:54.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:54.943 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T08:13:54.958 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-09-11T08:13:54.975 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:55.001 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:55.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:55.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:55.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:54 smithi181 ceph-mon[32701]: 5.1b scrub starts 2024-09-11T08:13:55.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:54 smithi181 ceph-mon[32701]: 5.1b scrub ok 2024-09-11T08:13:55.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:54 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-09-11T08:13:55.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:54 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-11T08:13:55.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:54 smithi181 ceph-mon[32701]: osdmap e159: 8 total, 7 up, 8 in 2024-09-11T08:13:55.103 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:55.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:55.130 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:55.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:55.231 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:55.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:54 smithi150 ceph-mon[29276]: 5.1b scrub starts 2024-09-11T08:13:55.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:54 smithi150 ceph-mon[29276]: 5.1b scrub ok 2024-09-11T08:13:55.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:54 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-09-11T08:13:55.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:54 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-11T08:13:55.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:54 smithi150 ceph-mon[29276]: osdmap e159: 8 total, 7 up, 8 in 2024-09-11T08:13:55.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:54 smithi150 ceph-mon[39612]: 5.1b scrub starts 2024-09-11T08:13:55.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:54 smithi150 ceph-mon[39612]: 5.1b scrub ok 2024-09-11T08:13:55.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:54 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-09-11T08:13:55.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:54 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-11T08:13:55.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:54 smithi150 ceph-mon[39612]: osdmap e159: 8 total, 7 up, 8 in 2024-09-11T08:13:55.253 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:55.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:55.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:55.292 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:13:55.354 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:55.377 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:55.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:55.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:55.478 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:55.505 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:55.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:55.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:55.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:13:55 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:13:55] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:13:55.607 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:55.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:55.630 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:55.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:55.731 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:55.915 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:55.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:55.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:56.017 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:56.041 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:56.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:56.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:56.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:55 smithi181 ceph-mon[32701]: 3.10 scrub starts 2024-09-11T08:13:56.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:55 smithi181 ceph-mon[32701]: 3.10 scrub ok 2024-09-11T08:13:56.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:55 smithi181 ceph-mon[32701]: pgmap v263: 148 pgs: 16 unknown, 8 active+undersized, 5 active+undersized+degraded, 119 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 15/588 objects degraded (2.551%) 2024-09-11T08:13:56.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:55 smithi181 ceph-mon[32701]: osdmap e160: 8 total, 7 up, 8 in 2024-09-11T08:13:56.088 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:55 smithi150 ceph-mon[29276]: 3.10 scrub starts 2024-09-11T08:13:56.089 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:55 smithi150 ceph-mon[29276]: 3.10 scrub ok 2024-09-11T08:13:56.089 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:55 smithi150 ceph-mon[29276]: pgmap v263: 148 pgs: 16 unknown, 8 active+undersized, 5 active+undersized+degraded, 119 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 15/588 objects degraded (2.551%) 2024-09-11T08:13:56.089 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:55 smithi150 ceph-mon[29276]: osdmap e160: 8 total, 7 up, 8 in 2024-09-11T08:13:56.089 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:55 smithi150 ceph-mon[39612]: 3.10 scrub starts 2024-09-11T08:13:56.089 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:55 smithi150 ceph-mon[39612]: 3.10 scrub ok 2024-09-11T08:13:56.089 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:55 smithi150 ceph-mon[39612]: pgmap v263: 148 pgs: 16 unknown, 8 active+undersized, 5 active+undersized+degraded, 119 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 15/588 objects degraded (2.551%) 2024-09-11T08:13:56.089 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:55 smithi150 ceph-mon[39612]: osdmap e160: 8 total, 7 up, 8 in 2024-09-11T08:13:56.144 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:56.168 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:56.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:56.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:56.270 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:56.293 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:56.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:56.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:56.394 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:56.416 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:56.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:56.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:56.518 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:56.544 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:56.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:56.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:56.646 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:56.669 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:56.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:56.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:56.771 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:56.797 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:56.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:56.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:56.899 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:56.920 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:56.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:56.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:56.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:56 smithi150 ceph-mon[29276]: 6.b scrub starts 2024-09-11T08:13:56.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:56 smithi150 ceph-mon[29276]: 6.b scrub ok 2024-09-11T08:13:56.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:56 smithi150 ceph-mon[29276]: 4.15 scrub starts 2024-09-11T08:13:56.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:56 smithi150 ceph-mon[29276]: 4.15 scrub ok 2024-09-11T08:13:56.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:56 smithi150 ceph-mon[29276]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-11T08:13:56.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:56 smithi150 ceph-mon[29276]: 3.14 scrub starts 2024-09-11T08:13:56.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:56 smithi150 ceph-mon[29276]: 3.14 scrub ok 2024-09-11T08:13:56.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:56 smithi150 ceph-mon[29276]: osdmap e161: 8 total, 7 up, 8 in 2024-09-11T08:13:56.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:56 smithi150 ceph-mon[39612]: 6.b scrub starts 2024-09-11T08:13:56.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:56 smithi150 ceph-mon[39612]: 6.b scrub ok 2024-09-11T08:13:56.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:56 smithi150 ceph-mon[39612]: 4.15 scrub starts 2024-09-11T08:13:56.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:56 smithi150 ceph-mon[39612]: 4.15 scrub ok 2024-09-11T08:13:56.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:56 smithi150 ceph-mon[39612]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-11T08:13:56.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:56 smithi150 ceph-mon[39612]: 3.14 scrub starts 2024-09-11T08:13:56.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:56 smithi150 ceph-mon[39612]: 3.14 scrub ok 2024-09-11T08:13:56.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:56 smithi150 ceph-mon[39612]: osdmap e161: 8 total, 7 up, 8 in 2024-09-11T08:13:57.022 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:57.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:57.044 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:57.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:57.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:56 smithi181 ceph-mon[32701]: 6.b scrub starts 2024-09-11T08:13:57.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:56 smithi181 ceph-mon[32701]: 6.b scrub ok 2024-09-11T08:13:57.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:56 smithi181 ceph-mon[32701]: 4.15 scrub starts 2024-09-11T08:13:57.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:56 smithi181 ceph-mon[32701]: 4.15 scrub ok 2024-09-11T08:13:57.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:56 smithi181 ceph-mon[32701]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-11T08:13:57.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:56 smithi181 ceph-mon[32701]: 3.14 scrub starts 2024-09-11T08:13:57.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:56 smithi181 ceph-mon[32701]: 3.14 scrub ok 2024-09-11T08:13:57.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:56 smithi181 ceph-mon[32701]: osdmap e161: 8 total, 7 up, 8 in 2024-09-11T08:13:57.145 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:57.171 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:57.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:57.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:57.273 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:57.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:57.295 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:57.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:57.397 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:57.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:57.419 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:57.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:57.521 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:57.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:57.544 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:57.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:57.645 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:57.665 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:57.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:57.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:57.768 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:57.794 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:57.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:57.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:57.896 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:57.921 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:57.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:57.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:57.954 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-09-11T08:13:57.954 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:13:58.023 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:13:58.026 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:57 smithi150 ceph-mon[29276]: pgmap v266: 148 pgs: 16 unknown, 8 active+undersized, 5 active+undersized+degraded, 119 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 15/588 objects degraded (2.551%) 2024-09-11T08:13:58.026 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:57 smithi150 ceph-mon[29276]: 6.8 scrub starts 2024-09-11T08:13:58.026 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:57 smithi150 ceph-mon[29276]: 6.8 scrub ok 2024-09-11T08:13:58.026 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:57 smithi150 ceph-mon[29276]: 3.4 scrub starts 2024-09-11T08:13:58.027 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:57 smithi150 ceph-mon[29276]: 3.4 scrub ok 2024-09-11T08:13:58.027 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:57 smithi150 ceph-mon[39612]: pgmap v266: 148 pgs: 16 unknown, 8 active+undersized, 5 active+undersized+degraded, 119 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 15/588 objects degraded (2.551%) 2024-09-11T08:13:58.027 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:57 smithi150 ceph-mon[39612]: 6.8 scrub starts 2024-09-11T08:13:58.027 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:57 smithi150 ceph-mon[39612]: 6.8 scrub ok 2024-09-11T08:13:58.027 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:57 smithi150 ceph-mon[39612]: 3.4 scrub starts 2024-09-11T08:13:58.027 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:57 smithi150 ceph-mon[39612]: 3.4 scrub ok 2024-09-11T08:13:58.046 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:58.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:58.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:13:58.148 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:58.173 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:58.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:58.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:58.275 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:58.298 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:58.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:58.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:58.323 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:13:58 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:13:58.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:57 smithi181 ceph-mon[32701]: pgmap v266: 148 pgs: 16 unknown, 8 active+undersized, 5 active+undersized+degraded, 119 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 15/588 objects degraded (2.551%) 2024-09-11T08:13:58.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:57 smithi181 ceph-mon[32701]: 6.8 scrub starts 2024-09-11T08:13:58.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:57 smithi181 ceph-mon[32701]: 6.8 scrub ok 2024-09-11T08:13:58.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:57 smithi181 ceph-mon[32701]: 3.4 scrub starts 2024-09-11T08:13:58.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:57 smithi181 ceph-mon[32701]: 3.4 scrub ok 2024-09-11T08:13:58.331 INFO:teuthology.orchestra.run.smithi150.stdout:moved 0 / 444 (0%) 2024-09-11T08:13:58.331 INFO:teuthology.orchestra.run.smithi150.stdout:avg 63.4286 2024-09-11T08:13:58.331 INFO:teuthology.orchestra.run.smithi150.stdout:stddev 7.47786 -> 7.47786 (expected baseline 7.37342) 2024-09-11T08:13:58.331 INFO:teuthology.orchestra.run.smithi150.stdout:min osd.2 with 49 -> 49 pgs (0.772523 -> 0.772523 * mean) 2024-09-11T08:13:58.331 INFO:teuthology.orchestra.run.smithi150.stdout:max osd.1 with 71 -> 71 pgs (1.11937 -> 1.11937 * mean) 2024-09-11T08:13:58.331 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:13:58.331 INFO:teuthology.orchestra.run.smithi150.stdout:oload 1000 2024-09-11T08:13:58.331 INFO:teuthology.orchestra.run.smithi150.stdout:max_change 1 2024-09-11T08:13:58.331 INFO:teuthology.orchestra.run.smithi150.stdout:max_change_osds 4 2024-09-11T08:13:58.332 INFO:teuthology.orchestra.run.smithi150.stdout:average_utilization 0.0051 2024-09-11T08:13:58.332 INFO:teuthology.orchestra.run.smithi150.stdout:overload_utilization 0.0507 2024-09-11T08:13:58.332 INFO:teuthology.orchestra.run.smithi150.stderr:no change 2024-09-11T08:13:58.400 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:58.423 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:58.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:58.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:58.525 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:13:58.548 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:58.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:58.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:13:58.593 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:13:58 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:13:58] "GET /metrics HTTP/1.1" 200 221542 "" "Prometheus/2.43.0" 2024-09-11T08:13:58.650 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:58.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:58.674 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:58.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:58.775 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:58.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:58.797 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:58.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:58.898 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:13:58.921 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:58.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:58.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:13:58.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:58 smithi150 ceph-mon[29276]: 4.3 scrub starts 2024-09-11T08:13:58.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:58 smithi150 ceph-mon[29276]: 4.3 scrub ok 2024-09-11T08:13:58.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:58 smithi150 ceph-mon[29276]: 5.1 scrub starts 2024-09-11T08:13:58.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:58 smithi150 ceph-mon[29276]: 5.1 scrub ok 2024-09-11T08:13:58.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:58 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:13:58.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:58 smithi150 ceph-mon[29276]: from='client.24947 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:13:58.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:58 smithi150 ceph-mon[39612]: 4.3 scrub starts 2024-09-11T08:13:58.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:58 smithi150 ceph-mon[39612]: 4.3 scrub ok 2024-09-11T08:13:58.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:58 smithi150 ceph-mon[39612]: 5.1 scrub starts 2024-09-11T08:13:58.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:58 smithi150 ceph-mon[39612]: 5.1 scrub ok 2024-09-11T08:13:58.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:58 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:13:58.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:58 smithi150 ceph-mon[39612]: from='client.24947 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:13:59.023 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:13:59.046 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:59.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:59.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:13:59.060 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-11T08:13:59.060 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd reweight-by-utilization 1000 0.05 2024-09-11T08:13:59.148 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:59.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:59.171 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:59.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:59.272 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:59.294 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:59.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:59.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:59.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:58 smithi181 ceph-mon[32701]: 4.3 scrub starts 2024-09-11T08:13:59.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:58 smithi181 ceph-mon[32701]: 4.3 scrub ok 2024-09-11T08:13:59.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:58 smithi181 ceph-mon[32701]: 5.1 scrub starts 2024-09-11T08:13:59.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:58 smithi181 ceph-mon[32701]: 5.1 scrub ok 2024-09-11T08:13:59.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:58 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:13:59.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:58 smithi181 ceph-mon[32701]: from='client.24947 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:13:59.396 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:13:59.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:59.418 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:59.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:13:59.519 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:59.544 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:59.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:59.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:59.646 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:59.646 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:13:59.647 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:13:59.671 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:59.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:59.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:13:59.772 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:13:59.798 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:13:59.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:59.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:13:59.831 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:59 smithi150 ceph-mon[39612]: pgmap v267: 148 pgs: 11 active+undersized, 5 active+undersized+degraded, 132 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 15/588 objects degraded (2.551%) 2024-09-11T08:13:59.831 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:59 smithi150 ceph-mon[39612]: 4.12 scrub starts 2024-09-11T08:13:59.831 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:59 smithi150 ceph-mon[39612]: 4.12 scrub ok 2024-09-11T08:13:59.831 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:59 smithi150 ceph-mon[29276]: pgmap v267: 148 pgs: 11 active+undersized, 5 active+undersized+degraded, 132 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 15/588 objects degraded (2.551%) 2024-09-11T08:13:59.832 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:59 smithi150 ceph-mon[29276]: 4.12 scrub starts 2024-09-11T08:13:59.832 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:59 smithi150 ceph-mon[29276]: 4.12 scrub ok 2024-09-11T08:13:59.832 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:59 smithi150 ceph-mon[29276]: 4.1f scrub starts 2024-09-11T08:13:59.832 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:13:59 smithi150 ceph-mon[29276]: 4.1f scrub ok 2024-09-11T08:13:59.900 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:13:59.922 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:13:59.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:13:59.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:00.024 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:00.047 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:00.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:00.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:00.065 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-11T08:14:00.126 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:59 smithi150 ceph-mon[39612]: 4.1f scrub starts 2024-09-11T08:14:00.126 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:13:59 smithi150 ceph-mon[39612]: 4.1f scrub ok 2024-09-11T08:14:00.149 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:00.170 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:00.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:00.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:00.272 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:00.298 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:00.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:00.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:00.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:59 smithi181 ceph-mon[32701]: pgmap v267: 148 pgs: 11 active+undersized, 5 active+undersized+degraded, 132 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 15/588 objects degraded (2.551%) 2024-09-11T08:14:00.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:59 smithi181 ceph-mon[32701]: 4.12 scrub starts 2024-09-11T08:14:00.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:59 smithi181 ceph-mon[32701]: 4.12 scrub ok 2024-09-11T08:14:00.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:59 smithi181 ceph-mon[32701]: 4.1f scrub starts 2024-09-11T08:14:00.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:13:59 smithi181 ceph-mon[32701]: 4.1f scrub ok 2024-09-11T08:14:00.399 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:14:00.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:00.425 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:00.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:14:00.526 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:00.553 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:00.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:00.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:00.654 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:14:00.676 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:00.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:00.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:14:00.778 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:00.799 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:00.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:00.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:00.890 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:00 smithi150 ceph-mon[29276]: 4.0 scrub starts 2024-09-11T08:14:00.890 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:00 smithi150 ceph-mon[29276]: 4.0 scrub ok 2024-09-11T08:14:00.890 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:00 smithi150 ceph-mon[29276]: 4.1c scrub starts 2024-09-11T08:14:00.890 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:00 smithi150 ceph-mon[29276]: 3.f scrub starts 2024-09-11T08:14:00.890 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:00 smithi150 ceph-mon[29276]: 4.1c scrub ok 2024-09-11T08:14:00.890 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:00 smithi150 ceph-mon[29276]: 3.f scrub ok 2024-09-11T08:14:00.891 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:00 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2965421969' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:14:00.891 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:00 smithi150 ceph-mon[39612]: 4.0 scrub starts 2024-09-11T08:14:00.891 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:00 smithi150 ceph-mon[39612]: 4.0 scrub ok 2024-09-11T08:14:00.891 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:00 smithi150 ceph-mon[39612]: 4.1c scrub starts 2024-09-11T08:14:00.891 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:00 smithi150 ceph-mon[39612]: 3.f scrub starts 2024-09-11T08:14:00.891 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:00 smithi150 ceph-mon[39612]: 4.1c scrub ok 2024-09-11T08:14:00.891 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:00 smithi150 ceph-mon[39612]: 3.f scrub ok 2024-09-11T08:14:00.891 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:00 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2965421969' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:14:00.901 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:00.927 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:00.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:00.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:01.029 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:14:01.052 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:01.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:01.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:14:01.153 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:01.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:01.175 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:01.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:01.229 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:00 smithi181 ceph-mon[32701]: 4.0 scrub starts 2024-09-11T08:14:01.229 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:00 smithi181 ceph-mon[32701]: 4.0 scrub ok 2024-09-11T08:14:01.229 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:00 smithi181 ceph-mon[32701]: 4.1c scrub starts 2024-09-11T08:14:01.229 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:00 smithi181 ceph-mon[32701]: 3.f scrub starts 2024-09-11T08:14:01.229 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:00 smithi181 ceph-mon[32701]: 4.1c scrub ok 2024-09-11T08:14:01.229 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:00 smithi181 ceph-mon[32701]: 3.f scrub ok 2024-09-11T08:14:01.229 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:00 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2965421969' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:14:01.277 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:01.298 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:01.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:01.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:01.400 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:01.426 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:01.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:01.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:01.464 INFO:teuthology.orchestra.run.smithi150.stderr:nodeep-scrub is unset 2024-09-11T08:14:01.528 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:01.556 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:01.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:01.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:01.658 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:14:01.681 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:01.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:01.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:14:01.783 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:01.809 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:01.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:01.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:01.911 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:01.937 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:01.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:01.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:02.039 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:02.061 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:02.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:02.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:02.162 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:02.165 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:01 smithi150 ceph-mon[29276]: 5.17 scrub starts 2024-09-11T08:14:02.166 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:01 smithi150 ceph-mon[29276]: 5.17 scrub ok 2024-09-11T08:14:02.166 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:01 smithi150 ceph-mon[29276]: 4.1e scrub starts 2024-09-11T08:14:02.166 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:01 smithi150 ceph-mon[29276]: 4.1e scrub ok 2024-09-11T08:14:02.166 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:01 smithi150 ceph-mon[29276]: pgmap v268: 148 pgs: 11 active+undersized, 5 active+undersized+degraded, 132 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 736 B/s rd, 0 op/s; 15/588 objects degraded (2.551%) 2024-09-11T08:14:02.166 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:01 smithi150 ceph-mon[29276]: 5.1a scrub starts 2024-09-11T08:14:02.166 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:01 smithi150 ceph-mon[29276]: 5.1a scrub ok 2024-09-11T08:14:02.166 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:01 smithi150 ceph-mon[29276]: 5.c scrub starts 2024-09-11T08:14:02.166 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:01 smithi150 ceph-mon[29276]: 5.c scrub ok 2024-09-11T08:14:02.166 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:01 smithi150 ceph-mon[29276]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:14:02.166 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:01 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2965421969' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:14:02.166 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:01 smithi150 ceph-mon[29276]: osdmap e162: 8 total, 7 up, 8 in 2024-09-11T08:14:02.167 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:01 smithi150 ceph-mon[39612]: 5.17 scrub starts 2024-09-11T08:14:02.167 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:01 smithi150 ceph-mon[39612]: 5.17 scrub ok 2024-09-11T08:14:02.167 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:01 smithi150 ceph-mon[39612]: 4.1e scrub starts 2024-09-11T08:14:02.167 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:01 smithi150 ceph-mon[39612]: 4.1e scrub ok 2024-09-11T08:14:02.167 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:01 smithi150 ceph-mon[39612]: pgmap v268: 148 pgs: 11 active+undersized, 5 active+undersized+degraded, 132 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 736 B/s rd, 0 op/s; 15/588 objects degraded (2.551%) 2024-09-11T08:14:02.167 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:01 smithi150 ceph-mon[39612]: 5.1a scrub starts 2024-09-11T08:14:02.167 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:01 smithi150 ceph-mon[39612]: 5.1a scrub ok 2024-09-11T08:14:02.167 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:01 smithi150 ceph-mon[39612]: 5.c scrub starts 2024-09-11T08:14:02.167 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:01 smithi150 ceph-mon[39612]: 5.c scrub ok 2024-09-11T08:14:02.167 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:01 smithi150 ceph-mon[39612]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:14:02.167 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:01 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2965421969' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:14:02.167 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:01 smithi150 ceph-mon[39612]: osdmap e162: 8 total, 7 up, 8 in 2024-09-11T08:14:02.185 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:02.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:02.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:02.286 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:02.312 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:02.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:02.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:02.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:01 smithi181 ceph-mon[32701]: 5.17 scrub starts 2024-09-11T08:14:02.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:01 smithi181 ceph-mon[32701]: 5.17 scrub ok 2024-09-11T08:14:02.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:01 smithi181 ceph-mon[32701]: 4.1e scrub starts 2024-09-11T08:14:02.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:01 smithi181 ceph-mon[32701]: 4.1e scrub ok 2024-09-11T08:14:02.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:01 smithi181 ceph-mon[32701]: pgmap v268: 148 pgs: 11 active+undersized, 5 active+undersized+degraded, 132 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 736 B/s rd, 0 op/s; 15/588 objects degraded (2.551%) 2024-09-11T08:14:02.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:01 smithi181 ceph-mon[32701]: 5.1a scrub starts 2024-09-11T08:14:02.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:01 smithi181 ceph-mon[32701]: 5.1a scrub ok 2024-09-11T08:14:02.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:01 smithi181 ceph-mon[32701]: 5.c scrub starts 2024-09-11T08:14:02.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:01 smithi181 ceph-mon[32701]: 5.c scrub ok 2024-09-11T08:14:02.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:01 smithi181 ceph-mon[32701]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:14:02.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:01 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2965421969' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:14:02.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:01 smithi181 ceph-mon[32701]: osdmap e162: 8 total, 7 up, 8 in 2024-09-11T08:14:02.414 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:02.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:02.437 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:02.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:02.538 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:14:02.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:02.561 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:02.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:14:02.662 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:02.683 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:02.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:02.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:02.785 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:02.811 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:02.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:02.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:02.913 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:02.938 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:02.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:02.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:03.040 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:03.065 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:03.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:03.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:03.168 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:03.193 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:03.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:03.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:03.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:02 smithi150 ceph-mon[39612]: 6.9 scrub starts 2024-09-11T08:14:03.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:02 smithi150 ceph-mon[39612]: 6.9 scrub ok 2024-09-11T08:14:03.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:02 smithi150 ceph-mon[39612]: 5.8 scrub starts 2024-09-11T08:14:03.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:02 smithi150 ceph-mon[39612]: 5.8 scrub ok 2024-09-11T08:14:03.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:02 smithi150 ceph-mon[39612]: 5.10 scrub starts 2024-09-11T08:14:03.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:02 smithi150 ceph-mon[39612]: 5.10 scrub ok 2024-09-11T08:14:03.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:02 smithi150 ceph-mon[39612]: 6.4 scrub starts 2024-09-11T08:14:03.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:02 smithi150 ceph-mon[39612]: 6.4 scrub ok 2024-09-11T08:14:03.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:02 smithi150 ceph-mon[29276]: 6.9 scrub starts 2024-09-11T08:14:03.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:02 smithi150 ceph-mon[29276]: 6.9 scrub ok 2024-09-11T08:14:03.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:02 smithi150 ceph-mon[29276]: 5.8 scrub starts 2024-09-11T08:14:03.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:02 smithi150 ceph-mon[29276]: 5.8 scrub ok 2024-09-11T08:14:03.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:02 smithi150 ceph-mon[29276]: 5.10 scrub starts 2024-09-11T08:14:03.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:02 smithi150 ceph-mon[29276]: 5.10 scrub ok 2024-09-11T08:14:03.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:02 smithi150 ceph-mon[29276]: 6.4 scrub starts 2024-09-11T08:14:03.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:02 smithi150 ceph-mon[29276]: 6.4 scrub ok 2024-09-11T08:14:03.295 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:03.320 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:03.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:03.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:03.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:02 smithi181 ceph-mon[32701]: 6.9 scrub starts 2024-09-11T08:14:03.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:02 smithi181 ceph-mon[32701]: 6.9 scrub ok 2024-09-11T08:14:03.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:02 smithi181 ceph-mon[32701]: 5.8 scrub starts 2024-09-11T08:14:03.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:02 smithi181 ceph-mon[32701]: 5.8 scrub ok 2024-09-11T08:14:03.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:02 smithi181 ceph-mon[32701]: 5.10 scrub starts 2024-09-11T08:14:03.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:02 smithi181 ceph-mon[32701]: 5.10 scrub ok 2024-09-11T08:14:03.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:02 smithi181 ceph-mon[32701]: 6.4 scrub starts 2024-09-11T08:14:03.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:02 smithi181 ceph-mon[32701]: 6.4 scrub ok 2024-09-11T08:14:03.367 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:14:03.422 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:03.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:03.445 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:03.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:03.545 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:03.567 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:03.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:03.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:03.669 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:03.695 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:03.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:03.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:03.797 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:03.822 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:03.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:03.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:03.901 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:03 smithi150 ceph-mon[29276]: 5.14 scrub starts 2024-09-11T08:14:03.901 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:03 smithi150 ceph-mon[29276]: 5.14 scrub ok 2024-09-11T08:14:03.901 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:03 smithi150 ceph-mon[29276]: pgmap v270: 148 pgs: 11 active+undersized, 5 active+undersized+degraded, 132 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 644 B/s rd, 0 op/s; 15/588 objects degraded (2.551%) 2024-09-11T08:14:03.901 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:03 smithi150 ceph-mon[39612]: 5.14 scrub starts 2024-09-11T08:14:03.901 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:03 smithi150 ceph-mon[39612]: 5.14 scrub ok 2024-09-11T08:14:03.902 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:03 smithi150 ceph-mon[39612]: pgmap v270: 148 pgs: 11 active+undersized, 5 active+undersized+degraded, 132 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 644 B/s rd, 0 op/s; 15/588 objects degraded (2.551%) 2024-09-11T08:14:03.924 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:03.950 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:03.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:03.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:04.052 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:04.078 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:04.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:04.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:04.180 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:04.192 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T08:14:04.205 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:04.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:04.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:04.307 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:04.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:03 smithi181 ceph-mon[32701]: 5.14 scrub starts 2024-09-11T08:14:04.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:03 smithi181 ceph-mon[32701]: 5.14 scrub ok 2024-09-11T08:14:04.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:03 smithi181 ceph-mon[32701]: pgmap v270: 148 pgs: 11 active+undersized, 5 active+undersized+degraded, 132 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 644 B/s rd, 0 op/s; 15/588 objects degraded (2.551%) 2024-09-11T08:14:04.332 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:04.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:04.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:04.336 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:14:04.337 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:14:04.339 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:14:04.355 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:14:04.356 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:14:04.371 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:14:04.371 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:14:04.386 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:14:04.387 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:14:04.403 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:14:04.403 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:14:04.419 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:14:04.419 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:14:04.434 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:04.435 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:14:04.435 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:14:04.435 INFO:teuthology.orchestra.run.smithi150.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-11T08:14:04.456 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:04.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:04.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:04.558 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:14:04.579 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:04.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:04.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:14:04.680 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:04.707 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:04.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:04.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:04.809 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:04.834 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:04.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:04.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:04.937 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:04.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:04.959 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:04.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:05.061 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:14:05.082 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:05.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:05.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:14:05.145 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:04 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2761269902' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:14:05.183 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:05.194 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:04 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2761269902' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:14:05.194 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:04 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2761269902' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:14:05.209 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:05.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:05.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:05.217 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-11T08:14:05.217 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-11T08:14:05.311 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:05.332 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:05.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:05.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:05.434 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:05.456 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:05.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:05.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:05.557 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:05.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:14:05 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:14:05] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:14:05.578 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:05.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:05.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:05.681 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:05.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:05.704 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:05.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:05.805 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:05.831 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:05.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:05.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:05.932 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:05.957 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:05.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:05.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:06.059 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:06.085 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:06.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:06.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:06.187 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:06.213 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:06.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:06.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:06.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:05 smithi150 ceph-mon[29276]: pgmap v271: 148 pgs: 11 active+undersized, 5 active+undersized+degraded, 132 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 15/588 objects degraded (2.551%) 2024-09-11T08:14:06.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:05 smithi150 ceph-mon[29276]: 6.2 deep-scrub starts 2024-09-11T08:14:06.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:05 smithi150 ceph-mon[29276]: 6.2 deep-scrub ok 2024-09-11T08:14:06.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:05 smithi150 ceph-mon[39612]: pgmap v271: 148 pgs: 11 active+undersized, 5 active+undersized+degraded, 132 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 15/588 objects degraded (2.551%) 2024-09-11T08:14:06.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:05 smithi150 ceph-mon[39612]: 6.2 deep-scrub starts 2024-09-11T08:14:06.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:05 smithi150 ceph-mon[39612]: 6.2 deep-scrub ok 2024-09-11T08:14:06.315 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:06.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:05 smithi181 ceph-mon[32701]: pgmap v271: 148 pgs: 11 active+undersized, 5 active+undersized+degraded, 132 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 15/588 objects degraded (2.551%) 2024-09-11T08:14:06.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:05 smithi181 ceph-mon[32701]: 6.2 deep-scrub starts 2024-09-11T08:14:06.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:05 smithi181 ceph-mon[32701]: 6.2 deep-scrub ok 2024-09-11T08:14:06.337 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:06.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:06.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:06.439 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:06.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:06.465 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:06.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:06.566 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:06.592 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:06.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:06.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:06.693 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:06.720 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:06.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:06.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:06.821 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:06.847 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:06.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:06.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:06.948 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:06.970 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:06.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:06.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:07.065 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:06 smithi150 ceph-mon[29276]: 6.6 scrub starts 2024-09-11T08:14:07.065 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:06 smithi150 ceph-mon[29276]: 6.6 scrub ok 2024-09-11T08:14:07.065 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:06 smithi150 ceph-mon[29276]: 4.16 scrub starts 2024-09-11T08:14:07.065 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:06 smithi150 ceph-mon[29276]: 4.16 scrub ok 2024-09-11T08:14:07.065 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:06 smithi150 ceph-mon[29276]: 5.a scrub starts 2024-09-11T08:14:07.065 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:06 smithi150 ceph-mon[29276]: 5.a scrub ok 2024-09-11T08:14:07.065 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:06 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:14:07.065 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:06 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:14:07.066 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:06 smithi150 ceph-mon[39612]: 6.6 scrub starts 2024-09-11T08:14:07.066 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:06 smithi150 ceph-mon[39612]: 6.6 scrub ok 2024-09-11T08:14:07.066 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:06 smithi150 ceph-mon[39612]: 4.16 scrub starts 2024-09-11T08:14:07.066 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:06 smithi150 ceph-mon[39612]: 4.16 scrub ok 2024-09-11T08:14:07.066 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:06 smithi150 ceph-mon[39612]: 5.a scrub starts 2024-09-11T08:14:07.066 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:06 smithi150 ceph-mon[39612]: 5.a scrub ok 2024-09-11T08:14:07.066 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:06 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:14:07.066 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:06 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:14:07.072 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:07.097 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:07.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:07.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:07.199 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:07.219 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T08:14:07.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:07.222 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:07.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:07.323 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:07.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:06 smithi181 ceph-mon[32701]: 6.6 scrub starts 2024-09-11T08:14:07.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:06 smithi181 ceph-mon[32701]: 6.6 scrub ok 2024-09-11T08:14:07.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:06 smithi181 ceph-mon[32701]: 4.16 scrub starts 2024-09-11T08:14:07.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:06 smithi181 ceph-mon[32701]: 4.16 scrub ok 2024-09-11T08:14:07.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:06 smithi181 ceph-mon[32701]: 5.a scrub starts 2024-09-11T08:14:07.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:06 smithi181 ceph-mon[32701]: 5.a scrub ok 2024-09-11T08:14:07.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:06 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:14:07.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:06 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:14:07.349 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:07.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:07.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:07.451 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:07.478 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:07.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:07.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:07.580 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:07.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:07.606 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:07.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:07.707 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:07.733 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:07.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:07.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:07.835 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:07.857 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:07.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:07.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:07.959 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:14:07.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:07.982 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:07.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:14:07.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:07 smithi150 ceph-mon[29276]: 5.7 scrub starts 2024-09-11T08:14:07.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:07 smithi150 ceph-mon[29276]: 5.7 scrub ok 2024-09-11T08:14:07.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:07 smithi150 ceph-mon[29276]: pgmap v272: 148 pgs: 11 active+undersized, 5 active+undersized+degraded, 132 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 15/588 objects degraded (2.551%) 2024-09-11T08:14:07.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:07 smithi150 ceph-mon[29276]: 5.2 scrub starts 2024-09-11T08:14:07.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:07 smithi150 ceph-mon[29276]: 5.2 scrub ok 2024-09-11T08:14:07.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:07 smithi150 ceph-mon[39612]: 5.7 scrub starts 2024-09-11T08:14:07.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:07 smithi150 ceph-mon[39612]: 5.7 scrub ok 2024-09-11T08:14:07.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:07 smithi150 ceph-mon[39612]: pgmap v272: 148 pgs: 11 active+undersized, 5 active+undersized+degraded, 132 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 15/588 objects degraded (2.551%) 2024-09-11T08:14:07.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:07 smithi150 ceph-mon[39612]: 5.2 scrub starts 2024-09-11T08:14:07.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:07 smithi150 ceph-mon[39612]: 5.2 scrub ok 2024-09-11T08:14:08.083 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:08.113 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:08.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:08.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:08.214 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:08.239 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:08.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:08.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:08.323 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:14:08 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:14:08.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:07 smithi181 ceph-mon[32701]: 5.7 scrub starts 2024-09-11T08:14:08.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:07 smithi181 ceph-mon[32701]: 5.7 scrub ok 2024-09-11T08:14:08.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:07 smithi181 ceph-mon[32701]: pgmap v272: 148 pgs: 11 active+undersized, 5 active+undersized+degraded, 132 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 15/588 objects degraded (2.551%) 2024-09-11T08:14:08.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:07 smithi181 ceph-mon[32701]: 5.2 scrub starts 2024-09-11T08:14:08.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:07 smithi181 ceph-mon[32701]: 5.2 scrub ok 2024-09-11T08:14:08.340 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:14:08.362 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:08.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:08.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:14:08.464 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:08.485 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:08.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:08.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:08.587 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:14:08.607 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:08.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:08.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:14:08.624 INFO:teuthology.orchestra.run.smithi150.stdout:moved 0 / 444 (0%) 2024-09-11T08:14:08.624 INFO:teuthology.orchestra.run.smithi150.stdout:avg 63.4286 2024-09-11T08:14:08.624 INFO:teuthology.orchestra.run.smithi150.stdout:stddev 7.47786 -> 7.47786 (expected baseline 7.37342) 2024-09-11T08:14:08.624 INFO:teuthology.orchestra.run.smithi150.stdout:min osd.2 with 49 -> 49 pgs (0.772523 -> 0.772523 * mean) 2024-09-11T08:14:08.624 INFO:teuthology.orchestra.run.smithi150.stdout:max osd.1 with 71 -> 71 pgs (1.11937 -> 1.11937 * mean) 2024-09-11T08:14:08.624 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:14:08.624 INFO:teuthology.orchestra.run.smithi150.stdout:oload 1000 2024-09-11T08:14:08.625 INFO:teuthology.orchestra.run.smithi150.stdout:max_change 0.05 2024-09-11T08:14:08.625 INFO:teuthology.orchestra.run.smithi150.stdout:max_change_osds 4 2024-09-11T08:14:08.629 INFO:teuthology.orchestra.run.smithi150.stdout:average_utilization 0.0051 2024-09-11T08:14:08.629 INFO:teuthology.orchestra.run.smithi150.stdout:overload_utilization 0.0507 2024-09-11T08:14:08.629 INFO:teuthology.orchestra.run.smithi150.stderr:no change 2024-09-11T08:14:08.629 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:14:08 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:14:08] "GET /metrics HTTP/1.1" 200 223867 "" "Prometheus/2.43.0" 2024-09-11T08:14:08.708 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:14:08.709 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:08.734 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:08.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:08.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:08.836 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:08.860 INFO:teuthology.orchestra.run.smithi150.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-09-11T08:14:08.862 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:08.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:08.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:08.964 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:08.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:08 smithi150 ceph-mon[29276]: 6.1d scrub starts 2024-09-11T08:14:08.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:08 smithi150 ceph-mon[29276]: 6.1d scrub ok 2024-09-11T08:14:08.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:08 smithi150 ceph-mon[29276]: 6.f deep-scrub starts 2024-09-11T08:14:08.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:08 smithi150 ceph-mon[29276]: 6.f deep-scrub ok 2024-09-11T08:14:08.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:08 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:14:08.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:08 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2562458701' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-11T08:14:08.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:08 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-11T08:14:08.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:08 smithi150 ceph-mon[39612]: 6.1d scrub starts 2024-09-11T08:14:08.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:08 smithi150 ceph-mon[39612]: 6.1d scrub ok 2024-09-11T08:14:08.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:08 smithi150 ceph-mon[39612]: 6.f deep-scrub starts 2024-09-11T08:14:08.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:08 smithi150 ceph-mon[39612]: 6.f deep-scrub ok 2024-09-11T08:14:08.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:08 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:14:08.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:08 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2562458701' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-11T08:14:08.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:08 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-11T08:14:08.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:08.991 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:08.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:09.092 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:09.117 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:09.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:09.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:09.218 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:09.241 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:09.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:09.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:09.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:08 smithi181 ceph-mon[32701]: 6.1d scrub starts 2024-09-11T08:14:09.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:08 smithi181 ceph-mon[32701]: 6.1d scrub ok 2024-09-11T08:14:09.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:08 smithi181 ceph-mon[32701]: 6.f deep-scrub starts 2024-09-11T08:14:09.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:08 smithi181 ceph-mon[32701]: 6.f deep-scrub ok 2024-09-11T08:14:09.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:08 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:14:09.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:08 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2562458701' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-11T08:14:09.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:08 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-11T08:14:09.343 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:09.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:09.371 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:09.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:09.472 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:14:09.495 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:09.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:09.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:14:09.597 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:14:09.620 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:09.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:09.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:14:09.721 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:09.748 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:09.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:09.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:09.850 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:09.873 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:09.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:09.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:09.975 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:09.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:09.997 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:09.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:10.098 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:14:10.120 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:10.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:10.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:14:10.222 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:10.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:09 smithi150 ceph-mon[29276]: 6.19 scrub starts 2024-09-11T08:14:10.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:09 smithi150 ceph-mon[29276]: 6.19 scrub ok 2024-09-11T08:14:10.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:09 smithi150 ceph-mon[29276]: from='client.24971 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:14:10.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:09 smithi150 ceph-mon[29276]: pgmap v273: 148 pgs: 11 active+undersized, 5 active+undersized+degraded, 132 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 15/588 objects degraded (2.551%) 2024-09-11T08:14:10.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:09 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-11T08:14:10.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:09 smithi150 ceph-mon[29276]: osdmap e163: 8 total, 7 up, 8 in 2024-09-11T08:14:10.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:09 smithi150 ceph-mon[39612]: 6.19 scrub starts 2024-09-11T08:14:10.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:09 smithi150 ceph-mon[39612]: 6.19 scrub ok 2024-09-11T08:14:10.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:09 smithi150 ceph-mon[39612]: from='client.24971 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:14:10.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:09 smithi150 ceph-mon[39612]: pgmap v273: 148 pgs: 11 active+undersized, 5 active+undersized+degraded, 132 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 15/588 objects degraded (2.551%) 2024-09-11T08:14:10.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:09 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-11T08:14:10.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:09 smithi150 ceph-mon[39612]: osdmap e163: 8 total, 7 up, 8 in 2024-09-11T08:14:10.248 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:10.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:10.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:10.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:09 smithi181 ceph-mon[32701]: 6.19 scrub starts 2024-09-11T08:14:10.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:09 smithi181 ceph-mon[32701]: 6.19 scrub ok 2024-09-11T08:14:10.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:09 smithi181 ceph-mon[32701]: from='client.24971 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:14:10.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:09 smithi181 ceph-mon[32701]: pgmap v273: 148 pgs: 11 active+undersized, 5 active+undersized+degraded, 132 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 15/588 objects degraded (2.551%) 2024-09-11T08:14:10.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:09 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-11T08:14:10.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:09 smithi181 ceph-mon[32701]: osdmap e163: 8 total, 7 up, 8 in 2024-09-11T08:14:10.351 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:14:10.372 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:10.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:10.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:14:10.430 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-11T08:14:10.430 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [], 2024-09-11T08:14:10.431 INFO:teuthology.orchestra.run.smithi150.stdout: "num_ops": 0 2024-09-11T08:14:10.431 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-11T08:14:10.474 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:10.495 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:10.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:10.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:10.597 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:10.618 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:10.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:10.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:10.720 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:10.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:10.747 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:10.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:10.848 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:10.874 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:10.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:10.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:10.976 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:10.998 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:10.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:10.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:11.100 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:11.122 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:11.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:11.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:11.126 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:10 smithi150 ceph-mon[29276]: 6.1 scrub starts 2024-09-11T08:14:11.126 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:10 smithi150 ceph-mon[29276]: 6.1 scrub ok 2024-09-11T08:14:11.127 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:10 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:14:11.127 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:10 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:14:11.127 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:10 smithi150 ceph-mon[29276]: 3.d scrub starts 2024-09-11T08:14:11.127 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:10 smithi150 ceph-mon[29276]: 3.d scrub ok 2024-09-11T08:14:11.127 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:10 smithi150 ceph-mon[29276]: osdmap e164: 8 total, 7 up, 8 in 2024-09-11T08:14:11.127 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:10 smithi150 ceph-mon[39612]: 6.1 scrub starts 2024-09-11T08:14:11.127 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:10 smithi150 ceph-mon[39612]: 6.1 scrub ok 2024-09-11T08:14:11.127 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:10 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:14:11.127 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:10 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:14:11.127 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:10 smithi150 ceph-mon[39612]: 3.d scrub starts 2024-09-11T08:14:11.127 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:10 smithi150 ceph-mon[39612]: 3.d scrub ok 2024-09-11T08:14:11.127 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:10 smithi150 ceph-mon[39612]: osdmap e164: 8 total, 7 up, 8 in 2024-09-11T08:14:11.224 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:11.251 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:11.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:11.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:11.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:10 smithi181 ceph-mon[32701]: 6.1 scrub starts 2024-09-11T08:14:11.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:10 smithi181 ceph-mon[32701]: 6.1 scrub ok 2024-09-11T08:14:11.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:10 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:14:11.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:10 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:14:11.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:10 smithi181 ceph-mon[32701]: 3.d scrub starts 2024-09-11T08:14:11.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:10 smithi181 ceph-mon[32701]: 3.d scrub ok 2024-09-11T08:14:11.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:10 smithi181 ceph-mon[32701]: osdmap e164: 8 total, 7 up, 8 in 2024-09-11T08:14:11.353 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:11.354 DEBUG:teuthology.orchestra.run.smithi150:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-09-11T08:14:11.357 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-11T08:14:11.358 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-11T08:14:11.376 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:11.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:11.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:11.404 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:hints = 1 2024-09-11T08:14:11.404 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-11T08:14:11.404 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Object prefix: benchmark_data_smithi150_107646 2024-09-11T08:14:11.404 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-11T08:14:11.405 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 0 0 0 0 0 0 - 0 2024-09-11T08:14:11.478 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:11.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:11.522 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:11.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:11.623 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:11.661 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:11.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:11.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:11.763 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:11.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:11.796 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:11.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:11.841 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:14:11.844 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-11T08:14:11.896 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:11.933 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:11.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:11.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:12.035 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:12.064 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:12.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:12.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:12.166 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:12.205 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:12.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:12.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:12.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:11 smithi150 ceph-mon[29276]: 4.11 scrub starts 2024-09-11T08:14:12.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:11 smithi150 ceph-mon[29276]: 4.11 scrub ok 2024-09-11T08:14:12.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:11 smithi150 ceph-mon[29276]: 1.0 scrub starts 2024-09-11T08:14:12.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:11 smithi150 ceph-mon[29276]: 1.0 scrub ok 2024-09-11T08:14:12.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:11 smithi150 ceph-mon[29276]: pgmap v276: 148 pgs: 11 active+undersized, 5 active+undersized+degraded, 132 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 15/588 objects degraded (2.551%) 2024-09-11T08:14:12.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:11 smithi150 ceph-mon[29276]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-11T08:14:12.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:11 smithi150 ceph-mon[39612]: 4.11 scrub starts 2024-09-11T08:14:12.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:11 smithi150 ceph-mon[39612]: 4.11 scrub ok 2024-09-11T08:14:12.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:11 smithi150 ceph-mon[39612]: 1.0 scrub starts 2024-09-11T08:14:12.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:11 smithi150 ceph-mon[39612]: 1.0 scrub ok 2024-09-11T08:14:12.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:11 smithi150 ceph-mon[39612]: pgmap v276: 148 pgs: 11 active+undersized, 5 active+undersized+degraded, 132 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 15/588 objects degraded (2.551%) 2024-09-11T08:14:12.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:11 smithi150 ceph-mon[39612]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-11T08:14:12.307 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:11 smithi181 ceph-mon[32701]: 4.11 scrub starts 2024-09-11T08:14:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:11 smithi181 ceph-mon[32701]: 4.11 scrub ok 2024-09-11T08:14:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:11 smithi181 ceph-mon[32701]: 1.0 scrub starts 2024-09-11T08:14:12.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:11 smithi181 ceph-mon[32701]: 1.0 scrub ok 2024-09-11T08:14:12.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:11 smithi181 ceph-mon[32701]: pgmap v276: 148 pgs: 11 active+undersized, 5 active+undersized+degraded, 132 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 15/588 objects degraded (2.551%) 2024-09-11T08:14:12.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:11 smithi181 ceph-mon[32701]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-11T08:14:12.336 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:12.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:12.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:12.404 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 1 15 3837 3822 238.862 238.875 0.00428695 0.00417245 2024-09-11T08:14:12.438 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:12.478 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:12.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:12.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:12.580 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:12.617 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:12.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:12.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:12.719 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:12.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:12.760 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:12.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:12.861 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:12.907 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:12.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:12.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:13.009 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:13.041 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:13.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:13.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:13.143 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:14:13.180 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:13.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:13.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:14:13.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:12 smithi150 ceph-mon[39612]: 5.3 scrub starts 2024-09-11T08:14:13.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:12 smithi150 ceph-mon[39612]: 5.3 scrub ok 2024-09-11T08:14:13.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:12 smithi150 ceph-mon[29276]: 5.3 scrub starts 2024-09-11T08:14:13.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:12 smithi150 ceph-mon[29276]: 5.3 scrub ok 2024-09-11T08:14:13.282 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:13.313 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:13.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:13.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:13.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:12 smithi181 ceph-mon[32701]: 5.3 scrub starts 2024-09-11T08:14:13.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:12 smithi181 ceph-mon[32701]: 5.3 scrub ok 2024-09-11T08:14:13.405 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 2 16 7426 7410 231.498 224.25 0.00403398 0.00431192 2024-09-11T08:14:13.415 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:13.456 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:13.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:13.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:13.558 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:13.583 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:13.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:13.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:13.685 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:13.719 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:13.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:13.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:13.820 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:13.851 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:13.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:13.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:13.953 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:13.959 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:14:13.980 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:13.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:13.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:14.082 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:14.111 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:14.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:14.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:14.213 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:14.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:13 smithi150 ceph-mon[39612]: 4.17 scrub starts 2024-09-11T08:14:14.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:13 smithi150 ceph-mon[39612]: 4.17 scrub ok 2024-09-11T08:14:14.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:13 smithi150 ceph-mon[39612]: pgmap v277: 148 pgs: 9 active+undersized, 7 active+undersized+degraded, 132 active+clean; 117 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 15 MiB/s wr, 233 op/s; 677/6186 objects degraded (10.944%) 2024-09-11T08:14:14.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:13 smithi150 ceph-mon[39612]: Health check update: Degraded data redundancy: 677/6186 objects degraded (10.944%), 7 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-09-11T08:14:14.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:13 smithi150 ceph-mon[39612]: 3.e scrub starts 2024-09-11T08:14:14.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:13 smithi150 ceph-mon[39612]: 3.e scrub ok 2024-09-11T08:14:14.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:13 smithi150 ceph-mon[29276]: 4.17 scrub starts 2024-09-11T08:14:14.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:13 smithi150 ceph-mon[29276]: 4.17 scrub ok 2024-09-11T08:14:14.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:13 smithi150 ceph-mon[29276]: pgmap v277: 148 pgs: 9 active+undersized, 7 active+undersized+degraded, 132 active+clean; 117 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 15 MiB/s wr, 233 op/s; 677/6186 objects degraded (10.944%) 2024-09-11T08:14:14.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:13 smithi150 ceph-mon[29276]: Health check update: Degraded data redundancy: 677/6186 objects degraded (10.944%), 7 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-09-11T08:14:14.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:13 smithi150 ceph-mon[29276]: 3.e scrub starts 2024-09-11T08:14:14.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:13 smithi150 ceph-mon[29276]: 3.e scrub ok 2024-09-11T08:14:14.241 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:14.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:14.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:14.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:13 smithi181 ceph-mon[32701]: 4.17 scrub starts 2024-09-11T08:14:14.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:13 smithi181 ceph-mon[32701]: 4.17 scrub ok 2024-09-11T08:14:14.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:13 smithi181 ceph-mon[32701]: pgmap v277: 148 pgs: 9 active+undersized, 7 active+undersized+degraded, 132 active+clean; 117 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 15 MiB/s wr, 233 op/s; 677/6186 objects degraded (10.944%) 2024-09-11T08:14:14.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:13 smithi181 ceph-mon[32701]: Health check update: Degraded data redundancy: 677/6186 objects degraded (10.944%), 7 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-09-11T08:14:14.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:13 smithi181 ceph-mon[32701]: 3.e scrub starts 2024-09-11T08:14:14.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:13 smithi181 ceph-mon[32701]: 3.e scrub ok 2024-09-11T08:14:14.343 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:14.373 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:14.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:14.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:14.405 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 3 15 10612 10597 220.724 199.188 0.00432114 0.00452408 2024-09-11T08:14:14.475 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:14.494 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [7] live_osds: [0, 1, 2, 3, 4, 5, 6] 2024-09-11T08:14:14.494 INFO:tasks.thrashosds.thrasher:Reviving osd 7 2024-09-11T08:14:14.494 INFO:tasks.cephadm.osd.7:Restarting osd.7 (starting--it wasn't running)... 2024-09-11T08:14:14.494 DEBUG:teuthology.orchestra.run.smithi181:osd.7> sudo journalctl -f -n 0 -u ceph-c1bddff4-7013-11ef-bcea-c7b262605968@osd.7.service 2024-09-11T08:14:14.497 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl start ceph-c1bddff4-7013-11ef-bcea-c7b262605968@osd.7 2024-09-11T08:14:14.512 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:14.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:14.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:14.615 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:14.646 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:14.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:14.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:14.747 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:14.782 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:14.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:14.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:14.823 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:14:14 smithi181 systemd[1]: Starting Ceph osd.7 for c1bddff4-7013-11ef-bcea-c7b262605968... 2024-09-11T08:14:14.884 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:14.913 INFO:teuthology.orchestra.run.smithi150.stderr:noscrub is set 2024-09-11T08:14:14.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:14.940 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:14.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:15.041 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:15.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:15.084 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:15.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:15.144 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:14 smithi181 ceph-mon[32701]: 3.19 scrub starts 2024-09-11T08:14:15.145 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:14 smithi181 ceph-mon[32701]: 3.19 scrub ok 2024-09-11T08:14:15.145 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:14 smithi181 ceph-mon[32701]: 3.1b scrub starts 2024-09-11T08:14:15.145 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:14 smithi181 ceph-mon[32701]: 3.1b scrub ok 2024-09-11T08:14:15.145 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:14 smithi181 ceph-mon[32701]: 6.e scrub starts 2024-09-11T08:14:15.145 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:14 smithi181 ceph-mon[32701]: 6.e scrub ok 2024-09-11T08:14:15.145 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:14 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3620289148' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:14:15.145 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:14:14 smithi181 podman[86348]: 2024-09-11 08:14:14.768713864 +0000 UTC m=+0.026979622 image pull 4a918172a49766385959227c53e75e2c302e923539324792cb5bee6cad0b2f2b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40 2024-09-11T08:14:15.145 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:14:14 smithi181 podman[86348]: 2024-09-11 08:14:14.948330732 +0000 UTC m=+0.206596476 container create 3887aa0de26e8abdafed89f9be1b4a331504307a8804153b5b23ccd6fb4b80fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7-activate, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-11T08:14:15.185 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:15.227 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:15.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:15.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:15.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:14 smithi150 ceph-mon[29276]: 3.19 scrub starts 2024-09-11T08:14:15.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:14 smithi150 ceph-mon[29276]: 3.19 scrub ok 2024-09-11T08:14:15.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:14 smithi150 ceph-mon[29276]: 3.1b scrub starts 2024-09-11T08:14:15.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:14 smithi150 ceph-mon[29276]: 3.1b scrub ok 2024-09-11T08:14:15.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:14 smithi150 ceph-mon[29276]: 6.e scrub starts 2024-09-11T08:14:15.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:14 smithi150 ceph-mon[29276]: 6.e scrub ok 2024-09-11T08:14:15.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:14 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3620289148' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:14:15.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:14 smithi150 ceph-mon[39612]: 3.19 scrub starts 2024-09-11T08:14:15.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:14 smithi150 ceph-mon[39612]: 3.19 scrub ok 2024-09-11T08:14:15.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:14 smithi150 ceph-mon[39612]: 3.1b scrub starts 2024-09-11T08:14:15.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:14 smithi150 ceph-mon[39612]: 3.1b scrub ok 2024-09-11T08:14:15.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:14 smithi150 ceph-mon[39612]: 6.e scrub starts 2024-09-11T08:14:15.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:14 smithi150 ceph-mon[39612]: 6.e scrub ok 2024-09-11T08:14:15.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:14 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3620289148' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:14:15.329 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:15.363 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:15.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:15.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:15.405 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 4 16 13645 13629 212.914 189.5 0.00412113 0.00468921 2024-09-11T08:14:15.457 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:14:15 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:14:15] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:14:15.458 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:14:15 smithi181 podman[86348]: 2024-09-11 08:14:15.357760844 +0000 UTC m=+0.616026595 container init 3887aa0de26e8abdafed89f9be1b4a331504307a8804153b5b23ccd6fb4b80fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7-activate, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828) 2024-09-11T08:14:15.458 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:14:15 smithi181 podman[86348]: 2024-09-11 08:14:15.363671155 +0000 UTC m=+0.621936897 container start 3887aa0de26e8abdafed89f9be1b4a331504307a8804153b5b23ccd6fb4b80fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7-activate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-09-11T08:14:15.464 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:15.500 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:15.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:15.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:15.602 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:14:15.637 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:15.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:15.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:14:15.739 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:14:15.767 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:14:15 smithi181 podman[86348]: 2024-09-11 08:14:15.457609889 +0000 UTC m=+0.715875653 container attach 3887aa0de26e8abdafed89f9be1b4a331504307a8804153b5b23ccd6fb4b80fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-11T08:14:15.777 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:15.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:15.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:14:15.879 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:15.914 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:15.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:15.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:16.015 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:16.035 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:15 smithi150 ceph-mon[29276]: pgmap v278: 148 pgs: 8 active+undersized, 8 active+undersized+degraded, 132 active+clean; 355 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 44 MiB/s wr, 710 op/s; 1041/17610 objects degraded (5.911%) 2024-09-11T08:14:16.035 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:15 smithi150 ceph-mon[29276]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:14:16.035 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:15 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3620289148' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:14:16.035 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:15 smithi150 ceph-mon[29276]: osdmap e165: 8 total, 7 up, 8 in 2024-09-11T08:14:16.036 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:15 smithi150 ceph-mon[39612]: pgmap v278: 148 pgs: 8 active+undersized, 8 active+undersized+degraded, 132 active+clean; 355 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 44 MiB/s wr, 710 op/s; 1041/17610 objects degraded (5.911%) 2024-09-11T08:14:16.036 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:15 smithi150 ceph-mon[39612]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:14:16.036 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:15 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3620289148' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:14:16.036 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:15 smithi150 ceph-mon[39612]: osdmap e165: 8 total, 7 up, 8 in 2024-09-11T08:14:16.056 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:16.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:16.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:16.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:15 smithi181 ceph-mon[32701]: pgmap v278: 148 pgs: 8 active+undersized, 8 active+undersized+degraded, 132 active+clean; 355 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 44 MiB/s wr, 710 op/s; 1041/17610 objects degraded (5.911%) 2024-09-11T08:14:16.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:15 smithi181 ceph-mon[32701]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:14:16.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:15 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3620289148' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:14:16.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:15 smithi181 ceph-mon[32701]: osdmap e165: 8 total, 7 up, 8 in 2024-09-11T08:14:16.074 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:14:15 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7-activate[86379]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-11T08:14:16.074 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:14:15 smithi181 bash[86348]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-11T08:14:16.074 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:14:15 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7-activate[86379]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-11T08:14:16.074 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:14:15 smithi181 bash[86348]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-11T08:14:16.074 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:14:15 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7-activate[86379]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-11T08:14:16.074 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:14:15 smithi181 bash[86348]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-11T08:14:16.074 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:14:15 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7-activate[86379]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-11T08:14:16.074 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:14:15 smithi181 bash[86348]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-11T08:14:16.074 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:14:15 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7-activate[86379]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-09-11T08:14:16.075 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:14:15 smithi181 bash[86348]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-09-11T08:14:16.075 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:14:15 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7-activate[86379]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-11T08:14:16.075 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:14:15 smithi181 bash[86348]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-11T08:14:16.075 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:14:15 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7-activate[86379]: --> ceph-volume raw activate successful for osd ID: 7 2024-09-11T08:14:16.075 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:14:15 smithi181 bash[86348]: --> ceph-volume raw activate successful for osd ID: 7 2024-09-11T08:14:16.075 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:14:15 smithi181 podman[86348]: 2024-09-11 08:14:15.836207165 +0000 UTC m=+1.094472914 container died 3887aa0de26e8abdafed89f9be1b4a331504307a8804153b5b23ccd6fb4b80fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7-activate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-11T08:14:16.157 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:16.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:16.196 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:16.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:16.298 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:16.333 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:16.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:16.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:16.405 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 5 16 16618 16602 207.491 185.812 0.00776342 0.00481329 2024-09-11T08:14:16.435 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:16.471 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:16.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:16.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:16.573 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:16.598 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:16.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:16.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:16.700 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:16.735 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:16.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:16.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:16.823 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:14:16 smithi181 podman[86526]: 2024-09-11 08:14:16.591730799 +0000 UTC m=+0.746518980 container remove 3887aa0de26e8abdafed89f9be1b4a331504307a8804153b5b23ccd6fb4b80fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7-activate, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-11T08:14:16.837 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:16.876 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:16.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:16.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:16.978 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:16.999 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:16.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:17.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:17.101 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:17.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:17.131 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:17.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:17.232 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:17.263 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:17.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:17.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:17.299 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:14:16 smithi181 podman[86654]: 2024-09-11 08:14:16.77259054 +0000 UTC m=+0.032017710 image pull 4a918172a49766385959227c53e75e2c302e923539324792cb5bee6cad0b2f2b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40 2024-09-11T08:14:17.299 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:14:16 smithi181 podman[86654]: 2024-09-11 08:14:16.901714482 +0000 UTC m=+0.161141645 container create f35c562f0f3767f3d65faacca7617f9cfd9ad2e30386448a1e84e8964f337643 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7, org.label-schema.build-date=20240731, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-11T08:14:17.366 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:17.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:17.398 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:17.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:17.405 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 6 15 18800 18785 195.647 136.438 0.00511456 0.00510537 2024-09-11T08:14:17.499 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:17.531 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:17.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:17.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:17.571 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-11T08:14:17.633 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:17.690 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:17.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:17.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:17.792 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:17.812 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T08:14:17.823 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:14:17 smithi181 podman[86654]: 2024-09-11 08:14:17.446682971 +0000 UTC m=+0.706110138 container init f35c562f0f3767f3d65faacca7617f9cfd9ad2e30386448a1e84e8964f337643 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-11T08:14:17.824 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:14:17 smithi181 podman[86654]: 2024-09-11 08:14:17.452149511 +0000 UTC m=+0.711576676 container start f35c562f0f3767f3d65faacca7617f9cfd9ad2e30386448a1e84e8964f337643 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, ceph=True, GIT_BRANCH=HEAD) 2024-09-11T08:14:17.824 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:14:17 smithi181 ceph-osd[86742]: -- 172.21.15.181:0/4076531247 <== mon.1 v2:172.21.15.181:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563426c36d00 con 0x563426c09000 2024-09-11T08:14:17.824 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:14:17 smithi181 bash[86654]: f35c562f0f3767f3d65faacca7617f9cfd9ad2e30386448a1e84e8964f337643 2024-09-11T08:14:17.824 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:14:17 smithi181 systemd[1]: Started Ceph osd.7 for c1bddff4-7013-11ef-bcea-c7b262605968. 2024-09-11T08:14:17.829 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:17.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:17.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:17.931 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:17.962 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:17.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:17.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:17.991 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:14:18.032 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:14:18.032 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:14:18.065 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:18.066 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:14:18.066 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:14:18.092 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:18.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:18.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:18.105 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:14:18.105 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:14:18.105 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:17 smithi150 ceph-mon[29276]: pgmap v280: 148 pgs: 8 active+undersized, 8 active+undersized+degraded, 132 active+clean; 355 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 642 B/s rd, 44 MiB/s wr, 712 op/s; 1041/17610 objects degraded (5.911%) 2024-09-11T08:14:18.105 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:17 smithi150 ceph-mon[39612]: pgmap v280: 148 pgs: 8 active+undersized, 8 active+undersized+degraded, 132 active+clean; 355 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 642 B/s rd, 44 MiB/s wr, 712 op/s; 1041/17610 objects degraded (5.911%) 2024-09-11T08:14:18.143 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:14:18.143 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:14:18.173 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:17 smithi181 ceph-mon[32701]: pgmap v280: 148 pgs: 8 active+undersized, 8 active+undersized+degraded, 132 active+clean; 355 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 642 B/s rd, 44 MiB/s wr, 712 op/s; 1041/17610 objects degraded (5.911%) 2024-09-11T08:14:18.173 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:14:18 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:14:18.002+0000 7f3164351440 -1 Falling back to public interface 2024-09-11T08:14:18.174 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:14:18 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:14:18.178 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:14:18.178 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:14:18.194 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:18.209 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:14:18.209 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:14:18.226 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:18.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:18.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:18.242 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:14:18.242 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:14:18.243 INFO:teuthology.orchestra.run.smithi150.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-11T08:14:18.329 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:18.360 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:18.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:18.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:18.405 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 7 16 21408 21392 190.973 162.938 0.0068881 0.00522966 2024-09-11T08:14:18.462 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:14:18.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:18.502 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:18.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:14:18.603 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:18.643 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:18.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:18.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:18.745 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:18.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:18.782 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:18.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:18.883 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:18.901 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:14:18 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:14:18] "GET /metrics HTTP/1.1" 200 223867 "" "Prometheus/2.43.0" 2024-09-11T08:14:18.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:18.925 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:18.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:19.026 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:19.058 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:19.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:19.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:19.160 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:19.195 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:19.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:19.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:19.224 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:18 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/417962569' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:14:19.224 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:18 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:14:19.225 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:18 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:14:19.225 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:18 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:14:19.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:18 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/417962569' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:14:19.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:18 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:14:19.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:18 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:14:19.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:18 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:14:19.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:18 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/417962569' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:14:19.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:18 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:14:19.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:18 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:14:19.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:18 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:14:19.287 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-11T08:14:19.287 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-11T08:14:19.297 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:19.329 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:19.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:19.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:19.341 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:14:19.406 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 8 16 24034 24018 187.594 164.125 0.00430282 0.00532455 2024-09-11T08:14:19.432 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:19.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:19.471 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:19.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:19.572 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:14:19.617 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:19.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:19.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:14:19.719 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:19.754 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:19.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:19.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:19.855 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:14:19.862 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T08:14:19.862 INFO:teuthology.orchestra.run.smithi181.stderr:0 2024-09-11T08:14:19.862 INFO:teuthology.orchestra.run.smithi181.stderr:1 2024-09-11T08:14:19.862 INFO:teuthology.orchestra.run.smithi181.stderr:2 2024-09-11T08:14:19.862 INFO:teuthology.orchestra.run.smithi181.stderr:abort 2024-09-11T08:14:19.862 INFO:teuthology.orchestra.run.smithi181.stderr:assert 2024-09-11T08:14:19.862 INFO:teuthology.orchestra.run.smithi181.stderr:config diff 2024-09-11T08:14:19.863 INFO:teuthology.orchestra.run.smithi181.stderr:config diff get 2024-09-11T08:14:19.863 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T08:14:19.863 INFO:teuthology.orchestra.run.smithi181.stderr:config help [] 2024-09-11T08:14:19.863 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T08:14:19.863 INFO:teuthology.orchestra.run.smithi181.stderr:admin_socket: invalid command 2024-09-11T08:14:19.891 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:19.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:19.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:14:19.993 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:14:20.034 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:20.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:20.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:14:20.135 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:14:20.176 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:20.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:20.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:14:20.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:19 smithi150 ceph-mon[39612]: pgmap v281: 148 pgs: 8 active+undersized, 8 active+undersized+degraded, 132 active+clean; 1.2 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 141 MiB/s wr, 2.26k op/s; 3887/61296 objects degraded (6.341%) 2024-09-11T08:14:20.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:19 smithi150 ceph-mon[39612]: Health check update: Degraded data redundancy: 3887/61296 objects degraded (6.341%), 8 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-09-11T08:14:20.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:19 smithi150 ceph-mon[39612]: osdmap e166: 8 total, 7 up, 8 in 2024-09-11T08:14:20.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:19 smithi150 ceph-mon[29276]: pgmap v281: 148 pgs: 8 active+undersized, 8 active+undersized+degraded, 132 active+clean; 1.2 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 141 MiB/s wr, 2.26k op/s; 3887/61296 objects degraded (6.341%) 2024-09-11T08:14:20.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:19 smithi150 ceph-mon[29276]: Health check update: Degraded data redundancy: 3887/61296 objects degraded (6.341%), 8 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-09-11T08:14:20.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:19 smithi150 ceph-mon[29276]: osdmap e166: 8 total, 7 up, 8 in 2024-09-11T08:14:20.279 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:20.313 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:20.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:20.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:20.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:19 smithi181 ceph-mon[32701]: pgmap v281: 148 pgs: 8 active+undersized, 8 active+undersized+degraded, 132 active+clean; 1.2 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 141 MiB/s wr, 2.26k op/s; 3887/61296 objects degraded (6.341%) 2024-09-11T08:14:20.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:19 smithi181 ceph-mon[32701]: Health check update: Degraded data redundancy: 3887/61296 objects degraded (6.341%), 8 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-09-11T08:14:20.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:19 smithi181 ceph-mon[32701]: osdmap e166: 8 total, 7 up, 8 in 2024-09-11T08:14:20.406 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 9 16 26566 26550 184.333 158.25 0.00462936 0.0054203 2024-09-11T08:14:20.415 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:20.436 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:20.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:20.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:20.510 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T08:14:20.511 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-7, ['dump_ops_in_flight'] 2024-09-11T08:14:20.537 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:20.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:20.572 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:20.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:20.673 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:20.709 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:20.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:20.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:20.811 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:14:20.840 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:20.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:20.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:14:20.941 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:20.973 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:20.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:20.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:21.075 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:14:21.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:21.118 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:21.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:14:21.219 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:21.252 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:21.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:21.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:21.288 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T08:14:21.354 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:21.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:21.399 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:21.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:21.407 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 10 16 28552 28536 178.304 124.125 0.00629921 0.00560288 2024-09-11T08:14:21.499 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:21.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:21.544 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:21.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:21.612 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:21 smithi150 ceph-mon[29276]: pgmap v283: 148 pgs: 8 active+undersized, 8 active+undersized+degraded, 132 active+clean; 1.2 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 143 MiB/s wr, 2.30k op/s; 3887/61296 objects degraded (6.341%) 2024-09-11T08:14:21.613 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:21 smithi150 ceph-mon[39612]: pgmap v283: 148 pgs: 8 active+undersized, 8 active+undersized+degraded, 132 active+clean; 1.2 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 143 MiB/s wr, 2.30k op/s; 3887/61296 objects degraded (6.341%) 2024-09-11T08:14:21.646 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:14:21.676 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-09-11T08:14:21.676 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:14:21.685 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:21.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:21.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:14:21.781 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:21 smithi181 ceph-mon[32701]: pgmap v283: 148 pgs: 8 active+undersized, 8 active+undersized+degraded, 132 active+clean; 1.2 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 143 MiB/s wr, 2.30k op/s; 3887/61296 objects degraded (6.341%) 2024-09-11T08:14:21.787 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:21.816 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:21.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:21.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:21.918 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:14:21.953 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:21.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:21.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:14:22.055 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:22.073 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:14:21 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:14:21.778+0000 7f3164351440 -1 osd.7 139 log_to_monitors true 2024-09-11T08:14:22.093 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:22.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:22.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:22.195 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:22.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:22.241 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:22.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:22.343 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:22.380 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:22.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:22.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:22.407 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 11 16 30817 30801 174.963 141.562 0.00554754 0.00571104 2024-09-11T08:14:22.483 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:22.512 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:22.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:22.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:22.614 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:22.636 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:22.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:22.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:22.737 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:22.773 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:22.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:22.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:22.875 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:22.911 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:22.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:22.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:22.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:22 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:14:22.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:22 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:14:22.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:22 smithi150 ceph-mon[29276]: from='osd.7 v2:172.21.15.181:6812/2706455857' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-11T08:14:22.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:22 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:14:22.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:22 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:14:22.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:22 smithi150 ceph-mon[39612]: from='osd.7 v2:172.21.15.181:6812/2706455857' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-11T08:14:23.014 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:14:23.035 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:23.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:23.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:14:23.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:22 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:14:23.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:22 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:14:23.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:22 smithi181 ceph-mon[32701]: from='osd.7 v2:172.21.15.181:6812/2706455857' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-11T08:14:23.073 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:14:22 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:14:22.770+0000 7f31588f6640 -1 osd.7 139 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T08:14:23.136 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:23.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:23.166 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:23.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:23.268 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:23.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:23.299 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:23.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:23.400 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:23.407 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 12 15 32607 32592 169.711 111.938 0.00775847 0.00588726 2024-09-11T08:14:23.429 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:23.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:23.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:23.532 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:23.559 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:23.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:23.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:23.661 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:23.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:23.692 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:23.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:23.793 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:23.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:23.829 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:23.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:23.931 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:14:23.953 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:23.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:23.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:14:24.055 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:24.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:23 smithi181 ceph-mon[32701]: from='osd.7 v2:172.21.15.181:6812/2706455857' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-11T08:14:24.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:23 smithi181 ceph-mon[32701]: osdmap e167: 8 total, 7 up, 8 in 2024-09-11T08:14:24.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:23 smithi181 ceph-mon[32701]: from='osd.7 v2:172.21.15.181:6812/2706455857' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-09-11T08:14:24.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:23 smithi181 ceph-mon[32701]: pgmap v285: 148 pgs: 8 active+undersized, 8 active+undersized+degraded, 132 active+clean; 1.5 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 645 B/s rd, 149 MiB/s wr, 2.38k op/s; 5372/74346 objects degraded (7.226%) 2024-09-11T08:14:24.080 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:24.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:24.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:24.183 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:24.208 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:24.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:24.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:24.231 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:23 smithi150 ceph-mon[39612]: from='osd.7 v2:172.21.15.181:6812/2706455857' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-11T08:14:24.231 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:23 smithi150 ceph-mon[39612]: osdmap e167: 8 total, 7 up, 8 in 2024-09-11T08:14:24.231 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:23 smithi150 ceph-mon[39612]: from='osd.7 v2:172.21.15.181:6812/2706455857' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-09-11T08:14:24.232 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:23 smithi150 ceph-mon[39612]: pgmap v285: 148 pgs: 8 active+undersized, 8 active+undersized+degraded, 132 active+clean; 1.5 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 645 B/s rd, 149 MiB/s wr, 2.38k op/s; 5372/74346 objects degraded (7.226%) 2024-09-11T08:14:24.232 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:23 smithi150 ceph-mon[29276]: from='osd.7 v2:172.21.15.181:6812/2706455857' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-11T08:14:24.232 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:23 smithi150 ceph-mon[29276]: osdmap e167: 8 total, 7 up, 8 in 2024-09-11T08:14:24.233 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:23 smithi150 ceph-mon[29276]: from='osd.7 v2:172.21.15.181:6812/2706455857' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-09-11T08:14:24.233 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:23 smithi150 ceph-mon[29276]: pgmap v285: 148 pgs: 8 active+undersized, 8 active+undersized+degraded, 132 active+clean; 1.5 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 645 B/s rd, 149 MiB/s wr, 2.38k op/s; 5372/74346 objects degraded (7.226%) 2024-09-11T08:14:24.310 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:24.332 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:24.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:24.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:24.407 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 13 16 33314 33298 160.051 44.125 0.00210341 0.00595511 2024-09-11T08:14:24.434 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:24.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:24.457 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:24.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:24.559 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:24.586 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:24.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:24.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:24.688 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:24.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:24.713 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:24.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:24.814 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:24.840 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:24.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:24.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:24.942 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:24.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:24.966 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:24.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:24.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:24 smithi150 ceph-mon[29276]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-11T08:14:24.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:24 smithi150 ceph-mon[29276]: osd.7 v2:172.21.15.181:6812/2706455857 boot 2024-09-11T08:14:24.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:24 smithi150 ceph-mon[29276]: osdmap e168: 8 total, 8 up, 8 in 2024-09-11T08:14:24.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:24 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-11T08:14:24.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:24 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:14:24.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:24 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:14:24.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:24 smithi150 ceph-mon[39612]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-11T08:14:24.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:24 smithi150 ceph-mon[39612]: osd.7 v2:172.21.15.181:6812/2706455857 boot 2024-09-11T08:14:24.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:24 smithi150 ceph-mon[39612]: osdmap e168: 8 total, 8 up, 8 in 2024-09-11T08:14:24.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:24 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-11T08:14:24.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:24 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:14:24.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:24 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:14:25.068 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:25.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:24 smithi181 ceph-mon[32701]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-11T08:14:25.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:24 smithi181 ceph-mon[32701]: osd.7 v2:172.21.15.181:6812/2706455857 boot 2024-09-11T08:14:25.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:24 smithi181 ceph-mon[32701]: osdmap e168: 8 total, 8 up, 8 in 2024-09-11T08:14:25.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:24 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-11T08:14:25.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:24 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:14:25.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:24 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:14:25.094 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:25.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:25.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:25.196 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:25.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:25.224 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:25.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:25.326 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:25.351 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:25.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:25.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:25.407 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 14 16 33314 33298 148.62 0 - 0.00595511 2024-09-11T08:14:25.453 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:25.480 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:25.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:25.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:25.511 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-11T08:14:25.538 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:14:25 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:14:25] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:14:25.581 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:25.604 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:25.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:25.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:25.706 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:25.732 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:25.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:25.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:25.768 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:14:25.834 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:25.861 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:25.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:25.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:25.963 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:14:25.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:25.999 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:25.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:14:26.058 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:25 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:14:26.058 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:25 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:14:26.058 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:25 smithi181 ceph-mon[32701]: osdmap e169: 8 total, 8 up, 8 in 2024-09-11T08:14:26.058 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:25 smithi181 ceph-mon[32701]: pgmap v288: 148 pgs: 8 active+undersized, 8 active+undersized+degraded, 132 active+clean; 1.9 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 866 B/s rd, 122 MiB/s wr, 1.95k op/s; 6084/95823 objects degraded (6.349%) 2024-09-11T08:14:26.058 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:25 smithi181 ceph-mon[32701]: Health check update: Degraded data redundancy: 5372/74346 objects degraded (7.226%), 8 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-09-11T08:14:26.101 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:26.129 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:26.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:26.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:26.231 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:14:26.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:25 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:14:26.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:25 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:14:26.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:25 smithi150 ceph-mon[29276]: osdmap e169: 8 total, 8 up, 8 in 2024-09-11T08:14:26.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:25 smithi150 ceph-mon[29276]: pgmap v288: 148 pgs: 8 active+undersized, 8 active+undersized+degraded, 132 active+clean; 1.9 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 866 B/s rd, 122 MiB/s wr, 1.95k op/s; 6084/95823 objects degraded (6.349%) 2024-09-11T08:14:26.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:25 smithi150 ceph-mon[29276]: Health check update: Degraded data redundancy: 5372/74346 objects degraded (7.226%), 8 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-09-11T08:14:26.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:25 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:14:26.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:25 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:14:26.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:25 smithi150 ceph-mon[39612]: osdmap e169: 8 total, 8 up, 8 in 2024-09-11T08:14:26.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:25 smithi150 ceph-mon[39612]: pgmap v288: 148 pgs: 8 active+undersized, 8 active+undersized+degraded, 132 active+clean; 1.9 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 866 B/s rd, 122 MiB/s wr, 1.95k op/s; 6084/95823 objects degraded (6.349%) 2024-09-11T08:14:26.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:25 smithi150 ceph-mon[39612]: Health check update: Degraded data redundancy: 5372/74346 objects degraded (7.226%), 8 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-09-11T08:14:26.267 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:26.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:26.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:14:26.369 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:26.400 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:26.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:26.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:26.407 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 15 16 34095 34079 141.967 24.4062 0.00289098 0.0070362 2024-09-11T08:14:26.502 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:26.517 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-11T08:14:26.542 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:26.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:26.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:26.572 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:14:26.644 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:26.684 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:26.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:26.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:26.786 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:26.817 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:26.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:26.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:26.919 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:26.955 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:26.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:26.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:27.057 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:27.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:26 smithi181 ceph-mon[32701]: osdmap e170: 8 total, 8 up, 8 in 2024-09-11T08:14:27.115 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:27.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:27.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:27.143 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:26 smithi150 ceph-mon[29276]: osdmap e170: 8 total, 8 up, 8 in 2024-09-11T08:14:27.144 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:26 smithi150 ceph-mon[39612]: osdmap e170: 8 total, 8 up, 8 in 2024-09-11T08:14:27.217 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:27.257 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:27.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:27.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:27.359 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:27.397 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:27.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:27.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:27.407 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 16 16 35754 35738 139.573 103.688 0.00324588 0.00715713 2024-09-11T08:14:27.499 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:27.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:27.543 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:27.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:27.644 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:27.660 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-11T08:14:27.661 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-09-11T08:14:27.661 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:14:27.661 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.15087.0:36115 8.6 e170/169 8:6311bf2f:::benchmark_data_smithi150_107646_object36114:head v 170'2252, mlcod=164'105)", 2024-09-11T08:14:27.661 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:14:27.651320+0000", 2024-09-11T08:14:27.661 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.0073143059999999996, 2024-09-11T08:14:27.661 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0073504440000000002, 2024-09-11T08:14:27.661 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:14:27.661 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-11T08:14:27.661 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:14:27.661 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:14:27.661 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:14:27.661 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:27.651320+0000", 2024-09-11T08:14:27.661 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:14:27.661 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:14:27.662 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:14:27.662 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:14:27.662 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:27.651320+0000", 2024-09-11T08:14:27.662 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:14:27.662 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:14:27.662 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:14:27.662 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:14:27.662 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:27.651319+0000", 2024-09-11T08:14:27.662 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-09-11T08:14:27.662 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:14:27.662 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:14:27.662 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:14:27.662 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:27.651374+0000", 2024-09-11T08:14:27.662 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5288999999999999e-05 2024-09-11T08:14:27.663 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:14:27.663 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:14:27.663 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:14:27.663 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:27.651388+0000", 2024-09-11T08:14:27.663 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3468e-05 2024-09-11T08:14:27.663 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:14:27.663 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:14:27.663 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:14:27.663 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:27.651393+0000", 2024-09-11T08:14:27.663 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9110000000000001e-06 2024-09-11T08:14:27.663 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:14:27.663 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:14:27.663 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:14:27.664 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:27.655114+0000", 2024-09-11T08:14:27.664 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0037210239999999999 2024-09-11T08:14:27.664 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:14:27.664 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:14:27.664 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:14:27.664 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:27.655156+0000", 2024-09-11T08:14:27.664 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2469999999999998e-05 2024-09-11T08:14:27.664 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:14:27.664 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:14:27.664 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:14:27.664 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:14:27.664 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:14:27.664 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDPGPush(8.8 170/169 [PushOp(8:18f9aae7:::benchmark_data_smithi150_107646_object2041:head, version: 164'134, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:18f9aae7:::benchmark_data_smithi150_107646_object2041:head@164'134, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-11T08:14:27.665 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:14:27.656256+0000", 2024-09-11T08:14:27.665 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.0023780479999999998, 2024-09-11T08:14:27.665 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0025079130000000001, 2024-09-11T08:14:27.665 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:14:27.665 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-11T08:14:27.665 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:14:27.665 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:14:27.665 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:14:27.665 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:27.656256+0000", 2024-09-11T08:14:27.665 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:14:27.665 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:14:27.665 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:14:27.665 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:14:27.665 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:27.656256+0000", 2024-09-11T08:14:27.666 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:14:27.666 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:14:27.666 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:14:27.666 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:14:27.666 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:27.656255+0000", 2024-09-11T08:14:27.666 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:14:27.666 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:14:27.666 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:14:27.666 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:14:27.666 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:27.656290+0000", 2024-09-11T08:14:27.666 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4860999999999997e-05 2024-09-11T08:14:27.666 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:14:27.666 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:14:27.666 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:14:27.666 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:27.656313+0000", 2024-09-11T08:14:27.667 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2823000000000002e-05 2024-09-11T08:14:27.667 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:14:27.667 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:14:27.667 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:14:27.667 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:27.656324+0000", 2024-09-11T08:14:27.667 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1178999999999999e-05 2024-09-11T08:14:27.667 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:14:27.667 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:14:27.667 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:14:27.667 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:27.657330+0000", 2024-09-11T08:14:27.667 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001006095 2024-09-11T08:14:27.667 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:14:27.667 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:14:27.667 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:14:27.668 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:27.657353+0000", 2024-09-11T08:14:27.668 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2996000000000001e-05 2024-09-11T08:14:27.668 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:14:27.668 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:14:27.668 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:14:27.668 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:14:27.668 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:14:27.668 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.15087.0:36129 8.6 e170/169)", 2024-09-11T08:14:27.668 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:14:27.657096+0000", 2024-09-11T08:14:27.668 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.001538071, 2024-09-11T08:14:27.668 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0017193499999999999, 2024-09-11T08:14:27.668 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:14:27.668 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "queued for pg", 2024-09-11T08:14:27.668 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:14:27.669 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:14:27.669 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:14:27.669 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:27.657096+0000", 2024-09-11T08:14:27.669 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:14:27.669 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:14:27.669 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:14:27.669 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:14:27.669 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:27.657096+0000", 2024-09-11T08:14:27.669 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:14:27.669 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:14:27.669 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:14:27.669 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:14:27.669 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:27.657095+0000", 2024-09-11T08:14:27.669 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:14:27.670 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:14:27.670 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:14:27.670 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:14:27.670 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:27.657141+0000", 2024-09-11T08:14:27.670 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5547999999999998e-05 2024-09-11T08:14:27.670 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:14:27.670 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:14:27.670 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:14:27.670 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:27.657153+0000", 2024-09-11T08:14:27.670 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.276e-05 2024-09-11T08:14:27.670 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:14:27.670 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:14:27.670 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:14:27.670 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:27.657159+0000", 2024-09-11T08:14:27.670 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7320000000000003e-06 2024-09-11T08:14:27.671 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:14:27.671 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:14:27.671 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:14:27.671 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:14:27.671 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:14:27.671 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.15087.0:36131 8.8 e170/169)", 2024-09-11T08:14:27.671 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:14:27.658022+0000", 2024-09-11T08:14:27.671 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.00061174299999999999, 2024-09-11T08:14:27.671 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00082963500000000001, 2024-09-11T08:14:27.671 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:14:27.671 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-11T08:14:27.671 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:14:27.671 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:14:27.671 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:14:27.672 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:27.658022+0000", 2024-09-11T08:14:27.672 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:14:27.672 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:14:27.672 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:14:27.672 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:14:27.672 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:27.658022+0000", 2024-09-11T08:14:27.672 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:14:27.672 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:14:27.672 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:14:27.672 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:14:27.672 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:27.658022+0000", 2024-09-11T08:14:27.672 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-09-11T08:14:27.672 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:14:27.672 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:14:27.673 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:14:27.673 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:27.658044+0000", 2024-09-11T08:14:27.673 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2107999999999999e-05 2024-09-11T08:14:27.673 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:14:27.673 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:14:27.673 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:14:27.673 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:27.658066+0000", 2024-09-11T08:14:27.673 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1897999999999999e-05 2024-09-11T08:14:27.673 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:14:27.673 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:14:27.673 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:14:27.673 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:27.658071+0000", 2024-09-11T08:14:27.673 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3920000000000002e-06 2024-09-11T08:14:27.673 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:14:27.674 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:14:27.674 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:14:27.674 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:27.658838+0000", 2024-09-11T08:14:27.674 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00076740900000000004 2024-09-11T08:14:27.674 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:14:27.674 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:14:27.674 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:14:27.674 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:14:27.674 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-09-11T08:14:27.674 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 4 2024-09-11T08:14:27.674 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-11T08:14:27.685 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:27.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:27.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:27.787 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:14:27.815 INFO:teuthology.orchestra.run.smithi150.stderr:nodeep-scrub is set 2024-09-11T08:14:27.829 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:27.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:27.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:14:27.931 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:14:27.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:27.955 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:27.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:14:28.056 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:28.074 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:14:28 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:14:28.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:27 smithi181 ceph-mon[32701]: pgmap v290: 148 pgs: 8 active+undersized, 8 active+undersized+degraded, 132 active+clean; 1.9 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 110 MiB/s wr, 1.75k op/s; 6084/95823 objects degraded (6.349%) 2024-09-11T08:14:28.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:27 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2638236312' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:14:28.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:27 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:14:28.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:28.093 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:28.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:28.194 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:28.229 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:28.229 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:27 smithi150 ceph-mon[29276]: pgmap v290: 148 pgs: 8 active+undersized, 8 active+undersized+degraded, 132 active+clean; 1.9 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 110 MiB/s wr, 1.75k op/s; 6084/95823 objects degraded (6.349%) 2024-09-11T08:14:28.229 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:27 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2638236312' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:14:28.229 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:27 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:14:28.230 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:27 smithi150 ceph-mon[39612]: pgmap v290: 148 pgs: 8 active+undersized, 8 active+undersized+degraded, 132 active+clean; 1.9 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 110 MiB/s wr, 1.75k op/s; 6084/95823 objects degraded (6.349%) 2024-09-11T08:14:28.230 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:27 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2638236312' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:14:28.230 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:27 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:14:28.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:28.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:28.331 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:28.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:28.362 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:28.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:28.407 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 17 16 37359 37343 137.263 100.312 0.00336334 0.00727907 2024-09-11T08:14:28.463 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:28.497 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:28.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:28.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:28.599 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:28.635 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:28.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:28.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:28.737 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:14:28.784 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:28.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:28.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:14:28.868 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:28 smithi150 ceph-mon[39612]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:14:28.868 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:28 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:14:28.868 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:28 smithi150 ceph-mon[39612]: osdmap e171: 8 total, 8 up, 8 in 2024-09-11T08:14:28.868 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:28 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:14:28.868 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:28 smithi150 ceph-mon[29276]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:14:28.868 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:28 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:14:28.869 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:28 smithi150 ceph-mon[29276]: osdmap e171: 8 total, 8 up, 8 in 2024-09-11T08:14:28.869 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:28 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:14:28.869 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:14:28 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:14:28] "GET /metrics HTTP/1.1" 200 224065 "" "Prometheus/2.43.0" 2024-09-11T08:14:28.869 INFO:journalctl@ceph.rgw.foo.a.smithi150.stdout:Sep 11 08:14:28 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-rgw-foo-a[65759]: 2024-09-11T08:14:28.634+0000 7fb971ffb640 -1 rgw watcher librados: RGWWatcher::handle_error cookie 94783556104128 err (110) Connection timed out 2024-09-11T08:14:28.886 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:14:28.909 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:28.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:28.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:14:29.010 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:29.044 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:29.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:29.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:29.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:28 smithi181 ceph-mon[32701]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:14:29.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:28 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:14:29.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:28 smithi181 ceph-mon[32701]: osdmap e171: 8 total, 8 up, 8 in 2024-09-11T08:14:29.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:28 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:14:29.146 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:29.184 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:29.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:29.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:29.220 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:14:29.286 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:29.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:29.322 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:29.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:29.408 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 18 16 38914 38898 135.037 97.1875 0.00317142 0.0073912 2024-09-11T08:14:29.422 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:29.460 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:29.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:29.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:29.562 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:14:29.599 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:29.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:29.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:14:29.701 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:29.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:29.736 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:29.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:29.837 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:29.872 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:29.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:29.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:29.974 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:30.009 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:30.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:30.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:30.043 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:29 smithi150 ceph-mon[29276]: pgmap v292: 148 pgs: 3 active+recovering+undersized+degraded+remapped, 145 active+clean; 2.2 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 59 MiB/s wr, 951 op/s; 5961/110307 objects degraded (5.404%); 5.1 MiB/s, 81 objects/s recovering 2024-09-11T08:14:30.043 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:29 smithi150 ceph-mon[39612]: pgmap v292: 148 pgs: 3 active+recovering+undersized+degraded+remapped, 145 active+clean; 2.2 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 59 MiB/s wr, 951 op/s; 5961/110307 objects degraded (5.404%); 5.1 MiB/s, 81 objects/s recovering 2024-09-11T08:14:30.057 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:29 smithi181 ceph-mon[32701]: pgmap v292: 148 pgs: 3 active+recovering+undersized+degraded+remapped, 145 active+clean; 2.2 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 59 MiB/s wr, 951 op/s; 5961/110307 objects degraded (5.404%); 5.1 MiB/s, 81 objects/s recovering 2024-09-11T08:14:30.105 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd reweight 0 1 2024-09-11T08:14:30.110 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:30.188 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:30.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:30.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:30.290 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:30.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:30.329 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:30.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:30.408 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 19 16 40498 40482 133.139 99 0.00501763 0.00750359 2024-09-11T08:14:30.431 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:30.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:30.472 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:30.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:30.573 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:14:30.617 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:30.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:30.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:14:30.722 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:30.759 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:30.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:30.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:30.860 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:14:30.895 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:30.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:30.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:14:30.997 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:14:31.034 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:31.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:31.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:14:31.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:30 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/791185005' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:14:31.094 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:30 smithi150 ceph-mon[29276]: Health check update: Degraded data redundancy: 5961/110307 objects degraded (5.404%), 3 pgs degraded (PG_DEGRADED) 2024-09-11T08:14:31.094 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:30 smithi150 ceph-mon[29276]: osdmap e172: 8 total, 8 up, 8 in 2024-09-11T08:14:31.094 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:14:31.094 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:14:31.094 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:14:31.094 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:14:31.094 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:14:31.094 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:14:31.094 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:14:31.095 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:14:31.095 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:14:31.095 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:14:31.095 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:14:31.095 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:14:31.095 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:14:31.095 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:14:31.095 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:14:31.095 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:14:31.095 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:30 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/791185005' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:14:31.095 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:30 smithi150 ceph-mon[39612]: Health check update: Degraded data redundancy: 5961/110307 objects degraded (5.404%), 3 pgs degraded (PG_DEGRADED) 2024-09-11T08:14:31.096 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:30 smithi150 ceph-mon[39612]: osdmap e172: 8 total, 8 up, 8 in 2024-09-11T08:14:31.136 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:31.170 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:31.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:31.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:31.259 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T08:14:31.272 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:14:31.306 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:31.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:31.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:14:31.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:30 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/791185005' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:14:31.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:30 smithi181 ceph-mon[32701]: Health check update: Degraded data redundancy: 5961/110307 objects degraded (5.404%), 3 pgs degraded (PG_DEGRADED) 2024-09-11T08:14:31.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:30 smithi181 ceph-mon[32701]: osdmap e172: 8 total, 8 up, 8 in 2024-09-11T08:14:31.408 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:2024-09-11T08:14:31.408600+0000 min lat: 0.0010471 max lat: 2.12372 avg lat: 0.00758897 2024-09-11T08:14:31.408 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-11T08:14:31.408 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 20 16 42160 42144 131.675 103.875 0.0139998 0.00758897 2024-09-11T08:14:31.408 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:31.448 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:31.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:31.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:31.550 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:31.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:31.585 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:31.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:31.686 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:14:31.726 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:31.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:31.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:14:31.828 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:14:31.869 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:31.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:31.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:14:31.972 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:31.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:31 smithi150 ceph-mon[39612]: pgmap v294: 148 pgs: 3 active+recovering+undersized+degraded+remapped, 145 active+clean; 2.2 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 50 MiB/s wr, 804 op/s; 5961/110307 objects degraded (5.404%); 4.3 MiB/s, 68 objects/s recovering 2024-09-11T08:14:31.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:31 smithi150 ceph-mon[29276]: pgmap v294: 148 pgs: 3 active+recovering+undersized+degraded+remapped, 145 active+clean; 2.2 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 50 MiB/s wr, 804 op/s; 5961/110307 objects degraded (5.404%); 4.3 MiB/s, 68 objects/s recovering 2024-09-11T08:14:32.005 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:32.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:32.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:32.107 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:32.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:32.148 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:32.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:32.249 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:32.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:32.291 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:32.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:32.302 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:14:32.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:31 smithi181 ceph-mon[32701]: pgmap v294: 148 pgs: 3 active+recovering+undersized+degraded+remapped, 145 active+clean; 2.2 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 50 MiB/s wr, 804 op/s; 5961/110307 objects degraded (5.404%); 4.3 MiB/s, 68 objects/s recovering 2024-09-11T08:14:32.327 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-11T08:14:32.327 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 20, 2024-09-11T08:14:32.327 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 600, 2024-09-11T08:14:32.327 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [ 2024-09-11T08:14:32.327 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.328 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:107976 7.3 7:c68b5db5:::benchmark_data_smithi150_78508_object107975:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-11T08:14:32.328 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:10.905908+0000", 2024-09-11T08:14:32.328 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 201.413260251, 2024-09-11T08:14:32.328 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.13184691800000001, 2024-09-11T08:14:32.328 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:14:32.328 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:14:32.328 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:14:32.329 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:14:32.329 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:14:32.329 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 107976 2024-09-11T08:14:32.329 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.329 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:14:32.329 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.329 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:14:32.329 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.905908+0000", 2024-09-11T08:14:32.330 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:14:32.330 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.330 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.330 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:14:32.330 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.905908+0000", 2024-09-11T08:14:32.330 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:14:32.331 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.331 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.331 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:14:32.331 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.905908+0000", 2024-09-11T08:14:32.331 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:14:32.331 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.331 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.332 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:14:32.332 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.905927+0000", 2024-09-11T08:14:32.332 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.9598000000000001e-05 2024-09-11T08:14:32.332 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.332 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.333 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:14:32.333 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.905933+0000", 2024-09-11T08:14:32.333 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.9920000000000004e-06 2024-09-11T08:14:32.333 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.333 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.334 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:14:32.334 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.905937+0000", 2024-09-11T08:14:32.334 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.4120000000000001e-06 2024-09-11T08:14:32.334 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.335 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.335 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:14:32.335 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.910154+0000", 2024-09-11T08:14:32.335 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0042179319999999998 2024-09-11T08:14:32.335 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.335 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.336 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:14:32.336 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.910266+0000", 2024-09-11T08:14:32.336 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000111703 2024-09-11T08:14:32.336 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.336 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.336 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 2,4", 2024-09-11T08:14:32.337 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.910399+0000", 2024-09-11T08:14:32.337 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000132442 2024-09-11T08:14:32.337 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.337 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.338 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:14:32.339 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.915798+0000", 2024-09-11T08:14:32.339 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0053989679999999996 2024-09-11T08:14:32.339 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.339 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.339 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:14:32.339 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.915819+0000", 2024-09-11T08:14:32.340 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.1834000000000001e-05 2024-09-11T08:14:32.340 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.340 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.340 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:14:32.340 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.037586+0000", 2024-09-11T08:14:32.340 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.121766551 2024-09-11T08:14:32.341 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.341 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.341 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:14:32.341 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.037714+0000", 2024-09-11T08:14:32.341 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012809400000000001 2024-09-11T08:14:32.341 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.341 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.342 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:14:32.342 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.037755+0000", 2024-09-11T08:14:32.342 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.0873e-05 2024-09-11T08:14:32.342 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:14:32.342 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:14:32.342 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:14:32.342 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.342 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.343 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:107979 7.2 7:40eeebfe:::benchmark_data_smithi150_78508_object107978:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-11T08:14:32.343 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:10.911896+0000", 2024-09-11T08:14:32.343 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 201.407272105, 2024-09-11T08:14:32.343 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.11813148900000001, 2024-09-11T08:14:32.343 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:14:32.343 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:14:32.343 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:14:32.344 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:14:32.344 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:14:32.344 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 107979 2024-09-11T08:14:32.344 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.344 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:14:32.344 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.344 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:14:32.344 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.911896+0000", 2024-09-11T08:14:32.345 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:14:32.345 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.345 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.345 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:14:32.345 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.911896+0000", 2024-09-11T08:14:32.345 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:14:32.345 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.345 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.346 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:14:32.346 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.911894+0000", 2024-09-11T08:14:32.346 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:14:32.346 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.346 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.346 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:14:32.347 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.911918+0000", 2024-09-11T08:14:32.347 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.3295e-05 2024-09-11T08:14:32.347 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.347 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.347 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:14:32.347 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.911926+0000", 2024-09-11T08:14:32.347 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.5550000000000003e-06 2024-09-11T08:14:32.348 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.348 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.348 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:14:32.348 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.911931+0000", 2024-09-11T08:14:32.348 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.9609999999999996e-06 2024-09-11T08:14:32.348 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.349 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.349 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:14:32.349 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.911998+0000", 2024-09-11T08:14:32.349 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.6533999999999997e-05 2024-09-11T08:14:32.349 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.349 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.350 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:14:32.350 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.912108+0000", 2024-09-11T08:14:32.350 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000110637 2024-09-11T08:14:32.351 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.351 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.351 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 1,2", 2024-09-11T08:14:32.351 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.912236+0000", 2024-09-11T08:14:32.351 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012750500000000001 2024-09-11T08:14:32.351 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.351 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.352 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:14:32.352 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.915433+0000", 2024-09-11T08:14:32.352 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0031969540000000001 2024-09-11T08:14:32.352 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.352 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.352 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:14:32.353 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.029826+0000", 2024-09-11T08:14:32.353 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.11439272 2024-09-11T08:14:32.353 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.353 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.353 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:14:32.353 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.029947+0000", 2024-09-11T08:14:32.354 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012175700000000001 2024-09-11T08:14:32.354 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.354 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.354 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:14:32.354 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.029989+0000", 2024-09-11T08:14:32.354 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1233000000000003e-05 2024-09-11T08:14:32.355 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.355 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.355 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:14:32.355 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.030028+0000", 2024-09-11T08:14:32.355 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.9057999999999998e-05 2024-09-11T08:14:32.355 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:14:32.356 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:14:32.356 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:14:32.356 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.356 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.356 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:107984 7.a 7:5f828c6a:::benchmark_data_smithi150_78508_object107983:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-11T08:14:32.356 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:10.912870+0000", 2024-09-11T08:14:32.357 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 201.40629777199999, 2024-09-11T08:14:32.357 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.117220537, 2024-09-11T08:14:32.357 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:14:32.357 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:14:32.357 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:14:32.358 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:14:32.358 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:14:32.358 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 107984 2024-09-11T08:14:32.358 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.358 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:14:32.358 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.359 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:14:32.359 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.912870+0000", 2024-09-11T08:14:32.359 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:14:32.359 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.359 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.359 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:14:32.360 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.912870+0000", 2024-09-11T08:14:32.360 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:14:32.360 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.360 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.360 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:14:32.361 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.912869+0000", 2024-09-11T08:14:32.361 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:14:32.361 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.361 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.361 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:14:32.361 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.912893+0000", 2024-09-11T08:14:32.362 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.3264e-05 2024-09-11T08:14:32.362 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.362 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.362 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:14:32.362 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.912903+0000", 2024-09-11T08:14:32.362 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0557000000000001e-05 2024-09-11T08:14:32.363 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.363 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.363 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:14:32.363 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.912909+0000", 2024-09-11T08:14:32.363 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.8649999999999998e-06 2024-09-11T08:14:32.363 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.363 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.363 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:14:32.364 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.915007+0000", 2024-09-11T08:14:32.364 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0020981469999999999 2024-09-11T08:14:32.364 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.364 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.364 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:14:32.364 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.915120+0000", 2024-09-11T08:14:32.364 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000112891 2024-09-11T08:14:32.365 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.365 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.365 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 2,5", 2024-09-11T08:14:32.365 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.915248+0000", 2024-09-11T08:14:32.365 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012757299999999999 2024-09-11T08:14:32.365 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.366 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.366 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:14:32.366 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.029913+0000", 2024-09-11T08:14:32.366 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.114665409 2024-09-11T08:14:32.366 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.366 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.366 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:14:32.366 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.029939+0000", 2024-09-11T08:14:32.367 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.6202000000000001e-05 2024-09-11T08:14:32.367 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.367 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.367 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:14:32.367 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.030034+0000", 2024-09-11T08:14:32.367 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.5186999999999995e-05 2024-09-11T08:14:32.367 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.367 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.367 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:14:32.368 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.030064+0000", 2024-09-11T08:14:32.368 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.9859e-05 2024-09-11T08:14:32.368 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.368 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.368 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:14:32.368 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.030091+0000", 2024-09-11T08:14:32.369 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.6653e-05 2024-09-11T08:14:32.369 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:14:32.369 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:14:32.369 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:14:32.369 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.369 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.370 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.14790.0:107982 7.6 e122/91 7:665c1af9:::benchmark_data_smithi150_78508_object107981:head v 122'6801, mlcod=122'6799)", 2024-09-11T08:14:32.370 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:10.915094+0000", 2024-09-11T08:14:32.370 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 201.40407409700001, 2024-09-11T08:14:32.370 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.114003855, 2024-09-11T08:14:32.370 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:14:32.370 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:14:32.371 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:14:32.371 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.371 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:14:32.371 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.915094+0000", 2024-09-11T08:14:32.371 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:14:32.371 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.372 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.372 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:14:32.372 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.915094+0000", 2024-09-11T08:14:32.372 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:14:32.372 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.372 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.373 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:14:32.373 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.915094+0000", 2024-09-11T08:14:32.373 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:14:32.373 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.373 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.373 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:14:32.374 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.915116+0000", 2024-09-11T08:14:32.374 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.2938e-05 2024-09-11T08:14:32.374 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.374 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.374 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:14:32.374 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.915126+0000", 2024-09-11T08:14:32.375 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.5719999999999992e-06 2024-09-11T08:14:32.375 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.375 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.375 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:14:32.375 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.915131+0000", 2024-09-11T08:14:32.375 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.6e-06 2024-09-11T08:14:32.375 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.376 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.376 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:14:32.376 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.915190+0000", 2024-09-11T08:14:32.376 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.9357000000000001e-05 2024-09-11T08:14:32.376 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.377 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.377 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:14:32.377 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.915208+0000", 2024-09-11T08:14:32.377 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.8294000000000001e-05 2024-09-11T08:14:32.377 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.377 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.378 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:14:32.378 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.029054+0000", 2024-09-11T08:14:32.378 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.113845218 2024-09-11T08:14:32.378 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.378 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.378 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:14:32.379 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.029098+0000", 2024-09-11T08:14:32.379 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.4490000000000003e-05 2024-09-11T08:14:32.379 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:14:32.379 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:14:32.379 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:14:32.379 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.380 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.380 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:107985 7.3 7:c59c4586:::benchmark_data_smithi150_78508_object107984:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-11T08:14:32.380 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:10.916263+0000", 2024-09-11T08:14:32.380 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 201.402904941, 2024-09-11T08:14:32.380 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.121624965, 2024-09-11T08:14:32.380 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:14:32.381 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:14:32.381 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:14:32.381 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:14:32.382 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:14:32.382 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 107985 2024-09-11T08:14:32.382 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.382 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:14:32.382 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.382 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:14:32.382 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.916263+0000", 2024-09-11T08:14:32.383 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:14:32.383 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.383 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.383 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:14:32.383 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.916263+0000", 2024-09-11T08:14:32.383 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:14:32.384 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.384 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.384 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:14:32.384 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.916262+0000", 2024-09-11T08:14:32.384 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:14:32.384 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.385 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.385 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:14:32.385 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.916284+0000", 2024-09-11T08:14:32.386 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.2646e-05 2024-09-11T08:14:32.386 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.386 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.386 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:14:32.386 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.916372+0000", 2024-09-11T08:14:32.386 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.7496000000000004e-05 2024-09-11T08:14:32.387 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.387 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.387 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:14:32.387 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.916379+0000", 2024-09-11T08:14:32.387 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.2910000000000002e-06 2024-09-11T08:14:32.387 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.388 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.388 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:14:32.388 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.916412+0000", 2024-09-11T08:14:32.388 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.3021000000000001e-05 2024-09-11T08:14:32.388 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.388 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.389 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:14:32.389 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.916534+0000", 2024-09-11T08:14:32.389 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000122096 2024-09-11T08:14:32.389 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.389 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.389 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 2,4", 2024-09-11T08:14:32.390 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.916661+0000", 2024-09-11T08:14:32.390 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000126811 2024-09-11T08:14:32.390 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.390 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.390 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:14:32.391 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.032290+0000", 2024-09-11T08:14:32.391 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.115629047 2024-09-11T08:14:32.391 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.391 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.391 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:14:32.391 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.032314+0000", 2024-09-11T08:14:32.391 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.4219e-05 2024-09-11T08:14:32.391 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.391 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.392 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:14:32.392 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.037765+0000", 2024-09-11T08:14:32.392 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0054504150000000001 2024-09-11T08:14:32.392 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.392 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.392 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:14:32.392 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.037850+0000", 2024-09-11T08:14:32.392 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.4890999999999997e-05 2024-09-11T08:14:32.392 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.392 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.392 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:14:32.392 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.037888+0000", 2024-09-11T08:14:32.392 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.8578000000000002e-05 2024-09-11T08:14:32.392 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:14:32.393 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:14:32.393 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:14:32.393 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.393 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.393 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop_reply(client.14790.0:107979 7.2 e122/91 ondisk, result = 0)", 2024-09-11T08:14:32.393 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:10.916478+0000", 2024-09-11T08:14:32.393 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 201.402690057, 2024-09-11T08:14:32.393 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.113354038, 2024-09-11T08:14:32.393 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:14:32.393 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "started", 2024-09-11T08:14:32.393 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:14:32.393 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.393 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:14:32.394 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.916478+0000", 2024-09-11T08:14:32.394 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:14:32.394 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.394 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.394 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:14:32.394 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.916478+0000", 2024-09-11T08:14:32.394 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:14:32.394 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.394 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.394 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:14:32.394 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.916478+0000", 2024-09-11T08:14:32.394 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:14:32.394 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.395 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.395 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:14:32.395 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.916482+0000", 2024-09-11T08:14:32.395 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.8300000000000003e-06 2024-09-11T08:14:32.395 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.395 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.395 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:14:32.395 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.916490+0000", 2024-09-11T08:14:32.395 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.6979999999999998e-06 2024-09-11T08:14:32.395 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.395 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.395 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:14:32.395 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.916495+0000", 2024-09-11T08:14:32.395 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.5290000000000002e-06 2024-09-11T08:14:32.396 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.396 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.396 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:14:32.396 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.916753+0000", 2024-09-11T08:14:32.396 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00025800699999999998 2024-09-11T08:14:32.396 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.396 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.396 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:14:32.396 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.916759+0000", 2024-09-11T08:14:32.396 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.2360000000000001e-06 2024-09-11T08:14:32.397 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.397 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.397 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:14:32.397 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.029832+0000", 2024-09-11T08:14:32.397 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.113073362 2024-09-11T08:14:32.397 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:14:32.397 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:14:32.397 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:14:32.397 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.397 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.397 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop_reply(client.14790.0:107984 7.a e122/91 ondisk, result = 0)", 2024-09-11T08:14:32.398 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:10.919328+0000", 2024-09-11T08:14:32.398 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 201.39983996800001, 2024-09-11T08:14:32.398 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.11076470099999999, 2024-09-11T08:14:32.398 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:14:32.398 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "started", 2024-09-11T08:14:32.398 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:14:32.398 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.398 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:14:32.398 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.919328+0000", 2024-09-11T08:14:32.398 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:14:32.398 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.399 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.399 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:14:32.399 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.919328+0000", 2024-09-11T08:14:32.399 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:14:32.399 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.399 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.399 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:14:32.399 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.919327+0000", 2024-09-11T08:14:32.399 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:14:32.399 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.399 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.400 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:14:32.400 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.919334+0000", 2024-09-11T08:14:32.400 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.5490000000000003e-06 2024-09-11T08:14:32.400 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.400 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.400 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:14:32.400 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.029134+0000", 2024-09-11T08:14:32.400 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.10979998000000001 2024-09-11T08:14:32.400 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.400 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.400 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:14:32.400 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.029140+0000", 2024-09-11T08:14:32.400 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.5799999999999997e-06 2024-09-11T08:14:32.401 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.401 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.401 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:14:32.401 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.029930+0000", 2024-09-11T08:14:32.401 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00078996000000000003 2024-09-11T08:14:32.401 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.401 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.401 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:14:32.401 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.029935+0000", 2024-09-11T08:14:32.401 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.9259999999999999e-06 2024-09-11T08:14:32.401 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.401 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.401 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:14:32.402 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.030093+0000", 2024-09-11T08:14:32.402 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000157844 2024-09-11T08:14:32.402 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:14:32.402 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:14:32.402 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:14:32.402 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.402 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.402 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.14790.0:107989 7.6 e122/91 7:6298c77c:::benchmark_data_smithi150_78508_object107988:head v 122'6802, mlcod=122'6799)", 2024-09-11T08:14:32.402 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:10.920278+0000", 2024-09-11T08:14:32.402 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 201.39889067199999, 2024-09-11T08:14:32.402 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.116243732, 2024-09-11T08:14:32.402 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:14:32.402 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:14:32.403 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:14:32.403 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.403 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:14:32.403 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.920278+0000", 2024-09-11T08:14:32.403 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:14:32.403 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.403 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.403 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:14:32.403 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.920278+0000", 2024-09-11T08:14:32.403 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:14:32.403 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.403 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.404 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:14:32.404 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.920277+0000", 2024-09-11T08:14:32.404 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:14:32.404 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.404 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.404 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:14:32.404 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.920302+0000", 2024-09-11T08:14:32.404 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.4824000000000002e-05 2024-09-11T08:14:32.404 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.404 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.404 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:14:32.404 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.030110+0000", 2024-09-11T08:14:32.404 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.10980841500000001 2024-09-11T08:14:32.405 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.405 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.405 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:14:32.405 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.030116+0000", 2024-09-11T08:14:32.405 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.6319999999999996e-06 2024-09-11T08:14:32.405 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.405 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.405 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:14:32.405 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.030163+0000", 2024-09-11T08:14:32.405 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.7346999999999999e-05 2024-09-11T08:14:32.405 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.405 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.405 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:14:32.406 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.030182+0000", 2024-09-11T08:14:32.406 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.8726999999999999e-05 2024-09-11T08:14:32.406 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.406 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.406 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:14:32.406 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.036491+0000", 2024-09-11T08:14:32.406 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0063095959999999998 2024-09-11T08:14:32.406 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.406 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.406 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:14:32.406 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:32.407 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.036521+0000", 2024-09-11T08:14:32.407 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.0018999999999998e-05 2024-09-11T08:14:32.407 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:14:32.408 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:14:32.408 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:14:32.408 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.408 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.408 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop_reply(client.14790.0:107985 7.3 e122/91 ondisk, result = 0)", 2024-09-11T08:14:32.408 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:10.922258+0000", 2024-09-11T08:14:32.408 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 201.396910026, 2024-09-11T08:14:32.408 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.110033302, 2024-09-11T08:14:32.408 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:14:32.409 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "started", 2024-09-11T08:14:32.409 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:14:32.409 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.409 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:14:32.409 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.922258+0000", 2024-09-11T08:14:32.409 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:14:32.409 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.409 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.409 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:14:32.409 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.922258+0000", 2024-09-11T08:14:32.409 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:14:32.409 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.409 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.409 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:14:32.409 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.922257+0000", 2024-09-11T08:14:32.410 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:14:32.410 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.410 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.410 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:14:32.410 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.922264+0000", 2024-09-11T08:14:32.410 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.3890000000000002e-06 2024-09-11T08:14:32.410 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.410 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.410 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:14:32.410 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.028986+0000", 2024-09-11T08:14:32.410 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.106722281 2024-09-11T08:14:32.410 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.410 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.410 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:14:32.411 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.029004+0000", 2024-09-11T08:14:32.411 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.7745e-05 2024-09-11T08:14:32.411 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.411 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.411 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:14:32.411 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.032279+0000", 2024-09-11T08:14:32.411 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0032751360000000001 2024-09-11T08:14:32.411 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.411 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.411 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:14:32.411 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.032285+0000", 2024-09-11T08:14:32.411 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.9959999999999999e-06 2024-09-11T08:14:32.411 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.411 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.412 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:14:32.412 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.032292+0000", 2024-09-11T08:14:32.412 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.6039999999999998e-06 2024-09-11T08:14:32.412 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:14:32.412 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:14:32.412 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:14:32.412 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.412 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.412 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.14790.0:108642 7.f e122/91 7:f6bc24da:::benchmark_data_smithi150_78508_object108641:head v 122'6743, mlcod=122'6741)", 2024-09-11T08:14:32.412 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:11.612981+0000", 2024-09-11T08:14:32.412 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 200.70618706799999, 2024-09-11T08:14:32.412 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.118402496, 2024-09-11T08:14:32.412 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:14:32.413 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:14:32.413 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:14:32.413 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.413 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:14:32.413 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.612981+0000", 2024-09-11T08:14:32.413 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:14:32.413 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.413 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.413 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:14:32.413 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.612981+0000", 2024-09-11T08:14:32.413 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:14:32.413 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.413 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.414 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:14:32.414 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.612981+0000", 2024-09-11T08:14:32.414 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:14:32.414 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.414 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.414 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:14:32.414 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.612999+0000", 2024-09-11T08:14:32.414 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.8606000000000001e-05 2024-09-11T08:14:32.414 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.414 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.414 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:14:32.414 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.613005+0000", 2024-09-11T08:14:32.414 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.7339999999999996e-06 2024-09-11T08:14:32.415 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.415 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.415 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:14:32.415 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.613009+0000", 2024-09-11T08:14:32.415 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.7919999999999999e-06 2024-09-11T08:14:32.415 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.415 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.415 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:14:32.415 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.613168+0000", 2024-09-11T08:14:32.415 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00015886399999999999 2024-09-11T08:14:32.415 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.415 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.415 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:14:32.415 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.613215+0000", 2024-09-11T08:14:32.416 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.7769999999999998e-05 2024-09-11T08:14:32.416 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.416 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.416 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:14:32.416 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.731349+0000", 2024-09-11T08:14:32.416 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.11813324 2024-09-11T08:14:32.416 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.416 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.416 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:14:32.416 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.731384+0000", 2024-09-11T08:14:32.416 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.4984000000000003e-05 2024-09-11T08:14:32.416 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:14:32.416 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:14:32.417 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:14:32.417 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.417 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.417 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:108646 7.2 7:4556b770:::benchmark_data_smithi150_78508_object108645:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-11T08:14:32.417 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:11.614265+0000", 2024-09-11T08:14:32.417 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 200.70490340399999, 2024-09-11T08:14:32.417 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.117338502, 2024-09-11T08:14:32.417 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:14:32.417 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:14:32.417 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:14:32.417 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:14:32.417 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:14:32.417 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 108646 2024-09-11T08:14:32.418 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.418 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:14:32.418 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.418 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:14:32.418 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.614265+0000", 2024-09-11T08:14:32.418 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:14:32.418 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.418 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.418 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:14:32.418 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.614265+0000", 2024-09-11T08:14:32.418 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:14:32.418 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.418 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.418 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:14:32.418 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.614264+0000", 2024-09-11T08:14:32.419 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:14:32.419 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.419 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.419 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:14:32.419 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.614288+0000", 2024-09-11T08:14:32.419 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.4450000000000001e-05 2024-09-11T08:14:32.419 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.419 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.419 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:14:32.419 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.614299+0000", 2024-09-11T08:14:32.419 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.041e-05 2024-09-11T08:14:32.419 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.419 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.419 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:14:32.420 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.614304+0000", 2024-09-11T08:14:32.420 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.6779999999999997e-06 2024-09-11T08:14:32.420 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.420 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.420 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:14:32.420 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.616626+0000", 2024-09-11T08:14:32.420 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00232132 2024-09-11T08:14:32.420 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.420 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.420 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:14:32.420 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.616732+0000", 2024-09-11T08:14:32.420 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000105967 2024-09-11T08:14:32.421 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.421 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.421 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 1,2", 2024-09-11T08:14:32.421 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.616870+0000", 2024-09-11T08:14:32.421 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000138367 2024-09-11T08:14:32.421 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.421 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.421 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:14:32.421 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.625171+0000", 2024-09-11T08:14:32.421 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0083011720000000008 2024-09-11T08:14:32.421 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.421 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.421 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:14:32.422 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.625209+0000", 2024-09-11T08:14:32.422 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.7443e-05 2024-09-11T08:14:32.422 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.422 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.422 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:14:32.422 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:14:32.423 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.731499+0000", 2024-09-11T08:14:32.423 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.106290017 2024-09-11T08:14:32.423 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.423 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.423 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:14:32.423 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.731563+0000", 2024-09-11T08:14:32.424 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.3905000000000006e-05 2024-09-11T08:14:32.424 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.424 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.424 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:14:32.424 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.731603+0000", 2024-09-11T08:14:32.424 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.0803000000000002e-05 2024-09-11T08:14:32.424 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:14:32.424 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:14:32.424 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:14:32.424 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.424 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.424 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:108650 7.a 7:51c95174:::benchmark_data_smithi150_78508_object108649:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-11T08:14:32.424 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:11.619466+0000", 2024-09-11T08:14:32.425 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 200.69970225500001, 2024-09-11T08:14:32.425 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.11220878500000001, 2024-09-11T08:14:32.425 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:14:32.425 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:14:32.425 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:14:32.425 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:14:32.425 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:14:32.425 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 108650 2024-09-11T08:14:32.425 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.425 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:14:32.425 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.425 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:14:32.425 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.619466+0000", 2024-09-11T08:14:32.425 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:14:32.426 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.426 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.426 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:14:32.426 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.619466+0000", 2024-09-11T08:14:32.426 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:14:32.426 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.426 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.426 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:14:32.426 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.619465+0000", 2024-09-11T08:14:32.426 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:14:32.426 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.426 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.426 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:14:32.426 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.619494+0000", 2024-09-11T08:14:32.427 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.9238999999999999e-05 2024-09-11T08:14:32.427 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.427 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.427 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:14:32.427 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.619504+0000", 2024-09-11T08:14:32.427 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.8709999999999992e-06 2024-09-11T08:14:32.427 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.427 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.427 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:14:32.427 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.619509+0000", 2024-09-11T08:14:32.427 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.3480000000000003e-06 2024-09-11T08:14:32.427 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.427 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.427 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:14:32.428 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.619569+0000", 2024-09-11T08:14:32.428 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.0109000000000003e-05 2024-09-11T08:14:32.428 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.428 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.428 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:14:32.428 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.619674+0000", 2024-09-11T08:14:32.428 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00010487799999999999 2024-09-11T08:14:32.428 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.428 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.428 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 2,5", 2024-09-11T08:14:32.428 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.619810+0000", 2024-09-11T08:14:32.428 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000135718 2024-09-11T08:14:32.428 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.428 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.429 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:14:32.429 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.622991+0000", 2024-09-11T08:14:32.429 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0031816779999999999 2024-09-11T08:14:32.429 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.429 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.429 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:14:32.429 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.625215+0000", 2024-09-11T08:14:32.429 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0022232390000000001 2024-09-11T08:14:32.429 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.429 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.429 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:14:32.429 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.731585+0000", 2024-09-11T08:14:32.429 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.10637010299999999 2024-09-11T08:14:32.429 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.429 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.430 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:14:32.430 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.731631+0000", 2024-09-11T08:14:32.430 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.6270000000000003e-05 2024-09-11T08:14:32.430 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.430 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.430 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:14:32.430 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.731675+0000", 2024-09-11T08:14:32.430 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.3745000000000002e-05 2024-09-11T08:14:32.430 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:14:32.430 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:14:32.430 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:14:32.430 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.430 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.430 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:108651 7.3 7:c42b618f:::benchmark_data_smithi150_78508_object108650:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-11T08:14:32.431 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:11.619772+0000", 2024-09-11T08:14:32.431 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 200.699396635, 2024-09-11T08:14:32.431 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.111855882, 2024-09-11T08:14:32.431 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:14:32.431 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:14:32.431 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:14:32.431 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:14:32.431 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:14:32.431 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 108651 2024-09-11T08:14:32.431 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.431 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:14:32.431 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.431 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:14:32.431 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.619772+0000", 2024-09-11T08:14:32.432 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:14:32.432 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.432 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.432 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:14:32.432 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.619772+0000", 2024-09-11T08:14:32.432 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:14:32.432 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.432 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.432 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:14:32.432 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.619771+0000", 2024-09-11T08:14:32.432 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:14:32.432 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.432 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.433 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:14:32.433 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.619797+0000", 2024-09-11T08:14:32.433 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.5629000000000001e-05 2024-09-11T08:14:32.433 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.433 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.433 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:14:32.433 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.619804+0000", 2024-09-11T08:14:32.433 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.3350000000000001e-06 2024-09-11T08:14:32.433 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.433 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.433 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:14:32.433 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.619808+0000", 2024-09-11T08:14:32.433 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.3479999999999997e-06 2024-09-11T08:14:32.433 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.434 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.434 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:14:32.434 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.622759+0000", 2024-09-11T08:14:32.434 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0029511770000000001 2024-09-11T08:14:32.434 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.434 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.434 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:14:32.434 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.622875+0000", 2024-09-11T08:14:32.434 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00011602 2024-09-11T08:14:32.435 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.435 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.435 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 2,4", 2024-09-11T08:14:32.435 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.623005+0000", 2024-09-11T08:14:32.435 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012960599999999999 2024-09-11T08:14:32.435 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.435 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.435 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:14:32.435 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.629997+0000", 2024-09-11T08:14:32.435 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0069922309999999998 2024-09-11T08:14:32.435 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.435 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.435 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:14:32.435 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.731333+0000", 2024-09-11T08:14:32.435 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.10133574200000001 2024-09-11T08:14:32.436 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.436 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.436 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:14:32.436 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.731487+0000", 2024-09-11T08:14:32.436 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00015422799999999999 2024-09-11T08:14:32.436 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.436 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.436 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:14:32.436 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.731550+0000", 2024-09-11T08:14:32.436 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.2887999999999998e-05 2024-09-11T08:14:32.436 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.436 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.436 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:14:32.436 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.731627+0000", 2024-09-11T08:14:32.437 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.7296e-05 2024-09-11T08:14:32.437 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:14:32.437 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:14:32.437 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:14:32.437 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.437 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.437 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.14790.0:108644 7.f e122/91 7:f306e571:::benchmark_data_smithi150_78508_object108643:head v 122'6744, mlcod=122'6741)", 2024-09-11T08:14:32.437 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:11.619850+0000", 2024-09-11T08:14:32.437 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 200.699318425, 2024-09-11T08:14:32.437 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.111552015, 2024-09-11T08:14:32.438 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:14:32.438 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:14:32.438 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:14:32.438 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.438 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:14:32.438 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.619850+0000", 2024-09-11T08:14:32.438 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:14:32.438 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.438 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.438 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:14:32.438 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.619850+0000", 2024-09-11T08:14:32.438 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:14:32.439 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.439 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.439 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:14:32.439 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.619849+0000", 2024-09-11T08:14:32.439 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:14:32.439 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.439 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.439 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:14:32.439 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.619875+0000", 2024-09-11T08:14:32.439 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.6069999999999999e-05 2024-09-11T08:14:32.439 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.439 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.439 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:14:32.439 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.619882+0000", 2024-09-11T08:14:32.439 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.4489999999999996e-06 2024-09-11T08:14:32.440 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.440 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.440 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:14:32.440 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.619885+0000", 2024-09-11T08:14:32.440 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.585e-06 2024-09-11T08:14:32.440 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.440 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.440 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:14:32.440 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.619934+0000", 2024-09-11T08:14:32.440 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.8798999999999998e-05 2024-09-11T08:14:32.440 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.440 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.440 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:14:32.441 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.619951+0000", 2024-09-11T08:14:32.441 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.6988999999999999e-05 2024-09-11T08:14:32.441 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.441 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.441 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:14:32.441 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.731387+0000", 2024-09-11T08:14:32.441 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.11143541999999999 2024-09-11T08:14:32.441 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.441 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.441 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:14:32.441 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.731402+0000", 2024-09-11T08:14:32.441 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.5228000000000001e-05 2024-09-11T08:14:32.441 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:14:32.442 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:14:32.442 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:14:32.442 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.442 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.442 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:108658 7.2 7:4d5b2886:::benchmark_data_smithi150_78508_object108657:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-11T08:14:32.442 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:11.623440+0000", 2024-09-11T08:14:32.442 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 200.69572870100001, 2024-09-11T08:14:32.442 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.108296905, 2024-09-11T08:14:32.442 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:14:32.442 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:14:32.442 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:14:32.442 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:14:32.442 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:14:32.442 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 108658 2024-09-11T08:14:32.443 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.443 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:14:32.443 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.443 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:14:32.443 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.623440+0000", 2024-09-11T08:14:32.443 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:14:32.443 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.443 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.443 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:14:32.443 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.623440+0000", 2024-09-11T08:14:32.443 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:14:32.443 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.443 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.443 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:14:32.443 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.623438+0000", 2024-09-11T08:14:32.444 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:14:32.444 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.444 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.444 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:14:32.444 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.623460+0000", 2024-09-11T08:14:32.444 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.1345e-05 2024-09-11T08:14:32.444 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.444 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.444 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:14:32.444 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.623470+0000", 2024-09-11T08:14:32.444 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0857e-05 2024-09-11T08:14:32.444 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.444 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.445 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:14:32.445 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.623476+0000", 2024-09-11T08:14:32.445 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.66e-06 2024-09-11T08:14:32.445 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.445 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.445 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:14:32.445 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.625234+0000", 2024-09-11T08:14:32.445 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.001758049 2024-09-11T08:14:32.445 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.445 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.445 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:14:32.445 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.625341+0000", 2024-09-11T08:14:32.445 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000106655 2024-09-11T08:14:32.445 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.445 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.446 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 1,2", 2024-09-11T08:14:32.446 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.625469+0000", 2024-09-11T08:14:32.446 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000128653 2024-09-11T08:14:32.446 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.446 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.446 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:14:32.446 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.628901+0000", 2024-09-11T08:14:32.446 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0034319989999999998 2024-09-11T08:14:32.446 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.446 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.446 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:14:32.446 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.731639+0000", 2024-09-11T08:14:32.446 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.102737068 2024-09-11T08:14:32.446 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.447 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.447 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:14:32.447 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.731665+0000", 2024-09-11T08:14:32.447 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.6012000000000001e-05 2024-09-11T08:14:32.447 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.447 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.447 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:14:32.447 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.731703+0000", 2024-09-11T08:14:32.447 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.8917e-05 2024-09-11T08:14:32.447 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.447 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.447 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:14:32.447 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.731736+0000", 2024-09-11T08:14:32.447 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.2960000000000003e-05 2024-09-11T08:14:32.447 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:14:32.448 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:14:32.448 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:14:32.448 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.448 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.448 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.14790.0:108652 7.f e122/91 7:f02b47c9:::benchmark_data_smithi150_78508_object108651:head v 122'6745, mlcod=122'6741)", 2024-09-11T08:14:32.448 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:11.623493+0000", 2024-09-11T08:14:32.448 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 200.69567534000001, 2024-09-11T08:14:32.448 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.113189462, 2024-09-11T08:14:32.448 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:14:32.448 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:14:32.448 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:14:32.448 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.448 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:14:32.448 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.623493+0000", 2024-09-11T08:14:32.449 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:14:32.449 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.449 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.449 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:14:32.449 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.623493+0000", 2024-09-11T08:14:32.449 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:14:32.449 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.449 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.449 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:14:32.449 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.623492+0000", 2024-09-11T08:14:32.449 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:14:32.449 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.449 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.449 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:14:32.449 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.623510+0000", 2024-09-11T08:14:32.450 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.7629e-05 2024-09-11T08:14:32.450 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.450 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.450 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:14:32.450 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.623516+0000", 2024-09-11T08:14:32.450 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.0929999999999999e-06 2024-09-11T08:14:32.450 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.450 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.450 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:14:32.450 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.623520+0000", 2024-09-11T08:14:32.450 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.709e-06 2024-09-11T08:14:32.450 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.450 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.450 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:14:32.450 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.630343+0000", 2024-09-11T08:14:32.451 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0068228769999999998 2024-09-11T08:14:32.451 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.451 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.451 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:14:32.451 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.630360+0000", 2024-09-11T08:14:32.451 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.7397999999999998e-05 2024-09-11T08:14:32.451 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.451 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.451 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:14:32.451 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.736620+0000", 2024-09-11T08:14:32.451 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.10625954999999999 2024-09-11T08:14:32.451 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.451 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.451 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:14:32.452 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.736682+0000", 2024-09-11T08:14:32.452 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.2648000000000001e-05 2024-09-11T08:14:32.452 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:14:32.452 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:14:32.452 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:14:32.452 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.452 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.452 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.14790.0:142816 7.c e129/91 7:35f8d9b8:::benchmark_data_smithi150_78508_object18375:head v 129'8819, mlcod=129'8817)", 2024-09-11T08:14:32.452 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:45.451357+0000", 2024-09-11T08:14:32.452 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 166.86781138699999, 2024-09-11T08:14:32.452 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.109437163, 2024-09-11T08:14:32.452 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:14:32.452 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:14:32.453 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:14:32.453 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.453 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:14:32.453 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.451357+0000", 2024-09-11T08:14:32.453 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:14:32.453 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.453 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.453 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:14:32.453 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.451357+0000", 2024-09-11T08:14:32.453 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:14:32.453 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.453 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.453 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:14:32.453 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.451353+0000", 2024-09-11T08:14:32.454 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999957 2024-09-11T08:14:32.454 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.454 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.454 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:14:32.454 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.451361+0000", 2024-09-11T08:14:32.454 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.8689999999999993e-06 2024-09-11T08:14:32.454 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.454 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.454 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:14:32.454 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.451369+0000", 2024-09-11T08:14:32.454 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.5480000000000004e-06 2024-09-11T08:14:32.454 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.454 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.454 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:14:32.455 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.451374+0000", 2024-09-11T08:14:32.455 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.843e-06 2024-09-11T08:14:32.455 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.455 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.455 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:14:32.455 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.451413+0000", 2024-09-11T08:14:32.455 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.8887999999999998e-05 2024-09-11T08:14:32.455 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.455 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.455 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:14:32.455 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.451431+0000", 2024-09-11T08:14:32.455 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.7952999999999999e-05 2024-09-11T08:14:32.455 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.455 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.456 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:14:32.456 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.560753+0000", 2024-09-11T08:14:32.456 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.109321953 2024-09-11T08:14:32.456 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.456 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.456 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:14:32.456 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.560794+0000", 2024-09-11T08:14:32.456 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1341999999999998e-05 2024-09-11T08:14:32.456 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:14:32.456 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:14:32.456 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:14:32.456 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.456 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.456 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:142827 7.3 7:c80263f7:::benchmark_data_smithi150_78508_object18386:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-11T08:14:32.456 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:45.460440+0000", 2024-09-11T08:14:32.457 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 166.85872830700001, 2024-09-11T08:14:32.457 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.110157146, 2024-09-11T08:14:32.457 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:14:32.457 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:14:32.457 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:14:32.457 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:14:32.457 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:14:32.457 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 142827 2024-09-11T08:14:32.457 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.457 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:14:32.457 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.457 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:14:32.457 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.460440+0000", 2024-09-11T08:14:32.457 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:14:32.458 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.458 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.458 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:14:32.458 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.460440+0000", 2024-09-11T08:14:32.458 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:14:32.458 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.458 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.458 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:14:32.458 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.460438+0000", 2024-09-11T08:14:32.458 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:14:32.458 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.458 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.458 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:14:32.458 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.460445+0000", 2024-09-11T08:14:32.459 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.5429999999999999e-06 2024-09-11T08:14:32.459 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.459 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.459 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:14:32.459 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.460455+0000", 2024-09-11T08:14:32.459 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0421000000000001e-05 2024-09-11T08:14:32.459 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.459 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.459 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:14:32.459 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.460460+0000", 2024-09-11T08:14:32.459 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.2000000000000002e-06 2024-09-11T08:14:32.459 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.459 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.459 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:14:32.459 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.460546+0000", 2024-09-11T08:14:32.460 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.6008999999999996e-05 2024-09-11T08:14:32.460 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.460 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.460 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:14:32.460 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.460656+0000", 2024-09-11T08:14:32.460 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000109898 2024-09-11T08:14:32.460 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.460 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.460 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 2,4", 2024-09-11T08:14:32.460 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.560816+0000", 2024-09-11T08:14:32.460 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.100160286 2024-09-11T08:14:32.460 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.460 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.461 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:14:32.461 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.566356+0000", 2024-09-11T08:14:32.461 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0055392159999999996 2024-09-11T08:14:32.461 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.461 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.461 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:14:32.461 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.566361+0000", 2024-09-11T08:14:32.461 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.5509999999999999e-06 2024-09-11T08:14:32.461 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.461 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.461 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:14:32.461 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.566384+0000", 2024-09-11T08:14:32.461 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.2819e-05 2024-09-11T08:14:32.461 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.462 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.462 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:14:32.462 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.566426+0000", 2024-09-11T08:14:32.462 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1788e-05 2024-09-11T08:14:32.462 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.462 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.462 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:14:32.462 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.570597+0000", 2024-09-11T08:14:32.462 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0041712590000000001 2024-09-11T08:14:32.462 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:14:32.462 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:14:32.462 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:14:32.462 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.462 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.463 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.24320.0:930 5.14 5:2b04a3e9:::notify.6:head [watch reconnect cookie 94783556106752 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-09-11T08:14:32.463 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:12:27.939524+0000", 2024-09-11T08:14:32.463 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 124.37964393, 2024-09-11T08:14:32.463 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.56717752799999999, 2024-09-11T08:14:32.463 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:14:32.463 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "started", 2024-09-11T08:14:32.463 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:14:32.463 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.24320", 2024-09-11T08:14:32.463 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1503848706", 2024-09-11T08:14:32.463 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 930 2024-09-11T08:14:32.463 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.463 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:14:32.463 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.463 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:14:32.463 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.939524+0000", 2024-09-11T08:14:32.464 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:14:32.464 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.464 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.464 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:14:32.464 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.939524+0000", 2024-09-11T08:14:32.464 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:14:32.464 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.464 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.464 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:14:32.464 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.939523+0000", 2024-09-11T08:14:32.464 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:14:32.464 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.464 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.464 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:14:32.465 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.939529+0000", 2024-09-11T08:14:32.465 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.5189999999999998e-06 2024-09-11T08:14:32.465 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.465 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.465 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:14:32.465 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.939545+0000", 2024-09-11T08:14:32.465 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.573e-05 2024-09-11T08:14:32.465 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.465 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.465 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:14:32.465 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.939553+0000", 2024-09-11T08:14:32.465 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.2290000000000003e-06 2024-09-11T08:14:32.465 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.465 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.466 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:14:32.466 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.939603+0000", 2024-09-11T08:14:32.466 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.9285e-05 2024-09-11T08:14:32.466 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.466 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.466 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:14:32.466 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.939611+0000", 2024-09-11T08:14:32.466 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.3280000000000006e-06 2024-09-11T08:14:32.466 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.466 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.466 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:14:32.466 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.502573+0000", 2024-09-11T08:14:32.466 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.56296208000000003 2024-09-11T08:14:32.466 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.466 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.467 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:14:32.467 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.502714+0000", 2024-09-11T08:14:32.467 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00014101700000000001 2024-09-11T08:14:32.467 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.467 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.467 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:14:32.467 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.506702+0000", 2024-09-11T08:14:32.467 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0039878780000000003 2024-09-11T08:14:32.467 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:14:32.467 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:14:32.467 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:14:32.467 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.467 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.467 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:239248 7.a 7:53cd58d4:::benchmark_data_smithi150_78508_object114807:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-11T08:14:32.468 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:13:19.500925+0000", 2024-09-11T08:14:32.468 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 72.818243206000005, 2024-09-11T08:14:32.468 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.106801588, 2024-09-11T08:14:32.468 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:14:32.468 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:14:32.468 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:14:32.468 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:14:32.468 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:14:32.468 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 239248 2024-09-11T08:14:32.468 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.468 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:14:32.468 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.468 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:14:32.468 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:19.500925+0000", 2024-09-11T08:14:32.468 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:14:32.469 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.469 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.469 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:14:32.469 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:19.500925+0000", 2024-09-11T08:14:32.469 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:14:32.469 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.469 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.469 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:14:32.469 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:19.500924+0000", 2024-09-11T08:14:32.469 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:14:32.469 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.469 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.469 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:14:32.469 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:19.500929+0000", 2024-09-11T08:14:32.470 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.2299999999999999e-06 2024-09-11T08:14:32.470 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.470 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.470 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:14:32.470 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:19.500938+0000", 2024-09-11T08:14:32.470 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.7250000000000003e-06 2024-09-11T08:14:32.470 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.470 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.470 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:14:32.470 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:19.500944+0000", 2024-09-11T08:14:32.470 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.976e-06 2024-09-11T08:14:32.470 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.470 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.470 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:14:32.471 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:19.501093+0000", 2024-09-11T08:14:32.471 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00014953300000000001 2024-09-11T08:14:32.471 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.471 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.471 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:14:32.471 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:19.501369+0000", 2024-09-11T08:14:32.471 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00027565800000000002 2024-09-11T08:14:32.471 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.471 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.471 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 2,5", 2024-09-11T08:14:32.471 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:19.501747+0000", 2024-09-11T08:14:32.471 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00037776599999999997 2024-09-11T08:14:32.471 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.471 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.472 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:14:32.472 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:19.504622+0000", 2024-09-11T08:14:32.472 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0028749769999999999 2024-09-11T08:14:32.472 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.472 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.472 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:14:32.472 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:19.510221+0000", 2024-09-11T08:14:32.472 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0055994560000000001 2024-09-11T08:14:32.472 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.472 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.472 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:14:32.472 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:19.607677+0000", 2024-09-11T08:14:32.472 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.097455579000000001 2024-09-11T08:14:32.472 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.473 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.473 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:14:32.473 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:19.607706+0000", 2024-09-11T08:14:32.473 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.9088e-05 2024-09-11T08:14:32.473 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:14:32.473 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:14:32.473 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:14:32.473 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:19.607727+0000", 2024-09-11T08:14:32.473 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.0893000000000001e-05 2024-09-11T08:14:32.473 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:14:32.473 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:14:32.473 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:14:32.473 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:14:32.473 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:14:32.474 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-11T08:14:32.474 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 21 16 43688 43672 129.952 95.5 0.00891154 0.00768891 2024-09-11T08:14:32.508 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:32.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:32.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:32.610 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:32.645 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:32.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:32.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:32.747 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:32.782 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:32.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:32.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:32.884 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:14:32.917 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:32.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:32.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:14:33.019 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:33.055 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:33.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:33.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:33.157 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:33.195 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:33.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:33.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:33.297 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:33.334 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:33.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:33.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:33.408 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 22 16 45337 45321 128.73 103.062 0.00293005 0.00776058 2024-09-11T08:14:33.435 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:33.468 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:33.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:33.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:33.570 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:33.605 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:33.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:33.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:33.707 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:33.747 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-11T08:14:33.757 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:33.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:33.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:33.765 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T08:14:33.859 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:14:33.915 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:33.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:33.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:14:33.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:33 smithi150 ceph-mon[39612]: pgmap v295: 148 pgs: 3 active+recovering+undersized+degraded+remapped, 145 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s; 5276/119508 objects degraded (4.415%); 9.7 MiB/s, 155 objects/s recovering 2024-09-11T08:14:33.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:33 smithi150 ceph-mon[29276]: pgmap v295: 148 pgs: 3 active+recovering+undersized+degraded+remapped, 145 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s; 5276/119508 objects degraded (4.415%); 9.7 MiB/s, 155 objects/s recovering 2024-09-11T08:14:34.017 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:14:34.052 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:34.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:34.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:14:34.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:33 smithi181 ceph-mon[32701]: pgmap v295: 148 pgs: 3 active+recovering+undersized+degraded+remapped, 145 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s; 5276/119508 objects degraded (4.415%); 9.7 MiB/s, 155 objects/s recovering 2024-09-11T08:14:34.154 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:34.163 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:14:34.199 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:34.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:34.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:34.301 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:34.338 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:34.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:34.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:34.408 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 23 16 46837 46821 127.209 93.75 0.00483378 0.00785591 2024-09-11T08:14:34.440 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:34.480 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:34.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:34.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:34.582 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:34.621 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:34.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:34.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:34.722 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:34.759 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:34.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:34.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:34.861 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:34.904 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:34.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:34.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:34.978 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:34 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:14:34.978 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:34 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:14:34.978 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:34 smithi181 ceph-mon[32701]: osdmap e173: 8 total, 8 up, 8 in 2024-09-11T08:14:34.978 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:34 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:14:34.978 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:34 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:14:35.006 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:35.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:35.046 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:35.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:35.147 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:35.178 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:35.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:35.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:35.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:34 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:14:35.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:34 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:14:35.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:34 smithi150 ceph-mon[39612]: osdmap e173: 8 total, 8 up, 8 in 2024-09-11T08:14:35.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:34 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:14:35.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:34 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:14:35.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:34 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:14:35.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:34 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:14:35.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:34 smithi150 ceph-mon[29276]: osdmap e173: 8 total, 8 up, 8 in 2024-09-11T08:14:35.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:34 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:14:35.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:34 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:14:35.280 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:35.317 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:35.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:35.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:35.323 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:14:35 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:14:35] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:14:35.408 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 24 16 48406 48390 125.994 98.0625 0.00449166 0.00793177 2024-09-11T08:14:35.419 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:35.451 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:35.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:35.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:35.552 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:35.592 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:35.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:35.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:35.694 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:35.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:35.734 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:35.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:35.835 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:35.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:35.880 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:35.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:35.981 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:14:36.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:36.023 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:36.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:14:36.124 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:36.158 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:36.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:36.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:36.188 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:35 smithi181 ceph-mon[32701]: pgmap v297: 148 pgs: 3 active+recovering+undersized+degraded+remapped, 145 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 728 B/s rd, 98 MiB/s wr, 1.57k op/s; 4961/134301 objects degraded (3.694%); 10 MiB/s, 161 objects/s recovering 2024-09-11T08:14:36.189 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:35 smithi181 ceph-mon[32701]: osdmap e174: 8 total, 8 up, 8 in 2024-09-11T08:14:36.189 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:35 smithi181 ceph-mon[32701]: Health check update: Degraded data redundancy: 4961/134301 objects degraded (3.694%), 3 pgs degraded (PG_DEGRADED) 2024-09-11T08:14:36.189 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:35 smithi181 ceph-mon[32701]: osdmap e175: 8 total, 8 up, 8 in 2024-09-11T08:14:36.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:35 smithi150 ceph-mon[29276]: pgmap v297: 148 pgs: 3 active+recovering+undersized+degraded+remapped, 145 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 728 B/s rd, 98 MiB/s wr, 1.57k op/s; 4961/134301 objects degraded (3.694%); 10 MiB/s, 161 objects/s recovering 2024-09-11T08:14:36.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:35 smithi150 ceph-mon[29276]: osdmap e174: 8 total, 8 up, 8 in 2024-09-11T08:14:36.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:35 smithi150 ceph-mon[29276]: Health check update: Degraded data redundancy: 4961/134301 objects degraded (3.694%), 3 pgs degraded (PG_DEGRADED) 2024-09-11T08:14:36.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:35 smithi150 ceph-mon[29276]: osdmap e175: 8 total, 8 up, 8 in 2024-09-11T08:14:36.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:35 smithi150 ceph-mon[39612]: pgmap v297: 148 pgs: 3 active+recovering+undersized+degraded+remapped, 145 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 728 B/s rd, 98 MiB/s wr, 1.57k op/s; 4961/134301 objects degraded (3.694%); 10 MiB/s, 161 objects/s recovering 2024-09-11T08:14:36.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:35 smithi150 ceph-mon[39612]: osdmap e174: 8 total, 8 up, 8 in 2024-09-11T08:14:36.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:35 smithi150 ceph-mon[39612]: Health check update: Degraded data redundancy: 4961/134301 objects degraded (3.694%), 3 pgs degraded (PG_DEGRADED) 2024-09-11T08:14:36.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:35 smithi150 ceph-mon[39612]: osdmap e175: 8 total, 8 up, 8 in 2024-09-11T08:14:36.260 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:36.284 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:36.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:36.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:36.330 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-11T08:14:36.331 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-09-11T08:14:36.331 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:14:36.331 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15087.0:49681 8.2 8:4468ddab:::benchmark_data_smithi150_107646_object49680:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-11T08:14:36.331 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:14:36.326074+0000", 2024-09-11T08:14:36.331 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.0025654850000000002, 2024-09-11T08:14:36.331 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0026027870000000001, 2024-09-11T08:14:36.331 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:14:36.331 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "waiting for sub ops", 2024-09-11T08:14:36.331 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:14:36.331 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15087", 2024-09-11T08:14:36.331 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/3299560913", 2024-09-11T08:14:36.331 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 49681 2024-09-11T08:14:36.331 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:14:36.332 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:14:36.332 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:14:36.332 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:14:36.332 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:36.326074+0000", 2024-09-11T08:14:36.332 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:14:36.332 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:14:36.332 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:14:36.332 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:14:36.333 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:36.326074+0000", 2024-09-11T08:14:36.333 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:14:36.333 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:14:36.333 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:14:36.333 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:14:36.333 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:36.326072+0000", 2024-09-11T08:14:36.333 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:14:36.333 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:14:36.333 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:14:36.333 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:14:36.333 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:36.326117+0000", 2024-09-11T08:14:36.333 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4353e-05 2024-09-11T08:14:36.333 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:14:36.334 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:14:36.334 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:14:36.334 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:36.326128+0000", 2024-09-11T08:14:36.334 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1212999999999999e-05 2024-09-11T08:14:36.334 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:14:36.334 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:14:36.334 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:14:36.334 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:36.326134+0000", 2024-09-11T08:14:36.334 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0109999999999997e-06 2024-09-11T08:14:36.334 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:14:36.334 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:14:36.334 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:14:36.334 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:36.326185+0000", 2024-09-11T08:14:36.335 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1653999999999997e-05 2024-09-11T08:14:36.335 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:14:36.335 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:14:36.335 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:14:36.335 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:36.326324+0000", 2024-09-11T08:14:36.335 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013814400000000001 2024-09-11T08:14:36.335 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:14:36.335 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:14:36.335 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,5", 2024-09-11T08:14:36.335 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:36.326493+0000", 2024-09-11T08:14:36.335 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016991700000000001 2024-09-11T08:14:36.335 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:14:36.335 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:14:36.335 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:14:36.336 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:36.328293+0000", 2024-09-11T08:14:36.336 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0017995280000000001 2024-09-11T08:14:36.336 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:14:36.336 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:14:36.336 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:14:36.336 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:14:36.336 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:14:36.336 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.15087.0:49679 8.5 e175/169 8:a6c3225f:::benchmark_data_smithi150_107646_object49678:head v 175'3206, mlcod=164'773)", 2024-09-11T08:14:36.336 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:14:36.327065+0000", 2024-09-11T08:14:36.336 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.0015743700000000001, 2024-09-11T08:14:36.336 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0016938470000000001, 2024-09-11T08:14:36.336 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:14:36.336 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-11T08:14:36.336 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:14:36.337 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:14:36.337 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:14:36.337 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:36.327065+0000", 2024-09-11T08:14:36.337 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:14:36.337 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:14:36.337 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:14:36.337 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:14:36.337 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:36.327065+0000", 2024-09-11T08:14:36.337 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:14:36.337 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:14:36.337 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:14:36.337 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:14:36.337 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:36.327064+0000", 2024-09-11T08:14:36.338 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-09-11T08:14:36.338 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:14:36.338 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:14:36.338 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:14:36.338 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:36.327086+0000", 2024-09-11T08:14:36.338 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1860000000000001e-05 2024-09-11T08:14:36.338 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:14:36.338 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:14:36.338 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:14:36.338 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:36.327096+0000", 2024-09-11T08:14:36.338 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.6069999999999997e-06 2024-09-11T08:14:36.338 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:14:36.338 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:14:36.338 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:14:36.338 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:36.327101+0000", 2024-09-11T08:14:36.339 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1880000000000001e-06 2024-09-11T08:14:36.339 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:14:36.339 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:14:36.339 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:14:36.339 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:36.327197+0000", 2024-09-11T08:14:36.339 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.5665e-05 2024-09-11T08:14:36.339 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:14:36.339 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:14:36.339 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:14:36.339 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:36.327224+0000", 2024-09-11T08:14:36.339 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7577999999999999e-05 2024-09-11T08:14:36.339 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:14:36.339 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:14:36.339 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:14:36.340 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:14:36.340 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-09-11T08:14:36.340 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 2 2024-09-11T08:14:36.340 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-11T08:14:36.386 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:36.408 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 25 16 49807 49791 124.457 87.5625 0.00937137 0.00802946 2024-09-11T08:14:36.423 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:36.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:36.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:36.525 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:36.559 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:36.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:36.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:36.661 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:36.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:36.700 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:36.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:36.801 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:36.838 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:36.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:36.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:36.939 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:36.978 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:36.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:36.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:37.079 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:37.118 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:37.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:37.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:37.219 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:37.245 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:37.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:37.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:37.347 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:14:37.365 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-11T08:14:37.379 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:37.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:37.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:14:37.409 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 26 16 51273 51257 123.194 91.625 0.0134243 0.00811081 2024-09-11T08:14:37.481 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:37.510 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:37.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:37.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:37.612 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:37.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:37.654 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:37.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:37.755 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:37.788 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:37.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:37.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:37.802 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:14:37.890 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:37.933 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:37.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:37.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:38.015 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:37 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:14:38.016 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:37 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:14:38.016 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:37 smithi150 ceph-mon[29276]: pgmap v300: 148 pgs: 3 active+recovering+undersized+degraded+remapped, 145 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 83 MiB/s wr, 1.33k op/s; 4961/134301 objects degraded (3.694%); 10 MiB/s, 166 objects/s recovering 2024-09-11T08:14:38.016 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:37 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:14:38.016 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:37 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:14:38.016 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:37 smithi150 ceph-mon[39612]: pgmap v300: 148 pgs: 3 active+recovering+undersized+degraded+remapped, 145 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 83 MiB/s wr, 1.33k op/s; 4961/134301 objects degraded (3.694%); 10 MiB/s, 166 objects/s recovering 2024-09-11T08:14:38.035 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:38.070 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:37 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:14:38.070 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:37 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:14:38.070 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:37 smithi181 ceph-mon[32701]: pgmap v300: 148 pgs: 3 active+recovering+undersized+degraded+remapped, 145 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 83 MiB/s wr, 1.33k op/s; 4961/134301 objects degraded (3.694%); 10 MiB/s, 166 objects/s recovering 2024-09-11T08:14:38.071 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:38.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:38.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:38.157 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:14:38.173 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:14:38.205 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:38.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:38.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:14:38.306 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:38.323 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:14:38 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:14:38.341 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:38.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:38.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:38.409 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 27 16 52700 52684 121.934 89.1875 0.00713356 0.00819578 2024-09-11T08:14:38.443 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:38.474 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:38.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:38.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:38.576 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:38.606 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:38.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:38.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:38.708 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:38.740 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:38.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:38.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:38.831 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:38 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:14:38.832 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:14:38 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:14:38] "GET /metrics HTTP/1.1" 200 224022 "" "Prometheus/2.43.0" 2024-09-11T08:14:38.832 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:38 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:14:38.842 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:38.880 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:38.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:38.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:38.981 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:14:39.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:39.027 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:39.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:14:39.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:38 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:14:39.129 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:39.169 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:39.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:39.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:39.271 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:39.312 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:39.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:39.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:39.409 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 28 16 53896 53880 120.249 74.75 0.0102502 0.00831067 2024-09-11T08:14:39.414 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:39.460 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:39.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:39.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:39.562 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:39.601 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:39.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:39.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:39.703 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:39.737 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:39.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:39.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:39.754 INFO:teuthology.orchestra.run.smithi150.stderr:reweighted osd.0 to 1 (10000) 2024-09-11T08:14:39.755 INFO:teuthology.orchestra.run.smithi150.stderr:noscrub is unset 2024-09-11T08:14:39.839 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:39.866 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:39.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:39.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:39.880 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-11T08:14:39.880 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-11T08:14:39.880 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-09-11T08:14:39.880 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-09-11T08:14:39.880 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-11T08:14:39.880 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:39 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1455640330' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-11T08:14:39.880 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:39 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-11T08:14:39.880 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:39 smithi181 ceph-mon[32701]: pgmap v301: 148 pgs: 3 active+recovering+undersized+degraded+remapped, 145 active+clean; 3.2 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 129 MiB/s wr, 2.07k op/s; 3822/156723 objects degraded (2.439%); 15 MiB/s, 242 objects/s recovering 2024-09-11T08:14:39.880 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:39 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3942711365' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:14:39.880 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:39 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:14:39.968 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:14:40.006 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:39 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1455640330' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-11T08:14:40.006 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:39 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-11T08:14:40.006 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:39 smithi150 ceph-mon[29276]: pgmap v301: 148 pgs: 3 active+recovering+undersized+degraded+remapped, 145 active+clean; 3.2 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 129 MiB/s wr, 2.07k op/s; 3822/156723 objects degraded (2.439%); 15 MiB/s, 242 objects/s recovering 2024-09-11T08:14:40.006 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:39 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3942711365' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:14:40.006 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:39 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:14:40.007 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:39 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1455640330' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-11T08:14:40.007 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:39 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-11T08:14:40.007 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:39 smithi150 ceph-mon[39612]: pgmap v301: 148 pgs: 3 active+recovering+undersized+degraded+remapped, 145 active+clean; 3.2 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 129 MiB/s wr, 2.07k op/s; 3822/156723 objects degraded (2.439%); 15 MiB/s, 242 objects/s recovering 2024-09-11T08:14:40.007 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:39 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3942711365' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:14:40.007 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:39 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:14:40.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:40.013 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:40.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:14:40.114 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:14:40.147 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:40.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:40.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:14:40.249 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:40.280 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:40.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:40.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:40.382 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:40.409 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 29 16 55141 55125 118.785 77.8125 0.0117237 0.00841355 2024-09-11T08:14:40.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:40.420 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:40.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:40.521 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:40.555 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:40.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:40.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:40.657 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:40.678 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:40.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:40.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:40.779 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:40.815 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd reweight 1 1 2024-09-11T08:14:40.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:40.817 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:40.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:40.919 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:40.959 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:40.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:40.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:41.061 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:41.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:40 smithi181 ceph-mon[32701]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:14:41.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:40 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-11T08:14:41.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:40 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:14:41.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:40 smithi181 ceph-mon[32701]: osdmap e176: 8 total, 8 up, 8 in 2024-09-11T08:14:41.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:40 smithi181 ceph-mon[32701]: 5.d scrub starts 2024-09-11T08:14:41.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:40 smithi181 ceph-mon[32701]: Health check update: Degraded data redundancy: 3822/156723 objects degraded (2.439%), 3 pgs degraded (PG_DEGRADED) 2024-09-11T08:14:41.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:40 smithi181 ceph-mon[32701]: osdmap e177: 8 total, 8 up, 8 in 2024-09-11T08:14:41.097 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:41.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:41.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:41.161 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:40 smithi150 ceph-mon[29276]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:14:41.161 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:40 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-11T08:14:41.161 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:40 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:14:41.161 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:40 smithi150 ceph-mon[29276]: osdmap e176: 8 total, 8 up, 8 in 2024-09-11T08:14:41.161 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:40 smithi150 ceph-mon[29276]: 5.d scrub starts 2024-09-11T08:14:41.161 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:40 smithi150 ceph-mon[29276]: Health check update: Degraded data redundancy: 3822/156723 objects degraded (2.439%), 3 pgs degraded (PG_DEGRADED) 2024-09-11T08:14:41.162 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:40 smithi150 ceph-mon[29276]: osdmap e177: 8 total, 8 up, 8 in 2024-09-11T08:14:41.163 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:40 smithi150 ceph-mon[39612]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:14:41.163 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:40 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-11T08:14:41.163 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:40 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:14:41.163 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:40 smithi150 ceph-mon[39612]: osdmap e176: 8 total, 8 up, 8 in 2024-09-11T08:14:41.163 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:40 smithi150 ceph-mon[39612]: 5.d scrub starts 2024-09-11T08:14:41.163 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:40 smithi150 ceph-mon[39612]: Health check update: Degraded data redundancy: 3822/156723 objects degraded (2.439%), 3 pgs degraded (PG_DEGRADED) 2024-09-11T08:14:41.163 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:40 smithi150 ceph-mon[39612]: osdmap e177: 8 total, 8 up, 8 in 2024-09-11T08:14:41.199 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:41.243 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:41.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:41.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:41.345 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:41.372 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:41.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:41.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:41.409 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 30 16 56285 56269 117.209 71.5 0.0153524 0.00852632 2024-09-11T08:14:41.474 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:41.504 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:41.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:41.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:41.606 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:41.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:41.654 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:41.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:41.755 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:41.793 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:41.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:41.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:41.895 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:41.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:41.925 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:41.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:42.026 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:14:42.060 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:42.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:42.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:14:42.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:41 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:14:42.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:41 smithi181 ceph-mon[32701]: pgmap v304: 148 pgs: 3 active+recovering+undersized+degraded+remapped, 145 active+clean; 3.2 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 859 B/s rd, 78 MiB/s wr, 1.25k op/s; 3822/156723 objects degraded (2.439%); 12 MiB/s, 191 objects/s recovering 2024-09-11T08:14:42.163 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:14:42.192 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:42.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:42.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:14:42.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:41 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:14:42.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:41 smithi150 ceph-mon[29276]: pgmap v304: 148 pgs: 3 active+recovering+undersized+degraded+remapped, 145 active+clean; 3.2 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 859 B/s rd, 78 MiB/s wr, 1.25k op/s; 3822/156723 objects degraded (2.439%); 12 MiB/s, 191 objects/s recovering 2024-09-11T08:14:42.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:41 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:14:42.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:41 smithi150 ceph-mon[39612]: pgmap v304: 148 pgs: 3 active+recovering+undersized+degraded+remapped, 145 active+clean; 3.2 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 859 B/s rd, 78 MiB/s wr, 1.25k op/s; 3822/156723 objects degraded (2.439%); 12 MiB/s, 191 objects/s recovering 2024-09-11T08:14:42.294 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:14:42.337 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:42.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:42.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:14:42.409 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 31 16 57524 57508 115.926 77.4375 0.0032278 0.00861284 2024-09-11T08:14:42.438 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:42.472 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:42.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:42.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:42.574 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:42.606 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:42.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:42.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:42.708 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:42.760 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:42.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:42.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:42.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:42 smithi181 ceph-mon[32701]: 3.0 scrub starts 2024-09-11T08:14:42.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:42 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:14:42.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:42 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:14:42.862 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:42.883 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:42.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:42.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:42.985 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:43.019 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:43.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:43.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:43.121 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:43.155 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:43.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:43.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:43.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:42 smithi150 ceph-mon[29276]: 3.0 scrub starts 2024-09-11T08:14:43.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:42 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:14:43.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:42 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:14:43.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:42 smithi150 ceph-mon[39612]: 3.0 scrub starts 2024-09-11T08:14:43.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:42 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:14:43.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:42 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:14:43.257 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:43.291 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:43.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:43.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:43.393 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:43.410 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 32 16 58810 58794 114.811 80.375 0.0178719 0.00870482 2024-09-11T08:14:43.424 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:43.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:43.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:43.439 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-11T08:14:43.474 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:14:43.526 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:14:43.558 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:43.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:43.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:14:43.660 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:43.698 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:43.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:43.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:43.800 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:43.834 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:43.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:43.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:43.864 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:14:43.936 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:43.974 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:43.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:43.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:44.041 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:43 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:14:44.041 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:43 smithi150 ceph-mon[29276]: pgmap v305: 148 pgs: 3 active+recovering+undersized+degraded+remapped, 145 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.4 KiB/s rd, 83 MiB/s wr, 1.32k op/s; 2824/163584 objects degraded (1.726%); 18 MiB/s, 289 objects/s recovering 2024-09-11T08:14:44.041 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:43 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:14:44.041 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:43 smithi150 ceph-mon[39612]: pgmap v305: 148 pgs: 3 active+recovering+undersized+degraded+remapped, 145 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.4 KiB/s rd, 83 MiB/s wr, 1.32k op/s; 2824/163584 objects degraded (1.726%); 18 MiB/s, 289 objects/s recovering 2024-09-11T08:14:44.043 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:43 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:14:44.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:43 smithi181 ceph-mon[32701]: pgmap v305: 148 pgs: 3 active+recovering+undersized+degraded+remapped, 145 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.4 KiB/s rd, 83 MiB/s wr, 1.32k op/s; 2824/163584 objects degraded (1.726%); 18 MiB/s, 289 objects/s recovering 2024-09-11T08:14:44.077 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:44.116 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:44.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:44.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:44.218 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:44.255 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:44.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:44.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:44.357 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:44.394 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:44.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:44.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:44.410 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 33 16 60177 60161 113.921 85.4375 0.00532043 0.00877237 2024-09-11T08:14:44.496 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:44.508 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T08:14:44.532 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:44.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:44.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:44.619 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:14:44.619 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:14:44.620 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:14:44.620 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:14:44.620 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:14:44.621 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:14:44.634 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:44.651 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:14:44.651 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:14:44.685 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:14:44.685 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:14:44.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:44.698 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:44.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:44.723 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:14:44.723 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:14:44.757 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:14:44.758 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:14:44.786 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:14:44.787 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:14:44.799 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:44.833 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:44.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:44.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:44.935 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:44.939 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:44 smithi150 ceph-mon[29276]: 5.6 scrub starts 2024-09-11T08:14:44.940 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:44 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2980917227' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:14:44.940 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:44 smithi150 ceph-mon[39612]: 5.6 scrub starts 2024-09-11T08:14:44.940 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:44 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2980917227' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:14:44.971 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:44.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:44.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:45.073 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:45.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:44 smithi181 ceph-mon[32701]: 5.6 scrub starts 2024-09-11T08:14:45.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:44 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2980917227' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:14:45.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:45.110 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:45.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:45.211 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:45.255 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:45.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:45.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:45.357 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:45.393 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:45.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:45.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:45.410 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 34 16 61542 61526 113.079 85.3125 0.0148346 0.00883803 2024-09-11T08:14:45.495 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:45.536 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:45.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:45.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:45.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:14:45 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:14:45] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:14:45.638 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:45.672 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:45.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:45.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:45.774 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:14:45.797 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:45.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:45.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:14:45.898 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:45.939 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:45.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:45.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:46.041 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:46.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:45 smithi181 ceph-mon[32701]: pgmap v306: 148 pgs: 3 active+recovering+undersized+degraded+remapped, 145 active+clean; 3.5 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 106 MiB/s wr, 1.70k op/s; 2436/175035 objects degraded (1.392%); 20 MiB/s, 315 objects/s recovering 2024-09-11T08:14:46.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:45 smithi181 ceph-mon[32701]: 5.b scrub starts 2024-09-11T08:14:46.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:45 smithi181 ceph-mon[32701]: Health check update: Degraded data redundancy: 2824/163584 objects degraded (1.726%), 3 pgs degraded (PG_DEGRADED) 2024-09-11T08:14:46.074 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:46.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:46.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:46.124 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:45 smithi150 ceph-mon[29276]: pgmap v306: 148 pgs: 3 active+recovering+undersized+degraded+remapped, 145 active+clean; 3.5 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 106 MiB/s wr, 1.70k op/s; 2436/175035 objects degraded (1.392%); 20 MiB/s, 315 objects/s recovering 2024-09-11T08:14:46.124 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:45 smithi150 ceph-mon[29276]: 5.b scrub starts 2024-09-11T08:14:46.124 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:45 smithi150 ceph-mon[29276]: Health check update: Degraded data redundancy: 2824/163584 objects degraded (1.726%), 3 pgs degraded (PG_DEGRADED) 2024-09-11T08:14:46.126 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:45 smithi150 ceph-mon[39612]: pgmap v306: 148 pgs: 3 active+recovering+undersized+degraded+remapped, 145 active+clean; 3.5 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 106 MiB/s wr, 1.70k op/s; 2436/175035 objects degraded (1.392%); 20 MiB/s, 315 objects/s recovering 2024-09-11T08:14:46.126 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:45 smithi150 ceph-mon[39612]: 5.b scrub starts 2024-09-11T08:14:46.126 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:45 smithi150 ceph-mon[39612]: Health check update: Degraded data redundancy: 2824/163584 objects degraded (1.726%), 3 pgs degraded (PG_DEGRADED) 2024-09-11T08:14:46.176 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:14:46.213 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:46.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:46.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:14:46.315 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:14:46.352 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:46.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:46.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:14:46.410 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 35 16 62922 62906 112.312 86.25 0.00838774 0.00889867 2024-09-11T08:14:46.454 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:46.487 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:46.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:46.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:46.589 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:46.626 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:46.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:46.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:46.728 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:46.762 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:46.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:46.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:46.864 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:46.897 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:46.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:46.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:46.999 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:47.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:47.038 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:47.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:47.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:46 smithi181 ceph-mon[32701]: 4.2 scrub starts 2024-09-11T08:14:47.138 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:47.162 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:47.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:47.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:47.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:46 smithi150 ceph-mon[29276]: 4.2 scrub starts 2024-09-11T08:14:47.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:46 smithi150 ceph-mon[39612]: 4.2 scrub starts 2024-09-11T08:14:47.264 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:47.300 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:47.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:47.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:47.402 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:47.411 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 36 15 64250 64235 111.497 83.0625 0.0115335 0.00896352 2024-09-11T08:14:47.436 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:47.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:47.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:47.538 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:14:47.566 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:47.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:47.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:14:47.668 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:47.699 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:47.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:47.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:47.801 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:14:47.841 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:47.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:47.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:14:47.942 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:47.977 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:47.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:47.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:48.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:47 smithi181 ceph-mon[32701]: 4.4 scrub starts 2024-09-11T08:14:48.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:47 smithi181 ceph-mon[32701]: pgmap v307: 148 pgs: 3 active+recovering+undersized+degraded+remapped, 145 active+clean; 3.5 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 48 MiB/s wr, 763 op/s; 2436/175035 objects degraded (1.392%); 11 MiB/s, 173 objects/s recovering 2024-09-11T08:14:48.080 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:48.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:48.119 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:48.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:48.220 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:48.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:47 smithi150 ceph-mon[29276]: 4.4 scrub starts 2024-09-11T08:14:48.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:47 smithi150 ceph-mon[29276]: pgmap v307: 148 pgs: 3 active+recovering+undersized+degraded+remapped, 145 active+clean; 3.5 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 48 MiB/s wr, 763 op/s; 2436/175035 objects degraded (1.392%); 11 MiB/s, 173 objects/s recovering 2024-09-11T08:14:48.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:47 smithi150 ceph-mon[39612]: 4.4 scrub starts 2024-09-11T08:14:48.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:47 smithi150 ceph-mon[39612]: pgmap v307: 148 pgs: 3 active+recovering+undersized+degraded+remapped, 145 active+clean; 3.5 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 48 MiB/s wr, 763 op/s; 2436/175035 objects degraded (1.392%); 11 MiB/s, 173 objects/s recovering 2024-09-11T08:14:48.252 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:48.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:48.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:48.354 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:48.394 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:48.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:48.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:48.411 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 37 16 65438 65422 110.489 74.1875 0.00634225 0.00904602 2024-09-11T08:14:48.496 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:48.507 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:14:48 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:14:48.539 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:48.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:48.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:48.594 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T08:14:48.640 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:48.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:48.678 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:48.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:48.779 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:48.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:48.815 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:48.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:48.835 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:14:48 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:14:48] "GET /metrics HTTP/1.1" 200 224022 "" "Prometheus/2.43.0" 2024-09-11T08:14:48.848 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-11T08:14:48.848 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:14:48.916 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:48.953 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:48.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:48.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:48.990 INFO:teuthology.orchestra.run.smithi150.stderr:reweighted osd.1 to 1 (10000) 2024-09-11T08:14:49.055 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:49.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:48 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:14:49.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:48 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3063958809' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-11T08:14:49.091 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:49.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:49.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:49.193 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:49.228 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:49.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:49.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:49.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:48 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:14:49.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:48 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3063958809' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-11T08:14:49.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:48 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:14:49.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:48 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3063958809' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-11T08:14:49.330 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:49.363 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:49.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:49.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:49.411 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 38 16 66643 66627 109.563 75.3125 0.010986 0.00912217 2024-09-11T08:14:49.465 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:14:49.503 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:49.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:49.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:14:49.521 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-11T08:14:49.605 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:49.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:49.654 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:49.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:49.755 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:49.811 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:49.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:49.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:49.913 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:49.940 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:49.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:49.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:49.950 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:14:50.042 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:14:50.064 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:50.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:50.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:14:50.149 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:49 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3063958809' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-11T08:14:50.149 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:49 smithi181 ceph-mon[32701]: osdmap e178: 8 total, 8 up, 8 in 2024-09-11T08:14:50.149 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:49 smithi181 ceph-mon[32701]: pgmap v309: 148 pgs: 3 active+recovering+undersized+degraded+remapped, 145 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.53k op/s; 870/195225 objects degraded (0.446%); 22 MiB/s, 352 objects/s recovering 2024-09-11T08:14:50.149 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:49 smithi181 ceph-mon[32701]: 5.12 scrub starts 2024-09-11T08:14:50.167 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:50.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:50.189 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:50.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:50.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:49 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3063958809' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-11T08:14:50.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:49 smithi150 ceph-mon[29276]: osdmap e178: 8 total, 8 up, 8 in 2024-09-11T08:14:50.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:49 smithi150 ceph-mon[29276]: pgmap v309: 148 pgs: 3 active+recovering+undersized+degraded+remapped, 145 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.53k op/s; 870/195225 objects degraded (0.446%); 22 MiB/s, 352 objects/s recovering 2024-09-11T08:14:50.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:49 smithi150 ceph-mon[29276]: 5.12 scrub starts 2024-09-11T08:14:50.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:49 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3063958809' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-11T08:14:50.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:49 smithi150 ceph-mon[39612]: osdmap e178: 8 total, 8 up, 8 in 2024-09-11T08:14:50.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:49 smithi150 ceph-mon[39612]: pgmap v309: 148 pgs: 3 active+recovering+undersized+degraded+remapped, 145 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.53k op/s; 870/195225 objects degraded (0.446%); 22 MiB/s, 352 objects/s recovering 2024-09-11T08:14:50.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:49 smithi150 ceph-mon[39612]: 5.12 scrub starts 2024-09-11T08:14:50.290 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:14:50.312 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:50.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:50.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:14:50.412 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 39 16 67294 67278 107.797 40.6875 0.00317291 0.00915348 2024-09-11T08:14:50.414 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:50.437 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:50.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:50.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:50.539 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:50.560 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:50.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:50.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:50.662 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:50.683 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:50.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:50.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:50.764 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:14:50.785 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:50.806 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:50.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:50.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:50.908 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:50.942 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:50.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:50.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:51.043 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:51.065 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:51.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:51.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:51.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:50 smithi181 ceph-mon[32701]: osdmap e179: 8 total, 8 up, 8 in 2024-09-11T08:14:51.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:50 smithi181 ceph-mon[32701]: 2.1 scrub starts 2024-09-11T08:14:51.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:50 smithi181 ceph-mon[32701]: Health check update: Degraded data redundancy: 870/195225 objects degraded (0.446%), 3 pgs degraded (PG_DEGRADED) 2024-09-11T08:14:51.167 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:51.188 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:50 smithi150 ceph-mon[29276]: osdmap e179: 8 total, 8 up, 8 in 2024-09-11T08:14:51.188 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:50 smithi150 ceph-mon[29276]: 2.1 scrub starts 2024-09-11T08:14:51.188 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:50 smithi150 ceph-mon[29276]: Health check update: Degraded data redundancy: 870/195225 objects degraded (0.446%), 3 pgs degraded (PG_DEGRADED) 2024-09-11T08:14:51.188 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:50 smithi150 ceph-mon[39612]: osdmap e179: 8 total, 8 up, 8 in 2024-09-11T08:14:51.188 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:50 smithi150 ceph-mon[39612]: 2.1 scrub starts 2024-09-11T08:14:51.188 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:50 smithi150 ceph-mon[39612]: Health check update: Degraded data redundancy: 870/195225 objects degraded (0.446%), 3 pgs degraded (PG_DEGRADED) 2024-09-11T08:14:51.189 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:51.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:51.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:51.290 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:51.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:51.313 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:51.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:51.412 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:2024-09-11T08:14:51.412211+0000 min lat: 0.0010471 max lat: 2.12372 avg lat: 0.00937894 2024-09-11T08:14:51.412 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-11T08:14:51.412 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 40 16 67439 67423 105.329 9.0625 0.00298649 0.00937894 2024-09-11T08:14:51.414 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:51.437 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:51.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:51.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:51.539 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:51.560 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:51.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:51.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:51.662 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:51.684 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:51.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:51.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:51.701 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-11T08:14:51.701 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:14:51.786 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:51.807 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:51.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:51.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:51.909 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:51.942 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:51.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:51.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:52.044 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:52.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:51 smithi181 ceph-mon[32701]: 3.1d scrub starts 2024-09-11T08:14:52.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:51 smithi181 ceph-mon[32701]: pgmap v311: 148 pgs: 3 active+recovering+undersized+degraded+remapped, 145 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 82 MiB/s wr, 1.32k op/s; 870/195225 objects degraded (0.446%); 15 MiB/s, 244 objects/s recovering 2024-09-11T08:14:52.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:51 smithi181 ceph-mon[32701]: osdmap e180: 8 total, 8 up, 8 in 2024-09-11T08:14:52.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:51 smithi181 ceph-mon[32701]: 4.5 scrub starts 2024-09-11T08:14:52.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:51 smithi181 ceph-mon[32701]: 4.5 scrub ok 2024-09-11T08:14:52.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:51 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:14:52.075 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:52.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:52.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:52.114 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:51 smithi150 ceph-mon[39612]: 3.1d scrub starts 2024-09-11T08:14:52.114 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:51 smithi150 ceph-mon[39612]: pgmap v311: 148 pgs: 3 active+recovering+undersized+degraded+remapped, 145 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 82 MiB/s wr, 1.32k op/s; 870/195225 objects degraded (0.446%); 15 MiB/s, 244 objects/s recovering 2024-09-11T08:14:52.114 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:51 smithi150 ceph-mon[39612]: osdmap e180: 8 total, 8 up, 8 in 2024-09-11T08:14:52.115 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:51 smithi150 ceph-mon[39612]: 4.5 scrub starts 2024-09-11T08:14:52.115 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:51 smithi150 ceph-mon[39612]: 4.5 scrub ok 2024-09-11T08:14:52.115 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:51 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:14:52.118 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:51 smithi150 ceph-mon[29276]: 3.1d scrub starts 2024-09-11T08:14:52.118 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:51 smithi150 ceph-mon[29276]: pgmap v311: 148 pgs: 3 active+recovering+undersized+degraded+remapped, 145 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 82 MiB/s wr, 1.32k op/s; 870/195225 objects degraded (0.446%); 15 MiB/s, 244 objects/s recovering 2024-09-11T08:14:52.118 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:51 smithi150 ceph-mon[29276]: osdmap e180: 8 total, 8 up, 8 in 2024-09-11T08:14:52.118 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:51 smithi150 ceph-mon[29276]: 4.5 scrub starts 2024-09-11T08:14:52.118 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:51 smithi150 ceph-mon[29276]: 4.5 scrub ok 2024-09-11T08:14:52.118 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:51 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:14:52.119 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd reweight 2 1 2024-09-11T08:14:52.176 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:52.219 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:52.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:52.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:52.321 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:52.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:52.359 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:52.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:52.402 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-11T08:14:52.412 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 41 16 68142 68126 103.832 43.9375 0.00955503 0.00962548 2024-09-11T08:14:52.460 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:52.498 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:52.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:52.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:52.600 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:52.645 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:52.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:52.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:52.747 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:52.797 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:52.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:52.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:52.817 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:14:52.840 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:14:52.900 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:52.934 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:52.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:52.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:52.984 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:52 smithi150 ceph-mon[29276]: 6.7 scrub starts 2024-09-11T08:14:52.984 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:52 smithi150 ceph-mon[29276]: 6.7 scrub ok 2024-09-11T08:14:52.984 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:52 smithi150 ceph-mon[29276]: osdmap e181: 8 total, 8 up, 8 in 2024-09-11T08:14:52.984 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:52 smithi150 ceph-mon[39612]: 6.7 scrub starts 2024-09-11T08:14:52.984 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:52 smithi150 ceph-mon[39612]: 6.7 scrub ok 2024-09-11T08:14:52.985 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:52 smithi150 ceph-mon[39612]: osdmap e181: 8 total, 8 up, 8 in 2024-09-11T08:14:53.036 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:14:53.059 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:53.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:53.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:14:53.161 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:53.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:53.207 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:53.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:53.219 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:52 smithi181 ceph-mon[32701]: 6.7 scrub starts 2024-09-11T08:14:53.219 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:52 smithi181 ceph-mon[32701]: 6.7 scrub ok 2024-09-11T08:14:53.219 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:52 smithi181 ceph-mon[32701]: osdmap e181: 8 total, 8 up, 8 in 2024-09-11T08:14:53.308 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:53.337 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:53.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:53.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:53.412 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 42 16 69397 69381 103.227 78.4375 0.00961035 0.00968118 2024-09-11T08:14:53.439 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:53.474 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:53.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:53.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:53.576 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:53.613 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:53.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:53.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:53.714 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:14:53.736 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:53.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:53.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:14:53.837 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:53.872 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:53.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:53.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:53.974 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:54.013 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:54.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:54.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:54.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:53 smithi181 ceph-mon[32701]: pgmap v314: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 24 MiB/s wr, 389 op/s; 477/199896 objects degraded (0.239%); 6.2 MiB/s, 98 objects/s recovering 2024-09-11T08:14:54.115 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:54.131 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:53 smithi150 ceph-mon[29276]: pgmap v314: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 24 MiB/s wr, 389 op/s; 477/199896 objects degraded (0.239%); 6.2 MiB/s, 98 objects/s recovering 2024-09-11T08:14:54.131 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:53 smithi150 ceph-mon[39612]: pgmap v314: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 24 MiB/s wr, 389 op/s; 477/199896 objects degraded (0.239%); 6.2 MiB/s, 98 objects/s recovering 2024-09-11T08:14:54.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:54.143 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:54.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:54.244 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:54.273 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:54.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:54.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:54.375 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:54.412 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 43 16 70590 70574 102.56 74.5625 0.03308 0.00974448 2024-09-11T08:14:54.418 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:54.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:54.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:54.520 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:54.549 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:54.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:54.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:54.618 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:14:54.650 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:54.678 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:54.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:54.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:54.779 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:54.830 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:54.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:54.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:54.920 INFO:teuthology.orchestra.run.smithi150.stderr:nodeep-scrub is unset 2024-09-11T08:14:54.932 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:54.961 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:54.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:54.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:54.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:54 smithi150 ceph-mon[29276]: 6.18 scrub starts 2024-09-11T08:14:54.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:54 smithi150 ceph-mon[29276]: 6.18 scrub ok 2024-09-11T08:14:54.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:54 smithi150 ceph-mon[29276]: 3.9 scrub starts 2024-09-11T08:14:54.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:54 smithi150 ceph-mon[29276]: 3.9 scrub ok 2024-09-11T08:14:54.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:54 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3874570579' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:14:54.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:54 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:14:54.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:54 smithi150 ceph-mon[39612]: 6.18 scrub starts 2024-09-11T08:14:54.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:54 smithi150 ceph-mon[39612]: 6.18 scrub ok 2024-09-11T08:14:54.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:54 smithi150 ceph-mon[39612]: 3.9 scrub starts 2024-09-11T08:14:54.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:54 smithi150 ceph-mon[39612]: 3.9 scrub ok 2024-09-11T08:14:54.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:54 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3874570579' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:14:54.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:54 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:14:54.996 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:54 smithi181 ceph-mon[32701]: 6.18 scrub starts 2024-09-11T08:14:54.996 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:54 smithi181 ceph-mon[32701]: 6.18 scrub ok 2024-09-11T08:14:54.996 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:54 smithi181 ceph-mon[32701]: 3.9 scrub starts 2024-09-11T08:14:54.996 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:54 smithi181 ceph-mon[32701]: 3.9 scrub ok 2024-09-11T08:14:54.996 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:54 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3874570579' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:14:54.996 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:54 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:14:55.019 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-11T08:14:55.020 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:14:55.063 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:55.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:55.099 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:55.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:55.200 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:55.233 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:55.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:55.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:55.323 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:14:55 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:14:55] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:14:55.335 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:55.360 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:55.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:55.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:55.412 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 44 16 71833 71817 101.994 77.6875 0.0127394 0.00979936 2024-09-11T08:14:55.463 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:55.512 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:55.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:55.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:55.614 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:55.656 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:55.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:55.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:55.758 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:55.794 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:55.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:55.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:55.826 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-11T08:14:55.896 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:55.948 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:55.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:55.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:56.050 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:56.086 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:56.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:56.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:56.188 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:56.231 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:56.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:56.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:56.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:55 smithi150 ceph-mon[29276]: pgmap v315: 148 pgs: 148 active+clean; 4.2 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 42 MiB/s wr, 666 op/s; 9.1 MiB/s, 145 objects/s recovering 2024-09-11T08:14:56.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:55 smithi150 ceph-mon[29276]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 477/199896 objects degraded (0.239%), 1 pg degraded) 2024-09-11T08:14:56.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:55 smithi150 ceph-mon[29276]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:14:56.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:55 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:14:56.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:55 smithi150 ceph-mon[29276]: osdmap e182: 8 total, 8 up, 8 in 2024-09-11T08:14:56.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:55 smithi150 ceph-mon[29276]: osdmap e183: 8 total, 8 up, 8 in 2024-09-11T08:14:56.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:55 smithi150 ceph-mon[39612]: pgmap v315: 148 pgs: 148 active+clean; 4.2 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 42 MiB/s wr, 666 op/s; 9.1 MiB/s, 145 objects/s recovering 2024-09-11T08:14:56.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:55 smithi150 ceph-mon[39612]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 477/199896 objects degraded (0.239%), 1 pg degraded) 2024-09-11T08:14:56.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:55 smithi150 ceph-mon[39612]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:14:56.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:55 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:14:56.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:55 smithi150 ceph-mon[39612]: osdmap e182: 8 total, 8 up, 8 in 2024-09-11T08:14:56.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:55 smithi150 ceph-mon[39612]: osdmap e183: 8 total, 8 up, 8 in 2024-09-11T08:14:56.272 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:14:56.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:55 smithi181 ceph-mon[32701]: pgmap v315: 148 pgs: 148 active+clean; 4.2 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 42 MiB/s wr, 666 op/s; 9.1 MiB/s, 145 objects/s recovering 2024-09-11T08:14:56.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:55 smithi181 ceph-mon[32701]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 477/199896 objects degraded (0.239%), 1 pg degraded) 2024-09-11T08:14:56.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:55 smithi181 ceph-mon[32701]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:14:56.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:55 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:14:56.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:55 smithi181 ceph-mon[32701]: osdmap e182: 8 total, 8 up, 8 in 2024-09-11T08:14:56.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:55 smithi181 ceph-mon[32701]: osdmap e183: 8 total, 8 up, 8 in 2024-09-11T08:14:56.333 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:56.373 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:56.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:56.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:56.412 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 45 16 73232 73216 101.671 87.4375 0.00984834 0.00983054 2024-09-11T08:14:56.475 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:56.513 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:56.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:56.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:56.615 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:56.655 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:56.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:56.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:56.757 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:56.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:56.792 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:56.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:56.893 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:56.935 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:56.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:56.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:57.037 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:14:57.068 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:57.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:57.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:14:57.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:56 smithi181 ceph-mon[32701]: 3.1c scrub starts 2024-09-11T08:14:57.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:56 smithi181 ceph-mon[32701]: 3.1c scrub ok 2024-09-11T08:14:57.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:56 smithi181 ceph-mon[32701]: 5.1e scrub starts 2024-09-11T08:14:57.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:56 smithi181 ceph-mon[32701]: 5.1e scrub ok 2024-09-11T08:14:57.170 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:57.199 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:57.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:57.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:57.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:56 smithi150 ceph-mon[29276]: 3.1c scrub starts 2024-09-11T08:14:57.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:56 smithi150 ceph-mon[29276]: 3.1c scrub ok 2024-09-11T08:14:57.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:56 smithi150 ceph-mon[29276]: 5.1e scrub starts 2024-09-11T08:14:57.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:56 smithi150 ceph-mon[29276]: 5.1e scrub ok 2024-09-11T08:14:57.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:56 smithi150 ceph-mon[39612]: 3.1c scrub starts 2024-09-11T08:14:57.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:56 smithi150 ceph-mon[39612]: 3.1c scrub ok 2024-09-11T08:14:57.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:56 smithi150 ceph-mon[39612]: 5.1e scrub starts 2024-09-11T08:14:57.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:56 smithi150 ceph-mon[39612]: 5.1e scrub ok 2024-09-11T08:14:57.301 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:57.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:57.335 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:57.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:57.412 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 46 16 74634 74618 101.365 87.625 0.00652178 0.00986072 2024-09-11T08:14:57.436 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:57.470 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:57.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:57.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:57.572 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:57.618 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:57.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:57.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:57.720 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:57.760 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:57.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:57.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:57.862 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:57.904 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:57.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:57.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:58.006 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:58.040 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:58.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:58.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:58.151 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:14:58.190 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:58.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:58.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:14:58.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:57 smithi150 ceph-mon[29276]: 4.14 scrub starts 2024-09-11T08:14:58.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:57 smithi150 ceph-mon[29276]: 4.14 scrub ok 2024-09-11T08:14:58.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:57 smithi150 ceph-mon[29276]: 6.15 scrub starts 2024-09-11T08:14:58.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:57 smithi150 ceph-mon[29276]: 6.15 scrub ok 2024-09-11T08:14:58.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:57 smithi150 ceph-mon[29276]: pgmap v318: 148 pgs: 148 active+clean; 4.2 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 856 B/s rd, 42 MiB/s wr, 668 op/s; 9.1 MiB/s, 145 objects/s recovering 2024-09-11T08:14:58.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:57 smithi150 ceph-mon[29276]: 5.d deep-scrub starts 2024-09-11T08:14:58.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:57 smithi150 ceph-mon[29276]: 5.d deep-scrub ok 2024-09-11T08:14:58.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:57 smithi150 ceph-mon[29276]: 5.19 scrub starts 2024-09-11T08:14:58.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:57 smithi150 ceph-mon[29276]: 5.19 scrub ok 2024-09-11T08:14:58.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:57 smithi150 ceph-mon[39612]: 4.14 scrub starts 2024-09-11T08:14:58.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:57 smithi150 ceph-mon[39612]: 4.14 scrub ok 2024-09-11T08:14:58.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:57 smithi150 ceph-mon[39612]: 6.15 scrub starts 2024-09-11T08:14:58.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:57 smithi150 ceph-mon[39612]: 6.15 scrub ok 2024-09-11T08:14:58.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:57 smithi150 ceph-mon[39612]: pgmap v318: 148 pgs: 148 active+clean; 4.2 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 856 B/s rd, 42 MiB/s wr, 668 op/s; 9.1 MiB/s, 145 objects/s recovering 2024-09-11T08:14:58.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:57 smithi150 ceph-mon[39612]: 5.d deep-scrub starts 2024-09-11T08:14:58.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:57 smithi150 ceph-mon[39612]: 5.d deep-scrub ok 2024-09-11T08:14:58.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:57 smithi150 ceph-mon[39612]: 5.19 scrub starts 2024-09-11T08:14:58.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:57 smithi150 ceph-mon[39612]: 5.19 scrub ok 2024-09-11T08:14:58.243 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:57 smithi181 ceph-mon[32701]: 4.14 scrub starts 2024-09-11T08:14:58.243 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:57 smithi181 ceph-mon[32701]: 4.14 scrub ok 2024-09-11T08:14:58.243 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:57 smithi181 ceph-mon[32701]: 6.15 scrub starts 2024-09-11T08:14:58.244 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:57 smithi181 ceph-mon[32701]: 6.15 scrub ok 2024-09-11T08:14:58.244 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:57 smithi181 ceph-mon[32701]: pgmap v318: 148 pgs: 148 active+clean; 4.2 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 856 B/s rd, 42 MiB/s wr, 668 op/s; 9.1 MiB/s, 145 objects/s recovering 2024-09-11T08:14:58.244 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:57 smithi181 ceph-mon[32701]: 5.d deep-scrub starts 2024-09-11T08:14:58.244 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:57 smithi181 ceph-mon[32701]: 5.d deep-scrub ok 2024-09-11T08:14:58.244 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:57 smithi181 ceph-mon[32701]: 5.19 scrub starts 2024-09-11T08:14:58.244 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:57 smithi181 ceph-mon[32701]: 5.19 scrub ok 2024-09-11T08:14:58.244 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:14:58 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:14:58.287 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-11T08:14:58.287 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:14:58.292 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:58.328 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:58.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:58.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:58.413 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 47 16 75986 75970 101.006 84.5 0.0132993 0.0098951 2024-09-11T08:14:58.430 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:14:58.443 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:14:58.443 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:14:58.458 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:58.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:58.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:14:58.480 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:14:58.480 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:14:58.512 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:14:58.512 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:14:58.545 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:14:58.546 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:14:58.560 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:58.577 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:14:58.577 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:14:58.589 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:58.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:58.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:58.609 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:14:58.609 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:14:58.641 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:14:58.641 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:14:58.676 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:14:58.676 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:14:58.690 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:14:58.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:58.740 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:58.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:14:58.841 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:14:58.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:58.875 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:58.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:14:58.941 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:14:58 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:14:58] "GET /metrics HTTP/1.1" 200 224136 "" "Prometheus/2.43.0" 2024-09-11T08:14:58.976 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:14:59.001 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:59.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:59.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:14:59.013 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-11T08:14:59.103 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:59.138 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:59.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:59.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:59.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:58 smithi150 ceph-mon[29276]: 6.c scrub starts 2024-09-11T08:14:59.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:58 smithi150 ceph-mon[29276]: 6.c scrub ok 2024-09-11T08:14:59.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:58 smithi150 ceph-mon[29276]: 2.2 scrub starts 2024-09-11T08:14:59.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:58 smithi150 ceph-mon[29276]: 2.2 scrub ok 2024-09-11T08:14:59.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:58 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:14:59.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:58 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2376148952' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:14:59.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:58 smithi150 ceph-mon[39612]: 6.c scrub starts 2024-09-11T08:14:59.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:58 smithi150 ceph-mon[39612]: 6.c scrub ok 2024-09-11T08:14:59.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:58 smithi150 ceph-mon[39612]: 2.2 scrub starts 2024-09-11T08:14:59.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:58 smithi150 ceph-mon[39612]: 2.2 scrub ok 2024-09-11T08:14:59.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:58 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:14:59.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:58 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2376148952' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:14:59.240 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:59.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:58 smithi181 ceph-mon[32701]: 6.c scrub starts 2024-09-11T08:14:59.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:58 smithi181 ceph-mon[32701]: 6.c scrub ok 2024-09-11T08:14:59.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:58 smithi181 ceph-mon[32701]: 2.2 scrub starts 2024-09-11T08:14:59.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:58 smithi181 ceph-mon[32701]: 2.2 scrub ok 2024-09-11T08:14:59.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:58 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:14:59.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:58 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2376148952' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:14:59.268 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:59.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:59.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:14:59.321 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T08:14:59.370 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:59.413 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 48 16 77186 77170 100.464 75 0.00318084 0.00994745 2024-09-11T08:14:59.413 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:59.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:59.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:59.435 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:14:59.515 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:14:59.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:59.551 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:14:59.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:14:59.652 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:14:59.688 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:59.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:59.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:14:59.789 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:14:59.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:59.833 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:59.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:14:59.934 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:14:59.972 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:14:59.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:14:59.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:00.074 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:15:00.111 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:59 smithi150 ceph-mon[29276]: 6.1b scrub starts 2024-09-11T08:15:00.112 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:59 smithi150 ceph-mon[29276]: 6.1b scrub ok 2024-09-11T08:15:00.112 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:59 smithi150 ceph-mon[29276]: 6.5 deep-scrub starts 2024-09-11T08:15:00.112 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:59 smithi150 ceph-mon[29276]: 6.5 deep-scrub ok 2024-09-11T08:15:00.112 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:59 smithi150 ceph-mon[29276]: pgmap v319: 148 pgs: 148 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.4 KiB/s rd, 94 MiB/s wr, 1.51k op/s; 7.8 MiB/s, 125 objects/s recovering 2024-09-11T08:15:00.112 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:59 smithi150 ceph-mon[29276]: 5.6 deep-scrub starts 2024-09-11T08:15:00.112 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:59 smithi150 ceph-mon[29276]: 5.6 deep-scrub ok 2024-09-11T08:15:00.112 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:59 smithi150 ceph-mon[29276]: 4.1 scrub starts 2024-09-11T08:15:00.112 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:14:59 smithi150 ceph-mon[29276]: 4.1 scrub ok 2024-09-11T08:15:00.112 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:59 smithi150 ceph-mon[39612]: 6.1b scrub starts 2024-09-11T08:15:00.113 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:59 smithi150 ceph-mon[39612]: 6.1b scrub ok 2024-09-11T08:15:00.113 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:59 smithi150 ceph-mon[39612]: 6.5 deep-scrub starts 2024-09-11T08:15:00.113 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:59 smithi150 ceph-mon[39612]: 6.5 deep-scrub ok 2024-09-11T08:15:00.113 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:59 smithi150 ceph-mon[39612]: pgmap v319: 148 pgs: 148 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.4 KiB/s rd, 94 MiB/s wr, 1.51k op/s; 7.8 MiB/s, 125 objects/s recovering 2024-09-11T08:15:00.113 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:59 smithi150 ceph-mon[39612]: 5.6 deep-scrub starts 2024-09-11T08:15:00.113 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:59 smithi150 ceph-mon[39612]: 5.6 deep-scrub ok 2024-09-11T08:15:00.113 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:59 smithi150 ceph-mon[39612]: 4.1 scrub starts 2024-09-11T08:15:00.113 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:14:59 smithi150 ceph-mon[39612]: 4.1 scrub ok 2024-09-11T08:15:00.114 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:00.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:00.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:15:00.148 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:59 smithi181 ceph-mon[32701]: 6.1b scrub starts 2024-09-11T08:15:00.148 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:59 smithi181 ceph-mon[32701]: 6.1b scrub ok 2024-09-11T08:15:00.148 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:59 smithi181 ceph-mon[32701]: 6.5 deep-scrub starts 2024-09-11T08:15:00.148 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:59 smithi181 ceph-mon[32701]: 6.5 deep-scrub ok 2024-09-11T08:15:00.149 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:59 smithi181 ceph-mon[32701]: pgmap v319: 148 pgs: 148 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.4 KiB/s rd, 94 MiB/s wr, 1.51k op/s; 7.8 MiB/s, 125 objects/s recovering 2024-09-11T08:15:00.149 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:59 smithi181 ceph-mon[32701]: 5.6 deep-scrub starts 2024-09-11T08:15:00.149 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:59 smithi181 ceph-mon[32701]: 5.6 deep-scrub ok 2024-09-11T08:15:00.149 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:59 smithi181 ceph-mon[32701]: 4.1 scrub starts 2024-09-11T08:15:00.149 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:14:59 smithi181 ceph-mon[32701]: 4.1 scrub ok 2024-09-11T08:15:00.216 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:00.243 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:00.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:00.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:00.345 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:00.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:00.382 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:00.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:00.413 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 49 16 78333 78317 99.8769 71.6875 0.0138772 0.0100074 2024-09-11T08:15:00.482 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:15:00.489 INFO:teuthology.orchestra.run.smithi150.stderr:reweighted osd.2 to 1 (10000) 2024-09-11T08:15:00.532 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:00.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:00.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:15:00.634 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:15:00.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:00.665 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:00.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:15:00.766 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:15:00.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:00.801 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:00.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:15:00.901 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:15:00.937 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:00.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:00.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:15:01.020 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:00 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:15:01.020 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:00 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:15:01.020 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:00 smithi181 ceph-mon[32701]: 5.b scrub starts 2024-09-11T08:15:01.020 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:00 smithi181 ceph-mon[32701]: 5.b scrub ok 2024-09-11T08:15:01.020 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:00 smithi181 ceph-mon[32701]: osdmap e184: 8 total, 8 up, 8 in 2024-09-11T08:15:01.020 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:00 smithi181 ceph-mon[32701]: 3.a scrub starts 2024-09-11T08:15:01.020 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:00 smithi181 ceph-mon[32701]: 3.a scrub ok 2024-09-11T08:15:01.020 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:00 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3425453229' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-11T08:15:01.020 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:00 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-11T08:15:01.021 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:00 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-11T08:15:01.021 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:00 smithi181 ceph-mon[32701]: osdmap e185: 8 total, 8 up, 8 in 2024-09-11T08:15:01.039 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:01.068 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:01.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:01.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:01.170 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:01.206 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:01.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:01.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:01.208 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:00 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:15:01.208 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:00 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:15:01.208 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:00 smithi150 ceph-mon[29276]: 5.b scrub starts 2024-09-11T08:15:01.208 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:00 smithi150 ceph-mon[29276]: 5.b scrub ok 2024-09-11T08:15:01.208 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:00 smithi150 ceph-mon[29276]: osdmap e184: 8 total, 8 up, 8 in 2024-09-11T08:15:01.208 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:00 smithi150 ceph-mon[29276]: 3.a scrub starts 2024-09-11T08:15:01.208 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:00 smithi150 ceph-mon[29276]: 3.a scrub ok 2024-09-11T08:15:01.209 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:00 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3425453229' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-11T08:15:01.209 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:00 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-11T08:15:01.209 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:00 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-11T08:15:01.209 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:00 smithi150 ceph-mon[29276]: osdmap e185: 8 total, 8 up, 8 in 2024-09-11T08:15:01.209 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:00 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:15:01.209 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:00 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:15:01.209 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:00 smithi150 ceph-mon[39612]: 5.b scrub starts 2024-09-11T08:15:01.209 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:00 smithi150 ceph-mon[39612]: 5.b scrub ok 2024-09-11T08:15:01.209 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:00 smithi150 ceph-mon[39612]: osdmap e184: 8 total, 8 up, 8 in 2024-09-11T08:15:01.209 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:00 smithi150 ceph-mon[39612]: 3.a scrub starts 2024-09-11T08:15:01.210 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:00 smithi150 ceph-mon[39612]: 3.a scrub ok 2024-09-11T08:15:01.210 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:00 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3425453229' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-11T08:15:01.210 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:00 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-11T08:15:01.210 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:00 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-11T08:15:01.210 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:00 smithi150 ceph-mon[39612]: osdmap e185: 8 total, 8 up, 8 in 2024-09-11T08:15:01.308 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:01.341 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:01.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:01.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:01.413 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 50 16 79677 79661 99.5593 84 0.00627329 0.0100396 2024-09-11T08:15:01.442 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:15:01.446 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-11T08:15:01.446 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-09-11T08:15:01.447 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.447 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15087.0:79716 8.9 8:92bc3cd9:::benchmark_data_smithi150_107646_object79715:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-11T08:15:01.447 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:15:01.440393+0000", 2024-09-11T08:15:01.447 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.0046392050000000004, 2024-09-11T08:15:01.447 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0046729759999999997, 2024-09-11T08:15:01.447 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:15:01.447 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "waiting for sub ops", 2024-09-11T08:15:01.447 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:15:01.447 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15087", 2024-09-11T08:15:01.447 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/3299560913", 2024-09-11T08:15:01.447 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 79716 2024-09-11T08:15:01.447 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.448 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:15:01.448 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.448 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:15:01.448 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.440393+0000", 2024-09-11T08:15:01.448 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:01.448 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.448 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.448 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:15:01.448 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.440393+0000", 2024-09-11T08:15:01.448 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:01.448 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.448 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.448 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:15:01.449 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.440392+0000", 2024-09-11T08:15:01.449 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:15:01.449 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.449 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.449 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:15:01.449 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.440421+0000", 2024-09-11T08:15:01.449 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9709999999999998e-05 2024-09-11T08:15:01.449 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.449 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.449 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:15:01.449 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.440437+0000", 2024-09-11T08:15:01.449 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6195999999999999e-05 2024-09-11T08:15:01.450 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.450 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.450 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:15:01.450 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.440450+0000", 2024-09-11T08:15:01.450 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2873e-05 2024-09-11T08:15:01.450 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.450 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.450 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:15:01.450 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.440489+0000", 2024-09-11T08:15:01.450 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8964000000000002e-05 2024-09-11T08:15:01.450 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.450 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.450 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:15:01.451 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.440631+0000", 2024-09-11T08:15:01.451 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000141454 2024-09-11T08:15:01.451 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.451 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.451 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,5", 2024-09-11T08:15:01.451 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.440792+0000", 2024-09-11T08:15:01.451 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000160877 2024-09-11T08:15:01.451 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:01.451 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:15:01.451 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:01.451 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.451 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.451 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.15087.0:79718 8.d e185/159 8:b32b5092:::benchmark_data_smithi150_107646_object79717:head v 185'4952, mlcod=185'4950)", 2024-09-11T08:15:01.452 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:15:01.441257+0000", 2024-09-11T08:15:01.452 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.0037746659999999999, 2024-09-11T08:15:01.452 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0038837170000000001, 2024-09-11T08:15:01.452 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:15:01.452 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-11T08:15:01.452 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:15:01.452 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.452 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:15:01.452 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.441257+0000", 2024-09-11T08:15:01.452 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:01.452 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.453 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.453 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:15:01.453 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.441257+0000", 2024-09-11T08:15:01.453 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:01.453 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.453 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.453 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:15:01.453 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.441257+0000", 2024-09-11T08:15:01.453 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:15:01.453 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.453 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.453 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:15:01.454 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.441278+0000", 2024-09-11T08:15:01.454 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1203e-05 2024-09-11T08:15:01.454 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.454 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.454 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:15:01.454 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.441288+0000", 2024-09-11T08:15:01.454 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0599e-05 2024-09-11T08:15:01.454 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.454 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.454 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:15:01.454 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.441294+0000", 2024-09-11T08:15:01.454 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4890000000000003e-06 2024-09-11T08:15:01.455 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.455 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.455 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:15:01.455 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.441323+0000", 2024-09-11T08:15:01.455 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8699000000000001e-05 2024-09-11T08:15:01.455 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.455 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.455 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:15:01.455 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.441340+0000", 2024-09-11T08:15:01.455 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7263999999999999e-05 2024-09-11T08:15:01.455 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:01.455 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:15:01.456 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:01.456 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.456 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.456 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15087.0:79719 8.9 8.659fe6b9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-11T08:15:01.456 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:15:01.441475+0000", 2024-09-11T08:15:01.456 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.0035565660000000002, 2024-09-11T08:15:01.456 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003717878, 2024-09-11T08:15:01.456 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:15:01.456 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "queued for pg", 2024-09-11T08:15:01.456 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:15:01.456 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15087", 2024-09-11T08:15:01.457 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/3299560913", 2024-09-11T08:15:01.457 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 79719 2024-09-11T08:15:01.457 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.457 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:15:01.457 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.457 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:15:01.457 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.441475+0000", 2024-09-11T08:15:01.457 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:01.457 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.457 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.457 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:15:01.457 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.441475+0000", 2024-09-11T08:15:01.457 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:01.458 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.458 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.458 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:15:01.458 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.441475+0000", 2024-09-11T08:15:01.458 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:15:01.458 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.458 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.458 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:15:01.458 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.441537+0000", 2024-09-11T08:15:01.458 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2050000000000004e-05 2024-09-11T08:15:01.458 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.458 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.459 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:15:01.459 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.441546+0000", 2024-09-11T08:15:01.459 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0450000000000006e-06 2024-09-11T08:15:01.459 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.459 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.459 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:15:01.459 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.441551+0000", 2024-09-11T08:15:01.459 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2580000000000003e-06 2024-09-11T08:15:01.459 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:01.459 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:15:01.459 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:01.459 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.459 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.460 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.15087.0:79712 8.6 e185/180 8:632049d0:::benchmark_data_smithi150_107646_object79711:head v 185'5028, mlcod=185'5025)", 2024-09-11T08:15:01.460 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:15:01.442877+0000", 2024-09-11T08:15:01.460 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.0021554590000000002, 2024-09-11T08:15:01.460 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0023660949999999999, 2024-09-11T08:15:01.460 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:15:01.460 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-11T08:15:01.460 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:15:01.460 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.460 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:15:01.460 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.442877+0000", 2024-09-11T08:15:01.460 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:01.460 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.460 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.461 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:15:01.461 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.442877+0000", 2024-09-11T08:15:01.461 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:01.461 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.461 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.461 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:15:01.461 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.442876+0000", 2024-09-11T08:15:01.461 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-09-11T08:15:01.461 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.461 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.461 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:15:01.461 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.442960+0000", 2024-09-11T08:15:01.461 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4340999999999998e-05 2024-09-11T08:15:01.462 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.462 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.462 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:15:01.462 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.442970+0000", 2024-09-11T08:15:01.462 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0234000000000001e-05 2024-09-11T08:15:01.462 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.462 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.462 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:15:01.463 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.442975+0000", 2024-09-11T08:15:01.463 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.211e-06 2024-09-11T08:15:01.463 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.463 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.463 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:15:01.463 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.443004+0000", 2024-09-11T08:15:01.463 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9564999999999999e-05 2024-09-11T08:15:01.463 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.463 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.463 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:15:01.464 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.443022+0000", 2024-09-11T08:15:01.464 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7504999999999999e-05 2024-09-11T08:15:01.464 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:01.464 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:15:01.464 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:01.464 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.464 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.464 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15087.0:79721 8.5 8:a292f06c:::benchmark_data_smithi150_107646_object79720:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-11T08:15:01.464 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:15:01.443409+0000", 2024-09-11T08:15:01.464 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.0016233669999999999, 2024-09-11T08:15:01.464 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001890777, 2024-09-11T08:15:01.464 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:15:01.465 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "waiting for sub ops", 2024-09-11T08:15:01.465 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:15:01.465 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15087", 2024-09-11T08:15:01.465 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/3299560913", 2024-09-11T08:15:01.465 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 79721 2024-09-11T08:15:01.465 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.465 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:15:01.465 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.465 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:15:01.465 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.443409+0000", 2024-09-11T08:15:01.465 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:01.465 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.466 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.466 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:15:01.466 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.443409+0000", 2024-09-11T08:15:01.466 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:01.466 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.466 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.466 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:15:01.466 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.443407+0000", 2024-09-11T08:15:01.466 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:15:01.466 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.466 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.466 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:15:01.466 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.443460+0000", 2024-09-11T08:15:01.467 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2692000000000003e-05 2024-09-11T08:15:01.467 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.467 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.467 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:15:01.467 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.443469+0000", 2024-09-11T08:15:01.467 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3489999999999992e-06 2024-09-11T08:15:01.467 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.467 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.467 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:15:01.467 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.443473+0000", 2024-09-11T08:15:01.467 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3209999999999998e-06 2024-09-11T08:15:01.467 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.468 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.468 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:15:01.468 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.443532+0000", 2024-09-11T08:15:01.468 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9063e-05 2024-09-11T08:15:01.468 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.468 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.468 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:15:01.468 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.443653+0000", 2024-09-11T08:15:01.468 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000121427 2024-09-11T08:15:01.468 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.468 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.468 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 4,7", 2024-09-11T08:15:01.468 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.443788+0000", 2024-09-11T08:15:01.469 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013425199999999999 2024-09-11T08:15:01.469 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:01.469 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:15:01.469 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:01.469 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.469 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.469 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop_reply(client.15087.0:79696 8.9 e185/159)", 2024-09-11T08:15:01.469 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:15:01.443824+0000", 2024-09-11T08:15:01.469 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.0012078220000000001, 2024-09-11T08:15:01.469 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0015354660000000001, 2024-09-11T08:15:01.469 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:15:01.469 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "queued for pg", 2024-09-11T08:15:01.470 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:15:01.470 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.470 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:15:01.470 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.443824+0000", 2024-09-11T08:15:01.470 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:01.470 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.470 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.470 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:15:01.470 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.443824+0000", 2024-09-11T08:15:01.470 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:01.470 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.470 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.470 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:15:01.471 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.443824+0000", 2024-09-11T08:15:01.471 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-09-11T08:15:01.471 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.471 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.471 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:15:01.471 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.443828+0000", 2024-09-11T08:15:01.471 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4569999999999998e-06 2024-09-11T08:15:01.471 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.471 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.471 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:15:01.471 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.443834+0000", 2024-09-11T08:15:01.471 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0719999999999996e-06 2024-09-11T08:15:01.472 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.472 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.472 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:15:01.472 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.443839+0000", 2024-09-11T08:15:01.472 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3170000000000001e-06 2024-09-11T08:15:01.472 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:01.472 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:15:01.472 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:01.472 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.472 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.472 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop_reply(client.15087.0:79716 8.9 e185/159)", 2024-09-11T08:15:01.472 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:15:01.443926+0000", 2024-09-11T08:15:01.472 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.0011063410000000001, 2024-09-11T08:15:01.473 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0014724849999999999, 2024-09-11T08:15:01.473 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:15:01.473 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "queued for pg", 2024-09-11T08:15:01.473 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:15:01.473 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.473 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:15:01.473 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.443926+0000", 2024-09-11T08:15:01.473 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:01.473 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.473 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.473 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:15:01.473 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.443926+0000", 2024-09-11T08:15:01.473 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:01.474 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.474 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.474 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:15:01.474 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.443925+0000", 2024-09-11T08:15:01.474 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:15:01.474 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.474 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.474 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:15:01.474 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.443930+0000", 2024-09-11T08:15:01.474 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8389999999999997e-06 2024-09-11T08:15:01.474 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.474 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.474 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:15:01.475 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.443940+0000", 2024-09-11T08:15:01.475 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7380000000000006e-06 2024-09-11T08:15:01.475 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.475 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.475 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:15:01.475 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.443945+0000", 2024-09-11T08:15:01.475 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1789999999999999e-06 2024-09-11T08:15:01.475 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:01.475 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:15:01.475 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:01.475 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.475 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.475 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15087.0:79696 8.9 8:9b46a9f9:::benchmark_data_smithi150_107646_object79695:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-11T08:15:01.475 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:15:01.426154+0000", 2024-09-11T08:15:01.476 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.018878286000000001, 2024-09-11T08:15:01.476 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.019309142000000001, 2024-09-11T08:15:01.476 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:15:01.476 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "waiting for sub ops", 2024-09-11T08:15:01.476 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:15:01.476 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15087", 2024-09-11T08:15:01.476 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/3299560913", 2024-09-11T08:15:01.476 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 79696 2024-09-11T08:15:01.476 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.476 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:15:01.476 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.476 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:15:01.476 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.426154+0000", 2024-09-11T08:15:01.477 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:01.477 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.477 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.477 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:15:01.477 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.426154+0000", 2024-09-11T08:15:01.477 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:01.477 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.477 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.477 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:15:01.477 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.426152+0000", 2024-09-11T08:15:01.477 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999981 2024-09-11T08:15:01.478 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.478 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.478 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:15:01.478 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.426206+0000", 2024-09-11T08:15:01.478 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3946000000000002e-05 2024-09-11T08:15:01.478 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.478 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.478 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:15:01.478 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.426222+0000", 2024-09-11T08:15:01.478 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6538999999999999e-05 2024-09-11T08:15:01.478 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.478 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.478 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:15:01.479 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.426230+0000", 2024-09-11T08:15:01.479 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8189999999999992e-06 2024-09-11T08:15:01.479 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.479 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.479 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:15:01.479 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.429924+0000", 2024-09-11T08:15:01.479 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0036939260000000002 2024-09-11T08:15:01.479 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.479 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.479 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:15:01.480 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.430030+0000", 2024-09-11T08:15:01.480 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010554 2024-09-11T08:15:01.480 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.480 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.480 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,5", 2024-09-11T08:15:01.480 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.430111+0000", 2024-09-11T08:15:01.480 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1080000000000003e-05 2024-09-11T08:15:01.480 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.480 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.480 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:15:01.481 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.433022+0000", 2024-09-11T08:15:01.481 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002910723 2024-09-11T08:15:01.481 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:01.481 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:01.481 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:01.481 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:01.436952+0000", 2024-09-11T08:15:01.481 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0039301520000000001 2024-09-11T08:15:01.481 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:01.481 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:15:01.481 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:01.481 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:01.481 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-09-11T08:15:01.482 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 8 2024-09-11T08:15:01.482 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-11T08:15:01.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:01.483 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:01.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:15:01.584 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:15:01.625 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:01.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:01.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:15:01.727 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:01.766 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:01.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:01.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:01.844 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd reweight 3 1 2024-09-11T08:15:01.868 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:01.941 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:01.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:01.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:01.960 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:15:02.043 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:02.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:02.088 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:02.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:02.105 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:01 smithi150 ceph-mon[29276]: pgmap v322: 148 pgs: 148 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 862 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-09-11T08:15:02.105 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:01 smithi150 ceph-mon[29276]: 5.1f scrub starts 2024-09-11T08:15:02.105 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:01 smithi150 ceph-mon[29276]: 5.1f scrub ok 2024-09-11T08:15:02.106 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:01 smithi150 ceph-mon[29276]: 6.16 scrub starts 2024-09-11T08:15:02.106 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:01 smithi150 ceph-mon[29276]: 6.16 scrub ok 2024-09-11T08:15:02.106 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:01 smithi150 ceph-mon[39612]: pgmap v322: 148 pgs: 148 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 862 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-09-11T08:15:02.106 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:01 smithi150 ceph-mon[39612]: 5.1f scrub starts 2024-09-11T08:15:02.106 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:01 smithi150 ceph-mon[39612]: 5.1f scrub ok 2024-09-11T08:15:02.106 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:01 smithi150 ceph-mon[39612]: 6.16 scrub starts 2024-09-11T08:15:02.106 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:01 smithi150 ceph-mon[39612]: 6.16 scrub ok 2024-09-11T08:15:02.189 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:02.197 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-11T08:15:02.220 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:02.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:02.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:02.255 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:01 smithi181 ceph-mon[32701]: pgmap v322: 148 pgs: 148 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 862 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-09-11T08:15:02.255 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:01 smithi181 ceph-mon[32701]: 5.1f scrub starts 2024-09-11T08:15:02.255 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:01 smithi181 ceph-mon[32701]: 5.1f scrub ok 2024-09-11T08:15:02.255 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:01 smithi181 ceph-mon[32701]: 6.16 scrub starts 2024-09-11T08:15:02.255 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:01 smithi181 ceph-mon[32701]: 6.16 scrub ok 2024-09-11T08:15:02.323 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:15:02.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:02.371 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:02.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:15:02.413 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 51 16 80958 80942 99.1769 80.0625 0.0115798 0.0100786 2024-09-11T08:15:02.473 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:15:02.513 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:02.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:02.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:15:02.610 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:15:02.614 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:15:02.656 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:02.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:02.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:15:02.758 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:02.800 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:02.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:02.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:02.902 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:02.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:02.943 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:02.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:02.971 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T08:15:03.016 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:02 smithi181 ceph-mon[32701]: 5.13 deep-scrub starts 2024-09-11T08:15:03.016 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:02 smithi181 ceph-mon[32701]: 5.13 deep-scrub ok 2024-09-11T08:15:03.016 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:02 smithi181 ceph-mon[32701]: 2.1 deep-scrub starts 2024-09-11T08:15:03.016 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:02 smithi181 ceph-mon[32701]: 2.1 deep-scrub ok 2024-09-11T08:15:03.044 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:15:03.072 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:03.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:03.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:15:03.174 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:15:03.219 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:02 smithi150 ceph-mon[29276]: 5.13 deep-scrub starts 2024-09-11T08:15:03.220 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:02 smithi150 ceph-mon[29276]: 5.13 deep-scrub ok 2024-09-11T08:15:03.220 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:02 smithi150 ceph-mon[29276]: 2.1 deep-scrub starts 2024-09-11T08:15:03.220 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:02 smithi150 ceph-mon[29276]: 2.1 deep-scrub ok 2024-09-11T08:15:03.220 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:02 smithi150 ceph-mon[39612]: 5.13 deep-scrub starts 2024-09-11T08:15:03.220 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:02 smithi150 ceph-mon[39612]: 5.13 deep-scrub ok 2024-09-11T08:15:03.220 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:02 smithi150 ceph-mon[39612]: 2.1 deep-scrub starts 2024-09-11T08:15:03.220 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:02 smithi150 ceph-mon[39612]: 2.1 deep-scrub ok 2024-09-11T08:15:03.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:03.237 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:03.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:15:03.292 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:15:03.338 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:15:03.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:03.417 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:03.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:15:03.420 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 52 16 82198 82182 98.7578 77.5 0.0138089 0.0101208 2024-09-11T08:15:03.519 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:15:03.568 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:03.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:03.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:15:03.670 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:03.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:03.707 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:03.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:03.808 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:15:03.847 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:03.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:03.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:15:03.950 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:03.992 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:03.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:03.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:04.094 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:15:04.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:04.137 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:04.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:15:04.238 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:04.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:03 smithi150 ceph-mon[29276]: 5.16 deep-scrub starts 2024-09-11T08:15:04.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:03 smithi150 ceph-mon[29276]: 5.16 deep-scrub ok 2024-09-11T08:15:04.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:03 smithi150 ceph-mon[29276]: pgmap v323: 148 pgs: 148 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.4 KiB/s rd, 75 MiB/s wr, 1.21k op/s 2024-09-11T08:15:04.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:03 smithi150 ceph-mon[29276]: 4.1b scrub starts 2024-09-11T08:15:04.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:03 smithi150 ceph-mon[29276]: 4.1b scrub ok 2024-09-11T08:15:04.242 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:03 smithi150 ceph-mon[39612]: 5.16 deep-scrub starts 2024-09-11T08:15:04.242 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:03 smithi150 ceph-mon[39612]: 5.16 deep-scrub ok 2024-09-11T08:15:04.242 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:03 smithi150 ceph-mon[39612]: pgmap v323: 148 pgs: 148 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.4 KiB/s rd, 75 MiB/s wr, 1.21k op/s 2024-09-11T08:15:04.243 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:03 smithi150 ceph-mon[39612]: 4.1b scrub starts 2024-09-11T08:15:04.243 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:03 smithi150 ceph-mon[39612]: 4.1b scrub ok 2024-09-11T08:15:04.265 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:04.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:04.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:04.308 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:03 smithi181 ceph-mon[32701]: 5.16 deep-scrub starts 2024-09-11T08:15:04.308 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:03 smithi181 ceph-mon[32701]: 5.16 deep-scrub ok 2024-09-11T08:15:04.308 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:03 smithi181 ceph-mon[32701]: pgmap v323: 148 pgs: 148 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.4 KiB/s rd, 75 MiB/s wr, 1.21k op/s 2024-09-11T08:15:04.308 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:03 smithi181 ceph-mon[32701]: 4.1b scrub starts 2024-09-11T08:15:04.309 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:03 smithi181 ceph-mon[32701]: 4.1b scrub ok 2024-09-11T08:15:04.367 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:04.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:04.403 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:04.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:04.414 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 53 16 83504 83488 98.4344 81.625 0.0116199 0.0101544 2024-09-11T08:15:04.504 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:04.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:04.536 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:04.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:04.637 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:04.677 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:04.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:04.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:04.778 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:04.803 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-11T08:15:04.803 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-11T08:15:04.803 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-09-11T08:15:04.803 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-09-11T08:15:04.803 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-11T08:15:04.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:04.822 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:04.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:04.923 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:04.955 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:04.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:04.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:05.056 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:15:05.104 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:05.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:05.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:15:05.148 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:04 smithi150 ceph-mon[29276]: 6.a deep-scrub starts 2024-09-11T08:15:05.148 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:04 smithi150 ceph-mon[29276]: 6.a deep-scrub ok 2024-09-11T08:15:05.148 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:04 smithi150 ceph-mon[29276]: 6.1c scrub starts 2024-09-11T08:15:05.148 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:04 smithi150 ceph-mon[29276]: 6.1c scrub ok 2024-09-11T08:15:05.148 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:04 smithi150 ceph-mon[29276]: 2.0 scrub starts 2024-09-11T08:15:05.148 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:04 smithi150 ceph-mon[39612]: 6.a deep-scrub starts 2024-09-11T08:15:05.148 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:04 smithi150 ceph-mon[39612]: 6.a deep-scrub ok 2024-09-11T08:15:05.149 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:04 smithi150 ceph-mon[39612]: 6.1c scrub starts 2024-09-11T08:15:05.149 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:04 smithi150 ceph-mon[39612]: 6.1c scrub ok 2024-09-11T08:15:05.149 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:04 smithi150 ceph-mon[39612]: 2.0 scrub starts 2024-09-11T08:15:05.206 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:05.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:05.243 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:05.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:05.323 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:15:05 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:15:05] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:15:05.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:04 smithi181 ceph-mon[32701]: 6.a deep-scrub starts 2024-09-11T08:15:05.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:04 smithi181 ceph-mon[32701]: 6.a deep-scrub ok 2024-09-11T08:15:05.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:04 smithi181 ceph-mon[32701]: 6.1c scrub starts 2024-09-11T08:15:05.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:04 smithi181 ceph-mon[32701]: 6.1c scrub ok 2024-09-11T08:15:05.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:04 smithi181 ceph-mon[32701]: 2.0 scrub starts 2024-09-11T08:15:05.344 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:15:05.386 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:05.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:05.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:15:05.415 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 54 16 84784 84768 98.0927 80 0.0192503 0.0101892 2024-09-11T08:15:05.488 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:15:05.509 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-11T08:15:05.529 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:05.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:05.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:15:05.631 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:05.672 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:05.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:05.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:05.744 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:15:05.774 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:15:05.804 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:05.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:05.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:15:05.878 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:15:05.906 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:15:05.944 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:05.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:05.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:15:06.046 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:15:06.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:06.089 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:06.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:15:06.190 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:15:06.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:06.232 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:06.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:15:06.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:05 smithi150 ceph-mon[29276]: 3.15 scrub starts 2024-09-11T08:15:06.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:05 smithi150 ceph-mon[29276]: 3.15 scrub ok 2024-09-11T08:15:06.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:05 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:15:06.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:05 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:15:06.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:05 smithi150 ceph-mon[29276]: 2.0 scrub ok 2024-09-11T08:15:06.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:05 smithi150 ceph-mon[29276]: pgmap v324: 148 pgs: 148 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-09-11T08:15:06.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:05 smithi150 ceph-mon[29276]: osdmap e186: 8 total, 8 up, 8 in 2024-09-11T08:15:06.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:05 smithi150 ceph-mon[29276]: 8.7 scrub starts 2024-09-11T08:15:06.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:05 smithi150 ceph-mon[29276]: osdmap e187: 8 total, 8 up, 8 in 2024-09-11T08:15:06.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:05 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1062118196' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:15:06.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:05 smithi150 ceph-mon[39612]: 3.15 scrub starts 2024-09-11T08:15:06.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:05 smithi150 ceph-mon[39612]: 3.15 scrub ok 2024-09-11T08:15:06.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:05 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:15:06.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:05 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:15:06.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:05 smithi150 ceph-mon[39612]: 2.0 scrub ok 2024-09-11T08:15:06.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:05 smithi150 ceph-mon[39612]: pgmap v324: 148 pgs: 148 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-09-11T08:15:06.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:05 smithi150 ceph-mon[39612]: osdmap e186: 8 total, 8 up, 8 in 2024-09-11T08:15:06.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:05 smithi150 ceph-mon[39612]: 8.7 scrub starts 2024-09-11T08:15:06.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:05 smithi150 ceph-mon[39612]: osdmap e187: 8 total, 8 up, 8 in 2024-09-11T08:15:06.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:05 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1062118196' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:15:06.321 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:05 smithi181 ceph-mon[32701]: 3.15 scrub starts 2024-09-11T08:15:06.321 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:05 smithi181 ceph-mon[32701]: 3.15 scrub ok 2024-09-11T08:15:06.321 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:05 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:15:06.321 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:05 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:15:06.321 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:05 smithi181 ceph-mon[32701]: 2.0 scrub ok 2024-09-11T08:15:06.321 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:05 smithi181 ceph-mon[32701]: pgmap v324: 148 pgs: 148 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-09-11T08:15:06.321 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:05 smithi181 ceph-mon[32701]: osdmap e186: 8 total, 8 up, 8 in 2024-09-11T08:15:06.321 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:05 smithi181 ceph-mon[32701]: 8.7 scrub starts 2024-09-11T08:15:06.321 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:05 smithi181 ceph-mon[32701]: osdmap e187: 8 total, 8 up, 8 in 2024-09-11T08:15:06.321 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:05 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1062118196' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:15:06.333 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:15:06.370 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:06.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:06.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:15:06.415 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 55 16 85945 85929 97.6285 72.5625 0.00582245 0.0102381 2024-09-11T08:15:06.473 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:15:06.495 INFO:teuthology.orchestra.run.smithi150.stderr:noscrub is set 2024-09-11T08:15:06.495 INFO:teuthology.orchestra.run.smithi150.stderr:reweighted osd.3 to 1 (10000) 2024-09-11T08:15:06.504 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:06.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:06.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:15:06.606 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:06.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:06.638 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:06.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:06.739 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:06.782 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:06.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:06.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:06.884 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:15:06.919 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:06.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:06.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:15:07.021 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:15:07.058 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:07.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:07.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:15:07.101 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:06 smithi150 ceph-mon[29276]: 6.17 deep-scrub starts 2024-09-11T08:15:07.101 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:06 smithi150 ceph-mon[29276]: 6.17 deep-scrub ok 2024-09-11T08:15:07.101 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:06 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/4047043582' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-11T08:15:07.101 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:06 smithi150 ceph-mon[29276]: 6.0 scrub starts 2024-09-11T08:15:07.101 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:06 smithi150 ceph-mon[29276]: 6.0 scrub ok 2024-09-11T08:15:07.101 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:06 smithi150 ceph-mon[29276]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:15:07.102 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:06 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1062118196' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:15:07.102 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:06 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/4047043582' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-11T08:15:07.102 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:06 smithi150 ceph-mon[29276]: osdmap e188: 8 total, 8 up, 8 in 2024-09-11T08:15:07.102 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:06 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:15:07.102 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:06 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:15:07.102 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:06 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 0]}]: dispatch 2024-09-11T08:15:07.102 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:06 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [1, 2]}]: dispatch 2024-09-11T08:15:07.102 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:06 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.3", "id": [0, 3]}]: dispatch 2024-09-11T08:15:07.102 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:06 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1e", "id": [0, 6]}]: dispatch 2024-09-11T08:15:07.103 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:06 smithi150 ceph-mon[39612]: 6.17 deep-scrub starts 2024-09-11T08:15:07.103 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:06 smithi150 ceph-mon[39612]: 6.17 deep-scrub ok 2024-09-11T08:15:07.103 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:06 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/4047043582' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-11T08:15:07.103 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:06 smithi150 ceph-mon[39612]: 6.0 scrub starts 2024-09-11T08:15:07.103 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:06 smithi150 ceph-mon[39612]: 6.0 scrub ok 2024-09-11T08:15:07.103 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:06 smithi150 ceph-mon[39612]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:15:07.103 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:06 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1062118196' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:15:07.103 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:06 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/4047043582' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-11T08:15:07.103 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:06 smithi150 ceph-mon[39612]: osdmap e188: 8 total, 8 up, 8 in 2024-09-11T08:15:07.103 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:06 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:15:07.104 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:06 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:15:07.104 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:06 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 0]}]: dispatch 2024-09-11T08:15:07.104 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:06 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [1, 2]}]: dispatch 2024-09-11T08:15:07.104 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:06 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.3", "id": [0, 3]}]: dispatch 2024-09-11T08:15:07.104 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:06 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1e", "id": [0, 6]}]: dispatch 2024-09-11T08:15:07.135 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:06 smithi181 ceph-mon[32701]: 6.17 deep-scrub starts 2024-09-11T08:15:07.135 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:06 smithi181 ceph-mon[32701]: 6.17 deep-scrub ok 2024-09-11T08:15:07.136 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:06 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/4047043582' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-11T08:15:07.136 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:06 smithi181 ceph-mon[32701]: 6.0 scrub starts 2024-09-11T08:15:07.136 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:06 smithi181 ceph-mon[32701]: 6.0 scrub ok 2024-09-11T08:15:07.136 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:06 smithi181 ceph-mon[32701]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:15:07.136 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:06 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1062118196' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:15:07.136 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:06 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/4047043582' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-11T08:15:07.136 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:06 smithi181 ceph-mon[32701]: osdmap e188: 8 total, 8 up, 8 in 2024-09-11T08:15:07.136 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:06 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:15:07.136 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:06 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:15:07.136 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:06 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 0]}]: dispatch 2024-09-11T08:15:07.136 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:06 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [1, 2]}]: dispatch 2024-09-11T08:15:07.136 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:06 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.3", "id": [0, 3]}]: dispatch 2024-09-11T08:15:07.136 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:06 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1e", "id": [0, 6]}]: dispatch 2024-09-11T08:15:07.161 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:07.197 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:07.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:07.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:07.299 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:07.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:07.338 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:07.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:07.415 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 56 16 87264 87248 97.3571 82.4375 0.00807589 0.0102668 2024-09-11T08:15:07.439 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:15:07.477 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:07.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:07.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:15:07.573 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd reweight 4 1 2024-09-11T08:15:07.579 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:07.611 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:07.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:07.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:07.713 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:15:07.745 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:07.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:07.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:15:07.788 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-11T08:15:07.788 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-09-11T08:15:07.788 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-09-11T08:15:07.788 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-09-11T08:15:07.788 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.788 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14790.0:107788 7.7 7:ee3b03b0:::benchmark_data_smithi150_78508_object107787:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-11T08:15:07.788 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:11:10.650185+0000", 2024-09-11T08:15:07.788 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 237.13449312899999, 2024-09-11T08:15:07.788 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.105404093, 2024-09-11T08:15:07.788 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:15:07.789 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:15:07.789 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:15:07.789 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14790", 2024-09-11T08:15:07.789 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:15:07.789 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 107788 2024-09-11T08:15:07.789 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.789 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:15:07.789 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.789 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:15:07.789 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.650185+0000", 2024-09-11T08:15:07.789 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:07.789 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.789 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.789 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:15:07.790 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.650185+0000", 2024-09-11T08:15:07.790 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:07.790 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.790 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.790 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:15:07.790 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.650183+0000", 2024-09-11T08:15:07.791 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:15:07.791 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.791 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.791 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:15:07.791 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.650234+0000", 2024-09-11T08:15:07.791 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1153000000000003e-05 2024-09-11T08:15:07.791 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.791 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.791 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:15:07.791 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.650249+0000", 2024-09-11T08:15:07.791 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4768e-05 2024-09-11T08:15:07.791 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.791 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.792 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:15:07.792 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.650255+0000", 2024-09-11T08:15:07.792 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9120000000000003e-06 2024-09-11T08:15:07.792 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.792 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.792 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:15:07.792 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.650300+0000", 2024-09-11T08:15:07.792 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5346000000000003e-05 2024-09-11T08:15:07.792 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.792 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.792 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:15:07.792 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.650426+0000", 2024-09-11T08:15:07.793 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012579299999999999 2024-09-11T08:15:07.793 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.793 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.793 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,3", 2024-09-11T08:15:07.793 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.650584+0000", 2024-09-11T08:15:07.793 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015783099999999999 2024-09-11T08:15:07.793 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.793 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.793 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:15:07.793 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.658381+0000", 2024-09-11T08:15:07.793 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0077973310000000002 2024-09-11T08:15:07.793 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.793 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.794 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:07.794 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.660410+0000", 2024-09-11T08:15:07.794 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0020288070000000001 2024-09-11T08:15:07.794 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.794 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.794 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:07.794 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.755507+0000", 2024-09-11T08:15:07.794 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.095097386000000006 2024-09-11T08:15:07.794 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.794 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.794 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:15:07.794 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.755549+0000", 2024-09-11T08:15:07.794 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1591999999999998e-05 2024-09-11T08:15:07.795 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.795 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.795 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:15:07.795 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.755589+0000", 2024-09-11T08:15:07.795 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9647999999999997e-05 2024-09-11T08:15:07.795 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:07.795 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:15:07.795 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:07.795 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.795 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.795 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14790.0:107789 7.7 7:e2eaa105:::benchmark_data_smithi150_78508_object107788:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-11T08:15:07.795 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:11:10.650273+0000", 2024-09-11T08:15:07.795 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 237.13440478999999, 2024-09-11T08:15:07.795 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.105446646, 2024-09-11T08:15:07.796 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:15:07.796 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:15:07.796 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:15:07.796 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14790", 2024-09-11T08:15:07.796 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:15:07.796 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 107789 2024-09-11T08:15:07.796 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.796 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:15:07.796 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.796 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:15:07.796 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.650273+0000", 2024-09-11T08:15:07.796 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:07.796 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.796 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.797 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:15:07.797 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.650273+0000", 2024-09-11T08:15:07.797 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:07.797 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.797 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.797 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:15:07.797 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.650272+0000", 2024-09-11T08:15:07.797 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-09-11T08:15:07.797 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.797 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.797 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:15:07.797 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.650343+0000", 2024-09-11T08:15:07.797 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0863000000000002e-05 2024-09-11T08:15:07.797 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.798 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.798 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:15:07.798 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.650355+0000", 2024-09-11T08:15:07.798 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2184999999999999e-05 2024-09-11T08:15:07.798 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.798 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.798 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:15:07.798 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.650361+0000", 2024-09-11T08:15:07.798 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4779999999999999e-06 2024-09-11T08:15:07.798 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.798 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.798 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:15:07.798 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.653411+0000", 2024-09-11T08:15:07.798 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003050044 2024-09-11T08:15:07.799 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.799 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.799 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:15:07.799 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.653568+0000", 2024-09-11T08:15:07.799 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000156914 2024-09-11T08:15:07.799 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.799 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.799 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,3", 2024-09-11T08:15:07.799 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.653705+0000", 2024-09-11T08:15:07.799 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013759899999999999 2024-09-11T08:15:07.799 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.799 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.800 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:15:07.800 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.658386+0000", 2024-09-11T08:15:07.800 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.004681073 2024-09-11T08:15:07.800 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.800 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.800 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:07.800 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.664223+0000", 2024-09-11T08:15:07.800 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0058361239999999998 2024-09-11T08:15:07.800 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.800 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.801 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:07.801 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.755671+0000", 2024-09-11T08:15:07.801 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.091448810000000005 2024-09-11T08:15:07.801 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.801 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.801 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:15:07.801 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.755699+0000", 2024-09-11T08:15:07.801 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7783000000000001e-05 2024-09-11T08:15:07.801 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.801 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.801 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:15:07.801 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.755720+0000", 2024-09-11T08:15:07.801 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0352000000000001e-05 2024-09-11T08:15:07.801 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:07.802 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:15:07.802 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:07.802 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.802 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.802 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14790.0:107790 7.7 7:e3ce7c04:::benchmark_data_smithi150_78508_object107789:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-11T08:15:07.802 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:11:10.651996+0000", 2024-09-11T08:15:07.802 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 237.13268214300001, 2024-09-11T08:15:07.802 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.103794942, 2024-09-11T08:15:07.802 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:15:07.802 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:15:07.802 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:15:07.802 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14790", 2024-09-11T08:15:07.803 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:15:07.803 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 107790 2024-09-11T08:15:07.803 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.803 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:15:07.803 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.803 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:15:07.803 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.651996+0000", 2024-09-11T08:15:07.804 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:07.804 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.804 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.804 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:15:07.804 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.651996+0000", 2024-09-11T08:15:07.804 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:07.804 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.804 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.804 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:15:07.804 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.651994+0000", 2024-09-11T08:15:07.804 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:15:07.804 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.804 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.805 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:15:07.805 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.652048+0000", 2024-09-11T08:15:07.805 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3866000000000002e-05 2024-09-11T08:15:07.805 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.805 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.805 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:15:07.805 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.652060+0000", 2024-09-11T08:15:07.805 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1552000000000001e-05 2024-09-11T08:15:07.805 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.805 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.805 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:15:07.805 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.652065+0000", 2024-09-11T08:15:07.805 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.648e-06 2024-09-11T08:15:07.805 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.806 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.806 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:15:07.806 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.655937+0000", 2024-09-11T08:15:07.806 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003872197 2024-09-11T08:15:07.806 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.806 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.806 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:15:07.806 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.656047+0000", 2024-09-11T08:15:07.806 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000110011 2024-09-11T08:15:07.806 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.806 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.806 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,3", 2024-09-11T08:15:07.806 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.656138+0000", 2024-09-11T08:15:07.806 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0533999999999998e-05 2024-09-11T08:15:07.807 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.807 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.807 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:15:07.807 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.658523+0000", 2024-09-11T08:15:07.807 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0023845720000000002 2024-09-11T08:15:07.807 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.807 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.807 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:07.807 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.755644+0000", 2024-09-11T08:15:07.807 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.097121115999999993 2024-09-11T08:15:07.807 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.807 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.807 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:07.807 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.755743+0000", 2024-09-11T08:15:07.808 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9698999999999995e-05 2024-09-11T08:15:07.808 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.808 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.808 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:15:07.808 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.755772+0000", 2024-09-11T08:15:07.808 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8643999999999999e-05 2024-09-11T08:15:07.808 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.808 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.808 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:15:07.808 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.755790+0000", 2024-09-11T08:15:07.808 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8491e-05 2024-09-11T08:15:07.808 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:07.808 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:15:07.809 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:07.809 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.809 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.809 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14790.0:107983 7.d 7:bdc4699e:::benchmark_data_smithi150_78508_object107982:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-11T08:15:07.809 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:11:10.912162+0000", 2024-09-11T08:15:07.809 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 236.87251559500001, 2024-09-11T08:15:07.809 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.117542387, 2024-09-11T08:15:07.809 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:15:07.809 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:15:07.809 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:15:07.809 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14790", 2024-09-11T08:15:07.809 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:15:07.809 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 107983 2024-09-11T08:15:07.810 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.810 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:15:07.810 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.810 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:15:07.810 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.912162+0000", 2024-09-11T08:15:07.810 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:07.810 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.810 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.810 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:15:07.810 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.912162+0000", 2024-09-11T08:15:07.810 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:07.810 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.810 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.810 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:15:07.811 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.912161+0000", 2024-09-11T08:15:07.811 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:15:07.811 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.811 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.811 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:15:07.811 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.912223+0000", 2024-09-11T08:15:07.811 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2510999999999998e-05 2024-09-11T08:15:07.811 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.811 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.811 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:15:07.811 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.912234+0000", 2024-09-11T08:15:07.811 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0761000000000001e-05 2024-09-11T08:15:07.811 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.811 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.812 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:15:07.812 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.912240+0000", 2024-09-11T08:15:07.812 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2940000000000002e-06 2024-09-11T08:15:07.812 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.812 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.812 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:15:07.812 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.912279+0000", 2024-09-11T08:15:07.812 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8994e-05 2024-09-11T08:15:07.812 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.812 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.812 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:15:07.812 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.912362+0000", 2024-09-11T08:15:07.813 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2207999999999996e-05 2024-09-11T08:15:07.813 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.813 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.813 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,1", 2024-09-11T08:15:07.813 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.912454+0000", 2024-09-11T08:15:07.813 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.2105e-05 2024-09-11T08:15:07.813 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.813 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.813 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:15:07.813 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.915437+0000", 2024-09-11T08:15:07.813 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0029833189999999999 2024-09-11T08:15:07.813 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.813 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.813 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:07.814 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.922564+0000", 2024-09-11T08:15:07.814 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0071273739999999997 2024-09-11T08:15:07.814 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.814 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.814 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:07.814 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:11.029614+0000", 2024-09-11T08:15:07.814 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.107049851 2024-09-11T08:15:07.814 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.814 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.814 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:15:07.814 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:11.029666+0000", 2024-09-11T08:15:07.814 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1480999999999997e-05 2024-09-11T08:15:07.814 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.814 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.815 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:15:07.815 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:11.029704+0000", 2024-09-11T08:15:07.815 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8756e-05 2024-09-11T08:15:07.815 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:07.815 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:15:07.815 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:07.815 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.815 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.815 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14790.0:107986 7.d 7:b325facf:::benchmark_data_smithi150_78508_object107985:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-11T08:15:07.815 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:11:10.916234+0000", 2024-09-11T08:15:07.815 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 236.86844374399999, 2024-09-11T08:15:07.815 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.113578731, 2024-09-11T08:15:07.815 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:15:07.816 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:15:07.816 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:15:07.816 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14790", 2024-09-11T08:15:07.816 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:15:07.816 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 107986 2024-09-11T08:15:07.816 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.816 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:15:07.816 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.816 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:15:07.816 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.916234+0000", 2024-09-11T08:15:07.816 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:07.816 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.816 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.816 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:15:07.817 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.916234+0000", 2024-09-11T08:15:07.817 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:07.817 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.817 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.817 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:15:07.817 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.916233+0000", 2024-09-11T08:15:07.817 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:15:07.817 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.817 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.817 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:15:07.817 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.916294+0000", 2024-09-11T08:15:07.817 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1434999999999997e-05 2024-09-11T08:15:07.817 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.818 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.818 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:15:07.818 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.916306+0000", 2024-09-11T08:15:07.818 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1652e-05 2024-09-11T08:15:07.818 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.818 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.818 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:15:07.818 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.916311+0000", 2024-09-11T08:15:07.818 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8720000000000001e-06 2024-09-11T08:15:07.818 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.818 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.818 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:15:07.818 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.916340+0000", 2024-09-11T08:15:07.818 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9699000000000001e-05 2024-09-11T08:15:07.819 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.819 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.819 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:15:07.819 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.916437+0000", 2024-09-11T08:15:07.819 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7107000000000004e-05 2024-09-11T08:15:07.819 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.819 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.819 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,1", 2024-09-11T08:15:07.819 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.916545+0000", 2024-09-11T08:15:07.819 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010762799999999999 2024-09-11T08:15:07.819 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.819 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.820 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:15:07.820 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.922581+0000", 2024-09-11T08:15:07.820 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0060361089999999996 2024-09-11T08:15:07.820 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.820 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.820 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:07.820 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.923928+0000", 2024-09-11T08:15:07.820 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001346634 2024-09-11T08:15:07.820 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.821 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.821 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:07.821 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:11.029753+0000", 2024-09-11T08:15:07.821 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10582517700000001 2024-09-11T08:15:07.821 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.821 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.821 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:15:07.821 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:11.029787+0000", 2024-09-11T08:15:07.822 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3998999999999998e-05 2024-09-11T08:15:07.822 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.822 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.822 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:15:07.822 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:11.029813+0000", 2024-09-11T08:15:07.822 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5718999999999999e-05 2024-09-11T08:15:07.822 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:07.822 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:15:07.822 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:07.822 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.822 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.822 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14790.0:107990 7.d 7:b289709b:::benchmark_data_smithi150_78508_object107989:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-11T08:15:07.822 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:11:10.917695+0000", 2024-09-11T08:15:07.822 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 236.86698218199999, 2024-09-11T08:15:07.823 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.121911354, 2024-09-11T08:15:07.823 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:15:07.823 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:15:07.823 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:15:07.823 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14790", 2024-09-11T08:15:07.823 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:15:07.823 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 107990 2024-09-11T08:15:07.823 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.823 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:15:07.823 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.823 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:15:07.823 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.917695+0000", 2024-09-11T08:15:07.823 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:07.823 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.824 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.824 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:15:07.824 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.917695+0000", 2024-09-11T08:15:07.824 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:07.824 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.824 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.824 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:15:07.824 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.917694+0000", 2024-09-11T08:15:07.824 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:15:07.824 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.824 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.824 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:15:07.824 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.917798+0000", 2024-09-11T08:15:07.824 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010350500000000001 2024-09-11T08:15:07.825 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.825 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.825 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:15:07.825 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.917808+0000", 2024-09-11T08:15:07.825 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0631e-05 2024-09-11T08:15:07.825 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.825 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.825 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:15:07.825 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.917814+0000", 2024-09-11T08:15:07.825 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7339999999999996e-06 2024-09-11T08:15:07.826 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.826 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.826 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:15:07.826 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.919793+0000", 2024-09-11T08:15:07.826 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0019785269999999999 2024-09-11T08:15:07.826 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.826 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.826 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:15:07.826 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.919921+0000", 2024-09-11T08:15:07.826 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012792200000000001 2024-09-11T08:15:07.826 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.826 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.826 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,1", 2024-09-11T08:15:07.826 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.920086+0000", 2024-09-11T08:15:07.826 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016528800000000001 2024-09-11T08:15:07.827 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.827 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.827 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:15:07.827 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.922668+0000", 2024-09-11T08:15:07.827 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0025824400000000001 2024-09-11T08:15:07.827 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.827 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.827 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:07.827 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.926179+0000", 2024-09-11T08:15:07.827 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0035110139999999998 2024-09-11T08:15:07.828 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.828 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.828 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:07.828 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:11.039525+0000", 2024-09-11T08:15:07.828 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.11334577 2024-09-11T08:15:07.828 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.828 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.828 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:15:07.828 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:11.039572+0000", 2024-09-11T08:15:07.828 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6471000000000003e-05 2024-09-11T08:15:07.828 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.828 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.828 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:15:07.829 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:11.039607+0000", 2024-09-11T08:15:07.829 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5212000000000001e-05 2024-09-11T08:15:07.829 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:07.829 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:15:07.829 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:07.829 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.829 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.829 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14790.0:108875 7.7 7:e4a96b17:::benchmark_data_smithi150_78508_object108874:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-11T08:15:07.829 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:11:11.920922+0000", 2024-09-11T08:15:07.829 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 235.86375524900001, 2024-09-11T08:15:07.829 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10167828499999999, 2024-09-11T08:15:07.830 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:15:07.830 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:15:07.830 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:15:07.830 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14790", 2024-09-11T08:15:07.830 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:15:07.830 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 108875 2024-09-11T08:15:07.830 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.830 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:15:07.830 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.830 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:15:07.830 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:11.920922+0000", 2024-09-11T08:15:07.830 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:07.831 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.831 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.831 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:15:07.831 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:11.920922+0000", 2024-09-11T08:15:07.831 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:07.831 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.831 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.831 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:15:07.831 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:11.920922+0000", 2024-09-11T08:15:07.831 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:15:07.831 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.832 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.832 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:15:07.832 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:11.920990+0000", 2024-09-11T08:15:07.832 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8547000000000006e-05 2024-09-11T08:15:07.832 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.832 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.832 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:15:07.832 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:11.920999+0000", 2024-09-11T08:15:07.832 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.9299999999999992e-06 2024-09-11T08:15:07.832 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.832 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.832 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:15:07.832 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:11.921003+0000", 2024-09-11T08:15:07.833 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6720000000000002e-06 2024-09-11T08:15:07.834 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.834 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.834 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:15:07.834 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:11.921748+0000", 2024-09-11T08:15:07.834 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00074566300000000003 2024-09-11T08:15:07.834 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.834 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.834 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:15:07.835 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:11.921841+0000", 2024-09-11T08:15:07.835 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.3129999999999998e-05 2024-09-11T08:15:07.835 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.835 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.835 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,3", 2024-09-11T08:15:07.835 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:11.921985+0000", 2024-09-11T08:15:07.835 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014328400000000001 2024-09-11T08:15:07.835 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.835 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.835 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:15:07.835 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:11.924214+0000", 2024-09-11T08:15:07.836 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0022292330000000002 2024-09-11T08:15:07.836 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.836 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.836 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:07.836 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:11.934164+0000", 2024-09-11T08:15:07.836 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0099501569999999994 2024-09-11T08:15:07.836 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.836 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.836 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:07.837 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:12.022543+0000", 2024-09-11T08:15:07.837 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.088378660999999997 2024-09-11T08:15:07.837 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.837 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.837 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:15:07.837 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:12.022576+0000", 2024-09-11T08:15:07.837 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3013999999999999e-05 2024-09-11T08:15:07.837 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.837 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.838 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:15:07.838 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:12.022601+0000", 2024-09-11T08:15:07.838 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4875000000000002e-05 2024-09-11T08:15:07.838 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:07.838 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:15:07.838 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:07.838 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.838 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.838 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14790.0:142816 7.c 7:35f8d9b8:::benchmark_data_smithi150_78508_object18375:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-11T08:15:07.838 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:11:45.449962+0000", 2024-09-11T08:15:07.838 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 202.334715797, 2024-09-11T08:15:07.838 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.111155043, 2024-09-11T08:15:07.838 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:15:07.838 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:15:07.838 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:15:07.839 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14790", 2024-09-11T08:15:07.839 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:15:07.839 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 142816 2024-09-11T08:15:07.839 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.839 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:15:07.839 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.839 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:15:07.839 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.449962+0000", 2024-09-11T08:15:07.839 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:07.839 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.839 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.839 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:15:07.839 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.449962+0000", 2024-09-11T08:15:07.839 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:07.840 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.840 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.840 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:15:07.840 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.449961+0000", 2024-09-11T08:15:07.840 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-09-11T08:15:07.840 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.840 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.840 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:15:07.840 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.449964+0000", 2024-09-11T08:15:07.840 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8289999999999999e-06 2024-09-11T08:15:07.840 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.840 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.840 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:15:07.840 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.449970+0000", 2024-09-11T08:15:07.841 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7590000000000003e-06 2024-09-11T08:15:07.841 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.841 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.841 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:15:07.841 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.449974+0000", 2024-09-11T08:15:07.841 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7670000000000001e-06 2024-09-11T08:15:07.841 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.841 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.841 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:15:07.841 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.450013+0000", 2024-09-11T08:15:07.841 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8921000000000002e-05 2024-09-11T08:15:07.841 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.841 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.841 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:15:07.842 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.450111+0000", 2024-09-11T08:15:07.842 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.815e-05 2024-09-11T08:15:07.842 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.842 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.842 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3,7", 2024-09-11T08:15:07.842 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.450207+0000", 2024-09-11T08:15:07.842 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.6020000000000006e-05 2024-09-11T08:15:07.842 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.842 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.842 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:07.842 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.453680+0000", 2024-09-11T08:15:07.842 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0034732059999999999 2024-09-11T08:15:07.842 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.843 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.843 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:15:07.843 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.453686+0000", 2024-09-11T08:15:07.843 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3799999999999999e-06 2024-09-11T08:15:07.843 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.843 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.843 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:07.843 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.561017+0000", 2024-09-11T08:15:07.843 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10733019000000001 2024-09-11T08:15:07.843 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.843 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.843 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:15:07.843 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.561070+0000", 2024-09-11T08:15:07.843 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2998999999999999e-05 2024-09-11T08:15:07.844 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.844 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.844 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:15:07.844 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.561117+0000", 2024-09-11T08:15:07.844 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7363e-05 2024-09-11T08:15:07.844 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:07.844 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:15:07.844 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:07.844 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.844 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.844 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14790.0:142817 7.7 7:e20eec82:::benchmark_data_smithi150_78508_object18376:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-11T08:15:07.844 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:11:45.450032+0000", 2024-09-11T08:15:07.844 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 202.33464614299999, 2024-09-11T08:15:07.844 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.111096497, 2024-09-11T08:15:07.845 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:15:07.845 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:15:07.845 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:15:07.845 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14790", 2024-09-11T08:15:07.845 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:15:07.845 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 142817 2024-09-11T08:15:07.845 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.845 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:15:07.845 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.845 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:15:07.845 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.450032+0000", 2024-09-11T08:15:07.845 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:07.845 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.845 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.846 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:15:07.846 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.450032+0000", 2024-09-11T08:15:07.846 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:07.846 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.846 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.846 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:15:07.846 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.450031+0000", 2024-09-11T08:15:07.846 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-09-11T08:15:07.846 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.846 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.846 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:15:07.846 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.450034+0000", 2024-09-11T08:15:07.846 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5689999999999999e-06 2024-09-11T08:15:07.847 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.847 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.847 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:15:07.847 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.450039+0000", 2024-09-11T08:15:07.847 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3759999999999999e-06 2024-09-11T08:15:07.847 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.847 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.847 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:15:07.847 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.450043+0000", 2024-09-11T08:15:07.847 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5140000000000002e-06 2024-09-11T08:15:07.847 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.847 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.847 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:15:07.847 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.450091+0000", 2024-09-11T08:15:07.848 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8729000000000001e-05 2024-09-11T08:15:07.848 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.848 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.848 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:15:07.848 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.450185+0000", 2024-09-11T08:15:07.848 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4172999999999994e-05 2024-09-11T08:15:07.848 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.848 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.848 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,3", 2024-09-11T08:15:07.848 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.450279+0000", 2024-09-11T08:15:07.848 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4011999999999994e-05 2024-09-11T08:15:07.848 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.848 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.848 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:15:07.848 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.453956+0000", 2024-09-11T08:15:07.849 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003676697 2024-09-11T08:15:07.849 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.849 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.849 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:07.849 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.459552+0000", 2024-09-11T08:15:07.849 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.005595961 2024-09-11T08:15:07.849 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.849 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.849 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:07.849 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.561022+0000", 2024-09-11T08:15:07.849 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101470352 2024-09-11T08:15:07.849 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.849 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.850 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:15:07.850 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.561064+0000", 2024-09-11T08:15:07.850 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1288999999999997e-05 2024-09-11T08:15:07.850 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.850 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.850 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:15:07.850 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.561128+0000", 2024-09-11T08:15:07.850 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4299000000000002e-05 2024-09-11T08:15:07.850 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:07.850 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:15:07.850 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:07.850 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.850 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.850 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14790.0:142824 7.c 7:35512376:::benchmark_data_smithi150_78508_object18383:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-11T08:15:07.851 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:11:45.457253+0000", 2024-09-11T08:15:07.851 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 202.327424933, 2024-09-11T08:15:07.851 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10395362599999999, 2024-09-11T08:15:07.851 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:15:07.851 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:15:07.851 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:15:07.851 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14790", 2024-09-11T08:15:07.851 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:15:07.851 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 142824 2024-09-11T08:15:07.851 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.851 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:15:07.851 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.851 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:15:07.851 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.457253+0000", 2024-09-11T08:15:07.851 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:07.852 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.852 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.852 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:15:07.852 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.457253+0000", 2024-09-11T08:15:07.852 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:07.852 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.852 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.852 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:15:07.852 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.457251+0000", 2024-09-11T08:15:07.852 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:15:07.852 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.852 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.853 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:15:07.853 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.457257+0000", 2024-09-11T08:15:07.853 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7370000000000003e-06 2024-09-11T08:15:07.853 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.853 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.853 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:15:07.853 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.457270+0000", 2024-09-11T08:15:07.853 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2897999999999999e-05 2024-09-11T08:15:07.853 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.853 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.853 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:15:07.853 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.457276+0000", 2024-09-11T08:15:07.853 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.604e-06 2024-09-11T08:15:07.853 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.854 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.854 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:15:07.854 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.457332+0000", 2024-09-11T08:15:07.854 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6005999999999999e-05 2024-09-11T08:15:07.854 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.854 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.854 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:15:07.854 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.457467+0000", 2024-09-11T08:15:07.854 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000135053 2024-09-11T08:15:07.854 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.854 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.854 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3,7", 2024-09-11T08:15:07.854 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.457582+0000", 2024-09-11T08:15:07.854 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000115771 2024-09-11T08:15:07.855 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.855 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.855 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:15:07.855 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.464428+0000", 2024-09-11T08:15:07.855 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.006845566 2024-09-11T08:15:07.855 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.855 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.855 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:07.855 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.464895+0000", 2024-09-11T08:15:07.855 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00046669099999999998 2024-09-11T08:15:07.855 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.855 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.855 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:07.855 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.561159+0000", 2024-09-11T08:15:07.856 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.096264058 2024-09-11T08:15:07.856 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.856 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.856 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:15:07.856 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.561187+0000", 2024-09-11T08:15:07.856 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7906e-05 2024-09-11T08:15:07.856 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.856 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.856 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:15:07.856 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.561206+0000", 2024-09-11T08:15:07.856 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9735e-05 2024-09-11T08:15:07.856 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:07.856 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:15:07.856 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:07.857 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.857 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.857 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14790.0:142828 7.7 7:e1b2372f:::benchmark_data_smithi150_78508_object18387:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-11T08:15:07.857 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:11:45.461016+0000", 2024-09-11T08:15:07.857 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 202.32366153000001, 2024-09-11T08:15:07.857 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.110292217, 2024-09-11T08:15:07.857 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:15:07.857 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:15:07.857 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:15:07.857 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14790", 2024-09-11T08:15:07.857 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:15:07.857 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 142828 2024-09-11T08:15:07.857 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.857 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:15:07.857 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.858 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:15:07.858 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.461016+0000", 2024-09-11T08:15:07.858 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:07.858 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.858 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.858 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:15:07.858 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.461016+0000", 2024-09-11T08:15:07.858 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:07.858 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.859 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.859 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:15:07.859 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.461015+0000", 2024-09-11T08:15:07.859 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:15:07.859 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.859 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.859 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:15:07.859 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.461020+0000", 2024-09-11T08:15:07.859 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4729999999999999e-06 2024-09-11T08:15:07.859 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.859 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.859 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:15:07.859 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.461031+0000", 2024-09-11T08:15:07.859 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0511e-05 2024-09-11T08:15:07.860 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.860 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.860 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:15:07.860 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.461036+0000", 2024-09-11T08:15:07.860 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7100000000000004e-06 2024-09-11T08:15:07.860 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.860 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.860 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:15:07.860 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.461096+0000", 2024-09-11T08:15:07.861 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9944000000000001e-05 2024-09-11T08:15:07.861 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.861 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.861 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:15:07.861 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.461195+0000", 2024-09-11T08:15:07.861 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.8783000000000006e-05 2024-09-11T08:15:07.861 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.861 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.861 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,3", 2024-09-11T08:15:07.861 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.461259+0000", 2024-09-11T08:15:07.861 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4015000000000003e-05 2024-09-11T08:15:07.861 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.861 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.862 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:07.862 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.464217+0000", 2024-09-11T08:15:07.862 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0029576469999999999 2024-09-11T08:15:07.862 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.862 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.862 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:15:07.862 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.464232+0000", 2024-09-11T08:15:07.862 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5150000000000001e-05 2024-09-11T08:15:07.862 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.862 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.862 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:07.863 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.571222+0000", 2024-09-11T08:15:07.863 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.106989614 2024-09-11T08:15:07.863 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.863 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.863 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:15:07.863 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.571267+0000", 2024-09-11T08:15:07.863 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5580000000000001e-05 2024-09-11T08:15:07.863 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.863 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.863 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:15:07.863 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.571308+0000", 2024-09-11T08:15:07.864 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1216e-05 2024-09-11T08:15:07.864 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:07.864 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:15:07.864 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:07.864 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.864 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.864 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14790.0:142829 7.c 7:35774efa:::benchmark_data_smithi150_78508_object18388:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-11T08:15:07.864 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:11:45.461366+0000", 2024-09-11T08:15:07.864 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 202.323311676, 2024-09-11T08:15:07.865 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.110791558, 2024-09-11T08:15:07.865 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:15:07.865 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:15:07.865 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:15:07.865 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14790", 2024-09-11T08:15:07.865 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:15:07.865 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 142829 2024-09-11T08:15:07.865 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.865 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:15:07.865 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.865 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:15:07.865 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.461366+0000", 2024-09-11T08:15:07.866 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:07.866 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.866 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.866 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:15:07.866 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.461366+0000", 2024-09-11T08:15:07.866 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:07.866 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.866 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.866 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:15:07.867 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.461365+0000", 2024-09-11T08:15:07.867 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:15:07.867 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.867 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.867 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:15:07.867 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.461371+0000", 2024-09-11T08:15:07.867 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6550000000000001e-06 2024-09-11T08:15:07.867 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.867 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.867 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:15:07.867 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.461381+0000", 2024-09-11T08:15:07.868 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.8840000000000006e-06 2024-09-11T08:15:07.868 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.868 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.868 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:15:07.868 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.461386+0000", 2024-09-11T08:15:07.868 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7209999999999999e-06 2024-09-11T08:15:07.868 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.868 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.868 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:15:07.868 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.461420+0000", 2024-09-11T08:15:07.868 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4113999999999999e-05 2024-09-11T08:15:07.868 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.868 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.869 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:15:07.869 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.461524+0000", 2024-09-11T08:15:07.869 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010385700000000001 2024-09-11T08:15:07.869 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.869 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.869 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3,7", 2024-09-11T08:15:07.869 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.461624+0000", 2024-09-11T08:15:07.869 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9512999999999994e-05 2024-09-11T08:15:07.869 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.869 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.869 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:15:07.869 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.464538+0000", 2024-09-11T08:15:07.869 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0029147000000000001 2024-09-11T08:15:07.870 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.870 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.870 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:07.870 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.465053+0000", 2024-09-11T08:15:07.870 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00051460999999999998 2024-09-11T08:15:07.870 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.870 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.870 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:07.870 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.572079+0000", 2024-09-11T08:15:07.871 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.107026432 2024-09-11T08:15:07.871 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.871 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.871 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:15:07.871 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.572119+0000", 2024-09-11T08:15:07.871 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9400000000000002e-05 2024-09-11T08:15:07.871 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.871 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.871 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:15:07.872 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.572158+0000", 2024-09-11T08:15:07.872 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8698000000000001e-05 2024-09-11T08:15:07.872 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:07.872 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:15:07.872 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:07.872 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.872 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.872 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14790.0:142832 7.7 7:e9a2ce0c:::benchmark_data_smithi150_78508_object18391:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-11T08:15:07.872 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:11:45.464904+0000", 2024-09-11T08:15:07.873 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 202.31977416699999, 2024-09-11T08:15:07.873 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.106509003, 2024-09-11T08:15:07.873 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:15:07.873 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:15:07.873 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:15:07.873 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14790", 2024-09-11T08:15:07.873 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:15:07.873 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 142832 2024-09-11T08:15:07.873 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.873 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:15:07.873 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.873 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:15:07.874 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.464904+0000", 2024-09-11T08:15:07.874 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:07.874 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.874 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.874 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:15:07.874 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.464904+0000", 2024-09-11T08:15:07.874 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:07.874 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.874 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.874 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:15:07.874 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.464902+0000", 2024-09-11T08:15:07.874 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:15:07.874 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.874 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.874 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:15:07.875 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.464907+0000", 2024-09-11T08:15:07.875 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7820000000000001e-06 2024-09-11T08:15:07.875 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.875 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.875 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:15:07.875 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.464915+0000", 2024-09-11T08:15:07.875 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9589999999999995e-06 2024-09-11T08:15:07.875 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.875 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.875 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:15:07.875 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.464920+0000", 2024-09-11T08:15:07.875 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1250000000000001e-06 2024-09-11T08:15:07.875 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.875 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.875 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:15:07.875 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.464990+0000", 2024-09-11T08:15:07.876 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0203000000000005e-05 2024-09-11T08:15:07.876 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.876 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.876 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:15:07.876 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.465085+0000", 2024-09-11T08:15:07.876 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4584000000000001e-05 2024-09-11T08:15:07.876 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.876 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.876 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,3", 2024-09-11T08:15:07.876 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.465164+0000", 2024-09-11T08:15:07.876 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8568999999999999e-05 2024-09-11T08:15:07.876 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.876 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.876 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:15:07.876 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.468390+0000", 2024-09-11T08:15:07.877 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0032267509999999999 2024-09-11T08:15:07.877 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.877 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.877 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:07.877 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.468918+0000", 2024-09-11T08:15:07.877 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00052711900000000005 2024-09-11T08:15:07.877 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.877 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.877 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:07.877 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.571350+0000", 2024-09-11T08:15:07.877 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10243265999999999 2024-09-11T08:15:07.877 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.877 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.877 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:15:07.878 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.571381+0000", 2024-09-11T08:15:07.878 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0326000000000001e-05 2024-09-11T08:15:07.878 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.878 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.878 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:15:07.878 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.571413+0000", 2024-09-11T08:15:07.878 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2002000000000003e-05 2024-09-11T08:15:07.878 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:07.878 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:15:07.878 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:07.878 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.878 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.878 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14790.0:142833 7.c 7:302c1671:::benchmark_data_smithi150_78508_object18392:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-11T08:15:07.878 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:11:45.465028+0000", 2024-09-11T08:15:07.878 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 202.31964950899999, 2024-09-11T08:15:07.879 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10812332500000001, 2024-09-11T08:15:07.879 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:15:07.879 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:15:07.879 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:15:07.879 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14790", 2024-09-11T08:15:07.879 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:15:07.879 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 142833 2024-09-11T08:15:07.879 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.879 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:15:07.879 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.879 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:15:07.879 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.465028+0000", 2024-09-11T08:15:07.879 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:07.879 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.879 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.880 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:15:07.880 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.465028+0000", 2024-09-11T08:15:07.880 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:07.880 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.880 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.880 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:15:07.880 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.465028+0000", 2024-09-11T08:15:07.880 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-09-11T08:15:07.880 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.880 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.880 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:15:07.880 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.465031+0000", 2024-09-11T08:15:07.880 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.185e-06 2024-09-11T08:15:07.880 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.881 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.881 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:15:07.881 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.465036+0000", 2024-09-11T08:15:07.881 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5770000000000001e-06 2024-09-11T08:15:07.881 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.881 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.881 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:15:07.881 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.465040+0000", 2024-09-11T08:15:07.881 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8179999999999997e-06 2024-09-11T08:15:07.881 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.881 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.881 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:15:07.881 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.465070+0000", 2024-09-11T08:15:07.882 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0105000000000001e-05 2024-09-11T08:15:07.882 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.882 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.882 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:15:07.882 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.465164+0000", 2024-09-11T08:15:07.882 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.3258999999999996e-05 2024-09-11T08:15:07.882 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.882 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.882 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3,7", 2024-09-11T08:15:07.882 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.465262+0000", 2024-09-11T08:15:07.882 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.8051e-05 2024-09-11T08:15:07.882 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.882 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.882 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:15:07.883 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.467933+0000", 2024-09-11T08:15:07.883 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0026711949999999999 2024-09-11T08:15:07.883 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.883 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.883 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:07.883 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.468879+0000", 2024-09-11T08:15:07.883 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00094654299999999999 2024-09-11T08:15:07.883 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.883 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.883 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:07.883 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.573079+0000", 2024-09-11T08:15:07.883 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10419961 2024-09-11T08:15:07.883 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.883 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.884 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:15:07.884 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.573120+0000", 2024-09-11T08:15:07.884 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0679000000000001e-05 2024-09-11T08:15:07.884 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.884 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.884 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:15:07.884 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.573151+0000", 2024-09-11T08:15:07.884 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1884000000000002e-05 2024-09-11T08:15:07.884 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:07.884 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:15:07.884 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:07.884 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.884 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.884 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14790.0:185902 7.c 7:347cced3:::benchmark_data_smithi150_78508_object61461:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-09-11T08:15:07.885 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:12:27.513442+0000", 2024-09-11T08:15:07.885 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 160.27123596600001, 2024-09-11T08:15:07.885 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0142799899999999, 2024-09-11T08:15:07.885 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:15:07.885 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:15:07.885 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:15:07.885 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14790", 2024-09-11T08:15:07.885 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:15:07.885 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 185902 2024-09-11T08:15:07.885 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.885 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:15:07.885 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.885 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:15:07.885 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:27.513442+0000", 2024-09-11T08:15:07.886 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:07.886 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.886 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.886 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:15:07.886 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:27.513442+0000", 2024-09-11T08:15:07.886 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:07.886 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.886 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.886 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:15:07.886 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:27.513441+0000", 2024-09-11T08:15:07.886 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:15:07.886 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.886 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.886 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:15:07.887 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:27.513450+0000", 2024-09-11T08:15:07.887 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.2720000000000003e-06 2024-09-11T08:15:07.887 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.887 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.887 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:15:07.887 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:27.513457+0000", 2024-09-11T08:15:07.887 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6590000000000001e-06 2024-09-11T08:15:07.887 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.887 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.887 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:15:07.887 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:27.513460+0000", 2024-09-11T08:15:07.887 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0510000000000001e-06 2024-09-11T08:15:07.887 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.887 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.888 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:15:07.888 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:27.513493+0000", 2024-09-11T08:15:07.888 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3589e-05 2024-09-11T08:15:07.888 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.888 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.888 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:15:07.888 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:27.513501+0000", 2024-09-11T08:15:07.888 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8129999999999996e-06 2024-09-11T08:15:07.888 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.888 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.888 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:15:07.888 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:28.519683+0000", 2024-09-11T08:15:07.888 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.006181762 2024-09-11T08:15:07.888 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.889 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.889 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:15:07.889 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:28.519817+0000", 2024-09-11T08:15:07.889 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013361499999999999 2024-09-11T08:15:07.889 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.889 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.889 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3", 2024-09-11T08:15:07.889 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:28.519948+0000", 2024-09-11T08:15:07.889 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013139 2024-09-11T08:15:07.889 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.889 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.889 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:07.889 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:28.527596+0000", 2024-09-11T08:15:07.889 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0076475889999999998 2024-09-11T08:15:07.889 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.890 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.890 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:15:07.890 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:28.527601+0000", 2024-09-11T08:15:07.890 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8420000000000001e-06 2024-09-11T08:15:07.890 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.890 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.890 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:15:07.890 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:28.527632+0000", 2024-09-11T08:15:07.890 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0453000000000001e-05 2024-09-11T08:15:07.890 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.890 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.890 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:15:07.890 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:28.527722+0000", 2024-09-11T08:15:07.890 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.9820000000000004e-05 2024-09-11T08:15:07.891 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:07.891 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:15:07.891 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:07.891 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.891 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.891 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14790.0:185911 7.c 7:36b4aa38:::benchmark_data_smithi150_78508_object61470:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-09-11T08:15:07.891 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:12:27.527305+0000", 2024-09-11T08:15:07.891 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 160.25737301300001, 2024-09-11T08:15:07.891 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.003321535, 2024-09-11T08:15:07.891 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:15:07.891 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:15:07.891 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:15:07.891 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14790", 2024-09-11T08:15:07.891 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:15:07.891 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 185911 2024-09-11T08:15:07.892 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.892 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:15:07.892 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.892 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:15:07.892 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:27.527305+0000", 2024-09-11T08:15:07.892 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:07.892 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.892 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.892 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:15:07.892 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:27.527305+0000", 2024-09-11T08:15:07.892 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:07.892 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.892 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.893 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:15:07.894 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:15:07.894 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:27.527303+0000", 2024-09-11T08:15:07.894 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:15:07.894 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.894 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.894 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:15:07.894 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:27.527309+0000", 2024-09-11T08:15:07.895 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7520000000000002e-06 2024-09-11T08:15:07.895 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.895 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.895 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:15:07.895 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:27.527320+0000", 2024-09-11T08:15:07.895 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0805000000000001e-05 2024-09-11T08:15:07.895 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.895 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.895 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:15:07.895 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:27.527325+0000", 2024-09-11T08:15:07.895 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5399999999999997e-06 2024-09-11T08:15:07.895 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.896 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.896 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:15:07.896 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:27.527367+0000", 2024-09-11T08:15:07.896 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.244e-05 2024-09-11T08:15:07.896 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.896 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.896 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:15:07.896 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:27.527375+0000", 2024-09-11T08:15:07.896 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3650000000000004e-06 2024-09-11T08:15:07.896 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.896 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.897 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:15:07.897 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:28.522600+0000", 2024-09-11T08:15:07.897 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.99522450799999995 2024-09-11T08:15:07.897 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.897 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.897 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:15:07.897 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:28.522682+0000", 2024-09-11T08:15:07.897 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1854999999999995e-05 2024-09-11T08:15:07.897 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.897 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.897 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3", 2024-09-11T08:15:07.897 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:28.522776+0000", 2024-09-11T08:15:07.898 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4300000000000002e-05 2024-09-11T08:15:07.898 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.898 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.898 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:15:07.898 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:28.527735+0000", 2024-09-11T08:15:07.898 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0049592580000000002 2024-09-11T08:15:07.898 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.898 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.898 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:07.898 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:28.530551+0000", 2024-09-11T08:15:07.898 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0028152609999999999 2024-09-11T08:15:07.898 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.899 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.899 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:15:07.899 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:28.530578+0000", 2024-09-11T08:15:07.899 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7807999999999999e-05 2024-09-11T08:15:07.899 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.899 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.899 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:15:07.899 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:28.530626+0000", 2024-09-11T08:15:07.899 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7800999999999999e-05 2024-09-11T08:15:07.899 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:07.900 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:15:07.900 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:07.900 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.900 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.900 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14790.0:185923 7.c 7:3f515920:::benchmark_data_smithi150_78508_object61482:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-09-11T08:15:07.900 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:12:27.544959+0000", 2024-09-11T08:15:07.900 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 160.2397182, 2024-09-11T08:15:07.900 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.98583623799999998, 2024-09-11T08:15:07.900 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:15:07.900 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:15:07.900 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:15:07.900 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14790", 2024-09-11T08:15:07.901 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:15:07.901 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 185923 2024-09-11T08:15:07.901 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.901 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:15:07.901 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.901 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:15:07.901 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:27.544959+0000", 2024-09-11T08:15:07.901 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:07.901 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.901 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.902 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:15:07.902 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:27.544959+0000", 2024-09-11T08:15:07.902 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:07.902 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.902 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.902 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:15:07.902 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:27.544959+0000", 2024-09-11T08:15:07.902 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:15:07.902 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.902 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.902 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:15:07.902 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:27.544962+0000", 2024-09-11T08:15:07.902 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6950000000000002e-06 2024-09-11T08:15:07.903 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.903 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.903 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:15:07.903 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:27.544969+0000", 2024-09-11T08:15:07.903 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1770000000000001e-06 2024-09-11T08:15:07.903 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.903 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.903 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:15:07.903 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:27.544975+0000", 2024-09-11T08:15:07.904 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1089999999999997e-06 2024-09-11T08:15:07.904 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.904 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.904 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:15:07.904 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:27.545008+0000", 2024-09-11T08:15:07.904 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3562000000000002e-05 2024-09-11T08:15:07.904 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.904 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.904 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:15:07.904 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:27.545014+0000", 2024-09-11T08:15:07.905 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9479999999999996e-06 2024-09-11T08:15:07.905 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.905 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.905 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:15:07.905 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:28.525074+0000", 2024-09-11T08:15:07.905 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.98006025799999996 2024-09-11T08:15:07.905 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.905 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.905 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:15:07.905 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:28.525201+0000", 2024-09-11T08:15:07.906 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012668299999999999 2024-09-11T08:15:07.906 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.906 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.906 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3", 2024-09-11T08:15:07.906 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:28.525332+0000", 2024-09-11T08:15:07.906 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013046100000000001 2024-09-11T08:15:07.906 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.906 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.906 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:15:07.906 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:28.527738+0000", 2024-09-11T08:15:07.906 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0024068560000000002 2024-09-11T08:15:07.906 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.906 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.906 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:07.907 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:28.530665+0000", 2024-09-11T08:15:07.907 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0029270490000000001 2024-09-11T08:15:07.907 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.907 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.907 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:15:07.907 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:28.530710+0000", 2024-09-11T08:15:07.907 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4335000000000002e-05 2024-09-11T08:15:07.907 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.907 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.907 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:15:07.907 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:28.530796+0000", 2024-09-11T08:15:07.907 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5958000000000006e-05 2024-09-11T08:15:07.907 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:07.907 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:15:07.908 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:07.908 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.908 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.908 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24320.0:931 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94783556094064 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-09-11T08:15:07.908 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:12:27.939487+0000", 2024-09-11T08:15:07.908 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 159.845190793, 2024-09-11T08:15:07.908 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.55492438600000005, 2024-09-11T08:15:07.908 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:15:07.908 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-11T08:15:07.908 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:15:07.908 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24320", 2024-09-11T08:15:07.908 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/1503848706", 2024-09-11T08:15:07.908 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 931 2024-09-11T08:15:07.909 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.909 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:15:07.909 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.909 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:15:07.909 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:27.939487+0000", 2024-09-11T08:15:07.909 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:07.909 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.909 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.909 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:15:07.909 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:27.939487+0000", 2024-09-11T08:15:07.909 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:07.909 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.909 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.909 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:15:07.910 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:27.939485+0000", 2024-09-11T08:15:07.910 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999976 2024-09-11T08:15:07.910 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.910 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.910 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:15:07.910 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:27.939497+0000", 2024-09-11T08:15:07.910 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2164000000000001e-05 2024-09-11T08:15:07.910 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.910 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.910 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:15:07.910 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:27.939515+0000", 2024-09-11T08:15:07.910 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8110000000000001e-05 2024-09-11T08:15:07.910 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.910 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.911 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:15:07.911 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:27.939524+0000", 2024-09-11T08:15:07.911 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.5659999999999996e-06 2024-09-11T08:15:07.911 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.911 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.911 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:15:07.911 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:27.939580+0000", 2024-09-11T08:15:07.911 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5500000000000001e-05 2024-09-11T08:15:07.911 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.911 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.911 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:15:07.911 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:27.939596+0000", 2024-09-11T08:15:07.911 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5696e-05 2024-09-11T08:15:07.912 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.912 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.912 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:15:07.912 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:28.491230+0000", 2024-09-11T08:15:07.912 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.55163408700000005 2024-09-11T08:15:07.912 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.912 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.912 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:15:07.912 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:28.491470+0000", 2024-09-11T08:15:07.912 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00024013300000000001 2024-09-11T08:15:07.912 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.912 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.912 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:15:07.912 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:28.494411+0000", 2024-09-11T08:15:07.913 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0029414319999999999 2024-09-11T08:15:07.913 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:07.913 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:15:07.913 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:07.913 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.913 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.913 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24320.0:932 5.9 5:93e5b521:::notify.7:head [watch reconnect cookie 94783556123344 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-09-11T08:15:07.913 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:12:27.939536+0000", 2024-09-11T08:15:07.913 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 159.84514177899999, 2024-09-11T08:15:07.913 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.55557662900000004, 2024-09-11T08:15:07.913 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:15:07.913 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-11T08:15:07.913 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:15:07.913 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24320", 2024-09-11T08:15:07.914 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/1503848706", 2024-09-11T08:15:07.914 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 932 2024-09-11T08:15:07.914 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.914 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:15:07.914 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.914 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:15:07.914 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:27.939536+0000", 2024-09-11T08:15:07.914 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:07.914 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.914 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.914 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:15:07.914 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:27.939536+0000", 2024-09-11T08:15:07.914 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:07.914 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.915 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.915 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:15:07.915 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:27.939535+0000", 2024-09-11T08:15:07.915 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-09-11T08:15:07.915 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.915 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.915 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:15:07.915 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:27.939538+0000", 2024-09-11T08:15:07.915 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0029999999999999e-06 2024-09-11T08:15:07.915 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.915 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.915 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:15:07.915 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:27.939543+0000", 2024-09-11T08:15:07.915 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7360000000000001e-06 2024-09-11T08:15:07.916 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.916 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.916 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:15:07.916 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:27.939547+0000", 2024-09-11T08:15:07.916 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.579e-06 2024-09-11T08:15:07.916 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.916 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.916 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:15:07.916 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:27.939604+0000", 2024-09-11T08:15:07.916 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6938999999999998e-05 2024-09-11T08:15:07.916 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.916 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.916 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:15:07.916 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:27.939612+0000", 2024-09-11T08:15:07.917 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.7120000000000006e-06 2024-09-11T08:15:07.917 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.917 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.917 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:15:07.917 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:28.490301+0000", 2024-09-11T08:15:07.917 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.55068880399999998 2024-09-11T08:15:07.917 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.917 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.917 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:15:07.917 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:28.490473+0000", 2024-09-11T08:15:07.917 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000171411 2024-09-11T08:15:07.917 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.917 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.917 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:15:07.917 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:12:28.495113+0000", 2024-09-11T08:15:07.918 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0046399320000000003 2024-09-11T08:15:07.918 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:07.918 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:15:07.918 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:07.918 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.918 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.918 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14790.0:239073 7.c 7:31c29a8c:::benchmark_data_smithi150_78508_object114632:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-11T08:15:07.918 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:13:19.269414+0000", 2024-09-11T08:15:07.918 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 108.51526357900001, 2024-09-11T08:15:07.918 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.145124375, 2024-09-11T08:15:07.918 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:15:07.918 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:15:07.918 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:15:07.918 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14790", 2024-09-11T08:15:07.919 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:15:07.919 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 239073 2024-09-11T08:15:07.919 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.919 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:15:07.919 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.919 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:15:07.919 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.269414+0000", 2024-09-11T08:15:07.919 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:07.919 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.919 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.919 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:15:07.919 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.269414+0000", 2024-09-11T08:15:07.919 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:15:07.919 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.920 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.920 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:15:07.920 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.269413+0000", 2024-09-11T08:15:07.920 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:15:07.920 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.920 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.920 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:15:07.920 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.269419+0000", 2024-09-11T08:15:07.920 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2389999999999999e-06 2024-09-11T08:15:07.920 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.920 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.920 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:15:07.920 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.269427+0000", 2024-09-11T08:15:07.920 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2570000000000007e-06 2024-09-11T08:15:07.921 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.921 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.921 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:15:07.921 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.269431+0000", 2024-09-11T08:15:07.921 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1400000000000002e-06 2024-09-11T08:15:07.921 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.921 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.921 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:15:07.921 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.269466+0000", 2024-09-11T08:15:07.921 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4610000000000002e-05 2024-09-11T08:15:07.922 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.922 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.922 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:15:07.922 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.352140+0000", 2024-09-11T08:15:07.922 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.082674391999999999 2024-09-11T08:15:07.922 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.922 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.922 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3", 2024-09-11T08:15:07.922 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.352277+0000", 2024-09-11T08:15:07.922 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000136537 2024-09-11T08:15:07.922 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.922 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.922 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:07.922 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.414431+0000", 2024-09-11T08:15:07.923 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.062154362999999997 2024-09-11T08:15:07.923 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.923 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.923 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:15:07.923 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.414439+0000", 2024-09-11T08:15:07.923 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9500000000000001e-06 2024-09-11T08:15:07.923 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.923 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.923 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:15:07.923 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.414473+0000", 2024-09-11T08:15:07.923 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3377999999999998e-05 2024-09-11T08:15:07.923 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:15:07.923 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:15:07.923 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:15:07.924 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.414538+0000", 2024-09-11T08:15:07.924 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5848999999999999e-05 2024-09-11T08:15:07.924 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:07.924 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:15:07.924 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:07.924 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:15:07.924 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:15:07.924 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-11T08:15:07.938 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:07.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:07.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:15:08.039 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:15:08.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:08.071 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:08.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:15:08.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:07 smithi181 ceph-mon[32701]: 3.2 deep-scrub starts 2024-09-11T08:15:08.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:07 smithi181 ceph-mon[32701]: 3.2 deep-scrub ok 2024-09-11T08:15:08.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:07 smithi181 ceph-mon[32701]: 5.f scrub starts 2024-09-11T08:15:08.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:07 smithi181 ceph-mon[32701]: 5.f scrub ok 2024-09-11T08:15:08.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:07 smithi181 ceph-mon[32701]: pgmap v328: 148 pgs: 148 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 65 MiB/s wr, 1.05k op/s 2024-09-11T08:15:08.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:07 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 0]}]': finished 2024-09-11T08:15:08.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:07 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [1, 2]}]': finished 2024-09-11T08:15:08.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:07 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.3", "id": [0, 3]}]': finished 2024-09-11T08:15:08.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:07 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1e", "id": [0, 6]}]': finished 2024-09-11T08:15:08.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:07 smithi181 ceph-mon[32701]: osdmap e189: 8 total, 8 up, 8 in 2024-09-11T08:15:08.173 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:08.213 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:08.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:08.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:08.300 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:07 smithi150 ceph-mon[29276]: 3.2 deep-scrub starts 2024-09-11T08:15:08.301 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:07 smithi150 ceph-mon[29276]: 3.2 deep-scrub ok 2024-09-11T08:15:08.301 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:07 smithi150 ceph-mon[29276]: 5.f scrub starts 2024-09-11T08:15:08.301 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:07 smithi150 ceph-mon[29276]: 5.f scrub ok 2024-09-11T08:15:08.301 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:07 smithi150 ceph-mon[29276]: pgmap v328: 148 pgs: 148 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 65 MiB/s wr, 1.05k op/s 2024-09-11T08:15:08.301 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:07 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 0]}]': finished 2024-09-11T08:15:08.301 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:07 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [1, 2]}]': finished 2024-09-11T08:15:08.301 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:07 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.3", "id": [0, 3]}]': finished 2024-09-11T08:15:08.301 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:07 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1e", "id": [0, 6]}]': finished 2024-09-11T08:15:08.301 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:07 smithi150 ceph-mon[29276]: osdmap e189: 8 total, 8 up, 8 in 2024-09-11T08:15:08.301 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:07 smithi150 ceph-mon[39612]: 3.2 deep-scrub starts 2024-09-11T08:15:08.302 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:07 smithi150 ceph-mon[39612]: 3.2 deep-scrub ok 2024-09-11T08:15:08.302 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:07 smithi150 ceph-mon[39612]: 5.f scrub starts 2024-09-11T08:15:08.302 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:07 smithi150 ceph-mon[39612]: 5.f scrub ok 2024-09-11T08:15:08.302 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:07 smithi150 ceph-mon[39612]: pgmap v328: 148 pgs: 148 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 65 MiB/s wr, 1.05k op/s 2024-09-11T08:15:08.302 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:07 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 0]}]': finished 2024-09-11T08:15:08.302 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:07 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [1, 2]}]': finished 2024-09-11T08:15:08.302 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:07 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.3", "id": [0, 3]}]': finished 2024-09-11T08:15:08.302 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:07 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1e", "id": [0, 6]}]': finished 2024-09-11T08:15:08.302 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:07 smithi150 ceph-mon[39612]: osdmap e189: 8 total, 8 up, 8 in 2024-09-11T08:15:08.314 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:15:08.354 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:08.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:08.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:15:08.415 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 57 16 88548 88532 97.0569 80.25 0.00688523 0.0102988 2024-09-11T08:15:08.456 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:08.485 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:08.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:08.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:08.573 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:15:08 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:15:08.587 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:15:08.621 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:08.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:08.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:15:08.719 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:15:08.723 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:08.779 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-11T08:15:08.790 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:08.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:08.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:08.837 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:15:08 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:15:08] "GET /metrics HTTP/1.1" 200 224121 "" "Prometheus/2.43.0" 2024-09-11T08:15:08.892 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:08.942 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:08.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:08.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:09.044 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:09.079 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:09.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:09.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:09.155 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:08 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:15:09.155 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:08 smithi150 ceph-mon[29276]: osdmap e190: 8 total, 8 up, 8 in 2024-09-11T08:15:09.155 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:08 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:15:09.155 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:08 smithi150 ceph-mon[39612]: osdmap e190: 8 total, 8 up, 8 in 2024-09-11T08:15:09.163 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:15:09.181 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:09.217 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:09.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:09.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:09.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:08 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:15:09.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:08 smithi181 ceph-mon[32701]: osdmap e190: 8 total, 8 up, 8 in 2024-09-11T08:15:09.318 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:15:09.351 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:09.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:09.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:15:09.415 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 58 16 89770 89754 96.7002 76.375 0.0307727 0.0103359 2024-09-11T08:15:09.454 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:15:09.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:09.499 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:09.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:15:09.599 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:09.625 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:09.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:09.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:09.727 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:15:09.750 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:09.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:09.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:15:09.833 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:15:09.833 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:15:09.833 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:15:09.834 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:15:09.834 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:15:09.834 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:15:09.834 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:15:09.834 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:15:09.834 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:15:09.835 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:15:09.835 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:15:09.835 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:15:09.851 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:15:09.851 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:15:09.852 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:09.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:09.885 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:09.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:09.886 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:15:09.886 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:15:09.986 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:10.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:10.029 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:10.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:10.071 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T08:15:10.109 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:10 smithi150 ceph-mon[29276]: pgmap v331: 148 pgs: 4 peering, 144 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-09-11T08:15:10.110 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:10 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/829632070' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:15:10.110 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:10 smithi150 ceph-mon[39612]: pgmap v331: 148 pgs: 4 peering, 144 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-09-11T08:15:10.110 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:10 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/829632070' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:15:10.130 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:10.155 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:10.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:10.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:10.257 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:10.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:10.307 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:10.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:10.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:09 smithi181 ceph-mon[32701]: pgmap v331: 148 pgs: 4 peering, 144 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-09-11T08:15:10.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:09 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/829632070' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:15:10.409 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:10.415 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 59 16 90722 90706 96.0696 59.5 0.0110429 0.0104042 2024-09-11T08:15:10.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:10.448 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:10.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:10.549 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:15:10.578 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:10.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:10.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:15:10.680 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:15:10.719 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:10.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:10.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:15:10.821 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:10.860 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:10.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:10.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:10.962 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:10.985 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:10.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:10.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:11.088 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:15:11.122 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:11.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:11.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:15:11.224 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:11.260 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:11.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:11.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:11.362 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:11.388 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:11.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:11.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:11.416 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:2024-09-11T08:15:11.416332+0000 min lat: 0.0010471 max lat: 2.12372 avg lat: 0.0104438 2024-09-11T08:15:11.416 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-11T08:15:11.416 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 60 16 91912 91896 95.7067 74.375 0.0216744 0.0104438 2024-09-11T08:15:11.490 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:11.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:11.543 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:11.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:11.644 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:11.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:11.681 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:11.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:11.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:11 smithi150 ceph-mon[29276]: osdmap e191: 8 total, 8 up, 8 in 2024-09-11T08:15:11.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:11 smithi150 ceph-mon[29276]: pgmap v333: 148 pgs: 4 peering, 144 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 90 MiB/s wr, 1.43k op/s 2024-09-11T08:15:11.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:11 smithi150 ceph-mon[39612]: osdmap e191: 8 total, 8 up, 8 in 2024-09-11T08:15:11.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:11 smithi150 ceph-mon[39612]: pgmap v333: 148 pgs: 4 peering, 144 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 90 MiB/s wr, 1.43k op/s 2024-09-11T08:15:11.777 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:11 smithi181 ceph-mon[32701]: osdmap e191: 8 total, 8 up, 8 in 2024-09-11T08:15:11.777 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:11 smithi181 ceph-mon[32701]: pgmap v333: 148 pgs: 4 peering, 144 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 90 MiB/s wr, 1.43k op/s 2024-09-11T08:15:11.782 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:11.817 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:11.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:11.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:11.919 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:15:11.955 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:11.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:11.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:15:12.057 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:12.096 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:12.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:12.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:12.198 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:12.230 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:12.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:12.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:12.332 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:15:12.363 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:12.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:12.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:15:12.416 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 61 16 93125 93109 95.3805 75.8125 0.0113723 0.0104798 2024-09-11T08:15:12.464 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:15:12.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:12.519 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:12.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:15:12.620 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:12.643 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T08:15:12.644 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:15:12.648 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:12.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:12.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:12.750 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:15:12.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:12.797 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:12.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:15:12.898 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:15:12.924 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:12.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:12.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:15:13.026 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:13.060 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:13.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:13.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:13.162 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:15:13.200 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:13.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:13.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:15:13.302 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:13.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:13.344 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:13.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:13.416 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 62 16 94341 94325 95.0678 76 0.0146783 0.0105143 2024-09-11T08:15:13.446 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:13.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:13.496 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:13.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:13.597 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:13.634 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:13.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:13.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:13.736 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:15:13.773 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:13.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:13.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:15:13.842 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-11T08:15:13.842 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:15:13.875 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:13.916 INFO:teuthology.orchestra.run.smithi150.stderr:reweighted osd.4 to 1 (10000) 2024-09-11T08:15:13.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:13.919 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:13.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:13.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:13 smithi150 ceph-mon[29276]: pgmap v334: 148 pgs: 3 peering, 145 active+clean; 5.5 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-09-11T08:15:13.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:13 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2409325976' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-11T08:15:13.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:13 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-11T08:15:13.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:13 smithi150 ceph-mon[39612]: pgmap v334: 148 pgs: 3 peering, 145 active+clean; 5.5 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-09-11T08:15:13.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:13 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2409325976' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-11T08:15:13.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:13 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-11T08:15:14.021 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:15:14.063 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:14.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:14.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:15:14.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:13 smithi181 ceph-mon[32701]: pgmap v334: 148 pgs: 3 peering, 145 active+clean; 5.5 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-09-11T08:15:14.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:13 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2409325976' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-11T08:15:14.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:13 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-11T08:15:14.166 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:15:14.195 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:14.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:14.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:15:14.297 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:15:14.331 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:14.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:14.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:15:14.375 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:15:14.416 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 63 16 95620 95604 94.8275 79.9375 0.00972607 0.010541 2024-09-11T08:15:14.433 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:15:14.464 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:14.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:14.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:15:14.519 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-11T08:15:14.567 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:14.607 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:14.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:14.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:14.709 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:14.744 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:14.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:14.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:14.846 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:14.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:14.895 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:14.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:14.925 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:15:14.954 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd reweight 5 1 2024-09-11T08:15:14.996 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:15:15.027 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:15.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:15.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:15:15.129 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:15.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:15.170 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:15.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:15.204 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:14 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:15:15.205 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:14 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:15:15.205 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:14 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-11T08:15:15.205 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:14 smithi181 ceph-mon[32701]: osdmap e192: 8 total, 8 up, 8 in 2024-09-11T08:15:15.205 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:15:15 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:15:15] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:15:15.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:14 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:15:15.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:14 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:15:15.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:14 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-11T08:15:15.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:14 smithi150 ceph-mon[29276]: osdmap e192: 8 total, 8 up, 8 in 2024-09-11T08:15:15.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:14 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:15:15.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:14 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:15:15.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:14 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-11T08:15:15.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:14 smithi150 ceph-mon[39612]: osdmap e192: 8 total, 8 up, 8 in 2024-09-11T08:15:15.271 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:15.303 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:15.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:15.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:15.404 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:15.418 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 64 16 96885 96869 94.581 79.0625 0.0103938 0.010568 2024-09-11T08:15:15.437 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:15.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:15.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:15.538 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:15.568 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:15.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:15.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:15.669 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:15.708 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:15.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:15.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:15.810 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:15.844 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:15.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:15.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:15.946 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:15.980 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:15.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:15.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:16.081 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:15:16.108 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:16.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:16.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:15:16.210 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:15:16.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:15 smithi150 ceph-mon[29276]: pgmap v336: 148 pgs: 148 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 6.2 KiB/s rd, 60 MiB/s wr, 965 op/s; 58 B/s, 2 objects/s recovering 2024-09-11T08:15:16.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:15 smithi150 ceph-mon[29276]: osdmap e193: 8 total, 8 up, 8 in 2024-09-11T08:15:16.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:15 smithi150 ceph-mon[39612]: pgmap v336: 148 pgs: 148 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 6.2 KiB/s rd, 60 MiB/s wr, 965 op/s; 58 B/s, 2 objects/s recovering 2024-09-11T08:15:16.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:15 smithi150 ceph-mon[39612]: osdmap e193: 8 total, 8 up, 8 in 2024-09-11T08:15:16.250 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:16.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:16.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:15:16.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:15 smithi181 ceph-mon[32701]: pgmap v336: 148 pgs: 148 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 6.2 KiB/s rd, 60 MiB/s wr, 965 op/s; 58 B/s, 2 objects/s recovering 2024-09-11T08:15:16.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:15 smithi181 ceph-mon[32701]: osdmap e193: 8 total, 8 up, 8 in 2024-09-11T08:15:16.352 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:16.385 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:16.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:16.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:16.416 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 65 16 98019 98003 94.2162 70.875 0.0203018 0.0106091 2024-09-11T08:15:16.487 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:16.520 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:16.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:16.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:16.621 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:16.657 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:16.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:16.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:16.759 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:16.792 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:16.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:16.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:16.894 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:15:16.921 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:16.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:16.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:15:17.023 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:15:17.049 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:17.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:17.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:15:17.151 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:15:17.188 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:17.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:17.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:15:17.290 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:15:17.331 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:17.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:17.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:15:17.416 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 66 16 99194 99178 93.9014 73.4375 0.00733879 0.0106448 2024-09-11T08:15:17.433 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:15:17.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:17.456 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:17.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:15:17.557 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:15:17.581 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:17.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:17.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:15:17.685 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:17.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:17.712 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:17.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:17.813 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:15:17.838 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:17.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:17.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:15:17.940 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:17.946 INFO:teuthology.orchestra.run.smithi150.stderr:nodeep-scrub is set 2024-09-11T08:15:17.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:17.973 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:17.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:18.074 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:18.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:18.095 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:18.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:18.196 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:15:18.216 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:18.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:18.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:15:18.224 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:15:18 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:15:18.227 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:17 smithi181 ceph-mon[32701]: pgmap v338: 148 pgs: 148 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 6.0 KiB/s rd, 57 MiB/s wr, 924 op/s; 56 B/s, 2 objects/s recovering 2024-09-11T08:15:18.227 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:17 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/868282103' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:15:18.227 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:17 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:15:18.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:17 smithi150 ceph-mon[29276]: pgmap v338: 148 pgs: 148 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 6.0 KiB/s rd, 57 MiB/s wr, 924 op/s; 56 B/s, 2 objects/s recovering 2024-09-11T08:15:18.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:17 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/868282103' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:15:18.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:17 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:15:18.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:17 smithi150 ceph-mon[39612]: pgmap v338: 148 pgs: 148 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 6.0 KiB/s rd, 57 MiB/s wr, 924 op/s; 56 B/s, 2 objects/s recovering 2024-09-11T08:15:18.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:17 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/868282103' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:15:18.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:17 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:15:18.318 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:18.347 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:18.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:18.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:18.416 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 67 16 99949 99933 93.2041 47.1875 0.00835223 0.0107246 2024-09-11T08:15:18.448 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:15:18.485 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:18.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:18.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:15:18.587 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:18.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:18.625 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:18.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:18.726 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:15:18.764 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:18.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:18.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:15:18.830 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:15:18 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:15:18] "GET /metrics HTTP/1.1" 200 224121 "" "Prometheus/2.43.0" 2024-09-11T08:15:18.866 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:18.897 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:18.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:18.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:18.999 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:15:19.045 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:19.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:19.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:15:19.147 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:19.191 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:19.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:19.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:19.195 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:18 smithi150 ceph-mon[29276]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:15:19.195 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:18 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:15:19.195 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:18 smithi150 ceph-mon[29276]: osdmap e194: 8 total, 8 up, 8 in 2024-09-11T08:15:19.195 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:18 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:15:19.195 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:18 smithi150 ceph-mon[39612]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:15:19.196 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:18 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:15:19.196 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:18 smithi150 ceph-mon[39612]: osdmap e194: 8 total, 8 up, 8 in 2024-09-11T08:15:19.196 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:18 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:15:19.213 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:18 smithi181 ceph-mon[32701]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:15:19.213 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:18 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:15:19.213 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:18 smithi181 ceph-mon[32701]: osdmap e194: 8 total, 8 up, 8 in 2024-09-11T08:15:19.213 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:18 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:15:19.293 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:15:19.335 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:19.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:19.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:15:19.417 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 68 16 101233 101217 93.0136 80.25 0.0118017 0.0107462 2024-09-11T08:15:19.437 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:19.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:19.482 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:19.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:19.583 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:19.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:19.627 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:19.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:19.728 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:19.767 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:19.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:19.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:19.774 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-11T08:15:19.774 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:15:19.869 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:15:19.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:19.908 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:19.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:15:20.008 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:20.020 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:15:20.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:20.043 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:20.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:20.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:19 smithi181 ceph-mon[32701]: pgmap v340: 148 pgs: 148 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 6.3 KiB/s rd, 98 MiB/s wr, 1.58k op/s; 59 B/s, 2 objects/s recovering 2024-09-11T08:15:20.135 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:19 smithi150 ceph-mon[29276]: pgmap v340: 148 pgs: 148 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 6.3 KiB/s rd, 98 MiB/s wr, 1.58k op/s; 59 B/s, 2 objects/s recovering 2024-09-11T08:15:20.135 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:19 smithi150 ceph-mon[39612]: pgmap v340: 148 pgs: 148 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 6.3 KiB/s rd, 98 MiB/s wr, 1.58k op/s; 59 B/s, 2 objects/s recovering 2024-09-11T08:15:20.144 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:20.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:20.183 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:20.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:20.284 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:20.321 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:20.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:20.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:20.417 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 69 16 102476 102460 92.7914 77.6875 0.0122166 0.0107722 2024-09-11T08:15:20.424 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:15:20.457 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:20.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:20.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:15:20.496 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-11T08:15:20.559 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:20.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:20.621 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:20.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:20.722 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:15:20.757 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:20.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:20.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:15:20.859 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:15:20.906 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:20.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:20.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:15:20.963 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:15:21.008 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:15:21.047 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:21.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:21.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:15:21.149 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:15:21.182 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:21.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:21.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:15:21.284 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:21.323 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:21.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:21.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:21.417 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 70 16 103778 103762 92.6282 81.375 0.0117463 0.0107914 2024-09-11T08:15:21.425 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:15:21.466 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:21.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:21.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:15:21.568 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:21.603 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:21.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:21.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:21.705 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:15:21.732 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:21.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:21.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:15:21.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:21 smithi181 ceph-mon[32701]: osdmap e195: 8 total, 8 up, 8 in 2024-09-11T08:15:21.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:21 smithi181 ceph-mon[32701]: pgmap v342: 148 pgs: 148 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 61 MiB/s wr, 975 op/s 2024-09-11T08:15:21.834 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:15:21.866 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:21.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:21.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:15:21.968 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:15:21.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:21 smithi150 ceph-mon[29276]: osdmap e195: 8 total, 8 up, 8 in 2024-09-11T08:15:21.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:21 smithi150 ceph-mon[29276]: pgmap v342: 148 pgs: 148 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 61 MiB/s wr, 975 op/s 2024-09-11T08:15:21.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:21 smithi150 ceph-mon[39612]: osdmap e195: 8 total, 8 up, 8 in 2024-09-11T08:15:21.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:21 smithi150 ceph-mon[39612]: pgmap v342: 148 pgs: 148 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 61 MiB/s wr, 975 op/s 2024-09-11T08:15:22.005 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:22.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:22.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:15:22.011 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T08:15:22.108 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:22.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:22.145 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:22.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:22.246 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:15:22.288 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:22.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:22.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:15:22.390 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:15:22.418 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 71 16 105013 104997 92.4103 77.1875 0.0174226 0.0108169 2024-09-11T08:15:22.431 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:22.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:22.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:15:22.533 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:15:22.566 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:22.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:22.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:15:22.668 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:15:22.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:22.696 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:22.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:15:22.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:22 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:15:22.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:22 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:15:22.753 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:22 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:15:22.797 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:15:22.846 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:22.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:22.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:15:22.901 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-11T08:15:22.901 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:15:22.949 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:22.981 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:22.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:22.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:23.083 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:23.115 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:23.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:23.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:23.217 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:15:23.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:23.258 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:23.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:15:23.359 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:15:23.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:23.390 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:23.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:15:23.417 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 72 16 106230 106214 92.1831 76.0625 0.00638242 0.0108434 2024-09-11T08:15:23.491 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:15:23.521 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:23.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:23.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:15:23.622 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:15:23.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:23.662 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:23.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:15:23.763 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:15:23.808 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:23.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:23.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:15:23.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:23 smithi181 ceph-mon[32701]: pgmap v343: 148 pgs: 148 active+clean; 6.2 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 69 MiB/s wr, 1.11k op/s 2024-09-11T08:15:23.849 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:23 smithi150 ceph-mon[29276]: pgmap v343: 148 pgs: 148 active+clean; 6.2 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 69 MiB/s wr, 1.11k op/s 2024-09-11T08:15:23.849 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:23 smithi150 ceph-mon[39612]: pgmap v343: 148 pgs: 148 active+clean; 6.2 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 69 MiB/s wr, 1.11k op/s 2024-09-11T08:15:23.910 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:23.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:23.940 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:23.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:24.040 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:24.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:24.077 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:24.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:24.178 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:24.200 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:24.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:24.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:24.302 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:15:24.331 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:24.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:24.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:15:24.372 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:15:24.373 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:15:24.412 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:15:24.412 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:15:24.419 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 73 16 107290 107274 91.8259 66.25 0.0299523 0.0108854 2024-09-11T08:15:24.433 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:24.454 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:15:24.454 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:15:24.458 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:24.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:24.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:24.487 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:15:24.487 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:15:24.497 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-11T08:15:24.518 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:15:24.519 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:15:24.552 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:15:24.553 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:15:24.562 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:24.587 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:24.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:24.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:24.595 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:15:24.595 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:15:24.629 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:15:24.629 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:15:24.689 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:24.713 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:24 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1406439832' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:15:24.713 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:24 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1406439832' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:15:24.717 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:24.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:24.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:24.819 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:15:24.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:24 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1406439832' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:15:24.845 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:24.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:24.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:15:24.947 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:24.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:24.993 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:24.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:25.094 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:25.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:25.138 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:25.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:25.239 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:15:25.266 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:25.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:25.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:15:25.368 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:25.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:25.407 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:25.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:25.419 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 74 16 108333 108317 91.4659 65.1875 0.0105396 0.0109289 2024-09-11T08:15:25.508 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:25.529 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:15:25 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:15:25] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:15:25.547 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:25.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:25.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:25.648 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:25.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:25.683 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:25.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:25.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:25 smithi150 ceph-mon[29276]: pgmap v344: 148 pgs: 148 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-09-11T08:15:25.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:25 smithi150 ceph-mon[39612]: pgmap v344: 148 pgs: 148 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-09-11T08:15:25.785 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:15:25.816 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:25.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:25.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:15:25.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:25 smithi181 ceph-mon[32701]: pgmap v344: 148 pgs: 148 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-09-11T08:15:25.837 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:15:25.904 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:15:25.918 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:25.945 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:25.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:25.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:26.047 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:15:26.096 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:26.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:26.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:15:26.197 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:26.235 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:26.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:26.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:26.337 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:15:26.370 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:26.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:26.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:15:26.419 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 75 16 109617 109601 91.3163 80.25 0.00768644 0.0109462 2024-09-11T08:15:26.472 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:15:26.509 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:26.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:26.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:15:26.545 INFO:teuthology.orchestra.run.smithi150.stderr:reweighted osd.5 to 1 (10000) 2024-09-11T08:15:26.611 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:15:26.640 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:26.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:26.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:15:26.699 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:26 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3562447514' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-11T08:15:26.699 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:26 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-11T08:15:26.699 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:26 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3562447514' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-11T08:15:26.699 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:26 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-11T08:15:26.742 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:26.776 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:26.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:26.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:26.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:26 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3562447514' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-11T08:15:26.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:26 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-11T08:15:26.878 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:26.912 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:26.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:26.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:27.014 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:27.051 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:27.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:27.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:27.153 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:27.179 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:27.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:27.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:27.281 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:27.308 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:27.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:27.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:27.409 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:27.421 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 76 16 110981 110965 91.236 85.25 0.0155694 0.0109564 2024-09-11T08:15:27.445 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:27.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:27.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:27.547 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:15:27.579 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:27.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:27.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:15:27.681 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:15:27.715 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:27.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:27.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:15:27.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:27 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:15:27.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:27 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:15:27.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:27 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-11T08:15:27.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:27 smithi150 ceph-mon[39612]: osdmap e196: 8 total, 8 up, 8 in 2024-09-11T08:15:27.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:27 smithi150 ceph-mon[39612]: pgmap v346: 148 pgs: 148 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 48 MiB/s wr, 761 op/s 2024-09-11T08:15:27.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:27 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:15:27.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:27 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:15:27.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:27 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-11T08:15:27.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:27 smithi150 ceph-mon[29276]: osdmap e196: 8 total, 8 up, 8 in 2024-09-11T08:15:27.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:27 smithi150 ceph-mon[29276]: pgmap v346: 148 pgs: 148 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 48 MiB/s wr, 761 op/s 2024-09-11T08:15:27.814 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T08:15:27.817 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:27.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:27 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:15:27.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:27 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:15:27.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:27 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-11T08:15:27.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:27 smithi181 ceph-mon[32701]: osdmap e196: 8 total, 8 up, 8 in 2024-09-11T08:15:27.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:27 smithi181 ceph-mon[32701]: pgmap v346: 148 pgs: 148 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 48 MiB/s wr, 761 op/s 2024-09-11T08:15:27.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:27.853 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:27.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:27.954 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:28.001 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:28.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:28.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:28.103 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:28.151 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:28.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:28.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:28.253 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:15:28.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:28.288 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:28.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:15:28.389 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:15:28.419 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 77 15 112293 112278 91.1167 82.0625 0.00979529 0.0109706 2024-09-11T08:15:28.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:28.421 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:28.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:15:28.522 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:28.551 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:28.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:28.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:28.573 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:15:28 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:15:28.599 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd reweight 6 1 2024-09-11T08:15:28.652 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:28.680 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:28.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:28.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:28.783 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:15:28.846 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:28.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:28.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:15:28.852 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:28 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:15:28.852 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:28 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:15:28.852 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:28 smithi150 ceph-mon[29276]: osdmap e197: 8 total, 8 up, 8 in 2024-09-11T08:15:28.853 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:28 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:15:28.853 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:15:28 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:15:28] "GET /metrics HTTP/1.1" 200 224144 "" "Prometheus/2.43.0" 2024-09-11T08:15:28.854 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:28 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:15:28.854 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:28 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:15:28.854 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:28 smithi150 ceph-mon[39612]: osdmap e197: 8 total, 8 up, 8 in 2024-09-11T08:15:28.854 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:28 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:15:28.948 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:15:28.992 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:28.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:28.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:15:29.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:28 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:15:29.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:28 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:15:29.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:28 smithi181 ceph-mon[32701]: osdmap e197: 8 total, 8 up, 8 in 2024-09-11T08:15:29.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:28 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:15:29.094 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:15:29.121 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:29.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:29.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:15:29.223 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:29.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:29.271 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:29.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:29.372 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:29.419 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 78 16 113520 113504 90.9309 76.625 0.00676501 0.0109929 2024-09-11T08:15:29.425 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:29.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:29.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:29.527 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:29.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:29.558 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:29.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:29.658 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:15:29.696 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:29.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:29.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:15:29.737 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-11T08:15:29.737 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [ 2024-09-11T08:15:29.737 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:29.737 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15087.0:113903 8.e 8:73cb7f84:::benchmark_data_smithi150_107646_object113902:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-09-11T08:15:29.737 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:15:29.716917+0000", 2024-09-11T08:15:29.737 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.018454630999999999, 2024-09-11T08:15:29.737 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.01848783, 2024-09-11T08:15:29.738 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:15:29.738 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "waiting for sub ops", 2024-09-11T08:15:29.738 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:15:29.738 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15087", 2024-09-11T08:15:29.738 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/3299560913", 2024-09-11T08:15:29.738 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 113903 2024-09-11T08:15:29.738 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:29.738 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:15:29.738 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:29.738 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:15:29.738 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:29.716917+0000", 2024-09-11T08:15:29.738 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:15:29.738 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:29.739 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:29.739 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:15:29.739 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:29.716917+0000", 2024-09-11T08:15:29.739 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:15:29.739 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:29.739 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:29.739 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:15:29.739 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:29.716915+0000", 2024-09-11T08:15:29.739 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:15:29.739 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:29.739 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:29.739 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:15:29.739 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:29.716945+0000", 2024-09-11T08:15:29.740 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.0468999999999999e-05 2024-09-11T08:15:29.740 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:29.740 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:29.740 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:15:29.740 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:29.716959+0000", 2024-09-11T08:15:29.740 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3379e-05 2024-09-11T08:15:29.740 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:29.740 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:29.740 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:15:29.740 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:29.716966+0000", 2024-09-11T08:15:29.740 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.8990000000000004e-06 2024-09-11T08:15:29.740 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:29.740 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:29.741 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:15:29.741 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:29.726299+0000", 2024-09-11T08:15:29.741 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.009333694 2024-09-11T08:15:29.741 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:29.741 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:29.741 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:15:29.741 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:29.726409+0000", 2024-09-11T08:15:29.741 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000109564 2024-09-11T08:15:29.741 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:29.741 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:29.741 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 2,4", 2024-09-11T08:15:29.741 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:29.726537+0000", 2024-09-11T08:15:29.741 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012853800000000001 2024-09-11T08:15:29.742 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:29.742 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:29.742 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:29.742 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:29.732574+0000", 2024-09-11T08:15:29.742 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0060367179999999999 2024-09-11T08:15:29.742 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:29.742 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:29.742 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:29.742 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:29.734670+0000", 2024-09-11T08:15:29.742 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0020962160000000001 2024-09-11T08:15:29.742 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:29.742 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:29.742 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:15:29.742 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:29.735350+0000", 2024-09-11T08:15:29.743 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00067923199999999999 2024-09-11T08:15:29.743 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:15:29.743 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:15:29.743 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:15:29.743 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:29.743 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:29.743 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15087.0:113904 8.6 8:6938d0cc:::benchmark_data_smithi150_107646_object113903:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-09-11T08:15:29.743 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:15:29.717872+0000", 2024-09-11T08:15:29.743 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.017499849000000001, 2024-09-11T08:15:29.743 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.017638587000000001, 2024-09-11T08:15:29.743 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:15:29.743 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "waiting for sub ops", 2024-09-11T08:15:29.743 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:15:29.744 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15087", 2024-09-11T08:15:29.744 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/3299560913", 2024-09-11T08:15:29.744 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 113904 2024-09-11T08:15:29.744 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:29.744 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:15:29.744 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:29.744 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:15:29.744 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:29.717872+0000", 2024-09-11T08:15:29.744 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:15:29.744 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:29.744 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:29.744 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:15:29.744 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:29.717872+0000", 2024-09-11T08:15:29.745 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:15:29.745 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:29.745 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:29.745 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:15:29.745 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:29.717871+0000", 2024-09-11T08:15:29.745 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:15:29.745 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:29.745 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:29.745 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:15:29.745 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:29.717895+0000", 2024-09-11T08:15:29.745 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.4156999999999999e-05 2024-09-11T08:15:29.745 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:29.745 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:29.745 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:15:29.746 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:29.717907+0000", 2024-09-11T08:15:29.746 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2002e-05 2024-09-11T08:15:29.746 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:29.746 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:29.746 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:15:29.746 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:29.717913+0000", 2024-09-11T08:15:29.746 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.0270000000000001e-06 2024-09-11T08:15:29.746 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:29.746 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:29.746 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:15:29.746 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:29.732897+0000", 2024-09-11T08:15:29.746 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.014984302 2024-09-11T08:15:29.747 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:29.747 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:29.747 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:15:29.747 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:29.733057+0000", 2024-09-11T08:15:29.747 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000159982 2024-09-11T08:15:29.747 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:29.747 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:29.747 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 6,7", 2024-09-11T08:15:29.747 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:29.733189+0000", 2024-09-11T08:15:29.747 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000131805 2024-09-11T08:15:29.747 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:15:29.747 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:15:29.747 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:15:29.747 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:15:29.748 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-09-11T08:15:29.748 INFO:teuthology.orchestra.run.smithi150.stdout: "num_ops": 2 2024-09-11T08:15:29.748 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-11T08:15:29.748 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:29 smithi150 ceph-mon[29276]: pgmap v348: 148 pgs: 148 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.51k op/s 2024-09-11T08:15:29.748 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:29 smithi150 ceph-mon[39612]: pgmap v348: 148 pgs: 148 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.51k op/s 2024-09-11T08:15:29.798 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:29.828 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:29.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:29.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:29.930 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:29.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:29.963 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:29.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:30.064 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:30.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:29 smithi181 ceph-mon[32701]: pgmap v348: 148 pgs: 148 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.51k op/s 2024-09-11T08:15:30.117 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:30.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:30.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:30.221 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:15:30.247 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:30.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:30.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:15:30.349 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:15:30.386 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:15:30.395 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:30.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:30.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:15:30.420 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 79 16 114775 114759 90.7726 78.4375 0.0138646 0.0110118 2024-09-11T08:15:30.497 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:30.541 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:30.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:30.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:30.643 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:30.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:30.681 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:30.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:30.761 INFO:teuthology.orchestra.run.smithi150.stderr:noscrub is unset 2024-09-11T08:15:30.782 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:30 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1326994633' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:15:30.782 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:30 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:15:30.782 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:30 smithi150 ceph-mon[29276]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:15:30.782 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:30 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1326994633' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:15:30.782 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:30 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:15:30.782 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:30 smithi150 ceph-mon[39612]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:15:30.783 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:15:30.814 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:30.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:30.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:15:30.916 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:15:30.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:30.954 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:30.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:15:31.055 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:31.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:30 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1326994633' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:15:31.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:30 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:15:31.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:30 smithi181 ceph-mon[32701]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:15:31.089 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:31.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:31.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:31.191 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:31.229 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:31.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:31.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:31.331 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:31.367 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:31.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:31.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:31.420 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:2024-09-11T08:15:31.420421+0000 min lat: 0.0010471 max lat: 2.12372 avg lat: 0.0110277 2024-09-11T08:15:31.420 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-11T08:15:31.420 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 80 16 116060 116044 90.6418 80.3125 0.0195528 0.0110277 2024-09-11T08:15:31.469 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:31.492 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-11T08:15:31.496 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:31.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:31.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:31.598 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:31.623 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:31.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:31.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:31.725 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:31.776 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:31.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:31.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:31.878 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:15:31.934 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:31.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:31.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:15:31.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:31 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:15:31.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:31 smithi150 ceph-mon[39612]: osdmap e198: 8 total, 8 up, 8 in 2024-09-11T08:15:31.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:31 smithi150 ceph-mon[39612]: pgmap v350: 148 pgs: 148 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 63 MiB/s wr, 1.00k op/s 2024-09-11T08:15:31.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:31 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:15:31.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:31 smithi150 ceph-mon[29276]: osdmap e198: 8 total, 8 up, 8 in 2024-09-11T08:15:31.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:31 smithi150 ceph-mon[29276]: pgmap v350: 148 pgs: 148 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 63 MiB/s wr, 1.00k op/s 2024-09-11T08:15:32.037 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:32.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:31 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:15:32.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:31 smithi181 ceph-mon[32701]: osdmap e198: 8 total, 8 up, 8 in 2024-09-11T08:15:32.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:31 smithi181 ceph-mon[32701]: pgmap v350: 148 pgs: 148 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 63 MiB/s wr, 1.00k op/s 2024-09-11T08:15:32.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:32.074 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:32.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:32.175 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:15:32.203 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:32.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:32.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:15:32.305 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:15:32.340 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:32.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:32.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:15:32.420 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 81 16 117280 117264 90.464 76.25 0.00672744 0.0110495 2024-09-11T08:15:32.441 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:32.478 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:32.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:32.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:32.580 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:32.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:32.618 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:32.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:32.719 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:15:32.751 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:32.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:32.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:15:32.853 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:32.908 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:32.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:32.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:33.010 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:33.049 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:33.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:33.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:33.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:32 smithi181 ceph-mon[32701]: 3.1e scrub starts 2024-09-11T08:15:33.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:32 smithi181 ceph-mon[32701]: 3.10 scrub starts 2024-09-11T08:15:33.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:32 smithi181 ceph-mon[32701]: 3.1e scrub ok 2024-09-11T08:15:33.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:32 smithi181 ceph-mon[32701]: 3.10 scrub ok 2024-09-11T08:15:33.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:32 smithi181 ceph-mon[32701]: 3.0 scrub starts 2024-09-11T08:15:33.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:32 smithi181 ceph-mon[32701]: 3.0 scrub ok 2024-09-11T08:15:33.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:32 smithi181 ceph-mon[32701]: 3.11 scrub starts 2024-09-11T08:15:33.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:32 smithi181 ceph-mon[32701]: 3.11 scrub ok 2024-09-11T08:15:33.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:32 smithi181 ceph-mon[32701]: 3.1d scrub starts 2024-09-11T08:15:33.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:32 smithi181 ceph-mon[32701]: 3.1d scrub ok 2024-09-11T08:15:33.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:32 smithi181 ceph-mon[32701]: 6.8 scrub starts 2024-09-11T08:15:33.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:32 smithi181 ceph-mon[32701]: 6.8 scrub ok 2024-09-11T08:15:33.150 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:33.190 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:33.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:33.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:33.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:32 smithi150 ceph-mon[29276]: 3.1e scrub starts 2024-09-11T08:15:33.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:32 smithi150 ceph-mon[29276]: 3.10 scrub starts 2024-09-11T08:15:33.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:32 smithi150 ceph-mon[29276]: 3.1e scrub ok 2024-09-11T08:15:33.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:32 smithi150 ceph-mon[29276]: 3.10 scrub ok 2024-09-11T08:15:33.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:32 smithi150 ceph-mon[29276]: 3.0 scrub starts 2024-09-11T08:15:33.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:32 smithi150 ceph-mon[29276]: 3.0 scrub ok 2024-09-11T08:15:33.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:32 smithi150 ceph-mon[29276]: 3.11 scrub starts 2024-09-11T08:15:33.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:32 smithi150 ceph-mon[29276]: 3.11 scrub ok 2024-09-11T08:15:33.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:32 smithi150 ceph-mon[29276]: 3.1d scrub starts 2024-09-11T08:15:33.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:32 smithi150 ceph-mon[29276]: 3.1d scrub ok 2024-09-11T08:15:33.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:32 smithi150 ceph-mon[29276]: 6.8 scrub starts 2024-09-11T08:15:33.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:32 smithi150 ceph-mon[29276]: 6.8 scrub ok 2024-09-11T08:15:33.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:32 smithi150 ceph-mon[39612]: 3.1e scrub starts 2024-09-11T08:15:33.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:32 smithi150 ceph-mon[39612]: 3.10 scrub starts 2024-09-11T08:15:33.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:32 smithi150 ceph-mon[39612]: 3.1e scrub ok 2024-09-11T08:15:33.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:32 smithi150 ceph-mon[39612]: 3.10 scrub ok 2024-09-11T08:15:33.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:32 smithi150 ceph-mon[39612]: 3.0 scrub starts 2024-09-11T08:15:33.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:32 smithi150 ceph-mon[39612]: 3.0 scrub ok 2024-09-11T08:15:33.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:32 smithi150 ceph-mon[39612]: 3.11 scrub starts 2024-09-11T08:15:33.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:32 smithi150 ceph-mon[39612]: 3.11 scrub ok 2024-09-11T08:15:33.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:32 smithi150 ceph-mon[39612]: 3.1d scrub starts 2024-09-11T08:15:33.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:32 smithi150 ceph-mon[39612]: 3.1d scrub ok 2024-09-11T08:15:33.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:32 smithi150 ceph-mon[39612]: 6.8 scrub starts 2024-09-11T08:15:33.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:32 smithi150 ceph-mon[39612]: 6.8 scrub ok 2024-09-11T08:15:33.292 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:33.328 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:33.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:33.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:33.420 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 82 16 118607 118591 90.3721 82.9375 0.00897798 0.0110611 2024-09-11T08:15:33.429 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:15:33.472 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:33.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:33.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:15:33.574 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:33.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:33.610 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:33.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:33.711 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:33.745 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:33.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:33.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:33.798 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:33 smithi150 ceph-mon[29276]: pgmap v351: 148 pgs: 148 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.6 KiB/s rd, 83 MiB/s wr, 1.32k op/s 2024-09-11T08:15:33.798 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:33 smithi150 ceph-mon[29276]: 3.13 scrub starts 2024-09-11T08:15:33.798 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:33 smithi150 ceph-mon[29276]: 3.13 scrub ok 2024-09-11T08:15:33.847 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:33.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:33.882 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:33.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:33.983 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:15:34.009 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:34.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:34.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:15:34.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:33 smithi181 ceph-mon[32701]: pgmap v351: 148 pgs: 148 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.6 KiB/s rd, 83 MiB/s wr, 1.32k op/s 2024-09-11T08:15:34.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:33 smithi181 ceph-mon[32701]: 3.13 scrub starts 2024-09-11T08:15:34.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:33 smithi181 ceph-mon[32701]: 3.13 scrub ok 2024-09-11T08:15:34.101 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:33 smithi150 ceph-mon[39612]: pgmap v351: 148 pgs: 148 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.6 KiB/s rd, 83 MiB/s wr, 1.32k op/s 2024-09-11T08:15:34.101 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:33 smithi150 ceph-mon[39612]: 3.13 scrub starts 2024-09-11T08:15:34.101 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:33 smithi150 ceph-mon[39612]: 3.13 scrub ok 2024-09-11T08:15:34.110 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:34.115 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:15:34.115 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:15:34.165 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:34.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:34.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:34.174 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:15:34.175 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:15:34.221 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:15:34.221 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:15:34.251 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:15:34.251 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:15:34.266 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:34.302 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:15:34.302 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:15:34.316 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:34.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:34.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:34.330 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:15:34.330 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:15:34.363 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:15:34.363 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:15:34.398 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:15:34.399 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:15:34.418 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:15:34.421 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 83 16 119840 119824 90.2117 77.0625 0.0046645 0.0110804 2024-09-11T08:15:34.430 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:15:34.431 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:15:34.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:34.466 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:34.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:15:34.568 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:15:34.606 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:34.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:34.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:15:34.708 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:15:34.743 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:34.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:34.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:15:34.844 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:15:34.872 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:34.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:34.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:15:34.974 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:35.011 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:35.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:35.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:35.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:34 smithi181 ceph-mon[32701]: 6.3 scrub starts 2024-09-11T08:15:35.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:34 smithi181 ceph-mon[32701]: 6.3 scrub ok 2024-09-11T08:15:35.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:34 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2060092513' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:15:35.113 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:35.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:35.155 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:35.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:35.213 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:34 smithi150 ceph-mon[29276]: 6.3 scrub starts 2024-09-11T08:15:35.213 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:34 smithi150 ceph-mon[29276]: 6.3 scrub ok 2024-09-11T08:15:35.213 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:34 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2060092513' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:15:35.214 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:34 smithi150 ceph-mon[39612]: 6.3 scrub starts 2024-09-11T08:15:35.214 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:34 smithi150 ceph-mon[39612]: 6.3 scrub ok 2024-09-11T08:15:35.214 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:34 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2060092513' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:15:35.257 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:35.291 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:35.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:35.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:35.393 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:35.420 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 84 15 121163 121148 90.1228 82.75 0.0107398 0.0110917 2024-09-11T08:15:35.425 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:35.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:35.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:35.527 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:15:35.544 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T08:15:35.564 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:35.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:35.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:15:35.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:15:35 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:15:35] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:15:35.667 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:15:35.700 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:35.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:35.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:15:35.802 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:15:35.831 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:35.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:35.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:15:35.899 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:35 smithi150 ceph-mon[29276]: 3.c scrub starts 2024-09-11T08:15:35.899 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:35 smithi150 ceph-mon[29276]: 3.c scrub ok 2024-09-11T08:15:35.900 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:35 smithi150 ceph-mon[29276]: pgmap v352: 148 pgs: 148 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-09-11T08:15:35.900 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:35 smithi150 ceph-mon[29276]: 6.12 scrub starts 2024-09-11T08:15:35.900 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:35 smithi150 ceph-mon[29276]: 6.12 scrub ok 2024-09-11T08:15:35.900 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:35 smithi150 ceph-mon[29276]: osdmap e199: 8 total, 8 up, 8 in 2024-09-11T08:15:35.900 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:35 smithi150 ceph-mon[39612]: 3.c scrub starts 2024-09-11T08:15:35.900 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:35 smithi150 ceph-mon[39612]: 3.c scrub ok 2024-09-11T08:15:35.900 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:35 smithi150 ceph-mon[39612]: pgmap v352: 148 pgs: 148 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-09-11T08:15:35.900 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:35 smithi150 ceph-mon[39612]: 6.12 scrub starts 2024-09-11T08:15:35.900 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:35 smithi150 ceph-mon[39612]: 6.12 scrub ok 2024-09-11T08:15:35.900 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:35 smithi150 ceph-mon[39612]: osdmap e199: 8 total, 8 up, 8 in 2024-09-11T08:15:35.933 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:35.958 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:35.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:35.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:36.060 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:36.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:35 smithi181 ceph-mon[32701]: 3.c scrub starts 2024-09-11T08:15:36.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:35 smithi181 ceph-mon[32701]: 3.c scrub ok 2024-09-11T08:15:36.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:35 smithi181 ceph-mon[32701]: pgmap v352: 148 pgs: 148 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-09-11T08:15:36.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:35 smithi181 ceph-mon[32701]: 6.12 scrub starts 2024-09-11T08:15:36.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:35 smithi181 ceph-mon[32701]: 6.12 scrub ok 2024-09-11T08:15:36.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:35 smithi181 ceph-mon[32701]: osdmap e199: 8 total, 8 up, 8 in 2024-09-11T08:15:36.095 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:36.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:36.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:36.197 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:15:36.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:36.232 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:36.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:15:36.333 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:36.371 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:36.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:36.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:36.421 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 85 16 122419 122403 89.9853 78.4375 0.00801657 0.0111086 2024-09-11T08:15:36.473 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:36.507 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:36.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:36.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:36.609 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:36.644 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:36.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:36.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:36.745 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:15:36.777 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:36.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:36.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:15:36.880 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:36.918 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:36.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:36.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:36.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:36 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:15:36.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:36 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:15:37.019 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:37.059 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:37.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:37.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:37.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:36 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:15:37.161 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:15:37.194 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:37.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:37.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:15:37.296 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:37.330 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:37.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:37.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:37.421 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 86 16 123761 123745 89.9141 83.875 0.00696655 0.0111175 2024-09-11T08:15:37.431 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:37.465 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:37.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:37.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:37.566 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:15:37.599 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:37.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:37.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:15:37.701 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:15:37.734 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:37.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:37.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:15:37.836 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:15:37.875 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:37.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:37.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:15:37.977 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:15:37.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:37 smithi150 ceph-mon[39612]: 8.5 scrub starts 2024-09-11T08:15:37.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:37 smithi150 ceph-mon[39612]: 4.2 scrub starts 2024-09-11T08:15:37.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:37 smithi150 ceph-mon[39612]: 4.2 scrub ok 2024-09-11T08:15:37.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:37 smithi150 ceph-mon[39612]: pgmap v354: 148 pgs: 148 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 49 MiB/s wr, 790 op/s 2024-09-11T08:15:37.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:37 smithi150 ceph-mon[39612]: 4.18 scrub starts 2024-09-11T08:15:37.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:37 smithi150 ceph-mon[39612]: 4.18 scrub ok 2024-09-11T08:15:37.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:37 smithi150 ceph-mon[39612]: 5.a scrub starts 2024-09-11T08:15:37.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:37 smithi150 ceph-mon[39612]: 5.a scrub ok 2024-09-11T08:15:37.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:37 smithi150 ceph-mon[29276]: 8.5 scrub starts 2024-09-11T08:15:37.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:37 smithi150 ceph-mon[29276]: 4.2 scrub starts 2024-09-11T08:15:37.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:37 smithi150 ceph-mon[29276]: 4.2 scrub ok 2024-09-11T08:15:37.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:37 smithi150 ceph-mon[29276]: pgmap v354: 148 pgs: 148 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 49 MiB/s wr, 790 op/s 2024-09-11T08:15:37.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:37 smithi150 ceph-mon[29276]: 4.18 scrub starts 2024-09-11T08:15:37.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:37 smithi150 ceph-mon[29276]: 4.18 scrub ok 2024-09-11T08:15:37.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:37 smithi150 ceph-mon[29276]: 5.a scrub starts 2024-09-11T08:15:37.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:37 smithi150 ceph-mon[29276]: 5.a scrub ok 2024-09-11T08:15:38.010 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:38.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:38.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:15:38.112 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:38.138 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:37 smithi181 ceph-mon[32701]: 8.5 scrub starts 2024-09-11T08:15:38.138 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:37 smithi181 ceph-mon[32701]: 4.2 scrub starts 2024-09-11T08:15:38.138 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:37 smithi181 ceph-mon[32701]: 4.2 scrub ok 2024-09-11T08:15:38.138 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:37 smithi181 ceph-mon[32701]: pgmap v354: 148 pgs: 148 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 49 MiB/s wr, 790 op/s 2024-09-11T08:15:38.138 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:37 smithi181 ceph-mon[32701]: 4.18 scrub starts 2024-09-11T08:15:38.138 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:37 smithi181 ceph-mon[32701]: 4.18 scrub ok 2024-09-11T08:15:38.139 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:37 smithi181 ceph-mon[32701]: 5.a scrub starts 2024-09-11T08:15:38.139 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:37 smithi181 ceph-mon[32701]: 5.a scrub ok 2024-09-11T08:15:38.149 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:38.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:38.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:38.251 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:38.288 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:38.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:38.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:38.390 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:15:38.420 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 87 16 125081 125065 89.8288 82.5 0.00852093 0.0111279 2024-09-11T08:15:38.429 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:38.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:38.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:15:38.431 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T08:15:38.531 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:38.573 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:15:38 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:15:38.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:38.575 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:38.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:38.676 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:15:38.718 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:38.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:38.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:15:38.820 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:38.825 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:15:38 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:15:38] "GET /metrics HTTP/1.1" 200 224151 "" "Prometheus/2.43.0" 2024-09-11T08:15:38.825 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:38 smithi150 ceph-mon[39612]: 5.11 scrub starts 2024-09-11T08:15:38.825 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:38 smithi150 ceph-mon[39612]: 5.11 scrub ok 2024-09-11T08:15:38.825 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:38 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:15:38.826 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:38 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:15:38.826 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:38 smithi150 ceph-mon[39612]: osdmap e200: 8 total, 8 up, 8 in 2024-09-11T08:15:38.826 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:38 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:15:38.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:38.865 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:38.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:38.966 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:39.010 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:39.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:39.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:39.075 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:38 smithi150 ceph-mon[29276]: 5.11 scrub starts 2024-09-11T08:15:39.075 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:38 smithi150 ceph-mon[29276]: 5.11 scrub ok 2024-09-11T08:15:39.076 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:38 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:15:39.076 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:38 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:15:39.076 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:38 smithi150 ceph-mon[29276]: osdmap e200: 8 total, 8 up, 8 in 2024-09-11T08:15:39.076 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:38 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:15:39.111 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:39.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:39.152 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:39.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:39.253 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:39.290 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:39.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:39.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:39.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:38 smithi181 ceph-mon[32701]: 5.11 scrub starts 2024-09-11T08:15:39.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:38 smithi181 ceph-mon[32701]: 5.11 scrub ok 2024-09-11T08:15:39.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:38 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:15:39.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:38 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:15:39.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:38 smithi181 ceph-mon[32701]: osdmap e200: 8 total, 8 up, 8 in 2024-09-11T08:15:39.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:38 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:15:39.392 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:39.418 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-09-11T08:15:39.418 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:15:39.421 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 88 16 126267 126251 89.6503 74.125 0.0166893 0.0111502 2024-09-11T08:15:39.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:39.422 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:39.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:39.523 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:15:39.562 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:39.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:39.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:15:39.663 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:15:39.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:39.708 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:39.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:15:39.809 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:39.843 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:39.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:39.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:39.945 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:15:39.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:39.981 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:39.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:15:40.082 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:40.120 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:39 smithi150 ceph-mon[39612]: 5.9 scrub starts 2024-09-11T08:15:40.120 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:39 smithi150 ceph-mon[39612]: 5.9 scrub ok 2024-09-11T08:15:40.120 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:39 smithi150 ceph-mon[39612]: 8.5 scrub ok 2024-09-11T08:15:40.120 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:39 smithi150 ceph-mon[39612]: 5.12 scrub starts 2024-09-11T08:15:40.120 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:39 smithi150 ceph-mon[39612]: 5.12 scrub ok 2024-09-11T08:15:40.120 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:39 smithi150 ceph-mon[39612]: pgmap v356: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.6 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-09-11T08:15:40.121 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:39 smithi150 ceph-mon[29276]: 5.9 scrub starts 2024-09-11T08:15:40.121 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:39 smithi150 ceph-mon[29276]: 5.9 scrub ok 2024-09-11T08:15:40.121 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:39 smithi150 ceph-mon[29276]: 8.5 scrub ok 2024-09-11T08:15:40.121 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:39 smithi150 ceph-mon[29276]: 5.12 scrub starts 2024-09-11T08:15:40.121 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:39 smithi150 ceph-mon[29276]: 5.12 scrub ok 2024-09-11T08:15:40.121 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:39 smithi150 ceph-mon[29276]: pgmap v356: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.6 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-09-11T08:15:40.128 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:40.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:40.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:40.230 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:40.264 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:40.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:40.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:40.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:39 smithi181 ceph-mon[32701]: 5.9 scrub starts 2024-09-11T08:15:40.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:39 smithi181 ceph-mon[32701]: 5.9 scrub ok 2024-09-11T08:15:40.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:39 smithi181 ceph-mon[32701]: 8.5 scrub ok 2024-09-11T08:15:40.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:39 smithi181 ceph-mon[32701]: 5.12 scrub starts 2024-09-11T08:15:40.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:39 smithi181 ceph-mon[32701]: 5.12 scrub ok 2024-09-11T08:15:40.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:39 smithi181 ceph-mon[32701]: pgmap v356: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.6 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-09-11T08:15:40.365 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:15:40.397 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:40.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:40.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:15:40.420 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 89 16 127591 127575 89.5727 82.75 0.00487577 0.0111595 2024-09-11T08:15:40.499 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:15:40.541 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:40.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:40.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:15:40.643 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:15:40.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:40.683 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:40.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:15:40.784 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:15:40.819 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:40.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:40.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:15:40.853 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:15:40.884 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-11T08:15:40.921 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:40.950 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:40.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:40.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:41.052 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:41.107 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:40 smithi150 ceph-mon[29276]: 3.17 scrub starts 2024-09-11T08:15:41.107 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:40 smithi150 ceph-mon[29276]: 3.17 scrub ok 2024-09-11T08:15:41.107 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:40 smithi150 ceph-mon[29276]: osdmap e201: 8 total, 8 up, 8 in 2024-09-11T08:15:41.107 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:40 smithi150 ceph-mon[39612]: 3.17 scrub starts 2024-09-11T08:15:41.107 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:40 smithi150 ceph-mon[39612]: 3.17 scrub ok 2024-09-11T08:15:41.108 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:40 smithi150 ceph-mon[39612]: osdmap e201: 8 total, 8 up, 8 in 2024-09-11T08:15:41.118 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:41.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:41.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:41.223 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:15:41.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:41.259 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:41.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:15:41.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:40 smithi181 ceph-mon[32701]: 3.17 scrub starts 2024-09-11T08:15:41.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:40 smithi181 ceph-mon[32701]: 3.17 scrub ok 2024-09-11T08:15:41.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:40 smithi181 ceph-mon[32701]: osdmap e201: 8 total, 8 up, 8 in 2024-09-11T08:15:41.361 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:41.395 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:41.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:41.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:41.421 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Total time run: 90.0085 2024-09-11T08:15:41.421 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Total writes made: 128829 2024-09-11T08:15:41.421 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Write size: 65536 2024-09-11T08:15:41.421 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Object size: 65536 2024-09-11T08:15:41.421 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Bandwidth (MB/sec): 89.4562 2024-09-11T08:15:41.421 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Stddev Bandwidth: 38.5942 2024-09-11T08:15:41.421 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Max bandwidth (MB/sec): 238.875 2024-09-11T08:15:41.421 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Min bandwidth (MB/sec): 0 2024-09-11T08:15:41.421 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Average IOPS: 1431 2024-09-11T08:15:41.422 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Stddev IOPS: 617.516 2024-09-11T08:15:41.422 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Max IOPS: 3822 2024-09-11T08:15:41.422 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Min IOPS: 0 2024-09-11T08:15:41.422 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Average Latency(s): 0.0111747 2024-09-11T08:15:41.422 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Stddev Latency(s): 0.0286331 2024-09-11T08:15:41.422 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Max latency(s): 2.12372 2024-09-11T08:15:41.422 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Min latency(s): 0.0010471 2024-09-11T08:15:41.425 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Cleaning up (deleting benchmark objects) 2024-09-11T08:15:41.497 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:41.526 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:41.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:41.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:41.628 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:41.655 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:15:41.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:41.675 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:41.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:41.777 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:15:41.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:41.824 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:41.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:15:41.925 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:41.960 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:41.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:41.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:42.063 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:15:42.090 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:42.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:42.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:15:42.175 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:41 smithi150 ceph-mon[29276]: pgmap v358: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.6 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 852 B/s rd, 68 MiB/s wr, 1.08k op/s 2024-09-11T08:15:42.178 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:41 smithi150 ceph-mon[39612]: pgmap v358: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.6 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 852 B/s rd, 68 MiB/s wr, 1.08k op/s 2024-09-11T08:15:42.191 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:42.222 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:42.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:42.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:42.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:41 smithi181 ceph-mon[32701]: pgmap v358: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.6 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 852 B/s rd, 68 MiB/s wr, 1.08k op/s 2024-09-11T08:15:42.324 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:15:42.358 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:42.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:42.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:15:42.460 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:15:42.497 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:42.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:42.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:15:42.599 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:15:42.647 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:42.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:42.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:15:42.749 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:42.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:42.779 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:42.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:42.872 INFO:teuthology.orchestra.run.smithi150.stderr:reweighted osd.6 to 1 (10000) 2024-09-11T08:15:42.881 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:15:42.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:42.907 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:42.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:15:42.991 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:42 smithi150 ceph-mon[29276]: 4.10 scrub starts 2024-09-11T08:15:42.991 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:42 smithi150 ceph-mon[29276]: 4.10 scrub ok 2024-09-11T08:15:42.991 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:42 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1977558606' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-11T08:15:42.991 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:42 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-11T08:15:42.991 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:42 smithi150 ceph-mon[29276]: 4.d scrub starts 2024-09-11T08:15:42.991 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:42 smithi150 ceph-mon[29276]: 4.d scrub ok 2024-09-11T08:15:42.992 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:42 smithi150 ceph-mon[39612]: 4.10 scrub starts 2024-09-11T08:15:42.992 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:42 smithi150 ceph-mon[39612]: 4.10 scrub ok 2024-09-11T08:15:42.992 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:42 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1977558606' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-11T08:15:42.992 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:42 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-11T08:15:42.992 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:42 smithi150 ceph-mon[39612]: 4.d scrub starts 2024-09-11T08:15:42.992 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:42 smithi150 ceph-mon[39612]: 4.d scrub ok 2024-09-11T08:15:43.009 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:43.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:43.041 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:43.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:43.142 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:15:43.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:43.181 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:43.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:15:43.282 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:43.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:43.310 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:43.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:43.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:42 smithi181 ceph-mon[32701]: 4.10 scrub starts 2024-09-11T08:15:43.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:42 smithi181 ceph-mon[32701]: 4.10 scrub ok 2024-09-11T08:15:43.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:42 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1977558606' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-11T08:15:43.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:42 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-11T08:15:43.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:42 smithi181 ceph-mon[32701]: 4.d scrub starts 2024-09-11T08:15:43.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:42 smithi181 ceph-mon[32701]: 4.d scrub ok 2024-09-11T08:15:43.411 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:15:43.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:43.450 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:43.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:15:43.551 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:43.593 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:43.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:43.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:43.631 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd reweight 7 1 2024-09-11T08:15:43.695 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:43.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:43.746 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:43.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:43.847 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:15:43.873 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:43.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:43.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:15:43.975 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:43.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:43 smithi150 ceph-mon[29276]: 4.f scrub starts 2024-09-11T08:15:43.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:43 smithi150 ceph-mon[29276]: 4.f scrub ok 2024-09-11T08:15:43.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:43 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:15:43.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:43 smithi150 ceph-mon[29276]: pgmap v359: 148 pgs: 148 active+clean; 7.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.5 KiB/s rd, 69 MiB/s wr, 1.20k op/s 2024-09-11T08:15:43.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:43 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-11T08:15:43.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:43 smithi150 ceph-mon[29276]: osdmap e202: 8 total, 8 up, 8 in 2024-09-11T08:15:43.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:43 smithi150 ceph-mon[29276]: 6.f scrub starts 2024-09-11T08:15:43.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:43 smithi150 ceph-mon[29276]: 6.f scrub ok 2024-09-11T08:15:43.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:43 smithi150 ceph-mon[39612]: 4.f scrub starts 2024-09-11T08:15:43.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:43 smithi150 ceph-mon[39612]: 4.f scrub ok 2024-09-11T08:15:43.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:43 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:15:43.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:43 smithi150 ceph-mon[39612]: pgmap v359: 148 pgs: 148 active+clean; 7.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.5 KiB/s rd, 69 MiB/s wr, 1.20k op/s 2024-09-11T08:15:43.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:43 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-11T08:15:43.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:43 smithi150 ceph-mon[39612]: osdmap e202: 8 total, 8 up, 8 in 2024-09-11T08:15:43.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:43 smithi150 ceph-mon[39612]: 6.f scrub starts 2024-09-11T08:15:43.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:43 smithi150 ceph-mon[39612]: 6.f scrub ok 2024-09-11T08:15:44.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:44.021 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:44.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:44.122 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:15:44.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:44.178 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:44.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:15:44.279 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:15:44.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:43 smithi181 ceph-mon[32701]: 4.f scrub starts 2024-09-11T08:15:44.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:43 smithi181 ceph-mon[32701]: 4.f scrub ok 2024-09-11T08:15:44.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:43 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:15:44.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:43 smithi181 ceph-mon[32701]: pgmap v359: 148 pgs: 148 active+clean; 7.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.5 KiB/s rd, 69 MiB/s wr, 1.20k op/s 2024-09-11T08:15:44.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:43 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-11T08:15:44.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:43 smithi181 ceph-mon[32701]: osdmap e202: 8 total, 8 up, 8 in 2024-09-11T08:15:44.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:43 smithi181 ceph-mon[32701]: 6.f scrub starts 2024-09-11T08:15:44.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:43 smithi181 ceph-mon[32701]: 6.f scrub ok 2024-09-11T08:15:44.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:44.329 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:44.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:15:44.430 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:44.463 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:44.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:44.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:44.565 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:15:44.597 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:44.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:44.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:15:44.699 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:44.733 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:44.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:44.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:44.835 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:44.863 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:44.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:44.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:44.965 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:45.001 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:45.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:45.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:45.103 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:45.129 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:45.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:45.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:45.143 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:44 smithi181 ceph-mon[32701]: 8.9 scrub starts 2024-09-11T08:15:45.143 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:44 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:15:45.143 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:44 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:15:45.143 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:44 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:15:45.149 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:15:45.204 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:15:45.231 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:15:45.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:44 smithi150 ceph-mon[29276]: 8.9 scrub starts 2024-09-11T08:15:45.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:44 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:15:45.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:44 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:15:45.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:44 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:15:45.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:44 smithi150 ceph-mon[39612]: 8.9 scrub starts 2024-09-11T08:15:45.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:44 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:15:45.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:44 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:15:45.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:44 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:15:45.270 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:45.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:45.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:15:45.372 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:15:45.416 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:45.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:45.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:15:45.518 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:15:45.538 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:45.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:45.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:15:45.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:15:45 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:15:45] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:15:45.641 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:15:45.672 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:45.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:45.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:15:45.774 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:45.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:45.808 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:45.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:45.909 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:15:45.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:45.937 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:45.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:15:45.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:45 smithi150 ceph-mon[29276]: 5.15 scrub starts 2024-09-11T08:15:45.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:45 smithi150 ceph-mon[29276]: 5.15 scrub ok 2024-09-11T08:15:45.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:45 smithi150 ceph-mon[29276]: 6.13 scrub starts 2024-09-11T08:15:45.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:45 smithi150 ceph-mon[29276]: 6.13 scrub ok 2024-09-11T08:15:45.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:45 smithi150 ceph-mon[29276]: pgmap v361: 148 pgs: 148 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 872 B/s rd, 74 MiB/s wr, 1.45k op/s 2024-09-11T08:15:45.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:45 smithi150 ceph-mon[29276]: 3.3 scrub starts 2024-09-11T08:15:45.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:45 smithi150 ceph-mon[29276]: 3.3 scrub ok 2024-09-11T08:15:45.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:45 smithi150 ceph-mon[29276]: osdmap e203: 8 total, 8 up, 8 in 2024-09-11T08:15:45.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:45 smithi150 ceph-mon[39612]: 5.15 scrub starts 2024-09-11T08:15:45.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:45 smithi150 ceph-mon[39612]: 5.15 scrub ok 2024-09-11T08:15:45.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:45 smithi150 ceph-mon[39612]: 6.13 scrub starts 2024-09-11T08:15:45.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:45 smithi150 ceph-mon[39612]: 6.13 scrub ok 2024-09-11T08:15:45.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:45 smithi150 ceph-mon[39612]: pgmap v361: 148 pgs: 148 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 872 B/s rd, 74 MiB/s wr, 1.45k op/s 2024-09-11T08:15:45.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:45 smithi150 ceph-mon[39612]: 3.3 scrub starts 2024-09-11T08:15:45.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:45 smithi150 ceph-mon[39612]: 3.3 scrub ok 2024-09-11T08:15:45.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:45 smithi150 ceph-mon[39612]: osdmap e203: 8 total, 8 up, 8 in 2024-09-11T08:15:46.038 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:15:46.074 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:46.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:46.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:15:46.176 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:46.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:46.211 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:46.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:46.312 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:15:46.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:45 smithi181 ceph-mon[32701]: 5.15 scrub starts 2024-09-11T08:15:46.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:45 smithi181 ceph-mon[32701]: 5.15 scrub ok 2024-09-11T08:15:46.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:45 smithi181 ceph-mon[32701]: 6.13 scrub starts 2024-09-11T08:15:46.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:45 smithi181 ceph-mon[32701]: 6.13 scrub ok 2024-09-11T08:15:46.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:45 smithi181 ceph-mon[32701]: pgmap v361: 148 pgs: 148 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 872 B/s rd, 74 MiB/s wr, 1.45k op/s 2024-09-11T08:15:46.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:45 smithi181 ceph-mon[32701]: 3.3 scrub starts 2024-09-11T08:15:46.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:45 smithi181 ceph-mon[32701]: 3.3 scrub ok 2024-09-11T08:15:46.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:45 smithi181 ceph-mon[32701]: osdmap e203: 8 total, 8 up, 8 in 2024-09-11T08:15:46.344 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:46.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:46.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:15:46.446 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:46.474 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:46.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:46.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:46.575 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:46.611 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:46.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:46.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:46.712 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:46.749 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:46.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:46.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:46.852 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:15:46.877 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:46.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:46.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:15:46.979 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:46.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:46 smithi150 ceph-mon[39612]: 6.d scrub starts 2024-09-11T08:15:46.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:46 smithi150 ceph-mon[39612]: 6.d scrub ok 2024-09-11T08:15:46.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:46 smithi150 ceph-mon[29276]: 6.d scrub starts 2024-09-11T08:15:46.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:46 smithi150 ceph-mon[29276]: 6.d scrub ok 2024-09-11T08:15:47.013 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:47.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:47.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:47.115 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:15:47.136 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:47.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:47.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:15:47.238 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:47.268 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:47.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:47.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:47.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:46 smithi181 ceph-mon[32701]: 6.d scrub starts 2024-09-11T08:15:47.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:46 smithi181 ceph-mon[32701]: 6.d scrub ok 2024-09-11T08:15:47.371 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:47.398 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:47.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:47.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:47.499 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:15:47.529 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:47.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:47.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:15:47.631 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:47.662 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:47.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:47.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:47.764 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:47.793 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:47.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:47.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:47.895 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:15:47.924 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:47.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:47.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:15:48.026 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:15:48.048 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:48.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:48.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:15:48.149 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:48.150 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:47 smithi181 ceph-mon[32701]: 3.5 scrub starts 2024-09-11T08:15:48.150 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:47 smithi181 ceph-mon[32701]: 3.5 scrub ok 2024-09-11T08:15:48.150 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:47 smithi181 ceph-mon[32701]: 5.1d scrub starts 2024-09-11T08:15:48.150 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:47 smithi181 ceph-mon[32701]: 5.1d scrub ok 2024-09-11T08:15:48.150 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:47 smithi181 ceph-mon[32701]: pgmap v363: 148 pgs: 148 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 965 B/s rd, 42 MiB/s wr, 966 op/s 2024-09-11T08:15:48.150 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:47 smithi181 ceph-mon[32701]: 8.9 scrub ok 2024-09-11T08:15:48.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:48.186 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:48.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:48.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:47 smithi150 ceph-mon[29276]: 3.5 scrub starts 2024-09-11T08:15:48.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:47 smithi150 ceph-mon[29276]: 3.5 scrub ok 2024-09-11T08:15:48.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:47 smithi150 ceph-mon[29276]: 5.1d scrub starts 2024-09-11T08:15:48.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:47 smithi150 ceph-mon[29276]: 5.1d scrub ok 2024-09-11T08:15:48.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:47 smithi150 ceph-mon[29276]: pgmap v363: 148 pgs: 148 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 965 B/s rd, 42 MiB/s wr, 966 op/s 2024-09-11T08:15:48.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:47 smithi150 ceph-mon[29276]: 8.9 scrub ok 2024-09-11T08:15:48.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:47 smithi150 ceph-mon[39612]: 3.5 scrub starts 2024-09-11T08:15:48.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:47 smithi150 ceph-mon[39612]: 3.5 scrub ok 2024-09-11T08:15:48.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:47 smithi150 ceph-mon[39612]: 5.1d scrub starts 2024-09-11T08:15:48.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:47 smithi150 ceph-mon[39612]: 5.1d scrub ok 2024-09-11T08:15:48.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:47 smithi150 ceph-mon[39612]: pgmap v363: 148 pgs: 148 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 965 B/s rd, 42 MiB/s wr, 966 op/s 2024-09-11T08:15:48.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:47 smithi150 ceph-mon[39612]: 8.9 scrub ok 2024-09-11T08:15:48.287 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:48.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:48.324 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:48.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:48.425 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:48.459 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:48.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:48.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:48.561 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:48.574 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:15:48 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:15:48.601 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:48.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:48.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:48.703 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:48.738 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:15:48 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:15:48] "GET /metrics HTTP/1.1" 200 224151 "" "Prometheus/2.43.0" 2024-09-11T08:15:48.747 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:48.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:48.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:48.848 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:15:48.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:48.887 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:48.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:15:48.989 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:15:49.021 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:48 smithi150 ceph-mon[29276]: 3.18 scrub starts 2024-09-11T08:15:49.021 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:48 smithi150 ceph-mon[29276]: 4.8 scrub starts 2024-09-11T08:15:49.021 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:48 smithi150 ceph-mon[29276]: 4.8 scrub ok 2024-09-11T08:15:49.021 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:48 smithi150 ceph-mon[29276]: 3.18 scrub ok 2024-09-11T08:15:49.022 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:48 smithi150 ceph-mon[29276]: 3.1a scrub starts 2024-09-11T08:15:49.022 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:48 smithi150 ceph-mon[29276]: 3.1a scrub ok 2024-09-11T08:15:49.022 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:48 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:15:49.022 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:48 smithi150 ceph-mon[39612]: 3.18 scrub starts 2024-09-11T08:15:49.022 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:48 smithi150 ceph-mon[39612]: 4.8 scrub starts 2024-09-11T08:15:49.022 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:48 smithi150 ceph-mon[39612]: 4.8 scrub ok 2024-09-11T08:15:49.022 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:48 smithi150 ceph-mon[39612]: 3.18 scrub ok 2024-09-11T08:15:49.022 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:48 smithi150 ceph-mon[39612]: 3.1a scrub starts 2024-09-11T08:15:49.022 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:48 smithi150 ceph-mon[39612]: 3.1a scrub ok 2024-09-11T08:15:49.022 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:48 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:15:49.024 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:49.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:49.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:15:49.034 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:48 smithi181 ceph-mon[32701]: 3.18 scrub starts 2024-09-11T08:15:49.034 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:48 smithi181 ceph-mon[32701]: 4.8 scrub starts 2024-09-11T08:15:49.034 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:48 smithi181 ceph-mon[32701]: 4.8 scrub ok 2024-09-11T08:15:49.034 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:48 smithi181 ceph-mon[32701]: 3.18 scrub ok 2024-09-11T08:15:49.034 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:48 smithi181 ceph-mon[32701]: 3.1a scrub starts 2024-09-11T08:15:49.034 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:48 smithi181 ceph-mon[32701]: 3.1a scrub ok 2024-09-11T08:15:49.035 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:48 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:15:49.126 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:49.170 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:49.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:49.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:49.272 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:15:49.310 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:49.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:49.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:15:49.412 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:49.441 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:49.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:49.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:49.542 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:49.577 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:49.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:49.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:49.679 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:49.713 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:49.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:49.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:49.806 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:15:49.806 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:15:49.806 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:15:49.806 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:15:49.806 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:15:49.806 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:15:49.806 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:15:49.806 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:15:49.807 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:15:49.807 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:15:49.807 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:15:49.807 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:15:49.807 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:15:49.807 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:15:49.807 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:15:49.807 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:15:49.814 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:15:49.851 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:49.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:49.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:15:49.918 INFO:teuthology.orchestra.run.smithi150.stderr:nodeep-scrub is unset 2024-09-11T08:15:49.953 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:49.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:49.992 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:49.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:50.093 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:50.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:50.136 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:50.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:50.194 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:49 smithi181 ceph-mon[32701]: 3.b scrub starts 2024-09-11T08:15:50.194 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:49 smithi181 ceph-mon[32701]: 3.b scrub ok 2024-09-11T08:15:50.195 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:49 smithi181 ceph-mon[32701]: 8.d scrub starts 2024-09-11T08:15:50.195 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:49 smithi181 ceph-mon[32701]: pgmap v364: 148 pgs: 148 active+clean; 7.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.4 KiB/s rd, 34 MiB/s wr, 1.55k op/s 2024-09-11T08:15:50.195 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:49 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2428447824' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:15:50.195 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:49 smithi181 ceph-mon[32701]: 4.a scrub starts 2024-09-11T08:15:50.195 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:49 smithi181 ceph-mon[32701]: 4.a scrub ok 2024-09-11T08:15:50.195 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:49 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/376370207' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:15:50.195 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:49 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:15:50.236 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:15:50.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:49 smithi150 ceph-mon[29276]: 3.b scrub starts 2024-09-11T08:15:50.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:49 smithi150 ceph-mon[29276]: 3.b scrub ok 2024-09-11T08:15:50.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:49 smithi150 ceph-mon[29276]: 8.d scrub starts 2024-09-11T08:15:50.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:49 smithi150 ceph-mon[29276]: pgmap v364: 148 pgs: 148 active+clean; 7.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.4 KiB/s rd, 34 MiB/s wr, 1.55k op/s 2024-09-11T08:15:50.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:49 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2428447824' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:15:50.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:49 smithi150 ceph-mon[29276]: 4.a scrub starts 2024-09-11T08:15:50.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:49 smithi150 ceph-mon[29276]: 4.a scrub ok 2024-09-11T08:15:50.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:49 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/376370207' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:15:50.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:49 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:15:50.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:49 smithi150 ceph-mon[39612]: 3.b scrub starts 2024-09-11T08:15:50.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:49 smithi150 ceph-mon[39612]: 3.b scrub ok 2024-09-11T08:15:50.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:49 smithi150 ceph-mon[39612]: 8.d scrub starts 2024-09-11T08:15:50.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:49 smithi150 ceph-mon[39612]: pgmap v364: 148 pgs: 148 active+clean; 7.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.4 KiB/s rd, 34 MiB/s wr, 1.55k op/s 2024-09-11T08:15:50.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:49 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2428447824' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:15:50.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:49 smithi150 ceph-mon[39612]: 4.a scrub starts 2024-09-11T08:15:50.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:49 smithi150 ceph-mon[39612]: 4.a scrub ok 2024-09-11T08:15:50.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:49 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/376370207' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:15:50.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:49 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:15:50.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:50.265 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:50.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:15:50.366 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:15:50.402 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:50.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:50.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:15:50.504 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:50.542 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:50.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:50.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:50.643 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:50.679 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:50.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:50.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:50.781 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:15:50.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:50.815 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:50.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:15:50.916 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:15:50.953 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:50.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:50.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:15:51.055 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:51.090 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:51.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:51.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:51.192 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:51.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:51.227 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:51.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:51.229 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:50 smithi150 ceph-mon[29276]: 3.16 scrub starts 2024-09-11T08:15:51.229 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:50 smithi150 ceph-mon[29276]: 3.16 scrub ok 2024-09-11T08:15:51.229 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:50 smithi150 ceph-mon[29276]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:15:51.229 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:50 smithi150 ceph-mon[29276]: Cluster is now healthy 2024-09-11T08:15:51.230 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:50 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:15:51.230 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:50 smithi150 ceph-mon[29276]: osdmap e204: 8 total, 8 up, 8 in 2024-09-11T08:15:51.230 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:50 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:15:51.230 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:50 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:15:51.230 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:50 smithi150 ceph-mon[29276]: osdmap e205: 8 total, 8 up, 8 in 2024-09-11T08:15:51.230 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:50 smithi150 ceph-mon[39612]: 3.16 scrub starts 2024-09-11T08:15:51.230 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:50 smithi150 ceph-mon[39612]: 3.16 scrub ok 2024-09-11T08:15:51.230 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:50 smithi150 ceph-mon[39612]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:15:51.230 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:50 smithi150 ceph-mon[39612]: Cluster is now healthy 2024-09-11T08:15:51.230 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:50 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:15:51.230 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:50 smithi150 ceph-mon[39612]: osdmap e204: 8 total, 8 up, 8 in 2024-09-11T08:15:51.231 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:50 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:15:51.231 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:50 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:15:51.231 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:50 smithi150 ceph-mon[39612]: osdmap e205: 8 total, 8 up, 8 in 2024-09-11T08:15:51.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:50 smithi181 ceph-mon[32701]: 3.16 scrub starts 2024-09-11T08:15:51.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:50 smithi181 ceph-mon[32701]: 3.16 scrub ok 2024-09-11T08:15:51.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:50 smithi181 ceph-mon[32701]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:15:51.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:50 smithi181 ceph-mon[32701]: Cluster is now healthy 2024-09-11T08:15:51.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:50 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:15:51.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:50 smithi181 ceph-mon[32701]: osdmap e204: 8 total, 8 up, 8 in 2024-09-11T08:15:51.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:50 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:15:51.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:50 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:15:51.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:50 smithi181 ceph-mon[32701]: osdmap e205: 8 total, 8 up, 8 in 2024-09-11T08:15:51.328 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:51.361 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:51.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:51.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:51.463 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:51.496 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:51.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:51.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:51.598 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:15:51.636 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:51.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:51.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:15:51.738 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:15:51.777 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:51.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:51.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:15:51.878 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:15:51.914 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:51.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:51.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:15:52.017 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:52.053 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:52.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:52.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:52.155 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:15:52.187 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:52.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:52.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:15:52.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:51 smithi150 ceph-mon[29276]: 5.1b scrub starts 2024-09-11T08:15:52.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:51 smithi150 ceph-mon[29276]: 5.1b scrub ok 2024-09-11T08:15:52.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:51 smithi150 ceph-mon[29276]: pgmap v367: 148 pgs: 148 active+clean; 7.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.04k op/s 2024-09-11T08:15:52.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:51 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:15:52.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:51 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:15:52.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:51 smithi150 ceph-mon[39612]: 5.1b scrub starts 2024-09-11T08:15:52.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:51 smithi150 ceph-mon[39612]: 5.1b scrub ok 2024-09-11T08:15:52.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:51 smithi150 ceph-mon[39612]: pgmap v367: 148 pgs: 148 active+clean; 7.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.04k op/s 2024-09-11T08:15:52.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:51 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:15:52.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:51 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:15:52.288 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:52.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:51 smithi181 ceph-mon[32701]: 5.1b scrub starts 2024-09-11T08:15:52.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:51 smithi181 ceph-mon[32701]: 5.1b scrub ok 2024-09-11T08:15:52.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:51 smithi181 ceph-mon[32701]: pgmap v367: 148 pgs: 148 active+clean; 7.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.04k op/s 2024-09-11T08:15:52.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:51 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:15:52.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:51 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:15:52.328 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:52.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:52.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:52.430 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:15:52.466 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:52.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:52.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:15:52.568 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:52.601 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:52.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:52.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:52.703 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:52.734 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:52.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:52.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:52.836 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:15:52.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:52.869 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:52.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:15:52.970 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:53.005 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:53.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:53.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:53.107 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:15:53.142 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:53.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:53.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:15:53.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:52 smithi150 ceph-mon[29276]: 8.d scrub ok 2024-09-11T08:15:53.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:52 smithi150 ceph-mon[29276]: 5.0 scrub starts 2024-09-11T08:15:53.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:52 smithi150 ceph-mon[29276]: 8.4 scrub starts 2024-09-11T08:15:53.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:52 smithi150 ceph-mon[29276]: 5.0 scrub ok 2024-09-11T08:15:53.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:52 smithi150 ceph-mon[29276]: 3.1 scrub starts 2024-09-11T08:15:53.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:52 smithi150 ceph-mon[29276]: 3.1 scrub ok 2024-09-11T08:15:53.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:52 smithi150 ceph-mon[29276]: 5.18 scrub starts 2024-09-11T08:15:53.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:52 smithi150 ceph-mon[29276]: 5.18 scrub ok 2024-09-11T08:15:53.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:52 smithi150 ceph-mon[39612]: 8.d scrub ok 2024-09-11T08:15:53.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:52 smithi150 ceph-mon[39612]: 5.0 scrub starts 2024-09-11T08:15:53.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:52 smithi150 ceph-mon[39612]: 8.4 scrub starts 2024-09-11T08:15:53.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:52 smithi150 ceph-mon[39612]: 5.0 scrub ok 2024-09-11T08:15:53.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:52 smithi150 ceph-mon[39612]: 3.1 scrub starts 2024-09-11T08:15:53.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:52 smithi150 ceph-mon[39612]: 3.1 scrub ok 2024-09-11T08:15:53.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:52 smithi150 ceph-mon[39612]: 5.18 scrub starts 2024-09-11T08:15:53.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:52 smithi150 ceph-mon[39612]: 5.18 scrub ok 2024-09-11T08:15:53.244 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:15:53.280 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:53.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:53.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:15:53.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:52 smithi181 ceph-mon[32701]: 8.d scrub ok 2024-09-11T08:15:53.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:52 smithi181 ceph-mon[32701]: 5.0 scrub starts 2024-09-11T08:15:53.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:52 smithi181 ceph-mon[32701]: 8.4 scrub starts 2024-09-11T08:15:53.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:52 smithi181 ceph-mon[32701]: 5.0 scrub ok 2024-09-11T08:15:53.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:52 smithi181 ceph-mon[32701]: 3.1 scrub starts 2024-09-11T08:15:53.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:52 smithi181 ceph-mon[32701]: 3.1 scrub ok 2024-09-11T08:15:53.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:52 smithi181 ceph-mon[32701]: 5.18 scrub starts 2024-09-11T08:15:53.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:52 smithi181 ceph-mon[32701]: 5.18 scrub ok 2024-09-11T08:15:53.381 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:15:53.410 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:53.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:53.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:15:53.512 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:53.546 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:53.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:53.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:53.648 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:15:53.677 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:53.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:53.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:15:53.779 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:53.808 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:53.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:53.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:53.910 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:15:53.952 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:53.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:53.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:15:54.054 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:54.072 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T08:15:54.087 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:54.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:54.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:54.189 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:15:54.214 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:54.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:54.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:15:54.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:53 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:15:54.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:53 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:15:54.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:53 smithi150 ceph-mon[39612]: 4.15 scrub starts 2024-09-11T08:15:54.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:53 smithi150 ceph-mon[39612]: 4.15 scrub ok 2024-09-11T08:15:54.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:53 smithi150 ceph-mon[39612]: 4.4 deep-scrub starts 2024-09-11T08:15:54.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:53 smithi150 ceph-mon[39612]: 4.4 deep-scrub ok 2024-09-11T08:15:54.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:53 smithi150 ceph-mon[39612]: pgmap v368: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.20k op/s 2024-09-11T08:15:54.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:53 smithi150 ceph-mon[39612]: osdmap e206: 8 total, 8 up, 8 in 2024-09-11T08:15:54.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:53 smithi150 ceph-mon[39612]: 4.9 scrub starts 2024-09-11T08:15:54.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:53 smithi150 ceph-mon[39612]: 4.9 scrub ok 2024-09-11T08:15:54.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:53 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:15:54.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:53 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:15:54.242 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:53 smithi150 ceph-mon[29276]: 4.15 scrub starts 2024-09-11T08:15:54.242 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:53 smithi150 ceph-mon[29276]: 4.15 scrub ok 2024-09-11T08:15:54.242 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:53 smithi150 ceph-mon[29276]: 4.4 deep-scrub starts 2024-09-11T08:15:54.242 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:53 smithi150 ceph-mon[29276]: 4.4 deep-scrub ok 2024-09-11T08:15:54.242 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:53 smithi150 ceph-mon[29276]: pgmap v368: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.20k op/s 2024-09-11T08:15:54.242 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:53 smithi150 ceph-mon[29276]: osdmap e206: 8 total, 8 up, 8 in 2024-09-11T08:15:54.242 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:53 smithi150 ceph-mon[29276]: 4.9 scrub starts 2024-09-11T08:15:54.242 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:53 smithi150 ceph-mon[29276]: 4.9 scrub ok 2024-09-11T08:15:54.316 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:15:54.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:53 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:15:54.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:53 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:15:54.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:53 smithi181 ceph-mon[32701]: 4.15 scrub starts 2024-09-11T08:15:54.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:53 smithi181 ceph-mon[32701]: 4.15 scrub ok 2024-09-11T08:15:54.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:53 smithi181 ceph-mon[32701]: 4.4 deep-scrub starts 2024-09-11T08:15:54.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:53 smithi181 ceph-mon[32701]: 4.4 deep-scrub ok 2024-09-11T08:15:54.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:53 smithi181 ceph-mon[32701]: pgmap v368: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.20k op/s 2024-09-11T08:15:54.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:53 smithi181 ceph-mon[32701]: osdmap e206: 8 total, 8 up, 8 in 2024-09-11T08:15:54.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:53 smithi181 ceph-mon[32701]: 4.9 scrub starts 2024-09-11T08:15:54.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:53 smithi181 ceph-mon[32701]: 4.9 scrub ok 2024-09-11T08:15:54.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:54.360 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:54.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:15:54.461 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:15:54.488 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:54.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:54.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:15:54.590 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:54.624 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:54.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:54.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:54.627 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T08:15:54.725 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:54.790 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:54.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:54.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:54.891 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:54.945 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:54.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:54.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:55.047 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:55.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:55.083 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:55.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:55.184 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:15:55.224 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:55.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:55.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:15:55.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:54 smithi150 ceph-mon[29276]: 8.4 scrub ok 2024-09-11T08:15:55.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:54 smithi150 ceph-mon[29276]: 6.4 scrub starts 2024-09-11T08:15:55.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:54 smithi150 ceph-mon[29276]: 6.4 scrub ok 2024-09-11T08:15:55.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:54 smithi150 ceph-mon[29276]: 4.1d scrub starts 2024-09-11T08:15:55.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:54 smithi150 ceph-mon[29276]: 4.1d scrub ok 2024-09-11T08:15:55.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:54 smithi150 ceph-mon[29276]: 4.6 scrub starts 2024-09-11T08:15:55.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:54 smithi150 ceph-mon[29276]: 4.6 scrub ok 2024-09-11T08:15:55.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:54 smithi150 ceph-mon[29276]: 6.14 deep-scrub starts 2024-09-11T08:15:55.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:54 smithi150 ceph-mon[29276]: 6.14 deep-scrub ok 2024-09-11T08:15:55.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:54 smithi150 ceph-mon[29276]: 5.2 scrub starts 2024-09-11T08:15:55.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:54 smithi150 ceph-mon[29276]: 5.2 scrub ok 2024-09-11T08:15:55.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:54 smithi150 ceph-mon[39612]: 8.4 scrub ok 2024-09-11T08:15:55.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:54 smithi150 ceph-mon[39612]: 6.4 scrub starts 2024-09-11T08:15:55.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:54 smithi150 ceph-mon[39612]: 6.4 scrub ok 2024-09-11T08:15:55.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:54 smithi150 ceph-mon[39612]: 4.1d scrub starts 2024-09-11T08:15:55.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:54 smithi150 ceph-mon[39612]: 4.1d scrub ok 2024-09-11T08:15:55.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:54 smithi150 ceph-mon[39612]: 4.6 scrub starts 2024-09-11T08:15:55.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:54 smithi150 ceph-mon[39612]: 4.6 scrub ok 2024-09-11T08:15:55.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:54 smithi150 ceph-mon[39612]: 6.14 deep-scrub starts 2024-09-11T08:15:55.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:54 smithi150 ceph-mon[39612]: 6.14 deep-scrub ok 2024-09-11T08:15:55.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:54 smithi150 ceph-mon[39612]: 5.2 scrub starts 2024-09-11T08:15:55.242 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:54 smithi150 ceph-mon[39612]: 5.2 scrub ok 2024-09-11T08:15:55.324 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:15:55 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:15:55] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:15:55.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:54 smithi181 ceph-mon[32701]: 8.4 scrub ok 2024-09-11T08:15:55.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:54 smithi181 ceph-mon[32701]: 6.4 scrub starts 2024-09-11T08:15:55.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:54 smithi181 ceph-mon[32701]: 6.4 scrub ok 2024-09-11T08:15:55.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:54 smithi181 ceph-mon[32701]: 4.1d scrub starts 2024-09-11T08:15:55.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:54 smithi181 ceph-mon[32701]: 4.1d scrub ok 2024-09-11T08:15:55.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:54 smithi181 ceph-mon[32701]: 4.6 scrub starts 2024-09-11T08:15:55.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:54 smithi181 ceph-mon[32701]: 4.6 scrub ok 2024-09-11T08:15:55.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:54 smithi181 ceph-mon[32701]: 6.14 deep-scrub starts 2024-09-11T08:15:55.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:54 smithi181 ceph-mon[32701]: 6.14 deep-scrub ok 2024-09-11T08:15:55.326 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:54 smithi181 ceph-mon[32701]: 5.2 scrub starts 2024-09-11T08:15:55.326 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:54 smithi181 ceph-mon[32701]: 5.2 scrub ok 2024-09-11T08:15:55.326 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:55.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:55.375 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:55.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:55.476 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:55.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:55.511 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:55.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:55.513 INFO:teuthology.orchestra.run.smithi150.stderr:reweighted osd.7 to 1 (10000) 2024-09-11T08:15:55.613 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:15:55.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:55.637 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:55.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:15:55.738 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:55.763 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:55.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:55.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:55.865 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:55.888 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:55.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:55.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:55.912 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-11T08:15:55.912 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 20, 2024-09-11T08:15:55.912 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 600, 2024-09-11T08:15:55.912 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [ 2024-09-11T08:15:55.912 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.912 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:89912 7.6 7:605a3b3f:::benchmark_data_smithi150_78508_object89911:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-11T08:15:55.912 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:10:53.653952+0000", 2024-09-11T08:15:55.912 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 301.994077887, 2024-09-11T08:15:55.912 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.11647966, 2024-09-11T08:15:55.912 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:15:55.912 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:15:55.913 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:15:55.913 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:15:55.913 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:15:55.913 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 89912 2024-09-11T08:15:55.913 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.913 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:15:55.913 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.913 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:15:55.913 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.653952+0000", 2024-09-11T08:15:55.913 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:15:55.913 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.913 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.913 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:15:55.913 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.653952+0000", 2024-09-11T08:15:55.914 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:15:55.914 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.914 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.914 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:15:55.914 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.653952+0000", 2024-09-11T08:15:55.914 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:15:55.914 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.914 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.914 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:15:55.914 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.653974+0000", 2024-09-11T08:15:55.914 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.2126000000000001e-05 2024-09-11T08:15:55.914 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.914 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.915 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:15:55.915 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.653980+0000", 2024-09-11T08:15:55.915 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.832e-06 2024-09-11T08:15:55.915 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.915 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.915 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:15:55.915 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.653985+0000", 2024-09-11T08:15:55.915 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.126e-06 2024-09-11T08:15:55.915 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.915 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.915 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:15:55.915 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.740027+0000", 2024-09-11T08:15:55.915 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.086042775000000002 2024-09-11T08:15:55.915 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.916 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.916 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:15:55.916 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.740148+0000", 2024-09-11T08:15:55.916 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000121044 2024-09-11T08:15:55.916 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.916 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.916 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 3,7", 2024-09-11T08:15:55.916 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.740282+0000", 2024-09-11T08:15:55.916 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00013378899999999999 2024-09-11T08:15:55.916 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.916 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.916 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:55.916 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.746948+0000", 2024-09-11T08:15:55.917 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0066659459999999999 2024-09-11T08:15:55.917 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.917 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.917 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:15:55.917 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.754354+0000", 2024-09-11T08:15:55.917 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0074061650000000001 2024-09-11T08:15:55.917 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.917 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.917 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:55.917 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.770354+0000", 2024-09-11T08:15:55.917 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.015999254000000001 2024-09-11T08:15:55.917 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.917 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.917 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:15:55.918 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.770393+0000", 2024-09-11T08:15:55.918 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.9536999999999998e-05 2024-09-11T08:15:55.918 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.918 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.918 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:15:55.918 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.770432+0000", 2024-09-11T08:15:55.918 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.8593000000000001e-05 2024-09-11T08:15:55.918 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:15:55.918 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:15:55.918 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:15:55.919 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.919 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.919 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:107769 7.4 7:248a60ac:::benchmark_data_smithi150_78508_object107768:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-11T08:15:55.919 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:10.631549+0000", 2024-09-11T08:15:55.919 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 285.01648135099998, 2024-09-11T08:15:55.919 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.124723374, 2024-09-11T08:15:55.919 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:15:55.919 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:15:55.919 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:15:55.919 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:15:55.919 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:15:55.919 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 107769 2024-09-11T08:15:55.919 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.920 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:15:55.920 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.920 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:15:55.920 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.631549+0000", 2024-09-11T08:15:55.920 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:15:55.920 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.920 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.920 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:15:55.920 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.631549+0000", 2024-09-11T08:15:55.920 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:15:55.920 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.920 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.920 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:15:55.920 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.631548+0000", 2024-09-11T08:15:55.921 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:15:55.921 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.921 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.921 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:15:55.921 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.631570+0000", 2024-09-11T08:15:55.921 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.2229e-05 2024-09-11T08:15:55.921 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.921 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.921 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:15:55.921 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.631579+0000", 2024-09-11T08:15:55.921 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.5939999999999994e-06 2024-09-11T08:15:55.921 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.921 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.921 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:15:55.921 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.631584+0000", 2024-09-11T08:15:55.922 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.9760000000000003e-06 2024-09-11T08:15:55.922 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.922 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.922 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:15:55.922 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.641627+0000", 2024-09-11T08:15:55.922 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.010043908000000001 2024-09-11T08:15:55.922 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.922 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.922 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:15:55.922 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.641739+0000", 2024-09-11T08:15:55.922 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000111038 2024-09-11T08:15:55.922 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.922 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.922 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 1,4", 2024-09-11T08:15:55.923 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.641876+0000", 2024-09-11T08:15:55.923 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00013711499999999999 2024-09-11T08:15:55.923 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.923 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.923 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:15:55.923 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.645066+0000", 2024-09-11T08:15:55.923 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0031904339999999998 2024-09-11T08:15:55.923 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.923 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.923 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:55.923 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.646736+0000", 2024-09-11T08:15:55.923 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0016699079999999999 2024-09-11T08:15:55.923 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.923 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.923 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:55.924 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.755409+0000", 2024-09-11T08:15:55.924 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.108673117 2024-09-11T08:15:55.924 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.924 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.924 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:15:55.924 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.756225+0000", 2024-09-11T08:15:55.924 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00081548700000000005 2024-09-11T08:15:55.924 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.924 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.924 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:15:55.924 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.756272+0000", 2024-09-11T08:15:55.924 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.7377999999999999e-05 2024-09-11T08:15:55.924 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:15:55.925 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:15:55.925 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:15:55.925 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.925 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.925 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:107982 7.6 7:665c1af9:::benchmark_data_smithi150_78508_object107981:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-11T08:15:55.925 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:10.912271+0000", 2024-09-11T08:15:55.925 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 284.73575891399997, 2024-09-11T08:15:55.925 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.11809383599999999, 2024-09-11T08:15:55.925 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:15:55.925 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:15:55.925 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:15:55.925 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:15:55.925 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:15:55.926 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 107982 2024-09-11T08:15:55.926 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.926 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:15:55.926 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.926 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:15:55.926 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.912271+0000", 2024-09-11T08:15:55.926 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:15:55.926 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.926 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.926 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:15:55.926 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.912271+0000", 2024-09-11T08:15:55.926 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:15:55.926 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.926 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.927 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:15:55.927 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.912270+0000", 2024-09-11T08:15:55.927 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:15:55.927 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.927 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.927 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:15:55.927 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.912443+0000", 2024-09-11T08:15:55.927 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00017241000000000001 2024-09-11T08:15:55.927 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.927 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.927 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:15:55.927 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.912453+0000", 2024-09-11T08:15:55.927 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0013999999999999e-05 2024-09-11T08:15:55.927 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.928 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.928 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:15:55.928 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.912458+0000", 2024-09-11T08:15:55.928 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.5500000000000002e-06 2024-09-11T08:15:55.928 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.928 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.928 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:15:55.928 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.913046+0000", 2024-09-11T08:15:55.928 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00058744500000000002 2024-09-11T08:15:55.928 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.928 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.928 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:15:55.928 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.913202+0000", 2024-09-11T08:15:55.928 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00015638800000000001 2024-09-11T08:15:55.929 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.929 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.929 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 3,7", 2024-09-11T08:15:55.929 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.913501+0000", 2024-09-11T08:15:55.929 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00029898300000000001 2024-09-11T08:15:55.929 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.929 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.929 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:55.929 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.917896+0000", 2024-09-11T08:15:55.929 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.004394663 2024-09-11T08:15:55.929 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.929 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.929 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:15:55.929 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.923581+0000", 2024-09-11T08:15:55.930 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0056846730000000003 2024-09-11T08:15:55.930 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.930 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.930 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:55.930 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.030272+0000", 2024-09-11T08:15:55.930 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.106691571 2024-09-11T08:15:55.930 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.930 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.930 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:15:55.930 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.030327+0000", 2024-09-11T08:15:55.930 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.4790999999999998e-05 2024-09-11T08:15:55.930 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.930 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.930 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:15:55.931 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.030365+0000", 2024-09-11T08:15:55.931 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.7981000000000001e-05 2024-09-11T08:15:55.931 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:15:55.931 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:15:55.931 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:15:55.931 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.931 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.931 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.14790.0:107980 7.5 e122/91 7:ab79afea:::benchmark_data_smithi150_78508_object107979:head v 122'6786, mlcod=122'6784)", 2024-09-11T08:15:55.931 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:10.913733+0000", 2024-09-11T08:15:55.931 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 284.73429657899999, 2024-09-11T08:15:55.931 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.11537257400000001, 2024-09-11T08:15:55.931 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:15:55.931 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:15:55.931 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:15:55.932 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.932 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:15:55.932 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.913733+0000", 2024-09-11T08:15:55.932 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:15:55.932 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.932 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.932 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:15:55.932 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.913733+0000", 2024-09-11T08:15:55.932 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:15:55.932 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.932 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.932 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:15:55.932 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.913732+0000", 2024-09-11T08:15:55.933 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:15:55.933 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.933 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.933 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:15:55.933 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.913773+0000", 2024-09-11T08:15:55.933 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.0438999999999997e-05 2024-09-11T08:15:55.933 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.933 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.933 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:15:55.933 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.913791+0000", 2024-09-11T08:15:55.933 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.8235000000000001e-05 2024-09-11T08:15:55.933 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.933 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.933 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:15:55.934 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.913797+0000", 2024-09-11T08:15:55.934 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.5110000000000003e-06 2024-09-11T08:15:55.934 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.934 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.934 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:15:55.934 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.913933+0000", 2024-09-11T08:15:55.934 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00013587199999999999 2024-09-11T08:15:55.934 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.934 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.934 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:15:55.934 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.913951+0000", 2024-09-11T08:15:55.934 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.8615e-05 2024-09-11T08:15:55.934 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.934 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.935 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:15:55.935 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.029064+0000", 2024-09-11T08:15:55.935 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.115112703 2024-09-11T08:15:55.935 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.935 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.935 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:15:55.935 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.029106+0000", 2024-09-11T08:15:55.935 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.2070000000000002e-05 2024-09-11T08:15:55.935 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:15:55.935 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:15:55.935 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:15:55.935 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.935 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.936 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.14790.0:107983 7.d e122/91 7:bdc4699e:::benchmark_data_smithi150_78508_object107982:head v 122'6700, mlcod=122'6698)", 2024-09-11T08:15:55.936 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:10.913818+0000", 2024-09-11T08:15:55.936 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 284.73421161200002, 2024-09-11T08:15:55.936 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.115314551, 2024-09-11T08:15:55.936 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:15:55.936 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:15:55.936 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:15:55.936 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.936 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:15:55.936 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.913818+0000", 2024-09-11T08:15:55.936 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:15:55.936 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.936 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.937 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:15:55.937 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.913818+0000", 2024-09-11T08:15:55.937 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:15:55.937 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.937 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.937 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:15:55.937 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.913818+0000", 2024-09-11T08:15:55.937 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:15:55.937 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.937 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.937 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:15:55.937 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.913853+0000", 2024-09-11T08:15:55.937 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.5345999999999997e-05 2024-09-11T08:15:55.937 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.938 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.938 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:15:55.938 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.913860+0000", 2024-09-11T08:15:55.938 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.2759999999999997e-06 2024-09-11T08:15:55.938 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.938 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.938 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:15:55.938 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.913863+0000", 2024-09-11T08:15:55.938 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.5999999999999998e-06 2024-09-11T08:15:55.938 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.938 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.938 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:15:55.938 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.914942+0000", 2024-09-11T08:15:55.938 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0010788320000000001 2024-09-11T08:15:55.938 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.939 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.939 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:15:55.939 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.914960+0000", 2024-09-11T08:15:55.939 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.7561e-05 2024-09-11T08:15:55.939 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.939 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.939 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:15:55.939 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.029110+0000", 2024-09-11T08:15:55.939 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.114150875 2024-09-11T08:15:55.939 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.939 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.939 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:15:55.939 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.029133+0000", 2024-09-11T08:15:55.939 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.2416e-05 2024-09-11T08:15:55.940 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:15:55.940 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:15:55.940 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:15:55.940 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.940 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.940 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:107989 7.6 7:6298c77c:::benchmark_data_smithi150_78508_object107988:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-11T08:15:55.940 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:10.917681+0000", 2024-09-11T08:15:55.940 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 284.73034892599998, 2024-09-11T08:15:55.940 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.119620217, 2024-09-11T08:15:55.940 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:15:55.940 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:15:55.940 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:15:55.940 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:15:55.941 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:15:55.941 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 107989 2024-09-11T08:15:55.941 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.941 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:15:55.941 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.941 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:15:55.941 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.917681+0000", 2024-09-11T08:15:55.941 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:15:55.941 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.941 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.941 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:15:55.941 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.917681+0000", 2024-09-11T08:15:55.941 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:15:55.941 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.942 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.942 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:15:55.942 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.917680+0000", 2024-09-11T08:15:55.942 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:15:55.942 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.942 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.942 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:15:55.942 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.917704+0000", 2024-09-11T08:15:55.942 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.4043e-05 2024-09-11T08:15:55.942 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.942 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.942 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:15:55.942 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.917712+0000", 2024-09-11T08:15:55.942 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.6160000000000002e-06 2024-09-11T08:15:55.943 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.943 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.943 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:15:55.943 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.917717+0000", 2024-09-11T08:15:55.943 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.9019999999999998e-06 2024-09-11T08:15:55.943 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.943 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.943 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:15:55.943 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.919492+0000", 2024-09-11T08:15:55.943 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.001774299 2024-09-11T08:15:55.943 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.943 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.943 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:15:55.943 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.919629+0000", 2024-09-11T08:15:55.944 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00013706600000000001 2024-09-11T08:15:55.944 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.944 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.944 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 3,7", 2024-09-11T08:15:55.944 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.919838+0000", 2024-09-11T08:15:55.944 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00020975799999999999 2024-09-11T08:15:55.944 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.944 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.944 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:55.944 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.923269+0000", 2024-09-11T08:15:55.944 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0034300749999999999 2024-09-11T08:15:55.944 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.944 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.944 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:15:55.945 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.923598+0000", 2024-09-11T08:15:55.945 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00032931300000000002 2024-09-11T08:15:55.945 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.945 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.945 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:55.945 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.037148+0000", 2024-09-11T08:15:55.945 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.113549925 2024-09-11T08:15:55.945 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.945 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.945 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:15:55.945 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.037213+0000", 2024-09-11T08:15:55.945 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.5084000000000002e-05 2024-09-11T08:15:55.945 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.945 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.946 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:15:55.946 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.037301+0000", 2024-09-11T08:15:55.946 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.8386000000000004e-05 2024-09-11T08:15:55.946 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:15:55.946 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:15:55.946 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:15:55.946 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.946 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.946 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:107993 7.9 7:910cd0f0:::benchmark_data_smithi150_78508_object107992:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-11T08:15:55.946 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:10.923155+0000", 2024-09-11T08:15:55.946 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 284.72487470099998, 2024-09-11T08:15:55.946 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.114438673, 2024-09-11T08:15:55.946 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:15:55.946 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:15:55.947 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:15:55.947 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:15:55.947 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:15:55.947 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 107993 2024-09-11T08:15:55.947 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.947 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:15:55.947 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.947 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:15:55.947 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.923155+0000", 2024-09-11T08:15:55.947 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:15:55.947 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.947 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.947 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:15:55.947 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.923155+0000", 2024-09-11T08:15:55.947 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:15:55.948 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.948 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.948 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:15:55.948 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.923154+0000", 2024-09-11T08:15:55.948 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:15:55.948 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.948 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.948 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:15:55.948 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.923206+0000", 2024-09-11T08:15:55.948 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.2302999999999997e-05 2024-09-11T08:15:55.948 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.948 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.948 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:15:55.948 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.923218+0000", 2024-09-11T08:15:55.949 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2225000000000001e-05 2024-09-11T08:15:55.949 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.949 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.949 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:15:55.949 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.923804+0000", 2024-09-11T08:15:55.949 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00058606800000000003 2024-09-11T08:15:55.949 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.949 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.949 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:15:55.949 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.029026+0000", 2024-09-11T08:15:55.949 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.105221881 2024-09-11T08:15:55.949 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.949 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.949 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:15:55.950 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.029186+0000", 2024-09-11T08:15:55.950 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00016024 2024-09-11T08:15:55.950 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.950 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.950 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 1,4", 2024-09-11T08:15:55.950 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.029331+0000", 2024-09-11T08:15:55.950 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00014485600000000001 2024-09-11T08:15:55.950 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.950 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.950 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:15:55.950 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.032466+0000", 2024-09-11T08:15:55.950 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0031345549999999998 2024-09-11T08:15:55.950 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.950 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.950 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:55.951 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.034570+0000", 2024-09-11T08:15:55.951 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0021038189999999998 2024-09-11T08:15:55.951 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.951 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.951 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:55.951 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.037515+0000", 2024-09-11T08:15:55.951 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00294582 2024-09-11T08:15:55.951 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.951 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.951 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:15:55.951 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.037561+0000", 2024-09-11T08:15:55.951 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.5556999999999998e-05 2024-09-11T08:15:55.951 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.951 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.952 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:15:55.952 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.037594+0000", 2024-09-11T08:15:55.952 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.3022000000000003e-05 2024-09-11T08:15:55.952 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:15:55.952 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:15:55.952 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:15:55.952 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.952 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.952 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:108642 7.f 7:f6bc24da:::benchmark_data_smithi150_78508_object108641:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-11T08:15:55.952 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:11.612413+0000", 2024-09-11T08:15:55.952 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 284.03561715799998, 2024-09-11T08:15:55.952 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.119793208, 2024-09-11T08:15:55.952 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:15:55.952 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:15:55.953 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:15:55.953 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:15:55.953 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:15:55.953 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 108642 2024-09-11T08:15:55.953 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.953 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:15:55.953 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.953 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:15:55.953 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.612413+0000", 2024-09-11T08:15:55.953 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:15:55.953 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.953 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.953 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:15:55.953 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.612413+0000", 2024-09-11T08:15:55.954 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:15:55.954 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.954 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.954 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:15:55.954 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.612411+0000", 2024-09-11T08:15:55.954 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:15:55.954 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.954 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.954 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:15:55.954 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.612436+0000", 2024-09-11T08:15:55.954 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.4426999999999998e-05 2024-09-11T08:15:55.954 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.954 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.954 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:15:55.955 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.612448+0000", 2024-09-11T08:15:55.955 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2514000000000001e-05 2024-09-11T08:15:55.955 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.955 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.955 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:15:55.955 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.612454+0000", 2024-09-11T08:15:55.955 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.8719999999999999e-06 2024-09-11T08:15:55.955 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.955 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.955 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:15:55.955 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.612511+0000", 2024-09-11T08:15:55.955 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.7213000000000003e-05 2024-09-11T08:15:55.955 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.955 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.956 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:15:55.956 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.612618+0000", 2024-09-11T08:15:55.956 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00010720799999999999 2024-09-11T08:15:55.956 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.956 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.956 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 3,7", 2024-09-11T08:15:55.956 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.612748+0000", 2024-09-11T08:15:55.956 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012908900000000001 2024-09-11T08:15:55.956 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.956 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.956 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:55.956 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.622005+0000", 2024-09-11T08:15:55.956 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0092576269999999992 2024-09-11T08:15:55.957 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.957 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.957 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:15:55.957 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.622067+0000", 2024-09-11T08:15:55.957 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.2236999999999994e-05 2024-09-11T08:15:55.957 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.957 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.957 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:55.957 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.732129+0000", 2024-09-11T08:15:55.957 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.11006160299999999 2024-09-11T08:15:55.957 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.957 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.957 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:15:55.957 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.732172+0000", 2024-09-11T08:15:55.958 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.3069999999999999e-05 2024-09-11T08:15:55.958 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.958 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.958 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:15:55.958 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.732206+0000", 2024-09-11T08:15:55.958 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.3917000000000001e-05 2024-09-11T08:15:55.958 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:15:55.958 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:15:55.958 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:15:55.958 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.958 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.958 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:108644 7.f 7:f306e571:::benchmark_data_smithi150_78508_object108643:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-11T08:15:55.958 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:11.613339+0000", 2024-09-11T08:15:55.958 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 284.03469103200001, 2024-09-11T08:15:55.959 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.119043579, 2024-09-11T08:15:55.959 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:15:55.959 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:15:55.959 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:15:55.959 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:15:55.959 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:15:55.959 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 108644 2024-09-11T08:15:55.959 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.959 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:15:55.959 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.959 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:15:55.959 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.613339+0000", 2024-09-11T08:15:55.959 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:15:55.959 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.959 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.960 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:15:55.960 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.613339+0000", 2024-09-11T08:15:55.960 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:15:55.960 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.960 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.960 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:15:55.960 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.613338+0000", 2024-09-11T08:15:55.960 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:15:55.960 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.960 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.960 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:15:55.960 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.613368+0000", 2024-09-11T08:15:55.960 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.0035e-05 2024-09-11T08:15:55.960 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.961 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.961 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:15:55.961 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.613378+0000", 2024-09-11T08:15:55.961 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.7799999999999995e-06 2024-09-11T08:15:55.961 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.961 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.961 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:15:55.961 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.613383+0000", 2024-09-11T08:15:55.961 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.1359999999999996e-06 2024-09-11T08:15:55.961 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.961 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.961 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:15:55.961 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.618494+0000", 2024-09-11T08:15:55.961 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0051113740000000001 2024-09-11T08:15:55.961 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.962 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.962 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:15:55.962 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.618602+0000", 2024-09-11T08:15:55.962 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0001075 2024-09-11T08:15:55.962 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.962 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.962 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 3,7", 2024-09-11T08:15:55.962 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.618764+0000", 2024-09-11T08:15:55.962 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00016228600000000001 2024-09-11T08:15:55.962 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.962 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.962 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:55.962 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.622033+0000", 2024-09-11T08:15:55.962 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0032686239999999999 2024-09-11T08:15:55.963 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.963 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.963 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:15:55.963 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.622071+0000", 2024-09-11T08:15:55.963 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.8862000000000002e-05 2024-09-11T08:15:55.963 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.963 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.963 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:55.963 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.732300+0000", 2024-09-11T08:15:55.963 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.110228601 2024-09-11T08:15:55.963 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.963 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.963 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:15:55.963 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.732336+0000", 2024-09-11T08:15:55.964 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.5880000000000002e-05 2024-09-11T08:15:55.964 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.964 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.964 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:15:55.964 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.732382+0000", 2024-09-11T08:15:55.964 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.6601000000000003e-05 2024-09-11T08:15:55.964 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:15:55.964 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:15:55.964 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:15:55.964 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.964 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.964 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:108647 7.6 7:6242fc83:::benchmark_data_smithi150_78508_object108646:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-11T08:15:55.964 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:11.614263+0000", 2024-09-11T08:15:55.964 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 284.03376702000003, 2024-09-11T08:15:55.964 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.117508529, 2024-09-11T08:15:55.965 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:15:55.965 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:15:55.965 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:15:55.965 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:15:55.965 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:15:55.965 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 108647 2024-09-11T08:15:55.965 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.965 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:15:55.965 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.965 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:15:55.965 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.614263+0000", 2024-09-11T08:15:55.965 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:15:55.965 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.965 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.966 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:15:55.966 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.614263+0000", 2024-09-11T08:15:55.966 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:15:55.966 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.966 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.966 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:15:55.967 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.614262+0000", 2024-09-11T08:15:55.967 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:15:55.967 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.967 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.967 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:15:55.967 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.614283+0000", 2024-09-11T08:15:55.967 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.1353999999999999e-05 2024-09-11T08:15:55.967 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.967 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.967 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:15:55.968 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.614292+0000", 2024-09-11T08:15:55.968 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.6240000000000008e-06 2024-09-11T08:15:55.968 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.968 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.968 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:15:55.968 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.614297+0000", 2024-09-11T08:15:55.968 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.5809999999999998e-06 2024-09-11T08:15:55.968 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.968 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.968 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:15:55.968 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.615418+0000", 2024-09-11T08:15:55.968 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0011213110000000001 2024-09-11T08:15:55.968 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.968 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.969 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:15:55.969 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.615534+0000", 2024-09-11T08:15:55.969 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000116434 2024-09-11T08:15:55.969 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.969 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.969 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 3,7", 2024-09-11T08:15:55.969 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.615672+0000", 2024-09-11T08:15:55.969 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00013768600000000001 2024-09-11T08:15:55.969 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.969 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.969 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:55.969 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.619046+0000", 2024-09-11T08:15:55.969 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0033743010000000001 2024-09-11T08:15:55.969 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.970 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.970 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:15:55.970 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.619056+0000", 2024-09-11T08:15:55.970 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.7110000000000007e-06 2024-09-11T08:15:55.970 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.970 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.970 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:55.970 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.731695+0000", 2024-09-11T08:15:55.970 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.112638391 2024-09-11T08:15:55.970 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.970 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.970 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:15:55.970 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.731737+0000", 2024-09-11T08:15:55.970 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.2836000000000001e-05 2024-09-11T08:15:55.971 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.971 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.971 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:15:55.971 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.731771+0000", 2024-09-11T08:15:55.971 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.4078000000000001e-05 2024-09-11T08:15:55.971 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:15:55.971 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:15:55.971 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:15:55.971 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.971 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.971 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:108652 7.f 7:f02b47c9:::benchmark_data_smithi150_78508_object108651:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-11T08:15:55.971 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:11.620633+0000", 2024-09-11T08:15:55.971 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 284.02739646600003, 2024-09-11T08:15:55.971 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.118188819, 2024-09-11T08:15:55.972 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:15:55.972 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:15:55.972 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:15:55.972 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:15:55.972 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:15:55.972 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 108652 2024-09-11T08:15:55.972 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.972 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:15:55.972 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.972 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:15:55.972 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.620633+0000", 2024-09-11T08:15:55.972 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:15:55.972 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.972 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.973 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:15:55.973 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.620633+0000", 2024-09-11T08:15:55.973 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:15:55.973 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.973 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.973 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:15:55.973 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.620632+0000", 2024-09-11T08:15:55.973 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:15:55.973 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.973 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.973 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:15:55.973 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.620653+0000", 2024-09-11T08:15:55.973 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.0843999999999999e-05 2024-09-11T08:15:55.973 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.974 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.974 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:15:55.974 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.620664+0000", 2024-09-11T08:15:55.974 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1139e-05 2024-09-11T08:15:55.974 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.974 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.974 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:15:55.974 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.620670+0000", 2024-09-11T08:15:55.974 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.6829999999999996e-06 2024-09-11T08:15:55.974 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.974 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.974 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:15:55.974 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.622090+0000", 2024-09-11T08:15:55.974 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0014208879999999999 2024-09-11T08:15:55.975 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.975 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.975 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:15:55.975 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.622197+0000", 2024-09-11T08:15:55.975 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000106925 2024-09-11T08:15:55.975 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.975 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.975 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 3,7", 2024-09-11T08:15:55.975 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.622328+0000", 2024-09-11T08:15:55.975 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00013085700000000001 2024-09-11T08:15:55.975 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.975 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.975 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:15:55.975 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.625382+0000", 2024-09-11T08:15:55.976 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0030533460000000002 2024-09-11T08:15:55.976 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.976 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.976 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:55.976 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.625551+0000", 2024-09-11T08:15:55.976 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000169335 2024-09-11T08:15:55.976 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.976 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.976 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:55.976 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.738719+0000", 2024-09-11T08:15:55.976 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.113167978 2024-09-11T08:15:55.976 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.976 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.976 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:15:55.977 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.738772+0000", 2024-09-11T08:15:55.977 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.3436000000000001e-05 2024-09-11T08:15:55.977 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.977 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.977 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:15:55.977 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.738822+0000", 2024-09-11T08:15:55.977 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.9913999999999997e-05 2024-09-11T08:15:55.977 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:15:55.977 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:15:55.977 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:15:55.977 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.977 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.977 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:185889 7.f 7:f20c7729:::benchmark_data_smithi150_78508_object61448:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-11T08:15:55.977 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:12:27.484849+0000", 2024-09-11T08:15:55.977 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 208.163180934, 2024-09-11T08:15:55.978 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.073658571, 2024-09-11T08:15:55.978 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:15:55.978 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-09-11T08:15:55.978 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:15:55.978 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:15:55.978 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:15:55.978 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 185889 2024-09-11T08:15:55.978 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.978 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:15:55.978 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.978 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:15:55.978 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.484849+0000", 2024-09-11T08:15:55.978 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:15:55.979 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.979 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.979 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:15:55.979 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.484849+0000", 2024-09-11T08:15:55.979 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:15:55.979 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.979 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.979 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:15:55.979 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.484847+0000", 2024-09-11T08:15:55.979 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:15:55.979 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.979 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.979 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:15:55.979 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.484855+0000", 2024-09-11T08:15:55.979 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.0190000000000006e-06 2024-09-11T08:15:55.980 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.980 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.980 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:15:55.980 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.484868+0000", 2024-09-11T08:15:55.980 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3104e-05 2024-09-11T08:15:55.980 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.980 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.980 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:15:55.980 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.484875+0000", 2024-09-11T08:15:55.980 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.7970000000000003e-06 2024-09-11T08:15:55.980 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.980 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.980 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:15:55.981 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.484924+0000", 2024-09-11T08:15:55.981 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.8618000000000001e-05 2024-09-11T08:15:55.981 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.981 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.981 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:15:55.981 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.485040+0000", 2024-09-11T08:15:55.981 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000116791 2024-09-11T08:15:55.981 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.981 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.981 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 3,7", 2024-09-11T08:15:55.981 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.485168+0000", 2024-09-11T08:15:55.981 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012775500000000001 2024-09-11T08:15:55.981 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.981 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.981 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:15:55.982 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.519648+0000", 2024-09-11T08:15:55.982 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.034479579000000003 2024-09-11T08:15:55.982 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.982 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.982 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:15:55.982 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.519655+0000", 2024-09-11T08:15:55.982 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.8050000000000001e-06 2024-09-11T08:15:55.982 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.982 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.982 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:15:55.982 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.547352+0000", 2024-09-11T08:15:55.982 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0276973220000001 2024-09-11T08:15:55.982 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.982 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.983 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for missing object", 2024-09-11T08:15:55.983 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.547738+0000", 2024-09-11T08:15:55.983 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00038661099999999998 2024-09-11T08:15:55.983 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.983 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.983 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:15:55.983 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.558442+0000", 2024-09-11T08:15:55.983 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.010703297000000001 2024-09-11T08:15:55.983 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.983 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.983 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:15:55.983 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.558508+0000", 2024-09-11T08:15:55.983 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.5822999999999996e-05 2024-09-11T08:15:55.983 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:15:55.983 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:15:55.984 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:15:55.984 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.984 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.984 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:185891 7.6 7:656e4927:::benchmark_data_smithi150_78508_object61450:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-11T08:15:55.984 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:12:27.487952+0000", 2024-09-11T08:15:55.984 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 208.16007809600001, 2024-09-11T08:15:55.984 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0469824720000001, 2024-09-11T08:15:55.984 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:15:55.984 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-09-11T08:15:55.984 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:15:55.984 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:15:55.984 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:15:55.984 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 185891 2024-09-11T08:15:55.985 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.985 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:15:55.985 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.985 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:15:55.985 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.487952+0000", 2024-09-11T08:15:55.985 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:15:55.985 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.985 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.985 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:15:55.985 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.487952+0000", 2024-09-11T08:15:55.985 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:15:55.985 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.985 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.985 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:15:55.985 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.487950+0000", 2024-09-11T08:15:55.986 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:15:55.986 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.986 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.986 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:15:55.986 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.487958+0000", 2024-09-11T08:15:55.986 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.055e-06 2024-09-11T08:15:55.986 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.986 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.986 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:15:55.986 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.487972+0000", 2024-09-11T08:15:55.986 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3557000000000001e-05 2024-09-11T08:15:55.986 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.986 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.986 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:15:55.986 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.487978+0000", 2024-09-11T08:15:55.987 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.3119999999999999e-06 2024-09-11T08:15:55.987 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.987 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.987 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:15:55.987 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.494882+0000", 2024-09-11T08:15:55.987 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.006903508 2024-09-11T08:15:55.987 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.987 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.987 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:15:55.987 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.495008+0000", 2024-09-11T08:15:55.987 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012622999999999999 2024-09-11T08:15:55.987 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.987 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.987 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 3,7", 2024-09-11T08:15:55.988 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.495118+0000", 2024-09-11T08:15:55.988 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000109858 2024-09-11T08:15:55.988 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.988 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.988 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:15:55.988 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.503221+0000", 2024-09-11T08:15:55.988 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0081029919999999998 2024-09-11T08:15:55.988 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.988 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.988 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:15:55.988 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.518414+0000", 2024-09-11T08:15:55.988 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.015193442999999999 2024-09-11T08:15:55.988 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.988 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.989 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:15:55.989 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.518421+0000", 2024-09-11T08:15:55.989 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.0829999999999998e-06 2024-09-11T08:15:55.989 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.989 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.989 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:15:55.989 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.534871+0000", 2024-09-11T08:15:55.989 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0164496430000001 2024-09-11T08:15:55.989 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.989 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.989 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:15:55.989 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.534934+0000", 2024-09-11T08:15:55.989 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.3515000000000004e-05 2024-09-11T08:15:55.989 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:15:55.990 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:15:55.990 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:15:55.990 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.990 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.990 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:185898 7.6 7:6ed37a8e:::benchmark_data_smithi150_78508_object61457:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-09-11T08:15:55.990 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:12:27.504908+0000", 2024-09-11T08:15:55.990 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 208.143121597, 2024-09-11T08:15:55.990 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0425820509999999, 2024-09-11T08:15:55.990 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:15:55.990 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:15:55.990 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:15:55.990 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:15:55.990 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:15:55.990 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 185898 2024-09-11T08:15:55.991 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.991 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:15:55.991 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.991 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:15:55.991 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.504908+0000", 2024-09-11T08:15:55.991 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:15:55.991 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.991 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.991 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:15:55.991 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.504908+0000", 2024-09-11T08:15:55.991 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:15:55.991 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.991 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.992 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:15:55.992 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.504906+0000", 2024-09-11T08:15:55.992 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:15:55.992 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.992 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.992 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:15:55.992 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.504914+0000", 2024-09-11T08:15:55.992 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.3869999999999997e-06 2024-09-11T08:15:55.992 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.992 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.992 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:15:55.992 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.507628+0000", 2024-09-11T08:15:55.992 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.002714396 2024-09-11T08:15:55.992 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.993 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.993 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:15:55.993 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.507641+0000", 2024-09-11T08:15:55.993 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2884000000000001e-05 2024-09-11T08:15:55.993 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.993 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.993 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:15:55.993 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.519272+0000", 2024-09-11T08:15:55.993 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.01163114 2024-09-11T08:15:55.993 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.993 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.993 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:15:55.993 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.519278+0000", 2024-09-11T08:15:55.994 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.6219999999999997e-06 2024-09-11T08:15:55.994 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.994 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.994 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:15:55.994 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.534959+0000", 2024-09-11T08:15:55.994 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.015681418 2024-09-11T08:15:55.994 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.994 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.994 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:15:55.994 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.535054+0000", 2024-09-11T08:15:55.994 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.5048000000000002e-05 2024-09-11T08:15:55.994 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.994 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.994 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 3", 2024-09-11T08:15:55.995 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.535175+0000", 2024-09-11T08:15:55.995 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000120974 2024-09-11T08:15:55.995 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.995 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.995 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:55.995 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.543859+0000", 2024-09-11T08:15:55.995 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0086840479999999998 2024-09-11T08:15:55.995 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.995 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.995 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:15:55.995 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.547313+0000", 2024-09-11T08:15:55.995 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0034536699999999998 2024-09-11T08:15:55.995 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.996 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.996 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:15:55.996 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.547411+0000", 2024-09-11T08:15:55.996 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.8193e-05 2024-09-11T08:15:55.996 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.996 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.996 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:15:55.996 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.547490+0000", 2024-09-11T08:15:55.996 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.9239999999999993e-05 2024-09-11T08:15:55.996 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:15:55.996 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:15:55.996 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:15:55.996 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.997 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.997 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:185899 7.f 7:f7fb7d91:::benchmark_data_smithi150_78508_object61458:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-09-11T08:15:55.997 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:12:27.508749+0000", 2024-09-11T08:15:55.997 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 208.13928073700001, 2024-09-11T08:15:55.997 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0498902720000001, 2024-09-11T08:15:55.997 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:15:55.997 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:15:55.997 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:15:55.997 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:15:55.997 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:15:55.997 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 185899 2024-09-11T08:15:55.997 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.997 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:15:55.997 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.998 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:15:55.998 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.508749+0000", 2024-09-11T08:15:55.998 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:15:55.998 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.998 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.998 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:15:55.998 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.508749+0000", 2024-09-11T08:15:55.998 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:15:55.998 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.998 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.998 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:15:55.998 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.508748+0000", 2024-09-11T08:15:55.998 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:15:55.998 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.999 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.999 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:15:55.999 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.508754+0000", 2024-09-11T08:15:55.999 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.5640000000000002e-06 2024-09-11T08:15:55.999 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.999 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.999 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:15:55.999 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.508766+0000", 2024-09-11T08:15:55.999 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1279e-05 2024-09-11T08:15:55.999 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:55.999 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:55.999 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:15:55.999 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.508771+0000", 2024-09-11T08:15:55.999 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.7130000000000002e-06 2024-09-11T08:15:56.000 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.000 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.000 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:15:56.000 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.520925+0000", 2024-09-11T08:15:56.000 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.012153746999999999 2024-09-11T08:15:56.000 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.000 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.000 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:15:56.000 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.520931+0000", 2024-09-11T08:15:56.000 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.2979999999999997e-06 2024-09-11T08:15:56.000 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.000 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.000 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:15:56.000 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.547767+0000", 2024-09-11T08:15:56.001 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.026835511 2024-09-11T08:15:56.001 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.001 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.001 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:15:56.001 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.548837+0000", 2024-09-11T08:15:56.001 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00106989 2024-09-11T08:15:56.001 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.001 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.001 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 3", 2024-09-11T08:15:56.001 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.548952+0000", 2024-09-11T08:15:56.001 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000114957 2024-09-11T08:15:56.001 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.001 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.002 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:15:56.002 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.558040+0000", 2024-09-11T08:15:56.002 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0090886339999999999 2024-09-11T08:15:56.002 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.002 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.002 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:56.002 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.558535+0000", 2024-09-11T08:15:56.002 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00049422300000000001 2024-09-11T08:15:56.002 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.002 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.002 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:15:56.002 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.558568+0000", 2024-09-11T08:15:56.002 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.3077000000000002e-05 2024-09-11T08:15:56.003 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.003 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.003 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:15:56.003 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.558639+0000", 2024-09-11T08:15:56.003 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.1802000000000001e-05 2024-09-11T08:15:56.003 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:15:56.003 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:15:56.003 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:15:56.003 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.003 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.003 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:185912 7.f 7:f4b5a1fc:::benchmark_data_smithi150_78508_object61471:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-09-11T08:15:56.003 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:12:27.527574+0000", 2024-09-11T08:15:56.003 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 208.12045560199999, 2024-09-11T08:15:56.003 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.041402859, 2024-09-11T08:15:56.004 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:15:56.004 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:15:56.004 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:15:56.004 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:15:56.004 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:15:56.004 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 185912 2024-09-11T08:15:56.004 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.004 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:15:56.004 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.004 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:15:56.004 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.527574+0000", 2024-09-11T08:15:56.004 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:15:56.004 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.005 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.005 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:15:56.005 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.527574+0000", 2024-09-11T08:15:56.005 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:15:56.005 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.005 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.005 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:15:56.005 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.527572+0000", 2024-09-11T08:15:56.005 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999976 2024-09-11T08:15:56.005 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.005 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.005 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:15:56.005 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.527581+0000", 2024-09-11T08:15:56.005 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.8580000000000006e-06 2024-09-11T08:15:56.005 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.006 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.006 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:15:56.006 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.527599+0000", 2024-09-11T08:15:56.006 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.8474e-05 2024-09-11T08:15:56.006 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.006 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.006 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:15:56.006 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.527606+0000", 2024-09-11T08:15:56.006 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.8140000000000003e-06 2024-09-11T08:15:56.006 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.006 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.006 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:15:56.006 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.527658+0000", 2024-09-11T08:15:56.006 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.1499000000000003e-05 2024-09-11T08:15:56.007 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.007 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.007 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:15:56.007 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.527667+0000", 2024-09-11T08:15:56.007 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.8779999999999993e-06 2024-09-11T08:15:56.007 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.007 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.007 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:15:56.007 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.551879+0000", 2024-09-11T08:15:56.007 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0242118979999999 2024-09-11T08:15:56.007 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.007 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.008 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:15:56.008 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.551981+0000", 2024-09-11T08:15:56.008 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000101856 2024-09-11T08:15:56.008 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.008 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.008 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 3", 2024-09-11T08:15:56.008 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.552086+0000", 2024-09-11T08:15:56.008 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00010490200000000001 2024-09-11T08:15:56.008 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.008 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.008 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:15:56.008 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.561353+0000", 2024-09-11T08:15:56.008 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0092671100000000003 2024-09-11T08:15:56.008 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.009 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.009 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:56.009 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.568687+0000", 2024-09-11T08:15:56.009 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0073335079999999999 2024-09-11T08:15:56.009 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.009 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.009 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:15:56.009 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.568880+0000", 2024-09-11T08:15:56.009 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00019370499999999999 2024-09-11T08:15:56.009 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.009 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.009 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:15:56.009 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.568977+0000", 2024-09-11T08:15:56.010 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.6769999999999997e-05 2024-09-11T08:15:56.010 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:15:56.010 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:15:56.010 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:15:56.010 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.010 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.010 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:185922 7.f 7:f82229b7:::benchmark_data_smithi150_78508_object61481:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-09-11T08:15:56.010 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:12:27.544971+0000", 2024-09-11T08:15:56.010 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 208.10305857500001, 2024-09-11T08:15:56.010 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0241932540000001, 2024-09-11T08:15:56.010 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:15:56.010 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:15:56.010 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:15:56.010 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:15:56.011 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:15:56.011 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 185922 2024-09-11T08:15:56.011 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.011 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:15:56.011 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.011 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:15:56.011 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.544971+0000", 2024-09-11T08:15:56.011 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:15:56.011 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.011 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.011 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:15:56.011 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.544971+0000", 2024-09-11T08:15:56.011 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:15:56.011 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.012 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.012 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:15:56.012 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.544970+0000", 2024-09-11T08:15:56.012 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:15:56.012 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.012 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.012 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:15:56.012 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.544977+0000", 2024-09-11T08:15:56.012 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.6710000000000002e-06 2024-09-11T08:15:56.012 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.012 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.012 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:15:56.012 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.544990+0000", 2024-09-11T08:15:56.013 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3091000000000001e-05 2024-09-11T08:15:56.013 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.013 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.013 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:15:56.013 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.544996+0000", 2024-09-11T08:15:56.013 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.891e-06 2024-09-11T08:15:56.013 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.013 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.013 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:15:56.013 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.553262+0000", 2024-09-11T08:15:56.013 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.008266252 2024-09-11T08:15:56.013 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.013 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.013 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:15:56.013 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.553269+0000", 2024-09-11T08:15:56.014 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.4449999999999999e-06 2024-09-11T08:15:56.014 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.014 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.014 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:15:56.014 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.554883+0000", 2024-09-11T08:15:56.014 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0016135500000001 2024-09-11T08:15:56.014 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.014 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.014 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:15:56.014 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.555000+0000", 2024-09-11T08:15:56.014 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000117419 2024-09-11T08:15:56.014 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.014 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.014 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 3", 2024-09-11T08:15:56.015 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.555128+0000", 2024-09-11T08:15:56.015 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000127615 2024-09-11T08:15:56.015 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.015 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.015 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:15:56.015 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.561359+0000", 2024-09-11T08:15:56.015 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0062312979999999997 2024-09-11T08:15:56.015 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.015 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.015 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:56.015 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.569013+0000", 2024-09-11T08:15:56.015 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0076535800000000001 2024-09-11T08:15:56.015 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.015 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.015 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:15:56.016 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.569087+0000", 2024-09-11T08:15:56.016 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.3750000000000004e-05 2024-09-11T08:15:56.016 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.016 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.016 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:15:56.016 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.569165+0000", 2024-09-11T08:15:56.016 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.8104999999999998e-05 2024-09-11T08:15:56.016 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:15:56.016 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:15:56.016 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:15:56.016 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.016 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.016 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:185926 7.6 7:6f05efb5:::benchmark_data_smithi150_78508_object61485:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-09-11T08:15:56.016 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:12:27.555807+0000", 2024-09-11T08:15:56.017 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 208.09222316500001, 2024-09-11T08:15:56.017 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.99179493699999999, 2024-09-11T08:15:56.017 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:15:56.017 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:15:56.017 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:15:56.017 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:15:56.017 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:15:56.017 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 185926 2024-09-11T08:15:56.017 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.017 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:15:56.017 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.017 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:15:56.017 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.555807+0000", 2024-09-11T08:15:56.017 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:15:56.018 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.018 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.018 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:15:56.018 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.555807+0000", 2024-09-11T08:15:56.018 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:15:56.018 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.018 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.018 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:15:56.018 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.555805+0000", 2024-09-11T08:15:56.018 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:15:56.018 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.018 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.018 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:15:56.018 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.555812+0000", 2024-09-11T08:15:56.019 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.4489999999999996e-06 2024-09-11T08:15:56.019 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.019 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.019 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:15:56.019 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.555827+0000", 2024-09-11T08:15:56.019 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4789e-05 2024-09-11T08:15:56.019 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.019 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.019 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:15:56.019 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.555832+0000", 2024-09-11T08:15:56.019 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.7209999999999999e-06 2024-09-11T08:15:56.019 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.019 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.019 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:15:56.020 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.555876+0000", 2024-09-11T08:15:56.020 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.3909000000000003e-05 2024-09-11T08:15:56.020 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.020 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.020 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:15:56.020 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.555884+0000", 2024-09-11T08:15:56.020 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.8099999999999998e-06 2024-09-11T08:15:56.020 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.020 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.020 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:15:56.020 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.538336+0000", 2024-09-11T08:15:56.020 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.98245216199999996 2024-09-11T08:15:56.020 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.020 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.021 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:15:56.021 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.538435+0000", 2024-09-11T08:15:56.021 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.8202000000000006e-05 2024-09-11T08:15:56.021 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.021 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.021 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 3", 2024-09-11T08:15:56.021 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.538546+0000", 2024-09-11T08:15:56.021 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000111648 2024-09-11T08:15:56.021 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.021 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.021 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:56.021 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.544018+0000", 2024-09-11T08:15:56.021 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0054713460000000002 2024-09-11T08:15:56.021 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.022 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.022 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:15:56.022 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.547499+0000", 2024-09-11T08:15:56.022 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0034812760000000002 2024-09-11T08:15:56.022 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.022 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.022 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:15:56.022 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.547535+0000", 2024-09-11T08:15:56.022 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.6482000000000001e-05 2024-09-11T08:15:56.022 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.022 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.022 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:15:56.022 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.547602+0000", 2024-09-11T08:15:56.022 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.6418e-05 2024-09-11T08:15:56.023 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:15:56.023 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:15:56.023 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:15:56.023 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.023 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.023 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15087.0:54336 8.e 8:791483ab:::benchmark_data_smithi150_107646_object54335:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-11T08:15:56.023 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:14:39.758178+0000", 2024-09-11T08:15:56.023 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 75.889851848999996, 2024-09-11T08:15:56.023 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.11803913000000001, 2024-09-11T08:15:56.023 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:15:56.023 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:15:56.023 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:15:56.023 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15087", 2024-09-11T08:15:56.024 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/3299560913", 2024-09-11T08:15:56.024 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 54336 2024-09-11T08:15:56.024 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.024 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:15:56.024 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.024 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:15:56.024 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:14:39.758178+0000", 2024-09-11T08:15:56.024 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:15:56.024 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.024 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.024 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:15:56.024 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:14:39.758178+0000", 2024-09-11T08:15:56.024 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:15:56.024 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.025 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.025 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:15:56.025 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:14:39.758177+0000", 2024-09-11T08:15:56.025 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:15:56.025 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.025 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.025 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:15:56.025 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:14:39.758199+0000", 2024-09-11T08:15:56.025 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.1393e-05 2024-09-11T08:15:56.025 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.025 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.025 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:15:56.025 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:14:39.758209+0000", 2024-09-11T08:15:56.026 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0148e-05 2024-09-11T08:15:56.026 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.026 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.026 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:15:56.026 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:14:39.758214+0000", 2024-09-11T08:15:56.026 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.0200000000000002e-06 2024-09-11T08:15:56.026 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.026 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.026 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:15:56.026 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:14:39.776714+0000", 2024-09-11T08:15:56.026 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.018500256999999999 2024-09-11T08:15:56.026 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.026 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.026 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:15:56.027 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:14:39.776859+0000", 2024-09-11T08:15:56.027 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00014440500000000001 2024-09-11T08:15:56.027 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.027 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.027 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 2,4", 2024-09-11T08:15:56.027 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:14:39.776986+0000", 2024-09-11T08:15:56.027 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012781899999999999 2024-09-11T08:15:56.027 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.027 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.027 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:15:56.027 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:14:39.783240+0000", 2024-09-11T08:15:56.027 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0062539290000000001 2024-09-11T08:15:56.027 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.027 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.028 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:56.028 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:14:39.787109+0000", 2024-09-11T08:15:56.028 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0038682389999999999 2024-09-11T08:15:56.028 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.028 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.028 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:15:56.028 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:14:39.876056+0000", 2024-09-11T08:15:56.028 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.088947717999999995 2024-09-11T08:15:56.028 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.028 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.028 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:15:56.028 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:14:39.876170+0000", 2024-09-11T08:15:56.028 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00011329500000000001 2024-09-11T08:15:56.029 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.029 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.029 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:15:56.029 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:14:39.876217+0000", 2024-09-11T08:15:56.029 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.7658999999999999e-05 2024-09-11T08:15:56.029 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:15:56.029 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:15:56.029 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:15:56.029 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.029 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.029 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.24320.0:1237 4.4 4:22d26bf9:::data_loggenerations_metadata:head [watch reconnect cookie 94783556145488 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-11T08:15:56.029 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:15:08.641855+0000", 2024-09-11T08:15:56.029 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 47.006175337999998, 2024-09-11T08:15:56.029 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.158061334, 2024-09-11T08:15:56.030 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:15:56.030 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "started", 2024-09-11T08:15:56.030 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:15:56.030 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.24320", 2024-09-11T08:15:56.030 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1503848706", 2024-09-11T08:15:56.030 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 1237 2024-09-11T08:15:56.030 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.030 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:15:56.030 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.030 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:15:56.030 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:08.641855+0000", 2024-09-11T08:15:56.030 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:15:56.030 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.030 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.031 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:15:56.031 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:08.641855+0000", 2024-09-11T08:15:56.031 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:15:56.031 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.031 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.031 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:15:56.031 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:08.641853+0000", 2024-09-11T08:15:56.031 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:15:56.031 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.031 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.031 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:15:56.031 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:08.641859+0000", 2024-09-11T08:15:56.031 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.7930000000000003e-06 2024-09-11T08:15:56.031 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.032 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.032 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:15:56.032 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:08.641870+0000", 2024-09-11T08:15:56.032 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.049e-05 2024-09-11T08:15:56.032 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.032 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.032 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:15:56.032 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:08.641876+0000", 2024-09-11T08:15:56.032 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.088e-06 2024-09-11T08:15:56.032 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.032 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.032 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:15:56.032 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:08.642231+0000", 2024-09-11T08:15:56.032 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00035531199999999999 2024-09-11T08:15:56.033 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.033 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.033 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:15:56.033 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:08.642237+0000", 2024-09-11T08:15:56.033 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.5579999999999997e-06 2024-09-11T08:15:56.033 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.033 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.033 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:15:56.033 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:08.775568+0000", 2024-09-11T08:15:56.033 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.133330278 2024-09-11T08:15:56.033 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.033 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.033 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for missing object", 2024-09-11T08:15:56.033 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:08.775731+0000", 2024-09-11T08:15:56.033 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00016296799999999999 2024-09-11T08:15:56.034 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.034 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.034 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:15:56.034 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:08.780713+0000", 2024-09-11T08:15:56.034 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0049824730000000003 2024-09-11T08:15:56.034 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.034 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.034 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for rw locks", 2024-09-11T08:15:56.034 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:08.780877+0000", 2024-09-11T08:15:56.034 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000163475 2024-09-11T08:15:56.034 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.034 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.034 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:15:56.034 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:08.793305+0000", 2024-09-11T08:15:56.034 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.012427858999999999 2024-09-11T08:15:56.035 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.035 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.035 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:15:56.035 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:08.793381+0000", 2024-09-11T08:15:56.035 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.6229999999999994e-05 2024-09-11T08:15:56.035 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:15:56.035 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:15:56.035 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:15:56.035 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:08.799916+0000", 2024-09-11T08:15:56.035 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0065351580000000001 2024-09-11T08:15:56.035 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:15:56.035 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:15:56.035 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:15:56.036 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:15:56.036 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:15:56.036 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-11T08:15:56.036 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:15:56.057 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:56.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:56.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:15:56.159 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:56.183 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:56.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:56.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:56.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:55 smithi150 ceph-mon[39612]: 8.8 scrub starts 2024-09-11T08:15:56.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:55 smithi150 ceph-mon[39612]: 3.4 scrub starts 2024-09-11T08:15:56.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:55 smithi150 ceph-mon[39612]: 3.4 scrub ok 2024-09-11T08:15:56.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:55 smithi150 ceph-mon[39612]: 3.6 scrub starts 2024-09-11T08:15:56.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:55 smithi150 ceph-mon[39612]: 3.6 scrub ok 2024-09-11T08:15:56.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:55 smithi150 ceph-mon[39612]: pgmap v370: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.11k op/s 2024-09-11T08:15:56.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:55 smithi150 ceph-mon[39612]: 5.4 scrub starts 2024-09-11T08:15:56.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:55 smithi150 ceph-mon[39612]: 5.4 scrub ok 2024-09-11T08:15:56.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:55 smithi150 ceph-mon[39612]: 4.1f scrub starts 2024-09-11T08:15:56.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:55 smithi150 ceph-mon[39612]: 4.1f scrub ok 2024-09-11T08:15:56.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:55 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1430067758' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-11T08:15:56.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:55 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-11T08:15:56.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:55 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-11T08:15:56.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:55 smithi150 ceph-mon[39612]: osdmap e207: 8 total, 8 up, 8 in 2024-09-11T08:15:56.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:55 smithi150 ceph-mon[29276]: 8.8 scrub starts 2024-09-11T08:15:56.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:55 smithi150 ceph-mon[29276]: 3.4 scrub starts 2024-09-11T08:15:56.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:55 smithi150 ceph-mon[29276]: 3.4 scrub ok 2024-09-11T08:15:56.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:55 smithi150 ceph-mon[29276]: 3.6 scrub starts 2024-09-11T08:15:56.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:55 smithi150 ceph-mon[29276]: 3.6 scrub ok 2024-09-11T08:15:56.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:55 smithi150 ceph-mon[29276]: pgmap v370: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.11k op/s 2024-09-11T08:15:56.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:55 smithi150 ceph-mon[29276]: 5.4 scrub starts 2024-09-11T08:15:56.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:55 smithi150 ceph-mon[29276]: 5.4 scrub ok 2024-09-11T08:15:56.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:55 smithi150 ceph-mon[29276]: 4.1f scrub starts 2024-09-11T08:15:56.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:55 smithi150 ceph-mon[29276]: 4.1f scrub ok 2024-09-11T08:15:56.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:55 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1430067758' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-11T08:15:56.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:55 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-11T08:15:56.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:55 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-11T08:15:56.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:55 smithi150 ceph-mon[29276]: osdmap e207: 8 total, 8 up, 8 in 2024-09-11T08:15:56.284 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:56.307 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:56.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:56.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:56.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:55 smithi181 ceph-mon[32701]: 8.8 scrub starts 2024-09-11T08:15:56.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:55 smithi181 ceph-mon[32701]: 3.4 scrub starts 2024-09-11T08:15:56.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:55 smithi181 ceph-mon[32701]: 3.4 scrub ok 2024-09-11T08:15:56.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:55 smithi181 ceph-mon[32701]: 3.6 scrub starts 2024-09-11T08:15:56.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:55 smithi181 ceph-mon[32701]: 3.6 scrub ok 2024-09-11T08:15:56.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:55 smithi181 ceph-mon[32701]: pgmap v370: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.11k op/s 2024-09-11T08:15:56.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:55 smithi181 ceph-mon[32701]: 5.4 scrub starts 2024-09-11T08:15:56.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:55 smithi181 ceph-mon[32701]: 5.4 scrub ok 2024-09-11T08:15:56.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:55 smithi181 ceph-mon[32701]: 4.1f scrub starts 2024-09-11T08:15:56.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:55 smithi181 ceph-mon[32701]: 4.1f scrub ok 2024-09-11T08:15:56.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:55 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1430067758' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-11T08:15:56.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:55 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-11T08:15:56.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:55 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-11T08:15:56.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:55 smithi181 ceph-mon[32701]: osdmap e207: 8 total, 8 up, 8 in 2024-09-11T08:15:56.409 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:15:56.430 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:56.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:56.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:15:56.531 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:15:56.559 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:56.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:56.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:15:56.660 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:56.682 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:56.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:56.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:56.783 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:56.807 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:56.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:56.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:56.909 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:56.932 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:56.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:56.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:57.033 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:57.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:57.057 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:57.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:57.157 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:57.161 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-11T08:15:57.179 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:57.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:57.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:57.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:56 smithi150 ceph-mon[29276]: 6.19 scrub starts 2024-09-11T08:15:57.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:56 smithi150 ceph-mon[29276]: 6.19 scrub ok 2024-09-11T08:15:57.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:56 smithi150 ceph-mon[29276]: 8.7 scrub starts 2024-09-11T08:15:57.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:56 smithi150 ceph-mon[29276]: 8.8 scrub ok 2024-09-11T08:15:57.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:56 smithi150 ceph-mon[29276]: osdmap e208: 8 total, 8 up, 8 in 2024-09-11T08:15:57.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:56 smithi150 ceph-mon[39612]: 6.19 scrub starts 2024-09-11T08:15:57.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:56 smithi150 ceph-mon[39612]: 6.19 scrub ok 2024-09-11T08:15:57.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:56 smithi150 ceph-mon[39612]: 8.7 scrub starts 2024-09-11T08:15:57.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:56 smithi150 ceph-mon[39612]: 8.8 scrub ok 2024-09-11T08:15:57.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:56 smithi150 ceph-mon[39612]: osdmap e208: 8 total, 8 up, 8 in 2024-09-11T08:15:57.281 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:57.302 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:57.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:57.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:57.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:56 smithi181 ceph-mon[32701]: 6.19 scrub starts 2024-09-11T08:15:57.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:56 smithi181 ceph-mon[32701]: 6.19 scrub ok 2024-09-11T08:15:57.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:56 smithi181 ceph-mon[32701]: 8.7 scrub starts 2024-09-11T08:15:57.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:56 smithi181 ceph-mon[32701]: 8.8 scrub ok 2024-09-11T08:15:57.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:56 smithi181 ceph-mon[32701]: osdmap e208: 8 total, 8 up, 8 in 2024-09-11T08:15:57.404 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:15:57.426 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:15:57.431 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:57.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:57.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:15:57.532 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:15:57.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:57.555 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:57.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:15:57.655 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:57.698 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:57.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:57.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:57.799 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:57.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:57.835 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:57.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:57.936 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:15:57.967 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:57.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:57.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:15:57.987 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-11T08:15:57.988 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T08:15:58.008 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:15:58.053 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:15:58.069 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:58.098 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:58.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:58.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:58.200 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:58.206 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:57 smithi150 ceph-mon[29276]: 8.7 scrub ok 2024-09-11T08:15:58.207 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:57 smithi150 ceph-mon[29276]: 8.1 scrub starts 2024-09-11T08:15:58.207 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:57 smithi150 ceph-mon[29276]: 6.11 scrub starts 2024-09-11T08:15:58.207 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:57 smithi150 ceph-mon[29276]: 6.11 scrub ok 2024-09-11T08:15:58.207 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:57 smithi150 ceph-mon[29276]: pgmap v373: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.11k op/s 2024-09-11T08:15:58.207 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:57 smithi150 ceph-mon[29276]: 8.1 scrub ok 2024-09-11T08:15:58.207 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:57 smithi150 ceph-mon[29276]: osdmap e209: 8 total, 8 up, 8 in 2024-09-11T08:15:58.207 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:57 smithi150 ceph-mon[39612]: 8.7 scrub ok 2024-09-11T08:15:58.207 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:57 smithi150 ceph-mon[39612]: 8.1 scrub starts 2024-09-11T08:15:58.207 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:57 smithi150 ceph-mon[39612]: 6.11 scrub starts 2024-09-11T08:15:58.208 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:57 smithi150 ceph-mon[39612]: 6.11 scrub ok 2024-09-11T08:15:58.208 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:57 smithi150 ceph-mon[39612]: pgmap v373: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.11k op/s 2024-09-11T08:15:58.208 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:57 smithi150 ceph-mon[39612]: 8.1 scrub ok 2024-09-11T08:15:58.208 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:57 smithi150 ceph-mon[39612]: osdmap e209: 8 total, 8 up, 8 in 2024-09-11T08:15:58.231 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:58.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:58.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:58.323 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:15:58 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:15:58.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:57 smithi181 ceph-mon[32701]: 8.7 scrub ok 2024-09-11T08:15:58.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:57 smithi181 ceph-mon[32701]: 8.1 scrub starts 2024-09-11T08:15:58.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:57 smithi181 ceph-mon[32701]: 6.11 scrub starts 2024-09-11T08:15:58.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:57 smithi181 ceph-mon[32701]: 6.11 scrub ok 2024-09-11T08:15:58.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:57 smithi181 ceph-mon[32701]: pgmap v373: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.11k op/s 2024-09-11T08:15:58.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:57 smithi181 ceph-mon[32701]: 8.1 scrub ok 2024-09-11T08:15:58.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:57 smithi181 ceph-mon[32701]: osdmap e209: 8 total, 8 up, 8 in 2024-09-11T08:15:58.332 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:58.366 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:58.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:58.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:58.468 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:58.526 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:58.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:58.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:58.628 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:15:58.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:58.671 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:58.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:15:58.772 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:58.808 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:58.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:58.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:58.909 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:15:58.945 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:58.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:58.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:15:58.963 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:15:58 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:15:58] "GET /metrics HTTP/1.1" 200 224162 "" "Prometheus/2.43.0" 2024-09-11T08:15:59.047 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:15:59.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:59.088 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:59.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:15:59.189 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:59.221 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:59.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:59.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:59.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:58 smithi150 ceph-mon[39612]: 4.19 deep-scrub starts 2024-09-11T08:15:59.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:58 smithi150 ceph-mon[39612]: 4.19 deep-scrub ok 2024-09-11T08:15:59.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:58 smithi150 ceph-mon[39612]: 8.c starting backfill to osd.7 from (0'0,0'0] MAX to 206'9115 2024-09-11T08:15:59.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:58 smithi150 ceph-mon[39612]: 8.d starting backfill to osd.7 from (0'0,0'0] MAX to 206'9195 2024-09-11T08:15:59.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:58 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:15:59.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:58 smithi150 ceph-mon[29276]: 4.19 deep-scrub starts 2024-09-11T08:15:59.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:58 smithi150 ceph-mon[29276]: 4.19 deep-scrub ok 2024-09-11T08:15:59.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:58 smithi150 ceph-mon[29276]: 8.c starting backfill to osd.7 from (0'0,0'0] MAX to 206'9115 2024-09-11T08:15:59.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:58 smithi150 ceph-mon[29276]: 8.d starting backfill to osd.7 from (0'0,0'0] MAX to 206'9195 2024-09-11T08:15:59.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:58 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:15:59.323 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:15:59.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:58 smithi181 ceph-mon[32701]: 4.19 deep-scrub starts 2024-09-11T08:15:59.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:58 smithi181 ceph-mon[32701]: 4.19 deep-scrub ok 2024-09-11T08:15:59.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:58 smithi181 ceph-mon[32701]: 8.c starting backfill to osd.7 from (0'0,0'0] MAX to 206'9115 2024-09-11T08:15:59.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:58 smithi181 ceph-mon[32701]: 8.d starting backfill to osd.7 from (0'0,0'0] MAX to 206'9195 2024-09-11T08:15:59.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:58 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:15:59.357 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:59.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:59.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:15:59.459 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:15:59.490 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:59.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:59.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:15:59.592 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:15:59.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:59.623 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:15:59.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:15:59.724 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:15:59.761 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:59.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:59.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:15:59.863 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:15:59.905 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:15:59.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:15:59.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:00.007 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:00.040 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:00.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:00.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:00.141 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:00.177 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:00.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:00.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:00.219 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:59 smithi150 ceph-mon[29276]: 6.1f scrub starts 2024-09-11T08:16:00.219 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:15:59 smithi150 ceph-mon[29276]: pgmap v375: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 863 B/s rd, 0 B/s wr, 1.36k op/s; 13639/330888 objects misplaced (4.122%); 1.4 MiB/s, 30 objects/s recovering 2024-09-11T08:16:00.219 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:59 smithi150 ceph-mon[39612]: 6.1f scrub starts 2024-09-11T08:16:00.219 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:15:59 smithi150 ceph-mon[39612]: pgmap v375: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 863 B/s rd, 0 B/s wr, 1.36k op/s; 13639/330888 objects misplaced (4.122%); 1.4 MiB/s, 30 objects/s recovering 2024-09-11T08:16:00.279 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:00.305 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:59 smithi181 ceph-mon[32701]: 6.1f scrub starts 2024-09-11T08:16:00.305 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:15:59 smithi181 ceph-mon[32701]: pgmap v375: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 863 B/s rd, 0 B/s wr, 1.36k op/s; 13639/330888 objects misplaced (4.122%); 1.4 MiB/s, 30 objects/s recovering 2024-09-11T08:16:00.312 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:00.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:00.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:00.413 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:16:00.444 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:00.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:00.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:16:00.546 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:00.586 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:00.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:00.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:00.687 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:00.724 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:00.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:00.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:00.826 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:00.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:00.866 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:00.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:00.968 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:16:00.995 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:00.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:00.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:16:01.097 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:01.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:01.137 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:01.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:01.207 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:00 smithi150 ceph-mon[39612]: 3.8 scrub starts 2024-09-11T08:16:01.207 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:00 smithi150 ceph-mon[29276]: 3.8 scrub starts 2024-09-11T08:16:01.238 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:16:01.271 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:01.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:01.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:16:01.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:00 smithi181 ceph-mon[32701]: 3.8 scrub starts 2024-09-11T08:16:01.375 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:01.414 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:01.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:01.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:01.515 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:16:01.556 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:01.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:01.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:16:01.658 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:01.691 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:01.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:01.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:01.793 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:16:01.822 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:01.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:01.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:16:01.923 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:01.954 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:01.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:01.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:02.056 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:02.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:02.100 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:02.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:02.176 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:01 smithi150 ceph-mon[29276]: 4.7 scrub starts 2024-09-11T08:16:02.176 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:01 smithi150 ceph-mon[29276]: pgmap v376: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 660 op/s; 13639/330888 objects misplaced (4.122%); 1.4 MiB/s, 30 objects/s recovering 2024-09-11T08:16:02.176 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:01 smithi150 ceph-mon[39612]: 4.7 scrub starts 2024-09-11T08:16:02.176 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:01 smithi150 ceph-mon[39612]: pgmap v376: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 660 op/s; 13639/330888 objects misplaced (4.122%); 1.4 MiB/s, 30 objects/s recovering 2024-09-11T08:16:02.202 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:02.234 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:02.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:02.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:02.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:01 smithi181 ceph-mon[32701]: 4.7 scrub starts 2024-09-11T08:16:02.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:01 smithi181 ceph-mon[32701]: pgmap v376: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 660 op/s; 13639/330888 objects misplaced (4.122%); 1.4 MiB/s, 30 objects/s recovering 2024-09-11T08:16:02.336 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:16:02.364 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:02.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:02.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:16:02.466 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:02.501 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:02.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:02.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:02.603 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:16:02.635 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:02.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:02.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:16:02.737 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:02.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:02.786 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:02.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:02.887 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:02.920 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:16:02.920 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:16:02.931 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:02.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:02.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:02.966 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:16:02.966 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:16:03.000 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:16:03.000 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:16:03.031 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:16:03.031 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:16:03.034 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:16:03.065 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:03.066 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:16:03.066 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:16:03.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:03.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:16:03.096 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:16:03.096 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:16:03.128 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:16:03.128 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:16:03.160 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:16:03.160 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:16:03.168 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:03.176 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:02 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1102000855' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:16:03.176 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:02 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1102000855' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:16:03.207 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:03.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:03.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:03.309 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:03.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:02 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1102000855' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:16:03.342 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:03.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:03.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:03.444 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:03.475 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:03.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:03.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:03.577 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:03.618 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:03.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:03.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:03.720 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:16:03.745 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:03.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:03.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:16:03.847 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:16:03.880 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:03.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:03.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:16:03.981 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:16:04.020 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:04.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:04.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:16:04.122 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:16:04.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:04.158 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:04.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:16:04.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:03 smithi150 ceph-mon[29276]: 8.b scrub starts 2024-09-11T08:16:04.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:03 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:16:04.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:03 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:16:04.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:03 smithi150 ceph-mon[29276]: pgmap v377: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 6.6 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 869 op/s; 12244/323592 objects misplaced (3.784%); 10 MiB/s, 168 objects/s recovering 2024-09-11T08:16:04.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:03 smithi150 ceph-mon[29276]: osdmap e210: 8 total, 8 up, 8 in 2024-09-11T08:16:04.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:03 smithi150 ceph-mon[39612]: 8.b scrub starts 2024-09-11T08:16:04.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:03 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:16:04.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:03 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:16:04.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:03 smithi150 ceph-mon[39612]: pgmap v377: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 6.6 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 869 op/s; 12244/323592 objects misplaced (3.784%); 10 MiB/s, 168 objects/s recovering 2024-09-11T08:16:04.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:03 smithi150 ceph-mon[39612]: osdmap e210: 8 total, 8 up, 8 in 2024-09-11T08:16:04.259 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:04.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:04.297 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:04.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:04.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:03 smithi181 ceph-mon[32701]: 8.b scrub starts 2024-09-11T08:16:04.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:03 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:16:04.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:03 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:16:04.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:03 smithi181 ceph-mon[32701]: pgmap v377: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 6.6 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 869 op/s; 12244/323592 objects misplaced (3.784%); 10 MiB/s, 168 objects/s recovering 2024-09-11T08:16:04.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:03 smithi181 ceph-mon[32701]: osdmap e210: 8 total, 8 up, 8 in 2024-09-11T08:16:04.398 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:04.433 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:04.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:04.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:04.536 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:16:04.568 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:04.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:04.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:16:04.670 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:04.703 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:04.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:04.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:04.792 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:16:04.805 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:16:04.812 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-09-11T08:16:04.812 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:16:04.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:04.835 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:04.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:16:04.936 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:04.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:04.979 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:04.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:05.080 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:16:05.110 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:05.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:05.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:16:05.212 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:05.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:04 smithi150 ceph-mon[29276]: 4.1c deep-scrub starts 2024-09-11T08:16:05.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:04 smithi150 ceph-mon[29276]: 4.1c deep-scrub ok 2024-09-11T08:16:05.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:04 smithi150 ceph-mon[39612]: 4.1c deep-scrub starts 2024-09-11T08:16:05.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:04 smithi150 ceph-mon[39612]: 4.1c deep-scrub ok 2024-09-11T08:16:05.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:05.253 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:05.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:05.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:04 smithi181 ceph-mon[32701]: 4.1c deep-scrub starts 2024-09-11T08:16:05.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:04 smithi181 ceph-mon[32701]: 4.1c deep-scrub ok 2024-09-11T08:16:05.323 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:16:05 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:16:05] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:16:05.354 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:16:05.386 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:05.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:05.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:16:05.487 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:16:05.516 INFO:teuthology.orchestra.run.smithi150.stderr:noscrub is set 2024-09-11T08:16:05.529 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:05.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:05.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:16:05.631 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:05.667 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:05.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:05.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:05.769 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:05.804 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:05.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:05.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:05.906 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:05.928 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:05.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:05.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:06.029 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:16:06.038 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-11T08:16:06.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:06.064 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:06.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:16:06.165 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:16:06.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:06.195 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:06.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:16:06.296 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:16:06.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:05 smithi181 ceph-mon[32701]: 4.12 scrub starts 2024-09-11T08:16:06.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:05 smithi181 ceph-mon[32701]: 4.12 scrub ok 2024-09-11T08:16:06.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:05 smithi181 ceph-mon[32701]: pgmap v379: 148 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 145 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s; 11165/311808 objects misplaced (3.581%); 15 MiB/s, 244 objects/s recovering 2024-09-11T08:16:06.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:05 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/4120257194' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:16:06.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:05 smithi181 ceph-mon[32701]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:16:06.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:05 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/4120257194' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:16:06.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:05 smithi181 ceph-mon[32701]: osdmap e211: 8 total, 8 up, 8 in 2024-09-11T08:16:06.330 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:06.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:06.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:16:06.432 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:16:06.453 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:05 smithi150 ceph-mon[29276]: 4.12 scrub starts 2024-09-11T08:16:06.453 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:05 smithi150 ceph-mon[29276]: 4.12 scrub ok 2024-09-11T08:16:06.453 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:05 smithi150 ceph-mon[29276]: pgmap v379: 148 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 145 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s; 11165/311808 objects misplaced (3.581%); 15 MiB/s, 244 objects/s recovering 2024-09-11T08:16:06.453 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:05 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/4120257194' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:16:06.453 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:05 smithi150 ceph-mon[29276]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:16:06.453 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:05 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/4120257194' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:16:06.454 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:05 smithi150 ceph-mon[29276]: osdmap e211: 8 total, 8 up, 8 in 2024-09-11T08:16:06.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:05 smithi150 ceph-mon[39612]: 4.12 scrub starts 2024-09-11T08:16:06.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:05 smithi150 ceph-mon[39612]: 4.12 scrub ok 2024-09-11T08:16:06.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:05 smithi150 ceph-mon[39612]: pgmap v379: 148 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 145 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s; 11165/311808 objects misplaced (3.581%); 15 MiB/s, 244 objects/s recovering 2024-09-11T08:16:06.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:05 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/4120257194' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:16:06.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:05 smithi150 ceph-mon[39612]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:16:06.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:05 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/4120257194' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:16:06.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:05 smithi150 ceph-mon[39612]: osdmap e211: 8 total, 8 up, 8 in 2024-09-11T08:16:06.461 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:06.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:06.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:16:06.563 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:06.608 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:06.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:06.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:06.710 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:16:06.745 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:06.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:06.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:16:06.846 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:06.883 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:06.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:06.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:06.988 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:07.023 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:07.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:07.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:07.124 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:07.161 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:07.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:07.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:07.211 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:06 smithi150 ceph-mon[29276]: 8.b scrub ok 2024-09-11T08:16:07.211 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:06 smithi150 ceph-mon[29276]: 3.7 scrub starts 2024-09-11T08:16:07.211 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:06 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:16:07.211 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:06 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:16:07.212 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:06 smithi150 ceph-mon[39612]: 8.b scrub ok 2024-09-11T08:16:07.212 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:06 smithi150 ceph-mon[39612]: 3.7 scrub starts 2024-09-11T08:16:07.212 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:06 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:16:07.212 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:06 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:16:07.263 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:07.296 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:07.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:07.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:07.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:06 smithi181 ceph-mon[32701]: 8.b scrub ok 2024-09-11T08:16:07.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:06 smithi181 ceph-mon[32701]: 3.7 scrub starts 2024-09-11T08:16:07.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:06 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:16:07.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:06 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:16:07.398 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:16:07.436 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:07.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:07.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:16:07.538 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:16:07.577 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:07.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:07.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:16:07.679 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:07.716 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:07.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:07.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:07.818 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:07.852 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:07.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:07.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:07.954 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:07.989 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:07.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:07.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:08.091 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:08.129 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:08.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:08.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:08.174 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:07 smithi150 ceph-mon[29276]: pgmap v381: 148 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 145 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 795 op/s; 11165/311808 objects misplaced (3.581%); 14 MiB/s, 222 objects/s recovering 2024-09-11T08:16:08.174 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:07 smithi150 ceph-mon[39612]: pgmap v381: 148 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 145 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 795 op/s; 11165/311808 objects misplaced (3.581%); 14 MiB/s, 222 objects/s recovering 2024-09-11T08:16:08.194 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:16:08.232 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:08.281 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:08.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:08.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:08.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:07 smithi181 ceph-mon[32701]: pgmap v381: 148 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 145 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 795 op/s; 11165/311808 objects misplaced (3.581%); 14 MiB/s, 222 objects/s recovering 2024-09-11T08:16:08.323 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:16:08 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:16:08.383 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:08.423 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:08.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:08.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:08.498 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T08:16:08.525 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:08.550 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:08.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:08.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:08.652 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:08.692 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:08.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:08.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:08.789 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:16:08 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:16:08] "GET /metrics HTTP/1.1" 200 224218 "" "Prometheus/2.43.0" 2024-09-11T08:16:08.794 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:16:08.819 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:08.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:08.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:16:08.921 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:16:08.951 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T08:16:08.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:08.953 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:08.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:16:09.054 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:09.092 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:09.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:09.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:09.194 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:09.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:09.237 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:09.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:09.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:08 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:16:09.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:08 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:16:09.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:08 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:16:09.338 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:09.370 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:09.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:09.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:09.472 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:09.504 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:09.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:09.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:09.606 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:09.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:09.646 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:09.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:09.747 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:16:09.775 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:09.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:09.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:16:09.877 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:09.899 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:09.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:09.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:10.001 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:10.036 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:10.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:10.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:10.138 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:10.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:10.169 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:10.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:10.270 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:10.310 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:10.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:10.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:10.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:10 smithi181 ceph-mon[32701]: pgmap v382: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.52k op/s; 8763/294381 objects misplaced (2.977%); 28 MiB/s, 452 objects/s recovering 2024-09-11T08:16:10.411 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:10.446 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:10.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:10.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:10.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:10 smithi150 ceph-mon[29276]: pgmap v382: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.52k op/s; 8763/294381 objects misplaced (2.977%); 28 MiB/s, 452 objects/s recovering 2024-09-11T08:16:10.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:10 smithi150 ceph-mon[39612]: pgmap v382: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.52k op/s; 8763/294381 objects misplaced (2.977%); 28 MiB/s, 452 objects/s recovering 2024-09-11T08:16:10.518 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:16:10.522 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-09-11T08:16:10.548 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:10.615 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:10.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:10.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:10.717 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:10.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:10.766 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:10.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:10.867 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:10.906 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:10.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:10.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:11.009 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:11.047 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:11.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:11.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:11.149 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:11.189 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:11.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:11.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:11.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:11 smithi150 ceph-mon[29276]: from='client.15207 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:16:11.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:11 smithi150 ceph-mon[39612]: from='client.15207 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:16:11.291 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:11.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:11 smithi181 ceph-mon[32701]: from='client.15207 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:16:11.329 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:11.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:11.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:11.430 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:16:11.466 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:11.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:11.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:16:11.568 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:16:11.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:11.597 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:11.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:16:11.698 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:11.732 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:11.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:11.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:11.834 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:16:11.874 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:11.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:11.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:16:11.976 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:12.016 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:12.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:12.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:12.117 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:16:12.151 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:12.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:12.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:16:12.253 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:12.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:12.290 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:12.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:12 smithi181 ceph-mon[32701]: pgmap v383: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.22k op/s; 8763/294381 objects misplaced (2.977%); 20 MiB/s, 319 objects/s recovering 2024-09-11T08:16:12.390 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:16:12.423 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:12.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:12.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:16:12.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:12 smithi150 ceph-mon[29276]: pgmap v383: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.22k op/s; 8763/294381 objects misplaced (2.977%); 20 MiB/s, 319 objects/s recovering 2024-09-11T08:16:12.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:12 smithi150 ceph-mon[39612]: pgmap v383: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.22k op/s; 8763/294381 objects misplaced (2.977%); 20 MiB/s, 319 objects/s recovering 2024-09-11T08:16:12.524 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:16:12.542 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":382,"stamp":"2024-09-11T08:16:08.872054+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6418536692,"num_objects":98127,"num_object_clones":0,"num_object_copies":294381,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8763,"num_objects_unfound":0,"num_objects_dirty":98127,"num_whiteouts":0,"num_read":934,"num_read_kb":793,"num_write":160272,"num_write_kb":8246728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10271,"num_bytes_recovered":664867332,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85795,"ondisk_log_size":85795,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":451,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":21911684,"kb_used_data":19449968,"kb_used_omap":19,"kb_used_meta":2461292,"kb_avail":728049532,"statfs":{"total":767960285184,"available":745522720768,"internally_reserved":0,"allocated":19916767232,"data_stored":19912386484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19955,"internal_metadata":2520363533},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-797507584,"num_objects":-12169,"num_object_clones":0,"num_object_copies":-36507,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-4876,"num_objects_unfound":0,"num_objects_dirty":-12169,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":12169,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3619,"num_bytes_recovered":237174784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002088"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":191,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.991339+0000","last_change":"2024-09-11T08:12:52.080544+0000","last_active":"2024-09-11T08:16:02.991339+0000","last_peered":"2024-09-11T08:16:02.991339+0000","last_clean":"2024-09-11T08:16:02.991339+0000","last_became_active":"2024-09-11T08:07:30.044976+0000","last_became_peered":"2024-09-11T08:07:30.044976+0000","last_unstale":"2024-09-11T08:16:02.991339+0000","last_undegraded":"2024-09-11T08:16:02.991339+0000","last_fullsized":"2024-09-11T08:16:02.991339+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:12:52.079881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:12:52.079881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.023612166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":205,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.984467+0000","last_change":"2024-09-11T08:15:37.106907+0000","last_active":"2024-09-11T08:16:02.984467+0000","last_peered":"2024-09-11T08:16:02.984467+0000","last_clean":"2024-09-11T08:16:02.984467+0000","last_became_active":"2024-09-11T08:07:32.048048+0000","last_became_peered":"2024-09-11T08:07:32.048048+0000","last_unstale":"2024-09-11T08:16:02.984467+0000","last_undegraded":"2024-09-11T08:16:02.984467+0000","last_fullsized":"2024-09-11T08:16:02.984467+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:15:37.106824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:15:37.106824+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:39.764009+0000","scrub_duration":0.0038285900000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":234,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.999174+0000","last_change":"2024-09-11T08:15:56.531271+0000","last_active":"2024-09-11T08:16:02.999174+0000","last_peered":"2024-09-11T08:16:02.999174+0000","last_clean":"2024-09-11T08:16:02.999174+0000","last_became_active":"2024-09-11T08:15:56.530946+0000","last_became_peered":"2024-09-11T08:15:56.530946+0000","last_unstale":"2024-09-11T08:16:02.999174+0000","last_undegraded":"2024-09-11T08:16:02.999174+0000","last_fullsized":"2024-09-11T08:16:02.999174+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:57.020218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:14:57.020218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:21.588027+0000","scrub_duration":0.0034881460000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":185,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.985430+0000","last_change":"2024-09-11T08:13:10.073583+0000","last_active":"2024-09-11T08:16:02.985430+0000","last_peered":"2024-09-11T08:16:02.985430+0000","last_clean":"2024-09-11T08:16:02.985430+0000","last_became_active":"2024-09-11T08:07:36.074789+0000","last_became_peered":"2024-09-11T08:07:36.074789+0000","last_unstale":"2024-09-11T08:16:02.985430+0000","last_undegraded":"2024-09-11T08:16:02.985430+0000","last_fullsized":"2024-09-11T08:16:02.985430+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:10.073490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:02.470649+0000","last_clean_scrub_stamp":"2024-09-11T08:13:10.073490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0098908899999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":212,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.992896+0000","last_change":"2024-09-11T08:15:56.521398+0000","last_active":"2024-09-11T08:16:02.992896+0000","last_peered":"2024-09-11T08:16:02.992896+0000","last_clean":"2024-09-11T08:16:02.992896+0000","last_became_active":"2024-09-11T08:15:56.521104+0000","last_became_peered":"2024-09-11T08:15:56.521104+0000","last_unstale":"2024-09-11T08:16:02.992896+0000","last_undegraded":"2024-09-11T08:16:02.992896+0000","last_fullsized":"2024-09-11T08:16:02.992896+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:58.222356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:14:58.222356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013526233,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":191,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.991985+0000","last_change":"2024-09-11T08:15:31.254961+0000","last_active":"2024-09-11T08:16:02.991985+0000","last_peered":"2024-09-11T08:16:02.991985+0000","last_clean":"2024-09-11T08:16:02.991985+0000","last_became_active":"2024-09-11T08:07:30.045579+0000","last_became_peered":"2024-09-11T08:07:30.045579+0000","last_unstale":"2024-09-11T08:16:02.991985+0000","last_undegraded":"2024-09-11T08:16:02.991985+0000","last_fullsized":"2024-09-11T08:16:02.991985+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:31.254885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:31.254885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:36.877052+0000","scrub_duration":0.0033689100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"145'22","reported_seq":231,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.991595+0000","last_change":"2024-09-11T08:15:57.478927+0000","last_active":"2024-09-11T08:16:02.991595+0000","last_peered":"2024-09-11T08:16:02.991595+0000","last_clean":"2024-09-11T08:16:02.991595+0000","last_became_active":"2024-09-11T08:07:32.045848+0000","last_became_peered":"2024-09-11T08:07:32.045848+0000","last_unstale":"2024-09-11T08:16:02.991595+0000","last_undegraded":"2024-09-11T08:16:02.991595+0000","last_fullsized":"2024-09-11T08:16:02.991595+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'22","last_scrub_stamp":"2024-09-11T08:15:57.478870+0000","last_deep_scrub":"145'22","last_deep_scrub_stamp":"2024-09-11T08:15:57.478870+0000","last_clean_scrub_stamp":"2024-09-11T08:15:57.478870+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:13.662088+0000","scrub_duration":0.0021529710000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":187,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.984383+0000","last_change":"2024-09-11T08:15:52.050101+0000","last_active":"2024-09-11T08:16:02.984383+0000","last_peered":"2024-09-11T08:16:02.984383+0000","last_clean":"2024-09-11T08:16:02.984383+0000","last_became_active":"2024-09-11T08:07:34.052243+0000","last_became_peered":"2024-09-11T08:07:34.052243+0000","last_unstale":"2024-09-11T08:16:02.984383+0000","last_undegraded":"2024-09-11T08:16:02.984383+0000","last_fullsized":"2024-09-11T08:16:02.984383+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:52.050017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:15:52.050017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:20.035807+0000","scrub_duration":0.0037049880000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":203,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.991428+0000","last_change":"2024-09-11T08:15:31.550213+0000","last_active":"2024-09-11T08:16:02.991428+0000","last_peered":"2024-09-11T08:16:02.991428+0000","last_clean":"2024-09-11T08:16:02.991428+0000","last_became_active":"2024-09-11T08:07:30.043950+0000","last_became_peered":"2024-09-11T08:07:30.043950+0000","last_unstale":"2024-09-11T08:16:02.991428+0000","last_undegraded":"2024-09-11T08:16:02.991428+0000","last_fullsized":"2024-09-11T08:16:02.991428+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:31.550091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:54.878409+0000","last_clean_scrub_stamp":"2024-09-11T08:15:31.550091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:56.355754+0000","scrub_duration":0.0017129999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":205,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.985698+0000","last_change":"2024-09-11T08:13:04.107341+0000","last_active":"2024-09-11T08:16:02.985698+0000","last_peered":"2024-09-11T08:16:02.985698+0000","last_clean":"2024-09-11T08:16:02.985698+0000","last_became_active":"2024-09-11T08:07:32.044626+0000","last_became_peered":"2024-09-11T08:07:32.044626+0000","last_unstale":"2024-09-11T08:16:02.985698+0000","last_undegraded":"2024-09-11T08:16:02.985698+0000","last_fullsized":"2024-09-11T08:16:02.985698+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:13:04.107254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:13:04.107254+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0056027890000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":223,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.990519+0000","last_change":"2024-09-11T08:15:56.531009+0000","last_active":"2024-09-11T08:16:02.990519+0000","last_peered":"2024-09-11T08:16:02.990519+0000","last_clean":"2024-09-11T08:16:02.990519+0000","last_became_active":"2024-09-11T08:15:56.530680+0000","last_became_peered":"2024-09-11T08:15:56.530680+0000","last_unstale":"2024-09-11T08:16:02.990519+0000","last_undegraded":"2024-09-11T08:16:02.990519+0000","last_fullsized":"2024-09-11T08:16:02.990519+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:50.469752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:15:50.469752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:08.638035+0000","scrub_duration":0.0033416409999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":202,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.990691+0000","last_change":"2024-09-11T08:14:53.516662+0000","last_active":"2024-09-11T08:16:02.990691+0000","last_peered":"2024-09-11T08:16:02.990691+0000","last_clean":"2024-09-11T08:16:02.990691+0000","last_became_active":"2024-09-11T08:14:24.792595+0000","last_became_peered":"2024-09-11T08:14:24.792595+0000","last_unstale":"2024-09-11T08:16:02.990691+0000","last_undegraded":"2024-09-11T08:16:02.990691+0000","last_fullsized":"2024-09-11T08:16:02.990691+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:53.516583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:14:53.516583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034487419999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":200,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.993351+0000","last_change":"2024-09-11T08:14:55.444867+0000","last_active":"2024-09-11T08:16:02.993351+0000","last_peered":"2024-09-11T08:16:02.993351+0000","last_clean":"2024-09-11T08:16:02.993351+0000","last_became_active":"2024-09-11T08:07:30.043277+0000","last_became_peered":"2024-09-11T08:07:30.043277+0000","last_unstale":"2024-09-11T08:16:02.993351+0000","last_undegraded":"2024-09-11T08:16:02.993351+0000","last_fullsized":"2024-09-11T08:16:02.993351+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:55.444806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:14:55.444806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0099859159999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"139'7","reported_seq":201,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.984814+0000","last_change":"2024-09-11T08:15:02.988616+0000","last_active":"2024-09-11T08:16:02.984814+0000","last_peered":"2024-09-11T08:16:02.984814+0000","last_clean":"2024-09-11T08:16:02.984814+0000","last_became_active":"2024-09-11T08:07:32.048661+0000","last_became_peered":"2024-09-11T08:07:32.048661+0000","last_unstale":"2024-09-11T08:16:02.984814+0000","last_undegraded":"2024-09-11T08:16:02.984814+0000","last_fullsized":"2024-09-11T08:16:02.984814+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'7","last_scrub_stamp":"2024-09-11T08:15:02.988425+0000","last_deep_scrub":"139'7","last_deep_scrub_stamp":"2024-09-11T08:12:24.869318+0000","last_clean_scrub_stamp":"2024-09-11T08:15:02.988425+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:09.144367+0000","scrub_duration":0.0033901550000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":13,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.985003+0000","last_change":"2024-09-11T08:15:56.527328+0000","last_active":"2024-09-11T08:16:02.985003+0000","last_peered":"2024-09-11T08:16:02.985003+0000","last_clean":"2024-09-11T08:16:02.985003+0000","last_became_active":"2024-09-11T08:15:56.526954+0000","last_became_peered":"2024-09-11T08:15:56.526954+0000","last_unstale":"2024-09-11T08:16:02.985003+0000","last_undegraded":"2024-09-11T08:16:02.985003+0000","last_fullsized":"2024-09-11T08:16:02.985003+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:00.885045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:14:00.885045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":204,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.990417+0000","last_change":"2024-09-11T08:15:55.371107+0000","last_active":"2024-09-11T08:16:02.990417+0000","last_peered":"2024-09-11T08:16:02.990417+0000","last_clean":"2024-09-11T08:16:02.990417+0000","last_became_active":"2024-09-11T08:07:36.077602+0000","last_became_peered":"2024-09-11T08:07:36.077602+0000","last_unstale":"2024-09-11T08:16:02.990417+0000","last_undegraded":"2024-09-11T08:16:02.990417+0000","last_fullsized":"2024-09-11T08:16:02.990417+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:55.371030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:04.782577+0000","last_clean_scrub_stamp":"2024-09-11T08:15:55.371030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:57.415606+0000","scrub_duration":0.0028204950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":185,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.986526+0000","last_change":"2024-09-11T08:13:11.061617+0000","last_active":"2024-09-11T08:16:02.986526+0000","last_peered":"2024-09-11T08:16:02.986526+0000","last_clean":"2024-09-11T08:16:02.986526+0000","last_became_active":"2024-09-11T08:07:36.633338+0000","last_became_peered":"2024-09-11T08:07:36.633338+0000","last_unstale":"2024-09-11T08:16:02.986526+0000","last_undegraded":"2024-09-11T08:16:02.986526+0000","last_fullsized":"2024-09-11T08:16:02.986526+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:11.061558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:13:11.061558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016408200000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":218,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.992028+0000","last_change":"2024-09-11T08:15:56.517613+0000","last_active":"2024-09-11T08:16:02.992028+0000","last_peered":"2024-09-11T08:16:02.992028+0000","last_clean":"2024-09-11T08:16:02.992028+0000","last_became_active":"2024-09-11T08:15:56.516941+0000","last_became_peered":"2024-09-11T08:15:56.516941+0000","last_unstale":"2024-09-11T08:16:02.992028+0000","last_undegraded":"2024-09-11T08:16:02.992028+0000","last_fullsized":"2024-09-11T08:16:02.992028+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:13.315410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:14:13.315410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030491400000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"107'22","reported_seq":240,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:04.159612+0000","last_change":"2024-09-11T08:16:04.159388+0000","last_active":"2024-09-11T08:16:04.159612+0000","last_peered":"2024-09-11T08:16:04.159612+0000","last_clean":"2024-09-11T08:16:04.159612+0000","last_became_active":"2024-09-11T08:07:32.046363+0000","last_became_peered":"2024-09-11T08:07:32.046363+0000","last_unstale":"2024-09-11T08:16:04.159612+0000","last_undegraded":"2024-09-11T08:16:04.159612+0000","last_fullsized":"2024-09-11T08:16:04.159612+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'22","last_scrub_stamp":"2024-09-11T08:16:04.159211+0000","last_deep_scrub":"107'22","last_deep_scrub_stamp":"2024-09-11T08:16:04.159211+0000","last_clean_scrub_stamp":"2024-09-11T08:16:04.159211+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:30.394200+0000","scrub_duration":0.0069586229999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":207,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.999536+0000","last_change":"2024-09-11T08:15:46.597272+0000","last_active":"2024-09-11T08:16:02.999536+0000","last_peered":"2024-09-11T08:16:02.999536+0000","last_clean":"2024-09-11T08:16:02.999536+0000","last_became_active":"2024-09-11T08:07:34.055410+0000","last_became_peered":"2024-09-11T08:07:34.055410+0000","last_unstale":"2024-09-11T08:16:02.999536+0000","last_undegraded":"2024-09-11T08:16:02.999536+0000","last_fullsized":"2024-09-11T08:16:02.999536+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:46.597198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:10:31.418873+0000","last_clean_scrub_stamp":"2024-09-11T08:15:46.597198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:00.695980+0000","scrub_duration":0.0032858620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":178,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.995150+0000","last_change":"2024-09-11T08:15:58.526429+0000","last_active":"2024-09-11T08:16:02.995150+0000","last_peered":"2024-09-11T08:16:02.995150+0000","last_clean":"2024-09-11T08:16:02.995150+0000","last_became_active":"2024-09-11T08:07:36.633258+0000","last_became_peered":"2024-09-11T08:07:36.633258+0000","last_unstale":"2024-09-11T08:16:02.995150+0000","last_undegraded":"2024-09-11T08:16:02.995150+0000","last_fullsized":"2024-09-11T08:16:02.995150+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:12:58.260306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:12:58.260306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015796729999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":201,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.985740+0000","last_change":"2024-09-11T08:15:48.081690+0000","last_active":"2024-09-11T08:16:02.985740+0000","last_peered":"2024-09-11T08:16:02.985740+0000","last_clean":"2024-09-11T08:16:02.985740+0000","last_became_active":"2024-09-11T08:07:30.046284+0000","last_became_peered":"2024-09-11T08:07:30.046284+0000","last_unstale":"2024-09-11T08:16:02.985740+0000","last_undegraded":"2024-09-11T08:16:02.985740+0000","last_fullsized":"2024-09-11T08:16:02.985740+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:48.081591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:48.081591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:03.932824+0000","scrub_duration":0.0059702619999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":244,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.991575+0000","last_change":"2024-09-11T08:15:53.518868+0000","last_active":"2024-09-11T08:16:02.991575+0000","last_peered":"2024-09-11T08:16:02.991575+0000","last_clean":"2024-09-11T08:16:02.991575+0000","last_became_active":"2024-09-11T08:09:07.878432+0000","last_became_peered":"2024-09-11T08:09:07.878432+0000","last_unstale":"2024-09-11T08:16:02.991575+0000","last_undegraded":"2024-09-11T08:16:02.991575+0000","last_fullsized":"2024-09-11T08:16:02.991575+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-11T08:15:53.518765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:15:53.518765+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:19.946047+0000","scrub_duration":0.0047796499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":187,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.985872+0000","last_change":"2024-09-11T08:13:06.131006+0000","last_active":"2024-09-11T08:16:02.985872+0000","last_peered":"2024-09-11T08:16:02.985872+0000","last_clean":"2024-09-11T08:16:02.985872+0000","last_became_active":"2024-09-11T08:07:34.056528+0000","last_became_peered":"2024-09-11T08:07:34.056528+0000","last_unstale":"2024-09-11T08:16:02.985872+0000","last_undegraded":"2024-09-11T08:16:02.985872+0000","last_fullsized":"2024-09-11T08:16:02.985872+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:06.130952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:10:40.110490+0000","last_clean_scrub_stamp":"2024-09-11T08:13:06.130952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050011930000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":140,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.986318+0000","last_change":"2024-09-11T08:15:03.838012+0000","last_active":"2024-09-11T08:16:02.986318+0000","last_peered":"2024-09-11T08:16:02.986318+0000","last_clean":"2024-09-11T08:16:02.986318+0000","last_became_active":"2024-09-11T08:14:24.791610+0000","last_became_peered":"2024-09-11T08:14:24.791610+0000","last_unstale":"2024-09-11T08:16:02.986318+0000","last_undegraded":"2024-09-11T08:16:02.986318+0000","last_fullsized":"2024-09-11T08:16:02.986318+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-11T08:15:03.837911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:15:03.837911+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:29.972319+0000","scrub_duration":0.0035450270000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":201,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.999286+0000","last_change":"2024-09-11T08:14:13.134615+0000","last_active":"2024-09-11T08:16:02.999286+0000","last_peered":"2024-09-11T08:16:02.999286+0000","last_clean":"2024-09-11T08:16:02.999286+0000","last_became_active":"2024-09-11T08:07:30.045932+0000","last_became_peered":"2024-09-11T08:07:30.045932+0000","last_unstale":"2024-09-11T08:16:02.999286+0000","last_undegraded":"2024-09-11T08:16:02.999286+0000","last_fullsized":"2024-09-11T08:16:02.999286+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:13.134086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:14:13.134086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0059195080000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":277,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.992231+0000","last_change":"2024-09-11T08:15:56.536927+0000","last_active":"2024-09-11T08:16:02.992231+0000","last_peered":"2024-09-11T08:16:02.992231+0000","last_clean":"2024-09-11T08:16:02.992231+0000","last_became_active":"2024-09-11T08:15:56.515817+0000","last_became_peered":"2024-09-11T08:15:56.515817+0000","last_unstale":"2024-09-11T08:16:02.992231+0000","last_undegraded":"2024-09-11T08:16:02.992231+0000","last_fullsized":"2024-09-11T08:16:02.992231+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-11T08:14:00.393362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:14:00.393362+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018583250000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"209'11","reported_seq":356,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:03.651266+0000","last_change":"2024-09-11T08:15:56.541377+0000","last_active":"2024-09-11T08:16:03.651266+0000","last_peered":"2024-09-11T08:16:03.651266+0000","last_clean":"2024-09-11T08:16:03.651266+0000","last_became_active":"2024-09-11T08:15:56.525498+0000","last_became_peered":"2024-09-11T08:15:56.525498+0000","last_unstale":"2024-09-11T08:16:03.651266+0000","last_undegraded":"2024-09-11T08:16:03.651266+0000","last_fullsized":"2024-09-11T08:16:03.651266+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"156'10","last_scrub_stamp":"2024-09-11T08:15:01.002908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:15:01.002908+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:10.643002+0000","scrub_duration":0.0026556840000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"211'10017","reported_seq":10077,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:08.132336+0000","last_change":"2024-09-11T08:13:54.883961+0000","last_active":"2024-09-11T08:16:08.132336+0000","last_peered":"2024-09-11T08:16:08.132336+0000","last_clean":"2024-09-11T08:16:08.132336+0000","last_became_active":"2024-09-11T08:13:54.883685+0000","last_became_peered":"2024-09-11T08:13:54.883685+0000","last_unstale":"2024-09-11T08:16:08.132336+0000","last_undegraded":"2024-09-11T08:16:08.132336+0000","last_fullsized":"2024-09-11T08:16:08.132336+0000","mapping_epoch":159,"log_start":"185'5000","ondisk_log_start":"185'5000","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:13:53.862306+0000","objects_scrubbed":0,"log_size":5017,"log_dups_size":0,"ondisk_log_size":5017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":401801216,"num_objects":6131,"num_object_clones":0,"num_object_copies":18393,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10017,"num_write_kb":516736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":234,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:03.004256+0000","last_change":"2024-09-11T08:15:42.534547+0000","last_active":"2024-09-11T08:16:03.004256+0000","last_peered":"2024-09-11T08:16:03.004256+0000","last_clean":"2024-09-11T08:16:03.004256+0000","last_became_active":"2024-09-11T08:07:32.054317+0000","last_became_peered":"2024-09-11T08:07:32.054317+0000","last_unstale":"2024-09-11T08:16:03.004256+0000","last_undegraded":"2024-09-11T08:16:03.004256+0000","last_fullsized":"2024-09-11T08:16:03.004256+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-11T08:15:42.534460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:15:42.534460+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:51.887358+0000","scrub_duration":0.0078017169999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":211,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.993517+0000","last_change":"2024-09-11T08:15:59.503489+0000","last_active":"2024-09-11T08:16:02.993517+0000","last_peered":"2024-09-11T08:16:02.993517+0000","last_clean":"2024-09-11T08:16:02.993517+0000","last_became_active":"2024-09-11T08:15:56.521943+0000","last_became_peered":"2024-09-11T08:15:56.521943+0000","last_unstale":"2024-09-11T08:16:02.993517+0000","last_undegraded":"2024-09-11T08:16:02.993517+0000","last_fullsized":"2024-09-11T08:16:02.993517+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:50.951830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:13:50.951830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011546670000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":298,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:03.651390+0000","last_change":"2024-09-11T08:13:12.092663+0000","last_active":"2024-09-11T08:16:03.651390+0000","last_peered":"2024-09-11T08:16:03.651390+0000","last_clean":"2024-09-11T08:16:03.651390+0000","last_became_active":"2024-09-11T08:07:34.052981+0000","last_became_peered":"2024-09-11T08:07:34.052981+0000","last_unstale":"2024-09-11T08:16:03.651390+0000","last_undegraded":"2024-09-11T08:16:03.651390+0000","last_fullsized":"2024-09-11T08:16:03.651390+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:13:12.092586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:13:12.092586+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012246072,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":204,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.991192+0000","last_change":"2024-09-11T08:15:45.411776+0000","last_active":"2024-09-11T08:16:02.991192+0000","last_peered":"2024-09-11T08:16:02.991192+0000","last_clean":"2024-09-11T08:16:02.991192+0000","last_became_active":"2024-09-11T08:07:36.073969+0000","last_became_peered":"2024-09-11T08:07:36.073969+0000","last_unstale":"2024-09-11T08:16:02.991192+0000","last_undegraded":"2024-09-11T08:16:02.991192+0000","last_fullsized":"2024-09-11T08:16:02.991192+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:45.411593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:06.616332+0000","last_clean_scrub_stamp":"2024-09-11T08:15:45.411593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:06.019721+0000","scrub_duration":0.0031499940000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"211'9917","reported_seq":13001,"reported_epoch":211,"state":"active+remapped+backfilling","last_fresh":"2024-09-11T08:16:08.231095+0000","last_change":"2024-09-11T08:15:57.607034+0000","last_active":"2024-09-11T08:16:08.231095+0000","last_peered":"2024-09-11T08:16:08.231095+0000","last_clean":"2024-09-11T08:15:55.461733+0000","last_became_active":"2024-09-11T08:15:57.579110+0000","last_became_peered":"2024-09-11T08:15:57.579110+0000","last_unstale":"2024-09-11T08:16:08.231095+0000","last_undegraded":"2024-09-11T08:16:08.231095+0000","last_fullsized":"2024-09-11T08:16:08.231095+0000","mapping_epoch":208,"log_start":"177'4032","ondisk_log_start":"177'4032","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:13:53.862306+0000","objects_scrubbed":0,"log_size":5885,"log_dups_size":0,"ondisk_log_size":5885,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398262272,"num_objects":6077,"num_object_clones":0,"num_object_copies":18231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4662,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9917,"num_write_kb":511808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1506,"num_bytes_recovered":98697216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,5],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":6077}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":284,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.993352+0000","last_change":"2024-09-11T08:15:56.542573+0000","last_active":"2024-09-11T08:16:02.993352+0000","last_peered":"2024-09-11T08:16:02.993352+0000","last_clean":"2024-09-11T08:16:02.993352+0000","last_became_active":"2024-09-11T08:15:56.517680+0000","last_became_peered":"2024-09-11T08:15:56.517680+0000","last_unstale":"2024-09-11T08:16:02.993352+0000","last_undegraded":"2024-09-11T08:16:02.993352+0000","last_fullsized":"2024-09-11T08:16:02.993352+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-11T08:13:58.979187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:13:58.979187+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001885091,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":211,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.435990+0000","last_change":"2024-09-11T08:16:05.435966+0000","last_active":"2024-09-11T08:16:05.435990+0000","last_peered":"2024-09-11T08:16:05.435990+0000","last_clean":"2024-09-11T08:16:05.435990+0000","last_became_active":"2024-09-11T08:15:56.527438+0000","last_became_peered":"2024-09-11T08:15:56.527438+0000","last_unstale":"2024-09-11T08:16:05.435990+0000","last_undegraded":"2024-09-11T08:16:05.435990+0000","last_fullsized":"2024-09-11T08:16:05.435990+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:46.896181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:13:46.896181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035672669999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":214,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.985320+0000","last_change":"2024-09-11T08:15:56.522907+0000","last_active":"2024-09-11T08:16:02.985320+0000","last_peered":"2024-09-11T08:16:02.985320+0000","last_clean":"2024-09-11T08:16:02.985320+0000","last_became_active":"2024-09-11T08:15:56.522606+0000","last_became_peered":"2024-09-11T08:15:56.522606+0000","last_unstale":"2024-09-11T08:16:02.985320+0000","last_undegraded":"2024-09-11T08:16:02.985320+0000","last_fullsized":"2024-09-11T08:16:02.985320+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:57.840457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:13:57.840457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014582009999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":185,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.985619+0000","last_change":"2024-09-11T08:14:04.937966+0000","last_active":"2024-09-11T08:16:02.985619+0000","last_peered":"2024-09-11T08:16:02.985619+0000","last_clean":"2024-09-11T08:16:02.985619+0000","last_became_active":"2024-09-11T08:07:36.069225+0000","last_became_peered":"2024-09-11T08:07:36.069225+0000","last_unstale":"2024-09-11T08:16:02.985619+0000","last_undegraded":"2024-09-11T08:16:02.985619+0000","last_fullsized":"2024-09-11T08:16:02.985619+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:04.937884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:04.937884+0000","last_clean_scrub_stamp":"2024-09-11T08:14:04.937884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018777710000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"211'9941","reported_seq":14798,"reported_epoch":211,"state":"active+remapped+backfilling","last_fresh":"2024-09-11T08:16:07.851895+0000","last_change":"2024-09-11T08:15:57.602250+0000","last_active":"2024-09-11T08:16:07.851895+0000","last_peered":"2024-09-11T08:16:07.851895+0000","last_clean":"2024-09-11T08:15:55.508898+0000","last_became_active":"2024-09-11T08:15:57.581916+0000","last_became_peered":"2024-09-11T08:15:57.581916+0000","last_unstale":"2024-09-11T08:16:07.851895+0000","last_undegraded":"2024-09-11T08:16:07.851895+0000","last_fullsized":"2024-09-11T08:16:07.851895+0000","mapping_epoch":208,"log_start":"183'4600","ondisk_log_start":"183'4600","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'8832","last_scrub_stamp":"2024-09-11T08:15:51.203402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:15:51.203402+0000","objects_scrubbed":7454,"log_size":5341,"log_dups_size":0,"ondisk_log_size":5341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:58.659559+0000","scrub_duration":2.7862675239999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":407371800,"num_objects":6217,"num_object_clones":0,"num_object_copies":18651,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4101,"num_objects_unfound":0,"num_objects_dirty":6217,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9941,"num_write_kb":516993,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2237,"num_bytes_recovered":146604032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,2,6],"avail_no_missing":["5","2","6"],"object_location_counts":[{"shards":"2,5,6","objects":6217}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":210,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.985558+0000","last_change":"2024-09-11T08:14:59.014008+0000","last_active":"2024-09-11T08:16:02.985558+0000","last_peered":"2024-09-11T08:16:02.985558+0000","last_clean":"2024-09-11T08:16:02.985558+0000","last_became_active":"2024-09-11T08:07:32.040069+0000","last_became_peered":"2024-09-11T08:07:32.040069+0000","last_unstale":"2024-09-11T08:16:02.985558+0000","last_undegraded":"2024-09-11T08:16:02.985558+0000","last_fullsized":"2024-09-11T08:16:02.985558+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-11T08:14:59.013949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:14:59.013949+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001898156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":201,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.992646+0000","last_change":"2024-09-11T08:15:54.794370+0000","last_active":"2024-09-11T08:16:02.992646+0000","last_peered":"2024-09-11T08:16:02.992646+0000","last_clean":"2024-09-11T08:16:02.992646+0000","last_became_active":"2024-09-11T08:07:30.050484+0000","last_became_peered":"2024-09-11T08:07:30.050484+0000","last_unstale":"2024-09-11T08:16:02.992646+0000","last_undegraded":"2024-09-11T08:16:02.992646+0000","last_fullsized":"2024-09-11T08:16:02.992646+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:54.793950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:54.793950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:06.106542+0000","scrub_duration":0.0059264870000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"82'9","reported_seq":312,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:03.656114+0000","last_change":"2024-09-11T08:15:51.456046+0000","last_active":"2024-09-11T08:16:03.656114+0000","last_peered":"2024-09-11T08:16:03.656114+0000","last_clean":"2024-09-11T08:16:03.656114+0000","last_became_active":"2024-09-11T08:07:34.056143+0000","last_became_peered":"2024-09-11T08:07:34.056143+0000","last_unstale":"2024-09-11T08:16:03.656114+0000","last_undegraded":"2024-09-11T08:16:03.656114+0000","last_fullsized":"2024-09-11T08:16:03.656114+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-11T08:15:51.455966+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-09-11T08:13:10.230703+0000","last_clean_scrub_stamp":"2024-09-11T08:15:51.455966+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:01.331170+0000","scrub_duration":0.00915995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":175,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.985451+0000","last_change":"2024-09-11T08:15:33.507691+0000","last_active":"2024-09-11T08:16:02.985451+0000","last_peered":"2024-09-11T08:16:02.985451+0000","last_clean":"2024-09-11T08:16:02.985451+0000","last_became_active":"2024-09-11T08:14:24.788968+0000","last_became_peered":"2024-09-11T08:14:24.788968+0000","last_unstale":"2024-09-11T08:16:02.985451+0000","last_undegraded":"2024-09-11T08:16:02.985451+0000","last_fullsized":"2024-09-11T08:16:02.985451+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:33.507613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:15:33.507613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:55.580423+0000","scrub_duration":0.0032985240000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.e","version":"211'9880","reported_seq":9940,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:08.188093+0000","last_change":"2024-09-11T08:13:55.705967+0000","last_active":"2024-09-11T08:16:08.188093+0000","last_peered":"2024-09-11T08:16:08.188093+0000","last_clean":"2024-09-11T08:16:08.188093+0000","last_became_active":"2024-09-11T08:13:55.705670+0000","last_became_peered":"2024-09-11T08:13:55.705670+0000","last_unstale":"2024-09-11T08:16:08.188093+0000","last_undegraded":"2024-09-11T08:16:08.188093+0000","last_fullsized":"2024-09-11T08:16:08.188093+0000","mapping_epoch":159,"log_start":"177'4000","ondisk_log_start":"177'4000","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:13:53.862306+0000","objects_scrubbed":0,"log_size":5880,"log_dups_size":0,"ondisk_log_size":5880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":397541376,"num_objects":6066,"num_object_clones":0,"num_object_copies":18198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9880,"num_write_kb":510272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":217,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.999666+0000","last_change":"2024-09-11T08:15:36.365980+0000","last_active":"2024-09-11T08:16:02.999666+0000","last_peered":"2024-09-11T08:16:02.999666+0000","last_clean":"2024-09-11T08:16:02.999666+0000","last_became_active":"2024-09-11T08:07:32.054294+0000","last_became_peered":"2024-09-11T08:07:32.054294+0000","last_unstale":"2024-09-11T08:16:02.999666+0000","last_undegraded":"2024-09-11T08:16:02.999666+0000","last_fullsized":"2024-09-11T08:16:02.999666+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-11T08:15:36.365891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:15:36.365891+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:05.688088+0000","scrub_duration":0.003716071,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":200,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.991457+0000","last_change":"2024-09-11T08:15:46.459225+0000","last_active":"2024-09-11T08:16:02.991457+0000","last_peered":"2024-09-11T08:16:02.991457+0000","last_clean":"2024-09-11T08:16:02.991457+0000","last_became_active":"2024-09-11T08:07:30.047497+0000","last_became_peered":"2024-09-11T08:07:30.047497+0000","last_unstale":"2024-09-11T08:16:02.991457+0000","last_undegraded":"2024-09-11T08:16:02.991457+0000","last_fullsized":"2024-09-11T08:16:02.991457+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:46.459143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:46.459143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:12.469916+0000","scrub_duration":0.0057785019999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"190'9","reported_seq":47,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:03.652766+0000","last_change":"2024-09-11T08:15:08.769903+0000","last_active":"2024-09-11T08:16:03.652766+0000","last_peered":"2024-09-11T08:16:03.652766+0000","last_clean":"2024-09-11T08:16:03.652766+0000","last_became_active":"2024-09-11T08:15:08.764876+0000","last_became_peered":"2024-09-11T08:15:08.764876+0000","last_unstale":"2024-09-11T08:16:03.652766+0000","last_undegraded":"2024-09-11T08:16:03.652766+0000","last_fullsized":"2024-09-11T08:16:03.652766+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:14:11.320174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:14:11.320174+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017353449999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"66'1","reported_seq":186,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.990717+0000","last_change":"2024-09-11T08:15:06.445409+0000","last_active":"2024-09-11T08:16:02.990717+0000","last_peered":"2024-09-11T08:16:02.990717+0000","last_clean":"2024-09-11T08:16:02.990717+0000","last_became_active":"2024-09-11T08:07:36.072593+0000","last_became_peered":"2024-09-11T08:07:36.072593+0000","last_unstale":"2024-09-11T08:16:02.990717+0000","last_undegraded":"2024-09-11T08:16:02.990717+0000","last_fullsized":"2024-09-11T08:16:02.990717+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-11T08:15:06.445301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:15:06.445301+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:09.835968+0000","scrub_duration":0.013971574,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.f","version":"211'9944","reported_seq":10004,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:08.020738+0000","last_change":"2024-09-11T08:13:55.706681+0000","last_active":"2024-09-11T08:16:08.020738+0000","last_peered":"2024-09-11T08:16:08.020738+0000","last_clean":"2024-09-11T08:16:08.020738+0000","last_became_active":"2024-09-11T08:13:55.706134+0000","last_became_peered":"2024-09-11T08:13:55.706134+0000","last_unstale":"2024-09-11T08:16:08.020738+0000","last_undegraded":"2024-09-11T08:16:08.020738+0000","last_fullsized":"2024-09-11T08:16:08.020738+0000","mapping_epoch":159,"log_start":"185'5109","ondisk_log_start":"185'5109","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:13:53.862306+0000","objects_scrubbed":0,"log_size":4835,"log_dups_size":0,"ondisk_log_size":4835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402128896,"num_objects":6136,"num_object_clones":0,"num_object_copies":18408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9944,"num_write_kb":514560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":288,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.991285+0000","last_change":"2024-09-11T08:15:56.540181+0000","last_active":"2024-09-11T08:16:02.991285+0000","last_peered":"2024-09-11T08:16:02.991285+0000","last_clean":"2024-09-11T08:16:02.991285+0000","last_became_active":"2024-09-11T08:15:56.518384+0000","last_became_peered":"2024-09-11T08:15:56.518384+0000","last_unstale":"2024-09-11T08:16:02.991285+0000","last_undegraded":"2024-09-11T08:16:02.991285+0000","last_fullsized":"2024-09-11T08:16:02.991285+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-11T08:13:57.387660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:13:57.387660+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025295550000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":211,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.999269+0000","last_change":"2024-09-11T08:15:54.517507+0000","last_active":"2024-09-11T08:16:02.999269+0000","last_peered":"2024-09-11T08:16:02.999269+0000","last_clean":"2024-09-11T08:16:02.999269+0000","last_became_active":"2024-09-11T08:07:30.045380+0000","last_became_peered":"2024-09-11T08:07:30.045380+0000","last_unstale":"2024-09-11T08:16:02.999269+0000","last_undegraded":"2024-09-11T08:16:02.999269+0000","last_fullsized":"2024-09-11T08:16:02.999269+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:54.517429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:54.517429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:05.165550+0000","scrub_duration":0.0055168639999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":206,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.994642+0000","last_change":"2024-09-11T08:15:54.238757+0000","last_active":"2024-09-11T08:16:02.994642+0000","last_peered":"2024-09-11T08:16:02.994642+0000","last_clean":"2024-09-11T08:16:02.994642+0000","last_became_active":"2024-09-11T08:07:34.051004+0000","last_became_peered":"2024-09-11T08:07:34.051004+0000","last_unstale":"2024-09-11T08:16:02.994642+0000","last_undegraded":"2024-09-11T08:16:02.994642+0000","last_fullsized":"2024-09-11T08:16:02.994642+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:54.238680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:15:54.238680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:58.812475+0000","scrub_duration":0.0019626420000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":195,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.999111+0000","last_change":"2024-09-11T08:14:09.068749+0000","last_active":"2024-09-11T08:16:02.999111+0000","last_peered":"2024-09-11T08:16:02.999111+0000","last_clean":"2024-09-11T08:16:02.999111+0000","last_became_active":"2024-09-11T08:07:36.631668+0000","last_became_peered":"2024-09-11T08:07:36.631668+0000","last_unstale":"2024-09-11T08:16:02.999111+0000","last_undegraded":"2024-09-11T08:16:02.999111+0000","last_fullsized":"2024-09-11T08:16:02.999111+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:09.068694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:14:09.068694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033190699999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"211'10131","reported_seq":14761,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:07.821665+0000","last_change":"2024-09-11T08:15:56.003569+0000","last_active":"2024-09-11T08:16:07.821665+0000","last_peered":"2024-09-11T08:16:07.821665+0000","last_clean":"2024-09-11T08:16:07.821665+0000","last_became_active":"2024-09-11T08:14:50.875690+0000","last_became_peered":"2024-09-11T08:14:50.875690+0000","last_unstale":"2024-09-11T08:16:07.821665+0000","last_undegraded":"2024-09-11T08:16:07.821665+0000","last_fullsized":"2024-09-11T08:16:07.821665+0000","mapping_epoch":179,"log_start":"183'4800","ondisk_log_start":"183'4800","created":159,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'9330","last_scrub_stamp":"2024-09-11T08:15:56.003501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:15:56.003501+0000","objects_scrubbed":7010,"log_size":5331,"log_dups_size":0,"ondisk_log_size":5331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:14.010825+0000","scrub_duration":1.6004849619999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404946944,"num_objects":6179,"num_object_clones":0,"num_object_copies":18537,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10131,"num_write_kb":521920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2130,"num_bytes_recovered":139591680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"190'39","reported_seq":75,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:03.651250+0000","last_change":"2024-09-11T08:15:52.848067+0000","last_active":"2024-09-11T08:16:03.651250+0000","last_peered":"2024-09-11T08:16:03.651250+0000","last_clean":"2024-09-11T08:16:03.651250+0000","last_became_active":"2024-09-11T08:15:08.775181+0000","last_became_peered":"2024-09-11T08:15:08.775181+0000","last_unstale":"2024-09-11T08:16:03.651250+0000","last_undegraded":"2024-09-11T08:16:03.651250+0000","last_fullsized":"2024-09-11T08:16:03.651250+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'39","last_scrub_stamp":"2024-09-11T08:15:52.847968+0000","last_deep_scrub":"190'39","last_deep_scrub_stamp":"2024-09-11T08:15:52.847968+0000","last_clean_scrub_stamp":"2024-09-11T08:15:52.847968+0000","objects_scrubbed":10,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:11.824582+0000","scrub_duration":0.012280474,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"59'1","reported_seq":210,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.985144+0000","last_change":"2024-09-11T08:15:44.992430+0000","last_active":"2024-09-11T08:16:02.985144+0000","last_peered":"2024-09-11T08:16:02.985144+0000","last_clean":"2024-09-11T08:16:02.985144+0000","last_became_active":"2024-09-11T08:07:30.047513+0000","last_became_peered":"2024-09-11T08:07:30.047513+0000","last_unstale":"2024-09-11T08:16:02.985144+0000","last_undegraded":"2024-09-11T08:16:02.985144+0000","last_fullsized":"2024-09-11T08:16:02.985144+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-11T08:15:44.992356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:44.992356+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:00.977258+0000","scrub_duration":0.0049464690000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":207,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.990714+0000","last_change":"2024-09-11T08:14:57.417159+0000","last_active":"2024-09-11T08:16:02.990714+0000","last_peered":"2024-09-11T08:16:02.990714+0000","last_clean":"2024-09-11T08:16:02.990714+0000","last_became_active":"2024-09-11T08:07:30.043130+0000","last_became_peered":"2024-09-11T08:07:30.043130+0000","last_unstale":"2024-09-11T08:16:02.990714+0000","last_undegraded":"2024-09-11T08:16:02.990714+0000","last_fullsized":"2024-09-11T08:16:02.990714+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-11T08:14:57.417078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:14:57.417078+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026993970000000001,"objects_trimmed":0,"snaptrim_duration":0.0026120850000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":187,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.991099+0000","last_change":"2024-09-11T08:13:01.147269+0000","last_active":"2024-09-11T08:16:02.991099+0000","last_peered":"2024-09-11T08:16:02.991099+0000","last_clean":"2024-09-11T08:16:02.991099+0000","last_became_active":"2024-09-11T08:07:34.057154+0000","last_became_peered":"2024-09-11T08:07:34.057154+0000","last_unstale":"2024-09-11T08:16:02.991099+0000","last_undegraded":"2024-09-11T08:16:02.991099+0000","last_fullsized":"2024-09-11T08:16:02.991099+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:01.147031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:10:37.957096+0000","last_clean_scrub_stamp":"2024-09-11T08:13:01.147031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.027172575000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":212,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.992243+0000","last_change":"2024-09-11T08:15:56.527473+0000","last_active":"2024-09-11T08:16:02.992243+0000","last_peered":"2024-09-11T08:16:02.992243+0000","last_clean":"2024-09-11T08:16:02.992243+0000","last_became_active":"2024-09-11T08:15:56.525099+0000","last_became_peered":"2024-09-11T08:15:56.525099+0000","last_unstale":"2024-09-11T08:16:02.992243+0000","last_undegraded":"2024-09-11T08:16:02.992243+0000","last_fullsized":"2024-09-11T08:16:02.992243+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:05.076884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:14:05.076884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001986065,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"211'9820","reported_seq":10141,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:08.028866+0000","last_change":"2024-09-11T08:16:05.402156+0000","last_active":"2024-09-11T08:16:08.028866+0000","last_peered":"2024-09-11T08:16:08.028866+0000","last_clean":"2024-09-11T08:16:08.028866+0000","last_became_active":"2024-09-11T08:13:54.882290+0000","last_became_peered":"2024-09-11T08:13:54.882290+0000","last_unstale":"2024-09-11T08:16:08.028866+0000","last_undegraded":"2024-09-11T08:16:08.028866+0000","last_fullsized":"2024-09-11T08:16:08.028866+0000","mapping_epoch":159,"log_start":"185'5005","ondisk_log_start":"185'5005","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'9635","last_scrub_stamp":"2024-09-11T08:16:05.401974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:16:05.401974+0000","objects_scrubbed":6255,"log_size":4815,"log_dups_size":0,"ondisk_log_size":4815,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:26.177859+0000","scrub_duration":2.9696664940000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":389939200,"num_objects":5950,"num_object_clones":0,"num_object_copies":17850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9820,"num_write_kb":504640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":229,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:03.004323+0000","last_change":"2024-09-11T08:16:00.399312+0000","last_active":"2024-09-11T08:16:03.004323+0000","last_peered":"2024-09-11T08:16:03.004323+0000","last_clean":"2024-09-11T08:16:03.004323+0000","last_became_active":"2024-09-11T08:07:32.050673+0000","last_became_peered":"2024-09-11T08:07:32.050673+0000","last_unstale":"2024-09-11T08:16:03.004323+0000","last_undegraded":"2024-09-11T08:16:03.004323+0000","last_fullsized":"2024-09-11T08:16:03.004323+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'13","last_scrub_stamp":"2024-09-11T08:13:52.948301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:13:52.948301+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021084820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":204,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.998455+0000","last_change":"2024-09-11T08:15:31.273498+0000","last_active":"2024-09-11T08:16:02.998455+0000","last_peered":"2024-09-11T08:16:02.998455+0000","last_clean":"2024-09-11T08:16:02.998455+0000","last_became_active":"2024-09-11T08:07:30.047014+0000","last_became_peered":"2024-09-11T08:07:30.047014+0000","last_unstale":"2024-09-11T08:16:02.998455+0000","last_undegraded":"2024-09-11T08:16:02.998455+0000","last_fullsized":"2024-09-11T08:16:02.998455+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:31.273415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:31.273415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:59.181535+0000","scrub_duration":0.0057528060000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"127'2","reported_seq":220,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.997387+0000","last_change":"2024-09-11T08:15:01.938952+0000","last_active":"2024-09-11T08:16:02.997387+0000","last_peered":"2024-09-11T08:16:02.997387+0000","last_clean":"2024-09-11T08:16:02.997387+0000","last_became_active":"2024-09-11T08:07:30.045009+0000","last_became_peered":"2024-09-11T08:07:30.045009+0000","last_unstale":"2024-09-11T08:16:02.997387+0000","last_undegraded":"2024-09-11T08:16:02.997387+0000","last_fullsized":"2024-09-11T08:16:02.997387+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'2","last_scrub_stamp":"2024-09-11T08:15:01.938497+0000","last_deep_scrub":"127'2","last_deep_scrub_stamp":"2024-09-11T08:15:01.938497+0000","last_clean_scrub_stamp":"2024-09-11T08:15:01.938497+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:09.383968+0000","scrub_duration":0.034714793000000001,"objects_trimmed":0,"snaptrim_duration":0.001859533,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":216,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:03.001505+0000","last_change":"2024-09-11T08:14:58.889181+0000","last_active":"2024-09-11T08:16:03.001505+0000","last_peered":"2024-09-11T08:16:03.001505+0000","last_clean":"2024-09-11T08:16:03.001505+0000","last_became_active":"2024-09-11T08:14:24.792279+0000","last_became_peered":"2024-09-11T08:14:24.792279+0000","last_unstale":"2024-09-11T08:16:03.001505+0000","last_undegraded":"2024-09-11T08:16:03.001505+0000","last_fullsized":"2024-09-11T08:16:03.001505+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:58.889094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:58.889094+0000","last_clean_scrub_stamp":"2024-09-11T08:14:58.889094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:16.725132+0000","scrub_duration":0.0040282030000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":131,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.987555+0000","last_change":"2024-09-11T08:14:58.796274+0000","last_active":"2024-09-11T08:16:02.987555+0000","last_peered":"2024-09-11T08:16:02.987555+0000","last_clean":"2024-09-11T08:16:02.987555+0000","last_became_active":"2024-09-11T08:14:24.792847+0000","last_became_peered":"2024-09-11T08:14:24.792847+0000","last_unstale":"2024-09-11T08:16:02.987555+0000","last_undegraded":"2024-09-11T08:16:02.987555+0000","last_fullsized":"2024-09-11T08:16:02.987555+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:58.796217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:58.796217+0000","last_clean_scrub_stamp":"2024-09-11T08:14:58.796217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:26.213711+0000","scrub_duration":0.0035819979999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"211'9901","reported_seq":9961,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:08.184963+0000","last_change":"2024-09-11T08:13:55.707851+0000","last_active":"2024-09-11T08:16:08.184963+0000","last_peered":"2024-09-11T08:16:08.184963+0000","last_clean":"2024-09-11T08:16:08.184963+0000","last_became_active":"2024-09-11T08:13:55.707556+0000","last_became_peered":"2024-09-11T08:13:55.707556+0000","last_unstale":"2024-09-11T08:16:08.184963+0000","last_undegraded":"2024-09-11T08:16:08.184963+0000","last_fullsized":"2024-09-11T08:16:08.184963+0000","mapping_epoch":159,"log_start":"185'4900","ondisk_log_start":"185'4900","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:13:53.862306+0000","objects_scrubbed":0,"log_size":5001,"log_dups_size":0,"ondisk_log_size":5001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400359424,"num_objects":6109,"num_object_clones":0,"num_object_copies":18327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9901,"num_write_kb":512320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":217,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.992307+0000","last_change":"2024-09-11T08:15:53.800579+0000","last_active":"2024-09-11T08:16:02.992307+0000","last_peered":"2024-09-11T08:16:02.992307+0000","last_clean":"2024-09-11T08:16:02.992307+0000","last_became_active":"2024-09-11T08:07:32.055407+0000","last_became_peered":"2024-09-11T08:07:32.055407+0000","last_unstale":"2024-09-11T08:16:02.992307+0000","last_undegraded":"2024-09-11T08:16:02.992307+0000","last_fullsized":"2024-09-11T08:16:02.992307+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-11T08:15:53.800494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:15:53.800494+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:54.617297+0000","scrub_duration":0.0070921359999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"59'1","reported_seq":208,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.992349+0000","last_change":"2024-09-11T08:15:51.807218+0000","last_active":"2024-09-11T08:16:02.992349+0000","last_peered":"2024-09-11T08:16:02.992349+0000","last_clean":"2024-09-11T08:16:02.992349+0000","last_became_active":"2024-09-11T08:07:30.050504+0000","last_became_peered":"2024-09-11T08:07:30.050504+0000","last_unstale":"2024-09-11T08:16:02.992349+0000","last_undegraded":"2024-09-11T08:16:02.992349+0000","last_fullsized":"2024-09-11T08:16:02.992349+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-11T08:15:51.806819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:51.806819+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:59.379579+0000","scrub_duration":0.012826908,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":495,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:06.055571+0000","last_change":"2024-09-11T08:15:04.793314+0000","last_active":"2024-09-11T08:16:06.055571+0000","last_peered":"2024-09-11T08:16:06.055571+0000","last_clean":"2024-09-11T08:16:06.055571+0000","last_became_active":"2024-09-11T08:14:24.793048+0000","last_became_peered":"2024-09-11T08:14:24.793048+0000","last_unstale":"2024-09-11T08:16:06.055571+0000","last_undegraded":"2024-09-11T08:16:06.055571+0000","last_fullsized":"2024-09-11T08:16:06.055571+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-11T08:15:04.793062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:04.793062+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:22.337286+0000","scrub_duration":0.0082004639999999993,"objects_trimmed":0,"snaptrim_duration":0.000318821,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":109,"num_read_kb":104,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":196,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.990872+0000","last_change":"2024-09-11T08:14:06.390698+0000","last_active":"2024-09-11T08:16:02.990872+0000","last_peered":"2024-09-11T08:16:02.990872+0000","last_clean":"2024-09-11T08:16:02.990872+0000","last_became_active":"2024-09-11T08:07:34.058962+0000","last_became_peered":"2024-09-11T08:07:34.058962+0000","last_unstale":"2024-09-11T08:16:02.990872+0000","last_undegraded":"2024-09-11T08:16:02.990872+0000","last_fullsized":"2024-09-11T08:16:02.990872+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:06.390632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:14:06.390632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017575690000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":205,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.999035+0000","last_change":"2024-09-11T08:15:53.506655+0000","last_active":"2024-09-11T08:16:02.999035+0000","last_peered":"2024-09-11T08:16:02.999035+0000","last_clean":"2024-09-11T08:16:02.999035+0000","last_became_active":"2024-09-11T08:07:36.077597+0000","last_became_peered":"2024-09-11T08:07:36.077597+0000","last_unstale":"2024-09-11T08:16:02.999035+0000","last_undegraded":"2024-09-11T08:16:02.999035+0000","last_fullsized":"2024-09-11T08:16:02.999035+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:53.506581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:15:53.506581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:15.129440+0000","scrub_duration":0.0028717209999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"211'10000","reported_seq":10377,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:07.820437+0000","last_change":"2024-09-11T08:15:47.174817+0000","last_active":"2024-09-11T08:16:07.820437+0000","last_peered":"2024-09-11T08:16:07.820437+0000","last_clean":"2024-09-11T08:16:07.820437+0000","last_became_active":"2024-09-11T08:13:54.882405+0000","last_became_peered":"2024-09-11T08:13:54.882405+0000","last_unstale":"2024-09-11T08:16:07.820437+0000","last_undegraded":"2024-09-11T08:16:07.820437+0000","last_fullsized":"2024-09-11T08:16:07.820437+0000","mapping_epoch":159,"log_start":"177'3900","ondisk_log_start":"177'3900","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'8519","last_scrub_stamp":"2024-09-11T08:15:47.174735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:15:47.174735+0000","objects_scrubbed":7692,"log_size":6100,"log_dups_size":0,"ondisk_log_size":6100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:12.130102+0000","scrub_duration":2.9660035420000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398589952,"num_objects":6082,"num_object_clones":0,"num_object_copies":18246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10000,"num_write_kb":514624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"84'21","reported_seq":244,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.995295+0000","last_change":"2024-09-11T08:14:51.029564+0000","last_active":"2024-09-11T08:16:02.995295+0000","last_peered":"2024-09-11T08:16:02.995295+0000","last_clean":"2024-09-11T08:16:02.995295+0000","last_became_active":"2024-09-11T08:14:24.789154+0000","last_became_peered":"2024-09-11T08:14:24.789154+0000","last_unstale":"2024-09-11T08:16:02.995295+0000","last_undegraded":"2024-09-11T08:16:02.995295+0000","last_fullsized":"2024-09-11T08:16:02.995295+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'21","last_scrub_stamp":"2024-09-11T08:14:51.029469+0000","last_deep_scrub":"84'21","last_deep_scrub_stamp":"2024-09-11T08:08:56.705306+0000","last_clean_scrub_stamp":"2024-09-11T08:14:51.029469+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001733975,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":191,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.992206+0000","last_change":"2024-09-11T08:15:06.236392+0000","last_active":"2024-09-11T08:16:02.992206+0000","last_peered":"2024-09-11T08:16:02.992206+0000","last_clean":"2024-09-11T08:16:02.992206+0000","last_became_active":"2024-09-11T08:07:30.047246+0000","last_became_peered":"2024-09-11T08:07:30.047246+0000","last_unstale":"2024-09-11T08:16:02.992206+0000","last_undegraded":"2024-09-11T08:16:02.992206+0000","last_fullsized":"2024-09-11T08:16:02.992206+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:06.236119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:06.236119+0000","last_clean_scrub_stamp":"2024-09-11T08:15:06.236119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:06.467298+0000","scrub_duration":0.0029121939999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":19,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.984548+0000","last_change":"2024-09-11T08:15:56.536262+0000","last_active":"2024-09-11T08:16:02.984548+0000","last_peered":"2024-09-11T08:16:02.984548+0000","last_clean":"2024-09-11T08:16:02.984548+0000","last_became_active":"2024-09-11T08:15:56.523430+0000","last_became_peered":"2024-09-11T08:15:56.523430+0000","last_unstale":"2024-09-11T08:16:02.984548+0000","last_undegraded":"2024-09-11T08:16:02.984548+0000","last_fullsized":"2024-09-11T08:16:02.984548+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-11T08:14:10.287968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:05:21.231856+0000","last_clean_scrub_stamp":"2024-09-11T08:14:10.287968+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023909769999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":13,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.986144+0000","last_change":"2024-09-11T08:15:56.529159+0000","last_active":"2024-09-11T08:16:02.986144+0000","last_peered":"2024-09-11T08:16:02.986144+0000","last_clean":"2024-09-11T08:16:02.986144+0000","last_became_active":"2024-09-11T08:15:56.528641+0000","last_became_peered":"2024-09-11T08:15:56.528641+0000","last_unstale":"2024-09-11T08:16:02.986144+0000","last_undegraded":"2024-09-11T08:16:02.986144+0000","last_fullsized":"2024-09-11T08:16:02.986144+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:55.140137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:51.289872+0000","last_clean_scrub_stamp":"2024-09-11T08:15:55.140137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:57.600999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":194,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.990810+0000","last_change":"2024-09-11T08:14:51.371546+0000","last_active":"2024-09-11T08:16:02.990810+0000","last_peered":"2024-09-11T08:16:02.990810+0000","last_clean":"2024-09-11T08:16:02.990810+0000","last_became_active":"2024-09-11T08:07:36.072508+0000","last_became_peered":"2024-09-11T08:07:36.072508+0000","last_unstale":"2024-09-11T08:16:02.990810+0000","last_undegraded":"2024-09-11T08:16:02.990810+0000","last_fullsized":"2024-09-11T08:16:02.990810+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:51.371488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:01.777429+0000","last_clean_scrub_stamp":"2024-09-11T08:14:51.371488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001247979,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"211'9899","reported_seq":9959,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:07.747937+0000","last_change":"2024-09-11T08:13:54.881747+0000","last_active":"2024-09-11T08:16:07.747937+0000","last_peered":"2024-09-11T08:16:07.747937+0000","last_clean":"2024-09-11T08:16:07.747937+0000","last_became_active":"2024-09-11T08:13:54.881423+0000","last_became_peered":"2024-09-11T08:13:54.881423+0000","last_unstale":"2024-09-11T08:16:07.747937+0000","last_undegraded":"2024-09-11T08:16:07.747937+0000","last_fullsized":"2024-09-11T08:16:07.747937+0000","mapping_epoch":159,"log_start":"185'5100","ondisk_log_start":"185'5100","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:13:53.862306+0000","objects_scrubbed":0,"log_size":4799,"log_dups_size":0,"ondisk_log_size":4799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404946944,"num_objects":6179,"num_object_clones":0,"num_object_copies":18537,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9899,"num_write_kb":514496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"88'16","reported_seq":213,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.986684+0000","last_change":"2024-09-11T08:12:55.025363+0000","last_active":"2024-09-11T08:16:02.986684+0000","last_peered":"2024-09-11T08:16:02.986684+0000","last_clean":"2024-09-11T08:16:02.986684+0000","last_became_active":"2024-09-11T08:07:32.047971+0000","last_became_peered":"2024-09-11T08:07:32.047971+0000","last_unstale":"2024-09-11T08:16:02.986684+0000","last_undegraded":"2024-09-11T08:16:02.986684+0000","last_fullsized":"2024-09-11T08:16:02.986684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'16","last_scrub_stamp":"2024-09-11T08:12:55.025272+0000","last_deep_scrub":"88'16","last_deep_scrub_stamp":"2024-09-11T08:09:39.635984+0000","last_clean_scrub_stamp":"2024-09-11T08:12:55.025272+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023373420000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":208,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.985685+0000","last_change":"2024-09-11T08:14:54.023658+0000","last_active":"2024-09-11T08:16:02.985685+0000","last_peered":"2024-09-11T08:16:02.985685+0000","last_clean":"2024-09-11T08:16:02.985685+0000","last_became_active":"2024-09-11T08:14:24.789888+0000","last_became_peered":"2024-09-11T08:14:24.789888+0000","last_unstale":"2024-09-11T08:16:02.985685+0000","last_undegraded":"2024-09-11T08:16:02.985685+0000","last_fullsized":"2024-09-11T08:16:02.985685+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:54.023556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:36.417911+0000","last_clean_scrub_stamp":"2024-09-11T08:14:54.023556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:09.994515+0000","scrub_duration":0.002726596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":196,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.990755+0000","last_change":"2024-09-11T08:15:06.434641+0000","last_active":"2024-09-11T08:16:02.990755+0000","last_peered":"2024-09-11T08:16:02.990755+0000","last_clean":"2024-09-11T08:16:02.990755+0000","last_became_active":"2024-09-11T08:07:34.053347+0000","last_became_peered":"2024-09-11T08:07:34.053347+0000","last_unstale":"2024-09-11T08:16:02.990755+0000","last_undegraded":"2024-09-11T08:16:02.990755+0000","last_fullsized":"2024-09-11T08:16:02.990755+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:06.434588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:01.708166+0000","last_clean_scrub_stamp":"2024-09-11T08:15:06.434588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:31.639996+0000","scrub_duration":0.0039794419999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":185,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.995699+0000","last_change":"2024-09-11T08:14:57.236619+0000","last_active":"2024-09-11T08:16:02.995699+0000","last_peered":"2024-09-11T08:16:02.995699+0000","last_clean":"2024-09-11T08:16:02.995699+0000","last_became_active":"2024-09-11T08:07:36.631687+0000","last_became_peered":"2024-09-11T08:07:36.631687+0000","last_unstale":"2024-09-11T08:16:02.995699+0000","last_undegraded":"2024-09-11T08:16:02.995699+0000","last_fullsized":"2024-09-11T08:16:02.995699+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:57.236432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:14:57.236432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029326859999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1","version":"211'9965","reported_seq":10308,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:08.159808+0000","last_change":"2024-09-11T08:15:57.075945+0000","last_active":"2024-09-11T08:16:08.159808+0000","last_peered":"2024-09-11T08:16:08.159808+0000","last_clean":"2024-09-11T08:16:08.159808+0000","last_became_active":"2024-09-11T08:13:55.707503+0000","last_became_peered":"2024-09-11T08:13:55.707503+0000","last_unstale":"2024-09-11T08:16:08.159808+0000","last_undegraded":"2024-09-11T08:16:08.159808+0000","last_fullsized":"2024-09-11T08:16:08.159808+0000","mapping_epoch":159,"log_start":"185'4900","ondisk_log_start":"185'4900","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'9162","last_scrub_stamp":"2024-09-11T08:15:57.075870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:15:57.075870+0000","objects_scrubbed":6840,"log_size":5065,"log_dups_size":0,"ondisk_log_size":5065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:10.709887+0000","scrub_duration":0.63951687599999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":395640832,"num_objects":6037,"num_object_clones":0,"num_object_copies":18111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9965,"num_write_kb":512064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"77'25","reported_seq":263,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.985658+0000","last_change":"2024-09-11T08:15:42.061711+0000","last_active":"2024-09-11T08:16:02.985658+0000","last_peered":"2024-09-11T08:16:02.985658+0000","last_clean":"2024-09-11T08:16:02.985658+0000","last_became_active":"2024-09-11T08:09:08.106771+0000","last_became_peered":"2024-09-11T08:09:08.106771+0000","last_unstale":"2024-09-11T08:16:02.985658+0000","last_undegraded":"2024-09-11T08:16:02.985658+0000","last_fullsized":"2024-09-11T08:16:02.985658+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'25","last_scrub_stamp":"2024-09-11T08:15:42.061632+0000","last_deep_scrub":"77'25","last_deep_scrub_stamp":"2024-09-11T08:08:32.458210+0000","last_clean_scrub_stamp":"2024-09-11T08:15:42.061632+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:06.943716+0000","scrub_duration":0.00500454,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":200,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.994722+0000","last_change":"2024-09-11T08:15:00.050885+0000","last_active":"2024-09-11T08:16:02.994722+0000","last_peered":"2024-09-11T08:16:02.994722+0000","last_clean":"2024-09-11T08:16:02.994722+0000","last_became_active":"2024-09-11T08:07:30.042555+0000","last_became_peered":"2024-09-11T08:07:30.042555+0000","last_unstale":"2024-09-11T08:16:02.994722+0000","last_undegraded":"2024-09-11T08:16:02.994722+0000","last_fullsized":"2024-09-11T08:16:02.994722+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:00.050803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:00.050803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:08.088041+0000","scrub_duration":0.0043077549999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":197,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.999107+0000","last_change":"2024-09-11T08:14:00.985127+0000","last_active":"2024-09-11T08:16:02.999107+0000","last_peered":"2024-09-11T08:16:02.999107+0000","last_clean":"2024-09-11T08:16:02.999107+0000","last_became_active":"2024-09-11T08:07:34.054961+0000","last_became_peered":"2024-09-11T08:07:34.054961+0000","last_unstale":"2024-09-11T08:16:02.999107+0000","last_undegraded":"2024-09-11T08:16:02.999107+0000","last_fullsized":"2024-09-11T08:16:02.999107+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:00.985075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:14:00.985075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015093109999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":214,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.998472+0000","last_change":"2024-09-11T08:15:43.160119+0000","last_active":"2024-09-11T08:16:02.998472+0000","last_peered":"2024-09-11T08:16:02.998472+0000","last_clean":"2024-09-11T08:16:02.998472+0000","last_became_active":"2024-09-11T08:07:36.071194+0000","last_became_peered":"2024-09-11T08:07:36.071194+0000","last_unstale":"2024-09-11T08:16:02.998472+0000","last_undegraded":"2024-09-11T08:16:02.998472+0000","last_fullsized":"2024-09-11T08:16:02.998472+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:43.158812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:07.930137+0000","last_clean_scrub_stamp":"2024-09-11T08:15:43.158812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:00.399779+0000","scrub_duration":0.0051752609999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"211'10042","reported_seq":10101,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:08.054727+0000","last_change":"2024-09-11T08:13:54.882472+0000","last_active":"2024-09-11T08:16:08.054727+0000","last_peered":"2024-09-11T08:16:08.054727+0000","last_clean":"2024-09-11T08:16:08.054727+0000","last_became_active":"2024-09-11T08:13:54.882196+0000","last_became_peered":"2024-09-11T08:13:54.882196+0000","last_unstale":"2024-09-11T08:16:08.054727+0000","last_undegraded":"2024-09-11T08:16:08.054727+0000","last_fullsized":"2024-09-11T08:16:08.054727+0000","mapping_epoch":159,"log_start":"185'5235","ondisk_log_start":"185'5235","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:13:53.862306+0000","objects_scrubbed":0,"log_size":4807,"log_dups_size":0,"ondisk_log_size":4807,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":397672448,"num_objects":6068,"num_object_clones":0,"num_object_copies":18204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10042,"num_write_kb":515520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":204,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.985927+0000","last_change":"2024-09-11T08:13:07.139492+0000","last_active":"2024-09-11T08:16:02.985927+0000","last_peered":"2024-09-11T08:16:02.985927+0000","last_clean":"2024-09-11T08:16:02.985927+0000","last_became_active":"2024-09-11T08:07:32.043314+0000","last_became_peered":"2024-09-11T08:07:32.043314+0000","last_unstale":"2024-09-11T08:16:02.985927+0000","last_undegraded":"2024-09-11T08:16:02.985927+0000","last_fullsized":"2024-09-11T08:16:02.985927+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-11T08:13:07.139430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:13:07.139430+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0075406340000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":201,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.993220+0000","last_change":"2024-09-11T08:15:48.405028+0000","last_active":"2024-09-11T08:16:02.993220+0000","last_peered":"2024-09-11T08:16:02.993220+0000","last_clean":"2024-09-11T08:16:02.993220+0000","last_became_active":"2024-09-11T08:07:30.046684+0000","last_became_peered":"2024-09-11T08:07:30.046684+0000","last_unstale":"2024-09-11T08:16:02.993220+0000","last_undegraded":"2024-09-11T08:16:02.993220+0000","last_fullsized":"2024-09-11T08:16:02.993220+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:48.404853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:48.404853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:01.459915+0000","scrub_duration":0.0048699770000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"171'12","reported_seq":331,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:03.651289+0000","last_change":"2024-09-11T08:14:56.881397+0000","last_active":"2024-09-11T08:16:03.651289+0000","last_peered":"2024-09-11T08:16:03.651289+0000","last_clean":"2024-09-11T08:16:03.651289+0000","last_became_active":"2024-09-11T08:14:24.794446+0000","last_became_peered":"2024-09-11T08:14:24.794446+0000","last_unstale":"2024-09-11T08:16:03.651289+0000","last_undegraded":"2024-09-11T08:16:03.651289+0000","last_fullsized":"2024-09-11T08:16:03.651289+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'12","last_scrub_stamp":"2024-09-11T08:14:56.881299+0000","last_deep_scrub":"171'12","last_deep_scrub_stamp":"2024-09-11T08:14:56.881299+0000","last_clean_scrub_stamp":"2024-09-11T08:14:56.881299+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:13.287491+0000","scrub_duration":0.0073519259999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":185,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.986228+0000","last_change":"2024-09-11T08:14:13.941569+0000","last_active":"2024-09-11T08:16:02.986228+0000","last_peered":"2024-09-11T08:16:02.986228+0000","last_clean":"2024-09-11T08:16:02.986228+0000","last_became_active":"2024-09-11T08:07:36.074684+0000","last_became_peered":"2024-09-11T08:07:36.074684+0000","last_unstale":"2024-09-11T08:16:02.986228+0000","last_undegraded":"2024-09-11T08:16:02.986228+0000","last_fullsized":"2024-09-11T08:16:02.986228+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:13.941492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:14.613532+0000","last_clean_scrub_stamp":"2024-09-11T08:14:13.941492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037661470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.7","version":"211'9939","reported_seq":10397,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:08.154377+0000","last_change":"2024-09-11T08:15:56.302099+0000","last_active":"2024-09-11T08:16:08.154377+0000","last_peered":"2024-09-11T08:16:08.154377+0000","last_clean":"2024-09-11T08:16:08.154377+0000","last_became_active":"2024-09-11T08:13:54.883624+0000","last_became_peered":"2024-09-11T08:13:54.883624+0000","last_unstale":"2024-09-11T08:16:08.154377+0000","last_undegraded":"2024-09-11T08:16:08.154377+0000","last_fullsized":"2024-09-11T08:16:08.154377+0000","mapping_epoch":159,"log_start":"184'4900","ondisk_log_start":"184'4900","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'9119","last_scrub_stamp":"2024-09-11T08:15:56.302032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:15:56.302032+0000","objects_scrubbed":6919,"log_size":5039,"log_dups_size":0,"ondisk_log_size":5039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:02.822566+0000","scrub_duration":0.82679451100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399704064,"num_objects":6099,"num_object_clones":0,"num_object_copies":18297,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9939,"num_write_kb":513216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":205,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.991199+0000","last_change":"2024-09-11T08:13:14.171710+0000","last_active":"2024-09-11T08:16:02.991199+0000","last_peered":"2024-09-11T08:16:02.991199+0000","last_clean":"2024-09-11T08:16:02.991199+0000","last_became_active":"2024-09-11T08:07:32.054885+0000","last_became_peered":"2024-09-11T08:07:32.054885+0000","last_unstale":"2024-09-11T08:16:02.991199+0000","last_undegraded":"2024-09-11T08:16:02.991199+0000","last_fullsized":"2024-09-11T08:16:02.991199+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:13:14.171623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:13:14.171623+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.01042096,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":210,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.993388+0000","last_change":"2024-09-11T08:15:34.497175+0000","last_active":"2024-09-11T08:16:02.993388+0000","last_peered":"2024-09-11T08:16:02.993388+0000","last_clean":"2024-09-11T08:16:02.993388+0000","last_became_active":"2024-09-11T08:07:30.045613+0000","last_became_peered":"2024-09-11T08:07:30.045613+0000","last_unstale":"2024-09-11T08:16:02.993388+0000","last_undegraded":"2024-09-11T08:16:02.993388+0000","last_fullsized":"2024-09-11T08:16:02.993388+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:34.497125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:34.497125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:45.232704+0000","scrub_duration":0.0052802250000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":216,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.997519+0000","last_change":"2024-09-11T08:15:37.123094+0000","last_active":"2024-09-11T08:16:02.997519+0000","last_peered":"2024-09-11T08:16:02.997519+0000","last_clean":"2024-09-11T08:16:02.997519+0000","last_became_active":"2024-09-11T08:07:34.056606+0000","last_became_peered":"2024-09-11T08:07:34.056606+0000","last_unstale":"2024-09-11T08:16:02.997519+0000","last_undegraded":"2024-09-11T08:16:02.997519+0000","last_fullsized":"2024-09-11T08:16:02.997519+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:37.123010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:15:37.123010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:05.989949+0000","scrub_duration":0.0042949169999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":212,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.992268+0000","last_change":"2024-09-11T08:15:56.520006+0000","last_active":"2024-09-11T08:16:02.992268+0000","last_peered":"2024-09-11T08:16:02.992268+0000","last_clean":"2024-09-11T08:16:02.992268+0000","last_became_active":"2024-09-11T08:15:56.518884+0000","last_became_peered":"2024-09-11T08:15:56.518884+0000","last_unstale":"2024-09-11T08:16:02.992268+0000","last_undegraded":"2024-09-11T08:16:02.992268+0000","last_fullsized":"2024-09-11T08:16:02.992268+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:01.376411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:14:01.376411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012067320000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"211'10126","reported_seq":14327,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:08.210642+0000","last_change":"2024-09-11T08:14:51.900945+0000","last_active":"2024-09-11T08:16:08.210642+0000","last_peered":"2024-09-11T08:16:08.210642+0000","last_clean":"2024-09-11T08:16:08.210642+0000","last_became_active":"2024-09-11T08:14:51.879122+0000","last_became_peered":"2024-09-11T08:14:51.879122+0000","last_unstale":"2024-09-11T08:16:08.210642+0000","last_undegraded":"2024-09-11T08:16:08.210642+0000","last_fullsized":"2024-09-11T08:16:08.210642+0000","mapping_epoch":180,"log_start":"181'4300","ondisk_log_start":"181'4300","created":159,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:13:53.862306+0000","objects_scrubbed":0,"log_size":5826,"log_dups_size":0,"ondisk_log_size":5826,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":408289280,"num_objects":6230,"num_object_clones":0,"num_object_copies":18690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10126,"num_write_kb":523392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2061,"num_bytes_recovered":135069696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":306,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.994725+0000","last_change":"2024-09-11T08:15:56.551439+0000","last_active":"2024-09-11T08:16:02.994725+0000","last_peered":"2024-09-11T08:16:02.994725+0000","last_clean":"2024-09-11T08:16:02.994725+0000","last_became_active":"2024-09-11T08:15:56.526961+0000","last_became_peered":"2024-09-11T08:15:56.526961+0000","last_unstale":"2024-09-11T08:16:02.994725+0000","last_undegraded":"2024-09-11T08:16:02.994725+0000","last_fullsized":"2024-09-11T08:16:02.994725+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-11T08:15:49.187186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:15:49.187186+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:17.036763+0000","scrub_duration":0.0054603430000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.987408+0000","last_change":"2024-09-11T08:15:56.530685+0000","last_active":"2024-09-11T08:16:02.987408+0000","last_peered":"2024-09-11T08:16:02.987408+0000","last_clean":"2024-09-11T08:16:02.987408+0000","last_became_active":"2024-09-11T08:15:56.530370+0000","last_became_peered":"2024-09-11T08:15:56.530370+0000","last_unstale":"2024-09-11T08:16:02.987408+0000","last_undegraded":"2024-09-11T08:16:02.987408+0000","last_fullsized":"2024-09-11T08:16:02.987408+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:09.433494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:44.385372+0000","last_clean_scrub_stamp":"2024-09-11T08:14:09.433494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":209,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.997833+0000","last_change":"2024-09-11T08:14:59.900979+0000","last_active":"2024-09-11T08:16:02.997833+0000","last_peered":"2024-09-11T08:16:02.997833+0000","last_clean":"2024-09-11T08:16:02.997833+0000","last_became_active":"2024-09-11T08:07:34.063589+0000","last_became_peered":"2024-09-11T08:07:34.063589+0000","last_unstale":"2024-09-11T08:16:02.997833+0000","last_undegraded":"2024-09-11T08:16:02.997833+0000","last_fullsized":"2024-09-11T08:16:02.997833+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:59.900898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:14:59.900898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:26.844256+0000","scrub_duration":0.006061121,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":13,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.984630+0000","last_change":"2024-09-11T08:15:56.518476+0000","last_active":"2024-09-11T08:16:02.984630+0000","last_peered":"2024-09-11T08:16:02.984630+0000","last_clean":"2024-09-11T08:16:02.984630+0000","last_became_active":"2024-09-11T08:15:56.518169+0000","last_became_peered":"2024-09-11T08:15:56.518169+0000","last_unstale":"2024-09-11T08:16:02.984630+0000","last_undegraded":"2024-09-11T08:16:02.984630+0000","last_fullsized":"2024-09-11T08:16:02.984630+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:32.131008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:15:32.131008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:59.362244+0000","scrub_duration":0.0064196519999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"211'10169","reported_seq":14937,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:07.824506+0000","last_change":"2024-09-11T08:15:38.333949+0000","last_active":"2024-09-11T08:16:07.824506+0000","last_peered":"2024-09-11T08:16:07.824506+0000","last_clean":"2024-09-11T08:16:07.824506+0000","last_became_active":"2024-09-11T08:14:50.875228+0000","last_became_peered":"2024-09-11T08:14:50.875228+0000","last_unstale":"2024-09-11T08:16:07.824506+0000","last_undegraded":"2024-09-11T08:16:07.824506+0000","last_fullsized":"2024-09-11T08:16:07.824506+0000","mapping_epoch":179,"log_start":"177'4076","ondisk_log_start":"177'4076","created":159,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"200'7896","last_scrub_stamp":"2024-09-11T08:15:38.333891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:15:38.333891+0000","objects_scrubbed":7806,"log_size":6093,"log_dups_size":0,"ondisk_log_size":6093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:43.473904+0000","scrub_duration":2.12463202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402325504,"num_objects":6139,"num_object_clones":0,"num_object_copies":18417,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10169,"num_write_kb":521856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2184,"num_bytes_recovered":143130624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"145'17","reported_seq":227,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.985760+0000","last_change":"2024-09-11T08:15:53.040339+0000","last_active":"2024-09-11T08:16:02.985760+0000","last_peered":"2024-09-11T08:16:02.985760+0000","last_clean":"2024-09-11T08:16:02.985760+0000","last_became_active":"2024-09-11T08:07:32.048164+0000","last_became_peered":"2024-09-11T08:07:32.048164+0000","last_unstale":"2024-09-11T08:16:02.985760+0000","last_undegraded":"2024-09-11T08:16:02.985760+0000","last_fullsized":"2024-09-11T08:16:02.985760+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'17","last_scrub_stamp":"2024-09-11T08:15:53.040223+0000","last_deep_scrub":"145'17","last_deep_scrub_stamp":"2024-09-11T08:13:08.160848+0000","last_clean_scrub_stamp":"2024-09-11T08:15:53.040223+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:12.632932+0000","scrub_duration":0.0039160820000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.986501+0000","last_change":"2024-09-11T08:15:56.526162+0000","last_active":"2024-09-11T08:16:02.986501+0000","last_peered":"2024-09-11T08:16:02.986501+0000","last_clean":"2024-09-11T08:16:02.986501+0000","last_became_active":"2024-09-11T08:15:56.525845+0000","last_became_peered":"2024-09-11T08:15:56.525845+0000","last_unstale":"2024-09-11T08:16:02.986501+0000","last_undegraded":"2024-09-11T08:16:02.986501+0000","last_fullsized":"2024-09-11T08:16:02.986501+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:12.966188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:53.247275+0000","last_clean_scrub_stamp":"2024-09-11T08:14:12.966188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":196,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.996770+0000","last_change":"2024-09-11T08:14:01.899688+0000","last_active":"2024-09-11T08:16:02.996770+0000","last_peered":"2024-09-11T08:16:02.996770+0000","last_clean":"2024-09-11T08:16:02.996770+0000","last_became_active":"2024-09-11T08:07:34.057479+0000","last_became_peered":"2024-09-11T08:07:34.057479+0000","last_unstale":"2024-09-11T08:16:02.996770+0000","last_undegraded":"2024-09-11T08:16:02.996770+0000","last_fullsized":"2024-09-11T08:16:02.996770+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:01.899633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:14:01.899633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011221180000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":212,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.992969+0000","last_change":"2024-09-11T08:15:56.522949+0000","last_active":"2024-09-11T08:16:02.992969+0000","last_peered":"2024-09-11T08:16:02.992969+0000","last_clean":"2024-09-11T08:16:02.992969+0000","last_became_active":"2024-09-11T08:15:56.522624+0000","last_became_peered":"2024-09-11T08:15:56.522624+0000","last_unstale":"2024-09-11T08:16:02.992969+0000","last_undegraded":"2024-09-11T08:16:02.992969+0000","last_fullsized":"2024-09-11T08:16:02.992969+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:54.938695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:13:54.938695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016854260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"211'10038","reported_seq":10399,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:07.842365+0000","last_change":"2024-09-11T08:15:53.462987+0000","last_active":"2024-09-11T08:16:07.842365+0000","last_peered":"2024-09-11T08:16:07.842365+0000","last_clean":"2024-09-11T08:16:07.842365+0000","last_became_active":"2024-09-11T08:13:55.705819+0000","last_became_peered":"2024-09-11T08:13:55.705819+0000","last_unstale":"2024-09-11T08:16:07.842365+0000","last_undegraded":"2024-09-11T08:16:07.842365+0000","last_fullsized":"2024-09-11T08:16:07.842365+0000","mapping_epoch":159,"log_start":"183'4700","ondisk_log_start":"183'4700","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"206'9066","last_scrub_stamp":"2024-09-11T08:15:53.462907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:15:53.462907+0000","objects_scrubbed":7268,"log_size":5338,"log_dups_size":0,"ondisk_log_size":5338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:18.668302+0000","scrub_duration":2.0133290810000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":408420352,"num_objects":6232,"num_object_clones":0,"num_object_copies":18696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10038,"num_write_kb":520640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":282,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.990335+0000","last_change":"2024-09-11T08:15:56.551197+0000","last_active":"2024-09-11T08:16:02.990335+0000","last_peered":"2024-09-11T08:16:02.990335+0000","last_clean":"2024-09-11T08:16:02.990335+0000","last_became_active":"2024-09-11T08:15:56.532431+0000","last_became_peered":"2024-09-11T08:15:56.532431+0000","last_unstale":"2024-09-11T08:16:02.990335+0000","last_undegraded":"2024-09-11T08:16:02.990335+0000","last_fullsized":"2024-09-11T08:16:02.990335+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-11T08:15:47.427884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:15:47.427884+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:52.822361+0000","scrub_duration":0.0016881800000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":19,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.985977+0000","last_change":"2024-09-11T08:15:56.538509+0000","last_active":"2024-09-11T08:16:02.985977+0000","last_peered":"2024-09-11T08:16:02.985977+0000","last_clean":"2024-09-11T08:16:02.985977+0000","last_became_active":"2024-09-11T08:15:56.515328+0000","last_became_peered":"2024-09-11T08:15:56.515328+0000","last_unstale":"2024-09-11T08:16:02.985977+0000","last_undegraded":"2024-09-11T08:16:02.985977+0000","last_fullsized":"2024-09-11T08:16:02.985977+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-11T08:13:59.909695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:13:59.909695+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"209'11","reported_seq":19,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:03.651272+0000","last_change":"2024-09-11T08:15:56.534905+0000","last_active":"2024-09-11T08:16:03.651272+0000","last_peered":"2024-09-11T08:16:03.651272+0000","last_clean":"2024-09-11T08:16:03.651272+0000","last_became_active":"2024-09-11T08:15:56.524513+0000","last_became_peered":"2024-09-11T08:15:56.524513+0000","last_unstale":"2024-09-11T08:16:03.651272+0000","last_undegraded":"2024-09-11T08:16:03.651272+0000","last_fullsized":"2024-09-11T08:16:03.651272+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"156'10","last_scrub_stamp":"2024-09-11T08:15:38.215399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:15:38.215399+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:00.740762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":194,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.990535+0000","last_change":"2024-09-11T08:15:03.484853+0000","last_active":"2024-09-11T08:16:02.990535+0000","last_peered":"2024-09-11T08:16:02.990535+0000","last_clean":"2024-09-11T08:16:02.990535+0000","last_became_active":"2024-09-11T08:07:36.631755+0000","last_became_peered":"2024-09-11T08:07:36.631755+0000","last_unstale":"2024-09-11T08:16:02.990535+0000","last_undegraded":"2024-09-11T08:16:02.990535+0000","last_fullsized":"2024-09-11T08:16:02.990535+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:03.484755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:03.484755+0000","last_clean_scrub_stamp":"2024-09-11T08:15:03.484755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:05.977841+0000","scrub_duration":0.0064296900000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":210,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.994454+0000","last_change":"2024-09-11T08:15:31.257899+0000","last_active":"2024-09-11T08:16:02.994454+0000","last_peered":"2024-09-11T08:16:02.994454+0000","last_clean":"2024-09-11T08:16:02.994454+0000","last_became_active":"2024-09-11T08:07:30.045910+0000","last_became_peered":"2024-09-11T08:07:30.045910+0000","last_unstale":"2024-09-11T08:16:02.994454+0000","last_undegraded":"2024-09-11T08:16:02.994454+0000","last_fullsized":"2024-09-11T08:16:02.994454+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:31.257821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:31.257821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:56.144543+0000","scrub_duration":0.0040247779999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"84'9","reported_seq":244,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.995659+0000","last_change":"2024-09-11T08:15:56.550790+0000","last_active":"2024-09-11T08:16:02.995659+0000","last_peered":"2024-09-11T08:16:02.995659+0000","last_clean":"2024-09-11T08:16:02.995659+0000","last_became_active":"2024-09-11T08:15:56.526325+0000","last_became_peered":"2024-09-11T08:15:56.526325+0000","last_unstale":"2024-09-11T08:16:02.995659+0000","last_undegraded":"2024-09-11T08:16:02.995659+0000","last_fullsized":"2024-09-11T08:16:02.995659+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-09-11T08:14:12.071950+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-09-11T08:09:03.418457+0000","last_clean_scrub_stamp":"2024-09-11T08:14:12.071950+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0068954059999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":196,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.990631+0000","last_change":"2024-09-11T08:15:02.463928+0000","last_active":"2024-09-11T08:16:02.990631+0000","last_peered":"2024-09-11T08:16:02.990631+0000","last_clean":"2024-09-11T08:16:02.990631+0000","last_became_active":"2024-09-11T08:07:34.057774+0000","last_became_peered":"2024-09-11T08:07:34.057774+0000","last_unstale":"2024-09-11T08:16:02.990631+0000","last_undegraded":"2024-09-11T08:16:02.990631+0000","last_fullsized":"2024-09-11T08:16:02.990631+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:02.463848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:02.463848+0000","last_clean_scrub_stamp":"2024-09-11T08:15:02.463848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034811849999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":138,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.987483+0000","last_change":"2024-09-11T08:14:56.815770+0000","last_active":"2024-09-11T08:16:02.987483+0000","last_peered":"2024-09-11T08:16:02.987483+0000","last_clean":"2024-09-11T08:16:02.987483+0000","last_became_active":"2024-09-11T08:14:24.790678+0000","last_became_peered":"2024-09-11T08:14:24.790678+0000","last_unstale":"2024-09-11T08:16:02.987483+0000","last_undegraded":"2024-09-11T08:16:02.987483+0000","last_fullsized":"2024-09-11T08:16:02.987483+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:56.815690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:14:56.815690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:09.682817+0000","scrub_duration":0.0016768309999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"84'13","reported_seq":256,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.992571+0000","last_change":"2024-09-11T08:15:56.532882+0000","last_active":"2024-09-11T08:16:02.992571+0000","last_peered":"2024-09-11T08:16:02.992571+0000","last_clean":"2024-09-11T08:16:02.992571+0000","last_became_active":"2024-09-11T08:15:56.514793+0000","last_became_peered":"2024-09-11T08:15:56.514793+0000","last_unstale":"2024-09-11T08:16:02.992571+0000","last_undegraded":"2024-09-11T08:16:02.992571+0000","last_fullsized":"2024-09-11T08:16:02.992571+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'13","last_scrub_stamp":"2024-09-11T08:14:05.339509+0000","last_deep_scrub":"84'13","last_deep_scrub_stamp":"2024-09-11T08:08:57.835634+0000","last_clean_scrub_stamp":"2024-09-11T08:14:05.339509+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001614047,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":171,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.984830+0000","last_change":"2024-09-11T08:15:31.534404+0000","last_active":"2024-09-11T08:16:02.984830+0000","last_peered":"2024-09-11T08:16:02.984830+0000","last_clean":"2024-09-11T08:16:02.984830+0000","last_became_active":"2024-09-11T08:14:24.786021+0000","last_became_peered":"2024-09-11T08:14:24.786021+0000","last_unstale":"2024-09-11T08:16:02.984830+0000","last_undegraded":"2024-09-11T08:16:02.984830+0000","last_fullsized":"2024-09-11T08:16:02.984830+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:31.534302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:30.471620+0000","last_clean_scrub_stamp":"2024-09-11T08:15:31.534302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:43.916804+0000","scrub_duration":0.0021837200000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":214,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.997429+0000","last_change":"2024-09-11T08:15:56.526834+0000","last_active":"2024-09-11T08:16:02.997429+0000","last_peered":"2024-09-11T08:16:02.997429+0000","last_clean":"2024-09-11T08:16:02.997429+0000","last_became_active":"2024-09-11T08:15:56.525717+0000","last_became_peered":"2024-09-11T08:15:56.525717+0000","last_unstale":"2024-09-11T08:16:02.997429+0000","last_undegraded":"2024-09-11T08:16:02.997429+0000","last_fullsized":"2024-09-11T08:16:02.997429+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:59.988381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:04.370734+0000","last_clean_scrub_stamp":"2024-09-11T08:13:59.988381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017033700000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":221,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.997691+0000","last_change":"2024-09-11T08:15:56.525429+0000","last_active":"2024-09-11T08:16:02.997691+0000","last_peered":"2024-09-11T08:16:02.997691+0000","last_clean":"2024-09-11T08:16:02.997691+0000","last_became_active":"2024-09-11T08:15:56.525120+0000","last_became_peered":"2024-09-11T08:15:56.525120+0000","last_unstale":"2024-09-11T08:16:02.997691+0000","last_undegraded":"2024-09-11T08:16:02.997691+0000","last_fullsized":"2024-09-11T08:16:02.997691+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:54.184372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:54.184372+0000","last_clean_scrub_stamp":"2024-09-11T08:15:54.184372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:21.919337+0000","scrub_duration":0.013922767000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":271,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.991360+0000","last_change":"2024-09-11T08:15:56.547635+0000","last_active":"2024-09-11T08:16:02.991360+0000","last_peered":"2024-09-11T08:16:02.991360+0000","last_clean":"2024-09-11T08:16:02.991360+0000","last_became_active":"2024-09-11T08:15:56.532513+0000","last_became_peered":"2024-09-11T08:15:56.532513+0000","last_unstale":"2024-09-11T08:16:02.991360+0000","last_undegraded":"2024-09-11T08:16:02.991360+0000","last_fullsized":"2024-09-11T08:16:02.991360+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:15:52.443189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:15:52.443189+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:53.586813+0000","scrub_duration":0.010882618,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":218,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.991909+0000","last_change":"2024-09-11T08:15:56.513604+0000","last_active":"2024-09-11T08:16:02.991909+0000","last_peered":"2024-09-11T08:16:02.991909+0000","last_clean":"2024-09-11T08:16:02.991909+0000","last_became_active":"2024-09-11T08:15:56.513278+0000","last_became_peered":"2024-09-11T08:15:56.513278+0000","last_unstale":"2024-09-11T08:16:02.991909+0000","last_undegraded":"2024-09-11T08:16:02.991909+0000","last_fullsized":"2024-09-11T08:16:02.991909+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:48.213172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:48.935387+0000","last_clean_scrub_stamp":"2024-09-11T08:13:48.213172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011628039999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"209'11","reported_seq":334,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:03.651450+0000","last_change":"2024-09-11T08:15:56.543025+0000","last_active":"2024-09-11T08:16:03.651450+0000","last_peered":"2024-09-11T08:16:03.651450+0000","last_clean":"2024-09-11T08:16:03.651450+0000","last_became_active":"2024-09-11T08:15:56.525253+0000","last_became_peered":"2024-09-11T08:15:56.525253+0000","last_unstale":"2024-09-11T08:16:03.651450+0000","last_undegraded":"2024-09-11T08:16:03.651450+0000","last_fullsized":"2024-09-11T08:16:03.651450+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"156'10","last_scrub_stamp":"2024-09-11T08:14:01.979463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:14:01.979463+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012487329999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":185,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.984948+0000","last_change":"2024-09-11T08:15:05.867100+0000","last_active":"2024-09-11T08:16:02.984948+0000","last_peered":"2024-09-11T08:16:02.984948+0000","last_clean":"2024-09-11T08:16:02.984948+0000","last_became_active":"2024-09-11T08:07:36.068493+0000","last_became_peered":"2024-09-11T08:07:36.068493+0000","last_unstale":"2024-09-11T08:16:02.984948+0000","last_undegraded":"2024-09-11T08:16:02.984948+0000","last_fullsized":"2024-09-11T08:16:02.984948+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:05.867013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:05.867013+0000","last_clean_scrub_stamp":"2024-09-11T08:15:05.867013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:11.060549+0000","scrub_duration":0.0061248279999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":243,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.996168+0000","last_change":"2024-09-11T08:14:56.223381+0000","last_active":"2024-09-11T08:16:02.996168+0000","last_peered":"2024-09-11T08:16:02.996168+0000","last_clean":"2024-09-11T08:16:02.996168+0000","last_became_active":"2024-09-11T08:14:24.791187+0000","last_became_peered":"2024-09-11T08:14:24.791187+0000","last_unstale":"2024-09-11T08:16:02.996168+0000","last_undegraded":"2024-09-11T08:16:02.996168+0000","last_fullsized":"2024-09-11T08:16:02.996168+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-11T08:14:56.222082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:14:56.222082+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:21.292272+0000","scrub_duration":0.011605691,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.985374+0000","last_change":"2024-09-11T08:15:56.521890+0000","last_active":"2024-09-11T08:16:02.985374+0000","last_peered":"2024-09-11T08:16:02.985374+0000","last_clean":"2024-09-11T08:16:02.985374+0000","last_became_active":"2024-09-11T08:15:56.518515+0000","last_became_peered":"2024-09-11T08:15:56.518515+0000","last_unstale":"2024-09-11T08:16:02.985374+0000","last_undegraded":"2024-09-11T08:16:02.985374+0000","last_fullsized":"2024-09-11T08:16:02.985374+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:33.021088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:33.021088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:40.824210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":307,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:03.651429+0000","last_change":"2024-09-11T08:15:44.429144+0000","last_active":"2024-09-11T08:16:03.651429+0000","last_peered":"2024-09-11T08:16:03.651429+0000","last_clean":"2024-09-11T08:16:03.651429+0000","last_became_active":"2024-09-11T08:07:34.060736+0000","last_became_peered":"2024-09-11T08:07:34.060736+0000","last_unstale":"2024-09-11T08:16:03.651429+0000","last_undegraded":"2024-09-11T08:16:03.651429+0000","last_fullsized":"2024-09-11T08:16:03.651429+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:15:44.429062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:15:44.429062+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:58.251305+0000","scrub_duration":0.0062457210000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":212,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.992014+0000","last_change":"2024-09-11T08:15:56.515344+0000","last_active":"2024-09-11T08:16:02.992014+0000","last_peered":"2024-09-11T08:16:02.992014+0000","last_clean":"2024-09-11T08:16:02.992014+0000","last_became_active":"2024-09-11T08:15:56.513949+0000","last_became_peered":"2024-09-11T08:15:56.513949+0000","last_unstale":"2024-09-11T08:16:02.992014+0000","last_undegraded":"2024-09-11T08:16:02.992014+0000","last_fullsized":"2024-09-11T08:16:02.992014+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:01.448261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:15:01.448261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:25.797475+0000","scrub_duration":0.021382708,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":208,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.984887+0000","last_change":"2024-09-11T08:13:13.047503+0000","last_active":"2024-09-11T08:16:02.984887+0000","last_peered":"2024-09-11T08:16:02.984887+0000","last_clean":"2024-09-11T08:16:02.984887+0000","last_became_active":"2024-09-11T08:07:32.047981+0000","last_became_peered":"2024-09-11T08:07:32.047981+0000","last_unstale":"2024-09-11T08:16:02.984887+0000","last_undegraded":"2024-09-11T08:16:02.984887+0000","last_fullsized":"2024-09-11T08:16:02.984887+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-11T08:13:13.047436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:13:13.047436+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012804688,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":218,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.986057+0000","last_change":"2024-09-11T08:15:56.523858+0000","last_active":"2024-09-11T08:16:02.986057+0000","last_peered":"2024-09-11T08:16:02.986057+0000","last_clean":"2024-09-11T08:16:02.986057+0000","last_became_active":"2024-09-11T08:15:56.523574+0000","last_became_peered":"2024-09-11T08:15:56.523574+0000","last_unstale":"2024-09-11T08:16:02.986057+0000","last_undegraded":"2024-09-11T08:16:02.986057+0000","last_fullsized":"2024-09-11T08:16:02.986057+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:55.845422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:13:55.845422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00143399,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":200,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.999531+0000","last_change":"2024-09-11T08:15:38.410767+0000","last_active":"2024-09-11T08:16:02.999531+0000","last_peered":"2024-09-11T08:16:02.999531+0000","last_clean":"2024-09-11T08:16:02.999531+0000","last_became_active":"2024-09-11T08:07:34.053137+0000","last_became_peered":"2024-09-11T08:07:34.053137+0000","last_unstale":"2024-09-11T08:16:02.999531+0000","last_undegraded":"2024-09-11T08:16:02.999531+0000","last_fullsized":"2024-09-11T08:16:02.999531+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:38.410625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:15:38.410625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:59.728039+0000","scrub_duration":0.021540422,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":185,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.991199+0000","last_change":"2024-09-11T08:15:56.494833+0000","last_active":"2024-09-11T08:16:02.991199+0000","last_peered":"2024-09-11T08:16:02.991199+0000","last_clean":"2024-09-11T08:16:02.991199+0000","last_became_active":"2024-09-11T08:07:36.076132+0000","last_became_peered":"2024-09-11T08:07:36.076132+0000","last_unstale":"2024-09-11T08:16:02.991199+0000","last_undegraded":"2024-09-11T08:16:02.991199+0000","last_fullsized":"2024-09-11T08:16:02.991199+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:56.494764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:15:56.494764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:18.392464+0000","scrub_duration":0.001948635,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":225,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:04.409777+0000","last_change":"2024-09-11T08:16:04.409705+0000","last_active":"2024-09-11T08:16:04.409777+0000","last_peered":"2024-09-11T08:16:04.409777+0000","last_clean":"2024-09-11T08:16:04.409777+0000","last_became_active":"2024-09-11T08:07:32.049368+0000","last_became_peered":"2024-09-11T08:07:32.049368+0000","last_unstale":"2024-09-11T08:16:04.409777+0000","last_undegraded":"2024-09-11T08:16:04.409777+0000","last_fullsized":"2024-09-11T08:16:04.409777+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:16:04.409608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:16:04.409608+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:19.702294+0000","scrub_duration":0.0094835470000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.987286+0000","last_change":"2024-09-11T08:15:56.521976+0000","last_active":"2024-09-11T08:16:02.987286+0000","last_peered":"2024-09-11T08:16:02.987286+0000","last_clean":"2024-09-11T08:16:02.987286+0000","last_became_active":"2024-09-11T08:15:56.521570+0000","last_became_peered":"2024-09-11T08:15:56.521570+0000","last_unstale":"2024-09-11T08:16:02.987286+0000","last_undegraded":"2024-09-11T08:16:02.987286+0000","last_fullsized":"2024-09-11T08:16:02.987286+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:04.256038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:04.256038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:06.812508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":187,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.993035+0000","last_change":"2024-09-11T08:15:01.218687+0000","last_active":"2024-09-11T08:16:02.993035+0000","last_peered":"2024-09-11T08:16:02.993035+0000","last_clean":"2024-09-11T08:16:02.993035+0000","last_became_active":"2024-09-11T08:07:34.057064+0000","last_became_peered":"2024-09-11T08:07:34.057064+0000","last_unstale":"2024-09-11T08:16:02.993035+0000","last_undegraded":"2024-09-11T08:16:02.993035+0000","last_fullsized":"2024-09-11T08:16:02.993035+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:01.218475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:01.218475+0000","last_clean_scrub_stamp":"2024-09-11T08:15:01.218475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:07.409489+0000","scrub_duration":0.0099033660000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":185,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.990458+0000","last_change":"2024-09-11T08:13:08.164389+0000","last_active":"2024-09-11T08:16:02.990458+0000","last_peered":"2024-09-11T08:16:02.990458+0000","last_clean":"2024-09-11T08:16:02.990458+0000","last_became_active":"2024-09-11T08:07:36.077406+0000","last_became_peered":"2024-09-11T08:07:36.077406+0000","last_unstale":"2024-09-11T08:16:02.990458+0000","last_undegraded":"2024-09-11T08:16:02.990458+0000","last_fullsized":"2024-09-11T08:16:02.990458+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:08.164302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:09.973943+0000","last_clean_scrub_stamp":"2024-09-11T08:13:08.164302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003643444,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":259,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.991738+0000","last_change":"2024-09-11T08:15:56.543699+0000","last_active":"2024-09-11T08:16:02.991738+0000","last_peered":"2024-09-11T08:16:02.991738+0000","last_clean":"2024-09-11T08:16:02.991738+0000","last_became_active":"2024-09-11T08:15:56.520058+0000","last_became_peered":"2024-09-11T08:15:56.520058+0000","last_unstale":"2024-09-11T08:16:02.991738+0000","last_undegraded":"2024-09-11T08:16:02.991738+0000","last_fullsized":"2024-09-11T08:16:02.991738+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-11T08:14:10.129903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:14:10.129903+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004657011,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":227,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.990721+0000","last_change":"2024-09-11T08:15:56.531914+0000","last_active":"2024-09-11T08:16:02.990721+0000","last_peered":"2024-09-11T08:16:02.990721+0000","last_clean":"2024-09-11T08:16:02.990721+0000","last_became_active":"2024-09-11T08:15:56.531612+0000","last_became_peered":"2024-09-11T08:15:56.531612+0000","last_unstale":"2024-09-11T08:16:02.990721+0000","last_undegraded":"2024-09-11T08:16:02.990721+0000","last_fullsized":"2024-09-11T08:16:02.990721+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:49.478359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:49.478359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:16.513888+0000","scrub_duration":0.012202396000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":13,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.984721+0000","last_change":"2024-09-11T08:15:56.524265+0000","last_active":"2024-09-11T08:16:02.984721+0000","last_peered":"2024-09-11T08:16:02.984721+0000","last_clean":"2024-09-11T08:16:02.984721+0000","last_became_active":"2024-09-11T08:15:56.523728+0000","last_became_peered":"2024-09-11T08:15:56.523728+0000","last_unstale":"2024-09-11T08:16:02.984721+0000","last_undegraded":"2024-09-11T08:16:02.984721+0000","last_fullsized":"2024-09-11T08:16:02.984721+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:01.931182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:11:34.159053+0000","last_clean_scrub_stamp":"2024-09-11T08:14:01.931182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":185,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.993153+0000","last_change":"2024-09-11T08:15:44.448007+0000","last_active":"2024-09-11T08:16:02.993153+0000","last_peered":"2024-09-11T08:16:02.993153+0000","last_clean":"2024-09-11T08:16:02.993153+0000","last_became_active":"2024-09-11T08:07:36.633368+0000","last_became_peered":"2024-09-11T08:07:36.633368+0000","last_unstale":"2024-09-11T08:16:02.993153+0000","last_undegraded":"2024-09-11T08:16:02.993153+0000","last_fullsized":"2024-09-11T08:16:02.993153+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:44.447924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:34.378734+0000","last_clean_scrub_stamp":"2024-09-11T08:15:44.447924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:50.350614+0000","scrub_duration":0.0029812609999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":195,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.993276+0000","last_change":"2024-09-11T08:15:41.389724+0000","last_active":"2024-09-11T08:16:02.993276+0000","last_peered":"2024-09-11T08:16:02.993276+0000","last_clean":"2024-09-11T08:16:02.993276+0000","last_became_active":"2024-09-11T08:07:32.045728+0000","last_became_peered":"2024-09-11T08:07:32.045728+0000","last_unstale":"2024-09-11T08:16:02.993276+0000","last_undegraded":"2024-09-11T08:16:02.993276+0000","last_fullsized":"2024-09-11T08:16:02.993276+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'4","last_scrub_stamp":"2024-09-11T08:15:41.389510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:15:41.389510+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:10.070852+0000","scrub_duration":0.0069641269999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":201,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.991431+0000","last_change":"2024-09-11T08:15:39.743690+0000","last_active":"2024-09-11T08:16:02.991431+0000","last_peered":"2024-09-11T08:16:02.991431+0000","last_clean":"2024-09-11T08:16:02.991431+0000","last_became_active":"2024-09-11T08:07:30.049409+0000","last_became_peered":"2024-09-11T08:07:30.049409+0000","last_unstale":"2024-09-11T08:16:02.991431+0000","last_undegraded":"2024-09-11T08:16:02.991431+0000","last_fullsized":"2024-09-11T08:16:02.991431+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:39.743604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:39.743604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:08.408228+0000","scrub_duration":0.0089953759999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":233,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.994653+0000","last_change":"2024-09-11T08:15:56.529068+0000","last_active":"2024-09-11T08:16:02.994653+0000","last_peered":"2024-09-11T08:16:02.994653+0000","last_clean":"2024-09-11T08:16:02.994653+0000","last_became_active":"2024-09-11T08:15:56.528607+0000","last_became_peered":"2024-09-11T08:15:56.528607+0000","last_unstale":"2024-09-11T08:16:02.994653+0000","last_undegraded":"2024-09-11T08:16:02.994653+0000","last_fullsized":"2024-09-11T08:16:02.994653+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:37.177042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:15:37.177042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:50.577161+0000","scrub_duration":0.011175861,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":13,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.985068+0000","last_change":"2024-09-11T08:15:56.522225+0000","last_active":"2024-09-11T08:16:02.985068+0000","last_peered":"2024-09-11T08:16:02.985068+0000","last_clean":"2024-09-11T08:16:02.985068+0000","last_became_active":"2024-09-11T08:15:56.521488+0000","last_became_peered":"2024-09-11T08:15:56.521488+0000","last_unstale":"2024-09-11T08:16:02.985068+0000","last_undegraded":"2024-09-11T08:16:02.985068+0000","last_fullsized":"2024-09-11T08:16:02.985068+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:35.183289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:15:35.183289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:51.893171+0000","scrub_duration":0.007179726,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":195,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.999348+0000","last_change":"2024-09-11T08:14:07.076355+0000","last_active":"2024-09-11T08:16:02.999348+0000","last_peered":"2024-09-11T08:16:02.999348+0000","last_clean":"2024-09-11T08:16:02.999348+0000","last_became_active":"2024-09-11T08:07:36.071468+0000","last_became_peered":"2024-09-11T08:07:36.071468+0000","last_unstale":"2024-09-11T08:16:02.999348+0000","last_undegraded":"2024-09-11T08:16:02.999348+0000","last_fullsized":"2024-09-11T08:16:02.999348+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:07.076300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:12:25.814048+0000","last_clean_scrub_stamp":"2024-09-11T08:14:07.076300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001297334,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":201,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.993483+0000","last_change":"2024-09-11T08:15:47.493538+0000","last_active":"2024-09-11T08:16:02.993483+0000","last_peered":"2024-09-11T08:16:02.993483+0000","last_clean":"2024-09-11T08:16:02.993483+0000","last_became_active":"2024-09-11T08:07:30.046702+0000","last_became_peered":"2024-09-11T08:07:30.046702+0000","last_unstale":"2024-09-11T08:16:02.993483+0000","last_undegraded":"2024-09-11T08:16:02.993483+0000","last_fullsized":"2024-09-11T08:16:02.993483+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:47.493466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:47.493466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:47.631872+0000","scrub_duration":0.071745623999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":227,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.995462+0000","last_change":"2024-09-11T08:15:55.255237+0000","last_active":"2024-09-11T08:16:02.995462+0000","last_peered":"2024-09-11T08:16:02.995462+0000","last_clean":"2024-09-11T08:16:02.995462+0000","last_became_active":"2024-09-11T08:07:32.054656+0000","last_became_peered":"2024-09-11T08:07:32.054656+0000","last_unstale":"2024-09-11T08:16:02.995462+0000","last_undegraded":"2024-09-11T08:16:02.995462+0000","last_fullsized":"2024-09-11T08:16:02.995462+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-11T08:15:55.255152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:15:55.255152+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:09.255947+0000","scrub_duration":0.0070337539999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":31,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T08:16:02.995259+0000","last_change":"2024-09-11T08:15:08.769594+0000","last_active":"2024-09-11T08:16:02.995259+0000","last_peered":"2024-09-11T08:16:02.995259+0000","last_clean":"2024-09-11T08:16:02.995259+0000","last_became_active":"2024-09-11T08:15:08.769133+0000","last_became_peered":"2024-09-11T08:15:08.769133+0000","last_unstale":"2024-09-11T08:16:02.995259+0000","last_undegraded":"2024-09-11T08:16:02.995259+0000","last_fullsized":"2024-09-11T08:16:02.995259+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:55.501694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:14:55.501694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:25.384157+0000","scrub_duration":0.024419420000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":6417940504,"num_objects":97931,"num_object_clones":0,"num_object_copies":293793,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8763,"num_objects_unfound":0,"num_objects_dirty":97931,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159729,"num_write_kb":8245057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10118,"num_bytes_recovered":663093248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19492122624,"data_stored":19492110408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85172,"ondisk_log_size":85172,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":139,"num_bytes_recovered":2888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3425,"internal_metadata":0},"log_size":450,"ondisk_log_size":450,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":109,"num_read_kb":104,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":168,"seq":721554505750,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1989556,"kb_used_data":1672044,"kb_used_omap":7,"kb_used_meta":317496,"kb_avail":91755596,"statfs":{"total":95995035648,"available":93957730304,"internally_reserved":0,"allocated":1712173056,"data_stored":1711627763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8027,"internal_metadata":325116069},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:15:25 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":2.9780000000000002,"5min":2.9780000000000002,"15min":2.9780000000000002},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.995,"5min":0.995,"15min":0.995},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":2.8069999999999999,"5min":2.8069999999999999,"15min":2.8069999999999999},"last":2.1280000000000001}]},{"osd":1,"last update":"Wed Sep 11 08:15:25 2024","interfaces":[{"interface":"back","average":{"1min":1.946,"5min":1.946,"15min":1.946},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":26.390999999999998,"5min":26.390999999999998,"15min":26.390999999999998},"last":2.2919999999999998},{"interface":"front","average":{"1min":2.0459999999999998,"5min":2.0459999999999998,"15min":2.0459999999999998},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":26.481000000000002,"5min":26.481000000000002,"15min":26.481000000000002},"last":0.57199999999999995}]},{"osd":2,"last update":"Wed Sep 11 08:15:25 2024","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.042,"15min":1.042},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":3.9630000000000001,"5min":3.9630000000000001,"15min":3.9630000000000001},"last":2.1600000000000001},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":2.601,"5min":2.601,"15min":2.601},"last":2.3119999999999998}]},{"osd":3,"last update":"Wed Sep 11 08:15:25 2024","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.073,"15min":1.073},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":3.702,"5min":3.702,"15min":3.702},"last":0.502},{"interface":"front","average":{"1min":1.425,"5min":1.425,"15min":1.425},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":3.8109999999999999,"5min":3.8109999999999999,"15min":3.8109999999999999},"last":2.1920000000000002}]},{"osd":4,"last update":"Wed Sep 11 08:15:25 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.7470000000000001,"5min":1.7470000000000001,"15min":1.7470000000000001},"last":2.2679999999999998},{"interface":"front","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":2.153,"5min":2.153,"15min":2.153},"last":0.45900000000000002}]},{"osd":5,"last update":"Wed Sep 11 08:15:25 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":2.6139999999999999,"5min":2.6139999999999999,"15min":2.6139999999999999},"last":2.3490000000000002},{"interface":"front","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":2.6240000000000001,"5min":2.6240000000000001,"15min":2.6240000000000001},"last":0.66300000000000003}]},{"osd":6,"last update":"Wed Sep 11 08:15:25 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":2.8370000000000002,"5min":2.8370000000000002,"15min":2.8370000000000002},"last":6.0609999999999999},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":2.0449999999999999,"5min":2.0449999999999999,"15min":2.0449999999999999},"last":2.3319999999999999}]}]},{"osd":6,"up_from":45,"seq":193273528433,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2689260,"kb_used_data":2387744,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":91055892,"statfs":{"total":95995035648,"available":93241233408,"internally_reserved":0,"allocated":2445049856,"data_stored":2444516698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":1.988,"15min":0.99099999999999999},"min":{"1min":0.20399999999999999,"5min":0.185,"15min":0.185},"max":{"1min":5.1449999999999996,"5min":45.066000000000003,"15min":45.066000000000003},"last":7.5810000000000004},{"interface":"front","average":{"1min":1.157,"5min":1.8700000000000001,"15min":0.95899999999999996},"min":{"1min":0.22,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":4.6230000000000002,"5min":43.899000000000001,"15min":43.899000000000001},"last":3.6960000000000002}]},{"osd":1,"last update":"Wed Sep 11 08:15:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.538,"15min":0.83099999999999996},"min":{"1min":0.246,"5min":0.217,"15min":0.217},"max":{"1min":4.9180000000000001,"5min":44.658000000000001,"15min":44.658000000000001},"last":3.4590000000000001},{"interface":"front","average":{"1min":0.93500000000000005,"5min":1.677,"15min":0.89100000000000001},"min":{"1min":0.30499999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":5.7949999999999999,"5min":44.694000000000003,"15min":44.694000000000003},"last":0.32500000000000001}]},{"osd":2,"last update":"Wed Sep 11 08:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.93999999999999995,"15min":0.64400000000000002},"min":{"1min":0.29099999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.1400000000000001,"5min":4.8399999999999999,"15min":4.8399999999999999},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.75900000000000001,"5min":1.262,"15min":0.74199999999999999},"min":{"1min":0.27400000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":2.5070000000000001,"5min":13.346,"15min":13.346},"last":3.6619999999999999}]},{"osd":3,"last update":"Wed Sep 11 08:15:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.8460000000000001,"15min":0.96199999999999997},"min":{"1min":0.40799999999999997,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":2.8220000000000001,"5min":36.061,"15min":36.061},"last":3.3980000000000001},{"interface":"front","average":{"1min":1.145,"5min":1.7889999999999999,"15min":0.95299999999999996},"min":{"1min":0.378,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":3.3330000000000002,"5min":36.093000000000004,"15min":36.093000000000004},"last":0.38800000000000001}]},{"osd":4,"last update":"Wed Sep 11 08:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.45200000000000001,"15min":0.41199999999999998},"min":{"1min":0.253,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":3.306,"5min":3.306,"15min":3.306},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.50800000000000001,"15min":0.44900000000000001},"min":{"1min":0.28000000000000003,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":3.278,"5min":3.278,"15min":3.278},"last":0.52400000000000002}]},{"osd":5,"last update":"Wed Sep 11 08:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.56000000000000005,"15min":0.51100000000000001},"min":{"1min":0.32400000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":3.2970000000000002,"5min":3.2970000000000002,"15min":3.2970000000000002},"last":0.37},{"interface":"front","average":{"1min":0.97799999999999998,"5min":0.60099999999999998,"15min":0.47799999999999998},"min":{"1min":0.28999999999999998,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":3.3319999999999999,"5min":3.3319999999999999,"15min":3.3319999999999999},"last":3.4329999999999998}]},{"osd":7,"last update":"Wed Sep 11 08:15:23 2024","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":2.9980000000000002,"5min":2.9980000000000002,"15min":2.9980000000000002},"last":8.4580000000000002},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":3.2919999999999998,"5min":3.2919999999999998,"15min":3.2919999999999998},"last":0.54200000000000004}]}]},{"osd":5,"up_from":38,"seq":163208757366,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3874092,"kb_used_data":3563936,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":89871060,"statfs":{"total":95995035648,"available":92027965440,"internally_reserved":0,"allocated":3649470464,"data_stored":3648936323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:15:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4430000000000001,"5min":1.254,"15min":0.78800000000000003},"min":{"1min":0.20999999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":4.327,"5min":7.9470000000000001,"15min":7.9470000000000001},"last":0.56299999999999994},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.222,"15min":0.77700000000000002},"min":{"1min":0.24399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":3.4820000000000002,"5min":7.6189999999999998,"15min":7.6189999999999998},"last":0.246}]},{"osd":1,"last update":"Wed Sep 11 08:15:48 2024","interfaces":[{"interface":"back","average":{"1min":2.77,"5min":1.5760000000000001,"15min":1.2829999999999999},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":26.709,"5min":26.709,"15min":26.709},"last":0.27700000000000002},{"interface":"front","average":{"1min":2.5339999999999998,"5min":1.5569999999999999,"15min":0.85899999999999999},"min":{"1min":0.25900000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":26.754000000000001,"5min":26.754000000000001,"15min":26.754000000000001},"last":0.30599999999999999}]},{"osd":2,"last update":"Wed Sep 11 08:15:48 2024","interfaces":[{"interface":"back","average":{"1min":1.891,"5min":1.2589999999999999,"15min":1.171},"min":{"1min":0.36799999999999999,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":10.891,"5min":10.891,"15min":16.018000000000001},"last":0.58299999999999996},{"interface":"front","average":{"1min":1.4330000000000001,"5min":0.99099999999999999,"15min":0.68500000000000005},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":7.5250000000000004,"5min":7.5250000000000004,"15min":7.5250000000000004},"last":0.28499999999999998}]},{"osd":3,"last update":"Wed Sep 11 08:15:48 2024","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.3819999999999999,"15min":0.85399999999999998},"min":{"1min":0.29999999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":4.3150000000000004,"5min":24.931000000000001,"15min":24.931000000000001},"last":0.25800000000000001},{"interface":"front","average":{"1min":1.544,"5min":1.514,"15min":0.879},"min":{"1min":0.38800000000000001,"5min":0.251,"15min":0.251},"max":{"1min":4.4710000000000001,"5min":24.861999999999998,"15min":24.861999999999998},"last":0.52200000000000002}]},{"osd":4,"last update":"Wed Sep 11 08:15:48 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.58199999999999996,"15min":0.46999999999999997},"min":{"1min":0.23499999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5979999999999999,"5min":3.5979999999999999,"15min":3.5979999999999999},"last":1.159},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.56200000000000006,"15min":0.46200000000000002},"min":{"1min":0.27300000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":2.706,"5min":2.706,"15min":2.706},"last":0.433}]},{"osd":6,"last update":"Wed Sep 11 08:15:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.48899999999999999,"15min":0.443},"min":{"1min":0.29599999999999999,"5min":0.252,"15min":0.23400000000000001},"max":{"1min":1.1180000000000001,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.748,"5min":0.57099999999999995,"15min":0.49199999999999999},"min":{"1min":0.33200000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":3.9489999999999998,"5min":3.9489999999999998,"15min":3.9489999999999998},"last":0.61199999999999999}]},{"osd":7,"last update":"Wed Sep 11 08:15:25 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":3.1320000000000001,"5min":3.1320000000000001,"15min":3.1320000000000001},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":3.0499999999999998,"5min":3.0499999999999998,"15min":3.0499999999999998},"last":0.60199999999999998}]}]},{"osd":4,"up_from":31,"seq":133143986299,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2295740,"kb_used_data":1985904,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":91449412,"statfs":{"total":95995035648,"available":93644197888,"internally_reserved":0,"allocated":2033565696,"data_stored":2033016461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:15:18 2024","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":1.4079999999999999,"15min":1.006},"min":{"1min":0.22700000000000001,"5min":0.19,"15min":0.19},"max":{"1min":1.6870000000000001,"5min":26.754999999999999,"15min":48.728999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.76300000000000001,"5min":1.1679999999999999,"15min":0.91300000000000003},"min":{"1min":0.22600000000000001,"5min":0.221,"15min":0.221},"max":{"1min":3.2909999999999999,"5min":27.055,"15min":49.168999999999997},"last":0.57299999999999995}]},{"osd":1,"last update":"Wed Sep 11 08:15:18 2024","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1,"15min":0.71499999999999997},"min":{"1min":0.245,"5min":0.217,"15min":0.217},"max":{"1min":3.508,"5min":4.7320000000000002,"15min":13.207000000000001},"last":0.504},{"interface":"front","average":{"1min":1.0089999999999999,"5min":0.94799999999999995,"15min":0.66700000000000004},"min":{"1min":0.27800000000000002,"5min":0.22900000000000001,"15min":0.22700000000000001},"max":{"1min":2.3090000000000002,"5min":4.2089999999999996,"15min":4.2089999999999996},"last":0.53600000000000003}]},{"osd":2,"last update":"Wed Sep 11 08:15:18 2024","interfaces":[{"interface":"back","average":{"1min":1.45,"5min":1.0629999999999999,"15min":0.71099999999999997},"min":{"1min":0.26000000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":7.976,"5min":7.976,"15min":7.976},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.98799999999999999,"15min":0.67800000000000005},"min":{"1min":0.26500000000000001,"5min":0.248,"15min":0.248},"max":{"1min":2.0030000000000001,"5min":5.5670000000000002,"15min":5.5670000000000002},"last":0.55200000000000005}]},{"osd":3,"last update":"Wed Sep 11 08:15:18 2024","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.468,"15min":0.93799999999999994},"min":{"1min":0.34999999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":5.6280000000000001,"5min":26.884,"15min":26.884},"last":0.435},{"interface":"front","average":{"1min":0.82199999999999995,"5min":1.6759999999999999,"15min":0.96599999999999997},"min":{"1min":0.311,"5min":0.246,"15min":0.246},"max":{"1min":2.9289999999999998,"5min":26.783000000000001,"15min":26.783000000000001},"last":0.36499999999999999}]},{"osd":5,"last update":"Wed Sep 11 08:15:45 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.497,"15min":0.42299999999999999},"min":{"1min":0.26400000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":2.5489999999999999,"5min":2.5489999999999999,"15min":2.5489999999999999},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.48299999999999998,"15min":0.41499999999999998},"min":{"1min":0.253,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.7390000000000001,"5min":1.9630000000000001,"15min":1.9630000000000001},"last":0.46800000000000003}]},{"osd":6,"last update":"Wed Sep 11 08:15:13 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.47199999999999998,"15min":0.41799999999999998},"min":{"1min":0.27000000000000002,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":1.012,"5min":2.1230000000000002,"15min":2.1230000000000002},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.54700000000000004,"15min":0.47799999999999998},"min":{"1min":0.33500000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":2.254,"5min":2.254,"15min":2.254},"last":0.377}]},{"osd":7,"last update":"Wed Sep 11 08:15:25 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":3.2770000000000001,"5min":3.2770000000000001,"15min":3.2770000000000001},"last":0.30299999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182528,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2661700,"kb_used_data":2356028,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":91083452,"statfs":{"total":95995035648,"available":93269454848,"internally_reserved":0,"allocated":2412572672,"data_stored":2412010620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:16:04 2024","interfaces":[{"interface":"back","average":{"1min":4.2320000000000002,"5min":2.363,"15min":1.099},"min":{"1min":0.20200000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":60.469999999999999,"5min":60.469999999999999,"15min":60.469999999999999},"last":1.7969999999999999},{"interface":"front","average":{"1min":4.4939999999999998,"5min":2.613,"15min":1.1990000000000001},"min":{"1min":0.245,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":60.488999999999997,"5min":60.488999999999997,"15min":60.488999999999997},"last":1.732}]},{"osd":1,"last update":"Wed Sep 11 08:16:04 2024","interfaces":[{"interface":"back","average":{"1min":4.0620000000000003,"5min":2.2679999999999998,"15min":1.0860000000000001},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.20899999999999999},"max":{"1min":60.578000000000003,"5min":60.578000000000003,"15min":60.578000000000003},"last":1.232},{"interface":"front","average":{"1min":4.5910000000000002,"5min":2.411,"15min":1.1830000000000001},"min":{"1min":0.23400000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":63.674999999999997,"5min":63.674999999999997,"15min":63.674999999999997},"last":3.5710000000000002}]},{"osd":2,"last update":"Wed Sep 11 08:16:04 2024","interfaces":[{"interface":"back","average":{"1min":4.6840000000000002,"5min":2.6139999999999999,"15min":1.222},"min":{"1min":0.22,"5min":0.185,"15min":0.185},"max":{"1min":60.645000000000003,"5min":60.645000000000003,"15min":60.645000000000003},"last":1.585},{"interface":"front","average":{"1min":4.6740000000000004,"5min":2.3740000000000001,"15min":1.1379999999999999},"min":{"1min":0.38600000000000001,"5min":0.20699999999999999,"15min":0.20599999999999999},"max":{"1min":60.723999999999997,"5min":60.723999999999997,"15min":60.723999999999997},"last":0.91800000000000004}]},{"osd":4,"last update":"Wed Sep 11 08:15:20 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":1.3999999999999999,"15min":0.83699999999999997},"min":{"1min":0.28499999999999998,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":5.1829999999999998,"5min":6.468,"15min":6.468},"last":0.495},{"interface":"front","average":{"1min":1.417,"5min":1.3460000000000001,"15min":0.80200000000000005},"min":{"1min":0.32700000000000001,"5min":0.27500000000000002,"15min":0.27200000000000002},"max":{"1min":5.0199999999999996,"5min":10.563000000000001,"15min":10.563000000000001},"last":0.89500000000000002}]},{"osd":5,"last update":"Wed Sep 11 08:15:47 2024","interfaces":[{"interface":"back","average":{"1min":6.5309999999999997,"5min":2.3620000000000001,"15min":1.159},"min":{"1min":0.40000000000000002,"5min":0.26600000000000001,"15min":0.26500000000000001},"max":{"1min":60.914999999999999,"5min":60.914999999999999,"15min":60.914999999999999},"last":0.873},{"interface":"front","average":{"1min":6.2590000000000003,"5min":2.3730000000000002,"15min":1.242},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":60.787999999999997,"5min":60.787999999999997,"15min":60.787999999999997},"last":0.82899999999999996}]},{"osd":6,"last update":"Wed Sep 11 08:15:06 2024","interfaces":[{"interface":"back","average":{"1min":3.4100000000000001,"5min":1.839,"15min":1.034},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33300000000000002},"max":{"1min":42.956000000000003,"5min":42.956000000000003,"15min":42.956000000000003},"last":0.60999999999999999},{"interface":"front","average":{"1min":3.1150000000000002,"5min":2.032,"15min":1.111},"min":{"1min":0.33100000000000002,"5min":0.307,"15min":0.307},"max":{"1min":42.850999999999999,"5min":42.850999999999999,"15min":42.850999999999999},"last":0.52800000000000002}]},{"osd":7,"last update":"Wed Sep 11 08:15:24 2024","interfaces":[{"interface":"back","average":{"1min":3.1419999999999999,"5min":3.1419999999999999,"15min":3.1419999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":32.000999999999998,"5min":32.000999999999998,"15min":32.000999999999998},"last":0.78000000000000003},{"interface":"front","average":{"1min":3.073,"5min":3.073,"15min":3.073},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":32.792999999999999,"5min":32.792999999999999,"15min":32.792999999999999},"last":0.84299999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411459,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2689192,"kb_used_data":2383520,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":91055960,"statfs":{"total":95995035648,"available":93241303040,"internally_reserved":0,"allocated":2440724480,"data_stored":2440190251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:15:42 2024","interfaces":[{"interface":"back","average":{"1min":4.8499999999999996,"5min":2.3260000000000001,"15min":1.071},"min":{"1min":0.32800000000000001,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":43.539999999999999,"5min":43.539999999999999,"15min":43.539999999999999},"last":8.0920000000000005},{"interface":"front","average":{"1min":5.2800000000000002,"5min":2.5870000000000002,"15min":1.139},"min":{"1min":0.20399999999999999,"5min":0.185,"15min":0.185},"max":{"1min":41.962000000000003,"5min":41.962000000000003,"15min":41.962000000000003},"last":5.9790000000000001}]},{"osd":1,"last update":"Wed Sep 11 08:15:42 2024","interfaces":[{"interface":"back","average":{"1min":1.992,"5min":1.734,"15min":0.88700000000000001},"min":{"1min":0.182,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":13.138999999999999,"5min":13.138999999999999,"15min":13.138999999999999},"last":0.53100000000000003},{"interface":"front","average":{"1min":2.7309999999999999,"5min":1.9019999999999999,"15min":0.94099999999999995},"min":{"1min":0.26600000000000001,"5min":0.23200000000000001,"15min":0.20200000000000001},"max":{"1min":10.167999999999999,"5min":10.167999999999999,"15min":10.167999999999999},"last":5.0220000000000002}]},{"osd":3,"last update":"Wed Sep 11 08:16:05 2024","interfaces":[{"interface":"back","average":{"1min":2.5070000000000001,"5min":1.97,"15min":0.998},"min":{"1min":0.21299999999999999,"5min":0.188,"15min":0.188},"max":{"1min":8.1989999999999998,"5min":8.1989999999999998,"15min":8.1989999999999998},"last":2.0179999999999998},{"interface":"front","average":{"1min":2.4289999999999998,"5min":1.879,"15min":0.97299999999999998},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.22500000000000001},"max":{"1min":8.0640000000000001,"5min":8.3219999999999992,"15min":8.3219999999999992},"last":4.681}]},{"osd":4,"last update":"Wed Sep 11 08:15:25 2024","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.242,"15min":0.77800000000000002},"min":{"1min":0.32300000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":4.923,"5min":7.806,"15min":7.806},"last":1.8220000000000001},{"interface":"front","average":{"1min":1.248,"5min":1.206,"15min":0.83099999999999996},"min":{"1min":0.28699999999999998,"5min":0.252,"15min":0.252},"max":{"1min":5.4740000000000002,"5min":6.8200000000000003,"15min":12.129},"last":0.309}]},{"osd":5,"last update":"Wed Sep 11 08:15:57 2024","interfaces":[{"interface":"back","average":{"1min":1.6120000000000001,"5min":1.3100000000000001,"15min":0.85199999999999998},"min":{"1min":0.39900000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":5.4539999999999997,"5min":6.7709999999999999,"15min":6.7709999999999999},"last":0.38100000000000001},{"interface":"front","average":{"1min":1.196,"5min":1.3160000000000001,"15min":0.84599999999999997},"min":{"1min":0.36299999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":4.024,"5min":12.869,"15min":12.869},"last":0.55800000000000005}]},{"osd":6,"last update":"Wed Sep 11 08:15:09 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":1.141,"15min":0.77200000000000002},"min":{"1min":0.39500000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.8109999999999999,"5min":7.2249999999999996,"15min":7.2249999999999996},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.91100000000000003,"5min":1.2629999999999999,"15min":0.81599999999999995},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.32100000000000001},"max":{"1min":3.1850000000000001,"5min":6.71,"15min":6.71},"last":0.35499999999999998}]},{"osd":7,"last update":"Wed Sep 11 08:15:25 2024","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.389,"15min":1.389},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":4.9569999999999999,"5min":4.9569999999999999,"15min":4.9569999999999999},"last":1.9179999999999999},{"interface":"front","average":{"1min":1.5369999999999999,"5min":1.5369999999999999,"15min":1.5369999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":5.3710000000000004,"5min":5.3710000000000004,"15min":5.3710000000000004},"last":1.736}]}]},{"osd":1,"up_from":13,"seq":55834574985,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3048444,"kb_used_data":2746928,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":90696708,"statfs":{"total":95995035648,"available":92873428992,"internally_reserved":0,"allocated":2812854272,"data_stored":2812304293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:15:17 2024","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.8540000000000001,"15min":0.84299999999999997},"min":{"1min":0.249,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":4.6820000000000004,"5min":32.478000000000002,"15min":32.478000000000002},"last":1.264},{"interface":"front","average":{"1min":1.9490000000000001,"5min":2.4670000000000001,"15min":1.0640000000000001},"min":{"1min":0.19500000000000001,"5min":0.186,"15min":0.186},"max":{"1min":6.8170000000000002,"5min":35.459000000000003,"15min":35.459000000000003},"last":5.4210000000000003}]},{"osd":2,"last update":"Wed Sep 11 08:15:49 2024","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.5800000000000001,"15min":0.79600000000000004},"min":{"1min":0.26400000000000001,"5min":0.216,"15min":0.21299999999999999},"max":{"1min":7.9699999999999998,"5min":8.3819999999999997,"15min":8.3819999999999997},"last":0.999},{"interface":"front","average":{"1min":1.5229999999999999,"5min":1.8779999999999999,"15min":0.92000000000000004},"min":{"1min":0.192,"5min":0.192,"15min":0.16300000000000001},"max":{"1min":5.1319999999999997,"5min":10.276,"15min":10.276},"last":1.026}]},{"osd":3,"last update":"Wed Sep 11 08:16:01 2024","interfaces":[{"interface":"back","average":{"1min":1.835,"5min":2.3479999999999999,"15min":1.339},"min":{"1min":0.38600000000000001,"5min":0.248,"15min":0.246},"max":{"1min":5.1689999999999996,"5min":16.783999999999999,"15min":32.759},"last":2.278},{"interface":"front","average":{"1min":1.0469999999999999,"5min":2.008,"15min":1.175},"min":{"1min":0.27000000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":4.2489999999999997,"5min":14.973000000000001,"15min":32.444000000000003},"last":0.97299999999999998}]},{"osd":4,"last update":"Wed Sep 11 08:15:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.3120000000000001,"15min":0.81499999999999995},"min":{"1min":0.26300000000000001,"5min":0.252,"15min":0.252},"max":{"1min":3.653,"5min":9.0139999999999993,"15min":9.0139999999999993},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.93400000000000005,"5min":1.266,"15min":0.79800000000000004},"min":{"1min":0.373,"5min":0.248,"15min":0.248},"max":{"1min":2.3540000000000001,"5min":8.2360000000000007,"15min":8.2360000000000007},"last":0.42399999999999999}]},{"osd":5,"last update":"Wed Sep 11 08:15:49 2024","interfaces":[{"interface":"back","average":{"1min":1.6859999999999999,"5min":1.4399999999999999,"15min":0.90900000000000003},"min":{"1min":0.316,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":5.3869999999999996,"5min":8.6769999999999996,"15min":9.0470000000000006},"last":0.45200000000000001},{"interface":"front","average":{"1min":1.8959999999999999,"5min":1.6399999999999999,"15min":0.95399999999999996},"min":{"1min":0.35199999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":4.5019999999999998,"5min":8.9860000000000007,"15min":11.967000000000001},"last":0.59099999999999997}]},{"osd":6,"last update":"Wed Sep 11 08:16:03 2024","interfaces":[{"interface":"back","average":{"1min":7.8250000000000002,"5min":2.7599999999999998,"15min":1.3859999999999999},"min":{"1min":0.32600000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":124.31999999999999,"5min":124.31999999999999,"15min":124.31999999999999},"last":0.626},{"interface":"front","average":{"1min":7.6020000000000003,"5min":2.6779999999999999,"15min":1.341},"min":{"1min":0.39200000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":124.958,"5min":124.958,"15min":124.958},"last":0.39200000000000002}]},{"osd":7,"last update":"Wed Sep 11 08:15:26 2024","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":3.6850000000000001,"5min":3.6850000000000001,"15min":3.6850000000000001},"last":0.496},{"interface":"front","average":{"1min":1.48,"5min":1.48,"15min":1.48},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":4.593,"5min":4.593,"15min":4.593},"last":0.65000000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738509,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2663700,"kb_used_data":2353864,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":91081452,"statfs":{"total":95995035648,"available":93267406848,"internally_reserved":0,"allocated":2410356736,"data_stored":2409784075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 11 08:15:19 2024","interfaces":[{"interface":"back","average":{"1min":1.5940000000000001,"5min":1.6850000000000001,"15min":0.78500000000000003},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":6.9580000000000002,"5min":7.2160000000000002,"15min":7.2160000000000002},"last":1.7969999999999999},{"interface":"front","average":{"1min":1.998,"5min":1.899,"15min":0.86499999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":6.8579999999999997,"5min":9.5679999999999996,"15min":9.5679999999999996},"last":0.29899999999999999}]},{"osd":2,"last update":"Wed Sep 11 08:15:43 2024","interfaces":[{"interface":"back","average":{"1min":1.607,"5min":1.5620000000000001,"15min":0.80000000000000004},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":6.875,"5min":10.351000000000001,"15min":10.351000000000001},"last":0.74299999999999999},{"interface":"front","average":{"1min":1.421,"5min":1.7829999999999999,"15min":0.88600000000000001},"min":{"1min":0.24299999999999999,"5min":0.186,"15min":0.186},"max":{"1min":6.2320000000000002,"5min":7.3090000000000002,"15min":7.3090000000000002},"last":2.6219999999999999}]},{"osd":3,"last update":"Wed Sep 11 08:16:05 2024","interfaces":[{"interface":"back","average":{"1min":1.546,"5min":2.2549999999999999,"15min":1.2130000000000001},"min":{"1min":0.29299999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":4.5679999999999996,"5min":35.491,"15min":35.491},"last":2.5630000000000002},{"interface":"front","average":{"1min":2.2240000000000002,"5min":2.6150000000000002,"15min":1.246},"min":{"1min":0.32900000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.7919999999999998,"5min":37.563000000000002,"15min":37.563000000000002},"last":3.1829999999999998}]},{"osd":4,"last update":"Wed Sep 11 08:15:24 2024","interfaces":[{"interface":"back","average":{"1min":1.6160000000000001,"5min":1.51,"15min":0.86899999999999999},"min":{"1min":0.26900000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":6.0190000000000001,"5min":9.4030000000000005,"15min":9.4030000000000005},"last":0.39300000000000002},{"interface":"front","average":{"1min":1.373,"5min":1.4650000000000001,"15min":0.86499999999999999},"min":{"1min":0.32400000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":4.2460000000000004,"5min":6.6619999999999999,"15min":6.6619999999999999},"last":2.5899999999999999}]},{"osd":5,"last update":"Wed Sep 11 08:15:53 2024","interfaces":[{"interface":"back","average":{"1min":2.4529999999999998,"5min":1.696,"15min":0.98899999999999999},"min":{"1min":0.33100000000000002,"5min":0.255,"15min":0.255},"max":{"1min":24.914999999999999,"5min":24.914999999999999,"15min":24.914999999999999},"last":2.6440000000000001},{"interface":"front","average":{"1min":3.173,"5min":1.9299999999999999,"15min":1.0720000000000001},"min":{"1min":0.34100000000000003,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":25.277000000000001,"5min":25.277000000000001,"15min":25.277000000000001},"last":0.36699999999999999}]},{"osd":6,"last update":"Wed Sep 11 08:15:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6619999999999999,"5min":2.4359999999999999,"15min":1.2130000000000001},"min":{"1min":0.32000000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":6.0549999999999997,"5min":78.522000000000006,"15min":78.522000000000006},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.2829999999999999,"5min":2.089,"15min":1.081},"min":{"1min":0.30599999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":3.2149999999999999,"5min":79.808000000000007,"15min":79.808000000000007},"last":0.77100000000000002}]},{"osd":7,"last update":"Wed Sep 11 08:15:24 2024","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.3620000000000001,"15min":1.3620000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":6.7750000000000004,"5min":6.7750000000000004,"15min":6.7750000000000004},"last":0.79700000000000004},{"interface":"front","average":{"1min":1.355,"5min":1.355,"15min":1.355},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":6.8179999999999996,"5min":6.8179999999999996,"15min":6.8179999999999996},"last":0.82399999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3425,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2397700096,"data_stored":2397700096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2784886784,"data_stored":2784886784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2413957120,"data_stored":2413953048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2392522752,"data_stored":2392522752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2007695360,"data_stored":2007695360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3627683840,"data_stored":3627679768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2423197696,"data_stored":2423193624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1444478976,"data_stored":1444478976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T08:16:12.553 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T08:16:12.558 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:12.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:12.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:16:12.660 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:12.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:12.705 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:12.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:12.806 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:12.850 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:12.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:12.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:12.952 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:13.006 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:13.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:13.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:13.082 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:16:13.107 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:16:13.114 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:16:13.136 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:13.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:13.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:16:13.238 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:13.283 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:13.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:13.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:13.385 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:13.414 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:13.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:13.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:13.516 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:13.560 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:13.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:13.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:13.662 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:13.700 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:13.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:13.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:13.782 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:16:13.795 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-09-11T08:16:13.795 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:16:13.803 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:13.832 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:13.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:13.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:13.934 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:16:13.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:13.970 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:13.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:16:14.071 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:14.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:14.103 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:14.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:14.204 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:16:14.239 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:14.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:14.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:16:14.285 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:14 smithi150 ceph-mon[29276]: pgmap v384: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 5.9 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.09k op/s; 8763/291177 objects misplaced (3.010%); 16 MiB/s, 258 objects/s recovering 2024-09-11T08:16:14.285 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:14 smithi150 ceph-mon[39612]: pgmap v384: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 5.9 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.09k op/s; 8763/291177 objects misplaced (3.010%); 16 MiB/s, 258 objects/s recovering 2024-09-11T08:16:14.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:14 smithi181 ceph-mon[32701]: pgmap v384: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 5.9 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.09k op/s; 8763/291177 objects misplaced (3.010%); 16 MiB/s, 258 objects/s recovering 2024-09-11T08:16:14.341 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:16:14.381 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:14.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:14.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:16:14.483 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:14.519 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:14.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:14.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:14.621 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:16:14.647 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:14.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:14.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:16:14.749 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:16:14.781 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:14.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:14.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:16:14.883 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:14.918 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:14.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:14.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:15.019 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:16:15.050 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:15.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:15.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:16:15.152 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:15.174 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:15.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:15.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:15.276 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:16:15.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:15.309 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:15.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:16:15.410 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:15.444 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:15.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:15.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:15.546 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:15.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:16:15 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:16:15] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:16:15.581 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:15.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:15.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:15.682 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:15.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:15.725 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:15.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:15.826 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:16:15.860 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:15.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:15.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:16:15.962 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:16:15.995 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:15.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:15.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:16:16.097 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:16:16.139 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:16.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:16.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:16:16.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:16 smithi150 ceph-mon[29276]: pgmap v385: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.16k op/s; 6414/277053 objects misplaced (2.315%); 23 MiB/s, 372 objects/s recovering 2024-09-11T08:16:16.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:16 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:16:16.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:16 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:16:16.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:16 smithi150 ceph-mon[39612]: pgmap v385: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.16k op/s; 6414/277053 objects misplaced (2.315%); 23 MiB/s, 372 objects/s recovering 2024-09-11T08:16:16.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:16 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:16:16.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:16 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:16:16.241 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:16:16.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:16.278 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:16.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:16:16.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:16 smithi181 ceph-mon[32701]: pgmap v385: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.16k op/s; 6414/277053 objects misplaced (2.315%); 23 MiB/s, 372 objects/s recovering 2024-09-11T08:16:16.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:16 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:16:16.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:16 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:16:16.379 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:16:16.410 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:16.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:16.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:16:16.512 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:16:16.544 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:16.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:16.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:16:16.646 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:16.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:16.680 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:16.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:16.781 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:16.818 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:16.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:16.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:16.920 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:16:16.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:16.957 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:16.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:16:17.058 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:17.088 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:17.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:17.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:17.190 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:17.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:17.226 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:17.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:17.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:17 smithi150 ceph-mon[29276]: osdmap e212: 8 total, 8 up, 8 in 2024-09-11T08:16:17.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:17 smithi150 ceph-mon[39612]: osdmap e212: 8 total, 8 up, 8 in 2024-09-11T08:16:17.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:17 smithi181 ceph-mon[32701]: osdmap e212: 8 total, 8 up, 8 in 2024-09-11T08:16:17.327 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:16:17.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:17.361 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:17.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:16:17.462 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:16:17.497 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:17.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:17.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:16:17.599 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:17.639 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:17.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:17.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:17.741 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:17.778 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:17.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:17.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:17.880 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:17.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:17.915 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:17.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:18.015 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:18.055 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:18.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:18.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:18.157 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:18.195 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:18.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:18.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:18.297 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:18.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:18 smithi181 ceph-mon[32701]: pgmap v387: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.16k op/s; 6414/277053 objects misplaced (2.315%); 23 MiB/s, 372 objects/s recovering 2024-09-11T08:16:18.323 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:16:18 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:16:18.331 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:18.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:18.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:18.433 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:16:18.468 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:18.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:18.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:16:18.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:18 smithi150 ceph-mon[29276]: pgmap v387: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.16k op/s; 6414/277053 objects misplaced (2.315%); 23 MiB/s, 372 objects/s recovering 2024-09-11T08:16:18.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:18 smithi150 ceph-mon[39612]: pgmap v387: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.16k op/s; 6414/277053 objects misplaced (2.315%); 23 MiB/s, 372 objects/s recovering 2024-09-11T08:16:18.570 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:18.599 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:18.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:18.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:18.701 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:18.733 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:18.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:18.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:18.835 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:18.863 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:18.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:18.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:18.946 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:16:18 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:16:18] "GET /metrics HTTP/1.1" 200 224218 "" "Prometheus/2.43.0" 2024-09-11T08:16:18.965 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:19.000 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:19.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:19.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:19.102 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:16:19.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:19.140 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:19.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:16:19.185 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:19 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:16:19.215 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:19 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:16:19.215 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:19 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:16:19.241 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:19.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:19.283 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:19.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:19.384 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:19.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:19.424 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:19.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:19.479 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-11T08:16:19.525 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:16:19.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:19.560 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:19.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:16:19.661 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:19.707 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:19.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:19.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:19.809 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:16:19.846 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:19.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:19.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:16:19.948 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:19.985 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:19.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:19.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:20.076 INFO:teuthology.orchestra.run.smithi150.stderr:nodeep-scrub is set 2024-09-11T08:16:20.087 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:20.130 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:20.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:20.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:20.232 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:16:20.269 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:20.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:20.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:16:20.309 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:20 smithi150 ceph-mon[29276]: pgmap v388: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s; 4242/259152 objects misplaced (1.637%); 23 MiB/s, 373 objects/s recovering 2024-09-11T08:16:20.309 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:20 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2051207836' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:16:20.310 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:20 smithi150 ceph-mon[39612]: pgmap v388: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s; 4242/259152 objects misplaced (1.637%); 23 MiB/s, 373 objects/s recovering 2024-09-11T08:16:20.310 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:20 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2051207836' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:16:20.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:20 smithi181 ceph-mon[32701]: pgmap v388: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s; 4242/259152 objects misplaced (1.637%); 23 MiB/s, 373 objects/s recovering 2024-09-11T08:16:20.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:20 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2051207836' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:16:20.371 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:20.406 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:20.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:20.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:20.507 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:16:20.551 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:20.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:20.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:16:20.653 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:20.687 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:20.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:20.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:20.789 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:20.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:20.828 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:20.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:20.929 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:16:20.967 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:20.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:20.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:16:21.069 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:21.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:21.113 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:21.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:21.214 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:21.252 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:21.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:21.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:21.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:21 smithi181 ceph-mon[32701]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:16:21.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:21 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2051207836' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:16:21.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:21 smithi181 ceph-mon[32701]: osdmap e213: 8 total, 8 up, 8 in 2024-09-11T08:16:21.330 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:21 smithi150 ceph-mon[29276]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:16:21.330 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:21 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2051207836' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:16:21.330 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:21 smithi150 ceph-mon[29276]: osdmap e213: 8 total, 8 up, 8 in 2024-09-11T08:16:21.330 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:21 smithi150 ceph-mon[39612]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:16:21.331 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:21 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2051207836' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:16:21.331 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:21 smithi150 ceph-mon[39612]: osdmap e213: 8 total, 8 up, 8 in 2024-09-11T08:16:21.354 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:21.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:21.407 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:21.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:21.508 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:16:21.541 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:21.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:21.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:16:21.642 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:21.672 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:21.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:21.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:21.773 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:21.807 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:21.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:21.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:21.908 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:16:21.943 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:21.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:21.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:16:22.045 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:22.082 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:22.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:22.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:22.184 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:22.215 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:22.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:22.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:22.318 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:22.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:22 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:16:22.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:22 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:16:22.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:22 smithi181 ceph-mon[32701]: pgmap v390: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.33k op/s; 4242/259152 objects misplaced (1.637%); 29 MiB/s, 467 objects/s recovering 2024-09-11T08:16:22.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:22 smithi181 ceph-mon[32701]: osdmap e214: 8 total, 8 up, 8 in 2024-09-11T08:16:22.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:22 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:16:22.354 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:22.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:22.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:22.456 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:22.467 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:22 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:16:22.467 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:22 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:16:22.467 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:22 smithi150 ceph-mon[29276]: pgmap v390: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.33k op/s; 4242/259152 objects misplaced (1.637%); 29 MiB/s, 467 objects/s recovering 2024-09-11T08:16:22.467 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:22 smithi150 ceph-mon[29276]: osdmap e214: 8 total, 8 up, 8 in 2024-09-11T08:16:22.467 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:22 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:16:22.468 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:22 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:16:22.468 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:22 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:16:22.468 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:22 smithi150 ceph-mon[39612]: pgmap v390: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.33k op/s; 4242/259152 objects misplaced (1.637%); 29 MiB/s, 467 objects/s recovering 2024-09-11T08:16:22.468 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:22 smithi150 ceph-mon[39612]: osdmap e214: 8 total, 8 up, 8 in 2024-09-11T08:16:22.468 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:22 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:16:22.503 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:22.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:22.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:22.605 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:22.639 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:22.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:22.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:22.741 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:22.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:22.778 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:22.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:22.879 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:22.911 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:22.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:22.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:23.013 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:23.034 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:23.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:23.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:23.136 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:23.170 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:23.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:23.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:23.272 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:23.309 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:23.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:23.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:23.411 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:23.451 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:23.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:23.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:23.552 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:23.590 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:23.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:23.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:23.692 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:23.727 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:23.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:23.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:23.829 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:16:23.860 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:23.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:23.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:16:23.962 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:16:24.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:24.001 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:24.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:16:24.102 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:24.127 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:24.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:24.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:24.229 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:24.259 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:24.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:24.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:24.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:24 smithi181 ceph-mon[32701]: pgmap v392: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 5.2 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.03k op/s; 4242/255948 objects misplaced (1.657%); 17 MiB/s, 271 objects/s recovering 2024-09-11T08:16:24.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:24 smithi181 ceph-mon[32701]: from='client.25024 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:16:24.361 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:24.382 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:24.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:24.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:24.484 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:24.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:24 smithi150 ceph-mon[29276]: pgmap v392: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 5.2 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.03k op/s; 4242/255948 objects misplaced (1.657%); 17 MiB/s, 271 objects/s recovering 2024-09-11T08:16:24.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:24 smithi150 ceph-mon[29276]: from='client.25024 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:16:24.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:24 smithi150 ceph-mon[39612]: pgmap v392: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 5.2 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.03k op/s; 4242/255948 objects misplaced (1.657%); 17 MiB/s, 271 objects/s recovering 2024-09-11T08:16:24.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:24 smithi150 ceph-mon[39612]: from='client.25024 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:16:24.505 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:24.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:24.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:24.607 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:24.630 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:24.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:24.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:24.732 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:16:24.753 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:24.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:24.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:16:24.825 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:16:24.827 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-09-11T08:16:24.855 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:24.879 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:24.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:24.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:24.980 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:24.981 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:16:24.981 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:16:24.981 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:16:24.981 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:16:24.981 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:16:24.981 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:16:24.981 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:16:24.981 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:16:24.982 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:16:24.982 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:16:24.982 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:16:24.982 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:16:24.982 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:16:24.982 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:16:24.982 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:16:24.982 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:16:25.003 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:25.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:25.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:25.105 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:25.135 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:25.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:25.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:25.238 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:25.268 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:25.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:25.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:25.346 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:25 smithi150 ceph-mon[29276]: osdmap e215: 8 total, 8 up, 8 in 2024-09-11T08:16:25.346 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:25 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/310182126' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:16:25.346 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:25 smithi150 ceph-mon[39612]: osdmap e215: 8 total, 8 up, 8 in 2024-09-11T08:16:25.346 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:25 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/310182126' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:16:25.370 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:25.408 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:25.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:25.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:25.510 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:25.547 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:25.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:25.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:25.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:25 smithi181 ceph-mon[32701]: osdmap e215: 8 total, 8 up, 8 in 2024-09-11T08:16:25.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:25 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/310182126' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:16:25.574 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:16:25 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:16:25] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:16:25.648 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:25.685 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:25.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:25.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:25.788 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:16:25.826 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:25.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:25.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:16:25.927 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:25.957 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:25.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:25.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:26.059 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:26.094 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:26.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:26.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:26.197 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:26.234 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:26.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:26.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:26.336 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:26.373 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:26.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:26.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:26.475 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:16:26.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:26 smithi150 ceph-mon[29276]: pgmap v394: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 922 op/s; 2375/242547 objects misplaced (0.979%); 17 MiB/s, 274 objects/s recovering 2024-09-11T08:16:26.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:26 smithi150 ceph-mon[29276]: osdmap e216: 8 total, 8 up, 8 in 2024-09-11T08:16:26.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:26 smithi150 ceph-mon[39612]: pgmap v394: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 922 op/s; 2375/242547 objects misplaced (0.979%); 17 MiB/s, 274 objects/s recovering 2024-09-11T08:16:26.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:26 smithi150 ceph-mon[39612]: osdmap e216: 8 total, 8 up, 8 in 2024-09-11T08:16:26.512 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:26.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:26.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:16:26.571 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":392,"stamp":"2024-09-11T08:16:22.876636+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5578954996,"num_objects":85316,"num_object_clones":0,"num_object_copies":255948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4242,"num_objects_unfound":0,"num_objects_dirty":85316,"num_whiteouts":0,"num_read":949,"num_read_kb":808,"num_write":173083,"num_write_kb":8246728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14012,"num_bytes_recovered":909971996,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86006,"ondisk_log_size":86006,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":451,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":19447476,"kb_used_data":16989316,"kb_used_omap":19,"kb_used_meta":2457132,"kb_avail":730513740,"statfs":{"total":767960285184,"available":748046069760,"internally_reserved":0,"allocated":17397059584,"data_stored":17393992925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19955,"internal_metadata":2516103693},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-461045760,"num_objects":-7035,"num_object_clones":0,"num_object_copies":-21105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2172,"num_objects_unfound":0,"num_objects_dirty":-7035,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":7035,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1857,"num_bytes_recovered":121700352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.838664"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":192,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.526976+0000","last_change":"2024-09-11T08:12:52.080544+0000","last_active":"2024-09-11T08:16:05.526976+0000","last_peered":"2024-09-11T08:16:05.526976+0000","last_clean":"2024-09-11T08:16:05.526976+0000","last_became_active":"2024-09-11T08:07:30.044976+0000","last_became_peered":"2024-09-11T08:07:30.044976+0000","last_unstale":"2024-09-11T08:16:05.526976+0000","last_undegraded":"2024-09-11T08:16:05.526976+0000","last_fullsized":"2024-09-11T08:16:05.526976+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:12:52.079881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:12:52.079881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.023612166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":208,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-11T08:16:20.074554+0000","last_change":"2024-09-11T08:15:37.106907+0000","last_active":"2024-09-11T08:16:20.074554+0000","last_peered":"2024-09-11T08:16:20.074554+0000","last_clean":"2024-09-11T08:16:20.074554+0000","last_became_active":"2024-09-11T08:07:32.048048+0000","last_became_peered":"2024-09-11T08:07:32.048048+0000","last_unstale":"2024-09-11T08:16:20.074554+0000","last_undegraded":"2024-09-11T08:16:20.074554+0000","last_fullsized":"2024-09-11T08:16:20.074554+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:15:37.106824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:15:37.106824+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:39.764009+0000","scrub_duration":0.0038285900000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":235,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.520843+0000","last_change":"2024-09-11T08:15:56.531271+0000","last_active":"2024-09-11T08:16:05.520843+0000","last_peered":"2024-09-11T08:16:05.520843+0000","last_clean":"2024-09-11T08:16:05.520843+0000","last_became_active":"2024-09-11T08:15:56.530946+0000","last_became_peered":"2024-09-11T08:15:56.530946+0000","last_unstale":"2024-09-11T08:16:05.520843+0000","last_undegraded":"2024-09-11T08:16:05.520843+0000","last_fullsized":"2024-09-11T08:16:05.520843+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:57.020218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:14:57.020218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:21.588027+0000","scrub_duration":0.0034881460000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":188,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-11T08:16:20.074661+0000","last_change":"2024-09-11T08:13:10.073583+0000","last_active":"2024-09-11T08:16:20.074661+0000","last_peered":"2024-09-11T08:16:20.074661+0000","last_clean":"2024-09-11T08:16:20.074661+0000","last_became_active":"2024-09-11T08:07:36.074789+0000","last_became_peered":"2024-09-11T08:07:36.074789+0000","last_unstale":"2024-09-11T08:16:20.074661+0000","last_undegraded":"2024-09-11T08:16:20.074661+0000","last_fullsized":"2024-09-11T08:16:20.074661+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:10.073490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:02.470649+0000","last_clean_scrub_stamp":"2024-09-11T08:13:10.073490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0098908899999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":213,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.515994+0000","last_change":"2024-09-11T08:15:56.521398+0000","last_active":"2024-09-11T08:16:05.515994+0000","last_peered":"2024-09-11T08:16:05.515994+0000","last_clean":"2024-09-11T08:16:05.515994+0000","last_became_active":"2024-09-11T08:15:56.521104+0000","last_became_peered":"2024-09-11T08:15:56.521104+0000","last_unstale":"2024-09-11T08:16:05.515994+0000","last_undegraded":"2024-09-11T08:16:05.515994+0000","last_fullsized":"2024-09-11T08:16:05.515994+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:58.222356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:14:58.222356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013526233,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":192,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.517025+0000","last_change":"2024-09-11T08:15:31.254961+0000","last_active":"2024-09-11T08:16:05.517025+0000","last_peered":"2024-09-11T08:16:05.517025+0000","last_clean":"2024-09-11T08:16:05.517025+0000","last_became_active":"2024-09-11T08:07:30.045579+0000","last_became_peered":"2024-09-11T08:07:30.045579+0000","last_unstale":"2024-09-11T08:16:05.517025+0000","last_undegraded":"2024-09-11T08:16:05.517025+0000","last_fullsized":"2024-09-11T08:16:05.517025+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:31.254885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:31.254885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:36.877052+0000","scrub_duration":0.0033689100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"145'22","reported_seq":232,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.515677+0000","last_change":"2024-09-11T08:15:57.478927+0000","last_active":"2024-09-11T08:16:05.515677+0000","last_peered":"2024-09-11T08:16:05.515677+0000","last_clean":"2024-09-11T08:16:05.515677+0000","last_became_active":"2024-09-11T08:07:32.045848+0000","last_became_peered":"2024-09-11T08:07:32.045848+0000","last_unstale":"2024-09-11T08:16:05.515677+0000","last_undegraded":"2024-09-11T08:16:05.515677+0000","last_fullsized":"2024-09-11T08:16:05.515677+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'22","last_scrub_stamp":"2024-09-11T08:15:57.478870+0000","last_deep_scrub":"145'22","last_deep_scrub_stamp":"2024-09-11T08:15:57.478870+0000","last_clean_scrub_stamp":"2024-09-11T08:15:57.478870+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:13.662088+0000","scrub_duration":0.0021529710000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":190,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-11T08:16:20.074474+0000","last_change":"2024-09-11T08:15:52.050101+0000","last_active":"2024-09-11T08:16:20.074474+0000","last_peered":"2024-09-11T08:16:20.074474+0000","last_clean":"2024-09-11T08:16:20.074474+0000","last_became_active":"2024-09-11T08:07:34.052243+0000","last_became_peered":"2024-09-11T08:07:34.052243+0000","last_unstale":"2024-09-11T08:16:20.074474+0000","last_undegraded":"2024-09-11T08:16:20.074474+0000","last_fullsized":"2024-09-11T08:16:20.074474+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:52.050017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:15:52.050017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:20.035807+0000","scrub_duration":0.0037049880000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":204,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.527761+0000","last_change":"2024-09-11T08:15:31.550213+0000","last_active":"2024-09-11T08:16:05.527761+0000","last_peered":"2024-09-11T08:16:05.527761+0000","last_clean":"2024-09-11T08:16:05.527761+0000","last_became_active":"2024-09-11T08:07:30.043950+0000","last_became_peered":"2024-09-11T08:07:30.043950+0000","last_unstale":"2024-09-11T08:16:05.527761+0000","last_undegraded":"2024-09-11T08:16:05.527761+0000","last_fullsized":"2024-09-11T08:16:05.527761+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:31.550091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:54.878409+0000","last_clean_scrub_stamp":"2024-09-11T08:15:31.550091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:56.355754+0000","scrub_duration":0.0017129999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":208,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-11T08:16:20.074901+0000","last_change":"2024-09-11T08:13:04.107341+0000","last_active":"2024-09-11T08:16:20.074901+0000","last_peered":"2024-09-11T08:16:20.074901+0000","last_clean":"2024-09-11T08:16:20.074901+0000","last_became_active":"2024-09-11T08:07:32.044626+0000","last_became_peered":"2024-09-11T08:07:32.044626+0000","last_unstale":"2024-09-11T08:16:20.074901+0000","last_undegraded":"2024-09-11T08:16:20.074901+0000","last_fullsized":"2024-09-11T08:16:20.074901+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:13:04.107254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:13:04.107254+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0056027890000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":224,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.526163+0000","last_change":"2024-09-11T08:15:56.531009+0000","last_active":"2024-09-11T08:16:05.526163+0000","last_peered":"2024-09-11T08:16:05.526163+0000","last_clean":"2024-09-11T08:16:05.526163+0000","last_became_active":"2024-09-11T08:15:56.530680+0000","last_became_peered":"2024-09-11T08:15:56.530680+0000","last_unstale":"2024-09-11T08:16:05.526163+0000","last_undegraded":"2024-09-11T08:16:05.526163+0000","last_fullsized":"2024-09-11T08:16:05.526163+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:50.469752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:15:50.469752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:08.638035+0000","scrub_duration":0.0033416409999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":203,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.524907+0000","last_change":"2024-09-11T08:14:53.516662+0000","last_active":"2024-09-11T08:16:05.524907+0000","last_peered":"2024-09-11T08:16:05.524907+0000","last_clean":"2024-09-11T08:16:05.524907+0000","last_became_active":"2024-09-11T08:14:24.792595+0000","last_became_peered":"2024-09-11T08:14:24.792595+0000","last_unstale":"2024-09-11T08:16:05.524907+0000","last_undegraded":"2024-09-11T08:16:05.524907+0000","last_fullsized":"2024-09-11T08:16:05.524907+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:53.516583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:14:53.516583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034487419999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":201,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.526989+0000","last_change":"2024-09-11T08:14:55.444867+0000","last_active":"2024-09-11T08:16:05.526989+0000","last_peered":"2024-09-11T08:16:05.526989+0000","last_clean":"2024-09-11T08:16:05.526989+0000","last_became_active":"2024-09-11T08:07:30.043277+0000","last_became_peered":"2024-09-11T08:07:30.043277+0000","last_unstale":"2024-09-11T08:16:05.526989+0000","last_undegraded":"2024-09-11T08:16:05.526989+0000","last_fullsized":"2024-09-11T08:16:05.526989+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:55.444806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:14:55.444806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0099859159999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"139'7","reported_seq":204,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-11T08:16:20.075636+0000","last_change":"2024-09-11T08:15:02.988616+0000","last_active":"2024-09-11T08:16:20.075636+0000","last_peered":"2024-09-11T08:16:20.075636+0000","last_clean":"2024-09-11T08:16:20.075636+0000","last_became_active":"2024-09-11T08:07:32.048661+0000","last_became_peered":"2024-09-11T08:07:32.048661+0000","last_unstale":"2024-09-11T08:16:20.075636+0000","last_undegraded":"2024-09-11T08:16:20.075636+0000","last_fullsized":"2024-09-11T08:16:20.075636+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'7","last_scrub_stamp":"2024-09-11T08:15:02.988425+0000","last_deep_scrub":"139'7","last_deep_scrub_stamp":"2024-09-11T08:12:24.869318+0000","last_clean_scrub_stamp":"2024-09-11T08:15:02.988425+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033901550000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":16,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-11T08:16:20.086427+0000","last_change":"2024-09-11T08:15:56.527328+0000","last_active":"2024-09-11T08:16:20.086427+0000","last_peered":"2024-09-11T08:16:20.086427+0000","last_clean":"2024-09-11T08:16:20.086427+0000","last_became_active":"2024-09-11T08:15:56.526954+0000","last_became_peered":"2024-09-11T08:15:56.526954+0000","last_unstale":"2024-09-11T08:16:20.086427+0000","last_undegraded":"2024-09-11T08:16:20.086427+0000","last_fullsized":"2024-09-11T08:16:20.086427+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:00.885045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:14:00.885045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":205,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.526564+0000","last_change":"2024-09-11T08:15:55.371107+0000","last_active":"2024-09-11T08:16:05.526564+0000","last_peered":"2024-09-11T08:16:05.526564+0000","last_clean":"2024-09-11T08:16:05.526564+0000","last_became_active":"2024-09-11T08:07:36.077602+0000","last_became_peered":"2024-09-11T08:07:36.077602+0000","last_unstale":"2024-09-11T08:16:05.526564+0000","last_undegraded":"2024-09-11T08:16:05.526564+0000","last_fullsized":"2024-09-11T08:16:05.526564+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:55.371030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:04.782577+0000","last_clean_scrub_stamp":"2024-09-11T08:15:55.371030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:57.415606+0000","scrub_duration":0.0028204950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":188,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-11T08:16:20.074773+0000","last_change":"2024-09-11T08:13:11.061617+0000","last_active":"2024-09-11T08:16:20.074773+0000","last_peered":"2024-09-11T08:16:20.074773+0000","last_clean":"2024-09-11T08:16:20.074773+0000","last_became_active":"2024-09-11T08:07:36.633338+0000","last_became_peered":"2024-09-11T08:07:36.633338+0000","last_unstale":"2024-09-11T08:16:20.074773+0000","last_undegraded":"2024-09-11T08:16:20.074773+0000","last_fullsized":"2024-09-11T08:16:20.074773+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:11.061558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:13:11.061558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016408200000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":219,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.525741+0000","last_change":"2024-09-11T08:15:56.517613+0000","last_active":"2024-09-11T08:16:05.525741+0000","last_peered":"2024-09-11T08:16:05.525741+0000","last_clean":"2024-09-11T08:16:05.525741+0000","last_became_active":"2024-09-11T08:15:56.516941+0000","last_became_peered":"2024-09-11T08:15:56.516941+0000","last_unstale":"2024-09-11T08:16:05.525741+0000","last_undegraded":"2024-09-11T08:16:05.525741+0000","last_fullsized":"2024-09-11T08:16:05.525741+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:13.315410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:14:13.315410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030491400000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"107'22","reported_seq":241,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.533403+0000","last_change":"2024-09-11T08:16:04.159388+0000","last_active":"2024-09-11T08:16:05.533403+0000","last_peered":"2024-09-11T08:16:05.533403+0000","last_clean":"2024-09-11T08:16:05.533403+0000","last_became_active":"2024-09-11T08:07:32.046363+0000","last_became_peered":"2024-09-11T08:07:32.046363+0000","last_unstale":"2024-09-11T08:16:05.533403+0000","last_undegraded":"2024-09-11T08:16:05.533403+0000","last_fullsized":"2024-09-11T08:16:05.533403+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'22","last_scrub_stamp":"2024-09-11T08:16:04.159211+0000","last_deep_scrub":"107'22","last_deep_scrub_stamp":"2024-09-11T08:16:04.159211+0000","last_clean_scrub_stamp":"2024-09-11T08:16:04.159211+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:30.394200+0000","scrub_duration":0.0069586229999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":208,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.522421+0000","last_change":"2024-09-11T08:15:46.597272+0000","last_active":"2024-09-11T08:16:05.522421+0000","last_peered":"2024-09-11T08:16:05.522421+0000","last_clean":"2024-09-11T08:16:05.522421+0000","last_became_active":"2024-09-11T08:07:34.055410+0000","last_became_peered":"2024-09-11T08:07:34.055410+0000","last_unstale":"2024-09-11T08:16:05.522421+0000","last_undegraded":"2024-09-11T08:16:05.522421+0000","last_fullsized":"2024-09-11T08:16:05.522421+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:46.597198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:10:31.418873+0000","last_clean_scrub_stamp":"2024-09-11T08:15:46.597198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:00.695980+0000","scrub_duration":0.0032858620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":179,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.519208+0000","last_change":"2024-09-11T08:15:58.526429+0000","last_active":"2024-09-11T08:16:05.519208+0000","last_peered":"2024-09-11T08:16:05.519208+0000","last_clean":"2024-09-11T08:16:05.519208+0000","last_became_active":"2024-09-11T08:07:36.633258+0000","last_became_peered":"2024-09-11T08:07:36.633258+0000","last_unstale":"2024-09-11T08:16:05.519208+0000","last_undegraded":"2024-09-11T08:16:05.519208+0000","last_fullsized":"2024-09-11T08:16:05.519208+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:12:58.260306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:12:58.260306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015796729999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":204,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-11T08:16:20.075477+0000","last_change":"2024-09-11T08:15:48.081690+0000","last_active":"2024-09-11T08:16:20.075477+0000","last_peered":"2024-09-11T08:16:20.075477+0000","last_clean":"2024-09-11T08:16:20.075477+0000","last_became_active":"2024-09-11T08:07:30.046284+0000","last_became_peered":"2024-09-11T08:07:30.046284+0000","last_unstale":"2024-09-11T08:16:20.075477+0000","last_undegraded":"2024-09-11T08:16:20.075477+0000","last_fullsized":"2024-09-11T08:16:20.075477+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:48.081591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:48.081591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:03.932824+0000","scrub_duration":0.0059702619999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":245,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.516194+0000","last_change":"2024-09-11T08:15:53.518868+0000","last_active":"2024-09-11T08:16:05.516194+0000","last_peered":"2024-09-11T08:16:05.516194+0000","last_clean":"2024-09-11T08:16:05.516194+0000","last_became_active":"2024-09-11T08:09:07.878432+0000","last_became_peered":"2024-09-11T08:09:07.878432+0000","last_unstale":"2024-09-11T08:16:05.516194+0000","last_undegraded":"2024-09-11T08:16:05.516194+0000","last_fullsized":"2024-09-11T08:16:05.516194+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-11T08:15:53.518765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:15:53.518765+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:19.946047+0000","scrub_duration":0.0047796499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":190,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-11T08:16:20.076144+0000","last_change":"2024-09-11T08:13:06.131006+0000","last_active":"2024-09-11T08:16:20.076144+0000","last_peered":"2024-09-11T08:16:20.076144+0000","last_clean":"2024-09-11T08:16:20.076144+0000","last_became_active":"2024-09-11T08:07:34.056528+0000","last_became_peered":"2024-09-11T08:07:34.056528+0000","last_unstale":"2024-09-11T08:16:20.076144+0000","last_undegraded":"2024-09-11T08:16:20.076144+0000","last_fullsized":"2024-09-11T08:16:20.076144+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:06.130952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:10:40.110490+0000","last_clean_scrub_stamp":"2024-09-11T08:13:06.130952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050011930000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":143,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-11T08:16:20.086963+0000","last_change":"2024-09-11T08:15:03.838012+0000","last_active":"2024-09-11T08:16:20.086963+0000","last_peered":"2024-09-11T08:16:20.086963+0000","last_clean":"2024-09-11T08:16:20.086963+0000","last_became_active":"2024-09-11T08:14:24.791610+0000","last_became_peered":"2024-09-11T08:14:24.791610+0000","last_unstale":"2024-09-11T08:16:20.086963+0000","last_undegraded":"2024-09-11T08:16:20.086963+0000","last_fullsized":"2024-09-11T08:16:20.086963+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-11T08:15:03.837911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:15:03.837911+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:29.972319+0000","scrub_duration":0.0035450270000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":202,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.521557+0000","last_change":"2024-09-11T08:14:13.134615+0000","last_active":"2024-09-11T08:16:05.521557+0000","last_peered":"2024-09-11T08:16:05.521557+0000","last_clean":"2024-09-11T08:16:05.521557+0000","last_became_active":"2024-09-11T08:07:30.045932+0000","last_became_peered":"2024-09-11T08:07:30.045932+0000","last_unstale":"2024-09-11T08:16:05.521557+0000","last_undegraded":"2024-09-11T08:16:05.521557+0000","last_fullsized":"2024-09-11T08:16:05.521557+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:13.134086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:14:13.134086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0059195080000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":278,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.538670+0000","last_change":"2024-09-11T08:15:56.536927+0000","last_active":"2024-09-11T08:16:05.538670+0000","last_peered":"2024-09-11T08:16:05.538670+0000","last_clean":"2024-09-11T08:16:05.538670+0000","last_became_active":"2024-09-11T08:15:56.515817+0000","last_became_peered":"2024-09-11T08:15:56.515817+0000","last_unstale":"2024-09-11T08:16:05.538670+0000","last_undegraded":"2024-09-11T08:16:05.538670+0000","last_fullsized":"2024-09-11T08:16:05.538670+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-11T08:14:00.393362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:14:00.393362+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018583250000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"209'11","reported_seq":362,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-11T08:16:20.081153+0000","last_change":"2024-09-11T08:15:56.541377+0000","last_active":"2024-09-11T08:16:20.081153+0000","last_peered":"2024-09-11T08:16:20.081153+0000","last_clean":"2024-09-11T08:16:20.081153+0000","last_became_active":"2024-09-11T08:15:56.525498+0000","last_became_peered":"2024-09-11T08:15:56.525498+0000","last_unstale":"2024-09-11T08:16:20.081153+0000","last_undegraded":"2024-09-11T08:16:20.081153+0000","last_fullsized":"2024-09-11T08:16:20.081153+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"156'10","last_scrub_stamp":"2024-09-11T08:15:01.002908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:15:01.002908+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026556840000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"212'10728","reported_seq":10789,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-11T08:16:18.192653+0000","last_change":"2024-09-11T08:13:54.883961+0000","last_active":"2024-09-11T08:16:18.192653+0000","last_peered":"2024-09-11T08:16:18.192653+0000","last_clean":"2024-09-11T08:16:18.192653+0000","last_became_active":"2024-09-11T08:13:54.883685+0000","last_became_peered":"2024-09-11T08:13:54.883685+0000","last_unstale":"2024-09-11T08:16:18.192653+0000","last_undegraded":"2024-09-11T08:16:18.192653+0000","last_fullsized":"2024-09-11T08:16:18.192653+0000","mapping_epoch":159,"log_start":"191'5700","ondisk_log_start":"191'5700","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:13:53.862306+0000","objects_scrubbed":0,"log_size":5028,"log_dups_size":0,"ondisk_log_size":5028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":355205120,"num_objects":5420,"num_object_clones":0,"num_object_copies":16260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10728,"num_write_kb":516736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":235,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.521805+0000","last_change":"2024-09-11T08:15:42.534547+0000","last_active":"2024-09-11T08:16:05.521805+0000","last_peered":"2024-09-11T08:16:05.521805+0000","last_clean":"2024-09-11T08:16:05.521805+0000","last_became_active":"2024-09-11T08:07:32.054317+0000","last_became_peered":"2024-09-11T08:07:32.054317+0000","last_unstale":"2024-09-11T08:16:05.521805+0000","last_undegraded":"2024-09-11T08:16:05.521805+0000","last_fullsized":"2024-09-11T08:16:05.521805+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-11T08:15:42.534460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:15:42.534460+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:51.887358+0000","scrub_duration":0.0078017169999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":212,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.515526+0000","last_change":"2024-09-11T08:15:59.503489+0000","last_active":"2024-09-11T08:16:05.515526+0000","last_peered":"2024-09-11T08:16:05.515526+0000","last_clean":"2024-09-11T08:16:05.515526+0000","last_became_active":"2024-09-11T08:15:56.521943+0000","last_became_peered":"2024-09-11T08:15:56.521943+0000","last_unstale":"2024-09-11T08:16:05.515526+0000","last_undegraded":"2024-09-11T08:16:05.515526+0000","last_fullsized":"2024-09-11T08:16:05.515526+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:50.951830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:13:50.951830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011546670000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":304,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-11T08:16:20.074842+0000","last_change":"2024-09-11T08:13:12.092663+0000","last_active":"2024-09-11T08:16:20.074842+0000","last_peered":"2024-09-11T08:16:20.074842+0000","last_clean":"2024-09-11T08:16:20.074842+0000","last_became_active":"2024-09-11T08:07:34.052981+0000","last_became_peered":"2024-09-11T08:07:34.052981+0000","last_unstale":"2024-09-11T08:16:20.074842+0000","last_undegraded":"2024-09-11T08:16:20.074842+0000","last_fullsized":"2024-09-11T08:16:20.074842+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:13:12.092586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:13:12.092586+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012246072,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":205,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.527363+0000","last_change":"2024-09-11T08:15:45.411776+0000","last_active":"2024-09-11T08:16:05.527363+0000","last_peered":"2024-09-11T08:16:05.527363+0000","last_clean":"2024-09-11T08:16:05.527363+0000","last_became_active":"2024-09-11T08:07:36.073969+0000","last_became_peered":"2024-09-11T08:07:36.073969+0000","last_unstale":"2024-09-11T08:16:05.527363+0000","last_undegraded":"2024-09-11T08:16:05.527363+0000","last_fullsized":"2024-09-11T08:16:05.527363+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:45.411593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:06.616332+0000","last_clean_scrub_stamp":"2024-09-11T08:15:45.411593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:06.019721+0000","scrub_duration":0.0031499940000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"212'10631","reported_seq":16477,"reported_epoch":212,"state":"active+remapped+backfilling","last_fresh":"2024-09-11T08:16:18.260814+0000","last_change":"2024-09-11T08:15:57.607034+0000","last_active":"2024-09-11T08:16:18.260814+0000","last_peered":"2024-09-11T08:16:18.260814+0000","last_clean":"2024-09-11T08:15:55.461733+0000","last_became_active":"2024-09-11T08:15:57.579110+0000","last_became_peered":"2024-09-11T08:15:57.579110+0000","last_unstale":"2024-09-11T08:16:18.260814+0000","last_undegraded":"2024-09-11T08:16:18.260814+0000","last_fullsized":"2024-09-11T08:16:18.260814+0000","mapping_epoch":208,"log_start":"183'4732","ondisk_log_start":"183'4732","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:13:53.862306+0000","objects_scrubbed":0,"log_size":5899,"log_dups_size":0,"ondisk_log_size":5899,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":351469568,"num_objects":5363,"num_object_clones":0,"num_object_copies":16089,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2818,"num_objects_unfound":0,"num_objects_dirty":5363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10631,"num_write_kb":511808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2886,"num_bytes_recovered":189136896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,5],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":5363}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":285,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.515491+0000","last_change":"2024-09-11T08:15:56.542573+0000","last_active":"2024-09-11T08:16:05.515491+0000","last_peered":"2024-09-11T08:16:05.515491+0000","last_clean":"2024-09-11T08:16:05.515491+0000","last_became_active":"2024-09-11T08:15:56.517680+0000","last_became_peered":"2024-09-11T08:15:56.517680+0000","last_unstale":"2024-09-11T08:16:05.515491+0000","last_undegraded":"2024-09-11T08:16:05.515491+0000","last_fullsized":"2024-09-11T08:16:05.515491+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-11T08:13:58.979187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:13:58.979187+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001885091,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":212,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.519559+0000","last_change":"2024-09-11T08:16:05.435966+0000","last_active":"2024-09-11T08:16:05.519559+0000","last_peered":"2024-09-11T08:16:05.519559+0000","last_clean":"2024-09-11T08:16:05.519559+0000","last_became_active":"2024-09-11T08:15:56.527438+0000","last_became_peered":"2024-09-11T08:15:56.527438+0000","last_unstale":"2024-09-11T08:16:05.519559+0000","last_undegraded":"2024-09-11T08:16:05.519559+0000","last_fullsized":"2024-09-11T08:16:05.519559+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:46.896181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:13:46.896181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035672669999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":217,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-11T08:16:20.074755+0000","last_change":"2024-09-11T08:15:56.522907+0000","last_active":"2024-09-11T08:16:20.074755+0000","last_peered":"2024-09-11T08:16:20.074755+0000","last_clean":"2024-09-11T08:16:20.074755+0000","last_became_active":"2024-09-11T08:15:56.522606+0000","last_became_peered":"2024-09-11T08:15:56.522606+0000","last_unstale":"2024-09-11T08:16:20.074755+0000","last_undegraded":"2024-09-11T08:16:20.074755+0000","last_fullsized":"2024-09-11T08:16:20.074755+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:57.840457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:13:57.840457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014582009999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":188,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-11T08:16:20.074832+0000","last_change":"2024-09-11T08:14:04.937966+0000","last_active":"2024-09-11T08:16:20.074832+0000","last_peered":"2024-09-11T08:16:20.074832+0000","last_clean":"2024-09-11T08:16:20.074832+0000","last_became_active":"2024-09-11T08:07:36.069225+0000","last_became_peered":"2024-09-11T08:07:36.069225+0000","last_unstale":"2024-09-11T08:16:20.074832+0000","last_undegraded":"2024-09-11T08:16:20.074832+0000","last_fullsized":"2024-09-11T08:16:20.074832+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:04.937884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:04.937884+0000","last_clean_scrub_stamp":"2024-09-11T08:14:04.937884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018777710000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"212'10669","reported_seq":20248,"reported_epoch":212,"state":"active+remapped+backfilling","last_fresh":"2024-09-11T08:16:17.902269+0000","last_change":"2024-09-11T08:15:57.602250+0000","last_active":"2024-09-11T08:16:17.902269+0000","last_peered":"2024-09-11T08:16:17.902269+0000","last_clean":"2024-09-11T08:15:55.508898+0000","last_became_active":"2024-09-11T08:15:57.581916+0000","last_became_peered":"2024-09-11T08:15:57.581916+0000","last_unstale":"2024-09-11T08:16:17.902269+0000","last_undegraded":"2024-09-11T08:16:17.902269+0000","last_fullsized":"2024-09-11T08:16:17.902269+0000","mapping_epoch":208,"log_start":"188'5400","ondisk_log_start":"188'5400","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'8832","last_scrub_stamp":"2024-09-11T08:15:51.203402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:15:51.203402+0000","objects_scrubbed":7454,"log_size":5269,"log_dups_size":0,"ondisk_log_size":5269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:58.659559+0000","scrub_duration":2.7862675239999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":359661592,"num_objects":5489,"num_object_clones":0,"num_object_copies":16467,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1424,"num_objects_unfound":0,"num_objects_dirty":5489,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10669,"num_write_kb":516993,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4598,"num_bytes_recovered":301269016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,2,6],"avail_no_missing":["5","2","6"],"object_location_counts":[{"shards":"2,5,6","objects":5489}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":213,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-11T08:16:20.075018+0000","last_change":"2024-09-11T08:14:59.014008+0000","last_active":"2024-09-11T08:16:20.075018+0000","last_peered":"2024-09-11T08:16:20.075018+0000","last_clean":"2024-09-11T08:16:20.075018+0000","last_became_active":"2024-09-11T08:07:32.040069+0000","last_became_peered":"2024-09-11T08:07:32.040069+0000","last_unstale":"2024-09-11T08:16:20.075018+0000","last_undegraded":"2024-09-11T08:16:20.075018+0000","last_fullsized":"2024-09-11T08:16:20.075018+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-11T08:14:59.013949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:14:59.013949+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001898156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":202,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.538770+0000","last_change":"2024-09-11T08:15:54.794370+0000","last_active":"2024-09-11T08:16:05.538770+0000","last_peered":"2024-09-11T08:16:05.538770+0000","last_clean":"2024-09-11T08:16:05.538770+0000","last_became_active":"2024-09-11T08:07:30.050484+0000","last_became_peered":"2024-09-11T08:07:30.050484+0000","last_unstale":"2024-09-11T08:16:05.538770+0000","last_undegraded":"2024-09-11T08:16:05.538770+0000","last_fullsized":"2024-09-11T08:16:05.538770+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:54.793950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:54.793950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:06.106542+0000","scrub_duration":0.0059264870000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"82'9","reported_seq":315,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:13.653321+0000","last_change":"2024-09-11T08:15:51.456046+0000","last_active":"2024-09-11T08:16:13.653321+0000","last_peered":"2024-09-11T08:16:13.653321+0000","last_clean":"2024-09-11T08:16:13.653321+0000","last_became_active":"2024-09-11T08:07:34.056143+0000","last_became_peered":"2024-09-11T08:07:34.056143+0000","last_unstale":"2024-09-11T08:16:13.653321+0000","last_undegraded":"2024-09-11T08:16:13.653321+0000","last_fullsized":"2024-09-11T08:16:13.653321+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-11T08:15:51.455966+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-09-11T08:13:10.230703+0000","last_clean_scrub_stamp":"2024-09-11T08:15:51.455966+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:01.331170+0000","scrub_duration":0.00915995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":178,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-11T08:16:20.086810+0000","last_change":"2024-09-11T08:15:33.507691+0000","last_active":"2024-09-11T08:16:20.086810+0000","last_peered":"2024-09-11T08:16:20.086810+0000","last_clean":"2024-09-11T08:16:20.086810+0000","last_became_active":"2024-09-11T08:14:24.788968+0000","last_became_peered":"2024-09-11T08:14:24.788968+0000","last_unstale":"2024-09-11T08:16:20.086810+0000","last_undegraded":"2024-09-11T08:16:20.086810+0000","last_fullsized":"2024-09-11T08:16:20.086810+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:33.507613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:15:33.507613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:55.580423+0000","scrub_duration":0.0032985240000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.e","version":"212'10643","reported_seq":10704,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-11T08:16:18.226600+0000","last_change":"2024-09-11T08:13:55.705967+0000","last_active":"2024-09-11T08:16:18.226600+0000","last_peered":"2024-09-11T08:16:18.226600+0000","last_clean":"2024-09-11T08:16:18.226600+0000","last_became_active":"2024-09-11T08:13:55.705670+0000","last_became_peered":"2024-09-11T08:13:55.705670+0000","last_unstale":"2024-09-11T08:16:18.226600+0000","last_undegraded":"2024-09-11T08:16:18.226600+0000","last_fullsized":"2024-09-11T08:16:18.226600+0000","mapping_epoch":159,"log_start":"183'4700","ondisk_log_start":"183'4700","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:13:53.862306+0000","objects_scrubbed":0,"log_size":5943,"log_dups_size":0,"ondisk_log_size":5943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":347537408,"num_objects":5303,"num_object_clones":0,"num_object_copies":15909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10643,"num_write_kb":510272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":218,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.521127+0000","last_change":"2024-09-11T08:15:36.365980+0000","last_active":"2024-09-11T08:16:05.521127+0000","last_peered":"2024-09-11T08:16:05.521127+0000","last_clean":"2024-09-11T08:16:05.521127+0000","last_became_active":"2024-09-11T08:07:32.054294+0000","last_became_peered":"2024-09-11T08:07:32.054294+0000","last_unstale":"2024-09-11T08:16:05.521127+0000","last_undegraded":"2024-09-11T08:16:05.521127+0000","last_fullsized":"2024-09-11T08:16:05.521127+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-11T08:15:36.365891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:15:36.365891+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:05.688088+0000","scrub_duration":0.003716071,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":201,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.527797+0000","last_change":"2024-09-11T08:15:46.459225+0000","last_active":"2024-09-11T08:16:05.527797+0000","last_peered":"2024-09-11T08:16:05.527797+0000","last_clean":"2024-09-11T08:16:05.527797+0000","last_became_active":"2024-09-11T08:07:30.047497+0000","last_became_peered":"2024-09-11T08:07:30.047497+0000","last_unstale":"2024-09-11T08:16:05.527797+0000","last_undegraded":"2024-09-11T08:16:05.527797+0000","last_fullsized":"2024-09-11T08:16:05.527797+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:46.459143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:46.459143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:12.469916+0000","scrub_duration":0.0057785019999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"190'9","reported_seq":50,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:13.662703+0000","last_change":"2024-09-11T08:15:08.769903+0000","last_active":"2024-09-11T08:16:13.662703+0000","last_peered":"2024-09-11T08:16:13.662703+0000","last_clean":"2024-09-11T08:16:13.662703+0000","last_became_active":"2024-09-11T08:15:08.764876+0000","last_became_peered":"2024-09-11T08:15:08.764876+0000","last_unstale":"2024-09-11T08:16:13.662703+0000","last_undegraded":"2024-09-11T08:16:13.662703+0000","last_fullsized":"2024-09-11T08:16:13.662703+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:14:11.320174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:14:11.320174+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017353449999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"66'1","reported_seq":187,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.525031+0000","last_change":"2024-09-11T08:15:06.445409+0000","last_active":"2024-09-11T08:16:05.525031+0000","last_peered":"2024-09-11T08:16:05.525031+0000","last_clean":"2024-09-11T08:16:05.525031+0000","last_became_active":"2024-09-11T08:07:36.072593+0000","last_became_peered":"2024-09-11T08:07:36.072593+0000","last_unstale":"2024-09-11T08:16:05.525031+0000","last_undegraded":"2024-09-11T08:16:05.525031+0000","last_fullsized":"2024-09-11T08:16:05.525031+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-11T08:15:06.445301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:15:06.445301+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013971574,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.f","version":"212'10664","reported_seq":10725,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-11T08:16:18.058526+0000","last_change":"2024-09-11T08:13:55.706681+0000","last_active":"2024-09-11T08:16:18.058526+0000","last_peered":"2024-09-11T08:16:18.058526+0000","last_clean":"2024-09-11T08:16:18.058526+0000","last_became_active":"2024-09-11T08:13:55.706134+0000","last_became_peered":"2024-09-11T08:13:55.706134+0000","last_unstale":"2024-09-11T08:16:18.058526+0000","last_undegraded":"2024-09-11T08:16:18.058526+0000","last_fullsized":"2024-09-11T08:16:18.058526+0000","mapping_epoch":159,"log_start":"191'5809","ondisk_log_start":"191'5809","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:13:53.862306+0000","objects_scrubbed":0,"log_size":4855,"log_dups_size":0,"ondisk_log_size":4855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":354942976,"num_objects":5416,"num_object_clones":0,"num_object_copies":16248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10664,"num_write_kb":514560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":289,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.525533+0000","last_change":"2024-09-11T08:15:56.540181+0000","last_active":"2024-09-11T08:16:05.525533+0000","last_peered":"2024-09-11T08:16:05.525533+0000","last_clean":"2024-09-11T08:16:05.525533+0000","last_became_active":"2024-09-11T08:15:56.518384+0000","last_became_peered":"2024-09-11T08:15:56.518384+0000","last_unstale":"2024-09-11T08:16:05.525533+0000","last_undegraded":"2024-09-11T08:16:05.525533+0000","last_fullsized":"2024-09-11T08:16:05.525533+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-11T08:13:57.387660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:13:57.387660+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025295550000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":212,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.523487+0000","last_change":"2024-09-11T08:15:54.517507+0000","last_active":"2024-09-11T08:16:05.523487+0000","last_peered":"2024-09-11T08:16:05.523487+0000","last_clean":"2024-09-11T08:16:05.523487+0000","last_became_active":"2024-09-11T08:07:30.045380+0000","last_became_peered":"2024-09-11T08:07:30.045380+0000","last_unstale":"2024-09-11T08:16:05.523487+0000","last_undegraded":"2024-09-11T08:16:05.523487+0000","last_fullsized":"2024-09-11T08:16:05.523487+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:54.517429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:54.517429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:05.165550+0000","scrub_duration":0.0055168639999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":209,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-11T08:16:20.080192+0000","last_change":"2024-09-11T08:15:54.238757+0000","last_active":"2024-09-11T08:16:20.080192+0000","last_peered":"2024-09-11T08:16:20.080192+0000","last_clean":"2024-09-11T08:16:20.080192+0000","last_became_active":"2024-09-11T08:07:34.051004+0000","last_became_peered":"2024-09-11T08:07:34.051004+0000","last_unstale":"2024-09-11T08:16:20.080192+0000","last_undegraded":"2024-09-11T08:16:20.080192+0000","last_fullsized":"2024-09-11T08:16:20.080192+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:54.238680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:15:54.238680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:58.812475+0000","scrub_duration":0.0019626420000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":196,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.520761+0000","last_change":"2024-09-11T08:14:09.068749+0000","last_active":"2024-09-11T08:16:05.520761+0000","last_peered":"2024-09-11T08:16:05.520761+0000","last_clean":"2024-09-11T08:16:05.520761+0000","last_became_active":"2024-09-11T08:07:36.631668+0000","last_became_peered":"2024-09-11T08:07:36.631668+0000","last_unstale":"2024-09-11T08:16:05.520761+0000","last_undegraded":"2024-09-11T08:16:05.520761+0000","last_fullsized":"2024-09-11T08:16:05.520761+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:09.068694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:14:09.068694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033190699999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"212'10892","reported_seq":15524,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-11T08:16:17.869167+0000","last_change":"2024-09-11T08:15:56.003569+0000","last_active":"2024-09-11T08:16:17.869167+0000","last_peered":"2024-09-11T08:16:17.869167+0000","last_clean":"2024-09-11T08:16:17.869167+0000","last_became_active":"2024-09-11T08:14:50.875690+0000","last_became_peered":"2024-09-11T08:14:50.875690+0000","last_unstale":"2024-09-11T08:16:17.869167+0000","last_undegraded":"2024-09-11T08:16:17.869167+0000","last_fullsized":"2024-09-11T08:16:17.869167+0000","mapping_epoch":179,"log_start":"188'5600","ondisk_log_start":"188'5600","created":159,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'9330","last_scrub_stamp":"2024-09-11T08:15:56.003501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:15:56.003501+0000","objects_scrubbed":7010,"log_size":5292,"log_dups_size":0,"ondisk_log_size":5292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:14.010825+0000","scrub_duration":1.6004849619999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":355074048,"num_objects":5418,"num_object_clones":0,"num_object_copies":16254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10892,"num_write_kb":521920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2130,"num_bytes_recovered":139591680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"190'39","reported_seq":78,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:13.657546+0000","last_change":"2024-09-11T08:15:52.848067+0000","last_active":"2024-09-11T08:16:13.657546+0000","last_peered":"2024-09-11T08:16:13.657546+0000","last_clean":"2024-09-11T08:16:13.657546+0000","last_became_active":"2024-09-11T08:15:08.775181+0000","last_became_peered":"2024-09-11T08:15:08.775181+0000","last_unstale":"2024-09-11T08:16:13.657546+0000","last_undegraded":"2024-09-11T08:16:13.657546+0000","last_fullsized":"2024-09-11T08:16:13.657546+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'39","last_scrub_stamp":"2024-09-11T08:15:52.847968+0000","last_deep_scrub":"190'39","last_deep_scrub_stamp":"2024-09-11T08:15:52.847968+0000","last_clean_scrub_stamp":"2024-09-11T08:15:52.847968+0000","objects_scrubbed":10,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:11.824582+0000","scrub_duration":0.012280474,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"59'1","reported_seq":213,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-11T08:16:20.075961+0000","last_change":"2024-09-11T08:15:44.992430+0000","last_active":"2024-09-11T08:16:20.075961+0000","last_peered":"2024-09-11T08:16:20.075961+0000","last_clean":"2024-09-11T08:16:20.075961+0000","last_became_active":"2024-09-11T08:07:30.047513+0000","last_became_peered":"2024-09-11T08:07:30.047513+0000","last_unstale":"2024-09-11T08:16:20.075961+0000","last_undegraded":"2024-09-11T08:16:20.075961+0000","last_fullsized":"2024-09-11T08:16:20.075961+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-11T08:15:44.992356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:44.992356+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:00.977258+0000","scrub_duration":0.0049464690000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":208,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.527600+0000","last_change":"2024-09-11T08:14:57.417159+0000","last_active":"2024-09-11T08:16:05.527600+0000","last_peered":"2024-09-11T08:16:05.527600+0000","last_clean":"2024-09-11T08:16:05.527600+0000","last_became_active":"2024-09-11T08:07:30.043130+0000","last_became_peered":"2024-09-11T08:07:30.043130+0000","last_unstale":"2024-09-11T08:16:05.527600+0000","last_undegraded":"2024-09-11T08:16:05.527600+0000","last_fullsized":"2024-09-11T08:16:05.527600+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-11T08:14:57.417078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:14:57.417078+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026993970000000001,"objects_trimmed":0,"snaptrim_duration":0.0026120850000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":188,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.525961+0000","last_change":"2024-09-11T08:13:01.147269+0000","last_active":"2024-09-11T08:16:05.525961+0000","last_peered":"2024-09-11T08:16:05.525961+0000","last_clean":"2024-09-11T08:16:05.525961+0000","last_became_active":"2024-09-11T08:07:34.057154+0000","last_became_peered":"2024-09-11T08:07:34.057154+0000","last_unstale":"2024-09-11T08:16:05.525961+0000","last_undegraded":"2024-09-11T08:16:05.525961+0000","last_fullsized":"2024-09-11T08:16:05.525961+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:01.147031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:10:37.957096+0000","last_clean_scrub_stamp":"2024-09-11T08:13:01.147031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.027172575000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":213,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.516829+0000","last_change":"2024-09-11T08:15:56.527473+0000","last_active":"2024-09-11T08:16:05.516829+0000","last_peered":"2024-09-11T08:16:05.516829+0000","last_clean":"2024-09-11T08:16:05.516829+0000","last_became_active":"2024-09-11T08:15:56.525099+0000","last_became_peered":"2024-09-11T08:15:56.525099+0000","last_unstale":"2024-09-11T08:16:05.516829+0000","last_undegraded":"2024-09-11T08:16:05.516829+0000","last_fullsized":"2024-09-11T08:16:05.516829+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:05.076884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:14:05.076884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001986065,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"212'10553","reported_seq":10875,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-11T08:16:18.040469+0000","last_change":"2024-09-11T08:16:05.402156+0000","last_active":"2024-09-11T08:16:18.040469+0000","last_peered":"2024-09-11T08:16:18.040469+0000","last_clean":"2024-09-11T08:16:18.040469+0000","last_became_active":"2024-09-11T08:13:54.882290+0000","last_became_peered":"2024-09-11T08:13:54.882290+0000","last_unstale":"2024-09-11T08:16:18.040469+0000","last_undegraded":"2024-09-11T08:16:18.040469+0000","last_fullsized":"2024-09-11T08:16:18.040469+0000","mapping_epoch":159,"log_start":"191'5705","ondisk_log_start":"191'5705","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'9635","last_scrub_stamp":"2024-09-11T08:16:05.401974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:16:05.401974+0000","objects_scrubbed":6255,"log_size":4848,"log_dups_size":0,"ondisk_log_size":4848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:26.177859+0000","scrub_duration":2.9696664940000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341901312,"num_objects":5217,"num_object_clones":0,"num_object_copies":15651,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10553,"num_write_kb":504640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":230,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.521939+0000","last_change":"2024-09-11T08:16:00.399312+0000","last_active":"2024-09-11T08:16:05.521939+0000","last_peered":"2024-09-11T08:16:05.521939+0000","last_clean":"2024-09-11T08:16:05.521939+0000","last_became_active":"2024-09-11T08:07:32.050673+0000","last_became_peered":"2024-09-11T08:07:32.050673+0000","last_unstale":"2024-09-11T08:16:05.521939+0000","last_undegraded":"2024-09-11T08:16:05.521939+0000","last_fullsized":"2024-09-11T08:16:05.521939+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'13","last_scrub_stamp":"2024-09-11T08:13:52.948301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:13:52.948301+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021084820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":205,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.520656+0000","last_change":"2024-09-11T08:15:31.273498+0000","last_active":"2024-09-11T08:16:05.520656+0000","last_peered":"2024-09-11T08:16:05.520656+0000","last_clean":"2024-09-11T08:16:05.520656+0000","last_became_active":"2024-09-11T08:07:30.047014+0000","last_became_peered":"2024-09-11T08:07:30.047014+0000","last_unstale":"2024-09-11T08:16:05.520656+0000","last_undegraded":"2024-09-11T08:16:05.520656+0000","last_fullsized":"2024-09-11T08:16:05.520656+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:31.273415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:31.273415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:59.181535+0000","scrub_duration":0.0057528060000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"127'2","reported_seq":221,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.533679+0000","last_change":"2024-09-11T08:15:01.938952+0000","last_active":"2024-09-11T08:16:05.533679+0000","last_peered":"2024-09-11T08:16:05.533679+0000","last_clean":"2024-09-11T08:16:05.533679+0000","last_became_active":"2024-09-11T08:07:30.045009+0000","last_became_peered":"2024-09-11T08:07:30.045009+0000","last_unstale":"2024-09-11T08:16:05.533679+0000","last_undegraded":"2024-09-11T08:16:05.533679+0000","last_fullsized":"2024-09-11T08:16:05.533679+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'2","last_scrub_stamp":"2024-09-11T08:15:01.938497+0000","last_deep_scrub":"127'2","last_deep_scrub_stamp":"2024-09-11T08:15:01.938497+0000","last_clean_scrub_stamp":"2024-09-11T08:15:01.938497+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.034714793000000001,"objects_trimmed":0,"snaptrim_duration":0.001859533,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":217,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.529532+0000","last_change":"2024-09-11T08:14:58.889181+0000","last_active":"2024-09-11T08:16:05.529532+0000","last_peered":"2024-09-11T08:16:05.529532+0000","last_clean":"2024-09-11T08:16:05.529532+0000","last_became_active":"2024-09-11T08:14:24.792279+0000","last_became_peered":"2024-09-11T08:14:24.792279+0000","last_unstale":"2024-09-11T08:16:05.529532+0000","last_undegraded":"2024-09-11T08:16:05.529532+0000","last_fullsized":"2024-09-11T08:16:05.529532+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:58.889094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:58.889094+0000","last_clean_scrub_stamp":"2024-09-11T08:14:58.889094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:16.725132+0000","scrub_duration":0.0040282030000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":134,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-11T08:16:20.087722+0000","last_change":"2024-09-11T08:14:58.796274+0000","last_active":"2024-09-11T08:16:20.087722+0000","last_peered":"2024-09-11T08:16:20.087722+0000","last_clean":"2024-09-11T08:16:20.087722+0000","last_became_active":"2024-09-11T08:14:24.792847+0000","last_became_peered":"2024-09-11T08:14:24.792847+0000","last_unstale":"2024-09-11T08:16:20.087722+0000","last_undegraded":"2024-09-11T08:16:20.087722+0000","last_fullsized":"2024-09-11T08:16:20.087722+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:58.796217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:58.796217+0000","last_clean_scrub_stamp":"2024-09-11T08:14:58.796217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:26.213711+0000","scrub_duration":0.0035819979999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"212'10650","reported_seq":10711,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-11T08:16:18.163897+0000","last_change":"2024-09-11T08:13:55.707851+0000","last_active":"2024-09-11T08:16:18.163897+0000","last_peered":"2024-09-11T08:16:18.163897+0000","last_clean":"2024-09-11T08:16:18.163897+0000","last_became_active":"2024-09-11T08:13:55.707556+0000","last_became_peered":"2024-09-11T08:13:55.707556+0000","last_unstale":"2024-09-11T08:16:18.163897+0000","last_undegraded":"2024-09-11T08:16:18.163897+0000","last_fullsized":"2024-09-11T08:16:18.163897+0000","mapping_epoch":159,"log_start":"190'5600","ondisk_log_start":"190'5600","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:13:53.862306+0000","objects_scrubbed":0,"log_size":5050,"log_dups_size":0,"ondisk_log_size":5050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":351272960,"num_objects":5360,"num_object_clones":0,"num_object_copies":16080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10650,"num_write_kb":512320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":218,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.538692+0000","last_change":"2024-09-11T08:15:53.800579+0000","last_active":"2024-09-11T08:16:05.538692+0000","last_peered":"2024-09-11T08:16:05.538692+0000","last_clean":"2024-09-11T08:16:05.538692+0000","last_became_active":"2024-09-11T08:07:32.055407+0000","last_became_peered":"2024-09-11T08:07:32.055407+0000","last_unstale":"2024-09-11T08:16:05.538692+0000","last_undegraded":"2024-09-11T08:16:05.538692+0000","last_fullsized":"2024-09-11T08:16:05.538692+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-11T08:15:53.800494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:15:53.800494+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:54.617297+0000","scrub_duration":0.0070921359999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"59'1","reported_seq":209,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.527348+0000","last_change":"2024-09-11T08:15:51.807218+0000","last_active":"2024-09-11T08:16:05.527348+0000","last_peered":"2024-09-11T08:16:05.527348+0000","last_clean":"2024-09-11T08:16:05.527348+0000","last_became_active":"2024-09-11T08:07:30.050504+0000","last_became_peered":"2024-09-11T08:07:30.050504+0000","last_unstale":"2024-09-11T08:16:05.527348+0000","last_undegraded":"2024-09-11T08:16:05.527348+0000","last_fullsized":"2024-09-11T08:16:05.527348+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-11T08:15:51.806819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:51.806819+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:59.379579+0000","scrub_duration":0.012826908,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":513,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-11T08:16:21.541249+0000","last_change":"2024-09-11T08:15:04.793314+0000","last_active":"2024-09-11T08:16:21.541249+0000","last_peered":"2024-09-11T08:16:21.541249+0000","last_clean":"2024-09-11T08:16:21.541249+0000","last_became_active":"2024-09-11T08:14:24.793048+0000","last_became_peered":"2024-09-11T08:14:24.793048+0000","last_unstale":"2024-09-11T08:16:21.541249+0000","last_undegraded":"2024-09-11T08:16:21.541249+0000","last_fullsized":"2024-09-11T08:16:21.541249+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-11T08:15:04.793062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:04.793062+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:22.337286+0000","scrub_duration":0.0082004639999999993,"objects_trimmed":0,"snaptrim_duration":0.000318821,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":124,"num_read_kb":119,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":197,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.526717+0000","last_change":"2024-09-11T08:14:06.390698+0000","last_active":"2024-09-11T08:16:05.526717+0000","last_peered":"2024-09-11T08:16:05.526717+0000","last_clean":"2024-09-11T08:16:05.526717+0000","last_became_active":"2024-09-11T08:07:34.058962+0000","last_became_peered":"2024-09-11T08:07:34.058962+0000","last_unstale":"2024-09-11T08:16:05.526717+0000","last_undegraded":"2024-09-11T08:16:05.526717+0000","last_fullsized":"2024-09-11T08:16:05.526717+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:06.390632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:14:06.390632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017575690000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":206,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.520807+0000","last_change":"2024-09-11T08:15:53.506655+0000","last_active":"2024-09-11T08:16:05.520807+0000","last_peered":"2024-09-11T08:16:05.520807+0000","last_clean":"2024-09-11T08:16:05.520807+0000","last_became_active":"2024-09-11T08:07:36.077597+0000","last_became_peered":"2024-09-11T08:07:36.077597+0000","last_unstale":"2024-09-11T08:16:05.520807+0000","last_undegraded":"2024-09-11T08:16:05.520807+0000","last_fullsized":"2024-09-11T08:16:05.520807+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:53.506581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:15:53.506581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:15.129440+0000","scrub_duration":0.0028717209999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"214'11045","reported_seq":11425,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-11T08:16:22.722101+0000","last_change":"2024-09-11T08:15:47.174817+0000","last_active":"2024-09-11T08:16:22.722101+0000","last_peered":"2024-09-11T08:16:22.722101+0000","last_clean":"2024-09-11T08:16:22.722101+0000","last_became_active":"2024-09-11T08:13:54.882405+0000","last_became_peered":"2024-09-11T08:13:54.882405+0000","last_unstale":"2024-09-11T08:16:22.722101+0000","last_undegraded":"2024-09-11T08:16:22.722101+0000","last_fullsized":"2024-09-11T08:16:22.722101+0000","mapping_epoch":159,"log_start":"185'5000","ondisk_log_start":"185'5000","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'8519","last_scrub_stamp":"2024-09-11T08:15:47.174735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:15:47.174735+0000","objects_scrubbed":7692,"log_size":6045,"log_dups_size":0,"ondisk_log_size":6045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:12.130102+0000","scrub_duration":2.9660035420000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330104832,"num_objects":5037,"num_object_clones":0,"num_object_copies":15111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11045,"num_write_kb":514624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"84'21","reported_seq":247,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-11T08:16:20.082957+0000","last_change":"2024-09-11T08:14:51.029564+0000","last_active":"2024-09-11T08:16:20.082957+0000","last_peered":"2024-09-11T08:16:20.082957+0000","last_clean":"2024-09-11T08:16:20.082957+0000","last_became_active":"2024-09-11T08:14:24.789154+0000","last_became_peered":"2024-09-11T08:14:24.789154+0000","last_unstale":"2024-09-11T08:16:20.082957+0000","last_undegraded":"2024-09-11T08:16:20.082957+0000","last_fullsized":"2024-09-11T08:16:20.082957+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'21","last_scrub_stamp":"2024-09-11T08:14:51.029469+0000","last_deep_scrub":"84'21","last_deep_scrub_stamp":"2024-09-11T08:08:56.705306+0000","last_clean_scrub_stamp":"2024-09-11T08:14:51.029469+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001733975,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":192,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.515804+0000","last_change":"2024-09-11T08:15:06.236392+0000","last_active":"2024-09-11T08:16:05.515804+0000","last_peered":"2024-09-11T08:16:05.515804+0000","last_clean":"2024-09-11T08:16:05.515804+0000","last_became_active":"2024-09-11T08:07:30.047246+0000","last_became_peered":"2024-09-11T08:07:30.047246+0000","last_unstale":"2024-09-11T08:16:05.515804+0000","last_undegraded":"2024-09-11T08:16:05.515804+0000","last_fullsized":"2024-09-11T08:16:05.515804+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:06.236119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:06.236119+0000","last_clean_scrub_stamp":"2024-09-11T08:15:06.236119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029121939999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":22,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-11T08:16:20.086984+0000","last_change":"2024-09-11T08:15:56.536262+0000","last_active":"2024-09-11T08:16:20.086984+0000","last_peered":"2024-09-11T08:16:20.086984+0000","last_clean":"2024-09-11T08:16:20.086984+0000","last_became_active":"2024-09-11T08:15:56.523430+0000","last_became_peered":"2024-09-11T08:15:56.523430+0000","last_unstale":"2024-09-11T08:16:20.086984+0000","last_undegraded":"2024-09-11T08:16:20.086984+0000","last_fullsized":"2024-09-11T08:16:20.086984+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-11T08:14:10.287968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:05:21.231856+0000","last_clean_scrub_stamp":"2024-09-11T08:14:10.287968+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023909769999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":16,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-11T08:16:20.086769+0000","last_change":"2024-09-11T08:15:56.529159+0000","last_active":"2024-09-11T08:16:20.086769+0000","last_peered":"2024-09-11T08:16:20.086769+0000","last_clean":"2024-09-11T08:16:20.086769+0000","last_became_active":"2024-09-11T08:15:56.528641+0000","last_became_peered":"2024-09-11T08:15:56.528641+0000","last_unstale":"2024-09-11T08:16:20.086769+0000","last_undegraded":"2024-09-11T08:16:20.086769+0000","last_fullsized":"2024-09-11T08:16:20.086769+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:55.140137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:51.289872+0000","last_clean_scrub_stamp":"2024-09-11T08:15:55.140137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:57.600999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":195,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.526563+0000","last_change":"2024-09-11T08:14:51.371546+0000","last_active":"2024-09-11T08:16:05.526563+0000","last_peered":"2024-09-11T08:16:05.526563+0000","last_clean":"2024-09-11T08:16:05.526563+0000","last_became_active":"2024-09-11T08:07:36.072508+0000","last_became_peered":"2024-09-11T08:07:36.072508+0000","last_unstale":"2024-09-11T08:16:05.526563+0000","last_undegraded":"2024-09-11T08:16:05.526563+0000","last_fullsized":"2024-09-11T08:16:05.526563+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:51.371488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:01.777429+0000","last_clean_scrub_stamp":"2024-09-11T08:14:51.371488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001247979,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"214'11035","reported_seq":11098,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-11T08:16:22.761399+0000","last_change":"2024-09-11T08:13:54.881747+0000","last_active":"2024-09-11T08:16:22.761399+0000","last_peered":"2024-09-11T08:16:22.761399+0000","last_clean":"2024-09-11T08:16:22.761399+0000","last_became_active":"2024-09-11T08:13:54.881423+0000","last_became_peered":"2024-09-11T08:13:54.881423+0000","last_unstale":"2024-09-11T08:16:22.761399+0000","last_undegraded":"2024-09-11T08:16:22.761399+0000","last_fullsized":"2024-09-11T08:16:22.761399+0000","mapping_epoch":159,"log_start":"193'6200","ondisk_log_start":"193'6200","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:13:53.862306+0000","objects_scrubbed":0,"log_size":4835,"log_dups_size":0,"ondisk_log_size":4835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330498048,"num_objects":5043,"num_object_clones":0,"num_object_copies":15129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11035,"num_write_kb":514496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"88'16","reported_seq":216,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-11T08:16:20.074918+0000","last_change":"2024-09-11T08:12:55.025363+0000","last_active":"2024-09-11T08:16:20.074918+0000","last_peered":"2024-09-11T08:16:20.074918+0000","last_clean":"2024-09-11T08:16:20.074918+0000","last_became_active":"2024-09-11T08:07:32.047971+0000","last_became_peered":"2024-09-11T08:07:32.047971+0000","last_unstale":"2024-09-11T08:16:20.074918+0000","last_undegraded":"2024-09-11T08:16:20.074918+0000","last_fullsized":"2024-09-11T08:16:20.074918+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'16","last_scrub_stamp":"2024-09-11T08:12:55.025272+0000","last_deep_scrub":"88'16","last_deep_scrub_stamp":"2024-09-11T08:09:39.635984+0000","last_clean_scrub_stamp":"2024-09-11T08:12:55.025272+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023373420000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":211,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-11T08:16:20.075139+0000","last_change":"2024-09-11T08:14:54.023658+0000","last_active":"2024-09-11T08:16:20.075139+0000","last_peered":"2024-09-11T08:16:20.075139+0000","last_clean":"2024-09-11T08:16:20.075139+0000","last_became_active":"2024-09-11T08:14:24.789888+0000","last_became_peered":"2024-09-11T08:14:24.789888+0000","last_unstale":"2024-09-11T08:16:20.075139+0000","last_undegraded":"2024-09-11T08:16:20.075139+0000","last_fullsized":"2024-09-11T08:16:20.075139+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:54.023556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:36.417911+0000","last_clean_scrub_stamp":"2024-09-11T08:14:54.023556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002726596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":197,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.526420+0000","last_change":"2024-09-11T08:15:06.434641+0000","last_active":"2024-09-11T08:16:05.526420+0000","last_peered":"2024-09-11T08:16:05.526420+0000","last_clean":"2024-09-11T08:16:05.526420+0000","last_became_active":"2024-09-11T08:07:34.053347+0000","last_became_peered":"2024-09-11T08:07:34.053347+0000","last_unstale":"2024-09-11T08:16:05.526420+0000","last_undegraded":"2024-09-11T08:16:05.526420+0000","last_fullsized":"2024-09-11T08:16:05.526420+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:06.434588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:01.708166+0000","last_clean_scrub_stamp":"2024-09-11T08:15:06.434588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:31.639996+0000","scrub_duration":0.0039794419999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":186,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.515894+0000","last_change":"2024-09-11T08:14:57.236619+0000","last_active":"2024-09-11T08:16:05.515894+0000","last_peered":"2024-09-11T08:16:05.515894+0000","last_clean":"2024-09-11T08:16:05.515894+0000","last_became_active":"2024-09-11T08:07:36.631687+0000","last_became_peered":"2024-09-11T08:07:36.631687+0000","last_unstale":"2024-09-11T08:16:05.515894+0000","last_undegraded":"2024-09-11T08:16:05.515894+0000","last_fullsized":"2024-09-11T08:16:05.515894+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:57.236432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:14:57.236432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029326859999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1","version":"212'10693","reported_seq":11037,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-11T08:16:18.179694+0000","last_change":"2024-09-11T08:15:57.075945+0000","last_active":"2024-09-11T08:16:18.179694+0000","last_peered":"2024-09-11T08:16:18.179694+0000","last_clean":"2024-09-11T08:16:18.179694+0000","last_became_active":"2024-09-11T08:13:55.707503+0000","last_became_peered":"2024-09-11T08:13:55.707503+0000","last_unstale":"2024-09-11T08:16:18.179694+0000","last_undegraded":"2024-09-11T08:16:18.179694+0000","last_fullsized":"2024-09-11T08:16:18.179694+0000","mapping_epoch":159,"log_start":"190'5600","ondisk_log_start":"190'5600","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'9162","last_scrub_stamp":"2024-09-11T08:15:57.075870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:15:57.075870+0000","objects_scrubbed":6840,"log_size":5093,"log_dups_size":0,"ondisk_log_size":5093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:10.709887+0000","scrub_duration":0.63951687599999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":347930624,"num_objects":5309,"num_object_clones":0,"num_object_copies":15927,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10693,"num_write_kb":512064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"77'25","reported_seq":266,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-11T08:16:20.075359+0000","last_change":"2024-09-11T08:15:42.061711+0000","last_active":"2024-09-11T08:16:20.075359+0000","last_peered":"2024-09-11T08:16:20.075359+0000","last_clean":"2024-09-11T08:16:20.075359+0000","last_became_active":"2024-09-11T08:09:08.106771+0000","last_became_peered":"2024-09-11T08:09:08.106771+0000","last_unstale":"2024-09-11T08:16:20.075359+0000","last_undegraded":"2024-09-11T08:16:20.075359+0000","last_fullsized":"2024-09-11T08:16:20.075359+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'25","last_scrub_stamp":"2024-09-11T08:15:42.061632+0000","last_deep_scrub":"77'25","last_deep_scrub_stamp":"2024-09-11T08:08:32.458210+0000","last_clean_scrub_stamp":"2024-09-11T08:15:42.061632+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:06.943716+0000","scrub_duration":0.00500454,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":203,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-11T08:16:20.080363+0000","last_change":"2024-09-11T08:15:00.050885+0000","last_active":"2024-09-11T08:16:20.080363+0000","last_peered":"2024-09-11T08:16:20.080363+0000","last_clean":"2024-09-11T08:16:20.080363+0000","last_became_active":"2024-09-11T08:07:30.042555+0000","last_became_peered":"2024-09-11T08:07:30.042555+0000","last_unstale":"2024-09-11T08:16:20.080363+0000","last_undegraded":"2024-09-11T08:16:20.080363+0000","last_fullsized":"2024-09-11T08:16:20.080363+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:00.050803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:00.050803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043077549999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":198,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.520881+0000","last_change":"2024-09-11T08:14:00.985127+0000","last_active":"2024-09-11T08:16:05.520881+0000","last_peered":"2024-09-11T08:16:05.520881+0000","last_clean":"2024-09-11T08:16:05.520881+0000","last_became_active":"2024-09-11T08:07:34.054961+0000","last_became_peered":"2024-09-11T08:07:34.054961+0000","last_unstale":"2024-09-11T08:16:05.520881+0000","last_undegraded":"2024-09-11T08:16:05.520881+0000","last_fullsized":"2024-09-11T08:16:05.520881+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:00.985075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:14:00.985075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015093109999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":215,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.529478+0000","last_change":"2024-09-11T08:15:43.160119+0000","last_active":"2024-09-11T08:16:05.529478+0000","last_peered":"2024-09-11T08:16:05.529478+0000","last_clean":"2024-09-11T08:16:05.529478+0000","last_became_active":"2024-09-11T08:07:36.071194+0000","last_became_peered":"2024-09-11T08:07:36.071194+0000","last_unstale":"2024-09-11T08:16:05.529478+0000","last_undegraded":"2024-09-11T08:16:05.529478+0000","last_fullsized":"2024-09-11T08:16:05.529478+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:43.158812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:07.930137+0000","last_clean_scrub_stamp":"2024-09-11T08:15:43.158812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:00.399779+0000","scrub_duration":0.0051752609999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"212'10734","reported_seq":10795,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-11T08:16:18.063195+0000","last_change":"2024-09-11T08:13:54.882472+0000","last_active":"2024-09-11T08:16:18.063195+0000","last_peered":"2024-09-11T08:16:18.063195+0000","last_clean":"2024-09-11T08:16:18.063195+0000","last_became_active":"2024-09-11T08:13:54.882196+0000","last_became_peered":"2024-09-11T08:13:54.882196+0000","last_unstale":"2024-09-11T08:16:18.063195+0000","last_undegraded":"2024-09-11T08:16:18.063195+0000","last_fullsized":"2024-09-11T08:16:18.063195+0000","mapping_epoch":159,"log_start":"191'5935","ondisk_log_start":"191'5935","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:13:53.862306+0000","objects_scrubbed":0,"log_size":4799,"log_dups_size":0,"ondisk_log_size":4799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352321536,"num_objects":5376,"num_object_clones":0,"num_object_copies":16128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10734,"num_write_kb":515520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":207,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-11T08:16:20.076210+0000","last_change":"2024-09-11T08:13:07.139492+0000","last_active":"2024-09-11T08:16:20.076210+0000","last_peered":"2024-09-11T08:16:20.076210+0000","last_clean":"2024-09-11T08:16:20.076210+0000","last_became_active":"2024-09-11T08:07:32.043314+0000","last_became_peered":"2024-09-11T08:07:32.043314+0000","last_unstale":"2024-09-11T08:16:20.076210+0000","last_undegraded":"2024-09-11T08:16:20.076210+0000","last_fullsized":"2024-09-11T08:16:20.076210+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-11T08:13:07.139430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:13:07.139430+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0075406340000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":202,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.516339+0000","last_change":"2024-09-11T08:15:48.405028+0000","last_active":"2024-09-11T08:16:05.516339+0000","last_peered":"2024-09-11T08:16:05.516339+0000","last_clean":"2024-09-11T08:16:05.516339+0000","last_became_active":"2024-09-11T08:07:30.046684+0000","last_became_peered":"2024-09-11T08:07:30.046684+0000","last_unstale":"2024-09-11T08:16:05.516339+0000","last_undegraded":"2024-09-11T08:16:05.516339+0000","last_fullsized":"2024-09-11T08:16:05.516339+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:48.404853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:48.404853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:01.459915+0000","scrub_duration":0.0048699770000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"171'12","reported_seq":334,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:13.654515+0000","last_change":"2024-09-11T08:14:56.881397+0000","last_active":"2024-09-11T08:16:13.654515+0000","last_peered":"2024-09-11T08:16:13.654515+0000","last_clean":"2024-09-11T08:16:13.654515+0000","last_became_active":"2024-09-11T08:14:24.794446+0000","last_became_peered":"2024-09-11T08:14:24.794446+0000","last_unstale":"2024-09-11T08:16:13.654515+0000","last_undegraded":"2024-09-11T08:16:13.654515+0000","last_fullsized":"2024-09-11T08:16:13.654515+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'12","last_scrub_stamp":"2024-09-11T08:14:56.881299+0000","last_deep_scrub":"171'12","last_deep_scrub_stamp":"2024-09-11T08:14:56.881299+0000","last_clean_scrub_stamp":"2024-09-11T08:14:56.881299+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0073519259999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":188,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-11T08:16:20.074703+0000","last_change":"2024-09-11T08:14:13.941569+0000","last_active":"2024-09-11T08:16:20.074703+0000","last_peered":"2024-09-11T08:16:20.074703+0000","last_clean":"2024-09-11T08:16:20.074703+0000","last_became_active":"2024-09-11T08:07:36.074684+0000","last_became_peered":"2024-09-11T08:07:36.074684+0000","last_unstale":"2024-09-11T08:16:20.074703+0000","last_undegraded":"2024-09-11T08:16:20.074703+0000","last_fullsized":"2024-09-11T08:16:20.074703+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:13.941492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:14.613532+0000","last_clean_scrub_stamp":"2024-09-11T08:14:13.941492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037661470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.7","version":"212'10683","reported_seq":11142,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-11T08:16:18.159285+0000","last_change":"2024-09-11T08:15:56.302099+0000","last_active":"2024-09-11T08:16:18.159285+0000","last_peered":"2024-09-11T08:16:18.159285+0000","last_clean":"2024-09-11T08:16:18.159285+0000","last_became_active":"2024-09-11T08:13:54.883624+0000","last_became_peered":"2024-09-11T08:13:54.883624+0000","last_unstale":"2024-09-11T08:16:18.159285+0000","last_undegraded":"2024-09-11T08:16:18.159285+0000","last_fullsized":"2024-09-11T08:16:18.159285+0000","mapping_epoch":159,"log_start":"190'5600","ondisk_log_start":"190'5600","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'9119","last_scrub_stamp":"2024-09-11T08:15:56.302032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:15:56.302032+0000","objects_scrubbed":6919,"log_size":5083,"log_dups_size":0,"ondisk_log_size":5083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:02.822566+0000","scrub_duration":0.82679451100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350945280,"num_objects":5355,"num_object_clones":0,"num_object_copies":16065,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10683,"num_write_kb":513216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":206,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.525459+0000","last_change":"2024-09-11T08:13:14.171710+0000","last_active":"2024-09-11T08:16:05.525459+0000","last_peered":"2024-09-11T08:16:05.525459+0000","last_clean":"2024-09-11T08:16:05.525459+0000","last_became_active":"2024-09-11T08:07:32.054885+0000","last_became_peered":"2024-09-11T08:07:32.054885+0000","last_unstale":"2024-09-11T08:16:05.525459+0000","last_undegraded":"2024-09-11T08:16:05.525459+0000","last_fullsized":"2024-09-11T08:16:05.525459+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:13:14.171623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:13:14.171623+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.01042096,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":211,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.527700+0000","last_change":"2024-09-11T08:15:34.497175+0000","last_active":"2024-09-11T08:16:05.527700+0000","last_peered":"2024-09-11T08:16:05.527700+0000","last_clean":"2024-09-11T08:16:05.527700+0000","last_became_active":"2024-09-11T08:07:30.045613+0000","last_became_peered":"2024-09-11T08:07:30.045613+0000","last_unstale":"2024-09-11T08:16:05.527700+0000","last_undegraded":"2024-09-11T08:16:05.527700+0000","last_fullsized":"2024-09-11T08:16:05.527700+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:34.497125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:34.497125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:45.232704+0000","scrub_duration":0.0052802250000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":217,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.528377+0000","last_change":"2024-09-11T08:15:37.123094+0000","last_active":"2024-09-11T08:16:05.528377+0000","last_peered":"2024-09-11T08:16:05.528377+0000","last_clean":"2024-09-11T08:16:05.528377+0000","last_became_active":"2024-09-11T08:07:34.056606+0000","last_became_peered":"2024-09-11T08:07:34.056606+0000","last_unstale":"2024-09-11T08:16:05.528377+0000","last_undegraded":"2024-09-11T08:16:05.528377+0000","last_fullsized":"2024-09-11T08:16:05.528377+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:37.123010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:15:37.123010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:05.989949+0000","scrub_duration":0.0042949169999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":213,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.527272+0000","last_change":"2024-09-11T08:15:56.520006+0000","last_active":"2024-09-11T08:16:05.527272+0000","last_peered":"2024-09-11T08:16:05.527272+0000","last_clean":"2024-09-11T08:16:05.527272+0000","last_became_active":"2024-09-11T08:15:56.518884+0000","last_became_peered":"2024-09-11T08:15:56.518884+0000","last_unstale":"2024-09-11T08:16:05.527272+0000","last_undegraded":"2024-09-11T08:16:05.527272+0000","last_fullsized":"2024-09-11T08:16:05.527272+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:01.376411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:14:01.376411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012067320000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"212'10860","reported_seq":15062,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-11T08:16:18.234814+0000","last_change":"2024-09-11T08:14:51.900945+0000","last_active":"2024-09-11T08:16:18.234814+0000","last_peered":"2024-09-11T08:16:18.234814+0000","last_clean":"2024-09-11T08:16:18.234814+0000","last_became_active":"2024-09-11T08:14:51.879122+0000","last_became_peered":"2024-09-11T08:14:51.879122+0000","last_unstale":"2024-09-11T08:16:18.234814+0000","last_undegraded":"2024-09-11T08:16:18.234814+0000","last_fullsized":"2024-09-11T08:16:18.234814+0000","mapping_epoch":180,"log_start":"183'4900","ondisk_log_start":"183'4900","created":159,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:13:53.862306+0000","objects_scrubbed":0,"log_size":5960,"log_dups_size":0,"ondisk_log_size":5960,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360185856,"num_objects":5496,"num_object_clones":0,"num_object_copies":16488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10860,"num_write_kb":523392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2061,"num_bytes_recovered":135069696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":309,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-11T08:16:20.080439+0000","last_change":"2024-09-11T08:15:56.551439+0000","last_active":"2024-09-11T08:16:20.080439+0000","last_peered":"2024-09-11T08:16:20.080439+0000","last_clean":"2024-09-11T08:16:20.080439+0000","last_became_active":"2024-09-11T08:15:56.526961+0000","last_became_peered":"2024-09-11T08:15:56.526961+0000","last_unstale":"2024-09-11T08:16:20.080439+0000","last_undegraded":"2024-09-11T08:16:20.080439+0000","last_fullsized":"2024-09-11T08:16:20.080439+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-11T08:15:49.187186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:15:49.187186+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:17.036763+0000","scrub_duration":0.0054603430000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":16,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-11T08:16:20.087501+0000","last_change":"2024-09-11T08:15:56.530685+0000","last_active":"2024-09-11T08:16:20.087501+0000","last_peered":"2024-09-11T08:16:20.087501+0000","last_clean":"2024-09-11T08:16:20.087501+0000","last_became_active":"2024-09-11T08:15:56.530370+0000","last_became_peered":"2024-09-11T08:15:56.530370+0000","last_unstale":"2024-09-11T08:16:20.087501+0000","last_undegraded":"2024-09-11T08:16:20.087501+0000","last_fullsized":"2024-09-11T08:16:20.087501+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:09.433494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:44.385372+0000","last_clean_scrub_stamp":"2024-09-11T08:14:09.433494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":210,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.528191+0000","last_change":"2024-09-11T08:14:59.900979+0000","last_active":"2024-09-11T08:16:05.528191+0000","last_peered":"2024-09-11T08:16:05.528191+0000","last_clean":"2024-09-11T08:16:05.528191+0000","last_became_active":"2024-09-11T08:07:34.063589+0000","last_became_peered":"2024-09-11T08:07:34.063589+0000","last_unstale":"2024-09-11T08:16:05.528191+0000","last_undegraded":"2024-09-11T08:16:05.528191+0000","last_fullsized":"2024-09-11T08:16:05.528191+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:59.900898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:14:59.900898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:26.844256+0000","scrub_duration":0.006061121,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":16,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-11T08:16:20.087059+0000","last_change":"2024-09-11T08:15:56.518476+0000","last_active":"2024-09-11T08:16:20.087059+0000","last_peered":"2024-09-11T08:16:20.087059+0000","last_clean":"2024-09-11T08:16:20.087059+0000","last_became_active":"2024-09-11T08:15:56.518169+0000","last_became_peered":"2024-09-11T08:15:56.518169+0000","last_unstale":"2024-09-11T08:16:20.087059+0000","last_undegraded":"2024-09-11T08:16:20.087059+0000","last_fullsized":"2024-09-11T08:16:20.087059+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:32.131008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:15:32.131008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:59.362244+0000","scrub_duration":0.0064196519999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"214'11246","reported_seq":16018,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-11T08:16:22.829126+0000","last_change":"2024-09-11T08:15:38.333949+0000","last_active":"2024-09-11T08:16:22.829126+0000","last_peered":"2024-09-11T08:16:22.829126+0000","last_clean":"2024-09-11T08:16:22.829126+0000","last_became_active":"2024-09-11T08:14:50.875228+0000","last_became_peered":"2024-09-11T08:14:50.875228+0000","last_unstale":"2024-09-11T08:16:22.829126+0000","last_undegraded":"2024-09-11T08:16:22.829126+0000","last_fullsized":"2024-09-11T08:16:22.829126+0000","mapping_epoch":179,"log_start":"185'5176","ondisk_log_start":"185'5176","created":159,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"200'7896","last_scrub_stamp":"2024-09-11T08:15:38.333891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:15:38.333891+0000","objects_scrubbed":7806,"log_size":6070,"log_dups_size":0,"ondisk_log_size":6070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:43.473904+0000","scrub_duration":2.12463202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331743232,"num_objects":5062,"num_object_clones":0,"num_object_copies":15186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11246,"num_write_kb":521856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2184,"num_bytes_recovered":143130624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"145'17","reported_seq":230,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-11T08:16:20.074888+0000","last_change":"2024-09-11T08:15:53.040339+0000","last_active":"2024-09-11T08:16:20.074888+0000","last_peered":"2024-09-11T08:16:20.074888+0000","last_clean":"2024-09-11T08:16:20.074888+0000","last_became_active":"2024-09-11T08:07:32.048164+0000","last_became_peered":"2024-09-11T08:07:32.048164+0000","last_unstale":"2024-09-11T08:16:20.074888+0000","last_undegraded":"2024-09-11T08:16:20.074888+0000","last_fullsized":"2024-09-11T08:16:20.074888+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'17","last_scrub_stamp":"2024-09-11T08:15:53.040223+0000","last_deep_scrub":"145'17","last_deep_scrub_stamp":"2024-09-11T08:13:08.160848+0000","last_clean_scrub_stamp":"2024-09-11T08:15:53.040223+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:12.632932+0000","scrub_duration":0.0039160820000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":16,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-11T08:16:20.087181+0000","last_change":"2024-09-11T08:15:56.526162+0000","last_active":"2024-09-11T08:16:20.087181+0000","last_peered":"2024-09-11T08:16:20.087181+0000","last_clean":"2024-09-11T08:16:20.087181+0000","last_became_active":"2024-09-11T08:15:56.525845+0000","last_became_peered":"2024-09-11T08:15:56.525845+0000","last_unstale":"2024-09-11T08:16:20.087181+0000","last_undegraded":"2024-09-11T08:16:20.087181+0000","last_fullsized":"2024-09-11T08:16:20.087181+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:12.966188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:53.247275+0000","last_clean_scrub_stamp":"2024-09-11T08:14:12.966188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":197,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.528105+0000","last_change":"2024-09-11T08:14:01.899688+0000","last_active":"2024-09-11T08:16:05.528105+0000","last_peered":"2024-09-11T08:16:05.528105+0000","last_clean":"2024-09-11T08:16:05.528105+0000","last_became_active":"2024-09-11T08:07:34.057479+0000","last_became_peered":"2024-09-11T08:07:34.057479+0000","last_unstale":"2024-09-11T08:16:05.528105+0000","last_undegraded":"2024-09-11T08:16:05.528105+0000","last_fullsized":"2024-09-11T08:16:05.528105+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:01.899633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:14:01.899633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011221180000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":213,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.516070+0000","last_change":"2024-09-11T08:15:56.522949+0000","last_active":"2024-09-11T08:16:05.516070+0000","last_peered":"2024-09-11T08:16:05.516070+0000","last_clean":"2024-09-11T08:16:05.516070+0000","last_became_active":"2024-09-11T08:15:56.522624+0000","last_became_peered":"2024-09-11T08:15:56.522624+0000","last_unstale":"2024-09-11T08:16:05.516070+0000","last_undegraded":"2024-09-11T08:16:05.516070+0000","last_fullsized":"2024-09-11T08:16:05.516070+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:54.938695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:13:54.938695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016854260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"212'10814","reported_seq":11176,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-11T08:16:17.895399+0000","last_change":"2024-09-11T08:15:53.462987+0000","last_active":"2024-09-11T08:16:17.895399+0000","last_peered":"2024-09-11T08:16:17.895399+0000","last_clean":"2024-09-11T08:16:17.895399+0000","last_became_active":"2024-09-11T08:13:55.705819+0000","last_became_peered":"2024-09-11T08:13:55.705819+0000","last_unstale":"2024-09-11T08:16:17.895399+0000","last_undegraded":"2024-09-11T08:16:17.895399+0000","last_fullsized":"2024-09-11T08:16:17.895399+0000","mapping_epoch":159,"log_start":"188'5500","ondisk_log_start":"188'5500","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"206'9066","last_scrub_stamp":"2024-09-11T08:15:53.462907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:15:53.462907+0000","objects_scrubbed":7268,"log_size":5314,"log_dups_size":0,"ondisk_log_size":5314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:18.668302+0000","scrub_duration":2.0133290810000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":357564416,"num_objects":5456,"num_object_clones":0,"num_object_copies":16368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10814,"num_write_kb":520640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":283,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.528360+0000","last_change":"2024-09-11T08:15:56.551197+0000","last_active":"2024-09-11T08:16:05.528360+0000","last_peered":"2024-09-11T08:16:05.528360+0000","last_clean":"2024-09-11T08:16:05.528360+0000","last_became_active":"2024-09-11T08:15:56.532431+0000","last_became_peered":"2024-09-11T08:15:56.532431+0000","last_unstale":"2024-09-11T08:16:05.528360+0000","last_undegraded":"2024-09-11T08:16:05.528360+0000","last_fullsized":"2024-09-11T08:16:05.528360+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-11T08:15:47.427884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:15:47.427884+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:52.822361+0000","scrub_duration":0.0016881800000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":22,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-11T08:16:20.086855+0000","last_change":"2024-09-11T08:15:56.538509+0000","last_active":"2024-09-11T08:16:20.086855+0000","last_peered":"2024-09-11T08:16:20.086855+0000","last_clean":"2024-09-11T08:16:20.086855+0000","last_became_active":"2024-09-11T08:15:56.515328+0000","last_became_peered":"2024-09-11T08:15:56.515328+0000","last_unstale":"2024-09-11T08:16:20.086855+0000","last_undegraded":"2024-09-11T08:16:20.086855+0000","last_fullsized":"2024-09-11T08:16:20.086855+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-11T08:13:59.909695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:13:59.909695+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"209'11","reported_seq":25,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-11T08:16:20.086358+0000","last_change":"2024-09-11T08:15:56.534905+0000","last_active":"2024-09-11T08:16:20.086358+0000","last_peered":"2024-09-11T08:16:20.086358+0000","last_clean":"2024-09-11T08:16:20.086358+0000","last_became_active":"2024-09-11T08:15:56.524513+0000","last_became_peered":"2024-09-11T08:15:56.524513+0000","last_unstale":"2024-09-11T08:16:20.086358+0000","last_undegraded":"2024-09-11T08:16:20.086358+0000","last_fullsized":"2024-09-11T08:16:20.086358+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"156'10","last_scrub_stamp":"2024-09-11T08:15:38.215399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:15:38.215399+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:00.740762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":195,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.527199+0000","last_change":"2024-09-11T08:15:03.484853+0000","last_active":"2024-09-11T08:16:05.527199+0000","last_peered":"2024-09-11T08:16:05.527199+0000","last_clean":"2024-09-11T08:16:05.527199+0000","last_became_active":"2024-09-11T08:07:36.631755+0000","last_became_peered":"2024-09-11T08:07:36.631755+0000","last_unstale":"2024-09-11T08:16:05.527199+0000","last_undegraded":"2024-09-11T08:16:05.527199+0000","last_fullsized":"2024-09-11T08:16:05.527199+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:03.484755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:03.484755+0000","last_clean_scrub_stamp":"2024-09-11T08:15:03.484755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0064296900000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":213,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-11T08:16:20.080531+0000","last_change":"2024-09-11T08:15:31.257899+0000","last_active":"2024-09-11T08:16:20.080531+0000","last_peered":"2024-09-11T08:16:20.080531+0000","last_clean":"2024-09-11T08:16:20.080531+0000","last_became_active":"2024-09-11T08:07:30.045910+0000","last_became_peered":"2024-09-11T08:07:30.045910+0000","last_unstale":"2024-09-11T08:16:20.080531+0000","last_undegraded":"2024-09-11T08:16:20.080531+0000","last_fullsized":"2024-09-11T08:16:20.080531+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:31.257821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:31.257821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:56.144543+0000","scrub_duration":0.0040247779999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"84'9","reported_seq":245,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.519385+0000","last_change":"2024-09-11T08:15:56.550790+0000","last_active":"2024-09-11T08:16:05.519385+0000","last_peered":"2024-09-11T08:16:05.519385+0000","last_clean":"2024-09-11T08:16:05.519385+0000","last_became_active":"2024-09-11T08:15:56.526325+0000","last_became_peered":"2024-09-11T08:15:56.526325+0000","last_unstale":"2024-09-11T08:16:05.519385+0000","last_undegraded":"2024-09-11T08:16:05.519385+0000","last_fullsized":"2024-09-11T08:16:05.519385+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-09-11T08:14:12.071950+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-09-11T08:09:03.418457+0000","last_clean_scrub_stamp":"2024-09-11T08:14:12.071950+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0068954059999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":197,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.526287+0000","last_change":"2024-09-11T08:15:02.463928+0000","last_active":"2024-09-11T08:16:05.526287+0000","last_peered":"2024-09-11T08:16:05.526287+0000","last_clean":"2024-09-11T08:16:05.526287+0000","last_became_active":"2024-09-11T08:07:34.057774+0000","last_became_peered":"2024-09-11T08:07:34.057774+0000","last_unstale":"2024-09-11T08:16:05.526287+0000","last_undegraded":"2024-09-11T08:16:05.526287+0000","last_fullsized":"2024-09-11T08:16:05.526287+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:02.463848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:02.463848+0000","last_clean_scrub_stamp":"2024-09-11T08:15:02.463848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034811849999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":141,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-11T08:16:20.087614+0000","last_change":"2024-09-11T08:14:56.815770+0000","last_active":"2024-09-11T08:16:20.087614+0000","last_peered":"2024-09-11T08:16:20.087614+0000","last_clean":"2024-09-11T08:16:20.087614+0000","last_became_active":"2024-09-11T08:14:24.790678+0000","last_became_peered":"2024-09-11T08:14:24.790678+0000","last_unstale":"2024-09-11T08:16:20.087614+0000","last_undegraded":"2024-09-11T08:16:20.087614+0000","last_fullsized":"2024-09-11T08:16:20.087614+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:56.815690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:14:56.815690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016768309999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"84'13","reported_seq":257,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.538753+0000","last_change":"2024-09-11T08:15:56.532882+0000","last_active":"2024-09-11T08:16:05.538753+0000","last_peered":"2024-09-11T08:16:05.538753+0000","last_clean":"2024-09-11T08:16:05.538753+0000","last_became_active":"2024-09-11T08:15:56.514793+0000","last_became_peered":"2024-09-11T08:15:56.514793+0000","last_unstale":"2024-09-11T08:16:05.538753+0000","last_undegraded":"2024-09-11T08:16:05.538753+0000","last_fullsized":"2024-09-11T08:16:05.538753+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'13","last_scrub_stamp":"2024-09-11T08:14:05.339509+0000","last_deep_scrub":"84'13","last_deep_scrub_stamp":"2024-09-11T08:08:57.835634+0000","last_clean_scrub_stamp":"2024-09-11T08:14:05.339509+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001614047,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":174,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-11T08:16:20.086930+0000","last_change":"2024-09-11T08:15:31.534404+0000","last_active":"2024-09-11T08:16:20.086930+0000","last_peered":"2024-09-11T08:16:20.086930+0000","last_clean":"2024-09-11T08:16:20.086930+0000","last_became_active":"2024-09-11T08:14:24.786021+0000","last_became_peered":"2024-09-11T08:14:24.786021+0000","last_unstale":"2024-09-11T08:16:20.086930+0000","last_undegraded":"2024-09-11T08:16:20.086930+0000","last_fullsized":"2024-09-11T08:16:20.086930+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:31.534302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:30.471620+0000","last_clean_scrub_stamp":"2024-09-11T08:15:31.534302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:43.916804+0000","scrub_duration":0.0021837200000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":215,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.519272+0000","last_change":"2024-09-11T08:15:56.526834+0000","last_active":"2024-09-11T08:16:05.519272+0000","last_peered":"2024-09-11T08:16:05.519272+0000","last_clean":"2024-09-11T08:16:05.519272+0000","last_became_active":"2024-09-11T08:15:56.525717+0000","last_became_peered":"2024-09-11T08:15:56.525717+0000","last_unstale":"2024-09-11T08:16:05.519272+0000","last_undegraded":"2024-09-11T08:16:05.519272+0000","last_fullsized":"2024-09-11T08:16:05.519272+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:59.988381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:04.370734+0000","last_clean_scrub_stamp":"2024-09-11T08:13:59.988381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017033700000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":222,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.533024+0000","last_change":"2024-09-11T08:15:56.525429+0000","last_active":"2024-09-11T08:16:05.533024+0000","last_peered":"2024-09-11T08:16:05.533024+0000","last_clean":"2024-09-11T08:16:05.533024+0000","last_became_active":"2024-09-11T08:15:56.525120+0000","last_became_peered":"2024-09-11T08:15:56.525120+0000","last_unstale":"2024-09-11T08:16:05.533024+0000","last_undegraded":"2024-09-11T08:16:05.533024+0000","last_fullsized":"2024-09-11T08:16:05.533024+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:54.184372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:54.184372+0000","last_clean_scrub_stamp":"2024-09-11T08:15:54.184372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:21.919337+0000","scrub_duration":0.013922767000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":272,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.527696+0000","last_change":"2024-09-11T08:15:56.547635+0000","last_active":"2024-09-11T08:16:05.527696+0000","last_peered":"2024-09-11T08:16:05.527696+0000","last_clean":"2024-09-11T08:16:05.527696+0000","last_became_active":"2024-09-11T08:15:56.532513+0000","last_became_peered":"2024-09-11T08:15:56.532513+0000","last_unstale":"2024-09-11T08:16:05.527696+0000","last_undegraded":"2024-09-11T08:16:05.527696+0000","last_fullsized":"2024-09-11T08:16:05.527696+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:15:52.443189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:15:52.443189+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:53.586813+0000","scrub_duration":0.010882618,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":219,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.526446+0000","last_change":"2024-09-11T08:15:56.513604+0000","last_active":"2024-09-11T08:16:05.526446+0000","last_peered":"2024-09-11T08:16:05.526446+0000","last_clean":"2024-09-11T08:16:05.526446+0000","last_became_active":"2024-09-11T08:15:56.513278+0000","last_became_peered":"2024-09-11T08:15:56.513278+0000","last_unstale":"2024-09-11T08:16:05.526446+0000","last_undegraded":"2024-09-11T08:16:05.526446+0000","last_fullsized":"2024-09-11T08:16:05.526446+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:48.213172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:48.935387+0000","last_clean_scrub_stamp":"2024-09-11T08:13:48.213172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011628039999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"209'11","reported_seq":337,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:13.653379+0000","last_change":"2024-09-11T08:15:56.543025+0000","last_active":"2024-09-11T08:16:13.653379+0000","last_peered":"2024-09-11T08:16:13.653379+0000","last_clean":"2024-09-11T08:16:13.653379+0000","last_became_active":"2024-09-11T08:15:56.525253+0000","last_became_peered":"2024-09-11T08:15:56.525253+0000","last_unstale":"2024-09-11T08:16:13.653379+0000","last_undegraded":"2024-09-11T08:16:13.653379+0000","last_fullsized":"2024-09-11T08:16:13.653379+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"156'10","last_scrub_stamp":"2024-09-11T08:14:01.979463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:14:01.979463+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012487329999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":188,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-11T08:16:20.074883+0000","last_change":"2024-09-11T08:15:05.867100+0000","last_active":"2024-09-11T08:16:20.074883+0000","last_peered":"2024-09-11T08:16:20.074883+0000","last_clean":"2024-09-11T08:16:20.074883+0000","last_became_active":"2024-09-11T08:07:36.068493+0000","last_became_peered":"2024-09-11T08:07:36.068493+0000","last_unstale":"2024-09-11T08:16:20.074883+0000","last_undegraded":"2024-09-11T08:16:20.074883+0000","last_fullsized":"2024-09-11T08:16:20.074883+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:05.867013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:05.867013+0000","last_clean_scrub_stamp":"2024-09-11T08:15:05.867013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0061248279999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":244,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.516142+0000","last_change":"2024-09-11T08:14:56.223381+0000","last_active":"2024-09-11T08:16:05.516142+0000","last_peered":"2024-09-11T08:16:05.516142+0000","last_clean":"2024-09-11T08:16:05.516142+0000","last_became_active":"2024-09-11T08:14:24.791187+0000","last_became_peered":"2024-09-11T08:14:24.791187+0000","last_unstale":"2024-09-11T08:16:05.516142+0000","last_undegraded":"2024-09-11T08:16:05.516142+0000","last_fullsized":"2024-09-11T08:16:05.516142+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-11T08:14:56.222082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:14:56.222082+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:21.292272+0000","scrub_duration":0.011605691,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":16,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-11T08:16:20.086740+0000","last_change":"2024-09-11T08:15:56.521890+0000","last_active":"2024-09-11T08:16:20.086740+0000","last_peered":"2024-09-11T08:16:20.086740+0000","last_clean":"2024-09-11T08:16:20.086740+0000","last_became_active":"2024-09-11T08:15:56.518515+0000","last_became_peered":"2024-09-11T08:15:56.518515+0000","last_unstale":"2024-09-11T08:16:20.086740+0000","last_undegraded":"2024-09-11T08:16:20.086740+0000","last_fullsized":"2024-09-11T08:16:20.086740+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:33.021088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:33.021088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:40.824210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":310,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:13.656426+0000","last_change":"2024-09-11T08:15:44.429144+0000","last_active":"2024-09-11T08:16:13.656426+0000","last_peered":"2024-09-11T08:16:13.656426+0000","last_clean":"2024-09-11T08:16:13.656426+0000","last_became_active":"2024-09-11T08:07:34.060736+0000","last_became_peered":"2024-09-11T08:07:34.060736+0000","last_unstale":"2024-09-11T08:16:13.656426+0000","last_undegraded":"2024-09-11T08:16:13.656426+0000","last_fullsized":"2024-09-11T08:16:13.656426+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:15:44.429062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:15:44.429062+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:58.251305+0000","scrub_duration":0.0062457210000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":213,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.529979+0000","last_change":"2024-09-11T08:15:56.515344+0000","last_active":"2024-09-11T08:16:05.529979+0000","last_peered":"2024-09-11T08:16:05.529979+0000","last_clean":"2024-09-11T08:16:05.529979+0000","last_became_active":"2024-09-11T08:15:56.513949+0000","last_became_peered":"2024-09-11T08:15:56.513949+0000","last_unstale":"2024-09-11T08:16:05.529979+0000","last_undegraded":"2024-09-11T08:16:05.529979+0000","last_fullsized":"2024-09-11T08:16:05.529979+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:01.448261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:15:01.448261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:25.797475+0000","scrub_duration":0.021382708,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":211,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-11T08:16:20.075702+0000","last_change":"2024-09-11T08:13:13.047503+0000","last_active":"2024-09-11T08:16:20.075702+0000","last_peered":"2024-09-11T08:16:20.075702+0000","last_clean":"2024-09-11T08:16:20.075702+0000","last_became_active":"2024-09-11T08:07:32.047981+0000","last_became_peered":"2024-09-11T08:07:32.047981+0000","last_unstale":"2024-09-11T08:16:20.075702+0000","last_undegraded":"2024-09-11T08:16:20.075702+0000","last_fullsized":"2024-09-11T08:16:20.075702+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-11T08:13:13.047436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:13:13.047436+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012804688,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":221,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-11T08:16:20.076356+0000","last_change":"2024-09-11T08:15:56.523858+0000","last_active":"2024-09-11T08:16:20.076356+0000","last_peered":"2024-09-11T08:16:20.076356+0000","last_clean":"2024-09-11T08:16:20.076356+0000","last_became_active":"2024-09-11T08:15:56.523574+0000","last_became_peered":"2024-09-11T08:15:56.523574+0000","last_unstale":"2024-09-11T08:16:20.076356+0000","last_undegraded":"2024-09-11T08:16:20.076356+0000","last_fullsized":"2024-09-11T08:16:20.076356+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:55.845422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:13:55.845422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00143399,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":201,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.520994+0000","last_change":"2024-09-11T08:15:38.410767+0000","last_active":"2024-09-11T08:16:05.520994+0000","last_peered":"2024-09-11T08:16:05.520994+0000","last_clean":"2024-09-11T08:16:05.520994+0000","last_became_active":"2024-09-11T08:07:34.053137+0000","last_became_peered":"2024-09-11T08:07:34.053137+0000","last_unstale":"2024-09-11T08:16:05.520994+0000","last_undegraded":"2024-09-11T08:16:05.520994+0000","last_fullsized":"2024-09-11T08:16:05.520994+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:38.410625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:15:38.410625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:59.728039+0000","scrub_duration":0.021540422,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":186,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.515407+0000","last_change":"2024-09-11T08:15:56.494833+0000","last_active":"2024-09-11T08:16:05.515407+0000","last_peered":"2024-09-11T08:16:05.515407+0000","last_clean":"2024-09-11T08:16:05.515407+0000","last_became_active":"2024-09-11T08:07:36.076132+0000","last_became_peered":"2024-09-11T08:07:36.076132+0000","last_unstale":"2024-09-11T08:16:05.515407+0000","last_undegraded":"2024-09-11T08:16:05.515407+0000","last_fullsized":"2024-09-11T08:16:05.515407+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:56.494764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:15:56.494764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:18.392464+0000","scrub_duration":0.001948635,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":226,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.521255+0000","last_change":"2024-09-11T08:16:04.409705+0000","last_active":"2024-09-11T08:16:05.521255+0000","last_peered":"2024-09-11T08:16:05.521255+0000","last_clean":"2024-09-11T08:16:05.521255+0000","last_became_active":"2024-09-11T08:07:32.049368+0000","last_became_peered":"2024-09-11T08:07:32.049368+0000","last_unstale":"2024-09-11T08:16:05.521255+0000","last_undegraded":"2024-09-11T08:16:05.521255+0000","last_fullsized":"2024-09-11T08:16:05.521255+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:16:04.409608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:16:04.409608+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:19.702294+0000","scrub_duration":0.0094835470000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":16,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-11T08:16:20.087166+0000","last_change":"2024-09-11T08:15:56.521976+0000","last_active":"2024-09-11T08:16:20.087166+0000","last_peered":"2024-09-11T08:16:20.087166+0000","last_clean":"2024-09-11T08:16:20.087166+0000","last_became_active":"2024-09-11T08:15:56.521570+0000","last_became_peered":"2024-09-11T08:15:56.521570+0000","last_unstale":"2024-09-11T08:16:20.087166+0000","last_undegraded":"2024-09-11T08:16:20.087166+0000","last_fullsized":"2024-09-11T08:16:20.087166+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:04.256038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:04.256038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":188,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.516146+0000","last_change":"2024-09-11T08:15:01.218687+0000","last_active":"2024-09-11T08:16:05.516146+0000","last_peered":"2024-09-11T08:16:05.516146+0000","last_clean":"2024-09-11T08:16:05.516146+0000","last_became_active":"2024-09-11T08:07:34.057064+0000","last_became_peered":"2024-09-11T08:07:34.057064+0000","last_unstale":"2024-09-11T08:16:05.516146+0000","last_undegraded":"2024-09-11T08:16:05.516146+0000","last_fullsized":"2024-09-11T08:16:05.516146+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:01.218475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:01.218475+0000","last_clean_scrub_stamp":"2024-09-11T08:15:01.218475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0099033660000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":186,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.524771+0000","last_change":"2024-09-11T08:13:08.164389+0000","last_active":"2024-09-11T08:16:05.524771+0000","last_peered":"2024-09-11T08:16:05.524771+0000","last_clean":"2024-09-11T08:16:05.524771+0000","last_became_active":"2024-09-11T08:07:36.077406+0000","last_became_peered":"2024-09-11T08:07:36.077406+0000","last_unstale":"2024-09-11T08:16:05.524771+0000","last_undegraded":"2024-09-11T08:16:05.524771+0000","last_fullsized":"2024-09-11T08:16:05.524771+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:08.164302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:09.973943+0000","last_clean_scrub_stamp":"2024-09-11T08:13:08.164302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003643444,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":260,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.515488+0000","last_change":"2024-09-11T08:15:56.543699+0000","last_active":"2024-09-11T08:16:05.515488+0000","last_peered":"2024-09-11T08:16:05.515488+0000","last_clean":"2024-09-11T08:16:05.515488+0000","last_became_active":"2024-09-11T08:15:56.520058+0000","last_became_peered":"2024-09-11T08:15:56.520058+0000","last_unstale":"2024-09-11T08:16:05.515488+0000","last_undegraded":"2024-09-11T08:16:05.515488+0000","last_fullsized":"2024-09-11T08:16:05.515488+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-11T08:14:10.129903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:14:10.129903+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004657011,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":228,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.526351+0000","last_change":"2024-09-11T08:15:56.531914+0000","last_active":"2024-09-11T08:16:05.526351+0000","last_peered":"2024-09-11T08:16:05.526351+0000","last_clean":"2024-09-11T08:16:05.526351+0000","last_became_active":"2024-09-11T08:15:56.531612+0000","last_became_peered":"2024-09-11T08:15:56.531612+0000","last_unstale":"2024-09-11T08:16:05.526351+0000","last_undegraded":"2024-09-11T08:16:05.526351+0000","last_fullsized":"2024-09-11T08:16:05.526351+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:49.478359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:49.478359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:16.513888+0000","scrub_duration":0.012202396000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":16,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-11T08:16:20.087124+0000","last_change":"2024-09-11T08:15:56.524265+0000","last_active":"2024-09-11T08:16:20.087124+0000","last_peered":"2024-09-11T08:16:20.087124+0000","last_clean":"2024-09-11T08:16:20.087124+0000","last_became_active":"2024-09-11T08:15:56.523728+0000","last_became_peered":"2024-09-11T08:15:56.523728+0000","last_unstale":"2024-09-11T08:16:20.087124+0000","last_undegraded":"2024-09-11T08:16:20.087124+0000","last_fullsized":"2024-09-11T08:16:20.087124+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:01.931182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:11:34.159053+0000","last_clean_scrub_stamp":"2024-09-11T08:14:01.931182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":186,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.516336+0000","last_change":"2024-09-11T08:15:44.448007+0000","last_active":"2024-09-11T08:16:05.516336+0000","last_peered":"2024-09-11T08:16:05.516336+0000","last_clean":"2024-09-11T08:16:05.516336+0000","last_became_active":"2024-09-11T08:07:36.633368+0000","last_became_peered":"2024-09-11T08:07:36.633368+0000","last_unstale":"2024-09-11T08:16:05.516336+0000","last_undegraded":"2024-09-11T08:16:05.516336+0000","last_fullsized":"2024-09-11T08:16:05.516336+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:44.447924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:34.378734+0000","last_clean_scrub_stamp":"2024-09-11T08:15:44.447924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:50.350614+0000","scrub_duration":0.0029812609999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":196,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.515404+0000","last_change":"2024-09-11T08:15:41.389724+0000","last_active":"2024-09-11T08:16:05.515404+0000","last_peered":"2024-09-11T08:16:05.515404+0000","last_clean":"2024-09-11T08:16:05.515404+0000","last_became_active":"2024-09-11T08:07:32.045728+0000","last_became_peered":"2024-09-11T08:07:32.045728+0000","last_unstale":"2024-09-11T08:16:05.515404+0000","last_undegraded":"2024-09-11T08:16:05.515404+0000","last_fullsized":"2024-09-11T08:16:05.515404+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'4","last_scrub_stamp":"2024-09-11T08:15:41.389510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:15:41.389510+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:10.070852+0000","scrub_duration":0.0069641269999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":202,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.527034+0000","last_change":"2024-09-11T08:15:39.743690+0000","last_active":"2024-09-11T08:16:05.527034+0000","last_peered":"2024-09-11T08:16:05.527034+0000","last_clean":"2024-09-11T08:16:05.527034+0000","last_became_active":"2024-09-11T08:07:30.049409+0000","last_became_peered":"2024-09-11T08:07:30.049409+0000","last_unstale":"2024-09-11T08:16:05.527034+0000","last_undegraded":"2024-09-11T08:16:05.527034+0000","last_fullsized":"2024-09-11T08:16:05.527034+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:39.743604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:39.743604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:08.408228+0000","scrub_duration":0.0089953759999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":236,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-11T08:16:20.082100+0000","last_change":"2024-09-11T08:15:56.529068+0000","last_active":"2024-09-11T08:16:20.082100+0000","last_peered":"2024-09-11T08:16:20.082100+0000","last_clean":"2024-09-11T08:16:20.082100+0000","last_became_active":"2024-09-11T08:15:56.528607+0000","last_became_peered":"2024-09-11T08:15:56.528607+0000","last_unstale":"2024-09-11T08:16:20.082100+0000","last_undegraded":"2024-09-11T08:16:20.082100+0000","last_fullsized":"2024-09-11T08:16:20.082100+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:37.177042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:15:37.177042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:50.577161+0000","scrub_duration":0.011175861,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":16,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-11T08:16:20.086504+0000","last_change":"2024-09-11T08:15:56.522225+0000","last_active":"2024-09-11T08:16:20.086504+0000","last_peered":"2024-09-11T08:16:20.086504+0000","last_clean":"2024-09-11T08:16:20.086504+0000","last_became_active":"2024-09-11T08:15:56.521488+0000","last_became_peered":"2024-09-11T08:15:56.521488+0000","last_unstale":"2024-09-11T08:16:20.086504+0000","last_undegraded":"2024-09-11T08:16:20.086504+0000","last_fullsized":"2024-09-11T08:16:20.086504+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:35.183289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:15:35.183289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:51.893171+0000","scrub_duration":0.007179726,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":196,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.521043+0000","last_change":"2024-09-11T08:14:07.076355+0000","last_active":"2024-09-11T08:16:05.521043+0000","last_peered":"2024-09-11T08:16:05.521043+0000","last_clean":"2024-09-11T08:16:05.521043+0000","last_became_active":"2024-09-11T08:07:36.071468+0000","last_became_peered":"2024-09-11T08:07:36.071468+0000","last_unstale":"2024-09-11T08:16:05.521043+0000","last_undegraded":"2024-09-11T08:16:05.521043+0000","last_fullsized":"2024-09-11T08:16:05.521043+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:07.076300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:12:25.814048+0000","last_clean_scrub_stamp":"2024-09-11T08:14:07.076300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001297334,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":202,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T08:16:05.515482+0000","last_change":"2024-09-11T08:15:47.493538+0000","last_active":"2024-09-11T08:16:05.515482+0000","last_peered":"2024-09-11T08:16:05.515482+0000","last_clean":"2024-09-11T08:16:05.515482+0000","last_became_active":"2024-09-11T08:07:30.046702+0000","last_became_peered":"2024-09-11T08:07:30.046702+0000","last_unstale":"2024-09-11T08:16:05.515482+0000","last_undegraded":"2024-09-11T08:16:05.515482+0000","last_fullsized":"2024-09-11T08:16:05.515482+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:47.493466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:47.493466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:47.631872+0000","scrub_duration":0.071745623999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":230,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-11T08:16:20.081226+0000","last_change":"2024-09-11T08:15:55.255237+0000","last_active":"2024-09-11T08:16:20.081226+0000","last_peered":"2024-09-11T08:16:20.081226+0000","last_clean":"2024-09-11T08:16:20.081226+0000","last_became_active":"2024-09-11T08:07:32.054656+0000","last_became_peered":"2024-09-11T08:07:32.054656+0000","last_unstale":"2024-09-11T08:16:20.081226+0000","last_undegraded":"2024-09-11T08:16:20.081226+0000","last_fullsized":"2024-09-11T08:16:20.081226+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-11T08:15:55.255152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:15:55.255152+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:09.255947+0000","scrub_duration":0.0070337539999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":34,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-11T08:16:20.080566+0000","last_change":"2024-09-11T08:15:08.769594+0000","last_active":"2024-09-11T08:16:20.080566+0000","last_peered":"2024-09-11T08:16:20.080566+0000","last_clean":"2024-09-11T08:16:20.080566+0000","last_became_active":"2024-09-11T08:15:08.769133+0000","last_became_peered":"2024-09-11T08:15:08.769133+0000","last_unstale":"2024-09-11T08:16:20.080566+0000","last_undegraded":"2024-09-11T08:16:20.080566+0000","last_fullsized":"2024-09-11T08:16:20.080566+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:55.501694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:14:55.501694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:25.384157+0000","scrub_duration":0.024419420000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":5578358808,"num_objects":85120,"num_object_clones":0,"num_object_copies":255360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4242,"num_objects_unfound":0,"num_objects_dirty":85120,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172540,"num_write_kb":8245057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13859,"num_bytes_recovered":908197912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17147052032,"data_stored":17147035744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85383,"ondisk_log_size":85383,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":139,"num_bytes_recovered":2888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3425,"internal_metadata":0},"log_size":450,"ondisk_log_size":450,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":124,"num_read_kb":119,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":168,"seq":721554505753,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1917616,"kb_used_data":1591780,"kb_used_omap":7,"kb_used_meta":325816,"kb_avail":91827536,"statfs":{"total":95995035648,"available":94031396864,"internally_reserved":0,"allocated":1629982720,"data_stored":1629612107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8027,"internal_metadata":333635749},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:15:25 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":2.9780000000000002,"5min":2.9780000000000002,"15min":2.9780000000000002},"last":0.223},{"interface":"front","average":{"1min":0.995,"5min":0.995,"15min":0.995},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":2.8069999999999999,"5min":2.8069999999999999,"15min":2.8069999999999999},"last":0.26300000000000001}]},{"osd":1,"last update":"Wed Sep 11 08:15:25 2024","interfaces":[{"interface":"back","average":{"1min":1.946,"5min":1.946,"15min":1.946},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":26.390999999999998,"5min":26.390999999999998,"15min":26.390999999999998},"last":4.7329999999999997},{"interface":"front","average":{"1min":2.0459999999999998,"5min":2.0459999999999998,"15min":2.0459999999999998},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":26.481000000000002,"5min":26.481000000000002,"15min":26.481000000000002},"last":0.47999999999999998}]},{"osd":2,"last update":"Wed Sep 11 08:15:25 2024","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.042,"15min":1.042},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":3.9630000000000001,"5min":3.9630000000000001,"15min":3.9630000000000001},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":2.601,"5min":2.601,"15min":2.601},"last":0.42199999999999999}]},{"osd":3,"last update":"Wed Sep 11 08:15:25 2024","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.073,"15min":1.073},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":3.702,"5min":3.702,"15min":3.702},"last":0.52900000000000003},{"interface":"front","average":{"1min":1.425,"5min":1.425,"15min":1.425},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":3.8109999999999999,"5min":3.8109999999999999,"15min":3.8109999999999999},"last":0.68700000000000006}]},{"osd":4,"last update":"Wed Sep 11 08:15:25 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.7470000000000001,"5min":1.7470000000000001,"15min":1.7470000000000001},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":2.153,"5min":2.153,"15min":2.153},"last":0.317}]},{"osd":5,"last update":"Wed Sep 11 08:15:25 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":2.6139999999999999,"5min":2.6139999999999999,"15min":2.6139999999999999},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":2.6240000000000001,"5min":2.6240000000000001,"15min":2.6240000000000001},"last":0.95399999999999996}]},{"osd":6,"last update":"Wed Sep 11 08:15:25 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":2.8370000000000002,"5min":2.8370000000000002,"15min":2.8370000000000002},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":2.0449999999999999,"5min":2.0449999999999999,"15min":2.0449999999999999},"last":0.73799999999999999}]}]},{"osd":6,"up_from":45,"seq":193273528436,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2277548,"kb_used_data":1967636,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":91467604,"statfs":{"total":95995035648,"available":93662826496,"internally_reserved":0,"allocated":2014859264,"data_stored":2014504858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:16:10 2024","interfaces":[{"interface":"back","average":{"1min":1.53,"5min":2.1360000000000001,"15min":1.0600000000000001},"min":{"1min":0.20899999999999999,"5min":0.20399999999999999,"15min":0.185},"max":{"1min":7.5810000000000004,"5min":45.066000000000003,"15min":45.066000000000003},"last":0.31},{"interface":"front","average":{"1min":1.5189999999999999,"5min":1.9510000000000001,"15min":1.026},"min":{"1min":0.224,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":4.1379999999999999,"5min":43.899000000000001,"15min":43.899000000000001},"last":0.629}]},{"osd":1,"last update":"Wed Sep 11 08:16:10 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.7010000000000001,"15min":0.89500000000000002},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":3.931,"5min":44.658000000000001,"15min":44.658000000000001},"last":0.32900000000000001},{"interface":"front","average":{"1min":1.506,"5min":1.827,"15min":0.95799999999999996},"min":{"1min":0.32500000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":6.4870000000000001,"5min":44.694000000000003,"15min":44.694000000000003},"last":0.53300000000000003}]},{"osd":2,"last update":"Wed Sep 11 08:16:10 2024","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.0169999999999999,"15min":0.70099999999999996},"min":{"1min":0.28699999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":5.9770000000000003,"5min":5.9770000000000003,"15min":5.9770000000000003},"last":0.58299999999999996},{"interface":"front","average":{"1min":1.9790000000000001,"5min":1.45,"15min":0.84199999999999997},"min":{"1min":0.29299999999999998,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":7.4809999999999999,"5min":13.346,"15min":13.346},"last":0.65700000000000003}]},{"osd":3,"last update":"Wed Sep 11 08:16:10 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.7509999999999999,"15min":0.996},"min":{"1min":0.23599999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":3.3980000000000001,"5min":36.061,"15min":36.061},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.90300000000000002,"5min":1.718,"15min":0.97699999999999998},"min":{"1min":0.30599999999999999,"5min":0.28399999999999997,"15min":0.26800000000000002},"max":{"1min":2.6749999999999998,"5min":36.093000000000004,"15min":36.093000000000004},"last":0.56200000000000006}]},{"osd":4,"last update":"Wed Sep 11 08:16:10 2024","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":0.57799999999999996,"15min":0.45700000000000002},"min":{"1min":0.23699999999999999,"5min":0.219,"15min":0.20200000000000001},"max":{"1min":4.1130000000000004,"5min":4.1130000000000004,"15min":4.1130000000000004},"last":2.8220000000000001},{"interface":"front","average":{"1min":1.1579999999999999,"5min":0.64700000000000002,"15min":0.499},"min":{"1min":0.27100000000000002,"5min":0.251,"15min":0.24099999999999999},"max":{"1min":4.468,"5min":4.468,"15min":4.468},"last":0.71599999999999997}]},{"osd":5,"last update":"Wed Sep 11 08:16:10 2024","interfaces":[{"interface":"back","average":{"1min":1.514,"5min":0.76400000000000001,"15min":0.57999999999999996},"min":{"1min":0.37,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":7.4459999999999997,"5min":7.4459999999999997,"15min":7.4459999999999997},"last":0.67800000000000005},{"interface":"front","average":{"1min":1.246,"5min":0.747,"15min":0.53400000000000003},"min":{"1min":0.28399999999999997,"5min":0.22800000000000001,"15min":0.22600000000000001},"max":{"1min":3.4329999999999998,"5min":3.4329999999999998,"15min":3.4329999999999998},"last":0.28199999999999997}]},{"osd":7,"last update":"Wed Sep 11 08:15:23 2024","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":2.9980000000000002,"5min":2.9980000000000002,"15min":2.9980000000000002},"last":0.999},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":3.2919999999999998,"5min":3.2919999999999998,"15min":3.2919999999999998},"last":3.121}]}]},{"osd":5,"up_from":38,"seq":163208757368,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3436940,"kb_used_data":3135428,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":90308212,"statfs":{"total":95995035648,"available":92475609088,"internally_reserved":0,"allocated":3210678272,"data_stored":3210300512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:15:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4430000000000001,"5min":1.254,"15min":0.78800000000000003},"min":{"1min":0.20999999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":4.327,"5min":7.9470000000000001,"15min":7.9470000000000001},"last":0.29799999999999999},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.222,"15min":0.77700000000000002},"min":{"1min":0.24399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":3.4820000000000002,"5min":7.6189999999999998,"15min":7.6189999999999998},"last":0.48099999999999998}]},{"osd":1,"last update":"Wed Sep 11 08:15:48 2024","interfaces":[{"interface":"back","average":{"1min":2.77,"5min":1.5760000000000001,"15min":1.2829999999999999},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":26.709,"5min":26.709,"15min":26.709},"last":0.59299999999999997},{"interface":"front","average":{"1min":2.5339999999999998,"5min":1.5569999999999999,"15min":0.85899999999999999},"min":{"1min":0.25900000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":26.754000000000001,"5min":26.754000000000001,"15min":26.754000000000001},"last":0.53200000000000003}]},{"osd":2,"last update":"Wed Sep 11 08:15:48 2024","interfaces":[{"interface":"back","average":{"1min":1.891,"5min":1.2589999999999999,"15min":1.171},"min":{"1min":0.36799999999999999,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":10.891,"5min":10.891,"15min":16.018000000000001},"last":0.374},{"interface":"front","average":{"1min":1.4330000000000001,"5min":0.99099999999999999,"15min":0.68500000000000005},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":7.5250000000000004,"5min":7.5250000000000004,"15min":7.5250000000000004},"last":0.49399999999999999}]},{"osd":3,"last update":"Wed Sep 11 08:15:48 2024","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.3819999999999999,"15min":0.85399999999999998},"min":{"1min":0.29999999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":4.3150000000000004,"5min":24.931000000000001,"15min":24.931000000000001},"last":0.91200000000000003},{"interface":"front","average":{"1min":1.544,"5min":1.514,"15min":0.879},"min":{"1min":0.38800000000000001,"5min":0.251,"15min":0.251},"max":{"1min":4.4710000000000001,"5min":24.861999999999998,"15min":24.861999999999998},"last":0.57099999999999995}]},{"osd":4,"last update":"Wed Sep 11 08:15:48 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.58199999999999996,"15min":0.46999999999999997},"min":{"1min":0.23499999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5979999999999999,"5min":3.5979999999999999,"15min":3.5979999999999999},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.56200000000000006,"15min":0.46200000000000002},"min":{"1min":0.27300000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":2.706,"5min":2.706,"15min":2.706},"last":0.42099999999999999}]},{"osd":6,"last update":"Wed Sep 11 08:16:12 2024","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":0.54300000000000004,"15min":0.47299999999999998},"min":{"1min":0.30399999999999999,"5min":0.252,"15min":0.23400000000000001},"max":{"1min":3.1869999999999998,"5min":3.1869999999999998,"15min":3.1869999999999998},"last":0.33700000000000002},{"interface":"front","average":{"1min":1.0129999999999999,"5min":0.64500000000000002,"15min":0.53100000000000003},"min":{"1min":0.35199999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":3.5169999999999999,"5min":3.9489999999999998,"15min":3.9489999999999998},"last":0.622}]},{"osd":7,"last update":"Wed Sep 11 08:15:25 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":3.1320000000000001,"5min":3.1320000000000001,"15min":3.1320000000000001},"last":0.754},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":3.0499999999999998,"5min":3.0499999999999998,"15min":3.0499999999999998},"last":0.54600000000000004}]}]},{"osd":4,"up_from":31,"seq":133143986302,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1922992,"kb_used_data":1625636,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":91822160,"statfs":{"total":95995035648,"available":94025891840,"internally_reserved":0,"allocated":1664651264,"data_stored":1664280781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:16:20 2024","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":1.0600000000000001,"15min":1.0660000000000001},"min":{"1min":0.23300000000000001,"5min":0.19,"15min":0.19},"max":{"1min":5.306,"5min":6.1159999999999997,"15min":48.728999999999999},"last":0.625},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.85099999999999998,"15min":0.93300000000000005},"min":{"1min":0.22900000000000001,"5min":0.221,"15min":0.221},"max":{"1min":2.8650000000000002,"5min":5.5599999999999996,"15min":49.168999999999997},"last":0.92300000000000004}]},{"osd":1,"last update":"Wed Sep 11 08:16:20 2024","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.0049999999999999,"15min":0.753},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":3.5470000000000002,"5min":4.7320000000000002,"15min":13.207000000000001},"last":0.78100000000000003},{"interface":"front","average":{"1min":1.3560000000000001,"5min":0.98599999999999999,"15min":0.72199999999999998},"min":{"1min":0.24299999999999999,"5min":0.22900000000000001,"15min":0.22700000000000001},"max":{"1min":5.8869999999999996,"5min":5.8869999999999996,"15min":5.8869999999999996},"last":0.59499999999999997}]},{"osd":2,"last update":"Wed Sep 11 08:16:20 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.0549999999999999,"15min":0.74199999999999999},"min":{"1min":0.245,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":3.3599999999999999,"5min":7.976,"15min":7.976},"last":0.76600000000000001},{"interface":"front","average":{"1min":1.085,"5min":0.871,"15min":0.71499999999999997},"min":{"1min":0.32700000000000001,"5min":0.248,"15min":0.248},"max":{"1min":3.1389999999999998,"5min":3.6989999999999998,"15min":5.5670000000000002},"last":0.85999999999999999}]},{"osd":3,"last update":"Wed Sep 11 08:16:20 2024","interfaces":[{"interface":"back","average":{"1min":1.986,"5min":1.3129999999999999,"15min":1.0349999999999999},"min":{"1min":0.27600000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":19.686,"5min":19.686,"15min":26.884},"last":0.98199999999999998},{"interface":"front","average":{"1min":1.8660000000000001,"5min":1.4550000000000001,"15min":1.0549999999999999},"min":{"1min":0.26600000000000001,"5min":0.246,"15min":0.246},"max":{"1min":19.882000000000001,"5min":19.882000000000001,"15min":26.783000000000001},"last":1.8029999999999999}]},{"osd":5,"last update":"Wed Sep 11 08:15:45 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.497,"15min":0.42299999999999999},"min":{"1min":0.26400000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":2.5489999999999999,"5min":2.5489999999999999,"15min":2.5489999999999999},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.48299999999999998,"15min":0.41499999999999998},"min":{"1min":0.253,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.7390000000000001,"5min":1.9630000000000001,"15min":1.9630000000000001},"last":0.90300000000000002}]},{"osd":6,"last update":"Wed Sep 11 08:16:14 2024","interfaces":[{"interface":"back","average":{"1min":3.048,"5min":1.004,"15min":0.59599999999999997},"min":{"1min":0.28599999999999998,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":48.808,"5min":48.808,"15min":48.808},"last":0.82599999999999996},{"interface":"front","average":{"1min":2.988,"5min":1.032,"15min":0.64800000000000002},"min":{"1min":0.27800000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":48.746000000000002,"5min":48.746000000000002,"15min":48.746000000000002},"last":0.317}]},{"osd":7,"last update":"Wed Sep 11 08:15:25 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.88300000000000001},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":3.2770000000000001,"5min":3.2770000000000001,"15min":3.2770000000000001},"last":0.747}]}]},{"osd":3,"up_from":25,"seq":107374182530,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2378068,"kb_used_data":2072032,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":91367084,"statfs":{"total":95995035648,"available":93559894016,"internally_reserved":0,"allocated":2121760768,"data_stored":2121355097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:16:04 2024","interfaces":[{"interface":"back","average":{"1min":4.2320000000000002,"5min":2.363,"15min":1.099},"min":{"1min":0.20200000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":60.469999999999999,"5min":60.469999999999999,"15min":60.469999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":4.4939999999999998,"5min":2.613,"15min":1.1990000000000001},"min":{"1min":0.245,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":60.488999999999997,"5min":60.488999999999997,"15min":60.488999999999997},"last":1.623}]},{"osd":1,"last update":"Wed Sep 11 08:16:04 2024","interfaces":[{"interface":"back","average":{"1min":4.0620000000000003,"5min":2.2679999999999998,"15min":1.0860000000000001},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.20899999999999999},"max":{"1min":60.578000000000003,"5min":60.578000000000003,"15min":60.578000000000003},"last":1.6830000000000001},{"interface":"front","average":{"1min":4.5910000000000002,"5min":2.411,"15min":1.1830000000000001},"min":{"1min":0.23400000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":63.674999999999997,"5min":63.674999999999997,"15min":63.674999999999997},"last":1.397}]},{"osd":2,"last update":"Wed Sep 11 08:16:04 2024","interfaces":[{"interface":"back","average":{"1min":4.6840000000000002,"5min":2.6139999999999999,"15min":1.222},"min":{"1min":0.22,"5min":0.185,"15min":0.185},"max":{"1min":60.645000000000003,"5min":60.645000000000003,"15min":60.645000000000003},"last":1.2130000000000001},{"interface":"front","average":{"1min":4.6740000000000004,"5min":2.3740000000000001,"15min":1.1379999999999999},"min":{"1min":0.38600000000000001,"5min":0.20699999999999999,"15min":0.20599999999999999},"max":{"1min":60.723999999999997,"5min":60.723999999999997,"15min":60.723999999999997},"last":0.316}]},{"osd":4,"last update":"Wed Sep 11 08:15:20 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":1.3999999999999999,"15min":0.83699999999999997},"min":{"1min":0.28499999999999998,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":5.1829999999999998,"5min":6.468,"15min":6.468},"last":1.25},{"interface":"front","average":{"1min":1.417,"5min":1.3460000000000001,"15min":0.80200000000000005},"min":{"1min":0.32700000000000001,"5min":0.27500000000000002,"15min":0.27200000000000002},"max":{"1min":5.0199999999999996,"5min":10.563000000000001,"15min":10.563000000000001},"last":0.41399999999999998}]},{"osd":5,"last update":"Wed Sep 11 08:15:47 2024","interfaces":[{"interface":"back","average":{"1min":6.5309999999999997,"5min":2.3620000000000001,"15min":1.159},"min":{"1min":0.40000000000000002,"5min":0.26600000000000001,"15min":0.26500000000000001},"max":{"1min":60.914999999999999,"5min":60.914999999999999,"15min":60.914999999999999},"last":0.34599999999999997},{"interface":"front","average":{"1min":6.2590000000000003,"5min":2.3730000000000002,"15min":1.242},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":60.787999999999997,"5min":60.787999999999997,"15min":60.787999999999997},"last":0.72699999999999998}]},{"osd":6,"last update":"Wed Sep 11 08:16:10 2024","interfaces":[{"interface":"back","average":{"1min":4.3019999999999996,"5min":2.4329999999999998,"15min":1.286},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":60.817,"5min":60.817,"15min":60.817},"last":0.75900000000000001},{"interface":"front","average":{"1min":4.4530000000000003,"5min":2.5169999999999999,"15min":1.371},"min":{"1min":0.505,"5min":0.307,"15min":0.307},"max":{"1min":61.264000000000003,"5min":61.264000000000003,"15min":61.264000000000003},"last":1.4239999999999999}]},{"osd":7,"last update":"Wed Sep 11 08:15:24 2024","interfaces":[{"interface":"back","average":{"1min":3.1419999999999999,"5min":3.1419999999999999,"15min":3.1419999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":32.000999999999998,"5min":32.000999999999998,"15min":32.000999999999998},"last":0.78000000000000003},{"interface":"front","average":{"1min":3.073,"5min":3.073,"15min":3.073},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":32.792999999999999,"5min":32.792999999999999,"15min":32.792999999999999},"last":0.502}]}]},{"osd":2,"up_from":18,"seq":77309411461,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2403768,"kb_used_data":2097732,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":91341384,"statfs":{"total":95995035648,"available":93533577216,"internally_reserved":0,"allocated":2148077568,"data_stored":2147699720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:15:42 2024","interfaces":[{"interface":"back","average":{"1min":4.8499999999999996,"5min":2.3260000000000001,"15min":1.071},"min":{"1min":0.32800000000000001,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":43.539999999999999,"5min":43.539999999999999,"15min":43.539999999999999},"last":0.33500000000000002},{"interface":"front","average":{"1min":5.2800000000000002,"5min":2.5870000000000002,"15min":1.139},"min":{"1min":0.20399999999999999,"5min":0.185,"15min":0.185},"max":{"1min":41.962000000000003,"5min":41.962000000000003,"15min":41.962000000000003},"last":1.214}]},{"osd":1,"last update":"Wed Sep 11 08:15:42 2024","interfaces":[{"interface":"back","average":{"1min":1.992,"5min":1.734,"15min":0.88700000000000001},"min":{"1min":0.182,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":13.138999999999999,"5min":13.138999999999999,"15min":13.138999999999999},"last":0.30199999999999999},{"interface":"front","average":{"1min":2.7309999999999999,"5min":1.9019999999999999,"15min":0.94099999999999995},"min":{"1min":0.26600000000000001,"5min":0.23200000000000001,"15min":0.20200000000000001},"max":{"1min":10.167999999999999,"5min":10.167999999999999,"15min":10.167999999999999},"last":0.92900000000000005}]},{"osd":3,"last update":"Wed Sep 11 08:16:05 2024","interfaces":[{"interface":"back","average":{"1min":2.5070000000000001,"5min":1.97,"15min":0.998},"min":{"1min":0.21299999999999999,"5min":0.188,"15min":0.188},"max":{"1min":8.1989999999999998,"5min":8.1989999999999998,"15min":8.1989999999999998},"last":1.1259999999999999},{"interface":"front","average":{"1min":2.4289999999999998,"5min":1.879,"15min":0.97299999999999998},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.22500000000000001},"max":{"1min":8.0640000000000001,"5min":8.3219999999999992,"15min":8.3219999999999992},"last":1.4259999999999999}]},{"osd":4,"last update":"Wed Sep 11 08:15:25 2024","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.242,"15min":0.77800000000000002},"min":{"1min":0.32300000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":4.923,"5min":7.806,"15min":7.806},"last":0.95099999999999996},{"interface":"front","average":{"1min":1.248,"5min":1.206,"15min":0.83099999999999996},"min":{"1min":0.28699999999999998,"5min":0.252,"15min":0.252},"max":{"1min":5.4740000000000002,"5min":6.8200000000000003,"15min":12.129},"last":0.97399999999999998}]},{"osd":5,"last update":"Wed Sep 11 08:15:57 2024","interfaces":[{"interface":"back","average":{"1min":1.6120000000000001,"5min":1.3100000000000001,"15min":0.85199999999999998},"min":{"1min":0.39900000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":5.4539999999999997,"5min":6.7709999999999999,"15min":6.7709999999999999},"last":1.069},{"interface":"front","average":{"1min":1.196,"5min":1.3160000000000001,"15min":0.84599999999999997},"min":{"1min":0.36299999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":4.024,"5min":12.869,"15min":12.869},"last":0.44500000000000001}]},{"osd":6,"last update":"Wed Sep 11 08:16:10 2024","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.2090000000000001,"15min":0.81599999999999995},"min":{"1min":0.44700000000000001,"5min":0.32600000000000001,"15min":0.26100000000000001},"max":{"1min":4.2539999999999996,"5min":7.2249999999999996,"15min":7.2249999999999996},"last":0.47199999999999998},{"interface":"front","average":{"1min":1.819,"5min":1.3839999999999999,"15min":0.89700000000000002},"min":{"1min":0.35499999999999998,"5min":0.33200000000000002,"15min":0.32100000000000001},"max":{"1min":5.4939999999999998,"5min":6.71,"15min":6.71},"last":1.1060000000000001}]},{"osd":7,"last update":"Wed Sep 11 08:15:25 2024","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.389,"15min":1.389},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":4.9569999999999999,"5min":4.9569999999999999,"15min":4.9569999999999999},"last":1.0169999999999999},{"interface":"front","average":{"1min":1.5369999999999999,"5min":1.5369999999999999,"15min":1.5369999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":5.3710000000000004,"5min":5.3710000000000004,"15min":5.3710000000000004},"last":1.0029999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574987,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2726876,"kb_used_data":2425300,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":91018276,"statfs":{"total":95995035648,"available":93202714624,"internally_reserved":0,"allocated":2483507200,"data_stored":2483113602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:16:17 2024","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.3779999999999999,"15min":0.90800000000000003},"min":{"1min":0.24099999999999999,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":4.2199999999999998,"5min":8.1509999999999998,"15min":32.478000000000002},"last":0.27900000000000003},{"interface":"front","average":{"1min":1.9970000000000001,"5min":1.9419999999999999,"15min":1.177},"min":{"1min":0.28000000000000003,"5min":0.186,"15min":0.186},"max":{"1min":5.4210000000000003,"5min":10.247,"15min":35.459000000000003},"last":0.74399999999999999}]},{"osd":2,"last update":"Wed Sep 11 08:15:49 2024","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.5800000000000001,"15min":0.79600000000000004},"min":{"1min":0.26400000000000001,"5min":0.216,"15min":0.21299999999999999},"max":{"1min":7.9699999999999998,"5min":8.3819999999999997,"15min":8.3819999999999997},"last":2.1269999999999998},{"interface":"front","average":{"1min":1.5229999999999999,"5min":1.8779999999999999,"15min":0.92000000000000004},"min":{"1min":0.192,"5min":0.192,"15min":0.16300000000000001},"max":{"1min":5.1319999999999997,"5min":10.276,"15min":10.276},"last":1.829}]},{"osd":3,"last update":"Wed Sep 11 08:16:01 2024","interfaces":[{"interface":"back","average":{"1min":1.835,"5min":2.3479999999999999,"15min":1.339},"min":{"1min":0.38600000000000001,"5min":0.248,"15min":0.246},"max":{"1min":5.1689999999999996,"5min":16.783999999999999,"15min":32.759},"last":0.97199999999999998},{"interface":"front","average":{"1min":1.0469999999999999,"5min":2.008,"15min":1.175},"min":{"1min":0.27000000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":4.2489999999999997,"5min":14.973000000000001,"15min":32.444000000000003},"last":0.49399999999999999}]},{"osd":4,"last update":"Wed Sep 11 08:15:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.3120000000000001,"15min":0.81499999999999995},"min":{"1min":0.26300000000000001,"5min":0.252,"15min":0.252},"max":{"1min":3.653,"5min":9.0139999999999993,"15min":9.0139999999999993},"last":2.8639999999999999},{"interface":"front","average":{"1min":0.93400000000000005,"5min":1.266,"15min":0.79800000000000004},"min":{"1min":0.373,"5min":0.248,"15min":0.248},"max":{"1min":2.3540000000000001,"5min":8.2360000000000007,"15min":8.2360000000000007},"last":0.71399999999999997}]},{"osd":5,"last update":"Wed Sep 11 08:15:49 2024","interfaces":[{"interface":"back","average":{"1min":1.6859999999999999,"5min":1.4399999999999999,"15min":0.90900000000000003},"min":{"1min":0.316,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":5.3869999999999996,"5min":8.6769999999999996,"15min":9.0470000000000006},"last":3.1819999999999999},{"interface":"front","average":{"1min":1.8959999999999999,"5min":1.6399999999999999,"15min":0.95399999999999996},"min":{"1min":0.35199999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":4.5019999999999998,"5min":8.9860000000000007,"15min":11.967000000000001},"last":3.8559999999999999}]},{"osd":6,"last update":"Wed Sep 11 08:16:03 2024","interfaces":[{"interface":"back","average":{"1min":7.8250000000000002,"5min":2.7599999999999998,"15min":1.3859999999999999},"min":{"1min":0.32600000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":124.31999999999999,"5min":124.31999999999999,"15min":124.31999999999999},"last":1.532},{"interface":"front","average":{"1min":7.6020000000000003,"5min":2.6779999999999999,"15min":1.341},"min":{"1min":0.39200000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":124.958,"5min":124.958,"15min":124.958},"last":0.46899999999999997}]},{"osd":7,"last update":"Wed Sep 11 08:15:26 2024","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":3.6850000000000001,"5min":3.6850000000000001,"15min":3.6850000000000001},"last":1.863},{"interface":"front","average":{"1min":1.48,"5min":1.48,"15min":1.48},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":4.593,"5min":4.593,"15min":4.593},"last":0.80900000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738511,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2383668,"kb_used_data":2073772,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":91361484,"statfs":{"total":95995035648,"available":93554159616,"internally_reserved":0,"allocated":2123542528,"data_stored":2123126248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 11 08:15:19 2024","interfaces":[{"interface":"back","average":{"1min":1.5940000000000001,"5min":1.6850000000000001,"15min":0.78500000000000003},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":6.9580000000000002,"5min":7.2160000000000002,"15min":7.2160000000000002},"last":0.873},{"interface":"front","average":{"1min":1.998,"5min":1.899,"15min":0.86499999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":6.8579999999999997,"5min":9.5679999999999996,"15min":9.5679999999999996},"last":1.399}]},{"osd":2,"last update":"Wed Sep 11 08:15:43 2024","interfaces":[{"interface":"back","average":{"1min":1.607,"5min":1.5620000000000001,"15min":0.80000000000000004},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":6.875,"5min":10.351000000000001,"15min":10.351000000000001},"last":0.80400000000000005},{"interface":"front","average":{"1min":1.421,"5min":1.7829999999999999,"15min":0.88600000000000001},"min":{"1min":0.24299999999999999,"5min":0.186,"15min":0.186},"max":{"1min":6.2320000000000002,"5min":7.3090000000000002,"15min":7.3090000000000002},"last":0.78300000000000003}]},{"osd":3,"last update":"Wed Sep 11 08:16:05 2024","interfaces":[{"interface":"back","average":{"1min":1.546,"5min":2.2549999999999999,"15min":1.2130000000000001},"min":{"1min":0.29299999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":4.5679999999999996,"5min":35.491,"15min":35.491},"last":0.82399999999999995},{"interface":"front","average":{"1min":2.2240000000000002,"5min":2.6150000000000002,"15min":1.246},"min":{"1min":0.32900000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.7919999999999998,"5min":37.563000000000002,"15min":37.563000000000002},"last":0.61199999999999999}]},{"osd":4,"last update":"Wed Sep 11 08:15:24 2024","interfaces":[{"interface":"back","average":{"1min":1.6160000000000001,"5min":1.51,"15min":0.86899999999999999},"min":{"1min":0.26900000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":6.0190000000000001,"5min":9.4030000000000005,"15min":9.4030000000000005},"last":0.69999999999999996},{"interface":"front","average":{"1min":1.373,"5min":1.4650000000000001,"15min":0.86499999999999999},"min":{"1min":0.32400000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":4.2460000000000004,"5min":6.6619999999999999,"15min":6.6619999999999999},"last":0.89200000000000002}]},{"osd":5,"last update":"Wed Sep 11 08:15:53 2024","interfaces":[{"interface":"back","average":{"1min":2.4529999999999998,"5min":1.696,"15min":0.98899999999999999},"min":{"1min":0.33100000000000002,"5min":0.255,"15min":0.255},"max":{"1min":24.914999999999999,"5min":24.914999999999999,"15min":24.914999999999999},"last":0.85199999999999998},{"interface":"front","average":{"1min":3.173,"5min":1.9299999999999999,"15min":1.0720000000000001},"min":{"1min":0.34100000000000003,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":25.277000000000001,"5min":25.277000000000001,"15min":25.277000000000001},"last":0.437}]},{"osd":6,"last update":"Wed Sep 11 08:16:10 2024","interfaces":[{"interface":"back","average":{"1min":2.0230000000000001,"5min":2.4950000000000001,"15min":1.306},"min":{"1min":0.29599999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":5.085,"5min":78.522000000000006,"15min":78.522000000000006},"last":0.46999999999999997},{"interface":"front","average":{"1min":1.514,"5min":1.986,"15min":1.145},"min":{"1min":0.29899999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":6.8520000000000003,"5min":79.808000000000007,"15min":79.808000000000007},"last":0.90400000000000003}]},{"osd":7,"last update":"Wed Sep 11 08:15:24 2024","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.3620000000000001,"15min":1.3620000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":6.7750000000000004,"5min":6.7750000000000004,"15min":6.7750000000000004},"last":1.286},{"interface":"front","average":{"1min":1.355,"5min":1.355,"15min":1.355},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":6.8179999999999996,"5min":6.8179999999999996,"15min":6.8179999999999996},"last":1.26}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3425,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2108096512,"data_stored":2108096512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2452226048,"data_stored":2452226048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2122387456,"data_stored":2122383384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2107441152,"data_stored":2107441152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1648754688,"data_stored":1648754688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3194490880,"data_stored":3194486808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2002718720,"data_stored":2002714648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1510936576,"data_stored":1510932504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T08:16:26.579 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph status --format=json 2024-09-11T08:16:26.579 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:26 smithi181 ceph-mon[32701]: pgmap v394: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 922 op/s; 2375/242547 objects misplaced (0.979%); 17 MiB/s, 274 objects/s recovering 2024-09-11T08:16:26.579 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:26 smithi181 ceph-mon[32701]: osdmap e216: 8 total, 8 up, 8 in 2024-09-11T08:16:26.605 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:16:26.613 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:26.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:26.680 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:26.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:26.781 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:26.805 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:26.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:26.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:26.907 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:26.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:26.941 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:26.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:27.042 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:16:27.073 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:27.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:27.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:16:27.174 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:27.214 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:27.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:27.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:27.316 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:27.353 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:27.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:27.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:27.454 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:27.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:27.494 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:27.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:27.595 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:16:27.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:27.630 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:27.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:16:27.731 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:27.762 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:27.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:27.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:27.864 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:16:27.865 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T08:16:27.898 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:27.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:27.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:16:27.999 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:28.047 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:28.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:28.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:28.149 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:16:28.174 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:16:28.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:28.179 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:28.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:16:28.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:28 smithi150 ceph-mon[29276]: pgmap v396: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 923 op/s; 2375/242547 objects misplaced (0.979%); 17 MiB/s, 274 objects/s recovering 2024-09-11T08:16:28.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:28 smithi150 ceph-mon[39612]: pgmap v396: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 923 op/s; 2375/242547 objects misplaced (0.979%); 17 MiB/s, 274 objects/s recovering 2024-09-11T08:16:28.280 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:16:28.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:28.317 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:28.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:16:28.418 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:28.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:28.452 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:28.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:28.553 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:16:28.573 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:16:28 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:16:28.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:28 smithi181 ceph-mon[32701]: pgmap v396: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 923 op/s; 2375/242547 objects misplaced (0.979%); 17 MiB/s, 274 objects/s recovering 2024-09-11T08:16:28.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:28.587 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:28.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:16:28.688 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:28.726 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:28.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:28.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:28.828 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:28.873 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:16:28.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:28.874 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:28.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:28.960 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:16:28 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:16:28] "GET /metrics HTTP/1.1" 200 224220 "" "Prometheus/2.43.0" 2024-09-11T08:16:28.975 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:29.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:29.020 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:29.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:29.112 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T08:16:29.122 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:29.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:29.214 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:29.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:29.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:29 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:16:29.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:29 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:16:29.316 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:29.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:29.374 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:29.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:29.475 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:16:29.508 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:29.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:29.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:16:29.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:29 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:16:29.610 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:16:29.645 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:29.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:29.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:16:29.747 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:29.786 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:29.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:29.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:29.888 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:29.921 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:29.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:29.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:30.023 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:30.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:30.053 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:30.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:30.154 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:16:30.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:30.195 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:30.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:16:30.296 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:16:30.329 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:30.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:30.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:16:30.431 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:30.477 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:30.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:30.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:30.486 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:30 smithi181 ceph-mon[32701]: pgmap v397: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.44k op/s; 853/227556 objects misplaced (0.375%); 26 MiB/s, 418 objects/s recovering 2024-09-11T08:16:30.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:30 smithi150 ceph-mon[29276]: pgmap v397: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.44k op/s; 853/227556 objects misplaced (0.375%); 26 MiB/s, 418 objects/s recovering 2024-09-11T08:16:30.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:30 smithi150 ceph-mon[39612]: pgmap v397: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.44k op/s; 853/227556 objects misplaced (0.375%); 26 MiB/s, 418 objects/s recovering 2024-09-11T08:16:30.579 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:30.600 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:30.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:30.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:30.702 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:30.738 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:30.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:30.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:30.839 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:30.883 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:30.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:30.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:30.985 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:31.018 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:31.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:31.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:31.120 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:31.163 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:31.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:31.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:31.265 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:16:31.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:31.336 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:31.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:16:31.437 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:31.452 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:31 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:16:31.452 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:31 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:16:31.452 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:31 smithi150 ceph-mon[29276]: osdmap e217: 8 total, 8 up, 8 in 2024-09-11T08:16:31.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:31 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:16:31.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:31 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:16:31.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:31 smithi150 ceph-mon[39612]: osdmap e217: 8 total, 8 up, 8 in 2024-09-11T08:16:31.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:31.481 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:31.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:31.569 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-11T08:16:31.570 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 20, 2024-09-11T08:16:31.570 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 600, 2024-09-11T08:16:31.570 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [ 2024-09-11T08:16:31.570 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.570 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:139331 7.b 7:dd362835:::benchmark_data_smithi150_78508_object14890:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-09-11T08:16:31.570 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:41.985463+0000", 2024-09-11T08:16:31.570 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 289.54047241400002, 2024-09-11T08:16:31.570 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.14882606800000001, 2024-09-11T08:16:31.570 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:16:31.570 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:16:31.570 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:16:31.571 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:16:31.571 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:16:31.571 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 139331 2024-09-11T08:16:31.571 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.571 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:16:31.571 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.572 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:16:31.572 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:41.985463+0000", 2024-09-11T08:16:31.572 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:16:31.572 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.572 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.572 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:16:31.572 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:41.985463+0000", 2024-09-11T08:16:31.572 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:16:31.573 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.573 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.573 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:16:31.573 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:41.985462+0000", 2024-09-11T08:16:31.573 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:16:31.573 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.573 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.573 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:16:31.573 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:41.985469+0000", 2024-09-11T08:16:31.574 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.0940000000000002e-06 2024-09-11T08:16:31.574 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.574 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.574 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:16:31.574 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:41.985484+0000", 2024-09-11T08:16:31.574 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.5404999999999999e-05 2024-09-11T08:16:31.574 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.574 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.574 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:16:31.574 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:41.985491+0000", 2024-09-11T08:16:31.575 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.6379999999999998e-06 2024-09-11T08:16:31.575 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.575 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.575 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:16:31.575 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:42.026460+0000", 2024-09-11T08:16:31.575 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.040969288 2024-09-11T08:16:31.575 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.575 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.575 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:16:31.575 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:42.026569+0000", 2024-09-11T08:16:31.576 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000108354 2024-09-11T08:16:31.576 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.576 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.576 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,6", 2024-09-11T08:16:31.576 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:42.026670+0000", 2024-09-11T08:16:31.576 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000101278 2024-09-11T08:16:31.576 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.576 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.576 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:16:31.576 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:42.134153+0000", 2024-09-11T08:16:31.576 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.10748302 2024-09-11T08:16:31.576 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.577 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.577 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:16:31.577 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:42.134160+0000", 2024-09-11T08:16:31.577 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.4000000000000003e-06 2024-09-11T08:16:31.577 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.577 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.577 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:16:31.577 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:42.134194+0000", 2024-09-11T08:16:31.577 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.3738999999999997e-05 2024-09-11T08:16:31.577 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.577 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.577 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:16:31.578 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:42.134244+0000", 2024-09-11T08:16:31.578 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.0364000000000001e-05 2024-09-11T08:16:31.578 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.578 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.578 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:16:31.578 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:42.134290+0000", 2024-09-11T08:16:31.578 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.5027000000000001e-05 2024-09-11T08:16:31.578 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:16:31.578 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:16:31.578 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:16:31.578 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.578 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.578 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:140147 7.b 7:d7db166a:::benchmark_data_smithi150_78508_object15706:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-11T08:16:31.578 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:42.853618+0000", 2024-09-11T08:16:31.578 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 288.67231766399999, 2024-09-11T08:16:31.579 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.13674162100000001, 2024-09-11T08:16:31.579 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:16:31.579 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:16:31.579 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:16:31.579 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:16:31.579 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:16:31.579 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 140147 2024-09-11T08:16:31.579 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.579 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:16:31.579 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.579 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:16:31.579 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:42.853618+0000", 2024-09-11T08:16:31.579 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:16:31.579 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.579 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.579 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:16:31.580 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:42.853618+0000", 2024-09-11T08:16:31.580 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:16:31.580 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.580 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.580 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:16:31.580 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:42.853616+0000", 2024-09-11T08:16:31.580 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:16:31.580 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.580 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.580 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:16:31.580 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:42.853623+0000", 2024-09-11T08:16:31.580 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.0369999999999997e-06 2024-09-11T08:16:31.580 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.580 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.581 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:16:31.581 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:42.853635+0000", 2024-09-11T08:16:31.581 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1864e-05 2024-09-11T08:16:31.581 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.581 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.581 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:16:31.581 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:42.853642+0000", 2024-09-11T08:16:31.581 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.3099999999999997e-06 2024-09-11T08:16:31.581 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.581 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.581 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:16:31.581 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:42.921192+0000", 2024-09-11T08:16:31.581 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.067550027999999998 2024-09-11T08:16:31.582 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.582 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.582 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:16:31.582 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:42.921297+0000", 2024-09-11T08:16:31.582 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000105146 2024-09-11T08:16:31.582 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.582 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.582 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,6", 2024-09-11T08:16:31.582 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:42.921417+0000", 2024-09-11T08:16:31.582 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000119739 2024-09-11T08:16:31.582 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.582 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.582 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:16:31.582 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:42.930351+0000", 2024-09-11T08:16:31.582 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0089349340000000003 2024-09-11T08:16:31.583 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.583 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.583 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:16:31.583 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:42.930378+0000", 2024-09-11T08:16:31.583 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.7041000000000001e-05 2024-09-11T08:16:31.583 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.583 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.583 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:16:31.583 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:42.990279+0000", 2024-09-11T08:16:31.583 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.059900025000000003 2024-09-11T08:16:31.583 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.583 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.584 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:16:31.584 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:42.990324+0000", 2024-09-11T08:16:31.584 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.5596000000000002e-05 2024-09-11T08:16:31.584 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.584 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.584 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:16:31.584 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:42.990360+0000", 2024-09-11T08:16:31.584 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.5709e-05 2024-09-11T08:16:31.584 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:16:31.584 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:16:31.584 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:16:31.584 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.584 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.584 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:142456 7.b 7:dc673c2d:::benchmark_data_smithi150_78508_object18015:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-11T08:16:31.585 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:45.043478+0000", 2024-09-11T08:16:31.585 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 286.482458108, 2024-09-11T08:16:31.585 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.16652083100000001, 2024-09-11T08:16:31.585 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:16:31.585 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:16:31.585 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:16:31.585 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:16:31.585 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:16:31.585 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 142456 2024-09-11T08:16:31.585 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.585 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:16:31.585 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.586 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:16:31.586 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.043478+0000", 2024-09-11T08:16:31.586 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:16:31.586 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.586 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.586 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:16:31.586 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.043478+0000", 2024-09-11T08:16:31.586 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:16:31.586 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.586 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.587 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:16:31.587 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.043476+0000", 2024-09-11T08:16:31.587 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:16:31.587 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.587 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.587 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:16:31.587 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.043483+0000", 2024-09-11T08:16:31.587 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.8920000000000003e-06 2024-09-11T08:16:31.587 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.588 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.588 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:16:31.588 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.043498+0000", 2024-09-11T08:16:31.588 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4486e-05 2024-09-11T08:16:31.588 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.588 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.588 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:16:31.588 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.043505+0000", 2024-09-11T08:16:31.588 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.9550000000000003e-06 2024-09-11T08:16:31.588 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.588 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.588 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:16:31.588 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.087948+0000", 2024-09-11T08:16:31.588 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.044443911000000003 2024-09-11T08:16:31.589 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.589 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.589 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:16:31.589 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.088060+0000", 2024-09-11T08:16:31.589 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000111759 2024-09-11T08:16:31.589 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.589 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.589 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,6", 2024-09-11T08:16:31.589 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.088171+0000", 2024-09-11T08:16:31.589 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00011094000000000001 2024-09-11T08:16:31.589 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.589 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.589 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:16:31.589 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.093696+0000", 2024-09-11T08:16:31.590 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0055247949999999999 2024-09-11T08:16:31.590 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.590 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.590 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:16:31.590 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.094800+0000", 2024-09-11T08:16:31.590 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.001104231 2024-09-11T08:16:31.590 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.590 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.590 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:16:31.590 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.209931+0000", 2024-09-11T08:16:31.590 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.11513032300000001 2024-09-11T08:16:31.590 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.590 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.590 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:16:31.591 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.209962+0000", 2024-09-11T08:16:31.591 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.1720999999999997e-05 2024-09-11T08:16:31.591 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.591 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.591 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:16:31.591 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.209999+0000", 2024-09-11T08:16:31.591 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.6347999999999998e-05 2024-09-11T08:16:31.591 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:16:31.591 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:16:31.591 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:16:31.591 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.591 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.591 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:142798 7.b 7:d133174b:::benchmark_data_smithi150_78508_object18357:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-11T08:16:31.591 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:45.427201+0000", 2024-09-11T08:16:31.592 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 286.09873469000001, 2024-09-11T08:16:31.592 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.158988447, 2024-09-11T08:16:31.592 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:16:31.592 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:16:31.592 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:16:31.592 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:16:31.592 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:16:31.592 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 142798 2024-09-11T08:16:31.592 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.592 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:16:31.592 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.592 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:16:31.592 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.427201+0000", 2024-09-11T08:16:31.593 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:16:31.593 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.593 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.593 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:16:31.593 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.427201+0000", 2024-09-11T08:16:31.593 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:16:31.593 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.593 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.593 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:16:31.593 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.427200+0000", 2024-09-11T08:16:31.593 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:16:31.593 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.593 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.593 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:16:31.593 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.427205+0000", 2024-09-11T08:16:31.594 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.809e-06 2024-09-11T08:16:31.594 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.594 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.594 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:16:31.594 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.427215+0000", 2024-09-11T08:16:31.594 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.3489999999999998e-06 2024-09-11T08:16:31.594 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.594 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.594 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:16:31.594 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.427220+0000", 2024-09-11T08:16:31.594 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.2730000000000002e-06 2024-09-11T08:16:31.595 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.595 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.595 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:16:31.595 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.433203+0000", 2024-09-11T08:16:31.595 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0059829389999999996 2024-09-11T08:16:31.595 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.595 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.595 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:16:31.595 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.433328+0000", 2024-09-11T08:16:31.595 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012522 2024-09-11T08:16:31.595 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.596 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.596 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,6", 2024-09-11T08:16:31.596 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.433467+0000", 2024-09-11T08:16:31.596 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00013929900000000001 2024-09-11T08:16:31.596 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.596 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.596 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:16:31.596 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.436705+0000", 2024-09-11T08:16:31.596 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0032379349999999999 2024-09-11T08:16:31.596 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.596 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.596 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:16:31.596 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.585196+0000", 2024-09-11T08:16:31.597 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.148491134 2024-09-11T08:16:31.597 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.597 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.597 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:16:31.597 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.586104+0000", 2024-09-11T08:16:31.597 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00090806800000000005 2024-09-11T08:16:31.597 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.597 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.597 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:16:31.597 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.586153+0000", 2024-09-11T08:16:31.597 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.8063999999999999e-05 2024-09-11T08:16:31.598 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.598 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.598 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:16:31.598 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.586190+0000", 2024-09-11T08:16:31.598 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.7064000000000003e-05 2024-09-11T08:16:31.598 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:16:31.598 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:16:31.598 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:16:31.598 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.598 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.598 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop_reply(client.14790.0:142798 7.b e129/91 ondisk, result = 0)", 2024-09-11T08:16:31.598 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:45.436863+0000", 2024-09-11T08:16:31.598 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 286.08907324900002, 2024-09-11T08:16:31.599 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.148335468, 2024-09-11T08:16:31.599 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:16:31.599 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "started", 2024-09-11T08:16:31.599 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:16:31.599 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.599 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:16:31.599 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.436863+0000", 2024-09-11T08:16:31.599 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:16:31.599 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.599 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.599 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:16:31.599 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.436863+0000", 2024-09-11T08:16:31.599 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:16:31.600 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.600 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.600 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:16:31.600 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.436862+0000", 2024-09-11T08:16:31.600 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:16:31.600 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.600 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.600 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:16:31.600 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.436866+0000", 2024-09-11T08:16:31.600 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.352e-06 2024-09-11T08:16:31.600 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.600 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.600 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:16:31.601 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.436875+0000", 2024-09-11T08:16:31.601 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.2579999999999995e-06 2024-09-11T08:16:31.601 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.601 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.601 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:16:31.601 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.436880+0000", 2024-09-11T08:16:31.601 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.1019999999999996e-06 2024-09-11T08:16:31.601 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.601 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.601 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:16:31.601 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.585184+0000", 2024-09-11T08:16:31.601 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.14830451 2024-09-11T08:16:31.602 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.602 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.602 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:16:31.602 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.585191+0000", 2024-09-11T08:16:31.602 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.2700000000000001e-06 2024-09-11T08:16:31.602 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.602 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.602 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:16:31.602 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.585198+0000", 2024-09-11T08:16:31.602 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.5789999999999998e-06 2024-09-11T08:16:31.602 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:16:31.602 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:16:31.602 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:16:31.602 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.603 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.603 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop_reply(osd.1.0:10057 7.b e129/91 ondisk, result = 0)", 2024-09-11T08:16:31.603 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:45.441449+0000", 2024-09-11T08:16:31.603 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 286.08448647799997, 2024-09-11T08:16:31.603 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.144143985, 2024-09-11T08:16:31.603 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:16:31.603 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "started", 2024-09-11T08:16:31.603 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:16:31.603 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.603 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:16:31.603 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.441449+0000", 2024-09-11T08:16:31.603 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:16:31.603 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.603 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.604 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:16:31.604 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.441449+0000", 2024-09-11T08:16:31.604 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:16:31.604 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.604 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.604 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:16:31.604 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.441449+0000", 2024-09-11T08:16:31.604 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:16:31.604 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.604 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.604 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:16:31.604 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.441454+0000", 2024-09-11T08:16:31.604 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.0089999999999998e-06 2024-09-11T08:16:31.605 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.605 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.605 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:16:31.605 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.441460+0000", 2024-09-11T08:16:31.605 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.122e-06 2024-09-11T08:16:31.605 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.605 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.605 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:16:31.605 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.441465+0000", 2024-09-11T08:16:31.605 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.3480000000000003e-06 2024-09-11T08:16:31.605 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.605 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.605 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:16:31.606 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.585579+0000", 2024-09-11T08:16:31.606 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.144113722 2024-09-11T08:16:31.606 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.606 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.606 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:16:31.606 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.585586+0000", 2024-09-11T08:16:31.606 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.0119999999999999e-06 2024-09-11T08:16:31.606 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.606 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.606 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:16:31.606 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.585593+0000", 2024-09-11T08:16:31.606 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.4669999999999998e-06 2024-09-11T08:16:31.606 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:16:31.607 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:16:31.607 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:16:31.607 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.607 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.607 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop_reply(osd.1.0:10057 7.b e129/91 ondisk, result = 0)", 2024-09-11T08:16:31.607 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:45.442198+0000", 2024-09-11T08:16:31.607 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 286.08373832799998, 2024-09-11T08:16:31.607 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.14342907599999999, 2024-09-11T08:16:31.607 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:16:31.607 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "started", 2024-09-11T08:16:31.607 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:16:31.607 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.607 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:16:31.607 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.442198+0000", 2024-09-11T08:16:31.608 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:16:31.608 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.608 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.608 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:16:31.608 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.442198+0000", 2024-09-11T08:16:31.608 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:16:31.608 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.608 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.608 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:16:31.608 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.442197+0000", 2024-09-11T08:16:31.608 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:16:31.608 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.608 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.608 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:16:31.608 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.442202+0000", 2024-09-11T08:16:31.609 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.2519999999999999e-06 2024-09-11T08:16:31.609 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.609 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.609 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:16:31.609 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.442213+0000", 2024-09-11T08:16:31.609 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0423e-05 2024-09-11T08:16:31.609 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.609 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.609 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:16:31.609 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.442218+0000", 2024-09-11T08:16:31.609 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.0900000000000004e-06 2024-09-11T08:16:31.609 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.609 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.609 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:16:31.610 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.585611+0000", 2024-09-11T08:16:31.610 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.14339310799999999 2024-09-11T08:16:31.610 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.610 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.610 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:16:31.610 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.585617+0000", 2024-09-11T08:16:31.610 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.8860000000000001e-06 2024-09-11T08:16:31.610 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.610 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.610 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:16:31.610 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.585627+0000", 2024-09-11T08:16:31.610 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.961e-06 2024-09-11T08:16:31.610 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:16:31.610 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:16:31.610 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:16:31.611 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.611 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.611 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop_reply(osd.1.0:10058 7.b e129/91 ondisk, result = 0)", 2024-09-11T08:16:31.611 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:45.443862+0000", 2024-09-11T08:16:31.611 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 286.08207362600001, 2024-09-11T08:16:31.611 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.14183115099999999, 2024-09-11T08:16:31.611 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:16:31.611 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "started", 2024-09-11T08:16:31.611 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:16:31.611 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.611 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:16:31.611 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.443862+0000", 2024-09-11T08:16:31.611 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:16:31.611 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.611 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.611 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:16:31.612 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.443862+0000", 2024-09-11T08:16:31.612 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:16:31.612 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.612 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.612 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:16:31.612 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.443861+0000", 2024-09-11T08:16:31.612 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:16:31.612 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.612 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.612 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:16:31.612 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.443866+0000", 2024-09-11T08:16:31.612 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.8459999999999999e-06 2024-09-11T08:16:31.612 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.612 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.613 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:16:31.613 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.443876+0000", 2024-09-11T08:16:31.613 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.3309999999999993e-06 2024-09-11T08:16:31.613 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.613 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.613 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:16:31.613 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.443881+0000", 2024-09-11T08:16:31.613 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.994e-06 2024-09-11T08:16:31.613 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.613 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.613 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:16:31.613 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.585680+0000", 2024-09-11T08:16:31.613 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.141799698 2024-09-11T08:16:31.613 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.613 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.614 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:16:31.614 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.585686+0000", 2024-09-11T08:16:31.614 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.5369999999999997e-06 2024-09-11T08:16:31.614 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.614 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.614 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:16:31.614 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.585693+0000", 2024-09-11T08:16:31.614 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.4970000000000004e-06 2024-09-11T08:16:31.614 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:16:31.614 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:16:31.614 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:16:31.614 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.614 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.614 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:142809 7.b 7:d8633ebc:::benchmark_data_smithi150_78508_object18368:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-11T08:16:31.614 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:45.445577+0000", 2024-09-11T08:16:31.614 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 286.08035867199999, 2024-09-11T08:16:31.615 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.15410784599999999, 2024-09-11T08:16:31.615 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:16:31.615 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:16:31.615 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:16:31.615 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:16:31.615 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:16:31.615 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 142809 2024-09-11T08:16:31.615 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.615 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:16:31.615 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.615 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:16:31.615 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.445577+0000", 2024-09-11T08:16:31.615 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:16:31.615 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.615 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.615 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:16:31.616 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.445577+0000", 2024-09-11T08:16:31.616 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:16:31.616 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.616 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.616 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:16:31.616 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.445575+0000", 2024-09-11T08:16:31.616 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:16:31.616 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.616 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.616 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:16:31.616 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.445583+0000", 2024-09-11T08:16:31.616 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.0360000000000006e-06 2024-09-11T08:16:31.616 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.616 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.616 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:16:31.617 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.445596+0000", 2024-09-11T08:16:31.617 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2255e-05 2024-09-11T08:16:31.617 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.617 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.617 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:16:31.617 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.445602+0000", 2024-09-11T08:16:31.617 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.3040000000000001e-06 2024-09-11T08:16:31.617 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.617 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.617 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:16:31.617 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.588455+0000", 2024-09-11T08:16:31.617 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.142852588 2024-09-11T08:16:31.618 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.618 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.618 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:16:31.618 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.588760+0000", 2024-09-11T08:16:31.618 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00030525000000000002 2024-09-11T08:16:31.618 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.618 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.618 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,6", 2024-09-11T08:16:31.618 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.588938+0000", 2024-09-11T08:16:31.618 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00017793600000000001 2024-09-11T08:16:31.618 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.618 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.619 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:16:31.619 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.598547+0000", 2024-09-11T08:16:31.619 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0096096610000000002 2024-09-11T08:16:31.619 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.619 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.619 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:16:31.619 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.598576+0000", 2024-09-11T08:16:31.619 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.8895999999999999e-05 2024-09-11T08:16:31.619 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.619 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.619 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:16:31.619 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.599586+0000", 2024-09-11T08:16:31.619 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0010097470000000001 2024-09-11T08:16:31.620 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.620 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.620 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:16:31.620 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.599642+0000", 2024-09-11T08:16:31.620 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.5642e-05 2024-09-11T08:16:31.620 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.620 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.620 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:16:31.620 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.599685+0000", 2024-09-11T08:16:31.620 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.3266999999999998e-05 2024-09-11T08:16:31.620 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:16:31.620 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:16:31.620 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:16:31.620 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.620 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.621 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop_reply(osd.1.0:10058 7.b e129/91 ondisk, result = 0)", 2024-09-11T08:16:31.621 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:45.446740+0000", 2024-09-11T08:16:31.621 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 286.07919605400002, 2024-09-11T08:16:31.621 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.13898244200000001, 2024-09-11T08:16:31.621 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:16:31.621 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "started", 2024-09-11T08:16:31.621 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:16:31.621 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.621 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:16:31.621 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.446740+0000", 2024-09-11T08:16:31.621 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:16:31.621 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.621 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.621 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:16:31.622 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.446740+0000", 2024-09-11T08:16:31.622 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:16:31.622 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.622 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.622 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:16:31.622 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.446739+0000", 2024-09-11T08:16:31.622 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:16:31.622 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.622 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.622 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:16:31.622 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.446742+0000", 2024-09-11T08:16:31.622 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.8430000000000001e-06 2024-09-11T08:16:31.622 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.622 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.622 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:16:31.622 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.446748+0000", 2024-09-11T08:16:31.623 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.9900000000000002e-06 2024-09-11T08:16:31.623 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.623 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.623 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:16:31.623 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.446752+0000", 2024-09-11T08:16:31.623 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.2760000000000001e-06 2024-09-11T08:16:31.623 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.623 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.623 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:16:31.623 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.585711+0000", 2024-09-11T08:16:31.623 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.138959422 2024-09-11T08:16:31.623 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.624 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.624 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:16:31.624 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.585716+0000", 2024-09-11T08:16:31.624 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.2089999999999996e-06 2024-09-11T08:16:31.624 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.624 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.624 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:16:31.624 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.585722+0000", 2024-09-11T08:16:31.624 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.1020000000000002e-06 2024-09-11T08:16:31.624 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:16:31.625 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:16:31.625 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:16:31.625 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.625 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.625 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop_reply(osd.1.0:10059 7.b e129/91 ondisk, result = 0)", 2024-09-11T08:16:31.625 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:45.446757+0000", 2024-09-11T08:16:31.625 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 286.079178609, 2024-09-11T08:16:31.625 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.13902146800000001, 2024-09-11T08:16:31.625 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:16:31.625 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "started", 2024-09-11T08:16:31.625 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:16:31.625 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.625 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:16:31.626 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.446757+0000", 2024-09-11T08:16:31.626 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:16:31.626 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.626 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.626 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:16:31.626 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.446757+0000", 2024-09-11T08:16:31.626 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:16:31.626 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.626 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.626 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:16:31.626 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.446757+0000", 2024-09-11T08:16:31.626 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967296 2024-09-11T08:16:31.626 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.626 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.626 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:16:31.627 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.446759+0000", 2024-09-11T08:16:31.627 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.216e-06 2024-09-11T08:16:31.627 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.627 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.627 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:16:31.627 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.446763+0000", 2024-09-11T08:16:31.627 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.0249999999999996e-06 2024-09-11T08:16:31.627 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.627 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.627 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:16:31.627 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.446766+0000", 2024-09-11T08:16:31.627 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.7439999999999999e-06 2024-09-11T08:16:31.627 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.627 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.628 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:16:31.628 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.585768+0000", 2024-09-11T08:16:31.628 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.13900195800000001 2024-09-11T08:16:31.628 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.628 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.628 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:16:31.628 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.585773+0000", 2024-09-11T08:16:31.628 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.147e-06 2024-09-11T08:16:31.628 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.628 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.628 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:16:31.628 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.585779+0000", 2024-09-11T08:16:31.628 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.5480000000000001e-06 2024-09-11T08:16:31.628 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:16:31.628 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:16:31.629 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:16:31.629 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.629 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.629 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop_reply(osd.1.0:10059 7.b e129/91 ondisk, result = 0)", 2024-09-11T08:16:31.629 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:45.446809+0000", 2024-09-11T08:16:31.629 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 286.07912649000002, 2024-09-11T08:16:31.629 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.13894177599999999, 2024-09-11T08:16:31.629 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:16:31.629 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "started", 2024-09-11T08:16:31.629 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:16:31.629 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.629 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:16:31.629 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.446809+0000", 2024-09-11T08:16:31.629 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:16:31.629 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.630 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.630 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:16:31.630 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.446809+0000", 2024-09-11T08:16:31.630 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:16:31.630 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.630 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.630 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:16:31.630 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.446809+0000", 2024-09-11T08:16:31.630 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:16:31.630 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.630 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.630 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:16:31.630 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.446813+0000", 2024-09-11T08:16:31.631 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.428e-06 2024-09-11T08:16:31.631 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.631 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.631 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:16:31.631 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.446819+0000", 2024-09-11T08:16:31.631 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.2559999999999999e-06 2024-09-11T08:16:31.631 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.631 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.631 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:16:31.631 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.446822+0000", 2024-09-11T08:16:31.631 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.5710000000000002e-06 2024-09-11T08:16:31.631 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.631 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.632 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:16:31.632 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.585740+0000", 2024-09-11T08:16:31.632 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.13891720799999999 2024-09-11T08:16:31.632 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.632 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.632 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:16:31.632 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.585745+0000", 2024-09-11T08:16:31.632 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.3820000000000003e-06 2024-09-11T08:16:31.632 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.632 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.632 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:16:31.632 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.585751+0000", 2024-09-11T08:16:31.632 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.2249999999999997e-06 2024-09-11T08:16:31.632 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:16:31.633 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:16:31.633 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:16:31.633 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.633 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.633 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:142813 7.b 7:d5c9fa5d:::benchmark_data_smithi150_78508_object18372:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-11T08:16:31.633 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:45.449052+0000", 2024-09-11T08:16:31.633 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 286.07688357299998, 2024-09-11T08:16:31.633 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.15070741700000001, 2024-09-11T08:16:31.633 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:16:31.633 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:16:31.633 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:16:31.633 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:16:31.633 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:16:31.634 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 142813 2024-09-11T08:16:31.634 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.634 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:16:31.634 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.634 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:16:31.634 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.449052+0000", 2024-09-11T08:16:31.634 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:16:31.634 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.634 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.634 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:16:31.634 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.449052+0000", 2024-09-11T08:16:31.634 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:16:31.634 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.634 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.635 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:16:31.635 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.449051+0000", 2024-09-11T08:16:31.635 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:16:31.635 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.635 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.635 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:16:31.635 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.449057+0000", 2024-09-11T08:16:31.635 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.9419999999999998e-06 2024-09-11T08:16:31.635 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.635 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.635 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:16:31.635 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.449070+0000", 2024-09-11T08:16:31.635 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2302e-05 2024-09-11T08:16:31.636 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.636 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.636 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:16:31.636 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.449075+0000", 2024-09-11T08:16:31.636 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.6289999999999998e-06 2024-09-11T08:16:31.636 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.636 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.636 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:16:31.636 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.593478+0000", 2024-09-11T08:16:31.636 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.14440255099999999 2024-09-11T08:16:31.636 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.636 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.636 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:16:31.637 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.593585+0000", 2024-09-11T08:16:31.637 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000106914 2024-09-11T08:16:31.637 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.637 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.637 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,6", 2024-09-11T08:16:31.637 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.593690+0000", 2024-09-11T08:16:31.637 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000105011 2024-09-11T08:16:31.637 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.637 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.637 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:16:31.638 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.598602+0000", 2024-09-11T08:16:31.638 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0049125799999999997 2024-09-11T08:16:31.638 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.638 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.638 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:16:31.638 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.598902+0000", 2024-09-11T08:16:31.638 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000299516 2024-09-11T08:16:31.638 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.638 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.638 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:16:31.638 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.599694+0000", 2024-09-11T08:16:31.638 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00079222500000000003 2024-09-11T08:16:31.638 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.639 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.639 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:16:31.639 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.599725+0000", 2024-09-11T08:16:31.639 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.0981e-05 2024-09-11T08:16:31.639 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.639 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.639 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:16:31.639 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.599760+0000", 2024-09-11T08:16:31.639 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.4529000000000001e-05 2024-09-11T08:16:31.639 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:16:31.639 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:16:31.639 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:16:31.640 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.640 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.640 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop_reply(osd.1.0:10060 7.b e129/91 ondisk, result = 0)", 2024-09-11T08:16:31.640 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:45.454567+0000", 2024-09-11T08:16:31.640 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 286.071368452, 2024-09-11T08:16:31.640 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.13127867600000001, 2024-09-11T08:16:31.640 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:16:31.640 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "started", 2024-09-11T08:16:31.640 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:16:31.640 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.640 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:16:31.640 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.454567+0000", 2024-09-11T08:16:31.640 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:16:31.641 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.641 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.641 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:16:31.641 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.454567+0000", 2024-09-11T08:16:31.641 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:16:31.641 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.641 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.641 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:16:31.641 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.454567+0000", 2024-09-11T08:16:31.641 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:16:31.641 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.641 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.642 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:16:31.642 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.454574+0000", 2024-09-11T08:16:31.642 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.6860000000000006e-06 2024-09-11T08:16:31.642 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.642 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.642 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:16:31.642 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.454583+0000", 2024-09-11T08:16:31.642 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.6719999999999993e-06 2024-09-11T08:16:31.642 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.642 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.642 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:16:31.642 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.454588+0000", 2024-09-11T08:16:31.642 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.9579999999999998e-06 2024-09-11T08:16:31.642 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.643 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.643 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:16:31.643 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.585834+0000", 2024-09-11T08:16:31.643 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.13124561800000001 2024-09-11T08:16:31.643 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.643 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.643 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:16:31.643 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.585839+0000", 2024-09-11T08:16:31.643 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.7780000000000004e-06 2024-09-11T08:16:31.643 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.643 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.643 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:16:31.644 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.585846+0000", 2024-09-11T08:16:31.644 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.6619999999999999e-06 2024-09-11T08:16:31.644 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:16:31.644 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:16:31.644 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:16:31.644 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.644 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.644 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop_reply(osd.1.0:10060 7.b e129/91 ondisk, result = 0)", 2024-09-11T08:16:31.644 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:45.456042+0000", 2024-09-11T08:16:31.644 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 286.06989371899999, 2024-09-11T08:16:31.644 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.129938254, 2024-09-11T08:16:31.644 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:16:31.644 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "started", 2024-09-11T08:16:31.644 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:16:31.645 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.645 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:16:31.645 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.456042+0000", 2024-09-11T08:16:31.645 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:16:31.645 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.645 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.645 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:31.647 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:31 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:16:31.647 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:31 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:16:31.647 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:31 smithi181 ceph-mon[32701]: osdmap e217: 8 total, 8 up, 8 in 2024-09-11T08:16:31.647 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:16:31.648 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.456042+0000", 2024-09-11T08:16:31.648 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:16:31.648 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.648 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.648 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:16:31.648 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.456042+0000", 2024-09-11T08:16:31.648 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:16:31.648 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.648 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.648 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:16:31.648 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.456044+0000", 2024-09-11T08:16:31.649 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.2189999999999998e-06 2024-09-11T08:16:31.649 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.649 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.649 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:16:31.649 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.456048+0000", 2024-09-11T08:16:31.652 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.3789999999999999e-06 2024-09-11T08:16:31.652 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.652 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.652 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:16:31.652 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.456051+0000", 2024-09-11T08:16:31.652 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.932e-06 2024-09-11T08:16:31.653 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.653 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.653 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:16:31.653 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.585864+0000", 2024-09-11T08:16:31.653 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.12981280000000001 2024-09-11T08:16:31.653 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.653 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.653 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:16:31.653 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.585869+0000", 2024-09-11T08:16:31.653 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.2059999999999998e-06 2024-09-11T08:16:31.653 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.653 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.653 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:16:31.653 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.585980+0000", 2024-09-11T08:16:31.653 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000110967 2024-09-11T08:16:31.654 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:16:31.654 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:16:31.654 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:16:31.654 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.654 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.654 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop_reply(osd.1.0:10061 7.b e129/91 ondisk, result = 0)", 2024-09-11T08:16:31.654 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:45.456055+0000", 2024-09-11T08:16:31.654 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 286.06988082300001, 2024-09-11T08:16:31.654 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.130014084, 2024-09-11T08:16:31.654 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:16:31.654 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "started", 2024-09-11T08:16:31.654 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:16:31.654 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.654 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:16:31.654 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.456055+0000", 2024-09-11T08:16:31.654 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:16:31.655 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.655 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.655 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:16:31.655 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.456055+0000", 2024-09-11T08:16:31.655 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:16:31.655 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.655 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.655 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:16:31.655 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.456055+0000", 2024-09-11T08:16:31.655 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967296 2024-09-11T08:16:31.655 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.655 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.655 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:16:31.655 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.456057+0000", 2024-09-11T08:16:31.655 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.6479999999999999e-06 2024-09-11T08:16:31.656 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.656 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.656 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:16:31.656 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.456060+0000", 2024-09-11T08:16:31.656 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.777e-06 2024-09-11T08:16:31.656 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.656 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.656 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:16:31.656 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.456063+0000", 2024-09-11T08:16:31.656 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.6460000000000002e-06 2024-09-11T08:16:31.657 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.657 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.657 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:16:31.657 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.586054+0000", 2024-09-11T08:16:31.657 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.12999131899999999 2024-09-11T08:16:31.657 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.657 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.657 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:16:31.657 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.586060+0000", 2024-09-11T08:16:31.657 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.8189999999999997e-06 2024-09-11T08:16:31.657 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.657 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.658 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:16:31.658 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:45.586069+0000", 2024-09-11T08:16:31.658 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.02e-06 2024-09-11T08:16:31.658 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:16:31.658 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:16:31.658 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:16:31.658 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.658 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.658 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:145587 7.b 7:dad37def:::benchmark_data_smithi150_78508_object21146:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-11T08:16:31.658 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:48.236747+0000", 2024-09-11T08:16:31.658 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 283.28918893299999, 2024-09-11T08:16:31.658 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.16628657899999999, 2024-09-11T08:16:31.658 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:16:31.658 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:16:31.658 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:16:31.659 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:16:31.659 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:16:31.659 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 145587 2024-09-11T08:16:31.659 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.659 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:16:31.659 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.659 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:16:31.659 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:48.236747+0000", 2024-09-11T08:16:31.659 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:16:31.659 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.659 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.659 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:16:31.659 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:48.236747+0000", 2024-09-11T08:16:31.659 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:16:31.660 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.660 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.660 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:16:31.660 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:48.236745+0000", 2024-09-11T08:16:31.660 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:16:31.660 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.660 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.660 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:16:31.660 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:48.236752+0000", 2024-09-11T08:16:31.660 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.4529999999999999e-06 2024-09-11T08:16:31.660 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.660 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.660 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:16:31.661 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:48.236763+0000", 2024-09-11T08:16:31.661 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1209999999999999e-05 2024-09-11T08:16:31.661 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.661 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.661 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:16:31.661 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:48.236769+0000", 2024-09-11T08:16:31.661 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.4569999999999996e-06 2024-09-11T08:16:31.661 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.661 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.661 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:16:31.661 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:48.307120+0000", 2024-09-11T08:16:31.661 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.070351643000000005 2024-09-11T08:16:31.662 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.662 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.662 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:16:31.662 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:48.307220+0000", 2024-09-11T08:16:31.662 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.9939999999999995e-05 2024-09-11T08:16:31.662 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.662 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.662 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,6", 2024-09-11T08:16:31.662 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:48.307324+0000", 2024-09-11T08:16:31.662 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00010423 2024-09-11T08:16:31.662 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.662 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.663 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:16:31.663 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:48.312376+0000", 2024-09-11T08:16:31.663 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0050518389999999998 2024-09-11T08:16:31.663 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.663 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.663 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:16:31.663 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:48.328777+0000", 2024-09-11T08:16:31.663 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.016400956000000001 2024-09-11T08:16:31.663 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.663 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.663 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:16:31.663 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:48.402929+0000", 2024-09-11T08:16:31.663 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.074151686999999994 2024-09-11T08:16:31.663 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.663 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.664 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:16:31.664 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:48.402995+0000", 2024-09-11T08:16:31.664 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.5809999999999995e-05 2024-09-11T08:16:31.664 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.664 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.664 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:16:31.664 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:48.403034+0000", 2024-09-11T08:16:31.664 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.8803000000000001e-05 2024-09-11T08:16:31.664 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:16:31.664 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:16:31.664 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:16:31.664 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.664 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.664 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:146886 7.b 7:dfff1ecb:::benchmark_data_smithi150_78508_object22445:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-11T08:16:31.665 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:49.570818+0000", 2024-09-11T08:16:31.665 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 281.95511797099999, 2024-09-11T08:16:31.665 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.133407004, 2024-09-11T08:16:31.665 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:16:31.665 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:16:31.665 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:16:31.665 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:16:31.665 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:16:31.665 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 146886 2024-09-11T08:16:31.665 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.665 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:16:31.665 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.665 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:16:31.665 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:49.570818+0000", 2024-09-11T08:16:31.666 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:16:31.666 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.666 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.666 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:16:31.666 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:49.570818+0000", 2024-09-11T08:16:31.666 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:16:31.666 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.666 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.666 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:16:31.666 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:49.570816+0000", 2024-09-11T08:16:31.666 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:16:31.666 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.666 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.666 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:16:31.666 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:49.570824+0000", 2024-09-11T08:16:31.666 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.3259999999999998e-06 2024-09-11T08:16:31.667 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.667 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.667 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:16:31.667 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:49.570834+0000", 2024-09-11T08:16:31.667 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0654e-05 2024-09-11T08:16:31.667 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.667 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.667 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:16:31.667 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:49.570840+0000", 2024-09-11T08:16:31.667 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.3430000000000003e-06 2024-09-11T08:16:31.667 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.667 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.668 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:16:31.668 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:49.606964+0000", 2024-09-11T08:16:31.668 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.036124244999999999 2024-09-11T08:16:31.668 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.668 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.668 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:16:31.668 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:49.607065+0000", 2024-09-11T08:16:31.668 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00010132 2024-09-11T08:16:31.668 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.668 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.668 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,6", 2024-09-11T08:16:31.668 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:49.607169+0000", 2024-09-11T08:16:31.668 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00010412999999999999 2024-09-11T08:16:31.668 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.669 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.669 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:16:31.669 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:49.613504+0000", 2024-09-11T08:16:31.669 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0063347409999999996 2024-09-11T08:16:31.669 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.669 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.669 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:16:31.669 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:49.614985+0000", 2024-09-11T08:16:31.669 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.001481062 2024-09-11T08:16:31.669 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.669 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.669 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:16:31.670 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:49.704131+0000", 2024-09-11T08:16:31.670 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.089145514999999995 2024-09-11T08:16:31.670 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.670 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.670 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:16:31.670 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:49.704180+0000", 2024-09-11T08:16:31.670 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.9759999999999998e-05 2024-09-11T08:16:31.670 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.670 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.670 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:16:31.670 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:49.704225+0000", 2024-09-11T08:16:31.671 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.4498e-05 2024-09-11T08:16:31.671 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:16:31.671 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:16:31.671 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:16:31.671 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.671 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.671 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.24421.0:285 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e140)", 2024-09-11T08:16:31.671 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:12:27.939875+0000", 2024-09-11T08:16:31.671 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 243.586061094, 2024-09-11T08:16:31.671 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.56415223599999997, 2024-09-11T08:16:31.671 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:16:31.671 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "started", 2024-09-11T08:16:31.671 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:16:31.671 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.24421", 2024-09-11T08:16:31.672 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.181:0/3861404665", 2024-09-11T08:16:31.672 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 285 2024-09-11T08:16:31.672 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.672 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:16:31.672 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.672 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:16:31.672 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.939875+0000", 2024-09-11T08:16:31.672 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:16:31.672 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.672 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.672 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:16:31.673 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.939875+0000", 2024-09-11T08:16:31.673 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:16:31.673 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.673 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.673 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:16:31.673 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.939874+0000", 2024-09-11T08:16:31.673 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:16:31.673 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.673 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.673 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:16:31.673 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.939879+0000", 2024-09-11T08:16:31.673 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.8490000000000002e-06 2024-09-11T08:16:31.673 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.673 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.674 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:16:31.674 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.939887+0000", 2024-09-11T08:16:31.674 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.7749999999999993e-06 2024-09-11T08:16:31.674 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.674 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.674 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:16:31.674 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.939892+0000", 2024-09-11T08:16:31.674 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.8450000000000002e-06 2024-09-11T08:16:31.674 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.674 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.674 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:16:31.674 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.939939+0000", 2024-09-11T08:16:31.674 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.7431e-05 2024-09-11T08:16:31.675 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.675 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.675 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:16:31.675 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.939945+0000", 2024-09-11T08:16:31.675 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.3260000000000003e-06 2024-09-11T08:16:31.675 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.675 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.675 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:16:31.675 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.503775+0000", 2024-09-11T08:16:31.675 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.56383043899999996 2024-09-11T08:16:31.675 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.675 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.675 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:16:31.675 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.503905+0000", 2024-09-11T08:16:31.675 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012999499999999999 2024-09-11T08:16:31.676 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.676 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.676 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:16:31.676 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.504027+0000", 2024-09-11T08:16:31.676 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012184300000000001 2024-09-11T08:16:31.676 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:16:31.676 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:16:31.676 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:16:31.676 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.676 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.676 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15087.0:143226 8.7 8:e32946fa:::benchmark_data_smithi150_107646_object14394:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-11T08:16:31.676 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:15:52.683710+0000", 2024-09-11T08:16:31.676 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 38.842225929999998, 2024-09-11T08:16:31.676 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.098271644000000005, 2024-09-11T08:16:31.677 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:16:31.677 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:16:31.677 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:16:31.677 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15087", 2024-09-11T08:16:31.677 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/3299560913", 2024-09-11T08:16:31.677 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 143226 2024-09-11T08:16:31.677 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.677 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:16:31.677 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.677 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:16:31.677 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:52.683710+0000", 2024-09-11T08:16:31.677 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:16:31.677 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.677 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.677 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:16:31.678 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:52.683710+0000", 2024-09-11T08:16:31.678 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:16:31.678 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.678 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.678 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:16:31.678 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:52.683708+0000", 2024-09-11T08:16:31.678 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:16:31.678 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.678 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.678 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:16:31.678 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:52.683715+0000", 2024-09-11T08:16:31.678 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.652e-06 2024-09-11T08:16:31.678 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.679 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.679 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:16:31.679 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:52.683725+0000", 2024-09-11T08:16:31.679 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0353999999999999e-05 2024-09-11T08:16:31.679 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.679 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.679 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:16:31.679 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:52.683731+0000", 2024-09-11T08:16:31.679 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.1939999999999997e-06 2024-09-11T08:16:31.679 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.679 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.679 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:16:31.679 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:52.683777+0000", 2024-09-11T08:16:31.680 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.6146000000000002e-05 2024-09-11T08:16:31.680 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.680 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.680 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:16:31.680 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:52.683885+0000", 2024-09-11T08:16:31.680 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000107892 2024-09-11T08:16:31.680 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.680 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.680 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 3,6", 2024-09-11T08:16:31.680 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:52.684460+0000", 2024-09-11T08:16:31.680 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00057576600000000002 2024-09-11T08:16:31.680 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.680 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.680 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:16:31.681 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:52.690921+0000", 2024-09-11T08:16:31.681 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0064602460000000002 2024-09-11T08:16:31.681 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.681 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.681 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:16:31.681 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:52.693578+0000", 2024-09-11T08:16:31.681 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.002657319 2024-09-11T08:16:31.681 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.681 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.681 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:16:31.681 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:52.781901+0000", 2024-09-11T08:16:31.681 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.088322620000000004 2024-09-11T08:16:31.681 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.681 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.681 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:16:31.682 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:52.781948+0000", 2024-09-11T08:16:31.682 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.7923999999999997e-05 2024-09-11T08:16:31.682 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:16:31.682 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:16:31.682 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:16:31.682 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:52.781982+0000", 2024-09-11T08:16:31.682 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.3136999999999998e-05 2024-09-11T08:16:31.682 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:16:31.682 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:16:31.682 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:16:31.682 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:16:31.682 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:16:31.682 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-11T08:16:31.719 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:31.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:31.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:31.821 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:16:31.857 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:31.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:31.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:16:31.959 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:31.994 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:31.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:31.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:32.096 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:32.132 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:32.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:32.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:32.233 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:32.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:32.273 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:32.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:32.374 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:16:32.405 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:32.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:32.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:16:32.420 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:32 smithi150 ceph-mon[29276]: pgmap v399: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 752 B/s rd, 0 B/s wr, 735 op/s; 853/227556 objects misplaced (0.375%); 13 MiB/s, 210 objects/s recovering 2024-09-11T08:16:32.420 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:32 smithi150 ceph-mon[39612]: pgmap v399: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 752 B/s rd, 0 B/s wr, 735 op/s; 853/227556 objects misplaced (0.375%); 13 MiB/s, 210 objects/s recovering 2024-09-11T08:16:32.507 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:32.556 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:32.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:32.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:32.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:32 smithi181 ceph-mon[32701]: pgmap v399: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 752 B/s rd, 0 B/s wr, 735 op/s; 853/227556 objects misplaced (0.375%); 13 MiB/s, 210 objects/s recovering 2024-09-11T08:16:32.658 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:32.751 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:32.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:32.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:32.853 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:16:32.889 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:32.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:32.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:16:32.991 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:33.024 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:33.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:33.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:33.126 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:33.165 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:33.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:33.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:33.267 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:33.305 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:33.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:33.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:33.407 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:33.444 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:33.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:33.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:33.546 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:33.585 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:33.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:33.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:33.630 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-11T08:16:33.634 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:16:33.687 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:33.712 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:33.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:33.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:33.814 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:33.861 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:33.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:33.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:33.963 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:16:33.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:33.992 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:33.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:16:34.093 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:16:34.127 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:34.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:34.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:16:34.229 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:34.261 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:34.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:34.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:34.364 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:16:34.385 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:34.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:34.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:16:34.487 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:34.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:34 smithi150 ceph-mon[29276]: pgmap v400: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 4.6 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 758 op/s; 853/224367 objects misplaced (0.380%); 11 MiB/s, 178 objects/s recovering 2024-09-11T08:16:34.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:34 smithi150 ceph-mon[39612]: pgmap v400: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 4.6 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 758 op/s; 853/224367 objects misplaced (0.380%); 11 MiB/s, 178 objects/s recovering 2024-09-11T08:16:34.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:34.509 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:34.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:34.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:34 smithi181 ceph-mon[32701]: pgmap v400: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 4.6 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 758 op/s; 853/224367 objects misplaced (0.380%); 11 MiB/s, 178 objects/s recovering 2024-09-11T08:16:34.610 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:16:34.632 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:34.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:34.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:16:34.735 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:16:34.757 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:34.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:34.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:16:34.859 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:34.881 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:34.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:34.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:34.983 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:35.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:35.005 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:35.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:35.106 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:35.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:35.129 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:35.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:35.230 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:16:35.264 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:35.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:35.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:16:35.366 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:35.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:35.401 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:35.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:35.442 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:35 smithi150 ceph-mon[29276]: osdmap e218: 8 total, 8 up, 8 in 2024-09-11T08:16:35.442 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:35 smithi150 ceph-mon[39612]: osdmap e218: 8 total, 8 up, 8 in 2024-09-11T08:16:35.502 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:35.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:35.545 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:35.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:35.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:35 smithi181 ceph-mon[32701]: osdmap e218: 8 total, 8 up, 8 in 2024-09-11T08:16:35.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:16:35 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:16:35] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:16:35.646 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:16:35.673 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:35.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:35.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:16:35.775 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:16:35.801 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:35.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:35.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:16:35.903 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:16:35.931 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:35.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:35.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:16:36.033 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:36.074 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:36.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:36.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:36.176 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:36.203 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:36.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:36.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:36.260 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:16:36.260 INFO:teuthology.orchestra.run.smithi150.stdout:{"fsid":"c1bddff4-7013-11ef-bcea-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":759,"monmap":{"epoch":3,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":219,"num_osds":8,"num_up_osds":8,"osd_up_since":1726042463,"num_in_osds":8,"osd_in_since":1726042020,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":147},{"state_name":"active+remapped+backfilling","count":1}],"num_pgs":148,"num_pools":7,"num_objects":69959,"data_bytes":4572518644,"bytes_used":17020743680,"bytes_avail":750939541504,"bytes_total":767960285184,"misplaced_objects":55,"misplaced_total":209877,"misplaced_ratio":0.00026205825316733135,"recovering_objects_per_sec":274,"recovering_bytes_per_sec":17966145,"recovering_keys_per_sec":0,"num_objects_recovered":2194,"num_bytes_recovered":143785984,"num_keys_recovered":0,"read_bytes_sec":1279,"write_bytes_sec":0,"read_op_per_sec":1,"write_op_per_sec":1360},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.150:8443/","prometheus":"http://172.21.15.150:9283/"}},"servicemap":{"epoch":15,"modified":"2024-09-11T08:15:22.852838+0000","services":{"rgw":{"daemons":{"summary":"","14451":{"start_epoch":12,"start_stamp":"2024-09-11T08:07:37.191516+0000","gid":14451,"addr":"172.21.15.150:0/3411048405","metadata":{"arch":"x86_64","ceph_release":"quincy","ceph_version":"ceph version 17.2.7-1623-gd0e6828a (d0e6828a2016d48cf25ad84064e50742bb1c39b9) quincy (stable)","ceph_version_short":"17.2.7-1623-gd0e6828a","container_hostname":"smithi150","container_image":"quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40","cpu":"Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz","distro":"centos","distro_description":"CentOS Stream 9","distro_version":"9","frontend_config#0":"beast port=80","frontend_type#0":"beast","hostname":"smithi150","id":"foo.a","kernel_description":"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024","kernel_version":"5.14.0-505.el9.x86_64","mem_swap_kb":"0","mem_total_kb":"32487300","num_handles":"1","os":"Linux","pid":"2","realm_id":"","realm_name":"","zone_id":"faa33b33-895b-44d2-8ef5-97d186d0231e","zone_name":"default","zonegroup_id":"247f1a8f-1665-4228-87e1-08d663c7ff54","zonegroup_name":"default"},"task_status":{}}}}}},"progress_events":{}} 2024-09-11T08:16:36.305 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:36.342 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:36.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:36.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:36.347 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:36 smithi150 ceph-mon[39612]: pgmap v402: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.36k op/s; 55/209877 objects misplaced (0.026%); 17 MiB/s, 274 objects/s recovering 2024-09-11T08:16:36.347 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:36 smithi150 ceph-mon[39612]: osdmap e219: 8 total, 8 up, 8 in 2024-09-11T08:16:36.347 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:36 smithi150 ceph-mon[29276]: pgmap v402: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.36k op/s; 55/209877 objects misplaced (0.026%); 17 MiB/s, 274 objects/s recovering 2024-09-11T08:16:36.347 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:36 smithi150 ceph-mon[29276]: osdmap e219: 8 total, 8 up, 8 in 2024-09-11T08:16:36.445 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:36.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:36.472 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:36.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:36.573 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:36.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:36 smithi181 ceph-mon[32701]: pgmap v402: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.36k op/s; 55/209877 objects misplaced (0.026%); 17 MiB/s, 274 objects/s recovering 2024-09-11T08:16:36.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:36 smithi181 ceph-mon[32701]: osdmap e219: 8 total, 8 up, 8 in 2024-09-11T08:16:36.617 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:36.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:36.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:36.720 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:16:36.752 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:36.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:36.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:16:36.854 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:36.889 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:36.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:36.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:36.992 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:16:37.019 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:37.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:37.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:16:37.121 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:16:37.132 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:16:37.147 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:37.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:37.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:16:37.187 INFO:teuthology.orchestra.run.smithi150.stderr:noscrub is unset 2024-09-11T08:16:37.249 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:37.275 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:37.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:37.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:37.377 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:37.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:37.414 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:37.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:37.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:37 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/123556414' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-09-11T08:16:37.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:37 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2174193956' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:16:37.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:37 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:16:37.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:37 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:16:37.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:37 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/123556414' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-09-11T08:16:37.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:37 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2174193956' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:16:37.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:37 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:16:37.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:37 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:16:37.515 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:37.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:37.548 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:37.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:37.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:37 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/123556414' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-09-11T08:16:37.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:37 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2174193956' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:16:37.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:37 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:16:37.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:37 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:16:37.650 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:16:37.681 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:37.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:37.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:16:37.741 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-11T08:16:37.742 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T08:16:37.783 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:37.807 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:37.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:37.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:37.909 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:16:37.935 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:37.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:37.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:16:38.037 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:38.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:38.083 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:38.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:38.184 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:38.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:38.225 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:38.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:38.326 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:38.357 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:38.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:38.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:38.444 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:38 smithi150 ceph-mon[29276]: pgmap v404: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 757 B/s rd, 0 B/s wr, 872 op/s; 55/209877 objects misplaced (0.026%); 7.0 MiB/s, 112 objects/s recovering 2024-09-11T08:16:38.444 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:38 smithi150 ceph-mon[29276]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:16:38.444 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:38 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:16:38.444 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:38 smithi150 ceph-mon[29276]: osdmap e220: 8 total, 8 up, 8 in 2024-09-11T08:16:38.445 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:38 smithi150 ceph-mon[29276]: 3.1f scrub starts 2024-09-11T08:16:38.445 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:38 smithi150 ceph-mon[29276]: 3.1f scrub ok 2024-09-11T08:16:38.445 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:38 smithi150 ceph-mon[39612]: pgmap v404: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 757 B/s rd, 0 B/s wr, 872 op/s; 55/209877 objects misplaced (0.026%); 7.0 MiB/s, 112 objects/s recovering 2024-09-11T08:16:38.445 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:38 smithi150 ceph-mon[39612]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:16:38.445 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:38 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:16:38.445 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:38 smithi150 ceph-mon[39612]: osdmap e220: 8 total, 8 up, 8 in 2024-09-11T08:16:38.445 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:38 smithi150 ceph-mon[39612]: 3.1f scrub starts 2024-09-11T08:16:38.445 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:38 smithi150 ceph-mon[39612]: 3.1f scrub ok 2024-09-11T08:16:38.459 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:38.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:38.487 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:38.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:38.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:38 smithi181 ceph-mon[32701]: pgmap v404: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 757 B/s rd, 0 B/s wr, 872 op/s; 55/209877 objects misplaced (0.026%); 7.0 MiB/s, 112 objects/s recovering 2024-09-11T08:16:38.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:38 smithi181 ceph-mon[32701]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:16:38.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:38 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:16:38.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:38 smithi181 ceph-mon[32701]: osdmap e220: 8 total, 8 up, 8 in 2024-09-11T08:16:38.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:38 smithi181 ceph-mon[32701]: 3.1f scrub starts 2024-09-11T08:16:38.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:38 smithi181 ceph-mon[32701]: 3.1f scrub ok 2024-09-11T08:16:38.574 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:16:38 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:16:38.588 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:38.627 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:38.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:38.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:38.729 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:16:38.738 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:16:38 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:16:38] "GET /metrics HTTP/1.1" 200 224237 "" "Prometheus/2.43.0" 2024-09-11T08:16:38.760 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:38.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:38.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:16:38.862 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:38.897 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:38.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:38.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:38.999 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:39.036 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:39.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:39.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:39.138 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:16:39.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:39.168 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:39.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:16:39.212 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:39 smithi150 ceph-mon[29276]: 5.7 scrub starts 2024-09-11T08:16:39.212 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:39 smithi150 ceph-mon[29276]: 5.7 scrub ok 2024-09-11T08:16:39.212 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:39 smithi150 ceph-mon[29276]: 5.8 scrub starts 2024-09-11T08:16:39.213 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:39 smithi150 ceph-mon[29276]: 5.8 scrub ok 2024-09-11T08:16:39.213 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:39 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:16:39.213 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:39 smithi150 ceph-mon[29276]: 5.10 scrub starts 2024-09-11T08:16:39.213 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:39 smithi150 ceph-mon[29276]: 5.10 scrub ok 2024-09-11T08:16:39.213 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:39 smithi150 ceph-mon[39612]: 5.7 scrub starts 2024-09-11T08:16:39.213 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:39 smithi150 ceph-mon[39612]: 5.7 scrub ok 2024-09-11T08:16:39.213 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:39 smithi150 ceph-mon[39612]: 5.8 scrub starts 2024-09-11T08:16:39.214 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:39 smithi150 ceph-mon[39612]: 5.8 scrub ok 2024-09-11T08:16:39.214 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:39 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:16:39.214 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:39 smithi150 ceph-mon[39612]: 5.10 scrub starts 2024-09-11T08:16:39.214 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:39 smithi150 ceph-mon[39612]: 5.10 scrub ok 2024-09-11T08:16:39.269 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:39.314 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:39.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:39.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:39.416 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:39.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:39.453 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:39.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:39.554 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:39.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:39 smithi181 ceph-mon[32701]: 5.7 scrub starts 2024-09-11T08:16:39.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:39 smithi181 ceph-mon[32701]: 5.7 scrub ok 2024-09-11T08:16:39.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:39 smithi181 ceph-mon[32701]: 5.8 scrub starts 2024-09-11T08:16:39.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:39 smithi181 ceph-mon[32701]: 5.8 scrub ok 2024-09-11T08:16:39.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:39 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:16:39.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:39 smithi181 ceph-mon[32701]: 5.10 scrub starts 2024-09-11T08:16:39.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:39 smithi181 ceph-mon[32701]: 5.10 scrub ok 2024-09-11T08:16:39.595 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:39.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:39.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:39.697 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:16:39.725 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:39.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:39.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:16:39.827 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:39.863 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:39.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:39.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:39.965 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:39.999 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:40.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:40.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:40.101 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:16:40.140 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:40.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:40.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:16:40.242 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:16:40.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:40.277 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:40.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:16:40.378 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:40.409 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:40.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:40.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:40.447 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T08:16:40.483 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:40 smithi150 ceph-mon[39612]: pgmap v406: 148 pgs: 148 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.67k op/s; 8.5 MiB/s, 135 objects/s recovering 2024-09-11T08:16:40.483 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:40 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:16:40.483 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:40 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:16:40.483 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:40 smithi150 ceph-mon[39612]: 5.1c scrub starts 2024-09-11T08:16:40.483 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:40 smithi150 ceph-mon[39612]: 5.1c scrub ok 2024-09-11T08:16:40.483 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:40 smithi150 ceph-mon[39612]: osdmap e221: 8 total, 8 up, 8 in 2024-09-11T08:16:40.483 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:40 smithi150 ceph-mon[39612]: 6.10 scrub starts 2024-09-11T08:16:40.483 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:40 smithi150 ceph-mon[39612]: 6.10 scrub ok 2024-09-11T08:16:40.483 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:40 smithi150 ceph-mon[39612]: 6.1f scrub starts 2024-09-11T08:16:40.483 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:40 smithi150 ceph-mon[39612]: 6.1f scrub ok 2024-09-11T08:16:40.484 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:40 smithi150 ceph-mon[29276]: pgmap v406: 148 pgs: 148 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.67k op/s; 8.5 MiB/s, 135 objects/s recovering 2024-09-11T08:16:40.484 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:40 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:16:40.484 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:40 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:16:40.484 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:40 smithi150 ceph-mon[29276]: 5.1c scrub starts 2024-09-11T08:16:40.484 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:40 smithi150 ceph-mon[29276]: 5.1c scrub ok 2024-09-11T08:16:40.484 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:40 smithi150 ceph-mon[29276]: osdmap e221: 8 total, 8 up, 8 in 2024-09-11T08:16:40.484 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:40 smithi150 ceph-mon[29276]: 6.10 scrub starts 2024-09-11T08:16:40.484 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:40 smithi150 ceph-mon[29276]: 6.10 scrub ok 2024-09-11T08:16:40.484 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:40 smithi150 ceph-mon[29276]: 6.1f scrub starts 2024-09-11T08:16:40.484 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:40 smithi150 ceph-mon[29276]: 6.1f scrub ok 2024-09-11T08:16:40.511 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:40.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:40 smithi181 ceph-mon[32701]: pgmap v406: 148 pgs: 148 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.67k op/s; 8.5 MiB/s, 135 objects/s recovering 2024-09-11T08:16:40.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:40 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:16:40.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:40 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:16:40.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:40 smithi181 ceph-mon[32701]: 5.1c scrub starts 2024-09-11T08:16:40.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:40 smithi181 ceph-mon[32701]: 5.1c scrub ok 2024-09-11T08:16:40.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:40 smithi181 ceph-mon[32701]: osdmap e221: 8 total, 8 up, 8 in 2024-09-11T08:16:40.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:40 smithi181 ceph-mon[32701]: 6.10 scrub starts 2024-09-11T08:16:40.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:40 smithi181 ceph-mon[32701]: 6.10 scrub ok 2024-09-11T08:16:40.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:40 smithi181 ceph-mon[32701]: 6.1f scrub starts 2024-09-11T08:16:40.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:40 smithi181 ceph-mon[32701]: 6.1f scrub ok 2024-09-11T08:16:40.581 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:40.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:40.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:40.683 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:40.717 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:40.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:40.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:40.820 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:40.861 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:40.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:40.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:40.963 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:41.003 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:41.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:41.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:41.105 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:16:41.114 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-09-11T08:16:41.114 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:16:41.135 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:41.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:41.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:16:41.237 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:41.278 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:41.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:41.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:41.380 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:16:41.413 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:41.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:41.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:16:41.515 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:41.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:41.553 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:41.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:41.654 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:16:41.693 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:41.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:41.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:16:41.794 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:41.823 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:41 smithi150 ceph-mon[29276]: 2.1 scrub starts 2024-09-11T08:16:41.823 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:41 smithi150 ceph-mon[29276]: 2.1 scrub ok 2024-09-11T08:16:41.823 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:41 smithi150 ceph-mon[29276]: 4.5 scrub starts 2024-09-11T08:16:41.823 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:41 smithi150 ceph-mon[29276]: 4.5 scrub ok 2024-09-11T08:16:41.823 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:41 smithi150 ceph-mon[29276]: pgmap v408: 148 pgs: 148 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 860 op/s; 576 KiB/s, 8 objects/s recovering 2024-09-11T08:16:41.823 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:41 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3365500545' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:16:41.824 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:41 smithi150 ceph-mon[39612]: 2.1 scrub starts 2024-09-11T08:16:41.824 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:41 smithi150 ceph-mon[39612]: 2.1 scrub ok 2024-09-11T08:16:41.824 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:41 smithi150 ceph-mon[39612]: 4.5 scrub starts 2024-09-11T08:16:41.824 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:41 smithi150 ceph-mon[39612]: 4.5 scrub ok 2024-09-11T08:16:41.824 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:41 smithi150 ceph-mon[39612]: pgmap v408: 148 pgs: 148 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 860 op/s; 576 KiB/s, 8 objects/s recovering 2024-09-11T08:16:41.824 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:41 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3365500545' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:16:41.825 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-11T08:16:41.825 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:41 smithi181 ceph-mon[32701]: 2.1 scrub starts 2024-09-11T08:16:41.826 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:41 smithi181 ceph-mon[32701]: 2.1 scrub ok 2024-09-11T08:16:41.826 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:41 smithi181 ceph-mon[32701]: 4.5 scrub starts 2024-09-11T08:16:41.826 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:41 smithi181 ceph-mon[32701]: 4.5 scrub ok 2024-09-11T08:16:41.826 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:41 smithi181 ceph-mon[32701]: pgmap v408: 148 pgs: 148 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 860 op/s; 576 KiB/s, 8 objects/s recovering 2024-09-11T08:16:41.826 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:41 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3365500545' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:16:41.842 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:41.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:41.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:41.943 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:16:41.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:41.973 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:41.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:16:42.074 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:42.090 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:16:42.090 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:16:42.090 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:16:42.090 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:16:42.090 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:16:42.091 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:16:42.091 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:16:42.091 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:16:42.091 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:16:42.091 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:16:42.091 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:16:42.091 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:16:42.091 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:16:42.091 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:16:42.091 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:16:42.091 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:16:42.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:42.110 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:42.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:42.211 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:16:42.243 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:42.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:42.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:16:42.345 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:42.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:42.379 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:42.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:42.480 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:42.511 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:42.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:42.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:42.613 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:42.654 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:42.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:42.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:42.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:42 smithi150 ceph-mon[29276]: 3.a scrub starts 2024-09-11T08:16:42.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:42 smithi150 ceph-mon[29276]: 4.7 scrub starts 2024-09-11T08:16:42.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:42 smithi150 ceph-mon[29276]: 3.a scrub ok 2024-09-11T08:16:42.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:42 smithi150 ceph-mon[29276]: 4.7 scrub ok 2024-09-11T08:16:42.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:42 smithi150 ceph-mon[29276]: 5.1a scrub starts 2024-09-11T08:16:42.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:42 smithi150 ceph-mon[29276]: 5.1a scrub ok 2024-09-11T08:16:42.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:42 smithi150 ceph-mon[39612]: 3.a scrub starts 2024-09-11T08:16:42.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:42 smithi150 ceph-mon[39612]: 4.7 scrub starts 2024-09-11T08:16:42.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:42 smithi150 ceph-mon[39612]: 3.a scrub ok 2024-09-11T08:16:42.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:42 smithi150 ceph-mon[39612]: 4.7 scrub ok 2024-09-11T08:16:42.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:42 smithi150 ceph-mon[39612]: 5.1a scrub starts 2024-09-11T08:16:42.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:42 smithi150 ceph-mon[39612]: 5.1a scrub ok 2024-09-11T08:16:42.756 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:42.789 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:42.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:42.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:42.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:42 smithi181 ceph-mon[32701]: 3.a scrub starts 2024-09-11T08:16:42.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:42 smithi181 ceph-mon[32701]: 4.7 scrub starts 2024-09-11T08:16:42.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:42 smithi181 ceph-mon[32701]: 3.a scrub ok 2024-09-11T08:16:42.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:42 smithi181 ceph-mon[32701]: 4.7 scrub ok 2024-09-11T08:16:42.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:42 smithi181 ceph-mon[32701]: 5.1a scrub starts 2024-09-11T08:16:42.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:42 smithi181 ceph-mon[32701]: 5.1a scrub ok 2024-09-11T08:16:42.886 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:16:42.891 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:42.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:42.965 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:42.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:43.066 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:43.111 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:43.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:43.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:43.213 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:43.247 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:43.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:43.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:43.349 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:43.386 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:43.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:43.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:43.487 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:16:43.524 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:43.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:43.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:16:43.626 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:16:43.657 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:43.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:43.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:16:43.759 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:16:43.783 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:43.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:43.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:16:43.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:43 smithi181 ceph-mon[32701]: 3.8 scrub starts 2024-09-11T08:16:43.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:43 smithi181 ceph-mon[32701]: 3.8 scrub ok 2024-09-11T08:16:43.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:43 smithi181 ceph-mon[32701]: 5.5 scrub starts 2024-09-11T08:16:43.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:43 smithi181 ceph-mon[32701]: 5.5 scrub ok 2024-09-11T08:16:43.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:43 smithi181 ceph-mon[32701]: 3.f scrub starts 2024-09-11T08:16:43.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:43 smithi181 ceph-mon[32701]: 3.f scrub ok 2024-09-11T08:16:43.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:43 smithi181 ceph-mon[32701]: pgmap v409: 148 pgs: 148 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 829 op/s; 447 KiB/s, 6 objects/s recovering 2024-09-11T08:16:43.885 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:16:43.911 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:43.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:43.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:16:43.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:43 smithi150 ceph-mon[29276]: 3.8 scrub starts 2024-09-11T08:16:43.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:43 smithi150 ceph-mon[29276]: 3.8 scrub ok 2024-09-11T08:16:43.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:43 smithi150 ceph-mon[29276]: 5.5 scrub starts 2024-09-11T08:16:43.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:43 smithi150 ceph-mon[29276]: 5.5 scrub ok 2024-09-11T08:16:43.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:43 smithi150 ceph-mon[29276]: 3.f scrub starts 2024-09-11T08:16:43.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:43 smithi150 ceph-mon[29276]: 3.f scrub ok 2024-09-11T08:16:43.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:43 smithi150 ceph-mon[29276]: pgmap v409: 148 pgs: 148 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 829 op/s; 447 KiB/s, 6 objects/s recovering 2024-09-11T08:16:43.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:43 smithi150 ceph-mon[39612]: 3.8 scrub starts 2024-09-11T08:16:43.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:43 smithi150 ceph-mon[39612]: 3.8 scrub ok 2024-09-11T08:16:43.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:43 smithi150 ceph-mon[39612]: 5.5 scrub starts 2024-09-11T08:16:43.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:43 smithi150 ceph-mon[39612]: 5.5 scrub ok 2024-09-11T08:16:43.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:43 smithi150 ceph-mon[39612]: 3.f scrub starts 2024-09-11T08:16:43.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:43 smithi150 ceph-mon[39612]: 3.f scrub ok 2024-09-11T08:16:43.991 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:43 smithi150 ceph-mon[39612]: pgmap v409: 148 pgs: 148 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 829 op/s; 447 KiB/s, 6 objects/s recovering 2024-09-11T08:16:44.013 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:44.014 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:16:44.060 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:44.060 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:16:44.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:44.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:44.162 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:44.204 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:44.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:44.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:44.306 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:44.337 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:44.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:44.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:44.439 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:44.472 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:44.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:44.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:44.574 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:16:44.609 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:44.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:44.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:16:44.711 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:44.736 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:44.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:44.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:44.818 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:44 smithi150 ceph-mon[29276]: 3.7 scrub starts 2024-09-11T08:16:44.818 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:44 smithi150 ceph-mon[29276]: 3.7 scrub ok 2024-09-11T08:16:44.818 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:44 smithi150 ceph-mon[29276]: 6.7 scrub starts 2024-09-11T08:16:44.818 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:44 smithi150 ceph-mon[29276]: 6.7 scrub ok 2024-09-11T08:16:44.818 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:44 smithi150 ceph-mon[29276]: 5.e scrub starts 2024-09-11T08:16:44.818 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:44 smithi150 ceph-mon[29276]: 5.e scrub ok 2024-09-11T08:16:44.818 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:44 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:16:44.819 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:44 smithi150 ceph-mon[39612]: 3.7 scrub starts 2024-09-11T08:16:44.819 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:44 smithi150 ceph-mon[39612]: 3.7 scrub ok 2024-09-11T08:16:44.819 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:44 smithi150 ceph-mon[39612]: 6.7 scrub starts 2024-09-11T08:16:44.819 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:44 smithi150 ceph-mon[39612]: 6.7 scrub ok 2024-09-11T08:16:44.819 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:44 smithi150 ceph-mon[39612]: 5.e scrub starts 2024-09-11T08:16:44.819 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:44 smithi150 ceph-mon[39612]: 5.e scrub ok 2024-09-11T08:16:44.819 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:44 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:16:44.826 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:44 smithi181 ceph-mon[32701]: 3.7 scrub starts 2024-09-11T08:16:44.826 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:44 smithi181 ceph-mon[32701]: 3.7 scrub ok 2024-09-11T08:16:44.826 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:44 smithi181 ceph-mon[32701]: 6.7 scrub starts 2024-09-11T08:16:44.826 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:44 smithi181 ceph-mon[32701]: 6.7 scrub ok 2024-09-11T08:16:44.826 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:44 smithi181 ceph-mon[32701]: 5.e scrub starts 2024-09-11T08:16:44.826 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:44 smithi181 ceph-mon[32701]: 5.e scrub ok 2024-09-11T08:16:44.826 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:44 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:16:44.837 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:16:44.877 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:44.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:44.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:16:44.940 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T08:16:44.980 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:45.053 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:45.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:45.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:45.155 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:45.218 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:45.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:45.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:45.320 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:45.363 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:45.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:45.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:45.465 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:45.503 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:45.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:45.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:45.572 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:16:45 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:16:45] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:16:45.605 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:45.652 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:45.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:45.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:45.754 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:16:45.789 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:45.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:45.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:16:45.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:45 smithi181 ceph-mon[32701]: 5.1f scrub starts 2024-09-11T08:16:45.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:45 smithi181 ceph-mon[32701]: 5.1f scrub ok 2024-09-11T08:16:45.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:45 smithi181 ceph-mon[32701]: 8.a scrub starts 2024-09-11T08:16:45.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:45 smithi181 ceph-mon[32701]: pgmap v410: 148 pgs: 148 active+clean; 3.6 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.44k op/s; 432 KiB/s, 6 objects/s recovering 2024-09-11T08:16:45.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:45 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:16:45.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:45 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:16:45.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:45 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:16:45.892 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:16:45.900 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:45 smithi150 ceph-mon[29276]: 5.1f scrub starts 2024-09-11T08:16:45.900 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:45 smithi150 ceph-mon[29276]: 5.1f scrub ok 2024-09-11T08:16:45.900 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:45 smithi150 ceph-mon[29276]: 8.a scrub starts 2024-09-11T08:16:45.900 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:45 smithi150 ceph-mon[29276]: pgmap v410: 148 pgs: 148 active+clean; 3.6 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.44k op/s; 432 KiB/s, 6 objects/s recovering 2024-09-11T08:16:45.900 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:45 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:16:45.900 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:45 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:16:45.900 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:45 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:16:45.900 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:45 smithi150 ceph-mon[39612]: 5.1f scrub starts 2024-09-11T08:16:45.901 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:45 smithi150 ceph-mon[39612]: 5.1f scrub ok 2024-09-11T08:16:45.901 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:45 smithi150 ceph-mon[39612]: 8.a scrub starts 2024-09-11T08:16:45.901 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:45 smithi150 ceph-mon[39612]: pgmap v410: 148 pgs: 148 active+clean; 3.6 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.44k op/s; 432 KiB/s, 6 objects/s recovering 2024-09-11T08:16:45.901 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:45 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:16:45.901 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:45 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:16:45.901 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:45 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:16:45.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:45.922 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:45.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:16:46.023 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:46.046 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:46.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:46.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:46.147 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:16:46.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:46.180 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:46.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:16:46.281 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:46.311 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:46.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:46.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:46.413 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:46.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:46.438 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:46.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:46.539 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:16:46.569 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:46.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:46.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:16:46.671 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:46.705 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:46.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:46.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:46.807 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:46.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:46 smithi181 ceph-mon[32701]: 1.0 scrub starts 2024-09-11T08:16:46.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:46 smithi181 ceph-mon[32701]: 1.0 scrub ok 2024-09-11T08:16:46.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:46 smithi181 ceph-mon[32701]: 4.e scrub starts 2024-09-11T08:16:46.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:46 smithi181 ceph-mon[32701]: 4.e scrub ok 2024-09-11T08:16:46.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:46 smithi181 ceph-mon[32701]: 8.a scrub ok 2024-09-11T08:16:46.844 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:46.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:46.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:46.906 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:46 smithi150 ceph-mon[29276]: 1.0 scrub starts 2024-09-11T08:16:46.906 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:46 smithi150 ceph-mon[29276]: 1.0 scrub ok 2024-09-11T08:16:46.906 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:46 smithi150 ceph-mon[29276]: 4.e scrub starts 2024-09-11T08:16:46.906 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:46 smithi150 ceph-mon[29276]: 4.e scrub ok 2024-09-11T08:16:46.907 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:46 smithi150 ceph-mon[29276]: 8.a scrub ok 2024-09-11T08:16:46.907 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:46 smithi150 ceph-mon[39612]: 1.0 scrub starts 2024-09-11T08:16:46.907 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:46 smithi150 ceph-mon[39612]: 1.0 scrub ok 2024-09-11T08:16:46.907 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:46 smithi150 ceph-mon[39612]: 4.e scrub starts 2024-09-11T08:16:46.907 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:46 smithi150 ceph-mon[39612]: 4.e scrub ok 2024-09-11T08:16:46.907 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:46 smithi150 ceph-mon[39612]: 8.a scrub ok 2024-09-11T08:16:46.946 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:16:46.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:46.982 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:46.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:16:47.083 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:16:47.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:47.122 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:47.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:16:47.223 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:16:47.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:47.251 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:47.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:16:47.352 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:47.399 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:47.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:47.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:47.501 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:47.536 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:47.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:47.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:47.638 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:16:47.670 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:47.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:47.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:16:47.772 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:47.806 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:47.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:47.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:47.873 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:47 smithi150 ceph-mon[29276]: 5.1e scrub starts 2024-09-11T08:16:47.873 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:47 smithi150 ceph-mon[29276]: 5.1e scrub ok 2024-09-11T08:16:47.873 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:47 smithi150 ceph-mon[29276]: 3.d scrub starts 2024-09-11T08:16:47.873 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:47 smithi150 ceph-mon[29276]: 3.d scrub ok 2024-09-11T08:16:47.873 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:47 smithi150 ceph-mon[29276]: pgmap v411: 148 pgs: 148 active+clean; 3.6 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.18k op/s; 356 KiB/s, 5 objects/s recovering 2024-09-11T08:16:47.874 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:47 smithi150 ceph-mon[39612]: 5.1e scrub starts 2024-09-11T08:16:47.874 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:47 smithi150 ceph-mon[39612]: 5.1e scrub ok 2024-09-11T08:16:47.874 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:47 smithi150 ceph-mon[39612]: 3.d scrub starts 2024-09-11T08:16:47.874 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:47 smithi150 ceph-mon[39612]: 3.d scrub ok 2024-09-11T08:16:47.874 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:47 smithi150 ceph-mon[39612]: pgmap v411: 148 pgs: 148 active+clean; 3.6 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.18k op/s; 356 KiB/s, 5 objects/s recovering 2024-09-11T08:16:47.909 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:47.953 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:48.084 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:47 smithi181 ceph-mon[32701]: 5.1e scrub starts 2024-09-11T08:16:48.084 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:47 smithi181 ceph-mon[32701]: 5.1e scrub ok 2024-09-11T08:16:48.084 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:47 smithi181 ceph-mon[32701]: 3.d scrub starts 2024-09-11T08:16:48.084 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:47 smithi181 ceph-mon[32701]: 3.d scrub ok 2024-09-11T08:16:48.084 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:47 smithi181 ceph-mon[32701]: pgmap v411: 148 pgs: 148 active+clean; 3.6 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.18k op/s; 356 KiB/s, 5 objects/s recovering 2024-09-11T08:16:48.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:48.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:48.186 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:16:48.222 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:48.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:48.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:16:48.324 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:48.330 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:16:48.352 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:48.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:48.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:48.454 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:48.502 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:48.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:48.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:48.573 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:16:48 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:16:48.604 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:48.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:48.644 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:48.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:48.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:48 smithi150 ceph-mon[29276]: 3.e scrub starts 2024-09-11T08:16:48.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:48 smithi150 ceph-mon[29276]: 3.e scrub ok 2024-09-11T08:16:48.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:48 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:16:48.739 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:16:48 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:16:48] "GET /metrics HTTP/1.1" 200 224237 "" "Prometheus/2.43.0" 2024-09-11T08:16:48.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:48 smithi150 ceph-mon[39612]: 3.e scrub starts 2024-09-11T08:16:48.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:48 smithi150 ceph-mon[39612]: 3.e scrub ok 2024-09-11T08:16:48.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:48 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:16:48.745 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:48.777 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:48.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:48.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:48.879 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:16:48.910 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:48.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:48.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:16:49.012 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:16:49.037 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:49.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:49.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:16:49.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:48 smithi181 ceph-mon[32701]: 3.e scrub starts 2024-09-11T08:16:49.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:48 smithi181 ceph-mon[32701]: 3.e scrub ok 2024-09-11T08:16:49.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:48 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:16:49.139 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:16:49.164 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:49.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:49.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:16:49.266 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:49.297 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:49.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:49.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:49.399 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:16:49.423 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:49.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:49.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:16:49.443 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:16:49.446 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-09-11T08:16:49.525 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:49.562 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:49.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:49.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:49.595 INFO:teuthology.orchestra.run.smithi150.stderr:nodeep-scrub is unset 2024-09-11T08:16:49.663 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:16:49.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:49.696 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:49.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:16:49.798 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:16:49.823 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:49.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:49.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:16:49.925 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:49.949 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:49.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:49.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:49.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:49 smithi150 ceph-mon[29276]: 8.5 scrub starts 2024-09-11T08:16:49.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:49 smithi150 ceph-mon[29276]: 8.3 scrub starts 2024-09-11T08:16:49.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:49 smithi150 ceph-mon[29276]: pgmap v412: 148 pgs: 148 active+clean; 3.2 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s 2024-09-11T08:16:49.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:49 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2478514926' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:16:49.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:49 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:16:49.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:49 smithi150 ceph-mon[39612]: 8.5 scrub starts 2024-09-11T08:16:49.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:49 smithi150 ceph-mon[39612]: 8.3 scrub starts 2024-09-11T08:16:49.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:49 smithi150 ceph-mon[39612]: pgmap v412: 148 pgs: 148 active+clean; 3.2 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s 2024-09-11T08:16:49.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:49 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2478514926' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:16:49.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:49 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:16:50.051 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:50.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:49 smithi181 ceph-mon[32701]: 8.5 scrub starts 2024-09-11T08:16:50.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:49 smithi181 ceph-mon[32701]: 8.3 scrub starts 2024-09-11T08:16:50.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:49 smithi181 ceph-mon[32701]: pgmap v412: 148 pgs: 148 active+clean; 3.2 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s 2024-09-11T08:16:50.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:49 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2478514926' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:16:50.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:49 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:16:50.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:50.087 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:50.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:50.189 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:50.222 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:50.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:50.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:50.323 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:50.361 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:50.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:50.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:50.462 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:16:50.496 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:50.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:50.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:16:50.597 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:16:50.626 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:50.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:50.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:16:50.728 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:50.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:50 smithi150 ceph-mon[29276]: 8.5 scrub ok 2024-09-11T08:16:50.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:50 smithi150 ceph-mon[29276]: from='client.25160 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:16:50.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:50 smithi150 ceph-mon[29276]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:16:50.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:50 smithi150 ceph-mon[29276]: Cluster is now healthy 2024-09-11T08:16:50.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:50 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:16:50.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:50 smithi150 ceph-mon[29276]: osdmap e222: 8 total, 8 up, 8 in 2024-09-11T08:16:50.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:50 smithi150 ceph-mon[29276]: 4.b scrub starts 2024-09-11T08:16:50.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:50 smithi150 ceph-mon[29276]: 4.b scrub ok 2024-09-11T08:16:50.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:50 smithi150 ceph-mon[29276]: 8.3 scrub ok 2024-09-11T08:16:50.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:50 smithi150 ceph-mon[39612]: 8.5 scrub ok 2024-09-11T08:16:50.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:50 smithi150 ceph-mon[39612]: from='client.25160 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:16:50.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:50 smithi150 ceph-mon[39612]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:16:50.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:50 smithi150 ceph-mon[39612]: Cluster is now healthy 2024-09-11T08:16:50.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:50 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:16:50.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:50 smithi150 ceph-mon[39612]: osdmap e222: 8 total, 8 up, 8 in 2024-09-11T08:16:50.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:50 smithi150 ceph-mon[39612]: 4.b scrub starts 2024-09-11T08:16:50.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:50 smithi150 ceph-mon[39612]: 4.b scrub ok 2024-09-11T08:16:50.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:50 smithi150 ceph-mon[39612]: 8.3 scrub ok 2024-09-11T08:16:50.765 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:50.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:50.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:50.867 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:50.899 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:50.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:50.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:51.001 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:51.027 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:51.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:51.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:51.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:50 smithi181 ceph-mon[32701]: 8.5 scrub ok 2024-09-11T08:16:51.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:50 smithi181 ceph-mon[32701]: from='client.25160 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:16:51.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:50 smithi181 ceph-mon[32701]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:16:51.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:50 smithi181 ceph-mon[32701]: Cluster is now healthy 2024-09-11T08:16:51.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:50 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:16:51.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:50 smithi181 ceph-mon[32701]: osdmap e222: 8 total, 8 up, 8 in 2024-09-11T08:16:51.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:50 smithi181 ceph-mon[32701]: 4.b scrub starts 2024-09-11T08:16:51.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:50 smithi181 ceph-mon[32701]: 4.b scrub ok 2024-09-11T08:16:51.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:50 smithi181 ceph-mon[32701]: 8.3 scrub ok 2024-09-11T08:16:51.129 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:16:51.161 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:51.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:51.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:16:51.263 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:16:51.298 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:51.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:51.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:16:51.400 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:51.434 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:51.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:51.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:51.535 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:51.570 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:51.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:51.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:51.672 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:51.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:51.707 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:51.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:51.808 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:16:51.844 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:51.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:51.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:16:51.946 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:51.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:51 smithi150 ceph-mon[39612]: 5.d scrub starts 2024-09-11T08:16:51.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:51 smithi150 ceph-mon[39612]: 5.d scrub ok 2024-09-11T08:16:51.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:51 smithi150 ceph-mon[39612]: 6.1d deep-scrub starts 2024-09-11T08:16:51.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:51 smithi150 ceph-mon[39612]: 6.1d deep-scrub ok 2024-09-11T08:16:51.956 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:51 smithi150 ceph-mon[39612]: 6.b scrub starts 2024-09-11T08:16:51.956 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:51 smithi150 ceph-mon[39612]: 6.b scrub ok 2024-09-11T08:16:51.956 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:51 smithi150 ceph-mon[39612]: pgmap v414: 148 pgs: 148 active+clean; 3.2 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s 2024-09-11T08:16:51.956 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:51 smithi150 ceph-mon[29276]: 5.d scrub starts 2024-09-11T08:16:51.956 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:51 smithi150 ceph-mon[29276]: 5.d scrub ok 2024-09-11T08:16:51.956 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:51 smithi150 ceph-mon[29276]: 6.1d deep-scrub starts 2024-09-11T08:16:51.956 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:51 smithi150 ceph-mon[29276]: 6.1d deep-scrub ok 2024-09-11T08:16:51.956 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:51 smithi150 ceph-mon[29276]: 6.b scrub starts 2024-09-11T08:16:51.956 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:51 smithi150 ceph-mon[29276]: 6.b scrub ok 2024-09-11T08:16:51.956 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:51 smithi150 ceph-mon[29276]: pgmap v414: 148 pgs: 148 active+clean; 3.2 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s 2024-09-11T08:16:51.969 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:51.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:51.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:52.070 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:16:52.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:51 smithi181 ceph-mon[32701]: 5.d scrub starts 2024-09-11T08:16:52.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:51 smithi181 ceph-mon[32701]: 5.d scrub ok 2024-09-11T08:16:52.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:51 smithi181 ceph-mon[32701]: 6.1d deep-scrub starts 2024-09-11T08:16:52.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:51 smithi181 ceph-mon[32701]: 6.1d deep-scrub ok 2024-09-11T08:16:52.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:51 smithi181 ceph-mon[32701]: 6.b scrub starts 2024-09-11T08:16:52.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:51 smithi181 ceph-mon[32701]: 6.b scrub ok 2024-09-11T08:16:52.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:51 smithi181 ceph-mon[32701]: pgmap v414: 148 pgs: 148 active+clean; 3.2 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s 2024-09-11T08:16:52.101 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:52.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:52.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:16:52.203 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:52.234 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:52.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:52.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:52.335 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:16:52.366 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:52.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:52.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:16:52.468 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:52.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:52.499 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:52.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:52.600 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:16:52.623 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:52.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:52.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:16:52.725 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:52.763 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:52.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:52.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:52.865 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:52.893 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:52.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:52.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:52.897 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-09-11T08:16:52.897 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:16:52.898 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:52 smithi150 ceph-mon[29276]: 5.6 scrub starts 2024-09-11T08:16:52.898 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:52 smithi150 ceph-mon[29276]: 5.6 scrub ok 2024-09-11T08:16:52.899 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:52 smithi150 ceph-mon[29276]: 5.11 scrub starts 2024-09-11T08:16:52.899 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:52 smithi150 ceph-mon[29276]: 5.11 scrub ok 2024-09-11T08:16:52.899 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:52 smithi150 ceph-mon[29276]: 3.1c scrub starts 2024-09-11T08:16:52.899 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:52 smithi150 ceph-mon[29276]: 3.1c scrub ok 2024-09-11T08:16:52.899 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:52 smithi150 ceph-mon[29276]: 3.15 scrub starts 2024-09-11T08:16:52.899 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:52 smithi150 ceph-mon[29276]: 3.15 scrub ok 2024-09-11T08:16:52.899 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:52 smithi150 ceph-mon[29276]: 8.e deep-scrub starts 2024-09-11T08:16:52.900 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:52 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:16:52.900 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:52 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:16:52.900 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:52 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:16:52.900 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:52 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:16:52.900 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:52 smithi150 ceph-mon[39612]: 5.6 scrub starts 2024-09-11T08:16:52.900 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:52 smithi150 ceph-mon[39612]: 5.6 scrub ok 2024-09-11T08:16:52.900 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:52 smithi150 ceph-mon[39612]: 5.11 scrub starts 2024-09-11T08:16:52.900 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:52 smithi150 ceph-mon[39612]: 5.11 scrub ok 2024-09-11T08:16:52.901 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:52 smithi150 ceph-mon[39612]: 3.1c scrub starts 2024-09-11T08:16:52.901 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:52 smithi150 ceph-mon[39612]: 3.1c scrub ok 2024-09-11T08:16:52.901 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:52 smithi150 ceph-mon[39612]: 3.15 scrub starts 2024-09-11T08:16:52.901 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:52 smithi150 ceph-mon[39612]: 3.15 scrub ok 2024-09-11T08:16:52.901 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:52 smithi150 ceph-mon[39612]: 8.e deep-scrub starts 2024-09-11T08:16:52.901 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:52 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:16:52.901 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:52 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:16:52.901 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:52 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:16:52.901 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:52 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:16:52.995 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:16:53.023 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:53.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:53.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:16:53.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:52 smithi181 ceph-mon[32701]: 5.6 scrub starts 2024-09-11T08:16:53.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:52 smithi181 ceph-mon[32701]: 5.6 scrub ok 2024-09-11T08:16:53.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:52 smithi181 ceph-mon[32701]: 5.11 scrub starts 2024-09-11T08:16:53.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:52 smithi181 ceph-mon[32701]: 5.11 scrub ok 2024-09-11T08:16:53.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:52 smithi181 ceph-mon[32701]: 3.1c scrub starts 2024-09-11T08:16:53.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:52 smithi181 ceph-mon[32701]: 3.1c scrub ok 2024-09-11T08:16:53.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:52 smithi181 ceph-mon[32701]: 3.15 scrub starts 2024-09-11T08:16:53.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:52 smithi181 ceph-mon[32701]: 3.15 scrub ok 2024-09-11T08:16:53.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:52 smithi181 ceph-mon[32701]: 8.e deep-scrub starts 2024-09-11T08:16:53.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:52 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:16:53.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:52 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:16:53.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:52 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:16:53.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:52 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:16:53.125 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:53.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:53.158 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:53.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:53.259 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:53.285 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:53.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:53.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:53.387 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:16:53.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:53.420 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:53.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:16:53.521 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:16:53.555 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:53.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:53.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:16:53.656 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:53.683 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:53.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:53.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:53.784 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:16:53.810 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:53.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:53.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:16:53.912 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:53.933 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:53.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:53.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:54.035 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:54.067 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:54.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:54.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:54.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:53 smithi181 ceph-mon[32701]: 5.c scrub starts 2024-09-11T08:16:54.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:53 smithi181 ceph-mon[32701]: 5.c scrub ok 2024-09-11T08:16:54.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:53 smithi181 ceph-mon[32701]: 5.14 deep-scrub starts 2024-09-11T08:16:54.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:53 smithi181 ceph-mon[32701]: 5.14 deep-scrub ok 2024-09-11T08:16:54.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:53 smithi181 ceph-mon[32701]: 3.12 deep-scrub starts 2024-09-11T08:16:54.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:53 smithi181 ceph-mon[32701]: 3.12 deep-scrub ok 2024-09-11T08:16:54.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:53 smithi181 ceph-mon[32701]: osdmap e223: 8 total, 8 up, 8 in 2024-09-11T08:16:54.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:53 smithi181 ceph-mon[32701]: pgmap v416: 148 pgs: 148 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 890 op/s 2024-09-11T08:16:54.168 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:54.199 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:54.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:54.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:54.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:53 smithi150 ceph-mon[29276]: 5.c scrub starts 2024-09-11T08:16:54.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:53 smithi150 ceph-mon[29276]: 5.c scrub ok 2024-09-11T08:16:54.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:53 smithi150 ceph-mon[29276]: 5.14 deep-scrub starts 2024-09-11T08:16:54.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:53 smithi150 ceph-mon[29276]: 5.14 deep-scrub ok 2024-09-11T08:16:54.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:53 smithi150 ceph-mon[29276]: 3.12 deep-scrub starts 2024-09-11T08:16:54.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:53 smithi150 ceph-mon[29276]: 3.12 deep-scrub ok 2024-09-11T08:16:54.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:53 smithi150 ceph-mon[29276]: osdmap e223: 8 total, 8 up, 8 in 2024-09-11T08:16:54.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:53 smithi150 ceph-mon[29276]: pgmap v416: 148 pgs: 148 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 890 op/s 2024-09-11T08:16:54.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:53 smithi150 ceph-mon[39612]: 5.c scrub starts 2024-09-11T08:16:54.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:53 smithi150 ceph-mon[39612]: 5.c scrub ok 2024-09-11T08:16:54.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:53 smithi150 ceph-mon[39612]: 5.14 deep-scrub starts 2024-09-11T08:16:54.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:53 smithi150 ceph-mon[39612]: 5.14 deep-scrub ok 2024-09-11T08:16:54.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:53 smithi150 ceph-mon[39612]: 3.12 deep-scrub starts 2024-09-11T08:16:54.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:53 smithi150 ceph-mon[39612]: 3.12 deep-scrub ok 2024-09-11T08:16:54.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:53 smithi150 ceph-mon[39612]: osdmap e223: 8 total, 8 up, 8 in 2024-09-11T08:16:54.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:53 smithi150 ceph-mon[39612]: pgmap v416: 148 pgs: 148 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 890 op/s 2024-09-11T08:16:54.300 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:16:54.324 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:54.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:54.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:16:54.426 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:54.457 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:54.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:54.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:54.491 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":412,"stamp":"2024-09-11T08:16:48.887847+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3418364148,"num_objects":52348,"num_object_clones":0,"num_object_copies":157044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52348,"num_whiteouts":0,"num_read":974,"num_read_kb":833,"num_write":206051,"num_write_kb":8246728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17905,"num_bytes_recovered":1165103644,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85874,"ondisk_log_size":85874,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":446,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":13154740,"kb_used_data":10701428,"kb_used_omap":19,"kb_used_meta":2452972,"kb_avail":736806476,"statfs":{"total":767960285184,"available":754489831424,"internally_reserved":0,"allocated":10958262272,"data_stored":10955007868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19955,"internal_metadata":2511843853},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-815792128,"num_objects":-12448,"num_object_clones":0,"num_object_copies":-37344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-12448,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":12448,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.005433"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":211,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.667468+0000","last_change":"2024-09-11T08:16:37.667413+0000","last_active":"2024-09-11T08:16:37.667468+0000","last_peered":"2024-09-11T08:16:37.667468+0000","last_clean":"2024-09-11T08:16:37.667468+0000","last_became_active":"2024-09-11T08:07:30.044976+0000","last_became_peered":"2024-09-11T08:07:30.044976+0000","last_unstale":"2024-09-11T08:16:37.667468+0000","last_undegraded":"2024-09-11T08:16:37.667468+0000","last_fullsized":"2024-09-11T08:16:37.667468+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:37.667335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:16:37.667335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:40.583337+0000","scrub_duration":0.0048271249999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":215,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.177298+0000","last_change":"2024-09-11T08:15:37.106907+0000","last_active":"2024-09-11T08:16:37.177298+0000","last_peered":"2024-09-11T08:16:37.177298+0000","last_clean":"2024-09-11T08:16:37.177298+0000","last_became_active":"2024-09-11T08:07:32.048048+0000","last_became_peered":"2024-09-11T08:07:32.048048+0000","last_unstale":"2024-09-11T08:16:37.177298+0000","last_undegraded":"2024-09-11T08:16:37.177298+0000","last_fullsized":"2024-09-11T08:16:37.177298+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:15:37.106824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:15:37.106824+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:39.764009+0000","scrub_duration":0.0038285900000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":244,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.177439+0000","last_change":"2024-09-11T08:15:56.531271+0000","last_active":"2024-09-11T08:16:37.177439+0000","last_peered":"2024-09-11T08:16:37.177439+0000","last_clean":"2024-09-11T08:16:37.177439+0000","last_became_active":"2024-09-11T08:15:56.530946+0000","last_became_peered":"2024-09-11T08:15:56.530946+0000","last_unstale":"2024-09-11T08:16:37.177439+0000","last_undegraded":"2024-09-11T08:16:37.177439+0000","last_fullsized":"2024-09-11T08:16:37.177439+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:57.020218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:14:57.020218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034881460000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":195,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.177539+0000","last_change":"2024-09-11T08:13:10.073583+0000","last_active":"2024-09-11T08:16:37.177539+0000","last_peered":"2024-09-11T08:16:37.177539+0000","last_clean":"2024-09-11T08:16:37.177539+0000","last_became_active":"2024-09-11T08:07:36.074789+0000","last_became_peered":"2024-09-11T08:07:36.074789+0000","last_unstale":"2024-09-11T08:16:37.177539+0000","last_undegraded":"2024-09-11T08:16:37.177539+0000","last_fullsized":"2024-09-11T08:16:37.177539+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:10.073490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:02.470649+0000","last_clean_scrub_stamp":"2024-09-11T08:13:10.073490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0098908899999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":222,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.174682+0000","last_change":"2024-09-11T08:15:56.521398+0000","last_active":"2024-09-11T08:16:37.174682+0000","last_peered":"2024-09-11T08:16:37.174682+0000","last_clean":"2024-09-11T08:16:37.174682+0000","last_became_active":"2024-09-11T08:15:56.521104+0000","last_became_peered":"2024-09-11T08:15:56.521104+0000","last_unstale":"2024-09-11T08:16:37.174682+0000","last_undegraded":"2024-09-11T08:16:37.174682+0000","last_fullsized":"2024-09-11T08:16:37.174682+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:58.222356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:14:58.222356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013526233,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":201,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.174778+0000","last_change":"2024-09-11T08:15:31.254961+0000","last_active":"2024-09-11T08:16:37.174778+0000","last_peered":"2024-09-11T08:16:37.174778+0000","last_clean":"2024-09-11T08:16:37.174778+0000","last_became_active":"2024-09-11T08:07:30.045579+0000","last_became_peered":"2024-09-11T08:07:30.045579+0000","last_unstale":"2024-09-11T08:16:37.174778+0000","last_undegraded":"2024-09-11T08:16:37.174778+0000","last_fullsized":"2024-09-11T08:16:37.174778+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:31.254885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:31.254885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033689100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"145'22","reported_seq":241,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.174370+0000","last_change":"2024-09-11T08:15:57.478927+0000","last_active":"2024-09-11T08:16:37.174370+0000","last_peered":"2024-09-11T08:16:37.174370+0000","last_clean":"2024-09-11T08:16:37.174370+0000","last_became_active":"2024-09-11T08:07:32.045848+0000","last_became_peered":"2024-09-11T08:07:32.045848+0000","last_unstale":"2024-09-11T08:16:37.174370+0000","last_undegraded":"2024-09-11T08:16:37.174370+0000","last_fullsized":"2024-09-11T08:16:37.174370+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'22","last_scrub_stamp":"2024-09-11T08:15:57.478870+0000","last_deep_scrub":"145'22","last_deep_scrub_stamp":"2024-09-11T08:15:57.478870+0000","last_clean_scrub_stamp":"2024-09-11T08:15:57.478870+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:13.662088+0000","scrub_duration":0.0021529710000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":197,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.177269+0000","last_change":"2024-09-11T08:15:52.050101+0000","last_active":"2024-09-11T08:16:37.177269+0000","last_peered":"2024-09-11T08:16:37.177269+0000","last_clean":"2024-09-11T08:16:37.177269+0000","last_became_active":"2024-09-11T08:07:34.052243+0000","last_became_peered":"2024-09-11T08:07:34.052243+0000","last_unstale":"2024-09-11T08:16:37.177269+0000","last_undegraded":"2024-09-11T08:16:37.177269+0000","last_fullsized":"2024-09-11T08:16:37.177269+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:52.050017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:15:52.050017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:20.035807+0000","scrub_duration":0.0037049880000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":213,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.184027+0000","last_change":"2024-09-11T08:15:31.550213+0000","last_active":"2024-09-11T08:16:37.184027+0000","last_peered":"2024-09-11T08:16:37.184027+0000","last_clean":"2024-09-11T08:16:37.184027+0000","last_became_active":"2024-09-11T08:07:30.043950+0000","last_became_peered":"2024-09-11T08:07:30.043950+0000","last_unstale":"2024-09-11T08:16:37.184027+0000","last_undegraded":"2024-09-11T08:16:37.184027+0000","last_fullsized":"2024-09-11T08:16:37.184027+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:31.550091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:54.878409+0000","last_clean_scrub_stamp":"2024-09-11T08:15:31.550091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:56.355754+0000","scrub_duration":0.0017129999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":215,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.177729+0000","last_change":"2024-09-11T08:13:04.107341+0000","last_active":"2024-09-11T08:16:37.177729+0000","last_peered":"2024-09-11T08:16:37.177729+0000","last_clean":"2024-09-11T08:16:37.177729+0000","last_became_active":"2024-09-11T08:07:32.044626+0000","last_became_peered":"2024-09-11T08:07:32.044626+0000","last_unstale":"2024-09-11T08:16:37.177729+0000","last_undegraded":"2024-09-11T08:16:37.177729+0000","last_fullsized":"2024-09-11T08:16:37.177729+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:13:04.107254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:13:04.107254+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0056027890000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":233,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.183502+0000","last_change":"2024-09-11T08:15:56.531009+0000","last_active":"2024-09-11T08:16:37.183502+0000","last_peered":"2024-09-11T08:16:37.183502+0000","last_clean":"2024-09-11T08:16:37.183502+0000","last_became_active":"2024-09-11T08:15:56.530680+0000","last_became_peered":"2024-09-11T08:15:56.530680+0000","last_unstale":"2024-09-11T08:16:37.183502+0000","last_undegraded":"2024-09-11T08:16:37.183502+0000","last_fullsized":"2024-09-11T08:16:37.183502+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:50.469752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:15:50.469752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:08.638035+0000","scrub_duration":0.0033416409999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":212,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.180052+0000","last_change":"2024-09-11T08:14:53.516662+0000","last_active":"2024-09-11T08:16:37.180052+0000","last_peered":"2024-09-11T08:16:37.180052+0000","last_clean":"2024-09-11T08:16:37.180052+0000","last_became_active":"2024-09-11T08:14:24.792595+0000","last_became_peered":"2024-09-11T08:14:24.792595+0000","last_unstale":"2024-09-11T08:16:37.180052+0000","last_undegraded":"2024-09-11T08:16:37.180052+0000","last_fullsized":"2024-09-11T08:16:37.180052+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:53.516583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:14:53.516583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034487419999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":210,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.185650+0000","last_change":"2024-09-11T08:14:55.444867+0000","last_active":"2024-09-11T08:16:37.185650+0000","last_peered":"2024-09-11T08:16:37.185650+0000","last_clean":"2024-09-11T08:16:37.185650+0000","last_became_active":"2024-09-11T08:07:30.043277+0000","last_became_peered":"2024-09-11T08:07:30.043277+0000","last_unstale":"2024-09-11T08:16:37.185650+0000","last_undegraded":"2024-09-11T08:16:37.185650+0000","last_fullsized":"2024-09-11T08:16:37.185650+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:55.444806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:14:55.444806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0099859159999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"139'7","reported_seq":211,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.177938+0000","last_change":"2024-09-11T08:15:02.988616+0000","last_active":"2024-09-11T08:16:37.177938+0000","last_peered":"2024-09-11T08:16:37.177938+0000","last_clean":"2024-09-11T08:16:37.177938+0000","last_became_active":"2024-09-11T08:07:32.048661+0000","last_became_peered":"2024-09-11T08:07:32.048661+0000","last_unstale":"2024-09-11T08:16:37.177938+0000","last_undegraded":"2024-09-11T08:16:37.177938+0000","last_fullsized":"2024-09-11T08:16:37.177938+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'7","last_scrub_stamp":"2024-09-11T08:15:02.988425+0000","last_deep_scrub":"139'7","last_deep_scrub_stamp":"2024-09-11T08:12:24.869318+0000","last_clean_scrub_stamp":"2024-09-11T08:15:02.988425+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033901550000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":33,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-11T08:16:41.727462+0000","last_change":"2024-09-11T08:16:41.727421+0000","last_active":"2024-09-11T08:16:41.727462+0000","last_peered":"2024-09-11T08:16:41.727462+0000","last_clean":"2024-09-11T08:16:41.727462+0000","last_became_active":"2024-09-11T08:15:56.526954+0000","last_became_peered":"2024-09-11T08:15:56.526954+0000","last_unstale":"2024-09-11T08:16:41.727462+0000","last_undegraded":"2024-09-11T08:16:41.727462+0000","last_fullsized":"2024-09-11T08:16:41.727462+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:41.727344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:16:41.727344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:50.309626+0000","scrub_duration":0.0011409409999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":214,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.186827+0000","last_change":"2024-09-11T08:15:55.371107+0000","last_active":"2024-09-11T08:16:37.186827+0000","last_peered":"2024-09-11T08:16:37.186827+0000","last_clean":"2024-09-11T08:16:37.186827+0000","last_became_active":"2024-09-11T08:07:36.077602+0000","last_became_peered":"2024-09-11T08:07:36.077602+0000","last_unstale":"2024-09-11T08:16:37.186827+0000","last_undegraded":"2024-09-11T08:16:37.186827+0000","last_fullsized":"2024-09-11T08:16:37.186827+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:55.371030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:04.782577+0000","last_clean_scrub_stamp":"2024-09-11T08:15:55.371030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:57.415606+0000","scrub_duration":0.0028204950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":195,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.178154+0000","last_change":"2024-09-11T08:13:11.061617+0000","last_active":"2024-09-11T08:16:37.178154+0000","last_peered":"2024-09-11T08:16:37.178154+0000","last_clean":"2024-09-11T08:16:37.178154+0000","last_became_active":"2024-09-11T08:07:36.633338+0000","last_became_peered":"2024-09-11T08:07:36.633338+0000","last_unstale":"2024-09-11T08:16:37.178154+0000","last_undegraded":"2024-09-11T08:16:37.178154+0000","last_fullsized":"2024-09-11T08:16:37.178154+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:11.061558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:13:11.061558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016408200000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":228,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.181428+0000","last_change":"2024-09-11T08:15:56.517613+0000","last_active":"2024-09-11T08:16:37.181428+0000","last_peered":"2024-09-11T08:16:37.181428+0000","last_clean":"2024-09-11T08:16:37.181428+0000","last_became_active":"2024-09-11T08:15:56.516941+0000","last_became_peered":"2024-09-11T08:15:56.516941+0000","last_unstale":"2024-09-11T08:16:37.181428+0000","last_undegraded":"2024-09-11T08:16:37.181428+0000","last_fullsized":"2024-09-11T08:16:37.181428+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:13.315410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:14:13.315410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030491400000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"107'22","reported_seq":250,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.185139+0000","last_change":"2024-09-11T08:16:04.159388+0000","last_active":"2024-09-11T08:16:37.185139+0000","last_peered":"2024-09-11T08:16:37.185139+0000","last_clean":"2024-09-11T08:16:37.185139+0000","last_became_active":"2024-09-11T08:07:32.046363+0000","last_became_peered":"2024-09-11T08:07:32.046363+0000","last_unstale":"2024-09-11T08:16:37.185139+0000","last_undegraded":"2024-09-11T08:16:37.185139+0000","last_fullsized":"2024-09-11T08:16:37.185139+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'22","last_scrub_stamp":"2024-09-11T08:16:04.159211+0000","last_deep_scrub":"107'22","last_deep_scrub_stamp":"2024-09-11T08:16:04.159211+0000","last_clean_scrub_stamp":"2024-09-11T08:16:04.159211+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:30.394200+0000","scrub_duration":0.0069586229999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":217,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.174676+0000","last_change":"2024-09-11T08:15:46.597272+0000","last_active":"2024-09-11T08:16:37.174676+0000","last_peered":"2024-09-11T08:16:37.174676+0000","last_clean":"2024-09-11T08:16:37.174676+0000","last_became_active":"2024-09-11T08:07:34.055410+0000","last_became_peered":"2024-09-11T08:07:34.055410+0000","last_unstale":"2024-09-11T08:16:37.174676+0000","last_undegraded":"2024-09-11T08:16:37.174676+0000","last_fullsized":"2024-09-11T08:16:37.174676+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:46.597198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:10:31.418873+0000","last_clean_scrub_stamp":"2024-09-11T08:15:46.597198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:00.695980+0000","scrub_duration":0.0032858620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":198,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-11T08:16:39.677566+0000","last_change":"2024-09-11T08:16:39.677479+0000","last_active":"2024-09-11T08:16:39.677566+0000","last_peered":"2024-09-11T08:16:39.677566+0000","last_clean":"2024-09-11T08:16:39.677566+0000","last_became_active":"2024-09-11T08:07:36.633258+0000","last_became_peered":"2024-09-11T08:07:36.633258+0000","last_unstale":"2024-09-11T08:16:39.677566+0000","last_undegraded":"2024-09-11T08:16:39.677566+0000","last_fullsized":"2024-09-11T08:16:39.677566+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:39.677399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:16:39.677399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:56.767305+0000","scrub_duration":0.0031390630000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":211,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.177849+0000","last_change":"2024-09-11T08:15:48.081690+0000","last_active":"2024-09-11T08:16:37.177849+0000","last_peered":"2024-09-11T08:16:37.177849+0000","last_clean":"2024-09-11T08:16:37.177849+0000","last_became_active":"2024-09-11T08:07:30.046284+0000","last_became_peered":"2024-09-11T08:07:30.046284+0000","last_unstale":"2024-09-11T08:16:37.177849+0000","last_undegraded":"2024-09-11T08:16:37.177849+0000","last_fullsized":"2024-09-11T08:16:37.177849+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:48.081591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:48.081591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:03.932824+0000","scrub_duration":0.0059702619999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":254,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.174264+0000","last_change":"2024-09-11T08:15:53.518868+0000","last_active":"2024-09-11T08:16:37.174264+0000","last_peered":"2024-09-11T08:16:37.174264+0000","last_clean":"2024-09-11T08:16:37.174264+0000","last_became_active":"2024-09-11T08:09:07.878432+0000","last_became_peered":"2024-09-11T08:09:07.878432+0000","last_unstale":"2024-09-11T08:16:37.174264+0000","last_undegraded":"2024-09-11T08:16:37.174264+0000","last_fullsized":"2024-09-11T08:16:37.174264+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-11T08:15:53.518765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:15:53.518765+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:19.946047+0000","scrub_duration":0.0047796499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":207,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:38.974541+0000","last_change":"2024-09-11T08:16:38.974464+0000","last_active":"2024-09-11T08:16:38.974541+0000","last_peered":"2024-09-11T08:16:38.974541+0000","last_clean":"2024-09-11T08:16:38.974541+0000","last_became_active":"2024-09-11T08:07:34.056528+0000","last_became_peered":"2024-09-11T08:07:34.056528+0000","last_unstale":"2024-09-11T08:16:38.974541+0000","last_undegraded":"2024-09-11T08:16:38.974541+0000","last_fullsized":"2024-09-11T08:16:38.974541+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:38.974372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:10:40.110490+0000","last_clean_scrub_stamp":"2024-09-11T08:16:38.974372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:43.756800+0000","scrub_duration":0.014148811000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":150,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.189469+0000","last_change":"2024-09-11T08:15:03.838012+0000","last_active":"2024-09-11T08:16:37.189469+0000","last_peered":"2024-09-11T08:16:37.189469+0000","last_clean":"2024-09-11T08:16:37.189469+0000","last_became_active":"2024-09-11T08:14:24.791610+0000","last_became_peered":"2024-09-11T08:14:24.791610+0000","last_unstale":"2024-09-11T08:16:37.189469+0000","last_undegraded":"2024-09-11T08:16:37.189469+0000","last_fullsized":"2024-09-11T08:16:37.189469+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-11T08:15:03.837911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:15:03.837911+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035450270000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":211,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.177559+0000","last_change":"2024-09-11T08:14:13.134615+0000","last_active":"2024-09-11T08:16:37.177559+0000","last_peered":"2024-09-11T08:16:37.177559+0000","last_clean":"2024-09-11T08:16:37.177559+0000","last_became_active":"2024-09-11T08:07:30.045932+0000","last_became_peered":"2024-09-11T08:07:30.045932+0000","last_unstale":"2024-09-11T08:16:37.177559+0000","last_undegraded":"2024-09-11T08:16:37.177559+0000","last_fullsized":"2024-09-11T08:16:37.177559+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:13.134086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:14:13.134086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0059195080000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":287,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.181406+0000","last_change":"2024-09-11T08:15:56.536927+0000","last_active":"2024-09-11T08:16:37.181406+0000","last_peered":"2024-09-11T08:16:37.181406+0000","last_clean":"2024-09-11T08:16:37.181406+0000","last_became_active":"2024-09-11T08:15:56.515817+0000","last_became_peered":"2024-09-11T08:15:56.515817+0000","last_unstale":"2024-09-11T08:16:37.181406+0000","last_undegraded":"2024-09-11T08:16:37.181406+0000","last_fullsized":"2024-09-11T08:16:37.181406+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-11T08:14:00.393362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:14:00.393362+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018583250000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"209'11","reported_seq":384,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-11T08:16:44.493415+0000","last_change":"2024-09-11T08:16:44.493380+0000","last_active":"2024-09-11T08:16:44.493415+0000","last_peered":"2024-09-11T08:16:44.493415+0000","last_clean":"2024-09-11T08:16:44.493415+0000","last_became_active":"2024-09-11T08:15:56.525498+0000","last_became_peered":"2024-09-11T08:15:56.525498+0000","last_unstale":"2024-09-11T08:16:44.493415+0000","last_undegraded":"2024-09-11T08:16:44.493415+0000","last_fullsized":"2024-09-11T08:16:44.493415+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"209'11","last_scrub_stamp":"2024-09-11T08:16:44.493296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:16:44.493296+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:06.576751+0000","scrub_duration":0.002213058,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"221'12848","reported_seq":12918,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-11T08:16:48.267575+0000","last_change":"2024-09-11T08:13:54.883961+0000","last_active":"2024-09-11T08:16:48.267575+0000","last_peered":"2024-09-11T08:16:48.267575+0000","last_clean":"2024-09-11T08:16:48.267575+0000","last_became_active":"2024-09-11T08:13:54.883685+0000","last_became_peered":"2024-09-11T08:13:54.883685+0000","last_unstale":"2024-09-11T08:16:48.267575+0000","last_undegraded":"2024-09-11T08:16:48.267575+0000","last_fullsized":"2024-09-11T08:16:48.267575+0000","mapping_epoch":159,"log_start":"200'7800","ondisk_log_start":"200'7800","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:13:53.862306+0000","objects_scrubbed":0,"log_size":5048,"log_dups_size":0,"ondisk_log_size":5048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216268800,"num_objects":3300,"num_object_clones":0,"num_object_copies":9900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12848,"num_write_kb":516736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":244,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.182540+0000","last_change":"2024-09-11T08:15:42.534547+0000","last_active":"2024-09-11T08:16:37.182540+0000","last_peered":"2024-09-11T08:16:37.182540+0000","last_clean":"2024-09-11T08:16:37.182540+0000","last_became_active":"2024-09-11T08:07:32.054317+0000","last_became_peered":"2024-09-11T08:07:32.054317+0000","last_unstale":"2024-09-11T08:16:37.182540+0000","last_undegraded":"2024-09-11T08:16:37.182540+0000","last_fullsized":"2024-09-11T08:16:37.182540+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-11T08:15:42.534460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:15:42.534460+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:51.887358+0000","scrub_duration":0.0078017169999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":231,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-11T08:16:42.624704+0000","last_change":"2024-09-11T08:16:42.624557+0000","last_active":"2024-09-11T08:16:42.624704+0000","last_peered":"2024-09-11T08:16:42.624704+0000","last_clean":"2024-09-11T08:16:42.624704+0000","last_became_active":"2024-09-11T08:15:56.521943+0000","last_became_peered":"2024-09-11T08:15:56.521943+0000","last_unstale":"2024-09-11T08:16:42.624704+0000","last_undegraded":"2024-09-11T08:16:42.624704+0000","last_fullsized":"2024-09-11T08:16:42.624704+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:42.624425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:16:42.624425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:05.345719+0000","scrub_duration":0.011367408000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":326,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-11T08:16:44.005158+0000","last_change":"2024-09-11T08:16:44.005131+0000","last_active":"2024-09-11T08:16:44.005158+0000","last_peered":"2024-09-11T08:16:44.005158+0000","last_clean":"2024-09-11T08:16:44.005158+0000","last_became_active":"2024-09-11T08:07:34.052981+0000","last_became_peered":"2024-09-11T08:07:34.052981+0000","last_unstale":"2024-09-11T08:16:44.005158+0000","last_undegraded":"2024-09-11T08:16:44.005158+0000","last_fullsized":"2024-09-11T08:16:44.005158+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:16:44.005077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:16:44.005077+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:04.080032+0000","scrub_duration":0.019154572000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":214,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.183597+0000","last_change":"2024-09-11T08:15:45.411776+0000","last_active":"2024-09-11T08:16:37.183597+0000","last_peered":"2024-09-11T08:16:37.183597+0000","last_clean":"2024-09-11T08:16:37.183597+0000","last_became_active":"2024-09-11T08:07:36.073969+0000","last_became_peered":"2024-09-11T08:07:36.073969+0000","last_unstale":"2024-09-11T08:16:37.183597+0000","last_undegraded":"2024-09-11T08:16:37.183597+0000","last_fullsized":"2024-09-11T08:16:37.183597+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:45.411593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:06.616332+0000","last_clean_scrub_stamp":"2024-09-11T08:15:45.411593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:06.019721+0000","scrub_duration":0.0031499940000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"221'12767","reported_seq":23706,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-11T08:16:48.298752+0000","last_change":"2024-09-11T08:16:35.218904+0000","last_active":"2024-09-11T08:16:48.298752+0000","last_peered":"2024-09-11T08:16:48.298752+0000","last_clean":"2024-09-11T08:16:48.298752+0000","last_became_active":"2024-09-11T08:16:35.162488+0000","last_became_peered":"2024-09-11T08:16:35.162488+0000","last_unstale":"2024-09-11T08:16:48.298752+0000","last_undegraded":"2024-09-11T08:16:48.298752+0000","last_fullsized":"2024-09-11T08:16:48.298752+0000","mapping_epoch":218,"log_start":"195'6832","ondisk_log_start":"195'6832","created":159,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:13:53.862306+0000","objects_scrubbed":0,"log_size":5935,"log_dups_size":0,"ondisk_log_size":5935,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211484672,"num_objects":3227,"num_object_clones":0,"num_object_copies":9681,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":511808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5422,"num_bytes_recovered":355336192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":294,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.174981+0000","last_change":"2024-09-11T08:15:56.542573+0000","last_active":"2024-09-11T08:16:37.174981+0000","last_peered":"2024-09-11T08:16:37.174981+0000","last_clean":"2024-09-11T08:16:37.174981+0000","last_became_active":"2024-09-11T08:15:56.517680+0000","last_became_peered":"2024-09-11T08:15:56.517680+0000","last_unstale":"2024-09-11T08:16:37.174981+0000","last_undegraded":"2024-09-11T08:16:37.174981+0000","last_fullsized":"2024-09-11T08:16:37.174981+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-11T08:13:58.979187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:13:58.979187+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001885091,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":231,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-11T08:16:43.614658+0000","last_change":"2024-09-11T08:16:43.614365+0000","last_active":"2024-09-11T08:16:43.614658+0000","last_peered":"2024-09-11T08:16:43.614658+0000","last_clean":"2024-09-11T08:16:43.614658+0000","last_became_active":"2024-09-11T08:15:56.527438+0000","last_became_peered":"2024-09-11T08:15:56.527438+0000","last_unstale":"2024-09-11T08:16:43.614658+0000","last_undegraded":"2024-09-11T08:16:43.614658+0000","last_fullsized":"2024-09-11T08:16:43.614658+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:43.614153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:16:43.614153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:05.791592+0000","scrub_duration":0.0022087069999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":224,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.177371+0000","last_change":"2024-09-11T08:15:56.522907+0000","last_active":"2024-09-11T08:16:37.177371+0000","last_peered":"2024-09-11T08:16:37.177371+0000","last_clean":"2024-09-11T08:16:37.177371+0000","last_became_active":"2024-09-11T08:15:56.522606+0000","last_became_peered":"2024-09-11T08:15:56.522606+0000","last_unstale":"2024-09-11T08:16:37.177371+0000","last_undegraded":"2024-09-11T08:16:37.177371+0000","last_fullsized":"2024-09-11T08:16:37.177371+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:57.840457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:13:57.840457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014582009999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":195,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.177716+0000","last_change":"2024-09-11T08:14:04.937966+0000","last_active":"2024-09-11T08:16:37.177716+0000","last_peered":"2024-09-11T08:16:37.177716+0000","last_clean":"2024-09-11T08:16:37.177716+0000","last_became_active":"2024-09-11T08:07:36.069225+0000","last_became_peered":"2024-09-11T08:07:36.069225+0000","last_unstale":"2024-09-11T08:16:37.177716+0000","last_undegraded":"2024-09-11T08:16:37.177716+0000","last_fullsized":"2024-09-11T08:16:37.177716+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:04.937884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:04.937884+0000","last_clean_scrub_stamp":"2024-09-11T08:14:04.937884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018777710000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"221'12829","reported_seq":25143,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-11T08:16:47.918838+0000","last_change":"2024-09-11T08:16:25.103862+0000","last_active":"2024-09-11T08:16:47.918838+0000","last_peered":"2024-09-11T08:16:47.918838+0000","last_clean":"2024-09-11T08:16:47.918838+0000","last_became_active":"2024-09-11T08:16:25.088438+0000","last_became_peered":"2024-09-11T08:16:25.088438+0000","last_unstale":"2024-09-11T08:16:47.918838+0000","last_undegraded":"2024-09-11T08:16:47.918838+0000","last_fullsized":"2024-09-11T08:16:47.918838+0000","mapping_epoch":215,"log_start":"198'7500","ondisk_log_start":"198'7500","created":159,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'8832","last_scrub_stamp":"2024-09-11T08:15:51.203402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:15:51.203402+0000","objects_scrubbed":7454,"log_size":5329,"log_dups_size":0,"ondisk_log_size":5329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:06.670584+0000","scrub_duration":2.7862675239999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218103832,"num_objects":3329,"num_object_clones":0,"num_object_copies":9987,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3329,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12829,"num_write_kb":516993,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5955,"num_bytes_recovered":390201368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":220,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.177613+0000","last_change":"2024-09-11T08:14:59.014008+0000","last_active":"2024-09-11T08:16:37.177613+0000","last_peered":"2024-09-11T08:16:37.177613+0000","last_clean":"2024-09-11T08:16:37.177613+0000","last_became_active":"2024-09-11T08:07:32.040069+0000","last_became_peered":"2024-09-11T08:07:32.040069+0000","last_unstale":"2024-09-11T08:16:37.177613+0000","last_undegraded":"2024-09-11T08:16:37.177613+0000","last_fullsized":"2024-09-11T08:16:37.177613+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-11T08:14:59.013949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:14:59.013949+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001898156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":211,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.181623+0000","last_change":"2024-09-11T08:15:54.794370+0000","last_active":"2024-09-11T08:16:37.181623+0000","last_peered":"2024-09-11T08:16:37.181623+0000","last_clean":"2024-09-11T08:16:37.181623+0000","last_became_active":"2024-09-11T08:07:30.050484+0000","last_became_peered":"2024-09-11T08:07:30.050484+0000","last_unstale":"2024-09-11T08:16:37.181623+0000","last_undegraded":"2024-09-11T08:16:37.181623+0000","last_fullsized":"2024-09-11T08:16:37.181623+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:54.793950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:54.793950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:06.106542+0000","scrub_duration":0.0059264870000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"82'9","reported_seq":330,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-11T08:16:39.199621+0000","last_change":"2024-09-11T08:15:51.456046+0000","last_active":"2024-09-11T08:16:39.199621+0000","last_peered":"2024-09-11T08:16:39.199621+0000","last_clean":"2024-09-11T08:16:39.199621+0000","last_became_active":"2024-09-11T08:07:34.056143+0000","last_became_peered":"2024-09-11T08:07:34.056143+0000","last_unstale":"2024-09-11T08:16:39.199621+0000","last_undegraded":"2024-09-11T08:16:39.199621+0000","last_fullsized":"2024-09-11T08:16:39.199621+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-11T08:15:51.455966+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-09-11T08:13:10.230703+0000","last_clean_scrub_stamp":"2024-09-11T08:15:51.455966+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:01.331170+0000","scrub_duration":0.00915995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":185,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.189254+0000","last_change":"2024-09-11T08:15:33.507691+0000","last_active":"2024-09-11T08:16:37.189254+0000","last_peered":"2024-09-11T08:16:37.189254+0000","last_clean":"2024-09-11T08:16:37.189254+0000","last_became_active":"2024-09-11T08:14:24.788968+0000","last_became_peered":"2024-09-11T08:14:24.788968+0000","last_unstale":"2024-09-11T08:16:37.189254+0000","last_undegraded":"2024-09-11T08:16:37.189254+0000","last_fullsized":"2024-09-11T08:16:37.189254+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:33.507613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:15:33.507613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:55.580423+0000","scrub_duration":0.0032985240000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.e","version":"221'12708","reported_seq":12778,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-11T08:16:48.304344+0000","last_change":"2024-09-11T08:13:55.705967+0000","last_active":"2024-09-11T08:16:48.304344+0000","last_peered":"2024-09-11T08:16:48.304344+0000","last_clean":"2024-09-11T08:16:48.304344+0000","last_became_active":"2024-09-11T08:13:55.705670+0000","last_became_peered":"2024-09-11T08:13:55.705670+0000","last_unstale":"2024-09-11T08:16:48.304344+0000","last_undegraded":"2024-09-11T08:16:48.304344+0000","last_fullsized":"2024-09-11T08:16:48.304344+0000","mapping_epoch":159,"log_start":"196'6800","ondisk_log_start":"196'6800","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:13:53.862306+0000","objects_scrubbed":0,"log_size":5908,"log_dups_size":0,"ondisk_log_size":5908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212205568,"num_objects":3238,"num_object_clones":0,"num_object_copies":9714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12708,"num_write_kb":510272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":227,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.175254+0000","last_change":"2024-09-11T08:15:36.365980+0000","last_active":"2024-09-11T08:16:37.175254+0000","last_peered":"2024-09-11T08:16:37.175254+0000","last_clean":"2024-09-11T08:16:37.175254+0000","last_became_active":"2024-09-11T08:07:32.054294+0000","last_became_peered":"2024-09-11T08:07:32.054294+0000","last_unstale":"2024-09-11T08:16:37.175254+0000","last_undegraded":"2024-09-11T08:16:37.175254+0000","last_fullsized":"2024-09-11T08:16:37.175254+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-11T08:15:36.365891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:15:36.365891+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:05.688088+0000","scrub_duration":0.003716071,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":210,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.184098+0000","last_change":"2024-09-11T08:15:46.459225+0000","last_active":"2024-09-11T08:16:37.184098+0000","last_peered":"2024-09-11T08:16:37.184098+0000","last_clean":"2024-09-11T08:16:37.184098+0000","last_became_active":"2024-09-11T08:07:30.047497+0000","last_became_peered":"2024-09-11T08:07:30.047497+0000","last_unstale":"2024-09-11T08:16:37.184098+0000","last_undegraded":"2024-09-11T08:16:37.184098+0000","last_fullsized":"2024-09-11T08:16:37.184098+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:46.459143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:46.459143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:12.469916+0000","scrub_duration":0.0057785019999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"190'9","reported_seq":65,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-11T08:16:39.192652+0000","last_change":"2024-09-11T08:15:08.769903+0000","last_active":"2024-09-11T08:16:39.192652+0000","last_peered":"2024-09-11T08:16:39.192652+0000","last_clean":"2024-09-11T08:16:39.192652+0000","last_became_active":"2024-09-11T08:15:08.764876+0000","last_became_peered":"2024-09-11T08:15:08.764876+0000","last_unstale":"2024-09-11T08:16:39.192652+0000","last_undegraded":"2024-09-11T08:16:39.192652+0000","last_fullsized":"2024-09-11T08:16:39.192652+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:14:11.320174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:14:11.320174+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017353449999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"66'1","reported_seq":196,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.180186+0000","last_change":"2024-09-11T08:15:06.445409+0000","last_active":"2024-09-11T08:16:37.180186+0000","last_peered":"2024-09-11T08:16:37.180186+0000","last_clean":"2024-09-11T08:16:37.180186+0000","last_became_active":"2024-09-11T08:07:36.072593+0000","last_became_peered":"2024-09-11T08:07:36.072593+0000","last_unstale":"2024-09-11T08:16:37.180186+0000","last_undegraded":"2024-09-11T08:16:37.180186+0000","last_fullsized":"2024-09-11T08:16:37.180186+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-11T08:15:06.445301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:15:06.445301+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013971574,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.f","version":"221'12832","reported_seq":12902,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-11T08:16:48.045883+0000","last_change":"2024-09-11T08:13:55.706681+0000","last_active":"2024-09-11T08:16:48.045883+0000","last_peered":"2024-09-11T08:16:48.045883+0000","last_clean":"2024-09-11T08:16:48.045883+0000","last_became_active":"2024-09-11T08:13:55.706134+0000","last_became_peered":"2024-09-11T08:13:55.706134+0000","last_unstale":"2024-09-11T08:16:48.045883+0000","last_undegraded":"2024-09-11T08:16:48.045883+0000","last_fullsized":"2024-09-11T08:16:48.045883+0000","mapping_epoch":159,"log_start":"201'8009","ondisk_log_start":"201'8009","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:13:53.862306+0000","objects_scrubbed":0,"log_size":4823,"log_dups_size":0,"ondisk_log_size":4823,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212860928,"num_objects":3248,"num_object_clones":0,"num_object_copies":9744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12832,"num_write_kb":514560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":298,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.181176+0000","last_change":"2024-09-11T08:15:56.540181+0000","last_active":"2024-09-11T08:16:37.181176+0000","last_peered":"2024-09-11T08:16:37.181176+0000","last_clean":"2024-09-11T08:16:37.181176+0000","last_became_active":"2024-09-11T08:15:56.518384+0000","last_became_peered":"2024-09-11T08:15:56.518384+0000","last_unstale":"2024-09-11T08:16:37.181176+0000","last_undegraded":"2024-09-11T08:16:37.181176+0000","last_fullsized":"2024-09-11T08:16:37.181176+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-11T08:13:57.387660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:13:57.387660+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025295550000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":221,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.174810+0000","last_change":"2024-09-11T08:15:54.517507+0000","last_active":"2024-09-11T08:16:37.174810+0000","last_peered":"2024-09-11T08:16:37.174810+0000","last_clean":"2024-09-11T08:16:37.174810+0000","last_became_active":"2024-09-11T08:07:30.045380+0000","last_became_peered":"2024-09-11T08:07:30.045380+0000","last_unstale":"2024-09-11T08:16:37.174810+0000","last_undegraded":"2024-09-11T08:16:37.174810+0000","last_fullsized":"2024-09-11T08:16:37.174810+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:54.517429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:54.517429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:05.165550+0000","scrub_duration":0.0055168639999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":216,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.181492+0000","last_change":"2024-09-11T08:15:54.238757+0000","last_active":"2024-09-11T08:16:37.181492+0000","last_peered":"2024-09-11T08:16:37.181492+0000","last_clean":"2024-09-11T08:16:37.181492+0000","last_became_active":"2024-09-11T08:07:34.051004+0000","last_became_peered":"2024-09-11T08:07:34.051004+0000","last_unstale":"2024-09-11T08:16:37.181492+0000","last_undegraded":"2024-09-11T08:16:37.181492+0000","last_fullsized":"2024-09-11T08:16:37.181492+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:54.238680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:15:54.238680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:58.812475+0000","scrub_duration":0.0019626420000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":205,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.177379+0000","last_change":"2024-09-11T08:14:09.068749+0000","last_active":"2024-09-11T08:16:37.177379+0000","last_peered":"2024-09-11T08:16:37.177379+0000","last_clean":"2024-09-11T08:16:37.177379+0000","last_became_active":"2024-09-11T08:07:36.631668+0000","last_became_peered":"2024-09-11T08:07:36.631668+0000","last_unstale":"2024-09-11T08:16:37.177379+0000","last_undegraded":"2024-09-11T08:16:37.177379+0000","last_fullsized":"2024-09-11T08:16:37.177379+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:09.068694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:14:09.068694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033190699999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"221'13068","reported_seq":17709,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-11T08:16:47.904133+0000","last_change":"2024-09-11T08:15:56.003569+0000","last_active":"2024-09-11T08:16:47.904133+0000","last_peered":"2024-09-11T08:16:47.904133+0000","last_clean":"2024-09-11T08:16:47.904133+0000","last_became_active":"2024-09-11T08:14:50.875690+0000","last_became_peered":"2024-09-11T08:14:50.875690+0000","last_unstale":"2024-09-11T08:16:47.904133+0000","last_undegraded":"2024-09-11T08:16:47.904133+0000","last_fullsized":"2024-09-11T08:16:47.904133+0000","mapping_epoch":179,"log_start":"199'7800","ondisk_log_start":"199'7800","created":159,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'9330","last_scrub_stamp":"2024-09-11T08:15:56.003501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:15:56.003501+0000","objects_scrubbed":7010,"log_size":5268,"log_dups_size":0,"ondisk_log_size":5268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:14.010825+0000","scrub_duration":1.6004849619999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212467712,"num_objects":3242,"num_object_clones":0,"num_object_copies":9726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13068,"num_write_kb":521920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2130,"num_bytes_recovered":139591680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"190'39","reported_seq":93,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-11T08:16:39.195635+0000","last_change":"2024-09-11T08:15:52.848067+0000","last_active":"2024-09-11T08:16:39.195635+0000","last_peered":"2024-09-11T08:16:39.195635+0000","last_clean":"2024-09-11T08:16:39.195635+0000","last_became_active":"2024-09-11T08:15:08.775181+0000","last_became_peered":"2024-09-11T08:15:08.775181+0000","last_unstale":"2024-09-11T08:16:39.195635+0000","last_undegraded":"2024-09-11T08:16:39.195635+0000","last_fullsized":"2024-09-11T08:16:39.195635+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'39","last_scrub_stamp":"2024-09-11T08:15:52.847968+0000","last_deep_scrub":"190'39","last_deep_scrub_stamp":"2024-09-11T08:15:52.847968+0000","last_clean_scrub_stamp":"2024-09-11T08:15:52.847968+0000","objects_scrubbed":10,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:11.824582+0000","scrub_duration":0.012280474,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"59'1","reported_seq":220,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.178204+0000","last_change":"2024-09-11T08:15:44.992430+0000","last_active":"2024-09-11T08:16:37.178204+0000","last_peered":"2024-09-11T08:16:37.178204+0000","last_clean":"2024-09-11T08:16:37.178204+0000","last_became_active":"2024-09-11T08:07:30.047513+0000","last_became_peered":"2024-09-11T08:07:30.047513+0000","last_unstale":"2024-09-11T08:16:37.178204+0000","last_undegraded":"2024-09-11T08:16:37.178204+0000","last_fullsized":"2024-09-11T08:16:37.178204+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-11T08:15:44.992356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:44.992356+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:00.977258+0000","scrub_duration":0.0049464690000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":217,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.184728+0000","last_change":"2024-09-11T08:14:57.417159+0000","last_active":"2024-09-11T08:16:37.184728+0000","last_peered":"2024-09-11T08:16:37.184728+0000","last_clean":"2024-09-11T08:16:37.184728+0000","last_became_active":"2024-09-11T08:07:30.043130+0000","last_became_peered":"2024-09-11T08:07:30.043130+0000","last_unstale":"2024-09-11T08:16:37.184728+0000","last_undegraded":"2024-09-11T08:16:37.184728+0000","last_fullsized":"2024-09-11T08:16:37.184728+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-11T08:14:57.417078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:14:57.417078+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026993970000000001,"objects_trimmed":0,"snaptrim_duration":0.0026120850000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":207,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-11T08:16:42.666165+0000","last_change":"2024-09-11T08:16:42.666121+0000","last_active":"2024-09-11T08:16:42.666165+0000","last_peered":"2024-09-11T08:16:42.666165+0000","last_clean":"2024-09-11T08:16:42.666165+0000","last_became_active":"2024-09-11T08:07:34.057154+0000","last_became_peered":"2024-09-11T08:07:34.057154+0000","last_unstale":"2024-09-11T08:16:42.666165+0000","last_undegraded":"2024-09-11T08:16:42.666165+0000","last_fullsized":"2024-09-11T08:16:42.666165+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:42.666035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:10:37.957096+0000","last_clean_scrub_stamp":"2024-09-11T08:16:42.666035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:11.328001+0000","scrub_duration":0.0077723870000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":222,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.174589+0000","last_change":"2024-09-11T08:15:56.527473+0000","last_active":"2024-09-11T08:16:37.174589+0000","last_peered":"2024-09-11T08:16:37.174589+0000","last_clean":"2024-09-11T08:16:37.174589+0000","last_became_active":"2024-09-11T08:15:56.525099+0000","last_became_peered":"2024-09-11T08:15:56.525099+0000","last_unstale":"2024-09-11T08:16:37.174589+0000","last_undegraded":"2024-09-11T08:16:37.174589+0000","last_fullsized":"2024-09-11T08:16:37.174589+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:05.076884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:14:05.076884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001986065,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"221'12619","reported_seq":12950,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-11T08:16:48.096367+0000","last_change":"2024-09-11T08:16:05.402156+0000","last_active":"2024-09-11T08:16:48.096367+0000","last_peered":"2024-09-11T08:16:48.096367+0000","last_clean":"2024-09-11T08:16:48.096367+0000","last_became_active":"2024-09-11T08:13:54.882290+0000","last_became_peered":"2024-09-11T08:13:54.882290+0000","last_unstale":"2024-09-11T08:16:48.096367+0000","last_undegraded":"2024-09-11T08:16:48.096367+0000","last_fullsized":"2024-09-11T08:16:48.096367+0000","mapping_epoch":159,"log_start":"201'7805","ondisk_log_start":"201'7805","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'9635","last_scrub_stamp":"2024-09-11T08:16:05.401974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:16:05.401974+0000","objects_scrubbed":6255,"log_size":4814,"log_dups_size":0,"ondisk_log_size":4814,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:26.177859+0000","scrub_duration":2.9696664940000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206503936,"num_objects":3151,"num_object_clones":0,"num_object_copies":9453,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12619,"num_write_kb":504640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":249,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-11T08:16:41.514990+0000","last_change":"2024-09-11T08:16:41.514946+0000","last_active":"2024-09-11T08:16:41.514990+0000","last_peered":"2024-09-11T08:16:41.514990+0000","last_clean":"2024-09-11T08:16:41.514990+0000","last_became_active":"2024-09-11T08:07:32.050673+0000","last_became_peered":"2024-09-11T08:07:32.050673+0000","last_unstale":"2024-09-11T08:16:41.514990+0000","last_undegraded":"2024-09-11T08:16:41.514990+0000","last_fullsized":"2024-09-11T08:16:41.514990+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'13","last_scrub_stamp":"2024-09-11T08:16:41.514853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:16:41.514853+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:50.626022+0000","scrub_duration":0.0093017910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":214,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.175905+0000","last_change":"2024-09-11T08:15:31.273498+0000","last_active":"2024-09-11T08:16:37.175905+0000","last_peered":"2024-09-11T08:16:37.175905+0000","last_clean":"2024-09-11T08:16:37.175905+0000","last_became_active":"2024-09-11T08:07:30.047014+0000","last_became_peered":"2024-09-11T08:07:30.047014+0000","last_unstale":"2024-09-11T08:16:37.175905+0000","last_undegraded":"2024-09-11T08:16:37.175905+0000","last_fullsized":"2024-09-11T08:16:37.175905+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:31.273415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:31.273415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:59.181535+0000","scrub_duration":0.0057528060000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"127'2","reported_seq":240,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-11T08:16:40.021316+0000","last_change":"2024-09-11T08:16:40.021161+0000","last_active":"2024-09-11T08:16:40.021316+0000","last_peered":"2024-09-11T08:16:40.021316+0000","last_clean":"2024-09-11T08:16:40.021316+0000","last_became_active":"2024-09-11T08:07:30.045009+0000","last_became_peered":"2024-09-11T08:07:30.045009+0000","last_unstale":"2024-09-11T08:16:40.021316+0000","last_undegraded":"2024-09-11T08:16:40.021316+0000","last_fullsized":"2024-09-11T08:16:40.021316+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'2","last_scrub_stamp":"2024-09-11T08:16:40.021082+0000","last_deep_scrub":"127'2","last_deep_scrub_stamp":"2024-09-11T08:15:01.938497+0000","last_clean_scrub_stamp":"2024-09-11T08:16:40.021082+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:50.557847+0000","scrub_duration":0.002170426,"objects_trimmed":0,"snaptrim_duration":0.001859533,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":226,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.189678+0000","last_change":"2024-09-11T08:14:58.889181+0000","last_active":"2024-09-11T08:16:37.189678+0000","last_peered":"2024-09-11T08:16:37.189678+0000","last_clean":"2024-09-11T08:16:37.189678+0000","last_became_active":"2024-09-11T08:14:24.792279+0000","last_became_peered":"2024-09-11T08:14:24.792279+0000","last_unstale":"2024-09-11T08:16:37.189678+0000","last_undegraded":"2024-09-11T08:16:37.189678+0000","last_fullsized":"2024-09-11T08:16:37.189678+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:58.889094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:58.889094+0000","last_clean_scrub_stamp":"2024-09-11T08:14:58.889094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040282030000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":141,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.189548+0000","last_change":"2024-09-11T08:14:58.796274+0000","last_active":"2024-09-11T08:16:37.189548+0000","last_peered":"2024-09-11T08:16:37.189548+0000","last_clean":"2024-09-11T08:16:37.189548+0000","last_became_active":"2024-09-11T08:14:24.792847+0000","last_became_peered":"2024-09-11T08:14:24.792847+0000","last_unstale":"2024-09-11T08:16:37.189548+0000","last_undegraded":"2024-09-11T08:16:37.189548+0000","last_fullsized":"2024-09-11T08:16:37.189548+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:58.796217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:58.796217+0000","last_clean_scrub_stamp":"2024-09-11T08:14:58.796217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035819979999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"221'12776","reported_seq":12993,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-11T08:16:48.277292+0000","last_change":"2024-09-11T08:16:46.178629+0000","last_active":"2024-09-11T08:16:48.277292+0000","last_peered":"2024-09-11T08:16:48.277292+0000","last_clean":"2024-09-11T08:16:48.277292+0000","last_became_active":"2024-09-11T08:13:55.707556+0000","last_became_peered":"2024-09-11T08:13:55.707556+0000","last_unstale":"2024-09-11T08:16:48.277292+0000","last_undegraded":"2024-09-11T08:16:48.277292+0000","last_fullsized":"2024-09-11T08:16:48.277292+0000","mapping_epoch":159,"log_start":"200'7700","ondisk_log_start":"200'7700","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"221'12598","last_scrub_stamp":"2024-09-11T08:16:46.177312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:16:46.177312+0000","objects_scrubbed":3466,"log_size":5076,"log_dups_size":0,"ondisk_log_size":5076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:15.364017+0000","scrub_duration":1.6312657339999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211943424,"num_objects":3234,"num_object_clones":0,"num_object_copies":9702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12776,"num_write_kb":512320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":227,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.181474+0000","last_change":"2024-09-11T08:15:53.800579+0000","last_active":"2024-09-11T08:16:37.181474+0000","last_peered":"2024-09-11T08:16:37.181474+0000","last_clean":"2024-09-11T08:16:37.181474+0000","last_became_active":"2024-09-11T08:07:32.055407+0000","last_became_peered":"2024-09-11T08:07:32.055407+0000","last_unstale":"2024-09-11T08:16:37.181474+0000","last_undegraded":"2024-09-11T08:16:37.181474+0000","last_fullsized":"2024-09-11T08:16:37.181474+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-11T08:15:53.800494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:15:53.800494+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:54.617297+0000","scrub_duration":0.0070921359999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"59'1","reported_seq":218,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.181341+0000","last_change":"2024-09-11T08:15:51.807218+0000","last_active":"2024-09-11T08:16:37.181341+0000","last_peered":"2024-09-11T08:16:37.181341+0000","last_clean":"2024-09-11T08:16:37.181341+0000","last_became_active":"2024-09-11T08:07:30.050504+0000","last_became_peered":"2024-09-11T08:07:30.050504+0000","last_unstale":"2024-09-11T08:16:37.181341+0000","last_undegraded":"2024-09-11T08:16:37.181341+0000","last_fullsized":"2024-09-11T08:16:37.181341+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-11T08:15:51.806819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:51.806819+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:59.379579+0000","scrub_duration":0.012826908,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":545,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-11T08:16:46.106773+0000","last_change":"2024-09-11T08:15:04.793314+0000","last_active":"2024-09-11T08:16:46.106773+0000","last_peered":"2024-09-11T08:16:46.106773+0000","last_clean":"2024-09-11T08:16:46.106773+0000","last_became_active":"2024-09-11T08:14:24.793048+0000","last_became_peered":"2024-09-11T08:14:24.793048+0000","last_unstale":"2024-09-11T08:16:46.106773+0000","last_undegraded":"2024-09-11T08:16:46.106773+0000","last_fullsized":"2024-09-11T08:16:46.106773+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-11T08:15:04.793062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:04.793062+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0082004639999999993,"objects_trimmed":0,"snaptrim_duration":0.000318821,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":149,"num_read_kb":144,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":216,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.787116+0000","last_change":"2024-09-11T08:16:37.787055+0000","last_active":"2024-09-11T08:16:37.787116+0000","last_peered":"2024-09-11T08:16:37.787116+0000","last_clean":"2024-09-11T08:16:37.787116+0000","last_became_active":"2024-09-11T08:07:34.058962+0000","last_became_peered":"2024-09-11T08:07:34.058962+0000","last_unstale":"2024-09-11T08:16:37.787116+0000","last_undegraded":"2024-09-11T08:16:37.787116+0000","last_fullsized":"2024-09-11T08:16:37.787116+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:37.786977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:16:37.786977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:48.840372+0000","scrub_duration":0.010499725999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":215,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.174689+0000","last_change":"2024-09-11T08:15:53.506655+0000","last_active":"2024-09-11T08:16:37.174689+0000","last_peered":"2024-09-11T08:16:37.174689+0000","last_clean":"2024-09-11T08:16:37.174689+0000","last_became_active":"2024-09-11T08:07:36.077597+0000","last_became_peered":"2024-09-11T08:07:36.077597+0000","last_unstale":"2024-09-11T08:16:37.174689+0000","last_undegraded":"2024-09-11T08:16:37.174689+0000","last_fullsized":"2024-09-11T08:16:37.174689+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:53.506581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:15:53.506581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:15.129440+0000","scrub_duration":0.0028717209999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"221'12777","reported_seq":13164,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-11T08:16:47.840636+0000","last_change":"2024-09-11T08:15:47.174817+0000","last_active":"2024-09-11T08:16:47.840636+0000","last_peered":"2024-09-11T08:16:47.840636+0000","last_clean":"2024-09-11T08:16:47.840636+0000","last_became_active":"2024-09-11T08:13:54.882405+0000","last_became_peered":"2024-09-11T08:13:54.882405+0000","last_unstale":"2024-09-11T08:16:47.840636+0000","last_undegraded":"2024-09-11T08:16:47.840636+0000","last_fullsized":"2024-09-11T08:16:47.840636+0000","mapping_epoch":159,"log_start":"195'6700","ondisk_log_start":"195'6700","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'8519","last_scrub_stamp":"2024-09-11T08:15:47.174735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:15:47.174735+0000","objects_scrubbed":7692,"log_size":6077,"log_dups_size":0,"ondisk_log_size":6077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:12.130102+0000","scrub_duration":2.9660035420000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216596480,"num_objects":3305,"num_object_clones":0,"num_object_copies":9915,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12777,"num_write_kb":514624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"84'21","reported_seq":264,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-11T08:16:40.472801+0000","last_change":"2024-09-11T08:16:40.472710+0000","last_active":"2024-09-11T08:16:40.472801+0000","last_peered":"2024-09-11T08:16:40.472801+0000","last_clean":"2024-09-11T08:16:40.472801+0000","last_became_active":"2024-09-11T08:14:24.789154+0000","last_became_peered":"2024-09-11T08:14:24.789154+0000","last_unstale":"2024-09-11T08:16:40.472801+0000","last_undegraded":"2024-09-11T08:16:40.472801+0000","last_fullsized":"2024-09-11T08:16:40.472801+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'21","last_scrub_stamp":"2024-09-11T08:16:40.472618+0000","last_deep_scrub":"84'21","last_deep_scrub_stamp":"2024-09-11T08:08:56.705306+0000","last_clean_scrub_stamp":"2024-09-11T08:16:40.472618+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:00.262854+0000","scrub_duration":0.0099327200000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":201,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.174133+0000","last_change":"2024-09-11T08:15:06.236392+0000","last_active":"2024-09-11T08:16:37.174133+0000","last_peered":"2024-09-11T08:16:37.174133+0000","last_clean":"2024-09-11T08:16:37.174133+0000","last_became_active":"2024-09-11T08:07:30.047246+0000","last_became_peered":"2024-09-11T08:07:30.047246+0000","last_unstale":"2024-09-11T08:16:37.174133+0000","last_undegraded":"2024-09-11T08:16:37.174133+0000","last_fullsized":"2024-09-11T08:16:37.174133+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:06.236119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:06.236119+0000","last_clean_scrub_stamp":"2024-09-11T08:15:06.236119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029121939999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":39,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-11T08:16:45.611896+0000","last_change":"2024-09-11T08:16:45.611831+0000","last_active":"2024-09-11T08:16:45.611896+0000","last_peered":"2024-09-11T08:16:45.611896+0000","last_clean":"2024-09-11T08:16:45.611896+0000","last_became_active":"2024-09-11T08:15:56.523430+0000","last_became_peered":"2024-09-11T08:15:56.523430+0000","last_unstale":"2024-09-11T08:16:45.611896+0000","last_undegraded":"2024-09-11T08:16:45.611896+0000","last_fullsized":"2024-09-11T08:16:45.611896+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-11T08:16:45.611772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:05:21.231856+0000","last_clean_scrub_stamp":"2024-09-11T08:16:45.611772+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:06.875825+0000","scrub_duration":0.0021764710000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":23,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.189295+0000","last_change":"2024-09-11T08:15:56.529159+0000","last_active":"2024-09-11T08:16:37.189295+0000","last_peered":"2024-09-11T08:16:37.189295+0000","last_clean":"2024-09-11T08:16:37.189295+0000","last_became_active":"2024-09-11T08:15:56.528641+0000","last_became_peered":"2024-09-11T08:15:56.528641+0000","last_unstale":"2024-09-11T08:16:37.189295+0000","last_undegraded":"2024-09-11T08:16:37.189295+0000","last_fullsized":"2024-09-11T08:16:37.189295+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:55.140137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:51.289872+0000","last_clean_scrub_stamp":"2024-09-11T08:15:55.140137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:57.600999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":214,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-11T08:16:43.782553+0000","last_change":"2024-09-11T08:16:43.782518+0000","last_active":"2024-09-11T08:16:43.782553+0000","last_peered":"2024-09-11T08:16:43.782553+0000","last_clean":"2024-09-11T08:16:43.782553+0000","last_became_active":"2024-09-11T08:07:36.072508+0000","last_became_peered":"2024-09-11T08:07:36.072508+0000","last_unstale":"2024-09-11T08:16:43.782553+0000","last_undegraded":"2024-09-11T08:16:43.782553+0000","last_fullsized":"2024-09-11T08:16:43.782553+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:43.782439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:01.777429+0000","last_clean_scrub_stamp":"2024-09-11T08:16:43.782439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:49.290220+0000","scrub_duration":0.0038441999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"221'12789","reported_seq":12859,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-11T08:16:47.762836+0000","last_change":"2024-09-11T08:13:54.881747+0000","last_active":"2024-09-11T08:16:47.762836+0000","last_peered":"2024-09-11T08:16:47.762836+0000","last_clean":"2024-09-11T08:16:47.762836+0000","last_became_active":"2024-09-11T08:13:54.881423+0000","last_became_peered":"2024-09-11T08:13:54.881423+0000","last_unstale":"2024-09-11T08:16:47.762836+0000","last_undegraded":"2024-09-11T08:16:47.762836+0000","last_fullsized":"2024-09-11T08:16:47.762836+0000","mapping_epoch":159,"log_start":"201'8000","ondisk_log_start":"201'8000","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:13:53.862306+0000","objects_scrubbed":0,"log_size":4789,"log_dups_size":0,"ondisk_log_size":4789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215547904,"num_objects":3289,"num_object_clones":0,"num_object_copies":9867,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12789,"num_write_kb":514496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"88'16","reported_seq":233,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-11T08:16:46.015918+0000","last_change":"2024-09-11T08:16:46.015835+0000","last_active":"2024-09-11T08:16:46.015918+0000","last_peered":"2024-09-11T08:16:46.015918+0000","last_clean":"2024-09-11T08:16:46.015918+0000","last_became_active":"2024-09-11T08:07:32.047971+0000","last_became_peered":"2024-09-11T08:07:32.047971+0000","last_unstale":"2024-09-11T08:16:46.015918+0000","last_undegraded":"2024-09-11T08:16:46.015918+0000","last_fullsized":"2024-09-11T08:16:46.015918+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'16","last_scrub_stamp":"2024-09-11T08:16:46.015776+0000","last_deep_scrub":"88'16","last_deep_scrub_stamp":"2024-09-11T08:09:39.635984+0000","last_clean_scrub_stamp":"2024-09-11T08:16:46.015776+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:03.636623+0000","scrub_duration":0.0030649929999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":218,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.177718+0000","last_change":"2024-09-11T08:14:54.023658+0000","last_active":"2024-09-11T08:16:37.177718+0000","last_peered":"2024-09-11T08:16:37.177718+0000","last_clean":"2024-09-11T08:16:37.177718+0000","last_became_active":"2024-09-11T08:14:24.789888+0000","last_became_peered":"2024-09-11T08:14:24.789888+0000","last_unstale":"2024-09-11T08:16:37.177718+0000","last_undegraded":"2024-09-11T08:16:37.177718+0000","last_fullsized":"2024-09-11T08:16:37.177718+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:54.023556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:36.417911+0000","last_clean_scrub_stamp":"2024-09-11T08:14:54.023556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002726596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":206,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.183888+0000","last_change":"2024-09-11T08:15:06.434641+0000","last_active":"2024-09-11T08:16:37.183888+0000","last_peered":"2024-09-11T08:16:37.183888+0000","last_clean":"2024-09-11T08:16:37.183888+0000","last_became_active":"2024-09-11T08:07:34.053347+0000","last_became_peered":"2024-09-11T08:07:34.053347+0000","last_unstale":"2024-09-11T08:16:37.183888+0000","last_undegraded":"2024-09-11T08:16:37.183888+0000","last_fullsized":"2024-09-11T08:16:37.183888+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:06.434588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:01.708166+0000","last_clean_scrub_stamp":"2024-09-11T08:15:06.434588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039794419999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":195,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.174875+0000","last_change":"2024-09-11T08:14:57.236619+0000","last_active":"2024-09-11T08:16:37.174875+0000","last_peered":"2024-09-11T08:16:37.174875+0000","last_clean":"2024-09-11T08:16:37.174875+0000","last_became_active":"2024-09-11T08:07:36.631687+0000","last_became_peered":"2024-09-11T08:07:36.631687+0000","last_unstale":"2024-09-11T08:16:37.174875+0000","last_undegraded":"2024-09-11T08:16:37.174875+0000","last_fullsized":"2024-09-11T08:16:37.174875+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:57.236432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:14:57.236432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029326859999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1","version":"221'12745","reported_seq":13098,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-11T08:16:48.227025+0000","last_change":"2024-09-11T08:15:57.075945+0000","last_active":"2024-09-11T08:16:48.227025+0000","last_peered":"2024-09-11T08:16:48.227025+0000","last_clean":"2024-09-11T08:16:48.227025+0000","last_became_active":"2024-09-11T08:13:55.707503+0000","last_became_peered":"2024-09-11T08:13:55.707503+0000","last_unstale":"2024-09-11T08:16:48.227025+0000","last_undegraded":"2024-09-11T08:16:48.227025+0000","last_fullsized":"2024-09-11T08:16:48.227025+0000","mapping_epoch":159,"log_start":"199'7700","ondisk_log_start":"199'7700","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'9162","last_scrub_stamp":"2024-09-11T08:15:57.075870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:15:57.075870+0000","objects_scrubbed":6840,"log_size":5045,"log_dups_size":0,"ondisk_log_size":5045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:10.709887+0000","scrub_duration":0.63951687599999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213450752,"num_objects":3257,"num_object_clones":0,"num_object_copies":9771,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12745,"num_write_kb":512064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"77'25","reported_seq":273,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.179853+0000","last_change":"2024-09-11T08:15:42.061711+0000","last_active":"2024-09-11T08:16:37.179853+0000","last_peered":"2024-09-11T08:16:37.179853+0000","last_clean":"2024-09-11T08:16:37.179853+0000","last_became_active":"2024-09-11T08:09:08.106771+0000","last_became_peered":"2024-09-11T08:09:08.106771+0000","last_unstale":"2024-09-11T08:16:37.179853+0000","last_undegraded":"2024-09-11T08:16:37.179853+0000","last_fullsized":"2024-09-11T08:16:37.179853+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'25","last_scrub_stamp":"2024-09-11T08:15:42.061632+0000","last_deep_scrub":"77'25","last_deep_scrub_stamp":"2024-09-11T08:08:32.458210+0000","last_clean_scrub_stamp":"2024-09-11T08:15:42.061632+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:06.943716+0000","scrub_duration":0.00500454,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":220,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-11T08:16:41.507063+0000","last_change":"2024-09-11T08:16:41.507028+0000","last_active":"2024-09-11T08:16:41.507063+0000","last_peered":"2024-09-11T08:16:41.507063+0000","last_clean":"2024-09-11T08:16:41.507063+0000","last_became_active":"2024-09-11T08:07:30.042555+0000","last_became_peered":"2024-09-11T08:07:30.042555+0000","last_unstale":"2024-09-11T08:16:41.507063+0000","last_undegraded":"2024-09-11T08:16:41.507063+0000","last_fullsized":"2024-09-11T08:16:41.507063+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:41.506939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:16:41.506939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:53.258601+0000","scrub_duration":0.0026028340000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":207,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.173643+0000","last_change":"2024-09-11T08:14:00.985127+0000","last_active":"2024-09-11T08:16:37.173643+0000","last_peered":"2024-09-11T08:16:37.173643+0000","last_clean":"2024-09-11T08:16:37.173643+0000","last_became_active":"2024-09-11T08:07:34.054961+0000","last_became_peered":"2024-09-11T08:07:34.054961+0000","last_unstale":"2024-09-11T08:16:37.173643+0000","last_undegraded":"2024-09-11T08:16:37.173643+0000","last_fullsized":"2024-09-11T08:16:37.173643+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:00.985075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:14:00.985075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015093109999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":224,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.185753+0000","last_change":"2024-09-11T08:15:43.160119+0000","last_active":"2024-09-11T08:16:37.185753+0000","last_peered":"2024-09-11T08:16:37.185753+0000","last_clean":"2024-09-11T08:16:37.185753+0000","last_became_active":"2024-09-11T08:07:36.071194+0000","last_became_peered":"2024-09-11T08:07:36.071194+0000","last_unstale":"2024-09-11T08:16:37.185753+0000","last_undegraded":"2024-09-11T08:16:37.185753+0000","last_fullsized":"2024-09-11T08:16:37.185753+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:43.158812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:07.930137+0000","last_clean_scrub_stamp":"2024-09-11T08:15:43.158812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:00.399779+0000","scrub_duration":0.0051752609999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"221'12902","reported_seq":12971,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-11T08:16:48.081936+0000","last_change":"2024-09-11T08:13:54.882472+0000","last_active":"2024-09-11T08:16:48.081936+0000","last_peered":"2024-09-11T08:16:48.081936+0000","last_clean":"2024-09-11T08:16:48.081936+0000","last_became_active":"2024-09-11T08:13:54.882196+0000","last_became_peered":"2024-09-11T08:13:54.882196+0000","last_unstale":"2024-09-11T08:16:48.081936+0000","last_undegraded":"2024-09-11T08:16:48.081936+0000","last_fullsized":"2024-09-11T08:16:48.081936+0000","mapping_epoch":159,"log_start":"201'8135","ondisk_log_start":"201'8135","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:13:53.862306+0000","objects_scrubbed":0,"log_size":4767,"log_dups_size":0,"ondisk_log_size":4767,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210239488,"num_objects":3208,"num_object_clones":0,"num_object_copies":9624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12902,"num_write_kb":515520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":214,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.178079+0000","last_change":"2024-09-11T08:13:07.139492+0000","last_active":"2024-09-11T08:16:37.178079+0000","last_peered":"2024-09-11T08:16:37.178079+0000","last_clean":"2024-09-11T08:16:37.178079+0000","last_became_active":"2024-09-11T08:07:32.043314+0000","last_became_peered":"2024-09-11T08:07:32.043314+0000","last_unstale":"2024-09-11T08:16:37.178079+0000","last_undegraded":"2024-09-11T08:16:37.178079+0000","last_fullsized":"2024-09-11T08:16:37.178079+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-11T08:13:07.139430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:13:07.139430+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0075406340000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":211,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.175009+0000","last_change":"2024-09-11T08:15:48.405028+0000","last_active":"2024-09-11T08:16:37.175009+0000","last_peered":"2024-09-11T08:16:37.175009+0000","last_clean":"2024-09-11T08:16:37.175009+0000","last_became_active":"2024-09-11T08:07:30.046684+0000","last_became_peered":"2024-09-11T08:07:30.046684+0000","last_unstale":"2024-09-11T08:16:37.175009+0000","last_undegraded":"2024-09-11T08:16:37.175009+0000","last_fullsized":"2024-09-11T08:16:37.175009+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:48.404853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:48.404853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:01.459915+0000","scrub_duration":0.0048699770000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"171'12","reported_seq":349,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-11T08:16:39.198903+0000","last_change":"2024-09-11T08:14:56.881397+0000","last_active":"2024-09-11T08:16:39.198903+0000","last_peered":"2024-09-11T08:16:39.198903+0000","last_clean":"2024-09-11T08:16:39.198903+0000","last_became_active":"2024-09-11T08:14:24.794446+0000","last_became_peered":"2024-09-11T08:14:24.794446+0000","last_unstale":"2024-09-11T08:16:39.198903+0000","last_undegraded":"2024-09-11T08:16:39.198903+0000","last_fullsized":"2024-09-11T08:16:39.198903+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'12","last_scrub_stamp":"2024-09-11T08:14:56.881299+0000","last_deep_scrub":"171'12","last_deep_scrub_stamp":"2024-09-11T08:14:56.881299+0000","last_clean_scrub_stamp":"2024-09-11T08:14:56.881299+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0073519259999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":195,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.178081+0000","last_change":"2024-09-11T08:14:13.941569+0000","last_active":"2024-09-11T08:16:37.178081+0000","last_peered":"2024-09-11T08:16:37.178081+0000","last_clean":"2024-09-11T08:16:37.178081+0000","last_became_active":"2024-09-11T08:07:36.074684+0000","last_became_peered":"2024-09-11T08:07:36.074684+0000","last_unstale":"2024-09-11T08:16:37.178081+0000","last_undegraded":"2024-09-11T08:16:37.178081+0000","last_fullsized":"2024-09-11T08:16:37.178081+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:13.941492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:14.613532+0000","last_clean_scrub_stamp":"2024-09-11T08:14:13.941492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037661470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.7","version":"221'12845","reported_seq":13313,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-11T08:16:48.270819+0000","last_change":"2024-09-11T08:15:56.302099+0000","last_active":"2024-09-11T08:16:48.270819+0000","last_peered":"2024-09-11T08:16:48.270819+0000","last_clean":"2024-09-11T08:16:48.270819+0000","last_became_active":"2024-09-11T08:13:54.883624+0000","last_became_peered":"2024-09-11T08:13:54.883624+0000","last_unstale":"2024-09-11T08:16:48.270819+0000","last_undegraded":"2024-09-11T08:16:48.270819+0000","last_fullsized":"2024-09-11T08:16:48.270819+0000","mapping_epoch":159,"log_start":"200'7800","ondisk_log_start":"200'7800","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'9119","last_scrub_stamp":"2024-09-11T08:15:56.302032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:15:56.302032+0000","objects_scrubbed":6919,"log_size":5045,"log_dups_size":0,"ondisk_log_size":5045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:02.822566+0000","scrub_duration":0.82679451100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209256448,"num_objects":3193,"num_object_clones":0,"num_object_copies":9579,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12845,"num_write_kb":513216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":215,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.181096+0000","last_change":"2024-09-11T08:13:14.171710+0000","last_active":"2024-09-11T08:16:37.181096+0000","last_peered":"2024-09-11T08:16:37.181096+0000","last_clean":"2024-09-11T08:16:37.181096+0000","last_became_active":"2024-09-11T08:07:32.054885+0000","last_became_peered":"2024-09-11T08:07:32.054885+0000","last_unstale":"2024-09-11T08:16:37.181096+0000","last_undegraded":"2024-09-11T08:16:37.181096+0000","last_fullsized":"2024-09-11T08:16:37.181096+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:13:14.171623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:13:14.171623+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.01042096,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":220,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.185750+0000","last_change":"2024-09-11T08:15:34.497175+0000","last_active":"2024-09-11T08:16:37.185750+0000","last_peered":"2024-09-11T08:16:37.185750+0000","last_clean":"2024-09-11T08:16:37.185750+0000","last_became_active":"2024-09-11T08:07:30.045613+0000","last_became_peered":"2024-09-11T08:07:30.045613+0000","last_unstale":"2024-09-11T08:16:37.185750+0000","last_undegraded":"2024-09-11T08:16:37.185750+0000","last_fullsized":"2024-09-11T08:16:37.185750+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:34.497125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:34.497125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:45.232704+0000","scrub_duration":0.0052802250000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":226,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.184149+0000","last_change":"2024-09-11T08:15:37.123094+0000","last_active":"2024-09-11T08:16:37.184149+0000","last_peered":"2024-09-11T08:16:37.184149+0000","last_clean":"2024-09-11T08:16:37.184149+0000","last_became_active":"2024-09-11T08:07:34.056606+0000","last_became_peered":"2024-09-11T08:07:34.056606+0000","last_unstale":"2024-09-11T08:16:37.184149+0000","last_undegraded":"2024-09-11T08:16:37.184149+0000","last_fullsized":"2024-09-11T08:16:37.184149+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:37.123010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:15:37.123010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:05.989949+0000","scrub_duration":0.0042949169999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":222,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.181488+0000","last_change":"2024-09-11T08:15:56.520006+0000","last_active":"2024-09-11T08:16:37.181488+0000","last_peered":"2024-09-11T08:16:37.181488+0000","last_clean":"2024-09-11T08:16:37.181488+0000","last_became_active":"2024-09-11T08:15:56.518884+0000","last_became_peered":"2024-09-11T08:15:56.518884+0000","last_unstale":"2024-09-11T08:16:37.181488+0000","last_undegraded":"2024-09-11T08:16:37.181488+0000","last_fullsized":"2024-09-11T08:16:37.181488+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:01.376411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:14:01.376411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012067320000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"221'13036","reported_seq":17246,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-11T08:16:48.319301+0000","last_change":"2024-09-11T08:14:51.900945+0000","last_active":"2024-09-11T08:16:48.319301+0000","last_peered":"2024-09-11T08:16:48.319301+0000","last_clean":"2024-09-11T08:16:48.319301+0000","last_became_active":"2024-09-11T08:14:51.879122+0000","last_became_peered":"2024-09-11T08:14:51.879122+0000","last_unstale":"2024-09-11T08:16:48.319301+0000","last_undegraded":"2024-09-11T08:16:48.319301+0000","last_fullsized":"2024-09-11T08:16:48.319301+0000","mapping_epoch":180,"log_start":"197'7100","ondisk_log_start":"197'7100","created":159,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:13:53.862306+0000","objects_scrubbed":0,"log_size":5936,"log_dups_size":0,"ondisk_log_size":5936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217579520,"num_objects":3320,"num_object_clones":0,"num_object_copies":9960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13036,"num_write_kb":523392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2061,"num_bytes_recovered":135069696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":316,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.181670+0000","last_change":"2024-09-11T08:15:56.551439+0000","last_active":"2024-09-11T08:16:37.181670+0000","last_peered":"2024-09-11T08:16:37.181670+0000","last_clean":"2024-09-11T08:16:37.181670+0000","last_became_active":"2024-09-11T08:15:56.526961+0000","last_became_peered":"2024-09-11T08:15:56.526961+0000","last_unstale":"2024-09-11T08:16:37.181670+0000","last_undegraded":"2024-09-11T08:16:37.181670+0000","last_fullsized":"2024-09-11T08:16:37.181670+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-11T08:15:49.187186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:15:49.187186+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:17.036763+0000","scrub_duration":0.0054603430000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":33,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-11T08:16:46.578872+0000","last_change":"2024-09-11T08:16:46.578643+0000","last_active":"2024-09-11T08:16:46.578872+0000","last_peered":"2024-09-11T08:16:46.578872+0000","last_clean":"2024-09-11T08:16:46.578872+0000","last_became_active":"2024-09-11T08:15:56.530370+0000","last_became_peered":"2024-09-11T08:15:56.530370+0000","last_unstale":"2024-09-11T08:16:46.578872+0000","last_undegraded":"2024-09-11T08:16:46.578872+0000","last_fullsized":"2024-09-11T08:16:46.578872+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:46.578575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:44.385372+0000","last_clean_scrub_stamp":"2024-09-11T08:16:46.578575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:15.109608+0000","scrub_duration":0.016632636999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":219,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.184387+0000","last_change":"2024-09-11T08:14:59.900979+0000","last_active":"2024-09-11T08:16:37.184387+0000","last_peered":"2024-09-11T08:16:37.184387+0000","last_clean":"2024-09-11T08:16:37.184387+0000","last_became_active":"2024-09-11T08:07:34.063589+0000","last_became_peered":"2024-09-11T08:07:34.063589+0000","last_unstale":"2024-09-11T08:16:37.184387+0000","last_undegraded":"2024-09-11T08:16:37.184387+0000","last_fullsized":"2024-09-11T08:16:37.184387+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:59.900898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:14:59.900898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.006061121,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":23,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.187138+0000","last_change":"2024-09-11T08:15:56.518476+0000","last_active":"2024-09-11T08:16:37.187138+0000","last_peered":"2024-09-11T08:16:37.187138+0000","last_clean":"2024-09-11T08:16:37.187138+0000","last_became_active":"2024-09-11T08:15:56.518169+0000","last_became_peered":"2024-09-11T08:15:56.518169+0000","last_unstale":"2024-09-11T08:16:37.187138+0000","last_undegraded":"2024-09-11T08:16:37.187138+0000","last_fullsized":"2024-09-11T08:16:37.187138+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:32.131008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:15:32.131008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:59.362244+0000","scrub_duration":0.0064196519999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"221'13035","reported_seq":17815,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-11T08:16:47.780850+0000","last_change":"2024-09-11T08:15:38.333949+0000","last_active":"2024-09-11T08:16:47.780850+0000","last_peered":"2024-09-11T08:16:47.780850+0000","last_clean":"2024-09-11T08:16:47.780850+0000","last_became_active":"2024-09-11T08:14:50.875228+0000","last_became_peered":"2024-09-11T08:14:50.875228+0000","last_unstale":"2024-09-11T08:16:47.780850+0000","last_undegraded":"2024-09-11T08:16:47.780850+0000","last_fullsized":"2024-09-11T08:16:47.780850+0000","mapping_epoch":179,"log_start":"196'6976","ondisk_log_start":"196'6976","created":159,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"200'7896","last_scrub_stamp":"2024-09-11T08:15:38.333891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:15:38.333891+0000","objects_scrubbed":7806,"log_size":6059,"log_dups_size":0,"ondisk_log_size":6059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2.12463202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":214499328,"num_objects":3273,"num_object_clones":0,"num_object_copies":9819,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13035,"num_write_kb":521856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2184,"num_bytes_recovered":143130624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"145'17","reported_seq":237,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.177502+0000","last_change":"2024-09-11T08:15:53.040339+0000","last_active":"2024-09-11T08:16:37.177502+0000","last_peered":"2024-09-11T08:16:37.177502+0000","last_clean":"2024-09-11T08:16:37.177502+0000","last_became_active":"2024-09-11T08:07:32.048164+0000","last_became_peered":"2024-09-11T08:07:32.048164+0000","last_unstale":"2024-09-11T08:16:37.177502+0000","last_undegraded":"2024-09-11T08:16:37.177502+0000","last_fullsized":"2024-09-11T08:16:37.177502+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'17","last_scrub_stamp":"2024-09-11T08:15:53.040223+0000","last_deep_scrub":"145'17","last_deep_scrub_stamp":"2024-09-11T08:13:08.160848+0000","last_clean_scrub_stamp":"2024-09-11T08:15:53.040223+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:12.632932+0000","scrub_duration":0.0039160820000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":33,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-11T08:16:47.557767+0000","last_change":"2024-09-11T08:16:47.557730+0000","last_active":"2024-09-11T08:16:47.557767+0000","last_peered":"2024-09-11T08:16:47.557767+0000","last_clean":"2024-09-11T08:16:47.557767+0000","last_became_active":"2024-09-11T08:15:56.525845+0000","last_became_peered":"2024-09-11T08:15:56.525845+0000","last_unstale":"2024-09-11T08:16:47.557767+0000","last_undegraded":"2024-09-11T08:16:47.557767+0000","last_fullsized":"2024-09-11T08:16:47.557767+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:47.557675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:53.247275+0000","last_clean_scrub_stamp":"2024-09-11T08:16:47.557675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:01.123588+0000","scrub_duration":0.0086777009999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":216,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:38.072454+0000","last_change":"2024-09-11T08:16:38.072401+0000","last_active":"2024-09-11T08:16:38.072454+0000","last_peered":"2024-09-11T08:16:38.072454+0000","last_clean":"2024-09-11T08:16:38.072454+0000","last_became_active":"2024-09-11T08:07:34.057479+0000","last_became_peered":"2024-09-11T08:07:34.057479+0000","last_unstale":"2024-09-11T08:16:38.072454+0000","last_undegraded":"2024-09-11T08:16:38.072454+0000","last_fullsized":"2024-09-11T08:16:38.072454+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:38.072323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:16:38.072323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:47.397010+0000","scrub_duration":0.005761909,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":222,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.174756+0000","last_change":"2024-09-11T08:15:56.522949+0000","last_active":"2024-09-11T08:16:37.174756+0000","last_peered":"2024-09-11T08:16:37.174756+0000","last_clean":"2024-09-11T08:16:37.174756+0000","last_became_active":"2024-09-11T08:15:56.522624+0000","last_became_peered":"2024-09-11T08:15:56.522624+0000","last_unstale":"2024-09-11T08:16:37.174756+0000","last_undegraded":"2024-09-11T08:16:37.174756+0000","last_fullsized":"2024-09-11T08:16:37.174756+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:54.938695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:13:54.938695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016854260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"221'12932","reported_seq":13303,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-11T08:16:47.908549+0000","last_change":"2024-09-11T08:15:53.462987+0000","last_active":"2024-09-11T08:16:47.908549+0000","last_peered":"2024-09-11T08:16:47.908549+0000","last_clean":"2024-09-11T08:16:47.908549+0000","last_became_active":"2024-09-11T08:13:55.705819+0000","last_became_peered":"2024-09-11T08:13:55.705819+0000","last_unstale":"2024-09-11T08:16:47.908549+0000","last_undegraded":"2024-09-11T08:16:47.908549+0000","last_fullsized":"2024-09-11T08:16:47.908549+0000","mapping_epoch":159,"log_start":"198'7600","ondisk_log_start":"198'7600","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"206'9066","last_scrub_stamp":"2024-09-11T08:15:53.462907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:15:53.462907+0000","objects_scrubbed":7268,"log_size":5332,"log_dups_size":0,"ondisk_log_size":5332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:18.668302+0000","scrub_duration":2.0133290810000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218759168,"num_objects":3338,"num_object_clones":0,"num_object_copies":10014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12932,"num_write_kb":520640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":292,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.188125+0000","last_change":"2024-09-11T08:15:56.551197+0000","last_active":"2024-09-11T08:16:37.188125+0000","last_peered":"2024-09-11T08:16:37.188125+0000","last_clean":"2024-09-11T08:16:37.188125+0000","last_became_active":"2024-09-11T08:15:56.532431+0000","last_became_peered":"2024-09-11T08:15:56.532431+0000","last_unstale":"2024-09-11T08:16:37.188125+0000","last_undegraded":"2024-09-11T08:16:37.188125+0000","last_fullsized":"2024-09-11T08:16:37.188125+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-11T08:15:47.427884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:15:47.427884+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:52.822361+0000","scrub_duration":0.0016881800000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":39,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-11T08:16:42.714148+0000","last_change":"2024-09-11T08:16:42.714111+0000","last_active":"2024-09-11T08:16:42.714148+0000","last_peered":"2024-09-11T08:16:42.714148+0000","last_clean":"2024-09-11T08:16:42.714148+0000","last_became_active":"2024-09-11T08:15:56.515328+0000","last_became_peered":"2024-09-11T08:15:56.515328+0000","last_unstale":"2024-09-11T08:16:42.714148+0000","last_undegraded":"2024-09-11T08:16:42.714148+0000","last_fullsized":"2024-09-11T08:16:42.714148+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-11T08:16:42.714038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:16:42.714038+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:06.366726+0000","scrub_duration":0.0083916919999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"209'11","reported_seq":37,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-11T08:16:39.187582+0000","last_change":"2024-09-11T08:15:56.534905+0000","last_active":"2024-09-11T08:16:39.187582+0000","last_peered":"2024-09-11T08:16:39.187582+0000","last_clean":"2024-09-11T08:16:39.187582+0000","last_became_active":"2024-09-11T08:15:56.524513+0000","last_became_peered":"2024-09-11T08:15:56.524513+0000","last_unstale":"2024-09-11T08:16:39.187582+0000","last_undegraded":"2024-09-11T08:16:39.187582+0000","last_fullsized":"2024-09-11T08:16:39.187582+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"156'10","last_scrub_stamp":"2024-09-11T08:15:38.215399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:15:38.215399+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:00.740762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":204,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.184423+0000","last_change":"2024-09-11T08:15:03.484853+0000","last_active":"2024-09-11T08:16:37.184423+0000","last_peered":"2024-09-11T08:16:37.184423+0000","last_clean":"2024-09-11T08:16:37.184423+0000","last_became_active":"2024-09-11T08:07:36.631755+0000","last_became_peered":"2024-09-11T08:07:36.631755+0000","last_unstale":"2024-09-11T08:16:37.184423+0000","last_undegraded":"2024-09-11T08:16:37.184423+0000","last_fullsized":"2024-09-11T08:16:37.184423+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:03.484755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:03.484755+0000","last_clean_scrub_stamp":"2024-09-11T08:15:03.484755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0064296900000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":220,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.182019+0000","last_change":"2024-09-11T08:15:31.257899+0000","last_active":"2024-09-11T08:16:37.182019+0000","last_peered":"2024-09-11T08:16:37.182019+0000","last_clean":"2024-09-11T08:16:37.182019+0000","last_became_active":"2024-09-11T08:07:30.045910+0000","last_became_peered":"2024-09-11T08:07:30.045910+0000","last_unstale":"2024-09-11T08:16:37.182019+0000","last_undegraded":"2024-09-11T08:16:37.182019+0000","last_fullsized":"2024-09-11T08:16:37.182019+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:31.257821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:31.257821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:56.144543+0000","scrub_duration":0.0040247779999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"84'9","reported_seq":254,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.176339+0000","last_change":"2024-09-11T08:15:56.550790+0000","last_active":"2024-09-11T08:16:37.176339+0000","last_peered":"2024-09-11T08:16:37.176339+0000","last_clean":"2024-09-11T08:16:37.176339+0000","last_became_active":"2024-09-11T08:15:56.526325+0000","last_became_peered":"2024-09-11T08:15:56.526325+0000","last_unstale":"2024-09-11T08:16:37.176339+0000","last_undegraded":"2024-09-11T08:16:37.176339+0000","last_fullsized":"2024-09-11T08:16:37.176339+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-09-11T08:14:12.071950+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-09-11T08:09:03.418457+0000","last_clean_scrub_stamp":"2024-09-11T08:14:12.071950+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0068954059999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":206,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.183708+0000","last_change":"2024-09-11T08:15:02.463928+0000","last_active":"2024-09-11T08:16:37.183708+0000","last_peered":"2024-09-11T08:16:37.183708+0000","last_clean":"2024-09-11T08:16:37.183708+0000","last_became_active":"2024-09-11T08:07:34.057774+0000","last_became_peered":"2024-09-11T08:07:34.057774+0000","last_unstale":"2024-09-11T08:16:37.183708+0000","last_undegraded":"2024-09-11T08:16:37.183708+0000","last_fullsized":"2024-09-11T08:16:37.183708+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:02.463848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:02.463848+0000","last_clean_scrub_stamp":"2024-09-11T08:15:02.463848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034811849999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":148,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.189476+0000","last_change":"2024-09-11T08:14:56.815770+0000","last_active":"2024-09-11T08:16:37.189476+0000","last_peered":"2024-09-11T08:16:37.189476+0000","last_clean":"2024-09-11T08:16:37.189476+0000","last_became_active":"2024-09-11T08:14:24.790678+0000","last_became_peered":"2024-09-11T08:14:24.790678+0000","last_unstale":"2024-09-11T08:16:37.189476+0000","last_undegraded":"2024-09-11T08:16:37.189476+0000","last_fullsized":"2024-09-11T08:16:37.189476+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:56.815690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:14:56.815690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016768309999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"84'13","reported_seq":266,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.181550+0000","last_change":"2024-09-11T08:15:56.532882+0000","last_active":"2024-09-11T08:16:37.181550+0000","last_peered":"2024-09-11T08:16:37.181550+0000","last_clean":"2024-09-11T08:16:37.181550+0000","last_became_active":"2024-09-11T08:15:56.514793+0000","last_became_peered":"2024-09-11T08:15:56.514793+0000","last_unstale":"2024-09-11T08:16:37.181550+0000","last_undegraded":"2024-09-11T08:16:37.181550+0000","last_fullsized":"2024-09-11T08:16:37.181550+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'13","last_scrub_stamp":"2024-09-11T08:14:05.339509+0000","last_deep_scrub":"84'13","last_deep_scrub_stamp":"2024-09-11T08:08:57.835634+0000","last_clean_scrub_stamp":"2024-09-11T08:14:05.339509+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001614047,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":181,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.189133+0000","last_change":"2024-09-11T08:15:31.534404+0000","last_active":"2024-09-11T08:16:37.189133+0000","last_peered":"2024-09-11T08:16:37.189133+0000","last_clean":"2024-09-11T08:16:37.189133+0000","last_became_active":"2024-09-11T08:14:24.786021+0000","last_became_peered":"2024-09-11T08:14:24.786021+0000","last_unstale":"2024-09-11T08:16:37.189133+0000","last_undegraded":"2024-09-11T08:16:37.189133+0000","last_fullsized":"2024-09-11T08:16:37.189133+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:31.534302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:30.471620+0000","last_clean_scrub_stamp":"2024-09-11T08:15:31.534302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:43.916804+0000","scrub_duration":0.0021837200000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":224,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.176226+0000","last_change":"2024-09-11T08:15:56.526834+0000","last_active":"2024-09-11T08:16:37.176226+0000","last_peered":"2024-09-11T08:16:37.176226+0000","last_clean":"2024-09-11T08:16:37.176226+0000","last_became_active":"2024-09-11T08:15:56.525717+0000","last_became_peered":"2024-09-11T08:15:56.525717+0000","last_unstale":"2024-09-11T08:16:37.176226+0000","last_undegraded":"2024-09-11T08:16:37.176226+0000","last_fullsized":"2024-09-11T08:16:37.176226+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:59.988381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:04.370734+0000","last_clean_scrub_stamp":"2024-09-11T08:13:59.988381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017033700000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":231,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.184798+0000","last_change":"2024-09-11T08:15:56.525429+0000","last_active":"2024-09-11T08:16:37.184798+0000","last_peered":"2024-09-11T08:16:37.184798+0000","last_clean":"2024-09-11T08:16:37.184798+0000","last_became_active":"2024-09-11T08:15:56.525120+0000","last_became_peered":"2024-09-11T08:15:56.525120+0000","last_unstale":"2024-09-11T08:16:37.184798+0000","last_undegraded":"2024-09-11T08:16:37.184798+0000","last_fullsized":"2024-09-11T08:16:37.184798+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:54.184372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:54.184372+0000","last_clean_scrub_stamp":"2024-09-11T08:15:54.184372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:21.919337+0000","scrub_duration":0.013922767000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":281,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.183963+0000","last_change":"2024-09-11T08:15:56.547635+0000","last_active":"2024-09-11T08:16:37.183963+0000","last_peered":"2024-09-11T08:16:37.183963+0000","last_clean":"2024-09-11T08:16:37.183963+0000","last_became_active":"2024-09-11T08:15:56.532513+0000","last_became_peered":"2024-09-11T08:15:56.532513+0000","last_unstale":"2024-09-11T08:16:37.183963+0000","last_undegraded":"2024-09-11T08:16:37.183963+0000","last_fullsized":"2024-09-11T08:16:37.183963+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:15:52.443189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:15:52.443189+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:53.586813+0000","scrub_duration":0.010882618,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":228,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.180885+0000","last_change":"2024-09-11T08:15:56.513604+0000","last_active":"2024-09-11T08:16:37.180885+0000","last_peered":"2024-09-11T08:16:37.180885+0000","last_clean":"2024-09-11T08:16:37.180885+0000","last_became_active":"2024-09-11T08:15:56.513278+0000","last_became_peered":"2024-09-11T08:15:56.513278+0000","last_unstale":"2024-09-11T08:16:37.180885+0000","last_undegraded":"2024-09-11T08:16:37.180885+0000","last_fullsized":"2024-09-11T08:16:37.180885+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:48.213172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:48.935387+0000","last_clean_scrub_stamp":"2024-09-11T08:13:48.213172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011628039999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"209'11","reported_seq":352,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-11T08:16:39.192246+0000","last_change":"2024-09-11T08:15:56.543025+0000","last_active":"2024-09-11T08:16:39.192246+0000","last_peered":"2024-09-11T08:16:39.192246+0000","last_clean":"2024-09-11T08:16:39.192246+0000","last_became_active":"2024-09-11T08:15:56.525253+0000","last_became_peered":"2024-09-11T08:15:56.525253+0000","last_unstale":"2024-09-11T08:16:39.192246+0000","last_undegraded":"2024-09-11T08:16:39.192246+0000","last_fullsized":"2024-09-11T08:16:39.192246+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"156'10","last_scrub_stamp":"2024-09-11T08:14:01.979463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:14:01.979463+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012487329999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":195,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.178217+0000","last_change":"2024-09-11T08:15:05.867100+0000","last_active":"2024-09-11T08:16:37.178217+0000","last_peered":"2024-09-11T08:16:37.178217+0000","last_clean":"2024-09-11T08:16:37.178217+0000","last_became_active":"2024-09-11T08:07:36.068493+0000","last_became_peered":"2024-09-11T08:07:36.068493+0000","last_unstale":"2024-09-11T08:16:37.178217+0000","last_undegraded":"2024-09-11T08:16:37.178217+0000","last_fullsized":"2024-09-11T08:16:37.178217+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:05.867013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:05.867013+0000","last_clean_scrub_stamp":"2024-09-11T08:15:05.867013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0061248279999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":253,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.174415+0000","last_change":"2024-09-11T08:14:56.223381+0000","last_active":"2024-09-11T08:16:37.174415+0000","last_peered":"2024-09-11T08:16:37.174415+0000","last_clean":"2024-09-11T08:16:37.174415+0000","last_became_active":"2024-09-11T08:14:24.791187+0000","last_became_peered":"2024-09-11T08:14:24.791187+0000","last_unstale":"2024-09-11T08:16:37.174415+0000","last_undegraded":"2024-09-11T08:16:37.174415+0000","last_fullsized":"2024-09-11T08:16:37.174415+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-11T08:14:56.222082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:14:56.222082+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011605691,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":23,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.189306+0000","last_change":"2024-09-11T08:15:56.521890+0000","last_active":"2024-09-11T08:16:37.189306+0000","last_peered":"2024-09-11T08:16:37.189306+0000","last_clean":"2024-09-11T08:16:37.189306+0000","last_became_active":"2024-09-11T08:15:56.518515+0000","last_became_peered":"2024-09-11T08:15:56.518515+0000","last_unstale":"2024-09-11T08:16:37.189306+0000","last_undegraded":"2024-09-11T08:16:37.189306+0000","last_fullsized":"2024-09-11T08:16:37.189306+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:33.021088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:33.021088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:40.824210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":325,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-11T08:16:39.192978+0000","last_change":"2024-09-11T08:15:44.429144+0000","last_active":"2024-09-11T08:16:39.192978+0000","last_peered":"2024-09-11T08:16:39.192978+0000","last_clean":"2024-09-11T08:16:39.192978+0000","last_became_active":"2024-09-11T08:07:34.060736+0000","last_became_peered":"2024-09-11T08:07:34.060736+0000","last_unstale":"2024-09-11T08:16:39.192978+0000","last_undegraded":"2024-09-11T08:16:39.192978+0000","last_fullsized":"2024-09-11T08:16:39.192978+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:15:44.429062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:15:44.429062+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:58.251305+0000","scrub_duration":0.0062457210000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":222,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.181201+0000","last_change":"2024-09-11T08:15:56.515344+0000","last_active":"2024-09-11T08:16:37.181201+0000","last_peered":"2024-09-11T08:16:37.181201+0000","last_clean":"2024-09-11T08:16:37.181201+0000","last_became_active":"2024-09-11T08:15:56.513949+0000","last_became_peered":"2024-09-11T08:15:56.513949+0000","last_unstale":"2024-09-11T08:16:37.181201+0000","last_undegraded":"2024-09-11T08:16:37.181201+0000","last_fullsized":"2024-09-11T08:16:37.181201+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:01.448261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:15:01.448261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.021382708,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":218,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.178016+0000","last_change":"2024-09-11T08:13:13.047503+0000","last_active":"2024-09-11T08:16:37.178016+0000","last_peered":"2024-09-11T08:16:37.178016+0000","last_clean":"2024-09-11T08:16:37.178016+0000","last_became_active":"2024-09-11T08:07:32.047981+0000","last_became_peered":"2024-09-11T08:07:32.047981+0000","last_unstale":"2024-09-11T08:16:37.178016+0000","last_undegraded":"2024-09-11T08:16:37.178016+0000","last_fullsized":"2024-09-11T08:16:37.178016+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-11T08:13:13.047436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:13:13.047436+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012804688,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":228,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.178253+0000","last_change":"2024-09-11T08:15:56.523858+0000","last_active":"2024-09-11T08:16:37.178253+0000","last_peered":"2024-09-11T08:16:37.178253+0000","last_clean":"2024-09-11T08:16:37.178253+0000","last_became_active":"2024-09-11T08:15:56.523574+0000","last_became_peered":"2024-09-11T08:15:56.523574+0000","last_unstale":"2024-09-11T08:16:37.178253+0000","last_undegraded":"2024-09-11T08:16:37.178253+0000","last_fullsized":"2024-09-11T08:16:37.178253+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:55.845422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:13:55.845422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00143399,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":210,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.175133+0000","last_change":"2024-09-11T08:15:38.410767+0000","last_active":"2024-09-11T08:16:37.175133+0000","last_peered":"2024-09-11T08:16:37.175133+0000","last_clean":"2024-09-11T08:16:37.175133+0000","last_became_active":"2024-09-11T08:07:34.053137+0000","last_became_peered":"2024-09-11T08:07:34.053137+0000","last_unstale":"2024-09-11T08:16:37.175133+0000","last_undegraded":"2024-09-11T08:16:37.175133+0000","last_fullsized":"2024-09-11T08:16:37.175133+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:38.410625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:15:38.410625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:59.728039+0000","scrub_duration":0.021540422,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":195,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.173858+0000","last_change":"2024-09-11T08:15:56.494833+0000","last_active":"2024-09-11T08:16:37.173858+0000","last_peered":"2024-09-11T08:16:37.173858+0000","last_clean":"2024-09-11T08:16:37.173858+0000","last_became_active":"2024-09-11T08:07:36.076132+0000","last_became_peered":"2024-09-11T08:07:36.076132+0000","last_unstale":"2024-09-11T08:16:37.173858+0000","last_undegraded":"2024-09-11T08:16:37.173858+0000","last_fullsized":"2024-09-11T08:16:37.173858+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:56.494764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:15:56.494764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:18.392464+0000","scrub_duration":0.001948635,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":235,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.175372+0000","last_change":"2024-09-11T08:16:04.409705+0000","last_active":"2024-09-11T08:16:37.175372+0000","last_peered":"2024-09-11T08:16:37.175372+0000","last_clean":"2024-09-11T08:16:37.175372+0000","last_became_active":"2024-09-11T08:07:32.049368+0000","last_became_peered":"2024-09-11T08:07:32.049368+0000","last_unstale":"2024-09-11T08:16:37.175372+0000","last_undegraded":"2024-09-11T08:16:37.175372+0000","last_fullsized":"2024-09-11T08:16:37.175372+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:16:04.409608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:16:04.409608+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:19.702294+0000","scrub_duration":0.0094835470000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":23,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.189336+0000","last_change":"2024-09-11T08:15:56.521976+0000","last_active":"2024-09-11T08:16:37.189336+0000","last_peered":"2024-09-11T08:16:37.189336+0000","last_clean":"2024-09-11T08:16:37.189336+0000","last_became_active":"2024-09-11T08:15:56.521570+0000","last_became_peered":"2024-09-11T08:15:56.521570+0000","last_unstale":"2024-09-11T08:16:37.189336+0000","last_undegraded":"2024-09-11T08:16:37.189336+0000","last_fullsized":"2024-09-11T08:16:37.189336+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:04.256038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:04.256038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":197,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.174830+0000","last_change":"2024-09-11T08:15:01.218687+0000","last_active":"2024-09-11T08:16:37.174830+0000","last_peered":"2024-09-11T08:16:37.174830+0000","last_clean":"2024-09-11T08:16:37.174830+0000","last_became_active":"2024-09-11T08:07:34.057064+0000","last_became_peered":"2024-09-11T08:07:34.057064+0000","last_unstale":"2024-09-11T08:16:37.174830+0000","last_undegraded":"2024-09-11T08:16:37.174830+0000","last_fullsized":"2024-09-11T08:16:37.174830+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:01.218475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:01.218475+0000","last_clean_scrub_stamp":"2024-09-11T08:15:01.218475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0099033660000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":205,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-11T08:16:39.635272+0000","last_change":"2024-09-11T08:16:39.635237+0000","last_active":"2024-09-11T08:16:39.635272+0000","last_peered":"2024-09-11T08:16:39.635272+0000","last_clean":"2024-09-11T08:16:39.635272+0000","last_became_active":"2024-09-11T08:07:36.077406+0000","last_became_peered":"2024-09-11T08:07:36.077406+0000","last_unstale":"2024-09-11T08:16:39.635272+0000","last_undegraded":"2024-09-11T08:16:39.635272+0000","last_fullsized":"2024-09-11T08:16:39.635272+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:39.635151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:09.973943+0000","last_clean_scrub_stamp":"2024-09-11T08:16:39.635151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:45.940617+0000","scrub_duration":0.0062349479999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":269,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.173985+0000","last_change":"2024-09-11T08:15:56.543699+0000","last_active":"2024-09-11T08:16:37.173985+0000","last_peered":"2024-09-11T08:16:37.173985+0000","last_clean":"2024-09-11T08:16:37.173985+0000","last_became_active":"2024-09-11T08:15:56.520058+0000","last_became_peered":"2024-09-11T08:15:56.520058+0000","last_unstale":"2024-09-11T08:16:37.173985+0000","last_undegraded":"2024-09-11T08:16:37.173985+0000","last_fullsized":"2024-09-11T08:16:37.173985+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-11T08:14:10.129903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:14:10.129903+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004657011,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":237,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.183718+0000","last_change":"2024-09-11T08:15:56.531914+0000","last_active":"2024-09-11T08:16:37.183718+0000","last_peered":"2024-09-11T08:16:37.183718+0000","last_clean":"2024-09-11T08:16:37.183718+0000","last_became_active":"2024-09-11T08:15:56.531612+0000","last_became_peered":"2024-09-11T08:15:56.531612+0000","last_unstale":"2024-09-11T08:16:37.183718+0000","last_undegraded":"2024-09-11T08:16:37.183718+0000","last_fullsized":"2024-09-11T08:16:37.183718+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:49.478359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:49.478359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:16.513888+0000","scrub_duration":0.012202396000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":33,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:38.643822+0000","last_change":"2024-09-11T08:16:38.643733+0000","last_active":"2024-09-11T08:16:38.643822+0000","last_peered":"2024-09-11T08:16:38.643822+0000","last_clean":"2024-09-11T08:16:38.643822+0000","last_became_active":"2024-09-11T08:15:56.523728+0000","last_became_peered":"2024-09-11T08:15:56.523728+0000","last_unstale":"2024-09-11T08:16:38.643822+0000","last_undegraded":"2024-09-11T08:16:38.643822+0000","last_fullsized":"2024-09-11T08:16:38.643822+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:38.643651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:11:34.159053+0000","last_clean_scrub_stamp":"2024-09-11T08:16:38.643651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:58.036696+0000","scrub_duration":0.002366469,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":195,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.174946+0000","last_change":"2024-09-11T08:15:44.448007+0000","last_active":"2024-09-11T08:16:37.174946+0000","last_peered":"2024-09-11T08:16:37.174946+0000","last_clean":"2024-09-11T08:16:37.174946+0000","last_became_active":"2024-09-11T08:07:36.633368+0000","last_became_peered":"2024-09-11T08:07:36.633368+0000","last_unstale":"2024-09-11T08:16:37.174946+0000","last_undegraded":"2024-09-11T08:16:37.174946+0000","last_fullsized":"2024-09-11T08:16:37.174946+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:44.447924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:34.378734+0000","last_clean_scrub_stamp":"2024-09-11T08:15:44.447924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:50.350614+0000","scrub_duration":0.0029812609999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":205,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.174904+0000","last_change":"2024-09-11T08:15:41.389724+0000","last_active":"2024-09-11T08:16:37.174904+0000","last_peered":"2024-09-11T08:16:37.174904+0000","last_clean":"2024-09-11T08:16:37.174904+0000","last_became_active":"2024-09-11T08:07:32.045728+0000","last_became_peered":"2024-09-11T08:07:32.045728+0000","last_unstale":"2024-09-11T08:16:37.174904+0000","last_undegraded":"2024-09-11T08:16:37.174904+0000","last_fullsized":"2024-09-11T08:16:37.174904+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'4","last_scrub_stamp":"2024-09-11T08:15:41.389510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:15:41.389510+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:10.070852+0000","scrub_duration":0.0069641269999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":211,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.181608+0000","last_change":"2024-09-11T08:15:39.743690+0000","last_active":"2024-09-11T08:16:37.181608+0000","last_peered":"2024-09-11T08:16:37.181608+0000","last_clean":"2024-09-11T08:16:37.181608+0000","last_became_active":"2024-09-11T08:07:30.049409+0000","last_became_peered":"2024-09-11T08:07:30.049409+0000","last_unstale":"2024-09-11T08:16:37.181608+0000","last_undegraded":"2024-09-11T08:16:37.181608+0000","last_fullsized":"2024-09-11T08:16:37.181608+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:39.743604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:39.743604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:08.408228+0000","scrub_duration":0.0089953759999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":243,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.182167+0000","last_change":"2024-09-11T08:15:56.529068+0000","last_active":"2024-09-11T08:16:37.182167+0000","last_peered":"2024-09-11T08:16:37.182167+0000","last_clean":"2024-09-11T08:16:37.182167+0000","last_became_active":"2024-09-11T08:15:56.528607+0000","last_became_peered":"2024-09-11T08:15:56.528607+0000","last_unstale":"2024-09-11T08:16:37.182167+0000","last_undegraded":"2024-09-11T08:16:37.182167+0000","last_fullsized":"2024-09-11T08:16:37.182167+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:37.177042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:15:37.177042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:50.577161+0000","scrub_duration":0.011175861,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":23,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.189015+0000","last_change":"2024-09-11T08:15:56.522225+0000","last_active":"2024-09-11T08:16:37.189015+0000","last_peered":"2024-09-11T08:16:37.189015+0000","last_clean":"2024-09-11T08:16:37.189015+0000","last_became_active":"2024-09-11T08:15:56.521488+0000","last_became_peered":"2024-09-11T08:15:56.521488+0000","last_unstale":"2024-09-11T08:16:37.189015+0000","last_undegraded":"2024-09-11T08:16:37.189015+0000","last_fullsized":"2024-09-11T08:16:37.189015+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:35.183289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:15:35.183289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:51.893171+0000","scrub_duration":0.007179726,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":205,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.174450+0000","last_change":"2024-09-11T08:14:07.076355+0000","last_active":"2024-09-11T08:16:37.174450+0000","last_peered":"2024-09-11T08:16:37.174450+0000","last_clean":"2024-09-11T08:16:37.174450+0000","last_became_active":"2024-09-11T08:07:36.071468+0000","last_became_peered":"2024-09-11T08:07:36.071468+0000","last_unstale":"2024-09-11T08:16:37.174450+0000","last_undegraded":"2024-09-11T08:16:37.174450+0000","last_fullsized":"2024-09-11T08:16:37.174450+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:07.076300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:12:25.814048+0000","last_clean_scrub_stamp":"2024-09-11T08:14:07.076300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001297334,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":211,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.175103+0000","last_change":"2024-09-11T08:15:47.493538+0000","last_active":"2024-09-11T08:16:37.175103+0000","last_peered":"2024-09-11T08:16:37.175103+0000","last_clean":"2024-09-11T08:16:37.175103+0000","last_became_active":"2024-09-11T08:07:30.046702+0000","last_became_peered":"2024-09-11T08:07:30.046702+0000","last_unstale":"2024-09-11T08:16:37.175103+0000","last_undegraded":"2024-09-11T08:16:37.175103+0000","last_fullsized":"2024-09-11T08:16:37.175103+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:47.493466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:47.493466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:47.631872+0000","scrub_duration":0.071745623999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":237,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-11T08:16:37.182689+0000","last_change":"2024-09-11T08:15:55.255237+0000","last_active":"2024-09-11T08:16:37.182689+0000","last_peered":"2024-09-11T08:16:37.182689+0000","last_clean":"2024-09-11T08:16:37.182689+0000","last_became_active":"2024-09-11T08:07:32.054656+0000","last_became_peered":"2024-09-11T08:07:32.054656+0000","last_unstale":"2024-09-11T08:16:37.182689+0000","last_undegraded":"2024-09-11T08:16:37.182689+0000","last_fullsized":"2024-09-11T08:16:37.182689+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-11T08:15:55.255152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:15:55.255152+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:09.255947+0000","scrub_duration":0.0070337539999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":51,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-11T08:16:46.497778+0000","last_change":"2024-09-11T08:16:46.497719+0000","last_active":"2024-09-11T08:16:46.497778+0000","last_peered":"2024-09-11T08:16:46.497778+0000","last_clean":"2024-09-11T08:16:46.497778+0000","last_became_active":"2024-09-11T08:15:08.769133+0000","last_became_peered":"2024-09-11T08:15:08.769133+0000","last_unstale":"2024-09-11T08:16:46.497778+0000","last_undegraded":"2024-09-11T08:16:46.497778+0000","last_fullsized":"2024-09-11T08:16:46.497778+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:46.497639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:16:46.497639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:14.479718+0000","scrub_duration":0.0048914830000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":3417767960,"num_objects":52152,"num_object_clones":0,"num_object_copies":156456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52152,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":205508,"num_write_kb":8245057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17752,"num_bytes_recovered":1163329560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10810638336,"data_stored":10810622048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85251,"ondisk_log_size":85251,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":139,"num_bytes_recovered":2888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3425,"internal_metadata":0},"log_size":450,"ondisk_log_size":450,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":149,"num_read_kb":144,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":168,"seq":721554505758,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1390976,"kb_used_data":1065144,"kb_used_omap":7,"kb_used_meta":325816,"kb_avail":92354176,"statfs":{"total":95995035648,"available":94570676224,"internally_reserved":0,"allocated":1090707456,"data_stored":1090299393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8027,"internal_metadata":333635749},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:16:26 2024","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":0.89800000000000002,"15min":0.84999999999999998},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":4.673,"5min":4.673,"15min":4.673},"last":2.7210000000000001},{"interface":"front","average":{"1min":0.97699999999999998,"5min":0.99099999999999999,"15min":0.99299999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":3.2559999999999998,"5min":3.2559999999999998,"15min":3.2559999999999998},"last":2.2690000000000001}]},{"osd":1,"last update":"Wed Sep 11 08:16:26 2024","interfaces":[{"interface":"back","average":{"1min":4.4729999999999999,"5min":2.4510000000000001,"15min":2.1139999999999999},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":42.082999999999998,"5min":42.082999999999998,"15min":42.082999999999998},"last":3.1760000000000002},{"interface":"front","average":{"1min":3.5390000000000001,"5min":2.3439999999999999,"15min":2.145},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":42.151000000000003,"5min":42.151000000000003,"15min":42.151000000000003},"last":2.5990000000000002}]},{"osd":2,"last update":"Wed Sep 11 08:16:26 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.97099999999999997,"15min":1.018},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":2.1600000000000001,"5min":3.9630000000000001,"15min":3.9630000000000001},"last":2.7349999999999999},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.73299999999999998,"15min":0.72499999999999998},"min":{"1min":0.28699999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":2.3119999999999998,"5min":2.601,"15min":2.601},"last":0.81699999999999995}]},{"osd":3,"last update":"Wed Sep 11 08:16:26 2024","interfaces":[{"interface":"back","average":{"1min":2.427,"5min":1.343,"15min":1.163},"min":{"1min":0.30499999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":21.460000000000001,"5min":21.460000000000001,"15min":21.460000000000001},"last":0.28499999999999998},{"interface":"front","average":{"1min":2.5169999999999999,"5min":1.643,"15min":1.4970000000000001},"min":{"1min":0.28999999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":21.440000000000001,"5min":21.440000000000001,"15min":21.440000000000001},"last":2.7770000000000001}]},{"osd":4,"last update":"Wed Sep 11 08:16:26 2024","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":0.629,"15min":0.56000000000000005},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":4.141,"5min":4.141,"15min":4.141},"last":0.88},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.73799999999999999,"15min":0.748},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":3.157,"5min":3.157,"15min":3.157},"last":0.749}]},{"osd":5,"last update":"Wed Sep 11 08:16:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":0.88900000000000001,"15min":0.82899999999999996},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":4.1379999999999999,"5min":4.1379999999999999,"15min":4.1379999999999999},"last":1.5289999999999999},{"interface":"front","average":{"1min":1.1120000000000001,"5min":0.82199999999999995,"15min":0.77400000000000002},"min":{"1min":0.41899999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":3.907,"5min":3.907,"15min":3.907},"last":0.32600000000000001}]},{"osd":6,"last update":"Wed Sep 11 08:16:26 2024","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":0.83299999999999996,"15min":0.80600000000000005},"min":{"1min":0.32000000000000001,"5min":0.312,"15min":0.312},"max":{"1min":6.0609999999999999,"5min":6.0609999999999999,"15min":6.0609999999999999},"last":0.73699999999999999},{"interface":"front","average":{"1min":1.0009999999999999,"5min":0.83199999999999996,"15min":0.80400000000000005},"min":{"1min":0.38800000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":3.6829999999999998,"5min":3.6829999999999998,"15min":3.6829999999999998},"last":0.84299999999999997}]}]},{"osd":6,"up_from":45,"seq":193273528441,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1655792,"kb_used_data":1345960,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":92089360,"statfs":{"total":95995035648,"available":94299504640,"internally_reserved":0,"allocated":1378263040,"data_stored":1377871184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:16:10 2024","interfaces":[{"interface":"back","average":{"1min":1.53,"5min":2.1360000000000001,"15min":1.0600000000000001},"min":{"1min":0.20899999999999999,"5min":0.20399999999999999,"15min":0.185},"max":{"1min":7.5810000000000004,"5min":45.066000000000003,"15min":45.066000000000003},"last":0.88},{"interface":"front","average":{"1min":1.5189999999999999,"5min":1.9510000000000001,"15min":1.026},"min":{"1min":0.224,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":4.1379999999999999,"5min":43.899000000000001,"15min":43.899000000000001},"last":0.46200000000000002}]},{"osd":1,"last update":"Wed Sep 11 08:16:10 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.7010000000000001,"15min":0.89500000000000002},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":3.931,"5min":44.658000000000001,"15min":44.658000000000001},"last":0.34300000000000003},{"interface":"front","average":{"1min":1.506,"5min":1.827,"15min":0.95799999999999996},"min":{"1min":0.32500000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":6.4870000000000001,"5min":44.694000000000003,"15min":44.694000000000003},"last":0.97299999999999998}]},{"osd":2,"last update":"Wed Sep 11 08:16:10 2024","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.0169999999999999,"15min":0.70099999999999996},"min":{"1min":0.28699999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":5.9770000000000003,"5min":5.9770000000000003,"15min":5.9770000000000003},"last":0.51800000000000002},{"interface":"front","average":{"1min":1.9790000000000001,"5min":1.45,"15min":0.84199999999999997},"min":{"1min":0.29299999999999998,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":7.4809999999999999,"5min":13.346,"15min":13.346},"last":0.33000000000000002}]},{"osd":3,"last update":"Wed Sep 11 08:16:10 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.7509999999999999,"15min":0.996},"min":{"1min":0.23599999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":3.3980000000000001,"5min":36.061,"15min":36.061},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.90300000000000002,"5min":1.718,"15min":0.97699999999999998},"min":{"1min":0.30599999999999999,"5min":0.28399999999999997,"15min":0.26800000000000002},"max":{"1min":2.6749999999999998,"5min":36.093000000000004,"15min":36.093000000000004},"last":0.432}]},{"osd":4,"last update":"Wed Sep 11 08:16:10 2024","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":0.57799999999999996,"15min":0.45700000000000002},"min":{"1min":0.23699999999999999,"5min":0.219,"15min":0.20200000000000001},"max":{"1min":4.1130000000000004,"5min":4.1130000000000004,"15min":4.1130000000000004},"last":0.251},{"interface":"front","average":{"1min":1.1579999999999999,"5min":0.64700000000000002,"15min":0.499},"min":{"1min":0.27100000000000002,"5min":0.251,"15min":0.24099999999999999},"max":{"1min":4.468,"5min":4.468,"15min":4.468},"last":0.48999999999999999}]},{"osd":5,"last update":"Wed Sep 11 08:16:10 2024","interfaces":[{"interface":"back","average":{"1min":1.514,"5min":0.76400000000000001,"15min":0.57999999999999996},"min":{"1min":0.37,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":7.4459999999999997,"5min":7.4459999999999997,"15min":7.4459999999999997},"last":0.41599999999999998},{"interface":"front","average":{"1min":1.246,"5min":0.747,"15min":0.53400000000000003},"min":{"1min":0.28399999999999997,"5min":0.22800000000000001,"15min":0.22600000000000001},"max":{"1min":3.4329999999999998,"5min":3.4329999999999998,"15min":3.4329999999999998},"last":0.39400000000000002}]},{"osd":7,"last update":"Wed Sep 11 08:16:25 2024","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":1.159,"15min":1.1499999999999999},"min":{"1min":0.28899999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":8.4580000000000002,"5min":8.4580000000000002,"15min":8.4580000000000002},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.98699999999999999,"5min":0.91900000000000004,"15min":0.90700000000000003},"min":{"1min":0.34000000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":3.593,"5min":3.593,"15min":3.593},"last":0.48099999999999998}]}]},{"osd":5,"up_from":38,"seq":163208757374,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2242288,"kb_used_data":1940776,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":91502864,"statfs":{"total":95995035648,"available":93698932736,"internally_reserved":0,"allocated":1987354624,"data_stored":1986961785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:15:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4430000000000001,"5min":1.254,"15min":0.78800000000000003},"min":{"1min":0.20999999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":4.327,"5min":7.9470000000000001,"15min":7.9470000000000001},"last":0.63800000000000001},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.222,"15min":0.77700000000000002},"min":{"1min":0.24399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":3.4820000000000002,"5min":7.6189999999999998,"15min":7.6189999999999998},"last":0.56200000000000006}]},{"osd":1,"last update":"Wed Sep 11 08:15:48 2024","interfaces":[{"interface":"back","average":{"1min":2.77,"5min":1.5760000000000001,"15min":1.2829999999999999},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":26.709,"5min":26.709,"15min":26.709},"last":0.29999999999999999},{"interface":"front","average":{"1min":2.5339999999999998,"5min":1.5569999999999999,"15min":0.85899999999999999},"min":{"1min":0.25900000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":26.754000000000001,"5min":26.754000000000001,"15min":26.754000000000001},"last":0.27400000000000002}]},{"osd":2,"last update":"Wed Sep 11 08:15:48 2024","interfaces":[{"interface":"back","average":{"1min":1.891,"5min":1.2589999999999999,"15min":1.171},"min":{"1min":0.36799999999999999,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":10.891,"5min":10.891,"15min":16.018000000000001},"last":0.73399999999999999},{"interface":"front","average":{"1min":1.4330000000000001,"5min":0.99099999999999999,"15min":0.68500000000000005},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":7.5250000000000004,"5min":7.5250000000000004,"15min":7.5250000000000004},"last":0.42099999999999999}]},{"osd":3,"last update":"Wed Sep 11 08:15:48 2024","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.3819999999999999,"15min":0.85399999999999998},"min":{"1min":0.29999999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":4.3150000000000004,"5min":24.931000000000001,"15min":24.931000000000001},"last":0.35799999999999998},{"interface":"front","average":{"1min":1.544,"5min":1.514,"15min":0.879},"min":{"1min":0.38800000000000001,"5min":0.251,"15min":0.251},"max":{"1min":4.4710000000000001,"5min":24.861999999999998,"15min":24.861999999999998},"last":0.58299999999999996}]},{"osd":4,"last update":"Wed Sep 11 08:15:48 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.58199999999999996,"15min":0.46999999999999997},"min":{"1min":0.23499999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5979999999999999,"5min":3.5979999999999999,"15min":3.5979999999999999},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.56200000000000006,"15min":0.46200000000000002},"min":{"1min":0.27300000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":2.706,"5min":2.706,"15min":2.706},"last":4.9210000000000003}]},{"osd":6,"last update":"Wed Sep 11 08:16:12 2024","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":0.54300000000000004,"15min":0.47299999999999998},"min":{"1min":0.30399999999999999,"5min":0.252,"15min":0.23400000000000001},"max":{"1min":3.1869999999999998,"5min":3.1869999999999998,"15min":3.1869999999999998},"last":0.60999999999999999},{"interface":"front","average":{"1min":1.0129999999999999,"5min":0.64500000000000002,"15min":0.53100000000000003},"min":{"1min":0.35199999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":3.5169999999999999,"5min":3.9489999999999998,"15min":3.9489999999999998},"last":0.47199999999999998}]},{"osd":7,"last update":"Wed Sep 11 08:16:28 2024","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.85299999999999998,"15min":0.83799999999999997},"min":{"1min":0.38500000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":3.5459999999999998,"5min":3.5459999999999998,"15min":3.5459999999999998},"last":1.5129999999999999},{"interface":"front","average":{"1min":1.0680000000000001,"5min":0.82399999999999995,"15min":0.78400000000000003},"min":{"1min":0.34899999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":3.1019999999999999,"5min":3.1019999999999999,"15min":3.1019999999999999},"last":1.486}]}]},{"osd":4,"up_from":31,"seq":133143986307,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1364928,"kb_used_data":1067512,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":92380224,"statfs":{"total":95995035648,"available":94597349376,"internally_reserved":0,"allocated":1093132288,"data_stored":1092724355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:16:20 2024","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":1.0600000000000001,"15min":1.0660000000000001},"min":{"1min":0.23300000000000001,"5min":0.19,"15min":0.19},"max":{"1min":5.306,"5min":6.1159999999999997,"15min":48.728999999999999},"last":0.375},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.85099999999999998,"15min":0.93300000000000005},"min":{"1min":0.22900000000000001,"5min":0.221,"15min":0.221},"max":{"1min":2.8650000000000002,"5min":5.5599999999999996,"15min":49.168999999999997},"last":0.53400000000000003}]},{"osd":1,"last update":"Wed Sep 11 08:16:20 2024","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.0049999999999999,"15min":0.753},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":3.5470000000000002,"5min":4.7320000000000002,"15min":13.207000000000001},"last":0.755},{"interface":"front","average":{"1min":1.3560000000000001,"5min":0.98599999999999999,"15min":0.72199999999999998},"min":{"1min":0.24299999999999999,"5min":0.22900000000000001,"15min":0.22700000000000001},"max":{"1min":5.8869999999999996,"5min":5.8869999999999996,"15min":5.8869999999999996},"last":0.33400000000000002}]},{"osd":2,"last update":"Wed Sep 11 08:16:20 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.0549999999999999,"15min":0.74199999999999999},"min":{"1min":0.245,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":3.3599999999999999,"5min":7.976,"15min":7.976},"last":0.46500000000000002},{"interface":"front","average":{"1min":1.085,"5min":0.871,"15min":0.71499999999999997},"min":{"1min":0.32700000000000001,"5min":0.248,"15min":0.248},"max":{"1min":3.1389999999999998,"5min":3.6989999999999998,"15min":5.5670000000000002},"last":0.34899999999999998}]},{"osd":3,"last update":"Wed Sep 11 08:16:20 2024","interfaces":[{"interface":"back","average":{"1min":1.986,"5min":1.3129999999999999,"15min":1.0349999999999999},"min":{"1min":0.27600000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":19.686,"5min":19.686,"15min":26.884},"last":0.28799999999999998},{"interface":"front","average":{"1min":1.8660000000000001,"5min":1.4550000000000001,"15min":1.0549999999999999},"min":{"1min":0.26600000000000001,"5min":0.246,"15min":0.246},"max":{"1min":19.882000000000001,"5min":19.882000000000001,"15min":26.783000000000001},"last":0.35999999999999999}]},{"osd":5,"last update":"Wed Sep 11 08:15:45 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.497,"15min":0.42299999999999999},"min":{"1min":0.26400000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":2.5489999999999999,"5min":2.5489999999999999,"15min":2.5489999999999999},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.48299999999999998,"15min":0.41499999999999998},"min":{"1min":0.253,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.7390000000000001,"5min":1.9630000000000001,"15min":1.9630000000000001},"last":0.32300000000000001}]},{"osd":6,"last update":"Wed Sep 11 08:16:14 2024","interfaces":[{"interface":"back","average":{"1min":3.048,"5min":1.004,"15min":0.59599999999999997},"min":{"1min":0.28599999999999998,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":48.808,"5min":48.808,"15min":48.808},"last":0.311},{"interface":"front","average":{"1min":2.988,"5min":1.032,"15min":0.64800000000000002},"min":{"1min":0.27800000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":48.746000000000002,"5min":48.746000000000002,"15min":48.746000000000002},"last":0.44}]},{"osd":7,"last update":"Wed Sep 11 08:16:30 2024","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.56499999999999995,"15min":0.50700000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":3.512,"5min":3.512,"15min":3.512},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.69699999999999995,"15min":0.67400000000000004},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":3.4950000000000001,"5min":3.4950000000000001,"15min":3.4950000000000001},"last":0.39600000000000002}]}]},{"osd":3,"up_from":25,"seq":107374182536,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1554316,"kb_used_data":1252612,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":92190836,"statfs":{"total":95995035648,"available":94403416064,"internally_reserved":0,"allocated":1282674688,"data_stored":1282253938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:16:04 2024","interfaces":[{"interface":"back","average":{"1min":4.2320000000000002,"5min":2.363,"15min":1.099},"min":{"1min":0.20200000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":60.469999999999999,"5min":60.469999999999999,"15min":60.469999999999999},"last":0.27800000000000002},{"interface":"front","average":{"1min":4.4939999999999998,"5min":2.613,"15min":1.1990000000000001},"min":{"1min":0.245,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":60.488999999999997,"5min":60.488999999999997,"15min":60.488999999999997},"last":1.5469999999999999}]},{"osd":1,"last update":"Wed Sep 11 08:16:04 2024","interfaces":[{"interface":"back","average":{"1min":4.0620000000000003,"5min":2.2679999999999998,"15min":1.0860000000000001},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.20899999999999999},"max":{"1min":60.578000000000003,"5min":60.578000000000003,"15min":60.578000000000003},"last":1.5680000000000001},{"interface":"front","average":{"1min":4.5910000000000002,"5min":2.411,"15min":1.1830000000000001},"min":{"1min":0.23400000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":63.674999999999997,"5min":63.674999999999997,"15min":63.674999999999997},"last":1.47}]},{"osd":2,"last update":"Wed Sep 11 08:16:04 2024","interfaces":[{"interface":"back","average":{"1min":4.6840000000000002,"5min":2.6139999999999999,"15min":1.222},"min":{"1min":0.22,"5min":0.185,"15min":0.185},"max":{"1min":60.645000000000003,"5min":60.645000000000003,"15min":60.645000000000003},"last":1.829},{"interface":"front","average":{"1min":4.6740000000000004,"5min":2.3740000000000001,"15min":1.1379999999999999},"min":{"1min":0.38600000000000001,"5min":0.20699999999999999,"15min":0.20599999999999999},"max":{"1min":60.723999999999997,"5min":60.723999999999997,"15min":60.723999999999997},"last":1.0129999999999999}]},{"osd":4,"last update":"Wed Sep 11 08:16:23 2024","interfaces":[{"interface":"back","average":{"1min":4.0970000000000004,"5min":1.9339999999999999,"15min":1.0740000000000001},"min":{"1min":0.28100000000000003,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":61.290999999999997,"5min":61.290999999999997,"15min":61.290999999999997},"last":0.628},{"interface":"front","average":{"1min":3.8740000000000001,"5min":1.831,"15min":1.024},"min":{"1min":0.28899999999999998,"5min":0.27500000000000002,"15min":0.27200000000000002},"max":{"1min":60.856000000000002,"5min":60.856000000000002,"15min":60.856000000000002},"last":0.53500000000000003}]},{"osd":5,"last update":"Wed Sep 11 08:15:47 2024","interfaces":[{"interface":"back","average":{"1min":6.5309999999999997,"5min":2.3620000000000001,"15min":1.159},"min":{"1min":0.40000000000000002,"5min":0.26600000000000001,"15min":0.26500000000000001},"max":{"1min":60.914999999999999,"5min":60.914999999999999,"15min":60.914999999999999},"last":2.1400000000000001},{"interface":"front","average":{"1min":6.2590000000000003,"5min":2.3730000000000002,"15min":1.242},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":60.787999999999997,"5min":60.787999999999997,"15min":60.787999999999997},"last":1.8660000000000001}]},{"osd":6,"last update":"Wed Sep 11 08:16:10 2024","interfaces":[{"interface":"back","average":{"1min":4.3019999999999996,"5min":2.4329999999999998,"15min":1.286},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":60.817,"5min":60.817,"15min":60.817},"last":1.657},{"interface":"front","average":{"1min":4.4530000000000003,"5min":2.5169999999999999,"15min":1.371},"min":{"1min":0.505,"5min":0.307,"15min":0.307},"max":{"1min":61.264000000000003,"5min":61.264000000000003,"15min":61.264000000000003},"last":1.9510000000000001}]},{"osd":7,"last update":"Wed Sep 11 08:16:26 2024","interfaces":[{"interface":"back","average":{"1min":3.8570000000000002,"5min":3.2850000000000001,"15min":3.1890000000000001},"min":{"1min":0.39500000000000002,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":60.847000000000001,"5min":60.847000000000001,"15min":60.847000000000001},"last":1.583},{"interface":"front","average":{"1min":3.8769999999999998,"5min":3.2330000000000001,"15min":3.1259999999999999},"min":{"1min":0.442,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":60.883000000000003,"5min":60.883000000000003,"15min":60.883000000000003},"last":0.56599999999999995}]}]},{"osd":2,"up_from":18,"seq":77309411467,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1594928,"kb_used_data":1289256,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":92150224,"statfs":{"total":95995035648,"available":94361829376,"internally_reserved":0,"allocated":1320198144,"data_stored":1319805217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:16:43 2024","interfaces":[{"interface":"back","average":{"1min":1.867,"5min":2.2679999999999998,"15min":1.1699999999999999},"min":{"1min":0.28100000000000003,"5min":0.21299999999999999,"15min":0.17199999999999999},"max":{"1min":8.0920000000000005,"5min":43.539999999999999,"15min":43.539999999999999},"last":1.6419999999999999},{"interface":"front","average":{"1min":1.401,"5min":2.4199999999999999,"15min":1.2090000000000001},"min":{"1min":0.19600000000000001,"5min":0.185,"15min":0.185},"max":{"1min":5.9790000000000001,"5min":41.962000000000003,"15min":41.962000000000003},"last":0.93300000000000005}]},{"osd":1,"last update":"Wed Sep 11 08:16:43 2024","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":1.659,"15min":0.95499999999999996},"min":{"1min":0.28699999999999998,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":5.1539999999999999,"5min":13.138999999999999,"15min":13.138999999999999},"last":1.3129999999999999},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.8360000000000001,"15min":1.0089999999999999},"min":{"1min":0.252,"5min":0.252,"15min":0.20200000000000001},"max":{"1min":5.0220000000000002,"5min":10.167999999999999,"15min":10.167999999999999},"last":0.68000000000000005}]},{"osd":3,"last update":"Wed Sep 11 08:16:05 2024","interfaces":[{"interface":"back","average":{"1min":2.5070000000000001,"5min":1.97,"15min":0.998},"min":{"1min":0.21299999999999999,"5min":0.188,"15min":0.188},"max":{"1min":8.1989999999999998,"5min":8.1989999999999998,"15min":8.1989999999999998},"last":1.2070000000000001},{"interface":"front","average":{"1min":2.4289999999999998,"5min":1.879,"15min":0.97299999999999998},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.22500000000000001},"max":{"1min":8.0640000000000001,"5min":8.3219999999999992,"15min":8.3219999999999992},"last":1.6739999999999999}]},{"osd":4,"last update":"Wed Sep 11 08:16:27 2024","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":1.1839999999999999,"15min":0.81999999999999995},"min":{"1min":0.29799999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":3.2999999999999998,"5min":7.2480000000000002,"15min":7.806},"last":0.48099999999999998},{"interface":"front","average":{"1min":1.1850000000000001,"5min":1.1519999999999999,"15min":0.871},"min":{"1min":0.309,"5min":0.252,"15min":0.252},"max":{"1min":3.855,"5min":5.4740000000000002,"15min":12.129},"last":0.36799999999999999}]},{"osd":5,"last update":"Wed Sep 11 08:15:57 2024","interfaces":[{"interface":"back","average":{"1min":1.6120000000000001,"5min":1.3100000000000001,"15min":0.85199999999999998},"min":{"1min":0.39900000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":5.4539999999999997,"5min":6.7709999999999999,"15min":6.7709999999999999},"last":1.431},{"interface":"front","average":{"1min":1.196,"5min":1.3160000000000001,"15min":0.84599999999999997},"min":{"1min":0.36299999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":4.024,"5min":12.869,"15min":12.869},"last":1.238}]},{"osd":6,"last update":"Wed Sep 11 08:16:10 2024","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.2090000000000001,"15min":0.81599999999999995},"min":{"1min":0.44700000000000001,"5min":0.32600000000000001,"15min":0.26100000000000001},"max":{"1min":4.2539999999999996,"5min":7.2249999999999996,"15min":7.2249999999999996},"last":1.262},{"interface":"front","average":{"1min":1.819,"5min":1.3839999999999999,"15min":0.89700000000000002},"min":{"1min":0.35499999999999998,"5min":0.33200000000000002,"15min":0.32100000000000001},"max":{"1min":5.4939999999999998,"5min":6.71,"15min":6.71},"last":0.42199999999999999}]},{"osd":7,"last update":"Wed Sep 11 08:16:27 2024","interfaces":[{"interface":"back","average":{"1min":1.468,"5min":1.4039999999999999,"15min":1.3939999999999999},"min":{"1min":0.375,"5min":0.37,"15min":0.37},"max":{"1min":4.0780000000000003,"5min":4.9569999999999999,"15min":4.9569999999999999},"last":0.45400000000000001},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.512,"15min":1.528},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":3.8300000000000001,"5min":5.3710000000000004,"15min":5.3710000000000004},"last":0.40200000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574993,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1774592,"kb_used_data":1468920,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":91970560,"statfs":{"total":95995035648,"available":94177853440,"internally_reserved":0,"allocated":1504174080,"data_stored":1503765403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:16:17 2024","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.3779999999999999,"15min":0.90800000000000003},"min":{"1min":0.24099999999999999,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":4.2199999999999998,"5min":8.1509999999999998,"15min":32.478000000000002},"last":0.35699999999999998},{"interface":"front","average":{"1min":1.9970000000000001,"5min":1.9419999999999999,"15min":1.177},"min":{"1min":0.28000000000000003,"5min":0.186,"15min":0.186},"max":{"1min":5.4210000000000003,"5min":10.247,"15min":35.459000000000003},"last":2.0659999999999998}]},{"osd":2,"last update":"Wed Sep 11 08:15:49 2024","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.5800000000000001,"15min":0.79600000000000004},"min":{"1min":0.26400000000000001,"5min":0.216,"15min":0.21299999999999999},"max":{"1min":7.9699999999999998,"5min":8.3819999999999997,"15min":8.3819999999999997},"last":1.827},{"interface":"front","average":{"1min":1.5229999999999999,"5min":1.8779999999999999,"15min":0.92000000000000004},"min":{"1min":0.192,"5min":0.192,"15min":0.16300000000000001},"max":{"1min":5.1319999999999997,"5min":10.276,"15min":10.276},"last":5.2640000000000002}]},{"osd":3,"last update":"Wed Sep 11 08:16:01 2024","interfaces":[{"interface":"back","average":{"1min":1.835,"5min":2.3479999999999999,"15min":1.339},"min":{"1min":0.38600000000000001,"5min":0.248,"15min":0.246},"max":{"1min":5.1689999999999996,"5min":16.783999999999999,"15min":32.759},"last":2.0939999999999999},{"interface":"front","average":{"1min":1.0469999999999999,"5min":2.008,"15min":1.175},"min":{"1min":0.27000000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":4.2489999999999997,"5min":14.973000000000001,"15min":32.444000000000003},"last":0.67900000000000005}]},{"osd":4,"last update":"Wed Sep 11 08:16:27 2024","interfaces":[{"interface":"back","average":{"1min":2.2669999999999999,"5min":1.333,"15min":0.93000000000000005},"min":{"1min":0.29999999999999999,"5min":0.252,"15min":0.252},"max":{"1min":5.8970000000000002,"5min":8.5640000000000001,"15min":9.0139999999999993},"last":1.6140000000000001},{"interface":"front","average":{"1min":1.1850000000000001,"5min":1.129,"15min":0.83999999999999997},"min":{"1min":0.38100000000000001,"5min":0.248,"15min":0.248},"max":{"1min":4.2949999999999999,"5min":8.0609999999999999,"15min":8.2360000000000007},"last":1.7949999999999999}]},{"osd":5,"last update":"Wed Sep 11 08:15:49 2024","interfaces":[{"interface":"back","average":{"1min":1.6859999999999999,"5min":1.4399999999999999,"15min":0.90900000000000003},"min":{"1min":0.316,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":5.3869999999999996,"5min":8.6769999999999996,"15min":9.0470000000000006},"last":1.704},{"interface":"front","average":{"1min":1.8959999999999999,"5min":1.6399999999999999,"15min":0.95399999999999996},"min":{"1min":0.35199999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":4.5019999999999998,"5min":8.9860000000000007,"15min":11.967000000000001},"last":2}]},{"osd":6,"last update":"Wed Sep 11 08:16:03 2024","interfaces":[{"interface":"back","average":{"1min":7.8250000000000002,"5min":2.7599999999999998,"15min":1.3859999999999999},"min":{"1min":0.32600000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":124.31999999999999,"5min":124.31999999999999,"15min":124.31999999999999},"last":2.1549999999999998},{"interface":"front","average":{"1min":7.6020000000000003,"5min":2.6779999999999999,"15min":1.341},"min":{"1min":0.39200000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":124.958,"5min":124.958,"15min":124.958},"last":1.5740000000000001}]},{"osd":7,"last update":"Wed Sep 11 08:16:27 2024","interfaces":[{"interface":"back","average":{"1min":2.5169999999999999,"5min":1.4510000000000001,"15min":1.2729999999999999},"min":{"1min":0.34100000000000003,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":5.9580000000000002,"5min":5.9580000000000002,"15min":5.9580000000000002},"last":1.6739999999999999},{"interface":"front","average":{"1min":2.0019999999999998,"5min":1.5840000000000001,"15min":1.514},"min":{"1min":0.5,"5min":0.377,"15min":0.377},"max":{"1min":4.452,"5min":4.593,"15min":4.593},"last":2.0339999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738517,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1576920,"kb_used_data":1271248,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":92168232,"statfs":{"total":95995035648,"available":94380269568,"internally_reserved":0,"allocated":1301757952,"data_stored":1301326593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 11 08:16:20 2024","interfaces":[{"interface":"back","average":{"1min":1.4770000000000001,"5min":1.5609999999999999,"15min":0.86299999999999999},"min":{"1min":0.19600000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":5.992,"5min":7.2160000000000002,"15min":7.2160000000000002},"last":0.88200000000000001},{"interface":"front","average":{"1min":2.9409999999999998,"5min":2.117,"15min":1.0389999999999999},"min":{"1min":0.29899999999999999,"5min":0.153,"15min":0.153},"max":{"1min":8.3070000000000004,"5min":9.5679999999999996,"15min":9.5679999999999996},"last":0.61699999999999999}]},{"osd":2,"last update":"Wed Sep 11 08:16:44 2024","interfaces":[{"interface":"back","average":{"1min":2.1709999999999998,"5min":1.5640000000000001,"15min":0.92200000000000004},"min":{"1min":0.23899999999999999,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":7.4340000000000002,"5min":10.351000000000001,"15min":10.351000000000001},"last":0.79600000000000004},{"interface":"front","average":{"1min":1.827,"5min":1.639,"15min":0.98299999999999998},"min":{"1min":0.28899999999999998,"5min":0.186,"15min":0.186},"max":{"1min":6.2690000000000001,"5min":7.3090000000000002,"15min":7.3090000000000002},"last":0.85799999999999998}]},{"osd":3,"last update":"Wed Sep 11 08:16:05 2024","interfaces":[{"interface":"back","average":{"1min":1.546,"5min":2.2549999999999999,"15min":1.2130000000000001},"min":{"1min":0.29299999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":4.5679999999999996,"5min":35.491,"15min":35.491},"last":1.3500000000000001},{"interface":"front","average":{"1min":2.2240000000000002,"5min":2.6150000000000002,"15min":1.246},"min":{"1min":0.32900000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.7919999999999998,"5min":37.563000000000002,"15min":37.563000000000002},"last":0.91300000000000003}]},{"osd":4,"last update":"Wed Sep 11 08:16:25 2024","interfaces":[{"interface":"back","average":{"1min":1.6759999999999999,"5min":1.502,"15min":0.94399999999999995},"min":{"1min":0.39300000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":5.0190000000000001,"5min":9.4030000000000005,"15min":9.4030000000000005},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.88900000000000001,"5min":1.2909999999999999,"15min":0.88600000000000001},"min":{"1min":0.34399999999999997,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":2.5899999999999999,"5min":6.6619999999999999,"15min":6.6619999999999999},"last":0.83099999999999996}]},{"osd":5,"last update":"Wed Sep 11 08:15:53 2024","interfaces":[{"interface":"back","average":{"1min":2.4529999999999998,"5min":1.696,"15min":0.98899999999999999},"min":{"1min":0.33100000000000002,"5min":0.255,"15min":0.255},"max":{"1min":24.914999999999999,"5min":24.914999999999999,"15min":24.914999999999999},"last":1.4379999999999999},{"interface":"front","average":{"1min":3.173,"5min":1.9299999999999999,"15min":1.0720000000000001},"min":{"1min":0.34100000000000003,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":25.277000000000001,"5min":25.277000000000001,"15min":25.277000000000001},"last":2.399}]},{"osd":6,"last update":"Wed Sep 11 08:16:10 2024","interfaces":[{"interface":"back","average":{"1min":2.0230000000000001,"5min":2.4950000000000001,"15min":1.306},"min":{"1min":0.29599999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":5.085,"5min":78.522000000000006,"15min":78.522000000000006},"last":0.63800000000000001},{"interface":"front","average":{"1min":1.514,"5min":1.986,"15min":1.145},"min":{"1min":0.29899999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":6.8520000000000003,"5min":79.808000000000007,"15min":79.808000000000007},"last":0.441}]},{"osd":7,"last update":"Wed Sep 11 08:16:25 2024","interfaces":[{"interface":"back","average":{"1min":1.978,"5min":1.4850000000000001,"15min":1.403},"min":{"1min":0.41499999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":5.976,"5min":6.7750000000000004,"15min":6.7750000000000004},"last":1.6619999999999999},{"interface":"front","average":{"1min":1.6299999999999999,"5min":1.4099999999999999,"15min":1.373},"min":{"1min":0.438,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":6.117,"5min":6.8179999999999996,"15min":6.8179999999999996},"last":1.9930000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3425,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1280245760,"data_stored":1280245760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1478164480,"data_stored":1478164480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1290801152,"data_stored":1290797080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1263665152,"data_stored":1263665152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1071972352,"data_stored":1071972352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1977159680,"data_stored":1977155608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1366495232,"data_stored":1366491160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1082134528,"data_stored":1082130456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T08:16:54.559 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:54.597 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:54.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:54.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:54.699 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:16:54.726 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:54.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:54.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:16:54.828 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:16:54.859 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:54.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:54.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:16:54.960 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:54.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:54 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:16:54.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:54 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:16:54.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:54 smithi150 ceph-mon[29276]: 5.16 deep-scrub starts 2024-09-11T08:16:54.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:54 smithi150 ceph-mon[29276]: 5.16 deep-scrub ok 2024-09-11T08:16:54.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:54 smithi150 ceph-mon[29276]: 4.11 scrub starts 2024-09-11T08:16:54.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:54 smithi150 ceph-mon[29276]: 4.11 scrub ok 2024-09-11T08:16:54.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:54 smithi150 ceph-mon[29276]: 6.15 scrub starts 2024-09-11T08:16:54.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:54 smithi150 ceph-mon[29276]: 6.15 scrub ok 2024-09-11T08:16:54.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:54 smithi150 ceph-mon[29276]: osdmap e224: 8 total, 8 up, 8 in 2024-09-11T08:16:54.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:54 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:16:54.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:54 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:16:54.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:54 smithi150 ceph-mon[39612]: 5.16 deep-scrub starts 2024-09-11T08:16:54.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:54 smithi150 ceph-mon[39612]: 5.16 deep-scrub ok 2024-09-11T08:16:54.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:54 smithi150 ceph-mon[39612]: 4.11 scrub starts 2024-09-11T08:16:54.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:54 smithi150 ceph-mon[39612]: 4.11 scrub ok 2024-09-11T08:16:54.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:54 smithi150 ceph-mon[39612]: 6.15 scrub starts 2024-09-11T08:16:54.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:54 smithi150 ceph-mon[39612]: 6.15 scrub ok 2024-09-11T08:16:54.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:54 smithi150 ceph-mon[39612]: osdmap e224: 8 total, 8 up, 8 in 2024-09-11T08:16:55.000 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:55.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:55.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:55.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:54 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:16:55.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:54 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:16:55.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:54 smithi181 ceph-mon[32701]: 5.16 deep-scrub starts 2024-09-11T08:16:55.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:54 smithi181 ceph-mon[32701]: 5.16 deep-scrub ok 2024-09-11T08:16:55.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:54 smithi181 ceph-mon[32701]: 4.11 scrub starts 2024-09-11T08:16:55.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:54 smithi181 ceph-mon[32701]: 4.11 scrub ok 2024-09-11T08:16:55.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:54 smithi181 ceph-mon[32701]: 6.15 scrub starts 2024-09-11T08:16:55.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:54 smithi181 ceph-mon[32701]: 6.15 scrub ok 2024-09-11T08:16:55.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:54 smithi181 ceph-mon[32701]: osdmap e224: 8 total, 8 up, 8 in 2024-09-11T08:16:55.101 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:55.139 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:55.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:55.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:55.241 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:55.268 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:55.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:55.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:55.370 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:16:55.397 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:55.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:55.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:16:55.499 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:55.536 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:55.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:55.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:55.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:16:55 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:16:55] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:16:55.639 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:55.663 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:55.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:55.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:55.765 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:55.801 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:55.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:55.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:55.903 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:55.931 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:55.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:55.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:55.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:55 smithi150 ceph-mon[39612]: 8.0 scrub starts 2024-09-11T08:16:55.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:55 smithi150 ceph-mon[39612]: pgmap v418: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 876 op/s 2024-09-11T08:16:55.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:55 smithi150 ceph-mon[39612]: 5.b scrub starts 2024-09-11T08:16:55.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:55 smithi150 ceph-mon[39612]: 5.b scrub ok 2024-09-11T08:16:55.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:55 smithi150 ceph-mon[39612]: 4.13 scrub starts 2024-09-11T08:16:55.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:55 smithi150 ceph-mon[39612]: 4.13 scrub ok 2024-09-11T08:16:55.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:55 smithi150 ceph-mon[29276]: 8.0 scrub starts 2024-09-11T08:16:55.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:55 smithi150 ceph-mon[29276]: pgmap v418: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 876 op/s 2024-09-11T08:16:55.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:55 smithi150 ceph-mon[29276]: 5.b scrub starts 2024-09-11T08:16:55.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:55 smithi150 ceph-mon[29276]: 5.b scrub ok 2024-09-11T08:16:55.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:55 smithi150 ceph-mon[29276]: 4.13 scrub starts 2024-09-11T08:16:55.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:55 smithi150 ceph-mon[29276]: 4.13 scrub ok 2024-09-11T08:16:56.032 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:16:56.059 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:56.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:56.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:16:56.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:55 smithi181 ceph-mon[32701]: 8.0 scrub starts 2024-09-11T08:16:56.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:55 smithi181 ceph-mon[32701]: pgmap v418: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 876 op/s 2024-09-11T08:16:56.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:55 smithi181 ceph-mon[32701]: 5.b scrub starts 2024-09-11T08:16:56.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:55 smithi181 ceph-mon[32701]: 5.b scrub ok 2024-09-11T08:16:56.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:55 smithi181 ceph-mon[32701]: 4.13 scrub starts 2024-09-11T08:16:56.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:55 smithi181 ceph-mon[32701]: 4.13 scrub ok 2024-09-11T08:16:56.162 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:56.197 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:56.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:56.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:56.299 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:16:56.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:56.339 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:56.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:16:56.440 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:16:56.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:56.467 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:56.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:16:56.568 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:16:56.605 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:56.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:56.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:16:56.706 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:56.738 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:56.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:56.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:56.840 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:56.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:56.877 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:56.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:56.978 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:16:56.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:56 smithi150 ceph-mon[29276]: 8.0 scrub ok 2024-09-11T08:16:56.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:56 smithi150 ceph-mon[29276]: 3.19 scrub starts 2024-09-11T08:16:56.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:56 smithi150 ceph-mon[29276]: 3.19 scrub ok 2024-09-11T08:16:56.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:56 smithi150 ceph-mon[29276]: 2.0 deep-scrub starts 2024-09-11T08:16:56.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:56 smithi150 ceph-mon[29276]: 2.0 deep-scrub ok 2024-09-11T08:16:56.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:56 smithi150 ceph-mon[29276]: 4.1a scrub starts 2024-09-11T08:16:56.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:56 smithi150 ceph-mon[29276]: 4.1a scrub ok 2024-09-11T08:16:56.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:56 smithi150 ceph-mon[39612]: 8.0 scrub ok 2024-09-11T08:16:56.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:56 smithi150 ceph-mon[39612]: 3.19 scrub starts 2024-09-11T08:16:56.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:56 smithi150 ceph-mon[39612]: 3.19 scrub ok 2024-09-11T08:16:56.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:56 smithi150 ceph-mon[39612]: 2.0 deep-scrub starts 2024-09-11T08:16:56.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:56 smithi150 ceph-mon[39612]: 2.0 deep-scrub ok 2024-09-11T08:16:56.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:56 smithi150 ceph-mon[39612]: 4.1a scrub starts 2024-09-11T08:16:56.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:56 smithi150 ceph-mon[39612]: 4.1a scrub ok 2024-09-11T08:16:57.009 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:57.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:57.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:16:57.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:56 smithi181 ceph-mon[32701]: 8.0 scrub ok 2024-09-11T08:16:57.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:56 smithi181 ceph-mon[32701]: 3.19 scrub starts 2024-09-11T08:16:57.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:56 smithi181 ceph-mon[32701]: 3.19 scrub ok 2024-09-11T08:16:57.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:56 smithi181 ceph-mon[32701]: 2.0 deep-scrub starts 2024-09-11T08:16:57.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:56 smithi181 ceph-mon[32701]: 2.0 deep-scrub ok 2024-09-11T08:16:57.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:56 smithi181 ceph-mon[32701]: 4.1a scrub starts 2024-09-11T08:16:57.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:56 smithi181 ceph-mon[32701]: 4.1a scrub ok 2024-09-11T08:16:57.111 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:57.142 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:57.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:57.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:57.244 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:57.276 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:57.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:57.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:57.354 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-11T08:16:57.378 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:57.408 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:57.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:57.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:57.510 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:57.540 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:57.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:57.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:57.559 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T08:16:57.642 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:57.702 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:57.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:57.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:57.755 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T08:16:57.805 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:57.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:57.910 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:57.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:57.956 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:16:57.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:57 smithi150 ceph-mon[29276]: 2.2 scrub starts 2024-09-11T08:16:57.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:57 smithi150 ceph-mon[29276]: 2.2 scrub ok 2024-09-11T08:16:57.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:57 smithi150 ceph-mon[29276]: 4.1e scrub starts 2024-09-11T08:16:57.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:57 smithi150 ceph-mon[29276]: 4.1e scrub ok 2024-09-11T08:16:57.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:57 smithi150 ceph-mon[29276]: pgmap v419: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 701 B/s rd, 0 B/s wr, 720 op/s 2024-09-11T08:16:57.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:57 smithi150 ceph-mon[39612]: 2.2 scrub starts 2024-09-11T08:16:57.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:57 smithi150 ceph-mon[39612]: 2.2 scrub ok 2024-09-11T08:16:57.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:57 smithi150 ceph-mon[39612]: 4.1e scrub starts 2024-09-11T08:16:57.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:57 smithi150 ceph-mon[39612]: 4.1e scrub ok 2024-09-11T08:16:57.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:57 smithi150 ceph-mon[39612]: pgmap v419: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 701 B/s rd, 0 B/s wr, 720 op/s 2024-09-11T08:16:58.011 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:58.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:58.046 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:58.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:58.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:57 smithi181 ceph-mon[32701]: 2.2 scrub starts 2024-09-11T08:16:58.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:57 smithi181 ceph-mon[32701]: 2.2 scrub ok 2024-09-11T08:16:58.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:57 smithi181 ceph-mon[32701]: 4.1e scrub starts 2024-09-11T08:16:58.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:57 smithi181 ceph-mon[32701]: 4.1e scrub ok 2024-09-11T08:16:58.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:57 smithi181 ceph-mon[32701]: pgmap v419: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 701 B/s rd, 0 B/s wr, 720 op/s 2024-09-11T08:16:58.147 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:58.178 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:58.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:58.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:58.280 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:16:58.305 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:58.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:58.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:16:58.408 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:58.446 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:58.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:58.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:58.548 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:16:58.557 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:16:58.557 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:16:58.570 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:16:58.573 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:16:58 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:16:58.574 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:58.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:58.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:16:58.581 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:16:58.598 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:16:58.598 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:16:58.599 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:16:58 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:16:58] "GET /metrics HTTP/1.1" 200 224202 "" "Prometheus/2.43.0" 2024-09-11T08:16:58.632 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:16:58.632 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:16:58.661 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:16:58.661 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:16:58.675 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:16:58.691 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:16:58.691 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:16:58.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:58.721 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:58.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:16:58.724 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:16:58.724 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:16:58.750 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:16:58.750 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:16:58.777 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:16:58.777 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:16:58.822 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:16:58.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:58.848 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:58.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:16:58.850 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:58 smithi150 ceph-mon[39612]: 3.10 scrub starts 2024-09-11T08:16:58.850 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:58 smithi150 ceph-mon[39612]: 3.10 scrub ok 2024-09-11T08:16:58.850 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:58 smithi150 ceph-mon[39612]: 6.1a scrub starts 2024-09-11T08:16:58.850 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:58 smithi150 ceph-mon[39612]: 6.1a scrub ok 2024-09-11T08:16:58.850 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:58 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:16:58.850 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:58 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3688992060' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:16:58.850 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:58 smithi150 ceph-mon[29276]: 3.10 scrub starts 2024-09-11T08:16:58.850 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:58 smithi150 ceph-mon[29276]: 3.10 scrub ok 2024-09-11T08:16:58.851 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:58 smithi150 ceph-mon[29276]: 6.1a scrub starts 2024-09-11T08:16:58.851 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:58 smithi150 ceph-mon[29276]: 6.1a scrub ok 2024-09-11T08:16:58.851 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:58 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:16:58.851 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:58 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3688992060' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:16:58.949 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:16:58.974 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:58.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:58.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:16:59.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:58 smithi181 ceph-mon[32701]: 3.10 scrub starts 2024-09-11T08:16:59.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:58 smithi181 ceph-mon[32701]: 3.10 scrub ok 2024-09-11T08:16:59.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:58 smithi181 ceph-mon[32701]: 6.1a scrub starts 2024-09-11T08:16:59.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:58 smithi181 ceph-mon[32701]: 6.1a scrub ok 2024-09-11T08:16:59.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:58 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:16:59.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:58 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3688992060' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:16:59.076 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:59.109 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:59.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:59.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:59.210 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:16:59.245 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:59.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:59.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:16:59.347 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:16:59.383 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:59.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:59.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:16:59.484 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:16:59.512 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:59.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:59.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:16:59.614 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:59.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:59.651 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:59.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:59.752 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:16:59.781 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:16:59.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:59.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:16:59.882 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:16:59.909 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:16:59.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:16:59.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:16:59.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:59 smithi150 ceph-mon[29276]: 6.1 deep-scrub starts 2024-09-11T08:16:59.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:59 smithi150 ceph-mon[29276]: 6.1 deep-scrub ok 2024-09-11T08:16:59.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:59 smithi150 ceph-mon[29276]: 8.f scrub starts 2024-09-11T08:16:59.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:59 smithi150 ceph-mon[29276]: 6.5 scrub starts 2024-09-11T08:16:59.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:59 smithi150 ceph-mon[29276]: 6.5 scrub ok 2024-09-11T08:16:59.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:59 smithi150 ceph-mon[29276]: 4.3 scrub starts 2024-09-11T08:16:59.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:59 smithi150 ceph-mon[29276]: 4.3 scrub ok 2024-09-11T08:16:59.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:16:59 smithi150 ceph-mon[29276]: pgmap v420: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.6 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-09-11T08:16:59.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:59 smithi150 ceph-mon[39612]: 6.1 deep-scrub starts 2024-09-11T08:16:59.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:59 smithi150 ceph-mon[39612]: 6.1 deep-scrub ok 2024-09-11T08:16:59.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:59 smithi150 ceph-mon[39612]: 8.f scrub starts 2024-09-11T08:16:59.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:59 smithi150 ceph-mon[39612]: 6.5 scrub starts 2024-09-11T08:16:59.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:59 smithi150 ceph-mon[39612]: 6.5 scrub ok 2024-09-11T08:16:59.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:59 smithi150 ceph-mon[39612]: 4.3 scrub starts 2024-09-11T08:16:59.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:59 smithi150 ceph-mon[39612]: 4.3 scrub ok 2024-09-11T08:16:59.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:16:59 smithi150 ceph-mon[39612]: pgmap v420: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.6 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-09-11T08:17:00.010 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:00.046 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:00.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:00.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:00.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:59 smithi181 ceph-mon[32701]: 6.1 deep-scrub starts 2024-09-11T08:17:00.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:59 smithi181 ceph-mon[32701]: 6.1 deep-scrub ok 2024-09-11T08:17:00.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:59 smithi181 ceph-mon[32701]: 8.f scrub starts 2024-09-11T08:17:00.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:59 smithi181 ceph-mon[32701]: 6.5 scrub starts 2024-09-11T08:17:00.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:59 smithi181 ceph-mon[32701]: 6.5 scrub ok 2024-09-11T08:17:00.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:59 smithi181 ceph-mon[32701]: 4.3 scrub starts 2024-09-11T08:17:00.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:59 smithi181 ceph-mon[32701]: 4.3 scrub ok 2024-09-11T08:17:00.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:16:59 smithi181 ceph-mon[32701]: pgmap v420: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.6 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-09-11T08:17:00.148 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:00.177 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:00.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:00.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:00.279 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:00.305 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:00.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:00.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:00.407 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:00.434 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:00.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:00.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:00.535 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:00.574 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:00.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:00.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:00.676 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:17:00.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:00.722 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:00.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:17:00.823 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:00.849 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:00.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:00.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:00.951 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:00.981 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:00.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:00.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:01.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:00 smithi181 ceph-mon[32701]: 8.f scrub ok 2024-09-11T08:17:01.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:00 smithi181 ceph-mon[32701]: 5.19 deep-scrub starts 2024-09-11T08:17:01.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:00 smithi181 ceph-mon[32701]: 5.19 deep-scrub ok 2024-09-11T08:17:01.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:00 smithi181 ceph-mon[32701]: 6.a scrub starts 2024-09-11T08:17:01.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:00 smithi181 ceph-mon[32701]: 6.a scrub ok 2024-09-11T08:17:01.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:00 smithi181 ceph-mon[32701]: 5.17 scrub starts 2024-09-11T08:17:01.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:00 smithi181 ceph-mon[32701]: 5.17 scrub ok 2024-09-11T08:17:01.083 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:01.117 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:01.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:01.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:01.219 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:17:01.230 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:00 smithi150 ceph-mon[29276]: 8.f scrub ok 2024-09-11T08:17:01.230 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:00 smithi150 ceph-mon[29276]: 5.19 deep-scrub starts 2024-09-11T08:17:01.230 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:00 smithi150 ceph-mon[29276]: 5.19 deep-scrub ok 2024-09-11T08:17:01.230 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:00 smithi150 ceph-mon[29276]: 6.a scrub starts 2024-09-11T08:17:01.230 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:00 smithi150 ceph-mon[29276]: 6.a scrub ok 2024-09-11T08:17:01.230 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:00 smithi150 ceph-mon[29276]: 5.17 scrub starts 2024-09-11T08:17:01.230 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:00 smithi150 ceph-mon[29276]: 5.17 scrub ok 2024-09-11T08:17:01.231 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:00 smithi150 ceph-mon[39612]: 8.f scrub ok 2024-09-11T08:17:01.231 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:00 smithi150 ceph-mon[39612]: 5.19 deep-scrub starts 2024-09-11T08:17:01.231 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:00 smithi150 ceph-mon[39612]: 5.19 deep-scrub ok 2024-09-11T08:17:01.231 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:00 smithi150 ceph-mon[39612]: 6.a scrub starts 2024-09-11T08:17:01.231 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:00 smithi150 ceph-mon[39612]: 6.a scrub ok 2024-09-11T08:17:01.231 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:00 smithi150 ceph-mon[39612]: 5.17 scrub starts 2024-09-11T08:17:01.231 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:00 smithi150 ceph-mon[39612]: 5.17 scrub ok 2024-09-11T08:17:01.247 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:01.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:01.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:17:01.349 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:17:01.386 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:01.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:01.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:17:01.487 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:17:01.518 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:01.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:01.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:17:01.620 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:01.649 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:01.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:01.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:01.651 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T08:17:01.750 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:01.778 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:01.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:01.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:01.879 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:17:01.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:01.921 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:01.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:17:01.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:01 smithi150 ceph-mon[29276]: 5.2 scrub starts 2024-09-11T08:17:01.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:01 smithi150 ceph-mon[29276]: 5.2 scrub ok 2024-09-11T08:17:01.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:01 smithi150 ceph-mon[29276]: 4.f scrub starts 2024-09-11T08:17:01.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:01 smithi150 ceph-mon[29276]: 4.f scrub ok 2024-09-11T08:17:01.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:01 smithi150 ceph-mon[29276]: pgmap v421: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.6 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.27k op/s 2024-09-11T08:17:01.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:01 smithi150 ceph-mon[29276]: 4.c deep-scrub starts 2024-09-11T08:17:01.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:01 smithi150 ceph-mon[29276]: 4.c deep-scrub ok 2024-09-11T08:17:01.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:01 smithi150 ceph-mon[39612]: 5.2 scrub starts 2024-09-11T08:17:01.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:01 smithi150 ceph-mon[39612]: 5.2 scrub ok 2024-09-11T08:17:01.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:01 smithi150 ceph-mon[39612]: 4.f scrub starts 2024-09-11T08:17:01.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:01 smithi150 ceph-mon[39612]: 4.f scrub ok 2024-09-11T08:17:01.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:01 smithi150 ceph-mon[39612]: pgmap v421: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.6 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.27k op/s 2024-09-11T08:17:01.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:01 smithi150 ceph-mon[39612]: 4.c deep-scrub starts 2024-09-11T08:17:01.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:01 smithi150 ceph-mon[39612]: 4.c deep-scrub ok 2024-09-11T08:17:02.022 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:02.047 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:02.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:02.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:02.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:01 smithi181 ceph-mon[32701]: 5.2 scrub starts 2024-09-11T08:17:02.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:01 smithi181 ceph-mon[32701]: 5.2 scrub ok 2024-09-11T08:17:02.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:01 smithi181 ceph-mon[32701]: 4.f scrub starts 2024-09-11T08:17:02.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:01 smithi181 ceph-mon[32701]: 4.f scrub ok 2024-09-11T08:17:02.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:01 smithi181 ceph-mon[32701]: pgmap v421: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.6 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.27k op/s 2024-09-11T08:17:02.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:01 smithi181 ceph-mon[32701]: 4.c deep-scrub starts 2024-09-11T08:17:02.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:01 smithi181 ceph-mon[32701]: 4.c deep-scrub ok 2024-09-11T08:17:02.148 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:17:02.181 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:02.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:02.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:17:02.283 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:17:02.320 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:02.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:02.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:17:02.422 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:02.458 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:02.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:02.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:02.560 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:02.586 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:02.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:02.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:02.688 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:02.722 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:02.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:02.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:02.824 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:02.861 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:02.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:02.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:02.915 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:02 smithi150 ceph-mon[29276]: 5.f deep-scrub starts 2024-09-11T08:17:02.915 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:02 smithi150 ceph-mon[29276]: 5.f deep-scrub ok 2024-09-11T08:17:02.915 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:02 smithi150 ceph-mon[29276]: 5.3 scrub starts 2024-09-11T08:17:02.915 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:02 smithi150 ceph-mon[29276]: 5.3 scrub ok 2024-09-11T08:17:02.915 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:02 smithi150 ceph-mon[29276]: 6.1c scrub starts 2024-09-11T08:17:02.915 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:02 smithi150 ceph-mon[29276]: 6.1c scrub ok 2024-09-11T08:17:02.915 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:02 smithi150 ceph-mon[29276]: 8.c scrub starts 2024-09-11T08:17:02.916 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:02 smithi150 ceph-mon[39612]: 5.f deep-scrub starts 2024-09-11T08:17:02.916 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:02 smithi150 ceph-mon[39612]: 5.f deep-scrub ok 2024-09-11T08:17:02.916 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:02 smithi150 ceph-mon[39612]: 5.3 scrub starts 2024-09-11T08:17:02.916 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:02 smithi150 ceph-mon[39612]: 5.3 scrub ok 2024-09-11T08:17:02.916 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:02 smithi150 ceph-mon[39612]: 6.1c scrub starts 2024-09-11T08:17:02.916 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:02 smithi150 ceph-mon[39612]: 6.1c scrub ok 2024-09-11T08:17:02.916 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:02 smithi150 ceph-mon[39612]: 8.c scrub starts 2024-09-11T08:17:02.963 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:02.991 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:02.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:02.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:03.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:02 smithi181 ceph-mon[32701]: 5.f deep-scrub starts 2024-09-11T08:17:03.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:02 smithi181 ceph-mon[32701]: 5.f deep-scrub ok 2024-09-11T08:17:03.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:02 smithi181 ceph-mon[32701]: 5.3 scrub starts 2024-09-11T08:17:03.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:02 smithi181 ceph-mon[32701]: 5.3 scrub ok 2024-09-11T08:17:03.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:02 smithi181 ceph-mon[32701]: 6.1c scrub starts 2024-09-11T08:17:03.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:02 smithi181 ceph-mon[32701]: 6.1c scrub ok 2024-09-11T08:17:03.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:02 smithi181 ceph-mon[32701]: 8.c scrub starts 2024-09-11T08:17:03.093 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:03.119 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:03.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:03.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:03.221 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:17:03.245 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:17:03.260 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:03.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:03.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:17:03.362 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:03.387 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:03.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:03.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:03.488 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:17:03.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:03.530 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:03.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:17:03.602 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:17:03.610 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-09-11T08:17:03.632 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:03.671 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:03.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:03.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:03.773 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:17:03.799 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:03.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:03.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:17:03.901 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:17:03.932 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:03.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:03.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:17:04.033 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:17:04.067 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:03 smithi150 ceph-mon[29276]: 3.0 scrub starts 2024-09-11T08:17:04.067 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:03 smithi150 ceph-mon[29276]: 3.0 scrub ok 2024-09-11T08:17:04.067 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:03 smithi150 ceph-mon[29276]: 8.c scrub ok 2024-09-11T08:17:04.067 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:03 smithi150 ceph-mon[29276]: 3.c scrub starts 2024-09-11T08:17:04.067 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:03 smithi150 ceph-mon[29276]: 3.c scrub ok 2024-09-11T08:17:04.067 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:03 smithi150 ceph-mon[29276]: pgmap v422: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s 2024-09-11T08:17:04.068 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:03 smithi150 ceph-mon[39612]: 3.0 scrub starts 2024-09-11T08:17:04.068 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:03 smithi150 ceph-mon[39612]: 3.0 scrub ok 2024-09-11T08:17:04.068 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:03 smithi150 ceph-mon[39612]: 8.c scrub ok 2024-09-11T08:17:04.068 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:03 smithi150 ceph-mon[39612]: 3.c scrub starts 2024-09-11T08:17:04.068 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:03 smithi150 ceph-mon[39612]: 3.c scrub ok 2024-09-11T08:17:04.068 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:03 smithi150 ceph-mon[39612]: pgmap v422: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s 2024-09-11T08:17:04.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:04.069 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:04.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:17:04.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:03 smithi181 ceph-mon[32701]: 3.0 scrub starts 2024-09-11T08:17:04.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:03 smithi181 ceph-mon[32701]: 3.0 scrub ok 2024-09-11T08:17:04.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:03 smithi181 ceph-mon[32701]: 8.c scrub ok 2024-09-11T08:17:04.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:03 smithi181 ceph-mon[32701]: 3.c scrub starts 2024-09-11T08:17:04.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:03 smithi181 ceph-mon[32701]: 3.c scrub ok 2024-09-11T08:17:04.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:03 smithi181 ceph-mon[32701]: pgmap v422: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s 2024-09-11T08:17:04.170 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:17:04.202 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:04.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:04.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:17:04.304 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:04.330 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:04.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:04.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:04.432 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:17:04.465 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:04.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:04.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:17:04.567 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:04.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:04.594 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:04.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:04.695 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:04.724 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:04.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:04.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:04.815 INFO:teuthology.orchestra.run.smithi150.stderr:noscrub is set 2024-09-11T08:17:04.826 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:04.849 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:04.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:04.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:04.951 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:04.998 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:04.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:04.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:05.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:04 smithi181 ceph-mon[32701]: 5.12 scrub starts 2024-09-11T08:17:05.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:04 smithi181 ceph-mon[32701]: 5.12 scrub ok 2024-09-11T08:17:05.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:04 smithi181 ceph-mon[32701]: from='client.25169 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:17:05.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:04 smithi181 ceph-mon[32701]: 4.0 scrub starts 2024-09-11T08:17:05.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:04 smithi181 ceph-mon[32701]: 4.0 scrub ok 2024-09-11T08:17:05.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:04 smithi181 ceph-mon[32701]: 3.13 scrub starts 2024-09-11T08:17:05.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:04 smithi181 ceph-mon[32701]: 3.13 scrub ok 2024-09-11T08:17:05.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:04 smithi181 ceph-mon[32701]: 6.9 scrub starts 2024-09-11T08:17:05.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:04 smithi181 ceph-mon[32701]: 6.9 scrub ok 2024-09-11T08:17:05.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:04 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2401913996' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:17:05.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:04 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:17:05.100 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:05.110 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:04 smithi150 ceph-mon[29276]: 5.12 scrub starts 2024-09-11T08:17:05.110 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:04 smithi150 ceph-mon[29276]: 5.12 scrub ok 2024-09-11T08:17:05.110 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:04 smithi150 ceph-mon[29276]: from='client.25169 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:17:05.110 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:04 smithi150 ceph-mon[29276]: 4.0 scrub starts 2024-09-11T08:17:05.110 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:04 smithi150 ceph-mon[29276]: 4.0 scrub ok 2024-09-11T08:17:05.111 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:04 smithi150 ceph-mon[29276]: 3.13 scrub starts 2024-09-11T08:17:05.111 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:04 smithi150 ceph-mon[29276]: 3.13 scrub ok 2024-09-11T08:17:05.111 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:04 smithi150 ceph-mon[29276]: 6.9 scrub starts 2024-09-11T08:17:05.111 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:04 smithi150 ceph-mon[29276]: 6.9 scrub ok 2024-09-11T08:17:05.111 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:04 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2401913996' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:17:05.111 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:04 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:17:05.111 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:04 smithi150 ceph-mon[39612]: 5.12 scrub starts 2024-09-11T08:17:05.111 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:04 smithi150 ceph-mon[39612]: 5.12 scrub ok 2024-09-11T08:17:05.111 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:04 smithi150 ceph-mon[39612]: from='client.25169 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:17:05.111 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:04 smithi150 ceph-mon[39612]: 4.0 scrub starts 2024-09-11T08:17:05.111 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:04 smithi150 ceph-mon[39612]: 4.0 scrub ok 2024-09-11T08:17:05.112 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:04 smithi150 ceph-mon[39612]: 3.13 scrub starts 2024-09-11T08:17:05.112 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:04 smithi150 ceph-mon[39612]: 3.13 scrub ok 2024-09-11T08:17:05.112 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:04 smithi150 ceph-mon[39612]: 6.9 scrub starts 2024-09-11T08:17:05.112 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:04 smithi150 ceph-mon[39612]: 6.9 scrub ok 2024-09-11T08:17:05.112 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:04 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2401913996' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:17:05.112 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:04 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:17:05.140 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:05.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:05.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:05.241 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:17:05.280 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:05.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:05.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:17:05.382 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:17:05.419 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:05.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:05.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:17:05.521 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:05.558 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:05.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:05.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:05.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:17:05 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:17:05] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:17:05.660 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:05.694 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:05.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:05.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:05.796 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:17:05.830 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:05.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:05.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:17:05.932 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:05.962 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:05.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:05.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:06.063 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:06.101 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:06.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:06.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:06.202 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:06.231 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:06.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:06.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:06.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:05 smithi150 ceph-mon[29276]: 8.6 deep-scrub starts 2024-09-11T08:17:06.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:05 smithi150 ceph-mon[29276]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:17:06.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:05 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:17:06.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:05 smithi150 ceph-mon[29276]: osdmap e225: 8 total, 8 up, 8 in 2024-09-11T08:17:06.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:05 smithi150 ceph-mon[29276]: pgmap v424: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.2 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-09-11T08:17:06.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:05 smithi150 ceph-mon[29276]: osdmap e226: 8 total, 8 up, 8 in 2024-09-11T08:17:06.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:05 smithi150 ceph-mon[39612]: 8.6 deep-scrub starts 2024-09-11T08:17:06.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:05 smithi150 ceph-mon[39612]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:17:06.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:05 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:17:06.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:05 smithi150 ceph-mon[39612]: osdmap e225: 8 total, 8 up, 8 in 2024-09-11T08:17:06.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:05 smithi150 ceph-mon[39612]: pgmap v424: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.2 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-09-11T08:17:06.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:05 smithi150 ceph-mon[39612]: osdmap e226: 8 total, 8 up, 8 in 2024-09-11T08:17:06.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:05 smithi181 ceph-mon[32701]: 8.6 deep-scrub starts 2024-09-11T08:17:06.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:05 smithi181 ceph-mon[32701]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:17:06.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:05 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:17:06.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:05 smithi181 ceph-mon[32701]: osdmap e225: 8 total, 8 up, 8 in 2024-09-11T08:17:06.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:05 smithi181 ceph-mon[32701]: pgmap v424: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.2 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-09-11T08:17:06.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:05 smithi181 ceph-mon[32701]: osdmap e226: 8 total, 8 up, 8 in 2024-09-11T08:17:06.333 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:06.364 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:06.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:06.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:06.466 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:06.500 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:06.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:06.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:06.603 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:17:06.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:06.635 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:06.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:17:06.636 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":422,"stamp":"2024-09-11T08:17:02.893130+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2681084148,"num_objects":41098,"num_object_clones":0,"num_object_copies":123294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41098,"num_whiteouts":0,"num_read":989,"num_read_kb":848,"num_write":217301,"num_write_kb":8246728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17905,"num_bytes_recovered":1165103644,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85867,"ondisk_log_size":85867,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":446,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":10769748,"kb_used_data":8308308,"kb_used_omap":19,"kb_used_meta":2461292,"kb_avail":739191468,"statfs":{"total":767960285184,"available":756932063232,"internally_reserved":0,"allocated":8507707392,"data_stored":8504324452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19955,"internal_metadata":2520363533},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-672727040,"num_objects":-10265,"num_object_clones":0,"num_object_copies":-30795,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-10265,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10265,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003586"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":214,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.755083+0000","last_change":"2024-09-11T08:16:37.667413+0000","last_active":"2024-09-11T08:16:52.755083+0000","last_peered":"2024-09-11T08:16:52.755083+0000","last_clean":"2024-09-11T08:16:52.755083+0000","last_became_active":"2024-09-11T08:07:30.044976+0000","last_became_peered":"2024-09-11T08:07:30.044976+0000","last_unstale":"2024-09-11T08:16:52.755083+0000","last_undegraded":"2024-09-11T08:16:52.755083+0000","last_fullsized":"2024-09-11T08:16:52.755083+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:37.667335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:16:37.667335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:40.583337+0000","scrub_duration":0.0048271249999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":218,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.745736+0000","last_change":"2024-09-11T08:15:37.106907+0000","last_active":"2024-09-11T08:16:52.745736+0000","last_peered":"2024-09-11T08:16:52.745736+0000","last_clean":"2024-09-11T08:16:52.745736+0000","last_became_active":"2024-09-11T08:07:32.048048+0000","last_became_peered":"2024-09-11T08:07:32.048048+0000","last_unstale":"2024-09-11T08:16:52.745736+0000","last_undegraded":"2024-09-11T08:16:52.745736+0000","last_fullsized":"2024-09-11T08:16:52.745736+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:15:37.106824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:15:37.106824+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038285900000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":247,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.742324+0000","last_change":"2024-09-11T08:15:56.531271+0000","last_active":"2024-09-11T08:16:52.742324+0000","last_peered":"2024-09-11T08:16:52.742324+0000","last_clean":"2024-09-11T08:16:52.742324+0000","last_became_active":"2024-09-11T08:15:56.530946+0000","last_became_peered":"2024-09-11T08:15:56.530946+0000","last_unstale":"2024-09-11T08:16:52.742324+0000","last_undegraded":"2024-09-11T08:16:52.742324+0000","last_fullsized":"2024-09-11T08:16:52.742324+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:57.020218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:14:57.020218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034881460000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":208,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T08:16:58.143031+0000","last_change":"2024-09-11T08:16:58.143005+0000","last_active":"2024-09-11T08:16:58.143031+0000","last_peered":"2024-09-11T08:16:58.143031+0000","last_clean":"2024-09-11T08:16:58.143031+0000","last_became_active":"2024-09-11T08:07:36.074789+0000","last_became_peered":"2024-09-11T08:07:36.074789+0000","last_unstale":"2024-09-11T08:16:58.143031+0000","last_undegraded":"2024-09-11T08:16:58.143031+0000","last_fullsized":"2024-09-11T08:16:58.143031+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:58.142957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:02.470649+0000","last_clean_scrub_stamp":"2024-09-11T08:16:58.142957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:22.869601+0000","scrub_duration":0.001328805,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":225,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.741649+0000","last_change":"2024-09-11T08:15:56.521398+0000","last_active":"2024-09-11T08:16:52.741649+0000","last_peered":"2024-09-11T08:16:52.741649+0000","last_clean":"2024-09-11T08:16:52.741649+0000","last_became_active":"2024-09-11T08:15:56.521104+0000","last_became_peered":"2024-09-11T08:15:56.521104+0000","last_unstale":"2024-09-11T08:16:52.741649+0000","last_undegraded":"2024-09-11T08:16:52.741649+0000","last_fullsized":"2024-09-11T08:16:52.741649+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:58.222356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:14:58.222356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013526233,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":204,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.743838+0000","last_change":"2024-09-11T08:15:31.254961+0000","last_active":"2024-09-11T08:16:52.743838+0000","last_peered":"2024-09-11T08:16:52.743838+0000","last_clean":"2024-09-11T08:16:52.743838+0000","last_became_active":"2024-09-11T08:07:30.045579+0000","last_became_peered":"2024-09-11T08:07:30.045579+0000","last_unstale":"2024-09-11T08:16:52.743838+0000","last_undegraded":"2024-09-11T08:16:52.743838+0000","last_fullsized":"2024-09-11T08:16:52.743838+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:31.254885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:31.254885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033689100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"145'22","reported_seq":244,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.740621+0000","last_change":"2024-09-11T08:15:57.478927+0000","last_active":"2024-09-11T08:16:52.740621+0000","last_peered":"2024-09-11T08:16:52.740621+0000","last_clean":"2024-09-11T08:16:52.740621+0000","last_became_active":"2024-09-11T08:07:32.045848+0000","last_became_peered":"2024-09-11T08:07:32.045848+0000","last_unstale":"2024-09-11T08:16:52.740621+0000","last_undegraded":"2024-09-11T08:16:52.740621+0000","last_fullsized":"2024-09-11T08:16:52.740621+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'22","last_scrub_stamp":"2024-09-11T08:15:57.478870+0000","last_deep_scrub":"145'22","last_deep_scrub_stamp":"2024-09-11T08:15:57.478870+0000","last_clean_scrub_stamp":"2024-09-11T08:15:57.478870+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:13.662088+0000","scrub_duration":0.0021529710000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":200,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.745708+0000","last_change":"2024-09-11T08:15:52.050101+0000","last_active":"2024-09-11T08:16:52.745708+0000","last_peered":"2024-09-11T08:16:52.745708+0000","last_clean":"2024-09-11T08:16:52.745708+0000","last_became_active":"2024-09-11T08:07:34.052243+0000","last_became_peered":"2024-09-11T08:07:34.052243+0000","last_unstale":"2024-09-11T08:16:52.745708+0000","last_undegraded":"2024-09-11T08:16:52.745708+0000","last_fullsized":"2024-09-11T08:16:52.745708+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:52.050017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:15:52.050017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:20.035807+0000","scrub_duration":0.0037049880000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":216,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.740557+0000","last_change":"2024-09-11T08:15:31.550213+0000","last_active":"2024-09-11T08:16:52.740557+0000","last_peered":"2024-09-11T08:16:52.740557+0000","last_clean":"2024-09-11T08:16:52.740557+0000","last_became_active":"2024-09-11T08:07:30.043950+0000","last_became_peered":"2024-09-11T08:07:30.043950+0000","last_unstale":"2024-09-11T08:16:52.740557+0000","last_undegraded":"2024-09-11T08:16:52.740557+0000","last_fullsized":"2024-09-11T08:16:52.740557+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:31.550091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:54.878409+0000","last_clean_scrub_stamp":"2024-09-11T08:15:31.550091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:56.355754+0000","scrub_duration":0.0017129999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":228,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T08:16:56.115610+0000","last_change":"2024-09-11T08:16:56.115531+0000","last_active":"2024-09-11T08:16:56.115610+0000","last_peered":"2024-09-11T08:16:56.115610+0000","last_clean":"2024-09-11T08:16:56.115610+0000","last_became_active":"2024-09-11T08:07:32.044626+0000","last_became_peered":"2024-09-11T08:07:32.044626+0000","last_unstale":"2024-09-11T08:16:56.115610+0000","last_undegraded":"2024-09-11T08:16:56.115610+0000","last_fullsized":"2024-09-11T08:16:56.115610+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:16:56.115436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:16:56.115436+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:15.933818+0000","scrub_duration":0.0060897449999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":236,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.741097+0000","last_change":"2024-09-11T08:15:56.531009+0000","last_active":"2024-09-11T08:16:52.741097+0000","last_peered":"2024-09-11T08:16:52.741097+0000","last_clean":"2024-09-11T08:16:52.741097+0000","last_became_active":"2024-09-11T08:15:56.530680+0000","last_became_peered":"2024-09-11T08:15:56.530680+0000","last_unstale":"2024-09-11T08:16:52.741097+0000","last_undegraded":"2024-09-11T08:16:52.741097+0000","last_fullsized":"2024-09-11T08:16:52.741097+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:50.469752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:15:50.469752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:08.638035+0000","scrub_duration":0.0033416409999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":215,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.755446+0000","last_change":"2024-09-11T08:14:53.516662+0000","last_active":"2024-09-11T08:16:52.755446+0000","last_peered":"2024-09-11T08:16:52.755446+0000","last_clean":"2024-09-11T08:16:52.755446+0000","last_became_active":"2024-09-11T08:14:24.792595+0000","last_became_peered":"2024-09-11T08:14:24.792595+0000","last_unstale":"2024-09-11T08:16:52.755446+0000","last_undegraded":"2024-09-11T08:16:52.755446+0000","last_fullsized":"2024-09-11T08:16:52.755446+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:53.516583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:14:53.516583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034487419999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":223,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.740426+0000","last_change":"2024-09-11T08:16:51.618963+0000","last_active":"2024-09-11T08:16:52.740426+0000","last_peered":"2024-09-11T08:16:52.740426+0000","last_clean":"2024-09-11T08:16:52.740426+0000","last_became_active":"2024-09-11T08:07:30.043277+0000","last_became_peered":"2024-09-11T08:07:30.043277+0000","last_unstale":"2024-09-11T08:16:52.740426+0000","last_undegraded":"2024-09-11T08:16:52.740426+0000","last_fullsized":"2024-09-11T08:16:52.740426+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:51.618910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:16:51.618910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:59.666435+0000","scrub_duration":0.011147119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"139'7","reported_seq":214,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.745993+0000","last_change":"2024-09-11T08:15:02.988616+0000","last_active":"2024-09-11T08:16:52.745993+0000","last_peered":"2024-09-11T08:16:52.745993+0000","last_clean":"2024-09-11T08:16:52.745993+0000","last_became_active":"2024-09-11T08:07:32.048661+0000","last_became_peered":"2024-09-11T08:07:32.048661+0000","last_unstale":"2024-09-11T08:16:52.745993+0000","last_undegraded":"2024-09-11T08:16:52.745993+0000","last_fullsized":"2024-09-11T08:16:52.745993+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'7","last_scrub_stamp":"2024-09-11T08:15:02.988425+0000","last_deep_scrub":"139'7","last_deep_scrub_stamp":"2024-09-11T08:12:24.869318+0000","last_clean_scrub_stamp":"2024-09-11T08:15:02.988425+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033901550000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":36,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.738104+0000","last_change":"2024-09-11T08:16:41.727421+0000","last_active":"2024-09-11T08:16:52.738104+0000","last_peered":"2024-09-11T08:16:52.738104+0000","last_clean":"2024-09-11T08:16:52.738104+0000","last_became_active":"2024-09-11T08:15:56.526954+0000","last_became_peered":"2024-09-11T08:15:56.526954+0000","last_unstale":"2024-09-11T08:16:52.738104+0000","last_undegraded":"2024-09-11T08:16:52.738104+0000","last_fullsized":"2024-09-11T08:16:52.738104+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:41.727344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:16:41.727344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:50.309626+0000","scrub_duration":0.0011409409999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":217,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.739967+0000","last_change":"2024-09-11T08:15:55.371107+0000","last_active":"2024-09-11T08:16:52.739967+0000","last_peered":"2024-09-11T08:16:52.739967+0000","last_clean":"2024-09-11T08:16:52.739967+0000","last_became_active":"2024-09-11T08:07:36.077602+0000","last_became_peered":"2024-09-11T08:07:36.077602+0000","last_unstale":"2024-09-11T08:16:52.739967+0000","last_undegraded":"2024-09-11T08:16:52.739967+0000","last_fullsized":"2024-09-11T08:16:52.739967+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:55.371030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:04.782577+0000","last_clean_scrub_stamp":"2024-09-11T08:15:55.371030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:57.415606+0000","scrub_duration":0.0028204950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":198,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.746117+0000","last_change":"2024-09-11T08:13:11.061617+0000","last_active":"2024-09-11T08:16:52.746117+0000","last_peered":"2024-09-11T08:16:52.746117+0000","last_clean":"2024-09-11T08:16:52.746117+0000","last_became_active":"2024-09-11T08:07:36.633338+0000","last_became_peered":"2024-09-11T08:07:36.633338+0000","last_unstale":"2024-09-11T08:16:52.746117+0000","last_undegraded":"2024-09-11T08:16:52.746117+0000","last_fullsized":"2024-09-11T08:16:52.746117+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:11.061558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:13:11.061558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016408200000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":231,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.755701+0000","last_change":"2024-09-11T08:15:56.517613+0000","last_active":"2024-09-11T08:16:52.755701+0000","last_peered":"2024-09-11T08:16:52.755701+0000","last_clean":"2024-09-11T08:16:52.755701+0000","last_became_active":"2024-09-11T08:15:56.516941+0000","last_became_peered":"2024-09-11T08:15:56.516941+0000","last_unstale":"2024-09-11T08:16:52.755701+0000","last_undegraded":"2024-09-11T08:16:52.755701+0000","last_fullsized":"2024-09-11T08:16:52.755701+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:13.315410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:14:13.315410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030491400000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"107'22","reported_seq":253,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.754614+0000","last_change":"2024-09-11T08:16:04.159388+0000","last_active":"2024-09-11T08:16:52.754614+0000","last_peered":"2024-09-11T08:16:52.754614+0000","last_clean":"2024-09-11T08:16:52.754614+0000","last_became_active":"2024-09-11T08:07:32.046363+0000","last_became_peered":"2024-09-11T08:07:32.046363+0000","last_unstale":"2024-09-11T08:16:52.754614+0000","last_undegraded":"2024-09-11T08:16:52.754614+0000","last_fullsized":"2024-09-11T08:16:52.754614+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'22","last_scrub_stamp":"2024-09-11T08:16:04.159211+0000","last_deep_scrub":"107'22","last_deep_scrub_stamp":"2024-09-11T08:16:04.159211+0000","last_clean_scrub_stamp":"2024-09-11T08:16:04.159211+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:30.394200+0000","scrub_duration":0.0069586229999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":220,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.742959+0000","last_change":"2024-09-11T08:15:46.597272+0000","last_active":"2024-09-11T08:16:52.742959+0000","last_peered":"2024-09-11T08:16:52.742959+0000","last_clean":"2024-09-11T08:16:52.742959+0000","last_became_active":"2024-09-11T08:07:34.055410+0000","last_became_peered":"2024-09-11T08:07:34.055410+0000","last_unstale":"2024-09-11T08:16:52.742959+0000","last_undegraded":"2024-09-11T08:16:52.742959+0000","last_fullsized":"2024-09-11T08:16:52.742959+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:46.597198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:10:31.418873+0000","last_clean_scrub_stamp":"2024-09-11T08:15:46.597198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:00.695980+0000","scrub_duration":0.0032858620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":201,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.743075+0000","last_change":"2024-09-11T08:16:39.677479+0000","last_active":"2024-09-11T08:16:52.743075+0000","last_peered":"2024-09-11T08:16:52.743075+0000","last_clean":"2024-09-11T08:16:52.743075+0000","last_became_active":"2024-09-11T08:07:36.633258+0000","last_became_peered":"2024-09-11T08:07:36.633258+0000","last_unstale":"2024-09-11T08:16:52.743075+0000","last_undegraded":"2024-09-11T08:16:52.743075+0000","last_fullsized":"2024-09-11T08:16:52.743075+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:39.677399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:16:39.677399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:56.767305+0000","scrub_duration":0.0031390630000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":214,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.746856+0000","last_change":"2024-09-11T08:15:48.081690+0000","last_active":"2024-09-11T08:16:52.746856+0000","last_peered":"2024-09-11T08:16:52.746856+0000","last_clean":"2024-09-11T08:16:52.746856+0000","last_became_active":"2024-09-11T08:07:30.046284+0000","last_became_peered":"2024-09-11T08:07:30.046284+0000","last_unstale":"2024-09-11T08:16:52.746856+0000","last_undegraded":"2024-09-11T08:16:52.746856+0000","last_fullsized":"2024-09-11T08:16:52.746856+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:48.081591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:48.081591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:03.932824+0000","scrub_duration":0.0059702619999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":257,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.741682+0000","last_change":"2024-09-11T08:15:53.518868+0000","last_active":"2024-09-11T08:16:52.741682+0000","last_peered":"2024-09-11T08:16:52.741682+0000","last_clean":"2024-09-11T08:16:52.741682+0000","last_became_active":"2024-09-11T08:09:07.878432+0000","last_became_peered":"2024-09-11T08:09:07.878432+0000","last_unstale":"2024-09-11T08:16:52.741682+0000","last_undegraded":"2024-09-11T08:16:52.741682+0000","last_fullsized":"2024-09-11T08:16:52.741682+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-11T08:15:53.518765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:15:53.518765+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:19.946047+0000","scrub_duration":0.0047796499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":210,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.746033+0000","last_change":"2024-09-11T08:16:38.974464+0000","last_active":"2024-09-11T08:16:52.746033+0000","last_peered":"2024-09-11T08:16:52.746033+0000","last_clean":"2024-09-11T08:16:52.746033+0000","last_became_active":"2024-09-11T08:07:34.056528+0000","last_became_peered":"2024-09-11T08:07:34.056528+0000","last_unstale":"2024-09-11T08:16:52.746033+0000","last_undegraded":"2024-09-11T08:16:52.746033+0000","last_fullsized":"2024-09-11T08:16:52.746033+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:38.974372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:10:40.110490+0000","last_clean_scrub_stamp":"2024-09-11T08:16:38.974372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:43.756800+0000","scrub_duration":0.014148811000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":163,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T08:17:01.693825+0000","last_change":"2024-09-11T08:17:01.693734+0000","last_active":"2024-09-11T08:17:01.693825+0000","last_peered":"2024-09-11T08:17:01.693825+0000","last_clean":"2024-09-11T08:17:01.693825+0000","last_became_active":"2024-09-11T08:14:24.791610+0000","last_became_peered":"2024-09-11T08:14:24.791610+0000","last_unstale":"2024-09-11T08:17:01.693825+0000","last_undegraded":"2024-09-11T08:17:01.693825+0000","last_fullsized":"2024-09-11T08:17:01.693825+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-11T08:17:01.693642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:17:01.693642+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:05.451908+0000","scrub_duration":0.0041332410000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":224,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T08:16:55.567063+0000","last_change":"2024-09-11T08:16:55.566928+0000","last_active":"2024-09-11T08:16:55.567063+0000","last_peered":"2024-09-11T08:16:55.567063+0000","last_clean":"2024-09-11T08:16:55.567063+0000","last_became_active":"2024-09-11T08:07:30.045932+0000","last_became_peered":"2024-09-11T08:07:30.045932+0000","last_unstale":"2024-09-11T08:16:55.567063+0000","last_undegraded":"2024-09-11T08:16:55.567063+0000","last_fullsized":"2024-09-11T08:16:55.567063+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:55.566790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:16:55.566790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:19.971541+0000","scrub_duration":0.004175102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":300,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T08:16:56.735547+0000","last_change":"2024-09-11T08:16:56.735441+0000","last_active":"2024-09-11T08:16:56.735547+0000","last_peered":"2024-09-11T08:16:56.735547+0000","last_clean":"2024-09-11T08:16:56.735547+0000","last_became_active":"2024-09-11T08:15:56.515817+0000","last_became_peered":"2024-09-11T08:15:56.515817+0000","last_unstale":"2024-09-11T08:16:56.735547+0000","last_undegraded":"2024-09-11T08:16:56.735547+0000","last_fullsized":"2024-09-11T08:16:56.735547+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-11T08:16:56.735310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:16:56.735310+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:22.326800+0000","scrub_duration":0.052749864,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"209'11","reported_seq":390,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T08:16:53.762658+0000","last_change":"2024-09-11T08:16:44.493380+0000","last_active":"2024-09-11T08:16:53.762658+0000","last_peered":"2024-09-11T08:16:53.762658+0000","last_clean":"2024-09-11T08:16:53.762658+0000","last_became_active":"2024-09-11T08:15:56.525498+0000","last_became_peered":"2024-09-11T08:15:56.525498+0000","last_unstale":"2024-09-11T08:16:53.762658+0000","last_undegraded":"2024-09-11T08:16:53.762658+0000","last_fullsized":"2024-09-11T08:16:53.762658+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"209'11","last_scrub_stamp":"2024-09-11T08:16:44.493296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:16:44.493296+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:06.576751+0000","scrub_duration":0.002213058,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"224'13500","reported_seq":13713,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T08:16:58.270485+0000","last_change":"2024-09-11T08:16:50.189580+0000","last_active":"2024-09-11T08:16:58.270485+0000","last_peered":"2024-09-11T08:16:58.270485+0000","last_clean":"2024-09-11T08:16:58.270485+0000","last_became_active":"2024-09-11T08:13:54.883685+0000","last_became_peered":"2024-09-11T08:13:54.883685+0000","last_unstale":"2024-09-11T08:16:58.270485+0000","last_undegraded":"2024-09-11T08:16:58.270485+0000","last_fullsized":"2024-09-11T08:16:58.270485+0000","mapping_epoch":159,"log_start":"203'8400","ondisk_log_start":"203'8400","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'12950","last_scrub_stamp":"2024-09-11T08:16:50.189134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:16:50.189134+0000","objects_scrubbed":3240,"log_size":5100,"log_dups_size":0,"ondisk_log_size":5100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:52.322069+0000","scrub_duration":1.6148215850000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173539328,"num_objects":2648,"num_object_clones":0,"num_object_copies":7944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13500,"num_write_kb":516736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":247,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.743887+0000","last_change":"2024-09-11T08:15:42.534547+0000","last_active":"2024-09-11T08:16:52.743887+0000","last_peered":"2024-09-11T08:16:52.743887+0000","last_clean":"2024-09-11T08:16:52.743887+0000","last_became_active":"2024-09-11T08:07:32.054317+0000","last_became_peered":"2024-09-11T08:07:32.054317+0000","last_unstale":"2024-09-11T08:16:52.743887+0000","last_undegraded":"2024-09-11T08:16:52.743887+0000","last_fullsized":"2024-09-11T08:16:52.743887+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-11T08:15:42.534460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:15:42.534460+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0078017169999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":234,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.741695+0000","last_change":"2024-09-11T08:16:42.624557+0000","last_active":"2024-09-11T08:16:52.741695+0000","last_peered":"2024-09-11T08:16:52.741695+0000","last_clean":"2024-09-11T08:16:52.741695+0000","last_became_active":"2024-09-11T08:15:56.521943+0000","last_became_peered":"2024-09-11T08:15:56.521943+0000","last_unstale":"2024-09-11T08:16:52.741695+0000","last_undegraded":"2024-09-11T08:16:52.741695+0000","last_fullsized":"2024-09-11T08:16:52.741695+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:42.624425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:16:42.624425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:05.345719+0000","scrub_duration":0.011367408000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":332,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T08:16:53.764514+0000","last_change":"2024-09-11T08:16:44.005131+0000","last_active":"2024-09-11T08:16:53.764514+0000","last_peered":"2024-09-11T08:16:53.764514+0000","last_clean":"2024-09-11T08:16:53.764514+0000","last_became_active":"2024-09-11T08:07:34.052981+0000","last_became_peered":"2024-09-11T08:07:34.052981+0000","last_unstale":"2024-09-11T08:16:53.764514+0000","last_undegraded":"2024-09-11T08:16:53.764514+0000","last_fullsized":"2024-09-11T08:16:53.764514+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:16:44.005077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:16:44.005077+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:04.080032+0000","scrub_duration":0.019154572000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":217,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.740312+0000","last_change":"2024-09-11T08:15:45.411776+0000","last_active":"2024-09-11T08:16:52.740312+0000","last_peered":"2024-09-11T08:16:52.740312+0000","last_clean":"2024-09-11T08:16:52.740312+0000","last_became_active":"2024-09-11T08:07:36.073969+0000","last_became_peered":"2024-09-11T08:07:36.073969+0000","last_unstale":"2024-09-11T08:16:52.740312+0000","last_undegraded":"2024-09-11T08:16:52.740312+0000","last_fullsized":"2024-09-11T08:16:52.740312+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:45.411593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:06.616332+0000","last_clean_scrub_stamp":"2024-09-11T08:15:45.411593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:06.019721+0000","scrub_duration":0.0031499940000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"224'13437","reported_seq":24379,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T08:16:58.322482+0000","last_change":"2024-09-11T08:16:35.218904+0000","last_active":"2024-09-11T08:16:58.322482+0000","last_peered":"2024-09-11T08:16:58.322482+0000","last_clean":"2024-09-11T08:16:58.322482+0000","last_became_active":"2024-09-11T08:16:35.162488+0000","last_became_peered":"2024-09-11T08:16:35.162488+0000","last_unstale":"2024-09-11T08:16:58.322482+0000","last_undegraded":"2024-09-11T08:16:58.322482+0000","last_fullsized":"2024-09-11T08:16:58.322482+0000","mapping_epoch":218,"log_start":"198'7532","ondisk_log_start":"198'7532","created":159,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:13:53.862306+0000","objects_scrubbed":0,"log_size":5905,"log_dups_size":0,"ondisk_log_size":5905,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167575552,"num_objects":2557,"num_object_clones":0,"num_object_copies":7671,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13437,"num_write_kb":511808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5422,"num_bytes_recovered":355336192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":297,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.741494+0000","last_change":"2024-09-11T08:15:56.542573+0000","last_active":"2024-09-11T08:16:52.741494+0000","last_peered":"2024-09-11T08:16:52.741494+0000","last_clean":"2024-09-11T08:16:52.741494+0000","last_became_active":"2024-09-11T08:15:56.517680+0000","last_became_peered":"2024-09-11T08:15:56.517680+0000","last_unstale":"2024-09-11T08:16:52.741494+0000","last_undegraded":"2024-09-11T08:16:52.741494+0000","last_fullsized":"2024-09-11T08:16:52.741494+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-11T08:13:58.979187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:13:58.979187+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001885091,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":234,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.745714+0000","last_change":"2024-09-11T08:16:43.614365+0000","last_active":"2024-09-11T08:16:52.745714+0000","last_peered":"2024-09-11T08:16:52.745714+0000","last_clean":"2024-09-11T08:16:52.745714+0000","last_became_active":"2024-09-11T08:15:56.527438+0000","last_became_peered":"2024-09-11T08:15:56.527438+0000","last_unstale":"2024-09-11T08:16:52.745714+0000","last_undegraded":"2024-09-11T08:16:52.745714+0000","last_fullsized":"2024-09-11T08:16:52.745714+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:43.614153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:16:43.614153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:05.791592+0000","scrub_duration":0.0022087069999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":227,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.745760+0000","last_change":"2024-09-11T08:15:56.522907+0000","last_active":"2024-09-11T08:16:52.745760+0000","last_peered":"2024-09-11T08:16:52.745760+0000","last_clean":"2024-09-11T08:16:52.745760+0000","last_became_active":"2024-09-11T08:15:56.522606+0000","last_became_peered":"2024-09-11T08:15:56.522606+0000","last_unstale":"2024-09-11T08:16:52.745760+0000","last_undegraded":"2024-09-11T08:16:52.745760+0000","last_fullsized":"2024-09-11T08:16:52.745760+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:57.840457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:13:57.840457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014582009999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":198,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.746763+0000","last_change":"2024-09-11T08:14:04.937966+0000","last_active":"2024-09-11T08:16:52.746763+0000","last_peered":"2024-09-11T08:16:52.746763+0000","last_clean":"2024-09-11T08:16:52.746763+0000","last_became_active":"2024-09-11T08:07:36.069225+0000","last_became_peered":"2024-09-11T08:07:36.069225+0000","last_unstale":"2024-09-11T08:16:52.746763+0000","last_undegraded":"2024-09-11T08:16:52.746763+0000","last_fullsized":"2024-09-11T08:16:52.746763+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:04.937884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:04.937884+0000","last_clean_scrub_stamp":"2024-09-11T08:14:04.937884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018777710000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"224'13469","reported_seq":25786,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T08:16:57.869826+0000","last_change":"2024-09-11T08:16:25.103862+0000","last_active":"2024-09-11T08:16:57.869826+0000","last_peered":"2024-09-11T08:16:57.869826+0000","last_clean":"2024-09-11T08:16:57.869826+0000","last_became_active":"2024-09-11T08:16:25.088438+0000","last_became_peered":"2024-09-11T08:16:25.088438+0000","last_unstale":"2024-09-11T08:16:57.869826+0000","last_undegraded":"2024-09-11T08:16:57.869826+0000","last_fullsized":"2024-09-11T08:16:57.869826+0000","mapping_epoch":215,"log_start":"202'8200","ondisk_log_start":"202'8200","created":159,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'8832","last_scrub_stamp":"2024-09-11T08:15:51.203402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:15:51.203402+0000","objects_scrubbed":7454,"log_size":5269,"log_dups_size":0,"ondisk_log_size":5269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:06.670584+0000","scrub_duration":2.7862675239999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176160792,"num_objects":2689,"num_object_clones":0,"num_object_copies":8067,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2689,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13469,"num_write_kb":516993,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5955,"num_bytes_recovered":390201368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":223,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.746314+0000","last_change":"2024-09-11T08:14:59.014008+0000","last_active":"2024-09-11T08:16:52.746314+0000","last_peered":"2024-09-11T08:16:52.746314+0000","last_clean":"2024-09-11T08:16:52.746314+0000","last_became_active":"2024-09-11T08:07:32.040069+0000","last_became_peered":"2024-09-11T08:07:32.040069+0000","last_unstale":"2024-09-11T08:16:52.746314+0000","last_undegraded":"2024-09-11T08:16:52.746314+0000","last_fullsized":"2024-09-11T08:16:52.746314+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-11T08:14:59.013949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:14:59.013949+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001898156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":214,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.817231+0000","last_change":"2024-09-11T08:15:54.794370+0000","last_active":"2024-09-11T08:16:52.817231+0000","last_peered":"2024-09-11T08:16:52.817231+0000","last_clean":"2024-09-11T08:16:52.817231+0000","last_became_active":"2024-09-11T08:07:30.050484+0000","last_became_peered":"2024-09-11T08:07:30.050484+0000","last_unstale":"2024-09-11T08:16:52.817231+0000","last_undegraded":"2024-09-11T08:16:52.817231+0000","last_fullsized":"2024-09-11T08:16:52.817231+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:54.793950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:54.793950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:06.106542+0000","scrub_duration":0.0059264870000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"82'9","reported_seq":335,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:53.663639+0000","last_change":"2024-09-11T08:15:51.456046+0000","last_active":"2024-09-11T08:16:53.663639+0000","last_peered":"2024-09-11T08:16:53.663639+0000","last_clean":"2024-09-11T08:16:53.663639+0000","last_became_active":"2024-09-11T08:07:34.056143+0000","last_became_peered":"2024-09-11T08:07:34.056143+0000","last_unstale":"2024-09-11T08:16:53.663639+0000","last_undegraded":"2024-09-11T08:16:53.663639+0000","last_fullsized":"2024-09-11T08:16:53.663639+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-11T08:15:51.455966+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-09-11T08:13:10.230703+0000","last_clean_scrub_stamp":"2024-09-11T08:15:51.455966+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:01.331170+0000","scrub_duration":0.00915995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":188,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.738519+0000","last_change":"2024-09-11T08:15:33.507691+0000","last_active":"2024-09-11T08:16:52.738519+0000","last_peered":"2024-09-11T08:16:52.738519+0000","last_clean":"2024-09-11T08:16:52.738519+0000","last_became_active":"2024-09-11T08:14:24.788968+0000","last_became_peered":"2024-09-11T08:14:24.788968+0000","last_unstale":"2024-09-11T08:16:52.738519+0000","last_undegraded":"2024-09-11T08:16:52.738519+0000","last_fullsized":"2024-09-11T08:16:52.738519+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:33.507613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:15:33.507613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:55.580423+0000","scrub_duration":0.0032985240000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.e","version":"224'14198","reported_seq":14306,"reported_epoch":224,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-11T08:16:58.345052+0000","last_change":"2024-09-11T08:16:51.642928+0000","last_active":"2024-09-11T08:16:58.345052+0000","last_peered":"2024-09-11T08:16:58.345052+0000","last_clean":"2024-09-11T08:16:58.345052+0000","last_became_active":"2024-09-11T08:13:55.705670+0000","last_became_peered":"2024-09-11T08:13:55.705670+0000","last_unstale":"2024-09-11T08:16:58.345052+0000","last_undegraded":"2024-09-11T08:16:58.345052+0000","last_fullsized":"2024-09-11T08:16:58.345052+0000","mapping_epoch":159,"log_start":"203'8300","ondisk_log_start":"203'8300","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:13:53.862306+0000","objects_scrubbed":836,"log_size":5898,"log_dups_size":0,"ondisk_log_size":5898,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 6s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169345024,"num_objects":2584,"num_object_clones":0,"num_object_copies":7752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13362,"num_write_kb":510272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":230,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.743828+0000","last_change":"2024-09-11T08:15:36.365980+0000","last_active":"2024-09-11T08:16:52.743828+0000","last_peered":"2024-09-11T08:16:52.743828+0000","last_clean":"2024-09-11T08:16:52.743828+0000","last_became_active":"2024-09-11T08:07:32.054294+0000","last_became_peered":"2024-09-11T08:07:32.054294+0000","last_unstale":"2024-09-11T08:16:52.743828+0000","last_undegraded":"2024-09-11T08:16:52.743828+0000","last_fullsized":"2024-09-11T08:16:52.743828+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-11T08:15:36.365891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:15:36.365891+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:05.688088+0000","scrub_duration":0.003716071,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":213,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.740592+0000","last_change":"2024-09-11T08:15:46.459225+0000","last_active":"2024-09-11T08:16:52.740592+0000","last_peered":"2024-09-11T08:16:52.740592+0000","last_clean":"2024-09-11T08:16:52.740592+0000","last_became_active":"2024-09-11T08:07:30.047497+0000","last_became_peered":"2024-09-11T08:07:30.047497+0000","last_unstale":"2024-09-11T08:16:52.740592+0000","last_undegraded":"2024-09-11T08:16:52.740592+0000","last_fullsized":"2024-09-11T08:16:52.740592+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:46.459143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:46.459143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:12.469916+0000","scrub_duration":0.0057785019999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"190'9","reported_seq":70,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:53.663418+0000","last_change":"2024-09-11T08:15:08.769903+0000","last_active":"2024-09-11T08:16:53.663418+0000","last_peered":"2024-09-11T08:16:53.663418+0000","last_clean":"2024-09-11T08:16:53.663418+0000","last_became_active":"2024-09-11T08:15:08.764876+0000","last_became_peered":"2024-09-11T08:15:08.764876+0000","last_unstale":"2024-09-11T08:16:53.663418+0000","last_undegraded":"2024-09-11T08:16:53.663418+0000","last_fullsized":"2024-09-11T08:16:53.663418+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:14:11.320174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:14:11.320174+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017353449999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"66'1","reported_seq":199,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.755605+0000","last_change":"2024-09-11T08:15:06.445409+0000","last_active":"2024-09-11T08:16:52.755605+0000","last_peered":"2024-09-11T08:16:52.755605+0000","last_clean":"2024-09-11T08:16:52.755605+0000","last_became_active":"2024-09-11T08:07:36.072593+0000","last_became_peered":"2024-09-11T08:07:36.072593+0000","last_unstale":"2024-09-11T08:16:52.755605+0000","last_undegraded":"2024-09-11T08:16:52.755605+0000","last_fullsized":"2024-09-11T08:16:52.755605+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-11T08:15:06.445301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:15:06.445301+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013971574,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.f","version":"224'13481","reported_seq":13554,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T08:16:58.086398+0000","last_change":"2024-09-11T08:13:55.706681+0000","last_active":"2024-09-11T08:16:58.086398+0000","last_peered":"2024-09-11T08:16:58.086398+0000","last_clean":"2024-09-11T08:16:58.086398+0000","last_became_active":"2024-09-11T08:13:55.706134+0000","last_became_peered":"2024-09-11T08:13:55.706134+0000","last_unstale":"2024-09-11T08:16:58.086398+0000","last_undegraded":"2024-09-11T08:16:58.086398+0000","last_fullsized":"2024-09-11T08:16:58.086398+0000","mapping_epoch":159,"log_start":"203'8709","ondisk_log_start":"203'8709","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:13:53.862306+0000","objects_scrubbed":0,"log_size":4772,"log_dups_size":0,"ondisk_log_size":4772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170328064,"num_objects":2599,"num_object_clones":0,"num_object_copies":7797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13481,"num_write_kb":514560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":301,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.755468+0000","last_change":"2024-09-11T08:15:56.540181+0000","last_active":"2024-09-11T08:16:52.755468+0000","last_peered":"2024-09-11T08:16:52.755468+0000","last_clean":"2024-09-11T08:16:52.755468+0000","last_became_active":"2024-09-11T08:15:56.518384+0000","last_became_peered":"2024-09-11T08:15:56.518384+0000","last_unstale":"2024-09-11T08:16:52.755468+0000","last_undegraded":"2024-09-11T08:16:52.755468+0000","last_fullsized":"2024-09-11T08:16:52.755468+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-11T08:13:57.387660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:13:57.387660+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025295550000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":224,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.742326+0000","last_change":"2024-09-11T08:15:54.517507+0000","last_active":"2024-09-11T08:16:52.742326+0000","last_peered":"2024-09-11T08:16:52.742326+0000","last_clean":"2024-09-11T08:16:52.742326+0000","last_became_active":"2024-09-11T08:07:30.045380+0000","last_became_peered":"2024-09-11T08:07:30.045380+0000","last_unstale":"2024-09-11T08:16:52.742326+0000","last_undegraded":"2024-09-11T08:16:52.742326+0000","last_fullsized":"2024-09-11T08:16:52.742326+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:54.517429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:54.517429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:05.165550+0000","scrub_duration":0.0055168639999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":229,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T08:17:00.522586+0000","last_change":"2024-09-11T08:17:00.522545+0000","last_active":"2024-09-11T08:17:00.522586+0000","last_peered":"2024-09-11T08:17:00.522586+0000","last_clean":"2024-09-11T08:17:00.522586+0000","last_became_active":"2024-09-11T08:07:34.051004+0000","last_became_peered":"2024-09-11T08:07:34.051004+0000","last_unstale":"2024-09-11T08:17:00.522586+0000","last_undegraded":"2024-09-11T08:17:00.522586+0000","last_fullsized":"2024-09-11T08:17:00.522586+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:00.522467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:17:00.522467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:11.280048+0000","scrub_duration":0.0017451980000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":208,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.742257+0000","last_change":"2024-09-11T08:14:09.068749+0000","last_active":"2024-09-11T08:16:52.742257+0000","last_peered":"2024-09-11T08:16:52.742257+0000","last_clean":"2024-09-11T08:16:52.742257+0000","last_became_active":"2024-09-11T08:07:36.631668+0000","last_became_peered":"2024-09-11T08:07:36.631668+0000","last_unstale":"2024-09-11T08:16:52.742257+0000","last_undegraded":"2024-09-11T08:16:52.742257+0000","last_fullsized":"2024-09-11T08:16:52.742257+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:09.068694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:14:09.068694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033190699999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"224'13710","reported_seq":18353,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T08:16:57.920118+0000","last_change":"2024-09-11T08:15:56.003569+0000","last_active":"2024-09-11T08:16:57.920118+0000","last_peered":"2024-09-11T08:16:57.920118+0000","last_clean":"2024-09-11T08:16:57.920118+0000","last_became_active":"2024-09-11T08:14:50.875690+0000","last_became_peered":"2024-09-11T08:14:50.875690+0000","last_unstale":"2024-09-11T08:16:57.920118+0000","last_undegraded":"2024-09-11T08:16:57.920118+0000","last_fullsized":"2024-09-11T08:16:57.920118+0000","mapping_epoch":179,"log_start":"202'8400","ondisk_log_start":"202'8400","created":159,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'9330","last_scrub_stamp":"2024-09-11T08:15:56.003501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:15:56.003501+0000","objects_scrubbed":7010,"log_size":5310,"log_dups_size":0,"ondisk_log_size":5310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:14.010825+0000","scrub_duration":1.6004849619999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170393600,"num_objects":2600,"num_object_clones":0,"num_object_copies":7800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13710,"num_write_kb":521920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2130,"num_bytes_recovered":139591680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"190'39","reported_seq":98,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:53.663761+0000","last_change":"2024-09-11T08:15:52.848067+0000","last_active":"2024-09-11T08:16:53.663761+0000","last_peered":"2024-09-11T08:16:53.663761+0000","last_clean":"2024-09-11T08:16:53.663761+0000","last_became_active":"2024-09-11T08:15:08.775181+0000","last_became_peered":"2024-09-11T08:15:08.775181+0000","last_unstale":"2024-09-11T08:16:53.663761+0000","last_undegraded":"2024-09-11T08:16:53.663761+0000","last_fullsized":"2024-09-11T08:16:53.663761+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'39","last_scrub_stamp":"2024-09-11T08:15:52.847968+0000","last_deep_scrub":"190'39","last_deep_scrub_stamp":"2024-09-11T08:15:52.847968+0000","last_clean_scrub_stamp":"2024-09-11T08:15:52.847968+0000","objects_scrubbed":10,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:11.824582+0000","scrub_duration":0.012280474,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"59'1","reported_seq":223,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.746580+0000","last_change":"2024-09-11T08:15:44.992430+0000","last_active":"2024-09-11T08:16:52.746580+0000","last_peered":"2024-09-11T08:16:52.746580+0000","last_clean":"2024-09-11T08:16:52.746580+0000","last_became_active":"2024-09-11T08:07:30.047513+0000","last_became_peered":"2024-09-11T08:07:30.047513+0000","last_unstale":"2024-09-11T08:16:52.746580+0000","last_undegraded":"2024-09-11T08:16:52.746580+0000","last_fullsized":"2024-09-11T08:16:52.746580+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-11T08:15:44.992356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:44.992356+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:00.977258+0000","scrub_duration":0.0049464690000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":230,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T08:16:56.558626+0000","last_change":"2024-09-11T08:16:56.558518+0000","last_active":"2024-09-11T08:16:56.558626+0000","last_peered":"2024-09-11T08:16:56.558626+0000","last_clean":"2024-09-11T08:16:56.558626+0000","last_became_active":"2024-09-11T08:07:30.043130+0000","last_became_peered":"2024-09-11T08:07:30.043130+0000","last_unstale":"2024-09-11T08:16:56.558626+0000","last_undegraded":"2024-09-11T08:16:56.558626+0000","last_fullsized":"2024-09-11T08:16:56.558626+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-11T08:16:56.558301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:16:56.558301+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:23.711468+0000","scrub_duration":0.0048858299999999999,"objects_trimmed":0,"snaptrim_duration":0.0026120850000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":210,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.754810+0000","last_change":"2024-09-11T08:16:42.666121+0000","last_active":"2024-09-11T08:16:52.754810+0000","last_peered":"2024-09-11T08:16:52.754810+0000","last_clean":"2024-09-11T08:16:52.754810+0000","last_became_active":"2024-09-11T08:07:34.057154+0000","last_became_peered":"2024-09-11T08:07:34.057154+0000","last_unstale":"2024-09-11T08:16:52.754810+0000","last_undegraded":"2024-09-11T08:16:52.754810+0000","last_fullsized":"2024-09-11T08:16:52.754810+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:42.666035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:10:37.957096+0000","last_clean_scrub_stamp":"2024-09-11T08:16:42.666035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:11.328001+0000","scrub_duration":0.0077723870000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":225,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.743671+0000","last_change":"2024-09-11T08:15:56.527473+0000","last_active":"2024-09-11T08:16:52.743671+0000","last_peered":"2024-09-11T08:16:52.743671+0000","last_clean":"2024-09-11T08:16:52.743671+0000","last_became_active":"2024-09-11T08:15:56.525099+0000","last_became_peered":"2024-09-11T08:15:56.525099+0000","last_unstale":"2024-09-11T08:16:52.743671+0000","last_undegraded":"2024-09-11T08:16:52.743671+0000","last_fullsized":"2024-09-11T08:16:52.743671+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:05.076884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:14:05.076884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001986065,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"224'13232","reported_seq":13566,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T08:16:58.090051+0000","last_change":"2024-09-11T08:16:05.402156+0000","last_active":"2024-09-11T08:16:58.090051+0000","last_peered":"2024-09-11T08:16:58.090051+0000","last_clean":"2024-09-11T08:16:58.090051+0000","last_became_active":"2024-09-11T08:13:54.882290+0000","last_became_peered":"2024-09-11T08:13:54.882290+0000","last_unstale":"2024-09-11T08:16:58.090051+0000","last_undegraded":"2024-09-11T08:16:58.090051+0000","last_fullsized":"2024-09-11T08:16:58.090051+0000","mapping_epoch":159,"log_start":"203'8405","ondisk_log_start":"203'8405","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'9635","last_scrub_stamp":"2024-09-11T08:16:05.401974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:16:05.401974+0000","objects_scrubbed":6255,"log_size":4827,"log_dups_size":0,"ondisk_log_size":4827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:26.177859+0000","scrub_duration":2.9696664940000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166330368,"num_objects":2538,"num_object_clones":0,"num_object_copies":7614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13232,"num_write_kb":504640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":252,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.743960+0000","last_change":"2024-09-11T08:16:41.514946+0000","last_active":"2024-09-11T08:16:52.743960+0000","last_peered":"2024-09-11T08:16:52.743960+0000","last_clean":"2024-09-11T08:16:52.743960+0000","last_became_active":"2024-09-11T08:07:32.050673+0000","last_became_peered":"2024-09-11T08:07:32.050673+0000","last_unstale":"2024-09-11T08:16:52.743960+0000","last_undegraded":"2024-09-11T08:16:52.743960+0000","last_fullsized":"2024-09-11T08:16:52.743960+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'13","last_scrub_stamp":"2024-09-11T08:16:41.514853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:16:41.514853+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:50.626022+0000","scrub_duration":0.0093017910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":217,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.742369+0000","last_change":"2024-09-11T08:15:31.273498+0000","last_active":"2024-09-11T08:16:52.742369+0000","last_peered":"2024-09-11T08:16:52.742369+0000","last_clean":"2024-09-11T08:16:52.742369+0000","last_became_active":"2024-09-11T08:07:30.047014+0000","last_became_peered":"2024-09-11T08:07:30.047014+0000","last_unstale":"2024-09-11T08:16:52.742369+0000","last_undegraded":"2024-09-11T08:16:52.742369+0000","last_fullsized":"2024-09-11T08:16:52.742369+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:31.273415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:31.273415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:59.181535+0000","scrub_duration":0.0057528060000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"127'2","reported_seq":243,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.746242+0000","last_change":"2024-09-11T08:16:40.021161+0000","last_active":"2024-09-11T08:16:52.746242+0000","last_peered":"2024-09-11T08:16:52.746242+0000","last_clean":"2024-09-11T08:16:52.746242+0000","last_became_active":"2024-09-11T08:07:30.045009+0000","last_became_peered":"2024-09-11T08:07:30.045009+0000","last_unstale":"2024-09-11T08:16:52.746242+0000","last_undegraded":"2024-09-11T08:16:52.746242+0000","last_fullsized":"2024-09-11T08:16:52.746242+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'2","last_scrub_stamp":"2024-09-11T08:16:40.021082+0000","last_deep_scrub":"127'2","last_deep_scrub_stamp":"2024-09-11T08:15:01.938497+0000","last_clean_scrub_stamp":"2024-09-11T08:16:40.021082+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:50.557847+0000","scrub_duration":0.002170426,"objects_trimmed":0,"snaptrim_duration":0.001859533,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":239,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.755840+0000","last_change":"2024-09-11T08:16:50.993227+0000","last_active":"2024-09-11T08:16:52.755840+0000","last_peered":"2024-09-11T08:16:52.755840+0000","last_clean":"2024-09-11T08:16:52.755840+0000","last_became_active":"2024-09-11T08:14:24.792279+0000","last_became_peered":"2024-09-11T08:14:24.792279+0000","last_unstale":"2024-09-11T08:16:52.755840+0000","last_undegraded":"2024-09-11T08:16:52.755840+0000","last_fullsized":"2024-09-11T08:16:52.755840+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:50.993154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:58.889094+0000","last_clean_scrub_stamp":"2024-09-11T08:16:50.993154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:13.412919+0000","scrub_duration":0.0011069389999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":154,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T08:16:58.685551+0000","last_change":"2024-09-11T08:16:58.685525+0000","last_active":"2024-09-11T08:16:58.685551+0000","last_peered":"2024-09-11T08:16:58.685551+0000","last_clean":"2024-09-11T08:16:58.685551+0000","last_became_active":"2024-09-11T08:14:24.792847+0000","last_became_peered":"2024-09-11T08:14:24.792847+0000","last_unstale":"2024-09-11T08:16:58.685551+0000","last_undegraded":"2024-09-11T08:16:58.685551+0000","last_fullsized":"2024-09-11T08:16:58.685551+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:58.685473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:58.796217+0000","last_clean_scrub_stamp":"2024-09-11T08:16:58.685473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:13.302984+0000","scrub_duration":0.0016333490000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"224'13390","reported_seq":13612,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T08:16:58.157613+0000","last_change":"2024-09-11T08:16:46.178629+0000","last_active":"2024-09-11T08:16:58.157613+0000","last_peered":"2024-09-11T08:16:58.157613+0000","last_clean":"2024-09-11T08:16:58.157613+0000","last_became_active":"2024-09-11T08:13:55.707556+0000","last_became_peered":"2024-09-11T08:13:55.707556+0000","last_unstale":"2024-09-11T08:16:58.157613+0000","last_undegraded":"2024-09-11T08:16:58.157613+0000","last_fullsized":"2024-09-11T08:16:58.157613+0000","mapping_epoch":159,"log_start":"202'8300","ondisk_log_start":"202'8300","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"221'12598","last_scrub_stamp":"2024-09-11T08:16:46.177312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:16:46.177312+0000","objects_scrubbed":3466,"log_size":5090,"log_dups_size":0,"ondisk_log_size":5090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:15.364017+0000","scrub_duration":1.6312657339999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171704320,"num_objects":2620,"num_object_clones":0,"num_object_copies":7860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13390,"num_write_kb":512320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":230,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.817152+0000","last_change":"2024-09-11T08:15:53.800579+0000","last_active":"2024-09-11T08:16:52.817152+0000","last_peered":"2024-09-11T08:16:52.817152+0000","last_clean":"2024-09-11T08:16:52.817152+0000","last_became_active":"2024-09-11T08:07:32.055407+0000","last_became_peered":"2024-09-11T08:07:32.055407+0000","last_unstale":"2024-09-11T08:16:52.817152+0000","last_undegraded":"2024-09-11T08:16:52.817152+0000","last_fullsized":"2024-09-11T08:16:52.817152+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-11T08:15:53.800494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:15:53.800494+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:54.617297+0000","scrub_duration":0.0070921359999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"59'1","reported_seq":221,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.756814+0000","last_change":"2024-09-11T08:15:51.807218+0000","last_active":"2024-09-11T08:16:52.756814+0000","last_peered":"2024-09-11T08:16:52.756814+0000","last_clean":"2024-09-11T08:16:52.756814+0000","last_became_active":"2024-09-11T08:07:30.050504+0000","last_became_peered":"2024-09-11T08:07:30.050504+0000","last_unstale":"2024-09-11T08:16:52.756814+0000","last_undegraded":"2024-09-11T08:16:52.756814+0000","last_fullsized":"2024-09-11T08:16:52.756814+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-11T08:15:51.806819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:51.806819+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:59.379579+0000","scrub_duration":0.012826908,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"224'6","reported_seq":574,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T08:17:01.131795+0000","last_change":"2024-09-11T08:16:55.778375+0000","last_active":"2024-09-11T08:17:01.131795+0000","last_peered":"2024-09-11T08:17:01.131795+0000","last_clean":"2024-09-11T08:17:01.131795+0000","last_became_active":"2024-09-11T08:14:24.793048+0000","last_became_peered":"2024-09-11T08:14:24.793048+0000","last_unstale":"2024-09-11T08:17:01.131795+0000","last_undegraded":"2024-09-11T08:17:01.131795+0000","last_fullsized":"2024-09-11T08:17:01.131795+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'6","last_scrub_stamp":"2024-09-11T08:16:55.778284+0000","last_deep_scrub":"224'6","last_deep_scrub_stamp":"2024-09-11T08:16:55.778284+0000","last_clean_scrub_stamp":"2024-09-11T08:16:55.778284+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:08.760919+0000","scrub_duration":0.018002499000000002,"objects_trimmed":0,"snaptrim_duration":0.000318821,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":164,"num_read_kb":159,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":219,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.740764+0000","last_change":"2024-09-11T08:16:37.787055+0000","last_active":"2024-09-11T08:16:52.740764+0000","last_peered":"2024-09-11T08:16:52.740764+0000","last_clean":"2024-09-11T08:16:52.740764+0000","last_became_active":"2024-09-11T08:07:34.058962+0000","last_became_peered":"2024-09-11T08:07:34.058962+0000","last_unstale":"2024-09-11T08:16:52.740764+0000","last_undegraded":"2024-09-11T08:16:52.740764+0000","last_fullsized":"2024-09-11T08:16:52.740764+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:37.786977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:16:37.786977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:48.840372+0000","scrub_duration":0.010499725999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":218,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.742094+0000","last_change":"2024-09-11T08:15:53.506655+0000","last_active":"2024-09-11T08:16:52.742094+0000","last_peered":"2024-09-11T08:16:52.742094+0000","last_clean":"2024-09-11T08:16:52.742094+0000","last_became_active":"2024-09-11T08:07:36.077597+0000","last_became_peered":"2024-09-11T08:07:36.077597+0000","last_unstale":"2024-09-11T08:16:52.742094+0000","last_undegraded":"2024-09-11T08:16:52.742094+0000","last_fullsized":"2024-09-11T08:16:52.742094+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:53.506581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:15:53.506581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:15.129440+0000","scrub_duration":0.0028717209999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"224'13730","reported_seq":14120,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T08:17:02.856125+0000","last_change":"2024-09-11T08:15:47.174817+0000","last_active":"2024-09-11T08:17:02.856125+0000","last_peered":"2024-09-11T08:17:02.856125+0000","last_clean":"2024-09-11T08:17:02.856125+0000","last_became_active":"2024-09-11T08:13:54.882405+0000","last_became_peered":"2024-09-11T08:13:54.882405+0000","last_unstale":"2024-09-11T08:17:02.856125+0000","last_undegraded":"2024-09-11T08:17:02.856125+0000","last_fullsized":"2024-09-11T08:17:02.856125+0000","mapping_epoch":159,"log_start":"199'7700","ondisk_log_start":"199'7700","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'8519","last_scrub_stamp":"2024-09-11T08:15:47.174735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:15:47.174735+0000","objects_scrubbed":7692,"log_size":6030,"log_dups_size":0,"ondisk_log_size":6030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:12.130102+0000","scrub_duration":2.9660035420000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154140672,"num_objects":2352,"num_object_clones":0,"num_object_copies":7056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13730,"num_write_kb":514624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"84'21","reported_seq":267,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.742477+0000","last_change":"2024-09-11T08:16:40.472710+0000","last_active":"2024-09-11T08:16:52.742477+0000","last_peered":"2024-09-11T08:16:52.742477+0000","last_clean":"2024-09-11T08:16:52.742477+0000","last_became_active":"2024-09-11T08:14:24.789154+0000","last_became_peered":"2024-09-11T08:14:24.789154+0000","last_unstale":"2024-09-11T08:16:52.742477+0000","last_undegraded":"2024-09-11T08:16:52.742477+0000","last_fullsized":"2024-09-11T08:16:52.742477+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'21","last_scrub_stamp":"2024-09-11T08:16:40.472618+0000","last_deep_scrub":"84'21","last_deep_scrub_stamp":"2024-09-11T08:08:56.705306+0000","last_clean_scrub_stamp":"2024-09-11T08:16:40.472618+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:00.262854+0000","scrub_duration":0.0099327200000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":204,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.740967+0000","last_change":"2024-09-11T08:15:06.236392+0000","last_active":"2024-09-11T08:16:52.740967+0000","last_peered":"2024-09-11T08:16:52.740967+0000","last_clean":"2024-09-11T08:16:52.740967+0000","last_became_active":"2024-09-11T08:07:30.047246+0000","last_became_peered":"2024-09-11T08:07:30.047246+0000","last_unstale":"2024-09-11T08:16:52.740967+0000","last_undegraded":"2024-09-11T08:16:52.740967+0000","last_fullsized":"2024-09-11T08:16:52.740967+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:06.236119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:06.236119+0000","last_clean_scrub_stamp":"2024-09-11T08:15:06.236119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029121939999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":42,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.737970+0000","last_change":"2024-09-11T08:16:45.611831+0000","last_active":"2024-09-11T08:16:52.737970+0000","last_peered":"2024-09-11T08:16:52.737970+0000","last_clean":"2024-09-11T08:16:52.737970+0000","last_became_active":"2024-09-11T08:15:56.523430+0000","last_became_peered":"2024-09-11T08:15:56.523430+0000","last_unstale":"2024-09-11T08:16:52.737970+0000","last_undegraded":"2024-09-11T08:16:52.737970+0000","last_fullsized":"2024-09-11T08:16:52.737970+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-11T08:16:45.611772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:05:21.231856+0000","last_clean_scrub_stamp":"2024-09-11T08:16:45.611772+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:06.875825+0000","scrub_duration":0.0021764710000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":26,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.738729+0000","last_change":"2024-09-11T08:15:56.529159+0000","last_active":"2024-09-11T08:16:52.738729+0000","last_peered":"2024-09-11T08:16:52.738729+0000","last_clean":"2024-09-11T08:16:52.738729+0000","last_became_active":"2024-09-11T08:15:56.528641+0000","last_became_peered":"2024-09-11T08:15:56.528641+0000","last_unstale":"2024-09-11T08:16:52.738729+0000","last_undegraded":"2024-09-11T08:16:52.738729+0000","last_fullsized":"2024-09-11T08:16:52.738729+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:55.140137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:51.289872+0000","last_clean_scrub_stamp":"2024-09-11T08:15:55.140137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:57.600999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":217,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.740678+0000","last_change":"2024-09-11T08:16:43.782518+0000","last_active":"2024-09-11T08:16:52.740678+0000","last_peered":"2024-09-11T08:16:52.740678+0000","last_clean":"2024-09-11T08:16:52.740678+0000","last_became_active":"2024-09-11T08:07:36.072508+0000","last_became_peered":"2024-09-11T08:07:36.072508+0000","last_unstale":"2024-09-11T08:16:52.740678+0000","last_undegraded":"2024-09-11T08:16:52.740678+0000","last_fullsized":"2024-09-11T08:16:52.740678+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:43.782439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:01.777429+0000","last_clean_scrub_stamp":"2024-09-11T08:16:43.782439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:49.290220+0000","scrub_duration":0.0038441999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"224'13764","reported_seq":13837,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T08:17:02.721041+0000","last_change":"2024-09-11T08:13:54.881747+0000","last_active":"2024-09-11T08:17:02.721041+0000","last_peered":"2024-09-11T08:17:02.721041+0000","last_clean":"2024-09-11T08:17:02.721041+0000","last_became_active":"2024-09-11T08:13:54.881423+0000","last_became_peered":"2024-09-11T08:13:54.881423+0000","last_unstale":"2024-09-11T08:17:02.721041+0000","last_undegraded":"2024-09-11T08:17:02.721041+0000","last_fullsized":"2024-09-11T08:17:02.721041+0000","mapping_epoch":159,"log_start":"206'9000","ondisk_log_start":"206'9000","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:13:53.862306+0000","objects_scrubbed":0,"log_size":4764,"log_dups_size":0,"ondisk_log_size":4764,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151650304,"num_objects":2314,"num_object_clones":0,"num_object_copies":6942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13764,"num_write_kb":514496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"88'16","reported_seq":236,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.746317+0000","last_change":"2024-09-11T08:16:46.015835+0000","last_active":"2024-09-11T08:16:52.746317+0000","last_peered":"2024-09-11T08:16:52.746317+0000","last_clean":"2024-09-11T08:16:52.746317+0000","last_became_active":"2024-09-11T08:07:32.047971+0000","last_became_peered":"2024-09-11T08:07:32.047971+0000","last_unstale":"2024-09-11T08:16:52.746317+0000","last_undegraded":"2024-09-11T08:16:52.746317+0000","last_fullsized":"2024-09-11T08:16:52.746317+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'16","last_scrub_stamp":"2024-09-11T08:16:46.015776+0000","last_deep_scrub":"88'16","last_deep_scrub_stamp":"2024-09-11T08:09:39.635984+0000","last_clean_scrub_stamp":"2024-09-11T08:16:46.015776+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:03.636623+0000","scrub_duration":0.0030649929999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":221,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.746006+0000","last_change":"2024-09-11T08:14:54.023658+0000","last_active":"2024-09-11T08:16:52.746006+0000","last_peered":"2024-09-11T08:16:52.746006+0000","last_clean":"2024-09-11T08:16:52.746006+0000","last_became_active":"2024-09-11T08:14:24.789888+0000","last_became_peered":"2024-09-11T08:14:24.789888+0000","last_unstale":"2024-09-11T08:16:52.746006+0000","last_undegraded":"2024-09-11T08:16:52.746006+0000","last_fullsized":"2024-09-11T08:16:52.746006+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:54.023556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:36.417911+0000","last_clean_scrub_stamp":"2024-09-11T08:14:54.023556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002726596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":209,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.740578+0000","last_change":"2024-09-11T08:15:06.434641+0000","last_active":"2024-09-11T08:16:52.740578+0000","last_peered":"2024-09-11T08:16:52.740578+0000","last_clean":"2024-09-11T08:16:52.740578+0000","last_became_active":"2024-09-11T08:07:34.053347+0000","last_became_peered":"2024-09-11T08:07:34.053347+0000","last_unstale":"2024-09-11T08:16:52.740578+0000","last_undegraded":"2024-09-11T08:16:52.740578+0000","last_fullsized":"2024-09-11T08:16:52.740578+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:06.434588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:01.708166+0000","last_clean_scrub_stamp":"2024-09-11T08:15:06.434588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039794419999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":198,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.742457+0000","last_change":"2024-09-11T08:14:57.236619+0000","last_active":"2024-09-11T08:16:52.742457+0000","last_peered":"2024-09-11T08:16:52.742457+0000","last_clean":"2024-09-11T08:16:52.742457+0000","last_became_active":"2024-09-11T08:07:36.631687+0000","last_became_peered":"2024-09-11T08:07:36.631687+0000","last_unstale":"2024-09-11T08:16:52.742457+0000","last_undegraded":"2024-09-11T08:16:52.742457+0000","last_fullsized":"2024-09-11T08:16:52.742457+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:57.236432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:14:57.236432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029326859999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1","version":"224'13396","reported_seq":13752,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T08:16:58.283308+0000","last_change":"2024-09-11T08:15:57.075945+0000","last_active":"2024-09-11T08:16:58.283308+0000","last_peered":"2024-09-11T08:16:58.283308+0000","last_clean":"2024-09-11T08:16:58.283308+0000","last_became_active":"2024-09-11T08:13:55.707503+0000","last_became_peered":"2024-09-11T08:13:55.707503+0000","last_unstale":"2024-09-11T08:16:58.283308+0000","last_undegraded":"2024-09-11T08:16:58.283308+0000","last_fullsized":"2024-09-11T08:16:58.283308+0000","mapping_epoch":159,"log_start":"202'8300","ondisk_log_start":"202'8300","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'9162","last_scrub_stamp":"2024-09-11T08:15:57.075870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:15:57.075870+0000","objects_scrubbed":6840,"log_size":5096,"log_dups_size":0,"ondisk_log_size":5096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:10.709887+0000","scrub_duration":0.63951687599999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170786816,"num_objects":2606,"num_object_clones":0,"num_object_copies":7818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13396,"num_write_kb":512064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"77'25","reported_seq":276,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.746398+0000","last_change":"2024-09-11T08:15:42.061711+0000","last_active":"2024-09-11T08:16:52.746398+0000","last_peered":"2024-09-11T08:16:52.746398+0000","last_clean":"2024-09-11T08:16:52.746398+0000","last_became_active":"2024-09-11T08:09:08.106771+0000","last_became_peered":"2024-09-11T08:09:08.106771+0000","last_unstale":"2024-09-11T08:16:52.746398+0000","last_undegraded":"2024-09-11T08:16:52.746398+0000","last_fullsized":"2024-09-11T08:16:52.746398+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'25","last_scrub_stamp":"2024-09-11T08:15:42.061632+0000","last_deep_scrub":"77'25","last_deep_scrub_stamp":"2024-09-11T08:08:32.458210+0000","last_clean_scrub_stamp":"2024-09-11T08:15:42.061632+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:06.943716+0000","scrub_duration":0.00500454,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":223,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.742306+0000","last_change":"2024-09-11T08:16:41.507028+0000","last_active":"2024-09-11T08:16:52.742306+0000","last_peered":"2024-09-11T08:16:52.742306+0000","last_clean":"2024-09-11T08:16:52.742306+0000","last_became_active":"2024-09-11T08:07:30.042555+0000","last_became_peered":"2024-09-11T08:07:30.042555+0000","last_unstale":"2024-09-11T08:16:52.742306+0000","last_undegraded":"2024-09-11T08:16:52.742306+0000","last_fullsized":"2024-09-11T08:16:52.742306+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:41.506939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:16:41.506939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:53.258601+0000","scrub_duration":0.0026028340000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":220,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.742166+0000","last_change":"2024-09-11T08:16:52.542153+0000","last_active":"2024-09-11T08:16:52.742166+0000","last_peered":"2024-09-11T08:16:52.742166+0000","last_clean":"2024-09-11T08:16:52.742166+0000","last_became_active":"2024-09-11T08:07:34.054961+0000","last_became_peered":"2024-09-11T08:07:34.054961+0000","last_unstale":"2024-09-11T08:16:52.742166+0000","last_undegraded":"2024-09-11T08:16:52.742166+0000","last_fullsized":"2024-09-11T08:16:52.742166+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:52.542072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:16:52.542072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:14.525068+0000","scrub_duration":0.0052953879999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":227,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.758878+0000","last_change":"2024-09-11T08:15:43.160119+0000","last_active":"2024-09-11T08:16:52.758878+0000","last_peered":"2024-09-11T08:16:52.758878+0000","last_clean":"2024-09-11T08:16:52.758878+0000","last_became_active":"2024-09-11T08:07:36.071194+0000","last_became_peered":"2024-09-11T08:07:36.071194+0000","last_unstale":"2024-09-11T08:16:52.758878+0000","last_undegraded":"2024-09-11T08:16:52.758878+0000","last_fullsized":"2024-09-11T08:16:52.758878+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:43.158812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:07.930137+0000","last_clean_scrub_stamp":"2024-09-11T08:15:43.158812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:00.399779+0000","scrub_duration":0.0051752609999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"224'13519","reported_seq":13713,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T08:16:58.108838+0000","last_change":"2024-09-11T08:16:55.299061+0000","last_active":"2024-09-11T08:16:58.108838+0000","last_peered":"2024-09-11T08:16:58.108838+0000","last_clean":"2024-09-11T08:16:58.108838+0000","last_became_active":"2024-09-11T08:13:54.882196+0000","last_became_peered":"2024-09-11T08:13:54.882196+0000","last_unstale":"2024-09-11T08:16:58.108838+0000","last_undegraded":"2024-09-11T08:16:58.108838+0000","last_fullsized":"2024-09-11T08:16:58.108838+0000","mapping_epoch":159,"log_start":"203'8735","ondisk_log_start":"203'8735","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'13345","last_scrub_stamp":"2024-09-11T08:16:55.298997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:16:55.298997+0000","objects_scrubbed":2788,"log_size":4784,"log_dups_size":0,"ondisk_log_size":4784,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:06.363400+0000","scrub_duration":0.74703533799999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169803776,"num_objects":2591,"num_object_clones":0,"num_object_copies":7773,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13519,"num_write_kb":515520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"224'15","reported_seq":232,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T08:17:01.126634+0000","last_change":"2024-09-11T08:17:01.126558+0000","last_active":"2024-09-11T08:17:01.126634+0000","last_peered":"2024-09-11T08:17:01.126634+0000","last_clean":"2024-09-11T08:17:01.126634+0000","last_became_active":"2024-09-11T08:07:32.043314+0000","last_became_peered":"2024-09-11T08:07:32.043314+0000","last_unstale":"2024-09-11T08:17:01.126634+0000","last_undegraded":"2024-09-11T08:17:01.126634+0000","last_fullsized":"2024-09-11T08:17:01.126634+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'15","last_scrub_stamp":"2024-09-11T08:17:01.126478+0000","last_deep_scrub":"224'15","last_deep_scrub_stamp":"2024-09-11T08:17:01.126478+0000","last_clean_scrub_stamp":"2024-09-11T08:17:01.126478+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:16.625548+0000","scrub_duration":0.0092237729999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":214,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.742092+0000","last_change":"2024-09-11T08:15:48.405028+0000","last_active":"2024-09-11T08:16:52.742092+0000","last_peered":"2024-09-11T08:16:52.742092+0000","last_clean":"2024-09-11T08:16:52.742092+0000","last_became_active":"2024-09-11T08:07:30.046684+0000","last_became_peered":"2024-09-11T08:07:30.046684+0000","last_unstale":"2024-09-11T08:16:52.742092+0000","last_undegraded":"2024-09-11T08:16:52.742092+0000","last_fullsized":"2024-09-11T08:16:52.742092+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:48.404853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:48.404853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:01.459915+0000","scrub_duration":0.0048699770000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"171'12","reported_seq":364,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:53.663526+0000","last_change":"2024-09-11T08:16:50.043791+0000","last_active":"2024-09-11T08:16:53.663526+0000","last_peered":"2024-09-11T08:16:53.663526+0000","last_clean":"2024-09-11T08:16:53.663526+0000","last_became_active":"2024-09-11T08:14:24.794446+0000","last_became_peered":"2024-09-11T08:14:24.794446+0000","last_unstale":"2024-09-11T08:16:53.663526+0000","last_undegraded":"2024-09-11T08:16:53.663526+0000","last_fullsized":"2024-09-11T08:16:53.663526+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'12","last_scrub_stamp":"2024-09-11T08:16:50.043698+0000","last_deep_scrub":"171'12","last_deep_scrub_stamp":"2024-09-11T08:14:56.881299+0000","last_clean_scrub_stamp":"2024-09-11T08:16:50.043698+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:57.810719+0000","scrub_duration":0.0069122760000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":198,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.746045+0000","last_change":"2024-09-11T08:14:13.941569+0000","last_active":"2024-09-11T08:16:52.746045+0000","last_peered":"2024-09-11T08:16:52.746045+0000","last_clean":"2024-09-11T08:16:52.746045+0000","last_became_active":"2024-09-11T08:07:36.074684+0000","last_became_peered":"2024-09-11T08:07:36.074684+0000","last_unstale":"2024-09-11T08:16:52.746045+0000","last_undegraded":"2024-09-11T08:16:52.746045+0000","last_fullsized":"2024-09-11T08:16:52.746045+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:13.941492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:14.613532+0000","last_clean_scrub_stamp":"2024-09-11T08:14:13.941492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037661470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.7","version":"224'13490","reported_seq":13961,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T08:16:58.283854+0000","last_change":"2024-09-11T08:15:56.302099+0000","last_active":"2024-09-11T08:16:58.283854+0000","last_peered":"2024-09-11T08:16:58.283854+0000","last_clean":"2024-09-11T08:16:58.283854+0000","last_became_active":"2024-09-11T08:13:54.883624+0000","last_became_peered":"2024-09-11T08:13:54.883624+0000","last_unstale":"2024-09-11T08:16:58.283854+0000","last_undegraded":"2024-09-11T08:16:58.283854+0000","last_fullsized":"2024-09-11T08:16:58.283854+0000","mapping_epoch":159,"log_start":"203'8400","ondisk_log_start":"203'8400","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'9119","last_scrub_stamp":"2024-09-11T08:15:56.302032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:15:56.302032+0000","objects_scrubbed":6919,"log_size":5090,"log_dups_size":0,"ondisk_log_size":5090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:02.822566+0000","scrub_duration":0.82679451100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166985728,"num_objects":2548,"num_object_clones":0,"num_object_copies":7644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13490,"num_write_kb":513216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":228,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.755387+0000","last_change":"2024-09-11T08:16:49.692191+0000","last_active":"2024-09-11T08:16:52.755387+0000","last_peered":"2024-09-11T08:16:52.755387+0000","last_clean":"2024-09-11T08:16:52.755387+0000","last_became_active":"2024-09-11T08:07:32.054885+0000","last_became_peered":"2024-09-11T08:07:32.054885+0000","last_unstale":"2024-09-11T08:16:52.755387+0000","last_undegraded":"2024-09-11T08:16:52.755387+0000","last_fullsized":"2024-09-11T08:16:52.755387+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:16:49.692128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:16:49.692128+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:01.355684+0000","scrub_duration":0.012332951999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":223,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.740453+0000","last_change":"2024-09-11T08:15:34.497175+0000","last_active":"2024-09-11T08:16:52.740453+0000","last_peered":"2024-09-11T08:16:52.740453+0000","last_clean":"2024-09-11T08:16:52.740453+0000","last_became_active":"2024-09-11T08:07:30.045613+0000","last_became_peered":"2024-09-11T08:07:30.045613+0000","last_unstale":"2024-09-11T08:16:52.740453+0000","last_undegraded":"2024-09-11T08:16:52.740453+0000","last_fullsized":"2024-09-11T08:16:52.740453+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:34.497125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:34.497125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052802250000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":229,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.746167+0000","last_change":"2024-09-11T08:15:37.123094+0000","last_active":"2024-09-11T08:16:52.746167+0000","last_peered":"2024-09-11T08:16:52.746167+0000","last_clean":"2024-09-11T08:16:52.746167+0000","last_became_active":"2024-09-11T08:07:34.056606+0000","last_became_peered":"2024-09-11T08:07:34.056606+0000","last_unstale":"2024-09-11T08:16:52.746167+0000","last_undegraded":"2024-09-11T08:16:52.746167+0000","last_fullsized":"2024-09-11T08:16:52.746167+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:37.123010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:15:37.123010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:05.989949+0000","scrub_duration":0.0042949169999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":225,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.756725+0000","last_change":"2024-09-11T08:15:56.520006+0000","last_active":"2024-09-11T08:16:52.756725+0000","last_peered":"2024-09-11T08:16:52.756725+0000","last_clean":"2024-09-11T08:16:52.756725+0000","last_became_active":"2024-09-11T08:15:56.518884+0000","last_became_peered":"2024-09-11T08:15:56.518884+0000","last_unstale":"2024-09-11T08:16:52.756725+0000","last_undegraded":"2024-09-11T08:16:52.756725+0000","last_fullsized":"2024-09-11T08:16:52.756725+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:14:01.376411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:14:01.376411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012067320000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"224'13673","reported_seq":17887,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T08:16:58.215441+0000","last_change":"2024-09-11T08:14:51.900945+0000","last_active":"2024-09-11T08:16:58.215441+0000","last_peered":"2024-09-11T08:16:58.215441+0000","last_clean":"2024-09-11T08:16:58.215441+0000","last_became_active":"2024-09-11T08:14:51.879122+0000","last_became_peered":"2024-09-11T08:14:51.879122+0000","last_unstale":"2024-09-11T08:16:58.215441+0000","last_undegraded":"2024-09-11T08:16:58.215441+0000","last_fullsized":"2024-09-11T08:16:58.215441+0000","mapping_epoch":180,"log_start":"199'7700","ondisk_log_start":"199'7700","created":159,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:13:53.862306+0000","objects_scrubbed":0,"log_size":5973,"log_dups_size":0,"ondisk_log_size":5973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175833088,"num_objects":2683,"num_object_clones":0,"num_object_copies":8049,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13673,"num_write_kb":523392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2061,"num_bytes_recovered":135069696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":319,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.742351+0000","last_change":"2024-09-11T08:15:56.551439+0000","last_active":"2024-09-11T08:16:52.742351+0000","last_peered":"2024-09-11T08:16:52.742351+0000","last_clean":"2024-09-11T08:16:52.742351+0000","last_became_active":"2024-09-11T08:15:56.526961+0000","last_became_peered":"2024-09-11T08:15:56.526961+0000","last_unstale":"2024-09-11T08:16:52.742351+0000","last_undegraded":"2024-09-11T08:16:52.742351+0000","last_fullsized":"2024-09-11T08:16:52.742351+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-11T08:15:49.187186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:15:49.187186+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:17.036763+0000","scrub_duration":0.0054603430000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":36,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.739090+0000","last_change":"2024-09-11T08:16:46.578643+0000","last_active":"2024-09-11T08:16:52.739090+0000","last_peered":"2024-09-11T08:16:52.739090+0000","last_clean":"2024-09-11T08:16:52.739090+0000","last_became_active":"2024-09-11T08:15:56.530370+0000","last_became_peered":"2024-09-11T08:15:56.530370+0000","last_unstale":"2024-09-11T08:16:52.739090+0000","last_undegraded":"2024-09-11T08:16:52.739090+0000","last_fullsized":"2024-09-11T08:16:52.739090+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:46.578575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:44.385372+0000","last_clean_scrub_stamp":"2024-09-11T08:16:46.578575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:15.109608+0000","scrub_duration":0.016632636999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":232,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T08:16:55.019445+0000","last_change":"2024-09-11T08:16:55.019406+0000","last_active":"2024-09-11T08:16:55.019445+0000","last_peered":"2024-09-11T08:16:55.019445+0000","last_clean":"2024-09-11T08:16:55.019445+0000","last_became_active":"2024-09-11T08:07:34.063589+0000","last_became_peered":"2024-09-11T08:07:34.063589+0000","last_unstale":"2024-09-11T08:16:55.019445+0000","last_undegraded":"2024-09-11T08:16:55.019445+0000","last_fullsized":"2024-09-11T08:16:55.019445+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:55.019351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:16:55.019351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:07.074171+0000","scrub_duration":0.001926295,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":26,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.738053+0000","last_change":"2024-09-11T08:15:56.518476+0000","last_active":"2024-09-11T08:16:52.738053+0000","last_peered":"2024-09-11T08:16:52.738053+0000","last_clean":"2024-09-11T08:16:52.738053+0000","last_became_active":"2024-09-11T08:15:56.518169+0000","last_became_peered":"2024-09-11T08:15:56.518169+0000","last_unstale":"2024-09-11T08:16:52.738053+0000","last_undegraded":"2024-09-11T08:16:52.738053+0000","last_fullsized":"2024-09-11T08:16:52.738053+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:32.131008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:15:32.131008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:59.362244+0000","scrub_duration":0.0064196519999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"224'14013","reported_seq":18933,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T08:17:02.798570+0000","last_change":"2024-09-11T08:16:48.953408+0000","last_active":"2024-09-11T08:17:02.798570+0000","last_peered":"2024-09-11T08:17:02.798570+0000","last_clean":"2024-09-11T08:17:02.798570+0000","last_became_active":"2024-09-11T08:14:50.875228+0000","last_became_peered":"2024-09-11T08:14:50.875228+0000","last_unstale":"2024-09-11T08:17:02.798570+0000","last_undegraded":"2024-09-11T08:17:02.798570+0000","last_fullsized":"2024-09-11T08:17:02.798570+0000","mapping_epoch":179,"log_start":"200'7976","ondisk_log_start":"200'7976","created":159,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"221'13122","last_scrub_stamp":"2024-09-11T08:16:48.953325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:16:48.953325+0000","objects_scrubbed":3197,"log_size":6037,"log_dups_size":0,"ondisk_log_size":6037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:17.019253+0000","scrub_duration":0.519289475,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150405120,"num_objects":2295,"num_object_clones":0,"num_object_copies":6885,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14013,"num_write_kb":521856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2184,"num_bytes_recovered":143130624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"145'17","reported_seq":240,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.745837+0000","last_change":"2024-09-11T08:15:53.040339+0000","last_active":"2024-09-11T08:16:52.745837+0000","last_peered":"2024-09-11T08:16:52.745837+0000","last_clean":"2024-09-11T08:16:52.745837+0000","last_became_active":"2024-09-11T08:07:32.048164+0000","last_became_peered":"2024-09-11T08:07:32.048164+0000","last_unstale":"2024-09-11T08:16:52.745837+0000","last_undegraded":"2024-09-11T08:16:52.745837+0000","last_fullsized":"2024-09-11T08:16:52.745837+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'17","last_scrub_stamp":"2024-09-11T08:15:53.040223+0000","last_deep_scrub":"145'17","last_deep_scrub_stamp":"2024-09-11T08:13:08.160848+0000","last_clean_scrub_stamp":"2024-09-11T08:15:53.040223+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:12.632932+0000","scrub_duration":0.0039160820000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":36,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.739116+0000","last_change":"2024-09-11T08:16:47.557730+0000","last_active":"2024-09-11T08:16:52.739116+0000","last_peered":"2024-09-11T08:16:52.739116+0000","last_clean":"2024-09-11T08:16:52.739116+0000","last_became_active":"2024-09-11T08:15:56.525845+0000","last_became_peered":"2024-09-11T08:15:56.525845+0000","last_unstale":"2024-09-11T08:16:52.739116+0000","last_undegraded":"2024-09-11T08:16:52.739116+0000","last_fullsized":"2024-09-11T08:16:52.739116+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:47.557675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:53.247275+0000","last_clean_scrub_stamp":"2024-09-11T08:16:47.557675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:01.123588+0000","scrub_duration":0.0086777009999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":219,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.745796+0000","last_change":"2024-09-11T08:16:38.072401+0000","last_active":"2024-09-11T08:16:52.745796+0000","last_peered":"2024-09-11T08:16:52.745796+0000","last_clean":"2024-09-11T08:16:52.745796+0000","last_became_active":"2024-09-11T08:07:34.057479+0000","last_became_peered":"2024-09-11T08:07:34.057479+0000","last_unstale":"2024-09-11T08:16:52.745796+0000","last_undegraded":"2024-09-11T08:16:52.745796+0000","last_fullsized":"2024-09-11T08:16:52.745796+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:38.072323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:16:38.072323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:47.397010+0000","scrub_duration":0.005761909,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":235,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.741735+0000","last_change":"2024-09-11T08:16:50.631547+0000","last_active":"2024-09-11T08:16:52.741735+0000","last_peered":"2024-09-11T08:16:52.741735+0000","last_clean":"2024-09-11T08:16:52.741735+0000","last_became_active":"2024-09-11T08:15:56.522624+0000","last_became_peered":"2024-09-11T08:15:56.522624+0000","last_unstale":"2024-09-11T08:16:52.741735+0000","last_undegraded":"2024-09-11T08:16:52.741735+0000","last_fullsized":"2024-09-11T08:16:52.741735+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:50.631471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:16:50.631471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:18.067947+0000","scrub_duration":0.0037889149999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"224'13592","reported_seq":13966,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T08:16:57.904813+0000","last_change":"2024-09-11T08:15:53.462987+0000","last_active":"2024-09-11T08:16:57.904813+0000","last_peered":"2024-09-11T08:16:57.904813+0000","last_clean":"2024-09-11T08:16:57.904813+0000","last_became_active":"2024-09-11T08:13:55.705819+0000","last_became_peered":"2024-09-11T08:13:55.705819+0000","last_unstale":"2024-09-11T08:16:57.904813+0000","last_undegraded":"2024-09-11T08:16:57.904813+0000","last_fullsized":"2024-09-11T08:16:57.904813+0000","mapping_epoch":159,"log_start":"202'8300","ondisk_log_start":"202'8300","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"206'9066","last_scrub_stamp":"2024-09-11T08:15:53.462907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:53.862306+0000","last_clean_scrub_stamp":"2024-09-11T08:15:53.462907+0000","objects_scrubbed":7268,"log_size":5292,"log_dups_size":0,"ondisk_log_size":5292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:18.668302+0000","scrub_duration":2.0133290810000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175505408,"num_objects":2678,"num_object_clones":0,"num_object_copies":8034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13592,"num_write_kb":520640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":295,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.739901+0000","last_change":"2024-09-11T08:15:56.551197+0000","last_active":"2024-09-11T08:16:52.739901+0000","last_peered":"2024-09-11T08:16:52.739901+0000","last_clean":"2024-09-11T08:16:52.739901+0000","last_became_active":"2024-09-11T08:15:56.532431+0000","last_became_peered":"2024-09-11T08:15:56.532431+0000","last_unstale":"2024-09-11T08:16:52.739901+0000","last_undegraded":"2024-09-11T08:16:52.739901+0000","last_fullsized":"2024-09-11T08:16:52.739901+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-11T08:15:47.427884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:15:47.427884+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016881800000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":42,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.738393+0000","last_change":"2024-09-11T08:16:42.714111+0000","last_active":"2024-09-11T08:16:52.738393+0000","last_peered":"2024-09-11T08:16:52.738393+0000","last_clean":"2024-09-11T08:16:52.738393+0000","last_became_active":"2024-09-11T08:15:56.515328+0000","last_became_peered":"2024-09-11T08:15:56.515328+0000","last_unstale":"2024-09-11T08:16:52.738393+0000","last_undegraded":"2024-09-11T08:16:52.738393+0000","last_fullsized":"2024-09-11T08:16:52.738393+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-11T08:16:42.714038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:16:42.714038+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:06.366726+0000","scrub_duration":0.0083916919999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"209'11","reported_seq":43,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T08:16:53.755104+0000","last_change":"2024-09-11T08:15:56.534905+0000","last_active":"2024-09-11T08:16:53.755104+0000","last_peered":"2024-09-11T08:16:53.755104+0000","last_clean":"2024-09-11T08:16:53.755104+0000","last_became_active":"2024-09-11T08:15:56.524513+0000","last_became_peered":"2024-09-11T08:15:56.524513+0000","last_unstale":"2024-09-11T08:16:53.755104+0000","last_undegraded":"2024-09-11T08:16:53.755104+0000","last_fullsized":"2024-09-11T08:16:53.755104+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"156'10","last_scrub_stamp":"2024-09-11T08:15:38.215399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:15:38.215399+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:00.740762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":207,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.740008+0000","last_change":"2024-09-11T08:15:03.484853+0000","last_active":"2024-09-11T08:16:52.740008+0000","last_peered":"2024-09-11T08:16:52.740008+0000","last_clean":"2024-09-11T08:16:52.740008+0000","last_became_active":"2024-09-11T08:07:36.631755+0000","last_became_peered":"2024-09-11T08:07:36.631755+0000","last_unstale":"2024-09-11T08:16:52.740008+0000","last_undegraded":"2024-09-11T08:16:52.740008+0000","last_fullsized":"2024-09-11T08:16:52.740008+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:03.484755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:03.484755+0000","last_clean_scrub_stamp":"2024-09-11T08:15:03.484755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0064296900000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":233,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T08:16:57.502036+0000","last_change":"2024-09-11T08:16:57.501989+0000","last_active":"2024-09-11T08:16:57.502036+0000","last_peered":"2024-09-11T08:16:57.502036+0000","last_clean":"2024-09-11T08:16:57.502036+0000","last_became_active":"2024-09-11T08:07:30.045910+0000","last_became_peered":"2024-09-11T08:07:30.045910+0000","last_unstale":"2024-09-11T08:16:57.502036+0000","last_undegraded":"2024-09-11T08:16:57.502036+0000","last_fullsized":"2024-09-11T08:16:57.502036+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:57.501885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:16:57.501885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:02.513820+0000","scrub_duration":0.0050562519999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"84'9","reported_seq":257,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.743289+0000","last_change":"2024-09-11T08:15:56.550790+0000","last_active":"2024-09-11T08:16:52.743289+0000","last_peered":"2024-09-11T08:16:52.743289+0000","last_clean":"2024-09-11T08:16:52.743289+0000","last_became_active":"2024-09-11T08:15:56.526325+0000","last_became_peered":"2024-09-11T08:15:56.526325+0000","last_unstale":"2024-09-11T08:16:52.743289+0000","last_undegraded":"2024-09-11T08:16:52.743289+0000","last_fullsized":"2024-09-11T08:16:52.743289+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-09-11T08:14:12.071950+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-09-11T08:09:03.418457+0000","last_clean_scrub_stamp":"2024-09-11T08:14:12.071950+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0068954059999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":219,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:53.549123+0000","last_change":"2024-09-11T08:16:53.549060+0000","last_active":"2024-09-11T08:16:53.549123+0000","last_peered":"2024-09-11T08:16:53.549123+0000","last_clean":"2024-09-11T08:16:53.549123+0000","last_became_active":"2024-09-11T08:07:34.057774+0000","last_became_peered":"2024-09-11T08:07:34.057774+0000","last_unstale":"2024-09-11T08:16:53.549123+0000","last_undegraded":"2024-09-11T08:16:53.549123+0000","last_fullsized":"2024-09-11T08:16:53.549123+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:53.548967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:53.548967+0000","last_clean_scrub_stamp":"2024-09-11T08:16:53.548967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:17.071272+0000","scrub_duration":0.0067671800000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":161,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:53.670788+0000","last_change":"2024-09-11T08:16:53.669111+0000","last_active":"2024-09-11T08:16:53.670788+0000","last_peered":"2024-09-11T08:16:53.670788+0000","last_clean":"2024-09-11T08:16:53.670788+0000","last_became_active":"2024-09-11T08:14:24.790678+0000","last_became_peered":"2024-09-11T08:14:24.790678+0000","last_unstale":"2024-09-11T08:16:53.670788+0000","last_undegraded":"2024-09-11T08:16:53.670788+0000","last_fullsized":"2024-09-11T08:16:53.670788+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:53.669034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:16:53.669034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:56.180252+0000","scrub_duration":0.0015615850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"84'13","reported_seq":269,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.817210+0000","last_change":"2024-09-11T08:15:56.532882+0000","last_active":"2024-09-11T08:16:52.817210+0000","last_peered":"2024-09-11T08:16:52.817210+0000","last_clean":"2024-09-11T08:16:52.817210+0000","last_became_active":"2024-09-11T08:15:56.514793+0000","last_became_peered":"2024-09-11T08:15:56.514793+0000","last_unstale":"2024-09-11T08:16:52.817210+0000","last_undegraded":"2024-09-11T08:16:52.817210+0000","last_fullsized":"2024-09-11T08:16:52.817210+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'13","last_scrub_stamp":"2024-09-11T08:14:05.339509+0000","last_deep_scrub":"84'13","last_deep_scrub_stamp":"2024-09-11T08:08:57.835634+0000","last_clean_scrub_stamp":"2024-09-11T08:14:05.339509+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001614047,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":184,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.738490+0000","last_change":"2024-09-11T08:15:31.534404+0000","last_active":"2024-09-11T08:16:52.738490+0000","last_peered":"2024-09-11T08:16:52.738490+0000","last_clean":"2024-09-11T08:16:52.738490+0000","last_became_active":"2024-09-11T08:14:24.786021+0000","last_became_peered":"2024-09-11T08:14:24.786021+0000","last_unstale":"2024-09-11T08:16:52.738490+0000","last_undegraded":"2024-09-11T08:16:52.738490+0000","last_fullsized":"2024-09-11T08:16:52.738490+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:31.534302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:30.471620+0000","last_clean_scrub_stamp":"2024-09-11T08:15:31.534302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021837200000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":227,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.743157+0000","last_change":"2024-09-11T08:15:56.526834+0000","last_active":"2024-09-11T08:16:52.743157+0000","last_peered":"2024-09-11T08:16:52.743157+0000","last_clean":"2024-09-11T08:16:52.743157+0000","last_became_active":"2024-09-11T08:15:56.525717+0000","last_became_peered":"2024-09-11T08:15:56.525717+0000","last_unstale":"2024-09-11T08:16:52.743157+0000","last_undegraded":"2024-09-11T08:16:52.743157+0000","last_fullsized":"2024-09-11T08:16:52.743157+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:59.988381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:04.370734+0000","last_clean_scrub_stamp":"2024-09-11T08:13:59.988381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017033700000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":234,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.745585+0000","last_change":"2024-09-11T08:15:56.525429+0000","last_active":"2024-09-11T08:16:52.745585+0000","last_peered":"2024-09-11T08:16:52.745585+0000","last_clean":"2024-09-11T08:16:52.745585+0000","last_became_active":"2024-09-11T08:15:56.525120+0000","last_became_peered":"2024-09-11T08:15:56.525120+0000","last_unstale":"2024-09-11T08:16:52.745585+0000","last_undegraded":"2024-09-11T08:16:52.745585+0000","last_fullsized":"2024-09-11T08:16:52.745585+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:54.184372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:54.184372+0000","last_clean_scrub_stamp":"2024-09-11T08:15:54.184372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:21.919337+0000","scrub_duration":0.013922767000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":284,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.740523+0000","last_change":"2024-09-11T08:15:56.547635+0000","last_active":"2024-09-11T08:16:52.740523+0000","last_peered":"2024-09-11T08:16:52.740523+0000","last_clean":"2024-09-11T08:16:52.740523+0000","last_became_active":"2024-09-11T08:15:56.532513+0000","last_became_peered":"2024-09-11T08:15:56.532513+0000","last_unstale":"2024-09-11T08:16:52.740523+0000","last_undegraded":"2024-09-11T08:16:52.740523+0000","last_fullsized":"2024-09-11T08:16:52.740523+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:15:52.443189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:15:52.443189+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010882618,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":241,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.754497+0000","last_change":"2024-09-11T08:16:52.661667+0000","last_active":"2024-09-11T08:16:52.754497+0000","last_peered":"2024-09-11T08:16:52.754497+0000","last_clean":"2024-09-11T08:16:52.754497+0000","last_became_active":"2024-09-11T08:15:56.513278+0000","last_became_peered":"2024-09-11T08:15:56.513278+0000","last_unstale":"2024-09-11T08:16:52.754497+0000","last_undegraded":"2024-09-11T08:16:52.754497+0000","last_fullsized":"2024-09-11T08:16:52.754497+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:52.661591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:52.661591+0000","last_clean_scrub_stamp":"2024-09-11T08:16:52.661591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:02.731607+0000","scrub_duration":0.0045159830000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"222'12","reported_seq":368,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:53.663839+0000","last_change":"2024-09-11T08:16:52.593122+0000","last_active":"2024-09-11T08:16:53.663839+0000","last_peered":"2024-09-11T08:16:53.663839+0000","last_clean":"2024-09-11T08:16:53.663839+0000","last_became_active":"2024-09-11T08:15:56.525253+0000","last_became_peered":"2024-09-11T08:15:56.525253+0000","last_unstale":"2024-09-11T08:16:53.663839+0000","last_undegraded":"2024-09-11T08:16:53.663839+0000","last_fullsized":"2024-09-11T08:16:53.663839+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'12","last_scrub_stamp":"2024-09-11T08:16:52.592856+0000","last_deep_scrub":"222'12","last_deep_scrub_stamp":"2024-09-11T08:16:52.592856+0000","last_clean_scrub_stamp":"2024-09-11T08:16:52.592856+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:19.514425+0000","scrub_duration":0.011996981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":198,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.746180+0000","last_change":"2024-09-11T08:15:05.867100+0000","last_active":"2024-09-11T08:16:52.746180+0000","last_peered":"2024-09-11T08:16:52.746180+0000","last_clean":"2024-09-11T08:16:52.746180+0000","last_became_active":"2024-09-11T08:07:36.068493+0000","last_became_peered":"2024-09-11T08:07:36.068493+0000","last_unstale":"2024-09-11T08:16:52.746180+0000","last_undegraded":"2024-09-11T08:16:52.746180+0000","last_fullsized":"2024-09-11T08:16:52.746180+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:05.867013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:05.867013+0000","last_clean_scrub_stamp":"2024-09-11T08:15:05.867013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0061248279999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":256,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.742843+0000","last_change":"2024-09-11T08:14:56.223381+0000","last_active":"2024-09-11T08:16:52.742843+0000","last_peered":"2024-09-11T08:16:52.742843+0000","last_clean":"2024-09-11T08:16:52.742843+0000","last_became_active":"2024-09-11T08:14:24.791187+0000","last_became_peered":"2024-09-11T08:14:24.791187+0000","last_unstale":"2024-09-11T08:16:52.742843+0000","last_undegraded":"2024-09-11T08:16:52.742843+0000","last_fullsized":"2024-09-11T08:16:52.742843+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-11T08:14:56.222082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:14:56.222082+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011605691,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":26,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.738436+0000","last_change":"2024-09-11T08:15:56.521890+0000","last_active":"2024-09-11T08:16:52.738436+0000","last_peered":"2024-09-11T08:16:52.738436+0000","last_clean":"2024-09-11T08:16:52.738436+0000","last_became_active":"2024-09-11T08:15:56.518515+0000","last_became_peered":"2024-09-11T08:15:56.518515+0000","last_unstale":"2024-09-11T08:16:52.738436+0000","last_undegraded":"2024-09-11T08:16:52.738436+0000","last_fullsized":"2024-09-11T08:16:52.738436+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:33.021088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:33.021088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":330,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:53.663188+0000","last_change":"2024-09-11T08:15:44.429144+0000","last_active":"2024-09-11T08:16:53.663188+0000","last_peered":"2024-09-11T08:16:53.663188+0000","last_clean":"2024-09-11T08:16:53.663188+0000","last_became_active":"2024-09-11T08:07:34.060736+0000","last_became_peered":"2024-09-11T08:07:34.060736+0000","last_unstale":"2024-09-11T08:16:53.663188+0000","last_undegraded":"2024-09-11T08:16:53.663188+0000","last_fullsized":"2024-09-11T08:16:53.663188+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:15:44.429062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:15:44.429062+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:58.251305+0000","scrub_duration":0.0062457210000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":225,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.816978+0000","last_change":"2024-09-11T08:15:56.515344+0000","last_active":"2024-09-11T08:16:52.816978+0000","last_peered":"2024-09-11T08:16:52.816978+0000","last_clean":"2024-09-11T08:16:52.816978+0000","last_became_active":"2024-09-11T08:15:56.513949+0000","last_became_peered":"2024-09-11T08:15:56.513949+0000","last_unstale":"2024-09-11T08:16:52.816978+0000","last_undegraded":"2024-09-11T08:16:52.816978+0000","last_fullsized":"2024-09-11T08:16:52.816978+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:01.448261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:15:01.448261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.021382708,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":231,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T08:16:55.089679+0000","last_change":"2024-09-11T08:16:55.089638+0000","last_active":"2024-09-11T08:16:55.089679+0000","last_peered":"2024-09-11T08:16:55.089679+0000","last_clean":"2024-09-11T08:16:55.089679+0000","last_became_active":"2024-09-11T08:07:32.047981+0000","last_became_peered":"2024-09-11T08:07:32.047981+0000","last_unstale":"2024-09-11T08:16:55.089679+0000","last_undegraded":"2024-09-11T08:16:55.089679+0000","last_fullsized":"2024-09-11T08:16:55.089679+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-11T08:16:55.089545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:16:55.089545+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:12.681553+0000","scrub_duration":0.0037132879999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":231,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.746250+0000","last_change":"2024-09-11T08:15:56.523858+0000","last_active":"2024-09-11T08:16:52.746250+0000","last_peered":"2024-09-11T08:16:52.746250+0000","last_clean":"2024-09-11T08:16:52.746250+0000","last_became_active":"2024-09-11T08:15:56.523574+0000","last_became_peered":"2024-09-11T08:15:56.523574+0000","last_unstale":"2024-09-11T08:16:52.746250+0000","last_undegraded":"2024-09-11T08:16:52.746250+0000","last_fullsized":"2024-09-11T08:16:52.746250+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:13:55.845422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:13:55.845422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00143399,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":213,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.743689+0000","last_change":"2024-09-11T08:15:38.410767+0000","last_active":"2024-09-11T08:16:52.743689+0000","last_peered":"2024-09-11T08:16:52.743689+0000","last_clean":"2024-09-11T08:16:52.743689+0000","last_became_active":"2024-09-11T08:07:34.053137+0000","last_became_peered":"2024-09-11T08:07:34.053137+0000","last_unstale":"2024-09-11T08:16:52.743689+0000","last_undegraded":"2024-09-11T08:16:52.743689+0000","last_fullsized":"2024-09-11T08:16:52.743689+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:38.410625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:15:38.410625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:16:59.728039+0000","scrub_duration":0.021540422,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":198,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.740317+0000","last_change":"2024-09-11T08:15:56.494833+0000","last_active":"2024-09-11T08:16:52.740317+0000","last_peered":"2024-09-11T08:16:52.740317+0000","last_clean":"2024-09-11T08:16:52.740317+0000","last_became_active":"2024-09-11T08:07:36.076132+0000","last_became_peered":"2024-09-11T08:07:36.076132+0000","last_unstale":"2024-09-11T08:16:52.740317+0000","last_undegraded":"2024-09-11T08:16:52.740317+0000","last_fullsized":"2024-09-11T08:16:52.740317+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:56.494764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:15:56.494764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:18.392464+0000","scrub_duration":0.001948635,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":238,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.743967+0000","last_change":"2024-09-11T08:16:04.409705+0000","last_active":"2024-09-11T08:16:52.743967+0000","last_peered":"2024-09-11T08:16:52.743967+0000","last_clean":"2024-09-11T08:16:52.743967+0000","last_became_active":"2024-09-11T08:07:32.049368+0000","last_became_peered":"2024-09-11T08:07:32.049368+0000","last_unstale":"2024-09-11T08:16:52.743967+0000","last_undegraded":"2024-09-11T08:16:52.743967+0000","last_fullsized":"2024-09-11T08:16:52.743967+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:16:04.409608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:16:04.409608+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:19.702294+0000","scrub_duration":0.0094835470000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":36,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.738779+0000","last_change":"2024-09-11T08:16:51.628468+0000","last_active":"2024-09-11T08:16:52.738779+0000","last_peered":"2024-09-11T08:16:52.738779+0000","last_clean":"2024-09-11T08:16:52.738779+0000","last_became_active":"2024-09-11T08:15:56.521570+0000","last_became_peered":"2024-09-11T08:15:56.521570+0000","last_unstale":"2024-09-11T08:16:52.738779+0000","last_undegraded":"2024-09-11T08:16:52.738779+0000","last_fullsized":"2024-09-11T08:16:52.738779+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:51.628378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:16:51.628378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:53.598838+0000","scrub_duration":0.001961422,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":200,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.741847+0000","last_change":"2024-09-11T08:15:01.218687+0000","last_active":"2024-09-11T08:16:52.741847+0000","last_peered":"2024-09-11T08:16:52.741847+0000","last_clean":"2024-09-11T08:16:52.741847+0000","last_became_active":"2024-09-11T08:07:34.057064+0000","last_became_peered":"2024-09-11T08:07:34.057064+0000","last_unstale":"2024-09-11T08:16:52.741847+0000","last_undegraded":"2024-09-11T08:16:52.741847+0000","last_fullsized":"2024-09-11T08:16:52.741847+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:01.218475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:01.218475+0000","last_clean_scrub_stamp":"2024-09-11T08:15:01.218475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0099033660000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":208,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.755311+0000","last_change":"2024-09-11T08:16:39.635237+0000","last_active":"2024-09-11T08:16:52.755311+0000","last_peered":"2024-09-11T08:16:52.755311+0000","last_clean":"2024-09-11T08:16:52.755311+0000","last_became_active":"2024-09-11T08:07:36.077406+0000","last_became_peered":"2024-09-11T08:07:36.077406+0000","last_unstale":"2024-09-11T08:16:52.755311+0000","last_undegraded":"2024-09-11T08:16:52.755311+0000","last_fullsized":"2024-09-11T08:16:52.755311+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:39.635151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:09.973943+0000","last_clean_scrub_stamp":"2024-09-11T08:16:39.635151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:45.940617+0000","scrub_duration":0.0062349479999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":282,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:53.599851+0000","last_change":"2024-09-11T08:16:53.599766+0000","last_active":"2024-09-11T08:16:53.599851+0000","last_peered":"2024-09-11T08:16:53.599851+0000","last_clean":"2024-09-11T08:16:53.599851+0000","last_became_active":"2024-09-11T08:15:56.520058+0000","last_became_peered":"2024-09-11T08:15:56.520058+0000","last_unstale":"2024-09-11T08:16:53.599851+0000","last_undegraded":"2024-09-11T08:16:53.599851+0000","last_fullsized":"2024-09-11T08:16:53.599851+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-11T08:16:53.599677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:16:53.599677+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:20.621343+0000","scrub_duration":0.0071859539999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":240,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.740395+0000","last_change":"2024-09-11T08:15:56.531914+0000","last_active":"2024-09-11T08:16:52.740395+0000","last_peered":"2024-09-11T08:16:52.740395+0000","last_clean":"2024-09-11T08:16:52.740395+0000","last_became_active":"2024-09-11T08:15:56.531612+0000","last_became_peered":"2024-09-11T08:15:56.531612+0000","last_unstale":"2024-09-11T08:16:52.740395+0000","last_undegraded":"2024-09-11T08:16:52.740395+0000","last_fullsized":"2024-09-11T08:16:52.740395+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:49.478359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:49.478359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:16.513888+0000","scrub_duration":0.012202396000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":36,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.738553+0000","last_change":"2024-09-11T08:16:38.643733+0000","last_active":"2024-09-11T08:16:52.738553+0000","last_peered":"2024-09-11T08:16:52.738553+0000","last_clean":"2024-09-11T08:16:52.738553+0000","last_became_active":"2024-09-11T08:15:56.523728+0000","last_became_peered":"2024-09-11T08:15:56.523728+0000","last_unstale":"2024-09-11T08:16:52.738553+0000","last_undegraded":"2024-09-11T08:16:52.738553+0000","last_fullsized":"2024-09-11T08:16:52.738553+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:38.643651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:11:34.159053+0000","last_clean_scrub_stamp":"2024-09-11T08:16:38.643651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:58.036696+0000","scrub_duration":0.002366469,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":198,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.742015+0000","last_change":"2024-09-11T08:15:44.448007+0000","last_active":"2024-09-11T08:16:52.742015+0000","last_peered":"2024-09-11T08:16:52.742015+0000","last_clean":"2024-09-11T08:16:52.742015+0000","last_became_active":"2024-09-11T08:07:36.633368+0000","last_became_peered":"2024-09-11T08:07:36.633368+0000","last_unstale":"2024-09-11T08:16:52.742015+0000","last_undegraded":"2024-09-11T08:16:52.742015+0000","last_fullsized":"2024-09-11T08:16:52.742015+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:44.447924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:34.378734+0000","last_clean_scrub_stamp":"2024-09-11T08:15:44.447924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029812609999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":208,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.741414+0000","last_change":"2024-09-11T08:15:41.389724+0000","last_active":"2024-09-11T08:16:52.741414+0000","last_peered":"2024-09-11T08:16:52.741414+0000","last_clean":"2024-09-11T08:16:52.741414+0000","last_became_active":"2024-09-11T08:07:32.045728+0000","last_became_peered":"2024-09-11T08:07:32.045728+0000","last_unstale":"2024-09-11T08:16:52.741414+0000","last_undegraded":"2024-09-11T08:16:52.741414+0000","last_fullsized":"2024-09-11T08:16:52.741414+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'4","last_scrub_stamp":"2024-09-11T08:15:41.389510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:15:41.389510+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:10.070852+0000","scrub_duration":0.0069641269999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":214,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.755798+0000","last_change":"2024-09-11T08:15:39.743690+0000","last_active":"2024-09-11T08:16:52.755798+0000","last_peered":"2024-09-11T08:16:52.755798+0000","last_clean":"2024-09-11T08:16:52.755798+0000","last_became_active":"2024-09-11T08:07:30.049409+0000","last_became_peered":"2024-09-11T08:07:30.049409+0000","last_unstale":"2024-09-11T08:16:52.755798+0000","last_undegraded":"2024-09-11T08:16:52.755798+0000","last_fullsized":"2024-09-11T08:16:52.755798+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:39.743604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:39.743604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:08.408228+0000","scrub_duration":0.0089953759999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":256,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.742237+0000","last_change":"2024-09-11T08:16:51.403463+0000","last_active":"2024-09-11T08:16:52.742237+0000","last_peered":"2024-09-11T08:16:52.742237+0000","last_clean":"2024-09-11T08:16:52.742237+0000","last_became_active":"2024-09-11T08:15:56.528607+0000","last_became_peered":"2024-09-11T08:15:56.528607+0000","last_unstale":"2024-09-11T08:16:52.742237+0000","last_undegraded":"2024-09-11T08:16:52.742237+0000","last_fullsized":"2024-09-11T08:16:52.742237+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:51.403370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:16:51.403370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:53.355166+0000","scrub_duration":0.002498068,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":26,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.738595+0000","last_change":"2024-09-11T08:15:56.522225+0000","last_active":"2024-09-11T08:16:52.738595+0000","last_peered":"2024-09-11T08:16:52.738595+0000","last_clean":"2024-09-11T08:16:52.738595+0000","last_became_active":"2024-09-11T08:15:56.521488+0000","last_became_peered":"2024-09-11T08:15:56.521488+0000","last_unstale":"2024-09-11T08:16:52.738595+0000","last_undegraded":"2024-09-11T08:16:52.738595+0000","last_fullsized":"2024-09-11T08:16:52.738595+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:35.183289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:15:35.183289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.007179726,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":218,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.742099+0000","last_change":"2024-09-11T08:16:50.585648+0000","last_active":"2024-09-11T08:16:52.742099+0000","last_peered":"2024-09-11T08:16:52.742099+0000","last_clean":"2024-09-11T08:16:52.742099+0000","last_became_active":"2024-09-11T08:07:36.071468+0000","last_became_peered":"2024-09-11T08:07:36.071468+0000","last_unstale":"2024-09-11T08:16:52.742099+0000","last_undegraded":"2024-09-11T08:16:52.742099+0000","last_fullsized":"2024-09-11T08:16:52.742099+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:50.585566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:50.585566+0000","last_clean_scrub_stamp":"2024-09-11T08:16:50.585566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:20.341597+0000","scrub_duration":0.0073189980000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":214,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.741625+0000","last_change":"2024-09-11T08:15:47.493538+0000","last_active":"2024-09-11T08:16:52.741625+0000","last_peered":"2024-09-11T08:16:52.741625+0000","last_clean":"2024-09-11T08:16:52.741625+0000","last_became_active":"2024-09-11T08:07:30.046702+0000","last_became_peered":"2024-09-11T08:07:30.046702+0000","last_unstale":"2024-09-11T08:16:52.741625+0000","last_undegraded":"2024-09-11T08:16:52.741625+0000","last_fullsized":"2024-09-11T08:16:52.741625+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:47.493466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:47.493466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.071745623999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":240,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.742614+0000","last_change":"2024-09-11T08:15:55.255237+0000","last_active":"2024-09-11T08:16:52.742614+0000","last_peered":"2024-09-11T08:16:52.742614+0000","last_clean":"2024-09-11T08:16:52.742614+0000","last_became_active":"2024-09-11T08:07:32.054656+0000","last_became_peered":"2024-09-11T08:07:32.054656+0000","last_unstale":"2024-09-11T08:16:52.742614+0000","last_undegraded":"2024-09-11T08:16:52.742614+0000","last_fullsized":"2024-09-11T08:16:52.742614+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-11T08:15:55.255152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:15:55.255152+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:09.255947+0000","scrub_duration":0.0070337539999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":54,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T08:16:52.742315+0000","last_change":"2024-09-11T08:16:46.497719+0000","last_active":"2024-09-11T08:16:52.742315+0000","last_peered":"2024-09-11T08:16:52.742315+0000","last_clean":"2024-09-11T08:16:52.742315+0000","last_became_active":"2024-09-11T08:15:08.769133+0000","last_became_peered":"2024-09-11T08:15:08.769133+0000","last_unstale":"2024-09-11T08:16:52.742315+0000","last_undegraded":"2024-09-11T08:16:52.742315+0000","last_fullsized":"2024-09-11T08:16:52.742315+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:46.497639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:16:46.497639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:14.479718+0000","scrub_duration":0.0048914830000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":2680487960,"num_objects":40902,"num_object_clones":0,"num_object_copies":122706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40902,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":216758,"num_write_kb":8245057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17752,"num_bytes_recovered":1163329560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8421916672,"data_stored":8421900384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85237,"ondisk_log_size":85237,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80,"ondisk_log_size":80,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":139,"num_bytes_recovered":2888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3425,"internal_metadata":0},"log_size":455,"ondisk_log_size":455,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":164,"num_read_kb":159,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":168,"seq":721554505761,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1077988,"kb_used_data":747996,"kb_used_omap":7,"kb_used_meta":329976,"kb_avail":92667164,"statfs":{"total":95995035648,"available":94891175936,"internally_reserved":0,"allocated":765947904,"data_stored":765523774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8027,"internal_metadata":337895589},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:16:26 2024","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":0.89800000000000002,"15min":0.84999999999999998},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":4.673,"5min":4.673,"15min":4.673},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.97699999999999998,"5min":0.99099999999999999,"15min":0.99299999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":3.2559999999999998,"5min":3.2559999999999998,"15min":3.2559999999999998},"last":5.2610000000000001}]},{"osd":1,"last update":"Wed Sep 11 08:16:26 2024","interfaces":[{"interface":"back","average":{"1min":4.4729999999999999,"5min":2.4510000000000001,"15min":2.1139999999999999},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":42.082999999999998,"5min":42.082999999999998,"15min":42.082999999999998},"last":0.40600000000000003},{"interface":"front","average":{"1min":3.5390000000000001,"5min":2.3439999999999999,"15min":2.145},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":42.151000000000003,"5min":42.151000000000003,"15min":42.151000000000003},"last":0.30599999999999999}]},{"osd":2,"last update":"Wed Sep 11 08:16:26 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.97099999999999997,"15min":1.018},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":2.1600000000000001,"5min":3.9630000000000001,"15min":3.9630000000000001},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.73299999999999998,"15min":0.72499999999999998},"min":{"1min":0.28699999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":2.3119999999999998,"5min":2.601,"15min":2.601},"last":2.1509999999999998}]},{"osd":3,"last update":"Wed Sep 11 08:16:26 2024","interfaces":[{"interface":"back","average":{"1min":2.427,"5min":1.343,"15min":1.163},"min":{"1min":0.30499999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":21.460000000000001,"5min":21.460000000000001,"15min":21.460000000000001},"last":0.42699999999999999},{"interface":"front","average":{"1min":2.5169999999999999,"5min":1.643,"15min":1.4970000000000001},"min":{"1min":0.28999999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":21.440000000000001,"5min":21.440000000000001,"15min":21.440000000000001},"last":0.27200000000000002}]},{"osd":4,"last update":"Wed Sep 11 08:16:26 2024","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":0.629,"15min":0.56000000000000005},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":4.141,"5min":4.141,"15min":4.141},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.73799999999999999,"15min":0.748},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":3.157,"5min":3.157,"15min":3.157},"last":0.34300000000000003}]},{"osd":5,"last update":"Wed Sep 11 08:16:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":0.88900000000000001,"15min":0.82899999999999996},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":4.1379999999999999,"5min":4.1379999999999999,"15min":4.1379999999999999},"last":0.36199999999999999},{"interface":"front","average":{"1min":1.1120000000000001,"5min":0.82199999999999995,"15min":0.77400000000000002},"min":{"1min":0.41899999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":3.907,"5min":3.907,"15min":3.907},"last":0.39000000000000001}]},{"osd":6,"last update":"Wed Sep 11 08:16:26 2024","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":0.83299999999999996,"15min":0.80600000000000005},"min":{"1min":0.32000000000000001,"5min":0.312,"15min":0.312},"max":{"1min":6.0609999999999999,"5min":6.0609999999999999,"15min":6.0609999999999999},"last":0.37},{"interface":"front","average":{"1min":1.0009999999999999,"5min":0.83199999999999996,"15min":0.80400000000000005},"min":{"1min":0.38800000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":3.6829999999999998,"5min":3.6829999999999998,"15min":3.6829999999999998},"last":0.38}]}]},{"osd":6,"up_from":45,"seq":193273528444,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1300564,"kb_used_data":1003212,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":92444588,"statfs":{"total":95995035648,"available":94663258112,"internally_reserved":0,"allocated":1027289088,"data_stored":1026881165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:16:10 2024","interfaces":[{"interface":"back","average":{"1min":1.53,"5min":2.1360000000000001,"15min":1.0600000000000001},"min":{"1min":0.20899999999999999,"5min":0.20399999999999999,"15min":0.185},"max":{"1min":7.5810000000000004,"5min":45.066000000000003,"15min":45.066000000000003},"last":1.407},{"interface":"front","average":{"1min":1.5189999999999999,"5min":1.9510000000000001,"15min":1.026},"min":{"1min":0.224,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":4.1379999999999999,"5min":43.899000000000001,"15min":43.899000000000001},"last":1.6100000000000001}]},{"osd":1,"last update":"Wed Sep 11 08:16:10 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.7010000000000001,"15min":0.89500000000000002},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":3.931,"5min":44.658000000000001,"15min":44.658000000000001},"last":0.97899999999999998},{"interface":"front","average":{"1min":1.506,"5min":1.827,"15min":0.95799999999999996},"min":{"1min":0.32500000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":6.4870000000000001,"5min":44.694000000000003,"15min":44.694000000000003},"last":0.75800000000000001}]},{"osd":2,"last update":"Wed Sep 11 08:16:10 2024","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.0169999999999999,"15min":0.70099999999999996},"min":{"1min":0.28699999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":5.9770000000000003,"5min":5.9770000000000003,"15min":5.9770000000000003},"last":0.26700000000000002},{"interface":"front","average":{"1min":1.9790000000000001,"5min":1.45,"15min":0.84199999999999997},"min":{"1min":0.29299999999999998,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":7.4809999999999999,"5min":13.346,"15min":13.346},"last":3.1019999999999999}]},{"osd":3,"last update":"Wed Sep 11 08:16:10 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.7509999999999999,"15min":0.996},"min":{"1min":0.23599999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":3.3980000000000001,"5min":36.061,"15min":36.061},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.90300000000000002,"5min":1.718,"15min":0.97699999999999998},"min":{"1min":0.30599999999999999,"5min":0.28399999999999997,"15min":0.26800000000000002},"max":{"1min":2.6749999999999998,"5min":36.093000000000004,"15min":36.093000000000004},"last":0.29799999999999999}]},{"osd":4,"last update":"Wed Sep 11 08:16:10 2024","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":0.57799999999999996,"15min":0.45700000000000002},"min":{"1min":0.23699999999999999,"5min":0.219,"15min":0.20200000000000001},"max":{"1min":4.1130000000000004,"5min":4.1130000000000004,"15min":4.1130000000000004},"last":0.73299999999999998},{"interface":"front","average":{"1min":1.1579999999999999,"5min":0.64700000000000002,"15min":0.499},"min":{"1min":0.27100000000000002,"5min":0.251,"15min":0.24099999999999999},"max":{"1min":4.468,"5min":4.468,"15min":4.468},"last":0.77400000000000002}]},{"osd":5,"last update":"Wed Sep 11 08:16:10 2024","interfaces":[{"interface":"back","average":{"1min":1.514,"5min":0.76400000000000001,"15min":0.57999999999999996},"min":{"1min":0.37,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":7.4459999999999997,"5min":7.4459999999999997,"15min":7.4459999999999997},"last":0.65200000000000002},{"interface":"front","average":{"1min":1.246,"5min":0.747,"15min":0.53400000000000003},"min":{"1min":0.28399999999999997,"5min":0.22800000000000001,"15min":0.22600000000000001},"max":{"1min":3.4329999999999998,"5min":3.4329999999999998,"15min":3.4329999999999998},"last":0.57299999999999995}]},{"osd":7,"last update":"Wed Sep 11 08:16:25 2024","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":1.159,"15min":1.1499999999999999},"min":{"1min":0.28899999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":8.4580000000000002,"5min":8.4580000000000002,"15min":8.4580000000000002},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.98699999999999999,"5min":0.91900000000000004,"15min":0.90700000000000003},"min":{"1min":0.34000000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":3.593,"5min":3.593,"15min":3.593},"last":0.31900000000000001}]}]},{"osd":5,"up_from":38,"seq":163208757376,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1900180,"kb_used_data":1594508,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":91844972,"statfs":{"total":95995035648,"available":94049251328,"internally_reserved":0,"allocated":1632776192,"data_stored":1632367286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:16:52 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":1.1799999999999999,"15min":0.81299999999999994},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":2.0739999999999998,"5min":7.9470000000000001,"15min":7.9470000000000001},"last":0.35499999999999998},{"interface":"front","average":{"1min":2.016,"5min":1.4199999999999999,"15min":0.878},"min":{"1min":0.246,"5min":0.23000000000000001,"15min":0.20399999999999999},"max":{"1min":13.077,"5min":13.077,"15min":13.077},"last":0.47299999999999998}]},{"osd":1,"last update":"Wed Sep 11 08:16:52 2024","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":1.431,"15min":1.2569999999999999},"min":{"1min":0.27700000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":3.4809999999999999,"5min":26.709,"15min":26.709},"last":0.29299999999999998},{"interface":"front","average":{"1min":1.0680000000000001,"5min":1.419,"15min":0.89600000000000002},"min":{"1min":0.248,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.8259999999999996,"5min":26.754000000000001,"15min":26.754000000000001},"last":0.309}]},{"osd":2,"last update":"Wed Sep 11 08:16:52 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":1.1919999999999999,"15min":1.1359999999999999},"min":{"1min":0.27500000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":2.2970000000000002,"5min":10.891,"15min":16.018000000000001},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.96499999999999997,"15min":0.70699999999999996},"min":{"1min":0.28499999999999998,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":2.903,"5min":7.5250000000000004,"15min":7.5250000000000004},"last":0.36599999999999999}]},{"osd":3,"last update":"Wed Sep 11 08:16:52 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":1.292,"15min":0.86699999999999999},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":2.0249999999999999,"5min":24.931000000000001,"15min":24.931000000000001},"last":0.34100000000000003},{"interface":"front","average":{"1min":1.0880000000000001,"5min":1.4830000000000001,"15min":0.91700000000000004},"min":{"1min":0.27700000000000002,"5min":0.255,"15min":0.251},"max":{"1min":3.9449999999999998,"5min":24.861999999999998,"15min":24.861999999999998},"last":0.52300000000000002}]},{"osd":4,"last update":"Wed Sep 11 08:16:52 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.64500000000000002,"15min":0.499},"min":{"1min":0.25800000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.0409999999999999,"5min":3.5979999999999999,"15min":3.5979999999999999},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.60099999999999998,"15min":0.48799999999999999},"min":{"1min":0.26900000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":4.9210000000000003,"5min":4.9210000000000003,"15min":4.9210000000000003},"last":0.26900000000000002}]},{"osd":6,"last update":"Wed Sep 11 08:16:12 2024","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":0.54300000000000004,"15min":0.47299999999999998},"min":{"1min":0.30399999999999999,"5min":0.252,"15min":0.23400000000000001},"max":{"1min":3.1869999999999998,"5min":3.1869999999999998,"15min":3.1869999999999998},"last":0.499},{"interface":"front","average":{"1min":1.0129999999999999,"5min":0.64500000000000002,"15min":0.53100000000000003},"min":{"1min":0.35199999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":3.5169999999999999,"5min":3.9489999999999998,"15min":3.9489999999999998},"last":0.375}]},{"osd":7,"last update":"Wed Sep 11 08:16:28 2024","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.85299999999999998,"15min":0.83799999999999997},"min":{"1min":0.38500000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":3.5459999999999998,"5min":3.5459999999999998,"15min":3.5459999999999998},"last":0.55200000000000005},{"interface":"front","average":{"1min":1.0680000000000001,"5min":0.82399999999999995,"15min":0.78400000000000003},"min":{"1min":0.34899999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":3.1019999999999999,"5min":3.1019999999999999,"15min":3.1019999999999999},"last":0.54500000000000004}]}]},{"osd":4,"up_from":31,"seq":133143986310,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1058404,"kb_used_data":752732,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":92686748,"statfs":{"total":95995035648,"available":94911229952,"internally_reserved":0,"allocated":770797568,"data_stored":770373568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:16:20 2024","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":1.0600000000000001,"15min":1.0660000000000001},"min":{"1min":0.23300000000000001,"5min":0.19,"15min":0.19},"max":{"1min":5.306,"5min":6.1159999999999997,"15min":48.728999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.85099999999999998,"15min":0.93300000000000005},"min":{"1min":0.22900000000000001,"5min":0.221,"15min":0.221},"max":{"1min":2.8650000000000002,"5min":5.5599999999999996,"15min":49.168999999999997},"last":0.41399999999999998}]},{"osd":1,"last update":"Wed Sep 11 08:16:20 2024","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.0049999999999999,"15min":0.753},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":3.5470000000000002,"5min":4.7320000000000002,"15min":13.207000000000001},"last":3.028},{"interface":"front","average":{"1min":1.3560000000000001,"5min":0.98599999999999999,"15min":0.72199999999999998},"min":{"1min":0.24299999999999999,"5min":0.22900000000000001,"15min":0.22700000000000001},"max":{"1min":5.8869999999999996,"5min":5.8869999999999996,"15min":5.8869999999999996},"last":0.29899999999999999}]},{"osd":2,"last update":"Wed Sep 11 08:16:20 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.0549999999999999,"15min":0.74199999999999999},"min":{"1min":0.245,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":3.3599999999999999,"5min":7.976,"15min":7.976},"last":0.255},{"interface":"front","average":{"1min":1.085,"5min":0.871,"15min":0.71499999999999997},"min":{"1min":0.32700000000000001,"5min":0.248,"15min":0.248},"max":{"1min":3.1389999999999998,"5min":3.6989999999999998,"15min":5.5670000000000002},"last":0.34399999999999997}]},{"osd":3,"last update":"Wed Sep 11 08:16:20 2024","interfaces":[{"interface":"back","average":{"1min":1.986,"5min":1.3129999999999999,"15min":1.0349999999999999},"min":{"1min":0.27600000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":19.686,"5min":19.686,"15min":26.884},"last":0.313},{"interface":"front","average":{"1min":1.8660000000000001,"5min":1.4550000000000001,"15min":1.0549999999999999},"min":{"1min":0.26600000000000001,"5min":0.246,"15min":0.246},"max":{"1min":19.882000000000001,"5min":19.882000000000001,"15min":26.783000000000001},"last":2.0960000000000001}]},{"osd":5,"last update":"Wed Sep 11 08:16:49 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.56999999999999995,"15min":0.44800000000000001},"min":{"1min":0.23999999999999999,"5min":0.23599999999999999,"15min":0.23499999999999999},"max":{"1min":2.7229999999999999,"5min":2.7229999999999999,"15min":2.7229999999999999},"last":0.505},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.53900000000000003,"15min":0.44800000000000001},"min":{"1min":0.28000000000000003,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":2.8090000000000002,"5min":2.8090000000000002,"15min":2.8090000000000002},"last":0.39300000000000002}]},{"osd":6,"last update":"Wed Sep 11 08:16:14 2024","interfaces":[{"interface":"back","average":{"1min":3.048,"5min":1.004,"15min":0.59599999999999997},"min":{"1min":0.28599999999999998,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":48.808,"5min":48.808,"15min":48.808},"last":0.59299999999999997},{"interface":"front","average":{"1min":2.988,"5min":1.032,"15min":0.64800000000000002},"min":{"1min":0.27800000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":48.746000000000002,"5min":48.746000000000002,"15min":48.746000000000002},"last":0.42199999999999999}]},{"osd":7,"last update":"Wed Sep 11 08:16:30 2024","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.56499999999999995,"15min":0.50700000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":3.512,"5min":3.512,"15min":3.512},"last":0.372},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.69699999999999995,"15min":0.67400000000000004},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":3.4950000000000001,"5min":3.4950000000000001,"15min":3.4950000000000001},"last":1.8819999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182538,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1299248,"kb_used_data":997672,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":92445904,"statfs":{"total":95995035648,"available":94664605696,"internally_reserved":0,"allocated":1021616128,"data_stored":1021179311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:16:04 2024","interfaces":[{"interface":"back","average":{"1min":4.2320000000000002,"5min":2.363,"15min":1.099},"min":{"1min":0.20200000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":60.469999999999999,"5min":60.469999999999999,"15min":60.469999999999999},"last":0.182},{"interface":"front","average":{"1min":4.4939999999999998,"5min":2.613,"15min":1.1990000000000001},"min":{"1min":0.245,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":60.488999999999997,"5min":60.488999999999997,"15min":60.488999999999997},"last":0.28999999999999998}]},{"osd":1,"last update":"Wed Sep 11 08:16:04 2024","interfaces":[{"interface":"back","average":{"1min":4.0620000000000003,"5min":2.2679999999999998,"15min":1.0860000000000001},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.20899999999999999},"max":{"1min":60.578000000000003,"5min":60.578000000000003,"15min":60.578000000000003},"last":0.26300000000000001},{"interface":"front","average":{"1min":4.5910000000000002,"5min":2.411,"15min":1.1830000000000001},"min":{"1min":0.23400000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":63.674999999999997,"5min":63.674999999999997,"15min":63.674999999999997},"last":0.57399999999999995}]},{"osd":2,"last update":"Wed Sep 11 08:16:04 2024","interfaces":[{"interface":"back","average":{"1min":4.6840000000000002,"5min":2.6139999999999999,"15min":1.222},"min":{"1min":0.22,"5min":0.185,"15min":0.185},"max":{"1min":60.645000000000003,"5min":60.645000000000003,"15min":60.645000000000003},"last":0.63500000000000001},{"interface":"front","average":{"1min":4.6740000000000004,"5min":2.3740000000000001,"15min":1.1379999999999999},"min":{"1min":0.38600000000000001,"5min":0.20699999999999999,"15min":0.20599999999999999},"max":{"1min":60.723999999999997,"5min":60.723999999999997,"15min":60.723999999999997},"last":0.35399999999999998}]},{"osd":4,"last update":"Wed Sep 11 08:16:23 2024","interfaces":[{"interface":"back","average":{"1min":4.0970000000000004,"5min":1.9339999999999999,"15min":1.0740000000000001},"min":{"1min":0.28100000000000003,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":61.290999999999997,"5min":61.290999999999997,"15min":61.290999999999997},"last":0.92000000000000004},{"interface":"front","average":{"1min":3.8740000000000001,"5min":1.831,"15min":1.024},"min":{"1min":0.28899999999999998,"5min":0.27500000000000002,"15min":0.27200000000000002},"max":{"1min":60.856000000000002,"5min":60.856000000000002,"15min":60.856000000000002},"last":0.311}]},{"osd":5,"last update":"Wed Sep 11 08:16:53 2024","interfaces":[{"interface":"back","average":{"1min":1.6910000000000001,"5min":2.3199999999999998,"15min":1.236},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.26500000000000001},"max":{"1min":11.755000000000001,"5min":60.914999999999999,"15min":60.914999999999999},"last":1.232},{"interface":"front","average":{"1min":1.6639999999999999,"5min":2.4390000000000001,"15min":1.3169999999999999},"min":{"1min":0.35699999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":12.26,"5min":60.787999999999997,"15min":60.787999999999997},"last":0.434}]},{"osd":6,"last update":"Wed Sep 11 08:16:10 2024","interfaces":[{"interface":"back","average":{"1min":4.3019999999999996,"5min":2.4329999999999998,"15min":1.286},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":60.817,"5min":60.817,"15min":60.817},"last":0.98599999999999999},{"interface":"front","average":{"1min":4.4530000000000003,"5min":2.5169999999999999,"15min":1.371},"min":{"1min":0.505,"5min":0.307,"15min":0.307},"max":{"1min":61.264000000000003,"5min":61.264000000000003,"15min":61.264000000000003},"last":0.60799999999999998}]},{"osd":7,"last update":"Wed Sep 11 08:16:26 2024","interfaces":[{"interface":"back","average":{"1min":3.8570000000000002,"5min":3.2850000000000001,"15min":3.1890000000000001},"min":{"1min":0.39500000000000002,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":60.847000000000001,"5min":60.847000000000001,"15min":60.847000000000001},"last":2.1440000000000001},{"interface":"front","average":{"1min":3.8769999999999998,"5min":3.2330000000000001,"15min":3.1259999999999999},"min":{"1min":0.442,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":60.883000000000003,"5min":60.883000000000003,"15min":60.883000000000003},"last":1.4890000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411469,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1319188,"kb_used_data":1013516,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":92425964,"statfs":{"total":95995035648,"available":94644187136,"internally_reserved":0,"allocated":1037840384,"data_stored":1037431390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:16:43 2024","interfaces":[{"interface":"back","average":{"1min":1.867,"5min":2.2679999999999998,"15min":1.1699999999999999},"min":{"1min":0.28100000000000003,"5min":0.21299999999999999,"15min":0.17199999999999999},"max":{"1min":8.0920000000000005,"5min":43.539999999999999,"15min":43.539999999999999},"last":1.833},{"interface":"front","average":{"1min":1.401,"5min":2.4199999999999999,"15min":1.2090000000000001},"min":{"1min":0.19600000000000001,"5min":0.185,"15min":0.185},"max":{"1min":5.9790000000000001,"5min":41.962000000000003,"15min":41.962000000000003},"last":1.845}]},{"osd":1,"last update":"Wed Sep 11 08:16:43 2024","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":1.659,"15min":0.95499999999999996},"min":{"1min":0.28699999999999998,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":5.1539999999999999,"5min":13.138999999999999,"15min":13.138999999999999},"last":2.6629999999999998},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.8360000000000001,"15min":1.0089999999999999},"min":{"1min":0.252,"5min":0.252,"15min":0.20200000000000001},"max":{"1min":5.0220000000000002,"5min":10.167999999999999,"15min":10.167999999999999},"last":1.903}]},{"osd":3,"last update":"Wed Sep 11 08:16:05 2024","interfaces":[{"interface":"back","average":{"1min":2.5070000000000001,"5min":1.97,"15min":0.998},"min":{"1min":0.21299999999999999,"5min":0.188,"15min":0.188},"max":{"1min":8.1989999999999998,"5min":8.1989999999999998,"15min":8.1989999999999998},"last":2.6339999999999999},{"interface":"front","average":{"1min":2.4289999999999998,"5min":1.879,"15min":0.97299999999999998},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.22500000000000001},"max":{"1min":8.0640000000000001,"5min":8.3219999999999992,"15min":8.3219999999999992},"last":1.8799999999999999}]},{"osd":4,"last update":"Wed Sep 11 08:16:27 2024","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":1.1839999999999999,"15min":0.81999999999999995},"min":{"1min":0.29799999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":3.2999999999999998,"5min":7.2480000000000002,"15min":7.806},"last":0.44700000000000001},{"interface":"front","average":{"1min":1.1850000000000001,"5min":1.1519999999999999,"15min":0.871},"min":{"1min":0.309,"5min":0.252,"15min":0.252},"max":{"1min":3.855,"5min":5.4740000000000002,"15min":12.129},"last":1.9399999999999999}]},{"osd":5,"last update":"Wed Sep 11 08:16:58 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.145,"15min":0.876},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":3.1139999999999999,"5min":5.4539999999999997,"15min":6.7709999999999999},"last":2.02},{"interface":"front","average":{"1min":1.292,"5min":1.3089999999999999,"15min":0.89200000000000002},"min":{"1min":0.34200000000000003,"5min":0.30399999999999999,"15min":0.28399999999999997},"max":{"1min":4.2249999999999996,"5min":12.869,"15min":12.869},"last":2.0019999999999998}]},{"osd":6,"last update":"Wed Sep 11 08:16:10 2024","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.2090000000000001,"15min":0.81599999999999995},"min":{"1min":0.44700000000000001,"5min":0.32600000000000001,"15min":0.26100000000000001},"max":{"1min":4.2539999999999996,"5min":7.2249999999999996,"15min":7.2249999999999996},"last":1.972},{"interface":"front","average":{"1min":1.819,"5min":1.3839999999999999,"15min":0.89700000000000002},"min":{"1min":0.35499999999999998,"5min":0.33200000000000002,"15min":0.32100000000000001},"max":{"1min":5.4939999999999998,"5min":6.71,"15min":6.71},"last":2.0499999999999998}]},{"osd":7,"last update":"Wed Sep 11 08:16:27 2024","interfaces":[{"interface":"back","average":{"1min":1.468,"5min":1.4039999999999999,"15min":1.3939999999999999},"min":{"1min":0.375,"5min":0.37,"15min":0.37},"max":{"1min":4.0780000000000003,"5min":4.9569999999999999,"15min":4.9569999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.512,"15min":1.528},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":3.8300000000000001,"5min":5.3710000000000004,"15min":5.3710000000000004},"last":2.0750000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574995,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1486564,"kb_used_data":1180892,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":92258588,"statfs":{"total":95995035648,"available":94472794112,"internally_reserved":0,"allocated":1209233408,"data_stored":1208808664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:16:17 2024","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.3779999999999999,"15min":0.90800000000000003},"min":{"1min":0.24099999999999999,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":4.2199999999999998,"5min":8.1509999999999998,"15min":32.478000000000002},"last":1.627},{"interface":"front","average":{"1min":1.9970000000000001,"5min":1.9419999999999999,"15min":1.177},"min":{"1min":0.28000000000000003,"5min":0.186,"15min":0.186},"max":{"1min":5.4210000000000003,"5min":10.247,"15min":35.459000000000003},"last":15.351000000000001}]},{"osd":2,"last update":"Wed Sep 11 08:16:52 2024","interfaces":[{"interface":"back","average":{"1min":1.5429999999999999,"5min":1.4019999999999999,"15min":0.876},"min":{"1min":0.223,"5min":0.216,"15min":0.21299999999999999},"max":{"1min":4.048,"5min":7.9699999999999998,"15min":8.3819999999999997},"last":1.6599999999999999},{"interface":"front","average":{"1min":2.121,"5min":1.6080000000000001,"15min":1.0389999999999999},"min":{"1min":0.20399999999999999,"5min":0.192,"15min":0.16300000000000001},"max":{"1min":5.8490000000000002,"5min":6.6349999999999998,"15min":10.276},"last":1.458}]},{"osd":3,"last update":"Wed Sep 11 08:16:01 2024","interfaces":[{"interface":"back","average":{"1min":1.835,"5min":2.3479999999999999,"15min":1.339},"min":{"1min":0.38600000000000001,"5min":0.248,"15min":0.246},"max":{"1min":5.1689999999999996,"5min":16.783999999999999,"15min":32.759},"last":8.4149999999999991},{"interface":"front","average":{"1min":1.0469999999999999,"5min":2.008,"15min":1.175},"min":{"1min":0.27000000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":4.2489999999999997,"5min":14.973000000000001,"15min":32.444000000000003},"last":0.45100000000000001}]},{"osd":4,"last update":"Wed Sep 11 08:16:27 2024","interfaces":[{"interface":"back","average":{"1min":2.2669999999999999,"5min":1.333,"15min":0.93000000000000005},"min":{"1min":0.29999999999999999,"5min":0.252,"15min":0.252},"max":{"1min":5.8970000000000002,"5min":8.5640000000000001,"15min":9.0139999999999993},"last":1.4990000000000001},{"interface":"front","average":{"1min":1.1850000000000001,"5min":1.129,"15min":0.83999999999999997},"min":{"1min":0.38100000000000001,"5min":0.248,"15min":0.248},"max":{"1min":4.2949999999999999,"5min":8.0609999999999999,"15min":8.2360000000000007},"last":0.48099999999999998}]},{"osd":5,"last update":"Wed Sep 11 08:16:52 2024","interfaces":[{"interface":"back","average":{"1min":2.0870000000000002,"5min":1.4099999999999999,"15min":1.0129999999999999},"min":{"1min":0.29499999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":5.9299999999999997,"5min":7.9420000000000002,"15min":9.0470000000000006},"last":1.532},{"interface":"front","average":{"1min":2.4550000000000001,"5min":1.6180000000000001,"15min":1.0820000000000001},"min":{"1min":0.39600000000000002,"5min":0.28799999999999998,"15min":0.28100000000000003},"max":{"1min":4.7530000000000001,"5min":7.0330000000000004,"15min":11.967000000000001},"last":5.3780000000000001}]},{"osd":6,"last update":"Wed Sep 11 08:16:03 2024","interfaces":[{"interface":"back","average":{"1min":7.8250000000000002,"5min":2.7599999999999998,"15min":1.3859999999999999},"min":{"1min":0.32600000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":124.31999999999999,"5min":124.31999999999999,"15min":124.31999999999999},"last":5.3479999999999999},{"interface":"front","average":{"1min":7.6020000000000003,"5min":2.6779999999999999,"15min":1.341},"min":{"1min":0.39200000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":124.958,"5min":124.958,"15min":124.958},"last":0.504}]},{"osd":7,"last update":"Wed Sep 11 08:16:27 2024","interfaces":[{"interface":"back","average":{"1min":2.5169999999999999,"5min":1.4510000000000001,"15min":1.2729999999999999},"min":{"1min":0.34100000000000003,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":5.9580000000000002,"5min":5.9580000000000002,"15min":5.9580000000000002},"last":1.556},{"interface":"front","average":{"1min":2.0019999999999998,"5min":1.5840000000000001,"15min":1.514},"min":{"1min":0.5,"5min":0.377,"15min":0.377},"max":{"1min":4.452,"5min":4.593,"15min":4.593},"last":5.2969999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738519,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1327612,"kb_used_data":1017780,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":92417540,"statfs":{"total":95995035648,"available":94635560960,"internally_reserved":0,"allocated":1042206720,"data_stored":1041759294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 11 08:16:20 2024","interfaces":[{"interface":"back","average":{"1min":1.4770000000000001,"5min":1.5609999999999999,"15min":0.86299999999999999},"min":{"1min":0.19600000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":5.992,"5min":7.2160000000000002,"15min":7.2160000000000002},"last":10.105},{"interface":"front","average":{"1min":2.9409999999999998,"5min":2.117,"15min":1.0389999999999999},"min":{"1min":0.29899999999999999,"5min":0.153,"15min":0.153},"max":{"1min":8.3070000000000004,"5min":9.5679999999999996,"15min":9.5679999999999996},"last":1.169}]},{"osd":2,"last update":"Wed Sep 11 08:16:44 2024","interfaces":[{"interface":"back","average":{"1min":2.1709999999999998,"5min":1.5640000000000001,"15min":0.92200000000000004},"min":{"1min":0.23899999999999999,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":7.4340000000000002,"5min":10.351000000000001,"15min":10.351000000000001},"last":10.025},{"interface":"front","average":{"1min":1.827,"5min":1.639,"15min":0.98299999999999998},"min":{"1min":0.28899999999999998,"5min":0.186,"15min":0.186},"max":{"1min":6.2690000000000001,"5min":7.3090000000000002,"15min":7.3090000000000002},"last":10.074999999999999}]},{"osd":3,"last update":"Wed Sep 11 08:16:05 2024","interfaces":[{"interface":"back","average":{"1min":1.546,"5min":2.2549999999999999,"15min":1.2130000000000001},"min":{"1min":0.29299999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":4.5679999999999996,"5min":35.491,"15min":35.491},"last":10.129},{"interface":"front","average":{"1min":2.2240000000000002,"5min":2.6150000000000002,"15min":1.246},"min":{"1min":0.32900000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.7919999999999998,"5min":37.563000000000002,"15min":37.563000000000002},"last":1.286}]},{"osd":4,"last update":"Wed Sep 11 08:16:25 2024","interfaces":[{"interface":"back","average":{"1min":1.6759999999999999,"5min":1.502,"15min":0.94399999999999995},"min":{"1min":0.39300000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":5.0190000000000001,"5min":9.4030000000000005,"15min":9.4030000000000005},"last":1.2350000000000001},{"interface":"front","average":{"1min":0.88900000000000001,"5min":1.2909999999999999,"15min":0.88600000000000001},"min":{"1min":0.34399999999999997,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":2.5899999999999999,"5min":6.6619999999999999,"15min":6.6619999999999999},"last":5.6399999999999997}]},{"osd":5,"last update":"Wed Sep 11 08:16:56 2024","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.6200000000000001,"15min":1.0469999999999999},"min":{"1min":0.42199999999999999,"5min":0.255,"15min":0.255},"max":{"1min":5.4240000000000004,"5min":24.914999999999999,"15min":24.914999999999999},"last":5.6100000000000003},{"interface":"front","average":{"1min":1.3480000000000001,"5min":1.9019999999999999,"15min":1.125},"min":{"1min":0.36699999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":8.2080000000000002,"5min":25.277000000000001,"15min":25.277000000000001},"last":1.196}]},{"osd":6,"last update":"Wed Sep 11 08:16:10 2024","interfaces":[{"interface":"back","average":{"1min":2.0230000000000001,"5min":2.4950000000000001,"15min":1.306},"min":{"1min":0.29599999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":5.085,"5min":78.522000000000006,"15min":78.522000000000006},"last":1.216},{"interface":"front","average":{"1min":1.514,"5min":1.986,"15min":1.145},"min":{"1min":0.29899999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":6.8520000000000003,"5min":79.808000000000007,"15min":79.808000000000007},"last":5.1589999999999998}]},{"osd":7,"last update":"Wed Sep 11 08:16:25 2024","interfaces":[{"interface":"back","average":{"1min":1.978,"5min":1.4850000000000001,"15min":1.403},"min":{"1min":0.41499999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":5.976,"5min":6.7750000000000004,"15min":6.7750000000000004},"last":5.1980000000000004},{"interface":"front","average":{"1min":1.6299999999999999,"5min":1.4099999999999999,"15min":1.373},"min":{"1min":0.438,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":6.117,"5min":6.8179999999999996,"15min":6.8179999999999996},"last":5.218}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3425,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1028456448,"data_stored":1028456448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1186725888,"data_stored":1186725888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1036718080,"data_stored":1036714008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1012269056,"data_stored":1012269056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":755695616,"data_stored":755695616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1622675456,"data_stored":1622671384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1018236928,"data_stored":1018232856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":761139200,"data_stored":761135128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T08:17:06.645 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-11T08:17:06.645 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T08:17:06.645 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-09-11T08:17:06.645 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-11T08:17:06.645 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-11T08:17:06.645 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.0 config set filestore_inject_stall 3 2024-09-11T08:17:06.736 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:06.790 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:06.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:06.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:06.892 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:17:06.920 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:06.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:06.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:17:07.022 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:17:07.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:07.054 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:07.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:17:07.155 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:07.190 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:07.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:07.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:07.292 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:07.322 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:07.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:07.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:07.424 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:07.456 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:07.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:07.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:07.558 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:07.594 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:07.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:07.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:07.696 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:07.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:07.728 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:07.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:07.829 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:17:07.864 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:07.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:07.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:17:07.937 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T08:17:07.966 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:08.027 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:08.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:08.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:08.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:07 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:17:08.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:07 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:17:08.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:07 smithi181 ceph-mon[32701]: pgmap v426: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.2 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 626 op/s 2024-09-11T08:17:08.074 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:07 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:17:08.074 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:07 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:17:08.074 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:07 smithi150 ceph-mon[29276]: pgmap v426: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.2 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 626 op/s 2024-09-11T08:17:08.074 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:07 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:17:08.074 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:07 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:17:08.074 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:07 smithi150 ceph-mon[39612]: pgmap v426: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.2 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 626 op/s 2024-09-11T08:17:08.106 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-11T08:17:08.106 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 20, 2024-09-11T08:17:08.106 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 600, 2024-09-11T08:17:08.106 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [ 2024-09-11T08:17:08.106 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.106 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:89904 7.e 7:70ba542c:::benchmark_data_smithi150_78508_object89903:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-11T08:17:08.106 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:10:53.646654+0000", 2024-09-11T08:17:08.106 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 374.45452058799998, 2024-09-11T08:17:08.106 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.087794550999999998, 2024-09-11T08:17:08.106 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:17:08.106 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:17:08.107 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:17:08.107 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:17:08.107 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:17:08.107 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 89904 2024-09-11T08:17:08.107 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.107 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:17:08.107 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.107 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:17:08.107 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.646654+0000", 2024-09-11T08:17:08.107 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:17:08.107 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.107 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.107 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:17:08.107 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.646654+0000", 2024-09-11T08:17:08.108 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:17:08.108 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.108 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.108 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:17:08.108 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.646652+0000", 2024-09-11T08:17:08.108 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:17:08.108 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.108 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.108 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:17:08.108 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.646681+0000", 2024-09-11T08:17:08.108 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.8467e-05 2024-09-11T08:17:08.108 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.108 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.109 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:17:08.109 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.646689+0000", 2024-09-11T08:17:08.109 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.799e-06 2024-09-11T08:17:08.109 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.109 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.109 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:17:08.109 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.646694+0000", 2024-09-11T08:17:08.109 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.1050000000000003e-06 2024-09-11T08:17:08.109 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.109 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.109 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:17:08.109 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.647684+0000", 2024-09-11T08:17:08.109 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0009899380000000001 2024-09-11T08:17:08.110 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.110 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.110 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:17:08.110 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.647791+0000", 2024-09-11T08:17:08.110 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00010665199999999999 2024-09-11T08:17:08.110 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.110 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.110 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 0,5", 2024-09-11T08:17:08.110 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.648274+0000", 2024-09-11T08:17:08.110 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00048331999999999997 2024-09-11T08:17:08.110 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.110 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.110 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:17:08.111 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.651957+0000", 2024-09-11T08:17:08.111 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0036826850000000002 2024-09-11T08:17:08.111 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.111 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.111 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:17:08.111 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.651969+0000", 2024-09-11T08:17:08.111 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2125e-05 2024-09-11T08:17:08.111 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.111 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.111 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:17:08.111 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.734196+0000", 2024-09-11T08:17:08.111 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.082226441999999997 2024-09-11T08:17:08.111 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.111 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.112 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:17:08.112 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.734317+0000", 2024-09-11T08:17:08.112 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012095900000000001 2024-09-11T08:17:08.112 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.112 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.112 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:17:08.112 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.734448+0000", 2024-09-11T08:17:08.112 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00013158899999999999 2024-09-11T08:17:08.112 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:08.112 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:08.112 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:08.112 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.112 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.113 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:89919 7.e 7:7341f89b:::benchmark_data_smithi150_78508_object89918:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-11T08:17:08.113 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:10:53.657447+0000", 2024-09-11T08:17:08.113 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 374.44372726900002, 2024-09-11T08:17:08.113 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.097698273000000002, 2024-09-11T08:17:08.113 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:17:08.113 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:17:08.113 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:17:08.113 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:17:08.113 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:17:08.113 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 89919 2024-09-11T08:17:08.113 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.113 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:17:08.113 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.113 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:17:08.114 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.657447+0000", 2024-09-11T08:17:08.114 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:17:08.114 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.114 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.114 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:17:08.114 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.657447+0000", 2024-09-11T08:17:08.114 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:17:08.114 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.114 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.114 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:17:08.114 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.657445+0000", 2024-09-11T08:17:08.114 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:17:08.114 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.115 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.115 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:17:08.115 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.657483+0000", 2024-09-11T08:17:08.115 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.7976999999999999e-05 2024-09-11T08:17:08.115 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.115 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.115 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:17:08.115 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.657498+0000", 2024-09-11T08:17:08.115 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4433999999999999e-05 2024-09-11T08:17:08.115 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.115 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.115 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:17:08.115 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.657504+0000", 2024-09-11T08:17:08.115 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.3149999999999997e-06 2024-09-11T08:17:08.115 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.116 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.116 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:17:08.116 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.657559+0000", 2024-09-11T08:17:08.116 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.4413999999999998e-05 2024-09-11T08:17:08.116 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.116 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.116 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:17:08.116 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.657941+0000", 2024-09-11T08:17:08.116 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00038252099999999998 2024-09-11T08:17:08.116 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.116 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.116 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 0,5", 2024-09-11T08:17:08.116 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.658503+0000", 2024-09-11T08:17:08.117 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00056179199999999998 2024-09-11T08:17:08.117 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.117 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.117 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:17:08.117 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.666225+0000", 2024-09-11T08:17:08.117 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0077220759999999996 2024-09-11T08:17:08.117 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.117 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.117 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:17:08.117 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.666593+0000", 2024-09-11T08:17:08.118 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00036758599999999999 2024-09-11T08:17:08.118 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.118 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.118 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:17:08.118 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.754441+0000", 2024-09-11T08:17:08.118 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.087848750000000003 2024-09-11T08:17:08.118 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.118 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.118 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:17:08.118 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.755013+0000", 2024-09-11T08:17:08.118 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00057142399999999996 2024-09-11T08:17:08.118 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.118 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.118 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:17:08.119 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.755145+0000", 2024-09-11T08:17:08.119 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00013243400000000001 2024-09-11T08:17:08.119 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:08.119 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:08.119 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:08.119 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.119 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.119 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:89929 7.e 7:7381e007:::benchmark_data_smithi150_78508_object89928:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-11T08:17:08.119 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:10:53.668595+0000", 2024-09-11T08:17:08.119 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 374.43257934000002, 2024-09-11T08:17:08.119 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.087111145000000001, 2024-09-11T08:17:08.119 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:17:08.119 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:17:08.119 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:17:08.120 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:17:08.120 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:17:08.120 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 89929 2024-09-11T08:17:08.121 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.121 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:17:08.121 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.121 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:17:08.121 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.668595+0000", 2024-09-11T08:17:08.121 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:17:08.121 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.121 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.121 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:17:08.121 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.668595+0000", 2024-09-11T08:17:08.121 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:17:08.121 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.122 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.122 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:17:08.122 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.668593+0000", 2024-09-11T08:17:08.122 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:17:08.122 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.122 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.122 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:17:08.122 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.668620+0000", 2024-09-11T08:17:08.122 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.7424999999999999e-05 2024-09-11T08:17:08.122 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.122 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.122 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:17:08.122 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.668639+0000", 2024-09-11T08:17:08.122 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.8552000000000001e-05 2024-09-11T08:17:08.123 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.123 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.123 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:17:08.123 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.668646+0000", 2024-09-11T08:17:08.123 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.3520000000000001e-06 2024-09-11T08:17:08.123 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.123 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.123 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:17:08.123 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.668696+0000", 2024-09-11T08:17:08.123 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.0087999999999999e-05 2024-09-11T08:17:08.123 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.123 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.123 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:17:08.123 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.668809+0000", 2024-09-11T08:17:08.124 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00011257800000000001 2024-09-11T08:17:08.124 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.124 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.124 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 0,5", 2024-09-11T08:17:08.124 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.668904+0000", 2024-09-11T08:17:08.124 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.4610000000000004e-05 2024-09-11T08:17:08.124 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.124 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.124 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:17:08.124 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.671197+0000", 2024-09-11T08:17:08.124 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0022935669999999998 2024-09-11T08:17:08.124 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.124 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.125 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:17:08.125 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.671250+0000", 2024-09-11T08:17:08.125 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.2488999999999998e-05 2024-09-11T08:17:08.125 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.125 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.125 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:17:08.125 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.755270+0000", 2024-09-11T08:17:08.125 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.084020513000000005 2024-09-11T08:17:08.125 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.125 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.125 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:17:08.125 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.755591+0000", 2024-09-11T08:17:08.125 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000320942 2024-09-11T08:17:08.126 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.126 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.126 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:17:08.126 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:10:53.755706+0000", 2024-09-11T08:17:08.126 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000114868 2024-09-11T08:17:08.126 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:08.126 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:08.126 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:08.126 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.126 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.126 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:107991 7.e 7:74ab0424:::benchmark_data_smithi150_78508_object107990:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-11T08:17:08.126 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:10.917994+0000", 2024-09-11T08:17:08.126 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 357.18318051300002, 2024-09-11T08:17:08.127 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.113263696, 2024-09-11T08:17:08.127 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:17:08.127 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:17:08.127 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:17:08.127 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:17:08.127 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:17:08.127 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 107991 2024-09-11T08:17:08.127 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.127 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:17:08.127 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.127 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:17:08.127 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.917994+0000", 2024-09-11T08:17:08.127 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:17:08.128 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.128 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.128 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:17:08.128 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.917994+0000", 2024-09-11T08:17:08.128 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:17:08.128 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.128 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.128 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:17:08.128 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.917992+0000", 2024-09-11T08:17:08.128 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:17:08.128 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.128 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.128 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:17:08.128 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.918015+0000", 2024-09-11T08:17:08.129 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.2524e-05 2024-09-11T08:17:08.129 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.129 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.129 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:17:08.129 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.918027+0000", 2024-09-11T08:17:08.129 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1789e-05 2024-09-11T08:17:08.129 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.129 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.129 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:17:08.129 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.918032+0000", 2024-09-11T08:17:08.129 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.806e-06 2024-09-11T08:17:08.129 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.129 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.129 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:17:08.130 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.918321+0000", 2024-09-11T08:17:08.130 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00028877299999999998 2024-09-11T08:17:08.130 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.130 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.130 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:17:08.130 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.918431+0000", 2024-09-11T08:17:08.130 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00010980399999999999 2024-09-11T08:17:08.130 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.130 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.130 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 0,5", 2024-09-11T08:17:08.130 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.918551+0000", 2024-09-11T08:17:08.130 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00011991500000000001 2024-09-11T08:17:08.131 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.131 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.131 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:17:08.131 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.923453+0000", 2024-09-11T08:17:08.131 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0049019279999999998 2024-09-11T08:17:08.131 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.131 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.131 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:17:08.131 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.926055+0000", 2024-09-11T08:17:08.131 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0026020879999999998 2024-09-11T08:17:08.131 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.131 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.131 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:17:08.132 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.031174+0000", 2024-09-11T08:17:08.132 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.10511904599999999 2024-09-11T08:17:08.132 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.132 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.132 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:17:08.132 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.031222+0000", 2024-09-11T08:17:08.132 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.8442000000000001e-05 2024-09-11T08:17:08.132 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.132 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.132 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:17:08.132 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.031257+0000", 2024-09-11T08:17:08.132 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.4888999999999997e-05 2024-09-11T08:17:08.132 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:08.132 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:08.133 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:08.133 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.133 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.133 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:114845 7.e 7:7f844ee0:::benchmark_data_smithi150_78508_object114844:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-09-11T08:17:08.133 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:17.777734+0000", 2024-09-11T08:17:08.133 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 350.32343994600001, 2024-09-11T08:17:08.133 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.086747156000000006, 2024-09-11T08:17:08.133 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:17:08.133 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:17:08.133 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:17:08.133 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:17:08.133 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:17:08.133 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 114845 2024-09-11T08:17:08.134 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.134 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:17:08.134 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.134 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:17:08.134 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.777734+0000", 2024-09-11T08:17:08.134 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:17:08.134 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.134 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.134 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:17:08.134 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.777734+0000", 2024-09-11T08:17:08.134 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:17:08.134 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.134 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.134 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:17:08.135 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.777733+0000", 2024-09-11T08:17:08.135 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:17:08.135 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.135 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.135 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:17:08.135 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.777771+0000", 2024-09-11T08:17:08.135 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.8012999999999997e-05 2024-09-11T08:17:08.135 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.135 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.135 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:17:08.135 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.777780+0000", 2024-09-11T08:17:08.135 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.7290000000000006e-06 2024-09-11T08:17:08.135 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.135 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.136 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:17:08.136 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.777794+0000", 2024-09-11T08:17:08.136 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4093999999999999e-05 2024-09-11T08:17:08.136 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.136 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.136 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:17:08.136 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.777894+0000", 2024-09-11T08:17:08.136 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000100695 2024-09-11T08:17:08.136 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.136 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.136 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:17:08.136 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.778009+0000", 2024-09-11T08:17:08.136 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000115081 2024-09-11T08:17:08.137 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.137 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.137 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 0,5", 2024-09-11T08:17:08.137 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.778151+0000", 2024-09-11T08:17:08.137 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00014143900000000001 2024-09-11T08:17:08.137 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.137 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.137 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:17:08.137 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.785147+0000", 2024-09-11T08:17:08.137 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0069962779999999999 2024-09-11T08:17:08.137 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.137 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.137 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:17:08.137 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.786065+0000", 2024-09-11T08:17:08.138 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00091810500000000001 2024-09-11T08:17:08.138 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.138 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.138 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:17:08.138 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.864393+0000", 2024-09-11T08:17:08.138 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.078327752 2024-09-11T08:17:08.138 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.138 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.138 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:17:08.138 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.864442+0000", 2024-09-11T08:17:08.138 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.8751000000000001e-05 2024-09-11T08:17:08.138 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.138 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.138 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:17:08.139 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.864481+0000", 2024-09-11T08:17:08.139 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.9648999999999999e-05 2024-09-11T08:17:08.139 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:08.139 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:08.139 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:08.139 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.139 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.139 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:114851 7.e 7:7efef43e:::benchmark_data_smithi150_78508_object114850:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-09-11T08:17:08.139 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:17.780925+0000", 2024-09-11T08:17:08.139 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 350.32024879800002, 2024-09-11T08:17:08.139 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.091782973000000004, 2024-09-11T08:17:08.139 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:17:08.139 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:17:08.139 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:17:08.140 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:17:08.140 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:17:08.140 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 114851 2024-09-11T08:17:08.140 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.140 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:17:08.140 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.141 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:17:08.141 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.780925+0000", 2024-09-11T08:17:08.141 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:17:08.141 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.141 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.141 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:17:08.142 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.780925+0000", 2024-09-11T08:17:08.142 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:17:08.142 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.142 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.142 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:17:08.142 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.780924+0000", 2024-09-11T08:17:08.142 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:17:08.142 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.142 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.143 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:17:08.143 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.780945+0000", 2024-09-11T08:17:08.143 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.1457000000000001e-05 2024-09-11T08:17:08.143 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.143 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.143 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:17:08.143 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.780955+0000", 2024-09-11T08:17:08.143 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.9699999999999994e-06 2024-09-11T08:17:08.143 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.143 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.143 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:17:08.143 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.780960+0000", 2024-09-11T08:17:08.144 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.8740000000000003e-06 2024-09-11T08:17:08.144 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.144 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.144 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:17:08.144 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.781504+0000", 2024-09-11T08:17:08.144 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00054328099999999995 2024-09-11T08:17:08.144 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.144 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.144 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:17:08.144 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.781619+0000", 2024-09-11T08:17:08.144 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000115808 2024-09-11T08:17:08.144 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.144 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.144 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 0,5", 2024-09-11T08:17:08.145 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.781752+0000", 2024-09-11T08:17:08.145 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00013221700000000001 2024-09-11T08:17:08.145 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.145 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.145 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:17:08.145 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.786059+0000", 2024-09-11T08:17:08.145 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0043070010000000004 2024-09-11T08:17:08.145 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.145 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.145 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:17:08.145 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.786072+0000", 2024-09-11T08:17:08.145 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3461e-05 2024-09-11T08:17:08.145 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.146 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.146 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:17:08.146 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.872493+0000", 2024-09-11T08:17:08.146 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.086420728000000002 2024-09-11T08:17:08.146 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.146 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.146 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:17:08.146 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.872581+0000", 2024-09-11T08:17:08.146 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.8572999999999994e-05 2024-09-11T08:17:08.146 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.146 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.146 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:17:08.146 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.872708+0000", 2024-09-11T08:17:08.146 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000127052 2024-09-11T08:17:08.147 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:08.147 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:08.147 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:08.147 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.147 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.147 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:114857 7.e 7:7b90298a:::benchmark_data_smithi150_78508_object114856:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-09-11T08:17:08.147 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:17.788110+0000", 2024-09-11T08:17:08.147 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 350.31306378699998, 2024-09-11T08:17:08.147 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.084835387999999998, 2024-09-11T08:17:08.147 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:17:08.147 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:17:08.147 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:17:08.148 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:17:08.148 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:17:08.148 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 114857 2024-09-11T08:17:08.148 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.148 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:17:08.148 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.148 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:17:08.148 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.788110+0000", 2024-09-11T08:17:08.148 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:17:08.148 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.148 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.148 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:17:08.148 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.788110+0000", 2024-09-11T08:17:08.149 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:17:08.149 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.149 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.149 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:17:08.149 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.788109+0000", 2024-09-11T08:17:08.149 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:17:08.149 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.149 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.149 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:17:08.149 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.788134+0000", 2024-09-11T08:17:08.149 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.514e-05 2024-09-11T08:17:08.149 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.149 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.150 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:17:08.150 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.788151+0000", 2024-09-11T08:17:08.150 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.6716999999999999e-05 2024-09-11T08:17:08.150 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.150 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.150 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:17:08.150 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.788157+0000", 2024-09-11T08:17:08.150 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.9079999999999998e-06 2024-09-11T08:17:08.150 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.150 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.150 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:17:08.150 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.788390+0000", 2024-09-11T08:17:08.150 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00023229199999999999 2024-09-11T08:17:08.151 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.151 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.151 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:17:08.151 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.788524+0000", 2024-09-11T08:17:08.151 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000133855 2024-09-11T08:17:08.151 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.151 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.151 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 0,5", 2024-09-11T08:17:08.151 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.788665+0000", 2024-09-11T08:17:08.151 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000141185 2024-09-11T08:17:08.151 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.151 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.151 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:17:08.151 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.798016+0000", 2024-09-11T08:17:08.151 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0093515319999999992 2024-09-11T08:17:08.152 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.152 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.152 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:17:08.152 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.798023+0000", 2024-09-11T08:17:08.152 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.81e-06 2024-09-11T08:17:08.152 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.152 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.152 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:17:08.152 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.872748+0000", 2024-09-11T08:17:08.152 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.074725134999999998 2024-09-11T08:17:08.152 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.152 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.152 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:17:08.152 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.872908+0000", 2024-09-11T08:17:08.153 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00015925600000000001 2024-09-11T08:17:08.153 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.153 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.153 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:17:08.153 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.872946+0000", 2024-09-11T08:17:08.153 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.8278000000000002e-05 2024-09-11T08:17:08.153 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:08.153 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:08.153 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:08.153 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.153 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.153 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:114858 7.e 7:727353af:::benchmark_data_smithi150_78508_object114857:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-09-11T08:17:08.153 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:17.788366+0000", 2024-09-11T08:17:08.153 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 350.31280774999999, 2024-09-11T08:17:08.154 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.084697621000000001, 2024-09-11T08:17:08.154 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:17:08.154 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:17:08.154 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:17:08.154 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:17:08.154 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:17:08.154 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 114858 2024-09-11T08:17:08.154 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.154 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:17:08.154 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.154 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:17:08.154 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.788366+0000", 2024-09-11T08:17:08.155 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:17:08.155 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.155 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.155 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:17:08.155 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.788366+0000", 2024-09-11T08:17:08.155 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:17:08.155 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.155 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.155 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:17:08.155 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.788366+0000", 2024-09-11T08:17:08.155 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:17:08.155 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.155 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.155 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:17:08.155 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.788492+0000", 2024-09-11T08:17:08.156 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000126384 2024-09-11T08:17:08.156 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.156 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.156 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:17:08.156 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.788502+0000", 2024-09-11T08:17:08.156 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.4520000000000003e-06 2024-09-11T08:17:08.156 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.156 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.156 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:17:08.156 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.788508+0000", 2024-09-11T08:17:08.156 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.0129999999999999e-06 2024-09-11T08:17:08.156 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.156 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.156 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:17:08.157 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.793843+0000", 2024-09-11T08:17:08.157 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0053355479999999999 2024-09-11T08:17:08.157 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.157 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.157 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:17:08.157 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.793959+0000", 2024-09-11T08:17:08.157 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000115758 2024-09-11T08:17:08.157 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.157 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.157 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 0,5", 2024-09-11T08:17:08.157 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.794096+0000", 2024-09-11T08:17:08.157 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0001372 2024-09-11T08:17:08.157 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.157 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.158 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:17:08.158 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.798547+0000", 2024-09-11T08:17:08.158 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0044509349999999996 2024-09-11T08:17:08.158 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.158 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.158 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:17:08.158 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.799019+0000", 2024-09-11T08:17:08.158 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000472287 2024-09-11T08:17:08.158 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.158 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.158 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:17:08.158 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.872987+0000", 2024-09-11T08:17:08.158 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.073967226999999997 2024-09-11T08:17:08.159 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.159 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.159 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:17:08.159 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.873035+0000", 2024-09-11T08:17:08.159 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.8316000000000003e-05 2024-09-11T08:17:08.159 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.159 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.159 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:17:08.159 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:17.873064+0000", 2024-09-11T08:17:08.159 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.9221e-05 2024-09-11T08:17:08.159 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:08.159 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:08.159 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:08.160 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.160 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.160 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:185885 7.8 7:16afb42a:::benchmark_data_smithi150_78508_object61444:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-11T08:17:08.160 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:12:27.481789+0000", 2024-09-11T08:17:08.160 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 280.61938471600001, 2024-09-11T08:17:08.160 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0640386980000001, 2024-09-11T08:17:08.160 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:17:08.160 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-09-11T08:17:08.160 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:17:08.160 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:17:08.160 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:17:08.160 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 185885 2024-09-11T08:17:08.160 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.161 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:17:08.161 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.161 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:17:08.161 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.481789+0000", 2024-09-11T08:17:08.161 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:17:08.161 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.161 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.161 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:17:08.161 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.481789+0000", 2024-09-11T08:17:08.161 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:17:08.161 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.161 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.161 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:17:08.162 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.481780+0000", 2024-09-11T08:17:08.162 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999905 2024-09-11T08:17:08.162 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.162 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.162 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:17:08.162 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.481794+0000", 2024-09-11T08:17:08.162 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4771e-05 2024-09-11T08:17:08.162 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.162 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.162 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:17:08.162 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.481809+0000", 2024-09-11T08:17:08.162 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4681000000000001e-05 2024-09-11T08:17:08.162 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.163 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.163 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:17:08.163 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.481815+0000", 2024-09-11T08:17:08.163 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.263e-06 2024-09-11T08:17:08.163 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.163 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.163 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:17:08.163 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.481957+0000", 2024-09-11T08:17:08.163 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00014108100000000001 2024-09-11T08:17:08.163 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.163 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.163 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:17:08.163 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.482320+0000", 2024-09-11T08:17:08.163 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000363397 2024-09-11T08:17:08.164 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.164 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.164 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 4,7", 2024-09-11T08:17:08.164 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.482701+0000", 2024-09-11T08:17:08.164 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00038093399999999998 2024-09-11T08:17:08.164 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.164 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.164 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:08.165 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:17:08.166 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.510037+0000", 2024-09-11T08:17:08.166 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.027335673000000001 2024-09-11T08:17:08.166 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.166 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.166 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:17:08.166 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.510044+0000", 2024-09-11T08:17:08.166 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.5660000000000001e-06 2024-09-11T08:17:08.166 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.166 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.166 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:17:08.166 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.542329+0000", 2024-09-11T08:17:08.166 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0322849839999999 2024-09-11T08:17:08.167 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.167 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.167 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for missing object", 2024-09-11T08:17:08.167 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.542426+0000", 2024-09-11T08:17:08.167 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.7194000000000005e-05 2024-09-11T08:17:08.167 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.167 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.167 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:17:08.167 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.545755+0000", 2024-09-11T08:17:08.167 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0033286380000000001 2024-09-11T08:17:08.167 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.167 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.168 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:17:08.168 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.545828+0000", 2024-09-11T08:17:08.168 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.3269000000000006e-05 2024-09-11T08:17:08.168 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:08.168 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:08.168 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:08.168 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.168 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.168 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:185886 7.5 7:a47178a9:::benchmark_data_smithi150_78508_object61445:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e140)", 2024-09-11T08:17:08.168 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:12:27.500199+0000", 2024-09-11T08:17:08.168 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 280.60097501400003, 2024-09-11T08:17:08.168 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0339040960000001, 2024-09-11T08:17:08.168 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:17:08.169 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-09-11T08:17:08.169 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:17:08.169 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:17:08.169 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:17:08.169 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 185886 2024-09-11T08:17:08.169 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.169 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:17:08.169 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.169 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:17:08.169 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.500199+0000", 2024-09-11T08:17:08.169 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:17:08.169 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.169 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.169 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:17:08.170 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.500199+0000", 2024-09-11T08:17:08.170 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:17:08.170 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.170 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.170 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:17:08.170 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.500198+0000", 2024-09-11T08:17:08.170 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:17:08.170 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.170 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.170 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:17:08.171 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.500204+0000", 2024-09-11T08:17:08.171 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.4670000000000001e-06 2024-09-11T08:17:08.171 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.171 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.171 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:17:08.171 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.500217+0000", 2024-09-11T08:17:08.171 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3188e-05 2024-09-11T08:17:08.171 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.171 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.171 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:17:08.171 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.500224+0000", 2024-09-11T08:17:08.172 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.1179999999999997e-06 2024-09-11T08:17:08.172 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.172 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.172 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:17:08.172 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.502530+0000", 2024-09-11T08:17:08.172 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0023060939999999999 2024-09-11T08:17:08.172 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.172 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.172 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:17:08.172 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.502537+0000", 2024-09-11T08:17:08.172 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.2540000000000003e-06 2024-09-11T08:17:08.172 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.173 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.173 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:17:08.173 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.534033+0000", 2024-09-11T08:17:08.173 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.031495638 2024-09-11T08:17:08.173 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.173 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.173 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:17:08.173 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.534103+0000", 2024-09-11T08:17:08.173 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.0760999999999996e-05 2024-09-11T08:17:08.173 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:08.173 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:08.173 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:08.174 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.174 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.174 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:185893 7.5 7:a8ebd761:::benchmark_data_smithi150_78508_object61452:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e140)", 2024-09-11T08:17:08.174 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:12:27.500232+0000", 2024-09-11T08:17:08.174 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 280.60094252499999, 2024-09-11T08:17:08.174 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0502388069999999, 2024-09-11T08:17:08.174 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:17:08.174 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-09-11T08:17:08.175 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:17:08.175 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:17:08.175 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:17:08.175 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 185893 2024-09-11T08:17:08.175 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.175 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:17:08.175 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.175 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:17:08.175 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.500232+0000", 2024-09-11T08:17:08.175 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:17:08.175 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.175 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.175 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:17:08.175 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.500232+0000", 2024-09-11T08:17:08.175 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:17:08.176 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.176 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.176 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:17:08.176 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.500231+0000", 2024-09-11T08:17:08.176 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:17:08.176 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.176 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.176 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:17:08.176 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.500234+0000", 2024-09-11T08:17:08.176 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.6699999999999998e-06 2024-09-11T08:17:08.176 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.176 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.176 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:17:08.177 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.500239+0000", 2024-09-11T08:17:08.177 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.7720000000000002e-06 2024-09-11T08:17:08.177 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.177 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.177 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:17:08.177 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.500242+0000", 2024-09-11T08:17:08.177 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.4249999999999998e-06 2024-09-11T08:17:08.177 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.177 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.177 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:17:08.177 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.502556+0000", 2024-09-11T08:17:08.177 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0023134829999999999 2024-09-11T08:17:08.177 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.177 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.177 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:17:08.178 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.502561+0000", 2024-09-11T08:17:08.178 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.967e-06 2024-09-11T08:17:08.178 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.178 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.178 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:17:08.178 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.534126+0000", 2024-09-11T08:17:08.178 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.031565501 2024-09-11T08:17:08.178 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.178 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.178 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for missing object", 2024-09-11T08:17:08.178 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.534214+0000", 2024-09-11T08:17:08.178 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.7558000000000004e-05 2024-09-11T08:17:08.178 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.179 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.179 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:17:08.179 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.550276+0000", 2024-09-11T08:17:08.179 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.016062779999999999 2024-09-11T08:17:08.179 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.179 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.179 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:17:08.179 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.550470+0000", 2024-09-11T08:17:08.179 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000194017 2024-09-11T08:17:08.179 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:08.179 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:08.179 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:08.179 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.179 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.180 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:185895 7.5 7:af92df4d:::benchmark_data_smithi150_78508_object61454:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-09-11T08:17:08.180 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:12:27.504076+0000", 2024-09-11T08:17:08.180 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 280.59709855099999, 2024-09-11T08:17:08.180 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.046040546, 2024-09-11T08:17:08.180 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:17:08.180 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:17:08.180 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:17:08.180 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:17:08.180 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:17:08.180 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 185895 2024-09-11T08:17:08.180 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.180 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:17:08.180 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.181 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:17:08.181 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.504076+0000", 2024-09-11T08:17:08.181 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:17:08.181 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.181 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.181 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:17:08.181 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.504076+0000", 2024-09-11T08:17:08.181 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:17:08.181 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.181 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.181 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:17:08.181 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.504074+0000", 2024-09-11T08:17:08.181 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:17:08.181 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.182 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.182 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:17:08.182 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.504081+0000", 2024-09-11T08:17:08.182 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.1319999999999996e-06 2024-09-11T08:17:08.182 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.182 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.182 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:17:08.182 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.504114+0000", 2024-09-11T08:17:08.182 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.2990000000000001e-05 2024-09-11T08:17:08.182 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.182 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.182 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:17:08.182 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.504121+0000", 2024-09-11T08:17:08.182 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.0330000000000003e-06 2024-09-11T08:17:08.183 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.183 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.183 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:17:08.183 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.504486+0000", 2024-09-11T08:17:08.183 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00036412 2024-09-11T08:17:08.183 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.183 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.183 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:17:08.183 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.504493+0000", 2024-09-11T08:17:08.183 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.4379999999999998e-06 2024-09-11T08:17:08.184 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.184 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.184 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:17:08.184 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.534238+0000", 2024-09-11T08:17:08.184 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0297454699999999 2024-09-11T08:17:08.184 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.184 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.184 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:17:08.184 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.534349+0000", 2024-09-11T08:17:08.184 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000110535 2024-09-11T08:17:08.184 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.184 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.184 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 0", 2024-09-11T08:17:08.185 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.534463+0000", 2024-09-11T08:17:08.185 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000114015 2024-09-11T08:17:08.185 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.185 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.185 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:17:08.185 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.546279+0000", 2024-09-11T08:17:08.185 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.01181631 2024-09-11T08:17:08.185 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.185 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.185 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:17:08.185 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.549934+0000", 2024-09-11T08:17:08.185 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.003655054 2024-09-11T08:17:08.185 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.185 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.186 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:17:08.186 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.550022+0000", 2024-09-11T08:17:08.186 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.7619000000000003e-05 2024-09-11T08:17:08.186 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.186 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.186 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:17:08.186 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.550116+0000", 2024-09-11T08:17:08.186 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.4242999999999999e-05 2024-09-11T08:17:08.186 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:08.186 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:08.186 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:08.186 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.186 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.186 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:185900 7.5 7:a3904075:::benchmark_data_smithi150_78508_object61459:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-09-11T08:17:08.187 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:12:27.517267+0000", 2024-09-11T08:17:08.187 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 280.58390671799998, 2024-09-11T08:17:08.187 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.03349158, 2024-09-11T08:17:08.187 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:17:08.187 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:17:08.187 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:17:08.187 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:17:08.187 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:17:08.187 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 185900 2024-09-11T08:17:08.187 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.187 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:17:08.187 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.187 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:17:08.187 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.517267+0000", 2024-09-11T08:17:08.188 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:17:08.188 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.188 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.188 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:17:08.188 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.517267+0000", 2024-09-11T08:17:08.188 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:17:08.188 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.188 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.188 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:17:08.188 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.517266+0000", 2024-09-11T08:17:08.188 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:17:08.188 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.188 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.188 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:17:08.188 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.517273+0000", 2024-09-11T08:17:08.189 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.4880000000000001e-06 2024-09-11T08:17:08.189 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.189 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.189 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:17:08.189 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.517289+0000", 2024-09-11T08:17:08.189 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.5449000000000001e-05 2024-09-11T08:17:08.189 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.189 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.189 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:17:08.189 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.517295+0000", 2024-09-11T08:17:08.189 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.2720000000000003e-06 2024-09-11T08:17:08.189 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.189 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.189 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:17:08.190 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.517914+0000", 2024-09-11T08:17:08.190 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00061895500000000003 2024-09-11T08:17:08.190 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.190 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.190 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:17:08.190 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.517921+0000", 2024-09-11T08:17:08.190 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.6579999999999996e-06 2024-09-11T08:17:08.190 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.190 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.190 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:17:08.190 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.537074+0000", 2024-09-11T08:17:08.190 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0191534790000001 2024-09-11T08:17:08.190 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.190 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.191 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:17:08.191 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.537191+0000", 2024-09-11T08:17:08.191 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00011654099999999999 2024-09-11T08:17:08.191 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.191 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.191 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 0", 2024-09-11T08:17:08.191 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.537306+0000", 2024-09-11T08:17:08.191 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000115671 2024-09-11T08:17:08.191 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.191 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.191 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:17:08.191 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.550128+0000", 2024-09-11T08:17:08.191 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.012822142 2024-09-11T08:17:08.192 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.192 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.192 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:17:08.192 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.550653+0000", 2024-09-11T08:17:08.192 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00052431500000000002 2024-09-11T08:17:08.192 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.192 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.192 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:17:08.192 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.550689+0000", 2024-09-11T08:17:08.192 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.6105000000000001e-05 2024-09-11T08:17:08.192 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.192 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.192 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:17:08.192 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.550759+0000", 2024-09-11T08:17:08.193 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.0135999999999994e-05 2024-09-11T08:17:08.193 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:08.193 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:08.193 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:08.193 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.193 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.193 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:185906 7.5 7:a0b92603:::benchmark_data_smithi150_78508_object61465:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-09-11T08:17:08.193 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:12:27.520019+0000", 2024-09-11T08:17:08.193 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 280.58115553200003, 2024-09-11T08:17:08.193 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.03434936, 2024-09-11T08:17:08.193 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:17:08.193 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:17:08.194 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:17:08.194 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:17:08.194 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:17:08.194 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 185906 2024-09-11T08:17:08.194 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.194 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:17:08.194 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.194 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:17:08.194 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.520019+0000", 2024-09-11T08:17:08.194 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:17:08.194 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.194 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.194 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:17:08.194 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.520019+0000", 2024-09-11T08:17:08.195 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:17:08.195 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.195 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.195 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:17:08.195 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.520017+0000", 2024-09-11T08:17:08.195 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:17:08.195 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.195 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.195 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:17:08.195 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.520025+0000", 2024-09-11T08:17:08.195 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.1100000000000003e-06 2024-09-11T08:17:08.195 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.195 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.196 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:17:08.196 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.520043+0000", 2024-09-11T08:17:08.196 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.7788999999999999e-05 2024-09-11T08:17:08.196 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.196 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.196 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:17:08.196 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.520050+0000", 2024-09-11T08:17:08.196 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.0650000000000001e-06 2024-09-11T08:17:08.196 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.196 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.196 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:17:08.196 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.520655+0000", 2024-09-11T08:17:08.196 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00060452800000000001 2024-09-11T08:17:08.196 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.197 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.197 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:17:08.197 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.520662+0000", 2024-09-11T08:17:08.197 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.5660000000000001e-06 2024-09-11T08:17:08.197 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.197 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.197 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:17:08.197 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.540811+0000", 2024-09-11T08:17:08.197 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.020148654 2024-09-11T08:17:08.197 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.197 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.197 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:17:08.197 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.540920+0000", 2024-09-11T08:17:08.198 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000109677 2024-09-11T08:17:08.198 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.198 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.198 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 0", 2024-09-11T08:17:08.198 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.541027+0000", 2024-09-11T08:17:08.198 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000106686 2024-09-11T08:17:08.198 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.198 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.198 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:17:08.198 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.550134+0000", 2024-09-11T08:17:08.198 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0091063440000000006 2024-09-11T08:17:08.198 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.198 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.198 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:17:08.199 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.554226+0000", 2024-09-11T08:17:08.199 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0040922190000000002 2024-09-11T08:17:08.199 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.199 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.199 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:17:08.199 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.554282+0000", 2024-09-11T08:17:08.199 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.5781e-05 2024-09-11T08:17:08.199 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.199 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.199 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:17:08.199 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.554368+0000", 2024-09-11T08:17:08.199 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.6502999999999995e-05 2024-09-11T08:17:08.199 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:08.199 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:08.200 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:08.200 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.200 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.200 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.24320.0:929 5.d 5:b4812045:::notify.4:head [watch reconnect cookie 94783556104128 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-09-11T08:17:08.200 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:12:27.939507+0000", 2024-09-11T08:17:08.200 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 280.16166722700001, 2024-09-11T08:17:08.200 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.611529616, 2024-09-11T08:17:08.200 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:17:08.200 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "started", 2024-09-11T08:17:08.200 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:17:08.200 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.24320", 2024-09-11T08:17:08.200 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1503848706", 2024-09-11T08:17:08.200 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 929 2024-09-11T08:17:08.200 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.201 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:17:08.201 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.201 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:17:08.201 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.939507+0000", 2024-09-11T08:17:08.201 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:17:08.201 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.201 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.201 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:17:08.201 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.939507+0000", 2024-09-11T08:17:08.201 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:17:08.201 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.201 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.201 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:17:08.201 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.939505+0000", 2024-09-11T08:17:08.202 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:17:08.202 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.202 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.202 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:17:08.202 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.939515+0000", 2024-09-11T08:17:08.202 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0223e-05 2024-09-11T08:17:08.202 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.202 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.202 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:17:08.202 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.939529+0000", 2024-09-11T08:17:08.202 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3655999999999999e-05 2024-09-11T08:17:08.202 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.202 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.203 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:17:08.203 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.939536+0000", 2024-09-11T08:17:08.203 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.2830000000000004e-06 2024-09-11T08:17:08.203 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.203 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.203 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:17:08.203 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.939589+0000", 2024-09-11T08:17:08.203 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.2796000000000001e-05 2024-09-11T08:17:08.203 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.203 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.204 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:17:08.204 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.939599+0000", 2024-09-11T08:17:08.204 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0128e-05 2024-09-11T08:17:08.204 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.204 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.204 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:17:08.204 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.546712+0000", 2024-09-11T08:17:08.204 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.607113299 2024-09-11T08:17:08.204 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.204 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.204 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:17:08.204 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.546873+0000", 2024-09-11T08:17:08.205 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000160932 2024-09-11T08:17:08.205 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.205 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.205 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:17:08.205 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.551037+0000", 2024-09-11T08:17:08.205 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0041632209999999999 2024-09-11T08:17:08.205 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:08.205 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:08.205 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:08.205 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.205 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.205 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:238414 7.5 7:ac6bf147:::benchmark_data_smithi150_78508_object113973:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-11T08:17:08.206 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:13:18.623096+0000", 2024-09-11T08:17:08.206 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 229.478078283, 2024-09-11T08:17:08.206 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.081778343000000003, 2024-09-11T08:17:08.206 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:17:08.206 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:17:08.206 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:17:08.206 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:17:08.206 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:17:08.206 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 238414 2024-09-11T08:17:08.206 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.206 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:17:08.206 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.206 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:17:08.207 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:18.623096+0000", 2024-09-11T08:17:08.207 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:17:08.207 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.207 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.207 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:17:08.207 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:18.623096+0000", 2024-09-11T08:17:08.207 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:17:08.207 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.207 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.207 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:17:08.207 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:18.623095+0000", 2024-09-11T08:17:08.207 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:17:08.207 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.207 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.208 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:17:08.208 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:18.623099+0000", 2024-09-11T08:17:08.208 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1570000000000002e-06 2024-09-11T08:17:08.208 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.208 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.208 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:17:08.208 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:18.623106+0000", 2024-09-11T08:17:08.208 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.2169999999999997e-06 2024-09-11T08:17:08.208 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.208 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.208 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:17:08.208 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:18.623110+0000", 2024-09-11T08:17:08.208 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.6009999999999999e-06 2024-09-11T08:17:08.208 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.208 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.209 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:17:08.209 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:18.623145+0000", 2024-09-11T08:17:08.209 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.4780000000000002e-05 2024-09-11T08:17:08.209 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.209 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.209 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:17:08.209 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:18.623217+0000", 2024-09-11T08:17:08.209 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.1808000000000001e-05 2024-09-11T08:17:08.209 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.209 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.209 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 0", 2024-09-11T08:17:08.209 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:18.623302+0000", 2024-09-11T08:17:08.209 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.5217999999999996e-05 2024-09-11T08:17:08.209 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.210 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.210 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:17:08.210 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:18.635140+0000", 2024-09-11T08:17:08.210 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.011837926 2024-09-11T08:17:08.210 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.210 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.210 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:17:08.210 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:18.704739+0000", 2024-09-11T08:17:08.210 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.069599108000000007 2024-09-11T08:17:08.210 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.210 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.210 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:17:08.210 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:18.704794+0000", 2024-09-11T08:17:08.211 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.5386000000000001e-05 2024-09-11T08:17:08.211 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.211 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.211 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:17:08.211 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:18.704874+0000", 2024-09-11T08:17:08.211 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.9948000000000001e-05 2024-09-11T08:17:08.211 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:08.211 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:08.211 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:08.211 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.211 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.211 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:239262 7.e 7:75393bd7:::benchmark_data_smithi150_78508_object114821:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-11T08:17:08.211 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:13:19.516813+0000", 2024-09-11T08:17:08.212 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 228.58436168399999, 2024-09-11T08:17:08.212 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.096861592999999996, 2024-09-11T08:17:08.212 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:17:08.212 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:17:08.212 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:17:08.212 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:17:08.212 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:17:08.212 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 239262 2024-09-11T08:17:08.212 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.212 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:17:08.212 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.212 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:17:08.212 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:19.516813+0000", 2024-09-11T08:17:08.212 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:17:08.213 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.213 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.213 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:17:08.213 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:19.516813+0000", 2024-09-11T08:17:08.213 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:17:08.213 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.213 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.213 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:17:08.213 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:19.516811+0000", 2024-09-11T08:17:08.213 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:17:08.213 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.213 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.213 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:17:08.214 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:19.516819+0000", 2024-09-11T08:17:08.214 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.5979999999999997e-06 2024-09-11T08:17:08.214 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.214 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.214 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:17:08.214 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:19.516833+0000", 2024-09-11T08:17:08.214 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3528e-05 2024-09-11T08:17:08.214 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.214 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.214 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:17:08.214 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:19.516839+0000", 2024-09-11T08:17:08.214 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.5520000000000001e-06 2024-09-11T08:17:08.214 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.214 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.215 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:17:08.215 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:19.516889+0000", 2024-09-11T08:17:08.215 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.9601999999999997e-05 2024-09-11T08:17:08.215 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.215 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.215 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:17:08.215 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:19.516994+0000", 2024-09-11T08:17:08.215 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000104959 2024-09-11T08:17:08.215 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.215 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.215 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 0,5", 2024-09-11T08:17:08.216 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:19.517096+0000", 2024-09-11T08:17:08.216 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00010181199999999999 2024-09-11T08:17:08.216 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.216 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.216 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:17:08.216 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:19.522339+0000", 2024-09-11T08:17:08.216 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.005243124 2024-09-11T08:17:08.216 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.216 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.216 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:17:08.216 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:19.523424+0000", 2024-09-11T08:17:08.216 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0010855999999999999 2024-09-11T08:17:08.216 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.217 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.217 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:17:08.217 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:19.613386+0000", 2024-09-11T08:17:08.217 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.089961429999999995 2024-09-11T08:17:08.217 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.217 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.217 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:17:08.217 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:19.613635+0000", 2024-09-11T08:17:08.217 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00024948599999999999 2024-09-11T08:17:08.217 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.217 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.217 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:17:08.217 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:13:19.613674+0000", 2024-09-11T08:17:08.217 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.8838999999999999e-05 2024-09-11T08:17:08.218 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:08.218 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:08.218 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:08.218 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.218 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.218 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.15087.0:54333 8.f e175/159 8:f9667130:::benchmark_data_smithi150_107646_object54332:head v 175'3422, mlcod=175'3420)", 2024-09-11T08:17:08.218 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:14:39.758366+0000", 2024-09-11T08:17:08.218 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 148.34280779299999, 2024-09-11T08:17:08.218 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.098538254000000006, 2024-09-11T08:17:08.218 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:17:08.218 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:17:08.218 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:17:08.218 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.219 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:17:08.219 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:14:39.758366+0000", 2024-09-11T08:17:08.219 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:17:08.219 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.219 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.219 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:17:08.219 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:14:39.758366+0000", 2024-09-11T08:17:08.219 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:17:08.219 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.219 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.220 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:17:08.220 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:14:39.758366+0000", 2024-09-11T08:17:08.220 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:17:08.220 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.220 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.220 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:17:08.221 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:14:39.758411+0000", 2024-09-11T08:17:08.221 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.5618000000000003e-05 2024-09-11T08:17:08.221 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.221 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.222 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:17:08.222 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:14:39.758422+0000", 2024-09-11T08:17:08.222 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0407999999999999e-05 2024-09-11T08:17:08.222 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.223 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.223 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:17:08.223 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:14:39.758428+0000", 2024-09-11T08:17:08.223 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.2809999999999997e-06 2024-09-11T08:17:08.223 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.224 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.224 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:17:08.224 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:14:39.758614+0000", 2024-09-11T08:17:08.224 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000186069 2024-09-11T08:17:08.224 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.224 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.224 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:17:08.224 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:14:39.758642+0000", 2024-09-11T08:17:08.225 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.7387000000000001e-05 2024-09-11T08:17:08.225 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.225 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.225 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:17:08.225 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:14:39.856870+0000", 2024-09-11T08:17:08.225 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.098228812999999998 2024-09-11T08:17:08.225 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.225 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.225 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:17:08.225 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:14:39.856905+0000", 2024-09-11T08:17:08.225 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.4248999999999997e-05 2024-09-11T08:17:08.225 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:08.225 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:08.226 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:08.226 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.226 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.226 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.15087.0:54337 8.4 e175/159 8:2e22a651:::benchmark_data_smithi150_107646_object54336:head v 175'3413, mlcod=175'3411)", 2024-09-11T08:17:08.226 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:14:39.762318+0000", 2024-09-11T08:17:08.226 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 148.33885647599999, 2024-09-11T08:17:08.226 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.090962681000000004, 2024-09-11T08:17:08.226 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:17:08.226 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:17:08.226 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:17:08.226 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.226 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:17:08.226 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:14:39.762318+0000", 2024-09-11T08:17:08.226 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:17:08.227 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.227 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.227 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:17:08.227 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:14:39.762318+0000", 2024-09-11T08:17:08.227 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:17:08.227 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.227 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.228 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:17:08.228 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:14:39.762317+0000", 2024-09-11T08:17:08.228 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:17:08.228 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.228 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.228 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:17:08.228 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:14:39.762414+0000", 2024-09-11T08:17:08.228 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.7273000000000002e-05 2024-09-11T08:17:08.229 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.229 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.229 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:17:08.229 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:14:39.849683+0000", 2024-09-11T08:17:08.229 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.087269237999999999 2024-09-11T08:17:08.229 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.229 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.229 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:17:08.229 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:14:39.849693+0000", 2024-09-11T08:17:08.229 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0190000000000001e-05 2024-09-11T08:17:08.229 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.229 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.229 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:17:08.230 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:14:39.849998+0000", 2024-09-11T08:17:08.230 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00030498299999999999 2024-09-11T08:17:08.230 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.230 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.230 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:17:08.230 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:14:39.850019+0000", 2024-09-11T08:17:08.230 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.031e-05 2024-09-11T08:17:08.230 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.230 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.230 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:17:08.230 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:14:39.853178+0000", 2024-09-11T08:17:08.230 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0031592370000000001 2024-09-11T08:17:08.231 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.231 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.231 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:17:08.231 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:14:39.853280+0000", 2024-09-11T08:17:08.231 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000102607 2024-09-11T08:17:08.231 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:08.231 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:08.231 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:08.231 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.231 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.231 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.15087.0:54334 8.f e175/159 8:f20ddaa1:::benchmark_data_smithi150_107646_object54333:head v 175'3423, mlcod=175'3420)", 2024-09-11T08:17:08.231 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:14:39.768855+0000", 2024-09-11T08:17:08.232 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 148.33231914000001, 2024-09-11T08:17:08.232 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.088069755999999999, 2024-09-11T08:17:08.232 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:17:08.232 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:17:08.232 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:17:08.232 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.232 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:17:08.232 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:14:39.768855+0000", 2024-09-11T08:17:08.232 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:17:08.232 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.232 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.232 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:17:08.232 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:14:39.768855+0000", 2024-09-11T08:17:08.232 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:17:08.233 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.233 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.233 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:17:08.233 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:14:39.768854+0000", 2024-09-11T08:17:08.233 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:17:08.233 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.233 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.233 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:17:08.233 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:14:39.768884+0000", 2024-09-11T08:17:08.233 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.0032e-05 2024-09-11T08:17:08.233 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.233 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.233 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:17:08.233 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:14:39.849669+0000", 2024-09-11T08:17:08.234 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.080784979000000007 2024-09-11T08:17:08.234 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.234 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.234 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:17:08.234 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:14:39.849680+0000", 2024-09-11T08:17:08.234 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0733e-05 2024-09-11T08:17:08.234 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.234 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.234 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:17:08.234 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:14:39.851499+0000", 2024-09-11T08:17:08.234 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.001818769 2024-09-11T08:17:08.234 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.234 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.234 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:17:08.234 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:14:39.851540+0000", 2024-09-11T08:17:08.235 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1749999999999998e-05 2024-09-11T08:17:08.235 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.235 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.235 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:17:08.235 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:14:39.856909+0000", 2024-09-11T08:17:08.235 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0053682629999999999 2024-09-11T08:17:08.235 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:08.235 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:08.235 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:17:08.235 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:14:39.856925+0000", 2024-09-11T08:17:08.235 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.6274999999999999e-05 2024-09-11T08:17:08.235 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:08.235 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:08.235 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:08.235 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:08.236 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:08.236 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-11T08:17:08.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:08.257 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:08.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:08.359 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:08.397 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:08.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:08.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:08.499 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:08.533 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:08.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:08.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:08.573 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:17:08 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:17:08.635 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:08.671 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:08.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:08.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:08.773 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:08.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:08.803 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:08.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:08.904 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:08.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:08.936 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:08.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:08.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:08 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:17:08.988 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:17:08 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:17:08] "GET /metrics HTTP/1.1" 200 224229 "" "Prometheus/2.43.0" 2024-09-11T08:17:08.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:08 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:17:09.037 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:09.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:09.069 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:09.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:09.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:08 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:17:09.171 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:17:09.201 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:09.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:09.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:17:09.303 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:17:09.335 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:09.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:09.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:17:09.437 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:09.466 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:09.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:09.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:09.568 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:09.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:09.603 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:09.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:09.660 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-11T08:17:09.704 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:09.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:09.734 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:09.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:09.835 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:09.857 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:09.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:09.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:09.959 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:09.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:09 smithi150 ceph-mon[39612]: pgmap v427: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 1.9 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-09-11T08:17:09.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:09 smithi150 ceph-mon[29276]: pgmap v427: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 1.9 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-09-11T08:17:09.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:09.993 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:09.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:10.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:09 smithi181 ceph-mon[32701]: pgmap v427: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 1.9 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-09-11T08:17:10.095 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:10.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:10.125 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:10.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:10.226 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:10.252 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:10.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:10.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:10.354 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:10.386 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:10.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:10.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:10.488 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:17:10.523 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:10.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:10.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:17:10.625 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:17:10.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:10.661 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:10.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:17:10.762 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:10.785 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:10.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:10.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:10.887 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:10.924 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:10.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:10.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:11.026 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:11.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:11.060 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:11.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:11.161 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:17:11.192 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:11.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:11.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:17:11.294 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:11.331 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:11.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:11.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:11.429 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:17:11.433 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:11.463 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:11.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:11.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:11.490 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:17:11.564 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:17:11.597 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:11.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:11.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:17:11.699 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:17:11.737 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:11.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:11.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:17:11.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:11 smithi150 ceph-mon[29276]: osdmap e227: 8 total, 8 up, 8 in 2024-09-11T08:17:11.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:11 smithi150 ceph-mon[29276]: pgmap v429: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 1.9 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 840 B/s rd, 0 B/s wr, 816 op/s 2024-09-11T08:17:11.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:11 smithi150 ceph-mon[39612]: osdmap e227: 8 total, 8 up, 8 in 2024-09-11T08:17:11.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:11 smithi150 ceph-mon[39612]: pgmap v429: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 1.9 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 840 B/s rd, 0 B/s wr, 816 op/s 2024-09-11T08:17:11.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:11 smithi181 ceph-mon[32701]: osdmap e227: 8 total, 8 up, 8 in 2024-09-11T08:17:11.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:11 smithi181 ceph-mon[32701]: pgmap v429: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 1.9 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 840 B/s rd, 0 B/s wr, 816 op/s 2024-09-11T08:17:11.839 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:11.876 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:11.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:11.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:11.978 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:12.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:12.014 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:12.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:12.114 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:17:12.147 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:12.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:12.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:17:12.249 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:12.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:12.285 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:12.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:12.386 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:12.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:12.417 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:12.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:12.518 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:17:12.560 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:12.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:12.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:17:12.662 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:12.694 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:12.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:12.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:12.795 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:17:12.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:12.833 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:12.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:17:12.934 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:12.959 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:12.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:12.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:13.061 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:13.096 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:13.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:13.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:13.198 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:13.224 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:13.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:13.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:13.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:12 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2734008652' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:17:13.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:12 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2734008652' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:17:13.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:12 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2734008652' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:17:13.325 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:17:13.363 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:13.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:13.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:17:13.465 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:13.498 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:13.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:13.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:13.600 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:13.635 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:13.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:13.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:13.736 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:13.772 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:13.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:13.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:13.873 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:17:13.899 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:13.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:13.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:17:14.001 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:14.032 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:14.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:14.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:14.133 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:14.169 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:14.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:14.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:14.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:13 smithi150 ceph-mon[29276]: pgmap v430: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 1.9 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 747 op/s 2024-09-11T08:17:14.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:13 smithi150 ceph-mon[39612]: pgmap v430: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 1.9 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 747 op/s 2024-09-11T08:17:14.271 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:14.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:14.302 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:14.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:14.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:13 smithi181 ceph-mon[32701]: pgmap v430: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 1.9 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 747 op/s 2024-09-11T08:17:14.403 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:14.434 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:14.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:14.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:14.535 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:14.565 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:14.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:14.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:14.667 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:14.706 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:14.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:14.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:14.806 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:17:14.806 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:17:14.807 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:17:14.807 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:17:14.807 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:17:14.807 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:17:14.807 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:17:14.807 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:17:14.807 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:17:14.807 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:17:14.807 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:17:14.807 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:17:14.808 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:17:14.808 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:17:14.808 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:17:14.808 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:17:14.808 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:14.849 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:14.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:14.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:14.951 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:17:14.986 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:14.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:14.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:17:15.088 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:17:15.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:15.123 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:15.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:17:15.224 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:15.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:14 smithi150 ceph-mon[29276]: 8.e deep-scrub ok 2024-09-11T08:17:15.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:14 smithi150 ceph-mon[39612]: 8.e deep-scrub ok 2024-09-11T08:17:15.251 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:15.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:15.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:15.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:14 smithi181 ceph-mon[32701]: 8.e deep-scrub ok 2024-09-11T08:17:15.323 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:17:15 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:17:15] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:17:15.353 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:15.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:15.391 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:15.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:15.492 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:17:15.527 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:15.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:15.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:17:15.628 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:15.666 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:15.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:15.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:15.768 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:17:15.795 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:15.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:15.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:17:15.897 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:17:15.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:15.927 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:15.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:17:16.028 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:16.061 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:16.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:16.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:16.163 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:16.203 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:16.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:16.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:16.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:15 smithi150 ceph-mon[29276]: pgmap v431: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 1.6 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.09k op/s 2024-09-11T08:17:16.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:15 smithi150 ceph-mon[29276]: osdmap e228: 8 total, 8 up, 8 in 2024-09-11T08:17:16.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:15 smithi150 ceph-mon[39612]: pgmap v431: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 1.6 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.09k op/s 2024-09-11T08:17:16.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:15 smithi150 ceph-mon[39612]: osdmap e228: 8 total, 8 up, 8 in 2024-09-11T08:17:16.305 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:17:16.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:15 smithi181 ceph-mon[32701]: pgmap v431: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 1.6 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.09k op/s 2024-09-11T08:17:16.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:15 smithi181 ceph-mon[32701]: osdmap e228: 8 total, 8 up, 8 in 2024-09-11T08:17:16.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:16.341 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:16.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:17:16.442 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:16.473 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:16.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:16.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:16.574 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:16.600 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:16.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:16.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:16.702 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:16.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:16.740 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:16.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:16.841 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:17:16.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:16.873 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:16.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:17:16.974 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:17.002 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:17.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:17.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:17.104 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:17.136 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:17.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:17.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:17.238 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:17:17.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:17.267 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:17.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:17:17.368 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:17.400 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:17.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:17.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:17.502 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:17:17.543 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:17.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:17.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:17:17.645 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:17.670 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:17.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:17.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:17.772 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:17.802 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:17.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:17.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:17.804 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-11T08:17:17.804 INFO:teuthology.orchestra.run.smithi150.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-11T08:17:17.804 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-11T08:17:17.903 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:17.931 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:17.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:17.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:17.985 INFO:teuthology.orchestra.run.smithi150.stderr:nodeep-scrub is set 2024-09-11T08:17:18.032 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:17:18.082 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:18.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:18.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:17:18.184 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:17:18.221 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:18.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:18.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:17:18.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:17 smithi150 ceph-mon[29276]: pgmap v433: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 1.6 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 659 op/s 2024-09-11T08:17:18.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:17 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1692159963' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:17:18.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:17 smithi150 ceph-mon[39612]: pgmap v433: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 1.6 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 659 op/s 2024-09-11T08:17:18.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:17 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1692159963' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:17:18.251 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:17 smithi181 ceph-mon[32701]: pgmap v433: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 1.6 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 659 op/s 2024-09-11T08:17:18.251 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:17 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1692159963' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:17:18.323 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:18.351 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:18.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:18.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:18.453 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:17:18.488 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:18.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:18.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:17:18.573 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:17:18 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:17:18.590 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:18.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:18.626 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:18.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:18.727 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:18.738 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:17:18 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:17:18] "GET /metrics HTTP/1.1" 200 224229 "" "Prometheus/2.43.0" 2024-09-11T08:17:18.763 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:18.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:18.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:18.865 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:17:18.903 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:18.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:18.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:17:18.992 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T08:17:19.005 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:17:19.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:19.082 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:19.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:17:19.183 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:19.210 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:19.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:19.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:19.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:18 smithi150 ceph-mon[29276]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:17:19.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:18 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1692159963' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:17:19.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:18 smithi150 ceph-mon[29276]: osdmap e229: 8 total, 8 up, 8 in 2024-09-11T08:17:19.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:18 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:17:19.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:18 smithi150 ceph-mon[39612]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:17:19.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:18 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1692159963' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:17:19.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:18 smithi150 ceph-mon[39612]: osdmap e229: 8 total, 8 up, 8 in 2024-09-11T08:17:19.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:18 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:17:19.312 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:19.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:18 smithi181 ceph-mon[32701]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:17:19.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:18 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1692159963' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:17:19.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:18 smithi181 ceph-mon[32701]: osdmap e229: 8 total, 8 up, 8 in 2024-09-11T08:17:19.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:18 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:17:19.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:19.345 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:19.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:19.446 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:17:19.478 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:19.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:19.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:17:19.545 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:17:19.580 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:17:19.613 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:19.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:19.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:17:19.715 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:17:19.737 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:19.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:19.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:17:19.839 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:17:19.872 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:19.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:19.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:17:19.974 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:20.002 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:20.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:20.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:20.104 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:20.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:20.142 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:20.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:20.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:19 smithi150 ceph-mon[29276]: pgmap v435: 148 pgs: 148 active+clean; 1.3 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.34k op/s 2024-09-11T08:17:20.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:19 smithi150 ceph-mon[39612]: pgmap v435: 148 pgs: 148 active+clean; 1.3 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.34k op/s 2024-09-11T08:17:20.243 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:20.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:20.275 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:20.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:20.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:19 smithi181 ceph-mon[32701]: pgmap v435: 148 pgs: 148 active+clean; 1.3 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.34k op/s 2024-09-11T08:17:20.376 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:20.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:20.414 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:20.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:20.515 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:20.552 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:20.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:20.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:20.654 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:20.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:20.690 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:20.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:20.791 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:17:20.821 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:20.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:20.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:17:20.923 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:20.951 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:20.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:20.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:21.053 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:21.080 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:21.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:21.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:21.142 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:21 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:17:21.142 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:21 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:17:21.142 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:21 smithi150 ceph-mon[29276]: osdmap e230: 8 total, 8 up, 8 in 2024-09-11T08:17:21.142 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:21 smithi150 ceph-mon[29276]: osdmap e231: 8 total, 8 up, 8 in 2024-09-11T08:17:21.142 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:21 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:17:21.142 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:21 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:17:21.143 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:21 smithi150 ceph-mon[39612]: osdmap e230: 8 total, 8 up, 8 in 2024-09-11T08:17:21.143 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:21 smithi150 ceph-mon[39612]: osdmap e231: 8 total, 8 up, 8 in 2024-09-11T08:17:21.182 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:21.200 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:17:21.234 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:21.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:21.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:21.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:21 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:17:21.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:21 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:17:21.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:21 smithi181 ceph-mon[32701]: osdmap e230: 8 total, 8 up, 8 in 2024-09-11T08:17:21.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:21 smithi181 ceph-mon[32701]: osdmap e231: 8 total, 8 up, 8 in 2024-09-11T08:17:21.335 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:21.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:21.367 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:21.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:21.468 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:21.508 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:21.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:21.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:21.580 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T08:17:21.609 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:21.637 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:21.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:21.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:21.739 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:17:21.779 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:21.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:21.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:17:21.881 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:21.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:21.924 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:21.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:22.026 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:22.053 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:22.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:22.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:22.155 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:22.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:22.195 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:22.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:22.296 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:22.322 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:22.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:22.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:22.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:22 smithi181 ceph-mon[32701]: pgmap v438: 148 pgs: 148 active+clean; 1.3 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 954 B/s rd, 0 B/s wr, 1.02k op/s 2024-09-11T08:17:22.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:22 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:17:22.330 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:22 smithi150 ceph-mon[29276]: pgmap v438: 148 pgs: 148 active+clean; 1.3 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 954 B/s rd, 0 B/s wr, 1.02k op/s 2024-09-11T08:17:22.330 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:22 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:17:22.331 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:22 smithi150 ceph-mon[39612]: pgmap v438: 148 pgs: 148 active+clean; 1.3 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 954 B/s rd, 0 B/s wr, 1.02k op/s 2024-09-11T08:17:22.331 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:22 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:17:22.376 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-09-11T08:17:22.376 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:17:22.424 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:22.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:22.465 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:22.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:22.566 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:17:22.597 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:22.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:22.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:17:22.699 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:22.734 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:22.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:22.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:22.837 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:17:22.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:22.880 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:22.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:17:22.981 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:23.014 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:23.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:23.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:23.116 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:17:23.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:23.158 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:23.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:17:23.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:23 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:17:23.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:23 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:17:23.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:23 smithi150 ceph-mon[29276]: osdmap e232: 8 total, 8 up, 8 in 2024-09-11T08:17:23.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:23 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:17:23.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:23 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:17:23.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:23 smithi150 ceph-mon[39612]: osdmap e232: 8 total, 8 up, 8 in 2024-09-11T08:17:23.259 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:23.289 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:23.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:23.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:23.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:23 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:17:23.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:23 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:17:23.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:23 smithi181 ceph-mon[32701]: osdmap e232: 8 total, 8 up, 8 in 2024-09-11T08:17:23.391 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:17:23.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:23.440 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:23.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:17:23.541 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:17:23.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:23.574 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:23.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:17:23.675 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:23.699 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:23.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:23.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:23.801 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:23.826 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:23.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:23.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:23.928 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:23.962 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:23.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:23.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:23.964 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-11T08:17:23.964 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T08:17:23.965 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-11T08:17:23.965 INFO:tasks.thrashosds.thrasher:Setting osd 3 primary_affinity to 0.611128 2024-09-11T08:17:23.965 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 3 0.611127613160871 2024-09-11T08:17:24.063 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:24.132 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:24.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:24.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:24.152 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-11T08:17:24.165 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:17:24.234 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:17:24.295 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:24 smithi150 ceph-mon[29276]: pgmap v440: 148 pgs: 148 active+clean; 1.2 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 924 op/s 2024-09-11T08:17:24.295 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:24 smithi150 ceph-mon[39612]: pgmap v440: 148 pgs: 148 active+clean; 1.2 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 924 op/s 2024-09-11T08:17:24.305 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:24.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:24.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:17:24.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:24 smithi181 ceph-mon[32701]: pgmap v440: 148 pgs: 148 active+clean; 1.2 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 924 op/s 2024-09-11T08:17:24.408 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:24.431 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:24.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:24.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:24.532 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:24.566 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:24.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:24.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:24.668 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:17:24.700 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:24.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:24.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:17:24.802 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:24.835 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:24.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:24.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:24.936 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:24.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:24.972 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:24.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:24.983 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:17:24.983 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:17:25.014 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:17:25.015 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:17:25.044 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:17:25.044 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:17:25.073 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:25.076 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:17:25.076 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:17:25.077 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:25 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2663964309' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:17:25.077 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:25 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2663964309' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:17:25.104 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:25.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:25.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:25.114 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:17:25.115 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:17:25.152 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:17:25.153 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:17:25.179 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:17:25.179 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:17:25.206 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:17:25.211 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:17:25.211 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:17:25.247 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:25.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:25.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:17:25.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:25 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2663964309' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:17:25.324 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:17:25 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:17:25] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:17:25.349 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:25.379 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:25.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:25.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:25.481 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:25.517 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:25.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:25.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:25.619 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:17:25.652 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:25.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:25.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:17:25.754 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:17:25.784 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:25.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:25.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:17:25.886 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:25.919 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:25.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:25.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:26.021 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:26.049 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:26.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:26.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:26.151 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:26.186 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:26.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:26.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:26.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:26 smithi150 ceph-mon[29276]: pgmap v441: 148 pgs: 148 active+clean; 940 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.01k op/s 2024-09-11T08:17:26.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:26 smithi150 ceph-mon[29276]: osdmap e233: 8 total, 8 up, 8 in 2024-09-11T08:17:26.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:26 smithi150 ceph-mon[39612]: pgmap v441: 148 pgs: 148 active+clean; 940 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.01k op/s 2024-09-11T08:17:26.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:26 smithi150 ceph-mon[39612]: osdmap e233: 8 total, 8 up, 8 in 2024-09-11T08:17:26.288 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:26.323 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:26.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:26 smithi181 ceph-mon[32701]: pgmap v441: 148 pgs: 148 active+clean; 940 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.01k op/s 2024-09-11T08:17:26.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:26 smithi181 ceph-mon[32701]: osdmap e233: 8 total, 8 up, 8 in 2024-09-11T08:17:26.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:26.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:26.425 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:26.478 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:26.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:26.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:26.580 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:26.608 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:26.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:26.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:26.710 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:26.746 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:26.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:26.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:26.848 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:26.880 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:26.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:26.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:26.982 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:27.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:27.027 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:27.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:27.128 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:27.157 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:27.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:27.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:27.259 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:17:27.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:27.290 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:27.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:17:27.391 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:17:27.415 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:27.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:27.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:17:27.421 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:17:27.518 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:27.549 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:27.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:27.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:27.651 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:17:27.683 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:27.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:27.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:17:27.785 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:27.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:27.823 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:27.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:27.924 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:17:27.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:27.960 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:27.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:17:28.061 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:28.106 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:28.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:28.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:28.208 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:28.242 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:28 smithi150 ceph-mon[29276]: pgmap v443: 148 pgs: 148 active+clean; 940 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 804 B/s rd, 0 B/s wr, 954 op/s 2024-09-11T08:17:28.243 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:28 smithi150 ceph-mon[39612]: pgmap v443: 148 pgs: 148 active+clean; 940 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 804 B/s rd, 0 B/s wr, 954 op/s 2024-09-11T08:17:28.255 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:28.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:28.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:28.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:28 smithi181 ceph-mon[32701]: pgmap v443: 148 pgs: 148 active+clean; 940 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 804 B/s rd, 0 B/s wr, 954 op/s 2024-09-11T08:17:28.324 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:17:28 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:17:28.357 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:28.398 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:28.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:28.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:28.500 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:28.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:28.546 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:28.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:28.647 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:28.685 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:28.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:28.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:28.687 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:17:28.738 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:17:28 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:17:28] "GET /metrics HTTP/1.1" 200 224246 "" "Prometheus/2.43.0" 2024-09-11T08:17:28.787 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:17:28.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:28.825 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:28.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:17:28.926 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:28.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:28.958 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:28.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:29.059 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:29.072 INFO:teuthology.orchestra.run.smithi150.stderr:noscrub is unset 2024-09-11T08:17:29.087 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:29 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:17:29.087 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:29 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/185093879' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:17:29.087 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:29 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:17:29.088 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:29 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:17:29.088 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:29 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/185093879' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:17:29.088 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:29 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:17:29.088 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:29.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:29.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:29.190 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:17:29.225 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:29.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:29.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:17:29.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:29 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:17:29.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:29 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/185093879' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:17:29.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:29 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:17:29.327 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:29.359 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:29.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:29.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:29.418 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T08:17:29.461 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:29.503 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:29.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:29.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:29.605 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:29.632 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:29.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:29.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:29.733 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:29.762 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:29.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:29.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:29.864 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:17:29.891 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:29.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:29.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:17:29.992 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:30.023 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:30.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:30.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:30.125 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:30.157 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:30.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:30.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:30.259 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:30.293 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:30.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:30.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:30.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:30 smithi181 ceph-mon[32701]: pgmap v444: 148 pgs: 148 active+clean; 563 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.51k op/s 2024-09-11T08:17:30.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:30 smithi181 ceph-mon[32701]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:17:30.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:30 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:17:30.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:30 smithi181 ceph-mon[32701]: osdmap e234: 8 total, 8 up, 8 in 2024-09-11T08:17:30.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:30 smithi181 ceph-mon[32701]: 3.11 scrub starts 2024-09-11T08:17:30.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:30 smithi181 ceph-mon[32701]: 3.11 scrub ok 2024-09-11T08:17:30.395 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:30.432 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:30.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:30.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:30.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:30 smithi150 ceph-mon[29276]: pgmap v444: 148 pgs: 148 active+clean; 563 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.51k op/s 2024-09-11T08:17:30.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:30 smithi150 ceph-mon[29276]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:17:30.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:30 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:17:30.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:30 smithi150 ceph-mon[29276]: osdmap e234: 8 total, 8 up, 8 in 2024-09-11T08:17:30.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:30 smithi150 ceph-mon[29276]: 3.11 scrub starts 2024-09-11T08:17:30.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:30 smithi150 ceph-mon[29276]: 3.11 scrub ok 2024-09-11T08:17:30.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:30 smithi150 ceph-mon[39612]: pgmap v444: 148 pgs: 148 active+clean; 563 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.51k op/s 2024-09-11T08:17:30.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:30 smithi150 ceph-mon[39612]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:17:30.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:30 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:17:30.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:30 smithi150 ceph-mon[39612]: osdmap e234: 8 total, 8 up, 8 in 2024-09-11T08:17:30.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:30 smithi150 ceph-mon[39612]: 3.11 scrub starts 2024-09-11T08:17:30.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:30 smithi150 ceph-mon[39612]: 3.11 scrub ok 2024-09-11T08:17:30.534 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:17:30.561 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:30.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:30.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:17:30.664 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:17:30.704 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:30.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:30.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:17:30.806 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:30.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:30.836 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:30.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:30.937 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:30.972 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:30.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:30.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:31.074 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:17:31.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:31.111 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:31.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:17:31.212 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:17:31.248 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:31.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:31.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:17:31.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:31 smithi181 ceph-mon[32701]: 8.6 scrub starts 2024-09-11T08:17:31.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:31 smithi181 ceph-mon[32701]: 6.1e scrub starts 2024-09-11T08:17:31.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:31 smithi181 ceph-mon[32701]: 6.1e scrub ok 2024-09-11T08:17:31.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:31 smithi181 ceph-mon[32701]: 4.1f scrub starts 2024-09-11T08:17:31.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:31 smithi181 ceph-mon[32701]: 4.1f scrub ok 2024-09-11T08:17:31.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:31 smithi181 ceph-mon[32701]: osdmap e235: 8 total, 8 up, 8 in 2024-09-11T08:17:31.349 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:31.383 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:31.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:31.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:31.485 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:31.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:31 smithi150 ceph-mon[29276]: 8.6 scrub starts 2024-09-11T08:17:31.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:31 smithi150 ceph-mon[29276]: 6.1e scrub starts 2024-09-11T08:17:31.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:31 smithi150 ceph-mon[29276]: 6.1e scrub ok 2024-09-11T08:17:31.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:31 smithi150 ceph-mon[29276]: 4.1f scrub starts 2024-09-11T08:17:31.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:31 smithi150 ceph-mon[29276]: 4.1f scrub ok 2024-09-11T08:17:31.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:31 smithi150 ceph-mon[29276]: osdmap e235: 8 total, 8 up, 8 in 2024-09-11T08:17:31.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:31 smithi150 ceph-mon[39612]: 8.6 scrub starts 2024-09-11T08:17:31.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:31 smithi150 ceph-mon[39612]: 6.1e scrub starts 2024-09-11T08:17:31.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:31 smithi150 ceph-mon[39612]: 6.1e scrub ok 2024-09-11T08:17:31.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:31 smithi150 ceph-mon[39612]: 4.1f scrub starts 2024-09-11T08:17:31.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:31 smithi150 ceph-mon[39612]: 4.1f scrub ok 2024-09-11T08:17:31.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:31 smithi150 ceph-mon[39612]: osdmap e235: 8 total, 8 up, 8 in 2024-09-11T08:17:31.512 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:31.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:31.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:31.614 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:17:31.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:31.649 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:31.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:17:31.750 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:17:31.772 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:31.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:31.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:17:31.874 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:17:31.918 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:31.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:31.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:17:32.020 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:32.043 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:32.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:32.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:32.145 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:32.181 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:32.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:32.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:32.283 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:32.316 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:32.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:32.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:32.418 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:32.454 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:32.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:32.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:32.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:32 smithi150 ceph-mon[29276]: 8.6 scrub ok 2024-09-11T08:17:32.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:32 smithi150 ceph-mon[29276]: 5.1d scrub starts 2024-09-11T08:17:32.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:32 smithi150 ceph-mon[29276]: 5.1d scrub ok 2024-09-11T08:17:32.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:32 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:17:32.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:32 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:17:32.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:32 smithi150 ceph-mon[29276]: pgmap v447: 148 pgs: 148 active+clean; 563 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.00k op/s 2024-09-11T08:17:32.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:32 smithi150 ceph-mon[29276]: 4.16 scrub starts 2024-09-11T08:17:32.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:32 smithi150 ceph-mon[29276]: 4.16 scrub ok 2024-09-11T08:17:32.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:32 smithi150 ceph-mon[29276]: 3.14 scrub starts 2024-09-11T08:17:32.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:32 smithi150 ceph-mon[29276]: 3.14 scrub ok 2024-09-11T08:17:32.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:32 smithi150 ceph-mon[29276]: 8.9 scrub starts 2024-09-11T08:17:32.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:32 smithi150 ceph-mon[29276]: 8.9 scrub ok 2024-09-11T08:17:32.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:32 smithi150 ceph-mon[29276]: osdmap e236: 8 total, 8 up, 8 in 2024-09-11T08:17:32.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:32 smithi150 ceph-mon[39612]: 8.6 scrub ok 2024-09-11T08:17:32.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:32 smithi150 ceph-mon[39612]: 5.1d scrub starts 2024-09-11T08:17:32.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:32 smithi150 ceph-mon[39612]: 5.1d scrub ok 2024-09-11T08:17:32.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:32 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:17:32.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:32 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:17:32.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:32 smithi150 ceph-mon[39612]: pgmap v447: 148 pgs: 148 active+clean; 563 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.00k op/s 2024-09-11T08:17:32.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:32 smithi150 ceph-mon[39612]: 4.16 scrub starts 2024-09-11T08:17:32.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:32 smithi150 ceph-mon[39612]: 4.16 scrub ok 2024-09-11T08:17:32.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:32 smithi150 ceph-mon[39612]: 3.14 scrub starts 2024-09-11T08:17:32.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:32 smithi150 ceph-mon[39612]: 3.14 scrub ok 2024-09-11T08:17:32.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:32 smithi150 ceph-mon[39612]: 8.9 scrub starts 2024-09-11T08:17:32.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:32 smithi150 ceph-mon[39612]: 8.9 scrub ok 2024-09-11T08:17:32.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:32 smithi150 ceph-mon[39612]: osdmap e236: 8 total, 8 up, 8 in 2024-09-11T08:17:32.556 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:32.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:32 smithi181 ceph-mon[32701]: 8.6 scrub ok 2024-09-11T08:17:32.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:32 smithi181 ceph-mon[32701]: 5.1d scrub starts 2024-09-11T08:17:32.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:32 smithi181 ceph-mon[32701]: 5.1d scrub ok 2024-09-11T08:17:32.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:32 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:17:32.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:32 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:17:32.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:32 smithi181 ceph-mon[32701]: pgmap v447: 148 pgs: 148 active+clean; 563 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.00k op/s 2024-09-11T08:17:32.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:32 smithi181 ceph-mon[32701]: 4.16 scrub starts 2024-09-11T08:17:32.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:32 smithi181 ceph-mon[32701]: 4.16 scrub ok 2024-09-11T08:17:32.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:32 smithi181 ceph-mon[32701]: 3.14 scrub starts 2024-09-11T08:17:32.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:32 smithi181 ceph-mon[32701]: 3.14 scrub ok 2024-09-11T08:17:32.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:32 smithi181 ceph-mon[32701]: 8.9 scrub starts 2024-09-11T08:17:32.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:32 smithi181 ceph-mon[32701]: 8.9 scrub ok 2024-09-11T08:17:32.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:32 smithi181 ceph-mon[32701]: osdmap e236: 8 total, 8 up, 8 in 2024-09-11T08:17:32.577 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:32.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:32.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:32.679 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:17:32.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:32.714 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:32.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:17:32.816 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:32.858 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:32.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:32.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:32.960 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:32.998 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:32.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:32.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:33.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:33 smithi150 ceph-mon[29276]: 6.6 scrub starts 2024-09-11T08:17:33.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:33 smithi150 ceph-mon[29276]: 6.6 scrub ok 2024-09-11T08:17:33.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:33 smithi150 ceph-mon[29276]: 6.12 scrub starts 2024-09-11T08:17:33.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:33 smithi150 ceph-mon[29276]: 6.12 scrub ok 2024-09-11T08:17:33.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:33 smithi150 ceph-mon[29276]: 3.1b scrub starts 2024-09-11T08:17:33.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:33 smithi150 ceph-mon[29276]: 3.1b scrub ok 2024-09-11T08:17:33.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:33 smithi150 ceph-mon[29276]: 6.f scrub starts 2024-09-11T08:17:33.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:33 smithi150 ceph-mon[29276]: 6.f scrub ok 2024-09-11T08:17:33.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:33 smithi150 ceph-mon[29276]: 4.a scrub starts 2024-09-11T08:17:33.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:33 smithi150 ceph-mon[29276]: 4.a scrub ok 2024-09-11T08:17:33.094 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:33 smithi150 ceph-mon[39612]: 6.6 scrub starts 2024-09-11T08:17:33.100 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:17:33.147 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:33.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:33.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:17:33.165 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T08:17:33.249 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:17:33.285 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:33.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:33.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:17:33.387 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:33.428 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:33.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:33.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:33.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:33 smithi150 ceph-mon[39612]: 6.6 scrub ok 2024-09-11T08:17:33.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:33 smithi150 ceph-mon[39612]: 6.12 scrub starts 2024-09-11T08:17:33.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:33 smithi150 ceph-mon[39612]: 6.12 scrub ok 2024-09-11T08:17:33.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:33 smithi150 ceph-mon[39612]: 3.1b scrub starts 2024-09-11T08:17:33.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:33 smithi150 ceph-mon[39612]: 3.1b scrub ok 2024-09-11T08:17:33.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:33 smithi150 ceph-mon[39612]: 6.f scrub starts 2024-09-11T08:17:33.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:33 smithi150 ceph-mon[39612]: 6.f scrub ok 2024-09-11T08:17:33.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:33 smithi150 ceph-mon[39612]: 4.a scrub starts 2024-09-11T08:17:33.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:33 smithi150 ceph-mon[39612]: 4.a scrub ok 2024-09-11T08:17:33.530 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:33.556 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:33.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:33.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:33.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:33 smithi181 ceph-mon[32701]: 6.6 scrub starts 2024-09-11T08:17:33.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:33 smithi181 ceph-mon[32701]: 6.6 scrub ok 2024-09-11T08:17:33.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:33 smithi181 ceph-mon[32701]: 6.12 scrub starts 2024-09-11T08:17:33.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:33 smithi181 ceph-mon[32701]: 6.12 scrub ok 2024-09-11T08:17:33.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:33 smithi181 ceph-mon[32701]: 3.1b scrub starts 2024-09-11T08:17:33.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:33 smithi181 ceph-mon[32701]: 3.1b scrub ok 2024-09-11T08:17:33.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:33 smithi181 ceph-mon[32701]: 6.f scrub starts 2024-09-11T08:17:33.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:33 smithi181 ceph-mon[32701]: 6.f scrub ok 2024-09-11T08:17:33.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:33 smithi181 ceph-mon[32701]: 4.a scrub starts 2024-09-11T08:17:33.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:33 smithi181 ceph-mon[32701]: 4.a scrub ok 2024-09-11T08:17:33.659 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:17:33.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:33.699 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:33.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:17:33.799 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:33.831 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:33.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:33.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:33.933 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:33.963 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:33.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:33.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:34.065 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:17:34.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:34.102 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:34.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:17:34.203 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:17:34.227 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:34.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:34.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:17:34.328 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:17:34.360 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:34.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:34.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:17:34.369 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:34 smithi150 ceph-mon[29276]: 8.7 scrub starts 2024-09-11T08:17:34.369 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:34 smithi150 ceph-mon[29276]: 8.7 scrub ok 2024-09-11T08:17:34.369 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:34 smithi150 ceph-mon[29276]: pgmap v449: 148 pgs: 148 active+clean; 495 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.19k op/s 2024-09-11T08:17:34.369 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:34 smithi150 ceph-mon[29276]: 6.18 scrub starts 2024-09-11T08:17:34.369 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:34 smithi150 ceph-mon[29276]: 6.18 scrub ok 2024-09-11T08:17:34.369 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:34 smithi150 ceph-mon[29276]: 8.2 scrub starts 2024-09-11T08:17:34.369 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:34 smithi150 ceph-mon[29276]: 8.2 scrub ok 2024-09-11T08:17:34.370 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:34 smithi150 ceph-mon[29276]: 5.a scrub starts 2024-09-11T08:17:34.370 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:34 smithi150 ceph-mon[29276]: 5.a scrub ok 2024-09-11T08:17:34.370 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:34 smithi150 ceph-mon[39612]: 8.7 scrub starts 2024-09-11T08:17:34.370 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:34 smithi150 ceph-mon[39612]: 8.7 scrub ok 2024-09-11T08:17:34.370 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:34 smithi150 ceph-mon[39612]: pgmap v449: 148 pgs: 148 active+clean; 495 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.19k op/s 2024-09-11T08:17:34.370 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:34 smithi150 ceph-mon[39612]: 6.18 scrub starts 2024-09-11T08:17:34.370 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:34 smithi150 ceph-mon[39612]: 6.18 scrub ok 2024-09-11T08:17:34.370 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:34 smithi150 ceph-mon[39612]: 8.2 scrub starts 2024-09-11T08:17:34.370 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:34 smithi150 ceph-mon[39612]: 8.2 scrub ok 2024-09-11T08:17:34.370 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:34 smithi150 ceph-mon[39612]: 5.a scrub starts 2024-09-11T08:17:34.370 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:34 smithi150 ceph-mon[39612]: 5.a scrub ok 2024-09-11T08:17:34.462 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:17:34.501 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:34.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:34.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:17:34.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:34 smithi181 ceph-mon[32701]: 8.7 scrub starts 2024-09-11T08:17:34.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:34 smithi181 ceph-mon[32701]: 8.7 scrub ok 2024-09-11T08:17:34.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:34 smithi181 ceph-mon[32701]: pgmap v449: 148 pgs: 148 active+clean; 495 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.19k op/s 2024-09-11T08:17:34.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:34 smithi181 ceph-mon[32701]: 6.18 scrub starts 2024-09-11T08:17:34.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:34 smithi181 ceph-mon[32701]: 6.18 scrub ok 2024-09-11T08:17:34.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:34 smithi181 ceph-mon[32701]: 8.2 scrub starts 2024-09-11T08:17:34.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:34 smithi181 ceph-mon[32701]: 8.2 scrub ok 2024-09-11T08:17:34.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:34 smithi181 ceph-mon[32701]: 5.a scrub starts 2024-09-11T08:17:34.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:34 smithi181 ceph-mon[32701]: 5.a scrub ok 2024-09-11T08:17:34.603 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:34.638 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:34.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:34.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:34.740 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:34.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:34.768 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:34.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:34.869 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:17:34.901 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:34.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:34.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:17:35.003 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:17:35.043 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:35.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:35.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:17:35.145 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:35.182 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:35.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:35.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:35.284 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:35.295 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:35 smithi150 ceph-mon[29276]: 4.8 scrub starts 2024-09-11T08:17:35.295 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:35 smithi150 ceph-mon[29276]: 4.8 scrub ok 2024-09-11T08:17:35.295 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:35 smithi150 ceph-mon[39612]: 4.8 scrub starts 2024-09-11T08:17:35.295 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:35 smithi150 ceph-mon[39612]: 4.8 scrub ok 2024-09-11T08:17:35.306 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:17:35.311 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:35.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:35.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:35.412 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:35.438 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:35.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:35.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:35.515 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Removed 128829 objects 2024-09-11T08:17:35.516 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Clean up completed and total clean up time :114.09 2024-09-11T08:17:35.540 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:35.560 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:35.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:35.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:35.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:35 smithi181 ceph-mon[32701]: 4.8 scrub starts 2024-09-11T08:17:35.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:35 smithi181 ceph-mon[32701]: 4.8 scrub ok 2024-09-11T08:17:35.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:17:35 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:17:35] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:17:35.662 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:17:35.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:35.684 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:35.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:17:35.785 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:35.806 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:35.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:35.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:35.898 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:17:35.908 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:35.929 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:35.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:35.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:36.030 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:36.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:36.052 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:36.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:36.153 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:17:36.175 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:36.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:36.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:17:36.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:36 smithi150 ceph-mon[29276]: 4.15 scrub starts 2024-09-11T08:17:36.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:36 smithi150 ceph-mon[29276]: 4.15 scrub ok 2024-09-11T08:17:36.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:36 smithi150 ceph-mon[29276]: 3.4 scrub starts 2024-09-11T08:17:36.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:36 smithi150 ceph-mon[29276]: 3.4 scrub ok 2024-09-11T08:17:36.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:36 smithi150 ceph-mon[29276]: pgmap v450: 148 pgs: 148 active+clean; 184 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.01k op/s 2024-09-11T08:17:36.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:36 smithi150 ceph-mon[29276]: osdmap e237: 8 total, 8 up, 8 in 2024-09-11T08:17:36.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:36 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/460859653' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.611127613160871}]: dispatch 2024-09-11T08:17:36.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:36 smithi150 ceph-mon[39612]: 4.15 scrub starts 2024-09-11T08:17:36.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:36 smithi150 ceph-mon[39612]: 4.15 scrub ok 2024-09-11T08:17:36.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:36 smithi150 ceph-mon[39612]: 3.4 scrub starts 2024-09-11T08:17:36.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:36 smithi150 ceph-mon[39612]: 3.4 scrub ok 2024-09-11T08:17:36.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:36 smithi150 ceph-mon[39612]: pgmap v450: 148 pgs: 148 active+clean; 184 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.01k op/s 2024-09-11T08:17:36.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:36 smithi150 ceph-mon[39612]: osdmap e237: 8 total, 8 up, 8 in 2024-09-11T08:17:36.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:36 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/460859653' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.611127613160871}]: dispatch 2024-09-11T08:17:36.277 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:36.299 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:36.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:36.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:36.401 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:36.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:36.423 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:36.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:36.524 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:17:36.543 INFO:teuthology.orchestra.run.smithi150.stderr:set osd.3 primary-affinity to 0.611128 (8400502) 2024-09-11T08:17:36.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:36.552 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:36.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:17:36.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:36 smithi181 ceph-mon[32701]: 4.15 scrub starts 2024-09-11T08:17:36.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:36 smithi181 ceph-mon[32701]: 4.15 scrub ok 2024-09-11T08:17:36.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:36 smithi181 ceph-mon[32701]: 3.4 scrub starts 2024-09-11T08:17:36.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:36 smithi181 ceph-mon[32701]: 3.4 scrub ok 2024-09-11T08:17:36.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:36 smithi181 ceph-mon[32701]: pgmap v450: 148 pgs: 148 active+clean; 184 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.01k op/s 2024-09-11T08:17:36.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:36 smithi181 ceph-mon[32701]: osdmap e237: 8 total, 8 up, 8 in 2024-09-11T08:17:36.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:36 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/460859653' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.611127613160871}]: dispatch 2024-09-11T08:17:36.652 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:36.677 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:36.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:36.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:36.779 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:36.805 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:36.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:36.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:36.906 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:36.932 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:36.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:36.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:37.034 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:17:37.056 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:37.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:37.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:17:37.081 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-11T08:17:37.082 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [], 2024-09-11T08:17:37.082 INFO:teuthology.orchestra.run.smithi150.stdout: "complaint_time": 30, 2024-09-11T08:17:37.082 INFO:teuthology.orchestra.run.smithi150.stdout: "num_blocked_ops": 0 2024-09-11T08:17:37.082 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-11T08:17:37.157 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:37.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:37.180 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:37.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:37.281 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:37.303 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:37.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:37.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:37.405 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:37.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:37.428 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:37.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:37.529 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:37.558 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:37.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:37.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:37.660 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:37.683 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:37.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:37.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:37.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:37 smithi150 ceph-mon[39612]: 6.3 scrub starts 2024-09-11T08:17:37.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:37 smithi150 ceph-mon[39612]: 6.3 scrub ok 2024-09-11T08:17:37.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:37 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/460859653' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.611127613160871}]': finished 2024-09-11T08:17:37.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:37 smithi150 ceph-mon[39612]: osdmap e238: 8 total, 8 up, 8 in 2024-09-11T08:17:37.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:37 smithi150 ceph-mon[39612]: 3.1d scrub starts 2024-09-11T08:17:37.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:37 smithi150 ceph-mon[39612]: 3.1d scrub ok 2024-09-11T08:17:37.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:37 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:17:37.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:37 smithi150 ceph-mon[39612]: pgmap v453: 148 pgs: 148 active+clean; 184 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.01k op/s 2024-09-11T08:17:37.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:37 smithi150 ceph-mon[29276]: 6.3 scrub starts 2024-09-11T08:17:37.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:37 smithi150 ceph-mon[29276]: 6.3 scrub ok 2024-09-11T08:17:37.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:37 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/460859653' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.611127613160871}]': finished 2024-09-11T08:17:37.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:37 smithi150 ceph-mon[29276]: osdmap e238: 8 total, 8 up, 8 in 2024-09-11T08:17:37.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:37 smithi150 ceph-mon[29276]: 3.1d scrub starts 2024-09-11T08:17:37.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:37 smithi150 ceph-mon[29276]: 3.1d scrub ok 2024-09-11T08:17:37.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:37 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:17:37.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:37 smithi150 ceph-mon[29276]: pgmap v453: 148 pgs: 148 active+clean; 184 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.01k op/s 2024-09-11T08:17:37.783 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-11T08:17:37.785 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:37.807 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:37.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:37.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:37.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:37 smithi181 ceph-mon[32701]: 6.3 scrub starts 2024-09-11T08:17:37.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:37 smithi181 ceph-mon[32701]: 6.3 scrub ok 2024-09-11T08:17:37.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:37 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/460859653' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.611127613160871}]': finished 2024-09-11T08:17:37.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:37 smithi181 ceph-mon[32701]: osdmap e238: 8 total, 8 up, 8 in 2024-09-11T08:17:37.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:37 smithi181 ceph-mon[32701]: 3.1d scrub starts 2024-09-11T08:17:37.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:37 smithi181 ceph-mon[32701]: 3.1d scrub ok 2024-09-11T08:17:37.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:37 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:17:37.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:37 smithi181 ceph-mon[32701]: pgmap v453: 148 pgs: 148 active+clean; 184 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.01k op/s 2024-09-11T08:17:37.909 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:37.930 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:37.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:37.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:38.032 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:17:38.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:38.055 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:38.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:17:38.156 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:17:38.177 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:38.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:38.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:17:38.279 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:38.300 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:38.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:38.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:38.403 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:17:38.424 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:38.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:38.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:17:38.526 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:17:38.543 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:17:38 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:17:38.548 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:38.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:38.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:17:38.650 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:38.672 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:38.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:38.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:38.774 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:38.796 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:38.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:38.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:38.801 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-09-11T08:17:38.801 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-09-11T08:17:38.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:38 smithi181 ceph-mon[32701]: osdmap e239: 8 total, 8 up, 8 in 2024-09-11T08:17:38.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:38 smithi181 ceph-mon[32701]: 4.2 scrub starts 2024-09-11T08:17:38.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:38 smithi181 ceph-mon[32701]: 4.2 scrub ok 2024-09-11T08:17:38.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:38 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:17:38.825 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:17:38 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:17:38] "GET /metrics HTTP/1.1" 200 224246 "" "Prometheus/2.43.0" 2024-09-11T08:17:38.826 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:38 smithi150 ceph-mon[29276]: osdmap e239: 8 total, 8 up, 8 in 2024-09-11T08:17:38.826 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:38 smithi150 ceph-mon[29276]: 4.2 scrub starts 2024-09-11T08:17:38.826 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:38 smithi150 ceph-mon[29276]: 4.2 scrub ok 2024-09-11T08:17:38.826 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:38 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:17:38.826 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:38 smithi150 ceph-mon[39612]: osdmap e239: 8 total, 8 up, 8 in 2024-09-11T08:17:38.826 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:38 smithi150 ceph-mon[39612]: 4.2 scrub starts 2024-09-11T08:17:38.826 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:38 smithi150 ceph-mon[39612]: 4.2 scrub ok 2024-09-11T08:17:38.826 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:38 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:17:38.899 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:38.921 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:38.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:38.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:39.023 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:17:39.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:39.045 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:39.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:17:39.146 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:39.167 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:39.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:39.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:39.269 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:39.289 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:39.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:39.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:39.391 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:39.412 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:39.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:39.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:39.514 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:39.543 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:39.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:39.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:39.645 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:39.667 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:39.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:39.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:39.678 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:39 smithi150 ceph-mon[29276]: 3.2 scrub starts 2024-09-11T08:17:39.678 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:39 smithi150 ceph-mon[29276]: 3.2 scrub ok 2024-09-11T08:17:39.679 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:39 smithi150 ceph-mon[29276]: pgmap v455: 148 pgs: 148 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 1.44k op/s 2024-09-11T08:17:39.679 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:39 smithi150 ceph-mon[29276]: 5.13 scrub starts 2024-09-11T08:17:39.679 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:39 smithi150 ceph-mon[29276]: 5.13 scrub ok 2024-09-11T08:17:39.679 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:39 smithi150 ceph-mon[29276]: 6.2 scrub starts 2024-09-11T08:17:39.679 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:39 smithi150 ceph-mon[29276]: 6.2 scrub ok 2024-09-11T08:17:39.679 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:39 smithi150 ceph-mon[39612]: 3.2 scrub starts 2024-09-11T08:17:39.679 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:39 smithi150 ceph-mon[39612]: 3.2 scrub ok 2024-09-11T08:17:39.679 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:39 smithi150 ceph-mon[39612]: pgmap v455: 148 pgs: 148 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 1.44k op/s 2024-09-11T08:17:39.679 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:39 smithi150 ceph-mon[39612]: 5.13 scrub starts 2024-09-11T08:17:39.679 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:39 smithi150 ceph-mon[39612]: 5.13 scrub ok 2024-09-11T08:17:39.679 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:39 smithi150 ceph-mon[39612]: 6.2 scrub starts 2024-09-11T08:17:39.680 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:39 smithi150 ceph-mon[39612]: 6.2 scrub ok 2024-09-11T08:17:39.769 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:39.790 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:39.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:39.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:39.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:39 smithi181 ceph-mon[32701]: 3.2 scrub starts 2024-09-11T08:17:39.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:39 smithi181 ceph-mon[32701]: 3.2 scrub ok 2024-09-11T08:17:39.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:39 smithi181 ceph-mon[32701]: pgmap v455: 148 pgs: 148 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 1.44k op/s 2024-09-11T08:17:39.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:39 smithi181 ceph-mon[32701]: 5.13 scrub starts 2024-09-11T08:17:39.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:39 smithi181 ceph-mon[32701]: 5.13 scrub ok 2024-09-11T08:17:39.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:39 smithi181 ceph-mon[32701]: 6.2 scrub starts 2024-09-11T08:17:39.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:39 smithi181 ceph-mon[32701]: 6.2 scrub ok 2024-09-11T08:17:39.892 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:39.915 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:39.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:39.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:40.017 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:17:40.039 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:40.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:40.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:17:40.106 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:17:40.107 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:17:40.122 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:17:40.122 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:17:40.137 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:17:40.137 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:17:40.141 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:40.152 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:17:40.152 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:17:40.163 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:40.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:40.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:40.169 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:17:40.169 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:17:40.184 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:17:40.185 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:17:40.200 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:17:40.200 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:17:40.215 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:17:40.215 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:17:40.265 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:17:40.286 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:40.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:40.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:17:40.388 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:40.411 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:40.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:40.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:40.513 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:17:40.536 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:40.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:40.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:17:40.638 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:17:40.659 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:40.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:40.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:17:40.761 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:17:40.782 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:40.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:40.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:17:40.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:40 smithi181 ceph-mon[32701]: 4.17 scrub starts 2024-09-11T08:17:40.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:40 smithi181 ceph-mon[32701]: 4.17 scrub ok 2024-09-11T08:17:40.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:40 smithi181 ceph-mon[32701]: 6.0 scrub starts 2024-09-11T08:17:40.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:40 smithi181 ceph-mon[32701]: 6.0 scrub ok 2024-09-11T08:17:40.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:40 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/115931712' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:17:40.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:40 smithi181 ceph-mon[32701]: 6.e scrub starts 2024-09-11T08:17:40.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:40 smithi181 ceph-mon[32701]: 6.e scrub ok 2024-09-11T08:17:40.883 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:17:40.904 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:40.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:40.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:17:40.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:40 smithi150 ceph-mon[29276]: 4.17 scrub starts 2024-09-11T08:17:40.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:40 smithi150 ceph-mon[29276]: 4.17 scrub ok 2024-09-11T08:17:40.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:40 smithi150 ceph-mon[29276]: 6.0 scrub starts 2024-09-11T08:17:40.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:40 smithi150 ceph-mon[29276]: 6.0 scrub ok 2024-09-11T08:17:40.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:40 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/115931712' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:17:40.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:40 smithi150 ceph-mon[29276]: 6.e scrub starts 2024-09-11T08:17:40.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:40 smithi150 ceph-mon[29276]: 6.e scrub ok 2024-09-11T08:17:40.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:40 smithi150 ceph-mon[39612]: 4.17 scrub starts 2024-09-11T08:17:40.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:40 smithi150 ceph-mon[39612]: 4.17 scrub ok 2024-09-11T08:17:40.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:40 smithi150 ceph-mon[39612]: 6.0 scrub starts 2024-09-11T08:17:40.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:40 smithi150 ceph-mon[39612]: 6.0 scrub ok 2024-09-11T08:17:40.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:40 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/115931712' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:17:40.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:40 smithi150 ceph-mon[39612]: 6.e scrub starts 2024-09-11T08:17:40.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:40 smithi150 ceph-mon[39612]: 6.e scrub ok 2024-09-11T08:17:41.006 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:41.030 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:41.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:41.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:41.131 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:41.155 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:41.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:41.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:41.257 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:41.279 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:41.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:41.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:41.381 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:17:41.404 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:41.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:41.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:17:41.506 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:17:41.529 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:41.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:41.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:17:41.630 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:41.653 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:41.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:41.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:41.755 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:41.777 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:41.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:41.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:41.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:41 smithi181 ceph-mon[32701]: 5.4 scrub starts 2024-09-11T08:17:41.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:41 smithi181 ceph-mon[32701]: 5.4 scrub ok 2024-09-11T08:17:41.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:41 smithi181 ceph-mon[32701]: pgmap v456: 148 pgs: 148 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 608 op/s 2024-09-11T08:17:41.864 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:41 smithi150 ceph-mon[29276]: 5.4 scrub starts 2024-09-11T08:17:41.864 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:41 smithi150 ceph-mon[29276]: 5.4 scrub ok 2024-09-11T08:17:41.865 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:41 smithi150 ceph-mon[29276]: pgmap v456: 148 pgs: 148 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 608 op/s 2024-09-11T08:17:41.865 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:41 smithi150 ceph-mon[39612]: 5.4 scrub starts 2024-09-11T08:17:41.865 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:41 smithi150 ceph-mon[39612]: 5.4 scrub ok 2024-09-11T08:17:41.865 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:41 smithi150 ceph-mon[39612]: pgmap v456: 148 pgs: 148 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 608 op/s 2024-09-11T08:17:41.879 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:17:41.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:41.903 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:41.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:17:42.004 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:42.027 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:42.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:42.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:42.129 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:42.150 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:42.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:42.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:42.252 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:42.278 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:42.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:42.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:42.332 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:17:42.335 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:17:42.380 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:42.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:42.402 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:42.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:42.503 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:42.524 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:42.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:42.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:42.570 INFO:teuthology.orchestra.run.smithi150.stderr:nodeep-scrub is unset 2024-09-11T08:17:42.627 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:42.652 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:42.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:42.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:42.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:42 smithi150 ceph-mon[29276]: 8.1 scrub starts 2024-09-11T08:17:42.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:42 smithi150 ceph-mon[29276]: 8.1 scrub ok 2024-09-11T08:17:42.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:42 smithi150 ceph-mon[29276]: 6.8 scrub starts 2024-09-11T08:17:42.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:42 smithi150 ceph-mon[29276]: 6.8 scrub ok 2024-09-11T08:17:42.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:42 smithi150 ceph-mon[29276]: 6.16 scrub starts 2024-09-11T08:17:42.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:42 smithi150 ceph-mon[29276]: 6.16 scrub ok 2024-09-11T08:17:42.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:42 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/528750502' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:17:42.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:42 smithi150 ceph-mon[39612]: 8.1 scrub starts 2024-09-11T08:17:42.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:42 smithi150 ceph-mon[39612]: 8.1 scrub ok 2024-09-11T08:17:42.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:42 smithi150 ceph-mon[39612]: 6.8 scrub starts 2024-09-11T08:17:42.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:42 smithi150 ceph-mon[39612]: 6.8 scrub ok 2024-09-11T08:17:42.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:42 smithi150 ceph-mon[39612]: 6.16 scrub starts 2024-09-11T08:17:42.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:42 smithi150 ceph-mon[39612]: 6.16 scrub ok 2024-09-11T08:17:42.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:42 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/528750502' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:17:42.754 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:17:42.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:42.777 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:42.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:17:42.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:42 smithi181 ceph-mon[32701]: 8.1 scrub starts 2024-09-11T08:17:42.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:42 smithi181 ceph-mon[32701]: 8.1 scrub ok 2024-09-11T08:17:42.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:42 smithi181 ceph-mon[32701]: 6.8 scrub starts 2024-09-11T08:17:42.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:42 smithi181 ceph-mon[32701]: 6.8 scrub ok 2024-09-11T08:17:42.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:42 smithi181 ceph-mon[32701]: 6.16 scrub starts 2024-09-11T08:17:42.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:42 smithi181 ceph-mon[32701]: 6.16 scrub ok 2024-09-11T08:17:42.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:42 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/528750502' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:17:42.878 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:42.904 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:42.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:42.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:43.005 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:43.027 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:43.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:43.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:43.129 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:43.155 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:43.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:43.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:43.256 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:43.278 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:43.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:43.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:43.380 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:43.406 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:43.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:43.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:43.508 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:43.533 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:43.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:43.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:43.635 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:43.639 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-11T08:17:43.640 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T08:17:43.640 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-11T08:17:43.640 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-11T08:17:43.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:43.657 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:43.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:43.677 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:43 smithi150 ceph-mon[29276]: 6.c scrub starts 2024-09-11T08:17:43.677 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:43 smithi150 ceph-mon[29276]: 6.c scrub ok 2024-09-11T08:17:43.678 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:43 smithi150 ceph-mon[29276]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:17:43.678 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:43 smithi150 ceph-mon[29276]: Cluster is now healthy 2024-09-11T08:17:43.678 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:43 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/528750502' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:17:43.678 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:43 smithi150 ceph-mon[29276]: osdmap e240: 8 total, 8 up, 8 in 2024-09-11T08:17:43.678 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:43 smithi150 ceph-mon[29276]: 6.4 scrub starts 2024-09-11T08:17:43.678 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:43 smithi150 ceph-mon[29276]: 6.4 scrub ok 2024-09-11T08:17:43.678 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:43 smithi150 ceph-mon[29276]: 6.19 scrub starts 2024-09-11T08:17:43.678 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:43 smithi150 ceph-mon[29276]: 6.19 scrub ok 2024-09-11T08:17:43.678 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:43 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:17:43.678 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:43 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:17:43.678 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:43 smithi150 ceph-mon[29276]: 5.9 scrub starts 2024-09-11T08:17:43.678 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:43 smithi150 ceph-mon[29276]: 5.9 scrub ok 2024-09-11T08:17:43.678 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:43 smithi150 ceph-mon[29276]: pgmap v458: 148 pgs: 148 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 573 op/s 2024-09-11T08:17:43.679 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:43 smithi150 ceph-mon[39612]: 6.c scrub starts 2024-09-11T08:17:43.679 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:43 smithi150 ceph-mon[39612]: 6.c scrub ok 2024-09-11T08:17:43.679 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:43 smithi150 ceph-mon[39612]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:17:43.679 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:43 smithi150 ceph-mon[39612]: Cluster is now healthy 2024-09-11T08:17:43.679 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:43 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/528750502' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:17:43.679 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:43 smithi150 ceph-mon[39612]: osdmap e240: 8 total, 8 up, 8 in 2024-09-11T08:17:43.679 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:43 smithi150 ceph-mon[39612]: 6.4 scrub starts 2024-09-11T08:17:43.679 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:43 smithi150 ceph-mon[39612]: 6.4 scrub ok 2024-09-11T08:17:43.679 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:43 smithi150 ceph-mon[39612]: 6.19 scrub starts 2024-09-11T08:17:43.679 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:43 smithi150 ceph-mon[39612]: 6.19 scrub ok 2024-09-11T08:17:43.679 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:43 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:17:43.679 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:43 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:17:43.679 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:43 smithi150 ceph-mon[39612]: 5.9 scrub starts 2024-09-11T08:17:43.680 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:43 smithi150 ceph-mon[39612]: 5.9 scrub ok 2024-09-11T08:17:43.680 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:43 smithi150 ceph-mon[39612]: pgmap v458: 148 pgs: 148 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 573 op/s 2024-09-11T08:17:43.758 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:43.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:43.780 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:43.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:43.881 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:43.908 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:43.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:43.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:44.010 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:44.032 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:44.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:44.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:44.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:43 smithi181 ceph-mon[32701]: 6.c scrub starts 2024-09-11T08:17:44.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:43 smithi181 ceph-mon[32701]: 6.c scrub ok 2024-09-11T08:17:44.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:43 smithi181 ceph-mon[32701]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:17:44.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:43 smithi181 ceph-mon[32701]: Cluster is now healthy 2024-09-11T08:17:44.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:43 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/528750502' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:17:44.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:43 smithi181 ceph-mon[32701]: osdmap e240: 8 total, 8 up, 8 in 2024-09-11T08:17:44.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:43 smithi181 ceph-mon[32701]: 6.4 scrub starts 2024-09-11T08:17:44.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:43 smithi181 ceph-mon[32701]: 6.4 scrub ok 2024-09-11T08:17:44.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:43 smithi181 ceph-mon[32701]: 6.19 scrub starts 2024-09-11T08:17:44.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:43 smithi181 ceph-mon[32701]: 6.19 scrub ok 2024-09-11T08:17:44.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:43 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:17:44.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:43 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:17:44.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:43 smithi181 ceph-mon[32701]: 5.9 scrub starts 2024-09-11T08:17:44.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:43 smithi181 ceph-mon[32701]: 5.9 scrub ok 2024-09-11T08:17:44.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:43 smithi181 ceph-mon[32701]: pgmap v458: 148 pgs: 148 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 573 op/s 2024-09-11T08:17:44.134 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:44.160 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:44.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:44.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:44.261 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:44.284 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:44.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:44.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:44.386 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:44.411 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:44.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:44.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:44.513 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:44.537 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:44.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:44.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:44.639 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:44.665 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:44.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:44.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:44.766 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:44.788 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:44.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:44.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:44.889 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:44.890 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:44 smithi150 ceph-mon[29276]: 3.1e scrub starts 2024-09-11T08:17:44.890 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:44 smithi150 ceph-mon[29276]: 3.1e scrub ok 2024-09-11T08:17:44.890 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:44 smithi150 ceph-mon[29276]: osdmap e241: 8 total, 8 up, 8 in 2024-09-11T08:17:44.890 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:44 smithi150 ceph-mon[29276]: 4.12 scrub starts 2024-09-11T08:17:44.890 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:44 smithi150 ceph-mon[29276]: 4.12 scrub ok 2024-09-11T08:17:44.891 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:44 smithi150 ceph-mon[29276]: 4.6 scrub starts 2024-09-11T08:17:44.891 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:44 smithi150 ceph-mon[29276]: 4.6 scrub ok 2024-09-11T08:17:44.891 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:44 smithi150 ceph-mon[39612]: 3.1e scrub starts 2024-09-11T08:17:44.891 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:44 smithi150 ceph-mon[39612]: 3.1e scrub ok 2024-09-11T08:17:44.891 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:44 smithi150 ceph-mon[39612]: osdmap e241: 8 total, 8 up, 8 in 2024-09-11T08:17:44.891 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:44 smithi150 ceph-mon[39612]: 4.12 scrub starts 2024-09-11T08:17:44.891 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:44 smithi150 ceph-mon[39612]: 4.12 scrub ok 2024-09-11T08:17:44.891 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:44 smithi150 ceph-mon[39612]: 4.6 scrub starts 2024-09-11T08:17:44.891 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:44 smithi150 ceph-mon[39612]: 4.6 scrub ok 2024-09-11T08:17:44.912 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:44.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:44.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:45.014 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:17:45.035 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:45.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:45.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:17:45.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:44 smithi181 ceph-mon[32701]: 3.1e scrub starts 2024-09-11T08:17:45.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:44 smithi181 ceph-mon[32701]: 3.1e scrub ok 2024-09-11T08:17:45.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:44 smithi181 ceph-mon[32701]: osdmap e241: 8 total, 8 up, 8 in 2024-09-11T08:17:45.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:44 smithi181 ceph-mon[32701]: 4.12 scrub starts 2024-09-11T08:17:45.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:44 smithi181 ceph-mon[32701]: 4.12 scrub ok 2024-09-11T08:17:45.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:44 smithi181 ceph-mon[32701]: 4.6 scrub starts 2024-09-11T08:17:45.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:44 smithi181 ceph-mon[32701]: 4.6 scrub ok 2024-09-11T08:17:45.137 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:17:45.144 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-09-11T08:17:45.144 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:17:45.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:45.161 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:45.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:17:45.262 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:45.288 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:45.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:45.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:45.379 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T08:17:45.389 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:45.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:45.412 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:45.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:45.513 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:45.534 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:45.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:45.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:45.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:17:45 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:17:45] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:17:45.636 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:17:45.658 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:45.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:45.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:17:45.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:45 smithi150 ceph-mon[39612]: 6.1b scrub starts 2024-09-11T08:17:45.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:45 smithi150 ceph-mon[39612]: 6.1b scrub ok 2024-09-11T08:17:45.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:45 smithi150 ceph-mon[39612]: 5.15 scrub starts 2024-09-11T08:17:45.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:45 smithi150 ceph-mon[39612]: 5.15 scrub ok 2024-09-11T08:17:45.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:45 smithi150 ceph-mon[39612]: pgmap v460: 148 pgs: 148 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 496 op/s 2024-09-11T08:17:45.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:45 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:17:45.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:45 smithi150 ceph-mon[29276]: 6.1b scrub starts 2024-09-11T08:17:45.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:45 smithi150 ceph-mon[29276]: 6.1b scrub ok 2024-09-11T08:17:45.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:45 smithi150 ceph-mon[29276]: 5.15 scrub starts 2024-09-11T08:17:45.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:45 smithi150 ceph-mon[29276]: 5.15 scrub ok 2024-09-11T08:17:45.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:45 smithi150 ceph-mon[29276]: pgmap v460: 148 pgs: 148 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 496 op/s 2024-09-11T08:17:45.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:45 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:17:45.760 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:17:45.781 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:45.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:45.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:17:45.883 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:45.904 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:45.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:45.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:46.006 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:17:46.027 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:46.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:46.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:17:46.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:45 smithi181 ceph-mon[32701]: 6.1b scrub starts 2024-09-11T08:17:46.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:45 smithi181 ceph-mon[32701]: 6.1b scrub ok 2024-09-11T08:17:46.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:45 smithi181 ceph-mon[32701]: 5.15 scrub starts 2024-09-11T08:17:46.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:45 smithi181 ceph-mon[32701]: 5.15 scrub ok 2024-09-11T08:17:46.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:45 smithi181 ceph-mon[32701]: pgmap v460: 148 pgs: 148 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 496 op/s 2024-09-11T08:17:46.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:45 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:17:46.129 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:46.150 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:46.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:46.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:46.252 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:46.274 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:46.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:46.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:46.376 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:46.397 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:46.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:46.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:46.404 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-11T08:17:46.498 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:17:46.520 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:46.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:46.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:17:46.622 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:17:46.644 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:46.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:46.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:17:46.683 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:17:46.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:46 smithi150 ceph-mon[29276]: 3.b deep-scrub starts 2024-09-11T08:17:46.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:46 smithi150 ceph-mon[29276]: 3.b deep-scrub ok 2024-09-11T08:17:46.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:46 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:17:46.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:46 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:17:46.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:46 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:17:46.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:46 smithi150 ceph-mon[39612]: 3.b deep-scrub starts 2024-09-11T08:17:46.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:46 smithi150 ceph-mon[39612]: 3.b deep-scrub ok 2024-09-11T08:17:46.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:46 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:17:46.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:46 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:17:46.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:46 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:17:46.746 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:46.767 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:46.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:46.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:46.803 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:46 smithi181 ceph-mon[32701]: 3.b deep-scrub starts 2024-09-11T08:17:46.803 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:46 smithi181 ceph-mon[32701]: 3.b deep-scrub ok 2024-09-11T08:17:46.804 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:46 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:17:46.804 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:46 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:17:46.804 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:46 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:17:46.868 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:46.890 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:46.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:46.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:46.992 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:47.014 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:47.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:47.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:47.116 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:47.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:47.139 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:47.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:47.240 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:47.262 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:47.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:47.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:47.277 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:17:47.309 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:17:47.364 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:47.385 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:47.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:47.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:47.487 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:47.509 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:47.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:47.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:47.611 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:47.632 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:47.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:47.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:47.734 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:47.756 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:47.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:47.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:47.795 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:47 smithi150 ceph-mon[29276]: 6.13 deep-scrub starts 2024-09-11T08:17:47.795 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:47 smithi150 ceph-mon[29276]: 6.13 deep-scrub ok 2024-09-11T08:17:47.795 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:47 smithi150 ceph-mon[29276]: pgmap v461: 148 pgs: 148 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-11T08:17:47.795 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:47 smithi150 ceph-mon[29276]: 3.1 scrub starts 2024-09-11T08:17:47.795 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:47 smithi150 ceph-mon[29276]: 3.1 scrub ok 2024-09-11T08:17:47.795 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:47 smithi150 ceph-mon[39612]: 6.13 deep-scrub starts 2024-09-11T08:17:47.795 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:47 smithi150 ceph-mon[39612]: 6.13 deep-scrub ok 2024-09-11T08:17:47.795 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:47 smithi150 ceph-mon[39612]: pgmap v461: 148 pgs: 148 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-11T08:17:47.796 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:47 smithi150 ceph-mon[39612]: 3.1 scrub starts 2024-09-11T08:17:47.796 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:47 smithi150 ceph-mon[39612]: 3.1 scrub ok 2024-09-11T08:17:47.857 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:47.867 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:47 smithi181 ceph-mon[32701]: 6.13 deep-scrub starts 2024-09-11T08:17:47.867 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:47 smithi181 ceph-mon[32701]: 6.13 deep-scrub ok 2024-09-11T08:17:47.867 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:47 smithi181 ceph-mon[32701]: pgmap v461: 148 pgs: 148 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-11T08:17:47.867 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:47 smithi181 ceph-mon[32701]: 3.1 scrub starts 2024-09-11T08:17:47.867 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:47 smithi181 ceph-mon[32701]: 3.1 scrub ok 2024-09-11T08:17:47.879 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:47.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:47.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:47.981 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:48.003 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:48.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:48.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:48.105 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:48.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:48.128 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:48.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:48.229 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:48.250 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:48.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:48.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:48.352 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:48.373 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:48.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:48.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:48.405 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-11T08:17:48.406 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:17:48.413 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:17:48 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:17:48.475 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:48.496 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:48.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:48.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:48.598 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:17:48.619 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:48.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:48.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:17:48.721 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:48.738 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:17:48 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:17:48] "GET /metrics HTTP/1.1" 200 224246 "" "Prometheus/2.43.0" 2024-09-11T08:17:48.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:48 smithi150 ceph-mon[29276]: 5.0 scrub starts 2024-09-11T08:17:48.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:48 smithi150 ceph-mon[29276]: 5.0 scrub ok 2024-09-11T08:17:48.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:48 smithi150 ceph-mon[29276]: 6.d scrub starts 2024-09-11T08:17:48.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:48 smithi150 ceph-mon[29276]: 6.d scrub ok 2024-09-11T08:17:48.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:48 smithi150 ceph-mon[29276]: 3.6 deep-scrub starts 2024-09-11T08:17:48.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:48 smithi150 ceph-mon[29276]: 3.6 deep-scrub ok 2024-09-11T08:17:48.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:48 smithi150 ceph-mon[29276]: 5.1 scrub starts 2024-09-11T08:17:48.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:48 smithi150 ceph-mon[29276]: 5.1 scrub ok 2024-09-11T08:17:48.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:48 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:17:48.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:48 smithi150 ceph-mon[39612]: 5.0 scrub starts 2024-09-11T08:17:48.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:48 smithi150 ceph-mon[39612]: 5.0 scrub ok 2024-09-11T08:17:48.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:48 smithi150 ceph-mon[39612]: 6.d scrub starts 2024-09-11T08:17:48.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:48 smithi150 ceph-mon[39612]: 6.d scrub ok 2024-09-11T08:17:48.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:48 smithi150 ceph-mon[39612]: 3.6 deep-scrub starts 2024-09-11T08:17:48.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:48 smithi150 ceph-mon[39612]: 3.6 deep-scrub ok 2024-09-11T08:17:48.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:48 smithi150 ceph-mon[39612]: 5.1 scrub starts 2024-09-11T08:17:48.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:48 smithi150 ceph-mon[39612]: 5.1 scrub ok 2024-09-11T08:17:48.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:48 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:17:48.743 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:48.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:48.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:48.798 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T08:17:48.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:48 smithi181 ceph-mon[32701]: 5.0 scrub starts 2024-09-11T08:17:48.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:48 smithi181 ceph-mon[32701]: 5.0 scrub ok 2024-09-11T08:17:48.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:48 smithi181 ceph-mon[32701]: 6.d scrub starts 2024-09-11T08:17:48.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:48 smithi181 ceph-mon[32701]: 6.d scrub ok 2024-09-11T08:17:48.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:48 smithi181 ceph-mon[32701]: 3.6 deep-scrub starts 2024-09-11T08:17:48.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:48 smithi181 ceph-mon[32701]: 3.6 deep-scrub ok 2024-09-11T08:17:48.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:48 smithi181 ceph-mon[32701]: 5.1 scrub starts 2024-09-11T08:17:48.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:48 smithi181 ceph-mon[32701]: 5.1 scrub ok 2024-09-11T08:17:48.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:48 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:17:48.844 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:17:48.866 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:48.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:48.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:17:48.968 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:17:48.989 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:48.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:48.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:17:49.074 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-11T08:17:49.091 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:17:49.138 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:49.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:49.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:17:49.239 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:49.262 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:49.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:49.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:49.329 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:17:49.364 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:49.384 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:49.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:49.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:49.526 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:17:49.548 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:49.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:49.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:17:49.650 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:49.672 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:49.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:49.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:49.774 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:17:49.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:49.796 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:49.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:17:49.897 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:49.918 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:49.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:49.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:49.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:49 smithi150 ceph-mon[29276]: pgmap v462: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:17:49.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:49 smithi150 ceph-mon[29276]: 3.17 scrub starts 2024-09-11T08:17:49.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:49 smithi150 ceph-mon[29276]: 3.17 scrub ok 2024-09-11T08:17:49.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:49 smithi150 ceph-mon[29276]: 4.1 scrub starts 2024-09-11T08:17:49.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:49 smithi150 ceph-mon[29276]: 4.1 scrub ok 2024-09-11T08:17:49.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:49 smithi150 ceph-mon[39612]: pgmap v462: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:17:49.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:49 smithi150 ceph-mon[39612]: 3.17 scrub starts 2024-09-11T08:17:49.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:49 smithi150 ceph-mon[39612]: 3.17 scrub ok 2024-09-11T08:17:49.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:49 smithi150 ceph-mon[39612]: 4.1 scrub starts 2024-09-11T08:17:49.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:49 smithi150 ceph-mon[39612]: 4.1 scrub ok 2024-09-11T08:17:50.020 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:50.042 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:50.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:50.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:50.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:49 smithi181 ceph-mon[32701]: pgmap v462: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:17:50.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:49 smithi181 ceph-mon[32701]: 3.17 scrub starts 2024-09-11T08:17:50.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:49 smithi181 ceph-mon[32701]: 3.17 scrub ok 2024-09-11T08:17:50.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:49 smithi181 ceph-mon[32701]: 4.1 scrub starts 2024-09-11T08:17:50.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:49 smithi181 ceph-mon[32701]: 4.1 scrub ok 2024-09-11T08:17:50.144 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:17:50.165 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:50.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:50.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:17:50.267 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:50.288 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:50.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:50.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:50.390 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:50.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:50.412 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:50.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:50.513 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:17:50.536 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:50.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:50.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:17:50.616 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:50 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1265119223' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-11T08:17:50.616 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:50 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-11T08:17:50.617 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'unique_pool_1' removed 2024-09-11T08:17:50.617 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:50 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1265119223' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-11T08:17:50.617 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:50 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-11T08:17:50.637 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:50.675 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:50.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:50.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:50.777 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:50.798 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:50.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:50.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:50.900 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:50.921 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:50.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:50.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:50.928 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:50 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1265119223' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-11T08:17:50.928 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:50 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-11T08:17:50.974 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:17:50.974 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-11T08:17:50.975 INFO:teuthology.orchestra.run.smithi150.stdout: "epoch": 242, 2024-09-11T08:17:50.975 INFO:teuthology.orchestra.run.smithi150.stdout: "fsid": "c1bddff4-7013-11ef-bcea-c7b262605968", 2024-09-11T08:17:50.975 INFO:teuthology.orchestra.run.smithi150.stdout: "created": "2024-09-11T08:02:15.344618+0000", 2024-09-11T08:17:50.975 INFO:teuthology.orchestra.run.smithi150.stdout: "modified": "2024-09-11T08:17:50.600744+0000", 2024-09-11T08:17:50.975 INFO:teuthology.orchestra.run.smithi150.stdout: "last_up_change": "2024-09-11T08:14:23.745449+0000", 2024-09-11T08:17:50.975 INFO:teuthology.orchestra.run.smithi150.stdout: "last_in_change": "2024-09-11T08:07:00.805384+0000", 2024-09-11T08:17:50.975 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-11T08:17:50.975 INFO:teuthology.orchestra.run.smithi150.stdout: "flags_num": 5799936, 2024-09-11T08:17:50.975 INFO:teuthology.orchestra.run.smithi150.stdout: "flags_set": [ 2024-09-11T08:17:50.975 INFO:teuthology.orchestra.run.smithi150.stdout: "pglog_hardlimit", 2024-09-11T08:17:50.975 INFO:teuthology.orchestra.run.smithi150.stdout: "purged_snapdirs", 2024-09-11T08:17:50.975 INFO:teuthology.orchestra.run.smithi150.stdout: "recovery_deletes", 2024-09-11T08:17:50.975 INFO:teuthology.orchestra.run.smithi150.stdout: "sortbitwise" 2024-09-11T08:17:50.975 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-09-11T08:17:50.976 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_version": 19, 2024-09-11T08:17:50.976 INFO:teuthology.orchestra.run.smithi150.stdout: "full_ratio": 0.94999998807907104, 2024-09-11T08:17:50.976 INFO:teuthology.orchestra.run.smithi150.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-11T08:17:50.976 INFO:teuthology.orchestra.run.smithi150.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-11T08:17:50.976 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_snapshot": "", 2024-09-11T08:17:50.976 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_max": 8, 2024-09-11T08:17:50.976 INFO:teuthology.orchestra.run.smithi150.stdout: "max_osd": 8, 2024-09-11T08:17:50.976 INFO:teuthology.orchestra.run.smithi150.stdout: "require_min_compat_client": "luminous", 2024-09-11T08:17:50.976 INFO:teuthology.orchestra.run.smithi150.stdout: "min_compat_client": "luminous", 2024-09-11T08:17:50.976 INFO:teuthology.orchestra.run.smithi150.stdout: "require_osd_release": "quincy", 2024-09-11T08:17:50.976 INFO:teuthology.orchestra.run.smithi150.stdout: "pools": [ 2024-09-11T08:17:50.976 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:50.976 INFO:teuthology.orchestra.run.smithi150.stdout: "pool": 1, 2024-09-11T08:17:50.977 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_name": ".mgr", 2024-09-11T08:17:50.977 INFO:teuthology.orchestra.run.smithi150.stdout: "create_time": "2024-09-11T08:05:21.044729+0000", 2024-09-11T08:17:50.977 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-09-11T08:17:50.977 INFO:teuthology.orchestra.run.smithi150.stdout: "flags_names": "hashpspool", 2024-09-11T08:17:50.977 INFO:teuthology.orchestra.run.smithi150.stdout: "type": 1, 2024-09-11T08:17:50.977 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 3, 2024-09-11T08:17:50.977 INFO:teuthology.orchestra.run.smithi150.stdout: "min_size": 2, 2024-09-11T08:17:50.977 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_rule": 0, 2024-09-11T08:17:50.977 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_count": 0, 2024-09-11T08:17:50.977 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_target": 0, 2024-09-11T08:17:50.977 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T08:17:50.977 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T08:17:50.977 INFO:teuthology.orchestra.run.smithi150.stdout: "object_hash": 2, 2024-09-11T08:17:50.977 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_autoscale_mode": "off", 2024-09-11T08:17:50.977 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num": 1, 2024-09-11T08:17:50.978 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num": 1, 2024-09-11T08:17:50.978 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num_target": 1, 2024-09-11T08:17:50.978 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_target": 1, 2024-09-11T08:17:50.978 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_pending": 1, 2024-09-11T08:17:50.978 INFO:teuthology.orchestra.run.smithi150.stdout: "last_pg_merge_meta": { 2024-09-11T08:17:50.978 INFO:teuthology.orchestra.run.smithi150.stdout: "source_pgid": "0.0", 2024-09-11T08:17:50.978 INFO:teuthology.orchestra.run.smithi150.stdout: "ready_epoch": 0, 2024-09-11T08:17:50.978 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_started": 0, 2024-09-11T08:17:50.978 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_clean": 0, 2024-09-11T08:17:50.978 INFO:teuthology.orchestra.run.smithi150.stdout: "source_version": "0'0", 2024-09-11T08:17:50.978 INFO:teuthology.orchestra.run.smithi150.stdout: "target_version": "0'0" 2024-09-11T08:17:50.978 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:50.978 INFO:teuthology.orchestra.run.smithi150.stdout: "last_change": "21", 2024-09-11T08:17:50.978 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend": "0", 2024-09-11T08:17:50.979 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T08:17:50.979 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T08:17:50.979 INFO:teuthology.orchestra.run.smithi150.stdout: "auid": 0, 2024-09-11T08:17:50.979 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_mode": "selfmanaged", 2024-09-11T08:17:50.979 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_seq": 0, 2024-09-11T08:17:50.979 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_epoch": 0, 2024-09-11T08:17:50.979 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_snaps": [], 2024-09-11T08:17:50.979 INFO:teuthology.orchestra.run.smithi150.stdout: "removed_snaps": "[]", 2024-09-11T08:17:50.979 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_bytes": 0, 2024-09-11T08:17:50.979 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_objects": 0, 2024-09-11T08:17:50.979 INFO:teuthology.orchestra.run.smithi150.stdout: "tiers": [], 2024-09-11T08:17:50.980 INFO:teuthology.orchestra.run.smithi150.stdout: "tier_of": -1, 2024-09-11T08:17:50.980 INFO:teuthology.orchestra.run.smithi150.stdout: "read_tier": -1, 2024-09-11T08:17:50.980 INFO:teuthology.orchestra.run.smithi150.stdout: "write_tier": -1, 2024-09-11T08:17:50.980 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_mode": "none", 2024-09-11T08:17:50.980 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_bytes": 0, 2024-09-11T08:17:50.980 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_objects": 0, 2024-09-11T08:17:50.980 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T08:17:50.981 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T08:17:50.981 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T08:17:50.981 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_flush_age": 0, 2024-09-11T08:17:50.981 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_evict_age": 0, 2024-09-11T08:17:50.981 INFO:teuthology.orchestra.run.smithi150.stdout: "erasure_code_profile": "", 2024-09-11T08:17:50.981 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_params": { 2024-09-11T08:17:50.982 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "none" 2024-09-11T08:17:50.982 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:50.982 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_period": 0, 2024-09-11T08:17:50.982 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_count": 0, 2024-09-11T08:17:50.982 INFO:teuthology.orchestra.run.smithi150.stdout: "use_gmt_hitset": true, 2024-09-11T08:17:50.982 INFO:teuthology.orchestra.run.smithi150.stdout: "min_read_recency_for_promote": 0, 2024-09-11T08:17:50.982 INFO:teuthology.orchestra.run.smithi150.stdout: "min_write_recency_for_promote": 0, 2024-09-11T08:17:50.983 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T08:17:50.983 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_search_last_n": 0, 2024-09-11T08:17:50.983 INFO:teuthology.orchestra.run.smithi150.stdout: "grade_table": [], 2024-09-11T08:17:50.983 INFO:teuthology.orchestra.run.smithi150.stdout: "stripe_width": 0, 2024-09-11T08:17:50.983 INFO:teuthology.orchestra.run.smithi150.stdout: "expected_num_objects": 0, 2024-09-11T08:17:50.983 INFO:teuthology.orchestra.run.smithi150.stdout: "fast_read": false, 2024-09-11T08:17:50.984 INFO:teuthology.orchestra.run.smithi150.stdout: "options": { 2024-09-11T08:17:50.984 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_max": 32, 2024-09-11T08:17:50.984 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_min": 1 2024-09-11T08:17:50.984 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:50.984 INFO:teuthology.orchestra.run.smithi150.stdout: "application_metadata": { 2024-09-11T08:17:50.984 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": {} 2024-09-11T08:17:50.985 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:50.985 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:50.985 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:50.985 INFO:teuthology.orchestra.run.smithi150.stdout: "pool": 2, 2024-09-11T08:17:50.985 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_name": "datapool", 2024-09-11T08:17:50.985 INFO:teuthology.orchestra.run.smithi150.stdout: "create_time": "2024-09-11T08:07:28.786572+0000", 2024-09-11T08:17:50.986 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 8193, 2024-09-11T08:17:50.986 INFO:teuthology.orchestra.run.smithi150.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-11T08:17:50.986 INFO:teuthology.orchestra.run.smithi150.stdout: "type": 1, 2024-09-11T08:17:50.986 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 3, 2024-09-11T08:17:50.986 INFO:teuthology.orchestra.run.smithi150.stdout: "min_size": 2, 2024-09-11T08:17:50.986 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_rule": 0, 2024-09-11T08:17:50.987 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_count": 0, 2024-09-11T08:17:50.987 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_target": 0, 2024-09-11T08:17:50.987 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T08:17:50.987 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T08:17:50.987 INFO:teuthology.orchestra.run.smithi150.stdout: "object_hash": 2, 2024-09-11T08:17:50.987 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_autoscale_mode": "off", 2024-09-11T08:17:50.988 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num": 3, 2024-09-11T08:17:50.988 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num": 3, 2024-09-11T08:17:50.988 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num_target": 3, 2024-09-11T08:17:50.988 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_target": 3, 2024-09-11T08:17:50.988 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_pending": 3, 2024-09-11T08:17:50.988 INFO:teuthology.orchestra.run.smithi150.stdout: "last_pg_merge_meta": { 2024-09-11T08:17:50.988 INFO:teuthology.orchestra.run.smithi150.stdout: "source_pgid": "0.0", 2024-09-11T08:17:50.989 INFO:teuthology.orchestra.run.smithi150.stdout: "ready_epoch": 0, 2024-09-11T08:17:50.989 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_started": 0, 2024-09-11T08:17:50.989 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_clean": 0, 2024-09-11T08:17:50.989 INFO:teuthology.orchestra.run.smithi150.stdout: "source_version": "0'0", 2024-09-11T08:17:50.989 INFO:teuthology.orchestra.run.smithi150.stdout: "target_version": "0'0" 2024-09-11T08:17:50.990 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:50.990 INFO:teuthology.orchestra.run.smithi150.stdout: "last_change": "63", 2024-09-11T08:17:50.990 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend": "0", 2024-09-11T08:17:50.990 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T08:17:50.990 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T08:17:50.990 INFO:teuthology.orchestra.run.smithi150.stdout: "auid": 0, 2024-09-11T08:17:50.991 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_mode": "selfmanaged", 2024-09-11T08:17:50.991 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_seq": 3, 2024-09-11T08:17:50.991 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_epoch": 63, 2024-09-11T08:17:50.991 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_snaps": [], 2024-09-11T08:17:50.991 INFO:teuthology.orchestra.run.smithi150.stdout: "removed_snaps": "[]", 2024-09-11T08:17:50.991 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_bytes": 0, 2024-09-11T08:17:50.991 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_objects": 0, 2024-09-11T08:17:50.992 INFO:teuthology.orchestra.run.smithi150.stdout: "tiers": [], 2024-09-11T08:17:50.992 INFO:teuthology.orchestra.run.smithi150.stdout: "tier_of": -1, 2024-09-11T08:17:50.992 INFO:teuthology.orchestra.run.smithi150.stdout: "read_tier": -1, 2024-09-11T08:17:50.992 INFO:teuthology.orchestra.run.smithi150.stdout: "write_tier": -1, 2024-09-11T08:17:50.992 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_mode": "none", 2024-09-11T08:17:50.992 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_bytes": 0, 2024-09-11T08:17:50.993 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_objects": 0, 2024-09-11T08:17:50.993 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T08:17:50.993 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T08:17:50.993 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T08:17:50.993 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_flush_age": 0, 2024-09-11T08:17:50.993 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_evict_age": 0, 2024-09-11T08:17:50.994 INFO:teuthology.orchestra.run.smithi150.stdout: "erasure_code_profile": "", 2024-09-11T08:17:50.994 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_params": { 2024-09-11T08:17:50.994 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "none" 2024-09-11T08:17:50.994 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:50.994 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_period": 0, 2024-09-11T08:17:50.994 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_count": 0, 2024-09-11T08:17:50.995 INFO:teuthology.orchestra.run.smithi150.stdout: "use_gmt_hitset": true, 2024-09-11T08:17:50.995 INFO:teuthology.orchestra.run.smithi150.stdout: "min_read_recency_for_promote": 0, 2024-09-11T08:17:50.995 INFO:teuthology.orchestra.run.smithi150.stdout: "min_write_recency_for_promote": 0, 2024-09-11T08:17:50.995 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T08:17:50.995 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_search_last_n": 0, 2024-09-11T08:17:50.995 INFO:teuthology.orchestra.run.smithi150.stdout: "grade_table": [], 2024-09-11T08:17:50.996 INFO:teuthology.orchestra.run.smithi150.stdout: "stripe_width": 0, 2024-09-11T08:17:50.996 INFO:teuthology.orchestra.run.smithi150.stdout: "expected_num_objects": 0, 2024-09-11T08:17:50.996 INFO:teuthology.orchestra.run.smithi150.stdout: "fast_read": false, 2024-09-11T08:17:50.996 INFO:teuthology.orchestra.run.smithi150.stdout: "options": {}, 2024-09-11T08:17:50.996 INFO:teuthology.orchestra.run.smithi150.stdout: "application_metadata": { 2024-09-11T08:17:50.996 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd": {} 2024-09-11T08:17:50.997 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:50.997 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:50.997 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:50.997 INFO:teuthology.orchestra.run.smithi150.stdout: "pool": 3, 2024-09-11T08:17:50.997 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_name": ".rgw.root", 2024-09-11T08:17:50.997 INFO:teuthology.orchestra.run.smithi150.stdout: "create_time": "2024-09-11T08:07:29.021286+0000", 2024-09-11T08:17:50.998 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-09-11T08:17:50.998 INFO:teuthology.orchestra.run.smithi150.stdout: "flags_names": "hashpspool", 2024-09-11T08:17:50.998 INFO:teuthology.orchestra.run.smithi150.stdout: "type": 1, 2024-09-11T08:17:50.998 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 3, 2024-09-11T08:17:50.998 INFO:teuthology.orchestra.run.smithi150.stdout: "min_size": 2, 2024-09-11T08:17:50.998 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_rule": 0, 2024-09-11T08:17:50.999 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_count": 0, 2024-09-11T08:17:50.999 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_target": 0, 2024-09-11T08:17:50.999 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T08:17:50.999 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T08:17:50.999 INFO:teuthology.orchestra.run.smithi150.stdout: "object_hash": 2, 2024-09-11T08:17:50.999 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_autoscale_mode": "off", 2024-09-11T08:17:51.000 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num": 32, 2024-09-11T08:17:51.000 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num": 32, 2024-09-11T08:17:51.000 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num_target": 32, 2024-09-11T08:17:51.000 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_target": 32, 2024-09-11T08:17:51.000 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_pending": 32, 2024-09-11T08:17:51.001 INFO:teuthology.orchestra.run.smithi150.stdout: "last_pg_merge_meta": { 2024-09-11T08:17:51.001 INFO:teuthology.orchestra.run.smithi150.stdout: "source_pgid": "0.0", 2024-09-11T08:17:51.001 INFO:teuthology.orchestra.run.smithi150.stdout: "ready_epoch": 0, 2024-09-11T08:17:51.001 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_started": 0, 2024-09-11T08:17:51.001 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_clean": 0, 2024-09-11T08:17:51.001 INFO:teuthology.orchestra.run.smithi150.stdout: "source_version": "0'0", 2024-09-11T08:17:51.002 INFO:teuthology.orchestra.run.smithi150.stdout: "target_version": "0'0" 2024-09-11T08:17:51.002 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.002 INFO:teuthology.orchestra.run.smithi150.stdout: "last_change": "60", 2024-09-11T08:17:51.002 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend": "0", 2024-09-11T08:17:51.002 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T08:17:51.002 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T08:17:51.003 INFO:teuthology.orchestra.run.smithi150.stdout: "auid": 0, 2024-09-11T08:17:51.003 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_mode": "selfmanaged", 2024-09-11T08:17:51.003 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_seq": 0, 2024-09-11T08:17:51.003 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_epoch": 0, 2024-09-11T08:17:51.003 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_snaps": [], 2024-09-11T08:17:51.004 INFO:teuthology.orchestra.run.smithi150.stdout: "removed_snaps": "[]", 2024-09-11T08:17:51.004 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_bytes": 0, 2024-09-11T08:17:51.004 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_objects": 0, 2024-09-11T08:17:51.004 INFO:teuthology.orchestra.run.smithi150.stdout: "tiers": [], 2024-09-11T08:17:51.004 INFO:teuthology.orchestra.run.smithi150.stdout: "tier_of": -1, 2024-09-11T08:17:51.004 INFO:teuthology.orchestra.run.smithi150.stdout: "read_tier": -1, 2024-09-11T08:17:51.005 INFO:teuthology.orchestra.run.smithi150.stdout: "write_tier": -1, 2024-09-11T08:17:51.005 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_mode": "none", 2024-09-11T08:17:51.005 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_bytes": 0, 2024-09-11T08:17:51.005 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_objects": 0, 2024-09-11T08:17:51.005 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T08:17:51.005 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T08:17:51.005 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T08:17:51.006 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_flush_age": 0, 2024-09-11T08:17:51.006 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_evict_age": 0, 2024-09-11T08:17:51.006 INFO:teuthology.orchestra.run.smithi150.stdout: "erasure_code_profile": "", 2024-09-11T08:17:51.006 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_params": { 2024-09-11T08:17:51.006 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "none" 2024-09-11T08:17:51.006 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.007 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_period": 0, 2024-09-11T08:17:51.007 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_count": 0, 2024-09-11T08:17:51.007 INFO:teuthology.orchestra.run.smithi150.stdout: "use_gmt_hitset": true, 2024-09-11T08:17:51.007 INFO:teuthology.orchestra.run.smithi150.stdout: "min_read_recency_for_promote": 0, 2024-09-11T08:17:51.007 INFO:teuthology.orchestra.run.smithi150.stdout: "min_write_recency_for_promote": 0, 2024-09-11T08:17:51.007 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T08:17:51.007 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_search_last_n": 0, 2024-09-11T08:17:51.007 INFO:teuthology.orchestra.run.smithi150.stdout: "grade_table": [], 2024-09-11T08:17:51.008 INFO:teuthology.orchestra.run.smithi150.stdout: "stripe_width": 0, 2024-09-11T08:17:51.008 INFO:teuthology.orchestra.run.smithi150.stdout: "expected_num_objects": 0, 2024-09-11T08:17:51.008 INFO:teuthology.orchestra.run.smithi150.stdout: "fast_read": false, 2024-09-11T08:17:51.008 INFO:teuthology.orchestra.run.smithi150.stdout: "options": {}, 2024-09-11T08:17:51.008 INFO:teuthology.orchestra.run.smithi150.stdout: "application_metadata": { 2024-09-11T08:17:51.008 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw": {} 2024-09-11T08:17:51.008 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:51.008 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.008 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.008 INFO:teuthology.orchestra.run.smithi150.stdout: "pool": 4, 2024-09-11T08:17:51.008 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_name": "default.rgw.log", 2024-09-11T08:17:51.008 INFO:teuthology.orchestra.run.smithi150.stdout: "create_time": "2024-09-11T08:07:30.334181+0000", 2024-09-11T08:17:51.008 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-09-11T08:17:51.009 INFO:teuthology.orchestra.run.smithi150.stdout: "flags_names": "hashpspool", 2024-09-11T08:17:51.009 INFO:teuthology.orchestra.run.smithi150.stdout: "type": 1, 2024-09-11T08:17:51.009 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 3, 2024-09-11T08:17:51.009 INFO:teuthology.orchestra.run.smithi150.stdout: "min_size": 2, 2024-09-11T08:17:51.009 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_rule": 0, 2024-09-11T08:17:51.009 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_count": 0, 2024-09-11T08:17:51.009 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_target": 0, 2024-09-11T08:17:51.009 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T08:17:51.009 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T08:17:51.009 INFO:teuthology.orchestra.run.smithi150.stdout: "object_hash": 2, 2024-09-11T08:17:51.009 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_autoscale_mode": "off", 2024-09-11T08:17:51.009 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num": 32, 2024-09-11T08:17:51.009 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num": 32, 2024-09-11T08:17:51.010 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num_target": 32, 2024-09-11T08:17:51.010 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_target": 32, 2024-09-11T08:17:51.010 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_pending": 32, 2024-09-11T08:17:51.010 INFO:teuthology.orchestra.run.smithi150.stdout: "last_pg_merge_meta": { 2024-09-11T08:17:51.010 INFO:teuthology.orchestra.run.smithi150.stdout: "source_pgid": "0.0", 2024-09-11T08:17:51.010 INFO:teuthology.orchestra.run.smithi150.stdout: "ready_epoch": 0, 2024-09-11T08:17:51.010 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_started": 0, 2024-09-11T08:17:51.010 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_clean": 0, 2024-09-11T08:17:51.010 INFO:teuthology.orchestra.run.smithi150.stdout: "source_version": "0'0", 2024-09-11T08:17:51.010 INFO:teuthology.orchestra.run.smithi150.stdout: "target_version": "0'0" 2024-09-11T08:17:51.010 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.010 INFO:teuthology.orchestra.run.smithi150.stdout: "last_change": "62", 2024-09-11T08:17:51.010 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend": "0", 2024-09-11T08:17:51.010 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T08:17:51.010 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T08:17:51.011 INFO:teuthology.orchestra.run.smithi150.stdout: "auid": 0, 2024-09-11T08:17:51.011 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_mode": "selfmanaged", 2024-09-11T08:17:51.011 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_seq": 0, 2024-09-11T08:17:51.011 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_epoch": 0, 2024-09-11T08:17:51.011 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_snaps": [], 2024-09-11T08:17:51.011 INFO:teuthology.orchestra.run.smithi150.stdout: "removed_snaps": "[]", 2024-09-11T08:17:51.011 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_bytes": 0, 2024-09-11T08:17:51.011 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_objects": 0, 2024-09-11T08:17:51.011 INFO:teuthology.orchestra.run.smithi150.stdout: "tiers": [], 2024-09-11T08:17:51.011 INFO:teuthology.orchestra.run.smithi150.stdout: "tier_of": -1, 2024-09-11T08:17:51.011 INFO:teuthology.orchestra.run.smithi150.stdout: "read_tier": -1, 2024-09-11T08:17:51.011 INFO:teuthology.orchestra.run.smithi150.stdout: "write_tier": -1, 2024-09-11T08:17:51.011 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_mode": "none", 2024-09-11T08:17:51.011 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_bytes": 0, 2024-09-11T08:17:51.011 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_objects": 0, 2024-09-11T08:17:51.012 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T08:17:51.012 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T08:17:51.012 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T08:17:51.012 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_flush_age": 0, 2024-09-11T08:17:51.012 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_evict_age": 0, 2024-09-11T08:17:51.012 INFO:teuthology.orchestra.run.smithi150.stdout: "erasure_code_profile": "", 2024-09-11T08:17:51.012 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_params": { 2024-09-11T08:17:51.012 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "none" 2024-09-11T08:17:51.012 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.012 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_period": 0, 2024-09-11T08:17:51.012 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_count": 0, 2024-09-11T08:17:51.012 INFO:teuthology.orchestra.run.smithi150.stdout: "use_gmt_hitset": true, 2024-09-11T08:17:51.012 INFO:teuthology.orchestra.run.smithi150.stdout: "min_read_recency_for_promote": 0, 2024-09-11T08:17:51.013 INFO:teuthology.orchestra.run.smithi150.stdout: "min_write_recency_for_promote": 0, 2024-09-11T08:17:51.013 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T08:17:51.013 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_search_last_n": 0, 2024-09-11T08:17:51.013 INFO:teuthology.orchestra.run.smithi150.stdout: "grade_table": [], 2024-09-11T08:17:51.013 INFO:teuthology.orchestra.run.smithi150.stdout: "stripe_width": 0, 2024-09-11T08:17:51.013 INFO:teuthology.orchestra.run.smithi150.stdout: "expected_num_objects": 0, 2024-09-11T08:17:51.013 INFO:teuthology.orchestra.run.smithi150.stdout: "fast_read": false, 2024-09-11T08:17:51.013 INFO:teuthology.orchestra.run.smithi150.stdout: "options": {}, 2024-09-11T08:17:51.013 INFO:teuthology.orchestra.run.smithi150.stdout: "application_metadata": { 2024-09-11T08:17:51.013 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw": {} 2024-09-11T08:17:51.013 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:51.013 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.013 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.013 INFO:teuthology.orchestra.run.smithi150.stdout: "pool": 5, 2024-09-11T08:17:51.014 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_name": "default.rgw.control", 2024-09-11T08:17:51.014 INFO:teuthology.orchestra.run.smithi150.stdout: "create_time": "2024-09-11T08:07:32.289626+0000", 2024-09-11T08:17:51.014 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-09-11T08:17:51.014 INFO:teuthology.orchestra.run.smithi150.stdout: "flags_names": "hashpspool", 2024-09-11T08:17:51.014 INFO:teuthology.orchestra.run.smithi150.stdout: "type": 1, 2024-09-11T08:17:51.014 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 3, 2024-09-11T08:17:51.014 INFO:teuthology.orchestra.run.smithi150.stdout: "min_size": 2, 2024-09-11T08:17:51.014 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_rule": 0, 2024-09-11T08:17:51.014 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_count": 0, 2024-09-11T08:17:51.014 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_target": 0, 2024-09-11T08:17:51.014 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T08:17:51.014 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T08:17:51.014 INFO:teuthology.orchestra.run.smithi150.stdout: "object_hash": 2, 2024-09-11T08:17:51.014 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_autoscale_mode": "off", 2024-09-11T08:17:51.015 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num": 32, 2024-09-11T08:17:51.015 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num": 32, 2024-09-11T08:17:51.015 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num_target": 32, 2024-09-11T08:17:51.015 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_target": 32, 2024-09-11T08:17:51.015 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_pending": 32, 2024-09-11T08:17:51.015 INFO:teuthology.orchestra.run.smithi150.stdout: "last_pg_merge_meta": { 2024-09-11T08:17:51.015 INFO:teuthology.orchestra.run.smithi150.stdout: "source_pgid": "0.0", 2024-09-11T08:17:51.015 INFO:teuthology.orchestra.run.smithi150.stdout: "ready_epoch": 0, 2024-09-11T08:17:51.015 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_started": 0, 2024-09-11T08:17:51.015 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_clean": 0, 2024-09-11T08:17:51.015 INFO:teuthology.orchestra.run.smithi150.stdout: "source_version": "0'0", 2024-09-11T08:17:51.015 INFO:teuthology.orchestra.run.smithi150.stdout: "target_version": "0'0" 2024-09-11T08:17:51.015 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.015 INFO:teuthology.orchestra.run.smithi150.stdout: "last_change": "64", 2024-09-11T08:17:51.015 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend": "0", 2024-09-11T08:17:51.016 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T08:17:51.016 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T08:17:51.016 INFO:teuthology.orchestra.run.smithi150.stdout: "auid": 0, 2024-09-11T08:17:51.016 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_mode": "selfmanaged", 2024-09-11T08:17:51.016 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_seq": 0, 2024-09-11T08:17:51.016 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_epoch": 0, 2024-09-11T08:17:51.016 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_snaps": [], 2024-09-11T08:17:51.016 INFO:teuthology.orchestra.run.smithi150.stdout: "removed_snaps": "[]", 2024-09-11T08:17:51.016 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_bytes": 0, 2024-09-11T08:17:51.016 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_objects": 0, 2024-09-11T08:17:51.016 INFO:teuthology.orchestra.run.smithi150.stdout: "tiers": [], 2024-09-11T08:17:51.016 INFO:teuthology.orchestra.run.smithi150.stdout: "tier_of": -1, 2024-09-11T08:17:51.016 INFO:teuthology.orchestra.run.smithi150.stdout: "read_tier": -1, 2024-09-11T08:17:51.016 INFO:teuthology.orchestra.run.smithi150.stdout: "write_tier": -1, 2024-09-11T08:17:51.016 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_mode": "none", 2024-09-11T08:17:51.017 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_bytes": 0, 2024-09-11T08:17:51.017 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_objects": 0, 2024-09-11T08:17:51.017 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T08:17:51.017 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T08:17:51.017 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T08:17:51.017 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_flush_age": 0, 2024-09-11T08:17:51.017 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_evict_age": 0, 2024-09-11T08:17:51.017 INFO:teuthology.orchestra.run.smithi150.stdout: "erasure_code_profile": "", 2024-09-11T08:17:51.017 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_params": { 2024-09-11T08:17:51.017 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "none" 2024-09-11T08:17:51.017 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.017 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_period": 0, 2024-09-11T08:17:51.017 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_count": 0, 2024-09-11T08:17:51.018 INFO:teuthology.orchestra.run.smithi150.stdout: "use_gmt_hitset": true, 2024-09-11T08:17:51.018 INFO:teuthology.orchestra.run.smithi150.stdout: "min_read_recency_for_promote": 0, 2024-09-11T08:17:51.018 INFO:teuthology.orchestra.run.smithi150.stdout: "min_write_recency_for_promote": 0, 2024-09-11T08:17:51.018 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T08:17:51.018 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_search_last_n": 0, 2024-09-11T08:17:51.018 INFO:teuthology.orchestra.run.smithi150.stdout: "grade_table": [], 2024-09-11T08:17:51.018 INFO:teuthology.orchestra.run.smithi150.stdout: "stripe_width": 0, 2024-09-11T08:17:51.018 INFO:teuthology.orchestra.run.smithi150.stdout: "expected_num_objects": 0, 2024-09-11T08:17:51.018 INFO:teuthology.orchestra.run.smithi150.stdout: "fast_read": false, 2024-09-11T08:17:51.018 INFO:teuthology.orchestra.run.smithi150.stdout: "options": {}, 2024-09-11T08:17:51.018 INFO:teuthology.orchestra.run.smithi150.stdout: "application_metadata": { 2024-09-11T08:17:51.018 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw": {} 2024-09-11T08:17:51.018 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:51.018 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.018 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.019 INFO:teuthology.orchestra.run.smithi150.stdout: "pool": 6, 2024-09-11T08:17:51.019 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_name": "default.rgw.meta", 2024-09-11T08:17:51.019 INFO:teuthology.orchestra.run.smithi150.stdout: "create_time": "2024-09-11T08:07:34.351861+0000", 2024-09-11T08:17:51.019 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-09-11T08:17:51.019 INFO:teuthology.orchestra.run.smithi150.stdout: "flags_names": "hashpspool", 2024-09-11T08:17:51.019 INFO:teuthology.orchestra.run.smithi150.stdout: "type": 1, 2024-09-11T08:17:51.019 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 3, 2024-09-11T08:17:51.019 INFO:teuthology.orchestra.run.smithi150.stdout: "min_size": 2, 2024-09-11T08:17:51.019 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_rule": 0, 2024-09-11T08:17:51.019 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_count": 0, 2024-09-11T08:17:51.019 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_target": 0, 2024-09-11T08:17:51.019 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T08:17:51.019 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T08:17:51.019 INFO:teuthology.orchestra.run.smithi150.stdout: "object_hash": 2, 2024-09-11T08:17:51.019 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_autoscale_mode": "off", 2024-09-11T08:17:51.019 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num": 32, 2024-09-11T08:17:51.020 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num": 32, 2024-09-11T08:17:51.020 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num_target": 32, 2024-09-11T08:17:51.020 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_target": 32, 2024-09-11T08:17:51.020 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_pending": 32, 2024-09-11T08:17:51.020 INFO:teuthology.orchestra.run.smithi150.stdout: "last_pg_merge_meta": { 2024-09-11T08:17:51.020 INFO:teuthology.orchestra.run.smithi150.stdout: "source_pgid": "0.0", 2024-09-11T08:17:51.020 INFO:teuthology.orchestra.run.smithi150.stdout: "ready_epoch": 0, 2024-09-11T08:17:51.020 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_started": 0, 2024-09-11T08:17:51.020 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_clean": 0, 2024-09-11T08:17:51.020 INFO:teuthology.orchestra.run.smithi150.stdout: "source_version": "0'0", 2024-09-11T08:17:51.020 INFO:teuthology.orchestra.run.smithi150.stdout: "target_version": "0'0" 2024-09-11T08:17:51.020 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.020 INFO:teuthology.orchestra.run.smithi150.stdout: "last_change": "66", 2024-09-11T08:17:51.020 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend": "0", 2024-09-11T08:17:51.021 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T08:17:51.021 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T08:17:51.021 INFO:teuthology.orchestra.run.smithi150.stdout: "auid": 0, 2024-09-11T08:17:51.021 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_mode": "selfmanaged", 2024-09-11T08:17:51.021 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_seq": 0, 2024-09-11T08:17:51.021 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_epoch": 0, 2024-09-11T08:17:51.021 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_snaps": [], 2024-09-11T08:17:51.021 INFO:teuthology.orchestra.run.smithi150.stdout: "removed_snaps": "[]", 2024-09-11T08:17:51.021 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_bytes": 0, 2024-09-11T08:17:51.021 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_objects": 0, 2024-09-11T08:17:51.021 INFO:teuthology.orchestra.run.smithi150.stdout: "tiers": [], 2024-09-11T08:17:51.021 INFO:teuthology.orchestra.run.smithi150.stdout: "tier_of": -1, 2024-09-11T08:17:51.022 INFO:teuthology.orchestra.run.smithi150.stdout: "read_tier": -1, 2024-09-11T08:17:51.022 INFO:teuthology.orchestra.run.smithi150.stdout: "write_tier": -1, 2024-09-11T08:17:51.022 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_mode": "none", 2024-09-11T08:17:51.022 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_bytes": 0, 2024-09-11T08:17:51.022 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_objects": 0, 2024-09-11T08:17:51.022 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T08:17:51.022 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T08:17:51.022 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T08:17:51.022 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_flush_age": 0, 2024-09-11T08:17:51.022 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_evict_age": 0, 2024-09-11T08:17:51.022 INFO:teuthology.orchestra.run.smithi150.stdout: "erasure_code_profile": "", 2024-09-11T08:17:51.022 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_params": { 2024-09-11T08:17:51.022 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "none" 2024-09-11T08:17:51.022 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.022 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_period": 0, 2024-09-11T08:17:51.023 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_count": 0, 2024-09-11T08:17:51.023 INFO:teuthology.orchestra.run.smithi150.stdout: "use_gmt_hitset": true, 2024-09-11T08:17:51.023 INFO:teuthology.orchestra.run.smithi150.stdout: "min_read_recency_for_promote": 0, 2024-09-11T08:17:51.023 INFO:teuthology.orchestra.run.smithi150.stdout: "min_write_recency_for_promote": 0, 2024-09-11T08:17:51.023 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T08:17:51.023 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_search_last_n": 0, 2024-09-11T08:17:51.023 INFO:teuthology.orchestra.run.smithi150.stdout: "grade_table": [], 2024-09-11T08:17:51.023 INFO:teuthology.orchestra.run.smithi150.stdout: "stripe_width": 0, 2024-09-11T08:17:51.023 INFO:teuthology.orchestra.run.smithi150.stdout: "expected_num_objects": 0, 2024-09-11T08:17:51.023 INFO:teuthology.orchestra.run.smithi150.stdout: "fast_read": false, 2024-09-11T08:17:51.023 INFO:teuthology.orchestra.run.smithi150.stdout: "options": { 2024-09-11T08:17:51.023 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_autoscale_bias": 4 2024-09-11T08:17:51.023 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.023 INFO:teuthology.orchestra.run.smithi150.stdout: "application_metadata": { 2024-09-11T08:17:51.023 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw": {} 2024-09-11T08:17:51.024 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:51.024 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:51.024 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-09-11T08:17:51.024 INFO:teuthology.orchestra.run.smithi150.stdout: "osds": [ 2024-09-11T08:17:51.024 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.024 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 0, 2024-09-11T08:17:51.024 INFO:teuthology.orchestra.run.smithi150.stdout: "uuid": "37b63db9-ed2a-4701-b63d-03ff9885cc0e", 2024-09-11T08:17:51.024 INFO:teuthology.orchestra.run.smithi150.stdout: "up": 1, 2024-09-11T08:17:51.024 INFO:teuthology.orchestra.run.smithi150.stdout: "in": 1, 2024-09-11T08:17:51.024 INFO:teuthology.orchestra.run.smithi150.stdout: "weight": 1, 2024-09-11T08:17:51.024 INFO:teuthology.orchestra.run.smithi150.stdout: "primary_affinity": 1, 2024-09-11T08:17:51.024 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_begin": 0, 2024-09-11T08:17:51.024 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_end": 0, 2024-09-11T08:17:51.025 INFO:teuthology.orchestra.run.smithi150.stdout: "up_from": 8, 2024-09-11T08:17:51.025 INFO:teuthology.orchestra.run.smithi150.stdout: "up_thru": 238, 2024-09-11T08:17:51.025 INFO:teuthology.orchestra.run.smithi150.stdout: "down_at": 0, 2024-09-11T08:17:51.025 INFO:teuthology.orchestra.run.smithi150.stdout: "lost_at": 0, 2024-09-11T08:17:51.025 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addrs": { 2024-09-11T08:17:51.025 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:17:51.025 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.025 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:17:51.025 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6801", 2024-09-11T08:17:51.025 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 334255279 2024-09-11T08:17:51.025 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:51.025 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:51.025 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.026 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addrs": { 2024-09-11T08:17:51.026 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:17:51.026 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.026 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:17:51.026 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6802", 2024-09-11T08:17:51.026 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 334255279 2024-09-11T08:17:51.026 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:51.026 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:51.026 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.026 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addrs": { 2024-09-11T08:17:51.026 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:17:51.026 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.026 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:17:51.026 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6804", 2024-09-11T08:17:51.026 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 334255279 2024-09-11T08:17:51.027 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:51.027 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:51.027 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.027 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addrs": { 2024-09-11T08:17:51.027 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:17:51.027 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.027 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:17:51.027 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6803", 2024-09-11T08:17:51.027 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 334255279 2024-09-11T08:17:51.027 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:51.027 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:51.027 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.027 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addr": "172.21.15.150:6801/334255279", 2024-09-11T08:17:51.027 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addr": "172.21.15.150:6802/334255279", 2024-09-11T08:17:51.028 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addr": "172.21.15.150:6804/334255279", 2024-09-11T08:17:51.028 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addr": "172.21.15.150:6803/334255279", 2024-09-11T08:17:51.028 INFO:teuthology.orchestra.run.smithi150.stdout: "state": [ 2024-09-11T08:17:51.028 INFO:teuthology.orchestra.run.smithi150.stdout: "exists", 2024-09-11T08:17:51.028 INFO:teuthology.orchestra.run.smithi150.stdout: "up" 2024-09-11T08:17:51.028 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:51.028 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.028 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.028 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 1, 2024-09-11T08:17:51.028 INFO:teuthology.orchestra.run.smithi150.stdout: "uuid": "80378442-02bc-48aa-b8ca-6113a4092079", 2024-09-11T08:17:51.028 INFO:teuthology.orchestra.run.smithi150.stdout: "up": 1, 2024-09-11T08:17:51.028 INFO:teuthology.orchestra.run.smithi150.stdout: "in": 1, 2024-09-11T08:17:51.029 INFO:teuthology.orchestra.run.smithi150.stdout: "weight": 1, 2024-09-11T08:17:51.029 INFO:teuthology.orchestra.run.smithi150.stdout: "primary_affinity": 1, 2024-09-11T08:17:51.029 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_begin": 0, 2024-09-11T08:17:51.029 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_end": 0, 2024-09-11T08:17:51.029 INFO:teuthology.orchestra.run.smithi150.stdout: "up_from": 13, 2024-09-11T08:17:51.029 INFO:teuthology.orchestra.run.smithi150.stdout: "up_thru": 207, 2024-09-11T08:17:51.029 INFO:teuthology.orchestra.run.smithi150.stdout: "down_at": 0, 2024-09-11T08:17:51.029 INFO:teuthology.orchestra.run.smithi150.stdout: "lost_at": 0, 2024-09-11T08:17:51.029 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addrs": { 2024-09-11T08:17:51.029 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:17:51.029 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.029 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:17:51.030 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6805", 2024-09-11T08:17:51.030 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1129510619 2024-09-11T08:17:51.030 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:51.030 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:51.030 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.030 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addrs": { 2024-09-11T08:17:51.030 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:17:51.030 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.030 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:17:51.030 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6806", 2024-09-11T08:17:51.030 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1129510619 2024-09-11T08:17:51.030 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:51.030 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:51.031 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.031 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addrs": { 2024-09-11T08:17:51.031 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:17:51.031 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.031 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:17:51.031 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6808", 2024-09-11T08:17:51.031 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1129510619 2024-09-11T08:17:51.031 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:51.031 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:51.031 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.032 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addrs": { 2024-09-11T08:17:51.032 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:17:51.032 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.032 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:17:51.032 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6807", 2024-09-11T08:17:51.032 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1129510619 2024-09-11T08:17:51.032 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:51.032 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:51.032 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.032 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addr": "172.21.15.150:6805/1129510619", 2024-09-11T08:17:51.032 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addr": "172.21.15.150:6806/1129510619", 2024-09-11T08:17:51.032 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addr": "172.21.15.150:6808/1129510619", 2024-09-11T08:17:51.033 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addr": "172.21.15.150:6807/1129510619", 2024-09-11T08:17:51.033 INFO:teuthology.orchestra.run.smithi150.stdout: "state": [ 2024-09-11T08:17:51.033 INFO:teuthology.orchestra.run.smithi150.stdout: "exists", 2024-09-11T08:17:51.033 INFO:teuthology.orchestra.run.smithi150.stdout: "up" 2024-09-11T08:17:51.033 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:51.033 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.033 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.033 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 2, 2024-09-11T08:17:51.033 INFO:teuthology.orchestra.run.smithi150.stdout: "uuid": "0955bb92-36a0-4e9a-98e1-b3a5af89f6d2", 2024-09-11T08:17:51.033 INFO:teuthology.orchestra.run.smithi150.stdout: "up": 1, 2024-09-11T08:17:51.033 INFO:teuthology.orchestra.run.smithi150.stdout: "in": 1, 2024-09-11T08:17:51.033 INFO:teuthology.orchestra.run.smithi150.stdout: "weight": 1, 2024-09-11T08:17:51.033 INFO:teuthology.orchestra.run.smithi150.stdout: "primary_affinity": 1, 2024-09-11T08:17:51.033 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_begin": 0, 2024-09-11T08:17:51.034 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_end": 0, 2024-09-11T08:17:51.034 INFO:teuthology.orchestra.run.smithi150.stdout: "up_from": 18, 2024-09-11T08:17:51.034 INFO:teuthology.orchestra.run.smithi150.stdout: "up_thru": 238, 2024-09-11T08:17:51.034 INFO:teuthology.orchestra.run.smithi150.stdout: "down_at": 0, 2024-09-11T08:17:51.034 INFO:teuthology.orchestra.run.smithi150.stdout: "lost_at": 0, 2024-09-11T08:17:51.034 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addrs": { 2024-09-11T08:17:51.034 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:17:51.034 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.034 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:17:51.034 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6809", 2024-09-11T08:17:51.034 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 719975422 2024-09-11T08:17:51.034 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:51.034 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:51.034 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.034 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addrs": { 2024-09-11T08:17:51.035 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:17:51.035 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.035 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:17:51.035 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6810", 2024-09-11T08:17:51.035 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 719975422 2024-09-11T08:17:51.035 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:51.035 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:51.035 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.035 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addrs": { 2024-09-11T08:17:51.035 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:17:51.035 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.035 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:17:51.035 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6812", 2024-09-11T08:17:51.035 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 719975422 2024-09-11T08:17:51.035 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:51.036 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:51.036 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.036 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addrs": { 2024-09-11T08:17:51.036 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:17:51.036 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.036 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:17:51.036 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6811", 2024-09-11T08:17:51.036 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 719975422 2024-09-11T08:17:51.036 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:51.036 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:51.036 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.036 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addr": "172.21.15.150:6809/719975422", 2024-09-11T08:17:51.036 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addr": "172.21.15.150:6810/719975422", 2024-09-11T08:17:51.036 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addr": "172.21.15.150:6812/719975422", 2024-09-11T08:17:51.036 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addr": "172.21.15.150:6811/719975422", 2024-09-11T08:17:51.037 INFO:teuthology.orchestra.run.smithi150.stdout: "state": [ 2024-09-11T08:17:51.037 INFO:teuthology.orchestra.run.smithi150.stdout: "exists", 2024-09-11T08:17:51.037 INFO:teuthology.orchestra.run.smithi150.stdout: "up" 2024-09-11T08:17:51.037 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:51.037 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.037 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.037 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 3, 2024-09-11T08:17:51.037 INFO:teuthology.orchestra.run.smithi150.stdout: "uuid": "1d04d4d3-62e8-4460-8a07-77f7e5c62929", 2024-09-11T08:17:51.037 INFO:teuthology.orchestra.run.smithi150.stdout: "up": 1, 2024-09-11T08:17:51.037 INFO:teuthology.orchestra.run.smithi150.stdout: "in": 1, 2024-09-11T08:17:51.037 INFO:teuthology.orchestra.run.smithi150.stdout: "weight": 1, 2024-09-11T08:17:51.037 INFO:teuthology.orchestra.run.smithi150.stdout: "primary_affinity": 0.611114501953125, 2024-09-11T08:17:51.037 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_begin": 0, 2024-09-11T08:17:51.037 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_end": 0, 2024-09-11T08:17:51.038 INFO:teuthology.orchestra.run.smithi150.stdout: "up_from": 25, 2024-09-11T08:17:51.038 INFO:teuthology.orchestra.run.smithi150.stdout: "up_thru": 207, 2024-09-11T08:17:51.038 INFO:teuthology.orchestra.run.smithi150.stdout: "down_at": 0, 2024-09-11T08:17:51.038 INFO:teuthology.orchestra.run.smithi150.stdout: "lost_at": 0, 2024-09-11T08:17:51.038 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addrs": { 2024-09-11T08:17:51.038 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:17:51.038 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.038 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:17:51.038 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6813", 2024-09-11T08:17:51.038 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 3101032500 2024-09-11T08:17:51.038 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:51.039 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:51.039 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.039 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addrs": { 2024-09-11T08:17:51.039 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:17:51.039 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.039 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:17:51.039 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6814", 2024-09-11T08:17:51.039 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 3101032500 2024-09-11T08:17:51.039 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:51.039 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:51.039 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.039 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addrs": { 2024-09-11T08:17:51.040 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:17:51.040 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.040 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:17:51.040 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6816", 2024-09-11T08:17:51.040 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 3101032500 2024-09-11T08:17:51.040 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:51.040 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:51.040 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.040 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addrs": { 2024-09-11T08:17:51.040 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:17:51.040 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.040 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:17:51.041 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6815", 2024-09-11T08:17:51.041 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 3101032500 2024-09-11T08:17:51.041 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:51.041 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:51.041 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.041 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addr": "172.21.15.150:6813/3101032500", 2024-09-11T08:17:51.041 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addr": "172.21.15.150:6814/3101032500", 2024-09-11T08:17:51.041 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addr": "172.21.15.150:6816/3101032500", 2024-09-11T08:17:51.041 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addr": "172.21.15.150:6815/3101032500", 2024-09-11T08:17:51.041 INFO:teuthology.orchestra.run.smithi150.stdout: "state": [ 2024-09-11T08:17:51.041 INFO:teuthology.orchestra.run.smithi150.stdout: "exists", 2024-09-11T08:17:51.041 INFO:teuthology.orchestra.run.smithi150.stdout: "up" 2024-09-11T08:17:51.042 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:51.042 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.042 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.042 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 4, 2024-09-11T08:17:51.042 INFO:teuthology.orchestra.run.smithi150.stdout: "uuid": "c5321a3c-d2f0-4440-8475-2c111d261139", 2024-09-11T08:17:51.042 INFO:teuthology.orchestra.run.smithi150.stdout: "up": 1, 2024-09-11T08:17:51.042 INFO:teuthology.orchestra.run.smithi150.stdout: "in": 1, 2024-09-11T08:17:51.042 INFO:teuthology.orchestra.run.smithi150.stdout: "weight": 1, 2024-09-11T08:17:51.042 INFO:teuthology.orchestra.run.smithi150.stdout: "primary_affinity": 1, 2024-09-11T08:17:51.042 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_begin": 0, 2024-09-11T08:17:51.042 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_end": 0, 2024-09-11T08:17:51.042 INFO:teuthology.orchestra.run.smithi150.stdout: "up_from": 31, 2024-09-11T08:17:51.043 INFO:teuthology.orchestra.run.smithi150.stdout: "up_thru": 207, 2024-09-11T08:17:51.043 INFO:teuthology.orchestra.run.smithi150.stdout: "down_at": 0, 2024-09-11T08:17:51.043 INFO:teuthology.orchestra.run.smithi150.stdout: "lost_at": 0, 2024-09-11T08:17:51.043 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addrs": { 2024-09-11T08:17:51.043 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:17:51.043 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.043 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:17:51.043 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6800", 2024-09-11T08:17:51.043 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2567452825 2024-09-11T08:17:51.043 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:51.043 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:51.043 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.043 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addrs": { 2024-09-11T08:17:51.043 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:17:51.044 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.044 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:17:51.044 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6801", 2024-09-11T08:17:51.044 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2567452825 2024-09-11T08:17:51.044 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:51.044 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:51.044 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.044 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addrs": { 2024-09-11T08:17:51.044 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:17:51.044 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.044 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:17:51.044 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6803", 2024-09-11T08:17:51.044 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2567452825 2024-09-11T08:17:51.045 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:51.045 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:51.045 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.045 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addrs": { 2024-09-11T08:17:51.045 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:17:51.045 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.045 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:17:51.045 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6802", 2024-09-11T08:17:51.045 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2567452825 2024-09-11T08:17:51.045 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:51.046 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:51.046 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.046 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addr": "172.21.15.181:6800/2567452825", 2024-09-11T08:17:51.046 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addr": "172.21.15.181:6801/2567452825", 2024-09-11T08:17:51.046 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addr": "172.21.15.181:6803/2567452825", 2024-09-11T08:17:51.046 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addr": "172.21.15.181:6802/2567452825", 2024-09-11T08:17:51.046 INFO:teuthology.orchestra.run.smithi150.stdout: "state": [ 2024-09-11T08:17:51.046 INFO:teuthology.orchestra.run.smithi150.stdout: "exists", 2024-09-11T08:17:51.046 INFO:teuthology.orchestra.run.smithi150.stdout: "up" 2024-09-11T08:17:51.046 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:51.046 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.046 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.046 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 5, 2024-09-11T08:17:51.046 INFO:teuthology.orchestra.run.smithi150.stdout: "uuid": "1262b2fe-f253-4a26-bfd4-c9dffb697c77", 2024-09-11T08:17:51.046 INFO:teuthology.orchestra.run.smithi150.stdout: "up": 1, 2024-09-11T08:17:51.047 INFO:teuthology.orchestra.run.smithi150.stdout: "in": 1, 2024-09-11T08:17:51.047 INFO:teuthology.orchestra.run.smithi150.stdout: "weight": 1, 2024-09-11T08:17:51.047 INFO:teuthology.orchestra.run.smithi150.stdout: "primary_affinity": 1, 2024-09-11T08:17:51.047 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_begin": 0, 2024-09-11T08:17:51.047 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_end": 0, 2024-09-11T08:17:51.047 INFO:teuthology.orchestra.run.smithi150.stdout: "up_from": 38, 2024-09-11T08:17:51.047 INFO:teuthology.orchestra.run.smithi150.stdout: "up_thru": 238, 2024-09-11T08:17:51.047 INFO:teuthology.orchestra.run.smithi150.stdout: "down_at": 0, 2024-09-11T08:17:51.047 INFO:teuthology.orchestra.run.smithi150.stdout: "lost_at": 0, 2024-09-11T08:17:51.047 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addrs": { 2024-09-11T08:17:51.047 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:17:51.047 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.047 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:17:51.047 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6804", 2024-09-11T08:17:51.047 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2043435508 2024-09-11T08:17:51.048 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:51.048 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:51.048 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.048 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addrs": { 2024-09-11T08:17:51.048 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:17:51.048 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.048 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:17:51.048 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6805", 2024-09-11T08:17:51.048 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2043435508 2024-09-11T08:17:51.048 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:51.048 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:51.048 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.048 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addrs": { 2024-09-11T08:17:51.049 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:17:51.049 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.049 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:17:51.049 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6807", 2024-09-11T08:17:51.049 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2043435508 2024-09-11T08:17:51.049 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:51.049 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:51.049 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.049 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addrs": { 2024-09-11T08:17:51.049 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:17:51.049 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.049 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:17:51.049 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6806", 2024-09-11T08:17:51.049 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2043435508 2024-09-11T08:17:51.050 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:51.050 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:51.050 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.050 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addr": "172.21.15.181:6804/2043435508", 2024-09-11T08:17:51.050 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addr": "172.21.15.181:6805/2043435508", 2024-09-11T08:17:51.050 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addr": "172.21.15.181:6807/2043435508", 2024-09-11T08:17:51.050 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addr": "172.21.15.181:6806/2043435508", 2024-09-11T08:17:51.050 INFO:teuthology.orchestra.run.smithi150.stdout: "state": [ 2024-09-11T08:17:51.050 INFO:teuthology.orchestra.run.smithi150.stdout: "exists", 2024-09-11T08:17:51.050 INFO:teuthology.orchestra.run.smithi150.stdout: "up" 2024-09-11T08:17:51.050 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:51.050 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.050 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.050 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 6, 2024-09-11T08:17:51.050 INFO:teuthology.orchestra.run.smithi150.stdout: "uuid": "7facca22-b3f1-4c20-833e-e3db798a08c4", 2024-09-11T08:17:51.051 INFO:teuthology.orchestra.run.smithi150.stdout: "up": 1, 2024-09-11T08:17:51.051 INFO:teuthology.orchestra.run.smithi150.stdout: "in": 1, 2024-09-11T08:17:51.051 INFO:teuthology.orchestra.run.smithi150.stdout: "weight": 1, 2024-09-11T08:17:51.051 INFO:teuthology.orchestra.run.smithi150.stdout: "primary_affinity": 1, 2024-09-11T08:17:51.051 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_begin": 0, 2024-09-11T08:17:51.051 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_end": 0, 2024-09-11T08:17:51.051 INFO:teuthology.orchestra.run.smithi150.stdout: "up_from": 45, 2024-09-11T08:17:51.052 INFO:teuthology.orchestra.run.smithi150.stdout: "up_thru": 238, 2024-09-11T08:17:51.052 INFO:teuthology.orchestra.run.smithi150.stdout: "down_at": 0, 2024-09-11T08:17:51.052 INFO:teuthology.orchestra.run.smithi150.stdout: "lost_at": 0, 2024-09-11T08:17:51.052 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addrs": { 2024-09-11T08:17:51.052 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:17:51.052 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.052 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:17:51.052 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6808", 2024-09-11T08:17:51.052 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1789880331 2024-09-11T08:17:51.052 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:51.052 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:51.053 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.053 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addrs": { 2024-09-11T08:17:51.053 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:17:51.053 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.053 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:17:51.053 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6809", 2024-09-11T08:17:51.053 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1789880331 2024-09-11T08:17:51.053 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:51.053 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:51.053 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.053 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addrs": { 2024-09-11T08:17:51.053 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:17:51.053 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.054 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:17:51.054 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6811", 2024-09-11T08:17:51.054 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1789880331 2024-09-11T08:17:51.054 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:51.054 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:51.054 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.054 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addrs": { 2024-09-11T08:17:51.054 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:17:51.054 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.054 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:17:51.054 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6810", 2024-09-11T08:17:51.054 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1789880331 2024-09-11T08:17:51.054 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:51.054 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:51.054 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.054 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addr": "172.21.15.181:6808/1789880331", 2024-09-11T08:17:51.055 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addr": "172.21.15.181:6809/1789880331", 2024-09-11T08:17:51.055 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addr": "172.21.15.181:6811/1789880331", 2024-09-11T08:17:51.055 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addr": "172.21.15.181:6810/1789880331", 2024-09-11T08:17:51.055 INFO:teuthology.orchestra.run.smithi150.stdout: "state": [ 2024-09-11T08:17:51.055 INFO:teuthology.orchestra.run.smithi150.stdout: "exists", 2024-09-11T08:17:51.055 INFO:teuthology.orchestra.run.smithi150.stdout: "up" 2024-09-11T08:17:51.055 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:51.055 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.055 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.055 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 7, 2024-09-11T08:17:51.055 INFO:teuthology.orchestra.run.smithi150.stdout: "uuid": "6ac1b06a-79bd-43ca-b91e-fcb0f4033a34", 2024-09-11T08:17:51.055 INFO:teuthology.orchestra.run.smithi150.stdout: "up": 1, 2024-09-11T08:17:51.055 INFO:teuthology.orchestra.run.smithi150.stdout: "in": 1, 2024-09-11T08:17:51.055 INFO:teuthology.orchestra.run.smithi150.stdout: "weight": 1, 2024-09-11T08:17:51.056 INFO:teuthology.orchestra.run.smithi150.stdout: "primary_affinity": 1, 2024-09-11T08:17:51.056 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_begin": 53, 2024-09-11T08:17:51.056 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_end": 139, 2024-09-11T08:17:51.056 INFO:teuthology.orchestra.run.smithi150.stdout: "up_from": 168, 2024-09-11T08:17:51.056 INFO:teuthology.orchestra.run.smithi150.stdout: "up_thru": 238, 2024-09-11T08:17:51.056 INFO:teuthology.orchestra.run.smithi150.stdout: "down_at": 140, 2024-09-11T08:17:51.056 INFO:teuthology.orchestra.run.smithi150.stdout: "lost_at": 0, 2024-09-11T08:17:51.056 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addrs": { 2024-09-11T08:17:51.056 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:17:51.056 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.056 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:17:51.056 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6812", 2024-09-11T08:17:51.057 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2706455857 2024-09-11T08:17:51.057 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:51.057 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:51.057 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.057 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addrs": { 2024-09-11T08:17:51.057 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:17:51.057 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.057 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:17:51.057 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6813", 2024-09-11T08:17:51.057 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2706455857 2024-09-11T08:17:51.057 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:51.057 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:51.057 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.058 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addrs": { 2024-09-11T08:17:51.058 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:17:51.058 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.058 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:17:51.058 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6815", 2024-09-11T08:17:51.058 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2706455857 2024-09-11T08:17:51.058 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:51.058 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:51.058 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.058 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addrs": { 2024-09-11T08:17:51.058 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:17:51.058 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.058 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:17:51.058 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6814", 2024-09-11T08:17:51.059 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2706455857 2024-09-11T08:17:51.059 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:51.059 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:51.059 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.059 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addr": "172.21.15.181:6812/2706455857", 2024-09-11T08:17:51.059 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addr": "172.21.15.181:6813/2706455857", 2024-09-11T08:17:51.059 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addr": "172.21.15.181:6815/2706455857", 2024-09-11T08:17:51.059 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addr": "172.21.15.181:6814/2706455857", 2024-09-11T08:17:51.059 INFO:teuthology.orchestra.run.smithi150.stdout: "state": [ 2024-09-11T08:17:51.059 INFO:teuthology.orchestra.run.smithi150.stdout: "exists", 2024-09-11T08:17:51.059 INFO:teuthology.orchestra.run.smithi150.stdout: "up" 2024-09-11T08:17:51.059 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:51.059 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:51.059 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-09-11T08:17:51.060 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_xinfo": [ 2024-09-11T08:17:51.060 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.060 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 0, 2024-09-11T08:17:51.060 INFO:teuthology.orchestra.run.smithi150.stdout: "down_stamp": "0.000000", 2024-09-11T08:17:51.060 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_probability": 0, 2024-09-11T08:17:51.060 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_interval": 0, 2024-09-11T08:17:51.060 INFO:teuthology.orchestra.run.smithi150.stdout: "features": 4540138320759226367, 2024-09-11T08:17:51.060 INFO:teuthology.orchestra.run.smithi150.stdout: "old_weight": 0, 2024-09-11T08:17:51.060 INFO:teuthology.orchestra.run.smithi150.stdout: "last_purged_snaps_scrub": "2024-09-11T08:17:30.741804+0000", 2024-09-11T08:17:51.060 INFO:teuthology.orchestra.run.smithi150.stdout: "dead_epoch": 0 2024-09-11T08:17:51.060 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.060 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.060 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 1, 2024-09-11T08:17:51.060 INFO:teuthology.orchestra.run.smithi150.stdout: "down_stamp": "0.000000", 2024-09-11T08:17:51.061 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_probability": 0, 2024-09-11T08:17:51.061 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_interval": 0, 2024-09-11T08:17:51.061 INFO:teuthology.orchestra.run.smithi150.stdout: "features": 4540138320759226367, 2024-09-11T08:17:51.061 INFO:teuthology.orchestra.run.smithi150.stdout: "old_weight": 0, 2024-09-11T08:17:51.061 INFO:teuthology.orchestra.run.smithi150.stdout: "last_purged_snaps_scrub": "2024-09-11T08:17:19.773293+0000", 2024-09-11T08:17:51.061 INFO:teuthology.orchestra.run.smithi150.stdout: "dead_epoch": 0 2024-09-11T08:17:51.061 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.061 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.061 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 2, 2024-09-11T08:17:51.061 INFO:teuthology.orchestra.run.smithi150.stdout: "down_stamp": "0.000000", 2024-09-11T08:17:51.061 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_probability": 0, 2024-09-11T08:17:51.061 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_interval": 0, 2024-09-11T08:17:51.062 INFO:teuthology.orchestra.run.smithi150.stdout: "features": 4540138320759226367, 2024-09-11T08:17:51.062 INFO:teuthology.orchestra.run.smithi150.stdout: "old_weight": 0, 2024-09-11T08:17:51.062 INFO:teuthology.orchestra.run.smithi150.stdout: "last_purged_snaps_scrub": "2024-09-11T08:16:38.895061+0000", 2024-09-11T08:17:51.062 INFO:teuthology.orchestra.run.smithi150.stdout: "dead_epoch": 0 2024-09-11T08:17:51.062 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.062 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.062 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 3, 2024-09-11T08:17:51.062 INFO:teuthology.orchestra.run.smithi150.stdout: "down_stamp": "0.000000", 2024-09-11T08:17:51.062 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_probability": 0, 2024-09-11T08:17:51.062 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_interval": 0, 2024-09-11T08:17:51.062 INFO:teuthology.orchestra.run.smithi150.stdout: "features": 4540138320759226367, 2024-09-11T08:17:51.062 INFO:teuthology.orchestra.run.smithi150.stdout: "old_weight": 0, 2024-09-11T08:17:51.062 INFO:teuthology.orchestra.run.smithi150.stdout: "last_purged_snaps_scrub": "2024-09-11T08:17:21.516348+0000", 2024-09-11T08:17:51.062 INFO:teuthology.orchestra.run.smithi150.stdout: "dead_epoch": 0 2024-09-11T08:17:51.062 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.063 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.063 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 4, 2024-09-11T08:17:51.063 INFO:teuthology.orchestra.run.smithi150.stdout: "down_stamp": "0.000000", 2024-09-11T08:17:51.063 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_probability": 0, 2024-09-11T08:17:51.063 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_interval": 0, 2024-09-11T08:17:51.063 INFO:teuthology.orchestra.run.smithi150.stdout: "features": 4540138320759226367, 2024-09-11T08:17:51.063 INFO:teuthology.orchestra.run.smithi150.stdout: "old_weight": 0, 2024-09-11T08:17:51.063 INFO:teuthology.orchestra.run.smithi150.stdout: "last_purged_snaps_scrub": "2024-09-11T08:16:52.243024+0000", 2024-09-11T08:17:51.063 INFO:teuthology.orchestra.run.smithi150.stdout: "dead_epoch": 0 2024-09-11T08:17:51.063 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.063 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.063 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 5, 2024-09-11T08:17:51.064 INFO:teuthology.orchestra.run.smithi150.stdout: "down_stamp": "0.000000", 2024-09-11T08:17:51.064 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_probability": 0, 2024-09-11T08:17:51.064 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_interval": 0, 2024-09-11T08:17:51.064 INFO:teuthology.orchestra.run.smithi150.stdout: "features": 4540138320759226367, 2024-09-11T08:17:51.064 INFO:teuthology.orchestra.run.smithi150.stdout: "old_weight": 0, 2024-09-11T08:17:51.064 INFO:teuthology.orchestra.run.smithi150.stdout: "last_purged_snaps_scrub": "2024-09-11T08:17:42.723306+0000", 2024-09-11T08:17:51.064 INFO:teuthology.orchestra.run.smithi150.stdout: "dead_epoch": 0 2024-09-11T08:17:51.064 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.064 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.064 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 6, 2024-09-11T08:17:51.064 INFO:teuthology.orchestra.run.smithi150.stdout: "down_stamp": "0.000000", 2024-09-11T08:17:51.064 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_probability": 0, 2024-09-11T08:17:51.064 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_interval": 0, 2024-09-11T08:17:51.064 INFO:teuthology.orchestra.run.smithi150.stdout: "features": 4540138320759226367, 2024-09-11T08:17:51.065 INFO:teuthology.orchestra.run.smithi150.stdout: "old_weight": 0, 2024-09-11T08:17:51.065 INFO:teuthology.orchestra.run.smithi150.stdout: "last_purged_snaps_scrub": "2024-09-11T08:16:53.143364+0000", 2024-09-11T08:17:51.065 INFO:teuthology.orchestra.run.smithi150.stdout: "dead_epoch": 0 2024-09-11T08:17:51.065 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.065 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.065 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 7, 2024-09-11T08:17:51.065 INFO:teuthology.orchestra.run.smithi150.stdout: "down_stamp": "0.000000", 2024-09-11T08:17:51.065 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_probability": 0, 2024-09-11T08:17:51.065 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_interval": 0, 2024-09-11T08:17:51.065 INFO:teuthology.orchestra.run.smithi150.stdout: "features": 4540138320759226367, 2024-09-11T08:17:51.065 INFO:teuthology.orchestra.run.smithi150.stdout: "old_weight": 0, 2024-09-11T08:17:51.065 INFO:teuthology.orchestra.run.smithi150.stdout: "last_purged_snaps_scrub": "2024-09-11T08:16:29.930906+0000", 2024-09-11T08:17:51.065 INFO:teuthology.orchestra.run.smithi150.stdout: "dead_epoch": 139 2024-09-11T08:17:51.065 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:51.066 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-09-11T08:17:51.066 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_upmap": [], 2024-09-11T08:17:51.066 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_upmap_items": [ 2024-09-11T08:17:51.066 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.066 INFO:teuthology.orchestra.run.smithi150.stdout: "pgid": "4.4", 2024-09-11T08:17:51.066 INFO:teuthology.orchestra.run.smithi150.stdout: "mappings": [ 2024-09-11T08:17:51.066 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.066 INFO:teuthology.orchestra.run.smithi150.stdout: "from": 1, 2024-09-11T08:17:51.066 INFO:teuthology.orchestra.run.smithi150.stdout: "to": 0 2024-09-11T08:17:51.066 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:51.067 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:51.067 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.067 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.067 INFO:teuthology.orchestra.run.smithi150.stdout: "pgid": "4.d", 2024-09-11T08:17:51.067 INFO:teuthology.orchestra.run.smithi150.stdout: "mappings": [ 2024-09-11T08:17:51.067 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.067 INFO:teuthology.orchestra.run.smithi150.stdout: "from": 1, 2024-09-11T08:17:51.067 INFO:teuthology.orchestra.run.smithi150.stdout: "to": 5 2024-09-11T08:17:51.067 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:51.067 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:51.067 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.067 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.067 INFO:teuthology.orchestra.run.smithi150.stdout: "pgid": "4.14", 2024-09-11T08:17:51.067 INFO:teuthology.orchestra.run.smithi150.stdout: "mappings": [ 2024-09-11T08:17:51.067 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.068 INFO:teuthology.orchestra.run.smithi150.stdout: "from": 1, 2024-09-11T08:17:51.068 INFO:teuthology.orchestra.run.smithi150.stdout: "to": 2 2024-09-11T08:17:51.068 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:51.068 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:51.068 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.068 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.068 INFO:teuthology.orchestra.run.smithi150.stdout: "pgid": "4.1d", 2024-09-11T08:17:51.068 INFO:teuthology.orchestra.run.smithi150.stdout: "mappings": [ 2024-09-11T08:17:51.068 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.068 INFO:teuthology.orchestra.run.smithi150.stdout: "from": 1, 2024-09-11T08:17:51.068 INFO:teuthology.orchestra.run.smithi150.stdout: "to": 2 2024-09-11T08:17:51.068 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:51.068 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:51.068 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.069 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.069 INFO:teuthology.orchestra.run.smithi150.stdout: "pgid": "5.3", 2024-09-11T08:17:51.069 INFO:teuthology.orchestra.run.smithi150.stdout: "mappings": [ 2024-09-11T08:17:51.069 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.069 INFO:teuthology.orchestra.run.smithi150.stdout: "from": 0, 2024-09-11T08:17:51.069 INFO:teuthology.orchestra.run.smithi150.stdout: "to": 3 2024-09-11T08:17:51.069 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:51.069 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:51.069 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.069 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.069 INFO:teuthology.orchestra.run.smithi150.stdout: "pgid": "5.1e", 2024-09-11T08:17:51.069 INFO:teuthology.orchestra.run.smithi150.stdout: "mappings": [ 2024-09-11T08:17:51.069 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:17:51.069 INFO:teuthology.orchestra.run.smithi150.stdout: "from": 0, 2024-09-11T08:17:51.070 INFO:teuthology.orchestra.run.smithi150.stdout: "to": 6 2024-09-11T08:17:51.070 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:51.070 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:17:51.070 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:51.070 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-09-11T08:17:51.070 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_temp": [], 2024-09-11T08:17:51.070 INFO:teuthology.orchestra.run.smithi150.stdout: "primary_temp": [], 2024-09-11T08:17:51.070 INFO:teuthology.orchestra.run.smithi150.stdout: "blocklist": { 2024-09-11T08:17:51.070 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/2620931070": "2024-09-12T08:08:06.649692+0000", 2024-09-11T08:17:51.070 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/1073581409": "2024-09-12T08:02:45.986151+0000", 2024-09-11T08:17:51.070 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/2363649697": "2024-09-12T08:08:06.649692+0000", 2024-09-11T08:17:51.070 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:6800/603451071": "2024-09-12T08:02:45.986151+0000", 2024-09-11T08:17:51.070 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/1614413548": "2024-09-12T08:02:45.986151+0000", 2024-09-11T08:17:51.070 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/3673218166": "2024-09-12T08:03:08.740828+0000", 2024-09-11T08:17:51.070 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:6800/556498589": "2024-09-12T08:03:08.740828+0000", 2024-09-11T08:17:51.071 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/3666733994": "2024-09-12T08:03:08.740828+0000", 2024-09-11T08:17:51.071 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/3271479010": "2024-09-12T08:03:08.740828+0000", 2024-09-11T08:17:51.071 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/3185731090": "2024-09-12T08:08:06.649692+0000", 2024-09-11T08:17:51.071 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/2539412006": "2024-09-12T08:08:06.649692+0000", 2024-09-11T08:17:51.071 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/2740222408": "2024-09-12T08:02:45.986151+0000", 2024-09-11T08:17:51.071 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:6800/127856313": "2024-09-12T08:08:06.649692+0000", 2024-09-11T08:17:51.071 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/2375121603": "2024-09-12T08:08:06.649692+0000" 2024-09-11T08:17:51.071 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.071 INFO:teuthology.orchestra.run.smithi150.stdout: "range_blocklist": {}, 2024-09-11T08:17:51.071 INFO:teuthology.orchestra.run.smithi150.stdout: "erasure_code_profiles": { 2024-09-11T08:17:51.071 INFO:teuthology.orchestra.run.smithi150.stdout: "default": { 2024-09-11T08:17:51.071 INFO:teuthology.orchestra.run.smithi150.stdout: "crush-failure-domain": "osd", 2024-09-11T08:17:51.071 INFO:teuthology.orchestra.run.smithi150.stdout: "k": "2", 2024-09-11T08:17:51.072 INFO:teuthology.orchestra.run.smithi150.stdout: "m": "1", 2024-09-11T08:17:51.072 INFO:teuthology.orchestra.run.smithi150.stdout: "plugin": "jerasure", 2024-09-11T08:17:51.072 INFO:teuthology.orchestra.run.smithi150.stdout: "technique": "reed_sol_van" 2024-09-11T08:17:51.072 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:51.072 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:17:51.072 INFO:teuthology.orchestra.run.smithi150.stdout: "removed_snaps_queue": [], 2024-09-11T08:17:51.072 INFO:teuthology.orchestra.run.smithi150.stdout: "new_removed_snaps": [], 2024-09-11T08:17:51.072 INFO:teuthology.orchestra.run.smithi150.stdout: "new_purged_snaps": [], 2024-09-11T08:17:51.072 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_node_flags": {}, 2024-09-11T08:17:51.072 INFO:teuthology.orchestra.run.smithi150.stdout: "device_class_flags": {}, 2024-09-11T08:17:51.072 INFO:teuthology.orchestra.run.smithi150.stdout: "stretch_mode": { 2024-09-11T08:17:51.072 INFO:teuthology.orchestra.run.smithi150.stdout: "stretch_mode_enabled": false, 2024-09-11T08:17:51.073 INFO:teuthology.orchestra.run.smithi150.stdout: "stretch_bucket_count": 0, 2024-09-11T08:17:51.073 INFO:teuthology.orchestra.run.smithi150.stdout: "degraded_stretch_mode": 0, 2024-09-11T08:17:51.073 INFO:teuthology.orchestra.run.smithi150.stdout: "recovering_stretch_mode": 0, 2024-09-11T08:17:51.073 INFO:teuthology.orchestra.run.smithi150.stdout: "stretch_mode_bucket": 0 2024-09-11T08:17:51.073 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:17:51.073 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-11T08:17:51.073 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:17:51.095 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:51.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:51.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:17:51.153 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-11T08:17:51.154 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:17:51.196 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:51.199 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:17:51.222 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:51.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:51.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:51.324 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:51.346 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:51.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:51.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:51.448 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:17:51.469 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:51.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:51.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:17:51.571 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:51.593 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:51.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:51.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:51.695 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:51.717 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:51.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:51.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:51.808 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:51 smithi181 ceph-mon[32701]: 8.d scrub starts 2024-09-11T08:17:51.808 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:51 smithi181 ceph-mon[32701]: 8.d scrub ok 2024-09-11T08:17:51.808 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:51 smithi181 ceph-mon[32701]: 4.10 scrub starts 2024-09-11T08:17:51.808 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:51 smithi181 ceph-mon[32701]: 4.10 scrub ok 2024-09-11T08:17:51.809 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:51 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-11T08:17:51.809 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:51 smithi181 ceph-mon[32701]: osdmap e242: 8 total, 8 up, 8 in 2024-09-11T08:17:51.809 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:51 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2002899437' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-11T08:17:51.809 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:51 smithi181 ceph-mon[32701]: pgmap v464: 132 pgs: 132 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-11T08:17:51.809 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:51 smithi181 ceph-mon[32701]: 4.4 scrub starts 2024-09-11T08:17:51.809 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:51 smithi181 ceph-mon[32701]: 4.4 scrub ok 2024-09-11T08:17:51.809 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:51 smithi181 ceph-mon[32701]: 4.1b scrub starts 2024-09-11T08:17:51.818 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:17:51.832 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-11T08:17:51.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:51.840 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:51.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:17:51.942 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:51.964 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:51.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:51.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:51.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:51 smithi150 ceph-mon[29276]: 8.d scrub starts 2024-09-11T08:17:51.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:51 smithi150 ceph-mon[29276]: 8.d scrub ok 2024-09-11T08:17:51.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:51 smithi150 ceph-mon[29276]: 4.10 scrub starts 2024-09-11T08:17:51.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:51 smithi150 ceph-mon[29276]: 4.10 scrub ok 2024-09-11T08:17:51.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:51 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-11T08:17:51.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:51 smithi150 ceph-mon[29276]: osdmap e242: 8 total, 8 up, 8 in 2024-09-11T08:17:51.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:51 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2002899437' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-11T08:17:51.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:51 smithi150 ceph-mon[29276]: pgmap v464: 132 pgs: 132 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-11T08:17:51.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:51 smithi150 ceph-mon[29276]: 4.4 scrub starts 2024-09-11T08:17:51.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:51 smithi150 ceph-mon[29276]: 4.4 scrub ok 2024-09-11T08:17:51.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:51 smithi150 ceph-mon[29276]: 4.1b scrub starts 2024-09-11T08:17:51.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:51 smithi150 ceph-mon[39612]: 8.d scrub starts 2024-09-11T08:17:51.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:51 smithi150 ceph-mon[39612]: 8.d scrub ok 2024-09-11T08:17:51.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:51 smithi150 ceph-mon[39612]: 4.10 scrub starts 2024-09-11T08:17:51.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:51 smithi150 ceph-mon[39612]: 4.10 scrub ok 2024-09-11T08:17:51.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:51 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-11T08:17:51.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:51 smithi150 ceph-mon[39612]: osdmap e242: 8 total, 8 up, 8 in 2024-09-11T08:17:51.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:51 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2002899437' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-11T08:17:51.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:51 smithi150 ceph-mon[39612]: pgmap v464: 132 pgs: 132 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-11T08:17:51.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:51 smithi150 ceph-mon[39612]: 4.4 scrub starts 2024-09-11T08:17:51.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:51 smithi150 ceph-mon[39612]: 4.4 scrub ok 2024-09-11T08:17:51.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:51 smithi150 ceph-mon[39612]: 4.1b scrub starts 2024-09-11T08:17:52.024 INFO:tasks.thrashosds.thrasher:j is {'epoch': 242, 'fsid': 'c1bddff4-7013-11ef-bcea-c7b262605968', 'created': '2024-09-11T08:02:15.344618+0000', 'modified': '2024-09-11T08:17:50.600744+0000', 'last_up_change': '2024-09-11T08:14:23.745449+0000', 'last_in_change': '2024-09-11T08:07:00.805384+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 8, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-11T08:05:21.044729+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-11T08:07:28.786572+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 63, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-11T08:07:29.021286+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-11T08:07:30.334181+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-11T08:07:32.289626+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-11T08:07:34.351861+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}], 'osds': [{'osd': 0, 'uuid': '37b63db9-ed2a-4701-b63d-03ff9885cc0e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 238, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6801', 'nonce': 334255279}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6802', 'nonce': 334255279}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6804', 'nonce': 334255279}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6803', 'nonce': 334255279}]}, 'public_addr': '172.21.15.150:6801/334255279', 'cluster_addr': '172.21.15.150:6802/334255279', 'heartbeat_back_addr': '172.21.15.150:6804/334255279', 'heartbeat_front_addr': '172.21.15.150:6803/334255279', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '80378442-02bc-48aa-b8ca-6113a4092079', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 207, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6805', 'nonce': 1129510619}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6806', 'nonce': 1129510619}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6808', 'nonce': 1129510619}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6807', 'nonce': 1129510619}]}, 'public_addr': '172.21.15.150:6805/1129510619', 'cluster_addr': '172.21.15.150:6806/1129510619', 'heartbeat_back_addr': '172.21.15.150:6808/1129510619', 'heartbeat_front_addr': '172.21.15.150:6807/1129510619', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '0955bb92-36a0-4e9a-98e1-b3a5af89f6d2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 238, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6809', 'nonce': 719975422}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6810', 'nonce': 719975422}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6812', 'nonce': 719975422}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6811', 'nonce': 719975422}]}, 'public_addr': '172.21.15.150:6809/719975422', 'cluster_addr': '172.21.15.150:6810/719975422', 'heartbeat_back_addr': '172.21.15.150:6812/719975422', 'heartbeat_front_addr': '172.21.15.150:6811/719975422', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '1d04d4d3-62e8-4460-8a07-77f7e5c62929', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.611114501953125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 207, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6813', 'nonce': 3101032500}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6814', 'nonce': 3101032500}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6816', 'nonce': 3101032500}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6815', 'nonce': 3101032500}]}, 'public_addr': '172.21.15.150:6813/3101032500', 'cluster_addr': '172.21.15.150:6814/3101032500', 'heartbeat_back_addr': '172.21.15.150:6816/3101032500', 'heartbeat_front_addr': '172.21.15.150:6815/3101032500', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'c5321a3c-d2f0-4440-8475-2c111d261139', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 207, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6800', 'nonce': 2567452825}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6801', 'nonce': 2567452825}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6803', 'nonce': 2567452825}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6802', 'nonce': 2567452825}]}, 'public_addr': '172.21.15.181:6800/2567452825', 'cluster_addr': '172.21.15.181:6801/2567452825', 'heartbeat_back_addr': '172.21.15.181:6803/2567452825', 'heartbeat_front_addr': '172.21.15.181:6802/2567452825', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '1262b2fe-f253-4a26-bfd4-c9dffb697c77', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 238, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6804', 'nonce': 2043435508}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6805', 'nonce': 2043435508}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6807', 'nonce': 2043435508}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6806', 'nonce': 2043435508}]}, 'public_addr': '172.21.15.181:6804/2043435508', 'cluster_addr': '172.21.15.181:6805/2043435508', 'heartbeat_back_addr': '172.21.15.181:6807/2043435508', 'heartbeat_front_addr': '172.21.15.181:6806/2043435508', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '7facca22-b3f1-4c20-833e-e3db798a08c4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 238, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6808', 'nonce': 1789880331}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6809', 'nonce': 1789880331}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6811', 'nonce': 1789880331}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6810', 'nonce': 1789880331}]}, 'public_addr': '172.21.15.181:6808/1789880331', 'cluster_addr': '172.21.15.181:6809/1789880331', 'heartbeat_back_addr': '172.21.15.181:6811/1789880331', 'heartbeat_front_addr': '172.21.15.181:6810/1789880331', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '6ac1b06a-79bd-43ca-b91e-fcb0f4033a34', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 53, 'last_clean_end': 139, 'up_from': 168, 'up_thru': 238, 'down_at': 140, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6812', 'nonce': 2706455857}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6813', 'nonce': 2706455857}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6815', 'nonce': 2706455857}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6814', 'nonce': 2706455857}]}, 'public_addr': '172.21.15.181:6812/2706455857', 'cluster_addr': '172.21.15.181:6813/2706455857', 'heartbeat_back_addr': '172.21.15.181:6815/2706455857', 'heartbeat_front_addr': '172.21.15.181:6814/2706455857', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T08:17:30.741804+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T08:17:19.773293+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T08:16:38.895061+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T08:17:21.516348+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T08:16:52.243024+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T08:17:42.723306+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T08:16:53.143364+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T08:16:29.930906+0000', 'dead_epoch': 139}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.4', 'mappings': [{'from': 1, 'to': 0}]}, {'pgid': '4.d', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.14', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1d', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '5.3', 'mappings': [{'from': 0, 'to': 3}]}, {'pgid': '5.1e', 'mappings': [{'from': 0, 'to': 6}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.150:0/2620931070': '2024-09-12T08:08:06.649692+0000', '172.21.15.150:0/1073581409': '2024-09-12T08:02:45.986151+0000', '172.21.15.150:0/2363649697': '2024-09-12T08:08:06.649692+0000', '172.21.15.150:6800/603451071': '2024-09-12T08:02:45.986151+0000', '172.21.15.150:0/1614413548': '2024-09-12T08:02:45.986151+0000', '172.21.15.150:0/3673218166': '2024-09-12T08:03:08.740828+0000', '172.21.15.150:6800/556498589': '2024-09-12T08:03:08.740828+0000', '172.21.15.150:0/3666733994': '2024-09-12T08:03:08.740828+0000', '172.21.15.150:0/3271479010': '2024-09-12T08:03:08.740828+0000', '172.21.15.150:0/3185731090': '2024-09-12T08:08:06.649692+0000', '172.21.15.150:0/2539412006': '2024-09-12T08:08:06.649692+0000', '172.21.15.150:0/2740222408': '2024-09-12T08:02:45.986151+0000', '172.21.15.150:6800/127856313': '2024-09-12T08:08:06.649692+0000', '172.21.15.150:0/2375121603': '2024-09-12T08:08:06.649692+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-11T08:17:52.024 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T08:17:52.066 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:17:52.088 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:17:52.089 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:52.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:52.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:17:52.191 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:17:52.213 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:52.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:52.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:17:52.315 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:17:52.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:52.337 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:52.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:17:52.438 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:52.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:52.462 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:52.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:52.563 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:52.585 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:52.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:52.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:52.629 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-11T08:17:52.630 INFO:tasks.radosbench:Beginning radosbench... 2024-09-11T08:17:52.630 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-09-11T08:17:52.630 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd pool create unique_pool_2 16 2024-09-11T08:17:52.687 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:52.708 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:52.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:52.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:52.810 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:52.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:52 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:17:52.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:52 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:17:52.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:52 smithi181 ceph-mon[32701]: 5.1b scrub starts 2024-09-11T08:17:52.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:52 smithi181 ceph-mon[32701]: 5.1b scrub ok 2024-09-11T08:17:52.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:52 smithi181 ceph-mon[32701]: osdmap e243: 8 total, 8 up, 8 in 2024-09-11T08:17:52.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:52 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:17:52.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:52 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:17:52.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:52 smithi181 ceph-mon[32701]: 3.18 scrub starts 2024-09-11T08:17:52.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:52 smithi181 ceph-mon[32701]: 3.18 scrub ok 2024-09-11T08:17:52.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:52 smithi181 ceph-mon[32701]: 3.9 scrub starts 2024-09-11T08:17:52.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:52 smithi181 ceph-mon[32701]: 3.9 scrub ok 2024-09-11T08:17:52.833 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:52.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:52.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:52.886 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:52 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:17:52.886 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:52 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:17:52.886 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:52 smithi150 ceph-mon[39612]: 5.1b scrub starts 2024-09-11T08:17:52.886 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:52 smithi150 ceph-mon[39612]: 5.1b scrub ok 2024-09-11T08:17:52.886 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:52 smithi150 ceph-mon[39612]: osdmap e243: 8 total, 8 up, 8 in 2024-09-11T08:17:52.886 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:52 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:17:52.886 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:52 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:17:52.887 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:52 smithi150 ceph-mon[39612]: 3.18 scrub starts 2024-09-11T08:17:52.887 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:52 smithi150 ceph-mon[39612]: 3.18 scrub ok 2024-09-11T08:17:52.887 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:52 smithi150 ceph-mon[39612]: 3.9 scrub starts 2024-09-11T08:17:52.887 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:52 smithi150 ceph-mon[39612]: 3.9 scrub ok 2024-09-11T08:17:52.887 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:52 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:17:52.887 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:52 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:17:52.887 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:52 smithi150 ceph-mon[29276]: 5.1b scrub starts 2024-09-11T08:17:52.887 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:52 smithi150 ceph-mon[29276]: 5.1b scrub ok 2024-09-11T08:17:52.887 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:52 smithi150 ceph-mon[29276]: osdmap e243: 8 total, 8 up, 8 in 2024-09-11T08:17:52.887 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:52 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:17:52.888 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:52 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:17:52.888 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:52 smithi150 ceph-mon[29276]: 3.18 scrub starts 2024-09-11T08:17:52.888 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:52 smithi150 ceph-mon[29276]: 3.18 scrub ok 2024-09-11T08:17:52.888 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:52 smithi150 ceph-mon[29276]: 3.9 scrub starts 2024-09-11T08:17:52.888 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:52 smithi150 ceph-mon[29276]: 3.9 scrub ok 2024-09-11T08:17:52.935 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:52.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:52.959 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:52.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:53.060 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:17:53.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:53.083 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:53.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:17:53.184 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:53.206 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:53.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:53.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:53.308 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:17:53.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:53.333 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:53.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:17:53.434 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:53.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:53.456 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:53.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:53.557 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:17:53.578 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:53.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:53.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:17:53.630 INFO:teuthology.orchestra.run.smithi150.stderr:noscrub is set 2024-09-11T08:17:53.681 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:53.702 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-11T08:17:53.702 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:17:53.704 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:53.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:53.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:53.806 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:17:53.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:53 smithi181 ceph-mon[32701]: pgmap v466: 132 pgs: 132 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-11T08:17:53.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:53 smithi181 ceph-mon[32701]: 6.11 scrub starts 2024-09-11T08:17:53.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:53 smithi181 ceph-mon[32701]: 6.11 scrub ok 2024-09-11T08:17:53.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:53 smithi181 ceph-mon[32701]: 6.17 deep-scrub starts 2024-09-11T08:17:53.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:53 smithi181 ceph-mon[32701]: 6.17 deep-scrub ok 2024-09-11T08:17:53.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:53 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2559911921' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:17:53.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:53 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:17:53.827 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:53.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:53.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:17:53.871 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:53 smithi150 ceph-mon[29276]: pgmap v466: 132 pgs: 132 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-11T08:17:53.871 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:53 smithi150 ceph-mon[29276]: 6.11 scrub starts 2024-09-11T08:17:53.871 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:53 smithi150 ceph-mon[29276]: 6.11 scrub ok 2024-09-11T08:17:53.871 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:53 smithi150 ceph-mon[29276]: 6.17 deep-scrub starts 2024-09-11T08:17:53.871 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:53 smithi150 ceph-mon[29276]: 6.17 deep-scrub ok 2024-09-11T08:17:53.871 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:53 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2559911921' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:17:53.871 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:53 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:17:53.872 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:53 smithi150 ceph-mon[39612]: pgmap v466: 132 pgs: 132 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-11T08:17:53.872 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:53 smithi150 ceph-mon[39612]: 6.11 scrub starts 2024-09-11T08:17:53.872 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:53 smithi150 ceph-mon[39612]: 6.11 scrub ok 2024-09-11T08:17:53.872 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:53 smithi150 ceph-mon[39612]: 6.17 deep-scrub starts 2024-09-11T08:17:53.872 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:53 smithi150 ceph-mon[39612]: 6.17 deep-scrub ok 2024-09-11T08:17:53.872 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:53 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2559911921' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:17:53.872 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:53 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:17:53.930 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:53.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:53.952 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:53.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:54.053 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:54.075 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:54.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:54.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:54.106 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:17:54.107 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:17:54.177 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:54.199 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:54.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:54.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:54.300 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:54.322 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:54.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:54.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:54.356 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-11T08:17:54.424 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:54.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:54.447 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:54.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:54.547 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:54.569 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:54.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:54.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:54.642 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:17:54.671 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:54.692 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:54.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:54.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:54.794 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:54.815 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:54.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:54.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:54.882 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:54 smithi181 ceph-mon[32701]: 4.14 scrub starts 2024-09-11T08:17:54.882 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:54 smithi181 ceph-mon[32701]: 4.14 scrub ok 2024-09-11T08:17:54.883 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:54 smithi181 ceph-mon[32701]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:17:54.883 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:54 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:17:54.883 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:54 smithi181 ceph-mon[32701]: osdmap e244: 8 total, 8 up, 8 in 2024-09-11T08:17:54.917 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:54.939 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:54.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:54.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:54.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:54 smithi150 ceph-mon[29276]: 4.14 scrub starts 2024-09-11T08:17:54.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:54 smithi150 ceph-mon[29276]: 4.14 scrub ok 2024-09-11T08:17:54.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:54 smithi150 ceph-mon[29276]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:17:54.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:54 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:17:54.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:54 smithi150 ceph-mon[29276]: osdmap e244: 8 total, 8 up, 8 in 2024-09-11T08:17:54.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:54 smithi150 ceph-mon[39612]: 4.14 scrub starts 2024-09-11T08:17:54.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:54 smithi150 ceph-mon[39612]: 4.14 scrub ok 2024-09-11T08:17:54.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:54 smithi150 ceph-mon[39612]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:17:54.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:54 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:17:54.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:54 smithi150 ceph-mon[39612]: osdmap e244: 8 total, 8 up, 8 in 2024-09-11T08:17:55.041 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:55.063 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:55.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:55.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:55.165 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:17:55.187 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:55.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:55.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:17:55.288 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:17:55.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:55.311 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:55.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:17:55.412 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:55.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:55.435 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:55.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:55.464 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:17:55 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:17:55] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:17:55.537 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:55.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:55.561 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:55.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:55.662 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:17:55.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:55.686 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:55.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:17:55.733 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:55 smithi150 ceph-mon[29276]: 3.a scrub starts 2024-09-11T08:17:55.734 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:55 smithi150 ceph-mon[29276]: 3.5 scrub starts 2024-09-11T08:17:55.734 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:55 smithi150 ceph-mon[29276]: pgmap v468: 132 pgs: 132 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-11T08:17:55.734 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:55 smithi150 ceph-mon[39612]: 3.a scrub starts 2024-09-11T08:17:55.734 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:55 smithi150 ceph-mon[39612]: 3.5 scrub starts 2024-09-11T08:17:55.734 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:55 smithi150 ceph-mon[39612]: pgmap v468: 132 pgs: 132 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-11T08:17:55.764 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:55 smithi181 ceph-mon[32701]: 3.a scrub starts 2024-09-11T08:17:55.764 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:55 smithi181 ceph-mon[32701]: 3.5 scrub starts 2024-09-11T08:17:55.764 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:55 smithi181 ceph-mon[32701]: pgmap v468: 132 pgs: 132 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-11T08:17:55.787 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:55.810 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:55.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:55.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:55.912 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:55.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:55.943 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:55.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:56.045 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:17:56.066 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:56.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:56.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:17:56.168 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:56.190 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:56.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:56.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:56.291 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:56.313 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:56.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:56.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:56.320 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-11T08:17:56.320 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:17:56.415 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:56.436 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:56.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:56.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:56.512 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:17:56.512 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:17:56.522 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:17:56.523 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:17:56.537 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:56.538 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:17:56.538 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:17:56.552 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:17:56.552 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:17:56.560 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:56.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:56.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:56.568 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:17:56.568 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:17:56.584 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:17:56.584 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:17:56.599 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:17:56.599 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:17:56.606 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T08:17:56.616 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:17:56.616 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:17:56.661 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:17:56.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:56.685 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:56.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:17:56.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:56 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:17:56.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:56 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:17:56.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:56 smithi150 ceph-mon[29276]: osdmap e245: 8 total, 8 up, 8 in 2024-09-11T08:17:56.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:56 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2529440344' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:17:56.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:56 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:17:56.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:56 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:17:56.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:56 smithi150 ceph-mon[39612]: osdmap e245: 8 total, 8 up, 8 in 2024-09-11T08:17:56.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:56 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2529440344' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:17:56.786 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:17:56.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:56.808 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:56.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:17:56.909 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:17:56.914 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:56 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:17:56.914 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:56 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:17:56.914 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:56 smithi181 ceph-mon[32701]: osdmap e245: 8 total, 8 up, 8 in 2024-09-11T08:17:56.914 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:56 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2529440344' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:17:56.931 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:56.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:56.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:17:56.937 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-11T08:17:57.032 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:17:57.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:57.055 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:57.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:17:57.156 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:17:57.177 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:57.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:57.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:17:57.179 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:17:57.278 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:17:57.300 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:57.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:57.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:17:57.401 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:17:57.423 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:57.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:57.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:17:57.525 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:17:57.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:57.549 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:57.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:17:57.650 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:57.672 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:57.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:57.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:57.774 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:17:57.795 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:57.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:57.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:17:57.897 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:57.918 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:57.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:57.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:57.920 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:57 smithi150 ceph-mon[29276]: pgmap v470: 132 pgs: 132 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:17:57.920 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:57 smithi150 ceph-mon[39612]: pgmap v470: 132 pgs: 132 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:17:58.020 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:17:58.043 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:58.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:58.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:17:58.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:57 smithi181 ceph-mon[32701]: pgmap v470: 132 pgs: 132 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:17:58.145 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:58.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:58.167 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:58.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:58.268 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:17:58.289 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:58.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:58.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:17:58.391 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:58.412 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:58.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:58.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:58.514 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:17:58.521 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:17:58 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:17:58.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:58.536 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:58.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:17:58.637 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:17:58.659 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:58.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:58.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:17:58.761 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:58.773 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:58 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:17:58.784 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:58.785 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:58 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:17:58.785 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:17:58 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:17:58] "GET /metrics HTTP/1.1" 200 221817 "" "Prometheus/2.43.0" 2024-09-11T08:17:58.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:58.786 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:58 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:17:58.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:58.790 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-11T08:17:58.790 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-11T08:17:58.790 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-09-11T08:17:58.790 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-09-11T08:17:58.790 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-11T08:17:58.887 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:17:58.908 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:58.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:58.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:17:59.010 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:17:59.032 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:59.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:59.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:17:59.134 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:17:59.157 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:59.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:59.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:17:59.258 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:59.280 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:59.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:59.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:59.381 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:17:59.403 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:59.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:59.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:17:59.456 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-11T08:17:59.504 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:59.525 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:59.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:59.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:59.627 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:17:59.648 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:59.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:59.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:17:59.699 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:17:59.750 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:17:59.772 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:17:59.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:59.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:17:59.818 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'unique_pool_2' created 2024-09-11T08:17:59.874 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:17:59.897 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:17:59.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:17:59.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:17:59.945 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T08:17:59.999 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:18:00.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:00.021 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:00.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:18:00.026 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:17:59 smithi181 ceph-mon[32701]: pgmap v471: 132 pgs: 132 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-11T08:18:00.120 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:17:59 smithi150 ceph-mon[29276]: pgmap v471: 132 pgs: 132 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-11T08:18:00.121 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:17:59 smithi150 ceph-mon[39612]: pgmap v471: 132 pgs: 132 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-11T08:18:00.126 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:18:00.129 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:18:00.129 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-09-11T08:18:00.154 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:00.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:00.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:18:00.256 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:00.279 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:00.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:00.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:00.380 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:00.403 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:00.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:00.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:00.505 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:18:00.527 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:00.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:00.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:18:00.629 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:00.651 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:00.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:00.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:00.753 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:00.778 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:00.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:00.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:00.880 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:18:00.902 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:00.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:00.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:18:00.945 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":472,"stamp":"2024-09-11T08:17:59.813429+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596188,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":1472,"num_read_kb":1331,"num_write":829,"num_write_kb":1671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":1773644,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":918,"ondisk_log_size":918,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":398,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2738040,"kb_used_data":271492,"kb_used_omap":19,"kb_used_meta":2465452,"kb_avail":747223176,"statfs":{"total":767960285184,"available":765156532224,"internally_reserved":0,"allocated":278007808,"data_stored":275152044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19955,"internal_metadata":2524623373},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.902001"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":234,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.722597+0000","last_change":"2024-09-11T08:16:37.667413+0000","last_active":"2024-09-11T08:17:54.722597+0000","last_peered":"2024-09-11T08:17:54.722597+0000","last_clean":"2024-09-11T08:17:54.722597+0000","last_became_active":"2024-09-11T08:07:30.044976+0000","last_became_peered":"2024-09-11T08:07:30.044976+0000","last_unstale":"2024-09-11T08:17:54.722597+0000","last_undegraded":"2024-09-11T08:17:54.722597+0000","last_fullsized":"2024-09-11T08:17:54.722597+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:37.667335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:16:37.667335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048271249999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"239'15","reported_seq":248,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:53.623359+0000","last_change":"2024-09-11T08:15:37.106907+0000","last_active":"2024-09-11T08:17:53.623359+0000","last_peered":"2024-09-11T08:17:53.623359+0000","last_clean":"2024-09-11T08:17:53.623359+0000","last_became_active":"2024-09-11T08:07:32.048048+0000","last_became_peered":"2024-09-11T08:07:32.048048+0000","last_unstale":"2024-09-11T08:17:53.623359+0000","last_undegraded":"2024-09-11T08:17:53.623359+0000","last_fullsized":"2024-09-11T08:17:53.623359+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:15:37.106824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:15:37.106824+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038285900000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":278,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.048012+0000","last_change":"2024-09-11T08:16:59.531358+0000","last_active":"2024-09-11T08:17:54.048012+0000","last_peered":"2024-09-11T08:17:54.048012+0000","last_clean":"2024-09-11T08:17:54.048012+0000","last_became_active":"2024-09-11T08:15:56.530946+0000","last_became_peered":"2024-09-11T08:15:56.530946+0000","last_unstale":"2024-09-11T08:17:54.048012+0000","last_undegraded":"2024-09-11T08:17:54.048012+0000","last_fullsized":"2024-09-11T08:17:54.048012+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:59.531202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:59.531202+0000","last_clean_scrub_stamp":"2024-09-11T08:16:59.531202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:04.694037+0000","scrub_duration":0.0017351910000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":229,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:53.623541+0000","last_change":"2024-09-11T08:16:58.143005+0000","last_active":"2024-09-11T08:17:53.623541+0000","last_peered":"2024-09-11T08:17:53.623541+0000","last_clean":"2024-09-11T08:17:53.623541+0000","last_became_active":"2024-09-11T08:07:36.074789+0000","last_became_peered":"2024-09-11T08:07:36.074789+0000","last_unstale":"2024-09-11T08:17:53.623541+0000","last_undegraded":"2024-09-11T08:17:53.623541+0000","last_fullsized":"2024-09-11T08:17:53.623541+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:58.142957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:02.470649+0000","last_clean_scrub_stamp":"2024-09-11T08:16:58.142957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:22.869601+0000","scrub_duration":0.001328805,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":256,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:53.628863+0000","last_change":"2024-09-11T08:17:44.578728+0000","last_active":"2024-09-11T08:17:53.628863+0000","last_peered":"2024-09-11T08:17:53.628863+0000","last_clean":"2024-09-11T08:17:53.628863+0000","last_became_active":"2024-09-11T08:15:56.521104+0000","last_became_peered":"2024-09-11T08:15:56.521104+0000","last_unstale":"2024-09-11T08:17:53.628863+0000","last_undegraded":"2024-09-11T08:17:53.628863+0000","last_fullsized":"2024-09-11T08:17:53.628863+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:44.578639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:17:44.578639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:13.051708+0000","scrub_duration":0.0046216080000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":29,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.046947+0000","last_change":"2024-09-11T08:17:43.158518+0000","last_active":"2024-09-11T08:17:54.046947+0000","last_peered":"2024-09-11T08:17:54.046947+0000","last_clean":"2024-09-11T08:17:54.046947+0000","last_became_active":"2024-09-11T08:17:37.550048+0000","last_became_peered":"2024-09-11T08:17:37.550048+0000","last_unstale":"2024-09-11T08:17:54.046947+0000","last_undegraded":"2024-09-11T08:17:54.046947+0000","last_fullsized":"2024-09-11T08:17:54.046947+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:43.158429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:17:43.158429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:55.231013+0000","scrub_duration":0.0013456760000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"239'34","reported_seq":179,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.179219+0000","last_change":"2024-09-11T08:17:37.554247+0000","last_active":"2024-09-11T08:17:54.179219+0000","last_peered":"2024-09-11T08:17:54.179219+0000","last_clean":"2024-09-11T08:17:54.179219+0000","last_became_active":"2024-09-11T08:17:37.553866+0000","last_became_peered":"2024-09-11T08:17:37.553866+0000","last_unstale":"2024-09-11T08:17:54.179219+0000","last_undegraded":"2024-09-11T08:17:54.179219+0000","last_fullsized":"2024-09-11T08:17:54.179219+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'22","last_scrub_stamp":"2024-09-11T08:15:57.478870+0000","last_deep_scrub":"145'22","last_deep_scrub_stamp":"2024-09-11T08:15:57.478870+0000","last_clean_scrub_stamp":"2024-09-11T08:15:57.478870+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.021858643000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":221,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:53.623341+0000","last_change":"2024-09-11T08:15:52.050101+0000","last_active":"2024-09-11T08:17:53.623341+0000","last_peered":"2024-09-11T08:17:53.623341+0000","last_clean":"2024-09-11T08:17:53.623341+0000","last_became_active":"2024-09-11T08:07:34.052243+0000","last_became_peered":"2024-09-11T08:07:34.052243+0000","last_unstale":"2024-09-11T08:17:53.623341+0000","last_undegraded":"2024-09-11T08:17:53.623341+0000","last_fullsized":"2024-09-11T08:17:53.623341+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:52.050017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:15:52.050017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037049880000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":247,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.474597+0000","last_change":"2024-09-11T08:17:36.593508+0000","last_active":"2024-09-11T08:17:54.474597+0000","last_peered":"2024-09-11T08:17:54.474597+0000","last_clean":"2024-09-11T08:17:54.474597+0000","last_became_active":"2024-09-11T08:07:30.043950+0000","last_became_peered":"2024-09-11T08:07:30.043950+0000","last_unstale":"2024-09-11T08:17:54.474597+0000","last_undegraded":"2024-09-11T08:17:54.474597+0000","last_fullsized":"2024-09-11T08:17:54.474597+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:36.593436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:54.878409+0000","last_clean_scrub_stamp":"2024-09-11T08:17:36.593436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:05.576254+0000","scrub_duration":0.001379564,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"239'15","reported_seq":258,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:53.623817+0000","last_change":"2024-09-11T08:16:56.115531+0000","last_active":"2024-09-11T08:17:53.623817+0000","last_peered":"2024-09-11T08:17:53.623817+0000","last_clean":"2024-09-11T08:17:53.623817+0000","last_became_active":"2024-09-11T08:07:32.044626+0000","last_became_peered":"2024-09-11T08:07:32.044626+0000","last_unstale":"2024-09-11T08:17:53.623817+0000","last_undegraded":"2024-09-11T08:17:53.623817+0000","last_fullsized":"2024-09-11T08:17:53.623817+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:16:56.115436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:16:56.115436+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:15.933818+0000","scrub_duration":0.0060897449999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":267,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.475071+0000","last_change":"2024-09-11T08:17:51.520223+0000","last_active":"2024-09-11T08:17:54.475071+0000","last_peered":"2024-09-11T08:17:54.475071+0000","last_clean":"2024-09-11T08:17:54.475071+0000","last_became_active":"2024-09-11T08:15:56.530680+0000","last_became_peered":"2024-09-11T08:15:56.530680+0000","last_unstale":"2024-09-11T08:17:54.475071+0000","last_undegraded":"2024-09-11T08:17:54.475071+0000","last_fullsized":"2024-09-11T08:17:54.475071+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:51.520165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:17:51.520165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:21.269411+0000","scrub_duration":0.0013849839999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":245,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.722060+0000","last_change":"2024-09-11T08:17:33.060925+0000","last_active":"2024-09-11T08:17:54.722060+0000","last_peered":"2024-09-11T08:17:54.722060+0000","last_clean":"2024-09-11T08:17:54.722060+0000","last_became_active":"2024-09-11T08:14:24.792595+0000","last_became_peered":"2024-09-11T08:14:24.792595+0000","last_unstale":"2024-09-11T08:17:54.722060+0000","last_undegraded":"2024-09-11T08:17:54.722060+0000","last_fullsized":"2024-09-11T08:17:54.722060+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:33.060860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:17:33.060860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:35.555628+0000","scrub_duration":0.025846780999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":244,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.474705+0000","last_change":"2024-09-11T08:16:51.618963+0000","last_active":"2024-09-11T08:17:54.474705+0000","last_peered":"2024-09-11T08:17:54.474705+0000","last_clean":"2024-09-11T08:17:54.474705+0000","last_became_active":"2024-09-11T08:07:30.043277+0000","last_became_peered":"2024-09-11T08:07:30.043277+0000","last_unstale":"2024-09-11T08:17:54.474705+0000","last_undegraded":"2024-09-11T08:17:54.474705+0000","last_fullsized":"2024-09-11T08:17:54.474705+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:51.618910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:16:51.618910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:59.666435+0000","scrub_duration":0.011147119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"239'9","reported_seq":241,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:53.623578+0000","last_change":"2024-09-11T08:17:51.154234+0000","last_active":"2024-09-11T08:17:53.623578+0000","last_peered":"2024-09-11T08:17:53.623578+0000","last_clean":"2024-09-11T08:17:53.623578+0000","last_became_active":"2024-09-11T08:07:32.048661+0000","last_became_peered":"2024-09-11T08:07:32.048661+0000","last_unstale":"2024-09-11T08:17:53.623578+0000","last_undegraded":"2024-09-11T08:17:53.623578+0000","last_fullsized":"2024-09-11T08:17:53.623578+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'7","last_scrub_stamp":"2024-09-11T08:15:02.988425+0000","last_deep_scrub":"139'7","last_deep_scrub_stamp":"2024-09-11T08:12:24.869318+0000","last_clean_scrub_stamp":"2024-09-11T08:15:02.988425+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033901550000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":57,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:53.628143+0000","last_change":"2024-09-11T08:16:41.727421+0000","last_active":"2024-09-11T08:17:53.628143+0000","last_peered":"2024-09-11T08:17:53.628143+0000","last_clean":"2024-09-11T08:17:53.628143+0000","last_became_active":"2024-09-11T08:15:56.526954+0000","last_became_peered":"2024-09-11T08:15:56.526954+0000","last_unstale":"2024-09-11T08:17:53.628143+0000","last_undegraded":"2024-09-11T08:17:53.628143+0000","last_fullsized":"2024-09-11T08:17:53.628143+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:41.727344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:16:41.727344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011409409999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":248,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.474723+0000","last_change":"2024-09-11T08:17:42.618987+0000","last_active":"2024-09-11T08:17:54.474723+0000","last_peered":"2024-09-11T08:17:54.474723+0000","last_clean":"2024-09-11T08:17:54.474723+0000","last_became_active":"2024-09-11T08:07:36.077602+0000","last_became_peered":"2024-09-11T08:07:36.077602+0000","last_unstale":"2024-09-11T08:17:54.474723+0000","last_undegraded":"2024-09-11T08:17:54.474723+0000","last_fullsized":"2024-09-11T08:17:54.474723+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:42.618920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:04.782577+0000","last_clean_scrub_stamp":"2024-09-11T08:17:42.618920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:57.267152+0000","scrub_duration":0.0015188039999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":229,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:53.624155+0000","last_change":"2024-09-11T08:17:30.235545+0000","last_active":"2024-09-11T08:17:53.624155+0000","last_peered":"2024-09-11T08:17:53.624155+0000","last_clean":"2024-09-11T08:17:53.624155+0000","last_became_active":"2024-09-11T08:07:36.633338+0000","last_became_peered":"2024-09-11T08:07:36.633338+0000","last_unstale":"2024-09-11T08:17:53.624155+0000","last_undegraded":"2024-09-11T08:17:53.624155+0000","last_fullsized":"2024-09-11T08:17:53.624155+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:30.235441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:17:30.235441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:50.613994+0000","scrub_duration":0.0028448620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":261,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.722466+0000","last_change":"2024-09-11T08:17:32.057243+0000","last_active":"2024-09-11T08:17:54.722466+0000","last_peered":"2024-09-11T08:17:54.722466+0000","last_clean":"2024-09-11T08:17:54.722466+0000","last_became_active":"2024-09-11T08:15:56.516941+0000","last_became_peered":"2024-09-11T08:15:56.516941+0000","last_unstale":"2024-09-11T08:17:54.722466+0000","last_undegraded":"2024-09-11T08:17:54.722466+0000","last_fullsized":"2024-09-11T08:17:54.722466+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:32.057157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:17:32.057157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:43.946489+0000","scrub_duration":0.0042801360000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"239'34","reported_seq":292,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.179636+0000","last_change":"2024-09-11T08:16:04.159388+0000","last_active":"2024-09-11T08:17:54.179636+0000","last_peered":"2024-09-11T08:17:54.179636+0000","last_clean":"2024-09-11T08:17:54.179636+0000","last_became_active":"2024-09-11T08:07:32.046363+0000","last_became_peered":"2024-09-11T08:07:32.046363+0000","last_unstale":"2024-09-11T08:17:54.179636+0000","last_undegraded":"2024-09-11T08:17:54.179636+0000","last_fullsized":"2024-09-11T08:17:54.179636+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'22","last_scrub_stamp":"2024-09-11T08:16:04.159211+0000","last_deep_scrub":"107'22","last_deep_scrub_stamp":"2024-09-11T08:16:04.159211+0000","last_clean_scrub_stamp":"2024-09-11T08:16:04.159211+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0069586229999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":251,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.048416+0000","last_change":"2024-09-11T08:17:30.539868+0000","last_active":"2024-09-11T08:17:54.048416+0000","last_peered":"2024-09-11T08:17:54.048416+0000","last_clean":"2024-09-11T08:17:54.048416+0000","last_became_active":"2024-09-11T08:07:34.055410+0000","last_became_peered":"2024-09-11T08:07:34.055410+0000","last_unstale":"2024-09-11T08:17:54.048416+0000","last_undegraded":"2024-09-11T08:17:54.048416+0000","last_fullsized":"2024-09-11T08:17:54.048416+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:30.539794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:10:31.418873+0000","last_clean_scrub_stamp":"2024-09-11T08:17:30.539794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:59.055342+0000","scrub_duration":0.0070061949999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":19,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.046925+0000","last_change":"2024-09-11T08:17:37.553907+0000","last_active":"2024-09-11T08:17:54.046925+0000","last_peered":"2024-09-11T08:17:54.046925+0000","last_clean":"2024-09-11T08:17:54.046925+0000","last_became_active":"2024-09-11T08:17:37.553559+0000","last_became_peered":"2024-09-11T08:17:37.553559+0000","last_unstale":"2024-09-11T08:17:54.046925+0000","last_undegraded":"2024-09-11T08:17:54.046925+0000","last_fullsized":"2024-09-11T08:17:54.046925+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:39.677399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:16:39.677399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:01.485791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":235,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:53.623904+0000","last_change":"2024-09-11T08:15:48.081690+0000","last_active":"2024-09-11T08:17:53.623904+0000","last_peered":"2024-09-11T08:17:53.623904+0000","last_clean":"2024-09-11T08:17:53.623904+0000","last_became_active":"2024-09-11T08:07:30.046284+0000","last_became_peered":"2024-09-11T08:07:30.046284+0000","last_unstale":"2024-09-11T08:17:53.623904+0000","last_undegraded":"2024-09-11T08:17:53.623904+0000","last_fullsized":"2024-09-11T08:17:53.623904+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:48.081591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:48.081591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0059702619999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"239'18","reported_seq":86,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.179436+0000","last_change":"2024-09-11T08:17:37.558062+0000","last_active":"2024-09-11T08:17:54.179436+0000","last_peered":"2024-09-11T08:17:54.179436+0000","last_clean":"2024-09-11T08:17:54.179436+0000","last_became_active":"2024-09-11T08:17:37.553402+0000","last_became_peered":"2024-09-11T08:17:37.553402+0000","last_unstale":"2024-09-11T08:17:54.179436+0000","last_undegraded":"2024-09-11T08:17:54.179436+0000","last_fullsized":"2024-09-11T08:17:54.179436+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-11T08:15:53.518765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:15:53.518765+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":231,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:53.623661+0000","last_change":"2024-09-11T08:16:38.974464+0000","last_active":"2024-09-11T08:17:53.623661+0000","last_peered":"2024-09-11T08:17:53.623661+0000","last_clean":"2024-09-11T08:17:53.623661+0000","last_became_active":"2024-09-11T08:07:34.056528+0000","last_became_peered":"2024-09-11T08:07:34.056528+0000","last_unstale":"2024-09-11T08:17:53.623661+0000","last_undegraded":"2024-09-11T08:17:53.623661+0000","last_fullsized":"2024-09-11T08:17:53.623661+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:38.974372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:10:40.110490+0000","last_clean_scrub_stamp":"2024-09-11T08:16:38.974372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014148811000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":184,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:53.628559+0000","last_change":"2024-09-11T08:17:01.693734+0000","last_active":"2024-09-11T08:17:53.628559+0000","last_peered":"2024-09-11T08:17:53.628559+0000","last_clean":"2024-09-11T08:17:53.628559+0000","last_became_active":"2024-09-11T08:14:24.791610+0000","last_became_peered":"2024-09-11T08:14:24.791610+0000","last_unstale":"2024-09-11T08:17:53.628559+0000","last_undegraded":"2024-09-11T08:17:53.628559+0000","last_fullsized":"2024-09-11T08:17:53.628559+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-11T08:17:01.693642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:17:01.693642+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:05.451908+0000","scrub_duration":0.0041332410000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":245,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.048178+0000","last_change":"2024-09-11T08:16:55.566928+0000","last_active":"2024-09-11T08:17:54.048178+0000","last_peered":"2024-09-11T08:17:54.048178+0000","last_clean":"2024-09-11T08:17:54.048178+0000","last_became_active":"2024-09-11T08:07:30.045932+0000","last_became_peered":"2024-09-11T08:07:30.045932+0000","last_unstale":"2024-09-11T08:17:54.048178+0000","last_undegraded":"2024-09-11T08:17:54.048178+0000","last_fullsized":"2024-09-11T08:17:54.048178+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:55.566790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:16:55.566790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:19.971541+0000","scrub_duration":0.004175102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"239'20","reported_seq":335,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.722578+0000","last_change":"2024-09-11T08:16:56.735441+0000","last_active":"2024-09-11T08:17:54.722578+0000","last_peered":"2024-09-11T08:17:54.722578+0000","last_clean":"2024-09-11T08:17:54.722578+0000","last_became_active":"2024-09-11T08:15:56.515817+0000","last_became_peered":"2024-09-11T08:15:56.515817+0000","last_unstale":"2024-09-11T08:17:54.722578+0000","last_undegraded":"2024-09-11T08:17:54.722578+0000","last_fullsized":"2024-09-11T08:17:54.722578+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-11T08:16:56.735310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:16:56.735310+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:22.326800+0000","scrub_duration":0.052749864,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"209'11","reported_seq":422,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.046993+0000","last_change":"2024-09-11T08:16:44.493380+0000","last_active":"2024-09-11T08:17:54.046993+0000","last_peered":"2024-09-11T08:17:54.046993+0000","last_clean":"2024-09-11T08:17:54.046993+0000","last_became_active":"2024-09-11T08:15:56.525498+0000","last_became_peered":"2024-09-11T08:15:56.525498+0000","last_unstale":"2024-09-11T08:17:54.046993+0000","last_undegraded":"2024-09-11T08:17:54.046993+0000","last_fullsized":"2024-09-11T08:17:54.046993+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"209'11","last_scrub_stamp":"2024-09-11T08:16:44.493296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:16:44.493296+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:06.576751+0000","scrub_duration":0.002213058,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T08:17:59.811675+0000","last_change":"2024-09-11T08:17:59.811675+0000","last_active":"2024-09-11T08:17:59.811675+0000","last_peered":"2024-09-11T08:17:59.811675+0000","last_clean":"2024-09-11T08:17:59.811675+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T08:17:59.811675+0000","last_undegraded":"2024-09-11T08:17:59.811675+0000","last_fullsized":"2024-09-11T08:17:59.811675+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:17:59.811675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.f","version":"239'27","reported_seq":296,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.048585+0000","last_change":"2024-09-11T08:17:00.559444+0000","last_active":"2024-09-11T08:17:54.048585+0000","last_peered":"2024-09-11T08:17:54.048585+0000","last_clean":"2024-09-11T08:17:54.048585+0000","last_became_active":"2024-09-11T08:07:32.054317+0000","last_became_peered":"2024-09-11T08:07:32.054317+0000","last_unstale":"2024-09-11T08:17:54.048585+0000","last_undegraded":"2024-09-11T08:17:54.048585+0000","last_fullsized":"2024-09-11T08:17:54.048585+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-11T08:17:00.559350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:17:00.559350+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:09.457044+0000","scrub_duration":0.013106273999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":255,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:53.628587+0000","last_change":"2024-09-11T08:16:42.624557+0000","last_active":"2024-09-11T08:17:53.628587+0000","last_peered":"2024-09-11T08:17:53.628587+0000","last_clean":"2024-09-11T08:17:53.628587+0000","last_became_active":"2024-09-11T08:15:56.521943+0000","last_became_peered":"2024-09-11T08:15:56.521943+0000","last_unstale":"2024-09-11T08:17:53.628587+0000","last_undegraded":"2024-09-11T08:17:53.628587+0000","last_fullsized":"2024-09-11T08:17:53.628587+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:42.624425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:16:42.624425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:05.345719+0000","scrub_duration":0.011367408000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":364,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:53.674334+0000","last_change":"2024-09-11T08:16:44.005131+0000","last_active":"2024-09-11T08:17:53.674334+0000","last_peered":"2024-09-11T08:17:53.674334+0000","last_clean":"2024-09-11T08:17:53.674334+0000","last_became_active":"2024-09-11T08:07:34.052981+0000","last_became_peered":"2024-09-11T08:07:34.052981+0000","last_unstale":"2024-09-11T08:17:53.674334+0000","last_undegraded":"2024-09-11T08:17:53.674334+0000","last_fullsized":"2024-09-11T08:17:53.674334+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:16:44.005077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:16:44.005077+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:04.080032+0000","scrub_duration":0.019154572000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":248,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.475265+0000","last_change":"2024-09-11T08:17:47.577994+0000","last_active":"2024-09-11T08:17:54.475265+0000","last_peered":"2024-09-11T08:17:54.475265+0000","last_clean":"2024-09-11T08:17:54.475265+0000","last_became_active":"2024-09-11T08:07:36.073969+0000","last_became_peered":"2024-09-11T08:07:36.073969+0000","last_unstale":"2024-09-11T08:17:54.475265+0000","last_undegraded":"2024-09-11T08:17:54.475265+0000","last_fullsized":"2024-09-11T08:17:54.475265+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:47.577941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:06.616332+0000","last_clean_scrub_stamp":"2024-09-11T08:17:47.577941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:48.235918+0000","scrub_duration":0.001202289,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T08:17:59.811675+0000","last_change":"2024-09-11T08:17:59.811675+0000","last_active":"2024-09-11T08:17:59.811675+0000","last_peered":"2024-09-11T08:17:59.811675+0000","last_clean":"2024-09-11T08:17:59.811675+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T08:17:59.811675+0000","last_undegraded":"2024-09-11T08:17:59.811675+0000","last_fullsized":"2024-09-11T08:17:59.811675+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:17:59.811675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.0","version":"239'30","reported_seq":291,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:53.628225+0000","last_change":"2024-09-11T08:17:37.556001+0000","last_active":"2024-09-11T08:17:53.628225+0000","last_peered":"2024-09-11T08:17:53.628225+0000","last_clean":"2024-09-11T08:17:53.628225+0000","last_became_active":"2024-09-11T08:17:37.555703+0000","last_became_peered":"2024-09-11T08:17:37.555703+0000","last_unstale":"2024-09-11T08:17:53.628225+0000","last_undegraded":"2024-09-11T08:17:53.628225+0000","last_fullsized":"2024-09-11T08:17:53.628225+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-11T08:17:03.650750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:17:03.650750+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:15.558999+0000","scrub_duration":0.001885091,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":215,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:53.628723+0000","last_change":"2024-09-11T08:17:37.554031+0000","last_active":"2024-09-11T08:17:53.628723+0000","last_peered":"2024-09-11T08:17:53.628723+0000","last_clean":"2024-09-11T08:17:53.628723+0000","last_became_active":"2024-09-11T08:17:37.553579+0000","last_became_peered":"2024-09-11T08:17:37.553579+0000","last_unstale":"2024-09-11T08:17:53.628723+0000","last_undegraded":"2024-09-11T08:17:53.628723+0000","last_fullsized":"2024-09-11T08:17:53.628723+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:43.614153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:16:43.614153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035672669999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":258,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:53.623416+0000","last_change":"2024-09-11T08:17:48.150197+0000","last_active":"2024-09-11T08:17:53.623416+0000","last_peered":"2024-09-11T08:17:53.623416+0000","last_clean":"2024-09-11T08:17:53.623416+0000","last_became_active":"2024-09-11T08:15:56.522606+0000","last_became_peered":"2024-09-11T08:15:56.522606+0000","last_unstale":"2024-09-11T08:17:53.623416+0000","last_undegraded":"2024-09-11T08:17:53.623416+0000","last_fullsized":"2024-09-11T08:17:53.623416+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:48.150140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:17:48.150140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:03.659795+0000","scrub_duration":0.001616889,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":229,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:53.623721+0000","last_change":"2024-09-11T08:17:39.096079+0000","last_active":"2024-09-11T08:17:53.623721+0000","last_peered":"2024-09-11T08:17:53.623721+0000","last_clean":"2024-09-11T08:17:53.623721+0000","last_became_active":"2024-09-11T08:07:36.069225+0000","last_became_peered":"2024-09-11T08:07:36.069225+0000","last_unstale":"2024-09-11T08:17:53.623721+0000","last_undegraded":"2024-09-11T08:17:53.623721+0000","last_fullsized":"2024-09-11T08:17:53.623721+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:39.096025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:04.937884+0000","last_clean_scrub_stamp":"2024-09-11T08:17:39.096025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:41.275131+0000","scrub_duration":0.001460954,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T08:17:59.811675+0000","last_change":"2024-09-11T08:17:59.811675+0000","last_active":"2024-09-11T08:17:59.811675+0000","last_peered":"2024-09-11T08:17:59.811675+0000","last_clean":"2024-09-11T08:17:59.811675+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T08:17:59.811675+0000","last_undegraded":"2024-09-11T08:17:59.811675+0000","last_fullsized":"2024-09-11T08:17:59.811675+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:17:59.811675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1","version":"239'28","reported_seq":275,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:53.623609+0000","last_change":"2024-09-11T08:17:49.184178+0000","last_active":"2024-09-11T08:17:53.623609+0000","last_peered":"2024-09-11T08:17:53.623609+0000","last_clean":"2024-09-11T08:17:53.623609+0000","last_became_active":"2024-09-11T08:07:32.040069+0000","last_became_peered":"2024-09-11T08:07:32.040069+0000","last_unstale":"2024-09-11T08:17:53.623609+0000","last_undegraded":"2024-09-11T08:17:53.623609+0000","last_fullsized":"2024-09-11T08:17:53.623609+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'28","last_scrub_stamp":"2024-09-11T08:17:49.184124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:17:49.184124+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:52.848465+0000","scrub_duration":0.0019943040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":244,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.721952+0000","last_change":"2024-09-11T08:17:47.960385+0000","last_active":"2024-09-11T08:17:54.721952+0000","last_peered":"2024-09-11T08:17:54.721952+0000","last_clean":"2024-09-11T08:17:54.721952+0000","last_became_active":"2024-09-11T08:07:30.050484+0000","last_became_peered":"2024-09-11T08:07:30.050484+0000","last_unstale":"2024-09-11T08:17:54.721952+0000","last_undegraded":"2024-09-11T08:17:54.721952+0000","last_fullsized":"2024-09-11T08:17:54.721952+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:47.960329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:47.960329+0000","last_clean_scrub_stamp":"2024-09-11T08:17:47.960329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:07.259809+0000","scrub_duration":0.001320681,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"82'9","reported_seq":378,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:53.674532+0000","last_change":"2024-09-11T08:17:47.485985+0000","last_active":"2024-09-11T08:17:53.674532+0000","last_peered":"2024-09-11T08:17:53.674532+0000","last_clean":"2024-09-11T08:17:53.674532+0000","last_became_active":"2024-09-11T08:07:34.056143+0000","last_became_peered":"2024-09-11T08:07:34.056143+0000","last_unstale":"2024-09-11T08:17:53.674532+0000","last_undegraded":"2024-09-11T08:17:53.674532+0000","last_fullsized":"2024-09-11T08:17:53.674532+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-11T08:17:47.485932+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-09-11T08:13:10.230703+0000","last_clean_scrub_stamp":"2024-09-11T08:17:47.485932+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:56.243066+0000","scrub_duration":0.0016186289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":219,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:53.628351+0000","last_change":"2024-09-11T08:17:35.867212+0000","last_active":"2024-09-11T08:17:53.628351+0000","last_peered":"2024-09-11T08:17:53.628351+0000","last_clean":"2024-09-11T08:17:53.628351+0000","last_became_active":"2024-09-11T08:14:24.788968+0000","last_became_peered":"2024-09-11T08:14:24.788968+0000","last_unstale":"2024-09-11T08:17:53.628351+0000","last_undegraded":"2024-09-11T08:17:53.628351+0000","last_fullsized":"2024-09-11T08:17:53.628351+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:35.867116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:17:35.867116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:55.885043+0000","scrub_duration":0.0021141390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T08:17:59.811675+0000","last_change":"2024-09-11T08:17:59.811675+0000","last_active":"2024-09-11T08:17:59.811675+0000","last_peered":"2024-09-11T08:17:59.811675+0000","last_clean":"2024-09-11T08:17:59.811675+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T08:17:59.811675+0000","last_undegraded":"2024-09-11T08:17:59.811675+0000","last_fullsized":"2024-09-11T08:17:59.811675+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:17:59.811675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2","version":"239'20","reported_seq":276,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.048288+0000","last_change":"2024-09-11T08:17:37.613078+0000","last_active":"2024-09-11T08:17:54.048288+0000","last_peered":"2024-09-11T08:17:54.048288+0000","last_clean":"2024-09-11T08:17:54.048288+0000","last_became_active":"2024-09-11T08:07:32.054294+0000","last_became_peered":"2024-09-11T08:07:32.054294+0000","last_unstale":"2024-09-11T08:17:54.048288+0000","last_undegraded":"2024-09-11T08:17:54.048288+0000","last_fullsized":"2024-09-11T08:17:54.048288+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'18","last_scrub_stamp":"2024-09-11T08:17:37.613023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:17:37.613023+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:05.205128+0000","scrub_duration":0.0017108679999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":240,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.475084+0000","last_change":"2024-09-11T08:17:54.475062+0000","last_active":"2024-09-11T08:17:54.475084+0000","last_peered":"2024-09-11T08:17:54.475084+0000","last_clean":"2024-09-11T08:17:54.475084+0000","last_became_active":"2024-09-11T08:07:30.047497+0000","last_became_peered":"2024-09-11T08:07:30.047497+0000","last_unstale":"2024-09-11T08:17:54.475084+0000","last_undegraded":"2024-09-11T08:17:54.475084+0000","last_fullsized":"2024-09-11T08:17:54.475084+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:46.459143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:46.459143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0057785019999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"239'10","reported_seq":43,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.046574+0000","last_change":"2024-09-11T08:17:37.554331+0000","last_active":"2024-09-11T08:17:54.046574+0000","last_peered":"2024-09-11T08:17:54.046574+0000","last_clean":"2024-09-11T08:17:54.046574+0000","last_became_active":"2024-09-11T08:17:37.552680+0000","last_became_peered":"2024-09-11T08:17:37.552680+0000","last_unstale":"2024-09-11T08:17:54.046574+0000","last_undegraded":"2024-09-11T08:17:54.046574+0000","last_fullsized":"2024-09-11T08:17:54.046574+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'9","last_scrub_stamp":"2024-09-11T08:17:01.640482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:17:01.640482+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:09.113810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"66'1","reported_seq":229,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.722167+0000","last_change":"2024-09-11T08:17:40.041132+0000","last_active":"2024-09-11T08:17:54.722167+0000","last_peered":"2024-09-11T08:17:54.722167+0000","last_clean":"2024-09-11T08:17:54.722167+0000","last_became_active":"2024-09-11T08:07:36.072593+0000","last_became_peered":"2024-09-11T08:07:36.072593+0000","last_unstale":"2024-09-11T08:17:54.722167+0000","last_undegraded":"2024-09-11T08:17:54.722167+0000","last_fullsized":"2024-09-11T08:17:54.722167+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-11T08:17:40.041070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:17:40.041070+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:07.811252+0000","scrub_duration":0.0016243379999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T08:17:59.811675+0000","last_change":"2024-09-11T08:17:59.811675+0000","last_active":"2024-09-11T08:17:59.811675+0000","last_peered":"2024-09-11T08:17:59.811675+0000","last_clean":"2024-09-11T08:17:59.811675+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T08:17:59.811675+0000","last_undegraded":"2024-09-11T08:17:59.811675+0000","last_fullsized":"2024-09-11T08:17:59.811675+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:17:59.811675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3","version":"239'29","reported_seq":346,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.722329+0000","last_change":"2024-09-11T08:16:58.691731+0000","last_active":"2024-09-11T08:17:54.722329+0000","last_peered":"2024-09-11T08:17:54.722329+0000","last_clean":"2024-09-11T08:17:54.722329+0000","last_became_active":"2024-09-11T08:15:56.518384+0000","last_became_peered":"2024-09-11T08:15:56.518384+0000","last_unstale":"2024-09-11T08:17:54.722329+0000","last_undegraded":"2024-09-11T08:17:54.722329+0000","last_fullsized":"2024-09-11T08:17:54.722329+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-11T08:16:58.691634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:16:58.691634+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:10.015330+0000","scrub_duration":0.003405576,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":255,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.048322+0000","last_change":"2024-09-11T08:17:34.670772+0000","last_active":"2024-09-11T08:17:54.048322+0000","last_peered":"2024-09-11T08:17:54.048322+0000","last_clean":"2024-09-11T08:17:54.048322+0000","last_became_active":"2024-09-11T08:07:30.045380+0000","last_became_peered":"2024-09-11T08:07:30.045380+0000","last_unstale":"2024-09-11T08:17:54.048322+0000","last_undegraded":"2024-09-11T08:17:54.048322+0000","last_fullsized":"2024-09-11T08:17:54.048322+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:34.670680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:17:34.670680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:00.877538+0000","scrub_duration":0.0042100260000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":250,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.046411+0000","last_change":"2024-09-11T08:17:00.522545+0000","last_active":"2024-09-11T08:17:54.046411+0000","last_peered":"2024-09-11T08:17:54.046411+0000","last_clean":"2024-09-11T08:17:54.046411+0000","last_became_active":"2024-09-11T08:07:34.051004+0000","last_became_peered":"2024-09-11T08:07:34.051004+0000","last_unstale":"2024-09-11T08:17:54.046411+0000","last_undegraded":"2024-09-11T08:17:54.046411+0000","last_fullsized":"2024-09-11T08:17:54.046411+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:00.522467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:17:00.522467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:11.280048+0000","scrub_duration":0.0017451980000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":239,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.047969+0000","last_change":"2024-09-11T08:16:58.517711+0000","last_active":"2024-09-11T08:17:54.047969+0000","last_peered":"2024-09-11T08:17:54.047969+0000","last_clean":"2024-09-11T08:17:54.047969+0000","last_became_active":"2024-09-11T08:07:36.631668+0000","last_became_peered":"2024-09-11T08:07:36.631668+0000","last_unstale":"2024-09-11T08:17:54.047969+0000","last_undegraded":"2024-09-11T08:17:54.047969+0000","last_fullsized":"2024-09-11T08:17:54.047969+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:58.517624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:58.517624+0000","last_clean_scrub_stamp":"2024-09-11T08:16:58.517624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:08.188321+0000","scrub_duration":0.0083486270000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T08:17:59.811675+0000","last_change":"2024-09-11T08:17:59.811675+0000","last_active":"2024-09-11T08:17:59.811675+0000","last_peered":"2024-09-11T08:17:59.811675+0000","last_clean":"2024-09-11T08:17:59.811675+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T08:17:59.811675+0000","last_undegraded":"2024-09-11T08:17:59.811675+0000","last_fullsized":"2024-09-11T08:17:59.811675+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:17:59.811675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.4","version":"239'51","reported_seq":158,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.722395+0000","last_change":"2024-09-11T08:17:50.928002+0000","last_active":"2024-09-11T08:17:54.722395+0000","last_peered":"2024-09-11T08:17:54.722395+0000","last_clean":"2024-09-11T08:17:54.722395+0000","last_became_active":"2024-09-11T08:15:08.775181+0000","last_became_peered":"2024-09-11T08:15:08.775181+0000","last_unstale":"2024-09-11T08:17:54.722395+0000","last_undegraded":"2024-09-11T08:17:54.722395+0000","last_fullsized":"2024-09-11T08:17:54.722395+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'51","last_scrub_stamp":"2024-09-11T08:17:50.927912+0000","last_deep_scrub":"190'39","last_deep_scrub_stamp":"2024-09-11T08:15:52.847968+0000","last_clean_scrub_stamp":"2024-09-11T08:17:50.927912+0000","objects_scrubbed":10,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:13.514824+0000","scrub_duration":0.0036015019999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"59'1","reported_seq":244,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:53.623991+0000","last_change":"2024-09-11T08:15:44.992430+0000","last_active":"2024-09-11T08:17:53.623991+0000","last_peered":"2024-09-11T08:17:53.623991+0000","last_clean":"2024-09-11T08:17:53.623991+0000","last_became_active":"2024-09-11T08:07:30.047513+0000","last_became_peered":"2024-09-11T08:07:30.047513+0000","last_unstale":"2024-09-11T08:17:53.623991+0000","last_undegraded":"2024-09-11T08:17:53.623991+0000","last_fullsized":"2024-09-11T08:17:53.623991+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-11T08:15:44.992356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:44.992356+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049464690000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":251,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.474504+0000","last_change":"2024-09-11T08:16:56.558518+0000","last_active":"2024-09-11T08:17:54.474504+0000","last_peered":"2024-09-11T08:17:54.474504+0000","last_clean":"2024-09-11T08:17:54.474504+0000","last_became_active":"2024-09-11T08:07:30.043130+0000","last_became_peered":"2024-09-11T08:07:30.043130+0000","last_unstale":"2024-09-11T08:17:54.474504+0000","last_undegraded":"2024-09-11T08:17:54.474504+0000","last_fullsized":"2024-09-11T08:17:54.474504+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-11T08:16:56.558301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:16:56.558301+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:23.711468+0000","scrub_duration":0.0048858299999999999,"objects_trimmed":0,"snaptrim_duration":0.0026120850000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":230,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.722493+0000","last_change":"2024-09-11T08:16:42.666121+0000","last_active":"2024-09-11T08:17:54.722493+0000","last_peered":"2024-09-11T08:17:54.722493+0000","last_clean":"2024-09-11T08:17:54.722493+0000","last_became_active":"2024-09-11T08:07:34.057154+0000","last_became_peered":"2024-09-11T08:07:34.057154+0000","last_unstale":"2024-09-11T08:17:54.722493+0000","last_undegraded":"2024-09-11T08:17:54.722493+0000","last_fullsized":"2024-09-11T08:17:54.722493+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:42.666035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:10:37.957096+0000","last_clean_scrub_stamp":"2024-09-11T08:16:42.666035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:11.328001+0000","scrub_duration":0.0077723870000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":256,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:53.629319+0000","last_change":"2024-09-11T08:17:31.602267+0000","last_active":"2024-09-11T08:17:53.629319+0000","last_peered":"2024-09-11T08:17:53.629319+0000","last_clean":"2024-09-11T08:17:53.629319+0000","last_became_active":"2024-09-11T08:15:56.525099+0000","last_became_peered":"2024-09-11T08:15:56.525099+0000","last_unstale":"2024-09-11T08:17:53.629319+0000","last_undegraded":"2024-09-11T08:17:53.629319+0000","last_fullsized":"2024-09-11T08:17:53.629319+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:31.602060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:17:31.602060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:55.062758+0000","scrub_duration":0.0059808200000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T08:17:59.811675+0000","last_change":"2024-09-11T08:17:59.811675+0000","last_active":"2024-09-11T08:17:59.811675+0000","last_peered":"2024-09-11T08:17:59.811675+0000","last_clean":"2024-09-11T08:17:59.811675+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T08:17:59.811675+0000","last_undegraded":"2024-09-11T08:17:59.811675+0000","last_fullsized":"2024-09-11T08:17:59.811675+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:17:59.811675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.7","version":"239'17","reported_seq":279,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.048616+0000","last_change":"2024-09-11T08:16:41.514946+0000","last_active":"2024-09-11T08:17:54.048616+0000","last_peered":"2024-09-11T08:17:54.048616+0000","last_clean":"2024-09-11T08:17:54.048616+0000","last_became_active":"2024-09-11T08:07:32.050673+0000","last_became_peered":"2024-09-11T08:07:32.050673+0000","last_unstale":"2024-09-11T08:17:54.048616+0000","last_undegraded":"2024-09-11T08:17:54.048616+0000","last_fullsized":"2024-09-11T08:17:54.048616+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'13","last_scrub_stamp":"2024-09-11T08:16:41.514853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:16:41.514853+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0093017910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":248,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.047890+0000","last_change":"2024-09-11T08:17:02.493714+0000","last_active":"2024-09-11T08:17:54.047890+0000","last_peered":"2024-09-11T08:17:54.047890+0000","last_clean":"2024-09-11T08:17:54.047890+0000","last_became_active":"2024-09-11T08:07:30.047014+0000","last_became_peered":"2024-09-11T08:07:30.047014+0000","last_unstale":"2024-09-11T08:17:54.047890+0000","last_undegraded":"2024-09-11T08:17:54.047890+0000","last_fullsized":"2024-09-11T08:17:54.047890+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:02.493627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:17:02.493627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:24.784652+0000","scrub_duration":0.0048819830000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"127'2","reported_seq":264,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.178996+0000","last_change":"2024-09-11T08:16:40.021161+0000","last_active":"2024-09-11T08:17:54.178996+0000","last_peered":"2024-09-11T08:17:54.178996+0000","last_clean":"2024-09-11T08:17:54.178996+0000","last_became_active":"2024-09-11T08:07:30.045009+0000","last_became_peered":"2024-09-11T08:07:30.045009+0000","last_unstale":"2024-09-11T08:17:54.178996+0000","last_undegraded":"2024-09-11T08:17:54.178996+0000","last_fullsized":"2024-09-11T08:17:54.178996+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'2","last_scrub_stamp":"2024-09-11T08:16:40.021082+0000","last_deep_scrub":"127'2","last_deep_scrub_stamp":"2024-09-11T08:15:01.938497+0000","last_clean_scrub_stamp":"2024-09-11T08:16:40.021082+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002170426,"objects_trimmed":0,"snaptrim_duration":0.001859533,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":260,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.179599+0000","last_change":"2024-09-11T08:16:50.993227+0000","last_active":"2024-09-11T08:17:54.179599+0000","last_peered":"2024-09-11T08:17:54.179599+0000","last_clean":"2024-09-11T08:17:54.179599+0000","last_became_active":"2024-09-11T08:14:24.792279+0000","last_became_peered":"2024-09-11T08:14:24.792279+0000","last_unstale":"2024-09-11T08:17:54.179599+0000","last_undegraded":"2024-09-11T08:17:54.179599+0000","last_fullsized":"2024-09-11T08:17:54.179599+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:50.993154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:58.889094+0000","last_clean_scrub_stamp":"2024-09-11T08:16:50.993154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:13.412919+0000","scrub_duration":0.0011069389999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":175,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:53.628742+0000","last_change":"2024-09-11T08:16:58.685525+0000","last_active":"2024-09-11T08:17:53.628742+0000","last_peered":"2024-09-11T08:17:53.628742+0000","last_clean":"2024-09-11T08:17:53.628742+0000","last_became_active":"2024-09-11T08:14:24.792847+0000","last_became_peered":"2024-09-11T08:14:24.792847+0000","last_unstale":"2024-09-11T08:17:53.628742+0000","last_undegraded":"2024-09-11T08:17:53.628742+0000","last_fullsized":"2024-09-11T08:17:53.628742+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:58.685473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:58.796217+0000","last_clean_scrub_stamp":"2024-09-11T08:16:58.685473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:13.302984+0000","scrub_duration":0.0016333490000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T08:17:59.811675+0000","last_change":"2024-09-11T08:17:59.811675+0000","last_active":"2024-09-11T08:17:59.811675+0000","last_peered":"2024-09-11T08:17:59.811675+0000","last_clean":"2024-09-11T08:17:59.811675+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T08:17:59.811675+0000","last_undegraded":"2024-09-11T08:17:59.811675+0000","last_fullsized":"2024-09-11T08:17:59.811675+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:17:59.811675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.6","version":"239'24","reported_seq":278,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.722630+0000","last_change":"2024-09-11T08:17:44.010492+0000","last_active":"2024-09-11T08:17:54.722630+0000","last_peered":"2024-09-11T08:17:54.722630+0000","last_clean":"2024-09-11T08:17:54.722630+0000","last_became_active":"2024-09-11T08:07:32.055407+0000","last_became_peered":"2024-09-11T08:07:32.055407+0000","last_unstale":"2024-09-11T08:17:54.722630+0000","last_undegraded":"2024-09-11T08:17:54.722630+0000","last_fullsized":"2024-09-11T08:17:54.722630+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'24","last_scrub_stamp":"2024-09-11T08:17:44.010434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:17:44.010434+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:06.755691+0000","scrub_duration":0.0041285339999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"59'1","reported_seq":251,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.721913+0000","last_change":"2024-09-11T08:17:46.998145+0000","last_active":"2024-09-11T08:17:54.721913+0000","last_peered":"2024-09-11T08:17:54.721913+0000","last_clean":"2024-09-11T08:17:54.721913+0000","last_became_active":"2024-09-11T08:07:30.050504+0000","last_became_peered":"2024-09-11T08:07:30.050504+0000","last_unstale":"2024-09-11T08:17:54.721913+0000","last_undegraded":"2024-09-11T08:17:54.721913+0000","last_fullsized":"2024-09-11T08:17:54.721913+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-11T08:17:46.998019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:17:46.998019+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:12.051489+0000","scrub_duration":0.0015225320000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"224'6","reported_seq":650,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-11T08:17:56.209003+0000","last_change":"2024-09-11T08:16:55.778375+0000","last_active":"2024-09-11T08:17:56.209003+0000","last_peered":"2024-09-11T08:17:56.209003+0000","last_clean":"2024-09-11T08:17:56.209003+0000","last_became_active":"2024-09-11T08:14:24.793048+0000","last_became_peered":"2024-09-11T08:14:24.793048+0000","last_unstale":"2024-09-11T08:17:56.209003+0000","last_undegraded":"2024-09-11T08:17:56.209003+0000","last_fullsized":"2024-09-11T08:17:56.209003+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'6","last_scrub_stamp":"2024-09-11T08:16:55.778284+0000","last_deep_scrub":"224'6","last_deep_scrub_stamp":"2024-09-11T08:16:55.778284+0000","last_clean_scrub_stamp":"2024-09-11T08:16:55.778284+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:08.760919+0000","scrub_duration":0.018002499000000002,"objects_trimmed":0,"snaptrim_duration":0.000318821,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":219,"num_read_kb":214,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":240,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.475584+0000","last_change":"2024-09-11T08:16:37.787055+0000","last_active":"2024-09-11T08:17:54.475584+0000","last_peered":"2024-09-11T08:17:54.475584+0000","last_clean":"2024-09-11T08:17:54.475584+0000","last_became_active":"2024-09-11T08:07:34.058962+0000","last_became_peered":"2024-09-11T08:07:34.058962+0000","last_unstale":"2024-09-11T08:17:54.475584+0000","last_undegraded":"2024-09-11T08:17:54.475584+0000","last_fullsized":"2024-09-11T08:17:54.475584+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:37.786977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:16:37.786977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010499725999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":249,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.048136+0000","last_change":"2024-09-11T08:17:42.607077+0000","last_active":"2024-09-11T08:17:54.048136+0000","last_peered":"2024-09-11T08:17:54.048136+0000","last_clean":"2024-09-11T08:17:54.048136+0000","last_became_active":"2024-09-11T08:07:36.077597+0000","last_became_peered":"2024-09-11T08:07:36.077597+0000","last_unstale":"2024-09-11T08:17:54.048136+0000","last_undegraded":"2024-09-11T08:17:54.048136+0000","last_fullsized":"2024-09-11T08:17:54.048136+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:42.606972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:17:42.606972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:08.110790+0000","scrub_duration":0.0020562559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T08:17:59.811675+0000","last_change":"2024-09-11T08:17:59.811675+0000","last_active":"2024-09-11T08:17:59.811675+0000","last_peered":"2024-09-11T08:17:59.811675+0000","last_clean":"2024-09-11T08:17:59.811675+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T08:17:59.811675+0000","last_undegraded":"2024-09-11T08:17:59.811675+0000","last_fullsized":"2024-09-11T08:17:59.811675+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:17:59.811675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.5","version":"239'33","reported_seq":306,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.046915+0000","last_change":"2024-09-11T08:16:40.472710+0000","last_active":"2024-09-11T08:17:54.046915+0000","last_peered":"2024-09-11T08:17:54.046915+0000","last_clean":"2024-09-11T08:17:54.046915+0000","last_became_active":"2024-09-11T08:14:24.789154+0000","last_became_peered":"2024-09-11T08:14:24.789154+0000","last_unstale":"2024-09-11T08:17:54.046915+0000","last_undegraded":"2024-09-11T08:17:54.046915+0000","last_fullsized":"2024-09-11T08:17:54.046915+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'21","last_scrub_stamp":"2024-09-11T08:16:40.472618+0000","last_deep_scrub":"84'21","last_deep_scrub_stamp":"2024-09-11T08:08:56.705306+0000","last_clean_scrub_stamp":"2024-09-11T08:16:40.472618+0000","objects_scrubbed":7,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:00.262854+0000","scrub_duration":0.0099327200000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":29,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.475048+0000","last_change":"2024-09-11T08:17:38.584894+0000","last_active":"2024-09-11T08:17:54.475048+0000","last_peered":"2024-09-11T08:17:54.475048+0000","last_clean":"2024-09-11T08:17:54.475048+0000","last_became_active":"2024-09-11T08:17:37.554397+0000","last_became_peered":"2024-09-11T08:17:37.554397+0000","last_unstale":"2024-09-11T08:17:54.475048+0000","last_undegraded":"2024-09-11T08:17:54.475048+0000","last_fullsized":"2024-09-11T08:17:54.475048+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:38.584839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:06.236119+0000","last_clean_scrub_stamp":"2024-09-11T08:17:38.584839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:42.517360+0000","scrub_duration":0.001652581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":63,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:53.627998+0000","last_change":"2024-09-11T08:16:45.611831+0000","last_active":"2024-09-11T08:17:53.627998+0000","last_peered":"2024-09-11T08:17:53.627998+0000","last_clean":"2024-09-11T08:17:53.627998+0000","last_became_active":"2024-09-11T08:15:56.523430+0000","last_became_peered":"2024-09-11T08:15:56.523430+0000","last_unstale":"2024-09-11T08:17:53.627998+0000","last_undegraded":"2024-09-11T08:17:53.627998+0000","last_fullsized":"2024-09-11T08:17:53.627998+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-11T08:16:45.611772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:05:21.231856+0000","last_clean_scrub_stamp":"2024-09-11T08:16:45.611772+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:06.875825+0000","scrub_duration":0.0021764710000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":57,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:53.628468+0000","last_change":"2024-09-11T08:17:40.793670+0000","last_active":"2024-09-11T08:17:53.628468+0000","last_peered":"2024-09-11T08:17:53.628468+0000","last_clean":"2024-09-11T08:17:53.628468+0000","last_became_active":"2024-09-11T08:15:56.528641+0000","last_became_peered":"2024-09-11T08:15:56.528641+0000","last_unstale":"2024-09-11T08:17:53.628468+0000","last_undegraded":"2024-09-11T08:17:53.628468+0000","last_fullsized":"2024-09-11T08:17:53.628468+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:40.793611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:51.289872+0000","last_clean_scrub_stamp":"2024-09-11T08:17:40.793611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:41.869181+0000","scrub_duration":0.0012304760000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":238,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.475544+0000","last_change":"2024-09-11T08:16:43.782518+0000","last_active":"2024-09-11T08:17:54.475544+0000","last_peered":"2024-09-11T08:17:54.475544+0000","last_clean":"2024-09-11T08:17:54.475544+0000","last_became_active":"2024-09-11T08:07:36.072508+0000","last_became_peered":"2024-09-11T08:07:36.072508+0000","last_unstale":"2024-09-11T08:17:54.475544+0000","last_undegraded":"2024-09-11T08:17:54.475544+0000","last_fullsized":"2024-09-11T08:17:54.475544+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:43.782439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:01.777429+0000","last_clean_scrub_stamp":"2024-09-11T08:16:43.782439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038441999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T08:17:59.811675+0000","last_change":"2024-09-11T08:17:59.811675+0000","last_active":"2024-09-11T08:17:59.811675+0000","last_peered":"2024-09-11T08:17:59.811675+0000","last_clean":"2024-09-11T08:17:59.811675+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T08:17:59.811675+0000","last_undegraded":"2024-09-11T08:17:59.811675+0000","last_fullsized":"2024-09-11T08:17:59.811675+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:17:59.811675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.e","version":"239'24","reported_seq":269,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:53.624275+0000","last_change":"2024-09-11T08:16:46.015835+0000","last_active":"2024-09-11T08:17:53.624275+0000","last_peered":"2024-09-11T08:17:53.624275+0000","last_clean":"2024-09-11T08:17:53.624275+0000","last_became_active":"2024-09-11T08:07:32.047971+0000","last_became_peered":"2024-09-11T08:07:32.047971+0000","last_unstale":"2024-09-11T08:17:53.624275+0000","last_undegraded":"2024-09-11T08:17:53.624275+0000","last_fullsized":"2024-09-11T08:17:53.624275+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'16","last_scrub_stamp":"2024-09-11T08:16:46.015776+0000","last_deep_scrub":"88'16","last_deep_scrub_stamp":"2024-09-11T08:09:39.635984+0000","last_clean_scrub_stamp":"2024-09-11T08:16:46.015776+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:03.636623+0000","scrub_duration":0.0030649929999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":252,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:53.623655+0000","last_change":"2024-09-11T08:17:52.133078+0000","last_active":"2024-09-11T08:17:53.623655+0000","last_peered":"2024-09-11T08:17:53.623655+0000","last_clean":"2024-09-11T08:17:53.623655+0000","last_became_active":"2024-09-11T08:14:24.789888+0000","last_became_peered":"2024-09-11T08:14:24.789888+0000","last_unstale":"2024-09-11T08:17:53.623655+0000","last_undegraded":"2024-09-11T08:17:53.623655+0000","last_fullsized":"2024-09-11T08:17:53.623655+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:52.133025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:36.417911+0000","last_clean_scrub_stamp":"2024-09-11T08:17:52.133025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:07.817831+0000","scrub_duration":0.0012243499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":240,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.475453+0000","last_change":"2024-09-11T08:17:01.558423+0000","last_active":"2024-09-11T08:17:54.475453+0000","last_peered":"2024-09-11T08:17:54.475453+0000","last_clean":"2024-09-11T08:17:54.475453+0000","last_became_active":"2024-09-11T08:07:34.053347+0000","last_became_peered":"2024-09-11T08:07:34.053347+0000","last_unstale":"2024-09-11T08:17:54.475453+0000","last_undegraded":"2024-09-11T08:17:54.475453+0000","last_fullsized":"2024-09-11T08:17:54.475453+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:01.558340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:01.558340+0000","last_clean_scrub_stamp":"2024-09-11T08:17:01.558340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:11.604522+0000","scrub_duration":0.0012693030000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":29,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.046645+0000","last_change":"2024-09-11T08:17:42.108402+0000","last_active":"2024-09-11T08:17:54.046645+0000","last_peered":"2024-09-11T08:17:54.046645+0000","last_clean":"2024-09-11T08:17:54.046645+0000","last_became_active":"2024-09-11T08:17:37.552895+0000","last_became_peered":"2024-09-11T08:17:37.552895+0000","last_unstale":"2024-09-11T08:17:54.046645+0000","last_undegraded":"2024-09-11T08:17:54.046645+0000","last_fullsized":"2024-09-11T08:17:54.046645+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:42.108342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:17:42.108342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:44.557067+0000","scrub_duration":0.001403751,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T08:17:59.811675+0000","last_change":"2024-09-11T08:17:59.811675+0000","last_active":"2024-09-11T08:17:59.811675+0000","last_peered":"2024-09-11T08:17:59.811675+0000","last_clean":"2024-09-11T08:17:59.811675+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T08:17:59.811675+0000","last_undegraded":"2024-09-11T08:17:59.811675+0000","last_fullsized":"2024-09-11T08:17:59.811675+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:17:59.811675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.d","version":"239'39","reported_seq":318,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:53.624104+0000","last_change":"2024-09-11T08:15:42.061711+0000","last_active":"2024-09-11T08:17:53.624104+0000","last_peered":"2024-09-11T08:17:53.624104+0000","last_clean":"2024-09-11T08:17:53.624104+0000","last_became_active":"2024-09-11T08:09:08.106771+0000","last_became_peered":"2024-09-11T08:09:08.106771+0000","last_unstale":"2024-09-11T08:17:53.624104+0000","last_undegraded":"2024-09-11T08:17:53.624104+0000","last_fullsized":"2024-09-11T08:17:53.624104+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'25","last_scrub_stamp":"2024-09-11T08:15:42.061632+0000","last_deep_scrub":"77'25","last_deep_scrub_stamp":"2024-09-11T08:08:32.458210+0000","last_clean_scrub_stamp":"2024-09-11T08:15:42.061632+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00500454,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":250,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.046988+0000","last_change":"2024-09-11T08:17:54.046967+0000","last_active":"2024-09-11T08:17:54.046988+0000","last_peered":"2024-09-11T08:17:54.046988+0000","last_clean":"2024-09-11T08:17:54.046988+0000","last_became_active":"2024-09-11T08:07:30.042555+0000","last_became_peered":"2024-09-11T08:07:30.042555+0000","last_unstale":"2024-09-11T08:17:54.046988+0000","last_undegraded":"2024-09-11T08:17:54.046988+0000","last_fullsized":"2024-09-11T08:17:54.046988+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:41.506939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:16:41.506939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026028340000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":241,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.048239+0000","last_change":"2024-09-11T08:16:52.542153+0000","last_active":"2024-09-11T08:17:54.048239+0000","last_peered":"2024-09-11T08:17:54.048239+0000","last_clean":"2024-09-11T08:17:54.048239+0000","last_became_active":"2024-09-11T08:07:34.054961+0000","last_became_peered":"2024-09-11T08:07:34.054961+0000","last_unstale":"2024-09-11T08:17:54.048239+0000","last_undegraded":"2024-09-11T08:17:54.048239+0000","last_fullsized":"2024-09-11T08:17:54.048239+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:52.542072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:16:52.542072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:14.525068+0000","scrub_duration":0.0052953879999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":258,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.179563+0000","last_change":"2024-09-11T08:17:32.226752+0000","last_active":"2024-09-11T08:17:54.179563+0000","last_peered":"2024-09-11T08:17:54.179563+0000","last_clean":"2024-09-11T08:17:54.179563+0000","last_became_active":"2024-09-11T08:07:36.071194+0000","last_became_peered":"2024-09-11T08:07:36.071194+0000","last_unstale":"2024-09-11T08:17:54.179563+0000","last_undegraded":"2024-09-11T08:17:54.179563+0000","last_fullsized":"2024-09-11T08:17:54.179563+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:32.226668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:07.930137+0000","last_clean_scrub_stamp":"2024-09-11T08:17:32.226668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:43.201485+0000","scrub_duration":0.0099076860000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T08:17:59.811675+0000","last_change":"2024-09-11T08:17:59.811675+0000","last_active":"2024-09-11T08:17:59.811675+0000","last_peered":"2024-09-11T08:17:59.811675+0000","last_clean":"2024-09-11T08:17:59.811675+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T08:17:59.811675+0000","last_undegraded":"2024-09-11T08:17:59.811675+0000","last_fullsized":"2024-09-11T08:17:59.811675+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:17:59.811675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.c","version":"239'25","reported_seq":268,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:53.623715+0000","last_change":"2024-09-11T08:17:01.126558+0000","last_active":"2024-09-11T08:17:53.623715+0000","last_peered":"2024-09-11T08:17:53.623715+0000","last_clean":"2024-09-11T08:17:53.623715+0000","last_became_active":"2024-09-11T08:07:32.043314+0000","last_became_peered":"2024-09-11T08:07:32.043314+0000","last_unstale":"2024-09-11T08:17:53.623715+0000","last_undegraded":"2024-09-11T08:17:53.623715+0000","last_fullsized":"2024-09-11T08:17:53.623715+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'15","last_scrub_stamp":"2024-09-11T08:17:01.126478+0000","last_deep_scrub":"224'15","last_deep_scrub_stamp":"2024-09-11T08:17:01.126478+0000","last_clean_scrub_stamp":"2024-09-11T08:17:01.126478+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:16.625548+0000","scrub_duration":0.0092237729999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":29,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.722399+0000","last_change":"2024-09-11T08:17:45.979903+0000","last_active":"2024-09-11T08:17:54.722399+0000","last_peered":"2024-09-11T08:17:54.722399+0000","last_clean":"2024-09-11T08:17:54.722399+0000","last_became_active":"2024-09-11T08:17:37.557278+0000","last_became_peered":"2024-09-11T08:17:37.557278+0000","last_unstale":"2024-09-11T08:17:54.722399+0000","last_undegraded":"2024-09-11T08:17:54.722399+0000","last_fullsized":"2024-09-11T08:17:54.722399+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:45.979847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:45.979847+0000","last_clean_scrub_stamp":"2024-09-11T08:17:45.979847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:52.778180+0000","scrub_duration":0.001534947,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"171'12","reported_seq":397,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.179447+0000","last_change":"2024-09-11T08:16:50.043791+0000","last_active":"2024-09-11T08:17:54.179447+0000","last_peered":"2024-09-11T08:17:54.179447+0000","last_clean":"2024-09-11T08:17:54.179447+0000","last_became_active":"2024-09-11T08:14:24.794446+0000","last_became_peered":"2024-09-11T08:14:24.794446+0000","last_unstale":"2024-09-11T08:17:54.179447+0000","last_undegraded":"2024-09-11T08:17:54.179447+0000","last_fullsized":"2024-09-11T08:17:54.179447+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'12","last_scrub_stamp":"2024-09-11T08:16:50.043698+0000","last_deep_scrub":"171'12","last_deep_scrub_stamp":"2024-09-11T08:14:56.881299+0000","last_clean_scrub_stamp":"2024-09-11T08:16:50.043698+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:57.810719+0000","scrub_duration":0.0069122760000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":229,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:53.624196+0000","last_change":"2024-09-11T08:17:40.124232+0000","last_active":"2024-09-11T08:17:53.624196+0000","last_peered":"2024-09-11T08:17:53.624196+0000","last_clean":"2024-09-11T08:17:53.624196+0000","last_became_active":"2024-09-11T08:07:36.074684+0000","last_became_peered":"2024-09-11T08:07:36.074684+0000","last_unstale":"2024-09-11T08:17:53.624196+0000","last_undegraded":"2024-09-11T08:17:53.624196+0000","last_fullsized":"2024-09-11T08:17:53.624196+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:40.124172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:14.613532+0000","last_clean_scrub_stamp":"2024-09-11T08:17:40.124172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:46.653254+0000","scrub_duration":0.001238495,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T08:17:59.811675+0000","last_change":"2024-09-11T08:17:59.811675+0000","last_active":"2024-09-11T08:17:59.811675+0000","last_peered":"2024-09-11T08:17:59.811675+0000","last_clean":"2024-09-11T08:17:59.811675+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T08:17:59.811675+0000","last_undegraded":"2024-09-11T08:17:59.811675+0000","last_fullsized":"2024-09-11T08:17:59.811675+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:17:59.811675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.b","version":"239'15","reported_seq":257,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.722294+0000","last_change":"2024-09-11T08:16:49.692191+0000","last_active":"2024-09-11T08:17:54.722294+0000","last_peered":"2024-09-11T08:17:54.722294+0000","last_clean":"2024-09-11T08:17:54.722294+0000","last_became_active":"2024-09-11T08:07:32.054885+0000","last_became_peered":"2024-09-11T08:07:32.054885+0000","last_unstale":"2024-09-11T08:17:54.722294+0000","last_undegraded":"2024-09-11T08:17:54.722294+0000","last_fullsized":"2024-09-11T08:17:54.722294+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:16:49.692128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:16:49.692128+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:01.355684+0000","scrub_duration":0.012332951999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":254,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.474575+0000","last_change":"2024-09-11T08:17:02.576325+0000","last_active":"2024-09-11T08:17:54.474575+0000","last_peered":"2024-09-11T08:17:54.474575+0000","last_clean":"2024-09-11T08:17:54.474575+0000","last_became_active":"2024-09-11T08:07:30.045613+0000","last_became_peered":"2024-09-11T08:07:30.045613+0000","last_unstale":"2024-09-11T08:17:54.474575+0000","last_undegraded":"2024-09-11T08:17:54.474575+0000","last_fullsized":"2024-09-11T08:17:54.474575+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:02.576236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:17:02.576236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:04.493673+0000","scrub_duration":0.003426503,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":260,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.179300+0000","last_change":"2024-09-11T08:17:33.268363+0000","last_active":"2024-09-11T08:17:54.179300+0000","last_peered":"2024-09-11T08:17:54.179300+0000","last_clean":"2024-09-11T08:17:54.179300+0000","last_became_active":"2024-09-11T08:07:34.056606+0000","last_became_peered":"2024-09-11T08:07:34.056606+0000","last_unstale":"2024-09-11T08:17:54.179300+0000","last_undegraded":"2024-09-11T08:17:54.179300+0000","last_fullsized":"2024-09-11T08:17:54.179300+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:33.268226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:17:33.268226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:56.735900+0000","scrub_duration":0.0024528100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":255,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.722194+0000","last_change":"2024-09-11T08:17:03.795482+0000","last_active":"2024-09-11T08:17:54.722194+0000","last_peered":"2024-09-11T08:17:54.722194+0000","last_clean":"2024-09-11T08:17:54.722194+0000","last_became_active":"2024-09-11T08:15:56.518884+0000","last_became_peered":"2024-09-11T08:15:56.518884+0000","last_unstale":"2024-09-11T08:17:54.722194+0000","last_undegraded":"2024-09-11T08:17:54.722194+0000","last_fullsized":"2024-09-11T08:17:54.722194+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:03.795360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:17:03.795360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:06.276216+0000","scrub_duration":0.0069097719999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T08:17:59.811675+0000","last_change":"2024-09-11T08:17:59.811675+0000","last_active":"2024-09-11T08:17:59.811675+0000","last_peered":"2024-09-11T08:17:59.811675+0000","last_clean":"2024-09-11T08:17:59.811675+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T08:17:59.811675+0000","last_undegraded":"2024-09-11T08:17:59.811675+0000","last_fullsized":"2024-09-11T08:17:59.811675+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:17:59.811675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.a","version":"239'35","reported_seq":374,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.046642+0000","last_change":"2024-09-11T08:17:32.266441+0000","last_active":"2024-09-11T08:17:54.046642+0000","last_peered":"2024-09-11T08:17:54.046642+0000","last_clean":"2024-09-11T08:17:54.046642+0000","last_became_active":"2024-09-11T08:15:56.526961+0000","last_became_peered":"2024-09-11T08:15:56.526961+0000","last_unstale":"2024-09-11T08:17:54.046642+0000","last_undegraded":"2024-09-11T08:17:54.046642+0000","last_fullsized":"2024-09-11T08:17:54.046642+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-11T08:17:32.266343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:17:32.266343+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:41.658630+0000","scrub_duration":0.0088220920000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":57,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:53.628558+0000","last_change":"2024-09-11T08:16:46.578643+0000","last_active":"2024-09-11T08:17:53.628558+0000","last_peered":"2024-09-11T08:17:53.628558+0000","last_clean":"2024-09-11T08:17:53.628558+0000","last_became_active":"2024-09-11T08:15:56.530370+0000","last_became_peered":"2024-09-11T08:15:56.530370+0000","last_unstale":"2024-09-11T08:17:53.628558+0000","last_undegraded":"2024-09-11T08:17:53.628558+0000","last_fullsized":"2024-09-11T08:17:53.628558+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:46.578575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:44.385372+0000","last_clean_scrub_stamp":"2024-09-11T08:16:46.578575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:15.109608+0000","scrub_duration":0.016632636999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":253,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.179310+0000","last_change":"2024-09-11T08:16:55.019406+0000","last_active":"2024-09-11T08:17:54.179310+0000","last_peered":"2024-09-11T08:17:54.179310+0000","last_clean":"2024-09-11T08:17:54.179310+0000","last_became_active":"2024-09-11T08:07:34.063589+0000","last_became_peered":"2024-09-11T08:07:34.063589+0000","last_unstale":"2024-09-11T08:17:54.179310+0000","last_undegraded":"2024-09-11T08:17:54.179310+0000","last_fullsized":"2024-09-11T08:17:54.179310+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:55.019351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:16:55.019351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:07.074171+0000","scrub_duration":0.001926295,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":57,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:53.628007+0000","last_change":"2024-09-11T08:17:41.786187+0000","last_active":"2024-09-11T08:17:53.628007+0000","last_peered":"2024-09-11T08:17:53.628007+0000","last_clean":"2024-09-11T08:17:53.628007+0000","last_became_active":"2024-09-11T08:15:56.518169+0000","last_became_peered":"2024-09-11T08:15:56.518169+0000","last_unstale":"2024-09-11T08:17:53.628007+0000","last_undegraded":"2024-09-11T08:17:53.628007+0000","last_fullsized":"2024-09-11T08:17:53.628007+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:41.786126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:17:41.786126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:50.563217+0000","scrub_duration":0.0015404170000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T08:17:59.811675+0000","last_change":"2024-09-11T08:17:59.811675+0000","last_active":"2024-09-11T08:17:59.811675+0000","last_peered":"2024-09-11T08:17:59.811675+0000","last_clean":"2024-09-11T08:17:59.811675+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T08:17:59.811675+0000","last_undegraded":"2024-09-11T08:17:59.811675+0000","last_fullsized":"2024-09-11T08:17:59.811675+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:17:59.811675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.9","version":"239'23","reported_seq":270,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:53.623511+0000","last_change":"2024-09-11T08:15:53.040339+0000","last_active":"2024-09-11T08:17:53.623511+0000","last_peered":"2024-09-11T08:17:53.623511+0000","last_clean":"2024-09-11T08:17:53.623511+0000","last_became_active":"2024-09-11T08:07:32.048164+0000","last_became_peered":"2024-09-11T08:07:32.048164+0000","last_unstale":"2024-09-11T08:17:53.623511+0000","last_undegraded":"2024-09-11T08:17:53.623511+0000","last_fullsized":"2024-09-11T08:17:53.623511+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'17","last_scrub_stamp":"2024-09-11T08:15:53.040223+0000","last_deep_scrub":"145'17","last_deep_scrub_stamp":"2024-09-11T08:13:08.160848+0000","last_clean_scrub_stamp":"2024-09-11T08:15:53.040223+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039160820000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":57,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:53.628637+0000","last_change":"2024-09-11T08:16:47.557730+0000","last_active":"2024-09-11T08:17:53.628637+0000","last_peered":"2024-09-11T08:17:53.628637+0000","last_clean":"2024-09-11T08:17:53.628637+0000","last_became_active":"2024-09-11T08:15:56.525845+0000","last_became_peered":"2024-09-11T08:15:56.525845+0000","last_unstale":"2024-09-11T08:17:53.628637+0000","last_undegraded":"2024-09-11T08:17:53.628637+0000","last_fullsized":"2024-09-11T08:17:53.628637+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:47.557675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:53.247275+0000","last_clean_scrub_stamp":"2024-09-11T08:16:47.557675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:01.123588+0000","scrub_duration":0.0086777009999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":240,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.179126+0000","last_change":"2024-09-11T08:16:38.072401+0000","last_active":"2024-09-11T08:17:54.179126+0000","last_peered":"2024-09-11T08:17:54.179126+0000","last_clean":"2024-09-11T08:17:54.179126+0000","last_became_active":"2024-09-11T08:07:34.057479+0000","last_became_peered":"2024-09-11T08:07:34.057479+0000","last_unstale":"2024-09-11T08:17:54.179126+0000","last_undegraded":"2024-09-11T08:17:54.179126+0000","last_fullsized":"2024-09-11T08:17:54.179126+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:38.072323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:16:38.072323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005761909,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":256,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:53.628948+0000","last_change":"2024-09-11T08:16:50.631547+0000","last_active":"2024-09-11T08:17:53.628948+0000","last_peered":"2024-09-11T08:17:53.628948+0000","last_clean":"2024-09-11T08:17:53.628948+0000","last_became_active":"2024-09-11T08:15:56.522624+0000","last_became_peered":"2024-09-11T08:15:56.522624+0000","last_unstale":"2024-09-11T08:17:53.628948+0000","last_undegraded":"2024-09-11T08:17:53.628948+0000","last_fullsized":"2024-09-11T08:17:53.628948+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:50.631471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:16:50.631471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:18.067947+0000","scrub_duration":0.0037889149999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T08:17:59.811675+0000","last_change":"2024-09-11T08:17:59.811675+0000","last_active":"2024-09-11T08:17:59.811675+0000","last_peered":"2024-09-11T08:17:59.811675+0000","last_clean":"2024-09-11T08:17:59.811675+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T08:17:59.811675+0000","last_undegraded":"2024-09-11T08:17:59.811675+0000","last_fullsized":"2024-09-11T08:17:59.811675+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:17:59.811675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.8","version":"239'27","reported_seq":344,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.474715+0000","last_change":"2024-09-11T08:17:33.548568+0000","last_active":"2024-09-11T08:17:54.474715+0000","last_peered":"2024-09-11T08:17:54.474715+0000","last_clean":"2024-09-11T08:17:54.474715+0000","last_became_active":"2024-09-11T08:15:56.532431+0000","last_became_peered":"2024-09-11T08:15:56.532431+0000","last_unstale":"2024-09-11T08:17:54.474715+0000","last_undegraded":"2024-09-11T08:17:54.474715+0000","last_fullsized":"2024-09-11T08:17:54.474715+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-11T08:17:33.548509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:17:33.548509+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:44.124562+0000","scrub_duration":0.010207872,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":63,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:53.628733+0000","last_change":"2024-09-11T08:16:42.714111+0000","last_active":"2024-09-11T08:17:53.628733+0000","last_peered":"2024-09-11T08:17:53.628733+0000","last_clean":"2024-09-11T08:17:53.628733+0000","last_became_active":"2024-09-11T08:15:56.515328+0000","last_became_peered":"2024-09-11T08:15:56.515328+0000","last_unstale":"2024-09-11T08:17:53.628733+0000","last_undegraded":"2024-09-11T08:17:53.628733+0000","last_fullsized":"2024-09-11T08:17:53.628733+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-11T08:16:42.714038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:16:42.714038+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:06.366726+0000","scrub_duration":0.0083916919999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"209'11","reported_seq":85,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:53.674315+0000","last_change":"2024-09-11T08:17:42.782853+0000","last_active":"2024-09-11T08:17:53.674315+0000","last_peered":"2024-09-11T08:17:53.674315+0000","last_clean":"2024-09-11T08:17:53.674315+0000","last_became_active":"2024-09-11T08:15:56.524513+0000","last_became_peered":"2024-09-11T08:15:56.524513+0000","last_unstale":"2024-09-11T08:17:53.674315+0000","last_undegraded":"2024-09-11T08:17:53.674315+0000","last_fullsized":"2024-09-11T08:17:53.674315+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"209'11","last_scrub_stamp":"2024-09-11T08:17:42.782786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:17:42.782786+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:09.663912+0000","scrub_duration":0.001693901,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":238,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.474895+0000","last_change":"2024-09-11T08:16:59.559972+0000","last_active":"2024-09-11T08:17:54.474895+0000","last_peered":"2024-09-11T08:17:54.474895+0000","last_clean":"2024-09-11T08:17:54.474895+0000","last_became_active":"2024-09-11T08:07:36.631755+0000","last_became_peered":"2024-09-11T08:07:36.631755+0000","last_unstale":"2024-09-11T08:17:54.474895+0000","last_undegraded":"2024-09-11T08:17:54.474895+0000","last_fullsized":"2024-09-11T08:17:54.474895+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:59.559893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:03.484755+0000","last_clean_scrub_stamp":"2024-09-11T08:16:59.559893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:28.125788+0000","scrub_duration":0.006203648,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":254,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.046557+0000","last_change":"2024-09-11T08:16:57.501989+0000","last_active":"2024-09-11T08:17:54.046557+0000","last_peered":"2024-09-11T08:17:54.046557+0000","last_clean":"2024-09-11T08:17:54.046557+0000","last_became_active":"2024-09-11T08:07:30.045910+0000","last_became_peered":"2024-09-11T08:07:30.045910+0000","last_unstale":"2024-09-11T08:17:54.046557+0000","last_undegraded":"2024-09-11T08:17:54.046557+0000","last_fullsized":"2024-09-11T08:17:54.046557+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:57.501885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:16:57.501885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:02.513820+0000","scrub_duration":0.0050562519999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"239'15","reported_seq":297,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:53.629161+0000","last_change":"2024-09-11T08:17:39.580332+0000","last_active":"2024-09-11T08:17:53.629161+0000","last_peered":"2024-09-11T08:17:53.629161+0000","last_clean":"2024-09-11T08:17:53.629161+0000","last_became_active":"2024-09-11T08:15:56.526325+0000","last_became_peered":"2024-09-11T08:15:56.526325+0000","last_unstale":"2024-09-11T08:17:53.629161+0000","last_undegraded":"2024-09-11T08:17:53.629161+0000","last_fullsized":"2024-09-11T08:17:53.629161+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'15","last_scrub_stamp":"2024-09-11T08:17:39.580272+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-09-11T08:09:03.418457+0000","last_clean_scrub_stamp":"2024-09-11T08:17:39.580272+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:54.540761+0000","scrub_duration":0.0019585050000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":240,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.475425+0000","last_change":"2024-09-11T08:16:53.549060+0000","last_active":"2024-09-11T08:17:54.475425+0000","last_peered":"2024-09-11T08:17:54.475425+0000","last_clean":"2024-09-11T08:17:54.475425+0000","last_became_active":"2024-09-11T08:07:34.057774+0000","last_became_peered":"2024-09-11T08:07:34.057774+0000","last_unstale":"2024-09-11T08:17:54.475425+0000","last_undegraded":"2024-09-11T08:17:54.475425+0000","last_fullsized":"2024-09-11T08:17:54.475425+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:53.548967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:53.548967+0000","last_clean_scrub_stamp":"2024-09-11T08:16:53.548967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:17.071272+0000","scrub_duration":0.0067671800000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":182,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:53.628662+0000","last_change":"2024-09-11T08:16:53.669111+0000","last_active":"2024-09-11T08:17:53.628662+0000","last_peered":"2024-09-11T08:17:53.628662+0000","last_clean":"2024-09-11T08:17:53.628662+0000","last_became_active":"2024-09-11T08:14:24.790678+0000","last_became_peered":"2024-09-11T08:14:24.790678+0000","last_unstale":"2024-09-11T08:17:53.628662+0000","last_undegraded":"2024-09-11T08:17:53.628662+0000","last_fullsized":"2024-09-11T08:17:53.628662+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:53.669034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:16:53.669034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:56.180252+0000","scrub_duration":0.0015615850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"239'19","reported_seq":308,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.722656+0000","last_change":"2024-09-11T08:17:31.038816+0000","last_active":"2024-09-11T08:17:54.722656+0000","last_peered":"2024-09-11T08:17:54.722656+0000","last_clean":"2024-09-11T08:17:54.722656+0000","last_became_active":"2024-09-11T08:15:56.514793+0000","last_became_peered":"2024-09-11T08:15:56.514793+0000","last_unstale":"2024-09-11T08:17:54.722656+0000","last_undegraded":"2024-09-11T08:17:54.722656+0000","last_fullsized":"2024-09-11T08:17:54.722656+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'13","last_scrub_stamp":"2024-09-11T08:17:31.038601+0000","last_deep_scrub":"84'13","last_deep_scrub_stamp":"2024-09-11T08:08:57.835634+0000","last_clean_scrub_stamp":"2024-09-11T08:17:31.038601+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:00.285015+0000","scrub_duration":0.0067181660000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":215,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:53.628316+0000","last_change":"2024-09-11T08:17:29.790967+0000","last_active":"2024-09-11T08:17:53.628316+0000","last_peered":"2024-09-11T08:17:53.628316+0000","last_clean":"2024-09-11T08:17:53.628316+0000","last_became_active":"2024-09-11T08:14:24.786021+0000","last_became_peered":"2024-09-11T08:14:24.786021+0000","last_unstale":"2024-09-11T08:17:53.628316+0000","last_undegraded":"2024-09-11T08:17:53.628316+0000","last_fullsized":"2024-09-11T08:17:53.628316+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:29.790887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:30.471620+0000","last_clean_scrub_stamp":"2024-09-11T08:17:29.790887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:49.599776+0000","scrub_duration":0.001553074,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":258,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:53.629088+0000","last_change":"2024-09-11T08:16:59.670424+0000","last_active":"2024-09-11T08:17:53.629088+0000","last_peered":"2024-09-11T08:17:53.629088+0000","last_clean":"2024-09-11T08:17:53.629088+0000","last_became_active":"2024-09-11T08:15:56.525717+0000","last_became_peered":"2024-09-11T08:15:56.525717+0000","last_unstale":"2024-09-11T08:17:53.629088+0000","last_undegraded":"2024-09-11T08:17:53.629088+0000","last_fullsized":"2024-09-11T08:17:53.629088+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:59.670299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:04.370734+0000","last_clean_scrub_stamp":"2024-09-11T08:16:59.670299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:04.460289+0000","scrub_duration":0.0026338139999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":255,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.179413+0000","last_change":"2024-09-11T08:15:56.525429+0000","last_active":"2024-09-11T08:17:54.179413+0000","last_peered":"2024-09-11T08:17:54.179413+0000","last_clean":"2024-09-11T08:17:54.179413+0000","last_became_active":"2024-09-11T08:15:56.525120+0000","last_became_peered":"2024-09-11T08:15:56.525120+0000","last_unstale":"2024-09-11T08:17:54.179413+0000","last_undegraded":"2024-09-11T08:17:54.179413+0000","last_fullsized":"2024-09-11T08:17:54.179413+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:54.184372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:54.184372+0000","last_clean_scrub_stamp":"2024-09-11T08:15:54.184372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013922767000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"239'15","reported_seq":324,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.475480+0000","last_change":"2024-09-11T08:17:34.566504+0000","last_active":"2024-09-11T08:17:54.475480+0000","last_peered":"2024-09-11T08:17:54.475480+0000","last_clean":"2024-09-11T08:17:54.475480+0000","last_became_active":"2024-09-11T08:15:56.532513+0000","last_became_peered":"2024-09-11T08:15:56.532513+0000","last_unstale":"2024-09-11T08:17:54.475480+0000","last_undegraded":"2024-09-11T08:17:54.475480+0000","last_fullsized":"2024-09-11T08:17:54.475480+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:17:34.566230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:17:34.566230+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:41.796251+0000","scrub_duration":0.0047860990000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":261,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.722342+0000","last_change":"2024-09-11T08:16:52.661667+0000","last_active":"2024-09-11T08:17:54.722342+0000","last_peered":"2024-09-11T08:17:54.722342+0000","last_clean":"2024-09-11T08:17:54.722342+0000","last_became_active":"2024-09-11T08:15:56.513278+0000","last_became_peered":"2024-09-11T08:15:56.513278+0000","last_unstale":"2024-09-11T08:17:54.722342+0000","last_undegraded":"2024-09-11T08:17:54.722342+0000","last_fullsized":"2024-09-11T08:17:54.722342+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:52.661591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:52.661591+0000","last_clean_scrub_stamp":"2024-09-11T08:16:52.661591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:02.731607+0000","scrub_duration":0.0045159830000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"239'13","reported_seq":382,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:53.674311+0000","last_change":"2024-09-11T08:17:37.558961+0000","last_active":"2024-09-11T08:17:53.674311+0000","last_peered":"2024-09-11T08:17:53.674311+0000","last_clean":"2024-09-11T08:17:53.674311+0000","last_became_active":"2024-09-11T08:17:37.558224+0000","last_became_peered":"2024-09-11T08:17:37.558224+0000","last_unstale":"2024-09-11T08:17:53.674311+0000","last_undegraded":"2024-09-11T08:17:53.674311+0000","last_fullsized":"2024-09-11T08:17:53.674311+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'12","last_scrub_stamp":"2024-09-11T08:16:52.592856+0000","last_deep_scrub":"222'12","last_deep_scrub_stamp":"2024-09-11T08:16:52.592856+0000","last_clean_scrub_stamp":"2024-09-11T08:16:52.592856+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:14.648828+0000","scrub_duration":0.0012487329999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":229,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:53.624290+0000","last_change":"2024-09-11T08:17:53.176053+0000","last_active":"2024-09-11T08:17:53.624290+0000","last_peered":"2024-09-11T08:17:53.624290+0000","last_clean":"2024-09-11T08:17:53.624290+0000","last_became_active":"2024-09-11T08:07:36.068493+0000","last_became_peered":"2024-09-11T08:07:36.068493+0000","last_unstale":"2024-09-11T08:17:53.624290+0000","last_undegraded":"2024-09-11T08:17:53.624290+0000","last_fullsized":"2024-09-11T08:17:53.624290+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:53.175994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:53.175994+0000","last_clean_scrub_stamp":"2024-09-11T08:17:53.175994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:04.570339+0000","scrub_duration":0.001353499,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"239'20","reported_seq":104,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.179355+0000","last_change":"2024-09-11T08:17:53.206017+0000","last_active":"2024-09-11T08:17:54.179355+0000","last_peered":"2024-09-11T08:17:54.179355+0000","last_clean":"2024-09-11T08:17:54.179355+0000","last_became_active":"2024-09-11T08:17:37.554357+0000","last_became_peered":"2024-09-11T08:17:37.554357+0000","last_unstale":"2024-09-11T08:17:54.179355+0000","last_undegraded":"2024-09-11T08:17:54.179355+0000","last_fullsized":"2024-09-11T08:17:54.179355+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'20","last_scrub_stamp":"2024-09-11T08:17:53.205922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:17:53.205922+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:22.709127+0000","scrub_duration":0.0017765369999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":57,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:53.628356+0000","last_change":"2024-09-11T08:17:03.708286+0000","last_active":"2024-09-11T08:17:53.628356+0000","last_peered":"2024-09-11T08:17:53.628356+0000","last_clean":"2024-09-11T08:17:53.628356+0000","last_became_active":"2024-09-11T08:15:56.518515+0000","last_became_peered":"2024-09-11T08:15:56.518515+0000","last_unstale":"2024-09-11T08:17:53.628356+0000","last_undegraded":"2024-09-11T08:17:53.628356+0000","last_fullsized":"2024-09-11T08:17:53.628356+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:03.708205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:17:03.708205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:30.336677+0000","scrub_duration":0.0039365750000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":373,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.475380+0000","last_change":"2024-09-11T08:17:44.632426+0000","last_active":"2024-09-11T08:17:54.475380+0000","last_peered":"2024-09-11T08:17:54.475380+0000","last_clean":"2024-09-11T08:17:54.475380+0000","last_became_active":"2024-09-11T08:07:34.060736+0000","last_became_peered":"2024-09-11T08:07:34.060736+0000","last_unstale":"2024-09-11T08:17:54.475380+0000","last_undegraded":"2024-09-11T08:17:54.475380+0000","last_fullsized":"2024-09-11T08:17:54.475380+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:17:44.632353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:17:44.632353+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:03.078582+0000","scrub_duration":0.001735232,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":255,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.722480+0000","last_change":"2024-09-11T08:17:42.030917+0000","last_active":"2024-09-11T08:17:54.722480+0000","last_peered":"2024-09-11T08:17:54.722480+0000","last_clean":"2024-09-11T08:17:54.722480+0000","last_became_active":"2024-09-11T08:15:56.513949+0000","last_became_peered":"2024-09-11T08:15:56.513949+0000","last_unstale":"2024-09-11T08:17:54.722480+0000","last_undegraded":"2024-09-11T08:17:54.722480+0000","last_fullsized":"2024-09-11T08:17:54.722480+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:42.030854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:17:42.030854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:05.644587+0000","scrub_duration":0.0017212379999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"239'19","reported_seq":264,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:53.623749+0000","last_change":"2024-09-11T08:16:55.089638+0000","last_active":"2024-09-11T08:17:53.623749+0000","last_peered":"2024-09-11T08:17:53.623749+0000","last_clean":"2024-09-11T08:17:53.623749+0000","last_became_active":"2024-09-11T08:07:32.047981+0000","last_became_peered":"2024-09-11T08:07:32.047981+0000","last_unstale":"2024-09-11T08:17:53.623749+0000","last_undegraded":"2024-09-11T08:17:53.623749+0000","last_fullsized":"2024-09-11T08:17:53.623749+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-11T08:16:55.089545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:16:55.089545+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:12.681553+0000","scrub_duration":0.0037132879999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":262,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:53.623814+0000","last_change":"2024-09-11T08:17:31.186217+0000","last_active":"2024-09-11T08:17:53.623814+0000","last_peered":"2024-09-11T08:17:53.623814+0000","last_clean":"2024-09-11T08:17:53.623814+0000","last_became_active":"2024-09-11T08:15:56.523574+0000","last_became_peered":"2024-09-11T08:15:56.523574+0000","last_unstale":"2024-09-11T08:17:53.623814+0000","last_undegraded":"2024-09-11T08:17:53.623814+0000","last_fullsized":"2024-09-11T08:17:53.623814+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:31.186045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:17:31.186045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:33.322805+0000","scrub_duration":0.0016617730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":244,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.048096+0000","last_change":"2024-09-11T08:17:03.475878+0000","last_active":"2024-09-11T08:17:54.048096+0000","last_peered":"2024-09-11T08:17:54.048096+0000","last_clean":"2024-09-11T08:17:54.048096+0000","last_became_active":"2024-09-11T08:07:34.053137+0000","last_became_peered":"2024-09-11T08:07:34.053137+0000","last_unstale":"2024-09-11T08:17:54.048096+0000","last_undegraded":"2024-09-11T08:17:54.048096+0000","last_fullsized":"2024-09-11T08:17:54.048096+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:03.475805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:17:03.475805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:26.468256+0000","scrub_duration":0.012934157,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":29,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.722094+0000","last_change":"2024-09-11T08:17:52.926082+0000","last_active":"2024-09-11T08:17:54.722094+0000","last_peered":"2024-09-11T08:17:54.722094+0000","last_clean":"2024-09-11T08:17:54.722094+0000","last_became_active":"2024-09-11T08:17:37.556871+0000","last_became_peered":"2024-09-11T08:17:37.556871+0000","last_unstale":"2024-09-11T08:17:54.722094+0000","last_undegraded":"2024-09-11T08:17:54.722094+0000","last_fullsized":"2024-09-11T08:17:54.722094+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:52.925992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:17:52.925992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:18.928220+0000","scrub_duration":0.0087812989999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"239'15","reported_seq":278,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.048271+0000","last_change":"2024-09-11T08:17:43.635561+0000","last_active":"2024-09-11T08:17:54.048271+0000","last_peered":"2024-09-11T08:17:54.048271+0000","last_clean":"2024-09-11T08:17:54.048271+0000","last_became_active":"2024-09-11T08:07:32.049368+0000","last_became_peered":"2024-09-11T08:07:32.049368+0000","last_unstale":"2024-09-11T08:17:54.048271+0000","last_undegraded":"2024-09-11T08:17:54.048271+0000","last_fullsized":"2024-09-11T08:17:54.048271+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'15","last_scrub_stamp":"2024-09-11T08:17:43.635506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:17:43.635506+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:07.068338+0000","scrub_duration":0.0022043800000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":57,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:53.628438+0000","last_change":"2024-09-11T08:16:51.628468+0000","last_active":"2024-09-11T08:17:53.628438+0000","last_peered":"2024-09-11T08:17:53.628438+0000","last_clean":"2024-09-11T08:17:53.628438+0000","last_became_active":"2024-09-11T08:15:56.521570+0000","last_became_peered":"2024-09-11T08:15:56.521570+0000","last_unstale":"2024-09-11T08:17:53.628438+0000","last_undegraded":"2024-09-11T08:17:53.628438+0000","last_fullsized":"2024-09-11T08:17:53.628438+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:51.628378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:16:51.628378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001961422,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":29,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.722175+0000","last_change":"2024-09-11T08:17:39.028807+0000","last_active":"2024-09-11T08:17:54.722175+0000","last_peered":"2024-09-11T08:17:54.722175+0000","last_clean":"2024-09-11T08:17:54.722175+0000","last_became_active":"2024-09-11T08:17:37.557645+0000","last_became_peered":"2024-09-11T08:17:37.557645+0000","last_unstale":"2024-09-11T08:17:54.722175+0000","last_undegraded":"2024-09-11T08:17:54.722175+0000","last_fullsized":"2024-09-11T08:17:54.722175+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:39.028741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:01.218475+0000","last_clean_scrub_stamp":"2024-09-11T08:17:39.028741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:02.357273+0000","scrub_duration":0.0012042629999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":228,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.722008+0000","last_change":"2024-09-11T08:16:39.635237+0000","last_active":"2024-09-11T08:17:54.722008+0000","last_peered":"2024-09-11T08:17:54.722008+0000","last_clean":"2024-09-11T08:17:54.722008+0000","last_became_active":"2024-09-11T08:07:36.077406+0000","last_became_peered":"2024-09-11T08:07:36.077406+0000","last_unstale":"2024-09-11T08:17:54.722008+0000","last_undegraded":"2024-09-11T08:17:54.722008+0000","last_fullsized":"2024-09-11T08:17:54.722008+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:39.635151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:09.973943+0000","last_clean_scrub_stamp":"2024-09-11T08:16:39.635151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0062349479999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"239'19","reported_seq":315,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:53.628562+0000","last_change":"2024-09-11T08:16:53.599766+0000","last_active":"2024-09-11T08:17:53.628562+0000","last_peered":"2024-09-11T08:17:53.628562+0000","last_clean":"2024-09-11T08:17:53.628562+0000","last_became_active":"2024-09-11T08:15:56.520058+0000","last_became_peered":"2024-09-11T08:15:56.520058+0000","last_unstale":"2024-09-11T08:17:53.628562+0000","last_undegraded":"2024-09-11T08:17:53.628562+0000","last_fullsized":"2024-09-11T08:17:53.628562+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-11T08:16:53.599677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:16:53.599677+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:20.621343+0000","scrub_duration":0.0071859539999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":261,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.475466+0000","last_change":"2024-09-11T08:15:56.531914+0000","last_active":"2024-09-11T08:17:54.475466+0000","last_peered":"2024-09-11T08:17:54.475466+0000","last_clean":"2024-09-11T08:17:54.475466+0000","last_became_active":"2024-09-11T08:15:56.531612+0000","last_became_peered":"2024-09-11T08:15:56.531612+0000","last_unstale":"2024-09-11T08:17:54.475466+0000","last_undegraded":"2024-09-11T08:17:54.475466+0000","last_fullsized":"2024-09-11T08:17:54.475466+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:49.478359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:49.478359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012202396000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":57,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:53.628203+0000","last_change":"2024-09-11T08:16:38.643733+0000","last_active":"2024-09-11T08:17:53.628203+0000","last_peered":"2024-09-11T08:17:53.628203+0000","last_clean":"2024-09-11T08:17:53.628203+0000","last_became_active":"2024-09-11T08:15:56.523728+0000","last_became_peered":"2024-09-11T08:15:56.523728+0000","last_unstale":"2024-09-11T08:17:53.628203+0000","last_undegraded":"2024-09-11T08:17:53.628203+0000","last_fullsized":"2024-09-11T08:17:53.628203+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:38.643651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:11:34.159053+0000","last_clean_scrub_stamp":"2024-09-11T08:16:38.643651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:17:58.036696+0000","scrub_duration":0.002366469,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":229,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:53.629129+0000","last_change":"2024-09-11T08:17:46.489320+0000","last_active":"2024-09-11T08:17:53.629129+0000","last_peered":"2024-09-11T08:17:53.629129+0000","last_clean":"2024-09-11T08:17:53.629129+0000","last_became_active":"2024-09-11T08:07:36.633368+0000","last_became_peered":"2024-09-11T08:07:36.633368+0000","last_unstale":"2024-09-11T08:17:53.629129+0000","last_undegraded":"2024-09-11T08:17:53.629129+0000","last_fullsized":"2024-09-11T08:17:53.629129+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:46.489220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:46.489220+0000","last_clean_scrub_stamp":"2024-09-11T08:17:46.489220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:04.268669+0000","scrub_duration":0.0015753690000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"239'8","reported_seq":245,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:53.628428+0000","last_change":"2024-09-11T08:17:50.580924+0000","last_active":"2024-09-11T08:17:53.628428+0000","last_peered":"2024-09-11T08:17:53.628428+0000","last_clean":"2024-09-11T08:17:53.628428+0000","last_became_active":"2024-09-11T08:07:32.045728+0000","last_became_peered":"2024-09-11T08:07:32.045728+0000","last_unstale":"2024-09-11T08:17:53.628428+0000","last_undegraded":"2024-09-11T08:17:53.628428+0000","last_fullsized":"2024-09-11T08:17:53.628428+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'8","last_scrub_stamp":"2024-09-11T08:17:50.580863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:17:50.580863+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:09.146709+0000","scrub_duration":0.001681075,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":244,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.721958+0000","last_change":"2024-09-11T08:17:48.965635+0000","last_active":"2024-09-11T08:17:54.721958+0000","last_peered":"2024-09-11T08:17:54.721958+0000","last_clean":"2024-09-11T08:17:54.721958+0000","last_became_active":"2024-09-11T08:07:30.049409+0000","last_became_peered":"2024-09-11T08:07:30.049409+0000","last_unstale":"2024-09-11T08:17:54.721958+0000","last_undegraded":"2024-09-11T08:17:54.721958+0000","last_fullsized":"2024-09-11T08:17:54.721958+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:48.965582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:17:48.965582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:52.654703+0000","scrub_duration":0.001302138,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":277,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.046492+0000","last_change":"2024-09-11T08:16:51.403463+0000","last_active":"2024-09-11T08:17:54.046492+0000","last_peered":"2024-09-11T08:17:54.046492+0000","last_clean":"2024-09-11T08:17:54.046492+0000","last_became_active":"2024-09-11T08:15:56.528607+0000","last_became_peered":"2024-09-11T08:15:56.528607+0000","last_unstale":"2024-09-11T08:17:54.046492+0000","last_undegraded":"2024-09-11T08:17:54.046492+0000","last_fullsized":"2024-09-11T08:17:54.046492+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:51.403370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:16:51.403370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002498068,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":57,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:53.628229+0000","last_change":"2024-09-11T08:17:31.857979+0000","last_active":"2024-09-11T08:17:53.628229+0000","last_peered":"2024-09-11T08:17:53.628229+0000","last_clean":"2024-09-11T08:17:53.628229+0000","last_became_active":"2024-09-11T08:15:56.521488+0000","last_became_peered":"2024-09-11T08:15:56.521488+0000","last_unstale":"2024-09-11T08:17:53.628229+0000","last_undegraded":"2024-09-11T08:17:53.628229+0000","last_fullsized":"2024-09-11T08:17:53.628229+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:31.857906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:17:31.857906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:43.754475+0000","scrub_duration":0.011557305,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":239,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.048151+0000","last_change":"2024-09-11T08:16:50.585648+0000","last_active":"2024-09-11T08:17:54.048151+0000","last_peered":"2024-09-11T08:17:54.048151+0000","last_clean":"2024-09-11T08:17:54.048151+0000","last_became_active":"2024-09-11T08:07:36.071468+0000","last_became_peered":"2024-09-11T08:07:36.071468+0000","last_unstale":"2024-09-11T08:17:54.048151+0000","last_undegraded":"2024-09-11T08:17:54.048151+0000","last_fullsized":"2024-09-11T08:17:54.048151+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:50.585566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:50.585566+0000","last_clean_scrub_stamp":"2024-09-11T08:16:50.585566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:20.341597+0000","scrub_duration":0.0073189980000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":29,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.722285+0000","last_change":"2024-09-11T08:17:51.950776+0000","last_active":"2024-09-11T08:17:54.722285+0000","last_peered":"2024-09-11T08:17:54.722285+0000","last_clean":"2024-09-11T08:17:54.722285+0000","last_became_active":"2024-09-11T08:17:37.554262+0000","last_became_peered":"2024-09-11T08:17:37.554262+0000","last_unstale":"2024-09-11T08:17:54.722285+0000","last_undegraded":"2024-09-11T08:17:54.722285+0000","last_fullsized":"2024-09-11T08:17:54.722285+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:51.950689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:17:51.950689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:08.838111+0000","scrub_duration":0.0014240730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"239'19","reported_seq":283,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.047007+0000","last_change":"2024-09-11T08:17:30.294601+0000","last_active":"2024-09-11T08:17:54.047007+0000","last_peered":"2024-09-11T08:17:54.047007+0000","last_clean":"2024-09-11T08:17:54.047007+0000","last_became_active":"2024-09-11T08:07:32.054656+0000","last_became_peered":"2024-09-11T08:07:32.054656+0000","last_unstale":"2024-09-11T08:17:54.047007+0000","last_undegraded":"2024-09-11T08:17:54.047007+0000","last_fullsized":"2024-09-11T08:17:54.047007+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-11T08:17:30.294514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:17:30.294514+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:44.544439+0000","scrub_duration":0.0051598069999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":75,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-11T08:17:54.046687+0000","last_change":"2024-09-11T08:16:46.497719+0000","last_active":"2024-09-11T08:17:54.046687+0000","last_peered":"2024-09-11T08:17:54.046687+0000","last_clean":"2024-09-11T08:17:54.046687+0000","last_became_active":"2024-09-11T08:15:08.769133+0000","last_became_peered":"2024-09-11T08:15:08.769133+0000","last_unstale":"2024-09-11T08:17:54.046687+0000","last_undegraded":"2024-09-11T08:17:54.046687+0000","last_fullsized":"2024-09-11T08:17:54.046687+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:46.497639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:16:46.497639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:14.479718+0000","scrub_duration":0.0048914830000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1127,"num_read_kb":884,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":2448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3425,"internal_metadata":0},"log_size":741,"ondisk_log_size":741,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":219,"num_read_kb":214,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":168,"seq":721554505772,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":318916,"kb_used_data":1280,"kb_used_omap":7,"kb_used_meta":317496,"kb_avail":93426236,"statfs":{"total":95995035648,"available":95668465664,"internally_reserved":0,"allocated":1310720,"data_stored":954523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8027,"internal_metadata":325116069},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:17:27 2024","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":0.99399999999999999,"15min":0.88200000000000001},"min":{"1min":0.24199999999999999,"5min":0.223,"15min":0.223},"max":{"1min":3.8260000000000001,"5min":4.673,"15min":4.673},"last":0.53000000000000003},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.0029999999999999,"15min":0.997},"min":{"1min":0.22600000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":5.2610000000000001,"5min":5.2610000000000001,"15min":5.2610000000000001},"last":0.49299999999999999}]},{"osd":1,"last update":"Wed Sep 11 08:17:27 2024","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":2.3119999999999998,"15min":2.0680000000000001},"min":{"1min":0.28799999999999998,"5min":0.25,"15min":0.25},"max":{"1min":5.431,"5min":42.082999999999998,"15min":42.082999999999998},"last":0.60699999999999998},{"interface":"front","average":{"1min":1.075,"5min":2.1499999999999999,"15min":2.0800000000000001},"min":{"1min":0.28699999999999998,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":2.5990000000000002,"5min":42.151000000000003,"15min":42.151000000000003},"last":0.50600000000000001}]},{"osd":2,"last update":"Wed Sep 11 08:17:27 2024","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.93600000000000005,"15min":1.006},"min":{"1min":0.26500000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":2.984,"5min":3.9630000000000001,"15min":3.9630000000000001},"last":0.51800000000000002},{"interface":"front","average":{"1min":1.1479999999999999,"5min":0.81799999999999995,"15min":0.753},"min":{"1min":0.30299999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":5.5170000000000003,"5min":5.5170000000000003,"15min":5.5170000000000003},"last":0.58499999999999996}]},{"osd":3,"last update":"Wed Sep 11 08:17:27 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":1.319,"15min":1.155},"min":{"1min":0.28499999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":3.0830000000000002,"5min":21.460000000000001,"15min":21.460000000000001},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.995,"5min":1.5569999999999999,"15min":1.4690000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":5.585,"5min":21.440000000000001,"15min":21.440000000000001},"last":0.57099999999999995}]},{"osd":4,"last update":"Wed Sep 11 08:17:27 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.66300000000000003,"15min":0.57099999999999995},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":1.544,"5min":4.141,"15min":4.141},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.63,"5min":0.71299999999999997,"15min":0.73999999999999999},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.423,"5min":3.157,"15min":3.157},"last":0.34399999999999997}]},{"osd":5,"last update":"Wed Sep 11 08:17:27 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.88,"15min":0.82599999999999996},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.8220000000000001,"5min":4.1379999999999999,"15min":4.1379999999999999},"last":0.372},{"interface":"front","average":{"1min":0.621,"5min":0.79600000000000004,"15min":0.76500000000000001},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.4470000000000001,"5min":3.907,"15min":3.907},"last":0.40200000000000002}]},{"osd":6,"last update":"Wed Sep 11 08:17:27 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.77800000000000002,"15min":0.78800000000000003},"min":{"1min":0.33400000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.92500000000000004,"5min":6.0609999999999999,"15min":6.0609999999999999},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.81200000000000006,"15min":0.79700000000000004},"min":{"1min":0.38,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.5229999999999999,"5min":3.6829999999999998,"15min":3.6829999999999998},"last":0.442}]}]},{"osd":6,"up_from":45,"seq":193273528455,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":450552,"kb_used_data":148912,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93294600,"statfs":{"total":95995035648,"available":95533670400,"internally_reserved":0,"allocated":152485888,"data_stored":152145898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:17:15 2024","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":1.4890000000000001,"15min":1.083},"min":{"1min":0.21299999999999999,"5min":0.20399999999999999,"15min":0.185},"max":{"1min":2.589,"5min":45.066000000000003,"15min":45.066000000000003},"last":0.443},{"interface":"front","average":{"1min":1.72,"5min":1.7609999999999999,"15min":1.107},"min":{"1min":0.27000000000000002,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":12.672000000000001,"5min":43.899000000000001,"15min":43.899000000000001},"last":0.38300000000000001}]},{"osd":1,"last update":"Wed Sep 11 08:17:15 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":1.5229999999999999,"15min":0.90500000000000003},"min":{"1min":0.24199999999999999,"5min":0.214,"15min":0.214},"max":{"1min":1.4119999999999999,"5min":44.658000000000001,"15min":44.658000000000001},"last":0.34300000000000003},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.788,"15min":1.012},"min":{"1min":0.22800000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":4.8419999999999996,"5min":44.694000000000003,"15min":44.694000000000003},"last":0.45100000000000001}]},{"osd":2,"last update":"Wed Sep 11 08:17:15 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":0.97799999999999998,"15min":0.72699999999999998},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":2.7749999999999999,"5min":5.9770000000000003,"15min":5.9770000000000003},"last":0.36699999999999999},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.302,"15min":0.90000000000000002},"min":{"1min":0.30199999999999999,"5min":0.252,"15min":0.23300000000000001},"max":{"1min":3.8239999999999998,"5min":7.4809999999999999,"15min":13.346},"last":0.433}]},{"osd":3,"last update":"Wed Sep 11 08:17:15 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":1.2509999999999999,"15min":1.028},"min":{"1min":0.27700000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":2.8700000000000001,"5min":24.850999999999999,"15min":36.061},"last":0.46400000000000002},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.2929999999999999,"15min":1.0249999999999999},"min":{"1min":0.29799999999999999,"5min":0.28399999999999997,"15min":0.26800000000000002},"max":{"1min":3.5640000000000001,"5min":24.876000000000001,"15min":36.093000000000004},"last":0.48099999999999998}]},{"osd":4,"last update":"Wed Sep 11 08:17:15 2024","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.65000000000000002,"15min":0.48699999999999999},"min":{"1min":0.251,"5min":0.223,"15min":0.20200000000000001},"max":{"1min":3.8780000000000001,"5min":4.1130000000000004,"15min":4.1130000000000004},"last":0.313},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.67700000000000005,"15min":0.51600000000000001},"min":{"1min":0.32400000000000001,"5min":0.254,"15min":0.24099999999999999},"max":{"1min":2.4590000000000001,"5min":4.468,"15min":4.468},"last":0.35499999999999998}]},{"osd":5,"last update":"Wed Sep 11 08:17:15 2024","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.83599999999999997,"15min":0.60899999999999999},"min":{"1min":0.34899999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":4.1210000000000004,"5min":7.4459999999999997,"15min":7.4459999999999997},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.75,"15min":0.55400000000000005},"min":{"1min":0.28199999999999997,"5min":0.22800000000000001,"15min":0.22600000000000001},"max":{"1min":1.8720000000000001,"5min":3.4329999999999998,"15min":3.4329999999999998},"last":0.42099999999999999}]},{"osd":7,"last update":"Wed Sep 11 08:17:25 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":1.0720000000000001,"15min":1.121},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":1.911,"5min":8.4580000000000002,"15min":8.4580000000000002},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.872,"15min":0.89200000000000002},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.8480000000000001,"5min":3.593,"15min":3.593},"last":0.40699999999999997}]}]},{"osd":5,"up_from":38,"seq":163208757388,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":422968,"kb_used_data":117168,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93322184,"statfs":{"total":95995035648,"available":95561916416,"internally_reserved":0,"allocated":119980032,"data_stored":119639059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:17:54 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":1.0860000000000001,"15min":0.82799999999999996},"min":{"1min":0.214,"5min":0.184,"15min":0.184},"max":{"1min":5.0990000000000002,"5min":7.9470000000000001,"15min":7.9470000000000001},"last":0.35299999999999998},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.2829999999999999,"15min":0.91900000000000004},"min":{"1min":0.311,"5min":0.23000000000000001,"15min":0.20399999999999999},"max":{"1min":5.6210000000000004,"5min":13.077,"15min":13.077},"last":0.435}]},{"osd":1,"last update":"Wed Sep 11 08:17:54 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":1.248,"15min":1.2130000000000001},"min":{"1min":0.251,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":2.105,"5min":26.709,"15min":26.709},"last":0.46100000000000002},{"interface":"front","average":{"1min":1.02,"5min":1.248,"15min":0.93000000000000005},"min":{"1min":0.311,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.8570000000000002,"5min":26.754000000000001,"15min":26.754000000000001},"last":0.36499999999999999}]},{"osd":2,"last update":"Wed Sep 11 08:17:54 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":1.0289999999999999,"15min":1.0760000000000001},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.3100000000000001,"5min":10.891,"15min":16.018000000000001},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.875,"15min":0.70899999999999996},"min":{"1min":0.33200000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":1.296,"5min":7.5250000000000004,"15min":7.5250000000000004},"last":0.33200000000000002}]},{"osd":3,"last update":"Wed Sep 11 08:17:54 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":1.1930000000000001,"15min":0.88300000000000001},"min":{"1min":0.25800000000000001,"5min":0.25,"15min":0.25},"max":{"1min":3.2120000000000002,"5min":24.931000000000001,"15min":24.931000000000001},"last":0.29599999999999999},{"interface":"front","average":{"1min":1.002,"5min":1.4179999999999999,"15min":0.94899999999999995},"min":{"1min":0.36499999999999999,"5min":0.27700000000000002,"15min":0.251},"max":{"1min":4.3150000000000004,"5min":24.861999999999998,"15min":24.861999999999998},"last":0.38}]},{"osd":4,"last update":"Wed Sep 11 08:17:54 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.627,"15min":0.51000000000000001},"min":{"1min":0.27200000000000002,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":2.1269999999999998,"5min":3.5979999999999999,"15min":3.5979999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.437,"5min":0.58799999999999997,"15min":0.49199999999999999},"min":{"1min":0.28100000000000003,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.93700000000000006,"5min":4.9210000000000003,"15min":4.9210000000000003},"last":0.39100000000000001}]},{"osd":6,"last update":"Wed Sep 11 08:17:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.56999999999999995,"15min":0.48299999999999998},"min":{"1min":0.29299999999999998,"5min":0.252,"15min":0.23400000000000001},"max":{"1min":1.988,"5min":3.1869999999999998,"15min":3.1869999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.68600000000000005,"15min":0.55000000000000004},"min":{"1min":0.375,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":2.4140000000000001,"5min":3.9489999999999998,"15min":3.9489999999999998},"last":0.48199999999999998}]},{"osd":7,"last update":"Wed Sep 11 08:17:28 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.84999999999999998,"15min":0.83699999999999997},"min":{"1min":0.33900000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":2.2759999999999998,"5min":3.5459999999999998,"15min":3.5459999999999998},"last":0.496},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.78700000000000003,"15min":0.77100000000000002},"min":{"1min":0.35799999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.486,"5min":3.1019999999999999,"15min":3.1019999999999999},"last":0.45100000000000001}]}]},{"osd":4,"up_from":31,"seq":133143986321,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":310664,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93434488,"statfs":{"total":95995035648,"available":95676915712,"internally_reserved":0,"allocated":724992,"data_stored":364853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:17:21 2024","interfaces":[{"interface":"back","average":{"1min":3.895,"5min":1.546,"15min":1.292},"min":{"1min":0.22800000000000001,"5min":0.19,"15min":0.19},"max":{"1min":60.497999999999998,"5min":60.497999999999998,"15min":60.497999999999998},"last":0.39400000000000002},{"interface":"front","average":{"1min":3.77,"5min":1.429,"15min":1.1479999999999999},"min":{"1min":0.24299999999999999,"5min":0.221,"15min":0.221},"max":{"1min":60.484999999999999,"5min":60.484999999999999,"15min":60.484999999999999},"last":0.54300000000000004}]},{"osd":1,"last update":"Wed Sep 11 08:17:21 2024","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":1.01,"15min":0.78900000000000003},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.214},"max":{"1min":3.1099999999999999,"5min":4.7320000000000002,"15min":13.207000000000001},"last":0.49399999999999999},{"interface":"front","average":{"1min":1.0600000000000001,"5min":0.94399999999999995,"15min":0.75700000000000001},"min":{"1min":0.26800000000000002,"5min":0.24299999999999999,"15min":0.22700000000000001},"max":{"1min":3.992,"5min":5.8869999999999996,"15min":5.8869999999999996},"last":0.44500000000000001}]},{"osd":2,"last update":"Wed Sep 11 08:17:21 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.91600000000000004,"15min":0.747},"min":{"1min":0.255,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":1.651,"5min":7.976,"15min":7.976},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.77900000000000003,"15min":0.71999999999999997},"min":{"1min":0.28399999999999997,"5min":0.248,"15min":0.248},"max":{"1min":1.802,"5min":3.1389999999999998,"15min":5.5670000000000002},"last":0.51000000000000001}]},{"osd":3,"last update":"Wed Sep 11 08:17:21 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":1.1830000000000001,"15min":1.054},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":1.8600000000000001,"5min":19.686,"15min":26.884},"last":0.57299999999999995},{"interface":"front","average":{"1min":1.3069999999999999,"5min":1.3759999999999999,"15min":1.107},"min":{"1min":0.35499999999999998,"5min":0.246,"15min":0.246},"max":{"1min":2.9510000000000001,"5min":19.882000000000001,"15min":26.783000000000001},"last":0.43099999999999999}]},{"osd":5,"last update":"Wed Sep 11 08:17:49 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.58299999999999996,"15min":0.46200000000000002},"min":{"1min":0.26500000000000001,"5min":0.23599999999999999,"15min":0.23499999999999999},"max":{"1min":1.6850000000000001,"5min":2.7229999999999999,"15min":2.7229999999999999},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.93799999999999994,"5min":0.64700000000000002,"15min":0.48599999999999999},"min":{"1min":0.33100000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":5.2549999999999999,"5min":5.2549999999999999,"15min":5.2549999999999999},"last":0.33600000000000002}]},{"osd":6,"last update":"Wed Sep 11 08:17:15 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.99299999999999999,"15min":0.61299999999999999},"min":{"1min":0.25,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":1.8260000000000001,"5min":48.808,"15min":48.808},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.62,"5min":1.0389999999999999,"15min":0.66100000000000003},"min":{"1min":0.26800000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.579,"5min":48.746000000000002,"15min":48.746000000000002},"last":0.35599999999999998}]},{"osd":7,"last update":"Wed Sep 11 08:17:34 2024","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.65700000000000003,"15min":0.53700000000000003},"min":{"1min":0.30499999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":5.3449999999999998,"5min":5.3449999999999998,"15min":5.3449999999999998},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.749,"15min":0.69099999999999995},"min":{"1min":0.307,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":4.7160000000000002,"5min":4.7160000000000002,"15min":4.7160000000000002},"last":0.66200000000000003}]}]},{"osd":3,"up_from":25,"seq":107374182550,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":310676,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93434476,"statfs":{"total":95995035648,"available":95676903424,"internally_reserved":0,"allocated":737280,"data_stored":364324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:17:07 2024","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":2.081,"15min":1.1439999999999999},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":5.4710000000000001,"5min":60.469999999999999,"15min":60.469999999999999},"last":0.311},{"interface":"front","average":{"1min":1.1060000000000001,"5min":2.2450000000000001,"15min":1.2450000000000001},"min":{"1min":0.23000000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":3.298,"5min":60.488999999999997,"15min":60.488999999999997},"last":0.28399999999999997}]},{"osd":1,"last update":"Wed Sep 11 08:17:07 2024","interfaces":[{"interface":"back","average":{"1min":1.24,"5min":2.1669999999999998,"15min":1.1379999999999999},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":4.0039999999999996,"5min":60.578000000000003,"15min":60.578000000000003},"last":0.34699999999999998},{"interface":"front","average":{"1min":3.3530000000000002,"5min":2.544,"15min":1.3779999999999999},"min":{"1min":0.216,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":46.868000000000002,"5min":63.674999999999997,"15min":63.674999999999997},"last":0.36299999999999999}]},{"osd":2,"last update":"Wed Sep 11 08:17:07 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":2.327,"15min":1.2689999999999999},"min":{"1min":0.26500000000000001,"5min":0.185,"15min":0.185},"max":{"1min":3.081,"5min":60.645000000000003,"15min":60.645000000000003},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.93700000000000006,"5min":2.0670000000000002,"15min":1.1719999999999999},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20599999999999999},"max":{"1min":3.012,"5min":60.723999999999997,"15min":60.723999999999997},"last":0.373}]},{"osd":4,"last update":"Wed Sep 11 08:17:23 2024","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.8169999999999999,"15min":1.105},"min":{"1min":0.29499999999999998,"5min":0.28100000000000003,"15min":0.24299999999999999},"max":{"1min":5.5650000000000004,"5min":61.290999999999997,"15min":61.290999999999997},"last":0.56100000000000005},{"interface":"front","average":{"1min":1.081,"5min":1.752,"15min":1.0609999999999999},"min":{"1min":0.311,"5min":0.28899999999999998,"15min":0.27200000000000002},"max":{"1min":6.1340000000000003,"5min":60.856000000000002,"15min":60.856000000000002},"last":0.52200000000000002}]},{"osd":5,"last update":"Wed Sep 11 08:17:53 2024","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":2.2869999999999999,"15min":1.276},"min":{"1min":0.34399999999999997,"5min":0.28199999999999997,"15min":0.26500000000000001},"max":{"1min":4.1509999999999998,"5min":60.914999999999999,"15min":60.914999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":1.3200000000000001,"5min":2.403,"15min":1.3680000000000001},"min":{"1min":0.38400000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.3380000000000001,"5min":60.787999999999997,"15min":60.787999999999997},"last":0.59499999999999997}]},{"osd":6,"last update":"Wed Sep 11 08:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":2.3889999999999998,"15min":1.3080000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":2.609,"5min":60.817,"15min":60.817},"last":0.57199999999999995},{"interface":"front","average":{"1min":1.49,"5min":2.4369999999999998,"15min":1.4330000000000001},"min":{"1min":0.34699999999999998,"5min":0.307,"15min":0.307},"max":{"1min":11.837,"5min":61.264000000000003,"15min":61.264000000000003},"last":0.505}]},{"osd":7,"last update":"Wed Sep 11 08:17:27 2024","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":2.9380000000000002,"15min":3.0739999999999998},"min":{"1min":0.40999999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":6.2770000000000001,"5min":60.847000000000001,"15min":60.847000000000001},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.95799999999999996,"5min":2.8100000000000001,"15min":2.9849999999999999},"min":{"1min":0.36699999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":4.0309999999999997,"5min":60.883000000000003,"15min":60.883000000000003},"last":0.48999999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411481,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306488,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93438664,"statfs":{"total":95995035648,"available":95681191936,"internally_reserved":0,"allocated":704512,"data_stored":363451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:17:49 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":1.9870000000000001,"15min":1.206},"min":{"1min":0.22600000000000001,"5min":0.21299999999999999,"15min":0.17199999999999999},"max":{"1min":2.9020000000000001,"5min":43.539999999999999,"15min":43.539999999999999},"last":0.32600000000000001},{"interface":"front","average":{"1min":1.093,"5min":2.101,"15min":1.258},"min":{"1min":0.27600000000000002,"5min":0.185,"15min":0.185},"max":{"1min":3.2970000000000002,"5min":41.962000000000003,"15min":41.962000000000003},"last":0.307}]},{"osd":1,"last update":"Wed Sep 11 08:17:49 2024","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.4630000000000001,"15min":1.012},"min":{"1min":0.28199999999999997,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":4.633,"5min":13.138999999999999,"15min":13.138999999999999},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.84999999999999998,"5min":1.532,"15min":1.04},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20200000000000001},"max":{"1min":3.3260000000000001,"5min":10.167999999999999,"15min":10.167999999999999},"last":0.35699999999999998}]},{"osd":3,"last update":"Wed Sep 11 08:17:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.7669999999999999,"15min":1.042},"min":{"1min":0.33700000000000002,"5min":0.21299999999999999,"15min":0.188},"max":{"1min":2.6339999999999999,"5min":8.1989999999999998,"15min":8.1989999999999998},"last":0.371},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.629,"15min":1.0089999999999999},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":3.0619999999999998,"5min":8.0640000000000001,"15min":8.3219999999999992},"last":0.34000000000000002}]},{"osd":4,"last update":"Wed Sep 11 08:17:29 2024","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":1.1339999999999999,"15min":0.84899999999999998},"min":{"1min":0.27900000000000003,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":3.0379999999999998,"5min":7.2480000000000002,"15min":7.806},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.83099999999999996,"5min":1.0129999999999999,"15min":0.88700000000000001},"min":{"1min":0.27500000000000002,"5min":0.252,"15min":0.252},"max":{"1min":2.1850000000000001,"5min":5.4740000000000002,"15min":12.129},"last":0.38500000000000001}]},{"osd":5,"last update":"Wed Sep 11 08:16:58 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.145,"15min":0.876},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":3.1139999999999999,"5min":5.4539999999999997,"15min":6.7709999999999999},"last":0.45900000000000002},{"interface":"front","average":{"1min":1.292,"5min":1.3089999999999999,"15min":0.89200000000000002},"min":{"1min":0.34200000000000003,"5min":0.30399999999999999,"15min":0.28399999999999997},"max":{"1min":4.2249999999999996,"5min":12.869,"15min":12.869},"last":0.39800000000000002}]},{"osd":6,"last update":"Wed Sep 11 08:17:16 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":1.05,"15min":0.83299999999999996},"min":{"1min":0.31,"5min":0.31,"15min":0.26100000000000001},"max":{"1min":2.1899999999999999,"5min":7.2249999999999996,"15min":7.2249999999999996},"last":0.437},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.1719999999999999,"15min":0.92700000000000005},"min":{"1min":0.35699999999999998,"5min":0.33200000000000002,"15min":0.32100000000000001},"max":{"1min":3.3490000000000002,"5min":5.4939999999999998,"15min":6.71},"last":0.42699999999999999}]},{"osd":7,"last update":"Wed Sep 11 08:17:29 2024","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.3440000000000001,"15min":1.3740000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":5.101,"5min":5.101,"15min":5.101},"last":0.48899999999999999},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.419,"15min":1.4970000000000001},"min":{"1min":0.40100000000000002,"5min":0.376,"15min":0.376},"max":{"1min":5.1310000000000002,"5min":5.3710000000000004,"15min":5.3710000000000004},"last":0.40699999999999997}]}]},{"osd":1,"up_from":13,"seq":55834575007,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306508,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93438644,"statfs":{"total":95995035648,"available":95681171456,"internally_reserved":0,"allocated":724992,"data_stored":364109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:17:18 2024","interfaces":[{"interface":"back","average":{"1min":1.843,"5min":1.3120000000000001,"15min":1.0109999999999999},"min":{"1min":0.19900000000000001,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":6.593,"5min":6.952,"15min":32.478000000000002},"last":0.30599999999999999},{"interface":"front","average":{"1min":2.6800000000000002,"5min":1.931,"15min":1.335},"min":{"1min":0.27700000000000002,"5min":0.186,"15min":0.186},"max":{"1min":15.351000000000001,"5min":15.351000000000001,"15min":35.459000000000003},"last":0.26300000000000001}]},{"osd":2,"last update":"Wed Sep 11 08:17:56 2024","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.1970000000000001,"15min":0.93200000000000005},"min":{"1min":0.221,"5min":0.221,"15min":0.21299999999999999},"max":{"1min":7.798,"5min":7.9699999999999998,"15min":8.3819999999999997},"last":0.246},{"interface":"front","average":{"1min":1.109,"5min":1.4119999999999999,"15min":1.091},"min":{"1min":0.248,"5min":0.192,"15min":0.16300000000000001},"max":{"1min":4.3220000000000001,"5min":5.8490000000000002,"15min":10.276},"last":0.28100000000000003}]},{"osd":3,"last update":"Wed Sep 11 08:17:01 2024","interfaces":[{"interface":"back","average":{"1min":2.4849999999999999,"5min":2.1909999999999998,"15min":1.4770000000000001},"min":{"1min":0.38400000000000001,"5min":0.248,"15min":0.246},"max":{"1min":8.4149999999999991,"5min":16.783999999999999,"15min":32.759},"last":0.33900000000000002},{"interface":"front","average":{"1min":1.833,"5min":1.841,"15min":1.268},"min":{"1min":0.30599999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":6.5940000000000003,"5min":14.973000000000001,"15min":32.444000000000003},"last":0.34899999999999998}]},{"osd":4,"last update":"Wed Sep 11 08:17:30 2024","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.2310000000000001,"15min":0.96699999999999997},"min":{"1min":0.26200000000000001,"5min":0.252,"15min":0.252},"max":{"1min":4.0970000000000004,"5min":5.8970000000000002,"15min":9.0139999999999993},"last":0.46400000000000002},{"interface":"front","average":{"1min":1.405,"5min":1.071,"15min":0.89700000000000002},"min":{"1min":0.29499999999999998,"5min":0.248,"15min":0.248},"max":{"1min":7.9560000000000004,"5min":7.9560000000000004,"15min":8.2360000000000007},"last":0.54300000000000004}]},{"osd":5,"last update":"Wed Sep 11 08:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":1.3440000000000001,"15min":1.044},"min":{"1min":0.32300000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":4.2919999999999998,"5min":5.9299999999999997,"15min":9.0470000000000006},"last":0.51400000000000001},{"interface":"front","average":{"1min":1.135,"5min":1.556,"15min":1.123},"min":{"1min":0.35999999999999999,"5min":0.35199999999999998,"15min":0.28100000000000003},"max":{"1min":5.3780000000000001,"5min":7.0330000000000004,"15min":11.967000000000001},"last":0.48199999999999998}]},{"osd":6,"last update":"Wed Sep 11 08:17:04 2024","interfaces":[{"interface":"back","average":{"1min":2.2440000000000002,"5min":2.7850000000000001,"15min":1.498},"min":{"1min":0.41199999999999998,"5min":0.307,"15min":0.29399999999999998},"max":{"1min":5.3479999999999999,"5min":124.31999999999999,"15min":124.31999999999999},"last":0.503},{"interface":"front","average":{"1min":1.522,"5min":2.5590000000000002,"15min":1.4019999999999999},"min":{"1min":0.35999999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":6.6680000000000001,"5min":124.958,"15min":124.958},"last":0.56100000000000005}]},{"osd":7,"last update":"Wed Sep 11 08:17:30 2024","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.454,"15min":1.274},"min":{"1min":0.34300000000000003,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":4.7649999999999997,"5min":5.9580000000000002,"15min":5.9580000000000002},"last":0.49199999999999999},{"interface":"front","average":{"1min":1.7390000000000001,"5min":1.6359999999999999,"15min":1.532},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":5.2969999999999997,"5min":5.2969999999999997,"15min":5.2969999999999997},"last":0.52900000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738531,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311268,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93433884,"statfs":{"total":95995035648,"available":95676297216,"internally_reserved":0,"allocated":1339392,"data_stored":955827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 11 08:17:22 2024","interfaces":[{"interface":"back","average":{"1min":1.49,"5min":1.4770000000000001,"15min":0.94199999999999995},"min":{"1min":0.221,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":10.105,"5min":10.105,"15min":10.105},"last":0.41599999999999998},{"interface":"front","average":{"1min":2.2519999999999998,"5min":2.0569999999999999,"15min":1.167},"min":{"1min":0.16600000000000001,"5min":0.153,"15min":0.153},"max":{"1min":19.852,"5min":19.852,"15min":19.852},"last":0.308}]},{"osd":2,"last update":"Wed Sep 11 08:17:47 2024","interfaces":[{"interface":"back","average":{"1min":1.855,"5min":1.6200000000000001,"15min":1.022},"min":{"1min":0.23699999999999999,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":10.025,"5min":10.351000000000001,"15min":10.351000000000001},"last":0.372},{"interface":"front","average":{"1min":1.873,"5min":1.6180000000000001,"15min":1.0840000000000001},"min":{"1min":0.29399999999999998,"5min":0.186,"15min":0.186},"max":{"1min":10.074999999999999,"5min":10.074999999999999,"15min":10.074999999999999},"last":0.44400000000000001}]},{"osd":3,"last update":"Wed Sep 11 08:17:07 2024","interfaces":[{"interface":"back","average":{"1min":1.9890000000000001,"5min":2.1509999999999998,"15min":1.3169999999999999},"min":{"1min":0.34799999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":10.129,"5min":35.491,"15min":35.491},"last":0.46100000000000002},{"interface":"front","average":{"1min":1.339,"5min":2.4399999999999999,"15min":1.3069999999999999},"min":{"1min":0.27100000000000002,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.282,"5min":37.563000000000002,"15min":37.563000000000002},"last":0.40300000000000002}]},{"osd":4,"last update":"Wed Sep 11 08:17:30 2024","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":1.2769999999999999,"15min":0.96299999999999997},"min":{"1min":0.313,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.929,"5min":6.0190000000000001,"15min":9.4030000000000005},"last":0.502},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.151,"15min":0.93999999999999995},"min":{"1min":0.40500000000000003,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":5.6399999999999997,"5min":6.6619999999999999,"15min":6.6619999999999999},"last":0.64200000000000002}]},{"osd":5,"last update":"Wed Sep 11 08:17:57 2024","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":1.5229999999999999,"15min":1.081},"min":{"1min":0.435,"5min":0.32700000000000001,"15min":0.255},"max":{"1min":5.6100000000000003,"5min":24.914999999999999,"15min":24.914999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.78700000000000003,"5min":1.5900000000000001,"15min":1.141},"min":{"1min":0.314,"5min":0.30399999999999999,"15min":0.27100000000000002},"max":{"1min":2.3119999999999998,"5min":25.277000000000001,"15min":25.277000000000001},"last":0.58499999999999996}]},{"osd":6,"last update":"Wed Sep 11 08:17:13 2024","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":2.3119999999999998,"15min":1.3260000000000001},"min":{"1min":0.29899999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":3.0609999999999999,"5min":78.522000000000006,"15min":78.522000000000006},"last":0.52000000000000002},{"interface":"front","average":{"1min":1.5760000000000001,"5min":2.089,"15min":1.2130000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":7.4820000000000002,"5min":79.808000000000007,"15min":79.808000000000007},"last":0.55500000000000005}]},{"osd":7,"last update":"Wed Sep 11 08:17:30 2024","interfaces":[{"interface":"back","average":{"1min":1.6439999999999999,"5min":1.5409999999999999,"15min":1.421},"min":{"1min":0.35499999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":7.5350000000000001,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.53500000000000003},{"interface":"front","average":{"1min":1.7230000000000001,"5min":1.4830000000000001,"15min":1.397},"min":{"1min":0.33700000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":7.5110000000000001,"5min":7.5110000000000001,"15min":7.5110000000000001},"last":0.51100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3425,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T08:18:00.957 INFO:tasks.thrashosds.thrasher:Setting 4.1a to [6, 3, 1, 2, 7, 5] 2024-09-11T08:18:00.957 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '4.1a', '6', '3', '1', '2', '7', '5'] 2024-09-11T08:18:00.957 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd pg-upmap-items 4.1a 6 3 1 2 7 5 2024-09-11T08:18:01.004 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:01.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:01.026 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:01.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:01.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:00 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/877772029' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-09-11T08:18:01.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:00 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/877772029' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-11T08:18:01.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:00 smithi181 ceph-mon[32701]: osdmap e246: 8 total, 8 up, 8 in 2024-09-11T08:18:01.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:00 smithi181 ceph-mon[32701]: from='client.25235 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:18:01.127 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:01.143 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:00 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/877772029' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-09-11T08:18:01.143 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:00 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/877772029' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-11T08:18:01.143 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:00 smithi150 ceph-mon[29276]: osdmap e246: 8 total, 8 up, 8 in 2024-09-11T08:18:01.143 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:00 smithi150 ceph-mon[29276]: from='client.25235 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:18:01.143 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:00 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/877772029' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-09-11T08:18:01.143 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:00 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/877772029' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-11T08:18:01.144 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:00 smithi150 ceph-mon[39612]: osdmap e246: 8 total, 8 up, 8 in 2024-09-11T08:18:01.144 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:00 smithi150 ceph-mon[39612]: from='client.25235 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:18:01.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:01.153 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:01.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:01.254 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:18:01.275 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:01.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:01.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:18:01.329 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-11T08:18:01.329 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-09-11T08:18:01.329 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-09-11T08:18:01.330 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-09-11T08:18:01.330 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.330 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14790.0:107799 7.0 7:0bf42be5:::benchmark_data_smithi150_78508_object107798:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-11T08:18:01.330 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:11:10.659500+0000", 2024-09-11T08:18:01.330 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 410.667643527, 2024-09-11T08:18:01.330 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.103845884, 2024-09-11T08:18:01.330 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:18:01.330 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:18:01.330 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:18:01.330 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14790", 2024-09-11T08:18:01.330 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:18:01.330 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 107799 2024-09-11T08:18:01.330 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.330 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:18:01.331 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.331 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:18:01.331 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.659500+0000", 2024-09-11T08:18:01.331 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:01.331 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.331 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.331 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:18:01.331 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.659500+0000", 2024-09-11T08:18:01.331 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:01.331 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.331 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.331 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:18:01.331 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.659499+0000", 2024-09-11T08:18:01.331 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:18:01.331 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.332 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.332 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:18:01.332 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.659547+0000", 2024-09-11T08:18:01.332 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7718e-05 2024-09-11T08:18:01.332 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.332 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.332 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:18:01.332 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.659559+0000", 2024-09-11T08:18:01.332 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2493999999999999e-05 2024-09-11T08:18:01.332 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.332 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.332 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:18:01.332 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.659566+0000", 2024-09-11T08:18:01.332 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2400000000000004e-06 2024-09-11T08:18:01.333 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.333 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.333 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:01.333 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.659607+0000", 2024-09-11T08:18:01.333 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1155999999999997e-05 2024-09-11T08:18:01.333 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.333 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.333 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:18:01.333 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.659695+0000", 2024-09-11T08:18:01.333 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8348000000000002e-05 2024-09-11T08:18:01.333 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.333 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.333 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-09-11T08:18:01.333 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.659793+0000", 2024-09-11T08:18:01.334 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.8227999999999995e-05 2024-09-11T08:18:01.334 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.334 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.334 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:18:01.334 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.662106+0000", 2024-09-11T08:18:01.334 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0023123620000000001 2024-09-11T08:18:01.334 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.334 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.334 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:01.334 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.668567+0000", 2024-09-11T08:18:01.334 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0064616379999999996 2024-09-11T08:18:01.334 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.334 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.334 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:01.334 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.763259+0000", 2024-09-11T08:18:01.334 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.094691701000000003 2024-09-11T08:18:01.335 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.335 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.335 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:18:01.335 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.763310+0000", 2024-09-11T08:18:01.335 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0872999999999999e-05 2024-09-11T08:18:01.335 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.335 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.335 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:18:01.335 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.763346+0000", 2024-09-11T08:18:01.335 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6442000000000001e-05 2024-09-11T08:18:01.335 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:01.335 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:18:01.335 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:01.336 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.336 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.336 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14790.0:107803 7.0 7:0f2f57ca:::benchmark_data_smithi150_78508_object107802:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-11T08:18:01.336 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:11:10.663869+0000", 2024-09-11T08:18:01.336 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 410.66327517100001, 2024-09-11T08:18:01.336 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10722504400000001, 2024-09-11T08:18:01.336 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:18:01.336 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:18:01.336 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:18:01.336 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14790", 2024-09-11T08:18:01.336 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:18:01.336 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 107803 2024-09-11T08:18:01.336 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.336 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:18:01.337 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.337 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:18:01.337 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.663869+0000", 2024-09-11T08:18:01.337 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:01.337 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.337 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.337 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:18:01.337 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.663869+0000", 2024-09-11T08:18:01.337 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:01.337 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.337 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.337 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:18:01.337 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.663861+0000", 2024-09-11T08:18:01.337 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999928 2024-09-11T08:18:01.338 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.338 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.338 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:18:01.338 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.663906+0000", 2024-09-11T08:18:01.338 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4628e-05 2024-09-11T08:18:01.338 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.338 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.338 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:18:01.338 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.663921+0000", 2024-09-11T08:18:01.338 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4742e-05 2024-09-11T08:18:01.338 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.338 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.338 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:18:01.338 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.663927+0000", 2024-09-11T08:18:01.338 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1850000000000001e-06 2024-09-11T08:18:01.339 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.339 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.339 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:01.339 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.663968+0000", 2024-09-11T08:18:01.339 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0895000000000001e-05 2024-09-11T08:18:01.339 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.339 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.339 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:18:01.339 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.664082+0000", 2024-09-11T08:18:01.339 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000113987 2024-09-11T08:18:01.339 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.339 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.339 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-09-11T08:18:01.339 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.664254+0000", 2024-09-11T08:18:01.339 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017227599999999999 2024-09-11T08:18:01.340 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.340 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.340 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:18:01.340 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.668526+0000", 2024-09-11T08:18:01.340 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0042720309999999999 2024-09-11T08:18:01.340 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.340 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.340 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:01.340 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.668696+0000", 2024-09-11T08:18:01.340 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017003299999999999 2024-09-11T08:18:01.340 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.340 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.340 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:01.340 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.771006+0000", 2024-09-11T08:18:01.341 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102309338 2024-09-11T08:18:01.341 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.341 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.341 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:18:01.341 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.771054+0000", 2024-09-11T08:18:01.341 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8668999999999998e-05 2024-09-11T08:18:01.341 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.341 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.341 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:18:01.341 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.771094+0000", 2024-09-11T08:18:01.341 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9622000000000001e-05 2024-09-11T08:18:01.341 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:01.341 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:18:01.342 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:01.342 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.342 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.342 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14790.0:107805 7.0 7:08b39077:::benchmark_data_smithi150_78508_object107804:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-11T08:18:01.342 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:11:10.664517+0000", 2024-09-11T08:18:01.342 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 410.662627439, 2024-09-11T08:18:01.342 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.106708476, 2024-09-11T08:18:01.342 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:18:01.342 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:18:01.342 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:18:01.342 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14790", 2024-09-11T08:18:01.342 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:18:01.342 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 107805 2024-09-11T08:18:01.343 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.343 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:18:01.343 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.343 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:18:01.343 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.664517+0000", 2024-09-11T08:18:01.343 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:01.343 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.343 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.343 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:18:01.343 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.664517+0000", 2024-09-11T08:18:01.343 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:01.343 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.343 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.344 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:18:01.344 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.664515+0000", 2024-09-11T08:18:01.344 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:18:01.344 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.344 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.344 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:18:01.344 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.664547+0000", 2024-09-11T08:18:01.344 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1572000000000002e-05 2024-09-11T08:18:01.344 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.344 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.344 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:18:01.344 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.664563+0000", 2024-09-11T08:18:01.344 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5824e-05 2024-09-11T08:18:01.345 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.345 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.345 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:18:01.345 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.664569+0000", 2024-09-11T08:18:01.345 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0569999999999998e-06 2024-09-11T08:18:01.345 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.345 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.345 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:01.345 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.666494+0000", 2024-09-11T08:18:01.345 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0019253860000000001 2024-09-11T08:18:01.345 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.345 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.346 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:18:01.346 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.666625+0000", 2024-09-11T08:18:01.346 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000130961 2024-09-11T08:18:01.346 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.346 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.346 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-09-11T08:18:01.346 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.666778+0000", 2024-09-11T08:18:01.346 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000153036 2024-09-11T08:18:01.346 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.346 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.346 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:18:01.346 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.668624+0000", 2024-09-11T08:18:01.346 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001845265 2024-09-11T08:18:01.346 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.347 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.347 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:01.347 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.668825+0000", 2024-09-11T08:18:01.347 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000201375 2024-09-11T08:18:01.347 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.347 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.347 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:01.347 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.771142+0000", 2024-09-11T08:18:01.347 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10231694099999999 2024-09-11T08:18:01.347 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.347 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.347 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:18:01.347 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.771188+0000", 2024-09-11T08:18:01.348 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6581e-05 2024-09-11T08:18:01.348 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.348 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.348 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:18:01.348 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.771225+0000", 2024-09-11T08:18:01.348 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6557999999999998e-05 2024-09-11T08:18:01.348 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:01.348 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:18:01.348 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:01.348 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.348 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.348 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14790.0:107988 7.0 7:0f2b6e5d:::benchmark_data_smithi150_78508_object107987:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-11T08:18:01.348 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:11:10.917620+0000", 2024-09-11T08:18:01.348 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 410.40952360699998, 2024-09-11T08:18:01.349 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.11916180699999999, 2024-09-11T08:18:01.349 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:18:01.349 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:18:01.349 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:18:01.349 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14790", 2024-09-11T08:18:01.349 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:18:01.349 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 107988 2024-09-11T08:18:01.349 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.349 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:18:01.349 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.349 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:18:01.349 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.917620+0000", 2024-09-11T08:18:01.349 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:01.349 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.350 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.350 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:18:01.350 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.917620+0000", 2024-09-11T08:18:01.350 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:01.350 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.350 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.350 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:18:01.350 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.917619+0000", 2024-09-11T08:18:01.350 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:18:01.350 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.350 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.350 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:18:01.350 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.917666+0000", 2024-09-11T08:18:01.350 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7317000000000001e-05 2024-09-11T08:18:01.351 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.351 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.351 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:18:01.351 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.917675+0000", 2024-09-11T08:18:01.351 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.551e-06 2024-09-11T08:18:01.351 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.351 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.351 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:18:01.351 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.917679+0000", 2024-09-11T08:18:01.351 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0110000000000002e-06 2024-09-11T08:18:01.351 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.351 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.351 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:01.352 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.917727+0000", 2024-09-11T08:18:01.352 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8718999999999999e-05 2024-09-11T08:18:01.352 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.352 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.352 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:18:01.352 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.917864+0000", 2024-09-11T08:18:01.352 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000136728 2024-09-11T08:18:01.352 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.352 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.352 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-09-11T08:18:01.352 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.917998+0000", 2024-09-11T08:18:01.352 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013343599999999999 2024-09-11T08:18:01.352 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.353 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.353 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:18:01.353 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.920823+0000", 2024-09-11T08:18:01.353 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0028258229999999999 2024-09-11T08:18:01.353 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.353 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.353 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:01.353 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.926218+0000", 2024-09-11T08:18:01.353 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0053950400000000003 2024-09-11T08:18:01.353 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.353 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.353 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:01.353 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:11.036697+0000", 2024-09-11T08:18:01.354 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.110478594 2024-09-11T08:18:01.354 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.354 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.354 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:18:01.354 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:11.036745+0000", 2024-09-11T08:18:01.354 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7484000000000001e-05 2024-09-11T08:18:01.354 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.354 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.354 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:18:01.354 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:11.036782+0000", 2024-09-11T08:18:01.354 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7648000000000003e-05 2024-09-11T08:18:01.354 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:01.354 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:18:01.354 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:01.355 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.355 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.355 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14790.0:107994 7.0 7:0ddff39c:::benchmark_data_smithi150_78508_object107993:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-11T08:18:01.355 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:11:10.923306+0000", 2024-09-11T08:18:01.355 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 410.40383754499999, 2024-09-11T08:18:01.355 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.11360466499999999, 2024-09-11T08:18:01.355 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:18:01.355 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:18:01.355 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:18:01.355 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14790", 2024-09-11T08:18:01.355 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:18:01.355 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 107994 2024-09-11T08:18:01.355 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.355 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:18:01.356 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.356 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:18:01.356 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.923306+0000", 2024-09-11T08:18:01.356 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:01.356 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.356 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.356 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:18:01.356 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.923306+0000", 2024-09-11T08:18:01.356 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:01.356 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.356 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.356 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:18:01.356 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.923304+0000", 2024-09-11T08:18:01.356 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999981 2024-09-11T08:18:01.357 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.357 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.358 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:18:01.358 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.923361+0000", 2024-09-11T08:18:01.358 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7018000000000002e-05 2024-09-11T08:18:01.358 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.358 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.358 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:18:01.358 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.923374+0000", 2024-09-11T08:18:01.358 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2734999999999999e-05 2024-09-11T08:18:01.358 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.358 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.358 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:18:01.358 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.923380+0000", 2024-09-11T08:18:01.358 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9150000000000001e-06 2024-09-11T08:18:01.358 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.359 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.359 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:01.359 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.923418+0000", 2024-09-11T08:18:01.359 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8067e-05 2024-09-11T08:18:01.359 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.359 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.359 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:18:01.359 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.923511+0000", 2024-09-11T08:18:01.359 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.3159999999999996e-05 2024-09-11T08:18:01.359 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.359 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.359 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-09-11T08:18:01.359 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.923600+0000", 2024-09-11T08:18:01.359 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8746999999999996e-05 2024-09-11T08:18:01.359 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.360 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.360 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:18:01.360 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.926348+0000", 2024-09-11T08:18:01.360 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0027479599999999998 2024-09-11T08:18:01.360 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.360 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.360 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:01.360 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:10.926634+0000", 2024-09-11T08:18:01.360 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00028572 2024-09-11T08:18:01.360 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.360 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.360 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:01.360 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:11.036824+0000", 2024-09-11T08:18:01.360 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.110190756 2024-09-11T08:18:01.361 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.361 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.361 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:18:01.361 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:11.036877+0000", 2024-09-11T08:18:01.361 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2358000000000003e-05 2024-09-11T08:18:01.361 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.361 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.361 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:18:01.361 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:11.036911+0000", 2024-09-11T08:18:01.361 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4359000000000001e-05 2024-09-11T08:18:01.361 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:01.361 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:18:01.361 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:01.361 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.364 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.364 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14790.0:108662 7.0 7:02eac7cd:::benchmark_data_smithi150_78508_object108661:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-11T08:18:01.364 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:11:11.631631+0000", 2024-09-11T08:18:01.364 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 409.69551282800001, 2024-09-11T08:18:01.364 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.116982267, 2024-09-11T08:18:01.364 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:18:01.365 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:18:01.365 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:18:01.365 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14790", 2024-09-11T08:18:01.365 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:18:01.365 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 108662 2024-09-11T08:18:01.365 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.365 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:18:01.365 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.365 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:18:01.365 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:11.631631+0000", 2024-09-11T08:18:01.365 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:01.365 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.366 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.366 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:18:01.366 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:11.631631+0000", 2024-09-11T08:18:01.366 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:01.366 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.366 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.366 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:18:01.366 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:11.631630+0000", 2024-09-11T08:18:01.366 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:18:01.366 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.366 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.366 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:18:01.366 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:11.631687+0000", 2024-09-11T08:18:01.366 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.77e-05 2024-09-11T08:18:01.366 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.367 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.367 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:18:01.367 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:11.631700+0000", 2024-09-11T08:18:01.367 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2384e-05 2024-09-11T08:18:01.367 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.367 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.367 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:18:01.367 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:11.631705+0000", 2024-09-11T08:18:01.367 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7409999999999998e-06 2024-09-11T08:18:01.367 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.367 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.367 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:01.367 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:11.631746+0000", 2024-09-11T08:18:01.367 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0040999999999998e-05 2024-09-11T08:18:01.368 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.368 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.368 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:18:01.368 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:11.631828+0000", 2024-09-11T08:18:01.368 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2344999999999998e-05 2024-09-11T08:18:01.368 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.368 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.368 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-09-11T08:18:01.368 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:11.631913+0000", 2024-09-11T08:18:01.368 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5284000000000005e-05 2024-09-11T08:18:01.368 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.368 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.368 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:18:01.368 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:11.634123+0000", 2024-09-11T08:18:01.369 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0022095970000000002 2024-09-11T08:18:01.369 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.369 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.369 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:01.369 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:11.635131+0000", 2024-09-11T08:18:01.369 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001008342 2024-09-11T08:18:01.369 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.369 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.369 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:01.369 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:11.748527+0000", 2024-09-11T08:18:01.369 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.113395396 2024-09-11T08:18:01.369 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.369 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.369 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:18:01.370 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:11.748562+0000", 2024-09-11T08:18:01.370 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5771e-05 2024-09-11T08:18:01.370 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.370 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.370 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:18:01.370 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:11.748613+0000", 2024-09-11T08:18:01.370 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1121000000000001e-05 2024-09-11T08:18:01.370 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:01.370 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:18:01.370 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:01.370 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.370 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.370 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14790.0:142831 7.0 7:0fc1e507:::benchmark_data_smithi150_78508_object18390:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-11T08:18:01.370 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:11:45.464717+0000", 2024-09-11T08:18:01.371 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 375.86242688700003, 2024-09-11T08:18:01.371 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10371095299999999, 2024-09-11T08:18:01.371 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:18:01.371 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:18:01.371 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:18:01.371 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14790", 2024-09-11T08:18:01.371 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:18:01.371 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 142831 2024-09-11T08:18:01.371 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.371 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:18:01.371 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.371 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:18:01.371 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.464717+0000", 2024-09-11T08:18:01.371 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:01.371 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.371 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.372 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:18:01.372 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.464717+0000", 2024-09-11T08:18:01.372 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:01.372 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.372 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.372 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:18:01.372 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.464716+0000", 2024-09-11T08:18:01.372 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:18:01.372 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.372 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.372 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:18:01.372 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.464723+0000", 2024-09-11T08:18:01.372 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1040000000000001e-06 2024-09-11T08:18:01.373 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.373 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.373 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:18:01.373 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.464737+0000", 2024-09-11T08:18:01.373 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4620999999999999e-05 2024-09-11T08:18:01.373 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.373 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.373 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:18:01.373 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.464744+0000", 2024-09-11T08:18:01.373 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6089999999999997e-06 2024-09-11T08:18:01.373 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.373 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.373 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:01.374 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.464797+0000", 2024-09-11T08:18:01.374 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3578000000000001e-05 2024-09-11T08:18:01.374 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.374 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.374 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:18:01.374 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.464920+0000", 2024-09-11T08:18:01.374 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012298699999999999 2024-09-11T08:18:01.374 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.374 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.374 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-09-11T08:18:01.374 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.465038+0000", 2024-09-11T08:18:01.374 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011740599999999999 2024-09-11T08:18:01.374 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.374 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.375 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:18:01.375 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.467855+0000", 2024-09-11T08:18:01.375 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0028174469999999998 2024-09-11T08:18:01.375 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.375 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.375 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:01.375 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.561355+0000", 2024-09-11T08:18:01.375 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.093499950999999998 2024-09-11T08:18:01.375 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.375 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.375 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:01.375 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.568370+0000", 2024-09-11T08:18:01.375 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0070143810000000001 2024-09-11T08:18:01.375 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.376 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.376 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:18:01.376 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.568402+0000", 2024-09-11T08:18:01.376 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2702e-05 2024-09-11T08:18:01.376 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.376 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.376 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:18:01.376 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:11:45.568428+0000", 2024-09-11T08:18:01.376 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5681999999999999e-05 2024-09-11T08:18:01.376 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:01.376 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:18:01.376 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:01.376 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.376 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.376 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14790.0:239245 7.0 7:0a6d89ff:::benchmark_data_smithi150_78508_object114804:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-11T08:18:01.377 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:13:19.497337+0000", 2024-09-11T08:18:01.377 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 281.82980655900002, 2024-09-11T08:18:01.377 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.108864397, 2024-09-11T08:18:01.377 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:18:01.377 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:18:01.377 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:18:01.377 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14790", 2024-09-11T08:18:01.377 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:18:01.377 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 239245 2024-09-11T08:18:01.377 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.377 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:18:01.377 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.377 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:18:01.377 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.497337+0000", 2024-09-11T08:18:01.378 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:01.378 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.378 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.378 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:18:01.378 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.497337+0000", 2024-09-11T08:18:01.378 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:01.378 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.378 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.378 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:18:01.378 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.497336+0000", 2024-09-11T08:18:01.378 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:18:01.378 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.378 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.378 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:18:01.378 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.497342+0000", 2024-09-11T08:18:01.379 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0619999999999997e-06 2024-09-11T08:18:01.379 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.379 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.379 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:18:01.379 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.497352+0000", 2024-09-11T08:18:01.379 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0016e-05 2024-09-11T08:18:01.379 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.379 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.379 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:18:01.379 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.497357+0000", 2024-09-11T08:18:01.379 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0799999999999996e-06 2024-09-11T08:18:01.379 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.379 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.379 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:01.379 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.497404+0000", 2024-09-11T08:18:01.380 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6993999999999998e-05 2024-09-11T08:18:01.380 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.380 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.380 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:18:01.380 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.497521+0000", 2024-09-11T08:18:01.380 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000117176 2024-09-11T08:18:01.380 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.380 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.380 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-09-11T08:18:01.380 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.497630+0000", 2024-09-11T08:18:01.380 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010885100000000001 2024-09-11T08:18:01.380 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.380 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.380 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:18:01.381 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.605944+0000", 2024-09-11T08:18:01.381 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.108313273 2024-09-11T08:18:01.381 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.381 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.381 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:01.381 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.606001+0000", 2024-09-11T08:18:01.381 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7309000000000003e-05 2024-09-11T08:18:01.381 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.381 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.381 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:01.381 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.606098+0000", 2024-09-11T08:18:01.381 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7348000000000004e-05 2024-09-11T08:18:01.381 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.382 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.382 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:18:01.382 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.606150+0000", 2024-09-11T08:18:01.382 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1984999999999998e-05 2024-09-11T08:18:01.382 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.382 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.382 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:18:01.382 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.606202+0000", 2024-09-11T08:18:01.382 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1643e-05 2024-09-11T08:18:01.382 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:01.382 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:18:01.382 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:01.382 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.382 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.383 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14790.0:239246 7.0 7:0f6221ed:::benchmark_data_smithi150_78508_object114805:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-11T08:18:01.383 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:13:19.497547+0000", 2024-09-11T08:18:01.383 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 281.82959673900001, 2024-09-11T08:18:01.383 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.11184465, 2024-09-11T08:18:01.383 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:18:01.383 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:18:01.383 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:18:01.383 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14790", 2024-09-11T08:18:01.383 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:18:01.383 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 239246 2024-09-11T08:18:01.383 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.383 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:18:01.383 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.383 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:18:01.384 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.497547+0000", 2024-09-11T08:18:01.384 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:01.384 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.384 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.384 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:18:01.384 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.497547+0000", 2024-09-11T08:18:01.384 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:01.384 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.384 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.384 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:18:01.384 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.497546+0000", 2024-09-11T08:18:01.384 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:18:01.384 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.384 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.384 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:18:01.385 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.497555+0000", 2024-09-11T08:18:01.385 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2660000000000001e-06 2024-09-11T08:18:01.385 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.385 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.385 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:18:01.385 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.497565+0000", 2024-09-11T08:18:01.385 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9599999999999995e-06 2024-09-11T08:18:01.385 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.385 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.385 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:18:01.385 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.497570+0000", 2024-09-11T08:18:01.385 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.553e-06 2024-09-11T08:18:01.385 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.385 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.386 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:01.386 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.500589+0000", 2024-09-11T08:18:01.386 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00301856 2024-09-11T08:18:01.386 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.386 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.386 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:18:01.386 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.500703+0000", 2024-09-11T08:18:01.386 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011435 2024-09-11T08:18:01.386 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.386 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.386 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-09-11T08:18:01.386 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.500817+0000", 2024-09-11T08:18:01.386 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000114141 2024-09-11T08:18:01.386 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.386 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.386 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:01.387 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.606052+0000", 2024-09-11T08:18:01.387 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.105234249 2024-09-11T08:18:01.387 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.387 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.387 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:01.387 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.606261+0000", 2024-09-11T08:18:01.387 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00020967800000000001 2024-09-11T08:18:01.387 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.387 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.387 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:18:01.387 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.609311+0000", 2024-09-11T08:18:01.387 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0030492879999999998 2024-09-11T08:18:01.387 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.387 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.387 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:18:01.388 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.609354+0000", 2024-09-11T08:18:01.388 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3831000000000001e-05 2024-09-11T08:18:01.388 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.388 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.388 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:18:01.388 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.609392+0000", 2024-09-11T08:18:01.388 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7533999999999997e-05 2024-09-11T08:18:01.388 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:01.388 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:18:01.388 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:01.388 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.388 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.388 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.14790.0:239241 7.b e148/91 7:d58052e7:::benchmark_data_smithi150_78508_object114800:head v 148'18333, mlcod=148'18331)", 2024-09-11T08:18:01.388 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:13:19.498600+0000", 2024-09-11T08:18:01.389 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 281.828543798, 2024-09-11T08:18:01.389 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.104656385, 2024-09-11T08:18:01.389 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:18:01.389 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:18:01.389 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:18:01.389 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.389 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:18:01.389 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.498600+0000", 2024-09-11T08:18:01.389 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:01.389 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.389 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.389 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:18:01.389 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.498600+0000", 2024-09-11T08:18:01.389 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:01.389 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.390 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.390 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:18:01.390 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.498599+0000", 2024-09-11T08:18:01.390 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-09-11T08:18:01.390 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.390 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.390 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:18:01.390 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.498606+0000", 2024-09-11T08:18:01.390 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5810000000000002e-06 2024-09-11T08:18:01.390 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.390 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.390 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:18:01.390 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.498613+0000", 2024-09-11T08:18:01.390 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.379e-06 2024-09-11T08:18:01.390 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.391 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.391 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:18:01.391 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.498617+0000", 2024-09-11T08:18:01.391 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7680000000000002e-06 2024-09-11T08:18:01.391 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.391 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.391 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:01.391 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.498654+0000", 2024-09-11T08:18:01.391 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6826999999999998e-05 2024-09-11T08:18:01.391 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.391 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.391 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:18:01.391 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.498671+0000", 2024-09-11T08:18:01.391 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7323e-05 2024-09-11T08:18:01.391 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.392 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.392 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:18:01.392 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.500947+0000", 2024-09-11T08:18:01.392 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0022751569999999999 2024-09-11T08:18:01.392 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.392 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.392 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:18:01.392 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.603257+0000", 2024-09-11T08:18:01.392 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102310021 2024-09-11T08:18:01.392 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:01.392 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:18:01.392 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:01.392 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.392 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.393 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop_reply(client.14790.0:239245 7.0 e148/91 ondisk, result = 0)", 2024-09-11T08:18:01.393 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:13:19.500883+0000", 2024-09-11T08:18:01.393 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 281.82626077600003, 2024-09-11T08:18:01.393 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.105122015, 2024-09-11T08:18:01.393 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:18:01.393 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-11T08:18:01.393 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:18:01.393 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.393 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:18:01.393 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.500883+0000", 2024-09-11T08:18:01.393 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:01.393 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.393 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.394 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:18:01.394 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.500883+0000", 2024-09-11T08:18:01.394 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:01.394 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.394 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.394 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:18:01.394 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.500882+0000", 2024-09-11T08:18:01.394 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:18:01.394 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.394 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.394 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:18:01.394 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.500887+0000", 2024-09-11T08:18:01.394 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.203e-06 2024-09-11T08:18:01.394 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.394 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.394 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:18:01.395 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.603088+0000", 2024-09-11T08:18:01.395 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10220096300000001 2024-09-11T08:18:01.395 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.395 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.395 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:18:01.395 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.603103+0000", 2024-09-11T08:18:01.395 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5167000000000001e-05 2024-09-11T08:18:01.395 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.395 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.395 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:01.395 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.605982+0000", 2024-09-11T08:18:01.395 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0028788809999999998 2024-09-11T08:18:01.395 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.396 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.396 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:18:01.396 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.605994+0000", 2024-09-11T08:18:01.396 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1304000000000001e-05 2024-09-11T08:18:01.396 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.396 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.396 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:18:01.396 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.606005+0000", 2024-09-11T08:18:01.396 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1513e-05 2024-09-11T08:18:01.396 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:01.396 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:18:01.396 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:01.396 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.396 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.397 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14790.0:239249 7.0 7:073a85af:::benchmark_data_smithi150_78508_object114808:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-11T08:18:01.397 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:13:19.502795+0000", 2024-09-11T08:18:01.397 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 281.82434877899999, 2024-09-11T08:18:01.397 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.12497079, 2024-09-11T08:18:01.397 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:18:01.397 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:18:01.397 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:18:01.397 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14790", 2024-09-11T08:18:01.397 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:18:01.397 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 239249 2024-09-11T08:18:01.397 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.397 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:18:01.397 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.397 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:18:01.397 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.502795+0000", 2024-09-11T08:18:01.398 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:01.398 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.398 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.398 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:18:01.398 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.502795+0000", 2024-09-11T08:18:01.398 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:01.398 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.398 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.398 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:18:01.398 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.502794+0000", 2024-09-11T08:18:01.398 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:18:01.398 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.398 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.398 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:18:01.399 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.502800+0000", 2024-09-11T08:18:01.399 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7729999999999996e-06 2024-09-11T08:18:01.399 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.399 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.399 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:18:01.399 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.603215+0000", 2024-09-11T08:18:01.399 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100414794 2024-09-11T08:18:01.399 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.399 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.399 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:18:01.399 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.603230+0000", 2024-09-11T08:18:01.399 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5133999999999999e-05 2024-09-11T08:18:01.399 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.399 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.399 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:01.399 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.606296+0000", 2024-09-11T08:18:01.400 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0030661949999999999 2024-09-11T08:18:01.400 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.400 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.400 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:18:01.400 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.606431+0000", 2024-09-11T08:18:01.400 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013462200000000001 2024-09-11T08:18:01.400 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.400 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.400 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-09-11T08:18:01.400 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.606580+0000", 2024-09-11T08:18:01.400 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014936999999999999 2024-09-11T08:18:01.400 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.400 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.400 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:01.401 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.616287+0000", 2024-09-11T08:18:01.401 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0097068099999999997 2024-09-11T08:18:01.401 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.401 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.401 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:18:01.401 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.621774+0000", 2024-09-11T08:18:01.401 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0054871659999999999 2024-09-11T08:18:01.401 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.401 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.401 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:01.401 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.627679+0000", 2024-09-11T08:18:01.401 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.005904683 2024-09-11T08:18:01.401 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.401 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.402 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:18:01.402 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.627727+0000", 2024-09-11T08:18:01.402 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8026999999999999e-05 2024-09-11T08:18:01.402 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.402 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.402 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:18:01.402 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:13:19.627766+0000", 2024-09-11T08:18:01.402 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9255999999999998e-05 2024-09-11T08:18:01.402 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:01.402 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:18:01.402 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:01.402 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.402 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.402 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15087.0:146932 8.d 8:b808578b:::benchmark_data_smithi150_107646_object18100:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206)", 2024-09-11T08:18:01.402 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:15:55.523397+0000", 2024-09-11T08:18:01.403 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 125.80374729899999, 2024-09-11T08:18:01.403 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0733922520000001, 2024-09-11T08:18:01.403 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:18:01.403 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:18:01.403 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:18:01.403 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15087", 2024-09-11T08:18:01.403 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/3299560913", 2024-09-11T08:18:01.403 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 146932 2024-09-11T08:18:01.403 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.403 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:18:01.403 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.403 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:18:01.403 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.523397+0000", 2024-09-11T08:18:01.404 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:01.404 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.404 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.404 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:18:01.404 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.523397+0000", 2024-09-11T08:18:01.404 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:01.404 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.404 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.404 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:18:01.404 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.523395+0000", 2024-09-11T08:18:01.404 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:18:01.404 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.404 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.405 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:18:01.405 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.523401+0000", 2024-09-11T08:18:01.405 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6099999999999997e-06 2024-09-11T08:18:01.405 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.405 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.405 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:18:01.405 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.525812+0000", 2024-09-11T08:18:01.405 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0024110960000000002 2024-09-11T08:18:01.405 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.405 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.405 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:18:01.405 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.525823+0000", 2024-09-11T08:18:01.405 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1066999999999999e-05 2024-09-11T08:18:01.406 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.406 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.406 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:01.406 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.531490+0000", 2024-09-11T08:18:01.406 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.005666684 2024-09-11T08:18:01.406 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.406 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.406 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:18:01.406 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.531500+0000", 2024-09-11T08:18:01.406 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0397000000000001e-05 2024-09-11T08:18:01.406 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.406 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.406 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:01.406 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:56.515950+0000", 2024-09-11T08:18:01.406 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.98444924 2024-09-11T08:18:01.407 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.407 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.407 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:18:01.407 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:56.515961+0000", 2024-09-11T08:18:01.407 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1625e-05 2024-09-11T08:18:01.407 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.407 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.407 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:01.407 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.591039+0000", 2024-09-11T08:18:01.407 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.075078188 2024-09-11T08:18:01.407 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.407 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.407 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:18:01.407 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.591134+0000", 2024-09-11T08:18:01.408 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4813000000000002e-05 2024-09-11T08:18:01.408 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.408 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.408 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2,6,7", 2024-09-11T08:18:01.408 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.591210+0000", 2024-09-11T08:18:01.408 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5374e-05 2024-09-11T08:18:01.408 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.408 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.408 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:01.408 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.594054+0000", 2024-09-11T08:18:01.408 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0028446740000000002 2024-09-11T08:18:01.408 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.408 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.408 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:01.409 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.594089+0000", 2024-09-11T08:18:01.409 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5151000000000003e-05 2024-09-11T08:18:01.409 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.409 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.409 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:18:01.409 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.594094+0000", 2024-09-11T08:18:01.409 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9830000000000004e-06 2024-09-11T08:18:01.409 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.409 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.409 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:01.409 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.596667+0000", 2024-09-11T08:18:01.409 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0025724390000000001 2024-09-11T08:18:01.409 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.410 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.410 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:18:01.410 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.596712+0000", 2024-09-11T08:18:01.410 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5667000000000002e-05 2024-09-11T08:18:01.410 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.410 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.410 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:18:01.410 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.596789+0000", 2024-09-11T08:18:01.410 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6490999999999997e-05 2024-09-11T08:18:01.410 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:01.410 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:18:01.410 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:01.410 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.410 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.411 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15087.0:146933 8.d 8:bbaddf40:::benchmark_data_smithi150_107646_object18101:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206)", 2024-09-11T08:18:01.411 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:15:55.525845+0000", 2024-09-11T08:18:01.411 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 125.80129941, 2024-09-11T08:18:01.411 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0767671750000001, 2024-09-11T08:18:01.411 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:18:01.411 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:18:01.411 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:18:01.411 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15087", 2024-09-11T08:18:01.411 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/3299560913", 2024-09-11T08:18:01.411 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 146933 2024-09-11T08:18:01.411 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.411 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:18:01.411 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.411 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:18:01.411 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.525845+0000", 2024-09-11T08:18:01.411 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:01.412 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.412 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.412 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:18:01.412 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.525845+0000", 2024-09-11T08:18:01.412 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:01.412 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.412 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.412 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:18:01.412 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.525843+0000", 2024-09-11T08:18:01.412 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:18:01.412 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.412 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.412 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:18:01.412 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.525850+0000", 2024-09-11T08:18:01.412 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7399999999999998e-06 2024-09-11T08:18:01.413 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.413 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.413 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:18:01.413 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.525859+0000", 2024-09-11T08:18:01.413 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.9539999999999993e-06 2024-09-11T08:18:01.413 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.413 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.413 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:18:01.413 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.525864+0000", 2024-09-11T08:18:01.413 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6759999999999998e-06 2024-09-11T08:18:01.413 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.413 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.414 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:01.414 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.531522+0000", 2024-09-11T08:18:01.414 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0056579910000000002 2024-09-11T08:18:01.414 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.414 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.414 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:18:01.414 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.531527+0000", 2024-09-11T08:18:01.414 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8769999999999998e-06 2024-09-11T08:18:01.414 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.414 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.414 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:01.414 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:56.516026+0000", 2024-09-11T08:18:01.414 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.98449826699999998 2024-09-11T08:18:01.414 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.415 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.415 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:18:01.415 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:56.516033+0000", 2024-09-11T08:18:01.415 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6480000000000003e-06 2024-09-11T08:18:01.415 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.415 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.415 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:01.415 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.594125+0000", 2024-09-11T08:18:01.415 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0780913569999999 2024-09-11T08:18:01.415 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.415 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.415 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:18:01.415 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.594242+0000", 2024-09-11T08:18:01.415 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000117521 2024-09-11T08:18:01.416 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.416 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.416 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2,6,7", 2024-09-11T08:18:01.416 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.594340+0000", 2024-09-11T08:18:01.416 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7344999999999997e-05 2024-09-11T08:18:01.416 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.416 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.416 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:01.416 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.596848+0000", 2024-09-11T08:18:01.416 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0025083340000000001 2024-09-11T08:18:01.416 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.416 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.416 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:18:01.417 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.599713+0000", 2024-09-11T08:18:01.417 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0028655120000000002 2024-09-11T08:18:01.417 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.417 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.417 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:01.417 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.602373+0000", 2024-09-11T08:18:01.417 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0026598630000000002 2024-09-11T08:18:01.417 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.417 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.417 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:01.417 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.602441+0000", 2024-09-11T08:18:01.417 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7607999999999994e-05 2024-09-11T08:18:01.417 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.417 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.418 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:18:01.418 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.602499+0000", 2024-09-11T08:18:01.418 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8362000000000001e-05 2024-09-11T08:18:01.418 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.418 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.418 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:18:01.418 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.602612+0000", 2024-09-11T08:18:01.418 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011246199999999999 2024-09-11T08:18:01.418 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:01.418 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:18:01.418 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:01.418 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.418 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.418 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15087.0:146943 8.d 8:bb9cf1f9:::benchmark_data_smithi150_107646_object18111:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-11T08:18:01.419 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:15:55.551894+0000", 2024-09-11T08:18:01.419 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 125.775249832, 2024-09-11T08:18:01.419 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0510072859999999, 2024-09-11T08:18:01.419 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:18:01.419 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:18:01.419 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:18:01.419 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15087", 2024-09-11T08:18:01.419 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/3299560913", 2024-09-11T08:18:01.419 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 146943 2024-09-11T08:18:01.419 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.419 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:18:01.419 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.419 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:18:01.419 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.551894+0000", 2024-09-11T08:18:01.420 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:01.420 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.420 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.420 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:18:01.420 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.551894+0000", 2024-09-11T08:18:01.420 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:01.420 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.420 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.420 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:18:01.420 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.551893+0000", 2024-09-11T08:18:01.420 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:18:01.420 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.420 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.420 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:18:01.421 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.551899+0000", 2024-09-11T08:18:01.421 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6300000000000003e-06 2024-09-11T08:18:01.421 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.421 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.421 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:18:01.421 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.551909+0000", 2024-09-11T08:18:01.421 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0361e-05 2024-09-11T08:18:01.421 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.421 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.421 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:18:01.421 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.551914+0000", 2024-09-11T08:18:01.421 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8550000000000001e-06 2024-09-11T08:18:01.421 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.421 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.422 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:01.422 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.551956+0000", 2024-09-11T08:18:01.422 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1820000000000003e-05 2024-09-11T08:18:01.422 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.422 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.422 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:18:01.422 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.551963+0000", 2024-09-11T08:18:01.422 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9140000000000002e-06 2024-09-11T08:18:01.422 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.422 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.422 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:01.422 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:56.516053+0000", 2024-09-11T08:18:01.422 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.964090471 2024-09-11T08:18:01.422 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.422 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.422 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:18:01.423 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:56.516059+0000", 2024-09-11T08:18:01.423 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3840000000000002e-06 2024-09-11T08:18:01.423 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.423 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.423 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:01.423 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.596886+0000", 2024-09-11T08:18:01.423 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0808264320000001 2024-09-11T08:18:01.423 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.423 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.423 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:18:01.423 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.597015+0000", 2024-09-11T08:18:01.423 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012892599999999999 2024-09-11T08:18:01.423 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.424 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.424 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2,6,7", 2024-09-11T08:18:01.424 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.597138+0000", 2024-09-11T08:18:01.424 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012275799999999999 2024-09-11T08:18:01.424 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.424 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.424 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:18:01.424 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.602333+0000", 2024-09-11T08:18:01.425 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0051956429999999998 2024-09-11T08:18:01.425 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.425 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.425 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:01.425 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.602666+0000", 2024-09-11T08:18:01.425 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00033239400000000002 2024-09-11T08:18:01.425 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.425 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.425 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:01.425 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.602719+0000", 2024-09-11T08:18:01.425 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3785000000000001e-05 2024-09-11T08:18:01.425 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.425 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.425 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:01.426 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.602761+0000", 2024-09-11T08:18:01.426 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1439000000000001e-05 2024-09-11T08:18:01.426 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.426 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.426 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:18:01.426 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.602809+0000", 2024-09-11T08:18:01.426 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8325999999999998e-05 2024-09-11T08:18:01.426 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.426 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.426 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:18:01.426 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.602901+0000", 2024-09-11T08:18:01.426 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.2319999999999997e-05 2024-09-11T08:18:01.426 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:01.426 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:18:01.427 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:01.427 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.427 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.427 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15087.0:146953 8.d 8:b9a7d93a:::benchmark_data_smithi150_107646_object18121:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-11T08:18:01.427 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:15:55.562297+0000", 2024-09-11T08:18:01.427 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 125.764846761, 2024-09-11T08:18:01.427 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0486156900000001, 2024-09-11T08:18:01.427 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:18:01.427 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:18:01.427 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:18:01.427 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15087", 2024-09-11T08:18:01.427 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/3299560913", 2024-09-11T08:18:01.427 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 146953 2024-09-11T08:18:01.427 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.428 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:18:01.428 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.428 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:18:01.429 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:18:01.429 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.562297+0000", 2024-09-11T08:18:01.429 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:01.429 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.430 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.430 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:18:01.430 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.562297+0000", 2024-09-11T08:18:01.430 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:01.430 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.430 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.430 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:18:01.430 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.562296+0000", 2024-09-11T08:18:01.430 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:18:01.430 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.430 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.430 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:18:01.430 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.562301+0000", 2024-09-11T08:18:01.430 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3159999999999996e-06 2024-09-11T08:18:01.431 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.431 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.431 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:18:01.431 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.562311+0000", 2024-09-11T08:18:01.431 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.1600000000000004e-06 2024-09-11T08:18:01.431 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.431 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.431 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:18:01.431 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.562314+0000", 2024-09-11T08:18:01.431 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8510000000000001e-06 2024-09-11T08:18:01.431 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.431 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.431 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:01.431 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.562357+0000", 2024-09-11T08:18:01.431 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2141000000000002e-05 2024-09-11T08:18:01.432 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.432 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.432 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:18:01.432 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.562367+0000", 2024-09-11T08:18:01.432 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.022e-05 2024-09-11T08:18:01.432 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.432 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.432 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:01.432 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:56.516092+0000", 2024-09-11T08:18:01.432 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.953725033 2024-09-11T08:18:01.432 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.432 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.432 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:18:01.432 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:56.516099+0000", 2024-09-11T08:18:01.433 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4989999999999997e-06 2024-09-11T08:18:01.433 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.433 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.433 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:01.433 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.599740+0000", 2024-09-11T08:18:01.433 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0836410590000001 2024-09-11T08:18:01.433 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.433 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.433 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:18:01.433 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.599852+0000", 2024-09-11T08:18:01.433 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011151500000000001 2024-09-11T08:18:01.433 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.433 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.434 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2,6,7", 2024-09-11T08:18:01.434 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.599965+0000", 2024-09-11T08:18:01.434 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000112868 2024-09-11T08:18:01.434 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.434 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.434 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:18:01.434 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.602674+0000", 2024-09-11T08:18:01.434 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0027091099999999998 2024-09-11T08:18:01.434 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.434 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.434 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:01.434 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.602939+0000", 2024-09-11T08:18:01.434 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00026491699999999998 2024-09-11T08:18:01.434 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.434 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.435 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:01.435 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.602969+0000", 2024-09-11T08:18:01.435 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9901999999999999e-05 2024-09-11T08:18:01.435 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.435 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.435 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:01.435 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.610819+0000", 2024-09-11T08:18:01.435 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0078507319999999992 2024-09-11T08:18:01.435 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.435 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.435 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:18:01.435 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.610852+0000", 2024-09-11T08:18:01.435 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2098000000000003e-05 2024-09-11T08:18:01.435 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.435 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.435 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:18:01.436 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.610913+0000", 2024-09-11T08:18:01.436 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1307000000000001e-05 2024-09-11T08:18:01.436 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:01.436 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:18:01.436 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:01.436 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.436 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.436 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15087.0:146954 8.d 8:b72d99d7:::benchmark_data_smithi150_107646_object18122:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-11T08:18:01.436 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:15:55.562323+0000", 2024-09-11T08:18:01.436 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 125.76482106100001, 2024-09-11T08:18:01.436 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0487770150000002, 2024-09-11T08:18:01.436 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:18:01.436 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:18:01.436 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:18:01.437 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15087", 2024-09-11T08:18:01.437 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/3299560913", 2024-09-11T08:18:01.437 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 146954 2024-09-11T08:18:01.437 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.437 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:18:01.437 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.437 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:18:01.437 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.562323+0000", 2024-09-11T08:18:01.437 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:01.437 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.437 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.437 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:18:01.437 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.562323+0000", 2024-09-11T08:18:01.437 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:01.438 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.438 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.438 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:18:01.438 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.562323+0000", 2024-09-11T08:18:01.438 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-09-11T08:18:01.438 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.438 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.438 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:18:01.438 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.562325+0000", 2024-09-11T08:18:01.438 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5770000000000001e-06 2024-09-11T08:18:01.438 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.438 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.438 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:18:01.439 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.562335+0000", 2024-09-11T08:18:01.439 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7699999999999996e-06 2024-09-11T08:18:01.439 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.439 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.439 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:18:01.439 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.562338+0000", 2024-09-11T08:18:01.439 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.36e-06 2024-09-11T08:18:01.439 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.439 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.439 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:01.439 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.562388+0000", 2024-09-11T08:18:01.439 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.99e-05 2024-09-11T08:18:01.439 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.440 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.440 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:18:01.440 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.562394+0000", 2024-09-11T08:18:01.440 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7629999999999997e-06 2024-09-11T08:18:01.440 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.440 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.440 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:01.440 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:56.516120+0000", 2024-09-11T08:18:01.440 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.95372561300000003 2024-09-11T08:18:01.440 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.440 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.440 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:18:01.440 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:56.516126+0000", 2024-09-11T08:18:01.440 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6470000000000001e-06 2024-09-11T08:18:01.441 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.441 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.441 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:01.441 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.602988+0000", 2024-09-11T08:18:01.441 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.086861898 2024-09-11T08:18:01.441 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.441 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.441 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:18:01.441 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.603088+0000", 2024-09-11T08:18:01.441 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010015599999999999 2024-09-11T08:18:01.441 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.441 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.441 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2,6,7", 2024-09-11T08:18:01.442 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.603195+0000", 2024-09-11T08:18:01.442 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010692900000000001 2024-09-11T08:18:01.442 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.442 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.442 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:01.442 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.610953+0000", 2024-09-11T08:18:01.442 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0077574510000000003 2024-09-11T08:18:01.442 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.442 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.442 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:01.442 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.610979+0000", 2024-09-11T08:18:01.442 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6463000000000001e-05 2024-09-11T08:18:01.442 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.443 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.443 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:18:01.443 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.610985+0000", 2024-09-11T08:18:01.443 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6869999999999999e-06 2024-09-11T08:18:01.443 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.443 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.443 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:01.443 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.611014+0000", 2024-09-11T08:18:01.443 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9213999999999998e-05 2024-09-11T08:18:01.443 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.443 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.443 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:18:01.443 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.611043+0000", 2024-09-11T08:18:01.444 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9257000000000001e-05 2024-09-11T08:18:01.444 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.444 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.444 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:18:01.444 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.611100+0000", 2024-09-11T08:18:01.444 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6637e-05 2024-09-11T08:18:01.444 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:01.444 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:18:01.444 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:01.444 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.444 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.444 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15087.0:146964 8.d 8:bfe3bcbc:::benchmark_data_smithi150_107646_object18132:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-11T08:18:01.444 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:15:55.572456+0000", 2024-09-11T08:18:01.445 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 125.7546876, 2024-09-11T08:18:01.445 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.045324017, 2024-09-11T08:18:01.445 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:18:01.445 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:18:01.445 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:18:01.445 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15087", 2024-09-11T08:18:01.445 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/3299560913", 2024-09-11T08:18:01.445 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 146964 2024-09-11T08:18:01.445 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.445 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:18:01.445 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.445 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:18:01.445 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.572456+0000", 2024-09-11T08:18:01.445 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:01.446 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.446 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.446 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:18:01.446 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.572456+0000", 2024-09-11T08:18:01.446 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:01.446 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.446 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.446 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:18:01.446 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.572455+0000", 2024-09-11T08:18:01.446 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:18:01.446 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.446 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.446 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:18:01.446 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.572460+0000", 2024-09-11T08:18:01.447 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1789999999999999e-06 2024-09-11T08:18:01.447 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.447 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.447 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:18:01.447 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.572472+0000", 2024-09-11T08:18:01.447 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1291000000000001e-05 2024-09-11T08:18:01.447 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.447 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.447 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:18:01.447 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.572477+0000", 2024-09-11T08:18:01.447 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2789999999999998e-06 2024-09-11T08:18:01.447 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.447 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.447 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:01.447 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.572528+0000", 2024-09-11T08:18:01.448 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0782999999999998e-05 2024-09-11T08:18:01.448 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.448 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.448 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:18:01.448 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.572535+0000", 2024-09-11T08:18:01.448 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0870000000000001e-06 2024-09-11T08:18:01.448 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.448 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.448 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:01.448 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:56.516144+0000", 2024-09-11T08:18:01.448 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.94360924099999999 2024-09-11T08:18:01.448 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.448 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.448 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:18:01.449 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:56.516151+0000", 2024-09-11T08:18:01.449 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.392e-06 2024-09-11T08:18:01.449 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.449 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.449 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:01.449 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.608431+0000", 2024-09-11T08:18:01.449 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.092280602 2024-09-11T08:18:01.449 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.449 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.449 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:18:01.449 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.608554+0000", 2024-09-11T08:18:01.449 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012311900000000001 2024-09-11T08:18:01.450 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.450 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.450 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2,6,7", 2024-09-11T08:18:01.450 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.608639+0000", 2024-09-11T08:18:01.450 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4315999999999997e-05 2024-09-11T08:18:01.450 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.450 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.450 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:18:01.450 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.610989+0000", 2024-09-11T08:18:01.450 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0023499670000000001 2024-09-11T08:18:01.450 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.450 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.450 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:01.450 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.611143+0000", 2024-09-11T08:18:01.451 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015454800000000001 2024-09-11T08:18:01.451 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.451 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.451 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:01.451 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.617527+0000", 2024-09-11T08:18:01.451 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.006384308 2024-09-11T08:18:01.451 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.451 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.451 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:01.451 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.617678+0000", 2024-09-11T08:18:01.451 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000150731 2024-09-11T08:18:01.451 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.451 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.451 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:18:01.451 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.617708+0000", 2024-09-11T08:18:01.452 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9770000000000001e-05 2024-09-11T08:18:01.452 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.452 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.452 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:18:01.452 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.617780+0000", 2024-09-11T08:18:01.452 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2453999999999994e-05 2024-09-11T08:18:01.452 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:01.452 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:18:01.452 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:01.452 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.452 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.452 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15087.0:146995 8.d 8:b04cb85d:::benchmark_data_smithi150_107646_object18163:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-11T08:18:01.452 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:15:55.613312+0000", 2024-09-11T08:18:01.452 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 125.71383173300001, 2024-09-11T08:18:01.453 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0097442380000001, 2024-09-11T08:18:01.453 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:18:01.453 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:18:01.453 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:18:01.453 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15087", 2024-09-11T08:18:01.453 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/3299560913", 2024-09-11T08:18:01.453 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 146995 2024-09-11T08:18:01.453 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.453 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:18:01.453 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.453 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:18:01.453 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.613312+0000", 2024-09-11T08:18:01.453 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:01.453 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.453 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.454 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:18:01.454 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.613312+0000", 2024-09-11T08:18:01.454 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:01.454 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.454 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.454 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:18:01.454 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.613311+0000", 2024-09-11T08:18:01.454 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:18:01.454 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.454 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.454 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:18:01.454 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.613317+0000", 2024-09-11T08:18:01.454 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.874e-06 2024-09-11T08:18:01.454 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.455 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.455 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:18:01.455 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.613327+0000", 2024-09-11T08:18:01.455 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0071999999999999e-05 2024-09-11T08:18:01.455 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.455 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.455 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:18:01.455 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.613332+0000", 2024-09-11T08:18:01.455 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0730000000000004e-06 2024-09-11T08:18:01.455 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.455 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.455 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:01.455 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.613376+0000", 2024-09-11T08:18:01.455 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4152e-05 2024-09-11T08:18:01.455 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.455 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.456 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:18:01.456 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.613383+0000", 2024-09-11T08:18:01.456 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0299999999999996e-06 2024-09-11T08:18:01.456 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.456 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.456 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:01.456 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:56.516173+0000", 2024-09-11T08:18:01.456 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.90278959400000003 2024-09-11T08:18:01.456 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.456 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.456 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:18:01.456 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:56.516179+0000", 2024-09-11T08:18:01.456 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2779999999999999e-06 2024-09-11T08:18:01.456 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.457 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.457 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:01.457 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.611173+0000", 2024-09-11T08:18:01.457 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.094993423 2024-09-11T08:18:01.457 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.457 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.457 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:18:01.457 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.611241+0000", 2024-09-11T08:18:01.457 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8659999999999997e-05 2024-09-11T08:18:01.457 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.457 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.457 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2,6,7", 2024-09-11T08:18:01.457 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.611365+0000", 2024-09-11T08:18:01.458 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012359200000000001 2024-09-11T08:18:01.458 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.458 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.458 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:01.458 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.617594+0000", 2024-09-11T08:18:01.458 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0062289559999999999 2024-09-11T08:18:01.458 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.458 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.458 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:01.458 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.617645+0000", 2024-09-11T08:18:01.458 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1273000000000002e-05 2024-09-11T08:18:01.458 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.458 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.458 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:18:01.459 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.617650+0000", 2024-09-11T08:18:01.459 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0529999999999997e-06 2024-09-11T08:18:01.459 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.459 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.459 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:01.459 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.622960+0000", 2024-09-11T08:18:01.459 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.005309635 2024-09-11T08:18:01.459 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.459 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.459 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:18:01.459 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.622997+0000", 2024-09-11T08:18:01.459 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7512000000000003e-05 2024-09-11T08:18:01.459 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.460 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.460 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:18:01.460 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.623056+0000", 2024-09-11T08:18:01.460 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9227000000000001e-05 2024-09-11T08:18:01.460 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:01.460 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:18:01.460 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:01.460 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.460 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.460 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15087.0:147023 8.d 8:bfef95ee:::benchmark_data_smithi150_107646_object18191:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-11T08:18:01.460 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:15:55.674484+0000", 2024-09-11T08:18:01.460 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 125.652659695, 2024-09-11T08:18:01.460 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9487520229999999, 2024-09-11T08:18:01.461 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:18:01.461 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:18:01.461 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:18:01.461 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15087", 2024-09-11T08:18:01.461 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/3299560913", 2024-09-11T08:18:01.461 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 147023 2024-09-11T08:18:01.461 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.461 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:18:01.461 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.461 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:18:01.461 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.674484+0000", 2024-09-11T08:18:01.461 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:01.461 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.462 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.462 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:18:01.462 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.674484+0000", 2024-09-11T08:18:01.462 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:01.462 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.462 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.462 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:18:01.462 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.674483+0000", 2024-09-11T08:18:01.462 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:18:01.462 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.462 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.462 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:18:01.462 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.674489+0000", 2024-09-11T08:18:01.462 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.088e-06 2024-09-11T08:18:01.463 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.463 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.463 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:18:01.463 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.674501+0000", 2024-09-11T08:18:01.463 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1855000000000001e-05 2024-09-11T08:18:01.463 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.463 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.463 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:18:01.463 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.674507+0000", 2024-09-11T08:18:01.463 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.117e-06 2024-09-11T08:18:01.463 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.463 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.463 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:01.463 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.674555+0000", 2024-09-11T08:18:01.464 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8077000000000001e-05 2024-09-11T08:18:01.464 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.464 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.464 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:18:01.464 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.674561+0000", 2024-09-11T08:18:01.464 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.161e-06 2024-09-11T08:18:01.464 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.464 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.464 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:01.464 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:56.516197+0000", 2024-09-11T08:18:01.464 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.84163608400000001 2024-09-11T08:18:01.464 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.464 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.465 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:18:01.465 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:56.516204+0000", 2024-09-11T08:18:01.465 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5379999999999999e-06 2024-09-11T08:18:01.465 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.465 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.465 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:01.465 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.614887+0000", 2024-09-11T08:18:01.465 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0986829709999999 2024-09-11T08:18:01.465 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.465 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.465 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:18:01.465 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.615027+0000", 2024-09-11T08:18:01.465 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014060200000000001 2024-09-11T08:18:01.465 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.466 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.466 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2,6,7", 2024-09-11T08:18:01.466 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.615154+0000", 2024-09-11T08:18:01.466 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012669299999999999 2024-09-11T08:18:01.466 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.466 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.466 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:01.466 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.617825+0000", 2024-09-11T08:18:01.466 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002670894 2024-09-11T08:18:01.466 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.466 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.466 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:18:01.466 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.617833+0000", 2024-09-11T08:18:01.466 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5439999999999999e-06 2024-09-11T08:18:01.467 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.467 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.467 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:01.467 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.622923+0000", 2024-09-11T08:18:01.467 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0050899059999999999 2024-09-11T08:18:01.467 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.467 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.467 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:01.467 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.623114+0000", 2024-09-11T08:18:01.467 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000190278 2024-09-11T08:18:01.467 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.467 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.467 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:18:01.468 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.623162+0000", 2024-09-11T08:18:01.468 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8381999999999999e-05 2024-09-11T08:18:01.468 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:01.468 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:01.468 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:18:01.468 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.623236+0000", 2024-09-11T08:18:01.468 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4342000000000001e-05 2024-09-11T08:18:01.468 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:01.468 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:18:01.468 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:01.468 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:01.468 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:18:01.468 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-11T08:18:01.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:01.470 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:01.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:18:01.500 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-09-11T08:18:01.529 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:18:01.571 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:01.593 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:01.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:01.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:01.695 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:01.717 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:01.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:01.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:01.807 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:18:01.819 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:01.844 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:01.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:01.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:01.946 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:01.967 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:01.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:01.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:02.068 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:02.089 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:02.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:02.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:02.191 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:02.197 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:01 smithi150 ceph-mon[29276]: osdmap e247: 8 total, 8 up, 8 in 2024-09-11T08:18:02.197 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:01 smithi150 ceph-mon[29276]: pgmap v474: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:18:02.197 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:01 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:18:02.197 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:01 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:18:02.197 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:01 smithi150 ceph-mon[39612]: osdmap e247: 8 total, 8 up, 8 in 2024-09-11T08:18:02.197 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:01 smithi150 ceph-mon[39612]: pgmap v474: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:18:02.197 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:01 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:18:02.197 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:01 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:18:02.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:02.214 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:02.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:02.315 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:02.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:01 smithi181 ceph-mon[32701]: osdmap e247: 8 total, 8 up, 8 in 2024-09-11T08:18:02.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:01 smithi181 ceph-mon[32701]: pgmap v474: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:18:02.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:01 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:18:02.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:01 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:18:02.336 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:02.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:02.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:02.438 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:02.459 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:02.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:02.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:02.498 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-11T08:18:02.561 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:18:02.582 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:02.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:02.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:18:02.684 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:02.706 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:02.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:02.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:02.783 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:18:02.807 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:02.830 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:02.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:02.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:02.890 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:18:02.914 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:18:02.931 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:02.953 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:02.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:02.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:03.055 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:18:03.077 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:03.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:03.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:18:03.127 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:02 smithi150 ceph-mon[39612]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-11T08:18:03.127 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:02 smithi150 ceph-mon[39612]: osdmap e248: 8 total, 8 up, 8 in 2024-09-11T08:18:03.127 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:02 smithi150 ceph-mon[29276]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-11T08:18:03.127 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:02 smithi150 ceph-mon[29276]: osdmap e248: 8 total, 8 up, 8 in 2024-09-11T08:18:03.136 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:02 smithi181 ceph-mon[32701]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-11T08:18:03.137 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:02 smithi181 ceph-mon[32701]: osdmap e248: 8 total, 8 up, 8 in 2024-09-11T08:18:03.179 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:03.201 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:03.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:03.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:03.303 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:03.326 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:03.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:03.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:03.428 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:18:03.450 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:03.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:03.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:18:03.551 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:18:03.573 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:03.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:03.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:18:03.675 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:03.696 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:03.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:03.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:03.798 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:18:03.819 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:03.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:03.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:18:03.921 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:03.943 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:03.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:03.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:04.045 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:18:04.066 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:04.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:04.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:18:04.110 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:03 smithi181 ceph-mon[32701]: pgmap v476: 148 pgs: 11 unknown, 137 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-11T08:18:04.168 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:18:04.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:04.191 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:04.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:18:04.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:03 smithi150 ceph-mon[29276]: pgmap v476: 148 pgs: 11 unknown, 137 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-11T08:18:04.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:03 smithi150 ceph-mon[39612]: pgmap v476: 148 pgs: 11 unknown, 137 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-11T08:18:04.293 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:04.317 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:04.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:04.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:04.362 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-11T08:18:04.362 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-11T08:18:04.362 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-09-11T08:18:04.362 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-11T08:18:04.420 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:04.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:04.445 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:04.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:04.546 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:04.567 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:04.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:04.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:04.669 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:04.691 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:04.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:04.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:04.792 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:04.814 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:04.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:04.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:04.915 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:04.936 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:04.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:04.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:05.038 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:05.048 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-11T08:18:05.059 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:05.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:05.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:05.161 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:05.182 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:05.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:05.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:05.284 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:05.305 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:18:05.307 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:05.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:05.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:05.323 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:18:05 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:18:05] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:18:05.409 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:05.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:05.432 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:05.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:05.533 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:05.554 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:05.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:05.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:05.656 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:18:05.678 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:05.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:05.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:18:05.780 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:18:05.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:05.802 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:05.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:18:05.903 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:05.927 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:05.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:05.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:06.028 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:06.050 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:06.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:06.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:06.064 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:05 smithi181 ceph-mon[32701]: pgmap v477: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:18:06.152 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:06.173 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:06.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:06.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:06.208 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:05 smithi150 ceph-mon[29276]: pgmap v477: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:18:06.208 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:05 smithi150 ceph-mon[39612]: pgmap v477: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:18:06.275 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:06.279 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:18:06.279 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:18:06.279 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:18:06.279 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:18:06.279 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:18:06.279 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:18:06.280 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:18:06.280 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:18:06.280 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:18:06.280 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:18:06.280 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:18:06.280 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:18:06.280 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:18:06.280 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:18:06.280 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:18:06.280 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:18:06.298 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:06.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:06.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:06.399 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:06.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:06.421 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:06.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:06.522 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:18:06.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:06.544 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:06.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:18:06.645 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:06.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:06.685 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:06.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:06.786 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:06.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:06.809 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:06.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:06.894 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-11T08:18:06.894 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:18:06.910 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:06.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:06.933 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:06.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:07.034 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:07.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:07.056 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:07.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:07.157 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:18:07.179 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:07.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:07.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:18:07.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:06 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:18:07.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:06 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:18:07.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:06 smithi150 ceph-mon[29276]: osdmap e249: 8 total, 8 up, 8 in 2024-09-11T08:18:07.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:06 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/437775958' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:18:07.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:06 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:18:07.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:06 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:18:07.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:06 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:18:07.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:06 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:18:07.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:06 smithi150 ceph-mon[39612]: osdmap e249: 8 total, 8 up, 8 in 2024-09-11T08:18:07.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:06 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/437775958' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:18:07.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:06 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:18:07.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:06 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:18:07.240 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:06 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:18:07.240 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:06 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:18:07.240 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:06 smithi181 ceph-mon[32701]: osdmap e249: 8 total, 8 up, 8 in 2024-09-11T08:18:07.240 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:06 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/437775958' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:18:07.240 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:06 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:18:07.240 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:06 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:18:07.280 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:07.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:07.303 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:07.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:07.404 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:07.426 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:07.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:07.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:07.528 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:07.552 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:07.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:07.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:07.566 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-11T08:18:07.653 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:07.675 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:07.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:07.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:07.776 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:18:07.798 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:07.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:07.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:18:07.855 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:18:07.899 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:07.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:07.922 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:07.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:07.990 INFO:teuthology.orchestra.run.smithi150.stderr:set 4.1a pg_upmap_items mapping to [6->3,1->2,7->5] 2024-09-11T08:18:07.990 INFO:teuthology.orchestra.run.smithi150.stderr:nodeep-scrub is set 2024-09-11T08:18:08.023 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:08.044 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:08.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:08.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:08.146 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:18:08.167 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:08.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:08.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:18:08.268 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:08.291 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:08.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:08.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:08.293 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:07 smithi181 ceph-mon[32701]: pgmap v479: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 839 B/s rd, 0 op/s 2024-09-11T08:18:08.293 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:07 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1706791762' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.1a", "id": [6, 3, 1, 2, 7, 5]}]: dispatch 2024-09-11T08:18:08.293 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:07 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2391983968' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:18:08.293 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:07 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:18:08.393 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:08.414 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:08.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:08.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:08.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:07 smithi150 ceph-mon[29276]: pgmap v479: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 839 B/s rd, 0 op/s 2024-09-11T08:18:08.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:07 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1706791762' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.1a", "id": [6, 3, 1, 2, 7, 5]}]: dispatch 2024-09-11T08:18:08.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:07 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2391983968' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:18:08.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:07 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:18:08.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:07 smithi150 ceph-mon[39612]: pgmap v479: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 839 B/s rd, 0 op/s 2024-09-11T08:18:08.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:07 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1706791762' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.1a", "id": [6, 3, 1, 2, 7, 5]}]: dispatch 2024-09-11T08:18:08.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:07 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2391983968' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:18:08.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:07 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:18:08.516 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:08.537 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:08.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:08.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:08.573 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:18:08 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:18:08.639 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:08.661 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:08.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:08.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:08.763 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:18:08.778 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:18:08 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:18:08] "GET /metrics HTTP/1.1" 200 224049 "" "Prometheus/2.43.0" 2024-09-11T08:18:08.786 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:08.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:08.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:18:08.888 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:08.909 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:08.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:08.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:09.011 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:09.031 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:09 smithi181 ceph-mon[32701]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:18:09.031 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:09 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1706791762' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "4.1a", "id": [6, 3, 1, 2, 7, 5]}]': finished 2024-09-11T08:18:09.031 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:09 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:18:09.032 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:09 smithi181 ceph-mon[32701]: osdmap e250: 8 total, 8 up, 8 in 2024-09-11T08:18:09.032 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:09 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:18:09.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:09.034 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:09.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:09.038 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:09 smithi150 ceph-mon[39612]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:18:09.039 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:09 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1706791762' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "4.1a", "id": [6, 3, 1, 2, 7, 5]}]': finished 2024-09-11T08:18:09.039 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:09 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:18:09.039 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:09 smithi150 ceph-mon[39612]: osdmap e250: 8 total, 8 up, 8 in 2024-09-11T08:18:09.039 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:09 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:18:09.039 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:09 smithi150 ceph-mon[29276]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:18:09.039 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:09 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1706791762' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "4.1a", "id": [6, 3, 1, 2, 7, 5]}]': finished 2024-09-11T08:18:09.039 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:09 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:18:09.039 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:09 smithi150 ceph-mon[29276]: osdmap e250: 8 total, 8 up, 8 in 2024-09-11T08:18:09.039 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:09 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:18:09.135 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:09.156 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:09.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:09.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:09.258 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:09.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:09.280 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:09.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:09.381 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:09.403 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:09.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:09.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:09.504 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:09.526 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:09.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:09.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:09.571 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-11T08:18:09.571 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:18:09.627 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:09.649 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:09.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:09.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:09.751 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:09.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:09.774 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:09.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:09.875 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:10.032 INFO:teuthology.orchestra.run.smithi150.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-09-11T08:18:10.056 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:10.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:10.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:10.158 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:10.180 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:10.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:10.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:10.188 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-11T08:18:10.282 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:10.304 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:10.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:10.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:10.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:10 smithi181 ceph-mon[32701]: pgmap v481: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-11T08:18:10.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:10 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2671093986' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-11T08:18:10.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:10 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-11T08:18:10.398 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:10 smithi150 ceph-mon[39612]: pgmap v481: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-11T08:18:10.398 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:10 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2671093986' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-11T08:18:10.398 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:10 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-11T08:18:10.399 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:10 smithi150 ceph-mon[29276]: pgmap v481: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-11T08:18:10.399 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:10 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2671093986' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-11T08:18:10.399 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:10 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-11T08:18:10.405 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:10.428 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:10.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:10.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:10.434 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:18:10.529 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:10.551 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:10.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:10.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:10.653 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:10.675 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:10.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:10.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:10.777 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:10.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:10.800 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:10.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:10.901 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:10.922 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:10.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:10.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:10.942 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T08:18:11.024 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:11.047 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:11.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:11.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:11.149 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:11.170 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:11.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:11.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:11.238 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:11 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-11T08:18:11.238 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:11 smithi181 ceph-mon[32701]: osdmap e251: 8 total, 8 up, 8 in 2024-09-11T08:18:11.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:11 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-11T08:18:11.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:11 smithi150 ceph-mon[29276]: osdmap e251: 8 total, 8 up, 8 in 2024-09-11T08:18:11.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:11 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-11T08:18:11.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:11 smithi150 ceph-mon[39612]: osdmap e251: 8 total, 8 up, 8 in 2024-09-11T08:18:11.271 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:11.292 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:11.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:11.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:11.339 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:18:11.396 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:11.418 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:11.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:11.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:11.519 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:18:11.542 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:11.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:11.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:18:11.643 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:18:11.666 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:11.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:11.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:18:11.715 DEBUG:teuthology.orchestra.run.smithi150:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-09-11T08:18:11.718 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-11T08:18:11.718 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-11T08:18:11.763 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:hints = 1 2024-09-11T08:18:11.763 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-11T08:18:11.763 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Object prefix: benchmark_data_smithi150_133616 2024-09-11T08:18:11.763 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-11T08:18:11.763 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 0 0 0 0 0 0 - 0 2024-09-11T08:18:11.768 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:11.805 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:11.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:11.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:11.907 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:11.943 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:11.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:11.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:12.045 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:12.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:12 smithi181 ceph-mon[32701]: pgmap v483: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:18:12.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:12 smithi181 ceph-mon[32701]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-11T08:18:12.082 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:12.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:12.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:12.144 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-11T08:18:12.144 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:18:12.184 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:12.223 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:12.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:12.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:12.325 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:18:12.362 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:12.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:12.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:18:12.464 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:12.473 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:12 smithi150 ceph-mon[29276]: pgmap v483: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:18:12.473 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:12 smithi150 ceph-mon[29276]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-11T08:18:12.474 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:12 smithi150 ceph-mon[39612]: pgmap v483: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:18:12.474 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:12 smithi150 ceph-mon[39612]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-11T08:18:12.496 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:12.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:12.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:12.598 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:12.632 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:12.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:12.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:12.734 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:12.763 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 1 16 4014 3998 249.862 249.875 0.0027226 0.00399031 2024-09-11T08:18:12.767 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:12.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:12.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:12.869 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:12.903 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:12.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:12.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:13.005 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:13.040 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:13.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:13.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:13.088 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-11T08:18:13.142 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:13.177 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:13.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:13.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:13.279 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:13.285 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T08:18:13.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:13.305 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:13.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:13.406 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:13.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:13.443 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:13.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:13.490 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:18:13.545 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:13.551 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:18:13.551 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:18:13.576 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:13.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:13.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:13.593 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:18:13.594 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:18:13.624 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:18:13.624 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:18:13.657 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:18:13.657 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:18:13.678 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:13.691 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:18:13.691 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:18:13.708 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:13.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:13.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:13.717 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:18:13.738 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:18:13.738 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:18:13.763 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 2 16 7145 7129 222.737 195.688 0.00266762 0.00447923 2024-09-11T08:18:13.771 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:18:13.772 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:18:13.802 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:18:13.802 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:18:13.809 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:13.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:13.843 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:13.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:13.944 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:13.980 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:13.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:13.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:14.085 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:14.114 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:14.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:14.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:14.215 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:14.246 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:14.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:14.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:14.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:14 smithi181 ceph-mon[32701]: pgmap v484: 148 pgs: 148 active+clean; 84 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 12 MiB/s wr, 193 op/s 2024-09-11T08:18:14.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:14 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3068061511' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:18:14.349 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:14.350 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:14 smithi150 ceph-mon[29276]: pgmap v484: 148 pgs: 148 active+clean; 84 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 12 MiB/s wr, 193 op/s 2024-09-11T08:18:14.350 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:14 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3068061511' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:18:14.350 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:14 smithi150 ceph-mon[39612]: pgmap v484: 148 pgs: 148 active+clean; 84 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 12 MiB/s wr, 193 op/s 2024-09-11T08:18:14.350 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:14 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3068061511' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:18:14.385 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:14.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:14.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:14.487 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:14.521 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:14.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:14.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:14.623 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:14.658 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:14.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:14.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:14.760 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:14.763 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 3 16 10286 10270 213.924 196.312 0.00455682 0.00466637 2024-09-11T08:18:14.792 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:14.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:14.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:14.894 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:18:14.934 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:14.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:14.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:18:15.036 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:15.072 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-11T08:18:15.072 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T08:18:15.072 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-11T08:18:15.073 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T08:18:15.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:15.074 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:15.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:15.175 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:15.208 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:15.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:15.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:15.311 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:15.323 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:18:15 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:18:15] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:18:15.343 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:15.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:15.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:15.444 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:18:15.466 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:15.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:15.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:18:15.568 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:15.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:15.605 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:15.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:15.706 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:18:15.743 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:15.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:15.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:18:15.764 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 4 16 13077 13061 204.032 174.438 0.00367019 0.00489498 2024-09-11T08:18:15.846 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:15.879 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:15.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:15.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:15.981 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:16.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:16.021 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:16.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:16.122 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:16.161 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:16.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:16.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:16.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:16 smithi150 ceph-mon[29276]: pgmap v485: 148 pgs: 148 active+clean; 323 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 40 MiB/s wr, 646 op/s 2024-09-11T08:18:16.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:16 smithi150 ceph-mon[39612]: pgmap v485: 148 pgs: 148 active+clean; 323 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 40 MiB/s wr, 646 op/s 2024-09-11T08:18:16.262 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:18:16.272 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:16 smithi181 ceph-mon[32701]: pgmap v485: 148 pgs: 148 active+clean; 323 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 40 MiB/s wr, 646 op/s 2024-09-11T08:18:16.294 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-11T08:18:16.294 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-11T08:18:16.294 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-09-11T08:18:16.294 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-09-11T08:18:16.294 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-11T08:18:16.301 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:16.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:16.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:18:16.403 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:18:16.454 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:16.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:16.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:18:16.556 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:16.593 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:16.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:16.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:16.695 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:16.729 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:16.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:16.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:16.764 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 5 16 16056 16040 200.458 186.188 0.00597899 0.00498136 2024-09-11T08:18:16.790 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:18:16.831 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:16.860 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:16.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:16.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:16.961 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:16.983 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:16.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:16.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:17.001 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-11T08:18:17.085 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:18:17.118 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:17.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:17.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:18:17.220 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:17.256 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:17.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:17.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:17.310 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T08:18:17.358 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:17.402 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:17.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:17.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:17.449 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:18:17.505 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:17.539 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:17.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:17.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:17.641 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:18:17.679 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:17.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:17.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:18:17.764 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 6 16 18331 18315 190.745 142.188 0.00289886 0.00523636 2024-09-11T08:18:17.782 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:17.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:17.815 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:17.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:17.916 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:17.952 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:17.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:17.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:18.048 INFO:teuthology.orchestra.run.smithi150.stderr:noscrub is unset 2024-09-11T08:18:18.053 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:18:18.094 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:18.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:18.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:18:18.112 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:18 smithi150 ceph-mon[29276]: pgmap v486: 148 pgs: 148 active+clean; 323 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 573 B/s rd, 36 MiB/s wr, 578 op/s 2024-09-11T08:18:18.113 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:18 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/598058582' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:18:18.113 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:18 smithi150 ceph-mon[39612]: pgmap v486: 148 pgs: 148 active+clean; 323 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 573 B/s rd, 36 MiB/s wr, 578 op/s 2024-09-11T08:18:18.113 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:18 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/598058582' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:18:18.183 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:18 smithi181 ceph-mon[32701]: pgmap v486: 148 pgs: 148 active+clean; 323 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 573 B/s rd, 36 MiB/s wr, 578 op/s 2024-09-11T08:18:18.183 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:18 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/598058582' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:18:18.197 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:18:18.231 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:18.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:18.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:18:18.333 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:18.371 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:18.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:18.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:18.473 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:18:18.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:18.510 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:18.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:18:18.573 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:18:18 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:18:18.611 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:18:18.643 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:18.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:18.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:18:18.744 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:18.764 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 7 16 21004 20988 187.36 167.062 0.00663868 0.00533126 2024-09-11T08:18:18.778 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:18.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:18.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:18.880 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:18.914 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:18.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:18.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:18.921 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:18:18 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:18:18] "GET /metrics HTTP/1.1" 200 224049 "" "Prometheus/2.43.0" 2024-09-11T08:18:19.017 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:19.057 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:19.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:19.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:19.158 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:19.193 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:19.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:19.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:19.216 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:19 smithi181 ceph-mon[32701]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:18:19.216 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:19 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/598058582' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:18:19.216 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:19 smithi181 ceph-mon[32701]: osdmap e252: 8 total, 8 up, 8 in 2024-09-11T08:18:19.216 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:19 smithi181 ceph-mon[32701]: 3.1f scrub starts 2024-09-11T08:18:19.216 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:19 smithi181 ceph-mon[32701]: 3.1f scrub ok 2024-09-11T08:18:19.216 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:19 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:18:19.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:19 smithi150 ceph-mon[29276]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:18:19.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:19 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/598058582' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:18:19.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:19 smithi150 ceph-mon[29276]: osdmap e252: 8 total, 8 up, 8 in 2024-09-11T08:18:19.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:19 smithi150 ceph-mon[29276]: 3.1f scrub starts 2024-09-11T08:18:19.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:19 smithi150 ceph-mon[29276]: 3.1f scrub ok 2024-09-11T08:18:19.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:19 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:18:19.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:19 smithi150 ceph-mon[39612]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:18:19.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:19 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/598058582' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:18:19.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:19 smithi150 ceph-mon[39612]: osdmap e252: 8 total, 8 up, 8 in 2024-09-11T08:18:19.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:19 smithi150 ceph-mon[39612]: 3.1f scrub starts 2024-09-11T08:18:19.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:19 smithi150 ceph-mon[39612]: 3.1f scrub ok 2024-09-11T08:18:19.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:19 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:18:19.295 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:19.335 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:19.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:19.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:19.437 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:18:19.463 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:19.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:19.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:18:19.565 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:18:19.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:19.599 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:19.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:18:19.700 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:19.744 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:19.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:19.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:19.764 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 8 16 23115 23099 180.429 131.938 0.00452207 0.00553673 2024-09-11T08:18:19.846 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:19.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:19.882 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:19.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:19.983 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:20.031 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:20.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:20.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:20.133 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:18:20.155 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:20.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:20.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:18:20.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:20 smithi150 ceph-mon[29276]: 5.17 scrub starts 2024-09-11T08:18:20.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:20 smithi150 ceph-mon[29276]: 5.17 scrub ok 2024-09-11T08:18:20.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:20 smithi150 ceph-mon[29276]: pgmap v488: 148 pgs: 148 active+clean; 1.2 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 135 MiB/s wr, 2.15k op/s 2024-09-11T08:18:20.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:20 smithi150 ceph-mon[39612]: 5.17 scrub starts 2024-09-11T08:18:20.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:20 smithi150 ceph-mon[39612]: 5.17 scrub ok 2024-09-11T08:18:20.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:20 smithi150 ceph-mon[39612]: pgmap v488: 148 pgs: 148 active+clean; 1.2 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 135 MiB/s wr, 2.15k op/s 2024-09-11T08:18:20.257 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:18:20.282 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:18:20.309 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:20.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:20.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:18:20.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:20 smithi181 ceph-mon[32701]: 5.17 scrub starts 2024-09-11T08:18:20.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:20 smithi181 ceph-mon[32701]: 5.17 scrub ok 2024-09-11T08:18:20.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:20 smithi181 ceph-mon[32701]: pgmap v488: 148 pgs: 148 active+clean; 1.2 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 135 MiB/s wr, 2.15k op/s 2024-09-11T08:18:20.411 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:20.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:20.452 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:20.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:20.553 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:20.561 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:18:20.564 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-09-11T08:18:20.582 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:20.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:20.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:20.684 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:20.716 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:20.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:20.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:20.765 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 9 16 25074 25058 173.985 122.438 0.00284682 0.00571951 2024-09-11T08:18:20.818 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:20.853 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:20.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:20.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:20.955 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:20.982 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:20.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:20.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:21.084 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:21.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:21.121 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:21.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:21.222 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:18:21.258 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:21.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:21.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:18:21.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:21 smithi181 ceph-mon[32701]: 3.8 scrub starts 2024-09-11T08:18:21.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:21 smithi181 ceph-mon[32701]: 3.8 scrub ok 2024-09-11T08:18:21.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:21 smithi181 ceph-mon[32701]: 4.7 scrub starts 2024-09-11T08:18:21.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:21 smithi181 ceph-mon[32701]: 4.7 scrub ok 2024-09-11T08:18:21.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:21 smithi181 ceph-mon[32701]: 5.1a scrub starts 2024-09-11T08:18:21.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:21 smithi181 ceph-mon[32701]: 5.1a scrub ok 2024-09-11T08:18:21.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:21 smithi181 ceph-mon[32701]: 6.10 scrub starts 2024-09-11T08:18:21.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:21 smithi181 ceph-mon[32701]: 6.10 scrub ok 2024-09-11T08:18:21.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:21 smithi181 ceph-mon[32701]: 3.a scrub starts 2024-09-11T08:18:21.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:21 smithi181 ceph-mon[32701]: 3.a scrub ok 2024-09-11T08:18:21.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:21 smithi181 ceph-mon[32701]: 4.1d scrub starts 2024-09-11T08:18:21.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:21 smithi181 ceph-mon[32701]: 4.1d scrub ok 2024-09-11T08:18:21.360 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:18:21.382 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:21.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:21.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:18:21.484 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:21.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:21 smithi150 ceph-mon[39612]: 3.8 scrub starts 2024-09-11T08:18:21.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:21 smithi150 ceph-mon[39612]: 3.8 scrub ok 2024-09-11T08:18:21.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:21 smithi150 ceph-mon[39612]: 4.7 scrub starts 2024-09-11T08:18:21.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:21 smithi150 ceph-mon[39612]: 4.7 scrub ok 2024-09-11T08:18:21.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:21 smithi150 ceph-mon[39612]: 5.1a scrub starts 2024-09-11T08:18:21.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:21 smithi150 ceph-mon[39612]: 5.1a scrub ok 2024-09-11T08:18:21.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:21 smithi150 ceph-mon[39612]: 6.10 scrub starts 2024-09-11T08:18:21.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:21 smithi150 ceph-mon[39612]: 6.10 scrub ok 2024-09-11T08:18:21.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:21 smithi150 ceph-mon[39612]: 3.a scrub starts 2024-09-11T08:18:21.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:21 smithi150 ceph-mon[39612]: 3.a scrub ok 2024-09-11T08:18:21.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:21 smithi150 ceph-mon[39612]: 4.1d scrub starts 2024-09-11T08:18:21.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:21 smithi150 ceph-mon[39612]: 4.1d scrub ok 2024-09-11T08:18:21.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:21 smithi150 ceph-mon[29276]: 3.8 scrub starts 2024-09-11T08:18:21.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:21 smithi150 ceph-mon[29276]: 3.8 scrub ok 2024-09-11T08:18:21.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:21 smithi150 ceph-mon[29276]: 4.7 scrub starts 2024-09-11T08:18:21.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:21 smithi150 ceph-mon[29276]: 4.7 scrub ok 2024-09-11T08:18:21.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:21 smithi150 ceph-mon[29276]: 5.1a scrub starts 2024-09-11T08:18:21.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:21 smithi150 ceph-mon[29276]: 5.1a scrub ok 2024-09-11T08:18:21.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:21 smithi150 ceph-mon[29276]: 6.10 scrub starts 2024-09-11T08:18:21.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:21 smithi150 ceph-mon[29276]: 6.10 scrub ok 2024-09-11T08:18:21.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:21 smithi150 ceph-mon[29276]: 3.a scrub starts 2024-09-11T08:18:21.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:21 smithi150 ceph-mon[29276]: 3.a scrub ok 2024-09-11T08:18:21.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:21 smithi150 ceph-mon[29276]: 4.1d scrub starts 2024-09-11T08:18:21.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:21 smithi150 ceph-mon[29276]: 4.1d scrub ok 2024-09-11T08:18:21.517 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:21.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:21.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:21.619 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:21.643 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:21.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:21.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:21.714 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T08:18:21.745 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:21.765 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 10 16 26787 26771 167.292 107.062 0.0054186 0.00597152 2024-09-11T08:18:21.771 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:21.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:21.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:21.873 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:18:21.916 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:21.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:21.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:18:22.018 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:22.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:22.040 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:22.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:22.141 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:22.174 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:22.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:22.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:22.276 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:18:22.310 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:22.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:22.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:18:22.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:22 smithi181 ceph-mon[32701]: 6.b scrub starts 2024-09-11T08:18:22.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:22 smithi181 ceph-mon[32701]: 5.19 scrub starts 2024-09-11T08:18:22.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:22 smithi181 ceph-mon[32701]: 6.b scrub ok 2024-09-11T08:18:22.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:22 smithi181 ceph-mon[32701]: 5.19 scrub ok 2024-09-11T08:18:22.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:22 smithi181 ceph-mon[32701]: from='client.25283 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:18:22.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:22 smithi181 ceph-mon[32701]: pgmap v489: 148 pgs: 148 active+clean; 1.2 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 120 MiB/s wr, 1.92k op/s 2024-09-11T08:18:22.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:22 smithi181 ceph-mon[32701]: 4.1b scrub starts 2024-09-11T08:18:22.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:22 smithi181 ceph-mon[32701]: 4.1b scrub ok 2024-09-11T08:18:22.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:22 smithi181 ceph-mon[32701]: 4.19 scrub starts 2024-09-11T08:18:22.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:22 smithi181 ceph-mon[32701]: 4.19 scrub ok 2024-09-11T08:18:22.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:22 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:18:22.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:22 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:18:22.412 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:22.445 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:22.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:22.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:22.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:22 smithi150 ceph-mon[29276]: 6.b scrub starts 2024-09-11T08:18:22.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:22 smithi150 ceph-mon[29276]: 5.19 scrub starts 2024-09-11T08:18:22.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:22 smithi150 ceph-mon[29276]: 6.b scrub ok 2024-09-11T08:18:22.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:22 smithi150 ceph-mon[29276]: 5.19 scrub ok 2024-09-11T08:18:22.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:22 smithi150 ceph-mon[29276]: from='client.25283 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:18:22.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:22 smithi150 ceph-mon[29276]: pgmap v489: 148 pgs: 148 active+clean; 1.2 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 120 MiB/s wr, 1.92k op/s 2024-09-11T08:18:22.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:22 smithi150 ceph-mon[29276]: 4.1b scrub starts 2024-09-11T08:18:22.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:22 smithi150 ceph-mon[29276]: 4.1b scrub ok 2024-09-11T08:18:22.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:22 smithi150 ceph-mon[29276]: 4.19 scrub starts 2024-09-11T08:18:22.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:22 smithi150 ceph-mon[29276]: 4.19 scrub ok 2024-09-11T08:18:22.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:22 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:18:22.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:22 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:18:22.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:22 smithi150 ceph-mon[39612]: 6.b scrub starts 2024-09-11T08:18:22.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:22 smithi150 ceph-mon[39612]: 5.19 scrub starts 2024-09-11T08:18:22.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:22 smithi150 ceph-mon[39612]: 6.b scrub ok 2024-09-11T08:18:22.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:22 smithi150 ceph-mon[39612]: 5.19 scrub ok 2024-09-11T08:18:22.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:22 smithi150 ceph-mon[39612]: from='client.25283 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:18:22.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:22 smithi150 ceph-mon[39612]: pgmap v489: 148 pgs: 148 active+clean; 1.2 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 120 MiB/s wr, 1.92k op/s 2024-09-11T08:18:22.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:22 smithi150 ceph-mon[39612]: 4.1b scrub starts 2024-09-11T08:18:22.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:22 smithi150 ceph-mon[39612]: 4.1b scrub ok 2024-09-11T08:18:22.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:22 smithi150 ceph-mon[39612]: 4.19 scrub starts 2024-09-11T08:18:22.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:22 smithi150 ceph-mon[39612]: 4.19 scrub ok 2024-09-11T08:18:22.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:22 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:18:22.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:22 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:18:22.547 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:22.570 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-11T08:18:22.571 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-09-11T08:18:22.571 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-09-11T08:18:22.571 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-09-11T08:18:22.571 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.571 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24421.0:400 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e168)", 2024-09-11T08:18:22.571 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:14:23.790029+0000", 2024-09-11T08:18:22.571 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 238.77622490300001, 2024-09-11T08:18:22.571 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0036556700000001, 2024-09-11T08:18:22.571 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:18:22.571 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-11T08:18:22.571 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:18:22.571 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24421", 2024-09-11T08:18:22.572 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/3861404665", 2024-09-11T08:18:22.572 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 400 2024-09-11T08:18:22.572 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.572 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:18:22.572 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.572 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:18:22.572 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:23.790029+0000", 2024-09-11T08:18:22.572 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:22.572 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.572 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.572 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:18:22.572 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:23.790029+0000", 2024-09-11T08:18:22.572 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:22.573 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.573 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.573 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:18:22.573 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:23.790027+0000", 2024-09-11T08:18:22.573 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:18:22.573 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.573 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.573 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:18:22.573 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:23.790033+0000", 2024-09-11T08:18:22.573 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1120000000000001e-06 2024-09-11T08:18:22.573 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.573 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.574 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:18:22.574 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:23.790041+0000", 2024-09-11T08:18:22.574 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1029999999999996e-06 2024-09-11T08:18:22.574 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.574 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.574 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:18:22.574 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:23.790046+0000", 2024-09-11T08:18:22.574 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7389999999999996e-06 2024-09-11T08:18:22.574 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.574 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.575 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:22.575 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:23.827398+0000", 2024-09-11T08:18:22.575 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.037351729 2024-09-11T08:18:22.575 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.575 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.575 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:18:22.575 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:23.827575+0000", 2024-09-11T08:18:22.575 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017671700000000001 2024-09-11T08:18:22.575 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.575 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.576 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:22.576 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:24.793336+0000", 2024-09-11T08:18:22.576 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.96576121400000003 2024-09-11T08:18:22.576 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.576 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.576 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:18:22.576 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:24.793476+0000", 2024-09-11T08:18:22.576 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000140543 2024-09-11T08:18:22.577 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.577 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.577 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:18:22.577 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:24.793684+0000", 2024-09-11T08:18:22.577 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000208035 2024-09-11T08:18:22.577 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:22.577 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:18:22.577 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:22.578 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.578 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.578 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24320.0:1504 5.14 5:2b04a3e9:::notify.6:head [watch reconnect cookie 94783556106752 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-11T08:18:22.578 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:17:37.418092+0000", 2024-09-11T08:18:22.578 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 45.148162089000003, 2024-09-11T08:18:22.579 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.14172105099999999, 2024-09-11T08:18:22.579 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:18:22.579 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-11T08:18:22.579 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:18:22.579 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24320", 2024-09-11T08:18:22.579 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/1503848706", 2024-09-11T08:18:22.579 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 1504 2024-09-11T08:18:22.580 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.580 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:18:22.580 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.580 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:18:22.580 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:17:37.418092+0000", 2024-09-11T08:18:22.580 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:22.580 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.581 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.581 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:18:22.581 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:17:37.418092+0000", 2024-09-11T08:18:22.581 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:22.581 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.581 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.581 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:18:22.581 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:17:37.418090+0000", 2024-09-11T08:18:22.582 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999981 2024-09-11T08:18:22.582 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.582 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.582 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:18:22.582 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:17:37.418099+0000", 2024-09-11T08:18:22.582 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.679e-06 2024-09-11T08:18:22.582 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.582 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.583 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:18:22.583 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:17:37.418116+0000", 2024-09-11T08:18:22.583 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6592e-05 2024-09-11T08:18:22.583 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.583 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.583 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:18:22.583 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:17:37.418122+0000", 2024-09-11T08:18:22.584 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1750000000000002e-06 2024-09-11T08:18:22.584 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.584 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.584 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:22.584 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:17:37.418216+0000", 2024-09-11T08:18:22.584 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4239000000000004e-05 2024-09-11T08:18:22.584 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.584 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.584 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:18:22.585 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:17:37.418230+0000", 2024-09-11T08:18:22.585 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3704e-05 2024-09-11T08:18:22.585 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.585 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.585 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:22.585 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:17:37.558419+0000", 2024-09-11T08:18:22.585 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.14018851500000001 2024-09-11T08:18:22.585 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.585 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.586 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:18:22.586 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:17:37.558546+0000", 2024-09-11T08:18:22.586 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012696800000000001 2024-09-11T08:18:22.586 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.586 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.586 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:18:22.586 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:17:37.559813+0000", 2024-09-11T08:18:22.586 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001267053 2024-09-11T08:18:22.587 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:22.587 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:18:22.587 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:22.587 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.587 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.587 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:12197 9.3 9:c8b7ca77:::benchmark_data_smithi150_133616_object12196:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e251)", 2024-09-11T08:18:22.587 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:18:15.367478+0000", 2024-09-11T08:18:22.587 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 7.1987756320000003, 2024-09-11T08:18:22.587 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.13322205500000001, 2024-09-11T08:18:22.588 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:18:22.588 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:18:22.588 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:18:22.588 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:18:22.588 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:18:22.588 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 12197 2024-09-11T08:18:22.588 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.589 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:18:22.589 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.589 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:18:22.589 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.367478+0000", 2024-09-11T08:18:22.589 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:22.589 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.589 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.589 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:18:22.590 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.367478+0000", 2024-09-11T08:18:22.590 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:22.590 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.590 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.590 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:18:22.590 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.367477+0000", 2024-09-11T08:18:22.590 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:18:22.590 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.591 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.591 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:18:22.591 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.367553+0000", 2024-09-11T08:18:22.591 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6239e-05 2024-09-11T08:18:22.591 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.591 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.592 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:18:22.592 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.367566+0000", 2024-09-11T08:18:22.592 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3067e-05 2024-09-11T08:18:22.592 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.592 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.592 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:18:22.592 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.367571+0000", 2024-09-11T08:18:22.593 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0069999999999997e-06 2024-09-11T08:18:22.593 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.593 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.593 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:22.593 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.367617+0000", 2024-09-11T08:18:22.593 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6031e-05 2024-09-11T08:18:22.593 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.594 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.594 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:18:22.594 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.367744+0000", 2024-09-11T08:18:22.594 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012690299999999999 2024-09-11T08:18:22.594 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.594 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.594 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-09-11T08:18:22.594 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.367907+0000", 2024-09-11T08:18:22.594 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000163115 2024-09-11T08:18:22.595 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.595 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.595 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:18:22.595 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.368497+0000", 2024-09-11T08:18:22.595 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00058975 2024-09-11T08:18:22.595 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.595 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.595 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:22.595 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.370719+0000", 2024-09-11T08:18:22.595 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0022221490000000001 2024-09-11T08:18:22.596 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.596 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.596 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:22.596 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.500427+0000", 2024-09-11T08:18:22.596 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.12970715399999999 2024-09-11T08:18:22.596 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.596 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.596 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:18:22.596 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.500592+0000", 2024-09-11T08:18:22.596 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016540700000000001 2024-09-11T08:18:22.596 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.596 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.596 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:18:22.597 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.500700+0000", 2024-09-11T08:18:22.597 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010820899999999999 2024-09-11T08:18:22.597 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:22.597 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:18:22.597 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:22.597 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.597 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.597 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:12204 9.d 9:b0125059:::benchmark_data_smithi150_133616_object12203:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e251)", 2024-09-11T08:18:22.597 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:18:15.370768+0000", 2024-09-11T08:18:22.597 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 7.1954859730000003, 2024-09-11T08:18:22.597 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.13136710900000001, 2024-09-11T08:18:22.597 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:18:22.597 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:18:22.597 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:18:22.598 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:18:22.598 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:18:22.598 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 12204 2024-09-11T08:18:22.598 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.598 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:18:22.598 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.598 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:18:22.598 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.370768+0000", 2024-09-11T08:18:22.598 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:22.598 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.598 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.598 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:18:22.598 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.370768+0000", 2024-09-11T08:18:22.599 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:22.599 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.599 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.599 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:18:22.599 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.370767+0000", 2024-09-11T08:18:22.599 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:18:22.599 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.599 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.599 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:18:22.599 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.370834+0000", 2024-09-11T08:18:22.599 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7513000000000002e-05 2024-09-11T08:18:22.599 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.599 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.600 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:18:22.600 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.370844+0000", 2024-09-11T08:18:22.600 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9040000000000004e-06 2024-09-11T08:18:22.600 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.600 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.600 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:18:22.600 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.370850+0000", 2024-09-11T08:18:22.600 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5310000000000001e-06 2024-09-11T08:18:22.600 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.600 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.600 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:22.600 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.370888+0000", 2024-09-11T08:18:22.601 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8380000000000002e-05 2024-09-11T08:18:22.601 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.601 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.601 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:18:22.601 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.371016+0000", 2024-09-11T08:18:22.601 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000127766 2024-09-11T08:18:22.601 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.601 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.601 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,5", 2024-09-11T08:18:22.601 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.371160+0000", 2024-09-11T08:18:22.601 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000143948 2024-09-11T08:18:22.601 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.601 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.601 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:18:22.602 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.373193+0000", 2024-09-11T08:18:22.602 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002032758 2024-09-11T08:18:22.602 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.602 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.602 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:22.602 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.373229+0000", 2024-09-11T08:18:22.602 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6480999999999999e-05 2024-09-11T08:18:22.602 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.602 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.602 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:22.602 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.502039+0000", 2024-09-11T08:18:22.602 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.12881031100000001 2024-09-11T08:18:22.603 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.603 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.603 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:18:22.603 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.502094+0000", 2024-09-11T08:18:22.603 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4369000000000001e-05 2024-09-11T08:18:22.603 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.603 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.603 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:18:22.603 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.502135+0000", 2024-09-11T08:18:22.603 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1161000000000001e-05 2024-09-11T08:18:22.603 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:22.603 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:18:22.603 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:22.604 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.604 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.604 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:12205 9.3 9:c9d3892a:::benchmark_data_smithi150_133616_object12204:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e251)", 2024-09-11T08:18:22.604 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:18:15.370883+0000", 2024-09-11T08:18:22.604 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 7.1953707109999998, 2024-09-11T08:18:22.604 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.13124997199999999, 2024-09-11T08:18:22.604 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:18:22.604 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:18:22.604 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:18:22.604 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:18:22.604 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:18:22.604 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 12205 2024-09-11T08:18:22.604 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.605 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:18:22.605 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.605 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:18:22.605 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.370883+0000", 2024-09-11T08:18:22.605 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:22.605 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.605 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.605 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:18:22.605 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.370883+0000", 2024-09-11T08:18:22.605 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:22.606 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.606 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.606 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:18:22.606 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.370882+0000", 2024-09-11T08:18:22.606 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:18:22.606 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.606 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.606 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:18:22.606 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.370994+0000", 2024-09-11T08:18:22.606 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000112465 2024-09-11T08:18:22.606 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.606 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.607 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:18:22.607 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.371006+0000", 2024-09-11T08:18:22.607 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1257000000000001e-05 2024-09-11T08:18:22.607 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.607 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.607 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:18:22.607 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.371011+0000", 2024-09-11T08:18:22.607 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1909999999999999e-06 2024-09-11T08:18:22.607 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.607 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.607 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:22.607 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.371052+0000", 2024-09-11T08:18:22.607 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0732000000000002e-05 2024-09-11T08:18:22.608 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.608 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.608 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:18:22.608 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.371176+0000", 2024-09-11T08:18:22.608 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012435699999999999 2024-09-11T08:18:22.608 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.608 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.608 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-09-11T08:18:22.608 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.371297+0000", 2024-09-11T08:18:22.608 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012065000000000001 2024-09-11T08:18:22.608 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.608 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.608 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:18:22.609 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.372639+0000", 2024-09-11T08:18:22.609 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0013422639999999999 2024-09-11T08:18:22.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:22.610 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:22.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:22.611 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.611 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.611 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:22.612 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.373379+0000", 2024-09-11T08:18:22.612 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00074013799999999999 2024-09-11T08:18:22.612 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.612 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.612 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:22.612 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.502052+0000", 2024-09-11T08:18:22.612 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.12867291 2024-09-11T08:18:22.612 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.612 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.612 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:18:22.612 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.502098+0000", 2024-09-11T08:18:22.612 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5838999999999999e-05 2024-09-11T08:18:22.612 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.613 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.613 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:18:22.613 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.502133+0000", 2024-09-11T08:18:22.613 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5151999999999998e-05 2024-09-11T08:18:22.613 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:22.613 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:18:22.613 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:22.613 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.613 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.613 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:12211 9.d 9:bd9642fa:::benchmark_data_smithi150_133616_object12210:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e251)", 2024-09-11T08:18:22.613 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:18:15.371864+0000", 2024-09-11T08:18:22.613 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 7.1943899079999998, 2024-09-11T08:18:22.613 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.130683779, 2024-09-11T08:18:22.613 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:18:22.614 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:18:22.614 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:18:22.614 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:18:22.614 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:18:22.614 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 12211 2024-09-11T08:18:22.614 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.614 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:18:22.614 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.614 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:18:22.614 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.371864+0000", 2024-09-11T08:18:22.614 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:22.614 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.614 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.614 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:18:22.615 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.371864+0000", 2024-09-11T08:18:22.615 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:22.615 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.615 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.615 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:18:22.615 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.371863+0000", 2024-09-11T08:18:22.615 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:18:22.615 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.615 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.615 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:18:22.615 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.371905+0000", 2024-09-11T08:18:22.615 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2409e-05 2024-09-11T08:18:22.615 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.615 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.615 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:18:22.616 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.371923+0000", 2024-09-11T08:18:22.616 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7824000000000001e-05 2024-09-11T08:18:22.616 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.616 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.616 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:18:22.616 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.371929+0000", 2024-09-11T08:18:22.616 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5779999999999998e-06 2024-09-11T08:18:22.616 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.616 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.616 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:22.616 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.372592+0000", 2024-09-11T08:18:22.616 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00066330700000000002 2024-09-11T08:18:22.616 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.616 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.617 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:18:22.617 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.372712+0000", 2024-09-11T08:18:22.617 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000119949 2024-09-11T08:18:22.617 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.617 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.617 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,5", 2024-09-11T08:18:22.617 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.372798+0000", 2024-09-11T08:18:22.617 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5790000000000004e-05 2024-09-11T08:18:22.617 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.617 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.618 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:18:22.618 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.373321+0000", 2024-09-11T08:18:22.618 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00052284800000000002 2024-09-11T08:18:22.618 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.618 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.618 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:22.618 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.373582+0000", 2024-09-11T08:18:22.619 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000261771 2024-09-11T08:18:22.619 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.619 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.619 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:22.619 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.502463+0000", 2024-09-11T08:18:22.619 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.12888034500000001 2024-09-11T08:18:22.619 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.619 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.620 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:18:22.620 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.502514+0000", 2024-09-11T08:18:22.620 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0769999999999997e-05 2024-09-11T08:18:22.620 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.621 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.621 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:18:22.621 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.502548+0000", 2024-09-11T08:18:22.621 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4087000000000001e-05 2024-09-11T08:18:22.621 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:22.621 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:18:22.621 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:22.621 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.621 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.621 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:12212 9.3 9:c37256f5:::benchmark_data_smithi150_133616_object12211:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e251)", 2024-09-11T08:18:22.622 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:18:15.371942+0000", 2024-09-11T08:18:22.622 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 7.1943119319999997, 2024-09-11T08:18:22.622 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.130603104, 2024-09-11T08:18:22.622 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:18:22.622 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:18:22.622 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:18:22.622 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:18:22.622 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:18:22.622 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 12212 2024-09-11T08:18:22.622 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.622 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:18:22.622 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.623 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:18:22.623 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.371942+0000", 2024-09-11T08:18:22.623 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:22.623 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.623 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.623 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:18:22.623 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.371942+0000", 2024-09-11T08:18:22.623 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:22.623 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.623 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.623 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:18:22.623 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.371941+0000", 2024-09-11T08:18:22.623 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-09-11T08:18:22.623 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.624 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.624 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:18:22.624 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.371979+0000", 2024-09-11T08:18:22.624 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8021000000000001e-05 2024-09-11T08:18:22.624 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.624 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.624 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:18:22.624 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.371989+0000", 2024-09-11T08:18:22.625 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0104e-05 2024-09-11T08:18:22.625 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.625 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.625 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:18:22.625 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.371994+0000", 2024-09-11T08:18:22.625 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5249999999999999e-06 2024-09-11T08:18:22.626 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.626 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.626 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:22.626 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.372047+0000", 2024-09-11T08:18:22.626 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3458000000000002e-05 2024-09-11T08:18:22.626 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.627 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.627 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:18:22.627 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.372150+0000", 2024-09-11T08:18:22.627 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000103102 2024-09-11T08:18:22.627 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.627 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.627 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-09-11T08:18:22.628 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.372264+0000", 2024-09-11T08:18:22.628 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011408000000000001 2024-09-11T08:18:22.628 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.628 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.628 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:18:22.628 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.372815+0000", 2024-09-11T08:18:22.629 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00055091899999999997 2024-09-11T08:18:22.629 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.629 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.629 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:22.629 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.373493+0000", 2024-09-11T08:18:22.629 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00067724599999999995 2024-09-11T08:18:22.629 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.630 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.630 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:22.630 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.502457+0000", 2024-09-11T08:18:22.630 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.128964149 2024-09-11T08:18:22.630 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.630 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.631 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:18:22.631 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.502508+0000", 2024-09-11T08:18:22.631 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1158e-05 2024-09-11T08:18:22.631 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.631 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.631 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:18:22.631 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:15.502545+0000", 2024-09-11T08:18:22.632 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6928999999999998e-05 2024-09-11T08:18:22.632 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:22.632 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:18:22.632 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:22.632 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.632 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.632 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:16444 9.d 9:bc01d1c4:::benchmark_data_smithi150_133616_object16443:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e251)", 2024-09-11T08:18:22.632 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:18:16.922466+0000", 2024-09-11T08:18:22.632 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.6437880119999999, 2024-09-11T08:18:22.632 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.104795101, 2024-09-11T08:18:22.632 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:18:22.632 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:18:22.632 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:18:22.633 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:18:22.633 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:18:22.633 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 16444 2024-09-11T08:18:22.633 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.633 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:18:22.633 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.633 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:18:22.633 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.922466+0000", 2024-09-11T08:18:22.633 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:22.633 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.633 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.633 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:18:22.633 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.922466+0000", 2024-09-11T08:18:22.633 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:22.634 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.634 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.634 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:18:22.634 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.922465+0000", 2024-09-11T08:18:22.634 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:18:22.634 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.634 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.634 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:18:22.634 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.922514+0000", 2024-09-11T08:18:22.634 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9178999999999998e-05 2024-09-11T08:18:22.634 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.634 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.634 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:18:22.634 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.922531+0000", 2024-09-11T08:18:22.635 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7218e-05 2024-09-11T08:18:22.635 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.635 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.635 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:18:22.635 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.922538+0000", 2024-09-11T08:18:22.635 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0029999999999997e-06 2024-09-11T08:18:22.635 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.635 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.635 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:22.635 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.924589+0000", 2024-09-11T08:18:22.635 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002051537 2024-09-11T08:18:22.635 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.635 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.636 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:18:22.636 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.924706+0000", 2024-09-11T08:18:22.636 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000116024 2024-09-11T08:18:22.636 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.636 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.636 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,5", 2024-09-11T08:18:22.636 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.924829+0000", 2024-09-11T08:18:22.636 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000123008 2024-09-11T08:18:22.636 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.636 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.636 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:18:22.636 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.927711+0000", 2024-09-11T08:18:22.636 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0028822660000000001 2024-09-11T08:18:22.637 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.637 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.637 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:22.637 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.929792+0000", 2024-09-11T08:18:22.637 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0020807780000000001 2024-09-11T08:18:22.637 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.637 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.637 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:22.637 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:17.027181+0000", 2024-09-11T08:18:22.637 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.097389394000000004 2024-09-11T08:18:22.638 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.638 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.638 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:18:22.638 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:17.027228+0000", 2024-09-11T08:18:22.638 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6819e-05 2024-09-11T08:18:22.638 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.638 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.638 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:18:22.638 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:17.027261+0000", 2024-09-11T08:18:22.638 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3031000000000003e-05 2024-09-11T08:18:22.638 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:22.638 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:18:22.638 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:22.638 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.639 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.639 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:16445 9.d 9:b3fcf579:::benchmark_data_smithi150_133616_object16444:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e251)", 2024-09-11T08:18:22.639 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:18:16.922546+0000", 2024-09-11T08:18:22.639 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.6437076489999995, 2024-09-11T08:18:22.639 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.1048598, 2024-09-11T08:18:22.639 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:18:22.639 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:18:22.639 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:18:22.639 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:18:22.639 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:18:22.640 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 16445 2024-09-11T08:18:22.640 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.640 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:18:22.640 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.640 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:18:22.640 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.922546+0000", 2024-09-11T08:18:22.640 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:22.641 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.641 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.641 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:18:22.641 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.922546+0000", 2024-09-11T08:18:22.641 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:22.641 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.641 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.641 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:18:22.641 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.922545+0000", 2024-09-11T08:18:22.641 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-09-11T08:18:22.642 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.642 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.642 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:18:22.642 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.922571+0000", 2024-09-11T08:18:22.642 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5143e-05 2024-09-11T08:18:22.642 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.642 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.642 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:18:22.642 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.922577+0000", 2024-09-11T08:18:22.642 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1340000000000001e-06 2024-09-11T08:18:22.642 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.642 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.642 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:18:22.643 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.922581+0000", 2024-09-11T08:18:22.643 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.929e-06 2024-09-11T08:18:22.643 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.643 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.643 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:22.643 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.925530+0000", 2024-09-11T08:18:22.643 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0029497030000000001 2024-09-11T08:18:22.643 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.643 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.643 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:18:22.643 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.925631+0000", 2024-09-11T08:18:22.643 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010099600000000001 2024-09-11T08:18:22.644 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.644 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.644 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,5", 2024-09-11T08:18:22.644 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.925759+0000", 2024-09-11T08:18:22.644 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000127362 2024-09-11T08:18:22.644 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.644 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.644 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:18:22.644 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.927717+0000", 2024-09-11T08:18:22.644 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001957978 2024-09-11T08:18:22.644 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.644 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.644 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:22.645 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.929829+0000", 2024-09-11T08:18:22.645 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0021125150000000001 2024-09-11T08:18:22.645 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.645 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.645 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:22.645 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:17.027330+0000", 2024-09-11T08:18:22.645 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.097500733000000006 2024-09-11T08:18:22.645 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.645 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.645 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:18:22.645 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:17.027372+0000", 2024-09-11T08:18:22.645 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2225999999999999e-05 2024-09-11T08:18:22.645 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.646 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.646 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:18:22.646 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:17.027406+0000", 2024-09-11T08:18:22.646 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3670999999999997e-05 2024-09-11T08:18:22.646 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:22.646 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:18:22.646 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:22.646 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.646 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.646 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:16452 9.d 9:bb6e44c5:::benchmark_data_smithi150_133616_object16451:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e251)", 2024-09-11T08:18:22.646 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:18:16.922904+0000", 2024-09-11T08:18:22.646 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.6433492669999996, 2024-09-11T08:18:22.646 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.105407274, 2024-09-11T08:18:22.646 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:18:22.647 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:18:22.647 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:18:22.647 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:18:22.647 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:18:22.647 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 16452 2024-09-11T08:18:22.647 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.647 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:18:22.647 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.647 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:18:22.647 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.922904+0000", 2024-09-11T08:18:22.647 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:22.647 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.647 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.647 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:18:22.647 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.922904+0000", 2024-09-11T08:18:22.648 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:22.648 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.648 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.648 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:18:22.648 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.922904+0000", 2024-09-11T08:18:22.648 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-09-11T08:18:22.648 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.648 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.648 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:18:22.648 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.922951+0000", 2024-09-11T08:18:22.648 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7142999999999999e-05 2024-09-11T08:18:22.648 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.649 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.649 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:18:22.649 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.922958+0000", 2024-09-11T08:18:22.649 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2069999999999998e-06 2024-09-11T08:18:22.649 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.649 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.649 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:18:22.649 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.922997+0000", 2024-09-11T08:18:22.649 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8260999999999998e-05 2024-09-11T08:18:22.649 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.650 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.650 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:22.650 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.926330+0000", 2024-09-11T08:18:22.650 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003333632 2024-09-11T08:18:22.650 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.650 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.650 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:18:22.650 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.926423+0000", 2024-09-11T08:18:22.651 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.2915000000000001e-05 2024-09-11T08:18:22.652 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.652 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.652 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,5", 2024-09-11T08:18:22.652 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.926551+0000", 2024-09-11T08:18:22.652 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000127448 2024-09-11T08:18:22.652 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.652 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.652 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:18:22.652 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.927721+0000", 2024-09-11T08:18:22.652 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00117029 2024-09-11T08:18:22.652 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.652 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.653 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:22.653 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.930476+0000", 2024-09-11T08:18:22.653 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00275538 2024-09-11T08:18:22.653 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.653 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.653 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:22.653 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:17.028234+0000", 2024-09-11T08:18:22.653 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.097757436000000003 2024-09-11T08:18:22.653 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.653 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.653 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:18:22.653 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:17.028278+0000", 2024-09-11T08:18:22.654 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3915999999999998e-05 2024-09-11T08:18:22.654 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.654 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.654 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:18:22.654 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:17.028312+0000", 2024-09-11T08:18:22.654 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4195e-05 2024-09-11T08:18:22.654 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:22.654 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:18:22.654 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:22.654 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.654 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.654 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:16456 9.d 9:b24ea1cf:::benchmark_data_smithi150_133616_object16455:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e251)", 2024-09-11T08:18:22.654 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:18:16.924531+0000", 2024-09-11T08:18:22.655 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.6417231760000002, 2024-09-11T08:18:22.655 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.103898583, 2024-09-11T08:18:22.655 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:18:22.655 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:18:22.655 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:18:22.655 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:18:22.656 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:18:22.656 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 16456 2024-09-11T08:18:22.656 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.656 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:18:22.656 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.656 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:18:22.656 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.924531+0000", 2024-09-11T08:18:22.656 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:22.656 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.657 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.657 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:18:22.657 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.924531+0000", 2024-09-11T08:18:22.657 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:22.657 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.658 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.658 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:18:22.658 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.924529+0000", 2024-09-11T08:18:22.658 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:18:22.658 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.659 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.659 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:18:22.659 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.924561+0000", 2024-09-11T08:18:22.659 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1523999999999998e-05 2024-09-11T08:18:22.659 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.659 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.659 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:18:22.659 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.924572+0000", 2024-09-11T08:18:22.660 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1260999999999999e-05 2024-09-11T08:18:22.660 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.660 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.660 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:18:22.660 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.924578+0000", 2024-09-11T08:18:22.660 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2890000000000003e-06 2024-09-11T08:18:22.661 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.661 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.661 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:22.661 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.927744+0000", 2024-09-11T08:18:22.661 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003165799 2024-09-11T08:18:22.661 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.661 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.661 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:18:22.661 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.927857+0000", 2024-09-11T08:18:22.661 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000112626 2024-09-11T08:18:22.661 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.661 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.661 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,5", 2024-09-11T08:18:22.662 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.927971+0000", 2024-09-11T08:18:22.662 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011407 2024-09-11T08:18:22.662 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.662 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.662 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:18:22.662 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.929798+0000", 2024-09-11T08:18:22.662 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0018272710000000001 2024-09-11T08:18:22.662 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.662 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.662 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:22.662 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.930504+0000", 2024-09-11T08:18:22.662 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00070602299999999996 2024-09-11T08:18:22.662 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.662 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.663 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:22.663 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:17.028358+0000", 2024-09-11T08:18:22.663 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.097854021999999999 2024-09-11T08:18:22.663 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.663 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.663 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:18:22.663 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:17.028398+0000", 2024-09-11T08:18:22.663 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9471999999999997e-05 2024-09-11T08:18:22.663 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.663 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.664 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:18:22.664 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:17.028429+0000", 2024-09-11T08:18:22.664 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1440999999999999e-05 2024-09-11T08:18:22.664 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:22.664 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:18:22.664 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:22.664 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.664 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.664 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:16458 9.e 9:73c005e4:::benchmark_data_smithi150_133616_object16457:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e251)", 2024-09-11T08:18:22.664 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:18:16.924586+0000", 2024-09-11T08:18:22.664 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.6416675730000003, 2024-09-11T08:18:22.665 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.099667811999999995, 2024-09-11T08:18:22.665 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:18:22.665 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:18:22.665 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:18:22.665 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:18:22.665 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:18:22.665 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 16458 2024-09-11T08:18:22.665 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.665 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:18:22.665 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.665 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:18:22.665 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.924586+0000", 2024-09-11T08:18:22.666 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:22.666 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.666 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.666 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:18:22.666 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.924586+0000", 2024-09-11T08:18:22.666 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:22.666 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.666 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.666 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:18:22.666 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.924586+0000", 2024-09-11T08:18:22.666 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-09-11T08:18:22.666 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.666 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.666 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:18:22.667 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.924614+0000", 2024-09-11T08:18:22.667 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8053999999999999e-05 2024-09-11T08:18:22.667 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.667 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.667 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:18:22.667 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.924621+0000", 2024-09-11T08:18:22.667 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2189999999999998e-06 2024-09-11T08:18:22.667 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.667 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.667 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:18:22.667 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.924626+0000", 2024-09-11T08:18:22.667 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6249999999999998e-06 2024-09-11T08:18:22.667 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.667 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.668 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:22.668 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.924955+0000", 2024-09-11T08:18:22.668 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00032996000000000002 2024-09-11T08:18:22.668 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.668 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.668 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:18:22.668 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.925077+0000", 2024-09-11T08:18:22.668 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012102799999999999 2024-09-11T08:18:22.668 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.668 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.669 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,4", 2024-09-11T08:18:22.669 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.925197+0000", 2024-09-11T08:18:22.669 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012089100000000001 2024-09-11T08:18:22.669 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.669 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.669 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:18:22.670 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.926361+0000", 2024-09-11T08:18:22.670 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0011640730000000001 2024-09-11T08:18:22.670 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.670 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.670 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:22.670 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.927240+0000", 2024-09-11T08:18:22.670 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00087828899999999998 2024-09-11T08:18:22.670 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.670 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.670 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:22.670 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:17.024154+0000", 2024-09-11T08:18:22.671 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.096914407999999994 2024-09-11T08:18:22.671 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.671 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.671 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:18:22.671 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:17.024210+0000", 2024-09-11T08:18:22.671 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5495000000000003e-05 2024-09-11T08:18:22.671 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.671 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.671 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:18:22.671 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:17.024254+0000", 2024-09-11T08:18:22.671 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4277999999999999e-05 2024-09-11T08:18:22.671 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:22.671 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:18:22.672 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:22.672 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.672 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.672 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:16467 9.d 9:b81afdc2:::benchmark_data_smithi150_133616_object16466:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e251)", 2024-09-11T08:18:22.672 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:18:16.928497+0000", 2024-09-11T08:18:22.672 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.6377567580000001, 2024-09-11T08:18:22.672 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100148178, 2024-09-11T08:18:22.672 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:18:22.672 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:18:22.672 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:18:22.672 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:18:22.672 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:18:22.672 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 16467 2024-09-11T08:18:22.673 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.673 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:18:22.673 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.673 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:18:22.673 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.928497+0000", 2024-09-11T08:18:22.673 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:22.673 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.673 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.673 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:18:22.673 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.928497+0000", 2024-09-11T08:18:22.673 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:22.674 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.674 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.674 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:18:22.674 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.928496+0000", 2024-09-11T08:18:22.674 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:18:22.674 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.675 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.675 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:18:22.675 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.928524+0000", 2024-09-11T08:18:22.675 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7781999999999999e-05 2024-09-11T08:18:22.675 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.675 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.675 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:18:22.676 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.928533+0000", 2024-09-11T08:18:22.676 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.662e-06 2024-09-11T08:18:22.676 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.676 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.676 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:18:22.676 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.928540+0000", 2024-09-11T08:18:22.676 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0949999999999998e-06 2024-09-11T08:18:22.676 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.676 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.676 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:22.676 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.928583+0000", 2024-09-11T08:18:22.676 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2765000000000001e-05 2024-09-11T08:18:22.676 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.676 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.677 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:18:22.677 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.928685+0000", 2024-09-11T08:18:22.677 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010180599999999999 2024-09-11T08:18:22.677 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.677 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.677 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,5", 2024-09-11T08:18:22.677 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.928801+0000", 2024-09-11T08:18:22.677 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000115918 2024-09-11T08:18:22.677 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.677 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.677 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:18:22.677 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.929803+0000", 2024-09-11T08:18:22.677 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001001825 2024-09-11T08:18:22.678 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.678 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.678 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:22.678 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.930710+0000", 2024-09-11T08:18:22.678 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00090773999999999998 2024-09-11T08:18:22.678 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.678 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.678 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:22.678 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:17.028552+0000", 2024-09-11T08:18:22.678 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.097841845999999996 2024-09-11T08:18:22.678 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.679 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.679 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:18:22.679 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:17.028609+0000", 2024-09-11T08:18:22.679 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6496000000000003e-05 2024-09-11T08:18:22.679 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.679 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.679 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:18:22.680 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:17.028645+0000", 2024-09-11T08:18:22.680 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6467999999999997e-05 2024-09-11T08:18:22.680 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:22.680 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:18:22.680 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:22.680 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.681 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.681 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:16479 9.d 9:ba01daac:::benchmark_data_smithi150_133616_object16478:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e251)", 2024-09-11T08:18:22.681 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:18:16.932961+0000", 2024-09-11T08:18:22.681 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.6332929060000003, 2024-09-11T08:18:22.681 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.097446579000000005, 2024-09-11T08:18:22.681 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:18:22.681 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:18:22.682 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:18:22.682 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:18:22.682 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:18:22.682 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 16479 2024-09-11T08:18:22.682 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.682 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:18:22.683 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.683 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:18:22.683 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.932961+0000", 2024-09-11T08:18:22.683 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:22.683 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.683 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.684 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:18:22.684 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.932961+0000", 2024-09-11T08:18:22.684 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:22.684 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.684 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.684 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:18:22.685 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.932960+0000", 2024-09-11T08:18:22.685 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-09-11T08:18:22.685 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.685 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.685 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:18:22.685 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.933055+0000", 2024-09-11T08:18:22.686 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.5289000000000002e-05 2024-09-11T08:18:22.686 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.686 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.686 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:18:22.686 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.933067+0000", 2024-09-11T08:18:22.686 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1469e-05 2024-09-11T08:18:22.687 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.687 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.687 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:18:22.687 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.933072+0000", 2024-09-11T08:18:22.687 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4199999999999998e-06 2024-09-11T08:18:22.688 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.688 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.688 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:22.688 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.933562+0000", 2024-09-11T08:18:22.688 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00048974700000000001 2024-09-11T08:18:22.688 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.688 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.689 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:18:22.689 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.933681+0000", 2024-09-11T08:18:22.689 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000118946 2024-09-11T08:18:22.689 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.689 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.689 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,5", 2024-09-11T08:18:22.689 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.933812+0000", 2024-09-11T08:18:22.690 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000131209 2024-09-11T08:18:22.690 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.690 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.690 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:18:22.690 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.934388+0000", 2024-09-11T08:18:22.691 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00057602200000000004 2024-09-11T08:18:22.691 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.691 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.691 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:22.691 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:16.934526+0000", 2024-09-11T08:18:22.691 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013814900000000001 2024-09-11T08:18:22.692 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.692 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.692 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:22.692 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:17.029531+0000", 2024-09-11T08:18:22.692 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.095004558000000003 2024-09-11T08:18:22.692 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.692 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.692 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:18:22.692 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:17.029668+0000", 2024-09-11T08:18:22.693 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000137 2024-09-11T08:18:22.693 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.693 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.693 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:18:22.693 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:17.030407+0000", 2024-09-11T08:18:22.694 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00073947699999999999 2024-09-11T08:18:22.694 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:22.694 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:18:22.694 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:22.694 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.694 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.694 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:22571 9.e 9:7e6ad66b:::benchmark_data_smithi150_133616_object22570:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e252)", 2024-09-11T08:18:22.694 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:18:19.413620+0000", 2024-09-11T08:18:22.694 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 3.152633244, 2024-09-11T08:18:22.694 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.15670378099999999, 2024-09-11T08:18:22.695 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:18:22.695 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:18:22.695 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:18:22.695 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:18:22.695 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:18:22.695 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 22571 2024-09-11T08:18:22.695 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.695 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:18:22.695 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.695 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:18:22.696 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.413620+0000", 2024-09-11T08:18:22.696 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:22.696 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.696 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.696 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:18:22.696 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.413620+0000", 2024-09-11T08:18:22.696 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:22.696 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.696 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.697 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:18:22.697 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.413619+0000", 2024-09-11T08:18:22.697 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:18:22.697 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.697 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.697 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:18:22.697 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.413705+0000", 2024-09-11T08:18:22.698 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6098000000000001e-05 2024-09-11T08:18:22.698 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.698 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.698 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:18:22.698 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.413718+0000", 2024-09-11T08:18:22.698 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3324e-05 2024-09-11T08:18:22.698 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.698 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.698 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:18:22.698 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.413725+0000", 2024-09-11T08:18:22.698 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2929999999999997e-06 2024-09-11T08:18:22.698 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.698 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.699 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:22.699 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.413782+0000", 2024-09-11T08:18:22.699 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7216000000000002e-05 2024-09-11T08:18:22.699 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.699 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.699 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:18:22.699 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.413998+0000", 2024-09-11T08:18:22.699 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00021641100000000001 2024-09-11T08:18:22.699 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.699 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.699 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,4", 2024-09-11T08:18:22.699 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.414139+0000", 2024-09-11T08:18:22.699 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014046299999999999 2024-09-11T08:18:22.700 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.700 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.700 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:18:22.700 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.415504+0000", 2024-09-11T08:18:22.700 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001365694 2024-09-11T08:18:22.700 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.700 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.700 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:22.700 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.418657+0000", 2024-09-11T08:18:22.700 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0031527249999999999 2024-09-11T08:18:22.700 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.700 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.700 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:22.701 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.570261+0000", 2024-09-11T08:18:22.701 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.15160364100000001 2024-09-11T08:18:22.701 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.701 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.701 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:18:22.701 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.570302+0000", 2024-09-11T08:18:22.701 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0738000000000002e-05 2024-09-11T08:18:22.701 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.701 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.701 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:18:22.701 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.570324+0000", 2024-09-11T08:18:22.701 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2659000000000001e-05 2024-09-11T08:18:22.701 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:22.701 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:18:22.702 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:22.702 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.702 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.702 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:22594 9.e 9:70755779:::benchmark_data_smithi150_133616_object22593:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e252)", 2024-09-11T08:18:22.702 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:18:19.421390+0000", 2024-09-11T08:18:22.702 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 3.1448635469999999, 2024-09-11T08:18:22.702 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.14909266500000001, 2024-09-11T08:18:22.702 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:18:22.702 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:18:22.702 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:18:22.702 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:18:22.703 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:18:22.703 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 22594 2024-09-11T08:18:22.703 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.703 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:18:22.703 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.703 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:18:22.703 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.421390+0000", 2024-09-11T08:18:22.703 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:22.703 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.703 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.703 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:18:22.703 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.421390+0000", 2024-09-11T08:18:22.703 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:22.703 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.704 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.704 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:18:22.704 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.421389+0000", 2024-09-11T08:18:22.704 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:18:22.704 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.704 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.704 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:18:22.704 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.421498+0000", 2024-09-11T08:18:22.704 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000108793 2024-09-11T08:18:22.704 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.704 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.704 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:18:22.704 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.421508+0000", 2024-09-11T08:18:22.705 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0159999999999999e-05 2024-09-11T08:18:22.705 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.705 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.705 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:18:22.705 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.421514+0000", 2024-09-11T08:18:22.705 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7960000000000001e-06 2024-09-11T08:18:22.705 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.705 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.705 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:22.705 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.421559+0000", 2024-09-11T08:18:22.705 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5238999999999998e-05 2024-09-11T08:18:22.705 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.705 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.705 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:18:22.706 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.422169+0000", 2024-09-11T08:18:22.706 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00060995199999999998 2024-09-11T08:18:22.706 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.706 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.706 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,4", 2024-09-11T08:18:22.706 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.422294+0000", 2024-09-11T08:18:22.706 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000124683 2024-09-11T08:18:22.706 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.706 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.706 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:18:22.706 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.424350+0000", 2024-09-11T08:18:22.707 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0020565660000000001 2024-09-11T08:18:22.707 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.707 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.707 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:22.707 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.426301+0000", 2024-09-11T08:18:22.707 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0019502930000000001 2024-09-11T08:18:22.707 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.707 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.707 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:22.707 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.570374+0000", 2024-09-11T08:18:22.707 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.144073283 2024-09-11T08:18:22.707 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.707 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.707 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:18:22.708 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.570423+0000", 2024-09-11T08:18:22.708 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9018999999999999e-05 2024-09-11T08:18:22.708 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.708 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.708 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:18:22.708 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.570483+0000", 2024-09-11T08:18:22.708 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9812000000000003e-05 2024-09-11T08:18:22.708 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:22.708 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:18:22.708 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:22.708 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.708 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.708 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:22606 9.e 9:745fd589:::benchmark_data_smithi150_133616_object22605:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e252)", 2024-09-11T08:18:22.708 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:18:19.426649+0000", 2024-09-11T08:18:22.709 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 3.1396046989999999, 2024-09-11T08:18:22.709 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.14399266199999999, 2024-09-11T08:18:22.709 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:18:22.709 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:18:22.709 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:18:22.709 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:18:22.709 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:18:22.709 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 22606 2024-09-11T08:18:22.709 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.709 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:18:22.709 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.709 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:18:22.709 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.426649+0000", 2024-09-11T08:18:22.709 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:22.710 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.710 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.710 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:18:22.710 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.426649+0000", 2024-09-11T08:18:22.710 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:22.710 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.710 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.710 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:18:22.710 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.426648+0000", 2024-09-11T08:18:22.710 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:18:22.710 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.710 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.710 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:18:22.710 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.426839+0000", 2024-09-11T08:18:22.711 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000191416 2024-09-11T08:18:22.711 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.711 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.711 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:18:22.711 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.426850+0000", 2024-09-11T08:18:22.711 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1182999999999999e-05 2024-09-11T08:18:22.711 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.711 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.711 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:18:22.711 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.426856+0000", 2024-09-11T08:18:22.711 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8100000000000003e-06 2024-09-11T08:18:22.711 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.711 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.712 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:22.712 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.426899+0000", 2024-09-11T08:18:22.712 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2375e-05 2024-09-11T08:18:22.712 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.712 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.712 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:18:22.712 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.427003+0000", 2024-09-11T08:18:22.712 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010464299999999999 2024-09-11T08:18:22.712 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.712 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.712 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,4", 2024-09-11T08:18:22.712 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.427863+0000", 2024-09-11T08:18:22.712 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00086016300000000005 2024-09-11T08:18:22.712 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.713 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.713 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:18:22.713 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.429820+0000", 2024-09-11T08:18:22.713 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0019562540000000002 2024-09-11T08:18:22.713 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.713 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.713 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:22.713 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.433171+0000", 2024-09-11T08:18:22.713 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0033519320000000002 2024-09-11T08:18:22.713 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.713 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.713 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:22.713 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.570532+0000", 2024-09-11T08:18:22.713 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.13736082399999999 2024-09-11T08:18:22.714 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.714 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.714 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:18:22.714 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.570606+0000", 2024-09-11T08:18:22.714 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.3650000000000001e-05 2024-09-11T08:18:22.714 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.714 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.714 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:18:22.714 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.570642+0000", 2024-09-11T08:18:22.714 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5707999999999998e-05 2024-09-11T08:18:22.714 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:22.714 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:18:22.714 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:22.714 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.714 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.715 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:22646 9.e 9:78974891:::benchmark_data_smithi150_133616_object22645:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e252)", 2024-09-11T08:18:22.715 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:18:19.444265+0000", 2024-09-11T08:18:22.715 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 3.121988945, 2024-09-11T08:18:22.715 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.12649845100000001, 2024-09-11T08:18:22.715 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:18:22.715 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:18:22.715 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:18:22.715 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:18:22.715 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:18:22.715 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 22646 2024-09-11T08:18:22.715 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.715 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:18:22.715 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.715 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:18:22.716 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.444265+0000", 2024-09-11T08:18:22.716 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:22.716 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.716 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.716 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:18:22.716 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.444265+0000", 2024-09-11T08:18:22.716 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:22.716 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.716 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.716 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:18:22.716 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.444264+0000", 2024-09-11T08:18:22.716 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:18:22.716 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.716 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.717 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:18:22.717 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.444301+0000", 2024-09-11T08:18:22.717 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7249999999999997e-05 2024-09-11T08:18:22.717 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.717 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.717 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:18:22.717 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.444309+0000", 2024-09-11T08:18:22.717 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5620000000000004e-06 2024-09-11T08:18:22.717 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.717 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.717 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:18:22.717 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.444314+0000", 2024-09-11T08:18:22.717 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7729999999999999e-06 2024-09-11T08:18:22.717 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.718 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.718 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:22.718 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.444375+0000", 2024-09-11T08:18:22.718 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1054000000000002e-05 2024-09-11T08:18:22.718 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.718 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.718 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:18:22.718 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.444514+0000", 2024-09-11T08:18:22.718 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013890199999999999 2024-09-11T08:18:22.718 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.718 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.718 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,4", 2024-09-11T08:18:22.718 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.444638+0000", 2024-09-11T08:18:22.719 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012376800000000001 2024-09-11T08:18:22.719 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.719 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.719 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:18:22.719 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.445580+0000", 2024-09-11T08:18:22.719 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00094214699999999995 2024-09-11T08:18:22.719 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.719 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.720 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:22.720 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.445870+0000", 2024-09-11T08:18:22.720 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00028957899999999999 2024-09-11T08:18:22.720 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.720 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.720 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:22.720 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.570695+0000", 2024-09-11T08:18:22.720 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.12482491599999999 2024-09-11T08:18:22.721 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.721 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.721 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:18:22.721 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.570744+0000", 2024-09-11T08:18:22.721 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9135000000000003e-05 2024-09-11T08:18:22.721 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.721 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.722 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:18:22.722 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.570763+0000", 2024-09-11T08:18:22.722 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9565999999999999e-05 2024-09-11T08:18:22.722 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:22.722 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:18:22.722 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:22.722 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.722 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.722 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:26035 9.d 9:be425f7c:::benchmark_data_smithi150_133616_object26034:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e252)", 2024-09-11T08:18:22.723 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:18:21.255273+0000", 2024-09-11T08:18:22.723 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 1.310980249, 2024-09-11T08:18:22.723 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.13714338000000001, 2024-09-11T08:18:22.723 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:18:22.723 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:18:22.723 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:18:22.723 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:18:22.723 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:18:22.723 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 26035 2024-09-11T08:18:22.723 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.723 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:18:22.723 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.723 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:18:22.723 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:21.255273+0000", 2024-09-11T08:18:22.724 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:22.724 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.724 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.724 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:18:22.724 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:21.255273+0000", 2024-09-11T08:18:22.724 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:22.724 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.724 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.725 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:18:22.725 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:21.255272+0000", 2024-09-11T08:18:22.725 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:18:22.725 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.725 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.726 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:18:22.726 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:21.255893+0000", 2024-09-11T08:18:22.726 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00062122800000000004 2024-09-11T08:18:22.726 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.726 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.727 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:18:22.727 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:21.255907+0000", 2024-09-11T08:18:22.727 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3274e-05 2024-09-11T08:18:22.727 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.727 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.727 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:18:22.727 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:21.255913+0000", 2024-09-11T08:18:22.727 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5780000000000004e-06 2024-09-11T08:18:22.727 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.728 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.728 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:22.728 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:21.256005+0000", 2024-09-11T08:18:22.728 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.1520999999999993e-05 2024-09-11T08:18:22.728 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.728 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.729 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:18:22.729 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:21.256148+0000", 2024-09-11T08:18:22.729 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014335800000000001 2024-09-11T08:18:22.729 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.729 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.729 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,5", 2024-09-11T08:18:22.729 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:21.256278+0000", 2024-09-11T08:18:22.729 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012948199999999999 2024-09-11T08:18:22.729 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.729 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.729 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:18:22.729 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:21.258454+0000", 2024-09-11T08:18:22.729 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002176449 2024-09-11T08:18:22.730 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.730 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.730 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:22.730 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:21.259075+0000", 2024-09-11T08:18:22.730 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00062095499999999997 2024-09-11T08:18:22.730 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.730 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.730 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:22.730 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:21.392357+0000", 2024-09-11T08:18:22.730 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.13328161299999999 2024-09-11T08:18:22.730 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.730 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.730 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:18:22.730 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:21.392393+0000", 2024-09-11T08:18:22.731 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6066000000000003e-05 2024-09-11T08:18:22.731 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.731 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.731 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:18:22.731 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:21.392417+0000", 2024-09-11T08:18:22.731 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.421e-05 2024-09-11T08:18:22.731 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:22.731 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:18:22.731 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:22.731 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.731 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.731 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:26041 9.d 9:bcb08d0f:::benchmark_data_smithi150_133616_object26040:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e252)", 2024-09-11T08:18:22.731 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:18:21.257000+0000", 2024-09-11T08:18:22.731 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 1.309253424, 2024-09-11T08:18:22.731 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.135512138, 2024-09-11T08:18:22.732 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:18:22.732 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:18:22.732 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:18:22.732 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:18:22.732 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:18:22.732 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 26041 2024-09-11T08:18:22.732 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.732 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:18:22.732 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.732 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:18:22.732 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:21.257000+0000", 2024-09-11T08:18:22.733 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:22.733 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.733 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.733 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:18:22.733 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:21.257000+0000", 2024-09-11T08:18:22.734 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:18:22.734 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.734 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.734 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:18:22.734 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:21.256999+0000", 2024-09-11T08:18:22.734 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:18:22.735 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.735 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.735 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:18:22.735 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:21.257062+0000", 2024-09-11T08:18:22.735 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2726999999999998e-05 2024-09-11T08:18:22.735 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.735 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.735 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:18:22.735 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:21.257077+0000", 2024-09-11T08:18:22.736 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5271999999999999e-05 2024-09-11T08:18:22.736 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.736 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.736 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:18:22.736 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:21.257083+0000", 2024-09-11T08:18:22.736 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.801e-06 2024-09-11T08:18:22.736 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.736 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.736 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:18:22.736 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:21.257347+0000", 2024-09-11T08:18:22.736 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000264205 2024-09-11T08:18:22.736 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.736 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.736 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:18:22.737 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:21.257468+0000", 2024-09-11T08:18:22.737 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000121474 2024-09-11T08:18:22.737 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.737 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.737 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,5", 2024-09-11T08:18:22.737 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:21.257598+0000", 2024-09-11T08:18:22.737 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013011899999999999 2024-09-11T08:18:22.737 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.737 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.737 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:18:22.737 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:21.258542+0000", 2024-09-11T08:18:22.737 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00094373299999999999 2024-09-11T08:18:22.737 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.738 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.738 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:22.738 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:21.259194+0000", 2024-09-11T08:18:22.738 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00065168400000000003 2024-09-11T08:18:22.738 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.738 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.738 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:18:22.738 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:21.392458+0000", 2024-09-11T08:18:22.738 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.13326395099999999 2024-09-11T08:18:22.738 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.738 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.738 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:18:22.739 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:21.392490+0000", 2024-09-11T08:18:22.739 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2146e-05 2024-09-11T08:18:22.739 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:18:22.739 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:18:22.739 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:18:22.739 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:21.392512+0000", 2024-09-11T08:18:22.739 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2442999999999998e-05 2024-09-11T08:18:22.739 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:22.739 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:18:22.739 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:22.739 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:18:22.739 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:18:22.739 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-11T08:18:22.740 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:18:22.765 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 11 15 28297 28282 160.669 94.4375 0.00328046 0.00621829 2024-09-11T08:18:22.777 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:22.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:22.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:18:22.879 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:22.908 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:22.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:22.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:23.010 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:23.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:23.044 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:23.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:23.145 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:23.178 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:23.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:23.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:23.280 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:23.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:23.302 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:23.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:23.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:23 smithi181 ceph-mon[32701]: 3.5 scrub starts 2024-09-11T08:18:23.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:23 smithi181 ceph-mon[32701]: 3.5 scrub ok 2024-09-11T08:18:23.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:23 smithi181 ceph-mon[32701]: 6.1 scrub starts 2024-09-11T08:18:23.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:23 smithi181 ceph-mon[32701]: 6.1 scrub ok 2024-09-11T08:18:23.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:23 smithi181 ceph-mon[32701]: 4.18 scrub starts 2024-09-11T08:18:23.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:23 smithi181 ceph-mon[32701]: 4.18 scrub ok 2024-09-11T08:18:23.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:23 smithi181 ceph-mon[32701]: 5.11 scrub starts 2024-09-11T08:18:23.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:23 smithi181 ceph-mon[32701]: 5.11 scrub ok 2024-09-11T08:18:23.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:23 smithi181 ceph-mon[32701]: 6.14 scrub starts 2024-09-11T08:18:23.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:23 smithi181 ceph-mon[32701]: 6.14 scrub ok 2024-09-11T08:18:23.402 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:23.432 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:23.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:23.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:23.480 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:23 smithi150 ceph-mon[29276]: 3.5 scrub starts 2024-09-11T08:18:23.480 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:23 smithi150 ceph-mon[29276]: 3.5 scrub ok 2024-09-11T08:18:23.480 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:23 smithi150 ceph-mon[29276]: 6.1 scrub starts 2024-09-11T08:18:23.480 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:23 smithi150 ceph-mon[29276]: 6.1 scrub ok 2024-09-11T08:18:23.481 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:23 smithi150 ceph-mon[29276]: 4.18 scrub starts 2024-09-11T08:18:23.481 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:23 smithi150 ceph-mon[29276]: 4.18 scrub ok 2024-09-11T08:18:23.481 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:23 smithi150 ceph-mon[29276]: 5.11 scrub starts 2024-09-11T08:18:23.481 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:23 smithi150 ceph-mon[29276]: 5.11 scrub ok 2024-09-11T08:18:23.481 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:23 smithi150 ceph-mon[29276]: 6.14 scrub starts 2024-09-11T08:18:23.481 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:23 smithi150 ceph-mon[29276]: 6.14 scrub ok 2024-09-11T08:18:23.481 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:23 smithi150 ceph-mon[39612]: 3.5 scrub starts 2024-09-11T08:18:23.482 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:23 smithi150 ceph-mon[39612]: 3.5 scrub ok 2024-09-11T08:18:23.482 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:23 smithi150 ceph-mon[39612]: 6.1 scrub starts 2024-09-11T08:18:23.482 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:23 smithi150 ceph-mon[39612]: 6.1 scrub ok 2024-09-11T08:18:23.482 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:23 smithi150 ceph-mon[39612]: 4.18 scrub starts 2024-09-11T08:18:23.482 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:23 smithi150 ceph-mon[39612]: 4.18 scrub ok 2024-09-11T08:18:23.482 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:23 smithi150 ceph-mon[39612]: 5.11 scrub starts 2024-09-11T08:18:23.482 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:23 smithi150 ceph-mon[39612]: 5.11 scrub ok 2024-09-11T08:18:23.482 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:23 smithi150 ceph-mon[39612]: 6.14 scrub starts 2024-09-11T08:18:23.482 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:23 smithi150 ceph-mon[39612]: 6.14 scrub ok 2024-09-11T08:18:23.534 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:23.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:23.571 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:23.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:23.672 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:18:23.709 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:23.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:23.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:18:23.765 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 12 16 30095 30079 156.638 112.312 0.00595704 0.00637746 2024-09-11T08:18:23.810 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:23.846 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-11T08:18:23.853 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:23.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:23.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:23.955 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:18:23.992 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:23.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:23.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:18:24.094 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:24.116 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:24.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:24.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:24.218 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:24.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:24 smithi150 ceph-mon[29276]: 3.16 scrub starts 2024-09-11T08:18:24.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:24 smithi150 ceph-mon[29276]: 3.16 scrub ok 2024-09-11T08:18:24.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:24 smithi150 ceph-mon[29276]: pgmap v490: 148 pgs: 148 active+clean; 1.4 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 131 MiB/s wr, 2.10k op/s 2024-09-11T08:18:24.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:24 smithi150 ceph-mon[29276]: 3.7 scrub starts 2024-09-11T08:18:24.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:24 smithi150 ceph-mon[29276]: 3.7 scrub ok 2024-09-11T08:18:24.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:24 smithi150 ceph-mon[29276]: 4.1c scrub starts 2024-09-11T08:18:24.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:24 smithi150 ceph-mon[29276]: 4.1c scrub ok 2024-09-11T08:18:24.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:24 smithi150 ceph-mon[39612]: 3.16 scrub starts 2024-09-11T08:18:24.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:24 smithi150 ceph-mon[39612]: 3.16 scrub ok 2024-09-11T08:18:24.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:24 smithi150 ceph-mon[39612]: pgmap v490: 148 pgs: 148 active+clean; 1.4 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 131 MiB/s wr, 2.10k op/s 2024-09-11T08:18:24.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:24 smithi150 ceph-mon[39612]: 3.7 scrub starts 2024-09-11T08:18:24.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:24 smithi150 ceph-mon[39612]: 3.7 scrub ok 2024-09-11T08:18:24.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:24 smithi150 ceph-mon[39612]: 4.1c scrub starts 2024-09-11T08:18:24.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:24 smithi150 ceph-mon[39612]: 4.1c scrub ok 2024-09-11T08:18:24.251 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:24.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:24.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:24.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:24 smithi181 ceph-mon[32701]: 3.16 scrub starts 2024-09-11T08:18:24.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:24 smithi181 ceph-mon[32701]: 3.16 scrub ok 2024-09-11T08:18:24.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:24 smithi181 ceph-mon[32701]: pgmap v490: 148 pgs: 148 active+clean; 1.4 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 131 MiB/s wr, 2.10k op/s 2024-09-11T08:18:24.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:24 smithi181 ceph-mon[32701]: 3.7 scrub starts 2024-09-11T08:18:24.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:24 smithi181 ceph-mon[32701]: 3.7 scrub ok 2024-09-11T08:18:24.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:24 smithi181 ceph-mon[32701]: 4.1c scrub starts 2024-09-11T08:18:24.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:24 smithi181 ceph-mon[32701]: 4.1c scrub ok 2024-09-11T08:18:24.352 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:18:24.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:24.388 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:24.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:18:24.489 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:18:24.510 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:24.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:24.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:18:24.612 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:24.650 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:24.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:24.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:24.752 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:24.766 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 13 16 31831 31815 152.923 108.5 0.0086416 0.00653087 2024-09-11T08:18:24.778 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:24.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:24.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:24.880 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:24.910 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:24.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:24.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:25.013 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:18:25.049 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:25.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:25.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:18:25.151 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:25.182 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:25.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:25.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:25.284 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:25.319 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:25.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:25.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:25.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:25 smithi181 ceph-mon[32701]: 4.11 scrub starts 2024-09-11T08:18:25.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:25 smithi181 ceph-mon[32701]: 4.11 scrub ok 2024-09-11T08:18:25.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:25 smithi181 ceph-mon[32701]: 5.8 scrub starts 2024-09-11T08:18:25.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:25 smithi181 ceph-mon[32701]: 5.8 scrub ok 2024-09-11T08:18:25.323 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:18:25 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:18:25] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:18:25.421 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:25.456 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:25.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:25.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:25.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:25 smithi150 ceph-mon[29276]: 4.11 scrub starts 2024-09-11T08:18:25.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:25 smithi150 ceph-mon[29276]: 4.11 scrub ok 2024-09-11T08:18:25.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:25 smithi150 ceph-mon[29276]: 5.8 scrub starts 2024-09-11T08:18:25.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:25 smithi150 ceph-mon[29276]: 5.8 scrub ok 2024-09-11T08:18:25.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:25 smithi150 ceph-mon[39612]: 4.11 scrub starts 2024-09-11T08:18:25.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:25 smithi150 ceph-mon[39612]: 4.11 scrub ok 2024-09-11T08:18:25.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:25 smithi150 ceph-mon[39612]: 5.8 scrub starts 2024-09-11T08:18:25.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:25 smithi150 ceph-mon[39612]: 5.8 scrub ok 2024-09-11T08:18:25.559 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:25.588 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:25.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:25.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:25.690 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:25.724 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:25.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:25.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:25.766 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 14 16 34062 34046 151.959 139.438 0.0100205 0.00657582 2024-09-11T08:18:25.826 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:18:25.864 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:25.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:25.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:18:25.966 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:25.993 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:25.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:25.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:26.005 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":488,"stamp":"2024-09-11T08:18:18.918409+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1258232028,"num_objects":19386,"num_object_clones":0,"num_object_copies":58158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19386,"num_whiteouts":0,"num_read":1492,"num_read_kb":1351,"num_write":20019,"num_write_kb":1229831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":1773644,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20108,"ondisk_log_size":20108,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":446,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":5986900,"kb_used_data":3529680,"kb_used_omap":19,"kb_used_meta":2457132,"kb_avail":743974316,"statfs":{"total":767960285184,"available":761829699584,"internally_reserved":0,"allocated":3614392320,"data_stored":3611283239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19955,"internal_metadata":2516103693},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,10,1,7],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1257635840,"num_objects":19190,"num_object_clones":0,"num_object_copies":57570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19190,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":19190,"num_write_kb":1228160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.908786"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":251,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-11T08:18:18.102260+0000","last_change":"2024-09-11T08:18:18.102212+0000","last_active":"2024-09-11T08:18:18.102260+0000","last_peered":"2024-09-11T08:18:18.102260+0000","last_clean":"2024-09-11T08:18:18.102260+0000","last_became_active":"2024-09-11T08:07:30.044976+0000","last_became_peered":"2024-09-11T08:07:30.044976+0000","last_unstale":"2024-09-11T08:18:18.102260+0000","last_undegraded":"2024-09-11T08:18:18.102260+0000","last_fullsized":"2024-09-11T08:18:18.102260+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:18.102122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:18:18.102122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:27.029766+0000","scrub_duration":0.0063791840000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"239'15","reported_seq":254,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:07.983383+0000","last_change":"2024-09-11T08:15:37.106907+0000","last_active":"2024-09-11T08:18:07.983383+0000","last_peered":"2024-09-11T08:18:07.983383+0000","last_clean":"2024-09-11T08:18:07.983383+0000","last_became_active":"2024-09-11T08:07:32.048048+0000","last_became_peered":"2024-09-11T08:07:32.048048+0000","last_unstale":"2024-09-11T08:18:07.983383+0000","last_undegraded":"2024-09-11T08:18:07.983383+0000","last_fullsized":"2024-09-11T08:18:07.983383+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:15:37.106824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:15:37.106824+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038285900000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":285,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.487960+0000","last_change":"2024-09-11T08:16:59.531358+0000","last_active":"2024-09-11T08:18:10.487960+0000","last_peered":"2024-09-11T08:18:10.487960+0000","last_clean":"2024-09-11T08:18:10.487960+0000","last_became_active":"2024-09-11T08:15:56.530946+0000","last_became_peered":"2024-09-11T08:15:56.530946+0000","last_unstale":"2024-09-11T08:18:10.487960+0000","last_undegraded":"2024-09-11T08:18:10.487960+0000","last_fullsized":"2024-09-11T08:18:10.487960+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:59.531202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:59.531202+0000","last_clean_scrub_stamp":"2024-09-11T08:16:59.531202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017351910000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":235,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:07.983501+0000","last_change":"2024-09-11T08:16:58.143005+0000","last_active":"2024-09-11T08:18:07.983501+0000","last_peered":"2024-09-11T08:18:07.983501+0000","last_clean":"2024-09-11T08:18:07.983501+0000","last_became_active":"2024-09-11T08:07:36.074789+0000","last_became_peered":"2024-09-11T08:07:36.074789+0000","last_unstale":"2024-09-11T08:18:07.983501+0000","last_undegraded":"2024-09-11T08:18:07.983501+0000","last_fullsized":"2024-09-11T08:18:07.983501+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:58.142957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:02.470649+0000","last_clean_scrub_stamp":"2024-09-11T08:16:58.142957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:22.869601+0000","scrub_duration":0.001328805,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":263,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.011239+0000","last_change":"2024-09-11T08:17:44.578728+0000","last_active":"2024-09-11T08:18:10.011239+0000","last_peered":"2024-09-11T08:18:10.011239+0000","last_clean":"2024-09-11T08:18:10.011239+0000","last_became_active":"2024-09-11T08:15:56.521104+0000","last_became_peered":"2024-09-11T08:15:56.521104+0000","last_unstale":"2024-09-11T08:18:10.011239+0000","last_undegraded":"2024-09-11T08:18:10.011239+0000","last_fullsized":"2024-09-11T08:18:10.011239+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:44.578639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:17:44.578639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:13.051708+0000","scrub_duration":0.0046216080000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":35,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:08.267705+0000","last_change":"2024-09-11T08:17:43.158518+0000","last_active":"2024-09-11T08:18:08.267705+0000","last_peered":"2024-09-11T08:18:08.267705+0000","last_clean":"2024-09-11T08:18:08.267705+0000","last_became_active":"2024-09-11T08:17:37.550048+0000","last_became_peered":"2024-09-11T08:17:37.550048+0000","last_unstale":"2024-09-11T08:18:08.267705+0000","last_undegraded":"2024-09-11T08:18:08.267705+0000","last_fullsized":"2024-09-11T08:18:08.267705+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:43.158429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:17:43.158429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:55.231013+0000","scrub_duration":0.0013456760000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"239'34","reported_seq":186,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.183543+0000","last_change":"2024-09-11T08:17:37.554247+0000","last_active":"2024-09-11T08:18:10.183543+0000","last_peered":"2024-09-11T08:18:10.183543+0000","last_clean":"2024-09-11T08:18:10.183543+0000","last_became_active":"2024-09-11T08:17:37.553866+0000","last_became_peered":"2024-09-11T08:17:37.553866+0000","last_unstale":"2024-09-11T08:18:10.183543+0000","last_undegraded":"2024-09-11T08:18:10.183543+0000","last_fullsized":"2024-09-11T08:18:10.183543+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'22","last_scrub_stamp":"2024-09-11T08:15:57.478870+0000","last_deep_scrub":"145'22","last_deep_scrub_stamp":"2024-09-11T08:15:57.478870+0000","last_clean_scrub_stamp":"2024-09-11T08:15:57.478870+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.021858643000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":227,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:07.983345+0000","last_change":"2024-09-11T08:15:52.050101+0000","last_active":"2024-09-11T08:18:07.983345+0000","last_peered":"2024-09-11T08:18:07.983345+0000","last_clean":"2024-09-11T08:18:07.983345+0000","last_became_active":"2024-09-11T08:07:34.052243+0000","last_became_peered":"2024-09-11T08:07:34.052243+0000","last_unstale":"2024-09-11T08:18:07.983345+0000","last_undegraded":"2024-09-11T08:18:07.983345+0000","last_fullsized":"2024-09-11T08:18:07.983345+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:52.050017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:15:52.050017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037049880000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":253,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:08.360617+0000","last_change":"2024-09-11T08:17:36.593508+0000","last_active":"2024-09-11T08:18:08.360617+0000","last_peered":"2024-09-11T08:18:08.360617+0000","last_clean":"2024-09-11T08:18:08.360617+0000","last_became_active":"2024-09-11T08:07:30.043950+0000","last_became_peered":"2024-09-11T08:07:30.043950+0000","last_unstale":"2024-09-11T08:18:08.360617+0000","last_undegraded":"2024-09-11T08:18:08.360617+0000","last_fullsized":"2024-09-11T08:18:08.360617+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:36.593436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:54.878409+0000","last_clean_scrub_stamp":"2024-09-11T08:17:36.593436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:05.576254+0000","scrub_duration":0.001379564,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"239'15","reported_seq":264,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:07.983611+0000","last_change":"2024-09-11T08:16:56.115531+0000","last_active":"2024-09-11T08:18:07.983611+0000","last_peered":"2024-09-11T08:18:07.983611+0000","last_clean":"2024-09-11T08:18:07.983611+0000","last_became_active":"2024-09-11T08:07:32.044626+0000","last_became_peered":"2024-09-11T08:07:32.044626+0000","last_unstale":"2024-09-11T08:18:07.983611+0000","last_undegraded":"2024-09-11T08:18:07.983611+0000","last_fullsized":"2024-09-11T08:18:07.983611+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:16:56.115436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:16:56.115436+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:15.933818+0000","scrub_duration":0.0060897449999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":273,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:08.330132+0000","last_change":"2024-09-11T08:17:51.520223+0000","last_active":"2024-09-11T08:18:08.330132+0000","last_peered":"2024-09-11T08:18:08.330132+0000","last_clean":"2024-09-11T08:18:08.330132+0000","last_became_active":"2024-09-11T08:15:56.530680+0000","last_became_peered":"2024-09-11T08:15:56.530680+0000","last_unstale":"2024-09-11T08:18:08.330132+0000","last_undegraded":"2024-09-11T08:18:08.330132+0000","last_fullsized":"2024-09-11T08:18:08.330132+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:51.520165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:17:51.520165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:21.269411+0000","scrub_duration":0.0013849839999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":252,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.540549+0000","last_change":"2024-09-11T08:17:33.060925+0000","last_active":"2024-09-11T08:18:10.540549+0000","last_peered":"2024-09-11T08:18:10.540549+0000","last_clean":"2024-09-11T08:18:10.540549+0000","last_became_active":"2024-09-11T08:14:24.792595+0000","last_became_peered":"2024-09-11T08:14:24.792595+0000","last_unstale":"2024-09-11T08:18:10.540549+0000","last_undegraded":"2024-09-11T08:18:10.540549+0000","last_fullsized":"2024-09-11T08:18:10.540549+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:33.060860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:17:33.060860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:35.555628+0000","scrub_duration":0.025846780999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":250,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:08.360327+0000","last_change":"2024-09-11T08:16:51.618963+0000","last_active":"2024-09-11T08:18:08.360327+0000","last_peered":"2024-09-11T08:18:08.360327+0000","last_clean":"2024-09-11T08:18:08.360327+0000","last_became_active":"2024-09-11T08:07:30.043277+0000","last_became_peered":"2024-09-11T08:07:30.043277+0000","last_unstale":"2024-09-11T08:18:08.360327+0000","last_undegraded":"2024-09-11T08:18:08.360327+0000","last_fullsized":"2024-09-11T08:18:08.360327+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:51.618910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:16:51.618910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011147119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"239'9","reported_seq":247,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:07.983530+0000","last_change":"2024-09-11T08:17:51.154234+0000","last_active":"2024-09-11T08:18:07.983530+0000","last_peered":"2024-09-11T08:18:07.983530+0000","last_clean":"2024-09-11T08:18:07.983530+0000","last_became_active":"2024-09-11T08:07:32.048661+0000","last_became_peered":"2024-09-11T08:07:32.048661+0000","last_unstale":"2024-09-11T08:18:07.983530+0000","last_undegraded":"2024-09-11T08:18:07.983530+0000","last_fullsized":"2024-09-11T08:18:07.983530+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'7","last_scrub_stamp":"2024-09-11T08:15:02.988425+0000","last_deep_scrub":"139'7","last_deep_scrub_stamp":"2024-09-11T08:12:24.869318+0000","last_clean_scrub_stamp":"2024-09-11T08:15:02.988425+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033901550000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":63,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:07.988151+0000","last_change":"2024-09-11T08:16:41.727421+0000","last_active":"2024-09-11T08:18:07.988151+0000","last_peered":"2024-09-11T08:18:07.988151+0000","last_clean":"2024-09-11T08:18:07.988151+0000","last_became_active":"2024-09-11T08:15:56.526954+0000","last_became_peered":"2024-09-11T08:15:56.526954+0000","last_unstale":"2024-09-11T08:18:07.988151+0000","last_undegraded":"2024-09-11T08:18:07.988151+0000","last_fullsized":"2024-09-11T08:18:07.988151+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:41.727344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:16:41.727344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011409409999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":254,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:08.329714+0000","last_change":"2024-09-11T08:17:42.618987+0000","last_active":"2024-09-11T08:18:08.329714+0000","last_peered":"2024-09-11T08:18:08.329714+0000","last_clean":"2024-09-11T08:18:08.329714+0000","last_became_active":"2024-09-11T08:07:36.077602+0000","last_became_peered":"2024-09-11T08:07:36.077602+0000","last_unstale":"2024-09-11T08:18:08.329714+0000","last_undegraded":"2024-09-11T08:18:08.329714+0000","last_fullsized":"2024-09-11T08:18:08.329714+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:42.618920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:04.782577+0000","last_clean_scrub_stamp":"2024-09-11T08:17:42.618920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:57.267152+0000","scrub_duration":0.0015188039999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":235,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:07.983976+0000","last_change":"2024-09-11T08:17:30.235545+0000","last_active":"2024-09-11T08:18:07.983976+0000","last_peered":"2024-09-11T08:18:07.983976+0000","last_clean":"2024-09-11T08:18:07.983976+0000","last_became_active":"2024-09-11T08:07:36.633338+0000","last_became_peered":"2024-09-11T08:07:36.633338+0000","last_unstale":"2024-09-11T08:18:07.983976+0000","last_undegraded":"2024-09-11T08:18:07.983976+0000","last_fullsized":"2024-09-11T08:18:07.983976+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:30.235441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:17:30.235441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:50.613994+0000","scrub_duration":0.0028448620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":268,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.541071+0000","last_change":"2024-09-11T08:17:32.057243+0000","last_active":"2024-09-11T08:18:10.541071+0000","last_peered":"2024-09-11T08:18:10.541071+0000","last_clean":"2024-09-11T08:18:10.541071+0000","last_became_active":"2024-09-11T08:15:56.516941+0000","last_became_peered":"2024-09-11T08:15:56.516941+0000","last_unstale":"2024-09-11T08:18:10.541071+0000","last_undegraded":"2024-09-11T08:18:10.541071+0000","last_fullsized":"2024-09-11T08:18:10.541071+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:32.057157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:17:32.057157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:43.946489+0000","scrub_duration":0.0042801360000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"239'34","reported_seq":299,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.183904+0000","last_change":"2024-09-11T08:16:04.159388+0000","last_active":"2024-09-11T08:18:10.183904+0000","last_peered":"2024-09-11T08:18:10.183904+0000","last_clean":"2024-09-11T08:18:10.183904+0000","last_became_active":"2024-09-11T08:07:32.046363+0000","last_became_peered":"2024-09-11T08:07:32.046363+0000","last_unstale":"2024-09-11T08:18:10.183904+0000","last_undegraded":"2024-09-11T08:18:10.183904+0000","last_fullsized":"2024-09-11T08:18:10.183904+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'22","last_scrub_stamp":"2024-09-11T08:16:04.159211+0000","last_deep_scrub":"107'22","last_deep_scrub_stamp":"2024-09-11T08:16:04.159211+0000","last_clean_scrub_stamp":"2024-09-11T08:16:04.159211+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0069586229999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":258,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.488462+0000","last_change":"2024-09-11T08:17:30.539868+0000","last_active":"2024-09-11T08:18:10.488462+0000","last_peered":"2024-09-11T08:18:10.488462+0000","last_clean":"2024-09-11T08:18:10.488462+0000","last_became_active":"2024-09-11T08:07:34.055410+0000","last_became_peered":"2024-09-11T08:07:34.055410+0000","last_unstale":"2024-09-11T08:18:10.488462+0000","last_undegraded":"2024-09-11T08:18:10.488462+0000","last_fullsized":"2024-09-11T08:18:10.488462+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:30.539794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:10:31.418873+0000","last_clean_scrub_stamp":"2024-09-11T08:17:30.539794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:59.055342+0000","scrub_duration":0.0070061949999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":25,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:08.267579+0000","last_change":"2024-09-11T08:17:37.553907+0000","last_active":"2024-09-11T08:18:08.267579+0000","last_peered":"2024-09-11T08:18:08.267579+0000","last_clean":"2024-09-11T08:18:08.267579+0000","last_became_active":"2024-09-11T08:17:37.553559+0000","last_became_peered":"2024-09-11T08:17:37.553559+0000","last_unstale":"2024-09-11T08:18:08.267579+0000","last_undegraded":"2024-09-11T08:18:08.267579+0000","last_fullsized":"2024-09-11T08:18:08.267579+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:39.677399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:16:39.677399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":241,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:07.983687+0000","last_change":"2024-09-11T08:15:48.081690+0000","last_active":"2024-09-11T08:18:07.983687+0000","last_peered":"2024-09-11T08:18:07.983687+0000","last_clean":"2024-09-11T08:18:07.983687+0000","last_became_active":"2024-09-11T08:07:30.046284+0000","last_became_peered":"2024-09-11T08:07:30.046284+0000","last_unstale":"2024-09-11T08:18:07.983687+0000","last_undegraded":"2024-09-11T08:18:07.983687+0000","last_fullsized":"2024-09-11T08:18:07.983687+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:48.081591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:48.081591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0059702619999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"239'18","reported_seq":93,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.183610+0000","last_change":"2024-09-11T08:17:37.558062+0000","last_active":"2024-09-11T08:18:10.183610+0000","last_peered":"2024-09-11T08:18:10.183610+0000","last_clean":"2024-09-11T08:18:10.183610+0000","last_became_active":"2024-09-11T08:17:37.553402+0000","last_became_peered":"2024-09-11T08:17:37.553402+0000","last_unstale":"2024-09-11T08:18:10.183610+0000","last_undegraded":"2024-09-11T08:18:10.183610+0000","last_fullsized":"2024-09-11T08:18:10.183610+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-11T08:15:53.518765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:15:53.518765+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":237,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:07.983790+0000","last_change":"2024-09-11T08:16:38.974464+0000","last_active":"2024-09-11T08:18:07.983790+0000","last_peered":"2024-09-11T08:18:07.983790+0000","last_clean":"2024-09-11T08:18:07.983790+0000","last_became_active":"2024-09-11T08:07:34.056528+0000","last_became_peered":"2024-09-11T08:07:34.056528+0000","last_unstale":"2024-09-11T08:18:07.983790+0000","last_undegraded":"2024-09-11T08:18:07.983790+0000","last_fullsized":"2024-09-11T08:18:07.983790+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:38.974372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:10:40.110490+0000","last_clean_scrub_stamp":"2024-09-11T08:16:38.974372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014148811000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":190,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:07.988420+0000","last_change":"2024-09-11T08:17:01.693734+0000","last_active":"2024-09-11T08:18:07.988420+0000","last_peered":"2024-09-11T08:18:07.988420+0000","last_clean":"2024-09-11T08:18:07.988420+0000","last_became_active":"2024-09-11T08:14:24.791610+0000","last_became_peered":"2024-09-11T08:14:24.791610+0000","last_unstale":"2024-09-11T08:18:07.988420+0000","last_undegraded":"2024-09-11T08:18:07.988420+0000","last_fullsized":"2024-09-11T08:18:07.988420+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-11T08:17:01.693642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:17:01.693642+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041332410000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":252,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.487736+0000","last_change":"2024-09-11T08:16:55.566928+0000","last_active":"2024-09-11T08:18:10.487736+0000","last_peered":"2024-09-11T08:18:10.487736+0000","last_clean":"2024-09-11T08:18:10.487736+0000","last_became_active":"2024-09-11T08:07:30.045932+0000","last_became_peered":"2024-09-11T08:07:30.045932+0000","last_unstale":"2024-09-11T08:18:10.487736+0000","last_undegraded":"2024-09-11T08:18:10.487736+0000","last_fullsized":"2024-09-11T08:18:10.487736+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:55.566790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:16:55.566790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:19.971541+0000","scrub_duration":0.004175102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"239'20","reported_seq":342,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.540992+0000","last_change":"2024-09-11T08:16:56.735441+0000","last_active":"2024-09-11T08:18:10.540992+0000","last_peered":"2024-09-11T08:18:10.540992+0000","last_clean":"2024-09-11T08:18:10.540992+0000","last_became_active":"2024-09-11T08:15:56.515817+0000","last_became_peered":"2024-09-11T08:15:56.515817+0000","last_unstale":"2024-09-11T08:18:10.540992+0000","last_undegraded":"2024-09-11T08:18:10.540992+0000","last_fullsized":"2024-09-11T08:18:10.540992+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-11T08:16:56.735310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:16:56.735310+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:22.326800+0000","scrub_duration":0.052749864,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"209'11","reported_seq":432,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:11.006435+0000","last_change":"2024-09-11T08:16:44.493380+0000","last_active":"2024-09-11T08:18:11.006435+0000","last_peered":"2024-09-11T08:18:11.006435+0000","last_clean":"2024-09-11T08:18:11.006435+0000","last_became_active":"2024-09-11T08:15:56.525498+0000","last_became_peered":"2024-09-11T08:15:56.525498+0000","last_unstale":"2024-09-11T08:18:11.006435+0000","last_undegraded":"2024-09-11T08:18:11.006435+0000","last_fullsized":"2024-09-11T08:18:11.006435+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"209'11","last_scrub_stamp":"2024-09-11T08:16:44.493296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:16:44.493296+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002213058,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2","version":"252'1199","reported_seq":1213,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-11T08:18:18.146050+0000","last_change":"2024-09-11T08:18:01.133843+0000","last_active":"2024-09-11T08:18:18.146050+0000","last_peered":"2024-09-11T08:18:18.146050+0000","last_clean":"2024-09-11T08:18:18.146050+0000","last_became_active":"2024-09-11T08:18:01.133542+0000","last_became_peered":"2024-09-11T08:18:01.133542+0000","last_unstale":"2024-09-11T08:18:18.146050+0000","last_undegraded":"2024-09-11T08:18:18.146050+0000","last_fullsized":"2024-09-11T08:18:18.146050+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:17:59.811675+0000","objects_scrubbed":0,"log_size":1199,"log_dups_size":0,"ondisk_log_size":1199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:01.172609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78577664,"num_objects":1199,"num_object_clones":0,"num_object_copies":3597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1199,"num_write_kb":76736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"239'27","reported_seq":303,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.488662+0000","last_change":"2024-09-11T08:17:00.559444+0000","last_active":"2024-09-11T08:18:10.488662+0000","last_peered":"2024-09-11T08:18:10.488662+0000","last_clean":"2024-09-11T08:18:10.488662+0000","last_became_active":"2024-09-11T08:07:32.054317+0000","last_became_peered":"2024-09-11T08:07:32.054317+0000","last_unstale":"2024-09-11T08:18:10.488662+0000","last_undegraded":"2024-09-11T08:18:10.488662+0000","last_fullsized":"2024-09-11T08:18:10.488662+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-11T08:17:00.559350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:17:00.559350+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013106273999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":262,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.011088+0000","last_change":"2024-09-11T08:16:42.624557+0000","last_active":"2024-09-11T08:18:10.011088+0000","last_peered":"2024-09-11T08:18:10.011088+0000","last_clean":"2024-09-11T08:18:10.011088+0000","last_became_active":"2024-09-11T08:15:56.521943+0000","last_became_peered":"2024-09-11T08:15:56.521943+0000","last_unstale":"2024-09-11T08:18:10.011088+0000","last_undegraded":"2024-09-11T08:18:10.011088+0000","last_fullsized":"2024-09-11T08:18:10.011088+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:42.624425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:16:42.624425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011367408000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":374,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.009128+0000","last_change":"2024-09-11T08:16:44.005131+0000","last_active":"2024-09-11T08:18:10.009128+0000","last_peered":"2024-09-11T08:18:10.009128+0000","last_clean":"2024-09-11T08:18:10.009128+0000","last_became_active":"2024-09-11T08:07:34.052981+0000","last_became_peered":"2024-09-11T08:07:34.052981+0000","last_unstale":"2024-09-11T08:18:10.009128+0000","last_undegraded":"2024-09-11T08:18:10.009128+0000","last_fullsized":"2024-09-11T08:18:10.009128+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:16:44.005077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:16:44.005077+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.019154572000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":254,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:08.360410+0000","last_change":"2024-09-11T08:17:47.577994+0000","last_active":"2024-09-11T08:18:08.360410+0000","last_peered":"2024-09-11T08:18:08.360410+0000","last_clean":"2024-09-11T08:18:08.360410+0000","last_became_active":"2024-09-11T08:07:36.073969+0000","last_became_peered":"2024-09-11T08:07:36.073969+0000","last_unstale":"2024-09-11T08:18:08.360410+0000","last_undegraded":"2024-09-11T08:18:08.360410+0000","last_fullsized":"2024-09-11T08:18:08.360410+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:47.577941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:06.616332+0000","last_clean_scrub_stamp":"2024-09-11T08:17:47.577941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:48.235918+0000","scrub_duration":0.001202289,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"251'1191","reported_seq":1204,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:17.830222+0000","last_change":"2024-09-11T08:18:00.829908+0000","last_active":"2024-09-11T08:18:17.830222+0000","last_peered":"2024-09-11T08:18:17.830222+0000","last_clean":"2024-09-11T08:18:17.830222+0000","last_became_active":"2024-09-11T08:18:00.829565+0000","last_became_peered":"2024-09-11T08:18:00.829565+0000","last_unstale":"2024-09-11T08:18:17.830222+0000","last_undegraded":"2024-09-11T08:18:17.830222+0000","last_fullsized":"2024-09-11T08:18:17.830222+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:17:59.811675+0000","objects_scrubbed":0,"log_size":1191,"log_dups_size":0,"ondisk_log_size":1191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:01.785398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78053376,"num_objects":1191,"num_object_clones":0,"num_object_copies":3573,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1191,"num_write_kb":76224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"239'30","reported_seq":297,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:07.988117+0000","last_change":"2024-09-11T08:17:37.556001+0000","last_active":"2024-09-11T08:18:07.988117+0000","last_peered":"2024-09-11T08:18:07.988117+0000","last_clean":"2024-09-11T08:18:07.988117+0000","last_became_active":"2024-09-11T08:17:37.555703+0000","last_became_peered":"2024-09-11T08:17:37.555703+0000","last_unstale":"2024-09-11T08:18:07.988117+0000","last_undegraded":"2024-09-11T08:18:07.988117+0000","last_fullsized":"2024-09-11T08:18:07.988117+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-11T08:17:03.650750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:17:03.650750+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:15.558999+0000","scrub_duration":0.001885091,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":221,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:07.988716+0000","last_change":"2024-09-11T08:17:37.554031+0000","last_active":"2024-09-11T08:18:07.988716+0000","last_peered":"2024-09-11T08:18:07.988716+0000","last_clean":"2024-09-11T08:18:07.988716+0000","last_became_active":"2024-09-11T08:17:37.553579+0000","last_became_peered":"2024-09-11T08:17:37.553579+0000","last_unstale":"2024-09-11T08:18:07.988716+0000","last_undegraded":"2024-09-11T08:18:07.988716+0000","last_fullsized":"2024-09-11T08:18:07.988716+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:43.614153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:16:43.614153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035672669999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":264,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:07.983419+0000","last_change":"2024-09-11T08:17:48.150197+0000","last_active":"2024-09-11T08:18:07.983419+0000","last_peered":"2024-09-11T08:18:07.983419+0000","last_clean":"2024-09-11T08:18:07.983419+0000","last_became_active":"2024-09-11T08:15:56.522606+0000","last_became_peered":"2024-09-11T08:15:56.522606+0000","last_unstale":"2024-09-11T08:18:07.983419+0000","last_undegraded":"2024-09-11T08:18:07.983419+0000","last_fullsized":"2024-09-11T08:18:07.983419+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:48.150140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:17:48.150140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:03.659795+0000","scrub_duration":0.001616889,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":235,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:07.983597+0000","last_change":"2024-09-11T08:17:39.096079+0000","last_active":"2024-09-11T08:18:07.983597+0000","last_peered":"2024-09-11T08:18:07.983597+0000","last_clean":"2024-09-11T08:18:07.983597+0000","last_became_active":"2024-09-11T08:07:36.069225+0000","last_became_peered":"2024-09-11T08:07:36.069225+0000","last_unstale":"2024-09-11T08:18:07.983597+0000","last_undegraded":"2024-09-11T08:18:07.983597+0000","last_fullsized":"2024-09-11T08:18:07.983597+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:39.096025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:04.937884+0000","last_clean_scrub_stamp":"2024-09-11T08:17:39.096025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:41.275131+0000","scrub_duration":0.001460954,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"251'1171","reported_seq":1184,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:17.791047+0000","last_change":"2024-09-11T08:18:00.823642+0000","last_active":"2024-09-11T08:18:17.791047+0000","last_peered":"2024-09-11T08:18:17.791047+0000","last_clean":"2024-09-11T08:18:17.791047+0000","last_became_active":"2024-09-11T08:18:00.823223+0000","last_became_peered":"2024-09-11T08:18:00.823223+0000","last_unstale":"2024-09-11T08:18:17.791047+0000","last_undegraded":"2024-09-11T08:18:17.791047+0000","last_fullsized":"2024-09-11T08:18:17.791047+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:17:59.811675+0000","objects_scrubbed":0,"log_size":1171,"log_dups_size":0,"ondisk_log_size":1171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:16.219581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76742656,"num_objects":1171,"num_object_clones":0,"num_object_copies":3513,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1171,"num_write_kb":74944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"239'28","reported_seq":281,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:07.983557+0000","last_change":"2024-09-11T08:17:49.184178+0000","last_active":"2024-09-11T08:18:07.983557+0000","last_peered":"2024-09-11T08:18:07.983557+0000","last_clean":"2024-09-11T08:18:07.983557+0000","last_became_active":"2024-09-11T08:07:32.040069+0000","last_became_peered":"2024-09-11T08:07:32.040069+0000","last_unstale":"2024-09-11T08:18:07.983557+0000","last_undegraded":"2024-09-11T08:18:07.983557+0000","last_fullsized":"2024-09-11T08:18:07.983557+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'28","last_scrub_stamp":"2024-09-11T08:17:49.184124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:17:49.184124+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:52.848465+0000","scrub_duration":0.0019943040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":251,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.540386+0000","last_change":"2024-09-11T08:17:47.960385+0000","last_active":"2024-09-11T08:18:10.540386+0000","last_peered":"2024-09-11T08:18:10.540386+0000","last_clean":"2024-09-11T08:18:10.540386+0000","last_became_active":"2024-09-11T08:07:30.050484+0000","last_became_peered":"2024-09-11T08:07:30.050484+0000","last_unstale":"2024-09-11T08:18:10.540386+0000","last_undegraded":"2024-09-11T08:18:10.540386+0000","last_fullsized":"2024-09-11T08:18:10.540386+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:47.960329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:47.960329+0000","last_clean_scrub_stamp":"2024-09-11T08:17:47.960329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:07.259809+0000","scrub_duration":0.001320681,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"82'9","reported_seq":389,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:13.678439+0000","last_change":"2024-09-11T08:17:47.485985+0000","last_active":"2024-09-11T08:18:13.678439+0000","last_peered":"2024-09-11T08:18:13.678439+0000","last_clean":"2024-09-11T08:18:13.678439+0000","last_became_active":"2024-09-11T08:07:34.056143+0000","last_became_peered":"2024-09-11T08:07:34.056143+0000","last_unstale":"2024-09-11T08:18:13.678439+0000","last_undegraded":"2024-09-11T08:18:13.678439+0000","last_fullsized":"2024-09-11T08:18:13.678439+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-11T08:17:47.485932+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-09-11T08:13:10.230703+0000","last_clean_scrub_stamp":"2024-09-11T08:17:47.485932+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:56.243066+0000","scrub_duration":0.0016186289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":225,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:07.988465+0000","last_change":"2024-09-11T08:17:35.867212+0000","last_active":"2024-09-11T08:18:07.988465+0000","last_peered":"2024-09-11T08:18:07.988465+0000","last_clean":"2024-09-11T08:18:07.988465+0000","last_became_active":"2024-09-11T08:14:24.788968+0000","last_became_peered":"2024-09-11T08:14:24.788968+0000","last_unstale":"2024-09-11T08:18:07.988465+0000","last_undegraded":"2024-09-11T08:18:07.988465+0000","last_fullsized":"2024-09-11T08:18:07.988465+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:35.867116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:17:35.867116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:55.885043+0000","scrub_duration":0.0021141390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"251'1205","reported_seq":1218,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:17.953455+0000","last_change":"2024-09-11T08:18:00.832311+0000","last_active":"2024-09-11T08:18:17.953455+0000","last_peered":"2024-09-11T08:18:17.953455+0000","last_clean":"2024-09-11T08:18:17.953455+0000","last_became_active":"2024-09-11T08:18:00.832055+0000","last_became_peered":"2024-09-11T08:18:00.832055+0000","last_unstale":"2024-09-11T08:18:17.953455+0000","last_undegraded":"2024-09-11T08:18:17.953455+0000","last_fullsized":"2024-09-11T08:18:17.953455+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:17:59.811675+0000","objects_scrubbed":0,"log_size":1205,"log_dups_size":0,"ondisk_log_size":1205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:06.728227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78970880,"num_objects":1205,"num_object_clones":0,"num_object_copies":3615,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1205,"num_write_kb":77120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"239'20","reported_seq":283,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.488246+0000","last_change":"2024-09-11T08:17:37.613078+0000","last_active":"2024-09-11T08:18:10.488246+0000","last_peered":"2024-09-11T08:18:10.488246+0000","last_clean":"2024-09-11T08:18:10.488246+0000","last_became_active":"2024-09-11T08:07:32.054294+0000","last_became_peered":"2024-09-11T08:07:32.054294+0000","last_unstale":"2024-09-11T08:18:10.488246+0000","last_undegraded":"2024-09-11T08:18:10.488246+0000","last_fullsized":"2024-09-11T08:18:10.488246+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'18","last_scrub_stamp":"2024-09-11T08:17:37.613023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:17:37.613023+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:05.205128+0000","scrub_duration":0.0017108679999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":246,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:08.360692+0000","last_change":"2024-09-11T08:17:54.475062+0000","last_active":"2024-09-11T08:18:08.360692+0000","last_peered":"2024-09-11T08:18:08.360692+0000","last_clean":"2024-09-11T08:18:08.360692+0000","last_became_active":"2024-09-11T08:07:30.047497+0000","last_became_peered":"2024-09-11T08:07:30.047497+0000","last_unstale":"2024-09-11T08:18:08.360692+0000","last_undegraded":"2024-09-11T08:18:08.360692+0000","last_fullsized":"2024-09-11T08:18:08.360692+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:46.459143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:46.459143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0057785019999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"239'10","reported_seq":53,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:11.006259+0000","last_change":"2024-09-11T08:17:37.554331+0000","last_active":"2024-09-11T08:18:11.006259+0000","last_peered":"2024-09-11T08:18:11.006259+0000","last_clean":"2024-09-11T08:18:11.006259+0000","last_became_active":"2024-09-11T08:17:37.552680+0000","last_became_peered":"2024-09-11T08:17:37.552680+0000","last_unstale":"2024-09-11T08:18:11.006259+0000","last_undegraded":"2024-09-11T08:18:11.006259+0000","last_fullsized":"2024-09-11T08:18:11.006259+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'9","last_scrub_stamp":"2024-09-11T08:17:01.640482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:17:01.640482+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"66'1","reported_seq":236,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.540665+0000","last_change":"2024-09-11T08:17:40.041132+0000","last_active":"2024-09-11T08:18:10.540665+0000","last_peered":"2024-09-11T08:18:10.540665+0000","last_clean":"2024-09-11T08:18:10.540665+0000","last_became_active":"2024-09-11T08:07:36.072593+0000","last_became_peered":"2024-09-11T08:07:36.072593+0000","last_unstale":"2024-09-11T08:18:10.540665+0000","last_undegraded":"2024-09-11T08:18:10.540665+0000","last_fullsized":"2024-09-11T08:18:10.540665+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-11T08:17:40.041070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:17:40.041070+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:07.811252+0000","scrub_duration":0.0016243379999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"251'1070","reported_seq":1083,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:17.846718+0000","last_change":"2024-09-11T08:18:00.826098+0000","last_active":"2024-09-11T08:18:17.846718+0000","last_peered":"2024-09-11T08:18:17.846718+0000","last_clean":"2024-09-11T08:18:17.846718+0000","last_became_active":"2024-09-11T08:18:00.825771+0000","last_became_peered":"2024-09-11T08:18:00.825771+0000","last_unstale":"2024-09-11T08:18:17.846718+0000","last_undegraded":"2024-09-11T08:18:17.846718+0000","last_fullsized":"2024-09-11T08:18:17.846718+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:17:59.811675+0000","objects_scrubbed":0,"log_size":1070,"log_dups_size":0,"ondisk_log_size":1070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:27.363446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70123520,"num_objects":1070,"num_object_clones":0,"num_object_copies":3210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1070,"num_write_kb":68480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"239'29","reported_seq":353,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.540991+0000","last_change":"2024-09-11T08:16:58.691731+0000","last_active":"2024-09-11T08:18:10.540991+0000","last_peered":"2024-09-11T08:18:10.540991+0000","last_clean":"2024-09-11T08:18:10.540991+0000","last_became_active":"2024-09-11T08:15:56.518384+0000","last_became_peered":"2024-09-11T08:15:56.518384+0000","last_unstale":"2024-09-11T08:18:10.540991+0000","last_undegraded":"2024-09-11T08:18:10.540991+0000","last_fullsized":"2024-09-11T08:18:10.540991+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-11T08:16:58.691634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:16:58.691634+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003405576,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":262,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.488471+0000","last_change":"2024-09-11T08:17:34.670772+0000","last_active":"2024-09-11T08:18:10.488471+0000","last_peered":"2024-09-11T08:18:10.488471+0000","last_clean":"2024-09-11T08:18:10.488471+0000","last_became_active":"2024-09-11T08:07:30.045380+0000","last_became_peered":"2024-09-11T08:07:30.045380+0000","last_unstale":"2024-09-11T08:18:10.488471+0000","last_undegraded":"2024-09-11T08:18:10.488471+0000","last_fullsized":"2024-09-11T08:18:10.488471+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:34.670680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:17:34.670680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:00.877538+0000","scrub_duration":0.0042100260000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":256,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:08.267286+0000","last_change":"2024-09-11T08:17:00.522545+0000","last_active":"2024-09-11T08:18:08.267286+0000","last_peered":"2024-09-11T08:18:08.267286+0000","last_clean":"2024-09-11T08:18:08.267286+0000","last_became_active":"2024-09-11T08:07:34.051004+0000","last_became_peered":"2024-09-11T08:07:34.051004+0000","last_unstale":"2024-09-11T08:18:08.267286+0000","last_undegraded":"2024-09-11T08:18:08.267286+0000","last_fullsized":"2024-09-11T08:18:08.267286+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:00.522467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:17:00.522467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:11.280048+0000","scrub_duration":0.0017451980000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":246,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.487928+0000","last_change":"2024-09-11T08:16:58.517711+0000","last_active":"2024-09-11T08:18:10.487928+0000","last_peered":"2024-09-11T08:18:10.487928+0000","last_clean":"2024-09-11T08:18:10.487928+0000","last_became_active":"2024-09-11T08:07:36.631668+0000","last_became_peered":"2024-09-11T08:07:36.631668+0000","last_unstale":"2024-09-11T08:18:10.487928+0000","last_undegraded":"2024-09-11T08:18:10.487928+0000","last_fullsized":"2024-09-11T08:18:10.487928+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:58.517624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:58.517624+0000","last_clean_scrub_stamp":"2024-09-11T08:16:58.517624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0083486270000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"252'1269","reported_seq":1283,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-11T08:18:18.374923+0000","last_change":"2024-09-11T08:18:00.828986+0000","last_active":"2024-09-11T08:18:18.374923+0000","last_peered":"2024-09-11T08:18:18.374923+0000","last_clean":"2024-09-11T08:18:18.374923+0000","last_became_active":"2024-09-11T08:18:00.828697+0000","last_became_peered":"2024-09-11T08:18:00.828697+0000","last_unstale":"2024-09-11T08:18:18.374923+0000","last_undegraded":"2024-09-11T08:18:18.374923+0000","last_fullsized":"2024-09-11T08:18:18.374923+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:17:59.811675+0000","objects_scrubbed":0,"log_size":1269,"log_dups_size":0,"ondisk_log_size":1269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:04.920227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83165184,"num_objects":1269,"num_object_clones":0,"num_object_copies":3807,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1269,"num_write_kb":81216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"239'51","reported_seq":169,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:13.678491+0000","last_change":"2024-09-11T08:17:50.928002+0000","last_active":"2024-09-11T08:18:13.678491+0000","last_peered":"2024-09-11T08:18:13.678491+0000","last_clean":"2024-09-11T08:18:13.678491+0000","last_became_active":"2024-09-11T08:15:08.775181+0000","last_became_peered":"2024-09-11T08:15:08.775181+0000","last_unstale":"2024-09-11T08:18:13.678491+0000","last_undegraded":"2024-09-11T08:18:13.678491+0000","last_fullsized":"2024-09-11T08:18:13.678491+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'51","last_scrub_stamp":"2024-09-11T08:17:50.927912+0000","last_deep_scrub":"190'39","last_deep_scrub_stamp":"2024-09-11T08:15:52.847968+0000","last_clean_scrub_stamp":"2024-09-11T08:17:50.927912+0000","objects_scrubbed":10,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:13.514824+0000","scrub_duration":0.0036015019999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"59'1","reported_seq":250,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:07.983758+0000","last_change":"2024-09-11T08:15:44.992430+0000","last_active":"2024-09-11T08:18:07.983758+0000","last_peered":"2024-09-11T08:18:07.983758+0000","last_clean":"2024-09-11T08:18:07.983758+0000","last_became_active":"2024-09-11T08:07:30.047513+0000","last_became_peered":"2024-09-11T08:07:30.047513+0000","last_unstale":"2024-09-11T08:18:07.983758+0000","last_undegraded":"2024-09-11T08:18:07.983758+0000","last_fullsized":"2024-09-11T08:18:07.983758+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-11T08:15:44.992356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:44.992356+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049464690000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":257,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:08.330223+0000","last_change":"2024-09-11T08:16:56.558518+0000","last_active":"2024-09-11T08:18:08.330223+0000","last_peered":"2024-09-11T08:18:08.330223+0000","last_clean":"2024-09-11T08:18:08.330223+0000","last_became_active":"2024-09-11T08:07:30.043130+0000","last_became_peered":"2024-09-11T08:07:30.043130+0000","last_unstale":"2024-09-11T08:18:08.330223+0000","last_undegraded":"2024-09-11T08:18:08.330223+0000","last_fullsized":"2024-09-11T08:18:08.330223+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-11T08:16:56.558301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:16:56.558301+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:23.711468+0000","scrub_duration":0.0048858299999999999,"objects_trimmed":0,"snaptrim_duration":0.0026120850000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":237,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.541014+0000","last_change":"2024-09-11T08:16:42.666121+0000","last_active":"2024-09-11T08:18:10.541014+0000","last_peered":"2024-09-11T08:18:10.541014+0000","last_clean":"2024-09-11T08:18:10.541014+0000","last_became_active":"2024-09-11T08:07:34.057154+0000","last_became_peered":"2024-09-11T08:07:34.057154+0000","last_unstale":"2024-09-11T08:18:10.541014+0000","last_undegraded":"2024-09-11T08:18:10.541014+0000","last_fullsized":"2024-09-11T08:18:10.541014+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:42.666035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:10:37.957096+0000","last_clean_scrub_stamp":"2024-09-11T08:16:42.666035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0077723870000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":263,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.011384+0000","last_change":"2024-09-11T08:17:31.602267+0000","last_active":"2024-09-11T08:18:10.011384+0000","last_peered":"2024-09-11T08:18:10.011384+0000","last_clean":"2024-09-11T08:18:10.011384+0000","last_became_active":"2024-09-11T08:15:56.525099+0000","last_became_peered":"2024-09-11T08:15:56.525099+0000","last_unstale":"2024-09-11T08:18:10.011384+0000","last_undegraded":"2024-09-11T08:18:10.011384+0000","last_fullsized":"2024-09-11T08:18:10.011384+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:31.602060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:17:31.602060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:55.062758+0000","scrub_duration":0.0059808200000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"251'1181","reported_seq":1194,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:17.960554+0000","last_change":"2024-09-11T08:18:00.831231+0000","last_active":"2024-09-11T08:18:17.960554+0000","last_peered":"2024-09-11T08:18:17.960554+0000","last_clean":"2024-09-11T08:18:17.960554+0000","last_became_active":"2024-09-11T08:18:00.830958+0000","last_became_peered":"2024-09-11T08:18:00.830958+0000","last_unstale":"2024-09-11T08:18:17.960554+0000","last_undegraded":"2024-09-11T08:18:17.960554+0000","last_fullsized":"2024-09-11T08:18:17.960554+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:17:59.811675+0000","objects_scrubbed":0,"log_size":1181,"log_dups_size":0,"ondisk_log_size":1181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:15.656645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77398016,"num_objects":1181,"num_object_clones":0,"num_object_copies":3543,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1181,"num_write_kb":75584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"239'17","reported_seq":286,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.488690+0000","last_change":"2024-09-11T08:16:41.514946+0000","last_active":"2024-09-11T08:18:10.488690+0000","last_peered":"2024-09-11T08:18:10.488690+0000","last_clean":"2024-09-11T08:18:10.488690+0000","last_became_active":"2024-09-11T08:07:32.050673+0000","last_became_peered":"2024-09-11T08:07:32.050673+0000","last_unstale":"2024-09-11T08:18:10.488690+0000","last_undegraded":"2024-09-11T08:18:10.488690+0000","last_fullsized":"2024-09-11T08:18:10.488690+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'13","last_scrub_stamp":"2024-09-11T08:16:41.514853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:16:41.514853+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0093017910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":255,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.487843+0000","last_change":"2024-09-11T08:17:02.493714+0000","last_active":"2024-09-11T08:18:10.487843+0000","last_peered":"2024-09-11T08:18:10.487843+0000","last_clean":"2024-09-11T08:18:10.487843+0000","last_became_active":"2024-09-11T08:07:30.047014+0000","last_became_peered":"2024-09-11T08:07:30.047014+0000","last_unstale":"2024-09-11T08:18:10.487843+0000","last_undegraded":"2024-09-11T08:18:10.487843+0000","last_fullsized":"2024-09-11T08:18:10.487843+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:02.493627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:17:02.493627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:24.784652+0000","scrub_duration":0.0048819830000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"127'2","reported_seq":271,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.183331+0000","last_change":"2024-09-11T08:16:40.021161+0000","last_active":"2024-09-11T08:18:10.183331+0000","last_peered":"2024-09-11T08:18:10.183331+0000","last_clean":"2024-09-11T08:18:10.183331+0000","last_became_active":"2024-09-11T08:07:30.045009+0000","last_became_peered":"2024-09-11T08:07:30.045009+0000","last_unstale":"2024-09-11T08:18:10.183331+0000","last_undegraded":"2024-09-11T08:18:10.183331+0000","last_fullsized":"2024-09-11T08:18:10.183331+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'2","last_scrub_stamp":"2024-09-11T08:16:40.021082+0000","last_deep_scrub":"127'2","last_deep_scrub_stamp":"2024-09-11T08:15:01.938497+0000","last_clean_scrub_stamp":"2024-09-11T08:16:40.021082+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002170426,"objects_trimmed":0,"snaptrim_duration":0.001859533,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":267,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.183854+0000","last_change":"2024-09-11T08:16:50.993227+0000","last_active":"2024-09-11T08:18:10.183854+0000","last_peered":"2024-09-11T08:18:10.183854+0000","last_clean":"2024-09-11T08:18:10.183854+0000","last_became_active":"2024-09-11T08:14:24.792279+0000","last_became_peered":"2024-09-11T08:14:24.792279+0000","last_unstale":"2024-09-11T08:18:10.183854+0000","last_undegraded":"2024-09-11T08:18:10.183854+0000","last_fullsized":"2024-09-11T08:18:10.183854+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:50.993154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:58.889094+0000","last_clean_scrub_stamp":"2024-09-11T08:16:50.993154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011069389999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":181,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:07.988765+0000","last_change":"2024-09-11T08:16:58.685525+0000","last_active":"2024-09-11T08:18:07.988765+0000","last_peered":"2024-09-11T08:18:07.988765+0000","last_clean":"2024-09-11T08:18:07.988765+0000","last_became_active":"2024-09-11T08:14:24.792847+0000","last_became_peered":"2024-09-11T08:14:24.792847+0000","last_unstale":"2024-09-11T08:18:07.988765+0000","last_undegraded":"2024-09-11T08:18:07.988765+0000","last_fullsized":"2024-09-11T08:18:07.988765+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:58.685473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:58.796217+0000","last_clean_scrub_stamp":"2024-09-11T08:16:58.685473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:13.302984+0000","scrub_duration":0.0016333490000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"252'1198","reported_seq":1212,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-11T08:18:18.367382+0000","last_change":"2024-09-11T08:18:00.832178+0000","last_active":"2024-09-11T08:18:18.367382+0000","last_peered":"2024-09-11T08:18:18.367382+0000","last_clean":"2024-09-11T08:18:18.367382+0000","last_became_active":"2024-09-11T08:18:00.831930+0000","last_became_peered":"2024-09-11T08:18:00.831930+0000","last_unstale":"2024-09-11T08:18:18.367382+0000","last_undegraded":"2024-09-11T08:18:18.367382+0000","last_fullsized":"2024-09-11T08:18:18.367382+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:17:59.811675+0000","objects_scrubbed":0,"log_size":1198,"log_dups_size":0,"ondisk_log_size":1198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:02.337796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78512128,"num_objects":1198,"num_object_clones":0,"num_object_copies":3594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1198,"num_write_kb":76672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"239'24","reported_seq":285,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.541050+0000","last_change":"2024-09-11T08:17:44.010492+0000","last_active":"2024-09-11T08:18:10.541050+0000","last_peered":"2024-09-11T08:18:10.541050+0000","last_clean":"2024-09-11T08:18:10.541050+0000","last_became_active":"2024-09-11T08:07:32.055407+0000","last_became_peered":"2024-09-11T08:07:32.055407+0000","last_unstale":"2024-09-11T08:18:10.541050+0000","last_undegraded":"2024-09-11T08:18:10.541050+0000","last_fullsized":"2024-09-11T08:18:10.541050+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'24","last_scrub_stamp":"2024-09-11T08:17:44.010434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:17:44.010434+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:06.755691+0000","scrub_duration":0.0041285339999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"59'1","reported_seq":258,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.540386+0000","last_change":"2024-09-11T08:17:46.998145+0000","last_active":"2024-09-11T08:18:10.540386+0000","last_peered":"2024-09-11T08:18:10.540386+0000","last_clean":"2024-09-11T08:18:10.540386+0000","last_became_active":"2024-09-11T08:07:30.050504+0000","last_became_peered":"2024-09-11T08:07:30.050504+0000","last_unstale":"2024-09-11T08:18:10.540386+0000","last_undegraded":"2024-09-11T08:18:10.540386+0000","last_fullsized":"2024-09-11T08:18:10.540386+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-11T08:17:46.998019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:17:46.998019+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:12.051489+0000","scrub_duration":0.0015225320000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"224'6","reported_seq":676,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:16.234172+0000","last_change":"2024-09-11T08:16:55.778375+0000","last_active":"2024-09-11T08:18:16.234172+0000","last_peered":"2024-09-11T08:18:16.234172+0000","last_clean":"2024-09-11T08:18:16.234172+0000","last_became_active":"2024-09-11T08:14:24.793048+0000","last_became_peered":"2024-09-11T08:14:24.793048+0000","last_unstale":"2024-09-11T08:18:16.234172+0000","last_undegraded":"2024-09-11T08:18:16.234172+0000","last_fullsized":"2024-09-11T08:18:16.234172+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'6","last_scrub_stamp":"2024-09-11T08:16:55.778284+0000","last_deep_scrub":"224'6","last_deep_scrub_stamp":"2024-09-11T08:16:55.778284+0000","last_clean_scrub_stamp":"2024-09-11T08:16:55.778284+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.018002499000000002,"objects_trimmed":0,"snaptrim_duration":0.000318821,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":239,"num_read_kb":234,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":246,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:08.360801+0000","last_change":"2024-09-11T08:16:37.787055+0000","last_active":"2024-09-11T08:18:08.360801+0000","last_peered":"2024-09-11T08:18:08.360801+0000","last_clean":"2024-09-11T08:18:08.360801+0000","last_became_active":"2024-09-11T08:07:34.058962+0000","last_became_peered":"2024-09-11T08:07:34.058962+0000","last_unstale":"2024-09-11T08:18:08.360801+0000","last_undegraded":"2024-09-11T08:18:08.360801+0000","last_fullsized":"2024-09-11T08:18:08.360801+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:37.786977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:16:37.786977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010499725999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":256,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.488315+0000","last_change":"2024-09-11T08:17:42.607077+0000","last_active":"2024-09-11T08:18:10.488315+0000","last_peered":"2024-09-11T08:18:10.488315+0000","last_clean":"2024-09-11T08:18:10.488315+0000","last_became_active":"2024-09-11T08:07:36.077597+0000","last_became_peered":"2024-09-11T08:07:36.077597+0000","last_unstale":"2024-09-11T08:18:10.488315+0000","last_undegraded":"2024-09-11T08:18:10.488315+0000","last_fullsized":"2024-09-11T08:18:10.488315+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:42.606972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:17:42.606972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:08.110790+0000","scrub_duration":0.0020562559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"252'1277","reported_seq":1291,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-11T08:18:18.374081+0000","last_change":"2024-09-11T08:18:00.829744+0000","last_active":"2024-09-11T08:18:18.374081+0000","last_peered":"2024-09-11T08:18:18.374081+0000","last_clean":"2024-09-11T08:18:18.374081+0000","last_became_active":"2024-09-11T08:18:00.829472+0000","last_became_peered":"2024-09-11T08:18:00.829472+0000","last_unstale":"2024-09-11T08:18:18.374081+0000","last_undegraded":"2024-09-11T08:18:18.374081+0000","last_fullsized":"2024-09-11T08:18:18.374081+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:17:59.811675+0000","objects_scrubbed":0,"log_size":1277,"log_dups_size":0,"ondisk_log_size":1277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:17.301601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83689472,"num_objects":1277,"num_object_clones":0,"num_object_copies":3831,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1277,"num_write_kb":81728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"239'33","reported_seq":312,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:08.267769+0000","last_change":"2024-09-11T08:16:40.472710+0000","last_active":"2024-09-11T08:18:08.267769+0000","last_peered":"2024-09-11T08:18:08.267769+0000","last_clean":"2024-09-11T08:18:08.267769+0000","last_became_active":"2024-09-11T08:14:24.789154+0000","last_became_peered":"2024-09-11T08:14:24.789154+0000","last_unstale":"2024-09-11T08:18:08.267769+0000","last_undegraded":"2024-09-11T08:18:08.267769+0000","last_fullsized":"2024-09-11T08:18:08.267769+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'21","last_scrub_stamp":"2024-09-11T08:16:40.472618+0000","last_deep_scrub":"84'21","last_deep_scrub_stamp":"2024-09-11T08:08:56.705306+0000","last_clean_scrub_stamp":"2024-09-11T08:16:40.472618+0000","objects_scrubbed":7,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0099327200000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":35,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:08.330174+0000","last_change":"2024-09-11T08:17:38.584894+0000","last_active":"2024-09-11T08:18:08.330174+0000","last_peered":"2024-09-11T08:18:08.330174+0000","last_clean":"2024-09-11T08:18:08.330174+0000","last_became_active":"2024-09-11T08:17:37.554397+0000","last_became_peered":"2024-09-11T08:17:37.554397+0000","last_unstale":"2024-09-11T08:18:08.330174+0000","last_undegraded":"2024-09-11T08:18:08.330174+0000","last_fullsized":"2024-09-11T08:18:08.330174+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:38.584839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:06.236119+0000","last_clean_scrub_stamp":"2024-09-11T08:17:38.584839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:42.517360+0000","scrub_duration":0.001652581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":69,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:07.987955+0000","last_change":"2024-09-11T08:16:45.611831+0000","last_active":"2024-09-11T08:18:07.987955+0000","last_peered":"2024-09-11T08:18:07.987955+0000","last_clean":"2024-09-11T08:18:07.987955+0000","last_became_active":"2024-09-11T08:15:56.523430+0000","last_became_peered":"2024-09-11T08:15:56.523430+0000","last_unstale":"2024-09-11T08:18:07.987955+0000","last_undegraded":"2024-09-11T08:18:07.987955+0000","last_fullsized":"2024-09-11T08:18:07.987955+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-11T08:16:45.611772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:05:21.231856+0000","last_clean_scrub_stamp":"2024-09-11T08:16:45.611772+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021764710000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":63,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:07.988337+0000","last_change":"2024-09-11T08:17:40.793670+0000","last_active":"2024-09-11T08:18:07.988337+0000","last_peered":"2024-09-11T08:18:07.988337+0000","last_clean":"2024-09-11T08:18:07.988337+0000","last_became_active":"2024-09-11T08:15:56.528641+0000","last_became_peered":"2024-09-11T08:15:56.528641+0000","last_unstale":"2024-09-11T08:18:07.988337+0000","last_undegraded":"2024-09-11T08:18:07.988337+0000","last_fullsized":"2024-09-11T08:18:07.988337+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:40.793611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:51.289872+0000","last_clean_scrub_stamp":"2024-09-11T08:17:40.793611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:41.869181+0000","scrub_duration":0.0012304760000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":244,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:08.329766+0000","last_change":"2024-09-11T08:16:43.782518+0000","last_active":"2024-09-11T08:18:08.329766+0000","last_peered":"2024-09-11T08:18:08.329766+0000","last_clean":"2024-09-11T08:18:08.329766+0000","last_became_active":"2024-09-11T08:07:36.072508+0000","last_became_peered":"2024-09-11T08:07:36.072508+0000","last_unstale":"2024-09-11T08:18:08.329766+0000","last_undegraded":"2024-09-11T08:18:08.329766+0000","last_fullsized":"2024-09-11T08:18:08.329766+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:43.782439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:01.777429+0000","last_clean_scrub_stamp":"2024-09-11T08:16:43.782439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038441999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"251'1152","reported_seq":1165,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:17.844484+0000","last_change":"2024-09-11T08:18:00.826408+0000","last_active":"2024-09-11T08:18:17.844484+0000","last_peered":"2024-09-11T08:18:17.844484+0000","last_clean":"2024-09-11T08:18:17.844484+0000","last_became_active":"2024-09-11T08:18:00.826126+0000","last_became_peered":"2024-09-11T08:18:00.826126+0000","last_unstale":"2024-09-11T08:18:17.844484+0000","last_undegraded":"2024-09-11T08:18:17.844484+0000","last_fullsized":"2024-09-11T08:18:17.844484+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:17:59.811675+0000","objects_scrubbed":0,"log_size":1152,"log_dups_size":0,"ondisk_log_size":1152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:04.332445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":1152,"num_object_clones":0,"num_object_copies":3456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1152,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"239'24","reported_seq":275,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:07.984066+0000","last_change":"2024-09-11T08:16:46.015835+0000","last_active":"2024-09-11T08:18:07.984066+0000","last_peered":"2024-09-11T08:18:07.984066+0000","last_clean":"2024-09-11T08:18:07.984066+0000","last_became_active":"2024-09-11T08:07:32.047971+0000","last_became_peered":"2024-09-11T08:07:32.047971+0000","last_unstale":"2024-09-11T08:18:07.984066+0000","last_undegraded":"2024-09-11T08:18:07.984066+0000","last_fullsized":"2024-09-11T08:18:07.984066+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'16","last_scrub_stamp":"2024-09-11T08:16:46.015776+0000","last_deep_scrub":"88'16","last_deep_scrub_stamp":"2024-09-11T08:09:39.635984+0000","last_clean_scrub_stamp":"2024-09-11T08:16:46.015776+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030649929999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":258,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:07.983652+0000","last_change":"2024-09-11T08:17:52.133078+0000","last_active":"2024-09-11T08:18:07.983652+0000","last_peered":"2024-09-11T08:18:07.983652+0000","last_clean":"2024-09-11T08:18:07.983652+0000","last_became_active":"2024-09-11T08:14:24.789888+0000","last_became_peered":"2024-09-11T08:14:24.789888+0000","last_unstale":"2024-09-11T08:18:07.983652+0000","last_undegraded":"2024-09-11T08:18:07.983652+0000","last_fullsized":"2024-09-11T08:18:07.983652+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:52.133025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:36.417911+0000","last_clean_scrub_stamp":"2024-09-11T08:17:52.133025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:07.817831+0000","scrub_duration":0.0012243499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":246,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:08.360693+0000","last_change":"2024-09-11T08:17:01.558423+0000","last_active":"2024-09-11T08:18:08.360693+0000","last_peered":"2024-09-11T08:18:08.360693+0000","last_clean":"2024-09-11T08:18:08.360693+0000","last_became_active":"2024-09-11T08:07:34.053347+0000","last_became_peered":"2024-09-11T08:07:34.053347+0000","last_unstale":"2024-09-11T08:18:08.360693+0000","last_undegraded":"2024-09-11T08:18:08.360693+0000","last_fullsized":"2024-09-11T08:18:08.360693+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:01.558340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:01.558340+0000","last_clean_scrub_stamp":"2024-09-11T08:17:01.558340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:11.604522+0000","scrub_duration":0.0012693030000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":35,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:08.267486+0000","last_change":"2024-09-11T08:17:42.108402+0000","last_active":"2024-09-11T08:18:08.267486+0000","last_peered":"2024-09-11T08:18:08.267486+0000","last_clean":"2024-09-11T08:18:08.267486+0000","last_became_active":"2024-09-11T08:17:37.552895+0000","last_became_peered":"2024-09-11T08:17:37.552895+0000","last_unstale":"2024-09-11T08:18:08.267486+0000","last_undegraded":"2024-09-11T08:18:08.267486+0000","last_fullsized":"2024-09-11T08:18:08.267486+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:42.108342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:17:42.108342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:44.557067+0000","scrub_duration":0.001403751,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.0","version":"252'1232","reported_seq":1246,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-11T08:18:18.337626+0000","last_change":"2024-09-11T08:18:01.134013+0000","last_active":"2024-09-11T08:18:18.337626+0000","last_peered":"2024-09-11T08:18:18.337626+0000","last_clean":"2024-09-11T08:18:18.337626+0000","last_became_active":"2024-09-11T08:18:01.133673+0000","last_became_peered":"2024-09-11T08:18:01.133673+0000","last_unstale":"2024-09-11T08:18:18.337626+0000","last_undegraded":"2024-09-11T08:18:18.337626+0000","last_fullsized":"2024-09-11T08:18:18.337626+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:17:59.811675+0000","objects_scrubbed":0,"log_size":1232,"log_dups_size":0,"ondisk_log_size":1232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:22.769468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80740352,"num_objects":1232,"num_object_clones":0,"num_object_copies":3696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1232,"num_write_kb":78848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"239'39","reported_seq":324,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:07.984108+0000","last_change":"2024-09-11T08:15:42.061711+0000","last_active":"2024-09-11T08:18:07.984108+0000","last_peered":"2024-09-11T08:18:07.984108+0000","last_clean":"2024-09-11T08:18:07.984108+0000","last_became_active":"2024-09-11T08:09:08.106771+0000","last_became_peered":"2024-09-11T08:09:08.106771+0000","last_unstale":"2024-09-11T08:18:07.984108+0000","last_undegraded":"2024-09-11T08:18:07.984108+0000","last_fullsized":"2024-09-11T08:18:07.984108+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'25","last_scrub_stamp":"2024-09-11T08:15:42.061632+0000","last_deep_scrub":"77'25","last_deep_scrub_stamp":"2024-09-11T08:08:32.458210+0000","last_clean_scrub_stamp":"2024-09-11T08:15:42.061632+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00500454,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":256,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:08.267436+0000","last_change":"2024-09-11T08:17:54.046967+0000","last_active":"2024-09-11T08:18:08.267436+0000","last_peered":"2024-09-11T08:18:08.267436+0000","last_clean":"2024-09-11T08:18:08.267436+0000","last_became_active":"2024-09-11T08:07:30.042555+0000","last_became_peered":"2024-09-11T08:07:30.042555+0000","last_unstale":"2024-09-11T08:18:08.267436+0000","last_undegraded":"2024-09-11T08:18:08.267436+0000","last_fullsized":"2024-09-11T08:18:08.267436+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:41.506939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:16:41.506939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026028340000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":248,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.488382+0000","last_change":"2024-09-11T08:16:52.542153+0000","last_active":"2024-09-11T08:18:10.488382+0000","last_peered":"2024-09-11T08:18:10.488382+0000","last_clean":"2024-09-11T08:18:10.488382+0000","last_became_active":"2024-09-11T08:07:34.054961+0000","last_became_peered":"2024-09-11T08:07:34.054961+0000","last_unstale":"2024-09-11T08:18:10.488382+0000","last_undegraded":"2024-09-11T08:18:10.488382+0000","last_fullsized":"2024-09-11T08:18:10.488382+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:52.542072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:16:52.542072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052953879999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":265,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.183871+0000","last_change":"2024-09-11T08:17:32.226752+0000","last_active":"2024-09-11T08:18:10.183871+0000","last_peered":"2024-09-11T08:18:10.183871+0000","last_clean":"2024-09-11T08:18:10.183871+0000","last_became_active":"2024-09-11T08:07:36.071194+0000","last_became_peered":"2024-09-11T08:07:36.071194+0000","last_unstale":"2024-09-11T08:18:10.183871+0000","last_undegraded":"2024-09-11T08:18:10.183871+0000","last_fullsized":"2024-09-11T08:18:10.183871+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:32.226668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:07.930137+0000","last_clean_scrub_stamp":"2024-09-11T08:17:32.226668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:43.201485+0000","scrub_duration":0.0099076860000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"251'1132","reported_seq":1145,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:17.858060+0000","last_change":"2024-09-11T08:18:00.828764+0000","last_active":"2024-09-11T08:18:17.858060+0000","last_peered":"2024-09-11T08:18:17.858060+0000","last_clean":"2024-09-11T08:18:17.858060+0000","last_became_active":"2024-09-11T08:18:00.828320+0000","last_became_peered":"2024-09-11T08:18:00.828320+0000","last_unstale":"2024-09-11T08:18:17.858060+0000","last_undegraded":"2024-09-11T08:18:17.858060+0000","last_fullsized":"2024-09-11T08:18:17.858060+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:17:59.811675+0000","objects_scrubbed":0,"log_size":1132,"log_dups_size":0,"ondisk_log_size":1132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:07.547456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74186752,"num_objects":1132,"num_object_clones":0,"num_object_copies":3396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1132,"num_write_kb":72448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"239'25","reported_seq":274,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:07.983815+0000","last_change":"2024-09-11T08:17:01.126558+0000","last_active":"2024-09-11T08:18:07.983815+0000","last_peered":"2024-09-11T08:18:07.983815+0000","last_clean":"2024-09-11T08:18:07.983815+0000","last_became_active":"2024-09-11T08:07:32.043314+0000","last_became_peered":"2024-09-11T08:07:32.043314+0000","last_unstale":"2024-09-11T08:18:07.983815+0000","last_undegraded":"2024-09-11T08:18:07.983815+0000","last_fullsized":"2024-09-11T08:18:07.983815+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'15","last_scrub_stamp":"2024-09-11T08:17:01.126478+0000","last_deep_scrub":"224'15","last_deep_scrub_stamp":"2024-09-11T08:17:01.126478+0000","last_clean_scrub_stamp":"2024-09-11T08:17:01.126478+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:16.625548+0000","scrub_duration":0.0092237729999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":36,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.541043+0000","last_change":"2024-09-11T08:17:45.979903+0000","last_active":"2024-09-11T08:18:10.541043+0000","last_peered":"2024-09-11T08:18:10.541043+0000","last_clean":"2024-09-11T08:18:10.541043+0000","last_became_active":"2024-09-11T08:17:37.557278+0000","last_became_peered":"2024-09-11T08:17:37.557278+0000","last_unstale":"2024-09-11T08:18:10.541043+0000","last_undegraded":"2024-09-11T08:18:10.541043+0000","last_fullsized":"2024-09-11T08:18:10.541043+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:45.979847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:45.979847+0000","last_clean_scrub_stamp":"2024-09-11T08:17:45.979847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:52.778180+0000","scrub_duration":0.001534947,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"171'12","reported_seq":408,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:13.678725+0000","last_change":"2024-09-11T08:16:50.043791+0000","last_active":"2024-09-11T08:18:13.678725+0000","last_peered":"2024-09-11T08:18:13.678725+0000","last_clean":"2024-09-11T08:18:13.678725+0000","last_became_active":"2024-09-11T08:14:24.794446+0000","last_became_peered":"2024-09-11T08:14:24.794446+0000","last_unstale":"2024-09-11T08:18:13.678725+0000","last_undegraded":"2024-09-11T08:18:13.678725+0000","last_fullsized":"2024-09-11T08:18:13.678725+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'12","last_scrub_stamp":"2024-09-11T08:16:50.043698+0000","last_deep_scrub":"171'12","last_deep_scrub_stamp":"2024-09-11T08:14:56.881299+0000","last_clean_scrub_stamp":"2024-09-11T08:16:50.043698+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0069122760000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":235,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:07.983959+0000","last_change":"2024-09-11T08:17:40.124232+0000","last_active":"2024-09-11T08:18:07.983959+0000","last_peered":"2024-09-11T08:18:07.983959+0000","last_clean":"2024-09-11T08:18:07.983959+0000","last_became_active":"2024-09-11T08:07:36.074684+0000","last_became_peered":"2024-09-11T08:07:36.074684+0000","last_unstale":"2024-09-11T08:18:07.983959+0000","last_undegraded":"2024-09-11T08:18:07.983959+0000","last_fullsized":"2024-09-11T08:18:07.983959+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:40.124172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:14.613532+0000","last_clean_scrub_stamp":"2024-09-11T08:17:40.124172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:46.653254+0000","scrub_duration":0.001238495,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"251'1268","reported_seq":1281,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:17.965943+0000","last_change":"2024-09-11T08:18:01.133833+0000","last_active":"2024-09-11T08:18:17.965943+0000","last_peered":"2024-09-11T08:18:17.965943+0000","last_clean":"2024-09-11T08:18:17.965943+0000","last_became_active":"2024-09-11T08:18:01.133521+0000","last_became_peered":"2024-09-11T08:18:01.133521+0000","last_unstale":"2024-09-11T08:18:17.965943+0000","last_undegraded":"2024-09-11T08:18:17.965943+0000","last_fullsized":"2024-09-11T08:18:17.965943+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:17:59.811675+0000","objects_scrubbed":0,"log_size":1268,"log_dups_size":0,"ondisk_log_size":1268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:12.206914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83099648,"num_objects":1268,"num_object_clones":0,"num_object_copies":3804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1268,"num_write_kb":81152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"239'15","reported_seq":264,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.540844+0000","last_change":"2024-09-11T08:16:49.692191+0000","last_active":"2024-09-11T08:18:10.540844+0000","last_peered":"2024-09-11T08:18:10.540844+0000","last_clean":"2024-09-11T08:18:10.540844+0000","last_became_active":"2024-09-11T08:07:32.054885+0000","last_became_peered":"2024-09-11T08:07:32.054885+0000","last_unstale":"2024-09-11T08:18:10.540844+0000","last_undegraded":"2024-09-11T08:18:10.540844+0000","last_fullsized":"2024-09-11T08:18:10.540844+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:16:49.692128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:16:49.692128+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012332951999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":260,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:08.360499+0000","last_change":"2024-09-11T08:17:02.576325+0000","last_active":"2024-09-11T08:18:08.360499+0000","last_peered":"2024-09-11T08:18:08.360499+0000","last_clean":"2024-09-11T08:18:08.360499+0000","last_became_active":"2024-09-11T08:07:30.045613+0000","last_became_peered":"2024-09-11T08:07:30.045613+0000","last_unstale":"2024-09-11T08:18:08.360499+0000","last_undegraded":"2024-09-11T08:18:08.360499+0000","last_fullsized":"2024-09-11T08:18:08.360499+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:02.576236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:17:02.576236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003426503,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":267,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.183618+0000","last_change":"2024-09-11T08:17:33.268363+0000","last_active":"2024-09-11T08:18:10.183618+0000","last_peered":"2024-09-11T08:18:10.183618+0000","last_clean":"2024-09-11T08:18:10.183618+0000","last_became_active":"2024-09-11T08:07:34.056606+0000","last_became_peered":"2024-09-11T08:07:34.056606+0000","last_unstale":"2024-09-11T08:18:10.183618+0000","last_undegraded":"2024-09-11T08:18:10.183618+0000","last_fullsized":"2024-09-11T08:18:10.183618+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:33.268226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:17:33.268226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:56.735900+0000","scrub_duration":0.0024528100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":262,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.540689+0000","last_change":"2024-09-11T08:17:03.795482+0000","last_active":"2024-09-11T08:18:10.540689+0000","last_peered":"2024-09-11T08:18:10.540689+0000","last_clean":"2024-09-11T08:18:10.540689+0000","last_became_active":"2024-09-11T08:15:56.518884+0000","last_became_peered":"2024-09-11T08:15:56.518884+0000","last_unstale":"2024-09-11T08:18:10.540689+0000","last_undegraded":"2024-09-11T08:18:10.540689+0000","last_fullsized":"2024-09-11T08:18:10.540689+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:03.795360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:17:03.795360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0069097719999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"251'1182","reported_seq":1195,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:17.956901+0000","last_change":"2024-09-11T08:18:00.831563+0000","last_active":"2024-09-11T08:18:17.956901+0000","last_peered":"2024-09-11T08:18:17.956901+0000","last_clean":"2024-09-11T08:18:17.956901+0000","last_became_active":"2024-09-11T08:18:00.831044+0000","last_became_peered":"2024-09-11T08:18:00.831044+0000","last_unstale":"2024-09-11T08:18:17.956901+0000","last_undegraded":"2024-09-11T08:18:17.956901+0000","last_fullsized":"2024-09-11T08:18:17.956901+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:17:59.811675+0000","objects_scrubbed":0,"log_size":1182,"log_dups_size":0,"ondisk_log_size":1182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:17.562328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77463552,"num_objects":1182,"num_object_clones":0,"num_object_copies":3546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1182,"num_write_kb":75648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"239'35","reported_seq":380,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:08.267482+0000","last_change":"2024-09-11T08:17:32.266441+0000","last_active":"2024-09-11T08:18:08.267482+0000","last_peered":"2024-09-11T08:18:08.267482+0000","last_clean":"2024-09-11T08:18:08.267482+0000","last_became_active":"2024-09-11T08:15:56.526961+0000","last_became_peered":"2024-09-11T08:15:56.526961+0000","last_unstale":"2024-09-11T08:18:08.267482+0000","last_undegraded":"2024-09-11T08:18:08.267482+0000","last_fullsized":"2024-09-11T08:18:08.267482+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-11T08:17:32.266343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:17:32.266343+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:41.658630+0000","scrub_duration":0.0088220920000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":63,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:07.988584+0000","last_change":"2024-09-11T08:16:46.578643+0000","last_active":"2024-09-11T08:18:07.988584+0000","last_peered":"2024-09-11T08:18:07.988584+0000","last_clean":"2024-09-11T08:18:07.988584+0000","last_became_active":"2024-09-11T08:15:56.530370+0000","last_became_peered":"2024-09-11T08:15:56.530370+0000","last_unstale":"2024-09-11T08:18:07.988584+0000","last_undegraded":"2024-09-11T08:18:07.988584+0000","last_fullsized":"2024-09-11T08:18:07.988584+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:46.578575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:44.385372+0000","last_clean_scrub_stamp":"2024-09-11T08:16:46.578575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:15.109608+0000","scrub_duration":0.016632636999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":260,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.183546+0000","last_change":"2024-09-11T08:16:55.019406+0000","last_active":"2024-09-11T08:18:10.183546+0000","last_peered":"2024-09-11T08:18:10.183546+0000","last_clean":"2024-09-11T08:18:10.183546+0000","last_became_active":"2024-09-11T08:07:34.063589+0000","last_became_peered":"2024-09-11T08:07:34.063589+0000","last_unstale":"2024-09-11T08:18:10.183546+0000","last_undegraded":"2024-09-11T08:18:10.183546+0000","last_fullsized":"2024-09-11T08:18:10.183546+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:55.019351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:16:55.019351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001926295,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":63,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:07.987994+0000","last_change":"2024-09-11T08:17:41.786187+0000","last_active":"2024-09-11T08:18:07.987994+0000","last_peered":"2024-09-11T08:18:07.987994+0000","last_clean":"2024-09-11T08:18:07.987994+0000","last_became_active":"2024-09-11T08:15:56.518169+0000","last_became_peered":"2024-09-11T08:15:56.518169+0000","last_unstale":"2024-09-11T08:18:07.987994+0000","last_undegraded":"2024-09-11T08:18:07.987994+0000","last_fullsized":"2024-09-11T08:18:07.987994+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:41.786126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:17:41.786126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:50.563217+0000","scrub_duration":0.0015404170000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"252'1208","reported_seq":1222,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-11T08:18:18.344185+0000","last_change":"2024-09-11T08:18:00.827444+0000","last_active":"2024-09-11T08:18:18.344185+0000","last_peered":"2024-09-11T08:18:18.344185+0000","last_clean":"2024-09-11T08:18:18.344185+0000","last_became_active":"2024-09-11T08:18:00.827140+0000","last_became_peered":"2024-09-11T08:18:00.827140+0000","last_unstale":"2024-09-11T08:18:18.344185+0000","last_undegraded":"2024-09-11T08:18:18.344185+0000","last_fullsized":"2024-09-11T08:18:18.344185+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:17:59.811675+0000","objects_scrubbed":0,"log_size":1208,"log_dups_size":0,"ondisk_log_size":1208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:00.818038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79167488,"num_objects":1208,"num_object_clones":0,"num_object_copies":3624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1208,"num_write_kb":77312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"239'23","reported_seq":276,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:07.983505+0000","last_change":"2024-09-11T08:15:53.040339+0000","last_active":"2024-09-11T08:18:07.983505+0000","last_peered":"2024-09-11T08:18:07.983505+0000","last_clean":"2024-09-11T08:18:07.983505+0000","last_became_active":"2024-09-11T08:07:32.048164+0000","last_became_peered":"2024-09-11T08:07:32.048164+0000","last_unstale":"2024-09-11T08:18:07.983505+0000","last_undegraded":"2024-09-11T08:18:07.983505+0000","last_fullsized":"2024-09-11T08:18:07.983505+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'17","last_scrub_stamp":"2024-09-11T08:15:53.040223+0000","last_deep_scrub":"145'17","last_deep_scrub_stamp":"2024-09-11T08:13:08.160848+0000","last_clean_scrub_stamp":"2024-09-11T08:15:53.040223+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039160820000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":63,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:07.988729+0000","last_change":"2024-09-11T08:16:47.557730+0000","last_active":"2024-09-11T08:18:07.988729+0000","last_peered":"2024-09-11T08:18:07.988729+0000","last_clean":"2024-09-11T08:18:07.988729+0000","last_became_active":"2024-09-11T08:15:56.525845+0000","last_became_peered":"2024-09-11T08:15:56.525845+0000","last_unstale":"2024-09-11T08:18:07.988729+0000","last_undegraded":"2024-09-11T08:18:07.988729+0000","last_fullsized":"2024-09-11T08:18:07.988729+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:47.557675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:53.247275+0000","last_clean_scrub_stamp":"2024-09-11T08:16:47.557675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0086777009999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":247,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.183456+0000","last_change":"2024-09-11T08:16:38.072401+0000","last_active":"2024-09-11T08:18:10.183456+0000","last_peered":"2024-09-11T08:18:10.183456+0000","last_clean":"2024-09-11T08:18:10.183456+0000","last_became_active":"2024-09-11T08:07:34.057479+0000","last_became_peered":"2024-09-11T08:07:34.057479+0000","last_unstale":"2024-09-11T08:18:10.183456+0000","last_undegraded":"2024-09-11T08:18:10.183456+0000","last_fullsized":"2024-09-11T08:18:10.183456+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:38.072323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:16:38.072323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005761909,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":263,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.011298+0000","last_change":"2024-09-11T08:16:50.631547+0000","last_active":"2024-09-11T08:18:10.011298+0000","last_peered":"2024-09-11T08:18:10.011298+0000","last_clean":"2024-09-11T08:18:10.011298+0000","last_became_active":"2024-09-11T08:15:56.522624+0000","last_became_peered":"2024-09-11T08:15:56.522624+0000","last_unstale":"2024-09-11T08:18:10.011298+0000","last_undegraded":"2024-09-11T08:18:10.011298+0000","last_fullsized":"2024-09-11T08:18:10.011298+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:50.631471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:16:50.631471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:18.067947+0000","scrub_duration":0.0037889149999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"252'1255","reported_seq":1269,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-11T08:18:18.378494+0000","last_change":"2024-09-11T08:18:00.829727+0000","last_active":"2024-09-11T08:18:18.378494+0000","last_peered":"2024-09-11T08:18:18.378494+0000","last_clean":"2024-09-11T08:18:18.378494+0000","last_became_active":"2024-09-11T08:18:00.829429+0000","last_became_peered":"2024-09-11T08:18:00.829429+0000","last_unstale":"2024-09-11T08:18:18.378494+0000","last_undegraded":"2024-09-11T08:18:18.378494+0000","last_fullsized":"2024-09-11T08:18:18.378494+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:17:59.811675+0000","objects_scrubbed":0,"log_size":1255,"log_dups_size":0,"ondisk_log_size":1255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:14.300992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82247680,"num_objects":1255,"num_object_clones":0,"num_object_copies":3765,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1255,"num_write_kb":80320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"239'27","reported_seq":350,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:08.329918+0000","last_change":"2024-09-11T08:17:33.548568+0000","last_active":"2024-09-11T08:18:08.329918+0000","last_peered":"2024-09-11T08:18:08.329918+0000","last_clean":"2024-09-11T08:18:08.329918+0000","last_became_active":"2024-09-11T08:15:56.532431+0000","last_became_peered":"2024-09-11T08:15:56.532431+0000","last_unstale":"2024-09-11T08:18:08.329918+0000","last_undegraded":"2024-09-11T08:18:08.329918+0000","last_fullsized":"2024-09-11T08:18:08.329918+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-11T08:17:33.548509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:17:33.548509+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:44.124562+0000","scrub_duration":0.010207872,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":69,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:07.988759+0000","last_change":"2024-09-11T08:16:42.714111+0000","last_active":"2024-09-11T08:18:07.988759+0000","last_peered":"2024-09-11T08:18:07.988759+0000","last_clean":"2024-09-11T08:18:07.988759+0000","last_became_active":"2024-09-11T08:15:56.515328+0000","last_became_peered":"2024-09-11T08:15:56.515328+0000","last_unstale":"2024-09-11T08:18:07.988759+0000","last_undegraded":"2024-09-11T08:18:07.988759+0000","last_fullsized":"2024-09-11T08:18:07.988759+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-11T08:16:42.714038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:16:42.714038+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0083916919999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"209'11","reported_seq":95,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.010529+0000","last_change":"2024-09-11T08:17:42.782853+0000","last_active":"2024-09-11T08:18:10.010529+0000","last_peered":"2024-09-11T08:18:10.010529+0000","last_clean":"2024-09-11T08:18:10.010529+0000","last_became_active":"2024-09-11T08:15:56.524513+0000","last_became_peered":"2024-09-11T08:15:56.524513+0000","last_unstale":"2024-09-11T08:18:10.010529+0000","last_undegraded":"2024-09-11T08:18:10.010529+0000","last_fullsized":"2024-09-11T08:18:10.010529+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"209'11","last_scrub_stamp":"2024-09-11T08:17:42.782786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:17:42.782786+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:09.663912+0000","scrub_duration":0.001693901,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":244,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:08.330016+0000","last_change":"2024-09-11T08:16:59.559972+0000","last_active":"2024-09-11T08:18:08.330016+0000","last_peered":"2024-09-11T08:18:08.330016+0000","last_clean":"2024-09-11T08:18:08.330016+0000","last_became_active":"2024-09-11T08:07:36.631755+0000","last_became_peered":"2024-09-11T08:07:36.631755+0000","last_unstale":"2024-09-11T08:18:08.330016+0000","last_undegraded":"2024-09-11T08:18:08.330016+0000","last_fullsized":"2024-09-11T08:18:08.330016+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:59.559893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:03.484755+0000","last_clean_scrub_stamp":"2024-09-11T08:16:59.559893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:28.125788+0000","scrub_duration":0.006203648,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":260,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:08.267361+0000","last_change":"2024-09-11T08:16:57.501989+0000","last_active":"2024-09-11T08:18:08.267361+0000","last_peered":"2024-09-11T08:18:08.267361+0000","last_clean":"2024-09-11T08:18:08.267361+0000","last_became_active":"2024-09-11T08:07:30.045910+0000","last_became_peered":"2024-09-11T08:07:30.045910+0000","last_unstale":"2024-09-11T08:18:08.267361+0000","last_undegraded":"2024-09-11T08:18:08.267361+0000","last_fullsized":"2024-09-11T08:18:08.267361+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:57.501885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:16:57.501885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050562519999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"239'15","reported_seq":304,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.011730+0000","last_change":"2024-09-11T08:17:39.580332+0000","last_active":"2024-09-11T08:18:10.011730+0000","last_peered":"2024-09-11T08:18:10.011730+0000","last_clean":"2024-09-11T08:18:10.011730+0000","last_became_active":"2024-09-11T08:15:56.526325+0000","last_became_peered":"2024-09-11T08:15:56.526325+0000","last_unstale":"2024-09-11T08:18:10.011730+0000","last_undegraded":"2024-09-11T08:18:10.011730+0000","last_fullsized":"2024-09-11T08:18:10.011730+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'15","last_scrub_stamp":"2024-09-11T08:17:39.580272+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-09-11T08:09:03.418457+0000","last_clean_scrub_stamp":"2024-09-11T08:17:39.580272+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:54.540761+0000","scrub_duration":0.0019585050000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":246,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:08.360621+0000","last_change":"2024-09-11T08:16:53.549060+0000","last_active":"2024-09-11T08:18:08.360621+0000","last_peered":"2024-09-11T08:18:08.360621+0000","last_clean":"2024-09-11T08:18:08.360621+0000","last_became_active":"2024-09-11T08:07:34.057774+0000","last_became_peered":"2024-09-11T08:07:34.057774+0000","last_unstale":"2024-09-11T08:18:08.360621+0000","last_undegraded":"2024-09-11T08:18:08.360621+0000","last_fullsized":"2024-09-11T08:18:08.360621+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:53.548967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:53.548967+0000","last_clean_scrub_stamp":"2024-09-11T08:16:53.548967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:17.071272+0000","scrub_duration":0.0067671800000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":188,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:07.988682+0000","last_change":"2024-09-11T08:16:53.669111+0000","last_active":"2024-09-11T08:18:07.988682+0000","last_peered":"2024-09-11T08:18:07.988682+0000","last_clean":"2024-09-11T08:18:07.988682+0000","last_became_active":"2024-09-11T08:14:24.790678+0000","last_became_peered":"2024-09-11T08:14:24.790678+0000","last_unstale":"2024-09-11T08:18:07.988682+0000","last_undegraded":"2024-09-11T08:18:07.988682+0000","last_fullsized":"2024-09-11T08:18:07.988682+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:53.669034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:16:53.669034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015615850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"239'19","reported_seq":315,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.541100+0000","last_change":"2024-09-11T08:17:31.038816+0000","last_active":"2024-09-11T08:18:10.541100+0000","last_peered":"2024-09-11T08:18:10.541100+0000","last_clean":"2024-09-11T08:18:10.541100+0000","last_became_active":"2024-09-11T08:15:56.514793+0000","last_became_peered":"2024-09-11T08:15:56.514793+0000","last_unstale":"2024-09-11T08:18:10.541100+0000","last_undegraded":"2024-09-11T08:18:10.541100+0000","last_fullsized":"2024-09-11T08:18:10.541100+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'13","last_scrub_stamp":"2024-09-11T08:17:31.038601+0000","last_deep_scrub":"84'13","last_deep_scrub_stamp":"2024-09-11T08:08:57.835634+0000","last_clean_scrub_stamp":"2024-09-11T08:17:31.038601+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:00.285015+0000","scrub_duration":0.0067181660000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":221,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:07.988357+0000","last_change":"2024-09-11T08:17:29.790967+0000","last_active":"2024-09-11T08:18:07.988357+0000","last_peered":"2024-09-11T08:18:07.988357+0000","last_clean":"2024-09-11T08:18:07.988357+0000","last_became_active":"2024-09-11T08:14:24.786021+0000","last_became_peered":"2024-09-11T08:14:24.786021+0000","last_unstale":"2024-09-11T08:18:07.988357+0000","last_undegraded":"2024-09-11T08:18:07.988357+0000","last_fullsized":"2024-09-11T08:18:07.988357+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:29.790887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:30.471620+0000","last_clean_scrub_stamp":"2024-09-11T08:17:29.790887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:49.599776+0000","scrub_duration":0.001553074,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":265,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.011618+0000","last_change":"2024-09-11T08:16:59.670424+0000","last_active":"2024-09-11T08:18:10.011618+0000","last_peered":"2024-09-11T08:18:10.011618+0000","last_clean":"2024-09-11T08:18:10.011618+0000","last_became_active":"2024-09-11T08:15:56.525717+0000","last_became_peered":"2024-09-11T08:15:56.525717+0000","last_unstale":"2024-09-11T08:18:10.011618+0000","last_undegraded":"2024-09-11T08:18:10.011618+0000","last_fullsized":"2024-09-11T08:18:10.011618+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:59.670299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:04.370734+0000","last_clean_scrub_stamp":"2024-09-11T08:16:59.670299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026338139999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":262,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.183632+0000","last_change":"2024-09-11T08:15:56.525429+0000","last_active":"2024-09-11T08:18:10.183632+0000","last_peered":"2024-09-11T08:18:10.183632+0000","last_clean":"2024-09-11T08:18:10.183632+0000","last_became_active":"2024-09-11T08:15:56.525120+0000","last_became_peered":"2024-09-11T08:15:56.525120+0000","last_unstale":"2024-09-11T08:18:10.183632+0000","last_undegraded":"2024-09-11T08:18:10.183632+0000","last_fullsized":"2024-09-11T08:18:10.183632+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:54.184372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:54.184372+0000","last_clean_scrub_stamp":"2024-09-11T08:15:54.184372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013922767000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"239'15","reported_seq":330,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:08.360610+0000","last_change":"2024-09-11T08:17:34.566504+0000","last_active":"2024-09-11T08:18:08.360610+0000","last_peered":"2024-09-11T08:18:08.360610+0000","last_clean":"2024-09-11T08:18:08.360610+0000","last_became_active":"2024-09-11T08:15:56.532513+0000","last_became_peered":"2024-09-11T08:15:56.532513+0000","last_unstale":"2024-09-11T08:18:08.360610+0000","last_undegraded":"2024-09-11T08:18:08.360610+0000","last_fullsized":"2024-09-11T08:18:08.360610+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:17:34.566230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:17:34.566230+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:41.796251+0000","scrub_duration":0.0047860990000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":268,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.540729+0000","last_change":"2024-09-11T08:16:52.661667+0000","last_active":"2024-09-11T08:18:10.540729+0000","last_peered":"2024-09-11T08:18:10.540729+0000","last_clean":"2024-09-11T08:18:10.540729+0000","last_became_active":"2024-09-11T08:15:56.513278+0000","last_became_peered":"2024-09-11T08:15:56.513278+0000","last_unstale":"2024-09-11T08:18:10.540729+0000","last_undegraded":"2024-09-11T08:18:10.540729+0000","last_fullsized":"2024-09-11T08:18:10.540729+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:52.661591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:52.661591+0000","last_clean_scrub_stamp":"2024-09-11T08:16:52.661591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045159830000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"239'13","reported_seq":392,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.011093+0000","last_change":"2024-09-11T08:17:37.558961+0000","last_active":"2024-09-11T08:18:10.011093+0000","last_peered":"2024-09-11T08:18:10.011093+0000","last_clean":"2024-09-11T08:18:10.011093+0000","last_became_active":"2024-09-11T08:17:37.558224+0000","last_became_peered":"2024-09-11T08:17:37.558224+0000","last_unstale":"2024-09-11T08:18:10.011093+0000","last_undegraded":"2024-09-11T08:18:10.011093+0000","last_fullsized":"2024-09-11T08:18:10.011093+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'12","last_scrub_stamp":"2024-09-11T08:16:52.592856+0000","last_deep_scrub":"222'12","last_deep_scrub_stamp":"2024-09-11T08:16:52.592856+0000","last_clean_scrub_stamp":"2024-09-11T08:16:52.592856+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:14.648828+0000","scrub_duration":0.0012487329999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":235,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:07.984077+0000","last_change":"2024-09-11T08:17:53.176053+0000","last_active":"2024-09-11T08:18:07.984077+0000","last_peered":"2024-09-11T08:18:07.984077+0000","last_clean":"2024-09-11T08:18:07.984077+0000","last_became_active":"2024-09-11T08:07:36.068493+0000","last_became_peered":"2024-09-11T08:07:36.068493+0000","last_unstale":"2024-09-11T08:18:07.984077+0000","last_undegraded":"2024-09-11T08:18:07.984077+0000","last_fullsized":"2024-09-11T08:18:07.984077+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:53.175994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:53.175994+0000","last_clean_scrub_stamp":"2024-09-11T08:17:53.175994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:04.570339+0000","scrub_duration":0.001353499,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"239'20","reported_seq":111,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.183597+0000","last_change":"2024-09-11T08:17:53.206017+0000","last_active":"2024-09-11T08:18:10.183597+0000","last_peered":"2024-09-11T08:18:10.183597+0000","last_clean":"2024-09-11T08:18:10.183597+0000","last_became_active":"2024-09-11T08:17:37.554357+0000","last_became_peered":"2024-09-11T08:17:37.554357+0000","last_unstale":"2024-09-11T08:18:10.183597+0000","last_undegraded":"2024-09-11T08:18:10.183597+0000","last_fullsized":"2024-09-11T08:18:10.183597+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'20","last_scrub_stamp":"2024-09-11T08:17:53.205922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:17:53.205922+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:22.709127+0000","scrub_duration":0.0017765369999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":63,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:07.988460+0000","last_change":"2024-09-11T08:17:03.708286+0000","last_active":"2024-09-11T08:18:07.988460+0000","last_peered":"2024-09-11T08:18:07.988460+0000","last_clean":"2024-09-11T08:18:07.988460+0000","last_became_active":"2024-09-11T08:15:56.518515+0000","last_became_peered":"2024-09-11T08:15:56.518515+0000","last_unstale":"2024-09-11T08:18:07.988460+0000","last_undegraded":"2024-09-11T08:18:07.988460+0000","last_fullsized":"2024-09-11T08:18:07.988460+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:03.708205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:17:03.708205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:30.336677+0000","scrub_duration":0.0039365750000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":383,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.324620+0000","last_change":"2024-09-11T08:17:44.632426+0000","last_active":"2024-09-11T08:18:10.324620+0000","last_peered":"2024-09-11T08:18:10.324620+0000","last_clean":"2024-09-11T08:18:10.324620+0000","last_became_active":"2024-09-11T08:07:34.060736+0000","last_became_peered":"2024-09-11T08:07:34.060736+0000","last_unstale":"2024-09-11T08:18:10.324620+0000","last_undegraded":"2024-09-11T08:18:10.324620+0000","last_fullsized":"2024-09-11T08:18:10.324620+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:17:44.632353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:17:44.632353+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:03.078582+0000","scrub_duration":0.001735232,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":262,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.540895+0000","last_change":"2024-09-11T08:17:42.030917+0000","last_active":"2024-09-11T08:18:10.540895+0000","last_peered":"2024-09-11T08:18:10.540895+0000","last_clean":"2024-09-11T08:18:10.540895+0000","last_became_active":"2024-09-11T08:15:56.513949+0000","last_became_peered":"2024-09-11T08:15:56.513949+0000","last_unstale":"2024-09-11T08:18:10.540895+0000","last_undegraded":"2024-09-11T08:18:10.540895+0000","last_fullsized":"2024-09-11T08:18:10.540895+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:42.030854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:17:42.030854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:05.644587+0000","scrub_duration":0.0017212379999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"239'19","reported_seq":270,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:07.983631+0000","last_change":"2024-09-11T08:16:55.089638+0000","last_active":"2024-09-11T08:18:07.983631+0000","last_peered":"2024-09-11T08:18:07.983631+0000","last_clean":"2024-09-11T08:18:07.983631+0000","last_became_active":"2024-09-11T08:07:32.047981+0000","last_became_peered":"2024-09-11T08:07:32.047981+0000","last_unstale":"2024-09-11T08:18:07.983631+0000","last_undegraded":"2024-09-11T08:18:07.983631+0000","last_fullsized":"2024-09-11T08:18:07.983631+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-11T08:16:55.089545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:16:55.089545+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:12.681553+0000","scrub_duration":0.0037132879999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":268,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:07.983973+0000","last_change":"2024-09-11T08:17:31.186217+0000","last_active":"2024-09-11T08:18:07.983973+0000","last_peered":"2024-09-11T08:18:07.983973+0000","last_clean":"2024-09-11T08:18:07.983973+0000","last_became_active":"2024-09-11T08:15:56.523574+0000","last_became_peered":"2024-09-11T08:15:56.523574+0000","last_unstale":"2024-09-11T08:18:07.983973+0000","last_undegraded":"2024-09-11T08:18:07.983973+0000","last_fullsized":"2024-09-11T08:18:07.983973+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:31.186045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:17:31.186045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:33.322805+0000","scrub_duration":0.0016617730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":251,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.488187+0000","last_change":"2024-09-11T08:17:03.475878+0000","last_active":"2024-09-11T08:18:10.488187+0000","last_peered":"2024-09-11T08:18:10.488187+0000","last_clean":"2024-09-11T08:18:10.488187+0000","last_became_active":"2024-09-11T08:07:34.053137+0000","last_became_peered":"2024-09-11T08:07:34.053137+0000","last_unstale":"2024-09-11T08:18:10.488187+0000","last_undegraded":"2024-09-11T08:18:10.488187+0000","last_fullsized":"2024-09-11T08:18:10.488187+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:03.475805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:17:03.475805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:26.468256+0000","scrub_duration":0.012934157,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":36,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.540548+0000","last_change":"2024-09-11T08:17:52.926082+0000","last_active":"2024-09-11T08:18:10.540548+0000","last_peered":"2024-09-11T08:18:10.540548+0000","last_clean":"2024-09-11T08:18:10.540548+0000","last_became_active":"2024-09-11T08:17:37.556871+0000","last_became_peered":"2024-09-11T08:17:37.556871+0000","last_unstale":"2024-09-11T08:18:10.540548+0000","last_undegraded":"2024-09-11T08:18:10.540548+0000","last_fullsized":"2024-09-11T08:18:10.540548+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:52.925992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:17:52.925992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:18.928220+0000","scrub_duration":0.0087812989999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"239'15","reported_seq":285,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.488370+0000","last_change":"2024-09-11T08:17:43.635561+0000","last_active":"2024-09-11T08:18:10.488370+0000","last_peered":"2024-09-11T08:18:10.488370+0000","last_clean":"2024-09-11T08:18:10.488370+0000","last_became_active":"2024-09-11T08:07:32.049368+0000","last_became_peered":"2024-09-11T08:07:32.049368+0000","last_unstale":"2024-09-11T08:18:10.488370+0000","last_undegraded":"2024-09-11T08:18:10.488370+0000","last_fullsized":"2024-09-11T08:18:10.488370+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'15","last_scrub_stamp":"2024-09-11T08:17:43.635506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:17:43.635506+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:07.068338+0000","scrub_duration":0.0022043800000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":63,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:07.988580+0000","last_change":"2024-09-11T08:16:51.628468+0000","last_active":"2024-09-11T08:18:07.988580+0000","last_peered":"2024-09-11T08:18:07.988580+0000","last_clean":"2024-09-11T08:18:07.988580+0000","last_became_active":"2024-09-11T08:15:56.521570+0000","last_became_peered":"2024-09-11T08:15:56.521570+0000","last_unstale":"2024-09-11T08:18:07.988580+0000","last_undegraded":"2024-09-11T08:18:07.988580+0000","last_fullsized":"2024-09-11T08:18:07.988580+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:51.628378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:16:51.628378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001961422,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":36,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.540721+0000","last_change":"2024-09-11T08:17:39.028807+0000","last_active":"2024-09-11T08:18:10.540721+0000","last_peered":"2024-09-11T08:18:10.540721+0000","last_clean":"2024-09-11T08:18:10.540721+0000","last_became_active":"2024-09-11T08:17:37.557645+0000","last_became_peered":"2024-09-11T08:17:37.557645+0000","last_unstale":"2024-09-11T08:18:10.540721+0000","last_undegraded":"2024-09-11T08:18:10.540721+0000","last_fullsized":"2024-09-11T08:18:10.540721+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:39.028741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:01.218475+0000","last_clean_scrub_stamp":"2024-09-11T08:17:39.028741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:02.357273+0000","scrub_duration":0.0012042629999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":235,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.540462+0000","last_change":"2024-09-11T08:16:39.635237+0000","last_active":"2024-09-11T08:18:10.540462+0000","last_peered":"2024-09-11T08:18:10.540462+0000","last_clean":"2024-09-11T08:18:10.540462+0000","last_became_active":"2024-09-11T08:07:36.077406+0000","last_became_peered":"2024-09-11T08:07:36.077406+0000","last_unstale":"2024-09-11T08:18:10.540462+0000","last_undegraded":"2024-09-11T08:18:10.540462+0000","last_fullsized":"2024-09-11T08:18:10.540462+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:39.635151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:09.973943+0000","last_clean_scrub_stamp":"2024-09-11T08:16:39.635151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0062349479999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"239'19","reported_seq":322,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.010999+0000","last_change":"2024-09-11T08:16:53.599766+0000","last_active":"2024-09-11T08:18:10.010999+0000","last_peered":"2024-09-11T08:18:10.010999+0000","last_clean":"2024-09-11T08:18:10.010999+0000","last_became_active":"2024-09-11T08:15:56.520058+0000","last_became_peered":"2024-09-11T08:15:56.520058+0000","last_unstale":"2024-09-11T08:18:10.010999+0000","last_undegraded":"2024-09-11T08:18:10.010999+0000","last_fullsized":"2024-09-11T08:18:10.010999+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-11T08:16:53.599677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:16:53.599677+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:20.621343+0000","scrub_duration":0.0071859539999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":267,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:08.360666+0000","last_change":"2024-09-11T08:15:56.531914+0000","last_active":"2024-09-11T08:18:08.360666+0000","last_peered":"2024-09-11T08:18:08.360666+0000","last_clean":"2024-09-11T08:18:08.360666+0000","last_became_active":"2024-09-11T08:15:56.531612+0000","last_became_peered":"2024-09-11T08:15:56.531612+0000","last_unstale":"2024-09-11T08:18:08.360666+0000","last_undegraded":"2024-09-11T08:18:08.360666+0000","last_fullsized":"2024-09-11T08:18:08.360666+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:15:49.478359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:15:49.478359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012202396000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":63,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:07.988027+0000","last_change":"2024-09-11T08:16:38.643733+0000","last_active":"2024-09-11T08:18:07.988027+0000","last_peered":"2024-09-11T08:18:07.988027+0000","last_clean":"2024-09-11T08:18:07.988027+0000","last_became_active":"2024-09-11T08:15:56.523728+0000","last_became_peered":"2024-09-11T08:15:56.523728+0000","last_unstale":"2024-09-11T08:18:07.988027+0000","last_undegraded":"2024-09-11T08:18:07.988027+0000","last_fullsized":"2024-09-11T08:18:07.988027+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:38.643651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:11:34.159053+0000","last_clean_scrub_stamp":"2024-09-11T08:16:38.643651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002366469,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":236,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.011413+0000","last_change":"2024-09-11T08:17:46.489320+0000","last_active":"2024-09-11T08:18:10.011413+0000","last_peered":"2024-09-11T08:18:10.011413+0000","last_clean":"2024-09-11T08:18:10.011413+0000","last_became_active":"2024-09-11T08:07:36.633368+0000","last_became_peered":"2024-09-11T08:07:36.633368+0000","last_unstale":"2024-09-11T08:18:10.011413+0000","last_undegraded":"2024-09-11T08:18:10.011413+0000","last_fullsized":"2024-09-11T08:18:10.011413+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:46.489220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:46.489220+0000","last_clean_scrub_stamp":"2024-09-11T08:17:46.489220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:04.268669+0000","scrub_duration":0.0015753690000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"239'8","reported_seq":252,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.010828+0000","last_change":"2024-09-11T08:17:50.580924+0000","last_active":"2024-09-11T08:18:10.010828+0000","last_peered":"2024-09-11T08:18:10.010828+0000","last_clean":"2024-09-11T08:18:10.010828+0000","last_became_active":"2024-09-11T08:07:32.045728+0000","last_became_peered":"2024-09-11T08:07:32.045728+0000","last_unstale":"2024-09-11T08:18:10.010828+0000","last_undegraded":"2024-09-11T08:18:10.010828+0000","last_fullsized":"2024-09-11T08:18:10.010828+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'8","last_scrub_stamp":"2024-09-11T08:17:50.580863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:17:50.580863+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:09.146709+0000","scrub_duration":0.001681075,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":251,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.540390+0000","last_change":"2024-09-11T08:17:48.965635+0000","last_active":"2024-09-11T08:18:10.540390+0000","last_peered":"2024-09-11T08:18:10.540390+0000","last_clean":"2024-09-11T08:18:10.540390+0000","last_became_active":"2024-09-11T08:07:30.049409+0000","last_became_peered":"2024-09-11T08:07:30.049409+0000","last_unstale":"2024-09-11T08:18:10.540390+0000","last_undegraded":"2024-09-11T08:18:10.540390+0000","last_fullsized":"2024-09-11T08:18:10.540390+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:48.965582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:17:48.965582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:52.654703+0000","scrub_duration":0.001302138,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":283,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:08.267262+0000","last_change":"2024-09-11T08:16:51.403463+0000","last_active":"2024-09-11T08:18:08.267262+0000","last_peered":"2024-09-11T08:18:08.267262+0000","last_clean":"2024-09-11T08:18:08.267262+0000","last_became_active":"2024-09-11T08:15:56.528607+0000","last_became_peered":"2024-09-11T08:15:56.528607+0000","last_unstale":"2024-09-11T08:18:08.267262+0000","last_undegraded":"2024-09-11T08:18:08.267262+0000","last_fullsized":"2024-09-11T08:18:08.267262+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:51.403370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:16:51.403370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002498068,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":63,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:07.988217+0000","last_change":"2024-09-11T08:17:31.857979+0000","last_active":"2024-09-11T08:18:07.988217+0000","last_peered":"2024-09-11T08:18:07.988217+0000","last_clean":"2024-09-11T08:18:07.988217+0000","last_became_active":"2024-09-11T08:15:56.521488+0000","last_became_peered":"2024-09-11T08:15:56.521488+0000","last_unstale":"2024-09-11T08:18:07.988217+0000","last_undegraded":"2024-09-11T08:18:07.988217+0000","last_fullsized":"2024-09-11T08:18:07.988217+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:31.857906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:17:31.857906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:43.754475+0000","scrub_duration":0.011557305,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":246,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.488267+0000","last_change":"2024-09-11T08:16:50.585648+0000","last_active":"2024-09-11T08:18:10.488267+0000","last_peered":"2024-09-11T08:18:10.488267+0000","last_clean":"2024-09-11T08:18:10.488267+0000","last_became_active":"2024-09-11T08:07:36.071468+0000","last_became_peered":"2024-09-11T08:07:36.071468+0000","last_unstale":"2024-09-11T08:18:10.488267+0000","last_undegraded":"2024-09-11T08:18:10.488267+0000","last_fullsized":"2024-09-11T08:18:10.488267+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:50.585566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:50.585566+0000","last_clean_scrub_stamp":"2024-09-11T08:16:50.585566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:20.341597+0000","scrub_duration":0.0073189980000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":36,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-11T08:18:10.540863+0000","last_change":"2024-09-11T08:17:51.950776+0000","last_active":"2024-09-11T08:18:10.540863+0000","last_peered":"2024-09-11T08:18:10.540863+0000","last_clean":"2024-09-11T08:18:10.540863+0000","last_became_active":"2024-09-11T08:17:37.554262+0000","last_became_peered":"2024-09-11T08:17:37.554262+0000","last_unstale":"2024-09-11T08:18:10.540863+0000","last_undegraded":"2024-09-11T08:18:10.540863+0000","last_fullsized":"2024-09-11T08:18:10.540863+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:51.950689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:17:51.950689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:08.838111+0000","scrub_duration":0.0014240730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"239'19","reported_seq":289,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:08.267689+0000","last_change":"2024-09-11T08:17:30.294601+0000","last_active":"2024-09-11T08:18:08.267689+0000","last_peered":"2024-09-11T08:18:08.267689+0000","last_clean":"2024-09-11T08:18:08.267689+0000","last_became_active":"2024-09-11T08:07:32.054656+0000","last_became_peered":"2024-09-11T08:07:32.054656+0000","last_unstale":"2024-09-11T08:18:08.267689+0000","last_undegraded":"2024-09-11T08:18:08.267689+0000","last_fullsized":"2024-09-11T08:18:08.267689+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-11T08:17:30.294514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:17:30.294514+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:44.544439+0000","scrub_duration":0.0051598069999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":81,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-11T08:18:08.267487+0000","last_change":"2024-09-11T08:16:46.497719+0000","last_active":"2024-09-11T08:18:08.267487+0000","last_peered":"2024-09-11T08:18:08.267487+0000","last_clean":"2024-09-11T08:18:08.267487+0000","last_became_active":"2024-09-11T08:15:08.769133+0000","last_became_peered":"2024-09-11T08:15:08.769133+0000","last_unstale":"2024-09-11T08:18:08.267487+0000","last_undegraded":"2024-09-11T08:18:08.267487+0000","last_fullsized":"2024-09-11T08:18:08.267487+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:46.497639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:16:46.497639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:18:14.479718+0000","scrub_duration":0.0048914830000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":1257635840,"num_objects":19190,"num_object_clones":0,"num_object_copies":57570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19190,"num_write_kb":1228160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3767205888,"data_stored":3767205888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19190,"ondisk_log_size":19190,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1127,"num_read_kb":884,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":2448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3425,"internal_metadata":0},"log_size":741,"ondisk_log_size":741,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":239,"num_read_kb":234,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":168,"seq":721554505776,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":921040,"kb_used_data":591048,"kb_used_omap":7,"kb_used_meta":329976,"kb_avail":92824112,"statfs":{"total":95995035648,"available":95051890688,"internally_reserved":0,"allocated":605233152,"data_stored":604845960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8027,"internal_metadata":337895589},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:17:27 2024","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":0.99399999999999999,"15min":0.88200000000000001},"min":{"1min":0.24199999999999999,"5min":0.223,"15min":0.223},"max":{"1min":3.8260000000000001,"5min":4.673,"15min":4.673},"last":1.73},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.0029999999999999,"15min":0.997},"min":{"1min":0.22600000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":5.2610000000000001,"5min":5.2610000000000001,"15min":5.2610000000000001},"last":0.39400000000000002}]},{"osd":1,"last update":"Wed Sep 11 08:17:27 2024","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":2.3119999999999998,"15min":2.0680000000000001},"min":{"1min":0.28799999999999998,"5min":0.25,"15min":0.25},"max":{"1min":5.431,"5min":42.082999999999998,"15min":42.082999999999998},"last":0.85899999999999999},{"interface":"front","average":{"1min":1.075,"5min":2.1499999999999999,"15min":2.0800000000000001},"min":{"1min":0.28699999999999998,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":2.5990000000000002,"5min":42.151000000000003,"15min":42.151000000000003},"last":1.8129999999999999}]},{"osd":2,"last update":"Wed Sep 11 08:17:27 2024","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.93600000000000005,"15min":1.006},"min":{"1min":0.26500000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":2.984,"5min":3.9630000000000001,"15min":3.9630000000000001},"last":0.35899999999999999},{"interface":"front","average":{"1min":1.1479999999999999,"5min":0.81799999999999995,"15min":0.753},"min":{"1min":0.30299999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":5.5170000000000003,"5min":5.5170000000000003,"15min":5.5170000000000003},"last":0.73899999999999999}]},{"osd":3,"last update":"Wed Sep 11 08:17:27 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":1.319,"15min":1.155},"min":{"1min":0.28499999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":3.0830000000000002,"5min":21.460000000000001,"15min":21.460000000000001},"last":1.8400000000000001},{"interface":"front","average":{"1min":0.995,"5min":1.5569999999999999,"15min":1.4690000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":5.585,"5min":21.440000000000001,"15min":21.440000000000001},"last":1.254}]},{"osd":4,"last update":"Wed Sep 11 08:17:27 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.66300000000000003,"15min":0.57099999999999995},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":1.544,"5min":4.141,"15min":4.141},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.63,"5min":0.71299999999999997,"15min":0.73999999999999999},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.423,"5min":3.157,"15min":3.157},"last":1.665}]},{"osd":5,"last update":"Wed Sep 11 08:17:27 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.88,"15min":0.82599999999999996},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.8220000000000001,"5min":4.1379999999999999,"15min":4.1379999999999999},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.621,"5min":0.79600000000000004,"15min":0.76500000000000001},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.4470000000000001,"5min":3.907,"15min":3.907},"last":1.7050000000000001}]},{"osd":6,"last update":"Wed Sep 11 08:17:27 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.77800000000000002,"15min":0.78800000000000003},"min":{"1min":0.33400000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.92500000000000004,"5min":6.0609999999999999,"15min":6.0609999999999999},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.81200000000000006,"15min":0.79700000000000004},"min":{"1min":0.38,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.5229999999999999,"5min":3.6829999999999998,"15min":3.6829999999999998},"last":0.88500000000000001}]}]},{"osd":6,"up_from":45,"seq":193273528459,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":613056,"kb_used_data":303224,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93132096,"statfs":{"total":95995035648,"available":95367266304,"internally_reserved":0,"allocated":310501376,"data_stored":310130391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:18:16 2024","interfaces":[{"interface":"back","average":{"1min":1.96,"5min":1.165,"15min":1.181},"min":{"1min":0.28299999999999997,"5min":0.20399999999999999,"15min":0.185},"max":{"1min":22.012,"5min":22.012,"15min":45.066000000000003},"last":0.74099999999999999},{"interface":"front","average":{"1min":1.968,"5min":1.4019999999999999,"15min":1.2050000000000001},"min":{"1min":0.25800000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":21.963999999999999,"5min":21.963999999999999,"15min":43.899000000000001},"last":0.45500000000000002}]},{"osd":1,"last update":"Wed Sep 11 08:18:16 2024","interfaces":[{"interface":"back","average":{"1min":1.7190000000000001,"5min":1.1259999999999999,"15min":0.98799999999999999},"min":{"1min":0.26100000000000001,"5min":0.214,"15min":0.214},"max":{"1min":20.321000000000002,"5min":20.321000000000002,"15min":44.658000000000001},"last":0.65200000000000002},{"interface":"front","average":{"1min":2.0449999999999999,"5min":1.347,"15min":1.115},"min":{"1min":0.316,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":19.652000000000001,"5min":19.652000000000001,"15min":44.694000000000003},"last":0.625}]},{"osd":2,"last update":"Wed Sep 11 08:18:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.84799999999999998,"15min":0.73099999999999998},"min":{"1min":0.30299999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":1.7869999999999999,"5min":5.9770000000000003,"15min":5.9770000000000003},"last":0.57499999999999996},{"interface":"front","average":{"1min":1.133,"5min":1.22,"15min":0.94399999999999995},"min":{"1min":0.26800000000000002,"5min":0.252,"15min":0.23300000000000001},"max":{"1min":7.0549999999999997,"5min":7.4809999999999999,"15min":13.346},"last":0.58899999999999997}]},{"osd":3,"last update":"Wed Sep 11 08:18:16 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.93500000000000005,"15min":1.036},"min":{"1min":0.316,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":2.7610000000000001,"5min":4.875,"15min":36.061},"last":0.503},{"interface":"front","average":{"1min":1.121,"5min":1.071,"15min":1.0640000000000001},"min":{"1min":0.35399999999999998,"5min":0.28699999999999998,"15min":0.26800000000000002},"max":{"1min":5.5720000000000001,"5min":5.5720000000000001,"15min":36.093000000000004},"last":0.60199999999999998}]},{"osd":4,"last update":"Wed Sep 11 08:18:16 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72599999999999998,"15min":0.51000000000000001},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20200000000000001},"max":{"1min":4.492,"5min":4.492,"15min":4.492},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.745,"5min":0.73099999999999998,"15min":0.53900000000000003},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":4.1529999999999996,"5min":4.468,"15min":4.468},"last":0.42299999999999999}]},{"osd":5,"last update":"Wed Sep 11 08:18:16 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.91700000000000004,"15min":0.629},"min":{"1min":0.29199999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":4.1760000000000002,"5min":7.4459999999999997,"15min":7.4459999999999997},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.76900000000000002,"15min":0.56000000000000005},"min":{"1min":0.28100000000000003,"5min":0.28000000000000003,"15min":0.22600000000000001},"max":{"1min":1.4670000000000001,"5min":3.4329999999999998,"15min":3.4329999999999998},"last":0.56000000000000005}]},{"osd":7,"last update":"Wed Sep 11 08:17:25 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":1.0720000000000001,"15min":1.121},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":1.911,"5min":8.4580000000000002,"15min":8.4580000000000002},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.872,"15min":0.89200000000000002},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.8480000000000001,"5min":3.593,"15min":3.593},"last":0.64500000000000002}]}]},{"osd":5,"up_from":38,"seq":163208757392,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1016064,"kb_used_data":710392,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":92729088,"statfs":{"total":95995035648,"available":94954586112,"internally_reserved":0,"allocated":727441408,"data_stored":727069440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:17:54 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":1.0860000000000001,"15min":0.82799999999999996},"min":{"1min":0.214,"5min":0.184,"15min":0.184},"max":{"1min":5.0990000000000002,"5min":7.9470000000000001,"15min":7.9470000000000001},"last":0.56100000000000005},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.2829999999999999,"15min":0.91900000000000004},"min":{"1min":0.311,"5min":0.23000000000000001,"15min":0.20399999999999999},"max":{"1min":5.6210000000000004,"5min":13.077,"15min":13.077},"last":1.1839999999999999}]},{"osd":1,"last update":"Wed Sep 11 08:17:54 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":1.248,"15min":1.2130000000000001},"min":{"1min":0.251,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":2.105,"5min":26.709,"15min":26.709},"last":1.294},{"interface":"front","average":{"1min":1.02,"5min":1.248,"15min":0.93000000000000005},"min":{"1min":0.311,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.8570000000000002,"5min":26.754000000000001,"15min":26.754000000000001},"last":0.376}]},{"osd":2,"last update":"Wed Sep 11 08:17:54 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":1.0289999999999999,"15min":1.0760000000000001},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.3100000000000001,"5min":10.891,"15min":16.018000000000001},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.875,"15min":0.70899999999999996},"min":{"1min":0.33200000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":1.296,"5min":7.5250000000000004,"15min":7.5250000000000004},"last":0.434}]},{"osd":3,"last update":"Wed Sep 11 08:17:54 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":1.1930000000000001,"15min":0.88300000000000001},"min":{"1min":0.25800000000000001,"5min":0.25,"15min":0.25},"max":{"1min":3.2120000000000002,"5min":24.931000000000001,"15min":24.931000000000001},"last":0.5},{"interface":"front","average":{"1min":1.002,"5min":1.4179999999999999,"15min":0.94899999999999995},"min":{"1min":0.36499999999999999,"5min":0.27700000000000002,"15min":0.251},"max":{"1min":4.3150000000000004,"5min":24.861999999999998,"15min":24.861999999999998},"last":0.89500000000000002}]},{"osd":4,"last update":"Wed Sep 11 08:17:54 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.627,"15min":0.51000000000000001},"min":{"1min":0.27200000000000002,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":2.1269999999999998,"5min":3.5979999999999999,"15min":3.5979999999999999},"last":1.0580000000000001},{"interface":"front","average":{"1min":0.437,"5min":0.58799999999999997,"15min":0.49199999999999999},"min":{"1min":0.28100000000000003,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.93700000000000006,"5min":4.9210000000000003,"15min":4.9210000000000003},"last":0.40600000000000003}]},{"osd":6,"last update":"Wed Sep 11 08:18:13 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.59099999999999997,"15min":0.48699999999999999},"min":{"1min":0.26900000000000002,"5min":0.253,"15min":0.23400000000000001},"max":{"1min":0.92500000000000004,"5min":3.1869999999999998,"15min":3.1869999999999998},"last":1.075},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.71699999999999997,"15min":0.56399999999999995},"min":{"1min":0.309,"5min":0.28599999999999998,"15min":0.26900000000000002},"max":{"1min":2.0499999999999998,"5min":3.9489999999999998,"15min":3.9489999999999998},"last":1.2390000000000001}]},{"osd":7,"last update":"Wed Sep 11 08:17:28 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.84999999999999998,"15min":0.83699999999999997},"min":{"1min":0.33900000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":2.2759999999999998,"5min":3.5459999999999998,"15min":3.5459999999999998},"last":1.609},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.78700000000000003,"15min":0.77100000000000002},"min":{"1min":0.35799999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.486,"5min":3.1019999999999999,"15min":3.1019999999999999},"last":0.92700000000000005}]}]},{"osd":4,"up_from":31,"seq":133143986325,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":501204,"kb_used_data":195532,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93243948,"statfs":{"total":95995035648,"available":95481802752,"internally_reserved":0,"allocated":200224768,"data_stored":199833634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:17:21 2024","interfaces":[{"interface":"back","average":{"1min":3.895,"5min":1.546,"15min":1.292},"min":{"1min":0.22800000000000001,"5min":0.19,"15min":0.19},"max":{"1min":60.497999999999998,"5min":60.497999999999998,"15min":60.497999999999998},"last":0.222},{"interface":"front","average":{"1min":3.77,"5min":1.429,"15min":1.1479999999999999},"min":{"1min":0.24299999999999999,"5min":0.221,"15min":0.221},"max":{"1min":60.484999999999999,"5min":60.484999999999999,"15min":60.484999999999999},"last":0.70399999999999996}]},{"osd":1,"last update":"Wed Sep 11 08:17:21 2024","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":1.01,"15min":0.78900000000000003},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.214},"max":{"1min":3.1099999999999999,"5min":4.7320000000000002,"15min":13.207000000000001},"last":0.28699999999999998},{"interface":"front","average":{"1min":1.0600000000000001,"5min":0.94399999999999995,"15min":0.75700000000000001},"min":{"1min":0.26800000000000002,"5min":0.24299999999999999,"15min":0.22700000000000001},"max":{"1min":3.992,"5min":5.8869999999999996,"15min":5.8869999999999996},"last":0.308}]},{"osd":2,"last update":"Wed Sep 11 08:17:21 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.91600000000000004,"15min":0.747},"min":{"1min":0.255,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":1.651,"5min":7.976,"15min":7.976},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.77900000000000003,"15min":0.71999999999999997},"min":{"1min":0.28399999999999997,"5min":0.248,"15min":0.248},"max":{"1min":1.802,"5min":3.1389999999999998,"15min":5.5670000000000002},"last":0.67000000000000004}]},{"osd":3,"last update":"Wed Sep 11 08:17:21 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":1.1830000000000001,"15min":1.054},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":1.8600000000000001,"5min":19.686,"15min":26.884},"last":0.755},{"interface":"front","average":{"1min":1.3069999999999999,"5min":1.3759999999999999,"15min":1.107},"min":{"1min":0.35499999999999998,"5min":0.246,"15min":0.246},"max":{"1min":2.9510000000000001,"5min":19.882000000000001,"15min":26.783000000000001},"last":0.33100000000000002}]},{"osd":5,"last update":"Wed Sep 11 08:17:49 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.58299999999999996,"15min":0.46200000000000002},"min":{"1min":0.26500000000000001,"5min":0.23599999999999999,"15min":0.23499999999999999},"max":{"1min":1.6850000000000001,"5min":2.7229999999999999,"15min":2.7229999999999999},"last":1.0169999999999999},{"interface":"front","average":{"1min":0.93799999999999994,"5min":0.64700000000000002,"15min":0.48599999999999999},"min":{"1min":0.33100000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":5.2549999999999999,"5min":5.2549999999999999,"15min":5.2549999999999999},"last":1.0009999999999999}]},{"osd":6,"last update":"Wed Sep 11 08:18:15 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":1.044,"15min":0.63100000000000001},"min":{"1min":0.27700000000000002,"5min":0.25,"15min":0.23899999999999999},"max":{"1min":4.6799999999999997,"5min":48.808,"15min":48.808},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.48499999999999999,"5min":1.0489999999999999,"15min":0.66500000000000004},"min":{"1min":0.32500000000000001,"5min":0.26800000000000002,"15min":0.26200000000000001},"max":{"1min":0.95299999999999996,"5min":48.746000000000002,"15min":48.746000000000002},"last":0.95299999999999996}]},{"osd":7,"last update":"Wed Sep 11 08:17:34 2024","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.65700000000000003,"15min":0.53700000000000003},"min":{"1min":0.30499999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":5.3449999999999998,"5min":5.3449999999999998,"15min":5.3449999999999998},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.749,"15min":0.69099999999999995},"min":{"1min":0.307,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":4.7160000000000002,"5min":4.7160000000000002,"15min":4.7160000000000002},"last":0.68600000000000005}]}]},{"osd":3,"up_from":25,"seq":107374182554,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":871840,"kb_used_data":566168,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":92873312,"statfs":{"total":95995035648,"available":95102271488,"internally_reserved":0,"allocated":579756032,"data_stored":579352081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:18:12 2024","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":1.8049999999999999,"15min":1.1799999999999999},"min":{"1min":0.17799999999999999,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":4.0739999999999998,"5min":60.469999999999999,"15min":60.469999999999999},"last":0.307},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.758,"15min":1.284},"min":{"1min":0.187,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":6.3620000000000001,"5min":60.488999999999997,"15min":60.488999999999997},"last":1.462}]},{"osd":1,"last update":"Wed Sep 11 08:18:12 2024","interfaces":[{"interface":"back","average":{"1min":1.093,"5min":1.919,"15min":1.181},"min":{"1min":0.219,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":6.3940000000000001,"5min":60.578000000000003,"15min":60.578000000000003},"last":0.249},{"interface":"front","average":{"1min":1.3380000000000001,"5min":2.3090000000000002,"15min":1.4379999999999999},"min":{"1min":0.21099999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":8.3819999999999997,"5min":63.674999999999997,"15min":63.674999999999997},"last":1.5760000000000001}]},{"osd":2,"last update":"Wed Sep 11 08:18:12 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.819,"15min":1.3280000000000001},"min":{"1min":0.27600000000000002,"5min":0.185,"15min":0.185},"max":{"1min":8.4179999999999993,"5min":60.645000000000003,"15min":60.645000000000003},"last":1.5489999999999999},{"interface":"front","average":{"1min":0.89200000000000002,"5min":1.667,"15min":1.2030000000000001},"min":{"1min":0.24199999999999999,"5min":0.20699999999999999,"15min":0.20599999999999999},"max":{"1min":4.0970000000000004,"5min":60.723999999999997,"15min":60.723999999999997},"last":0.54800000000000004}]},{"osd":4,"last update":"Wed Sep 11 08:17:23 2024","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.8169999999999999,"15min":1.105},"min":{"1min":0.29499999999999998,"5min":0.28100000000000003,"15min":0.24299999999999999},"max":{"1min":5.5650000000000004,"5min":61.290999999999997,"15min":61.290999999999997},"last":1.3919999999999999},{"interface":"front","average":{"1min":1.081,"5min":1.752,"15min":1.0609999999999999},"min":{"1min":0.311,"5min":0.28899999999999998,"15min":0.27200000000000002},"max":{"1min":6.1340000000000003,"5min":60.856000000000002,"15min":60.856000000000002},"last":0.33500000000000002}]},{"osd":5,"last update":"Wed Sep 11 08:17:53 2024","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":2.2869999999999999,"15min":1.276},"min":{"1min":0.34399999999999997,"5min":0.28199999999999997,"15min":0.26500000000000001},"max":{"1min":4.1509999999999998,"5min":60.914999999999999,"15min":60.914999999999999},"last":1.1080000000000001},{"interface":"front","average":{"1min":1.3200000000000001,"5min":2.403,"15min":1.3680000000000001},"min":{"1min":0.38400000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.3380000000000001,"5min":60.787999999999997,"15min":60.787999999999997},"last":1.2969999999999999}]},{"osd":6,"last update":"Wed Sep 11 08:18:12 2024","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":2.1949999999999998,"15min":1.3540000000000001},"min":{"1min":0.33400000000000002,"5min":0.316,"15min":0.316},"max":{"1min":6.3150000000000004,"5min":60.817,"15min":60.817},"last":1.264},{"interface":"front","average":{"1min":1.484,"5min":2.242,"15min":1.4950000000000001},"min":{"1min":0.40699999999999997,"5min":0.33100000000000002,"15min":0.307},"max":{"1min":6.8200000000000003,"5min":61.264000000000003,"15min":61.264000000000003},"last":1.518}]},{"osd":7,"last update":"Wed Sep 11 08:17:27 2024","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":2.9380000000000002,"15min":3.0739999999999998},"min":{"1min":0.40999999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":6.2770000000000001,"5min":60.847000000000001,"15min":60.847000000000001},"last":1.3360000000000001},{"interface":"front","average":{"1min":0.95799999999999996,"5min":2.8100000000000001,"15min":2.9849999999999999},"min":{"1min":0.36699999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":4.0309999999999997,"5min":60.883000000000003,"15min":60.883000000000003},"last":0.57999999999999996}]}]},{"osd":2,"up_from":18,"seq":77309411485,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":512512,"kb_used_data":211000,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93232640,"statfs":{"total":95995035648,"available":95470223360,"internally_reserved":0,"allocated":216064000,"data_stored":215691944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:17:49 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":1.9870000000000001,"15min":1.206},"min":{"1min":0.22600000000000001,"5min":0.21299999999999999,"15min":0.17199999999999999},"max":{"1min":2.9020000000000001,"5min":43.539999999999999,"15min":43.539999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":1.093,"5min":2.101,"15min":1.258},"min":{"1min":0.27600000000000002,"5min":0.185,"15min":0.185},"max":{"1min":3.2970000000000002,"5min":41.962000000000003,"15min":41.962000000000003},"last":0.51400000000000001}]},{"osd":1,"last update":"Wed Sep 11 08:17:49 2024","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.4630000000000001,"15min":1.012},"min":{"1min":0.28199999999999997,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":4.633,"5min":13.138999999999999,"15min":13.138999999999999},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.84999999999999998,"5min":1.532,"15min":1.04},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20200000000000001},"max":{"1min":3.3260000000000001,"5min":10.167999999999999,"15min":10.167999999999999},"last":0.83899999999999997}]},{"osd":3,"last update":"Wed Sep 11 08:18:09 2024","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":1.3029999999999999,"15min":1.069},"min":{"1min":0.23599999999999999,"5min":0.21299999999999999,"15min":0.188},"max":{"1min":3.1419999999999999,"5min":8.1989999999999998,"15min":8.1989999999999998},"last":0.94199999999999995},{"interface":"front","average":{"1min":0.50800000000000001,"5min":1.226,"15min":1.0109999999999999},"min":{"1min":0.24399999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":1.391,"5min":8.0640000000000001,"15min":8.3219999999999992},"last":0.77400000000000002}]},{"osd":4,"last update":"Wed Sep 11 08:17:29 2024","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":1.1339999999999999,"15min":0.84899999999999998},"min":{"1min":0.27900000000000003,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":3.0379999999999998,"5min":7.2480000000000002,"15min":7.806},"last":0.88500000000000001},{"interface":"front","average":{"1min":0.83099999999999996,"5min":1.0129999999999999,"15min":0.88700000000000001},"min":{"1min":0.27500000000000002,"5min":0.252,"15min":0.252},"max":{"1min":2.1850000000000001,"5min":5.4740000000000002,"15min":12.129},"last":0.60099999999999998}]},{"osd":5,"last update":"Wed Sep 11 08:18:02 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":1.0469999999999999,"15min":0.89100000000000001},"min":{"1min":0.45900000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":2.0430000000000001,"5min":5.4539999999999997,"15min":6.7709999999999999},"last":1.407},{"interface":"front","average":{"1min":0.96499999999999997,"5min":1.1819999999999999,"15min":0.91600000000000004},"min":{"1min":0.39000000000000001,"5min":0.30399999999999999,"15min":0.28399999999999997},"max":{"1min":3.4300000000000002,"5min":12.869,"15min":12.869},"last":0.65600000000000003}]},{"osd":6,"last update":"Wed Sep 11 08:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.876,"15min":0.83299999999999996},"min":{"1min":0.41999999999999998,"5min":0.31,"15min":0.26100000000000001},"max":{"1min":1.24,"5min":4.8799999999999999,"15min":7.2249999999999996},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.73599999999999999,"5min":1.0660000000000001,"15min":0.93600000000000005},"min":{"1min":0.42699999999999999,"5min":0.33200000000000002,"15min":0.32100000000000001},"max":{"1min":3.7410000000000001,"5min":5.4939999999999998,"15min":6.71},"last":0.626}]},{"osd":7,"last update":"Wed Sep 11 08:17:29 2024","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.3440000000000001,"15min":1.3740000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":5.101,"5min":5.101,"15min":5.101},"last":0.61299999999999999},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.419,"15min":1.4970000000000001},"min":{"1min":0.40100000000000002,"5min":0.376,"15min":0.376},"max":{"1min":5.1310000000000002,"5min":5.3710000000000004,"15min":5.3710000000000004},"last":0.57499999999999996}]}]},{"osd":1,"up_from":13,"seq":55834575011,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":945556,"kb_used_data":644044,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":92799596,"statfs":{"total":95995035648,"available":95026786304,"internally_reserved":0,"allocated":659501056,"data_stored":659109178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:17:18 2024","interfaces":[{"interface":"back","average":{"1min":1.843,"5min":1.3120000000000001,"15min":1.0109999999999999},"min":{"1min":0.19900000000000001,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":6.593,"5min":6.952,"15min":32.478000000000002},"last":0.34000000000000002},{"interface":"front","average":{"1min":2.6800000000000002,"5min":1.931,"15min":1.335},"min":{"1min":0.27700000000000002,"5min":0.186,"15min":0.186},"max":{"1min":15.351000000000001,"5min":15.351000000000001,"15min":35.459000000000003},"last":0.223}]},{"osd":2,"last update":"Wed Sep 11 08:17:56 2024","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.1970000000000001,"15min":0.93200000000000005},"min":{"1min":0.221,"5min":0.221,"15min":0.21299999999999999},"max":{"1min":7.798,"5min":7.9699999999999998,"15min":8.3819999999999997},"last":0.26100000000000001},{"interface":"front","average":{"1min":1.109,"5min":1.4119999999999999,"15min":1.091},"min":{"1min":0.248,"5min":0.192,"15min":0.16300000000000001},"max":{"1min":4.3220000000000001,"5min":5.8490000000000002,"15min":10.276},"last":0.21099999999999999}]},{"osd":3,"last update":"Wed Sep 11 08:18:03 2024","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":1.6890000000000001,"15min":1.514},"min":{"1min":0.253,"5min":0.248,"15min":0.246},"max":{"1min":4.4199999999999999,"5min":16.783999999999999,"15min":32.759},"last":0.246},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.4990000000000001,"15min":1.3220000000000001},"min":{"1min":0.26200000000000001,"5min":0.25900000000000001,"15min":0.22600000000000001},"max":{"1min":8.0009999999999994,"5min":14.973000000000001,"15min":32.444000000000003},"last":0.31}]},{"osd":4,"last update":"Wed Sep 11 08:17:30 2024","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.2310000000000001,"15min":0.96699999999999997},"min":{"1min":0.26200000000000001,"5min":0.252,"15min":0.252},"max":{"1min":4.0970000000000004,"5min":5.8970000000000002,"15min":9.0139999999999993},"last":0.29199999999999998},{"interface":"front","average":{"1min":1.405,"5min":1.071,"15min":0.89700000000000002},"min":{"1min":0.29499999999999998,"5min":0.248,"15min":0.248},"max":{"1min":7.9560000000000004,"5min":7.9560000000000004,"15min":8.2360000000000007},"last":0.38300000000000001}]},{"osd":5,"last update":"Wed Sep 11 08:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":1.3440000000000001,"15min":1.044},"min":{"1min":0.32300000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":4.2919999999999998,"5min":5.9299999999999997,"15min":9.0470000000000006},"last":0.77600000000000002},{"interface":"front","average":{"1min":1.135,"5min":1.556,"15min":1.123},"min":{"1min":0.35999999999999999,"5min":0.35199999999999998,"15min":0.28100000000000003},"max":{"1min":5.3780000000000001,"5min":7.0330000000000004,"15min":11.967000000000001},"last":0.83899999999999997}]},{"osd":6,"last update":"Wed Sep 11 08:18:08 2024","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":2.6139999999999999,"15min":1.53},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.29399999999999998},"max":{"1min":4.2249999999999996,"5min":124.31999999999999,"15min":124.31999999999999},"last":0.40799999999999997},{"interface":"front","average":{"1min":1.026,"5min":2.4209999999999998,"15min":1.4299999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":7.9279999999999999,"5min":124.958,"15min":124.958},"last":0.499}]},{"osd":7,"last update":"Wed Sep 11 08:17:30 2024","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.454,"15min":1.274},"min":{"1min":0.34300000000000003,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":4.7649999999999997,"5min":5.9580000000000002,"15min":5.9580000000000002},"last":0.73899999999999999},{"interface":"front","average":{"1min":1.7390000000000001,"5min":1.6359999999999999,"15min":1.532},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":5.2969999999999997,"5min":5.2969999999999997,"15min":5.2969999999999997},"last":2.1890000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738535,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":605628,"kb_used_data":308272,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93139524,"statfs":{"total":95995035648,"available":95374872576,"internally_reserved":0,"allocated":315670528,"data_stored":315250611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,0,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 11 08:17:22 2024","interfaces":[{"interface":"back","average":{"1min":1.49,"5min":1.4770000000000001,"15min":0.94199999999999995},"min":{"1min":0.221,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":10.105,"5min":10.105,"15min":10.105},"last":1.1419999999999999},{"interface":"front","average":{"1min":2.2519999999999998,"5min":2.0569999999999999,"15min":1.167},"min":{"1min":0.16600000000000001,"5min":0.153,"15min":0.153},"max":{"1min":19.852,"5min":19.852,"15min":19.852},"last":0.70999999999999996}]},{"osd":2,"last update":"Wed Sep 11 08:17:47 2024","interfaces":[{"interface":"back","average":{"1min":1.855,"5min":1.6200000000000001,"15min":1.022},"min":{"1min":0.23699999999999999,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":10.025,"5min":10.351000000000001,"15min":10.351000000000001},"last":1.8169999999999999},{"interface":"front","average":{"1min":1.873,"5min":1.6180000000000001,"15min":1.0840000000000001},"min":{"1min":0.29399999999999998,"5min":0.186,"15min":0.186},"max":{"1min":10.074999999999999,"5min":10.074999999999999,"15min":10.074999999999999},"last":2.0739999999999998}]},{"osd":3,"last update":"Wed Sep 11 08:18:08 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":1.796,"15min":1.3340000000000001},"min":{"1min":0.316,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":3.0049999999999999,"5min":35.491,"15min":35.491},"last":0.97399999999999998},{"interface":"front","average":{"1min":0.58599999999999997,"5min":1.7589999999999999,"15min":1.3180000000000001},"min":{"1min":0.27700000000000002,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":2.1440000000000001,"5min":37.563000000000002,"15min":37.563000000000002},"last":1.681}]},{"osd":4,"last update":"Wed Sep 11 08:17:30 2024","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":1.2769999999999999,"15min":0.96299999999999997},"min":{"1min":0.313,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.929,"5min":6.0190000000000001,"15min":9.4030000000000005},"last":0.42099999999999999},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.151,"15min":0.93999999999999995},"min":{"1min":0.40500000000000003,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":5.6399999999999997,"5min":6.6619999999999999,"15min":6.6619999999999999},"last":0.53800000000000003}]},{"osd":5,"last update":"Wed Sep 11 08:17:57 2024","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":1.5229999999999999,"15min":1.081},"min":{"1min":0.435,"5min":0.32700000000000001,"15min":0.255},"max":{"1min":5.6100000000000003,"5min":24.914999999999999,"15min":24.914999999999999},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.78700000000000003,"5min":1.5900000000000001,"15min":1.141},"min":{"1min":0.314,"5min":0.30399999999999999,"15min":0.27100000000000002},"max":{"1min":2.3119999999999998,"5min":25.277000000000001,"15min":25.277000000000001},"last":0.38500000000000001}]},{"osd":6,"last update":"Wed Sep 11 08:18:15 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":1.9690000000000001,"15min":1.3300000000000001},"min":{"1min":0.39000000000000001,"5min":0.29599999999999999,"15min":0.29099999999999998},"max":{"1min":2.1669999999999998,"5min":78.522000000000006,"15min":78.522000000000006},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.84799999999999998,"5min":1.9910000000000001,"15min":1.232},"min":{"1min":0.42399999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":3.0339999999999998,"5min":79.808000000000007,"15min":79.808000000000007},"last":0.73399999999999999}]},{"osd":7,"last update":"Wed Sep 11 08:17:30 2024","interfaces":[{"interface":"back","average":{"1min":1.6439999999999999,"5min":1.5409999999999999,"15min":1.421},"min":{"1min":0.35499999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":7.5350000000000001,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.76600000000000001},{"interface":"front","average":{"1min":1.7230000000000001,"5min":1.4830000000000001,"15min":1.397},"min":{"1min":0.33700000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":7.5110000000000001,"5min":7.5110000000000001,"15min":7.5110000000000001},"last":0.79300000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3425,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327614464,"data_stored":327614464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":734330880,"data_stored":734330880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":240975872,"data_stored":240975872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":633470976,"data_stored":633470976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":220266496,"data_stored":220266496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":706609152,"data_stored":706609152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":223477760,"data_stored":223477760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":680460288,"data_stored":680460288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T08:18:26.013 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd pool set .mgr pgp_num 1 2024-09-11T08:18:26.095 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:26.127 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:26.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:26.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:26.227 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:26 smithi150 ceph-mon[29276]: 5.7 scrub starts 2024-09-11T08:18:26.227 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:26 smithi150 ceph-mon[29276]: 5.7 scrub ok 2024-09-11T08:18:26.227 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:26 smithi150 ceph-mon[29276]: 4.f scrub starts 2024-09-11T08:18:26.227 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:26 smithi150 ceph-mon[29276]: 4.f scrub ok 2024-09-11T08:18:26.227 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:26 smithi150 ceph-mon[29276]: pgmap v491: 148 pgs: 148 active+clean; 1.8 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 148 MiB/s wr, 2.37k op/s 2024-09-11T08:18:26.227 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:26 smithi150 ceph-mon[29276]: 2.1 scrub starts 2024-09-11T08:18:26.227 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:26 smithi150 ceph-mon[29276]: 2.1 scrub ok 2024-09-11T08:18:26.228 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:26 smithi150 ceph-mon[39612]: 5.7 scrub starts 2024-09-11T08:18:26.228 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:26 smithi150 ceph-mon[39612]: 5.7 scrub ok 2024-09-11T08:18:26.228 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:26 smithi150 ceph-mon[39612]: 4.f scrub starts 2024-09-11T08:18:26.228 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:26 smithi150 ceph-mon[39612]: 4.f scrub ok 2024-09-11T08:18:26.228 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:26 smithi150 ceph-mon[39612]: pgmap v491: 148 pgs: 148 active+clean; 1.8 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 148 MiB/s wr, 2.37k op/s 2024-09-11T08:18:26.228 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:26 smithi150 ceph-mon[39612]: 2.1 scrub starts 2024-09-11T08:18:26.228 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:26 smithi150 ceph-mon[39612]: 2.1 scrub ok 2024-09-11T08:18:26.228 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:26.259 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:26.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:26.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:26.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:26 smithi181 ceph-mon[32701]: 5.7 scrub starts 2024-09-11T08:18:26.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:26 smithi181 ceph-mon[32701]: 5.7 scrub ok 2024-09-11T08:18:26.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:26 smithi181 ceph-mon[32701]: 4.f scrub starts 2024-09-11T08:18:26.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:26 smithi181 ceph-mon[32701]: 4.f scrub ok 2024-09-11T08:18:26.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:26 smithi181 ceph-mon[32701]: pgmap v491: 148 pgs: 148 active+clean; 1.8 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 148 MiB/s wr, 2.37k op/s 2024-09-11T08:18:26.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:26 smithi181 ceph-mon[32701]: 2.1 scrub starts 2024-09-11T08:18:26.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:26 smithi181 ceph-mon[32701]: 2.1 scrub ok 2024-09-11T08:18:26.360 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:26.392 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:26.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:26.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:26.495 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:26.526 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:26.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:26.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:26.628 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:26.681 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:18:26.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:26.689 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:26.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:26.767 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 15 16 36114 36098 150.369 128.25 0.00737785 0.00664506 2024-09-11T08:18:26.791 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:26.828 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:26.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:26.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:26.930 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:26.969 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:26.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:26.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:27.070 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:27.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:27.106 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:27.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:27.153 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:27 smithi150 ceph-mon[29276]: 6.7 scrub starts 2024-09-11T08:18:27.154 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:27 smithi150 ceph-mon[29276]: 6.7 scrub ok 2024-09-11T08:18:27.154 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:27 smithi150 ceph-mon[39612]: 6.7 scrub starts 2024-09-11T08:18:27.154 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:27 smithi150 ceph-mon[39612]: 6.7 scrub ok 2024-09-11T08:18:27.206 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:27.238 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:27.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:27.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:27.340 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:27.378 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:27.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:27.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:27.482 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:27.496 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:18:27.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:27.536 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:27.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:27.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:27 smithi181 ceph-mon[32701]: 6.7 scrub starts 2024-09-11T08:18:27.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:27 smithi181 ceph-mon[32701]: 6.7 scrub ok 2024-09-11T08:18:27.637 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:27.675 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:27.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:27.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:27.767 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 16 16 38246 38230 149.297 133.25 0.00411921 0.0066915 2024-09-11T08:18:27.778 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:27.804 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:27.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:27.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:27.906 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:18:27.945 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:27.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:27.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:18:28.047 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:28.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:28.086 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:28.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:28.187 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:28.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:28.221 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:28.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:28.322 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:28.324 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:28 smithi150 ceph-mon[39612]: 3.1c scrub starts 2024-09-11T08:18:28.324 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:28 smithi150 ceph-mon[39612]: 3.1c scrub ok 2024-09-11T08:18:28.324 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:28 smithi150 ceph-mon[39612]: 5.c scrub starts 2024-09-11T08:18:28.324 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:28 smithi150 ceph-mon[39612]: 5.c scrub ok 2024-09-11T08:18:28.324 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:28 smithi150 ceph-mon[39612]: pgmap v492: 148 pgs: 148 active+clean; 1.8 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 148 MiB/s wr, 2.37k op/s 2024-09-11T08:18:28.324 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:28 smithi150 ceph-mon[39612]: 4.5 scrub starts 2024-09-11T08:18:28.324 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:28 smithi150 ceph-mon[39612]: 4.5 scrub ok 2024-09-11T08:18:28.324 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:28 smithi150 ceph-mon[39612]: 4.b scrub starts 2024-09-11T08:18:28.325 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:28 smithi150 ceph-mon[39612]: 4.b scrub ok 2024-09-11T08:18:28.325 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:28 smithi150 ceph-mon[39612]: 5.d scrub starts 2024-09-11T08:18:28.325 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:28 smithi150 ceph-mon[39612]: 5.d scrub ok 2024-09-11T08:18:28.325 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:28 smithi150 ceph-mon[29276]: 3.1c scrub starts 2024-09-11T08:18:28.325 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:28 smithi150 ceph-mon[29276]: 3.1c scrub ok 2024-09-11T08:18:28.325 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:28 smithi150 ceph-mon[29276]: 5.c scrub starts 2024-09-11T08:18:28.325 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:28 smithi150 ceph-mon[29276]: 5.c scrub ok 2024-09-11T08:18:28.325 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:28 smithi150 ceph-mon[29276]: pgmap v492: 148 pgs: 148 active+clean; 1.8 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 148 MiB/s wr, 2.37k op/s 2024-09-11T08:18:28.325 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:28 smithi150 ceph-mon[29276]: 4.5 scrub starts 2024-09-11T08:18:28.325 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:28 smithi150 ceph-mon[29276]: 4.5 scrub ok 2024-09-11T08:18:28.325 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:28 smithi150 ceph-mon[29276]: 4.b scrub starts 2024-09-11T08:18:28.325 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:28 smithi150 ceph-mon[29276]: 4.b scrub ok 2024-09-11T08:18:28.326 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:28 smithi150 ceph-mon[29276]: 5.d scrub starts 2024-09-11T08:18:28.326 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:28 smithi150 ceph-mon[29276]: 5.d scrub ok 2024-09-11T08:18:28.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:28.355 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:28.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:28.456 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:28.462 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:18:28.484 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:28.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:28.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:28.502 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:18:28.502 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:18:28.535 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:18:28.535 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:18:28.564 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:18:28.564 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:18:28.573 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:18:28 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:18:28.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:28 smithi181 ceph-mon[32701]: 3.1c scrub starts 2024-09-11T08:18:28.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:28 smithi181 ceph-mon[32701]: 3.1c scrub ok 2024-09-11T08:18:28.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:28 smithi181 ceph-mon[32701]: 5.c scrub starts 2024-09-11T08:18:28.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:28 smithi181 ceph-mon[32701]: 5.c scrub ok 2024-09-11T08:18:28.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:28 smithi181 ceph-mon[32701]: pgmap v492: 148 pgs: 148 active+clean; 1.8 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 148 MiB/s wr, 2.37k op/s 2024-09-11T08:18:28.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:28 smithi181 ceph-mon[32701]: 4.5 scrub starts 2024-09-11T08:18:28.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:28 smithi181 ceph-mon[32701]: 4.5 scrub ok 2024-09-11T08:18:28.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:28 smithi181 ceph-mon[32701]: 4.b scrub starts 2024-09-11T08:18:28.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:28 smithi181 ceph-mon[32701]: 4.b scrub ok 2024-09-11T08:18:28.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:28 smithi181 ceph-mon[32701]: 5.d scrub starts 2024-09-11T08:18:28.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:28 smithi181 ceph-mon[32701]: 5.d scrub ok 2024-09-11T08:18:28.586 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:28.600 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:18:28.600 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:18:28.601 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:18:28 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:18:28] "GET /metrics HTTP/1.1" 200 224255 "" "Prometheus/2.43.0" 2024-09-11T08:18:28.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:28.627 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:28.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:28.631 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:18:28.632 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:18:28.663 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:18:28.664 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:18:28.696 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:18:28.697 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:18:28.727 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:18:28.728 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:18:28.728 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:28.761 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:28.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:28.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:28.767 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 17 16 40186 40170 147.646 121.25 0.00556893 0.00676875 2024-09-11T08:18:28.863 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:28.898 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:28.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:28.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:29.000 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:29.026 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:29.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:29.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:29.128 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:29.158 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:29.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:29.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:29.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:29 smithi150 ceph-mon[29276]: 3.19 scrub starts 2024-09-11T08:18:29.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:29 smithi150 ceph-mon[29276]: 3.19 scrub ok 2024-09-11T08:18:29.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:29 smithi150 ceph-mon[29276]: 3.3 scrub starts 2024-09-11T08:18:29.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:29 smithi150 ceph-mon[29276]: 3.3 scrub ok 2024-09-11T08:18:29.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:29 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:18:29.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:29 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1278295315' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:18:29.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:29 smithi150 ceph-mon[39612]: 3.19 scrub starts 2024-09-11T08:18:29.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:29 smithi150 ceph-mon[39612]: 3.19 scrub ok 2024-09-11T08:18:29.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:29 smithi150 ceph-mon[39612]: 3.3 scrub starts 2024-09-11T08:18:29.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:29 smithi150 ceph-mon[39612]: 3.3 scrub ok 2024-09-11T08:18:29.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:29 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:18:29.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:29 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1278295315' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:18:29.259 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:29.297 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:29.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:29.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:29.399 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:29.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:29.443 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:29.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:29.544 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:29.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:29 smithi181 ceph-mon[32701]: 3.19 scrub starts 2024-09-11T08:18:29.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:29 smithi181 ceph-mon[32701]: 3.19 scrub ok 2024-09-11T08:18:29.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:29 smithi181 ceph-mon[32701]: 3.3 scrub starts 2024-09-11T08:18:29.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:29 smithi181 ceph-mon[32701]: 3.3 scrub ok 2024-09-11T08:18:29.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:29 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:18:29.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:29 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1278295315' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:18:29.577 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:29.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:29.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:29.679 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:29.721 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:29.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:29.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:29.767 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 18 16 42098 42082 146.082 119.5 0.00804876 0.00684033 2024-09-11T08:18:29.823 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:29.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:29.860 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:29.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:29.961 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:29.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:29.999 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:29.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:30.100 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:30.121 INFO:teuthology.orchestra.run.smithi150.stderr:nodeep-scrub is unset 2024-09-11T08:18:30.130 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:30.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:30.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:30.232 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:30.266 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:30.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:30.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:30.369 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:18:30.404 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:30.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:30.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:18:30.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:30 smithi150 ceph-mon[29276]: 3.12 scrub starts 2024-09-11T08:18:30.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:30 smithi150 ceph-mon[29276]: 3.12 scrub ok 2024-09-11T08:18:30.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:30 smithi150 ceph-mon[29276]: pgmap v493: 148 pgs: 148 active+clean; 2.4 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 941 B/s rd, 148 MiB/s wr, 2.36k op/s 2024-09-11T08:18:30.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:30 smithi150 ceph-mon[29276]: 3.15 scrub starts 2024-09-11T08:18:30.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:30 smithi150 ceph-mon[29276]: 3.15 scrub ok 2024-09-11T08:18:30.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:30 smithi150 ceph-mon[29276]: 3.1a scrub starts 2024-09-11T08:18:30.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:30 smithi150 ceph-mon[29276]: 3.1a scrub ok 2024-09-11T08:18:30.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:30 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2330962155' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:18:30.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:30 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:18:30.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:30 smithi150 ceph-mon[39612]: 3.12 scrub starts 2024-09-11T08:18:30.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:30 smithi150 ceph-mon[39612]: 3.12 scrub ok 2024-09-11T08:18:30.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:30 smithi150 ceph-mon[39612]: pgmap v493: 148 pgs: 148 active+clean; 2.4 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 941 B/s rd, 148 MiB/s wr, 2.36k op/s 2024-09-11T08:18:30.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:30 smithi150 ceph-mon[39612]: 3.15 scrub starts 2024-09-11T08:18:30.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:30 smithi150 ceph-mon[39612]: 3.15 scrub ok 2024-09-11T08:18:30.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:30 smithi150 ceph-mon[39612]: 3.1a scrub starts 2024-09-11T08:18:30.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:30 smithi150 ceph-mon[39612]: 3.1a scrub ok 2024-09-11T08:18:30.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:30 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2330962155' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:18:30.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:30 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:18:30.506 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:30.527 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:30.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:30.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:30.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:30 smithi181 ceph-mon[32701]: 3.12 scrub starts 2024-09-11T08:18:30.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:30 smithi181 ceph-mon[32701]: 3.12 scrub ok 2024-09-11T08:18:30.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:30 smithi181 ceph-mon[32701]: pgmap v493: 148 pgs: 148 active+clean; 2.4 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 941 B/s rd, 148 MiB/s wr, 2.36k op/s 2024-09-11T08:18:30.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:30 smithi181 ceph-mon[32701]: 3.15 scrub starts 2024-09-11T08:18:30.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:30 smithi181 ceph-mon[32701]: 3.15 scrub ok 2024-09-11T08:18:30.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:30 smithi181 ceph-mon[32701]: 3.1a scrub starts 2024-09-11T08:18:30.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:30 smithi181 ceph-mon[32701]: 3.1a scrub ok 2024-09-11T08:18:30.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:30 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2330962155' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:18:30.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:30 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:18:30.630 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:30.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:30.653 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:30.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:30.754 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:30.768 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 19 16 43822 43806 144.065 107.75 0.00516959 0.0069372 2024-09-11T08:18:30.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:30.788 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:30.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:30.889 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:30.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:30.924 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:30.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:31.026 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:18:31.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:31.065 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:31.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:18:31.166 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:31.200 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:31.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:31.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:31.302 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:31.336 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:31.337 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:31 smithi150 ceph-mon[29276]: 3.c scrub starts 2024-09-11T08:18:31.337 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:31 smithi150 ceph-mon[29276]: 3.c scrub ok 2024-09-11T08:18:31.337 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:31 smithi150 ceph-mon[29276]: 6.15 scrub starts 2024-09-11T08:18:31.337 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:31 smithi150 ceph-mon[29276]: 6.15 scrub ok 2024-09-11T08:18:31.337 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:31 smithi150 ceph-mon[29276]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:18:31.337 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:31 smithi150 ceph-mon[29276]: Cluster is now healthy 2024-09-11T08:18:31.337 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:31 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:18:31.337 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:31 smithi150 ceph-mon[29276]: osdmap e253: 8 total, 8 up, 8 in 2024-09-11T08:18:31.337 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:31 smithi150 ceph-mon[29276]: 6.1f scrub starts 2024-09-11T08:18:31.337 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:31 smithi150 ceph-mon[29276]: 6.1f scrub ok 2024-09-11T08:18:31.338 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:31 smithi150 ceph-mon[39612]: 3.c scrub starts 2024-09-11T08:18:31.338 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:31 smithi150 ceph-mon[39612]: 3.c scrub ok 2024-09-11T08:18:31.338 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:31 smithi150 ceph-mon[39612]: 6.15 scrub starts 2024-09-11T08:18:31.338 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:31 smithi150 ceph-mon[39612]: 6.15 scrub ok 2024-09-11T08:18:31.338 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:31 smithi150 ceph-mon[39612]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:18:31.338 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:31 smithi150 ceph-mon[39612]: Cluster is now healthy 2024-09-11T08:18:31.338 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:31 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:18:31.338 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:31 smithi150 ceph-mon[39612]: osdmap e253: 8 total, 8 up, 8 in 2024-09-11T08:18:31.338 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:31 smithi150 ceph-mon[39612]: 6.1f scrub starts 2024-09-11T08:18:31.339 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:31 smithi150 ceph-mon[39612]: 6.1f scrub ok 2024-09-11T08:18:31.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:31.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:31.440 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:31.468 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:31.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:31.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:31.570 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:31.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:31 smithi181 ceph-mon[32701]: 3.c scrub starts 2024-09-11T08:18:31.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:31 smithi181 ceph-mon[32701]: 3.c scrub ok 2024-09-11T08:18:31.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:31 smithi181 ceph-mon[32701]: 6.15 scrub starts 2024-09-11T08:18:31.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:31 smithi181 ceph-mon[32701]: 6.15 scrub ok 2024-09-11T08:18:31.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:31 smithi181 ceph-mon[32701]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:18:31.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:31 smithi181 ceph-mon[32701]: Cluster is now healthy 2024-09-11T08:18:31.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:31 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:18:31.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:31 smithi181 ceph-mon[32701]: osdmap e253: 8 total, 8 up, 8 in 2024-09-11T08:18:31.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:31 smithi181 ceph-mon[32701]: 6.1f scrub starts 2024-09-11T08:18:31.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:31 smithi181 ceph-mon[32701]: 6.1f scrub ok 2024-09-11T08:18:31.607 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:31.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:31.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:31.709 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:31.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:31.744 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:31.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:31.768 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:2024-09-11T08:18:31.768345+0000 min lat: 0.00117502 max lat: 0.179893 avg lat: 0.00698951 2024-09-11T08:18:31.768 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-11T08:18:31.768 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 20 16 45780 45764 142.98 122.375 0.00673565 0.00698951 2024-09-11T08:18:31.845 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:31.882 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:31.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:31.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:31.984 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:32.018 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:32.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:32.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:32.046 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T08:18:32.120 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:32.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:32.158 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:32.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:32.259 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:32.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:32.300 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:32.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:32.401 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:32.435 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:32 smithi150 ceph-mon[39612]: 5.f scrub starts 2024-09-11T08:18:32.435 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:32 smithi150 ceph-mon[39612]: 5.f scrub ok 2024-09-11T08:18:32.435 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:32 smithi150 ceph-mon[39612]: 5.10 scrub starts 2024-09-11T08:18:32.435 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:32 smithi150 ceph-mon[39612]: 5.10 scrub ok 2024-09-11T08:18:32.435 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:32 smithi150 ceph-mon[39612]: pgmap v495: 148 pgs: 148 active+clean; 2.4 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 123 MiB/s wr, 1.97k op/s 2024-09-11T08:18:32.436 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:32 smithi150 ceph-mon[39612]: 4.d scrub starts 2024-09-11T08:18:32.436 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:32 smithi150 ceph-mon[39612]: 4.d scrub ok 2024-09-11T08:18:32.436 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:32 smithi150 ceph-mon[39612]: 5.b scrub starts 2024-09-11T08:18:32.436 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:32 smithi150 ceph-mon[39612]: 5.b scrub ok 2024-09-11T08:18:32.436 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:32 smithi150 ceph-mon[29276]: 5.f scrub starts 2024-09-11T08:18:32.436 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:32 smithi150 ceph-mon[29276]: 5.f scrub ok 2024-09-11T08:18:32.436 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:32 smithi150 ceph-mon[29276]: 5.10 scrub starts 2024-09-11T08:18:32.436 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:32 smithi150 ceph-mon[29276]: 5.10 scrub ok 2024-09-11T08:18:32.436 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:32 smithi150 ceph-mon[29276]: pgmap v495: 148 pgs: 148 active+clean; 2.4 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 123 MiB/s wr, 1.97k op/s 2024-09-11T08:18:32.436 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:32 smithi150 ceph-mon[29276]: 4.d scrub starts 2024-09-11T08:18:32.436 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:32 smithi150 ceph-mon[29276]: 4.d scrub ok 2024-09-11T08:18:32.437 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:32 smithi150 ceph-mon[29276]: 5.b scrub starts 2024-09-11T08:18:32.437 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:32 smithi150 ceph-mon[29276]: 5.b scrub ok 2024-09-11T08:18:32.442 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:32.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:32.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:32.544 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:32.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:32 smithi181 ceph-mon[32701]: 5.f scrub starts 2024-09-11T08:18:32.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:32 smithi181 ceph-mon[32701]: 5.f scrub ok 2024-09-11T08:18:32.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:32 smithi181 ceph-mon[32701]: 5.10 scrub starts 2024-09-11T08:18:32.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:32 smithi181 ceph-mon[32701]: 5.10 scrub ok 2024-09-11T08:18:32.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:32 smithi181 ceph-mon[32701]: pgmap v495: 148 pgs: 148 active+clean; 2.4 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 123 MiB/s wr, 1.97k op/s 2024-09-11T08:18:32.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:32 smithi181 ceph-mon[32701]: 4.d scrub starts 2024-09-11T08:18:32.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:32 smithi181 ceph-mon[32701]: 4.d scrub ok 2024-09-11T08:18:32.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:32 smithi181 ceph-mon[32701]: 5.b scrub starts 2024-09-11T08:18:32.575 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:32 smithi181 ceph-mon[32701]: 5.b scrub ok 2024-09-11T08:18:32.596 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:32.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:32.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:32.699 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:32.732 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:32.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:32.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:32.768 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 21 16 47604 47588 141.6 114 0.00814377 0.00705725 2024-09-11T08:18:32.833 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:32.866 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:32.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:32.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:32.968 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:33.000 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:33.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:33.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:33.102 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:18:33.141 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:33.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:33.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:18:33.243 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:33.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:33.278 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:33.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:33.379 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:33.411 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:33.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:33.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:33.487 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:33 smithi150 ceph-mon[29276]: 3.10 scrub starts 2024-09-11T08:18:33.487 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:33 smithi150 ceph-mon[29276]: 3.10 scrub ok 2024-09-11T08:18:33.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:33 smithi150 ceph-mon[39612]: 3.10 scrub starts 2024-09-11T08:18:33.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:33 smithi150 ceph-mon[39612]: 3.10 scrub ok 2024-09-11T08:18:33.513 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:33.548 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:33.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:33.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:33.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:33 smithi181 ceph-mon[32701]: 3.10 scrub starts 2024-09-11T08:18:33.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:33 smithi181 ceph-mon[32701]: 3.10 scrub ok 2024-09-11T08:18:33.651 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:18:33.692 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:33.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:33.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:18:33.768 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 22 16 49494 49478 140.532 118.125 0.00638858 0.00711126 2024-09-11T08:18:33.795 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:33.825 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:33.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:33.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:33.928 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:33.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:33.966 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:33.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:34.022 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:18:34.068 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:34.105 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:34.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:34.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:34.207 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:34.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:34 smithi150 ceph-mon[29276]: 6.9 deep-scrub starts 2024-09-11T08:18:34.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:34 smithi150 ceph-mon[29276]: 6.9 deep-scrub ok 2024-09-11T08:18:34.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:34 smithi150 ceph-mon[29276]: 5.16 scrub starts 2024-09-11T08:18:34.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:34 smithi150 ceph-mon[29276]: 5.16 scrub ok 2024-09-11T08:18:34.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:34 smithi150 ceph-mon[29276]: pgmap v496: 148 pgs: 148 active+clean; 2.6 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 122 MiB/s wr, 1.95k op/s 2024-09-11T08:18:34.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:34 smithi150 ceph-mon[39612]: 6.9 deep-scrub starts 2024-09-11T08:18:34.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:34 smithi150 ceph-mon[39612]: 6.9 deep-scrub ok 2024-09-11T08:18:34.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:34 smithi150 ceph-mon[39612]: 5.16 scrub starts 2024-09-11T08:18:34.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:34 smithi150 ceph-mon[39612]: 5.16 scrub ok 2024-09-11T08:18:34.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:34 smithi150 ceph-mon[39612]: pgmap v496: 148 pgs: 148 active+clean; 2.6 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 122 MiB/s wr, 1.95k op/s 2024-09-11T08:18:34.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:34.245 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:34.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:34.346 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:18:34.381 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:34.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:34.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:18:34.483 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:34.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:34.515 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:34.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:34.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:34 smithi181 ceph-mon[32701]: 6.9 deep-scrub starts 2024-09-11T08:18:34.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:34 smithi181 ceph-mon[32701]: 6.9 deep-scrub ok 2024-09-11T08:18:34.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:34 smithi181 ceph-mon[32701]: 5.16 scrub starts 2024-09-11T08:18:34.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:34 smithi181 ceph-mon[32701]: 5.16 scrub ok 2024-09-11T08:18:34.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:34 smithi181 ceph-mon[32701]: pgmap v496: 148 pgs: 148 active+clean; 2.6 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 122 MiB/s wr, 1.95k op/s 2024-09-11T08:18:34.616 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:34.654 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:34.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:34.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:34.756 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:34.768 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 23 16 51230 51214 139.139 108.5 0.00487244 0.00718129 2024-09-11T08:18:34.783 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:34.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:34.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:34.885 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:34.918 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:34.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:34.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:35.020 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:18:35.060 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:35.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:35.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:18:35.162 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:35.191 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:35.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:35.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:35.293 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:35.318 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:35 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:18:35.318 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:35 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:18:35.318 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:35 smithi150 ceph-mon[29276]: osdmap e254: 8 total, 8 up, 8 in 2024-09-11T08:18:35.318 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:35 smithi150 ceph-mon[29276]: 5.1f deep-scrub starts 2024-09-11T08:18:35.319 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:35 smithi150 ceph-mon[29276]: 5.1f deep-scrub ok 2024-09-11T08:18:35.319 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:35 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:18:35.319 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:35 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:18:35.319 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:35 smithi150 ceph-mon[39612]: osdmap e254: 8 total, 8 up, 8 in 2024-09-11T08:18:35.319 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:35 smithi150 ceph-mon[39612]: 5.1f deep-scrub starts 2024-09-11T08:18:35.319 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:35 smithi150 ceph-mon[39612]: 5.1f deep-scrub ok 2024-09-11T08:18:35.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:35.328 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:35.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:35.400 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-09-11T08:18:35.400 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:18:35.429 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:18:35.465 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:35.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:35.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:18:35.567 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:35.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:18:35 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:18:35] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:18:35.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:35 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:18:35.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:35 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:18:35.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:35 smithi181 ceph-mon[32701]: osdmap e254: 8 total, 8 up, 8 in 2024-09-11T08:18:35.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:35 smithi181 ceph-mon[32701]: 5.1f deep-scrub starts 2024-09-11T08:18:35.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:35 smithi181 ceph-mon[32701]: 5.1f deep-scrub ok 2024-09-11T08:18:35.601 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:35.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:35.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:35.703 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:35.739 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:35.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:35.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:35.768 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 24 16 52878 52862 137.633 103 0.00908684 0.007261 2024-09-11T08:18:35.841 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:18:35.870 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:35.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:35.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:18:35.972 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:35.990 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T08:18:36.007 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:36.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:36.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:36.109 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:18:36.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:36.154 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:36.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:18:36.166 INFO:teuthology.orchestra.run.smithi150.stderr:set pool 1 pgp_num to 1 2024-09-11T08:18:36.255 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:36.289 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:36.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:36.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:36.391 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:36.423 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:36.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:36.425 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:36.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:36 smithi150 ceph-mon[29276]: 4.3 scrub starts 2024-09-11T08:18:36.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:36 smithi150 ceph-mon[29276]: 4.3 scrub ok 2024-09-11T08:18:36.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:36 smithi150 ceph-mon[29276]: pgmap v498: 148 pgs: 148 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 151 MiB/s wr, 2.42k op/s 2024-09-11T08:18:36.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:36 smithi150 ceph-mon[29276]: 4.9 scrub starts 2024-09-11T08:18:36.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:36 smithi150 ceph-mon[29276]: 4.9 scrub ok 2024-09-11T08:18:36.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:36 smithi150 ceph-mon[29276]: 5.6 scrub starts 2024-09-11T08:18:36.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:36 smithi150 ceph-mon[29276]: 5.6 scrub ok 2024-09-11T08:18:36.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:36 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1546214162' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-09-11T08:18:36.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:36 smithi150 ceph-mon[29276]: 5.3 scrub starts 2024-09-11T08:18:36.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:36 smithi150 ceph-mon[29276]: 5.3 scrub ok 2024-09-11T08:18:36.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:36 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:18:36.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:36 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:18:36.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:36 smithi150 ceph-mon[39612]: 4.3 scrub starts 2024-09-11T08:18:36.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:36 smithi150 ceph-mon[39612]: 4.3 scrub ok 2024-09-11T08:18:36.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:36 smithi150 ceph-mon[39612]: pgmap v498: 148 pgs: 148 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 151 MiB/s wr, 2.42k op/s 2024-09-11T08:18:36.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:36 smithi150 ceph-mon[39612]: 4.9 scrub starts 2024-09-11T08:18:36.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:36 smithi150 ceph-mon[39612]: 4.9 scrub ok 2024-09-11T08:18:36.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:36 smithi150 ceph-mon[39612]: 5.6 scrub starts 2024-09-11T08:18:36.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:36 smithi150 ceph-mon[39612]: 5.6 scrub ok 2024-09-11T08:18:36.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:36 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1546214162' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-09-11T08:18:36.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:36 smithi150 ceph-mon[39612]: 5.3 scrub starts 2024-09-11T08:18:36.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:36 smithi150 ceph-mon[39612]: 5.3 scrub ok 2024-09-11T08:18:36.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:36 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:18:36.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:36 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:18:36.525 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:36.565 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:36.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:36.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:36.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:36 smithi181 ceph-mon[32701]: 4.3 scrub starts 2024-09-11T08:18:36.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:36 smithi181 ceph-mon[32701]: 4.3 scrub ok 2024-09-11T08:18:36.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:36 smithi181 ceph-mon[32701]: pgmap v498: 148 pgs: 148 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 151 MiB/s wr, 2.42k op/s 2024-09-11T08:18:36.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:36 smithi181 ceph-mon[32701]: 4.9 scrub starts 2024-09-11T08:18:36.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:36 smithi181 ceph-mon[32701]: 4.9 scrub ok 2024-09-11T08:18:36.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:36 smithi181 ceph-mon[32701]: 5.6 scrub starts 2024-09-11T08:18:36.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:36 smithi181 ceph-mon[32701]: 5.6 scrub ok 2024-09-11T08:18:36.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:36 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1546214162' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-09-11T08:18:36.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:36 smithi181 ceph-mon[32701]: 5.3 scrub starts 2024-09-11T08:18:36.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:36 smithi181 ceph-mon[32701]: 5.3 scrub ok 2024-09-11T08:18:36.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:36 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:18:36.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:36 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:18:36.666 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:18:36.678 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-11T08:18:36.714 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:36.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:36.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:18:36.769 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 25 16 54492 54476 136.157 100.875 0.00948598 0.00733808 2024-09-11T08:18:36.817 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:36.849 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:36.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:36.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:36.951 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:36.984 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:36.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:36.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:37.085 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:37.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:37.125 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:37.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:37.226 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:37.264 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:37.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:37.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:37.366 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:37.405 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:37.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:37.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:37.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:37 smithi150 ceph-mon[29276]: 2.2 scrub starts 2024-09-11T08:18:37.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:37 smithi150 ceph-mon[29276]: 2.2 scrub ok 2024-09-11T08:18:37.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:37 smithi150 ceph-mon[29276]: 6.1d scrub starts 2024-09-11T08:18:37.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:37 smithi150 ceph-mon[29276]: 6.1d scrub ok 2024-09-11T08:18:37.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:37 smithi150 ceph-mon[29276]: 3.e scrub starts 2024-09-11T08:18:37.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:37 smithi150 ceph-mon[29276]: 3.e scrub ok 2024-09-11T08:18:37.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:37 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1546214162' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-09-11T08:18:37.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:37 smithi150 ceph-mon[29276]: osdmap e255: 8 total, 8 up, 8 in 2024-09-11T08:18:37.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:37 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:18:37.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:37 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:18:37.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:37 smithi150 ceph-mon[39612]: 2.2 scrub starts 2024-09-11T08:18:37.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:37 smithi150 ceph-mon[39612]: 2.2 scrub ok 2024-09-11T08:18:37.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:37 smithi150 ceph-mon[39612]: 6.1d scrub starts 2024-09-11T08:18:37.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:37 smithi150 ceph-mon[39612]: 6.1d scrub ok 2024-09-11T08:18:37.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:37 smithi150 ceph-mon[39612]: 3.e scrub starts 2024-09-11T08:18:37.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:37 smithi150 ceph-mon[39612]: 3.e scrub ok 2024-09-11T08:18:37.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:37 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1546214162' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-09-11T08:18:37.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:37 smithi150 ceph-mon[39612]: osdmap e255: 8 total, 8 up, 8 in 2024-09-11T08:18:37.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:37 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:18:37.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:37 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:18:37.507 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:37.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:37.542 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:37.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:37.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:37 smithi181 ceph-mon[32701]: 2.2 scrub starts 2024-09-11T08:18:37.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:37 smithi181 ceph-mon[32701]: 2.2 scrub ok 2024-09-11T08:18:37.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:37 smithi181 ceph-mon[32701]: 6.1d scrub starts 2024-09-11T08:18:37.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:37 smithi181 ceph-mon[32701]: 6.1d scrub ok 2024-09-11T08:18:37.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:37 smithi181 ceph-mon[32701]: 3.e scrub starts 2024-09-11T08:18:37.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:37 smithi181 ceph-mon[32701]: 3.e scrub ok 2024-09-11T08:18:37.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:37 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1546214162' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-09-11T08:18:37.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:37 smithi181 ceph-mon[32701]: osdmap e255: 8 total, 8 up, 8 in 2024-09-11T08:18:37.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:37 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:18:37.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:37 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:18:37.643 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:18:37.684 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:37.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:37.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:18:37.769 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 26 16 56152 56136 134.911 103.75 0.00895559 0.00740693 2024-09-11T08:18:37.786 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:37.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:37.821 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:37.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:37.922 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:37.961 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:37.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:37.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:38.063 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:38.090 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:18:38.099 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:18:38.116 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:38.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:38.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:38.218 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:38.252 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:38.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:38.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:38.354 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:18:38.392 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:38.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:38.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:18:38.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:38 smithi150 ceph-mon[29276]: pgmap v500: 148 pgs: 148 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 751 B/s rd, 86 MiB/s wr, 1.38k op/s 2024-09-11T08:18:38.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:38 smithi150 ceph-mon[29276]: 5.18 deep-scrub starts 2024-09-11T08:18:38.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:38 smithi150 ceph-mon[29276]: 5.18 deep-scrub ok 2024-09-11T08:18:38.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:38 smithi150 ceph-mon[29276]: 6.1c scrub starts 2024-09-11T08:18:38.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:38 smithi150 ceph-mon[29276]: 6.1c scrub ok 2024-09-11T08:18:38.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:38 smithi150 ceph-mon[39612]: pgmap v500: 148 pgs: 148 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 751 B/s rd, 86 MiB/s wr, 1.38k op/s 2024-09-11T08:18:38.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:38 smithi150 ceph-mon[39612]: 5.18 deep-scrub starts 2024-09-11T08:18:38.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:38 smithi150 ceph-mon[39612]: 5.18 deep-scrub ok 2024-09-11T08:18:38.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:38 smithi150 ceph-mon[39612]: 6.1c scrub starts 2024-09-11T08:18:38.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:38 smithi150 ceph-mon[39612]: 6.1c scrub ok 2024-09-11T08:18:38.495 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:38.529 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:38.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:38.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:38.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:38 smithi181 ceph-mon[32701]: pgmap v500: 148 pgs: 148 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 751 B/s rd, 86 MiB/s wr, 1.38k op/s 2024-09-11T08:18:38.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:38 smithi181 ceph-mon[32701]: 5.18 deep-scrub starts 2024-09-11T08:18:38.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:38 smithi181 ceph-mon[32701]: 5.18 deep-scrub ok 2024-09-11T08:18:38.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:38 smithi181 ceph-mon[32701]: 6.1c scrub starts 2024-09-11T08:18:38.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:38 smithi181 ceph-mon[32701]: 6.1c scrub ok 2024-09-11T08:18:38.574 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:18:38 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:18:38.632 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:38.671 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:38.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:38.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:38.769 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 27 15 57820 57805 133.777 104.312 0.00643798 0.00747034 2024-09-11T08:18:38.773 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:38.805 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:38.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:38.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:38.828 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:18:38 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:18:38] "GET /metrics HTTP/1.1" 200 224341 "" "Prometheus/2.43.0" 2024-09-11T08:18:38.907 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:38.939 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:38.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:38.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:39.041 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:39.076 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:39.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:39.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:39.178 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:39.217 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:39.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:39.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:39.319 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:39.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:39 smithi150 ceph-mon[29276]: 5.1c scrub starts 2024-09-11T08:18:39.343 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:39 smithi150 ceph-mon[29276]: 5.1c scrub ok 2024-09-11T08:18:39.343 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:39 smithi150 ceph-mon[29276]: 5.2 scrub starts 2024-09-11T08:18:39.343 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:39 smithi150 ceph-mon[29276]: 5.2 scrub ok 2024-09-11T08:18:39.343 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:39 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:18:39.343 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:39 smithi150 ceph-mon[39612]: 5.1c scrub starts 2024-09-11T08:18:39.343 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:39 smithi150 ceph-mon[39612]: 5.1c scrub ok 2024-09-11T08:18:39.343 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:39 smithi150 ceph-mon[39612]: 5.2 scrub starts 2024-09-11T08:18:39.344 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:39 smithi150 ceph-mon[39612]: 5.2 scrub ok 2024-09-11T08:18:39.344 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:39 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:18:39.347 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:39.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:39.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:39.449 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:39.475 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:18:39.475 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:18:39.483 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:39.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:39.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:39.504 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:18:39.504 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:18:39.535 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:18:39.536 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:18:39.568 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:18:39.568 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:18:39.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:39 smithi181 ceph-mon[32701]: 5.1c scrub starts 2024-09-11T08:18:39.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:39 smithi181 ceph-mon[32701]: 5.1c scrub ok 2024-09-11T08:18:39.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:39 smithi181 ceph-mon[32701]: 5.2 scrub starts 2024-09-11T08:18:39.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:39 smithi181 ceph-mon[32701]: 5.2 scrub ok 2024-09-11T08:18:39.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:39 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:18:39.586 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:39.600 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:18:39.601 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:18:39.625 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:39.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:39.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:39.630 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:18:39.630 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:18:39.660 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:18:39.660 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:18:39.691 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:18:39.692 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:18:39.727 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:39.759 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:39.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:39.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:39.769 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 28 16 59338 59322 132.385 94.8125 0.00979616 0.00754898 2024-09-11T08:18:39.861 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:39.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:39.883 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:39.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:39.984 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:18:40.019 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:40.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:40.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:18:40.121 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:40.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:40.159 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:40.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:40.260 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:40.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:40.296 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:40.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:40.397 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:18:40.433 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:40.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:40.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:18:40.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:40 smithi150 ceph-mon[29276]: pgmap v501: 148 pgs: 148 active+clean; 3.5 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 139 MiB/s wr, 2.23k op/s 2024-09-11T08:18:40.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:40 smithi150 ceph-mon[29276]: 3.f scrub starts 2024-09-11T08:18:40.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:40 smithi150 ceph-mon[29276]: 3.f scrub ok 2024-09-11T08:18:40.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:40 smithi150 ceph-mon[29276]: 5.1e deep-scrub starts 2024-09-11T08:18:40.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:40 smithi150 ceph-mon[29276]: 5.1e deep-scrub ok 2024-09-11T08:18:40.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:40 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1744621142' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:18:40.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:40 smithi150 ceph-mon[39612]: pgmap v501: 148 pgs: 148 active+clean; 3.5 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 139 MiB/s wr, 2.23k op/s 2024-09-11T08:18:40.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:40 smithi150 ceph-mon[39612]: 3.f scrub starts 2024-09-11T08:18:40.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:40 smithi150 ceph-mon[39612]: 3.f scrub ok 2024-09-11T08:18:40.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:40 smithi150 ceph-mon[39612]: 5.1e deep-scrub starts 2024-09-11T08:18:40.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:40 smithi150 ceph-mon[39612]: 5.1e deep-scrub ok 2024-09-11T08:18:40.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:40 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1744621142' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:18:40.535 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:18:40.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:40 smithi181 ceph-mon[32701]: pgmap v501: 148 pgs: 148 active+clean; 3.5 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 139 MiB/s wr, 2.23k op/s 2024-09-11T08:18:40.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:40 smithi181 ceph-mon[32701]: 3.f scrub starts 2024-09-11T08:18:40.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:40 smithi181 ceph-mon[32701]: 3.f scrub ok 2024-09-11T08:18:40.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:40 smithi181 ceph-mon[32701]: 5.1e deep-scrub starts 2024-09-11T08:18:40.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:40 smithi181 ceph-mon[32701]: 5.1e deep-scrub ok 2024-09-11T08:18:40.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:40 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1744621142' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:18:40.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:40.654 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:40.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:18:40.755 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:18:40.770 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 29 16 60860 60844 131.098 95.125 0.0123676 0.0076229 2024-09-11T08:18:40.799 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:40.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:40.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:18:40.901 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:40.936 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:40.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:40.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:41.038 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:41.071 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:41.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:41.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:41.172 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:41.205 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:41.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:41.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:41.307 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:18:41.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:41.345 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:41.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:18:41.446 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:18:41.484 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:41.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:41.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:18:41.586 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:41.626 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:41.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:41.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:41.728 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:41.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:41 smithi150 ceph-mon[29276]: 6.a scrub starts 2024-09-11T08:18:41.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:41 smithi150 ceph-mon[29276]: 6.a scrub ok 2024-09-11T08:18:41.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:41 smithi150 ceph-mon[29276]: 5.5 scrub starts 2024-09-11T08:18:41.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:41 smithi150 ceph-mon[29276]: 5.5 scrub ok 2024-09-11T08:18:41.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:41 smithi150 ceph-mon[29276]: 3.0 scrub starts 2024-09-11T08:18:41.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:41 smithi150 ceph-mon[29276]: 3.0 scrub ok 2024-09-11T08:18:41.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:41 smithi150 ceph-mon[29276]: pgmap v502: 148 pgs: 148 active+clean; 3.5 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 117 MiB/s wr, 1.86k op/s 2024-09-11T08:18:41.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:41 smithi150 ceph-mon[39612]: 6.a scrub starts 2024-09-11T08:18:41.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:41 smithi150 ceph-mon[39612]: 6.a scrub ok 2024-09-11T08:18:41.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:41 smithi150 ceph-mon[39612]: 5.5 scrub starts 2024-09-11T08:18:41.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:41 smithi150 ceph-mon[39612]: 5.5 scrub ok 2024-09-11T08:18:41.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:41 smithi150 ceph-mon[39612]: 3.0 scrub starts 2024-09-11T08:18:41.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:41 smithi150 ceph-mon[39612]: 3.0 scrub ok 2024-09-11T08:18:41.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:41 smithi150 ceph-mon[39612]: pgmap v502: 148 pgs: 148 active+clean; 3.5 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 117 MiB/s wr, 1.86k op/s 2024-09-11T08:18:41.761 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:41.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:41.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:41.770 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 30 16 62474 62458 130.09 100.875 0.00953646 0.00768163 2024-09-11T08:18:41.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:41 smithi181 ceph-mon[32701]: 6.a scrub starts 2024-09-11T08:18:41.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:41 smithi181 ceph-mon[32701]: 6.a scrub ok 2024-09-11T08:18:41.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:41 smithi181 ceph-mon[32701]: 5.5 scrub starts 2024-09-11T08:18:41.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:41 smithi181 ceph-mon[32701]: 5.5 scrub ok 2024-09-11T08:18:41.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:41 smithi181 ceph-mon[32701]: 3.0 scrub starts 2024-09-11T08:18:41.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:41 smithi181 ceph-mon[32701]: 3.0 scrub ok 2024-09-11T08:18:41.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:41 smithi181 ceph-mon[32701]: pgmap v502: 148 pgs: 148 active+clean; 3.5 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 117 MiB/s wr, 1.86k op/s 2024-09-11T08:18:41.863 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:41.899 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:41.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:41.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:42.001 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:42.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:42.046 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:42.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:42.147 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:42.189 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:42.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:42.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:42.291 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:42.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:42.326 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:42.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:42.428 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:18:42.465 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:42.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:42.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:18:42.567 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:42.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:42.609 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:42.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:42.710 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:18:42.751 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:42.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:42.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:18:42.770 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 31 16 64014 63998 128.998 96.25 0.0076229 0.00774732 2024-09-11T08:18:42.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:42 smithi181 ceph-mon[32701]: 4.e scrub starts 2024-09-11T08:18:42.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:42 smithi181 ceph-mon[32701]: 4.e scrub ok 2024-09-11T08:18:42.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:42 smithi181 ceph-mon[32701]: 1.0 scrub starts 2024-09-11T08:18:42.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:42 smithi181 ceph-mon[32701]: 1.0 scrub ok 2024-09-11T08:18:42.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:42 smithi181 ceph-mon[32701]: 5.12 scrub starts 2024-09-11T08:18:42.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:42 smithi181 ceph-mon[32701]: 5.12 scrub ok 2024-09-11T08:18:42.853 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:42.889 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:42.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:42.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:42.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:42 smithi150 ceph-mon[29276]: 4.e scrub starts 2024-09-11T08:18:42.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:42 smithi150 ceph-mon[29276]: 4.e scrub ok 2024-09-11T08:18:42.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:42 smithi150 ceph-mon[29276]: 1.0 scrub starts 2024-09-11T08:18:42.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:42 smithi150 ceph-mon[29276]: 1.0 scrub ok 2024-09-11T08:18:42.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:42 smithi150 ceph-mon[29276]: 5.12 scrub starts 2024-09-11T08:18:42.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:42 smithi150 ceph-mon[29276]: 5.12 scrub ok 2024-09-11T08:18:42.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:42 smithi150 ceph-mon[39612]: 4.e scrub starts 2024-09-11T08:18:42.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:42 smithi150 ceph-mon[39612]: 4.e scrub ok 2024-09-11T08:18:42.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:42 smithi150 ceph-mon[39612]: 1.0 scrub starts 2024-09-11T08:18:42.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:42 smithi150 ceph-mon[39612]: 1.0 scrub ok 2024-09-11T08:18:42.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:42 smithi150 ceph-mon[39612]: 5.12 scrub starts 2024-09-11T08:18:42.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:42 smithi150 ceph-mon[39612]: 5.12 scrub ok 2024-09-11T08:18:42.991 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:43.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:43.029 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:43.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:43.130 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:43.165 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T08:18:43.175 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:43.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:43.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:43.279 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:43.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:43.336 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:43.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:43.398 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-11T08:18:43.398 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [], 2024-09-11T08:18:43.398 INFO:teuthology.orchestra.run.smithi150.stdout: "complaint_time": 30, 2024-09-11T08:18:43.398 INFO:teuthology.orchestra.run.smithi150.stdout: "num_blocked_ops": 0 2024-09-11T08:18:43.398 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-11T08:18:43.438 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:43.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:43.468 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:43.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:43.569 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:18:43.582 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-11T08:18:43.582 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T08:18:43.582 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-09-11T08:18:43.582 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-11T08:18:43.582 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-11T08:18:43.583 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-09-11T08:18:43.605 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:43.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:43.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:18:43.707 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:43.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:43 smithi150 ceph-mon[29276]: pgmap v503: 148 pgs: 148 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.1 KiB/s rd, 78 MiB/s wr, 1.24k op/s 2024-09-11T08:18:43.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:43 smithi150 ceph-mon[39612]: pgmap v503: 148 pgs: 148 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.1 KiB/s rd, 78 MiB/s wr, 1.24k op/s 2024-09-11T08:18:43.749 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:43.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:43.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:43.770 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 32 16 65467 65451 127.805 90.8125 0.00834251 0.00781999 2024-09-11T08:18:43.851 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:43.892 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:43.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:43.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:43.994 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:44.023 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:18:44.029 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:44.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:44.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:44.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:43 smithi181 ceph-mon[32701]: pgmap v503: 148 pgs: 148 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.1 KiB/s rd, 78 MiB/s wr, 1.24k op/s 2024-09-11T08:18:44.126 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-11T08:18:44.130 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:44.160 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:44.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:44.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:44.262 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:44.291 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:44.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:44.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:44.393 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:44.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:44.459 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:44.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:44.560 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:18:44.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:44.609 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:44.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:18:44.692 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:18:44.710 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:44.727 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:18:44.731 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:44.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:44.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:44.770 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 33 16 66560 66544 126.002 68.3125 0.00371271 0.00790112 2024-09-11T08:18:44.833 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:44.865 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:44.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:44.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:44.960 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:44 smithi150 ceph-mon[29276]: 6.f scrub starts 2024-09-11T08:18:44.960 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:44 smithi150 ceph-mon[29276]: 6.f scrub ok 2024-09-11T08:18:44.960 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:44 smithi150 ceph-mon[39612]: 6.f scrub starts 2024-09-11T08:18:44.960 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:44 smithi150 ceph-mon[39612]: 6.f scrub ok 2024-09-11T08:18:44.968 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:44.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:44.990 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:44.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:45.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:44 smithi181 ceph-mon[32701]: 6.f scrub starts 2024-09-11T08:18:45.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:44 smithi181 ceph-mon[32701]: 6.f scrub ok 2024-09-11T08:18:45.092 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:45.133 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:45.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:45.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:45.235 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:18:45.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:45.257 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:45.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:18:45.358 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:45.396 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:45.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:45.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:45.498 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:45.519 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:45.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:45.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:45.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:18:45 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:18:45] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:18:45.622 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:45.655 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:45.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:45.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:45.758 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:18:45.770 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 34 16 67766 67750 124.513 75.375 0.025538 0.00802631 2024-09-11T08:18:45.772 INFO:teuthology.orchestra.run.smithi150.stderr:noscrub is set 2024-09-11T08:18:45.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:45.792 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:45.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:18:45.893 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:45.914 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:45.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:45.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:46.016 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:46.052 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:46.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:46.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:46.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:45 smithi181 ceph-mon[32701]: 4.a scrub starts 2024-09-11T08:18:46.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:45 smithi181 ceph-mon[32701]: 4.1e scrub starts 2024-09-11T08:18:46.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:45 smithi181 ceph-mon[32701]: 4.15 scrub starts 2024-09-11T08:18:46.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:45 smithi181 ceph-mon[32701]: 4.a scrub ok 2024-09-11T08:18:46.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:45 smithi181 ceph-mon[32701]: 4.15 scrub ok 2024-09-11T08:18:46.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:45 smithi181 ceph-mon[32701]: 4.1e scrub ok 2024-09-11T08:18:46.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:45 smithi181 ceph-mon[32701]: 5.e scrub starts 2024-09-11T08:18:46.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:45 smithi181 ceph-mon[32701]: 5.e scrub ok 2024-09-11T08:18:46.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:45 smithi181 ceph-mon[32701]: pgmap v504: 148 pgs: 148 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 102 MiB/s wr, 1.63k op/s 2024-09-11T08:18:46.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:45 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3152113368' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:18:46.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:45 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:18:46.155 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:46.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:46.179 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:46.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:46.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:45 smithi150 ceph-mon[39612]: 4.a scrub starts 2024-09-11T08:18:46.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:45 smithi150 ceph-mon[39612]: 4.1e scrub starts 2024-09-11T08:18:46.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:45 smithi150 ceph-mon[39612]: 4.15 scrub starts 2024-09-11T08:18:46.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:45 smithi150 ceph-mon[39612]: 4.a scrub ok 2024-09-11T08:18:46.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:45 smithi150 ceph-mon[39612]: 4.15 scrub ok 2024-09-11T08:18:46.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:45 smithi150 ceph-mon[39612]: 4.1e scrub ok 2024-09-11T08:18:46.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:45 smithi150 ceph-mon[39612]: 5.e scrub starts 2024-09-11T08:18:46.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:45 smithi150 ceph-mon[39612]: 5.e scrub ok 2024-09-11T08:18:46.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:45 smithi150 ceph-mon[39612]: pgmap v504: 148 pgs: 148 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 102 MiB/s wr, 1.63k op/s 2024-09-11T08:18:46.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:45 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3152113368' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:18:46.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:45 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:18:46.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:45 smithi150 ceph-mon[29276]: 4.a scrub starts 2024-09-11T08:18:46.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:45 smithi150 ceph-mon[29276]: 4.1e scrub starts 2024-09-11T08:18:46.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:45 smithi150 ceph-mon[29276]: 4.15 scrub starts 2024-09-11T08:18:46.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:45 smithi150 ceph-mon[29276]: 4.a scrub ok 2024-09-11T08:18:46.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:45 smithi150 ceph-mon[29276]: 4.15 scrub ok 2024-09-11T08:18:46.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:45 smithi150 ceph-mon[29276]: 4.1e scrub ok 2024-09-11T08:18:46.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:45 smithi150 ceph-mon[29276]: 5.e scrub starts 2024-09-11T08:18:46.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:45 smithi150 ceph-mon[29276]: 5.e scrub ok 2024-09-11T08:18:46.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:45 smithi150 ceph-mon[29276]: pgmap v504: 148 pgs: 148 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 102 MiB/s wr, 1.63k op/s 2024-09-11T08:18:46.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:45 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3152113368' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:18:46.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:45 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:18:46.280 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:46.313 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:46.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:46.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:46.415 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:46.450 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:46.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:46.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:46.552 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:46.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:46.574 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:46.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:46.675 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:46.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:46.708 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:46.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:46.771 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 35 16 68867 68851 122.922 68.8125 0.00426826 0.0081303 2024-09-11T08:18:46.808 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:46.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:46.837 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:46.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:46.938 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:46.968 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:46.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:46.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:47.070 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:18:47.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:46 smithi181 ceph-mon[32701]: 4.1f scrub starts 2024-09-11T08:18:47.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:46 smithi181 ceph-mon[32701]: 4.1f scrub ok 2024-09-11T08:18:47.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:46 smithi181 ceph-mon[32701]: 3.2 scrub starts 2024-09-11T08:18:47.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:46 smithi181 ceph-mon[32701]: 3.2 scrub ok 2024-09-11T08:18:47.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:46 smithi181 ceph-mon[32701]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:18:47.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:46 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:18:47.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:46 smithi181 ceph-mon[32701]: osdmap e256: 8 total, 8 up, 8 in 2024-09-11T08:18:47.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:46 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:18:47.092 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:47.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:47.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:18:47.194 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:47.217 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:47.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:47.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:47.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:46 smithi150 ceph-mon[39612]: 4.1f scrub starts 2024-09-11T08:18:47.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:46 smithi150 ceph-mon[39612]: 4.1f scrub ok 2024-09-11T08:18:47.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:46 smithi150 ceph-mon[39612]: 3.2 scrub starts 2024-09-11T08:18:47.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:46 smithi150 ceph-mon[39612]: 3.2 scrub ok 2024-09-11T08:18:47.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:46 smithi150 ceph-mon[39612]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:18:47.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:46 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:18:47.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:46 smithi150 ceph-mon[39612]: osdmap e256: 8 total, 8 up, 8 in 2024-09-11T08:18:47.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:46 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:18:47.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:46 smithi150 ceph-mon[29276]: 4.1f scrub starts 2024-09-11T08:18:47.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:46 smithi150 ceph-mon[29276]: 4.1f scrub ok 2024-09-11T08:18:47.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:46 smithi150 ceph-mon[29276]: 3.2 scrub starts 2024-09-11T08:18:47.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:46 smithi150 ceph-mon[29276]: 3.2 scrub ok 2024-09-11T08:18:47.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:46 smithi150 ceph-mon[29276]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:18:47.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:46 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:18:47.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:46 smithi150 ceph-mon[29276]: osdmap e256: 8 total, 8 up, 8 in 2024-09-11T08:18:47.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:46 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:18:47.318 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:47.350 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:47.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:47.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:47.452 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:47.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:47.485 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:47.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:47.587 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:47.623 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:47.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:47.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:47.725 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:47.745 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:47.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:47.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:47.771 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 36 16 69920 69904 121.335 65.8125 0.00398637 0.00822692 2024-09-11T08:18:47.847 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:47.869 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:47.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:47.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:47.971 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:48.007 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:48.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:48.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:48.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:47 smithi181 ceph-mon[32701]: pgmap v506: 148 pgs: 148 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 102 MiB/s wr, 1.63k op/s 2024-09-11T08:18:48.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:47 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:18:48.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:47 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:18:48.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:47 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:18:48.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:47 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:18:48.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:47 smithi181 ceph-mon[32701]: Adjusting osd_memory_target on smithi181 to 3247M 2024-09-11T08:18:48.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:47 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:18:48.109 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:48.130 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:48.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:48.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:48.232 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:48.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:47 smithi150 ceph-mon[29276]: pgmap v506: 148 pgs: 148 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 102 MiB/s wr, 1.63k op/s 2024-09-11T08:18:48.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:47 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:18:48.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:47 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:18:48.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:47 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:18:48.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:47 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:18:48.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:47 smithi150 ceph-mon[29276]: Adjusting osd_memory_target on smithi181 to 3247M 2024-09-11T08:18:48.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:47 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:18:48.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:47 smithi150 ceph-mon[39612]: pgmap v506: 148 pgs: 148 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 102 MiB/s wr, 1.63k op/s 2024-09-11T08:18:48.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:47 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:18:48.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:47 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:18:48.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:47 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:18:48.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:47 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:18:48.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:47 smithi150 ceph-mon[39612]: Adjusting osd_memory_target on smithi181 to 3247M 2024-09-11T08:18:48.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:47 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:18:48.266 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:48.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:48.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:48.368 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:48.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:48.391 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:48.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:48.493 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:48.528 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:48.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:48.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:48.630 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:48.657 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:48.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:48.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:48.758 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:18:48.762 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:18:48 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:18:48.771 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 37 16 70977 70961 119.839 66.0625 0.00956934 0.0083393 2024-09-11T08:18:48.794 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:48.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:48.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:18:48.874 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:18:48 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:18:48] "GET /metrics HTTP/1.1" 200 224341 "" "Prometheus/2.43.0" 2024-09-11T08:18:48.874 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:48 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:18:48.874 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:48 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:18:48.874 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:48 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:18:48.874 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:48 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:18:48.874 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:48 smithi150 ceph-mon[39612]: Adjusting osd_memory_target on smithi150 to 3248M 2024-09-11T08:18:48.874 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:48 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:18:48.874 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:48 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:18:48.875 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:48 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:18:48.875 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:48 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:18:48.875 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:48 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:18:48.875 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:48 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:18:48.875 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:48 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:18:48.875 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:48 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:18:48.875 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:48 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:18:48.875 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:48 smithi150 ceph-mon[29276]: Adjusting osd_memory_target on smithi150 to 3248M 2024-09-11T08:18:48.875 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:48 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:18:48.875 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:48 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:18:48.875 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:48 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:18:48.876 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:48 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:18:48.876 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:48 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:18:48.896 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:48.931 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:48.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:48.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:49.033 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:49.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:48 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:18:49.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:48 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:18:49.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:48 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:18:49.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:48 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:18:49.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:48 smithi181 ceph-mon[32701]: Adjusting osd_memory_target on smithi150 to 3248M 2024-09-11T08:18:49.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:48 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:18:49.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:48 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:18:49.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:48 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:18:49.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:48 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:18:49.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:48 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:18:49.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:49.079 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:49.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:49.180 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:49.213 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:49.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:49.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:49.315 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:49.356 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:49.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:49.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:49.458 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:49.496 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:49.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:49.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:49.601 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:49.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:49.643 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:49.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:49.744 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:49.771 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 38 16 72332 72316 118.914 84.6875 0.014012 0.00840404 2024-09-11T08:18:49.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:49.807 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:49.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:49.908 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:18:49.947 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:49.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:49.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:18:50.049 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:50.071 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:49 smithi150 ceph-mon[29276]: pgmap v507: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.35k op/s 2024-09-11T08:18:50.071 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:49 smithi150 ceph-mon[39612]: pgmap v507: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.35k op/s 2024-09-11T08:18:50.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:49 smithi181 ceph-mon[32701]: pgmap v507: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.35k op/s 2024-09-11T08:18:50.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:50.084 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:50.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:50.185 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:50.226 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:50.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:50.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:50.328 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:50.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:50.366 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:50.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:50.467 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:18:50.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:50.505 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:50.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:18:50.606 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:18:50.645 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:50.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:50.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:18:50.711 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-11T08:18:50.711 INFO:teuthology.orchestra.run.smithi181.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-11T08:18:50.711 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-11T08:18:50.746 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:50.771 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 39 16 73620 73604 117.929 80.5 0.0160045 0.0084746 2024-09-11T08:18:50.786 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:50.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:50.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:50.887 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:50.922 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:50.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:50.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:51.024 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:18:51.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:51.059 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:51.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:18:51.160 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:51.195 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:51.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:51.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:51.297 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:51.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:51.341 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:51.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:51.393 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:18:51.393 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:18:51.430 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:18:51.431 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:18:51.442 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:18:51.466 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:18:51.466 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:18:51.482 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:51.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:51.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:18:51.498 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:18:51.498 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:18:51.527 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:18:51.527 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:18:51.558 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:18:51.558 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:18:51.582 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:51 smithi150 ceph-mon[29276]: pgmap v508: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.35k op/s 2024-09-11T08:18:51.584 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:51.591 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:18:51.592 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:18:51.622 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:51.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:51.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:51.624 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:18:51.624 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:18:51.724 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:51.759 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:51.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:51.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:51.772 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:2024-09-11T08:18:51.772492+0000 min lat: 0.00117502 max lat: 0.179893 avg lat: 0.00853925 2024-09-11T08:18:51.772 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-11T08:18:51.772 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 40 16 74942 74926 117.046 82.625 0.00800614 0.00853925 2024-09-11T08:18:51.861 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:51.895 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:51.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:51.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:51.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:51 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2162024794' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:18:51.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:51 smithi150 ceph-mon[39612]: pgmap v508: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.35k op/s 2024-09-11T08:18:51.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:51 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2162024794' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:18:51.997 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:18:52.034 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:52.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:52.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:18:52.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:51 smithi181 ceph-mon[32701]: pgmap v508: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.35k op/s 2024-09-11T08:18:52.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:51 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2162024794' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:18:52.136 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:52.168 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:52.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:52.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:52.270 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:52.319 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:52.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:52.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:52.420 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:18:52.461 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:52.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:52.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:18:52.563 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:18:52.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:52.600 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:52.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:18:52.701 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:52.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:52.736 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:52.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:52.772 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 41 15 76373 76358 116.374 89.5 0.0186244 0.0085889 2024-09-11T08:18:52.797 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T08:18:52.838 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:52.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:52.886 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:52.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:52.987 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:18:53.021 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:52 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:18:53.022 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:52 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:18:53.022 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:52 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:18:53.022 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:52 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:18:53.038 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:53.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:53.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:18:53.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:52 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:18:53.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:52 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:18:53.141 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:18:53.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:53.184 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:53.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:18:53.285 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:18:53.324 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:53.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:53.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:18:53.426 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:53.457 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:53.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:53.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:53.558 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:18:53.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:53.598 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:53.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:18:53.699 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:53.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:53.733 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:53.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:53.772 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 42 16 77545 77529 115.346 73.1875 0.00847193 0.00866369 2024-09-11T08:18:53.790 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-09-11T08:18:53.790 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:18:53.835 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:53.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:53.876 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:53.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:53.977 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:18:53.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:53 smithi150 ceph-mon[29276]: pgmap v509: 148 pgs: 148 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-09-11T08:18:53.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:53 smithi150 ceph-mon[39612]: pgmap v509: 148 pgs: 148 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-09-11T08:18:54.015 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:54.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:54.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:18:54.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:53 smithi181 ceph-mon[32701]: pgmap v509: 148 pgs: 148 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-09-11T08:18:54.117 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:18:54.152 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:54.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:54.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:18:54.254 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:18:54.287 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:54.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:54.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:18:54.389 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:18:54.425 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:54.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:54.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:18:54.527 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:54.559 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:54.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:54.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:54.625 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-11T08:18:54.651 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:18:54.662 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:54.690 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:54.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:54.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:54.772 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 43 16 78688 78672 114.325 71.4375 0.00805993 0.00874103 2024-09-11T08:18:54.793 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:54.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:54.847 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:54.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:54.947 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:54 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:18:54.947 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:54 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:18:54.947 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:54 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:18:54.947 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:54 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:18:54.948 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:18:54.999 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:55.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:55.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:18:55.068 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T08:18:55.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:54 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:18:55.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:54 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:18:55.101 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:55.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:55.177 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:55.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:55.278 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:55.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:55.319 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:55.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:55.420 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:55.453 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:55.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:55.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:55.555 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:18:55.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:18:55 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:18:55] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:18:55.598 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:55.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:55.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:18:55.700 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:55.741 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:55.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:55.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:55.772 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 44 16 79908 79892 113.459 76.25 0.0178706 0.00880923 2024-09-11T08:18:55.843 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:18:55.878 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:55.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:55.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:18:55.980 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:18:56.012 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:56.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:56.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:18:56.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:55 smithi181 ceph-mon[32701]: osdmap e257: 8 total, 8 up, 8 in 2024-09-11T08:18:56.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:55 smithi181 ceph-mon[32701]: pgmap v511: 148 pgs: 148 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 83 MiB/s wr, 1.34k op/s 2024-09-11T08:18:56.090 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:55 smithi150 ceph-mon[39612]: osdmap e257: 8 total, 8 up, 8 in 2024-09-11T08:18:56.091 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:55 smithi150 ceph-mon[39612]: pgmap v511: 148 pgs: 148 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 83 MiB/s wr, 1.34k op/s 2024-09-11T08:18:56.096 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:55 smithi150 ceph-mon[29276]: osdmap e257: 8 total, 8 up, 8 in 2024-09-11T08:18:56.096 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:55 smithi150 ceph-mon[29276]: pgmap v511: 148 pgs: 148 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 83 MiB/s wr, 1.34k op/s 2024-09-11T08:18:56.114 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:18:56.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:56.161 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:56.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:18:56.180 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:18:56.224 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:18:56.262 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:56.294 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:56.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:56.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:56.396 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:18:56.443 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:56.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:56.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:18:56.545 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:56.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:56.574 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:56.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:56.675 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:56.715 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-11T08:18:56.715 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T08:18:56.715 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-11T08:18:56.715 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd out 5 2024-09-11T08:18:56.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:56.717 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:56.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:56.773 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 45 16 81184 81168 112.709 79.75 0.00802556 0.00886781 2024-09-11T08:18:56.819 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:56.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:56.851 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:56.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:56.952 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:18:56.995 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:56.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:56.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:18:57.097 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:57.150 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:57.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:57.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:57.252 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:57.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:57.288 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:57.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:57.389 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:57.423 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:57.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:57.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:57.525 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:18:57.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:57.565 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:57.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:18:57.666 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:18:57.702 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:57.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:57.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:18:57.773 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 46 16 82383 82367 111.888 74.9375 0.0068461 0.00893323 2024-09-11T08:18:57.803 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:57.827 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:57.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:57.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:57.929 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:57.965 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:57.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:57.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:57.993 INFO:teuthology.orchestra.run.smithi150.stderr:nodeep-scrub is set 2024-09-11T08:18:58.068 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:58.109 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:58.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:58.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:58.211 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:18:58.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:58.249 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:58.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:18:58.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:57 smithi181 ceph-mon[32701]: pgmap v512: 148 pgs: 148 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-09-11T08:18:58.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:57 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1104972522' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:18:58.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:57 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:18:58.349 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:18:58.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:58.384 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:58.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:18:58.485 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:18:58.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:57 smithi150 ceph-mon[29276]: pgmap v512: 148 pgs: 148 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-09-11T08:18:58.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:57 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1104972522' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:18:58.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:57 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:18:58.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:57 smithi150 ceph-mon[39612]: pgmap v512: 148 pgs: 148 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-09-11T08:18:58.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:57 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1104972522' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:18:58.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:57 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:18:58.524 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:58.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:58.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:18:58.626 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:18:58.662 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:58.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:58.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:18:58.764 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:58.774 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 47 16 83698 83682 111.253 82.1875 0.00638684 0.00898395 2024-09-11T08:18:58.810 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:58.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:58.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:58.823 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:18:58 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:18:58.912 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:58.940 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:58.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:58.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:58.988 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:18:58 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:18:58] "GET /metrics HTTP/1.1" 200 224374 "" "Prometheus/2.43.0" 2024-09-11T08:18:59.042 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:59.079 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:59.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:59.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:59.181 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:59.215 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:59.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:59.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:59.317 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:18:59.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:58 smithi181 ceph-mon[32701]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:18:59.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:58 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:18:59.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:58 smithi181 ceph-mon[32701]: osdmap e258: 8 total, 8 up, 8 in 2024-09-11T08:18:59.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:18:58 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:18:59.353 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:59.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:59.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:18:59.455 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:18:59.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:58 smithi150 ceph-mon[29276]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:18:59.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:58 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:18:59.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:58 smithi150 ceph-mon[29276]: osdmap e258: 8 total, 8 up, 8 in 2024-09-11T08:18:59.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:18:58 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:18:59.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:58 smithi150 ceph-mon[39612]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:18:59.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:58 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:18:59.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:58 smithi150 ceph-mon[39612]: osdmap e258: 8 total, 8 up, 8 in 2024-09-11T08:18:59.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:18:58 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:18:59.494 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:59.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:59.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:18:59.596 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:18:59.623 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:18:59.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:59.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:18:59.725 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:18:59.756 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:59.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:59.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:18:59.774 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 48 16 85025 85009 110.663 82.9375 0.0160946 0.00903174 2024-09-11T08:18:59.858 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:18:59.892 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:18:59.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:18:59.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:18:59.994 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:00.033 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:00.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:00.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:00.135 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:00.166 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:00.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:00.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:00.268 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:19:00.301 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:00.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:00.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:19:00.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:00 smithi181 ceph-mon[32701]: pgmap v514: 148 pgs: 148 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s 2024-09-11T08:19:00.402 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:00.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:00.436 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:00.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:00.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:00 smithi150 ceph-mon[29276]: pgmap v514: 148 pgs: 148 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s 2024-09-11T08:19:00.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:00 smithi150 ceph-mon[39612]: pgmap v514: 148 pgs: 148 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s 2024-09-11T08:19:00.536 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:00.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:00.594 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:00.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:00.695 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:00.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:00.731 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:00.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:00.774 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 49 16 86352 86336 110.097 82.9375 0.00904844 0.00907875 2024-09-11T08:19:00.832 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:19:00.870 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:00.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:00.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:19:00.972 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:01.000 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:01.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:01.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:01.101 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:01.140 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:01.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:01.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:01.242 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:01.277 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:01.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:01.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:01.379 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:19:01.414 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:01.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:01.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:19:01.516 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:19:01.549 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:19:01.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:01.551 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:01.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:19:01.651 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:01.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:01.691 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:01.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:01.776 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 50 16 87677 87661 109.551 82.8125 0.00833826 0.0091237 2024-09-11T08:19:01.793 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:19:01.801 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:01 smithi150 ceph-mon[29276]: pgmap v515: 148 pgs: 148 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-09-11T08:19:01.801 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:01 smithi150 ceph-mon[39612]: pgmap v515: 148 pgs: 148 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-09-11T08:19:01.825 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:01.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:01.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:19:01.927 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:01.958 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:01.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:01.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:02.060 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:02.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:01 smithi181 ceph-mon[32701]: pgmap v515: 148 pgs: 148 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-09-11T08:19:02.093 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-09-11T08:19:02.094 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:19:02.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:02.100 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:02.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:02.201 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:02.239 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:02.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:02.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:02.342 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:02.373 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:02.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:02.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:02.475 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:02.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:02.513 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:02.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:02.614 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:19:02.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:02.651 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:02.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:19:02.752 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:19:02.775 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 51 16 88947 88931 108.96 79.375 0.0103346 0.00917335 2024-09-11T08:19:02.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:02.790 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:02.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:19:02.891 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:02.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:02.926 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:02.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:03.027 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:03.036 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T08:19:03.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:03.062 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:03.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:03.163 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:19:03.193 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:03.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:03.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:19:03.295 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:19:03.335 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:03.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:03.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:19:03.437 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:19:03.467 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:03.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:03.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:19:03.540 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-11T08:19:03.568 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:19:03.601 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:03.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:03.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:19:03.703 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:03.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:03.768 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:03.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:03.774 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 52 16 90173 90157 108.338 76.625 0.0291671 0.00922553 2024-09-11T08:19:03.869 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:03.890 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:19:03.890 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:19:03.915 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:03.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:03.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:03.925 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:19:03.926 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:19:03.957 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:19:03.957 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:19:03.995 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:19:03.995 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:19:04.017 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:19:04.025 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:19:04.026 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:19:04.051 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:04.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:04.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:19:04.056 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:19:04.056 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:19:04.058 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:03 smithi150 ceph-mon[29276]: pgmap v516: 148 pgs: 148 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 62 MiB/s wr, 985 op/s 2024-09-11T08:19:04.058 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:03 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1362910888' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:19:04.058 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:03 smithi150 ceph-mon[39612]: pgmap v516: 148 pgs: 148 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 62 MiB/s wr, 985 op/s 2024-09-11T08:19:04.058 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:03 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1362910888' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:19:04.090 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:19:04.090 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:19:04.125 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:19:04.125 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:19:04.153 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:04.195 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:04.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:04.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:04.297 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:04.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:03 smithi181 ceph-mon[32701]: pgmap v516: 148 pgs: 148 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 62 MiB/s wr, 985 op/s 2024-09-11T08:19:04.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:03 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1362910888' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:19:04.335 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:04.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:04.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:04.440 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:04.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:04.474 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:04.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:04.575 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:04.611 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:04.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:04.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:04.714 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:04.754 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:04.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:04.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:04.774 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 53 16 91431 91415 107.777 78.625 0.0145004 0.00927397 2024-09-11T08:19:04.823 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:19:04 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=cleanup t=2024-09-11T08:19:04.496979097Z level=info msg="Completed cleanup jobs" duration=12.747523ms 2024-09-11T08:19:04.856 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:19:04.887 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:04.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:04.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:19:04.991 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:05.031 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:05.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:05.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:05.133 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:05.170 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:05.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:05.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:05.271 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:05.321 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:05.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:05.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:05.424 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:19:05.461 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:05.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:05.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:19:05.562 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:19:05.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:19:05 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:19:05] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:19:05.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:05.607 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:05.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:19:05.709 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:19:05.738 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:05.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:05.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:19:05.770 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:19:05.775 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 54 16 92709 92693 107.26 79.875 0.00617852 0.00931915 2024-09-11T08:19:05.840 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:19:05.878 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:05.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:05.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:19:05.980 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:06.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:06.022 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:06.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:06.123 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:19:06.158 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:06.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:06.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:19:06.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:05 smithi150 ceph-mon[29276]: pgmap v517: 148 pgs: 148 active+clean; 5.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-09-11T08:19:06.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:06 smithi150 ceph-mon[39612]: pgmap v517: 148 pgs: 148 active+clean; 5.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-09-11T08:19:06.260 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:19:06.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:06.291 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:06.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:19:06.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:05 smithi181 ceph-mon[32701]: pgmap v517: 148 pgs: 148 active+clean; 5.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-09-11T08:19:06.393 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:06.434 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:06.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:06.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:06.535 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:19:06.576 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:06.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:06.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:19:06.678 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:19:06.714 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:06.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:06.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:19:06.775 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 55 16 93983 93967 106.758 79.625 0.00859107 0.00936268 2024-09-11T08:19:06.817 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:06.855 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:06.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:06.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:06.957 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:19:06.991 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:06.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:06.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:19:07.093 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:07.126 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:07.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:07.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:07.228 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:19:07.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:07 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:19:07.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:07 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:19:07.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:07 smithi150 ceph-mon[29276]: osdmap e259: 8 total, 8 up, 8 in 2024-09-11T08:19:07.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:07 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:19:07.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:07 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:19:07.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:07 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:19:07.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:07 smithi150 ceph-mon[39612]: osdmap e259: 8 total, 8 up, 8 in 2024-09-11T08:19:07.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:07 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:19:07.264 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:07.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:07.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:19:07.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:07 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:19:07.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:07 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:19:07.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:07 smithi181 ceph-mon[32701]: osdmap e259: 8 total, 8 up, 8 in 2024-09-11T08:19:07.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:07 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:19:07.366 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:07.398 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:07.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:07.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:07.499 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:19:07.532 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:07.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:07.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:19:07.634 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:07.671 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:07.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:07.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:07.756 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T08:19:07.773 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:19:07.776 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 56 16 95227 95211 106.238 77.75 0.0131119 0.00940799 2024-09-11T08:19:07.802 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:07.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:07.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:19:07.904 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:07.931 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:07.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:07.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:08.032 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:19:08.091 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:08.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:08.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:19:08.175 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:19:08.193 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:19:08.222 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:08.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:08.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:19:08.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:08 smithi181 ceph-mon[32701]: pgmap v519: 148 pgs: 148 active+clean; 5.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 572 B/s rd, 75 MiB/s wr, 1.19k op/s 2024-09-11T08:19:08.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:08 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2473883246' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-09-11T08:19:08.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:08 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-09-11T08:19:08.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:08 smithi181 ceph-mon[32701]: Client client.admin marked osd.5 out, while it was still marked up 2024-09-11T08:19:08.324 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:19:08.344 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:07 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mon-a[29272]: 2024-09-11T08:19:07.986+0000 7f1b17b61640 -1 mon.a@0(leader).osd e259 definitely_dead 0 2024-09-11T08:19:08.344 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:08 smithi150 ceph-mon[29276]: pgmap v519: 148 pgs: 148 active+clean; 5.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 572 B/s rd, 75 MiB/s wr, 1.19k op/s 2024-09-11T08:19:08.345 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:08 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2473883246' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-09-11T08:19:08.345 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:08 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-09-11T08:19:08.345 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:08 smithi150 ceph-mon[29276]: Client client.admin marked osd.5 out, while it was still marked up 2024-09-11T08:19:08.345 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:08 smithi150 ceph-mon[39612]: pgmap v519: 148 pgs: 148 active+clean; 5.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 572 B/s rd, 75 MiB/s wr, 1.19k op/s 2024-09-11T08:19:08.345 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:08 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2473883246' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-09-11T08:19:08.345 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:08 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-09-11T08:19:08.345 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:08 smithi150 ceph-mon[39612]: Client client.admin marked osd.5 out, while it was still marked up 2024-09-11T08:19:08.363 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:08.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:08.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:19:08.465 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:19:08.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:08.502 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:08.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:19:08.594 INFO:teuthology.orchestra.run.smithi150.stderr:marked out osd.5. 2024-09-11T08:19:08.594 INFO:teuthology.orchestra.run.smithi150.stderr:noscrub is unset 2024-09-11T08:19:08.603 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:19:08.618 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:19:08 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:19:08] "GET /metrics HTTP/1.1" 200 224382 "" "Prometheus/2.43.0" 2024-09-11T08:19:08.630 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:08.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:08.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:19:08.732 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:08.756 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:08.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:08.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:08.776 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 57 16 96118 96102 105.351 55.6875 0.00410687 0.00945952 2024-09-11T08:19:08.823 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:19:08 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:19:08.858 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:19:08.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:08.881 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:08.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:19:08.982 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:09.004 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:09.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:09.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:09.106 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:19:09.127 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:09.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:09.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:19:09.229 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:19:09.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:09 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:19:09.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:09 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2479792188' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:19:09.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:09 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:19:09.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:09 smithi150 ceph-mon[29276]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:19:09.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:09 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:19:09.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:09 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:19:09.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:09 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-09-11T08:19:09.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:09 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:19:09.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:09 smithi150 ceph-mon[29276]: osdmap e260: 8 total, 8 up, 7 in 2024-09-11T08:19:09.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:09 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:19:09.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:09 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2479792188' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:19:09.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:09 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:19:09.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:09 smithi150 ceph-mon[39612]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:19:09.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:09 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:19:09.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:09 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:19:09.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:09 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-09-11T08:19:09.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:09 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:19:09.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:09 smithi150 ceph-mon[39612]: osdmap e260: 8 total, 8 up, 7 in 2024-09-11T08:19:09.250 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:09.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:09.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:19:09.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:09 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:19:09.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:09 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2479792188' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:19:09.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:09 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:19:09.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:09 smithi181 ceph-mon[32701]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:19:09.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:09 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:19:09.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:09 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:19:09.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:09 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-09-11T08:19:09.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:09 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:19:09.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:09 smithi181 ceph-mon[32701]: osdmap e260: 8 total, 8 up, 7 in 2024-09-11T08:19:09.353 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:09.375 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:09.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:09.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:09.476 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:19:09.498 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:09.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:09.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:19:09.600 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:19:09.631 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:09.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:09.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:19:09.733 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:09.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:09.754 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:09.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:09.776 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 58 16 96118 96102 103.535 0 - 0.00945952 2024-09-11T08:19:09.855 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:09.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:09.878 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:09.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:09.979 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:10.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:10.003 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:10.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:10.104 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:10.130 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:10.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:10.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:10.232 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:19:10.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:10 smithi150 ceph-mon[29276]: 4.17 scrub starts 2024-09-11T08:19:10.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:10 smithi150 ceph-mon[29276]: 4.17 scrub ok 2024-09-11T08:19:10.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:10 smithi150 ceph-mon[29276]: pgmap v521: 148 pgs: 148 active+clean; 5.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-09-11T08:19:10.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:10 smithi150 ceph-mon[29276]: osdmap e261: 8 total, 8 up, 7 in 2024-09-11T08:19:10.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:10 smithi150 ceph-mon[39612]: 4.17 scrub starts 2024-09-11T08:19:10.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:10 smithi150 ceph-mon[39612]: 4.17 scrub ok 2024-09-11T08:19:10.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:10 smithi150 ceph-mon[39612]: pgmap v521: 148 pgs: 148 active+clean; 5.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-09-11T08:19:10.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:10 smithi150 ceph-mon[39612]: osdmap e261: 8 total, 8 up, 7 in 2024-09-11T08:19:10.258 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:10.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:10.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:19:10.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:10 smithi181 ceph-mon[32701]: 4.17 scrub starts 2024-09-11T08:19:10.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:10 smithi181 ceph-mon[32701]: 4.17 scrub ok 2024-09-11T08:19:10.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:10 smithi181 ceph-mon[32701]: pgmap v521: 148 pgs: 148 active+clean; 5.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-09-11T08:19:10.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:10 smithi181 ceph-mon[32701]: osdmap e261: 8 total, 8 up, 7 in 2024-09-11T08:19:10.360 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:19:10.386 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:10.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:10.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:19:10.488 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:19:10.514 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:10.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:10.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:19:10.615 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:19:10.640 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:10.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:10.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:19:10.741 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:10.776 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 59 16 96121 96105 101.783 0.09375 0.0205175 0.00950314 2024-09-11T08:19:10.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:10.782 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:10.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:10.883 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:19:10.934 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:10.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:10.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:19:11.036 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:11.063 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:11.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:11.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:11.165 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:19:11.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:11.207 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:11.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:19:11.308 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:11.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:11 smithi181 ceph-mon[32701]: 6.18 scrub starts 2024-09-11T08:19:11.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:11 smithi181 ceph-mon[32701]: 6.18 scrub ok 2024-09-11T08:19:11.326 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:11 smithi181 ceph-mon[32701]: 5.a scrub starts 2024-09-11T08:19:11.326 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:11 smithi181 ceph-mon[32701]: 5.a scrub ok 2024-09-11T08:19:11.326 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:11 smithi181 ceph-mon[32701]: 3.1e scrub starts 2024-09-11T08:19:11.326 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:11 smithi181 ceph-mon[32701]: 3.1e scrub ok 2024-09-11T08:19:11.326 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:11 smithi181 ceph-mon[32701]: 9.d starting backfill to osd.4 from (0'0,0'0] MAX to 259'5962 2024-09-11T08:19:11.326 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:11 smithi181 ceph-mon[32701]: 9.7 starting backfill to osd.2 from (0'0,0'0] MAX to 259'6030 2024-09-11T08:19:11.327 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:11 smithi181 ceph-mon[32701]: osdmap e262: 8 total, 8 up, 7 in 2024-09-11T08:19:11.327 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:11 smithi181 ceph-mon[32701]: 9.6 starting backfill to osd.6 from (0'0,0'0] MAX to 259'6056 2024-09-11T08:19:11.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:11.364 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:11.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:11.465 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:19:11.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:11 smithi150 ceph-mon[29276]: 6.18 scrub starts 2024-09-11T08:19:11.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:11 smithi150 ceph-mon[29276]: 6.18 scrub ok 2024-09-11T08:19:11.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:11 smithi150 ceph-mon[29276]: 5.a scrub starts 2024-09-11T08:19:11.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:11 smithi150 ceph-mon[29276]: 5.a scrub ok 2024-09-11T08:19:11.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:11 smithi150 ceph-mon[29276]: 3.1e scrub starts 2024-09-11T08:19:11.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:11 smithi150 ceph-mon[29276]: 3.1e scrub ok 2024-09-11T08:19:11.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:11 smithi150 ceph-mon[29276]: 9.d starting backfill to osd.4 from (0'0,0'0] MAX to 259'5962 2024-09-11T08:19:11.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:11 smithi150 ceph-mon[29276]: 9.7 starting backfill to osd.2 from (0'0,0'0] MAX to 259'6030 2024-09-11T08:19:11.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:11 smithi150 ceph-mon[29276]: osdmap e262: 8 total, 8 up, 7 in 2024-09-11T08:19:11.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:11 smithi150 ceph-mon[29276]: 9.6 starting backfill to osd.6 from (0'0,0'0] MAX to 259'6056 2024-09-11T08:19:11.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:11 smithi150 ceph-mon[39612]: 6.18 scrub starts 2024-09-11T08:19:11.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:11 smithi150 ceph-mon[39612]: 6.18 scrub ok 2024-09-11T08:19:11.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:11 smithi150 ceph-mon[39612]: 5.a scrub starts 2024-09-11T08:19:11.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:11 smithi150 ceph-mon[39612]: 5.a scrub ok 2024-09-11T08:19:11.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:11 smithi150 ceph-mon[39612]: 3.1e scrub starts 2024-09-11T08:19:11.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:11 smithi150 ceph-mon[39612]: 3.1e scrub ok 2024-09-11T08:19:11.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:11 smithi150 ceph-mon[39612]: 9.d starting backfill to osd.4 from (0'0,0'0] MAX to 259'5962 2024-09-11T08:19:11.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:11 smithi150 ceph-mon[39612]: 9.7 starting backfill to osd.2 from (0'0,0'0] MAX to 259'6030 2024-09-11T08:19:11.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:11 smithi150 ceph-mon[39612]: osdmap e262: 8 total, 8 up, 7 in 2024-09-11T08:19:11.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:11 smithi150 ceph-mon[39612]: 9.6 starting backfill to osd.6 from (0'0,0'0] MAX to 259'6056 2024-09-11T08:19:11.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:11.505 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:11.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:19:11.606 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:11.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:11.669 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:11.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:11.770 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:19:11.776 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:2024-09-11T08:19:11.777228+0000 min lat: 0.00117502 max lat: 2.31172 avg lat: 0.00992674 2024-09-11T08:19:11.776 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-11T08:19:11.777 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 60 16 96682 96666 100.671 35.0625 0.0171375 0.00992674 2024-09-11T08:19:11.797 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:11.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:11.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:19:11.899 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:11.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:11.926 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:11.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:12.027 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:19:12.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:12.072 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:12.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:19:12.173 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:19:12.198 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:12.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:12.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:19:12.300 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:19:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:12 smithi181 ceph-mon[32701]: 9.a starting backfill to osd.3 from (0'0,0'0] MAX to 259'6076 2024-09-11T08:19:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:12 smithi181 ceph-mon[32701]: 9.f starting backfill to osd.2 from (0'0,0'0] MAX to 259'6053 2024-09-11T08:19:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:12 smithi181 ceph-mon[32701]: pgmap v524: 148 pgs: 148 active+clean; 5.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.1 KiB/s rd, 89 MiB/s wr, 1.43k op/s 2024-09-11T08:19:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:12 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:19:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:12 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:19:12.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:12 smithi181 ceph-mon[32701]: osdmap e263: 8 total, 8 up, 7 in 2024-09-11T08:19:12.343 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:12.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:12.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:19:12.445 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:19:12.482 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:12.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:12.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:19:12.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:12 smithi150 ceph-mon[29276]: 9.a starting backfill to osd.3 from (0'0,0'0] MAX to 259'6076 2024-09-11T08:19:12.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:12 smithi150 ceph-mon[29276]: 9.f starting backfill to osd.2 from (0'0,0'0] MAX to 259'6053 2024-09-11T08:19:12.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:12 smithi150 ceph-mon[29276]: pgmap v524: 148 pgs: 148 active+clean; 5.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.1 KiB/s rd, 89 MiB/s wr, 1.43k op/s 2024-09-11T08:19:12.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:12 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:19:12.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:12 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:19:12.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:12 smithi150 ceph-mon[29276]: osdmap e263: 8 total, 8 up, 7 in 2024-09-11T08:19:12.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:12 smithi150 ceph-mon[39612]: 9.a starting backfill to osd.3 from (0'0,0'0] MAX to 259'6076 2024-09-11T08:19:12.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:12 smithi150 ceph-mon[39612]: 9.f starting backfill to osd.2 from (0'0,0'0] MAX to 259'6053 2024-09-11T08:19:12.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:12 smithi150 ceph-mon[39612]: pgmap v524: 148 pgs: 148 active+clean; 5.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.1 KiB/s rd, 89 MiB/s wr, 1.43k op/s 2024-09-11T08:19:12.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:12 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:19:12.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:12 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:19:12.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:12 smithi150 ceph-mon[39612]: osdmap e263: 8 total, 8 up, 7 in 2024-09-11T08:19:12.514 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:19:12.584 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:12.611 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:12.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:12.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:12.713 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:19:12.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:12.767 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:12.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:19:12.777 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 61 16 97281 97265 99.6342 37.4375 0.0766846 0.0100281 2024-09-11T08:19:12.869 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:12.917 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:12.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:12.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:13.018 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:13.064 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:13.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:13.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:13.166 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:19:13.193 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:13.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:13.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:19:13.260 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-11T08:19:13.260 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [], 2024-09-11T08:19:13.260 INFO:teuthology.orchestra.run.smithi150.stdout: "complaint_time": 30, 2024-09-11T08:19:13.260 INFO:teuthology.orchestra.run.smithi150.stdout: "num_blocked_ops": 0 2024-09-11T08:19:13.260 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-11T08:19:13.295 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:13.322 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:13.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:13.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:13.424 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:19:13.468 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:13.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:13.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:19:13.570 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:13.612 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:13.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:13.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:13.714 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:19:13.742 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:13.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:13.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:19:13.777 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 62 16 97866 97850 98.617 36.5625 0.0676964 0.0101314 2024-09-11T08:19:13.844 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:13.884 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:13.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:13.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:13.986 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:14.014 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:14.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:14.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:14.116 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:19:14.145 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:14.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:14.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:19:14.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:14 smithi150 ceph-mon[29276]: pgmap v526: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 5.9 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 705 B/s rd, 10 MiB/s wr, 161 op/s; 10771/289452 objects misplaced (3.721%); 20 MiB/s, 320 objects/s recovering 2024-09-11T08:19:14.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:14 smithi150 ceph-mon[39612]: pgmap v526: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 5.9 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 705 B/s rd, 10 MiB/s wr, 161 op/s; 10771/289452 objects misplaced (3.721%); 20 MiB/s, 320 objects/s recovering 2024-09-11T08:19:14.247 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:19:14.276 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:14.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:14.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:19:14.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:14 smithi181 ceph-mon[32701]: pgmap v526: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 5.9 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 705 B/s rd, 10 MiB/s wr, 161 op/s; 10771/289452 objects misplaced (3.721%); 20 MiB/s, 320 objects/s recovering 2024-09-11T08:19:14.378 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:14.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:14.420 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:14.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:14.521 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:19:14.562 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T08:19:14.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:14.571 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:14.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:19:14.672 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:14.722 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:14.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:14.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:14.777 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 63 16 98485 98469 97.6658 38.6875 0.0148829 0.0102318 2024-09-11T08:19:14.787 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-11T08:19:14.824 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:14.851 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:14.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:14.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:14.953 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:14.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:14.980 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:14.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:15.081 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:15.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:15.135 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:15.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:15.236 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:19:15.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:15 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:19:15.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:15 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:19:15.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:15 smithi150 ceph-mon[29276]: osdmap e264: 8 total, 8 up, 7 in 2024-09-11T08:19:15.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:15 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:19:15.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:15 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:19:15.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:15 smithi150 ceph-mon[39612]: osdmap e264: 8 total, 8 up, 7 in 2024-09-11T08:19:15.285 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:15.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:15.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:19:15.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:15 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:19:15.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:15 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:19:15.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:15 smithi181 ceph-mon[32701]: osdmap e264: 8 total, 8 up, 7 in 2024-09-11T08:19:15.323 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:19:15 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:19:15] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:19:15.387 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:19:15.424 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:15.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:15.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:19:15.526 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:15.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:15.596 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:15.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:15.697 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:15.726 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:15.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:15.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:15.777 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 64 16 99061 99045 96.7022 36 0.0258815 0.0103349 2024-09-11T08:19:15.828 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:15.876 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:15.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:15.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:15.978 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:19:16.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:16.037 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:16.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:19:16.138 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:16.166 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:16.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:16.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:16.194 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:19:16.268 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:16.295 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:16.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:16.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:16.298 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:16 smithi150 ceph-mon[29276]: pgmap v528: 148 pgs: 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 139 active+clean; 6.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 573 B/s rd, 23 MiB/s wr, 362 op/s; 6070/293160 objects degraded (2.071%); 46537/293160 objects misplaced (15.874%); 22 MiB/s, 368 objects/s recovering 2024-09-11T08:19:16.298 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:16 smithi150 ceph-mon[29276]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-11T08:19:16.298 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:16 smithi150 ceph-mon[29276]: Health check failed: Degraded data redundancy: 6070/293160 objects degraded (2.071%), 1 pg degraded (PG_DEGRADED) 2024-09-11T08:19:16.298 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:16 smithi150 ceph-mon[29276]: osdmap e265: 8 total, 8 up, 7 in 2024-09-11T08:19:16.299 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:16 smithi150 ceph-mon[39612]: pgmap v528: 148 pgs: 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 139 active+clean; 6.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 573 B/s rd, 23 MiB/s wr, 362 op/s; 6070/293160 objects degraded (2.071%); 46537/293160 objects misplaced (15.874%); 22 MiB/s, 368 objects/s recovering 2024-09-11T08:19:16.299 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:16 smithi150 ceph-mon[39612]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-11T08:19:16.299 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:16 smithi150 ceph-mon[39612]: Health check failed: Degraded data redundancy: 6070/293160 objects degraded (2.071%), 1 pg degraded (PG_DEGRADED) 2024-09-11T08:19:16.299 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:16 smithi150 ceph-mon[39612]: osdmap e265: 8 total, 8 up, 7 in 2024-09-11T08:19:16.299 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:19:16.300 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:19:16.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:16 smithi181 ceph-mon[32701]: pgmap v528: 148 pgs: 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 139 active+clean; 6.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 573 B/s rd, 23 MiB/s wr, 362 op/s; 6070/293160 objects degraded (2.071%); 46537/293160 objects misplaced (15.874%); 22 MiB/s, 368 objects/s recovering 2024-09-11T08:19:16.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:16 smithi181 ceph-mon[32701]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-11T08:19:16.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:16 smithi181 ceph-mon[32701]: Health check failed: Degraded data redundancy: 6070/293160 objects degraded (2.071%), 1 pg degraded (PG_DEGRADED) 2024-09-11T08:19:16.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:16 smithi181 ceph-mon[32701]: osdmap e265: 8 total, 8 up, 7 in 2024-09-11T08:19:16.334 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:19:16.334 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:19:16.371 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:19:16.371 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:19:16.397 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:19:16.413 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:19:16.414 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:19:16.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:16.441 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:16.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:19:16.449 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:19:16.449 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:19:16.481 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:19:16.482 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:19:16.497 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:19:16.516 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:19:16.517 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:19:16.542 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:19:16.551 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:19:16.552 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:19:16.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:16.618 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:16.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:19:16.622 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-11T08:19:16.622 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T08:19:16.622 INFO:tasks.thrashosds.thrasher:Adding osd 5 2024-09-11T08:19:16.622 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd in 5 2024-09-11T08:19:16.719 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:16.748 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:16.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:16.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:16.778 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 65 16 99633 99617 95.7637 35.75 0.0253107 0.0104366 2024-09-11T08:19:16.849 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:19:16.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:16.875 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:16.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:19:16.976 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:17.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:17.057 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:17.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:17.158 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:19:17.196 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:17.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:17.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:19:17.298 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:19:17.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:17 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3393579462' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:19:17.326 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:17.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:17.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:19:17.428 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:19:17.457 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:17.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:17.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:19:17.476 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:17 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3393579462' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:19:17.476 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:17 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3393579462' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:19:17.559 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:17.598 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:17.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:17.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:17.700 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:17.741 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:17.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:17.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:17.778 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 66 16 100268 100252 94.9141 39.6875 0.00845032 0.0105292 2024-09-11T08:19:17.844 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:19:17.890 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:17.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:17.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:19:17.994 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:19:18.021 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:18.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:18.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:19:18.123 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:18.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:18.162 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:18.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:18.263 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:19:18.292 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:18.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:18.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:19:18.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:18 smithi181 ceph-mon[32701]: pgmap v530: 148 pgs: 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 139 active+clean; 6.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 511 B/s rd, 20 MiB/s wr, 323 op/s; 6070/293160 objects degraded (2.071%); 46537/293160 objects misplaced (15.874%); 20 MiB/s, 328 objects/s recovering 2024-09-11T08:19:18.384 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:18 smithi150 ceph-mon[29276]: pgmap v530: 148 pgs: 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 139 active+clean; 6.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 511 B/s rd, 20 MiB/s wr, 323 op/s; 6070/293160 objects degraded (2.071%); 46537/293160 objects misplaced (15.874%); 20 MiB/s, 328 objects/s recovering 2024-09-11T08:19:18.384 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:18 smithi150 ceph-mon[39612]: pgmap v530: 148 pgs: 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 139 active+clean; 6.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 511 B/s rd, 20 MiB/s wr, 323 op/s; 6070/293160 objects degraded (2.071%); 46537/293160 objects misplaced (15.874%); 20 MiB/s, 328 objects/s recovering 2024-09-11T08:19:18.393 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:18.423 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:18.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:18.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:18.525 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:18.555 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:18.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:18.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:18.648 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:19:18 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:19:18.658 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:19:18.701 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:18.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:18.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:19:18.738 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:19:18 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:19:18] "GET /metrics HTTP/1.1" 200 224382 "" "Prometheus/2.43.0" 2024-09-11T08:19:18.778 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 67 16 100898 100882 94.0853 39.375 0.051489 0.0106222 2024-09-11T08:19:18.802 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:18.832 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:18.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:18.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:18.934 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:18.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:18.992 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:18.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:19.093 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:19.124 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:19.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:19.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:19.226 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:19.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:19 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:19:19.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:19 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:19:19.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:19 smithi150 ceph-mon[29276]: osdmap e266: 8 total, 8 up, 7 in 2024-09-11T08:19:19.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:19 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:19:19.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:19 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:19:19.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:19 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:19:19.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:19 smithi150 ceph-mon[39612]: osdmap e266: 8 total, 8 up, 7 in 2024-09-11T08:19:19.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:19 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:19:19.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:19.258 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:19.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:19.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:19 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:19:19.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:19 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:19:19.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:19 smithi181 ceph-mon[32701]: osdmap e266: 8 total, 8 up, 7 in 2024-09-11T08:19:19.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:19 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:19:19.359 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:19.396 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:19.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:19.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:19.498 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:19.539 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:19.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:19.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:19.641 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:19:19.673 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:19.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:19.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:19:19.775 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:19.779 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 68 16 101515 101499 93.2688 38.5625 0.0146206 0.0107146 2024-09-11T08:19:19.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:19.815 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:19.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:19.916 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:19.948 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:19.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:19.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:20.050 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:20.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:20.098 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:20.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:20.199 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:20.227 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:20.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:20.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:20.262 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:19:20.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:20 smithi181 ceph-mon[32701]: pgmap v532: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 139 active+clean; 6.1 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 44 MiB/s wr, 705 op/s; 6070/302139 objects degraded (2.009%); 43125/302139 objects misplaced (14.273%); 49 MiB/s, 795 objects/s recovering 2024-09-11T08:19:20.330 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:20.356 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:20.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:20.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:20.458 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:19:20.487 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:20.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:20.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:19:20.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:20 smithi150 ceph-mon[29276]: pgmap v532: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 139 active+clean; 6.1 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 44 MiB/s wr, 705 op/s; 6070/302139 objects degraded (2.009%); 43125/302139 objects misplaced (14.273%); 49 MiB/s, 795 objects/s recovering 2024-09-11T08:19:20.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:20 smithi150 ceph-mon[39612]: pgmap v532: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 139 active+clean; 6.1 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 44 MiB/s wr, 705 op/s; 6070/302139 objects degraded (2.009%); 43125/302139 objects misplaced (14.273%); 49 MiB/s, 795 objects/s recovering 2024-09-11T08:19:20.590 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:20.612 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:20.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:20.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:20.714 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:20.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:20.756 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:20.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:20.778 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 69 16 102095 102079 92.4425 36.25 0.0356419 0.0108092 2024-09-11T08:19:20.857 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:19:20.879 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:20.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:20.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:19:20.943 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T08:19:20.981 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:21.007 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:21.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:21.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:21.109 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:19:21.137 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:21.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:21.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:19:21.239 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:19:21.294 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:21.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:21.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:19:21.313 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:21 smithi150 ceph-mon[29276]: Health check update: Degraded data redundancy: 6070/302139 objects degraded (2.009%), 1 pg degraded (PG_DEGRADED) 2024-09-11T08:19:21.313 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:21 smithi150 ceph-mon[29276]: osdmap e267: 8 total, 8 up, 7 in 2024-09-11T08:19:21.313 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:21 smithi150 ceph-mon[39612]: Health check update: Degraded data redundancy: 6070/302139 objects degraded (2.009%), 1 pg degraded (PG_DEGRADED) 2024-09-11T08:19:21.314 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:21 smithi150 ceph-mon[39612]: osdmap e267: 8 total, 8 up, 7 in 2024-09-11T08:19:21.396 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:19:21.424 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:21.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:21.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:19:21.526 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:21.557 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:21.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:21.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:21.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:21 smithi181 ceph-mon[32701]: Health check update: Degraded data redundancy: 6070/302139 objects degraded (2.009%), 1 pg degraded (PG_DEGRADED) 2024-09-11T08:19:21.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:21 smithi181 ceph-mon[32701]: osdmap e267: 8 total, 8 up, 7 in 2024-09-11T08:19:21.660 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:21.703 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:21.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:21.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:21.778 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 70 16 102632 102616 91.6014 33.5625 0.00859335 0.0109097 2024-09-11T08:19:21.805 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:19:21.832 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:21.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:21.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:19:21.934 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:21.972 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:21.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:21.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:22.074 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:22.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:22.122 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:22.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:22.222 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:22.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:22.265 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:22.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:22.366 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:22.394 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:22.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:22.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:22.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:22 smithi150 ceph-mon[39612]: pgmap v534: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 139 active+clean; 6.1 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 31 MiB/s wr, 499 op/s; 6070/302139 objects degraded (2.009%); 43125/302139 objects misplaced (14.273%); 44 MiB/s, 699 objects/s recovering 2024-09-11T08:19:22.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:22 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:19:22.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:22 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:19:22.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:22 smithi150 ceph-mon[29276]: pgmap v534: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 139 active+clean; 6.1 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 31 MiB/s wr, 499 op/s; 6070/302139 objects degraded (2.009%); 43125/302139 objects misplaced (14.273%); 44 MiB/s, 699 objects/s recovering 2024-09-11T08:19:22.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:22 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:19:22.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:22 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:19:22.496 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:22.525 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:22.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:22.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:22.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:22 smithi181 ceph-mon[32701]: pgmap v534: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 139 active+clean; 6.1 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 31 MiB/s wr, 499 op/s; 6070/302139 objects degraded (2.009%); 43125/302139 objects misplaced (14.273%); 44 MiB/s, 699 objects/s recovering 2024-09-11T08:19:22.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:22 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:19:22.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:22 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:19:22.627 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:19:22.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:22.682 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:22.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:19:22.778 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 71 16 103251 103235 90.8562 38.6875 0.037763 0.0109993 2024-09-11T08:19:22.784 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:22.810 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:22.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:22.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:22.911 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:22.939 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:22.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:22.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:23.042 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:23.070 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:23.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:23.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:23.172 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:19:23.200 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:23.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:23.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:19:23.302 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:23.330 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:23.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:23.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:23.432 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:19:23.458 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:23.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:23.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:19:23.560 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:23.585 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:23.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:23.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:23.687 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:23.734 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:23.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:23.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:23.779 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 72 16 103819 103803 90.0874 35.5 0.00934095 0.0110941 2024-09-11T08:19:23.836 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:19:23.881 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:23.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:23.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:19:23.982 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:24.007 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:24.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:24.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:24.109 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:19:24.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:24.154 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:24.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:19:24.255 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:24.278 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:24.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:24.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:24.380 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:24.403 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:24.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:24.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:24.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:24 smithi150 ceph-mon[29276]: pgmap v535: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 139 active+clean; 6.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.4 KiB/s rd, 35 MiB/s wr, 561 op/s; 6070/305538 objects degraded (1.987%); 40173/305538 objects misplaced (13.148%); 62 MiB/s, 999 objects/s recovering 2024-09-11T08:19:24.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:24 smithi150 ceph-mon[39612]: pgmap v535: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 139 active+clean; 6.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.4 KiB/s rd, 35 MiB/s wr, 561 op/s; 6070/305538 objects degraded (1.987%); 40173/305538 objects misplaced (13.148%); 62 MiB/s, 999 objects/s recovering 2024-09-11T08:19:24.505 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:19:24.527 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:24.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:24.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:19:24.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:24 smithi181 ceph-mon[32701]: pgmap v535: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 139 active+clean; 6.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.4 KiB/s rd, 35 MiB/s wr, 561 op/s; 6070/305538 objects degraded (1.987%); 40173/305538 objects misplaced (13.148%); 62 MiB/s, 999 objects/s recovering 2024-09-11T08:19:24.629 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:24.665 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:24.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:24.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:24.767 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:24.778 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 73 16 104217 104201 89.1942 24.875 0.0162634 0.0111524 2024-09-11T08:19:24.790 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:24.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:24.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:24.892 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:19:24.929 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:24.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:24.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:19:24.979 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-09-11T08:19:24.979 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:19:25.032 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:25.053 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:25.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:25.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:25.124 INFO:teuthology.orchestra.run.smithi150.stderr:nodeep-scrub is unset 2024-09-11T08:19:25.155 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:19:25.178 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:25.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:25.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:19:25.280 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:19:25.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:25.325 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:25.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:19:25.426 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:25.450 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:25.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:25.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:25.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:25 smithi150 ceph-mon[29276]: osdmap e268: 8 total, 8 up, 7 in 2024-09-11T08:19:25.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:25 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1904264966' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:19:25.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:25 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:19:25.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:25 smithi150 ceph-mon[39612]: osdmap e268: 8 total, 8 up, 7 in 2024-09-11T08:19:25.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:25 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1904264966' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:19:25.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:25 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:19:25.552 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:25.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:19:25 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:19:25] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:19:25.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:25 smithi181 ceph-mon[32701]: osdmap e268: 8 total, 8 up, 7 in 2024-09-11T08:19:25.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:25 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1904264966' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:19:25.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:25 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:19:25.576 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:25.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:25.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:25.678 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:19:25.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:25.737 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:25.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:19:25.779 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 74 16 104669 104653 88.3706 28.25 0.0330026 0.0113103 2024-09-11T08:19:25.839 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:25.866 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:25.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:25.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:25.968 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:19:25.992 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:25.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:25.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:19:26.093 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-11T08:19:26.093 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:26.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:26.154 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:26.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:26.255 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:26.305 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:26.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:26.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:26.311 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:26 smithi150 ceph-mon[29276]: 4.13 scrub starts 2024-09-11T08:19:26.312 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:26 smithi150 ceph-mon[29276]: pgmap v537: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 139 active+clean; 6.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 744 B/s rd, 44 MiB/s wr, 705 op/s; 6070/311016 objects degraded (1.952%); 39562/311016 objects misplaced (12.720%); 48 MiB/s, 765 objects/s recovering 2024-09-11T08:19:26.312 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:26 smithi150 ceph-mon[29276]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:19:26.312 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:26 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:19:26.312 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:26 smithi150 ceph-mon[29276]: osdmap e269: 8 total, 8 up, 7 in 2024-09-11T08:19:26.312 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:26 smithi150 ceph-mon[29276]: Health check update: Degraded data redundancy: 6070/311016 objects degraded (1.952%), 1 pg degraded (PG_DEGRADED) 2024-09-11T08:19:26.312 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:26 smithi150 ceph-mon[39612]: 4.13 scrub starts 2024-09-11T08:19:26.312 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:26 smithi150 ceph-mon[39612]: pgmap v537: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 139 active+clean; 6.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 744 B/s rd, 44 MiB/s wr, 705 op/s; 6070/311016 objects degraded (1.952%); 39562/311016 objects misplaced (12.720%); 48 MiB/s, 765 objects/s recovering 2024-09-11T08:19:26.312 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:26 smithi150 ceph-mon[39612]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:19:26.312 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:26 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:19:26.312 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:26 smithi150 ceph-mon[39612]: osdmap e269: 8 total, 8 up, 7 in 2024-09-11T08:19:26.312 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:26 smithi150 ceph-mon[39612]: Health check update: Degraded data redundancy: 6070/311016 objects degraded (1.952%), 1 pg degraded (PG_DEGRADED) 2024-09-11T08:19:26.407 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:26.431 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:26.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:26.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:26.532 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:19:26.557 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:26.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:26.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:19:26.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:26 smithi181 ceph-mon[32701]: 4.13 scrub starts 2024-09-11T08:19:26.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:26 smithi181 ceph-mon[32701]: pgmap v537: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 139 active+clean; 6.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 744 B/s rd, 44 MiB/s wr, 705 op/s; 6070/311016 objects degraded (1.952%); 39562/311016 objects misplaced (12.720%); 48 MiB/s, 765 objects/s recovering 2024-09-11T08:19:26.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:26 smithi181 ceph-mon[32701]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:19:26.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:26 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:19:26.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:26 smithi181 ceph-mon[32701]: osdmap e269: 8 total, 8 up, 7 in 2024-09-11T08:19:26.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:26 smithi181 ceph-mon[32701]: Health check update: Degraded data redundancy: 6070/311016 objects degraded (1.952%), 1 pg degraded (PG_DEGRADED) 2024-09-11T08:19:26.659 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:19:26.755 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:26.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:26.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:19:26.772 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:19:26.780 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 75 16 105306 105290 87.7218 39.8125 0.0144437 0.0113939 2024-09-11T08:19:26.857 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:26.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:26.917 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:26.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:27.018 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:19:27.042 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:27.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:27.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:19:27.144 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:19:27.188 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:27.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:27.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:19:27.293 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:19:27.338 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:27.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:27.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:19:27.356 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:27 smithi150 ceph-mon[29276]: 4.1a scrub starts 2024-09-11T08:19:27.356 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:27 smithi150 ceph-mon[39612]: 4.1a scrub starts 2024-09-11T08:19:27.440 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:19:27.463 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:27.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:27.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:19:27.565 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:27.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:27 smithi181 ceph-mon[32701]: 4.1a scrub starts 2024-09-11T08:19:27.590 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:27.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:27.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:27.692 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:27.715 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:27.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:27.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:27.780 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 76 16 105877 105861 87.0371 35.6875 0.0561626 0.0114811 2024-09-11T08:19:27.817 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:19:27.863 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:27.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:27.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:19:27.965 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:28.016 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:28.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:28.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:28.118 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:28.154 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:28.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:28.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:28.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:28 smithi150 ceph-mon[39612]: pgmap v539: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 139 active+clean; 6.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 804 B/s rd, 29 MiB/s wr, 465 op/s; 6070/311016 objects degraded (1.952%); 39562/311016 objects misplaced (12.720%); 42 MiB/s, 667 objects/s recovering 2024-09-11T08:19:28.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:28 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3567046301' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-09-11T08:19:28.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:28 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-09-11T08:19:28.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:27 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mon-a[29272]: 2024-09-11T08:19:27.892+0000 7f1b17b61640 -1 mon.a@0(leader).osd e269 definitely_dead 0 2024-09-11T08:19:28.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:28 smithi150 ceph-mon[29276]: pgmap v539: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 139 active+clean; 6.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 804 B/s rd, 29 MiB/s wr, 465 op/s; 6070/311016 objects degraded (1.952%); 39562/311016 objects misplaced (12.720%); 42 MiB/s, 667 objects/s recovering 2024-09-11T08:19:28.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:28 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3567046301' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-09-11T08:19:28.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:28 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-09-11T08:19:28.255 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:19:28.277 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:28.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:28.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:19:28.378 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:28.394 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:28 smithi181 ceph-mon[32701]: pgmap v539: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 139 active+clean; 6.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 804 B/s rd, 29 MiB/s wr, 465 op/s; 6070/311016 objects degraded (1.952%); 39562/311016 objects misplaced (12.720%); 42 MiB/s, 667 objects/s recovering 2024-09-11T08:19:28.394 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:28 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3567046301' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-09-11T08:19:28.394 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:28 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-09-11T08:19:28.401 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:28.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:28.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:28.505 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:19:28.527 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:28.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:28.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:19:28.628 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:19:28.651 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:28.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:28.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:19:28.753 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:28.776 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:28.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:28.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:28.780 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 77 16 106095 106079 86.0838 13.625 0.0132691 0.011514 2024-09-11T08:19:28.816 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T08:19:28.823 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:19:28 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:19:28.840 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:19:28 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:19:28] "GET /metrics HTTP/1.1" 200 224586 "" "Prometheus/2.43.0" 2024-09-11T08:19:28.878 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:28.899 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:28.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:28.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:29.001 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:29.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:29.023 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:29.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:29.124 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:29.158 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:29.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:29.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:29.260 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:29.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:29.293 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:29.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:29.394 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:29.416 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:29.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:29.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:29.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:29 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-09-11T08:19:29.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:29 smithi150 ceph-mon[29276]: osdmap e270: 8 total, 8 up, 8 in 2024-09-11T08:19:29.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:29 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:19:29.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:29 smithi150 ceph-mon[29276]: osdmap e271: 8 total, 8 up, 8 in 2024-09-11T08:19:29.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:29 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-09-11T08:19:29.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:29 smithi150 ceph-mon[39612]: osdmap e270: 8 total, 8 up, 8 in 2024-09-11T08:19:29.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:29 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:19:29.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:29 smithi150 ceph-mon[39612]: osdmap e271: 8 total, 8 up, 8 in 2024-09-11T08:19:29.518 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:19:29.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:29.541 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:29.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:19:29.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:29 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-09-11T08:19:29.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:29 smithi181 ceph-mon[32701]: osdmap e270: 8 total, 8 up, 8 in 2024-09-11T08:19:29.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:29 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:19:29.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:29 smithi181 ceph-mon[32701]: osdmap e271: 8 total, 8 up, 8 in 2024-09-11T08:19:29.642 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:19:29.664 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:29.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:29.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:19:29.766 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:19:29.780 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 78 16 106130 106114 85.0083 2.1875 0.00286054 0.0116086 2024-09-11T08:19:29.788 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:29.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:29.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:19:29.890 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:29.913 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:29.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:29.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:30.015 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:30.037 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:30.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:30.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:30.139 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:19:30.169 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:30.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:30.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:19:30.271 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:19:30.303 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:30.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:30.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:19:30.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:30 smithi181 ceph-mon[32701]: 6.6 scrub starts 2024-09-11T08:19:30.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:30 smithi181 ceph-mon[32701]: 6.6 scrub ok 2024-09-11T08:19:30.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:30 smithi181 ceph-mon[32701]: 5.1d deep-scrub starts 2024-09-11T08:19:30.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:30 smithi181 ceph-mon[32701]: 5.1d deep-scrub ok 2024-09-11T08:19:30.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:30 smithi181 ceph-mon[32701]: 2.0 scrub starts 2024-09-11T08:19:30.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:30 smithi181 ceph-mon[32701]: 2.0 scrub ok 2024-09-11T08:19:30.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:30 smithi181 ceph-mon[32701]: pgmap v541: 148 pgs: 7 remapped+peering, 15 peering, 1 active+remapped+backfilling, 125 active+clean; 6.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 46 MiB/s wr, 730 op/s; 3751/318669 objects misplaced (1.177%); 38 MiB/s, 607 objects/s recovering 2024-09-11T08:19:30.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:30 smithi181 ceph-mon[32701]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-11T08:19:30.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:30 smithi181 ceph-mon[32701]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6070/311016 objects degraded (1.952%), 1 pg degraded) 2024-09-11T08:19:30.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:30 smithi181 ceph-mon[32701]: Cluster is now healthy 2024-09-11T08:19:30.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:30 smithi181 ceph-mon[32701]: osdmap e272: 8 total, 8 up, 8 in 2024-09-11T08:19:30.406 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:30.448 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:30.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:30.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:30.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:30 smithi150 ceph-mon[29276]: 6.6 scrub starts 2024-09-11T08:19:30.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:30 smithi150 ceph-mon[29276]: 6.6 scrub ok 2024-09-11T08:19:30.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:30 smithi150 ceph-mon[29276]: 5.1d deep-scrub starts 2024-09-11T08:19:30.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:30 smithi150 ceph-mon[29276]: 5.1d deep-scrub ok 2024-09-11T08:19:30.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:30 smithi150 ceph-mon[29276]: 2.0 scrub starts 2024-09-11T08:19:30.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:30 smithi150 ceph-mon[29276]: 2.0 scrub ok 2024-09-11T08:19:30.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:30 smithi150 ceph-mon[29276]: pgmap v541: 148 pgs: 7 remapped+peering, 15 peering, 1 active+remapped+backfilling, 125 active+clean; 6.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 46 MiB/s wr, 730 op/s; 3751/318669 objects misplaced (1.177%); 38 MiB/s, 607 objects/s recovering 2024-09-11T08:19:30.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:30 smithi150 ceph-mon[29276]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-11T08:19:30.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:30 smithi150 ceph-mon[29276]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6070/311016 objects degraded (1.952%), 1 pg degraded) 2024-09-11T08:19:30.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:30 smithi150 ceph-mon[29276]: Cluster is now healthy 2024-09-11T08:19:30.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:30 smithi150 ceph-mon[29276]: osdmap e272: 8 total, 8 up, 8 in 2024-09-11T08:19:30.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:30 smithi150 ceph-mon[39612]: 6.6 scrub starts 2024-09-11T08:19:30.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:30 smithi150 ceph-mon[39612]: 6.6 scrub ok 2024-09-11T08:19:30.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:30 smithi150 ceph-mon[39612]: 5.1d deep-scrub starts 2024-09-11T08:19:30.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:30 smithi150 ceph-mon[39612]: 5.1d deep-scrub ok 2024-09-11T08:19:30.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:30 smithi150 ceph-mon[39612]: 2.0 scrub starts 2024-09-11T08:19:30.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:30 smithi150 ceph-mon[39612]: 2.0 scrub ok 2024-09-11T08:19:30.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:30 smithi150 ceph-mon[39612]: pgmap v541: 148 pgs: 7 remapped+peering, 15 peering, 1 active+remapped+backfilling, 125 active+clean; 6.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 46 MiB/s wr, 730 op/s; 3751/318669 objects misplaced (1.177%); 38 MiB/s, 607 objects/s recovering 2024-09-11T08:19:30.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:30 smithi150 ceph-mon[39612]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-11T08:19:30.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:30 smithi150 ceph-mon[39612]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6070/311016 objects degraded (1.952%), 1 pg degraded) 2024-09-11T08:19:30.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:30 smithi150 ceph-mon[39612]: Cluster is now healthy 2024-09-11T08:19:30.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:30 smithi150 ceph-mon[39612]: osdmap e272: 8 total, 8 up, 8 in 2024-09-11T08:19:30.553 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:30.577 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:30.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:30.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:30.679 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:19:30.712 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:30.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:30.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:19:30.781 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 79 16 106776 106760 84.4434 40.375 0.00525979 0.0118371 2024-09-11T08:19:30.814 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:19:30.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:30.836 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:30.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:19:30.937 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:30.972 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:30.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:30.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:31.074 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:31.095 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:31.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:31.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:31.197 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:19:31.233 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:31.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:31.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:19:31.335 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:31.376 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:31.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:31.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:31.478 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:31.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:31 smithi150 ceph-mon[29276]: 6.4 scrub starts 2024-09-11T08:19:31.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:31 smithi150 ceph-mon[29276]: 4.14 deep-scrub starts 2024-09-11T08:19:31.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:31 smithi150 ceph-mon[29276]: 9.d starting backfill to osd.5 from (252'2051,267'6471] MIN to 269'6588 2024-09-11T08:19:31.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:31 smithi150 ceph-mon[29276]: 6.1a scrub starts 2024-09-11T08:19:31.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:31 smithi150 ceph-mon[39612]: 6.4 scrub starts 2024-09-11T08:19:31.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:31 smithi150 ceph-mon[39612]: 4.14 deep-scrub starts 2024-09-11T08:19:31.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:31 smithi150 ceph-mon[39612]: 9.d starting backfill to osd.5 from (252'2051,267'6471] MIN to 269'6588 2024-09-11T08:19:31.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:31 smithi150 ceph-mon[39612]: 6.1a scrub starts 2024-09-11T08:19:31.513 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:31.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:31.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:31.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:31 smithi181 ceph-mon[32701]: 6.4 scrub starts 2024-09-11T08:19:31.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:31 smithi181 ceph-mon[32701]: 4.14 deep-scrub starts 2024-09-11T08:19:31.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:31 smithi181 ceph-mon[32701]: 9.d starting backfill to osd.5 from (252'2051,267'6471] MIN to 269'6588 2024-09-11T08:19:31.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:31 smithi181 ceph-mon[32701]: 6.1a scrub starts 2024-09-11T08:19:31.615 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:31.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:31.654 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:31.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:31.754 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:19:31.780 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:2024-09-11T08:19:31.781313+0000 min lat: 0.00117502 max lat: 2.31172 avg lat: 0.0118793 2024-09-11T08:19:31.781 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-11T08:19:31.781 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 80 16 107726 107710 84.13 59.375 0.00771701 0.0118793 2024-09-11T08:19:31.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:31.782 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:31.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:19:31.883 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:31.912 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:31.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:31.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:32.014 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:19:32.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:32.049 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:32.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:19:32.150 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:19:32.185 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:32.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:32.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:19:32.287 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:32.308 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:32.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:32.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:32.410 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:32.445 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:32.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:32.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:32.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:32 smithi150 ceph-mon[29276]: pgmap v544: 148 pgs: 7 remapped+peering, 15 peering, 1 active+remapped+backfilling, 125 active+clean; 6.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 877 B/s rd, 27 MiB/s wr, 437 op/s; 3751/318669 objects misplaced (1.177%); 27 MiB/s, 437 objects/s recovering 2024-09-11T08:19:32.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:32 smithi150 ceph-mon[39612]: pgmap v544: 148 pgs: 7 remapped+peering, 15 peering, 1 active+remapped+backfilling, 125 active+clean; 6.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 877 B/s rd, 27 MiB/s wr, 437 op/s; 3751/318669 objects misplaced (1.177%); 27 MiB/s, 437 objects/s recovering 2024-09-11T08:19:32.547 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:32.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:32 smithi181 ceph-mon[32701]: pgmap v544: 148 pgs: 7 remapped+peering, 15 peering, 1 active+remapped+backfilling, 125 active+clean; 6.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 877 B/s rd, 27 MiB/s wr, 437 op/s; 3751/318669 objects misplaced (1.177%); 27 MiB/s, 437 objects/s recovering 2024-09-11T08:19:32.581 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:32.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:32.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:32.683 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:32.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:32.711 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:32.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:32.781 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 81 16 108658 108642 83.8105 58.25 0.00581381 0.0119253 2024-09-11T08:19:32.812 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:19:32.852 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:32.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:32.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:19:32.954 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:32.988 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:32.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:32.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:33.091 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:19:33.112 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:33.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:33.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:19:33.214 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:19:33.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:33 smithi150 ceph-mon[29276]: 4.c scrub starts 2024-09-11T08:19:33.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:33 smithi150 ceph-mon[39612]: 4.c scrub starts 2024-09-11T08:19:33.250 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:33.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:33.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:19:33.352 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:33.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:33.379 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:33.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:33.480 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:33.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:33.502 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:33.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:33.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:33 smithi181 ceph-mon[32701]: 4.c scrub starts 2024-09-11T08:19:33.603 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:19:33.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:33.638 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:33.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:19:33.739 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:33.776 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:33.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:33.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:33.781 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 82 16 109488 109472 83.421 51.875 0.0135704 0.011983 2024-09-11T08:19:33.878 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:19:33.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:33.912 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:33.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:19:34.013 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:34.050 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:34.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:34.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:34.152 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:19:34.175 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:34.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:34.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:19:34.277 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:19:34.314 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:34.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:34.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:19:34.416 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:34.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:34.454 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:34.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:34.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:34 smithi150 ceph-mon[29276]: pgmap v545: 148 pgs: 1 active+remapped+backfill_wait, 7 remapped+peering, 15 peering, 1 active+remapped+backfilling, 124 active+clean; 6.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.7 KiB/s rd, 36 MiB/s wr, 577 op/s; 10518/321390 objects misplaced (3.273%); 27 MiB/s, 428 objects/s recovering 2024-09-11T08:19:34.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:34 smithi150 ceph-mon[29276]: 6.14 scrub starts 2024-09-11T08:19:34.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:34 smithi150 ceph-mon[29276]: 3.14 scrub starts 2024-09-11T08:19:34.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:34 smithi150 ceph-mon[39612]: pgmap v545: 148 pgs: 1 active+remapped+backfill_wait, 7 remapped+peering, 15 peering, 1 active+remapped+backfilling, 124 active+clean; 6.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.7 KiB/s rd, 36 MiB/s wr, 577 op/s; 10518/321390 objects misplaced (3.273%); 27 MiB/s, 428 objects/s recovering 2024-09-11T08:19:34.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:34 smithi150 ceph-mon[39612]: 6.14 scrub starts 2024-09-11T08:19:34.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:34 smithi150 ceph-mon[39612]: 3.14 scrub starts 2024-09-11T08:19:34.554 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:34.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:34 smithi181 ceph-mon[32701]: pgmap v545: 148 pgs: 1 active+remapped+backfill_wait, 7 remapped+peering, 15 peering, 1 active+remapped+backfilling, 124 active+clean; 6.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.7 KiB/s rd, 36 MiB/s wr, 577 op/s; 10518/321390 objects misplaced (3.273%); 27 MiB/s, 428 objects/s recovering 2024-09-11T08:19:34.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:34 smithi181 ceph-mon[32701]: 6.14 scrub starts 2024-09-11T08:19:34.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:34 smithi181 ceph-mon[32701]: 3.14 scrub starts 2024-09-11T08:19:34.576 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:34.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:34.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:34.677 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:34.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:34.711 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:34.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:34.781 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 83 16 110541 110525 83.2088 65.8125 0.0546704 0.0120106 2024-09-11T08:19:34.812 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:34.847 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:34.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:34.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:34.949 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:19:34.984 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:34.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:34.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:19:35.086 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:19:35.124 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:35.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:35.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:19:35.226 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:19:35.264 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:35.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:35.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:19:35.366 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:19:35.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:35.412 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:35.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:19:35.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:35 smithi150 ceph-mon[29276]: 4.2 scrub starts 2024-09-11T08:19:35.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:35 smithi150 ceph-mon[29276]: 6.2 scrub starts 2024-09-11T08:19:35.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:35 smithi150 ceph-mon[39612]: 4.2 scrub starts 2024-09-11T08:19:35.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:35 smithi150 ceph-mon[39612]: 6.2 scrub starts 2024-09-11T08:19:35.512 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:35.547 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:35.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:35.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:35.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:35 smithi181 ceph-mon[32701]: 4.2 scrub starts 2024-09-11T08:19:35.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:35 smithi181 ceph-mon[32701]: 6.2 scrub starts 2024-09-11T08:19:35.574 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:19:35 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:19:35] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:19:35.648 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:35.686 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:35.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:35.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:35.781 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 84 16 111689 111673 83.0723 71.75 0.00826218 0.0120331 2024-09-11T08:19:35.787 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:35.816 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:35.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:35.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:35.918 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:19:35.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:35.952 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:35.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:19:36.053 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:36.100 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:36.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:36.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:36.202 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:36.233 INFO:teuthology.orchestra.run.smithi150.stderr:marked in osd.5. 2024-09-11T08:19:36.238 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:36.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:36.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:36.341 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:19:36.384 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:36.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:36.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:19:36.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:36 smithi150 ceph-mon[29276]: 3.4 scrub starts 2024-09-11T08:19:36.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:36 smithi150 ceph-mon[29276]: pgmap v546: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 143 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 750 B/s rd, 42 MiB/s wr, 667 op/s; 592/327693 objects degraded (0.181%); 7675/327693 objects misplaced (2.342%); 22 MiB/s, 349 objects/s recovering 2024-09-11T08:19:36.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:36 smithi150 ceph-mon[29276]: Health check failed: Degraded data redundancy: 592/327693 objects degraded (0.181%), 1 pg degraded (PG_DEGRADED) 2024-09-11T08:19:36.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:36 smithi150 ceph-mon[29276]: 4.1d scrub starts 2024-09-11T08:19:36.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:36 smithi150 ceph-mon[39612]: 3.4 scrub starts 2024-09-11T08:19:36.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:36 smithi150 ceph-mon[39612]: pgmap v546: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 143 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 750 B/s rd, 42 MiB/s wr, 667 op/s; 592/327693 objects degraded (0.181%); 7675/327693 objects misplaced (2.342%); 22 MiB/s, 349 objects/s recovering 2024-09-11T08:19:36.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:36 smithi150 ceph-mon[39612]: Health check failed: Degraded data redundancy: 592/327693 objects degraded (0.181%), 1 pg degraded (PG_DEGRADED) 2024-09-11T08:19:36.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:36 smithi150 ceph-mon[39612]: 4.1d scrub starts 2024-09-11T08:19:36.491 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:36.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:36.527 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:36.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:36.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:36 smithi181 ceph-mon[32701]: 3.4 scrub starts 2024-09-11T08:19:36.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:36 smithi181 ceph-mon[32701]: pgmap v546: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 143 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 750 B/s rd, 42 MiB/s wr, 667 op/s; 592/327693 objects degraded (0.181%); 7675/327693 objects misplaced (2.342%); 22 MiB/s, 349 objects/s recovering 2024-09-11T08:19:36.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:36 smithi181 ceph-mon[32701]: Health check failed: Degraded data redundancy: 592/327693 objects degraded (0.181%), 1 pg degraded (PG_DEGRADED) 2024-09-11T08:19:36.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:36 smithi181 ceph-mon[32701]: 4.1d scrub starts 2024-09-11T08:19:36.628 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:36.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:36.665 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:36.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:36.766 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:19:36.781 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 85 16 112884 112868 82.9733 74.6875 0.0147684 0.0120473 2024-09-11T08:19:36.810 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:36.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:36.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:19:36.912 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:19:36.947 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:36.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:36.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:19:37.050 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:37.086 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:37.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:37.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:37.189 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:37.220 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:37.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:37.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:37.224 INFO:tasks.thrashosds.thrasher:Added osd 5 2024-09-11T08:19:37.254 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:19:37.323 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:19:37.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:37.369 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:37.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:19:37.470 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:19:37.508 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:37.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:37.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:19:37.609 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:37.644 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:37.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:37.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:37.689 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:19:37.746 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:37.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:37.782 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:37.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:37.782 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 86 15 113994 113979 82.8159 69.4375 0.00428695 0.0120707 2024-09-11T08:19:37.798 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:37 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:19:37.798 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:37 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:19:37.798 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:37 smithi150 ceph-mon[29276]: pgmap v547: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 143 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 23 MiB/s wr, 376 op/s; 592/327693 objects degraded (0.181%); 7675/327693 objects misplaced (2.342%); 8.2 MiB/s, 134 objects/s recovering 2024-09-11T08:19:37.798 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:37 smithi150 ceph-mon[29276]: 9.4 scrub starts 2024-09-11T08:19:37.798 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:37 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:19:37.799 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:37 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:19:37.799 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:37 smithi150 ceph-mon[39612]: pgmap v547: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 143 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 23 MiB/s wr, 376 op/s; 592/327693 objects degraded (0.181%); 7675/327693 objects misplaced (2.342%); 8.2 MiB/s, 134 objects/s recovering 2024-09-11T08:19:37.799 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:37 smithi150 ceph-mon[39612]: 9.4 scrub starts 2024-09-11T08:19:37.864 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:19:37.865 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:19:37.882 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:37.893 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:19:37.893 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:19:37.915 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:37.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:37.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:37.926 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:19:37.926 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:19:37.952 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:19:37.952 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:19:37.973 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:19:37.973 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:19:37.992 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:19:37.993 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:19:38.010 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:19:38.011 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:19:38.017 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:38.029 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:19:38.029 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:19:38.044 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:38.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:38.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:38.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:37 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:19:38.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:37 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:19:38.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:37 smithi181 ceph-mon[32701]: pgmap v547: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 143 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 23 MiB/s wr, 376 op/s; 592/327693 objects degraded (0.181%); 7675/327693 objects misplaced (2.342%); 8.2 MiB/s, 134 objects/s recovering 2024-09-11T08:19:38.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:37 smithi181 ceph-mon[32701]: 9.4 scrub starts 2024-09-11T08:19:38.146 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:38.168 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:38.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:38.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:38.270 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:19:38.293 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:38.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:38.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:19:38.395 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:38.418 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:38.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:38.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:38.520 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:19:38.544 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:38.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:38.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:19:38.646 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:38.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:38.668 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:38.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:38.738 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:19:38 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:19:38] "GET /metrics HTTP/1.1" 200 224565 "" "Prometheus/2.43.0" 2024-09-11T08:19:38.757 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:19:38 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:19:38.769 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:19:38.781 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 87 16 114146 114130 81.9725 9.4375 0.00390473 0.0120683 2024-09-11T08:19:38.792 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:38.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:38.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:19:38.894 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:19:38.921 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:38.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:38.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:19:39.022 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:39.044 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:39.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:39.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:39.058 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:38 smithi150 ceph-mon[29276]: osdmap e273: 8 total, 8 up, 8 in 2024-09-11T08:19:39.058 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:38 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3040763897' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:19:39.058 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:38 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:19:39.058 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:38 smithi150 ceph-mon[39612]: osdmap e273: 8 total, 8 up, 8 in 2024-09-11T08:19:39.058 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:38 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3040763897' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:19:39.059 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:38 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:19:39.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:38 smithi181 ceph-mon[32701]: osdmap e273: 8 total, 8 up, 8 in 2024-09-11T08:19:39.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:38 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3040763897' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:19:39.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:38 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:19:39.146 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:39.169 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:39.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:39.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:39.271 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:39.294 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:39.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:39.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:39.396 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:39.418 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:39.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:39.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:39.520 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:19:39.542 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:39.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:39.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:19:39.644 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:19:39.666 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:39.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:39.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:19:39.768 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:19:39.781 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 88 16 114269 114253 81.1285 7.6875 0.00310589 0.0121944 2024-09-11T08:19:39.795 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:39.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:39.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:19:39.897 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:39.933 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:39.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:39.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:40.035 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:19:40.067 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:40.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:40.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:19:40.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:39 smithi181 ceph-mon[32701]: osdmap e274: 8 total, 8 up, 8 in 2024-09-11T08:19:40.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:39 smithi181 ceph-mon[32701]: pgmap v550: 148 pgs: 1 active+remapped+backfill_wait, 1 peering, 2 active+undersized+remapped, 1 active+recovering+undersized+remapped, 143 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.01k op/s; 7469/342936 objects misplaced (2.178%); 17 MiB/s, 275 objects/s recovering 2024-09-11T08:19:40.169 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:40.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:40.208 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:40.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:40.223 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:39 smithi150 ceph-mon[39612]: osdmap e274: 8 total, 8 up, 8 in 2024-09-11T08:19:40.224 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:39 smithi150 ceph-mon[39612]: pgmap v550: 148 pgs: 1 active+remapped+backfill_wait, 1 peering, 2 active+undersized+remapped, 1 active+recovering+undersized+remapped, 143 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.01k op/s; 7469/342936 objects misplaced (2.178%); 17 MiB/s, 275 objects/s recovering 2024-09-11T08:19:40.226 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:39 smithi150 ceph-mon[29276]: osdmap e274: 8 total, 8 up, 8 in 2024-09-11T08:19:40.226 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:39 smithi150 ceph-mon[29276]: pgmap v550: 148 pgs: 1 active+remapped+backfill_wait, 1 peering, 2 active+undersized+remapped, 1 active+recovering+undersized+remapped, 143 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.01k op/s; 7469/342936 objects misplaced (2.178%); 17 MiB/s, 275 objects/s recovering 2024-09-11T08:19:40.309 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:19:40.361 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:40.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:40.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:19:40.463 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:40.500 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:40.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:40.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:40.602 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:40.629 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:40.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:40.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:40.731 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:40.757 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:40.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:40.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:40.782 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 89 16 115405 115389 81.0147 71 0.00341543 0.0123291 2024-09-11T08:19:40.859 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:19:40.883 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:40.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:40.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:19:40.985 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:41.008 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:41.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:41.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:41.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:40 smithi181 ceph-mon[32701]: 6.c scrub starts 2024-09-11T08:19:41.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:40 smithi181 ceph-mon[32701]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 592/327693 objects degraded (0.181%), 1 pg degraded) 2024-09-11T08:19:41.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:40 smithi181 ceph-mon[32701]: Cluster is now healthy 2024-09-11T08:19:41.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:40 smithi181 ceph-mon[32701]: osdmap e275: 8 total, 8 up, 8 in 2024-09-11T08:19:41.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:40 smithi181 ceph-mon[32701]: osdmap e276: 8 total, 8 up, 8 in 2024-09-11T08:19:41.110 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:41.127 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:40 smithi150 ceph-mon[29276]: 6.c scrub starts 2024-09-11T08:19:41.127 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:40 smithi150 ceph-mon[29276]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 592/327693 objects degraded (0.181%), 1 pg degraded) 2024-09-11T08:19:41.127 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:40 smithi150 ceph-mon[29276]: Cluster is now healthy 2024-09-11T08:19:41.127 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:40 smithi150 ceph-mon[29276]: osdmap e275: 8 total, 8 up, 8 in 2024-09-11T08:19:41.127 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:40 smithi150 ceph-mon[29276]: osdmap e276: 8 total, 8 up, 8 in 2024-09-11T08:19:41.127 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:40 smithi150 ceph-mon[39612]: 6.c scrub starts 2024-09-11T08:19:41.127 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:40 smithi150 ceph-mon[39612]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 592/327693 objects degraded (0.181%), 1 pg degraded) 2024-09-11T08:19:41.127 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:40 smithi150 ceph-mon[39612]: Cluster is now healthy 2024-09-11T08:19:41.128 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:40 smithi150 ceph-mon[39612]: osdmap e275: 8 total, 8 up, 8 in 2024-09-11T08:19:41.128 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:40 smithi150 ceph-mon[39612]: osdmap e276: 8 total, 8 up, 8 in 2024-09-11T08:19:41.132 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:41.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:41.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:41.234 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:41.256 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:41.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:41.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:41.358 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:41.379 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:41.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:41.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:41.481 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:19:41.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:41.505 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:41.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:19:41.606 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:19:41.629 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:41.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:41.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:19:41.731 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:19:41.770 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:41.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:41.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:19:41.782 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 90 4 115603 115599 80.2604 13.125 0.0364657 0.0124537 2024-09-11T08:19:41.794 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-11T08:19:41.794 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [], 2024-09-11T08:19:41.794 INFO:teuthology.orchestra.run.smithi150.stdout: "num_ops": 0 2024-09-11T08:19:41.794 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-11T08:19:41.872 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:41.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:41.894 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:41.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:41.995 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:19:42.009 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:41 smithi150 ceph-mon[29276]: 4.12 scrub starts 2024-09-11T08:19:42.009 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:41 smithi150 ceph-mon[29276]: 4.12 scrub ok 2024-09-11T08:19:42.009 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:41 smithi150 ceph-mon[29276]: pgmap v553: 148 pgs: 1 active+remapped+backfill_wait, 1 peering, 2 active+undersized+remapped, 1 active+recovering+undersized+remapped, 143 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.27k op/s; 7469/342936 objects misplaced (2.178%); 18 MiB/s, 280 objects/s recovering 2024-09-11T08:19:42.009 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:41 smithi150 ceph-mon[29276]: 6.1e scrub starts 2024-09-11T08:19:42.009 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:41 smithi150 ceph-mon[29276]: osdmap e277: 8 total, 8 up, 8 in 2024-09-11T08:19:42.010 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:41 smithi150 ceph-mon[39612]: 4.12 scrub starts 2024-09-11T08:19:42.010 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:41 smithi150 ceph-mon[39612]: 4.12 scrub ok 2024-09-11T08:19:42.010 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:41 smithi150 ceph-mon[39612]: pgmap v553: 148 pgs: 1 active+remapped+backfill_wait, 1 peering, 2 active+undersized+remapped, 1 active+recovering+undersized+remapped, 143 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.27k op/s; 7469/342936 objects misplaced (2.178%); 18 MiB/s, 280 objects/s recovering 2024-09-11T08:19:42.010 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:41 smithi150 ceph-mon[39612]: 6.1e scrub starts 2024-09-11T08:19:42.010 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:41 smithi150 ceph-mon[39612]: osdmap e277: 8 total, 8 up, 8 in 2024-09-11T08:19:42.016 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:42.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:42.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:19:42.104 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:41 smithi181 ceph-mon[32701]: 4.12 scrub starts 2024-09-11T08:19:42.104 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:41 smithi181 ceph-mon[32701]: 4.12 scrub ok 2024-09-11T08:19:42.104 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:41 smithi181 ceph-mon[32701]: pgmap v553: 148 pgs: 1 active+remapped+backfill_wait, 1 peering, 2 active+undersized+remapped, 1 active+recovering+undersized+remapped, 143 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.27k op/s; 7469/342936 objects misplaced (2.178%); 18 MiB/s, 280 objects/s recovering 2024-09-11T08:19:42.104 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:41 smithi181 ceph-mon[32701]: 6.1e scrub starts 2024-09-11T08:19:42.105 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:41 smithi181 ceph-mon[32701]: osdmap e277: 8 total, 8 up, 8 in 2024-09-11T08:19:42.205 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:19:42.225 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-11T08:19:42.226 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T08:19:42.226 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-11T08:19:42.226 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-11T08:19:42.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:42.228 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:42.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:19:42.329 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:19:42.352 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:42.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:42.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:19:42.453 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:42.475 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:42.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:42.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:42.577 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:42.599 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:42.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:42.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:42.700 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:42.724 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:42.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:42.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:42.782 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Total time run: 90.0192 2024-09-11T08:19:42.782 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Total writes made: 115603 2024-09-11T08:19:42.782 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Write size: 65536 2024-09-11T08:19:42.782 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Object size: 65536 2024-09-11T08:19:42.782 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Bandwidth (MB/sec): 80.2627 2024-09-11T08:19:42.783 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Stddev Bandwidth: 46.5275 2024-09-11T08:19:42.783 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Max bandwidth (MB/sec): 249.875 2024-09-11T08:19:42.783 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Min bandwidth (MB/sec): 0 2024-09-11T08:19:42.783 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Average IOPS: 1284 2024-09-11T08:19:42.783 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Stddev IOPS: 744.449 2024-09-11T08:19:42.783 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Max IOPS: 3998 2024-09-11T08:19:42.783 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Min IOPS: 0 2024-09-11T08:19:42.783 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Average Latency(s): 0.0124543 2024-09-11T08:19:42.783 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Stddev Latency(s): 0.0409674 2024-09-11T08:19:42.783 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Max latency(s): 2.31172 2024-09-11T08:19:42.783 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Min latency(s): 0.00117502 2024-09-11T08:19:42.784 INFO:teuthology.orchestra.run.smithi150.stderr:noscrub is set 2024-09-11T08:19:42.825 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:42.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:42.848 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:42.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:42.949 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:19:42.975 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:42.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:42.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:19:42.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:42 smithi150 ceph-mon[29276]: 9.1 scrub starts 2024-09-11T08:19:42.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:42 smithi150 ceph-mon[29276]: 3.1b scrub starts 2024-09-11T08:19:42.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:42 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/271145828' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:19:42.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:42 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:19:42.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:42 smithi150 ceph-mon[39612]: 9.1 scrub starts 2024-09-11T08:19:42.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:42 smithi150 ceph-mon[39612]: 3.1b scrub starts 2024-09-11T08:19:42.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:42 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/271145828' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:19:42.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:42 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:19:43.046 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-11T08:19:43.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:42 smithi181 ceph-mon[32701]: 9.1 scrub starts 2024-09-11T08:19:43.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:42 smithi181 ceph-mon[32701]: 3.1b scrub starts 2024-09-11T08:19:43.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:42 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/271145828' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:19:43.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:42 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:19:43.078 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:43.111 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:43.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:43.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:43.213 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:43.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:43.235 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:43.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:43.337 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:19:43.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:43.361 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:43.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:19:43.462 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:19:43.488 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:43.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:43.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:19:43.589 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:43.611 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:43.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:43.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:43.713 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:43.738 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:43.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:43.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:43.783 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:19:43.804 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Cleaning up (deleting benchmark objects) 2024-09-11T08:19:43.841 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:19:43.884 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:43.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:43.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:19:43.985 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:43 smithi150 ceph-mon[29276]: 3.1d scrub starts 2024-09-11T08:19:43.985 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:43 smithi150 ceph-mon[29276]: 3.1d scrub ok 2024-09-11T08:19:43.985 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:43 smithi150 ceph-mon[29276]: 5.0 scrub starts 2024-09-11T08:19:43.985 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:43 smithi150 ceph-mon[29276]: 5.0 scrub ok 2024-09-11T08:19:43.985 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:43 smithi150 ceph-mon[29276]: 5.11 scrub starts 2024-09-11T08:19:43.985 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:43 smithi150 ceph-mon[29276]: 5.11 scrub ok 2024-09-11T08:19:43.985 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:43 smithi150 ceph-mon[29276]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:19:43.985 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:43 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:19:43.986 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:43 smithi150 ceph-mon[29276]: osdmap e278: 8 total, 8 up, 8 in 2024-09-11T08:19:43.986 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:43 smithi150 ceph-mon[29276]: pgmap v556: 148 pgs: 2 peering, 2 active+undersized+remapped, 1 active+recovering+undersized+remapped, 143 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 7.5 MiB/s wr, 121 op/s; 378/344382 objects misplaced (0.110%); 12 MiB/s, 186 objects/s recovering 2024-09-11T08:19:43.986 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:43 smithi150 ceph-mon[39612]: 3.1d scrub starts 2024-09-11T08:19:43.986 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:43 smithi150 ceph-mon[39612]: 3.1d scrub ok 2024-09-11T08:19:43.986 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:43 smithi150 ceph-mon[39612]: 5.0 scrub starts 2024-09-11T08:19:43.986 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:43 smithi150 ceph-mon[39612]: 5.0 scrub ok 2024-09-11T08:19:43.986 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:43 smithi150 ceph-mon[39612]: 5.11 scrub starts 2024-09-11T08:19:43.986 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:43 smithi150 ceph-mon[39612]: 5.11 scrub ok 2024-09-11T08:19:43.986 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:43 smithi150 ceph-mon[39612]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:19:43.986 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:43 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:19:43.986 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:43 smithi150 ceph-mon[39612]: osdmap e278: 8 total, 8 up, 8 in 2024-09-11T08:19:43.987 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:43 smithi150 ceph-mon[39612]: pgmap v556: 148 pgs: 2 peering, 2 active+undersized+remapped, 1 active+recovering+undersized+remapped, 143 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 7.5 MiB/s wr, 121 op/s; 378/344382 objects misplaced (0.110%); 12 MiB/s, 186 objects/s recovering 2024-09-11T08:19:43.987 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:19:44.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:44.024 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:44.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:19:44.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:43 smithi181 ceph-mon[32701]: 3.1d scrub starts 2024-09-11T08:19:44.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:43 smithi181 ceph-mon[32701]: 3.1d scrub ok 2024-09-11T08:19:44.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:43 smithi181 ceph-mon[32701]: 5.0 scrub starts 2024-09-11T08:19:44.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:43 smithi181 ceph-mon[32701]: 5.0 scrub ok 2024-09-11T08:19:44.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:43 smithi181 ceph-mon[32701]: 5.11 scrub starts 2024-09-11T08:19:44.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:43 smithi181 ceph-mon[32701]: 5.11 scrub ok 2024-09-11T08:19:44.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:43 smithi181 ceph-mon[32701]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:19:44.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:43 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:19:44.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:43 smithi181 ceph-mon[32701]: osdmap e278: 8 total, 8 up, 8 in 2024-09-11T08:19:44.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:43 smithi181 ceph-mon[32701]: pgmap v556: 148 pgs: 2 peering, 2 active+undersized+remapped, 1 active+recovering+undersized+remapped, 143 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 7.5 MiB/s wr, 121 op/s; 378/344382 objects misplaced (0.110%); 12 MiB/s, 186 objects/s recovering 2024-09-11T08:19:44.125 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:44.158 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:44.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:44.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:44.260 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:44.297 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:44.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:44.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:44.375 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T08:19:44.400 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:19:44.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:44.465 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:44.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:19:44.566 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:44.595 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:19:44.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:44.600 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:44.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:44.701 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:44.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:44.740 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:44.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:44.841 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:19:44.876 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:44.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:44.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:19:44.977 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:19:44.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:44 smithi150 ceph-mon[29276]: osdmap e279: 8 total, 8 up, 8 in 2024-09-11T08:19:44.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:44 smithi150 ceph-mon[39612]: osdmap e279: 8 total, 8 up, 8 in 2024-09-11T08:19:45.014 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:45.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:45.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:19:45.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:44 smithi181 ceph-mon[32701]: osdmap e279: 8 total, 8 up, 8 in 2024-09-11T08:19:45.116 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:19:45.152 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:45.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:45.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:19:45.254 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:45.285 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:45.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:45.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:45.387 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:19:45.440 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:45.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:45.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:19:45.542 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:45.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:19:45 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:19:45] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:19:45.577 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:45.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:45.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:45.679 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:45.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:45.715 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:45.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:45.722 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T08:19:45.817 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:19:45.852 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:45.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:45.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:19:45.954 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:46.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:46.018 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:46.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:46.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:45 smithi181 ceph-mon[32701]: pgmap v558: 148 pgs: 1 peering, 147 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 21 MiB/s wr, 341 op/s; 16 MiB/s, 256 objects/s recovering 2024-09-11T08:19:46.119 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:19:46.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:46.154 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:46.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:19:46.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:45 smithi150 ceph-mon[29276]: pgmap v558: 148 pgs: 1 peering, 147 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 21 MiB/s wr, 341 op/s; 16 MiB/s, 256 objects/s recovering 2024-09-11T08:19:46.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:45 smithi150 ceph-mon[39612]: pgmap v558: 148 pgs: 1 peering, 147 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 21 MiB/s wr, 341 op/s; 16 MiB/s, 256 objects/s recovering 2024-09-11T08:19:46.255 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:19:46.298 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:46.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:46.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:19:46.400 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:46.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:46.440 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:46.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:46.541 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:46.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:46.576 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:46.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:46.677 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:46.715 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:46.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:46.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:46.817 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:19:46.833 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:19:46.835 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-11T08:19:46.835 INFO:teuthology.orchestra.run.smithi150.stdout: "epoch": 279, 2024-09-11T08:19:46.835 INFO:teuthology.orchestra.run.smithi150.stdout: "fsid": "c1bddff4-7013-11ef-bcea-c7b262605968", 2024-09-11T08:19:46.835 INFO:teuthology.orchestra.run.smithi150.stdout: "created": "2024-09-11T08:02:15.344618+0000", 2024-09-11T08:19:46.836 INFO:teuthology.orchestra.run.smithi150.stdout: "modified": "2024-09-11T08:19:43.779587+0000", 2024-09-11T08:19:46.836 INFO:teuthology.orchestra.run.smithi150.stdout: "last_up_change": "2024-09-11T08:14:23.745449+0000", 2024-09-11T08:19:46.836 INFO:teuthology.orchestra.run.smithi150.stdout: "last_in_change": "2024-09-11T08:19:28.121267+0000", 2024-09-11T08:19:46.836 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-11T08:19:46.836 INFO:teuthology.orchestra.run.smithi150.stdout: "flags_num": 5801984, 2024-09-11T08:19:46.836 INFO:teuthology.orchestra.run.smithi150.stdout: "flags_set": [ 2024-09-11T08:19:46.836 INFO:teuthology.orchestra.run.smithi150.stdout: "noscrub", 2024-09-11T08:19:46.836 INFO:teuthology.orchestra.run.smithi150.stdout: "pglog_hardlimit", 2024-09-11T08:19:46.836 INFO:teuthology.orchestra.run.smithi150.stdout: "purged_snapdirs", 2024-09-11T08:19:46.836 INFO:teuthology.orchestra.run.smithi150.stdout: "recovery_deletes", 2024-09-11T08:19:46.836 INFO:teuthology.orchestra.run.smithi150.stdout: "sortbitwise" 2024-09-11T08:19:46.836 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-09-11T08:19:46.836 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_version": 19, 2024-09-11T08:19:46.837 INFO:teuthology.orchestra.run.smithi150.stdout: "full_ratio": 0.94999998807907104, 2024-09-11T08:19:46.837 INFO:teuthology.orchestra.run.smithi150.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-11T08:19:46.837 INFO:teuthology.orchestra.run.smithi150.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-11T08:19:46.837 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_snapshot": "", 2024-09-11T08:19:46.837 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_max": 9, 2024-09-11T08:19:46.837 INFO:teuthology.orchestra.run.smithi150.stdout: "max_osd": 8, 2024-09-11T08:19:46.837 INFO:teuthology.orchestra.run.smithi150.stdout: "require_min_compat_client": "luminous", 2024-09-11T08:19:46.837 INFO:teuthology.orchestra.run.smithi150.stdout: "min_compat_client": "luminous", 2024-09-11T08:19:46.837 INFO:teuthology.orchestra.run.smithi150.stdout: "require_osd_release": "quincy", 2024-09-11T08:19:46.837 INFO:teuthology.orchestra.run.smithi150.stdout: "pools": [ 2024-09-11T08:19:46.837 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.837 INFO:teuthology.orchestra.run.smithi150.stdout: "pool": 1, 2024-09-11T08:19:46.837 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_name": ".mgr", 2024-09-11T08:19:46.838 INFO:teuthology.orchestra.run.smithi150.stdout: "create_time": "2024-09-11T08:05:21.044729+0000", 2024-09-11T08:19:46.838 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-09-11T08:19:46.838 INFO:teuthology.orchestra.run.smithi150.stdout: "flags_names": "hashpspool", 2024-09-11T08:19:46.838 INFO:teuthology.orchestra.run.smithi150.stdout: "type": 1, 2024-09-11T08:19:46.838 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 3, 2024-09-11T08:19:46.838 INFO:teuthology.orchestra.run.smithi150.stdout: "min_size": 2, 2024-09-11T08:19:46.838 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_rule": 0, 2024-09-11T08:19:46.838 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_count": 0, 2024-09-11T08:19:46.838 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_target": 0, 2024-09-11T08:19:46.838 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T08:19:46.838 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T08:19:46.838 INFO:teuthology.orchestra.run.smithi150.stdout: "object_hash": 2, 2024-09-11T08:19:46.838 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_autoscale_mode": "off", 2024-09-11T08:19:46.839 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num": 1, 2024-09-11T08:19:46.839 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num": 1, 2024-09-11T08:19:46.839 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num_target": 1, 2024-09-11T08:19:46.839 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_target": 1, 2024-09-11T08:19:46.839 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_pending": 1, 2024-09-11T08:19:46.839 INFO:teuthology.orchestra.run.smithi150.stdout: "last_pg_merge_meta": { 2024-09-11T08:19:46.839 INFO:teuthology.orchestra.run.smithi150.stdout: "source_pgid": "0.0", 2024-09-11T08:19:46.839 INFO:teuthology.orchestra.run.smithi150.stdout: "ready_epoch": 0, 2024-09-11T08:19:46.839 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_started": 0, 2024-09-11T08:19:46.839 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_clean": 0, 2024-09-11T08:19:46.839 INFO:teuthology.orchestra.run.smithi150.stdout: "source_version": "0'0", 2024-09-11T08:19:46.839 INFO:teuthology.orchestra.run.smithi150.stdout: "target_version": "0'0" 2024-09-11T08:19:46.839 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.840 INFO:teuthology.orchestra.run.smithi150.stdout: "last_change": "255", 2024-09-11T08:19:46.840 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend": "0", 2024-09-11T08:19:46.840 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T08:19:46.840 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T08:19:46.840 INFO:teuthology.orchestra.run.smithi150.stdout: "auid": 0, 2024-09-11T08:19:46.840 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_mode": "selfmanaged", 2024-09-11T08:19:46.840 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_seq": 0, 2024-09-11T08:19:46.840 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_epoch": 0, 2024-09-11T08:19:46.840 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_snaps": [], 2024-09-11T08:19:46.840 INFO:teuthology.orchestra.run.smithi150.stdout: "removed_snaps": "[]", 2024-09-11T08:19:46.840 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_bytes": 0, 2024-09-11T08:19:46.841 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_objects": 0, 2024-09-11T08:19:46.841 INFO:teuthology.orchestra.run.smithi150.stdout: "tiers": [], 2024-09-11T08:19:46.841 INFO:teuthology.orchestra.run.smithi150.stdout: "tier_of": -1, 2024-09-11T08:19:46.841 INFO:teuthology.orchestra.run.smithi150.stdout: "read_tier": -1, 2024-09-11T08:19:46.841 INFO:teuthology.orchestra.run.smithi150.stdout: "write_tier": -1, 2024-09-11T08:19:46.841 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_mode": "none", 2024-09-11T08:19:46.841 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_bytes": 0, 2024-09-11T08:19:46.841 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_objects": 0, 2024-09-11T08:19:46.841 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T08:19:46.841 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T08:19:46.841 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T08:19:46.841 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_flush_age": 0, 2024-09-11T08:19:46.841 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_evict_age": 0, 2024-09-11T08:19:46.841 INFO:teuthology.orchestra.run.smithi150.stdout: "erasure_code_profile": "", 2024-09-11T08:19:46.842 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_params": { 2024-09-11T08:19:46.842 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "none" 2024-09-11T08:19:46.842 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.842 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_period": 0, 2024-09-11T08:19:46.842 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_count": 0, 2024-09-11T08:19:46.842 INFO:teuthology.orchestra.run.smithi150.stdout: "use_gmt_hitset": true, 2024-09-11T08:19:46.842 INFO:teuthology.orchestra.run.smithi150.stdout: "min_read_recency_for_promote": 0, 2024-09-11T08:19:46.842 INFO:teuthology.orchestra.run.smithi150.stdout: "min_write_recency_for_promote": 0, 2024-09-11T08:19:46.842 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T08:19:46.842 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_search_last_n": 0, 2024-09-11T08:19:46.842 INFO:teuthology.orchestra.run.smithi150.stdout: "grade_table": [], 2024-09-11T08:19:46.842 INFO:teuthology.orchestra.run.smithi150.stdout: "stripe_width": 0, 2024-09-11T08:19:46.842 INFO:teuthology.orchestra.run.smithi150.stdout: "expected_num_objects": 0, 2024-09-11T08:19:46.843 INFO:teuthology.orchestra.run.smithi150.stdout: "fast_read": false, 2024-09-11T08:19:46.843 INFO:teuthology.orchestra.run.smithi150.stdout: "options": { 2024-09-11T08:19:46.843 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_max": 32, 2024-09-11T08:19:46.843 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_min": 1 2024-09-11T08:19:46.843 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.843 INFO:teuthology.orchestra.run.smithi150.stdout: "application_metadata": { 2024-09-11T08:19:46.843 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": {} 2024-09-11T08:19:46.843 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:19:46.843 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.843 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.843 INFO:teuthology.orchestra.run.smithi150.stdout: "pool": 2, 2024-09-11T08:19:46.843 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_name": "datapool", 2024-09-11T08:19:46.843 INFO:teuthology.orchestra.run.smithi150.stdout: "create_time": "2024-09-11T08:07:28.786572+0000", 2024-09-11T08:19:46.844 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 8193, 2024-09-11T08:19:46.844 INFO:teuthology.orchestra.run.smithi150.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-11T08:19:46.844 INFO:teuthology.orchestra.run.smithi150.stdout: "type": 1, 2024-09-11T08:19:46.844 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 3, 2024-09-11T08:19:46.844 INFO:teuthology.orchestra.run.smithi150.stdout: "min_size": 2, 2024-09-11T08:19:46.844 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_rule": 0, 2024-09-11T08:19:46.844 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_count": 0, 2024-09-11T08:19:46.844 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_target": 0, 2024-09-11T08:19:46.844 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T08:19:46.844 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T08:19:46.844 INFO:teuthology.orchestra.run.smithi150.stdout: "object_hash": 2, 2024-09-11T08:19:46.844 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_autoscale_mode": "off", 2024-09-11T08:19:46.844 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num": 3, 2024-09-11T08:19:46.844 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num": 3, 2024-09-11T08:19:46.844 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num_target": 3, 2024-09-11T08:19:46.845 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_target": 3, 2024-09-11T08:19:46.845 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_pending": 3, 2024-09-11T08:19:46.845 INFO:teuthology.orchestra.run.smithi150.stdout: "last_pg_merge_meta": { 2024-09-11T08:19:46.845 INFO:teuthology.orchestra.run.smithi150.stdout: "source_pgid": "0.0", 2024-09-11T08:19:46.845 INFO:teuthology.orchestra.run.smithi150.stdout: "ready_epoch": 0, 2024-09-11T08:19:46.845 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_started": 0, 2024-09-11T08:19:46.845 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_clean": 0, 2024-09-11T08:19:46.845 INFO:teuthology.orchestra.run.smithi150.stdout: "source_version": "0'0", 2024-09-11T08:19:46.845 INFO:teuthology.orchestra.run.smithi150.stdout: "target_version": "0'0" 2024-09-11T08:19:46.845 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.845 INFO:teuthology.orchestra.run.smithi150.stdout: "last_change": "63", 2024-09-11T08:19:46.845 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend": "0", 2024-09-11T08:19:46.845 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T08:19:46.845 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T08:19:46.846 INFO:teuthology.orchestra.run.smithi150.stdout: "auid": 0, 2024-09-11T08:19:46.846 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_mode": "selfmanaged", 2024-09-11T08:19:46.846 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_seq": 3, 2024-09-11T08:19:46.846 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_epoch": 63, 2024-09-11T08:19:46.846 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_snaps": [], 2024-09-11T08:19:46.846 INFO:teuthology.orchestra.run.smithi150.stdout: "removed_snaps": "[]", 2024-09-11T08:19:46.846 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_bytes": 0, 2024-09-11T08:19:46.846 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_objects": 0, 2024-09-11T08:19:46.846 INFO:teuthology.orchestra.run.smithi150.stdout: "tiers": [], 2024-09-11T08:19:46.846 INFO:teuthology.orchestra.run.smithi150.stdout: "tier_of": -1, 2024-09-11T08:19:46.846 INFO:teuthology.orchestra.run.smithi150.stdout: "read_tier": -1, 2024-09-11T08:19:46.846 INFO:teuthology.orchestra.run.smithi150.stdout: "write_tier": -1, 2024-09-11T08:19:46.846 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_mode": "none", 2024-09-11T08:19:46.846 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_bytes": 0, 2024-09-11T08:19:46.847 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_objects": 0, 2024-09-11T08:19:46.847 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T08:19:46.847 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T08:19:46.847 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T08:19:46.847 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_flush_age": 0, 2024-09-11T08:19:46.847 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_evict_age": 0, 2024-09-11T08:19:46.847 INFO:teuthology.orchestra.run.smithi150.stdout: "erasure_code_profile": "", 2024-09-11T08:19:46.847 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_params": { 2024-09-11T08:19:46.847 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "none" 2024-09-11T08:19:46.847 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.847 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_period": 0, 2024-09-11T08:19:46.847 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_count": 0, 2024-09-11T08:19:46.847 INFO:teuthology.orchestra.run.smithi150.stdout: "use_gmt_hitset": true, 2024-09-11T08:19:46.848 INFO:teuthology.orchestra.run.smithi150.stdout: "min_read_recency_for_promote": 0, 2024-09-11T08:19:46.848 INFO:teuthology.orchestra.run.smithi150.stdout: "min_write_recency_for_promote": 0, 2024-09-11T08:19:46.848 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T08:19:46.848 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_search_last_n": 0, 2024-09-11T08:19:46.848 INFO:teuthology.orchestra.run.smithi150.stdout: "grade_table": [], 2024-09-11T08:19:46.848 INFO:teuthology.orchestra.run.smithi150.stdout: "stripe_width": 0, 2024-09-11T08:19:46.848 INFO:teuthology.orchestra.run.smithi150.stdout: "expected_num_objects": 0, 2024-09-11T08:19:46.848 INFO:teuthology.orchestra.run.smithi150.stdout: "fast_read": false, 2024-09-11T08:19:46.848 INFO:teuthology.orchestra.run.smithi150.stdout: "options": {}, 2024-09-11T08:19:46.848 INFO:teuthology.orchestra.run.smithi150.stdout: "application_metadata": { 2024-09-11T08:19:46.848 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd": {} 2024-09-11T08:19:46.848 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:19:46.848 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.848 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.849 INFO:teuthology.orchestra.run.smithi150.stdout: "pool": 3, 2024-09-11T08:19:46.849 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_name": ".rgw.root", 2024-09-11T08:19:46.849 INFO:teuthology.orchestra.run.smithi150.stdout: "create_time": "2024-09-11T08:07:29.021286+0000", 2024-09-11T08:19:46.849 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-09-11T08:19:46.849 INFO:teuthology.orchestra.run.smithi150.stdout: "flags_names": "hashpspool", 2024-09-11T08:19:46.849 INFO:teuthology.orchestra.run.smithi150.stdout: "type": 1, 2024-09-11T08:19:46.849 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 3, 2024-09-11T08:19:46.849 INFO:teuthology.orchestra.run.smithi150.stdout: "min_size": 2, 2024-09-11T08:19:46.849 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_rule": 0, 2024-09-11T08:19:46.849 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_count": 0, 2024-09-11T08:19:46.849 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_target": 0, 2024-09-11T08:19:46.849 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T08:19:46.849 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T08:19:46.850 INFO:teuthology.orchestra.run.smithi150.stdout: "object_hash": 2, 2024-09-11T08:19:46.850 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_autoscale_mode": "off", 2024-09-11T08:19:46.850 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num": 32, 2024-09-11T08:19:46.850 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num": 32, 2024-09-11T08:19:46.850 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num_target": 32, 2024-09-11T08:19:46.850 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_target": 32, 2024-09-11T08:19:46.850 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_pending": 32, 2024-09-11T08:19:46.850 INFO:teuthology.orchestra.run.smithi150.stdout: "last_pg_merge_meta": { 2024-09-11T08:19:46.850 INFO:teuthology.orchestra.run.smithi150.stdout: "source_pgid": "0.0", 2024-09-11T08:19:46.850 INFO:teuthology.orchestra.run.smithi150.stdout: "ready_epoch": 0, 2024-09-11T08:19:46.850 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_started": 0, 2024-09-11T08:19:46.850 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_clean": 0, 2024-09-11T08:19:46.850 INFO:teuthology.orchestra.run.smithi150.stdout: "source_version": "0'0", 2024-09-11T08:19:46.850 INFO:teuthology.orchestra.run.smithi150.stdout: "target_version": "0'0" 2024-09-11T08:19:46.851 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.851 INFO:teuthology.orchestra.run.smithi150.stdout: "last_change": "60", 2024-09-11T08:19:46.851 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend": "0", 2024-09-11T08:19:46.851 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T08:19:46.851 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T08:19:46.851 INFO:teuthology.orchestra.run.smithi150.stdout: "auid": 0, 2024-09-11T08:19:46.851 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_mode": "selfmanaged", 2024-09-11T08:19:46.851 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_seq": 0, 2024-09-11T08:19:46.851 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_epoch": 0, 2024-09-11T08:19:46.851 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_snaps": [], 2024-09-11T08:19:46.851 INFO:teuthology.orchestra.run.smithi150.stdout: "removed_snaps": "[]", 2024-09-11T08:19:46.851 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_bytes": 0, 2024-09-11T08:19:46.851 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_objects": 0, 2024-09-11T08:19:46.851 INFO:teuthology.orchestra.run.smithi150.stdout: "tiers": [], 2024-09-11T08:19:46.852 INFO:teuthology.orchestra.run.smithi150.stdout: "tier_of": -1, 2024-09-11T08:19:46.852 INFO:teuthology.orchestra.run.smithi150.stdout: "read_tier": -1, 2024-09-11T08:19:46.852 INFO:teuthology.orchestra.run.smithi150.stdout: "write_tier": -1, 2024-09-11T08:19:46.852 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_mode": "none", 2024-09-11T08:19:46.852 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_bytes": 0, 2024-09-11T08:19:46.852 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_objects": 0, 2024-09-11T08:19:46.852 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T08:19:46.852 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T08:19:46.852 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T08:19:46.852 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_flush_age": 0, 2024-09-11T08:19:46.852 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_evict_age": 0, 2024-09-11T08:19:46.852 INFO:teuthology.orchestra.run.smithi150.stdout: "erasure_code_profile": "", 2024-09-11T08:19:46.853 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_params": { 2024-09-11T08:19:46.853 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "none" 2024-09-11T08:19:46.853 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.853 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_period": 0, 2024-09-11T08:19:46.853 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_count": 0, 2024-09-11T08:19:46.853 INFO:teuthology.orchestra.run.smithi150.stdout: "use_gmt_hitset": true, 2024-09-11T08:19:46.853 INFO:teuthology.orchestra.run.smithi150.stdout: "min_read_recency_for_promote": 0, 2024-09-11T08:19:46.853 INFO:teuthology.orchestra.run.smithi150.stdout: "min_write_recency_for_promote": 0, 2024-09-11T08:19:46.853 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T08:19:46.853 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_search_last_n": 0, 2024-09-11T08:19:46.853 INFO:teuthology.orchestra.run.smithi150.stdout: "grade_table": [], 2024-09-11T08:19:46.853 INFO:teuthology.orchestra.run.smithi150.stdout: "stripe_width": 0, 2024-09-11T08:19:46.853 INFO:teuthology.orchestra.run.smithi150.stdout: "expected_num_objects": 0, 2024-09-11T08:19:46.854 INFO:teuthology.orchestra.run.smithi150.stdout: "fast_read": false, 2024-09-11T08:19:46.854 INFO:teuthology.orchestra.run.smithi150.stdout: "options": {}, 2024-09-11T08:19:46.854 INFO:teuthology.orchestra.run.smithi150.stdout: "application_metadata": { 2024-09-11T08:19:46.854 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw": {} 2024-09-11T08:19:46.854 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:19:46.854 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.854 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.854 INFO:teuthology.orchestra.run.smithi150.stdout: "pool": 4, 2024-09-11T08:19:46.854 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_name": "default.rgw.log", 2024-09-11T08:19:46.854 INFO:teuthology.orchestra.run.smithi150.stdout: "create_time": "2024-09-11T08:07:30.334181+0000", 2024-09-11T08:19:46.854 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-09-11T08:19:46.854 INFO:teuthology.orchestra.run.smithi150.stdout: "flags_names": "hashpspool", 2024-09-11T08:19:46.854 INFO:teuthology.orchestra.run.smithi150.stdout: "type": 1, 2024-09-11T08:19:46.855 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 3, 2024-09-11T08:19:46.855 INFO:teuthology.orchestra.run.smithi150.stdout: "min_size": 2, 2024-09-11T08:19:46.855 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_rule": 0, 2024-09-11T08:19:46.855 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_count": 0, 2024-09-11T08:19:46.855 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_target": 0, 2024-09-11T08:19:46.855 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T08:19:46.855 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T08:19:46.855 INFO:teuthology.orchestra.run.smithi150.stdout: "object_hash": 2, 2024-09-11T08:19:46.855 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_autoscale_mode": "off", 2024-09-11T08:19:46.855 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num": 32, 2024-09-11T08:19:46.855 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num": 32, 2024-09-11T08:19:46.855 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num_target": 32, 2024-09-11T08:19:46.855 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_target": 32, 2024-09-11T08:19:46.856 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_pending": 32, 2024-09-11T08:19:46.856 INFO:teuthology.orchestra.run.smithi150.stdout: "last_pg_merge_meta": { 2024-09-11T08:19:46.856 INFO:teuthology.orchestra.run.smithi150.stdout: "source_pgid": "0.0", 2024-09-11T08:19:46.856 INFO:teuthology.orchestra.run.smithi150.stdout: "ready_epoch": 0, 2024-09-11T08:19:46.856 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_started": 0, 2024-09-11T08:19:46.856 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_clean": 0, 2024-09-11T08:19:46.856 INFO:teuthology.orchestra.run.smithi150.stdout: "source_version": "0'0", 2024-09-11T08:19:46.856 INFO:teuthology.orchestra.run.smithi150.stdout: "target_version": "0'0" 2024-09-11T08:19:46.856 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.856 INFO:teuthology.orchestra.run.smithi150.stdout: "last_change": "62", 2024-09-11T08:19:46.856 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend": "0", 2024-09-11T08:19:46.856 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T08:19:46.856 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T08:19:46.856 INFO:teuthology.orchestra.run.smithi150.stdout: "auid": 0, 2024-09-11T08:19:46.857 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_mode": "selfmanaged", 2024-09-11T08:19:46.857 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_seq": 0, 2024-09-11T08:19:46.857 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_epoch": 0, 2024-09-11T08:19:46.857 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_snaps": [], 2024-09-11T08:19:46.857 INFO:teuthology.orchestra.run.smithi150.stdout: "removed_snaps": "[]", 2024-09-11T08:19:46.857 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_bytes": 0, 2024-09-11T08:19:46.857 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_objects": 0, 2024-09-11T08:19:46.857 INFO:teuthology.orchestra.run.smithi150.stdout: "tiers": [], 2024-09-11T08:19:46.857 INFO:teuthology.orchestra.run.smithi150.stdout: "tier_of": -1, 2024-09-11T08:19:46.857 INFO:teuthology.orchestra.run.smithi150.stdout: "read_tier": -1, 2024-09-11T08:19:46.857 INFO:teuthology.orchestra.run.smithi150.stdout: "write_tier": -1, 2024-09-11T08:19:46.857 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_mode": "none", 2024-09-11T08:19:46.857 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_bytes": 0, 2024-09-11T08:19:46.858 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_objects": 0, 2024-09-11T08:19:46.858 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T08:19:46.858 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T08:19:46.858 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T08:19:46.858 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_flush_age": 0, 2024-09-11T08:19:46.858 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_evict_age": 0, 2024-09-11T08:19:46.858 INFO:teuthology.orchestra.run.smithi150.stdout: "erasure_code_profile": "", 2024-09-11T08:19:46.858 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_params": { 2024-09-11T08:19:46.858 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "none" 2024-09-11T08:19:46.858 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.858 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_period": 0, 2024-09-11T08:19:46.858 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_count": 0, 2024-09-11T08:19:46.858 INFO:teuthology.orchestra.run.smithi150.stdout: "use_gmt_hitset": true, 2024-09-11T08:19:46.858 INFO:teuthology.orchestra.run.smithi150.stdout: "min_read_recency_for_promote": 0, 2024-09-11T08:19:46.859 INFO:teuthology.orchestra.run.smithi150.stdout: "min_write_recency_for_promote": 0, 2024-09-11T08:19:46.859 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T08:19:46.859 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_search_last_n": 0, 2024-09-11T08:19:46.859 INFO:teuthology.orchestra.run.smithi150.stdout: "grade_table": [], 2024-09-11T08:19:46.859 INFO:teuthology.orchestra.run.smithi150.stdout: "stripe_width": 0, 2024-09-11T08:19:46.859 INFO:teuthology.orchestra.run.smithi150.stdout: "expected_num_objects": 0, 2024-09-11T08:19:46.859 INFO:teuthology.orchestra.run.smithi150.stdout: "fast_read": false, 2024-09-11T08:19:46.859 INFO:teuthology.orchestra.run.smithi150.stdout: "options": {}, 2024-09-11T08:19:46.859 INFO:teuthology.orchestra.run.smithi150.stdout: "application_metadata": { 2024-09-11T08:19:46.859 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw": {} 2024-09-11T08:19:46.859 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:19:46.859 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.860 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.860 INFO:teuthology.orchestra.run.smithi150.stdout: "pool": 5, 2024-09-11T08:19:46.860 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_name": "default.rgw.control", 2024-09-11T08:19:46.860 INFO:teuthology.orchestra.run.smithi150.stdout: "create_time": "2024-09-11T08:07:32.289626+0000", 2024-09-11T08:19:46.860 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-09-11T08:19:46.860 INFO:teuthology.orchestra.run.smithi150.stdout: "flags_names": "hashpspool", 2024-09-11T08:19:46.860 INFO:teuthology.orchestra.run.smithi150.stdout: "type": 1, 2024-09-11T08:19:46.860 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 3, 2024-09-11T08:19:46.860 INFO:teuthology.orchestra.run.smithi150.stdout: "min_size": 2, 2024-09-11T08:19:46.860 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_rule": 0, 2024-09-11T08:19:46.860 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_count": 0, 2024-09-11T08:19:46.860 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_target": 0, 2024-09-11T08:19:46.860 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T08:19:46.861 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T08:19:46.861 INFO:teuthology.orchestra.run.smithi150.stdout: "object_hash": 2, 2024-09-11T08:19:46.861 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_autoscale_mode": "off", 2024-09-11T08:19:46.861 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num": 32, 2024-09-11T08:19:46.861 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num": 32, 2024-09-11T08:19:46.861 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num_target": 32, 2024-09-11T08:19:46.861 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_target": 32, 2024-09-11T08:19:46.861 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_pending": 32, 2024-09-11T08:19:46.861 INFO:teuthology.orchestra.run.smithi150.stdout: "last_pg_merge_meta": { 2024-09-11T08:19:46.861 INFO:teuthology.orchestra.run.smithi150.stdout: "source_pgid": "0.0", 2024-09-11T08:19:46.861 INFO:teuthology.orchestra.run.smithi150.stdout: "ready_epoch": 0, 2024-09-11T08:19:46.861 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_started": 0, 2024-09-11T08:19:46.861 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_clean": 0, 2024-09-11T08:19:46.861 INFO:teuthology.orchestra.run.smithi150.stdout: "source_version": "0'0", 2024-09-11T08:19:46.862 INFO:teuthology.orchestra.run.smithi150.stdout: "target_version": "0'0" 2024-09-11T08:19:46.862 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.862 INFO:teuthology.orchestra.run.smithi150.stdout: "last_change": "64", 2024-09-11T08:19:46.862 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend": "0", 2024-09-11T08:19:46.862 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T08:19:46.862 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T08:19:46.862 INFO:teuthology.orchestra.run.smithi150.stdout: "auid": 0, 2024-09-11T08:19:46.862 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_mode": "selfmanaged", 2024-09-11T08:19:46.862 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_seq": 0, 2024-09-11T08:19:46.863 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_epoch": 0, 2024-09-11T08:19:46.863 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_snaps": [], 2024-09-11T08:19:46.863 INFO:teuthology.orchestra.run.smithi150.stdout: "removed_snaps": "[]", 2024-09-11T08:19:46.863 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_bytes": 0, 2024-09-11T08:19:46.863 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_objects": 0, 2024-09-11T08:19:46.863 INFO:teuthology.orchestra.run.smithi150.stdout: "tiers": [], 2024-09-11T08:19:46.863 INFO:teuthology.orchestra.run.smithi150.stdout: "tier_of": -1, 2024-09-11T08:19:46.863 INFO:teuthology.orchestra.run.smithi150.stdout: "read_tier": -1, 2024-09-11T08:19:46.863 INFO:teuthology.orchestra.run.smithi150.stdout: "write_tier": -1, 2024-09-11T08:19:46.863 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_mode": "none", 2024-09-11T08:19:46.863 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_bytes": 0, 2024-09-11T08:19:46.863 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_objects": 0, 2024-09-11T08:19:46.863 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T08:19:46.863 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T08:19:46.864 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T08:19:46.864 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_flush_age": 0, 2024-09-11T08:19:46.864 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_evict_age": 0, 2024-09-11T08:19:46.864 INFO:teuthology.orchestra.run.smithi150.stdout: "erasure_code_profile": "", 2024-09-11T08:19:46.864 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_params": { 2024-09-11T08:19:46.864 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "none" 2024-09-11T08:19:46.864 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.864 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_period": 0, 2024-09-11T08:19:46.864 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_count": 0, 2024-09-11T08:19:46.864 INFO:teuthology.orchestra.run.smithi150.stdout: "use_gmt_hitset": true, 2024-09-11T08:19:46.864 INFO:teuthology.orchestra.run.smithi150.stdout: "min_read_recency_for_promote": 0, 2024-09-11T08:19:46.864 INFO:teuthology.orchestra.run.smithi150.stdout: "min_write_recency_for_promote": 0, 2024-09-11T08:19:46.864 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T08:19:46.864 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_search_last_n": 0, 2024-09-11T08:19:46.865 INFO:teuthology.orchestra.run.smithi150.stdout: "grade_table": [], 2024-09-11T08:19:46.865 INFO:teuthology.orchestra.run.smithi150.stdout: "stripe_width": 0, 2024-09-11T08:19:46.865 INFO:teuthology.orchestra.run.smithi150.stdout: "expected_num_objects": 0, 2024-09-11T08:19:46.865 INFO:teuthology.orchestra.run.smithi150.stdout: "fast_read": false, 2024-09-11T08:19:46.865 INFO:teuthology.orchestra.run.smithi150.stdout: "options": {}, 2024-09-11T08:19:46.865 INFO:teuthology.orchestra.run.smithi150.stdout: "application_metadata": { 2024-09-11T08:19:46.865 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw": {} 2024-09-11T08:19:46.865 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:19:46.865 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.865 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.865 INFO:teuthology.orchestra.run.smithi150.stdout: "pool": 6, 2024-09-11T08:19:46.865 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_name": "default.rgw.meta", 2024-09-11T08:19:46.865 INFO:teuthology.orchestra.run.smithi150.stdout: "create_time": "2024-09-11T08:07:34.351861+0000", 2024-09-11T08:19:46.866 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-09-11T08:19:46.866 INFO:teuthology.orchestra.run.smithi150.stdout: "flags_names": "hashpspool", 2024-09-11T08:19:46.866 INFO:teuthology.orchestra.run.smithi150.stdout: "type": 1, 2024-09-11T08:19:46.866 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 3, 2024-09-11T08:19:46.866 INFO:teuthology.orchestra.run.smithi150.stdout: "min_size": 2, 2024-09-11T08:19:46.866 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_rule": 0, 2024-09-11T08:19:46.866 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_count": 0, 2024-09-11T08:19:46.866 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_target": 0, 2024-09-11T08:19:46.866 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T08:19:46.866 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T08:19:46.866 INFO:teuthology.orchestra.run.smithi150.stdout: "object_hash": 2, 2024-09-11T08:19:46.866 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_autoscale_mode": "off", 2024-09-11T08:19:46.866 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num": 32, 2024-09-11T08:19:46.867 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num": 32, 2024-09-11T08:19:46.867 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num_target": 32, 2024-09-11T08:19:46.867 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_target": 32, 2024-09-11T08:19:46.867 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_pending": 32, 2024-09-11T08:19:46.867 INFO:teuthology.orchestra.run.smithi150.stdout: "last_pg_merge_meta": { 2024-09-11T08:19:46.867 INFO:teuthology.orchestra.run.smithi150.stdout: "source_pgid": "0.0", 2024-09-11T08:19:46.867 INFO:teuthology.orchestra.run.smithi150.stdout: "ready_epoch": 0, 2024-09-11T08:19:46.867 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_started": 0, 2024-09-11T08:19:46.867 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_clean": 0, 2024-09-11T08:19:46.867 INFO:teuthology.orchestra.run.smithi150.stdout: "source_version": "0'0", 2024-09-11T08:19:46.867 INFO:teuthology.orchestra.run.smithi150.stdout: "target_version": "0'0" 2024-09-11T08:19:46.867 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.867 INFO:teuthology.orchestra.run.smithi150.stdout: "last_change": "66", 2024-09-11T08:19:46.867 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend": "0", 2024-09-11T08:19:46.868 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T08:19:46.868 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T08:19:46.868 INFO:teuthology.orchestra.run.smithi150.stdout: "auid": 0, 2024-09-11T08:19:46.868 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_mode": "selfmanaged", 2024-09-11T08:19:46.868 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_seq": 0, 2024-09-11T08:19:46.868 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_epoch": 0, 2024-09-11T08:19:46.868 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_snaps": [], 2024-09-11T08:19:46.868 INFO:teuthology.orchestra.run.smithi150.stdout: "removed_snaps": "[]", 2024-09-11T08:19:46.868 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_bytes": 0, 2024-09-11T08:19:46.868 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_objects": 0, 2024-09-11T08:19:46.868 INFO:teuthology.orchestra.run.smithi150.stdout: "tiers": [], 2024-09-11T08:19:46.868 INFO:teuthology.orchestra.run.smithi150.stdout: "tier_of": -1, 2024-09-11T08:19:46.868 INFO:teuthology.orchestra.run.smithi150.stdout: "read_tier": -1, 2024-09-11T08:19:46.869 INFO:teuthology.orchestra.run.smithi150.stdout: "write_tier": -1, 2024-09-11T08:19:46.869 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_mode": "none", 2024-09-11T08:19:46.869 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_bytes": 0, 2024-09-11T08:19:46.869 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_objects": 0, 2024-09-11T08:19:46.869 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T08:19:46.869 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T08:19:46.869 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T08:19:46.869 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_flush_age": 0, 2024-09-11T08:19:46.869 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_evict_age": 0, 2024-09-11T08:19:46.869 INFO:teuthology.orchestra.run.smithi150.stdout: "erasure_code_profile": "", 2024-09-11T08:19:46.869 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_params": { 2024-09-11T08:19:46.869 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "none" 2024-09-11T08:19:46.869 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.870 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_period": 0, 2024-09-11T08:19:46.870 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_count": 0, 2024-09-11T08:19:46.870 INFO:teuthology.orchestra.run.smithi150.stdout: "use_gmt_hitset": true, 2024-09-11T08:19:46.870 INFO:teuthology.orchestra.run.smithi150.stdout: "min_read_recency_for_promote": 0, 2024-09-11T08:19:46.870 INFO:teuthology.orchestra.run.smithi150.stdout: "min_write_recency_for_promote": 0, 2024-09-11T08:19:46.870 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T08:19:46.870 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_search_last_n": 0, 2024-09-11T08:19:46.870 INFO:teuthology.orchestra.run.smithi150.stdout: "grade_table": [], 2024-09-11T08:19:46.870 INFO:teuthology.orchestra.run.smithi150.stdout: "stripe_width": 0, 2024-09-11T08:19:46.870 INFO:teuthology.orchestra.run.smithi150.stdout: "expected_num_objects": 0, 2024-09-11T08:19:46.870 INFO:teuthology.orchestra.run.smithi150.stdout: "fast_read": false, 2024-09-11T08:19:46.870 INFO:teuthology.orchestra.run.smithi150.stdout: "options": { 2024-09-11T08:19:46.870 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_autoscale_bias": 4 2024-09-11T08:19:46.871 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.871 INFO:teuthology.orchestra.run.smithi150.stdout: "application_metadata": { 2024-09-11T08:19:46.871 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw": {} 2024-09-11T08:19:46.871 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:19:46.871 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.871 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.872 INFO:teuthology.orchestra.run.smithi150.stdout: "pool": 9, 2024-09-11T08:19:46.872 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_name": "unique_pool_2", 2024-09-11T08:19:46.872 INFO:teuthology.orchestra.run.smithi150.stdout: "create_time": "2024-09-11T08:17:59.761532+0000", 2024-09-11T08:19:46.872 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-09-11T08:19:46.872 INFO:teuthology.orchestra.run.smithi150.stdout: "flags_names": "hashpspool", 2024-09-11T08:19:46.872 INFO:teuthology.orchestra.run.smithi150.stdout: "type": 1, 2024-09-11T08:19:46.872 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 3, 2024-09-11T08:19:46.872 INFO:teuthology.orchestra.run.smithi150.stdout: "min_size": 2, 2024-09-11T08:19:46.872 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_rule": 0, 2024-09-11T08:19:46.872 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_count": 0, 2024-09-11T08:19:46.872 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_target": 0, 2024-09-11T08:19:46.872 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T08:19:46.872 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T08:19:46.873 INFO:teuthology.orchestra.run.smithi150.stdout: "object_hash": 2, 2024-09-11T08:19:46.873 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_autoscale_mode": "off", 2024-09-11T08:19:46.873 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num": 16, 2024-09-11T08:19:46.873 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num": 16, 2024-09-11T08:19:46.873 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num_target": 16, 2024-09-11T08:19:46.873 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_target": 16, 2024-09-11T08:19:46.873 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_pending": 16, 2024-09-11T08:19:46.873 INFO:teuthology.orchestra.run.smithi150.stdout: "last_pg_merge_meta": { 2024-09-11T08:19:46.873 INFO:teuthology.orchestra.run.smithi150.stdout: "source_pgid": "0.0", 2024-09-11T08:19:46.873 INFO:teuthology.orchestra.run.smithi150.stdout: "ready_epoch": 0, 2024-09-11T08:19:46.873 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_started": 0, 2024-09-11T08:19:46.873 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_clean": 0, 2024-09-11T08:19:46.873 INFO:teuthology.orchestra.run.smithi150.stdout: "source_version": "0'0", 2024-09-11T08:19:46.874 INFO:teuthology.orchestra.run.smithi150.stdout: "target_version": "0'0" 2024-09-11T08:19:46.874 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.874 INFO:teuthology.orchestra.run.smithi150.stdout: "last_change": "251", 2024-09-11T08:19:46.874 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend": "0", 2024-09-11T08:19:46.874 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T08:19:46.874 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T08:19:46.874 INFO:teuthology.orchestra.run.smithi150.stdout: "auid": 0, 2024-09-11T08:19:46.874 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_mode": "selfmanaged", 2024-09-11T08:19:46.874 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_seq": 0, 2024-09-11T08:19:46.874 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_epoch": 0, 2024-09-11T08:19:46.874 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_snaps": [], 2024-09-11T08:19:46.874 INFO:teuthology.orchestra.run.smithi150.stdout: "removed_snaps": "[]", 2024-09-11T08:19:46.874 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_bytes": 0, 2024-09-11T08:19:46.874 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_objects": 0, 2024-09-11T08:19:46.875 INFO:teuthology.orchestra.run.smithi150.stdout: "tiers": [], 2024-09-11T08:19:46.875 INFO:teuthology.orchestra.run.smithi150.stdout: "tier_of": -1, 2024-09-11T08:19:46.875 INFO:teuthology.orchestra.run.smithi150.stdout: "read_tier": -1, 2024-09-11T08:19:46.875 INFO:teuthology.orchestra.run.smithi150.stdout: "write_tier": -1, 2024-09-11T08:19:46.875 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_mode": "none", 2024-09-11T08:19:46.875 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_bytes": 0, 2024-09-11T08:19:46.875 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_objects": 0, 2024-09-11T08:19:46.875 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T08:19:46.875 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T08:19:46.875 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T08:19:46.875 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_flush_age": 0, 2024-09-11T08:19:46.875 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_evict_age": 0, 2024-09-11T08:19:46.875 INFO:teuthology.orchestra.run.smithi150.stdout: "erasure_code_profile": "", 2024-09-11T08:19:46.875 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_params": { 2024-09-11T08:19:46.876 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "none" 2024-09-11T08:19:46.876 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.876 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_period": 0, 2024-09-11T08:19:46.876 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_count": 0, 2024-09-11T08:19:46.876 INFO:teuthology.orchestra.run.smithi150.stdout: "use_gmt_hitset": true, 2024-09-11T08:19:46.876 INFO:teuthology.orchestra.run.smithi150.stdout: "min_read_recency_for_promote": 0, 2024-09-11T08:19:46.876 INFO:teuthology.orchestra.run.smithi150.stdout: "min_write_recency_for_promote": 0, 2024-09-11T08:19:46.876 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T08:19:46.876 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_search_last_n": 0, 2024-09-11T08:19:46.876 INFO:teuthology.orchestra.run.smithi150.stdout: "grade_table": [], 2024-09-11T08:19:46.876 INFO:teuthology.orchestra.run.smithi150.stdout: "stripe_width": 0, 2024-09-11T08:19:46.876 INFO:teuthology.orchestra.run.smithi150.stdout: "expected_num_objects": 0, 2024-09-11T08:19:46.876 INFO:teuthology.orchestra.run.smithi150.stdout: "fast_read": false, 2024-09-11T08:19:46.877 INFO:teuthology.orchestra.run.smithi150.stdout: "options": {}, 2024-09-11T08:19:46.877 INFO:teuthology.orchestra.run.smithi150.stdout: "application_metadata": { 2024-09-11T08:19:46.877 INFO:teuthology.orchestra.run.smithi150.stdout: "rados": {} 2024-09-11T08:19:46.877 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:19:46.877 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:19:46.877 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-09-11T08:19:46.877 INFO:teuthology.orchestra.run.smithi150.stdout: "osds": [ 2024-09-11T08:19:46.877 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.877 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 0, 2024-09-11T08:19:46.877 INFO:teuthology.orchestra.run.smithi150.stdout: "uuid": "37b63db9-ed2a-4701-b63d-03ff9885cc0e", 2024-09-11T08:19:46.877 INFO:teuthology.orchestra.run.smithi150.stdout: "up": 1, 2024-09-11T08:19:46.877 INFO:teuthology.orchestra.run.smithi150.stdout: "in": 1, 2024-09-11T08:19:46.877 INFO:teuthology.orchestra.run.smithi150.stdout: "weight": 1, 2024-09-11T08:19:46.878 INFO:teuthology.orchestra.run.smithi150.stdout: "primary_affinity": 1, 2024-09-11T08:19:46.878 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_begin": 0, 2024-09-11T08:19:46.878 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_end": 0, 2024-09-11T08:19:46.878 INFO:teuthology.orchestra.run.smithi150.stdout: "up_from": 8, 2024-09-11T08:19:46.878 INFO:teuthology.orchestra.run.smithi150.stdout: "up_thru": 276, 2024-09-11T08:19:46.878 INFO:teuthology.orchestra.run.smithi150.stdout: "down_at": 0, 2024-09-11T08:19:46.878 INFO:teuthology.orchestra.run.smithi150.stdout: "lost_at": 0, 2024-09-11T08:19:46.878 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addrs": { 2024-09-11T08:19:46.878 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:19:46.878 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.878 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:19:46.878 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6801", 2024-09-11T08:19:46.878 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 334255279 2024-09-11T08:19:46.878 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:19:46.879 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:19:46.879 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.879 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addrs": { 2024-09-11T08:19:46.879 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:19:46.879 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.879 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:19:46.879 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6802", 2024-09-11T08:19:46.879 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 334255279 2024-09-11T08:19:46.879 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:19:46.879 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:19:46.879 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.879 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addrs": { 2024-09-11T08:19:46.879 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:19:46.880 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.880 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:19:46.880 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6804", 2024-09-11T08:19:46.880 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 334255279 2024-09-11T08:19:46.880 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:19:46.880 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:19:46.880 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.880 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addrs": { 2024-09-11T08:19:46.880 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:19:46.880 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.880 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:19:46.880 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6803", 2024-09-11T08:19:46.880 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 334255279 2024-09-11T08:19:46.881 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:19:46.881 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:19:46.881 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.881 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addr": "172.21.15.150:6801/334255279", 2024-09-11T08:19:46.881 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addr": "172.21.15.150:6802/334255279", 2024-09-11T08:19:46.881 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addr": "172.21.15.150:6804/334255279", 2024-09-11T08:19:46.881 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addr": "172.21.15.150:6803/334255279", 2024-09-11T08:19:46.881 INFO:teuthology.orchestra.run.smithi150.stdout: "state": [ 2024-09-11T08:19:46.881 INFO:teuthology.orchestra.run.smithi150.stdout: "exists", 2024-09-11T08:19:46.881 INFO:teuthology.orchestra.run.smithi150.stdout: "up" 2024-09-11T08:19:46.881 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:19:46.881 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.881 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.882 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 1, 2024-09-11T08:19:46.882 INFO:teuthology.orchestra.run.smithi150.stdout: "uuid": "80378442-02bc-48aa-b8ca-6113a4092079", 2024-09-11T08:19:46.882 INFO:teuthology.orchestra.run.smithi150.stdout: "up": 1, 2024-09-11T08:19:46.882 INFO:teuthology.orchestra.run.smithi150.stdout: "in": 1, 2024-09-11T08:19:46.882 INFO:teuthology.orchestra.run.smithi150.stdout: "weight": 1, 2024-09-11T08:19:46.882 INFO:teuthology.orchestra.run.smithi150.stdout: "primary_affinity": 1, 2024-09-11T08:19:46.882 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_begin": 0, 2024-09-11T08:19:46.882 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_end": 0, 2024-09-11T08:19:46.882 INFO:teuthology.orchestra.run.smithi150.stdout: "up_from": 13, 2024-09-11T08:19:46.882 INFO:teuthology.orchestra.run.smithi150.stdout: "up_thru": 270, 2024-09-11T08:19:46.882 INFO:teuthology.orchestra.run.smithi150.stdout: "down_at": 0, 2024-09-11T08:19:46.882 INFO:teuthology.orchestra.run.smithi150.stdout: "lost_at": 0, 2024-09-11T08:19:46.882 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addrs": { 2024-09-11T08:19:46.882 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:19:46.883 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.883 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:19:46.883 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6805", 2024-09-11T08:19:46.883 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1129510619 2024-09-11T08:19:46.883 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:19:46.883 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:19:46.883 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.883 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addrs": { 2024-09-11T08:19:46.883 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:19:46.883 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.883 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:19:46.883 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6806", 2024-09-11T08:19:46.883 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1129510619 2024-09-11T08:19:46.883 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:19:46.884 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:19:46.884 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.884 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addrs": { 2024-09-11T08:19:46.884 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:19:46.884 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.884 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:19:46.884 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6808", 2024-09-11T08:19:46.884 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1129510619 2024-09-11T08:19:46.884 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:19:46.884 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:19:46.884 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.884 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addrs": { 2024-09-11T08:19:46.884 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:19:46.884 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.885 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:19:46.885 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6807", 2024-09-11T08:19:46.885 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1129510619 2024-09-11T08:19:46.885 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:19:46.885 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:19:46.885 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.885 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addr": "172.21.15.150:6805/1129510619", 2024-09-11T08:19:46.885 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addr": "172.21.15.150:6806/1129510619", 2024-09-11T08:19:46.885 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addr": "172.21.15.150:6808/1129510619", 2024-09-11T08:19:46.885 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addr": "172.21.15.150:6807/1129510619", 2024-09-11T08:19:46.885 INFO:teuthology.orchestra.run.smithi150.stdout: "state": [ 2024-09-11T08:19:46.885 INFO:teuthology.orchestra.run.smithi150.stdout: "exists", 2024-09-11T08:19:46.885 INFO:teuthology.orchestra.run.smithi150.stdout: "up" 2024-09-11T08:19:46.885 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:19:46.886 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.886 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.886 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 2, 2024-09-11T08:19:46.886 INFO:teuthology.orchestra.run.smithi150.stdout: "uuid": "0955bb92-36a0-4e9a-98e1-b3a5af89f6d2", 2024-09-11T08:19:46.886 INFO:teuthology.orchestra.run.smithi150.stdout: "up": 1, 2024-09-11T08:19:46.886 INFO:teuthology.orchestra.run.smithi150.stdout: "in": 1, 2024-09-11T08:19:46.886 INFO:teuthology.orchestra.run.smithi150.stdout: "weight": 1, 2024-09-11T08:19:46.886 INFO:teuthology.orchestra.run.smithi150.stdout: "primary_affinity": 1, 2024-09-11T08:19:46.886 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_begin": 0, 2024-09-11T08:19:46.886 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_end": 0, 2024-09-11T08:19:46.886 INFO:teuthology.orchestra.run.smithi150.stdout: "up_from": 18, 2024-09-11T08:19:46.886 INFO:teuthology.orchestra.run.smithi150.stdout: "up_thru": 270, 2024-09-11T08:19:46.886 INFO:teuthology.orchestra.run.smithi150.stdout: "down_at": 0, 2024-09-11T08:19:46.886 INFO:teuthology.orchestra.run.smithi150.stdout: "lost_at": 0, 2024-09-11T08:19:46.887 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addrs": { 2024-09-11T08:19:46.887 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:19:46.887 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.887 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:19:46.887 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6809", 2024-09-11T08:19:46.887 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 719975422 2024-09-11T08:19:46.887 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:19:46.887 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:19:46.887 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.887 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addrs": { 2024-09-11T08:19:46.887 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:19:46.887 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.887 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:19:46.887 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6810", 2024-09-11T08:19:46.888 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 719975422 2024-09-11T08:19:46.888 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:19:46.888 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:19:46.888 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.888 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addrs": { 2024-09-11T08:19:46.888 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:19:46.888 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.888 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:19:46.888 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6812", 2024-09-11T08:19:46.888 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 719975422 2024-09-11T08:19:46.888 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:19:46.888 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:19:46.888 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.888 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addrs": { 2024-09-11T08:19:46.889 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:19:46.889 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.889 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:19:46.889 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6811", 2024-09-11T08:19:46.889 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 719975422 2024-09-11T08:19:46.889 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:19:46.889 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:19:46.889 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.889 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addr": "172.21.15.150:6809/719975422", 2024-09-11T08:19:46.889 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addr": "172.21.15.150:6810/719975422", 2024-09-11T08:19:46.889 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addr": "172.21.15.150:6812/719975422", 2024-09-11T08:19:46.889 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addr": "172.21.15.150:6811/719975422", 2024-09-11T08:19:46.889 INFO:teuthology.orchestra.run.smithi150.stdout: "state": [ 2024-09-11T08:19:46.889 INFO:teuthology.orchestra.run.smithi150.stdout: "exists", 2024-09-11T08:19:46.890 INFO:teuthology.orchestra.run.smithi150.stdout: "up" 2024-09-11T08:19:46.890 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:19:46.890 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.890 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.890 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 3, 2024-09-11T08:19:46.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:46.891 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:46.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:19:46.892 INFO:teuthology.orchestra.run.smithi150.stdout: "uuid": "1d04d4d3-62e8-4460-8a07-77f7e5c62929", 2024-09-11T08:19:46.892 INFO:teuthology.orchestra.run.smithi150.stdout: "up": 1, 2024-09-11T08:19:46.892 INFO:teuthology.orchestra.run.smithi150.stdout: "in": 1, 2024-09-11T08:19:46.892 INFO:teuthology.orchestra.run.smithi150.stdout: "weight": 1, 2024-09-11T08:19:46.892 INFO:teuthology.orchestra.run.smithi150.stdout: "primary_affinity": 0.611114501953125, 2024-09-11T08:19:46.892 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_begin": 0, 2024-09-11T08:19:46.892 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_end": 0, 2024-09-11T08:19:46.892 INFO:teuthology.orchestra.run.smithi150.stdout: "up_from": 25, 2024-09-11T08:19:46.892 INFO:teuthology.orchestra.run.smithi150.stdout: "up_thru": 260, 2024-09-11T08:19:46.892 INFO:teuthology.orchestra.run.smithi150.stdout: "down_at": 0, 2024-09-11T08:19:46.892 INFO:teuthology.orchestra.run.smithi150.stdout: "lost_at": 0, 2024-09-11T08:19:46.892 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addrs": { 2024-09-11T08:19:46.893 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:19:46.893 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.893 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:19:46.893 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6813", 2024-09-11T08:19:46.893 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 3101032500 2024-09-11T08:19:46.893 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:19:46.893 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:19:46.893 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.893 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addrs": { 2024-09-11T08:19:46.893 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:19:46.893 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.893 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:19:46.893 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6814", 2024-09-11T08:19:46.893 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 3101032500 2024-09-11T08:19:46.894 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:19:46.894 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:19:46.894 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.894 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addrs": { 2024-09-11T08:19:46.894 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:19:46.894 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.894 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:19:46.894 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6816", 2024-09-11T08:19:46.894 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 3101032500 2024-09-11T08:19:46.894 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:19:46.894 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:19:46.894 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.894 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addrs": { 2024-09-11T08:19:46.894 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:19:46.895 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.895 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:19:46.895 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6815", 2024-09-11T08:19:46.895 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 3101032500 2024-09-11T08:19:46.895 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:19:46.895 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:19:46.895 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.895 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addr": "172.21.15.150:6813/3101032500", 2024-09-11T08:19:46.895 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addr": "172.21.15.150:6814/3101032500", 2024-09-11T08:19:46.895 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addr": "172.21.15.150:6816/3101032500", 2024-09-11T08:19:46.895 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addr": "172.21.15.150:6815/3101032500", 2024-09-11T08:19:46.895 INFO:teuthology.orchestra.run.smithi150.stdout: "state": [ 2024-09-11T08:19:46.895 INFO:teuthology.orchestra.run.smithi150.stdout: "exists", 2024-09-11T08:19:46.896 INFO:teuthology.orchestra.run.smithi150.stdout: "up" 2024-09-11T08:19:46.896 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:19:46.896 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.896 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.896 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 4, 2024-09-11T08:19:46.896 INFO:teuthology.orchestra.run.smithi150.stdout: "uuid": "c5321a3c-d2f0-4440-8475-2c111d261139", 2024-09-11T08:19:46.896 INFO:teuthology.orchestra.run.smithi150.stdout: "up": 1, 2024-09-11T08:19:46.896 INFO:teuthology.orchestra.run.smithi150.stdout: "in": 1, 2024-09-11T08:19:46.896 INFO:teuthology.orchestra.run.smithi150.stdout: "weight": 1, 2024-09-11T08:19:46.896 INFO:teuthology.orchestra.run.smithi150.stdout: "primary_affinity": 1, 2024-09-11T08:19:46.896 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_begin": 0, 2024-09-11T08:19:46.896 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_end": 0, 2024-09-11T08:19:46.896 INFO:teuthology.orchestra.run.smithi150.stdout: "up_from": 31, 2024-09-11T08:19:46.896 INFO:teuthology.orchestra.run.smithi150.stdout: "up_thru": 270, 2024-09-11T08:19:46.897 INFO:teuthology.orchestra.run.smithi150.stdout: "down_at": 0, 2024-09-11T08:19:46.897 INFO:teuthology.orchestra.run.smithi150.stdout: "lost_at": 0, 2024-09-11T08:19:46.897 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addrs": { 2024-09-11T08:19:46.897 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:19:46.897 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.897 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:19:46.897 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6800", 2024-09-11T08:19:46.897 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2567452825 2024-09-11T08:19:46.897 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:19:46.897 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:19:46.897 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.897 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addrs": { 2024-09-11T08:19:46.897 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:19:46.897 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.898 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:19:46.898 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6801", 2024-09-11T08:19:46.898 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2567452825 2024-09-11T08:19:46.898 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:19:46.898 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:19:46.898 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.898 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addrs": { 2024-09-11T08:19:46.898 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:19:46.898 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.898 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:19:46.898 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6803", 2024-09-11T08:19:46.898 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2567452825 2024-09-11T08:19:46.898 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:19:46.899 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:19:46.899 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.899 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addrs": { 2024-09-11T08:19:46.899 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:19:46.899 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.899 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:19:46.899 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6802", 2024-09-11T08:19:46.899 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2567452825 2024-09-11T08:19:46.899 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:19:46.899 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:19:46.899 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.899 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addr": "172.21.15.181:6800/2567452825", 2024-09-11T08:19:46.899 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addr": "172.21.15.181:6801/2567452825", 2024-09-11T08:19:46.899 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addr": "172.21.15.181:6803/2567452825", 2024-09-11T08:19:46.900 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addr": "172.21.15.181:6802/2567452825", 2024-09-11T08:19:46.900 INFO:teuthology.orchestra.run.smithi150.stdout: "state": [ 2024-09-11T08:19:46.900 INFO:teuthology.orchestra.run.smithi150.stdout: "exists", 2024-09-11T08:19:46.900 INFO:teuthology.orchestra.run.smithi150.stdout: "up" 2024-09-11T08:19:46.900 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:19:46.900 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.900 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.900 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 5, 2024-09-11T08:19:46.900 INFO:teuthology.orchestra.run.smithi150.stdout: "uuid": "1262b2fe-f253-4a26-bfd4-c9dffb697c77", 2024-09-11T08:19:46.900 INFO:teuthology.orchestra.run.smithi150.stdout: "up": 1, 2024-09-11T08:19:46.900 INFO:teuthology.orchestra.run.smithi150.stdout: "in": 1, 2024-09-11T08:19:46.901 INFO:teuthology.orchestra.run.smithi150.stdout: "weight": 1, 2024-09-11T08:19:46.901 INFO:teuthology.orchestra.run.smithi150.stdout: "primary_affinity": 1, 2024-09-11T08:19:46.901 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_begin": 0, 2024-09-11T08:19:46.901 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_end": 0, 2024-09-11T08:19:46.901 INFO:teuthology.orchestra.run.smithi150.stdout: "up_from": 38, 2024-09-11T08:19:46.901 INFO:teuthology.orchestra.run.smithi150.stdout: "up_thru": 271, 2024-09-11T08:19:46.901 INFO:teuthology.orchestra.run.smithi150.stdout: "down_at": 0, 2024-09-11T08:19:46.901 INFO:teuthology.orchestra.run.smithi150.stdout: "lost_at": 0, 2024-09-11T08:19:46.902 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addrs": { 2024-09-11T08:19:46.902 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:19:46.902 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.902 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:19:46.902 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6804", 2024-09-11T08:19:46.902 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2043435508 2024-09-11T08:19:46.902 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:19:46.902 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:19:46.902 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.902 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addrs": { 2024-09-11T08:19:46.902 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:19:46.902 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.902 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:19:46.903 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6805", 2024-09-11T08:19:46.903 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2043435508 2024-09-11T08:19:46.903 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:19:46.903 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:19:46.903 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.903 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addrs": { 2024-09-11T08:19:46.903 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:19:46.903 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.903 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:19:46.903 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6807", 2024-09-11T08:19:46.903 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2043435508 2024-09-11T08:19:46.903 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:19:46.903 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:19:46.903 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.904 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addrs": { 2024-09-11T08:19:46.904 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:19:46.904 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.904 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:19:46.904 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6806", 2024-09-11T08:19:46.904 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2043435508 2024-09-11T08:19:46.904 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:19:46.904 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:19:46.904 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.904 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addr": "172.21.15.181:6804/2043435508", 2024-09-11T08:19:46.904 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addr": "172.21.15.181:6805/2043435508", 2024-09-11T08:19:46.904 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addr": "172.21.15.181:6807/2043435508", 2024-09-11T08:19:46.904 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addr": "172.21.15.181:6806/2043435508", 2024-09-11T08:19:46.905 INFO:teuthology.orchestra.run.smithi150.stdout: "state": [ 2024-09-11T08:19:46.905 INFO:teuthology.orchestra.run.smithi150.stdout: "exists", 2024-09-11T08:19:46.905 INFO:teuthology.orchestra.run.smithi150.stdout: "up" 2024-09-11T08:19:46.905 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:19:46.905 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.905 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.905 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 6, 2024-09-11T08:19:46.905 INFO:teuthology.orchestra.run.smithi150.stdout: "uuid": "7facca22-b3f1-4c20-833e-e3db798a08c4", 2024-09-11T08:19:46.905 INFO:teuthology.orchestra.run.smithi150.stdout: "up": 1, 2024-09-11T08:19:46.905 INFO:teuthology.orchestra.run.smithi150.stdout: "in": 1, 2024-09-11T08:19:46.905 INFO:teuthology.orchestra.run.smithi150.stdout: "weight": 1, 2024-09-11T08:19:46.905 INFO:teuthology.orchestra.run.smithi150.stdout: "primary_affinity": 1, 2024-09-11T08:19:46.905 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_begin": 0, 2024-09-11T08:19:46.905 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_end": 0, 2024-09-11T08:19:46.906 INFO:teuthology.orchestra.run.smithi150.stdout: "up_from": 45, 2024-09-11T08:19:46.906 INFO:teuthology.orchestra.run.smithi150.stdout: "up_thru": 270, 2024-09-11T08:19:46.906 INFO:teuthology.orchestra.run.smithi150.stdout: "down_at": 0, 2024-09-11T08:19:46.906 INFO:teuthology.orchestra.run.smithi150.stdout: "lost_at": 0, 2024-09-11T08:19:46.906 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addrs": { 2024-09-11T08:19:46.906 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:19:46.906 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.906 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:19:46.906 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6808", 2024-09-11T08:19:46.906 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1789880331 2024-09-11T08:19:46.906 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:19:46.906 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:19:46.906 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.907 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addrs": { 2024-09-11T08:19:46.907 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:19:46.907 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.907 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:19:46.907 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6809", 2024-09-11T08:19:46.907 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1789880331 2024-09-11T08:19:46.907 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:19:46.907 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:19:46.907 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.907 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addrs": { 2024-09-11T08:19:46.907 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:19:46.907 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.907 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:19:46.908 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6811", 2024-09-11T08:19:46.908 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1789880331 2024-09-11T08:19:46.908 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:19:46.908 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:19:46.908 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.908 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addrs": { 2024-09-11T08:19:46.908 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:19:46.908 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.908 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:19:46.908 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6810", 2024-09-11T08:19:46.908 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1789880331 2024-09-11T08:19:46.908 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:19:46.908 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:19:46.909 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.909 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addr": "172.21.15.181:6808/1789880331", 2024-09-11T08:19:46.909 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addr": "172.21.15.181:6809/1789880331", 2024-09-11T08:19:46.909 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addr": "172.21.15.181:6811/1789880331", 2024-09-11T08:19:46.909 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addr": "172.21.15.181:6810/1789880331", 2024-09-11T08:19:46.909 INFO:teuthology.orchestra.run.smithi150.stdout: "state": [ 2024-09-11T08:19:46.909 INFO:teuthology.orchestra.run.smithi150.stdout: "exists", 2024-09-11T08:19:46.909 INFO:teuthology.orchestra.run.smithi150.stdout: "up" 2024-09-11T08:19:46.909 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:19:46.909 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.909 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.909 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 7, 2024-09-11T08:19:46.909 INFO:teuthology.orchestra.run.smithi150.stdout: "uuid": "6ac1b06a-79bd-43ca-b91e-fcb0f4033a34", 2024-09-11T08:19:46.909 INFO:teuthology.orchestra.run.smithi150.stdout: "up": 1, 2024-09-11T08:19:46.910 INFO:teuthology.orchestra.run.smithi150.stdout: "in": 1, 2024-09-11T08:19:46.910 INFO:teuthology.orchestra.run.smithi150.stdout: "weight": 1, 2024-09-11T08:19:46.910 INFO:teuthology.orchestra.run.smithi150.stdout: "primary_affinity": 1, 2024-09-11T08:19:46.910 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_begin": 53, 2024-09-11T08:19:46.910 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_end": 139, 2024-09-11T08:19:46.910 INFO:teuthology.orchestra.run.smithi150.stdout: "up_from": 168, 2024-09-11T08:19:46.910 INFO:teuthology.orchestra.run.smithi150.stdout: "up_thru": 278, 2024-09-11T08:19:46.910 INFO:teuthology.orchestra.run.smithi150.stdout: "down_at": 140, 2024-09-11T08:19:46.910 INFO:teuthology.orchestra.run.smithi150.stdout: "lost_at": 0, 2024-09-11T08:19:46.910 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addrs": { 2024-09-11T08:19:46.910 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:19:46.910 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.910 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:19:46.911 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6812", 2024-09-11T08:19:46.911 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2706455857 2024-09-11T08:19:46.911 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:19:46.911 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:19:46.911 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.911 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addrs": { 2024-09-11T08:19:46.911 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:19:46.911 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.911 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:19:46.911 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6813", 2024-09-11T08:19:46.911 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2706455857 2024-09-11T08:19:46.911 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:19:46.911 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:19:46.912 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.912 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addrs": { 2024-09-11T08:19:46.912 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:19:46.912 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.912 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:19:46.912 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6815", 2024-09-11T08:19:46.912 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2706455857 2024-09-11T08:19:46.912 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:19:46.912 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:19:46.912 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.912 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addrs": { 2024-09-11T08:19:46.912 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:19:46.912 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.913 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:19:46.913 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6814", 2024-09-11T08:19:46.913 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2706455857 2024-09-11T08:19:46.913 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:19:46.913 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:19:46.913 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.913 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addr": "172.21.15.181:6812/2706455857", 2024-09-11T08:19:46.913 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addr": "172.21.15.181:6813/2706455857", 2024-09-11T08:19:46.913 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addr": "172.21.15.181:6815/2706455857", 2024-09-11T08:19:46.913 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addr": "172.21.15.181:6814/2706455857", 2024-09-11T08:19:46.913 INFO:teuthology.orchestra.run.smithi150.stdout: "state": [ 2024-09-11T08:19:46.913 INFO:teuthology.orchestra.run.smithi150.stdout: "exists", 2024-09-11T08:19:46.913 INFO:teuthology.orchestra.run.smithi150.stdout: "up" 2024-09-11T08:19:46.913 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:19:46.914 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:19:46.914 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-09-11T08:19:46.914 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_xinfo": [ 2024-09-11T08:19:46.914 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.914 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 0, 2024-09-11T08:19:46.914 INFO:teuthology.orchestra.run.smithi150.stdout: "down_stamp": "0.000000", 2024-09-11T08:19:46.914 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_probability": 0, 2024-09-11T08:19:46.914 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_interval": 0, 2024-09-11T08:19:46.914 INFO:teuthology.orchestra.run.smithi150.stdout: "features": 4540138320759226367, 2024-09-11T08:19:46.914 INFO:teuthology.orchestra.run.smithi150.stdout: "old_weight": 0, 2024-09-11T08:19:46.914 INFO:teuthology.orchestra.run.smithi150.stdout: "last_purged_snaps_scrub": "2024-09-11T08:18:53.944858+0000", 2024-09-11T08:19:46.914 INFO:teuthology.orchestra.run.smithi150.stdout: "dead_epoch": 0 2024-09-11T08:19:46.914 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.914 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.915 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 1, 2024-09-11T08:19:46.915 INFO:teuthology.orchestra.run.smithi150.stdout: "down_stamp": "0.000000", 2024-09-11T08:19:46.915 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_probability": 0, 2024-09-11T08:19:46.915 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_interval": 0, 2024-09-11T08:19:46.915 INFO:teuthology.orchestra.run.smithi150.stdout: "features": 4540138320759226367, 2024-09-11T08:19:46.915 INFO:teuthology.orchestra.run.smithi150.stdout: "old_weight": 0, 2024-09-11T08:19:46.915 INFO:teuthology.orchestra.run.smithi150.stdout: "last_purged_snaps_scrub": "2024-09-11T08:18:33.903577+0000", 2024-09-11T08:19:46.915 INFO:teuthology.orchestra.run.smithi150.stdout: "dead_epoch": 0 2024-09-11T08:19:46.915 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.915 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.915 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 2, 2024-09-11T08:19:46.915 INFO:teuthology.orchestra.run.smithi150.stdout: "down_stamp": "0.000000", 2024-09-11T08:19:46.915 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_probability": 0, 2024-09-11T08:19:46.915 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_interval": 0, 2024-09-11T08:19:46.916 INFO:teuthology.orchestra.run.smithi150.stdout: "features": 4540138320759226367, 2024-09-11T08:19:46.916 INFO:teuthology.orchestra.run.smithi150.stdout: "old_weight": 0, 2024-09-11T08:19:46.916 INFO:teuthology.orchestra.run.smithi150.stdout: "last_purged_snaps_scrub": "2024-09-11T08:19:13.704246+0000", 2024-09-11T08:19:46.916 INFO:teuthology.orchestra.run.smithi150.stdout: "dead_epoch": 0 2024-09-11T08:19:46.916 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.916 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.916 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 3, 2024-09-11T08:19:46.916 INFO:teuthology.orchestra.run.smithi150.stdout: "down_stamp": "0.000000", 2024-09-11T08:19:46.916 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_probability": 0, 2024-09-11T08:19:46.916 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_interval": 0, 2024-09-11T08:19:46.916 INFO:teuthology.orchestra.run.smithi150.stdout: "features": 4540138320759226367, 2024-09-11T08:19:46.916 INFO:teuthology.orchestra.run.smithi150.stdout: "old_weight": 0, 2024-09-11T08:19:46.916 INFO:teuthology.orchestra.run.smithi150.stdout: "last_purged_snaps_scrub": "2024-09-11T08:18:36.090888+0000", 2024-09-11T08:19:46.916 INFO:teuthology.orchestra.run.smithi150.stdout: "dead_epoch": 0 2024-09-11T08:19:46.917 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.917 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.917 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 4, 2024-09-11T08:19:46.917 INFO:teuthology.orchestra.run.smithi150.stdout: "down_stamp": "0.000000", 2024-09-11T08:19:46.917 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_probability": 0, 2024-09-11T08:19:46.917 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_interval": 0, 2024-09-11T08:19:46.917 INFO:teuthology.orchestra.run.smithi150.stdout: "features": 4540138320759226367, 2024-09-11T08:19:46.917 INFO:teuthology.orchestra.run.smithi150.stdout: "old_weight": 0, 2024-09-11T08:19:46.917 INFO:teuthology.orchestra.run.smithi150.stdout: "last_purged_snaps_scrub": "2024-09-11T08:19:10.979682+0000", 2024-09-11T08:19:46.917 INFO:teuthology.orchestra.run.smithi150.stdout: "dead_epoch": 0 2024-09-11T08:19:46.917 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.917 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.917 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 5, 2024-09-11T08:19:46.918 INFO:teuthology.orchestra.run.smithi150.stdout: "down_stamp": "0.000000", 2024-09-11T08:19:46.918 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_probability": 0, 2024-09-11T08:19:46.918 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_interval": 0, 2024-09-11T08:19:46.918 INFO:teuthology.orchestra.run.smithi150.stdout: "features": 4540138320759226367, 2024-09-11T08:19:46.918 INFO:teuthology.orchestra.run.smithi150.stdout: "old_weight": 0, 2024-09-11T08:19:46.918 INFO:teuthology.orchestra.run.smithi150.stdout: "last_purged_snaps_scrub": "2024-09-11T08:19:05.764536+0000", 2024-09-11T08:19:46.918 INFO:teuthology.orchestra.run.smithi150.stdout: "dead_epoch": 0 2024-09-11T08:19:46.918 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.918 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.918 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 6, 2024-09-11T08:19:46.918 INFO:teuthology.orchestra.run.smithi150.stdout: "down_stamp": "0.000000", 2024-09-11T08:19:46.918 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_probability": 0, 2024-09-11T08:19:46.918 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_interval": 0, 2024-09-11T08:19:46.918 INFO:teuthology.orchestra.run.smithi150.stdout: "features": 4540138320759226367, 2024-09-11T08:19:46.919 INFO:teuthology.orchestra.run.smithi150.stdout: "old_weight": 0, 2024-09-11T08:19:46.919 INFO:teuthology.orchestra.run.smithi150.stdout: "last_purged_snaps_scrub": "2024-09-11T08:19:08.583336+0000", 2024-09-11T08:19:46.919 INFO:teuthology.orchestra.run.smithi150.stdout: "dead_epoch": 0 2024-09-11T08:19:46.919 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.919 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.919 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 7, 2024-09-11T08:19:46.919 INFO:teuthology.orchestra.run.smithi150.stdout: "down_stamp": "0.000000", 2024-09-11T08:19:46.919 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_probability": 0, 2024-09-11T08:19:46.919 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_interval": 0, 2024-09-11T08:19:46.919 INFO:teuthology.orchestra.run.smithi150.stdout: "features": 4540138320759226367, 2024-09-11T08:19:46.919 INFO:teuthology.orchestra.run.smithi150.stdout: "old_weight": 0, 2024-09-11T08:19:46.919 INFO:teuthology.orchestra.run.smithi150.stdout: "last_purged_snaps_scrub": "2024-09-11T08:19:17.980963+0000", 2024-09-11T08:19:46.919 INFO:teuthology.orchestra.run.smithi150.stdout: "dead_epoch": 139 2024-09-11T08:19:46.919 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:19:46.920 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-09-11T08:19:46.920 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_upmap": [], 2024-09-11T08:19:46.920 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_upmap_items": [ 2024-09-11T08:19:46.920 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.920 INFO:teuthology.orchestra.run.smithi150.stdout: "pgid": "4.4", 2024-09-11T08:19:46.920 INFO:teuthology.orchestra.run.smithi150.stdout: "mappings": [ 2024-09-11T08:19:46.920 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.920 INFO:teuthology.orchestra.run.smithi150.stdout: "from": 1, 2024-09-11T08:19:46.920 INFO:teuthology.orchestra.run.smithi150.stdout: "to": 0 2024-09-11T08:19:46.920 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:19:46.920 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:19:46.920 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.920 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.920 INFO:teuthology.orchestra.run.smithi150.stdout: "pgid": "4.14", 2024-09-11T08:19:46.921 INFO:teuthology.orchestra.run.smithi150.stdout: "mappings": [ 2024-09-11T08:19:46.921 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.921 INFO:teuthology.orchestra.run.smithi150.stdout: "from": 1, 2024-09-11T08:19:46.921 INFO:teuthology.orchestra.run.smithi150.stdout: "to": 2 2024-09-11T08:19:46.921 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:19:46.921 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:19:46.921 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.921 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.921 INFO:teuthology.orchestra.run.smithi150.stdout: "pgid": "4.1d", 2024-09-11T08:19:46.921 INFO:teuthology.orchestra.run.smithi150.stdout: "mappings": [ 2024-09-11T08:19:46.921 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.921 INFO:teuthology.orchestra.run.smithi150.stdout: "from": 1, 2024-09-11T08:19:46.921 INFO:teuthology.orchestra.run.smithi150.stdout: "to": 2 2024-09-11T08:19:46.922 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:19:46.922 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:19:46.922 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.922 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.922 INFO:teuthology.orchestra.run.smithi150.stdout: "pgid": "5.3", 2024-09-11T08:19:46.922 INFO:teuthology.orchestra.run.smithi150.stdout: "mappings": [ 2024-09-11T08:19:46.922 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.922 INFO:teuthology.orchestra.run.smithi150.stdout: "from": 0, 2024-09-11T08:19:46.922 INFO:teuthology.orchestra.run.smithi150.stdout: "to": 3 2024-09-11T08:19:46.922 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:19:46.922 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:19:46.922 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.922 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.922 INFO:teuthology.orchestra.run.smithi150.stdout: "pgid": "5.1e", 2024-09-11T08:19:46.923 INFO:teuthology.orchestra.run.smithi150.stdout: "mappings": [ 2024-09-11T08:19:46.923 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:19:46.923 INFO:teuthology.orchestra.run.smithi150.stdout: "from": 0, 2024-09-11T08:19:46.923 INFO:teuthology.orchestra.run.smithi150.stdout: "to": 6 2024-09-11T08:19:46.923 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:19:46.923 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:19:46.923 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:19:46.923 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-09-11T08:19:46.923 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_temp": [], 2024-09-11T08:19:46.923 INFO:teuthology.orchestra.run.smithi150.stdout: "primary_temp": [], 2024-09-11T08:19:46.923 INFO:teuthology.orchestra.run.smithi150.stdout: "blocklist": { 2024-09-11T08:19:46.923 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/2620931070": "2024-09-12T08:08:06.649692+0000", 2024-09-11T08:19:46.923 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/1073581409": "2024-09-12T08:02:45.986151+0000", 2024-09-11T08:19:46.923 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/2363649697": "2024-09-12T08:08:06.649692+0000", 2024-09-11T08:19:46.924 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:6800/603451071": "2024-09-12T08:02:45.986151+0000", 2024-09-11T08:19:46.924 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/1614413548": "2024-09-12T08:02:45.986151+0000", 2024-09-11T08:19:46.924 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/3673218166": "2024-09-12T08:03:08.740828+0000", 2024-09-11T08:19:46.924 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:6800/556498589": "2024-09-12T08:03:08.740828+0000", 2024-09-11T08:19:46.924 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/3666733994": "2024-09-12T08:03:08.740828+0000", 2024-09-11T08:19:46.924 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/3271479010": "2024-09-12T08:03:08.740828+0000", 2024-09-11T08:19:46.924 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/3185731090": "2024-09-12T08:08:06.649692+0000", 2024-09-11T08:19:46.924 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/2539412006": "2024-09-12T08:08:06.649692+0000", 2024-09-11T08:19:46.924 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/2740222408": "2024-09-12T08:02:45.986151+0000", 2024-09-11T08:19:46.924 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:6800/127856313": "2024-09-12T08:08:06.649692+0000", 2024-09-11T08:19:46.924 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/2375121603": "2024-09-12T08:08:06.649692+0000" 2024-09-11T08:19:46.924 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.924 INFO:teuthology.orchestra.run.smithi150.stdout: "range_blocklist": {}, 2024-09-11T08:19:46.924 INFO:teuthology.orchestra.run.smithi150.stdout: "erasure_code_profiles": { 2024-09-11T08:19:46.925 INFO:teuthology.orchestra.run.smithi150.stdout: "default": { 2024-09-11T08:19:46.925 INFO:teuthology.orchestra.run.smithi150.stdout: "crush-failure-domain": "osd", 2024-09-11T08:19:46.925 INFO:teuthology.orchestra.run.smithi150.stdout: "k": "2", 2024-09-11T08:19:46.925 INFO:teuthology.orchestra.run.smithi150.stdout: "m": "1", 2024-09-11T08:19:46.925 INFO:teuthology.orchestra.run.smithi150.stdout: "plugin": "jerasure", 2024-09-11T08:19:46.925 INFO:teuthology.orchestra.run.smithi150.stdout: "technique": "reed_sol_van" 2024-09-11T08:19:46.925 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:19:46.925 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:19:46.925 INFO:teuthology.orchestra.run.smithi150.stdout: "removed_snaps_queue": [], 2024-09-11T08:19:46.925 INFO:teuthology.orchestra.run.smithi150.stdout: "new_removed_snaps": [], 2024-09-11T08:19:46.925 INFO:teuthology.orchestra.run.smithi150.stdout: "new_purged_snaps": [], 2024-09-11T08:19:46.925 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_node_flags": {}, 2024-09-11T08:19:46.925 INFO:teuthology.orchestra.run.smithi150.stdout: "device_class_flags": {}, 2024-09-11T08:19:46.925 INFO:teuthology.orchestra.run.smithi150.stdout: "stretch_mode": { 2024-09-11T08:19:46.926 INFO:teuthology.orchestra.run.smithi150.stdout: "stretch_mode_enabled": false, 2024-09-11T08:19:46.926 INFO:teuthology.orchestra.run.smithi150.stdout: "stretch_bucket_count": 0, 2024-09-11T08:19:46.926 INFO:teuthology.orchestra.run.smithi150.stdout: "degraded_stretch_mode": 0, 2024-09-11T08:19:46.926 INFO:teuthology.orchestra.run.smithi150.stdout: "recovering_stretch_mode": 0, 2024-09-11T08:19:46.926 INFO:teuthology.orchestra.run.smithi150.stdout: "stretch_mode_bucket": 0 2024-09-11T08:19:46.926 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:19:46.926 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-11T08:19:46.993 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:47.024 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:47.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:47.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:47.126 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:47.159 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:47.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:47.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:47.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:46 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1085650141' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-11T08:19:47.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:46 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1085650141' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-11T08:19:47.261 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:19:47.300 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:47.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:47.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:19:47.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:46 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1085650141' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-11T08:19:47.403 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:47.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:47.437 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:47.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:47.538 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:47.572 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:47.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:47.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:47.676 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:19:47.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:47.711 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:47.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:19:47.812 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:47.843 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:47.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:47.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:47.945 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:19:47.965 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:47 smithi150 ceph-mon[29276]: pgmap v559: 148 pgs: 1 peering, 147 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 852 B/s rd, 15 MiB/s wr, 248 op/s; 12 MiB/s, 187 objects/s recovering 2024-09-11T08:19:47.965 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:47 smithi150 ceph-mon[39612]: pgmap v559: 148 pgs: 1 peering, 147 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 852 B/s rd, 15 MiB/s wr, 248 op/s; 12 MiB/s, 187 objects/s recovering 2024-09-11T08:19:47.975 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:47.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:47.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:19:47.979 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:19:47.998 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-11T08:19:47.998 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [], 2024-09-11T08:19:47.998 INFO:teuthology.orchestra.run.smithi150.stdout: "complaint_time": 30, 2024-09-11T08:19:47.998 INFO:teuthology.orchestra.run.smithi150.stdout: "num_blocked_ops": 0 2024-09-11T08:19:47.998 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-11T08:19:48.077 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:48.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:48.119 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:48.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:48.219 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:48.224 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:47 smithi181 ceph-mon[32701]: pgmap v559: 148 pgs: 1 peering, 147 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 852 B/s rd, 15 MiB/s wr, 248 op/s; 12 MiB/s, 187 objects/s recovering 2024-09-11T08:19:48.268 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:48.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:48.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:48.370 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:48.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:48.425 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:48.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:48.526 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:48.566 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:48.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:48.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:48.573 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:19:48 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:19:48.669 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:48.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:48.707 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:48.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:48.770 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:19:48 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:19:48] "GET /metrics HTTP/1.1" 200 224565 "" "Prometheus/2.43.0" 2024-09-11T08:19:48.808 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:19:48.810 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-11T08:19:48.844 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:48.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:48.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:19:48.946 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:19:48.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:48.976 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:48.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:19:49.078 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:19:49.136 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:49.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:49.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:19:49.238 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:49.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:48 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:19:49.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:48 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:19:49.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:48 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:19:49.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:48 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:19:49.279 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:49.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:49.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:49.321 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:19:49.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:48 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:19:49.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:48 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:19:49.346 INFO:tasks.thrashosds.thrasher:j is {'epoch': 279, 'fsid': 'c1bddff4-7013-11ef-bcea-c7b262605968', 'created': '2024-09-11T08:02:15.344618+0000', 'modified': '2024-09-11T08:19:43.779587+0000', 'last_up_change': '2024-09-11T08:14:23.745449+0000', 'last_in_change': '2024-09-11T08:19:28.121267+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-11T08:05:21.044729+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '255', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-11T08:07:28.786572+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 63, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-11T08:07:29.021286+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-11T08:07:30.334181+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-11T08:07:32.289626+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-11T08:07:34.351861+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 9, 'pool_name': 'unique_pool_2', 'create_time': '2024-09-11T08:17:59.761532+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '251', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': '37b63db9-ed2a-4701-b63d-03ff9885cc0e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 276, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6801', 'nonce': 334255279}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6802', 'nonce': 334255279}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6804', 'nonce': 334255279}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6803', 'nonce': 334255279}]}, 'public_addr': '172.21.15.150:6801/334255279', 'cluster_addr': '172.21.15.150:6802/334255279', 'heartbeat_back_addr': '172.21.15.150:6804/334255279', 'heartbeat_front_addr': '172.21.15.150:6803/334255279', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '80378442-02bc-48aa-b8ca-6113a4092079', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 270, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6805', 'nonce': 1129510619}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6806', 'nonce': 1129510619}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6808', 'nonce': 1129510619}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6807', 'nonce': 1129510619}]}, 'public_addr': '172.21.15.150:6805/1129510619', 'cluster_addr': '172.21.15.150:6806/1129510619', 'heartbeat_back_addr': '172.21.15.150:6808/1129510619', 'heartbeat_front_addr': '172.21.15.150:6807/1129510619', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '0955bb92-36a0-4e9a-98e1-b3a5af89f6d2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 270, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6809', 'nonce': 719975422}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6810', 'nonce': 719975422}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6812', 'nonce': 719975422}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6811', 'nonce': 719975422}]}, 'public_addr': '172.21.15.150:6809/719975422', 'cluster_addr': '172.21.15.150:6810/719975422', 'heartbeat_back_addr': '172.21.15.150:6812/719975422', 'heartbeat_front_addr': '172.21.15.150:6811/719975422', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '1d04d4d3-62e8-4460-8a07-77f7e5c62929', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.611114501953125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 260, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6813', 'nonce': 3101032500}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6814', 'nonce': 3101032500}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6816', 'nonce': 3101032500}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6815', 'nonce': 3101032500}]}, 'public_addr': '172.21.15.150:6813/3101032500', 'cluster_addr': '172.21.15.150:6814/3101032500', 'heartbeat_back_addr': '172.21.15.150:6816/3101032500', 'heartbeat_front_addr': '172.21.15.150:6815/3101032500', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'c5321a3c-d2f0-4440-8475-2c111d261139', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 270, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6800', 'nonce': 2567452825}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6801', 'nonce': 2567452825}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6803', 'nonce': 2567452825}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6802', 'nonce': 2567452825}]}, 'public_addr': '172.21.15.181:6800/2567452825', 'cluster_addr': '172.21.15.181:6801/2567452825', 'heartbeat_back_addr': '172.21.15.181:6803/2567452825', 'heartbeat_front_addr': '172.21.15.181:6802/2567452825', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '1262b2fe-f253-4a26-bfd4-c9dffb697c77', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 271, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6804', 'nonce': 2043435508}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6805', 'nonce': 2043435508}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6807', 'nonce': 2043435508}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6806', 'nonce': 2043435508}]}, 'public_addr': '172.21.15.181:6804/2043435508', 'cluster_addr': '172.21.15.181:6805/2043435508', 'heartbeat_back_addr': '172.21.15.181:6807/2043435508', 'heartbeat_front_addr': '172.21.15.181:6806/2043435508', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '7facca22-b3f1-4c20-833e-e3db798a08c4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 270, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6808', 'nonce': 1789880331}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6809', 'nonce': 1789880331}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6811', 'nonce': 1789880331}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6810', 'nonce': 1789880331}]}, 'public_addr': '172.21.15.181:6808/1789880331', 'cluster_addr': '172.21.15.181:6809/1789880331', 'heartbeat_back_addr': '172.21.15.181:6811/1789880331', 'heartbeat_front_addr': '172.21.15.181:6810/1789880331', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '6ac1b06a-79bd-43ca-b91e-fcb0f4033a34', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 53, 'last_clean_end': 139, 'up_from': 168, 'up_thru': 278, 'down_at': 140, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6812', 'nonce': 2706455857}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6813', 'nonce': 2706455857}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6815', 'nonce': 2706455857}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6814', 'nonce': 2706455857}]}, 'public_addr': '172.21.15.181:6812/2706455857', 'cluster_addr': '172.21.15.181:6813/2706455857', 'heartbeat_back_addr': '172.21.15.181:6815/2706455857', 'heartbeat_front_addr': '172.21.15.181:6814/2706455857', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T08:18:53.944858+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T08:18:33.903577+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T08:19:13.704246+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T08:18:36.090888+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T08:19:10.979682+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T08:19:05.764536+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T08:19:08.583336+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T08:19:17.980963+0000', 'dead_epoch': 139}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.4', 'mappings': [{'from': 1, 'to': 0}]}, {'pgid': '4.14', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1d', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '5.3', 'mappings': [{'from': 0, 'to': 3}]}, {'pgid': '5.1e', 'mappings': [{'from': 0, 'to': 6}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.150:0/2620931070': '2024-09-12T08:08:06.649692+0000', '172.21.15.150:0/1073581409': '2024-09-12T08:02:45.986151+0000', '172.21.15.150:0/2363649697': '2024-09-12T08:08:06.649692+0000', '172.21.15.150:6800/603451071': '2024-09-12T08:02:45.986151+0000', '172.21.15.150:0/1614413548': '2024-09-12T08:02:45.986151+0000', '172.21.15.150:0/3673218166': '2024-09-12T08:03:08.740828+0000', '172.21.15.150:6800/556498589': '2024-09-12T08:03:08.740828+0000', '172.21.15.150:0/3666733994': '2024-09-12T08:03:08.740828+0000', '172.21.15.150:0/3271479010': '2024-09-12T08:03:08.740828+0000', '172.21.15.150:0/3185731090': '2024-09-12T08:08:06.649692+0000', '172.21.15.150:0/2539412006': '2024-09-12T08:08:06.649692+0000', '172.21.15.150:0/2740222408': '2024-09-12T08:02:45.986151+0000', '172.21.15.150:6800/127856313': '2024-09-12T08:08:06.649692+0000', '172.21.15.150:0/2375121603': '2024-09-12T08:08:06.649692+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-11T08:19:49.346 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T08:19:49.381 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:19:49.435 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:49.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:49.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:19:49.538 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:49.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:49.589 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:49.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:49.690 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:49.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:49.731 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:49.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:49.832 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:49.864 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:49.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:49.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:49.965 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:19:50.011 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:50.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:50.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:19:50.113 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:50.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:50.150 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:50.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:50.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:49 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:19:50.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:49 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:19:50.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:49 smithi150 ceph-mon[29276]: osdmap e280: 8 total, 8 up, 8 in 2024-09-11T08:19:50.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:49 smithi150 ceph-mon[29276]: pgmap v561: 148 pgs: 148 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.8 KiB/s rd, 15 MiB/s wr, 1.18k op/s; 11 MiB/s, 181 objects/s recovering 2024-09-11T08:19:50.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:49 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:19:50.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:49 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:19:50.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:49 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:19:50.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:49 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:19:50.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:49 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:19:50.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:49 smithi150 ceph-mon[39612]: osdmap e280: 8 total, 8 up, 8 in 2024-09-11T08:19:50.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:49 smithi150 ceph-mon[39612]: pgmap v561: 148 pgs: 148 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.8 KiB/s rd, 15 MiB/s wr, 1.18k op/s; 11 MiB/s, 181 objects/s recovering 2024-09-11T08:19:50.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:49 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:19:50.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:49 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:19:50.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:49 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:19:50.251 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:50.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:50.291 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:50.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:50.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:49 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:19:50.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:49 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:19:50.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:49 smithi181 ceph-mon[32701]: osdmap e280: 8 total, 8 up, 8 in 2024-09-11T08:19:50.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:49 smithi181 ceph-mon[32701]: pgmap v561: 148 pgs: 148 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.8 KiB/s rd, 15 MiB/s wr, 1.18k op/s; 11 MiB/s, 181 objects/s recovering 2024-09-11T08:19:50.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:49 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:19:50.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:49 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:19:50.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:49 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:19:50.393 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:50.433 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:50.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:50.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:50.535 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:19:50.587 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:50.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:50.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:19:50.689 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:50.728 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:50.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:50.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:50.830 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:50.865 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:50.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:50.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:50.967 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:19:51.006 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:51.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:51.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:19:51.108 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:19:51.145 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:51.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:51.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:19:51.247 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:51.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:51.283 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:51.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:51.384 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:51.418 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:51.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:51.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:51.520 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:19:51.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:51.558 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:51.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:19:51.659 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:51.697 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:51.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:51.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:51.798 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:19:51.832 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:51.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:51.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:19:51.934 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:19:51.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:51.968 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:51.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:19:51.970 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:51 smithi150 ceph-mon[29276]: pgmap v562: 148 pgs: 148 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 767 B/s rd, 7.9 MiB/s wr, 846 op/s; 3.0 MiB/s, 47 objects/s recovering 2024-09-11T08:19:51.971 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:51 smithi150 ceph-mon[39612]: pgmap v562: 148 pgs: 148 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 767 B/s rd, 7.9 MiB/s wr, 846 op/s; 3.0 MiB/s, 47 objects/s recovering 2024-09-11T08:19:52.069 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:52.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:51 smithi181 ceph-mon[32701]: pgmap v562: 148 pgs: 148 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 767 B/s rd, 7.9 MiB/s wr, 846 op/s; 3.0 MiB/s, 47 objects/s recovering 2024-09-11T08:19:52.107 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:52.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:52.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:52.210 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:19:52.256 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:52.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:52.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:19:52.420 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:52.451 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:52.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:52.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:52.553 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:52.594 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:52.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:52.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:52.696 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:19:52.740 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:52.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:52.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:19:52.743 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:19:52.743 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:19:52.743 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:19:52.744 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:19:52.770 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:19:52.771 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:19:52.771 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:52 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:19:52.771 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:52 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:19:52.771 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:52 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/256636031' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:19:52.772 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:52 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:19:52.772 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:52 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:19:52.772 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:52 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/256636031' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:19:52.809 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:19:52.810 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:19:52.837 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:19:52.838 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:19:52.842 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:52.872 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:19:52.873 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:19:52.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:52.898 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:52.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:52.912 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:19:52.912 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:19:52.951 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:19:52.952 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:19:52.999 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:19:53.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:53.025 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:53.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:19:53.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:52 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:19:53.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:52 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:19:53.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:52 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/256636031' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:19:53.126 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:53.154 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:53.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:53.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:53.256 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:19:53.284 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:53.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:53.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:19:53.385 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:19:53.415 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:53.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:53.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:19:53.518 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:53.555 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:53.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:53.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:53.657 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:19:53.689 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:53.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:53.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:19:53.737 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:19:53.760 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:19:53.779 INFO:teuthology.orchestra.run.smithi150.stderr:nodeep-scrub is set 2024-09-11T08:19:53.790 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:19:53.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:53.859 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:53.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:19:53.960 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:19:53.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:53 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:19:53.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:53 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:19:53.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:53 smithi150 ceph-mon[29276]: osdmap e281: 8 total, 8 up, 8 in 2024-09-11T08:19:53.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:53 smithi150 ceph-mon[29276]: pgmap v564: 148 pgs: 148 active+clean; 6.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 961 op/s 2024-09-11T08:19:53.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:53 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2247306926' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:19:53.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:53 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:19:53.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:53 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:19:53.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:53 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:19:53.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:53 smithi150 ceph-mon[39612]: osdmap e281: 8 total, 8 up, 8 in 2024-09-11T08:19:53.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:53 smithi150 ceph-mon[39612]: pgmap v564: 148 pgs: 148 active+clean; 6.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 961 op/s 2024-09-11T08:19:53.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:53 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2247306926' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:19:53.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:53 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:19:53.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:53.994 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:53.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:19:54.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:53 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:19:54.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:53 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:19:54.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:53 smithi181 ceph-mon[32701]: osdmap e281: 8 total, 8 up, 8 in 2024-09-11T08:19:54.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:53 smithi181 ceph-mon[32701]: pgmap v564: 148 pgs: 148 active+clean; 6.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 961 op/s 2024-09-11T08:19:54.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:53 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2247306926' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:19:54.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:53 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:19:54.096 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:19:54.130 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:54.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:54.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:19:54.233 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:54.263 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:54.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:54.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:54.365 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:54.400 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:54.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:54.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:54.502 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:19:54.538 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:54.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:54.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:19:54.640 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:54.683 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:54.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:54.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:54.785 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:54.817 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:54.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:54.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:54.919 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:54.920 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:54 smithi150 ceph-mon[29276]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:19:54.920 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:54 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:19:54.921 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:54 smithi150 ceph-mon[29276]: osdmap e282: 8 total, 8 up, 8 in 2024-09-11T08:19:54.921 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:54 smithi150 ceph-mon[39612]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:19:54.921 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:54 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:19:54.921 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:54 smithi150 ceph-mon[39612]: osdmap e282: 8 total, 8 up, 8 in 2024-09-11T08:19:54.955 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:54.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:54.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:55.056 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:55.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:54 smithi181 ceph-mon[32701]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:19:55.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:54 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:19:55.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:54 smithi181 ceph-mon[32701]: osdmap e282: 8 total, 8 up, 8 in 2024-09-11T08:19:55.087 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:55.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:55.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:55.189 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:19:55.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:55.230 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:55.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:19:55.331 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:19:55.366 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:55.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:55.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:19:55.469 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:55.501 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:55.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:55.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:55.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:19:55 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:19:55] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:19:55.603 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:55.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:55.632 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:55.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:55.733 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:55.755 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T08:19:55.762 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:55.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:55.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:55.864 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:19:55.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:55.889 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:55.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:19:55.990 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:19:56.032 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:56.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:56.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:19:56.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:55 smithi181 ceph-mon[32701]: pgmap v566: 148 pgs: 148 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 850 B/s rd, 0 B/s wr, 1.03k op/s 2024-09-11T08:19:56.133 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:55 smithi150 ceph-mon[29276]: pgmap v566: 148 pgs: 148 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 850 B/s rd, 0 B/s wr, 1.03k op/s 2024-09-11T08:19:56.134 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:55 smithi150 ceph-mon[39612]: pgmap v566: 148 pgs: 148 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 850 B/s rd, 0 B/s wr, 1.03k op/s 2024-09-11T08:19:56.134 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:56.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:56.163 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:56.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:56.264 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:56.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:56.298 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:56.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:56.399 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:56.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:56.433 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:56.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:56.534 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:19:56.563 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:56.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:56.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:19:56.606 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T08:19:56.665 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:56.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:56.708 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:56.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:56.809 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:19:56.839 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:56.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:56.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:19:56.942 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:19:56.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:56.980 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:56.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:19:57.081 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:19:57.116 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:57.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:57.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:19:57.218 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:19:57.251 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:57.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:57.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:19:57.353 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:57.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:57.388 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:57.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:57.490 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:57.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:57.520 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:57.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:57.621 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:57.657 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:57.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:57.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:57.759 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:19:57.794 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:57.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:57.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:19:57.896 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:57.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:57.927 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:57.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:58.028 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:19:58.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:58.088 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:58.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:19:58.189 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:19:58.226 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:58.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:58.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:19:58.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:58 smithi150 ceph-mon[29276]: pgmap v567: 148 pgs: 148 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 776 op/s 2024-09-11T08:19:58.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:58 smithi150 ceph-mon[39612]: pgmap v567: 148 pgs: 148 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 776 op/s 2024-09-11T08:19:58.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:58 smithi181 ceph-mon[32701]: pgmap v567: 148 pgs: 148 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 776 op/s 2024-09-11T08:19:58.327 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:19:58.363 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:58.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:58.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:19:58.464 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:19:58.493 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:58.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:58.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:19:58.537 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:19:58.567 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:19:58.595 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:19:58.630 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:58.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:58.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:19:58.708 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:19:58 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:19:58] "GET /metrics HTTP/1.1" 200 224519 "" "Prometheus/2.43.0" 2024-09-11T08:19:58.732 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:19:58.767 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:58.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:58.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:19:58.823 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:19:58 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:19:58.870 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:19:58.903 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:58.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:58.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:19:59.005 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:19:59.043 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:59.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:59.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:19:59.145 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:19:59.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:59.182 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:59.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:19:59.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:19:59 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:19:59.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:19:59 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:19:59.283 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:59.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:19:59 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:19:59.328 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:59.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:59.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:59.398 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-09-11T08:19:59.398 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:19:59.431 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:19:59.460 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:59.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:59.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:19:59.562 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:19:59.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:59.601 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:19:59.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:19:59.702 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:19:59.738 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:59.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:59.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:19:59.840 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:19:59.876 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:19:59.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:19:59.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:19:59.978 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:00.018 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:00.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:00.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:00.119 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:00.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:00.155 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:00.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:00.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:00 smithi150 ceph-mon[39612]: pgmap v568: 148 pgs: 148 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.58k op/s 2024-09-11T08:20:00.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:00 smithi150 ceph-mon[39612]: overall HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-09-11T08:20:00.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:00 smithi150 ceph-mon[29276]: pgmap v568: 148 pgs: 148 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.58k op/s 2024-09-11T08:20:00.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:00 smithi150 ceph-mon[29276]: overall HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-09-11T08:20:00.255 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:20:00.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:00.293 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:00.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:20:00.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:00 smithi181 ceph-mon[32701]: pgmap v568: 148 pgs: 148 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.58k op/s 2024-09-11T08:20:00.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:00 smithi181 ceph-mon[32701]: overall HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-09-11T08:20:00.394 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:20:00.430 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:00.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:00.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:20:00.532 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:00.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:00.567 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:00.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:00.669 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:20:00.700 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:00.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:00.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:20:00.802 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:20:00.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:00.838 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:00.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:20:00.939 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:00.974 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:00.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:00.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:01.076 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:20:01.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:01.110 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:01.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:20:01.212 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:01.241 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:01.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:01.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:01.249 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-11T08:20:01.343 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:01.378 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:01.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:01.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:01.481 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:01.512 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:01.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:01.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:01.614 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:20:01.636 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:01.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:01.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:20:01.739 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:20:01.770 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:01.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:01.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:20:01.872 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:01.910 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:01.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:01.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:01.928 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:01 smithi150 ceph-mon[29276]: pgmap v569: 148 pgs: 148 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.54k op/s 2024-09-11T08:20:01.928 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:01 smithi150 ceph-mon[39612]: pgmap v569: 148 pgs: 148 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.54k op/s 2024-09-11T08:20:02.012 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:02.044 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:02.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:02.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:02.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:01 smithi181 ceph-mon[32701]: pgmap v569: 148 pgs: 148 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.54k op/s 2024-09-11T08:20:02.146 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:02.181 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:02.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:02.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:02.283 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:02.317 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:02.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:02.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:02.419 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:20:02.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:02.458 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:02.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:20:02.559 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:02.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:02.589 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:02.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:02.690 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:02.721 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:20:02.724 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-09-11T08:20:02.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:02.727 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:02.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:02.828 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:02.849 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:02.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:02.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:02.952 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:20:02.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:02.990 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:02.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:20:03.091 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:03.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:03.133 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:03.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:03.234 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:03.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:03.267 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:03.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:03.368 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:20:03.403 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:03.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:03.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:20:03.505 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:20:03.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:03.538 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:03.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:20:03.639 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:03.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:03.674 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:03.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:03.775 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:03.805 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:03.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:03.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:03.907 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:20:03.945 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:03.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:03.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:20:04.046 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:04.069 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:04.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:04.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:04.170 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:04.207 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:04.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:04.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:04.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:04 smithi150 ceph-mon[29276]: from='client.25376 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:20:04.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:04 smithi150 ceph-mon[29276]: pgmap v570: 148 pgs: 148 active+clean; 5.8 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.28k op/s 2024-09-11T08:20:04.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:04 smithi150 ceph-mon[39612]: from='client.25376 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:20:04.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:04 smithi150 ceph-mon[39612]: pgmap v570: 148 pgs: 148 active+clean; 5.8 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.28k op/s 2024-09-11T08:20:04.309 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:04.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:04 smithi181 ceph-mon[32701]: from='client.25376 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:20:04.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:04 smithi181 ceph-mon[32701]: pgmap v570: 148 pgs: 148 active+clean; 5.8 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.28k op/s 2024-09-11T08:20:04.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:04.341 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:04.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:04.442 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:04.476 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:04.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:04.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:04.577 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:20:04.627 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:04.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:04.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:20:04.729 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:20:04.764 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:04.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:04.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:20:04.865 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:04.900 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:04.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:04.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:05.004 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:05.042 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:05.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:05.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:05.144 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:05.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:05.173 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:05.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:05.274 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:05.304 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:05.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:05.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:05.406 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:05.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:05.442 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:05.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:05.543 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:05.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:20:05 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:20:05] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:20:05.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:05.580 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:05.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:05.641 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":569,"stamp":"2024-09-11T08:20:00.957052+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6370236660,"num_objects":97390,"num_object_clones":0,"num_object_copies":292170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97390,"num_whiteouts":0,"num_read":1591,"num_read_kb":1450,"num_write":134843,"num_write_kb":7400264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16435,"num_bytes_recovered":1064311439,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86939,"ondisk_log_size":86939,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":452,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":21992724,"kb_used_data":19522836,"kb_used_omap":19,"kb_used_meta":2469612,"kb_avail":727968492,"statfs":{"total":767960285184,"available":745439735808,"internally_reserved":0,"allocated":19991384064,"data_stored":19988475916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19955,"internal_metadata":2528883213},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-828899328,"num_objects":-12648,"num_object_clones":0,"num_object_copies":-37944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-12648,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":12648,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.203483"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":301,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.771694+0000","last_change":"2024-09-11T08:19:29.169750+0000","last_active":"2024-09-11T08:19:52.771694+0000","last_peered":"2024-09-11T08:19:52.771694+0000","last_clean":"2024-09-11T08:19:52.771694+0000","last_became_active":"2024-09-11T08:19:29.169363+0000","last_became_peered":"2024-09-11T08:19:29.169363+0000","last_unstale":"2024-09-11T08:19:52.771694+0000","last_undegraded":"2024-09-11T08:19:52.771694+0000","last_fullsized":"2024-09-11T08:19:52.771694+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:18.102122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:18:18.102122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0063791840000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"239'15","reported_seq":295,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.753025+0000","last_change":"2024-09-11T08:18:22.028235+0000","last_active":"2024-09-11T08:19:52.753025+0000","last_peered":"2024-09-11T08:19:52.753025+0000","last_clean":"2024-09-11T08:19:52.753025+0000","last_became_active":"2024-09-11T08:07:32.048048+0000","last_became_peered":"2024-09-11T08:07:32.048048+0000","last_unstale":"2024-09-11T08:19:52.753025+0000","last_undegraded":"2024-09-11T08:19:52.753025+0000","last_fullsized":"2024-09-11T08:19:52.753025+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'15","last_scrub_stamp":"2024-09-11T08:18:22.028181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:18:22.028181+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015902539999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":345,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.765290+0000","last_change":"2024-09-11T08:19:29.141302+0000","last_active":"2024-09-11T08:19:52.765290+0000","last_peered":"2024-09-11T08:19:52.765290+0000","last_clean":"2024-09-11T08:19:52.765290+0000","last_became_active":"2024-09-11T08:19:29.140997+0000","last_became_peered":"2024-09-11T08:19:29.140997+0000","last_unstale":"2024-09-11T08:19:52.765290+0000","last_undegraded":"2024-09-11T08:19:52.765290+0000","last_fullsized":"2024-09-11T08:19:52.765290+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:20.487750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:59.531202+0000","last_clean_scrub_stamp":"2024-09-11T08:18:20.487750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0088346129999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":289,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.752989+0000","last_change":"2024-09-11T08:19:30.506229+0000","last_active":"2024-09-11T08:19:52.752989+0000","last_peered":"2024-09-11T08:19:52.752989+0000","last_clean":"2024-09-11T08:19:52.752989+0000","last_became_active":"2024-09-11T08:19:29.147339+0000","last_became_peered":"2024-09-11T08:19:29.147339+0000","last_unstale":"2024-09-11T08:19:52.752989+0000","last_undegraded":"2024-09-11T08:19:52.752989+0000","last_fullsized":"2024-09-11T08:19:52.752989+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:58.142957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:02.470649+0000","last_clean_scrub_stamp":"2024-09-11T08:16:58.142957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001328805,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":293,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.772217+0000","last_change":"2024-09-11T08:17:44.578728+0000","last_active":"2024-09-11T08:19:52.772217+0000","last_peered":"2024-09-11T08:19:52.772217+0000","last_clean":"2024-09-11T08:19:52.772217+0000","last_became_active":"2024-09-11T08:15:56.521104+0000","last_became_peered":"2024-09-11T08:15:56.521104+0000","last_unstale":"2024-09-11T08:19:52.772217+0000","last_undegraded":"2024-09-11T08:19:52.772217+0000","last_fullsized":"2024-09-11T08:19:52.772217+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:44.578639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:17:44.578639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046216080000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":76,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.768224+0000","last_change":"2024-09-11T08:19:10.533897+0000","last_active":"2024-09-11T08:19:52.768224+0000","last_peered":"2024-09-11T08:19:52.768224+0000","last_clean":"2024-09-11T08:19:52.768224+0000","last_became_active":"2024-09-11T08:17:37.550048+0000","last_became_peered":"2024-09-11T08:17:37.550048+0000","last_unstale":"2024-09-11T08:19:52.768224+0000","last_undegraded":"2024-09-11T08:19:52.768224+0000","last_fullsized":"2024-09-11T08:19:52.768224+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:19:10.533821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:19:10.533821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:20:33.086593+0000","scrub_duration":0.0017821180000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"239'34","reported_seq":226,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.779328+0000","last_change":"2024-09-11T08:18:21.226194+0000","last_active":"2024-09-11T08:19:52.779328+0000","last_peered":"2024-09-11T08:19:52.779328+0000","last_clean":"2024-09-11T08:19:52.779328+0000","last_became_active":"2024-09-11T08:17:37.553866+0000","last_became_peered":"2024-09-11T08:17:37.553866+0000","last_unstale":"2024-09-11T08:19:52.779328+0000","last_undegraded":"2024-09-11T08:19:52.779328+0000","last_fullsized":"2024-09-11T08:19:52.779328+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'34","last_scrub_stamp":"2024-09-11T08:18:21.225670+0000","last_deep_scrub":"145'22","last_deep_scrub_stamp":"2024-09-11T08:15:57.478870+0000","last_clean_scrub_stamp":"2024-09-11T08:18:21.225670+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046176129999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":268,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.752929+0000","last_change":"2024-09-11T08:18:36.998586+0000","last_active":"2024-09-11T08:19:52.752929+0000","last_peered":"2024-09-11T08:19:52.752929+0000","last_clean":"2024-09-11T08:19:52.752929+0000","last_became_active":"2024-09-11T08:07:34.052243+0000","last_became_peered":"2024-09-11T08:07:34.052243+0000","last_unstale":"2024-09-11T08:19:52.752929+0000","last_undegraded":"2024-09-11T08:19:52.752929+0000","last_fullsized":"2024-09-11T08:19:52.752929+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:36.998503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:18:36.998503+0000","last_clean_scrub_stamp":"2024-09-11T08:18:36.998503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:20:03.057651+0000","scrub_duration":0.0067563429999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":31,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.760160+0000","last_change":"2024-09-11T08:19:42.490622+0000","last_active":"2024-09-11T08:19:52.760160+0000","last_peered":"2024-09-11T08:19:52.760160+0000","last_clean":"2024-09-11T08:19:52.760160+0000","last_became_active":"2024-09-11T08:19:29.142228+0000","last_became_peered":"2024-09-11T08:19:29.142228+0000","last_unstale":"2024-09-11T08:19:52.760160+0000","last_undegraded":"2024-09-11T08:19:52.760160+0000","last_fullsized":"2024-09-11T08:19:52.760160+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:19:42.490567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:54.878409+0000","last_clean_scrub_stamp":"2024-09-11T08:19:42.490567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:08.029051+0000","scrub_duration":0.001226284,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"239'15","reported_seq":298,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.753270+0000","last_change":"2024-09-11T08:19:25.601839+0000","last_active":"2024-09-11T08:19:52.753270+0000","last_peered":"2024-09-11T08:19:52.753270+0000","last_clean":"2024-09-11T08:19:52.753270+0000","last_became_active":"2024-09-11T08:07:32.044626+0000","last_became_peered":"2024-09-11T08:07:32.044626+0000","last_unstale":"2024-09-11T08:19:52.753270+0000","last_undegraded":"2024-09-11T08:19:52.753270+0000","last_fullsized":"2024-09-11T08:19:52.753270+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-11T08:16:56.115436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:16:56.115436+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0060897449999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":21,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.760005+0000","last_change":"2024-09-11T08:19:29.192823+0000","last_active":"2024-09-11T08:19:52.760005+0000","last_peered":"2024-09-11T08:19:52.760005+0000","last_clean":"2024-09-11T08:19:52.760005+0000","last_became_active":"2024-09-11T08:19:29.192476+0000","last_became_peered":"2024-09-11T08:19:29.192476+0000","last_unstale":"2024-09-11T08:19:52.760005+0000","last_undegraded":"2024-09-11T08:19:52.760005+0000","last_fullsized":"2024-09-11T08:19:52.760005+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:51.520165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:17:51.520165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033416409999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":292,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.768911+0000","last_change":"2024-09-11T08:19:09.244391+0000","last_active":"2024-09-11T08:19:52.768911+0000","last_peered":"2024-09-11T08:19:52.768911+0000","last_clean":"2024-09-11T08:19:52.768911+0000","last_became_active":"2024-09-11T08:14:24.792595+0000","last_became_peered":"2024-09-11T08:14:24.792595+0000","last_unstale":"2024-09-11T08:19:52.768911+0000","last_undegraded":"2024-09-11T08:19:52.768911+0000","last_fullsized":"2024-09-11T08:19:52.768911+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:19:09.244337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:19:09.244337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:20:38.552764+0000","scrub_duration":0.0013304670000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.759920+0000","last_change":"2024-09-11T08:19:29.140825+0000","last_active":"2024-09-11T08:19:52.759920+0000","last_peered":"2024-09-11T08:19:52.759920+0000","last_clean":"2024-09-11T08:19:52.759920+0000","last_became_active":"2024-09-11T08:19:29.140493+0000","last_became_peered":"2024-09-11T08:19:29.140493+0000","last_unstale":"2024-09-11T08:19:52.759920+0000","last_undegraded":"2024-09-11T08:19:52.759920+0000","last_fullsized":"2024-09-11T08:19:52.759920+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:26.321692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:18:26.321692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"239'9","reported_seq":288,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.753032+0000","last_change":"2024-09-11T08:18:21.019771+0000","last_active":"2024-09-11T08:19:52.753032+0000","last_peered":"2024-09-11T08:19:52.753032+0000","last_clean":"2024-09-11T08:19:52.753032+0000","last_became_active":"2024-09-11T08:07:32.048661+0000","last_became_peered":"2024-09-11T08:07:32.048661+0000","last_unstale":"2024-09-11T08:19:52.753032+0000","last_undegraded":"2024-09-11T08:19:52.753032+0000","last_fullsized":"2024-09-11T08:19:52.753032+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'9","last_scrub_stamp":"2024-09-11T08:18:21.019685+0000","last_deep_scrub":"139'7","last_deep_scrub_stamp":"2024-09-11T08:12:24.869318+0000","last_clean_scrub_stamp":"2024-09-11T08:18:21.019685+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.021195169,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":104,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.758301+0000","last_change":"2024-09-11T08:18:19.990097+0000","last_active":"2024-09-11T08:19:52.758301+0000","last_peered":"2024-09-11T08:19:52.758301+0000","last_clean":"2024-09-11T08:19:52.758301+0000","last_became_active":"2024-09-11T08:15:56.526954+0000","last_became_peered":"2024-09-11T08:15:56.526954+0000","last_unstale":"2024-09-11T08:19:52.758301+0000","last_undegraded":"2024-09-11T08:19:52.758301+0000","last_fullsized":"2024-09-11T08:19:52.758301+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:19.990017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:18:19.990017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037464999999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":21,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.759558+0000","last_change":"2024-09-11T08:19:29.143931+0000","last_active":"2024-09-11T08:19:52.759558+0000","last_peered":"2024-09-11T08:19:52.759558+0000","last_clean":"2024-09-11T08:19:52.759558+0000","last_became_active":"2024-09-11T08:19:29.143572+0000","last_became_peered":"2024-09-11T08:19:29.143572+0000","last_unstale":"2024-09-11T08:19:52.759558+0000","last_undegraded":"2024-09-11T08:19:52.759558+0000","last_fullsized":"2024-09-11T08:19:52.759558+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:42.618920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:04.782577+0000","last_clean_scrub_stamp":"2024-09-11T08:17:42.618920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":289,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.753244+0000","last_change":"2024-09-11T08:19:41.441141+0000","last_active":"2024-09-11T08:19:52.753244+0000","last_peered":"2024-09-11T08:19:52.753244+0000","last_clean":"2024-09-11T08:19:52.753244+0000","last_became_active":"2024-09-11T08:19:29.149081+0000","last_became_peered":"2024-09-11T08:19:29.149081+0000","last_unstale":"2024-09-11T08:19:52.753244+0000","last_undegraded":"2024-09-11T08:19:52.753244+0000","last_fullsized":"2024-09-11T08:19:52.753244+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:30.235441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:17:30.235441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028448620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":301,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.772242+0000","last_change":"2024-09-11T08:19:41.668645+0000","last_active":"2024-09-11T08:19:52.772242+0000","last_peered":"2024-09-11T08:19:52.772242+0000","last_clean":"2024-09-11T08:19:52.772242+0000","last_became_active":"2024-09-11T08:15:56.516941+0000","last_became_peered":"2024-09-11T08:15:56.516941+0000","last_unstale":"2024-09-11T08:19:52.772242+0000","last_undegraded":"2024-09-11T08:19:52.772242+0000","last_fullsized":"2024-09-11T08:19:52.772242+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:32.057157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:17:32.057157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042801360000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"239'34","reported_seq":339,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.779968+0000","last_change":"2024-09-11T08:18:23.258845+0000","last_active":"2024-09-11T08:19:52.779968+0000","last_peered":"2024-09-11T08:19:52.779968+0000","last_clean":"2024-09-11T08:19:52.779968+0000","last_became_active":"2024-09-11T08:07:32.046363+0000","last_became_peered":"2024-09-11T08:07:32.046363+0000","last_unstale":"2024-09-11T08:19:52.779968+0000","last_undegraded":"2024-09-11T08:19:52.779968+0000","last_fullsized":"2024-09-11T08:19:52.779968+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'34","last_scrub_stamp":"2024-09-11T08:18:23.258742+0000","last_deep_scrub":"107'22","last_deep_scrub_stamp":"2024-09-11T08:16:04.159211+0000","last_clean_scrub_stamp":"2024-09-11T08:18:23.258742+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0064364280000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":298,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.766078+0000","last_change":"2024-09-11T08:19:28.870267+0000","last_active":"2024-09-11T08:19:52.766078+0000","last_peered":"2024-09-11T08:19:52.766078+0000","last_clean":"2024-09-11T08:19:52.766078+0000","last_became_active":"2024-09-11T08:07:34.055410+0000","last_became_peered":"2024-09-11T08:07:34.055410+0000","last_unstale":"2024-09-11T08:19:52.766078+0000","last_undegraded":"2024-09-11T08:19:52.766078+0000","last_fullsized":"2024-09-11T08:19:52.766078+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:19:28.870211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:19:28.870211+0000","last_clean_scrub_stamp":"2024-09-11T08:19:28.870211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:20:49.866042+0000","scrub_duration":0.001404073,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":86,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.768132+0000","last_change":"2024-09-11T08:19:29.145374+0000","last_active":"2024-09-11T08:19:52.768132+0000","last_peered":"2024-09-11T08:19:52.768132+0000","last_clean":"2024-09-11T08:19:52.768132+0000","last_became_active":"2024-09-11T08:19:29.144780+0000","last_became_peered":"2024-09-11T08:19:29.144780+0000","last_unstale":"2024-09-11T08:19:52.768132+0000","last_undegraded":"2024-09-11T08:19:52.768132+0000","last_fullsized":"2024-09-11T08:19:52.768132+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:30.149736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:18:30.149736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0068267229999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":282,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.753388+0000","last_change":"2024-09-11T08:18:29.124017+0000","last_active":"2024-09-11T08:19:52.753388+0000","last_peered":"2024-09-11T08:19:52.753388+0000","last_clean":"2024-09-11T08:19:52.753388+0000","last_became_active":"2024-09-11T08:07:30.046284+0000","last_became_peered":"2024-09-11T08:07:30.046284+0000","last_unstale":"2024-09-11T08:19:52.753388+0000","last_undegraded":"2024-09-11T08:19:52.753388+0000","last_fullsized":"2024-09-11T08:19:52.753388+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:29.123933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:18:29.123933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026333300000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"239'18","reported_seq":136,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.780031+0000","last_change":"2024-09-11T08:19:35.218539+0000","last_active":"2024-09-11T08:19:52.780031+0000","last_peered":"2024-09-11T08:19:52.780031+0000","last_clean":"2024-09-11T08:19:52.780031+0000","last_became_active":"2024-09-11T08:17:37.553402+0000","last_became_peered":"2024-09-11T08:17:37.553402+0000","last_unstale":"2024-09-11T08:19:52.780031+0000","last_undegraded":"2024-09-11T08:19:52.780031+0000","last_fullsized":"2024-09-11T08:19:52.780031+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'18","last_scrub_stamp":"2024-09-11T08:18:20.235149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:18:20.235149+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002732099,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":278,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.753579+0000","last_change":"2024-09-11T08:18:37.965178+0000","last_active":"2024-09-11T08:19:52.753579+0000","last_peered":"2024-09-11T08:19:52.753579+0000","last_clean":"2024-09-11T08:19:52.753579+0000","last_became_active":"2024-09-11T08:07:34.056528+0000","last_became_peered":"2024-09-11T08:07:34.056528+0000","last_unstale":"2024-09-11T08:19:52.753579+0000","last_undegraded":"2024-09-11T08:19:52.753579+0000","last_fullsized":"2024-09-11T08:19:52.753579+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:37.964413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:10:40.110490+0000","last_clean_scrub_stamp":"2024-09-11T08:18:37.964413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042785210000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":259,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.756758+0000","last_change":"2024-09-11T08:19:29.163145+0000","last_active":"2024-09-11T08:19:52.756758+0000","last_peered":"2024-09-11T08:19:52.756758+0000","last_clean":"2024-09-11T08:19:52.756758+0000","last_became_active":"2024-09-11T08:19:29.149754+0000","last_became_peered":"2024-09-11T08:19:29.149754+0000","last_unstale":"2024-09-11T08:19:52.756758+0000","last_undegraded":"2024-09-11T08:19:52.756758+0000","last_fullsized":"2024-09-11T08:19:52.756758+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-11T08:18:37.045229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:18:37.045229+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032630189999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":292,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.765407+0000","last_change":"2024-09-11T08:18:27.575426+0000","last_active":"2024-09-11T08:19:52.765407+0000","last_peered":"2024-09-11T08:19:52.765407+0000","last_clean":"2024-09-11T08:19:52.765407+0000","last_became_active":"2024-09-11T08:07:30.045932+0000","last_became_peered":"2024-09-11T08:07:30.045932+0000","last_unstale":"2024-09-11T08:19:52.765407+0000","last_undegraded":"2024-09-11T08:19:52.765407+0000","last_fullsized":"2024-09-11T08:19:52.765407+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:27.575291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:18:27.575291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018750450000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"239'20","reported_seq":382,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.770958+0000","last_change":"2024-09-11T08:18:44.371881+0000","last_active":"2024-09-11T08:19:52.770958+0000","last_peered":"2024-09-11T08:19:52.770958+0000","last_clean":"2024-09-11T08:19:52.770958+0000","last_became_active":"2024-09-11T08:15:56.515817+0000","last_became_peered":"2024-09-11T08:15:56.515817+0000","last_unstale":"2024-09-11T08:19:52.770958+0000","last_undegraded":"2024-09-11T08:19:52.770958+0000","last_fullsized":"2024-09-11T08:19:52.770958+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'20","last_scrub_stamp":"2024-09-11T08:18:44.371760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:18:44.371760+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:54.696485+0000","scrub_duration":0.035522666000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"254'12","reported_seq":494,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:53.701115+0000","last_change":"2024-09-11T08:18:34.253955+0000","last_active":"2024-09-11T08:19:53.701115+0000","last_peered":"2024-09-11T08:19:53.701115+0000","last_clean":"2024-09-11T08:19:53.701115+0000","last_became_active":"2024-09-11T08:15:56.525498+0000","last_became_peered":"2024-09-11T08:15:56.525498+0000","last_unstale":"2024-09-11T08:19:53.701115+0000","last_undegraded":"2024-09-11T08:19:53.701115+0000","last_fullsized":"2024-09-11T08:19:53.701115+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'12","last_scrub_stamp":"2024-09-11T08:18:34.253865+0000","last_deep_scrub":"254'12","last_deep_scrub_stamp":"2024-09-11T08:18:34.253865+0000","last_clean_scrub_stamp":"2024-09-11T08:18:34.253865+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010288669,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2","version":"282'8363","reported_seq":8407,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-11T08:19:58.227961+0000","last_change":"2024-09-11T08:18:01.133843+0000","last_active":"2024-09-11T08:19:58.227961+0000","last_peered":"2024-09-11T08:19:58.227961+0000","last_clean":"2024-09-11T08:19:58.227961+0000","last_became_active":"2024-09-11T08:18:01.133542+0000","last_became_peered":"2024-09-11T08:18:01.133542+0000","last_unstale":"2024-09-11T08:19:58.227961+0000","last_undegraded":"2024-09-11T08:19:58.227961+0000","last_fullsized":"2024-09-11T08:19:58.227961+0000","mapping_epoch":246,"log_start":"255'3863","ondisk_log_start":"255'3863","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:17:59.811675+0000","objects_scrubbed":0,"log_size":4500,"log_dups_size":0,"ondisk_log_size":4500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":396951552,"num_objects":6057,"num_object_clones":0,"num_object_copies":18171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8363,"num_write_kb":461440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"239'27","reported_seq":343,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.770281+0000","last_change":"2024-09-11T08:18:24.523456+0000","last_active":"2024-09-11T08:19:52.770281+0000","last_peered":"2024-09-11T08:19:52.770281+0000","last_clean":"2024-09-11T08:19:52.770281+0000","last_became_active":"2024-09-11T08:07:32.054317+0000","last_became_peered":"2024-09-11T08:07:32.054317+0000","last_unstale":"2024-09-11T08:19:52.770281+0000","last_undegraded":"2024-09-11T08:19:52.770281+0000","last_fullsized":"2024-09-11T08:19:52.770281+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'27","last_scrub_stamp":"2024-09-11T08:18:24.523380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:18:24.523380+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0092214930000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":302,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.770451+0000","last_change":"2024-09-11T08:18:19.427778+0000","last_active":"2024-09-11T08:19:52.770451+0000","last_peered":"2024-09-11T08:19:52.770451+0000","last_clean":"2024-09-11T08:19:52.770451+0000","last_became_active":"2024-09-11T08:15:56.521943+0000","last_became_peered":"2024-09-11T08:15:56.521943+0000","last_unstale":"2024-09-11T08:19:52.770451+0000","last_undegraded":"2024-09-11T08:19:52.770451+0000","last_fullsized":"2024-09-11T08:19:52.770451+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:19.427701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:18:19.427701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003273855,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"271'10","reported_seq":463,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:53.701178+0000","last_change":"2024-09-11T08:19:29.165123+0000","last_active":"2024-09-11T08:19:53.701178+0000","last_peered":"2024-09-11T08:19:53.701178+0000","last_clean":"2024-09-11T08:19:53.701178+0000","last_became_active":"2024-09-11T08:19:29.149773+0000","last_became_peered":"2024-09-11T08:19:29.149773+0000","last_unstale":"2024-09-11T08:19:53.701178+0000","last_undegraded":"2024-09-11T08:19:53.701178+0000","last_fullsized":"2024-09-11T08:19:53.701178+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:18:44.817179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:18:44.817179+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:20:02.672361+0000","scrub_duration":0.0022442809999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":21,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.759871+0000","last_change":"2024-09-11T08:19:29.165731+0000","last_active":"2024-09-11T08:19:52.759871+0000","last_peered":"2024-09-11T08:19:52.759871+0000","last_clean":"2024-09-11T08:19:52.759871+0000","last_became_active":"2024-09-11T08:19:29.164735+0000","last_became_peered":"2024-09-11T08:19:29.164735+0000","last_unstale":"2024-09-11T08:19:52.759871+0000","last_undegraded":"2024-09-11T08:19:52.759871+0000","last_fullsized":"2024-09-11T08:19:52.759871+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:47.577941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:06.616332+0000","last_clean_scrub_stamp":"2024-09-11T08:17:47.577941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"282'8331","reported_seq":22307,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-11T08:19:57.879695+0000","last_change":"2024-09-11T08:19:43.799273+0000","last_active":"2024-09-11T08:19:57.879695+0000","last_peered":"2024-09-11T08:19:57.879695+0000","last_clean":"2024-09-11T08:19:57.879695+0000","last_became_active":"2024-09-11T08:19:43.795082+0000","last_became_peered":"2024-09-11T08:19:43.795082+0000","last_unstale":"2024-09-11T08:19:57.879695+0000","last_undegraded":"2024-09-11T08:19:57.879695+0000","last_fullsized":"2024-09-11T08:19:57.879695+0000","mapping_epoch":278,"log_start":"255'3551","ondisk_log_start":"255'3551","created":246,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:17:59.811675+0000","objects_scrubbed":0,"log_size":4780,"log_dups_size":0,"ondisk_log_size":4780,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":395051032,"num_objects":6029,"num_object_clones":0,"num_object_copies":18087,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6029,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8331,"num_write_kb":459457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6940,"num_bytes_recovered":454819840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"239'30","reported_seq":328,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.756483+0000","last_change":"2024-09-11T08:17:37.556001+0000","last_active":"2024-09-11T08:19:52.756483+0000","last_peered":"2024-09-11T08:19:52.756483+0000","last_clean":"2024-09-11T08:19:52.756483+0000","last_became_active":"2024-09-11T08:17:37.555703+0000","last_became_peered":"2024-09-11T08:17:37.555703+0000","last_unstale":"2024-09-11T08:19:52.756483+0000","last_undegraded":"2024-09-11T08:19:52.756483+0000","last_fullsized":"2024-09-11T08:19:52.756483+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-11T08:17:03.650750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:17:03.650750+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001885091,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":262,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.756985+0000","last_change":"2024-09-11T08:18:23.039080+0000","last_active":"2024-09-11T08:19:52.756985+0000","last_peered":"2024-09-11T08:19:52.756985+0000","last_clean":"2024-09-11T08:19:52.756985+0000","last_became_active":"2024-09-11T08:17:37.553579+0000","last_became_peered":"2024-09-11T08:17:37.553579+0000","last_unstale":"2024-09-11T08:19:52.756985+0000","last_undegraded":"2024-09-11T08:19:52.756985+0000","last_fullsized":"2024-09-11T08:19:52.756985+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:23.039004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:18:23.039004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003148269,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":295,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.753515+0000","last_change":"2024-09-11T08:17:48.150197+0000","last_active":"2024-09-11T08:19:52.753515+0000","last_peered":"2024-09-11T08:19:52.753515+0000","last_clean":"2024-09-11T08:19:52.753515+0000","last_became_active":"2024-09-11T08:15:56.522606+0000","last_became_peered":"2024-09-11T08:15:56.522606+0000","last_unstale":"2024-09-11T08:19:52.753515+0000","last_undegraded":"2024-09-11T08:19:52.753515+0000","last_fullsized":"2024-09-11T08:19:52.753515+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:48.150140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:17:48.150140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001616889,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":269,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.753200+0000","last_change":"2024-09-11T08:19:34.501201+0000","last_active":"2024-09-11T08:19:52.753200+0000","last_peered":"2024-09-11T08:19:52.753200+0000","last_clean":"2024-09-11T08:19:52.753200+0000","last_became_active":"2024-09-11T08:07:36.069225+0000","last_became_peered":"2024-09-11T08:07:36.069225+0000","last_unstale":"2024-09-11T08:19:52.753200+0000","last_undegraded":"2024-09-11T08:19:52.753200+0000","last_fullsized":"2024-09-11T08:19:52.753200+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:39.096025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:04.937884+0000","last_clean_scrub_stamp":"2024-09-11T08:17:39.096025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001460954,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"282'8334","reported_seq":8378,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-11T08:19:57.831357+0000","last_change":"2024-09-11T08:18:00.823642+0000","last_active":"2024-09-11T08:19:57.831357+0000","last_peered":"2024-09-11T08:19:57.831357+0000","last_clean":"2024-09-11T08:19:57.831357+0000","last_became_active":"2024-09-11T08:18:00.823223+0000","last_became_peered":"2024-09-11T08:18:00.823223+0000","last_unstale":"2024-09-11T08:19:57.831357+0000","last_undegraded":"2024-09-11T08:19:57.831357+0000","last_fullsized":"2024-09-11T08:19:57.831357+0000","mapping_epoch":246,"log_start":"255'3402","ondisk_log_start":"255'3402","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:17:59.811675+0000","objects_scrubbed":0,"log_size":4932,"log_dups_size":0,"ondisk_log_size":4932,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":397279232,"num_objects":6062,"num_object_clones":0,"num_object_copies":18186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8334,"num_write_kb":460672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"239'28","reported_seq":364,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.753759+0000","last_change":"2024-09-11T08:19:29.163323+0000","last_active":"2024-09-11T08:19:52.753759+0000","last_peered":"2024-09-11T08:19:52.753759+0000","last_clean":"2024-09-11T08:19:52.753759+0000","last_became_active":"2024-09-11T08:19:29.148311+0000","last_became_peered":"2024-09-11T08:19:29.148311+0000","last_unstale":"2024-09-11T08:19:52.753759+0000","last_undegraded":"2024-09-11T08:19:52.753759+0000","last_fullsized":"2024-09-11T08:19:52.753759+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'28","last_scrub_stamp":"2024-09-11T08:17:49.184124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:17:49.184124+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019943040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":281,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.771189+0000","last_change":"2024-09-11T08:17:47.960385+0000","last_active":"2024-09-11T08:19:52.771189+0000","last_peered":"2024-09-11T08:19:52.771189+0000","last_clean":"2024-09-11T08:19:52.771189+0000","last_became_active":"2024-09-11T08:07:30.050484+0000","last_became_peered":"2024-09-11T08:07:30.050484+0000","last_unstale":"2024-09-11T08:19:52.771189+0000","last_undegraded":"2024-09-11T08:19:52.771189+0000","last_fullsized":"2024-09-11T08:19:52.771189+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:47.960329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:47.960329+0000","last_clean_scrub_stamp":"2024-09-11T08:17:47.960329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001320681,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"82'9","reported_seq":449,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:53.702445+0000","last_change":"2024-09-11T08:19:42.519806+0000","last_active":"2024-09-11T08:19:53.702445+0000","last_peered":"2024-09-11T08:19:53.702445+0000","last_clean":"2024-09-11T08:19:53.702445+0000","last_became_active":"2024-09-11T08:07:34.056143+0000","last_became_peered":"2024-09-11T08:07:34.056143+0000","last_unstale":"2024-09-11T08:19:53.702445+0000","last_undegraded":"2024-09-11T08:19:53.702445+0000","last_fullsized":"2024-09-11T08:19:53.702445+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-11T08:19:42.519736+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-09-11T08:13:10.230703+0000","last_clean_scrub_stamp":"2024-09-11T08:19:42.519736+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:20:59.733447+0000","scrub_duration":0.0016558810000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":256,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.757246+0000","last_change":"2024-09-11T08:17:35.867212+0000","last_active":"2024-09-11T08:19:52.757246+0000","last_peered":"2024-09-11T08:19:52.757246+0000","last_clean":"2024-09-11T08:19:52.757246+0000","last_became_active":"2024-09-11T08:14:24.788968+0000","last_became_peered":"2024-09-11T08:14:24.788968+0000","last_unstale":"2024-09-11T08:19:52.757246+0000","last_undegraded":"2024-09-11T08:19:52.757246+0000","last_fullsized":"2024-09-11T08:19:52.757246+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:35.867116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:17:35.867116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021141390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"282'8473","reported_seq":11804,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-11T08:19:58.027570+0000","last_change":"2024-09-11T08:19:30.182877+0000","last_active":"2024-09-11T08:19:58.027570+0000","last_peered":"2024-09-11T08:19:58.027570+0000","last_clean":"2024-09-11T08:19:58.027570+0000","last_became_active":"2024-09-11T08:19:30.147415+0000","last_became_peered":"2024-09-11T08:19:30.147415+0000","last_unstale":"2024-09-11T08:19:58.027570+0000","last_undegraded":"2024-09-11T08:19:58.027570+0000","last_fullsized":"2024-09-11T08:19:58.027570+0000","mapping_epoch":271,"log_start":"253'3106","ondisk_log_start":"253'3106","created":246,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:17:59.811675+0000","objects_scrubbed":0,"log_size":5367,"log_dups_size":0,"ondisk_log_size":5367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403243008,"num_objects":6153,"num_object_clones":0,"num_object_copies":18459,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8473,"num_write_kb":468032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1634,"num_bytes_recovered":107085824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"239'20","reported_seq":360,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.766796+0000","last_change":"2024-09-11T08:19:33.883860+0000","last_active":"2024-09-11T08:19:52.766796+0000","last_peered":"2024-09-11T08:19:52.766796+0000","last_clean":"2024-09-11T08:19:52.766796+0000","last_became_active":"2024-09-11T08:19:29.143486+0000","last_became_peered":"2024-09-11T08:19:29.143486+0000","last_unstale":"2024-09-11T08:19:52.766796+0000","last_undegraded":"2024-09-11T08:19:52.766796+0000","last_fullsized":"2024-09-11T08:19:52.766796+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'18","last_scrub_stamp":"2024-09-11T08:17:37.613023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:17:37.613023+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017108679999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.759939+0000","last_change":"2024-09-11T08:19:29.147626+0000","last_active":"2024-09-11T08:19:52.759939+0000","last_peered":"2024-09-11T08:19:52.759939+0000","last_clean":"2024-09-11T08:19:52.759939+0000","last_became_active":"2024-09-11T08:19:29.143310+0000","last_became_peered":"2024-09-11T08:19:29.143310+0000","last_unstale":"2024-09-11T08:19:52.759939+0000","last_undegraded":"2024-09-11T08:19:52.759939+0000","last_fullsized":"2024-09-11T08:19:52.759939+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:21.290292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:18:21.290292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"271'12","reported_seq":114,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:53.701161+0000","last_change":"2024-09-11T08:19:29.164191+0000","last_active":"2024-09-11T08:19:53.701161+0000","last_peered":"2024-09-11T08:19:53.701161+0000","last_clean":"2024-09-11T08:19:53.701161+0000","last_became_active":"2024-09-11T08:19:29.144614+0000","last_became_peered":"2024-09-11T08:19:29.144614+0000","last_unstale":"2024-09-11T08:19:53.701161+0000","last_undegraded":"2024-09-11T08:19:53.701161+0000","last_fullsized":"2024-09-11T08:19:53.701161+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'10","last_scrub_stamp":"2024-09-11T08:18:35.294580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:18:35.294580+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:58.906158+0000","scrub_duration":0.0052494120000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"66'1","reported_seq":266,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.769027+0000","last_change":"2024-09-11T08:17:40.041132+0000","last_active":"2024-09-11T08:19:52.769027+0000","last_peered":"2024-09-11T08:19:52.769027+0000","last_clean":"2024-09-11T08:19:52.769027+0000","last_became_active":"2024-09-11T08:07:36.072593+0000","last_became_peered":"2024-09-11T08:07:36.072593+0000","last_unstale":"2024-09-11T08:19:52.769027+0000","last_undegraded":"2024-09-11T08:19:52.769027+0000","last_fullsized":"2024-09-11T08:19:52.769027+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-11T08:17:40.041070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:17:40.041070+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016243379999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"282'8243","reported_seq":8287,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-11T08:19:57.904665+0000","last_change":"2024-09-11T08:18:00.826098+0000","last_active":"2024-09-11T08:19:57.904665+0000","last_peered":"2024-09-11T08:19:57.904665+0000","last_clean":"2024-09-11T08:19:57.904665+0000","last_became_active":"2024-09-11T08:18:00.825771+0000","last_became_peered":"2024-09-11T08:18:00.825771+0000","last_unstale":"2024-09-11T08:19:57.904665+0000","last_undegraded":"2024-09-11T08:19:57.904665+0000","last_fullsized":"2024-09-11T08:19:57.904665+0000","mapping_epoch":246,"log_start":"255'3465","ondisk_log_start":"255'3465","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:17:59.811675+0000","objects_scrubbed":0,"log_size":4778,"log_dups_size":0,"ondisk_log_size":4778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402718720,"num_objects":6145,"num_object_clones":0,"num_object_copies":18435,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8243,"num_write_kb":460416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"239'29","reported_seq":449,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.771970+0000","last_change":"2024-09-11T08:19:29.249993+0000","last_active":"2024-09-11T08:19:52.771970+0000","last_peered":"2024-09-11T08:19:52.771970+0000","last_clean":"2024-09-11T08:19:52.771970+0000","last_became_active":"2024-09-11T08:19:29.195995+0000","last_became_peered":"2024-09-11T08:19:29.195995+0000","last_unstale":"2024-09-11T08:19:52.771970+0000","last_undegraded":"2024-09-11T08:19:52.771970+0000","last_fullsized":"2024-09-11T08:19:52.771970+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'29","last_scrub_stamp":"2024-09-11T08:18:34.333462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:18:34.333462+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011867379000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":315,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.766950+0000","last_change":"2024-09-11T08:19:34.924691+0000","last_active":"2024-09-11T08:19:52.766950+0000","last_peered":"2024-09-11T08:19:52.766950+0000","last_clean":"2024-09-11T08:19:52.766950+0000","last_became_active":"2024-09-11T08:19:29.141999+0000","last_became_peered":"2024-09-11T08:19:29.141999+0000","last_unstale":"2024-09-11T08:19:52.766950+0000","last_undegraded":"2024-09-11T08:19:52.766950+0000","last_fullsized":"2024-09-11T08:19:52.766950+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:34.670680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:17:34.670680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042100260000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":317,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.767306+0000","last_change":"2024-09-11T08:19:29.166900+0000","last_active":"2024-09-11T08:19:52.767306+0000","last_peered":"2024-09-11T08:19:52.767306+0000","last_clean":"2024-09-11T08:19:52.767306+0000","last_became_active":"2024-09-11T08:19:29.166609+0000","last_became_peered":"2024-09-11T08:19:29.166609+0000","last_unstale":"2024-09-11T08:19:52.767306+0000","last_undegraded":"2024-09-11T08:19:52.767306+0000","last_fullsized":"2024-09-11T08:19:52.767306+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:38.233219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:18:38.233219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:20:03.324811+0000","scrub_duration":0.0041324580000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":286,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.765223+0000","last_change":"2024-09-11T08:18:21.482471+0000","last_active":"2024-09-11T08:19:52.765223+0000","last_peered":"2024-09-11T08:19:52.765223+0000","last_clean":"2024-09-11T08:19:52.765223+0000","last_became_active":"2024-09-11T08:07:36.631668+0000","last_became_peered":"2024-09-11T08:07:36.631668+0000","last_unstale":"2024-09-11T08:19:52.765223+0000","last_undegraded":"2024-09-11T08:19:52.765223+0000","last_fullsized":"2024-09-11T08:19:52.765223+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:21.482387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:58.517624+0000","last_clean_scrub_stamp":"2024-09-11T08:18:21.482387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047634690000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"282'8185","reported_seq":10002,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-11T08:19:58.653610+0000","last_change":"2024-09-11T08:19:41.632459+0000","last_active":"2024-09-11T08:19:58.653610+0000","last_peered":"2024-09-11T08:19:58.653610+0000","last_clean":"2024-09-11T08:19:58.653610+0000","last_became_active":"2024-09-11T08:19:41.594593+0000","last_became_peered":"2024-09-11T08:19:41.594593+0000","last_unstale":"2024-09-11T08:19:58.653610+0000","last_undegraded":"2024-09-11T08:19:58.653610+0000","last_fullsized":"2024-09-11T08:19:58.653610+0000","mapping_epoch":276,"log_start":"252'2041","ondisk_log_start":"252'2041","created":246,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:17:59.811675+0000","objects_scrubbed":0,"log_size":6144,"log_dups_size":0,"ondisk_log_size":6144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":378732544,"num_objects":5779,"num_object_clones":0,"num_object_copies":17337,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8185,"num_write_kb":446848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":875,"num_bytes_recovered":57344000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"239'51","reported_seq":219,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:53.711462+0000","last_change":"2024-09-11T08:17:50.928002+0000","last_active":"2024-09-11T08:19:53.711462+0000","last_peered":"2024-09-11T08:19:53.711462+0000","last_clean":"2024-09-11T08:19:53.711462+0000","last_became_active":"2024-09-11T08:15:08.775181+0000","last_became_peered":"2024-09-11T08:15:08.775181+0000","last_unstale":"2024-09-11T08:19:53.711462+0000","last_undegraded":"2024-09-11T08:19:53.711462+0000","last_fullsized":"2024-09-11T08:19:53.711462+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'51","last_scrub_stamp":"2024-09-11T08:17:50.927912+0000","last_deep_scrub":"190'39","last_deep_scrub_stamp":"2024-09-11T08:15:52.847968+0000","last_clean_scrub_stamp":"2024-09-11T08:17:50.927912+0000","objects_scrubbed":10,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036015019999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"59'1","reported_seq":291,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.753348+0000","last_change":"2024-09-11T08:18:28.131832+0000","last_active":"2024-09-11T08:19:52.753348+0000","last_peered":"2024-09-11T08:19:52.753348+0000","last_clean":"2024-09-11T08:19:52.753348+0000","last_became_active":"2024-09-11T08:07:30.047513+0000","last_became_peered":"2024-09-11T08:07:30.047513+0000","last_unstale":"2024-09-11T08:19:52.753348+0000","last_undegraded":"2024-09-11T08:19:52.753348+0000","last_fullsized":"2024-09-11T08:19:52.753348+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-11T08:18:28.131735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:18:28.131735+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0055372269999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":25,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.759668+0000","last_change":"2024-09-11T08:19:29.157831+0000","last_active":"2024-09-11T08:19:52.759668+0000","last_peered":"2024-09-11T08:19:52.759668+0000","last_clean":"2024-09-11T08:19:52.759668+0000","last_became_active":"2024-09-11T08:19:29.142111+0000","last_became_peered":"2024-09-11T08:19:29.142111+0000","last_unstale":"2024-09-11T08:19:52.759668+0000","last_undegraded":"2024-09-11T08:19:52.759668+0000","last_fullsized":"2024-09-11T08:19:52.759668+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-11T08:18:35.267235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:18:35.267235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":277,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.768813+0000","last_change":"2024-09-11T08:18:40.347163+0000","last_active":"2024-09-11T08:19:52.768813+0000","last_peered":"2024-09-11T08:19:52.768813+0000","last_clean":"2024-09-11T08:19:52.768813+0000","last_became_active":"2024-09-11T08:07:34.057154+0000","last_became_peered":"2024-09-11T08:07:34.057154+0000","last_unstale":"2024-09-11T08:19:52.768813+0000","last_undegraded":"2024-09-11T08:19:52.768813+0000","last_fullsized":"2024-09-11T08:19:52.768813+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:40.346999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:10:37.957096+0000","last_clean_scrub_stamp":"2024-09-11T08:18:40.346999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:20:06.326118+0000","scrub_duration":0.0055147520000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":303,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.770468+0000","last_change":"2024-09-11T08:19:28.588367+0000","last_active":"2024-09-11T08:19:52.770468+0000","last_peered":"2024-09-11T08:19:52.770468+0000","last_clean":"2024-09-11T08:19:52.770468+0000","last_became_active":"2024-09-11T08:15:56.525099+0000","last_became_peered":"2024-09-11T08:15:56.525099+0000","last_unstale":"2024-09-11T08:19:52.770468+0000","last_undegraded":"2024-09-11T08:19:52.770468+0000","last_fullsized":"2024-09-11T08:19:52.770468+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:19:28.588307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:19:28.588307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:20:43.040114+0000","scrub_duration":0.0018132090000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"282'8388","reported_seq":11463,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-11T08:19:58.030112+0000","last_change":"2024-09-11T08:19:30.146064+0000","last_active":"2024-09-11T08:19:58.030112+0000","last_peered":"2024-09-11T08:19:58.030112+0000","last_clean":"2024-09-11T08:19:58.030112+0000","last_became_active":"2024-09-11T08:19:30.136920+0000","last_became_peered":"2024-09-11T08:19:30.136920+0000","last_unstale":"2024-09-11T08:19:58.030112+0000","last_undegraded":"2024-09-11T08:19:58.030112+0000","last_fullsized":"2024-09-11T08:19:58.030112+0000","mapping_epoch":271,"log_start":"253'3029","ondisk_log_start":"253'3029","created":246,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:17:59.811675+0000","objects_scrubbed":0,"log_size":5359,"log_dups_size":0,"ondisk_log_size":5359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":401080320,"num_objects":6120,"num_object_clones":0,"num_object_copies":18360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8388,"num_write_kb":464256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1507,"num_bytes_recovered":98762752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"239'17","reported_seq":370,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.770362+0000","last_change":"2024-09-11T08:19:29.176403+0000","last_active":"2024-09-11T08:19:52.770362+0000","last_peered":"2024-09-11T08:19:52.770362+0000","last_clean":"2024-09-11T08:19:52.770362+0000","last_became_active":"2024-09-11T08:19:29.161554+0000","last_became_peered":"2024-09-11T08:19:29.161554+0000","last_unstale":"2024-09-11T08:19:52.770362+0000","last_undegraded":"2024-09-11T08:19:52.770362+0000","last_fullsized":"2024-09-11T08:19:52.770362+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'17","last_scrub_stamp":"2024-09-11T08:18:19.498943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:18:19.498943+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018008900000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":295,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.765087+0000","last_change":"2024-09-11T08:18:40.719738+0000","last_active":"2024-09-11T08:19:52.765087+0000","last_peered":"2024-09-11T08:19:52.765087+0000","last_clean":"2024-09-11T08:19:52.765087+0000","last_became_active":"2024-09-11T08:07:30.047014+0000","last_became_peered":"2024-09-11T08:07:30.047014+0000","last_unstale":"2024-09-11T08:19:52.765087+0000","last_undegraded":"2024-09-11T08:19:52.765087+0000","last_fullsized":"2024-09-11T08:19:52.765087+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:40.719536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:18:40.719536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004534344,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"127'2","reported_seq":311,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.779407+0000","last_change":"2024-09-11T08:18:25.216828+0000","last_active":"2024-09-11T08:19:52.779407+0000","last_peered":"2024-09-11T08:19:52.779407+0000","last_clean":"2024-09-11T08:19:52.779407+0000","last_became_active":"2024-09-11T08:07:30.045009+0000","last_became_peered":"2024-09-11T08:07:30.045009+0000","last_unstale":"2024-09-11T08:19:52.779407+0000","last_undegraded":"2024-09-11T08:19:52.779407+0000","last_fullsized":"2024-09-11T08:19:52.779407+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'2","last_scrub_stamp":"2024-09-11T08:18:25.216046+0000","last_deep_scrub":"127'2","last_deep_scrub_stamp":"2024-09-11T08:15:01.938497+0000","last_clean_scrub_stamp":"2024-09-11T08:18:25.216046+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0051161380000000001,"objects_trimmed":0,"snaptrim_duration":0.001859533,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":327,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.782342+0000","last_change":"2024-09-11T08:19:29.145544+0000","last_active":"2024-09-11T08:19:52.782342+0000","last_peered":"2024-09-11T08:19:52.782342+0000","last_clean":"2024-09-11T08:19:52.782342+0000","last_became_active":"2024-09-11T08:19:29.145168+0000","last_became_peered":"2024-09-11T08:19:29.145168+0000","last_unstale":"2024-09-11T08:19:52.782342+0000","last_undegraded":"2024-09-11T08:19:52.782342+0000","last_fullsized":"2024-09-11T08:19:52.782342+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:35.146832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:58.889094+0000","last_clean_scrub_stamp":"2024-09-11T08:18:35.146832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035033009999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":212,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.756827+0000","last_change":"2024-09-11T08:16:58.685525+0000","last_active":"2024-09-11T08:19:52.756827+0000","last_peered":"2024-09-11T08:19:52.756827+0000","last_clean":"2024-09-11T08:19:52.756827+0000","last_became_active":"2024-09-11T08:14:24.792847+0000","last_became_peered":"2024-09-11T08:14:24.792847+0000","last_unstale":"2024-09-11T08:19:52.756827+0000","last_undegraded":"2024-09-11T08:19:52.756827+0000","last_fullsized":"2024-09-11T08:19:52.756827+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:58.685473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:58.796217+0000","last_clean_scrub_stamp":"2024-09-11T08:16:58.685473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016333490000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"282'8352","reported_seq":10222,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-11T08:19:58.639634+0000","last_change":"2024-09-11T08:19:38.803832+0000","last_active":"2024-09-11T08:19:58.639634+0000","last_peered":"2024-09-11T08:19:58.639634+0000","last_clean":"2024-09-11T08:19:58.639634+0000","last_became_active":"2024-09-11T08:19:38.767293+0000","last_became_peered":"2024-09-11T08:19:38.767293+0000","last_unstale":"2024-09-11T08:19:58.639634+0000","last_undegraded":"2024-09-11T08:19:58.639634+0000","last_fullsized":"2024-09-11T08:19:58.639634+0000","mapping_epoch":273,"log_start":"252'2194","ondisk_log_start":"252'2194","created":246,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:17:59.811675+0000","objects_scrubbed":0,"log_size":6158,"log_dups_size":0,"ondisk_log_size":6158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":397934592,"num_objects":6072,"num_object_clones":0,"num_object_copies":18216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8352,"num_write_kb":461568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":897,"num_bytes_recovered":58785792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"239'24","reported_seq":315,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.771042+0000","last_change":"2024-09-11T08:17:44.010492+0000","last_active":"2024-09-11T08:19:52.771042+0000","last_peered":"2024-09-11T08:19:52.771042+0000","last_clean":"2024-09-11T08:19:52.771042+0000","last_became_active":"2024-09-11T08:07:32.055407+0000","last_became_peered":"2024-09-11T08:07:32.055407+0000","last_unstale":"2024-09-11T08:19:52.771042+0000","last_undegraded":"2024-09-11T08:19:52.771042+0000","last_fullsized":"2024-09-11T08:19:52.771042+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'24","last_scrub_stamp":"2024-09-11T08:17:44.010434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:17:44.010434+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041285339999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"59'1","reported_seq":288,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.768564+0000","last_change":"2024-09-11T08:17:46.998145+0000","last_active":"2024-09-11T08:19:52.768564+0000","last_peered":"2024-09-11T08:19:52.768564+0000","last_clean":"2024-09-11T08:19:52.768564+0000","last_became_active":"2024-09-11T08:07:30.050504+0000","last_became_peered":"2024-09-11T08:07:30.050504+0000","last_unstale":"2024-09-11T08:19:52.768564+0000","last_undegraded":"2024-09-11T08:19:52.768564+0000","last_fullsized":"2024-09-11T08:19:52.768564+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-11T08:17:46.998019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:17:46.998019+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015225320000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"224'6","reported_seq":817,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-11T08:19:56.444766+0000","last_change":"2024-09-11T08:19:28.913001+0000","last_active":"2024-09-11T08:19:56.444766+0000","last_peered":"2024-09-11T08:19:56.444766+0000","last_clean":"2024-09-11T08:19:56.444766+0000","last_became_active":"2024-09-11T08:14:24.793048+0000","last_became_peered":"2024-09-11T08:14:24.793048+0000","last_unstale":"2024-09-11T08:19:56.444766+0000","last_undegraded":"2024-09-11T08:19:56.444766+0000","last_fullsized":"2024-09-11T08:19:56.444766+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'6","last_scrub_stamp":"2024-09-11T08:19:28.912944+0000","last_deep_scrub":"224'6","last_deep_scrub_stamp":"2024-09-11T08:16:55.778284+0000","last_clean_scrub_stamp":"2024-09-11T08:19:28.912944+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:20:50.155575+0000","scrub_duration":0.001538764,"objects_trimmed":0,"snaptrim_duration":0.000318821,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":339,"num_read_kb":334,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":21,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.760075+0000","last_change":"2024-09-11T08:19:29.155260+0000","last_active":"2024-09-11T08:19:52.760075+0000","last_peered":"2024-09-11T08:19:52.760075+0000","last_clean":"2024-09-11T08:19:52.760075+0000","last_became_active":"2024-09-11T08:19:29.154711+0000","last_became_peered":"2024-09-11T08:19:29.154711+0000","last_unstale":"2024-09-11T08:19:52.760075+0000","last_undegraded":"2024-09-11T08:19:52.760075+0000","last_fullsized":"2024-09-11T08:19:52.760075+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:24.366953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:18:24.366953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":309,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.766707+0000","last_change":"2024-09-11T08:19:29.838657+0000","last_active":"2024-09-11T08:19:52.766707+0000","last_peered":"2024-09-11T08:19:52.766707+0000","last_clean":"2024-09-11T08:19:52.766707+0000","last_became_active":"2024-09-11T08:19:29.141234+0000","last_became_peered":"2024-09-11T08:19:29.141234+0000","last_unstale":"2024-09-11T08:19:52.766707+0000","last_undegraded":"2024-09-11T08:19:52.766707+0000","last_fullsized":"2024-09-11T08:19:52.766707+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:42.606972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:17:42.606972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020562559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"282'8560","reported_seq":10514,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-11T08:19:58.669327+0000","last_change":"2024-09-11T08:19:39.795970+0000","last_active":"2024-09-11T08:19:58.669327+0000","last_peered":"2024-09-11T08:19:58.669327+0000","last_clean":"2024-09-11T08:19:58.669327+0000","last_became_active":"2024-09-11T08:19:39.774204+0000","last_became_peered":"2024-09-11T08:19:39.774204+0000","last_unstale":"2024-09-11T08:19:58.669327+0000","last_undegraded":"2024-09-11T08:19:58.669327+0000","last_fullsized":"2024-09-11T08:19:58.669327+0000","mapping_epoch":274,"log_start":"252'2412","ondisk_log_start":"252'2412","created":246,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:17:59.811675+0000","objects_scrubbed":0,"log_size":6148,"log_dups_size":0,"ondisk_log_size":6148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400818176,"num_objects":6116,"num_object_clones":0,"num_object_copies":18348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8560,"num_write_kb":469632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":940,"num_bytes_recovered":61603840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"239'33","reported_seq":353,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.767961+0000","last_change":"2024-09-11T08:18:27.136851+0000","last_active":"2024-09-11T08:19:52.767961+0000","last_peered":"2024-09-11T08:19:52.767961+0000","last_clean":"2024-09-11T08:19:52.767961+0000","last_became_active":"2024-09-11T08:14:24.789154+0000","last_became_peered":"2024-09-11T08:14:24.789154+0000","last_unstale":"2024-09-11T08:19:52.767961+0000","last_undegraded":"2024-09-11T08:19:52.767961+0000","last_fullsized":"2024-09-11T08:19:52.767961+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'33","last_scrub_stamp":"2024-09-11T08:18:27.136763+0000","last_deep_scrub":"84'21","last_deep_scrub_stamp":"2024-09-11T08:08:56.705306+0000","last_clean_scrub_stamp":"2024-09-11T08:18:27.136763+0000","objects_scrubbed":7,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0051223700000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.759690+0000","last_change":"2024-09-11T08:19:29.144089+0000","last_active":"2024-09-11T08:19:52.759690+0000","last_peered":"2024-09-11T08:19:52.759690+0000","last_clean":"2024-09-11T08:19:52.759690+0000","last_became_active":"2024-09-11T08:19:29.143505+0000","last_became_peered":"2024-09-11T08:19:29.143505+0000","last_unstale":"2024-09-11T08:19:52.759690+0000","last_undegraded":"2024-09-11T08:19:52.759690+0000","last_fullsized":"2024-09-11T08:19:52.759690+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:45.423258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:06.236119+0000","last_clean_scrub_stamp":"2024-09-11T08:18:45.423258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:20:13.167739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":110,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.756098+0000","last_change":"2024-09-11T08:18:41.048985+0000","last_active":"2024-09-11T08:19:52.756098+0000","last_peered":"2024-09-11T08:19:52.756098+0000","last_clean":"2024-09-11T08:19:52.756098+0000","last_became_active":"2024-09-11T08:15:56.523430+0000","last_became_peered":"2024-09-11T08:15:56.523430+0000","last_unstale":"2024-09-11T08:19:52.756098+0000","last_undegraded":"2024-09-11T08:19:52.756098+0000","last_fullsized":"2024-09-11T08:19:52.756098+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-11T08:18:41.048903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:05:21.231856+0000","last_clean_scrub_stamp":"2024-09-11T08:18:41.048903+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:20:05.808917+0000","scrub_duration":0.003255703,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":114,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.756592+0000","last_change":"2024-09-11T08:19:29.147905+0000","last_active":"2024-09-11T08:19:52.756592+0000","last_peered":"2024-09-11T08:19:52.756592+0000","last_clean":"2024-09-11T08:19:52.756592+0000","last_became_active":"2024-09-11T08:19:29.147576+0000","last_became_peered":"2024-09-11T08:19:29.147576+0000","last_unstale":"2024-09-11T08:19:52.756592+0000","last_undegraded":"2024-09-11T08:19:52.756592+0000","last_fullsized":"2024-09-11T08:19:52.756592+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:40.793611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:51.289872+0000","last_clean_scrub_stamp":"2024-09-11T08:17:40.793611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012304760000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":21,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.760214+0000","last_change":"2024-09-11T08:19:29.152791+0000","last_active":"2024-09-11T08:19:52.760214+0000","last_peered":"2024-09-11T08:19:52.760214+0000","last_clean":"2024-09-11T08:19:52.760214+0000","last_became_active":"2024-09-11T08:19:29.152071+0000","last_became_peered":"2024-09-11T08:19:29.152071+0000","last_unstale":"2024-09-11T08:19:52.760214+0000","last_undegraded":"2024-09-11T08:19:52.760214+0000","last_fullsized":"2024-09-11T08:19:52.760214+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:25.356936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:01.777429+0000","last_clean_scrub_stamp":"2024-09-11T08:18:25.356936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"282'8466","reported_seq":8510,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-11T08:19:57.904745+0000","last_change":"2024-09-11T08:18:00.826408+0000","last_active":"2024-09-11T08:19:57.904745+0000","last_peered":"2024-09-11T08:19:57.904745+0000","last_clean":"2024-09-11T08:19:57.904745+0000","last_became_active":"2024-09-11T08:18:00.826126+0000","last_became_peered":"2024-09-11T08:18:00.826126+0000","last_unstale":"2024-09-11T08:19:57.904745+0000","last_undegraded":"2024-09-11T08:19:57.904745+0000","last_fullsized":"2024-09-11T08:19:57.904745+0000","mapping_epoch":246,"log_start":"255'3632","ondisk_log_start":"255'3632","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:17:59.811675+0000","objects_scrubbed":0,"log_size":4834,"log_dups_size":0,"ondisk_log_size":4834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":408813568,"num_objects":6238,"num_object_clones":0,"num_object_copies":18714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8466,"num_write_kb":470528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"239'24","reported_seq":316,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.753330+0000","last_change":"2024-09-11T08:18:40.895400+0000","last_active":"2024-09-11T08:19:52.753330+0000","last_peered":"2024-09-11T08:19:52.753330+0000","last_clean":"2024-09-11T08:19:52.753330+0000","last_became_active":"2024-09-11T08:07:32.047971+0000","last_became_peered":"2024-09-11T08:07:32.047971+0000","last_unstale":"2024-09-11T08:19:52.753330+0000","last_undegraded":"2024-09-11T08:19:52.753330+0000","last_fullsized":"2024-09-11T08:19:52.753330+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'24","last_scrub_stamp":"2024-09-11T08:18:40.895293+0000","last_deep_scrub":"88'16","last_deep_scrub_stamp":"2024-09-11T08:09:39.635984+0000","last_clean_scrub_stamp":"2024-09-11T08:18:40.895293+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010336458999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":289,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.753873+0000","last_change":"2024-09-11T08:17:52.133078+0000","last_active":"2024-09-11T08:19:52.753873+0000","last_peered":"2024-09-11T08:19:52.753873+0000","last_clean":"2024-09-11T08:19:52.753873+0000","last_became_active":"2024-09-11T08:14:24.789888+0000","last_became_peered":"2024-09-11T08:14:24.789888+0000","last_unstale":"2024-09-11T08:19:52.753873+0000","last_undegraded":"2024-09-11T08:19:52.753873+0000","last_fullsized":"2024-09-11T08:19:52.753873+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:52.133025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:36.417911+0000","last_clean_scrub_stamp":"2024-09-11T08:17:52.133025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012243499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":21,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.760074+0000","last_change":"2024-09-11T08:19:29.154565+0000","last_active":"2024-09-11T08:19:52.760074+0000","last_peered":"2024-09-11T08:19:52.760074+0000","last_clean":"2024-09-11T08:19:52.760074+0000","last_became_active":"2024-09-11T08:19:29.153976+0000","last_became_peered":"2024-09-11T08:19:29.153976+0000","last_unstale":"2024-09-11T08:19:52.760074+0000","last_undegraded":"2024-09-11T08:19:52.760074+0000","last_fullsized":"2024-09-11T08:19:52.760074+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:30.247009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:01.558340+0000","last_clean_scrub_stamp":"2024-09-11T08:18:30.247009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:56.954326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":89,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.767682+0000","last_change":"2024-09-11T08:19:39.590247+0000","last_active":"2024-09-11T08:19:52.767682+0000","last_peered":"2024-09-11T08:19:52.767682+0000","last_clean":"2024-09-11T08:19:52.767682+0000","last_became_active":"2024-09-11T08:19:29.144592+0000","last_became_peered":"2024-09-11T08:19:29.144592+0000","last_unstale":"2024-09-11T08:19:52.767682+0000","last_undegraded":"2024-09-11T08:19:52.767682+0000","last_fullsized":"2024-09-11T08:19:52.767682+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:42.108342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:17:42.108342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001403751,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.0","version":"282'8396","reported_seq":8440,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-11T08:19:58.547285+0000","last_change":"2024-09-11T08:18:01.134013+0000","last_active":"2024-09-11T08:19:58.547285+0000","last_peered":"2024-09-11T08:19:58.547285+0000","last_clean":"2024-09-11T08:19:58.547285+0000","last_became_active":"2024-09-11T08:18:01.133673+0000","last_became_peered":"2024-09-11T08:18:01.133673+0000","last_unstale":"2024-09-11T08:19:58.547285+0000","last_undegraded":"2024-09-11T08:19:58.547285+0000","last_fullsized":"2024-09-11T08:19:58.547285+0000","mapping_epoch":246,"log_start":"255'3662","ondisk_log_start":"255'3662","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:17:59.811675+0000","objects_scrubbed":0,"log_size":4734,"log_dups_size":0,"ondisk_log_size":4734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":396099584,"num_objects":6044,"num_object_clones":0,"num_object_copies":18132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8396,"num_write_kb":462080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"239'39","reported_seq":393,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.755214+0000","last_change":"2024-09-11T08:19:09.625136+0000","last_active":"2024-09-11T08:19:52.755214+0000","last_peered":"2024-09-11T08:19:52.755214+0000","last_clean":"2024-09-11T08:19:52.755214+0000","last_became_active":"2024-09-11T08:19:09.604525+0000","last_became_peered":"2024-09-11T08:19:09.604525+0000","last_unstale":"2024-09-11T08:19:52.755214+0000","last_undegraded":"2024-09-11T08:19:52.755214+0000","last_fullsized":"2024-09-11T08:19:52.755214+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'39","last_scrub_stamp":"2024-09-11T08:18:31.141746+0000","last_deep_scrub":"77'25","last_deep_scrub_stamp":"2024-09-11T08:08:32.458210+0000","last_clean_scrub_stamp":"2024-09-11T08:18:31.141746+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0076700900000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":297,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.767504+0000","last_change":"2024-09-11T08:18:20.101952+0000","last_active":"2024-09-11T08:19:52.767504+0000","last_peered":"2024-09-11T08:19:52.767504+0000","last_clean":"2024-09-11T08:19:52.767504+0000","last_became_active":"2024-09-11T08:07:30.042555+0000","last_became_peered":"2024-09-11T08:07:30.042555+0000","last_unstale":"2024-09-11T08:19:52.767504+0000","last_undegraded":"2024-09-11T08:19:52.767504+0000","last_fullsized":"2024-09-11T08:19:52.767504+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:20.101868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:18:20.101868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017604610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":288,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.766775+0000","last_change":"2024-09-11T08:18:26.543459+0000","last_active":"2024-09-11T08:19:52.766775+0000","last_peered":"2024-09-11T08:19:52.766775+0000","last_clean":"2024-09-11T08:19:52.766775+0000","last_became_active":"2024-09-11T08:07:34.054961+0000","last_became_peered":"2024-09-11T08:07:34.054961+0000","last_unstale":"2024-09-11T08:19:52.766775+0000","last_undegraded":"2024-09-11T08:19:52.766775+0000","last_fullsized":"2024-09-11T08:19:52.766775+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:26.543286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:18:26.543286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0060933999999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":305,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.779909+0000","last_change":"2024-09-11T08:18:44.258085+0000","last_active":"2024-09-11T08:19:52.779909+0000","last_peered":"2024-09-11T08:19:52.779909+0000","last_clean":"2024-09-11T08:19:52.779909+0000","last_became_active":"2024-09-11T08:07:36.071194+0000","last_became_peered":"2024-09-11T08:07:36.071194+0000","last_unstale":"2024-09-11T08:19:52.779909+0000","last_undegraded":"2024-09-11T08:19:52.779909+0000","last_fullsized":"2024-09-11T08:19:52.779909+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:44.257990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:07.930137+0000","last_clean_scrub_stamp":"2024-09-11T08:18:44.257990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:20:00.548386+0000","scrub_duration":0.0049959540000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"282'8229","reported_seq":8276,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-11T08:19:57.870251+0000","last_change":"2024-09-11T08:19:41.617160+0000","last_active":"2024-09-11T08:19:57.870251+0000","last_peered":"2024-09-11T08:19:57.870251+0000","last_clean":"2024-09-11T08:19:57.870251+0000","last_became_active":"2024-09-11T08:18:00.828320+0000","last_became_peered":"2024-09-11T08:18:00.828320+0000","last_unstale":"2024-09-11T08:19:57.870251+0000","last_undegraded":"2024-09-11T08:19:57.870251+0000","last_fullsized":"2024-09-11T08:19:57.870251+0000","mapping_epoch":246,"log_start":"252'1758","ondisk_log_start":"252'1758","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:17:59.811675+0000","objects_scrubbed":0,"log_size":6471,"log_dups_size":0,"ondisk_log_size":6471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":396165120,"num_objects":6045,"num_object_clones":0,"num_object_copies":18135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8229,"num_write_kb":456768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"239'25","reported_seq":308,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.753649+0000","last_change":"2024-09-11T08:19:32.545790+0000","last_active":"2024-09-11T08:19:52.753649+0000","last_peered":"2024-09-11T08:19:52.753649+0000","last_clean":"2024-09-11T08:19:52.753649+0000","last_became_active":"2024-09-11T08:07:32.043314+0000","last_became_peered":"2024-09-11T08:07:32.043314+0000","last_unstale":"2024-09-11T08:19:52.753649+0000","last_undegraded":"2024-09-11T08:19:52.753649+0000","last_fullsized":"2024-09-11T08:19:52.753649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'15","last_scrub_stamp":"2024-09-11T08:17:01.126478+0000","last_deep_scrub":"224'15","last_deep_scrub_stamp":"2024-09-11T08:17:01.126478+0000","last_clean_scrub_stamp":"2024-09-11T08:17:01.126478+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0092237729999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":66,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.772164+0000","last_change":"2024-09-11T08:17:45.979903+0000","last_active":"2024-09-11T08:19:52.772164+0000","last_peered":"2024-09-11T08:19:52.772164+0000","last_clean":"2024-09-11T08:19:52.772164+0000","last_became_active":"2024-09-11T08:17:37.557278+0000","last_became_peered":"2024-09-11T08:17:37.557278+0000","last_unstale":"2024-09-11T08:19:52.772164+0000","last_undegraded":"2024-09-11T08:19:52.772164+0000","last_fullsized":"2024-09-11T08:19:52.772164+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:45.979847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:45.979847+0000","last_clean_scrub_stamp":"2024-09-11T08:17:45.979847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001534947,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"271'14","reported_seq":496,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:53.701745+0000","last_change":"2024-09-11T08:19:29.168426+0000","last_active":"2024-09-11T08:19:53.701745+0000","last_peered":"2024-09-11T08:19:53.701745+0000","last_clean":"2024-09-11T08:19:53.701745+0000","last_became_active":"2024-09-11T08:19:29.153997+0000","last_became_peered":"2024-09-11T08:19:29.153997+0000","last_unstale":"2024-09-11T08:19:53.701745+0000","last_undegraded":"2024-09-11T08:19:53.701745+0000","last_fullsized":"2024-09-11T08:19:53.701745+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'12","last_scrub_stamp":"2024-09-11T08:18:27.225741+0000","last_deep_scrub":"171'12","last_deep_scrub_stamp":"2024-09-11T08:14:56.881299+0000","last_clean_scrub_stamp":"2024-09-11T08:18:27.225741+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028345940000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":266,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.753190+0000","last_change":"2024-09-11T08:17:40.124232+0000","last_active":"2024-09-11T08:19:52.753190+0000","last_peered":"2024-09-11T08:19:52.753190+0000","last_clean":"2024-09-11T08:19:52.753190+0000","last_became_active":"2024-09-11T08:07:36.074684+0000","last_became_peered":"2024-09-11T08:07:36.074684+0000","last_unstale":"2024-09-11T08:19:52.753190+0000","last_undegraded":"2024-09-11T08:19:52.753190+0000","last_fullsized":"2024-09-11T08:19:52.753190+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:40.124172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:14.613532+0000","last_clean_scrub_stamp":"2024-09-11T08:17:40.124172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001238495,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"282'8474","reported_seq":8537,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-11T08:19:58.021097+0000","last_change":"2024-09-11T08:19:30.145830+0000","last_active":"2024-09-11T08:19:58.021097+0000","last_peered":"2024-09-11T08:19:58.021097+0000","last_clean":"2024-09-11T08:19:58.021097+0000","last_became_active":"2024-09-11T08:19:30.134196+0000","last_became_peered":"2024-09-11T08:19:30.134196+0000","last_unstale":"2024-09-11T08:19:58.021097+0000","last_undegraded":"2024-09-11T08:19:58.021097+0000","last_fullsized":"2024-09-11T08:19:58.021097+0000","mapping_epoch":271,"log_start":"253'3020","ondisk_log_start":"253'3020","created":246,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:17:59.811675+0000","objects_scrubbed":0,"log_size":5454,"log_dups_size":0,"ondisk_log_size":5454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":396361728,"num_objects":6048,"num_object_clones":0,"num_object_copies":18144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8474,"num_write_kb":464704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"239'15","reported_seq":304,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.771893+0000","last_change":"2024-09-11T08:18:27.143512+0000","last_active":"2024-09-11T08:19:52.771893+0000","last_peered":"2024-09-11T08:19:52.771893+0000","last_clean":"2024-09-11T08:19:52.771893+0000","last_became_active":"2024-09-11T08:07:32.054885+0000","last_became_peered":"2024-09-11T08:07:32.054885+0000","last_unstale":"2024-09-11T08:19:52.771893+0000","last_undegraded":"2024-09-11T08:19:52.771893+0000","last_fullsized":"2024-09-11T08:19:52.771893+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'15","last_scrub_stamp":"2024-09-11T08:18:27.143303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:18:27.143303+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00417571,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.759849+0000","last_change":"2024-09-11T08:19:29.146357+0000","last_active":"2024-09-11T08:19:52.759849+0000","last_peered":"2024-09-11T08:19:52.759849+0000","last_clean":"2024-09-11T08:19:52.759849+0000","last_became_active":"2024-09-11T08:19:29.143174+0000","last_became_peered":"2024-09-11T08:19:29.143174+0000","last_unstale":"2024-09-11T08:19:52.759849+0000","last_undegraded":"2024-09-11T08:19:52.759849+0000","last_fullsized":"2024-09-11T08:19:52.759849+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:29.234651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:18:29.234651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":307,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.783176+0000","last_change":"2024-09-11T08:19:10.193992+0000","last_active":"2024-09-11T08:19:52.783176+0000","last_peered":"2024-09-11T08:19:52.783176+0000","last_clean":"2024-09-11T08:19:52.783176+0000","last_became_active":"2024-09-11T08:07:34.056606+0000","last_became_peered":"2024-09-11T08:07:34.056606+0000","last_unstale":"2024-09-11T08:19:52.783176+0000","last_undegraded":"2024-09-11T08:19:52.783176+0000","last_fullsized":"2024-09-11T08:19:52.783176+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:19:10.193940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:19:10.193940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:20:31.158855+0000","scrub_duration":0.0015764850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":302,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.768500+0000","last_change":"2024-09-11T08:18:32.278327+0000","last_active":"2024-09-11T08:19:52.768500+0000","last_peered":"2024-09-11T08:19:52.768500+0000","last_clean":"2024-09-11T08:19:52.768500+0000","last_became_active":"2024-09-11T08:15:56.518884+0000","last_became_peered":"2024-09-11T08:15:56.518884+0000","last_unstale":"2024-09-11T08:19:52.768500+0000","last_undegraded":"2024-09-11T08:19:52.768500+0000","last_fullsized":"2024-09-11T08:19:52.768500+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:32.278257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:18:32.278257+0000","last_clean_scrub_stamp":"2024-09-11T08:18:32.278257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.021912361000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"282'8361","reported_seq":13998,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-11T08:19:58.020957+0000","last_change":"2024-09-11T08:19:30.144513+0000","last_active":"2024-09-11T08:19:58.020957+0000","last_peered":"2024-09-11T08:19:58.020957+0000","last_clean":"2024-09-11T08:19:58.020957+0000","last_became_active":"2024-09-11T08:19:30.138233+0000","last_became_peered":"2024-09-11T08:19:30.138233+0000","last_unstale":"2024-09-11T08:19:58.020957+0000","last_undegraded":"2024-09-11T08:19:58.020957+0000","last_fullsized":"2024-09-11T08:19:58.020957+0000","mapping_epoch":271,"log_start":"253'2919","ondisk_log_start":"253'2919","created":246,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:17:59.811675+0000","objects_scrubbed":0,"log_size":5442,"log_dups_size":0,"ondisk_log_size":5442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398393344,"num_objects":6079,"num_object_clones":0,"num_object_copies":18237,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8361,"num_write_kb":462080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2787,"num_bytes_recovered":182648832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"239'35","reported_seq":421,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.767551+0000","last_change":"2024-09-11T08:18:44.368363+0000","last_active":"2024-09-11T08:19:52.767551+0000","last_peered":"2024-09-11T08:19:52.767551+0000","last_clean":"2024-09-11T08:19:52.767551+0000","last_became_active":"2024-09-11T08:15:56.526961+0000","last_became_peered":"2024-09-11T08:15:56.526961+0000","last_unstale":"2024-09-11T08:19:52.767551+0000","last_undegraded":"2024-09-11T08:19:52.767551+0000","last_fullsized":"2024-09-11T08:19:52.767551+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'35","last_scrub_stamp":"2024-09-11T08:18:44.368266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:18:44.368266+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:55.917488+0000","scrub_duration":0.047744310999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":114,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.756618+0000","last_change":"2024-09-11T08:19:29.152635+0000","last_active":"2024-09-11T08:19:52.756618+0000","last_peered":"2024-09-11T08:19:52.756618+0000","last_clean":"2024-09-11T08:19:52.756618+0000","last_became_active":"2024-09-11T08:19:29.152114+0000","last_became_peered":"2024-09-11T08:19:29.152114+0000","last_unstale":"2024-09-11T08:19:52.756618+0000","last_undegraded":"2024-09-11T08:19:52.756618+0000","last_fullsized":"2024-09-11T08:19:52.756618+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:16:46.578575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:44.385372+0000","last_clean_scrub_stamp":"2024-09-11T08:16:46.578575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.016632636999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":320,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.779494+0000","last_change":"2024-09-11T08:19:29.179345+0000","last_active":"2024-09-11T08:19:52.779494+0000","last_peered":"2024-09-11T08:19:52.779494+0000","last_clean":"2024-09-11T08:19:52.779494+0000","last_became_active":"2024-09-11T08:19:29.178705+0000","last_became_peered":"2024-09-11T08:19:29.178705+0000","last_unstale":"2024-09-11T08:19:52.779494+0000","last_undegraded":"2024-09-11T08:19:52.779494+0000","last_fullsized":"2024-09-11T08:19:52.779494+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:31.165253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:18:31.165253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0053270899999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":94,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.756366+0000","last_change":"2024-09-11T08:17:41.786187+0000","last_active":"2024-09-11T08:19:52.756366+0000","last_peered":"2024-09-11T08:19:52.756366+0000","last_clean":"2024-09-11T08:19:52.756366+0000","last_became_active":"2024-09-11T08:15:56.518169+0000","last_became_peered":"2024-09-11T08:15:56.518169+0000","last_unstale":"2024-09-11T08:19:52.756366+0000","last_undegraded":"2024-09-11T08:19:52.756366+0000","last_fullsized":"2024-09-11T08:19:52.756366+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:41.786126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:17:41.786126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015404170000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"282'8348","reported_seq":8395,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-11T08:19:58.546979+0000","last_change":"2024-09-11T08:19:36.950084+0000","last_active":"2024-09-11T08:19:58.546979+0000","last_peered":"2024-09-11T08:19:58.546979+0000","last_clean":"2024-09-11T08:19:58.546979+0000","last_became_active":"2024-09-11T08:18:00.827140+0000","last_became_peered":"2024-09-11T08:18:00.827140+0000","last_unstale":"2024-09-11T08:19:58.546979+0000","last_undegraded":"2024-09-11T08:19:58.546979+0000","last_fullsized":"2024-09-11T08:19:58.546979+0000","mapping_epoch":246,"log_start":"255'3650","ondisk_log_start":"255'3650","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:17:59.811675+0000","objects_scrubbed":0,"log_size":4698,"log_dups_size":0,"ondisk_log_size":4698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":397410304,"num_objects":6064,"num_object_clones":0,"num_object_copies":18192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8348,"num_write_kb":461184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"239'23","reported_seq":317,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.753639+0000","last_change":"2024-09-11T08:18:35.057979+0000","last_active":"2024-09-11T08:19:52.753639+0000","last_peered":"2024-09-11T08:19:52.753639+0000","last_clean":"2024-09-11T08:19:52.753639+0000","last_became_active":"2024-09-11T08:07:32.048164+0000","last_became_peered":"2024-09-11T08:07:32.048164+0000","last_unstale":"2024-09-11T08:19:52.753639+0000","last_undegraded":"2024-09-11T08:19:52.753639+0000","last_fullsized":"2024-09-11T08:19:52.753639+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'23","last_scrub_stamp":"2024-09-11T08:18:35.057874+0000","last_deep_scrub":"145'17","last_deep_scrub_stamp":"2024-09-11T08:13:08.160848+0000","last_clean_scrub_stamp":"2024-09-11T08:18:35.057874+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:20:01.339446+0000","scrub_duration":0.0043775849999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":104,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.756489+0000","last_change":"2024-09-11T08:18:36.008359+0000","last_active":"2024-09-11T08:19:52.756489+0000","last_peered":"2024-09-11T08:19:52.756489+0000","last_clean":"2024-09-11T08:19:52.756489+0000","last_became_active":"2024-09-11T08:15:56.525845+0000","last_became_peered":"2024-09-11T08:15:56.525845+0000","last_unstale":"2024-09-11T08:19:52.756489+0000","last_undegraded":"2024-09-11T08:19:52.756489+0000","last_fullsized":"2024-09-11T08:19:52.756489+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:36.008191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:53.247275+0000","last_clean_scrub_stamp":"2024-09-11T08:18:36.008191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0094233570000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":287,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.779266+0000","last_change":"2024-09-11T08:18:24.261712+0000","last_active":"2024-09-11T08:19:52.779266+0000","last_peered":"2024-09-11T08:19:52.779266+0000","last_clean":"2024-09-11T08:19:52.779266+0000","last_became_active":"2024-09-11T08:07:34.057479+0000","last_became_peered":"2024-09-11T08:07:34.057479+0000","last_unstale":"2024-09-11T08:19:52.779266+0000","last_undegraded":"2024-09-11T08:19:52.779266+0000","last_fullsized":"2024-09-11T08:19:52.779266+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:24.261618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:18:24.261618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010616029000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":303,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.772279+0000","last_change":"2024-09-11T08:18:20.484274+0000","last_active":"2024-09-11T08:19:52.772279+0000","last_peered":"2024-09-11T08:19:52.772279+0000","last_clean":"2024-09-11T08:19:52.772279+0000","last_became_active":"2024-09-11T08:15:56.522624+0000","last_became_peered":"2024-09-11T08:15:56.522624+0000","last_unstale":"2024-09-11T08:19:52.772279+0000","last_undegraded":"2024-09-11T08:19:52.772279+0000","last_fullsized":"2024-09-11T08:19:52.772279+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:20.484184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:18:20.484184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.029691083,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"282'8511","reported_seq":9852,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-11T08:19:58.647033+0000","last_change":"2024-09-11T08:19:39.814176+0000","last_active":"2024-09-11T08:19:58.647033+0000","last_peered":"2024-09-11T08:19:58.647033+0000","last_clean":"2024-09-11T08:19:58.647033+0000","last_became_active":"2024-09-11T08:19:39.780479+0000","last_became_peered":"2024-09-11T08:19:39.780479+0000","last_unstale":"2024-09-11T08:19:58.647033+0000","last_undegraded":"2024-09-11T08:19:58.647033+0000","last_fullsized":"2024-09-11T08:19:58.647033+0000","mapping_epoch":274,"log_start":"252'2298","ondisk_log_start":"252'2298","created":246,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:17:59.811675+0000","objects_scrubbed":0,"log_size":6213,"log_dups_size":0,"ondisk_log_size":6213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402587648,"num_objects":6143,"num_object_clones":0,"num_object_copies":18429,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8511,"num_write_kb":468928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":633,"num_bytes_recovered":41484288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"239'27","reported_seq":37,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.759481+0000","last_change":"2024-09-11T08:19:29.165913+0000","last_active":"2024-09-11T08:19:52.759481+0000","last_peered":"2024-09-11T08:19:52.759481+0000","last_clean":"2024-09-11T08:19:52.759481+0000","last_became_active":"2024-09-11T08:19:29.147000+0000","last_became_peered":"2024-09-11T08:19:29.147000+0000","last_unstale":"2024-09-11T08:19:52.759481+0000","last_undegraded":"2024-09-11T08:19:52.759481+0000","last_fullsized":"2024-09-11T08:19:52.759481+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-11T08:17:33.548509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:17:33.548509+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010207872,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":110,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.757027+0000","last_change":"2024-09-11T08:18:39.012762+0000","last_active":"2024-09-11T08:19:52.757027+0000","last_peered":"2024-09-11T08:19:52.757027+0000","last_clean":"2024-09-11T08:19:52.757027+0000","last_became_active":"2024-09-11T08:15:56.515328+0000","last_became_peered":"2024-09-11T08:15:56.515328+0000","last_unstale":"2024-09-11T08:19:52.757027+0000","last_undegraded":"2024-09-11T08:19:52.757027+0000","last_fullsized":"2024-09-11T08:19:52.757027+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-11T08:18:39.012616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:18:39.012616+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025254959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"209'11","reported_seq":146,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:53.701419+0000","last_change":"2024-09-11T08:17:42.782853+0000","last_active":"2024-09-11T08:19:53.701419+0000","last_peered":"2024-09-11T08:19:53.701419+0000","last_clean":"2024-09-11T08:19:53.701419+0000","last_became_active":"2024-09-11T08:15:56.524513+0000","last_became_peered":"2024-09-11T08:15:56.524513+0000","last_unstale":"2024-09-11T08:19:53.701419+0000","last_undegraded":"2024-09-11T08:19:53.701419+0000","last_fullsized":"2024-09-11T08:19:53.701419+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"209'11","last_scrub_stamp":"2024-09-11T08:17:42.782786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:17:42.782786+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001693901,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":21,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.759770+0000","last_change":"2024-09-11T08:19:29.164149+0000","last_active":"2024-09-11T08:19:52.759770+0000","last_peered":"2024-09-11T08:19:52.759770+0000","last_clean":"2024-09-11T08:19:52.759770+0000","last_became_active":"2024-09-11T08:19:29.163872+0000","last_became_peered":"2024-09-11T08:19:29.163872+0000","last_unstale":"2024-09-11T08:19:52.759770+0000","last_undegraded":"2024-09-11T08:19:52.759770+0000","last_fullsized":"2024-09-11T08:19:52.759770+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:40.294136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:03.484755+0000","last_clean_scrub_stamp":"2024-09-11T08:18:40.294136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:57.919867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":321,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.767306+0000","last_change":"2024-09-11T08:19:29.193674+0000","last_active":"2024-09-11T08:19:52.767306+0000","last_peered":"2024-09-11T08:19:52.767306+0000","last_clean":"2024-09-11T08:19:52.767306+0000","last_became_active":"2024-09-11T08:19:29.193317+0000","last_became_peered":"2024-09-11T08:19:29.193317+0000","last_unstale":"2024-09-11T08:19:52.767306+0000","last_undegraded":"2024-09-11T08:19:52.767306+0000","last_fullsized":"2024-09-11T08:19:52.767306+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:32.166229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:18:32.166229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:57.653790+0000","scrub_duration":0.0052910539999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"239'15","reported_seq":344,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.770870+0000","last_change":"2024-09-11T08:19:08.696550+0000","last_active":"2024-09-11T08:19:52.770870+0000","last_peered":"2024-09-11T08:19:52.770870+0000","last_clean":"2024-09-11T08:19:52.770870+0000","last_became_active":"2024-09-11T08:15:56.526325+0000","last_became_peered":"2024-09-11T08:15:56.526325+0000","last_unstale":"2024-09-11T08:19:52.770870+0000","last_undegraded":"2024-09-11T08:19:52.770870+0000","last_fullsized":"2024-09-11T08:19:52.770870+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'15","last_scrub_stamp":"2024-09-11T08:19:08.696491+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-09-11T08:09:03.418457+0000","last_clean_scrub_stamp":"2024-09-11T08:19:08.696491+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:20:29.531284+0000","scrub_duration":0.0016320600000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":21,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.759958+0000","last_change":"2024-09-11T08:19:29.147871+0000","last_active":"2024-09-11T08:19:52.759958+0000","last_peered":"2024-09-11T08:19:52.759958+0000","last_clean":"2024-09-11T08:19:52.759958+0000","last_became_active":"2024-09-11T08:19:29.143925+0000","last_became_peered":"2024-09-11T08:19:29.143925+0000","last_unstale":"2024-09-11T08:19:52.759958+0000","last_undegraded":"2024-09-11T08:19:52.759958+0000","last_fullsized":"2024-09-11T08:19:52.759958+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:32.284664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:53.548967+0000","last_clean_scrub_stamp":"2024-09-11T08:18:32.284664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:57.836482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":229,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.756876+0000","last_change":"2024-09-11T08:18:29.927008+0000","last_active":"2024-09-11T08:19:52.756876+0000","last_peered":"2024-09-11T08:19:52.756876+0000","last_clean":"2024-09-11T08:19:52.756876+0000","last_became_active":"2024-09-11T08:14:24.790678+0000","last_became_peered":"2024-09-11T08:14:24.790678+0000","last_unstale":"2024-09-11T08:19:52.756876+0000","last_undegraded":"2024-09-11T08:19:52.756876+0000","last_fullsized":"2024-09-11T08:19:52.756876+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:29.926950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:18:29.926950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020333719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"239'19","reported_seq":345,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.771117+0000","last_change":"2024-09-11T08:17:31.038816+0000","last_active":"2024-09-11T08:19:52.771117+0000","last_peered":"2024-09-11T08:19:52.771117+0000","last_clean":"2024-09-11T08:19:52.771117+0000","last_became_active":"2024-09-11T08:15:56.514793+0000","last_became_peered":"2024-09-11T08:15:56.514793+0000","last_unstale":"2024-09-11T08:19:52.771117+0000","last_undegraded":"2024-09-11T08:19:52.771117+0000","last_fullsized":"2024-09-11T08:19:52.771117+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'13","last_scrub_stamp":"2024-09-11T08:17:31.038601+0000","last_deep_scrub":"84'13","last_deep_scrub_stamp":"2024-09-11T08:08:57.835634+0000","last_clean_scrub_stamp":"2024-09-11T08:17:31.038601+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0067181660000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":252,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.756810+0000","last_change":"2024-09-11T08:17:29.790967+0000","last_active":"2024-09-11T08:19:52.756810+0000","last_peered":"2024-09-11T08:19:52.756810+0000","last_clean":"2024-09-11T08:19:52.756810+0000","last_became_active":"2024-09-11T08:14:24.786021+0000","last_became_peered":"2024-09-11T08:14:24.786021+0000","last_unstale":"2024-09-11T08:19:52.756810+0000","last_undegraded":"2024-09-11T08:19:52.756810+0000","last_fullsized":"2024-09-11T08:19:52.756810+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:29.790887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:30.471620+0000","last_clean_scrub_stamp":"2024-09-11T08:17:29.790887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001553074,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":305,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.770761+0000","last_change":"2024-09-11T08:18:18.416464+0000","last_active":"2024-09-11T08:19:52.770761+0000","last_peered":"2024-09-11T08:19:52.770761+0000","last_clean":"2024-09-11T08:19:52.770761+0000","last_became_active":"2024-09-11T08:15:56.525717+0000","last_became_peered":"2024-09-11T08:15:56.525717+0000","last_unstale":"2024-09-11T08:19:52.770761+0000","last_undegraded":"2024-09-11T08:19:52.770761+0000","last_fullsized":"2024-09-11T08:19:52.770761+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:18.416382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:04.370734+0000","last_clean_scrub_stamp":"2024-09-11T08:18:18.416382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031105590000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":305,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.779462+0000","last_change":"2024-09-11T08:19:33.129123+0000","last_active":"2024-09-11T08:19:52.779462+0000","last_peered":"2024-09-11T08:19:52.779462+0000","last_clean":"2024-09-11T08:19:52.779462+0000","last_became_active":"2024-09-11T08:15:56.525120+0000","last_became_peered":"2024-09-11T08:15:56.525120+0000","last_unstale":"2024-09-11T08:19:52.779462+0000","last_undegraded":"2024-09-11T08:19:52.779462+0000","last_fullsized":"2024-09-11T08:19:52.779462+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:22.251118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:54.184372+0000","last_clean_scrub_stamp":"2024-09-11T08:18:22.251118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023643549999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"239'15","reported_seq":31,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.760155+0000","last_change":"2024-09-11T08:19:29.168879+0000","last_active":"2024-09-11T08:19:52.760155+0000","last_peered":"2024-09-11T08:19:52.760155+0000","last_clean":"2024-09-11T08:19:52.760155+0000","last_became_active":"2024-09-11T08:19:29.153740+0000","last_became_peered":"2024-09-11T08:19:29.153740+0000","last_unstale":"2024-09-11T08:19:52.760155+0000","last_undegraded":"2024-09-11T08:19:52.760155+0000","last_fullsized":"2024-09-11T08:19:52.760155+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'15","last_scrub_stamp":"2024-09-11T08:18:44.371100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:18:44.371100+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:20:03.634691+0000","scrub_duration":0.0047860990000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":308,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.769984+0000","last_change":"2024-09-11T08:18:28.168735+0000","last_active":"2024-09-11T08:19:52.769984+0000","last_peered":"2024-09-11T08:19:52.769984+0000","last_clean":"2024-09-11T08:19:52.769984+0000","last_became_active":"2024-09-11T08:15:56.513278+0000","last_became_peered":"2024-09-11T08:15:56.513278+0000","last_unstale":"2024-09-11T08:19:52.769984+0000","last_undegraded":"2024-09-11T08:19:52.769984+0000","last_fullsized":"2024-09-11T08:19:52.769984+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:28.168679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:52.661591+0000","last_clean_scrub_stamp":"2024-09-11T08:18:28.168679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031651969999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"239'13","reported_seq":443,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:53.701522+0000","last_change":"2024-09-11T08:17:37.558961+0000","last_active":"2024-09-11T08:19:53.701522+0000","last_peered":"2024-09-11T08:19:53.701522+0000","last_clean":"2024-09-11T08:19:53.701522+0000","last_became_active":"2024-09-11T08:17:37.558224+0000","last_became_peered":"2024-09-11T08:17:37.558224+0000","last_unstale":"2024-09-11T08:19:53.701522+0000","last_undegraded":"2024-09-11T08:19:53.701522+0000","last_fullsized":"2024-09-11T08:19:53.701522+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'12","last_scrub_stamp":"2024-09-11T08:16:52.592856+0000","last_deep_scrub":"222'12","last_deep_scrub_stamp":"2024-09-11T08:16:52.592856+0000","last_clean_scrub_stamp":"2024-09-11T08:16:52.592856+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012487329999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":286,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.755107+0000","last_change":"2024-09-11T08:19:29.148744+0000","last_active":"2024-09-11T08:19:52.755107+0000","last_peered":"2024-09-11T08:19:52.755107+0000","last_clean":"2024-09-11T08:19:52.755107+0000","last_became_active":"2024-09-11T08:19:29.147682+0000","last_became_peered":"2024-09-11T08:19:29.147682+0000","last_unstale":"2024-09-11T08:19:52.755107+0000","last_undegraded":"2024-09-11T08:19:52.755107+0000","last_fullsized":"2024-09-11T08:19:52.755107+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:53.175994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:53.175994+0000","last_clean_scrub_stamp":"2024-09-11T08:17:53.175994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001353499,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"239'20","reported_seq":144,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.779388+0000","last_change":"2024-09-11T08:19:30.080842+0000","last_active":"2024-09-11T08:19:52.779388+0000","last_peered":"2024-09-11T08:19:52.779388+0000","last_clean":"2024-09-11T08:19:52.779388+0000","last_became_active":"2024-09-11T08:17:37.554357+0000","last_became_peered":"2024-09-11T08:17:37.554357+0000","last_unstale":"2024-09-11T08:19:52.779388+0000","last_undegraded":"2024-09-11T08:19:52.779388+0000","last_fullsized":"2024-09-11T08:19:52.779388+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'20","last_scrub_stamp":"2024-09-11T08:17:53.205922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:17:53.205922+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017765369999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":94,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.757183+0000","last_change":"2024-09-11T08:17:03.708286+0000","last_active":"2024-09-11T08:19:52.757183+0000","last_peered":"2024-09-11T08:19:52.757183+0000","last_clean":"2024-09-11T08:19:52.757183+0000","last_became_active":"2024-09-11T08:15:56.518515+0000","last_became_peered":"2024-09-11T08:15:56.518515+0000","last_unstale":"2024-09-11T08:19:52.757183+0000","last_undegraded":"2024-09-11T08:19:52.757183+0000","last_fullsized":"2024-09-11T08:19:52.757183+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:03.708205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:17:03.708205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039365750000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"271'10","reported_seq":31,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:53.701555+0000","last_change":"2024-09-11T08:19:29.171309+0000","last_active":"2024-09-11T08:19:53.701555+0000","last_peered":"2024-09-11T08:19:53.701555+0000","last_clean":"2024-09-11T08:19:53.701555+0000","last_became_active":"2024-09-11T08:19:29.165183+0000","last_became_peered":"2024-09-11T08:19:29.165183+0000","last_unstale":"2024-09-11T08:19:53.701555+0000","last_undegraded":"2024-09-11T08:19:53.701555+0000","last_fullsized":"2024-09-11T08:19:53.701555+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:17:44.632353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:17:44.632353+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":292,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.770884+0000","last_change":"2024-09-11T08:17:42.030917+0000","last_active":"2024-09-11T08:19:52.770884+0000","last_peered":"2024-09-11T08:19:52.770884+0000","last_clean":"2024-09-11T08:19:52.770884+0000","last_became_active":"2024-09-11T08:15:56.513949+0000","last_became_peered":"2024-09-11T08:15:56.513949+0000","last_unstale":"2024-09-11T08:19:52.770884+0000","last_undegraded":"2024-09-11T08:19:52.770884+0000","last_fullsized":"2024-09-11T08:19:52.770884+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:42.030854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:17:42.030854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017212379999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"239'19","reported_seq":304,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.753121+0000","last_change":"2024-09-11T08:19:24.644764+0000","last_active":"2024-09-11T08:19:52.753121+0000","last_peered":"2024-09-11T08:19:52.753121+0000","last_clean":"2024-09-11T08:19:52.753121+0000","last_became_active":"2024-09-11T08:07:32.047981+0000","last_became_peered":"2024-09-11T08:07:32.047981+0000","last_unstale":"2024-09-11T08:19:52.753121+0000","last_undegraded":"2024-09-11T08:19:52.753121+0000","last_fullsized":"2024-09-11T08:19:52.753121+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-11T08:16:55.089545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:16:55.089545+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037132879999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":302,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.753886+0000","last_change":"2024-09-11T08:19:33.518218+0000","last_active":"2024-09-11T08:19:52.753886+0000","last_peered":"2024-09-11T08:19:52.753886+0000","last_clean":"2024-09-11T08:19:52.753886+0000","last_became_active":"2024-09-11T08:15:56.523574+0000","last_became_peered":"2024-09-11T08:15:56.523574+0000","last_unstale":"2024-09-11T08:19:52.753886+0000","last_undegraded":"2024-09-11T08:19:52.753886+0000","last_fullsized":"2024-09-11T08:19:52.753886+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:31.186045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:17:31.186045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016617730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":311,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.766681+0000","last_change":"2024-09-11T08:19:29.142149+0000","last_active":"2024-09-11T08:19:52.766681+0000","last_peered":"2024-09-11T08:19:52.766681+0000","last_clean":"2024-09-11T08:19:52.766681+0000","last_became_active":"2024-09-11T08:19:29.141857+0000","last_became_peered":"2024-09-11T08:19:29.141857+0000","last_unstale":"2024-09-11T08:19:52.766681+0000","last_undegraded":"2024-09-11T08:19:52.766681+0000","last_fullsized":"2024-09-11T08:19:52.766681+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:41.689207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:18:41.689207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:20:08.660920+0000","scrub_duration":0.0076107090000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":86,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.768317+0000","last_change":"2024-09-11T08:19:29.180337+0000","last_active":"2024-09-11T08:19:52.768317+0000","last_peered":"2024-09-11T08:19:52.768317+0000","last_clean":"2024-09-11T08:19:52.768317+0000","last_became_active":"2024-09-11T08:19:29.180111+0000","last_became_peered":"2024-09-11T08:19:29.180111+0000","last_unstale":"2024-09-11T08:19:52.768317+0000","last_undegraded":"2024-09-11T08:19:52.768317+0000","last_fullsized":"2024-09-11T08:19:52.768317+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:52.925992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:17:52.925992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0087812989999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"239'15","reported_seq":325,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.766876+0000","last_change":"2024-09-11T08:19:40.867691+0000","last_active":"2024-09-11T08:19:52.766876+0000","last_peered":"2024-09-11T08:19:52.766876+0000","last_clean":"2024-09-11T08:19:52.766876+0000","last_became_active":"2024-09-11T08:07:32.049368+0000","last_became_peered":"2024-09-11T08:07:32.049368+0000","last_unstale":"2024-09-11T08:19:52.766876+0000","last_undegraded":"2024-09-11T08:19:52.766876+0000","last_fullsized":"2024-09-11T08:19:52.766876+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'15","last_scrub_stamp":"2024-09-11T08:19:40.867599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:19:40.867599+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:20:41.389946+0000","scrub_duration":0.0023960169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":104,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.756597+0000","last_change":"2024-09-11T08:18:28.949104+0000","last_active":"2024-09-11T08:19:52.756597+0000","last_peered":"2024-09-11T08:19:52.756597+0000","last_clean":"2024-09-11T08:19:52.756597+0000","last_became_active":"2024-09-11T08:15:56.521570+0000","last_became_peered":"2024-09-11T08:15:56.521570+0000","last_unstale":"2024-09-11T08:19:52.756597+0000","last_undegraded":"2024-09-11T08:19:52.756597+0000","last_fullsized":"2024-09-11T08:19:52.756597+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:28.949027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:18:28.949027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030272900000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":66,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.771673+0000","last_change":"2024-09-11T08:17:39.028807+0000","last_active":"2024-09-11T08:19:52.771673+0000","last_peered":"2024-09-11T08:19:52.771673+0000","last_clean":"2024-09-11T08:19:52.771673+0000","last_became_active":"2024-09-11T08:17:37.557645+0000","last_became_peered":"2024-09-11T08:17:37.557645+0000","last_unstale":"2024-09-11T08:19:52.771673+0000","last_undegraded":"2024-09-11T08:19:52.771673+0000","last_fullsized":"2024-09-11T08:19:52.771673+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:39.028741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:01.218475+0000","last_clean_scrub_stamp":"2024-09-11T08:17:39.028741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012042629999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":295,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.768795+0000","last_change":"2024-09-11T08:19:29.198728+0000","last_active":"2024-09-11T08:19:52.768795+0000","last_peered":"2024-09-11T08:19:52.768795+0000","last_clean":"2024-09-11T08:19:52.768795+0000","last_became_active":"2024-09-11T08:19:29.198224+0000","last_became_peered":"2024-09-11T08:19:29.198224+0000","last_unstale":"2024-09-11T08:19:52.768795+0000","last_undegraded":"2024-09-11T08:19:52.768795+0000","last_fullsized":"2024-09-11T08:19:52.768795+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:20.077741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:09.973943+0000","last_clean_scrub_stamp":"2024-09-11T08:18:20.077741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035943939999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"239'19","reported_seq":362,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.772550+0000","last_change":"2024-09-11T08:18:23.532028+0000","last_active":"2024-09-11T08:19:52.772550+0000","last_peered":"2024-09-11T08:19:52.772550+0000","last_clean":"2024-09-11T08:19:52.772550+0000","last_became_active":"2024-09-11T08:15:56.520058+0000","last_became_peered":"2024-09-11T08:15:56.520058+0000","last_unstale":"2024-09-11T08:19:52.772550+0000","last_undegraded":"2024-09-11T08:19:52.772550+0000","last_fullsized":"2024-09-11T08:19:52.772550+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'19","last_scrub_stamp":"2024-09-11T08:18:23.531935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:18:23.531935+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0053374360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.760030+0000","last_change":"2024-09-11T08:19:29.144075+0000","last_active":"2024-09-11T08:19:52.760030+0000","last_peered":"2024-09-11T08:19:52.760030+0000","last_clean":"2024-09-11T08:19:52.760030+0000","last_became_active":"2024-09-11T08:19:29.143511+0000","last_became_peered":"2024-09-11T08:19:29.143511+0000","last_unstale":"2024-09-11T08:19:52.760030+0000","last_undegraded":"2024-09-11T08:19:52.760030+0000","last_fullsized":"2024-09-11T08:19:52.760030+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:22.308183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:18:22.308183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012202396000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":104,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.756177+0000","last_change":"2024-09-11T08:18:30.917483+0000","last_active":"2024-09-11T08:19:52.756177+0000","last_peered":"2024-09-11T08:19:52.756177+0000","last_clean":"2024-09-11T08:19:52.756177+0000","last_became_active":"2024-09-11T08:15:56.523728+0000","last_became_peered":"2024-09-11T08:15:56.523728+0000","last_unstale":"2024-09-11T08:19:52.756177+0000","last_undegraded":"2024-09-11T08:19:52.756177+0000","last_fullsized":"2024-09-11T08:19:52.756177+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:30.917391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:11:34.159053+0000","last_clean_scrub_stamp":"2024-09-11T08:18:30.917391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015403719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":266,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.772443+0000","last_change":"2024-09-11T08:17:46.489320+0000","last_active":"2024-09-11T08:19:52.772443+0000","last_peered":"2024-09-11T08:19:52.772443+0000","last_clean":"2024-09-11T08:19:52.772443+0000","last_became_active":"2024-09-11T08:07:36.633368+0000","last_became_peered":"2024-09-11T08:07:36.633368+0000","last_unstale":"2024-09-11T08:19:52.772443+0000","last_undegraded":"2024-09-11T08:19:52.772443+0000","last_fullsized":"2024-09-11T08:19:52.772443+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:46.489220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:46.489220+0000","last_clean_scrub_stamp":"2024-09-11T08:17:46.489220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015753690000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"239'8","reported_seq":282,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.770173+0000","last_change":"2024-09-11T08:17:50.580924+0000","last_active":"2024-09-11T08:19:52.770173+0000","last_peered":"2024-09-11T08:19:52.770173+0000","last_clean":"2024-09-11T08:19:52.770173+0000","last_became_active":"2024-09-11T08:07:32.045728+0000","last_became_peered":"2024-09-11T08:07:32.045728+0000","last_unstale":"2024-09-11T08:19:52.770173+0000","last_undegraded":"2024-09-11T08:19:52.770173+0000","last_fullsized":"2024-09-11T08:19:52.770173+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'8","last_scrub_stamp":"2024-09-11T08:17:50.580863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:17:50.580863+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001681075,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":301,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.771769+0000","last_change":"2024-09-11T08:19:29.162691+0000","last_active":"2024-09-11T08:19:52.771769+0000","last_peered":"2024-09-11T08:19:52.771769+0000","last_clean":"2024-09-11T08:19:52.771769+0000","last_became_active":"2024-09-11T08:19:29.161558+0000","last_became_peered":"2024-09-11T08:19:29.161558+0000","last_unstale":"2024-09-11T08:19:52.771769+0000","last_undegraded":"2024-09-11T08:19:52.771769+0000","last_fullsized":"2024-09-11T08:19:52.771769+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:48.965582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:17:48.965582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001302138,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":334,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.767331+0000","last_change":"2024-09-11T08:19:42.654620+0000","last_active":"2024-09-11T08:19:52.767331+0000","last_peered":"2024-09-11T08:19:52.767331+0000","last_clean":"2024-09-11T08:19:52.767331+0000","last_became_active":"2024-09-11T08:15:56.528607+0000","last_became_peered":"2024-09-11T08:15:56.528607+0000","last_unstale":"2024-09-11T08:19:52.767331+0000","last_undegraded":"2024-09-11T08:19:52.767331+0000","last_fullsized":"2024-09-11T08:19:52.767331+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:19:42.654538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:19:42.654538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:08.778702+0000","scrub_duration":0.001327426,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":94,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.758289+0000","last_change":"2024-09-11T08:17:31.857979+0000","last_active":"2024-09-11T08:19:52.758289+0000","last_peered":"2024-09-11T08:19:52.758289+0000","last_clean":"2024-09-11T08:19:52.758289+0000","last_became_active":"2024-09-11T08:15:56.521488+0000","last_became_peered":"2024-09-11T08:15:56.521488+0000","last_unstale":"2024-09-11T08:19:52.758289+0000","last_undegraded":"2024-09-11T08:19:52.758289+0000","last_fullsized":"2024-09-11T08:19:52.758289+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:31.857906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:17:31.857906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011557305,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":306,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.765892+0000","last_change":"2024-09-11T08:19:29.153334+0000","last_active":"2024-09-11T08:19:52.765892+0000","last_peered":"2024-09-11T08:19:52.765892+0000","last_clean":"2024-09-11T08:19:52.765892+0000","last_became_active":"2024-09-11T08:19:29.153017+0000","last_became_peered":"2024-09-11T08:19:29.153017+0000","last_unstale":"2024-09-11T08:19:52.765892+0000","last_undegraded":"2024-09-11T08:19:52.765892+0000","last_fullsized":"2024-09-11T08:19:52.765892+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:35.662817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:50.585566+0000","last_clean_scrub_stamp":"2024-09-11T08:18:35.662817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0093949970000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":66,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.769248+0000","last_change":"2024-09-11T08:17:51.950776+0000","last_active":"2024-09-11T08:19:52.769248+0000","last_peered":"2024-09-11T08:19:52.769248+0000","last_clean":"2024-09-11T08:19:52.769248+0000","last_became_active":"2024-09-11T08:17:37.554262+0000","last_became_peered":"2024-09-11T08:17:37.554262+0000","last_unstale":"2024-09-11T08:19:52.769248+0000","last_undegraded":"2024-09-11T08:19:52.769248+0000","last_fullsized":"2024-09-11T08:19:52.769248+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:51.950689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:17:51.950689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014240730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"239'19","reported_seq":374,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.768220+0000","last_change":"2024-09-11T08:19:29.173712+0000","last_active":"2024-09-11T08:19:52.768220+0000","last_peered":"2024-09-11T08:19:52.768220+0000","last_clean":"2024-09-11T08:19:52.768220+0000","last_became_active":"2024-09-11T08:19:29.145221+0000","last_became_peered":"2024-09-11T08:19:29.145221+0000","last_unstale":"2024-09-11T08:19:52.768220+0000","last_undegraded":"2024-09-11T08:19:52.768220+0000","last_fullsized":"2024-09-11T08:19:52.768220+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'19","last_scrub_stamp":"2024-09-11T08:18:45.330873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:18:45.330873+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:20:03.177726+0000","scrub_duration":0.034413234000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":122,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-11T08:19:52.768094+0000","last_change":"2024-09-11T08:18:39.206134+0000","last_active":"2024-09-11T08:19:52.768094+0000","last_peered":"2024-09-11T08:19:52.768094+0000","last_clean":"2024-09-11T08:19:52.768094+0000","last_became_active":"2024-09-11T08:15:08.769133+0000","last_became_peered":"2024-09-11T08:15:08.769133+0000","last_unstale":"2024-09-11T08:19:52.768094+0000","last_undegraded":"2024-09-11T08:19:52.768094+0000","last_fullsized":"2024-09-11T08:19:52.768094+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:39.206016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:18:39.206016+0000","last_clean_scrub_stamp":"2024-09-11T08:18:39.206016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047896750000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":6369640472,"num_objects":97194,"num_object_clones":0,"num_object_copies":291582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97194,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134014,"num_write_kb":7398593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16213,"num_bytes_recovered":1062535168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19878658048,"data_stored":19878641760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86012,"ondisk_log_size":86012,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1125,"num_read_kb":882,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":200,"num_bytes_recovered":4318,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3425,"internal_metadata":0},"log_size":741,"ondisk_log_size":741,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":339,"num_read_kb":334,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":168,"seq":721554505796,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3855204,"kb_used_data":3533532,"kb_used_omap":7,"kb_used_meta":321656,"kb_avail":89889948,"statfs":{"total":95995035648,"available":92047306752,"internally_reserved":0,"allocated":3618336768,"data_stored":3617971987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8027,"internal_metadata":329375909},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:19:27 2024","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.034,"15min":0.89500000000000002},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":4.8540000000000001,"5min":4.8540000000000001,"15min":4.8540000000000001},"last":0.60899999999999999},{"interface":"front","average":{"1min":1.3260000000000001,"5min":0.96799999999999997,"15min":0.98599999999999999},"min":{"1min":0.26000000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":4.4939999999999998,"5min":5.2610000000000001,"15min":5.2610000000000001},"last":0.42399999999999999}]},{"osd":1,"last update":"Wed Sep 11 08:19:27 2024","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":2.3109999999999999,"15min":2.0670000000000002},"min":{"1min":0.25600000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":5.5890000000000004,"5min":42.082999999999998,"15min":42.082999999999998},"last":0.871},{"interface":"front","average":{"1min":1.1299999999999999,"5min":2.089,"15min":2.0600000000000001},"min":{"1min":0.30599999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":2.8900000000000001,"5min":42.151000000000003,"15min":42.151000000000003},"last":0.68999999999999995}]},{"osd":2,"last update":"Wed Sep 11 08:19:27 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":0.91200000000000003,"15min":0.998},"min":{"1min":0.28699999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.444,"5min":6.444,"15min":6.444},"last":0.46100000000000002},{"interface":"front","average":{"1min":1.242,"5min":0.91500000000000004,"15min":0.78500000000000003},"min":{"1min":0.30099999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":6.3460000000000001,"5min":6.3460000000000001,"15min":6.3460000000000001},"last":0.93500000000000005}]},{"osd":3,"last update":"Wed Sep 11 08:19:27 2024","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":1.302,"15min":1.149},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":4.4489999999999998,"5min":21.460000000000001,"15min":21.460000000000001},"last":0.74399999999999999},{"interface":"front","average":{"1min":1.262,"5min":1.371,"15min":1.407},"min":{"1min":0.32300000000000001,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":6.4660000000000002,"5min":21.440000000000001,"15min":21.440000000000001},"last":0.442}]},{"osd":4,"last update":"Wed Sep 11 08:19:27 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.69899999999999995,"15min":0.58299999999999996},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":2.2730000000000001,"5min":4.141,"15min":4.141},"last":0.89900000000000002},{"interface":"front","average":{"1min":0.873,"5min":0.69899999999999995,"15min":0.73499999999999999},"min":{"1min":0.27000000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":3.0179999999999998,"5min":3.157,"15min":3.157},"last":0.72099999999999997}]},{"osd":5,"last update":"Wed Sep 11 08:19:27 2024","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":1.611,"15min":1.069},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":5.2110000000000003,"5min":65.866,"15min":65.866},"last":0.95299999999999996},{"interface":"front","average":{"1min":1.23,"5min":1.5980000000000001,"15min":1.032},"min":{"1min":0.31,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":3.3969999999999998,"5min":66.019000000000005,"15min":66.019000000000005},"last":0.77600000000000002}]},{"osd":6,"last update":"Wed Sep 11 08:19:27 2024","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.73599999999999999,"15min":0.77400000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":3.3740000000000001,"5min":6.0609999999999999,"15min":6.0609999999999999},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.79400000000000004,"15min":0.79100000000000004},"min":{"1min":0.29499999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":2.456,"5min":3.6829999999999998,"15min":3.6829999999999998},"last":0.91700000000000004}]}]},{"osd":6,"up_from":45,"seq":193273528479,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1478736,"kb_used_data":1177160,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":92266416,"statfs":{"total":95995035648,"available":94480809984,"internally_reserved":0,"allocated":1205411840,"data_stored":1205067338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:19:17 2024","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.264,"15min":1.2150000000000001},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":4.399,"5min":22.012,"15min":45.066000000000003},"last":1.994},{"interface":"front","average":{"1min":0.92200000000000004,"5min":1.4570000000000001,"15min":1.2330000000000001},"min":{"1min":0.23699999999999999,"5min":0.22,"15min":0.20599999999999999},"max":{"1min":4.4630000000000001,"5min":21.963999999999999,"15min":43.899000000000001},"last":0.65900000000000003}]},{"osd":1,"last update":"Wed Sep 11 08:19:17 2024","interfaces":[{"interface":"back","average":{"1min":2.1280000000000001,"5min":1.3959999999999999,"15min":1.0980000000000001},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.214},"max":{"1min":13.817,"5min":20.321000000000002,"15min":44.658000000000001},"last":0.23699999999999999},{"interface":"front","average":{"1min":1.911,"5min":1.5429999999999999,"15min":1.208},"min":{"1min":0.30599999999999999,"5min":0.22800000000000001,"15min":0.22600000000000001},"max":{"1min":14.114000000000001,"5min":19.652000000000001,"15min":44.694000000000003},"last":0.371}]},{"osd":2,"last update":"Wed Sep 11 08:19:17 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.84899999999999998,"15min":0.745},"min":{"1min":0.27100000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":2.3330000000000002,"5min":5.9770000000000003,"15min":5.9770000000000003},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.68300000000000005,"5min":1.1819999999999999,"15min":0.95699999999999996},"min":{"1min":0.27000000000000002,"5min":0.26800000000000002,"15min":0.23300000000000001},"max":{"1min":1.9299999999999999,"5min":7.4809999999999999,"15min":13.346},"last":0.45100000000000001}]},{"osd":3,"last update":"Wed Sep 11 08:19:17 2024","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":0.97499999999999998,"15min":1.0760000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.2889999999999997,"5min":4.2889999999999997,"15min":36.061},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.94199999999999995,"5min":1.073,"15min":1.091},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.26800000000000002},"max":{"1min":2.7090000000000001,"5min":5.5720000000000001,"15min":36.093000000000004},"last":0.68500000000000005}]},{"osd":4,"last update":"Wed Sep 11 08:19:17 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.748,"15min":0.51300000000000001},"min":{"1min":0.217,"5min":0.20899999999999999,"15min":0.20200000000000001},"max":{"1min":0.78100000000000003,"5min":4.492,"15min":4.492},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.74199999999999999,"15min":0.54000000000000004},"min":{"1min":0.26300000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.65300000000000002,"5min":4.468,"15min":4.468},"last":0.39000000000000001}]},{"osd":5,"last update":"Wed Sep 11 08:19:17 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.0549999999999999,"15min":0.67100000000000004},"min":{"1min":0.32400000000000001,"5min":0.29199999999999998,"15min":0.27700000000000002},"max":{"1min":10.49,"5min":10.49,"15min":10.49},"last":0.63400000000000001},{"interface":"front","average":{"1min":1.1919999999999999,"5min":0.92600000000000005,"15min":0.61199999999999999},"min":{"1min":0.35199999999999998,"5min":0.28100000000000003,"15min":0.22600000000000001},"max":{"1min":10.468,"5min":10.468,"15min":10.468},"last":0.59699999999999998}]},{"osd":7,"last update":"Wed Sep 11 08:19:26 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.86599999999999999,"15min":1.052},"min":{"1min":0.255,"5min":0.253,"15min":0.253},"max":{"1min":2.9929999999999999,"5min":8.4580000000000002,"15min":8.4580000000000002},"last":0.96899999999999997},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.72699999999999998,"15min":0.84299999999999997},"min":{"1min":0.27100000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.0609999999999999,"5min":3.593,"15min":3.593},"last":0.83999999999999997}]}]},{"osd":5,"up_from":38,"seq":163208757412,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3835536,"kb_used_data":3525704,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":89909616,"statfs":{"total":95995035648,"available":92067446784,"internally_reserved":0,"allocated":3610320896,"data_stored":3609975325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:19:57 2024","interfaces":[{"interface":"back","average":{"1min":5.9480000000000004,"5min":2.0209999999999999,"15min":1.2250000000000001},"min":{"1min":0.22900000000000001,"5min":0.184,"15min":0.184},"max":{"1min":92.924000000000007,"5min":92.924000000000007,"15min":92.924000000000007},"last":0.22900000000000001},{"interface":"front","average":{"1min":5.782,"5min":2.1930000000000001,"15min":1.3029999999999999},"min":{"1min":0.29399999999999998,"5min":0.24399999999999999,"15min":0.20399999999999999},"max":{"1min":93.173000000000002,"5min":93.173000000000002,"15min":93.173000000000002},"last":0.34100000000000003}]},{"osd":1,"last update":"Wed Sep 11 08:19:57 2024","interfaces":[{"interface":"back","average":{"1min":6.2969999999999997,"5min":2.3420000000000001,"15min":1.5169999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":96.899000000000001,"5min":96.899000000000001,"15min":96.899000000000001},"last":0.65500000000000003},{"interface":"front","average":{"1min":5.7329999999999997,"5min":2.234,"15min":1.2989999999999999},"min":{"1min":0.28999999999999998,"5min":0.248,"15min":0.22700000000000001},"max":{"1min":92.337000000000003,"5min":92.337000000000003,"15min":92.337000000000003},"last":0.29799999999999999}]},{"osd":2,"last update":"Wed Sep 11 08:19:57 2024","interfaces":[{"interface":"back","average":{"1min":1.748,"5min":1.2130000000000001,"15min":1.0800000000000001},"min":{"1min":0.25700000000000001,"5min":0.255,"15min":0.255},"max":{"1min":15.622,"5min":15.622,"15min":16.018000000000001},"last":0.438},{"interface":"front","average":{"1min":1.006,"5min":0.89100000000000001,"15min":0.748},"min":{"1min":0.316,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":3.9239999999999999,"5min":7.5250000000000004,"15min":7.5250000000000004},"last":0.36799999999999999}]},{"osd":3,"last update":"Wed Sep 11 08:19:57 2024","interfaces":[{"interface":"back","average":{"1min":5.4930000000000003,"5min":1.8009999999999999,"15min":1.2390000000000001},"min":{"1min":0.34100000000000003,"5min":0.25,"15min":0.25},"max":{"1min":92.245000000000005,"5min":92.245000000000005,"15min":92.245000000000005},"last":0.40200000000000002},{"interface":"front","average":{"1min":5.9539999999999997,"5min":2.161,"15min":1.3580000000000001},"min":{"1min":0.29699999999999999,"5min":0.27700000000000002,"15min":0.251},"max":{"1min":92.274000000000001,"5min":92.274000000000001,"15min":92.274000000000001},"last":0.41299999999999998}]},{"osd":4,"last update":"Wed Sep 11 08:19:57 2024","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":0.78200000000000003,"15min":0.57299999999999995},"min":{"1min":0.28699999999999998,"5min":0.23499999999999999,"15min":0.21299999999999999},"max":{"1min":3.5030000000000001,"5min":3.5979999999999999,"15min":3.5979999999999999},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.69699999999999995,"15min":0.53800000000000003},"min":{"1min":0.28899999999999998,"5min":0.23599999999999999,"15min":0.23100000000000001},"max":{"1min":3.0739999999999998,"5min":4.9210000000000003,"15min":4.9210000000000003},"last":0.32900000000000001}]},{"osd":6,"last update":"Wed Sep 11 08:19:14 2024","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":0.78400000000000003,"15min":0.55300000000000005},"min":{"1min":0.31900000000000001,"5min":0.26900000000000002,"15min":0.23400000000000001},"max":{"1min":7.5679999999999996,"5min":7.5679999999999996,"15min":7.5679999999999996},"last":0.28100000000000003},{"interface":"front","average":{"1min":1.1799999999999999,"5min":0.86599999999999999,"15min":0.61299999999999999},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.26900000000000002},"max":{"1min":3.5470000000000002,"5min":3.9489999999999998,"15min":3.9489999999999998},"last":2.8410000000000002}]},{"osd":7,"last update":"Wed Sep 11 08:19:33 2024","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":0.92800000000000005,"15min":0.86299999999999999},"min":{"1min":0.36299999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":3.5800000000000001,"5min":3.5800000000000001,"15min":3.5800000000000001},"last":1.4550000000000001},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.93799999999999994,"15min":0.82199999999999995},"min":{"1min":0.32600000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":7.5339999999999998,"5min":7.5339999999999998,"15min":7.5339999999999998},"last":3.141}]}]},{"osd":4,"up_from":31,"seq":133143986345,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1924136,"kb_used_data":1614240,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":91821016,"statfs":{"total":95995035648,"available":94024720384,"internally_reserved":0,"allocated":1652981760,"data_stored":1652613037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:19:28 2024","interfaces":[{"interface":"back","average":{"1min":3.0259999999999998,"5min":1.929,"15min":1.4570000000000001},"min":{"1min":0.23400000000000001,"5min":0.222,"15min":0.19},"max":{"1min":36.920000000000002,"5min":60.497999999999998,"15min":60.497999999999998},"last":0.309},{"interface":"front","average":{"1min":2.7709999999999999,"5min":1.7989999999999999,"15min":1.3180000000000001},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.221},"max":{"1min":37.820999999999998,"5min":60.484999999999999,"15min":60.484999999999999},"last":0.42499999999999999}]},{"osd":1,"last update":"Wed Sep 11 08:19:28 2024","interfaces":[{"interface":"back","average":{"1min":2.3149999999999999,"5min":1.302,"15min":0.93000000000000005},"min":{"1min":0.24299999999999999,"5min":0.214,"15min":0.214},"max":{"1min":20.471,"5min":20.471,"15min":20.471},"last":0.47399999999999998},{"interface":"front","average":{"1min":2.1549999999999998,"5min":1.2170000000000001,"15min":0.86299999999999999},"min":{"1min":0.245,"5min":0.23100000000000001,"15min":0.22700000000000001},"max":{"1min":15.744999999999999,"5min":15.744999999999999,"15min":15.744999999999999},"last":0.249}]},{"osd":2,"last update":"Wed Sep 11 08:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.89100000000000001,"15min":0.76900000000000002},"min":{"1min":0.253,"5min":0.245,"15min":0.23400000000000001},"max":{"1min":2.8959999999999999,"5min":7.976,"15min":7.976},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.85299999999999998,"15min":0.77500000000000002},"min":{"1min":0.28899999999999998,"5min":0.26500000000000001,"15min":0.248},"max":{"1min":7.0590000000000002,"5min":7.0590000000000002,"15min":7.0590000000000002},"last":0.29399999999999998}]},{"osd":3,"last update":"Wed Sep 11 08:19:28 2024","interfaces":[{"interface":"back","average":{"1min":1.536,"5min":1.492,"15min":1.226},"min":{"1min":0.30599999999999999,"5min":0.253,"15min":0.253},"max":{"1min":8.7840000000000007,"5min":19.686,"15min":26.884},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.95899999999999996,"5min":1.127,"15min":1.1459999999999999},"min":{"1min":0.30399999999999999,"5min":0.26600000000000001,"15min":0.246},"max":{"1min":5.8460000000000001,"5min":19.882000000000001,"15min":26.783000000000001},"last":0.38500000000000001}]},{"osd":5,"last update":"Wed Sep 11 08:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.64600000000000002,"15min":0.49099999999999999},"min":{"1min":0.26800000000000002,"5min":0.23999999999999999,"15min":0.23499999999999999},"max":{"1min":3.4670000000000001,"5min":3.4670000000000001,"15min":3.4670000000000001},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.76500000000000001,"15min":0.53200000000000003},"min":{"1min":0.29799999999999999,"5min":0.253,"15min":0.24199999999999999},"max":{"1min":3.4159999999999999,"5min":5.2549999999999999,"15min":5.2549999999999999},"last":0.35199999999999998}]},{"osd":6,"last update":"Wed Sep 11 08:19:17 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":1.089,"15min":0.64600000000000002},"min":{"1min":0.26800000000000002,"5min":0.25,"15min":0.23899999999999999},"max":{"1min":1.9399999999999999,"5min":48.808,"15min":48.808},"last":0.439},{"interface":"front","average":{"1min":0.65900000000000003,"5min":1.097,"15min":0.68000000000000005},"min":{"1min":0.32900000000000001,"5min":0.26800000000000002,"15min":0.26200000000000001},"max":{"1min":3.8999999999999999,"5min":48.746000000000002,"15min":48.746000000000002},"last":0.40300000000000002}]},{"osd":7,"last update":"Wed Sep 11 08:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.66900000000000004,"15min":0.54100000000000004},"min":{"1min":0.36699999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.81399999999999995,"5min":5.3449999999999998,"15min":5.3449999999999998},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.72499999999999998,"15min":0.68300000000000005},"min":{"1min":0.33800000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":2.29,"5min":4.7160000000000002,"15min":4.7160000000000002},"last":0.63600000000000001}]}]},{"osd":3,"up_from":25,"seq":107374182574,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3514032,"kb_used_data":3208360,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":90231120,"statfs":{"total":95995035648,"available":92396666880,"internally_reserved":0,"allocated":3285360640,"data_stored":3284983172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:19:15 2024","interfaces":[{"interface":"back","average":{"1min":1.897,"5min":1.996,"15min":1.2809999999999999},"min":{"1min":0.19,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":7.3710000000000004,"5min":60.469999999999999,"15min":60.469999999999999},"last":0.89700000000000002},{"interface":"front","average":{"1min":1.7669999999999999,"5min":1.8460000000000001,"15min":1.3740000000000001},"min":{"1min":0.249,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":8.1229999999999993,"5min":60.488999999999997,"15min":60.488999999999997},"last":0.71799999999999997}]},{"osd":1,"last update":"Wed Sep 11 08:19:15 2024","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.913,"15min":1.2350000000000001},"min":{"1min":0.249,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":2.9449999999999998,"5min":60.578000000000003,"15min":60.578000000000003},"last":0.745},{"interface":"front","average":{"1min":1.198,"5min":2.3410000000000002,"15min":1.4890000000000001},"min":{"1min":0.20499999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":4.4930000000000003,"5min":63.674999999999997,"15min":63.674999999999997},"last":0.52100000000000002}]},{"osd":2,"last update":"Wed Sep 11 08:19:15 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.8340000000000001,"15min":1.3819999999999999},"min":{"1min":0.23599999999999999,"5min":0.22,"15min":0.185},"max":{"1min":4.9889999999999999,"5min":60.645000000000003,"15min":60.645000000000003},"last":0.33800000000000002},{"interface":"front","average":{"1min":1.5509999999999999,"5min":1.8620000000000001,"15min":1.278},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.20599999999999999},"max":{"1min":5.4649999999999999,"5min":60.723999999999997,"15min":60.723999999999997},"last":0.871}]},{"osd":4,"last update":"Wed Sep 11 08:19:23 2024","interfaces":[{"interface":"back","average":{"1min":1.4630000000000001,"5min":1.76,"15min":1.2090000000000001},"min":{"1min":0.29499999999999998,"5min":0.28100000000000003,"15min":0.24299999999999999},"max":{"1min":4.532,"5min":61.290999999999997,"15min":61.290999999999997},"last":0.372},{"interface":"front","average":{"1min":1.482,"5min":1.7010000000000001,"15min":1.131},"min":{"1min":0.38200000000000001,"5min":0.28899999999999998,"15min":0.27200000000000002},"max":{"1min":4.0830000000000002,"5min":60.856000000000002,"15min":60.856000000000002},"last":1.0129999999999999}]},{"osd":5,"last update":"Wed Sep 11 08:19:56 2024","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":2.8180000000000001,"15min":1.52},"min":{"1min":0.39400000000000002,"5min":0.28199999999999997,"15min":0.26500000000000001},"max":{"1min":3.516,"5min":60.914999999999999,"15min":60.914999999999999},"last":3.516},{"interface":"front","average":{"1min":1.3220000000000001,"5min":2.6880000000000002,"15min":1.5760000000000001},"min":{"1min":0.307,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":5.0579999999999998,"5min":60.787999999999997,"15min":60.787999999999997},"last":0.97599999999999998}]},{"osd":6,"last update":"Wed Sep 11 08:19:15 2024","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":2.1760000000000002,"15min":1.3919999999999999},"min":{"1min":0.318,"5min":0.316,"15min":0.316},"max":{"1min":3.0369999999999999,"5min":60.817,"15min":60.817},"last":1.256},{"interface":"front","average":{"1min":1.371,"5min":2.3820000000000001,"15min":1.5489999999999999},"min":{"1min":0.40899999999999997,"5min":0.33100000000000002,"15min":0.307},"max":{"1min":4.556,"5min":61.264000000000003,"15min":61.264000000000003},"last":0.39200000000000002}]},{"osd":7,"last update":"Wed Sep 11 08:19:30 2024","interfaces":[{"interface":"back","average":{"1min":3.0619999999999998,"5min":2.4630000000000001,"15min":2.915},"min":{"1min":0.38700000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":31.071000000000002,"5min":60.847000000000001,"15min":60.847000000000001},"last":3.2440000000000002},{"interface":"front","average":{"1min":3.1949999999999998,"5min":2.4199999999999999,"15min":2.855},"min":{"1min":0.434,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":31.667000000000002,"5min":60.883000000000003,"15min":60.883000000000003},"last":1.1200000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411505,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1693264,"kb_used_data":1387528,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":92051888,"statfs":{"total":95995035648,"available":94261133312,"internally_reserved":0,"allocated":1420828672,"data_stored":1420483099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:19:51 2024","interfaces":[{"interface":"back","average":{"1min":2.4729999999999999,"5min":2.3879999999999999,"15min":1.4419999999999999},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17199999999999999},"max":{"1min":23.629000000000001,"5min":43.539999999999999,"15min":43.539999999999999},"last":0.32700000000000001},{"interface":"front","average":{"1min":4.8440000000000003,"5min":2.907,"15min":1.6619999999999999},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":44.424999999999997,"5min":44.424999999999997,"15min":44.424999999999997},"last":1.369}]},{"osd":1,"last update":"Wed Sep 11 08:19:51 2024","interfaces":[{"interface":"back","average":{"1min":2.8740000000000001,"5min":1.7090000000000001,"15min":1.2250000000000001},"min":{"1min":0.20200000000000001,"5min":0.182,"15min":0.17599999999999999},"max":{"1min":23.748999999999999,"5min":23.748999999999999,"15min":23.748999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":2.4700000000000002,"5min":1.7170000000000001,"15min":1.2290000000000001},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20200000000000001},"max":{"1min":23.513999999999999,"5min":23.513999999999999,"15min":23.513999999999999},"last":0.34100000000000003}]},{"osd":3,"last update":"Wed Sep 11 08:19:11 2024","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":1.2589999999999999,"15min":1.099},"min":{"1min":0.26600000000000001,"5min":0.21299999999999999,"15min":0.188},"max":{"1min":4.3300000000000001,"5min":8.1989999999999998,"15min":8.1989999999999998},"last":1.1639999999999999},{"interface":"front","average":{"1min":0.81999999999999995,"5min":1.1299999999999999,"15min":1.0349999999999999},"min":{"1min":0.245,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":3.3149999999999999,"5min":8.0640000000000001,"15min":8.3219999999999992},"last":1.0529999999999999}]},{"osd":4,"last update":"Wed Sep 11 08:19:34 2024","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.0089999999999999,"15min":0.89500000000000002},"min":{"1min":0.32400000000000001,"5min":0.27900000000000003,"15min":0.25900000000000001},"max":{"1min":5.5970000000000004,"5min":5.5970000000000004,"15min":7.806},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.97599999999999998,"15min":0.91600000000000004},"min":{"1min":0.30099999999999999,"5min":0.27500000000000002,"15min":0.252},"max":{"1min":3.4049999999999998,"5min":5.4740000000000002,"15min":12.129},"last":0.47299999999999998}]},{"osd":5,"last update":"Wed Sep 11 08:19:02 2024","interfaces":[{"interface":"back","average":{"1min":4.1180000000000003,"5min":1.718,"15min":1.1259999999999999},"min":{"1min":0.29599999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":35.963000000000001,"5min":35.963000000000001,"15min":35.963000000000001},"last":0.40799999999999997},{"interface":"front","average":{"1min":4.4249999999999998,"5min":1.889,"15min":1.1699999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28399999999999997},"max":{"1min":35.104999999999997,"5min":35.104999999999997,"15min":35.104999999999997},"last":0.48699999999999999}]},{"osd":6,"last update":"Wed Sep 11 08:19:22 2024","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.875,"15min":0.84699999999999998},"min":{"1min":0.315,"5min":0.31,"15min":0.26100000000000001},"max":{"1min":3.3050000000000002,"5min":4.2539999999999996,"15min":7.2249999999999996},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.89000000000000001,"5min":1.0820000000000001,"15min":0.95599999999999996},"min":{"1min":0.38200000000000001,"5min":0.33200000000000002,"15min":0.32100000000000001},"max":{"1min":3.4279999999999999,"5min":5.4939999999999998,"15min":6.71},"last":0.372}]},{"osd":7,"last update":"Wed Sep 11 08:19:34 2024","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.583,"15min":1.4530000000000001},"min":{"1min":0.36199999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":5.633,"5min":35.274999999999999,"15min":35.274999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":1.161,"5min":1.591,"15min":1.5549999999999999},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":3.6269999999999998,"5min":35.265000000000001,"15min":35.265000000000001},"last":0.443}]}]},{"osd":1,"up_from":13,"seq":55834575031,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3841388,"kb_used_data":3531556,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":89903764,"statfs":{"total":95995035648,"available":92061454336,"internally_reserved":0,"allocated":3616313344,"data_stored":3615939891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:19:21 2024","interfaces":[{"interface":"back","average":{"1min":2.3730000000000002,"5min":1.5309999999999999,"15min":1.204},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":8.6720000000000006,"5min":8.6720000000000006,"15min":32.478000000000002},"last":0.55800000000000005},{"interface":"front","average":{"1min":1.78,"5min":1.8460000000000001,"15min":1.468},"min":{"1min":0.34499999999999997,"5min":0.193,"15min":0.186},"max":{"1min":5.327,"5min":15.351000000000001,"15min":35.459000000000003},"last":0.495}]},{"osd":2,"last update":"Wed Sep 11 08:18:57 2024","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.2170000000000001,"15min":1},"min":{"1min":0.218,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":7.3330000000000002,"5min":7.9699999999999998,"15min":8.3819999999999997},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.66900000000000004,"5min":1.2549999999999999,"15min":1.113},"min":{"1min":0.21099999999999999,"5min":0.192,"15min":0.16300000000000001},"max":{"1min":2.105,"5min":5.8490000000000002,"15min":10.276},"last":1.1140000000000001}]},{"osd":3,"last update":"Wed Sep 11 08:19:05 2024","interfaces":[{"interface":"back","average":{"1min":1.524,"5min":1.766,"15min":1.587},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.2489999999999997,"5min":16.783999999999999,"15min":32.759},"last":0.59699999999999998},{"interface":"front","average":{"1min":2.3149999999999999,"5min":1.5820000000000001,"15min":1.448},"min":{"1min":0.30299999999999999,"5min":0.25900000000000001,"15min":0.22600000000000001},"max":{"1min":9.0060000000000002,"5min":14.973000000000001,"15min":32.444000000000003},"last":0.87}]},{"osd":4,"last update":"Wed Sep 11 08:19:31 2024","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":2.8849999999999998,"15min":1.552},"min":{"1min":0.30499999999999999,"5min":0.26200000000000001,"15min":0.252},"max":{"1min":4.2119999999999997,"5min":110.86499999999999,"15min":110.86499999999999},"last":2.9550000000000001},{"interface":"front","average":{"1min":1.669,"5min":2.75,"15min":1.5049999999999999},"min":{"1min":0.33100000000000002,"5min":0.29499999999999998,"15min":0.248},"max":{"1min":5.1139999999999999,"5min":110.831,"15min":110.831},"last":1.623}]},{"osd":5,"last update":"Wed Sep 11 08:18:57 2024","interfaces":[{"interface":"back","average":{"1min":8.0299999999999994,"5min":2.7410000000000001,"15min":1.544},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":109.208,"5min":109.208,"15min":109.208},"last":2.3220000000000001},{"interface":"front","average":{"1min":8.1649999999999991,"5min":2.9209999999999998,"15min":1.6319999999999999},"min":{"1min":0.435,"5min":0.35199999999999998,"15min":0.28100000000000003},"max":{"1min":111.254,"5min":111.254,"15min":111.254},"last":1.9059999999999999}]},{"osd":6,"last update":"Wed Sep 11 08:19:10 2024","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":2.7149999999999999,"15min":1.5780000000000001},"min":{"1min":0.39500000000000002,"5min":0.30499999999999999,"15min":0.29399999999999998},"max":{"1min":4.5759999999999996,"5min":124.31999999999999,"15min":124.31999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":1.948,"5min":2.5939999999999999,"15min":1.52},"min":{"1min":0.35299999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":5.1399999999999997,"5min":124.958,"15min":124.958},"last":0.61099999999999999}]},{"osd":7,"last update":"Wed Sep 11 08:19:31 2024","interfaces":[{"interface":"back","average":{"1min":7.1909999999999998,"5min":2.5859999999999999,"15min":1.6519999999999999},"min":{"1min":0.33400000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":109.149,"5min":109.149,"15min":109.149},"last":3.6859999999999999},{"interface":"front","average":{"1min":7.2439999999999998,"5min":2.6520000000000001,"15min":1.8700000000000001},"min":{"1min":0.41999999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":109.309,"5min":109.309,"15min":109.309},"last":1.9390000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738555,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1850428,"kb_used_data":1544756,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":91894724,"statfs":{"total":95995035648,"available":94100197376,"internally_reserved":0,"allocated":1581830144,"data_stored":1581442067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 11 08:19:26 2024","interfaces":[{"interface":"back","average":{"1min":1.788,"5min":1.4510000000000001,"15min":1.081},"min":{"1min":0.33000000000000002,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":5.3550000000000004,"5min":10.105,"15min":10.105},"last":0.99199999999999999},{"interface":"front","average":{"1min":1.6970000000000001,"5min":1.9339999999999999,"15min":1.2889999999999999},"min":{"1min":0.26700000000000002,"5min":0.153,"15min":0.153},"max":{"1min":7.8760000000000003,"5min":19.852,"15min":19.852},"last":2.9580000000000002}]},{"osd":2,"last update":"Wed Sep 11 08:19:51 2024","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.4830000000000001,"15min":1.095},"min":{"1min":0.22500000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":5.1970000000000001,"5min":10.025,"15min":10.351000000000001},"last":0.79400000000000004},{"interface":"front","average":{"1min":1.6200000000000001,"5min":1.456,"15min":1.179},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.186},"max":{"1min":5.3659999999999997,"5min":10.074999999999999,"15min":10.074999999999999},"last":1.0229999999999999}]},{"osd":3,"last update":"Wed Sep 11 08:19:10 2024","interfaces":[{"interface":"back","average":{"1min":2.0030000000000001,"5min":1.516,"15min":1.4379999999999999},"min":{"1min":0.251,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":8.4149999999999991,"5min":10.129,"15min":35.491},"last":1.486},{"interface":"front","average":{"1min":1.393,"5min":1.4419999999999999,"15min":1.3819999999999999},"min":{"1min":0.311,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.899,"5min":8.282,"15min":37.563000000000002},"last":3.2269999999999999}]},{"osd":4,"last update":"Wed Sep 11 08:19:34 2024","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.1559999999999999,"15min":1.0009999999999999},"min":{"1min":0.29799999999999999,"5min":0.26900000000000002,"15min":0.24099999999999999},"max":{"1min":4.3410000000000002,"5min":6.0190000000000001,"15min":9.4030000000000005},"last":4.5949999999999998},{"interface":"front","average":{"1min":1.6930000000000001,"5min":1.1739999999999999,"15min":1.0109999999999999},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.24399999999999999},"max":{"1min":8.3510000000000009,"5min":8.3510000000000009,"15min":8.3510000000000009},"last":1.0509999999999999}]},{"osd":5,"last update":"Wed Sep 11 08:18:58 2024","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.5029999999999999,"15min":1.1240000000000001},"min":{"1min":0.32800000000000001,"5min":0.32700000000000001,"15min":0.255},"max":{"1min":4.7229999999999999,"5min":24.914999999999999,"15min":24.914999999999999},"last":1.1120000000000001},{"interface":"front","average":{"1min":1.845,"5min":1.7589999999999999,"15min":1.2270000000000001},"min":{"1min":0.34100000000000003,"5min":0.30399999999999999,"15min":0.27100000000000002},"max":{"1min":13.885,"5min":25.277000000000001,"15min":25.277000000000001},"last":3.1970000000000001}]},{"osd":6,"last update":"Wed Sep 11 08:19:17 2024","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":1.2410000000000001,"15min":1.3500000000000001},"min":{"1min":0.36599999999999999,"5min":0.29599999999999999,"15min":0.29099999999999998},"max":{"1min":4.2670000000000003,"5min":6.0549999999999997,"15min":78.522000000000006},"last":2.996},{"interface":"front","average":{"1min":0.92200000000000004,"5min":1.228,"15min":1.2569999999999999},"min":{"1min":0.30299999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":2.657,"5min":7.4820000000000002,"15min":79.808000000000007},"last":0.82299999999999995}]},{"osd":7,"last update":"Wed Sep 11 08:19:34 2024","interfaces":[{"interface":"back","average":{"1min":1.2250000000000001,"5min":1.375,"15min":1.3660000000000001},"min":{"1min":0.35499999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":4.7839999999999998,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.84399999999999997},{"interface":"front","average":{"1min":1.115,"5min":1.486,"15min":1.3979999999999999},"min":{"1min":0.33600000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":4.8049999999999997,"5min":13.914,"15min":13.914},"last":0.86499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3425,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1579941888,"data_stored":1579941888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3581939712,"data_stored":3581935640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1406599168,"data_stored":1406599168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3267493888,"data_stored":3267493888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1642139648,"data_stored":1642135576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3587903488,"data_stored":3587899416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1198129152,"data_stored":1198129152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3614511104,"data_stored":3614507032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T08:20:05.650 INFO:tasks.thrashosds.thrasher:Setting 6.5 to [2, 4, 7] 2024-09-11T08:20:05.650 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '6.5', '2', '4', '7'] 2024-09-11T08:20:05.650 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd pg-upmap 6.5 2 4 7 2024-09-11T08:20:05.681 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:20:05.730 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:05.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:05.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:20:05.832 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:05.866 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:05.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:05.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:05.968 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:06.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:06.002 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:06.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:06.104 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:06.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:06.136 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:06.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:06.237 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:20:06.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:06 smithi150 ceph-mon[29276]: pgmap v571: 148 pgs: 148 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 915 B/s rd, 0 B/s wr, 1.15k op/s 2024-09-11T08:20:06.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:06 smithi150 ceph-mon[39612]: pgmap v571: 148 pgs: 148 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 915 B/s rd, 0 B/s wr, 1.15k op/s 2024-09-11T08:20:06.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:06.285 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:06.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:20:06.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:06 smithi181 ceph-mon[32701]: pgmap v571: 148 pgs: 148 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 915 B/s rd, 0 B/s wr, 1.15k op/s 2024-09-11T08:20:06.385 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:06.419 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:06.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:06.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:06.521 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:20:06.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:06.559 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:06.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:20:06.660 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:06.695 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:06.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:06.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:06.797 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:06.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:06.834 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:06.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:06.935 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:06.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:06.968 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:06.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:07.070 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:20:07.108 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:07.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:07.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:20:07.210 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:07.247 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:07.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:07.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:07.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:07 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/268035205' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:20:07.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:07 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:20:07.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:07 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:20:07.332 INFO:teuthology.orchestra.run.smithi150.stderr:noscrub is unset 2024-09-11T08:20:07.337 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:07 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/268035205' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:20:07.337 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:07 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:20:07.337 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:07 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:20:07.337 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:07 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/268035205' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:20:07.337 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:07 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:20:07.337 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:07 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:20:07.350 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:20:07.389 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:07.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:07.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:20:07.491 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:20:07.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:07.526 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:07.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:20:07.627 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:07.661 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:07.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:07.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:07.763 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:20:07.801 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:07.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:07.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:20:07.903 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:20:07.938 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:07.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:07.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:20:08.040 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:08.073 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:08.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:08.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:08.175 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:08.176 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:20:08.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:08.205 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:08.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:08.306 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:08.307 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:08 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:20:08.307 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:08 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:20:08.307 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:08 smithi150 ceph-mon[29276]: pgmap v572: 148 pgs: 148 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.07k op/s 2024-09-11T08:20:08.307 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:08 smithi150 ceph-mon[29276]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:20:08.308 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:08 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:20:08.308 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:08 smithi150 ceph-mon[29276]: osdmap e283: 8 total, 8 up, 8 in 2024-09-11T08:20:08.308 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:08 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:20:08.308 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:08 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:20:08.308 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:08 smithi150 ceph-mon[39612]: pgmap v572: 148 pgs: 148 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.07k op/s 2024-09-11T08:20:08.308 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:08 smithi150 ceph-mon[39612]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:20:08.308 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:08 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:20:08.308 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:08 smithi150 ceph-mon[39612]: osdmap e283: 8 total, 8 up, 8 in 2024-09-11T08:20:08.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:08 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:20:08.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:08 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:20:08.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:08 smithi181 ceph-mon[32701]: pgmap v572: 148 pgs: 148 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.07k op/s 2024-09-11T08:20:08.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:08 smithi181 ceph-mon[32701]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:20:08.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:08 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:20:08.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:08 smithi181 ceph-mon[32701]: osdmap e283: 8 total, 8 up, 8 in 2024-09-11T08:20:08.336 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:08.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:08.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:08.438 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:20:08.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:08.487 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:08.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:20:08.588 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:08.621 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:08.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:08.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:08.723 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:08.759 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:08.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:08.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:08.824 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:20:08 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:20:08.861 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:08.892 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:08.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:08.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:08.920 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:20:08 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:20:08] "GET /metrics HTTP/1.1" 200 224504 "" "Prometheus/2.43.0" 2024-09-11T08:20:08.997 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:09.033 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:09.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:09.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:09.077 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:20:09.077 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:20:09.108 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:20:09.108 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:20:09.135 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:09.142 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:20:09.143 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:20:09.166 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:09.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:09.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:09.181 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:20:09.182 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:20:09.210 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:20:09.210 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:20:09.239 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:20:09.240 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:20:09.268 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:09.270 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:20:09.270 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:20:09.298 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:09.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:09.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:09.302 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:20:09.302 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:20:09.400 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:20:09.435 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:09.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:09.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:20:09.537 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:20:09.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:09.577 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:09.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:20:09.678 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:09.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:09.718 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:09.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:09.819 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:09.848 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:09.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:09.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:09.950 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:09.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:09.986 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:09.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:10.043 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:20:10.087 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:20:10.138 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T08:20:10.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:10.144 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:10.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:20:10.149 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:10 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:20:10.149 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:10 smithi150 ceph-mon[29276]: 6.4 scrub starts 2024-09-11T08:20:10.149 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:10 smithi150 ceph-mon[29276]: 6.4 scrub ok 2024-09-11T08:20:10.149 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:10 smithi150 ceph-mon[29276]: 4.8 scrub starts 2024-09-11T08:20:10.149 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:10 smithi150 ceph-mon[29276]: 4.8 scrub ok 2024-09-11T08:20:10.149 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:10 smithi150 ceph-mon[29276]: pgmap v574: 148 pgs: 148 active+clean; 5.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.31k op/s 2024-09-11T08:20:10.150 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:10 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1825632361' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:20:10.150 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:10 smithi150 ceph-mon[29276]: 4.13 scrub starts 2024-09-11T08:20:10.150 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:10 smithi150 ceph-mon[29276]: 4.13 scrub ok 2024-09-11T08:20:10.150 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:10 smithi150 ceph-mon[29276]: 3.d scrub starts 2024-09-11T08:20:10.150 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:10 smithi150 ceph-mon[29276]: 3.d scrub ok 2024-09-11T08:20:10.150 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:10 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:20:10.150 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:10 smithi150 ceph-mon[39612]: 6.4 scrub starts 2024-09-11T08:20:10.150 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:10 smithi150 ceph-mon[39612]: 6.4 scrub ok 2024-09-11T08:20:10.150 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:10 smithi150 ceph-mon[39612]: 4.8 scrub starts 2024-09-11T08:20:10.150 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:10 smithi150 ceph-mon[39612]: 4.8 scrub ok 2024-09-11T08:20:10.150 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:10 smithi150 ceph-mon[39612]: pgmap v574: 148 pgs: 148 active+clean; 5.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.31k op/s 2024-09-11T08:20:10.150 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:10 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1825632361' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:20:10.151 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:10 smithi150 ceph-mon[39612]: 4.13 scrub starts 2024-09-11T08:20:10.151 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:10 smithi150 ceph-mon[39612]: 4.13 scrub ok 2024-09-11T08:20:10.151 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:10 smithi150 ceph-mon[39612]: 3.d scrub starts 2024-09-11T08:20:10.151 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:10 smithi150 ceph-mon[39612]: 3.d scrub ok 2024-09-11T08:20:10.245 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:10.274 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:10.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:10.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:10.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:10 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:20:10.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:10 smithi181 ceph-mon[32701]: 6.4 scrub starts 2024-09-11T08:20:10.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:10 smithi181 ceph-mon[32701]: 6.4 scrub ok 2024-09-11T08:20:10.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:10 smithi181 ceph-mon[32701]: 4.8 scrub starts 2024-09-11T08:20:10.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:10 smithi181 ceph-mon[32701]: 4.8 scrub ok 2024-09-11T08:20:10.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:10 smithi181 ceph-mon[32701]: pgmap v574: 148 pgs: 148 active+clean; 5.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.31k op/s 2024-09-11T08:20:10.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:10 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1825632361' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:20:10.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:10 smithi181 ceph-mon[32701]: 4.13 scrub starts 2024-09-11T08:20:10.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:10 smithi181 ceph-mon[32701]: 4.13 scrub ok 2024-09-11T08:20:10.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:10 smithi181 ceph-mon[32701]: 3.d scrub starts 2024-09-11T08:20:10.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:10 smithi181 ceph-mon[32701]: 3.d scrub ok 2024-09-11T08:20:10.376 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:10.406 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:10.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:10.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:10.508 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:10.544 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:10.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:10.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:10.646 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:20:10.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:10.683 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:10.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:20:10.784 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:10.814 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:10.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:10.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:10.916 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:10.947 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:10.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:10.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:11.048 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:11.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:11.081 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:11.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:11.182 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:11.215 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:11.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:11.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:11.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:11 smithi150 ceph-mon[29276]: 3.1b scrub starts 2024-09-11T08:20:11.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:11 smithi150 ceph-mon[29276]: 3.1b scrub ok 2024-09-11T08:20:11.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:11 smithi150 ceph-mon[39612]: 3.1b scrub starts 2024-09-11T08:20:11.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:11 smithi150 ceph-mon[39612]: 3.1b scrub ok 2024-09-11T08:20:11.317 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:11.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:11 smithi181 ceph-mon[32701]: 3.1b scrub starts 2024-09-11T08:20:11.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:11 smithi181 ceph-mon[32701]: 3.1b scrub ok 2024-09-11T08:20:11.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:11.358 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:11.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:11.459 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:11.494 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:11.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:11.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:11.596 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:20:11.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:11.636 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:11.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:20:11.737 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:20:11.782 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:11.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:11.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:20:11.884 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:11.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:11.925 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:11.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:12.026 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:12.076 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:12.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:12.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:12.178 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:12.212 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:12.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:12.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:12.314 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:12 smithi181 ceph-mon[32701]: 3.b scrub starts 2024-09-11T08:20:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:12 smithi181 ceph-mon[32701]: 3.b scrub ok 2024-09-11T08:20:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:12 smithi181 ceph-mon[32701]: 6.d scrub starts 2024-09-11T08:20:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:12 smithi181 ceph-mon[32701]: 6.d scrub ok 2024-09-11T08:20:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:12 smithi181 ceph-mon[32701]: pgmap v575: 148 pgs: 148 active+clean; 5.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.31k op/s 2024-09-11T08:20:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:12 smithi181 ceph-mon[32701]: 9.2 scrub starts 2024-09-11T08:20:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:12 smithi181 ceph-mon[32701]: 6.1a scrub starts 2024-09-11T08:20:12.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:12 smithi181 ceph-mon[32701]: 6.1a scrub ok 2024-09-11T08:20:12.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:12 smithi181 ceph-mon[32701]: 6.c scrub starts 2024-09-11T08:20:12.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:12 smithi181 ceph-mon[32701]: 6.c scrub ok 2024-09-11T08:20:12.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:12 smithi181 ceph-mon[32701]: 6.5 scrub starts 2024-09-11T08:20:12.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:12 smithi181 ceph-mon[32701]: 6.5 scrub ok 2024-09-11T08:20:12.335 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:12.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:12.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:12.436 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:12.468 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:12.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:12.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:12.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:12 smithi150 ceph-mon[29276]: 3.b scrub starts 2024-09-11T08:20:12.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:12 smithi150 ceph-mon[29276]: 3.b scrub ok 2024-09-11T08:20:12.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:12 smithi150 ceph-mon[29276]: 6.d scrub starts 2024-09-11T08:20:12.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:12 smithi150 ceph-mon[29276]: 6.d scrub ok 2024-09-11T08:20:12.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:12 smithi150 ceph-mon[29276]: pgmap v575: 148 pgs: 148 active+clean; 5.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.31k op/s 2024-09-11T08:20:12.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:12 smithi150 ceph-mon[29276]: 9.2 scrub starts 2024-09-11T08:20:12.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:12 smithi150 ceph-mon[29276]: 6.1a scrub starts 2024-09-11T08:20:12.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:12 smithi150 ceph-mon[29276]: 6.1a scrub ok 2024-09-11T08:20:12.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:12 smithi150 ceph-mon[29276]: 6.c scrub starts 2024-09-11T08:20:12.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:12 smithi150 ceph-mon[29276]: 6.c scrub ok 2024-09-11T08:20:12.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:12 smithi150 ceph-mon[29276]: 6.5 scrub starts 2024-09-11T08:20:12.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:12 smithi150 ceph-mon[29276]: 6.5 scrub ok 2024-09-11T08:20:12.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:12 smithi150 ceph-mon[39612]: 3.b scrub starts 2024-09-11T08:20:12.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:12 smithi150 ceph-mon[39612]: 3.b scrub ok 2024-09-11T08:20:12.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:12 smithi150 ceph-mon[39612]: 6.d scrub starts 2024-09-11T08:20:12.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:12 smithi150 ceph-mon[39612]: 6.d scrub ok 2024-09-11T08:20:12.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:12 smithi150 ceph-mon[39612]: pgmap v575: 148 pgs: 148 active+clean; 5.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.31k op/s 2024-09-11T08:20:12.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:12 smithi150 ceph-mon[39612]: 9.2 scrub starts 2024-09-11T08:20:12.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:12 smithi150 ceph-mon[39612]: 6.1a scrub starts 2024-09-11T08:20:12.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:12 smithi150 ceph-mon[39612]: 6.1a scrub ok 2024-09-11T08:20:12.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:12 smithi150 ceph-mon[39612]: 6.c scrub starts 2024-09-11T08:20:12.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:12 smithi150 ceph-mon[39612]: 6.c scrub ok 2024-09-11T08:20:12.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:12 smithi150 ceph-mon[39612]: 6.5 scrub starts 2024-09-11T08:20:12.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:12 smithi150 ceph-mon[39612]: 6.5 scrub ok 2024-09-11T08:20:12.571 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:12.601 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:12.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:12.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:12.647 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T08:20:12.703 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:20:12.773 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:12.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:12.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:20:12.875 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:12.903 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:12.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:12.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:13.005 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:20:13.043 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:13.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:13.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:20:13.144 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:13.182 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:13.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:13.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:13.284 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:13.323 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:13.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:13 smithi181 ceph-mon[32701]: 6.11 scrub starts 2024-09-11T08:20:13.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:13 smithi181 ceph-mon[32701]: 6.11 scrub ok 2024-09-11T08:20:13.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:13.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:13.364 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:13 smithi150 ceph-mon[29276]: 6.11 scrub starts 2024-09-11T08:20:13.365 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:13 smithi150 ceph-mon[29276]: 6.11 scrub ok 2024-09-11T08:20:13.365 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:13 smithi150 ceph-mon[39612]: 6.11 scrub starts 2024-09-11T08:20:13.365 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:13 smithi150 ceph-mon[39612]: 6.11 scrub ok 2024-09-11T08:20:13.400 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:20:13.425 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:13.455 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:13.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:13.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:13.557 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:13.590 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:13.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:13.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:13.692 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:13.721 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:13.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:13.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:13.823 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:13.856 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:13.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:13.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:13.958 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:13.990 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:13.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:13.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:14.092 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:20:14.126 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:14.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:14.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:20:14.228 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:14.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:14 smithi150 ceph-mon[29276]: 3.17 scrub starts 2024-09-11T08:20:14.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:14 smithi150 ceph-mon[29276]: 3.17 scrub ok 2024-09-11T08:20:14.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:14 smithi150 ceph-mon[29276]: pgmap v576: 148 pgs: 148 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.29k op/s 2024-09-11T08:20:14.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:14 smithi150 ceph-mon[29276]: 9.2 scrub ok 2024-09-11T08:20:14.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:14 smithi150 ceph-mon[29276]: 4.1a scrub starts 2024-09-11T08:20:14.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:14 smithi150 ceph-mon[29276]: 4.1a scrub ok 2024-09-11T08:20:14.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:14 smithi150 ceph-mon[29276]: 4.14 scrub starts 2024-09-11T08:20:14.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:14 smithi150 ceph-mon[29276]: 4.14 scrub ok 2024-09-11T08:20:14.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:14 smithi150 ceph-mon[39612]: 3.17 scrub starts 2024-09-11T08:20:14.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:14 smithi150 ceph-mon[39612]: 3.17 scrub ok 2024-09-11T08:20:14.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:14 smithi150 ceph-mon[39612]: pgmap v576: 148 pgs: 148 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.29k op/s 2024-09-11T08:20:14.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:14 smithi150 ceph-mon[39612]: 9.2 scrub ok 2024-09-11T08:20:14.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:14 smithi150 ceph-mon[39612]: 4.1a scrub starts 2024-09-11T08:20:14.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:14 smithi150 ceph-mon[39612]: 4.1a scrub ok 2024-09-11T08:20:14.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:14 smithi150 ceph-mon[39612]: 4.14 scrub starts 2024-09-11T08:20:14.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:14 smithi150 ceph-mon[39612]: 4.14 scrub ok 2024-09-11T08:20:14.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:14.264 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:14.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:14.365 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:20:14.415 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:14.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:14.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:20:14.517 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:14.533 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-11T08:20:14.533 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [ 2024-09-11T08:20:14.533 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:14.533 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15369.0:154840 9.2 9:4eb11e92:::benchmark_data_smithi150_133616_object39234:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e283)", 2024-09-11T08:20:14.533 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:20:14.516851+0000", 2024-09-11T08:20:14.533 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.013857815000000001, 2024-09-11T08:20:14.534 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.013889855, 2024-09-11T08:20:14.534 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:20:14.534 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "waiting for sub ops", 2024-09-11T08:20:14.534 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:20:14.534 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15369", 2024-09-11T08:20:14.534 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:20:14.534 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 154840 2024-09-11T08:20:14.534 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:14.534 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:20:14.534 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:14.534 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:20:14.534 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:20:14.516851+0000", 2024-09-11T08:20:14.534 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:20:14.534 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:14.535 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:14.535 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:20:14.535 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:20:14.516851+0000", 2024-09-11T08:20:14.535 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:20:14.535 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:14.535 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:14.535 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:20:14.535 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:20:14.516849+0000", 2024-09-11T08:20:14.535 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:20:14.535 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:14.535 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:14.535 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:20:14.535 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:20:14.516857+0000", 2024-09-11T08:20:14.535 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.2719999999999997e-06 2024-09-11T08:20:14.535 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:14.535 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:14.536 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:20:14.536 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:20:14.516870+0000", 2024-09-11T08:20:14.536 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3056e-05 2024-09-11T08:20:14.536 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:14.536 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:14.536 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:20:14.536 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:20:14.516876+0000", 2024-09-11T08:20:14.536 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.789e-06 2024-09-11T08:20:14.536 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:14.536 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:14.536 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:20:14.536 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:20:14.518382+0000", 2024-09-11T08:20:14.536 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0015060029999999999 2024-09-11T08:20:14.536 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:14.536 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:14.537 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:20:14.537 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:20:14.518833+0000", 2024-09-11T08:20:14.537 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00045086200000000001 2024-09-11T08:20:14.537 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:14.537 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:14.537 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 1,2", 2024-09-11T08:20:14.537 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:20:14.519026+0000", 2024-09-11T08:20:14.537 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00019279 2024-09-11T08:20:14.537 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:14.537 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:14.537 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:20:14.537 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:20:14.522794+0000", 2024-09-11T08:20:14.537 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.003767683 2024-09-11T08:20:14.537 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:14.537 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:14.538 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:20:14.538 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:20:14.525837+0000", 2024-09-11T08:20:14.538 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0030437939999999998 2024-09-11T08:20:14.538 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:14.538 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:14.538 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:14.538 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:14.538 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:14.538 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15369.0:154844 9.2 9:4ddfec91:::benchmark_data_smithi150_133616_object39238:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e283)", 2024-09-11T08:20:14.538 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:20:14.521311+0000", 2024-09-11T08:20:14.538 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.0093976560000000008, 2024-09-11T08:20:14.538 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0095225819999999999, 2024-09-11T08:20:14.538 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:20:14.538 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "waiting for sub ops", 2024-09-11T08:20:14.538 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:20:14.539 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15369", 2024-09-11T08:20:14.539 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:20:14.539 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 154844 2024-09-11T08:20:14.539 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:14.539 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:20:14.539 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:14.539 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:20:14.539 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:20:14.521311+0000", 2024-09-11T08:20:14.539 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:20:14.539 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:14.539 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:14.539 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:20:14.539 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:20:14.521311+0000", 2024-09-11T08:20:14.539 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:20:14.539 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:14.540 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:14.540 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:20:14.540 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:20:14.521309+0000", 2024-09-11T08:20:14.540 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:20:14.540 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:14.540 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:14.540 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:20:14.540 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:20:14.521318+0000", 2024-09-11T08:20:14.540 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.7420000000000003e-06 2024-09-11T08:20:14.540 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:14.540 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:14.540 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:20:14.540 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:20:14.521339+0000", 2024-09-11T08:20:14.540 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.1154000000000001e-05 2024-09-11T08:20:14.541 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:14.541 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:14.541 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:20:14.541 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:20:14.521347+0000", 2024-09-11T08:20:14.541 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.9850000000000006e-06 2024-09-11T08:20:14.541 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:14.541 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:14.541 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:20:14.541 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:20:14.522838+0000", 2024-09-11T08:20:14.541 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.001490681 2024-09-11T08:20:14.541 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:14.541 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:14.541 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:20:14.541 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:20:14.522956+0000", 2024-09-11T08:20:14.541 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000117939 2024-09-11T08:20:14.542 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:14.542 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:14.542 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 1,2", 2024-09-11T08:20:14.542 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:20:14.523110+0000", 2024-09-11T08:20:14.542 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000154628 2024-09-11T08:20:14.542 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:14.542 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:14.542 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:20:14.542 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:20:14.526292+0000", 2024-09-11T08:20:14.542 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0031819320000000002 2024-09-11T08:20:14.542 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:14.542 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:14.542 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:14.542 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:14.543 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:14.543 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15369.0:154853 9.2 9:40b46c62:::benchmark_data_smithi150_133616_object39247:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e283)", 2024-09-11T08:20:14.543 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:20:14.526890+0000", 2024-09-11T08:20:14.543 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.0038185879999999999, 2024-09-11T08:20:14.543 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0040170479999999996, 2024-09-11T08:20:14.543 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:20:14.543 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "waiting for sub ops", 2024-09-11T08:20:14.543 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:20:14.543 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15369", 2024-09-11T08:20:14.543 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:20:14.543 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 154853 2024-09-11T08:20:14.543 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:14.543 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:20:14.543 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:14.544 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:20:14.544 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:20:14.526890+0000", 2024-09-11T08:20:14.544 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:20:14.544 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:14.544 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:14.544 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:20:14.544 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:20:14.526890+0000", 2024-09-11T08:20:14.544 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:20:14.544 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:14.544 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:14.544 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:20:14.544 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:20:14.526888+0000", 2024-09-11T08:20:14.544 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:20:14.544 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:14.545 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:14.545 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:20:14.545 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:20:14.526895+0000", 2024-09-11T08:20:14.545 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.233e-06 2024-09-11T08:20:14.545 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:14.545 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:14.545 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:20:14.545 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:20:14.526912+0000", 2024-09-11T08:20:14.545 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.6166000000000001e-05 2024-09-11T08:20:14.545 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:14.545 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:14.545 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:20:14.545 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:20:14.526919+0000", 2024-09-11T08:20:14.545 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.0509999999999999e-06 2024-09-11T08:20:14.546 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:14.546 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:14.546 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:20:14.546 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:20:14.527075+0000", 2024-09-11T08:20:14.546 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000156467 2024-09-11T08:20:14.546 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:14.546 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:14.546 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:20:14.546 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:20:14.527413+0000", 2024-09-11T08:20:14.546 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00033807299999999999 2024-09-11T08:20:14.546 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:14.546 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:14.546 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 1,2", 2024-09-11T08:20:14.546 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:20:14.527810+0000", 2024-09-11T08:20:14.547 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00039664700000000003 2024-09-11T08:20:14.547 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:14.547 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:14.547 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:14.547 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:14.547 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:14.547 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.15369.0:154852 9.7 e283/271 9:ecc20194:::benchmark_data_smithi150_133616_object39246:head v 283'9671, mlcod=283'9669)", 2024-09-11T08:20:14.547 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:20:14.528468+0000", 2024-09-11T08:20:14.547 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.0022408969999999999, 2024-09-11T08:20:14.547 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0025060450000000001, 2024-09-11T08:20:14.547 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:20:14.547 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "started", 2024-09-11T08:20:14.547 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:20:14.548 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:14.548 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:20:14.548 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:20:14.528468+0000", 2024-09-11T08:20:14.548 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:20:14.548 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:14.548 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:14.548 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:20:14.548 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:20:14.528468+0000", 2024-09-11T08:20:14.548 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:20:14.548 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:14.548 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:14.548 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:20:14.548 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:20:14.528467+0000", 2024-09-11T08:20:14.548 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:20:14.549 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:14.549 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:14.549 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:20:14.549 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:20:14.528475+0000", 2024-09-11T08:20:14.549 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.3010000000000007e-06 2024-09-11T08:20:14.549 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:14.549 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:14.549 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:20:14.549 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:20:14.528491+0000", 2024-09-11T08:20:14.549 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.5647000000000001e-05 2024-09-11T08:20:14.549 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:14.549 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:14.549 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:20:14.549 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:20:14.528497+0000", 2024-09-11T08:20:14.550 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.0850000000000002e-06 2024-09-11T08:20:14.550 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:14.550 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:14.550 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:20:14.550 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:20:14.528692+0000", 2024-09-11T08:20:14.550 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00019473099999999999 2024-09-11T08:20:14.550 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:14.550 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:14.550 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:20:14.550 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:20:14.528740+0000", 2024-09-11T08:20:14.550 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.8245000000000003e-05 2024-09-11T08:20:14.550 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:14.550 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:14.550 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:14.550 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:14.551 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:14.551 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.15369.0:154851 9.f e283/271 9:fdbff8c7:::benchmark_data_smithi150_133616_object39245:head v 283'9787, mlcod=283'9785)", 2024-09-11T08:20:14.551 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:20:14.528510+0000", 2024-09-11T08:20:14.551 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.0021986929999999998, 2024-09-11T08:20:14.551 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.002515827, 2024-09-11T08:20:14.551 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:20:14.551 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "started", 2024-09-11T08:20:14.551 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:20:14.551 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:14.551 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:20:14.551 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:20:14.528510+0000", 2024-09-11T08:20:14.551 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:20:14.551 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:14.551 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:14.551 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:20:14.552 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:20:14.528510+0000", 2024-09-11T08:20:14.552 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:20:14.552 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:14.552 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:14.552 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:20:14.552 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:20:14.528510+0000", 2024-09-11T08:20:14.552 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:20:14.552 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:14.552 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:14.552 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:20:14.552 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:20:14.528514+0000", 2024-09-11T08:20:14.552 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.155e-06 2024-09-11T08:20:14.552 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:14.552 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:14.553 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:20:14.553 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:20:14.528520+0000", 2024-09-11T08:20:14.553 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.6300000000000003e-06 2024-09-11T08:20:14.553 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:14.553 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:14.553 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:20:14.553 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:20:14.528523+0000", 2024-09-11T08:20:14.553 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.5939999999999998e-06 2024-09-11T08:20:14.553 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:14.553 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:14.553 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:20:14.553 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:20:14.530728+0000", 2024-09-11T08:20:14.553 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.002204971 2024-09-11T08:20:14.553 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:14.553 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:14.554 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:20:14.554 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:20:14.530780+0000", 2024-09-11T08:20:14.554 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.1681000000000002e-05 2024-09-11T08:20:14.554 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:14.554 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:14.554 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:14.554 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:14.554 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-09-11T08:20:14.554 INFO:teuthology.orchestra.run.smithi150.stdout: "num_ops": 5 2024-09-11T08:20:14.554 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-11T08:20:14.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:14.555 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:14.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:14.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:14 smithi181 ceph-mon[32701]: 3.17 scrub starts 2024-09-11T08:20:14.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:14 smithi181 ceph-mon[32701]: 3.17 scrub ok 2024-09-11T08:20:14.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:14 smithi181 ceph-mon[32701]: pgmap v576: 148 pgs: 148 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.29k op/s 2024-09-11T08:20:14.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:14 smithi181 ceph-mon[32701]: 9.2 scrub ok 2024-09-11T08:20:14.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:14 smithi181 ceph-mon[32701]: 4.1a scrub starts 2024-09-11T08:20:14.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:14 smithi181 ceph-mon[32701]: 4.1a scrub ok 2024-09-11T08:20:14.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:14 smithi181 ceph-mon[32701]: 4.14 scrub starts 2024-09-11T08:20:14.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:14 smithi181 ceph-mon[32701]: 4.14 scrub ok 2024-09-11T08:20:14.656 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:14.691 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:14.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:14.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:14.793 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:14.823 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:14.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:14.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:14.925 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:14.958 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:14.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:14.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:15.060 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:20:15.096 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:15.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:15.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:20:15.105 INFO:teuthology.orchestra.run.smithi150.stderr:set 6.5 pg_upmap mapping to [2,4,7] 2024-09-11T08:20:15.198 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:15.235 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:15.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:15.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:15.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:15 smithi150 ceph-mon[29276]: 4.2 scrub starts 2024-09-11T08:20:15.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:15 smithi150 ceph-mon[29276]: 4.2 scrub ok 2024-09-11T08:20:15.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:15 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3838042562' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.5", "id": [2, 4, 7]}]: dispatch 2024-09-11T08:20:15.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:15 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.5", "id": [2, 4, 7]}]: dispatch 2024-09-11T08:20:15.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:15 smithi150 ceph-mon[29276]: 5.14 scrub starts 2024-09-11T08:20:15.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:15 smithi150 ceph-mon[29276]: 5.14 scrub ok 2024-09-11T08:20:15.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:15 smithi150 ceph-mon[39612]: 4.2 scrub starts 2024-09-11T08:20:15.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:15 smithi150 ceph-mon[39612]: 4.2 scrub ok 2024-09-11T08:20:15.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:15 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3838042562' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.5", "id": [2, 4, 7]}]: dispatch 2024-09-11T08:20:15.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:15 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.5", "id": [2, 4, 7]}]: dispatch 2024-09-11T08:20:15.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:15 smithi150 ceph-mon[39612]: 5.14 scrub starts 2024-09-11T08:20:15.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:15 smithi150 ceph-mon[39612]: 5.14 scrub ok 2024-09-11T08:20:15.337 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:20:15.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:15.376 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:15.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:20:15.477 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:20:15.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:15.513 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:15.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:20:15.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:20:15 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:20:15] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:20:15.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:15 smithi181 ceph-mon[32701]: 4.2 scrub starts 2024-09-11T08:20:15.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:15 smithi181 ceph-mon[32701]: 4.2 scrub ok 2024-09-11T08:20:15.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:15 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3838042562' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.5", "id": [2, 4, 7]}]: dispatch 2024-09-11T08:20:15.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:15 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.5", "id": [2, 4, 7]}]: dispatch 2024-09-11T08:20:15.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:15 smithi181 ceph-mon[32701]: 5.14 scrub starts 2024-09-11T08:20:15.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:15 smithi181 ceph-mon[32701]: 5.14 scrub ok 2024-09-11T08:20:15.613 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:15.649 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:15.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:15.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:15.751 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:20:15.789 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:15.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:15.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:20:15.891 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:15.926 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:15.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:15.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:16.028 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:16.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:16.062 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:16.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:16.162 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:16.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:16.194 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:16.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:16.294 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:20:16.333 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:16.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:16.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:20:16.434 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:20:16.473 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:16.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:16.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:20:16.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:16 smithi150 ceph-mon[29276]: 4.16 scrub starts 2024-09-11T08:20:16.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:16 smithi150 ceph-mon[29276]: 4.16 scrub ok 2024-09-11T08:20:16.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:16 smithi150 ceph-mon[29276]: pgmap v577: 148 pgs: 148 active+clean; 4.8 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.27k op/s 2024-09-11T08:20:16.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:16 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.5", "id": [2, 4, 7]}]': finished 2024-09-11T08:20:16.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:16 smithi150 ceph-mon[29276]: osdmap e284: 8 total, 8 up, 8 in 2024-09-11T08:20:16.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:16 smithi150 ceph-mon[29276]: 6.14 scrub starts 2024-09-11T08:20:16.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:16 smithi150 ceph-mon[29276]: 6.14 scrub ok 2024-09-11T08:20:16.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:16 smithi150 ceph-mon[29276]: 9.1 scrub starts 2024-09-11T08:20:16.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:16 smithi150 ceph-mon[29276]: 4.0 scrub starts 2024-09-11T08:20:16.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:16 smithi150 ceph-mon[29276]: 4.0 scrub ok 2024-09-11T08:20:16.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:16 smithi150 ceph-mon[39612]: 4.16 scrub starts 2024-09-11T08:20:16.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:16 smithi150 ceph-mon[39612]: 4.16 scrub ok 2024-09-11T08:20:16.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:16 smithi150 ceph-mon[39612]: pgmap v577: 148 pgs: 148 active+clean; 4.8 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.27k op/s 2024-09-11T08:20:16.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:16 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.5", "id": [2, 4, 7]}]': finished 2024-09-11T08:20:16.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:16 smithi150 ceph-mon[39612]: osdmap e284: 8 total, 8 up, 8 in 2024-09-11T08:20:16.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:16 smithi150 ceph-mon[39612]: 6.14 scrub starts 2024-09-11T08:20:16.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:16 smithi150 ceph-mon[39612]: 6.14 scrub ok 2024-09-11T08:20:16.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:16 smithi150 ceph-mon[39612]: 9.1 scrub starts 2024-09-11T08:20:16.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:16 smithi150 ceph-mon[39612]: 4.0 scrub starts 2024-09-11T08:20:16.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:16 smithi150 ceph-mon[39612]: 4.0 scrub ok 2024-09-11T08:20:16.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:16 smithi181 ceph-mon[32701]: 4.16 scrub starts 2024-09-11T08:20:16.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:16 smithi181 ceph-mon[32701]: 4.16 scrub ok 2024-09-11T08:20:16.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:16 smithi181 ceph-mon[32701]: pgmap v577: 148 pgs: 148 active+clean; 4.8 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.27k op/s 2024-09-11T08:20:16.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:16 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.5", "id": [2, 4, 7]}]': finished 2024-09-11T08:20:16.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:16 smithi181 ceph-mon[32701]: osdmap e284: 8 total, 8 up, 8 in 2024-09-11T08:20:16.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:16 smithi181 ceph-mon[32701]: 6.14 scrub starts 2024-09-11T08:20:16.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:16 smithi181 ceph-mon[32701]: 6.14 scrub ok 2024-09-11T08:20:16.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:16 smithi181 ceph-mon[32701]: 9.1 scrub starts 2024-09-11T08:20:16.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:16 smithi181 ceph-mon[32701]: 4.0 scrub starts 2024-09-11T08:20:16.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:16 smithi181 ceph-mon[32701]: 4.0 scrub ok 2024-09-11T08:20:16.574 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:16.606 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:16.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:16.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:16.708 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:16.750 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:16.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:16.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:16.851 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:20:16.880 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:16.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:16.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:20:16.982 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:20:17.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:17.014 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:17.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:20:17.115 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:17.153 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:17.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:17.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:17.254 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:20:17.292 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:17.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:17.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:20:17.394 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:20:17.425 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:17.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:17.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:20:17.487 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:17 smithi150 ceph-mon[29276]: 6.13 scrub starts 2024-09-11T08:20:17.487 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:17 smithi150 ceph-mon[29276]: 6.13 scrub ok 2024-09-11T08:20:17.487 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:17 smithi150 ceph-mon[29276]: osdmap e285: 8 total, 8 up, 8 in 2024-09-11T08:20:17.487 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:17 smithi150 ceph-mon[29276]: 4.c scrub starts 2024-09-11T08:20:17.487 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:17 smithi150 ceph-mon[29276]: 4.c scrub ok 2024-09-11T08:20:17.487 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:17 smithi150 ceph-mon[29276]: 4.5 scrub starts 2024-09-11T08:20:17.487 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:17 smithi150 ceph-mon[29276]: 4.5 scrub ok 2024-09-11T08:20:17.487 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:17 smithi150 ceph-mon[39612]: 6.13 scrub starts 2024-09-11T08:20:17.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:17 smithi150 ceph-mon[39612]: 6.13 scrub ok 2024-09-11T08:20:17.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:17 smithi150 ceph-mon[39612]: osdmap e285: 8 total, 8 up, 8 in 2024-09-11T08:20:17.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:17 smithi150 ceph-mon[39612]: 4.c scrub starts 2024-09-11T08:20:17.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:17 smithi150 ceph-mon[39612]: 4.c scrub ok 2024-09-11T08:20:17.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:17 smithi150 ceph-mon[39612]: 4.5 scrub starts 2024-09-11T08:20:17.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:17 smithi150 ceph-mon[39612]: 4.5 scrub ok 2024-09-11T08:20:17.527 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:17.529 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-11T08:20:17.559 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:17.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:17.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:17.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:17 smithi181 ceph-mon[32701]: 6.13 scrub starts 2024-09-11T08:20:17.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:17 smithi181 ceph-mon[32701]: 6.13 scrub ok 2024-09-11T08:20:17.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:17 smithi181 ceph-mon[32701]: osdmap e285: 8 total, 8 up, 8 in 2024-09-11T08:20:17.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:17 smithi181 ceph-mon[32701]: 4.c scrub starts 2024-09-11T08:20:17.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:17 smithi181 ceph-mon[32701]: 4.c scrub ok 2024-09-11T08:20:17.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:17 smithi181 ceph-mon[32701]: 4.5 scrub starts 2024-09-11T08:20:17.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:17 smithi181 ceph-mon[32701]: 4.5 scrub ok 2024-09-11T08:20:17.661 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:20:17.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:17.708 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:17.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:20:17.744 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:20:17.810 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:17.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:17.861 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:17.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:17.961 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:17.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:17.995 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:17.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:18.096 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:18.134 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:18.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:18.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:18.137 INFO:teuthology.orchestra.run.smithi150.stderr:nodeep-scrub is unset 2024-09-11T08:20:18.137 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:18 smithi150 ceph-mon[29276]: 5.13 scrub starts 2024-09-11T08:20:18.137 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:18 smithi150 ceph-mon[29276]: 5.13 scrub ok 2024-09-11T08:20:18.137 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:18 smithi150 ceph-mon[29276]: 5.15 scrub starts 2024-09-11T08:20:18.138 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:18 smithi150 ceph-mon[29276]: 5.15 scrub ok 2024-09-11T08:20:18.138 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:18 smithi150 ceph-mon[29276]: 4.10 scrub starts 2024-09-11T08:20:18.138 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:18 smithi150 ceph-mon[29276]: 4.10 scrub ok 2024-09-11T08:20:18.138 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:18 smithi150 ceph-mon[29276]: pgmap v580: 148 pgs: 148 active+clean; 4.8 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 758 op/s 2024-09-11T08:20:18.138 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:18 smithi150 ceph-mon[29276]: 9.1 scrub ok 2024-09-11T08:20:18.138 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:18 smithi150 ceph-mon[29276]: 4.1d scrub starts 2024-09-11T08:20:18.138 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:18 smithi150 ceph-mon[29276]: 4.1d scrub ok 2024-09-11T08:20:18.138 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:18 smithi150 ceph-mon[29276]: 3.a scrub starts 2024-09-11T08:20:18.138 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:18 smithi150 ceph-mon[29276]: 3.a scrub ok 2024-09-11T08:20:18.138 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:18 smithi150 ceph-mon[29276]: 3.13 scrub starts 2024-09-11T08:20:18.138 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:18 smithi150 ceph-mon[29276]: 3.13 scrub ok 2024-09-11T08:20:18.138 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:18 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1870236629' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:20:18.138 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:18 smithi150 ceph-mon[39612]: 5.13 scrub starts 2024-09-11T08:20:18.139 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:18 smithi150 ceph-mon[39612]: 5.13 scrub ok 2024-09-11T08:20:18.139 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:18 smithi150 ceph-mon[39612]: 5.15 scrub starts 2024-09-11T08:20:18.139 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:18 smithi150 ceph-mon[39612]: 5.15 scrub ok 2024-09-11T08:20:18.139 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:18 smithi150 ceph-mon[39612]: 4.10 scrub starts 2024-09-11T08:20:18.139 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:18 smithi150 ceph-mon[39612]: 4.10 scrub ok 2024-09-11T08:20:18.139 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:18 smithi150 ceph-mon[39612]: pgmap v580: 148 pgs: 148 active+clean; 4.8 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 758 op/s 2024-09-11T08:20:18.139 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:18 smithi150 ceph-mon[39612]: 9.1 scrub ok 2024-09-11T08:20:18.139 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:18 smithi150 ceph-mon[39612]: 4.1d scrub starts 2024-09-11T08:20:18.139 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:18 smithi150 ceph-mon[39612]: 4.1d scrub ok 2024-09-11T08:20:18.139 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:18 smithi150 ceph-mon[39612]: 3.a scrub starts 2024-09-11T08:20:18.139 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:18 smithi150 ceph-mon[39612]: 3.a scrub ok 2024-09-11T08:20:18.139 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:18 smithi150 ceph-mon[39612]: 3.13 scrub starts 2024-09-11T08:20:18.139 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:18 smithi150 ceph-mon[39612]: 3.13 scrub ok 2024-09-11T08:20:18.139 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:18 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1870236629' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:20:18.235 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:18.274 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:18.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:18.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:18.376 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:20:18.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:18.417 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:18.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:20:18.448 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:18 smithi181 ceph-mon[32701]: 5.13 scrub starts 2024-09-11T08:20:18.448 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:18 smithi181 ceph-mon[32701]: 5.13 scrub ok 2024-09-11T08:20:18.448 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:18 smithi181 ceph-mon[32701]: 5.15 scrub starts 2024-09-11T08:20:18.448 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:18 smithi181 ceph-mon[32701]: 5.15 scrub ok 2024-09-11T08:20:18.448 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:18 smithi181 ceph-mon[32701]: 4.10 scrub starts 2024-09-11T08:20:18.448 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:18 smithi181 ceph-mon[32701]: 4.10 scrub ok 2024-09-11T08:20:18.448 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:18 smithi181 ceph-mon[32701]: pgmap v580: 148 pgs: 148 active+clean; 4.8 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 758 op/s 2024-09-11T08:20:18.448 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:18 smithi181 ceph-mon[32701]: 9.1 scrub ok 2024-09-11T08:20:18.448 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:18 smithi181 ceph-mon[32701]: 4.1d scrub starts 2024-09-11T08:20:18.448 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:18 smithi181 ceph-mon[32701]: 4.1d scrub ok 2024-09-11T08:20:18.449 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:18 smithi181 ceph-mon[32701]: 3.a scrub starts 2024-09-11T08:20:18.449 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:18 smithi181 ceph-mon[32701]: 3.a scrub ok 2024-09-11T08:20:18.449 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:18 smithi181 ceph-mon[32701]: 3.13 scrub starts 2024-09-11T08:20:18.449 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:18 smithi181 ceph-mon[32701]: 3.13 scrub ok 2024-09-11T08:20:18.449 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:18 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1870236629' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:20:18.517 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:18.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:18.553 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:18.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:18.654 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:18.686 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:18.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:18.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:18.788 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:18.823 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:20:18 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:20:18.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:18.827 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:18.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:18.928 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:20:18 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:20:18] "GET /metrics HTTP/1.1" 200 224504 "" "Prometheus/2.43.0" 2024-09-11T08:20:18.928 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:20:18.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:18.974 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:18.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:20:19.075 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:20:19.108 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:19.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:19.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:20:19.210 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:19.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:19 smithi150 ceph-mon[29276]: 6.1b scrub starts 2024-09-11T08:20:19.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:19 smithi150 ceph-mon[29276]: 6.1b scrub ok 2024-09-11T08:20:19.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:19 smithi150 ceph-mon[29276]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:20:19.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:19 smithi150 ceph-mon[29276]: Cluster is now healthy 2024-09-11T08:20:19.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:19 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1870236629' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:20:19.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:19 smithi150 ceph-mon[29276]: osdmap e286: 8 total, 8 up, 8 in 2024-09-11T08:20:19.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:19 smithi150 ceph-mon[29276]: 4.19 scrub starts 2024-09-11T08:20:19.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:19 smithi150 ceph-mon[29276]: 4.19 scrub ok 2024-09-11T08:20:19.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:19 smithi150 ceph-mon[29276]: 6.1f scrub starts 2024-09-11T08:20:19.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:19 smithi150 ceph-mon[29276]: 6.1f scrub ok 2024-09-11T08:20:19.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:19 smithi150 ceph-mon[29276]: 6.12 deep-scrub starts 2024-09-11T08:20:19.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:19 smithi150 ceph-mon[29276]: 6.12 deep-scrub ok 2024-09-11T08:20:19.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:19 smithi150 ceph-mon[39612]: 6.1b scrub starts 2024-09-11T08:20:19.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:19 smithi150 ceph-mon[39612]: 6.1b scrub ok 2024-09-11T08:20:19.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:19 smithi150 ceph-mon[39612]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:20:19.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:19 smithi150 ceph-mon[39612]: Cluster is now healthy 2024-09-11T08:20:19.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:19 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1870236629' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:20:19.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:19 smithi150 ceph-mon[39612]: osdmap e286: 8 total, 8 up, 8 in 2024-09-11T08:20:19.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:19 smithi150 ceph-mon[39612]: 4.19 scrub starts 2024-09-11T08:20:19.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:19 smithi150 ceph-mon[39612]: 4.19 scrub ok 2024-09-11T08:20:19.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:19 smithi150 ceph-mon[39612]: 6.1f scrub starts 2024-09-11T08:20:19.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:19 smithi150 ceph-mon[39612]: 6.1f scrub ok 2024-09-11T08:20:19.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:19 smithi150 ceph-mon[39612]: 6.12 deep-scrub starts 2024-09-11T08:20:19.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:19 smithi150 ceph-mon[39612]: 6.12 deep-scrub ok 2024-09-11T08:20:19.244 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:19.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:19.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:19.346 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:19.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:19.384 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:19.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:19.485 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:20:19.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:19.534 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:19.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:20:19.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:19 smithi181 ceph-mon[32701]: 6.1b scrub starts 2024-09-11T08:20:19.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:19 smithi181 ceph-mon[32701]: 6.1b scrub ok 2024-09-11T08:20:19.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:19 smithi181 ceph-mon[32701]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:20:19.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:19 smithi181 ceph-mon[32701]: Cluster is now healthy 2024-09-11T08:20:19.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:19 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1870236629' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:20:19.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:19 smithi181 ceph-mon[32701]: osdmap e286: 8 total, 8 up, 8 in 2024-09-11T08:20:19.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:19 smithi181 ceph-mon[32701]: 4.19 scrub starts 2024-09-11T08:20:19.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:19 smithi181 ceph-mon[32701]: 4.19 scrub ok 2024-09-11T08:20:19.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:19 smithi181 ceph-mon[32701]: 6.1f scrub starts 2024-09-11T08:20:19.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:19 smithi181 ceph-mon[32701]: 6.1f scrub ok 2024-09-11T08:20:19.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:19 smithi181 ceph-mon[32701]: 6.12 deep-scrub starts 2024-09-11T08:20:19.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:19 smithi181 ceph-mon[32701]: 6.12 deep-scrub ok 2024-09-11T08:20:19.635 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:20:19.671 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:19.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:19.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:20:19.774 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:20:19.806 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:19.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:19.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:20:19.908 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:20:19.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:19.944 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:19.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:20:20.046 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:20.077 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:20.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:20.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:20.131 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:20:20.179 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:20.211 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:20.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:20.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:20.313 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:20.348 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:20.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:20.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:20.396 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:20 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:20:20.396 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:20 smithi150 ceph-mon[29276]: 5.1b deep-scrub starts 2024-09-11T08:20:20.396 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:20 smithi150 ceph-mon[29276]: 5.1b deep-scrub ok 2024-09-11T08:20:20.396 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:20 smithi150 ceph-mon[29276]: pgmap v582: 148 pgs: 148 active+clean; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.77k op/s 2024-09-11T08:20:20.396 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:20 smithi150 ceph-mon[29276]: 5.1f scrub starts 2024-09-11T08:20:20.396 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:20 smithi150 ceph-mon[29276]: 5.1f scrub ok 2024-09-11T08:20:20.397 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:20 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:20:20.397 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:20 smithi150 ceph-mon[39612]: 5.1b deep-scrub starts 2024-09-11T08:20:20.397 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:20 smithi150 ceph-mon[39612]: 5.1b deep-scrub ok 2024-09-11T08:20:20.397 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:20 smithi150 ceph-mon[39612]: pgmap v582: 148 pgs: 148 active+clean; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.77k op/s 2024-09-11T08:20:20.397 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:20 smithi150 ceph-mon[39612]: 5.1f scrub starts 2024-09-11T08:20:20.397 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:20 smithi150 ceph-mon[39612]: 5.1f scrub ok 2024-09-11T08:20:20.450 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:20:20.486 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:20.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:20.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:20:20.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:20 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:20:20.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:20 smithi181 ceph-mon[32701]: 5.1b deep-scrub starts 2024-09-11T08:20:20.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:20 smithi181 ceph-mon[32701]: 5.1b deep-scrub ok 2024-09-11T08:20:20.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:20 smithi181 ceph-mon[32701]: pgmap v582: 148 pgs: 148 active+clean; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.77k op/s 2024-09-11T08:20:20.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:20 smithi181 ceph-mon[32701]: 5.1f scrub starts 2024-09-11T08:20:20.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:20 smithi181 ceph-mon[32701]: 5.1f scrub ok 2024-09-11T08:20:20.588 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:20:20.632 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:20.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:20.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:20:20.734 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:20.775 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:20.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:20.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:20.876 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:20.915 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:20.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:20.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:20.973 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:20:20.974 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:20:21.004 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:20:21.004 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:20:21.018 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:21.040 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:20:21.041 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:20:21.049 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:21.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:21.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:21.076 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:20:21.076 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:20:21.111 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:20:21.111 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:20:21.146 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:20:21.147 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:20:21.151 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:21.183 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:21.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:21.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:21.184 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:20:21.185 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:20:21.224 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:21 smithi150 ceph-mon[29276]: 3.4 scrub starts 2024-09-11T08:20:21.224 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:21 smithi150 ceph-mon[29276]: 3.4 scrub ok 2024-09-11T08:20:21.224 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:21 smithi150 ceph-mon[29276]: 3.8 scrub starts 2024-09-11T08:20:21.224 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:21 smithi150 ceph-mon[29276]: 3.8 scrub ok 2024-09-11T08:20:21.225 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:21 smithi150 ceph-mon[29276]: 6.19 deep-scrub starts 2024-09-11T08:20:21.225 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:21 smithi150 ceph-mon[29276]: 6.19 deep-scrub ok 2024-09-11T08:20:21.225 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:21 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:20:21.225 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:21 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:20:21.225 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:21 smithi150 ceph-mon[29276]: 5.1e scrub starts 2024-09-11T08:20:21.225 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:21 smithi150 ceph-mon[29276]: 3.11 scrub starts 2024-09-11T08:20:21.225 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:21 smithi150 ceph-mon[29276]: 5.1e scrub ok 2024-09-11T08:20:21.225 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:21 smithi150 ceph-mon[29276]: 3.11 scrub ok 2024-09-11T08:20:21.225 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:21 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/979546084' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:20:21.225 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:21 smithi150 ceph-mon[39612]: 3.4 scrub starts 2024-09-11T08:20:21.225 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:21 smithi150 ceph-mon[39612]: 3.4 scrub ok 2024-09-11T08:20:21.226 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:21 smithi150 ceph-mon[39612]: 3.8 scrub starts 2024-09-11T08:20:21.226 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:21 smithi150 ceph-mon[39612]: 3.8 scrub ok 2024-09-11T08:20:21.226 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:21 smithi150 ceph-mon[39612]: 6.19 deep-scrub starts 2024-09-11T08:20:21.226 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:21 smithi150 ceph-mon[39612]: 6.19 deep-scrub ok 2024-09-11T08:20:21.226 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:21 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:20:21.226 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:21 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:20:21.226 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:21 smithi150 ceph-mon[39612]: 5.1e scrub starts 2024-09-11T08:20:21.226 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:21 smithi150 ceph-mon[39612]: 3.11 scrub starts 2024-09-11T08:20:21.226 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:21 smithi150 ceph-mon[39612]: 5.1e scrub ok 2024-09-11T08:20:21.226 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:21 smithi150 ceph-mon[39612]: 3.11 scrub ok 2024-09-11T08:20:21.226 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:21 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/979546084' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:20:21.227 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:20:21.227 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:20:21.284 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:21.313 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:21.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:21.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:21.415 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:21.451 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:21.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:21.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:21.553 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:21.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:21 smithi181 ceph-mon[32701]: 3.4 scrub starts 2024-09-11T08:20:21.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:21 smithi181 ceph-mon[32701]: 3.4 scrub ok 2024-09-11T08:20:21.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:21 smithi181 ceph-mon[32701]: 3.8 scrub starts 2024-09-11T08:20:21.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:21 smithi181 ceph-mon[32701]: 3.8 scrub ok 2024-09-11T08:20:21.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:21 smithi181 ceph-mon[32701]: 6.19 deep-scrub starts 2024-09-11T08:20:21.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:21 smithi181 ceph-mon[32701]: 6.19 deep-scrub ok 2024-09-11T08:20:21.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:21 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:20:21.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:21 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:20:21.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:21 smithi181 ceph-mon[32701]: 5.1e scrub starts 2024-09-11T08:20:21.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:21 smithi181 ceph-mon[32701]: 3.11 scrub starts 2024-09-11T08:20:21.575 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:21 smithi181 ceph-mon[32701]: 5.1e scrub ok 2024-09-11T08:20:21.575 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:21 smithi181 ceph-mon[32701]: 3.11 scrub ok 2024-09-11T08:20:21.575 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:21 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/979546084' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:20:21.584 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:21.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:21.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:21.603 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-11T08:20:21.603 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T08:20:21.603 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-09-11T08:20:21.603 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-11T08:20:21.603 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-11T08:20:21.603 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-09-11T08:20:21.686 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:20:21.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:21.717 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:21.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:20:21.818 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:21.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:21.856 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:21.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:21.957 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:20:21.986 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:21.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:21.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:20:22.057 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:20:22.067 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T08:20:22.088 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:20:22.157 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:22.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:22.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:20:22.259 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:22.296 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:22.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:22.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:22.398 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:22.438 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:22.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:22.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:22.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:22 smithi150 ceph-mon[29276]: 9.b scrub starts 2024-09-11T08:20:22.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:22 smithi150 ceph-mon[29276]: 9.7 scrub starts 2024-09-11T08:20:22.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:22 smithi150 ceph-mon[29276]: pgmap v583: 148 pgs: 148 active+clean; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.07k op/s 2024-09-11T08:20:22.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:22 smithi150 ceph-mon[29276]: osdmap e287: 8 total, 8 up, 8 in 2024-09-11T08:20:22.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:22 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:20:22.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:22 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:20:22.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:22 smithi150 ceph-mon[29276]: 6.8 scrub starts 2024-09-11T08:20:22.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:22 smithi150 ceph-mon[29276]: 6.8 scrub ok 2024-09-11T08:20:22.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:22 smithi150 ceph-mon[39612]: 9.b scrub starts 2024-09-11T08:20:22.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:22 smithi150 ceph-mon[39612]: 9.7 scrub starts 2024-09-11T08:20:22.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:22 smithi150 ceph-mon[39612]: pgmap v583: 148 pgs: 148 active+clean; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.07k op/s 2024-09-11T08:20:22.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:22 smithi150 ceph-mon[39612]: osdmap e287: 8 total, 8 up, 8 in 2024-09-11T08:20:22.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:22 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:20:22.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:22 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:20:22.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:22 smithi150 ceph-mon[39612]: 6.8 scrub starts 2024-09-11T08:20:22.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:22 smithi150 ceph-mon[39612]: 6.8 scrub ok 2024-09-11T08:20:22.540 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:22.557 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:22 smithi181 ceph-mon[32701]: 9.b scrub starts 2024-09-11T08:20:22.557 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:22 smithi181 ceph-mon[32701]: 9.7 scrub starts 2024-09-11T08:20:22.557 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:22 smithi181 ceph-mon[32701]: pgmap v583: 148 pgs: 148 active+clean; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.07k op/s 2024-09-11T08:20:22.557 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:22 smithi181 ceph-mon[32701]: osdmap e287: 8 total, 8 up, 8 in 2024-09-11T08:20:22.557 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:22 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:20:22.557 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:22 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:20:22.557 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:22 smithi181 ceph-mon[32701]: 6.8 scrub starts 2024-09-11T08:20:22.557 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:22 smithi181 ceph-mon[32701]: 6.8 scrub ok 2024-09-11T08:20:22.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:22.577 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:22.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:22.678 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:22.714 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:22.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:22.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:22.816 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:20:22.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:22.854 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:22.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:20:22.955 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:22.985 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:23.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:23.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:23.125 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:20:23.178 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:23.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:23.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:20:23.279 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:23.313 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:23.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:23.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:23.414 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:20:23.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:23.454 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:23.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:20:23.458 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:23 smithi150 ceph-mon[29276]: 6.b scrub starts 2024-09-11T08:20:23.458 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:23 smithi150 ceph-mon[29276]: 6.b scrub ok 2024-09-11T08:20:23.458 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:23 smithi150 ceph-mon[29276]: 9.6 scrub starts 2024-09-11T08:20:23.459 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:23 smithi150 ceph-mon[29276]: 5.6 scrub starts 2024-09-11T08:20:23.459 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:23 smithi150 ceph-mon[29276]: 6.3 scrub starts 2024-09-11T08:20:23.459 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:23 smithi150 ceph-mon[29276]: 6.3 scrub ok 2024-09-11T08:20:23.459 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:23 smithi150 ceph-mon[39612]: 6.b scrub starts 2024-09-11T08:20:23.459 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:23 smithi150 ceph-mon[39612]: 6.b scrub ok 2024-09-11T08:20:23.459 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:23 smithi150 ceph-mon[39612]: 9.6 scrub starts 2024-09-11T08:20:23.459 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:23 smithi150 ceph-mon[39612]: 5.6 scrub starts 2024-09-11T08:20:23.459 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:23 smithi150 ceph-mon[39612]: 6.3 scrub starts 2024-09-11T08:20:23.459 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:23 smithi150 ceph-mon[39612]: 6.3 scrub ok 2024-09-11T08:20:23.556 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:23.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:23 smithi181 ceph-mon[32701]: 6.b scrub starts 2024-09-11T08:20:23.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:23 smithi181 ceph-mon[32701]: 6.b scrub ok 2024-09-11T08:20:23.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:23 smithi181 ceph-mon[32701]: 9.6 scrub starts 2024-09-11T08:20:23.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:23 smithi181 ceph-mon[32701]: 5.6 scrub starts 2024-09-11T08:20:23.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:23 smithi181 ceph-mon[32701]: 6.3 scrub starts 2024-09-11T08:20:23.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:23 smithi181 ceph-mon[32701]: 6.3 scrub ok 2024-09-11T08:20:23.588 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:23.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:23.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:23.617 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-09-11T08:20:23.617 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:20:23.689 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:20:23.743 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:23.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:23.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:20:23.845 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:23.881 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:23.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:23.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:23.889 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:20:23.983 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:24.020 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:24.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:24.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:24.122 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:24.153 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:24.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:24.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:24.188 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:24 smithi181 ceph-mon[32701]: 9.b scrub ok 2024-09-11T08:20:24.188 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:24 smithi181 ceph-mon[32701]: 9.9 scrub starts 2024-09-11T08:20:24.189 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:24 smithi181 ceph-mon[32701]: 5.17 scrub starts 2024-09-11T08:20:24.189 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:24 smithi181 ceph-mon[32701]: 5.17 scrub ok 2024-09-11T08:20:24.189 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:24 smithi181 ceph-mon[32701]: pgmap v585: 148 pgs: 148 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.20k op/s 2024-09-11T08:20:24.189 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:24 smithi181 ceph-mon[32701]: 9.7 scrub ok 2024-09-11T08:20:24.228 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-11T08:20:24.228 INFO:teuthology.orchestra.run.smithi181.stdout: "success": "" 2024-09-11T08:20:24.228 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-11T08:20:24.243 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T08:20:24.254 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:24.290 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:24.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:24.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:24.392 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:24.419 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:24.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:24.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:24.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:24 smithi150 ceph-mon[29276]: 9.b scrub ok 2024-09-11T08:20:24.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:24 smithi150 ceph-mon[29276]: 9.9 scrub starts 2024-09-11T08:20:24.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:24 smithi150 ceph-mon[29276]: 5.17 scrub starts 2024-09-11T08:20:24.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:24 smithi150 ceph-mon[29276]: 5.17 scrub ok 2024-09-11T08:20:24.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:24 smithi150 ceph-mon[29276]: pgmap v585: 148 pgs: 148 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.20k op/s 2024-09-11T08:20:24.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:24 smithi150 ceph-mon[29276]: 9.7 scrub ok 2024-09-11T08:20:24.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:24 smithi150 ceph-mon[39612]: 9.b scrub ok 2024-09-11T08:20:24.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:24 smithi150 ceph-mon[39612]: 9.9 scrub starts 2024-09-11T08:20:24.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:24 smithi150 ceph-mon[39612]: 5.17 scrub starts 2024-09-11T08:20:24.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:24 smithi150 ceph-mon[39612]: 5.17 scrub ok 2024-09-11T08:20:24.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:24 smithi150 ceph-mon[39612]: pgmap v585: 148 pgs: 148 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.20k op/s 2024-09-11T08:20:24.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:24 smithi150 ceph-mon[39612]: 9.7 scrub ok 2024-09-11T08:20:24.521 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:24.533 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-11T08:20:24.558 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:24.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:24.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:24.660 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:24.692 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:24.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:24.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:24.794 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:20:24.843 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:24.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:24.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:20:24.945 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:24.987 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:24.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:24.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:25.089 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:25.117 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:25.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:25.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:25.218 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:25.253 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:25.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:25.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:25.323 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:20:25 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:20:25] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:20:25.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:25 smithi181 ceph-mon[32701]: 6.16 scrub starts 2024-09-11T08:20:25.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:25 smithi181 ceph-mon[32701]: 6.16 scrub ok 2024-09-11T08:20:25.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:25 smithi181 ceph-mon[32701]: 9.6 scrub ok 2024-09-11T08:20:25.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:25 smithi181 ceph-mon[32701]: 4.11 scrub starts 2024-09-11T08:20:25.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:25 smithi181 ceph-mon[32701]: 4.11 scrub ok 2024-09-11T08:20:25.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:25 smithi181 ceph-mon[32701]: 4.a scrub starts 2024-09-11T08:20:25.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:25 smithi181 ceph-mon[32701]: 4.a scrub ok 2024-09-11T08:20:25.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:25 smithi181 ceph-mon[32701]: 9.9 scrub ok 2024-09-11T08:20:25.354 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:25.395 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:25.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:25.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:25.455 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:20:25.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:25 smithi150 ceph-mon[29276]: 6.16 scrub starts 2024-09-11T08:20:25.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:25 smithi150 ceph-mon[29276]: 6.16 scrub ok 2024-09-11T08:20:25.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:25 smithi150 ceph-mon[29276]: 9.6 scrub ok 2024-09-11T08:20:25.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:25 smithi150 ceph-mon[29276]: 4.11 scrub starts 2024-09-11T08:20:25.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:25 smithi150 ceph-mon[29276]: 4.11 scrub ok 2024-09-11T08:20:25.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:25 smithi150 ceph-mon[29276]: 4.a scrub starts 2024-09-11T08:20:25.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:25 smithi150 ceph-mon[29276]: 4.a scrub ok 2024-09-11T08:20:25.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:25 smithi150 ceph-mon[29276]: 9.9 scrub ok 2024-09-11T08:20:25.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:25 smithi150 ceph-mon[39612]: 6.16 scrub starts 2024-09-11T08:20:25.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:25 smithi150 ceph-mon[39612]: 6.16 scrub ok 2024-09-11T08:20:25.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:25 smithi150 ceph-mon[39612]: 9.6 scrub ok 2024-09-11T08:20:25.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:25 smithi150 ceph-mon[39612]: 4.11 scrub starts 2024-09-11T08:20:25.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:25 smithi150 ceph-mon[39612]: 4.11 scrub ok 2024-09-11T08:20:25.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:25 smithi150 ceph-mon[39612]: 4.a scrub starts 2024-09-11T08:20:25.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:25 smithi150 ceph-mon[39612]: 4.a scrub ok 2024-09-11T08:20:25.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:25 smithi150 ceph-mon[39612]: 9.9 scrub ok 2024-09-11T08:20:25.497 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:20:25.536 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:25.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:25.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:20:25.638 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:25.667 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:25.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:25.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:25.769 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:20:25.803 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:25.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:25.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:20:25.905 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:20:25.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:25.946 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:25.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:20:26.047 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:20:26.099 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:26.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:26.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:20:26.201 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:26.233 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:26.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:26.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:26.297 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:26 smithi150 ceph-mon[29276]: 4.6 deep-scrub starts 2024-09-11T08:20:26.297 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:26 smithi150 ceph-mon[29276]: 4.6 deep-scrub ok 2024-09-11T08:20:26.297 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:26 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:20:26.297 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:26 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:20:26.297 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:26 smithi150 ceph-mon[29276]: 3.10 scrub starts 2024-09-11T08:20:26.297 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:26 smithi150 ceph-mon[29276]: 3.10 scrub ok 2024-09-11T08:20:26.297 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:26 smithi150 ceph-mon[29276]: pgmap v586: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.55k op/s 2024-09-11T08:20:26.297 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:26 smithi150 ceph-mon[29276]: osdmap e288: 8 total, 8 up, 8 in 2024-09-11T08:20:26.297 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:26 smithi150 ceph-mon[29276]: 5.4 deep-scrub starts 2024-09-11T08:20:26.297 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:26 smithi150 ceph-mon[29276]: 5.4 deep-scrub ok 2024-09-11T08:20:26.297 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:26 smithi150 ceph-mon[29276]: 5.3 scrub starts 2024-09-11T08:20:26.297 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:26 smithi150 ceph-mon[29276]: 5.3 scrub ok 2024-09-11T08:20:26.298 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:26 smithi150 ceph-mon[39612]: 4.6 deep-scrub starts 2024-09-11T08:20:26.298 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:26 smithi150 ceph-mon[39612]: 4.6 deep-scrub ok 2024-09-11T08:20:26.298 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:26 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:20:26.298 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:26 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:20:26.298 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:26 smithi150 ceph-mon[39612]: 3.10 scrub starts 2024-09-11T08:20:26.298 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:26 smithi150 ceph-mon[39612]: 3.10 scrub ok 2024-09-11T08:20:26.298 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:26 smithi150 ceph-mon[39612]: pgmap v586: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.55k op/s 2024-09-11T08:20:26.298 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:26 smithi150 ceph-mon[39612]: osdmap e288: 8 total, 8 up, 8 in 2024-09-11T08:20:26.298 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:26 smithi150 ceph-mon[39612]: 5.4 deep-scrub starts 2024-09-11T08:20:26.298 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:26 smithi150 ceph-mon[39612]: 5.4 deep-scrub ok 2024-09-11T08:20:26.298 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:26 smithi150 ceph-mon[39612]: 5.3 scrub starts 2024-09-11T08:20:26.298 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:26 smithi150 ceph-mon[39612]: 5.3 scrub ok 2024-09-11T08:20:26.335 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:20:26.376 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:26.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:26.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:20:26.477 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:26.505 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:26.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:26.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:26.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:26 smithi181 ceph-mon[32701]: 4.6 deep-scrub starts 2024-09-11T08:20:26.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:26 smithi181 ceph-mon[32701]: 4.6 deep-scrub ok 2024-09-11T08:20:26.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:26 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:20:26.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:26 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:20:26.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:26 smithi181 ceph-mon[32701]: 3.10 scrub starts 2024-09-11T08:20:26.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:26 smithi181 ceph-mon[32701]: 3.10 scrub ok 2024-09-11T08:20:26.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:26 smithi181 ceph-mon[32701]: pgmap v586: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.55k op/s 2024-09-11T08:20:26.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:26 smithi181 ceph-mon[32701]: osdmap e288: 8 total, 8 up, 8 in 2024-09-11T08:20:26.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:26 smithi181 ceph-mon[32701]: 5.4 deep-scrub starts 2024-09-11T08:20:26.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:26 smithi181 ceph-mon[32701]: 5.4 deep-scrub ok 2024-09-11T08:20:26.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:26 smithi181 ceph-mon[32701]: 5.3 scrub starts 2024-09-11T08:20:26.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:26 smithi181 ceph-mon[32701]: 5.3 scrub ok 2024-09-11T08:20:26.607 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:20:26.641 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:26.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:26.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:20:26.743 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:26.764 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:20:26.780 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:26.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:26.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:26.882 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:20:26.917 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:26.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:26.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:20:27.020 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:20:27.061 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:27.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:27.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:20:27.162 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:27.193 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:27.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:27.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:27.295 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:27.327 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:27.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:27.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:27.429 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:27.461 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:27.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:27.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:27.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:27 smithi150 ceph-mon[29276]: 9.4 scrub starts 2024-09-11T08:20:27.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:27 smithi150 ceph-mon[29276]: 9.f scrub starts 2024-09-11T08:20:27.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:27 smithi150 ceph-mon[39612]: 9.4 scrub starts 2024-09-11T08:20:27.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:27 smithi150 ceph-mon[39612]: 9.f scrub starts 2024-09-11T08:20:27.563 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:27.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:27 smithi181 ceph-mon[32701]: 9.4 scrub starts 2024-09-11T08:20:27.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:27 smithi181 ceph-mon[32701]: 9.f scrub starts 2024-09-11T08:20:27.593 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:27.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:27.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:27.695 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:27.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:27.729 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:27.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:27.830 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:27.862 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:27.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:27.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:27.964 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:20:27.998 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:27.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:27.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:20:28.100 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:20:28.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:28.136 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:28.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:20:28.237 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:28.277 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:28.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:28.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:28.379 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:28.417 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:28.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:28.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:28.458 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:28 smithi181 ceph-mon[32701]: 3.6 scrub starts 2024-09-11T08:20:28.458 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:28 smithi181 ceph-mon[32701]: 3.6 scrub ok 2024-09-11T08:20:28.458 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:28 smithi181 ceph-mon[32701]: pgmap v588: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 748 op/s 2024-09-11T08:20:28.458 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:28 smithi181 ceph-mon[32701]: 9.a deep-scrub starts 2024-09-11T08:20:28.458 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:28 smithi181 ceph-mon[32701]: 9.4 scrub ok 2024-09-11T08:20:28.458 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:28 smithi181 ceph-mon[32701]: 3.14 scrub starts 2024-09-11T08:20:28.458 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:28 smithi181 ceph-mon[32701]: 3.14 scrub ok 2024-09-11T08:20:28.458 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:28 smithi181 ceph-mon[32701]: 4.7 scrub starts 2024-09-11T08:20:28.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:28 smithi150 ceph-mon[29276]: 3.6 scrub starts 2024-09-11T08:20:28.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:28 smithi150 ceph-mon[29276]: 3.6 scrub ok 2024-09-11T08:20:28.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:28 smithi150 ceph-mon[29276]: pgmap v588: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 748 op/s 2024-09-11T08:20:28.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:28 smithi150 ceph-mon[29276]: 9.a deep-scrub starts 2024-09-11T08:20:28.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:28 smithi150 ceph-mon[29276]: 9.4 scrub ok 2024-09-11T08:20:28.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:28 smithi150 ceph-mon[29276]: 3.14 scrub starts 2024-09-11T08:20:28.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:28 smithi150 ceph-mon[29276]: 3.14 scrub ok 2024-09-11T08:20:28.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:28 smithi150 ceph-mon[29276]: 4.7 scrub starts 2024-09-11T08:20:28.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:28 smithi150 ceph-mon[39612]: 3.6 scrub starts 2024-09-11T08:20:28.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:28 smithi150 ceph-mon[39612]: 3.6 scrub ok 2024-09-11T08:20:28.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:28 smithi150 ceph-mon[39612]: pgmap v588: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 748 op/s 2024-09-11T08:20:28.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:28 smithi150 ceph-mon[39612]: 9.a deep-scrub starts 2024-09-11T08:20:28.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:28 smithi150 ceph-mon[39612]: 9.4 scrub ok 2024-09-11T08:20:28.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:28 smithi150 ceph-mon[39612]: 3.14 scrub starts 2024-09-11T08:20:28.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:28 smithi150 ceph-mon[39612]: 3.14 scrub ok 2024-09-11T08:20:28.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:28 smithi150 ceph-mon[39612]: 4.7 scrub starts 2024-09-11T08:20:28.519 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:20:28.552 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:28.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:28.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:20:28.654 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:28.685 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:28.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:28.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:28.787 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:28.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:28.820 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:28.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:28.823 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:20:28 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:20:28.922 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:28.958 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:28.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:28.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:28.988 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:20:28 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:20:28] "GET /metrics HTTP/1.1" 200 224555 "" "Prometheus/2.43.0" 2024-09-11T08:20:29.059 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:29.098 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:29.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:29.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:29.200 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:20:29.224 INFO:teuthology.orchestra.run.smithi150.stderr:noscrub is set 2024-09-11T08:20:29.232 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:29.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:29.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:20:29.335 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:20:29.363 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:29.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:29.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:20:29.465 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:29.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:29 smithi150 ceph-mon[29276]: 9.f scrub ok 2024-09-11T08:20:29.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:29 smithi150 ceph-mon[29276]: 4.7 scrub ok 2024-09-11T08:20:29.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:29 smithi150 ceph-mon[29276]: 6.2 scrub starts 2024-09-11T08:20:29.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:29 smithi150 ceph-mon[29276]: 6.2 scrub ok 2024-09-11T08:20:29.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:29 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2767179735' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:20:29.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:29 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:20:29.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:29 smithi150 ceph-mon[29276]: 9.3 deep-scrub starts 2024-09-11T08:20:29.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:29 smithi150 ceph-mon[29276]: 4.1f deep-scrub starts 2024-09-11T08:20:29.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:29 smithi150 ceph-mon[29276]: 4.1f deep-scrub ok 2024-09-11T08:20:29.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:29 smithi150 ceph-mon[39612]: 9.f scrub ok 2024-09-11T08:20:29.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:29 smithi150 ceph-mon[39612]: 4.7 scrub ok 2024-09-11T08:20:29.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:29 smithi150 ceph-mon[39612]: 6.2 scrub starts 2024-09-11T08:20:29.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:29 smithi150 ceph-mon[39612]: 6.2 scrub ok 2024-09-11T08:20:29.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:29 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2767179735' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:20:29.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:29 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:20:29.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:29 smithi150 ceph-mon[39612]: 9.3 deep-scrub starts 2024-09-11T08:20:29.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:29 smithi150 ceph-mon[39612]: 4.1f deep-scrub starts 2024-09-11T08:20:29.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:29 smithi150 ceph-mon[39612]: 4.1f deep-scrub ok 2024-09-11T08:20:29.495 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:29.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:29.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:29.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:29 smithi181 ceph-mon[32701]: 9.f scrub ok 2024-09-11T08:20:29.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:29 smithi181 ceph-mon[32701]: 4.7 scrub ok 2024-09-11T08:20:29.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:29 smithi181 ceph-mon[32701]: 6.2 scrub starts 2024-09-11T08:20:29.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:29 smithi181 ceph-mon[32701]: 6.2 scrub ok 2024-09-11T08:20:29.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:29 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2767179735' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:20:29.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:29 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:20:29.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:29 smithi181 ceph-mon[32701]: 9.3 deep-scrub starts 2024-09-11T08:20:29.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:29 smithi181 ceph-mon[32701]: 4.1f deep-scrub starts 2024-09-11T08:20:29.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:29 smithi181 ceph-mon[32701]: 4.1f deep-scrub ok 2024-09-11T08:20:29.597 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:20:29.634 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:29.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:29.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:20:29.736 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:29.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:29.769 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:29.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:29.870 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:20:29.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:29.903 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:29.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:20:29.989 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-11T08:20:29.990 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T08:20:29.990 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-09-11T08:20:29.991 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-11T08:20:29.991 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-11T08:20:29.991 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-09-11T08:20:30.005 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:30.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:30.074 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:30.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:30.175 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:30.212 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:30.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:30.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:30.314 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:30.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:30 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:20:30.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:30 smithi181 ceph-mon[32701]: 6.0 scrub starts 2024-09-11T08:20:30.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:30 smithi181 ceph-mon[32701]: 6.0 scrub ok 2024-09-11T08:20:30.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:30 smithi181 ceph-mon[32701]: pgmap v589: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 3.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.46k op/s 2024-09-11T08:20:30.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:30 smithi181 ceph-mon[32701]: 6.7 scrub starts 2024-09-11T08:20:30.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:30 smithi181 ceph-mon[32701]: 6.7 scrub ok 2024-09-11T08:20:30.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:30 smithi181 ceph-mon[32701]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:20:30.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:30 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:20:30.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:30 smithi181 ceph-mon[32701]: osdmap e289: 8 total, 8 up, 8 in 2024-09-11T08:20:30.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:30 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:20:30.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:30 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:20:30.350 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:30.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:30.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:30.419 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:20:30.453 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:30.480 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:30 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:20:30.480 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:30 smithi150 ceph-mon[39612]: 6.0 scrub starts 2024-09-11T08:20:30.480 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:30 smithi150 ceph-mon[39612]: 6.0 scrub ok 2024-09-11T08:20:30.480 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:30 smithi150 ceph-mon[39612]: pgmap v589: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 3.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.46k op/s 2024-09-11T08:20:30.480 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:30 smithi150 ceph-mon[39612]: 6.7 scrub starts 2024-09-11T08:20:30.480 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:30 smithi150 ceph-mon[39612]: 6.7 scrub ok 2024-09-11T08:20:30.480 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:30 smithi150 ceph-mon[39612]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:20:30.480 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:30 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:20:30.480 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:30 smithi150 ceph-mon[39612]: osdmap e289: 8 total, 8 up, 8 in 2024-09-11T08:20:30.481 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:30 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:20:30.481 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:30 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:20:30.481 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:30 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:20:30.481 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:30 smithi150 ceph-mon[29276]: 6.0 scrub starts 2024-09-11T08:20:30.481 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:30 smithi150 ceph-mon[29276]: 6.0 scrub ok 2024-09-11T08:20:30.481 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:30 smithi150 ceph-mon[29276]: pgmap v589: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 3.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.46k op/s 2024-09-11T08:20:30.481 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:30 smithi150 ceph-mon[29276]: 6.7 scrub starts 2024-09-11T08:20:30.481 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:30 smithi150 ceph-mon[29276]: 6.7 scrub ok 2024-09-11T08:20:30.481 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:30 smithi150 ceph-mon[29276]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:20:30.482 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:30 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:20:30.482 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:30 smithi150 ceph-mon[29276]: osdmap e289: 8 total, 8 up, 8 in 2024-09-11T08:20:30.482 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:30 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:20:30.482 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:30 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:20:30.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:30.509 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:30.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:30.610 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:20:30.640 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:30.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:30.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:20:30.742 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:30.779 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:30.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:30.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:30.881 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:30.914 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:30.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:30.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:31.016 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:31.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:31.051 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:31.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:31.152 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:20:31.187 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:31.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:31.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:20:31.289 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:31.326 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:31.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:31.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:31.428 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:31.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:31.466 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:31.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:31.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:31 smithi150 ceph-mon[29276]: osdmap e290: 8 total, 8 up, 8 in 2024-09-11T08:20:31.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:31 smithi150 ceph-mon[39612]: osdmap e290: 8 total, 8 up, 8 in 2024-09-11T08:20:31.567 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:20:31.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:31 smithi181 ceph-mon[32701]: osdmap e290: 8 total, 8 up, 8 in 2024-09-11T08:20:31.598 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:31.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:31.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:20:31.700 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:31.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:31.742 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:31.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:31.843 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:20:31.876 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:31.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:31.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:20:31.978 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:32.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:32.017 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:32.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:32.118 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:32.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:32.157 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:32.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:32.258 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:32.296 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:32.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:32.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:32.329 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-11T08:20:32.329 INFO:teuthology.orchestra.run.smithi181.stdout: "success": "" 2024-09-11T08:20:32.329 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-11T08:20:32.399 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:32.426 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:32.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:32.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:32.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:32 smithi150 ceph-mon[29276]: pgmap v592: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 3.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 950 op/s 2024-09-11T08:20:32.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:32 smithi150 ceph-mon[39612]: pgmap v592: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 3.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 950 op/s 2024-09-11T08:20:32.528 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:32.572 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:32.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:32 smithi181 ceph-mon[32701]: pgmap v592: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 3.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 950 op/s 2024-09-11T08:20:32.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:32.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:32.675 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:20:32.710 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:32.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:32.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:20:32.730 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:20:32.730 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:20:32.730 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:20:32.730 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:20:32.747 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:20:32.747 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:20:32.779 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:20:32.780 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:20:32.808 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:20:32.808 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:20:32.812 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:20:32.839 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:20:32.839 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:20:32.851 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:32.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:32.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:20:32.870 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:20:32.870 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:20:32.899 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:20:32.900 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:20:32.938 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T08:20:32.952 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:33.017 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:33.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:33.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:33.119 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:33.155 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:33.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:33.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:33.257 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:20:33.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:33.300 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:33.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:20:33.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:33 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2681339556' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:20:33.401 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:33.435 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:33.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:33.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:33.537 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:33.569 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:33.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:33.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:33.671 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:20:33.672 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:33 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2681339556' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:20:33.672 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:33 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2681339556' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:20:33.704 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:33.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:33.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:20:33.806 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:33.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:33.837 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:33.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:33.938 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:33.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:33.973 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:33.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:34.073 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:34.109 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:34.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:34.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:34.211 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:34.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:34.244 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:34.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:34.345 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:34.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:34.381 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:34.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:34.482 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:34.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:34.521 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:34.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:34.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:34 smithi181 ceph-mon[32701]: pgmap v593: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 921 op/s 2024-09-11T08:20:34.622 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:34.661 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:34.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:34.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:34.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:34 smithi150 ceph-mon[29276]: pgmap v593: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 921 op/s 2024-09-11T08:20:34.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:34 smithi150 ceph-mon[39612]: pgmap v593: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 921 op/s 2024-09-11T08:20:34.763 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:34.795 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:34.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:34.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:34.897 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:34.927 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:34.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:34.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:35.029 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:35.057 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:35.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:35.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:35.159 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:35.195 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:35.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:35.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:35.297 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:20:35.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:35.319 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:35.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:20:35.421 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:35.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:35.453 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:35.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:35.554 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:35.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:20:35 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:20:35] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:20:35.587 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:35.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:35.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:35.647 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:20:35.689 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:35.731 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:35.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:35.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:35.833 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:20:35.865 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:35.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:35.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:20:35.920 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-11T08:20:35.921 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 20, 2024-09-11T08:20:35.921 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 600, 2024-09-11T08:20:35.921 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [ 2024-09-11T08:20:35.921 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.921 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:107976 7.3 7:c68b5db5:::benchmark_data_smithi150_78508_object107975:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-11T08:20:35.921 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:10.905908+0000", 2024-09-11T08:20:35.921 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 565.01136807900002, 2024-09-11T08:20:35.921 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.13184691800000001, 2024-09-11T08:20:35.921 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:20:35.921 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:20:35.921 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:20:35.921 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:20:35.921 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:20:35.922 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 107976 2024-09-11T08:20:35.922 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.922 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:20:35.922 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.922 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:20:35.922 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.905908+0000", 2024-09-11T08:20:35.922 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:20:35.922 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.922 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.922 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:20:35.922 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.905908+0000", 2024-09-11T08:20:35.922 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:20:35.922 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.923 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.923 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:20:35.923 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.905908+0000", 2024-09-11T08:20:35.923 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:20:35.923 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.923 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.923 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:20:35.923 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.905927+0000", 2024-09-11T08:20:35.923 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.9598000000000001e-05 2024-09-11T08:20:35.923 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.923 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.923 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:20:35.923 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.905933+0000", 2024-09-11T08:20:35.924 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.9920000000000004e-06 2024-09-11T08:20:35.924 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.924 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.924 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:20:35.924 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.905937+0000", 2024-09-11T08:20:35.924 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.4120000000000001e-06 2024-09-11T08:20:35.924 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.924 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.924 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:20:35.924 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.910154+0000", 2024-09-11T08:20:35.924 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0042179319999999998 2024-09-11T08:20:35.924 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.925 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.925 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:20:35.925 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.910266+0000", 2024-09-11T08:20:35.925 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000111703 2024-09-11T08:20:35.925 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.925 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.925 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 2,4", 2024-09-11T08:20:35.925 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.910399+0000", 2024-09-11T08:20:35.925 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000132442 2024-09-11T08:20:35.925 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.925 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.925 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:20:35.925 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.915798+0000", 2024-09-11T08:20:35.925 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0053989679999999996 2024-09-11T08:20:35.926 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.926 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.926 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:20:35.926 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.915819+0000", 2024-09-11T08:20:35.926 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.1834000000000001e-05 2024-09-11T08:20:35.926 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.926 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.926 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:20:35.926 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.037586+0000", 2024-09-11T08:20:35.926 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.121766551 2024-09-11T08:20:35.926 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.926 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.926 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:20:35.927 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.037714+0000", 2024-09-11T08:20:35.927 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012809400000000001 2024-09-11T08:20:35.927 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.927 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.927 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:20:35.927 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.037755+0000", 2024-09-11T08:20:35.927 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.0873e-05 2024-09-11T08:20:35.927 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:35.927 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:35.927 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:35.927 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.927 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.927 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:107979 7.2 7:40eeebfe:::benchmark_data_smithi150_78508_object107978:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-11T08:20:35.927 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:10.911896+0000", 2024-09-11T08:20:35.928 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 565.00537993299997, 2024-09-11T08:20:35.928 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.11813148900000001, 2024-09-11T08:20:35.928 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:20:35.928 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:20:35.928 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:20:35.928 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:20:35.928 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:20:35.928 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 107979 2024-09-11T08:20:35.928 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.928 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:20:35.928 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.928 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:20:35.928 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.911896+0000", 2024-09-11T08:20:35.928 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:20:35.929 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.929 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.929 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:20:35.929 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.911896+0000", 2024-09-11T08:20:35.929 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:20:35.929 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.929 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.929 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:20:35.929 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.911894+0000", 2024-09-11T08:20:35.929 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:20:35.929 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.929 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.929 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:20:35.929 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.911918+0000", 2024-09-11T08:20:35.930 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.3295e-05 2024-09-11T08:20:35.930 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.930 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.930 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:20:35.930 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.911926+0000", 2024-09-11T08:20:35.930 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.5550000000000003e-06 2024-09-11T08:20:35.930 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.930 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.930 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:20:35.930 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.911931+0000", 2024-09-11T08:20:35.930 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.9609999999999996e-06 2024-09-11T08:20:35.930 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.930 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.930 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:20:35.931 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.911998+0000", 2024-09-11T08:20:35.931 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.6533999999999997e-05 2024-09-11T08:20:35.931 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.931 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.931 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:20:35.931 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.912108+0000", 2024-09-11T08:20:35.931 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000110637 2024-09-11T08:20:35.931 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.931 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.931 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 1,2", 2024-09-11T08:20:35.931 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.912236+0000", 2024-09-11T08:20:35.931 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012750500000000001 2024-09-11T08:20:35.931 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.931 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.932 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:20:35.932 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.915433+0000", 2024-09-11T08:20:35.932 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0031969540000000001 2024-09-11T08:20:35.932 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.932 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.932 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:20:35.932 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.029826+0000", 2024-09-11T08:20:35.932 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.11439272 2024-09-11T08:20:35.932 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.932 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.932 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:20:35.932 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.029947+0000", 2024-09-11T08:20:35.932 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012175700000000001 2024-09-11T08:20:35.932 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.933 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.933 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:20:35.933 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.029989+0000", 2024-09-11T08:20:35.933 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1233000000000003e-05 2024-09-11T08:20:35.933 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.933 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.933 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:20:35.933 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.030028+0000", 2024-09-11T08:20:35.933 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.9057999999999998e-05 2024-09-11T08:20:35.933 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:35.933 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:35.933 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:35.933 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.933 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.934 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:107984 7.a 7:5f828c6a:::benchmark_data_smithi150_78508_object107983:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-11T08:20:35.934 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:10.912870+0000", 2024-09-11T08:20:35.934 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 565.00440560000004, 2024-09-11T08:20:35.934 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.117220537, 2024-09-11T08:20:35.934 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:20:35.934 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:20:35.934 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:20:35.934 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:20:35.935 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:20:35.935 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 107984 2024-09-11T08:20:35.935 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.935 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:20:35.935 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.935 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:20:35.935 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.912870+0000", 2024-09-11T08:20:35.935 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:20:35.935 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.935 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.935 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:20:35.935 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.912870+0000", 2024-09-11T08:20:35.936 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:20:35.936 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.936 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.936 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:20:35.936 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.912869+0000", 2024-09-11T08:20:35.936 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:20:35.936 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.936 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.936 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:20:35.936 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.912893+0000", 2024-09-11T08:20:35.936 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.3264e-05 2024-09-11T08:20:35.936 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.936 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.936 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:20:35.937 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.912903+0000", 2024-09-11T08:20:35.937 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0557000000000001e-05 2024-09-11T08:20:35.937 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.937 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.937 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:20:35.937 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.912909+0000", 2024-09-11T08:20:35.937 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.8649999999999998e-06 2024-09-11T08:20:35.937 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.937 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.937 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:20:35.937 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.915007+0000", 2024-09-11T08:20:35.937 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0020981469999999999 2024-09-11T08:20:35.937 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.938 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.938 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:20:35.938 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.915120+0000", 2024-09-11T08:20:35.938 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000112891 2024-09-11T08:20:35.938 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.938 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.938 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 2,5", 2024-09-11T08:20:35.938 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.915248+0000", 2024-09-11T08:20:35.938 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012757299999999999 2024-09-11T08:20:35.938 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.938 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.938 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:20:35.938 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.029913+0000", 2024-09-11T08:20:35.938 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.114665409 2024-09-11T08:20:35.939 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.939 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.939 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:20:35.939 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.029939+0000", 2024-09-11T08:20:35.939 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.6202000000000001e-05 2024-09-11T08:20:35.939 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.939 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.939 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:20:35.939 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.030034+0000", 2024-09-11T08:20:35.939 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.5186999999999995e-05 2024-09-11T08:20:35.939 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.939 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.939 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:20:35.939 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.030064+0000", 2024-09-11T08:20:35.939 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.9859e-05 2024-09-11T08:20:35.940 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.940 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.940 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:20:35.940 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.030091+0000", 2024-09-11T08:20:35.940 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.6653e-05 2024-09-11T08:20:35.940 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:35.940 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:35.940 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:35.940 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.940 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.940 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.14790.0:107982 7.6 e122/91 7:665c1af9:::benchmark_data_smithi150_78508_object107981:head v 122'6801, mlcod=122'6799)", 2024-09-11T08:20:35.940 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:10.915094+0000", 2024-09-11T08:20:35.940 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 565.00218192499995, 2024-09-11T08:20:35.940 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.114003855, 2024-09-11T08:20:35.941 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:20:35.941 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:20:35.941 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:20:35.941 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.941 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:20:35.941 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.915094+0000", 2024-09-11T08:20:35.941 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:20:35.941 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.941 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.941 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:20:35.941 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.915094+0000", 2024-09-11T08:20:35.941 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:20:35.941 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.942 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.942 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:20:35.942 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.915094+0000", 2024-09-11T08:20:35.942 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:20:35.942 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.942 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.942 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:20:35.942 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.915116+0000", 2024-09-11T08:20:35.942 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.2938e-05 2024-09-11T08:20:35.942 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.942 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.942 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:20:35.942 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.915126+0000", 2024-09-11T08:20:35.942 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.5719999999999992e-06 2024-09-11T08:20:35.943 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.943 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.943 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:20:35.943 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.915131+0000", 2024-09-11T08:20:35.943 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.6e-06 2024-09-11T08:20:35.943 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.943 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.943 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:20:35.943 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.915190+0000", 2024-09-11T08:20:35.943 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.9357000000000001e-05 2024-09-11T08:20:35.943 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.943 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.943 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:20:35.943 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.915208+0000", 2024-09-11T08:20:35.944 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.8294000000000001e-05 2024-09-11T08:20:35.944 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.944 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.944 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:20:35.944 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.029054+0000", 2024-09-11T08:20:35.944 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.113845218 2024-09-11T08:20:35.944 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.944 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.944 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:20:35.944 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.029098+0000", 2024-09-11T08:20:35.944 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.4490000000000003e-05 2024-09-11T08:20:35.944 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:35.944 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:35.945 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:35.945 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.945 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.945 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:107985 7.3 7:c59c4586:::benchmark_data_smithi150_78508_object107984:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-11T08:20:35.945 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:10.916263+0000", 2024-09-11T08:20:35.945 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 565.001012769, 2024-09-11T08:20:35.945 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.121624965, 2024-09-11T08:20:35.945 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:20:35.945 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:20:35.945 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:20:35.945 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:20:35.945 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:20:35.945 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 107985 2024-09-11T08:20:35.946 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.946 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:20:35.946 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.946 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:20:35.946 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.916263+0000", 2024-09-11T08:20:35.946 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:20:35.946 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.946 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.946 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:20:35.946 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.916263+0000", 2024-09-11T08:20:35.946 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:20:35.946 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.946 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.946 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:20:35.947 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.916262+0000", 2024-09-11T08:20:35.947 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:20:35.947 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.947 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.947 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:20:35.947 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.916284+0000", 2024-09-11T08:20:35.947 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.2646e-05 2024-09-11T08:20:35.947 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.947 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.947 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:20:35.947 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.916372+0000", 2024-09-11T08:20:35.947 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.7496000000000004e-05 2024-09-11T08:20:35.947 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.947 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.948 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:20:35.948 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.916379+0000", 2024-09-11T08:20:35.948 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.2910000000000002e-06 2024-09-11T08:20:35.948 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.948 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.948 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:20:35.948 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.916412+0000", 2024-09-11T08:20:35.948 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.3021000000000001e-05 2024-09-11T08:20:35.948 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.948 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.948 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:20:35.948 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.916534+0000", 2024-09-11T08:20:35.948 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000122096 2024-09-11T08:20:35.948 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.949 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.949 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 2,4", 2024-09-11T08:20:35.949 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.916661+0000", 2024-09-11T08:20:35.949 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000126811 2024-09-11T08:20:35.949 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.949 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.949 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:20:35.949 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.032290+0000", 2024-09-11T08:20:35.949 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.115629047 2024-09-11T08:20:35.949 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.949 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.949 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:20:35.949 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.032314+0000", 2024-09-11T08:20:35.950 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.4219e-05 2024-09-11T08:20:35.950 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.950 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.950 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:20:35.950 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.037765+0000", 2024-09-11T08:20:35.950 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0054504150000000001 2024-09-11T08:20:35.950 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.950 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.950 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:20:35.950 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.037850+0000", 2024-09-11T08:20:35.950 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.4890999999999997e-05 2024-09-11T08:20:35.950 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.950 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.950 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:20:35.951 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.037888+0000", 2024-09-11T08:20:35.951 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.8578000000000002e-05 2024-09-11T08:20:35.951 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:35.951 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:35.951 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:35.951 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.951 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.951 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop_reply(client.14790.0:107979 7.2 e122/91 ondisk, result = 0)", 2024-09-11T08:20:35.951 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:10.916478+0000", 2024-09-11T08:20:35.951 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 565.000797885, 2024-09-11T08:20:35.951 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.113354038, 2024-09-11T08:20:35.951 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:20:35.951 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "started", 2024-09-11T08:20:35.951 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:20:35.952 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.952 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:20:35.952 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.916478+0000", 2024-09-11T08:20:35.952 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:20:35.952 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.952 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.952 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:20:35.952 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.916478+0000", 2024-09-11T08:20:35.952 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:20:35.952 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.952 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.952 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:20:35.952 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.916478+0000", 2024-09-11T08:20:35.952 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:20:35.953 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.953 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.953 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:20:35.953 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.916482+0000", 2024-09-11T08:20:35.953 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.8300000000000003e-06 2024-09-11T08:20:35.953 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.953 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.953 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:20:35.953 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.916490+0000", 2024-09-11T08:20:35.953 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.6979999999999998e-06 2024-09-11T08:20:35.953 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.953 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.953 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:20:35.954 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.916495+0000", 2024-09-11T08:20:35.954 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.5290000000000002e-06 2024-09-11T08:20:35.954 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.954 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.954 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:20:35.954 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.916753+0000", 2024-09-11T08:20:35.954 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00025800699999999998 2024-09-11T08:20:35.954 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.954 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.954 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:20:35.954 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.916759+0000", 2024-09-11T08:20:35.954 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.2360000000000001e-06 2024-09-11T08:20:35.954 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.954 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.955 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:20:35.955 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.029832+0000", 2024-09-11T08:20:35.955 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.113073362 2024-09-11T08:20:35.955 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:35.955 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:35.955 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:35.955 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.955 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.955 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.14790.0:107989 7.6 e122/91 7:6298c77c:::benchmark_data_smithi150_78508_object107988:head v 122'6802, mlcod=122'6799)", 2024-09-11T08:20:35.955 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:10.920278+0000", 2024-09-11T08:20:35.955 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 564.99699850000002, 2024-09-11T08:20:35.955 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.116243732, 2024-09-11T08:20:35.955 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:20:35.955 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:20:35.956 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:20:35.956 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.956 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:20:35.956 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.920278+0000", 2024-09-11T08:20:35.956 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:20:35.956 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.956 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.956 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:20:35.956 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.920278+0000", 2024-09-11T08:20:35.956 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:20:35.956 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.956 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.956 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:20:35.956 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.920277+0000", 2024-09-11T08:20:35.957 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:20:35.957 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.957 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.957 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:20:35.957 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:10.920302+0000", 2024-09-11T08:20:35.957 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.4824000000000002e-05 2024-09-11T08:20:35.957 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.957 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.957 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:20:35.957 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.030110+0000", 2024-09-11T08:20:35.957 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.10980841500000001 2024-09-11T08:20:35.957 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.957 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.958 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:20:35.958 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.030116+0000", 2024-09-11T08:20:35.958 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.6319999999999996e-06 2024-09-11T08:20:35.958 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.958 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.958 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:20:35.958 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.030163+0000", 2024-09-11T08:20:35.958 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.7346999999999999e-05 2024-09-11T08:20:35.958 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.958 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.958 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:20:35.958 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.030182+0000", 2024-09-11T08:20:35.958 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.8726999999999999e-05 2024-09-11T08:20:35.958 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.959 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.959 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:20:35.959 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.036491+0000", 2024-09-11T08:20:35.959 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0063095959999999998 2024-09-11T08:20:35.959 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.959 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.959 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:20:35.959 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.036521+0000", 2024-09-11T08:20:35.959 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.0018999999999998e-05 2024-09-11T08:20:35.959 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:35.959 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:35.959 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:35.959 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.959 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.959 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.14790.0:108642 7.f e122/91 7:f6bc24da:::benchmark_data_smithi150_78508_object108641:head v 122'6743, mlcod=122'6741)", 2024-09-11T08:20:35.960 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:11.612981+0000", 2024-09-11T08:20:35.960 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 564.30429489599999, 2024-09-11T08:20:35.960 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.118402496, 2024-09-11T08:20:35.960 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:20:35.960 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:20:35.960 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:20:35.960 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.960 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:20:35.960 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.612981+0000", 2024-09-11T08:20:35.960 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:20:35.960 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.960 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.960 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:20:35.961 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.612981+0000", 2024-09-11T08:20:35.961 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:20:35.961 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.961 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.961 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:20:35.961 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.612981+0000", 2024-09-11T08:20:35.961 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:20:35.961 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.961 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.961 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:20:35.961 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.612999+0000", 2024-09-11T08:20:35.961 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.8606000000000001e-05 2024-09-11T08:20:35.961 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.961 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.962 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:20:35.962 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.613005+0000", 2024-09-11T08:20:35.962 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.7339999999999996e-06 2024-09-11T08:20:35.962 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.962 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.962 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:20:35.962 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.613009+0000", 2024-09-11T08:20:35.962 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.7919999999999999e-06 2024-09-11T08:20:35.962 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.962 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.962 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:20:35.962 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.613168+0000", 2024-09-11T08:20:35.962 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00015886399999999999 2024-09-11T08:20:35.962 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.963 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.963 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:20:35.963 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.613215+0000", 2024-09-11T08:20:35.963 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.7769999999999998e-05 2024-09-11T08:20:35.963 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.964 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.964 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:20:35.964 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.731349+0000", 2024-09-11T08:20:35.964 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.11813324 2024-09-11T08:20:35.964 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.964 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.964 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:20:35.964 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.731384+0000", 2024-09-11T08:20:35.964 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.4984000000000003e-05 2024-09-11T08:20:35.964 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:35.964 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:35.964 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:35.964 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.965 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.965 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:108646 7.2 7:4556b770:::benchmark_data_smithi150_78508_object108645:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-11T08:20:35.965 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:11.614265+0000", 2024-09-11T08:20:35.965 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 564.30301123200002, 2024-09-11T08:20:35.965 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.117338502, 2024-09-11T08:20:35.965 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:20:35.965 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:20:35.965 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:20:35.965 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:20:35.965 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:20:35.965 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 108646 2024-09-11T08:20:35.965 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.965 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:20:35.965 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.966 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:20:35.966 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.614265+0000", 2024-09-11T08:20:35.966 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:20:35.966 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.966 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.966 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:20:35.966 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.614265+0000", 2024-09-11T08:20:35.966 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:20:35.966 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.966 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.966 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:20:35.966 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.614264+0000", 2024-09-11T08:20:35.966 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:20:35.966 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.967 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.967 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:20:35.967 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.614288+0000", 2024-09-11T08:20:35.967 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.4450000000000001e-05 2024-09-11T08:20:35.967 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.967 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.967 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:20:35.967 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.614299+0000", 2024-09-11T08:20:35.967 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.041e-05 2024-09-11T08:20:35.967 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.967 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.967 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:20:35.967 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.614304+0000", 2024-09-11T08:20:35.968 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.6779999999999997e-06 2024-09-11T08:20:35.968 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.968 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.968 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:20:35.968 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.616626+0000", 2024-09-11T08:20:35.968 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00232132 2024-09-11T08:20:35.968 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.968 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.968 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:20:35.968 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.616732+0000", 2024-09-11T08:20:35.968 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000105967 2024-09-11T08:20:35.968 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.968 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.969 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 1,2", 2024-09-11T08:20:35.969 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.616870+0000", 2024-09-11T08:20:35.969 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000138367 2024-09-11T08:20:35.969 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.969 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.969 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:20:35.969 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.625171+0000", 2024-09-11T08:20:35.969 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0083011720000000008 2024-09-11T08:20:35.969 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.969 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.969 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:20:35.969 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.625209+0000", 2024-09-11T08:20:35.969 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.7443e-05 2024-09-11T08:20:35.969 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.970 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.970 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:20:35.970 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.731499+0000", 2024-09-11T08:20:35.970 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.106290017 2024-09-11T08:20:35.970 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.970 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.970 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:20:35.970 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.731563+0000", 2024-09-11T08:20:35.970 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.3905000000000006e-05 2024-09-11T08:20:35.970 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.970 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.970 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:20:35.970 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.731603+0000", 2024-09-11T08:20:35.971 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.0803000000000002e-05 2024-09-11T08:20:35.971 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:35.971 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:35.971 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:35.971 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.971 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.971 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:108650 7.a 7:51c95174:::benchmark_data_smithi150_78508_object108649:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-11T08:20:35.971 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:11.619466+0000", 2024-09-11T08:20:35.971 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 564.29781008299994, 2024-09-11T08:20:35.971 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.11220878500000001, 2024-09-11T08:20:35.971 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:20:35.971 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:20:35.971 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:20:35.971 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:20:35.972 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:20:35.972 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 108650 2024-09-11T08:20:35.972 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.972 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:20:35.972 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.972 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:20:35.972 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.619466+0000", 2024-09-11T08:20:35.972 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:20:35.972 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.972 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.972 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:20:35.972 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.619466+0000", 2024-09-11T08:20:35.972 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:20:35.972 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.973 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.973 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:20:35.973 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.619465+0000", 2024-09-11T08:20:35.973 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:20:35.973 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.973 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.973 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:20:35.973 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.619494+0000", 2024-09-11T08:20:35.973 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.9238999999999999e-05 2024-09-11T08:20:35.973 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.973 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.973 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:20:35.973 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.619504+0000", 2024-09-11T08:20:35.973 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.8709999999999992e-06 2024-09-11T08:20:35.974 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.974 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.974 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:20:35.974 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.619509+0000", 2024-09-11T08:20:35.974 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.3480000000000003e-06 2024-09-11T08:20:35.974 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.974 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.974 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:20:35.974 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.619569+0000", 2024-09-11T08:20:35.974 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.0109000000000003e-05 2024-09-11T08:20:35.974 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.974 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.974 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:20:35.974 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.619674+0000", 2024-09-11T08:20:35.975 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00010487799999999999 2024-09-11T08:20:35.975 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.975 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.975 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 2,5", 2024-09-11T08:20:35.975 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.619810+0000", 2024-09-11T08:20:35.975 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000135718 2024-09-11T08:20:35.975 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.975 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.975 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:20:35.975 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.622991+0000", 2024-09-11T08:20:35.975 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0031816779999999999 2024-09-11T08:20:35.975 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.975 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.976 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:20:35.976 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.625215+0000", 2024-09-11T08:20:35.976 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0022232390000000001 2024-09-11T08:20:35.976 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.976 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.976 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:20:35.976 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.731585+0000", 2024-09-11T08:20:35.976 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.10637010299999999 2024-09-11T08:20:35.976 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.976 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.976 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:20:35.976 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.731631+0000", 2024-09-11T08:20:35.976 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.6270000000000003e-05 2024-09-11T08:20:35.976 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.977 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.977 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:20:35.977 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.731675+0000", 2024-09-11T08:20:35.977 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.3745000000000002e-05 2024-09-11T08:20:35.977 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:35.977 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:35.977 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:35.977 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.977 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.977 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14790.0:108651 7.3 7:c42b618f:::benchmark_data_smithi150_78508_object108650:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-11T08:20:35.977 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:11.619772+0000", 2024-09-11T08:20:35.977 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 564.297504463, 2024-09-11T08:20:35.977 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.111855882, 2024-09-11T08:20:35.978 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:20:35.978 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:20:35.978 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:20:35.978 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14790", 2024-09-11T08:20:35.978 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/2554736957", 2024-09-11T08:20:35.978 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 108651 2024-09-11T08:20:35.978 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.978 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:20:35.978 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.978 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:20:35.978 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.619772+0000", 2024-09-11T08:20:35.978 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:20:35.978 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.978 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.978 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:20:35.979 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.619772+0000", 2024-09-11T08:20:35.979 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:20:35.979 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.979 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.979 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:20:35.979 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.619771+0000", 2024-09-11T08:20:35.979 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:20:35.979 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.979 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.979 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:20:35.979 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.619797+0000", 2024-09-11T08:20:35.979 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.5629000000000001e-05 2024-09-11T08:20:35.979 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.979 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.980 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:20:35.980 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.619804+0000", 2024-09-11T08:20:35.980 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.3350000000000001e-06 2024-09-11T08:20:35.980 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.980 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.980 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:20:35.980 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.619808+0000", 2024-09-11T08:20:35.980 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.3479999999999997e-06 2024-09-11T08:20:35.980 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.980 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.980 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:20:35.980 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.622759+0000", 2024-09-11T08:20:35.980 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0029511770000000001 2024-09-11T08:20:35.981 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.981 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.981 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:20:35.981 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.622875+0000", 2024-09-11T08:20:35.981 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00011602 2024-09-11T08:20:35.981 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.981 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.981 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 2,4", 2024-09-11T08:20:35.981 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.623005+0000", 2024-09-11T08:20:35.981 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012960599999999999 2024-09-11T08:20:35.981 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.981 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.981 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:20:35.982 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.629997+0000", 2024-09-11T08:20:35.982 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0069922309999999998 2024-09-11T08:20:35.982 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.982 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.982 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:20:35.982 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.731333+0000", 2024-09-11T08:20:35.982 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.10133574200000001 2024-09-11T08:20:35.982 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.982 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.982 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:20:35.982 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.731487+0000", 2024-09-11T08:20:35.982 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00015422799999999999 2024-09-11T08:20:35.982 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.982 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.983 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:20:35.983 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.731550+0000", 2024-09-11T08:20:35.983 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.2887999999999998e-05 2024-09-11T08:20:35.983 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.983 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.983 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:20:35.983 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.731627+0000", 2024-09-11T08:20:35.983 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.7296e-05 2024-09-11T08:20:35.983 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:35.983 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:35.983 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:35.983 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.983 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.983 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.14790.0:108652 7.f e122/91 7:f02b47c9:::benchmark_data_smithi150_78508_object108651:head v 122'6745, mlcod=122'6741)", 2024-09-11T08:20:35.984 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:11:11.623493+0000", 2024-09-11T08:20:35.984 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 564.293783168, 2024-09-11T08:20:35.984 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.113189462, 2024-09-11T08:20:35.984 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:20:35.984 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:20:35.984 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:20:35.984 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.984 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:20:35.984 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.623493+0000", 2024-09-11T08:20:35.984 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:20:35.984 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.984 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.984 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:20:35.985 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.623493+0000", 2024-09-11T08:20:35.985 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:20:35.985 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.985 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.985 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:20:35.985 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.623492+0000", 2024-09-11T08:20:35.985 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:20:35.985 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.985 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.985 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:20:35.985 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.623510+0000", 2024-09-11T08:20:35.985 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.7629e-05 2024-09-11T08:20:35.985 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.985 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.986 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:20:35.986 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.623516+0000", 2024-09-11T08:20:35.986 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.0929999999999999e-06 2024-09-11T08:20:35.986 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.986 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.986 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:20:35.986 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.623520+0000", 2024-09-11T08:20:35.986 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.709e-06 2024-09-11T08:20:35.986 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.986 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.986 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:20:35.986 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.630343+0000", 2024-09-11T08:20:35.986 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0068228769999999998 2024-09-11T08:20:35.986 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.986 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.987 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:20:35.987 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.630360+0000", 2024-09-11T08:20:35.987 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.7397999999999998e-05 2024-09-11T08:20:35.987 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.987 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.987 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:20:35.987 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.736620+0000", 2024-09-11T08:20:35.987 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.10625954999999999 2024-09-11T08:20:35.987 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.987 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.987 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:20:35.987 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:11:11.736682+0000", 2024-09-11T08:20:35.987 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.2648000000000001e-05 2024-09-11T08:20:35.987 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:35.987 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:35.988 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:35.988 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.988 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.988 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.24320.0:930 5.14 5:2b04a3e9:::notify.6:head [watch reconnect cookie 94783556106752 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-09-11T08:20:35.988 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:12:27.939524+0000", 2024-09-11T08:20:35.988 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 487.97775175800001, 2024-09-11T08:20:35.988 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.56717752799999999, 2024-09-11T08:20:35.988 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:20:35.988 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "started", 2024-09-11T08:20:35.988 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:20:35.988 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.24320", 2024-09-11T08:20:35.988 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1503848706", 2024-09-11T08:20:35.988 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 930 2024-09-11T08:20:35.988 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.989 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:20:35.989 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.989 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:20:35.989 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.939524+0000", 2024-09-11T08:20:35.989 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:20:35.989 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.989 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.989 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:20:35.989 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.939524+0000", 2024-09-11T08:20:35.989 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:20:35.989 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.989 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.989 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:20:35.989 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.939523+0000", 2024-09-11T08:20:35.990 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:20:35.990 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.990 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.990 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:20:35.990 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.939529+0000", 2024-09-11T08:20:35.990 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.5189999999999998e-06 2024-09-11T08:20:35.990 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.990 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.990 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:20:35.990 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.939545+0000", 2024-09-11T08:20:35.990 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.573e-05 2024-09-11T08:20:35.990 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.990 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.991 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:20:35.991 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.939553+0000", 2024-09-11T08:20:35.991 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.2290000000000003e-06 2024-09-11T08:20:35.991 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.991 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.991 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:20:35.991 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.939603+0000", 2024-09-11T08:20:35.991 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.9285e-05 2024-09-11T08:20:35.991 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.991 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.991 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:20:35.991 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.939611+0000", 2024-09-11T08:20:35.991 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.3280000000000006e-06 2024-09-11T08:20:35.991 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.992 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.992 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:20:35.992 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.502573+0000", 2024-09-11T08:20:35.992 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.56296208000000003 2024-09-11T08:20:35.992 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.992 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.992 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:20:35.992 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.502714+0000", 2024-09-11T08:20:35.992 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00014101700000000001 2024-09-11T08:20:35.992 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.992 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.992 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:20:35.992 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.506702+0000", 2024-09-11T08:20:35.992 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0039878780000000003 2024-09-11T08:20:35.993 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:35.993 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:35.993 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:35.993 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.993 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.993 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.24320.0:1236 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 94783556120960 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-11T08:20:35.993 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:15:08.641645+0000", 2024-09-11T08:20:35.993 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 327.27563130599998, 2024-09-11T08:20:35.993 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.12799823699999999, 2024-09-11T08:20:35.993 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:20:35.993 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "started", 2024-09-11T08:20:35.993 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:20:35.993 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.24320", 2024-09-11T08:20:35.993 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1503848706", 2024-09-11T08:20:35.994 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 1236 2024-09-11T08:20:35.994 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.994 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:20:35.994 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.994 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:20:35.994 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:08.641645+0000", 2024-09-11T08:20:35.994 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:20:35.994 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.994 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.994 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:20:35.994 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:08.641645+0000", 2024-09-11T08:20:35.994 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:20:35.994 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.994 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.995 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:20:35.995 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:08.641644+0000", 2024-09-11T08:20:35.995 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:20:35.995 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.995 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.995 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:20:35.995 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:08.641649+0000", 2024-09-11T08:20:35.995 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.6550000000000001e-06 2024-09-11T08:20:35.995 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.995 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.995 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:20:35.995 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:08.641660+0000", 2024-09-11T08:20:35.995 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1073999999999999e-05 2024-09-11T08:20:35.995 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.995 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.996 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:20:35.996 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:08.641670+0000", 2024-09-11T08:20:35.996 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.3449999999999995e-06 2024-09-11T08:20:35.996 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.996 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.996 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:20:35.996 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:08.641711+0000", 2024-09-11T08:20:35.996 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.0988000000000001e-05 2024-09-11T08:20:35.996 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.996 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.996 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:20:35.996 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:08.641718+0000", 2024-09-11T08:20:35.996 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.2019999999999998e-06 2024-09-11T08:20:35.996 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.997 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.997 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:20:35.997 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:08.765266+0000", 2024-09-11T08:20:35.997 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.123548318 2024-09-11T08:20:35.997 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.997 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.997 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for missing object", 2024-09-11T08:20:35.997 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:08.765380+0000", 2024-09-11T08:20:35.997 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000113319 2024-09-11T08:20:35.997 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.997 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.997 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:20:35.997 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:08.766828+0000", 2024-09-11T08:20:35.997 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0014484369999999999 2024-09-11T08:20:35.998 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.998 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.998 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for rw locks", 2024-09-11T08:20:35.998 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:08.766916+0000", 2024-09-11T08:20:35.998 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.7818000000000005e-05 2024-09-11T08:20:35.998 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.998 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.998 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:20:35.998 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:08.767676+0000", 2024-09-11T08:20:35.998 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00076046600000000001 2024-09-11T08:20:35.998 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.998 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.998 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:20:35.998 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:08.767737+0000", 2024-09-11T08:20:35.998 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.0696999999999998e-05 2024-09-11T08:20:35.999 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.999 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.999 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:20:35.999 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:08.769643+0000", 2024-09-11T08:20:35.999 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0019059260000000001 2024-09-11T08:20:35.999 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:35.999 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:35.999 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:35.999 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:35.999 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:35.999 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15087.0:254113 8.b 8:d32650ea:::benchmark_data_smithi150_107646_object125281:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-11T08:20:35.999 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:17:32.516600+0000", 2024-09-11T08:20:35.999 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 183.40067608699999, 2024-09-11T08:20:36.000 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.13402002099999999, 2024-09-11T08:20:36.000 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:20:36.000 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:20:36.000 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:20:36.000 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15087", 2024-09-11T08:20:36.000 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/3299560913", 2024-09-11T08:20:36.000 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 254113 2024-09-11T08:20:36.000 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.000 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:20:36.000 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.000 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:20:36.000 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:32.516600+0000", 2024-09-11T08:20:36.000 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:20:36.001 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.001 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.001 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:20:36.001 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:32.516600+0000", 2024-09-11T08:20:36.001 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:20:36.001 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.001 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.001 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:20:36.001 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:32.516599+0000", 2024-09-11T08:20:36.001 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:20:36.001 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.001 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.001 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:20:36.001 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:32.516604+0000", 2024-09-11T08:20:36.002 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.057e-06 2024-09-11T08:20:36.002 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.002 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.002 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:20:36.002 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:32.516614+0000", 2024-09-11T08:20:36.002 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.7049999999999994e-06 2024-09-11T08:20:36.002 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.002 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.002 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:20:36.002 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:32.516619+0000", 2024-09-11T08:20:36.002 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.8640000000000004e-06 2024-09-11T08:20:36.002 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.002 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.002 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:20:36.002 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:32.516656+0000", 2024-09-11T08:20:36.003 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.6983000000000001e-05 2024-09-11T08:20:36.003 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.003 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.003 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:20:36.003 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:32.516759+0000", 2024-09-11T08:20:36.003 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00010391399999999999 2024-09-11T08:20:36.003 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.003 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.003 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 1,4", 2024-09-11T08:20:36.003 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:32.516869+0000", 2024-09-11T08:20:36.003 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00010964900000000001 2024-09-11T08:20:36.003 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.003 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.003 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:20:36.004 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:32.518985+0000", 2024-09-11T08:20:36.004 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0021154310000000001 2024-09-11T08:20:36.004 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.004 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.004 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:20:36.004 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:32.520229+0000", 2024-09-11T08:20:36.004 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0012447039999999999 2024-09-11T08:20:36.004 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.004 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.004 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:20:36.004 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:32.650412+0000", 2024-09-11T08:20:36.004 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.13018290800000001 2024-09-11T08:20:36.004 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.004 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.005 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:20:36.005 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:36.006 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:32.650579+0000", 2024-09-11T08:20:36.006 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000166782 2024-09-11T08:20:36.006 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.006 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.006 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:20:36.006 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:32.650620+0000", 2024-09-11T08:20:36.007 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1097000000000003e-05 2024-09-11T08:20:36.007 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:36.007 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:36.007 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:36.007 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.007 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.007 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15087.0:254125 8.0 8:025b1613:::benchmark_data_smithi150_107646_object125293:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-11T08:20:36.007 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:17:32.531680+0000", 2024-09-11T08:20:36.007 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 183.385596161, 2024-09-11T08:20:36.007 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.12198445400000001, 2024-09-11T08:20:36.007 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:20:36.007 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:20:36.008 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:20:36.008 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15087", 2024-09-11T08:20:36.008 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/3299560913", 2024-09-11T08:20:36.008 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 254125 2024-09-11T08:20:36.008 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.008 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:20:36.008 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.008 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:20:36.008 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:32.531680+0000", 2024-09-11T08:20:36.008 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:20:36.008 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.008 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.008 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:20:36.009 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:32.531680+0000", 2024-09-11T08:20:36.009 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:20:36.009 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.009 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.009 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:20:36.009 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:32.531679+0000", 2024-09-11T08:20:36.009 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:20:36.009 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.009 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.009 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:20:36.009 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:32.531684+0000", 2024-09-11T08:20:36.009 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.3199999999999999e-06 2024-09-11T08:20:36.009 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.009 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.010 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:20:36.010 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:32.531693+0000", 2024-09-11T08:20:36.010 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.9490000000000002e-06 2024-09-11T08:20:36.010 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.010 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.010 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:20:36.010 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:32.531697+0000", 2024-09-11T08:20:36.010 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1309999999999999e-06 2024-09-11T08:20:36.010 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.010 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.010 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:20:36.010 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:32.531730+0000", 2024-09-11T08:20:36.010 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.2422000000000002e-05 2024-09-11T08:20:36.010 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.011 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.011 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:20:36.011 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:32.531807+0000", 2024-09-11T08:20:36.011 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.7921000000000002e-05 2024-09-11T08:20:36.011 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.011 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.011 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 1,5", 2024-09-11T08:20:36.011 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:32.531872+0000", 2024-09-11T08:20:36.011 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.4197999999999998e-05 2024-09-11T08:20:36.011 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.011 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.011 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:20:36.011 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:32.533748+0000", 2024-09-11T08:20:36.011 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.001876663 2024-09-11T08:20:36.012 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.012 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.012 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:20:36.012 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:32.534305+0000", 2024-09-11T08:20:36.012 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000556822 2024-09-11T08:20:36.012 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.012 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.012 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:20:36.012 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:32.653581+0000", 2024-09-11T08:20:36.012 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.119275855 2024-09-11T08:20:36.012 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.012 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.012 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:20:36.012 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:32.653628+0000", 2024-09-11T08:20:36.013 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.6519e-05 2024-09-11T08:20:36.013 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.013 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.013 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:20:36.013 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:32.653664+0000", 2024-09-11T08:20:36.013 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.6853000000000001e-05 2024-09-11T08:20:36.013 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:36.013 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:36.013 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:36.013 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.013 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.013 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15369.0:12201 9.2 9:4c4254ed:::benchmark_data_smithi150_133616_object12200:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e251)", 2024-09-11T08:20:36.013 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:18:15.369136+0000", 2024-09-11T08:20:36.013 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 140.54813991099999, 2024-09-11T08:20:36.014 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.13206040399999999, 2024-09-11T08:20:36.014 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:20:36.014 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:20:36.014 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:20:36.014 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15369", 2024-09-11T08:20:36.014 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:20:36.014 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 12201 2024-09-11T08:20:36.014 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.014 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:20:36.014 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.014 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:20:36.014 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:15.369136+0000", 2024-09-11T08:20:36.014 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:20:36.015 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.015 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.015 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:20:36.015 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:15.369136+0000", 2024-09-11T08:20:36.015 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:20:36.015 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.015 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.015 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:20:36.015 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:15.369134+0000", 2024-09-11T08:20:36.015 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:20:36.015 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.015 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.015 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:20:36.015 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:15.369158+0000", 2024-09-11T08:20:36.015 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.3726999999999998e-05 2024-09-11T08:20:36.016 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.016 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.016 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:20:36.016 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:15.369168+0000", 2024-09-11T08:20:36.016 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0557000000000001e-05 2024-09-11T08:20:36.016 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.016 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.016 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:20:36.016 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:15.369174+0000", 2024-09-11T08:20:36.016 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.9540000000000001e-06 2024-09-11T08:20:36.016 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.016 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.016 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:20:36.017 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:15.369273+0000", 2024-09-11T08:20:36.017 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.8696999999999994e-05 2024-09-11T08:20:36.017 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.017 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.017 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:20:36.017 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:15.369378+0000", 2024-09-11T08:20:36.017 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00010511899999999999 2024-09-11T08:20:36.017 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.017 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.017 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 1,2", 2024-09-11T08:20:36.017 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:15.369498+0000", 2024-09-11T08:20:36.017 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000120227 2024-09-11T08:20:36.017 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.017 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.018 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:20:36.018 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:15.370977+0000", 2024-09-11T08:20:36.018 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0014787139999999999 2024-09-11T08:20:36.018 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.018 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.018 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:20:36.018 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:15.371012+0000", 2024-09-11T08:20:36.018 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.5176999999999999e-05 2024-09-11T08:20:36.018 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.018 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.018 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:20:36.018 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:15.501045+0000", 2024-09-11T08:20:36.018 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.130032386 2024-09-11T08:20:36.018 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.018 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.019 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:20:36.019 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:15.501152+0000", 2024-09-11T08:20:36.019 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00010757899999999999 2024-09-11T08:20:36.019 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.019 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.019 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:20:36.019 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:15.501197+0000", 2024-09-11T08:20:36.019 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.4372999999999997e-05 2024-09-11T08:20:36.019 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:36.019 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:36.019 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:36.019 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.019 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.020 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15369.0:12209 9.2 9:4f986861:::benchmark_data_smithi150_133616_object12208:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e251)", 2024-09-11T08:20:36.020 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:18:15.371687+0000", 2024-09-11T08:20:36.020 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 140.545588885, 2024-09-11T08:20:36.020 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.131822787, 2024-09-11T08:20:36.020 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:20:36.020 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:20:36.020 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:20:36.020 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15369", 2024-09-11T08:20:36.020 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:20:36.020 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 12209 2024-09-11T08:20:36.020 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.020 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:20:36.020 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.020 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:20:36.021 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:15.371687+0000", 2024-09-11T08:20:36.021 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:20:36.021 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.021 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.021 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:20:36.021 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:15.371687+0000", 2024-09-11T08:20:36.021 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:20:36.021 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.021 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.021 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:20:36.021 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:15.371686+0000", 2024-09-11T08:20:36.021 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:20:36.021 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.022 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.022 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:20:36.022 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:15.371707+0000", 2024-09-11T08:20:36.022 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.122e-05 2024-09-11T08:20:36.022 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.022 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.022 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:20:36.022 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:15.371717+0000", 2024-09-11T08:20:36.022 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0039e-05 2024-09-11T08:20:36.022 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.022 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.022 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:20:36.022 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:15.371721+0000", 2024-09-11T08:20:36.022 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.442e-06 2024-09-11T08:20:36.023 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.023 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.023 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:20:36.023 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:15.371757+0000", 2024-09-11T08:20:36.023 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.5917999999999997e-05 2024-09-11T08:20:36.023 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.023 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.023 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:20:36.023 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:15.371851+0000", 2024-09-11T08:20:36.023 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.3630999999999999e-05 2024-09-11T08:20:36.023 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.023 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.023 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 1,2", 2024-09-11T08:20:36.023 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:15.371956+0000", 2024-09-11T08:20:36.023 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000104874 2024-09-11T08:20:36.024 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.024 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.024 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:20:36.024 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:15.372822+0000", 2024-09-11T08:20:36.024 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00086610699999999997 2024-09-11T08:20:36.024 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.024 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.024 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:20:36.024 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:15.373111+0000", 2024-09-11T08:20:36.024 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00028872799999999998 2024-09-11T08:20:36.024 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.024 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.024 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:20:36.025 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:15.503365+0000", 2024-09-11T08:20:36.025 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.130254605 2024-09-11T08:20:36.025 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.025 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.025 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:20:36.025 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:15.503454+0000", 2024-09-11T08:20:36.025 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.8919999999999996e-05 2024-09-11T08:20:36.025 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.025 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.025 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:20:36.025 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:15.503510+0000", 2024-09-11T08:20:36.025 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.5902000000000001e-05 2024-09-11T08:20:36.025 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:36.026 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:36.026 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:36.026 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.026 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.026 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15369.0:28035 9.2 9:48f7c719:::benchmark_data_smithi150_133616_object28034:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e252)", 2024-09-11T08:20:36.026 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:18:22.532044+0000", 2024-09-11T08:20:36.026 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 133.38523175899999, 2024-09-11T08:20:36.026 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.138700556, 2024-09-11T08:20:36.026 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:20:36.026 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:20:36.026 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:20:36.026 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15369", 2024-09-11T08:20:36.026 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:20:36.026 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 28035 2024-09-11T08:20:36.027 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.027 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:20:36.027 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.027 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:20:36.027 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:22.532044+0000", 2024-09-11T08:20:36.027 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:20:36.027 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.027 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.027 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:20:36.027 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:22.532044+0000", 2024-09-11T08:20:36.027 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:20:36.027 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.027 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.027 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:20:36.028 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:22.532043+0000", 2024-09-11T08:20:36.028 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:20:36.028 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.028 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.028 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:20:36.028 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:22.532066+0000", 2024-09-11T08:20:36.028 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.3572999999999999e-05 2024-09-11T08:20:36.028 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.028 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.028 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:20:36.028 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:22.532079+0000", 2024-09-11T08:20:36.028 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2116000000000001e-05 2024-09-11T08:20:36.028 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.028 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.029 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:20:36.029 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:22.532085+0000", 2024-09-11T08:20:36.029 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.144e-06 2024-09-11T08:20:36.029 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.029 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.029 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:20:36.029 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:22.532126+0000", 2024-09-11T08:20:36.029 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1316000000000002e-05 2024-09-11T08:20:36.029 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.029 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.029 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:20:36.029 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:22.532220+0000", 2024-09-11T08:20:36.029 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.4155999999999998e-05 2024-09-11T08:20:36.029 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.030 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.030 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 1,2", 2024-09-11T08:20:36.030 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:22.532394+0000", 2024-09-11T08:20:36.030 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00017396800000000001 2024-09-11T08:20:36.030 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.030 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.030 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:20:36.030 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:22.533473+0000", 2024-09-11T08:20:36.030 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0010786470000000001 2024-09-11T08:20:36.030 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.030 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.030 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:20:36.030 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:22.533652+0000", 2024-09-11T08:20:36.030 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000179242 2024-09-11T08:20:36.031 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.031 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.031 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:20:36.031 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:22.670604+0000", 2024-09-11T08:20:36.031 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.13695171 2024-09-11T08:20:36.031 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.031 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.031 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:20:36.031 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:22.670670+0000", 2024-09-11T08:20:36.031 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.6481000000000003e-05 2024-09-11T08:20:36.031 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.031 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.031 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:20:36.031 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:22.670745+0000", 2024-09-11T08:20:36.032 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.4596000000000002e-05 2024-09-11T08:20:36.032 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:36.032 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:36.032 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:36.032 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.032 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.032 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15369.0:30534 9.2 9:4209b46e:::benchmark_data_smithi150_133616_object30533:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e252)", 2024-09-11T08:20:36.032 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:18:23.989103+0000", 2024-09-11T08:20:36.032 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 131.92817332499999, 2024-09-11T08:20:36.032 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.162855108, 2024-09-11T08:20:36.032 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:20:36.032 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:20:36.032 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:20:36.033 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15369", 2024-09-11T08:20:36.033 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:20:36.033 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 30534 2024-09-11T08:20:36.033 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.033 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:20:36.033 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.033 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:20:36.033 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:23.989103+0000", 2024-09-11T08:20:36.033 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:20:36.033 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.033 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.033 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:20:36.033 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:23.989103+0000", 2024-09-11T08:20:36.033 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:20:36.034 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.034 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.034 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:20:36.034 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:23.989101+0000", 2024-09-11T08:20:36.034 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:20:36.034 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.034 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.034 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:20:36.034 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:23.989126+0000", 2024-09-11T08:20:36.034 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.4474e-05 2024-09-11T08:20:36.034 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.034 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.034 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:20:36.034 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:23.989137+0000", 2024-09-11T08:20:36.034 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1109e-05 2024-09-11T08:20:36.035 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.035 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.035 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:20:36.035 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:23.989143+0000", 2024-09-11T08:20:36.035 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.9440000000000002e-06 2024-09-11T08:20:36.035 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.035 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.035 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:20:36.035 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:23.989193+0000", 2024-09-11T08:20:36.035 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.0766000000000001e-05 2024-09-11T08:20:36.035 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.035 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.035 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:20:36.035 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:23.989402+0000", 2024-09-11T08:20:36.036 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000208839 2024-09-11T08:20:36.036 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.036 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.036 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 1,2", 2024-09-11T08:20:36.036 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:23.989548+0000", 2024-09-11T08:20:36.036 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00014581199999999999 2024-09-11T08:20:36.036 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.036 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.036 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:20:36.036 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:23.991428+0000", 2024-09-11T08:20:36.036 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.001880147 2024-09-11T08:20:36.036 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.036 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.037 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:20:36.037 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:23.991525+0000", 2024-09-11T08:20:36.037 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.6718000000000005e-05 2024-09-11T08:20:36.037 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.037 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.037 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:20:36.037 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:24.151867+0000", 2024-09-11T08:20:36.037 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.160342019 2024-09-11T08:20:36.037 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.037 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.037 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:20:36.037 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:24.151917+0000", 2024-09-11T08:20:36.037 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.9935999999999998e-05 2024-09-11T08:20:36.037 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:36.038 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:36.038 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:20:36.038 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:24.151958+0000", 2024-09-11T08:20:36.038 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.0936000000000002e-05 2024-09-11T08:20:36.038 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:36.038 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:36.038 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:36.038 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:36.038 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:36.038 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-11T08:20:36.038 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T08:20:36.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:36.040 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:36.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:36.141 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:20:36.186 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:36.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:36.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:20:36.288 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:36.322 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:36.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:36.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:36.423 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:36.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:36.454 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:36.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:36.544 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:36 smithi150 ceph-mon[29276]: pgmap v594: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-09-11T08:20:36.544 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:36 smithi150 ceph-mon[39612]: pgmap v594: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-09-11T08:20:36.555 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:36.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:36 smithi181 ceph-mon[32701]: pgmap v594: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-09-11T08:20:36.584 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:36.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:36.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:36.686 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:36.717 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:36.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:36.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:36.819 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:20:36.860 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:36.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:36.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:20:36.961 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:36.996 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:36.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:36.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:37.098 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:37.131 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:37.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:37.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:37.233 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:37.265 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:37.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:37.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:37.367 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:20:37.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:37.401 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:37.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:20:37.502 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:37.534 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:37.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:37.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:37.635 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:37.666 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:37.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:37.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:37.768 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:37.798 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:37.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:37.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:37.891 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:37 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:20:37.891 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:37 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:20:37.892 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:37 smithi150 ceph-mon[29276]: pgmap v595: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 578 op/s 2024-09-11T08:20:37.892 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:37 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:20:37.892 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:37 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:20:37.892 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:37 smithi150 ceph-mon[39612]: pgmap v595: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 578 op/s 2024-09-11T08:20:37.899 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:20:37.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:37.930 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:37.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:20:38.031 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:20:38.067 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:38.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:38.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:20:38.068 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-11T08:20:38.068 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T08:20:38.069 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-11T08:20:38.069 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-11T08:20:38.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:37 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:20:38.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:37 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:20:38.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:37 smithi181 ceph-mon[32701]: pgmap v595: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 578 op/s 2024-09-11T08:20:38.170 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:20:38.211 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-11T08:20:38.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:38.227 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:38.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:20:38.328 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:38.367 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:38.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:38.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:38.436 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:20:38.468 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:38.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:38.510 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:38.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:38.575 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:20:38 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:20:38.611 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:38.616 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:20:38.671 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:38.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:38.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:38.773 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:38.816 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:38.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:38.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:38.913 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:20:38 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:20:38] "GET /metrics HTTP/1.1" 200 224570 "" "Prometheus/2.43.0" 2024-09-11T08:20:38.914 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:38 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:20:38.914 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:38 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:20:38.914 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:38 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:20:38.914 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:38 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:20:38.920 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:20:38.959 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:38.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:38.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:20:39.038 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:38 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:20:39.038 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:38 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:20:39.061 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:39.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:39.105 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:39.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:39.206 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:20:39.234 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:20:39.241 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:39.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:39.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:20:39.343 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:39.378 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:39.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:39.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:39.480 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:20:39.514 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:39.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:39.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:20:39.616 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:20:39.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:39.656 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:39.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:20:39.757 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:39.791 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:39.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:39.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:39.893 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:39.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:39.930 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:39.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:39.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:39 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:20:39.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:39 smithi150 ceph-mon[29276]: osdmap e291: 8 total, 8 up, 8 in 2024-09-11T08:20:39.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:39 smithi150 ceph-mon[29276]: pgmap v597: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-09-11T08:20:39.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:39 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:20:39.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:39 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:20:39.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:39 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:20:39.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:39 smithi150 ceph-mon[39612]: osdmap e291: 8 total, 8 up, 8 in 2024-09-11T08:20:39.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:39 smithi150 ceph-mon[39612]: pgmap v597: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-09-11T08:20:39.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:39 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:20:39.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:39 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:20:40.030 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:40.060 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:40.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:40.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:40.069 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:39 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:20:40.069 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:39 smithi181 ceph-mon[32701]: osdmap e291: 8 total, 8 up, 8 in 2024-09-11T08:20:40.069 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:39 smithi181 ceph-mon[32701]: pgmap v597: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-09-11T08:20:40.069 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:39 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:20:40.069 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:39 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:20:40.162 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:40.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:40.189 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:40.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:40.290 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:40.339 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:40.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:40.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:40.440 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:40.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:40.479 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:40.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:40.540 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-11T08:20:40.540 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-11T08:20:40.540 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-09-11T08:20:40.540 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-11T08:20:40.579 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:20:40.617 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:40.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:40.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:20:40.719 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:40.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:40.755 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:40.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:40.798 INFO:teuthology.orchestra.run.smithi150.stderr:nodeep-scrub is set 2024-09-11T08:20:40.856 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:40.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:40.890 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:40.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:40.921 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:40 smithi150 ceph-mon[29276]: osdmap e292: 8 total, 8 up, 8 in 2024-09-11T08:20:40.921 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:40 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1428445130' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:20:40.921 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:40 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:20:40.921 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:40 smithi150 ceph-mon[39612]: osdmap e292: 8 total, 8 up, 8 in 2024-09-11T08:20:40.921 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:40 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1428445130' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:20:40.921 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:40 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:20:40.991 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:41.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:41.023 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:41.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:41.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:40 smithi181 ceph-mon[32701]: osdmap e292: 8 total, 8 up, 8 in 2024-09-11T08:20:41.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:40 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1428445130' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:20:41.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:40 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:20:41.124 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:20:41.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:41.157 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:41.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:20:41.254 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-11T08:20:41.258 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:41.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:41.331 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:41.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:41.432 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:20:41.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:41.470 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:41.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:20:41.571 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:20:41.606 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:41.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:41.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:20:41.673 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:20:41.708 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:41.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:41.744 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:41.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:41.845 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:20:41.879 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:41.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:41.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:20:41.916 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:41 smithi150 ceph-mon[29276]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:20:41.916 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:41 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:20:41.916 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:41 smithi150 ceph-mon[29276]: osdmap e293: 8 total, 8 up, 8 in 2024-09-11T08:20:41.916 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:41 smithi150 ceph-mon[29276]: pgmap v600: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 760 op/s 2024-09-11T08:20:41.916 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:41 smithi150 ceph-mon[39612]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:20:41.916 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:41 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:20:41.916 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:41 smithi150 ceph-mon[39612]: osdmap e293: 8 total, 8 up, 8 in 2024-09-11T08:20:41.916 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:41 smithi150 ceph-mon[39612]: pgmap v600: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 760 op/s 2024-09-11T08:20:41.982 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:20:42.014 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:42.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:42.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:20:42.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:41 smithi181 ceph-mon[32701]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:20:42.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:41 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:20:42.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:41 smithi181 ceph-mon[32701]: osdmap e293: 8 total, 8 up, 8 in 2024-09-11T08:20:42.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:41 smithi181 ceph-mon[32701]: pgmap v600: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 760 op/s 2024-09-11T08:20:42.116 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:42.161 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:42.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:42.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:42.263 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:20:42.297 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:42.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:42.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:20:42.398 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:42.435 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:42.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:42.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:42.537 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:42.571 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:42.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:42.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:42.673 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:42.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:42.707 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:42.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:42.807 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:42.852 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:42.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:42.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:42.954 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:42.991 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:42.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:42.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:43.093 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:20:43.105 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:42 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/797036743' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:20:43.126 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:43.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:43.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:20:43.179 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:42 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/797036743' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:20:43.181 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:42 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/797036743' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:20:43.228 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:43.253 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:43.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:43.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:43.297 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:20:43.297 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:20:43.297 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:20:43.297 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:20:43.297 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:20:43.297 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:20:43.297 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:20:43.297 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:20:43.298 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:20:43.298 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:20:43.298 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:20:43.298 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:20:43.298 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:20:43.298 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:20:43.298 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:20:43.298 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:20:43.355 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:43.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:43.393 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:43.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:43.494 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:20:43.538 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:43.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:43.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:20:43.639 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:43.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:43.684 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:43.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:43.785 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:20:43.818 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:43.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:43.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:20:43.920 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:43.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:43.955 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:43.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:44.049 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-11T08:20:44.049 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:20:44.057 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:44.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:44.091 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:44.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:44.193 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:20:44.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:44.231 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:44.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:20:44.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:43 smithi150 ceph-mon[29276]: pgmap v601: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 870 op/s 2024-09-11T08:20:44.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:43 smithi150 ceph-mon[39612]: pgmap v601: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 870 op/s 2024-09-11T08:20:44.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:43 smithi181 ceph-mon[32701]: pgmap v601: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 870 op/s 2024-09-11T08:20:44.332 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:44.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:44.363 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:44.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:44.375 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:20:44.376 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-11T08:20:44.376 INFO:teuthology.orchestra.run.smithi150.stdout: "epoch": 293, 2024-09-11T08:20:44.376 INFO:teuthology.orchestra.run.smithi150.stdout: "fsid": "c1bddff4-7013-11ef-bcea-c7b262605968", 2024-09-11T08:20:44.376 INFO:teuthology.orchestra.run.smithi150.stdout: "created": "2024-09-11T08:02:15.344618+0000", 2024-09-11T08:20:44.376 INFO:teuthology.orchestra.run.smithi150.stdout: "modified": "2024-09-11T08:20:40.780751+0000", 2024-09-11T08:20:44.376 INFO:teuthology.orchestra.run.smithi150.stdout: "last_up_change": "2024-09-11T08:14:23.745449+0000", 2024-09-11T08:20:44.376 INFO:teuthology.orchestra.run.smithi150.stdout: "last_in_change": "2024-09-11T08:19:28.121267+0000", 2024-09-11T08:20:44.377 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-11T08:20:44.377 INFO:teuthology.orchestra.run.smithi150.stdout: "flags_num": 5806080, 2024-09-11T08:20:44.377 INFO:teuthology.orchestra.run.smithi150.stdout: "flags_set": [ 2024-09-11T08:20:44.377 INFO:teuthology.orchestra.run.smithi150.stdout: "nodeep-scrub", 2024-09-11T08:20:44.377 INFO:teuthology.orchestra.run.smithi150.stdout: "noscrub", 2024-09-11T08:20:44.377 INFO:teuthology.orchestra.run.smithi150.stdout: "pglog_hardlimit", 2024-09-11T08:20:44.377 INFO:teuthology.orchestra.run.smithi150.stdout: "purged_snapdirs", 2024-09-11T08:20:44.378 INFO:teuthology.orchestra.run.smithi150.stdout: "recovery_deletes", 2024-09-11T08:20:44.378 INFO:teuthology.orchestra.run.smithi150.stdout: "sortbitwise" 2024-09-11T08:20:44.378 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-09-11T08:20:44.378 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_version": 19, 2024-09-11T08:20:44.378 INFO:teuthology.orchestra.run.smithi150.stdout: "full_ratio": 0.94999998807907104, 2024-09-11T08:20:44.378 INFO:teuthology.orchestra.run.smithi150.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-11T08:20:44.379 INFO:teuthology.orchestra.run.smithi150.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-11T08:20:44.379 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_snapshot": "", 2024-09-11T08:20:44.379 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_max": 9, 2024-09-11T08:20:44.379 INFO:teuthology.orchestra.run.smithi150.stdout: "max_osd": 8, 2024-09-11T08:20:44.379 INFO:teuthology.orchestra.run.smithi150.stdout: "require_min_compat_client": "luminous", 2024-09-11T08:20:44.379 INFO:teuthology.orchestra.run.smithi150.stdout: "min_compat_client": "luminous", 2024-09-11T08:20:44.379 INFO:teuthology.orchestra.run.smithi150.stdout: "require_osd_release": "quincy", 2024-09-11T08:20:44.380 INFO:teuthology.orchestra.run.smithi150.stdout: "pools": [ 2024-09-11T08:20:44.380 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.380 INFO:teuthology.orchestra.run.smithi150.stdout: "pool": 1, 2024-09-11T08:20:44.380 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_name": ".mgr", 2024-09-11T08:20:44.380 INFO:teuthology.orchestra.run.smithi150.stdout: "create_time": "2024-09-11T08:05:21.044729+0000", 2024-09-11T08:20:44.380 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-09-11T08:20:44.380 INFO:teuthology.orchestra.run.smithi150.stdout: "flags_names": "hashpspool", 2024-09-11T08:20:44.380 INFO:teuthology.orchestra.run.smithi150.stdout: "type": 1, 2024-09-11T08:20:44.380 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 3, 2024-09-11T08:20:44.380 INFO:teuthology.orchestra.run.smithi150.stdout: "min_size": 2, 2024-09-11T08:20:44.380 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_rule": 0, 2024-09-11T08:20:44.380 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_count": 0, 2024-09-11T08:20:44.380 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_target": 0, 2024-09-11T08:20:44.381 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T08:20:44.381 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T08:20:44.381 INFO:teuthology.orchestra.run.smithi150.stdout: "object_hash": 2, 2024-09-11T08:20:44.381 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_autoscale_mode": "off", 2024-09-11T08:20:44.381 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num": 1, 2024-09-11T08:20:44.381 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num": 1, 2024-09-11T08:20:44.381 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num_target": 1, 2024-09-11T08:20:44.381 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_target": 1, 2024-09-11T08:20:44.381 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_pending": 1, 2024-09-11T08:20:44.381 INFO:teuthology.orchestra.run.smithi150.stdout: "last_pg_merge_meta": { 2024-09-11T08:20:44.381 INFO:teuthology.orchestra.run.smithi150.stdout: "source_pgid": "0.0", 2024-09-11T08:20:44.381 INFO:teuthology.orchestra.run.smithi150.stdout: "ready_epoch": 0, 2024-09-11T08:20:44.381 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_started": 0, 2024-09-11T08:20:44.382 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_clean": 0, 2024-09-11T08:20:44.382 INFO:teuthology.orchestra.run.smithi150.stdout: "source_version": "0'0", 2024-09-11T08:20:44.382 INFO:teuthology.orchestra.run.smithi150.stdout: "target_version": "0'0" 2024-09-11T08:20:44.382 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.382 INFO:teuthology.orchestra.run.smithi150.stdout: "last_change": "255", 2024-09-11T08:20:44.382 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend": "0", 2024-09-11T08:20:44.382 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T08:20:44.382 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T08:20:44.382 INFO:teuthology.orchestra.run.smithi150.stdout: "auid": 0, 2024-09-11T08:20:44.382 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_mode": "selfmanaged", 2024-09-11T08:20:44.382 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_seq": 0, 2024-09-11T08:20:44.382 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_epoch": 0, 2024-09-11T08:20:44.382 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_snaps": [], 2024-09-11T08:20:44.383 INFO:teuthology.orchestra.run.smithi150.stdout: "removed_snaps": "[]", 2024-09-11T08:20:44.383 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_bytes": 0, 2024-09-11T08:20:44.383 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_objects": 0, 2024-09-11T08:20:44.383 INFO:teuthology.orchestra.run.smithi150.stdout: "tiers": [], 2024-09-11T08:20:44.383 INFO:teuthology.orchestra.run.smithi150.stdout: "tier_of": -1, 2024-09-11T08:20:44.383 INFO:teuthology.orchestra.run.smithi150.stdout: "read_tier": -1, 2024-09-11T08:20:44.383 INFO:teuthology.orchestra.run.smithi150.stdout: "write_tier": -1, 2024-09-11T08:20:44.383 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_mode": "none", 2024-09-11T08:20:44.383 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_bytes": 0, 2024-09-11T08:20:44.383 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_objects": 0, 2024-09-11T08:20:44.383 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T08:20:44.383 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T08:20:44.383 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T08:20:44.383 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_flush_age": 0, 2024-09-11T08:20:44.384 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_evict_age": 0, 2024-09-11T08:20:44.384 INFO:teuthology.orchestra.run.smithi150.stdout: "erasure_code_profile": "", 2024-09-11T08:20:44.384 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_params": { 2024-09-11T08:20:44.384 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "none" 2024-09-11T08:20:44.384 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.384 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_period": 0, 2024-09-11T08:20:44.384 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_count": 0, 2024-09-11T08:20:44.384 INFO:teuthology.orchestra.run.smithi150.stdout: "use_gmt_hitset": true, 2024-09-11T08:20:44.384 INFO:teuthology.orchestra.run.smithi150.stdout: "min_read_recency_for_promote": 0, 2024-09-11T08:20:44.384 INFO:teuthology.orchestra.run.smithi150.stdout: "min_write_recency_for_promote": 0, 2024-09-11T08:20:44.384 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T08:20:44.384 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_search_last_n": 0, 2024-09-11T08:20:44.384 INFO:teuthology.orchestra.run.smithi150.stdout: "grade_table": [], 2024-09-11T08:20:44.384 INFO:teuthology.orchestra.run.smithi150.stdout: "stripe_width": 0, 2024-09-11T08:20:44.385 INFO:teuthology.orchestra.run.smithi150.stdout: "expected_num_objects": 0, 2024-09-11T08:20:44.385 INFO:teuthology.orchestra.run.smithi150.stdout: "fast_read": false, 2024-09-11T08:20:44.385 INFO:teuthology.orchestra.run.smithi150.stdout: "options": { 2024-09-11T08:20:44.385 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_max": 32, 2024-09-11T08:20:44.385 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_min": 1 2024-09-11T08:20:44.385 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.385 INFO:teuthology.orchestra.run.smithi150.stdout: "application_metadata": { 2024-09-11T08:20:44.385 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": {} 2024-09-11T08:20:44.385 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:44.385 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.385 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.385 INFO:teuthology.orchestra.run.smithi150.stdout: "pool": 2, 2024-09-11T08:20:44.386 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_name": "datapool", 2024-09-11T08:20:44.386 INFO:teuthology.orchestra.run.smithi150.stdout: "create_time": "2024-09-11T08:07:28.786572+0000", 2024-09-11T08:20:44.386 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 8193, 2024-09-11T08:20:44.386 INFO:teuthology.orchestra.run.smithi150.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-11T08:20:44.386 INFO:teuthology.orchestra.run.smithi150.stdout: "type": 1, 2024-09-11T08:20:44.386 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 3, 2024-09-11T08:20:44.386 INFO:teuthology.orchestra.run.smithi150.stdout: "min_size": 2, 2024-09-11T08:20:44.386 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_rule": 0, 2024-09-11T08:20:44.386 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_count": 0, 2024-09-11T08:20:44.386 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_target": 0, 2024-09-11T08:20:44.386 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T08:20:44.386 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T08:20:44.386 INFO:teuthology.orchestra.run.smithi150.stdout: "object_hash": 2, 2024-09-11T08:20:44.387 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_autoscale_mode": "off", 2024-09-11T08:20:44.387 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num": 3, 2024-09-11T08:20:44.387 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num": 3, 2024-09-11T08:20:44.387 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num_target": 3, 2024-09-11T08:20:44.387 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_target": 3, 2024-09-11T08:20:44.387 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_pending": 3, 2024-09-11T08:20:44.387 INFO:teuthology.orchestra.run.smithi150.stdout: "last_pg_merge_meta": { 2024-09-11T08:20:44.387 INFO:teuthology.orchestra.run.smithi150.stdout: "source_pgid": "0.0", 2024-09-11T08:20:44.387 INFO:teuthology.orchestra.run.smithi150.stdout: "ready_epoch": 0, 2024-09-11T08:20:44.387 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_started": 0, 2024-09-11T08:20:44.387 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_clean": 0, 2024-09-11T08:20:44.387 INFO:teuthology.orchestra.run.smithi150.stdout: "source_version": "0'0", 2024-09-11T08:20:44.388 INFO:teuthology.orchestra.run.smithi150.stdout: "target_version": "0'0" 2024-09-11T08:20:44.388 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.388 INFO:teuthology.orchestra.run.smithi150.stdout: "last_change": "63", 2024-09-11T08:20:44.388 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend": "0", 2024-09-11T08:20:44.388 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T08:20:44.388 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T08:20:44.388 INFO:teuthology.orchestra.run.smithi150.stdout: "auid": 0, 2024-09-11T08:20:44.388 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_mode": "selfmanaged", 2024-09-11T08:20:44.388 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_seq": 3, 2024-09-11T08:20:44.388 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_epoch": 63, 2024-09-11T08:20:44.388 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_snaps": [], 2024-09-11T08:20:44.388 INFO:teuthology.orchestra.run.smithi150.stdout: "removed_snaps": "[]", 2024-09-11T08:20:44.389 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_bytes": 0, 2024-09-11T08:20:44.389 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_objects": 0, 2024-09-11T08:20:44.389 INFO:teuthology.orchestra.run.smithi150.stdout: "tiers": [], 2024-09-11T08:20:44.389 INFO:teuthology.orchestra.run.smithi150.stdout: "tier_of": -1, 2024-09-11T08:20:44.389 INFO:teuthology.orchestra.run.smithi150.stdout: "read_tier": -1, 2024-09-11T08:20:44.389 INFO:teuthology.orchestra.run.smithi150.stdout: "write_tier": -1, 2024-09-11T08:20:44.389 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_mode": "none", 2024-09-11T08:20:44.390 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_bytes": 0, 2024-09-11T08:20:44.390 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_objects": 0, 2024-09-11T08:20:44.390 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T08:20:44.390 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T08:20:44.390 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T08:20:44.390 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_flush_age": 0, 2024-09-11T08:20:44.390 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_evict_age": 0, 2024-09-11T08:20:44.390 INFO:teuthology.orchestra.run.smithi150.stdout: "erasure_code_profile": "", 2024-09-11T08:20:44.390 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_params": { 2024-09-11T08:20:44.390 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "none" 2024-09-11T08:20:44.390 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.390 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_period": 0, 2024-09-11T08:20:44.391 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_count": 0, 2024-09-11T08:20:44.391 INFO:teuthology.orchestra.run.smithi150.stdout: "use_gmt_hitset": true, 2024-09-11T08:20:44.391 INFO:teuthology.orchestra.run.smithi150.stdout: "min_read_recency_for_promote": 0, 2024-09-11T08:20:44.391 INFO:teuthology.orchestra.run.smithi150.stdout: "min_write_recency_for_promote": 0, 2024-09-11T08:20:44.391 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T08:20:44.391 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_search_last_n": 0, 2024-09-11T08:20:44.391 INFO:teuthology.orchestra.run.smithi150.stdout: "grade_table": [], 2024-09-11T08:20:44.391 INFO:teuthology.orchestra.run.smithi150.stdout: "stripe_width": 0, 2024-09-11T08:20:44.391 INFO:teuthology.orchestra.run.smithi150.stdout: "expected_num_objects": 0, 2024-09-11T08:20:44.391 INFO:teuthology.orchestra.run.smithi150.stdout: "fast_read": false, 2024-09-11T08:20:44.391 INFO:teuthology.orchestra.run.smithi150.stdout: "options": {}, 2024-09-11T08:20:44.392 INFO:teuthology.orchestra.run.smithi150.stdout: "application_metadata": { 2024-09-11T08:20:44.392 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd": {} 2024-09-11T08:20:44.392 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:44.392 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.392 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.392 INFO:teuthology.orchestra.run.smithi150.stdout: "pool": 3, 2024-09-11T08:20:44.392 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_name": ".rgw.root", 2024-09-11T08:20:44.392 INFO:teuthology.orchestra.run.smithi150.stdout: "create_time": "2024-09-11T08:07:29.021286+0000", 2024-09-11T08:20:44.392 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-09-11T08:20:44.392 INFO:teuthology.orchestra.run.smithi150.stdout: "flags_names": "hashpspool", 2024-09-11T08:20:44.393 INFO:teuthology.orchestra.run.smithi150.stdout: "type": 1, 2024-09-11T08:20:44.393 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 3, 2024-09-11T08:20:44.393 INFO:teuthology.orchestra.run.smithi150.stdout: "min_size": 2, 2024-09-11T08:20:44.393 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_rule": 0, 2024-09-11T08:20:44.393 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_count": 0, 2024-09-11T08:20:44.393 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_target": 0, 2024-09-11T08:20:44.393 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T08:20:44.393 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T08:20:44.393 INFO:teuthology.orchestra.run.smithi150.stdout: "object_hash": 2, 2024-09-11T08:20:44.393 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_autoscale_mode": "off", 2024-09-11T08:20:44.393 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num": 32, 2024-09-11T08:20:44.393 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num": 32, 2024-09-11T08:20:44.394 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num_target": 32, 2024-09-11T08:20:44.394 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_target": 32, 2024-09-11T08:20:44.394 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_pending": 32, 2024-09-11T08:20:44.394 INFO:teuthology.orchestra.run.smithi150.stdout: "last_pg_merge_meta": { 2024-09-11T08:20:44.394 INFO:teuthology.orchestra.run.smithi150.stdout: "source_pgid": "0.0", 2024-09-11T08:20:44.394 INFO:teuthology.orchestra.run.smithi150.stdout: "ready_epoch": 0, 2024-09-11T08:20:44.394 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_started": 0, 2024-09-11T08:20:44.394 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_clean": 0, 2024-09-11T08:20:44.394 INFO:teuthology.orchestra.run.smithi150.stdout: "source_version": "0'0", 2024-09-11T08:20:44.394 INFO:teuthology.orchestra.run.smithi150.stdout: "target_version": "0'0" 2024-09-11T08:20:44.394 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.394 INFO:teuthology.orchestra.run.smithi150.stdout: "last_change": "60", 2024-09-11T08:20:44.394 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend": "0", 2024-09-11T08:20:44.395 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T08:20:44.395 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T08:20:44.395 INFO:teuthology.orchestra.run.smithi150.stdout: "auid": 0, 2024-09-11T08:20:44.395 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_mode": "selfmanaged", 2024-09-11T08:20:44.395 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_seq": 0, 2024-09-11T08:20:44.395 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_epoch": 0, 2024-09-11T08:20:44.395 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_snaps": [], 2024-09-11T08:20:44.395 INFO:teuthology.orchestra.run.smithi150.stdout: "removed_snaps": "[]", 2024-09-11T08:20:44.395 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_bytes": 0, 2024-09-11T08:20:44.395 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_objects": 0, 2024-09-11T08:20:44.395 INFO:teuthology.orchestra.run.smithi150.stdout: "tiers": [], 2024-09-11T08:20:44.395 INFO:teuthology.orchestra.run.smithi150.stdout: "tier_of": -1, 2024-09-11T08:20:44.395 INFO:teuthology.orchestra.run.smithi150.stdout: "read_tier": -1, 2024-09-11T08:20:44.395 INFO:teuthology.orchestra.run.smithi150.stdout: "write_tier": -1, 2024-09-11T08:20:44.395 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_mode": "none", 2024-09-11T08:20:44.396 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_bytes": 0, 2024-09-11T08:20:44.396 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_objects": 0, 2024-09-11T08:20:44.396 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T08:20:44.396 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T08:20:44.396 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T08:20:44.396 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_flush_age": 0, 2024-09-11T08:20:44.396 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_evict_age": 0, 2024-09-11T08:20:44.396 INFO:teuthology.orchestra.run.smithi150.stdout: "erasure_code_profile": "", 2024-09-11T08:20:44.396 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_params": { 2024-09-11T08:20:44.396 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "none" 2024-09-11T08:20:44.396 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T08:20:44.397 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.397 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_period": 0, 2024-09-11T08:20:44.398 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_count": 0, 2024-09-11T08:20:44.398 INFO:teuthology.orchestra.run.smithi150.stdout: "use_gmt_hitset": true, 2024-09-11T08:20:44.398 INFO:teuthology.orchestra.run.smithi150.stdout: "min_read_recency_for_promote": 0, 2024-09-11T08:20:44.398 INFO:teuthology.orchestra.run.smithi150.stdout: "min_write_recency_for_promote": 0, 2024-09-11T08:20:44.398 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T08:20:44.398 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_search_last_n": 0, 2024-09-11T08:20:44.398 INFO:teuthology.orchestra.run.smithi150.stdout: "grade_table": [], 2024-09-11T08:20:44.398 INFO:teuthology.orchestra.run.smithi150.stdout: "stripe_width": 0, 2024-09-11T08:20:44.398 INFO:teuthology.orchestra.run.smithi150.stdout: "expected_num_objects": 0, 2024-09-11T08:20:44.398 INFO:teuthology.orchestra.run.smithi150.stdout: "fast_read": false, 2024-09-11T08:20:44.398 INFO:teuthology.orchestra.run.smithi150.stdout: "options": {}, 2024-09-11T08:20:44.398 INFO:teuthology.orchestra.run.smithi150.stdout: "application_metadata": { 2024-09-11T08:20:44.399 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw": {} 2024-09-11T08:20:44.399 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:44.399 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.399 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.399 INFO:teuthology.orchestra.run.smithi150.stdout: "pool": 4, 2024-09-11T08:20:44.399 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_name": "default.rgw.log", 2024-09-11T08:20:44.399 INFO:teuthology.orchestra.run.smithi150.stdout: "create_time": "2024-09-11T08:07:30.334181+0000", 2024-09-11T08:20:44.399 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-09-11T08:20:44.399 INFO:teuthology.orchestra.run.smithi150.stdout: "flags_names": "hashpspool", 2024-09-11T08:20:44.399 INFO:teuthology.orchestra.run.smithi150.stdout: "type": 1, 2024-09-11T08:20:44.399 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 3, 2024-09-11T08:20:44.399 INFO:teuthology.orchestra.run.smithi150.stdout: "min_size": 2, 2024-09-11T08:20:44.399 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_rule": 0, 2024-09-11T08:20:44.399 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_count": 0, 2024-09-11T08:20:44.399 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_target": 0, 2024-09-11T08:20:44.400 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T08:20:44.400 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T08:20:44.400 INFO:teuthology.orchestra.run.smithi150.stdout: "object_hash": 2, 2024-09-11T08:20:44.400 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_autoscale_mode": "off", 2024-09-11T08:20:44.400 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num": 32, 2024-09-11T08:20:44.400 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num": 32, 2024-09-11T08:20:44.400 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num_target": 32, 2024-09-11T08:20:44.400 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_target": 32, 2024-09-11T08:20:44.400 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_pending": 32, 2024-09-11T08:20:44.400 INFO:teuthology.orchestra.run.smithi150.stdout: "last_pg_merge_meta": { 2024-09-11T08:20:44.400 INFO:teuthology.orchestra.run.smithi150.stdout: "source_pgid": "0.0", 2024-09-11T08:20:44.400 INFO:teuthology.orchestra.run.smithi150.stdout: "ready_epoch": 0, 2024-09-11T08:20:44.400 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_started": 0, 2024-09-11T08:20:44.400 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_clean": 0, 2024-09-11T08:20:44.401 INFO:teuthology.orchestra.run.smithi150.stdout: "source_version": "0'0", 2024-09-11T08:20:44.401 INFO:teuthology.orchestra.run.smithi150.stdout: "target_version": "0'0" 2024-09-11T08:20:44.401 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.401 INFO:teuthology.orchestra.run.smithi150.stdout: "last_change": "62", 2024-09-11T08:20:44.401 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend": "0", 2024-09-11T08:20:44.401 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T08:20:44.401 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T08:20:44.401 INFO:teuthology.orchestra.run.smithi150.stdout: "auid": 0, 2024-09-11T08:20:44.401 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_mode": "selfmanaged", 2024-09-11T08:20:44.401 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_seq": 0, 2024-09-11T08:20:44.401 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_epoch": 0, 2024-09-11T08:20:44.401 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_snaps": [], 2024-09-11T08:20:44.401 INFO:teuthology.orchestra.run.smithi150.stdout: "removed_snaps": "[]", 2024-09-11T08:20:44.401 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_bytes": 0, 2024-09-11T08:20:44.402 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_objects": 0, 2024-09-11T08:20:44.402 INFO:teuthology.orchestra.run.smithi150.stdout: "tiers": [], 2024-09-11T08:20:44.402 INFO:teuthology.orchestra.run.smithi150.stdout: "tier_of": -1, 2024-09-11T08:20:44.402 INFO:teuthology.orchestra.run.smithi150.stdout: "read_tier": -1, 2024-09-11T08:20:44.402 INFO:teuthology.orchestra.run.smithi150.stdout: "write_tier": -1, 2024-09-11T08:20:44.402 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_mode": "none", 2024-09-11T08:20:44.402 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_bytes": 0, 2024-09-11T08:20:44.402 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_objects": 0, 2024-09-11T08:20:44.402 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T08:20:44.402 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T08:20:44.402 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T08:20:44.402 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_flush_age": 0, 2024-09-11T08:20:44.402 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_evict_age": 0, 2024-09-11T08:20:44.403 INFO:teuthology.orchestra.run.smithi150.stdout: "erasure_code_profile": "", 2024-09-11T08:20:44.403 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_params": { 2024-09-11T08:20:44.403 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "none" 2024-09-11T08:20:44.403 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.403 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_period": 0, 2024-09-11T08:20:44.403 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_count": 0, 2024-09-11T08:20:44.403 INFO:teuthology.orchestra.run.smithi150.stdout: "use_gmt_hitset": true, 2024-09-11T08:20:44.403 INFO:teuthology.orchestra.run.smithi150.stdout: "min_read_recency_for_promote": 0, 2024-09-11T08:20:44.403 INFO:teuthology.orchestra.run.smithi150.stdout: "min_write_recency_for_promote": 0, 2024-09-11T08:20:44.403 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T08:20:44.403 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_search_last_n": 0, 2024-09-11T08:20:44.403 INFO:teuthology.orchestra.run.smithi150.stdout: "grade_table": [], 2024-09-11T08:20:44.403 INFO:teuthology.orchestra.run.smithi150.stdout: "stripe_width": 0, 2024-09-11T08:20:44.404 INFO:teuthology.orchestra.run.smithi150.stdout: "expected_num_objects": 0, 2024-09-11T08:20:44.404 INFO:teuthology.orchestra.run.smithi150.stdout: "fast_read": false, 2024-09-11T08:20:44.404 INFO:teuthology.orchestra.run.smithi150.stdout: "options": {}, 2024-09-11T08:20:44.404 INFO:teuthology.orchestra.run.smithi150.stdout: "application_metadata": { 2024-09-11T08:20:44.404 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw": {} 2024-09-11T08:20:44.404 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:44.404 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.404 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.404 INFO:teuthology.orchestra.run.smithi150.stdout: "pool": 5, 2024-09-11T08:20:44.404 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_name": "default.rgw.control", 2024-09-11T08:20:44.404 INFO:teuthology.orchestra.run.smithi150.stdout: "create_time": "2024-09-11T08:07:32.289626+0000", 2024-09-11T08:20:44.404 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-09-11T08:20:44.405 INFO:teuthology.orchestra.run.smithi150.stdout: "flags_names": "hashpspool", 2024-09-11T08:20:44.405 INFO:teuthology.orchestra.run.smithi150.stdout: "type": 1, 2024-09-11T08:20:44.405 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 3, 2024-09-11T08:20:44.405 INFO:teuthology.orchestra.run.smithi150.stdout: "min_size": 2, 2024-09-11T08:20:44.405 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_rule": 0, 2024-09-11T08:20:44.405 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_count": 0, 2024-09-11T08:20:44.405 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_target": 0, 2024-09-11T08:20:44.405 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T08:20:44.405 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T08:20:44.405 INFO:teuthology.orchestra.run.smithi150.stdout: "object_hash": 2, 2024-09-11T08:20:44.405 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_autoscale_mode": "off", 2024-09-11T08:20:44.405 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num": 32, 2024-09-11T08:20:44.405 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num": 32, 2024-09-11T08:20:44.406 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num_target": 32, 2024-09-11T08:20:44.406 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_target": 32, 2024-09-11T08:20:44.406 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_pending": 32, 2024-09-11T08:20:44.406 INFO:teuthology.orchestra.run.smithi150.stdout: "last_pg_merge_meta": { 2024-09-11T08:20:44.406 INFO:teuthology.orchestra.run.smithi150.stdout: "source_pgid": "0.0", 2024-09-11T08:20:44.406 INFO:teuthology.orchestra.run.smithi150.stdout: "ready_epoch": 0, 2024-09-11T08:20:44.406 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_started": 0, 2024-09-11T08:20:44.406 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_clean": 0, 2024-09-11T08:20:44.406 INFO:teuthology.orchestra.run.smithi150.stdout: "source_version": "0'0", 2024-09-11T08:20:44.406 INFO:teuthology.orchestra.run.smithi150.stdout: "target_version": "0'0" 2024-09-11T08:20:44.406 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.406 INFO:teuthology.orchestra.run.smithi150.stdout: "last_change": "64", 2024-09-11T08:20:44.407 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend": "0", 2024-09-11T08:20:44.407 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T08:20:44.407 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T08:20:44.407 INFO:teuthology.orchestra.run.smithi150.stdout: "auid": 0, 2024-09-11T08:20:44.407 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_mode": "selfmanaged", 2024-09-11T08:20:44.407 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_seq": 0, 2024-09-11T08:20:44.407 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_epoch": 0, 2024-09-11T08:20:44.407 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_snaps": [], 2024-09-11T08:20:44.407 INFO:teuthology.orchestra.run.smithi150.stdout: "removed_snaps": "[]", 2024-09-11T08:20:44.407 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_bytes": 0, 2024-09-11T08:20:44.407 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_objects": 0, 2024-09-11T08:20:44.407 INFO:teuthology.orchestra.run.smithi150.stdout: "tiers": [], 2024-09-11T08:20:44.407 INFO:teuthology.orchestra.run.smithi150.stdout: "tier_of": -1, 2024-09-11T08:20:44.407 INFO:teuthology.orchestra.run.smithi150.stdout: "read_tier": -1, 2024-09-11T08:20:44.408 INFO:teuthology.orchestra.run.smithi150.stdout: "write_tier": -1, 2024-09-11T08:20:44.408 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_mode": "none", 2024-09-11T08:20:44.408 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_bytes": 0, 2024-09-11T08:20:44.408 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_objects": 0, 2024-09-11T08:20:44.408 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T08:20:44.408 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T08:20:44.408 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T08:20:44.408 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_flush_age": 0, 2024-09-11T08:20:44.408 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_evict_age": 0, 2024-09-11T08:20:44.408 INFO:teuthology.orchestra.run.smithi150.stdout: "erasure_code_profile": "", 2024-09-11T08:20:44.408 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_params": { 2024-09-11T08:20:44.408 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "none" 2024-09-11T08:20:44.408 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.409 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_period": 0, 2024-09-11T08:20:44.409 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_count": 0, 2024-09-11T08:20:44.409 INFO:teuthology.orchestra.run.smithi150.stdout: "use_gmt_hitset": true, 2024-09-11T08:20:44.409 INFO:teuthology.orchestra.run.smithi150.stdout: "min_read_recency_for_promote": 0, 2024-09-11T08:20:44.409 INFO:teuthology.orchestra.run.smithi150.stdout: "min_write_recency_for_promote": 0, 2024-09-11T08:20:44.409 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T08:20:44.409 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_search_last_n": 0, 2024-09-11T08:20:44.409 INFO:teuthology.orchestra.run.smithi150.stdout: "grade_table": [], 2024-09-11T08:20:44.409 INFO:teuthology.orchestra.run.smithi150.stdout: "stripe_width": 0, 2024-09-11T08:20:44.409 INFO:teuthology.orchestra.run.smithi150.stdout: "expected_num_objects": 0, 2024-09-11T08:20:44.409 INFO:teuthology.orchestra.run.smithi150.stdout: "fast_read": false, 2024-09-11T08:20:44.409 INFO:teuthology.orchestra.run.smithi150.stdout: "options": {}, 2024-09-11T08:20:44.409 INFO:teuthology.orchestra.run.smithi150.stdout: "application_metadata": { 2024-09-11T08:20:44.410 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw": {} 2024-09-11T08:20:44.410 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:44.410 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.410 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.410 INFO:teuthology.orchestra.run.smithi150.stdout: "pool": 6, 2024-09-11T08:20:44.410 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_name": "default.rgw.meta", 2024-09-11T08:20:44.410 INFO:teuthology.orchestra.run.smithi150.stdout: "create_time": "2024-09-11T08:07:34.351861+0000", 2024-09-11T08:20:44.410 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-09-11T08:20:44.410 INFO:teuthology.orchestra.run.smithi150.stdout: "flags_names": "hashpspool", 2024-09-11T08:20:44.410 INFO:teuthology.orchestra.run.smithi150.stdout: "type": 1, 2024-09-11T08:20:44.411 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 3, 2024-09-11T08:20:44.411 INFO:teuthology.orchestra.run.smithi150.stdout: "min_size": 2, 2024-09-11T08:20:44.411 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_rule": 0, 2024-09-11T08:20:44.411 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_count": 0, 2024-09-11T08:20:44.411 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_target": 0, 2024-09-11T08:20:44.411 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T08:20:44.411 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T08:20:44.411 INFO:teuthology.orchestra.run.smithi150.stdout: "object_hash": 2, 2024-09-11T08:20:44.411 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_autoscale_mode": "off", 2024-09-11T08:20:44.411 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num": 32, 2024-09-11T08:20:44.411 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num": 32, 2024-09-11T08:20:44.411 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num_target": 32, 2024-09-11T08:20:44.411 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_target": 32, 2024-09-11T08:20:44.412 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_pending": 32, 2024-09-11T08:20:44.412 INFO:teuthology.orchestra.run.smithi150.stdout: "last_pg_merge_meta": { 2024-09-11T08:20:44.412 INFO:teuthology.orchestra.run.smithi150.stdout: "source_pgid": "0.0", 2024-09-11T08:20:44.412 INFO:teuthology.orchestra.run.smithi150.stdout: "ready_epoch": 0, 2024-09-11T08:20:44.412 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_started": 0, 2024-09-11T08:20:44.412 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_clean": 0, 2024-09-11T08:20:44.412 INFO:teuthology.orchestra.run.smithi150.stdout: "source_version": "0'0", 2024-09-11T08:20:44.412 INFO:teuthology.orchestra.run.smithi150.stdout: "target_version": "0'0" 2024-09-11T08:20:44.412 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.412 INFO:teuthology.orchestra.run.smithi150.stdout: "last_change": "66", 2024-09-11T08:20:44.412 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend": "0", 2024-09-11T08:20:44.412 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T08:20:44.412 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T08:20:44.413 INFO:teuthology.orchestra.run.smithi150.stdout: "auid": 0, 2024-09-11T08:20:44.413 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_mode": "selfmanaged", 2024-09-11T08:20:44.413 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_seq": 0, 2024-09-11T08:20:44.413 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_epoch": 0, 2024-09-11T08:20:44.413 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_snaps": [], 2024-09-11T08:20:44.413 INFO:teuthology.orchestra.run.smithi150.stdout: "removed_snaps": "[]", 2024-09-11T08:20:44.413 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_bytes": 0, 2024-09-11T08:20:44.413 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_objects": 0, 2024-09-11T08:20:44.413 INFO:teuthology.orchestra.run.smithi150.stdout: "tiers": [], 2024-09-11T08:20:44.413 INFO:teuthology.orchestra.run.smithi150.stdout: "tier_of": -1, 2024-09-11T08:20:44.413 INFO:teuthology.orchestra.run.smithi150.stdout: "read_tier": -1, 2024-09-11T08:20:44.413 INFO:teuthology.orchestra.run.smithi150.stdout: "write_tier": -1, 2024-09-11T08:20:44.413 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_mode": "none", 2024-09-11T08:20:44.414 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_bytes": 0, 2024-09-11T08:20:44.414 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_objects": 0, 2024-09-11T08:20:44.414 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T08:20:44.414 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T08:20:44.414 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T08:20:44.414 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_flush_age": 0, 2024-09-11T08:20:44.414 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_evict_age": 0, 2024-09-11T08:20:44.414 INFO:teuthology.orchestra.run.smithi150.stdout: "erasure_code_profile": "", 2024-09-11T08:20:44.414 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_params": { 2024-09-11T08:20:44.414 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "none" 2024-09-11T08:20:44.414 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.414 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_period": 0, 2024-09-11T08:20:44.414 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_count": 0, 2024-09-11T08:20:44.414 INFO:teuthology.orchestra.run.smithi150.stdout: "use_gmt_hitset": true, 2024-09-11T08:20:44.415 INFO:teuthology.orchestra.run.smithi150.stdout: "min_read_recency_for_promote": 0, 2024-09-11T08:20:44.415 INFO:teuthology.orchestra.run.smithi150.stdout: "min_write_recency_for_promote": 0, 2024-09-11T08:20:44.415 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T08:20:44.415 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_search_last_n": 0, 2024-09-11T08:20:44.415 INFO:teuthology.orchestra.run.smithi150.stdout: "grade_table": [], 2024-09-11T08:20:44.415 INFO:teuthology.orchestra.run.smithi150.stdout: "stripe_width": 0, 2024-09-11T08:20:44.415 INFO:teuthology.orchestra.run.smithi150.stdout: "expected_num_objects": 0, 2024-09-11T08:20:44.415 INFO:teuthology.orchestra.run.smithi150.stdout: "fast_read": false, 2024-09-11T08:20:44.415 INFO:teuthology.orchestra.run.smithi150.stdout: "options": { 2024-09-11T08:20:44.415 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_autoscale_bias": 4 2024-09-11T08:20:44.415 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.416 INFO:teuthology.orchestra.run.smithi150.stdout: "application_metadata": { 2024-09-11T08:20:44.416 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw": {} 2024-09-11T08:20:44.416 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:44.416 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.416 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.416 INFO:teuthology.orchestra.run.smithi150.stdout: "pool": 9, 2024-09-11T08:20:44.416 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_name": "unique_pool_2", 2024-09-11T08:20:44.416 INFO:teuthology.orchestra.run.smithi150.stdout: "create_time": "2024-09-11T08:17:59.761532+0000", 2024-09-11T08:20:44.416 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-09-11T08:20:44.416 INFO:teuthology.orchestra.run.smithi150.stdout: "flags_names": "hashpspool", 2024-09-11T08:20:44.416 INFO:teuthology.orchestra.run.smithi150.stdout: "type": 1, 2024-09-11T08:20:44.416 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 3, 2024-09-11T08:20:44.416 INFO:teuthology.orchestra.run.smithi150.stdout: "min_size": 2, 2024-09-11T08:20:44.417 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_rule": 0, 2024-09-11T08:20:44.417 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_count": 0, 2024-09-11T08:20:44.417 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_target": 0, 2024-09-11T08:20:44.417 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T08:20:44.417 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T08:20:44.417 INFO:teuthology.orchestra.run.smithi150.stdout: "object_hash": 2, 2024-09-11T08:20:44.417 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_autoscale_mode": "off", 2024-09-11T08:20:44.417 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num": 16, 2024-09-11T08:20:44.417 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num": 16, 2024-09-11T08:20:44.417 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num_target": 16, 2024-09-11T08:20:44.417 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_target": 16, 2024-09-11T08:20:44.417 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_pending": 16, 2024-09-11T08:20:44.417 INFO:teuthology.orchestra.run.smithi150.stdout: "last_pg_merge_meta": { 2024-09-11T08:20:44.417 INFO:teuthology.orchestra.run.smithi150.stdout: "source_pgid": "0.0", 2024-09-11T08:20:44.418 INFO:teuthology.orchestra.run.smithi150.stdout: "ready_epoch": 0, 2024-09-11T08:20:44.418 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_started": 0, 2024-09-11T08:20:44.418 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_clean": 0, 2024-09-11T08:20:44.418 INFO:teuthology.orchestra.run.smithi150.stdout: "source_version": "0'0", 2024-09-11T08:20:44.418 INFO:teuthology.orchestra.run.smithi150.stdout: "target_version": "0'0" 2024-09-11T08:20:44.418 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.418 INFO:teuthology.orchestra.run.smithi150.stdout: "last_change": "251", 2024-09-11T08:20:44.418 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend": "0", 2024-09-11T08:20:44.418 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T08:20:44.418 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T08:20:44.418 INFO:teuthology.orchestra.run.smithi150.stdout: "auid": 0, 2024-09-11T08:20:44.418 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_mode": "selfmanaged", 2024-09-11T08:20:44.418 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_seq": 0, 2024-09-11T08:20:44.418 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_epoch": 0, 2024-09-11T08:20:44.419 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_snaps": [], 2024-09-11T08:20:44.419 INFO:teuthology.orchestra.run.smithi150.stdout: "removed_snaps": "[]", 2024-09-11T08:20:44.419 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_bytes": 0, 2024-09-11T08:20:44.419 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_objects": 0, 2024-09-11T08:20:44.419 INFO:teuthology.orchestra.run.smithi150.stdout: "tiers": [], 2024-09-11T08:20:44.419 INFO:teuthology.orchestra.run.smithi150.stdout: "tier_of": -1, 2024-09-11T08:20:44.419 INFO:teuthology.orchestra.run.smithi150.stdout: "read_tier": -1, 2024-09-11T08:20:44.419 INFO:teuthology.orchestra.run.smithi150.stdout: "write_tier": -1, 2024-09-11T08:20:44.419 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_mode": "none", 2024-09-11T08:20:44.419 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_bytes": 0, 2024-09-11T08:20:44.419 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_objects": 0, 2024-09-11T08:20:44.419 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T08:20:44.419 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T08:20:44.419 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T08:20:44.420 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_flush_age": 0, 2024-09-11T08:20:44.420 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_evict_age": 0, 2024-09-11T08:20:44.420 INFO:teuthology.orchestra.run.smithi150.stdout: "erasure_code_profile": "", 2024-09-11T08:20:44.420 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_params": { 2024-09-11T08:20:44.420 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "none" 2024-09-11T08:20:44.420 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.420 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_period": 0, 2024-09-11T08:20:44.420 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_count": 0, 2024-09-11T08:20:44.420 INFO:teuthology.orchestra.run.smithi150.stdout: "use_gmt_hitset": true, 2024-09-11T08:20:44.420 INFO:teuthology.orchestra.run.smithi150.stdout: "min_read_recency_for_promote": 0, 2024-09-11T08:20:44.420 INFO:teuthology.orchestra.run.smithi150.stdout: "min_write_recency_for_promote": 0, 2024-09-11T08:20:44.421 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T08:20:44.421 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_search_last_n": 0, 2024-09-11T08:20:44.421 INFO:teuthology.orchestra.run.smithi150.stdout: "grade_table": [], 2024-09-11T08:20:44.421 INFO:teuthology.orchestra.run.smithi150.stdout: "stripe_width": 0, 2024-09-11T08:20:44.421 INFO:teuthology.orchestra.run.smithi150.stdout: "expected_num_objects": 0, 2024-09-11T08:20:44.421 INFO:teuthology.orchestra.run.smithi150.stdout: "fast_read": false, 2024-09-11T08:20:44.421 INFO:teuthology.orchestra.run.smithi150.stdout: "options": {}, 2024-09-11T08:20:44.421 INFO:teuthology.orchestra.run.smithi150.stdout: "application_metadata": { 2024-09-11T08:20:44.422 INFO:teuthology.orchestra.run.smithi150.stdout: "rados": {} 2024-09-11T08:20:44.422 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:44.422 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:44.422 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-09-11T08:20:44.422 INFO:teuthology.orchestra.run.smithi150.stdout: "osds": [ 2024-09-11T08:20:44.422 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.422 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 0, 2024-09-11T08:20:44.422 INFO:teuthology.orchestra.run.smithi150.stdout: "uuid": "37b63db9-ed2a-4701-b63d-03ff9885cc0e", 2024-09-11T08:20:44.422 INFO:teuthology.orchestra.run.smithi150.stdout: "up": 1, 2024-09-11T08:20:44.422 INFO:teuthology.orchestra.run.smithi150.stdout: "in": 1, 2024-09-11T08:20:44.423 INFO:teuthology.orchestra.run.smithi150.stdout: "weight": 1, 2024-09-11T08:20:44.423 INFO:teuthology.orchestra.run.smithi150.stdout: "primary_affinity": 1, 2024-09-11T08:20:44.423 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_begin": 0, 2024-09-11T08:20:44.423 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_end": 0, 2024-09-11T08:20:44.423 INFO:teuthology.orchestra.run.smithi150.stdout: "up_from": 8, 2024-09-11T08:20:44.423 INFO:teuthology.orchestra.run.smithi150.stdout: "up_thru": 276, 2024-09-11T08:20:44.423 INFO:teuthology.orchestra.run.smithi150.stdout: "down_at": 0, 2024-09-11T08:20:44.423 INFO:teuthology.orchestra.run.smithi150.stdout: "lost_at": 0, 2024-09-11T08:20:44.423 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addrs": { 2024-09-11T08:20:44.423 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:20:44.424 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.424 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:20:44.424 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6801", 2024-09-11T08:20:44.424 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 334255279 2024-09-11T08:20:44.424 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:44.424 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:44.424 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.424 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addrs": { 2024-09-11T08:20:44.424 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:20:44.425 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.425 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:20:44.425 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6802", 2024-09-11T08:20:44.425 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 334255279 2024-09-11T08:20:44.425 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:44.425 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:44.425 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.425 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addrs": { 2024-09-11T08:20:44.425 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:20:44.425 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.425 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:20:44.425 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6804", 2024-09-11T08:20:44.426 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 334255279 2024-09-11T08:20:44.426 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:44.426 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:44.426 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.426 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addrs": { 2024-09-11T08:20:44.426 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:20:44.426 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.426 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:20:44.426 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6803", 2024-09-11T08:20:44.426 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 334255279 2024-09-11T08:20:44.426 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:44.426 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:44.426 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.427 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addr": "172.21.15.150:6801/334255279", 2024-09-11T08:20:44.427 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addr": "172.21.15.150:6802/334255279", 2024-09-11T08:20:44.427 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addr": "172.21.15.150:6804/334255279", 2024-09-11T08:20:44.427 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addr": "172.21.15.150:6803/334255279", 2024-09-11T08:20:44.427 INFO:teuthology.orchestra.run.smithi150.stdout: "state": [ 2024-09-11T08:20:44.427 INFO:teuthology.orchestra.run.smithi150.stdout: "exists", 2024-09-11T08:20:44.427 INFO:teuthology.orchestra.run.smithi150.stdout: "up" 2024-09-11T08:20:44.428 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:44.428 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.428 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.428 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 1, 2024-09-11T08:20:44.428 INFO:teuthology.orchestra.run.smithi150.stdout: "uuid": "80378442-02bc-48aa-b8ca-6113a4092079", 2024-09-11T08:20:44.428 INFO:teuthology.orchestra.run.smithi150.stdout: "up": 1, 2024-09-11T08:20:44.428 INFO:teuthology.orchestra.run.smithi150.stdout: "in": 1, 2024-09-11T08:20:44.428 INFO:teuthology.orchestra.run.smithi150.stdout: "weight": 1, 2024-09-11T08:20:44.428 INFO:teuthology.orchestra.run.smithi150.stdout: "primary_affinity": 1, 2024-09-11T08:20:44.428 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_begin": 0, 2024-09-11T08:20:44.429 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_end": 0, 2024-09-11T08:20:44.429 INFO:teuthology.orchestra.run.smithi150.stdout: "up_from": 13, 2024-09-11T08:20:44.429 INFO:teuthology.orchestra.run.smithi150.stdout: "up_thru": 270, 2024-09-11T08:20:44.429 INFO:teuthology.orchestra.run.smithi150.stdout: "down_at": 0, 2024-09-11T08:20:44.429 INFO:teuthology.orchestra.run.smithi150.stdout: "lost_at": 0, 2024-09-11T08:20:44.429 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addrs": { 2024-09-11T08:20:44.429 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:20:44.429 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.429 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:20:44.429 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6805", 2024-09-11T08:20:44.429 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1129510619 2024-09-11T08:20:44.429 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:44.429 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:44.429 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.430 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addrs": { 2024-09-11T08:20:44.430 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:20:44.430 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.430 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:20:44.430 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6806", 2024-09-11T08:20:44.430 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1129510619 2024-09-11T08:20:44.430 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:44.430 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:44.430 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.430 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addrs": { 2024-09-11T08:20:44.430 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:20:44.430 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.430 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:20:44.430 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6808", 2024-09-11T08:20:44.431 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1129510619 2024-09-11T08:20:44.431 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:44.431 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:44.431 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.431 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addrs": { 2024-09-11T08:20:44.431 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:20:44.431 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.431 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:20:44.431 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6807", 2024-09-11T08:20:44.431 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1129510619 2024-09-11T08:20:44.431 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:44.431 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:44.431 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.431 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addr": "172.21.15.150:6805/1129510619", 2024-09-11T08:20:44.432 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addr": "172.21.15.150:6806/1129510619", 2024-09-11T08:20:44.432 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addr": "172.21.15.150:6808/1129510619", 2024-09-11T08:20:44.432 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addr": "172.21.15.150:6807/1129510619", 2024-09-11T08:20:44.432 INFO:teuthology.orchestra.run.smithi150.stdout: "state": [ 2024-09-11T08:20:44.432 INFO:teuthology.orchestra.run.smithi150.stdout: "exists", 2024-09-11T08:20:44.432 INFO:teuthology.orchestra.run.smithi150.stdout: "up" 2024-09-11T08:20:44.432 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:44.432 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.432 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.432 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 2, 2024-09-11T08:20:44.432 INFO:teuthology.orchestra.run.smithi150.stdout: "uuid": "0955bb92-36a0-4e9a-98e1-b3a5af89f6d2", 2024-09-11T08:20:44.432 INFO:teuthology.orchestra.run.smithi150.stdout: "up": 1, 2024-09-11T08:20:44.433 INFO:teuthology.orchestra.run.smithi150.stdout: "in": 1, 2024-09-11T08:20:44.433 INFO:teuthology.orchestra.run.smithi150.stdout: "weight": 1, 2024-09-11T08:20:44.433 INFO:teuthology.orchestra.run.smithi150.stdout: "primary_affinity": 1, 2024-09-11T08:20:44.433 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_begin": 0, 2024-09-11T08:20:44.433 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_end": 0, 2024-09-11T08:20:44.433 INFO:teuthology.orchestra.run.smithi150.stdout: "up_from": 18, 2024-09-11T08:20:44.433 INFO:teuthology.orchestra.run.smithi150.stdout: "up_thru": 284, 2024-09-11T08:20:44.433 INFO:teuthology.orchestra.run.smithi150.stdout: "down_at": 0, 2024-09-11T08:20:44.433 INFO:teuthology.orchestra.run.smithi150.stdout: "lost_at": 0, 2024-09-11T08:20:44.433 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addrs": { 2024-09-11T08:20:44.433 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:20:44.433 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.433 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:20:44.434 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6809", 2024-09-11T08:20:44.434 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 719975422 2024-09-11T08:20:44.434 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:44.434 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:44.434 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.434 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addrs": { 2024-09-11T08:20:44.434 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:20:44.434 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.434 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:20:44.434 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6810", 2024-09-11T08:20:44.434 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 719975422 2024-09-11T08:20:44.434 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:44.434 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:44.434 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.435 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addrs": { 2024-09-11T08:20:44.435 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:20:44.435 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.435 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:20:44.435 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6812", 2024-09-11T08:20:44.435 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 719975422 2024-09-11T08:20:44.435 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:44.435 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:44.435 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.435 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addrs": { 2024-09-11T08:20:44.435 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:20:44.436 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.436 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:20:44.436 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6811", 2024-09-11T08:20:44.436 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 719975422 2024-09-11T08:20:44.436 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:44.436 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:44.436 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.436 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addr": "172.21.15.150:6809/719975422", 2024-09-11T08:20:44.436 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addr": "172.21.15.150:6810/719975422", 2024-09-11T08:20:44.436 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addr": "172.21.15.150:6812/719975422", 2024-09-11T08:20:44.436 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addr": "172.21.15.150:6811/719975422", 2024-09-11T08:20:44.436 INFO:teuthology.orchestra.run.smithi150.stdout: "state": [ 2024-09-11T08:20:44.436 INFO:teuthology.orchestra.run.smithi150.stdout: "exists", 2024-09-11T08:20:44.437 INFO:teuthology.orchestra.run.smithi150.stdout: "up" 2024-09-11T08:20:44.437 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:44.437 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.438 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.438 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 3, 2024-09-11T08:20:44.438 INFO:teuthology.orchestra.run.smithi150.stdout: "uuid": "1d04d4d3-62e8-4460-8a07-77f7e5c62929", 2024-09-11T08:20:44.438 INFO:teuthology.orchestra.run.smithi150.stdout: "up": 1, 2024-09-11T08:20:44.438 INFO:teuthology.orchestra.run.smithi150.stdout: "in": 1, 2024-09-11T08:20:44.438 INFO:teuthology.orchestra.run.smithi150.stdout: "weight": 1, 2024-09-11T08:20:44.438 INFO:teuthology.orchestra.run.smithi150.stdout: "primary_affinity": 0.611114501953125, 2024-09-11T08:20:44.438 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_begin": 0, 2024-09-11T08:20:44.438 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_end": 0, 2024-09-11T08:20:44.438 INFO:teuthology.orchestra.run.smithi150.stdout: "up_from": 25, 2024-09-11T08:20:44.439 INFO:teuthology.orchestra.run.smithi150.stdout: "up_thru": 260, 2024-09-11T08:20:44.439 INFO:teuthology.orchestra.run.smithi150.stdout: "down_at": 0, 2024-09-11T08:20:44.439 INFO:teuthology.orchestra.run.smithi150.stdout: "lost_at": 0, 2024-09-11T08:20:44.439 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addrs": { 2024-09-11T08:20:44.439 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:20:44.439 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.439 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:20:44.439 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6813", 2024-09-11T08:20:44.439 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 3101032500 2024-09-11T08:20:44.439 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:44.439 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:44.439 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.439 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addrs": { 2024-09-11T08:20:44.440 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:20:44.440 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.440 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:20:44.440 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6814", 2024-09-11T08:20:44.440 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 3101032500 2024-09-11T08:20:44.440 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:44.440 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:44.440 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.440 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addrs": { 2024-09-11T08:20:44.440 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:20:44.440 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.440 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:20:44.440 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6816", 2024-09-11T08:20:44.441 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 3101032500 2024-09-11T08:20:44.441 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:44.441 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:44.441 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.441 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addrs": { 2024-09-11T08:20:44.441 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:20:44.441 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.441 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:20:44.441 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6815", 2024-09-11T08:20:44.441 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 3101032500 2024-09-11T08:20:44.441 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:44.442 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:44.442 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.442 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addr": "172.21.15.150:6813/3101032500", 2024-09-11T08:20:44.442 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addr": "172.21.15.150:6814/3101032500", 2024-09-11T08:20:44.442 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addr": "172.21.15.150:6816/3101032500", 2024-09-11T08:20:44.442 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addr": "172.21.15.150:6815/3101032500", 2024-09-11T08:20:44.442 INFO:teuthology.orchestra.run.smithi150.stdout: "state": [ 2024-09-11T08:20:44.442 INFO:teuthology.orchestra.run.smithi150.stdout: "exists", 2024-09-11T08:20:44.442 INFO:teuthology.orchestra.run.smithi150.stdout: "up" 2024-09-11T08:20:44.442 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:44.442 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.442 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.442 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 4, 2024-09-11T08:20:44.442 INFO:teuthology.orchestra.run.smithi150.stdout: "uuid": "c5321a3c-d2f0-4440-8475-2c111d261139", 2024-09-11T08:20:44.443 INFO:teuthology.orchestra.run.smithi150.stdout: "up": 1, 2024-09-11T08:20:44.443 INFO:teuthology.orchestra.run.smithi150.stdout: "in": 1, 2024-09-11T08:20:44.443 INFO:teuthology.orchestra.run.smithi150.stdout: "weight": 1, 2024-09-11T08:20:44.443 INFO:teuthology.orchestra.run.smithi150.stdout: "primary_affinity": 1, 2024-09-11T08:20:44.443 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_begin": 0, 2024-09-11T08:20:44.443 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_end": 0, 2024-09-11T08:20:44.443 INFO:teuthology.orchestra.run.smithi150.stdout: "up_from": 31, 2024-09-11T08:20:44.443 INFO:teuthology.orchestra.run.smithi150.stdout: "up_thru": 270, 2024-09-11T08:20:44.443 INFO:teuthology.orchestra.run.smithi150.stdout: "down_at": 0, 2024-09-11T08:20:44.443 INFO:teuthology.orchestra.run.smithi150.stdout: "lost_at": 0, 2024-09-11T08:20:44.443 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addrs": { 2024-09-11T08:20:44.443 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:20:44.443 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.443 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:20:44.444 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6800", 2024-09-11T08:20:44.444 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2567452825 2024-09-11T08:20:44.444 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:44.444 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:44.444 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.444 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addrs": { 2024-09-11T08:20:44.444 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:20:44.445 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.445 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:20:44.445 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6801", 2024-09-11T08:20:44.445 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2567452825 2024-09-11T08:20:44.445 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:44.446 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:44.446 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.446 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addrs": { 2024-09-11T08:20:44.446 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:20:44.446 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.446 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:20:44.446 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6803", 2024-09-11T08:20:44.447 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2567452825 2024-09-11T08:20:44.447 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:44.447 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:44.447 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.447 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addrs": { 2024-09-11T08:20:44.447 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:20:44.447 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.448 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:20:44.448 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6802", 2024-09-11T08:20:44.448 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2567452825 2024-09-11T08:20:44.448 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:44.448 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:44.448 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.448 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addr": "172.21.15.181:6800/2567452825", 2024-09-11T08:20:44.449 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addr": "172.21.15.181:6801/2567452825", 2024-09-11T08:20:44.449 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addr": "172.21.15.181:6803/2567452825", 2024-09-11T08:20:44.449 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addr": "172.21.15.181:6802/2567452825", 2024-09-11T08:20:44.449 INFO:teuthology.orchestra.run.smithi150.stdout: "state": [ 2024-09-11T08:20:44.449 INFO:teuthology.orchestra.run.smithi150.stdout: "exists", 2024-09-11T08:20:44.449 INFO:teuthology.orchestra.run.smithi150.stdout: "up" 2024-09-11T08:20:44.449 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:44.449 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.450 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.450 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 5, 2024-09-11T08:20:44.450 INFO:teuthology.orchestra.run.smithi150.stdout: "uuid": "1262b2fe-f253-4a26-bfd4-c9dffb697c77", 2024-09-11T08:20:44.450 INFO:teuthology.orchestra.run.smithi150.stdout: "up": 1, 2024-09-11T08:20:44.450 INFO:teuthology.orchestra.run.smithi150.stdout: "in": 1, 2024-09-11T08:20:44.451 INFO:teuthology.orchestra.run.smithi150.stdout: "weight": 1, 2024-09-11T08:20:44.451 INFO:teuthology.orchestra.run.smithi150.stdout: "primary_affinity": 1, 2024-09-11T08:20:44.451 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_begin": 0, 2024-09-11T08:20:44.451 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_end": 0, 2024-09-11T08:20:44.451 INFO:teuthology.orchestra.run.smithi150.stdout: "up_from": 38, 2024-09-11T08:20:44.451 INFO:teuthology.orchestra.run.smithi150.stdout: "up_thru": 271, 2024-09-11T08:20:44.451 INFO:teuthology.orchestra.run.smithi150.stdout: "down_at": 0, 2024-09-11T08:20:44.452 INFO:teuthology.orchestra.run.smithi150.stdout: "lost_at": 0, 2024-09-11T08:20:44.452 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addrs": { 2024-09-11T08:20:44.452 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:20:44.452 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.452 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:20:44.452 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6804", 2024-09-11T08:20:44.453 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2043435508 2024-09-11T08:20:44.453 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:44.453 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:44.453 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.453 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addrs": { 2024-09-11T08:20:44.453 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:20:44.453 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.453 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:20:44.453 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6805", 2024-09-11T08:20:44.454 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2043435508 2024-09-11T08:20:44.454 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:44.454 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:44.454 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.454 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addrs": { 2024-09-11T08:20:44.454 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:20:44.454 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.454 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:20:44.454 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6807", 2024-09-11T08:20:44.454 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2043435508 2024-09-11T08:20:44.454 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:44.455 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:44.455 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.455 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addrs": { 2024-09-11T08:20:44.455 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:20:44.455 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.455 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:20:44.455 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6806", 2024-09-11T08:20:44.455 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2043435508 2024-09-11T08:20:44.455 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:44.455 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:44.455 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.455 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addr": "172.21.15.181:6804/2043435508", 2024-09-11T08:20:44.455 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addr": "172.21.15.181:6805/2043435508", 2024-09-11T08:20:44.455 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addr": "172.21.15.181:6807/2043435508", 2024-09-11T08:20:44.456 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addr": "172.21.15.181:6806/2043435508", 2024-09-11T08:20:44.456 INFO:teuthology.orchestra.run.smithi150.stdout: "state": [ 2024-09-11T08:20:44.456 INFO:teuthology.orchestra.run.smithi150.stdout: "exists", 2024-09-11T08:20:44.456 INFO:teuthology.orchestra.run.smithi150.stdout: "up" 2024-09-11T08:20:44.456 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:44.456 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.456 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.456 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 6, 2024-09-11T08:20:44.457 INFO:teuthology.orchestra.run.smithi150.stdout: "uuid": "7facca22-b3f1-4c20-833e-e3db798a08c4", 2024-09-11T08:20:44.457 INFO:teuthology.orchestra.run.smithi150.stdout: "up": 1, 2024-09-11T08:20:44.457 INFO:teuthology.orchestra.run.smithi150.stdout: "in": 1, 2024-09-11T08:20:44.457 INFO:teuthology.orchestra.run.smithi150.stdout: "weight": 1, 2024-09-11T08:20:44.457 INFO:teuthology.orchestra.run.smithi150.stdout: "primary_affinity": 1, 2024-09-11T08:20:44.457 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_begin": 0, 2024-09-11T08:20:44.457 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_end": 0, 2024-09-11T08:20:44.457 INFO:teuthology.orchestra.run.smithi150.stdout: "up_from": 45, 2024-09-11T08:20:44.457 INFO:teuthology.orchestra.run.smithi150.stdout: "up_thru": 270, 2024-09-11T08:20:44.457 INFO:teuthology.orchestra.run.smithi150.stdout: "down_at": 0, 2024-09-11T08:20:44.457 INFO:teuthology.orchestra.run.smithi150.stdout: "lost_at": 0, 2024-09-11T08:20:44.457 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addrs": { 2024-09-11T08:20:44.458 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:20:44.458 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.458 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:20:44.458 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6808", 2024-09-11T08:20:44.458 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1789880331 2024-09-11T08:20:44.458 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:44.458 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:44.458 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.458 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addrs": { 2024-09-11T08:20:44.458 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:20:44.458 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.458 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:20:44.458 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6809", 2024-09-11T08:20:44.458 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1789880331 2024-09-11T08:20:44.459 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:44.459 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:44.459 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.459 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addrs": { 2024-09-11T08:20:44.459 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:20:44.459 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.459 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:20:44.459 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6811", 2024-09-11T08:20:44.459 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1789880331 2024-09-11T08:20:44.459 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:44.459 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:44.459 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.459 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addrs": { 2024-09-11T08:20:44.459 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:20:44.460 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.460 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:20:44.460 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6810", 2024-09-11T08:20:44.460 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1789880331 2024-09-11T08:20:44.460 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:44.460 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:44.460 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.460 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addr": "172.21.15.181:6808/1789880331", 2024-09-11T08:20:44.460 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addr": "172.21.15.181:6809/1789880331", 2024-09-11T08:20:44.460 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addr": "172.21.15.181:6811/1789880331", 2024-09-11T08:20:44.460 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addr": "172.21.15.181:6810/1789880331", 2024-09-11T08:20:44.460 INFO:teuthology.orchestra.run.smithi150.stdout: "state": [ 2024-09-11T08:20:44.460 INFO:teuthology.orchestra.run.smithi150.stdout: "exists", 2024-09-11T08:20:44.461 INFO:teuthology.orchestra.run.smithi150.stdout: "up" 2024-09-11T08:20:44.461 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:44.461 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.461 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.461 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 7, 2024-09-11T08:20:44.461 INFO:teuthology.orchestra.run.smithi150.stdout: "uuid": "6ac1b06a-79bd-43ca-b91e-fcb0f4033a34", 2024-09-11T08:20:44.461 INFO:teuthology.orchestra.run.smithi150.stdout: "up": 1, 2024-09-11T08:20:44.461 INFO:teuthology.orchestra.run.smithi150.stdout: "in": 1, 2024-09-11T08:20:44.461 INFO:teuthology.orchestra.run.smithi150.stdout: "weight": 1, 2024-09-11T08:20:44.461 INFO:teuthology.orchestra.run.smithi150.stdout: "primary_affinity": 1, 2024-09-11T08:20:44.461 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_begin": 53, 2024-09-11T08:20:44.461 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_end": 139, 2024-09-11T08:20:44.461 INFO:teuthology.orchestra.run.smithi150.stdout: "up_from": 168, 2024-09-11T08:20:44.462 INFO:teuthology.orchestra.run.smithi150.stdout: "up_thru": 278, 2024-09-11T08:20:44.462 INFO:teuthology.orchestra.run.smithi150.stdout: "down_at": 140, 2024-09-11T08:20:44.462 INFO:teuthology.orchestra.run.smithi150.stdout: "lost_at": 0, 2024-09-11T08:20:44.462 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addrs": { 2024-09-11T08:20:44.462 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:20:44.462 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.462 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:20:44.462 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6812", 2024-09-11T08:20:44.462 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2706455857 2024-09-11T08:20:44.462 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:44.462 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:44.462 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.462 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addrs": { 2024-09-11T08:20:44.462 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:20:44.463 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.463 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:20:44.463 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6813", 2024-09-11T08:20:44.463 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2706455857 2024-09-11T08:20:44.463 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:44.463 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:44.463 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.463 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addrs": { 2024-09-11T08:20:44.463 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:20:44.463 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.463 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:20:44.463 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6815", 2024-09-11T08:20:44.464 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2706455857 2024-09-11T08:20:44.464 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:44.464 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:44.464 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.464 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addrs": { 2024-09-11T08:20:44.464 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:20:44.464 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.464 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:20:44.464 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6814", 2024-09-11T08:20:44.464 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2706455857 2024-09-11T08:20:44.464 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:44.464 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:44.465 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.465 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addr": "172.21.15.181:6812/2706455857", 2024-09-11T08:20:44.465 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addr": "172.21.15.181:6813/2706455857", 2024-09-11T08:20:44.465 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addr": "172.21.15.181:6815/2706455857", 2024-09-11T08:20:44.465 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addr": "172.21.15.181:6814/2706455857", 2024-09-11T08:20:44.465 INFO:teuthology.orchestra.run.smithi150.stdout: "state": [ 2024-09-11T08:20:44.465 INFO:teuthology.orchestra.run.smithi150.stdout: "exists", 2024-09-11T08:20:44.465 INFO:teuthology.orchestra.run.smithi150.stdout: "up" 2024-09-11T08:20:44.465 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:44.465 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:44.465 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-09-11T08:20:44.466 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_xinfo": [ 2024-09-11T08:20:44.466 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.466 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 0, 2024-09-11T08:20:44.466 INFO:teuthology.orchestra.run.smithi150.stdout: "down_stamp": "0.000000", 2024-09-11T08:20:44.466 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_probability": 0, 2024-09-11T08:20:44.466 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_interval": 0, 2024-09-11T08:20:44.466 INFO:teuthology.orchestra.run.smithi150.stdout: "features": 4540138320759226367, 2024-09-11T08:20:44.466 INFO:teuthology.orchestra.run.smithi150.stdout: "old_weight": 0, 2024-09-11T08:20:44.466 INFO:teuthology.orchestra.run.smithi150.stdout: "last_purged_snaps_scrub": "2024-09-11T08:20:06.930131+0000", 2024-09-11T08:20:44.466 INFO:teuthology.orchestra.run.smithi150.stdout: "dead_epoch": 0 2024-09-11T08:20:44.466 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.466 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.466 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 1, 2024-09-11T08:20:44.467 INFO:teuthology.orchestra.run.smithi150.stdout: "down_stamp": "0.000000", 2024-09-11T08:20:44.467 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_probability": 0, 2024-09-11T08:20:44.467 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_interval": 0, 2024-09-11T08:20:44.467 INFO:teuthology.orchestra.run.smithi150.stdout: "features": 4540138320759226367, 2024-09-11T08:20:44.467 INFO:teuthology.orchestra.run.smithi150.stdout: "old_weight": 0, 2024-09-11T08:20:44.467 INFO:teuthology.orchestra.run.smithi150.stdout: "last_purged_snaps_scrub": "2024-09-11T08:19:48.635292+0000", 2024-09-11T08:20:44.467 INFO:teuthology.orchestra.run.smithi150.stdout: "dead_epoch": 0 2024-09-11T08:20:44.467 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.467 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.467 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 2, 2024-09-11T08:20:44.467 INFO:teuthology.orchestra.run.smithi150.stdout: "down_stamp": "0.000000", 2024-09-11T08:20:44.467 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_probability": 0, 2024-09-11T08:20:44.468 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_interval": 0, 2024-09-11T08:20:44.468 INFO:teuthology.orchestra.run.smithi150.stdout: "features": 4540138320759226367, 2024-09-11T08:20:44.468 INFO:teuthology.orchestra.run.smithi150.stdout: "old_weight": 0, 2024-09-11T08:20:44.468 INFO:teuthology.orchestra.run.smithi150.stdout: "last_purged_snaps_scrub": "2024-09-11T08:20:29.633594+0000", 2024-09-11T08:20:44.468 INFO:teuthology.orchestra.run.smithi150.stdout: "dead_epoch": 0 2024-09-11T08:20:44.469 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.469 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.469 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 3, 2024-09-11T08:20:44.469 INFO:teuthology.orchestra.run.smithi150.stdout: "down_stamp": "0.000000", 2024-09-11T08:20:44.469 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_probability": 0, 2024-09-11T08:20:44.469 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_interval": 0, 2024-09-11T08:20:44.469 INFO:teuthology.orchestra.run.smithi150.stdout: "features": 4540138320759226367, 2024-09-11T08:20:44.470 INFO:teuthology.orchestra.run.smithi150.stdout: "old_weight": 0, 2024-09-11T08:20:44.470 INFO:teuthology.orchestra.run.smithi150.stdout: "last_purged_snaps_scrub": "2024-09-11T08:19:52.222318+0000", 2024-09-11T08:20:44.470 INFO:teuthology.orchestra.run.smithi150.stdout: "dead_epoch": 0 2024-09-11T08:20:44.470 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.470 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.470 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 4, 2024-09-11T08:20:44.470 INFO:teuthology.orchestra.run.smithi150.stdout: "down_stamp": "0.000000", 2024-09-11T08:20:44.470 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_probability": 0, 2024-09-11T08:20:44.470 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_interval": 0, 2024-09-11T08:20:44.470 INFO:teuthology.orchestra.run.smithi150.stdout: "features": 4540138320759226367, 2024-09-11T08:20:44.470 INFO:teuthology.orchestra.run.smithi150.stdout: "old_weight": 0, 2024-09-11T08:20:44.470 INFO:teuthology.orchestra.run.smithi150.stdout: "last_purged_snaps_scrub": "2024-09-11T08:20:39.570253+0000", 2024-09-11T08:20:44.470 INFO:teuthology.orchestra.run.smithi150.stdout: "dead_epoch": 0 2024-09-11T08:20:44.471 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.471 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.471 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 5, 2024-09-11T08:20:44.471 INFO:teuthology.orchestra.run.smithi150.stdout: "down_stamp": "0.000000", 2024-09-11T08:20:44.471 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_probability": 0, 2024-09-11T08:20:44.471 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_interval": 0, 2024-09-11T08:20:44.471 INFO:teuthology.orchestra.run.smithi150.stdout: "features": 4540138320759226367, 2024-09-11T08:20:44.471 INFO:teuthology.orchestra.run.smithi150.stdout: "old_weight": 0, 2024-09-11T08:20:44.471 INFO:teuthology.orchestra.run.smithi150.stdout: "last_purged_snaps_scrub": "2024-09-11T08:20:24.756605+0000", 2024-09-11T08:20:44.471 INFO:teuthology.orchestra.run.smithi150.stdout: "dead_epoch": 0 2024-09-11T08:20:44.471 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.471 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.471 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 6, 2024-09-11T08:20:44.472 INFO:teuthology.orchestra.run.smithi150.stdout: "down_stamp": "0.000000", 2024-09-11T08:20:44.472 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_probability": 0, 2024-09-11T08:20:44.472 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_interval": 0, 2024-09-11T08:20:44.472 INFO:teuthology.orchestra.run.smithi150.stdout: "features": 4540138320759226367, 2024-09-11T08:20:44.472 INFO:teuthology.orchestra.run.smithi150.stdout: "old_weight": 0, 2024-09-11T08:20:44.472 INFO:teuthology.orchestra.run.smithi150.stdout: "last_purged_snaps_scrub": "2024-09-11T08:20:37.768917+0000", 2024-09-11T08:20:44.472 INFO:teuthology.orchestra.run.smithi150.stdout: "dead_epoch": 0 2024-09-11T08:20:44.472 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.472 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.472 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 7, 2024-09-11T08:20:44.472 INFO:teuthology.orchestra.run.smithi150.stdout: "down_stamp": "0.000000", 2024-09-11T08:20:44.472 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_probability": 0, 2024-09-11T08:20:44.472 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_interval": 0, 2024-09-11T08:20:44.472 INFO:teuthology.orchestra.run.smithi150.stdout: "features": 4540138320759226367, 2024-09-11T08:20:44.473 INFO:teuthology.orchestra.run.smithi150.stdout: "old_weight": 0, 2024-09-11T08:20:44.473 INFO:teuthology.orchestra.run.smithi150.stdout: "last_purged_snaps_scrub": "2024-09-11T08:20:20.325086+0000", 2024-09-11T08:20:44.473 INFO:teuthology.orchestra.run.smithi150.stdout: "dead_epoch": 139 2024-09-11T08:20:44.473 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:44.473 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-09-11T08:20:44.473 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_upmap": [ 2024-09-11T08:20:44.473 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.473 INFO:teuthology.orchestra.run.smithi150.stdout: "pgid": "6.5", 2024-09-11T08:20:44.473 INFO:teuthology.orchestra.run.smithi150.stdout: "osds": [ 2024-09-11T08:20:44.473 INFO:teuthology.orchestra.run.smithi150.stdout: 2, 2024-09-11T08:20:44.473 INFO:teuthology.orchestra.run.smithi150.stdout: 4, 2024-09-11T08:20:44.473 INFO:teuthology.orchestra.run.smithi150.stdout: 7 2024-09-11T08:20:44.473 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:44.474 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:44.474 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-09-11T08:20:44.474 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_upmap_items": [ 2024-09-11T08:20:44.474 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.474 INFO:teuthology.orchestra.run.smithi150.stdout: "pgid": "4.4", 2024-09-11T08:20:44.474 INFO:teuthology.orchestra.run.smithi150.stdout: "mappings": [ 2024-09-11T08:20:44.474 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.474 INFO:teuthology.orchestra.run.smithi150.stdout: "from": 1, 2024-09-11T08:20:44.474 INFO:teuthology.orchestra.run.smithi150.stdout: "to": 0 2024-09-11T08:20:44.474 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:44.474 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:44.474 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.474 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.474 INFO:teuthology.orchestra.run.smithi150.stdout: "pgid": "4.14", 2024-09-11T08:20:44.475 INFO:teuthology.orchestra.run.smithi150.stdout: "mappings": [ 2024-09-11T08:20:44.475 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.475 INFO:teuthology.orchestra.run.smithi150.stdout: "from": 1, 2024-09-11T08:20:44.475 INFO:teuthology.orchestra.run.smithi150.stdout: "to": 2 2024-09-11T08:20:44.475 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:44.475 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:44.475 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.475 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.475 INFO:teuthology.orchestra.run.smithi150.stdout: "pgid": "4.1d", 2024-09-11T08:20:44.475 INFO:teuthology.orchestra.run.smithi150.stdout: "mappings": [ 2024-09-11T08:20:44.475 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.475 INFO:teuthology.orchestra.run.smithi150.stdout: "from": 1, 2024-09-11T08:20:44.475 INFO:teuthology.orchestra.run.smithi150.stdout: "to": 2 2024-09-11T08:20:44.476 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:44.476 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:44.476 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.476 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.476 INFO:teuthology.orchestra.run.smithi150.stdout: "pgid": "5.3", 2024-09-11T08:20:44.476 INFO:teuthology.orchestra.run.smithi150.stdout: "mappings": [ 2024-09-11T08:20:44.476 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.476 INFO:teuthology.orchestra.run.smithi150.stdout: "from": 0, 2024-09-11T08:20:44.476 INFO:teuthology.orchestra.run.smithi150.stdout: "to": 3 2024-09-11T08:20:44.476 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:44.476 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:44.476 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.477 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.477 INFO:teuthology.orchestra.run.smithi150.stdout: "pgid": "5.1e", 2024-09-11T08:20:44.477 INFO:teuthology.orchestra.run.smithi150.stdout: "mappings": [ 2024-09-11T08:20:44.477 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:20:44.477 INFO:teuthology.orchestra.run.smithi150.stdout: "from": 0, 2024-09-11T08:20:44.477 INFO:teuthology.orchestra.run.smithi150.stdout: "to": 6 2024-09-11T08:20:44.477 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:44.477 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:20:44.477 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:44.477 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-09-11T08:20:44.477 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_temp": [], 2024-09-11T08:20:44.477 INFO:teuthology.orchestra.run.smithi150.stdout: "primary_temp": [], 2024-09-11T08:20:44.477 INFO:teuthology.orchestra.run.smithi150.stdout: "blocklist": { 2024-09-11T08:20:44.477 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/2620931070": "2024-09-12T08:08:06.649692+0000", 2024-09-11T08:20:44.478 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/1073581409": "2024-09-12T08:02:45.986151+0000", 2024-09-11T08:20:44.478 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/2363649697": "2024-09-12T08:08:06.649692+0000", 2024-09-11T08:20:44.478 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:6800/603451071": "2024-09-12T08:02:45.986151+0000", 2024-09-11T08:20:44.478 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/1614413548": "2024-09-12T08:02:45.986151+0000", 2024-09-11T08:20:44.478 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/3673218166": "2024-09-12T08:03:08.740828+0000", 2024-09-11T08:20:44.478 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:6800/556498589": "2024-09-12T08:03:08.740828+0000", 2024-09-11T08:20:44.478 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/3666733994": "2024-09-12T08:03:08.740828+0000", 2024-09-11T08:20:44.478 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/3271479010": "2024-09-12T08:03:08.740828+0000", 2024-09-11T08:20:44.478 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/3185731090": "2024-09-12T08:08:06.649692+0000", 2024-09-11T08:20:44.478 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/2539412006": "2024-09-12T08:08:06.649692+0000", 2024-09-11T08:20:44.478 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/2740222408": "2024-09-12T08:02:45.986151+0000", 2024-09-11T08:20:44.478 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:6800/127856313": "2024-09-12T08:08:06.649692+0000", 2024-09-11T08:20:44.478 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/2375121603": "2024-09-12T08:08:06.649692+0000" 2024-09-11T08:20:44.478 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.478 INFO:teuthology.orchestra.run.smithi150.stdout: "range_blocklist": {}, 2024-09-11T08:20:44.479 INFO:teuthology.orchestra.run.smithi150.stdout: "erasure_code_profiles": { 2024-09-11T08:20:44.479 INFO:teuthology.orchestra.run.smithi150.stdout: "default": { 2024-09-11T08:20:44.479 INFO:teuthology.orchestra.run.smithi150.stdout: "crush-failure-domain": "osd", 2024-09-11T08:20:44.479 INFO:teuthology.orchestra.run.smithi150.stdout: "k": "2", 2024-09-11T08:20:44.479 INFO:teuthology.orchestra.run.smithi150.stdout: "m": "1", 2024-09-11T08:20:44.479 INFO:teuthology.orchestra.run.smithi150.stdout: "plugin": "jerasure", 2024-09-11T08:20:44.479 INFO:teuthology.orchestra.run.smithi150.stdout: "technique": "reed_sol_van" 2024-09-11T08:20:44.479 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:44.479 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:20:44.479 INFO:teuthology.orchestra.run.smithi150.stdout: "removed_snaps_queue": [], 2024-09-11T08:20:44.479 INFO:teuthology.orchestra.run.smithi150.stdout: "new_removed_snaps": [], 2024-09-11T08:20:44.479 INFO:teuthology.orchestra.run.smithi150.stdout: "new_purged_snaps": [], 2024-09-11T08:20:44.479 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_node_flags": {}, 2024-09-11T08:20:44.479 INFO:teuthology.orchestra.run.smithi150.stdout: "device_class_flags": {}, 2024-09-11T08:20:44.479 INFO:teuthology.orchestra.run.smithi150.stdout: "stretch_mode": { 2024-09-11T08:20:44.479 INFO:teuthology.orchestra.run.smithi150.stdout: "stretch_mode_enabled": false, 2024-09-11T08:20:44.480 INFO:teuthology.orchestra.run.smithi150.stdout: "stretch_bucket_count": 0, 2024-09-11T08:20:44.480 INFO:teuthology.orchestra.run.smithi150.stdout: "degraded_stretch_mode": 0, 2024-09-11T08:20:44.480 INFO:teuthology.orchestra.run.smithi150.stdout: "recovering_stretch_mode": 0, 2024-09-11T08:20:44.480 INFO:teuthology.orchestra.run.smithi150.stdout: "stretch_mode_bucket": 0 2024-09-11T08:20:44.480 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:20:44.480 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-11T08:20:44.480 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:44.513 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:44.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:44.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:44.615 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:20:44.657 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:44.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:44.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:20:44.759 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:44.769 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-11T08:20:44.797 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:44.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:44.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:44.899 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:44.922 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:44.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:44.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:45.024 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:45.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:45.074 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:45.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:45.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:44 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/95501954' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-11T08:20:45.175 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:45.193 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:20:45.222 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:45.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:45.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:45.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:44 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/95501954' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-11T08:20:45.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:44 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/95501954' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-11T08:20:45.325 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:20:45.365 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:45.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:45.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:20:45.467 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:45.475 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:20:45 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:20:45] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:20:45.511 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:45.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:45.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:45.613 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:45.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:45.649 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:45.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:45.686 INFO:tasks.thrashosds.thrasher:j is {'epoch': 293, 'fsid': 'c1bddff4-7013-11ef-bcea-c7b262605968', 'created': '2024-09-11T08:02:15.344618+0000', 'modified': '2024-09-11T08:20:40.780751+0000', 'last_up_change': '2024-09-11T08:14:23.745449+0000', 'last_in_change': '2024-09-11T08:19:28.121267+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-11T08:05:21.044729+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '255', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-11T08:07:28.786572+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 63, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-11T08:07:29.021286+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-11T08:07:30.334181+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-11T08:07:32.289626+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-11T08:07:34.351861+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 9, 'pool_name': 'unique_pool_2', 'create_time': '2024-09-11T08:17:59.761532+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '251', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': '37b63db9-ed2a-4701-b63d-03ff9885cc0e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 276, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6801', 'nonce': 334255279}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6802', 'nonce': 334255279}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6804', 'nonce': 334255279}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6803', 'nonce': 334255279}]}, 'public_addr': '172.21.15.150:6801/334255279', 'cluster_addr': '172.21.15.150:6802/334255279', 'heartbeat_back_addr': '172.21.15.150:6804/334255279', 'heartbeat_front_addr': '172.21.15.150:6803/334255279', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '80378442-02bc-48aa-b8ca-6113a4092079', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 270, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6805', 'nonce': 1129510619}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6806', 'nonce': 1129510619}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6808', 'nonce': 1129510619}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6807', 'nonce': 1129510619}]}, 'public_addr': '172.21.15.150:6805/1129510619', 'cluster_addr': '172.21.15.150:6806/1129510619', 'heartbeat_back_addr': '172.21.15.150:6808/1129510619', 'heartbeat_front_addr': '172.21.15.150:6807/1129510619', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '0955bb92-36a0-4e9a-98e1-b3a5af89f6d2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 284, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6809', 'nonce': 719975422}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6810', 'nonce': 719975422}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6812', 'nonce': 719975422}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6811', 'nonce': 719975422}]}, 'public_addr': '172.21.15.150:6809/719975422', 'cluster_addr': '172.21.15.150:6810/719975422', 'heartbeat_back_addr': '172.21.15.150:6812/719975422', 'heartbeat_front_addr': '172.21.15.150:6811/719975422', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '1d04d4d3-62e8-4460-8a07-77f7e5c62929', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.611114501953125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 260, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6813', 'nonce': 3101032500}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6814', 'nonce': 3101032500}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6816', 'nonce': 3101032500}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6815', 'nonce': 3101032500}]}, 'public_addr': '172.21.15.150:6813/3101032500', 'cluster_addr': '172.21.15.150:6814/3101032500', 'heartbeat_back_addr': '172.21.15.150:6816/3101032500', 'heartbeat_front_addr': '172.21.15.150:6815/3101032500', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'c5321a3c-d2f0-4440-8475-2c111d261139', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 270, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6800', 'nonce': 2567452825}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6801', 'nonce': 2567452825}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6803', 'nonce': 2567452825}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6802', 'nonce': 2567452825}]}, 'public_addr': '172.21.15.181:6800/2567452825', 'cluster_addr': '172.21.15.181:6801/2567452825', 'heartbeat_back_addr': '172.21.15.181:6803/2567452825', 'heartbeat_front_addr': '172.21.15.181:6802/2567452825', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '1262b2fe-f253-4a26-bfd4-c9dffb697c77', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 271, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6804', 'nonce': 2043435508}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6805', 'nonce': 2043435508}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6807', 'nonce': 2043435508}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6806', 'nonce': 2043435508}]}, 'public_addr': '172.21.15.181:6804/2043435508', 'cluster_addr': '172.21.15.181:6805/2043435508', 'heartbeat_back_addr': '172.21.15.181:6807/2043435508', 'heartbeat_front_addr': '172.21.15.181:6806/2043435508', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '7facca22-b3f1-4c20-833e-e3db798a08c4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 270, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6808', 'nonce': 1789880331}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6809', 'nonce': 1789880331}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6811', 'nonce': 1789880331}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6810', 'nonce': 1789880331}]}, 'public_addr': '172.21.15.181:6808/1789880331', 'cluster_addr': '172.21.15.181:6809/1789880331', 'heartbeat_back_addr': '172.21.15.181:6811/1789880331', 'heartbeat_front_addr': '172.21.15.181:6810/1789880331', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '6ac1b06a-79bd-43ca-b91e-fcb0f4033a34', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 53, 'last_clean_end': 139, 'up_from': 168, 'up_thru': 278, 'down_at': 140, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6812', 'nonce': 2706455857}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6813', 'nonce': 2706455857}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6815', 'nonce': 2706455857}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6814', 'nonce': 2706455857}]}, 'public_addr': '172.21.15.181:6812/2706455857', 'cluster_addr': '172.21.15.181:6813/2706455857', 'heartbeat_back_addr': '172.21.15.181:6815/2706455857', 'heartbeat_front_addr': '172.21.15.181:6814/2706455857', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T08:20:06.930131+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T08:19:48.635292+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T08:20:29.633594+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T08:19:52.222318+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T08:20:39.570253+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T08:20:24.756605+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T08:20:37.768917+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T08:20:20.325086+0000', 'dead_epoch': 139}], 'pg_upmap': [{'pgid': '6.5', 'osds': [2, 4, 7]}], 'pg_upmap_items': [{'pgid': '4.4', 'mappings': [{'from': 1, 'to': 0}]}, {'pgid': '4.14', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1d', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '5.3', 'mappings': [{'from': 0, 'to': 3}]}, {'pgid': '5.1e', 'mappings': [{'from': 0, 'to': 6}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.150:0/2620931070': '2024-09-12T08:08:06.649692+0000', '172.21.15.150:0/1073581409': '2024-09-12T08:02:45.986151+0000', '172.21.15.150:0/2363649697': '2024-09-12T08:08:06.649692+0000', '172.21.15.150:6800/603451071': '2024-09-12T08:02:45.986151+0000', '172.21.15.150:0/1614413548': '2024-09-12T08:02:45.986151+0000', '172.21.15.150:0/3673218166': '2024-09-12T08:03:08.740828+0000', '172.21.15.150:6800/556498589': '2024-09-12T08:03:08.740828+0000', '172.21.15.150:0/3666733994': '2024-09-12T08:03:08.740828+0000', '172.21.15.150:0/3271479010': '2024-09-12T08:03:08.740828+0000', '172.21.15.150:0/3185731090': '2024-09-12T08:08:06.649692+0000', '172.21.15.150:0/2539412006': '2024-09-12T08:08:06.649692+0000', '172.21.15.150:0/2740222408': '2024-09-12T08:02:45.986151+0000', '172.21.15.150:6800/127856313': '2024-09-12T08:08:06.649692+0000', '172.21.15.150:0/2375121603': '2024-09-12T08:08:06.649692+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-11T08:20:45.686 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 4.4 2024-09-11T08:20:45.686 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd rm-pg-upmap-items 4.4 2024-09-11T08:20:45.750 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:45.778 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:45.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:45.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:45.806 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:20:45.880 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:45.914 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:45.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:45.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:46.017 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:46.047 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:46.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:46.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:46.149 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:46.190 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:46.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:46.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:46.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:45 smithi150 ceph-mon[29276]: pgmap v602: 148 pgs: 148 active+clean; 2.8 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 824 B/s rd, 0 B/s wr, 860 op/s 2024-09-11T08:20:46.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:45 smithi150 ceph-mon[39612]: pgmap v602: 148 pgs: 148 active+clean; 2.8 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 824 B/s rd, 0 B/s wr, 860 op/s 2024-09-11T08:20:46.292 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:46.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:45 smithi181 ceph-mon[32701]: pgmap v602: 148 pgs: 148 active+clean; 2.8 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 824 B/s rd, 0 B/s wr, 860 op/s 2024-09-11T08:20:46.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:46.324 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:46.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:46.425 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:20:46.463 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:46.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:46.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:20:46.565 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:46.600 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:46.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:46.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:46.611 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T08:20:46.701 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:46.751 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:46.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:46.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:46.853 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:46.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:46.889 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:46.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:46.990 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:47.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:47.029 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:47.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:47.104 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-11T08:20:47.104 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:20:47.130 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:47.173 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:47.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:47.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:47.275 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:20:47.351 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:20:47.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:47.359 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:47.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:20:47.461 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:47.494 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:47.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:47.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:47.596 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:47.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:47.631 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:47.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:47.732 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:47.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:47.767 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:47.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:47.818 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-11T08:20:47.868 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:20:47.899 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:47.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:47.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:20:48.001 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:20:48.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:48.032 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:48.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:20:48.133 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:48.152 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:20:48.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:48.171 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:48.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:48.251 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:20:48.272 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:48.284 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:48 smithi150 ceph-mon[29276]: pgmap v603: 148 pgs: 148 active+clean; 2.8 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 667 op/s 2024-09-11T08:20:48.285 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:48 smithi150 ceph-mon[39612]: pgmap v603: 148 pgs: 148 active+clean; 2.8 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 667 op/s 2024-09-11T08:20:48.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:48.321 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:48.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:48.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:48 smithi181 ceph-mon[32701]: pgmap v603: 148 pgs: 148 active+clean; 2.8 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 667 op/s 2024-09-11T08:20:48.422 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:48.454 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:48.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:48.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:48.556 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:48.588 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:48.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:48.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:48.690 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:48.716 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:48.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:48.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:48.818 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:20:48.823 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:20:48 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:20:48.857 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:48.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:48.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:20:48.959 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:48.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:48.986 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:48.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:48.988 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:20:48 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:20:48] "GET /metrics HTTP/1.1" 200 224570 "" "Prometheus/2.43.0" 2024-09-11T08:20:49.042 INFO:teuthology.orchestra.run.smithi150.stderr:noscrub is unset 2024-09-11T08:20:49.087 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:49.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:49.119 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:49.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:49.220 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:20:49.268 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:49.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:49.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:20:49.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:49 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1353940595' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:20:49.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:49 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:20:49.370 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:20:49.412 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:49.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:49.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:20:49.440 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:49 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1353940595' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:20:49.440 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:49 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:20:49.440 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:49 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1353940595' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:20:49.441 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:49 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:20:49.514 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:49.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:49.564 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:49.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:49.665 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:20:49.687 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:49.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:49.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:20:49.792 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:49.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:49.835 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:49.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:49.936 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:49.970 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:49.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:49.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:50.073 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:20:50.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:50.113 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:50.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:20:50.214 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:50.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:50 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:20:50.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:50 smithi150 ceph-mon[39612]: pgmap v604: 148 pgs: 148 active+clean; 2.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.24k op/s 2024-09-11T08:20:50.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:50 smithi150 ceph-mon[39612]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:20:50.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:50 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:20:50.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:50 smithi150 ceph-mon[39612]: osdmap e294: 8 total, 8 up, 8 in 2024-09-11T08:20:50.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:50 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:20:50.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:50 smithi150 ceph-mon[39612]: 6.1e scrub starts 2024-09-11T08:20:50.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:50 smithi150 ceph-mon[39612]: 6.1e scrub ok 2024-09-11T08:20:50.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:50 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:20:50.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:50 smithi150 ceph-mon[29276]: pgmap v604: 148 pgs: 148 active+clean; 2.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.24k op/s 2024-09-11T08:20:50.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:50 smithi150 ceph-mon[29276]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:20:50.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:50 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:20:50.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:50 smithi150 ceph-mon[29276]: osdmap e294: 8 total, 8 up, 8 in 2024-09-11T08:20:50.242 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:50 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:20:50.242 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:50 smithi150 ceph-mon[29276]: 6.1e scrub starts 2024-09-11T08:20:50.242 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:50 smithi150 ceph-mon[29276]: 6.1e scrub ok 2024-09-11T08:20:50.256 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:50.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:50.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:50.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:50 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:20:50.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:50 smithi181 ceph-mon[32701]: pgmap v604: 148 pgs: 148 active+clean; 2.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.24k op/s 2024-09-11T08:20:50.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:50 smithi181 ceph-mon[32701]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:20:50.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:50 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:20:50.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:50 smithi181 ceph-mon[32701]: osdmap e294: 8 total, 8 up, 8 in 2024-09-11T08:20:50.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:50 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:20:50.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:50 smithi181 ceph-mon[32701]: 6.1e scrub starts 2024-09-11T08:20:50.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:50 smithi181 ceph-mon[32701]: 6.1e scrub ok 2024-09-11T08:20:50.358 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:50.395 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:50.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:50.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:50.497 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:50.529 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:50.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:50.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:50.631 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:50.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:50.664 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:50.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:50.765 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:50.787 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:50.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:50.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:50.888 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:50.926 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:50.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:50.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:51.028 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:20:51.047 INFO:teuthology.orchestra.run.smithi150.stderr:clear 4.4 pg_upmap_items mapping 2024-09-11T08:20:51.058 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:51.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:51.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:20:51.160 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:20:51.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:51.205 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:51.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:20:51.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:51 smithi150 ceph-mon[29276]: 3.18 scrub starts 2024-09-11T08:20:51.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:51 smithi150 ceph-mon[29276]: 3.18 scrub ok 2024-09-11T08:20:51.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:51 smithi150 ceph-mon[29276]: 9.3 scrub starts 2024-09-11T08:20:51.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:51 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2437125279' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.4"}]: dispatch 2024-09-11T08:20:51.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:51 smithi150 ceph-mon[29276]: 6.e scrub starts 2024-09-11T08:20:51.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:51 smithi150 ceph-mon[29276]: 6.e scrub ok 2024-09-11T08:20:51.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:51 smithi150 ceph-mon[39612]: 3.18 scrub starts 2024-09-11T08:20:51.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:51 smithi150 ceph-mon[39612]: 3.18 scrub ok 2024-09-11T08:20:51.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:51 smithi150 ceph-mon[39612]: 9.3 scrub starts 2024-09-11T08:20:51.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:51 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2437125279' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.4"}]: dispatch 2024-09-11T08:20:51.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:51 smithi150 ceph-mon[39612]: 6.e scrub starts 2024-09-11T08:20:51.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:51 smithi150 ceph-mon[39612]: 6.e scrub ok 2024-09-11T08:20:51.307 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:51.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:51 smithi181 ceph-mon[32701]: 3.18 scrub starts 2024-09-11T08:20:51.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:51 smithi181 ceph-mon[32701]: 3.18 scrub ok 2024-09-11T08:20:51.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:51 smithi181 ceph-mon[32701]: 9.3 scrub starts 2024-09-11T08:20:51.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:51 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2437125279' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.4"}]: dispatch 2024-09-11T08:20:51.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:51 smithi181 ceph-mon[32701]: 6.e scrub starts 2024-09-11T08:20:51.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:51 smithi181 ceph-mon[32701]: 6.e scrub ok 2024-09-11T08:20:51.340 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:51.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:51.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:51.442 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:51.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:51.483 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:51.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:51.584 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:51.614 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:51.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:51.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:51.716 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:51.749 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:51.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:51.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:51.851 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:20:51.891 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:51.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:51.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:20:51.993 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:20:52.022 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:52.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:52.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:20:52.124 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:52.157 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:52.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:52.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:52.236 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:52 smithi181 ceph-mon[32701]: pgmap v606: 148 pgs: 148 active+clean; 2.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s 2024-09-11T08:20:52.236 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:52 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2437125279' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.4"}]': finished 2024-09-11T08:20:52.236 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:52 smithi181 ceph-mon[32701]: osdmap e295: 8 total, 8 up, 8 in 2024-09-11T08:20:52.237 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:52 smithi181 ceph-mon[32701]: 5.9 scrub starts 2024-09-11T08:20:52.237 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:52 smithi181 ceph-mon[32701]: 5.9 scrub ok 2024-09-11T08:20:52.237 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:52 smithi181 ceph-mon[32701]: 9.3 scrub ok 2024-09-11T08:20:52.237 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:52 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:20:52.259 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:52.310 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:52.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:52.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:52.371 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:52 smithi150 ceph-mon[29276]: pgmap v606: 148 pgs: 148 active+clean; 2.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s 2024-09-11T08:20:52.371 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:52 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2437125279' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.4"}]': finished 2024-09-11T08:20:52.372 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:52 smithi150 ceph-mon[29276]: osdmap e295: 8 total, 8 up, 8 in 2024-09-11T08:20:52.372 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:52 smithi150 ceph-mon[29276]: 5.9 scrub starts 2024-09-11T08:20:52.372 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:52 smithi150 ceph-mon[29276]: 5.9 scrub ok 2024-09-11T08:20:52.372 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:52 smithi150 ceph-mon[29276]: 9.3 scrub ok 2024-09-11T08:20:52.372 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:52 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:20:52.373 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:52 smithi150 ceph-mon[39612]: pgmap v606: 148 pgs: 148 active+clean; 2.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s 2024-09-11T08:20:52.373 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:52 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2437125279' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.4"}]': finished 2024-09-11T08:20:52.373 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:52 smithi150 ceph-mon[39612]: osdmap e295: 8 total, 8 up, 8 in 2024-09-11T08:20:52.373 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:52 smithi150 ceph-mon[39612]: 5.9 scrub starts 2024-09-11T08:20:52.374 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:52 smithi150 ceph-mon[39612]: 5.9 scrub ok 2024-09-11T08:20:52.374 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:52 smithi150 ceph-mon[39612]: 9.3 scrub ok 2024-09-11T08:20:52.374 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:52 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:20:52.412 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:52.449 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:52.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:52.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:52.534 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T08:20:52.550 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-11T08:20:52.550 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:20:52.551 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:52.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:52.618 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:52.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:52.719 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:52.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:52.755 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:52.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:52.856 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:20:52.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:52.901 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:52.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:20:53.002 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:20:53.038 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:53.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:53.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:20:53.140 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:20:53.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:53.175 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:53.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:20:53.276 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:20:53.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:53.309 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:53.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:20:53.338 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:53 smithi150 ceph-mon[29276]: 5.2 scrub starts 2024-09-11T08:20:53.338 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:53 smithi150 ceph-mon[29276]: 5.2 scrub ok 2024-09-11T08:20:53.338 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:53 smithi150 ceph-mon[29276]: osdmap e296: 8 total, 8 up, 8 in 2024-09-11T08:20:53.338 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:53 smithi150 ceph-mon[39612]: 5.2 scrub starts 2024-09-11T08:20:53.339 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:53 smithi150 ceph-mon[39612]: 5.2 scrub ok 2024-09-11T08:20:53.339 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:53 smithi150 ceph-mon[39612]: osdmap e296: 8 total, 8 up, 8 in 2024-09-11T08:20:53.410 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:53.443 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:53.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:53.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:53.545 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:53.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:53 smithi181 ceph-mon[32701]: 5.2 scrub starts 2024-09-11T08:20:53.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:53 smithi181 ceph-mon[32701]: 5.2 scrub ok 2024-09-11T08:20:53.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:53 smithi181 ceph-mon[32701]: osdmap e296: 8 total, 8 up, 8 in 2024-09-11T08:20:53.579 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:53.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:53.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:53.681 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:20:53.711 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:53.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:53.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:20:53.862 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:20:53.905 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:53.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:53.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:20:54.006 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:20:54.047 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:54.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:54.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:20:54.149 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:20:54.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:54.194 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:54.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:20:54.218 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:54 smithi150 ceph-mon[29276]: 5.6 scrub starts 2024-09-11T08:20:54.219 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:54 smithi150 ceph-mon[29276]: 5.6 scrub ok 2024-09-11T08:20:54.219 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:54 smithi150 ceph-mon[29276]: 3.1e scrub starts 2024-09-11T08:20:54.219 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:54 smithi150 ceph-mon[29276]: 3.1e scrub ok 2024-09-11T08:20:54.219 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:54 smithi150 ceph-mon[29276]: pgmap v609: 148 pgs: 148 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.12k op/s 2024-09-11T08:20:54.219 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:54 smithi150 ceph-mon[29276]: 5.1a scrub starts 2024-09-11T08:20:54.219 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:54 smithi150 ceph-mon[29276]: 5.1a scrub ok 2024-09-11T08:20:54.219 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:54 smithi150 ceph-mon[39612]: 5.6 scrub starts 2024-09-11T08:20:54.219 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:54 smithi150 ceph-mon[39612]: 5.6 scrub ok 2024-09-11T08:20:54.220 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:54 smithi150 ceph-mon[39612]: 3.1e scrub starts 2024-09-11T08:20:54.220 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:54 smithi150 ceph-mon[39612]: 3.1e scrub ok 2024-09-11T08:20:54.220 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:54 smithi150 ceph-mon[39612]: pgmap v609: 148 pgs: 148 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.12k op/s 2024-09-11T08:20:54.220 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:54 smithi150 ceph-mon[39612]: 5.1a scrub starts 2024-09-11T08:20:54.220 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:54 smithi150 ceph-mon[39612]: 5.1a scrub ok 2024-09-11T08:20:54.296 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:20:54.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:54.334 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:54.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:20:54.392 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:20:54.393 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:20:54.405 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:54 smithi181 ceph-mon[32701]: 5.6 scrub starts 2024-09-11T08:20:54.405 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:54 smithi181 ceph-mon[32701]: 5.6 scrub ok 2024-09-11T08:20:54.405 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:54 smithi181 ceph-mon[32701]: 3.1e scrub starts 2024-09-11T08:20:54.405 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:54 smithi181 ceph-mon[32701]: 3.1e scrub ok 2024-09-11T08:20:54.405 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:54 smithi181 ceph-mon[32701]: pgmap v609: 148 pgs: 148 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.12k op/s 2024-09-11T08:20:54.405 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:54 smithi181 ceph-mon[32701]: 5.1a scrub starts 2024-09-11T08:20:54.405 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:54 smithi181 ceph-mon[32701]: 5.1a scrub ok 2024-09-11T08:20:54.423 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:20:54.435 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:54.442 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-11T08:20:54.443 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:20:54.443 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:20:54.480 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:20:54.481 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:20:54.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:54.509 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:54.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:54.523 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:20:54.524 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:20:54.550 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:20:54.550 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:20:54.580 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:20:54.580 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:20:54.609 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:20:54.609 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:20:54.611 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:54.649 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:20:54.650 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:20:54.666 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:54.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:54.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:54.768 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:20:54.802 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:54.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:54.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:20:54.904 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:54.944 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:54.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:54.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:55.045 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:55.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:55.079 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:55.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:55.180 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:55.199 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:20:55.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:55.239 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:55.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:55.334 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:55 smithi150 ceph-mon[29276]: 9.a scrub starts 2024-09-11T08:20:55.334 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:55 smithi150 ceph-mon[29276]: 4.4 scrub starts 2024-09-11T08:20:55.334 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:55 smithi150 ceph-mon[29276]: 4.4 scrub ok 2024-09-11T08:20:55.334 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:55 smithi150 ceph-mon[29276]: 3.1 scrub starts 2024-09-11T08:20:55.334 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:55 smithi150 ceph-mon[29276]: 3.1 scrub ok 2024-09-11T08:20:55.334 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:55 smithi150 ceph-mon[29276]: 5.b scrub starts 2024-09-11T08:20:55.335 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:55 smithi150 ceph-mon[29276]: 5.b scrub ok 2024-09-11T08:20:55.335 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:55 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1261032514' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:20:55.335 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:55 smithi150 ceph-mon[29276]: 9.a scrub ok 2024-09-11T08:20:55.335 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:55 smithi150 ceph-mon[29276]: 6.17 scrub starts 2024-09-11T08:20:55.335 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:55 smithi150 ceph-mon[29276]: 6.17 scrub ok 2024-09-11T08:20:55.335 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:55 smithi150 ceph-mon[39612]: 9.a scrub starts 2024-09-11T08:20:55.335 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:55 smithi150 ceph-mon[39612]: 4.4 scrub starts 2024-09-11T08:20:55.335 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:55 smithi150 ceph-mon[39612]: 4.4 scrub ok 2024-09-11T08:20:55.335 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:55 smithi150 ceph-mon[39612]: 3.1 scrub starts 2024-09-11T08:20:55.335 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:55 smithi150 ceph-mon[39612]: 3.1 scrub ok 2024-09-11T08:20:55.335 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:55 smithi150 ceph-mon[39612]: 5.b scrub starts 2024-09-11T08:20:55.336 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:55 smithi150 ceph-mon[39612]: 5.b scrub ok 2024-09-11T08:20:55.336 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:55 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1261032514' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:20:55.336 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:55 smithi150 ceph-mon[39612]: 9.a scrub ok 2024-09-11T08:20:55.336 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:55 smithi150 ceph-mon[39612]: 6.17 scrub starts 2024-09-11T08:20:55.336 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:55 smithi150 ceph-mon[39612]: 6.17 scrub ok 2024-09-11T08:20:55.341 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:55.374 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:55.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:55.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:55.476 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:55.517 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:55.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:55.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:55.559 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:55 smithi181 ceph-mon[32701]: 9.a scrub starts 2024-09-11T08:20:55.559 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:55 smithi181 ceph-mon[32701]: 4.4 scrub starts 2024-09-11T08:20:55.559 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:55 smithi181 ceph-mon[32701]: 4.4 scrub ok 2024-09-11T08:20:55.559 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:55 smithi181 ceph-mon[32701]: 3.1 scrub starts 2024-09-11T08:20:55.559 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:55 smithi181 ceph-mon[32701]: 3.1 scrub ok 2024-09-11T08:20:55.559 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:55 smithi181 ceph-mon[32701]: 5.b scrub starts 2024-09-11T08:20:55.560 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:55 smithi181 ceph-mon[32701]: 5.b scrub ok 2024-09-11T08:20:55.560 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:55 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1261032514' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:20:55.560 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:55 smithi181 ceph-mon[32701]: 9.a scrub ok 2024-09-11T08:20:55.560 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:55 smithi181 ceph-mon[32701]: 6.17 scrub starts 2024-09-11T08:20:55.560 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:55 smithi181 ceph-mon[32701]: 6.17 scrub ok 2024-09-11T08:20:55.560 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:20:55 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:20:55] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:20:55.619 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:55.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:55.651 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:55.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:55.752 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:20:55.783 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:55.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:55.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:20:55.885 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:55.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:55.918 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:55.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:56.019 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:56.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:56.072 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:56.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:56.173 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:56.189 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:56 smithi181 ceph-mon[32701]: 5.7 scrub starts 2024-09-11T08:20:56.189 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:56 smithi181 ceph-mon[32701]: 5.7 scrub ok 2024-09-11T08:20:56.189 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:56 smithi181 ceph-mon[32701]: 9.0 scrub starts 2024-09-11T08:20:56.189 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:56 smithi181 ceph-mon[32701]: 4.17 scrub starts 2024-09-11T08:20:56.190 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:56 smithi181 ceph-mon[32701]: 4.17 scrub ok 2024-09-11T08:20:56.190 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:56 smithi181 ceph-mon[32701]: 5.d scrub starts 2024-09-11T08:20:56.190 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:56 smithi181 ceph-mon[32701]: 5.d scrub ok 2024-09-11T08:20:56.190 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:56 smithi181 ceph-mon[32701]: pgmap v610: 148 pgs: 148 active+clean; 2.0 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 964 op/s; 59 B/s, 1 objects/s recovering 2024-09-11T08:20:56.190 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:56 smithi181 ceph-mon[32701]: 9.e scrub starts 2024-09-11T08:20:56.190 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:56 smithi181 ceph-mon[32701]: 9.0 scrub ok 2024-09-11T08:20:56.190 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:56 smithi181 ceph-mon[32701]: 4.1 scrub starts 2024-09-11T08:20:56.190 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:56 smithi181 ceph-mon[32701]: 4.1 scrub ok 2024-09-11T08:20:56.190 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:56 smithi181 ceph-mon[32701]: 9.e scrub ok 2024-09-11T08:20:56.210 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:56.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:56.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:56.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:56 smithi150 ceph-mon[29276]: 5.7 scrub starts 2024-09-11T08:20:56.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:56 smithi150 ceph-mon[29276]: 5.7 scrub ok 2024-09-11T08:20:56.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:56 smithi150 ceph-mon[29276]: 9.0 scrub starts 2024-09-11T08:20:56.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:56 smithi150 ceph-mon[29276]: 4.17 scrub starts 2024-09-11T08:20:56.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:56 smithi150 ceph-mon[29276]: 4.17 scrub ok 2024-09-11T08:20:56.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:56 smithi150 ceph-mon[29276]: 5.d scrub starts 2024-09-11T08:20:56.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:56 smithi150 ceph-mon[29276]: 5.d scrub ok 2024-09-11T08:20:56.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:56 smithi150 ceph-mon[29276]: pgmap v610: 148 pgs: 148 active+clean; 2.0 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 964 op/s; 59 B/s, 1 objects/s recovering 2024-09-11T08:20:56.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:56 smithi150 ceph-mon[29276]: 9.e scrub starts 2024-09-11T08:20:56.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:56 smithi150 ceph-mon[29276]: 9.0 scrub ok 2024-09-11T08:20:56.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:56 smithi150 ceph-mon[29276]: 4.1 scrub starts 2024-09-11T08:20:56.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:56 smithi150 ceph-mon[29276]: 4.1 scrub ok 2024-09-11T08:20:56.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:56 smithi150 ceph-mon[29276]: 9.e scrub ok 2024-09-11T08:20:56.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:56 smithi150 ceph-mon[39612]: 5.7 scrub starts 2024-09-11T08:20:56.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:56 smithi150 ceph-mon[39612]: 5.7 scrub ok 2024-09-11T08:20:56.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:56 smithi150 ceph-mon[39612]: 9.0 scrub starts 2024-09-11T08:20:56.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:56 smithi150 ceph-mon[39612]: 4.17 scrub starts 2024-09-11T08:20:56.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:56 smithi150 ceph-mon[39612]: 4.17 scrub ok 2024-09-11T08:20:56.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:56 smithi150 ceph-mon[39612]: 5.d scrub starts 2024-09-11T08:20:56.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:56 smithi150 ceph-mon[39612]: 5.d scrub ok 2024-09-11T08:20:56.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:56 smithi150 ceph-mon[39612]: pgmap v610: 148 pgs: 148 active+clean; 2.0 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 964 op/s; 59 B/s, 1 objects/s recovering 2024-09-11T08:20:56.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:56 smithi150 ceph-mon[39612]: 9.e scrub starts 2024-09-11T08:20:56.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:56 smithi150 ceph-mon[39612]: 9.0 scrub ok 2024-09-11T08:20:56.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:56 smithi150 ceph-mon[39612]: 4.1 scrub starts 2024-09-11T08:20:56.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:56 smithi150 ceph-mon[39612]: 4.1 scrub ok 2024-09-11T08:20:56.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:56 smithi150 ceph-mon[39612]: 9.e scrub ok 2024-09-11T08:20:56.312 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:56.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:56.344 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:56.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:56.445 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:20:56.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:56.475 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:56.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:20:56.577 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:56.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:56.610 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:56.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:56.711 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:56.758 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:56.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:56.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:56.860 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:56.891 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:56.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:56.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:56.993 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:57.027 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:57.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:57.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:57.128 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:20:57.168 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:57.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:57.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:20:57.206 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-11T08:20:57.206 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T08:20:57.206 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T08:20:57.274 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:20:57.331 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:57 smithi150 ceph-mon[29276]: 3.5 scrub starts 2024-09-11T08:20:57.331 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:57 smithi150 ceph-mon[29276]: 3.5 scrub ok 2024-09-11T08:20:57.331 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:57 smithi150 ceph-mon[29276]: 6.6 scrub starts 2024-09-11T08:20:57.331 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:57 smithi150 ceph-mon[29276]: 6.6 scrub ok 2024-09-11T08:20:57.332 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:57 smithi150 ceph-mon[39612]: 3.5 scrub starts 2024-09-11T08:20:57.332 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:57 smithi150 ceph-mon[39612]: 3.5 scrub ok 2024-09-11T08:20:57.332 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:57 smithi150 ceph-mon[39612]: 6.6 scrub starts 2024-09-11T08:20:57.333 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:57 smithi150 ceph-mon[39612]: 6.6 scrub ok 2024-09-11T08:20:57.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:57.338 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:57.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:20:57.439 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:20:57.504 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:57.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:57.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:20:57.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:57 smithi181 ceph-mon[32701]: 3.5 scrub starts 2024-09-11T08:20:57.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:57 smithi181 ceph-mon[32701]: 3.5 scrub ok 2024-09-11T08:20:57.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:57 smithi181 ceph-mon[32701]: 6.6 scrub starts 2024-09-11T08:20:57.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:57 smithi181 ceph-mon[32701]: 6.6 scrub ok 2024-09-11T08:20:57.606 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:57.643 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:57.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:57.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:57.745 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:20:57.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:57.787 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:57.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:20:57.888 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:20:57.920 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T08:20:57.922 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:57.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:57.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:20:58.024 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:20:58.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:58.061 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:58.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:20:58.139 INFO:teuthology.orchestra.run.smithi150.stderr:nodeep-scrub is unset 2024-09-11T08:20:58.162 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:58.191 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:58.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:58.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:58.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:58 smithi150 ceph-mon[29276]: 6.1 scrub starts 2024-09-11T08:20:58.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:58 smithi150 ceph-mon[29276]: 6.1 scrub ok 2024-09-11T08:20:58.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:58 smithi150 ceph-mon[29276]: pgmap v611: 148 pgs: 148 active+clean; 2.0 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 644 B/s rd, 0 B/s wr, 728 op/s; 45 B/s, 1 objects/s recovering 2024-09-11T08:20:58.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:58 smithi150 ceph-mon[29276]: 5.1 scrub starts 2024-09-11T08:20:58.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:58 smithi150 ceph-mon[29276]: 5.1 scrub ok 2024-09-11T08:20:58.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:58 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/912951236' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:20:58.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:58 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:20:58.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:58 smithi150 ceph-mon[39612]: 6.1 scrub starts 2024-09-11T08:20:58.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:58 smithi150 ceph-mon[39612]: 6.1 scrub ok 2024-09-11T08:20:58.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:58 smithi150 ceph-mon[39612]: pgmap v611: 148 pgs: 148 active+clean; 2.0 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 644 B/s rd, 0 B/s wr, 728 op/s; 45 B/s, 1 objects/s recovering 2024-09-11T08:20:58.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:58 smithi150 ceph-mon[39612]: 5.1 scrub starts 2024-09-11T08:20:58.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:58 smithi150 ceph-mon[39612]: 5.1 scrub ok 2024-09-11T08:20:58.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:58 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/912951236' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:20:58.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:58 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:20:58.293 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:20:58.325 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:58.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:58.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:20:58.427 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:58.464 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:58.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:58.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:58.492 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:58 smithi181 ceph-mon[32701]: 6.1 scrub starts 2024-09-11T08:20:58.492 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:58 smithi181 ceph-mon[32701]: 6.1 scrub ok 2024-09-11T08:20:58.492 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:58 smithi181 ceph-mon[32701]: pgmap v611: 148 pgs: 148 active+clean; 2.0 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 644 B/s rd, 0 B/s wr, 728 op/s; 45 B/s, 1 objects/s recovering 2024-09-11T08:20:58.492 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:58 smithi181 ceph-mon[32701]: 5.1 scrub starts 2024-09-11T08:20:58.492 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:58 smithi181 ceph-mon[32701]: 5.1 scrub ok 2024-09-11T08:20:58.493 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:58 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/912951236' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:20:58.493 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:58 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:20:58.566 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:20:58.598 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:58.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:58.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:20:58.700 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:20:58.733 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:58.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:58.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:20:58.823 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:20:58 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:20:58.835 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:20:58.871 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:58.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:58.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:20:58.973 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:20:58.988 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:20:58 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:20:58] "GET /metrics HTTP/1.1" 200 224546 "" "Prometheus/2.43.0" 2024-09-11T08:20:59.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:59.015 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:59.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:20:59.116 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:20:59.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:59.154 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:59.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:20:59.255 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:59.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:59.302 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:59.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:59.403 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:20:59.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:59.437 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:20:59.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:20:59.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:59 smithi150 ceph-mon[29276]: 2.1 scrub starts 2024-09-11T08:20:59.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:59 smithi150 ceph-mon[29276]: 2.1 scrub ok 2024-09-11T08:20:59.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:59 smithi150 ceph-mon[29276]: 9.d scrub starts 2024-09-11T08:20:59.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:59 smithi150 ceph-mon[29276]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:20:59.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:59 smithi150 ceph-mon[29276]: Cluster is now healthy 2024-09-11T08:20:59.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:59 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:20:59.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:59 smithi150 ceph-mon[29276]: osdmap e297: 8 total, 8 up, 8 in 2024-09-11T08:20:59.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:20:59 smithi150 ceph-mon[29276]: 9.d scrub ok 2024-09-11T08:20:59.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:59 smithi150 ceph-mon[39612]: 2.1 scrub starts 2024-09-11T08:20:59.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:59 smithi150 ceph-mon[39612]: 2.1 scrub ok 2024-09-11T08:20:59.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:59 smithi150 ceph-mon[39612]: 9.d scrub starts 2024-09-11T08:20:59.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:59 smithi150 ceph-mon[39612]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:20:59.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:59 smithi150 ceph-mon[39612]: Cluster is now healthy 2024-09-11T08:20:59.491 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:59 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:20:59.492 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:59 smithi150 ceph-mon[39612]: osdmap e297: 8 total, 8 up, 8 in 2024-09-11T08:20:59.492 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:20:59 smithi150 ceph-mon[39612]: 9.d scrub ok 2024-09-11T08:20:59.538 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:59.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:59 smithi181 ceph-mon[32701]: 2.1 scrub starts 2024-09-11T08:20:59.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:59 smithi181 ceph-mon[32701]: 2.1 scrub ok 2024-09-11T08:20:59.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:59 smithi181 ceph-mon[32701]: 9.d scrub starts 2024-09-11T08:20:59.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:59 smithi181 ceph-mon[32701]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:20:59.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:59 smithi181 ceph-mon[32701]: Cluster is now healthy 2024-09-11T08:20:59.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:59 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:20:59.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:59 smithi181 ceph-mon[32701]: osdmap e297: 8 total, 8 up, 8 in 2024-09-11T08:20:59.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:20:59 smithi181 ceph-mon[32701]: 9.d scrub ok 2024-09-11T08:20:59.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:59.577 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:59.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:59.678 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:59.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:59.700 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:59.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:20:59.801 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:20:59.844 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:59.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:59.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:20:59.945 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:20:59.978 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-11T08:20:59.979 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-11T08:20:59.979 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-09-11T08:20:59.979 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-09-11T08:20:59.979 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-11T08:20:59.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:20:59.982 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:20:59.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:00.083 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:00.116 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:00.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:00.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:00.218 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:00.245 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:21:00.246 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:00.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:00.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:00.348 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:00.381 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:00.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:00.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:00.475 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:00 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:21:00.476 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:00 smithi150 ceph-mon[39612]: pgmap v613: 148 pgs: 148 active+clean; 1.7 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.46k op/s; 45 B/s, 1 objects/s recovering 2024-09-11T08:21:00.477 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:00 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:21:00.477 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:00 smithi150 ceph-mon[29276]: pgmap v613: 148 pgs: 148 active+clean; 1.7 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.46k op/s; 45 B/s, 1 objects/s recovering 2024-09-11T08:21:00.483 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:00.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:00.512 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:00.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:00.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:00 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:21:00.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:00 smithi181 ceph-mon[32701]: pgmap v613: 148 pgs: 148 active+clean; 1.7 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.46k op/s; 45 B/s, 1 objects/s recovering 2024-09-11T08:21:00.611 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:21:00.613 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:00.666 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:00.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:00.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:00.689 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-11T08:21:00.768 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:00.831 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:00.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:00.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:00.933 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:00.967 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:00.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:00.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:01.069 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:01.095 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:01.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:01.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:01.197 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:01.225 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:01.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:01.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:01.250 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:21:01.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:01 smithi181 ceph-mon[32701]: 9.5 scrub starts 2024-09-11T08:21:01.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:01 smithi181 ceph-mon[32701]: 5.8 scrub starts 2024-09-11T08:21:01.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:01 smithi181 ceph-mon[32701]: 5.8 scrub ok 2024-09-11T08:21:01.327 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:01.350 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:01.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:01.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:01.452 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:01.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:01.474 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:01.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:01.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:01 smithi150 ceph-mon[39612]: 9.5 scrub starts 2024-09-11T08:21:01.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:01 smithi150 ceph-mon[39612]: 5.8 scrub starts 2024-09-11T08:21:01.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:01 smithi150 ceph-mon[39612]: 5.8 scrub ok 2024-09-11T08:21:01.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:01 smithi150 ceph-mon[29276]: 9.5 scrub starts 2024-09-11T08:21:01.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:01 smithi150 ceph-mon[29276]: 5.8 scrub starts 2024-09-11T08:21:01.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:01 smithi150 ceph-mon[29276]: 5.8 scrub ok 2024-09-11T08:21:01.575 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:21:01.596 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:01.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:01.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:21:01.698 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:01.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:01.737 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:01.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:01.838 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:21:01.869 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:01.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:01.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:21:01.971 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:21:02.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:02.006 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:02.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:21:02.107 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:02.140 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:02.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:02.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:02.230 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T08:21:02.244 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:21:02.313 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:02.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:02.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:21:02.415 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:02 smithi181 ceph-mon[32701]: 9.5 scrub ok 2024-09-11T08:21:02.415 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:02 smithi181 ceph-mon[32701]: 3.c scrub starts 2024-09-11T08:21:02.415 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:02 smithi181 ceph-mon[32701]: 3.c scrub ok 2024-09-11T08:21:02.415 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:02 smithi181 ceph-mon[32701]: 5.0 deep-scrub starts 2024-09-11T08:21:02.415 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:02 smithi181 ceph-mon[32701]: 5.0 deep-scrub ok 2024-09-11T08:21:02.415 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:02 smithi181 ceph-mon[32701]: pgmap v614: 148 pgs: 148 active+clean; 1.7 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.30k op/s; 40 B/s, 1 objects/s recovering 2024-09-11T08:21:02.416 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:02.453 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:02.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:02.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:02.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:02 smithi150 ceph-mon[29276]: 9.5 scrub ok 2024-09-11T08:21:02.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:02 smithi150 ceph-mon[29276]: 3.c scrub starts 2024-09-11T08:21:02.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:02 smithi150 ceph-mon[29276]: 3.c scrub ok 2024-09-11T08:21:02.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:02 smithi150 ceph-mon[29276]: 5.0 deep-scrub starts 2024-09-11T08:21:02.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:02 smithi150 ceph-mon[29276]: 5.0 deep-scrub ok 2024-09-11T08:21:02.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:02 smithi150 ceph-mon[29276]: pgmap v614: 148 pgs: 148 active+clean; 1.7 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.30k op/s; 40 B/s, 1 objects/s recovering 2024-09-11T08:21:02.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:02 smithi150 ceph-mon[39612]: 9.5 scrub ok 2024-09-11T08:21:02.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:02 smithi150 ceph-mon[39612]: 3.c scrub starts 2024-09-11T08:21:02.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:02 smithi150 ceph-mon[39612]: 3.c scrub ok 2024-09-11T08:21:02.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:02 smithi150 ceph-mon[39612]: 5.0 deep-scrub starts 2024-09-11T08:21:02.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:02 smithi150 ceph-mon[39612]: 5.0 deep-scrub ok 2024-09-11T08:21:02.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:02 smithi150 ceph-mon[39612]: pgmap v614: 148 pgs: 148 active+clean; 1.7 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.30k op/s; 40 B/s, 1 objects/s recovering 2024-09-11T08:21:02.555 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:02.599 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:02.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:02.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:02.701 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:02.731 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:02.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:02.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:02.833 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:21:02.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:02.892 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:02.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:21:02.993 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:03.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:03.029 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:03.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:03.130 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:03.165 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:03.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:03.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:03.267 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:03.320 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:03.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:03.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:03.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:03 smithi181 ceph-mon[32701]: 4.1c scrub starts 2024-09-11T08:21:03.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:03 smithi181 ceph-mon[32701]: 4.1c scrub ok 2024-09-11T08:21:03.326 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:03 smithi181 ceph-mon[32701]: 3.7 scrub starts 2024-09-11T08:21:03.326 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:03 smithi181 ceph-mon[32701]: 3.7 scrub ok 2024-09-11T08:21:03.424 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:21:03.458 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:03.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:03.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:21:03.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:03 smithi150 ceph-mon[29276]: 4.1c scrub starts 2024-09-11T08:21:03.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:03 smithi150 ceph-mon[29276]: 4.1c scrub ok 2024-09-11T08:21:03.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:03 smithi150 ceph-mon[29276]: 3.7 scrub starts 2024-09-11T08:21:03.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:03 smithi150 ceph-mon[29276]: 3.7 scrub ok 2024-09-11T08:21:03.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:03 smithi150 ceph-mon[39612]: 4.1c scrub starts 2024-09-11T08:21:03.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:03 smithi150 ceph-mon[39612]: 4.1c scrub ok 2024-09-11T08:21:03.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:03 smithi150 ceph-mon[39612]: 3.7 scrub starts 2024-09-11T08:21:03.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:03 smithi150 ceph-mon[39612]: 3.7 scrub ok 2024-09-11T08:21:03.560 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:21:03.592 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:03.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:03.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:21:03.694 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:03.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:03.736 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:03.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:03.837 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:03.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:03.895 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:03.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:03.996 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:21:04.025 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:04.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:04.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:21:04.127 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:04.165 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:04.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:04.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:04.266 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:04.301 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:04.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:04.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:04.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:04 smithi181 ceph-mon[32701]: 9.8 scrub starts 2024-09-11T08:21:04.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:04 smithi181 ceph-mon[32701]: 6.f scrub starts 2024-09-11T08:21:04.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:04 smithi181 ceph-mon[32701]: 6.f scrub ok 2024-09-11T08:21:04.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:04 smithi181 ceph-mon[32701]: pgmap v615: 148 pgs: 148 active+clean; 1.7 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.15k op/s; 35 B/s, 0 objects/s recovering 2024-09-11T08:21:04.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:04 smithi181 ceph-mon[32701]: 6.15 scrub starts 2024-09-11T08:21:04.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:04 smithi181 ceph-mon[32701]: 6.15 scrub ok 2024-09-11T08:21:04.403 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:04.433 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:04.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:04.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:04.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:04 smithi150 ceph-mon[29276]: 9.8 scrub starts 2024-09-11T08:21:04.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:04 smithi150 ceph-mon[29276]: 6.f scrub starts 2024-09-11T08:21:04.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:04 smithi150 ceph-mon[29276]: 6.f scrub ok 2024-09-11T08:21:04.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:04 smithi150 ceph-mon[29276]: pgmap v615: 148 pgs: 148 active+clean; 1.7 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.15k op/s; 35 B/s, 0 objects/s recovering 2024-09-11T08:21:04.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:04 smithi150 ceph-mon[29276]: 6.15 scrub starts 2024-09-11T08:21:04.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:04 smithi150 ceph-mon[29276]: 6.15 scrub ok 2024-09-11T08:21:04.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:04 smithi150 ceph-mon[39612]: 9.8 scrub starts 2024-09-11T08:21:04.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:04 smithi150 ceph-mon[39612]: 6.f scrub starts 2024-09-11T08:21:04.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:04 smithi150 ceph-mon[39612]: 6.f scrub ok 2024-09-11T08:21:04.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:04 smithi150 ceph-mon[39612]: pgmap v615: 148 pgs: 148 active+clean; 1.7 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.15k op/s; 35 B/s, 0 objects/s recovering 2024-09-11T08:21:04.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:04 smithi150 ceph-mon[39612]: 6.15 scrub starts 2024-09-11T08:21:04.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:04 smithi150 ceph-mon[39612]: 6.15 scrub ok 2024-09-11T08:21:04.536 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:04.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:04.571 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:04.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:04.672 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:21:04.710 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:04.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:04.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:21:04.755 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-11T08:21:04.755 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-09-11T08:21:04.755 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-09-11T08:21:04.755 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-09-11T08:21:04.755 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.756 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15087.0:146932 8.d 8:b808578b:::benchmark_data_smithi150_107646_object18100:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206)", 2024-09-11T08:21:04.756 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:15:55.523397+0000", 2024-09-11T08:21:04.756 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 309.22885135799999, 2024-09-11T08:21:04.756 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0733922520000001, 2024-09-11T08:21:04.756 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:21:04.756 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:21:04.756 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:21:04.756 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15087", 2024-09-11T08:21:04.756 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/3299560913", 2024-09-11T08:21:04.756 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 146932 2024-09-11T08:21:04.756 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.756 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:21:04.756 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.756 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:21:04.757 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.523397+0000", 2024-09-11T08:21:04.757 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:04.757 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.757 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.757 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:21:04.757 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.523397+0000", 2024-09-11T08:21:04.757 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:04.757 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.757 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.757 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:21:04.757 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.523395+0000", 2024-09-11T08:21:04.757 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:21:04.757 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.758 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.758 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:21:04.758 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.523401+0000", 2024-09-11T08:21:04.758 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6099999999999997e-06 2024-09-11T08:21:04.758 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.758 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.758 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:21:04.758 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.525812+0000", 2024-09-11T08:21:04.758 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0024110960000000002 2024-09-11T08:21:04.758 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.758 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.758 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:21:04.758 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.525823+0000", 2024-09-11T08:21:04.758 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1066999999999999e-05 2024-09-11T08:21:04.759 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.759 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.759 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:04.759 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.531490+0000", 2024-09-11T08:21:04.759 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.005666684 2024-09-11T08:21:04.759 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.759 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.759 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:21:04.759 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.531500+0000", 2024-09-11T08:21:04.759 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0397000000000001e-05 2024-09-11T08:21:04.759 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.759 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.760 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:04.760 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:56.515950+0000", 2024-09-11T08:21:04.760 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.98444924 2024-09-11T08:21:04.760 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.760 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.760 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:21:04.760 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:56.515961+0000", 2024-09-11T08:21:04.760 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1625e-05 2024-09-11T08:21:04.760 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.760 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.760 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:04.760 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.591039+0000", 2024-09-11T08:21:04.760 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.075078188 2024-09-11T08:21:04.761 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.761 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.761 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:21:04.761 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.591134+0000", 2024-09-11T08:21:04.761 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4813000000000002e-05 2024-09-11T08:21:04.761 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.761 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.761 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2,6,7", 2024-09-11T08:21:04.761 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.591210+0000", 2024-09-11T08:21:04.761 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5374e-05 2024-09-11T08:21:04.761 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.761 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.761 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:04.762 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.594054+0000", 2024-09-11T08:21:04.762 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0028446740000000002 2024-09-11T08:21:04.762 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.762 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.762 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:04.762 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.594089+0000", 2024-09-11T08:21:04.762 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5151000000000003e-05 2024-09-11T08:21:04.762 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.762 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.762 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:21:04.762 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.594094+0000", 2024-09-11T08:21:04.762 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9830000000000004e-06 2024-09-11T08:21:04.762 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.763 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.763 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:04.763 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.596667+0000", 2024-09-11T08:21:04.763 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0025724390000000001 2024-09-11T08:21:04.763 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.763 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.763 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:21:04.763 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.596712+0000", 2024-09-11T08:21:04.763 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5667000000000002e-05 2024-09-11T08:21:04.763 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.763 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.763 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:21:04.763 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.596789+0000", 2024-09-11T08:21:04.763 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6490999999999997e-05 2024-09-11T08:21:04.764 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:04.764 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:21:04.764 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:04.764 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.764 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.764 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15087.0:146933 8.d 8:bbaddf40:::benchmark_data_smithi150_107646_object18101:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206)", 2024-09-11T08:21:04.764 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:15:55.525845+0000", 2024-09-11T08:21:04.764 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 309.22640346899999, 2024-09-11T08:21:04.764 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0767671750000001, 2024-09-11T08:21:04.764 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:21:04.764 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:21:04.764 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:21:04.764 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15087", 2024-09-11T08:21:04.765 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/3299560913", 2024-09-11T08:21:04.765 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 146933 2024-09-11T08:21:04.765 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.765 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:21:04.765 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.765 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:21:04.765 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.525845+0000", 2024-09-11T08:21:04.765 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:04.765 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.765 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.765 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:21:04.765 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.525845+0000", 2024-09-11T08:21:04.765 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:04.765 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.766 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.766 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:21:04.766 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.525843+0000", 2024-09-11T08:21:04.766 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:21:04.766 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.766 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.766 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:21:04.766 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.525850+0000", 2024-09-11T08:21:04.766 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7399999999999998e-06 2024-09-11T08:21:04.766 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.766 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.766 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:21:04.766 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.525859+0000", 2024-09-11T08:21:04.767 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.9539999999999993e-06 2024-09-11T08:21:04.767 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.767 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.767 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:21:04.767 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.525864+0000", 2024-09-11T08:21:04.767 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6759999999999998e-06 2024-09-11T08:21:04.767 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.767 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.767 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:04.767 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.531522+0000", 2024-09-11T08:21:04.767 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0056579910000000002 2024-09-11T08:21:04.767 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.767 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.767 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:21:04.767 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.531527+0000", 2024-09-11T08:21:04.768 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8769999999999998e-06 2024-09-11T08:21:04.768 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.768 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.768 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:04.768 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:56.516026+0000", 2024-09-11T08:21:04.768 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.98449826699999998 2024-09-11T08:21:04.768 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.768 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.768 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:21:04.768 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:56.516033+0000", 2024-09-11T08:21:04.768 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6480000000000003e-06 2024-09-11T08:21:04.768 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.768 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.769 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:04.769 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.594125+0000", 2024-09-11T08:21:04.769 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0780913569999999 2024-09-11T08:21:04.769 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.769 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.769 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:21:04.769 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.594242+0000", 2024-09-11T08:21:04.770 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000117521 2024-09-11T08:21:04.770 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.770 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.770 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2,6,7", 2024-09-11T08:21:04.770 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.594340+0000", 2024-09-11T08:21:04.770 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7344999999999997e-05 2024-09-11T08:21:04.770 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.770 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.770 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:04.770 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.596848+0000", 2024-09-11T08:21:04.770 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0025083340000000001 2024-09-11T08:21:04.770 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.770 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.771 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:21:04.771 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.599713+0000", 2024-09-11T08:21:04.771 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0028655120000000002 2024-09-11T08:21:04.771 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.771 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.771 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:04.771 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.602373+0000", 2024-09-11T08:21:04.771 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0026598630000000002 2024-09-11T08:21:04.771 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.771 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.771 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:04.771 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.602441+0000", 2024-09-11T08:21:04.771 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7607999999999994e-05 2024-09-11T08:21:04.771 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.771 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.772 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:21:04.772 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.602499+0000", 2024-09-11T08:21:04.772 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8362000000000001e-05 2024-09-11T08:21:04.772 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.772 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.772 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:21:04.772 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.602612+0000", 2024-09-11T08:21:04.772 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011246199999999999 2024-09-11T08:21:04.772 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:04.772 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:21:04.772 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:04.772 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.772 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.773 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15087.0:146943 8.d 8:bb9cf1f9:::benchmark_data_smithi150_107646_object18111:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-11T08:21:04.773 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:15:55.551894+0000", 2024-09-11T08:21:04.773 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 309.20035389100002, 2024-09-11T08:21:04.773 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0510072859999999, 2024-09-11T08:21:04.773 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:21:04.773 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:21:04.773 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:21:04.773 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15087", 2024-09-11T08:21:04.773 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/3299560913", 2024-09-11T08:21:04.773 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 146943 2024-09-11T08:21:04.773 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.773 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:21:04.773 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.773 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:21:04.774 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.551894+0000", 2024-09-11T08:21:04.774 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:04.774 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.774 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.774 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:21:04.774 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.551894+0000", 2024-09-11T08:21:04.774 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:04.774 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.774 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.774 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:21:04.774 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.551893+0000", 2024-09-11T08:21:04.774 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:21:04.774 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.775 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.775 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:21:04.775 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.551899+0000", 2024-09-11T08:21:04.775 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6300000000000003e-06 2024-09-11T08:21:04.775 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.775 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.775 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:21:04.775 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.551909+0000", 2024-09-11T08:21:04.775 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0361e-05 2024-09-11T08:21:04.775 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.775 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.775 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:21:04.775 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.551914+0000", 2024-09-11T08:21:04.775 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8550000000000001e-06 2024-09-11T08:21:04.776 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.776 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.776 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:04.776 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.551956+0000", 2024-09-11T08:21:04.776 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1820000000000003e-05 2024-09-11T08:21:04.776 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.776 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.776 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:21:04.776 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.551963+0000", 2024-09-11T08:21:04.776 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9140000000000002e-06 2024-09-11T08:21:04.776 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.776 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.776 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:04.776 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:56.516053+0000", 2024-09-11T08:21:04.777 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.964090471 2024-09-11T08:21:04.777 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.777 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.777 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:21:04.777 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:56.516059+0000", 2024-09-11T08:21:04.777 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3840000000000002e-06 2024-09-11T08:21:04.777 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.777 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.777 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:04.777 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.596886+0000", 2024-09-11T08:21:04.777 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0808264320000001 2024-09-11T08:21:04.777 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.777 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.778 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:21:04.778 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.597015+0000", 2024-09-11T08:21:04.778 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012892599999999999 2024-09-11T08:21:04.778 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.778 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.778 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2,6,7", 2024-09-11T08:21:04.778 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.597138+0000", 2024-09-11T08:21:04.778 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012275799999999999 2024-09-11T08:21:04.778 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.778 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.778 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:21:04.778 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.602333+0000", 2024-09-11T08:21:04.778 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0051956429999999998 2024-09-11T08:21:04.778 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.779 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.779 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:04.779 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.602666+0000", 2024-09-11T08:21:04.779 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00033239400000000002 2024-09-11T08:21:04.779 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.779 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.779 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:04.779 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.602719+0000", 2024-09-11T08:21:04.779 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3785000000000001e-05 2024-09-11T08:21:04.779 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.779 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.779 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:04.779 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.602761+0000", 2024-09-11T08:21:04.779 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1439000000000001e-05 2024-09-11T08:21:04.780 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.780 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.780 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:21:04.780 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.602809+0000", 2024-09-11T08:21:04.780 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8325999999999998e-05 2024-09-11T08:21:04.780 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.780 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.780 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:21:04.780 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.602901+0000", 2024-09-11T08:21:04.780 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.2319999999999997e-05 2024-09-11T08:21:04.781 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:04.781 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:21:04.781 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:04.781 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.781 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.781 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15087.0:146953 8.d 8:b9a7d93a:::benchmark_data_smithi150_107646_object18121:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-11T08:21:04.781 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:15:55.562297+0000", 2024-09-11T08:21:04.781 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 309.18995081999998, 2024-09-11T08:21:04.781 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0486156900000001, 2024-09-11T08:21:04.781 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:21:04.781 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:21:04.781 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:21:04.781 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15087", 2024-09-11T08:21:04.782 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/3299560913", 2024-09-11T08:21:04.782 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 146953 2024-09-11T08:21:04.782 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.782 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:21:04.782 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.782 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:21:04.782 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.562297+0000", 2024-09-11T08:21:04.782 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:04.782 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.782 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.782 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:21:04.782 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.562297+0000", 2024-09-11T08:21:04.782 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:04.782 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.783 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.783 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:21:04.783 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.562296+0000", 2024-09-11T08:21:04.783 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:21:04.783 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.783 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.783 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:21:04.783 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.562301+0000", 2024-09-11T08:21:04.783 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3159999999999996e-06 2024-09-11T08:21:04.783 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.783 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.783 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:21:04.783 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.562311+0000", 2024-09-11T08:21:04.783 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.1600000000000004e-06 2024-09-11T08:21:04.783 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.784 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.784 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:21:04.784 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.562314+0000", 2024-09-11T08:21:04.785 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8510000000000001e-06 2024-09-11T08:21:04.785 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.785 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.785 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:04.785 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.562357+0000", 2024-09-11T08:21:04.785 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2141000000000002e-05 2024-09-11T08:21:04.785 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.785 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.785 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:21:04.786 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.562367+0000", 2024-09-11T08:21:04.786 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.022e-05 2024-09-11T08:21:04.786 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.786 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.786 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:04.786 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:56.516092+0000", 2024-09-11T08:21:04.786 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.953725033 2024-09-11T08:21:04.786 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.786 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.786 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:21:04.786 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:56.516099+0000", 2024-09-11T08:21:04.786 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4989999999999997e-06 2024-09-11T08:21:04.786 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.787 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.787 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:04.787 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.599740+0000", 2024-09-11T08:21:04.787 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0836410590000001 2024-09-11T08:21:04.787 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.787 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.787 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:21:04.787 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.599852+0000", 2024-09-11T08:21:04.787 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011151500000000001 2024-09-11T08:21:04.787 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.787 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.787 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2,6,7", 2024-09-11T08:21:04.787 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.599965+0000", 2024-09-11T08:21:04.787 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000112868 2024-09-11T08:21:04.788 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.788 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.788 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:21:04.788 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.602674+0000", 2024-09-11T08:21:04.788 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0027091099999999998 2024-09-11T08:21:04.788 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.788 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.788 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:04.788 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.602939+0000", 2024-09-11T08:21:04.788 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00026491699999999998 2024-09-11T08:21:04.788 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.789 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.789 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:04.789 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.602969+0000", 2024-09-11T08:21:04.789 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9901999999999999e-05 2024-09-11T08:21:04.789 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.789 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.789 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:04.789 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.610819+0000", 2024-09-11T08:21:04.789 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0078507319999999992 2024-09-11T08:21:04.789 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.789 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.789 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:21:04.789 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.610852+0000", 2024-09-11T08:21:04.790 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2098000000000003e-05 2024-09-11T08:21:04.790 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.790 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.790 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:21:04.790 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.610913+0000", 2024-09-11T08:21:04.790 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1307000000000001e-05 2024-09-11T08:21:04.790 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:04.790 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:21:04.790 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:04.790 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.790 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.790 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15087.0:146954 8.d 8:b72d99d7:::benchmark_data_smithi150_107646_object18122:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-11T08:21:04.790 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:15:55.562323+0000", 2024-09-11T08:21:04.790 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 309.18992512, 2024-09-11T08:21:04.791 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0487770150000002, 2024-09-11T08:21:04.791 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:21:04.791 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:21:04.791 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:21:04.791 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15087", 2024-09-11T08:21:04.791 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/3299560913", 2024-09-11T08:21:04.791 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 146954 2024-09-11T08:21:04.791 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.791 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:21:04.791 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.791 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:21:04.791 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.562323+0000", 2024-09-11T08:21:04.791 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:04.792 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.792 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.792 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:21:04.792 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.562323+0000", 2024-09-11T08:21:04.792 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:04.792 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.792 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.792 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:21:04.792 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.562323+0000", 2024-09-11T08:21:04.792 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-09-11T08:21:04.792 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.792 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.792 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:21:04.792 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.562325+0000", 2024-09-11T08:21:04.793 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5770000000000001e-06 2024-09-11T08:21:04.793 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.793 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.793 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:21:04.793 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.562335+0000", 2024-09-11T08:21:04.793 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7699999999999996e-06 2024-09-11T08:21:04.793 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.793 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.793 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:21:04.793 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.562338+0000", 2024-09-11T08:21:04.793 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.36e-06 2024-09-11T08:21:04.793 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.793 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.793 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:04.794 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.562388+0000", 2024-09-11T08:21:04.794 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.99e-05 2024-09-11T08:21:04.794 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.794 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.794 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:21:04.794 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.562394+0000", 2024-09-11T08:21:04.794 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7629999999999997e-06 2024-09-11T08:21:04.794 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.795 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.795 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:04.795 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:56.516120+0000", 2024-09-11T08:21:04.795 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.95372561300000003 2024-09-11T08:21:04.795 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.795 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.795 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:21:04.795 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:56.516126+0000", 2024-09-11T08:21:04.795 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6470000000000001e-06 2024-09-11T08:21:04.795 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.795 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.795 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:04.795 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.602988+0000", 2024-09-11T08:21:04.795 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.086861898 2024-09-11T08:21:04.796 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.796 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.796 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:21:04.796 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.603088+0000", 2024-09-11T08:21:04.796 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010015599999999999 2024-09-11T08:21:04.796 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.796 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.796 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2,6,7", 2024-09-11T08:21:04.796 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.603195+0000", 2024-09-11T08:21:04.796 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010692900000000001 2024-09-11T08:21:04.796 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.796 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.796 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:04.796 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.610953+0000", 2024-09-11T08:21:04.797 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0077574510000000003 2024-09-11T08:21:04.797 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.797 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.797 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:04.797 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.610979+0000", 2024-09-11T08:21:04.797 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6463000000000001e-05 2024-09-11T08:21:04.797 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.797 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.797 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:21:04.797 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.610985+0000", 2024-09-11T08:21:04.797 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6869999999999999e-06 2024-09-11T08:21:04.797 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.797 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.797 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:04.797 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.611014+0000", 2024-09-11T08:21:04.798 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9213999999999998e-05 2024-09-11T08:21:04.798 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.798 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.798 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:21:04.798 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.611043+0000", 2024-09-11T08:21:04.798 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9257000000000001e-05 2024-09-11T08:21:04.798 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.798 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.798 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:21:04.798 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.611100+0000", 2024-09-11T08:21:04.798 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6637e-05 2024-09-11T08:21:04.798 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:04.798 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:21:04.798 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:04.799 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.799 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.799 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15087.0:146964 8.d 8:bfe3bcbc:::benchmark_data_smithi150_107646_object18132:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-11T08:21:04.799 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:15:55.572456+0000", 2024-09-11T08:21:04.799 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 309.17979165899999, 2024-09-11T08:21:04.799 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.045324017, 2024-09-11T08:21:04.799 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:21:04.799 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:21:04.799 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:21:04.799 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15087", 2024-09-11T08:21:04.799 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/3299560913", 2024-09-11T08:21:04.799 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 146964 2024-09-11T08:21:04.799 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.799 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:21:04.800 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.800 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:21:04.800 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.572456+0000", 2024-09-11T08:21:04.800 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:04.800 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.800 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.800 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:21:04.800 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.572456+0000", 2024-09-11T08:21:04.800 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:04.800 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.800 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.800 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:21:04.800 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.572455+0000", 2024-09-11T08:21:04.800 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:21:04.801 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.801 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.801 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:21:04.801 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.572460+0000", 2024-09-11T08:21:04.801 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1789999999999999e-06 2024-09-11T08:21:04.801 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.801 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.801 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:21:04.801 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.572472+0000", 2024-09-11T08:21:04.801 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1291000000000001e-05 2024-09-11T08:21:04.801 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.801 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.801 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:21:04.801 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.572477+0000", 2024-09-11T08:21:04.802 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2789999999999998e-06 2024-09-11T08:21:04.802 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.802 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.802 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:04.802 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.572528+0000", 2024-09-11T08:21:04.802 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0782999999999998e-05 2024-09-11T08:21:04.802 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.802 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.802 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:21:04.802 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.572535+0000", 2024-09-11T08:21:04.802 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0870000000000001e-06 2024-09-11T08:21:04.802 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.802 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.803 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:04.803 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:56.516144+0000", 2024-09-11T08:21:04.803 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.94360924099999999 2024-09-11T08:21:04.803 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.803 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.803 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:21:04.803 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:56.516151+0000", 2024-09-11T08:21:04.803 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.392e-06 2024-09-11T08:21:04.803 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.803 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.803 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:04.803 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.608431+0000", 2024-09-11T08:21:04.803 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.092280602 2024-09-11T08:21:04.803 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.803 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.804 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:21:04.804 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.608554+0000", 2024-09-11T08:21:04.804 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012311900000000001 2024-09-11T08:21:04.804 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.804 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.804 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2,6,7", 2024-09-11T08:21:04.804 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.608639+0000", 2024-09-11T08:21:04.804 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4315999999999997e-05 2024-09-11T08:21:04.804 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.804 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.804 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:21:04.804 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.610989+0000", 2024-09-11T08:21:04.804 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0023499670000000001 2024-09-11T08:21:04.804 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.805 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.805 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:04.805 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.611143+0000", 2024-09-11T08:21:04.805 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015454800000000001 2024-09-11T08:21:04.805 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.805 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.805 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:04.805 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.617527+0000", 2024-09-11T08:21:04.805 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.006384308 2024-09-11T08:21:04.805 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.805 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.805 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:04.806 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.617678+0000", 2024-09-11T08:21:04.806 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000150731 2024-09-11T08:21:04.806 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.806 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.806 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:21:04.806 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.617708+0000", 2024-09-11T08:21:04.806 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9770000000000001e-05 2024-09-11T08:21:04.806 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.806 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.806 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:21:04.806 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.617780+0000", 2024-09-11T08:21:04.806 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2453999999999994e-05 2024-09-11T08:21:04.806 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:04.806 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:21:04.807 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:04.807 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.807 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.807 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15087.0:146995 8.d 8:b04cb85d:::benchmark_data_smithi150_107646_object18163:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-11T08:21:04.807 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:15:55.613312+0000", 2024-09-11T08:21:04.807 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 309.13893579199998, 2024-09-11T08:21:04.807 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0097442380000001, 2024-09-11T08:21:04.807 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:21:04.807 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:21:04.807 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:21:04.807 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15087", 2024-09-11T08:21:04.807 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/3299560913", 2024-09-11T08:21:04.807 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 146995 2024-09-11T08:21:04.807 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.807 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:21:04.808 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.808 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:21:04.808 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.613312+0000", 2024-09-11T08:21:04.808 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:04.808 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.808 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.808 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:21:04.808 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.613312+0000", 2024-09-11T08:21:04.808 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:04.808 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.808 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.808 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:21:04.808 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.613311+0000", 2024-09-11T08:21:04.808 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:21:04.808 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.809 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.809 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:21:04.809 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.613317+0000", 2024-09-11T08:21:04.809 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.874e-06 2024-09-11T08:21:04.809 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.809 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.809 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:21:04.809 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.613327+0000", 2024-09-11T08:21:04.809 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0071999999999999e-05 2024-09-11T08:21:04.809 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.809 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.809 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:21:04.809 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.613332+0000", 2024-09-11T08:21:04.809 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0730000000000004e-06 2024-09-11T08:21:04.809 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.810 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.810 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:04.810 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.613376+0000", 2024-09-11T08:21:04.810 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4152e-05 2024-09-11T08:21:04.810 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.810 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.810 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:21:04.810 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.613383+0000", 2024-09-11T08:21:04.810 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0299999999999996e-06 2024-09-11T08:21:04.810 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.810 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.810 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:04.810 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:56.516173+0000", 2024-09-11T08:21:04.810 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.90278959400000003 2024-09-11T08:21:04.810 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.811 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.811 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:21:04.811 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:56.516179+0000", 2024-09-11T08:21:04.811 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2779999999999999e-06 2024-09-11T08:21:04.811 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.811 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.811 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:04.811 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.611173+0000", 2024-09-11T08:21:04.811 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.094993423 2024-09-11T08:21:04.811 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.811 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.811 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:21:04.811 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.611241+0000", 2024-09-11T08:21:04.811 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8659999999999997e-05 2024-09-11T08:21:04.812 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.812 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.812 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2,6,7", 2024-09-11T08:21:04.812 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.611365+0000", 2024-09-11T08:21:04.812 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012359200000000001 2024-09-11T08:21:04.812 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.812 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.812 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:04.812 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.617594+0000", 2024-09-11T08:21:04.812 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0062289559999999999 2024-09-11T08:21:04.812 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.812 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.812 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:04.813 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.617645+0000", 2024-09-11T08:21:04.813 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1273000000000002e-05 2024-09-11T08:21:04.813 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.813 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.813 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:21:04.813 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.617650+0000", 2024-09-11T08:21:04.813 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0529999999999997e-06 2024-09-11T08:21:04.813 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.813 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.813 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:04.813 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.622960+0000", 2024-09-11T08:21:04.813 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.005309635 2024-09-11T08:21:04.813 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.814 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.814 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:21:04.814 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.622997+0000", 2024-09-11T08:21:04.814 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7512000000000003e-05 2024-09-11T08:21:04.814 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.814 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.814 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:21:04.814 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.623056+0000", 2024-09-11T08:21:04.814 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9227000000000001e-05 2024-09-11T08:21:04.814 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:04.814 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:21:04.814 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:04.814 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.814 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.815 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15087.0:147023 8.d 8:bfef95ee:::benchmark_data_smithi150_107646_object18191:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-11T08:21:04.815 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:15:55.674484+0000", 2024-09-11T08:21:04.815 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 309.07776375399999, 2024-09-11T08:21:04.815 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9487520229999999, 2024-09-11T08:21:04.815 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:21:04.815 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:21:04.815 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:21:04.815 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15087", 2024-09-11T08:21:04.815 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/3299560913", 2024-09-11T08:21:04.815 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 147023 2024-09-11T08:21:04.815 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.815 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:21:04.815 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.815 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:21:04.816 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.674484+0000", 2024-09-11T08:21:04.816 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:04.816 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.816 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.816 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:21:04.816 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.674484+0000", 2024-09-11T08:21:04.816 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:04.816 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.816 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.816 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:21:04.816 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.674483+0000", 2024-09-11T08:21:04.816 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:21:04.816 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.816 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.817 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:21:04.817 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.674489+0000", 2024-09-11T08:21:04.817 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.088e-06 2024-09-11T08:21:04.817 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.817 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.817 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:21:04.817 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.674501+0000", 2024-09-11T08:21:04.817 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1855000000000001e-05 2024-09-11T08:21:04.817 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.817 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.817 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:21:04.817 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.674507+0000", 2024-09-11T08:21:04.817 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.117e-06 2024-09-11T08:21:04.817 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.818 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.818 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:04.818 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.674555+0000", 2024-09-11T08:21:04.818 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8077000000000001e-05 2024-09-11T08:21:04.818 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.818 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.818 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:21:04.818 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:55.674561+0000", 2024-09-11T08:21:04.818 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.161e-06 2024-09-11T08:21:04.818 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.818 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.818 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:04.818 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:56.516197+0000", 2024-09-11T08:21:04.818 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.84163608400000001 2024-09-11T08:21:04.819 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.819 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.819 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:21:04.819 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:56.516204+0000", 2024-09-11T08:21:04.819 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5379999999999999e-06 2024-09-11T08:21:04.819 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.819 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.819 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:04.819 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.614887+0000", 2024-09-11T08:21:04.819 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0986829709999999 2024-09-11T08:21:04.819 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.819 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.819 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:21:04.820 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.615027+0000", 2024-09-11T08:21:04.820 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014060200000000001 2024-09-11T08:21:04.820 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.820 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.820 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2,6,7", 2024-09-11T08:21:04.820 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.615154+0000", 2024-09-11T08:21:04.820 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012669299999999999 2024-09-11T08:21:04.820 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.820 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.820 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:04.820 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.617825+0000", 2024-09-11T08:21:04.820 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002670894 2024-09-11T08:21:04.820 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.821 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.821 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:21:04.821 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.617833+0000", 2024-09-11T08:21:04.821 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5439999999999999e-06 2024-09-11T08:21:04.821 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.821 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.821 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:04.821 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.622923+0000", 2024-09-11T08:21:04.821 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0050899059999999999 2024-09-11T08:21:04.821 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.821 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.821 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:04.821 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.623114+0000", 2024-09-11T08:21:04.821 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000190278 2024-09-11T08:21:04.822 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.822 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.822 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:21:04.822 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.623162+0000", 2024-09-11T08:21:04.822 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8381999999999999e-05 2024-09-11T08:21:04.822 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.822 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.822 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:21:04.822 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:15:57.623236+0000", 2024-09-11T08:21:04.822 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4342000000000001e-05 2024-09-11T08:21:04.822 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:04.822 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:21:04.822 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:04.822 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.823 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.823 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:27113 9.7 9:ee94abee:::benchmark_data_smithi150_133616_object27112:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e252)", 2024-09-11T08:21:04.823 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:18:21.927675+0000", 2024-09-11T08:21:04.823 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 162.82457347900001, 2024-09-11T08:21:04.823 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.15996824200000001, 2024-09-11T08:21:04.823 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:21:04.823 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:21:04.823 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:21:04.823 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:21:04.823 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:21:04.823 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 27113 2024-09-11T08:21:04.823 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.823 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:21:04.824 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.824 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:21:04.824 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:21.927675+0000", 2024-09-11T08:21:04.824 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:04.824 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.824 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.824 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:21:04.824 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:21.927675+0000", 2024-09-11T08:21:04.824 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:04.824 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.824 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.824 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:21:04.824 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:21.927673+0000", 2024-09-11T08:21:04.825 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:21:04.825 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.825 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.825 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:21:04.825 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:21.927714+0000", 2024-09-11T08:21:04.825 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0843000000000002e-05 2024-09-11T08:21:04.825 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.825 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.825 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:21:04.825 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:21.927729+0000", 2024-09-11T08:21:04.825 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5642999999999999e-05 2024-09-11T08:21:04.825 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.825 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.825 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:21:04.825 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:21.927735+0000", 2024-09-11T08:21:04.826 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8050000000000003e-06 2024-09-11T08:21:04.826 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.826 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.826 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:04.826 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:21.927779+0000", 2024-09-11T08:21:04.826 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3378999999999999e-05 2024-09-11T08:21:04.826 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.826 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.826 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:21:04.826 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:21.927902+0000", 2024-09-11T08:21:04.826 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012371300000000001 2024-09-11T08:21:04.826 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.826 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.826 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3,7", 2024-09-11T08:21:04.826 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:21.928045+0000", 2024-09-11T08:21:04.827 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014262000000000001 2024-09-11T08:21:04.827 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.827 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.827 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:21:04.827 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:21.929147+0000", 2024-09-11T08:21:04.827 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0011016019999999999 2024-09-11T08:21:04.827 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.827 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.827 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:04.827 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:21.929308+0000", 2024-09-11T08:21:04.827 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016125499999999999 2024-09-11T08:21:04.827 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.827 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.828 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:04.828 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:22.087548+0000", 2024-09-11T08:21:04.828 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.15823992000000001 2024-09-11T08:21:04.828 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.828 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.828 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:21:04.828 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:22.087597+0000", 2024-09-11T08:21:04.828 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9653e-05 2024-09-11T08:21:04.828 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.828 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.828 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:21:04.828 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:22.087643+0000", 2024-09-11T08:21:04.828 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5377000000000003e-05 2024-09-11T08:21:04.828 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:04.829 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:21:04.829 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:04.829 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.829 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.829 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:30523 9.a 9:5d6dc0d6:::benchmark_data_smithi150_133616_object30522:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e252)", 2024-09-11T08:21:04.829 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:18:23.983743+0000", 2024-09-11T08:21:04.829 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 160.76850497000001, 2024-09-11T08:21:04.829 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.16520641799999999, 2024-09-11T08:21:04.829 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:21:04.829 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:21:04.829 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:21:04.829 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:21:04.829 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:21:04.830 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 30523 2024-09-11T08:21:04.830 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.830 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:21:04.830 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.830 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:21:04.830 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.983743+0000", 2024-09-11T08:21:04.830 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:04.830 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.830 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.830 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:21:04.830 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.983743+0000", 2024-09-11T08:21:04.830 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:04.830 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.830 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.830 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:21:04.831 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.983742+0000", 2024-09-11T08:21:04.831 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:21:04.831 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.831 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.831 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:21:04.831 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.983804+0000", 2024-09-11T08:21:04.831 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1736000000000007e-05 2024-09-11T08:21:04.831 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.831 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.831 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:21:04.831 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.983812+0000", 2024-09-11T08:21:04.831 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7389999999999999e-06 2024-09-11T08:21:04.831 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.831 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.832 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:21:04.832 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.983815+0000", 2024-09-11T08:21:04.832 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.331e-06 2024-09-11T08:21:04.832 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.832 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.832 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:04.832 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.983838+0000", 2024-09-11T08:21:04.832 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3122999999999999e-05 2024-09-11T08:21:04.832 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.832 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.832 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:21:04.832 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.983915+0000", 2024-09-11T08:21:04.832 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7330000000000007e-05 2024-09-11T08:21:04.833 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.833 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.833 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,7", 2024-09-11T08:21:04.833 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.983996+0000", 2024-09-11T08:21:04.833 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0784000000000004e-05 2024-09-11T08:21:04.833 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.833 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.833 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:04.833 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.985079+0000", 2024-09-11T08:21:04.833 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0010832540000000001 2024-09-11T08:21:04.833 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.833 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.833 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:21:04.833 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.985094+0000", 2024-09-11T08:21:04.834 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4626e-05 2024-09-11T08:21:04.834 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.834 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.834 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:04.834 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:24.148871+0000", 2024-09-11T08:21:04.834 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.16377673000000001 2024-09-11T08:21:04.834 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.834 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.834 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:21:04.834 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:24.148909+0000", 2024-09-11T08:21:04.834 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8002e-05 2024-09-11T08:21:04.834 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.834 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.834 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:21:04.835 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:24.148949+0000", 2024-09-11T08:21:04.835 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.07e-05 2024-09-11T08:21:04.835 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:04.835 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:21:04.835 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:04.835 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.835 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.835 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:30526 9.6 9:698820d5:::benchmark_data_smithi150_133616_object30525:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e252)", 2024-09-11T08:21:04.835 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:18:23.986749+0000", 2024-09-11T08:21:04.835 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 160.76549878700001, 2024-09-11T08:21:04.835 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.17424029699999999, 2024-09-11T08:21:04.835 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:21:04.835 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:21:04.835 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:21:04.836 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:21:04.836 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:21:04.836 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 30526 2024-09-11T08:21:04.836 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.836 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:21:04.836 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.836 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:21:04.836 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.986749+0000", 2024-09-11T08:21:04.836 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:04.836 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.836 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.836 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:21:04.836 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.986749+0000", 2024-09-11T08:21:04.836 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:04.837 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.837 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.837 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:21:04.837 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.986745+0000", 2024-09-11T08:21:04.837 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999962 2024-09-11T08:21:04.837 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.837 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.837 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:21:04.837 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.986803+0000", 2024-09-11T08:21:04.837 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7525999999999998e-05 2024-09-11T08:21:04.837 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.837 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.837 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:21:04.837 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.986820+0000", 2024-09-11T08:21:04.838 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7201999999999999e-05 2024-09-11T08:21:04.838 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.838 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.838 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:21:04.838 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.986826+0000", 2024-09-11T08:21:04.838 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3779999999999998e-06 2024-09-11T08:21:04.838 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.838 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.838 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:04.838 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.986878+0000", 2024-09-11T08:21:04.838 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1669999999999998e-05 2024-09-11T08:21:04.838 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.838 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.838 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:21:04.839 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.986979+0000", 2024-09-11T08:21:04.839 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000101086 2024-09-11T08:21:04.839 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.839 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.839 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,2", 2024-09-11T08:21:04.839 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.987116+0000", 2024-09-11T08:21:04.839 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000137274 2024-09-11T08:21:04.839 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.839 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.839 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:21:04.839 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.989086+0000", 2024-09-11T08:21:04.839 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0019691169999999998 2024-09-11T08:21:04.839 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.839 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.839 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:04.840 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.989227+0000", 2024-09-11T08:21:04.840 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014167499999999999 2024-09-11T08:21:04.840 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.840 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.840 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:21:04.842 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:04.842 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:24.160896+0000", 2024-09-11T08:21:04.842 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.171668971 2024-09-11T08:21:04.842 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.842 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.842 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:21:04.842 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:24.160952+0000", 2024-09-11T08:21:04.842 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6175999999999999e-05 2024-09-11T08:21:04.842 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.843 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.843 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:21:04.843 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:24.160990+0000", 2024-09-11T08:21:04.843 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7113999999999997e-05 2024-09-11T08:21:04.843 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:04.843 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:21:04.843 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:04.843 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.843 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.843 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:30527 9.a 9:5115829a:::benchmark_data_smithi150_133616_object30526:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e252)", 2024-09-11T08:21:04.843 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:18:23.986848+0000", 2024-09-11T08:21:04.843 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 160.76540029500001, 2024-09-11T08:21:04.843 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.16484121900000001, 2024-09-11T08:21:04.844 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:21:04.844 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:21:04.844 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:21:04.844 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:21:04.844 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:21:04.844 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 30527 2024-09-11T08:21:04.844 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.844 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:21:04.844 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.844 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:21:04.844 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.986848+0000", 2024-09-11T08:21:04.844 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:04.844 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.845 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.845 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:21:04.845 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.986848+0000", 2024-09-11T08:21:04.845 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:04.845 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.845 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.845 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:21:04.845 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.986847+0000", 2024-09-11T08:21:04.845 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-09-11T08:21:04.845 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.845 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.845 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:21:04.845 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.986912+0000", 2024-09-11T08:21:04.845 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5154999999999995e-05 2024-09-11T08:21:04.846 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.846 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.846 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:21:04.846 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.986922+0000", 2024-09-11T08:21:04.846 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7019999999999996e-06 2024-09-11T08:21:04.846 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.846 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.846 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:21:04.846 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.986927+0000", 2024-09-11T08:21:04.846 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9599999999999999e-06 2024-09-11T08:21:04.846 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.846 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.846 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:04.846 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.987027+0000", 2024-09-11T08:21:04.847 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000100127 2024-09-11T08:21:04.847 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.847 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.847 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:21:04.847 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.987151+0000", 2024-09-11T08:21:04.847 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012369799999999999 2024-09-11T08:21:04.847 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.847 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.847 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,7", 2024-09-11T08:21:04.847 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.987288+0000", 2024-09-11T08:21:04.847 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013676699999999999 2024-09-11T08:21:04.847 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.847 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.847 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:21:04.848 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.989137+0000", 2024-09-11T08:21:04.848 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0018497380000000001 2024-09-11T08:21:04.848 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.848 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.848 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:04.848 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.989645+0000", 2024-09-11T08:21:04.848 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00050749200000000001 2024-09-11T08:21:04.848 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.848 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.848 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:04.848 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:24.151633+0000", 2024-09-11T08:21:04.848 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.16198802300000001 2024-09-11T08:21:04.848 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.848 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.849 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:21:04.849 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:24.151666+0000", 2024-09-11T08:21:04.849 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3191000000000001e-05 2024-09-11T08:21:04.849 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.849 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.849 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:21:04.849 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:24.151689+0000", 2024-09-11T08:21:04.849 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2931000000000001e-05 2024-09-11T08:21:04.849 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:04.849 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:21:04.849 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:04.849 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.849 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.849 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:30533 9.6 9:62416b4c:::benchmark_data_smithi150_133616_object30532:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e252)", 2024-09-11T08:21:04.850 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:18:23.989271+0000", 2024-09-11T08:21:04.850 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 160.76297668500001, 2024-09-11T08:21:04.850 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.17184012300000001, 2024-09-11T08:21:04.850 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:21:04.850 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:21:04.850 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:21:04.850 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:21:04.850 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:21:04.850 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 30533 2024-09-11T08:21:04.850 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.850 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:21:04.850 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.850 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:21:04.850 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.989271+0000", 2024-09-11T08:21:04.851 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:04.851 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.851 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.851 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:21:04.851 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.989271+0000", 2024-09-11T08:21:04.851 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:04.851 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.851 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.851 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:21:04.851 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.989270+0000", 2024-09-11T08:21:04.851 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:21:04.851 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.851 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.851 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:21:04.852 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.989294+0000", 2024-09-11T08:21:04.852 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4000999999999999e-05 2024-09-11T08:21:04.852 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.852 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.852 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:21:04.852 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.989303+0000", 2024-09-11T08:21:04.852 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.1500000000000005e-06 2024-09-11T08:21:04.852 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.852 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.852 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:21:04.852 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.989316+0000", 2024-09-11T08:21:04.852 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2507000000000001e-05 2024-09-11T08:21:04.852 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.852 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.853 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:04.853 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.989349+0000", 2024-09-11T08:21:04.853 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2941999999999997e-05 2024-09-11T08:21:04.853 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.853 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.853 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:21:04.853 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.989419+0000", 2024-09-11T08:21:04.853 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0797000000000007e-05 2024-09-11T08:21:04.853 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.853 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.853 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,2", 2024-09-11T08:21:04.853 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.989518+0000", 2024-09-11T08:21:04.853 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.8867000000000001e-05 2024-09-11T08:21:04.854 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.854 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.854 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:21:04.854 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.990675+0000", 2024-09-11T08:21:04.854 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001156642 2024-09-11T08:21:04.854 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.854 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.854 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:04.854 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.991357+0000", 2024-09-11T08:21:04.854 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00068165100000000002 2024-09-11T08:21:04.854 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.854 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.854 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:04.854 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:24.161034+0000", 2024-09-11T08:21:04.855 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.169677775 2024-09-11T08:21:04.855 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.855 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.855 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:21:04.855 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:24.161077+0000", 2024-09-11T08:21:04.855 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2407999999999998e-05 2024-09-11T08:21:04.855 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.855 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.855 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:21:04.855 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:24.161111+0000", 2024-09-11T08:21:04.855 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4672000000000003e-05 2024-09-11T08:21:04.855 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:04.855 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:21:04.856 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:04.856 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.856 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.856 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:30536 9.a 9:55453339:::benchmark_data_smithi150_133616_object30535:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e252)", 2024-09-11T08:21:04.856 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:18:23.992420+0000", 2024-09-11T08:21:04.856 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 160.75982836599999, 2024-09-11T08:21:04.856 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.15934726599999999, 2024-09-11T08:21:04.856 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:21:04.856 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:21:04.856 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:21:04.856 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:21:04.856 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:21:04.856 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 30536 2024-09-11T08:21:04.856 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.857 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:21:04.857 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.857 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:21:04.857 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.992420+0000", 2024-09-11T08:21:04.857 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:04.857 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.857 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.857 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:21:04.857 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.992420+0000", 2024-09-11T08:21:04.857 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:04.857 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.857 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.857 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:21:04.857 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.992419+0000", 2024-09-11T08:21:04.858 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:21:04.858 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.858 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.858 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:21:04.858 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.992474+0000", 2024-09-11T08:21:04.858 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4676999999999998e-05 2024-09-11T08:21:04.858 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.858 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.858 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:21:04.858 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.992483+0000", 2024-09-11T08:21:04.858 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4360000000000008e-06 2024-09-11T08:21:04.858 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.858 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.858 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:21:04.858 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.992487+0000", 2024-09-11T08:21:04.859 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2039999999999999e-06 2024-09-11T08:21:04.859 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.859 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.859 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:04.859 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.992515+0000", 2024-09-11T08:21:04.859 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8314e-05 2024-09-11T08:21:04.859 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.859 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.859 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:21:04.859 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.992598+0000", 2024-09-11T08:21:04.859 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2783999999999999e-05 2024-09-11T08:21:04.859 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.859 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.859 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,7", 2024-09-11T08:21:04.860 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.992680+0000", 2024-09-11T08:21:04.860 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1605000000000002e-05 2024-09-11T08:21:04.860 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.860 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.860 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:21:04.860 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.993618+0000", 2024-09-11T08:21:04.860 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00093792099999999998 2024-09-11T08:21:04.860 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.860 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.860 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:04.860 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.993853+0000", 2024-09-11T08:21:04.860 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00023529599999999999 2024-09-11T08:21:04.860 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.860 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.861 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:04.861 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:24.151718+0000", 2024-09-11T08:21:04.861 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.157864743 2024-09-11T08:21:04.861 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.861 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.861 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:21:04.861 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:24.151747+0000", 2024-09-11T08:21:04.861 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9482e-05 2024-09-11T08:21:04.861 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.861 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.861 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:21:04.861 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:24.151767+0000", 2024-09-11T08:21:04.861 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9604999999999999e-05 2024-09-11T08:21:04.861 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:04.862 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:21:04.862 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:04.862 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.862 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.862 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:177573 9.a 9:53019ca4:::benchmark_data_smithi150_133616_object61967:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-11T08:21:04.862 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:20:35.017278+0000", 2024-09-11T08:21:04.862 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 29.734970368999999, 2024-09-11T08:21:04.862 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.15801214999999999, 2024-09-11T08:21:04.862 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:21:04.862 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:21:04.862 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:21:04.862 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:21:04.862 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:21:04.862 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 177573 2024-09-11T08:21:04.863 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.863 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:21:04.863 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.863 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:21:04.863 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.017278+0000", 2024-09-11T08:21:04.863 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:04.863 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.863 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.863 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:21:04.863 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.017278+0000", 2024-09-11T08:21:04.863 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:04.863 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.863 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.863 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:21:04.864 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.017276+0000", 2024-09-11T08:21:04.864 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999981 2024-09-11T08:21:04.864 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.864 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.864 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:21:04.864 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.017281+0000", 2024-09-11T08:21:04.864 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2889999999999997e-06 2024-09-11T08:21:04.864 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.864 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.864 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:21:04.864 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.017291+0000", 2024-09-11T08:21:04.864 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.3049999999999998e-06 2024-09-11T08:21:04.864 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.865 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.865 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:21:04.865 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.017294+0000", 2024-09-11T08:21:04.865 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.873e-06 2024-09-11T08:21:04.865 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.865 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.865 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:04.865 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.070461+0000", 2024-09-11T08:21:04.865 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.053166351000000001 2024-09-11T08:21:04.865 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.865 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.865 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:21:04.865 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.070535+0000", 2024-09-11T08:21:04.865 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4441000000000001e-05 2024-09-11T08:21:04.866 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.866 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.866 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,7", 2024-09-11T08:21:04.866 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.070626+0000", 2024-09-11T08:21:04.866 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0519000000000005e-05 2024-09-11T08:21:04.866 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.866 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.866 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:21:04.866 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.075615+0000", 2024-09-11T08:21:04.866 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0049894989999999997 2024-09-11T08:21:04.866 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.866 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.866 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:04.866 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.075677+0000", 2024-09-11T08:21:04.866 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1272000000000006e-05 2024-09-11T08:21:04.866 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.867 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.867 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:04.867 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.175216+0000", 2024-09-11T08:21:04.867 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.099539459999999996 2024-09-11T08:21:04.867 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.867 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.867 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:21:04.867 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.175254+0000", 2024-09-11T08:21:04.867 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8479999999999997e-05 2024-09-11T08:21:04.867 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.867 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.867 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:21:04.867 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.175290+0000", 2024-09-11T08:21:04.867 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5346999999999999e-05 2024-09-11T08:21:04.867 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:04.868 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:21:04.868 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:04.868 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.868 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.868 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:177781 9.a 9:55e8c722:::benchmark_data_smithi150_133616_object62175:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-11T08:21:04.868 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:20:35.255144+0000", 2024-09-11T08:21:04.868 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 29.497104049000001, 2024-09-11T08:21:04.868 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.18980965699999999, 2024-09-11T08:21:04.868 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:21:04.868 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:21:04.868 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:21:04.868 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:21:04.868 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:21:04.868 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 177781 2024-09-11T08:21:04.869 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.869 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:21:04.869 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.869 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:21:04.869 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.255144+0000", 2024-09-11T08:21:04.869 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:04.869 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.869 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.869 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:21:04.869 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.255144+0000", 2024-09-11T08:21:04.869 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:04.869 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.869 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.869 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:21:04.870 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.255143+0000", 2024-09-11T08:21:04.870 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:21:04.870 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.870 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.870 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:21:04.870 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.255148+0000", 2024-09-11T08:21:04.870 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2079999999999999e-06 2024-09-11T08:21:04.870 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.870 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.870 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:21:04.870 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.255157+0000", 2024-09-11T08:21:04.870 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.3819999999999993e-06 2024-09-11T08:21:04.870 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.870 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.870 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:21:04.870 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.255161+0000", 2024-09-11T08:21:04.871 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0679999999999998e-06 2024-09-11T08:21:04.871 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.871 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.871 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:04.871 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.305941+0000", 2024-09-11T08:21:04.871 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.050779455000000001 2024-09-11T08:21:04.871 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.871 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.871 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:21:04.871 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.306012+0000", 2024-09-11T08:21:04.871 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1384000000000006e-05 2024-09-11T08:21:04.871 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.871 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.871 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,7", 2024-09-11T08:21:04.871 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.306093+0000", 2024-09-11T08:21:04.872 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0558999999999999e-05 2024-09-11T08:21:04.872 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.872 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.872 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:21:04.872 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.308734+0000", 2024-09-11T08:21:04.872 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0026410890000000001 2024-09-11T08:21:04.872 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.872 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.872 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:04.872 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.308755+0000", 2024-09-11T08:21:04.872 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1044999999999999e-05 2024-09-11T08:21:04.872 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.872 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.872 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:04.873 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.444872+0000", 2024-09-11T08:21:04.873 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.13611683899999999 2024-09-11T08:21:04.873 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.873 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.873 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:21:04.873 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.444919+0000", 2024-09-11T08:21:04.873 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7605000000000002e-05 2024-09-11T08:21:04.873 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.873 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.873 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:21:04.873 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.444954+0000", 2024-09-11T08:21:04.873 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4298999999999998e-05 2024-09-11T08:21:04.873 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:04.873 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:21:04.874 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:04.874 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.874 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.874 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:177782 9.a 9:564471e4:::benchmark_data_smithi150_133616_object62176:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-11T08:21:04.874 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:20:35.256257+0000", 2024-09-11T08:21:04.874 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 29.495990733999999, 2024-09-11T08:21:04.874 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20652247300000001, 2024-09-11T08:21:04.874 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:21:04.874 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:21:04.874 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:21:04.874 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:21:04.874 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:21:04.874 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 177782 2024-09-11T08:21:04.874 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.874 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:21:04.874 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.875 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:21:04.875 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.256257+0000", 2024-09-11T08:21:04.875 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:04.875 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.875 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.875 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:21:04.875 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.256257+0000", 2024-09-11T08:21:04.875 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:04.875 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.875 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.875 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:21:04.875 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.256257+0000", 2024-09-11T08:21:04.875 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-09-11T08:21:04.876 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.876 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.876 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:21:04.876 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.256260+0000", 2024-09-11T08:21:04.876 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4410000000000002e-06 2024-09-11T08:21:04.876 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.876 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.876 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:21:04.876 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.256268+0000", 2024-09-11T08:21:04.876 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8089999999999993e-06 2024-09-11T08:21:04.876 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.876 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.876 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:21:04.876 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.256272+0000", 2024-09-11T08:21:04.877 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.709e-06 2024-09-11T08:21:04.877 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.877 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.877 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:04.877 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.307344+0000", 2024-09-11T08:21:04.877 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.051072332999999998 2024-09-11T08:21:04.877 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.877 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.877 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:21:04.877 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.307414+0000", 2024-09-11T08:21:04.877 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9989999999999999e-05 2024-09-11T08:21:04.877 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.877 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.878 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,7", 2024-09-11T08:21:04.878 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.307510+0000", 2024-09-11T08:21:04.878 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.6374999999999998e-05 2024-09-11T08:21:04.878 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.878 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.878 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:21:04.878 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.308815+0000", 2024-09-11T08:21:04.878 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001304915 2024-09-11T08:21:04.878 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.878 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.878 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:04.878 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.308973+0000", 2024-09-11T08:21:04.878 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015808 2024-09-11T08:21:04.878 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.878 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.879 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:04.879 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.462678+0000", 2024-09-11T08:21:04.879 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.153704694 2024-09-11T08:21:04.879 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.879 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.879 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:21:04.879 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.462741+0000", 2024-09-11T08:21:04.879 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2949999999999999e-05 2024-09-11T08:21:04.879 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.879 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.879 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:21:04.879 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.462780+0000", 2024-09-11T08:21:04.879 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8760999999999997e-05 2024-09-11T08:21:04.879 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:04.879 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:21:04.879 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:04.880 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.880 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.880 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:180019 9.a 9:51dd620f:::benchmark_data_smithi150_133616_object64413:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-11T08:21:04.880 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:20:37.768464+0000", 2024-09-11T08:21:04.880 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 26.983784437000001, 2024-09-11T08:21:04.880 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.15530835800000001, 2024-09-11T08:21:04.880 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:21:04.880 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:21:04.880 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:21:04.880 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:21:04.880 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:21:04.880 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 180019 2024-09-11T08:21:04.880 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.880 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:21:04.880 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.881 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:21:04.881 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:37.768464+0000", 2024-09-11T08:21:04.881 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:04.881 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.881 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.881 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:21:04.881 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:37.768464+0000", 2024-09-11T08:21:04.881 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:04.881 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.881 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.881 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:21:04.881 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:37.768462+0000", 2024-09-11T08:21:04.881 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999981 2024-09-11T08:21:04.881 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.882 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.882 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:21:04.882 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:37.768468+0000", 2024-09-11T08:21:04.882 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4729999999999997e-06 2024-09-11T08:21:04.882 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.882 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.882 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:21:04.882 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:37.768481+0000", 2024-09-11T08:21:04.882 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3003000000000001e-05 2024-09-11T08:21:04.882 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.882 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.882 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:21:04.882 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:37.768487+0000", 2024-09-11T08:21:04.882 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5339999999999999e-06 2024-09-11T08:21:04.883 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.883 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.883 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:04.883 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:37.818643+0000", 2024-09-11T08:21:04.883 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.050156232000000002 2024-09-11T08:21:04.883 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.883 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.883 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:21:04.883 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:37.818954+0000", 2024-09-11T08:21:04.883 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00031072800000000002 2024-09-11T08:21:04.883 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.883 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.883 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,7", 2024-09-11T08:21:04.883 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:37.819084+0000", 2024-09-11T08:21:04.884 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012989100000000001 2024-09-11T08:21:04.884 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.884 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.884 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:21:04.884 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:37.825423+0000", 2024-09-11T08:21:04.884 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0063389900000000001 2024-09-11T08:21:04.884 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.884 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.884 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:04.884 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:37.842941+0000", 2024-09-11T08:21:04.884 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.017518138999999999 2024-09-11T08:21:04.884 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.884 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.884 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:04.884 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:37.923688+0000", 2024-09-11T08:21:04.885 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.080747330000000006 2024-09-11T08:21:04.885 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.885 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.885 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:21:04.885 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:37.923733+0000", 2024-09-11T08:21:04.885 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5334999999999999e-05 2024-09-11T08:21:04.885 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.885 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.885 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:21:04.885 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:37.923772+0000", 2024-09-11T08:21:04.885 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8593000000000001e-05 2024-09-11T08:21:04.885 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:04.885 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:21:04.885 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:04.885 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.886 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.886 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:180499 9.a 9:5d92d97a:::benchmark_data_smithi150_133616_object64893:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-11T08:21:04.886 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:20:38.311081+0000", 2024-09-11T08:21:04.886 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 26.441167148000002, 2024-09-11T08:21:04.886 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.16502208099999999, 2024-09-11T08:21:04.886 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:21:04.886 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:21:04.886 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:21:04.886 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:21:04.886 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:21:04.886 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 180499 2024-09-11T08:21:04.886 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.886 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:21:04.887 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.887 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:21:04.887 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:38.311081+0000", 2024-09-11T08:21:04.887 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:04.887 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.887 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.887 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:21:04.887 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:38.311081+0000", 2024-09-11T08:21:04.887 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:04.887 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.887 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.887 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:21:04.887 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:38.311079+0000", 2024-09-11T08:21:04.887 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:21:04.887 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.888 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.888 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:21:04.888 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:38.311086+0000", 2024-09-11T08:21:04.888 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1600000000000001e-06 2024-09-11T08:21:04.888 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.888 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.888 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:21:04.888 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:38.311101+0000", 2024-09-11T08:21:04.888 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4877e-05 2024-09-11T08:21:04.888 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.888 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.888 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:21:04.888 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:38.311107+0000", 2024-09-11T08:21:04.889 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8810000000000001e-06 2024-09-11T08:21:04.889 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.889 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.889 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:04.889 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:38.365484+0000", 2024-09-11T08:21:04.889 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.054376431000000003 2024-09-11T08:21:04.889 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.889 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.889 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:21:04.889 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:38.365582+0000", 2024-09-11T08:21:04.889 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.8296999999999998e-05 2024-09-11T08:21:04.889 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.889 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.889 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,7", 2024-09-11T08:21:04.890 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:38.365684+0000", 2024-09-11T08:21:04.890 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000102383 2024-09-11T08:21:04.890 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.890 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.890 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:21:04.890 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:38.372178+0000", 2024-09-11T08:21:04.890 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.006494054 2024-09-11T08:21:04.890 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.890 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.890 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:04.890 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:38.376433+0000", 2024-09-11T08:21:04.890 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0042548880000000001 2024-09-11T08:21:04.890 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.891 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.891 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:04.891 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:38.476009+0000", 2024-09-11T08:21:04.891 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.099575935000000004 2024-09-11T08:21:04.891 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.891 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.891 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:21:04.891 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:38.476063+0000", 2024-09-11T08:21:04.891 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3581000000000001e-05 2024-09-11T08:21:04.891 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.891 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.891 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:21:04.891 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:38.476103+0000", 2024-09-11T08:21:04.891 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0159e-05 2024-09-11T08:21:04.892 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:04.892 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:21:04.892 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:04.892 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.892 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.892 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:206711 9.6 9:67241916:::benchmark_data_smithi150_133616_object91105:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-09-11T08:21:04.892 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:21:01.169424+0000", 2024-09-11T08:21:04.892 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 3.582823893, 2024-09-11T08:21:04.892 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.18378388100000001, 2024-09-11T08:21:04.892 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:21:04.892 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:21:04.892 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:21:04.892 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:21:04.893 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:21:04.893 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 206711 2024-09-11T08:21:04.893 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.893 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:21:04.893 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.893 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:21:04.893 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:01.169424+0000", 2024-09-11T08:21:04.893 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:04.893 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.893 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.893 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:21:04.893 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:01.169424+0000", 2024-09-11T08:21:04.893 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:04.893 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.894 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.894 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:21:04.894 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:01.169422+0000", 2024-09-11T08:21:04.894 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999981 2024-09-11T08:21:04.894 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.894 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.894 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:21:04.894 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:01.169429+0000", 2024-09-11T08:21:04.895 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5060000000000001e-06 2024-09-11T08:21:04.895 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.895 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.895 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:21:04.895 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:01.169440+0000", 2024-09-11T08:21:04.895 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0926e-05 2024-09-11T08:21:04.895 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.895 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.895 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:21:04.895 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:01.169452+0000", 2024-09-11T08:21:04.895 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2035000000000001e-05 2024-09-11T08:21:04.895 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.895 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.895 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:04.896 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:01.169496+0000", 2024-09-11T08:21:04.896 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4459999999999998e-05 2024-09-11T08:21:04.896 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.896 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.896 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:21:04.896 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:01.169609+0000", 2024-09-11T08:21:04.896 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000112742 2024-09-11T08:21:04.896 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.896 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.896 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,2", 2024-09-11T08:21:04.896 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:01.169758+0000", 2024-09-11T08:21:04.896 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014909600000000001 2024-09-11T08:21:04.896 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.897 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.897 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:21:04.897 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:01.174993+0000", 2024-09-11T08:21:04.897 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.005235511 2024-09-11T08:21:04.897 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.897 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.897 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:04.897 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:01.175805+0000", 2024-09-11T08:21:04.897 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00081153799999999999 2024-09-11T08:21:04.897 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.897 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.897 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:04.897 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:01.353113+0000", 2024-09-11T08:21:04.897 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.177308457 2024-09-11T08:21:04.898 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.898 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.898 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:21:04.898 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:01.353165+0000", 2024-09-11T08:21:04.898 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1326000000000003e-05 2024-09-11T08:21:04.898 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:04.898 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:04.898 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:21:04.898 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:01.353208+0000", 2024-09-11T08:21:04.898 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3333e-05 2024-09-11T08:21:04.898 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:04.898 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:21:04.898 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:04.899 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:04.899 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:21:04.899 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-11T08:21:04.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:04.900 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:04.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:21:05.001 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:21:05.032 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:05.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:05.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:21:05.134 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:05.169 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:05.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:05.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:05.271 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:05.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:05.304 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:05.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:05.405 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:05.429 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:21:05 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:21:05] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:21:05.429 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:05 smithi181 ceph-mon[32701]: 9.8 scrub ok 2024-09-11T08:21:05.429 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:05 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:21:05.429 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:05 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:21:05.429 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:05 smithi181 ceph-mon[32701]: 6.1d scrub starts 2024-09-11T08:21:05.429 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:05 smithi181 ceph-mon[32701]: 6.1d scrub ok 2024-09-11T08:21:05.430 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:05 smithi181 ceph-mon[32701]: 3.9 scrub starts 2024-09-11T08:21:05.430 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:05 smithi181 ceph-mon[32701]: 3.9 scrub ok 2024-09-11T08:21:05.430 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:05 smithi181 ceph-mon[32701]: osdmap e298: 8 total, 8 up, 8 in 2024-09-11T08:21:05.430 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:05 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:21:05.430 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:05 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:21:05.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:05.446 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:05.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:05.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:05 smithi150 ceph-mon[29276]: 9.8 scrub ok 2024-09-11T08:21:05.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:05 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:21:05.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:05 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:21:05.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:05 smithi150 ceph-mon[29276]: 6.1d scrub starts 2024-09-11T08:21:05.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:05 smithi150 ceph-mon[29276]: 6.1d scrub ok 2024-09-11T08:21:05.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:05 smithi150 ceph-mon[29276]: 3.9 scrub starts 2024-09-11T08:21:05.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:05 smithi150 ceph-mon[29276]: 3.9 scrub ok 2024-09-11T08:21:05.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:05 smithi150 ceph-mon[29276]: osdmap e298: 8 total, 8 up, 8 in 2024-09-11T08:21:05.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:05 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:21:05.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:05 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:21:05.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:05 smithi150 ceph-mon[39612]: 9.8 scrub ok 2024-09-11T08:21:05.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:05 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:21:05.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:05 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:21:05.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:05 smithi150 ceph-mon[39612]: 6.1d scrub starts 2024-09-11T08:21:05.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:05 smithi150 ceph-mon[39612]: 6.1d scrub ok 2024-09-11T08:21:05.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:05 smithi150 ceph-mon[39612]: 3.9 scrub starts 2024-09-11T08:21:05.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:05 smithi150 ceph-mon[39612]: 3.9 scrub ok 2024-09-11T08:21:05.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:05 smithi150 ceph-mon[39612]: osdmap e298: 8 total, 8 up, 8 in 2024-09-11T08:21:05.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:05 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:21:05.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:05 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:21:05.547 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:05.592 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:05.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:05.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:05.694 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:05.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:05.730 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:05.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:05.831 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:05.867 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:05.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:05.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:05.969 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:06.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:06.002 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:06.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:06.103 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:21:06.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:06.139 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:06.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:21:06.240 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:06.265 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-11T08:21:06.286 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:06.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:06.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:06.388 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:21:06.432 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:06.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:06.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:21:06.480 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:06 smithi150 ceph-mon[29276]: 5.a scrub starts 2024-09-11T08:21:06.480 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:06 smithi150 ceph-mon[29276]: 5.a scrub ok 2024-09-11T08:21:06.481 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:06 smithi150 ceph-mon[29276]: pgmap v617: 148 pgs: 148 active+clean; 1.4 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-09-11T08:21:06.481 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:06 smithi150 ceph-mon[29276]: 6.1c scrub starts 2024-09-11T08:21:06.481 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:06 smithi150 ceph-mon[29276]: 6.1c scrub ok 2024-09-11T08:21:06.481 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:06 smithi150 ceph-mon[39612]: 5.a scrub starts 2024-09-11T08:21:06.481 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:06 smithi150 ceph-mon[39612]: 5.a scrub ok 2024-09-11T08:21:06.481 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:06 smithi150 ceph-mon[39612]: pgmap v617: 148 pgs: 148 active+clean; 1.4 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-09-11T08:21:06.481 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:06 smithi150 ceph-mon[39612]: 6.1c scrub starts 2024-09-11T08:21:06.481 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:06 smithi150 ceph-mon[39612]: 6.1c scrub ok 2024-09-11T08:21:06.534 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:06.555 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:06.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:06.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:06.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:06 smithi181 ceph-mon[32701]: 5.a scrub starts 2024-09-11T08:21:06.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:06 smithi181 ceph-mon[32701]: 5.a scrub ok 2024-09-11T08:21:06.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:06 smithi181 ceph-mon[32701]: pgmap v617: 148 pgs: 148 active+clean; 1.4 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-09-11T08:21:06.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:06 smithi181 ceph-mon[32701]: 6.1c scrub starts 2024-09-11T08:21:06.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:06 smithi181 ceph-mon[32701]: 6.1c scrub ok 2024-09-11T08:21:06.646 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:21:06.658 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:06.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:06.681 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:06.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:06.782 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:06.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:06.805 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:06.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:06.906 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:06.938 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:06.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:06.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:07.040 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:07.077 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:07.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:07.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:07.179 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:07.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:07.207 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:07.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:07.308 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:21:07.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:07 smithi181 ceph-mon[32701]: 5.19 scrub starts 2024-09-11T08:21:07.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:07 smithi181 ceph-mon[32701]: 5.19 scrub ok 2024-09-11T08:21:07.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:07 smithi181 ceph-mon[32701]: 3.12 scrub starts 2024-09-11T08:21:07.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:07 smithi181 ceph-mon[32701]: 3.12 scrub ok 2024-09-11T08:21:07.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:07 smithi181 ceph-mon[32701]: 3.e deep-scrub starts 2024-09-11T08:21:07.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:07 smithi181 ceph-mon[32701]: 3.e deep-scrub ok 2024-09-11T08:21:07.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:07 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:21:07.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:07 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:21:07.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:07 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:21:07.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:07 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:21:07.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:07.358 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:07.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:21:07.455 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:07 smithi150 ceph-mon[29276]: 5.19 scrub starts 2024-09-11T08:21:07.456 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:07 smithi150 ceph-mon[29276]: 5.19 scrub ok 2024-09-11T08:21:07.456 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:07 smithi150 ceph-mon[29276]: 3.12 scrub starts 2024-09-11T08:21:07.456 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:07 smithi150 ceph-mon[29276]: 3.12 scrub ok 2024-09-11T08:21:07.456 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:07 smithi150 ceph-mon[29276]: 3.e deep-scrub starts 2024-09-11T08:21:07.456 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:07 smithi150 ceph-mon[29276]: 3.e deep-scrub ok 2024-09-11T08:21:07.456 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:07 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:21:07.456 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:07 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:21:07.456 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:07 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:21:07.456 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:07 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:21:07.457 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:07 smithi150 ceph-mon[39612]: 5.19 scrub starts 2024-09-11T08:21:07.457 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:07 smithi150 ceph-mon[39612]: 5.19 scrub ok 2024-09-11T08:21:07.457 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:07 smithi150 ceph-mon[39612]: 3.12 scrub starts 2024-09-11T08:21:07.457 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:07 smithi150 ceph-mon[39612]: 3.12 scrub ok 2024-09-11T08:21:07.457 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:07 smithi150 ceph-mon[39612]: 3.e deep-scrub starts 2024-09-11T08:21:07.457 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:07 smithi150 ceph-mon[39612]: 3.e deep-scrub ok 2024-09-11T08:21:07.457 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:07 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:21:07.457 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:07 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:21:07.457 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:07 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:21:07.457 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:07 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:21:07.460 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:07.489 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:07.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:07.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:07.591 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:07.592 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:21:07.639 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:07.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:07.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:07.742 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:07.774 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:07.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:07.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:07.876 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:07.900 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:07.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:07.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:07.927 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:21:07.930 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-09-11T08:21:08.003 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:21:08.049 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:08.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:08.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:21:08.151 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:21:08.186 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:08.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:08.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:21:08.291 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:08.323 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:08.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:08.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:08.425 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:21:08.458 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:08.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:08.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:21:08.470 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:08 smithi150 ceph-mon[29276]: 3.19 scrub starts 2024-09-11T08:21:08.470 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:08 smithi150 ceph-mon[29276]: 3.19 scrub ok 2024-09-11T08:21:08.470 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:08 smithi150 ceph-mon[29276]: pgmap v618: 148 pgs: 148 active+clean; 1.4 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 578 B/s rd, 0 B/s wr, 1.05k op/s 2024-09-11T08:21:08.470 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:08 smithi150 ceph-mon[29276]: 5.10 scrub starts 2024-09-11T08:21:08.470 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:08 smithi150 ceph-mon[29276]: 5.10 scrub ok 2024-09-11T08:21:08.470 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:08 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:21:08.470 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:08 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:21:08.470 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:08 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:21:08.470 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:08 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]}]: dispatch 2024-09-11T08:21:08.471 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:08 smithi150 ceph-mon[39612]: 3.19 scrub starts 2024-09-11T08:21:08.471 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:08 smithi150 ceph-mon[39612]: 3.19 scrub ok 2024-09-11T08:21:08.471 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:08 smithi150 ceph-mon[39612]: pgmap v618: 148 pgs: 148 active+clean; 1.4 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 578 B/s rd, 0 B/s wr, 1.05k op/s 2024-09-11T08:21:08.471 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:08 smithi150 ceph-mon[39612]: 5.10 scrub starts 2024-09-11T08:21:08.471 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:08 smithi150 ceph-mon[39612]: 5.10 scrub ok 2024-09-11T08:21:08.471 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:08 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:21:08.471 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:08 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:21:08.471 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:08 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:21:08.471 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:08 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]}]: dispatch 2024-09-11T08:21:08.480 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:08 smithi181 ceph-mon[32701]: 3.19 scrub starts 2024-09-11T08:21:08.480 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:08 smithi181 ceph-mon[32701]: 3.19 scrub ok 2024-09-11T08:21:08.480 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:08 smithi181 ceph-mon[32701]: pgmap v618: 148 pgs: 148 active+clean; 1.4 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 578 B/s rd, 0 B/s wr, 1.05k op/s 2024-09-11T08:21:08.481 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:08 smithi181 ceph-mon[32701]: 5.10 scrub starts 2024-09-11T08:21:08.481 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:08 smithi181 ceph-mon[32701]: 5.10 scrub ok 2024-09-11T08:21:08.481 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:08 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:21:08.481 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:08 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:21:08.481 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:08 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:21:08.481 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:08 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]}]: dispatch 2024-09-11T08:21:08.561 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:08.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:08.595 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:08.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:08.696 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:08.728 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:08.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:08.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:08.738 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:21:08 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:21:08] "GET /metrics HTTP/1.1" 200 224552 "" "Prometheus/2.43.0" 2024-09-11T08:21:08.758 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:21:08 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:21:08.830 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:08.838 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":618,"stamp":"2024-09-11T08:21:06.983820+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1471879412,"num_objects":22647,"num_object_clones":0,"num_object_copies":67941,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22647,"num_whiteouts":0,"num_read":1656,"num_read_kb":1515,"num_write":209586,"num_write_kb":7400264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16445,"num_bytes_recovered":1064311797,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88293,"ondisk_log_size":88293,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":446,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":8070996,"kb_used_data":4735956,"kb_used_omap":19,"kb_used_meta":3334764,"kb_avail":741890220,"statfs":{"total":767960285184,"available":759695585280,"internally_reserved":0,"allocated":4849618944,"data_stored":4846542540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19955,"internal_metadata":3414798861},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,2,11,5,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-610336768,"num_objects":-9313,"num_object_clones":0,"num_object_copies":-27939,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9313,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":9313,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.855703"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":316,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.076136+0000","last_change":"2024-09-11T08:19:29.169750+0000","last_active":"2024-09-11T08:20:52.076136+0000","last_peered":"2024-09-11T08:20:52.076136+0000","last_clean":"2024-09-11T08:20:52.076136+0000","last_became_active":"2024-09-11T08:19:29.169363+0000","last_became_peered":"2024-09-11T08:19:29.169363+0000","last_unstale":"2024-09-11T08:20:52.076136+0000","last_undegraded":"2024-09-11T08:20:52.076136+0000","last_fullsized":"2024-09-11T08:20:52.076136+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:18.102122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:18:18.102122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0063791840000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"239'15","reported_seq":310,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.070224+0000","last_change":"2024-09-11T08:18:22.028235+0000","last_active":"2024-09-11T08:20:52.070224+0000","last_peered":"2024-09-11T08:20:52.070224+0000","last_clean":"2024-09-11T08:20:52.070224+0000","last_became_active":"2024-09-11T08:07:32.048048+0000","last_became_peered":"2024-09-11T08:07:32.048048+0000","last_unstale":"2024-09-11T08:20:52.070224+0000","last_undegraded":"2024-09-11T08:20:52.070224+0000","last_fullsized":"2024-09-11T08:20:52.070224+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'15","last_scrub_stamp":"2024-09-11T08:18:22.028181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:18:22.028181+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015902539999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":360,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.075709+0000","last_change":"2024-09-11T08:19:29.141302+0000","last_active":"2024-09-11T08:20:52.075709+0000","last_peered":"2024-09-11T08:20:52.075709+0000","last_clean":"2024-09-11T08:20:52.075709+0000","last_became_active":"2024-09-11T08:19:29.140997+0000","last_became_peered":"2024-09-11T08:19:29.140997+0000","last_unstale":"2024-09-11T08:20:52.075709+0000","last_undegraded":"2024-09-11T08:20:52.075709+0000","last_fullsized":"2024-09-11T08:20:52.075709+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:20.487750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:59.531202+0000","last_clean_scrub_stamp":"2024-09-11T08:18:20.487750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0088346129999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":314,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.068971+0000","last_change":"2024-09-11T08:20:11.523247+0000","last_active":"2024-09-11T08:20:52.068971+0000","last_peered":"2024-09-11T08:20:52.068971+0000","last_clean":"2024-09-11T08:20:52.068971+0000","last_became_active":"2024-09-11T08:19:29.147339+0000","last_became_peered":"2024-09-11T08:19:29.147339+0000","last_unstale":"2024-09-11T08:20:52.068971+0000","last_undegraded":"2024-09-11T08:20:52.068971+0000","last_fullsized":"2024-09-11T08:20:52.068971+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:11.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:02.470649+0000","last_clean_scrub_stamp":"2024-09-11T08:20:11.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:14.306367+0000","scrub_duration":0.0056491429999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":318,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.075000+0000","last_change":"2024-09-11T08:20:17.919584+0000","last_active":"2024-09-11T08:20:52.075000+0000","last_peered":"2024-09-11T08:20:52.075000+0000","last_clean":"2024-09-11T08:20:52.075000+0000","last_became_active":"2024-09-11T08:15:56.521104+0000","last_became_peered":"2024-09-11T08:15:56.521104+0000","last_unstale":"2024-09-11T08:20:52.075000+0000","last_undegraded":"2024-09-11T08:20:52.075000+0000","last_fullsized":"2024-09-11T08:20:52.075000+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:17.919504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:20:17.919504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:47.631226+0000","scrub_duration":0.0022382589999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":101,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.936295+0000","last_change":"2024-09-11T08:20:52.936220+0000","last_active":"2024-09-11T08:20:52.936295+0000","last_peered":"2024-09-11T08:20:52.936295+0000","last_clean":"2024-09-11T08:20:52.936295+0000","last_became_active":"2024-09-11T08:17:37.550048+0000","last_became_peered":"2024-09-11T08:17:37.550048+0000","last_unstale":"2024-09-11T08:20:52.936295+0000","last_undegraded":"2024-09-11T08:20:52.936295+0000","last_fullsized":"2024-09-11T08:20:52.936295+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:52.936088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:20:52.936088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:04.291152+0000","scrub_duration":0.0046527160000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"239'34","reported_seq":251,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.078913+0000","last_change":"2024-09-11T08:20:18.632224+0000","last_active":"2024-09-11T08:20:52.078913+0000","last_peered":"2024-09-11T08:20:52.078913+0000","last_clean":"2024-09-11T08:20:52.078913+0000","last_became_active":"2024-09-11T08:17:37.553866+0000","last_became_peered":"2024-09-11T08:17:37.553866+0000","last_unstale":"2024-09-11T08:20:52.078913+0000","last_undegraded":"2024-09-11T08:20:52.078913+0000","last_fullsized":"2024-09-11T08:20:52.078913+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'34","last_scrub_stamp":"2024-09-11T08:20:18.632165+0000","last_deep_scrub":"145'22","last_deep_scrub_stamp":"2024-09-11T08:15:57.478870+0000","last_clean_scrub_stamp":"2024-09-11T08:20:18.632165+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:26.354479+0000","scrub_duration":0.0063155520000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":283,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.070111+0000","last_change":"2024-09-11T08:18:36.998586+0000","last_active":"2024-09-11T08:20:52.070111+0000","last_peered":"2024-09-11T08:20:52.070111+0000","last_clean":"2024-09-11T08:20:52.070111+0000","last_became_active":"2024-09-11T08:07:34.052243+0000","last_became_peered":"2024-09-11T08:07:34.052243+0000","last_unstale":"2024-09-11T08:20:52.070111+0000","last_undegraded":"2024-09-11T08:20:52.070111+0000","last_fullsized":"2024-09-11T08:20:52.070111+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:36.998503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:18:36.998503+0000","last_clean_scrub_stamp":"2024-09-11T08:18:36.998503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0067563429999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":46,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.077199+0000","last_change":"2024-09-11T08:19:42.490622+0000","last_active":"2024-09-11T08:20:52.077199+0000","last_peered":"2024-09-11T08:20:52.077199+0000","last_clean":"2024-09-11T08:20:52.077199+0000","last_became_active":"2024-09-11T08:19:29.142228+0000","last_became_peered":"2024-09-11T08:19:29.142228+0000","last_unstale":"2024-09-11T08:20:52.077199+0000","last_undegraded":"2024-09-11T08:20:52.077199+0000","last_fullsized":"2024-09-11T08:20:52.077199+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:19:42.490567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:54.878409+0000","last_clean_scrub_stamp":"2024-09-11T08:19:42.490567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:08.029051+0000","scrub_duration":0.001226284,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"239'15","reported_seq":323,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.069196+0000","last_change":"2024-09-11T08:20:13.492673+0000","last_active":"2024-09-11T08:20:52.069196+0000","last_peered":"2024-09-11T08:20:52.069196+0000","last_clean":"2024-09-11T08:20:52.069196+0000","last_became_active":"2024-09-11T08:07:32.044626+0000","last_became_peered":"2024-09-11T08:07:32.044626+0000","last_unstale":"2024-09-11T08:20:52.069196+0000","last_undegraded":"2024-09-11T08:20:52.069196+0000","last_fullsized":"2024-09-11T08:20:52.069196+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'15","last_scrub_stamp":"2024-09-11T08:20:13.492593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:20:13.492593+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:24.083450+0000","scrub_duration":0.0023810210000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":46,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.076467+0000","last_change":"2024-09-11T08:20:18.900646+0000","last_active":"2024-09-11T08:20:52.076467+0000","last_peered":"2024-09-11T08:20:52.076467+0000","last_clean":"2024-09-11T08:20:52.076467+0000","last_became_active":"2024-09-11T08:19:29.192476+0000","last_became_peered":"2024-09-11T08:19:29.192476+0000","last_unstale":"2024-09-11T08:20:52.076467+0000","last_undegraded":"2024-09-11T08:20:52.076467+0000","last_fullsized":"2024-09-11T08:20:52.076467+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:18.900482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:20:18.900482+0000","last_clean_scrub_stamp":"2024-09-11T08:20:18.900482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:43.273628+0000","scrub_duration":0.027755386,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":307,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.074640+0000","last_change":"2024-09-11T08:19:09.244391+0000","last_active":"2024-09-11T08:20:52.074640+0000","last_peered":"2024-09-11T08:20:52.074640+0000","last_clean":"2024-09-11T08:20:52.074640+0000","last_became_active":"2024-09-11T08:14:24.792595+0000","last_became_peered":"2024-09-11T08:14:24.792595+0000","last_unstale":"2024-09-11T08:20:52.074640+0000","last_undegraded":"2024-09-11T08:20:52.074640+0000","last_fullsized":"2024-09-11T08:20:52.074640+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:19:09.244337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:19:09.244337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013304670000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":36,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.076618+0000","last_change":"2024-09-11T08:19:29.140825+0000","last_active":"2024-09-11T08:20:52.076618+0000","last_peered":"2024-09-11T08:20:52.076618+0000","last_clean":"2024-09-11T08:20:52.076618+0000","last_became_active":"2024-09-11T08:19:29.140493+0000","last_became_peered":"2024-09-11T08:19:29.140493+0000","last_unstale":"2024-09-11T08:20:52.076618+0000","last_undegraded":"2024-09-11T08:20:52.076618+0000","last_fullsized":"2024-09-11T08:20:52.076618+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:26.321692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:18:26.321692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"239'9","reported_seq":303,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.069469+0000","last_change":"2024-09-11T08:18:21.019771+0000","last_active":"2024-09-11T08:20:52.069469+0000","last_peered":"2024-09-11T08:20:52.069469+0000","last_clean":"2024-09-11T08:20:52.069469+0000","last_became_active":"2024-09-11T08:07:32.048661+0000","last_became_peered":"2024-09-11T08:07:32.048661+0000","last_unstale":"2024-09-11T08:20:52.069469+0000","last_undegraded":"2024-09-11T08:20:52.069469+0000","last_fullsized":"2024-09-11T08:20:52.069469+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'9","last_scrub_stamp":"2024-09-11T08:18:21.019685+0000","last_deep_scrub":"139'7","last_deep_scrub_stamp":"2024-09-11T08:12:24.869318+0000","last_clean_scrub_stamp":"2024-09-11T08:18:21.019685+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.021195169,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":129,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:53.087907+0000","last_change":"2024-09-11T08:20:53.087846+0000","last_active":"2024-09-11T08:20:53.087907+0000","last_peered":"2024-09-11T08:20:53.087907+0000","last_clean":"2024-09-11T08:20:53.087907+0000","last_became_active":"2024-09-11T08:15:56.526954+0000","last_became_peered":"2024-09-11T08:15:56.526954+0000","last_unstale":"2024-09-11T08:20:53.087907+0000","last_undegraded":"2024-09-11T08:20:53.087907+0000","last_fullsized":"2024-09-11T08:20:53.087907+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:53.087699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:20:53.087699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:22.711960+0000","scrub_duration":0.0024133610000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":46,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.076127+0000","last_change":"2024-09-11T08:20:19.930386+0000","last_active":"2024-09-11T08:20:52.076127+0000","last_peered":"2024-09-11T08:20:52.076127+0000","last_clean":"2024-09-11T08:20:52.076127+0000","last_became_active":"2024-09-11T08:19:29.143572+0000","last_became_peered":"2024-09-11T08:19:29.143572+0000","last_unstale":"2024-09-11T08:20:52.076127+0000","last_undegraded":"2024-09-11T08:20:52.076127+0000","last_fullsized":"2024-09-11T08:20:52.076127+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:19.927616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:20:19.927616+0000","last_clean_scrub_stamp":"2024-09-11T08:20:19.927616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:28.201605+0000","scrub_duration":0.0062725719999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":314,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.070674+0000","last_change":"2024-09-11T08:20:49.471916+0000","last_active":"2024-09-11T08:20:52.070674+0000","last_peered":"2024-09-11T08:20:52.070674+0000","last_clean":"2024-09-11T08:20:52.070674+0000","last_became_active":"2024-09-11T08:19:29.149081+0000","last_became_peered":"2024-09-11T08:19:29.149081+0000","last_unstale":"2024-09-11T08:20:52.070674+0000","last_undegraded":"2024-09-11T08:20:52.070674+0000","last_fullsized":"2024-09-11T08:20:52.070674+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:49.471510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:20:49.471510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:15.385674+0000","scrub_duration":0.0048182779999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":326,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.082073+0000","last_change":"2024-09-11T08:20:09.565068+0000","last_active":"2024-09-11T08:20:52.082073+0000","last_peered":"2024-09-11T08:20:52.082073+0000","last_clean":"2024-09-11T08:20:52.082073+0000","last_became_active":"2024-09-11T08:15:56.516941+0000","last_became_peered":"2024-09-11T08:15:56.516941+0000","last_unstale":"2024-09-11T08:20:52.082073+0000","last_undegraded":"2024-09-11T08:20:52.082073+0000","last_fullsized":"2024-09-11T08:20:52.082073+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:09.564941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:20:09.564941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:28.602005+0000","scrub_duration":0.0018499059999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"239'34","reported_seq":364,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-11T08:21:01.847580+0000","last_change":"2024-09-11T08:21:01.847533+0000","last_active":"2024-09-11T08:21:01.847580+0000","last_peered":"2024-09-11T08:21:01.847580+0000","last_clean":"2024-09-11T08:21:01.847580+0000","last_became_active":"2024-09-11T08:07:32.046363+0000","last_became_peered":"2024-09-11T08:07:32.046363+0000","last_unstale":"2024-09-11T08:21:01.847580+0000","last_undegraded":"2024-09-11T08:21:01.847580+0000","last_fullsized":"2024-09-11T08:21:01.847580+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'34","last_scrub_stamp":"2024-09-11T08:21:01.847438+0000","last_deep_scrub":"107'22","last_deep_scrub_stamp":"2024-09-11T08:16:04.159211+0000","last_clean_scrub_stamp":"2024-09-11T08:21:01.847438+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:16.033858+0000","scrub_duration":0.0079036850000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":313,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.076418+0000","last_change":"2024-09-11T08:19:28.870267+0000","last_active":"2024-09-11T08:20:52.076418+0000","last_peered":"2024-09-11T08:20:52.076418+0000","last_clean":"2024-09-11T08:20:52.076418+0000","last_became_active":"2024-09-11T08:07:34.055410+0000","last_became_peered":"2024-09-11T08:07:34.055410+0000","last_unstale":"2024-09-11T08:20:52.076418+0000","last_undegraded":"2024-09-11T08:20:52.076418+0000","last_fullsized":"2024-09-11T08:20:52.076418+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:19:28.870211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:19:28.870211+0000","last_clean_scrub_stamp":"2024-09-11T08:19:28.870211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001404073,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":111,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.072099+0000","last_change":"2024-09-11T08:20:18.753207+0000","last_active":"2024-09-11T08:20:52.072099+0000","last_peered":"2024-09-11T08:20:52.072099+0000","last_clean":"2024-09-11T08:20:52.072099+0000","last_became_active":"2024-09-11T08:19:29.144780+0000","last_became_peered":"2024-09-11T08:19:29.144780+0000","last_unstale":"2024-09-11T08:20:52.072099+0000","last_undegraded":"2024-09-11T08:20:52.072099+0000","last_fullsized":"2024-09-11T08:20:52.072099+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:18.753143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:20:18.753143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:33.458131+0000","scrub_duration":0.013030929,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":297,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.069295+0000","last_change":"2024-09-11T08:18:29.124017+0000","last_active":"2024-09-11T08:20:52.069295+0000","last_peered":"2024-09-11T08:20:52.069295+0000","last_clean":"2024-09-11T08:20:52.069295+0000","last_became_active":"2024-09-11T08:07:30.046284+0000","last_became_peered":"2024-09-11T08:07:30.046284+0000","last_unstale":"2024-09-11T08:20:52.069295+0000","last_undegraded":"2024-09-11T08:20:52.069295+0000","last_fullsized":"2024-09-11T08:20:52.069295+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:29.123933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:18:29.123933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026333300000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"239'18","reported_seq":161,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.079447+0000","last_change":"2024-09-11T08:20:17.626639+0000","last_active":"2024-09-11T08:20:52.079447+0000","last_peered":"2024-09-11T08:20:52.079447+0000","last_clean":"2024-09-11T08:20:52.079447+0000","last_became_active":"2024-09-11T08:17:37.553402+0000","last_became_peered":"2024-09-11T08:17:37.553402+0000","last_unstale":"2024-09-11T08:20:52.079447+0000","last_undegraded":"2024-09-11T08:20:52.079447+0000","last_fullsized":"2024-09-11T08:20:52.079447+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'18","last_scrub_stamp":"2024-09-11T08:20:17.626550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:20:17.626550+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:43.490840+0000","scrub_duration":0.015854423999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":293,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.070456+0000","last_change":"2024-09-11T08:18:37.965178+0000","last_active":"2024-09-11T08:20:52.070456+0000","last_peered":"2024-09-11T08:20:52.070456+0000","last_clean":"2024-09-11T08:20:52.070456+0000","last_became_active":"2024-09-11T08:07:34.056528+0000","last_became_peered":"2024-09-11T08:07:34.056528+0000","last_unstale":"2024-09-11T08:20:52.070456+0000","last_undegraded":"2024-09-11T08:20:52.070456+0000","last_fullsized":"2024-09-11T08:20:52.070456+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:37.964413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:10:40.110490+0000","last_clean_scrub_stamp":"2024-09-11T08:18:37.964413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042785210000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":274,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.068588+0000","last_change":"2024-09-11T08:19:29.163145+0000","last_active":"2024-09-11T08:20:52.068588+0000","last_peered":"2024-09-11T08:20:52.068588+0000","last_clean":"2024-09-11T08:20:52.068588+0000","last_became_active":"2024-09-11T08:19:29.149754+0000","last_became_peered":"2024-09-11T08:19:29.149754+0000","last_unstale":"2024-09-11T08:20:52.068588+0000","last_undegraded":"2024-09-11T08:20:52.068588+0000","last_fullsized":"2024-09-11T08:20:52.068588+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-11T08:18:37.045229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:18:37.045229+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032630189999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":307,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.075844+0000","last_change":"2024-09-11T08:18:27.575426+0000","last_active":"2024-09-11T08:20:52.075844+0000","last_peered":"2024-09-11T08:20:52.075844+0000","last_clean":"2024-09-11T08:20:52.075844+0000","last_became_active":"2024-09-11T08:07:30.045932+0000","last_became_peered":"2024-09-11T08:07:30.045932+0000","last_unstale":"2024-09-11T08:20:52.075844+0000","last_undegraded":"2024-09-11T08:20:52.075844+0000","last_fullsized":"2024-09-11T08:20:52.075844+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:27.575291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:18:27.575291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018750450000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"239'20","reported_seq":397,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.076721+0000","last_change":"2024-09-11T08:18:44.371881+0000","last_active":"2024-09-11T08:20:52.076721+0000","last_peered":"2024-09-11T08:20:52.076721+0000","last_clean":"2024-09-11T08:20:52.076721+0000","last_became_active":"2024-09-11T08:15:56.515817+0000","last_became_peered":"2024-09-11T08:15:56.515817+0000","last_unstale":"2024-09-11T08:20:52.076721+0000","last_undegraded":"2024-09-11T08:20:52.076721+0000","last_fullsized":"2024-09-11T08:20:52.076721+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'20","last_scrub_stamp":"2024-09-11T08:18:44.371760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:18:44.371760+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.035522666000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"254'12","reported_seq":532,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-11T08:20:58.141963+0000","last_change":"2024-09-11T08:20:19.770940+0000","last_active":"2024-09-11T08:20:58.141963+0000","last_peered":"2024-09-11T08:20:58.141963+0000","last_clean":"2024-09-11T08:20:58.141963+0000","last_became_active":"2024-09-11T08:15:56.525498+0000","last_became_peered":"2024-09-11T08:15:56.525498+0000","last_unstale":"2024-09-11T08:20:58.141963+0000","last_undegraded":"2024-09-11T08:20:58.141963+0000","last_fullsized":"2024-09-11T08:20:58.141963+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'12","last_scrub_stamp":"2024-09-11T08:20:19.770841+0000","last_deep_scrub":"254'12","last_deep_scrub_stamp":"2024-09-11T08:18:34.253865+0000","last_clean_scrub_stamp":"2024-09-11T08:20:19.770841+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:37.378623+0000","scrub_duration":0.0026189569999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2","version":"297'13022","reported_seq":13291,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-11T08:21:03.351417+0000","last_change":"2024-09-11T08:20:13.017981+0000","last_active":"2024-09-11T08:21:03.351417+0000","last_peered":"2024-09-11T08:21:03.351417+0000","last_clean":"2024-09-11T08:21:03.351417+0000","last_became_active":"2024-09-11T08:18:01.133542+0000","last_became_peered":"2024-09-11T08:18:01.133542+0000","last_unstale":"2024-09-11T08:21:03.351417+0000","last_undegraded":"2024-09-11T08:21:03.351417+0000","last_fullsized":"2024-09-11T08:21:03.351417+0000","mapping_epoch":246,"log_start":"282'8263","ondisk_log_start":"282'8263","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'9512","last_scrub_stamp":"2024-09-11T08:20:13.017898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:20:13.017898+0000","objects_scrubbed":4986,"log_size":4759,"log_dups_size":0,"ondisk_log_size":4759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:30.441437+0000","scrub_duration":2.0105805170000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91619328,"num_objects":1398,"num_object_clones":0,"num_object_copies":4194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13022,"num_write_kb":461440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"239'27","reported_seq":358,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.078171+0000","last_change":"2024-09-11T08:18:24.523456+0000","last_active":"2024-09-11T08:20:52.078171+0000","last_peered":"2024-09-11T08:20:52.078171+0000","last_clean":"2024-09-11T08:20:52.078171+0000","last_became_active":"2024-09-11T08:07:32.054317+0000","last_became_peered":"2024-09-11T08:07:32.054317+0000","last_unstale":"2024-09-11T08:20:52.078171+0000","last_undegraded":"2024-09-11T08:20:52.078171+0000","last_fullsized":"2024-09-11T08:20:52.078171+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'27","last_scrub_stamp":"2024-09-11T08:18:24.523380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:18:24.523380+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0092214930000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":327,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.077985+0000","last_change":"2024-09-11T08:20:19.839288+0000","last_active":"2024-09-11T08:20:52.077985+0000","last_peered":"2024-09-11T08:20:52.077985+0000","last_clean":"2024-09-11T08:20:52.077985+0000","last_became_active":"2024-09-11T08:15:56.521943+0000","last_became_peered":"2024-09-11T08:15:56.521943+0000","last_unstale":"2024-09-11T08:20:52.077985+0000","last_undegraded":"2024-09-11T08:20:52.077985+0000","last_fullsized":"2024-09-11T08:20:52.077985+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:19.839221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:20:19.839221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:30.887348+0000","scrub_duration":0.0023374770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"271'10","reported_seq":491,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-11T08:20:58.138818+0000","last_change":"2024-09-11T08:19:29.165123+0000","last_active":"2024-09-11T08:20:58.138818+0000","last_peered":"2024-09-11T08:20:58.138818+0000","last_clean":"2024-09-11T08:20:58.138818+0000","last_became_active":"2024-09-11T08:19:29.149773+0000","last_became_peered":"2024-09-11T08:19:29.149773+0000","last_unstale":"2024-09-11T08:20:58.138818+0000","last_undegraded":"2024-09-11T08:20:58.138818+0000","last_fullsized":"2024-09-11T08:20:58.138818+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:18:44.817179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:18:44.817179+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022442809999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":46,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.076956+0000","last_change":"2024-09-11T08:20:10.875817+0000","last_active":"2024-09-11T08:20:52.076956+0000","last_peered":"2024-09-11T08:20:52.076956+0000","last_clean":"2024-09-11T08:20:52.076956+0000","last_became_active":"2024-09-11T08:19:29.164735+0000","last_became_peered":"2024-09-11T08:19:29.164735+0000","last_unstale":"2024-09-11T08:20:52.076956+0000","last_undegraded":"2024-09-11T08:20:52.076956+0000","last_fullsized":"2024-09-11T08:20:52.076956+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:10.875651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:06.616332+0000","last_clean_scrub_stamp":"2024-09-11T08:20:10.875651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:22.781185+0000","scrub_duration":0.018223294000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"297'12951","reported_seq":27020,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-11T08:21:02.980474+0000","last_change":"2024-09-11T08:20:58.754746+0000","last_active":"2024-09-11T08:21:02.980474+0000","last_peered":"2024-09-11T08:21:02.980474+0000","last_clean":"2024-09-11T08:21:02.980474+0000","last_became_active":"2024-09-11T08:19:43.795082+0000","last_became_peered":"2024-09-11T08:19:43.795082+0000","last_unstale":"2024-09-11T08:21:02.980474+0000","last_undegraded":"2024-09-11T08:21:02.980474+0000","last_fullsized":"2024-09-11T08:21:02.980474+0000","mapping_epoch":278,"log_start":"282'8051","ondisk_log_start":"282'8051","created":246,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"297'12668","last_scrub_stamp":"2024-09-11T08:20:58.754649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:20:58.754649+0000","objects_scrubbed":1702,"log_size":4900,"log_dups_size":0,"ondisk_log_size":4900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:27.480813+0000","scrub_duration":0.64335482200000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274712,"num_objects":1409,"num_object_clones":0,"num_object_copies":4227,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1409,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12951,"num_write_kb":459457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6940,"num_bytes_recovered":454819840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"239'30","reported_seq":353,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.067985+0000","last_change":"2024-09-11T08:20:15.829563+0000","last_active":"2024-09-11T08:20:52.067985+0000","last_peered":"2024-09-11T08:20:52.067985+0000","last_clean":"2024-09-11T08:20:52.067985+0000","last_became_active":"2024-09-11T08:17:37.555703+0000","last_became_peered":"2024-09-11T08:17:37.555703+0000","last_unstale":"2024-09-11T08:20:52.067985+0000","last_undegraded":"2024-09-11T08:20:52.067985+0000","last_fullsized":"2024-09-11T08:20:52.067985+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'30","last_scrub_stamp":"2024-09-11T08:20:15.829476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:20:15.829476+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:39.569146+0000","scrub_duration":0.0058827109999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":287,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-11T08:21:02.160105+0000","last_change":"2024-09-11T08:21:02.160052+0000","last_active":"2024-09-11T08:21:02.160105+0000","last_peered":"2024-09-11T08:21:02.160105+0000","last_clean":"2024-09-11T08:21:02.160105+0000","last_became_active":"2024-09-11T08:17:37.553579+0000","last_became_peered":"2024-09-11T08:17:37.553579+0000","last_unstale":"2024-09-11T08:21:02.160105+0000","last_undegraded":"2024-09-11T08:21:02.160105+0000","last_fullsized":"2024-09-11T08:21:02.160105+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:02.159910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:21:02.159910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:20.376173+0000","scrub_duration":0.0086008690000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":320,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:57.523776+0000","last_change":"2024-09-11T08:20:57.523721+0000","last_active":"2024-09-11T08:20:57.523776+0000","last_peered":"2024-09-11T08:20:57.523776+0000","last_clean":"2024-09-11T08:20:57.523776+0000","last_became_active":"2024-09-11T08:15:56.522606+0000","last_became_peered":"2024-09-11T08:15:56.522606+0000","last_unstale":"2024-09-11T08:20:57.523776+0000","last_undegraded":"2024-09-11T08:20:57.523776+0000","last_fullsized":"2024-09-11T08:20:57.523776+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:57.523652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:20:57.523652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:19.553362+0000","scrub_duration":0.0041561580000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":294,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.069130+0000","last_change":"2024-09-11T08:20:28.457738+0000","last_active":"2024-09-11T08:20:52.069130+0000","last_peered":"2024-09-11T08:20:52.069130+0000","last_clean":"2024-09-11T08:20:52.069130+0000","last_became_active":"2024-09-11T08:07:36.069225+0000","last_became_peered":"2024-09-11T08:07:36.069225+0000","last_unstale":"2024-09-11T08:20:52.069130+0000","last_undegraded":"2024-09-11T08:20:52.069130+0000","last_fullsized":"2024-09-11T08:20:52.069130+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:28.457660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:04.937884+0000","last_clean_scrub_stamp":"2024-09-11T08:20:28.457660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:33.621713+0000","scrub_duration":0.0028279630000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"297'12936","reported_seq":12995,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-11T08:21:02.825960+0000","last_change":"2024-09-11T08:18:00.823642+0000","last_active":"2024-09-11T08:21:02.825960+0000","last_peered":"2024-09-11T08:21:02.825960+0000","last_clean":"2024-09-11T08:21:02.825960+0000","last_became_active":"2024-09-11T08:18:00.823223+0000","last_became_peered":"2024-09-11T08:18:00.823223+0000","last_unstale":"2024-09-11T08:21:02.825960+0000","last_undegraded":"2024-09-11T08:21:02.825960+0000","last_fullsized":"2024-09-11T08:21:02.825960+0000","mapping_epoch":246,"log_start":"282'8122","ondisk_log_start":"282'8122","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:17:59.811675+0000","objects_scrubbed":0,"log_size":4814,"log_dups_size":0,"ondisk_log_size":4814,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95682560,"num_objects":1460,"num_object_clones":0,"num_object_copies":4380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12936,"num_write_kb":460672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"239'28","reported_seq":389,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:55.458192+0000","last_change":"2024-09-11T08:20:55.458131+0000","last_active":"2024-09-11T08:20:55.458192+0000","last_peered":"2024-09-11T08:20:55.458192+0000","last_clean":"2024-09-11T08:20:55.458192+0000","last_became_active":"2024-09-11T08:19:29.148311+0000","last_became_peered":"2024-09-11T08:19:29.148311+0000","last_unstale":"2024-09-11T08:20:55.458192+0000","last_undegraded":"2024-09-11T08:20:55.458192+0000","last_fullsized":"2024-09-11T08:20:55.458192+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'28","last_scrub_stamp":"2024-09-11T08:20:55.458034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:20:55.458034+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:08.620710+0000","scrub_duration":0.0051959880000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":306,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.076616+0000","last_change":"2024-09-11T08:20:26.665113+0000","last_active":"2024-09-11T08:20:52.076616+0000","last_peered":"2024-09-11T08:20:52.076616+0000","last_clean":"2024-09-11T08:20:52.076616+0000","last_became_active":"2024-09-11T08:07:30.050484+0000","last_became_peered":"2024-09-11T08:07:30.050484+0000","last_unstale":"2024-09-11T08:20:52.076616+0000","last_undegraded":"2024-09-11T08:20:52.076616+0000","last_fullsized":"2024-09-11T08:20:52.076616+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:26.665013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:47.960329+0000","last_clean_scrub_stamp":"2024-09-11T08:20:26.665013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:26.753491+0000","scrub_duration":0.0089047229999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"82'9","reported_seq":487,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-11T08:21:00.841592+0000","last_change":"2024-09-11T08:21:00.841525+0000","last_active":"2024-09-11T08:21:00.841592+0000","last_peered":"2024-09-11T08:21:00.841592+0000","last_clean":"2024-09-11T08:21:00.841592+0000","last_became_active":"2024-09-11T08:07:34.056143+0000","last_became_peered":"2024-09-11T08:07:34.056143+0000","last_unstale":"2024-09-11T08:21:00.841592+0000","last_undegraded":"2024-09-11T08:21:00.841592+0000","last_fullsized":"2024-09-11T08:21:00.841592+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-11T08:21:00.841426+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-09-11T08:21:00.841426+0000","last_clean_scrub_stamp":"2024-09-11T08:21:00.841426+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:04.644815+0000","scrub_duration":0.0051535979999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":281,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.068967+0000","last_change":"2024-09-11T08:20:22.768959+0000","last_active":"2024-09-11T08:20:52.068967+0000","last_peered":"2024-09-11T08:20:52.068967+0000","last_clean":"2024-09-11T08:20:52.068967+0000","last_became_active":"2024-09-11T08:14:24.788968+0000","last_became_peered":"2024-09-11T08:14:24.788968+0000","last_unstale":"2024-09-11T08:20:52.068967+0000","last_undegraded":"2024-09-11T08:20:52.068967+0000","last_fullsized":"2024-09-11T08:20:52.068967+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:22.768817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:20:22.768817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:28.060784+0000","scrub_duration":0.006813398,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"297'13166","reported_seq":16678,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-11T08:21:03.141435+0000","last_change":"2024-09-11T08:20:27.495067+0000","last_active":"2024-09-11T08:21:03.141435+0000","last_peered":"2024-09-11T08:21:03.141435+0000","last_clean":"2024-09-11T08:21:03.141435+0000","last_became_active":"2024-09-11T08:19:30.147415+0000","last_became_peered":"2024-09-11T08:19:30.147415+0000","last_unstale":"2024-09-11T08:21:03.141435+0000","last_undegraded":"2024-09-11T08:21:03.141435+0000","last_fullsized":"2024-09-11T08:21:03.141435+0000","mapping_epoch":271,"log_start":"280'7806","ondisk_log_start":"280'7806","created":246,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'10749","last_scrub_stamp":"2024-09-11T08:20:27.495011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:20:27.495011+0000","objects_scrubbed":3912,"log_size":5360,"log_dups_size":0,"ondisk_log_size":5360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:54.389445+0000","scrub_duration":1.5107946880000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95682560,"num_objects":1460,"num_object_clones":0,"num_object_copies":4380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13166,"num_write_kb":468032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1634,"num_bytes_recovered":107085824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"239'20","reported_seq":385,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.079827+0000","last_change":"2024-09-11T08:20:13.659040+0000","last_active":"2024-09-11T08:20:52.079827+0000","last_peered":"2024-09-11T08:20:52.079827+0000","last_clean":"2024-09-11T08:20:52.079827+0000","last_became_active":"2024-09-11T08:19:29.143486+0000","last_became_peered":"2024-09-11T08:19:29.143486+0000","last_unstale":"2024-09-11T08:20:52.079827+0000","last_undegraded":"2024-09-11T08:20:52.079827+0000","last_fullsized":"2024-09-11T08:20:52.079827+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'20","last_scrub_stamp":"2024-09-11T08:20:13.658885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:20:13.658885+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:41.943049+0000","scrub_duration":0.012626671000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":46,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:55.691533+0000","last_change":"2024-09-11T08:20:55.691331+0000","last_active":"2024-09-11T08:20:55.691533+0000","last_peered":"2024-09-11T08:20:55.691533+0000","last_clean":"2024-09-11T08:20:55.691533+0000","last_became_active":"2024-09-11T08:19:29.143310+0000","last_became_peered":"2024-09-11T08:19:29.143310+0000","last_unstale":"2024-09-11T08:20:55.691533+0000","last_undegraded":"2024-09-11T08:20:55.691533+0000","last_fullsized":"2024-09-11T08:20:55.691533+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:55.690500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:20:55.690500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:09.255593+0000","scrub_duration":0.013778218,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"271'12","reported_seq":152,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-11T08:20:58.141705+0000","last_change":"2024-09-11T08:20:25.901406+0000","last_active":"2024-09-11T08:20:58.141705+0000","last_peered":"2024-09-11T08:20:58.141705+0000","last_clean":"2024-09-11T08:20:58.141705+0000","last_became_active":"2024-09-11T08:19:29.144614+0000","last_became_peered":"2024-09-11T08:19:29.144614+0000","last_unstale":"2024-09-11T08:20:58.141705+0000","last_undegraded":"2024-09-11T08:20:58.141705+0000","last_fullsized":"2024-09-11T08:20:58.141705+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'12","last_scrub_stamp":"2024-09-11T08:20:25.900976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:20:25.900976+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:33.218880+0000","scrub_duration":0.0053349850000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"66'1","reported_seq":291,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.074800+0000","last_change":"2024-09-11T08:20:28.587056+0000","last_active":"2024-09-11T08:20:52.074800+0000","last_peered":"2024-09-11T08:20:52.074800+0000","last_clean":"2024-09-11T08:20:52.074800+0000","last_became_active":"2024-09-11T08:07:36.072593+0000","last_became_peered":"2024-09-11T08:07:36.072593+0000","last_unstale":"2024-09-11T08:20:52.074800+0000","last_undegraded":"2024-09-11T08:20:52.074800+0000","last_fullsized":"2024-09-11T08:20:52.074800+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-11T08:20:28.586987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:20:28.586987+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:28.588685+0000","scrub_duration":0.0034752110000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"297'12979","reported_seq":13125,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-11T08:21:03.012841+0000","last_change":"2024-09-11T08:20:55.862684+0000","last_active":"2024-09-11T08:21:03.012841+0000","last_peered":"2024-09-11T08:21:03.012841+0000","last_clean":"2024-09-11T08:21:03.012841+0000","last_became_active":"2024-09-11T08:18:00.825771+0000","last_became_peered":"2024-09-11T08:18:00.825771+0000","last_unstale":"2024-09-11T08:21:03.012841+0000","last_undegraded":"2024-09-11T08:21:03.012841+0000","last_fullsized":"2024-09-11T08:21:03.012841+0000","mapping_epoch":246,"log_start":"282'8065","ondisk_log_start":"282'8065","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'12465","last_scrub_stamp":"2024-09-11T08:20:55.862600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:20:55.862600+0000","objects_scrubbed":1941,"log_size":4914,"log_dups_size":0,"ondisk_log_size":4914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:56.821139+0000","scrub_duration":0.724694862,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92340224,"num_objects":1409,"num_object_clones":0,"num_object_copies":4227,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12979,"num_write_kb":460416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"239'29","reported_seq":464,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.081851+0000","last_change":"2024-09-11T08:19:29.249993+0000","last_active":"2024-09-11T08:20:52.081851+0000","last_peered":"2024-09-11T08:20:52.081851+0000","last_clean":"2024-09-11T08:20:52.081851+0000","last_became_active":"2024-09-11T08:19:29.195995+0000","last_became_peered":"2024-09-11T08:19:29.195995+0000","last_unstale":"2024-09-11T08:20:52.081851+0000","last_undegraded":"2024-09-11T08:20:52.081851+0000","last_fullsized":"2024-09-11T08:20:52.081851+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'29","last_scrub_stamp":"2024-09-11T08:18:34.333462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:18:34.333462+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011867379000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":340,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.077702+0000","last_change":"2024-09-11T08:20:19.622253+0000","last_active":"2024-09-11T08:20:52.077702+0000","last_peered":"2024-09-11T08:20:52.077702+0000","last_clean":"2024-09-11T08:20:52.077702+0000","last_became_active":"2024-09-11T08:19:29.141999+0000","last_became_peered":"2024-09-11T08:19:29.141999+0000","last_unstale":"2024-09-11T08:20:52.077702+0000","last_undegraded":"2024-09-11T08:20:52.077702+0000","last_fullsized":"2024-09-11T08:20:52.077702+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:19.622169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:20:19.622169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:31.390592+0000","scrub_duration":0.0054018440000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":342,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.071211+0000","last_change":"2024-09-11T08:20:51.918821+0000","last_active":"2024-09-11T08:20:52.071211+0000","last_peered":"2024-09-11T08:20:52.071211+0000","last_clean":"2024-09-11T08:20:52.071211+0000","last_became_active":"2024-09-11T08:19:29.166609+0000","last_became_peered":"2024-09-11T08:19:29.166609+0000","last_unstale":"2024-09-11T08:20:52.071211+0000","last_undegraded":"2024-09-11T08:20:52.071211+0000","last_fullsized":"2024-09-11T08:20:52.071211+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:51.918741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:20:51.918741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:59.899889+0000","scrub_duration":0.0090788139999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":311,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:56.715408+0000","last_change":"2024-09-11T08:20:56.715356+0000","last_active":"2024-09-11T08:20:56.715408+0000","last_peered":"2024-09-11T08:20:56.715408+0000","last_clean":"2024-09-11T08:20:56.715408+0000","last_became_active":"2024-09-11T08:07:36.631668+0000","last_became_peered":"2024-09-11T08:07:36.631668+0000","last_unstale":"2024-09-11T08:20:56.715408+0000","last_undegraded":"2024-09-11T08:20:56.715408+0000","last_fullsized":"2024-09-11T08:20:56.715408+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:56.715273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:58.517624+0000","last_clean_scrub_stamp":"2024-09-11T08:20:56.715273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:25.260940+0000","scrub_duration":0.0021168839999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"297'12651","reported_seq":14649,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-11T08:21:03.722261+0000","last_change":"2024-09-11T08:20:24.076273+0000","last_active":"2024-09-11T08:21:03.722261+0000","last_peered":"2024-09-11T08:21:03.722261+0000","last_clean":"2024-09-11T08:21:03.722261+0000","last_became_active":"2024-09-11T08:19:41.594593+0000","last_became_peered":"2024-09-11T08:19:41.594593+0000","last_unstale":"2024-09-11T08:21:03.722261+0000","last_undegraded":"2024-09-11T08:21:03.722261+0000","last_fullsized":"2024-09-11T08:21:03.722261+0000","mapping_epoch":276,"log_start":"269'6341","ondisk_log_start":"269'6341","created":246,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"287'10084","last_scrub_stamp":"2024-09-11T08:20:24.076187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:20:24.076187+0000","objects_scrubbed":3917,"log_size":6310,"log_dups_size":0,"ondisk_log_size":6310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:48.089673+0000","scrub_duration":1.449697636,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86048768,"num_objects":1313,"num_object_clones":0,"num_object_copies":3939,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12651,"num_write_kb":446848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":875,"num_bytes_recovered":57344000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"296'52","reported_seq":45,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-11T08:20:58.153771+0000","last_change":"2024-09-11T08:20:53.645194+0000","last_active":"2024-09-11T08:20:58.153771+0000","last_peered":"2024-09-11T08:20:58.153771+0000","last_clean":"2024-09-11T08:20:58.153771+0000","last_became_active":"2024-09-11T08:20:52.086843+0000","last_became_peered":"2024-09-11T08:20:52.086843+0000","last_unstale":"2024-09-11T08:20:58.153771+0000","last_undegraded":"2024-09-11T08:20:58.153771+0000","last_fullsized":"2024-09-11T08:20:58.153771+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'51","last_scrub_stamp":"2024-09-11T08:20:53.645121+0000","last_deep_scrub":"190'39","last_deep_scrub_stamp":"2024-09-11T08:15:52.847968+0000","last_clean_scrub_stamp":"2024-09-11T08:20:53.645121+0000","objects_scrubbed":10,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:09.785967+0000","scrub_duration":0.0034138189999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"59'1","reported_seq":306,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.069741+0000","last_change":"2024-09-11T08:18:28.131832+0000","last_active":"2024-09-11T08:20:52.069741+0000","last_peered":"2024-09-11T08:20:52.069741+0000","last_clean":"2024-09-11T08:20:52.069741+0000","last_became_active":"2024-09-11T08:07:30.047513+0000","last_became_peered":"2024-09-11T08:07:30.047513+0000","last_unstale":"2024-09-11T08:20:52.069741+0000","last_undegraded":"2024-09-11T08:20:52.069741+0000","last_fullsized":"2024-09-11T08:20:52.069741+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-11T08:18:28.131735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:18:28.131735+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0055372269999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":40,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.076685+0000","last_change":"2024-09-11T08:19:29.157831+0000","last_active":"2024-09-11T08:20:52.076685+0000","last_peered":"2024-09-11T08:20:52.076685+0000","last_clean":"2024-09-11T08:20:52.076685+0000","last_became_active":"2024-09-11T08:19:29.142111+0000","last_became_peered":"2024-09-11T08:19:29.142111+0000","last_unstale":"2024-09-11T08:20:52.076685+0000","last_undegraded":"2024-09-11T08:20:52.076685+0000","last_fullsized":"2024-09-11T08:20:52.076685+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-11T08:18:35.267235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:18:35.267235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":292,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.075966+0000","last_change":"2024-09-11T08:18:40.347163+0000","last_active":"2024-09-11T08:20:52.075966+0000","last_peered":"2024-09-11T08:20:52.075966+0000","last_clean":"2024-09-11T08:20:52.075966+0000","last_became_active":"2024-09-11T08:07:34.057154+0000","last_became_peered":"2024-09-11T08:07:34.057154+0000","last_unstale":"2024-09-11T08:20:52.075966+0000","last_undegraded":"2024-09-11T08:20:52.075966+0000","last_fullsized":"2024-09-11T08:20:52.075966+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:40.346999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:10:37.957096+0000","last_clean_scrub_stamp":"2024-09-11T08:18:40.346999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0055147520000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":328,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:55.801028+0000","last_change":"2024-09-11T08:20:55.800959+0000","last_active":"2024-09-11T08:20:55.801028+0000","last_peered":"2024-09-11T08:20:55.801028+0000","last_clean":"2024-09-11T08:20:55.801028+0000","last_became_active":"2024-09-11T08:15:56.525099+0000","last_became_peered":"2024-09-11T08:15:56.525099+0000","last_unstale":"2024-09-11T08:20:55.801028+0000","last_undegraded":"2024-09-11T08:20:55.801028+0000","last_fullsized":"2024-09-11T08:20:55.801028+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:55.800856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:20:55.800856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:07.233576+0000","scrub_duration":0.008329276,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"297'15583","reported_seq":18869,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-11T08:21:03.173328+0000","last_change":"2024-09-11T08:20:54.440180+0000","last_active":"2024-09-11T08:21:03.173328+0000","last_peered":"2024-09-11T08:21:03.173328+0000","last_clean":"2024-09-11T08:21:03.173328+0000","last_became_active":"2024-09-11T08:19:30.136920+0000","last_became_peered":"2024-09-11T08:19:30.136920+0000","last_unstale":"2024-09-11T08:21:03.173328+0000","last_undegraded":"2024-09-11T08:21:03.173328+0000","last_fullsized":"2024-09-11T08:21:03.173328+0000","mapping_epoch":271,"log_start":"286'10132","ondisk_log_start":"286'10132","created":246,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'14982","last_scrub_stamp":"2024-09-11T08:20:54.438550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:20:54.438550+0000","objects_scrubbed":2007,"log_size":5451,"log_dups_size":0,"ondisk_log_size":5451,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:59.543999+0000","scrub_duration":0.80989290199999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90374144,"num_objects":1379,"num_object_clones":0,"num_object_copies":4137,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13129,"num_write_kb":464256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1507,"num_bytes_recovered":98762752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"239'17","reported_seq":395,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.078255+0000","last_change":"2024-09-11T08:20:27.578967+0000","last_active":"2024-09-11T08:20:52.078255+0000","last_peered":"2024-09-11T08:20:52.078255+0000","last_clean":"2024-09-11T08:20:52.078255+0000","last_became_active":"2024-09-11T08:19:29.161554+0000","last_became_peered":"2024-09-11T08:19:29.161554+0000","last_unstale":"2024-09-11T08:20:52.078255+0000","last_undegraded":"2024-09-11T08:20:52.078255+0000","last_fullsized":"2024-09-11T08:20:52.078255+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'17","last_scrub_stamp":"2024-09-11T08:20:27.578887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:20:27.578887+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:37.678491+0000","scrub_duration":0.0043509350000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":310,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.078358+0000","last_change":"2024-09-11T08:18:40.719738+0000","last_active":"2024-09-11T08:20:52.078358+0000","last_peered":"2024-09-11T08:20:52.078358+0000","last_clean":"2024-09-11T08:20:52.078358+0000","last_became_active":"2024-09-11T08:07:30.047014+0000","last_became_peered":"2024-09-11T08:07:30.047014+0000","last_unstale":"2024-09-11T08:20:52.078358+0000","last_undegraded":"2024-09-11T08:20:52.078358+0000","last_fullsized":"2024-09-11T08:20:52.078358+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:40.719536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:18:40.719536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004534344,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"127'2","reported_seq":336,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:57.886421+0000","last_change":"2024-09-11T08:20:57.886361+0000","last_active":"2024-09-11T08:20:57.886421+0000","last_peered":"2024-09-11T08:20:57.886421+0000","last_clean":"2024-09-11T08:20:57.886421+0000","last_became_active":"2024-09-11T08:07:30.045009+0000","last_became_peered":"2024-09-11T08:07:30.045009+0000","last_unstale":"2024-09-11T08:20:57.886421+0000","last_undegraded":"2024-09-11T08:20:57.886421+0000","last_fullsized":"2024-09-11T08:20:57.886421+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'2","last_scrub_stamp":"2024-09-11T08:20:57.886279+0000","last_deep_scrub":"127'2","last_deep_scrub_stamp":"2024-09-11T08:15:01.938497+0000","last_clean_scrub_stamp":"2024-09-11T08:20:57.886279+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:02.854851+0000","scrub_duration":0.0047039409999999997,"objects_trimmed":0,"snaptrim_duration":0.001859533,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":355,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.830895+0000","last_change":"2024-09-11T08:20:52.830754+0000","last_active":"2024-09-11T08:20:52.830895+0000","last_peered":"2024-09-11T08:20:52.830895+0000","last_clean":"2024-09-11T08:20:52.830895+0000","last_became_active":"2024-09-11T08:19:29.145168+0000","last_became_peered":"2024-09-11T08:19:29.145168+0000","last_unstale":"2024-09-11T08:20:52.830895+0000","last_undegraded":"2024-09-11T08:20:52.830895+0000","last_fullsized":"2024-09-11T08:20:52.830895+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:52.830600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:58.889094+0000","last_clean_scrub_stamp":"2024-09-11T08:20:52.830600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:58.857313+0000","scrub_duration":0.024812523,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":22,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.079379+0000","last_change":"2024-09-11T08:20:16.108984+0000","last_active":"2024-09-11T08:20:52.079379+0000","last_peered":"2024-09-11T08:20:52.079379+0000","last_clean":"2024-09-11T08:20:52.079379+0000","last_became_active":"2024-09-11T08:20:16.108674+0000","last_became_peered":"2024-09-11T08:20:16.108674+0000","last_unstale":"2024-09-11T08:20:52.079379+0000","last_undegraded":"2024-09-11T08:20:52.079379+0000","last_fullsized":"2024-09-11T08:20:52.079379+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:11.804556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:58.796217+0000","last_clean_scrub_stamp":"2024-09-11T08:20:11.804556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:18.829561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.b","version":"297'13005","reported_seq":15070,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-11T08:21:03.709273+0000","last_change":"2024-09-11T08:20:22.228081+0000","last_active":"2024-09-11T08:21:03.709273+0000","last_peered":"2024-09-11T08:21:03.709273+0000","last_clean":"2024-09-11T08:21:03.709273+0000","last_became_active":"2024-09-11T08:19:38.767293+0000","last_became_peered":"2024-09-11T08:19:38.767293+0000","last_unstale":"2024-09-11T08:21:03.709273+0000","last_undegraded":"2024-09-11T08:21:03.709273+0000","last_fullsized":"2024-09-11T08:21:03.709273+0000","mapping_epoch":273,"log_start":"272'6694","ondisk_log_start":"272'6694","created":246,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"287'10198","last_scrub_stamp":"2024-09-11T08:20:22.228004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:20:22.228004+0000","objects_scrubbed":4270,"log_size":6311,"log_dups_size":0,"ondisk_log_size":6311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:46.795828+0000","scrub_duration":1.647098065,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92995584,"num_objects":1419,"num_object_clones":0,"num_object_copies":4257,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13005,"num_write_kb":461568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":897,"num_bytes_recovered":58785792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"287'30","reported_seq":346,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.076464+0000","last_change":"2024-09-11T08:20:24.667229+0000","last_active":"2024-09-11T08:20:52.076464+0000","last_peered":"2024-09-11T08:20:52.076464+0000","last_clean":"2024-09-11T08:20:52.076464+0000","last_became_active":"2024-09-11T08:07:32.055407+0000","last_became_peered":"2024-09-11T08:07:32.055407+0000","last_unstale":"2024-09-11T08:20:52.076464+0000","last_undegraded":"2024-09-11T08:20:52.076464+0000","last_fullsized":"2024-09-11T08:20:52.076464+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"287'30","last_scrub_stamp":"2024-09-11T08:20:24.667139+0000","last_deep_scrub":"287'30","last_deep_scrub_stamp":"2024-09-11T08:20:24.667139+0000","last_clean_scrub_stamp":"2024-09-11T08:20:24.667139+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:28.622072+0000","scrub_duration":0.030013558999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"59'1","reported_seq":313,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:53.780117+0000","last_change":"2024-09-11T08:20:53.779883+0000","last_active":"2024-09-11T08:20:53.780117+0000","last_peered":"2024-09-11T08:20:53.780117+0000","last_clean":"2024-09-11T08:20:53.780117+0000","last_became_active":"2024-09-11T08:07:30.050504+0000","last_became_peered":"2024-09-11T08:07:30.050504+0000","last_unstale":"2024-09-11T08:20:53.780117+0000","last_undegraded":"2024-09-11T08:20:53.780117+0000","last_fullsized":"2024-09-11T08:20:53.780117+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-11T08:20:53.779281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:20:53.779281+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:21.136677+0000","scrub_duration":0.0097612610000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"224'6","reported_seq":897,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-11T08:21:01.571941+0000","last_change":"2024-09-11T08:19:28.913001+0000","last_active":"2024-09-11T08:21:01.571941+0000","last_peered":"2024-09-11T08:21:01.571941+0000","last_clean":"2024-09-11T08:21:01.571941+0000","last_became_active":"2024-09-11T08:14:24.793048+0000","last_became_peered":"2024-09-11T08:14:24.793048+0000","last_unstale":"2024-09-11T08:21:01.571941+0000","last_undegraded":"2024-09-11T08:21:01.571941+0000","last_fullsized":"2024-09-11T08:21:01.571941+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'6","last_scrub_stamp":"2024-09-11T08:19:28.912944+0000","last_deep_scrub":"224'6","last_deep_scrub_stamp":"2024-09-11T08:16:55.778284+0000","last_clean_scrub_stamp":"2024-09-11T08:19:28.912944+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001538764,"objects_trimmed":0,"snaptrim_duration":0.000318821,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":404,"num_read_kb":399,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":46,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:54.655152+0000","last_change":"2024-09-11T08:20:54.655099+0000","last_active":"2024-09-11T08:20:54.655152+0000","last_peered":"2024-09-11T08:20:54.655152+0000","last_clean":"2024-09-11T08:20:54.655152+0000","last_became_active":"2024-09-11T08:19:29.154711+0000","last_became_peered":"2024-09-11T08:19:29.154711+0000","last_unstale":"2024-09-11T08:20:54.655152+0000","last_undegraded":"2024-09-11T08:20:54.655152+0000","last_fullsized":"2024-09-11T08:20:54.655152+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:54.655033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:20:54.655033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:58.452585+0000","scrub_duration":0.017642073000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":334,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.075964+0000","last_change":"2024-09-11T08:20:08.606909+0000","last_active":"2024-09-11T08:20:52.075964+0000","last_peered":"2024-09-11T08:20:52.075964+0000","last_clean":"2024-09-11T08:20:52.075964+0000","last_became_active":"2024-09-11T08:19:29.141234+0000","last_became_peered":"2024-09-11T08:19:29.141234+0000","last_unstale":"2024-09-11T08:20:52.075964+0000","last_undegraded":"2024-09-11T08:20:52.075964+0000","last_fullsized":"2024-09-11T08:20:52.075964+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:08.606839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:20:08.606839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:37.940143+0000","scrub_duration":0.0020913960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"297'13260","reported_seq":15299,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-11T08:21:03.728214+0000","last_change":"2024-09-11T08:21:03.411596+0000","last_active":"2024-09-11T08:21:03.728214+0000","last_peered":"2024-09-11T08:21:03.728214+0000","last_clean":"2024-09-11T08:21:03.728214+0000","last_became_active":"2024-09-11T08:19:39.774204+0000","last_became_peered":"2024-09-11T08:19:39.774204+0000","last_unstale":"2024-09-11T08:21:03.728214+0000","last_undegraded":"2024-09-11T08:21:03.728214+0000","last_fullsized":"2024-09-11T08:21:03.728214+0000","mapping_epoch":274,"log_start":"272'6912","ondisk_log_start":"272'6912","created":246,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"297'13236","last_scrub_stamp":"2024-09-11T08:21:03.411511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:21:03.411511+0000","objects_scrubbed":1460,"log_size":6348,"log_dups_size":0,"ondisk_log_size":6348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:08.063302+0000","scrub_duration":0.67056985499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92798976,"num_objects":1416,"num_object_clones":0,"num_object_copies":4248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13260,"num_write_kb":469632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":940,"num_bytes_recovered":61603840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"239'33","reported_seq":378,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.071850+0000","last_change":"2024-09-11T08:20:16.770435+0000","last_active":"2024-09-11T08:20:52.071850+0000","last_peered":"2024-09-11T08:20:52.071850+0000","last_clean":"2024-09-11T08:20:52.071850+0000","last_became_active":"2024-09-11T08:14:24.789154+0000","last_became_peered":"2024-09-11T08:14:24.789154+0000","last_unstale":"2024-09-11T08:20:52.071850+0000","last_undegraded":"2024-09-11T08:20:52.071850+0000","last_fullsized":"2024-09-11T08:20:52.071850+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'33","last_scrub_stamp":"2024-09-11T08:20:16.770344+0000","last_deep_scrub":"84'21","last_deep_scrub_stamp":"2024-09-11T08:08:56.705306+0000","last_clean_scrub_stamp":"2024-09-11T08:20:16.770344+0000","objects_scrubbed":7,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:45.920805+0000","scrub_duration":0.037009359999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":36,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.076677+0000","last_change":"2024-09-11T08:19:29.144089+0000","last_active":"2024-09-11T08:20:52.076677+0000","last_peered":"2024-09-11T08:20:52.076677+0000","last_clean":"2024-09-11T08:20:52.076677+0000","last_became_active":"2024-09-11T08:19:29.143505+0000","last_became_peered":"2024-09-11T08:19:29.143505+0000","last_unstale":"2024-09-11T08:20:52.076677+0000","last_undegraded":"2024-09-11T08:20:52.076677+0000","last_fullsized":"2024-09-11T08:20:52.076677+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:45.423258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:06.236119+0000","last_clean_scrub_stamp":"2024-09-11T08:18:45.423258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":125,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.067817+0000","last_change":"2024-09-11T08:18:41.048985+0000","last_active":"2024-09-11T08:20:52.067817+0000","last_peered":"2024-09-11T08:20:52.067817+0000","last_clean":"2024-09-11T08:20:52.067817+0000","last_became_active":"2024-09-11T08:15:56.523430+0000","last_became_peered":"2024-09-11T08:15:56.523430+0000","last_unstale":"2024-09-11T08:20:52.067817+0000","last_undegraded":"2024-09-11T08:20:52.067817+0000","last_fullsized":"2024-09-11T08:20:52.067817+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-11T08:18:41.048903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:05:21.231856+0000","last_clean_scrub_stamp":"2024-09-11T08:18:41.048903+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003255703,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":139,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.068458+0000","last_change":"2024-09-11T08:20:25.775943+0000","last_active":"2024-09-11T08:20:52.068458+0000","last_peered":"2024-09-11T08:20:52.068458+0000","last_clean":"2024-09-11T08:20:52.068458+0000","last_became_active":"2024-09-11T08:19:29.147576+0000","last_became_peered":"2024-09-11T08:19:29.147576+0000","last_unstale":"2024-09-11T08:20:52.068458+0000","last_undegraded":"2024-09-11T08:20:52.068458+0000","last_fullsized":"2024-09-11T08:20:52.068458+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:25.775869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:20:25.775869+0000","last_clean_scrub_stamp":"2024-09-11T08:20:25.775869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:39.906509+0000","scrub_duration":0.004676014,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":46,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.077739+0000","last_change":"2024-09-11T08:20:29.154000+0000","last_active":"2024-09-11T08:20:52.077739+0000","last_peered":"2024-09-11T08:20:52.077739+0000","last_clean":"2024-09-11T08:20:52.077739+0000","last_became_active":"2024-09-11T08:19:29.152071+0000","last_became_peered":"2024-09-11T08:19:29.152071+0000","last_unstale":"2024-09-11T08:20:52.077739+0000","last_undegraded":"2024-09-11T08:20:52.077739+0000","last_fullsized":"2024-09-11T08:20:52.077739+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:29.153871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:01.777429+0000","last_clean_scrub_stamp":"2024-09-11T08:20:29.153871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:31.775315+0000","scrub_duration":0.0015196540000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"297'15615","reported_seq":15878,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-11T08:21:02.987578+0000","last_change":"2024-09-11T08:20:51.170996+0000","last_active":"2024-09-11T08:21:02.987578+0000","last_peered":"2024-09-11T08:21:02.987578+0000","last_clean":"2024-09-11T08:21:02.987578+0000","last_became_active":"2024-09-11T08:18:00.826126+0000","last_became_peered":"2024-09-11T08:18:00.826126+0000","last_unstale":"2024-09-11T08:21:02.987578+0000","last_undegraded":"2024-09-11T08:21:02.987578+0000","last_fullsized":"2024-09-11T08:21:02.987578+0000","mapping_epoch":246,"log_start":"288'10734","ondisk_log_start":"288'10734","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'14731","last_scrub_stamp":"2024-09-11T08:20:51.170867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:20:51.170867+0000","objects_scrubbed":2353,"log_size":4881,"log_dups_size":0,"ondisk_log_size":4881,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:14.132698+0000","scrub_duration":1.062925823,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94240768,"num_objects":1438,"num_object_clones":0,"num_object_copies":4314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13266,"num_write_kb":470528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"239'24","reported_seq":331,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.070988+0000","last_change":"2024-09-11T08:18:40.895400+0000","last_active":"2024-09-11T08:20:52.070988+0000","last_peered":"2024-09-11T08:20:52.070988+0000","last_clean":"2024-09-11T08:20:52.070988+0000","last_became_active":"2024-09-11T08:07:32.047971+0000","last_became_peered":"2024-09-11T08:07:32.047971+0000","last_unstale":"2024-09-11T08:20:52.070988+0000","last_undegraded":"2024-09-11T08:20:52.070988+0000","last_fullsized":"2024-09-11T08:20:52.070988+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'24","last_scrub_stamp":"2024-09-11T08:18:40.895293+0000","last_deep_scrub":"88'16","last_deep_scrub_stamp":"2024-09-11T08:09:39.635984+0000","last_clean_scrub_stamp":"2024-09-11T08:18:40.895293+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010336458999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":304,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.068793+0000","last_change":"2024-09-11T08:17:52.133078+0000","last_active":"2024-09-11T08:20:52.068793+0000","last_peered":"2024-09-11T08:20:52.068793+0000","last_clean":"2024-09-11T08:20:52.068793+0000","last_became_active":"2024-09-11T08:14:24.789888+0000","last_became_peered":"2024-09-11T08:14:24.789888+0000","last_unstale":"2024-09-11T08:20:52.068793+0000","last_undegraded":"2024-09-11T08:20:52.068793+0000","last_fullsized":"2024-09-11T08:20:52.068793+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:17:52.133025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:36.417911+0000","last_clean_scrub_stamp":"2024-09-11T08:17:52.133025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012243499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":36,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.077612+0000","last_change":"2024-09-11T08:19:29.154565+0000","last_active":"2024-09-11T08:20:52.077612+0000","last_peered":"2024-09-11T08:20:52.077612+0000","last_clean":"2024-09-11T08:20:52.077612+0000","last_became_active":"2024-09-11T08:19:29.153976+0000","last_became_peered":"2024-09-11T08:19:29.153976+0000","last_unstale":"2024-09-11T08:20:52.077612+0000","last_undegraded":"2024-09-11T08:20:52.077612+0000","last_fullsized":"2024-09-11T08:20:52.077612+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:30.247009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:01.558340+0000","last_clean_scrub_stamp":"2024-09-11T08:18:30.247009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":114,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.071711+0000","last_change":"2024-09-11T08:20:11.704845+0000","last_active":"2024-09-11T08:20:52.071711+0000","last_peered":"2024-09-11T08:20:52.071711+0000","last_clean":"2024-09-11T08:20:52.071711+0000","last_became_active":"2024-09-11T08:19:29.144592+0000","last_became_peered":"2024-09-11T08:19:29.144592+0000","last_unstale":"2024-09-11T08:20:52.071711+0000","last_undegraded":"2024-09-11T08:20:52.071711+0000","last_fullsized":"2024-09-11T08:20:52.071711+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:11.704755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:20:11.704755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:19.157575+0000","scrub_duration":0.0078689060000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.0","version":"297'13073","reported_seq":13220,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-11T08:21:03.621769+0000","last_change":"2024-09-11T08:20:55.368446+0000","last_active":"2024-09-11T08:21:03.621769+0000","last_peered":"2024-09-11T08:21:03.621769+0000","last_clean":"2024-09-11T08:21:03.621769+0000","last_became_active":"2024-09-11T08:18:01.133673+0000","last_became_peered":"2024-09-11T08:18:01.133673+0000","last_unstale":"2024-09-11T08:21:03.621769+0000","last_undegraded":"2024-09-11T08:21:03.621769+0000","last_fullsized":"2024-09-11T08:21:03.621769+0000","mapping_epoch":246,"log_start":"282'8302","ondisk_log_start":"282'8302","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'12508","last_scrub_stamp":"2024-09-11T08:20:55.368362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:20:55.368362+0000","objects_scrubbed":1955,"log_size":4771,"log_dups_size":0,"ondisk_log_size":4771,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:01.578532+0000","scrub_duration":0.68437722499999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89587712,"num_objects":1367,"num_object_clones":0,"num_object_copies":4101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13073,"num_write_kb":462080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"239'39","reported_seq":408,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.070348+0000","last_change":"2024-09-11T08:19:09.625136+0000","last_active":"2024-09-11T08:20:52.070348+0000","last_peered":"2024-09-11T08:20:52.070348+0000","last_clean":"2024-09-11T08:20:52.070348+0000","last_became_active":"2024-09-11T08:19:09.604525+0000","last_became_peered":"2024-09-11T08:19:09.604525+0000","last_unstale":"2024-09-11T08:20:52.070348+0000","last_undegraded":"2024-09-11T08:20:52.070348+0000","last_fullsized":"2024-09-11T08:20:52.070348+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'39","last_scrub_stamp":"2024-09-11T08:18:31.141746+0000","last_deep_scrub":"77'25","last_deep_scrub_stamp":"2024-09-11T08:08:32.458210+0000","last_clean_scrub_stamp":"2024-09-11T08:18:31.141746+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0076700900000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":322,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.071408+0000","last_change":"2024-09-11T08:20:17.724434+0000","last_active":"2024-09-11T08:20:52.071408+0000","last_peered":"2024-09-11T08:20:52.071408+0000","last_clean":"2024-09-11T08:20:52.071408+0000","last_became_active":"2024-09-11T08:07:30.042555+0000","last_became_peered":"2024-09-11T08:07:30.042555+0000","last_unstale":"2024-09-11T08:20:52.071408+0000","last_undegraded":"2024-09-11T08:20:52.071408+0000","last_fullsized":"2024-09-11T08:20:52.071408+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:17.724372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:20:17.724372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:39.478392+0000","scrub_duration":0.003123678,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":303,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.076046+0000","last_change":"2024-09-11T08:18:26.543459+0000","last_active":"2024-09-11T08:20:52.076046+0000","last_peered":"2024-09-11T08:20:52.076046+0000","last_clean":"2024-09-11T08:20:52.076046+0000","last_became_active":"2024-09-11T08:07:34.054961+0000","last_became_peered":"2024-09-11T08:07:34.054961+0000","last_unstale":"2024-09-11T08:20:52.076046+0000","last_undegraded":"2024-09-11T08:20:52.076046+0000","last_fullsized":"2024-09-11T08:20:52.076046+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:26.543286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:18:26.543286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0060933999999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":330,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-11T08:21:02.823126+0000","last_change":"2024-09-11T08:21:02.823083+0000","last_active":"2024-09-11T08:21:02.823126+0000","last_peered":"2024-09-11T08:21:02.823126+0000","last_clean":"2024-09-11T08:21:02.823126+0000","last_became_active":"2024-09-11T08:07:36.071194+0000","last_became_peered":"2024-09-11T08:07:36.071194+0000","last_unstale":"2024-09-11T08:21:02.823126+0000","last_undegraded":"2024-09-11T08:21:02.823126+0000","last_fullsized":"2024-09-11T08:21:02.823126+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:02.822996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:07.930137+0000","last_clean_scrub_stamp":"2024-09-11T08:21:02.822996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:13.236302+0000","scrub_duration":0.005271441,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"297'12864","reported_seq":13117,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-11T08:21:02.898515+0000","last_change":"2024-09-11T08:20:17.488165+0000","last_active":"2024-09-11T08:21:02.898515+0000","last_peered":"2024-09-11T08:21:02.898515+0000","last_clean":"2024-09-11T08:21:02.898515+0000","last_became_active":"2024-09-11T08:18:00.828320+0000","last_became_peered":"2024-09-11T08:18:00.828320+0000","last_unstale":"2024-09-11T08:21:02.898515+0000","last_undegraded":"2024-09-11T08:21:02.898515+0000","last_fullsized":"2024-09-11T08:21:02.898515+0000","mapping_epoch":246,"log_start":"266'6258","ondisk_log_start":"266'6258","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"285'9834","last_scrub_stamp":"2024-09-11T08:20:17.488079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:20:17.488079+0000","objects_scrubbed":4527,"log_size":6606,"log_dups_size":0,"ondisk_log_size":6606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:26.015958+0000","scrub_duration":1.748475096,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92405760,"num_objects":1410,"num_object_clones":0,"num_object_copies":4230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12864,"num_write_kb":456768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"239'25","reported_seq":333,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.070568+0000","last_change":"2024-09-11T08:20:16.527717+0000","last_active":"2024-09-11T08:20:52.070568+0000","last_peered":"2024-09-11T08:20:52.070568+0000","last_clean":"2024-09-11T08:20:52.070568+0000","last_became_active":"2024-09-11T08:07:32.043314+0000","last_became_peered":"2024-09-11T08:07:32.043314+0000","last_unstale":"2024-09-11T08:20:52.070568+0000","last_undegraded":"2024-09-11T08:20:52.070568+0000","last_fullsized":"2024-09-11T08:20:52.070568+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'25","last_scrub_stamp":"2024-09-11T08:20:16.527630+0000","last_deep_scrub":"224'15","last_deep_scrub_stamp":"2024-09-11T08:17:01.126478+0000","last_clean_scrub_stamp":"2024-09-11T08:20:16.527630+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:37.247145+0000","scrub_duration":0.0059554439999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":91,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.082014+0000","last_change":"2024-09-11T08:20:10.609260+0000","last_active":"2024-09-11T08:20:52.082014+0000","last_peered":"2024-09-11T08:20:52.082014+0000","last_clean":"2024-09-11T08:20:52.082014+0000","last_became_active":"2024-09-11T08:17:37.557278+0000","last_became_peered":"2024-09-11T08:17:37.557278+0000","last_unstale":"2024-09-11T08:20:52.082014+0000","last_undegraded":"2024-09-11T08:20:52.082014+0000","last_fullsized":"2024-09-11T08:20:52.082014+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:10.609170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:45.979847+0000","last_clean_scrub_stamp":"2024-09-11T08:20:10.609170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:31.049454+0000","scrub_duration":0.017294936,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"271'14","reported_seq":534,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-11T08:20:58.140544+0000","last_change":"2024-09-11T08:20:54.836617+0000","last_active":"2024-09-11T08:20:58.140544+0000","last_peered":"2024-09-11T08:20:58.140544+0000","last_clean":"2024-09-11T08:20:58.140544+0000","last_became_active":"2024-09-11T08:19:29.153997+0000","last_became_peered":"2024-09-11T08:19:29.153997+0000","last_unstale":"2024-09-11T08:20:58.140544+0000","last_undegraded":"2024-09-11T08:20:58.140544+0000","last_fullsized":"2024-09-11T08:20:58.140544+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'14","last_scrub_stamp":"2024-09-11T08:20:54.836531+0000","last_deep_scrub":"171'12","last_deep_scrub_stamp":"2024-09-11T08:14:56.881299+0000","last_clean_scrub_stamp":"2024-09-11T08:20:54.836531+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:56.071165+0000","scrub_duration":0.0063520010000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":291,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.068459+0000","last_change":"2024-09-11T08:20:50.427308+0000","last_active":"2024-09-11T08:20:52.068459+0000","last_peered":"2024-09-11T08:20:52.068459+0000","last_clean":"2024-09-11T08:20:52.068459+0000","last_became_active":"2024-09-11T08:07:36.074684+0000","last_became_peered":"2024-09-11T08:07:36.074684+0000","last_unstale":"2024-09-11T08:20:52.068459+0000","last_undegraded":"2024-09-11T08:20:52.068459+0000","last_fullsized":"2024-09-11T08:20:52.068459+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:50.427224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:14.613532+0000","last_clean_scrub_stamp":"2024-09-11T08:20:50.427224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:59.232555+0000","scrub_duration":0.0028770409999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"297'13111","reported_seq":13363,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-11T08:21:03.146055+0000","last_change":"2024-09-11T08:20:23.741617+0000","last_active":"2024-09-11T08:21:03.146055+0000","last_peered":"2024-09-11T08:21:03.146055+0000","last_clean":"2024-09-11T08:21:03.146055+0000","last_became_active":"2024-09-11T08:19:30.134196+0000","last_became_peered":"2024-09-11T08:19:30.134196+0000","last_unstale":"2024-09-11T08:21:03.146055+0000","last_undegraded":"2024-09-11T08:21:03.146055+0000","last_fullsized":"2024-09-11T08:21:03.146055+0000","mapping_epoch":271,"log_start":"280'7720","ondisk_log_start":"280'7720","created":246,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"287'10492","last_scrub_stamp":"2024-09-11T08:20:23.741530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:20:23.741530+0000","objects_scrubbed":4087,"log_size":5391,"log_dups_size":0,"ondisk_log_size":5391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:43.872652+0000","scrub_duration":1.814425969,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92471296,"num_objects":1411,"num_object_clones":0,"num_object_copies":4233,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13111,"num_write_kb":464704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"239'15","reported_seq":319,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.081766+0000","last_change":"2024-09-11T08:18:27.143512+0000","last_active":"2024-09-11T08:20:52.081766+0000","last_peered":"2024-09-11T08:20:52.081766+0000","last_clean":"2024-09-11T08:20:52.081766+0000","last_became_active":"2024-09-11T08:07:32.054885+0000","last_became_peered":"2024-09-11T08:07:32.054885+0000","last_unstale":"2024-09-11T08:20:52.081766+0000","last_undegraded":"2024-09-11T08:20:52.081766+0000","last_fullsized":"2024-09-11T08:20:52.081766+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'15","last_scrub_stamp":"2024-09-11T08:18:27.143303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:18:27.143303+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00417571,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":46,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-11T08:21:00.730596+0000","last_change":"2024-09-11T08:21:00.730548+0000","last_active":"2024-09-11T08:21:00.730596+0000","last_peered":"2024-09-11T08:21:00.730596+0000","last_clean":"2024-09-11T08:21:00.730596+0000","last_became_active":"2024-09-11T08:19:29.143174+0000","last_became_peered":"2024-09-11T08:19:29.143174+0000","last_unstale":"2024-09-11T08:21:00.730596+0000","last_undegraded":"2024-09-11T08:21:00.730596+0000","last_fullsized":"2024-09-11T08:21:00.730596+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:00.730460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:21:00.730460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:17.761419+0000","scrub_duration":0.0102811,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":322,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.079116+0000","last_change":"2024-09-11T08:19:10.193992+0000","last_active":"2024-09-11T08:20:52.079116+0000","last_peered":"2024-09-11T08:20:52.079116+0000","last_clean":"2024-09-11T08:20:52.079116+0000","last_became_active":"2024-09-11T08:07:34.056606+0000","last_became_peered":"2024-09-11T08:07:34.056606+0000","last_unstale":"2024-09-11T08:20:52.079116+0000","last_undegraded":"2024-09-11T08:20:52.079116+0000","last_fullsized":"2024-09-11T08:20:52.079116+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:19:10.193940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:19:10.193940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015764850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":317,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.075420+0000","last_change":"2024-09-11T08:18:32.278327+0000","last_active":"2024-09-11T08:20:52.075420+0000","last_peered":"2024-09-11T08:20:52.075420+0000","last_clean":"2024-09-11T08:20:52.075420+0000","last_became_active":"2024-09-11T08:15:56.518884+0000","last_became_peered":"2024-09-11T08:15:56.518884+0000","last_unstale":"2024-09-11T08:20:52.075420+0000","last_undegraded":"2024-09-11T08:20:52.075420+0000","last_fullsized":"2024-09-11T08:20:52.075420+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:32.278257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:18:32.278257+0000","last_clean_scrub_stamp":"2024-09-11T08:18:32.278257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.021912361000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"297'13051","reported_seq":18881,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-11T08:21:03.123232+0000","last_change":"2024-09-11T08:20:23.147847+0000","last_active":"2024-09-11T08:21:03.123232+0000","last_peered":"2024-09-11T08:21:03.123232+0000","last_clean":"2024-09-11T08:21:03.123232+0000","last_became_active":"2024-09-11T08:19:30.138233+0000","last_became_peered":"2024-09-11T08:19:30.138233+0000","last_unstale":"2024-09-11T08:21:03.123232+0000","last_undegraded":"2024-09-11T08:21:03.123232+0000","last_fullsized":"2024-09-11T08:21:03.123232+0000","mapping_epoch":271,"log_start":"279'7619","ondisk_log_start":"279'7619","created":246,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"287'10327","last_scrub_stamp":"2024-09-11T08:20:23.147735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:20:23.147735+0000","objects_scrubbed":4205,"log_size":5432,"log_dups_size":0,"ondisk_log_size":5432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:51.571739+0000","scrub_duration":2.2078867029999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91029504,"num_objects":1389,"num_object_clones":0,"num_object_copies":4167,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13051,"num_write_kb":462080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2787,"num_bytes_recovered":182648832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"239'35","reported_seq":446,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.071510+0000","last_change":"2024-09-11T08:20:23.930556+0000","last_active":"2024-09-11T08:20:52.071510+0000","last_peered":"2024-09-11T08:20:52.071510+0000","last_clean":"2024-09-11T08:20:52.071510+0000","last_became_active":"2024-09-11T08:15:56.526961+0000","last_became_peered":"2024-09-11T08:15:56.526961+0000","last_unstale":"2024-09-11T08:20:52.071510+0000","last_undegraded":"2024-09-11T08:20:52.071510+0000","last_fullsized":"2024-09-11T08:20:52.071510+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'35","last_scrub_stamp":"2024-09-11T08:20:23.930464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:20:23.930464+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:24.598234+0000","scrub_duration":0.003826734,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":139,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.069816+0000","last_change":"2024-09-11T08:20:09.859624+0000","last_active":"2024-09-11T08:20:52.069816+0000","last_peered":"2024-09-11T08:20:52.069816+0000","last_clean":"2024-09-11T08:20:52.069816+0000","last_became_active":"2024-09-11T08:19:29.152114+0000","last_became_peered":"2024-09-11T08:19:29.152114+0000","last_unstale":"2024-09-11T08:20:52.069816+0000","last_undegraded":"2024-09-11T08:20:52.069816+0000","last_fullsized":"2024-09-11T08:20:52.069816+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:09.859549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:44.385372+0000","last_clean_scrub_stamp":"2024-09-11T08:20:09.859549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:32.516863+0000","scrub_duration":0.0055679850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":345,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:53.860689+0000","last_change":"2024-09-11T08:20:53.860652+0000","last_active":"2024-09-11T08:20:53.860689+0000","last_peered":"2024-09-11T08:20:53.860689+0000","last_clean":"2024-09-11T08:20:53.860689+0000","last_became_active":"2024-09-11T08:19:29.178705+0000","last_became_peered":"2024-09-11T08:19:29.178705+0000","last_unstale":"2024-09-11T08:20:53.860689+0000","last_undegraded":"2024-09-11T08:20:53.860689+0000","last_fullsized":"2024-09-11T08:20:53.860689+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:53.860586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:20:53.860586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:19.805109+0000","scrub_duration":0.030572999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":119,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.067832+0000","last_change":"2024-09-11T08:20:21.780282+0000","last_active":"2024-09-11T08:20:52.067832+0000","last_peered":"2024-09-11T08:20:52.067832+0000","last_clean":"2024-09-11T08:20:52.067832+0000","last_became_active":"2024-09-11T08:15:56.518169+0000","last_became_peered":"2024-09-11T08:15:56.518169+0000","last_unstale":"2024-09-11T08:20:52.067832+0000","last_undegraded":"2024-09-11T08:20:52.067832+0000","last_fullsized":"2024-09-11T08:20:52.067832+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:21.780224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:20:21.780224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:39.541601+0000","scrub_duration":0.0071716779999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"297'13049","reported_seq":13275,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-11T08:21:03.608589+0000","last_change":"2024-09-11T08:20:27.162668+0000","last_active":"2024-09-11T08:21:03.608589+0000","last_peered":"2024-09-11T08:21:03.608589+0000","last_clean":"2024-09-11T08:21:03.608589+0000","last_became_active":"2024-09-11T08:18:00.827140+0000","last_became_peered":"2024-09-11T08:18:00.827140+0000","last_unstale":"2024-09-11T08:21:03.608589+0000","last_undegraded":"2024-09-11T08:21:03.608589+0000","last_fullsized":"2024-09-11T08:21:03.608589+0000","mapping_epoch":246,"log_start":"282'8289","ondisk_log_start":"282'8289","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'10610","last_scrub_stamp":"2024-09-11T08:20:27.162586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:20:27.162586+0000","objects_scrubbed":3852,"log_size":4760,"log_dups_size":0,"ondisk_log_size":4760,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:55.304374+0000","scrub_duration":1.512301039,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89325568,"num_objects":1363,"num_object_clones":0,"num_object_copies":4089,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13049,"num_write_kb":461184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"239'23","reported_seq":332,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.068581+0000","last_change":"2024-09-11T08:18:35.057979+0000","last_active":"2024-09-11T08:20:52.068581+0000","last_peered":"2024-09-11T08:20:52.068581+0000","last_clean":"2024-09-11T08:20:52.068581+0000","last_became_active":"2024-09-11T08:07:32.048164+0000","last_became_peered":"2024-09-11T08:07:32.048164+0000","last_unstale":"2024-09-11T08:20:52.068581+0000","last_undegraded":"2024-09-11T08:20:52.068581+0000","last_fullsized":"2024-09-11T08:20:52.068581+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'23","last_scrub_stamp":"2024-09-11T08:18:35.057874+0000","last_deep_scrub":"145'17","last_deep_scrub_stamp":"2024-09-11T08:13:08.160848+0000","last_clean_scrub_stamp":"2024-09-11T08:18:35.057874+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043775849999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":119,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.068820+0000","last_change":"2024-09-11T08:18:36.008359+0000","last_active":"2024-09-11T08:20:52.068820+0000","last_peered":"2024-09-11T08:20:52.068820+0000","last_clean":"2024-09-11T08:20:52.068820+0000","last_became_active":"2024-09-11T08:15:56.525845+0000","last_became_peered":"2024-09-11T08:15:56.525845+0000","last_unstale":"2024-09-11T08:20:52.068820+0000","last_undegraded":"2024-09-11T08:20:52.068820+0000","last_fullsized":"2024-09-11T08:20:52.068820+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:36.008191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:53.247275+0000","last_clean_scrub_stamp":"2024-09-11T08:18:36.008191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0094233570000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":312,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-11T08:20:59.876641+0000","last_change":"2024-09-11T08:20:59.876602+0000","last_active":"2024-09-11T08:20:59.876641+0000","last_peered":"2024-09-11T08:20:59.876641+0000","last_clean":"2024-09-11T08:20:59.876641+0000","last_became_active":"2024-09-11T08:07:34.057479+0000","last_became_peered":"2024-09-11T08:07:34.057479+0000","last_unstale":"2024-09-11T08:20:59.876641+0000","last_undegraded":"2024-09-11T08:20:59.876641+0000","last_fullsized":"2024-09-11T08:20:59.876641+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:59.876505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:20:59.876505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:24.461114+0000","scrub_duration":0.0064578550000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":328,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.075963+0000","last_change":"2024-09-11T08:20:21.834803+0000","last_active":"2024-09-11T08:20:52.075963+0000","last_peered":"2024-09-11T08:20:52.075963+0000","last_clean":"2024-09-11T08:20:52.075963+0000","last_became_active":"2024-09-11T08:15:56.522624+0000","last_became_peered":"2024-09-11T08:15:56.522624+0000","last_unstale":"2024-09-11T08:20:52.075963+0000","last_undegraded":"2024-09-11T08:20:52.075963+0000","last_fullsized":"2024-09-11T08:20:52.075963+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:21.834736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:20:21.834736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:22.373155+0000","scrub_duration":0.0012504840000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"297'13244","reported_seq":14675,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-11T08:21:03.657927+0000","last_change":"2024-09-11T08:21:00.497322+0000","last_active":"2024-09-11T08:21:03.657927+0000","last_peered":"2024-09-11T08:21:03.657927+0000","last_clean":"2024-09-11T08:21:03.657927+0000","last_became_active":"2024-09-11T08:19:39.780479+0000","last_became_peered":"2024-09-11T08:19:39.780479+0000","last_unstale":"2024-09-11T08:21:03.657927+0000","last_undegraded":"2024-09-11T08:21:03.657927+0000","last_fullsized":"2024-09-11T08:21:03.657927+0000","mapping_epoch":274,"log_start":"272'6898","ondisk_log_start":"272'6898","created":246,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"297'13040","last_scrub_stamp":"2024-09-11T08:21:00.497229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:21:00.497229+0000","objects_scrubbed":1638,"log_size":6346,"log_dups_size":0,"ondisk_log_size":6346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:11.290954+0000","scrub_duration":0.78454671399999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92405760,"num_objects":1410,"num_object_clones":0,"num_object_copies":4230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13244,"num_write_kb":468928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":633,"num_bytes_recovered":41484288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"239'27","reported_seq":62,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.076141+0000","last_change":"2024-09-11T08:20:08.785799+0000","last_active":"2024-09-11T08:20:52.076141+0000","last_peered":"2024-09-11T08:20:52.076141+0000","last_clean":"2024-09-11T08:20:52.076141+0000","last_became_active":"2024-09-11T08:19:29.147000+0000","last_became_peered":"2024-09-11T08:19:29.147000+0000","last_unstale":"2024-09-11T08:20:52.076141+0000","last_undegraded":"2024-09-11T08:20:52.076141+0000","last_fullsized":"2024-09-11T08:20:52.076141+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'27","last_scrub_stamp":"2024-09-11T08:20:08.785703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:20:08.785703+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:18.100187+0000","scrub_duration":0.005828789,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":125,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.070545+0000","last_change":"2024-09-11T08:18:39.012762+0000","last_active":"2024-09-11T08:20:52.070545+0000","last_peered":"2024-09-11T08:20:52.070545+0000","last_clean":"2024-09-11T08:20:52.070545+0000","last_became_active":"2024-09-11T08:15:56.515328+0000","last_became_peered":"2024-09-11T08:15:56.515328+0000","last_unstale":"2024-09-11T08:20:52.070545+0000","last_undegraded":"2024-09-11T08:20:52.070545+0000","last_fullsized":"2024-09-11T08:20:52.070545+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-11T08:18:39.012616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:18:39.012616+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025254959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"209'11","reported_seq":184,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-11T08:20:58.131376+0000","last_change":"2024-09-11T08:20:51.158341+0000","last_active":"2024-09-11T08:20:58.131376+0000","last_peered":"2024-09-11T08:20:58.131376+0000","last_clean":"2024-09-11T08:20:58.131376+0000","last_became_active":"2024-09-11T08:15:56.524513+0000","last_became_peered":"2024-09-11T08:15:56.524513+0000","last_unstale":"2024-09-11T08:20:58.131376+0000","last_undegraded":"2024-09-11T08:20:58.131376+0000","last_fullsized":"2024-09-11T08:20:58.131376+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"209'11","last_scrub_stamp":"2024-09-11T08:20:51.158199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:20:51.158199+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:17.301119+0000","scrub_duration":0.015653383,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":36,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.076766+0000","last_change":"2024-09-11T08:19:29.164149+0000","last_active":"2024-09-11T08:20:52.076766+0000","last_peered":"2024-09-11T08:20:52.076766+0000","last_clean":"2024-09-11T08:20:52.076766+0000","last_became_active":"2024-09-11T08:19:29.163872+0000","last_became_peered":"2024-09-11T08:19:29.163872+0000","last_unstale":"2024-09-11T08:20:52.076766+0000","last_undegraded":"2024-09-11T08:20:52.076766+0000","last_fullsized":"2024-09-11T08:20:52.076766+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:40.294136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:03.484755+0000","last_clean_scrub_stamp":"2024-09-11T08:18:40.294136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":346,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.073564+0000","last_change":"2024-09-11T08:20:24.921477+0000","last_active":"2024-09-11T08:20:52.073564+0000","last_peered":"2024-09-11T08:20:52.073564+0000","last_clean":"2024-09-11T08:20:52.073564+0000","last_became_active":"2024-09-11T08:19:29.193317+0000","last_became_peered":"2024-09-11T08:19:29.193317+0000","last_unstale":"2024-09-11T08:20:52.073564+0000","last_undegraded":"2024-09-11T08:20:52.073564+0000","last_fullsized":"2024-09-11T08:20:52.073564+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:24.921042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:20:24.921042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:41.162286+0000","scrub_duration":0.0069516710000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"239'15","reported_seq":369,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:54.764486+0000","last_change":"2024-09-11T08:20:54.764391+0000","last_active":"2024-09-11T08:20:54.764486+0000","last_peered":"2024-09-11T08:20:54.764486+0000","last_clean":"2024-09-11T08:20:54.764486+0000","last_became_active":"2024-09-11T08:15:56.526325+0000","last_became_peered":"2024-09-11T08:15:56.526325+0000","last_unstale":"2024-09-11T08:20:54.764486+0000","last_undegraded":"2024-09-11T08:20:54.764486+0000","last_fullsized":"2024-09-11T08:20:54.764486+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'15","last_scrub_stamp":"2024-09-11T08:20:54.764293+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-09-11T08:09:03.418457+0000","last_clean_scrub_stamp":"2024-09-11T08:20:54.764293+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:05.029319+0000","scrub_duration":0.0039192439999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":36,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.077190+0000","last_change":"2024-09-11T08:19:29.147871+0000","last_active":"2024-09-11T08:20:52.077190+0000","last_peered":"2024-09-11T08:20:52.077190+0000","last_clean":"2024-09-11T08:20:52.077190+0000","last_became_active":"2024-09-11T08:19:29.143925+0000","last_became_peered":"2024-09-11T08:19:29.143925+0000","last_unstale":"2024-09-11T08:20:52.077190+0000","last_undegraded":"2024-09-11T08:20:52.077190+0000","last_fullsized":"2024-09-11T08:20:52.077190+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:32.284664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:53.548967+0000","last_clean_scrub_stamp":"2024-09-11T08:18:32.284664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":244,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.069945+0000","last_change":"2024-09-11T08:18:29.927008+0000","last_active":"2024-09-11T08:20:52.069945+0000","last_peered":"2024-09-11T08:20:52.069945+0000","last_clean":"2024-09-11T08:20:52.069945+0000","last_became_active":"2024-09-11T08:14:24.790678+0000","last_became_peered":"2024-09-11T08:14:24.790678+0000","last_unstale":"2024-09-11T08:20:52.069945+0000","last_undegraded":"2024-09-11T08:20:52.069945+0000","last_fullsized":"2024-09-11T08:20:52.069945+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:29.926950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:18:29.926950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020333719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"239'19","reported_seq":370,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.076545+0000","last_change":"2024-09-11T08:20:14.603209+0000","last_active":"2024-09-11T08:20:52.076545+0000","last_peered":"2024-09-11T08:20:52.076545+0000","last_clean":"2024-09-11T08:20:52.076545+0000","last_became_active":"2024-09-11T08:15:56.514793+0000","last_became_peered":"2024-09-11T08:15:56.514793+0000","last_unstale":"2024-09-11T08:20:52.076545+0000","last_undegraded":"2024-09-11T08:20:52.076545+0000","last_fullsized":"2024-09-11T08:20:52.076545+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'19","last_scrub_stamp":"2024-09-11T08:20:14.603118+0000","last_deep_scrub":"84'13","last_deep_scrub_stamp":"2024-09-11T08:08:57.835634+0000","last_clean_scrub_stamp":"2024-09-11T08:20:14.603118+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:35.007041+0000","scrub_duration":0.0042720240000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":277,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.069777+0000","last_change":"2024-09-11T08:20:20.819257+0000","last_active":"2024-09-11T08:20:52.069777+0000","last_peered":"2024-09-11T08:20:52.069777+0000","last_clean":"2024-09-11T08:20:52.069777+0000","last_became_active":"2024-09-11T08:14:24.786021+0000","last_became_peered":"2024-09-11T08:14:24.786021+0000","last_unstale":"2024-09-11T08:20:52.069777+0000","last_undegraded":"2024-09-11T08:20:52.069777+0000","last_fullsized":"2024-09-11T08:20:52.069777+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:20.819154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:30.471620+0000","last_clean_scrub_stamp":"2024-09-11T08:20:20.819154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:39.435896+0000","scrub_duration":0.0018806490000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":330,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.076964+0000","last_change":"2024-09-11T08:20:22.889011+0000","last_active":"2024-09-11T08:20:52.076964+0000","last_peered":"2024-09-11T08:20:52.076964+0000","last_clean":"2024-09-11T08:20:52.076964+0000","last_became_active":"2024-09-11T08:15:56.525717+0000","last_became_peered":"2024-09-11T08:15:56.525717+0000","last_unstale":"2024-09-11T08:20:52.076964+0000","last_undegraded":"2024-09-11T08:20:52.076964+0000","last_fullsized":"2024-09-11T08:20:52.076964+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:22.888936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:04.370734+0000","last_clean_scrub_stamp":"2024-09-11T08:20:22.888936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:52.204024+0000","scrub_duration":0.01485916,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":330,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.079753+0000","last_change":"2024-09-11T08:20:15.554813+0000","last_active":"2024-09-11T08:20:52.079753+0000","last_peered":"2024-09-11T08:20:52.079753+0000","last_clean":"2024-09-11T08:20:52.079753+0000","last_became_active":"2024-09-11T08:15:56.525120+0000","last_became_peered":"2024-09-11T08:15:56.525120+0000","last_unstale":"2024-09-11T08:20:52.079753+0000","last_undegraded":"2024-09-11T08:20:52.079753+0000","last_fullsized":"2024-09-11T08:20:52.079753+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:15.554724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:54.184372+0000","last_clean_scrub_stamp":"2024-09-11T08:20:15.554724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:16.777929+0000","scrub_duration":0.0033311510000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"239'15","reported_seq":46,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.077264+0000","last_change":"2024-09-11T08:19:29.168879+0000","last_active":"2024-09-11T08:20:52.077264+0000","last_peered":"2024-09-11T08:20:52.077264+0000","last_clean":"2024-09-11T08:20:52.077264+0000","last_became_active":"2024-09-11T08:19:29.153740+0000","last_became_peered":"2024-09-11T08:19:29.153740+0000","last_unstale":"2024-09-11T08:20:52.077264+0000","last_undegraded":"2024-09-11T08:20:52.077264+0000","last_fullsized":"2024-09-11T08:20:52.077264+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'15","last_scrub_stamp":"2024-09-11T08:18:44.371100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:18:44.371100+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047860990000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":323,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.076415+0000","last_change":"2024-09-11T08:18:28.168735+0000","last_active":"2024-09-11T08:20:52.076415+0000","last_peered":"2024-09-11T08:20:52.076415+0000","last_clean":"2024-09-11T08:20:52.076415+0000","last_became_active":"2024-09-11T08:15:56.513278+0000","last_became_peered":"2024-09-11T08:15:56.513278+0000","last_unstale":"2024-09-11T08:20:52.076415+0000","last_undegraded":"2024-09-11T08:20:52.076415+0000","last_fullsized":"2024-09-11T08:20:52.076415+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:28.168679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:52.661591+0000","last_clean_scrub_stamp":"2024-09-11T08:18:28.168679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031651969999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"239'13","reported_seq":481,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-11T08:20:58.132288+0000","last_change":"2024-09-11T08:20:14.871683+0000","last_active":"2024-09-11T08:20:58.132288+0000","last_peered":"2024-09-11T08:20:58.132288+0000","last_clean":"2024-09-11T08:20:58.132288+0000","last_became_active":"2024-09-11T08:17:37.558224+0000","last_became_peered":"2024-09-11T08:17:37.558224+0000","last_unstale":"2024-09-11T08:20:58.132288+0000","last_undegraded":"2024-09-11T08:20:58.132288+0000","last_fullsized":"2024-09-11T08:20:58.132288+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'13","last_scrub_stamp":"2024-09-11T08:20:14.871608+0000","last_deep_scrub":"222'12","last_deep_scrub_stamp":"2024-09-11T08:16:52.592856+0000","last_clean_scrub_stamp":"2024-09-11T08:20:14.871608+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:29.721925+0000","scrub_duration":0.0023994300000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":311,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:54.449926+0000","last_change":"2024-09-11T08:20:54.449873+0000","last_active":"2024-09-11T08:20:54.449926+0000","last_peered":"2024-09-11T08:20:54.449926+0000","last_clean":"2024-09-11T08:20:54.449926+0000","last_became_active":"2024-09-11T08:19:29.147682+0000","last_became_peered":"2024-09-11T08:19:29.147682+0000","last_unstale":"2024-09-11T08:20:54.449926+0000","last_undegraded":"2024-09-11T08:20:54.449926+0000","last_fullsized":"2024-09-11T08:20:54.449926+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:54.449786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:53.175994+0000","last_clean_scrub_stamp":"2024-09-11T08:20:54.449786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:58.294035+0000","scrub_duration":0.0015389500000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"239'20","reported_seq":169,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.079687+0000","last_change":"2024-09-11T08:20:13.568730+0000","last_active":"2024-09-11T08:20:52.079687+0000","last_peered":"2024-09-11T08:20:52.079687+0000","last_clean":"2024-09-11T08:20:52.079687+0000","last_became_active":"2024-09-11T08:17:37.554357+0000","last_became_peered":"2024-09-11T08:17:37.554357+0000","last_unstale":"2024-09-11T08:20:52.079687+0000","last_undegraded":"2024-09-11T08:20:52.079687+0000","last_fullsized":"2024-09-11T08:20:52.079687+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'20","last_scrub_stamp":"2024-09-11T08:20:13.568619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:20:13.568619+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:17.454653+0000","scrub_duration":0.0099769119999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":119,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.068886+0000","last_change":"2024-09-11T08:20:17.777970+0000","last_active":"2024-09-11T08:20:52.068886+0000","last_peered":"2024-09-11T08:20:52.068886+0000","last_clean":"2024-09-11T08:20:52.068886+0000","last_became_active":"2024-09-11T08:15:56.518515+0000","last_became_peered":"2024-09-11T08:15:56.518515+0000","last_unstale":"2024-09-11T08:20:52.068886+0000","last_undegraded":"2024-09-11T08:20:52.068886+0000","last_fullsized":"2024-09-11T08:20:52.068886+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:17.777890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:20:17.777890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:30.228005+0000","scrub_duration":0.0086416670000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"271'10","reported_seq":69,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-11T08:20:58.137375+0000","last_change":"2024-09-11T08:20:16.922523+0000","last_active":"2024-09-11T08:20:58.137375+0000","last_peered":"2024-09-11T08:20:58.137375+0000","last_clean":"2024-09-11T08:20:58.137375+0000","last_became_active":"2024-09-11T08:19:29.165183+0000","last_became_peered":"2024-09-11T08:19:29.165183+0000","last_unstale":"2024-09-11T08:20:58.137375+0000","last_undegraded":"2024-09-11T08:20:58.137375+0000","last_fullsized":"2024-09-11T08:20:58.137375+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'10","last_scrub_stamp":"2024-09-11T08:20:16.922384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:20:16.922384+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:24.103214+0000","scrub_duration":0.0084094979999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":317,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.076258+0000","last_change":"2024-09-11T08:20:23.600147+0000","last_active":"2024-09-11T08:20:52.076258+0000","last_peered":"2024-09-11T08:20:52.076258+0000","last_clean":"2024-09-11T08:20:52.076258+0000","last_became_active":"2024-09-11T08:15:56.513949+0000","last_became_peered":"2024-09-11T08:15:56.513949+0000","last_unstale":"2024-09-11T08:20:52.076258+0000","last_undegraded":"2024-09-11T08:20:52.076258+0000","last_fullsized":"2024-09-11T08:20:52.076258+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:23.600056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:20:23.600056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:37.601061+0000","scrub_duration":0.003973341,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"239'19","reported_seq":329,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.069539+0000","last_change":"2024-09-11T08:20:09.525188+0000","last_active":"2024-09-11T08:20:52.069539+0000","last_peered":"2024-09-11T08:20:52.069539+0000","last_clean":"2024-09-11T08:20:52.069539+0000","last_became_active":"2024-09-11T08:07:32.047981+0000","last_became_peered":"2024-09-11T08:07:32.047981+0000","last_unstale":"2024-09-11T08:20:52.069539+0000","last_undegraded":"2024-09-11T08:20:52.069539+0000","last_fullsized":"2024-09-11T08:20:52.069539+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'19","last_scrub_stamp":"2024-09-11T08:20:09.525095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:20:09.525095+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:38.535885+0000","scrub_duration":0.0063068040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":327,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.070859+0000","last_change":"2024-09-11T08:20:27.515065+0000","last_active":"2024-09-11T08:20:52.070859+0000","last_peered":"2024-09-11T08:20:52.070859+0000","last_clean":"2024-09-11T08:20:52.070859+0000","last_became_active":"2024-09-11T08:15:56.523574+0000","last_became_peered":"2024-09-11T08:15:56.523574+0000","last_unstale":"2024-09-11T08:20:52.070859+0000","last_undegraded":"2024-09-11T08:20:52.070859+0000","last_fullsized":"2024-09-11T08:20:52.070859+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:27.514983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:20:27.514983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:43.256061+0000","scrub_duration":0.019346374,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":326,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.079701+0000","last_change":"2024-09-11T08:19:29.142149+0000","last_active":"2024-09-11T08:20:52.079701+0000","last_peered":"2024-09-11T08:20:52.079701+0000","last_clean":"2024-09-11T08:20:52.079701+0000","last_became_active":"2024-09-11T08:19:29.141857+0000","last_became_peered":"2024-09-11T08:19:29.141857+0000","last_unstale":"2024-09-11T08:20:52.079701+0000","last_undegraded":"2024-09-11T08:20:52.079701+0000","last_fullsized":"2024-09-11T08:20:52.079701+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:41.689207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:18:41.689207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0076107090000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":111,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.074612+0000","last_change":"2024-09-11T08:20:11.653639+0000","last_active":"2024-09-11T08:20:52.074612+0000","last_peered":"2024-09-11T08:20:52.074612+0000","last_clean":"2024-09-11T08:20:52.074612+0000","last_became_active":"2024-09-11T08:19:29.180111+0000","last_became_peered":"2024-09-11T08:19:29.180111+0000","last_unstale":"2024-09-11T08:20:52.074612+0000","last_undegraded":"2024-09-11T08:20:52.074612+0000","last_fullsized":"2024-09-11T08:20:52.074612+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:11.653502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:20:11.653502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:18.440265+0000","scrub_duration":0.017208757000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"239'15","reported_seq":340,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.079944+0000","last_change":"2024-09-11T08:19:40.867691+0000","last_active":"2024-09-11T08:20:52.079944+0000","last_peered":"2024-09-11T08:20:52.079944+0000","last_clean":"2024-09-11T08:20:52.079944+0000","last_became_active":"2024-09-11T08:07:32.049368+0000","last_became_peered":"2024-09-11T08:07:32.049368+0000","last_unstale":"2024-09-11T08:20:52.079944+0000","last_undegraded":"2024-09-11T08:20:52.079944+0000","last_fullsized":"2024-09-11T08:20:52.079944+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'15","last_scrub_stamp":"2024-09-11T08:19:40.867599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:19:40.867599+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023960169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":119,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.069747+0000","last_change":"2024-09-11T08:18:28.949104+0000","last_active":"2024-09-11T08:20:52.069747+0000","last_peered":"2024-09-11T08:20:52.069747+0000","last_clean":"2024-09-11T08:20:52.069747+0000","last_became_active":"2024-09-11T08:15:56.521570+0000","last_became_peered":"2024-09-11T08:15:56.521570+0000","last_unstale":"2024-09-11T08:20:52.069747+0000","last_undegraded":"2024-09-11T08:20:52.069747+0000","last_fullsized":"2024-09-11T08:20:52.069747+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:28.949027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:18:28.949027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030272900000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":91,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.075632+0000","last_change":"2024-09-11T08:20:16.552108+0000","last_active":"2024-09-11T08:20:52.075632+0000","last_peered":"2024-09-11T08:20:52.075632+0000","last_clean":"2024-09-11T08:20:52.075632+0000","last_became_active":"2024-09-11T08:17:37.557645+0000","last_became_peered":"2024-09-11T08:17:37.557645+0000","last_unstale":"2024-09-11T08:20:52.075632+0000","last_undegraded":"2024-09-11T08:20:52.075632+0000","last_fullsized":"2024-09-11T08:20:52.075632+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:16.551797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:01.218475+0000","last_clean_scrub_stamp":"2024-09-11T08:20:16.551797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:20.669628+0000","scrub_duration":0.0026502240000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":310,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.074503+0000","last_change":"2024-09-11T08:19:29.198728+0000","last_active":"2024-09-11T08:20:52.074503+0000","last_peered":"2024-09-11T08:20:52.074503+0000","last_clean":"2024-09-11T08:20:52.074503+0000","last_became_active":"2024-09-11T08:19:29.198224+0000","last_became_peered":"2024-09-11T08:19:29.198224+0000","last_unstale":"2024-09-11T08:20:52.074503+0000","last_undegraded":"2024-09-11T08:20:52.074503+0000","last_fullsized":"2024-09-11T08:20:52.074503+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:20.077741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:09.973943+0000","last_clean_scrub_stamp":"2024-09-11T08:18:20.077741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035943939999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"239'19","reported_seq":387,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.076481+0000","last_change":"2024-09-11T08:20:23.842392+0000","last_active":"2024-09-11T08:20:52.076481+0000","last_peered":"2024-09-11T08:20:52.076481+0000","last_clean":"2024-09-11T08:20:52.076481+0000","last_became_active":"2024-09-11T08:15:56.520058+0000","last_became_peered":"2024-09-11T08:15:56.520058+0000","last_unstale":"2024-09-11T08:20:52.076481+0000","last_undegraded":"2024-09-11T08:20:52.076481+0000","last_fullsized":"2024-09-11T08:20:52.076481+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'19","last_scrub_stamp":"2024-09-11T08:20:23.842243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:20:23.842243+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:25.267175+0000","scrub_duration":0.013871632,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":36,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.077258+0000","last_change":"2024-09-11T08:19:29.144075+0000","last_active":"2024-09-11T08:20:52.077258+0000","last_peered":"2024-09-11T08:20:52.077258+0000","last_clean":"2024-09-11T08:20:52.077258+0000","last_became_active":"2024-09-11T08:19:29.143511+0000","last_became_peered":"2024-09-11T08:19:29.143511+0000","last_unstale":"2024-09-11T08:20:52.077258+0000","last_undegraded":"2024-09-11T08:20:52.077258+0000","last_fullsized":"2024-09-11T08:20:52.077258+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:22.308183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:18:22.308183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012202396000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":119,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.067907+0000","last_change":"2024-09-11T08:18:30.917483+0000","last_active":"2024-09-11T08:20:52.067907+0000","last_peered":"2024-09-11T08:20:52.067907+0000","last_clean":"2024-09-11T08:20:52.067907+0000","last_became_active":"2024-09-11T08:15:56.523728+0000","last_became_peered":"2024-09-11T08:15:56.523728+0000","last_unstale":"2024-09-11T08:20:52.067907+0000","last_undegraded":"2024-09-11T08:20:52.067907+0000","last_fullsized":"2024-09-11T08:20:52.067907+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:30.917391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:11:34.159053+0000","last_clean_scrub_stamp":"2024-09-11T08:18:30.917391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015403719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":291,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.076138+0000","last_change":"2024-09-11T08:20:16.004116+0000","last_active":"2024-09-11T08:20:52.076138+0000","last_peered":"2024-09-11T08:20:52.076138+0000","last_clean":"2024-09-11T08:20:52.076138+0000","last_became_active":"2024-09-11T08:07:36.633368+0000","last_became_peered":"2024-09-11T08:07:36.633368+0000","last_unstale":"2024-09-11T08:20:52.076138+0000","last_undegraded":"2024-09-11T08:20:52.076138+0000","last_fullsized":"2024-09-11T08:20:52.076138+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:16.003821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:46.489220+0000","last_clean_scrub_stamp":"2024-09-11T08:20:16.003821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:22.490239+0000","scrub_duration":0.010466069,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"239'8","reported_seq":307,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.077684+0000","last_change":"2024-09-11T08:20:16.957169+0000","last_active":"2024-09-11T08:20:52.077684+0000","last_peered":"2024-09-11T08:20:52.077684+0000","last_clean":"2024-09-11T08:20:52.077684+0000","last_became_active":"2024-09-11T08:07:32.045728+0000","last_became_peered":"2024-09-11T08:07:32.045728+0000","last_unstale":"2024-09-11T08:20:52.077684+0000","last_undegraded":"2024-09-11T08:20:52.077684+0000","last_fullsized":"2024-09-11T08:20:52.077684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'8","last_scrub_stamp":"2024-09-11T08:20:16.957056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:20:16.957056+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:28.504779+0000","scrub_duration":0.0030711110000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":326,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.076207+0000","last_change":"2024-09-11T08:20:12.651491+0000","last_active":"2024-09-11T08:20:52.076207+0000","last_peered":"2024-09-11T08:20:52.076207+0000","last_clean":"2024-09-11T08:20:52.076207+0000","last_became_active":"2024-09-11T08:19:29.161558+0000","last_became_peered":"2024-09-11T08:19:29.161558+0000","last_unstale":"2024-09-11T08:20:52.076207+0000","last_undegraded":"2024-09-11T08:20:52.076207+0000","last_fullsized":"2024-09-11T08:20:52.076207+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:12.651408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:20:12.651408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:29.728488+0000","scrub_duration":0.0071009350000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":349,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.072188+0000","last_change":"2024-09-11T08:19:42.654620+0000","last_active":"2024-09-11T08:20:52.072188+0000","last_peered":"2024-09-11T08:20:52.072188+0000","last_clean":"2024-09-11T08:20:52.072188+0000","last_became_active":"2024-09-11T08:15:56.528607+0000","last_became_peered":"2024-09-11T08:15:56.528607+0000","last_unstale":"2024-09-11T08:20:52.072188+0000","last_undegraded":"2024-09-11T08:20:52.072188+0000","last_fullsized":"2024-09-11T08:20:52.072188+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:19:42.654538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:19:42.654538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:08.778702+0000","scrub_duration":0.001327426,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":119,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.068561+0000","last_change":"2024-09-11T08:20:18.827796+0000","last_active":"2024-09-11T08:20:52.068561+0000","last_peered":"2024-09-11T08:20:52.068561+0000","last_clean":"2024-09-11T08:20:52.068561+0000","last_became_active":"2024-09-11T08:15:56.521488+0000","last_became_peered":"2024-09-11T08:15:56.521488+0000","last_unstale":"2024-09-11T08:20:52.068561+0000","last_undegraded":"2024-09-11T08:20:52.068561+0000","last_fullsized":"2024-09-11T08:20:52.068561+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:18.827709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:20:18.827709+0000","last_clean_scrub_stamp":"2024-09-11T08:20:18.827709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:33.930866+0000","scrub_duration":0.014060774999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":331,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-11T08:21:03.627890+0000","last_change":"2024-09-11T08:21:03.627814+0000","last_active":"2024-09-11T08:21:03.627890+0000","last_peered":"2024-09-11T08:21:03.627890+0000","last_clean":"2024-09-11T08:21:03.627890+0000","last_became_active":"2024-09-11T08:19:29.153017+0000","last_became_peered":"2024-09-11T08:19:29.153017+0000","last_unstale":"2024-09-11T08:21:03.627890+0000","last_undegraded":"2024-09-11T08:21:03.627890+0000","last_fullsized":"2024-09-11T08:21:03.627890+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:03.627721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:50.585566+0000","last_clean_scrub_stamp":"2024-09-11T08:21:03.627721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:31.922693+0000","scrub_duration":0.003783294,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":91,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.075078+0000","last_change":"2024-09-11T08:20:49.725564+0000","last_active":"2024-09-11T08:20:52.075078+0000","last_peered":"2024-09-11T08:20:52.075078+0000","last_clean":"2024-09-11T08:20:52.075078+0000","last_became_active":"2024-09-11T08:17:37.554262+0000","last_became_peered":"2024-09-11T08:17:37.554262+0000","last_unstale":"2024-09-11T08:20:52.075078+0000","last_undegraded":"2024-09-11T08:20:52.075078+0000","last_fullsized":"2024-09-11T08:20:52.075078+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:49.725505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:20:49.725505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:51.072704+0000","scrub_duration":0.0012049350000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"288'24","reported_seq":404,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.072125+0000","last_change":"2024-09-11T08:20:28.885398+0000","last_active":"2024-09-11T08:20:52.072125+0000","last_peered":"2024-09-11T08:20:52.072125+0000","last_clean":"2024-09-11T08:20:52.072125+0000","last_became_active":"2024-09-11T08:19:29.145221+0000","last_became_peered":"2024-09-11T08:19:29.145221+0000","last_unstale":"2024-09-11T08:20:52.072125+0000","last_undegraded":"2024-09-11T08:20:52.072125+0000","last_fullsized":"2024-09-11T08:20:52.072125+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'24","last_scrub_stamp":"2024-09-11T08:20:28.885295+0000","last_deep_scrub":"288'24","last_deep_scrub_stamp":"2024-09-11T08:20:28.885295+0000","last_clean_scrub_stamp":"2024-09-11T08:20:28.885295+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:52.041380+0000","scrub_duration":0.0080800170000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":147,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-11T08:20:52.071837+0000","last_change":"2024-09-11T08:20:20.817249+0000","last_active":"2024-09-11T08:20:52.071837+0000","last_peered":"2024-09-11T08:20:52.071837+0000","last_clean":"2024-09-11T08:20:52.071837+0000","last_became_active":"2024-09-11T08:15:08.769133+0000","last_became_peered":"2024-09-11T08:15:08.769133+0000","last_unstale":"2024-09-11T08:20:52.071837+0000","last_undegraded":"2024-09-11T08:20:52.071837+0000","last_fullsized":"2024-09-11T08:20:52.071837+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:20.817194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:18:39.206016+0000","last_clean_scrub_stamp":"2024-09-11T08:20:20.817194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:48.465266+0000","scrub_duration":0.0018451699999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":1471283224,"num_objects":22451,"num_object_clones":0,"num_object_copies":67353,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22451,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":208757,"num_write_kb":7398593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16213,"num_bytes_recovered":1062535168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4749737984,"data_stored":4749721696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87354,"ondisk_log_size":87354,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1125,"num_read_kb":882,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":4676,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3425,"internal_metadata":0},"log_size":753,"ondisk_log_size":753,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":404,"num_read_kb":399,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":168,"seq":721554505809,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1171604,"kb_used_data":849932,"kb_used_omap":7,"kb_used_meta":321656,"kb_avail":92573548,"statfs":{"total":95995035648,"available":94795313152,"internally_reserved":0,"allocated":870330368,"data_stored":869944555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8027,"internal_metadata":329375909},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:20:30 2024","interfaces":[{"interface":"back","average":{"1min":5.3049999999999997,"5min":1.929,"15min":1.1930000000000001},"min":{"1min":0.22800000000000001,"5min":0.223,"15min":0.223},"max":{"1min":67.409999999999997,"5min":67.409999999999997,"15min":67.409999999999997},"last":1.0580000000000001},{"interface":"front","average":{"1min":5.0330000000000004,"5min":1.776,"15min":1.2549999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":66.959999999999994,"5min":66.959999999999994,"15min":66.959999999999994},"last":1.2210000000000001}]},{"osd":1,"last update":"Wed Sep 11 08:20:30 2024","interfaces":[{"interface":"back","average":{"1min":5.819,"5min":3.0859999999999999,"15min":2.3260000000000001},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":66.971000000000004,"5min":66.971000000000004,"15min":66.971000000000004},"last":1.1120000000000001},{"interface":"front","average":{"1min":5.3339999999999996,"5min":2.746,"15min":2.2789999999999999},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":67.185000000000002,"5min":67.185000000000002,"15min":67.185000000000002},"last":6.798}]},{"osd":2,"last update":"Wed Sep 11 08:20:30 2024","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":0.91400000000000003,"15min":0.999},"min":{"1min":0.25700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":2.9729999999999999,"5min":6.444,"15min":6.444},"last":1.548},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.0509999999999999,"15min":0.83099999999999996},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":4.5540000000000003,"5min":6.3460000000000001,"15min":6.3460000000000001},"last":1.643}]},{"osd":3,"last update":"Wed Sep 11 08:20:30 2024","interfaces":[{"interface":"back","average":{"1min":1.3440000000000001,"5min":1.3560000000000001,"15min":1.167},"min":{"1min":0.311,"5min":0.251,"15min":0.251},"max":{"1min":5.2149999999999999,"5min":21.460000000000001,"15min":21.460000000000001},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.96499999999999997,"5min":1.2789999999999999,"15min":1.3759999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":3.0019999999999998,"5min":21.440000000000001,"15min":21.440000000000001},"last":1.254}]},{"osd":4,"last update":"Wed Sep 11 08:20:30 2024","interfaces":[{"interface":"back","average":{"1min":1.448,"5min":0.88400000000000001,"15min":0.64500000000000002},"min":{"1min":0.25800000000000001,"5min":0.252,"15min":0.252},"max":{"1min":4.5289999999999999,"5min":4.5289999999999999,"15min":4.5289999999999999},"last":0.30299999999999999},{"interface":"front","average":{"1min":1.375,"5min":0.82299999999999995,"15min":0.77700000000000002},"min":{"1min":0.27200000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":5.2649999999999997,"5min":5.2649999999999997,"15min":5.2649999999999997},"last":0.376}]},{"osd":5,"last update":"Wed Sep 11 08:20:30 2024","interfaces":[{"interface":"back","average":{"1min":1.5640000000000001,"5min":1.764,"15min":1.1200000000000001},"min":{"1min":0.28699999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":5.1559999999999997,"5min":65.866,"15min":65.866},"last":0.46800000000000003},{"interface":"front","average":{"1min":1.9019999999999999,"5min":1.8280000000000001,"15min":1.109},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":6.3289999999999997,"5min":66.019000000000005,"15min":66.019000000000005},"last":0.34499999999999997}]},{"osd":6,"last update":"Wed Sep 11 08:20:30 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.73999999999999999,"15min":0.77500000000000002},"min":{"1min":0.34799999999999998,"5min":0.248,"15min":0.248},"max":{"1min":3.0259999999999998,"5min":6.0609999999999999,"15min":6.0609999999999999},"last":0.40000000000000002},{"interface":"front","average":{"1min":1.4930000000000001,"5min":0.93500000000000005,"15min":0.83799999999999997},"min":{"1min":0.379,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":4.907,"5min":4.907,"15min":4.907},"last":0.434}]}]},{"osd":6,"up_from":45,"seq":193273528492,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":599872,"kb_used_data":290040,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93145280,"statfs":{"total":95995035648,"available":95380766720,"internally_reserved":0,"allocated":297000960,"data_stored":296635426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:20:18 2024","interfaces":[{"interface":"back","average":{"1min":2.8820000000000001,"5min":1.645,"15min":1.3740000000000001},"min":{"1min":0.245,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":25.494,"5min":25.494,"15min":45.066000000000003},"last":0.47999999999999998},{"interface":"front","average":{"1min":2.0790000000000002,"5min":1.641,"15min":1.337},"min":{"1min":0.245,"5min":0.224,"15min":0.20599999999999999},"max":{"1min":25.48,"5min":25.48,"15min":43.899000000000001},"last":0.95199999999999996}]},{"osd":1,"last update":"Wed Sep 11 08:20:18 2024","interfaces":[{"interface":"back","average":{"1min":1.651,"5min":1.512,"15min":1.1759999999999999},"min":{"1min":0.23699999999999999,"5min":0.214,"15min":0.214},"max":{"1min":3.9660000000000002,"5min":20.321000000000002,"15min":44.658000000000001},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.64900000000000002,"5min":1.4850000000000001,"15min":1.218},"min":{"1min":0.252,"5min":0.22800000000000001,"15min":0.22600000000000001},"max":{"1min":2.641,"5min":19.652000000000001,"15min":44.694000000000003},"last":0.97999999999999998}]},{"osd":2,"last update":"Wed Sep 11 08:20:18 2024","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":0.90600000000000003,"15min":0.77800000000000002},"min":{"1min":0.29499999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":3.8759999999999999,"5min":5.9770000000000003,"15min":5.9770000000000003},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.78500000000000003,"5min":1.1879999999999999,"15min":0.97699999999999998},"min":{"1min":0.29499999999999998,"5min":0.26800000000000002,"15min":0.23300000000000001},"max":{"1min":2.089,"5min":7.4809999999999999,"15min":13.346},"last":0.252}]},{"osd":3,"last update":"Wed Sep 11 08:20:18 2024","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":0.96399999999999997,"15min":1.1080000000000001},"min":{"1min":0.28199999999999997,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":3.2410000000000001,"5min":4.2889999999999997,"15min":36.061},"last":5.2969999999999997},{"interface":"front","average":{"1min":0.92500000000000004,"5min":1.0289999999999999,"15min":1.117},"min":{"1min":0.29199999999999998,"5min":0.28999999999999998,"15min":0.26800000000000002},"max":{"1min":3.2559999999999998,"5min":5.5720000000000001,"15min":36.093000000000004},"last":5.2519999999999998}]},{"osd":4,"last update":"Wed Sep 11 08:20:18 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.69999999999999996,"15min":0.51700000000000002},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20200000000000001},"max":{"1min":1.6299999999999999,"5min":4.492,"15min":4.492},"last":0.318},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.69199999999999995,"15min":0.54300000000000004},"min":{"1min":0.23799999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.609,"5min":4.468,"15min":4.468},"last":0.34100000000000003}]},{"osd":5,"last update":"Wed Sep 11 08:20:18 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.97599999999999998,"15min":0.67800000000000005},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.6559999999999999,"5min":10.49,"15min":10.49},"last":0.63},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.877,"15min":0.63400000000000001},"min":{"1min":0.29199999999999998,"5min":0.28100000000000003,"15min":0.22600000000000001},"max":{"1min":4.4329999999999998,"5min":10.468,"15min":10.468},"last":0.57299999999999995}]},{"osd":7,"last update":"Wed Sep 11 08:20:27 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.77200000000000002,"15min":1.0209999999999999},"min":{"1min":0.254,"5min":0.253,"15min":0.253},"max":{"1min":1.986,"5min":8.4580000000000002,"15min":8.4580000000000002},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.435,"5min":0.63400000000000001,"15min":0.81200000000000006},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.83999999999999997,"5min":3.593,"15min":3.593},"last":1.909}]}]},{"osd":5,"up_from":38,"seq":163208757425,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1431872,"kb_used_data":833592,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":92313280,"statfs":{"total":95995035648,"available":94528798720,"internally_reserved":0,"allocated":853598208,"data_stored":853231605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:20:58 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":2.0619999999999998,"15min":1.3},"min":{"1min":0.28599999999999998,"5min":0.184,"15min":0.184},"max":{"1min":16.652000000000001,"5min":92.924000000000007,"15min":92.924000000000007},"last":2.3759999999999999},{"interface":"front","average":{"1min":1.329,"5min":2.246,"15min":1.359},"min":{"1min":0.25,"5min":0.246,"15min":0.20399999999999999},"max":{"1min":3.8679999999999999,"5min":93.173000000000002,"15min":93.173000000000002},"last":5.4950000000000001}]},{"osd":1,"last update":"Wed Sep 11 08:20:58 2024","interfaces":[{"interface":"back","average":{"1min":2.6040000000000001,"5min":2.3090000000000002,"15min":1.6000000000000001},"min":{"1min":0.32900000000000001,"5min":0.22,"15min":0.22},"max":{"1min":16.265000000000001,"5min":96.899000000000001,"15min":96.899000000000001},"last":0.46400000000000002},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.9390000000000001,"15min":1.3360000000000001},"min":{"1min":0.25700000000000001,"5min":0.248,"15min":0.22700000000000001},"max":{"1min":2.4700000000000002,"5min":92.337000000000003,"15min":92.337000000000003},"last":0.88200000000000001}]},{"osd":2,"last update":"Wed Sep 11 08:20:58 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.0569999999999999,"15min":1.0620000000000001},"min":{"1min":0.35599999999999998,"5min":0.255,"15min":0.255},"max":{"1min":6.0860000000000003,"5min":15.622,"15min":16.018000000000001},"last":0.53100000000000003},{"interface":"front","average":{"1min":1.238,"5min":0.85199999999999998,"15min":0.79500000000000004},"min":{"1min":0.317,"5min":0.27300000000000002,"15min":0.22800000000000001},"max":{"1min":4.9370000000000003,"5min":4.9370000000000003,"15min":7.5250000000000004},"last":0.49099999999999999}]},{"osd":3,"last update":"Wed Sep 11 08:20:58 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.8560000000000001,"15min":1.2969999999999999},"min":{"1min":0.28799999999999998,"5min":0.25,"15min":0.25},"max":{"1min":7.4880000000000004,"5min":92.245000000000005,"15min":92.245000000000005},"last":0.51700000000000002},{"interface":"front","average":{"1min":2.319,"5min":2.3159999999999998,"15min":1.478},"min":{"1min":0.35099999999999998,"5min":0.27700000000000002,"15min":0.251},"max":{"1min":16.457999999999998,"5min":92.274000000000001,"15min":92.274000000000001},"last":0.45200000000000001}]},{"osd":4,"last update":"Wed Sep 11 08:20:58 2024","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":0.82199999999999995,"15min":0.61399999999999999},"min":{"1min":0.27100000000000002,"5min":0.25800000000000001,"15min":0.21299999999999999},"max":{"1min":4.6719999999999997,"5min":4.6719999999999997,"15min":4.6719999999999997},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.72399999999999998,"15min":0.57599999999999996},"min":{"1min":0.24199999999999999,"5min":0.23599999999999999,"15min":0.23100000000000001},"max":{"1min":4.9720000000000004,"5min":4.9720000000000004,"15min":4.9720000000000004},"last":0.438}]},{"osd":6,"last update":"Wed Sep 11 08:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.82399999999999995,"15min":0.57799999999999996},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.23400000000000001},"max":{"1min":2.3719999999999999,"5min":7.5679999999999996,"15min":7.5679999999999996},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.90000000000000002,"15min":0.64500000000000002},"min":{"1min":0.28299999999999997,"5min":0.27500000000000002,"15min":0.26900000000000002},"max":{"1min":2.8410000000000002,"5min":3.5470000000000002,"15min":3.9489999999999998},"last":5.4349999999999996}]},{"osd":7,"last update":"Wed Sep 11 08:20:37 2024","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":0.97499999999999998,"15min":0.879},"min":{"1min":0.44800000000000001,"5min":0.33600000000000002,"15min":0.30399999999999999},"max":{"1min":3.0209999999999999,"5min":3.5800000000000001,"15min":3.5800000000000001},"last":5.4720000000000004},{"interface":"front","average":{"1min":1.0669999999999999,"5min":0.999,"15min":0.84199999999999997},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":3.141,"5min":7.5339999999999998,"15min":7.5339999999999998},"last":0.42199999999999999}]}]},{"osd":4,"up_from":31,"seq":133143986358,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":888664,"kb_used_data":587152,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":92856488,"statfs":{"total":95995035648,"available":95085043712,"internally_reserved":0,"allocated":601243648,"data_stored":600853893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:20:29 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":1.9279999999999999,"15min":1.478},"min":{"1min":0.23400000000000001,"5min":0.222,"15min":0.19},"max":{"1min":2.8639999999999999,"5min":60.497999999999998,"15min":60.497999999999998},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.88500000000000001,"5min":1.8240000000000001,"15min":1.341},"min":{"1min":0.247,"5min":0.22600000000000001,"15min":0.221},"max":{"1min":5.9870000000000001,"5min":60.484999999999999,"15min":60.484999999999999},"last":0.51700000000000002}]},{"osd":1,"last update":"Wed Sep 11 08:20:29 2024","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":1.24,"15min":0.95599999999999996},"min":{"1min":0.33000000000000002,"5min":0.214,"15min":0.214},"max":{"1min":2.556,"5min":20.471,"15min":20.471},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.97299999999999998,"5min":1.21,"15min":0.89300000000000002},"min":{"1min":0.249,"5min":0.23100000000000001,"15min":0.22700000000000001},"max":{"1min":2.8180000000000001,"5min":15.744999999999999,"15min":15.744999999999999},"last":0.63100000000000001}]},{"osd":2,"last update":"Wed Sep 11 08:20:29 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.76200000000000001,"15min":0.78700000000000003},"min":{"1min":0.25600000000000001,"5min":0.245,"15min":0.23400000000000001},"max":{"1min":2.9220000000000002,"5min":3.3599999999999999,"15min":7.976},"last":40.993000000000002},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.88400000000000001,"15min":0.79600000000000004},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.248},"max":{"1min":3.5289999999999999,"5min":7.0590000000000002,"15min":7.0590000000000002},"last":41.152000000000001}]},{"osd":3,"last update":"Wed Sep 11 08:20:29 2024","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.48,"15min":1.258},"min":{"1min":0.27600000000000002,"5min":0.253,"15min":0.253},"max":{"1min":3.7610000000000001,"5min":19.686,"15min":26.884},"last":0.54200000000000004},{"interface":"front","average":{"1min":1.169,"5min":1.196,"15min":1.1879999999999999},"min":{"1min":0.29499999999999998,"5min":0.26600000000000001,"15min":0.246},"max":{"1min":4.0570000000000004,"5min":19.882000000000001,"15min":26.783000000000001},"last":0.373}]},{"osd":5,"last update":"Wed Sep 11 08:20:56 2024","interfaces":[{"interface":"back","average":{"1min":2.6459999999999999,"5min":1.024,"15min":0.64300000000000002},"min":{"1min":0.30299999999999999,"5min":0.23999999999999999,"15min":0.23499999999999999},"max":{"1min":33.384999999999998,"5min":33.384999999999998,"15min":33.384999999999998},"last":0.55700000000000005},{"interface":"front","average":{"1min":2.3929999999999998,"5min":1.121,"15min":0.66700000000000004},"min":{"1min":0.253,"5min":0.253,"15min":0.24199999999999999},"max":{"1min":33.856000000000002,"5min":33.856000000000002,"15min":33.856000000000002},"last":0.84499999999999997}]},{"osd":6,"last update":"Wed Sep 11 08:20:18 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":1.1020000000000001,"15min":0.65900000000000003},"min":{"1min":0.313,"5min":0.25,"15min":0.23899999999999999},"max":{"1min":2.4969999999999999,"5min":48.808,"15min":48.808},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.89700000000000002,"5min":1.129,"15min":0.71099999999999997},"min":{"1min":0.28899999999999998,"5min":0.26800000000000002,"15min":0.26200000000000001},"max":{"1min":4.0499999999999998,"5min":48.746000000000002,"15min":48.746000000000002},"last":0.58299999999999996}]},{"osd":7,"last update":"Wed Sep 11 08:20:39 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.72899999999999998,"15min":0.56100000000000005},"min":{"1min":0.29199999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":2.8660000000000001,"5min":5.3449999999999998,"15min":5.3449999999999998},"last":1.2050000000000001},{"interface":"front","average":{"1min":1.3680000000000001,"5min":0.86599999999999999,"15min":0.72999999999999998},"min":{"1min":0.35899999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":4.0910000000000002,"5min":4.7160000000000002,"15min":4.7160000000000002},"last":1.224}]}]},{"osd":3,"up_from":25,"seq":107374182587,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1329632,"kb_used_data":739608,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":92415520,"statfs":{"total":95995035648,"available":94633492480,"internally_reserved":0,"allocated":757358592,"data_stored":756960092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,4],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:20:18 2024","interfaces":[{"interface":"back","average":{"1min":1.607,"5min":1.9419999999999999,"15min":1.363},"min":{"1min":0.21299999999999999,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":5.9089999999999998,"5min":60.469999999999999,"15min":60.469999999999999},"last":58.880000000000003},{"interface":"front","average":{"1min":1.5349999999999999,"5min":1.9850000000000001,"15min":1.448},"min":{"1min":0.27300000000000002,"5min":0.187,"15min":0.17399999999999999},"max":{"1min":5.5289999999999999,"5min":60.488999999999997,"15min":60.488999999999997},"last":59.246000000000002}]},{"osd":1,"last update":"Wed Sep 11 08:20:18 2024","interfaces":[{"interface":"back","average":{"1min":1.6439999999999999,"5min":1.857,"15min":1.3140000000000001},"min":{"1min":0.255,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.6029999999999998,"5min":60.578000000000003,"15min":60.578000000000003},"last":0.48499999999999999},{"interface":"front","average":{"1min":1.4970000000000001,"5min":2.395,"15min":1.5589999999999999},"min":{"1min":0.29599999999999999,"5min":0.20499999999999999,"15min":0.19500000000000001},"max":{"1min":6.1219999999999999,"5min":63.674999999999997,"15min":63.674999999999997},"last":0.47199999999999998}]},{"osd":2,"last update":"Wed Sep 11 08:20:18 2024","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.96,"15min":1.4410000000000001},"min":{"1min":0.251,"5min":0.22,"15min":0.185},"max":{"1min":5.7949999999999999,"5min":60.645000000000003,"15min":60.645000000000003},"last":0.35799999999999998},{"interface":"front","average":{"1min":1.278,"5min":1.8660000000000001,"15min":1.335},"min":{"1min":0.35299999999999998,"5min":0.22800000000000001,"15min":0.20599999999999999},"max":{"1min":5.9500000000000002,"5min":60.723999999999997,"15min":60.723999999999997},"last":0.22800000000000001}]},{"osd":4,"last update":"Wed Sep 11 08:20:25 2024","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.7749999999999999,"15min":1.246},"min":{"1min":0.372,"5min":0.28100000000000003,"15min":0.24299999999999999},"max":{"1min":2.7839999999999998,"5min":61.290999999999997,"15min":61.290999999999997},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.84699999999999998,"5min":1.587,"15min":1.151},"min":{"1min":0.40300000000000002,"5min":0.28899999999999998,"15min":0.27200000000000002},"max":{"1min":1.875,"5min":60.856000000000002,"15min":60.856000000000002},"last":0.63}]},{"osd":5,"last update":"Wed Sep 11 08:21:02 2024","interfaces":[{"interface":"back","average":{"1min":2.4199999999999999,"5min":1.9950000000000001,"15min":1.645},"min":{"1min":0.29999999999999999,"5min":0.28199999999999997,"15min":0.26500000000000001},"max":{"1min":25.754999999999999,"5min":31.460999999999999,"15min":60.914999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":2.5750000000000002,"5min":1.9510000000000001,"15min":1.7110000000000001},"min":{"1min":0.35799999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":25.577999999999999,"5min":31.216000000000001,"15min":60.787999999999997},"last":0.53900000000000003}]},{"osd":6,"last update":"Wed Sep 11 08:20:18 2024","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.7470000000000001,"15min":1.4410000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":5.0970000000000004,"5min":60.817,"15min":60.817},"last":0.56100000000000005},{"interface":"front","average":{"1min":1.357,"5min":2.0310000000000001,"15min":1.6020000000000001},"min":{"1min":0.38300000000000001,"5min":0.34699999999999998,"15min":0.307},"max":{"1min":6.0709999999999997,"5min":61.264000000000003,"15min":61.264000000000003},"last":0.43099999999999999}]},{"osd":7,"last update":"Wed Sep 11 08:20:33 2024","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":2.0699999999999998,"15min":2.7839999999999998},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":3.8479999999999999,"5min":60.847000000000001,"15min":60.847000000000001},"last":0.51600000000000001},{"interface":"front","average":{"1min":1.075,"5min":2.02,"15min":2.722},"min":{"1min":0.48899999999999999,"5min":0.36699999999999999,"15min":0.34499999999999997},"max":{"1min":2.351,"5min":60.883000000000003,"15min":60.883000000000003},"last":58.820999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411518,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":583104,"kb_used_data":285752,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93162048,"statfs":{"total":95995035648,"available":95397937152,"internally_reserved":0,"allocated":292610048,"data_stored":292243443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:20:52 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":1.55,"15min":1.4610000000000001},"min":{"1min":0.183,"5min":0.17799999999999999,"15min":0.17199999999999999},"max":{"1min":3.1699999999999999,"5min":25.427,"15min":43.539999999999999},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.84599999999999997,"5min":2.02,"15min":1.694},"min":{"1min":0.19700000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":2.2650000000000001,"5min":44.424999999999997,"15min":44.424999999999997},"last":0.39000000000000001}]},{"osd":1,"last update":"Wed Sep 11 08:20:52 2024","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.556,"15min":1.282},"min":{"1min":0.20799999999999999,"5min":0.20200000000000001,"15min":0.17599999999999999},"max":{"1min":5.2519999999999998,"5min":23.748999999999999,"15min":23.748999999999999},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.78500000000000003,"5min":1.3280000000000001,"15min":1.256},"min":{"1min":0.23000000000000001,"5min":0.20799999999999999,"15min":0.20200000000000001},"max":{"1min":6.0570000000000004,"5min":23.513999999999999,"15min":23.513999999999999},"last":0.60299999999999998}]},{"osd":3,"last update":"Wed Sep 11 08:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":1.276,"15min":1.1319999999999999},"min":{"1min":0.221,"5min":0.21299999999999999,"15min":0.188},"max":{"1min":4.5529999999999999,"5min":8.1989999999999998,"15min":8.1989999999999998},"last":0.628},{"interface":"front","average":{"1min":0.78700000000000003,"5min":1.1120000000000001,"15min":1.056},"min":{"1min":0.21299999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":2.702,"5min":8.0640000000000001,"15min":8.3219999999999992},"last":0.64400000000000002}]},{"osd":4,"last update":"Wed Sep 11 08:20:35 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.90200000000000002,"15min":0.90100000000000002},"min":{"1min":0.308,"5min":0.27900000000000003,"15min":0.25900000000000001},"max":{"1min":2.125,"5min":5.5970000000000004,"15min":7.806},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.84999999999999998,"15min":0.91800000000000004},"min":{"1min":0.29299999999999998,"5min":0.27500000000000002,"15min":0.252},"max":{"1min":2.5139999999999998,"5min":3.855,"15min":12.129},"last":0.80800000000000005}]},{"osd":5,"last update":"Wed Sep 11 08:20:04 2024","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.76,"15min":1.175},"min":{"1min":0.35099999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":3.3780000000000001,"5min":35.963000000000001,"15min":35.963000000000001},"last":1.2470000000000001},{"interface":"front","average":{"1min":1.25,"5min":1.825,"15min":1.2130000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":4.2619999999999996,"5min":35.104999999999997,"15min":35.104999999999997},"last":0.84899999999999998}]},{"osd":6,"last update":"Wed Sep 11 08:20:23 2024","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":0.91400000000000003,"15min":0.877},"min":{"1min":0.31,"5min":0.31,"15min":0.26100000000000001},"max":{"1min":4.2169999999999996,"5min":4.2539999999999996,"15min":7.2249999999999996},"last":0.872},{"interface":"front","average":{"1min":0.90000000000000002,"5min":1.0800000000000001,"15min":0.97599999999999998},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.32100000000000001},"max":{"1min":2.8010000000000002,"5min":5.4939999999999998,"15min":6.71},"last":1.286}]},{"osd":7,"last update":"Wed Sep 11 08:20:35 2024","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.5680000000000001,"15min":1.448},"min":{"1min":0.34599999999999997,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":11.19,"5min":35.274999999999999,"15min":35.274999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.5409999999999999,"15min":1.538},"min":{"1min":0.40100000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":11.172000000000001,"5min":35.265000000000001,"15min":35.265000000000001},"last":0.77700000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575044,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1395116,"kb_used_data":792740,"kb_used_omap":1,"kb_used_meta":602366,"kb_avail":92350036,"statfs":{"total":95995035648,"available":94566436864,"internally_reserved":0,"allocated":811765760,"data_stored":811355249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":616823128},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:20:23 2024","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.573,"15min":1.2709999999999999},"min":{"1min":0.222,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":5.8609999999999998,"5min":8.6720000000000006,"15min":32.478000000000002},"last":63.932000000000002},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.71,"15min":1.5329999999999999},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":5.6310000000000002,"5min":15.351000000000001,"15min":35.459000000000003},"last":64.269000000000005}]},{"osd":2,"last update":"Wed Sep 11 08:20:02 2024","interfaces":[{"interface":"back","average":{"1min":1.716,"5min":1.4370000000000001,"15min":1.0920000000000001},"min":{"1min":0.30499999999999999,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":5.8369999999999997,"5min":7.9699999999999998,"15min":8.3819999999999997},"last":0.27300000000000002},{"interface":"front","average":{"1min":1.49,"5min":1.3819999999999999,"15min":1.1899999999999999},"min":{"1min":0.29999999999999999,"5min":0.192,"15min":0.16300000000000001},"max":{"1min":5.6289999999999996,"5min":5.8490000000000002,"15min":10.276},"last":0.23699999999999999}]},{"osd":3,"last update":"Wed Sep 11 08:20:07 2024","interfaces":[{"interface":"back","average":{"1min":1.867,"5min":1.7390000000000001,"15min":1.6830000000000001},"min":{"1min":0.23000000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.8209999999999997,"5min":8.4149999999999991,"15min":32.759},"last":0.26200000000000001},{"interface":"front","average":{"1min":2.0819999999999999,"5min":1.706,"15min":1.5569999999999999},"min":{"1min":0.25,"5min":0.25,"15min":0.22600000000000001},"max":{"1min":8.8439999999999994,"5min":9.0060000000000002,"15min":32.444000000000003},"last":0.29899999999999999}]},{"osd":4,"last update":"Wed Sep 11 08:20:33 2024","interfaces":[{"interface":"back","average":{"1min":2.1869999999999998,"5min":3.0819999999999999,"15min":1.6619999999999999},"min":{"1min":0.41499999999999998,"5min":0.26200000000000001,"15min":0.252},"max":{"1min":17.856000000000002,"5min":110.86499999999999,"15min":110.86499999999999},"last":0.45100000000000001},{"interface":"front","average":{"1min":1.8080000000000001,"5min":2.9249999999999998,"15min":1.5880000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.248},"max":{"1min":5.7949999999999999,"5min":110.831,"15min":110.831},"last":0.43099999999999999}]},{"osd":5,"last update":"Wed Sep 11 08:20:02 2024","interfaces":[{"interface":"back","average":{"1min":1.7010000000000001,"5min":2.8999999999999999,"15min":1.623},"min":{"1min":0.35499999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":4.274,"5min":109.208,"15min":109.208},"last":0.48899999999999999},{"interface":"front","average":{"1min":1.865,"5min":3.1030000000000002,"15min":1.7210000000000001},"min":{"1min":0.39800000000000002,"5min":0.35199999999999998,"15min":0.28100000000000003},"max":{"1min":4.5999999999999996,"5min":111.254,"15min":111.254},"last":0.47999999999999998}]},{"osd":6,"last update":"Wed Sep 11 08:20:12 2024","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":2.7269999999999999,"15min":1.6240000000000001},"min":{"1min":0.316,"5min":0.30499999999999999,"15min":0.29399999999999998},"max":{"1min":4.0910000000000002,"5min":124.31999999999999,"15min":124.31999999999999},"last":0.441},{"interface":"front","average":{"1min":1.3300000000000001,"5min":2.6850000000000001,"15min":1.569},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":5.8159999999999998,"5min":124.958,"15min":124.958},"last":0.46200000000000002}]},{"osd":7,"last update":"Wed Sep 11 08:20:33 2024","interfaces":[{"interface":"back","average":{"1min":1.8959999999999999,"5min":2.7280000000000002,"15min":1.6990000000000001},"min":{"1min":0.435,"5min":0.33400000000000002,"15min":0.29399999999999998},"max":{"1min":5.0650000000000004,"5min":109.149,"15min":109.149},"last":0.42199999999999999},{"interface":"front","average":{"1min":1.9259999999999999,"5min":2.742,"15min":1.8999999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":5.7590000000000003,"5min":109.309,"15min":109.309},"last":0.504}]}]},{"osd":0,"up_from":8,"seq":34359738568,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":671132,"kb_used_data":357140,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":93074020,"statfs":{"total":95995035648,"available":95307796480,"internally_reserved":0,"allocated":365711360,"data_stored":365318277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 11 08:20:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":1.3200000000000001,"15min":1.1240000000000001},"min":{"1min":0.216,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":3.306,"5min":10.105,"15min":10.105},"last":0.214},{"interface":"front","average":{"1min":1.363,"5min":1.8069999999999999,"15min":1.3580000000000001},"min":{"1min":0.17999999999999999,"5min":0.16600000000000001,"15min":0.153},"max":{"1min":3.0870000000000002,"5min":19.852,"15min":19.852},"last":0.52500000000000002}]},{"osd":2,"last update":"Wed Sep 11 08:20:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":1.3640000000000001,"15min":1.1399999999999999},"min":{"1min":0.20799999999999999,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":3.0110000000000001,"5min":10.025,"15min":10.351000000000001},"last":0.30599999999999999},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.391,"15min":1.228},"min":{"1min":0.26300000000000001,"5min":0.23699999999999999,"15min":0.186},"max":{"1min":4.883,"5min":10.074999999999999,"15min":10.074999999999999},"last":0.44600000000000001}]},{"osd":3,"last update":"Wed Sep 11 08:20:15 2024","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":1.488,"15min":1.4890000000000001},"min":{"1min":0.27400000000000002,"5min":0.251,"15min":0.22900000000000001},"max":{"1min":3.6960000000000002,"5min":10.129,"15min":35.491},"last":0.46999999999999997},{"interface":"front","average":{"1min":1.3340000000000001,"5min":1.375,"15min":1.4430000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":4.0570000000000004,"5min":8.282,"15min":37.563000000000002},"last":0.25}]},{"osd":4,"last update":"Wed Sep 11 08:20:38 2024","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.999,"15min":1.02},"min":{"1min":0.252,"5min":0.252,"15min":0.24099999999999999},"max":{"1min":4.5949999999999998,"5min":5.0190000000000001,"15min":9.4030000000000005},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.91600000000000004,"5min":1.0820000000000001,"15min":1.0329999999999999},"min":{"1min":0.32500000000000001,"5min":0.27200000000000002,"15min":0.24399999999999999},"max":{"1min":3.0430000000000001,"5min":8.3510000000000009,"15min":8.3510000000000009},"last":0.38800000000000001}]},{"osd":5,"last update":"Wed Sep 11 08:20:59 2024","interfaces":[{"interface":"back","average":{"1min":1.5720000000000001,"5min":1.347,"15min":1.244},"min":{"1min":0.42599999999999999,"5min":0.28599999999999998,"15min":0.255},"max":{"1min":10.486000000000001,"5min":10.486000000000001,"15min":24.914999999999999},"last":0.42199999999999999},{"interface":"front","average":{"1min":2.3769999999999998,"5min":1.5229999999999999,"15min":1.3959999999999999},"min":{"1min":0.35699999999999998,"5min":0.28199999999999997,"15min":0.27100000000000002},"max":{"1min":12.536,"5min":13.885,"15min":25.277000000000001},"last":0.27900000000000003}]},{"osd":6,"last update":"Wed Sep 11 08:20:17 2024","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.121,"15min":1.379},"min":{"1min":0.32900000000000001,"5min":0.29599999999999999,"15min":0.29099999999999998},"max":{"1min":3.8530000000000002,"5min":5.085,"15min":78.522000000000006},"last":0.40500000000000003},{"interface":"front","average":{"1min":1.194,"5min":1.21,"15min":1.3},"min":{"1min":0.315,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":4.7539999999999996,"5min":7.4820000000000002,"15min":79.808000000000007},"last":1.1679999999999999}]},{"osd":7,"last update":"Wed Sep 11 08:20:38 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.292,"15min":1.3380000000000001},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":3.0680000000000001,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":2.6179999999999999},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.4450000000000001,"15min":1.385},"min":{"1min":0.33800000000000002,"5min":0.33600000000000002,"15min":0.32000000000000001},"max":{"1min":3.1859999999999999,"5min":13.914,"15min":13.914},"last":0.51200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3425,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":364183552,"data_stored":364183552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":809897984,"data_stored":809893912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":286851072,"data_stored":286851072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":739573760,"data_stored":739573760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":597430272,"data_stored":597426200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":835325952,"data_stored":835321880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":282263552,"data_stored":282263552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":834211840,"data_stored":834207768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T08:21:08.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:08.869 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:08.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:08.915 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-11T08:21:08.915 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-09-11T08:21:08.915 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.916 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:215204 9.1 9:803df01d:::benchmark_data_smithi150_133616_object99598:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e299)", 2024-09-11T08:21:08.916 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:21:08.900780+0000", 2024-09-11T08:21:08.916 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.012871685000000001, 2024-09-11T08:21:08.916 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.012900501999999999, 2024-09-11T08:21:08.916 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:21:08.916 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "waiting for sub ops", 2024-09-11T08:21:08.916 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:21:08.916 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:21:08.916 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:21:08.916 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 215204 2024-09-11T08:21:08.916 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.916 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:21:08.917 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.917 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:21:08.917 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.900780+0000", 2024-09-11T08:21:08.917 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:08.917 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.917 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.917 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:21:08.917 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.900780+0000", 2024-09-11T08:21:08.917 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:08.917 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.917 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.917 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:21:08.917 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.900778+0000", 2024-09-11T08:21:08.918 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999981 2024-09-11T08:21:08.918 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.918 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.918 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:21:08.918 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.900790+0000", 2024-09-11T08:21:08.918 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2425999999999999e-05 2024-09-11T08:21:08.918 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.918 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.918 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:21:08.918 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.900810+0000", 2024-09-11T08:21:08.918 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0123e-05 2024-09-11T08:21:08.918 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.918 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.918 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:21:08.919 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.900819+0000", 2024-09-11T08:21:08.919 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5150000000000006e-06 2024-09-11T08:21:08.919 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.919 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.919 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:08.919 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.902488+0000", 2024-09-11T08:21:08.919 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001669146 2024-09-11T08:21:08.919 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.919 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.919 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:21:08.919 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.902758+0000", 2024-09-11T08:21:08.919 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00026962200000000001 2024-09-11T08:21:08.919 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.919 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.920 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3,7", 2024-09-11T08:21:08.920 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.903084+0000", 2024-09-11T08:21:08.920 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00032662499999999998 2024-09-11T08:21:08.920 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.920 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.920 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:08.920 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.908076+0000", 2024-09-11T08:21:08.920 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0049919439999999999 2024-09-11T08:21:08.920 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.920 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.920 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:21:08.920 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.908138+0000", 2024-09-11T08:21:08.920 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1601999999999997e-05 2024-09-11T08:21:08.920 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:08.921 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:21:08.921 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:08.921 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.921 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.921 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.15369.0:215215 9.0 e299/246 9:0f7a7438:::benchmark_data_smithi150_133616_object99609:head v 299'13455, mlcod=299'13453)", 2024-09-11T08:21:08.921 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:21:08.911027+0000", 2024-09-11T08:21:08.921 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.002624702, 2024-09-11T08:21:08.921 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0027436700000000001, 2024-09-11T08:21:08.921 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:21:08.921 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-11T08:21:08.921 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:21:08.921 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.921 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:21:08.921 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.911027+0000", 2024-09-11T08:21:08.921 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:08.922 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.922 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.922 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:21:08.922 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.911027+0000", 2024-09-11T08:21:08.922 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:08.922 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.922 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.922 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:21:08.922 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.911025+0000", 2024-09-11T08:21:08.922 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:21:08.922 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.922 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.922 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:21:08.922 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.911035+0000", 2024-09-11T08:21:08.922 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7149999999999993e-06 2024-09-11T08:21:08.922 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.923 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.923 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:21:08.923 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.911055+0000", 2024-09-11T08:21:08.923 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9681e-05 2024-09-11T08:21:08.923 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.923 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.923 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:21:08.923 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.911063+0000", 2024-09-11T08:21:08.923 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0949999999999996e-06 2024-09-11T08:21:08.923 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.923 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.923 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:08.923 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.911261+0000", 2024-09-11T08:21:08.923 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00019830600000000001 2024-09-11T08:21:08.923 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.923 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.924 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:21:08.924 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.911279+0000", 2024-09-11T08:21:08.924 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7489000000000001e-05 2024-09-11T08:21:08.924 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:08.924 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:21:08.924 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:08.924 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.924 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.924 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:215214 9.1 9:869d9ae4:::benchmark_data_smithi150_133616_object99608:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e299)", 2024-09-11T08:21:08.924 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:21:08.911106+0000", 2024-09-11T08:21:08.924 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.0025448189999999998, 2024-09-11T08:21:08.924 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0027248369999999999, 2024-09-11T08:21:08.924 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:21:08.924 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "waiting for sub ops", 2024-09-11T08:21:08.924 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:21:08.925 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:21:08.925 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:21:08.925 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 215214 2024-09-11T08:21:08.925 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.925 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:21:08.925 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.925 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:21:08.925 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.911106+0000", 2024-09-11T08:21:08.925 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:08.925 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.925 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.925 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:21:08.925 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.911106+0000", 2024-09-11T08:21:08.925 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:08.925 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.926 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.926 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:21:08.926 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.911105+0000", 2024-09-11T08:21:08.926 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:21:08.926 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.926 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.926 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:21:08.926 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.911112+0000", 2024-09-11T08:21:08.926 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.579e-06 2024-09-11T08:21:08.926 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.926 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.926 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:21:08.926 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.911123+0000", 2024-09-11T08:21:08.926 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1321999999999999e-05 2024-09-11T08:21:08.926 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.926 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.927 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:21:08.927 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.911129+0000", 2024-09-11T08:21:08.927 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3799999999999999e-06 2024-09-11T08:21:08.927 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.927 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.927 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:08.927 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.911180+0000", 2024-09-11T08:21:08.927 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0813999999999998e-05 2024-09-11T08:21:08.927 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.927 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.927 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:21:08.927 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.912632+0000", 2024-09-11T08:21:08.927 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0014520959999999999 2024-09-11T08:21:08.927 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.928 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.928 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3,7", 2024-09-11T08:21:08.928 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.912746+0000", 2024-09-11T08:21:08.928 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000113731 2024-09-11T08:21:08.928 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:08.928 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:21:08.928 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:08.928 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.928 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.928 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop_reply(client.15369.0:215184 9.1 e299/246)", 2024-09-11T08:21:08.928 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:21:08.912588+0000", 2024-09-11T08:21:08.928 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.0010636720000000001, 2024-09-11T08:21:08.928 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001333975, 2024-09-11T08:21:08.928 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:21:08.928 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "queued for pg", 2024-09-11T08:21:08.929 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:21:08.929 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.929 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:21:08.929 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.912588+0000", 2024-09-11T08:21:08.929 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:08.929 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.930 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.930 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:21:08.930 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.912588+0000", 2024-09-11T08:21:08.930 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:08.930 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.930 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.930 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:21:08.930 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.912587+0000", 2024-09-11T08:21:08.930 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:21:08.930 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.930 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.930 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:21:08.930 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.912593+0000", 2024-09-11T08:21:08.930 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4130000000000003e-06 2024-09-11T08:21:08.930 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.931 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.931 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:21:08.931 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.912603+0000", 2024-09-11T08:21:08.931 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0146000000000001e-05 2024-09-11T08:21:08.931 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.931 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.931 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:21:08.931 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.912609+0000", 2024-09-11T08:21:08.931 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.711e-06 2024-09-11T08:21:08.931 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:08.932 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:21:08.932 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:08.932 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.932 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.932 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop_reply(client.15369.0:215186 9.1 e299/246)", 2024-09-11T08:21:08.932 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:21:08.913936+0000", 2024-09-11T08:21:08.932 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4294967295.9997149, 2024-09-11T08:21:08.932 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2245e-05, 2024-09-11T08:21:08.932 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:21:08.932 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "queued for pg", 2024-09-11T08:21:08.932 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:21:08.932 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.932 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:21:08.933 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.913936+0000", 2024-09-11T08:21:08.933 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:08.933 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.933 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.933 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:21:08.933 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.913936+0000", 2024-09-11T08:21:08.933 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:08.933 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.933 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.933 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:21:08.933 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.913936+0000", 2024-09-11T08:21:08.933 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:21:08.933 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.933 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.934 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:21:08.934 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.913941+0000", 2024-09-11T08:21:08.934 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9949999999999996e-06 2024-09-11T08:21:08.934 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.934 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.934 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:21:08.934 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.913949+0000", 2024-09-11T08:21:08.934 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4009999999999997e-06 2024-09-11T08:21:08.934 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.934 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.934 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:21:08.934 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.913954+0000", 2024-09-11T08:21:08.934 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0250000000000002e-06 2024-09-11T08:21:08.934 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:08.934 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:21:08.935 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:08.935 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.935 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.935 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:215184 9.1 9:848160e2:::benchmark_data_smithi150_133616_object99578:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e299)", 2024-09-11T08:21:08.935 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:21:08.885189+0000", 2024-09-11T08:21:08.935 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.028462390000000001, 2024-09-11T08:21:08.935 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.028830415000000002, 2024-09-11T08:21:08.935 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:21:08.935 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "waiting for sub ops", 2024-09-11T08:21:08.935 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:21:08.935 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:21:08.935 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:21:08.936 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 215184 2024-09-11T08:21:08.936 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.936 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:21:08.936 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.936 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:21:08.936 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.885189+0000", 2024-09-11T08:21:08.936 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:08.936 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.936 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.936 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:21:08.936 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.885189+0000", 2024-09-11T08:21:08.936 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:08.937 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.937 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.937 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:21:08.937 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.885187+0000", 2024-09-11T08:21:08.937 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999981 2024-09-11T08:21:08.937 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.937 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.937 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:21:08.937 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.885196+0000", 2024-09-11T08:21:08.938 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8559999999999996e-06 2024-09-11T08:21:08.938 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.938 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.938 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:21:08.938 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.885214+0000", 2024-09-11T08:21:08.938 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7878000000000001e-05 2024-09-11T08:21:08.938 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.938 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.938 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:21:08.938 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.885221+0000", 2024-09-11T08:21:08.938 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6229999999999997e-06 2024-09-11T08:21:08.938 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.938 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.938 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:08.939 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.885269+0000", 2024-09-11T08:21:08.939 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8128999999999999e-05 2024-09-11T08:21:08.939 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.939 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.939 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:21:08.939 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.885401+0000", 2024-09-11T08:21:08.939 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000131626 2024-09-11T08:21:08.939 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.939 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.939 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3,7", 2024-09-11T08:21:08.939 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.885536+0000", 2024-09-11T08:21:08.939 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013449499999999999 2024-09-11T08:21:08.939 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.939 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.939 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:21:08.940 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.894291+0000", 2024-09-11T08:21:08.940 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0087555440000000005 2024-09-11T08:21:08.940 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.940 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.940 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:08.940 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.894416+0000", 2024-09-11T08:21:08.940 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012469099999999999 2024-09-11T08:21:08.940 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:08.940 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:21:08.940 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:08.940 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.940 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.940 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:215186 9.1 9:89dfe167:::benchmark_data_smithi150_133616_object99580:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e299)", 2024-09-11T08:21:08.940 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:21:08.887515+0000", 2024-09-11T08:21:08.941 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.026135912000000001, 2024-09-11T08:21:08.941 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.026584796000000001, 2024-09-11T08:21:08.941 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:21:08.941 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "waiting for sub ops", 2024-09-11T08:21:08.941 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:21:08.941 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:21:08.941 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:21:08.941 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 215186 2024-09-11T08:21:08.941 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.941 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:21:08.941 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.941 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:21:08.941 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.887515+0000", 2024-09-11T08:21:08.941 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:08.942 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.942 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.942 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:21:08.942 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.887515+0000", 2024-09-11T08:21:08.942 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:08.942 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.942 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.942 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:21:08.942 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.887514+0000", 2024-09-11T08:21:08.942 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:21:08.942 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.942 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.942 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:21:08.942 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.887520+0000", 2024-09-11T08:21:08.942 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6830000000000002e-06 2024-09-11T08:21:08.943 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.943 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.943 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:21:08.943 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.887538+0000", 2024-09-11T08:21:08.943 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7243000000000001e-05 2024-09-11T08:21:08.943 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.943 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.943 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:21:08.943 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.887545+0000", 2024-09-11T08:21:08.943 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8530000000000003e-06 2024-09-11T08:21:08.943 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.943 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.943 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:08.943 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.889359+0000", 2024-09-11T08:21:08.944 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0018145150000000001 2024-09-11T08:21:08.944 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.944 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.944 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:21:08.944 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.889627+0000", 2024-09-11T08:21:08.944 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00026823299999999999 2024-09-11T08:21:08.944 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.944 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.944 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3,7", 2024-09-11T08:21:08.944 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.889954+0000", 2024-09-11T08:21:08.944 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00032678500000000001 2024-09-11T08:21:08.944 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.944 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.944 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:21:08.945 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.894311+0000", 2024-09-11T08:21:08.945 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0043571759999999999 2024-09-11T08:21:08.945 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:08.945 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:08.945 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:08.945 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:08.895661+0000", 2024-09-11T08:21:08.945 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00134961 2024-09-11T08:21:08.945 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:08.945 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:21:08.945 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:08.945 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:08.945 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-09-11T08:21:08.945 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 7 2024-09-11T08:21:08.945 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-11T08:21:08.970 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:09.004 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:09.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:09.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:09.106 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:09.146 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:09.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:09.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:09.252 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:21:09.287 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:09.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:09.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:21:09.325 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:09 smithi150 ceph-mon[29276]: 9.c scrub starts 2024-09-11T08:21:09.326 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:09 smithi150 ceph-mon[29276]: from='client.25321 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:21:09.326 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:09 smithi150 ceph-mon[29276]: 9.c scrub ok 2024-09-11T08:21:09.326 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:09 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]}]': finished 2024-09-11T08:21:09.326 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:09 smithi150 ceph-mon[29276]: osdmap e299: 8 total, 8 up, 8 in 2024-09-11T08:21:09.326 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:09 smithi150 ceph-mon[39612]: 9.c scrub starts 2024-09-11T08:21:09.326 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:09 smithi150 ceph-mon[39612]: from='client.25321 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:21:09.326 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:09 smithi150 ceph-mon[39612]: 9.c scrub ok 2024-09-11T08:21:09.326 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:09 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]}]': finished 2024-09-11T08:21:09.326 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:09 smithi150 ceph-mon[39612]: osdmap e299: 8 total, 8 up, 8 in 2024-09-11T08:21:09.389 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:21:09.421 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:09.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:09.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:21:09.523 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:09.558 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:09.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:09.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:09.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:09 smithi181 ceph-mon[32701]: 9.c scrub starts 2024-09-11T08:21:09.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:09 smithi181 ceph-mon[32701]: from='client.25321 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:21:09.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:09 smithi181 ceph-mon[32701]: 9.c scrub ok 2024-09-11T08:21:09.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:09 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]}]': finished 2024-09-11T08:21:09.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:09 smithi181 ceph-mon[32701]: osdmap e299: 8 total, 8 up, 8 in 2024-09-11T08:21:09.660 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:09.693 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:09.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:09.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:09.698 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-11T08:21:09.794 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:09.835 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:09.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:09.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:09.937 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:09.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:09.988 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:09.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:10.089 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:21:10.124 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:10.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:10.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:21:10.139 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:21:10.227 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:21:10.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:10.279 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:10.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:21:10.380 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:10.422 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:10.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:10.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:10.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:10 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:21:10.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:10 smithi150 ceph-mon[29276]: 6.10 scrub starts 2024-09-11T08:21:10.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:10 smithi150 ceph-mon[29276]: 6.10 scrub ok 2024-09-11T08:21:10.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:10 smithi150 ceph-mon[29276]: pgmap v620: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.0 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-09-11T08:21:10.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:10 smithi150 ceph-mon[29276]: osdmap e300: 8 total, 8 up, 8 in 2024-09-11T08:21:10.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:10 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:21:10.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:10 smithi150 ceph-mon[39612]: 6.10 scrub starts 2024-09-11T08:21:10.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:10 smithi150 ceph-mon[39612]: 6.10 scrub ok 2024-09-11T08:21:10.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:10 smithi150 ceph-mon[39612]: pgmap v620: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.0 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-09-11T08:21:10.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:10 smithi150 ceph-mon[39612]: osdmap e300: 8 total, 8 up, 8 in 2024-09-11T08:21:10.524 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:21:10.553 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:10 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:21:10.553 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:10 smithi181 ceph-mon[32701]: 6.10 scrub starts 2024-09-11T08:21:10.553 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:10 smithi181 ceph-mon[32701]: 6.10 scrub ok 2024-09-11T08:21:10.554 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:10 smithi181 ceph-mon[32701]: pgmap v620: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.0 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-09-11T08:21:10.554 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:10 smithi181 ceph-mon[32701]: osdmap e300: 8 total, 8 up, 8 in 2024-09-11T08:21:10.596 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:10.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:10.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:21:10.698 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:10.732 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:10.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:10.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:10.834 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:10.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:10.873 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:10.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:10.940 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:21:10.941 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:21:10.969 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:21:10.970 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:21:10.975 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:11.001 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:21:11.002 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:21:11.011 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:11.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:11.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:11.038 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:21:11.038 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:21:11.067 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:21:11.067 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:21:11.106 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:21:11.106 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:21:11.114 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:11.139 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:21:11.139 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:21:11.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:11.144 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:11.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:11.176 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:21:11.176 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:21:11.245 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:11.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:11.276 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:11.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:11.376 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:11 smithi150 ceph-mon[29276]: 5.c scrub starts 2024-09-11T08:21:11.376 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:11 smithi150 ceph-mon[29276]: 5.c scrub ok 2024-09-11T08:21:11.376 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:11 smithi150 ceph-mon[29276]: 3.1f scrub starts 2024-09-11T08:21:11.376 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:11 smithi150 ceph-mon[29276]: 3.1f scrub ok 2024-09-11T08:21:11.376 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:11 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2366511042' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:21:11.377 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:11 smithi150 ceph-mon[39612]: 5.c scrub starts 2024-09-11T08:21:11.377 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:11 smithi150 ceph-mon[39612]: 5.c scrub ok 2024-09-11T08:21:11.377 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:11 smithi150 ceph-mon[39612]: 3.1f scrub starts 2024-09-11T08:21:11.377 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:11 smithi150 ceph-mon[39612]: 3.1f scrub ok 2024-09-11T08:21:11.377 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:11 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2366511042' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:21:11.377 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:11.417 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:11.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:11.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:11.519 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:21:11.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:11.557 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:11.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:21:11.567 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:11 smithi181 ceph-mon[32701]: 5.c scrub starts 2024-09-11T08:21:11.567 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:11 smithi181 ceph-mon[32701]: 5.c scrub ok 2024-09-11T08:21:11.568 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:11 smithi181 ceph-mon[32701]: 3.1f scrub starts 2024-09-11T08:21:11.568 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:11 smithi181 ceph-mon[32701]: 3.1f scrub ok 2024-09-11T08:21:11.568 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:11 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2366511042' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:21:11.659 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:11.692 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:11.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:11.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:11.794 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:11.831 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:11.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:11.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:11.932 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:11.973 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:11.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:11.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:12.075 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:12.108 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:12.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:12.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:12.210 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:12.226 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-11T08:21:12.226 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:21:12.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:12.242 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:12.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:12.343 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:21:12.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:12.378 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:12.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:21:12.479 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:12.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:12 smithi150 ceph-mon[29276]: 3.0 scrub starts 2024-09-11T08:21:12.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:12 smithi150 ceph-mon[29276]: 3.0 scrub ok 2024-09-11T08:21:12.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:12 smithi150 ceph-mon[29276]: 4.b scrub starts 2024-09-11T08:21:12.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:12 smithi150 ceph-mon[29276]: 4.b scrub ok 2024-09-11T08:21:12.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:12 smithi150 ceph-mon[29276]: pgmap v622: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.0 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 753 B/s rd, 0 B/s wr, 846 op/s 2024-09-11T08:21:12.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:12 smithi150 ceph-mon[29276]: 5.11 scrub starts 2024-09-11T08:21:12.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:12 smithi150 ceph-mon[29276]: 5.11 scrub ok 2024-09-11T08:21:12.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:12 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2238027589' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:21:12.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:12 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:21:12.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:12 smithi150 ceph-mon[39612]: 3.0 scrub starts 2024-09-11T08:21:12.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:12 smithi150 ceph-mon[39612]: 3.0 scrub ok 2024-09-11T08:21:12.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:12 smithi150 ceph-mon[39612]: 4.b scrub starts 2024-09-11T08:21:12.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:12 smithi150 ceph-mon[39612]: 4.b scrub ok 2024-09-11T08:21:12.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:12 smithi150 ceph-mon[39612]: pgmap v622: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.0 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 753 B/s rd, 0 B/s wr, 846 op/s 2024-09-11T08:21:12.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:12 smithi150 ceph-mon[39612]: 5.11 scrub starts 2024-09-11T08:21:12.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:12 smithi150 ceph-mon[39612]: 5.11 scrub ok 2024-09-11T08:21:12.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:12 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2238027589' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:21:12.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:12 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:21:12.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:12.515 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:12.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:12.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:12 smithi181 ceph-mon[32701]: 3.0 scrub starts 2024-09-11T08:21:12.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:12 smithi181 ceph-mon[32701]: 3.0 scrub ok 2024-09-11T08:21:12.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:12 smithi181 ceph-mon[32701]: 4.b scrub starts 2024-09-11T08:21:12.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:12 smithi181 ceph-mon[32701]: 4.b scrub ok 2024-09-11T08:21:12.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:12 smithi181 ceph-mon[32701]: pgmap v622: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.0 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 753 B/s rd, 0 B/s wr, 846 op/s 2024-09-11T08:21:12.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:12 smithi181 ceph-mon[32701]: 5.11 scrub starts 2024-09-11T08:21:12.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:12 smithi181 ceph-mon[32701]: 5.11 scrub ok 2024-09-11T08:21:12.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:12 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2238027589' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:21:12.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:12 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:21:12.616 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:21:12.644 INFO:teuthology.orchestra.run.smithi150.stderr:noscrub is set 2024-09-11T08:21:12.656 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:12.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:12.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:21:12.758 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:12.795 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:12.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:12.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:12.897 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:12.933 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:12.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:12.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:12.989 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-11T08:21:13.038 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:13.072 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:13.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:13.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:13.174 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:13.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:13.211 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:13.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:13.312 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:21:13.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:13.346 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:13.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:21:13.426 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:21:13.447 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:21:13.483 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:13.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:13.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:21:13.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:13 smithi150 ceph-mon[29276]: 3.16 scrub starts 2024-09-11T08:21:13.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:13 smithi150 ceph-mon[29276]: 3.16 scrub ok 2024-09-11T08:21:13.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:13 smithi150 ceph-mon[29276]: 4.f scrub starts 2024-09-11T08:21:13.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:13 smithi150 ceph-mon[29276]: 4.f scrub ok 2024-09-11T08:21:13.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:13 smithi150 ceph-mon[29276]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:21:13.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:13 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:21:13.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:13 smithi150 ceph-mon[29276]: osdmap e301: 8 total, 8 up, 8 in 2024-09-11T08:21:13.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:13 smithi150 ceph-mon[39612]: 3.16 scrub starts 2024-09-11T08:21:13.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:13 smithi150 ceph-mon[39612]: 3.16 scrub ok 2024-09-11T08:21:13.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:13 smithi150 ceph-mon[39612]: 4.f scrub starts 2024-09-11T08:21:13.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:13 smithi150 ceph-mon[39612]: 4.f scrub ok 2024-09-11T08:21:13.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:13 smithi150 ceph-mon[39612]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:21:13.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:13 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:21:13.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:13 smithi150 ceph-mon[39612]: osdmap e301: 8 total, 8 up, 8 in 2024-09-11T08:21:13.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:13 smithi181 ceph-mon[32701]: 3.16 scrub starts 2024-09-11T08:21:13.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:13 smithi181 ceph-mon[32701]: 3.16 scrub ok 2024-09-11T08:21:13.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:13 smithi181 ceph-mon[32701]: 4.f scrub starts 2024-09-11T08:21:13.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:13 smithi181 ceph-mon[32701]: 4.f scrub ok 2024-09-11T08:21:13.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:13 smithi181 ceph-mon[32701]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:21:13.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:13 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:21:13.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:13 smithi181 ceph-mon[32701]: osdmap e301: 8 total, 8 up, 8 in 2024-09-11T08:21:13.584 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:21:13.618 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:13.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:13.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:21:13.720 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:21:13.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:13.762 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:13.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:21:13.851 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-11T08:21:13.851 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T08:21:13.851 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-11T08:21:13.851 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-11T08:21:13.864 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:13.897 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:13.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:13.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:13.999 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:14.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:14.036 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:14.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:14.136 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:14.165 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:14.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:14.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:14.267 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:14.276 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:21:14.307 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:14.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:14.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:14.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:14 smithi181 ceph-mon[32701]: pgmap v624: 148 pgs: 148 active+clean; 996 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.09k op/s; 18 B/s, 0 objects/s recovering 2024-09-11T08:21:14.411 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:14.450 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:14.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:14.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:14.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:14 smithi150 ceph-mon[29276]: pgmap v624: 148 pgs: 148 active+clean; 996 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.09k op/s; 18 B/s, 0 objects/s recovering 2024-09-11T08:21:14.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:14 smithi150 ceph-mon[39612]: pgmap v624: 148 pgs: 148 active+clean; 996 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.09k op/s; 18 B/s, 0 objects/s recovering 2024-09-11T08:21:14.552 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:14.589 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:14.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:14.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:14.691 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:14.726 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:14.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:14.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:14.828 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:21:14.855 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T08:21:14.866 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:14.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:14.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:21:14.969 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:21:15.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:15.003 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:15.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:21:15.104 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:21:15.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:15.143 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:15.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:21:15.244 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:15.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:15.278 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:15.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:15.323 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:21:15 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:21:15] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:21:15.379 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:21:15.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:15.418 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:15.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:21:15.519 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:21:15.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:15.556 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:15.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:21:15.589 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T08:21:15.657 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:15.693 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:15.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:15.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:15.794 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:21:15.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:15.837 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:15.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:21:15.938 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:21:15.939 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-11T08:21:15.939 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:21:15.975 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:15.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:15.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:21:16.076 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:16.108 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:16.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:16.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:16.213 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:21:16.253 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:16.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:16.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:21:16.354 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:16.389 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:16.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:16.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:16.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:16 smithi150 ceph-mon[39612]: pgmap v625: 148 pgs: 148 active+clean; 653 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 820 B/s rd, 0 B/s wr, 1.25k op/s; 17 B/s, 0 objects/s recovering 2024-09-11T08:21:16.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:16 smithi150 ceph-mon[29276]: pgmap v625: 148 pgs: 148 active+clean; 653 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 820 B/s rd, 0 B/s wr, 1.25k op/s; 17 B/s, 0 objects/s recovering 2024-09-11T08:21:16.491 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:16.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:16.527 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:16.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:16.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:16 smithi181 ceph-mon[32701]: pgmap v625: 148 pgs: 148 active+clean; 653 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 820 B/s rd, 0 B/s wr, 1.25k op/s; 17 B/s, 0 objects/s recovering 2024-09-11T08:21:16.629 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:21:16.663 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:16.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:16.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:21:16.665 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-11T08:21:16.765 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:21:16.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:16.812 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:16.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:21:16.913 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:16.947 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:16.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:16.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:17.048 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:21:17.084 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:17.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:17.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:21:17.133 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:21:17.186 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:17.223 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:17.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:17.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:17.306 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:21:17.324 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:17.355 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:17.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:17.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:17.445 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:17 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:21:17.445 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:17 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:21:17.446 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:17 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:21:17.446 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:17 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:21:17.457 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:17.500 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:17.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:17.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:17.555 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:17 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:21:17.555 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:17 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:21:17.561 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:21:17.562 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-11T08:21:17.562 INFO:teuthology.orchestra.run.smithi150.stdout: "epoch": 302, 2024-09-11T08:21:17.562 INFO:teuthology.orchestra.run.smithi150.stdout: "fsid": "c1bddff4-7013-11ef-bcea-c7b262605968", 2024-09-11T08:21:17.562 INFO:teuthology.orchestra.run.smithi150.stdout: "created": "2024-09-11T08:02:15.344618+0000", 2024-09-11T08:21:17.562 INFO:teuthology.orchestra.run.smithi150.stdout: "modified": "2024-09-11T08:21:17.224617+0000", 2024-09-11T08:21:17.562 INFO:teuthology.orchestra.run.smithi150.stdout: "last_up_change": "2024-09-11T08:14:23.745449+0000", 2024-09-11T08:21:17.562 INFO:teuthology.orchestra.run.smithi150.stdout: "last_in_change": "2024-09-11T08:19:28.121267+0000", 2024-09-11T08:21:17.562 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-11T08:21:17.562 INFO:teuthology.orchestra.run.smithi150.stdout: "flags_num": 5801984, 2024-09-11T08:21:17.562 INFO:teuthology.orchestra.run.smithi150.stdout: "flags_set": [ 2024-09-11T08:21:17.562 INFO:teuthology.orchestra.run.smithi150.stdout: "noscrub", 2024-09-11T08:21:17.562 INFO:teuthology.orchestra.run.smithi150.stdout: "pglog_hardlimit", 2024-09-11T08:21:17.562 INFO:teuthology.orchestra.run.smithi150.stdout: "purged_snapdirs", 2024-09-11T08:21:17.562 INFO:teuthology.orchestra.run.smithi150.stdout: "recovery_deletes", 2024-09-11T08:21:17.563 INFO:teuthology.orchestra.run.smithi150.stdout: "sortbitwise" 2024-09-11T08:21:17.563 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-09-11T08:21:17.563 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_version": 19, 2024-09-11T08:21:17.563 INFO:teuthology.orchestra.run.smithi150.stdout: "full_ratio": 0.94999998807907104, 2024-09-11T08:21:17.563 INFO:teuthology.orchestra.run.smithi150.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-11T08:21:17.563 INFO:teuthology.orchestra.run.smithi150.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-11T08:21:17.563 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_snapshot": "", 2024-09-11T08:21:17.563 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_max": 9, 2024-09-11T08:21:17.563 INFO:teuthology.orchestra.run.smithi150.stdout: "max_osd": 8, 2024-09-11T08:21:17.563 INFO:teuthology.orchestra.run.smithi150.stdout: "require_min_compat_client": "luminous", 2024-09-11T08:21:17.563 INFO:teuthology.orchestra.run.smithi150.stdout: "min_compat_client": "luminous", 2024-09-11T08:21:17.563 INFO:teuthology.orchestra.run.smithi150.stdout: "require_osd_release": "quincy", 2024-09-11T08:21:17.563 INFO:teuthology.orchestra.run.smithi150.stdout: "pools": [ 2024-09-11T08:21:17.564 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.564 INFO:teuthology.orchestra.run.smithi150.stdout: "pool": 1, 2024-09-11T08:21:17.564 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_name": ".mgr", 2024-09-11T08:21:17.564 INFO:teuthology.orchestra.run.smithi150.stdout: "create_time": "2024-09-11T08:05:21.044729+0000", 2024-09-11T08:21:17.564 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-09-11T08:21:17.564 INFO:teuthology.orchestra.run.smithi150.stdout: "flags_names": "hashpspool", 2024-09-11T08:21:17.564 INFO:teuthology.orchestra.run.smithi150.stdout: "type": 1, 2024-09-11T08:21:17.564 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 3, 2024-09-11T08:21:17.564 INFO:teuthology.orchestra.run.smithi150.stdout: "min_size": 2, 2024-09-11T08:21:17.564 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_rule": 0, 2024-09-11T08:21:17.564 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_count": 0, 2024-09-11T08:21:17.564 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_target": 0, 2024-09-11T08:21:17.564 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T08:21:17.565 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T08:21:17.565 INFO:teuthology.orchestra.run.smithi150.stdout: "object_hash": 2, 2024-09-11T08:21:17.565 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_autoscale_mode": "off", 2024-09-11T08:21:17.565 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num": 1, 2024-09-11T08:21:17.565 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num": 1, 2024-09-11T08:21:17.565 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num_target": 1, 2024-09-11T08:21:17.565 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_target": 1, 2024-09-11T08:21:17.565 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_pending": 1, 2024-09-11T08:21:17.565 INFO:teuthology.orchestra.run.smithi150.stdout: "last_pg_merge_meta": { 2024-09-11T08:21:17.565 INFO:teuthology.orchestra.run.smithi150.stdout: "source_pgid": "0.0", 2024-09-11T08:21:17.565 INFO:teuthology.orchestra.run.smithi150.stdout: "ready_epoch": 0, 2024-09-11T08:21:17.565 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_started": 0, 2024-09-11T08:21:17.565 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_clean": 0, 2024-09-11T08:21:17.565 INFO:teuthology.orchestra.run.smithi150.stdout: "source_version": "0'0", 2024-09-11T08:21:17.566 INFO:teuthology.orchestra.run.smithi150.stdout: "target_version": "0'0" 2024-09-11T08:21:17.566 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.566 INFO:teuthology.orchestra.run.smithi150.stdout: "last_change": "255", 2024-09-11T08:21:17.566 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend": "0", 2024-09-11T08:21:17.566 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T08:21:17.566 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T08:21:17.566 INFO:teuthology.orchestra.run.smithi150.stdout: "auid": 0, 2024-09-11T08:21:17.566 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_mode": "selfmanaged", 2024-09-11T08:21:17.566 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_seq": 0, 2024-09-11T08:21:17.566 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_epoch": 0, 2024-09-11T08:21:17.566 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_snaps": [], 2024-09-11T08:21:17.566 INFO:teuthology.orchestra.run.smithi150.stdout: "removed_snaps": "[]", 2024-09-11T08:21:17.566 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_bytes": 0, 2024-09-11T08:21:17.566 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_objects": 0, 2024-09-11T08:21:17.566 INFO:teuthology.orchestra.run.smithi150.stdout: "tiers": [], 2024-09-11T08:21:17.567 INFO:teuthology.orchestra.run.smithi150.stdout: "tier_of": -1, 2024-09-11T08:21:17.567 INFO:teuthology.orchestra.run.smithi150.stdout: "read_tier": -1, 2024-09-11T08:21:17.567 INFO:teuthology.orchestra.run.smithi150.stdout: "write_tier": -1, 2024-09-11T08:21:17.567 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_mode": "none", 2024-09-11T08:21:17.567 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_bytes": 0, 2024-09-11T08:21:17.567 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_objects": 0, 2024-09-11T08:21:17.567 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T08:21:17.567 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T08:21:17.567 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T08:21:17.567 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_flush_age": 0, 2024-09-11T08:21:17.567 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_evict_age": 0, 2024-09-11T08:21:17.567 INFO:teuthology.orchestra.run.smithi150.stdout: "erasure_code_profile": "", 2024-09-11T08:21:17.567 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_params": { 2024-09-11T08:21:17.567 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "none" 2024-09-11T08:21:17.567 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.567 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_period": 0, 2024-09-11T08:21:17.568 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_count": 0, 2024-09-11T08:21:17.568 INFO:teuthology.orchestra.run.smithi150.stdout: "use_gmt_hitset": true, 2024-09-11T08:21:17.568 INFO:teuthology.orchestra.run.smithi150.stdout: "min_read_recency_for_promote": 0, 2024-09-11T08:21:17.568 INFO:teuthology.orchestra.run.smithi150.stdout: "min_write_recency_for_promote": 0, 2024-09-11T08:21:17.568 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T08:21:17.568 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_search_last_n": 0, 2024-09-11T08:21:17.568 INFO:teuthology.orchestra.run.smithi150.stdout: "grade_table": [], 2024-09-11T08:21:17.568 INFO:teuthology.orchestra.run.smithi150.stdout: "stripe_width": 0, 2024-09-11T08:21:17.568 INFO:teuthology.orchestra.run.smithi150.stdout: "expected_num_objects": 0, 2024-09-11T08:21:17.568 INFO:teuthology.orchestra.run.smithi150.stdout: "fast_read": false, 2024-09-11T08:21:17.568 INFO:teuthology.orchestra.run.smithi150.stdout: "options": { 2024-09-11T08:21:17.568 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_max": 32, 2024-09-11T08:21:17.568 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_min": 1 2024-09-11T08:21:17.569 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.569 INFO:teuthology.orchestra.run.smithi150.stdout: "application_metadata": { 2024-09-11T08:21:17.569 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": {} 2024-09-11T08:21:17.569 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:21:17.569 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.569 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.569 INFO:teuthology.orchestra.run.smithi150.stdout: "pool": 2, 2024-09-11T08:21:17.569 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_name": "datapool", 2024-09-11T08:21:17.569 INFO:teuthology.orchestra.run.smithi150.stdout: "create_time": "2024-09-11T08:07:28.786572+0000", 2024-09-11T08:21:17.569 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 8193, 2024-09-11T08:21:17.569 INFO:teuthology.orchestra.run.smithi150.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-11T08:21:17.569 INFO:teuthology.orchestra.run.smithi150.stdout: "type": 1, 2024-09-11T08:21:17.569 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 3, 2024-09-11T08:21:17.570 INFO:teuthology.orchestra.run.smithi150.stdout: "min_size": 2, 2024-09-11T08:21:17.570 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_rule": 0, 2024-09-11T08:21:17.570 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_count": 0, 2024-09-11T08:21:17.570 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_target": 0, 2024-09-11T08:21:17.570 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T08:21:17.570 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T08:21:17.570 INFO:teuthology.orchestra.run.smithi150.stdout: "object_hash": 2, 2024-09-11T08:21:17.570 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_autoscale_mode": "off", 2024-09-11T08:21:17.570 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num": 3, 2024-09-11T08:21:17.570 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num": 3, 2024-09-11T08:21:17.570 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num_target": 3, 2024-09-11T08:21:17.570 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_target": 3, 2024-09-11T08:21:17.570 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_pending": 3, 2024-09-11T08:21:17.570 INFO:teuthology.orchestra.run.smithi150.stdout: "last_pg_merge_meta": { 2024-09-11T08:21:17.571 INFO:teuthology.orchestra.run.smithi150.stdout: "source_pgid": "0.0", 2024-09-11T08:21:17.571 INFO:teuthology.orchestra.run.smithi150.stdout: "ready_epoch": 0, 2024-09-11T08:21:17.571 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_started": 0, 2024-09-11T08:21:17.571 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_clean": 0, 2024-09-11T08:21:17.571 INFO:teuthology.orchestra.run.smithi150.stdout: "source_version": "0'0", 2024-09-11T08:21:17.571 INFO:teuthology.orchestra.run.smithi150.stdout: "target_version": "0'0" 2024-09-11T08:21:17.571 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.571 INFO:teuthology.orchestra.run.smithi150.stdout: "last_change": "63", 2024-09-11T08:21:17.571 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend": "0", 2024-09-11T08:21:17.571 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T08:21:17.571 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T08:21:17.571 INFO:teuthology.orchestra.run.smithi150.stdout: "auid": 0, 2024-09-11T08:21:17.571 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_mode": "selfmanaged", 2024-09-11T08:21:17.571 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_seq": 3, 2024-09-11T08:21:17.572 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_epoch": 63, 2024-09-11T08:21:17.572 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_snaps": [], 2024-09-11T08:21:17.572 INFO:teuthology.orchestra.run.smithi150.stdout: "removed_snaps": "[]", 2024-09-11T08:21:17.572 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_bytes": 0, 2024-09-11T08:21:17.572 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_objects": 0, 2024-09-11T08:21:17.572 INFO:teuthology.orchestra.run.smithi150.stdout: "tiers": [], 2024-09-11T08:21:17.572 INFO:teuthology.orchestra.run.smithi150.stdout: "tier_of": -1, 2024-09-11T08:21:17.572 INFO:teuthology.orchestra.run.smithi150.stdout: "read_tier": -1, 2024-09-11T08:21:17.572 INFO:teuthology.orchestra.run.smithi150.stdout: "write_tier": -1, 2024-09-11T08:21:17.572 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_mode": "none", 2024-09-11T08:21:17.572 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_bytes": 0, 2024-09-11T08:21:17.572 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_objects": 0, 2024-09-11T08:21:17.572 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T08:21:17.572 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T08:21:17.573 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T08:21:17.573 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_flush_age": 0, 2024-09-11T08:21:17.573 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_evict_age": 0, 2024-09-11T08:21:17.573 INFO:teuthology.orchestra.run.smithi150.stdout: "erasure_code_profile": "", 2024-09-11T08:21:17.573 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_params": { 2024-09-11T08:21:17.573 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "none" 2024-09-11T08:21:17.573 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.573 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_period": 0, 2024-09-11T08:21:17.573 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_count": 0, 2024-09-11T08:21:17.573 INFO:teuthology.orchestra.run.smithi150.stdout: "use_gmt_hitset": true, 2024-09-11T08:21:17.573 INFO:teuthology.orchestra.run.smithi150.stdout: "min_read_recency_for_promote": 0, 2024-09-11T08:21:17.573 INFO:teuthology.orchestra.run.smithi150.stdout: "min_write_recency_for_promote": 0, 2024-09-11T08:21:17.573 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T08:21:17.573 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_search_last_n": 0, 2024-09-11T08:21:17.574 INFO:teuthology.orchestra.run.smithi150.stdout: "grade_table": [], 2024-09-11T08:21:17.574 INFO:teuthology.orchestra.run.smithi150.stdout: "stripe_width": 0, 2024-09-11T08:21:17.574 INFO:teuthology.orchestra.run.smithi150.stdout: "expected_num_objects": 0, 2024-09-11T08:21:17.574 INFO:teuthology.orchestra.run.smithi150.stdout: "fast_read": false, 2024-09-11T08:21:17.574 INFO:teuthology.orchestra.run.smithi150.stdout: "options": {}, 2024-09-11T08:21:17.574 INFO:teuthology.orchestra.run.smithi150.stdout: "application_metadata": { 2024-09-11T08:21:17.574 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd": {} 2024-09-11T08:21:17.574 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:21:17.574 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.574 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.574 INFO:teuthology.orchestra.run.smithi150.stdout: "pool": 3, 2024-09-11T08:21:17.574 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_name": ".rgw.root", 2024-09-11T08:21:17.574 INFO:teuthology.orchestra.run.smithi150.stdout: "create_time": "2024-09-11T08:07:29.021286+0000", 2024-09-11T08:21:17.574 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-09-11T08:21:17.574 INFO:teuthology.orchestra.run.smithi150.stdout: "flags_names": "hashpspool", 2024-09-11T08:21:17.575 INFO:teuthology.orchestra.run.smithi150.stdout: "type": 1, 2024-09-11T08:21:17.575 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 3, 2024-09-11T08:21:17.575 INFO:teuthology.orchestra.run.smithi150.stdout: "min_size": 2, 2024-09-11T08:21:17.575 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_rule": 0, 2024-09-11T08:21:17.575 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_count": 0, 2024-09-11T08:21:17.575 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_target": 0, 2024-09-11T08:21:17.575 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T08:21:17.575 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T08:21:17.575 INFO:teuthology.orchestra.run.smithi150.stdout: "object_hash": 2, 2024-09-11T08:21:17.575 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_autoscale_mode": "off", 2024-09-11T08:21:17.575 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num": 32, 2024-09-11T08:21:17.575 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num": 32, 2024-09-11T08:21:17.575 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num_target": 32, 2024-09-11T08:21:17.575 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_target": 32, 2024-09-11T08:21:17.575 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_pending": 32, 2024-09-11T08:21:17.576 INFO:teuthology.orchestra.run.smithi150.stdout: "last_pg_merge_meta": { 2024-09-11T08:21:17.576 INFO:teuthology.orchestra.run.smithi150.stdout: "source_pgid": "0.0", 2024-09-11T08:21:17.576 INFO:teuthology.orchestra.run.smithi150.stdout: "ready_epoch": 0, 2024-09-11T08:21:17.576 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_started": 0, 2024-09-11T08:21:17.576 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_clean": 0, 2024-09-11T08:21:17.576 INFO:teuthology.orchestra.run.smithi150.stdout: "source_version": "0'0", 2024-09-11T08:21:17.576 INFO:teuthology.orchestra.run.smithi150.stdout: "target_version": "0'0" 2024-09-11T08:21:17.576 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.576 INFO:teuthology.orchestra.run.smithi150.stdout: "last_change": "60", 2024-09-11T08:21:17.576 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend": "0", 2024-09-11T08:21:17.576 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T08:21:17.576 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T08:21:17.576 INFO:teuthology.orchestra.run.smithi150.stdout: "auid": 0, 2024-09-11T08:21:17.577 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_mode": "selfmanaged", 2024-09-11T08:21:17.577 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_seq": 0, 2024-09-11T08:21:17.577 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_epoch": 0, 2024-09-11T08:21:17.577 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_snaps": [], 2024-09-11T08:21:17.577 INFO:teuthology.orchestra.run.smithi150.stdout: "removed_snaps": "[]", 2024-09-11T08:21:17.577 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_bytes": 0, 2024-09-11T08:21:17.577 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_objects": 0, 2024-09-11T08:21:17.577 INFO:teuthology.orchestra.run.smithi150.stdout: "tiers": [], 2024-09-11T08:21:17.577 INFO:teuthology.orchestra.run.smithi150.stdout: "tier_of": -1, 2024-09-11T08:21:17.577 INFO:teuthology.orchestra.run.smithi150.stdout: "read_tier": -1, 2024-09-11T08:21:17.577 INFO:teuthology.orchestra.run.smithi150.stdout: "write_tier": -1, 2024-09-11T08:21:17.577 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_mode": "none", 2024-09-11T08:21:17.577 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_bytes": 0, 2024-09-11T08:21:17.578 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_objects": 0, 2024-09-11T08:21:17.578 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T08:21:17.578 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T08:21:17.578 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T08:21:17.578 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_flush_age": 0, 2024-09-11T08:21:17.578 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_evict_age": 0, 2024-09-11T08:21:17.578 INFO:teuthology.orchestra.run.smithi150.stdout: "erasure_code_profile": "", 2024-09-11T08:21:17.578 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_params": { 2024-09-11T08:21:17.578 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "none" 2024-09-11T08:21:17.578 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.579 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_period": 0, 2024-09-11T08:21:17.579 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_count": 0, 2024-09-11T08:21:17.579 INFO:teuthology.orchestra.run.smithi150.stdout: "use_gmt_hitset": true, 2024-09-11T08:21:17.579 INFO:teuthology.orchestra.run.smithi150.stdout: "min_read_recency_for_promote": 0, 2024-09-11T08:21:17.579 INFO:teuthology.orchestra.run.smithi150.stdout: "min_write_recency_for_promote": 0, 2024-09-11T08:21:17.579 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T08:21:17.579 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_search_last_n": 0, 2024-09-11T08:21:17.579 INFO:teuthology.orchestra.run.smithi150.stdout: "grade_table": [], 2024-09-11T08:21:17.579 INFO:teuthology.orchestra.run.smithi150.stdout: "stripe_width": 0, 2024-09-11T08:21:17.580 INFO:teuthology.orchestra.run.smithi150.stdout: "expected_num_objects": 0, 2024-09-11T08:21:17.580 INFO:teuthology.orchestra.run.smithi150.stdout: "fast_read": false, 2024-09-11T08:21:17.580 INFO:teuthology.orchestra.run.smithi150.stdout: "options": {}, 2024-09-11T08:21:17.580 INFO:teuthology.orchestra.run.smithi150.stdout: "application_metadata": { 2024-09-11T08:21:17.580 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw": {} 2024-09-11T08:21:17.580 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:21:17.580 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.580 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.580 INFO:teuthology.orchestra.run.smithi150.stdout: "pool": 4, 2024-09-11T08:21:17.580 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_name": "default.rgw.log", 2024-09-11T08:21:17.580 INFO:teuthology.orchestra.run.smithi150.stdout: "create_time": "2024-09-11T08:07:30.334181+0000", 2024-09-11T08:21:17.580 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-09-11T08:21:17.580 INFO:teuthology.orchestra.run.smithi150.stdout: "flags_names": "hashpspool", 2024-09-11T08:21:17.580 INFO:teuthology.orchestra.run.smithi150.stdout: "type": 1, 2024-09-11T08:21:17.581 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 3, 2024-09-11T08:21:17.581 INFO:teuthology.orchestra.run.smithi150.stdout: "min_size": 2, 2024-09-11T08:21:17.581 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_rule": 0, 2024-09-11T08:21:17.581 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_count": 0, 2024-09-11T08:21:17.581 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_target": 0, 2024-09-11T08:21:17.581 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T08:21:17.581 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T08:21:17.581 INFO:teuthology.orchestra.run.smithi150.stdout: "object_hash": 2, 2024-09-11T08:21:17.581 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_autoscale_mode": "off", 2024-09-11T08:21:17.581 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num": 32, 2024-09-11T08:21:17.581 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num": 32, 2024-09-11T08:21:17.581 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num_target": 32, 2024-09-11T08:21:17.581 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_target": 32, 2024-09-11T08:21:17.582 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_pending": 32, 2024-09-11T08:21:17.582 INFO:teuthology.orchestra.run.smithi150.stdout: "last_pg_merge_meta": { 2024-09-11T08:21:17.582 INFO:teuthology.orchestra.run.smithi150.stdout: "source_pgid": "0.0", 2024-09-11T08:21:17.582 INFO:teuthology.orchestra.run.smithi150.stdout: "ready_epoch": 0, 2024-09-11T08:21:17.582 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_started": 0, 2024-09-11T08:21:17.582 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_clean": 0, 2024-09-11T08:21:17.582 INFO:teuthology.orchestra.run.smithi150.stdout: "source_version": "0'0", 2024-09-11T08:21:17.582 INFO:teuthology.orchestra.run.smithi150.stdout: "target_version": "0'0" 2024-09-11T08:21:17.582 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.582 INFO:teuthology.orchestra.run.smithi150.stdout: "last_change": "62", 2024-09-11T08:21:17.582 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend": "0", 2024-09-11T08:21:17.582 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T08:21:17.582 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T08:21:17.583 INFO:teuthology.orchestra.run.smithi150.stdout: "auid": 0, 2024-09-11T08:21:17.583 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_mode": "selfmanaged", 2024-09-11T08:21:17.583 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_seq": 0, 2024-09-11T08:21:17.583 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_epoch": 0, 2024-09-11T08:21:17.583 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_snaps": [], 2024-09-11T08:21:17.583 INFO:teuthology.orchestra.run.smithi150.stdout: "removed_snaps": "[]", 2024-09-11T08:21:17.583 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_bytes": 0, 2024-09-11T08:21:17.583 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_objects": 0, 2024-09-11T08:21:17.583 INFO:teuthology.orchestra.run.smithi150.stdout: "tiers": [], 2024-09-11T08:21:17.583 INFO:teuthology.orchestra.run.smithi150.stdout: "tier_of": -1, 2024-09-11T08:21:17.583 INFO:teuthology.orchestra.run.smithi150.stdout: "read_tier": -1, 2024-09-11T08:21:17.583 INFO:teuthology.orchestra.run.smithi150.stdout: "write_tier": -1, 2024-09-11T08:21:17.583 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_mode": "none", 2024-09-11T08:21:17.583 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_bytes": 0, 2024-09-11T08:21:17.583 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_objects": 0, 2024-09-11T08:21:17.584 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T08:21:17.584 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T08:21:17.584 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T08:21:17.584 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_flush_age": 0, 2024-09-11T08:21:17.584 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_evict_age": 0, 2024-09-11T08:21:17.584 INFO:teuthology.orchestra.run.smithi150.stdout: "erasure_code_profile": "", 2024-09-11T08:21:17.584 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_params": { 2024-09-11T08:21:17.584 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "none" 2024-09-11T08:21:17.584 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.584 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_period": 0, 2024-09-11T08:21:17.584 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_count": 0, 2024-09-11T08:21:17.584 INFO:teuthology.orchestra.run.smithi150.stdout: "use_gmt_hitset": true, 2024-09-11T08:21:17.584 INFO:teuthology.orchestra.run.smithi150.stdout: "min_read_recency_for_promote": 0, 2024-09-11T08:21:17.584 INFO:teuthology.orchestra.run.smithi150.stdout: "min_write_recency_for_promote": 0, 2024-09-11T08:21:17.585 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T08:21:17.585 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_search_last_n": 0, 2024-09-11T08:21:17.585 INFO:teuthology.orchestra.run.smithi150.stdout: "grade_table": [], 2024-09-11T08:21:17.585 INFO:teuthology.orchestra.run.smithi150.stdout: "stripe_width": 0, 2024-09-11T08:21:17.585 INFO:teuthology.orchestra.run.smithi150.stdout: "expected_num_objects": 0, 2024-09-11T08:21:17.585 INFO:teuthology.orchestra.run.smithi150.stdout: "fast_read": false, 2024-09-11T08:21:17.585 INFO:teuthology.orchestra.run.smithi150.stdout: "options": {}, 2024-09-11T08:21:17.585 INFO:teuthology.orchestra.run.smithi150.stdout: "application_metadata": { 2024-09-11T08:21:17.585 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw": {} 2024-09-11T08:21:17.585 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:21:17.585 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.585 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.585 INFO:teuthology.orchestra.run.smithi150.stdout: "pool": 5, 2024-09-11T08:21:17.585 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_name": "default.rgw.control", 2024-09-11T08:21:17.586 INFO:teuthology.orchestra.run.smithi150.stdout: "create_time": "2024-09-11T08:07:32.289626+0000", 2024-09-11T08:21:17.586 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-09-11T08:21:17.586 INFO:teuthology.orchestra.run.smithi150.stdout: "flags_names": "hashpspool", 2024-09-11T08:21:17.586 INFO:teuthology.orchestra.run.smithi150.stdout: "type": 1, 2024-09-11T08:21:17.586 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 3, 2024-09-11T08:21:17.586 INFO:teuthology.orchestra.run.smithi150.stdout: "min_size": 2, 2024-09-11T08:21:17.586 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_rule": 0, 2024-09-11T08:21:17.586 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_count": 0, 2024-09-11T08:21:17.586 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_target": 0, 2024-09-11T08:21:17.586 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T08:21:17.586 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T08:21:17.586 INFO:teuthology.orchestra.run.smithi150.stdout: "object_hash": 2, 2024-09-11T08:21:17.586 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_autoscale_mode": "off", 2024-09-11T08:21:17.586 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num": 32, 2024-09-11T08:21:17.587 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num": 32, 2024-09-11T08:21:17.587 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num_target": 32, 2024-09-11T08:21:17.587 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_target": 32, 2024-09-11T08:21:17.587 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_pending": 32, 2024-09-11T08:21:17.587 INFO:teuthology.orchestra.run.smithi150.stdout: "last_pg_merge_meta": { 2024-09-11T08:21:17.587 INFO:teuthology.orchestra.run.smithi150.stdout: "source_pgid": "0.0", 2024-09-11T08:21:17.587 INFO:teuthology.orchestra.run.smithi150.stdout: "ready_epoch": 0, 2024-09-11T08:21:17.587 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_started": 0, 2024-09-11T08:21:17.587 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_clean": 0, 2024-09-11T08:21:17.587 INFO:teuthology.orchestra.run.smithi150.stdout: "source_version": "0'0", 2024-09-11T08:21:17.587 INFO:teuthology.orchestra.run.smithi150.stdout: "target_version": "0'0" 2024-09-11T08:21:17.587 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.587 INFO:teuthology.orchestra.run.smithi150.stdout: "last_change": "64", 2024-09-11T08:21:17.587 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend": "0", 2024-09-11T08:21:17.588 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T08:21:17.588 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T08:21:17.588 INFO:teuthology.orchestra.run.smithi150.stdout: "auid": 0, 2024-09-11T08:21:17.588 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_mode": "selfmanaged", 2024-09-11T08:21:17.588 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_seq": 0, 2024-09-11T08:21:17.588 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_epoch": 0, 2024-09-11T08:21:17.588 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_snaps": [], 2024-09-11T08:21:17.588 INFO:teuthology.orchestra.run.smithi150.stdout: "removed_snaps": "[]", 2024-09-11T08:21:17.588 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_bytes": 0, 2024-09-11T08:21:17.588 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_objects": 0, 2024-09-11T08:21:17.588 INFO:teuthology.orchestra.run.smithi150.stdout: "tiers": [], 2024-09-11T08:21:17.588 INFO:teuthology.orchestra.run.smithi150.stdout: "tier_of": -1, 2024-09-11T08:21:17.588 INFO:teuthology.orchestra.run.smithi150.stdout: "read_tier": -1, 2024-09-11T08:21:17.588 INFO:teuthology.orchestra.run.smithi150.stdout: "write_tier": -1, 2024-09-11T08:21:17.589 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_mode": "none", 2024-09-11T08:21:17.589 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_bytes": 0, 2024-09-11T08:21:17.589 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_objects": 0, 2024-09-11T08:21:17.589 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T08:21:17.589 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T08:21:17.589 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T08:21:17.589 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_flush_age": 0, 2024-09-11T08:21:17.589 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_evict_age": 0, 2024-09-11T08:21:17.589 INFO:teuthology.orchestra.run.smithi150.stdout: "erasure_code_profile": "", 2024-09-11T08:21:17.589 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_params": { 2024-09-11T08:21:17.589 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "none" 2024-09-11T08:21:17.589 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.589 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_period": 0, 2024-09-11T08:21:17.589 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_count": 0, 2024-09-11T08:21:17.590 INFO:teuthology.orchestra.run.smithi150.stdout: "use_gmt_hitset": true, 2024-09-11T08:21:17.590 INFO:teuthology.orchestra.run.smithi150.stdout: "min_read_recency_for_promote": 0, 2024-09-11T08:21:17.590 INFO:teuthology.orchestra.run.smithi150.stdout: "min_write_recency_for_promote": 0, 2024-09-11T08:21:17.590 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T08:21:17.590 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_search_last_n": 0, 2024-09-11T08:21:17.590 INFO:teuthology.orchestra.run.smithi150.stdout: "grade_table": [], 2024-09-11T08:21:17.590 INFO:teuthology.orchestra.run.smithi150.stdout: "stripe_width": 0, 2024-09-11T08:21:17.590 INFO:teuthology.orchestra.run.smithi150.stdout: "expected_num_objects": 0, 2024-09-11T08:21:17.590 INFO:teuthology.orchestra.run.smithi150.stdout: "fast_read": false, 2024-09-11T08:21:17.590 INFO:teuthology.orchestra.run.smithi150.stdout: "options": {}, 2024-09-11T08:21:17.590 INFO:teuthology.orchestra.run.smithi150.stdout: "application_metadata": { 2024-09-11T08:21:17.590 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw": {} 2024-09-11T08:21:17.590 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:21:17.590 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.590 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.591 INFO:teuthology.orchestra.run.smithi150.stdout: "pool": 6, 2024-09-11T08:21:17.591 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_name": "default.rgw.meta", 2024-09-11T08:21:17.591 INFO:teuthology.orchestra.run.smithi150.stdout: "create_time": "2024-09-11T08:07:34.351861+0000", 2024-09-11T08:21:17.591 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-09-11T08:21:17.591 INFO:teuthology.orchestra.run.smithi150.stdout: "flags_names": "hashpspool", 2024-09-11T08:21:17.591 INFO:teuthology.orchestra.run.smithi150.stdout: "type": 1, 2024-09-11T08:21:17.591 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 3, 2024-09-11T08:21:17.591 INFO:teuthology.orchestra.run.smithi150.stdout: "min_size": 2, 2024-09-11T08:21:17.591 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_rule": 0, 2024-09-11T08:21:17.591 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_count": 0, 2024-09-11T08:21:17.591 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_target": 0, 2024-09-11T08:21:17.591 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T08:21:17.591 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T08:21:17.592 INFO:teuthology.orchestra.run.smithi150.stdout: "object_hash": 2, 2024-09-11T08:21:17.592 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_autoscale_mode": "off", 2024-09-11T08:21:17.592 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num": 32, 2024-09-11T08:21:17.592 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num": 32, 2024-09-11T08:21:17.592 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num_target": 32, 2024-09-11T08:21:17.592 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_target": 32, 2024-09-11T08:21:17.592 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_pending": 32, 2024-09-11T08:21:17.592 INFO:teuthology.orchestra.run.smithi150.stdout: "last_pg_merge_meta": { 2024-09-11T08:21:17.592 INFO:teuthology.orchestra.run.smithi150.stdout: "source_pgid": "0.0", 2024-09-11T08:21:17.592 INFO:teuthology.orchestra.run.smithi150.stdout: "ready_epoch": 0, 2024-09-11T08:21:17.592 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_started": 0, 2024-09-11T08:21:17.592 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_clean": 0, 2024-09-11T08:21:17.592 INFO:teuthology.orchestra.run.smithi150.stdout: "source_version": "0'0", 2024-09-11T08:21:17.593 INFO:teuthology.orchestra.run.smithi150.stdout: "target_version": "0'0" 2024-09-11T08:21:17.593 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.593 INFO:teuthology.orchestra.run.smithi150.stdout: "last_change": "66", 2024-09-11T08:21:17.593 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend": "0", 2024-09-11T08:21:17.593 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T08:21:17.593 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T08:21:17.593 INFO:teuthology.orchestra.run.smithi150.stdout: "auid": 0, 2024-09-11T08:21:17.593 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_mode": "selfmanaged", 2024-09-11T08:21:17.593 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_seq": 0, 2024-09-11T08:21:17.593 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_epoch": 0, 2024-09-11T08:21:17.593 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_snaps": [], 2024-09-11T08:21:17.593 INFO:teuthology.orchestra.run.smithi150.stdout: "removed_snaps": "[]", 2024-09-11T08:21:17.593 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_bytes": 0, 2024-09-11T08:21:17.593 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_objects": 0, 2024-09-11T08:21:17.594 INFO:teuthology.orchestra.run.smithi150.stdout: "tiers": [], 2024-09-11T08:21:17.594 INFO:teuthology.orchestra.run.smithi150.stdout: "tier_of": -1, 2024-09-11T08:21:17.594 INFO:teuthology.orchestra.run.smithi150.stdout: "read_tier": -1, 2024-09-11T08:21:17.594 INFO:teuthology.orchestra.run.smithi150.stdout: "write_tier": -1, 2024-09-11T08:21:17.594 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_mode": "none", 2024-09-11T08:21:17.594 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_bytes": 0, 2024-09-11T08:21:17.594 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_objects": 0, 2024-09-11T08:21:17.594 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T08:21:17.594 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T08:21:17.594 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T08:21:17.594 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_flush_age": 0, 2024-09-11T08:21:17.594 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_evict_age": 0, 2024-09-11T08:21:17.594 INFO:teuthology.orchestra.run.smithi150.stdout: "erasure_code_profile": "", 2024-09-11T08:21:17.594 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_params": { 2024-09-11T08:21:17.595 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "none" 2024-09-11T08:21:17.595 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.595 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_period": 0, 2024-09-11T08:21:17.595 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_count": 0, 2024-09-11T08:21:17.595 INFO:teuthology.orchestra.run.smithi150.stdout: "use_gmt_hitset": true, 2024-09-11T08:21:17.595 INFO:teuthology.orchestra.run.smithi150.stdout: "min_read_recency_for_promote": 0, 2024-09-11T08:21:17.595 INFO:teuthology.orchestra.run.smithi150.stdout: "min_write_recency_for_promote": 0, 2024-09-11T08:21:17.595 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T08:21:17.595 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_search_last_n": 0, 2024-09-11T08:21:17.595 INFO:teuthology.orchestra.run.smithi150.stdout: "grade_table": [], 2024-09-11T08:21:17.595 INFO:teuthology.orchestra.run.smithi150.stdout: "stripe_width": 0, 2024-09-11T08:21:17.595 INFO:teuthology.orchestra.run.smithi150.stdout: "expected_num_objects": 0, 2024-09-11T08:21:17.595 INFO:teuthology.orchestra.run.smithi150.stdout: "fast_read": false, 2024-09-11T08:21:17.595 INFO:teuthology.orchestra.run.smithi150.stdout: "options": { 2024-09-11T08:21:17.596 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_autoscale_bias": 4 2024-09-11T08:21:17.596 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.596 INFO:teuthology.orchestra.run.smithi150.stdout: "application_metadata": { 2024-09-11T08:21:17.596 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw": {} 2024-09-11T08:21:17.596 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:21:17.596 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.596 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.596 INFO:teuthology.orchestra.run.smithi150.stdout: "pool": 9, 2024-09-11T08:21:17.596 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_name": "unique_pool_2", 2024-09-11T08:21:17.596 INFO:teuthology.orchestra.run.smithi150.stdout: "create_time": "2024-09-11T08:17:59.761532+0000", 2024-09-11T08:21:17.596 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-09-11T08:21:17.596 INFO:teuthology.orchestra.run.smithi150.stdout: "flags_names": "hashpspool", 2024-09-11T08:21:17.596 INFO:teuthology.orchestra.run.smithi150.stdout: "type": 1, 2024-09-11T08:21:17.597 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 3, 2024-09-11T08:21:17.597 INFO:teuthology.orchestra.run.smithi150.stdout: "min_size": 2, 2024-09-11T08:21:17.597 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_rule": 0, 2024-09-11T08:21:17.597 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_count": 0, 2024-09-11T08:21:17.597 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_target": 0, 2024-09-11T08:21:17.597 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T08:21:17.597 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T08:21:17.597 INFO:teuthology.orchestra.run.smithi150.stdout: "object_hash": 2, 2024-09-11T08:21:17.597 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_autoscale_mode": "off", 2024-09-11T08:21:17.597 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num": 16, 2024-09-11T08:21:17.597 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num": 16, 2024-09-11T08:21:17.597 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num_target": 16, 2024-09-11T08:21:17.597 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_target": 16, 2024-09-11T08:21:17.597 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_pending": 16, 2024-09-11T08:21:17.598 INFO:teuthology.orchestra.run.smithi150.stdout: "last_pg_merge_meta": { 2024-09-11T08:21:17.598 INFO:teuthology.orchestra.run.smithi150.stdout: "source_pgid": "0.0", 2024-09-11T08:21:17.598 INFO:teuthology.orchestra.run.smithi150.stdout: "ready_epoch": 0, 2024-09-11T08:21:17.598 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_started": 0, 2024-09-11T08:21:17.598 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_clean": 0, 2024-09-11T08:21:17.598 INFO:teuthology.orchestra.run.smithi150.stdout: "source_version": "0'0", 2024-09-11T08:21:17.598 INFO:teuthology.orchestra.run.smithi150.stdout: "target_version": "0'0" 2024-09-11T08:21:17.598 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.598 INFO:teuthology.orchestra.run.smithi150.stdout: "last_change": "251", 2024-09-11T08:21:17.598 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend": "0", 2024-09-11T08:21:17.598 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T08:21:17.598 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T08:21:17.598 INFO:teuthology.orchestra.run.smithi150.stdout: "auid": 0, 2024-09-11T08:21:17.598 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_mode": "selfmanaged", 2024-09-11T08:21:17.599 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_seq": 0, 2024-09-11T08:21:17.599 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_epoch": 0, 2024-09-11T08:21:17.599 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_snaps": [], 2024-09-11T08:21:17.599 INFO:teuthology.orchestra.run.smithi150.stdout: "removed_snaps": "[]", 2024-09-11T08:21:17.599 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_bytes": 0, 2024-09-11T08:21:17.599 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_objects": 0, 2024-09-11T08:21:17.599 INFO:teuthology.orchestra.run.smithi150.stdout: "tiers": [], 2024-09-11T08:21:17.599 INFO:teuthology.orchestra.run.smithi150.stdout: "tier_of": -1, 2024-09-11T08:21:17.599 INFO:teuthology.orchestra.run.smithi150.stdout: "read_tier": -1, 2024-09-11T08:21:17.599 INFO:teuthology.orchestra.run.smithi150.stdout: "write_tier": -1, 2024-09-11T08:21:17.599 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_mode": "none", 2024-09-11T08:21:17.599 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_bytes": 0, 2024-09-11T08:21:17.599 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_objects": 0, 2024-09-11T08:21:17.599 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T08:21:17.599 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T08:21:17.600 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T08:21:17.600 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_flush_age": 0, 2024-09-11T08:21:17.600 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_evict_age": 0, 2024-09-11T08:21:17.600 INFO:teuthology.orchestra.run.smithi150.stdout: "erasure_code_profile": "", 2024-09-11T08:21:17.600 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_params": { 2024-09-11T08:21:17.600 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "none" 2024-09-11T08:21:17.600 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.600 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_period": 0, 2024-09-11T08:21:17.600 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_count": 0, 2024-09-11T08:21:17.600 INFO:teuthology.orchestra.run.smithi150.stdout: "use_gmt_hitset": true, 2024-09-11T08:21:17.600 INFO:teuthology.orchestra.run.smithi150.stdout: "min_read_recency_for_promote": 0, 2024-09-11T08:21:17.600 INFO:teuthology.orchestra.run.smithi150.stdout: "min_write_recency_for_promote": 0, 2024-09-11T08:21:17.600 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T08:21:17.601 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_search_last_n": 0, 2024-09-11T08:21:17.601 INFO:teuthology.orchestra.run.smithi150.stdout: "grade_table": [], 2024-09-11T08:21:17.601 INFO:teuthology.orchestra.run.smithi150.stdout: "stripe_width": 0, 2024-09-11T08:21:17.601 INFO:teuthology.orchestra.run.smithi150.stdout: "expected_num_objects": 0, 2024-09-11T08:21:17.601 INFO:teuthology.orchestra.run.smithi150.stdout: "fast_read": false, 2024-09-11T08:21:17.601 INFO:teuthology.orchestra.run.smithi150.stdout: "options": {}, 2024-09-11T08:21:17.601 INFO:teuthology.orchestra.run.smithi150.stdout: "application_metadata": { 2024-09-11T08:21:17.601 INFO:teuthology.orchestra.run.smithi150.stdout: "rados": {} 2024-09-11T08:21:17.601 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:21:17.601 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:21:17.601 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-09-11T08:21:17.601 INFO:teuthology.orchestra.run.smithi150.stdout: "osds": [ 2024-09-11T08:21:17.601 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.601 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 0, 2024-09-11T08:21:17.602 INFO:teuthology.orchestra.run.smithi150.stdout: "uuid": "37b63db9-ed2a-4701-b63d-03ff9885cc0e", 2024-09-11T08:21:17.602 INFO:teuthology.orchestra.run.smithi150.stdout: "up": 1, 2024-09-11T08:21:17.602 INFO:teuthology.orchestra.run.smithi150.stdout: "in": 1, 2024-09-11T08:21:17.602 INFO:teuthology.orchestra.run.smithi150.stdout: "weight": 1, 2024-09-11T08:21:17.602 INFO:teuthology.orchestra.run.smithi150.stdout: "primary_affinity": 1, 2024-09-11T08:21:17.602 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_begin": 0, 2024-09-11T08:21:17.602 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_end": 0, 2024-09-11T08:21:17.602 INFO:teuthology.orchestra.run.smithi150.stdout: "up_from": 8, 2024-09-11T08:21:17.602 INFO:teuthology.orchestra.run.smithi150.stdout: "up_thru": 276, 2024-09-11T08:21:17.602 INFO:teuthology.orchestra.run.smithi150.stdout: "down_at": 0, 2024-09-11T08:21:17.602 INFO:teuthology.orchestra.run.smithi150.stdout: "lost_at": 0, 2024-09-11T08:21:17.602 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addrs": { 2024-09-11T08:21:17.602 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:21:17.602 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.602 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:21:17.603 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6801", 2024-09-11T08:21:17.603 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 334255279 2024-09-11T08:21:17.603 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:21:17.603 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:21:17.603 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.603 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addrs": { 2024-09-11T08:21:17.603 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:21:17.603 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.603 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:21:17.603 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6802", 2024-09-11T08:21:17.603 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 334255279 2024-09-11T08:21:17.603 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:21:17.603 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:21:17.603 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.604 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addrs": { 2024-09-11T08:21:17.604 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:21:17.604 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.604 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:21:17.604 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6804", 2024-09-11T08:21:17.604 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 334255279 2024-09-11T08:21:17.604 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:21:17.604 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:21:17.604 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.604 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addrs": { 2024-09-11T08:21:17.604 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:21:17.604 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.604 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:21:17.604 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6803", 2024-09-11T08:21:17.605 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 334255279 2024-09-11T08:21:17.605 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:21:17.605 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:21:17.605 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.605 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addr": "172.21.15.150:6801/334255279", 2024-09-11T08:21:17.605 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addr": "172.21.15.150:6802/334255279", 2024-09-11T08:21:17.605 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addr": "172.21.15.150:6804/334255279", 2024-09-11T08:21:17.605 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addr": "172.21.15.150:6803/334255279", 2024-09-11T08:21:17.605 INFO:teuthology.orchestra.run.smithi150.stdout: "state": [ 2024-09-11T08:21:17.605 INFO:teuthology.orchestra.run.smithi150.stdout: "exists", 2024-09-11T08:21:17.605 INFO:teuthology.orchestra.run.smithi150.stdout: "up" 2024-09-11T08:21:17.605 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:21:17.605 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.605 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.606 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 1, 2024-09-11T08:21:17.606 INFO:teuthology.orchestra.run.smithi150.stdout: "uuid": "80378442-02bc-48aa-b8ca-6113a4092079", 2024-09-11T08:21:17.606 INFO:teuthology.orchestra.run.smithi150.stdout: "up": 1, 2024-09-11T08:21:17.606 INFO:teuthology.orchestra.run.smithi150.stdout: "in": 1, 2024-09-11T08:21:17.606 INFO:teuthology.orchestra.run.smithi150.stdout: "weight": 1, 2024-09-11T08:21:17.606 INFO:teuthology.orchestra.run.smithi150.stdout: "primary_affinity": 1, 2024-09-11T08:21:17.606 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_begin": 0, 2024-09-11T08:21:17.606 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_end": 0, 2024-09-11T08:21:17.606 INFO:teuthology.orchestra.run.smithi150.stdout: "up_from": 13, 2024-09-11T08:21:17.606 INFO:teuthology.orchestra.run.smithi150.stdout: "up_thru": 295, 2024-09-11T08:21:17.606 INFO:teuthology.orchestra.run.smithi150.stdout: "down_at": 0, 2024-09-11T08:21:17.606 INFO:teuthology.orchestra.run.smithi150.stdout: "lost_at": 0, 2024-09-11T08:21:17.606 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addrs": { 2024-09-11T08:21:17.606 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:21:17.607 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.607 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:21:17.607 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6805", 2024-09-11T08:21:17.607 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1129510619 2024-09-11T08:21:17.607 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:21:17.607 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:21:17.607 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.607 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addrs": { 2024-09-11T08:21:17.607 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:21:17.607 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.607 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:21:17.607 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6806", 2024-09-11T08:21:17.607 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1129510619 2024-09-11T08:21:17.607 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:21:17.607 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:21:17.608 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.608 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addrs": { 2024-09-11T08:21:17.608 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:21:17.608 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.608 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:21:17.608 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6808", 2024-09-11T08:21:17.608 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1129510619 2024-09-11T08:21:17.608 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:21:17.608 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:21:17.608 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.608 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addrs": { 2024-09-11T08:21:17.608 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:21:17.608 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.608 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:21:17.609 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6807", 2024-09-11T08:21:17.609 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1129510619 2024-09-11T08:21:17.609 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:21:17.609 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:21:17.609 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.609 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addr": "172.21.15.150:6805/1129510619", 2024-09-11T08:21:17.609 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addr": "172.21.15.150:6806/1129510619", 2024-09-11T08:21:17.609 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addr": "172.21.15.150:6808/1129510619", 2024-09-11T08:21:17.609 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addr": "172.21.15.150:6807/1129510619", 2024-09-11T08:21:17.609 INFO:teuthology.orchestra.run.smithi150.stdout: "state": [ 2024-09-11T08:21:17.609 INFO:teuthology.orchestra.run.smithi150.stdout: "exists", 2024-09-11T08:21:17.609 INFO:teuthology.orchestra.run.smithi150.stdout: "up" 2024-09-11T08:21:17.609 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:21:17.610 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.610 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.610 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 2, 2024-09-11T08:21:17.610 INFO:teuthology.orchestra.run.smithi150.stdout: "uuid": "0955bb92-36a0-4e9a-98e1-b3a5af89f6d2", 2024-09-11T08:21:17.610 INFO:teuthology.orchestra.run.smithi150.stdout: "up": 1, 2024-09-11T08:21:17.610 INFO:teuthology.orchestra.run.smithi150.stdout: "in": 1, 2024-09-11T08:21:17.610 INFO:teuthology.orchestra.run.smithi150.stdout: "weight": 1, 2024-09-11T08:21:17.610 INFO:teuthology.orchestra.run.smithi150.stdout: "primary_affinity": 1, 2024-09-11T08:21:17.610 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_begin": 0, 2024-09-11T08:21:17.610 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_end": 0, 2024-09-11T08:21:17.610 INFO:teuthology.orchestra.run.smithi150.stdout: "up_from": 18, 2024-09-11T08:21:17.610 INFO:teuthology.orchestra.run.smithi150.stdout: "up_thru": 284, 2024-09-11T08:21:17.610 INFO:teuthology.orchestra.run.smithi150.stdout: "down_at": 0, 2024-09-11T08:21:17.610 INFO:teuthology.orchestra.run.smithi150.stdout: "lost_at": 0, 2024-09-11T08:21:17.610 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addrs": { 2024-09-11T08:21:17.611 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:21:17.611 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.611 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:21:17.611 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6809", 2024-09-11T08:21:17.611 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 719975422 2024-09-11T08:21:17.611 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:21:17.611 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:21:17.611 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.611 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addrs": { 2024-09-11T08:21:17.611 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:21:17.611 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.611 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:21:17.611 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6810", 2024-09-11T08:21:17.611 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 719975422 2024-09-11T08:21:17.612 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:21:17.612 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:21:17.612 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.612 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addrs": { 2024-09-11T08:21:17.612 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:21:17.612 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.612 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:21:17.612 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6812", 2024-09-11T08:21:17.612 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 719975422 2024-09-11T08:21:17.612 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:21:17.612 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:21:17.612 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.612 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addrs": { 2024-09-11T08:21:17.612 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:21:17.612 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.613 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:21:17.613 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6811", 2024-09-11T08:21:17.613 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 719975422 2024-09-11T08:21:17.613 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:21:17.613 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:21:17.613 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.613 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addr": "172.21.15.150:6809/719975422", 2024-09-11T08:21:17.613 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addr": "172.21.15.150:6810/719975422", 2024-09-11T08:21:17.613 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addr": "172.21.15.150:6812/719975422", 2024-09-11T08:21:17.613 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addr": "172.21.15.150:6811/719975422", 2024-09-11T08:21:17.613 INFO:teuthology.orchestra.run.smithi150.stdout: "state": [ 2024-09-11T08:21:17.613 INFO:teuthology.orchestra.run.smithi150.stdout: "exists", 2024-09-11T08:21:17.613 INFO:teuthology.orchestra.run.smithi150.stdout: "up" 2024-09-11T08:21:17.613 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:21:17.614 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.614 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.614 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 3, 2024-09-11T08:21:17.614 INFO:teuthology.orchestra.run.smithi150.stdout: "uuid": "1d04d4d3-62e8-4460-8a07-77f7e5c62929", 2024-09-11T08:21:17.614 INFO:teuthology.orchestra.run.smithi150.stdout: "up": 1, 2024-09-11T08:21:17.614 INFO:teuthology.orchestra.run.smithi150.stdout: "in": 1, 2024-09-11T08:21:17.614 INFO:teuthology.orchestra.run.smithi150.stdout: "weight": 1, 2024-09-11T08:21:17.614 INFO:teuthology.orchestra.run.smithi150.stdout: "primary_affinity": 0.611114501953125, 2024-09-11T08:21:17.614 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_begin": 0, 2024-09-11T08:21:17.614 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_end": 0, 2024-09-11T08:21:17.614 INFO:teuthology.orchestra.run.smithi150.stdout: "up_from": 25, 2024-09-11T08:21:17.614 INFO:teuthology.orchestra.run.smithi150.stdout: "up_thru": 260, 2024-09-11T08:21:17.614 INFO:teuthology.orchestra.run.smithi150.stdout: "down_at": 0, 2024-09-11T08:21:17.614 INFO:teuthology.orchestra.run.smithi150.stdout: "lost_at": 0, 2024-09-11T08:21:17.615 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addrs": { 2024-09-11T08:21:17.615 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:21:17.615 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.615 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:21:17.615 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6813", 2024-09-11T08:21:17.615 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 3101032500 2024-09-11T08:21:17.615 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:21:17.615 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:21:17.615 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.615 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addrs": { 2024-09-11T08:21:17.615 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:21:17.615 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.615 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:21:17.615 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6814", 2024-09-11T08:21:17.616 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 3101032500 2024-09-11T08:21:17.616 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:21:17.616 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:21:17.616 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.616 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addrs": { 2024-09-11T08:21:17.616 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:21:17.616 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.616 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:21:17.616 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6816", 2024-09-11T08:21:17.616 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 3101032500 2024-09-11T08:21:17.616 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:21:17.616 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:21:17.616 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.616 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addrs": { 2024-09-11T08:21:17.617 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:21:17.617 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.617 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:21:17.617 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6815", 2024-09-11T08:21:17.617 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 3101032500 2024-09-11T08:21:17.617 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:21:17.617 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:21:17.617 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.617 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addr": "172.21.15.150:6813/3101032500", 2024-09-11T08:21:17.617 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addr": "172.21.15.150:6814/3101032500", 2024-09-11T08:21:17.617 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addr": "172.21.15.150:6816/3101032500", 2024-09-11T08:21:17.617 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addr": "172.21.15.150:6815/3101032500", 2024-09-11T08:21:17.617 INFO:teuthology.orchestra.run.smithi150.stdout: "state": [ 2024-09-11T08:21:17.618 INFO:teuthology.orchestra.run.smithi150.stdout: "exists", 2024-09-11T08:21:17.618 INFO:teuthology.orchestra.run.smithi150.stdout: "up" 2024-09-11T08:21:17.618 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:21:17.618 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.618 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.618 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 4, 2024-09-11T08:21:17.618 INFO:teuthology.orchestra.run.smithi150.stdout: "uuid": "c5321a3c-d2f0-4440-8475-2c111d261139", 2024-09-11T08:21:17.618 INFO:teuthology.orchestra.run.smithi150.stdout: "up": 1, 2024-09-11T08:21:17.618 INFO:teuthology.orchestra.run.smithi150.stdout: "in": 1, 2024-09-11T08:21:17.618 INFO:teuthology.orchestra.run.smithi150.stdout: "weight": 1, 2024-09-11T08:21:17.618 INFO:teuthology.orchestra.run.smithi150.stdout: "primary_affinity": 1, 2024-09-11T08:21:17.618 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_begin": 0, 2024-09-11T08:21:17.618 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_end": 0, 2024-09-11T08:21:17.619 INFO:teuthology.orchestra.run.smithi150.stdout: "up_from": 31, 2024-09-11T08:21:17.619 INFO:teuthology.orchestra.run.smithi150.stdout: "up_thru": 299, 2024-09-11T08:21:17.619 INFO:teuthology.orchestra.run.smithi150.stdout: "down_at": 0, 2024-09-11T08:21:17.619 INFO:teuthology.orchestra.run.smithi150.stdout: "lost_at": 0, 2024-09-11T08:21:17.619 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addrs": { 2024-09-11T08:21:17.619 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:21:17.619 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.619 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:21:17.619 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6800", 2024-09-11T08:21:17.619 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2567452825 2024-09-11T08:21:17.619 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:21:17.619 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:21:17.619 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.619 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addrs": { 2024-09-11T08:21:17.619 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:21:17.619 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.620 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:21:17.620 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6801", 2024-09-11T08:21:17.620 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2567452825 2024-09-11T08:21:17.620 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:21:17.620 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:21:17.620 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.620 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addrs": { 2024-09-11T08:21:17.620 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:21:17.620 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.620 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:21:17.620 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6803", 2024-09-11T08:21:17.620 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2567452825 2024-09-11T08:21:17.620 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:21:17.620 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:21:17.620 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.620 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addrs": { 2024-09-11T08:21:17.621 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:21:17.621 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.621 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:21:17.621 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6802", 2024-09-11T08:21:17.621 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2567452825 2024-09-11T08:21:17.621 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:21:17.621 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:21:17.621 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.621 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addr": "172.21.15.181:6800/2567452825", 2024-09-11T08:21:17.621 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addr": "172.21.15.181:6801/2567452825", 2024-09-11T08:21:17.621 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addr": "172.21.15.181:6803/2567452825", 2024-09-11T08:21:17.621 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addr": "172.21.15.181:6802/2567452825", 2024-09-11T08:21:17.621 INFO:teuthology.orchestra.run.smithi150.stdout: "state": [ 2024-09-11T08:21:17.621 INFO:teuthology.orchestra.run.smithi150.stdout: "exists", 2024-09-11T08:21:17.621 INFO:teuthology.orchestra.run.smithi150.stdout: "up" 2024-09-11T08:21:17.622 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:21:17.622 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.622 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.622 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 5, 2024-09-11T08:21:17.622 INFO:teuthology.orchestra.run.smithi150.stdout: "uuid": "1262b2fe-f253-4a26-bfd4-c9dffb697c77", 2024-09-11T08:21:17.622 INFO:teuthology.orchestra.run.smithi150.stdout: "up": 1, 2024-09-11T08:21:17.622 INFO:teuthology.orchestra.run.smithi150.stdout: "in": 1, 2024-09-11T08:21:17.622 INFO:teuthology.orchestra.run.smithi150.stdout: "weight": 1, 2024-09-11T08:21:17.622 INFO:teuthology.orchestra.run.smithi150.stdout: "primary_affinity": 1, 2024-09-11T08:21:17.622 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_begin": 0, 2024-09-11T08:21:17.622 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_end": 0, 2024-09-11T08:21:17.622 INFO:teuthology.orchestra.run.smithi150.stdout: "up_from": 38, 2024-09-11T08:21:17.622 INFO:teuthology.orchestra.run.smithi150.stdout: "up_thru": 271, 2024-09-11T08:21:17.623 INFO:teuthology.orchestra.run.smithi150.stdout: "down_at": 0, 2024-09-11T08:21:17.623 INFO:teuthology.orchestra.run.smithi150.stdout: "lost_at": 0, 2024-09-11T08:21:17.623 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addrs": { 2024-09-11T08:21:17.623 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:21:17.623 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.623 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:21:17.623 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6804", 2024-09-11T08:21:17.623 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2043435508 2024-09-11T08:21:17.623 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:21:17.623 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:21:17.623 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.623 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addrs": { 2024-09-11T08:21:17.623 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:21:17.623 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.623 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:21:17.624 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6805", 2024-09-11T08:21:17.624 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2043435508 2024-09-11T08:21:17.624 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:21:17.624 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:21:17.624 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.624 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addrs": { 2024-09-11T08:21:17.624 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:21:17.624 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.624 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:21:17.624 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6807", 2024-09-11T08:21:17.624 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2043435508 2024-09-11T08:21:17.624 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:21:17.624 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:21:17.624 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.625 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addrs": { 2024-09-11T08:21:17.625 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:21:17.625 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.625 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:21:17.625 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6806", 2024-09-11T08:21:17.625 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2043435508 2024-09-11T08:21:17.625 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:21:17.625 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:21:17.625 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.625 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addr": "172.21.15.181:6804/2043435508", 2024-09-11T08:21:17.625 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addr": "172.21.15.181:6805/2043435508", 2024-09-11T08:21:17.625 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addr": "172.21.15.181:6807/2043435508", 2024-09-11T08:21:17.625 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addr": "172.21.15.181:6806/2043435508", 2024-09-11T08:21:17.625 INFO:teuthology.orchestra.run.smithi150.stdout: "state": [ 2024-09-11T08:21:17.626 INFO:teuthology.orchestra.run.smithi150.stdout: "exists", 2024-09-11T08:21:17.626 INFO:teuthology.orchestra.run.smithi150.stdout: "up" 2024-09-11T08:21:17.626 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:21:17.626 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.626 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.626 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 6, 2024-09-11T08:21:17.626 INFO:teuthology.orchestra.run.smithi150.stdout: "uuid": "7facca22-b3f1-4c20-833e-e3db798a08c4", 2024-09-11T08:21:17.626 INFO:teuthology.orchestra.run.smithi150.stdout: "up": 1, 2024-09-11T08:21:17.626 INFO:teuthology.orchestra.run.smithi150.stdout: "in": 1, 2024-09-11T08:21:17.626 INFO:teuthology.orchestra.run.smithi150.stdout: "weight": 1, 2024-09-11T08:21:17.626 INFO:teuthology.orchestra.run.smithi150.stdout: "primary_affinity": 1, 2024-09-11T08:21:17.626 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_begin": 0, 2024-09-11T08:21:17.626 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_end": 0, 2024-09-11T08:21:17.626 INFO:teuthology.orchestra.run.smithi150.stdout: "up_from": 45, 2024-09-11T08:21:17.627 INFO:teuthology.orchestra.run.smithi150.stdout: "up_thru": 270, 2024-09-11T08:21:17.627 INFO:teuthology.orchestra.run.smithi150.stdout: "down_at": 0, 2024-09-11T08:21:17.627 INFO:teuthology.orchestra.run.smithi150.stdout: "lost_at": 0, 2024-09-11T08:21:17.627 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addrs": { 2024-09-11T08:21:17.627 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:21:17.627 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.627 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:21:17.627 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6808", 2024-09-11T08:21:17.627 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1789880331 2024-09-11T08:21:17.627 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:21:17.627 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:21:17.627 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.627 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addrs": { 2024-09-11T08:21:17.627 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:21:17.628 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.628 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:21:17.628 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6809", 2024-09-11T08:21:17.628 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1789880331 2024-09-11T08:21:17.628 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:21:17.628 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:21:17.628 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.628 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addrs": { 2024-09-11T08:21:17.628 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:21:17.628 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.628 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:21:17.628 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6811", 2024-09-11T08:21:17.628 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1789880331 2024-09-11T08:21:17.628 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:21:17.629 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:21:17.629 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.629 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addrs": { 2024-09-11T08:21:17.629 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:21:17.629 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.629 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:21:17.629 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6810", 2024-09-11T08:21:17.629 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1789880331 2024-09-11T08:21:17.629 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:21:17.629 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:21:17.629 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.629 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addr": "172.21.15.181:6808/1789880331", 2024-09-11T08:21:17.629 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addr": "172.21.15.181:6809/1789880331", 2024-09-11T08:21:17.629 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addr": "172.21.15.181:6811/1789880331", 2024-09-11T08:21:17.630 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addr": "172.21.15.181:6810/1789880331", 2024-09-11T08:21:17.630 INFO:teuthology.orchestra.run.smithi150.stdout: "state": [ 2024-09-11T08:21:17.630 INFO:teuthology.orchestra.run.smithi150.stdout: "exists", 2024-09-11T08:21:17.630 INFO:teuthology.orchestra.run.smithi150.stdout: "up" 2024-09-11T08:21:17.630 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:21:17.630 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.630 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.630 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 7, 2024-09-11T08:21:17.630 INFO:teuthology.orchestra.run.smithi150.stdout: "uuid": "6ac1b06a-79bd-43ca-b91e-fcb0f4033a34", 2024-09-11T08:21:17.630 INFO:teuthology.orchestra.run.smithi150.stdout: "up": 1, 2024-09-11T08:21:17.630 INFO:teuthology.orchestra.run.smithi150.stdout: "in": 1, 2024-09-11T08:21:17.630 INFO:teuthology.orchestra.run.smithi150.stdout: "weight": 1, 2024-09-11T08:21:17.630 INFO:teuthology.orchestra.run.smithi150.stdout: "primary_affinity": 1, 2024-09-11T08:21:17.630 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_begin": 53, 2024-09-11T08:21:17.630 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_end": 139, 2024-09-11T08:21:17.630 INFO:teuthology.orchestra.run.smithi150.stdout: "up_from": 168, 2024-09-11T08:21:17.631 INFO:teuthology.orchestra.run.smithi150.stdout: "up_thru": 278, 2024-09-11T08:21:17.631 INFO:teuthology.orchestra.run.smithi150.stdout: "down_at": 140, 2024-09-11T08:21:17.631 INFO:teuthology.orchestra.run.smithi150.stdout: "lost_at": 0, 2024-09-11T08:21:17.631 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addrs": { 2024-09-11T08:21:17.631 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:21:17.631 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.631 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:21:17.631 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6812", 2024-09-11T08:21:17.631 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2706455857 2024-09-11T08:21:17.631 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:21:17.631 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:21:17.631 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.631 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addrs": { 2024-09-11T08:21:17.631 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:21:17.631 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.631 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:21:17.632 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6813", 2024-09-11T08:21:17.632 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2706455857 2024-09-11T08:21:17.632 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:21:17.632 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:21:17.632 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:17.633 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.633 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addrs": { 2024-09-11T08:21:17.633 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:21:17.633 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.633 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:21:17.633 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6815", 2024-09-11T08:21:17.633 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2706455857 2024-09-11T08:21:17.633 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:21:17.634 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:21:17.634 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.634 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addrs": { 2024-09-11T08:21:17.634 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:21:17.634 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.634 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:21:17.634 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6814", 2024-09-11T08:21:17.634 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2706455857 2024-09-11T08:21:17.634 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:21:17.634 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:21:17.634 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.634 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addr": "172.21.15.181:6812/2706455857", 2024-09-11T08:21:17.634 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addr": "172.21.15.181:6813/2706455857", 2024-09-11T08:21:17.634 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addr": "172.21.15.181:6815/2706455857", 2024-09-11T08:21:17.634 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addr": "172.21.15.181:6814/2706455857", 2024-09-11T08:21:17.635 INFO:teuthology.orchestra.run.smithi150.stdout: "state": [ 2024-09-11T08:21:17.635 INFO:teuthology.orchestra.run.smithi150.stdout: "exists", 2024-09-11T08:21:17.635 INFO:teuthology.orchestra.run.smithi150.stdout: "up" 2024-09-11T08:21:17.635 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:21:17.635 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:21:17.635 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-09-11T08:21:17.635 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_xinfo": [ 2024-09-11T08:21:17.635 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.635 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 0, 2024-09-11T08:21:17.635 INFO:teuthology.orchestra.run.smithi150.stdout: "down_stamp": "0.000000", 2024-09-11T08:21:17.635 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_probability": 0, 2024-09-11T08:21:17.635 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_interval": 0, 2024-09-11T08:21:17.635 INFO:teuthology.orchestra.run.smithi150.stdout: "features": 4540138320759226367, 2024-09-11T08:21:17.635 INFO:teuthology.orchestra.run.smithi150.stdout: "old_weight": 0, 2024-09-11T08:21:17.635 INFO:teuthology.orchestra.run.smithi150.stdout: "last_purged_snaps_scrub": "2024-09-11T08:21:17.155333+0000", 2024-09-11T08:21:17.636 INFO:teuthology.orchestra.run.smithi150.stdout: "dead_epoch": 0 2024-09-11T08:21:17.636 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.636 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.636 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 1, 2024-09-11T08:21:17.636 INFO:teuthology.orchestra.run.smithi150.stdout: "down_stamp": "0.000000", 2024-09-11T08:21:17.636 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_probability": 0, 2024-09-11T08:21:17.636 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_interval": 0, 2024-09-11T08:21:17.636 INFO:teuthology.orchestra.run.smithi150.stdout: "features": 4540138320759226367, 2024-09-11T08:21:17.636 INFO:teuthology.orchestra.run.smithi150.stdout: "old_weight": 0, 2024-09-11T08:21:17.636 INFO:teuthology.orchestra.run.smithi150.stdout: "last_purged_snaps_scrub": "2024-09-11T08:21:16.393280+0000", 2024-09-11T08:21:17.636 INFO:teuthology.orchestra.run.smithi150.stdout: "dead_epoch": 0 2024-09-11T08:21:17.636 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.636 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.636 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 2, 2024-09-11T08:21:17.637 INFO:teuthology.orchestra.run.smithi150.stdout: "down_stamp": "0.000000", 2024-09-11T08:21:17.637 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_probability": 0, 2024-09-11T08:21:17.637 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_interval": 0, 2024-09-11T08:21:17.637 INFO:teuthology.orchestra.run.smithi150.stdout: "features": 4540138320759226367, 2024-09-11T08:21:17.637 INFO:teuthology.orchestra.run.smithi150.stdout: "old_weight": 0, 2024-09-11T08:21:17.637 INFO:teuthology.orchestra.run.smithi150.stdout: "last_purged_snaps_scrub": "2024-09-11T08:20:29.633594+0000", 2024-09-11T08:21:17.637 INFO:teuthology.orchestra.run.smithi150.stdout: "dead_epoch": 0 2024-09-11T08:21:17.637 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.637 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.637 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 3, 2024-09-11T08:21:17.637 INFO:teuthology.orchestra.run.smithi150.stdout: "down_stamp": "0.000000", 2024-09-11T08:21:17.637 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_probability": 0, 2024-09-11T08:21:17.637 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_interval": 0, 2024-09-11T08:21:17.638 INFO:teuthology.orchestra.run.smithi150.stdout: "features": 4540138320759226367, 2024-09-11T08:21:17.638 INFO:teuthology.orchestra.run.smithi150.stdout: "old_weight": 0, 2024-09-11T08:21:17.638 INFO:teuthology.orchestra.run.smithi150.stdout: "last_purged_snaps_scrub": "2024-09-11T08:21:03.587927+0000", 2024-09-11T08:21:17.638 INFO:teuthology.orchestra.run.smithi150.stdout: "dead_epoch": 0 2024-09-11T08:21:17.638 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.638 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.638 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 4, 2024-09-11T08:21:17.638 INFO:teuthology.orchestra.run.smithi150.stdout: "down_stamp": "0.000000", 2024-09-11T08:21:17.638 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_probability": 0, 2024-09-11T08:21:17.638 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_interval": 0, 2024-09-11T08:21:17.638 INFO:teuthology.orchestra.run.smithi150.stdout: "features": 4540138320759226367, 2024-09-11T08:21:17.638 INFO:teuthology.orchestra.run.smithi150.stdout: "old_weight": 0, 2024-09-11T08:21:17.638 INFO:teuthology.orchestra.run.smithi150.stdout: "last_purged_snaps_scrub": "2024-09-11T08:20:39.570253+0000", 2024-09-11T08:21:17.638 INFO:teuthology.orchestra.run.smithi150.stdout: "dead_epoch": 0 2024-09-11T08:21:17.639 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.639 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.639 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 5, 2024-09-11T08:21:17.639 INFO:teuthology.orchestra.run.smithi150.stdout: "down_stamp": "0.000000", 2024-09-11T08:21:17.639 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_probability": 0, 2024-09-11T08:21:17.639 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_interval": 0, 2024-09-11T08:21:17.639 INFO:teuthology.orchestra.run.smithi150.stdout: "features": 4540138320759226367, 2024-09-11T08:21:17.639 INFO:teuthology.orchestra.run.smithi150.stdout: "old_weight": 0, 2024-09-11T08:21:17.639 INFO:teuthology.orchestra.run.smithi150.stdout: "last_purged_snaps_scrub": "2024-09-11T08:20:24.756605+0000", 2024-09-11T08:21:17.639 INFO:teuthology.orchestra.run.smithi150.stdout: "dead_epoch": 0 2024-09-11T08:21:17.639 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.639 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.639 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 6, 2024-09-11T08:21:17.639 INFO:teuthology.orchestra.run.smithi150.stdout: "down_stamp": "0.000000", 2024-09-11T08:21:17.639 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_probability": 0, 2024-09-11T08:21:17.639 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_interval": 0, 2024-09-11T08:21:17.640 INFO:teuthology.orchestra.run.smithi150.stdout: "features": 4540138320759226367, 2024-09-11T08:21:17.640 INFO:teuthology.orchestra.run.smithi150.stdout: "old_weight": 0, 2024-09-11T08:21:17.640 INFO:teuthology.orchestra.run.smithi150.stdout: "last_purged_snaps_scrub": "2024-09-11T08:20:37.768917+0000", 2024-09-11T08:21:17.640 INFO:teuthology.orchestra.run.smithi150.stdout: "dead_epoch": 0 2024-09-11T08:21:17.640 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.640 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.640 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 7, 2024-09-11T08:21:17.640 INFO:teuthology.orchestra.run.smithi150.stdout: "down_stamp": "0.000000", 2024-09-11T08:21:17.640 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_probability": 0, 2024-09-11T08:21:17.640 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_interval": 0, 2024-09-11T08:21:17.640 INFO:teuthology.orchestra.run.smithi150.stdout: "features": 4540138320759226367, 2024-09-11T08:21:17.640 INFO:teuthology.orchestra.run.smithi150.stdout: "old_weight": 0, 2024-09-11T08:21:17.640 INFO:teuthology.orchestra.run.smithi150.stdout: "last_purged_snaps_scrub": "2024-09-11T08:20:20.325086+0000", 2024-09-11T08:21:17.640 INFO:teuthology.orchestra.run.smithi150.stdout: "dead_epoch": 139 2024-09-11T08:21:17.640 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:21:17.641 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-09-11T08:21:17.641 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_upmap": [ 2024-09-11T08:21:17.641 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.641 INFO:teuthology.orchestra.run.smithi150.stdout: "pgid": "6.5", 2024-09-11T08:21:17.641 INFO:teuthology.orchestra.run.smithi150.stdout: "osds": [ 2024-09-11T08:21:17.641 INFO:teuthology.orchestra.run.smithi150.stdout: 2, 2024-09-11T08:21:17.641 INFO:teuthology.orchestra.run.smithi150.stdout: 4, 2024-09-11T08:21:17.641 INFO:teuthology.orchestra.run.smithi150.stdout: 7 2024-09-11T08:21:17.641 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:21:17.641 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:21:17.641 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-09-11T08:21:17.641 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_upmap_items": [ 2024-09-11T08:21:17.641 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.641 INFO:teuthology.orchestra.run.smithi150.stdout: "pgid": "4.e", 2024-09-11T08:21:17.641 INFO:teuthology.orchestra.run.smithi150.stdout: "mappings": [ 2024-09-11T08:21:17.642 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.642 INFO:teuthology.orchestra.run.smithi150.stdout: "from": 1, 2024-09-11T08:21:17.642 INFO:teuthology.orchestra.run.smithi150.stdout: "to": 5 2024-09-11T08:21:17.642 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:21:17.642 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:21:17.642 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.642 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.642 INFO:teuthology.orchestra.run.smithi150.stdout: "pgid": "4.14", 2024-09-11T08:21:17.642 INFO:teuthology.orchestra.run.smithi150.stdout: "mappings": [ 2024-09-11T08:21:17.642 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.642 INFO:teuthology.orchestra.run.smithi150.stdout: "from": 1, 2024-09-11T08:21:17.642 INFO:teuthology.orchestra.run.smithi150.stdout: "to": 2 2024-09-11T08:21:17.642 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:21:17.642 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:21:17.642 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.643 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.643 INFO:teuthology.orchestra.run.smithi150.stdout: "pgid": "4.1d", 2024-09-11T08:21:17.643 INFO:teuthology.orchestra.run.smithi150.stdout: "mappings": [ 2024-09-11T08:21:17.643 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.643 INFO:teuthology.orchestra.run.smithi150.stdout: "from": 1, 2024-09-11T08:21:17.643 INFO:teuthology.orchestra.run.smithi150.stdout: "to": 2 2024-09-11T08:21:17.643 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:21:17.643 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:21:17.643 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.643 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.643 INFO:teuthology.orchestra.run.smithi150.stdout: "pgid": "5.3", 2024-09-11T08:21:17.643 INFO:teuthology.orchestra.run.smithi150.stdout: "mappings": [ 2024-09-11T08:21:17.643 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.643 INFO:teuthology.orchestra.run.smithi150.stdout: "from": 0, 2024-09-11T08:21:17.643 INFO:teuthology.orchestra.run.smithi150.stdout: "to": 3 2024-09-11T08:21:17.644 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:21:17.644 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:21:17.644 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.644 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.644 INFO:teuthology.orchestra.run.smithi150.stdout: "pgid": "5.1e", 2024-09-11T08:21:17.644 INFO:teuthology.orchestra.run.smithi150.stdout: "mappings": [ 2024-09-11T08:21:17.644 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:21:17.644 INFO:teuthology.orchestra.run.smithi150.stdout: "from": 0, 2024-09-11T08:21:17.644 INFO:teuthology.orchestra.run.smithi150.stdout: "to": 6 2024-09-11T08:21:17.644 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:21:17.644 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:21:17.644 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:21:17.644 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-09-11T08:21:17.644 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_temp": [], 2024-09-11T08:21:17.644 INFO:teuthology.orchestra.run.smithi150.stdout: "primary_temp": [], 2024-09-11T08:21:17.645 INFO:teuthology.orchestra.run.smithi150.stdout: "blocklist": { 2024-09-11T08:21:17.645 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/2620931070": "2024-09-12T08:08:06.649692+0000", 2024-09-11T08:21:17.645 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/1073581409": "2024-09-12T08:02:45.986151+0000", 2024-09-11T08:21:17.645 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/2363649697": "2024-09-12T08:08:06.649692+0000", 2024-09-11T08:21:17.645 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:6800/603451071": "2024-09-12T08:02:45.986151+0000", 2024-09-11T08:21:17.645 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/1614413548": "2024-09-12T08:02:45.986151+0000", 2024-09-11T08:21:17.645 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/3673218166": "2024-09-12T08:03:08.740828+0000", 2024-09-11T08:21:17.649 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:6800/556498589": "2024-09-12T08:03:08.740828+0000", 2024-09-11T08:21:17.649 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/3666733994": "2024-09-12T08:03:08.740828+0000", 2024-09-11T08:21:17.649 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/3271479010": "2024-09-12T08:03:08.740828+0000", 2024-09-11T08:21:17.649 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/3185731090": "2024-09-12T08:08:06.649692+0000", 2024-09-11T08:21:17.649 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/2539412006": "2024-09-12T08:08:06.649692+0000", 2024-09-11T08:21:17.649 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/2740222408": "2024-09-12T08:02:45.986151+0000", 2024-09-11T08:21:17.649 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:6800/127856313": "2024-09-12T08:08:06.649692+0000", 2024-09-11T08:21:17.649 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/2375121603": "2024-09-12T08:08:06.649692+0000" 2024-09-11T08:21:17.649 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.649 INFO:teuthology.orchestra.run.smithi150.stdout: "range_blocklist": {}, 2024-09-11T08:21:17.649 INFO:teuthology.orchestra.run.smithi150.stdout: "erasure_code_profiles": { 2024-09-11T08:21:17.650 INFO:teuthology.orchestra.run.smithi150.stdout: "default": { 2024-09-11T08:21:17.650 INFO:teuthology.orchestra.run.smithi150.stdout: "crush-failure-domain": "osd", 2024-09-11T08:21:17.650 INFO:teuthology.orchestra.run.smithi150.stdout: "k": "2", 2024-09-11T08:21:17.650 INFO:teuthology.orchestra.run.smithi150.stdout: "m": "1", 2024-09-11T08:21:17.650 INFO:teuthology.orchestra.run.smithi150.stdout: "plugin": "jerasure", 2024-09-11T08:21:17.650 INFO:teuthology.orchestra.run.smithi150.stdout: "technique": "reed_sol_van" 2024-09-11T08:21:17.650 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:21:17.650 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:21:17.650 INFO:teuthology.orchestra.run.smithi150.stdout: "removed_snaps_queue": [], 2024-09-11T08:21:17.650 INFO:teuthology.orchestra.run.smithi150.stdout: "new_removed_snaps": [], 2024-09-11T08:21:17.650 INFO:teuthology.orchestra.run.smithi150.stdout: "new_purged_snaps": [], 2024-09-11T08:21:17.650 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_node_flags": {}, 2024-09-11T08:21:17.651 INFO:teuthology.orchestra.run.smithi150.stdout: "device_class_flags": {}, 2024-09-11T08:21:17.651 INFO:teuthology.orchestra.run.smithi150.stdout: "stretch_mode": { 2024-09-11T08:21:17.651 INFO:teuthology.orchestra.run.smithi150.stdout: "stretch_mode_enabled": false, 2024-09-11T08:21:17.651 INFO:teuthology.orchestra.run.smithi150.stdout: "stretch_bucket_count": 0, 2024-09-11T08:21:17.651 INFO:teuthology.orchestra.run.smithi150.stdout: "degraded_stretch_mode": 0, 2024-09-11T08:21:17.651 INFO:teuthology.orchestra.run.smithi150.stdout: "recovering_stretch_mode": 0, 2024-09-11T08:21:17.651 INFO:teuthology.orchestra.run.smithi150.stdout: "stretch_mode_bucket": 0 2024-09-11T08:21:17.651 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:21:17.651 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-11T08:21:17.685 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:17.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:17.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:17.787 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:21:17.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:17.828 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:17.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:21:17.929 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:17.964 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:17.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:17.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:18.066 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:18.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:18.100 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:18.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:18.201 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:21:18.231 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:18.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:18.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:21:18.333 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:18.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:18.383 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:18.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:18.484 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:18.520 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:18 smithi181 ceph-mon[32701]: pgmap v626: 148 pgs: 148 active+clean; 653 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 638 B/s rd, 0 B/s wr, 781 op/s; 13 B/s, 0 objects/s recovering 2024-09-11T08:21:18.520 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:18 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:21:18.520 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:18 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:21:18.520 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:18 smithi181 ceph-mon[32701]: osdmap e302: 8 total, 8 up, 8 in 2024-09-11T08:21:18.520 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:18 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/967691133' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-11T08:21:18.521 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:21:18 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:21:18.524 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:18.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:18.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:18.569 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:18 smithi150 ceph-mon[39612]: pgmap v626: 148 pgs: 148 active+clean; 653 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 638 B/s rd, 0 B/s wr, 781 op/s; 13 B/s, 0 objects/s recovering 2024-09-11T08:21:18.569 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:18 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:21:18.570 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:18 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:21:18.570 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:18 smithi150 ceph-mon[39612]: osdmap e302: 8 total, 8 up, 8 in 2024-09-11T08:21:18.570 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:18 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/967691133' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-11T08:21:18.570 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:18 smithi150 ceph-mon[29276]: pgmap v626: 148 pgs: 148 active+clean; 653 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 638 B/s rd, 0 B/s wr, 781 op/s; 13 B/s, 0 objects/s recovering 2024-09-11T08:21:18.570 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:18 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:21:18.570 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:18 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:21:18.570 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:18 smithi150 ceph-mon[29276]: osdmap e302: 8 total, 8 up, 8 in 2024-09-11T08:21:18.570 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:18 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/967691133' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-11T08:21:18.626 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:18.658 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:18.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:18.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:18.760 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:21:18.789 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:18.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:18.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:21:18.891 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:21:18.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:18.924 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:18.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:21:18.988 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:21:18 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:21:18] "GET /metrics HTTP/1.1" 200 224552 "" "Prometheus/2.43.0" 2024-09-11T08:21:19.026 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:19.059 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-11T08:21:19.059 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:21:19.071 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:19.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:19.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:19.173 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:19.210 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:19.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:19.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:19.312 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:19.347 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:19.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:19.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:19.448 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:19.483 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:19.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:19.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:19.585 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:21:19.619 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:19.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:19.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:21:19.721 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:19.761 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:19.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:19.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:19.863 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:19.898 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:19.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:19.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:20.000 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:20.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:20.032 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:20.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:20.134 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:20.167 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:20.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:20.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:20.269 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:21:20.299 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:20.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:20.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:21:20.401 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:20.434 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:20.435 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:20 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:21:20.435 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:20 smithi150 ceph-mon[29276]: pgmap v628: 148 pgs: 148 active+clean; 276 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.53k op/s; 13 B/s, 0 objects/s recovering 2024-09-11T08:21:20.435 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:20 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:21:20.435 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:20 smithi150 ceph-mon[39612]: pgmap v628: 148 pgs: 148 active+clean; 276 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.53k op/s; 13 B/s, 0 objects/s recovering 2024-09-11T08:21:20.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:20.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:20.537 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:21:20.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:20 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:21:20.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:20 smithi181 ceph-mon[32701]: pgmap v628: 148 pgs: 148 active+clean; 276 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.53k op/s; 13 B/s, 0 objects/s recovering 2024-09-11T08:21:20.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:20.578 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:20.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:21:20.679 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:20.720 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:20.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:20.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:20.821 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:20.865 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:20.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:20.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:20.910 INFO:tasks.thrashosds.thrasher:j is {'epoch': 302, 'fsid': 'c1bddff4-7013-11ef-bcea-c7b262605968', 'created': '2024-09-11T08:02:15.344618+0000', 'modified': '2024-09-11T08:21:17.224617+0000', 'last_up_change': '2024-09-11T08:14:23.745449+0000', 'last_in_change': '2024-09-11T08:19:28.121267+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-11T08:05:21.044729+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '255', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-11T08:07:28.786572+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 63, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-11T08:07:29.021286+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-11T08:07:30.334181+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-11T08:07:32.289626+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-11T08:07:34.351861+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 9, 'pool_name': 'unique_pool_2', 'create_time': '2024-09-11T08:17:59.761532+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '251', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': '37b63db9-ed2a-4701-b63d-03ff9885cc0e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 276, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6801', 'nonce': 334255279}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6802', 'nonce': 334255279}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6804', 'nonce': 334255279}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6803', 'nonce': 334255279}]}, 'public_addr': '172.21.15.150:6801/334255279', 'cluster_addr': '172.21.15.150:6802/334255279', 'heartbeat_back_addr': '172.21.15.150:6804/334255279', 'heartbeat_front_addr': '172.21.15.150:6803/334255279', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '80378442-02bc-48aa-b8ca-6113a4092079', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 295, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6805', 'nonce': 1129510619}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6806', 'nonce': 1129510619}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6808', 'nonce': 1129510619}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6807', 'nonce': 1129510619}]}, 'public_addr': '172.21.15.150:6805/1129510619', 'cluster_addr': '172.21.15.150:6806/1129510619', 'heartbeat_back_addr': '172.21.15.150:6808/1129510619', 'heartbeat_front_addr': '172.21.15.150:6807/1129510619', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '0955bb92-36a0-4e9a-98e1-b3a5af89f6d2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 284, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6809', 'nonce': 719975422}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6810', 'nonce': 719975422}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6812', 'nonce': 719975422}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6811', 'nonce': 719975422}]}, 'public_addr': '172.21.15.150:6809/719975422', 'cluster_addr': '172.21.15.150:6810/719975422', 'heartbeat_back_addr': '172.21.15.150:6812/719975422', 'heartbeat_front_addr': '172.21.15.150:6811/719975422', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '1d04d4d3-62e8-4460-8a07-77f7e5c62929', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.611114501953125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 260, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6813', 'nonce': 3101032500}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6814', 'nonce': 3101032500}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6816', 'nonce': 3101032500}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6815', 'nonce': 3101032500}]}, 'public_addr': '172.21.15.150:6813/3101032500', 'cluster_addr': '172.21.15.150:6814/3101032500', 'heartbeat_back_addr': '172.21.15.150:6816/3101032500', 'heartbeat_front_addr': '172.21.15.150:6815/3101032500', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'c5321a3c-d2f0-4440-8475-2c111d261139', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 299, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6800', 'nonce': 2567452825}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6801', 'nonce': 2567452825}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6803', 'nonce': 2567452825}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6802', 'nonce': 2567452825}]}, 'public_addr': '172.21.15.181:6800/2567452825', 'cluster_addr': '172.21.15.181:6801/2567452825', 'heartbeat_back_addr': '172.21.15.181:6803/2567452825', 'heartbeat_front_addr': '172.21.15.181:6802/2567452825', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '1262b2fe-f253-4a26-bfd4-c9dffb697c77', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 271, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6804', 'nonce': 2043435508}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6805', 'nonce': 2043435508}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6807', 'nonce': 2043435508}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6806', 'nonce': 2043435508}]}, 'public_addr': '172.21.15.181:6804/2043435508', 'cluster_addr': '172.21.15.181:6805/2043435508', 'heartbeat_back_addr': '172.21.15.181:6807/2043435508', 'heartbeat_front_addr': '172.21.15.181:6806/2043435508', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '7facca22-b3f1-4c20-833e-e3db798a08c4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 270, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6808', 'nonce': 1789880331}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6809', 'nonce': 1789880331}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6811', 'nonce': 1789880331}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6810', 'nonce': 1789880331}]}, 'public_addr': '172.21.15.181:6808/1789880331', 'cluster_addr': '172.21.15.181:6809/1789880331', 'heartbeat_back_addr': '172.21.15.181:6811/1789880331', 'heartbeat_front_addr': '172.21.15.181:6810/1789880331', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '6ac1b06a-79bd-43ca-b91e-fcb0f4033a34', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 53, 'last_clean_end': 139, 'up_from': 168, 'up_thru': 278, 'down_at': 140, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6812', 'nonce': 2706455857}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6813', 'nonce': 2706455857}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6815', 'nonce': 2706455857}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6814', 'nonce': 2706455857}]}, 'public_addr': '172.21.15.181:6812/2706455857', 'cluster_addr': '172.21.15.181:6813/2706455857', 'heartbeat_back_addr': '172.21.15.181:6815/2706455857', 'heartbeat_front_addr': '172.21.15.181:6814/2706455857', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T08:21:17.155333+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T08:21:16.393280+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T08:20:29.633594+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T08:21:03.587927+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T08:20:39.570253+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T08:20:24.756605+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T08:20:37.768917+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T08:20:20.325086+0000', 'dead_epoch': 139}], 'pg_upmap': [{'pgid': '6.5', 'osds': [2, 4, 7]}], 'pg_upmap_items': [{'pgid': '4.e', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.14', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1d', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '5.3', 'mappings': [{'from': 0, 'to': 3}]}, {'pgid': '5.1e', 'mappings': [{'from': 0, 'to': 6}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.150:0/2620931070': '2024-09-12T08:08:06.649692+0000', '172.21.15.150:0/1073581409': '2024-09-12T08:02:45.986151+0000', '172.21.15.150:0/2363649697': '2024-09-12T08:08:06.649692+0000', '172.21.15.150:6800/603451071': '2024-09-12T08:02:45.986151+0000', '172.21.15.150:0/1614413548': '2024-09-12T08:02:45.986151+0000', '172.21.15.150:0/3673218166': '2024-09-12T08:03:08.740828+0000', '172.21.15.150:6800/556498589': '2024-09-12T08:03:08.740828+0000', '172.21.15.150:0/3666733994': '2024-09-12T08:03:08.740828+0000', '172.21.15.150:0/3271479010': '2024-09-12T08:03:08.740828+0000', '172.21.15.150:0/3185731090': '2024-09-12T08:08:06.649692+0000', '172.21.15.150:0/2539412006': '2024-09-12T08:08:06.649692+0000', '172.21.15.150:0/2740222408': '2024-09-12T08:02:45.986151+0000', '172.21.15.150:6800/127856313': '2024-09-12T08:08:06.649692+0000', '172.21.15.150:0/2375121603': '2024-09-12T08:08:06.649692+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-11T08:21:20.910 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T08:21:20.967 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:20.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:20.988 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:20.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:21.089 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:21.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:21.134 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:21.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:21.235 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:21:21.267 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:21.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:21.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:21:21.369 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:21.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:21.408 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:21.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:21.453 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:21:21.491 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-11T08:21:21.492 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:21:21.492 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:21:21.510 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:21.532 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:21:21.533 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:21:21.561 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:21:21.561 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:21:21.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:21.582 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:21.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:21.587 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:21:21.588 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:21:21.618 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:21:21.618 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:21:21.654 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:21:21.654 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:21:21.683 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:21.684 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:21:21.685 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:21:21.707 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:21:21.707 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:21:21.708 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:21 smithi150 ceph-mon[29276]: pgmap v629: 148 pgs: 148 active+clean; 276 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.47k op/s; 13 B/s, 0 objects/s recovering 2024-09-11T08:21:21.708 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:21 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3162550162' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:21:21.708 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:21 smithi150 ceph-mon[39612]: pgmap v629: 148 pgs: 148 active+clean; 276 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.47k op/s; 13 B/s, 0 objects/s recovering 2024-09-11T08:21:21.708 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:21 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3162550162' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:21:21.736 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:21.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:21.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:21.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:21 smithi181 ceph-mon[32701]: pgmap v629: 148 pgs: 148 active+clean; 276 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.47k op/s; 13 B/s, 0 objects/s recovering 2024-09-11T08:21:21.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:21 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3162550162' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:21:21.838 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:21.871 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:21.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:21.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:21.901 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:21:21.973 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:22.010 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:22.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:22.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:22.112 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:22.154 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:22.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:22.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:22.255 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:21:22.285 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:22.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:22.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:21:22.345 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Removed 115603 objects 2024-09-11T08:21:22.345 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Clean up completed and total clean up time :98.5413 2024-09-11T08:21:22.387 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:21:22.409 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:22.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:22.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:21:22.510 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:22.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:22.533 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:22.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:22.634 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:21:22.655 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:22.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:22.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:21:22.757 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:22.778 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:22.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:22.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:22.872 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:21:22.881 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:22.903 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:22.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:22.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:23.005 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:23.019 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:22 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:21:23.019 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:22 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:21:23.020 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:22 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:21:23.020 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:22 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:21:23.026 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:23.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:23.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:23.072 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:22 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:21:23.072 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:22 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:21:23.128 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:23.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:23.151 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:23.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:23.252 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:23.273 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:23.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:23.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:23.374 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:23.396 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:23.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:23.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:23.498 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:23.519 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:23.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:23.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:23.621 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:21:23.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:23.643 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:23.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:21:23.744 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:21:23.767 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:23.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:23.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:21:23.868 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:21:23.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:23.891 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:23.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:21:23.935 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-11T08:21:23.935 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-11T08:21:23.936 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-09-11T08:21:23.936 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-09-11T08:21:23.936 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-11T08:21:23.992 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:24.014 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:24.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:24.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:24.043 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:23 smithi150 ceph-mon[29276]: pgmap v630: 148 pgs: 148 active+clean; 237 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 1.21k op/s 2024-09-11T08:21:24.043 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:23 smithi150 ceph-mon[39612]: pgmap v630: 148 pgs: 148 active+clean; 237 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 1.21k op/s 2024-09-11T08:21:24.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:23 smithi181 ceph-mon[32701]: pgmap v630: 148 pgs: 148 active+clean; 237 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 1.21k op/s 2024-09-11T08:21:24.116 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:21:24.140 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:24.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:24.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:21:24.242 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:24.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:24.264 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:24.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:24.365 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:24.387 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:24.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:24.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:24.488 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:24.510 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:24.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:24.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:24.611 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:24.633 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:24.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:24.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:24.654 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-11T08:21:24.735 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:24.758 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:24.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:24.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:24.860 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:24.881 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:24.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:24.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:24.898 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:21:24.898 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T08:21:24.983 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:21:25.005 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:25.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:25.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:21:25.107 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:25.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:25.129 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:25.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:25.230 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:25.252 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:25.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:25.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:25.276 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:21:25 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:21:25] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:21:25.354 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:21:25.375 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:25.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:25.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:21:25.477 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:25.498 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:25.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:25.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:25.541 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:21:25.600 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:25.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:25.623 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:25.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:25.647 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:21:25.649 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-09-11T08:21:25.724 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:21:25.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:25.746 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:25.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:21:25.847 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:25.868 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:25.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:25.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:25.970 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:21:25.992 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:25.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:25.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:21:26.080 INFO:teuthology.orchestra.run.smithi150.stderr:nodeep-scrub is set 2024-09-11T08:21:26.094 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:26.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:26.116 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:26.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:26.217 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:21:26.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:26.241 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:26.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:21:26.289 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:26 smithi181 ceph-mon[32701]: pgmap v631: 148 pgs: 148 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 1.04k op/s 2024-09-11T08:21:26.290 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:26 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1242407277' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:21:26.342 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:26.366 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:26.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:26.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:26.457 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:26 smithi150 ceph-mon[29276]: pgmap v631: 148 pgs: 148 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 1.04k op/s 2024-09-11T08:21:26.457 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:26 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1242407277' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:21:26.457 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:26 smithi150 ceph-mon[39612]: pgmap v631: 148 pgs: 148 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 1.04k op/s 2024-09-11T08:21:26.457 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:26 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1242407277' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:21:26.469 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:21:26.491 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:26.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:26.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:21:26.592 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:26.614 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:26.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:26.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:26.715 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:26.739 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:26.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:26.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:26.793 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-11T08:21:26.794 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-09-11T08:21:26.794 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-09-11T08:21:26.794 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-09-11T08:21:26.794 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.794 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24421.0:400 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e168)", 2024-09-11T08:21:26.794 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:14:23.790029+0000", 2024-09-11T08:21:26.794 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 422.72239439200001, 2024-09-11T08:21:26.794 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0036556700000001, 2024-09-11T08:21:26.794 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:21:26.794 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-11T08:21:26.794 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:21:26.794 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24421", 2024-09-11T08:21:26.795 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/3861404665", 2024-09-11T08:21:26.795 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 400 2024-09-11T08:21:26.795 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.795 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:21:26.795 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.795 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:21:26.795 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:23.790029+0000", 2024-09-11T08:21:26.795 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:26.795 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.795 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.795 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:21:26.795 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:23.790029+0000", 2024-09-11T08:21:26.795 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:26.796 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.796 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.796 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:21:26.796 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:23.790027+0000", 2024-09-11T08:21:26.796 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:21:26.796 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.796 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.796 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:21:26.796 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:23.790033+0000", 2024-09-11T08:21:26.796 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1120000000000001e-06 2024-09-11T08:21:26.796 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.796 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.796 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:21:26.797 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:23.790041+0000", 2024-09-11T08:21:26.797 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1029999999999996e-06 2024-09-11T08:21:26.797 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.797 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.797 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:21:26.797 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:23.790046+0000", 2024-09-11T08:21:26.797 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7389999999999996e-06 2024-09-11T08:21:26.797 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.797 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.797 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:26.797 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:23.827398+0000", 2024-09-11T08:21:26.797 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.037351729 2024-09-11T08:21:26.798 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.798 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.798 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:21:26.798 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:23.827575+0000", 2024-09-11T08:21:26.798 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017671700000000001 2024-09-11T08:21:26.798 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.798 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.798 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:26.798 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:24.793336+0000", 2024-09-11T08:21:26.798 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.96576121400000003 2024-09-11T08:21:26.798 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.798 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.798 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:21:26.799 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:24.793476+0000", 2024-09-11T08:21:26.799 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000140543 2024-09-11T08:21:26.799 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.799 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.799 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:21:26.799 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:14:24.793684+0000", 2024-09-11T08:21:26.799 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000208035 2024-09-11T08:21:26.799 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:26.799 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:21:26.799 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:26.799 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.799 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.799 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:22571 9.e 9:7e6ad66b:::benchmark_data_smithi150_133616_object22570:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e252)", 2024-09-11T08:21:26.799 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:18:19.413620+0000", 2024-09-11T08:21:26.800 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 187.09880273300001, 2024-09-11T08:21:26.800 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.15670378099999999, 2024-09-11T08:21:26.800 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:21:26.800 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:21:26.800 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:21:26.800 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:21:26.800 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:21:26.800 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 22571 2024-09-11T08:21:26.800 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.800 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:21:26.800 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.800 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:21:26.800 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.413620+0000", 2024-09-11T08:21:26.800 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:26.801 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.801 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.801 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:21:26.801 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.413620+0000", 2024-09-11T08:21:26.801 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:26.801 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.801 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.801 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:21:26.801 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.413619+0000", 2024-09-11T08:21:26.801 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:21:26.801 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.801 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.801 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:21:26.801 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.413705+0000", 2024-09-11T08:21:26.802 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6098000000000001e-05 2024-09-11T08:21:26.802 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.802 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.802 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:21:26.802 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.413718+0000", 2024-09-11T08:21:26.802 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3324e-05 2024-09-11T08:21:26.802 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.802 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.802 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:21:26.802 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.413725+0000", 2024-09-11T08:21:26.802 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2929999999999997e-06 2024-09-11T08:21:26.802 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.802 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.802 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:26.802 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.413782+0000", 2024-09-11T08:21:26.803 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7216000000000002e-05 2024-09-11T08:21:26.803 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.803 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.803 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:21:26.803 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.413998+0000", 2024-09-11T08:21:26.803 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00021641100000000001 2024-09-11T08:21:26.803 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.803 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.803 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,4", 2024-09-11T08:21:26.803 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.414139+0000", 2024-09-11T08:21:26.803 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014046299999999999 2024-09-11T08:21:26.803 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.803 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.803 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:21:26.803 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.415504+0000", 2024-09-11T08:21:26.804 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001365694 2024-09-11T08:21:26.804 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.804 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.804 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:26.804 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.418657+0000", 2024-09-11T08:21:26.804 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0031527249999999999 2024-09-11T08:21:26.804 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.804 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.804 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:26.804 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.570261+0000", 2024-09-11T08:21:26.804 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.15160364100000001 2024-09-11T08:21:26.804 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.804 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.805 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:21:26.805 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.570302+0000", 2024-09-11T08:21:26.805 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0738000000000002e-05 2024-09-11T08:21:26.805 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.805 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.805 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:21:26.805 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.570324+0000", 2024-09-11T08:21:26.805 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2659000000000001e-05 2024-09-11T08:21:26.805 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:26.805 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:21:26.805 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:26.805 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.805 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.805 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:30518 9.d 9:bcdd8f8d:::benchmark_data_smithi150_133616_object30517:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e252)", 2024-09-11T08:21:26.805 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:18:23.980586+0000", 2024-09-11T08:21:26.806 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 182.531837346, 2024-09-11T08:21:26.806 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.16594493399999999, 2024-09-11T08:21:26.806 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:21:26.806 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:21:26.806 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:21:26.806 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:21:26.806 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:21:26.806 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 30518 2024-09-11T08:21:26.806 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.806 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:21:26.806 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.806 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:21:26.806 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.980586+0000", 2024-09-11T08:21:26.806 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:26.807 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.807 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.807 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:21:26.807 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.980586+0000", 2024-09-11T08:21:26.807 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:26.807 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.807 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.807 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:21:26.807 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.980585+0000", 2024-09-11T08:21:26.807 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-09-11T08:21:26.807 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.807 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.807 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:21:26.807 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.980621+0000", 2024-09-11T08:21:26.808 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5203999999999997e-05 2024-09-11T08:21:26.808 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.808 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.808 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:21:26.808 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.980625+0000", 2024-09-11T08:21:26.808 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.887e-06 2024-09-11T08:21:26.808 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.808 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.808 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:21:26.808 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.980628+0000", 2024-09-11T08:21:26.808 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8739999999999999e-06 2024-09-11T08:21:26.808 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.808 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.809 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:26.809 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.981775+0000", 2024-09-11T08:21:26.809 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0011470849999999999 2024-09-11T08:21:26.809 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.809 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.809 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:21:26.809 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.981889+0000", 2024-09-11T08:21:26.809 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011341999999999999 2024-09-11T08:21:26.809 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.809 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.809 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,5", 2024-09-11T08:21:26.809 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.981980+0000", 2024-09-11T08:21:26.809 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.1347999999999993e-05 2024-09-11T08:21:26.809 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.810 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.810 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:21:26.810 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.982814+0000", 2024-09-11T08:21:26.810 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00083360400000000005 2024-09-11T08:21:26.810 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.810 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.810 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:26.810 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:23.982981+0000", 2024-09-11T08:21:26.810 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016702199999999999 2024-09-11T08:21:26.810 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.810 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.810 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:26.810 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:24.146430+0000", 2024-09-11T08:21:26.810 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.16344957900000001 2024-09-11T08:21:26.811 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.811 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.811 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:21:26.811 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:24.146489+0000", 2024-09-11T08:21:26.811 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8635999999999999e-05 2024-09-11T08:21:26.811 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.811 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.811 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:21:26.811 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:24.146531+0000", 2024-09-11T08:21:26.811 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1828e-05 2024-09-11T08:21:26.811 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:26.811 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:21:26.811 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:26.811 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.812 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.812 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:96100 9.d 9:b6837c16:::benchmark_data_smithi150_133616_object96099:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e259)", 2024-09-11T08:21:26.812 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:19:08.589253+0000", 2024-09-11T08:21:26.812 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 137.92317016499999, 2024-09-11T08:21:26.812 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.241510726, 2024-09-11T08:21:26.812 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:21:26.812 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:21:26.812 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:21:26.812 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:21:26.812 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:21:26.812 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 96100 2024-09-11T08:21:26.812 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.812 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:21:26.812 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.813 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:21:26.813 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.589253+0000", 2024-09-11T08:21:26.813 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:26.813 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.813 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.813 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:21:26.813 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.589253+0000", 2024-09-11T08:21:26.813 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:26.813 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.813 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.813 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:21:26.813 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.589251+0000", 2024-09-11T08:21:26.813 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:21:26.813 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.813 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.814 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:21:26.814 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.589287+0000", 2024-09-11T08:21:26.814 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5831999999999998e-05 2024-09-11T08:21:26.814 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.814 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.814 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:21:26.814 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.589299+0000", 2024-09-11T08:21:26.814 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2143e-05 2024-09-11T08:21:26.814 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.814 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.814 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:21:26.814 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.589306+0000", 2024-09-11T08:21:26.814 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1800000000000001e-06 2024-09-11T08:21:26.814 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.814 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.815 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:26.815 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.615074+0000", 2024-09-11T08:21:26.815 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.025768183 2024-09-11T08:21:26.815 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.815 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.815 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:21:26.815 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.615093+0000", 2024-09-11T08:21:26.815 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9256e-05 2024-09-11T08:21:26.815 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.815 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.815 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:26.815 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:09.593235+0000", 2024-09-11T08:21:26.815 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.97814237000000004 2024-09-11T08:21:26.815 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.815 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.816 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:21:26.816 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:09.593247+0000", 2024-09-11T08:21:26.816 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1423999999999999e-05 2024-09-11T08:21:26.816 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.816 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.816 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:26.816 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:10.734531+0000", 2024-09-11T08:21:26.816 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.141284204 2024-09-11T08:21:26.816 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.816 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.816 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:21:26.816 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:10.734645+0000", 2024-09-11T08:21:26.816 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000114392 2024-09-11T08:21:26.816 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.817 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.817 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,4,5", 2024-09-11T08:21:26.817 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:10.735075+0000", 2024-09-11T08:21:26.817 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00042998299999999999 2024-09-11T08:21:26.817 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.817 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.817 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:21:26.817 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:10.745751+0000", 2024-09-11T08:21:26.817 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.010675326000000001 2024-09-11T08:21:26.817 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.817 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.817 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:26.817 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:10.745850+0000", 2024-09-11T08:21:26.817 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9628000000000002e-05 2024-09-11T08:21:26.818 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.818 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.818 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:26.818 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:10.745917+0000", 2024-09-11T08:21:26.818 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6192999999999995e-05 2024-09-11T08:21:26.818 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.818 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.818 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:26.818 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:10.830643+0000", 2024-09-11T08:21:26.818 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.084726478999999993 2024-09-11T08:21:26.818 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.818 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.818 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:21:26.818 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:10.830687+0000", 2024-09-11T08:21:26.819 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4487999999999998e-05 2024-09-11T08:21:26.819 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.819 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.819 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:21:26.819 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:10.830764+0000", 2024-09-11T08:21:26.819 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6274999999999997e-05 2024-09-11T08:21:26.819 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:26.819 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:21:26.819 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:26.819 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.819 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.819 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:96096 9.7 9:ee1b318e:::benchmark_data_smithi150_133616_object96095:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e260)", 2024-09-11T08:21:26.819 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:19:08.595939+0000", 2024-09-11T08:21:26.819 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 137.91648469200001, 2024-09-11T08:21:26.820 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.187328741, 2024-09-11T08:21:26.820 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:21:26.820 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-11T08:21:26.820 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:21:26.820 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:21:26.820 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:21:26.820 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 96096 2024-09-11T08:21:26.820 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.820 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:21:26.820 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.820 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:21:26.820 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.595939+0000", 2024-09-11T08:21:26.821 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:26.821 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.821 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.821 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:21:26.821 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.595939+0000", 2024-09-11T08:21:26.821 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:26.821 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.821 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.821 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:21:26.821 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.595937+0000", 2024-09-11T08:21:26.821 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999981 2024-09-11T08:21:26.821 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.821 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.822 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:21:26.822 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.595973+0000", 2024-09-11T08:21:26.822 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5794000000000003e-05 2024-09-11T08:21:26.822 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.822 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.822 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:21:26.822 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.596800+0000", 2024-09-11T08:21:26.822 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00082782999999999999 2024-09-11T08:21:26.822 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.822 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.822 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:21:26.822 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.596810+0000", 2024-09-11T08:21:26.822 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7969999999999995e-06 2024-09-11T08:21:26.822 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.823 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.823 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:26.823 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.605998+0000", 2024-09-11T08:21:26.823 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0091875180000000004 2024-09-11T08:21:26.823 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.823 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.823 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:21:26.823 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.606008+0000", 2024-09-11T08:21:26.823 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0223e-05 2024-09-11T08:21:26.823 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.823 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.823 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:26.823 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:09.594589+0000", 2024-09-11T08:21:26.823 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.988581402 2024-09-11T08:21:26.823 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.824 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.824 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:21:26.824 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:09.594601+0000", 2024-09-11T08:21:26.824 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1715999999999999e-05 2024-09-11T08:21:26.824 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.824 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.824 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:26.824 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:10.782301+0000", 2024-09-11T08:21:26.824 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1876996 2024-09-11T08:21:26.824 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.824 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.824 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:21:26.824 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:10.783267+0000", 2024-09-11T08:21:26.824 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00096663700000000001 2024-09-11T08:21:26.825 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:26.825 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:21:26.825 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:26.825 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.825 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.825 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:96109 9.d 9:b94774bb:::benchmark_data_smithi150_133616_object96108:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-11T08:21:26.825 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:19:08.614378+0000", 2024-09-11T08:21:26.825 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 137.89804560600001, 2024-09-11T08:21:26.825 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2165278509999999, 2024-09-11T08:21:26.825 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:21:26.825 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:21:26.825 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:21:26.825 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:21:26.825 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:21:26.826 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 96109 2024-09-11T08:21:26.826 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.826 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:21:26.826 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.826 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:21:26.826 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.614378+0000", 2024-09-11T08:21:26.826 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:26.826 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.826 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.826 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:21:26.826 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.614378+0000", 2024-09-11T08:21:26.826 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:26.826 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.827 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.827 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:21:26.827 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.614376+0000", 2024-09-11T08:21:26.827 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999981 2024-09-11T08:21:26.827 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.827 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.827 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:21:26.827 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.614423+0000", 2024-09-11T08:21:26.827 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6887e-05 2024-09-11T08:21:26.827 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.827 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.827 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:21:26.827 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.614433+0000", 2024-09-11T08:21:26.827 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.049e-05 2024-09-11T08:21:26.828 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.828 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.828 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:21:26.828 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.614440+0000", 2024-09-11T08:21:26.828 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7129999999999999e-06 2024-09-11T08:21:26.828 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.828 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.828 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:26.828 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.615331+0000", 2024-09-11T08:21:26.828 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00089052399999999996 2024-09-11T08:21:26.828 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.828 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.828 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:21:26.828 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.615370+0000", 2024-09-11T08:21:26.829 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9857999999999997e-05 2024-09-11T08:21:26.829 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.829 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.829 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:26.829 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:09.593268+0000", 2024-09-11T08:21:26.829 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.97789712600000001 2024-09-11T08:21:26.829 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.829 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.829 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:21:26.829 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:09.593274+0000", 2024-09-11T08:21:26.829 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7000000000000002e-06 2024-09-11T08:21:26.829 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.829 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.829 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:26.830 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:10.740372+0000", 2024-09-11T08:21:26.830 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.147097665 2024-09-11T08:21:26.830 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.830 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.830 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:21:26.830 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:10.740532+0000", 2024-09-11T08:21:26.830 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016001500000000001 2024-09-11T08:21:26.830 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.830 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.830 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,4,5", 2024-09-11T08:21:26.830 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:10.740723+0000", 2024-09-11T08:21:26.830 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00019085900000000001 2024-09-11T08:21:26.830 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.831 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.831 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:21:26.831 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:10.745859+0000", 2024-09-11T08:21:26.831 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0051359630000000003 2024-09-11T08:21:26.831 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.831 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.831 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:26.831 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:10.745892+0000", 2024-09-11T08:21:26.831 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3126000000000001e-05 2024-09-11T08:21:26.831 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.831 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.831 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:26.831 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:10.745963+0000", 2024-09-11T08:21:26.831 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1587000000000004e-05 2024-09-11T08:21:26.832 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.832 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.832 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:26.832 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:10.830805+0000", 2024-09-11T08:21:26.832 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.084841406999999994 2024-09-11T08:21:26.832 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.832 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.832 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:21:26.832 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:10.830843+0000", 2024-09-11T08:21:26.832 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7951999999999998e-05 2024-09-11T08:21:26.832 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.832 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.832 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:21:26.833 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:10.830905+0000", 2024-09-11T08:21:26.833 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2671999999999999e-05 2024-09-11T08:21:26.833 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:26.833 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:21:26.833 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:26.833 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.833 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.833 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:115604 9.d 9:b8615e71:::benchmark_last_metadata:head [writefull 0~24] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e277)", 2024-09-11T08:21:26.833 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:19:42.782701+0000", 2024-09-11T08:21:26.833 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 103.72972231200001, 2024-09-11T08:21:26.833 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0163880489999999, 2024-09-11T08:21:26.833 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:21:26.833 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-11T08:21:26.833 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:21:26.834 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:21:26.834 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:21:26.834 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 115604 2024-09-11T08:21:26.834 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.834 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:21:26.834 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.834 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:21:26.834 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:42.782701+0000", 2024-09-11T08:21:26.834 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:26.834 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.834 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.834 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:21:26.834 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:42.782701+0000", 2024-09-11T08:21:26.834 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:26.834 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.835 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.835 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:21:26.835 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:42.782699+0000", 2024-09-11T08:21:26.835 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999981 2024-09-11T08:21:26.835 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.835 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.835 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:21:26.835 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:42.782711+0000", 2024-09-11T08:21:26.835 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1805e-05 2024-09-11T08:21:26.835 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.835 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.835 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:21:26.835 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:42.782732+0000", 2024-09-11T08:21:26.835 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0863e-05 2024-09-11T08:21:26.836 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.836 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.836 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:21:26.836 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:42.782740+0000", 2024-09-11T08:21:26.836 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.9299999999999992e-06 2024-09-11T08:21:26.836 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.836 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.836 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:26.836 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:42.782800+0000", 2024-09-11T08:21:26.836 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9763999999999999e-05 2024-09-11T08:21:26.836 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.836 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.836 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:21:26.836 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:42.782925+0000", 2024-09-11T08:21:26.837 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012444800000000001 2024-09-11T08:21:26.837 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.837 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.837 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,4,5", 2024-09-11T08:21:26.837 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:42.783031+0000", 2024-09-11T08:21:26.837 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000106807 2024-09-11T08:21:26.837 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.837 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.837 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:26.837 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:42.786504+0000", 2024-09-11T08:21:26.837 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003472709 2024-09-11T08:21:26.837 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.837 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.837 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:26.837 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:42.787835+0000", 2024-09-11T08:21:26.838 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0013306469999999999 2024-09-11T08:21:26.838 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.838 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.838 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:21:26.838 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:42.787875+0000", 2024-09-11T08:21:26.838 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0329e-05 2024-09-11T08:21:26.838 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.838 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.838 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:26.838 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:43.799012+0000", 2024-09-11T08:21:26.838 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0111368759999999 2024-09-11T08:21:26.838 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.838 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.838 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:21:26.838 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:43.799089+0000", 2024-09-11T08:21:26.839 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6925e-05 2024-09-11T08:21:26.839 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:26.839 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:21:26.839 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:26.839 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.839 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.839 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:177468 9.3 9:c8e70765:::benchmark_data_smithi150_133616_object61862:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-11T08:21:26.839 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:20:34.901684+0000", 2024-09-11T08:21:26.839 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 51.610739131999999, 2024-09-11T08:21:26.839 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.16031762899999999, 2024-09-11T08:21:26.839 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:21:26.839 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:21:26.839 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:21:26.839 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:21:26.839 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:21:26.840 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 177468 2024-09-11T08:21:26.840 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.840 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:21:26.840 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.840 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:21:26.840 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:34.901684+0000", 2024-09-11T08:21:26.840 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:26.840 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.840 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.840 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:21:26.840 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:34.901684+0000", 2024-09-11T08:21:26.840 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:26.840 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.840 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.841 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:21:26.841 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:34.901682+0000", 2024-09-11T08:21:26.841 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:21:26.841 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.841 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.841 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:21:26.841 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:34.901688+0000", 2024-09-11T08:21:26.841 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8279999999999999e-06 2024-09-11T08:21:26.841 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.841 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.841 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:21:26.841 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:34.901697+0000", 2024-09-11T08:21:26.841 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6799999999999999e-06 2024-09-11T08:21:26.841 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.842 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.842 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:21:26.842 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:34.901702+0000", 2024-09-11T08:21:26.842 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9200000000000003e-06 2024-09-11T08:21:26.842 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.842 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.842 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:26.842 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:34.966623+0000", 2024-09-11T08:21:26.842 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.064920874000000003 2024-09-11T08:21:26.842 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.842 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.842 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:21:26.842 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:34.966751+0000", 2024-09-11T08:21:26.842 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012806800000000001 2024-09-11T08:21:26.843 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.843 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.843 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-09-11T08:21:26.843 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:34.966898+0000", 2024-09-11T08:21:26.843 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014682900000000001 2024-09-11T08:21:26.843 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.843 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.843 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:21:26.843 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:34.973097+0000", 2024-09-11T08:21:26.843 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.006199174 2024-09-11T08:21:26.843 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.843 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.843 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:26.843 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:34.978011+0000", 2024-09-11T08:21:26.843 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0049137959999999998 2024-09-11T08:21:26.844 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.844 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.844 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:26.844 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.061917+0000", 2024-09-11T08:21:26.844 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.083906071999999998 2024-09-11T08:21:26.844 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.844 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.844 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:21:26.844 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.061964+0000", 2024-09-11T08:21:26.844 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7737000000000001e-05 2024-09-11T08:21:26.844 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.844 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.844 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:21:26.844 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.062002+0000", 2024-09-11T08:21:26.845 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7268000000000003e-05 2024-09-11T08:21:26.845 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:26.845 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:21:26.845 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:26.845 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.845 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.845 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:177470 9.3 9:c0c92d91:::benchmark_data_smithi150_133616_object61864:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-11T08:21:26.845 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:20:34.903390+0000", 2024-09-11T08:21:26.845 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 51.609033136999997, 2024-09-11T08:21:26.845 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.15876438700000001, 2024-09-11T08:21:26.845 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:21:26.845 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:21:26.845 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:21:26.845 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:21:26.845 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:21:26.846 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 177470 2024-09-11T08:21:26.846 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.846 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:21:26.846 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.846 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:21:26.846 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:34.903390+0000", 2024-09-11T08:21:26.846 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:26.846 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.846 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.846 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:21:26.846 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:34.903390+0000", 2024-09-11T08:21:26.846 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:26.847 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.847 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.847 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:21:26.847 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:34.903390+0000", 2024-09-11T08:21:26.847 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-09-11T08:21:26.847 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.847 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.847 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:21:26.847 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:34.903393+0000", 2024-09-11T08:21:26.847 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.377e-06 2024-09-11T08:21:26.847 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.847 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.847 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:21:26.847 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:34.903399+0000", 2024-09-11T08:21:26.848 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3319999999999997e-06 2024-09-11T08:21:26.848 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.848 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.848 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:21:26.848 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:34.903404+0000", 2024-09-11T08:21:26.848 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3789999999999999e-06 2024-09-11T08:21:26.848 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.848 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.848 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:26.848 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:34.968986+0000", 2024-09-11T08:21:26.848 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.065582266 2024-09-11T08:21:26.848 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.848 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.849 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:21:26.849 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:34.969058+0000", 2024-09-11T08:21:26.849 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2317999999999994e-05 2024-09-11T08:21:26.849 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.849 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.849 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-09-11T08:21:26.849 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:34.969136+0000", 2024-09-11T08:21:26.849 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7825999999999997e-05 2024-09-11T08:21:26.849 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.849 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.849 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:21:26.849 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:34.975667+0000", 2024-09-11T08:21:26.849 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0065308880000000003 2024-09-11T08:21:26.849 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.850 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.850 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:26.850 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:34.978051+0000", 2024-09-11T08:21:26.850 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0023839299999999998 2024-09-11T08:21:26.850 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.850 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.850 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:26.850 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.062061+0000", 2024-09-11T08:21:26.850 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.084010178000000005 2024-09-11T08:21:26.850 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.850 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.850 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:21:26.851 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.062108+0000", 2024-09-11T08:21:26.851 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7481999999999997e-05 2024-09-11T08:21:26.851 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.851 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.851 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:21:26.851 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.062154+0000", 2024-09-11T08:21:26.851 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5961999999999998e-05 2024-09-11T08:21:26.851 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:26.851 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:21:26.851 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:26.851 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.851 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.851 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:177753 9.3 9:c8a38ab2:::benchmark_data_smithi150_133616_object62147:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-11T08:21:26.852 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:20:35.221804+0000", 2024-09-11T08:21:26.852 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 51.290619063000001, 2024-09-11T08:21:26.852 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.216098815, 2024-09-11T08:21:26.852 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:21:26.852 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:21:26.852 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:21:26.852 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:21:26.852 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:21:26.852 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 177753 2024-09-11T08:21:26.852 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.852 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:21:26.853 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.853 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:21:26.853 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.221804+0000", 2024-09-11T08:21:26.853 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:26.853 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.853 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.853 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:21:26.853 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.221804+0000", 2024-09-11T08:21:26.853 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:26.853 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.853 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.853 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:21:26.853 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.221803+0000", 2024-09-11T08:21:26.853 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:21:26.853 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.854 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.854 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:21:26.854 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.221809+0000", 2024-09-11T08:21:26.854 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4600000000000002e-06 2024-09-11T08:21:26.854 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.854 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.854 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:21:26.854 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.221817+0000", 2024-09-11T08:21:26.854 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6200000000000005e-06 2024-09-11T08:21:26.854 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.854 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.854 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:21:26.854 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.221822+0000", 2024-09-11T08:21:26.854 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3699999999999997e-06 2024-09-11T08:21:26.854 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.855 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.855 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:26.855 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.289187+0000", 2024-09-11T08:21:26.855 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.067365507000000005 2024-09-11T08:21:26.855 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.855 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.855 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:21:26.855 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.289301+0000", 2024-09-11T08:21:26.855 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011382099999999999 2024-09-11T08:21:26.855 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.855 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.855 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-09-11T08:21:26.855 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.289418+0000", 2024-09-11T08:21:26.855 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011716499999999999 2024-09-11T08:21:26.855 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.856 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.856 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:21:26.856 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.292415+0000", 2024-09-11T08:21:26.856 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0029972290000000001 2024-09-11T08:21:26.856 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.856 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.856 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:26.856 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.348999+0000", 2024-09-11T08:21:26.856 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.056583889999999998 2024-09-11T08:21:26.856 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.856 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.856 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:26.856 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.437818+0000", 2024-09-11T08:21:26.856 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.088818580999999994 2024-09-11T08:21:26.856 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.857 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.857 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:21:26.857 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.437865+0000", 2024-09-11T08:21:26.857 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7534999999999998e-05 2024-09-11T08:21:26.857 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.857 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.857 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:21:26.857 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.437903+0000", 2024-09-11T08:21:26.857 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7653e-05 2024-09-11T08:21:26.857 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:26.857 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:21:26.857 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:26.857 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.857 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.858 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:177772 9.3 9:c215674e:::benchmark_data_smithi150_133616_object62166:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-11T08:21:26.858 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:20:35.239776+0000", 2024-09-11T08:21:26.858 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 51.272647286999998, 2024-09-11T08:21:26.858 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.198239623, 2024-09-11T08:21:26.858 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:21:26.858 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:21:26.858 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:21:26.858 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:21:26.858 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:21:26.858 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 177772 2024-09-11T08:21:26.858 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.858 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:21:26.858 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.859 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:21:26.859 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.239776+0000", 2024-09-11T08:21:26.859 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:26.859 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.859 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.859 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:21:26.859 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.239776+0000", 2024-09-11T08:21:26.859 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:26.859 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.859 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.859 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:21:26.859 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.239775+0000", 2024-09-11T08:21:26.859 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-09-11T08:21:26.859 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.859 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.860 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:21:26.860 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.239779+0000", 2024-09-11T08:21:26.860 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4470000000000002e-06 2024-09-11T08:21:26.860 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.860 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.860 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:21:26.860 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.239787+0000", 2024-09-11T08:21:26.860 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5040000000000002e-06 2024-09-11T08:21:26.861 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.861 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.861 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:21:26.861 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.239792+0000", 2024-09-11T08:21:26.861 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1150000000000002e-06 2024-09-11T08:21:26.862 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.862 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.862 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:26.862 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.293496+0000", 2024-09-11T08:21:26.862 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.053704196000000003 2024-09-11T08:21:26.862 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.862 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.862 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:21:26.862 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.293601+0000", 2024-09-11T08:21:26.862 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000104697 2024-09-11T08:21:26.863 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.863 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.863 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-09-11T08:21:26.863 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.293731+0000", 2024-09-11T08:21:26.863 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012956900000000001 2024-09-11T08:21:26.863 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.863 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.863 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:21:26.863 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.299167+0000", 2024-09-11T08:21:26.863 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0054358160000000004 2024-09-11T08:21:26.863 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.863 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.863 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:26.864 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.349022+0000", 2024-09-11T08:21:26.864 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.049855162000000001 2024-09-11T08:21:26.864 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.864 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.864 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:26.864 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.437953+0000", 2024-09-11T08:21:26.864 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.088931099 2024-09-11T08:21:26.864 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.864 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.864 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:21:26.864 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.437980+0000", 2024-09-11T08:21:26.864 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7481e-05 2024-09-11T08:21:26.864 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.864 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.865 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:21:26.865 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.438016+0000", 2024-09-11T08:21:26.865 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5225000000000002e-05 2024-09-11T08:21:26.865 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:26.865 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:21:26.865 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:26.865 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.865 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.865 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:178226 9.3 9:ce264e8e:::benchmark_data_smithi150_133616_object62620:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-11T08:21:26.865 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:20:35.833790+0000", 2024-09-11T08:21:26.865 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 50.678633482999999, 2024-09-11T08:21:26.865 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.155602196, 2024-09-11T08:21:26.865 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:21:26.866 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:21:26.866 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:21:26.866 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:21:26.866 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:21:26.866 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 178226 2024-09-11T08:21:26.866 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.866 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:21:26.866 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.866 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:21:26.867 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.833790+0000", 2024-09-11T08:21:26.867 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:26.867 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.867 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.867 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:21:26.867 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.833790+0000", 2024-09-11T08:21:26.867 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:26.867 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.867 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.867 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:21:26.867 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.833789+0000", 2024-09-11T08:21:26.867 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-09-11T08:21:26.867 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.868 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.868 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:21:26.868 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.833792+0000", 2024-09-11T08:21:26.868 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3139999999999999e-06 2024-09-11T08:21:26.868 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.868 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.868 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:21:26.868 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.833800+0000", 2024-09-11T08:21:26.868 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5800000000000003e-06 2024-09-11T08:21:26.868 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.868 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.868 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:21:26.868 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.833804+0000", 2024-09-11T08:21:26.869 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2629999999999997e-06 2024-09-11T08:21:26.869 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.869 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.869 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:26.869 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.876315+0000", 2024-09-11T08:21:26.869 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.042510494000000003 2024-09-11T08:21:26.869 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.869 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.869 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:21:26.869 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.876419+0000", 2024-09-11T08:21:26.869 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000104362 2024-09-11T08:21:26.869 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.870 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.870 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-09-11T08:21:26.870 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.876539+0000", 2024-09-11T08:21:26.870 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011982899999999999 2024-09-11T08:21:26.870 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.870 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.870 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:21:26.870 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.882861+0000", 2024-09-11T08:21:26.870 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0063216670000000004 2024-09-11T08:21:26.870 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.870 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.870 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:26.870 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.909493+0000", 2024-09-11T08:21:26.870 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.026632302 2024-09-11T08:21:26.870 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.871 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.871 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:26.871 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.989306+0000", 2024-09-11T08:21:26.871 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.079812696000000002 2024-09-11T08:21:26.871 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.871 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.871 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:21:26.871 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.989355+0000", 2024-09-11T08:21:26.871 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9829e-05 2024-09-11T08:21:26.871 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.871 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.871 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:21:26.871 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:35.989392+0000", 2024-09-11T08:21:26.872 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6554999999999998e-05 2024-09-11T08:21:26.872 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:26.872 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:21:26.872 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:26.872 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.872 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.872 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:182130 9.3 9:c9506341:::benchmark_data_smithi150_133616_object66524:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-11T08:21:26.872 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:20:40.141473+0000", 2024-09-11T08:21:26.872 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 46.370949985999999, 2024-09-11T08:21:26.872 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.16435117199999999, 2024-09-11T08:21:26.872 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:21:26.872 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:21:26.872 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:21:26.872 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:21:26.873 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:21:26.873 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 182130 2024-09-11T08:21:26.873 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.873 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:21:26.873 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.873 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:21:26.873 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:40.141473+0000", 2024-09-11T08:21:26.873 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:26.873 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.873 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.873 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:21:26.873 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:40.141473+0000", 2024-09-11T08:21:26.873 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:26.873 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.874 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.874 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:21:26.874 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:40.141472+0000", 2024-09-11T08:21:26.874 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:21:26.874 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.874 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.874 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:21:26.874 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:40.141478+0000", 2024-09-11T08:21:26.874 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8289999999999996e-06 2024-09-11T08:21:26.874 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.874 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.874 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:21:26.874 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:40.141493+0000", 2024-09-11T08:21:26.875 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5135e-05 2024-09-11T08:21:26.875 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.875 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.875 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:21:26.875 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:40.141499+0000", 2024-09-11T08:21:26.875 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.251e-06 2024-09-11T08:21:26.875 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.875 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.875 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:26.875 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:40.198151+0000", 2024-09-11T08:21:26.875 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.056651975 2024-09-11T08:21:26.875 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.875 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.875 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:21:26.876 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:40.198265+0000", 2024-09-11T08:21:26.876 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011421000000000001 2024-09-11T08:21:26.876 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.876 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.876 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-09-11T08:21:26.876 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:40.198401+0000", 2024-09-11T08:21:26.876 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000135432 2024-09-11T08:21:26.876 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.876 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.876 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:21:26.876 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:40.204307+0000", 2024-09-11T08:21:26.876 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0059062480000000002 2024-09-11T08:21:26.876 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.876 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.877 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:26.877 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:40.220622+0000", 2024-09-11T08:21:26.877 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.016315446000000001 2024-09-11T08:21:26.877 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.877 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.877 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:26.877 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:40.305760+0000", 2024-09-11T08:21:26.877 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.085137745000000001 2024-09-11T08:21:26.877 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.877 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.877 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:21:26.877 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:40.305789+0000", 2024-09-11T08:21:26.877 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8917000000000001e-05 2024-09-11T08:21:26.878 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.878 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.878 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:21:26.878 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:20:40.305824+0000", 2024-09-11T08:21:26.878 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5404999999999997e-05 2024-09-11T08:21:26.878 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:26.878 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:21:26.878 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:26.878 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.878 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.878 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:212763 9.3 9:ce43ae85:::benchmark_data_smithi150_133616_object97157:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-11T08:21:26.878 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:21:06.528719+0000", 2024-09-11T08:21:26.879 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 19.983704268, 2024-09-11T08:21:26.879 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.27869297399999998, 2024-09-11T08:21:26.879 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:21:26.879 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:21:26.879 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:21:26.879 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:21:26.879 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:21:26.879 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 212763 2024-09-11T08:21:26.879 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.879 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:21:26.879 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.879 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:21:26.879 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.528719+0000", 2024-09-11T08:21:26.880 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:26.880 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.880 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.880 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:21:26.880 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.528719+0000", 2024-09-11T08:21:26.880 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:26.880 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.880 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.880 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:21:26.880 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.528718+0000", 2024-09-11T08:21:26.880 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:21:26.880 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.880 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.881 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:21:26.881 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.528724+0000", 2024-09-11T08:21:26.881 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0059999999999998e-06 2024-09-11T08:21:26.881 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.881 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.881 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:21:26.881 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.528737+0000", 2024-09-11T08:21:26.881 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3701e-05 2024-09-11T08:21:26.881 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.881 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.881 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:21:26.881 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.528743+0000", 2024-09-11T08:21:26.881 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5480000000000001e-06 2024-09-11T08:21:26.882 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.882 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.882 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:26.882 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.529022+0000", 2024-09-11T08:21:26.882 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000279499 2024-09-11T08:21:26.882 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.882 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.882 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:21:26.883 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.529119+0000", 2024-09-11T08:21:26.883 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.6080000000000002e-05 2024-09-11T08:21:26.883 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.883 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.883 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-09-11T08:21:26.883 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.529218+0000", 2024-09-11T08:21:26.883 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9675999999999999e-05 2024-09-11T08:21:26.883 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.883 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.883 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:21:26.883 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.537360+0000", 2024-09-11T08:21:26.883 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0081422119999999994 2024-09-11T08:21:26.883 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.883 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.884 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:26.884 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.616690+0000", 2024-09-11T08:21:26.884 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.079329685999999996 2024-09-11T08:21:26.884 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.884 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.884 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:26.884 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.807330+0000", 2024-09-11T08:21:26.884 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.190640169 2024-09-11T08:21:26.884 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.884 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.884 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:21:26.884 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.807372+0000", 2024-09-11T08:21:26.884 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1925000000000003e-05 2024-09-11T08:21:26.884 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.885 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.885 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:21:26.885 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.807412+0000", 2024-09-11T08:21:26.885 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9736e-05 2024-09-11T08:21:26.885 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:26.885 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:21:26.885 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:26.885 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.885 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.885 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:212766 9.3 9:ca00ffc2:::benchmark_data_smithi150_133616_object97160:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-11T08:21:26.885 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:21:06.528973+0000", 2024-09-11T08:21:26.885 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 19.983449846999999, 2024-09-11T08:21:26.885 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.28338277899999997, 2024-09-11T08:21:26.886 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:21:26.886 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:21:26.886 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:21:26.886 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:21:26.886 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:21:26.886 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 212766 2024-09-11T08:21:26.886 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.886 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:21:26.886 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.886 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:21:26.886 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.528973+0000", 2024-09-11T08:21:26.886 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:26.886 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.886 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.886 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:21:26.887 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.528973+0000", 2024-09-11T08:21:26.887 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:26.887 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.887 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.887 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:21:26.887 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.528973+0000", 2024-09-11T08:21:26.887 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:21:26.887 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.887 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.887 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:21:26.887 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.528977+0000", 2024-09-11T08:21:26.887 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.262e-06 2024-09-11T08:21:26.887 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.887 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.888 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:21:26.888 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.528983+0000", 2024-09-11T08:21:26.888 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2169999999999999e-06 2024-09-11T08:21:26.888 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.888 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.888 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:21:26.888 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.528988+0000", 2024-09-11T08:21:26.888 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5419999999999999e-06 2024-09-11T08:21:26.888 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.888 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.888 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:26.888 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.532155+0000", 2024-09-11T08:21:26.888 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0031670380000000001 2024-09-11T08:21:26.888 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.889 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.889 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:21:26.889 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.532257+0000", 2024-09-11T08:21:26.889 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000102718 2024-09-11T08:21:26.889 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.889 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.889 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-09-11T08:21:26.889 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.532366+0000", 2024-09-11T08:21:26.889 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000108285 2024-09-11T08:21:26.889 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.889 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.889 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:21:26.889 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.539502+0000", 2024-09-11T08:21:26.890 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0071359320000000002 2024-09-11T08:21:26.890 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.890 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.890 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:26.890 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.616774+0000", 2024-09-11T08:21:26.890 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.077272399000000005 2024-09-11T08:21:26.890 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.890 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.890 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:26.890 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.812218+0000", 2024-09-11T08:21:26.890 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19544420500000001 2024-09-11T08:21:26.890 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.890 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.890 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:21:26.891 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.812309+0000", 2024-09-11T08:21:26.891 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0586000000000003e-05 2024-09-11T08:21:26.891 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.891 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.891 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:21:26.891 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.812356+0000", 2024-09-11T08:21:26.891 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7338e-05 2024-09-11T08:21:26.891 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:26.891 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:21:26.891 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:26.891 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.891 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.891 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:212772 9.3 9:cfdf609d:::benchmark_data_smithi150_133616_object97166:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-11T08:21:26.892 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:21:06.535489+0000", 2024-09-11T08:21:26.892 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 19.976933828, 2024-09-11T08:21:26.892 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.27700612099999999, 2024-09-11T08:21:26.892 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:21:26.892 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:21:26.892 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:21:26.892 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:21:26.892 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:21:26.892 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 212772 2024-09-11T08:21:26.892 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.892 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:21:26.892 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.892 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:21:26.893 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.535489+0000", 2024-09-11T08:21:26.893 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:26.893 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.893 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.893 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:21:26.893 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.535489+0000", 2024-09-11T08:21:26.893 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:26.893 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.893 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.893 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:21:26.893 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.535488+0000", 2024-09-11T08:21:26.893 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:21:26.893 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.894 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.894 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:21:26.894 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.535494+0000", 2024-09-11T08:21:26.894 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7939999999999999e-06 2024-09-11T08:21:26.894 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.894 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.894 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:21:26.894 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.535506+0000", 2024-09-11T08:21:26.894 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2506e-05 2024-09-11T08:21:26.894 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.894 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.894 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:21:26.894 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.535513+0000", 2024-09-11T08:21:26.894 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3199999999999996e-06 2024-09-11T08:21:26.895 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.895 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.895 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:26.895 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.537384+0000", 2024-09-11T08:21:26.895 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001871511 2024-09-11T08:21:26.895 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.895 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.895 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:21:26.895 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.537501+0000", 2024-09-11T08:21:26.895 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000116897 2024-09-11T08:21:26.895 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.895 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.895 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-09-11T08:21:26.896 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.537609+0000", 2024-09-11T08:21:26.896 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010806099999999999 2024-09-11T08:21:26.896 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.896 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.896 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:21:26.896 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.539605+0000", 2024-09-11T08:21:26.896 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0019953240000000001 2024-09-11T08:21:26.896 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.896 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.896 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:26.896 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.618116+0000", 2024-09-11T08:21:26.896 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.078511023999999999 2024-09-11T08:21:26.896 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.897 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.897 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:26.897 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.812401+0000", 2024-09-11T08:21:26.897 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.194285285 2024-09-11T08:21:26.897 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.897 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.897 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:21:26.897 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.812473+0000", 2024-09-11T08:21:26.897 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1606999999999994e-05 2024-09-11T08:21:26.897 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.898 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.898 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:21:26.898 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.812495+0000", 2024-09-11T08:21:26.898 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2994999999999999e-05 2024-09-11T08:21:26.898 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:26.898 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:21:26.898 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:26.898 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.898 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.898 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:212776 9.3 9:cbebc502:::benchmark_data_smithi150_133616_object97170:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-11T08:21:26.898 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:21:06.612911+0000", 2024-09-11T08:21:26.898 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 19.899512491999999, 2024-09-11T08:21:26.898 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19970517700000001, 2024-09-11T08:21:26.899 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:21:26.899 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:21:26.899 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:21:26.899 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:21:26.899 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:21:26.899 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 212776 2024-09-11T08:21:26.899 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.899 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:21:26.899 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.899 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:21:26.899 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.612911+0000", 2024-09-11T08:21:26.899 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:26.899 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.900 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.900 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:21:26.900 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.612911+0000", 2024-09-11T08:21:26.900 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:26.900 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.900 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.900 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:21:26.900 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.612910+0000", 2024-09-11T08:21:26.900 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:21:26.900 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.900 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.900 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:21:26.901 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.612914+0000", 2024-09-11T08:21:26.901 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1880000000000004e-06 2024-09-11T08:21:26.901 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.901 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.901 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:21:26.901 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.612921+0000", 2024-09-11T08:21:26.901 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2799999999999998e-06 2024-09-11T08:21:26.901 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.901 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.901 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:21:26.901 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.612927+0000", 2024-09-11T08:21:26.901 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7080000000000002e-06 2024-09-11T08:21:26.902 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.902 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.902 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:26.902 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.612971+0000", 2024-09-11T08:21:26.902 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3569999999999998e-05 2024-09-11T08:21:26.902 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.902 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.902 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:21:26.902 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.613066+0000", 2024-09-11T08:21:26.902 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.5131999999999997e-05 2024-09-11T08:21:26.902 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.902 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.903 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-09-11T08:21:26.903 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.613171+0000", 2024-09-11T08:21:26.903 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000105519 2024-09-11T08:21:26.903 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.903 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.903 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:21:26.903 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.617015+0000", 2024-09-11T08:21:26.903 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0038432459999999998 2024-09-11T08:21:26.903 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.903 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.903 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:26.903 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.618176+0000", 2024-09-11T08:21:26.903 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001161126 2024-09-11T08:21:26.904 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.904 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.904 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:26.904 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.812540+0000", 2024-09-11T08:21:26.904 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.194363903 2024-09-11T08:21:26.904 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.904 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.904 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:21:26.904 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.812581+0000", 2024-09-11T08:21:26.904 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1440000000000003e-05 2024-09-11T08:21:26.904 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.904 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.905 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:21:26.905 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.812616+0000", 2024-09-11T08:21:26.905 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4876000000000003e-05 2024-09-11T08:21:26.905 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:26.905 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:21:26.905 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:26.905 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.905 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.905 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:212780 9.d 9:b9df0a16:::benchmark_data_smithi150_133616_object97174:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-11T08:21:26.905 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:21:06.619165+0000", 2024-09-11T08:21:26.905 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 19.893258221, 2024-09-11T08:21:26.905 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.18824692700000001, 2024-09-11T08:21:26.905 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:21:26.905 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:21:26.906 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:21:26.906 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:21:26.906 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:21:26.906 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 212780 2024-09-11T08:21:26.906 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.906 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:21:26.906 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.906 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:21:26.906 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.619165+0000", 2024-09-11T08:21:26.906 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:26.906 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.906 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.906 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:21:26.906 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.619165+0000", 2024-09-11T08:21:26.907 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:26.907 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.907 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.907 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:21:26.907 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.619163+0000", 2024-09-11T08:21:26.907 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999981 2024-09-11T08:21:26.907 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.907 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.907 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:21:26.907 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.619171+0000", 2024-09-11T08:21:26.907 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8290000000000008e-06 2024-09-11T08:21:26.907 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.907 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.908 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:21:26.908 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.619184+0000", 2024-09-11T08:21:26.908 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3159000000000001e-05 2024-09-11T08:21:26.908 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.908 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.908 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:21:26.908 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.619191+0000", 2024-09-11T08:21:26.908 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6660000000000002e-06 2024-09-11T08:21:26.908 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.908 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.908 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:26.908 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.623803+0000", 2024-09-11T08:21:26.908 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.004612372 2024-09-11T08:21:26.908 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.909 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.909 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:21:26.909 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.623979+0000", 2024-09-11T08:21:26.909 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000176016 2024-09-11T08:21:26.909 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.909 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.909 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,5", 2024-09-11T08:21:26.909 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.624091+0000", 2024-09-11T08:21:26.909 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000111927 2024-09-11T08:21:26.909 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.909 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.909 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:21:26.909 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.627203+0000", 2024-09-11T08:21:26.910 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0031118299999999999 2024-09-11T08:21:26.910 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.910 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.910 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:26.910 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.627229+0000", 2024-09-11T08:21:26.910 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6786000000000001e-05 2024-09-11T08:21:26.910 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.910 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.910 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:26.910 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.807262+0000", 2024-09-11T08:21:26.910 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.18003235200000001 2024-09-11T08:21:26.910 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.911 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.911 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:21:26.911 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.807360+0000", 2024-09-11T08:21:26.911 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.8128000000000006e-05 2024-09-11T08:21:26.911 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.911 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.911 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:21:26.911 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.807412+0000", 2024-09-11T08:21:26.911 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1952e-05 2024-09-11T08:21:26.911 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:26.911 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:21:26.911 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:26.911 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.912 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.912 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:212791 9.d 9:bdbfd58b:::benchmark_data_smithi150_133616_object97185:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-11T08:21:26.912 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:21:06.632995+0000", 2024-09-11T08:21:26.912 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 19.879428274999999, 2024-09-11T08:21:26.912 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.18305706899999999, 2024-09-11T08:21:26.912 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:21:26.912 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:21:26.912 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:21:26.912 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:21:26.912 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:21:26.912 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 212791 2024-09-11T08:21:26.912 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.912 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:21:26.913 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.913 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:21:26.913 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.632995+0000", 2024-09-11T08:21:26.913 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:26.913 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.913 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.913 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:21:26.913 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.632995+0000", 2024-09-11T08:21:26.913 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:26.913 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.913 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.913 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:21:26.913 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.632993+0000", 2024-09-11T08:21:26.914 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:21:26.914 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.914 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.914 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:21:26.914 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.633002+0000", 2024-09-11T08:21:26.914 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1680000000000004e-06 2024-09-11T08:21:26.914 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.914 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.914 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:21:26.914 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.633016+0000", 2024-09-11T08:21:26.914 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4365000000000001e-05 2024-09-11T08:21:26.914 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.914 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.914 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:21:26.915 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.633023+0000", 2024-09-11T08:21:26.915 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8229999999999997e-06 2024-09-11T08:21:26.915 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.915 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.915 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:26.915 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.633068+0000", 2024-09-11T08:21:26.915 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5062999999999999e-05 2024-09-11T08:21:26.915 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.915 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.915 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:21:26.915 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.633168+0000", 2024-09-11T08:21:26.915 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000100147 2024-09-11T08:21:26.915 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.916 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.916 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,5", 2024-09-11T08:21:26.916 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.633254+0000", 2024-09-11T08:21:26.916 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5950000000000002e-05 2024-09-11T08:21:26.916 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.916 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.916 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:21:26.916 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.636179+0000", 2024-09-11T08:21:26.916 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0029254760000000002 2024-09-11T08:21:26.916 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.916 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.916 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:26.916 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.639119+0000", 2024-09-11T08:21:26.916 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0029396549999999998 2024-09-11T08:21:26.917 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.917 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.917 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:26.917 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.815930+0000", 2024-09-11T08:21:26.917 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.17681124100000001 2024-09-11T08:21:26.917 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.917 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.917 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:21:26.917 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.816015+0000", 2024-09-11T08:21:26.917 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5113999999999999e-05 2024-09-11T08:21:26.917 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.917 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.917 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:21:26.918 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.816052+0000", 2024-09-11T08:21:26.918 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6661e-05 2024-09-11T08:21:26.918 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:26.918 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:21:26.918 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:26.918 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.918 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.918 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:212792 9.3 9:cdce40cc:::benchmark_data_smithi150_133616_object97186:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-11T08:21:26.918 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:21:06.633278+0000", 2024-09-11T08:21:26.918 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 19.879145087000001, 2024-09-11T08:21:26.918 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.179419886, 2024-09-11T08:21:26.918 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:21:26.918 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:21:26.918 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:21:26.919 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:21:26.919 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:21:26.919 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 212792 2024-09-11T08:21:26.919 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.919 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:21:26.919 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.919 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:21:26.919 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.633278+0000", 2024-09-11T08:21:26.919 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:26.919 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.919 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.919 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:21:26.919 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.633278+0000", 2024-09-11T08:21:26.919 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:21:26.920 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.920 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.920 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:21:26.920 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.633277+0000", 2024-09-11T08:21:26.920 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:21:26.920 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.920 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.920 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:21:26.920 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.633296+0000", 2024-09-11T08:21:26.920 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8334000000000001e-05 2024-09-11T08:21:26.920 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.920 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.920 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:21:26.921 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.633336+0000", 2024-09-11T08:21:26.921 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0899999999999998e-05 2024-09-11T08:21:26.921 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.921 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.921 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:21:26.921 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.633343+0000", 2024-09-11T08:21:26.921 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3350000000000003e-06 2024-09-11T08:21:26.921 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.921 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.921 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:21:26.921 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.633395+0000", 2024-09-11T08:21:26.921 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2694000000000001e-05 2024-09-11T08:21:26.921 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.922 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.922 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:21:26.922 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.633508+0000", 2024-09-11T08:21:26.922 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000112656 2024-09-11T08:21:26.922 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.922 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.922 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-09-11T08:21:26.922 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.633610+0000", 2024-09-11T08:21:26.922 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000101535 2024-09-11T08:21:26.922 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.922 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.922 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:26.922 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.636713+0000", 2024-09-11T08:21:26.923 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0031028739999999998 2024-09-11T08:21:26.923 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.923 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.923 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:21:26.923 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.636823+0000", 2024-09-11T08:21:26.923 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011084199999999999 2024-09-11T08:21:26.923 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.923 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.923 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:21:26.923 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.812644+0000", 2024-09-11T08:21:26.923 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.17582038 2024-09-11T08:21:26.923 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.923 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.923 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:21:26.924 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.812668+0000", 2024-09-11T08:21:26.924 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3770000000000001e-05 2024-09-11T08:21:26.924 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:21:26.924 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:21:26.924 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:21:26.924 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.812698+0000", 2024-09-11T08:21:26.924 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0488999999999999e-05 2024-09-11T08:21:26.924 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:26.924 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:21:26.924 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:26.924 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:21:26.924 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:21:26.924 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-11T08:21:26.925 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:26.947 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:26.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:26.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:26.966 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-09-11T08:21:26.966 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-09-11T08:21:27.049 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:21:27.071 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:27.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:27.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:21:27.173 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:21:27.195 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:27.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:27.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:21:27.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:27 smithi150 ceph-mon[29276]: from='client.15567 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:21:27.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:27 smithi150 ceph-mon[29276]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:21:27.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:27 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1242407277' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:21:27.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:27 smithi150 ceph-mon[29276]: osdmap e303: 8 total, 8 up, 8 in 2024-09-11T08:21:27.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:27 smithi150 ceph-mon[39612]: from='client.15567 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:21:27.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:27 smithi150 ceph-mon[39612]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:21:27.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:27 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1242407277' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:21:27.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:27 smithi150 ceph-mon[39612]: osdmap e303: 8 total, 8 up, 8 in 2024-09-11T08:21:27.297 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:21:27.320 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:27.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:27.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:21:27.384 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:27 smithi181 ceph-mon[32701]: from='client.15567 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:21:27.385 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:27 smithi181 ceph-mon[32701]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:21:27.385 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:27 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1242407277' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:21:27.385 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:27 smithi181 ceph-mon[32701]: osdmap e303: 8 total, 8 up, 8 in 2024-09-11T08:21:27.421 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:27.443 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:27.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:27.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:27.545 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:27.566 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:27.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:27.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:27.668 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:27.688 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:27.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:27.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:27.790 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:27.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:27.813 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:27.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:27.900 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-11T08:21:27.914 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:27.935 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:27.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:27.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:28.036 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:28.057 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:28.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:28.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:28.159 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:28.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:28.186 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:28.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:28.287 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:28.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:28.309 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:28.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:28.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:28 smithi181 ceph-mon[32701]: pgmap v633: 148 pgs: 148 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-09-11T08:21:28.410 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:21:28.432 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:28.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:28.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:21:28.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:28 smithi150 ceph-mon[29276]: pgmap v633: 148 pgs: 148 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-09-11T08:21:28.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:28 smithi150 ceph-mon[39612]: pgmap v633: 148 pgs: 148 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-09-11T08:21:28.533 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:28.555 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:28.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:28.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:28.562 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":631,"stamp":"2024-09-11T08:21:25.005583+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596188,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":1676,"num_read_kb":1535,"num_write":232037,"num_write_kb":7400264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16450,"num_bytes_recovered":1064311907,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87943,"ondisk_log_size":87943,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":445,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":3593388,"kb_used_data":279340,"kb_used_omap":19,"kb_used_meta":3313964,"kb_avail":746367828,"statfs":{"total":767960285184,"available":764280655872,"internally_reserved":0,"allocated":286044160,"data_stored":283195500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19955,"internal_metadata":3393499661},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-684130328,"num_objects":-10440,"num_object_clones":0,"num_object_copies":-31320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-10440,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10440,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.017483"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":331,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.653137+0000","last_change":"2024-09-11T08:21:09.740632+0000","last_active":"2024-09-11T08:21:12.653137+0000","last_peered":"2024-09-11T08:21:12.653137+0000","last_clean":"2024-09-11T08:21:12.653137+0000","last_became_active":"2024-09-11T08:19:29.169363+0000","last_became_peered":"2024-09-11T08:19:29.169363+0000","last_unstale":"2024-09-11T08:21:12.653137+0000","last_undegraded":"2024-09-11T08:21:12.653137+0000","last_fullsized":"2024-09-11T08:21:12.653137+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:09.740551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:21:09.740551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:11.000164+0000","scrub_duration":0.0095522019999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"239'15","reported_seq":315,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.639640+0000","last_change":"2024-09-11T08:18:22.028235+0000","last_active":"2024-09-11T08:21:12.639640+0000","last_peered":"2024-09-11T08:21:12.639640+0000","last_clean":"2024-09-11T08:21:12.639640+0000","last_became_active":"2024-09-11T08:07:32.048048+0000","last_became_peered":"2024-09-11T08:07:32.048048+0000","last_unstale":"2024-09-11T08:21:12.639640+0000","last_undegraded":"2024-09-11T08:21:12.639640+0000","last_fullsized":"2024-09-11T08:21:12.639640+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'15","last_scrub_stamp":"2024-09-11T08:18:22.028181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:18:22.028181+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015902539999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":375,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.652379+0000","last_change":"2024-09-11T08:21:05.601324+0000","last_active":"2024-09-11T08:21:12.652379+0000","last_peered":"2024-09-11T08:21:12.652379+0000","last_clean":"2024-09-11T08:21:12.652379+0000","last_became_active":"2024-09-11T08:19:29.140997+0000","last_became_peered":"2024-09-11T08:19:29.140997+0000","last_unstale":"2024-09-11T08:21:12.652379+0000","last_undegraded":"2024-09-11T08:21:12.652379+0000","last_fullsized":"2024-09-11T08:21:12.652379+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:05.601245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:59.531202+0000","last_clean_scrub_stamp":"2024-09-11T08:21:05.601245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:30.011735+0000","scrub_duration":0.006405127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":319,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.640166+0000","last_change":"2024-09-11T08:20:11.523247+0000","last_active":"2024-09-11T08:21:12.640166+0000","last_peered":"2024-09-11T08:21:12.640166+0000","last_clean":"2024-09-11T08:21:12.640166+0000","last_became_active":"2024-09-11T08:19:29.147339+0000","last_became_peered":"2024-09-11T08:19:29.147339+0000","last_unstale":"2024-09-11T08:21:12.640166+0000","last_undegraded":"2024-09-11T08:21:12.640166+0000","last_fullsized":"2024-09-11T08:21:12.640166+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:11.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:02.470649+0000","last_clean_scrub_stamp":"2024-09-11T08:20:11.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0056491429999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":323,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.646751+0000","last_change":"2024-09-11T08:20:17.919584+0000","last_active":"2024-09-11T08:21:12.646751+0000","last_peered":"2024-09-11T08:21:12.646751+0000","last_clean":"2024-09-11T08:21:12.646751+0000","last_became_active":"2024-09-11T08:15:56.521104+0000","last_became_peered":"2024-09-11T08:15:56.521104+0000","last_unstale":"2024-09-11T08:21:12.646751+0000","last_undegraded":"2024-09-11T08:21:12.646751+0000","last_fullsized":"2024-09-11T08:21:12.646751+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:17.919504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:20:17.919504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:47.631226+0000","scrub_duration":0.0022382589999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":106,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.651310+0000","last_change":"2024-09-11T08:20:52.936220+0000","last_active":"2024-09-11T08:21:12.651310+0000","last_peered":"2024-09-11T08:21:12.651310+0000","last_clean":"2024-09-11T08:21:12.651310+0000","last_became_active":"2024-09-11T08:17:37.550048+0000","last_became_peered":"2024-09-11T08:17:37.550048+0000","last_unstale":"2024-09-11T08:21:12.651310+0000","last_undegraded":"2024-09-11T08:21:12.651310+0000","last_fullsized":"2024-09-11T08:21:12.651310+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:52.936088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:20:52.936088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:04.291152+0000","scrub_duration":0.0046527160000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"239'34","reported_seq":256,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.656253+0000","last_change":"2024-09-11T08:20:18.632224+0000","last_active":"2024-09-11T08:21:12.656253+0000","last_peered":"2024-09-11T08:21:12.656253+0000","last_clean":"2024-09-11T08:21:12.656253+0000","last_became_active":"2024-09-11T08:17:37.553866+0000","last_became_peered":"2024-09-11T08:17:37.553866+0000","last_unstale":"2024-09-11T08:21:12.656253+0000","last_undegraded":"2024-09-11T08:21:12.656253+0000","last_fullsized":"2024-09-11T08:21:12.656253+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'34","last_scrub_stamp":"2024-09-11T08:20:18.632165+0000","last_deep_scrub":"145'22","last_deep_scrub_stamp":"2024-09-11T08:15:57.478870+0000","last_clean_scrub_stamp":"2024-09-11T08:20:18.632165+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:26.354479+0000","scrub_duration":0.0063155520000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":288,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.639562+0000","last_change":"2024-09-11T08:18:36.998586+0000","last_active":"2024-09-11T08:21:12.639562+0000","last_peered":"2024-09-11T08:21:12.639562+0000","last_clean":"2024-09-11T08:21:12.639562+0000","last_became_active":"2024-09-11T08:07:34.052243+0000","last_became_peered":"2024-09-11T08:07:34.052243+0000","last_unstale":"2024-09-11T08:21:12.639562+0000","last_undegraded":"2024-09-11T08:21:12.639562+0000","last_fullsized":"2024-09-11T08:21:12.639562+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:36.998503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:18:36.998503+0000","last_clean_scrub_stamp":"2024-09-11T08:18:36.998503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0067563429999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":51,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.656326+0000","last_change":"2024-09-11T08:19:42.490622+0000","last_active":"2024-09-11T08:21:12.656326+0000","last_peered":"2024-09-11T08:21:12.656326+0000","last_clean":"2024-09-11T08:21:12.656326+0000","last_became_active":"2024-09-11T08:19:29.142228+0000","last_became_peered":"2024-09-11T08:19:29.142228+0000","last_unstale":"2024-09-11T08:21:12.656326+0000","last_undegraded":"2024-09-11T08:21:12.656326+0000","last_fullsized":"2024-09-11T08:21:12.656326+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:19:42.490567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:54.878409+0000","last_clean_scrub_stamp":"2024-09-11T08:19:42.490567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001226284,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"239'15","reported_seq":328,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.640419+0000","last_change":"2024-09-11T08:20:13.492673+0000","last_active":"2024-09-11T08:21:12.640419+0000","last_peered":"2024-09-11T08:21:12.640419+0000","last_clean":"2024-09-11T08:21:12.640419+0000","last_became_active":"2024-09-11T08:07:32.044626+0000","last_became_peered":"2024-09-11T08:07:32.044626+0000","last_unstale":"2024-09-11T08:21:12.640419+0000","last_undegraded":"2024-09-11T08:21:12.640419+0000","last_fullsized":"2024-09-11T08:21:12.640419+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'15","last_scrub_stamp":"2024-09-11T08:20:13.492593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:20:13.492593+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:24.083450+0000","scrub_duration":0.0023810210000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":51,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.655805+0000","last_change":"2024-09-11T08:20:18.900646+0000","last_active":"2024-09-11T08:21:12.655805+0000","last_peered":"2024-09-11T08:21:12.655805+0000","last_clean":"2024-09-11T08:21:12.655805+0000","last_became_active":"2024-09-11T08:19:29.192476+0000","last_became_peered":"2024-09-11T08:19:29.192476+0000","last_unstale":"2024-09-11T08:21:12.655805+0000","last_undegraded":"2024-09-11T08:21:12.655805+0000","last_fullsized":"2024-09-11T08:21:12.655805+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:18.900482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:20:18.900482+0000","last_clean_scrub_stamp":"2024-09-11T08:20:18.900482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:43.273628+0000","scrub_duration":0.027755386,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":312,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.651904+0000","last_change":"2024-09-11T08:19:09.244391+0000","last_active":"2024-09-11T08:21:12.651904+0000","last_peered":"2024-09-11T08:21:12.651904+0000","last_clean":"2024-09-11T08:21:12.651904+0000","last_became_active":"2024-09-11T08:14:24.792595+0000","last_became_peered":"2024-09-11T08:14:24.792595+0000","last_unstale":"2024-09-11T08:21:12.651904+0000","last_undegraded":"2024-09-11T08:21:12.651904+0000","last_fullsized":"2024-09-11T08:21:12.651904+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:19:09.244337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:19:09.244337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013304670000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":41,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.654523+0000","last_change":"2024-09-11T08:19:29.140825+0000","last_active":"2024-09-11T08:21:12.654523+0000","last_peered":"2024-09-11T08:21:12.654523+0000","last_clean":"2024-09-11T08:21:12.654523+0000","last_became_active":"2024-09-11T08:19:29.140493+0000","last_became_peered":"2024-09-11T08:19:29.140493+0000","last_unstale":"2024-09-11T08:21:12.654523+0000","last_undegraded":"2024-09-11T08:21:12.654523+0000","last_fullsized":"2024-09-11T08:21:12.654523+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:26.321692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:18:26.321692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"239'9","reported_seq":308,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.640672+0000","last_change":"2024-09-11T08:18:21.019771+0000","last_active":"2024-09-11T08:21:12.640672+0000","last_peered":"2024-09-11T08:21:12.640672+0000","last_clean":"2024-09-11T08:21:12.640672+0000","last_became_active":"2024-09-11T08:07:32.048661+0000","last_became_peered":"2024-09-11T08:07:32.048661+0000","last_unstale":"2024-09-11T08:21:12.640672+0000","last_undegraded":"2024-09-11T08:21:12.640672+0000","last_fullsized":"2024-09-11T08:21:12.640672+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'9","last_scrub_stamp":"2024-09-11T08:18:21.019685+0000","last_deep_scrub":"139'7","last_deep_scrub_stamp":"2024-09-11T08:12:24.869318+0000","last_clean_scrub_stamp":"2024-09-11T08:18:21.019685+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.021195169,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":134,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.645845+0000","last_change":"2024-09-11T08:20:53.087846+0000","last_active":"2024-09-11T08:21:12.645845+0000","last_peered":"2024-09-11T08:21:12.645845+0000","last_clean":"2024-09-11T08:21:12.645845+0000","last_became_active":"2024-09-11T08:15:56.526954+0000","last_became_peered":"2024-09-11T08:15:56.526954+0000","last_unstale":"2024-09-11T08:21:12.645845+0000","last_undegraded":"2024-09-11T08:21:12.645845+0000","last_fullsized":"2024-09-11T08:21:12.645845+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:53.087699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:20:53.087699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:22.711960+0000","scrub_duration":0.0024133610000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":51,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.655080+0000","last_change":"2024-09-11T08:20:19.930386+0000","last_active":"2024-09-11T08:21:12.655080+0000","last_peered":"2024-09-11T08:21:12.655080+0000","last_clean":"2024-09-11T08:21:12.655080+0000","last_became_active":"2024-09-11T08:19:29.143572+0000","last_became_peered":"2024-09-11T08:19:29.143572+0000","last_unstale":"2024-09-11T08:21:12.655080+0000","last_undegraded":"2024-09-11T08:21:12.655080+0000","last_fullsized":"2024-09-11T08:21:12.655080+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:19.927616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:20:19.927616+0000","last_clean_scrub_stamp":"2024-09-11T08:20:19.927616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:28.201605+0000","scrub_duration":0.0062725719999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":319,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.640208+0000","last_change":"2024-09-11T08:20:49.471916+0000","last_active":"2024-09-11T08:21:12.640208+0000","last_peered":"2024-09-11T08:21:12.640208+0000","last_clean":"2024-09-11T08:21:12.640208+0000","last_became_active":"2024-09-11T08:19:29.149081+0000","last_became_peered":"2024-09-11T08:19:29.149081+0000","last_unstale":"2024-09-11T08:21:12.640208+0000","last_undegraded":"2024-09-11T08:21:12.640208+0000","last_fullsized":"2024-09-11T08:21:12.640208+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:49.471510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:20:49.471510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:15.385674+0000","scrub_duration":0.0048182779999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":331,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.653811+0000","last_change":"2024-09-11T08:20:09.565068+0000","last_active":"2024-09-11T08:21:12.653811+0000","last_peered":"2024-09-11T08:21:12.653811+0000","last_clean":"2024-09-11T08:21:12.653811+0000","last_became_active":"2024-09-11T08:15:56.516941+0000","last_became_peered":"2024-09-11T08:15:56.516941+0000","last_unstale":"2024-09-11T08:21:12.653811+0000","last_undegraded":"2024-09-11T08:21:12.653811+0000","last_fullsized":"2024-09-11T08:21:12.653811+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:09.564941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:20:09.564941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:28.602005+0000","scrub_duration":0.0018499059999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"239'34","reported_seq":369,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.657460+0000","last_change":"2024-09-11T08:21:01.847533+0000","last_active":"2024-09-11T08:21:12.657460+0000","last_peered":"2024-09-11T08:21:12.657460+0000","last_clean":"2024-09-11T08:21:12.657460+0000","last_became_active":"2024-09-11T08:07:32.046363+0000","last_became_peered":"2024-09-11T08:07:32.046363+0000","last_unstale":"2024-09-11T08:21:12.657460+0000","last_undegraded":"2024-09-11T08:21:12.657460+0000","last_fullsized":"2024-09-11T08:21:12.657460+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'34","last_scrub_stamp":"2024-09-11T08:21:01.847438+0000","last_deep_scrub":"107'22","last_deep_scrub_stamp":"2024-09-11T08:16:04.159211+0000","last_clean_scrub_stamp":"2024-09-11T08:21:01.847438+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:16.033858+0000","scrub_duration":0.0079036850000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":318,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.653570+0000","last_change":"2024-09-11T08:19:28.870267+0000","last_active":"2024-09-11T08:21:12.653570+0000","last_peered":"2024-09-11T08:21:12.653570+0000","last_clean":"2024-09-11T08:21:12.653570+0000","last_became_active":"2024-09-11T08:07:34.055410+0000","last_became_peered":"2024-09-11T08:07:34.055410+0000","last_unstale":"2024-09-11T08:21:12.653570+0000","last_undegraded":"2024-09-11T08:21:12.653570+0000","last_fullsized":"2024-09-11T08:21:12.653570+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:19:28.870211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:19:28.870211+0000","last_clean_scrub_stamp":"2024-09-11T08:19:28.870211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001404073,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":116,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.651992+0000","last_change":"2024-09-11T08:20:18.753207+0000","last_active":"2024-09-11T08:21:12.651992+0000","last_peered":"2024-09-11T08:21:12.651992+0000","last_clean":"2024-09-11T08:21:12.651992+0000","last_became_active":"2024-09-11T08:19:29.144780+0000","last_became_peered":"2024-09-11T08:19:29.144780+0000","last_unstale":"2024-09-11T08:21:12.651992+0000","last_undegraded":"2024-09-11T08:21:12.651992+0000","last_fullsized":"2024-09-11T08:21:12.651992+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:18.753143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:20:18.753143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:33.458131+0000","scrub_duration":0.013030929,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":302,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.640539+0000","last_change":"2024-09-11T08:18:29.124017+0000","last_active":"2024-09-11T08:21:12.640539+0000","last_peered":"2024-09-11T08:21:12.640539+0000","last_clean":"2024-09-11T08:21:12.640539+0000","last_became_active":"2024-09-11T08:07:30.046284+0000","last_became_peered":"2024-09-11T08:07:30.046284+0000","last_unstale":"2024-09-11T08:21:12.640539+0000","last_undegraded":"2024-09-11T08:21:12.640539+0000","last_fullsized":"2024-09-11T08:21:12.640539+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:29.123933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:18:29.123933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026333300000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"239'18","reported_seq":166,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.657169+0000","last_change":"2024-09-11T08:20:17.626639+0000","last_active":"2024-09-11T08:21:12.657169+0000","last_peered":"2024-09-11T08:21:12.657169+0000","last_clean":"2024-09-11T08:21:12.657169+0000","last_became_active":"2024-09-11T08:17:37.553402+0000","last_became_peered":"2024-09-11T08:17:37.553402+0000","last_unstale":"2024-09-11T08:21:12.657169+0000","last_undegraded":"2024-09-11T08:21:12.657169+0000","last_fullsized":"2024-09-11T08:21:12.657169+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'18","last_scrub_stamp":"2024-09-11T08:20:17.626550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:20:17.626550+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:43.490840+0000","scrub_duration":0.015854423999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":298,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.643605+0000","last_change":"2024-09-11T08:18:37.965178+0000","last_active":"2024-09-11T08:21:12.643605+0000","last_peered":"2024-09-11T08:21:12.643605+0000","last_clean":"2024-09-11T08:21:12.643605+0000","last_became_active":"2024-09-11T08:07:34.056528+0000","last_became_peered":"2024-09-11T08:07:34.056528+0000","last_unstale":"2024-09-11T08:21:12.643605+0000","last_undegraded":"2024-09-11T08:21:12.643605+0000","last_fullsized":"2024-09-11T08:21:12.643605+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:37.964413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:10:40.110490+0000","last_clean_scrub_stamp":"2024-09-11T08:18:37.964413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042785210000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":289,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.647499+0000","last_change":"2024-09-11T08:21:05.279818+0000","last_active":"2024-09-11T08:21:12.647499+0000","last_peered":"2024-09-11T08:21:12.647499+0000","last_clean":"2024-09-11T08:21:12.647499+0000","last_became_active":"2024-09-11T08:19:29.149754+0000","last_became_peered":"2024-09-11T08:19:29.149754+0000","last_unstale":"2024-09-11T08:21:12.647499+0000","last_undegraded":"2024-09-11T08:21:12.647499+0000","last_fullsized":"2024-09-11T08:21:12.647499+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-11T08:21:05.279671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:21:05.279671+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:09.854685+0000","scrub_duration":0.0033543840000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":322,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.652329+0000","last_change":"2024-09-11T08:21:06.624368+0000","last_active":"2024-09-11T08:21:12.652329+0000","last_peered":"2024-09-11T08:21:12.652329+0000","last_clean":"2024-09-11T08:21:12.652329+0000","last_became_active":"2024-09-11T08:07:30.045932+0000","last_became_peered":"2024-09-11T08:07:30.045932+0000","last_unstale":"2024-09-11T08:21:12.652329+0000","last_undegraded":"2024-09-11T08:21:12.652329+0000","last_fullsized":"2024-09-11T08:21:12.652329+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:06.624287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:21:06.624287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:27.183588+0000","scrub_duration":0.0049003370000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"239'20","reported_seq":402,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.652660+0000","last_change":"2024-09-11T08:18:44.371881+0000","last_active":"2024-09-11T08:21:12.652660+0000","last_peered":"2024-09-11T08:21:12.652660+0000","last_clean":"2024-09-11T08:21:12.652660+0000","last_became_active":"2024-09-11T08:15:56.515817+0000","last_became_peered":"2024-09-11T08:15:56.515817+0000","last_unstale":"2024-09-11T08:21:12.652660+0000","last_undegraded":"2024-09-11T08:21:12.652660+0000","last_fullsized":"2024-09-11T08:21:12.652660+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'20","last_scrub_stamp":"2024-09-11T08:18:44.371760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:18:44.371760+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.035522666000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"254'12","reported_seq":541,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-11T08:21:17.259957+0000","last_change":"2024-09-11T08:20:19.770940+0000","last_active":"2024-09-11T08:21:17.259957+0000","last_peered":"2024-09-11T08:21:17.259957+0000","last_clean":"2024-09-11T08:21:17.259957+0000","last_became_active":"2024-09-11T08:15:56.525498+0000","last_became_peered":"2024-09-11T08:15:56.525498+0000","last_unstale":"2024-09-11T08:21:17.259957+0000","last_undegraded":"2024-09-11T08:21:17.259957+0000","last_fullsized":"2024-09-11T08:21:17.259957+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'12","last_scrub_stamp":"2024-09-11T08:20:19.770841+0000","last_deep_scrub":"254'12","last_deep_scrub_stamp":"2024-09-11T08:18:34.253865+0000","last_clean_scrub_stamp":"2024-09-11T08:20:19.770841+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:37.378623+0000","scrub_duration":0.0026189569999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2","version":"302'14420","reported_seq":14694,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-11T08:21:22.334216+0000","last_change":"2024-09-11T08:20:13.017981+0000","last_active":"2024-09-11T08:21:22.334216+0000","last_peered":"2024-09-11T08:21:22.334216+0000","last_clean":"2024-09-11T08:21:22.334216+0000","last_became_active":"2024-09-11T08:18:01.133542+0000","last_became_peered":"2024-09-11T08:18:01.133542+0000","last_unstale":"2024-09-11T08:21:22.334216+0000","last_undegraded":"2024-09-11T08:21:22.334216+0000","last_fullsized":"2024-09-11T08:21:22.334216+0000","mapping_epoch":246,"log_start":"283'9663","ondisk_log_start":"283'9663","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'9512","last_scrub_stamp":"2024-09-11T08:20:13.017898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:20:13.017898+0000","objects_scrubbed":4986,"log_size":4757,"log_dups_size":0,"ondisk_log_size":4757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:30.441437+0000","scrub_duration":2.0105805170000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14420,"num_write_kb":461440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"239'27","reported_seq":373,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.653129+0000","last_change":"2024-09-11T08:21:11.723564+0000","last_active":"2024-09-11T08:21:12.653129+0000","last_peered":"2024-09-11T08:21:12.653129+0000","last_clean":"2024-09-11T08:21:12.653129+0000","last_became_active":"2024-09-11T08:07:32.054317+0000","last_became_peered":"2024-09-11T08:07:32.054317+0000","last_unstale":"2024-09-11T08:21:12.653129+0000","last_undegraded":"2024-09-11T08:21:12.653129+0000","last_fullsized":"2024-09-11T08:21:12.653129+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'27","last_scrub_stamp":"2024-09-11T08:21:11.722291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:21:11.722291+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:25.479711+0000","scrub_duration":0.024639298,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":332,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.645476+0000","last_change":"2024-09-11T08:20:19.839288+0000","last_active":"2024-09-11T08:21:12.645476+0000","last_peered":"2024-09-11T08:21:12.645476+0000","last_clean":"2024-09-11T08:21:12.645476+0000","last_became_active":"2024-09-11T08:15:56.521943+0000","last_became_peered":"2024-09-11T08:15:56.521943+0000","last_unstale":"2024-09-11T08:21:12.645476+0000","last_undegraded":"2024-09-11T08:21:12.645476+0000","last_fullsized":"2024-09-11T08:21:12.645476+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:19.839221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:20:19.839221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:30.887348+0000","scrub_duration":0.0023374770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"271'10","reported_seq":500,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-11T08:21:17.270420+0000","last_change":"2024-09-11T08:19:29.165123+0000","last_active":"2024-09-11T08:21:17.270420+0000","last_peered":"2024-09-11T08:21:17.270420+0000","last_clean":"2024-09-11T08:21:17.270420+0000","last_became_active":"2024-09-11T08:19:29.149773+0000","last_became_peered":"2024-09-11T08:19:29.149773+0000","last_unstale":"2024-09-11T08:21:17.270420+0000","last_undegraded":"2024-09-11T08:21:17.270420+0000","last_fullsized":"2024-09-11T08:21:17.270420+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:18:44.817179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:18:44.817179+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022442809999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":51,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.656022+0000","last_change":"2024-09-11T08:20:10.875817+0000","last_active":"2024-09-11T08:21:12.656022+0000","last_peered":"2024-09-11T08:21:12.656022+0000","last_clean":"2024-09-11T08:21:12.656022+0000","last_became_active":"2024-09-11T08:19:29.164735+0000","last_became_peered":"2024-09-11T08:19:29.164735+0000","last_unstale":"2024-09-11T08:21:12.656022+0000","last_undegraded":"2024-09-11T08:21:12.656022+0000","last_fullsized":"2024-09-11T08:21:12.656022+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:10.875651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:06.616332+0000","last_clean_scrub_stamp":"2024-09-11T08:20:10.875651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:22.781185+0000","scrub_duration":0.018223294000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"302'14360","reported_seq":28434,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-11T08:21:22.318295+0000","last_change":"2024-09-11T08:20:58.754746+0000","last_active":"2024-09-11T08:21:22.318295+0000","last_peered":"2024-09-11T08:21:22.318295+0000","last_clean":"2024-09-11T08:21:22.318295+0000","last_became_active":"2024-09-11T08:19:43.795082+0000","last_became_peered":"2024-09-11T08:19:43.795082+0000","last_unstale":"2024-09-11T08:21:22.318295+0000","last_undegraded":"2024-09-11T08:21:22.318295+0000","last_fullsized":"2024-09-11T08:21:22.318295+0000","mapping_epoch":278,"log_start":"283'9451","ondisk_log_start":"283'9451","created":246,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"297'12668","last_scrub_stamp":"2024-09-11T08:20:58.754649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:20:58.754649+0000","objects_scrubbed":1702,"log_size":4909,"log_dups_size":0,"ondisk_log_size":4909,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:27.480813+0000","scrub_duration":0.64335482200000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14360,"num_write_kb":459457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6940,"num_bytes_recovered":454819840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"239'30","reported_seq":358,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.644822+0000","last_change":"2024-09-11T08:20:15.829563+0000","last_active":"2024-09-11T08:21:12.644822+0000","last_peered":"2024-09-11T08:21:12.644822+0000","last_clean":"2024-09-11T08:21:12.644822+0000","last_became_active":"2024-09-11T08:17:37.555703+0000","last_became_peered":"2024-09-11T08:17:37.555703+0000","last_unstale":"2024-09-11T08:21:12.644822+0000","last_undegraded":"2024-09-11T08:21:12.644822+0000","last_fullsized":"2024-09-11T08:21:12.644822+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'30","last_scrub_stamp":"2024-09-11T08:20:15.829476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:20:15.829476+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:39.569146+0000","scrub_duration":0.0058827109999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":292,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.645382+0000","last_change":"2024-09-11T08:21:02.160052+0000","last_active":"2024-09-11T08:21:12.645382+0000","last_peered":"2024-09-11T08:21:12.645382+0000","last_clean":"2024-09-11T08:21:12.645382+0000","last_became_active":"2024-09-11T08:17:37.553579+0000","last_became_peered":"2024-09-11T08:17:37.553579+0000","last_unstale":"2024-09-11T08:21:12.645382+0000","last_undegraded":"2024-09-11T08:21:12.645382+0000","last_fullsized":"2024-09-11T08:21:12.645382+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:02.159910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:21:02.159910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:20.376173+0000","scrub_duration":0.0086008690000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":325,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.641931+0000","last_change":"2024-09-11T08:20:57.523721+0000","last_active":"2024-09-11T08:21:12.641931+0000","last_peered":"2024-09-11T08:21:12.641931+0000","last_clean":"2024-09-11T08:21:12.641931+0000","last_became_active":"2024-09-11T08:15:56.522606+0000","last_became_peered":"2024-09-11T08:15:56.522606+0000","last_unstale":"2024-09-11T08:21:12.641931+0000","last_undegraded":"2024-09-11T08:21:12.641931+0000","last_fullsized":"2024-09-11T08:21:12.641931+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:57.523652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:20:57.523652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:19.553362+0000","scrub_duration":0.0041561580000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":299,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.640332+0000","last_change":"2024-09-11T08:20:28.457738+0000","last_active":"2024-09-11T08:21:12.640332+0000","last_peered":"2024-09-11T08:21:12.640332+0000","last_clean":"2024-09-11T08:21:12.640332+0000","last_became_active":"2024-09-11T08:07:36.069225+0000","last_became_peered":"2024-09-11T08:07:36.069225+0000","last_unstale":"2024-09-11T08:21:12.640332+0000","last_undegraded":"2024-09-11T08:21:12.640332+0000","last_fullsized":"2024-09-11T08:21:12.640332+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:28.457660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:04.937884+0000","last_clean_scrub_stamp":"2024-09-11T08:20:28.457660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:33.621713+0000","scrub_duration":0.0028279630000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"302'14396","reported_seq":14514,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-11T08:21:22.326504+0000","last_change":"2024-09-11T08:21:08.264334+0000","last_active":"2024-09-11T08:21:22.326504+0000","last_peered":"2024-09-11T08:21:22.326504+0000","last_clean":"2024-09-11T08:21:22.326504+0000","last_became_active":"2024-09-11T08:18:00.823223+0000","last_became_peered":"2024-09-11T08:18:00.823223+0000","last_unstale":"2024-09-11T08:21:22.326504+0000","last_undegraded":"2024-09-11T08:21:22.326504+0000","last_fullsized":"2024-09-11T08:21:22.326504+0000","mapping_epoch":246,"log_start":"283'9622","ondisk_log_start":"283'9622","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"298'13336","last_scrub_stamp":"2024-09-11T08:21:08.264166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:21:08.264166+0000","objects_scrubbed":1078,"log_size":4774,"log_dups_size":0,"ondisk_log_size":4774,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:14.949962+0000","scrub_duration":0.48416262700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14396,"num_write_kb":460672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"239'28","reported_seq":394,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.643609+0000","last_change":"2024-09-11T08:20:55.458131+0000","last_active":"2024-09-11T08:21:12.643609+0000","last_peered":"2024-09-11T08:21:12.643609+0000","last_clean":"2024-09-11T08:21:12.643609+0000","last_became_active":"2024-09-11T08:19:29.148311+0000","last_became_peered":"2024-09-11T08:19:29.148311+0000","last_unstale":"2024-09-11T08:21:12.643609+0000","last_undegraded":"2024-09-11T08:21:12.643609+0000","last_fullsized":"2024-09-11T08:21:12.643609+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'28","last_scrub_stamp":"2024-09-11T08:20:55.458034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:20:55.458034+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:08.620710+0000","scrub_duration":0.0051959880000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":311,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.652889+0000","last_change":"2024-09-11T08:20:26.665113+0000","last_active":"2024-09-11T08:21:12.652889+0000","last_peered":"2024-09-11T08:21:12.652889+0000","last_clean":"2024-09-11T08:21:12.652889+0000","last_became_active":"2024-09-11T08:07:30.050484+0000","last_became_peered":"2024-09-11T08:07:30.050484+0000","last_unstale":"2024-09-11T08:21:12.652889+0000","last_undegraded":"2024-09-11T08:21:12.652889+0000","last_fullsized":"2024-09-11T08:21:12.652889+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:26.665013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:47.960329+0000","last_clean_scrub_stamp":"2024-09-11T08:20:26.665013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:26.753491+0000","scrub_duration":0.0089047229999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"82'9","reported_seq":496,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-11T08:21:17.254540+0000","last_change":"2024-09-11T08:21:00.841525+0000","last_active":"2024-09-11T08:21:17.254540+0000","last_peered":"2024-09-11T08:21:17.254540+0000","last_clean":"2024-09-11T08:21:17.254540+0000","last_became_active":"2024-09-11T08:07:34.056143+0000","last_became_peered":"2024-09-11T08:07:34.056143+0000","last_unstale":"2024-09-11T08:21:17.254540+0000","last_undegraded":"2024-09-11T08:21:17.254540+0000","last_fullsized":"2024-09-11T08:21:17.254540+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-11T08:21:00.841426+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-09-11T08:21:00.841426+0000","last_clean_scrub_stamp":"2024-09-11T08:21:00.841426+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:04.644815+0000","scrub_duration":0.0051535979999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":286,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.644806+0000","last_change":"2024-09-11T08:20:22.768959+0000","last_active":"2024-09-11T08:21:12.644806+0000","last_peered":"2024-09-11T08:21:12.644806+0000","last_clean":"2024-09-11T08:21:12.644806+0000","last_became_active":"2024-09-11T08:14:24.788968+0000","last_became_peered":"2024-09-11T08:14:24.788968+0000","last_unstale":"2024-09-11T08:21:12.644806+0000","last_undegraded":"2024-09-11T08:21:12.644806+0000","last_fullsized":"2024-09-11T08:21:12.644806+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:22.768817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:20:22.768817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:28.060784+0000","scrub_duration":0.006813398,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"302'14626","reported_seq":18144,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-11T08:21:22.319891+0000","last_change":"2024-09-11T08:20:27.495067+0000","last_active":"2024-09-11T08:21:22.319891+0000","last_peered":"2024-09-11T08:21:22.319891+0000","last_clean":"2024-09-11T08:21:22.319891+0000","last_became_active":"2024-09-11T08:19:30.147415+0000","last_became_peered":"2024-09-11T08:19:30.147415+0000","last_unstale":"2024-09-11T08:21:22.319891+0000","last_undegraded":"2024-09-11T08:21:22.319891+0000","last_fullsized":"2024-09-11T08:21:22.319891+0000","mapping_epoch":271,"log_start":"283'9269","ondisk_log_start":"283'9269","created":246,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'10749","last_scrub_stamp":"2024-09-11T08:20:27.495011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:20:27.495011+0000","objects_scrubbed":3912,"log_size":5357,"log_dups_size":0,"ondisk_log_size":5357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:54.389445+0000","scrub_duration":1.5107946880000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14626,"num_write_kb":468032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1634,"num_bytes_recovered":107085824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"239'20","reported_seq":390,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.653223+0000","last_change":"2024-09-11T08:20:13.659040+0000","last_active":"2024-09-11T08:21:12.653223+0000","last_peered":"2024-09-11T08:21:12.653223+0000","last_clean":"2024-09-11T08:21:12.653223+0000","last_became_active":"2024-09-11T08:19:29.143486+0000","last_became_peered":"2024-09-11T08:19:29.143486+0000","last_unstale":"2024-09-11T08:21:12.653223+0000","last_undegraded":"2024-09-11T08:21:12.653223+0000","last_fullsized":"2024-09-11T08:21:12.653223+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'20","last_scrub_stamp":"2024-09-11T08:20:13.658885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:20:13.658885+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:41.943049+0000","scrub_duration":0.012626671000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":51,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.656121+0000","last_change":"2024-09-11T08:20:55.691331+0000","last_active":"2024-09-11T08:21:12.656121+0000","last_peered":"2024-09-11T08:21:12.656121+0000","last_clean":"2024-09-11T08:21:12.656121+0000","last_became_active":"2024-09-11T08:19:29.143310+0000","last_became_peered":"2024-09-11T08:19:29.143310+0000","last_unstale":"2024-09-11T08:21:12.656121+0000","last_undegraded":"2024-09-11T08:21:12.656121+0000","last_fullsized":"2024-09-11T08:21:12.656121+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:55.690500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:20:55.690500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:09.255593+0000","scrub_duration":0.013778218,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"271'12","reported_seq":161,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-11T08:21:17.259570+0000","last_change":"2024-09-11T08:20:25.901406+0000","last_active":"2024-09-11T08:21:17.259570+0000","last_peered":"2024-09-11T08:21:17.259570+0000","last_clean":"2024-09-11T08:21:17.259570+0000","last_became_active":"2024-09-11T08:19:29.144614+0000","last_became_peered":"2024-09-11T08:19:29.144614+0000","last_unstale":"2024-09-11T08:21:17.259570+0000","last_undegraded":"2024-09-11T08:21:17.259570+0000","last_fullsized":"2024-09-11T08:21:17.259570+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'12","last_scrub_stamp":"2024-09-11T08:20:25.900976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:20:25.900976+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:33.218880+0000","scrub_duration":0.0053349850000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"66'1","reported_seq":296,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.652079+0000","last_change":"2024-09-11T08:20:28.587056+0000","last_active":"2024-09-11T08:21:12.652079+0000","last_peered":"2024-09-11T08:21:12.652079+0000","last_clean":"2024-09-11T08:21:12.652079+0000","last_became_active":"2024-09-11T08:07:36.072593+0000","last_became_peered":"2024-09-11T08:07:36.072593+0000","last_unstale":"2024-09-11T08:21:12.652079+0000","last_undegraded":"2024-09-11T08:21:12.652079+0000","last_fullsized":"2024-09-11T08:21:12.652079+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-11T08:20:28.586987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:20:28.586987+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:28.588685+0000","scrub_duration":0.0034752110000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"302'14388","reported_seq":14539,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-11T08:21:22.335026+0000","last_change":"2024-09-11T08:20:55.862684+0000","last_active":"2024-09-11T08:21:22.335026+0000","last_peered":"2024-09-11T08:21:22.335026+0000","last_clean":"2024-09-11T08:21:22.335026+0000","last_became_active":"2024-09-11T08:18:00.825771+0000","last_became_peered":"2024-09-11T08:18:00.825771+0000","last_unstale":"2024-09-11T08:21:22.335026+0000","last_undegraded":"2024-09-11T08:21:22.335026+0000","last_fullsized":"2024-09-11T08:21:22.335026+0000","mapping_epoch":246,"log_start":"283'9465","ondisk_log_start":"283'9465","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'12465","last_scrub_stamp":"2024-09-11T08:20:55.862600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:20:55.862600+0000","objects_scrubbed":1941,"log_size":4923,"log_dups_size":0,"ondisk_log_size":4923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:56.821139+0000","scrub_duration":0.724694862,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14388,"num_write_kb":460416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"239'29","reported_seq":469,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.653583+0000","last_change":"2024-09-11T08:19:29.249993+0000","last_active":"2024-09-11T08:21:12.653583+0000","last_peered":"2024-09-11T08:21:12.653583+0000","last_clean":"2024-09-11T08:21:12.653583+0000","last_became_active":"2024-09-11T08:19:29.195995+0000","last_became_peered":"2024-09-11T08:19:29.195995+0000","last_unstale":"2024-09-11T08:21:12.653583+0000","last_undegraded":"2024-09-11T08:21:12.653583+0000","last_fullsized":"2024-09-11T08:21:12.653583+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'29","last_scrub_stamp":"2024-09-11T08:18:34.333462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:18:34.333462+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011867379000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":345,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.652904+0000","last_change":"2024-09-11T08:20:19.622253+0000","last_active":"2024-09-11T08:21:12.652904+0000","last_peered":"2024-09-11T08:21:12.652904+0000","last_clean":"2024-09-11T08:21:12.652904+0000","last_became_active":"2024-09-11T08:19:29.141999+0000","last_became_peered":"2024-09-11T08:19:29.141999+0000","last_unstale":"2024-09-11T08:21:12.652904+0000","last_undegraded":"2024-09-11T08:21:12.652904+0000","last_fullsized":"2024-09-11T08:21:12.652904+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:19.622169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:20:19.622169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:31.390592+0000","scrub_duration":0.0054018440000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":347,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.651190+0000","last_change":"2024-09-11T08:20:51.918821+0000","last_active":"2024-09-11T08:21:12.651190+0000","last_peered":"2024-09-11T08:21:12.651190+0000","last_clean":"2024-09-11T08:21:12.651190+0000","last_became_active":"2024-09-11T08:19:29.166609+0000","last_became_peered":"2024-09-11T08:19:29.166609+0000","last_unstale":"2024-09-11T08:21:12.651190+0000","last_undegraded":"2024-09-11T08:21:12.651190+0000","last_fullsized":"2024-09-11T08:21:12.651190+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:51.918741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:20:51.918741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:59.899889+0000","scrub_duration":0.0090788139999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":316,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.651694+0000","last_change":"2024-09-11T08:20:56.715356+0000","last_active":"2024-09-11T08:21:12.651694+0000","last_peered":"2024-09-11T08:21:12.651694+0000","last_clean":"2024-09-11T08:21:12.651694+0000","last_became_active":"2024-09-11T08:07:36.631668+0000","last_became_peered":"2024-09-11T08:07:36.631668+0000","last_unstale":"2024-09-11T08:21:12.651694+0000","last_undegraded":"2024-09-11T08:21:12.651694+0000","last_fullsized":"2024-09-11T08:21:12.651694+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:56.715273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:58.517624+0000","last_clean_scrub_stamp":"2024-09-11T08:20:56.715273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:25.260940+0000","scrub_duration":0.0021168839999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"302'13964","reported_seq":15967,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-11T08:21:22.323135+0000","last_change":"2024-09-11T08:20:24.076273+0000","last_active":"2024-09-11T08:21:22.323135+0000","last_peered":"2024-09-11T08:21:22.323135+0000","last_clean":"2024-09-11T08:21:22.323135+0000","last_became_active":"2024-09-11T08:19:41.594593+0000","last_became_peered":"2024-09-11T08:19:41.594593+0000","last_unstale":"2024-09-11T08:21:22.323135+0000","last_undegraded":"2024-09-11T08:21:22.323135+0000","last_fullsized":"2024-09-11T08:21:22.323135+0000","mapping_epoch":276,"log_start":"280'7641","ondisk_log_start":"280'7641","created":246,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"287'10084","last_scrub_stamp":"2024-09-11T08:20:24.076187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:20:24.076187+0000","objects_scrubbed":3917,"log_size":6323,"log_dups_size":0,"ondisk_log_size":6323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:48.089673+0000","scrub_duration":1.449697636,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13964,"num_write_kb":446848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":875,"num_bytes_recovered":57344000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"296'52","reported_seq":54,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-11T08:21:17.269765+0000","last_change":"2024-09-11T08:20:53.645194+0000","last_active":"2024-09-11T08:21:17.269765+0000","last_peered":"2024-09-11T08:21:17.269765+0000","last_clean":"2024-09-11T08:21:17.269765+0000","last_became_active":"2024-09-11T08:20:52.086843+0000","last_became_peered":"2024-09-11T08:20:52.086843+0000","last_unstale":"2024-09-11T08:21:17.269765+0000","last_undegraded":"2024-09-11T08:21:17.269765+0000","last_fullsized":"2024-09-11T08:21:17.269765+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'51","last_scrub_stamp":"2024-09-11T08:20:53.645121+0000","last_deep_scrub":"190'39","last_deep_scrub_stamp":"2024-09-11T08:15:52.847968+0000","last_clean_scrub_stamp":"2024-09-11T08:20:53.645121+0000","objects_scrubbed":10,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:09.785967+0000","scrub_duration":0.0034138189999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"59'1","reported_seq":311,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.640933+0000","last_change":"2024-09-11T08:18:28.131832+0000","last_active":"2024-09-11T08:21:12.640933+0000","last_peered":"2024-09-11T08:21:12.640933+0000","last_clean":"2024-09-11T08:21:12.640933+0000","last_became_active":"2024-09-11T08:07:30.047513+0000","last_became_peered":"2024-09-11T08:07:30.047513+0000","last_unstale":"2024-09-11T08:21:12.640933+0000","last_undegraded":"2024-09-11T08:21:12.640933+0000","last_fullsized":"2024-09-11T08:21:12.640933+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-11T08:18:28.131735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:18:28.131735+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0055372269999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":45,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.654860+0000","last_change":"2024-09-11T08:19:29.157831+0000","last_active":"2024-09-11T08:21:12.654860+0000","last_peered":"2024-09-11T08:21:12.654860+0000","last_clean":"2024-09-11T08:21:12.654860+0000","last_became_active":"2024-09-11T08:19:29.142111+0000","last_became_peered":"2024-09-11T08:19:29.142111+0000","last_unstale":"2024-09-11T08:21:12.654860+0000","last_undegraded":"2024-09-11T08:21:12.654860+0000","last_fullsized":"2024-09-11T08:21:12.654860+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-11T08:18:35.267235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:18:35.267235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":297,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.654185+0000","last_change":"2024-09-11T08:18:40.347163+0000","last_active":"2024-09-11T08:21:12.654185+0000","last_peered":"2024-09-11T08:21:12.654185+0000","last_clean":"2024-09-11T08:21:12.654185+0000","last_became_active":"2024-09-11T08:07:34.057154+0000","last_became_peered":"2024-09-11T08:07:34.057154+0000","last_unstale":"2024-09-11T08:21:12.654185+0000","last_undegraded":"2024-09-11T08:21:12.654185+0000","last_fullsized":"2024-09-11T08:21:12.654185+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:40.346999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:10:37.957096+0000","last_clean_scrub_stamp":"2024-09-11T08:18:40.346999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0055147520000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":333,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.646935+0000","last_change":"2024-09-11T08:20:55.800959+0000","last_active":"2024-09-11T08:21:12.646935+0000","last_peered":"2024-09-11T08:21:12.646935+0000","last_clean":"2024-09-11T08:21:12.646935+0000","last_became_active":"2024-09-11T08:15:56.525099+0000","last_became_peered":"2024-09-11T08:15:56.525099+0000","last_unstale":"2024-09-11T08:21:12.646935+0000","last_undegraded":"2024-09-11T08:21:12.646935+0000","last_fullsized":"2024-09-11T08:21:12.646935+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:55.800856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:20:55.800856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:07.233576+0000","scrub_duration":0.008329276,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"302'16962","reported_seq":20253,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-11T08:21:22.313197+0000","last_change":"2024-09-11T08:20:54.440180+0000","last_active":"2024-09-11T08:21:22.313197+0000","last_peered":"2024-09-11T08:21:22.313197+0000","last_clean":"2024-09-11T08:21:22.313197+0000","last_became_active":"2024-09-11T08:19:30.136920+0000","last_became_peered":"2024-09-11T08:19:30.136920+0000","last_unstale":"2024-09-11T08:21:22.313197+0000","last_undegraded":"2024-09-11T08:21:22.313197+0000","last_fullsized":"2024-09-11T08:21:22.313197+0000","mapping_epoch":271,"log_start":"290'11604","ondisk_log_start":"290'11604","created":246,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'14982","last_scrub_stamp":"2024-09-11T08:20:54.438550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:20:54.438550+0000","objects_scrubbed":2007,"log_size":5358,"log_dups_size":0,"ondisk_log_size":5358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:59.543999+0000","scrub_duration":0.80989290199999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14508,"num_write_kb":464256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1507,"num_bytes_recovered":98762752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"239'17","reported_seq":400,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.653428+0000","last_change":"2024-09-11T08:20:27.578967+0000","last_active":"2024-09-11T08:21:12.653428+0000","last_peered":"2024-09-11T08:21:12.653428+0000","last_clean":"2024-09-11T08:21:12.653428+0000","last_became_active":"2024-09-11T08:19:29.161554+0000","last_became_peered":"2024-09-11T08:19:29.161554+0000","last_unstale":"2024-09-11T08:21:12.653428+0000","last_undegraded":"2024-09-11T08:21:12.653428+0000","last_fullsized":"2024-09-11T08:21:12.653428+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'17","last_scrub_stamp":"2024-09-11T08:20:27.578887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:20:27.578887+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:37.678491+0000","scrub_duration":0.0043509350000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":325,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.651531+0000","last_change":"2024-09-11T08:21:10.701859+0000","last_active":"2024-09-11T08:21:12.651531+0000","last_peered":"2024-09-11T08:21:12.651531+0000","last_clean":"2024-09-11T08:21:12.651531+0000","last_became_active":"2024-09-11T08:07:30.047014+0000","last_became_peered":"2024-09-11T08:07:30.047014+0000","last_unstale":"2024-09-11T08:21:12.651531+0000","last_undegraded":"2024-09-11T08:21:12.651531+0000","last_fullsized":"2024-09-11T08:21:12.651531+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:10.701767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:21:10.701767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:20.537530+0000","scrub_duration":0.0056318200000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"127'2","reported_seq":341,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.656323+0000","last_change":"2024-09-11T08:20:57.886361+0000","last_active":"2024-09-11T08:21:12.656323+0000","last_peered":"2024-09-11T08:21:12.656323+0000","last_clean":"2024-09-11T08:21:12.656323+0000","last_became_active":"2024-09-11T08:07:30.045009+0000","last_became_peered":"2024-09-11T08:07:30.045009+0000","last_unstale":"2024-09-11T08:21:12.656323+0000","last_undegraded":"2024-09-11T08:21:12.656323+0000","last_fullsized":"2024-09-11T08:21:12.656323+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'2","last_scrub_stamp":"2024-09-11T08:20:57.886279+0000","last_deep_scrub":"127'2","last_deep_scrub_stamp":"2024-09-11T08:15:01.938497+0000","last_clean_scrub_stamp":"2024-09-11T08:20:57.886279+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:02.854851+0000","scrub_duration":0.0047039409999999997,"objects_trimmed":0,"snaptrim_duration":0.001859533,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":360,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.657649+0000","last_change":"2024-09-11T08:20:52.830754+0000","last_active":"2024-09-11T08:21:12.657649+0000","last_peered":"2024-09-11T08:21:12.657649+0000","last_clean":"2024-09-11T08:21:12.657649+0000","last_became_active":"2024-09-11T08:19:29.145168+0000","last_became_peered":"2024-09-11T08:19:29.145168+0000","last_unstale":"2024-09-11T08:21:12.657649+0000","last_undegraded":"2024-09-11T08:21:12.657649+0000","last_fullsized":"2024-09-11T08:21:12.657649+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:52.830600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:58.889094+0000","last_clean_scrub_stamp":"2024-09-11T08:20:52.830600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:58.857313+0000","scrub_duration":0.024812523,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":27,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.657102+0000","last_change":"2024-09-11T08:20:16.108984+0000","last_active":"2024-09-11T08:21:12.657102+0000","last_peered":"2024-09-11T08:21:12.657102+0000","last_clean":"2024-09-11T08:21:12.657102+0000","last_became_active":"2024-09-11T08:20:16.108674+0000","last_became_peered":"2024-09-11T08:20:16.108674+0000","last_unstale":"2024-09-11T08:21:12.657102+0000","last_undegraded":"2024-09-11T08:21:12.657102+0000","last_fullsized":"2024-09-11T08:21:12.657102+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:11.804556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:58.796217+0000","last_clean_scrub_stamp":"2024-09-11T08:20:11.804556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:18.829561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.b","version":"302'14424","reported_seq":16494,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-11T08:21:22.325245+0000","last_change":"2024-09-11T08:20:22.228081+0000","last_active":"2024-09-11T08:21:22.325245+0000","last_peered":"2024-09-11T08:21:22.325245+0000","last_clean":"2024-09-11T08:21:22.325245+0000","last_became_active":"2024-09-11T08:19:38.767293+0000","last_became_peered":"2024-09-11T08:19:38.767293+0000","last_unstale":"2024-09-11T08:21:22.325245+0000","last_undegraded":"2024-09-11T08:21:22.325245+0000","last_fullsized":"2024-09-11T08:21:22.325245+0000","mapping_epoch":273,"log_start":"282'8094","ondisk_log_start":"282'8094","created":246,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"287'10198","last_scrub_stamp":"2024-09-11T08:20:22.228004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:20:22.228004+0000","objects_scrubbed":4270,"log_size":6330,"log_dups_size":0,"ondisk_log_size":6330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:46.795828+0000","scrub_duration":1.647098065,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14424,"num_write_kb":461568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":897,"num_bytes_recovered":58785792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"287'30","reported_seq":351,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.652736+0000","last_change":"2024-09-11T08:20:24.667229+0000","last_active":"2024-09-11T08:21:12.652736+0000","last_peered":"2024-09-11T08:21:12.652736+0000","last_clean":"2024-09-11T08:21:12.652736+0000","last_became_active":"2024-09-11T08:07:32.055407+0000","last_became_peered":"2024-09-11T08:07:32.055407+0000","last_unstale":"2024-09-11T08:21:12.652736+0000","last_undegraded":"2024-09-11T08:21:12.652736+0000","last_fullsized":"2024-09-11T08:21:12.652736+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"287'30","last_scrub_stamp":"2024-09-11T08:20:24.667139+0000","last_deep_scrub":"287'30","last_deep_scrub_stamp":"2024-09-11T08:20:24.667139+0000","last_clean_scrub_stamp":"2024-09-11T08:20:24.667139+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:28.622072+0000","scrub_duration":0.030013558999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"59'1","reported_seq":318,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.652188+0000","last_change":"2024-09-11T08:20:53.779883+0000","last_active":"2024-09-11T08:21:12.652188+0000","last_peered":"2024-09-11T08:21:12.652188+0000","last_clean":"2024-09-11T08:21:12.652188+0000","last_became_active":"2024-09-11T08:07:30.050504+0000","last_became_peered":"2024-09-11T08:07:30.050504+0000","last_unstale":"2024-09-11T08:21:12.652188+0000","last_undegraded":"2024-09-11T08:21:12.652188+0000","last_fullsized":"2024-09-11T08:21:12.652188+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-11T08:20:53.779281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:20:53.779281+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:21.136677+0000","scrub_duration":0.0097612610000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"224'6","reported_seq":922,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-11T08:21:21.626003+0000","last_change":"2024-09-11T08:19:28.913001+0000","last_active":"2024-09-11T08:21:21.626003+0000","last_peered":"2024-09-11T08:21:21.626003+0000","last_clean":"2024-09-11T08:21:21.626003+0000","last_became_active":"2024-09-11T08:14:24.793048+0000","last_became_peered":"2024-09-11T08:14:24.793048+0000","last_unstale":"2024-09-11T08:21:21.626003+0000","last_undegraded":"2024-09-11T08:21:21.626003+0000","last_fullsized":"2024-09-11T08:21:21.626003+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'6","last_scrub_stamp":"2024-09-11T08:19:28.912944+0000","last_deep_scrub":"224'6","last_deep_scrub_stamp":"2024-09-11T08:16:55.778284+0000","last_clean_scrub_stamp":"2024-09-11T08:19:28.912944+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001538764,"objects_trimmed":0,"snaptrim_duration":0.000318821,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":424,"num_read_kb":419,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":51,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.654850+0000","last_change":"2024-09-11T08:20:54.655099+0000","last_active":"2024-09-11T08:21:12.654850+0000","last_peered":"2024-09-11T08:21:12.654850+0000","last_clean":"2024-09-11T08:21:12.654850+0000","last_became_active":"2024-09-11T08:19:29.154711+0000","last_became_peered":"2024-09-11T08:19:29.154711+0000","last_unstale":"2024-09-11T08:21:12.654850+0000","last_undegraded":"2024-09-11T08:21:12.654850+0000","last_fullsized":"2024-09-11T08:21:12.654850+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:54.655033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:20:54.655033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:58.452585+0000","scrub_duration":0.017642073000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":339,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.652752+0000","last_change":"2024-09-11T08:20:08.606909+0000","last_active":"2024-09-11T08:21:12.652752+0000","last_peered":"2024-09-11T08:21:12.652752+0000","last_clean":"2024-09-11T08:21:12.652752+0000","last_became_active":"2024-09-11T08:19:29.141234+0000","last_became_peered":"2024-09-11T08:19:29.141234+0000","last_unstale":"2024-09-11T08:21:12.652752+0000","last_undegraded":"2024-09-11T08:21:12.652752+0000","last_fullsized":"2024-09-11T08:21:12.652752+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:08.606839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:20:08.606839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:37.940143+0000","scrub_duration":0.0020913960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"302'14676","reported_seq":16720,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-11T08:21:22.345772+0000","last_change":"2024-09-11T08:21:03.411596+0000","last_active":"2024-09-11T08:21:22.345772+0000","last_peered":"2024-09-11T08:21:22.345772+0000","last_clean":"2024-09-11T08:21:22.345772+0000","last_became_active":"2024-09-11T08:19:39.774204+0000","last_became_peered":"2024-09-11T08:19:39.774204+0000","last_unstale":"2024-09-11T08:21:22.345772+0000","last_undegraded":"2024-09-11T08:21:22.345772+0000","last_fullsized":"2024-09-11T08:21:22.345772+0000","mapping_epoch":274,"log_start":"282'8412","ondisk_log_start":"282'8412","created":246,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"297'13236","last_scrub_stamp":"2024-09-11T08:21:03.411511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:21:03.411511+0000","objects_scrubbed":1460,"log_size":6264,"log_dups_size":0,"ondisk_log_size":6264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:08.063302+0000","scrub_duration":0.67056985499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14676,"num_write_kb":469632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":940,"num_bytes_recovered":61603840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"239'33","reported_seq":383,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.651865+0000","last_change":"2024-09-11T08:20:16.770435+0000","last_active":"2024-09-11T08:21:12.651865+0000","last_peered":"2024-09-11T08:21:12.651865+0000","last_clean":"2024-09-11T08:21:12.651865+0000","last_became_active":"2024-09-11T08:14:24.789154+0000","last_became_peered":"2024-09-11T08:14:24.789154+0000","last_unstale":"2024-09-11T08:21:12.651865+0000","last_undegraded":"2024-09-11T08:21:12.651865+0000","last_fullsized":"2024-09-11T08:21:12.651865+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'33","last_scrub_stamp":"2024-09-11T08:20:16.770344+0000","last_deep_scrub":"84'21","last_deep_scrub_stamp":"2024-09-11T08:08:56.705306+0000","last_clean_scrub_stamp":"2024-09-11T08:20:16.770344+0000","objects_scrubbed":7,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:45.920805+0000","scrub_duration":0.037009359999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":41,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.654790+0000","last_change":"2024-09-11T08:19:29.144089+0000","last_active":"2024-09-11T08:21:12.654790+0000","last_peered":"2024-09-11T08:21:12.654790+0000","last_clean":"2024-09-11T08:21:12.654790+0000","last_became_active":"2024-09-11T08:19:29.143505+0000","last_became_peered":"2024-09-11T08:19:29.143505+0000","last_unstale":"2024-09-11T08:21:12.654790+0000","last_undegraded":"2024-09-11T08:21:12.654790+0000","last_fullsized":"2024-09-11T08:21:12.654790+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:45.423258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:06.236119+0000","last_clean_scrub_stamp":"2024-09-11T08:18:45.423258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":130,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.644532+0000","last_change":"2024-09-11T08:18:41.048985+0000","last_active":"2024-09-11T08:21:12.644532+0000","last_peered":"2024-09-11T08:21:12.644532+0000","last_clean":"2024-09-11T08:21:12.644532+0000","last_became_active":"2024-09-11T08:15:56.523430+0000","last_became_peered":"2024-09-11T08:15:56.523430+0000","last_unstale":"2024-09-11T08:21:12.644532+0000","last_undegraded":"2024-09-11T08:21:12.644532+0000","last_fullsized":"2024-09-11T08:21:12.644532+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-11T08:18:41.048903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:05:21.231856+0000","last_clean_scrub_stamp":"2024-09-11T08:18:41.048903+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003255703,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":144,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.647272+0000","last_change":"2024-09-11T08:20:25.775943+0000","last_active":"2024-09-11T08:21:12.647272+0000","last_peered":"2024-09-11T08:21:12.647272+0000","last_clean":"2024-09-11T08:21:12.647272+0000","last_became_active":"2024-09-11T08:19:29.147576+0000","last_became_peered":"2024-09-11T08:19:29.147576+0000","last_unstale":"2024-09-11T08:21:12.647272+0000","last_undegraded":"2024-09-11T08:21:12.647272+0000","last_fullsized":"2024-09-11T08:21:12.647272+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:25.775869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:20:25.775869+0000","last_clean_scrub_stamp":"2024-09-11T08:20:25.775869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:39.906509+0000","scrub_duration":0.004676014,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":51,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.655122+0000","last_change":"2024-09-11T08:20:29.154000+0000","last_active":"2024-09-11T08:21:12.655122+0000","last_peered":"2024-09-11T08:21:12.655122+0000","last_clean":"2024-09-11T08:21:12.655122+0000","last_became_active":"2024-09-11T08:19:29.152071+0000","last_became_peered":"2024-09-11T08:19:29.152071+0000","last_unstale":"2024-09-11T08:21:12.655122+0000","last_undegraded":"2024-09-11T08:21:12.655122+0000","last_fullsized":"2024-09-11T08:21:12.655122+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:29.153871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:01.777429+0000","last_clean_scrub_stamp":"2024-09-11T08:20:29.153871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:31.775315+0000","scrub_duration":0.0015196540000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"302'17053","reported_seq":17321,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-11T08:21:22.335399+0000","last_change":"2024-09-11T08:20:51.170996+0000","last_active":"2024-09-11T08:21:22.335399+0000","last_peered":"2024-09-11T08:21:22.335399+0000","last_clean":"2024-09-11T08:21:22.335399+0000","last_became_active":"2024-09-11T08:18:00.826126+0000","last_became_peered":"2024-09-11T08:18:00.826126+0000","last_unstale":"2024-09-11T08:21:22.335399+0000","last_undegraded":"2024-09-11T08:21:22.335399+0000","last_fullsized":"2024-09-11T08:21:22.335399+0000","mapping_epoch":246,"log_start":"290'12134","ondisk_log_start":"290'12134","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'14731","last_scrub_stamp":"2024-09-11T08:20:51.170867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:20:51.170867+0000","objects_scrubbed":2353,"log_size":4919,"log_dups_size":0,"ondisk_log_size":4919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:14.132698+0000","scrub_duration":1.062925823,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14704,"num_write_kb":470528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"239'24","reported_seq":358,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.640356+0000","last_change":"2024-09-11T08:21:09.811217+0000","last_active":"2024-09-11T08:21:12.640356+0000","last_peered":"2024-09-11T08:21:12.640356+0000","last_clean":"2024-09-11T08:21:12.640356+0000","last_became_active":"2024-09-11T08:21:09.782223+0000","last_became_peered":"2024-09-11T08:21:09.782223+0000","last_unstale":"2024-09-11T08:21:12.640356+0000","last_undegraded":"2024-09-11T08:21:12.640356+0000","last_fullsized":"2024-09-11T08:21:12.640356+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'24","last_scrub_stamp":"2024-09-11T08:18:40.895293+0000","last_deep_scrub":"88'16","last_deep_scrub_stamp":"2024-09-11T08:09:39.635984+0000","last_clean_scrub_stamp":"2024-09-11T08:18:40.895293+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010336458999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":319,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.643751+0000","last_change":"2024-09-11T08:21:03.653241+0000","last_active":"2024-09-11T08:21:12.643751+0000","last_peered":"2024-09-11T08:21:12.643751+0000","last_clean":"2024-09-11T08:21:12.643751+0000","last_became_active":"2024-09-11T08:14:24.789888+0000","last_became_peered":"2024-09-11T08:14:24.789888+0000","last_unstale":"2024-09-11T08:21:12.643751+0000","last_undegraded":"2024-09-11T08:21:12.643751+0000","last_fullsized":"2024-09-11T08:21:12.643751+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:03.653157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:36.417911+0000","last_clean_scrub_stamp":"2024-09-11T08:21:03.653157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:18.943216+0000","scrub_duration":0.0039236879999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":41,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.654916+0000","last_change":"2024-09-11T08:19:29.154565+0000","last_active":"2024-09-11T08:21:12.654916+0000","last_peered":"2024-09-11T08:21:12.654916+0000","last_clean":"2024-09-11T08:21:12.654916+0000","last_became_active":"2024-09-11T08:19:29.153976+0000","last_became_peered":"2024-09-11T08:19:29.153976+0000","last_unstale":"2024-09-11T08:21:12.654916+0000","last_undegraded":"2024-09-11T08:21:12.654916+0000","last_fullsized":"2024-09-11T08:21:12.654916+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:30.247009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:01.558340+0000","last_clean_scrub_stamp":"2024-09-11T08:18:30.247009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":119,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.651648+0000","last_change":"2024-09-11T08:20:11.704845+0000","last_active":"2024-09-11T08:21:12.651648+0000","last_peered":"2024-09-11T08:21:12.651648+0000","last_clean":"2024-09-11T08:21:12.651648+0000","last_became_active":"2024-09-11T08:19:29.144592+0000","last_became_peered":"2024-09-11T08:19:29.144592+0000","last_unstale":"2024-09-11T08:21:12.651648+0000","last_undegraded":"2024-09-11T08:21:12.651648+0000","last_fullsized":"2024-09-11T08:21:12.651648+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:11.704755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:20:11.704755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:19.157575+0000","scrub_duration":0.0078689060000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.0","version":"302'14440","reported_seq":14592,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-11T08:21:22.292655+0000","last_change":"2024-09-11T08:20:55.368446+0000","last_active":"2024-09-11T08:21:22.292655+0000","last_peered":"2024-09-11T08:21:22.292655+0000","last_clean":"2024-09-11T08:21:22.292655+0000","last_became_active":"2024-09-11T08:18:01.133673+0000","last_became_peered":"2024-09-11T08:18:01.133673+0000","last_unstale":"2024-09-11T08:21:22.292655+0000","last_undegraded":"2024-09-11T08:21:22.292655+0000","last_fullsized":"2024-09-11T08:21:22.292655+0000","mapping_epoch":246,"log_start":"283'9602","ondisk_log_start":"283'9602","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'12508","last_scrub_stamp":"2024-09-11T08:20:55.368362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:20:55.368362+0000","objects_scrubbed":1955,"log_size":4838,"log_dups_size":0,"ondisk_log_size":4838,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:01.578532+0000","scrub_duration":0.68437722499999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14440,"num_write_kb":462080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"239'39","reported_seq":413,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.641448+0000","last_change":"2024-09-11T08:19:09.625136+0000","last_active":"2024-09-11T08:21:12.641448+0000","last_peered":"2024-09-11T08:21:12.641448+0000","last_clean":"2024-09-11T08:21:12.641448+0000","last_became_active":"2024-09-11T08:19:09.604525+0000","last_became_peered":"2024-09-11T08:19:09.604525+0000","last_unstale":"2024-09-11T08:21:12.641448+0000","last_undegraded":"2024-09-11T08:21:12.641448+0000","last_fullsized":"2024-09-11T08:21:12.641448+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'39","last_scrub_stamp":"2024-09-11T08:18:31.141746+0000","last_deep_scrub":"77'25","last_deep_scrub_stamp":"2024-09-11T08:08:32.458210+0000","last_clean_scrub_stamp":"2024-09-11T08:18:31.141746+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0076700900000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":327,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.653144+0000","last_change":"2024-09-11T08:20:17.724434+0000","last_active":"2024-09-11T08:21:12.653144+0000","last_peered":"2024-09-11T08:21:12.653144+0000","last_clean":"2024-09-11T08:21:12.653144+0000","last_became_active":"2024-09-11T08:07:30.042555+0000","last_became_peered":"2024-09-11T08:07:30.042555+0000","last_unstale":"2024-09-11T08:21:12.653144+0000","last_undegraded":"2024-09-11T08:21:12.653144+0000","last_fullsized":"2024-09-11T08:21:12.653144+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:17.724372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:20:17.724372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:39.478392+0000","scrub_duration":0.003123678,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":318,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.652742+0000","last_change":"2024-09-11T08:21:09.706729+0000","last_active":"2024-09-11T08:21:12.652742+0000","last_peered":"2024-09-11T08:21:12.652742+0000","last_clean":"2024-09-11T08:21:12.652742+0000","last_became_active":"2024-09-11T08:07:34.054961+0000","last_became_peered":"2024-09-11T08:07:34.054961+0000","last_unstale":"2024-09-11T08:21:12.652742+0000","last_undegraded":"2024-09-11T08:21:12.652742+0000","last_fullsized":"2024-09-11T08:21:12.652742+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:09.706678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:21:09.706678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:25.130807+0000","scrub_duration":0.0038244720000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":335,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.657625+0000","last_change":"2024-09-11T08:21:02.823083+0000","last_active":"2024-09-11T08:21:12.657625+0000","last_peered":"2024-09-11T08:21:12.657625+0000","last_clean":"2024-09-11T08:21:12.657625+0000","last_became_active":"2024-09-11T08:07:36.071194+0000","last_became_peered":"2024-09-11T08:07:36.071194+0000","last_unstale":"2024-09-11T08:21:12.657625+0000","last_undegraded":"2024-09-11T08:21:12.657625+0000","last_fullsized":"2024-09-11T08:21:12.657625+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:02.822996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:07.930137+0000","last_clean_scrub_stamp":"2024-09-11T08:21:02.822996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:13.236302+0000","scrub_duration":0.005271441,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"302'14274","reported_seq":14532,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-11T08:21:22.305029+0000","last_change":"2024-09-11T08:20:17.488165+0000","last_active":"2024-09-11T08:21:22.305029+0000","last_peered":"2024-09-11T08:21:22.305029+0000","last_clean":"2024-09-11T08:21:22.305029+0000","last_became_active":"2024-09-11T08:18:00.828320+0000","last_became_peered":"2024-09-11T08:18:00.828320+0000","last_unstale":"2024-09-11T08:21:22.305029+0000","last_undegraded":"2024-09-11T08:21:22.305029+0000","last_fullsized":"2024-09-11T08:21:22.305029+0000","mapping_epoch":246,"log_start":"280'7658","ondisk_log_start":"280'7658","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"285'9834","last_scrub_stamp":"2024-09-11T08:20:17.488079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:20:17.488079+0000","objects_scrubbed":4527,"log_size":6616,"log_dups_size":0,"ondisk_log_size":6616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:26.015958+0000","scrub_duration":1.748475096,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14274,"num_write_kb":456768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"239'25","reported_seq":338,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.643058+0000","last_change":"2024-09-11T08:20:16.527717+0000","last_active":"2024-09-11T08:21:12.643058+0000","last_peered":"2024-09-11T08:21:12.643058+0000","last_clean":"2024-09-11T08:21:12.643058+0000","last_became_active":"2024-09-11T08:07:32.043314+0000","last_became_peered":"2024-09-11T08:07:32.043314+0000","last_unstale":"2024-09-11T08:21:12.643058+0000","last_undegraded":"2024-09-11T08:21:12.643058+0000","last_fullsized":"2024-09-11T08:21:12.643058+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'25","last_scrub_stamp":"2024-09-11T08:20:16.527630+0000","last_deep_scrub":"224'15","last_deep_scrub_stamp":"2024-09-11T08:17:01.126478+0000","last_clean_scrub_stamp":"2024-09-11T08:20:16.527630+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:37.247145+0000","scrub_duration":0.0059554439999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":96,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.653745+0000","last_change":"2024-09-11T08:20:10.609260+0000","last_active":"2024-09-11T08:21:12.653745+0000","last_peered":"2024-09-11T08:21:12.653745+0000","last_clean":"2024-09-11T08:21:12.653745+0000","last_became_active":"2024-09-11T08:17:37.557278+0000","last_became_peered":"2024-09-11T08:17:37.557278+0000","last_unstale":"2024-09-11T08:21:12.653745+0000","last_undegraded":"2024-09-11T08:21:12.653745+0000","last_fullsized":"2024-09-11T08:21:12.653745+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:10.609170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:45.979847+0000","last_clean_scrub_stamp":"2024-09-11T08:20:10.609170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:31.049454+0000","scrub_duration":0.017294936,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"271'14","reported_seq":543,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-11T08:21:17.267522+0000","last_change":"2024-09-11T08:20:54.836617+0000","last_active":"2024-09-11T08:21:17.267522+0000","last_peered":"2024-09-11T08:21:17.267522+0000","last_clean":"2024-09-11T08:21:17.267522+0000","last_became_active":"2024-09-11T08:19:29.153997+0000","last_became_peered":"2024-09-11T08:19:29.153997+0000","last_unstale":"2024-09-11T08:21:17.267522+0000","last_undegraded":"2024-09-11T08:21:17.267522+0000","last_fullsized":"2024-09-11T08:21:17.267522+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'14","last_scrub_stamp":"2024-09-11T08:20:54.836531+0000","last_deep_scrub":"171'12","last_deep_scrub_stamp":"2024-09-11T08:14:56.881299+0000","last_clean_scrub_stamp":"2024-09-11T08:20:54.836531+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:56.071165+0000","scrub_duration":0.0063520010000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":296,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.640137+0000","last_change":"2024-09-11T08:20:50.427308+0000","last_active":"2024-09-11T08:21:12.640137+0000","last_peered":"2024-09-11T08:21:12.640137+0000","last_clean":"2024-09-11T08:21:12.640137+0000","last_became_active":"2024-09-11T08:07:36.074684+0000","last_became_peered":"2024-09-11T08:07:36.074684+0000","last_unstale":"2024-09-11T08:21:12.640137+0000","last_undegraded":"2024-09-11T08:21:12.640137+0000","last_fullsized":"2024-09-11T08:21:12.640137+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:50.427224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:14.613532+0000","last_clean_scrub_stamp":"2024-09-11T08:20:50.427224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:59.232555+0000","scrub_duration":0.0028770409999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"302'14522","reported_seq":14779,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-11T08:21:22.301614+0000","last_change":"2024-09-11T08:20:23.741617+0000","last_active":"2024-09-11T08:21:22.301614+0000","last_peered":"2024-09-11T08:21:22.301614+0000","last_clean":"2024-09-11T08:21:22.301614+0000","last_became_active":"2024-09-11T08:19:30.134196+0000","last_became_peered":"2024-09-11T08:19:30.134196+0000","last_unstale":"2024-09-11T08:21:22.301614+0000","last_undegraded":"2024-09-11T08:21:22.301614+0000","last_fullsized":"2024-09-11T08:21:22.301614+0000","mapping_epoch":271,"log_start":"283'9240","ondisk_log_start":"283'9240","created":246,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"287'10492","last_scrub_stamp":"2024-09-11T08:20:23.741530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:20:23.741530+0000","objects_scrubbed":4087,"log_size":5282,"log_dups_size":0,"ondisk_log_size":5282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:43.872652+0000","scrub_duration":1.814425969,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14522,"num_write_kb":464704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"239'15","reported_seq":334,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.653508+0000","last_change":"2024-09-11T08:21:10.722246+0000","last_active":"2024-09-11T08:21:12.653508+0000","last_peered":"2024-09-11T08:21:12.653508+0000","last_clean":"2024-09-11T08:21:12.653508+0000","last_became_active":"2024-09-11T08:07:32.054885+0000","last_became_peered":"2024-09-11T08:07:32.054885+0000","last_unstale":"2024-09-11T08:21:12.653508+0000","last_undegraded":"2024-09-11T08:21:12.653508+0000","last_fullsized":"2024-09-11T08:21:12.653508+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'15","last_scrub_stamp":"2024-09-11T08:21:10.722062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:21:10.722062+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:23.225006+0000","scrub_duration":0.010706927,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":51,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.654456+0000","last_change":"2024-09-11T08:21:00.730548+0000","last_active":"2024-09-11T08:21:12.654456+0000","last_peered":"2024-09-11T08:21:12.654456+0000","last_clean":"2024-09-11T08:21:12.654456+0000","last_became_active":"2024-09-11T08:19:29.143174+0000","last_became_peered":"2024-09-11T08:19:29.143174+0000","last_unstale":"2024-09-11T08:21:12.654456+0000","last_undegraded":"2024-09-11T08:21:12.654456+0000","last_fullsized":"2024-09-11T08:21:12.654456+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:00.730460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:21:00.730460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:17.761419+0000","scrub_duration":0.0102811,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":337,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.659567+0000","last_change":"2024-09-11T08:21:04.894391+0000","last_active":"2024-09-11T08:21:12.659567+0000","last_peered":"2024-09-11T08:21:12.659567+0000","last_clean":"2024-09-11T08:21:12.659567+0000","last_became_active":"2024-09-11T08:07:34.056606+0000","last_became_peered":"2024-09-11T08:07:34.056606+0000","last_unstale":"2024-09-11T08:21:12.659567+0000","last_undegraded":"2024-09-11T08:21:12.659567+0000","last_fullsized":"2024-09-11T08:21:12.659567+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:04.894259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:21:04.894259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:27.176768+0000","scrub_duration":0.013493084000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":322,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.652118+0000","last_change":"2024-09-11T08:18:32.278327+0000","last_active":"2024-09-11T08:21:12.652118+0000","last_peered":"2024-09-11T08:21:12.652118+0000","last_clean":"2024-09-11T08:21:12.652118+0000","last_became_active":"2024-09-11T08:15:56.518884+0000","last_became_peered":"2024-09-11T08:15:56.518884+0000","last_unstale":"2024-09-11T08:21:12.652118+0000","last_undegraded":"2024-09-11T08:21:12.652118+0000","last_fullsized":"2024-09-11T08:21:12.652118+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:32.278257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:18:32.278257+0000","last_clean_scrub_stamp":"2024-09-11T08:18:32.278257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.021912361000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"302'14440","reported_seq":20275,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-11T08:21:22.309731+0000","last_change":"2024-09-11T08:20:23.147847+0000","last_active":"2024-09-11T08:21:22.309731+0000","last_peered":"2024-09-11T08:21:22.309731+0000","last_clean":"2024-09-11T08:21:22.309731+0000","last_became_active":"2024-09-11T08:19:30.138233+0000","last_became_peered":"2024-09-11T08:19:30.138233+0000","last_unstale":"2024-09-11T08:21:22.309731+0000","last_undegraded":"2024-09-11T08:21:22.309731+0000","last_fullsized":"2024-09-11T08:21:22.309731+0000","mapping_epoch":271,"log_start":"283'9165","ondisk_log_start":"283'9165","created":246,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"287'10327","last_scrub_stamp":"2024-09-11T08:20:23.147735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:20:23.147735+0000","objects_scrubbed":4205,"log_size":5275,"log_dups_size":0,"ondisk_log_size":5275,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:51.571739+0000","scrub_duration":2.2078867029999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14440,"num_write_kb":462080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2787,"num_bytes_recovered":182648832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"239'35","reported_seq":451,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.651415+0000","last_change":"2024-09-11T08:20:23.930556+0000","last_active":"2024-09-11T08:21:12.651415+0000","last_peered":"2024-09-11T08:21:12.651415+0000","last_clean":"2024-09-11T08:21:12.651415+0000","last_became_active":"2024-09-11T08:15:56.526961+0000","last_became_peered":"2024-09-11T08:15:56.526961+0000","last_unstale":"2024-09-11T08:21:12.651415+0000","last_undegraded":"2024-09-11T08:21:12.651415+0000","last_fullsized":"2024-09-11T08:21:12.651415+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'35","last_scrub_stamp":"2024-09-11T08:20:23.930464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:20:23.930464+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:24.598234+0000","scrub_duration":0.003826734,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":144,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.648711+0000","last_change":"2024-09-11T08:20:09.859624+0000","last_active":"2024-09-11T08:21:12.648711+0000","last_peered":"2024-09-11T08:21:12.648711+0000","last_clean":"2024-09-11T08:21:12.648711+0000","last_became_active":"2024-09-11T08:19:29.152114+0000","last_became_peered":"2024-09-11T08:19:29.152114+0000","last_unstale":"2024-09-11T08:21:12.648711+0000","last_undegraded":"2024-09-11T08:21:12.648711+0000","last_fullsized":"2024-09-11T08:21:12.648711+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:09.859549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:44.385372+0000","last_clean_scrub_stamp":"2024-09-11T08:20:09.859549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:32.516863+0000","scrub_duration":0.0055679850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":350,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.657283+0000","last_change":"2024-09-11T08:20:53.860652+0000","last_active":"2024-09-11T08:21:12.657283+0000","last_peered":"2024-09-11T08:21:12.657283+0000","last_clean":"2024-09-11T08:21:12.657283+0000","last_became_active":"2024-09-11T08:19:29.178705+0000","last_became_peered":"2024-09-11T08:19:29.178705+0000","last_unstale":"2024-09-11T08:21:12.657283+0000","last_undegraded":"2024-09-11T08:21:12.657283+0000","last_fullsized":"2024-09-11T08:21:12.657283+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:53.860586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:20:53.860586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:19.805109+0000","scrub_duration":0.030572999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":124,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.644608+0000","last_change":"2024-09-11T08:20:21.780282+0000","last_active":"2024-09-11T08:21:12.644608+0000","last_peered":"2024-09-11T08:21:12.644608+0000","last_clean":"2024-09-11T08:21:12.644608+0000","last_became_active":"2024-09-11T08:15:56.518169+0000","last_became_peered":"2024-09-11T08:15:56.518169+0000","last_unstale":"2024-09-11T08:21:12.644608+0000","last_undegraded":"2024-09-11T08:21:12.644608+0000","last_fullsized":"2024-09-11T08:21:12.644608+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:21.780224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:20:21.780224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:39.541601+0000","scrub_duration":0.0071716779999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"302'14412","reported_seq":14643,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-11T08:21:22.318543+0000","last_change":"2024-09-11T08:20:27.162668+0000","last_active":"2024-09-11T08:21:22.318543+0000","last_peered":"2024-09-11T08:21:22.318543+0000","last_clean":"2024-09-11T08:21:22.318543+0000","last_became_active":"2024-09-11T08:18:00.827140+0000","last_became_peered":"2024-09-11T08:18:00.827140+0000","last_unstale":"2024-09-11T08:21:22.318543+0000","last_undegraded":"2024-09-11T08:21:22.318543+0000","last_fullsized":"2024-09-11T08:21:22.318543+0000","mapping_epoch":246,"log_start":"283'9589","ondisk_log_start":"283'9589","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'10610","last_scrub_stamp":"2024-09-11T08:20:27.162586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:20:27.162586+0000","objects_scrubbed":3852,"log_size":4823,"log_dups_size":0,"ondisk_log_size":4823,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:55.304374+0000","scrub_duration":1.512301039,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14412,"num_write_kb":461184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"239'23","reported_seq":337,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.642051+0000","last_change":"2024-09-11T08:18:35.057979+0000","last_active":"2024-09-11T08:21:12.642051+0000","last_peered":"2024-09-11T08:21:12.642051+0000","last_clean":"2024-09-11T08:21:12.642051+0000","last_became_active":"2024-09-11T08:07:32.048164+0000","last_became_peered":"2024-09-11T08:07:32.048164+0000","last_unstale":"2024-09-11T08:21:12.642051+0000","last_undegraded":"2024-09-11T08:21:12.642051+0000","last_fullsized":"2024-09-11T08:21:12.642051+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'23","last_scrub_stamp":"2024-09-11T08:18:35.057874+0000","last_deep_scrub":"145'17","last_deep_scrub_stamp":"2024-09-11T08:13:08.160848+0000","last_clean_scrub_stamp":"2024-09-11T08:18:35.057874+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043775849999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":134,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.645197+0000","last_change":"2024-09-11T08:21:06.262697+0000","last_active":"2024-09-11T08:21:12.645197+0000","last_peered":"2024-09-11T08:21:12.645197+0000","last_clean":"2024-09-11T08:21:12.645197+0000","last_became_active":"2024-09-11T08:15:56.525845+0000","last_became_peered":"2024-09-11T08:15:56.525845+0000","last_unstale":"2024-09-11T08:21:12.645197+0000","last_undegraded":"2024-09-11T08:21:12.645197+0000","last_fullsized":"2024-09-11T08:21:12.645197+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:06.262572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:21:06.262572+0000","last_clean_scrub_stamp":"2024-09-11T08:21:06.262572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:08.699491+0000","scrub_duration":0.007163272,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":317,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.656061+0000","last_change":"2024-09-11T08:20:59.876602+0000","last_active":"2024-09-11T08:21:12.656061+0000","last_peered":"2024-09-11T08:21:12.656061+0000","last_clean":"2024-09-11T08:21:12.656061+0000","last_became_active":"2024-09-11T08:07:34.057479+0000","last_became_peered":"2024-09-11T08:07:34.057479+0000","last_unstale":"2024-09-11T08:21:12.656061+0000","last_undegraded":"2024-09-11T08:21:12.656061+0000","last_fullsized":"2024-09-11T08:21:12.656061+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:59.876505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:20:59.876505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:24.461114+0000","scrub_duration":0.0064578550000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":333,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.646991+0000","last_change":"2024-09-11T08:20:21.834803+0000","last_active":"2024-09-11T08:21:12.646991+0000","last_peered":"2024-09-11T08:21:12.646991+0000","last_clean":"2024-09-11T08:21:12.646991+0000","last_became_active":"2024-09-11T08:15:56.522624+0000","last_became_peered":"2024-09-11T08:15:56.522624+0000","last_unstale":"2024-09-11T08:21:12.646991+0000","last_undegraded":"2024-09-11T08:21:12.646991+0000","last_fullsized":"2024-09-11T08:21:12.646991+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:21.834736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:20:21.834736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:22.373155+0000","scrub_duration":0.0012504840000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"302'14654","reported_seq":16090,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-11T08:21:22.277004+0000","last_change":"2024-09-11T08:21:00.497322+0000","last_active":"2024-09-11T08:21:22.277004+0000","last_peered":"2024-09-11T08:21:22.277004+0000","last_clean":"2024-09-11T08:21:22.277004+0000","last_became_active":"2024-09-11T08:19:39.780479+0000","last_became_peered":"2024-09-11T08:19:39.780479+0000","last_unstale":"2024-09-11T08:21:22.277004+0000","last_undegraded":"2024-09-11T08:21:22.277004+0000","last_fullsized":"2024-09-11T08:21:22.277004+0000","mapping_epoch":274,"log_start":"282'8398","ondisk_log_start":"282'8398","created":246,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"297'13040","last_scrub_stamp":"2024-09-11T08:21:00.497229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:59.811675+0000","last_clean_scrub_stamp":"2024-09-11T08:21:00.497229+0000","objects_scrubbed":1638,"log_size":6256,"log_dups_size":0,"ondisk_log_size":6256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:11.290954+0000","scrub_duration":0.78454671399999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14654,"num_write_kb":468928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":633,"num_bytes_recovered":41484288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"239'27","reported_seq":67,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.654080+0000","last_change":"2024-09-11T08:20:08.785799+0000","last_active":"2024-09-11T08:21:12.654080+0000","last_peered":"2024-09-11T08:21:12.654080+0000","last_clean":"2024-09-11T08:21:12.654080+0000","last_became_active":"2024-09-11T08:19:29.147000+0000","last_became_peered":"2024-09-11T08:19:29.147000+0000","last_unstale":"2024-09-11T08:21:12.654080+0000","last_undegraded":"2024-09-11T08:21:12.654080+0000","last_fullsized":"2024-09-11T08:21:12.654080+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'27","last_scrub_stamp":"2024-09-11T08:20:08.785703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:20:08.785703+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:18.100187+0000","scrub_duration":0.005828789,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":130,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.645462+0000","last_change":"2024-09-11T08:18:39.012762+0000","last_active":"2024-09-11T08:21:12.645462+0000","last_peered":"2024-09-11T08:21:12.645462+0000","last_clean":"2024-09-11T08:21:12.645462+0000","last_became_active":"2024-09-11T08:15:56.515328+0000","last_became_peered":"2024-09-11T08:15:56.515328+0000","last_unstale":"2024-09-11T08:21:12.645462+0000","last_undegraded":"2024-09-11T08:21:12.645462+0000","last_fullsized":"2024-09-11T08:21:12.645462+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-11T08:18:39.012616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:18:39.012616+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025254959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"209'11","reported_seq":193,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-11T08:21:17.245705+0000","last_change":"2024-09-11T08:20:51.158341+0000","last_active":"2024-09-11T08:21:17.245705+0000","last_peered":"2024-09-11T08:21:17.245705+0000","last_clean":"2024-09-11T08:21:17.245705+0000","last_became_active":"2024-09-11T08:15:56.524513+0000","last_became_peered":"2024-09-11T08:15:56.524513+0000","last_unstale":"2024-09-11T08:21:17.245705+0000","last_undegraded":"2024-09-11T08:21:17.245705+0000","last_fullsized":"2024-09-11T08:21:17.245705+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"209'11","last_scrub_stamp":"2024-09-11T08:20:51.158199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:20:51.158199+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:17.301119+0000","scrub_duration":0.015653383,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":41,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.654928+0000","last_change":"2024-09-11T08:19:29.164149+0000","last_active":"2024-09-11T08:21:12.654928+0000","last_peered":"2024-09-11T08:21:12.654928+0000","last_clean":"2024-09-11T08:21:12.654928+0000","last_became_active":"2024-09-11T08:19:29.163872+0000","last_became_peered":"2024-09-11T08:19:29.163872+0000","last_unstale":"2024-09-11T08:21:12.654928+0000","last_undegraded":"2024-09-11T08:21:12.654928+0000","last_fullsized":"2024-09-11T08:21:12.654928+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:40.294136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:03.484755+0000","last_clean_scrub_stamp":"2024-09-11T08:18:40.294136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":351,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.651099+0000","last_change":"2024-09-11T08:20:24.921477+0000","last_active":"2024-09-11T08:21:12.651099+0000","last_peered":"2024-09-11T08:21:12.651099+0000","last_clean":"2024-09-11T08:21:12.651099+0000","last_became_active":"2024-09-11T08:19:29.193317+0000","last_became_peered":"2024-09-11T08:19:29.193317+0000","last_unstale":"2024-09-11T08:21:12.651099+0000","last_undegraded":"2024-09-11T08:21:12.651099+0000","last_fullsized":"2024-09-11T08:21:12.651099+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:24.921042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:20:24.921042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:41.162286+0000","scrub_duration":0.0069516710000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"239'15","reported_seq":374,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.646841+0000","last_change":"2024-09-11T08:20:54.764391+0000","last_active":"2024-09-11T08:21:12.646841+0000","last_peered":"2024-09-11T08:21:12.646841+0000","last_clean":"2024-09-11T08:21:12.646841+0000","last_became_active":"2024-09-11T08:15:56.526325+0000","last_became_peered":"2024-09-11T08:15:56.526325+0000","last_unstale":"2024-09-11T08:21:12.646841+0000","last_undegraded":"2024-09-11T08:21:12.646841+0000","last_fullsized":"2024-09-11T08:21:12.646841+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'15","last_scrub_stamp":"2024-09-11T08:20:54.764293+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-09-11T08:09:03.418457+0000","last_clean_scrub_stamp":"2024-09-11T08:20:54.764293+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:05.029319+0000","scrub_duration":0.0039192439999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":41,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.654405+0000","last_change":"2024-09-11T08:19:29.147871+0000","last_active":"2024-09-11T08:21:12.654405+0000","last_peered":"2024-09-11T08:21:12.654405+0000","last_clean":"2024-09-11T08:21:12.654405+0000","last_became_active":"2024-09-11T08:19:29.143925+0000","last_became_peered":"2024-09-11T08:19:29.143925+0000","last_unstale":"2024-09-11T08:21:12.654405+0000","last_undegraded":"2024-09-11T08:21:12.654405+0000","last_fullsized":"2024-09-11T08:21:12.654405+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:32.284664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:53.548967+0000","last_clean_scrub_stamp":"2024-09-11T08:18:32.284664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":259,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.651227+0000","last_change":"2024-09-11T08:21:03.199721+0000","last_active":"2024-09-11T08:21:12.651227+0000","last_peered":"2024-09-11T08:21:12.651227+0000","last_clean":"2024-09-11T08:21:12.651227+0000","last_became_active":"2024-09-11T08:14:24.790678+0000","last_became_peered":"2024-09-11T08:14:24.790678+0000","last_unstale":"2024-09-11T08:21:12.651227+0000","last_undegraded":"2024-09-11T08:21:12.651227+0000","last_fullsized":"2024-09-11T08:21:12.651227+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:03.199669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:21:03.199669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:27.957305+0000","scrub_duration":0.0011985959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"239'19","reported_seq":375,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.652825+0000","last_change":"2024-09-11T08:20:14.603209+0000","last_active":"2024-09-11T08:21:12.652825+0000","last_peered":"2024-09-11T08:21:12.652825+0000","last_clean":"2024-09-11T08:21:12.652825+0000","last_became_active":"2024-09-11T08:15:56.514793+0000","last_became_peered":"2024-09-11T08:15:56.514793+0000","last_unstale":"2024-09-11T08:21:12.652825+0000","last_undegraded":"2024-09-11T08:21:12.652825+0000","last_fullsized":"2024-09-11T08:21:12.652825+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'19","last_scrub_stamp":"2024-09-11T08:20:14.603118+0000","last_deep_scrub":"84'13","last_deep_scrub_stamp":"2024-09-11T08:08:57.835634+0000","last_clean_scrub_stamp":"2024-09-11T08:20:14.603118+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:35.007041+0000","scrub_duration":0.0042720240000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":282,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.646095+0000","last_change":"2024-09-11T08:20:20.819257+0000","last_active":"2024-09-11T08:21:12.646095+0000","last_peered":"2024-09-11T08:21:12.646095+0000","last_clean":"2024-09-11T08:21:12.646095+0000","last_became_active":"2024-09-11T08:14:24.786021+0000","last_became_peered":"2024-09-11T08:14:24.786021+0000","last_unstale":"2024-09-11T08:21:12.646095+0000","last_undegraded":"2024-09-11T08:21:12.646095+0000","last_fullsized":"2024-09-11T08:21:12.646095+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:20.819154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:30.471620+0000","last_clean_scrub_stamp":"2024-09-11T08:20:20.819154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:39.435896+0000","scrub_duration":0.0018806490000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":335,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.646643+0000","last_change":"2024-09-11T08:20:22.889011+0000","last_active":"2024-09-11T08:21:12.646643+0000","last_peered":"2024-09-11T08:21:12.646643+0000","last_clean":"2024-09-11T08:21:12.646643+0000","last_became_active":"2024-09-11T08:15:56.525717+0000","last_became_peered":"2024-09-11T08:15:56.525717+0000","last_unstale":"2024-09-11T08:21:12.646643+0000","last_undegraded":"2024-09-11T08:21:12.646643+0000","last_fullsized":"2024-09-11T08:21:12.646643+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:22.888936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:04.370734+0000","last_clean_scrub_stamp":"2024-09-11T08:20:22.888936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:52.204024+0000","scrub_duration":0.01485916,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":335,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.656635+0000","last_change":"2024-09-11T08:20:15.554813+0000","last_active":"2024-09-11T08:21:12.656635+0000","last_peered":"2024-09-11T08:21:12.656635+0000","last_clean":"2024-09-11T08:21:12.656635+0000","last_became_active":"2024-09-11T08:15:56.525120+0000","last_became_peered":"2024-09-11T08:15:56.525120+0000","last_unstale":"2024-09-11T08:21:12.656635+0000","last_undegraded":"2024-09-11T08:21:12.656635+0000","last_fullsized":"2024-09-11T08:21:12.656635+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:15.554724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:54.184372+0000","last_clean_scrub_stamp":"2024-09-11T08:20:15.554724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033311510000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"239'15","reported_seq":51,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.656399+0000","last_change":"2024-09-11T08:19:29.168879+0000","last_active":"2024-09-11T08:21:12.656399+0000","last_peered":"2024-09-11T08:21:12.656399+0000","last_clean":"2024-09-11T08:21:12.656399+0000","last_became_active":"2024-09-11T08:19:29.153740+0000","last_became_peered":"2024-09-11T08:19:29.153740+0000","last_unstale":"2024-09-11T08:21:12.656399+0000","last_undegraded":"2024-09-11T08:21:12.656399+0000","last_fullsized":"2024-09-11T08:21:12.656399+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'15","last_scrub_stamp":"2024-09-11T08:18:44.371100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:18:44.371100+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047860990000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":338,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.652382+0000","last_change":"2024-09-11T08:21:05.777148+0000","last_active":"2024-09-11T08:21:12.652382+0000","last_peered":"2024-09-11T08:21:12.652382+0000","last_clean":"2024-09-11T08:21:12.652382+0000","last_became_active":"2024-09-11T08:15:56.513278+0000","last_became_peered":"2024-09-11T08:15:56.513278+0000","last_unstale":"2024-09-11T08:21:12.652382+0000","last_undegraded":"2024-09-11T08:21:12.652382+0000","last_fullsized":"2024-09-11T08:21:12.652382+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:05.777069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:52.661591+0000","last_clean_scrub_stamp":"2024-09-11T08:21:05.777069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:33.267208+0000","scrub_duration":0.00237024,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"239'13","reported_seq":490,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-11T08:21:17.246063+0000","last_change":"2024-09-11T08:20:14.871683+0000","last_active":"2024-09-11T08:21:17.246063+0000","last_peered":"2024-09-11T08:21:17.246063+0000","last_clean":"2024-09-11T08:21:17.246063+0000","last_became_active":"2024-09-11T08:17:37.558224+0000","last_became_peered":"2024-09-11T08:17:37.558224+0000","last_unstale":"2024-09-11T08:21:17.246063+0000","last_undegraded":"2024-09-11T08:21:17.246063+0000","last_fullsized":"2024-09-11T08:21:17.246063+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'13","last_scrub_stamp":"2024-09-11T08:20:14.871608+0000","last_deep_scrub":"222'12","last_deep_scrub_stamp":"2024-09-11T08:16:52.592856+0000","last_clean_scrub_stamp":"2024-09-11T08:20:14.871608+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:29.721925+0000","scrub_duration":0.0023994300000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":316,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.641660+0000","last_change":"2024-09-11T08:20:54.449873+0000","last_active":"2024-09-11T08:21:12.641660+0000","last_peered":"2024-09-11T08:21:12.641660+0000","last_clean":"2024-09-11T08:21:12.641660+0000","last_became_active":"2024-09-11T08:19:29.147682+0000","last_became_peered":"2024-09-11T08:19:29.147682+0000","last_unstale":"2024-09-11T08:21:12.641660+0000","last_undegraded":"2024-09-11T08:21:12.641660+0000","last_fullsized":"2024-09-11T08:21:12.641660+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:54.449786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:53.175994+0000","last_clean_scrub_stamp":"2024-09-11T08:20:54.449786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:58.294035+0000","scrub_duration":0.0015389500000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"239'20","reported_seq":174,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.656560+0000","last_change":"2024-09-11T08:20:13.568730+0000","last_active":"2024-09-11T08:21:12.656560+0000","last_peered":"2024-09-11T08:21:12.656560+0000","last_clean":"2024-09-11T08:21:12.656560+0000","last_became_active":"2024-09-11T08:17:37.554357+0000","last_became_peered":"2024-09-11T08:17:37.554357+0000","last_unstale":"2024-09-11T08:21:12.656560+0000","last_undegraded":"2024-09-11T08:21:12.656560+0000","last_fullsized":"2024-09-11T08:21:12.656560+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'20","last_scrub_stamp":"2024-09-11T08:20:13.568619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:20:13.568619+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:17.454653+0000","scrub_duration":0.0099769119999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":124,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.644729+0000","last_change":"2024-09-11T08:20:17.777970+0000","last_active":"2024-09-11T08:21:12.644729+0000","last_peered":"2024-09-11T08:21:12.644729+0000","last_clean":"2024-09-11T08:21:12.644729+0000","last_became_active":"2024-09-11T08:15:56.518515+0000","last_became_peered":"2024-09-11T08:15:56.518515+0000","last_unstale":"2024-09-11T08:21:12.644729+0000","last_undegraded":"2024-09-11T08:21:12.644729+0000","last_fullsized":"2024-09-11T08:21:12.644729+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:17.777890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:20:17.777890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:30.228005+0000","scrub_duration":0.0086416670000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"271'10","reported_seq":78,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-11T08:21:17.245379+0000","last_change":"2024-09-11T08:20:16.922523+0000","last_active":"2024-09-11T08:21:17.245379+0000","last_peered":"2024-09-11T08:21:17.245379+0000","last_clean":"2024-09-11T08:21:17.245379+0000","last_became_active":"2024-09-11T08:19:29.165183+0000","last_became_peered":"2024-09-11T08:19:29.165183+0000","last_unstale":"2024-09-11T08:21:17.245379+0000","last_undegraded":"2024-09-11T08:21:17.245379+0000","last_fullsized":"2024-09-11T08:21:17.245379+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'10","last_scrub_stamp":"2024-09-11T08:20:16.922384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:20:16.922384+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:24.103214+0000","scrub_duration":0.0084094979999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":322,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.652532+0000","last_change":"2024-09-11T08:20:23.600147+0000","last_active":"2024-09-11T08:21:12.652532+0000","last_peered":"2024-09-11T08:21:12.652532+0000","last_clean":"2024-09-11T08:21:12.652532+0000","last_became_active":"2024-09-11T08:15:56.513949+0000","last_became_peered":"2024-09-11T08:15:56.513949+0000","last_unstale":"2024-09-11T08:21:12.652532+0000","last_undegraded":"2024-09-11T08:21:12.652532+0000","last_fullsized":"2024-09-11T08:21:12.652532+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:23.600056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:20:23.600056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:37.601061+0000","scrub_duration":0.003973341,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"239'19","reported_seq":334,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.640740+0000","last_change":"2024-09-11T08:20:09.525188+0000","last_active":"2024-09-11T08:21:12.640740+0000","last_peered":"2024-09-11T08:21:12.640740+0000","last_clean":"2024-09-11T08:21:12.640740+0000","last_became_active":"2024-09-11T08:07:32.047981+0000","last_became_peered":"2024-09-11T08:07:32.047981+0000","last_unstale":"2024-09-11T08:21:12.640740+0000","last_undegraded":"2024-09-11T08:21:12.640740+0000","last_fullsized":"2024-09-11T08:21:12.640740+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'19","last_scrub_stamp":"2024-09-11T08:20:09.525095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:20:09.525095+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:38.535885+0000","scrub_duration":0.0063068040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":332,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.643324+0000","last_change":"2024-09-11T08:20:27.515065+0000","last_active":"2024-09-11T08:21:12.643324+0000","last_peered":"2024-09-11T08:21:12.643324+0000","last_clean":"2024-09-11T08:21:12.643324+0000","last_became_active":"2024-09-11T08:15:56.523574+0000","last_became_peered":"2024-09-11T08:15:56.523574+0000","last_unstale":"2024-09-11T08:21:12.643324+0000","last_undegraded":"2024-09-11T08:21:12.643324+0000","last_fullsized":"2024-09-11T08:21:12.643324+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:27.514983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:20:27.514983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:43.256061+0000","scrub_duration":0.019346374,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":331,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.652382+0000","last_change":"2024-09-11T08:19:29.142149+0000","last_active":"2024-09-11T08:21:12.652382+0000","last_peered":"2024-09-11T08:21:12.652382+0000","last_clean":"2024-09-11T08:21:12.652382+0000","last_became_active":"2024-09-11T08:19:29.141857+0000","last_became_peered":"2024-09-11T08:19:29.141857+0000","last_unstale":"2024-09-11T08:21:12.652382+0000","last_undegraded":"2024-09-11T08:21:12.652382+0000","last_fullsized":"2024-09-11T08:21:12.652382+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:41.689207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:18:41.689207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0076107090000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":116,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.651864+0000","last_change":"2024-09-11T08:20:11.653639+0000","last_active":"2024-09-11T08:21:12.651864+0000","last_peered":"2024-09-11T08:21:12.651864+0000","last_clean":"2024-09-11T08:21:12.651864+0000","last_became_active":"2024-09-11T08:19:29.180111+0000","last_became_peered":"2024-09-11T08:19:29.180111+0000","last_unstale":"2024-09-11T08:21:12.651864+0000","last_undegraded":"2024-09-11T08:21:12.651864+0000","last_fullsized":"2024-09-11T08:21:12.651864+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:11.653502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:20:11.653502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:18.440265+0000","scrub_duration":0.017208757000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"239'15","reported_seq":345,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.653021+0000","last_change":"2024-09-11T08:19:40.867691+0000","last_active":"2024-09-11T08:21:12.653021+0000","last_peered":"2024-09-11T08:21:12.653021+0000","last_clean":"2024-09-11T08:21:12.653021+0000","last_became_active":"2024-09-11T08:07:32.049368+0000","last_became_peered":"2024-09-11T08:07:32.049368+0000","last_unstale":"2024-09-11T08:21:12.653021+0000","last_undegraded":"2024-09-11T08:21:12.653021+0000","last_fullsized":"2024-09-11T08:21:12.653021+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'15","last_scrub_stamp":"2024-09-11T08:19:40.867599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:19:40.867599+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023960169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":124,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.648651+0000","last_change":"2024-09-11T08:18:28.949104+0000","last_active":"2024-09-11T08:21:12.648651+0000","last_peered":"2024-09-11T08:21:12.648651+0000","last_clean":"2024-09-11T08:21:12.648651+0000","last_became_active":"2024-09-11T08:15:56.521570+0000","last_became_peered":"2024-09-11T08:15:56.521570+0000","last_unstale":"2024-09-11T08:21:12.648651+0000","last_undegraded":"2024-09-11T08:21:12.648651+0000","last_fullsized":"2024-09-11T08:21:12.648651+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:28.949027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:18:28.949027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030272900000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":96,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.653325+0000","last_change":"2024-09-11T08:20:16.552108+0000","last_active":"2024-09-11T08:21:12.653325+0000","last_peered":"2024-09-11T08:21:12.653325+0000","last_clean":"2024-09-11T08:21:12.653325+0000","last_became_active":"2024-09-11T08:17:37.557645+0000","last_became_peered":"2024-09-11T08:17:37.557645+0000","last_unstale":"2024-09-11T08:21:12.653325+0000","last_undegraded":"2024-09-11T08:21:12.653325+0000","last_fullsized":"2024-09-11T08:21:12.653325+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:16.551797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:01.218475+0000","last_clean_scrub_stamp":"2024-09-11T08:20:16.551797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:20.669628+0000","scrub_duration":0.0026502240000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":325,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.651763+0000","last_change":"2024-09-11T08:21:08.731138+0000","last_active":"2024-09-11T08:21:12.651763+0000","last_peered":"2024-09-11T08:21:12.651763+0000","last_clean":"2024-09-11T08:21:12.651763+0000","last_became_active":"2024-09-11T08:19:29.198224+0000","last_became_peered":"2024-09-11T08:19:29.198224+0000","last_unstale":"2024-09-11T08:21:12.651763+0000","last_undegraded":"2024-09-11T08:21:12.651763+0000","last_fullsized":"2024-09-11T08:21:12.651763+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:08.731053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:09.973943+0000","last_clean_scrub_stamp":"2024-09-11T08:21:08.731053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:26.737814+0000","scrub_duration":0.0019130169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"239'19","reported_seq":392,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.649502+0000","last_change":"2024-09-11T08:20:23.842392+0000","last_active":"2024-09-11T08:21:12.649502+0000","last_peered":"2024-09-11T08:21:12.649502+0000","last_clean":"2024-09-11T08:21:12.649502+0000","last_became_active":"2024-09-11T08:15:56.520058+0000","last_became_peered":"2024-09-11T08:15:56.520058+0000","last_unstale":"2024-09-11T08:21:12.649502+0000","last_undegraded":"2024-09-11T08:21:12.649502+0000","last_fullsized":"2024-09-11T08:21:12.649502+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'19","last_scrub_stamp":"2024-09-11T08:20:23.842243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:20:23.842243+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:25.267175+0000","scrub_duration":0.013871632,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":51,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.654480+0000","last_change":"2024-09-11T08:21:11.652549+0000","last_active":"2024-09-11T08:21:12.654480+0000","last_peered":"2024-09-11T08:21:12.654480+0000","last_clean":"2024-09-11T08:21:12.654480+0000","last_became_active":"2024-09-11T08:19:29.143511+0000","last_became_peered":"2024-09-11T08:19:29.143511+0000","last_unstale":"2024-09-11T08:21:12.654480+0000","last_undegraded":"2024-09-11T08:21:12.654480+0000","last_fullsized":"2024-09-11T08:21:12.654480+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:11.652456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:21:11.652456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:24.596654+0000","scrub_duration":0.011171949,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":134,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.644674+0000","last_change":"2024-09-11T08:21:07.260652+0000","last_active":"2024-09-11T08:21:12.644674+0000","last_peered":"2024-09-11T08:21:12.644674+0000","last_clean":"2024-09-11T08:21:12.644674+0000","last_became_active":"2024-09-11T08:15:56.523728+0000","last_became_peered":"2024-09-11T08:15:56.523728+0000","last_unstale":"2024-09-11T08:21:12.644674+0000","last_undegraded":"2024-09-11T08:21:12.644674+0000","last_fullsized":"2024-09-11T08:21:12.644674+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:07.260572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:11:34.159053+0000","last_clean_scrub_stamp":"2024-09-11T08:21:07.260572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:14.872091+0000","scrub_duration":0.0061691999999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":296,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.647121+0000","last_change":"2024-09-11T08:20:16.004116+0000","last_active":"2024-09-11T08:21:12.647121+0000","last_peered":"2024-09-11T08:21:12.647121+0000","last_clean":"2024-09-11T08:21:12.647121+0000","last_became_active":"2024-09-11T08:07:36.633368+0000","last_became_peered":"2024-09-11T08:07:36.633368+0000","last_unstale":"2024-09-11T08:21:12.647121+0000","last_undegraded":"2024-09-11T08:21:12.647121+0000","last_fullsized":"2024-09-11T08:21:12.647121+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:16.003821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:46.489220+0000","last_clean_scrub_stamp":"2024-09-11T08:20:16.003821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:22.490239+0000","scrub_duration":0.010466069,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"239'8","reported_seq":312,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.645288+0000","last_change":"2024-09-11T08:20:16.957169+0000","last_active":"2024-09-11T08:21:12.645288+0000","last_peered":"2024-09-11T08:21:12.645288+0000","last_clean":"2024-09-11T08:21:12.645288+0000","last_became_active":"2024-09-11T08:07:32.045728+0000","last_became_peered":"2024-09-11T08:07:32.045728+0000","last_unstale":"2024-09-11T08:21:12.645288+0000","last_undegraded":"2024-09-11T08:21:12.645288+0000","last_fullsized":"2024-09-11T08:21:12.645288+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'8","last_scrub_stamp":"2024-09-11T08:20:16.957056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:20:16.957056+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:28.504779+0000","scrub_duration":0.0030711110000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":331,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.653198+0000","last_change":"2024-09-11T08:20:12.651491+0000","last_active":"2024-09-11T08:21:12.653198+0000","last_peered":"2024-09-11T08:21:12.653198+0000","last_clean":"2024-09-11T08:21:12.653198+0000","last_became_active":"2024-09-11T08:19:29.161558+0000","last_became_peered":"2024-09-11T08:19:29.161558+0000","last_unstale":"2024-09-11T08:21:12.653198+0000","last_undegraded":"2024-09-11T08:21:12.653198+0000","last_fullsized":"2024-09-11T08:21:12.653198+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:12.651408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:20:12.651408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:29.728488+0000","scrub_duration":0.0071009350000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":364,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.653908+0000","last_change":"2024-09-11T08:21:11.229626+0000","last_active":"2024-09-11T08:21:12.653908+0000","last_peered":"2024-09-11T08:21:12.653908+0000","last_clean":"2024-09-11T08:21:12.653908+0000","last_became_active":"2024-09-11T08:15:56.528607+0000","last_became_peered":"2024-09-11T08:15:56.528607+0000","last_unstale":"2024-09-11T08:21:12.653908+0000","last_undegraded":"2024-09-11T08:21:12.653908+0000","last_fullsized":"2024-09-11T08:21:12.653908+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:11.229540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:21:11.229540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:36.062439+0000","scrub_duration":0.002377456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":124,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.645921+0000","last_change":"2024-09-11T08:20:18.827796+0000","last_active":"2024-09-11T08:21:12.645921+0000","last_peered":"2024-09-11T08:21:12.645921+0000","last_clean":"2024-09-11T08:21:12.645921+0000","last_became_active":"2024-09-11T08:15:56.521488+0000","last_became_peered":"2024-09-11T08:15:56.521488+0000","last_unstale":"2024-09-11T08:21:12.645921+0000","last_undegraded":"2024-09-11T08:21:12.645921+0000","last_fullsized":"2024-09-11T08:21:12.645921+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:18.827709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:20:18.827709+0000","last_clean_scrub_stamp":"2024-09-11T08:20:18.827709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:33.930866+0000","scrub_duration":0.014060774999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":336,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.653633+0000","last_change":"2024-09-11T08:21:03.627814+0000","last_active":"2024-09-11T08:21:12.653633+0000","last_peered":"2024-09-11T08:21:12.653633+0000","last_clean":"2024-09-11T08:21:12.653633+0000","last_became_active":"2024-09-11T08:19:29.153017+0000","last_became_peered":"2024-09-11T08:19:29.153017+0000","last_unstale":"2024-09-11T08:21:12.653633+0000","last_undegraded":"2024-09-11T08:21:12.653633+0000","last_fullsized":"2024-09-11T08:21:12.653633+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:03.627721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:50.585566+0000","last_clean_scrub_stamp":"2024-09-11T08:21:03.627721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:31.922693+0000","scrub_duration":0.003783294,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":96,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.652317+0000","last_change":"2024-09-11T08:20:49.725564+0000","last_active":"2024-09-11T08:21:12.652317+0000","last_peered":"2024-09-11T08:21:12.652317+0000","last_clean":"2024-09-11T08:21:12.652317+0000","last_became_active":"2024-09-11T08:17:37.554262+0000","last_became_peered":"2024-09-11T08:17:37.554262+0000","last_unstale":"2024-09-11T08:21:12.652317+0000","last_undegraded":"2024-09-11T08:21:12.652317+0000","last_fullsized":"2024-09-11T08:21:12.652317+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:49.725505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:20:49.725505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:51.072704+0000","scrub_duration":0.0012049350000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"288'24","reported_seq":409,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.652169+0000","last_change":"2024-09-11T08:20:28.885398+0000","last_active":"2024-09-11T08:21:12.652169+0000","last_peered":"2024-09-11T08:21:12.652169+0000","last_clean":"2024-09-11T08:21:12.652169+0000","last_became_active":"2024-09-11T08:19:29.145221+0000","last_became_peered":"2024-09-11T08:19:29.145221+0000","last_unstale":"2024-09-11T08:21:12.652169+0000","last_undegraded":"2024-09-11T08:21:12.652169+0000","last_fullsized":"2024-09-11T08:21:12.652169+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'24","last_scrub_stamp":"2024-09-11T08:20:28.885295+0000","last_deep_scrub":"288'24","last_deep_scrub_stamp":"2024-09-11T08:20:28.885295+0000","last_clean_scrub_stamp":"2024-09-11T08:20:28.885295+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:52.041380+0000","scrub_duration":0.0080800170000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":152,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-11T08:21:12.651121+0000","last_change":"2024-09-11T08:20:20.817249+0000","last_active":"2024-09-11T08:21:12.651121+0000","last_peered":"2024-09-11T08:21:12.651121+0000","last_clean":"2024-09-11T08:21:12.651121+0000","last_became_active":"2024-09-11T08:15:08.769133+0000","last_became_peered":"2024-09-11T08:15:08.769133+0000","last_unstale":"2024-09-11T08:21:12.651121+0000","last_undegraded":"2024-09-11T08:21:12.651121+0000","last_fullsized":"2024-09-11T08:21:12.651121+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:20.817194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:18:39.206016+0000","last_clean_scrub_stamp":"2024-09-11T08:20:20.817194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:21:48.465266+0000","scrub_duration":0.0018451699999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":231208,"num_write_kb":7398593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16213,"num_bytes_recovered":1062535168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":275910656,"data_stored":275906584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87004,"ondisk_log_size":87004,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1125,"num_read_kb":882,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":215,"num_bytes_recovered":4786,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3425,"internal_metadata":0},"log_size":753,"ondisk_log_size":753,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":424,"num_read_kb":419,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":168,"seq":721554505813,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":321240,"kb_used_data":3728,"kb_used_omap":7,"kb_used_meta":317496,"kb_avail":93423912,"statfs":{"total":95995035648,"available":95666085888,"internally_reserved":0,"allocated":3817472,"data_stored":3459109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8027,"internal_metadata":325116069},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:20:30 2024","interfaces":[{"interface":"back","average":{"1min":5.3049999999999997,"5min":1.929,"15min":1.1930000000000001},"min":{"1min":0.22800000000000001,"5min":0.223,"15min":0.223},"max":{"1min":67.409999999999997,"5min":67.409999999999997,"15min":67.409999999999997},"last":0.46899999999999997},{"interface":"front","average":{"1min":5.0330000000000004,"5min":1.776,"15min":1.2549999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":66.959999999999994,"5min":66.959999999999994,"15min":66.959999999999994},"last":0.33100000000000002}]},{"osd":1,"last update":"Wed Sep 11 08:20:30 2024","interfaces":[{"interface":"back","average":{"1min":5.819,"5min":3.0859999999999999,"15min":2.3260000000000001},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":66.971000000000004,"5min":66.971000000000004,"15min":66.971000000000004},"last":0.51700000000000002},{"interface":"front","average":{"1min":5.3339999999999996,"5min":2.746,"15min":2.2789999999999999},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":67.185000000000002,"5min":67.185000000000002,"15min":67.185000000000002},"last":0.57299999999999995}]},{"osd":2,"last update":"Wed Sep 11 08:20:30 2024","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":0.91400000000000003,"15min":0.999},"min":{"1min":0.25700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":2.9729999999999999,"5min":6.444,"15min":6.444},"last":0.39400000000000002},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.0509999999999999,"15min":0.83099999999999996},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":4.5540000000000003,"5min":6.3460000000000001,"15min":6.3460000000000001},"last":0.45100000000000001}]},{"osd":3,"last update":"Wed Sep 11 08:20:30 2024","interfaces":[{"interface":"back","average":{"1min":1.3440000000000001,"5min":1.3560000000000001,"15min":1.167},"min":{"1min":0.311,"5min":0.251,"15min":0.251},"max":{"1min":5.2149999999999999,"5min":21.460000000000001,"15min":21.460000000000001},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.96499999999999997,"5min":1.2789999999999999,"15min":1.3759999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":3.0019999999999998,"5min":21.440000000000001,"15min":21.440000000000001},"last":0.41999999999999998}]},{"osd":4,"last update":"Wed Sep 11 08:20:30 2024","interfaces":[{"interface":"back","average":{"1min":1.448,"5min":0.88400000000000001,"15min":0.64500000000000002},"min":{"1min":0.25800000000000001,"5min":0.252,"15min":0.252},"max":{"1min":4.5289999999999999,"5min":4.5289999999999999,"15min":4.5289999999999999},"last":0.53900000000000003},{"interface":"front","average":{"1min":1.375,"5min":0.82299999999999995,"15min":0.77700000000000002},"min":{"1min":0.27200000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":5.2649999999999997,"5min":5.2649999999999997,"15min":5.2649999999999997},"last":0.504}]},{"osd":5,"last update":"Wed Sep 11 08:20:30 2024","interfaces":[{"interface":"back","average":{"1min":1.5640000000000001,"5min":1.764,"15min":1.1200000000000001},"min":{"1min":0.28699999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":5.1559999999999997,"5min":65.866,"15min":65.866},"last":0.56200000000000006},{"interface":"front","average":{"1min":1.9019999999999999,"5min":1.8280000000000001,"15min":1.109},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":6.3289999999999997,"5min":66.019000000000005,"15min":66.019000000000005},"last":1.417}]},{"osd":6,"last update":"Wed Sep 11 08:20:30 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.73999999999999999,"15min":0.77500000000000002},"min":{"1min":0.34799999999999998,"5min":0.248,"15min":0.248},"max":{"1min":3.0259999999999998,"5min":6.0609999999999999,"15min":6.0609999999999999},"last":0.44},{"interface":"front","average":{"1min":1.4930000000000001,"5min":0.93500000000000005,"15min":0.83799999999999997},"min":{"1min":0.379,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":4.907,"5min":4.907,"15min":4.907},"last":0.71399999999999997}]}]},{"osd":6,"up_from":45,"seq":193273528496,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311108,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93434044,"statfs":{"total":95995035648,"available":95676461056,"internally_reserved":0,"allocated":1306624,"data_stored":968540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:21:19 2024","interfaces":[{"interface":"back","average":{"1min":1.0600000000000001,"5min":1.5509999999999999,"15min":1.411},"min":{"1min":0.19900000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":2.7309999999999999,"5min":25.494,"15min":45.066000000000003},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.92000000000000004,"5min":1.5209999999999999,"15min":1.365},"min":{"1min":0.254,"5min":0.23699999999999999,"15min":0.20599999999999999},"max":{"1min":2.8090000000000002,"5min":25.48,"15min":43.899000000000001},"last":0.64600000000000002}]},{"osd":1,"last update":"Wed Sep 11 08:21:19 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":1.3879999999999999,"15min":1.198},"min":{"1min":0.218,"5min":0.218,"15min":0.214},"max":{"1min":2.758,"5min":20.321000000000002,"15min":44.658000000000001},"last":0.52000000000000002},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.391,"15min":1.2529999999999999},"min":{"1min":0.26400000000000001,"5min":0.22800000000000001,"15min":0.22600000000000001},"max":{"1min":5.4800000000000004,"5min":19.652000000000001,"15min":44.694000000000003},"last":0.58899999999999997}]},{"osd":2,"last update":"Wed Sep 11 08:21:19 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.751,"15min":0.78300000000000003},"min":{"1min":0.254,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":2.0449999999999999,"5min":3.8759999999999999,"15min":5.9770000000000003},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.94999999999999996,"15min":0.998},"min":{"1min":0.252,"5min":0.252,"15min":0.23300000000000001},"max":{"1min":2.1549999999999998,"5min":7.0549999999999997,"15min":13.346},"last":0.58099999999999996}]},{"osd":3,"last update":"Wed Sep 11 08:21:19 2024","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":1.024,"15min":1.161},"min":{"1min":0.308,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.2969999999999997,"5min":5.2969999999999997,"15min":36.061},"last":0.66000000000000003},{"interface":"front","average":{"1min":1.115,"5min":1.071,"15min":1.155},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.26800000000000002},"max":{"1min":5.2519999999999998,"5min":5.5720000000000001,"15min":36.093000000000004},"last":0.62}]},{"osd":4,"last update":"Wed Sep 11 08:21:19 2024","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.66600000000000004,"15min":0.55000000000000004},"min":{"1min":0.24299999999999999,"5min":0.20499999999999999,"15min":0.20200000000000001},"max":{"1min":4.3440000000000003,"5min":4.492,"15min":4.492},"last":0.496},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.61799999999999999,"15min":0.56799999999999995},"min":{"1min":0.32200000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":2.5819999999999999,"5min":4.1529999999999996,"15min":4.468},"last":0.55600000000000005}]},{"osd":5,"last update":"Wed Sep 11 08:21:19 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.82699999999999996,"15min":0.69699999999999995},"min":{"1min":0.309,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":2.3029999999999999,"5min":10.49,"15min":10.49},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.79500000000000004,"15min":0.66200000000000003},"min":{"1min":0.32000000000000001,"5min":0.28100000000000003,"15min":0.22600000000000001},"max":{"1min":2.7810000000000001,"5min":10.468,"15min":10.468},"last":0.628}]},{"osd":7,"last update":"Wed Sep 11 08:20:27 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.77200000000000002,"15min":1.0209999999999999},"min":{"1min":0.254,"5min":0.253,"15min":0.253},"max":{"1min":1.986,"5min":8.4580000000000002,"15min":8.4580000000000002},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.435,"5min":0.63400000000000001,"15min":0.81200000000000006},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.83999999999999997,"5min":3.593,"15min":3.593},"last":0.63800000000000001}]}]},{"osd":5,"up_from":38,"seq":163208757429,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":594756,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":93150396,"statfs":{"total":95995035648,"available":95386005504,"internally_reserved":0,"allocated":716800,"data_stored":377733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:20:58 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":2.0619999999999998,"15min":1.3},"min":{"1min":0.28599999999999998,"5min":0.184,"15min":0.184},"max":{"1min":16.652000000000001,"5min":92.924000000000007,"15min":92.924000000000007},"last":0.311},{"interface":"front","average":{"1min":1.329,"5min":2.246,"15min":1.359},"min":{"1min":0.25,"5min":0.246,"15min":0.20399999999999999},"max":{"1min":3.8679999999999999,"5min":93.173000000000002,"15min":93.173000000000002},"last":0.41199999999999998}]},{"osd":1,"last update":"Wed Sep 11 08:20:58 2024","interfaces":[{"interface":"back","average":{"1min":2.6040000000000001,"5min":2.3090000000000002,"15min":1.6000000000000001},"min":{"1min":0.32900000000000001,"5min":0.22,"15min":0.22},"max":{"1min":16.265000000000001,"5min":96.899000000000001,"15min":96.899000000000001},"last":4.5510000000000002},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.9390000000000001,"15min":1.3360000000000001},"min":{"1min":0.25700000000000001,"5min":0.248,"15min":0.22700000000000001},"max":{"1min":2.4700000000000002,"5min":92.337000000000003,"15min":92.337000000000003},"last":2.5499999999999998}]},{"osd":2,"last update":"Wed Sep 11 08:20:58 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.0569999999999999,"15min":1.0620000000000001},"min":{"1min":0.35599999999999998,"5min":0.255,"15min":0.255},"max":{"1min":6.0860000000000003,"5min":15.622,"15min":16.018000000000001},"last":0.48599999999999999},{"interface":"front","average":{"1min":1.238,"5min":0.85199999999999998,"15min":0.79500000000000004},"min":{"1min":0.317,"5min":0.27300000000000002,"15min":0.22800000000000001},"max":{"1min":4.9370000000000003,"5min":4.9370000000000003,"15min":7.5250000000000004},"last":4.9800000000000004}]},{"osd":3,"last update":"Wed Sep 11 08:20:58 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.8560000000000001,"15min":1.2969999999999999},"min":{"1min":0.28799999999999998,"5min":0.25,"15min":0.25},"max":{"1min":7.4880000000000004,"5min":92.245000000000005,"15min":92.245000000000005},"last":3.7170000000000001},{"interface":"front","average":{"1min":2.319,"5min":2.3159999999999998,"15min":1.478},"min":{"1min":0.35099999999999998,"5min":0.27700000000000002,"15min":0.251},"max":{"1min":16.457999999999998,"5min":92.274000000000001,"15min":92.274000000000001},"last":0.53800000000000003}]},{"osd":4,"last update":"Wed Sep 11 08:20:58 2024","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":0.82199999999999995,"15min":0.61399999999999999},"min":{"1min":0.27100000000000002,"5min":0.25800000000000001,"15min":0.21299999999999999},"max":{"1min":4.6719999999999997,"5min":4.6719999999999997,"15min":4.6719999999999997},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.72399999999999998,"15min":0.57599999999999996},"min":{"1min":0.24199999999999999,"5min":0.23599999999999999,"15min":0.23100000000000001},"max":{"1min":4.9720000000000004,"5min":4.9720000000000004,"15min":4.9720000000000004},"last":2.5750000000000002}]},{"osd":6,"last update":"Wed Sep 11 08:21:17 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.82299999999999995,"15min":0.60799999999999998},"min":{"1min":0.311,"5min":0.24399999999999999,"15min":0.23400000000000001},"max":{"1min":5.2869999999999999,"5min":7.5679999999999996,"15min":7.5679999999999996},"last":0.51100000000000001},{"interface":"front","average":{"1min":1.46,"5min":0.98899999999999999,"15min":0.71299999999999997},"min":{"1min":0.375,"5min":0.27500000000000002,"15min":0.26900000000000002},"max":{"1min":6.4470000000000001,"5min":6.4470000000000001,"15min":6.4470000000000001},"last":0.44600000000000001}]},{"osd":7,"last update":"Wed Sep 11 08:20:37 2024","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":0.97499999999999998,"15min":0.879},"min":{"1min":0.44800000000000001,"5min":0.33600000000000002,"15min":0.30399999999999999},"max":{"1min":3.0209999999999999,"5min":3.5800000000000001,"15min":3.5800000000000001},"last":0.47199999999999998},{"interface":"front","average":{"1min":1.0669999999999999,"5min":0.999,"15min":0.84199999999999997},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":3.141,"5min":7.5339999999999998,"15min":7.5339999999999998},"last":1.998}]}]},{"osd":4,"up_from":31,"seq":133143986362,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":571676,"kb_used_data":270164,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93173476,"statfs":{"total":95995035648,"available":95409639424,"internally_reserved":0,"allocated":276647936,"data_stored":276285631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:20:29 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":1.9279999999999999,"15min":1.478},"min":{"1min":0.23400000000000001,"5min":0.222,"15min":0.19},"max":{"1min":2.8639999999999999,"5min":60.497999999999998,"15min":60.497999999999998},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.88500000000000001,"5min":1.8240000000000001,"15min":1.341},"min":{"1min":0.247,"5min":0.22600000000000001,"15min":0.221},"max":{"1min":5.9870000000000001,"5min":60.484999999999999,"15min":60.484999999999999},"last":0.374}]},{"osd":1,"last update":"Wed Sep 11 08:20:29 2024","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":1.24,"15min":0.95599999999999996},"min":{"1min":0.33000000000000002,"5min":0.214,"15min":0.214},"max":{"1min":2.556,"5min":20.471,"15min":20.471},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.97299999999999998,"5min":1.21,"15min":0.89300000000000002},"min":{"1min":0.249,"5min":0.23100000000000001,"15min":0.22700000000000001},"max":{"1min":2.8180000000000001,"5min":15.744999999999999,"15min":15.744999999999999},"last":0.38900000000000001}]},{"osd":2,"last update":"Wed Sep 11 08:20:29 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.76200000000000001,"15min":0.78700000000000003},"min":{"1min":0.25600000000000001,"5min":0.245,"15min":0.23400000000000001},"max":{"1min":2.9220000000000002,"5min":3.3599999999999999,"15min":7.976},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.88400000000000001,"15min":0.79600000000000004},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.248},"max":{"1min":3.5289999999999999,"5min":7.0590000000000002,"15min":7.0590000000000002},"last":0.40200000000000002}]},{"osd":3,"last update":"Wed Sep 11 08:20:29 2024","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.48,"15min":1.258},"min":{"1min":0.27600000000000002,"5min":0.253,"15min":0.253},"max":{"1min":3.7610000000000001,"5min":19.686,"15min":26.884},"last":0.313},{"interface":"front","average":{"1min":1.169,"5min":1.196,"15min":1.1879999999999999},"min":{"1min":0.29499999999999998,"5min":0.26600000000000001,"15min":0.246},"max":{"1min":4.0570000000000004,"5min":19.882000000000001,"15min":26.783000000000001},"last":0.36299999999999999}]},{"osd":5,"last update":"Wed Sep 11 08:20:56 2024","interfaces":[{"interface":"back","average":{"1min":2.6459999999999999,"5min":1.024,"15min":0.64300000000000002},"min":{"1min":0.30299999999999999,"5min":0.23999999999999999,"15min":0.23499999999999999},"max":{"1min":33.384999999999998,"5min":33.384999999999998,"15min":33.384999999999998},"last":0.32000000000000001},{"interface":"front","average":{"1min":2.3929999999999998,"5min":1.121,"15min":0.66700000000000004},"min":{"1min":0.253,"5min":0.253,"15min":0.24199999999999999},"max":{"1min":33.856000000000002,"5min":33.856000000000002,"15min":33.856000000000002},"last":0.255}]},{"osd":6,"last update":"Wed Sep 11 08:21:19 2024","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.66000000000000003,"15min":0.68899999999999995},"min":{"1min":0.34300000000000003,"5min":0.25,"15min":0.23899999999999999},"max":{"1min":2.7589999999999999,"5min":4.6799999999999997,"15min":48.808},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.70199999999999996,"15min":0.73999999999999999},"min":{"1min":0.35199999999999998,"5min":0.26800000000000002,"15min":0.26200000000000001},"max":{"1min":1.948,"5min":4.0499999999999998,"15min":48.746000000000002},"last":0.41699999999999998}]},{"osd":7,"last update":"Wed Sep 11 08:20:39 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.72899999999999998,"15min":0.56100000000000005},"min":{"1min":0.29199999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":2.8660000000000001,"5min":5.3449999999999998,"15min":5.3449999999999998},"last":0.34399999999999997},{"interface":"front","average":{"1min":1.3680000000000001,"5min":0.86599999999999999,"15min":0.72999999999999998},"min":{"1min":0.35899999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":4.0910000000000002,"5min":4.7160000000000002,"15min":4.7160000000000002},"last":0.27400000000000002}]}]},{"osd":3,"up_from":25,"seq":107374182591,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":598948,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":93146204,"statfs":{"total":95995035648,"available":95381712896,"internally_reserved":0,"allocated":749568,"data_stored":378518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:21:19 2024","interfaces":[{"interface":"back","average":{"1min":3.282,"5min":1.752,"15min":1.5569999999999999},"min":{"1min":0.14999999999999999,"5min":0.14999999999999999,"15min":0.14999999999999999},"max":{"1min":58.880000000000003,"5min":58.880000000000003,"15min":60.469999999999999},"last":0.61499999999999999},{"interface":"front","average":{"1min":3.5390000000000001,"5min":1.794,"15min":1.655},"min":{"1min":0.184,"5min":0.184,"15min":0.17399999999999999},"max":{"1min":59.246000000000002,"5min":59.246000000000002,"15min":60.488999999999997},"last":0.30199999999999999}]},{"osd":1,"last update":"Wed Sep 11 08:21:19 2024","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.3759999999999999,"15min":1.395},"min":{"1min":0.27500000000000002,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":11.148999999999999,"5min":11.148999999999999,"15min":60.578000000000003},"last":0.33400000000000002},{"interface":"front","average":{"1min":2.2429999999999999,"5min":1.925,"15min":1.6799999999999999},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.19500000000000001},"max":{"1min":11.853,"5min":46.868000000000002,"15min":63.674999999999997},"last":2.7349999999999999}]},{"osd":2,"last update":"Wed Sep 11 08:21:19 2024","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.2390000000000001,"15min":1.482},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.185},"max":{"1min":4.6100000000000003,"5min":8.4179999999999993,"15min":60.645000000000003},"last":2.8090000000000002},{"interface":"front","average":{"1min":0.90400000000000003,"5min":1.1120000000000001,"15min":1.3660000000000001},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":3.3999999999999999,"5min":5.9500000000000002,"15min":60.723999999999997},"last":0.48199999999999998}]},{"osd":4,"last update":"Wed Sep 11 08:20:25 2024","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.7749999999999999,"15min":1.246},"min":{"1min":0.372,"5min":0.28100000000000003,"15min":0.24299999999999999},"max":{"1min":2.7839999999999998,"5min":61.290999999999997,"15min":61.290999999999997},"last":2.6760000000000002},{"interface":"front","average":{"1min":0.84699999999999998,"5min":1.587,"15min":1.151},"min":{"1min":0.40300000000000002,"5min":0.28899999999999998,"15min":0.27200000000000002},"max":{"1min":1.875,"5min":60.856000000000002,"15min":60.856000000000002},"last":0.64700000000000002}]},{"osd":5,"last update":"Wed Sep 11 08:21:02 2024","interfaces":[{"interface":"back","average":{"1min":2.4199999999999999,"5min":1.9950000000000001,"15min":1.645},"min":{"1min":0.29999999999999999,"5min":0.28199999999999997,"15min":0.26500000000000001},"max":{"1min":25.754999999999999,"5min":31.460999999999999,"15min":60.914999999999999},"last":3.7120000000000002},{"interface":"front","average":{"1min":2.5750000000000002,"5min":1.9510000000000001,"15min":1.7110000000000001},"min":{"1min":0.35799999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":25.577999999999999,"5min":31.216000000000001,"15min":60.787999999999997},"last":2.9089999999999998}]},{"osd":6,"last update":"Wed Sep 11 08:21:19 2024","interfaces":[{"interface":"back","average":{"1min":1.627,"5min":1.212,"15min":1.514},"min":{"1min":0.33200000000000002,"5min":0.308,"15min":0.308},"max":{"1min":10.949999999999999,"5min":10.949999999999999,"15min":60.817},"last":0.38300000000000001},{"interface":"front","average":{"1min":2.0779999999999998,"5min":1.556,"15min":1.7030000000000001},"min":{"1min":0.35399999999999998,"5min":0.34699999999999998,"15min":0.307},"max":{"1min":11.750999999999999,"5min":11.837,"15min":61.264000000000003},"last":2.71}]},{"osd":7,"last update":"Wed Sep 11 08:20:33 2024","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":2.0699999999999998,"15min":2.7839999999999998},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":3.8479999999999999,"5min":60.847000000000001,"15min":60.847000000000001},"last":0.35999999999999999},{"interface":"front","average":{"1min":1.075,"5min":2.02,"15min":2.722},"min":{"1min":0.48899999999999999,"5min":0.36699999999999999,"15min":0.34499999999999997},"max":{"1min":2.351,"5min":60.883000000000003,"15min":60.883000000000003},"last":0.94599999999999995}]}]},{"osd":2,"up_from":18,"seq":77309411522,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298052,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447100,"statfs":{"total":95995035648,"available":95689830400,"internally_reserved":0,"allocated":716800,"data_stored":377645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:20:52 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":1.55,"15min":1.4610000000000001},"min":{"1min":0.183,"5min":0.17799999999999999,"15min":0.17199999999999999},"max":{"1min":3.1699999999999999,"5min":25.427,"15min":43.539999999999999},"last":1.395},{"interface":"front","average":{"1min":0.84599999999999997,"5min":2.02,"15min":1.694},"min":{"1min":0.19700000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":2.2650000000000001,"5min":44.424999999999997,"15min":44.424999999999997},"last":5.6660000000000004}]},{"osd":1,"last update":"Wed Sep 11 08:20:52 2024","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.556,"15min":1.282},"min":{"1min":0.20799999999999999,"5min":0.20200000000000001,"15min":0.17599999999999999},"max":{"1min":5.2519999999999998,"5min":23.748999999999999,"15min":23.748999999999999},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.78500000000000003,"5min":1.3280000000000001,"15min":1.256},"min":{"1min":0.23000000000000001,"5min":0.20799999999999999,"15min":0.20200000000000001},"max":{"1min":6.0570000000000004,"5min":23.513999999999999,"15min":23.513999999999999},"last":5.7039999999999997}]},{"osd":3,"last update":"Wed Sep 11 08:21:19 2024","interfaces":[{"interface":"back","average":{"1min":3.0630000000000002,"5min":1.3879999999999999,"15min":1.3049999999999999},"min":{"1min":0.224,"5min":0.221,"15min":0.188},"max":{"1min":46.070999999999998,"5min":46.070999999999998,"15min":46.070999999999998},"last":1.3720000000000001},{"interface":"front","average":{"1min":3.3530000000000002,"5min":1.2969999999999999,"15min":1.248},"min":{"1min":0.26700000000000002,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":45.790999999999997,"5min":45.790999999999997,"15min":45.790999999999997},"last":6.3070000000000004}]},{"osd":4,"last update":"Wed Sep 11 08:20:35 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.90200000000000002,"15min":0.90100000000000002},"min":{"1min":0.308,"5min":0.27900000000000003,"15min":0.25900000000000001},"max":{"1min":2.125,"5min":5.5970000000000004,"15min":7.806},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.84999999999999998,"15min":0.91800000000000004},"min":{"1min":0.29299999999999998,"5min":0.27500000000000002,"15min":0.252},"max":{"1min":2.5139999999999998,"5min":3.855,"15min":12.129},"last":5.7279999999999998}]},{"osd":5,"last update":"Wed Sep 11 08:21:06 2024","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":1.7290000000000001,"15min":1.2330000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":7.9059999999999997,"5min":35.963000000000001,"15min":35.963000000000001},"last":5.7530000000000001},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.8140000000000001,"15min":1.2490000000000001},"min":{"1min":0.32700000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":4.0149999999999997,"5min":35.104999999999997,"15min":35.104999999999997},"last":0.745}]},{"osd":6,"last update":"Wed Sep 11 08:20:23 2024","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":0.91400000000000003,"15min":0.877},"min":{"1min":0.31,"5min":0.31,"15min":0.26100000000000001},"max":{"1min":4.2169999999999996,"5min":4.2539999999999996,"15min":7.2249999999999996},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.90000000000000002,"5min":1.0800000000000001,"15min":0.97599999999999998},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.32100000000000001},"max":{"1min":2.8010000000000002,"5min":5.4939999999999998,"15min":6.71},"last":5.8010000000000002}]},{"osd":7,"last update":"Wed Sep 11 08:20:35 2024","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.5680000000000001,"15min":1.448},"min":{"1min":0.34599999999999997,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":11.19,"5min":35.274999999999999,"15min":35.274999999999999},"last":1.613},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.5409999999999999,"15min":1.538},"min":{"1min":0.40100000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":11.172000000000001,"5min":35.265000000000001,"15min":35.265000000000001},"last":5.7770000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575048,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":594792,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":93150360,"statfs":{"total":95995035648,"available":95385968640,"internally_reserved":0,"allocated":753664,"data_stored":378661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:20:23 2024","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.573,"15min":1.2709999999999999},"min":{"1min":0.222,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":5.8609999999999998,"5min":8.6720000000000006,"15min":32.478000000000002},"last":0.36799999999999999},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.71,"15min":1.5329999999999999},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":5.6310000000000002,"5min":15.351000000000001,"15min":35.459000000000003},"last":0.51300000000000001}]},{"osd":2,"last update":"Wed Sep 11 08:21:06 2024","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.3759999999999999,"15min":1.141},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":4.5439999999999996,"5min":7.798,"15min":8.3819999999999997},"last":0.624},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.248,"15min":1.2250000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.16300000000000001},"max":{"1min":3.4660000000000002,"5min":5.8490000000000002,"15min":10.276},"last":0.47799999999999998}]},{"osd":3,"last update":"Wed Sep 11 08:21:13 2024","interfaces":[{"interface":"back","average":{"1min":1.726,"5min":1.7170000000000001,"15min":1.77},"min":{"1min":0.23599999999999999,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":8.0909999999999993,"5min":8.4149999999999991,"15min":32.759},"last":0.64300000000000002},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.7969999999999999,"15min":1.6279999999999999},"min":{"1min":0.29899999999999999,"5min":0.25,"15min":0.22600000000000001},"max":{"1min":5.069,"5min":9.0060000000000002,"15min":32.444000000000003},"last":0.72099999999999997}]},{"osd":4,"last update":"Wed Sep 11 08:20:33 2024","interfaces":[{"interface":"back","average":{"1min":2.1869999999999998,"5min":3.0819999999999999,"15min":1.6619999999999999},"min":{"1min":0.41499999999999998,"5min":0.26200000000000001,"15min":0.252},"max":{"1min":17.856000000000002,"5min":110.86499999999999,"15min":110.86499999999999},"last":0.53700000000000003},{"interface":"front","average":{"1min":1.8080000000000001,"5min":2.9249999999999998,"15min":1.5880000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.248},"max":{"1min":5.7949999999999999,"5min":110.831,"15min":110.831},"last":0.69499999999999995}]},{"osd":5,"last update":"Wed Sep 11 08:21:06 2024","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":2.839,"15min":1.6799999999999999},"min":{"1min":0.38,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":4.0650000000000004,"5min":109.208,"15min":109.208},"last":1.675},{"interface":"front","average":{"1min":1.27,"5min":2.9780000000000002,"15min":1.7709999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28100000000000003},"max":{"1min":5.7149999999999999,"5min":111.254,"15min":111.254},"last":1.242}]},{"osd":6,"last update":"Wed Sep 11 08:21:13 2024","interfaces":[{"interface":"back","average":{"1min":1.4430000000000001,"5min":1.4510000000000001,"15min":1.6830000000000001},"min":{"1min":0.307,"5min":0.30499999999999999,"15min":0.29399999999999998},"max":{"1min":7.9809999999999999,"5min":7.9809999999999999,"15min":124.31999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":2.117,"5min":1.5880000000000001,"15min":1.6699999999999999},"min":{"1min":0.42299999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":8.6099999999999994,"5min":8.6099999999999994,"15min":124.958},"last":0.66700000000000004}]},{"osd":7,"last update":"Wed Sep 11 08:20:33 2024","interfaces":[{"interface":"back","average":{"1min":1.8959999999999999,"5min":2.7280000000000002,"15min":1.6990000000000001},"min":{"1min":0.435,"5min":0.33400000000000002,"15min":0.29399999999999998},"max":{"1min":5.0650000000000004,"5min":109.149,"15min":109.149},"last":0.84999999999999998},{"interface":"front","average":{"1min":1.9259999999999999,"5min":2.742,"15min":1.8999999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":5.7590000000000003,"5min":109.309,"15min":109.309},"last":0.61399999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738572,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":302816,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93442336,"statfs":{"total":95995035648,"available":95684952064,"internally_reserved":0,"allocated":1335296,"data_stored":969663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 11 08:20:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":1.3200000000000001,"15min":1.1240000000000001},"min":{"1min":0.216,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":3.306,"5min":10.105,"15min":10.105},"last":0.247},{"interface":"front","average":{"1min":1.363,"5min":1.8069999999999999,"15min":1.3580000000000001},"min":{"1min":0.17999999999999999,"5min":0.16600000000000001,"15min":0.153},"max":{"1min":3.0870000000000002,"5min":19.852,"15min":19.852},"last":0.377}]},{"osd":2,"last update":"Wed Sep 11 08:20:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":1.3640000000000001,"15min":1.1399999999999999},"min":{"1min":0.20799999999999999,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":3.0110000000000001,"5min":10.025,"15min":10.351000000000001},"last":0.27400000000000002},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.391,"15min":1.228},"min":{"1min":0.26300000000000001,"5min":0.23699999999999999,"15min":0.186},"max":{"1min":4.883,"5min":10.074999999999999,"15min":10.074999999999999},"last":0.313}]},{"osd":3,"last update":"Wed Sep 11 08:21:16 2024","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":1.3779999999999999,"15min":1.526},"min":{"1min":0.27700000000000002,"5min":0.251,"15min":0.22900000000000001},"max":{"1min":3.2749999999999999,"5min":10.129,"15min":35.491},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.624,"5min":1.0549999999999999,"15min":1.456},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":2.5230000000000001,"5min":8.282,"15min":37.563000000000002},"last":0.30099999999999999}]},{"osd":4,"last update":"Wed Sep 11 08:20:38 2024","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.999,"15min":1.02},"min":{"1min":0.252,"5min":0.252,"15min":0.24099999999999999},"max":{"1min":4.5949999999999998,"5min":5.0190000000000001,"15min":9.4030000000000005},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.91600000000000004,"5min":1.0820000000000001,"15min":1.0329999999999999},"min":{"1min":0.32500000000000001,"5min":0.27200000000000002,"15min":0.24399999999999999},"max":{"1min":3.0430000000000001,"5min":8.3510000000000009,"15min":8.3510000000000009},"last":0.36799999999999999}]},{"osd":5,"last update":"Wed Sep 11 08:20:59 2024","interfaces":[{"interface":"back","average":{"1min":1.5720000000000001,"5min":1.347,"15min":1.244},"min":{"1min":0.42599999999999999,"5min":0.28599999999999998,"15min":0.255},"max":{"1min":10.486000000000001,"5min":10.486000000000001,"15min":24.914999999999999},"last":0.40300000000000002},{"interface":"front","average":{"1min":2.3769999999999998,"5min":1.5229999999999999,"15min":1.3959999999999999},"min":{"1min":0.35699999999999998,"5min":0.28199999999999997,"15min":0.27100000000000002},"max":{"1min":12.536,"5min":13.885,"15min":25.277000000000001},"last":0.499}]},{"osd":6,"last update":"Wed Sep 11 08:21:22 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.874,"15min":1.3899999999999999},"min":{"1min":0.32600000000000001,"5min":0.29899999999999999,"15min":0.29099999999999998},"max":{"1min":3.2890000000000001,"5min":4.2670000000000003,"15min":78.522000000000006},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.77700000000000002,"5min":1.0629999999999999,"15min":1.3140000000000001},"min":{"1min":0.32900000000000001,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.8620000000000001,"5min":7.4820000000000002,"15min":79.808000000000007},"last":0.439}]},{"osd":7,"last update":"Wed Sep 11 08:20:38 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.292,"15min":1.3380000000000001},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":3.0680000000000001,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.41199999999999998},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.4450000000000001,"15min":1.385},"min":{"1min":0.33800000000000002,"5min":0.33600000000000002,"15min":0.32000000000000001},"max":{"1min":3.1859999999999999,"5min":13.914,"15min":13.914},"last":0.48699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3425,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":275910656,"data_stored":275906584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T08:21:28.569 INFO:tasks.thrashosds.thrasher:Setting 9.6 to [0, 1, 3, 4, 6, 7] 2024-09-11T08:21:28.570 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '9.6', '0', '1', '3', '4', '6', '7'] 2024-09-11T08:21:28.570 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd pg-upmap-items 9.6 0 1 3 4 6 7 2024-09-11T08:21:28.657 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:28.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:28.680 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:28.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:28.781 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:28.802 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:28.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:28.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:28.823 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:21:28 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:21:28.873 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:21:28 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:21:28] "GET /metrics HTTP/1.1" 200 224533 "" "Prometheus/2.43.0" 2024-09-11T08:21:28.877 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:21:28.878 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:21:28.904 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:28.926 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:28.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:28.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:29.028 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:21:29.049 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:29.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:29.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:21:29.124 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:21:29.124 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:21:29.137 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:21:29.137 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:21:29.151 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:29.152 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:21:29.152 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:21:29.168 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:21:29.168 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:21:29.174 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:29.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:29.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:29.183 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:21:29.184 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:21:29.199 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:21:29.199 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:21:29.214 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:21:29.214 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:21:29.230 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:21:29.231 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:21:29.275 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:21:29.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:29.298 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:29.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:21:29.400 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:29.425 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:29.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:29.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:29.526 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:21:29.548 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:29.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:29.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:21:29.650 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:29.675 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:29.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:29.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:29.772 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T08:21:29.777 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:21:29.803 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:29.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:29.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:21:29.905 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:29.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:29.932 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:29.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:30.033 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:30.058 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:30.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:30.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:30.160 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:30.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:30.183 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:30.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:30.284 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:30.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:30.309 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:30.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:30.410 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:30.432 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:30.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:30.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:30.434 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:30 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:21:30.434 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:30 smithi150 ceph-mon[29276]: pgmap v634: 148 pgs: 148 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 441 op/s 2024-09-11T08:21:30.434 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:30 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3489556699' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:21:30.434 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:30 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:21:30.434 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:30 smithi150 ceph-mon[39612]: pgmap v634: 148 pgs: 148 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 441 op/s 2024-09-11T08:21:30.434 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:30 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3489556699' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:21:30.534 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:30.558 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:30.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:30.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:30.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:30 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:21:30.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:30 smithi181 ceph-mon[32701]: pgmap v634: 148 pgs: 148 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 441 op/s 2024-09-11T08:21:30.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:30 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3489556699' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:21:30.661 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:30.686 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:30.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:30.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:30.788 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:21:30.810 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:30.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:30.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:21:30.912 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:21:30.938 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:30.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:30.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:21:31.039 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:31.062 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:31.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:31.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:31.089 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:21:31.163 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:31.187 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:31.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:31.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:31.288 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:21:31.311 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:31.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:31.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:21:31.412 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:31.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:31.436 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:31.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:31.537 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:31.558 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:31.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:31.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:31.660 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:21:31.683 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:31.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:31.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:21:31.785 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:31.806 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:31.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:31.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:31.908 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:31.934 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T08:21:31.935 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:31.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:31.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:32.037 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:21:32.059 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:32.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:32.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:21:32.161 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:32.183 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:32.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:32.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:32.201 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:32 smithi150 ceph-mon[39612]: pgmap v635: 148 pgs: 148 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 441 op/s 2024-09-11T08:21:32.201 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:32 smithi150 ceph-mon[29276]: pgmap v635: 148 pgs: 148 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 441 op/s 2024-09-11T08:21:32.285 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:32.307 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:32.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:32.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:32.409 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:21:32.430 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:32.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:32.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:21:32.531 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:32.553 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:32.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:32.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:32.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:32 smithi181 ceph-mon[32701]: pgmap v635: 148 pgs: 148 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 441 op/s 2024-09-11T08:21:32.655 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:21:32.677 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:32.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:32.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:21:32.778 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:32.801 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:32.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:32.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:32.903 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:21:32.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:32.926 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:32.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:21:33.027 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:33.050 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:33.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:33.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:33.152 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:33.175 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:33.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:33.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:33.277 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:33.290 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:21:33.299 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:33.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:33.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:33.400 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:21:33.423 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:33.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:33.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:21:33.525 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:33.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:33.547 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:33.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:33.648 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:21:33.670 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:33.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:33.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:21:33.771 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:33.793 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:33.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:33.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:33.894 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:21:33.916 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:33.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:33.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:21:34.018 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:34.040 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:34.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:34.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:34.058 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:21:34.141 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:34.163 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:34.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:34.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:34.265 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:34.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:34.292 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:34.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:34.293 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:34 smithi150 ceph-mon[29276]: pgmap v636: 148 pgs: 148 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 378 op/s 2024-09-11T08:21:34.293 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:34 smithi150 ceph-mon[39612]: pgmap v636: 148 pgs: 148 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 378 op/s 2024-09-11T08:21:34.393 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:21:34.415 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:34.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:34.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:21:34.517 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:34.519 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-09-11T08:21:34.519 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:21:34.543 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:34.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:34.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:34.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:34 smithi181 ceph-mon[32701]: pgmap v636: 148 pgs: 148 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 378 op/s 2024-09-11T08:21:34.645 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:34.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:34.668 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:34.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:34.769 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:34.795 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:34.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:34.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:34.896 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:34.918 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:34.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:34.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:35.019 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:21:35.041 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:35.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:35.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:21:35.145 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:35.169 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:35.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:35.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:35.270 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:35.293 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:35.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:35.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:35.395 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:21:35.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:35.417 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:35.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:21:35.518 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:21:35.539 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:35.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:35.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:21:35.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:21:35 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:21:35] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:21:35.642 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:21:35.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:35.664 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:35.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:21:35.765 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:35.791 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:35.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:35.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:35.892 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:21:35.919 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:35.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:35.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:21:36.021 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:36.044 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:36.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:36.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:36.092 INFO:teuthology.orchestra.run.smithi150.stderr:set 9.6 pg_upmap_items mapping to [0->1,3->4,6->7] 2024-09-11T08:21:36.145 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:36.171 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:36.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:36.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:36.273 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:21:36.301 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:36.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:36.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:21:36.403 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:21:36.428 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:36.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:36.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:21:36.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:36 smithi150 ceph-mon[29276]: pgmap v637: 148 pgs: 148 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-11T08:21:36.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:36 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3421100269' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "9.6", "id": [0, 1, 3, 4, 6, 7]}]: dispatch 2024-09-11T08:21:36.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:36 smithi150 ceph-mon[39612]: pgmap v637: 148 pgs: 148 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-11T08:21:36.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:36 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3421100269' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "9.6", "id": [0, 1, 3, 4, 6, 7]}]: dispatch 2024-09-11T08:21:36.530 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:36.555 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:36.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:36.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:36.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:36 smithi181 ceph-mon[32701]: pgmap v637: 148 pgs: 148 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-11T08:21:36.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:36 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3421100269' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "9.6", "id": [0, 1, 3, 4, 6, 7]}]: dispatch 2024-09-11T08:21:36.658 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:21:36.683 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:36.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:36.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:21:36.785 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:36.807 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:36.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:36.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:36.909 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:21:36.935 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:36.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:36.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:21:37.036 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:37.061 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:37.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:37.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:37.108 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'unique_pool_2' removed 2024-09-11T08:21:37.163 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:37.185 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:37.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:37.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:37.287 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:37.311 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:37.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:37.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:37.328 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:37 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3421100269' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "9.6", "id": [0, 1, 3, 4, 6, 7]}]': finished 2024-09-11T08:21:37.328 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:37 smithi150 ceph-mon[29276]: osdmap e304: 8 total, 8 up, 8 in 2024-09-11T08:21:37.328 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:37 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/571044771' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-11T08:21:37.328 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:37 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-11T08:21:37.328 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:37 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:21:37.329 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:37 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3421100269' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "9.6", "id": [0, 1, 3, 4, 6, 7]}]': finished 2024-09-11T08:21:37.329 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:37 smithi150 ceph-mon[39612]: osdmap e304: 8 total, 8 up, 8 in 2024-09-11T08:21:37.329 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:37 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/571044771' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-11T08:21:37.329 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:37 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-11T08:21:37.329 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:37 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:21:37.413 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:21:37.434 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:37.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:37.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:21:37.536 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:37.558 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:37.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:37.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:37.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:37 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3421100269' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "9.6", "id": [0, 1, 3, 4, 6, 7]}]': finished 2024-09-11T08:21:37.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:37 smithi181 ceph-mon[32701]: osdmap e304: 8 total, 8 up, 8 in 2024-09-11T08:21:37.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:37 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/571044771' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-11T08:21:37.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:37 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-11T08:21:37.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:37 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:21:37.660 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:37.681 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:37.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:37.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:37.783 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:37.805 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:37.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:37.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:37.906 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:37.930 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:37.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:37.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:38.031 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:38.054 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:38.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:38.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:38.116 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-11T08:21:38.156 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:21:38.179 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:38.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:38.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:21:38.281 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:21:38.304 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:38.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:38.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:21:38.406 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:38.428 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:38.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:38.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:38.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:38 smithi150 ceph-mon[29276]: pgmap v639: 148 pgs: 148 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-11T08:21:38.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:38 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-11T08:21:38.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:38 smithi150 ceph-mon[29276]: osdmap e305: 8 total, 8 up, 8 in 2024-09-11T08:21:38.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:38 smithi150 ceph-mon[39612]: pgmap v639: 148 pgs: 148 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-11T08:21:38.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:38 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-11T08:21:38.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:38 smithi150 ceph-mon[39612]: osdmap e305: 8 total, 8 up, 8 in 2024-09-11T08:21:38.529 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:21:38.533 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:38 smithi181 ceph-mon[32701]: pgmap v639: 148 pgs: 148 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-11T08:21:38.533 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:38 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-11T08:21:38.533 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:38 smithi181 ceph-mon[32701]: osdmap e305: 8 total, 8 up, 8 in 2024-09-11T08:21:38.553 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:38.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:38.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:21:38.656 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:38.678 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:38.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:38.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:38.779 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:21:38.801 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:38.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:38.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:21:38.823 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:21:38 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:21:38.839 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:21:38 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:21:38] "GET /metrics HTTP/1.1" 200 224392 "" "Prometheus/2.43.0" 2024-09-11T08:21:38.903 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:38.925 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:38.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:38.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:39.026 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:39.048 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:39.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:39.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:39.150 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:21:39.164 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-11T08:21:39.165 INFO:tasks.radosbench:Beginning radosbench... 2024-09-11T08:21:39.166 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-09-11T08:21:39.166 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd pool create unique_pool_3 16 2024-09-11T08:21:39.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:39.173 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:39.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:21:39.274 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:39.296 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:39.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:39.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:39.398 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:39.419 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:39.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:39.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:39.521 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:39.542 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:39.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:39.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:39.644 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:21:39.665 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:39.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:39.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:21:39.767 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:39.788 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:39.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:39.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:39.890 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:39.912 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:39.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:39.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:40.013 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:40.034 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:40.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:40.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:40.136 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:40.158 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:40.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:40.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:40.260 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:40.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:40.282 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:40.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:40.294 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:40 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:21:40.294 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:40 smithi150 ceph-mon[39612]: pgmap v641: 132 pgs: 132 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-11T08:21:40.295 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:40 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:21:40.295 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:40 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:21:40.295 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:40 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:21:40.295 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:40 smithi150 ceph-mon[29276]: pgmap v641: 132 pgs: 132 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-11T08:21:40.295 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:40 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:21:40.295 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:40 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:21:40.297 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:21:40.383 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:40.405 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:40.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:40.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:40.487 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:21:40.507 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:21:40.528 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:40.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:40.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:21:40.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:40 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:21:40.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:40 smithi181 ceph-mon[32701]: pgmap v641: 132 pgs: 132 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-11T08:21:40.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:40 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:21:40.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:40 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:21:40.631 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:21:40.652 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:40.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:40.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:21:40.691 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:21:40.692 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:21:40.698 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:21:40.698 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:21:40.713 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:21:40.713 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:21:40.728 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:21:40.728 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:21:40.744 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:21:40.744 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:21:40.754 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:40.759 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:21:40.759 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:21:40.775 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:21:40.775 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:21:40.775 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:40.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:40.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:40.790 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:21:40.791 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:21:40.877 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:21:40.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:40.900 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:40.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:21:41.001 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:41.023 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:41.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:41.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:41.125 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:41.146 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:41.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:41.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:41.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:41 smithi150 ceph-mon[29276]: osdmap e306: 8 total, 8 up, 8 in 2024-09-11T08:21:41.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:41 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2382434552' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:21:41.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:41 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:21:41.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:41 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1204320498' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:21:41.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:41 smithi150 ceph-mon[39612]: osdmap e306: 8 total, 8 up, 8 in 2024-09-11T08:21:41.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:41 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2382434552' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:21:41.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:41 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:21:41.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:41 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1204320498' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:21:41.248 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:21:41.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:41.271 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:41.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:21:41.372 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:41.394 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:41.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:41.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:41.496 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:41.519 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:41.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:41.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:41.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:41 smithi181 ceph-mon[32701]: osdmap e306: 8 total, 8 up, 8 in 2024-09-11T08:21:41.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:41 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2382434552' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:21:41.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:41 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:21:41.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:41 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1204320498' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:21:41.621 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:41.633 INFO:teuthology.orchestra.run.smithi150.stderr:noscrub is unset 2024-09-11T08:21:41.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:41.645 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:41.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:41.746 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:41.768 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:41.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:41.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:41.870 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:21:41.892 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:41.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:41.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:21:41.994 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:42.018 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:42.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:42.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:42.026 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-11T08:21:42.026 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T08:21:42.119 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:42.142 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:42.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:42.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:42.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:42 smithi150 ceph-mon[29276]: pgmap v643: 132 pgs: 132 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-11T08:21:42.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:42 smithi150 ceph-mon[29276]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:21:42.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:42 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:21:42.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:42 smithi150 ceph-mon[29276]: osdmap e307: 8 total, 8 up, 8 in 2024-09-11T08:21:42.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:42 smithi150 ceph-mon[39612]: pgmap v643: 132 pgs: 132 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-11T08:21:42.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:42 smithi150 ceph-mon[39612]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:21:42.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:42 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:21:42.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:42 smithi150 ceph-mon[39612]: osdmap e307: 8 total, 8 up, 8 in 2024-09-11T08:21:42.244 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:42.266 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:42.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:42.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:42.367 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:42.389 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:42.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:42.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:42.491 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:42.517 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:42.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:42.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:42.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:42 smithi181 ceph-mon[32701]: pgmap v643: 132 pgs: 132 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-11T08:21:42.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:42 smithi181 ceph-mon[32701]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:21:42.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:42 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:21:42.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:42 smithi181 ceph-mon[32701]: osdmap e307: 8 total, 8 up, 8 in 2024-09-11T08:21:42.620 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:42.645 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:42.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:42.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:42.747 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:42.769 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:42.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:42.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:42.871 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:21:42.896 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:42.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:42.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:21:42.998 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:43.024 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:43.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:43.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:43.126 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:21:43.152 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:43.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:43.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:21:43.254 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:43.279 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:43.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:43.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:43.381 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:21:43.405 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:43.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:43.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:21:43.507 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:21:43.528 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:43.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:43.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:21:43.630 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:43.655 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:43.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:43.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:43.665 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T08:21:43.757 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:21:43.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:43.779 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:43.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:21:43.880 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:43.906 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:43.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:43.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:44.008 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:21:44.033 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:44.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:44.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:21:44.135 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:44.156 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:44.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:44.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:44.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:44 smithi150 ceph-mon[29276]: pgmap v645: 132 pgs: 132 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-11T08:21:44.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:44 smithi150 ceph-mon[29276]: 6.14 scrub starts 2024-09-11T08:21:44.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:44 smithi150 ceph-mon[29276]: 6.14 scrub ok 2024-09-11T08:21:44.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:44 smithi150 ceph-mon[29276]: 6.c scrub starts 2024-09-11T08:21:44.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:44 smithi150 ceph-mon[29276]: 6.c scrub ok 2024-09-11T08:21:44.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:44 smithi150 ceph-mon[29276]: 3.15 scrub starts 2024-09-11T08:21:44.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:44 smithi150 ceph-mon[29276]: 3.15 scrub ok 2024-09-11T08:21:44.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:44 smithi150 ceph-mon[39612]: pgmap v645: 132 pgs: 132 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-11T08:21:44.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:44 smithi150 ceph-mon[39612]: 6.14 scrub starts 2024-09-11T08:21:44.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:44 smithi150 ceph-mon[39612]: 6.14 scrub ok 2024-09-11T08:21:44.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:44 smithi150 ceph-mon[39612]: 6.c scrub starts 2024-09-11T08:21:44.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:44 smithi150 ceph-mon[39612]: 6.c scrub ok 2024-09-11T08:21:44.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:44 smithi150 ceph-mon[39612]: 3.15 scrub starts 2024-09-11T08:21:44.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:44 smithi150 ceph-mon[39612]: 3.15 scrub ok 2024-09-11T08:21:44.260 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:44.284 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:44.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:44.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:44.386 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:44.410 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:44.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:44.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:44.512 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:21:44.533 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:44.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:44.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:21:44.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:44 smithi181 ceph-mon[32701]: pgmap v645: 132 pgs: 132 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-11T08:21:44.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:44 smithi181 ceph-mon[32701]: 6.14 scrub starts 2024-09-11T08:21:44.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:44 smithi181 ceph-mon[32701]: 6.14 scrub ok 2024-09-11T08:21:44.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:44 smithi181 ceph-mon[32701]: 6.c scrub starts 2024-09-11T08:21:44.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:44 smithi181 ceph-mon[32701]: 6.c scrub ok 2024-09-11T08:21:44.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:44 smithi181 ceph-mon[32701]: 3.15 scrub starts 2024-09-11T08:21:44.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:44 smithi181 ceph-mon[32701]: 3.15 scrub ok 2024-09-11T08:21:44.635 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:44.661 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:44.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:44.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:44.763 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:44.788 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:44.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:44.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:44.889 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:44.913 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:44.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:44.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:44.980 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T08:21:45.065 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:45.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:45.090 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:45.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:45.163 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-11T08:21:45.163 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [], 2024-09-11T08:21:45.163 INFO:teuthology.orchestra.run.smithi150.stdout: "complaint_time": 30, 2024-09-11T08:21:45.163 INFO:teuthology.orchestra.run.smithi150.stdout: "num_blocked_ops": 0 2024-09-11T08:21:45.163 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-11T08:21:45.191 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:21:45.213 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:45.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:45.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:21:45.314 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:21:45.337 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:45.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:45.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:21:45.439 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:45.449 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:45 smithi150 ceph-mon[29276]: 6.b scrub starts 2024-09-11T08:21:45.449 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:45 smithi150 ceph-mon[29276]: 6.b scrub ok 2024-09-11T08:21:45.449 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:45 smithi150 ceph-mon[29276]: 5.12 scrub starts 2024-09-11T08:21:45.449 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:45 smithi150 ceph-mon[29276]: 5.12 scrub ok 2024-09-11T08:21:45.449 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:45 smithi150 ceph-mon[29276]: 4.14 scrub starts 2024-09-11T08:21:45.449 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:45 smithi150 ceph-mon[29276]: 4.14 scrub ok 2024-09-11T08:21:45.449 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:45 smithi150 ceph-mon[29276]: 4.a scrub starts 2024-09-11T08:21:45.449 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:45 smithi150 ceph-mon[29276]: 4.a scrub ok 2024-09-11T08:21:45.449 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:45 smithi150 ceph-mon[39612]: 6.b scrub starts 2024-09-11T08:21:45.450 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:45 smithi150 ceph-mon[39612]: 6.b scrub ok 2024-09-11T08:21:45.450 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:45 smithi150 ceph-mon[39612]: 5.12 scrub starts 2024-09-11T08:21:45.450 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:45 smithi150 ceph-mon[39612]: 5.12 scrub ok 2024-09-11T08:21:45.450 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:45 smithi150 ceph-mon[39612]: 4.14 scrub starts 2024-09-11T08:21:45.450 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:45 smithi150 ceph-mon[39612]: 4.14 scrub ok 2024-09-11T08:21:45.450 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:45 smithi150 ceph-mon[39612]: 4.a scrub starts 2024-09-11T08:21:45.450 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:45 smithi150 ceph-mon[39612]: 4.a scrub ok 2024-09-11T08:21:45.463 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:45.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:45.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:45.563 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:21:45.565 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:21:45.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:21:45 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:21:45] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:21:45.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:45 smithi181 ceph-mon[32701]: 6.b scrub starts 2024-09-11T08:21:45.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:45 smithi181 ceph-mon[32701]: 6.b scrub ok 2024-09-11T08:21:45.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:45 smithi181 ceph-mon[32701]: 5.12 scrub starts 2024-09-11T08:21:45.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:45 smithi181 ceph-mon[32701]: 5.12 scrub ok 2024-09-11T08:21:45.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:45 smithi181 ceph-mon[32701]: 4.14 scrub starts 2024-09-11T08:21:45.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:45 smithi181 ceph-mon[32701]: 4.14 scrub ok 2024-09-11T08:21:45.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:45 smithi181 ceph-mon[32701]: 4.a scrub starts 2024-09-11T08:21:45.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:45 smithi181 ceph-mon[32701]: 4.a scrub ok 2024-09-11T08:21:45.588 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:45.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:45.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:21:45.690 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:45.713 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:45.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:45.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:45.815 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:45.838 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:45.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:45.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:45.940 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:45.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:45.965 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:45.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:46.066 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:46.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:46.092 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:46.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:46.128 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'unique_pool_3' created 2024-09-11T08:21:46.192 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:46.219 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:46.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:46.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:46.321 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:21:46.343 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:46.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:46.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:21:46.444 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:21:46.466 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:46.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:46.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:21:46.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:46 smithi150 ceph-mon[29276]: 4.3 scrub starts 2024-09-11T08:21:46.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:46 smithi150 ceph-mon[29276]: 4.3 scrub ok 2024-09-11T08:21:46.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:46 smithi150 ceph-mon[29276]: 4.12 scrub starts 2024-09-11T08:21:46.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:46 smithi150 ceph-mon[29276]: 4.12 scrub ok 2024-09-11T08:21:46.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:46 smithi150 ceph-mon[29276]: pgmap v646: 132 pgs: 132 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-11T08:21:46.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:46 smithi150 ceph-mon[29276]: 6.5 scrub starts 2024-09-11T08:21:46.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:46 smithi150 ceph-mon[29276]: 6.5 scrub ok 2024-09-11T08:21:46.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:46 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:21:46.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:46 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:21:46.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:46 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:21:46.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:46 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:21:46.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:46 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3854033535' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-09-11T08:21:46.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:46 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-09-11T08:21:46.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:46 smithi150 ceph-mon[39612]: 4.3 scrub starts 2024-09-11T08:21:46.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:46 smithi150 ceph-mon[39612]: 4.3 scrub ok 2024-09-11T08:21:46.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:46 smithi150 ceph-mon[39612]: 4.12 scrub starts 2024-09-11T08:21:46.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:46 smithi150 ceph-mon[39612]: 4.12 scrub ok 2024-09-11T08:21:46.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:46 smithi150 ceph-mon[39612]: pgmap v646: 132 pgs: 132 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-11T08:21:46.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:46 smithi150 ceph-mon[39612]: 6.5 scrub starts 2024-09-11T08:21:46.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:46 smithi150 ceph-mon[39612]: 6.5 scrub ok 2024-09-11T08:21:46.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:46 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:21:46.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:46 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:21:46.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:46 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:21:46.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:46 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:21:46.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:46 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3854033535' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-09-11T08:21:46.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:46 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-09-11T08:21:46.568 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:21:46.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:46 smithi181 ceph-mon[32701]: 4.3 scrub starts 2024-09-11T08:21:46.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:46 smithi181 ceph-mon[32701]: 4.3 scrub ok 2024-09-11T08:21:46.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:46 smithi181 ceph-mon[32701]: 4.12 scrub starts 2024-09-11T08:21:46.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:46 smithi181 ceph-mon[32701]: 4.12 scrub ok 2024-09-11T08:21:46.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:46 smithi181 ceph-mon[32701]: pgmap v646: 132 pgs: 132 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-11T08:21:46.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:46 smithi181 ceph-mon[32701]: 6.5 scrub starts 2024-09-11T08:21:46.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:46 smithi181 ceph-mon[32701]: 6.5 scrub ok 2024-09-11T08:21:46.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:46 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:21:46.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:46 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:21:46.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:46 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:21:46.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:46 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:21:46.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:46 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3854033535' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-09-11T08:21:46.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:46 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-09-11T08:21:46.594 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:46.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:46.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:21:46.696 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:46.717 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:46.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:46.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:46.819 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:46.845 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:46.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:46.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:46.947 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:46.970 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:46.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:46.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:46.983 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-11T08:21:47.072 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:47.095 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:47.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:47.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:47.196 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:21:47.222 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:47.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:47.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:21:47.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:47 smithi150 ceph-mon[29276]: 6.9 scrub starts 2024-09-11T08:21:47.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:47 smithi150 ceph-mon[29276]: 6.9 scrub ok 2024-09-11T08:21:47.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:47 smithi150 ceph-mon[29276]: 3.3 scrub starts 2024-09-11T08:21:47.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:47 smithi150 ceph-mon[29276]: 3.3 scrub ok 2024-09-11T08:21:47.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:47 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-11T08:21:47.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:47 smithi150 ceph-mon[29276]: osdmap e308: 8 total, 8 up, 8 in 2024-09-11T08:21:47.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:47 smithi150 ceph-mon[29276]: 3.f scrub starts 2024-09-11T08:21:47.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:47 smithi150 ceph-mon[29276]: 3.f scrub ok 2024-09-11T08:21:47.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:47 smithi150 ceph-mon[39612]: 6.9 scrub starts 2024-09-11T08:21:47.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:47 smithi150 ceph-mon[39612]: 6.9 scrub ok 2024-09-11T08:21:47.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:47 smithi150 ceph-mon[39612]: 3.3 scrub starts 2024-09-11T08:21:47.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:47 smithi150 ceph-mon[39612]: 3.3 scrub ok 2024-09-11T08:21:47.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:47 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-11T08:21:47.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:47 smithi150 ceph-mon[39612]: osdmap e308: 8 total, 8 up, 8 in 2024-09-11T08:21:47.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:47 smithi150 ceph-mon[39612]: 3.f scrub starts 2024-09-11T08:21:47.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:47 smithi150 ceph-mon[39612]: 3.f scrub ok 2024-09-11T08:21:47.324 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:47.350 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:47.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:47.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:47.452 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:47.463 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-09-11T08:21:47.478 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:47.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:47.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:47.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:47 smithi181 ceph-mon[32701]: 6.9 scrub starts 2024-09-11T08:21:47.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:47 smithi181 ceph-mon[32701]: 6.9 scrub ok 2024-09-11T08:21:47.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:47 smithi181 ceph-mon[32701]: 3.3 scrub starts 2024-09-11T08:21:47.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:47 smithi181 ceph-mon[32701]: 3.3 scrub ok 2024-09-11T08:21:47.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:47 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-11T08:21:47.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:47 smithi181 ceph-mon[32701]: osdmap e308: 8 total, 8 up, 8 in 2024-09-11T08:21:47.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:47 smithi181 ceph-mon[32701]: 3.f scrub starts 2024-09-11T08:21:47.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:47 smithi181 ceph-mon[32701]: 3.f scrub ok 2024-09-11T08:21:47.579 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:21:47.601 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:47.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:47.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:21:47.702 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:47.724 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:47.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:47.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:47.826 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:47.852 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:47.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:47.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:47.953 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:47.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:47.980 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:47.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:48.081 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:48.105 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:48.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:48.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:48.207 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:48 smithi150 ceph-mon[39612]: 4.1e scrub starts 2024-09-11T08:21:48.207 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:48 smithi150 ceph-mon[39612]: 4.1e scrub ok 2024-09-11T08:21:48.207 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:48 smithi150 ceph-mon[39612]: 6.13 scrub starts 2024-09-11T08:21:48.207 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:48 smithi150 ceph-mon[39612]: 6.13 scrub ok 2024-09-11T08:21:48.207 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:48 smithi150 ceph-mon[39612]: 5.1d scrub starts 2024-09-11T08:21:48.208 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:48 smithi150 ceph-mon[39612]: 5.1d scrub ok 2024-09-11T08:21:48.208 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:48 smithi150 ceph-mon[39612]: pgmap v648: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 741 B/s rd, 0 op/s 2024-09-11T08:21:48.208 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:48 smithi150 ceph-mon[39612]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-11T08:21:48.208 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:48 smithi150 ceph-mon[39612]: osdmap e309: 8 total, 8 up, 8 in 2024-09-11T08:21:48.208 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:48 smithi150 ceph-mon[39612]: 5.3 scrub starts 2024-09-11T08:21:48.208 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:48 smithi150 ceph-mon[39612]: 5.3 scrub ok 2024-09-11T08:21:48.208 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:48 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:21:48.208 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:48 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:21:48.208 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:48 smithi150 ceph-mon[29276]: 4.1e scrub starts 2024-09-11T08:21:48.208 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:48 smithi150 ceph-mon[29276]: 4.1e scrub ok 2024-09-11T08:21:48.208 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:48 smithi150 ceph-mon[29276]: 6.13 scrub starts 2024-09-11T08:21:48.208 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:48 smithi150 ceph-mon[29276]: 6.13 scrub ok 2024-09-11T08:21:48.209 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:48 smithi150 ceph-mon[29276]: 5.1d scrub starts 2024-09-11T08:21:48.209 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:48 smithi150 ceph-mon[29276]: 5.1d scrub ok 2024-09-11T08:21:48.209 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:48 smithi150 ceph-mon[29276]: pgmap v648: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 741 B/s rd, 0 op/s 2024-09-11T08:21:48.209 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:48 smithi150 ceph-mon[29276]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-11T08:21:48.209 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:48 smithi150 ceph-mon[29276]: osdmap e309: 8 total, 8 up, 8 in 2024-09-11T08:21:48.209 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:48 smithi150 ceph-mon[29276]: 5.3 scrub starts 2024-09-11T08:21:48.209 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:48 smithi150 ceph-mon[29276]: 5.3 scrub ok 2024-09-11T08:21:48.209 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:48 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:21:48.209 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:48 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:21:48.209 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:48.233 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:48.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:48.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:48.335 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:48.358 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:48.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:48.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:48.450 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:21:48.450 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:21:48.458 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:21:48.459 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:21:48.460 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:48.466 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:21:48.467 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:21:48.474 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:48 smithi181 ceph-mon[32701]: 4.1e scrub starts 2024-09-11T08:21:48.474 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:48 smithi181 ceph-mon[32701]: 4.1e scrub ok 2024-09-11T08:21:48.474 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:48 smithi181 ceph-mon[32701]: 6.13 scrub starts 2024-09-11T08:21:48.474 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:48 smithi181 ceph-mon[32701]: 6.13 scrub ok 2024-09-11T08:21:48.474 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:48 smithi181 ceph-mon[32701]: 5.1d scrub starts 2024-09-11T08:21:48.475 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:48 smithi181 ceph-mon[32701]: 5.1d scrub ok 2024-09-11T08:21:48.475 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:48 smithi181 ceph-mon[32701]: pgmap v648: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 741 B/s rd, 0 op/s 2024-09-11T08:21:48.475 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:48 smithi181 ceph-mon[32701]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-11T08:21:48.475 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:48 smithi181 ceph-mon[32701]: osdmap e309: 8 total, 8 up, 8 in 2024-09-11T08:21:48.475 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:48 smithi181 ceph-mon[32701]: 5.3 scrub starts 2024-09-11T08:21:48.475 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:48 smithi181 ceph-mon[32701]: 5.3 scrub ok 2024-09-11T08:21:48.475 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:48 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:21:48.475 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:48 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:21:48.484 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:48.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:48.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:48.487 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:21:48.487 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:21:48.504 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:21:48.504 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:21:48.519 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:21:48.520 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:21:48.535 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:21:48.535 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:21:48.550 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:21:48.550 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:21:48.565 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:21:48.566 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:21:48.585 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:21:48.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:48.609 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:48.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:21:48.710 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:21:48.733 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:48.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:48.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:21:48.737 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:21:48 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:21:48] "GET /metrics HTTP/1.1" 200 224392 "" "Prometheus/2.43.0" 2024-09-11T08:21:48.761 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:21:48 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:21:48.835 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:21:48.857 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:48.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:48.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:21:48.960 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:21:48.982 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:48.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:48.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:21:49.083 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:21:49.105 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:49.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:49.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:21:49.207 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:49.228 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:49.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:49.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:49.330 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:49.351 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:49.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:49.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:49.445 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:21:49.452 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:21:49.474 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:49.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:49.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:21:49.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:49 smithi150 ceph-mon[29276]: 3.4 scrub starts 2024-09-11T08:21:49.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:49 smithi150 ceph-mon[29276]: 3.4 scrub ok 2024-09-11T08:21:49.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:49 smithi150 ceph-mon[29276]: osdmap e310: 8 total, 8 up, 8 in 2024-09-11T08:21:49.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:49 smithi150 ceph-mon[29276]: 4.19 scrub starts 2024-09-11T08:21:49.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:49 smithi150 ceph-mon[29276]: 4.19 scrub ok 2024-09-11T08:21:49.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:49 smithi150 ceph-mon[29276]: 3.1c scrub starts 2024-09-11T08:21:49.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:49 smithi150 ceph-mon[29276]: 3.1c scrub ok 2024-09-11T08:21:49.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:49 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2228523022' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:21:49.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:49 smithi150 ceph-mon[29276]: 1.0 scrub starts 2024-09-11T08:21:49.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:49 smithi150 ceph-mon[29276]: 1.0 scrub ok 2024-09-11T08:21:49.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:49 smithi150 ceph-mon[39612]: 3.4 scrub starts 2024-09-11T08:21:49.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:49 smithi150 ceph-mon[39612]: 3.4 scrub ok 2024-09-11T08:21:49.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:49 smithi150 ceph-mon[39612]: osdmap e310: 8 total, 8 up, 8 in 2024-09-11T08:21:49.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:49 smithi150 ceph-mon[39612]: 4.19 scrub starts 2024-09-11T08:21:49.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:49 smithi150 ceph-mon[39612]: 4.19 scrub ok 2024-09-11T08:21:49.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:49 smithi150 ceph-mon[39612]: 3.1c scrub starts 2024-09-11T08:21:49.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:49 smithi150 ceph-mon[39612]: 3.1c scrub ok 2024-09-11T08:21:49.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:49 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2228523022' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:21:49.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:49 smithi150 ceph-mon[39612]: 1.0 scrub starts 2024-09-11T08:21:49.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:49 smithi150 ceph-mon[39612]: 1.0 scrub ok 2024-09-11T08:21:49.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:49 smithi181 ceph-mon[32701]: 3.4 scrub starts 2024-09-11T08:21:49.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:49 smithi181 ceph-mon[32701]: 3.4 scrub ok 2024-09-11T08:21:49.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:49 smithi181 ceph-mon[32701]: osdmap e310: 8 total, 8 up, 8 in 2024-09-11T08:21:49.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:49 smithi181 ceph-mon[32701]: 4.19 scrub starts 2024-09-11T08:21:49.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:49 smithi181 ceph-mon[32701]: 4.19 scrub ok 2024-09-11T08:21:49.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:49 smithi181 ceph-mon[32701]: 3.1c scrub starts 2024-09-11T08:21:49.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:49 smithi181 ceph-mon[32701]: 3.1c scrub ok 2024-09-11T08:21:49.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:49 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2228523022' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:21:49.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:49 smithi181 ceph-mon[32701]: 1.0 scrub starts 2024-09-11T08:21:49.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:49 smithi181 ceph-mon[32701]: 1.0 scrub ok 2024-09-11T08:21:49.576 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:49.597 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:49.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:49.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:49.699 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:21:49.720 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:49.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:49.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:21:49.822 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:21:49.843 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:49.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:49.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:21:49.945 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:49.965 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:49.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:49.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:50.067 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:50.088 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:50.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:50.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:50.180 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:50 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:21:50.181 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:50 smithi150 ceph-mon[39612]: 5.5 scrub starts 2024-09-11T08:21:50.181 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:50 smithi150 ceph-mon[39612]: 5.5 scrub ok 2024-09-11T08:21:50.181 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:50 smithi150 ceph-mon[39612]: pgmap v651: 148 pgs: 148 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-11T08:21:50.181 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:50 smithi150 ceph-mon[39612]: 2.2 scrub starts 2024-09-11T08:21:50.181 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:50 smithi150 ceph-mon[39612]: 2.2 scrub ok 2024-09-11T08:21:50.181 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:50 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:21:50.181 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:50 smithi150 ceph-mon[29276]: 5.5 scrub starts 2024-09-11T08:21:50.181 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:50 smithi150 ceph-mon[29276]: 5.5 scrub ok 2024-09-11T08:21:50.181 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:50 smithi150 ceph-mon[29276]: pgmap v651: 148 pgs: 148 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-11T08:21:50.181 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:50 smithi150 ceph-mon[29276]: 2.2 scrub starts 2024-09-11T08:21:50.182 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:50 smithi150 ceph-mon[29276]: 2.2 scrub ok 2024-09-11T08:21:50.190 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:50.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:50.212 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:50.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:50.313 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:50.335 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:50.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:50.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:50.437 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:50.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:50.459 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:50.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:50.560 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:50.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:50 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:21:50.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:50 smithi181 ceph-mon[32701]: 5.5 scrub starts 2024-09-11T08:21:50.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:50 smithi181 ceph-mon[32701]: 5.5 scrub ok 2024-09-11T08:21:50.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:50 smithi181 ceph-mon[32701]: pgmap v651: 148 pgs: 148 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-11T08:21:50.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:50 smithi181 ceph-mon[32701]: 2.2 scrub starts 2024-09-11T08:21:50.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:50 smithi181 ceph-mon[32701]: 2.2 scrub ok 2024-09-11T08:21:50.581 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:50.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:50.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:50.683 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:50.706 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:50.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:50.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:50.807 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:21:50.829 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:50.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:50.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:21:50.931 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:21:50.952 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:50.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:50.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:21:51.054 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:51.078 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:51.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:51.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:51.180 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:21:51.201 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:51.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:51.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:21:51.303 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:21:51.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:51.326 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:51.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:21:51.427 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:51.448 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:51.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:51.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:51.469 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:51 smithi150 ceph-mon[29276]: 4.1d scrub starts 2024-09-11T08:21:51.469 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:51 smithi150 ceph-mon[29276]: 4.1d scrub ok 2024-09-11T08:21:51.470 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:51 smithi150 ceph-mon[29276]: 5.f scrub starts 2024-09-11T08:21:51.470 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:51 smithi150 ceph-mon[29276]: 5.f scrub ok 2024-09-11T08:21:51.470 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:51 smithi150 ceph-mon[29276]: 2.0 scrub starts 2024-09-11T08:21:51.470 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:51 smithi150 ceph-mon[29276]: 2.0 scrub ok 2024-09-11T08:21:51.470 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:51 smithi150 ceph-mon[39612]: 4.1d scrub starts 2024-09-11T08:21:51.470 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:51 smithi150 ceph-mon[39612]: 4.1d scrub ok 2024-09-11T08:21:51.470 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:51 smithi150 ceph-mon[39612]: 5.f scrub starts 2024-09-11T08:21:51.470 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:51 smithi150 ceph-mon[39612]: 5.f scrub ok 2024-09-11T08:21:51.470 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:51 smithi150 ceph-mon[39612]: 2.0 scrub starts 2024-09-11T08:21:51.470 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:51 smithi150 ceph-mon[39612]: 2.0 scrub ok 2024-09-11T08:21:51.550 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:51.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:51 smithi181 ceph-mon[32701]: 4.1d scrub starts 2024-09-11T08:21:51.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:51 smithi181 ceph-mon[32701]: 4.1d scrub ok 2024-09-11T08:21:51.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:51 smithi181 ceph-mon[32701]: 5.f scrub starts 2024-09-11T08:21:51.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:51 smithi181 ceph-mon[32701]: 5.f scrub ok 2024-09-11T08:21:51.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:51 smithi181 ceph-mon[32701]: 2.0 scrub starts 2024-09-11T08:21:51.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:51 smithi181 ceph-mon[32701]: 2.0 scrub ok 2024-09-11T08:21:51.576 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:51.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:51.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:51.678 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:21:51.700 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:51.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:51.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:21:51.723 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-09-11T08:21:51.723 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:21:51.802 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:51.828 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:51.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:51.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:51.930 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:51.937 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T08:21:51.955 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:51.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:51.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:52.057 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:21:52.079 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:52.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:52.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:21:52.180 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:52.203 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:52.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:52.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:52.305 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:21:52.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:52.328 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:52.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:21:52.429 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:52.450 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:52.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:52.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:52.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:52 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:21:52.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:52 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:21:52.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:52 smithi150 ceph-mon[29276]: pgmap v652: 148 pgs: 148 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:21:52.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:52 smithi150 ceph-mon[29276]: 4.d scrub starts 2024-09-11T08:21:52.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:52 smithi150 ceph-mon[29276]: 4.d scrub ok 2024-09-11T08:21:52.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:52 smithi150 ceph-mon[29276]: osdmap e311: 8 total, 8 up, 8 in 2024-09-11T08:21:52.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:52 smithi150 ceph-mon[29276]: 6.1f scrub starts 2024-09-11T08:21:52.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:52 smithi150 ceph-mon[29276]: 6.1f scrub ok 2024-09-11T08:21:52.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:52 smithi150 ceph-mon[29276]: 6.3 scrub starts 2024-09-11T08:21:52.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:52 smithi150 ceph-mon[29276]: 6.3 scrub ok 2024-09-11T08:21:52.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:52 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:21:52.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:52 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:21:52.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:52 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:21:52.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:52 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:21:52.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:52 smithi150 ceph-mon[39612]: pgmap v652: 148 pgs: 148 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:21:52.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:52 smithi150 ceph-mon[39612]: 4.d scrub starts 2024-09-11T08:21:52.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:52 smithi150 ceph-mon[39612]: 4.d scrub ok 2024-09-11T08:21:52.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:52 smithi150 ceph-mon[39612]: osdmap e311: 8 total, 8 up, 8 in 2024-09-11T08:21:52.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:52 smithi150 ceph-mon[39612]: 6.1f scrub starts 2024-09-11T08:21:52.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:52 smithi150 ceph-mon[39612]: 6.1f scrub ok 2024-09-11T08:21:52.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:52 smithi150 ceph-mon[39612]: 6.3 scrub starts 2024-09-11T08:21:52.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:52 smithi150 ceph-mon[39612]: 6.3 scrub ok 2024-09-11T08:21:52.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:52 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:21:52.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:52 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:21:52.552 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:21:52.573 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:52.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:52 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:21:52.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:52 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:21:52.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:52 smithi181 ceph-mon[32701]: pgmap v652: 148 pgs: 148 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:21:52.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:52 smithi181 ceph-mon[32701]: 4.d scrub starts 2024-09-11T08:21:52.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:52 smithi181 ceph-mon[32701]: 4.d scrub ok 2024-09-11T08:21:52.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:52 smithi181 ceph-mon[32701]: osdmap e311: 8 total, 8 up, 8 in 2024-09-11T08:21:52.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:52 smithi181 ceph-mon[32701]: 6.1f scrub starts 2024-09-11T08:21:52.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:52 smithi181 ceph-mon[32701]: 6.1f scrub ok 2024-09-11T08:21:52.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:52 smithi181 ceph-mon[32701]: 6.3 scrub starts 2024-09-11T08:21:52.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:52 smithi181 ceph-mon[32701]: 6.3 scrub ok 2024-09-11T08:21:52.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:52 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:21:52.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:52 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:21:52.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:52.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:21:52.676 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:21:52.698 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:52.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:52.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:21:52.755 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-11T08:21:52.800 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:52.821 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:52.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:52.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:52.923 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:52.946 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:52.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:52.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:53.047 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:53.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:53.070 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:53.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:53.142 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:21:53.171 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:53.195 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:53.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:53.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:53.297 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:53.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:53.320 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:53.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:53.421 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:53.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:53.444 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:53.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:53.447 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:53 smithi150 ceph-mon[29276]: 3.1a scrub starts 2024-09-11T08:21:53.448 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:53 smithi150 ceph-mon[29276]: 3.1a scrub ok 2024-09-11T08:21:53.448 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:53 smithi150 ceph-mon[29276]: 5.1f scrub starts 2024-09-11T08:21:53.448 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:53 smithi150 ceph-mon[29276]: 5.1f scrub ok 2024-09-11T08:21:53.448 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:53 smithi150 ceph-mon[29276]: 5.14 scrub starts 2024-09-11T08:21:53.448 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:53 smithi150 ceph-mon[29276]: 5.14 scrub ok 2024-09-11T08:21:53.448 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:53 smithi150 ceph-mon[39612]: 3.1a scrub starts 2024-09-11T08:21:53.448 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:53 smithi150 ceph-mon[39612]: 3.1a scrub ok 2024-09-11T08:21:53.448 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:53 smithi150 ceph-mon[39612]: 5.1f scrub starts 2024-09-11T08:21:53.448 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:53 smithi150 ceph-mon[39612]: 5.1f scrub ok 2024-09-11T08:21:53.449 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:53 smithi150 ceph-mon[39612]: 5.14 scrub starts 2024-09-11T08:21:53.449 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:53 smithi150 ceph-mon[39612]: 5.14 scrub ok 2024-09-11T08:21:53.546 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:53.572 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:53.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:53.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:53.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:53 smithi181 ceph-mon[32701]: 3.1a scrub starts 2024-09-11T08:21:53.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:53 smithi181 ceph-mon[32701]: 3.1a scrub ok 2024-09-11T08:21:53.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:53 smithi181 ceph-mon[32701]: 5.1f scrub starts 2024-09-11T08:21:53.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:53 smithi181 ceph-mon[32701]: 5.1f scrub ok 2024-09-11T08:21:53.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:53 smithi181 ceph-mon[32701]: 5.14 scrub starts 2024-09-11T08:21:53.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:53 smithi181 ceph-mon[32701]: 5.14 scrub ok 2024-09-11T08:21:53.674 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:21:53.696 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:53.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:53.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:21:53.798 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:53.820 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:53.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:53.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:53.921 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:53.947 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:53.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:53.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:54.049 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:54.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:54.075 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:54.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:54.161 INFO:teuthology.orchestra.run.smithi150.stderr:nodeep-scrub is unset 2024-09-11T08:21:54.177 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:54.198 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:54.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:54.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:54.300 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:21:54.322 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:54.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:54.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:21:54.424 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:54.434 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:54 smithi150 ceph-mon[29276]: 6.18 scrub starts 2024-09-11T08:21:54.434 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:54 smithi150 ceph-mon[29276]: 6.18 scrub ok 2024-09-11T08:21:54.434 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:54 smithi150 ceph-mon[29276]: pgmap v654: 148 pgs: 148 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:21:54.434 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:54 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3632355806' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:21:54.434 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:54 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:21:54.434 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:54 smithi150 ceph-mon[29276]: 3.13 scrub starts 2024-09-11T08:21:54.434 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:54 smithi150 ceph-mon[29276]: 3.13 scrub ok 2024-09-11T08:21:54.434 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:54 smithi150 ceph-mon[39612]: 6.18 scrub starts 2024-09-11T08:21:54.435 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:54 smithi150 ceph-mon[39612]: 6.18 scrub ok 2024-09-11T08:21:54.435 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:54 smithi150 ceph-mon[39612]: pgmap v654: 148 pgs: 148 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:21:54.435 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:54 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3632355806' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:21:54.435 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:54 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:21:54.435 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:54 smithi150 ceph-mon[39612]: 3.13 scrub starts 2024-09-11T08:21:54.435 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:54 smithi150 ceph-mon[39612]: 3.13 scrub ok 2024-09-11T08:21:54.446 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:54.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:54.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:54.548 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:54.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:54 smithi181 ceph-mon[32701]: 6.18 scrub starts 2024-09-11T08:21:54.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:54 smithi181 ceph-mon[32701]: 6.18 scrub ok 2024-09-11T08:21:54.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:54 smithi181 ceph-mon[32701]: pgmap v654: 148 pgs: 148 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:21:54.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:54 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3632355806' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:21:54.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:54 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:21:54.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:54 smithi181 ceph-mon[32701]: 3.13 scrub starts 2024-09-11T08:21:54.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:54 smithi181 ceph-mon[32701]: 3.13 scrub ok 2024-09-11T08:21:54.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:54.575 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:54.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:54.676 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:54.702 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:54.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:54.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:54.804 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:54.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:54.831 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:54.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:54.932 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:54.956 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:54.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:54.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:55.058 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:21:55.081 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:55.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:55.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:21:55.183 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:55.209 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:55.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:55.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:55.310 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:55.335 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:55.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:55.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:55.437 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:21:55.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:55.461 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:55.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:21:55.476 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:55 smithi150 ceph-mon[39612]: 4.11 scrub starts 2024-09-11T08:21:55.476 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:55 smithi150 ceph-mon[39612]: 4.11 scrub ok 2024-09-11T08:21:55.476 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:55 smithi150 ceph-mon[39612]: 4.7 scrub starts 2024-09-11T08:21:55.476 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:55 smithi150 ceph-mon[39612]: 4.7 scrub ok 2024-09-11T08:21:55.476 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:55 smithi150 ceph-mon[39612]: 5.1c scrub starts 2024-09-11T08:21:55.476 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:55 smithi150 ceph-mon[39612]: 5.1c scrub ok 2024-09-11T08:21:55.476 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:55 smithi150 ceph-mon[39612]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:21:55.477 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:55 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:21:55.477 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:55 smithi150 ceph-mon[39612]: osdmap e312: 8 total, 8 up, 8 in 2024-09-11T08:21:55.477 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:55 smithi150 ceph-mon[39612]: 5.16 scrub starts 2024-09-11T08:21:55.477 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:55 smithi150 ceph-mon[39612]: 5.16 scrub ok 2024-09-11T08:21:55.477 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:55 smithi150 ceph-mon[29276]: 4.11 scrub starts 2024-09-11T08:21:55.477 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:55 smithi150 ceph-mon[29276]: 4.11 scrub ok 2024-09-11T08:21:55.477 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:55 smithi150 ceph-mon[29276]: 4.7 scrub starts 2024-09-11T08:21:55.477 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:55 smithi150 ceph-mon[29276]: 4.7 scrub ok 2024-09-11T08:21:55.477 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:55 smithi150 ceph-mon[29276]: 5.1c scrub starts 2024-09-11T08:21:55.477 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:55 smithi150 ceph-mon[29276]: 5.1c scrub ok 2024-09-11T08:21:55.477 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:55 smithi150 ceph-mon[29276]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:21:55.478 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:55 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:21:55.478 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:55 smithi150 ceph-mon[29276]: osdmap e312: 8 total, 8 up, 8 in 2024-09-11T08:21:55.478 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:55 smithi150 ceph-mon[29276]: 5.16 scrub starts 2024-09-11T08:21:55.478 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:55 smithi150 ceph-mon[29276]: 5.16 scrub ok 2024-09-11T08:21:55.562 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:55.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:21:55 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:21:55] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:21:55.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:55 smithi181 ceph-mon[32701]: 4.11 scrub starts 2024-09-11T08:21:55.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:55 smithi181 ceph-mon[32701]: 4.11 scrub ok 2024-09-11T08:21:55.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:55 smithi181 ceph-mon[32701]: 4.7 scrub starts 2024-09-11T08:21:55.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:55 smithi181 ceph-mon[32701]: 4.7 scrub ok 2024-09-11T08:21:55.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:55 smithi181 ceph-mon[32701]: 5.1c scrub starts 2024-09-11T08:21:55.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:55 smithi181 ceph-mon[32701]: 5.1c scrub ok 2024-09-11T08:21:55.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:55 smithi181 ceph-mon[32701]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:21:55.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:55 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:21:55.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:55 smithi181 ceph-mon[32701]: osdmap e312: 8 total, 8 up, 8 in 2024-09-11T08:21:55.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:55 smithi181 ceph-mon[32701]: 5.16 scrub starts 2024-09-11T08:21:55.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:55 smithi181 ceph-mon[32701]: 5.16 scrub ok 2024-09-11T08:21:55.587 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:55.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:55.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:55.689 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:55.711 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:55.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:55.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:55.813 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:55.838 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:55.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:55.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:55.847 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:21:55.940 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:21:55.965 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:55.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:55.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:21:56.066 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:21:56.088 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:56.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:56.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:21:56.190 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:56.214 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:56.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:56.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:56.316 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:56.341 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:56.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:56.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:56.443 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:56.467 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:56.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:56.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:56.471 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:56 smithi150 ceph-mon[29276]: 6.11 scrub starts 2024-09-11T08:21:56.471 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:56 smithi150 ceph-mon[29276]: 6.11 scrub ok 2024-09-11T08:21:56.472 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:56 smithi150 ceph-mon[29276]: pgmap v656: 148 pgs: 148 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 744 B/s rd, 0 op/s 2024-09-11T08:21:56.472 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:56 smithi150 ceph-mon[29276]: 3.a scrub starts 2024-09-11T08:21:56.472 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:56 smithi150 ceph-mon[29276]: 3.a scrub ok 2024-09-11T08:21:56.472 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:56 smithi150 ceph-mon[39612]: 6.11 scrub starts 2024-09-11T08:21:56.472 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:56 smithi150 ceph-mon[39612]: 6.11 scrub ok 2024-09-11T08:21:56.472 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:56 smithi150 ceph-mon[39612]: pgmap v656: 148 pgs: 148 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 744 B/s rd, 0 op/s 2024-09-11T08:21:56.472 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:56 smithi150 ceph-mon[39612]: 3.a scrub starts 2024-09-11T08:21:56.472 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:56 smithi150 ceph-mon[39612]: 3.a scrub ok 2024-09-11T08:21:56.567 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:21:56.568 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:21:56.569 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:56.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:56 smithi181 ceph-mon[32701]: 6.11 scrub starts 2024-09-11T08:21:56.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:56 smithi181 ceph-mon[32701]: 6.11 scrub ok 2024-09-11T08:21:56.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:56 smithi181 ceph-mon[32701]: pgmap v656: 148 pgs: 148 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 744 B/s rd, 0 op/s 2024-09-11T08:21:56.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:56 smithi181 ceph-mon[32701]: 3.a scrub starts 2024-09-11T08:21:56.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:56 smithi181 ceph-mon[32701]: 3.a scrub ok 2024-09-11T08:21:56.588 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:21:56.588 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:21:56.591 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:56.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:56.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:56.602 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:21:56.602 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:21:56.617 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:21:56.617 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:21:56.633 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:21:56.633 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:21:56.650 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:21:56.650 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:21:56.665 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:21:56.666 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:21:56.681 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:21:56.681 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:21:56.694 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:21:56.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:56.720 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:56.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:21:56.821 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:21:56.845 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:56.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:56.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:21:56.947 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:56.972 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:56.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:56.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:57.073 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:57.095 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:57.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:57.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:57.197 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:57.218 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:57.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:57.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:57.320 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:21:57.346 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:57.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:57.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:21:57.429 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:57 smithi150 ceph-mon[29276]: 6.4 scrub starts 2024-09-11T08:21:57.429 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:57 smithi150 ceph-mon[29276]: 6.4 scrub ok 2024-09-11T08:21:57.430 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:57 smithi150 ceph-mon[29276]: 4.10 deep-scrub starts 2024-09-11T08:21:57.430 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:57 smithi150 ceph-mon[29276]: 4.10 deep-scrub ok 2024-09-11T08:21:57.430 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:57 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3808383890' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:21:57.430 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:57 smithi150 ceph-mon[39612]: 6.4 scrub starts 2024-09-11T08:21:57.430 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:57 smithi150 ceph-mon[39612]: 6.4 scrub ok 2024-09-11T08:21:57.430 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:57 smithi150 ceph-mon[39612]: 4.10 deep-scrub starts 2024-09-11T08:21:57.430 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:57 smithi150 ceph-mon[39612]: 4.10 deep-scrub ok 2024-09-11T08:21:57.430 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:57 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3808383890' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:21:57.448 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:57.468 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:57.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:57.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:57.570 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:21:57.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:57 smithi181 ceph-mon[32701]: 6.4 scrub starts 2024-09-11T08:21:57.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:57 smithi181 ceph-mon[32701]: 6.4 scrub ok 2024-09-11T08:21:57.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:57 smithi181 ceph-mon[32701]: 4.10 deep-scrub starts 2024-09-11T08:21:57.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:57 smithi181 ceph-mon[32701]: 4.10 deep-scrub ok 2024-09-11T08:21:57.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:57 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3808383890' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:21:57.595 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:57.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:57.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:21:57.697 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:57.722 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:57.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:57.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:57.808 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T08:21:57.824 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:57.849 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:57.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:57.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:57.950 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:57.976 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:57.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:57.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:58.077 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:58.102 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:58.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:58.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:58.205 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:21:58.228 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:58.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:58.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:21:58.329 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:58.356 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:58.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:58.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:58.393 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:58 smithi150 ceph-mon[29276]: 4.2 scrub starts 2024-09-11T08:21:58.393 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:58 smithi150 ceph-mon[29276]: 4.2 scrub ok 2024-09-11T08:21:58.393 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:58 smithi150 ceph-mon[29276]: pgmap v657: 148 pgs: 148 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-11T08:21:58.393 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:58 smithi150 ceph-mon[39612]: 4.2 scrub starts 2024-09-11T08:21:58.393 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:58 smithi150 ceph-mon[39612]: 4.2 scrub ok 2024-09-11T08:21:58.393 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:58 smithi150 ceph-mon[39612]: pgmap v657: 148 pgs: 148 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-11T08:21:58.458 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:21:58.484 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:58.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:58.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:21:58.553 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:58 smithi181 ceph-mon[32701]: 4.2 scrub starts 2024-09-11T08:21:58.553 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:58 smithi181 ceph-mon[32701]: 4.2 scrub ok 2024-09-11T08:21:58.554 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:58 smithi181 ceph-mon[32701]: pgmap v657: 148 pgs: 148 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-11T08:21:58.586 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:58.612 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:58.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:58.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:58.684 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:21:58 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:21:58] "GET /metrics HTTP/1.1" 200 224406 "" "Prometheus/2.43.0" 2024-09-11T08:21:58.714 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:21:58.739 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:58.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:58.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:21:58.823 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:21:58 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:21:58.841 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:21:58.863 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:58.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:58.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:21:58.965 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:58.990 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:58.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:58.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:59.092 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:21:59.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:59.115 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:59.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:21:59.177 INFO:teuthology.orchestra.run.smithi150.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-09-11T08:21:59.216 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:21:59.241 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:59.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:59.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:21:59.343 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:59.366 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:59.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:59.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:59.468 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:21:59.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:59 smithi150 ceph-mon[29276]: 3.10 scrub starts 2024-09-11T08:21:59.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:59 smithi150 ceph-mon[29276]: 3.10 scrub ok 2024-09-11T08:21:59.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:59 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/395680216' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-11T08:21:59.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:21:59 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-11T08:21:59.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:59 smithi150 ceph-mon[39612]: 3.10 scrub starts 2024-09-11T08:21:59.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:59 smithi150 ceph-mon[39612]: 3.10 scrub ok 2024-09-11T08:21:59.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:59 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/395680216' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-11T08:21:59.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:21:59 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-11T08:21:59.490 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:59.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:59.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:21:59.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:59 smithi181 ceph-mon[32701]: 3.10 scrub starts 2024-09-11T08:21:59.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:59 smithi181 ceph-mon[32701]: 3.10 scrub ok 2024-09-11T08:21:59.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:59 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/395680216' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-11T08:21:59.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:21:59 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-11T08:21:59.592 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:59.615 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:59.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:59.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:21:59.717 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:21:59.740 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:59.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:59.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:21:59.841 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:21:59.863 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:21:59.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:59.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:21:59.964 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:21:59.987 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:21:59.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:21:59.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:22:00.023 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:22:00.089 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:22:00.105 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T08:22:00.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:00.111 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:00.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:22:00.212 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:22:00.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:00.238 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:00.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:22:00.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:00 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:22:00.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:00 smithi150 ceph-mon[29276]: pgmap v658: 148 pgs: 148 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:22:00.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:00 smithi150 ceph-mon[29276]: 4.18 deep-scrub starts 2024-09-11T08:22:00.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:00 smithi150 ceph-mon[29276]: 4.18 deep-scrub ok 2024-09-11T08:22:00.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:00 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-11T08:22:00.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:00 smithi150 ceph-mon[29276]: osdmap e313: 8 total, 8 up, 8 in 2024-09-11T08:22:00.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:00 smithi150 ceph-mon[29276]: 5.d scrub starts 2024-09-11T08:22:00.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:00 smithi150 ceph-mon[29276]: 5.d scrub ok 2024-09-11T08:22:00.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:00 smithi150 ceph-mon[29276]: 6.a scrub starts 2024-09-11T08:22:00.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:00 smithi150 ceph-mon[29276]: 6.a scrub ok 2024-09-11T08:22:00.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:00 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:22:00.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:00 smithi150 ceph-mon[39612]: pgmap v658: 148 pgs: 148 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:22:00.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:00 smithi150 ceph-mon[39612]: 4.18 deep-scrub starts 2024-09-11T08:22:00.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:00 smithi150 ceph-mon[39612]: 4.18 deep-scrub ok 2024-09-11T08:22:00.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:00 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-11T08:22:00.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:00 smithi150 ceph-mon[39612]: osdmap e313: 8 total, 8 up, 8 in 2024-09-11T08:22:00.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:00 smithi150 ceph-mon[39612]: 5.d scrub starts 2024-09-11T08:22:00.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:00 smithi150 ceph-mon[39612]: 5.d scrub ok 2024-09-11T08:22:00.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:00 smithi150 ceph-mon[39612]: 6.a scrub starts 2024-09-11T08:22:00.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:00 smithi150 ceph-mon[39612]: 6.a scrub ok 2024-09-11T08:22:00.339 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:22:00.365 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:00.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:00.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:22:00.467 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:22:00.489 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:00.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:00.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:22:00.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:00 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:22:00.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:00 smithi181 ceph-mon[32701]: pgmap v658: 148 pgs: 148 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:22:00.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:00 smithi181 ceph-mon[32701]: 4.18 deep-scrub starts 2024-09-11T08:22:00.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:00 smithi181 ceph-mon[32701]: 4.18 deep-scrub ok 2024-09-11T08:22:00.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:00 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-11T08:22:00.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:00 smithi181 ceph-mon[32701]: osdmap e313: 8 total, 8 up, 8 in 2024-09-11T08:22:00.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:00 smithi181 ceph-mon[32701]: 5.d scrub starts 2024-09-11T08:22:00.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:00 smithi181 ceph-mon[32701]: 5.d scrub ok 2024-09-11T08:22:00.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:00 smithi181 ceph-mon[32701]: 6.a scrub starts 2024-09-11T08:22:00.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:00 smithi181 ceph-mon[32701]: 6.a scrub ok 2024-09-11T08:22:00.591 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:00.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:00.621 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:00.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:00.722 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:22:00.743 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:00.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:00.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:22:00.825 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-11T08:22:00.825 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [], 2024-09-11T08:22:00.825 INFO:teuthology.orchestra.run.smithi150.stdout: "num_ops": 0 2024-09-11T08:22:00.825 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-11T08:22:00.846 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:22:00.867 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:00.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:00.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:22:00.969 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:22:00.974 DEBUG:teuthology.orchestra.run.smithi150:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-09-11T08:22:00.974 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-09-11T08:22:00.974 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-09-11T08:22:00.975 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T08:22:00.977 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-11T08:22:00.977 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-11T08:22:00.995 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:00.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:00.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:22:01.024 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:hints = 1 2024-09-11T08:22:01.024 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-11T08:22:01.024 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Object prefix: benchmark_data_smithi150_158067 2024-09-11T08:22:01.024 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-11T08:22:01.025 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 0 0 0 0 0 0 - 0 2024-09-11T08:22:01.097 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:01.133 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:01.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:01.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:01.235 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:22:01.270 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:01.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:01.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:22:01.371 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:22:01.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:01.408 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:01.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:22:01.509 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:01.542 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:01.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:01.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:01.644 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:01.683 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:01.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:01.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:01.785 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:22:01.821 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:01.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:01.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:22:01.890 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:01 smithi150 ceph-mon[29276]: 5.6 deep-scrub starts 2024-09-11T08:22:01.891 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:01 smithi150 ceph-mon[29276]: 5.6 deep-scrub ok 2024-09-11T08:22:01.891 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:01 smithi150 ceph-mon[29276]: 3.8 deep-scrub starts 2024-09-11T08:22:01.891 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:01 smithi150 ceph-mon[29276]: 3.8 deep-scrub ok 2024-09-11T08:22:01.891 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:01 smithi150 ceph-mon[29276]: pgmap v660: 148 pgs: 148 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:22:01.891 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:01 smithi150 ceph-mon[29276]: 4.1b scrub starts 2024-09-11T08:22:01.891 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:01 smithi150 ceph-mon[29276]: 4.1b scrub ok 2024-09-11T08:22:01.891 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:01 smithi150 ceph-mon[39612]: 5.6 deep-scrub starts 2024-09-11T08:22:01.891 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:01 smithi150 ceph-mon[39612]: 5.6 deep-scrub ok 2024-09-11T08:22:01.891 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:01 smithi150 ceph-mon[39612]: 3.8 deep-scrub starts 2024-09-11T08:22:01.892 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:01 smithi150 ceph-mon[39612]: 3.8 deep-scrub ok 2024-09-11T08:22:01.892 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:01 smithi150 ceph-mon[39612]: pgmap v660: 148 pgs: 148 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:22:01.892 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:01 smithi150 ceph-mon[39612]: 4.1b scrub starts 2024-09-11T08:22:01.892 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:01 smithi150 ceph-mon[39612]: 4.1b scrub ok 2024-09-11T08:22:01.923 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:22:01.939 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-11T08:22:01.963 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:01.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:01.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:22:02.025 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 1 16 3966 3950 246.821 246.875 0.00286082 0.00403674 2024-09-11T08:22:02.067 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:22:02.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:01 smithi181 ceph-mon[32701]: 5.6 deep-scrub starts 2024-09-11T08:22:02.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:01 smithi181 ceph-mon[32701]: 5.6 deep-scrub ok 2024-09-11T08:22:02.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:01 smithi181 ceph-mon[32701]: 3.8 deep-scrub starts 2024-09-11T08:22:02.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:01 smithi181 ceph-mon[32701]: 3.8 deep-scrub ok 2024-09-11T08:22:02.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:01 smithi181 ceph-mon[32701]: pgmap v660: 148 pgs: 148 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:22:02.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:01 smithi181 ceph-mon[32701]: 4.1b scrub starts 2024-09-11T08:22:02.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:01 smithi181 ceph-mon[32701]: 4.1b scrub ok 2024-09-11T08:22:02.101 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:02.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:02.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:22:02.203 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:22:02.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:02.243 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:02.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:22:02.344 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:02.378 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:02.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:02.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:02.480 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:22:02.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:02.511 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:02.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:22:02.540 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:22:02.612 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:22:02.644 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:02.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:02.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:22:02.684 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:22:02.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:02 smithi150 ceph-mon[39612]: 4.5 scrub starts 2024-09-11T08:22:02.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:02 smithi150 ceph-mon[39612]: 4.5 scrub ok 2024-09-11T08:22:02.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:02 smithi150 ceph-mon[39612]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-11T08:22:02.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:02 smithi150 ceph-mon[39612]: Cluster is now healthy 2024-09-11T08:22:02.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:02 smithi150 ceph-mon[29276]: 4.5 scrub starts 2024-09-11T08:22:02.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:02 smithi150 ceph-mon[29276]: 4.5 scrub ok 2024-09-11T08:22:02.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:02 smithi150 ceph-mon[29276]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-11T08:22:02.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:02 smithi150 ceph-mon[29276]: Cluster is now healthy 2024-09-11T08:22:02.745 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:22:02.747 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:22:02.794 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:02.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:02.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:22:02.896 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:22:02.934 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:02.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:02.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:22:03.025 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 2 15 7471 7456 232.965 219.125 0.00554967 0.004284 2024-09-11T08:22:03.037 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:03.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:02 smithi181 ceph-mon[32701]: 4.5 scrub starts 2024-09-11T08:22:03.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:02 smithi181 ceph-mon[32701]: 4.5 scrub ok 2024-09-11T08:22:03.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:02 smithi181 ceph-mon[32701]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-11T08:22:03.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:02 smithi181 ceph-mon[32701]: Cluster is now healthy 2024-09-11T08:22:03.074 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:03.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:03.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:03.176 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:22:03.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:03.212 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:03.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:22:03.313 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:03.350 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:03.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:03.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:03.452 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:03.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:03.489 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:03.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:03.590 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:22:03.626 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:03.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:03.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:22:03.729 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:22:03.765 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:03.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:03.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:22:03.867 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:22:03.903 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:03.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:03.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:22:03.914 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:03 smithi150 ceph-mon[29276]: 5.1e scrub starts 2024-09-11T08:22:03.914 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:03 smithi150 ceph-mon[29276]: 5.1e scrub ok 2024-09-11T08:22:03.914 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:03 smithi150 ceph-mon[29276]: 5.13 deep-scrub starts 2024-09-11T08:22:03.914 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:03 smithi150 ceph-mon[29276]: 5.13 deep-scrub ok 2024-09-11T08:22:03.914 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:03 smithi150 ceph-mon[29276]: pgmap v661: 148 pgs: 148 active+clean; 220 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 577 B/s rd, 25 MiB/s wr, 396 op/s 2024-09-11T08:22:03.914 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:03 smithi150 ceph-mon[29276]: 4.15 deep-scrub starts 2024-09-11T08:22:03.914 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:03 smithi150 ceph-mon[29276]: 4.15 deep-scrub ok 2024-09-11T08:22:03.914 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:03 smithi150 ceph-mon[39612]: 5.1e scrub starts 2024-09-11T08:22:03.914 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:03 smithi150 ceph-mon[39612]: 5.1e scrub ok 2024-09-11T08:22:03.914 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:03 smithi150 ceph-mon[39612]: 5.13 deep-scrub starts 2024-09-11T08:22:03.915 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:03 smithi150 ceph-mon[39612]: 5.13 deep-scrub ok 2024-09-11T08:22:03.915 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:03 smithi150 ceph-mon[39612]: pgmap v661: 148 pgs: 148 active+clean; 220 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 577 B/s rd, 25 MiB/s wr, 396 op/s 2024-09-11T08:22:03.915 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:03 smithi150 ceph-mon[39612]: 4.15 deep-scrub starts 2024-09-11T08:22:03.915 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:03 smithi150 ceph-mon[39612]: 4.15 deep-scrub ok 2024-09-11T08:22:04.004 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:22:04.025 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 3 16 10780 10764 224.221 206.75 0.00737507 0.00445381 2024-09-11T08:22:04.043 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:04.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:04.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:22:04.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:03 smithi181 ceph-mon[32701]: 5.1e scrub starts 2024-09-11T08:22:04.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:03 smithi181 ceph-mon[32701]: 5.1e scrub ok 2024-09-11T08:22:04.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:03 smithi181 ceph-mon[32701]: 5.13 deep-scrub starts 2024-09-11T08:22:04.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:03 smithi181 ceph-mon[32701]: 5.13 deep-scrub ok 2024-09-11T08:22:04.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:03 smithi181 ceph-mon[32701]: pgmap v661: 148 pgs: 148 active+clean; 220 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 577 B/s rd, 25 MiB/s wr, 396 op/s 2024-09-11T08:22:04.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:03 smithi181 ceph-mon[32701]: 4.15 deep-scrub starts 2024-09-11T08:22:04.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:03 smithi181 ceph-mon[32701]: 4.15 deep-scrub ok 2024-09-11T08:22:04.145 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:04.179 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:04.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:04.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:04.281 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:22:04.316 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:04.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:04.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:22:04.418 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:04.452 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:04.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:04.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:04.554 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:22:04.591 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:04.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:04.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:22:04.687 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:04 smithi150 ceph-mon[29276]: 2.1 scrub starts 2024-09-11T08:22:04.687 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:04 smithi150 ceph-mon[29276]: 2.1 scrub ok 2024-09-11T08:22:04.687 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:04 smithi150 ceph-mon[29276]: 3.6 scrub starts 2024-09-11T08:22:04.687 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:04 smithi150 ceph-mon[29276]: 3.6 scrub ok 2024-09-11T08:22:04.688 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:04 smithi150 ceph-mon[29276]: 4.9 scrub starts 2024-09-11T08:22:04.688 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:04 smithi150 ceph-mon[29276]: 4.9 scrub ok 2024-09-11T08:22:04.688 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:04 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3506616702' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:22:04.688 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:04 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:22:04.688 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:04 smithi150 ceph-mon[39612]: 2.1 scrub starts 2024-09-11T08:22:04.688 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:04 smithi150 ceph-mon[39612]: 2.1 scrub ok 2024-09-11T08:22:04.688 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:04 smithi150 ceph-mon[39612]: 3.6 scrub starts 2024-09-11T08:22:04.688 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:04 smithi150 ceph-mon[39612]: 3.6 scrub ok 2024-09-11T08:22:04.688 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:04 smithi150 ceph-mon[39612]: 4.9 scrub starts 2024-09-11T08:22:04.688 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:04 smithi150 ceph-mon[39612]: 4.9 scrub ok 2024-09-11T08:22:04.689 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:04 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3506616702' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:22:04.689 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:04 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:22:04.693 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:22:04.731 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:04.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:04.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:22:04.763 INFO:teuthology.orchestra.run.smithi150.stderr:noscrub is set 2024-09-11T08:22:04.833 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:22:04.872 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:04.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:04.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:22:04.974 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:05.016 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:05.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:05.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:05.025 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 4 16 13904 13888 216.974 195.25 0.00223444 0.00460232 2024-09-11T08:22:05.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:04 smithi181 ceph-mon[32701]: 2.1 scrub starts 2024-09-11T08:22:05.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:04 smithi181 ceph-mon[32701]: 2.1 scrub ok 2024-09-11T08:22:05.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:04 smithi181 ceph-mon[32701]: 3.6 scrub starts 2024-09-11T08:22:05.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:04 smithi181 ceph-mon[32701]: 3.6 scrub ok 2024-09-11T08:22:05.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:04 smithi181 ceph-mon[32701]: 4.9 scrub starts 2024-09-11T08:22:05.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:04 smithi181 ceph-mon[32701]: 4.9 scrub ok 2024-09-11T08:22:05.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:04 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3506616702' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:22:05.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:04 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:22:05.118 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:22:05.157 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:05.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:05.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:22:05.259 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:05.289 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:05.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:05.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:05.390 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:05.428 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:05.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:05.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:05.530 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:22:05.565 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:05.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:05.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:22:05.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:22:05 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:22:05] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:22:05.666 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:05.699 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:05.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:05.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:05.801 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:22:05.837 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:05.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:05.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:22:05.940 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:22:05.980 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:05.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:05.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:22:05.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:05 smithi150 ceph-mon[29276]: 6.0 scrub starts 2024-09-11T08:22:05.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:05 smithi150 ceph-mon[29276]: 6.0 scrub ok 2024-09-11T08:22:05.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:05 smithi150 ceph-mon[29276]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:22:05.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:05 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:22:05.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:05 smithi150 ceph-mon[29276]: osdmap e314: 8 total, 8 up, 8 in 2024-09-11T08:22:05.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:05 smithi150 ceph-mon[29276]: pgmap v663: 148 pgs: 148 active+clean; 490 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 980 op/s 2024-09-11T08:22:05.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:05 smithi150 ceph-mon[39612]: 6.0 scrub starts 2024-09-11T08:22:05.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:05 smithi150 ceph-mon[39612]: 6.0 scrub ok 2024-09-11T08:22:05.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:05 smithi150 ceph-mon[39612]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:22:05.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:05 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:22:05.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:05 smithi150 ceph-mon[39612]: osdmap e314: 8 total, 8 up, 8 in 2024-09-11T08:22:05.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:05 smithi150 ceph-mon[39612]: pgmap v663: 148 pgs: 148 active+clean; 490 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 980 op/s 2024-09-11T08:22:06.025 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 5 15 16833 16818 210.201 183.125 0.00551359 0.0047513 2024-09-11T08:22:06.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:05 smithi181 ceph-mon[32701]: 6.0 scrub starts 2024-09-11T08:22:06.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:05 smithi181 ceph-mon[32701]: 6.0 scrub ok 2024-09-11T08:22:06.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:05 smithi181 ceph-mon[32701]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:22:06.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:05 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:22:06.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:05 smithi181 ceph-mon[32701]: osdmap e314: 8 total, 8 up, 8 in 2024-09-11T08:22:06.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:05 smithi181 ceph-mon[32701]: pgmap v663: 148 pgs: 148 active+clean; 490 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 980 op/s 2024-09-11T08:22:06.082 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:06.117 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:06.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:06.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:06.219 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:22:06.255 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:06.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:06.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:22:06.357 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:22:06.394 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:06.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:06.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:22:06.496 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:22:06.530 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:06.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:06.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:22:06.632 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:22:06.667 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:06.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:06.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:22:06.769 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:22:06.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:06.816 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:06.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:22:06.917 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:06.948 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:06.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:06.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:07.025 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 6 16 19493 19477 202.863 166.188 0.00282278 0.00492037 2024-09-11T08:22:07.050 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:07.085 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:07.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:07.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:07.187 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:07.224 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:07.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:07.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:07.326 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:07.362 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:07.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:07.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:07.463 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:22:07.498 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:07.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:07.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:22:07.600 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:22:07.636 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:07.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:07.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:22:07.737 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:22:07.772 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:07.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:07.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:22:07.874 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:07.889 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T08:22:07.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:07.912 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:07.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:08.013 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:08.025 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 7 16 21782 21766 194.318 143.062 0.00453442 0.00513985 2024-09-11T08:22:08.053 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:08.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:08.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:08.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:07 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:22:08.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:07 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:22:08.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:07 smithi181 ceph-mon[32701]: pgmap v664: 148 pgs: 148 active+clean; 490 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 61 MiB/s wr, 979 op/s 2024-09-11T08:22:08.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:07 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:22:08.156 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:08.177 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:08.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:08.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:08.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:07 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:22:08.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:07 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:22:08.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:07 smithi150 ceph-mon[29276]: pgmap v664: 148 pgs: 148 active+clean; 490 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 61 MiB/s wr, 979 op/s 2024-09-11T08:22:08.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:07 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:22:08.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:07 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:22:08.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:07 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:22:08.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:07 smithi150 ceph-mon[39612]: pgmap v664: 148 pgs: 148 active+clean; 490 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 61 MiB/s wr, 979 op/s 2024-09-11T08:22:08.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:07 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:22:08.279 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:22:08.316 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:08.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:08.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:22:08.418 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:22:08.439 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:08.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:08.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:22:08.541 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:08.575 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:08.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:08.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:08.677 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:22:08.715 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:08.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:08.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:22:08.816 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:22:08.823 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:22:08 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:22:08.846 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:08.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:08.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:22:08.949 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:22:08.984 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:08.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:08.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:22:08.988 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:22:08 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:22:08] "GET /metrics HTTP/1.1" 200 224528 "" "Prometheus/2.43.0" 2024-09-11T08:22:09.025 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 8 16 23666 23650 184.746 117.75 0.00696055 0.00540786 2024-09-11T08:22:09.087 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:09.121 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:09.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:09.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:09.223 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:09.258 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:09.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:09.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:09.360 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:09.393 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:09.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:09.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:09.495 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:22:09.530 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:09.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:09.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:22:09.632 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:22:09.667 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:09.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:09.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:22:09.770 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:09.805 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:09.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:09.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:09.907 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:22:09.939 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:09.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:09.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:22:10.026 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 9 16 25852 25836 179.398 136.625 0.099152 0.00555033 2024-09-11T08:22:10.041 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:22:10.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:10.079 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:10.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:22:10.180 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:10.217 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:10.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:10.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:10.319 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:22:10.351 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:10.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:10.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:22:10.456 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:10.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:10 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:22:10.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:10 smithi150 ceph-mon[29276]: pgmap v665: 148 pgs: 148 active+clean; 1.3 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.0 KiB/s rd, 139 MiB/s wr, 2.23k op/s 2024-09-11T08:22:10.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:10 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:22:10.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:10 smithi150 ceph-mon[39612]: pgmap v665: 148 pgs: 148 active+clean; 1.3 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.0 KiB/s rd, 139 MiB/s wr, 2.23k op/s 2024-09-11T08:22:10.493 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:10.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:10.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:10.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:10 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:22:10.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:10 smithi181 ceph-mon[32701]: pgmap v665: 148 pgs: 148 active+clean; 1.3 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.0 KiB/s rd, 139 MiB/s wr, 2.23k op/s 2024-09-11T08:22:10.595 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:10.629 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:10.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:10.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:10.731 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:10.764 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:10.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:10.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:10.866 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:22:10.889 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:10.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:10.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:22:10.991 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:22:11.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:11.013 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:11.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:22:11.025 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 10 16 27768 27752 173.432 119.75 0.00217773 0.00571268 2024-09-11T08:22:11.114 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:22:11.147 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:11.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:11.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:22:11.249 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:22:11.285 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:11.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:11.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:22:11.288 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:22:11.288 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:22:11.288 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:22:11.288 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:22:11.289 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:22:11.289 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:22:11.289 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:22:11.289 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:22:11.289 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:22:11.289 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:22:11.289 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:22:11.289 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:22:11.290 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:22:11.290 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:22:11.290 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:22:11.290 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:22:11.387 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:22:11.424 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:11.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:11.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:22:11.525 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:22:11.568 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:11.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:11.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:22:11.670 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:11.705 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:11.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:11.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:11.807 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:22:11.842 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:11.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:11.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:22:11.945 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:22:11.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:11.982 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:11.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:22:11.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:11 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3855017199' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:22:11.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:11 smithi150 ceph-mon[29276]: pgmap v666: 148 pgs: 148 active+clean; 1.3 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 137 MiB/s wr, 2.19k op/s 2024-09-11T08:22:11.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:11 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3855017199' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:22:11.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:11 smithi150 ceph-mon[39612]: pgmap v666: 148 pgs: 148 active+clean; 1.3 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 137 MiB/s wr, 2.19k op/s 2024-09-11T08:22:12.025 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 11 16 29995 29979 170.317 139.188 0.00662815 0.00586579 2024-09-11T08:22:12.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:11 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3855017199' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:22:12.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:11 smithi181 ceph-mon[32701]: pgmap v666: 148 pgs: 148 active+clean; 1.3 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 137 MiB/s wr, 2.19k op/s 2024-09-11T08:22:12.083 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:22:12.121 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:12.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:12.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:22:12.224 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:22:12.258 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:12.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:12.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:22:12.360 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:12.392 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:12.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:12.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:12.494 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:12.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:12.528 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:12.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:12.629 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:12.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:12.666 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:12.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:12.767 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:22:12.805 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:12.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:12.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:22:12.908 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:12.945 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:12.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:12.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:12.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:12 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:22:12.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:12 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:22:12.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:12 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:22:12.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:12 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:22:12.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:12 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:22:12.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:12 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:22:13.025 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 12 16 32272 32256 167.983 142.312 0.00279481 0.00594635 2024-09-11T08:22:13.047 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:22:13.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:12 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:22:13.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:12 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:22:13.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:12 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:22:13.082 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:13.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:13.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:22:13.184 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:13.215 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:13.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:13.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:13.223 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:22:13.318 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:22:13.355 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:13.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:13.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:22:13.457 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:22:13.496 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:13.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:13.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:22:13.598 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:13.624 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:13.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:13.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:13.726 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:13.757 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:13.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:13.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:13.859 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:22:13.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:13.899 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:13.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:22:13.909 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-09-11T08:22:13.909 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:22:13.909 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:13 smithi150 ceph-mon[29276]: pgmap v667: 148 pgs: 148 active+clean; 1.7 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 148 MiB/s wr, 2.37k op/s 2024-09-11T08:22:13.910 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:13 smithi150 ceph-mon[39612]: pgmap v667: 148 pgs: 148 active+clean; 1.7 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 148 MiB/s wr, 2.37k op/s 2024-09-11T08:22:14.001 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:14.026 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 13 16 34424 34408 165.405 134.5 0.00666381 0.00604084 2024-09-11T08:22:14.038 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:14.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:14.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:14.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:13 smithi181 ceph-mon[32701]: pgmap v667: 148 pgs: 148 active+clean; 1.7 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 148 MiB/s wr, 2.37k op/s 2024-09-11T08:22:14.098 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T08:22:14.140 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:14.190 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:14.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:14.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:14.292 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:22:14.327 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:14.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:14.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:22:14.429 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:22:14.470 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:14.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:14.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:22:14.571 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:22:14.604 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:14.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:14.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:22:14.706 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:22:14.734 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:14.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:14.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:22:14.817 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:22:14.821 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-09-11T08:22:14.835 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:22:14.864 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:14.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:14.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:22:14.965 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:14.990 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:14.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:14.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:15.026 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 14 16 36230 36214 161.653 112.875 0.0039121 0.00618156 2024-09-11T08:22:15.092 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:15.131 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:15.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:15.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:15.233 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:22:15.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:15.268 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:15.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:22:15.369 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:22:15.405 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:15.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:15.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:22:15.507 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:15.542 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:15.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:15.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:15.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:22:15 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:22:15] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:22:15.643 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:22:15.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:15.675 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:15.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:22:15.776 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:15.797 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:15.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:15.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:15.899 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:15.927 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:15.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:15.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:16.026 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 15 16 37913 37897 157.888 105.188 0.00533645 0.00632887 2024-09-11T08:22:16.028 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:22:16.063 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:16.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:16.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:22:16.165 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:22:16.188 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:16.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:16.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:22:16.290 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:16.327 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:16.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:16.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:16.429 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:22:16.464 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:16.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:16.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:22:16.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:16 smithi150 ceph-mon[29276]: from='client.25414 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:22:16.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:16 smithi150 ceph-mon[29276]: pgmap v668: 148 pgs: 148 active+clean; 2.0 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 987 B/s rd, 149 MiB/s wr, 2.38k op/s 2024-09-11T08:22:16.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:16 smithi150 ceph-mon[39612]: from='client.25414 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:22:16.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:16 smithi150 ceph-mon[39612]: pgmap v668: 148 pgs: 148 active+clean; 2.0 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 987 B/s rd, 149 MiB/s wr, 2.38k op/s 2024-09-11T08:22:16.565 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:22:16.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:16 smithi181 ceph-mon[32701]: from='client.25414 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:22:16.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:16 smithi181 ceph-mon[32701]: pgmap v668: 148 pgs: 148 active+clean; 2.0 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 987 B/s rd, 149 MiB/s wr, 2.38k op/s 2024-09-11T08:22:16.587 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:16.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:16.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:22:16.689 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:16.721 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:16.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:16.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:16.822 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:22:16.861 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:16.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:16.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:22:16.963 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:22:16.993 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-11T08:22:17.002 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:17.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:17.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:22:17.026 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 16 16 39822 39806 155.476 119.312 0.00693413 0.00642675 2024-09-11T08:22:17.104 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:17.154 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:17.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:17.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:17.255 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:22:17.291 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:17.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:17.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:22:17.393 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:17.440 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:17.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:17.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:17.542 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:17.578 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:17.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:17.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:17.680 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:17.715 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:17.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:17.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:17.817 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:22:17.848 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:17.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:17.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:22:17.950 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:17.985 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:17.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:17.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:18.026 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 17 16 41713 41697 153.282 118.188 0.00362195 0.00650357 2024-09-11T08:22:18.087 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:18.119 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:18.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:18.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:18.222 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:22:18.256 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:18.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:18.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:22:18.358 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:18.375 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:18 smithi181 ceph-mon[32701]: pgmap v669: 148 pgs: 148 active+clean; 2.0 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 129 MiB/s wr, 2.06k op/s 2024-09-11T08:22:18.375 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:18 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:22:18.375 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:18 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:22:18.392 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:18.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:18.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:18.484 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:18 smithi150 ceph-mon[29276]: pgmap v669: 148 pgs: 148 active+clean; 2.0 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 129 MiB/s wr, 2.06k op/s 2024-09-11T08:22:18.485 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:18 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:22:18.485 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:18 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:22:18.485 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:18 smithi150 ceph-mon[39612]: pgmap v669: 148 pgs: 148 active+clean; 2.0 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 129 MiB/s wr, 2.06k op/s 2024-09-11T08:22:18.485 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:18 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:22:18.485 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:18 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:22:18.493 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:22:18.528 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:18.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:18.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:22:18.630 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:22:18.668 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:18.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:18.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:22:18.738 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:22:18 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:22:18] "GET /metrics HTTP/1.1" 200 224528 "" "Prometheus/2.43.0" 2024-09-11T08:22:18.769 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:18.805 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:18.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:18.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:18.823 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:22:18 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:22:18.907 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:18.940 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:18.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:18.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:19.026 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 18 16 43691 43675 151.633 123.625 0.0065276 0.0065901 2024-09-11T08:22:19.042 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:19.075 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:19.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:19.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:19.177 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:22:19.214 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:19.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:19.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:22:19.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:19 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:22:19.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:19 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:22:19.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:19 smithi150 ceph-mon[29276]: osdmap e315: 8 total, 8 up, 8 in 2024-09-11T08:22:19.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:19 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:22:19.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:19 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:22:19.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:19 smithi150 ceph-mon[39612]: osdmap e315: 8 total, 8 up, 8 in 2024-09-11T08:22:19.317 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:22:19.350 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:19.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:19.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:22:19.410 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":667,"stamp":"2024-09-11T08:22:13.018680+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1782847708,"num_objects":27391,"num_object_clones":0,"num_object_copies":82173,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27391,"num_whiteouts":0,"num_read":1720,"num_read_kb":1579,"num_write":28024,"num_write_kb":1742151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":237,"num_bytes_recovered":1776739,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":28144,"ondisk_log_size":28144,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":445,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":8151752,"kb_used_data":4829380,"kb_used_omap":19,"kb_used_meta":3322284,"kb_avail":741809464,"statfs":{"total":767960285184,"available":759612891136,"internally_reserved":0,"allocated":4945285120,"data_stored":4941902548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19955,"internal_metadata":3402019341},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,5,6,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1552547840,"num_objects":23690,"num_object_clones":0,"num_object_copies":71070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23690,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":23690,"num_write_kb":1516160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003003"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":343,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.447795+0000","last_change":"2024-09-11T08:21:09.740632+0000","last_active":"2024-09-11T08:21:59.447795+0000","last_peered":"2024-09-11T08:21:59.447795+0000","last_clean":"2024-09-11T08:21:59.447795+0000","last_became_active":"2024-09-11T08:19:29.169363+0000","last_became_peered":"2024-09-11T08:19:29.169363+0000","last_unstale":"2024-09-11T08:21:59.447795+0000","last_undegraded":"2024-09-11T08:21:59.447795+0000","last_fullsized":"2024-09-11T08:21:59.447795+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:09.740551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:21:09.740551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:11.000164+0000","scrub_duration":0.0095522019999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"312'19","reported_seq":341,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.173682+0000","last_change":"2024-09-11T08:21:59.122216+0000","last_active":"2024-09-11T08:21:59.173682+0000","last_peered":"2024-09-11T08:21:59.173682+0000","last_clean":"2024-09-11T08:21:59.173682+0000","last_became_active":"2024-09-11T08:07:32.048048+0000","last_became_peered":"2024-09-11T08:07:32.048048+0000","last_unstale":"2024-09-11T08:21:59.173682+0000","last_undegraded":"2024-09-11T08:21:59.173682+0000","last_fullsized":"2024-09-11T08:21:59.173682+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"312'19","last_scrub_stamp":"2024-09-11T08:21:59.122122+0000","last_deep_scrub":"312'19","last_deep_scrub_stamp":"2024-09-11T08:21:59.122122+0000","last_clean_scrub_stamp":"2024-09-11T08:21:59.122122+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:19.552550+0000","scrub_duration":0.0040323809999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":387,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:22:00.122901+0000","last_change":"2024-09-11T08:21:05.601324+0000","last_active":"2024-09-11T08:22:00.122901+0000","last_peered":"2024-09-11T08:22:00.122901+0000","last_clean":"2024-09-11T08:22:00.122901+0000","last_became_active":"2024-09-11T08:19:29.140997+0000","last_became_peered":"2024-09-11T08:19:29.140997+0000","last_unstale":"2024-09-11T08:22:00.122901+0000","last_undegraded":"2024-09-11T08:22:00.122901+0000","last_fullsized":"2024-09-11T08:22:00.122901+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:05.601245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:59.531202+0000","last_clean_scrub_stamp":"2024-09-11T08:21:05.601245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:30.011735+0000","scrub_duration":0.006405127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":331,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.173908+0000","last_change":"2024-09-11T08:20:11.523247+0000","last_active":"2024-09-11T08:21:59.173908+0000","last_peered":"2024-09-11T08:21:59.173908+0000","last_clean":"2024-09-11T08:21:59.173908+0000","last_became_active":"2024-09-11T08:19:29.147339+0000","last_became_peered":"2024-09-11T08:19:29.147339+0000","last_unstale":"2024-09-11T08:21:59.173908+0000","last_undegraded":"2024-09-11T08:21:59.173908+0000","last_fullsized":"2024-09-11T08:21:59.173908+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:11.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:02.470649+0000","last_clean_scrub_stamp":"2024-09-11T08:20:11.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0056491429999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":335,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.175217+0000","last_change":"2024-09-11T08:20:17.919584+0000","last_active":"2024-09-11T08:21:59.175217+0000","last_peered":"2024-09-11T08:21:59.175217+0000","last_clean":"2024-09-11T08:21:59.175217+0000","last_became_active":"2024-09-11T08:15:56.521104+0000","last_became_peered":"2024-09-11T08:15:56.521104+0000","last_unstale":"2024-09-11T08:21:59.175217+0000","last_undegraded":"2024-09-11T08:21:59.175217+0000","last_fullsized":"2024-09-11T08:21:59.175217+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:17.919504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:20:17.919504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022382589999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":118,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.447459+0000","last_change":"2024-09-11T08:20:52.936220+0000","last_active":"2024-09-11T08:21:59.447459+0000","last_peered":"2024-09-11T08:21:59.447459+0000","last_clean":"2024-09-11T08:21:59.447459+0000","last_became_active":"2024-09-11T08:17:37.550048+0000","last_became_peered":"2024-09-11T08:17:37.550048+0000","last_unstale":"2024-09-11T08:21:59.447459+0000","last_undegraded":"2024-09-11T08:21:59.447459+0000","last_fullsized":"2024-09-11T08:21:59.447459+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:52.936088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:20:52.936088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046527160000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"239'34","reported_seq":278,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.246809+0000","last_change":"2024-09-11T08:21:48.170435+0000","last_active":"2024-09-11T08:21:59.246809+0000","last_peered":"2024-09-11T08:21:59.246809+0000","last_clean":"2024-09-11T08:21:59.246809+0000","last_became_active":"2024-09-11T08:17:37.553866+0000","last_became_peered":"2024-09-11T08:17:37.553866+0000","last_unstale":"2024-09-11T08:21:59.246809+0000","last_undegraded":"2024-09-11T08:21:59.246809+0000","last_fullsized":"2024-09-11T08:21:59.246809+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'34","last_scrub_stamp":"2024-09-11T08:21:48.170378+0000","last_deep_scrub":"145'22","last_deep_scrub_stamp":"2024-09-11T08:15:57.478870+0000","last_clean_scrub_stamp":"2024-09-11T08:21:48.170378+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:51.705516+0000","scrub_duration":0.002279292,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":300,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.173548+0000","last_change":"2024-09-11T08:18:36.998586+0000","last_active":"2024-09-11T08:21:59.173548+0000","last_peered":"2024-09-11T08:21:59.173548+0000","last_clean":"2024-09-11T08:21:59.173548+0000","last_became_active":"2024-09-11T08:07:34.052243+0000","last_became_peered":"2024-09-11T08:07:34.052243+0000","last_unstale":"2024-09-11T08:21:59.173548+0000","last_undegraded":"2024-09-11T08:21:59.173548+0000","last_fullsized":"2024-09-11T08:21:59.173548+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:36.998503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:18:36.998503+0000","last_clean_scrub_stamp":"2024-09-11T08:18:36.998503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0067563429999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":63,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.249076+0000","last_change":"2024-09-11T08:19:42.490622+0000","last_active":"2024-09-11T08:21:59.249076+0000","last_peered":"2024-09-11T08:21:59.249076+0000","last_clean":"2024-09-11T08:21:59.249076+0000","last_became_active":"2024-09-11T08:19:29.142228+0000","last_became_peered":"2024-09-11T08:19:29.142228+0000","last_unstale":"2024-09-11T08:21:59.249076+0000","last_undegraded":"2024-09-11T08:21:59.249076+0000","last_fullsized":"2024-09-11T08:21:59.249076+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:19:42.490567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:54.878409+0000","last_clean_scrub_stamp":"2024-09-11T08:19:42.490567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001226284,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"239'15","reported_seq":340,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.174429+0000","last_change":"2024-09-11T08:20:13.492673+0000","last_active":"2024-09-11T08:21:59.174429+0000","last_peered":"2024-09-11T08:21:59.174429+0000","last_clean":"2024-09-11T08:21:59.174429+0000","last_became_active":"2024-09-11T08:07:32.044626+0000","last_became_peered":"2024-09-11T08:07:32.044626+0000","last_unstale":"2024-09-11T08:21:59.174429+0000","last_undegraded":"2024-09-11T08:21:59.174429+0000","last_fullsized":"2024-09-11T08:21:59.174429+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'15","last_scrub_stamp":"2024-09-11T08:20:13.492593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:20:13.492593+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023810210000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":63,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.248828+0000","last_change":"2024-09-11T08:20:18.900646+0000","last_active":"2024-09-11T08:21:59.248828+0000","last_peered":"2024-09-11T08:21:59.248828+0000","last_clean":"2024-09-11T08:21:59.248828+0000","last_became_active":"2024-09-11T08:19:29.192476+0000","last_became_peered":"2024-09-11T08:19:29.192476+0000","last_unstale":"2024-09-11T08:21:59.248828+0000","last_undegraded":"2024-09-11T08:21:59.248828+0000","last_fullsized":"2024-09-11T08:21:59.248828+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:18.900482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:20:18.900482+0000","last_clean_scrub_stamp":"2024-09-11T08:20:18.900482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.027755386,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":334,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.447092+0000","last_change":"2024-09-11T08:21:52.687247+0000","last_active":"2024-09-11T08:21:59.447092+0000","last_peered":"2024-09-11T08:21:59.447092+0000","last_clean":"2024-09-11T08:21:59.447092+0000","last_became_active":"2024-09-11T08:14:24.792595+0000","last_became_peered":"2024-09-11T08:14:24.792595+0000","last_unstale":"2024-09-11T08:21:59.447092+0000","last_undegraded":"2024-09-11T08:21:59.447092+0000","last_fullsized":"2024-09-11T08:21:59.447092+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:52.687195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:21:52.687195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:12.642688+0000","scrub_duration":0.0017906930000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":63,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.249126+0000","last_change":"2024-09-11T08:21:48.418367+0000","last_active":"2024-09-11T08:21:59.249126+0000","last_peered":"2024-09-11T08:21:59.249126+0000","last_clean":"2024-09-11T08:21:59.249126+0000","last_became_active":"2024-09-11T08:19:29.140493+0000","last_became_peered":"2024-09-11T08:19:29.140493+0000","last_unstale":"2024-09-11T08:21:59.249126+0000","last_undegraded":"2024-09-11T08:21:59.249126+0000","last_fullsized":"2024-09-11T08:21:59.249126+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:48.418276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:21:48.418276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:51.125352+0000","scrub_duration":0.0015283429999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"239'9","reported_seq":330,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:22:01.123205+0000","last_change":"2024-09-11T08:22:01.123132+0000","last_active":"2024-09-11T08:22:01.123205+0000","last_peered":"2024-09-11T08:22:01.123205+0000","last_clean":"2024-09-11T08:22:01.123205+0000","last_became_active":"2024-09-11T08:07:32.048661+0000","last_became_peered":"2024-09-11T08:07:32.048661+0000","last_unstale":"2024-09-11T08:22:01.123205+0000","last_undegraded":"2024-09-11T08:22:01.123205+0000","last_fullsized":"2024-09-11T08:22:01.123205+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'9","last_scrub_stamp":"2024-09-11T08:22:01.123040+0000","last_deep_scrub":"139'7","last_deep_scrub_stamp":"2024-09-11T08:12:24.869318+0000","last_clean_scrub_stamp":"2024-09-11T08:22:01.123040+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:01.721028+0000","scrub_duration":0.004407467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":146,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.175698+0000","last_change":"2024-09-11T08:20:53.087846+0000","last_active":"2024-09-11T08:21:59.175698+0000","last_peered":"2024-09-11T08:21:59.175698+0000","last_clean":"2024-09-11T08:21:59.175698+0000","last_became_active":"2024-09-11T08:15:56.526954+0000","last_became_peered":"2024-09-11T08:15:56.526954+0000","last_unstale":"2024-09-11T08:21:59.175698+0000","last_undegraded":"2024-09-11T08:21:59.175698+0000","last_fullsized":"2024-09-11T08:21:59.175698+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:53.087699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:20:53.087699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:22.711960+0000","scrub_duration":0.0024133610000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":63,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.248597+0000","last_change":"2024-09-11T08:20:19.930386+0000","last_active":"2024-09-11T08:21:59.248597+0000","last_peered":"2024-09-11T08:21:59.248597+0000","last_clean":"2024-09-11T08:21:59.248597+0000","last_became_active":"2024-09-11T08:19:29.143572+0000","last_became_peered":"2024-09-11T08:19:29.143572+0000","last_unstale":"2024-09-11T08:21:59.248597+0000","last_undegraded":"2024-09-11T08:21:59.248597+0000","last_fullsized":"2024-09-11T08:21:59.248597+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:19.927616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:20:19.927616+0000","last_clean_scrub_stamp":"2024-09-11T08:20:19.927616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0062725719999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":331,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.174240+0000","last_change":"2024-09-11T08:20:49.471916+0000","last_active":"2024-09-11T08:21:59.174240+0000","last_peered":"2024-09-11T08:21:59.174240+0000","last_clean":"2024-09-11T08:21:59.174240+0000","last_became_active":"2024-09-11T08:19:29.149081+0000","last_became_peered":"2024-09-11T08:19:29.149081+0000","last_unstale":"2024-09-11T08:21:59.174240+0000","last_undegraded":"2024-09-11T08:21:59.174240+0000","last_fullsized":"2024-09-11T08:21:59.174240+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:49.471510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:20:49.471510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:15.385674+0000","scrub_duration":0.0048182779999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":343,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.447758+0000","last_change":"2024-09-11T08:20:09.565068+0000","last_active":"2024-09-11T08:21:59.447758+0000","last_peered":"2024-09-11T08:21:59.447758+0000","last_clean":"2024-09-11T08:21:59.447758+0000","last_became_active":"2024-09-11T08:15:56.516941+0000","last_became_peered":"2024-09-11T08:15:56.516941+0000","last_unstale":"2024-09-11T08:21:59.447758+0000","last_undegraded":"2024-09-11T08:21:59.447758+0000","last_fullsized":"2024-09-11T08:21:59.447758+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:09.564941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:20:09.564941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018499059999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"239'34","reported_seq":381,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.247143+0000","last_change":"2024-09-11T08:21:01.847533+0000","last_active":"2024-09-11T08:21:59.247143+0000","last_peered":"2024-09-11T08:21:59.247143+0000","last_clean":"2024-09-11T08:21:59.247143+0000","last_became_active":"2024-09-11T08:07:32.046363+0000","last_became_peered":"2024-09-11T08:07:32.046363+0000","last_unstale":"2024-09-11T08:21:59.247143+0000","last_undegraded":"2024-09-11T08:21:59.247143+0000","last_fullsized":"2024-09-11T08:21:59.247143+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'34","last_scrub_stamp":"2024-09-11T08:21:01.847438+0000","last_deep_scrub":"107'22","last_deep_scrub_stamp":"2024-09-11T08:16:04.159211+0000","last_clean_scrub_stamp":"2024-09-11T08:21:01.847438+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:16.033858+0000","scrub_duration":0.0079036850000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":340,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:22:00.123560+0000","last_change":"2024-09-11T08:21:46.819968+0000","last_active":"2024-09-11T08:22:00.123560+0000","last_peered":"2024-09-11T08:22:00.123560+0000","last_clean":"2024-09-11T08:22:00.123560+0000","last_became_active":"2024-09-11T08:07:34.055410+0000","last_became_peered":"2024-09-11T08:07:34.055410+0000","last_unstale":"2024-09-11T08:22:00.123560+0000","last_undegraded":"2024-09-11T08:22:00.123560+0000","last_fullsized":"2024-09-11T08:22:00.123560+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:46.819905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:19:28.870211+0000","last_clean_scrub_stamp":"2024-09-11T08:21:46.819905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:01.293541+0000","scrub_duration":0.0018277110000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":138,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.447567+0000","last_change":"2024-09-11T08:21:51.420044+0000","last_active":"2024-09-11T08:21:59.447567+0000","last_peered":"2024-09-11T08:21:59.447567+0000","last_clean":"2024-09-11T08:21:59.447567+0000","last_became_active":"2024-09-11T08:19:29.144780+0000","last_became_peered":"2024-09-11T08:19:29.144780+0000","last_unstale":"2024-09-11T08:21:59.447567+0000","last_undegraded":"2024-09-11T08:21:59.447567+0000","last_fullsized":"2024-09-11T08:21:59.447567+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:51.419976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:21:51.419976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:51.718685+0000","scrub_duration":0.0015353280000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":324,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.174547+0000","last_change":"2024-09-11T08:21:52.110853+0000","last_active":"2024-09-11T08:21:59.174547+0000","last_peered":"2024-09-11T08:21:59.174547+0000","last_clean":"2024-09-11T08:21:59.174547+0000","last_became_active":"2024-09-11T08:07:30.046284+0000","last_became_peered":"2024-09-11T08:07:30.046284+0000","last_unstale":"2024-09-11T08:21:59.174547+0000","last_undegraded":"2024-09-11T08:21:59.174547+0000","last_fullsized":"2024-09-11T08:21:59.174547+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:52.110756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:21:52.110756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:12.473550+0000","scrub_duration":0.001648369,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"239'18","reported_seq":188,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.247016+0000","last_change":"2024-09-11T08:21:50.171673+0000","last_active":"2024-09-11T08:21:59.247016+0000","last_peered":"2024-09-11T08:21:59.247016+0000","last_clean":"2024-09-11T08:21:59.247016+0000","last_became_active":"2024-09-11T08:17:37.553402+0000","last_became_peered":"2024-09-11T08:17:37.553402+0000","last_unstale":"2024-09-11T08:21:59.247016+0000","last_undegraded":"2024-09-11T08:21:59.247016+0000","last_fullsized":"2024-09-11T08:21:59.247016+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'18","last_scrub_stamp":"2024-09-11T08:21:50.171618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:21:50.171618+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:05.493899+0000","scrub_duration":0.001971993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":320,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.174565+0000","last_change":"2024-09-11T08:21:54.144410+0000","last_active":"2024-09-11T08:21:59.174565+0000","last_peered":"2024-09-11T08:21:59.174565+0000","last_clean":"2024-09-11T08:21:59.174565+0000","last_became_active":"2024-09-11T08:07:34.056528+0000","last_became_peered":"2024-09-11T08:07:34.056528+0000","last_unstale":"2024-09-11T08:21:59.174565+0000","last_undegraded":"2024-09-11T08:21:59.174565+0000","last_fullsized":"2024-09-11T08:21:59.174565+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:54.144345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:10:40.110490+0000","last_clean_scrub_stamp":"2024-09-11T08:21:54.144345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:10.135298+0000","scrub_duration":0.0015696499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":301,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.176272+0000","last_change":"2024-09-11T08:21:05.279818+0000","last_active":"2024-09-11T08:21:59.176272+0000","last_peered":"2024-09-11T08:21:59.176272+0000","last_clean":"2024-09-11T08:21:59.176272+0000","last_became_active":"2024-09-11T08:19:29.149754+0000","last_became_peered":"2024-09-11T08:19:29.149754+0000","last_unstale":"2024-09-11T08:21:59.176272+0000","last_undegraded":"2024-09-11T08:21:59.176272+0000","last_fullsized":"2024-09-11T08:21:59.176272+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-11T08:21:05.279671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:21:05.279671+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:09.854685+0000","scrub_duration":0.0033543840000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":334,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:22:00.122981+0000","last_change":"2024-09-11T08:21:06.624368+0000","last_active":"2024-09-11T08:22:00.122981+0000","last_peered":"2024-09-11T08:22:00.122981+0000","last_clean":"2024-09-11T08:22:00.122981+0000","last_became_active":"2024-09-11T08:07:30.045932+0000","last_became_peered":"2024-09-11T08:07:30.045932+0000","last_unstale":"2024-09-11T08:22:00.122981+0000","last_undegraded":"2024-09-11T08:22:00.122981+0000","last_fullsized":"2024-09-11T08:22:00.122981+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:06.624287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:21:06.624287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:27.183588+0000","scrub_duration":0.0049003370000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"239'20","reported_seq":424,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.447694+0000","last_change":"2024-09-11T08:21:46.541430+0000","last_active":"2024-09-11T08:21:59.447694+0000","last_peered":"2024-09-11T08:21:59.447694+0000","last_clean":"2024-09-11T08:21:59.447694+0000","last_became_active":"2024-09-11T08:15:56.515817+0000","last_became_peered":"2024-09-11T08:15:56.515817+0000","last_unstale":"2024-09-11T08:21:59.447694+0000","last_undegraded":"2024-09-11T08:21:59.447694+0000","last_fullsized":"2024-09-11T08:21:59.447694+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'20","last_scrub_stamp":"2024-09-11T08:21:46.541374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:21:46.541374+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:11.112350+0000","scrub_duration":0.0020197930000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"254'12","reported_seq":573,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-11T08:22:04.659788+0000","last_change":"2024-09-11T08:21:52.380933+0000","last_active":"2024-09-11T08:22:04.659788+0000","last_peered":"2024-09-11T08:22:04.659788+0000","last_clean":"2024-09-11T08:22:04.659788+0000","last_became_active":"2024-09-11T08:15:56.525498+0000","last_became_peered":"2024-09-11T08:15:56.525498+0000","last_unstale":"2024-09-11T08:22:04.659788+0000","last_undegraded":"2024-09-11T08:22:04.659788+0000","last_fullsized":"2024-09-11T08:22:04.659788+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'12","last_scrub_stamp":"2024-09-11T08:21:52.380812+0000","last_deep_scrub":"254'12","last_deep_scrub_stamp":"2024-09-11T08:18:34.253865+0000","last_clean_scrub_stamp":"2024-09-11T08:21:52.380812+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:03.869144+0000","scrub_duration":0.0017919870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"314'1950","reported_seq":1964,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-11T08:22:12.854164+0000","last_change":"2024-09-11T08:21:47.139820+0000","last_active":"2024-09-11T08:22:12.854164+0000","last_peered":"2024-09-11T08:22:12.854164+0000","last_clean":"2024-09-11T08:22:12.854164+0000","last_became_active":"2024-09-11T08:21:47.139412+0000","last_became_peered":"2024-09-11T08:21:47.139412+0000","last_unstale":"2024-09-11T08:22:12.854164+0000","last_undegraded":"2024-09-11T08:22:12.854164+0000","last_fullsized":"2024-09-11T08:22:12.854164+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":308,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_clean_scrub_stamp":"2024-09-11T08:21:46.117341+0000","objects_scrubbed":0,"log_size":1950,"log_dups_size":0,"ondisk_log_size":1950,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:58.457112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127795200,"num_objects":1950,"num_object_clones":0,"num_object_copies":5850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1950,"num_write_kb":124800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"239'27","reported_seq":385,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:22:00.123650+0000","last_change":"2024-09-11T08:21:11.723564+0000","last_active":"2024-09-11T08:22:00.123650+0000","last_peered":"2024-09-11T08:22:00.123650+0000","last_clean":"2024-09-11T08:22:00.123650+0000","last_became_active":"2024-09-11T08:07:32.054317+0000","last_became_peered":"2024-09-11T08:07:32.054317+0000","last_unstale":"2024-09-11T08:22:00.123650+0000","last_undegraded":"2024-09-11T08:22:00.123650+0000","last_fullsized":"2024-09-11T08:22:00.123650+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'27","last_scrub_stamp":"2024-09-11T08:21:11.722291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:21:11.722291+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:25.479711+0000","scrub_duration":0.024639298,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":354,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:22:00.708089+0000","last_change":"2024-09-11T08:22:00.708044+0000","last_active":"2024-09-11T08:22:00.708089+0000","last_peered":"2024-09-11T08:22:00.708089+0000","last_clean":"2024-09-11T08:22:00.708089+0000","last_became_active":"2024-09-11T08:15:56.521943+0000","last_became_peered":"2024-09-11T08:15:56.521943+0000","last_unstale":"2024-09-11T08:22:00.708089+0000","last_undegraded":"2024-09-11T08:22:00.708089+0000","last_fullsized":"2024-09-11T08:22:00.708089+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:22:00.707990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:22:00.707990+0000","last_clean_scrub_stamp":"2024-09-11T08:22:00.707990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:02.700790+0000","scrub_duration":0.001504299,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"271'10","reported_seq":522,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-11T08:22:04.657588+0000","last_change":"2024-09-11T08:19:29.165123+0000","last_active":"2024-09-11T08:22:04.657588+0000","last_peered":"2024-09-11T08:22:04.657588+0000","last_clean":"2024-09-11T08:22:04.657588+0000","last_became_active":"2024-09-11T08:19:29.149773+0000","last_became_peered":"2024-09-11T08:19:29.149773+0000","last_unstale":"2024-09-11T08:22:04.657588+0000","last_undegraded":"2024-09-11T08:22:04.657588+0000","last_fullsized":"2024-09-11T08:22:04.657588+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:18:44.817179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:18:44.817179+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022442809999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":63,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.248944+0000","last_change":"2024-09-11T08:20:10.875817+0000","last_active":"2024-09-11T08:21:59.248944+0000","last_peered":"2024-09-11T08:21:59.248944+0000","last_clean":"2024-09-11T08:21:59.248944+0000","last_became_active":"2024-09-11T08:19:29.164735+0000","last_became_peered":"2024-09-11T08:19:29.164735+0000","last_unstale":"2024-09-11T08:21:59.248944+0000","last_undegraded":"2024-09-11T08:21:59.248944+0000","last_fullsized":"2024-09-11T08:21:59.248944+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:10.875651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:06.616332+0000","last_clean_scrub_stamp":"2024-09-11T08:20:10.875651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.018223294000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"314'1939","reported_seq":1953,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-11T08:22:12.863140+0000","last_change":"2024-09-11T08:21:47.141276+0000","last_active":"2024-09-11T08:22:12.863140+0000","last_peered":"2024-09-11T08:22:12.863140+0000","last_clean":"2024-09-11T08:22:12.863140+0000","last_became_active":"2024-09-11T08:21:47.140961+0000","last_became_peered":"2024-09-11T08:21:47.140961+0000","last_unstale":"2024-09-11T08:22:12.863140+0000","last_undegraded":"2024-09-11T08:22:12.863140+0000","last_fullsized":"2024-09-11T08:22:12.863140+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":308,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_clean_scrub_stamp":"2024-09-11T08:21:46.117341+0000","objects_scrubbed":0,"log_size":1939,"log_dups_size":0,"ondisk_log_size":1939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:57.094119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127074304,"num_objects":1939,"num_object_clones":0,"num_object_copies":5817,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1939,"num_write_kb":124096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"239'30","reported_seq":370,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.175821+0000","last_change":"2024-09-11T08:20:15.829563+0000","last_active":"2024-09-11T08:21:59.175821+0000","last_peered":"2024-09-11T08:21:59.175821+0000","last_clean":"2024-09-11T08:21:59.175821+0000","last_became_active":"2024-09-11T08:17:37.555703+0000","last_became_peered":"2024-09-11T08:17:37.555703+0000","last_unstale":"2024-09-11T08:21:59.175821+0000","last_undegraded":"2024-09-11T08:21:59.175821+0000","last_fullsized":"2024-09-11T08:21:59.175821+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'30","last_scrub_stamp":"2024-09-11T08:20:15.829476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:20:15.829476+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0058827109999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":304,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.176199+0000","last_change":"2024-09-11T08:21:02.160052+0000","last_active":"2024-09-11T08:21:59.176199+0000","last_peered":"2024-09-11T08:21:59.176199+0000","last_clean":"2024-09-11T08:21:59.176199+0000","last_became_active":"2024-09-11T08:17:37.553579+0000","last_became_peered":"2024-09-11T08:17:37.553579+0000","last_unstale":"2024-09-11T08:21:59.176199+0000","last_undegraded":"2024-09-11T08:21:59.176199+0000","last_fullsized":"2024-09-11T08:21:59.176199+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:02.159910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:21:02.159910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:20.376173+0000","scrub_duration":0.0086008690000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":337,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.173680+0000","last_change":"2024-09-11T08:20:57.523721+0000","last_active":"2024-09-11T08:21:59.173680+0000","last_peered":"2024-09-11T08:21:59.173680+0000","last_clean":"2024-09-11T08:21:59.173680+0000","last_became_active":"2024-09-11T08:15:56.522606+0000","last_became_peered":"2024-09-11T08:15:56.522606+0000","last_unstale":"2024-09-11T08:21:59.173680+0000","last_undegraded":"2024-09-11T08:21:59.173680+0000","last_fullsized":"2024-09-11T08:21:59.173680+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:57.523652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:20:57.523652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:19.553362+0000","scrub_duration":0.0041561580000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":311,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.174037+0000","last_change":"2024-09-11T08:20:28.457738+0000","last_active":"2024-09-11T08:21:59.174037+0000","last_peered":"2024-09-11T08:21:59.174037+0000","last_clean":"2024-09-11T08:21:59.174037+0000","last_became_active":"2024-09-11T08:07:36.069225+0000","last_became_peered":"2024-09-11T08:07:36.069225+0000","last_unstale":"2024-09-11T08:21:59.174037+0000","last_undegraded":"2024-09-11T08:21:59.174037+0000","last_fullsized":"2024-09-11T08:21:59.174037+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:28.457660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:04.937884+0000","last_clean_scrub_stamp":"2024-09-11T08:20:28.457660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028279630000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"314'1942","reported_seq":1956,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-11T08:22:12.901822+0000","last_change":"2024-09-11T08:21:47.139560+0000","last_active":"2024-09-11T08:22:12.901822+0000","last_peered":"2024-09-11T08:22:12.901822+0000","last_clean":"2024-09-11T08:22:12.901822+0000","last_became_active":"2024-09-11T08:21:47.139248+0000","last_became_peered":"2024-09-11T08:21:47.139248+0000","last_unstale":"2024-09-11T08:22:12.901822+0000","last_undegraded":"2024-09-11T08:22:12.901822+0000","last_fullsized":"2024-09-11T08:22:12.901822+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":308,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_clean_scrub_stamp":"2024-09-11T08:21:46.117341+0000","objects_scrubbed":0,"log_size":1942,"log_dups_size":0,"ondisk_log_size":1942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:54.521250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127270912,"num_objects":1942,"num_object_clones":0,"num_object_copies":5826,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1942,"num_write_kb":124288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"239'28","reported_seq":406,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.174112+0000","last_change":"2024-09-11T08:20:55.458131+0000","last_active":"2024-09-11T08:21:59.174112+0000","last_peered":"2024-09-11T08:21:59.174112+0000","last_clean":"2024-09-11T08:21:59.174112+0000","last_became_active":"2024-09-11T08:19:29.148311+0000","last_became_peered":"2024-09-11T08:19:29.148311+0000","last_unstale":"2024-09-11T08:21:59.174112+0000","last_undegraded":"2024-09-11T08:21:59.174112+0000","last_fullsized":"2024-09-11T08:21:59.174112+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'28","last_scrub_stamp":"2024-09-11T08:20:55.458034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:20:55.458034+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:08.620710+0000","scrub_duration":0.0051959880000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":333,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:22:03.564404+0000","last_change":"2024-09-11T08:22:03.564355+0000","last_active":"2024-09-11T08:22:03.564404+0000","last_peered":"2024-09-11T08:22:03.564404+0000","last_clean":"2024-09-11T08:22:03.564404+0000","last_became_active":"2024-09-11T08:07:30.050484+0000","last_became_peered":"2024-09-11T08:07:30.050484+0000","last_unstale":"2024-09-11T08:22:03.564404+0000","last_undegraded":"2024-09-11T08:22:03.564404+0000","last_fullsized":"2024-09-11T08:22:03.564404+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:22:03.564268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:47.960329+0000","last_clean_scrub_stamp":"2024-09-11T08:22:03.564268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:18.333900+0000","scrub_duration":0.0043127169999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"82'9","reported_seq":517,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:22:03.724468+0000","last_change":"2024-09-11T08:21:00.841525+0000","last_active":"2024-09-11T08:22:03.724468+0000","last_peered":"2024-09-11T08:22:03.724468+0000","last_clean":"2024-09-11T08:22:03.724468+0000","last_became_active":"2024-09-11T08:07:34.056143+0000","last_became_peered":"2024-09-11T08:07:34.056143+0000","last_unstale":"2024-09-11T08:22:03.724468+0000","last_undegraded":"2024-09-11T08:22:03.724468+0000","last_fullsized":"2024-09-11T08:22:03.724468+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-11T08:21:00.841426+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-09-11T08:21:00.841426+0000","last_clean_scrub_stamp":"2024-09-11T08:21:00.841426+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0051535979999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":308,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.176391+0000","last_change":"2024-09-11T08:21:51.600406+0000","last_active":"2024-09-11T08:21:59.176391+0000","last_peered":"2024-09-11T08:21:59.176391+0000","last_clean":"2024-09-11T08:21:59.176391+0000","last_became_active":"2024-09-11T08:14:24.788968+0000","last_became_peered":"2024-09-11T08:14:24.788968+0000","last_unstale":"2024-09-11T08:21:59.176391+0000","last_undegraded":"2024-09-11T08:21:59.176391+0000","last_fullsized":"2024-09-11T08:21:59.176391+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:51.600338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:21:51.600338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:08.612553+0000","scrub_duration":0.0016909780000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.c","version":"314'1333","reported_seq":1347,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-11T08:22:08.034039+0000","last_change":"2024-09-11T08:21:47.139581+0000","last_active":"2024-09-11T08:22:08.034039+0000","last_peered":"2024-09-11T08:22:08.034039+0000","last_clean":"2024-09-11T08:22:08.034039+0000","last_became_active":"2024-09-11T08:21:47.139258+0000","last_became_peered":"2024-09-11T08:21:47.139258+0000","last_unstale":"2024-09-11T08:22:08.034039+0000","last_undegraded":"2024-09-11T08:22:08.034039+0000","last_fullsized":"2024-09-11T08:22:08.034039+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":308,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_clean_scrub_stamp":"2024-09-11T08:21:46.117341+0000","objects_scrubbed":0,"log_size":1333,"log_dups_size":0,"ondisk_log_size":1333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:06.530718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87359488,"num_objects":1333,"num_object_clones":0,"num_object_copies":3999,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1333,"num_write_kb":85312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"239'20","reported_seq":412,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:22:00.123283+0000","last_change":"2024-09-11T08:21:56.861643+0000","last_active":"2024-09-11T08:22:00.123283+0000","last_peered":"2024-09-11T08:22:00.123283+0000","last_clean":"2024-09-11T08:22:00.123283+0000","last_became_active":"2024-09-11T08:19:29.143486+0000","last_became_peered":"2024-09-11T08:19:29.143486+0000","last_unstale":"2024-09-11T08:22:00.123283+0000","last_undegraded":"2024-09-11T08:22:00.123283+0000","last_fullsized":"2024-09-11T08:22:00.123283+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'20","last_scrub_stamp":"2024-09-11T08:21:56.861587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:21:56.861587+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:21.934014+0000","scrub_duration":0.0021882690000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":63,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.248996+0000","last_change":"2024-09-11T08:20:55.691331+0000","last_active":"2024-09-11T08:21:59.248996+0000","last_peered":"2024-09-11T08:21:59.248996+0000","last_clean":"2024-09-11T08:21:59.248996+0000","last_became_active":"2024-09-11T08:19:29.143310+0000","last_became_peered":"2024-09-11T08:19:29.143310+0000","last_unstale":"2024-09-11T08:21:59.248996+0000","last_undegraded":"2024-09-11T08:21:59.248996+0000","last_fullsized":"2024-09-11T08:21:59.248996+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:55.690500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:20:55.690500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:09.255593+0000","scrub_duration":0.013778218,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"271'12","reported_seq":193,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-11T08:22:04.659321+0000","last_change":"2024-09-11T08:21:47.298873+0000","last_active":"2024-09-11T08:22:04.659321+0000","last_peered":"2024-09-11T08:22:04.659321+0000","last_clean":"2024-09-11T08:22:04.659321+0000","last_became_active":"2024-09-11T08:19:29.144614+0000","last_became_peered":"2024-09-11T08:19:29.144614+0000","last_unstale":"2024-09-11T08:22:04.659321+0000","last_undegraded":"2024-09-11T08:22:04.659321+0000","last_fullsized":"2024-09-11T08:22:04.659321+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'12","last_scrub_stamp":"2024-09-11T08:21:47.298801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:21:47.298801+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:12.005882+0000","scrub_duration":0.0018774289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"66'1","reported_seq":318,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:22:04.543921+0000","last_change":"2024-09-11T08:22:04.543317+0000","last_active":"2024-09-11T08:22:04.543921+0000","last_peered":"2024-09-11T08:22:04.543921+0000","last_clean":"2024-09-11T08:22:04.543921+0000","last_became_active":"2024-09-11T08:07:36.072593+0000","last_became_peered":"2024-09-11T08:07:36.072593+0000","last_unstale":"2024-09-11T08:22:04.543921+0000","last_undegraded":"2024-09-11T08:22:04.543921+0000","last_fullsized":"2024-09-11T08:22:04.543921+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-11T08:22:04.543159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:22:04.543159+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:27.526410+0000","scrub_duration":0.002903625,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.d","version":"314'1977","reported_seq":1991,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-11T08:22:12.927039+0000","last_change":"2024-09-11T08:21:47.143484+0000","last_active":"2024-09-11T08:22:12.927039+0000","last_peered":"2024-09-11T08:22:12.927039+0000","last_clean":"2024-09-11T08:22:12.927039+0000","last_became_active":"2024-09-11T08:21:47.143172+0000","last_became_peered":"2024-09-11T08:21:47.143172+0000","last_unstale":"2024-09-11T08:22:12.927039+0000","last_undegraded":"2024-09-11T08:22:12.927039+0000","last_fullsized":"2024-09-11T08:22:12.927039+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":308,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_clean_scrub_stamp":"2024-09-11T08:21:46.117341+0000","objects_scrubbed":0,"log_size":1977,"log_dups_size":0,"ondisk_log_size":1977,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:56.640436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129564672,"num_objects":1977,"num_object_clones":0,"num_object_copies":5931,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1977,"num_write_kb":126528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"239'29","reported_seq":491,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.447548+0000","last_change":"2024-09-11T08:21:44.561967+0000","last_active":"2024-09-11T08:21:59.447548+0000","last_peered":"2024-09-11T08:21:59.447548+0000","last_clean":"2024-09-11T08:21:59.447548+0000","last_became_active":"2024-09-11T08:19:29.195995+0000","last_became_peered":"2024-09-11T08:19:29.195995+0000","last_unstale":"2024-09-11T08:21:59.447548+0000","last_undegraded":"2024-09-11T08:21:59.447548+0000","last_fullsized":"2024-09-11T08:21:59.447548+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'29","last_scrub_stamp":"2024-09-11T08:21:44.561906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:21:44.561906+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:02.503186+0000","scrub_duration":0.0024798810000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":367,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:22:00.123429+0000","last_change":"2024-09-11T08:21:47.825500+0000","last_active":"2024-09-11T08:22:00.123429+0000","last_peered":"2024-09-11T08:22:00.123429+0000","last_clean":"2024-09-11T08:22:00.123429+0000","last_became_active":"2024-09-11T08:19:29.141999+0000","last_became_peered":"2024-09-11T08:19:29.141999+0000","last_unstale":"2024-09-11T08:22:00.123429+0000","last_undegraded":"2024-09-11T08:22:00.123429+0000","last_fullsized":"2024-09-11T08:22:00.123429+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:47.825446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:21:47.825446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:17.004135+0000","scrub_duration":0.0015083430000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":359,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.447011+0000","last_change":"2024-09-11T08:20:51.918821+0000","last_active":"2024-09-11T08:21:59.447011+0000","last_peered":"2024-09-11T08:21:59.447011+0000","last_clean":"2024-09-11T08:21:59.447011+0000","last_became_active":"2024-09-11T08:19:29.166609+0000","last_became_peered":"2024-09-11T08:19:29.166609+0000","last_unstale":"2024-09-11T08:21:59.447011+0000","last_undegraded":"2024-09-11T08:21:59.447011+0000","last_fullsized":"2024-09-11T08:21:59.447011+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:51.918741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:20:51.918741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0090788139999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":328,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:22:00.122870+0000","last_change":"2024-09-11T08:20:56.715356+0000","last_active":"2024-09-11T08:22:00.122870+0000","last_peered":"2024-09-11T08:22:00.122870+0000","last_clean":"2024-09-11T08:22:00.122870+0000","last_became_active":"2024-09-11T08:07:36.631668+0000","last_became_peered":"2024-09-11T08:07:36.631668+0000","last_unstale":"2024-09-11T08:22:00.122870+0000","last_undegraded":"2024-09-11T08:22:00.122870+0000","last_fullsized":"2024-09-11T08:22:00.122870+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:56.715273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:58.517624+0000","last_clean_scrub_stamp":"2024-09-11T08:20:56.715273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:25.260940+0000","scrub_duration":0.0021168839999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"314'2042","reported_seq":2056,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-11T08:22:12.919719+0000","last_change":"2024-09-11T08:21:47.140617+0000","last_active":"2024-09-11T08:22:12.919719+0000","last_peered":"2024-09-11T08:22:12.919719+0000","last_clean":"2024-09-11T08:22:12.919719+0000","last_became_active":"2024-09-11T08:21:47.140341+0000","last_became_peered":"2024-09-11T08:21:47.140341+0000","last_unstale":"2024-09-11T08:22:12.919719+0000","last_undegraded":"2024-09-11T08:22:12.919719+0000","last_fullsized":"2024-09-11T08:22:12.919719+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":308,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_clean_scrub_stamp":"2024-09-11T08:21:46.117341+0000","objects_scrubbed":0,"log_size":2042,"log_dups_size":0,"ondisk_log_size":2042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:55.206551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133824512,"num_objects":2042,"num_object_clones":0,"num_object_copies":6126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2042,"num_write_kb":130688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"296'52","reported_seq":75,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:22:03.726116+0000","last_change":"2024-09-11T08:20:53.645194+0000","last_active":"2024-09-11T08:22:03.726116+0000","last_peered":"2024-09-11T08:22:03.726116+0000","last_clean":"2024-09-11T08:22:03.726116+0000","last_became_active":"2024-09-11T08:20:52.086843+0000","last_became_peered":"2024-09-11T08:20:52.086843+0000","last_unstale":"2024-09-11T08:22:03.726116+0000","last_undegraded":"2024-09-11T08:22:03.726116+0000","last_fullsized":"2024-09-11T08:22:03.726116+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'51","last_scrub_stamp":"2024-09-11T08:20:53.645121+0000","last_deep_scrub":"190'39","last_deep_scrub_stamp":"2024-09-11T08:15:52.847968+0000","last_clean_scrub_stamp":"2024-09-11T08:20:53.645121+0000","objects_scrubbed":10,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:09.785967+0000","scrub_duration":0.0034138189999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"59'1","reported_seq":333,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.174550+0000","last_change":"2024-09-11T08:21:46.049356+0000","last_active":"2024-09-11T08:21:59.174550+0000","last_peered":"2024-09-11T08:21:59.174550+0000","last_clean":"2024-09-11T08:21:59.174550+0000","last_became_active":"2024-09-11T08:07:30.047513+0000","last_became_peered":"2024-09-11T08:07:30.047513+0000","last_unstale":"2024-09-11T08:21:59.174550+0000","last_undegraded":"2024-09-11T08:21:59.174550+0000","last_fullsized":"2024-09-11T08:21:59.174550+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-11T08:21:46.049283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:21:46.049283+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:46.493529+0000","scrub_duration":0.0017569549999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":67,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.248767+0000","last_change":"2024-09-11T08:21:49.427128+0000","last_active":"2024-09-11T08:21:59.248767+0000","last_peered":"2024-09-11T08:21:59.248767+0000","last_clean":"2024-09-11T08:21:59.248767+0000","last_became_active":"2024-09-11T08:19:29.142111+0000","last_became_peered":"2024-09-11T08:19:29.142111+0000","last_unstale":"2024-09-11T08:21:59.248767+0000","last_undegraded":"2024-09-11T08:21:59.248767+0000","last_fullsized":"2024-09-11T08:21:59.248767+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-11T08:21:49.427071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:21:49.427071+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:16.477524+0000","scrub_duration":0.001771604,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":319,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.447701+0000","last_change":"2024-09-11T08:21:48.600637+0000","last_active":"2024-09-11T08:21:59.447701+0000","last_peered":"2024-09-11T08:21:59.447701+0000","last_clean":"2024-09-11T08:21:59.447701+0000","last_became_active":"2024-09-11T08:07:34.057154+0000","last_became_peered":"2024-09-11T08:07:34.057154+0000","last_unstale":"2024-09-11T08:21:59.447701+0000","last_undegraded":"2024-09-11T08:21:59.447701+0000","last_fullsized":"2024-09-11T08:21:59.447701+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:48.600546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:10:37.957096+0000","last_clean_scrub_stamp":"2024-09-11T08:21:48.600546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:55.974658+0000","scrub_duration":0.00136414,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":345,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.175595+0000","last_change":"2024-09-11T08:20:55.800959+0000","last_active":"2024-09-11T08:21:59.175595+0000","last_peered":"2024-09-11T08:21:59.175595+0000","last_clean":"2024-09-11T08:21:59.175595+0000","last_became_active":"2024-09-11T08:15:56.525099+0000","last_became_peered":"2024-09-11T08:15:56.525099+0000","last_unstale":"2024-09-11T08:21:59.175595+0000","last_undegraded":"2024-09-11T08:21:59.175595+0000","last_fullsized":"2024-09-11T08:21:59.175595+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:55.800856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:20:55.800856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:07.233576+0000","scrub_duration":0.008329276,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"314'1329","reported_seq":1343,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-11T08:22:08.217034+0000","last_change":"2024-09-11T08:21:47.139031+0000","last_active":"2024-09-11T08:22:08.217034+0000","last_peered":"2024-09-11T08:22:08.217034+0000","last_clean":"2024-09-11T08:22:08.217034+0000","last_became_active":"2024-09-11T08:21:47.138720+0000","last_became_peered":"2024-09-11T08:21:47.138720+0000","last_unstale":"2024-09-11T08:22:08.217034+0000","last_undegraded":"2024-09-11T08:22:08.217034+0000","last_fullsized":"2024-09-11T08:22:08.217034+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":308,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_clean_scrub_stamp":"2024-09-11T08:21:46.117341+0000","objects_scrubbed":0,"log_size":1329,"log_dups_size":0,"ondisk_log_size":1329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:53.290403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87097344,"num_objects":1329,"num_object_clones":0,"num_object_copies":3987,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1329,"num_write_kb":85056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"239'17","reported_seq":422,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:22:00.123708+0000","last_change":"2024-09-11T08:21:53.919207+0000","last_active":"2024-09-11T08:22:00.123708+0000","last_peered":"2024-09-11T08:22:00.123708+0000","last_clean":"2024-09-11T08:22:00.123708+0000","last_became_active":"2024-09-11T08:19:29.161554+0000","last_became_peered":"2024-09-11T08:19:29.161554+0000","last_unstale":"2024-09-11T08:22:00.123708+0000","last_undegraded":"2024-09-11T08:22:00.123708+0000","last_fullsized":"2024-09-11T08:22:00.123708+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'17","last_scrub_stamp":"2024-09-11T08:21:53.919145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:21:53.919145+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:14.995825+0000","scrub_duration":0.0026285179999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":337,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:22:00.122730+0000","last_change":"2024-09-11T08:21:10.701859+0000","last_active":"2024-09-11T08:22:00.122730+0000","last_peered":"2024-09-11T08:22:00.122730+0000","last_clean":"2024-09-11T08:22:00.122730+0000","last_became_active":"2024-09-11T08:07:30.047014+0000","last_became_peered":"2024-09-11T08:07:30.047014+0000","last_unstale":"2024-09-11T08:22:00.122730+0000","last_undegraded":"2024-09-11T08:22:00.122730+0000","last_fullsized":"2024-09-11T08:22:00.122730+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:10.701767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:21:10.701767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:20.537530+0000","scrub_duration":0.0056318200000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"127'2","reported_seq":363,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:22:03.267413+0000","last_change":"2024-09-11T08:22:03.267356+0000","last_active":"2024-09-11T08:22:03.267413+0000","last_peered":"2024-09-11T08:22:03.267413+0000","last_clean":"2024-09-11T08:22:03.267413+0000","last_became_active":"2024-09-11T08:07:30.045009+0000","last_became_peered":"2024-09-11T08:07:30.045009+0000","last_unstale":"2024-09-11T08:22:03.267413+0000","last_undegraded":"2024-09-11T08:22:03.267413+0000","last_fullsized":"2024-09-11T08:22:03.267413+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'2","last_scrub_stamp":"2024-09-11T08:22:03.267269+0000","last_deep_scrub":"127'2","last_deep_scrub_stamp":"2024-09-11T08:15:01.938497+0000","last_clean_scrub_stamp":"2024-09-11T08:22:03.267269+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:04.866068+0000","scrub_duration":0.0040114060000000003,"objects_trimmed":0,"snaptrim_duration":0.001859533,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":382,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:22:00.200991+0000","last_change":"2024-09-11T08:22:00.200933+0000","last_active":"2024-09-11T08:22:00.200991+0000","last_peered":"2024-09-11T08:22:00.200991+0000","last_clean":"2024-09-11T08:22:00.200991+0000","last_became_active":"2024-09-11T08:19:29.145168+0000","last_became_peered":"2024-09-11T08:19:29.145168+0000","last_unstale":"2024-09-11T08:22:00.200991+0000","last_undegraded":"2024-09-11T08:22:00.200991+0000","last_fullsized":"2024-09-11T08:22:00.200991+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:22:00.200872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:22:00.200872+0000","last_clean_scrub_stamp":"2024-09-11T08:22:00.200872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:01.894647+0000","scrub_duration":0.0020630689999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":49,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.247009+0000","last_change":"2024-09-11T08:21:45.173035+0000","last_active":"2024-09-11T08:21:59.247009+0000","last_peered":"2024-09-11T08:21:59.247009+0000","last_clean":"2024-09-11T08:21:59.247009+0000","last_became_active":"2024-09-11T08:20:16.108674+0000","last_became_peered":"2024-09-11T08:20:16.108674+0000","last_unstale":"2024-09-11T08:21:59.247009+0000","last_undegraded":"2024-09-11T08:21:59.247009+0000","last_fullsized":"2024-09-11T08:21:59.247009+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:45.172938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:58.796217+0000","last_clean_scrub_stamp":"2024-09-11T08:21:45.172938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:11.469254+0000","scrub_duration":0.0013497769999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.8","version":"314'1455","reported_seq":1469,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-11T08:22:08.761717+0000","last_change":"2024-09-11T08:21:47.139819+0000","last_active":"2024-09-11T08:22:08.761717+0000","last_peered":"2024-09-11T08:22:08.761717+0000","last_clean":"2024-09-11T08:22:08.761717+0000","last_became_active":"2024-09-11T08:21:47.139475+0000","last_became_peered":"2024-09-11T08:21:47.139475+0000","last_unstale":"2024-09-11T08:22:08.761717+0000","last_undegraded":"2024-09-11T08:22:08.761717+0000","last_fullsized":"2024-09-11T08:22:08.761717+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":308,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_clean_scrub_stamp":"2024-09-11T08:21:46.117341+0000","objects_scrubbed":0,"log_size":1455,"log_dups_size":0,"ondisk_log_size":1455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:00.885764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95354880,"num_objects":1455,"num_object_clones":0,"num_object_copies":4365,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1455,"num_write_kb":93120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"287'30","reported_seq":363,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.447768+0000","last_change":"2024-09-11T08:20:24.667229+0000","last_active":"2024-09-11T08:21:59.447768+0000","last_peered":"2024-09-11T08:21:59.447768+0000","last_clean":"2024-09-11T08:21:59.447768+0000","last_became_active":"2024-09-11T08:07:32.055407+0000","last_became_peered":"2024-09-11T08:07:32.055407+0000","last_unstale":"2024-09-11T08:21:59.447768+0000","last_undegraded":"2024-09-11T08:21:59.447768+0000","last_fullsized":"2024-09-11T08:21:59.447768+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"287'30","last_scrub_stamp":"2024-09-11T08:20:24.667139+0000","last_deep_scrub":"287'30","last_deep_scrub_stamp":"2024-09-11T08:20:24.667139+0000","last_clean_scrub_stamp":"2024-09-11T08:20:24.667139+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.030013558999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"59'1","reported_seq":330,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.447308+0000","last_change":"2024-09-11T08:20:53.779883+0000","last_active":"2024-09-11T08:21:59.447308+0000","last_peered":"2024-09-11T08:21:59.447308+0000","last_clean":"2024-09-11T08:21:59.447308+0000","last_became_active":"2024-09-11T08:07:30.050504+0000","last_became_peered":"2024-09-11T08:07:30.050504+0000","last_unstale":"2024-09-11T08:21:59.447308+0000","last_undegraded":"2024-09-11T08:21:59.447308+0000","last_fullsized":"2024-09-11T08:21:59.447308+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-11T08:20:53.779281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:20:53.779281+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:21.136677+0000","scrub_duration":0.0097612610000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"224'6","reported_seq":989,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-11T08:22:06.688828+0000","last_change":"2024-09-11T08:21:50.580561+0000","last_active":"2024-09-11T08:22:06.688828+0000","last_peered":"2024-09-11T08:22:06.688828+0000","last_clean":"2024-09-11T08:22:06.688828+0000","last_became_active":"2024-09-11T08:14:24.793048+0000","last_became_peered":"2024-09-11T08:14:24.793048+0000","last_unstale":"2024-09-11T08:22:06.688828+0000","last_undegraded":"2024-09-11T08:22:06.688828+0000","last_fullsized":"2024-09-11T08:22:06.688828+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'6","last_scrub_stamp":"2024-09-11T08:21:50.580504+0000","last_deep_scrub":"224'6","last_deep_scrub_stamp":"2024-09-11T08:16:55.778284+0000","last_clean_scrub_stamp":"2024-09-11T08:21:50.580504+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:07.461120+0000","scrub_duration":0.0015158470000000001,"objects_trimmed":0,"snaptrim_duration":0.000318821,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":469,"num_read_kb":464,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":63,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.249270+0000","last_change":"2024-09-11T08:20:54.655099+0000","last_active":"2024-09-11T08:21:59.249270+0000","last_peered":"2024-09-11T08:21:59.249270+0000","last_clean":"2024-09-11T08:21:59.249270+0000","last_became_active":"2024-09-11T08:19:29.154711+0000","last_became_peered":"2024-09-11T08:19:29.154711+0000","last_unstale":"2024-09-11T08:21:59.249270+0000","last_undegraded":"2024-09-11T08:21:59.249270+0000","last_fullsized":"2024-09-11T08:21:59.249270+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:54.655033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:20:54.655033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.017642073000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":361,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:22:00.123277+0000","last_change":"2024-09-11T08:21:55.844761+0000","last_active":"2024-09-11T08:22:00.123277+0000","last_peered":"2024-09-11T08:22:00.123277+0000","last_clean":"2024-09-11T08:22:00.123277+0000","last_became_active":"2024-09-11T08:19:29.141234+0000","last_became_peered":"2024-09-11T08:19:29.141234+0000","last_unstale":"2024-09-11T08:22:00.123277+0000","last_undegraded":"2024-09-11T08:22:00.123277+0000","last_fullsized":"2024-09-11T08:22:00.123277+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:55.844704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:21:55.844704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:56.481011+0000","scrub_duration":0.001596515,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"314'2032","reported_seq":2045,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-11T08:22:12.927507+0000","last_change":"2024-09-11T08:21:47.143520+0000","last_active":"2024-09-11T08:22:12.927507+0000","last_peered":"2024-09-11T08:22:12.927507+0000","last_clean":"2024-09-11T08:22:12.927507+0000","last_became_active":"2024-09-11T08:21:47.143210+0000","last_became_peered":"2024-09-11T08:21:47.143210+0000","last_unstale":"2024-09-11T08:22:12.927507+0000","last_undegraded":"2024-09-11T08:22:12.927507+0000","last_fullsized":"2024-09-11T08:22:12.927507+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":308,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_clean_scrub_stamp":"2024-09-11T08:21:46.117341+0000","objects_scrubbed":0,"log_size":2032,"log_dups_size":0,"ondisk_log_size":2032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:05.437537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133169152,"num_objects":2032,"num_object_clones":0,"num_object_copies":6096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2032,"num_write_kb":130048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"239'33","reported_seq":405,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:22:01.316992+0000","last_change":"2024-09-11T08:22:01.316777+0000","last_active":"2024-09-11T08:22:01.316992+0000","last_peered":"2024-09-11T08:22:01.316992+0000","last_clean":"2024-09-11T08:22:01.316992+0000","last_became_active":"2024-09-11T08:14:24.789154+0000","last_became_peered":"2024-09-11T08:14:24.789154+0000","last_unstale":"2024-09-11T08:22:01.316992+0000","last_undegraded":"2024-09-11T08:22:01.316992+0000","last_fullsized":"2024-09-11T08:22:01.316992+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'33","last_scrub_stamp":"2024-09-11T08:22:01.316685+0000","last_deep_scrub":"84'21","last_deep_scrub_stamp":"2024-09-11T08:08:56.705306+0000","last_clean_scrub_stamp":"2024-09-11T08:22:01.316685+0000","objects_scrubbed":7,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:15.626047+0000","scrub_duration":0.0038599789999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":53,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.248680+0000","last_change":"2024-09-11T08:19:29.144089+0000","last_active":"2024-09-11T08:21:59.248680+0000","last_peered":"2024-09-11T08:21:59.248680+0000","last_clean":"2024-09-11T08:21:59.248680+0000","last_became_active":"2024-09-11T08:19:29.143505+0000","last_became_peered":"2024-09-11T08:19:29.143505+0000","last_unstale":"2024-09-11T08:21:59.248680+0000","last_undegraded":"2024-09-11T08:21:59.248680+0000","last_fullsized":"2024-09-11T08:21:59.248680+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:45.423258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:06.236119+0000","last_clean_scrub_stamp":"2024-09-11T08:18:45.423258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":152,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.175382+0000","last_change":"2024-09-11T08:21:48.553887+0000","last_active":"2024-09-11T08:21:59.175382+0000","last_peered":"2024-09-11T08:21:59.175382+0000","last_clean":"2024-09-11T08:21:59.175382+0000","last_became_active":"2024-09-11T08:15:56.523430+0000","last_became_peered":"2024-09-11T08:15:56.523430+0000","last_unstale":"2024-09-11T08:21:59.175382+0000","last_undegraded":"2024-09-11T08:21:59.175382+0000","last_fullsized":"2024-09-11T08:21:59.175382+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-11T08:21:48.553784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:05:21.231856+0000","last_clean_scrub_stamp":"2024-09-11T08:21:48.553784+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:57.895097+0000","scrub_duration":0.0021209100000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":156,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.176151+0000","last_change":"2024-09-11T08:20:25.775943+0000","last_active":"2024-09-11T08:21:59.176151+0000","last_peered":"2024-09-11T08:21:59.176151+0000","last_clean":"2024-09-11T08:21:59.176151+0000","last_became_active":"2024-09-11T08:19:29.147576+0000","last_became_peered":"2024-09-11T08:19:29.147576+0000","last_unstale":"2024-09-11T08:21:59.176151+0000","last_undegraded":"2024-09-11T08:21:59.176151+0000","last_fullsized":"2024-09-11T08:21:59.176151+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:25.775869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:20:25.775869+0000","last_clean_scrub_stamp":"2024-09-11T08:20:25.775869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004676014,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":63,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.249346+0000","last_change":"2024-09-11T08:20:29.154000+0000","last_active":"2024-09-11T08:21:59.249346+0000","last_peered":"2024-09-11T08:21:59.249346+0000","last_clean":"2024-09-11T08:21:59.249346+0000","last_became_active":"2024-09-11T08:19:29.152071+0000","last_became_peered":"2024-09-11T08:19:29.152071+0000","last_unstale":"2024-09-11T08:21:59.249346+0000","last_undegraded":"2024-09-11T08:21:59.249346+0000","last_fullsized":"2024-09-11T08:21:59.249346+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:29.153871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:01.777429+0000","last_clean_scrub_stamp":"2024-09-11T08:20:29.153871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015196540000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"314'1995","reported_seq":2009,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-11T08:22:12.917067+0000","last_change":"2024-09-11T08:21:47.143063+0000","last_active":"2024-09-11T08:22:12.917067+0000","last_peered":"2024-09-11T08:22:12.917067+0000","last_clean":"2024-09-11T08:22:12.917067+0000","last_became_active":"2024-09-11T08:21:47.142740+0000","last_became_peered":"2024-09-11T08:21:47.142740+0000","last_unstale":"2024-09-11T08:22:12.917067+0000","last_undegraded":"2024-09-11T08:22:12.917067+0000","last_fullsized":"2024-09-11T08:22:12.917067+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":308,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_clean_scrub_stamp":"2024-09-11T08:21:46.117341+0000","objects_scrubbed":0,"log_size":1995,"log_dups_size":0,"ondisk_log_size":1995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:49.481125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130744320,"num_objects":1995,"num_object_clones":0,"num_object_copies":5985,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1995,"num_write_kb":127680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"239'24","reported_seq":370,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.174335+0000","last_change":"2024-09-11T08:21:09.811217+0000","last_active":"2024-09-11T08:21:59.174335+0000","last_peered":"2024-09-11T08:21:59.174335+0000","last_clean":"2024-09-11T08:21:59.174335+0000","last_became_active":"2024-09-11T08:21:09.782223+0000","last_became_peered":"2024-09-11T08:21:09.782223+0000","last_unstale":"2024-09-11T08:21:59.174335+0000","last_undegraded":"2024-09-11T08:21:59.174335+0000","last_fullsized":"2024-09-11T08:21:59.174335+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'24","last_scrub_stamp":"2024-09-11T08:18:40.895293+0000","last_deep_scrub":"88'16","last_deep_scrub_stamp":"2024-09-11T08:09:39.635984+0000","last_clean_scrub_stamp":"2024-09-11T08:18:40.895293+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010336458999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":331,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.174314+0000","last_change":"2024-09-11T08:21:03.653241+0000","last_active":"2024-09-11T08:21:59.174314+0000","last_peered":"2024-09-11T08:21:59.174314+0000","last_clean":"2024-09-11T08:21:59.174314+0000","last_became_active":"2024-09-11T08:14:24.789888+0000","last_became_peered":"2024-09-11T08:14:24.789888+0000","last_unstale":"2024-09-11T08:21:59.174314+0000","last_undegraded":"2024-09-11T08:21:59.174314+0000","last_fullsized":"2024-09-11T08:21:59.174314+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:03.653157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:36.417911+0000","last_clean_scrub_stamp":"2024-09-11T08:21:03.653157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:18.943216+0000","scrub_duration":0.0039236879999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":63,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.249255+0000","last_change":"2024-09-11T08:21:50.380816+0000","last_active":"2024-09-11T08:21:59.249255+0000","last_peered":"2024-09-11T08:21:59.249255+0000","last_clean":"2024-09-11T08:21:59.249255+0000","last_became_active":"2024-09-11T08:19:29.153976+0000","last_became_peered":"2024-09-11T08:19:29.153976+0000","last_unstale":"2024-09-11T08:21:59.249255+0000","last_undegraded":"2024-09-11T08:21:59.249255+0000","last_fullsized":"2024-09-11T08:21:59.249255+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:50.380756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:01.558340+0000","last_clean_scrub_stamp":"2024-09-11T08:21:50.380756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:10.607097+0000","scrub_duration":0.001327805,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":141,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.447616+0000","last_change":"2024-09-11T08:21:43.326785+0000","last_active":"2024-09-11T08:21:59.447616+0000","last_peered":"2024-09-11T08:21:59.447616+0000","last_clean":"2024-09-11T08:21:59.447616+0000","last_became_active":"2024-09-11T08:19:29.144592+0000","last_became_peered":"2024-09-11T08:19:29.144592+0000","last_unstale":"2024-09-11T08:21:59.447616+0000","last_undegraded":"2024-09-11T08:21:59.447616+0000","last_fullsized":"2024-09-11T08:21:59.447616+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:43.326717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:21:43.326717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:43.654572+0000","scrub_duration":0.0016517909999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3","version":"314'1394","reported_seq":1408,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-11T08:22:08.110130+0000","last_change":"2024-09-11T08:21:47.139861+0000","last_active":"2024-09-11T08:22:08.110130+0000","last_peered":"2024-09-11T08:22:08.110130+0000","last_clean":"2024-09-11T08:22:08.110130+0000","last_became_active":"2024-09-11T08:21:47.139581+0000","last_became_peered":"2024-09-11T08:21:47.139581+0000","last_unstale":"2024-09-11T08:22:08.110130+0000","last_undegraded":"2024-09-11T08:22:08.110130+0000","last_fullsized":"2024-09-11T08:22:08.110130+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":308,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_clean_scrub_stamp":"2024-09-11T08:21:46.117341+0000","objects_scrubbed":0,"log_size":1394,"log_dups_size":0,"ondisk_log_size":1394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:03.247374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91357184,"num_objects":1394,"num_object_clones":0,"num_object_copies":4182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1394,"num_write_kb":89216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"239'39","reported_seq":435,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.174872+0000","last_change":"2024-09-11T08:21:51.079782+0000","last_active":"2024-09-11T08:21:59.174872+0000","last_peered":"2024-09-11T08:21:59.174872+0000","last_clean":"2024-09-11T08:21:59.174872+0000","last_became_active":"2024-09-11T08:19:09.604525+0000","last_became_peered":"2024-09-11T08:19:09.604525+0000","last_unstale":"2024-09-11T08:21:59.174872+0000","last_undegraded":"2024-09-11T08:21:59.174872+0000","last_fullsized":"2024-09-11T08:21:59.174872+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'39","last_scrub_stamp":"2024-09-11T08:21:51.079716+0000","last_deep_scrub":"77'25","last_deep_scrub_stamp":"2024-09-11T08:08:32.458210+0000","last_clean_scrub_stamp":"2024-09-11T08:21:51.079716+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:59.763626+0000","scrub_duration":0.0024068399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":349,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.447331+0000","last_change":"2024-09-11T08:21:55.322957+0000","last_active":"2024-09-11T08:21:59.447331+0000","last_peered":"2024-09-11T08:21:59.447331+0000","last_clean":"2024-09-11T08:21:59.447331+0000","last_became_active":"2024-09-11T08:07:30.042555+0000","last_became_peered":"2024-09-11T08:07:30.042555+0000","last_unstale":"2024-09-11T08:21:59.447331+0000","last_undegraded":"2024-09-11T08:21:59.447331+0000","last_fullsized":"2024-09-11T08:21:59.447331+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:55.322891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:21:55.322891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:18.917383+0000","scrub_duration":0.00164072,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":330,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:22:00.123323+0000","last_change":"2024-09-11T08:21:09.706729+0000","last_active":"2024-09-11T08:22:00.123323+0000","last_peered":"2024-09-11T08:22:00.123323+0000","last_clean":"2024-09-11T08:22:00.123323+0000","last_became_active":"2024-09-11T08:07:34.054961+0000","last_became_peered":"2024-09-11T08:07:34.054961+0000","last_unstale":"2024-09-11T08:22:00.123323+0000","last_undegraded":"2024-09-11T08:22:00.123323+0000","last_fullsized":"2024-09-11T08:22:00.123323+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:09.706678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:21:09.706678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:25.130807+0000","scrub_duration":0.0038244720000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":347,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.247349+0000","last_change":"2024-09-11T08:21:02.823083+0000","last_active":"2024-09-11T08:21:59.247349+0000","last_peered":"2024-09-11T08:21:59.247349+0000","last_clean":"2024-09-11T08:21:59.247349+0000","last_became_active":"2024-09-11T08:07:36.071194+0000","last_became_peered":"2024-09-11T08:07:36.071194+0000","last_unstale":"2024-09-11T08:21:59.247349+0000","last_undegraded":"2024-09-11T08:21:59.247349+0000","last_fullsized":"2024-09-11T08:21:59.247349+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:02.822996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:07.930137+0000","last_clean_scrub_stamp":"2024-09-11T08:21:02.822996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:13.236302+0000","scrub_duration":0.005271441,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"314'1494","reported_seq":1508,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-11T08:22:08.656753+0000","last_change":"2024-09-11T08:21:47.139153+0000","last_active":"2024-09-11T08:22:08.656753+0000","last_peered":"2024-09-11T08:22:08.656753+0000","last_clean":"2024-09-11T08:22:08.656753+0000","last_became_active":"2024-09-11T08:21:47.138901+0000","last_became_peered":"2024-09-11T08:21:47.138901+0000","last_unstale":"2024-09-11T08:22:08.656753+0000","last_undegraded":"2024-09-11T08:22:08.656753+0000","last_fullsized":"2024-09-11T08:22:08.656753+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":308,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_clean_scrub_stamp":"2024-09-11T08:21:46.117341+0000","objects_scrubbed":0,"log_size":1494,"log_dups_size":0,"ondisk_log_size":1494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:15.143012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97910784,"num_objects":1494,"num_object_clones":0,"num_object_copies":4482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1494,"num_write_kb":95616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"239'25","reported_seq":350,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.174677+0000","last_change":"2024-09-11T08:20:16.527717+0000","last_active":"2024-09-11T08:21:59.174677+0000","last_peered":"2024-09-11T08:21:59.174677+0000","last_clean":"2024-09-11T08:21:59.174677+0000","last_became_active":"2024-09-11T08:07:32.043314+0000","last_became_peered":"2024-09-11T08:07:32.043314+0000","last_unstale":"2024-09-11T08:21:59.174677+0000","last_undegraded":"2024-09-11T08:21:59.174677+0000","last_fullsized":"2024-09-11T08:21:59.174677+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'25","last_scrub_stamp":"2024-09-11T08:20:16.527630+0000","last_deep_scrub":"224'15","last_deep_scrub_stamp":"2024-09-11T08:17:01.126478+0000","last_clean_scrub_stamp":"2024-09-11T08:20:16.527630+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0059554439999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":108,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.447735+0000","last_change":"2024-09-11T08:20:10.609260+0000","last_active":"2024-09-11T08:21:59.447735+0000","last_peered":"2024-09-11T08:21:59.447735+0000","last_clean":"2024-09-11T08:21:59.447735+0000","last_became_active":"2024-09-11T08:17:37.557278+0000","last_became_peered":"2024-09-11T08:17:37.557278+0000","last_unstale":"2024-09-11T08:21:59.447735+0000","last_undegraded":"2024-09-11T08:21:59.447735+0000","last_fullsized":"2024-09-11T08:21:59.447735+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:10.609170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:45.979847+0000","last_clean_scrub_stamp":"2024-09-11T08:20:10.609170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.017294936,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"271'14","reported_seq":574,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:22:03.725300+0000","last_change":"2024-09-11T08:21:59.247799+0000","last_active":"2024-09-11T08:22:03.725300+0000","last_peered":"2024-09-11T08:22:03.725300+0000","last_clean":"2024-09-11T08:22:03.725300+0000","last_became_active":"2024-09-11T08:19:29.153997+0000","last_became_peered":"2024-09-11T08:19:29.153997+0000","last_unstale":"2024-09-11T08:22:03.725300+0000","last_undegraded":"2024-09-11T08:22:03.725300+0000","last_fullsized":"2024-09-11T08:22:03.725300+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'14","last_scrub_stamp":"2024-09-11T08:21:59.247702+0000","last_deep_scrub":"171'12","last_deep_scrub_stamp":"2024-09-11T08:14:56.881299+0000","last_clean_scrub_stamp":"2024-09-11T08:21:59.247702+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:26.503385+0000","scrub_duration":0.0021085589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":308,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.174143+0000","last_change":"2024-09-11T08:20:50.427308+0000","last_active":"2024-09-11T08:21:59.174143+0000","last_peered":"2024-09-11T08:21:59.174143+0000","last_clean":"2024-09-11T08:21:59.174143+0000","last_became_active":"2024-09-11T08:07:36.074684+0000","last_became_peered":"2024-09-11T08:07:36.074684+0000","last_unstale":"2024-09-11T08:21:59.174143+0000","last_undegraded":"2024-09-11T08:21:59.174143+0000","last_fullsized":"2024-09-11T08:21:59.174143+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:50.427224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:14.613532+0000","last_clean_scrub_stamp":"2024-09-11T08:20:50.427224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028770409999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.5","version":"314'1359","reported_seq":1372,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-11T08:22:08.121168+0000","last_change":"2024-09-11T08:21:47.140488+0000","last_active":"2024-09-11T08:22:08.121168+0000","last_peered":"2024-09-11T08:22:08.121168+0000","last_clean":"2024-09-11T08:22:08.121168+0000","last_became_active":"2024-09-11T08:21:47.140224+0000","last_became_peered":"2024-09-11T08:21:47.140224+0000","last_unstale":"2024-09-11T08:22:08.121168+0000","last_undegraded":"2024-09-11T08:22:08.121168+0000","last_fullsized":"2024-09-11T08:22:08.121168+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":308,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_clean_scrub_stamp":"2024-09-11T08:21:46.117341+0000","objects_scrubbed":0,"log_size":1359,"log_dups_size":0,"ondisk_log_size":1359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:05.175594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89063424,"num_objects":1359,"num_object_clones":0,"num_object_copies":4077,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1359,"num_write_kb":86976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"239'15","reported_seq":346,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.447481+0000","last_change":"2024-09-11T08:21:10.722246+0000","last_active":"2024-09-11T08:21:59.447481+0000","last_peered":"2024-09-11T08:21:59.447481+0000","last_clean":"2024-09-11T08:21:59.447481+0000","last_became_active":"2024-09-11T08:07:32.054885+0000","last_became_peered":"2024-09-11T08:07:32.054885+0000","last_unstale":"2024-09-11T08:21:59.447481+0000","last_undegraded":"2024-09-11T08:21:59.447481+0000","last_fullsized":"2024-09-11T08:21:59.447481+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'15","last_scrub_stamp":"2024-09-11T08:21:10.722062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:21:10.722062+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:23.225006+0000","scrub_duration":0.010706927,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":63,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.248942+0000","last_change":"2024-09-11T08:21:00.730548+0000","last_active":"2024-09-11T08:21:59.248942+0000","last_peered":"2024-09-11T08:21:59.248942+0000","last_clean":"2024-09-11T08:21:59.248942+0000","last_became_active":"2024-09-11T08:19:29.143174+0000","last_became_peered":"2024-09-11T08:19:29.143174+0000","last_unstale":"2024-09-11T08:21:59.248942+0000","last_undegraded":"2024-09-11T08:21:59.248942+0000","last_fullsized":"2024-09-11T08:21:59.248942+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:00.730460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:21:00.730460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:17.761419+0000","scrub_duration":0.0102811,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":349,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.246913+0000","last_change":"2024-09-11T08:21:04.894391+0000","last_active":"2024-09-11T08:21:59.246913+0000","last_peered":"2024-09-11T08:21:59.246913+0000","last_clean":"2024-09-11T08:21:59.246913+0000","last_became_active":"2024-09-11T08:07:34.056606+0000","last_became_peered":"2024-09-11T08:07:34.056606+0000","last_unstale":"2024-09-11T08:21:59.246913+0000","last_undegraded":"2024-09-11T08:21:59.246913+0000","last_fullsized":"2024-09-11T08:21:59.246913+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:04.894259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:21:04.894259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:27.176768+0000","scrub_duration":0.013493084000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":344,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.447300+0000","last_change":"2024-09-11T08:21:45.541234+0000","last_active":"2024-09-11T08:21:59.447300+0000","last_peered":"2024-09-11T08:21:59.447300+0000","last_clean":"2024-09-11T08:21:59.447300+0000","last_became_active":"2024-09-11T08:15:56.518884+0000","last_became_peered":"2024-09-11T08:15:56.518884+0000","last_unstale":"2024-09-11T08:21:59.447300+0000","last_undegraded":"2024-09-11T08:21:59.447300+0000","last_fullsized":"2024-09-11T08:21:59.447300+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:45.541171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:18:32.278257+0000","last_clean_scrub_stamp":"2024-09-11T08:21:45.541171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:48.020855+0000","scrub_duration":0.0013210780000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"314'1486","reported_seq":1500,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-11T08:22:08.757965+0000","last_change":"2024-09-11T08:21:47.143324+0000","last_active":"2024-09-11T08:22:08.757965+0000","last_peered":"2024-09-11T08:22:08.757965+0000","last_clean":"2024-09-11T08:22:08.757965+0000","last_became_active":"2024-09-11T08:21:47.143034+0000","last_became_peered":"2024-09-11T08:21:47.143034+0000","last_unstale":"2024-09-11T08:22:08.757965+0000","last_undegraded":"2024-09-11T08:22:08.757965+0000","last_fullsized":"2024-09-11T08:22:08.757965+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":308,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_clean_scrub_stamp":"2024-09-11T08:21:46.117341+0000","objects_scrubbed":0,"log_size":1486,"log_dups_size":0,"ondisk_log_size":1486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:55.046330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97386496,"num_objects":1486,"num_object_clones":0,"num_object_copies":4458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1486,"num_write_kb":95104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"239'35","reported_seq":473,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.447255+0000","last_change":"2024-09-11T08:21:44.294440+0000","last_active":"2024-09-11T08:21:59.447255+0000","last_peered":"2024-09-11T08:21:59.447255+0000","last_clean":"2024-09-11T08:21:59.447255+0000","last_became_active":"2024-09-11T08:15:56.526961+0000","last_became_peered":"2024-09-11T08:15:56.526961+0000","last_unstale":"2024-09-11T08:21:59.447255+0000","last_undegraded":"2024-09-11T08:21:59.447255+0000","last_fullsized":"2024-09-11T08:21:59.447255+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'35","last_scrub_stamp":"2024-09-11T08:21:44.294374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:21:44.294374+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:59.282750+0000","scrub_duration":0.0025259789999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":156,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.175993+0000","last_change":"2024-09-11T08:20:09.859624+0000","last_active":"2024-09-11T08:21:59.175993+0000","last_peered":"2024-09-11T08:21:59.175993+0000","last_clean":"2024-09-11T08:21:59.175993+0000","last_became_active":"2024-09-11T08:19:29.152114+0000","last_became_peered":"2024-09-11T08:19:29.152114+0000","last_unstale":"2024-09-11T08:21:59.175993+0000","last_undegraded":"2024-09-11T08:21:59.175993+0000","last_fullsized":"2024-09-11T08:21:59.175993+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:09.859549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:44.385372+0000","last_clean_scrub_stamp":"2024-09-11T08:20:09.859549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0055679850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":362,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.246944+0000","last_change":"2024-09-11T08:20:53.860652+0000","last_active":"2024-09-11T08:21:59.246944+0000","last_peered":"2024-09-11T08:21:59.246944+0000","last_clean":"2024-09-11T08:21:59.246944+0000","last_became_active":"2024-09-11T08:19:29.178705+0000","last_became_peered":"2024-09-11T08:19:29.178705+0000","last_unstale":"2024-09-11T08:21:59.246944+0000","last_undegraded":"2024-09-11T08:21:59.246944+0000","last_fullsized":"2024-09-11T08:21:59.246944+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:53.860586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:20:53.860586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:19.805109+0000","scrub_duration":0.030572999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":136,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.175408+0000","last_change":"2024-09-11T08:20:21.780282+0000","last_active":"2024-09-11T08:21:59.175408+0000","last_peered":"2024-09-11T08:21:59.175408+0000","last_clean":"2024-09-11T08:21:59.175408+0000","last_became_active":"2024-09-11T08:15:56.518169+0000","last_became_peered":"2024-09-11T08:15:56.518169+0000","last_unstale":"2024-09-11T08:21:59.175408+0000","last_undegraded":"2024-09-11T08:21:59.175408+0000","last_fullsized":"2024-09-11T08:21:59.175408+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:21.780224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:20:21.780224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0071716779999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"314'2024","reported_seq":2038,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-11T08:22:12.932527+0000","last_change":"2024-09-11T08:21:47.141356+0000","last_active":"2024-09-11T08:22:12.932527+0000","last_peered":"2024-09-11T08:22:12.932527+0000","last_clean":"2024-09-11T08:22:12.932527+0000","last_became_active":"2024-09-11T08:21:47.141093+0000","last_became_peered":"2024-09-11T08:21:47.141093+0000","last_unstale":"2024-09-11T08:22:12.932527+0000","last_undegraded":"2024-09-11T08:22:12.932527+0000","last_fullsized":"2024-09-11T08:22:12.932527+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":308,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_clean_scrub_stamp":"2024-09-11T08:21:46.117341+0000","objects_scrubbed":0,"log_size":2024,"log_dups_size":0,"ondisk_log_size":2024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:01.240099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132644864,"num_objects":2024,"num_object_clones":0,"num_object_copies":6072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2024,"num_write_kb":129536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"239'23","reported_seq":359,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:22:04.128735+0000","last_change":"2024-09-11T08:22:04.128665+0000","last_active":"2024-09-11T08:22:04.128735+0000","last_peered":"2024-09-11T08:22:04.128735+0000","last_clean":"2024-09-11T08:22:04.128735+0000","last_became_active":"2024-09-11T08:07:32.048164+0000","last_became_peered":"2024-09-11T08:07:32.048164+0000","last_unstale":"2024-09-11T08:22:04.128735+0000","last_undegraded":"2024-09-11T08:22:04.128735+0000","last_fullsized":"2024-09-11T08:22:04.128735+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'23","last_scrub_stamp":"2024-09-11T08:22:04.128578+0000","last_deep_scrub":"145'17","last_deep_scrub_stamp":"2024-09-11T08:13:08.160848+0000","last_clean_scrub_stamp":"2024-09-11T08:22:04.128578+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:11.066610+0000","scrub_duration":0.0059359520000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":146,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.176288+0000","last_change":"2024-09-11T08:21:06.262697+0000","last_active":"2024-09-11T08:21:59.176288+0000","last_peered":"2024-09-11T08:21:59.176288+0000","last_clean":"2024-09-11T08:21:59.176288+0000","last_became_active":"2024-09-11T08:15:56.525845+0000","last_became_peered":"2024-09-11T08:15:56.525845+0000","last_unstale":"2024-09-11T08:21:59.176288+0000","last_undegraded":"2024-09-11T08:21:59.176288+0000","last_fullsized":"2024-09-11T08:21:59.176288+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:06.262572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:21:06.262572+0000","last_clean_scrub_stamp":"2024-09-11T08:21:06.262572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:08.699491+0000","scrub_duration":0.007163272,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":329,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.246680+0000","last_change":"2024-09-11T08:20:59.876602+0000","last_active":"2024-09-11T08:21:59.246680+0000","last_peered":"2024-09-11T08:21:59.246680+0000","last_clean":"2024-09-11T08:21:59.246680+0000","last_became_active":"2024-09-11T08:07:34.057479+0000","last_became_peered":"2024-09-11T08:07:34.057479+0000","last_unstale":"2024-09-11T08:21:59.246680+0000","last_undegraded":"2024-09-11T08:21:59.246680+0000","last_fullsized":"2024-09-11T08:21:59.246680+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:59.876505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:20:59.876505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:24.461114+0000","scrub_duration":0.0064578550000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":355,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.175207+0000","last_change":"2024-09-11T08:21:43.750897+0000","last_active":"2024-09-11T08:21:59.175207+0000","last_peered":"2024-09-11T08:21:59.175207+0000","last_clean":"2024-09-11T08:21:59.175207+0000","last_became_active":"2024-09-11T08:15:56.522624+0000","last_became_peered":"2024-09-11T08:15:56.522624+0000","last_unstale":"2024-09-11T08:21:59.175207+0000","last_undegraded":"2024-09-11T08:21:59.175207+0000","last_fullsized":"2024-09-11T08:21:59.175207+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:43.750844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:21:43.750844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:01.039674+0000","scrub_duration":0.0015776570000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"314'1444","reported_seq":1458,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-11T08:22:08.211601+0000","last_change":"2024-09-11T08:21:47.141892+0000","last_active":"2024-09-11T08:22:08.211601+0000","last_peered":"2024-09-11T08:22:08.211601+0000","last_clean":"2024-09-11T08:22:08.211601+0000","last_became_active":"2024-09-11T08:21:47.141518+0000","last_became_peered":"2024-09-11T08:21:47.141518+0000","last_unstale":"2024-09-11T08:22:08.211601+0000","last_undegraded":"2024-09-11T08:22:08.211601+0000","last_fullsized":"2024-09-11T08:22:08.211601+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":308,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_clean_scrub_stamp":"2024-09-11T08:21:46.117341+0000","objects_scrubbed":0,"log_size":1444,"log_dups_size":0,"ondisk_log_size":1444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:57.785263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94633984,"num_objects":1444,"num_object_clones":0,"num_object_copies":4332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1444,"num_write_kb":92416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"239'27","reported_seq":79,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.248477+0000","last_change":"2024-09-11T08:20:08.785799+0000","last_active":"2024-09-11T08:21:59.248477+0000","last_peered":"2024-09-11T08:21:59.248477+0000","last_clean":"2024-09-11T08:21:59.248477+0000","last_became_active":"2024-09-11T08:19:29.147000+0000","last_became_peered":"2024-09-11T08:19:29.147000+0000","last_unstale":"2024-09-11T08:21:59.248477+0000","last_undegraded":"2024-09-11T08:21:59.248477+0000","last_fullsized":"2024-09-11T08:21:59.248477+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'27","last_scrub_stamp":"2024-09-11T08:20:08.785703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:20:08.785703+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005828789,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":152,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.176266+0000","last_change":"2024-09-11T08:21:46.555670+0000","last_active":"2024-09-11T08:21:59.176266+0000","last_peered":"2024-09-11T08:21:59.176266+0000","last_clean":"2024-09-11T08:21:59.176266+0000","last_became_active":"2024-09-11T08:15:56.515328+0000","last_became_peered":"2024-09-11T08:15:56.515328+0000","last_unstale":"2024-09-11T08:21:59.176266+0000","last_undegraded":"2024-09-11T08:21:59.176266+0000","last_fullsized":"2024-09-11T08:21:59.176266+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-11T08:21:46.555601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:21:46.555601+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:04.642167+0000","scrub_duration":0.0017840829999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"209'11","reported_seq":214,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:22:03.724165+0000","last_change":"2024-09-11T08:20:51.158341+0000","last_active":"2024-09-11T08:22:03.724165+0000","last_peered":"2024-09-11T08:22:03.724165+0000","last_clean":"2024-09-11T08:22:03.724165+0000","last_became_active":"2024-09-11T08:15:56.524513+0000","last_became_peered":"2024-09-11T08:15:56.524513+0000","last_unstale":"2024-09-11T08:22:03.724165+0000","last_undegraded":"2024-09-11T08:22:03.724165+0000","last_fullsized":"2024-09-11T08:22:03.724165+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"209'11","last_scrub_stamp":"2024-09-11T08:20:51.158199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:20:51.158199+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:17.301119+0000","scrub_duration":0.015653383,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":63,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.448659+0000","last_change":"2024-09-11T08:21:59.448626+0000","last_active":"2024-09-11T08:21:59.448659+0000","last_peered":"2024-09-11T08:21:59.448659+0000","last_clean":"2024-09-11T08:21:59.448659+0000","last_became_active":"2024-09-11T08:19:29.163872+0000","last_became_peered":"2024-09-11T08:19:29.163872+0000","last_unstale":"2024-09-11T08:21:59.448659+0000","last_undegraded":"2024-09-11T08:21:59.448659+0000","last_fullsized":"2024-09-11T08:21:59.448659+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:59.448573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:03.484755+0000","last_clean_scrub_stamp":"2024-09-11T08:21:59.448573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:25.059461+0000","scrub_duration":0.0028812519999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":373,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.447203+0000","last_change":"2024-09-11T08:21:58.281147+0000","last_active":"2024-09-11T08:21:59.447203+0000","last_peered":"2024-09-11T08:21:59.447203+0000","last_clean":"2024-09-11T08:21:59.447203+0000","last_became_active":"2024-09-11T08:19:29.193317+0000","last_became_peered":"2024-09-11T08:19:29.193317+0000","last_unstale":"2024-09-11T08:21:59.447203+0000","last_undegraded":"2024-09-11T08:21:59.447203+0000","last_fullsized":"2024-09-11T08:21:59.447203+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:58.281082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:21:58.281082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:12.638469+0000","scrub_duration":0.001499371,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"239'15","reported_seq":386,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.175843+0000","last_change":"2024-09-11T08:20:54.764391+0000","last_active":"2024-09-11T08:21:59.175843+0000","last_peered":"2024-09-11T08:21:59.175843+0000","last_clean":"2024-09-11T08:21:59.175843+0000","last_became_active":"2024-09-11T08:15:56.526325+0000","last_became_peered":"2024-09-11T08:15:56.526325+0000","last_unstale":"2024-09-11T08:21:59.175843+0000","last_undegraded":"2024-09-11T08:21:59.175843+0000","last_fullsized":"2024-09-11T08:21:59.175843+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'15","last_scrub_stamp":"2024-09-11T08:20:54.764293+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-09-11T08:09:03.418457+0000","last_clean_scrub_stamp":"2024-09-11T08:20:54.764293+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:05.029319+0000","scrub_duration":0.0039192439999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":63,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.249128+0000","last_change":"2024-09-11T08:21:54.344679+0000","last_active":"2024-09-11T08:21:59.249128+0000","last_peered":"2024-09-11T08:21:59.249128+0000","last_clean":"2024-09-11T08:21:59.249128+0000","last_became_active":"2024-09-11T08:19:29.143925+0000","last_became_peered":"2024-09-11T08:19:29.143925+0000","last_unstale":"2024-09-11T08:21:59.249128+0000","last_undegraded":"2024-09-11T08:21:59.249128+0000","last_fullsized":"2024-09-11T08:21:59.249128+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:54.344520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:53.548967+0000","last_clean_scrub_stamp":"2024-09-11T08:21:54.344520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:03.623437+0000","scrub_duration":0.001658814,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":271,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.176199+0000","last_change":"2024-09-11T08:21:03.199721+0000","last_active":"2024-09-11T08:21:59.176199+0000","last_peered":"2024-09-11T08:21:59.176199+0000","last_clean":"2024-09-11T08:21:59.176199+0000","last_became_active":"2024-09-11T08:14:24.790678+0000","last_became_peered":"2024-09-11T08:14:24.790678+0000","last_unstale":"2024-09-11T08:21:59.176199+0000","last_undegraded":"2024-09-11T08:21:59.176199+0000","last_fullsized":"2024-09-11T08:21:59.176199+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:03.199669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:21:03.199669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:27.957305+0000","scrub_duration":0.0011985959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"239'19","reported_seq":387,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.447798+0000","last_change":"2024-09-11T08:20:14.603209+0000","last_active":"2024-09-11T08:21:59.447798+0000","last_peered":"2024-09-11T08:21:59.447798+0000","last_clean":"2024-09-11T08:21:59.447798+0000","last_became_active":"2024-09-11T08:15:56.514793+0000","last_became_peered":"2024-09-11T08:15:56.514793+0000","last_unstale":"2024-09-11T08:21:59.447798+0000","last_undegraded":"2024-09-11T08:21:59.447798+0000","last_fullsized":"2024-09-11T08:21:59.447798+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'19","last_scrub_stamp":"2024-09-11T08:20:14.603118+0000","last_deep_scrub":"84'13","last_deep_scrub_stamp":"2024-09-11T08:08:57.835634+0000","last_clean_scrub_stamp":"2024-09-11T08:20:14.603118+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042720240000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":294,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.175904+0000","last_change":"2024-09-11T08:20:20.819257+0000","last_active":"2024-09-11T08:21:59.175904+0000","last_peered":"2024-09-11T08:21:59.175904+0000","last_clean":"2024-09-11T08:21:59.175904+0000","last_became_active":"2024-09-11T08:14:24.786021+0000","last_became_peered":"2024-09-11T08:14:24.786021+0000","last_unstale":"2024-09-11T08:21:59.175904+0000","last_undegraded":"2024-09-11T08:21:59.175904+0000","last_fullsized":"2024-09-11T08:21:59.175904+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:20.819154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:30.471620+0000","last_clean_scrub_stamp":"2024-09-11T08:20:20.819154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018806490000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":347,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.175799+0000","last_change":"2024-09-11T08:20:22.889011+0000","last_active":"2024-09-11T08:21:59.175799+0000","last_peered":"2024-09-11T08:21:59.175799+0000","last_clean":"2024-09-11T08:21:59.175799+0000","last_became_active":"2024-09-11T08:15:56.525717+0000","last_became_peered":"2024-09-11T08:15:56.525717+0000","last_unstale":"2024-09-11T08:21:59.175799+0000","last_undegraded":"2024-09-11T08:21:59.175799+0000","last_fullsized":"2024-09-11T08:21:59.175799+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:22.888936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:04.370734+0000","last_clean_scrub_stamp":"2024-09-11T08:20:22.888936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.01485916,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":357,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.246843+0000","last_change":"2024-09-11T08:21:43.156155+0000","last_active":"2024-09-11T08:21:59.246843+0000","last_peered":"2024-09-11T08:21:59.246843+0000","last_clean":"2024-09-11T08:21:59.246843+0000","last_became_active":"2024-09-11T08:15:56.525120+0000","last_became_peered":"2024-09-11T08:15:56.525120+0000","last_unstale":"2024-09-11T08:21:59.246843+0000","last_undegraded":"2024-09-11T08:21:59.246843+0000","last_fullsized":"2024-09-11T08:21:59.246843+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:43.156101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:54.184372+0000","last_clean_scrub_stamp":"2024-09-11T08:21:43.156101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:07.325849+0000","scrub_duration":0.0017268610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"313'19","reported_seq":77,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:22:03.458574+0000","last_change":"2024-09-11T08:22:03.458497+0000","last_active":"2024-09-11T08:22:03.458574+0000","last_peered":"2024-09-11T08:22:03.458574+0000","last_clean":"2024-09-11T08:22:03.458574+0000","last_became_active":"2024-09-11T08:19:29.153740+0000","last_became_peered":"2024-09-11T08:19:29.153740+0000","last_unstale":"2024-09-11T08:22:03.458574+0000","last_undegraded":"2024-09-11T08:22:03.458574+0000","last_fullsized":"2024-09-11T08:22:03.458574+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"313'19","last_scrub_stamp":"2024-09-11T08:22:03.458402+0000","last_deep_scrub":"313'19","last_deep_scrub_stamp":"2024-09-11T08:22:03.458402+0000","last_clean_scrub_stamp":"2024-09-11T08:22:03.458402+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:06.902001+0000","scrub_duration":0.0061927930000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":350,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.447454+0000","last_change":"2024-09-11T08:21:05.777148+0000","last_active":"2024-09-11T08:21:59.447454+0000","last_peered":"2024-09-11T08:21:59.447454+0000","last_clean":"2024-09-11T08:21:59.447454+0000","last_became_active":"2024-09-11T08:15:56.513278+0000","last_became_peered":"2024-09-11T08:15:56.513278+0000","last_unstale":"2024-09-11T08:21:59.447454+0000","last_undegraded":"2024-09-11T08:21:59.447454+0000","last_fullsized":"2024-09-11T08:21:59.447454+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:05.777069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:52.661591+0000","last_clean_scrub_stamp":"2024-09-11T08:21:05.777069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:33.267208+0000","scrub_duration":0.00237024,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"239'13","reported_seq":521,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:22:03.724339+0000","last_change":"2024-09-11T08:21:52.595127+0000","last_active":"2024-09-11T08:22:03.724339+0000","last_peered":"2024-09-11T08:22:03.724339+0000","last_clean":"2024-09-11T08:22:03.724339+0000","last_became_active":"2024-09-11T08:17:37.558224+0000","last_became_peered":"2024-09-11T08:17:37.558224+0000","last_unstale":"2024-09-11T08:22:03.724339+0000","last_undegraded":"2024-09-11T08:22:03.724339+0000","last_fullsized":"2024-09-11T08:22:03.724339+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'13","last_scrub_stamp":"2024-09-11T08:21:52.595045+0000","last_deep_scrub":"222'12","last_deep_scrub_stamp":"2024-09-11T08:16:52.592856+0000","last_clean_scrub_stamp":"2024-09-11T08:21:52.595045+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:55.458513+0000","scrub_duration":0.00191333,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":328,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.174306+0000","last_change":"2024-09-11T08:20:54.449873+0000","last_active":"2024-09-11T08:21:59.174306+0000","last_peered":"2024-09-11T08:21:59.174306+0000","last_clean":"2024-09-11T08:21:59.174306+0000","last_became_active":"2024-09-11T08:19:29.147682+0000","last_became_peered":"2024-09-11T08:19:29.147682+0000","last_unstale":"2024-09-11T08:21:59.174306+0000","last_undegraded":"2024-09-11T08:21:59.174306+0000","last_fullsized":"2024-09-11T08:21:59.174306+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:54.449786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:53.175994+0000","last_clean_scrub_stamp":"2024-09-11T08:20:54.449786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015389500000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"239'20","reported_seq":196,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.246823+0000","last_change":"2024-09-11T08:21:44.200657+0000","last_active":"2024-09-11T08:21:59.246823+0000","last_peered":"2024-09-11T08:21:59.246823+0000","last_clean":"2024-09-11T08:21:59.246823+0000","last_became_active":"2024-09-11T08:17:37.554357+0000","last_became_peered":"2024-09-11T08:17:37.554357+0000","last_unstale":"2024-09-11T08:21:59.246823+0000","last_undegraded":"2024-09-11T08:21:59.246823+0000","last_fullsized":"2024-09-11T08:21:59.246823+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'20","last_scrub_stamp":"2024-09-11T08:21:44.200602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:21:44.200602+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:01.573512+0000","scrub_duration":0.0020864519999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":146,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.176293+0000","last_change":"2024-09-11T08:21:53.562461+0000","last_active":"2024-09-11T08:21:59.176293+0000","last_peered":"2024-09-11T08:21:59.176293+0000","last_clean":"2024-09-11T08:21:59.176293+0000","last_became_active":"2024-09-11T08:15:56.518515+0000","last_became_peered":"2024-09-11T08:15:56.518515+0000","last_unstale":"2024-09-11T08:21:59.176293+0000","last_undegraded":"2024-09-11T08:21:59.176293+0000","last_fullsized":"2024-09-11T08:21:59.176293+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:53.562379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:21:53.562379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:59.004320+0000","scrub_duration":0.0015591660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"271'10","reported_seq":99,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:22:03.724303+0000","last_change":"2024-09-11T08:20:16.922523+0000","last_active":"2024-09-11T08:22:03.724303+0000","last_peered":"2024-09-11T08:22:03.724303+0000","last_clean":"2024-09-11T08:22:03.724303+0000","last_became_active":"2024-09-11T08:19:29.165183+0000","last_became_peered":"2024-09-11T08:19:29.165183+0000","last_unstale":"2024-09-11T08:22:03.724303+0000","last_undegraded":"2024-09-11T08:22:03.724303+0000","last_fullsized":"2024-09-11T08:22:03.724303+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'10","last_scrub_stamp":"2024-09-11T08:20:16.922384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:20:16.922384+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0084094979999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":334,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.447591+0000","last_change":"2024-09-11T08:20:23.600147+0000","last_active":"2024-09-11T08:21:59.447591+0000","last_peered":"2024-09-11T08:21:59.447591+0000","last_clean":"2024-09-11T08:21:59.447591+0000","last_became_active":"2024-09-11T08:15:56.513949+0000","last_became_peered":"2024-09-11T08:15:56.513949+0000","last_unstale":"2024-09-11T08:21:59.447591+0000","last_undegraded":"2024-09-11T08:21:59.447591+0000","last_fullsized":"2024-09-11T08:21:59.447591+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:23.600056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:20:23.600056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003973341,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"239'19","reported_seq":346,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.174342+0000","last_change":"2024-09-11T08:20:09.525188+0000","last_active":"2024-09-11T08:21:59.174342+0000","last_peered":"2024-09-11T08:21:59.174342+0000","last_clean":"2024-09-11T08:21:59.174342+0000","last_became_active":"2024-09-11T08:07:32.047981+0000","last_became_peered":"2024-09-11T08:07:32.047981+0000","last_unstale":"2024-09-11T08:21:59.174342+0000","last_undegraded":"2024-09-11T08:21:59.174342+0000","last_fullsized":"2024-09-11T08:21:59.174342+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'19","last_scrub_stamp":"2024-09-11T08:20:09.525095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:20:09.525095+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0063068040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":344,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.174763+0000","last_change":"2024-09-11T08:20:27.515065+0000","last_active":"2024-09-11T08:21:59.174763+0000","last_peered":"2024-09-11T08:21:59.174763+0000","last_clean":"2024-09-11T08:21:59.174763+0000","last_became_active":"2024-09-11T08:15:56.523574+0000","last_became_peered":"2024-09-11T08:15:56.523574+0000","last_unstale":"2024-09-11T08:21:59.174763+0000","last_undegraded":"2024-09-11T08:21:59.174763+0000","last_fullsized":"2024-09-11T08:21:59.174763+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:27.514983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:20:27.514983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.019346374,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":353,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:22:00.123142+0000","last_change":"2024-09-11T08:21:43.836231+0000","last_active":"2024-09-11T08:22:00.123142+0000","last_peered":"2024-09-11T08:22:00.123142+0000","last_clean":"2024-09-11T08:22:00.123142+0000","last_became_active":"2024-09-11T08:19:29.141857+0000","last_became_peered":"2024-09-11T08:19:29.141857+0000","last_unstale":"2024-09-11T08:22:00.123142+0000","last_undegraded":"2024-09-11T08:22:00.123142+0000","last_fullsized":"2024-09-11T08:22:00.123142+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:43.836166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:21:43.836166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:00.285779+0000","scrub_duration":0.001475074,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":138,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.447137+0000","last_change":"2024-09-11T08:21:54.636380+0000","last_active":"2024-09-11T08:21:59.447137+0000","last_peered":"2024-09-11T08:21:59.447137+0000","last_clean":"2024-09-11T08:21:59.447137+0000","last_became_active":"2024-09-11T08:19:29.180111+0000","last_became_peered":"2024-09-11T08:19:29.180111+0000","last_unstale":"2024-09-11T08:21:59.447137+0000","last_undegraded":"2024-09-11T08:21:59.447137+0000","last_fullsized":"2024-09-11T08:21:59.447137+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:54.636326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:21:54.636326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:18.539494+0000","scrub_duration":0.001465472,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"239'15","reported_seq":367,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:22:00.123454+0000","last_change":"2024-09-11T08:21:44.864734+0000","last_active":"2024-09-11T08:22:00.123454+0000","last_peered":"2024-09-11T08:22:00.123454+0000","last_clean":"2024-09-11T08:22:00.123454+0000","last_became_active":"2024-09-11T08:07:32.049368+0000","last_became_peered":"2024-09-11T08:07:32.049368+0000","last_unstale":"2024-09-11T08:22:00.123454+0000","last_undegraded":"2024-09-11T08:22:00.123454+0000","last_fullsized":"2024-09-11T08:22:00.123454+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'15","last_scrub_stamp":"2024-09-11T08:21:44.864672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:21:44.864672+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:06.795734+0000","scrub_duration":0.0019068259999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":146,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.175914+0000","last_change":"2024-09-11T08:21:43.492563+0000","last_active":"2024-09-11T08:21:59.175914+0000","last_peered":"2024-09-11T08:21:59.175914+0000","last_clean":"2024-09-11T08:21:59.175914+0000","last_became_active":"2024-09-11T08:15:56.521570+0000","last_became_peered":"2024-09-11T08:15:56.521570+0000","last_unstale":"2024-09-11T08:21:59.175914+0000","last_undegraded":"2024-09-11T08:21:59.175914+0000","last_fullsized":"2024-09-11T08:21:59.175914+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:43.492497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:21:43.492497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:59.430693+0000","scrub_duration":0.001773766,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":118,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:22:02.568906+0000","last_change":"2024-09-11T08:22:02.568857+0000","last_active":"2024-09-11T08:22:02.568906+0000","last_peered":"2024-09-11T08:22:02.568906+0000","last_clean":"2024-09-11T08:22:02.568906+0000","last_became_active":"2024-09-11T08:17:37.557645+0000","last_became_peered":"2024-09-11T08:17:37.557645+0000","last_unstale":"2024-09-11T08:22:02.568906+0000","last_undegraded":"2024-09-11T08:22:02.568906+0000","last_fullsized":"2024-09-11T08:22:02.568906+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:22:02.568761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:22:02.568761+0000","last_clean_scrub_stamp":"2024-09-11T08:22:02.568761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:23.299593+0000","scrub_duration":0.0057637030000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":337,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.447051+0000","last_change":"2024-09-11T08:21:08.731138+0000","last_active":"2024-09-11T08:21:59.447051+0000","last_peered":"2024-09-11T08:21:59.447051+0000","last_clean":"2024-09-11T08:21:59.447051+0000","last_became_active":"2024-09-11T08:19:29.198224+0000","last_became_peered":"2024-09-11T08:19:29.198224+0000","last_unstale":"2024-09-11T08:21:59.447051+0000","last_undegraded":"2024-09-11T08:21:59.447051+0000","last_fullsized":"2024-09-11T08:21:59.447051+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:08.731053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:09.973943+0000","last_clean_scrub_stamp":"2024-09-11T08:21:08.731053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:26.737814+0000","scrub_duration":0.0019130169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"239'19","reported_seq":414,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.175066+0000","last_change":"2024-09-11T08:21:53.814597+0000","last_active":"2024-09-11T08:21:59.175066+0000","last_peered":"2024-09-11T08:21:59.175066+0000","last_clean":"2024-09-11T08:21:59.175066+0000","last_became_active":"2024-09-11T08:15:56.520058+0000","last_became_peered":"2024-09-11T08:15:56.520058+0000","last_unstale":"2024-09-11T08:21:59.175066+0000","last_undegraded":"2024-09-11T08:21:59.175066+0000","last_fullsized":"2024-09-11T08:21:59.175066+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'19","last_scrub_stamp":"2024-09-11T08:21:53.814502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:21:53.814502+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:55.121172+0000","scrub_duration":0.0020894770000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":63,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.249197+0000","last_change":"2024-09-11T08:21:11.652549+0000","last_active":"2024-09-11T08:21:59.249197+0000","last_peered":"2024-09-11T08:21:59.249197+0000","last_clean":"2024-09-11T08:21:59.249197+0000","last_became_active":"2024-09-11T08:19:29.143511+0000","last_became_peered":"2024-09-11T08:19:29.143511+0000","last_unstale":"2024-09-11T08:21:59.249197+0000","last_undegraded":"2024-09-11T08:21:59.249197+0000","last_fullsized":"2024-09-11T08:21:59.249197+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:11.652456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:21:11.652456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:24.596654+0000","scrub_duration":0.011171949,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":146,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.175510+0000","last_change":"2024-09-11T08:21:07.260652+0000","last_active":"2024-09-11T08:21:59.175510+0000","last_peered":"2024-09-11T08:21:59.175510+0000","last_clean":"2024-09-11T08:21:59.175510+0000","last_became_active":"2024-09-11T08:15:56.523728+0000","last_became_peered":"2024-09-11T08:15:56.523728+0000","last_unstale":"2024-09-11T08:21:59.175510+0000","last_undegraded":"2024-09-11T08:21:59.175510+0000","last_fullsized":"2024-09-11T08:21:59.175510+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:07.260572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:11:34.159053+0000","last_clean_scrub_stamp":"2024-09-11T08:21:07.260572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:14.872091+0000","scrub_duration":0.0061691999999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":318,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.175375+0000","last_change":"2024-09-11T08:21:46.689250+0000","last_active":"2024-09-11T08:21:59.175375+0000","last_peered":"2024-09-11T08:21:59.175375+0000","last_clean":"2024-09-11T08:21:59.175375+0000","last_became_active":"2024-09-11T08:07:36.633368+0000","last_became_peered":"2024-09-11T08:07:36.633368+0000","last_unstale":"2024-09-11T08:21:59.175375+0000","last_undegraded":"2024-09-11T08:21:59.175375+0000","last_fullsized":"2024-09-11T08:21:59.175375+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:46.689190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:46.489220+0000","last_clean_scrub_stamp":"2024-09-11T08:21:46.689190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:48.254640+0000","scrub_duration":0.001726811,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"312'10","reported_seq":336,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.174996+0000","last_change":"2024-09-11T08:21:55.859919+0000","last_active":"2024-09-11T08:21:59.174996+0000","last_peered":"2024-09-11T08:21:59.174996+0000","last_clean":"2024-09-11T08:21:59.174996+0000","last_became_active":"2024-09-11T08:07:32.045728+0000","last_became_peered":"2024-09-11T08:07:32.045728+0000","last_unstale":"2024-09-11T08:21:59.174996+0000","last_undegraded":"2024-09-11T08:21:59.174996+0000","last_fullsized":"2024-09-11T08:21:59.174996+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"312'10","last_scrub_stamp":"2024-09-11T08:21:55.859866+0000","last_deep_scrub":"312'10","last_deep_scrub_stamp":"2024-09-11T08:21:55.859866+0000","last_clean_scrub_stamp":"2024-09-11T08:21:55.859866+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:00.807147+0000","scrub_duration":0.0027238100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":343,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.447816+0000","last_change":"2024-09-11T08:20:12.651491+0000","last_active":"2024-09-11T08:21:59.447816+0000","last_peered":"2024-09-11T08:21:59.447816+0000","last_clean":"2024-09-11T08:21:59.447816+0000","last_became_active":"2024-09-11T08:19:29.161558+0000","last_became_peered":"2024-09-11T08:19:29.161558+0000","last_unstale":"2024-09-11T08:21:59.447816+0000","last_undegraded":"2024-09-11T08:21:59.447816+0000","last_fullsized":"2024-09-11T08:21:59.447816+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:12.651408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:20:12.651408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0071009350000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":376,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.446982+0000","last_change":"2024-09-11T08:21:11.229626+0000","last_active":"2024-09-11T08:21:59.446982+0000","last_peered":"2024-09-11T08:21:59.446982+0000","last_clean":"2024-09-11T08:21:59.446982+0000","last_became_active":"2024-09-11T08:15:56.528607+0000","last_became_peered":"2024-09-11T08:15:56.528607+0000","last_unstale":"2024-09-11T08:21:59.446982+0000","last_undegraded":"2024-09-11T08:21:59.446982+0000","last_fullsized":"2024-09-11T08:21:59.446982+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:11.229540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:21:11.229540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:36.062439+0000","scrub_duration":0.002377456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":136,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.175766+0000","last_change":"2024-09-11T08:20:18.827796+0000","last_active":"2024-09-11T08:21:59.175766+0000","last_peered":"2024-09-11T08:21:59.175766+0000","last_clean":"2024-09-11T08:21:59.175766+0000","last_became_active":"2024-09-11T08:15:56.521488+0000","last_became_peered":"2024-09-11T08:15:56.521488+0000","last_unstale":"2024-09-11T08:21:59.175766+0000","last_undegraded":"2024-09-11T08:21:59.175766+0000","last_fullsized":"2024-09-11T08:21:59.175766+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:18.827709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:20:18.827709+0000","last_clean_scrub_stamp":"2024-09-11T08:20:18.827709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014060774999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":348,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:22:00.123382+0000","last_change":"2024-09-11T08:21:03.627814+0000","last_active":"2024-09-11T08:22:00.123382+0000","last_peered":"2024-09-11T08:22:00.123382+0000","last_clean":"2024-09-11T08:22:00.123382+0000","last_became_active":"2024-09-11T08:19:29.153017+0000","last_became_peered":"2024-09-11T08:19:29.153017+0000","last_unstale":"2024-09-11T08:22:00.123382+0000","last_undegraded":"2024-09-11T08:22:00.123382+0000","last_fullsized":"2024-09-11T08:22:00.123382+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:03.627721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:50.585566+0000","last_clean_scrub_stamp":"2024-09-11T08:21:03.627721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:31.922693+0000","scrub_duration":0.003783294,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":108,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.447457+0000","last_change":"2024-09-11T08:20:49.725564+0000","last_active":"2024-09-11T08:21:59.447457+0000","last_peered":"2024-09-11T08:21:59.447457+0000","last_clean":"2024-09-11T08:21:59.447457+0000","last_became_active":"2024-09-11T08:17:37.554262+0000","last_became_peered":"2024-09-11T08:17:37.554262+0000","last_unstale":"2024-09-11T08:21:59.447457+0000","last_undegraded":"2024-09-11T08:21:59.447457+0000","last_fullsized":"2024-09-11T08:21:59.447457+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:49.725505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:20:49.725505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012049350000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"288'24","reported_seq":421,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:21:59.447701+0000","last_change":"2024-09-11T08:20:28.885398+0000","last_active":"2024-09-11T08:21:59.447701+0000","last_peered":"2024-09-11T08:21:59.447701+0000","last_clean":"2024-09-11T08:21:59.447701+0000","last_became_active":"2024-09-11T08:19:29.145221+0000","last_became_peered":"2024-09-11T08:19:29.145221+0000","last_unstale":"2024-09-11T08:21:59.447701+0000","last_undegraded":"2024-09-11T08:21:59.447701+0000","last_fullsized":"2024-09-11T08:21:59.447701+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'24","last_scrub_stamp":"2024-09-11T08:20:28.885295+0000","last_deep_scrub":"288'24","last_deep_scrub_stamp":"2024-09-11T08:20:28.885295+0000","last_clean_scrub_stamp":"2024-09-11T08:20:28.885295+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0080800170000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":174,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-11T08:22:02.333557+0000","last_change":"2024-09-11T08:22:02.333467+0000","last_active":"2024-09-11T08:22:02.333557+0000","last_peered":"2024-09-11T08:22:02.333557+0000","last_clean":"2024-09-11T08:22:02.333557+0000","last_became_active":"2024-09-11T08:15:08.769133+0000","last_became_peered":"2024-09-11T08:15:08.769133+0000","last_unstale":"2024-09-11T08:22:02.333557+0000","last_undegraded":"2024-09-11T08:22:02.333557+0000","last_fullsized":"2024-09-11T08:22:02.333557+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:22:02.333362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:18:39.206016+0000","last_clean_scrub_stamp":"2024-09-11T08:22:02.333362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:32.251869+0000","scrub_duration":0.0035016740000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":1782251520,"num_objects":27195,"num_object_clones":0,"num_object_copies":81585,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27195,"num_write_kb":1740480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4999806976,"data_stored":4999806976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27195,"ondisk_log_size":27195,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1125,"num_read_kb":882,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":215,"num_bytes_recovered":4786,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3425,"internal_metadata":0},"log_size":763,"ondisk_log_size":763,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":469,"num_read_kb":464,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":168,"seq":721554505822,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":653864,"kb_used_data":332192,"kb_used_omap":7,"kb_used_meta":321656,"kb_avail":93091288,"statfs":{"total":95995035648,"available":95325478912,"internally_reserved":0,"allocated":340164608,"data_stored":339742560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8027,"internal_metadata":329375909},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:21:30 2024","interfaces":[{"interface":"back","average":{"1min":2.855,"5min":2.262,"15min":1.329},"min":{"1min":0.32800000000000001,"5min":0.223,"15min":0.223},"max":{"1min":32.942999999999998,"5min":67.409999999999997,"15min":67.409999999999997},"last":0.68700000000000006},{"interface":"front","average":{"1min":2.96,"5min":2.1720000000000002,"15min":1.3859999999999999},"min":{"1min":0.28299999999999997,"5min":0.219,"15min":0.219},"max":{"1min":32.665999999999997,"5min":66.959999999999994,"15min":66.959999999999994},"last":0.442}]},{"osd":1,"last update":"Wed Sep 11 08:21:30 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":2.3660000000000001,"15min":2.254},"min":{"1min":0.26000000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":4.2880000000000003,"5min":66.971000000000004,"15min":66.971000000000004},"last":0.48099999999999998},{"interface":"front","average":{"1min":1.226,"5min":2.2839999999999998,"15min":2.2240000000000002},"min":{"1min":0.32700000000000001,"5min":0.23699999999999999,"15min":0.23000000000000001},"max":{"1min":6.798,"5min":67.185000000000002,"15min":67.185000000000002},"last":0.85199999999999998}]},{"osd":2,"last update":"Wed Sep 11 08:21:30 2024","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.02,"15min":1.0109999999999999},"min":{"1min":0.35199999999999998,"5min":0.25700000000000001,"15min":0.22700000000000001},"max":{"1min":4.8920000000000003,"5min":6.444,"15min":6.444},"last":0.754},{"interface":"front","average":{"1min":0.84599999999999997,"5min":1.0640000000000001,"15min":0.83899999999999997},"min":{"1min":0.28599999999999998,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":4.4939999999999998,"5min":6.3460000000000001,"15min":6.3460000000000001},"last":0.39700000000000002}]},{"osd":3,"last update":"Wed Sep 11 08:21:30 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":1.042,"15min":1.153},"min":{"1min":0.28299999999999997,"5min":0.251,"15min":0.251},"max":{"1min":2.532,"5min":5.2149999999999999,"15min":21.460000000000001},"last":0.32000000000000001},{"interface":"front","average":{"1min":1.6020000000000001,"5min":1.0960000000000001,"15min":1.3879999999999999},"min":{"1min":0.34300000000000003,"5min":0.251,"15min":0.251},"max":{"1min":5.7030000000000003,"5min":6.4660000000000002,"15min":21.440000000000001},"last":0.373}]},{"osd":4,"last update":"Wed Sep 11 08:21:30 2024","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.85499999999999998,"15min":0.67000000000000004},"min":{"1min":0.28599999999999998,"5min":0.252,"15min":0.252},"max":{"1min":4.3390000000000004,"5min":4.5289999999999999,"15min":4.5289999999999999},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.80900000000000005,"15min":0.76700000000000002},"min":{"1min":0.251,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":1.891,"5min":5.2649999999999997,"15min":5.2649999999999997},"last":0.35899999999999999}]},{"osd":5,"last update":"Wed Sep 11 08:21:30 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":1.7010000000000001,"15min":1.129},"min":{"1min":0.28799999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":4.3710000000000004,"5min":65.866,"15min":65.866},"last":0.501},{"interface":"front","average":{"1min":0.86699999999999999,"5min":1.7789999999999999,"15min":1.117},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":2.968,"5min":66.019000000000005,"15min":66.019000000000005},"last":0.42099999999999999}]},{"osd":6,"last update":"Wed Sep 11 08:21:30 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":0.77000000000000002,"15min":0.79900000000000004},"min":{"1min":0.311,"5min":0.248,"15min":0.248},"max":{"1min":7.1699999999999999,"5min":7.1699999999999999,"15min":7.1699999999999999},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.92800000000000005,"15min":0.84999999999999998},"min":{"1min":0.36399999999999999,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":4.468,"5min":4.907,"15min":4.907},"last":0.42999999999999999}]}]},{"osd":6,"up_from":45,"seq":193273528506,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1157336,"kb_used_data":851664,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":92587816,"statfs":{"total":95995035648,"available":94809923584,"internally_reserved":0,"allocated":872103936,"data_stored":871698095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:21:19 2024","interfaces":[{"interface":"back","average":{"1min":1.0600000000000001,"5min":1.5509999999999999,"15min":1.411},"min":{"1min":0.19900000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":2.7309999999999999,"5min":25.494,"15min":45.066000000000003},"last":1.5409999999999999},{"interface":"front","average":{"1min":0.92000000000000004,"5min":1.5209999999999999,"15min":1.365},"min":{"1min":0.254,"5min":0.23699999999999999,"15min":0.20599999999999999},"max":{"1min":2.8090000000000002,"5min":25.48,"15min":43.899000000000001},"last":0.56899999999999995}]},{"osd":1,"last update":"Wed Sep 11 08:21:19 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":1.3879999999999999,"15min":1.198},"min":{"1min":0.218,"5min":0.218,"15min":0.214},"max":{"1min":2.758,"5min":20.321000000000002,"15min":44.658000000000001},"last":0.76100000000000001},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.391,"15min":1.2529999999999999},"min":{"1min":0.26400000000000001,"5min":0.22800000000000001,"15min":0.22600000000000001},"max":{"1min":5.4800000000000004,"5min":19.652000000000001,"15min":44.694000000000003},"last":0.69999999999999996}]},{"osd":2,"last update":"Wed Sep 11 08:21:19 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.751,"15min":0.78300000000000003},"min":{"1min":0.254,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":2.0449999999999999,"5min":3.8759999999999999,"15min":5.9770000000000003},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.94999999999999996,"15min":0.998},"min":{"1min":0.252,"5min":0.252,"15min":0.23300000000000001},"max":{"1min":2.1549999999999998,"5min":7.0549999999999997,"15min":13.346},"last":0.85299999999999998}]},{"osd":3,"last update":"Wed Sep 11 08:21:19 2024","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":1.024,"15min":1.161},"min":{"1min":0.308,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.2969999999999997,"5min":5.2969999999999997,"15min":36.061},"last":0.54300000000000004},{"interface":"front","average":{"1min":1.115,"5min":1.071,"15min":1.155},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.26800000000000002},"max":{"1min":5.2519999999999998,"5min":5.5720000000000001,"15min":36.093000000000004},"last":0.65300000000000002}]},{"osd":4,"last update":"Wed Sep 11 08:21:19 2024","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.66600000000000004,"15min":0.55000000000000004},"min":{"1min":0.24299999999999999,"5min":0.20499999999999999,"15min":0.20200000000000001},"max":{"1min":4.3440000000000003,"5min":4.492,"15min":4.492},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.61799999999999999,"15min":0.56799999999999995},"min":{"1min":0.32200000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":2.5819999999999999,"5min":4.1529999999999996,"15min":4.468},"last":0.72299999999999998}]},{"osd":5,"last update":"Wed Sep 11 08:21:19 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.82699999999999996,"15min":0.69699999999999995},"min":{"1min":0.309,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":2.3029999999999999,"5min":10.49,"15min":10.49},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.79500000000000004,"15min":0.66200000000000003},"min":{"1min":0.32000000000000001,"5min":0.28100000000000003,"15min":0.22600000000000001},"max":{"1min":2.7810000000000001,"5min":10.468,"15min":10.468},"last":0.623}]},{"osd":7,"last update":"Wed Sep 11 08:21:27 2024","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":0.74399999999999999,"15min":1.016},"min":{"1min":0.317,"5min":0.253,"15min":0.253},"max":{"1min":3.8809999999999998,"5min":3.8809999999999998,"15min":8.4580000000000002},"last":0.33800000000000002},{"interface":"front","average":{"1min":1,"5min":0.63600000000000001,"15min":0.81899999999999995},"min":{"1min":0.28599999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":3.0870000000000002,"5min":3.0870000000000002,"15min":3.593},"last":0.81200000000000006}]}]},{"osd":5,"up_from":38,"seq":163208757438,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":997020,"kb_used_data":411280,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":92748132,"statfs":{"total":95995035648,"available":94974087168,"internally_reserved":0,"allocated":421150720,"data_stored":420743896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:21:58 2024","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":2.0979999999999999,"15min":1.337},"min":{"1min":0.251,"5min":0.214,"15min":0.184},"max":{"1min":5.1200000000000001,"5min":92.924000000000007,"15min":92.924000000000007},"last":0.32000000000000001},{"interface":"front","average":{"1min":1.2490000000000001,"5min":2.0920000000000001,"15min":1.4079999999999999},"min":{"1min":0.221,"5min":0.221,"15min":0.20399999999999999},"max":{"1min":6.468,"5min":93.173000000000002,"15min":93.173000000000002},"last":0.69299999999999995}]},{"osd":1,"last update":"Wed Sep 11 08:21:58 2024","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":2.351,"15min":1.5880000000000001},"min":{"1min":0.253,"5min":0.22,"15min":0.22},"max":{"1min":6.423,"5min":96.899000000000001,"15min":96.899000000000001},"last":0.53100000000000003},{"interface":"front","average":{"1min":1.236,"5min":1.9730000000000001,"15min":1.385},"min":{"1min":0.35799999999999998,"5min":0.25700000000000001,"15min":0.22700000000000001},"max":{"1min":5.0780000000000003,"5min":92.337000000000003,"15min":92.337000000000003},"last":0.33800000000000002}]},{"osd":2,"last update":"Wed Sep 11 08:21:58 2024","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":1.0780000000000001,"15min":1.0349999999999999},"min":{"1min":0.28000000000000003,"5min":0.255,"15min":0.255},"max":{"1min":5.085,"5min":15.622,"15min":16.018000000000001},"last":0.34799999999999998},{"interface":"front","average":{"1min":1.9139999999999999,"5min":1.0649999999999999,"15min":0.88800000000000001},"min":{"1min":0.39000000000000001,"5min":0.27300000000000002,"15min":0.22800000000000001},"max":{"1min":16.713999999999999,"5min":16.713999999999999,"15min":16.713999999999999},"last":0.47599999999999998}]},{"osd":3,"last update":"Wed Sep 11 08:21:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":1.9179999999999999,"15min":1.3300000000000001},"min":{"1min":0.35299999999999998,"5min":0.254,"15min":0.25},"max":{"1min":3.7170000000000001,"5min":92.245000000000005,"15min":92.245000000000005},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.99399999999999999,"5min":2.298,"15min":1.51},"min":{"1min":0.39000000000000001,"5min":0.29699999999999999,"15min":0.251},"max":{"1min":5.1429999999999998,"5min":92.274000000000001,"15min":92.274000000000001},"last":0.41399999999999998}]},{"osd":4,"last update":"Wed Sep 11 08:21:58 2024","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":0.874,"15min":0.66000000000000003},"min":{"1min":0.28000000000000003,"5min":0.27100000000000002,"15min":0.21299999999999999},"max":{"1min":6.3879999999999999,"5min":6.3879999999999999,"15min":6.3879999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.872,"5min":0.74299999999999999,"15min":0.60899999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23100000000000001},"max":{"1min":3.1480000000000001,"5min":4.9720000000000004,"15min":4.9720000000000004},"last":0.443}]},{"osd":6,"last update":"Wed Sep 11 08:21:17 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.82299999999999995,"15min":0.60799999999999998},"min":{"1min":0.311,"5min":0.24399999999999999,"15min":0.23400000000000001},"max":{"1min":5.2869999999999999,"5min":7.5679999999999996,"15min":7.5679999999999996},"last":0.58599999999999997},{"interface":"front","average":{"1min":1.46,"5min":0.98899999999999999,"15min":0.71299999999999997},"min":{"1min":0.375,"5min":0.27500000000000002,"15min":0.26900000000000002},"max":{"1min":6.4470000000000001,"5min":6.4470000000000001,"15min":6.4470000000000001},"last":1.0069999999999999}]},{"osd":7,"last update":"Wed Sep 11 08:21:39 2024","interfaces":[{"interface":"back","average":{"1min":1.913,"5min":1.169,"15min":0.95099999999999996},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":6.5179999999999998,"5min":6.5179999999999998,"15min":6.5179999999999998},"last":0.40200000000000002},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.0720000000000001,"15min":0.88700000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":5.1989999999999998,"5min":7.5339999999999998,"15min":7.5339999999999998},"last":0.51400000000000001}]}]},{"osd":4,"up_from":31,"seq":133143986372,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1442668,"kb_used_data":1136996,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":92302484,"statfs":{"total":95995035648,"available":94517743616,"internally_reserved":0,"allocated":1164283904,"data_stored":1163857914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:21:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.905,"15min":1.53},"min":{"1min":0.224,"5min":0.222,"15min":0.19},"max":{"1min":8.1880000000000006,"5min":60.497999999999998,"15min":60.497999999999998},"last":0.498},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.8620000000000001,"15min":1.3740000000000001},"min":{"1min":0.24099999999999999,"5min":0.22600000000000001,"15min":0.221},"max":{"1min":7.5380000000000003,"5min":60.484999999999999,"15min":60.484999999999999},"last":1.091}]},{"osd":1,"last update":"Wed Sep 11 08:21:31 2024","interfaces":[{"interface":"back","average":{"1min":1.7989999999999999,"5min":1.381,"15min":1.042},"min":{"1min":0.254,"5min":0.24299999999999999,"15min":0.214},"max":{"1min":15.127000000000001,"5min":20.471,"15min":20.471},"last":1.591},{"interface":"front","average":{"1min":1.6910000000000001,"5min":1.2769999999999999,"15min":0.96999999999999997},"min":{"1min":0.247,"5min":0.23100000000000001,"15min":0.22700000000000001},"max":{"1min":14.843,"5min":15.744999999999999,"15min":15.744999999999999},"last":0.38600000000000001}]},{"osd":2,"last update":"Wed Sep 11 08:21:31 2024","interfaces":[{"interface":"back","average":{"1min":2.7879999999999998,"5min":1.119,"15min":0.93799999999999994},"min":{"1min":0.28100000000000003,"5min":0.253,"15min":0.23400000000000001},"max":{"1min":40.993000000000002,"5min":40.993000000000002,"15min":40.993000000000002},"last":0.41299999999999998},{"interface":"front","average":{"1min":2.621,"5min":1.1910000000000001,"15min":0.93500000000000005},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.248},"max":{"1min":41.152000000000001,"5min":41.152000000000001,"15min":41.152000000000001},"last":0.36699999999999999}]},{"osd":3,"last update":"Wed Sep 11 08:21:31 2024","interfaces":[{"interface":"back","average":{"1min":1.6439999999999999,"5min":1.4119999999999999,"15min":1.333},"min":{"1min":0.312,"5min":0.253,"15min":0.253},"max":{"1min":7.8150000000000004,"5min":19.102,"15min":26.884},"last":1.1120000000000001},{"interface":"front","average":{"1min":1.1479999999999999,"5min":1.052,"15min":1.2290000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.246},"max":{"1min":7.5869999999999997,"5min":7.5869999999999997,"15min":26.783000000000001},"last":0.66000000000000003}]},{"osd":5,"last update":"Wed Sep 11 08:21:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.98299999999999998,"15min":0.65400000000000003},"min":{"1min":0.28000000000000003,"5min":0.26500000000000001,"15min":0.23499999999999999},"max":{"1min":1.4319999999999999,"5min":33.384999999999998,"15min":33.384999999999998},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.60199999999999998,"5min":1.0700000000000001,"15min":0.68300000000000005},"min":{"1min":0.255,"5min":0.253,"15min":0.24199999999999999},"max":{"1min":1.5209999999999999,"5min":33.856000000000002,"15min":33.856000000000002},"last":1.0629999999999999}]},{"osd":6,"last update":"Wed Sep 11 08:21:19 2024","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.66000000000000003,"15min":0.68899999999999995},"min":{"1min":0.34300000000000003,"5min":0.25,"15min":0.23899999999999999},"max":{"1min":2.7589999999999999,"5min":4.6799999999999997,"15min":48.808},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.70199999999999996,"15min":0.73999999999999999},"min":{"1min":0.35199999999999998,"5min":0.26800000000000002,"15min":0.26200000000000001},"max":{"1min":1.948,"5min":4.0499999999999998,"15min":48.746000000000002},"last":0.443}]},{"osd":7,"last update":"Wed Sep 11 08:21:40 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.68100000000000005,"15min":0.57399999999999995},"min":{"1min":0.33500000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":2.052,"5min":5.3449999999999998,"15min":5.3449999999999998},"last":1.131},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.84299999999999997,"15min":0.73399999999999999},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":1.911,"5min":4.7160000000000002,"15min":4.7160000000000002},"last":0.52000000000000002}]}]},{"osd":3,"up_from":25,"seq":107374182600,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1017912,"kb_used_data":428016,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":92727240,"statfs":{"total":95995035648,"available":94952693760,"internally_reserved":0,"allocated":438288384,"data_stored":437849577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:21:19 2024","interfaces":[{"interface":"back","average":{"1min":3.282,"5min":1.752,"15min":1.5569999999999999},"min":{"1min":0.14999999999999999,"5min":0.14999999999999999,"15min":0.14999999999999999},"max":{"1min":58.880000000000003,"5min":58.880000000000003,"15min":60.469999999999999},"last":0.80600000000000005},{"interface":"front","average":{"1min":3.5390000000000001,"5min":1.794,"15min":1.655},"min":{"1min":0.184,"5min":0.184,"15min":0.17399999999999999},"max":{"1min":59.246000000000002,"5min":59.246000000000002,"15min":60.488999999999997},"last":0.222}]},{"osd":1,"last update":"Wed Sep 11 08:21:19 2024","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.3759999999999999,"15min":1.395},"min":{"1min":0.27500000000000002,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":11.148999999999999,"5min":11.148999999999999,"15min":60.578000000000003},"last":0.54100000000000004},{"interface":"front","average":{"1min":2.2429999999999999,"5min":1.925,"15min":1.6799999999999999},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.19500000000000001},"max":{"1min":11.853,"5min":46.868000000000002,"15min":63.674999999999997},"last":0.48299999999999998}]},{"osd":2,"last update":"Wed Sep 11 08:21:19 2024","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.2390000000000001,"15min":1.482},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.185},"max":{"1min":4.6100000000000003,"5min":8.4179999999999993,"15min":60.645000000000003},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.90400000000000003,"5min":1.1120000000000001,"15min":1.3660000000000001},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":3.3999999999999999,"5min":5.9500000000000002,"15min":60.723999999999997},"last":0.59199999999999997}]},{"osd":4,"last update":"Wed Sep 11 08:21:26 2024","interfaces":[{"interface":"back","average":{"1min":1.8819999999999999,"5min":1.3320000000000001,"15min":1.335},"min":{"1min":0.307,"5min":0.29499999999999998,"15min":0.24299999999999999},"max":{"1min":11.802,"5min":11.802,"15min":61.290999999999997},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.93200000000000005,"5min":0.998,"15min":1.1779999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.27200000000000002},"max":{"1min":3.9780000000000002,"5min":6.1340000000000003,"15min":60.856000000000002},"last":2.1840000000000002}]},{"osd":5,"last update":"Wed Sep 11 08:22:03 2024","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.8879999999999999,"15min":1.6870000000000001},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.26500000000000001},"max":{"1min":10.885999999999999,"5min":31.460999999999999,"15min":60.914999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.755,"5min":1.77,"15min":1.7250000000000001},"min":{"1min":0.34200000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":3.5659999999999998,"5min":31.216000000000001,"15min":60.787999999999997},"last":0.33500000000000002}]},{"osd":6,"last update":"Wed Sep 11 08:21:19 2024","interfaces":[{"interface":"back","average":{"1min":1.627,"5min":1.212,"15min":1.514},"min":{"1min":0.33200000000000002,"5min":0.308,"15min":0.308},"max":{"1min":10.949999999999999,"5min":10.949999999999999,"15min":60.817},"last":0.878},{"interface":"front","average":{"1min":2.0779999999999998,"5min":1.556,"15min":1.7030000000000001},"min":{"1min":0.35399999999999998,"5min":0.34699999999999998,"15min":0.307},"max":{"1min":11.750999999999999,"5min":11.837,"15min":61.264000000000003},"last":0.67800000000000005}]},{"osd":7,"last update":"Wed Sep 11 08:21:37 2024","interfaces":[{"interface":"back","average":{"1min":2.004,"5min":1.6990000000000001,"15min":2.7080000000000002},"min":{"1min":0.35999999999999999,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":11.016999999999999,"5min":31.071000000000002,"15min":60.847000000000001},"last":0.36799999999999999},{"interface":"front","average":{"1min":4.032,"5min":2.0510000000000002,"15min":2.786},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":58.820999999999998,"5min":58.820999999999998,"15min":60.883000000000003},"last":0.56499999999999995}]}]},{"osd":2,"up_from":18,"seq":77309411531,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":785688,"kb_used_data":475856,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":92959464,"statfs":{"total":95995035648,"available":95190491136,"internally_reserved":0,"allocated":487276544,"data_stored":486869632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:21:57 2024","interfaces":[{"interface":"back","average":{"1min":3.0569999999999999,"5min":1.788,"15min":1.639},"min":{"1min":0.221,"5min":0.17799999999999999,"15min":0.17199999999999999},"max":{"1min":45.838000000000001,"5min":45.838000000000001,"15min":45.838000000000001},"last":1.306},{"interface":"front","average":{"1min":2.8479999999999999,"5min":2.3100000000000001,"15min":1.8600000000000001},"min":{"1min":0.22500000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":46.508000000000003,"5min":46.508000000000003,"15min":46.508000000000003},"last":1.0329999999999999}]},{"osd":1,"last update":"Wed Sep 11 08:21:57 2024","interfaces":[{"interface":"back","average":{"1min":3.4159999999999999,"5min":1.9610000000000001,"15min":1.4850000000000001},"min":{"1min":0.23200000000000001,"5min":0.20200000000000001,"15min":0.17599999999999999},"max":{"1min":53.347000000000001,"5min":53.347000000000001,"15min":53.347000000000001},"last":1.1779999999999999},{"interface":"front","average":{"1min":3.5099999999999998,"5min":1.7490000000000001,"15min":1.4650000000000001},"min":{"1min":0.246,"5min":0.20799999999999999,"15min":0.20200000000000001},"max":{"1min":53.082999999999998,"5min":53.082999999999998,"15min":53.082999999999998},"last":0.45800000000000002}]},{"osd":3,"last update":"Wed Sep 11 08:21:19 2024","interfaces":[{"interface":"back","average":{"1min":3.0630000000000002,"5min":1.3879999999999999,"15min":1.3049999999999999},"min":{"1min":0.224,"5min":0.221,"15min":0.188},"max":{"1min":46.070999999999998,"5min":46.070999999999998,"15min":46.070999999999998},"last":1.2390000000000001},{"interface":"front","average":{"1min":3.3530000000000002,"5min":1.2969999999999999,"15min":1.248},"min":{"1min":0.26700000000000002,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":45.790999999999997,"5min":45.790999999999997,"15min":45.790999999999997},"last":0.42399999999999999}]},{"osd":4,"last update":"Wed Sep 11 08:21:36 2024","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":0.86799999999999999,"15min":0.93200000000000005},"min":{"1min":0.33100000000000002,"5min":0.27900000000000003,"15min":0.25900000000000001},"max":{"1min":5.8680000000000003,"5min":5.8680000000000003,"15min":7.806},"last":0.54100000000000004},{"interface":"front","average":{"1min":1.052,"5min":0.82299999999999995,"15min":0.94899999999999995},"min":{"1min":0.30299999999999999,"5min":0.27500000000000002,"15min":0.252},"max":{"1min":5.7279999999999998,"5min":5.7279999999999998,"15min":12.129},"last":0.67400000000000004}]},{"osd":5,"last update":"Wed Sep 11 08:22:06 2024","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.7809999999999999,"15min":1.274},"min":{"1min":0.35299999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":5.7530000000000001,"5min":35.963000000000001,"15min":35.963000000000001},"last":1.776},{"interface":"front","average":{"1min":1.494,"5min":1.855,"15min":1.3080000000000001},"min":{"1min":0.29899999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":15.032,"5min":35.104999999999997,"15min":35.104999999999997},"last":1.925}]},{"osd":6,"last update":"Wed Sep 11 08:21:25 2024","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":0.89900000000000002,"15min":0.91600000000000004},"min":{"1min":0.311,"5min":0.31,"15min":0.26100000000000001},"max":{"1min":4.4989999999999997,"5min":4.4989999999999997,"15min":7.2249999999999996},"last":1.4370000000000001},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.89700000000000002,"15min":0.996},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32100000000000001},"max":{"1min":5.8010000000000002,"5min":5.8010000000000002,"15min":6.71},"last":1.474}]},{"osd":7,"last update":"Wed Sep 11 08:21:36 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":1.431,"15min":1.4079999999999999},"min":{"1min":0.33300000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":2.6880000000000002,"5min":35.274999999999999,"15min":35.274999999999999},"last":0.76400000000000001},{"interface":"front","average":{"1min":1.349,"5min":1.5289999999999999,"15min":1.526},"min":{"1min":0.34599999999999997,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":5.7770000000000001,"5min":35.265000000000001,"15min":35.265000000000001},"last":0.99199999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575057,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":862332,"kb_used_data":268276,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":92882820,"statfs":{"total":95995035648,"available":95112007680,"internally_reserved":0,"allocated":274714624,"data_stored":274271864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:21:28 2024","interfaces":[{"interface":"back","average":{"1min":3.9609999999999999,"5min":2.1120000000000001,"15min":1.516},"min":{"1min":0.185,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":63.932000000000002,"5min":63.932000000000002,"15min":63.932000000000002},"last":1.8069999999999999},{"interface":"front","average":{"1min":4.5099999999999998,"5min":2.2120000000000002,"15min":1.8129999999999999},"min":{"1min":0.17799999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":64.269000000000005,"5min":64.269000000000005,"15min":64.269000000000005},"last":0.64100000000000001}]},{"osd":2,"last update":"Wed Sep 11 08:22:08 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":1.1599999999999999,"15min":1.149},"min":{"1min":0.23000000000000001,"5min":0.189,"15min":0.189},"max":{"1min":1.1659999999999999,"5min":7.798,"15min":8.3819999999999997},"last":1.1659999999999999},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.98299999999999998,"15min":1.2549999999999999},"min":{"1min":0.224,"5min":0.19,"15min":0.16300000000000001},"max":{"1min":4.9790000000000001,"5min":5.6289999999999996,"15min":10.276},"last":0.90800000000000003}]},{"osd":3,"last update":"Wed Sep 11 08:21:13 2024","interfaces":[{"interface":"back","average":{"1min":1.726,"5min":1.7170000000000001,"15min":1.77},"min":{"1min":0.23599999999999999,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":8.0909999999999993,"5min":8.4149999999999991,"15min":32.759},"last":0.39700000000000002},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.7969999999999999,"15min":1.6279999999999999},"min":{"1min":0.29899999999999999,"5min":0.25,"15min":0.22600000000000001},"max":{"1min":5.069,"5min":9.0060000000000002,"15min":32.444000000000003},"last":0.81499999999999995}]},{"osd":4,"last update":"Wed Sep 11 08:21:34 2024","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":2.8700000000000001,"15min":1.706},"min":{"1min":0.308,"5min":0.26200000000000001,"15min":0.252},"max":{"1min":5.7939999999999996,"5min":110.86499999999999,"15min":110.86499999999999},"last":0.27000000000000002},{"interface":"front","average":{"1min":1.5309999999999999,"5min":2.9940000000000002,"15min":1.653},"min":{"1min":0.29299999999999998,"5min":0.28000000000000003,"15min":0.248},"max":{"1min":10.468999999999999,"5min":110.831,"15min":110.831},"last":0.86199999999999999}]},{"osd":5,"last update":"Wed Sep 11 08:22:08 2024","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":2.625,"15min":1.7130000000000001},"min":{"1min":0.41799999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":5.8300000000000001,"5min":109.208,"15min":109.208},"last":0.66400000000000003},{"interface":"front","average":{"1min":1.272,"5min":2.7410000000000001,"15min":1.821},"min":{"1min":0.36799999999999999,"5min":0.28299999999999997,"15min":0.28100000000000003},"max":{"1min":7.9359999999999999,"5min":111.254,"15min":111.254},"last":0.55700000000000005}]},{"osd":6,"last update":"Wed Sep 11 08:21:13 2024","interfaces":[{"interface":"back","average":{"1min":1.4430000000000001,"5min":1.4510000000000001,"15min":1.6830000000000001},"min":{"1min":0.307,"5min":0.30499999999999999,"15min":0.29399999999999998},"max":{"1min":7.9809999999999999,"5min":7.9809999999999999,"15min":124.31999999999999},"last":0.69199999999999995},{"interface":"front","average":{"1min":2.117,"5min":1.5880000000000001,"15min":1.6699999999999999},"min":{"1min":0.42299999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":8.6099999999999994,"5min":8.6099999999999994,"15min":124.958},"last":0.89300000000000002}]},{"osd":7,"last update":"Wed Sep 11 08:21:34 2024","interfaces":[{"interface":"back","average":{"1min":1.514,"5min":2.528,"15min":1.7210000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":8.5419999999999998,"5min":109.149,"15min":109.149},"last":0.36099999999999999},{"interface":"front","average":{"1min":2.0529999999999999,"5min":2.7519999999999998,"15min":1.9379999999999999},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":10.695,"5min":109.309,"15min":109.309},"last":0.60899999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738581,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1234932,"kb_used_data":925100,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":92510220,"statfs":{"total":95995035648,"available":94730465280,"internally_reserved":0,"allocated":947302400,"data_stored":946869010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,4],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 11 08:21:28 2024","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.252,"15min":1.1799999999999999},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17000000000000001},"max":{"1min":4.883,"5min":10.105,"15min":10.105},"last":1.8100000000000001},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.429,"15min":1.4059999999999999},"min":{"1min":0.23400000000000001,"5min":0.16600000000000001,"15min":0.153},"max":{"1min":3.4119999999999999,"5min":19.852,"15min":19.852},"last":0.39000000000000001}]},{"osd":2,"last update":"Wed Sep 11 08:21:55 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":1,"15min":1.1399999999999999},"min":{"1min":0.21099999999999999,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.77600000000000002,"5min":10.025,"15min":10.351000000000001},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.48599999999999999,"5min":1.123,"15min":1.236},"min":{"1min":0.245,"5min":0.23699999999999999,"15min":0.186},"max":{"1min":1.526,"5min":10.074999999999999,"15min":10.074999999999999},"last":1.714}]},{"osd":3,"last update":"Wed Sep 11 08:21:16 2024","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":1.3779999999999999,"15min":1.526},"min":{"1min":0.27700000000000002,"5min":0.251,"15min":0.22900000000000001},"max":{"1min":3.2749999999999999,"5min":10.129,"15min":35.491},"last":1.746},{"interface":"front","average":{"1min":0.624,"5min":1.0549999999999999,"15min":1.456},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":2.5230000000000001,"5min":8.282,"15min":37.563000000000002},"last":0.41599999999999998}]},{"osd":4,"last update":"Wed Sep 11 08:21:40 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.79000000000000004,"15min":1.026},"min":{"1min":0.29599999999999999,"5min":0.252,"15min":0.24099999999999999},"max":{"1min":3.266,"5min":4.5949999999999998,"15min":9.4030000000000005},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.70399999999999996,"5min":1.0449999999999999,"15min":1.0409999999999999},"min":{"1min":0.29299999999999998,"5min":0.27200000000000002,"15min":0.24399999999999999},"max":{"1min":1.5669999999999999,"5min":8.3510000000000009,"15min":8.3510000000000009},"last":1.77}]},{"osd":5,"last update":"Wed Sep 11 08:22:00 2024","interfaces":[{"interface":"back","average":{"1min":1.6459999999999999,"5min":1.3839999999999999,"15min":1.3140000000000001},"min":{"1min":0.40300000000000002,"5min":0.28599999999999998,"15min":0.255},"max":{"1min":15.754,"5min":15.754,"15min":24.914999999999999},"last":1.7909999999999999},{"interface":"front","average":{"1min":0.59499999999999997,"5min":1.3720000000000001,"15min":1.399},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27100000000000002},"max":{"1min":1.8180000000000001,"5min":13.885,"15min":25.277000000000001},"last":0.91400000000000003}]},{"osd":6,"last update":"Wed Sep 11 08:21:22 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.874,"15min":1.3899999999999999},"min":{"1min":0.32600000000000001,"5min":0.29899999999999999,"15min":0.29099999999999998},"max":{"1min":3.2890000000000001,"5min":4.2670000000000003,"15min":78.522000000000006},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.77700000000000002,"5min":1.0629999999999999,"15min":1.3140000000000001},"min":{"1min":0.32900000000000001,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.8620000000000001,"5min":7.4820000000000002,"15min":79.808000000000007},"last":0.68899999999999995}]},{"osd":7,"last update":"Wed Sep 11 08:21:40 2024","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":1.0880000000000001,"15min":1.3109999999999999},"min":{"1min":0.39800000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":2.6179999999999999,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":1.1839999999999999},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.349,"15min":1.371},"min":{"1min":0.41799999999999998,"5min":0.33600000000000002,"15min":0.32000000000000001},"max":{"1min":8.2780000000000005,"5min":13.914,"15min":13.914},"last":1.212}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3425,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":958136320,"data_stored":958136320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":289210368,"data_stored":289210368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":533528576,"data_stored":533528576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":459145216,"data_stored":459145216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1042022400,"data_stored":1042022400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":453771264,"data_stored":453771264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":915341312,"data_stored":915341312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":348651520,"data_stored":348651520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T08:22:19.418 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj -9, currently 1 2024-09-11T08:22:19.452 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:19.483 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:19.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:19.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:19.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:19 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:22:19.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:19 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:22:19.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:19 smithi181 ceph-mon[32701]: osdmap e315: 8 total, 8 up, 8 in 2024-09-11T08:22:19.585 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:22:19.622 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:19.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:19.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:22:19.723 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:19.746 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:22:19.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:19.756 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:19.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:19.856 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:22:19.892 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:19.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:19.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:22:19.993 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:20.022 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:22:20.026 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 19 15 45549 45534 149.767 116.188 0.00905476 0.00667179 2024-09-11T08:22:20.027 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:20.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:20.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:20.129 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:20.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:20.175 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:20.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:20.260 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:20 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:22:20.260 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:20 smithi150 ceph-mon[29276]: pgmap v671: 148 pgs: 148 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 125 MiB/s wr, 2.01k op/s 2024-09-11T08:22:20.260 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:20 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:22:20.260 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:20 smithi150 ceph-mon[39612]: pgmap v671: 148 pgs: 148 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 125 MiB/s wr, 2.01k op/s 2024-09-11T08:22:20.277 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:20.311 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:20.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:20.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:20.413 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:20.447 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:20.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:20.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:20.549 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:20.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:20 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:22:20.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:20 smithi181 ceph-mon[32701]: pgmap v671: 148 pgs: 148 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 125 MiB/s wr, 2.01k op/s 2024-09-11T08:22:20.585 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:20.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:20.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:20.644 INFO:teuthology.orchestra.run.smithi150.stderr:nodeep-scrub is set 2024-09-11T08:22:20.687 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:20.719 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:20.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:20.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:20.821 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:22:20.853 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:20.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:20.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:22:20.955 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:20.994 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:20.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:20.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:21.026 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:2024-09-11T08:22:21.027313+0000 min lat: 0.00123278 max lat: 0.187209 avg lat: 0.00675018 2024-09-11T08:22:21.026 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-11T08:22:21.027 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 20 15 47392 47377 148.037 115.188 0.0107653 0.00675018 2024-09-11T08:22:21.096 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:22:21.136 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:21.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:21.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:22:21.238 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:22:21.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:21.274 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:21.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:22:21.375 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:22:21.408 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:21.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:21.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:22:21.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:21 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2594533172' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:22:21.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:21 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:22:21.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:21 smithi150 ceph-mon[29276]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:22:21.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:21 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:22:21.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:21 smithi150 ceph-mon[29276]: osdmap e316: 8 total, 8 up, 8 in 2024-09-11T08:22:21.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:21 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2594533172' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:22:21.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:21 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:22:21.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:21 smithi150 ceph-mon[39612]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:22:21.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:21 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:22:21.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:21 smithi150 ceph-mon[39612]: osdmap e316: 8 total, 8 up, 8 in 2024-09-11T08:22:21.510 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:22:21.543 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:21.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:21.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:22:21.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:21 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2594533172' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:22:21.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:21 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:22:21.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:21 smithi181 ceph-mon[32701]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:22:21.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:21 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:22:21.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:21 smithi181 ceph-mon[32701]: osdmap e316: 8 total, 8 up, 8 in 2024-09-11T08:22:21.645 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:21.675 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:21.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:21.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:21.777 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:21.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:21.812 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:21.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:21.913 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:22:21.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:21.947 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:21.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:22:22.027 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 21 16 49264 49248 146.556 116.938 0.00718439 0.00681819 2024-09-11T08:22:22.049 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:22:22.085 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:22.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:22.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:22:22.187 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:22.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:22.224 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:22.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:22.325 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:22:22.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:22.358 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:22.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:22:22.459 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:22.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:22 smithi150 ceph-mon[29276]: pgmap v673: 148 pgs: 148 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 116 MiB/s wr, 1.85k op/s 2024-09-11T08:22:22.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:22 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:22:22.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:22 smithi150 ceph-mon[39612]: pgmap v673: 148 pgs: 148 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 116 MiB/s wr, 1.85k op/s 2024-09-11T08:22:22.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:22 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:22:22.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:22.500 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:22.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:22.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:22 smithi181 ceph-mon[32701]: pgmap v673: 148 pgs: 148 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 116 MiB/s wr, 1.85k op/s 2024-09-11T08:22:22.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:22 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:22:22.601 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:22:22.637 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:22.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:22.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:22:22.739 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:22:22.774 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:22.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:22.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:22:22.877 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:22.912 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:22.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:22.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:23.018 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:23.028 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 22 16 50956 50940 144.692 105.75 0.0100505 0.00690639 2024-09-11T08:22:23.055 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:23.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:23.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:23.157 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:22:23.191 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:23.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:23.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:22:23.292 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:23.324 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:23.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:23.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:23.427 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:22:23.454 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:23.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:23.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:22:23.556 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:22:23.590 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:23.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:23.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:22:23.691 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:22:23.719 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:23.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:23.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:22:23.821 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:23.857 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:23.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:23.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:23.958 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:23.990 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:23.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:23.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:24.028 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 23 15 52491 52476 142.575 96 0.00298906 0.00700947 2024-09-11T08:22:24.092 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:24.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:24.133 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:24.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:24.234 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:24.255 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:24.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:24.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:24.356 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:22:24.368 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:24 smithi150 ceph-mon[29276]: pgmap v674: 148 pgs: 148 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 110 MiB/s wr, 1.76k op/s 2024-09-11T08:22:24.368 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:24 smithi150 ceph-mon[39612]: pgmap v674: 148 pgs: 148 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 110 MiB/s wr, 1.76k op/s 2024-09-11T08:22:24.396 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:24.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:24.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:22:24.419 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-11T08:22:24.419 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T08:22:24.419 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_3 2024-09-11T08:22:24.419 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool unique_pool_3 by 10 min 8 2024-09-11T08:22:24.419 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T08:22:24.498 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:24.519 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:24.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:24.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:24.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:24 smithi181 ceph-mon[32701]: pgmap v674: 148 pgs: 148 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 110 MiB/s wr, 1.76k op/s 2024-09-11T08:22:24.621 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:24.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:24.657 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:24.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:24.759 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:22:24.790 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:24.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:24.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:22:24.892 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:22:24.924 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:24.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:24.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:22:25.026 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:25.028 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 24 16 53959 53943 140.454 91.6875 0.0100888 0.00711475 2024-09-11T08:22:25.062 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:25.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:25.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:25.164 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:25.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:25.201 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:25.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:25.302 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:22:25.335 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:25.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:25.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:22:25.403 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T08:22:25.437 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:22:25.509 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:25.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:25.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:22:25.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:22:25 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:22:25] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:22:25.611 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:25.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:25.664 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:25.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:25.765 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:22:25.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:25.816 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:25.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:22:25.917 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:25.952 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:25.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:25.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:26.028 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 25 16 55516 55500 138.728 97.3125 0.00902136 0.00720339 2024-09-11T08:22:26.054 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:26.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:26.088 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:26.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:26.178 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:22:26.179 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:22:26.189 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:22:26.211 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:22:26.211 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:22:26.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:26.232 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:26.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:22:26.250 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:22:26.250 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:22:26.284 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:26 smithi150 ceph-mon[29276]: pgmap v675: 148 pgs: 148 active+clean; 3.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 145 MiB/s wr, 2.33k op/s 2024-09-11T08:22:26.284 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:26 smithi150 ceph-mon[29276]: osdmap e317: 8 total, 8 up, 8 in 2024-09-11T08:22:26.284 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:26 smithi150 ceph-mon[39612]: pgmap v675: 148 pgs: 148 active+clean; 3.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 145 MiB/s wr, 2.33k op/s 2024-09-11T08:22:26.285 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:26 smithi150 ceph-mon[39612]: osdmap e317: 8 total, 8 up, 8 in 2024-09-11T08:22:26.285 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:22:26.285 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:22:26.316 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:22:26.317 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:22:26.333 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:22:26.350 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:22:26.351 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:22:26.374 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:26.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:26.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:22:26.387 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:22:26.388 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:22:26.425 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:22:26.425 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:22:26.476 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:22:26.514 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:26.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:26.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:22:26.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:26 smithi181 ceph-mon[32701]: pgmap v675: 148 pgs: 148 active+clean; 3.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 145 MiB/s wr, 2.33k op/s 2024-09-11T08:22:26.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:26 smithi181 ceph-mon[32701]: osdmap e317: 8 total, 8 up, 8 in 2024-09-11T08:22:26.615 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:22:26.654 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:26.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:26.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:22:26.757 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:26.801 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:26.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:26.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:26.887 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-11T08:22:26.888 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 20, 2024-09-11T08:22:26.888 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 600, 2024-09-11T08:22:26.888 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [ 2024-09-11T08:22:26.888 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.889 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.24421.0:285 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e140)", 2024-09-11T08:22:26.889 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:12:27.939875+0000", 2024-09-11T08:22:26.889 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 598.76638124399994, 2024-09-11T08:22:26.889 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.56415223599999997, 2024-09-11T08:22:26.889 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:22:26.889 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "started", 2024-09-11T08:22:26.889 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:22:26.889 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.24421", 2024-09-11T08:22:26.889 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.181:0/3861404665", 2024-09-11T08:22:26.889 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 285 2024-09-11T08:22:26.890 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.890 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:22:26.890 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.890 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:22:26.890 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.939875+0000", 2024-09-11T08:22:26.890 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:26.890 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.890 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.890 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:22:26.890 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.939875+0000", 2024-09-11T08:22:26.890 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:26.891 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.891 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.891 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:22:26.891 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.939874+0000", 2024-09-11T08:22:26.891 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:22:26.891 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.891 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.891 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:22:26.891 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.939879+0000", 2024-09-11T08:22:26.891 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.8490000000000002e-06 2024-09-11T08:22:26.891 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.891 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.892 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:22:26.892 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.939887+0000", 2024-09-11T08:22:26.892 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.7749999999999993e-06 2024-09-11T08:22:26.892 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.892 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.892 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:22:26.892 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.939892+0000", 2024-09-11T08:22:26.892 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.8450000000000002e-06 2024-09-11T08:22:26.892 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.892 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.892 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:26.892 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.939939+0000", 2024-09-11T08:22:26.892 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.7431e-05 2024-09-11T08:22:26.893 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.893 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.893 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:22:26.893 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:27.939945+0000", 2024-09-11T08:22:26.893 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.3260000000000003e-06 2024-09-11T08:22:26.893 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.893 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.893 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:26.893 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.503775+0000", 2024-09-11T08:22:26.893 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.56383043899999996 2024-09-11T08:22:26.893 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.893 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.893 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:22:26.893 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.503905+0000", 2024-09-11T08:22:26.894 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012999499999999999 2024-09-11T08:22:26.894 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.894 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.894 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:22:26.894 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:12:28.504027+0000", 2024-09-11T08:22:26.894 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012184300000000001 2024-09-11T08:22:26.894 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:26.894 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:22:26.894 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:26.894 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.894 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.894 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15369.0:30513 9.4 9:2497afae:::benchmark_data_smithi150_133616_object30512:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e252)", 2024-09-11T08:22:26.894 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:18:23.979024+0000", 2024-09-11T08:22:26.895 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 242.72723181200001, 2024-09-11T08:22:26.895 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.17581966600000001, 2024-09-11T08:22:26.895 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:22:26.895 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:22:26.895 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:22:26.895 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15369", 2024-09-11T08:22:26.895 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:22:26.895 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 30513 2024-09-11T08:22:26.895 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.895 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:22:26.895 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.895 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:22:26.895 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:23.979024+0000", 2024-09-11T08:22:26.896 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:26.896 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.896 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.896 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:22:26.896 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:23.979024+0000", 2024-09-11T08:22:26.896 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:26.896 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.896 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.896 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:22:26.896 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:23.979024+0000", 2024-09-11T08:22:26.896 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:22:26.896 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.896 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.896 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:22:26.897 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:23.979045+0000", 2024-09-11T08:22:26.897 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.0971e-05 2024-09-11T08:22:26.897 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.897 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.897 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:22:26.897 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:23.979051+0000", 2024-09-11T08:22:26.897 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.8279999999999997e-06 2024-09-11T08:22:26.897 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.897 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.897 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:22:26.897 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:23.979056+0000", 2024-09-11T08:22:26.897 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.357e-06 2024-09-11T08:22:26.898 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.898 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.898 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:26.898 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:23.980657+0000", 2024-09-11T08:22:26.898 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.001601599 2024-09-11T08:22:26.898 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.898 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.898 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:22:26.898 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:23.980770+0000", 2024-09-11T08:22:26.898 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000112942 2024-09-11T08:22:26.898 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.898 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.898 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 3,4", 2024-09-11T08:22:26.898 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:23.980941+0000", 2024-09-11T08:22:26.899 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00017040100000000001 2024-09-11T08:22:26.899 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.899 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.899 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:22:26.899 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:23.986392+0000", 2024-09-11T08:22:26.899 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0054511500000000001 2024-09-11T08:22:26.899 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.899 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.899 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:22:26.899 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:24.154716+0000", 2024-09-11T08:22:26.899 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.168323836 2024-09-11T08:22:26.899 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.899 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.899 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:22:26.900 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:24.154749+0000", 2024-09-11T08:22:26.900 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.3497000000000001e-05 2024-09-11T08:22:26.900 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.900 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.900 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:22:26.900 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:24.154802+0000", 2024-09-11T08:22:26.900 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.3016999999999997e-05 2024-09-11T08:22:26.900 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.900 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.900 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:22:26.900 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:24.154844+0000", 2024-09-11T08:22:26.900 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1658e-05 2024-09-11T08:22:26.900 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:26.900 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:22:26.901 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:26.901 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.901 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.901 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15369.0:96087 9.a 9:5eafb6c5:::benchmark_data_smithi150_133616_object96086:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e260)", 2024-09-11T08:22:26.901 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:19:08.596504+0000", 2024-09-11T08:22:26.901 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 198.10975173700001, 2024-09-11T08:22:26.901 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.0948638279999998, 2024-09-11T08:22:26.901 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:22:26.901 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-09-11T08:22:26.901 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:22:26.901 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15369", 2024-09-11T08:22:26.901 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:22:26.901 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 96087 2024-09-11T08:22:26.901 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.902 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:22:26.902 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.902 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:22:26.902 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.596504+0000", 2024-09-11T08:22:26.902 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:26.902 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.902 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.902 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:22:26.902 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.596504+0000", 2024-09-11T08:22:26.902 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:26.902 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.902 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.902 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:22:26.902 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.596503+0000", 2024-09-11T08:22:26.902 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:22:26.903 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.903 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.903 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:22:26.903 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.596526+0000", 2024-09-11T08:22:26.903 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.3686e-05 2024-09-11T08:22:26.903 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.903 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.903 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:22:26.903 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.596537+0000", 2024-09-11T08:22:26.903 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0324e-05 2024-09-11T08:22:26.903 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.903 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.904 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:22:26.904 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.596542+0000", 2024-09-11T08:22:26.904 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.6729999999999997e-06 2024-09-11T08:22:26.904 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.904 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.904 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:26.904 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.601303+0000", 2024-09-11T08:22:26.904 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0047603510000000003 2024-09-11T08:22:26.904 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.904 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.904 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:22:26.904 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.601314+0000", 2024-09-11T08:22:26.904 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1639e-05 2024-09-11T08:22:26.904 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.905 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.905 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:26.905 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.591260+0000", 2024-09-11T08:22:26.905 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.98994515299999997 2024-09-11T08:22:26.905 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.905 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.905 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:22:26.905 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.591270+0000", 2024-09-11T08:22:26.905 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0226e-05 2024-09-11T08:22:26.905 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.905 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.905 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:26.905 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.689068+0000", 2024-09-11T08:22:26.905 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.097797911 2024-09-11T08:22:26.906 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.906 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.906 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:22:26.906 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.691368+0000", 2024-09-11T08:22:26.906 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0023004119999999999 2024-09-11T08:22:26.906 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:26.906 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:22:26.906 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:26.906 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.906 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.906 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15369.0:96116 9.a 9:59cffadf:::benchmark_data_smithi150_133616_object96115:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-11T08:22:26.906 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:19:08.628434+0000", 2024-09-11T08:22:26.906 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 198.077821621, 2024-09-11T08:22:26.906 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.082338676, 2024-09-11T08:22:26.907 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:22:26.907 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:22:26.907 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:22:26.907 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15369", 2024-09-11T08:22:26.907 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:22:26.907 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 96116 2024-09-11T08:22:26.907 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.907 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:22:26.907 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.907 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:22:26.907 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.628434+0000", 2024-09-11T08:22:26.907 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:26.907 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.908 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.908 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:22:26.908 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.628434+0000", 2024-09-11T08:22:26.908 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:26.908 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.908 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.908 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:22:26.908 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.628432+0000", 2024-09-11T08:22:26.908 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:22:26.908 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.908 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.908 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:22:26.908 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.628461+0000", 2024-09-11T08:22:26.909 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.8496999999999998e-05 2024-09-11T08:22:26.909 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.909 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.909 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:22:26.909 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.628471+0000", 2024-09-11T08:22:26.909 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0362e-05 2024-09-11T08:22:26.909 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.909 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.909 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:22:26.909 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.628477+0000", 2024-09-11T08:22:26.909 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.1859999999999997e-06 2024-09-11T08:22:26.909 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.909 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.910 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:26.910 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.628547+0000", 2024-09-11T08:22:26.910 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.9460000000000002e-05 2024-09-11T08:22:26.910 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.910 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.910 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:22:26.910 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.628558+0000", 2024-09-11T08:22:26.910 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0811e-05 2024-09-11T08:22:26.910 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.910 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.910 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:26.910 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.591329+0000", 2024-09-11T08:22:26.910 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.96277097700000003 2024-09-11T08:22:26.911 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.911 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.911 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:22:26.911 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.591336+0000", 2024-09-11T08:22:26.911 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.5129999999999999e-06 2024-09-11T08:22:26.911 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.911 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.911 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:26.911 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.691674+0000", 2024-09-11T08:22:26.911 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.100337771 2024-09-11T08:22:26.911 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.911 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.911 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:22:26.911 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.692428+0000", 2024-09-11T08:22:26.911 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00075426699999999998 2024-09-11T08:22:26.912 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.912 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.912 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 3,5,7", 2024-09-11T08:22:26.912 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.692745+0000", 2024-09-11T08:22:26.912 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000317003 2024-09-11T08:22:26.912 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.912 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.912 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:22:26.912 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.702335+0000", 2024-09-11T08:22:26.912 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0095893450000000009 2024-09-11T08:22:26.912 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.912 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.912 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:22:26.912 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.702358+0000", 2024-09-11T08:22:26.913 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.3431e-05 2024-09-11T08:22:26.913 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.913 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.913 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:22:26.913 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.702626+0000", 2024-09-11T08:22:26.913 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00026750599999999998 2024-09-11T08:22:26.913 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.913 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.913 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:22:26.913 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.710136+0000", 2024-09-11T08:22:26.913 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0075106670000000004 2024-09-11T08:22:26.913 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.913 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.913 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:22:26.914 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.710657+0000", 2024-09-11T08:22:26.914 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00052111099999999999 2024-09-11T08:22:26.914 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.914 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.914 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:22:26.914 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.710773+0000", 2024-09-11T08:22:26.914 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000115777 2024-09-11T08:22:26.914 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:26.914 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:22:26.914 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:26.914 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.914 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.914 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15369.0:96117 9.f 9:fa88c3cd:::benchmark_data_smithi150_133616_object96116:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-11T08:22:26.914 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:19:08.628715+0000", 2024-09-11T08:22:26.915 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 198.07754127199999, 2024-09-11T08:22:26.915 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.2202831299999999, 2024-09-11T08:22:26.915 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:22:26.915 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:22:26.915 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:22:26.915 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15369", 2024-09-11T08:22:26.915 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:22:26.915 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 96117 2024-09-11T08:22:26.915 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.915 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:22:26.915 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.915 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:22:26.915 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.628715+0000", 2024-09-11T08:22:26.915 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:26.916 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.916 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.916 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:22:26.916 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.628715+0000", 2024-09-11T08:22:26.916 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:26.916 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.916 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.916 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:22:26.916 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.628714+0000", 2024-09-11T08:22:26.916 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:22:26.916 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.916 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.916 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:22:26.916 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.628738+0000", 2024-09-11T08:22:26.917 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.4105999999999999e-05 2024-09-11T08:22:26.917 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.917 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.917 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:22:26.917 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.628752+0000", 2024-09-11T08:22:26.917 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3909e-05 2024-09-11T08:22:26.917 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.917 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.917 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:22:26.917 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.628758+0000", 2024-09-11T08:22:26.917 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.9830000000000001e-06 2024-09-11T08:22:26.917 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.917 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.918 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:26.918 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.628814+0000", 2024-09-11T08:22:26.918 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.6297e-05 2024-09-11T08:22:26.918 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.918 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.918 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:22:26.918 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.628820+0000", 2024-09-11T08:22:26.918 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.4200000000000004e-06 2024-09-11T08:22:26.918 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.918 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.918 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:26.918 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.601901+0000", 2024-09-11T08:22:26.918 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.97308029200000001 2024-09-11T08:22:26.918 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.919 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.919 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:22:26.919 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.601916+0000", 2024-09-11T08:22:26.919 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4888e-05 2024-09-11T08:22:26.919 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.919 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.919 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:26.919 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.787726+0000", 2024-09-11T08:22:26.919 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.185809914 2024-09-11T08:22:26.919 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.919 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.919 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:22:26.919 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.790484+0000", 2024-09-11T08:22:26.919 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0027581490000000001 2024-09-11T08:22:26.919 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.920 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.920 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 2,3,5", 2024-09-11T08:22:26.920 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.790769+0000", 2024-09-11T08:22:26.920 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00028509599999999998 2024-09-11T08:22:26.920 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.920 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.920 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:22:26.920 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.835253+0000", 2024-09-11T08:22:26.920 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.044484502000000002 2024-09-11T08:22:26.920 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.920 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.920 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:22:26.920 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.835360+0000", 2024-09-11T08:22:26.921 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00010631799999999999 2024-09-11T08:22:26.921 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.921 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.921 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:22:26.921 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.835369+0000", 2024-09-11T08:22:26.921 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.55e-06 2024-09-11T08:22:26.921 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.921 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.921 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:22:26.921 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.848835+0000", 2024-09-11T08:22:26.921 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.013465886999999999 2024-09-11T08:22:26.921 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.921 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.921 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:22:26.921 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.848891+0000", 2024-09-11T08:22:26.922 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.5890999999999997e-05 2024-09-11T08:22:26.922 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.922 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.922 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:22:26.922 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.848998+0000", 2024-09-11T08:22:26.922 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000106939 2024-09-11T08:22:26.922 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:26.922 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:22:26.922 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:26.922 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.922 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.922 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.24320.0:1804 5.15 5:a93a5511:::notify.2:head [watch reconnect cookie 94783556101344 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-11T08:22:26.922 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:19:08.688799+0000", 2024-09-11T08:22:26.922 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 198.017456565, 2024-09-11T08:22:26.923 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.92763364500000001, 2024-09-11T08:22:26.923 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:22:26.923 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "started", 2024-09-11T08:22:26.923 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:22:26.923 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.24320", 2024-09-11T08:22:26.923 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1503848706", 2024-09-11T08:22:26.923 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 1804 2024-09-11T08:22:26.923 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.923 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:22:26.923 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.923 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:22:26.923 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.688799+0000", 2024-09-11T08:22:26.923 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:26.923 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.924 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.924 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:22:26.924 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.688799+0000", 2024-09-11T08:22:26.924 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:26.924 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.924 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.924 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:22:26.924 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.688797+0000", 2024-09-11T08:22:26.924 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:22:26.924 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.924 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.924 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:22:26.924 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.688807+0000", 2024-09-11T08:22:26.924 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.1889999999999996e-06 2024-09-11T08:22:26.925 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.925 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.925 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:22:26.925 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.688826+0000", 2024-09-11T08:22:26.925 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.9813000000000002e-05 2024-09-11T08:22:26.925 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.925 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.925 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:22:26.925 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.688834+0000", 2024-09-11T08:22:26.925 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.7819999999999994e-06 2024-09-11T08:22:26.925 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.925 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.926 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:26.926 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.688883+0000", 2024-09-11T08:22:26.926 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.8806e-05 2024-09-11T08:22:26.926 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.926 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.926 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:22:26.926 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.688890+0000", 2024-09-11T08:22:26.926 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.9280000000000004e-06 2024-09-11T08:22:26.926 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.926 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.926 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:26.926 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.612372+0000", 2024-09-11T08:22:26.926 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.92348205699999997 2024-09-11T08:22:26.926 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.927 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.927 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for degraded object", 2024-09-11T08:22:26.927 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.612676+0000", 2024-09-11T08:22:26.927 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00030383 2024-09-11T08:22:26.927 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.927 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.927 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:26.927 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.613796+0000", 2024-09-11T08:22:26.927 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0011199339999999999 2024-09-11T08:22:26.927 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.927 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.927 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:22:26.927 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.613872+0000", 2024-09-11T08:22:26.927 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.6050000000000005e-05 2024-09-11T08:22:26.928 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.928 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.928 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:22:26.928 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.616433+0000", 2024-09-11T08:22:26.928 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0025613749999999998 2024-09-11T08:22:26.928 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:26.928 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:22:26.928 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:26.928 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.928 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.928 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15369.0:206716 9.0 9:057b3767:::benchmark_data_smithi150_133616_object91110:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-09-11T08:22:26.928 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:21:01.171560+0000", 2024-09-11T08:22:26.928 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 85.534695580000005, 2024-09-11T08:22:26.928 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.18158703600000001, 2024-09-11T08:22:26.929 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:22:26.929 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:22:26.929 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:22:26.929 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15369", 2024-09-11T08:22:26.929 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:22:26.929 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 206716 2024-09-11T08:22:26.929 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.929 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:22:26.929 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.929 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:22:26.929 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.171560+0000", 2024-09-11T08:22:26.929 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:26.929 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.929 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.930 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:22:26.930 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.171560+0000", 2024-09-11T08:22:26.930 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:26.930 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.930 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.930 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:22:26.930 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.171559+0000", 2024-09-11T08:22:26.930 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:22:26.930 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.930 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.930 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:22:26.930 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.171565+0000", 2024-09-11T08:22:26.930 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.7209999999999997e-06 2024-09-11T08:22:26.931 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.931 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.931 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:22:26.931 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.171580+0000", 2024-09-11T08:22:26.931 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4669e-05 2024-09-11T08:22:26.931 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.931 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.931 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:22:26.931 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.171586+0000", 2024-09-11T08:22:26.931 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.5470000000000004e-06 2024-09-11T08:22:26.931 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.931 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.931 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:26.931 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.171634+0000", 2024-09-11T08:22:26.932 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.8143000000000003e-05 2024-09-11T08:22:26.932 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.932 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.932 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:22:26.932 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.171738+0000", 2024-09-11T08:22:26.932 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00010432700000000001 2024-09-11T08:22:26.932 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.932 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.932 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 2,6", 2024-09-11T08:22:26.932 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.171846+0000", 2024-09-11T08:22:26.932 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000108279 2024-09-11T08:22:26.932 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.932 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.933 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:22:26.933 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.174408+0000", 2024-09-11T08:22:26.933 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0025620809999999999 2024-09-11T08:22:26.933 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.933 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.933 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:22:26.933 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.178140+0000", 2024-09-11T08:22:26.933 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.003731767 2024-09-11T08:22:26.933 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.933 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.933 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:22:26.933 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.353055+0000", 2024-09-11T08:22:26.933 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.17491469800000001 2024-09-11T08:22:26.933 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.934 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.934 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:22:26.934 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.353107+0000", 2024-09-11T08:22:26.934 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.2479999999999999e-05 2024-09-11T08:22:26.934 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.934 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.934 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:22:26.934 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.353147+0000", 2024-09-11T08:22:26.934 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.0077000000000003e-05 2024-09-11T08:22:26.934 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:26.934 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:22:26.934 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:26.934 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.934 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.934 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.15369.0:212763 9.3 e298/246 9:ce43ae85:::benchmark_data_smithi150_133616_object97157:head v 298'15904, mlcod=298'15902)", 2024-09-11T08:22:26.935 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:21:06.533414+0000", 2024-09-11T08:22:26.935 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 80.172842234000001, 2024-09-11T08:22:26.935 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.27374648499999998, 2024-09-11T08:22:26.935 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:22:26.935 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:22:26.935 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:22:26.935 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.935 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:22:26.935 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.533414+0000", 2024-09-11T08:22:26.935 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:26.935 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.935 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.935 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:22:26.936 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.533414+0000", 2024-09-11T08:22:26.936 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:26.936 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.936 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.936 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:22:26.936 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.533413+0000", 2024-09-11T08:22:26.936 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:22:26.936 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.936 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.936 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:22:26.936 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.533420+0000", 2024-09-11T08:22:26.936 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.2239999999999998e-06 2024-09-11T08:22:26.936 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.937 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.937 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:22:26.937 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.618921+0000", 2024-09-11T08:22:26.937 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.085500823000000004 2024-09-11T08:22:26.937 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.937 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.937 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:22:26.937 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.618927+0000", 2024-09-11T08:22:26.937 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.4979999999999997e-06 2024-09-11T08:22:26.937 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.937 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.937 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:26.937 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.618972+0000", 2024-09-11T08:22:26.937 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.5472000000000001e-05 2024-09-11T08:22:26.938 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.938 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.938 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:22:26.938 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.618994+0000", 2024-09-11T08:22:26.938 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.1676e-05 2024-09-11T08:22:26.938 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.938 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.938 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:22:26.938 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.807131+0000", 2024-09-11T08:22:26.938 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.18813698000000001 2024-09-11T08:22:26.938 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.938 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.938 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:22:26.939 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.807160+0000", 2024-09-11T08:22:26.939 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.9496999999999999e-05 2024-09-11T08:22:26.939 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:26.939 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:22:26.939 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:26.939 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.939 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.939 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.15369.0:212766 9.3 e298/246 9:ca00ffc2:::benchmark_data_smithi150_133616_object97160:head v 298'15905, mlcod=298'15902)", 2024-09-11T08:22:26.939 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:21:06.618944+0000", 2024-09-11T08:22:26.939 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 80.087312010999995, 2024-09-11T08:22:26.939 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.19080667700000001, 2024-09-11T08:22:26.939 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:22:26.939 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:22:26.940 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:22:26.940 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.940 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:22:26.940 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.618944+0000", 2024-09-11T08:22:26.940 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:26.940 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.940 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.940 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:22:26.940 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.618944+0000", 2024-09-11T08:22:26.940 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:26.940 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.940 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.940 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:22:26.941 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.618943+0000", 2024-09-11T08:22:26.941 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:22:26.941 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.941 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.941 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:22:26.941 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.618951+0000", 2024-09-11T08:22:26.941 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.6199999999999999e-06 2024-09-11T08:22:26.941 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.941 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.941 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:22:26.941 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.618957+0000", 2024-09-11T08:22:26.941 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.6189999999999996e-06 2024-09-11T08:22:26.941 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.941 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.942 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:22:26.942 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.618961+0000", 2024-09-11T08:22:26.942 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.2329999999999998e-06 2024-09-11T08:22:26.942 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.942 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.942 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:26.942 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.621622+0000", 2024-09-11T08:22:26.942 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0026617519999999999 2024-09-11T08:22:26.942 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.942 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.942 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:22:26.942 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.621639+0000", 2024-09-11T08:22:26.942 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.6543000000000001e-05 2024-09-11T08:22:26.942 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.943 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.943 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:22:26.943 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.809723+0000", 2024-09-11T08:22:26.943 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.188084476 2024-09-11T08:22:26.943 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.943 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.943 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:22:26.943 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.809751+0000", 2024-09-11T08:22:26.943 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.7189999999999999e-05 2024-09-11T08:22:26.944 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:26.944 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:22:26.944 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:26.944 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.944 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.944 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.15369.0:212770 9.e e298/246 9:79e6cbd3:::benchmark_data_smithi150_133616_object97164:head v 298'13227, mlcod=298'13225)", 2024-09-11T08:22:26.944 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:21:06.618975+0000", 2024-09-11T08:22:26.944 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 80.087281114999996, 2024-09-11T08:22:26.944 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.18815554900000001, 2024-09-11T08:22:26.944 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:22:26.944 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:22:26.944 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:22:26.944 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.945 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:22:26.945 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.618975+0000", 2024-09-11T08:22:26.945 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:26.945 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.945 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.945 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:22:26.945 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.618975+0000", 2024-09-11T08:22:26.945 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:26.945 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.945 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.945 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:22:26.945 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.618975+0000", 2024-09-11T08:22:26.945 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:22:26.946 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.946 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.946 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:22:26.946 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.618978+0000", 2024-09-11T08:22:26.946 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.3129999999999999e-06 2024-09-11T08:22:26.946 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.946 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.946 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:22:26.946 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.618983+0000", 2024-09-11T08:22:26.946 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.6e-06 2024-09-11T08:22:26.946 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.946 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.946 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:22:26.946 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.618987+0000", 2024-09-11T08:22:26.947 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.9890000000000003e-06 2024-09-11T08:22:26.947 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.947 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.947 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:26.947 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.619438+0000", 2024-09-11T08:22:26.947 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00045184400000000002 2024-09-11T08:22:26.947 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.947 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.947 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:22:26.947 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.619457+0000", 2024-09-11T08:22:26.947 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.8578999999999999e-05 2024-09-11T08:22:26.947 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.947 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.948 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:22:26.948 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.625854+0000", 2024-09-11T08:22:26.948 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0063966759999999996 2024-09-11T08:22:26.948 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.948 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.948 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:22:26.948 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.807130+0000", 2024-09-11T08:22:26.948 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.18127680099999999 2024-09-11T08:22:26.948 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:26.948 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:22:26.948 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:26.948 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.948 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.949 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.15369.0:212772 9.3 e298/246 9:cfdf609d:::benchmark_data_smithi150_133616_object97166:head v 298'15906, mlcod=298'15902)", 2024-09-11T08:22:26.949 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:21:06.618997+0000", 2024-09-11T08:22:26.949 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 80.087259036999995, 2024-09-11T08:22:26.949 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.190769993, 2024-09-11T08:22:26.949 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:22:26.949 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:22:26.949 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:22:26.949 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.949 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:22:26.949 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.618997+0000", 2024-09-11T08:22:26.949 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:26.949 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.949 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.949 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:22:26.950 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.618997+0000", 2024-09-11T08:22:26.950 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:26.950 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.950 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.950 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:22:26.950 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.618997+0000", 2024-09-11T08:22:26.950 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967296 2024-09-11T08:22:26.950 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.950 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.950 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:22:26.950 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.619000+0000", 2024-09-11T08:22:26.950 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.3790000000000002e-06 2024-09-11T08:22:26.950 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.951 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.951 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:22:26.951 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.619004+0000", 2024-09-11T08:22:26.951 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.0239999999999999e-06 2024-09-11T08:22:26.951 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.951 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.951 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:22:26.951 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.619007+0000", 2024-09-11T08:22:26.951 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.1e-06 2024-09-11T08:22:26.951 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.951 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.951 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:26.951 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.624034+0000", 2024-09-11T08:22:26.951 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.005027129 2024-09-11T08:22:26.951 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.952 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.952 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:22:26.952 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.624052+0000", 2024-09-11T08:22:26.952 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.8068999999999999e-05 2024-09-11T08:22:26.952 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.952 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.952 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:22:26.952 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.809753+0000", 2024-09-11T08:22:26.952 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.18570086899999999 2024-09-11T08:22:26.952 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.952 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.952 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:22:26.952 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.809767+0000", 2024-09-11T08:22:26.953 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3621999999999999e-05 2024-09-11T08:22:26.953 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:26.953 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:22:26.953 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:26.953 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.953 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.953 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.15369.0:212774 9.6 e298/271 9:6c4ab803:::benchmark_data_smithi150_133616_object97168:head v 298'13385, mlcod=298'13383)", 2024-09-11T08:22:26.953 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:21:06.619006+0000", 2024-09-11T08:22:26.953 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 80.087249693999993, 2024-09-11T08:22:26.953 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.18817231600000001, 2024-09-11T08:22:26.953 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:22:26.953 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:22:26.953 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:22:26.953 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.953 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:22:26.954 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.619006+0000", 2024-09-11T08:22:26.954 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:26.954 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.954 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.954 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:22:26.954 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.619006+0000", 2024-09-11T08:22:26.954 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:26.954 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.954 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.954 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:22:26.954 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.619006+0000", 2024-09-11T08:22:26.954 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:22:26.954 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.954 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.955 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:22:26.955 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.619013+0000", 2024-09-11T08:22:26.955 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.8399999999999997e-06 2024-09-11T08:22:26.955 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.955 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.955 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:22:26.955 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.619019+0000", 2024-09-11T08:22:26.955 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.5119999999999997e-06 2024-09-11T08:22:26.955 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.955 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.955 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:22:26.955 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.619023+0000", 2024-09-11T08:22:26.955 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1160000000000001e-06 2024-09-11T08:22:26.955 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.956 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.956 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:26.956 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.619452+0000", 2024-09-11T08:22:26.956 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00042851400000000001 2024-09-11T08:22:26.956 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.956 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.956 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:22:26.956 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.619469+0000", 2024-09-11T08:22:26.956 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.7521999999999999e-05 2024-09-11T08:22:26.956 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.956 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.956 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:22:26.956 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.807155+0000", 2024-09-11T08:22:26.956 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.18768555200000001 2024-09-11T08:22:26.956 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.957 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.957 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:22:26.957 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.807179+0000", 2024-09-11T08:22:26.957 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.3921e-05 2024-09-11T08:22:26.957 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:26.957 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:22:26.957 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:26.957 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.957 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.957 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.15369.0:212776 9.3 e298/246 9:cbebc502:::benchmark_data_smithi150_133616_object97170:head v 298'15907, mlcod=298'15902)", 2024-09-11T08:22:26.957 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:21:06.619011+0000", 2024-09-11T08:22:26.957 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 80.087244995999995, 2024-09-11T08:22:26.957 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.19077562100000001, 2024-09-11T08:22:26.957 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:22:26.958 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:22:26.958 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:22:26.958 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.958 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:22:26.958 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.619011+0000", 2024-09-11T08:22:26.958 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:26.958 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.958 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.958 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:22:26.958 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.619011+0000", 2024-09-11T08:22:26.958 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:26.958 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.958 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.958 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:22:26.959 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.619011+0000", 2024-09-11T08:22:26.959 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:22:26.959 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.959 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.959 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:22:26.959 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.619019+0000", 2024-09-11T08:22:26.959 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.3860000000000007e-06 2024-09-11T08:22:26.959 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.959 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.959 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:22:26.959 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.619023+0000", 2024-09-11T08:22:26.959 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.2200000000000003e-06 2024-09-11T08:22:26.959 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.959 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.960 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:22:26.960 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.619026+0000", 2024-09-11T08:22:26.960 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.05e-06 2024-09-11T08:22:26.960 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.960 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.960 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:26.960 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.626513+0000", 2024-09-11T08:22:26.960 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0074867429999999997 2024-09-11T08:22:26.960 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.960 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.960 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:22:26.960 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.626532+0000", 2024-09-11T08:22:26.960 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.8998999999999999e-05 2024-09-11T08:22:26.960 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.961 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.961 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:22:26.961 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.809769+0000", 2024-09-11T08:22:26.961 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.183236497 2024-09-11T08:22:26.961 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.961 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.961 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:22:26.961 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.809787+0000", 2024-09-11T08:22:26.961 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.7992999999999999e-05 2024-09-11T08:22:26.961 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:26.961 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:22:26.961 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:26.961 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.961 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.962 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15369.0:212781 9.0 9:0e872012:::benchmark_data_smithi150_133616_object97175:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-11T08:22:26.962 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:21:06.619479+0000", 2024-09-11T08:22:26.962 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 80.086776975999996, 2024-09-11T08:22:26.962 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.18786146400000001, 2024-09-11T08:22:26.962 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:22:26.962 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:22:26.962 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:22:26.962 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15369", 2024-09-11T08:22:26.962 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:22:26.962 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 212781 2024-09-11T08:22:26.962 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.962 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:22:26.962 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.962 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:22:26.963 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.619479+0000", 2024-09-11T08:22:26.963 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:26.963 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.963 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.963 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:22:26.963 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.619479+0000", 2024-09-11T08:22:26.963 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:26.963 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.963 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.963 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:22:26.963 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.619479+0000", 2024-09-11T08:22:26.963 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:22:26.963 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.963 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.964 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:22:26.964 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.619481+0000", 2024-09-11T08:22:26.964 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.5919999999999999e-06 2024-09-11T08:22:26.964 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.964 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.964 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:22:26.964 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.619487+0000", 2024-09-11T08:22:26.964 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.3299999999999998e-06 2024-09-11T08:22:26.964 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.964 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.964 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:22:26.964 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.619490+0000", 2024-09-11T08:22:26.964 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.8940000000000003e-06 2024-09-11T08:22:26.964 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.965 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.965 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:26.965 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.622440+0000", 2024-09-11T08:22:26.965 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0029497939999999999 2024-09-11T08:22:26.965 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.965 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.965 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:22:26.965 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.622558+0000", 2024-09-11T08:22:26.965 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000117433 2024-09-11T08:22:26.965 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.965 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.965 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 2,6", 2024-09-11T08:22:26.965 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.622661+0000", 2024-09-11T08:22:26.965 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00010323 2024-09-11T08:22:26.966 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.966 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.966 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:22:26.966 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.628517+0000", 2024-09-11T08:22:26.966 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0058560330000000001 2024-09-11T08:22:26.966 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.966 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.966 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:22:26.966 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.807157+0000", 2024-09-11T08:22:26.966 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.17864007700000001 2024-09-11T08:22:26.966 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.966 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.966 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:22:26.966 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.807252+0000", 2024-09-11T08:22:26.967 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.4641000000000004e-05 2024-09-11T08:22:26.967 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.967 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.967 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:22:26.967 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.807299+0000", 2024-09-11T08:22:26.967 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.6944999999999999e-05 2024-09-11T08:22:26.967 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.967 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.967 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:22:26.967 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.807341+0000", 2024-09-11T08:22:26.967 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1922999999999998e-05 2024-09-11T08:22:26.967 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:26.967 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:22:26.967 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:26.968 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.968 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.968 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.15369.0:212782 9.6 e298/271 9:625a7268:::benchmark_data_smithi150_133616_object97176:head v 298'13386, mlcod=298'13383)", 2024-09-11T08:22:26.968 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:21:06.620315+0000", 2024-09-11T08:22:26.968 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 80.085941169999998, 2024-09-11T08:22:26.968 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.18688198, 2024-09-11T08:22:26.968 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:22:26.968 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:22:26.968 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:22:26.968 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.968 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:22:26.968 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.620315+0000", 2024-09-11T08:22:26.968 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:26.968 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.969 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.969 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:22:26.969 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.620315+0000", 2024-09-11T08:22:26.969 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:26.969 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.969 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.969 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:22:26.969 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.620314+0000", 2024-09-11T08:22:26.969 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:22:26.969 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.969 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.969 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:22:26.969 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.620319+0000", 2024-09-11T08:22:26.970 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.2839999999999997e-06 2024-09-11T08:22:26.970 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.970 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.970 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:22:26.970 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.620328+0000", 2024-09-11T08:22:26.970 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.5359999999999992e-06 2024-09-11T08:22:26.970 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.970 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.970 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:22:26.970 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.620333+0000", 2024-09-11T08:22:26.970 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.079e-06 2024-09-11T08:22:26.970 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.970 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.970 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:26.970 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.625927+0000", 2024-09-11T08:22:26.971 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0055937030000000002 2024-09-11T08:22:26.971 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.971 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.971 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:22:26.971 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.625943+0000", 2024-09-11T08:22:26.971 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.6362e-05 2024-09-11T08:22:26.971 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.971 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.971 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:22:26.971 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.807182+0000", 2024-09-11T08:22:26.971 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.18123858400000001 2024-09-11T08:22:26.971 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.971 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.972 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:22:26.972 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.807197+0000", 2024-09-11T08:22:26.972 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.5098e-05 2024-09-11T08:22:26.972 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:26.972 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:22:26.972 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:26.972 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.972 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.972 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.15369.0:212780 9.d e298/278 9:b9df0a16:::benchmark_data_smithi150_133616_object97174:head v 298'13216, mlcod=298'13212)", 2024-09-11T08:22:26.972 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:21:06.624298+0000", 2024-09-11T08:22:26.972 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 80.081958475999997, 2024-09-11T08:22:26.972 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.18285809, 2024-09-11T08:22:26.972 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:22:26.972 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:22:26.973 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:22:26.973 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.973 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:22:26.973 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.624298+0000", 2024-09-11T08:22:26.973 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:26.973 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.973 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.973 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:22:26.973 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.624298+0000", 2024-09-11T08:22:26.973 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:26.973 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.973 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.973 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:22:26.974 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.624297+0000", 2024-09-11T08:22:26.974 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:22:26.974 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.974 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.974 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:22:26.974 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.624304+0000", 2024-09-11T08:22:26.974 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.0700000000000001e-06 2024-09-11T08:22:26.974 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.974 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.974 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:22:26.974 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.624313+0000", 2024-09-11T08:22:26.974 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.2790000000000004e-06 2024-09-11T08:22:26.974 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.974 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.975 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:22:26.975 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.624318+0000", 2024-09-11T08:22:26.975 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.0030000000000002e-06 2024-09-11T08:22:26.975 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.975 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.975 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:26.975 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.625409+0000", 2024-09-11T08:22:26.975 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0010904109999999999 2024-09-11T08:22:26.975 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.975 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.975 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:22:26.975 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.625425+0000", 2024-09-11T08:22:26.975 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.5931e-05 2024-09-11T08:22:26.975 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.975 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.976 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:22:26.976 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.807124+0000", 2024-09-11T08:22:26.976 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.181699369 2024-09-11T08:22:26.976 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.976 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.976 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:22:26.976 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.807156+0000", 2024-09-11T08:22:26.976 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.1640999999999997e-05 2024-09-11T08:22:26.976 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:26.976 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:22:26.976 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:26.976 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.976 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.976 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.15369.0:212786 9.f e298/271 9:fe1a4c9f:::benchmark_data_smithi150_133616_object97180:head v 298'13423, mlcod=298'13421)", 2024-09-11T08:22:26.976 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:21:06.625201+0000", 2024-09-11T08:22:26.977 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 80.081055441000004, 2024-09-11T08:22:26.977 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.18195381199999999, 2024-09-11T08:22:26.977 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:22:26.977 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:22:26.977 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:22:26.977 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.977 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:22:26.977 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.625201+0000", 2024-09-11T08:22:26.977 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:26.977 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.977 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.977 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:22:26.977 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.625201+0000", 2024-09-11T08:22:26.977 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:26.978 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.978 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.978 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:22:26.978 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.625200+0000", 2024-09-11T08:22:26.978 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:22:26.978 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.978 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.978 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:22:26.978 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.625214+0000", 2024-09-11T08:22:26.978 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4024e-05 2024-09-11T08:22:26.978 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.978 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.978 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:22:26.978 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.625223+0000", 2024-09-11T08:22:26.978 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.3829999999999998e-06 2024-09-11T08:22:26.979 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.979 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.979 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:22:26.979 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.625229+0000", 2024-09-11T08:22:26.979 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.9270000000000002e-06 2024-09-11T08:22:26.979 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.979 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.979 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:26.979 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.625272+0000", 2024-09-11T08:22:26.979 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.3035999999999999e-05 2024-09-11T08:22:26.979 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.979 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.979 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:22:26.979 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.625291+0000", 2024-09-11T08:22:26.980 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.8793000000000001e-05 2024-09-11T08:22:26.980 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.980 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.980 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:22:26.980 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.807117+0000", 2024-09-11T08:22:26.980 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.181825986 2024-09-11T08:22:26.980 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.980 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.980 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:22:26.980 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.807154+0000", 2024-09-11T08:22:26.980 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.7305999999999998e-05 2024-09-11T08:22:26.980 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:26.980 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:22:26.980 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:26.981 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.981 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.981 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop_reply(client.15369.0:212781 9.0 e298/246 ondisk, result = 0)", 2024-09-11T08:22:26.981 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:21:06.626377+0000", 2024-09-11T08:22:26.981 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 80.079879250000005, 2024-09-11T08:22:26.981 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.18078471800000001, 2024-09-11T08:22:26.981 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:22:26.981 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "started", 2024-09-11T08:22:26.981 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:22:26.981 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.981 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:22:26.981 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.626377+0000", 2024-09-11T08:22:26.981 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:26.982 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.982 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.982 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:22:26.982 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.626377+0000", 2024-09-11T08:22:26.982 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:26.982 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.982 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.982 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:22:26.982 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.626376+0000", 2024-09-11T08:22:26.982 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:22:26.982 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.982 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.982 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:22:26.982 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.626379+0000", 2024-09-11T08:22:26.982 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.1439999999999999e-06 2024-09-11T08:22:26.983 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.983 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.983 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:22:26.983 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.626386+0000", 2024-09-11T08:22:26.983 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.4119999999999998e-06 2024-09-11T08:22:26.983 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.983 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.983 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:22:26.983 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.626390+0000", 2024-09-11T08:22:26.983 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.4819999999999996e-06 2024-09-11T08:22:26.983 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.983 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.983 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:26.983 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.807122+0000", 2024-09-11T08:22:26.984 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.180732112 2024-09-11T08:22:26.984 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.984 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.984 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:22:26.984 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.807149+0000", 2024-09-11T08:22:26.984 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.7002999999999999e-05 2024-09-11T08:22:26.984 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.984 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.984 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:22:26.984 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.807161+0000", 2024-09-11T08:22:26.984 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2046e-05 2024-09-11T08:22:26.984 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:26.984 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:22:26.984 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:26.985 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.985 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.985 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.15369.0:212789 9.2 e298/246 9:4d7a08f9:::benchmark_data_smithi150_133616_object97183:head v 298'13254, mlcod=298'13252)", 2024-09-11T08:22:26.985 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:21:06.629297+0000", 2024-09-11T08:22:26.985 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 80.076959054, 2024-09-11T08:22:26.985 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.18916815200000001, 2024-09-11T08:22:26.985 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:22:26.985 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:22:26.985 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:22:26.985 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.985 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:22:26.985 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.629297+0000", 2024-09-11T08:22:26.985 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:26.985 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.986 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.986 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:22:26.986 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.629297+0000", 2024-09-11T08:22:26.986 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:26.986 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.986 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.986 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:22:26.986 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.629296+0000", 2024-09-11T08:22:26.986 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:22:26.986 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.986 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.986 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:22:26.986 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.629305+0000", 2024-09-11T08:22:26.986 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.5150000000000006e-06 2024-09-11T08:22:26.986 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.986 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.987 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:22:26.987 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.807125+0000", 2024-09-11T08:22:26.987 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.1778209 2024-09-11T08:22:26.987 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.987 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.987 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:22:26.987 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.807134+0000", 2024-09-11T08:22:26.987 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.0259999999999996e-06 2024-09-11T08:22:26.987 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.987 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.987 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:26.987 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.807191+0000", 2024-09-11T08:22:26.987 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.6659000000000001e-05 2024-09-11T08:22:26.987 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.987 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.987 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:22:26.988 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.807216+0000", 2024-09-11T08:22:26.988 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.4712999999999999e-05 2024-09-11T08:22:26.988 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.988 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.988 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:22:26.988 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.818435+0000", 2024-09-11T08:22:26.988 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.011218756999999999 2024-09-11T08:22:26.988 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.988 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.988 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:22:26.988 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.818465+0000", 2024-09-11T08:22:26.988 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.0519999999999999e-05 2024-09-11T08:22:26.988 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:26.989 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:22:26.989 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:26.989 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.989 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.989 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.15369.0:212791 9.d e298/278 9:bdbfd58b:::benchmark_data_smithi150_133616_object97185:head v 298'13217, mlcod=298'13214)", 2024-09-11T08:22:26.989 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:21:06.633455+0000", 2024-09-11T08:22:26.989 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 80.072801143000007, 2024-09-11T08:22:26.989 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.18205613700000001, 2024-09-11T08:22:26.989 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:22:26.990 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:22:26.990 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:22:26.990 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.990 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:22:26.990 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.633455+0000", 2024-09-11T08:22:26.990 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:26.990 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.990 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.990 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:22:26.990 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.633455+0000", 2024-09-11T08:22:26.990 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:26.990 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.991 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.991 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:22:26.991 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.633454+0000", 2024-09-11T08:22:26.991 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:22:26.991 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.991 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.991 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:22:26.991 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.633462+0000", 2024-09-11T08:22:26.991 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.7299999999999994e-06 2024-09-11T08:22:26.991 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.991 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.991 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:22:26.991 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.807103+0000", 2024-09-11T08:22:26.991 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.173640141 2024-09-11T08:22:26.992 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.992 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.992 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:22:26.992 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.807117+0000", 2024-09-11T08:22:26.992 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4664999999999999e-05 2024-09-11T08:22:26.992 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.992 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.992 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:26.992 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.807179+0000", 2024-09-11T08:22:26.992 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.1740000000000002e-05 2024-09-11T08:22:26.992 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.992 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.993 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:22:26.993 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.807199+0000", 2024-09-11T08:22:26.993 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.0452e-05 2024-09-11T08:22:26.993 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.993 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.993 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:22:26.993 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.815443+0000", 2024-09-11T08:22:26.993 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0082432909999999998 2024-09-11T08:22:26.993 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:26.993 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:26.993 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:22:26.993 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.815511+0000", 2024-09-11T08:22:26.993 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.8223000000000001e-05 2024-09-11T08:22:26.994 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:26.994 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:22:26.994 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:26.994 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:26.994 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:22:26.994 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-11T08:22:26.994 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:22:27.029 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:27.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:27.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:22:27.031 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 26 16 57062 57046 137.1 96.625 0.0101826 0.00728873 2024-09-11T08:22:27.131 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:22:27.137 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:27 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3949167334' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:22:27.167 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:27.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:27.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:22:27.269 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:27.300 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:27.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:27.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:27.402 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:27.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:27.436 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:27.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:27.449 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:27 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3949167334' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:22:27.537 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:22:27.573 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:27.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:27 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3949167334' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:22:27.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:27.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:22:27.675 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:22:27.711 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:27.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:27.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:22:27.813 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:27.847 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:27.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:27.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:27.949 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:27.985 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:27.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:27.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:28.030 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 27 16 58721 58705 135.862 103.688 0.00326554 0.0073548 2024-09-11T08:22:28.045 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-11T08:22:28.054 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:22:28.087 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:22:28.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:28.126 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:28.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:22:28.227 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:22:28.271 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:28.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:28.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:22:28.373 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:28.411 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:28.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:28.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:28.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:28 smithi150 ceph-mon[29276]: pgmap v677: 148 pgs: 148 active+clean; 3.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-09-11T08:22:28.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:28 smithi150 ceph-mon[39612]: pgmap v677: 148 pgs: 148 active+clean; 3.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-09-11T08:22:28.513 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:22:28.546 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:28.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:28.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:22:28.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:28 smithi181 ceph-mon[32701]: pgmap v677: 148 pgs: 148 active+clean; 3.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-09-11T08:22:28.648 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:28.686 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:28.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:28.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:28.779 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:22:28.787 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:22:28.814 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:28.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:28.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:22:28.822 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:22:28 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:22:28] "GET /metrics HTTP/1.1" 200 224626 "" "Prometheus/2.43.0" 2024-09-11T08:22:28.916 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:22:28.945 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:28.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:28.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:22:29.030 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 28 16 60277 60261 134.483 97.25 0.0116281 0.00743133 2024-09-11T08:22:29.047 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:22:29.073 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:22:28 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:22:29.080 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:29.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:29.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:22:29.182 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:29.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:29.222 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:29.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:29.323 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:22:29.361 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:29.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:29.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:22:29.464 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:22:29.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:29.501 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:29.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:22:29.602 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:22:29.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:29.651 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:29.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:22:29.752 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:29.792 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:29.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:29.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:29.895 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:29.931 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:29.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:29.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:30.030 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 29 15 61908 61893 133.363 102 0.0132211 0.00749406 2024-09-11T08:22:30.033 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:22:30.071 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:30.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:30.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:22:30.173 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:30.214 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:30.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:30.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:30.316 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:22:30.345 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:22:30.351 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:30.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:30.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:22:30.452 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:30.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:30 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:22:30.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:30 smithi150 ceph-mon[29276]: pgmap v678: 148 pgs: 148 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 126 MiB/s wr, 2.02k op/s 2024-09-11T08:22:30.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:30 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:22:30.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:30 smithi150 ceph-mon[39612]: pgmap v678: 148 pgs: 148 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 126 MiB/s wr, 2.02k op/s 2024-09-11T08:22:30.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:30.490 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:30.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:30.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:30 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:22:30.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:30 smithi181 ceph-mon[32701]: pgmap v678: 148 pgs: 148 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 126 MiB/s wr, 2.02k op/s 2024-09-11T08:22:30.592 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:22:30.631 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:30.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:30.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:22:30.733 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:30.771 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:30.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:30.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:30.827 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T08:22:30.873 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:30.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:30.918 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:30.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:31.019 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:22:31.031 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 30 16 63417 63401 132.056 94.25 0.0123602 0.00756675 2024-09-11T08:22:31.063 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:31.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:31.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:22:31.165 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:31.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:31.221 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:31.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:31.322 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:22:31.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:31.363 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:31.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:22:31.463 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:22:31.499 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:31.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:31.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:22:31.601 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:22:31.640 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:31.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:31.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:22:31.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:31 smithi150 ceph-mon[29276]: osdmap e318: 8 total, 8 up, 8 in 2024-09-11T08:22:31.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:31 smithi150 ceph-mon[29276]: pgmap v680: 148 pgs: 148 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-09-11T08:22:31.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:31 smithi150 ceph-mon[39612]: osdmap e318: 8 total, 8 up, 8 in 2024-09-11T08:22:31.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:31 smithi150 ceph-mon[39612]: pgmap v680: 148 pgs: 148 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-09-11T08:22:31.742 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:22:31.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:31.779 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:31.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:22:31.880 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:22:31.912 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:31.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:31.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:22:32.013 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:32.031 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 31 16 64917 64901 130.821 93.75 0.0126716 0.00763933 2024-09-11T08:22:32.050 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:32.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:32.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:32.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:31 smithi181 ceph-mon[32701]: osdmap e318: 8 total, 8 up, 8 in 2024-09-11T08:22:32.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:31 smithi181 ceph-mon[32701]: pgmap v680: 148 pgs: 148 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-09-11T08:22:32.151 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:22:32.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:32.198 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:32.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:22:32.299 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:32.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:32.345 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:32.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:32.446 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:32.481 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:32.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:32.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:32.531 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:22:32.534 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-09-11T08:22:32.583 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:22:32.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:32.617 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:32.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:22:32.718 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:22:32.754 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:32.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:32.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:22:32.856 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:32.891 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:32.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:32.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:32.928 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:32 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:22:32.928 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:32 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:22:32.928 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:32 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:22:32.928 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:32 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:22:32.992 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:22:33.022 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:33.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:33.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:22:33.031 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 32 16 66373 66357 129.576 91 0.00789124 0.00771329 2024-09-11T08:22:33.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:32 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:22:33.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:32 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:22:33.124 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:33.155 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:33.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:33.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:33.256 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:22:33.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:33.290 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:33.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:22:33.391 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:22:33.420 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":680,"stamp":"2024-09-11T08:22:31.026956+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3861387484,"num_objects":59107,"num_object_clones":0,"num_object_copies":177321,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59107,"num_whiteouts":0,"num_read":1740,"num_read_kb":1599,"num_write":59740,"num_write_kb":3771975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":237,"num_bytes_recovered":1776739,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59860,"ondisk_log_size":59860,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":445,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":14893552,"kb_used_data":11299428,"kb_used_omap":19,"kb_used_meta":3594028,"kb_avail":735067664,"statfs":{"total":767960285184,"available":752709287936,"internally_reserved":0,"allocated":11570614272,"data_stored":11567104028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19955,"internal_metadata":3680285197},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,2,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":806813696,"num_objects":12311,"num_object_clones":0,"num_object_copies":36933,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12311,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":12311,"num_write_kb":787904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.005238"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":346,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.638692+0000","last_change":"2024-09-11T08:21:09.740632+0000","last_active":"2024-09-11T08:22:20.638692+0000","last_peered":"2024-09-11T08:22:20.638692+0000","last_clean":"2024-09-11T08:22:20.638692+0000","last_became_active":"2024-09-11T08:19:29.169363+0000","last_became_peered":"2024-09-11T08:19:29.169363+0000","last_unstale":"2024-09-11T08:22:20.638692+0000","last_undegraded":"2024-09-11T08:22:20.638692+0000","last_fullsized":"2024-09-11T08:22:20.638692+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:09.740551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:21:09.740551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0095522019999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"312'19","reported_seq":344,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.636601+0000","last_change":"2024-09-11T08:21:59.122216+0000","last_active":"2024-09-11T08:22:20.636601+0000","last_peered":"2024-09-11T08:22:20.636601+0000","last_clean":"2024-09-11T08:22:20.636601+0000","last_became_active":"2024-09-11T08:07:32.048048+0000","last_became_peered":"2024-09-11T08:07:32.048048+0000","last_unstale":"2024-09-11T08:22:20.636601+0000","last_undegraded":"2024-09-11T08:22:20.636601+0000","last_fullsized":"2024-09-11T08:22:20.636601+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"312'19","last_scrub_stamp":"2024-09-11T08:21:59.122122+0000","last_deep_scrub":"312'19","last_deep_scrub_stamp":"2024-09-11T08:21:59.122122+0000","last_clean_scrub_stamp":"2024-09-11T08:21:59.122122+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:19.552550+0000","scrub_duration":0.0040323809999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":390,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.648358+0000","last_change":"2024-09-11T08:21:05.601324+0000","last_active":"2024-09-11T08:22:20.648358+0000","last_peered":"2024-09-11T08:22:20.648358+0000","last_clean":"2024-09-11T08:22:20.648358+0000","last_became_active":"2024-09-11T08:19:29.140997+0000","last_became_peered":"2024-09-11T08:19:29.140997+0000","last_unstale":"2024-09-11T08:22:20.648358+0000","last_undegraded":"2024-09-11T08:22:20.648358+0000","last_fullsized":"2024-09-11T08:22:20.648358+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:05.601245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:59.531202+0000","last_clean_scrub_stamp":"2024-09-11T08:21:05.601245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:30.011735+0000","scrub_duration":0.006405127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":334,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.635668+0000","last_change":"2024-09-11T08:20:11.523247+0000","last_active":"2024-09-11T08:22:20.635668+0000","last_peered":"2024-09-11T08:22:20.635668+0000","last_clean":"2024-09-11T08:22:20.635668+0000","last_became_active":"2024-09-11T08:19:29.147339+0000","last_became_peered":"2024-09-11T08:19:29.147339+0000","last_unstale":"2024-09-11T08:22:20.635668+0000","last_undegraded":"2024-09-11T08:22:20.635668+0000","last_fullsized":"2024-09-11T08:22:20.635668+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:11.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:02.470649+0000","last_clean_scrub_stamp":"2024-09-11T08:20:11.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0056491429999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":338,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.642817+0000","last_change":"2024-09-11T08:20:17.919584+0000","last_active":"2024-09-11T08:22:20.642817+0000","last_peered":"2024-09-11T08:22:20.642817+0000","last_clean":"2024-09-11T08:22:20.642817+0000","last_became_active":"2024-09-11T08:15:56.521104+0000","last_became_peered":"2024-09-11T08:15:56.521104+0000","last_unstale":"2024-09-11T08:22:20.642817+0000","last_undegraded":"2024-09-11T08:22:20.642817+0000","last_fullsized":"2024-09-11T08:22:20.642817+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:17.919504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:20:17.919504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022382589999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":121,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.641896+0000","last_change":"2024-09-11T08:20:52.936220+0000","last_active":"2024-09-11T08:22:20.641896+0000","last_peered":"2024-09-11T08:22:20.641896+0000","last_clean":"2024-09-11T08:22:20.641896+0000","last_became_active":"2024-09-11T08:17:37.550048+0000","last_became_peered":"2024-09-11T08:17:37.550048+0000","last_unstale":"2024-09-11T08:22:20.641896+0000","last_undegraded":"2024-09-11T08:22:20.641896+0000","last_fullsized":"2024-09-11T08:22:20.641896+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:52.936088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:20:52.936088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046527160000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"239'34","reported_seq":281,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.638803+0000","last_change":"2024-09-11T08:21:48.170435+0000","last_active":"2024-09-11T08:22:20.638803+0000","last_peered":"2024-09-11T08:22:20.638803+0000","last_clean":"2024-09-11T08:22:20.638803+0000","last_became_active":"2024-09-11T08:17:37.553866+0000","last_became_peered":"2024-09-11T08:17:37.553866+0000","last_unstale":"2024-09-11T08:22:20.638803+0000","last_undegraded":"2024-09-11T08:22:20.638803+0000","last_fullsized":"2024-09-11T08:22:20.638803+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'34","last_scrub_stamp":"2024-09-11T08:21:48.170378+0000","last_deep_scrub":"145'22","last_deep_scrub_stamp":"2024-09-11T08:15:57.478870+0000","last_clean_scrub_stamp":"2024-09-11T08:21:48.170378+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:51.705516+0000","scrub_duration":0.002279292,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":303,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.636522+0000","last_change":"2024-09-11T08:18:36.998586+0000","last_active":"2024-09-11T08:22:20.636522+0000","last_peered":"2024-09-11T08:22:20.636522+0000","last_clean":"2024-09-11T08:22:20.636522+0000","last_became_active":"2024-09-11T08:07:34.052243+0000","last_became_peered":"2024-09-11T08:07:34.052243+0000","last_unstale":"2024-09-11T08:22:20.636522+0000","last_undegraded":"2024-09-11T08:22:20.636522+0000","last_fullsized":"2024-09-11T08:22:20.636522+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:36.998503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:18:36.998503+0000","last_clean_scrub_stamp":"2024-09-11T08:18:36.998503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0067563429999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":66,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.632832+0000","last_change":"2024-09-11T08:19:42.490622+0000","last_active":"2024-09-11T08:22:20.632832+0000","last_peered":"2024-09-11T08:22:20.632832+0000","last_clean":"2024-09-11T08:22:20.632832+0000","last_became_active":"2024-09-11T08:19:29.142228+0000","last_became_peered":"2024-09-11T08:19:29.142228+0000","last_unstale":"2024-09-11T08:22:20.632832+0000","last_undegraded":"2024-09-11T08:22:20.632832+0000","last_fullsized":"2024-09-11T08:22:20.632832+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:19:42.490567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:54.878409+0000","last_clean_scrub_stamp":"2024-09-11T08:19:42.490567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001226284,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"239'15","reported_seq":343,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.635906+0000","last_change":"2024-09-11T08:20:13.492673+0000","last_active":"2024-09-11T08:22:20.635906+0000","last_peered":"2024-09-11T08:22:20.635906+0000","last_clean":"2024-09-11T08:22:20.635906+0000","last_became_active":"2024-09-11T08:07:32.044626+0000","last_became_peered":"2024-09-11T08:07:32.044626+0000","last_unstale":"2024-09-11T08:22:20.635906+0000","last_undegraded":"2024-09-11T08:22:20.635906+0000","last_fullsized":"2024-09-11T08:22:20.635906+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'15","last_scrub_stamp":"2024-09-11T08:20:13.492593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:20:13.492593+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023810210000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":66,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.633159+0000","last_change":"2024-09-11T08:20:18.900646+0000","last_active":"2024-09-11T08:22:20.633159+0000","last_peered":"2024-09-11T08:22:20.633159+0000","last_clean":"2024-09-11T08:22:20.633159+0000","last_became_active":"2024-09-11T08:19:29.192476+0000","last_became_peered":"2024-09-11T08:19:29.192476+0000","last_unstale":"2024-09-11T08:22:20.633159+0000","last_undegraded":"2024-09-11T08:22:20.633159+0000","last_fullsized":"2024-09-11T08:22:20.633159+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:18.900482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:20:18.900482+0000","last_clean_scrub_stamp":"2024-09-11T08:20:18.900482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.027755386,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":337,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.640512+0000","last_change":"2024-09-11T08:21:52.687247+0000","last_active":"2024-09-11T08:22:20.640512+0000","last_peered":"2024-09-11T08:22:20.640512+0000","last_clean":"2024-09-11T08:22:20.640512+0000","last_became_active":"2024-09-11T08:14:24.792595+0000","last_became_peered":"2024-09-11T08:14:24.792595+0000","last_unstale":"2024-09-11T08:22:20.640512+0000","last_undegraded":"2024-09-11T08:22:20.640512+0000","last_fullsized":"2024-09-11T08:22:20.640512+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:52.687195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:21:52.687195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:12.642688+0000","scrub_duration":0.0017906930000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":66,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.634531+0000","last_change":"2024-09-11T08:21:48.418367+0000","last_active":"2024-09-11T08:22:20.634531+0000","last_peered":"2024-09-11T08:22:20.634531+0000","last_clean":"2024-09-11T08:22:20.634531+0000","last_became_active":"2024-09-11T08:19:29.140493+0000","last_became_peered":"2024-09-11T08:19:29.140493+0000","last_unstale":"2024-09-11T08:22:20.634531+0000","last_undegraded":"2024-09-11T08:22:20.634531+0000","last_fullsized":"2024-09-11T08:22:20.634531+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:48.418276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:21:48.418276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:51.125352+0000","scrub_duration":0.0015283429999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"239'9","reported_seq":333,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.635696+0000","last_change":"2024-09-11T08:22:01.123132+0000","last_active":"2024-09-11T08:22:20.635696+0000","last_peered":"2024-09-11T08:22:20.635696+0000","last_clean":"2024-09-11T08:22:20.635696+0000","last_became_active":"2024-09-11T08:07:32.048661+0000","last_became_peered":"2024-09-11T08:07:32.048661+0000","last_unstale":"2024-09-11T08:22:20.635696+0000","last_undegraded":"2024-09-11T08:22:20.635696+0000","last_fullsized":"2024-09-11T08:22:20.635696+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'9","last_scrub_stamp":"2024-09-11T08:22:01.123040+0000","last_deep_scrub":"139'7","last_deep_scrub_stamp":"2024-09-11T08:12:24.869318+0000","last_clean_scrub_stamp":"2024-09-11T08:22:01.123040+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:01.721028+0000","scrub_duration":0.004407467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":149,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.634678+0000","last_change":"2024-09-11T08:20:53.087846+0000","last_active":"2024-09-11T08:22:20.634678+0000","last_peered":"2024-09-11T08:22:20.634678+0000","last_clean":"2024-09-11T08:22:20.634678+0000","last_became_active":"2024-09-11T08:15:56.526954+0000","last_became_peered":"2024-09-11T08:15:56.526954+0000","last_unstale":"2024-09-11T08:22:20.634678+0000","last_undegraded":"2024-09-11T08:22:20.634678+0000","last_fullsized":"2024-09-11T08:22:20.634678+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:53.087699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:20:53.087699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024133610000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":66,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.634992+0000","last_change":"2024-09-11T08:20:19.930386+0000","last_active":"2024-09-11T08:22:20.634992+0000","last_peered":"2024-09-11T08:22:20.634992+0000","last_clean":"2024-09-11T08:22:20.634992+0000","last_became_active":"2024-09-11T08:19:29.143572+0000","last_became_peered":"2024-09-11T08:19:29.143572+0000","last_unstale":"2024-09-11T08:22:20.634992+0000","last_undegraded":"2024-09-11T08:22:20.634992+0000","last_fullsized":"2024-09-11T08:22:20.634992+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:19.927616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:20:19.927616+0000","last_clean_scrub_stamp":"2024-09-11T08:20:19.927616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0062725719999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":334,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.638051+0000","last_change":"2024-09-11T08:20:49.471916+0000","last_active":"2024-09-11T08:22:20.638051+0000","last_peered":"2024-09-11T08:22:20.638051+0000","last_clean":"2024-09-11T08:22:20.638051+0000","last_became_active":"2024-09-11T08:19:29.149081+0000","last_became_peered":"2024-09-11T08:19:29.149081+0000","last_unstale":"2024-09-11T08:22:20.638051+0000","last_undegraded":"2024-09-11T08:22:20.638051+0000","last_fullsized":"2024-09-11T08:22:20.638051+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:49.471510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:20:49.471510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048182779999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":346,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.639016+0000","last_change":"2024-09-11T08:20:09.565068+0000","last_active":"2024-09-11T08:22:20.639016+0000","last_peered":"2024-09-11T08:22:20.639016+0000","last_clean":"2024-09-11T08:22:20.639016+0000","last_became_active":"2024-09-11T08:15:56.516941+0000","last_became_peered":"2024-09-11T08:15:56.516941+0000","last_unstale":"2024-09-11T08:22:20.639016+0000","last_undegraded":"2024-09-11T08:22:20.639016+0000","last_fullsized":"2024-09-11T08:22:20.639016+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:09.564941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:20:09.564941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018499059999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"239'34","reported_seq":384,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.639548+0000","last_change":"2024-09-11T08:21:01.847533+0000","last_active":"2024-09-11T08:22:20.639548+0000","last_peered":"2024-09-11T08:22:20.639548+0000","last_clean":"2024-09-11T08:22:20.639548+0000","last_became_active":"2024-09-11T08:07:32.046363+0000","last_became_peered":"2024-09-11T08:07:32.046363+0000","last_unstale":"2024-09-11T08:22:20.639548+0000","last_undegraded":"2024-09-11T08:22:20.639548+0000","last_fullsized":"2024-09-11T08:22:20.639548+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'34","last_scrub_stamp":"2024-09-11T08:21:01.847438+0000","last_deep_scrub":"107'22","last_deep_scrub_stamp":"2024-09-11T08:16:04.159211+0000","last_clean_scrub_stamp":"2024-09-11T08:21:01.847438+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0079036850000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":343,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.650618+0000","last_change":"2024-09-11T08:21:46.819968+0000","last_active":"2024-09-11T08:22:20.650618+0000","last_peered":"2024-09-11T08:22:20.650618+0000","last_clean":"2024-09-11T08:22:20.650618+0000","last_became_active":"2024-09-11T08:07:34.055410+0000","last_became_peered":"2024-09-11T08:07:34.055410+0000","last_unstale":"2024-09-11T08:22:20.650618+0000","last_undegraded":"2024-09-11T08:22:20.650618+0000","last_fullsized":"2024-09-11T08:22:20.650618+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:46.819905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:19:28.870211+0000","last_clean_scrub_stamp":"2024-09-11T08:21:46.819905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:01.293541+0000","scrub_duration":0.0018277110000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":141,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.641279+0000","last_change":"2024-09-11T08:21:51.420044+0000","last_active":"2024-09-11T08:22:20.641279+0000","last_peered":"2024-09-11T08:22:20.641279+0000","last_clean":"2024-09-11T08:22:20.641279+0000","last_became_active":"2024-09-11T08:19:29.144780+0000","last_became_peered":"2024-09-11T08:19:29.144780+0000","last_unstale":"2024-09-11T08:22:20.641279+0000","last_undegraded":"2024-09-11T08:22:20.641279+0000","last_fullsized":"2024-09-11T08:22:20.641279+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:51.419976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:21:51.419976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:51.718685+0000","scrub_duration":0.0015353280000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":327,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.636033+0000","last_change":"2024-09-11T08:21:52.110853+0000","last_active":"2024-09-11T08:22:20.636033+0000","last_peered":"2024-09-11T08:22:20.636033+0000","last_clean":"2024-09-11T08:22:20.636033+0000","last_became_active":"2024-09-11T08:07:30.046284+0000","last_became_peered":"2024-09-11T08:07:30.046284+0000","last_unstale":"2024-09-11T08:22:20.636033+0000","last_undegraded":"2024-09-11T08:22:20.636033+0000","last_fullsized":"2024-09-11T08:22:20.636033+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:52.110756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:21:52.110756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:12.473550+0000","scrub_duration":0.001648369,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"239'18","reported_seq":191,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.641565+0000","last_change":"2024-09-11T08:21:50.171673+0000","last_active":"2024-09-11T08:22:20.641565+0000","last_peered":"2024-09-11T08:22:20.641565+0000","last_clean":"2024-09-11T08:22:20.641565+0000","last_became_active":"2024-09-11T08:17:37.553402+0000","last_became_peered":"2024-09-11T08:17:37.553402+0000","last_unstale":"2024-09-11T08:22:20.641565+0000","last_undegraded":"2024-09-11T08:22:20.641565+0000","last_fullsized":"2024-09-11T08:22:20.641565+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'18","last_scrub_stamp":"2024-09-11T08:21:50.171618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:21:50.171618+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:05.493899+0000","scrub_duration":0.001971993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":323,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.638470+0000","last_change":"2024-09-11T08:21:54.144410+0000","last_active":"2024-09-11T08:22:20.638470+0000","last_peered":"2024-09-11T08:22:20.638470+0000","last_clean":"2024-09-11T08:22:20.638470+0000","last_became_active":"2024-09-11T08:07:34.056528+0000","last_became_peered":"2024-09-11T08:07:34.056528+0000","last_unstale":"2024-09-11T08:22:20.638470+0000","last_undegraded":"2024-09-11T08:22:20.638470+0000","last_fullsized":"2024-09-11T08:22:20.638470+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:54.144345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:10:40.110490+0000","last_clean_scrub_stamp":"2024-09-11T08:21:54.144345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:10.135298+0000","scrub_duration":0.0015696499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":304,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.636370+0000","last_change":"2024-09-11T08:21:05.279818+0000","last_active":"2024-09-11T08:22:20.636370+0000","last_peered":"2024-09-11T08:22:20.636370+0000","last_clean":"2024-09-11T08:22:20.636370+0000","last_became_active":"2024-09-11T08:19:29.149754+0000","last_became_peered":"2024-09-11T08:19:29.149754+0000","last_unstale":"2024-09-11T08:22:20.636370+0000","last_undegraded":"2024-09-11T08:22:20.636370+0000","last_fullsized":"2024-09-11T08:22:20.636370+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-11T08:21:05.279671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:21:05.279671+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033543840000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":337,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.648309+0000","last_change":"2024-09-11T08:21:06.624368+0000","last_active":"2024-09-11T08:22:20.648309+0000","last_peered":"2024-09-11T08:22:20.648309+0000","last_clean":"2024-09-11T08:22:20.648309+0000","last_became_active":"2024-09-11T08:07:30.045932+0000","last_became_peered":"2024-09-11T08:07:30.045932+0000","last_unstale":"2024-09-11T08:22:20.648309+0000","last_undegraded":"2024-09-11T08:22:20.648309+0000","last_fullsized":"2024-09-11T08:22:20.648309+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:06.624287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:21:06.624287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:27.183588+0000","scrub_duration":0.0049003370000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"239'20","reported_seq":427,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.638173+0000","last_change":"2024-09-11T08:21:46.541430+0000","last_active":"2024-09-11T08:22:20.638173+0000","last_peered":"2024-09-11T08:22:20.638173+0000","last_clean":"2024-09-11T08:22:20.638173+0000","last_became_active":"2024-09-11T08:15:56.515817+0000","last_became_peered":"2024-09-11T08:15:56.515817+0000","last_unstale":"2024-09-11T08:22:20.638173+0000","last_undegraded":"2024-09-11T08:22:20.638173+0000","last_fullsized":"2024-09-11T08:22:20.638173+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'20","last_scrub_stamp":"2024-09-11T08:21:46.541374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:21:46.541374+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:11.112350+0000","scrub_duration":0.0020197930000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"254'12","reported_seq":579,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:23.730543+0000","last_change":"2024-09-11T08:21:52.380933+0000","last_active":"2024-09-11T08:22:23.730543+0000","last_peered":"2024-09-11T08:22:23.730543+0000","last_clean":"2024-09-11T08:22:23.730543+0000","last_became_active":"2024-09-11T08:15:56.525498+0000","last_became_peered":"2024-09-11T08:15:56.525498+0000","last_unstale":"2024-09-11T08:22:23.730543+0000","last_undegraded":"2024-09-11T08:22:23.730543+0000","last_fullsized":"2024-09-11T08:22:23.730543+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'12","last_scrub_stamp":"2024-09-11T08:21:52.380812+0000","last_deep_scrub":"254'12","last_deep_scrub_stamp":"2024-09-11T08:18:34.253865+0000","last_clean_scrub_stamp":"2024-09-11T08:21:52.380812+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:03.869144+0000","scrub_duration":0.0017919870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"317'3646","reported_seq":3663,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-11T08:22:27.871360+0000","last_change":"2024-09-11T08:21:47.139820+0000","last_active":"2024-09-11T08:22:27.871360+0000","last_peered":"2024-09-11T08:22:27.871360+0000","last_clean":"2024-09-11T08:22:27.871360+0000","last_became_active":"2024-09-11T08:21:47.139412+0000","last_became_peered":"2024-09-11T08:21:47.139412+0000","last_unstale":"2024-09-11T08:22:27.871360+0000","last_undegraded":"2024-09-11T08:22:27.871360+0000","last_fullsized":"2024-09-11T08:22:27.871360+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":308,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_clean_scrub_stamp":"2024-09-11T08:21:46.117341+0000","objects_scrubbed":0,"log_size":3646,"log_dups_size":0,"ondisk_log_size":3646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:58.457112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238944256,"num_objects":3646,"num_object_clones":0,"num_object_copies":10938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3646,"num_write_kb":233344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"239'27","reported_seq":388,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.651576+0000","last_change":"2024-09-11T08:21:11.723564+0000","last_active":"2024-09-11T08:22:20.651576+0000","last_peered":"2024-09-11T08:22:20.651576+0000","last_clean":"2024-09-11T08:22:20.651576+0000","last_became_active":"2024-09-11T08:07:32.054317+0000","last_became_peered":"2024-09-11T08:07:32.054317+0000","last_unstale":"2024-09-11T08:22:20.651576+0000","last_undegraded":"2024-09-11T08:22:20.651576+0000","last_fullsized":"2024-09-11T08:22:20.651576+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'27","last_scrub_stamp":"2024-09-11T08:21:11.722291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:21:11.722291+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.024639298,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":357,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.642066+0000","last_change":"2024-09-11T08:22:00.708044+0000","last_active":"2024-09-11T08:22:20.642066+0000","last_peered":"2024-09-11T08:22:20.642066+0000","last_clean":"2024-09-11T08:22:20.642066+0000","last_became_active":"2024-09-11T08:15:56.521943+0000","last_became_peered":"2024-09-11T08:15:56.521943+0000","last_unstale":"2024-09-11T08:22:20.642066+0000","last_undegraded":"2024-09-11T08:22:20.642066+0000","last_fullsized":"2024-09-11T08:22:20.642066+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:22:00.707990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:22:00.707990+0000","last_clean_scrub_stamp":"2024-09-11T08:22:00.707990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:02.700790+0000","scrub_duration":0.001504299,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"271'10","reported_seq":528,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:23.728610+0000","last_change":"2024-09-11T08:19:29.165123+0000","last_active":"2024-09-11T08:22:23.728610+0000","last_peered":"2024-09-11T08:22:23.728610+0000","last_clean":"2024-09-11T08:22:23.728610+0000","last_became_active":"2024-09-11T08:19:29.149773+0000","last_became_peered":"2024-09-11T08:19:29.149773+0000","last_unstale":"2024-09-11T08:22:23.728610+0000","last_undegraded":"2024-09-11T08:22:23.728610+0000","last_fullsized":"2024-09-11T08:22:23.728610+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-11T08:18:44.817179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:18:44.817179+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022442809999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":66,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.632629+0000","last_change":"2024-09-11T08:20:10.875817+0000","last_active":"2024-09-11T08:22:20.632629+0000","last_peered":"2024-09-11T08:22:20.632629+0000","last_clean":"2024-09-11T08:22:20.632629+0000","last_became_active":"2024-09-11T08:19:29.164735+0000","last_became_peered":"2024-09-11T08:19:29.164735+0000","last_unstale":"2024-09-11T08:22:20.632629+0000","last_undegraded":"2024-09-11T08:22:20.632629+0000","last_fullsized":"2024-09-11T08:22:20.632629+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:10.875651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:06.616332+0000","last_clean_scrub_stamp":"2024-09-11T08:20:10.875651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.018223294000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"317'3524","reported_seq":3541,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-11T08:22:27.882583+0000","last_change":"2024-09-11T08:21:47.141276+0000","last_active":"2024-09-11T08:22:27.882583+0000","last_peered":"2024-09-11T08:22:27.882583+0000","last_clean":"2024-09-11T08:22:27.882583+0000","last_became_active":"2024-09-11T08:21:47.140961+0000","last_became_peered":"2024-09-11T08:21:47.140961+0000","last_unstale":"2024-09-11T08:22:27.882583+0000","last_undegraded":"2024-09-11T08:22:27.882583+0000","last_fullsized":"2024-09-11T08:22:27.882583+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":308,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_clean_scrub_stamp":"2024-09-11T08:21:46.117341+0000","objects_scrubbed":0,"log_size":3524,"log_dups_size":0,"ondisk_log_size":3524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:57.094119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230948864,"num_objects":3524,"num_object_clones":0,"num_object_copies":10572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3524,"num_write_kb":225536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"239'30","reported_seq":373,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.634000+0000","last_change":"2024-09-11T08:20:15.829563+0000","last_active":"2024-09-11T08:22:20.634000+0000","last_peered":"2024-09-11T08:22:20.634000+0000","last_clean":"2024-09-11T08:22:20.634000+0000","last_became_active":"2024-09-11T08:17:37.555703+0000","last_became_peered":"2024-09-11T08:17:37.555703+0000","last_unstale":"2024-09-11T08:22:20.634000+0000","last_undegraded":"2024-09-11T08:22:20.634000+0000","last_fullsized":"2024-09-11T08:22:20.634000+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'30","last_scrub_stamp":"2024-09-11T08:20:15.829476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:20:15.829476+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0058827109999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":307,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.634118+0000","last_change":"2024-09-11T08:21:02.160052+0000","last_active":"2024-09-11T08:22:20.634118+0000","last_peered":"2024-09-11T08:22:20.634118+0000","last_clean":"2024-09-11T08:22:20.634118+0000","last_became_active":"2024-09-11T08:17:37.553579+0000","last_became_peered":"2024-09-11T08:17:37.553579+0000","last_unstale":"2024-09-11T08:22:20.634118+0000","last_undegraded":"2024-09-11T08:22:20.634118+0000","last_fullsized":"2024-09-11T08:22:20.634118+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:02.159910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:21:02.159910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0086008690000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":340,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.635822+0000","last_change":"2024-09-11T08:20:57.523721+0000","last_active":"2024-09-11T08:22:20.635822+0000","last_peered":"2024-09-11T08:22:20.635822+0000","last_clean":"2024-09-11T08:22:20.635822+0000","last_became_active":"2024-09-11T08:15:56.522606+0000","last_became_peered":"2024-09-11T08:15:56.522606+0000","last_unstale":"2024-09-11T08:22:20.635822+0000","last_undegraded":"2024-09-11T08:22:20.635822+0000","last_fullsized":"2024-09-11T08:22:20.635822+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:57.523652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:20:57.523652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041561580000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":314,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.635836+0000","last_change":"2024-09-11T08:20:28.457738+0000","last_active":"2024-09-11T08:22:20.635836+0000","last_peered":"2024-09-11T08:22:20.635836+0000","last_clean":"2024-09-11T08:22:20.635836+0000","last_became_active":"2024-09-11T08:07:36.069225+0000","last_became_peered":"2024-09-11T08:07:36.069225+0000","last_unstale":"2024-09-11T08:22:20.635836+0000","last_undegraded":"2024-09-11T08:22:20.635836+0000","last_fullsized":"2024-09-11T08:22:20.635836+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:28.457660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:04.937884+0000","last_clean_scrub_stamp":"2024-09-11T08:20:28.457660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028279630000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"317'3583","reported_seq":3600,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-11T08:22:27.904214+0000","last_change":"2024-09-11T08:21:47.139560+0000","last_active":"2024-09-11T08:22:27.904214+0000","last_peered":"2024-09-11T08:22:27.904214+0000","last_clean":"2024-09-11T08:22:27.904214+0000","last_became_active":"2024-09-11T08:21:47.139248+0000","last_became_peered":"2024-09-11T08:21:47.139248+0000","last_unstale":"2024-09-11T08:22:27.904214+0000","last_undegraded":"2024-09-11T08:22:27.904214+0000","last_fullsized":"2024-09-11T08:22:27.904214+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":308,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_clean_scrub_stamp":"2024-09-11T08:21:46.117341+0000","objects_scrubbed":0,"log_size":3583,"log_dups_size":0,"ondisk_log_size":3583,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:54.521250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234815488,"num_objects":3583,"num_object_clones":0,"num_object_copies":10749,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3583,"num_write_kb":229312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"239'28","reported_seq":409,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.636075+0000","last_change":"2024-09-11T08:20:55.458131+0000","last_active":"2024-09-11T08:22:20.636075+0000","last_peered":"2024-09-11T08:22:20.636075+0000","last_clean":"2024-09-11T08:22:20.636075+0000","last_became_active":"2024-09-11T08:19:29.148311+0000","last_became_peered":"2024-09-11T08:19:29.148311+0000","last_unstale":"2024-09-11T08:22:20.636075+0000","last_undegraded":"2024-09-11T08:22:20.636075+0000","last_fullsized":"2024-09-11T08:22:20.636075+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'28","last_scrub_stamp":"2024-09-11T08:20:55.458034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:20:55.458034+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0051959880000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":336,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.638431+0000","last_change":"2024-09-11T08:22:03.564355+0000","last_active":"2024-09-11T08:22:20.638431+0000","last_peered":"2024-09-11T08:22:20.638431+0000","last_clean":"2024-09-11T08:22:20.638431+0000","last_became_active":"2024-09-11T08:07:30.050484+0000","last_became_peered":"2024-09-11T08:07:30.050484+0000","last_unstale":"2024-09-11T08:22:20.638431+0000","last_undegraded":"2024-09-11T08:22:20.638431+0000","last_fullsized":"2024-09-11T08:22:20.638431+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:22:03.564268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:47.960329+0000","last_clean_scrub_stamp":"2024-09-11T08:22:03.564268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:18.333900+0000","scrub_duration":0.0043127169999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"82'9","reported_seq":524,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:23.728845+0000","last_change":"2024-09-11T08:21:00.841525+0000","last_active":"2024-09-11T08:22:23.728845+0000","last_peered":"2024-09-11T08:22:23.728845+0000","last_clean":"2024-09-11T08:22:23.728845+0000","last_became_active":"2024-09-11T08:07:34.056143+0000","last_became_peered":"2024-09-11T08:07:34.056143+0000","last_unstale":"2024-09-11T08:22:23.728845+0000","last_undegraded":"2024-09-11T08:22:23.728845+0000","last_fullsized":"2024-09-11T08:22:23.728845+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-11T08:21:00.841426+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-09-11T08:21:00.841426+0000","last_clean_scrub_stamp":"2024-09-11T08:21:00.841426+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0051535979999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":311,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.633999+0000","last_change":"2024-09-11T08:21:51.600406+0000","last_active":"2024-09-11T08:22:20.633999+0000","last_peered":"2024-09-11T08:22:20.633999+0000","last_clean":"2024-09-11T08:22:20.633999+0000","last_became_active":"2024-09-11T08:14:24.788968+0000","last_became_peered":"2024-09-11T08:14:24.788968+0000","last_unstale":"2024-09-11T08:22:20.633999+0000","last_undegraded":"2024-09-11T08:22:20.633999+0000","last_fullsized":"2024-09-11T08:22:20.633999+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:51.600338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:21:51.600338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:08.612553+0000","scrub_duration":0.0016909780000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.c","version":"317'3635","reported_seq":3652,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-11T08:22:28.042283+0000","last_change":"2024-09-11T08:21:47.139581+0000","last_active":"2024-09-11T08:22:28.042283+0000","last_peered":"2024-09-11T08:22:28.042283+0000","last_clean":"2024-09-11T08:22:28.042283+0000","last_became_active":"2024-09-11T08:21:47.139258+0000","last_became_peered":"2024-09-11T08:21:47.139258+0000","last_unstale":"2024-09-11T08:22:28.042283+0000","last_undegraded":"2024-09-11T08:22:28.042283+0000","last_fullsized":"2024-09-11T08:22:28.042283+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":308,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_clean_scrub_stamp":"2024-09-11T08:21:46.117341+0000","objects_scrubbed":0,"log_size":3635,"log_dups_size":0,"ondisk_log_size":3635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:06.530718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238223360,"num_objects":3635,"num_object_clones":0,"num_object_copies":10905,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3635,"num_write_kb":232640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"239'20","reported_seq":415,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.650157+0000","last_change":"2024-09-11T08:21:56.861643+0000","last_active":"2024-09-11T08:22:20.650157+0000","last_peered":"2024-09-11T08:22:20.650157+0000","last_clean":"2024-09-11T08:22:20.650157+0000","last_became_active":"2024-09-11T08:19:29.143486+0000","last_became_peered":"2024-09-11T08:19:29.143486+0000","last_unstale":"2024-09-11T08:22:20.650157+0000","last_undegraded":"2024-09-11T08:22:20.650157+0000","last_fullsized":"2024-09-11T08:22:20.650157+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'20","last_scrub_stamp":"2024-09-11T08:21:56.861587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:21:56.861587+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:21.934014+0000","scrub_duration":0.0021882690000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":66,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.632693+0000","last_change":"2024-09-11T08:20:55.691331+0000","last_active":"2024-09-11T08:22:20.632693+0000","last_peered":"2024-09-11T08:22:20.632693+0000","last_clean":"2024-09-11T08:22:20.632693+0000","last_became_active":"2024-09-11T08:19:29.143310+0000","last_became_peered":"2024-09-11T08:19:29.143310+0000","last_unstale":"2024-09-11T08:22:20.632693+0000","last_undegraded":"2024-09-11T08:22:20.632693+0000","last_fullsized":"2024-09-11T08:22:20.632693+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:55.690500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:20:55.690500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013778218,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"271'12","reported_seq":199,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:23.728238+0000","last_change":"2024-09-11T08:21:47.298873+0000","last_active":"2024-09-11T08:22:23.728238+0000","last_peered":"2024-09-11T08:22:23.728238+0000","last_clean":"2024-09-11T08:22:23.728238+0000","last_became_active":"2024-09-11T08:19:29.144614+0000","last_became_peered":"2024-09-11T08:19:29.144614+0000","last_unstale":"2024-09-11T08:22:23.728238+0000","last_undegraded":"2024-09-11T08:22:23.728238+0000","last_fullsized":"2024-09-11T08:22:23.728238+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'12","last_scrub_stamp":"2024-09-11T08:21:47.298801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:21:47.298801+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:12.005882+0000","scrub_duration":0.0018774289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"66'1","reported_seq":321,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.640592+0000","last_change":"2024-09-11T08:22:04.543317+0000","last_active":"2024-09-11T08:22:20.640592+0000","last_peered":"2024-09-11T08:22:20.640592+0000","last_clean":"2024-09-11T08:22:20.640592+0000","last_became_active":"2024-09-11T08:07:36.072593+0000","last_became_peered":"2024-09-11T08:07:36.072593+0000","last_unstale":"2024-09-11T08:22:20.640592+0000","last_undegraded":"2024-09-11T08:22:20.640592+0000","last_fullsized":"2024-09-11T08:22:20.640592+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-11T08:22:04.543159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:22:04.543159+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:27.526410+0000","scrub_duration":0.002903625,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.d","version":"317'3696","reported_seq":3713,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-11T08:22:27.917033+0000","last_change":"2024-09-11T08:21:47.143484+0000","last_active":"2024-09-11T08:22:27.917033+0000","last_peered":"2024-09-11T08:22:27.917033+0000","last_clean":"2024-09-11T08:22:27.917033+0000","last_became_active":"2024-09-11T08:21:47.143172+0000","last_became_peered":"2024-09-11T08:21:47.143172+0000","last_unstale":"2024-09-11T08:22:27.917033+0000","last_undegraded":"2024-09-11T08:22:27.917033+0000","last_fullsized":"2024-09-11T08:22:27.917033+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":308,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_clean_scrub_stamp":"2024-09-11T08:21:46.117341+0000","objects_scrubbed":0,"log_size":3696,"log_dups_size":0,"ondisk_log_size":3696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:56.640436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242221056,"num_objects":3696,"num_object_clones":0,"num_object_copies":11088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3696,"num_write_kb":236544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"239'29","reported_seq":494,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.637610+0000","last_change":"2024-09-11T08:21:44.561967+0000","last_active":"2024-09-11T08:22:20.637610+0000","last_peered":"2024-09-11T08:22:20.637610+0000","last_clean":"2024-09-11T08:22:20.637610+0000","last_became_active":"2024-09-11T08:19:29.195995+0000","last_became_peered":"2024-09-11T08:19:29.195995+0000","last_unstale":"2024-09-11T08:22:20.637610+0000","last_undegraded":"2024-09-11T08:22:20.637610+0000","last_fullsized":"2024-09-11T08:22:20.637610+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'29","last_scrub_stamp":"2024-09-11T08:21:44.561906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:21:44.561906+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:02.503186+0000","scrub_duration":0.0024798810000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":370,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.651374+0000","last_change":"2024-09-11T08:21:47.825500+0000","last_active":"2024-09-11T08:22:20.651374+0000","last_peered":"2024-09-11T08:22:20.651374+0000","last_clean":"2024-09-11T08:22:20.651374+0000","last_became_active":"2024-09-11T08:19:29.141999+0000","last_became_peered":"2024-09-11T08:19:29.141999+0000","last_unstale":"2024-09-11T08:22:20.651374+0000","last_undegraded":"2024-09-11T08:22:20.651374+0000","last_fullsized":"2024-09-11T08:22:20.651374+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:47.825446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:21:47.825446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:17.004135+0000","scrub_duration":0.0015083430000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":362,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.641970+0000","last_change":"2024-09-11T08:20:51.918821+0000","last_active":"2024-09-11T08:22:20.641970+0000","last_peered":"2024-09-11T08:22:20.641970+0000","last_clean":"2024-09-11T08:22:20.641970+0000","last_became_active":"2024-09-11T08:19:29.166609+0000","last_became_peered":"2024-09-11T08:19:29.166609+0000","last_unstale":"2024-09-11T08:22:20.641970+0000","last_undegraded":"2024-09-11T08:22:20.641970+0000","last_fullsized":"2024-09-11T08:22:20.641970+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:51.918741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:20:51.918741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0090788139999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":331,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.648142+0000","last_change":"2024-09-11T08:20:56.715356+0000","last_active":"2024-09-11T08:22:20.648142+0000","last_peered":"2024-09-11T08:22:20.648142+0000","last_clean":"2024-09-11T08:22:20.648142+0000","last_became_active":"2024-09-11T08:07:36.631668+0000","last_became_peered":"2024-09-11T08:07:36.631668+0000","last_unstale":"2024-09-11T08:22:20.648142+0000","last_undegraded":"2024-09-11T08:22:20.648142+0000","last_fullsized":"2024-09-11T08:22:20.648142+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:56.715273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:58.517624+0000","last_clean_scrub_stamp":"2024-09-11T08:20:56.715273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021168839999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"317'3710","reported_seq":3727,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-11T08:22:27.943428+0000","last_change":"2024-09-11T08:21:47.140617+0000","last_active":"2024-09-11T08:22:27.943428+0000","last_peered":"2024-09-11T08:22:27.943428+0000","last_clean":"2024-09-11T08:22:27.943428+0000","last_became_active":"2024-09-11T08:21:47.140341+0000","last_became_peered":"2024-09-11T08:21:47.140341+0000","last_unstale":"2024-09-11T08:22:27.943428+0000","last_undegraded":"2024-09-11T08:22:27.943428+0000","last_fullsized":"2024-09-11T08:22:27.943428+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":308,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_clean_scrub_stamp":"2024-09-11T08:21:46.117341+0000","objects_scrubbed":0,"log_size":3710,"log_dups_size":0,"ondisk_log_size":3710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:55.206551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243138560,"num_objects":3710,"num_object_clones":0,"num_object_copies":11130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3710,"num_write_kb":237440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"296'52","reported_seq":82,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:23.728188+0000","last_change":"2024-09-11T08:20:53.645194+0000","last_active":"2024-09-11T08:22:23.728188+0000","last_peered":"2024-09-11T08:22:23.728188+0000","last_clean":"2024-09-11T08:22:23.728188+0000","last_became_active":"2024-09-11T08:20:52.086843+0000","last_became_peered":"2024-09-11T08:20:52.086843+0000","last_unstale":"2024-09-11T08:22:23.728188+0000","last_undegraded":"2024-09-11T08:22:23.728188+0000","last_fullsized":"2024-09-11T08:22:23.728188+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'51","last_scrub_stamp":"2024-09-11T08:20:53.645121+0000","last_deep_scrub":"190'39","last_deep_scrub_stamp":"2024-09-11T08:15:52.847968+0000","last_clean_scrub_stamp":"2024-09-11T08:20:53.645121+0000","objects_scrubbed":10,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034138189999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"59'1","reported_seq":336,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.636123+0000","last_change":"2024-09-11T08:21:46.049356+0000","last_active":"2024-09-11T08:22:20.636123+0000","last_peered":"2024-09-11T08:22:20.636123+0000","last_clean":"2024-09-11T08:22:20.636123+0000","last_became_active":"2024-09-11T08:07:30.047513+0000","last_became_peered":"2024-09-11T08:07:30.047513+0000","last_unstale":"2024-09-11T08:22:20.636123+0000","last_undegraded":"2024-09-11T08:22:20.636123+0000","last_fullsized":"2024-09-11T08:22:20.636123+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-11T08:21:46.049283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:21:46.049283+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:46.493529+0000","scrub_duration":0.0017569549999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":70,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.632662+0000","last_change":"2024-09-11T08:21:49.427128+0000","last_active":"2024-09-11T08:22:20.632662+0000","last_peered":"2024-09-11T08:22:20.632662+0000","last_clean":"2024-09-11T08:22:20.632662+0000","last_became_active":"2024-09-11T08:19:29.142111+0000","last_became_peered":"2024-09-11T08:19:29.142111+0000","last_unstale":"2024-09-11T08:22:20.632662+0000","last_undegraded":"2024-09-11T08:22:20.632662+0000","last_fullsized":"2024-09-11T08:22:20.632662+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-11T08:21:49.427071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:21:49.427071+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:16.477524+0000","scrub_duration":0.001771604,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":322,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.639578+0000","last_change":"2024-09-11T08:21:48.600637+0000","last_active":"2024-09-11T08:22:20.639578+0000","last_peered":"2024-09-11T08:22:20.639578+0000","last_clean":"2024-09-11T08:22:20.639578+0000","last_became_active":"2024-09-11T08:07:34.057154+0000","last_became_peered":"2024-09-11T08:07:34.057154+0000","last_unstale":"2024-09-11T08:22:20.639578+0000","last_undegraded":"2024-09-11T08:22:20.639578+0000","last_fullsized":"2024-09-11T08:22:20.639578+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:48.600546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:10:37.957096+0000","last_clean_scrub_stamp":"2024-09-11T08:21:48.600546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:55.974658+0000","scrub_duration":0.00136414,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":348,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.642915+0000","last_change":"2024-09-11T08:20:55.800959+0000","last_active":"2024-09-11T08:22:20.642915+0000","last_peered":"2024-09-11T08:22:20.642915+0000","last_clean":"2024-09-11T08:22:20.642915+0000","last_became_active":"2024-09-11T08:15:56.525099+0000","last_became_peered":"2024-09-11T08:15:56.525099+0000","last_unstale":"2024-09-11T08:22:20.642915+0000","last_undegraded":"2024-09-11T08:22:20.642915+0000","last_fullsized":"2024-09-11T08:22:20.642915+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:55.800856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:20:55.800856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.008329276,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"317'3695","reported_seq":3712,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-11T08:22:28.229831+0000","last_change":"2024-09-11T08:21:47.139031+0000","last_active":"2024-09-11T08:22:28.229831+0000","last_peered":"2024-09-11T08:22:28.229831+0000","last_clean":"2024-09-11T08:22:28.229831+0000","last_became_active":"2024-09-11T08:21:47.138720+0000","last_became_peered":"2024-09-11T08:21:47.138720+0000","last_unstale":"2024-09-11T08:22:28.229831+0000","last_undegraded":"2024-09-11T08:22:28.229831+0000","last_fullsized":"2024-09-11T08:22:28.229831+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":308,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_clean_scrub_stamp":"2024-09-11T08:21:46.117341+0000","objects_scrubbed":0,"log_size":3695,"log_dups_size":0,"ondisk_log_size":3695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:53.290403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242155520,"num_objects":3695,"num_object_clones":0,"num_object_copies":11085,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3695,"num_write_kb":236480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"239'17","reported_seq":425,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.651652+0000","last_change":"2024-09-11T08:21:53.919207+0000","last_active":"2024-09-11T08:22:20.651652+0000","last_peered":"2024-09-11T08:22:20.651652+0000","last_clean":"2024-09-11T08:22:20.651652+0000","last_became_active":"2024-09-11T08:19:29.161554+0000","last_became_peered":"2024-09-11T08:19:29.161554+0000","last_unstale":"2024-09-11T08:22:20.651652+0000","last_undegraded":"2024-09-11T08:22:20.651652+0000","last_fullsized":"2024-09-11T08:22:20.651652+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'17","last_scrub_stamp":"2024-09-11T08:21:53.919145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:21:53.919145+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:14.995825+0000","scrub_duration":0.0026285179999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":340,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.647948+0000","last_change":"2024-09-11T08:21:10.701859+0000","last_active":"2024-09-11T08:22:20.647948+0000","last_peered":"2024-09-11T08:22:20.647948+0000","last_clean":"2024-09-11T08:22:20.647948+0000","last_became_active":"2024-09-11T08:07:30.047014+0000","last_became_peered":"2024-09-11T08:07:30.047014+0000","last_unstale":"2024-09-11T08:22:20.647948+0000","last_undegraded":"2024-09-11T08:22:20.647948+0000","last_fullsized":"2024-09-11T08:22:20.647948+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:10.701767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:21:10.701767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0056318200000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"127'2","reported_seq":366,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.638980+0000","last_change":"2024-09-11T08:22:03.267356+0000","last_active":"2024-09-11T08:22:20.638980+0000","last_peered":"2024-09-11T08:22:20.638980+0000","last_clean":"2024-09-11T08:22:20.638980+0000","last_became_active":"2024-09-11T08:07:30.045009+0000","last_became_peered":"2024-09-11T08:07:30.045009+0000","last_unstale":"2024-09-11T08:22:20.638980+0000","last_undegraded":"2024-09-11T08:22:20.638980+0000","last_fullsized":"2024-09-11T08:22:20.638980+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'2","last_scrub_stamp":"2024-09-11T08:22:03.267269+0000","last_deep_scrub":"127'2","last_deep_scrub_stamp":"2024-09-11T08:15:01.938497+0000","last_clean_scrub_stamp":"2024-09-11T08:22:03.267269+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:04.866068+0000","scrub_duration":0.0040114060000000003,"objects_trimmed":0,"snaptrim_duration":0.001859533,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":385,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.638605+0000","last_change":"2024-09-11T08:22:00.200933+0000","last_active":"2024-09-11T08:22:20.638605+0000","last_peered":"2024-09-11T08:22:20.638605+0000","last_clean":"2024-09-11T08:22:20.638605+0000","last_became_active":"2024-09-11T08:19:29.145168+0000","last_became_peered":"2024-09-11T08:19:29.145168+0000","last_unstale":"2024-09-11T08:22:20.638605+0000","last_undegraded":"2024-09-11T08:22:20.638605+0000","last_fullsized":"2024-09-11T08:22:20.638605+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:22:00.200872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:22:00.200872+0000","last_clean_scrub_stamp":"2024-09-11T08:22:00.200872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:01.894647+0000","scrub_duration":0.0020630689999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":52,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.641495+0000","last_change":"2024-09-11T08:21:45.173035+0000","last_active":"2024-09-11T08:22:20.641495+0000","last_peered":"2024-09-11T08:22:20.641495+0000","last_clean":"2024-09-11T08:22:20.641495+0000","last_became_active":"2024-09-11T08:20:16.108674+0000","last_became_peered":"2024-09-11T08:20:16.108674+0000","last_unstale":"2024-09-11T08:22:20.641495+0000","last_undegraded":"2024-09-11T08:22:20.641495+0000","last_fullsized":"2024-09-11T08:22:20.641495+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:45.172938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:58.796217+0000","last_clean_scrub_stamp":"2024-09-11T08:21:45.172938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:11.469254+0000","scrub_duration":0.0013497769999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.8","version":"317'3726","reported_seq":3743,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-11T08:22:28.775348+0000","last_change":"2024-09-11T08:21:47.139819+0000","last_active":"2024-09-11T08:22:28.775348+0000","last_peered":"2024-09-11T08:22:28.775348+0000","last_clean":"2024-09-11T08:22:28.775348+0000","last_became_active":"2024-09-11T08:21:47.139475+0000","last_became_peered":"2024-09-11T08:21:47.139475+0000","last_unstale":"2024-09-11T08:22:28.775348+0000","last_undegraded":"2024-09-11T08:22:28.775348+0000","last_fullsized":"2024-09-11T08:22:28.775348+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":308,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_clean_scrub_stamp":"2024-09-11T08:21:46.117341+0000","objects_scrubbed":0,"log_size":3726,"log_dups_size":0,"ondisk_log_size":3726,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:00.885764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244187136,"num_objects":3726,"num_object_clones":0,"num_object_copies":11178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3726,"num_write_kb":238464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"287'30","reported_seq":366,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.638262+0000","last_change":"2024-09-11T08:20:24.667229+0000","last_active":"2024-09-11T08:22:20.638262+0000","last_peered":"2024-09-11T08:22:20.638262+0000","last_clean":"2024-09-11T08:22:20.638262+0000","last_became_active":"2024-09-11T08:07:32.055407+0000","last_became_peered":"2024-09-11T08:07:32.055407+0000","last_unstale":"2024-09-11T08:22:20.638262+0000","last_undegraded":"2024-09-11T08:22:20.638262+0000","last_fullsized":"2024-09-11T08:22:20.638262+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"287'30","last_scrub_stamp":"2024-09-11T08:20:24.667139+0000","last_deep_scrub":"287'30","last_deep_scrub_stamp":"2024-09-11T08:20:24.667139+0000","last_clean_scrub_stamp":"2024-09-11T08:20:24.667139+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.030013558999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"59'1","reported_seq":333,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.638140+0000","last_change":"2024-09-11T08:20:53.779883+0000","last_active":"2024-09-11T08:22:20.638140+0000","last_peered":"2024-09-11T08:22:20.638140+0000","last_clean":"2024-09-11T08:22:20.638140+0000","last_became_active":"2024-09-11T08:07:30.050504+0000","last_became_peered":"2024-09-11T08:07:30.050504+0000","last_unstale":"2024-09-11T08:22:20.638140+0000","last_undegraded":"2024-09-11T08:22:20.638140+0000","last_fullsized":"2024-09-11T08:22:20.638140+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-11T08:20:53.779281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:20:53.779281+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0097612610000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"224'6","reported_seq":1012,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-11T08:22:26.714737+0000","last_change":"2024-09-11T08:21:50.580561+0000","last_active":"2024-09-11T08:22:26.714737+0000","last_peered":"2024-09-11T08:22:26.714737+0000","last_clean":"2024-09-11T08:22:26.714737+0000","last_became_active":"2024-09-11T08:14:24.793048+0000","last_became_peered":"2024-09-11T08:14:24.793048+0000","last_unstale":"2024-09-11T08:22:26.714737+0000","last_undegraded":"2024-09-11T08:22:26.714737+0000","last_fullsized":"2024-09-11T08:22:26.714737+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'6","last_scrub_stamp":"2024-09-11T08:21:50.580504+0000","last_deep_scrub":"224'6","last_deep_scrub_stamp":"2024-09-11T08:16:55.778284+0000","last_clean_scrub_stamp":"2024-09-11T08:21:50.580504+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:07.461120+0000","scrub_duration":0.0015158470000000001,"objects_trimmed":0,"snaptrim_duration":0.000318821,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":489,"num_read_kb":484,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":66,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.637527+0000","last_change":"2024-09-11T08:20:54.655099+0000","last_active":"2024-09-11T08:22:20.637527+0000","last_peered":"2024-09-11T08:22:20.637527+0000","last_clean":"2024-09-11T08:22:20.637527+0000","last_became_active":"2024-09-11T08:19:29.154711+0000","last_became_peered":"2024-09-11T08:19:29.154711+0000","last_unstale":"2024-09-11T08:22:20.637527+0000","last_undegraded":"2024-09-11T08:22:20.637527+0000","last_fullsized":"2024-09-11T08:22:20.637527+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:54.655033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:20:54.655033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.017642073000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":364,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.651143+0000","last_change":"2024-09-11T08:21:55.844761+0000","last_active":"2024-09-11T08:22:20.651143+0000","last_peered":"2024-09-11T08:22:20.651143+0000","last_clean":"2024-09-11T08:22:20.651143+0000","last_became_active":"2024-09-11T08:19:29.141234+0000","last_became_peered":"2024-09-11T08:19:29.141234+0000","last_unstale":"2024-09-11T08:22:20.651143+0000","last_undegraded":"2024-09-11T08:22:20.651143+0000","last_fullsized":"2024-09-11T08:22:20.651143+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:55.844704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:21:55.844704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:56.481011+0000","scrub_duration":0.001596515,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"317'3742","reported_seq":3759,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-11T08:22:27.913955+0000","last_change":"2024-09-11T08:21:47.143520+0000","last_active":"2024-09-11T08:22:27.913955+0000","last_peered":"2024-09-11T08:22:27.913955+0000","last_clean":"2024-09-11T08:22:27.913955+0000","last_became_active":"2024-09-11T08:21:47.143210+0000","last_became_peered":"2024-09-11T08:21:47.143210+0000","last_unstale":"2024-09-11T08:22:27.913955+0000","last_undegraded":"2024-09-11T08:22:27.913955+0000","last_fullsized":"2024-09-11T08:22:27.913955+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":308,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_clean_scrub_stamp":"2024-09-11T08:21:46.117341+0000","objects_scrubbed":0,"log_size":3742,"log_dups_size":0,"ondisk_log_size":3742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:05.437537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":245235712,"num_objects":3742,"num_object_clones":0,"num_object_copies":11226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3742,"num_write_kb":239488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"239'33","reported_seq":408,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.641355+0000","last_change":"2024-09-11T08:22:01.316777+0000","last_active":"2024-09-11T08:22:20.641355+0000","last_peered":"2024-09-11T08:22:20.641355+0000","last_clean":"2024-09-11T08:22:20.641355+0000","last_became_active":"2024-09-11T08:14:24.789154+0000","last_became_peered":"2024-09-11T08:14:24.789154+0000","last_unstale":"2024-09-11T08:22:20.641355+0000","last_undegraded":"2024-09-11T08:22:20.641355+0000","last_fullsized":"2024-09-11T08:22:20.641355+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'33","last_scrub_stamp":"2024-09-11T08:22:01.316685+0000","last_deep_scrub":"84'21","last_deep_scrub_stamp":"2024-09-11T08:08:56.705306+0000","last_clean_scrub_stamp":"2024-09-11T08:22:01.316685+0000","objects_scrubbed":7,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:15.626047+0000","scrub_duration":0.0038599789999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":56,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.632526+0000","last_change":"2024-09-11T08:19:29.144089+0000","last_active":"2024-09-11T08:22:20.632526+0000","last_peered":"2024-09-11T08:22:20.632526+0000","last_clean":"2024-09-11T08:22:20.632526+0000","last_became_active":"2024-09-11T08:19:29.143505+0000","last_became_peered":"2024-09-11T08:19:29.143505+0000","last_unstale":"2024-09-11T08:22:20.632526+0000","last_undegraded":"2024-09-11T08:22:20.632526+0000","last_fullsized":"2024-09-11T08:22:20.632526+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:18:45.423258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:06.236119+0000","last_clean_scrub_stamp":"2024-09-11T08:18:45.423258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":155,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.633624+0000","last_change":"2024-09-11T08:21:48.553887+0000","last_active":"2024-09-11T08:22:20.633624+0000","last_peered":"2024-09-11T08:22:20.633624+0000","last_clean":"2024-09-11T08:22:20.633624+0000","last_became_active":"2024-09-11T08:15:56.523430+0000","last_became_peered":"2024-09-11T08:15:56.523430+0000","last_unstale":"2024-09-11T08:22:20.633624+0000","last_undegraded":"2024-09-11T08:22:20.633624+0000","last_fullsized":"2024-09-11T08:22:20.633624+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-11T08:21:48.553784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:05:21.231856+0000","last_clean_scrub_stamp":"2024-09-11T08:21:48.553784+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:57.895097+0000","scrub_duration":0.0021209100000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":159,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.635394+0000","last_change":"2024-09-11T08:20:25.775943+0000","last_active":"2024-09-11T08:22:20.635394+0000","last_peered":"2024-09-11T08:22:20.635394+0000","last_clean":"2024-09-11T08:22:20.635394+0000","last_became_active":"2024-09-11T08:19:29.147576+0000","last_became_peered":"2024-09-11T08:19:29.147576+0000","last_unstale":"2024-09-11T08:22:20.635394+0000","last_undegraded":"2024-09-11T08:22:20.635394+0000","last_fullsized":"2024-09-11T08:22:20.635394+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:25.775869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:20:25.775869+0000","last_clean_scrub_stamp":"2024-09-11T08:20:25.775869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004676014,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":66,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.637728+0000","last_change":"2024-09-11T08:20:29.154000+0000","last_active":"2024-09-11T08:22:20.637728+0000","last_peered":"2024-09-11T08:22:20.637728+0000","last_clean":"2024-09-11T08:22:20.637728+0000","last_became_active":"2024-09-11T08:19:29.152071+0000","last_became_peered":"2024-09-11T08:19:29.152071+0000","last_unstale":"2024-09-11T08:22:20.637728+0000","last_undegraded":"2024-09-11T08:22:20.637728+0000","last_fullsized":"2024-09-11T08:22:20.637728+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:29.153871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:01.777429+0000","last_clean_scrub_stamp":"2024-09-11T08:20:29.153871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015196540000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"317'3611","reported_seq":3628,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-11T08:22:27.940084+0000","last_change":"2024-09-11T08:21:47.143063+0000","last_active":"2024-09-11T08:22:27.940084+0000","last_peered":"2024-09-11T08:22:27.940084+0000","last_clean":"2024-09-11T08:22:27.940084+0000","last_became_active":"2024-09-11T08:21:47.142740+0000","last_became_peered":"2024-09-11T08:21:47.142740+0000","last_unstale":"2024-09-11T08:22:27.940084+0000","last_undegraded":"2024-09-11T08:22:27.940084+0000","last_fullsized":"2024-09-11T08:22:27.940084+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":308,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_clean_scrub_stamp":"2024-09-11T08:21:46.117341+0000","objects_scrubbed":0,"log_size":3611,"log_dups_size":0,"ondisk_log_size":3611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:49.481125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236650496,"num_objects":3611,"num_object_clones":0,"num_object_copies":10833,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3611,"num_write_kb":231104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"239'24","reported_seq":373,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.638304+0000","last_change":"2024-09-11T08:21:09.811217+0000","last_active":"2024-09-11T08:22:20.638304+0000","last_peered":"2024-09-11T08:22:20.638304+0000","last_clean":"2024-09-11T08:22:20.638304+0000","last_became_active":"2024-09-11T08:21:09.782223+0000","last_became_peered":"2024-09-11T08:21:09.782223+0000","last_unstale":"2024-09-11T08:22:20.638304+0000","last_undegraded":"2024-09-11T08:22:20.638304+0000","last_fullsized":"2024-09-11T08:22:20.638304+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'24","last_scrub_stamp":"2024-09-11T08:18:40.895293+0000","last_deep_scrub":"88'16","last_deep_scrub_stamp":"2024-09-11T08:09:39.635984+0000","last_clean_scrub_stamp":"2024-09-11T08:18:40.895293+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010336458999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":334,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.636200+0000","last_change":"2024-09-11T08:21:03.653241+0000","last_active":"2024-09-11T08:22:20.636200+0000","last_peered":"2024-09-11T08:22:20.636200+0000","last_clean":"2024-09-11T08:22:20.636200+0000","last_became_active":"2024-09-11T08:14:24.789888+0000","last_became_peered":"2024-09-11T08:14:24.789888+0000","last_unstale":"2024-09-11T08:22:20.636200+0000","last_undegraded":"2024-09-11T08:22:20.636200+0000","last_fullsized":"2024-09-11T08:22:20.636200+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:03.653157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:36.417911+0000","last_clean_scrub_stamp":"2024-09-11T08:21:03.653157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039236879999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":66,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.637611+0000","last_change":"2024-09-11T08:21:50.380816+0000","last_active":"2024-09-11T08:22:20.637611+0000","last_peered":"2024-09-11T08:22:20.637611+0000","last_clean":"2024-09-11T08:22:20.637611+0000","last_became_active":"2024-09-11T08:19:29.153976+0000","last_became_peered":"2024-09-11T08:19:29.153976+0000","last_unstale":"2024-09-11T08:22:20.637611+0000","last_undegraded":"2024-09-11T08:22:20.637611+0000","last_fullsized":"2024-09-11T08:22:20.637611+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:50.380756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:01.558340+0000","last_clean_scrub_stamp":"2024-09-11T08:21:50.380756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:10.607097+0000","scrub_duration":0.001327805,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":144,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.641740+0000","last_change":"2024-09-11T08:21:43.326785+0000","last_active":"2024-09-11T08:22:20.641740+0000","last_peered":"2024-09-11T08:22:20.641740+0000","last_clean":"2024-09-11T08:22:20.641740+0000","last_became_active":"2024-09-11T08:19:29.144592+0000","last_became_peered":"2024-09-11T08:19:29.144592+0000","last_unstale":"2024-09-11T08:22:20.641740+0000","last_undegraded":"2024-09-11T08:22:20.641740+0000","last_fullsized":"2024-09-11T08:22:20.641740+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:43.326717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:21:43.326717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:43.654572+0000","scrub_duration":0.0016517909999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3","version":"317'3725","reported_seq":3742,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-11T08:22:28.204911+0000","last_change":"2024-09-11T08:21:47.139861+0000","last_active":"2024-09-11T08:22:28.204911+0000","last_peered":"2024-09-11T08:22:28.204911+0000","last_clean":"2024-09-11T08:22:28.204911+0000","last_became_active":"2024-09-11T08:21:47.139581+0000","last_became_peered":"2024-09-11T08:21:47.139581+0000","last_unstale":"2024-09-11T08:22:28.204911+0000","last_undegraded":"2024-09-11T08:22:28.204911+0000","last_fullsized":"2024-09-11T08:22:28.204911+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":308,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_clean_scrub_stamp":"2024-09-11T08:21:46.117341+0000","objects_scrubbed":0,"log_size":3725,"log_dups_size":0,"ondisk_log_size":3725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:03.247374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244121600,"num_objects":3725,"num_object_clones":0,"num_object_copies":11175,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3725,"num_write_kb":238400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"239'39","reported_seq":438,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.637525+0000","last_change":"2024-09-11T08:21:51.079782+0000","last_active":"2024-09-11T08:22:20.637525+0000","last_peered":"2024-09-11T08:22:20.637525+0000","last_clean":"2024-09-11T08:22:20.637525+0000","last_became_active":"2024-09-11T08:19:09.604525+0000","last_became_peered":"2024-09-11T08:19:09.604525+0000","last_unstale":"2024-09-11T08:22:20.637525+0000","last_undegraded":"2024-09-11T08:22:20.637525+0000","last_fullsized":"2024-09-11T08:22:20.637525+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'39","last_scrub_stamp":"2024-09-11T08:21:51.079716+0000","last_deep_scrub":"77'25","last_deep_scrub_stamp":"2024-09-11T08:08:32.458210+0000","last_clean_scrub_stamp":"2024-09-11T08:21:51.079716+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:59.763626+0000","scrub_duration":0.0024068399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":352,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.642196+0000","last_change":"2024-09-11T08:21:55.322957+0000","last_active":"2024-09-11T08:22:20.642196+0000","last_peered":"2024-09-11T08:22:20.642196+0000","last_clean":"2024-09-11T08:22:20.642196+0000","last_became_active":"2024-09-11T08:07:30.042555+0000","last_became_peered":"2024-09-11T08:07:30.042555+0000","last_unstale":"2024-09-11T08:22:20.642196+0000","last_undegraded":"2024-09-11T08:22:20.642196+0000","last_fullsized":"2024-09-11T08:22:20.642196+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:55.322891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:21:55.322891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:18.917383+0000","scrub_duration":0.00164072,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":333,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.651206+0000","last_change":"2024-09-11T08:21:09.706729+0000","last_active":"2024-09-11T08:22:20.651206+0000","last_peered":"2024-09-11T08:22:20.651206+0000","last_clean":"2024-09-11T08:22:20.651206+0000","last_became_active":"2024-09-11T08:07:34.054961+0000","last_became_peered":"2024-09-11T08:07:34.054961+0000","last_unstale":"2024-09-11T08:22:20.651206+0000","last_undegraded":"2024-09-11T08:22:20.651206+0000","last_fullsized":"2024-09-11T08:22:20.651206+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:09.706678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:21:09.706678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038244720000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":350,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.640345+0000","last_change":"2024-09-11T08:21:02.823083+0000","last_active":"2024-09-11T08:22:20.640345+0000","last_peered":"2024-09-11T08:22:20.640345+0000","last_clean":"2024-09-11T08:22:20.640345+0000","last_became_active":"2024-09-11T08:07:36.071194+0000","last_became_peered":"2024-09-11T08:07:36.071194+0000","last_unstale":"2024-09-11T08:22:20.640345+0000","last_undegraded":"2024-09-11T08:22:20.640345+0000","last_fullsized":"2024-09-11T08:22:20.640345+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:02.822996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:07.930137+0000","last_clean_scrub_stamp":"2024-09-11T08:21:02.822996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005271441,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"317'3815","reported_seq":3832,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-11T08:22:28.654681+0000","last_change":"2024-09-11T08:21:47.139153+0000","last_active":"2024-09-11T08:22:28.654681+0000","last_peered":"2024-09-11T08:22:28.654681+0000","last_clean":"2024-09-11T08:22:28.654681+0000","last_became_active":"2024-09-11T08:21:47.138901+0000","last_became_peered":"2024-09-11T08:21:47.138901+0000","last_unstale":"2024-09-11T08:22:28.654681+0000","last_undegraded":"2024-09-11T08:22:28.654681+0000","last_fullsized":"2024-09-11T08:22:28.654681+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":308,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_clean_scrub_stamp":"2024-09-11T08:21:46.117341+0000","objects_scrubbed":0,"log_size":3815,"log_dups_size":0,"ondisk_log_size":3815,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:15.143012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250019840,"num_objects":3815,"num_object_clones":0,"num_object_copies":11445,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3815,"num_write_kb":244160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"239'25","reported_seq":353,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.638504+0000","last_change":"2024-09-11T08:20:16.527717+0000","last_active":"2024-09-11T08:22:20.638504+0000","last_peered":"2024-09-11T08:22:20.638504+0000","last_clean":"2024-09-11T08:22:20.638504+0000","last_became_active":"2024-09-11T08:07:32.043314+0000","last_became_peered":"2024-09-11T08:07:32.043314+0000","last_unstale":"2024-09-11T08:22:20.638504+0000","last_undegraded":"2024-09-11T08:22:20.638504+0000","last_fullsized":"2024-09-11T08:22:20.638504+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'25","last_scrub_stamp":"2024-09-11T08:20:16.527630+0000","last_deep_scrub":"224'15","last_deep_scrub_stamp":"2024-09-11T08:17:01.126478+0000","last_clean_scrub_stamp":"2024-09-11T08:20:16.527630+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0059554439999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":111,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.638939+0000","last_change":"2024-09-11T08:20:10.609260+0000","last_active":"2024-09-11T08:22:20.638939+0000","last_peered":"2024-09-11T08:22:20.638939+0000","last_clean":"2024-09-11T08:22:20.638939+0000","last_became_active":"2024-09-11T08:17:37.557278+0000","last_became_peered":"2024-09-11T08:17:37.557278+0000","last_unstale":"2024-09-11T08:22:20.638939+0000","last_undegraded":"2024-09-11T08:22:20.638939+0000","last_fullsized":"2024-09-11T08:22:20.638939+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:10.609170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:45.979847+0000","last_clean_scrub_stamp":"2024-09-11T08:20:10.609170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.017294936,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"271'14","reported_seq":581,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:23.728342+0000","last_change":"2024-09-11T08:21:59.247799+0000","last_active":"2024-09-11T08:22:23.728342+0000","last_peered":"2024-09-11T08:22:23.728342+0000","last_clean":"2024-09-11T08:22:23.728342+0000","last_became_active":"2024-09-11T08:19:29.153997+0000","last_became_peered":"2024-09-11T08:19:29.153997+0000","last_unstale":"2024-09-11T08:22:23.728342+0000","last_undegraded":"2024-09-11T08:22:23.728342+0000","last_fullsized":"2024-09-11T08:22:23.728342+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'14","last_scrub_stamp":"2024-09-11T08:21:59.247702+0000","last_deep_scrub":"171'12","last_deep_scrub_stamp":"2024-09-11T08:14:56.881299+0000","last_clean_scrub_stamp":"2024-09-11T08:21:59.247702+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:26.503385+0000","scrub_duration":0.0021085589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":311,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.637977+0000","last_change":"2024-09-11T08:20:50.427308+0000","last_active":"2024-09-11T08:22:20.637977+0000","last_peered":"2024-09-11T08:22:20.637977+0000","last_clean":"2024-09-11T08:22:20.637977+0000","last_became_active":"2024-09-11T08:07:36.074684+0000","last_became_peered":"2024-09-11T08:07:36.074684+0000","last_unstale":"2024-09-11T08:22:20.637977+0000","last_undegraded":"2024-09-11T08:22:20.637977+0000","last_fullsized":"2024-09-11T08:22:20.637977+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:50.427224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:14.613532+0000","last_clean_scrub_stamp":"2024-09-11T08:20:50.427224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028770409999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.5","version":"317'3656","reported_seq":3673,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-11T08:22:28.175000+0000","last_change":"2024-09-11T08:21:47.140488+0000","last_active":"2024-09-11T08:22:28.175000+0000","last_peered":"2024-09-11T08:22:28.175000+0000","last_clean":"2024-09-11T08:22:28.175000+0000","last_became_active":"2024-09-11T08:21:47.140224+0000","last_became_peered":"2024-09-11T08:21:47.140224+0000","last_unstale":"2024-09-11T08:22:28.175000+0000","last_undegraded":"2024-09-11T08:22:28.175000+0000","last_fullsized":"2024-09-11T08:22:28.175000+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":308,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_clean_scrub_stamp":"2024-09-11T08:21:46.117341+0000","objects_scrubbed":0,"log_size":3656,"log_dups_size":0,"ondisk_log_size":3656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:05.175594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239599616,"num_objects":3656,"num_object_clones":0,"num_object_copies":10968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3656,"num_write_kb":233984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"239'15","reported_seq":349,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.637531+0000","last_change":"2024-09-11T08:21:10.722246+0000","last_active":"2024-09-11T08:22:20.637531+0000","last_peered":"2024-09-11T08:22:20.637531+0000","last_clean":"2024-09-11T08:22:20.637531+0000","last_became_active":"2024-09-11T08:07:32.054885+0000","last_became_peered":"2024-09-11T08:07:32.054885+0000","last_unstale":"2024-09-11T08:22:20.637531+0000","last_undegraded":"2024-09-11T08:22:20.637531+0000","last_fullsized":"2024-09-11T08:22:20.637531+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'15","last_scrub_stamp":"2024-09-11T08:21:10.722062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:21:10.722062+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010706927,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":66,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.634470+0000","last_change":"2024-09-11T08:21:00.730548+0000","last_active":"2024-09-11T08:22:20.634470+0000","last_peered":"2024-09-11T08:22:20.634470+0000","last_clean":"2024-09-11T08:22:20.634470+0000","last_became_active":"2024-09-11T08:19:29.143174+0000","last_became_peered":"2024-09-11T08:19:29.143174+0000","last_unstale":"2024-09-11T08:22:20.634470+0000","last_undegraded":"2024-09-11T08:22:20.634470+0000","last_fullsized":"2024-09-11T08:22:20.634470+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:00.730460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:21:00.730460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0102811,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":352,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.639730+0000","last_change":"2024-09-11T08:21:04.894391+0000","last_active":"2024-09-11T08:22:20.639730+0000","last_peered":"2024-09-11T08:22:20.639730+0000","last_clean":"2024-09-11T08:22:20.639730+0000","last_became_active":"2024-09-11T08:07:34.056606+0000","last_became_peered":"2024-09-11T08:07:34.056606+0000","last_unstale":"2024-09-11T08:22:20.639730+0000","last_undegraded":"2024-09-11T08:22:20.639730+0000","last_fullsized":"2024-09-11T08:22:20.639730+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:04.894259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:21:04.894259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:27.176768+0000","scrub_duration":0.013493084000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":347,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.638037+0000","last_change":"2024-09-11T08:21:45.541234+0000","last_active":"2024-09-11T08:22:20.638037+0000","last_peered":"2024-09-11T08:22:20.638037+0000","last_clean":"2024-09-11T08:22:20.638037+0000","last_became_active":"2024-09-11T08:15:56.518884+0000","last_became_peered":"2024-09-11T08:15:56.518884+0000","last_unstale":"2024-09-11T08:22:20.638037+0000","last_undegraded":"2024-09-11T08:22:20.638037+0000","last_fullsized":"2024-09-11T08:22:20.638037+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:45.541171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:18:32.278257+0000","last_clean_scrub_stamp":"2024-09-11T08:21:45.541171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:48.020855+0000","scrub_duration":0.0013210780000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"317'3763","reported_seq":3780,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-11T08:22:28.775557+0000","last_change":"2024-09-11T08:21:47.143324+0000","last_active":"2024-09-11T08:22:28.775557+0000","last_peered":"2024-09-11T08:22:28.775557+0000","last_clean":"2024-09-11T08:22:28.775557+0000","last_became_active":"2024-09-11T08:21:47.143034+0000","last_became_peered":"2024-09-11T08:21:47.143034+0000","last_unstale":"2024-09-11T08:22:28.775557+0000","last_undegraded":"2024-09-11T08:22:28.775557+0000","last_fullsized":"2024-09-11T08:22:28.775557+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":308,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_clean_scrub_stamp":"2024-09-11T08:21:46.117341+0000","objects_scrubbed":0,"log_size":3763,"log_dups_size":0,"ondisk_log_size":3763,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:55.046330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":246611968,"num_objects":3763,"num_object_clones":0,"num_object_copies":11289,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3763,"num_write_kb":240832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"239'35","reported_seq":476,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.642262+0000","last_change":"2024-09-11T08:21:44.294440+0000","last_active":"2024-09-11T08:22:20.642262+0000","last_peered":"2024-09-11T08:22:20.642262+0000","last_clean":"2024-09-11T08:22:20.642262+0000","last_became_active":"2024-09-11T08:15:56.526961+0000","last_became_peered":"2024-09-11T08:15:56.526961+0000","last_unstale":"2024-09-11T08:22:20.642262+0000","last_undegraded":"2024-09-11T08:22:20.642262+0000","last_fullsized":"2024-09-11T08:22:20.642262+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'35","last_scrub_stamp":"2024-09-11T08:21:44.294374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:21:44.294374+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:59.282750+0000","scrub_duration":0.0025259789999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":159,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.633787+0000","last_change":"2024-09-11T08:20:09.859624+0000","last_active":"2024-09-11T08:22:20.633787+0000","last_peered":"2024-09-11T08:22:20.633787+0000","last_clean":"2024-09-11T08:22:20.633787+0000","last_became_active":"2024-09-11T08:19:29.152114+0000","last_became_peered":"2024-09-11T08:19:29.152114+0000","last_unstale":"2024-09-11T08:22:20.633787+0000","last_undegraded":"2024-09-11T08:22:20.633787+0000","last_fullsized":"2024-09-11T08:22:20.633787+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:09.859549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:44.385372+0000","last_clean_scrub_stamp":"2024-09-11T08:20:09.859549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0055679850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":365,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.638864+0000","last_change":"2024-09-11T08:20:53.860652+0000","last_active":"2024-09-11T08:22:20.638864+0000","last_peered":"2024-09-11T08:22:20.638864+0000","last_clean":"2024-09-11T08:22:20.638864+0000","last_became_active":"2024-09-11T08:19:29.178705+0000","last_became_peered":"2024-09-11T08:19:29.178705+0000","last_unstale":"2024-09-11T08:22:20.638864+0000","last_undegraded":"2024-09-11T08:22:20.638864+0000","last_fullsized":"2024-09-11T08:22:20.638864+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:53.860586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:20:53.860586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.030572999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":139,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.633700+0000","last_change":"2024-09-11T08:20:21.780282+0000","last_active":"2024-09-11T08:22:20.633700+0000","last_peered":"2024-09-11T08:22:20.633700+0000","last_clean":"2024-09-11T08:22:20.633700+0000","last_became_active":"2024-09-11T08:15:56.518169+0000","last_became_peered":"2024-09-11T08:15:56.518169+0000","last_unstale":"2024-09-11T08:22:20.633700+0000","last_undegraded":"2024-09-11T08:22:20.633700+0000","last_fullsized":"2024-09-11T08:22:20.633700+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:21.780224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:20:21.780224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0071716779999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"317'3709","reported_seq":3726,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-11T08:22:27.918109+0000","last_change":"2024-09-11T08:21:47.141356+0000","last_active":"2024-09-11T08:22:27.918109+0000","last_peered":"2024-09-11T08:22:27.918109+0000","last_clean":"2024-09-11T08:22:27.918109+0000","last_became_active":"2024-09-11T08:21:47.141093+0000","last_became_peered":"2024-09-11T08:21:47.141093+0000","last_unstale":"2024-09-11T08:22:27.918109+0000","last_undegraded":"2024-09-11T08:22:27.918109+0000","last_fullsized":"2024-09-11T08:22:27.918109+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":308,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_clean_scrub_stamp":"2024-09-11T08:21:46.117341+0000","objects_scrubbed":0,"log_size":3709,"log_dups_size":0,"ondisk_log_size":3709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:01.240099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243073024,"num_objects":3709,"num_object_clones":0,"num_object_copies":11127,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3709,"num_write_kb":237376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"239'23","reported_seq":362,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.635928+0000","last_change":"2024-09-11T08:22:04.128665+0000","last_active":"2024-09-11T08:22:20.635928+0000","last_peered":"2024-09-11T08:22:20.635928+0000","last_clean":"2024-09-11T08:22:20.635928+0000","last_became_active":"2024-09-11T08:07:32.048164+0000","last_became_peered":"2024-09-11T08:07:32.048164+0000","last_unstale":"2024-09-11T08:22:20.635928+0000","last_undegraded":"2024-09-11T08:22:20.635928+0000","last_fullsized":"2024-09-11T08:22:20.635928+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'23","last_scrub_stamp":"2024-09-11T08:22:04.128578+0000","last_deep_scrub":"145'17","last_deep_scrub_stamp":"2024-09-11T08:13:08.160848+0000","last_clean_scrub_stamp":"2024-09-11T08:22:04.128578+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:11.066610+0000","scrub_duration":0.0059359520000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":149,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.634833+0000","last_change":"2024-09-11T08:21:06.262697+0000","last_active":"2024-09-11T08:22:20.634833+0000","last_peered":"2024-09-11T08:22:20.634833+0000","last_clean":"2024-09-11T08:22:20.634833+0000","last_became_active":"2024-09-11T08:15:56.525845+0000","last_became_peered":"2024-09-11T08:15:56.525845+0000","last_unstale":"2024-09-11T08:22:20.634833+0000","last_undegraded":"2024-09-11T08:22:20.634833+0000","last_fullsized":"2024-09-11T08:22:20.634833+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:06.262572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:21:06.262572+0000","last_clean_scrub_stamp":"2024-09-11T08:21:06.262572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.007163272,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":332,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.639778+0000","last_change":"2024-09-11T08:20:59.876602+0000","last_active":"2024-09-11T08:22:20.639778+0000","last_peered":"2024-09-11T08:22:20.639778+0000","last_clean":"2024-09-11T08:22:20.639778+0000","last_became_active":"2024-09-11T08:07:34.057479+0000","last_became_peered":"2024-09-11T08:07:34.057479+0000","last_unstale":"2024-09-11T08:22:20.639778+0000","last_undegraded":"2024-09-11T08:22:20.639778+0000","last_fullsized":"2024-09-11T08:22:20.639778+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:59.876505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:20:59.876505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0064578550000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":358,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.641620+0000","last_change":"2024-09-11T08:21:43.750897+0000","last_active":"2024-09-11T08:22:20.641620+0000","last_peered":"2024-09-11T08:22:20.641620+0000","last_clean":"2024-09-11T08:22:20.641620+0000","last_became_active":"2024-09-11T08:15:56.522624+0000","last_became_peered":"2024-09-11T08:15:56.522624+0000","last_unstale":"2024-09-11T08:22:20.641620+0000","last_undegraded":"2024-09-11T08:22:20.641620+0000","last_fullsized":"2024-09-11T08:22:20.641620+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:43.750844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:21:43.750844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:01.039674+0000","scrub_duration":0.0015776570000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"317'3675","reported_seq":3692,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-11T08:22:28.210301+0000","last_change":"2024-09-11T08:21:47.141892+0000","last_active":"2024-09-11T08:22:28.210301+0000","last_peered":"2024-09-11T08:22:28.210301+0000","last_clean":"2024-09-11T08:22:28.210301+0000","last_became_active":"2024-09-11T08:21:47.141518+0000","last_became_peered":"2024-09-11T08:21:47.141518+0000","last_unstale":"2024-09-11T08:22:28.210301+0000","last_undegraded":"2024-09-11T08:22:28.210301+0000","last_fullsized":"2024-09-11T08:22:28.210301+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":308,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:21:46.117341+0000","last_clean_scrub_stamp":"2024-09-11T08:21:46.117341+0000","objects_scrubbed":0,"log_size":3675,"log_dups_size":0,"ondisk_log_size":3675,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:57.785263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240844800,"num_objects":3675,"num_object_clones":0,"num_object_copies":11025,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3675,"num_write_kb":235200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"239'27","reported_seq":82,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.632231+0000","last_change":"2024-09-11T08:20:08.785799+0000","last_active":"2024-09-11T08:22:20.632231+0000","last_peered":"2024-09-11T08:22:20.632231+0000","last_clean":"2024-09-11T08:22:20.632231+0000","last_became_active":"2024-09-11T08:19:29.147000+0000","last_became_peered":"2024-09-11T08:19:29.147000+0000","last_unstale":"2024-09-11T08:22:20.632231+0000","last_undegraded":"2024-09-11T08:22:20.632231+0000","last_fullsized":"2024-09-11T08:22:20.632231+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'27","last_scrub_stamp":"2024-09-11T08:20:08.785703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:20:08.785703+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005828789,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":155,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.634358+0000","last_change":"2024-09-11T08:21:46.555670+0000","last_active":"2024-09-11T08:22:20.634358+0000","last_peered":"2024-09-11T08:22:20.634358+0000","last_clean":"2024-09-11T08:22:20.634358+0000","last_became_active":"2024-09-11T08:15:56.515328+0000","last_became_peered":"2024-09-11T08:15:56.515328+0000","last_unstale":"2024-09-11T08:22:20.634358+0000","last_undegraded":"2024-09-11T08:22:20.634358+0000","last_fullsized":"2024-09-11T08:22:20.634358+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-11T08:21:46.555601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:21:46.555601+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:04.642167+0000","scrub_duration":0.0017840829999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"209'11","reported_seq":221,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:23.729586+0000","last_change":"2024-09-11T08:20:51.158341+0000","last_active":"2024-09-11T08:22:23.729586+0000","last_peered":"2024-09-11T08:22:23.729586+0000","last_clean":"2024-09-11T08:22:23.729586+0000","last_became_active":"2024-09-11T08:15:56.524513+0000","last_became_peered":"2024-09-11T08:15:56.524513+0000","last_unstale":"2024-09-11T08:22:23.729586+0000","last_undegraded":"2024-09-11T08:22:23.729586+0000","last_fullsized":"2024-09-11T08:22:23.729586+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"209'11","last_scrub_stamp":"2024-09-11T08:20:51.158199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:20:51.158199+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015653383,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":66,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.632743+0000","last_change":"2024-09-11T08:21:59.448626+0000","last_active":"2024-09-11T08:22:20.632743+0000","last_peered":"2024-09-11T08:22:20.632743+0000","last_clean":"2024-09-11T08:22:20.632743+0000","last_became_active":"2024-09-11T08:19:29.163872+0000","last_became_peered":"2024-09-11T08:19:29.163872+0000","last_unstale":"2024-09-11T08:22:20.632743+0000","last_undegraded":"2024-09-11T08:22:20.632743+0000","last_fullsized":"2024-09-11T08:22:20.632743+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:59.448573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:03.484755+0000","last_clean_scrub_stamp":"2024-09-11T08:21:59.448573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:25.059461+0000","scrub_duration":0.0028812519999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":376,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.640963+0000","last_change":"2024-09-11T08:21:58.281147+0000","last_active":"2024-09-11T08:22:20.640963+0000","last_peered":"2024-09-11T08:22:20.640963+0000","last_clean":"2024-09-11T08:22:20.640963+0000","last_became_active":"2024-09-11T08:19:29.193317+0000","last_became_peered":"2024-09-11T08:19:29.193317+0000","last_unstale":"2024-09-11T08:22:20.640963+0000","last_undegraded":"2024-09-11T08:22:20.640963+0000","last_fullsized":"2024-09-11T08:22:20.640963+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:58.281082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:21:58.281082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:12.638469+0000","scrub_duration":0.001499371,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"239'15","reported_seq":389,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.644523+0000","last_change":"2024-09-11T08:20:54.764391+0000","last_active":"2024-09-11T08:22:20.644523+0000","last_peered":"2024-09-11T08:22:20.644523+0000","last_clean":"2024-09-11T08:22:20.644523+0000","last_became_active":"2024-09-11T08:15:56.526325+0000","last_became_peered":"2024-09-11T08:15:56.526325+0000","last_unstale":"2024-09-11T08:22:20.644523+0000","last_undegraded":"2024-09-11T08:22:20.644523+0000","last_fullsized":"2024-09-11T08:22:20.644523+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'15","last_scrub_stamp":"2024-09-11T08:20:54.764293+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-09-11T08:09:03.418457+0000","last_clean_scrub_stamp":"2024-09-11T08:20:54.764293+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039192439999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":66,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.633566+0000","last_change":"2024-09-11T08:21:54.344679+0000","last_active":"2024-09-11T08:22:20.633566+0000","last_peered":"2024-09-11T08:22:20.633566+0000","last_clean":"2024-09-11T08:22:20.633566+0000","last_became_active":"2024-09-11T08:19:29.143925+0000","last_became_peered":"2024-09-11T08:19:29.143925+0000","last_unstale":"2024-09-11T08:22:20.633566+0000","last_undegraded":"2024-09-11T08:22:20.633566+0000","last_fullsized":"2024-09-11T08:22:20.633566+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:54.344520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:53.548967+0000","last_clean_scrub_stamp":"2024-09-11T08:21:54.344520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:03.623437+0000","scrub_duration":0.001658814,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":274,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.633916+0000","last_change":"2024-09-11T08:21:03.199721+0000","last_active":"2024-09-11T08:22:20.633916+0000","last_peered":"2024-09-11T08:22:20.633916+0000","last_clean":"2024-09-11T08:22:20.633916+0000","last_became_active":"2024-09-11T08:14:24.790678+0000","last_became_peered":"2024-09-11T08:14:24.790678+0000","last_unstale":"2024-09-11T08:22:20.633916+0000","last_undegraded":"2024-09-11T08:22:20.633916+0000","last_fullsized":"2024-09-11T08:22:20.633916+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:03.199669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:21:03.199669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:27.957305+0000","scrub_duration":0.0011985959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"239'19","reported_seq":390,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.638345+0000","last_change":"2024-09-11T08:20:14.603209+0000","last_active":"2024-09-11T08:22:20.638345+0000","last_peered":"2024-09-11T08:22:20.638345+0000","last_clean":"2024-09-11T08:22:20.638345+0000","last_became_active":"2024-09-11T08:15:56.514793+0000","last_became_peered":"2024-09-11T08:15:56.514793+0000","last_unstale":"2024-09-11T08:22:20.638345+0000","last_undegraded":"2024-09-11T08:22:20.638345+0000","last_fullsized":"2024-09-11T08:22:20.638345+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'19","last_scrub_stamp":"2024-09-11T08:20:14.603118+0000","last_deep_scrub":"84'13","last_deep_scrub_stamp":"2024-09-11T08:08:57.835634+0000","last_clean_scrub_stamp":"2024-09-11T08:20:14.603118+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042720240000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":297,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.636281+0000","last_change":"2024-09-11T08:20:20.819257+0000","last_active":"2024-09-11T08:22:20.636281+0000","last_peered":"2024-09-11T08:22:20.636281+0000","last_clean":"2024-09-11T08:22:20.636281+0000","last_became_active":"2024-09-11T08:14:24.786021+0000","last_became_peered":"2024-09-11T08:14:24.786021+0000","last_unstale":"2024-09-11T08:22:20.636281+0000","last_undegraded":"2024-09-11T08:22:20.636281+0000","last_fullsized":"2024-09-11T08:22:20.636281+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:20.819154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:30.471620+0000","last_clean_scrub_stamp":"2024-09-11T08:20:20.819154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018806490000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":350,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.644395+0000","last_change":"2024-09-11T08:20:22.889011+0000","last_active":"2024-09-11T08:22:20.644395+0000","last_peered":"2024-09-11T08:22:20.644395+0000","last_clean":"2024-09-11T08:22:20.644395+0000","last_became_active":"2024-09-11T08:15:56.525717+0000","last_became_peered":"2024-09-11T08:15:56.525717+0000","last_unstale":"2024-09-11T08:22:20.644395+0000","last_undegraded":"2024-09-11T08:22:20.644395+0000","last_fullsized":"2024-09-11T08:22:20.644395+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:22.888936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:04.370734+0000","last_clean_scrub_stamp":"2024-09-11T08:20:22.888936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.01485916,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":360,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.639230+0000","last_change":"2024-09-11T08:21:43.156155+0000","last_active":"2024-09-11T08:22:20.639230+0000","last_peered":"2024-09-11T08:22:20.639230+0000","last_clean":"2024-09-11T08:22:20.639230+0000","last_became_active":"2024-09-11T08:15:56.525120+0000","last_became_peered":"2024-09-11T08:15:56.525120+0000","last_unstale":"2024-09-11T08:22:20.639230+0000","last_undegraded":"2024-09-11T08:22:20.639230+0000","last_fullsized":"2024-09-11T08:22:20.639230+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:43.156101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:54.184372+0000","last_clean_scrub_stamp":"2024-09-11T08:21:43.156101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:07.325849+0000","scrub_duration":0.0017268610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"313'19","reported_seq":80,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.632907+0000","last_change":"2024-09-11T08:22:03.458497+0000","last_active":"2024-09-11T08:22:20.632907+0000","last_peered":"2024-09-11T08:22:20.632907+0000","last_clean":"2024-09-11T08:22:20.632907+0000","last_became_active":"2024-09-11T08:19:29.153740+0000","last_became_peered":"2024-09-11T08:19:29.153740+0000","last_unstale":"2024-09-11T08:22:20.632907+0000","last_undegraded":"2024-09-11T08:22:20.632907+0000","last_fullsized":"2024-09-11T08:22:20.632907+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"313'19","last_scrub_stamp":"2024-09-11T08:22:03.458402+0000","last_deep_scrub":"313'19","last_deep_scrub_stamp":"2024-09-11T08:22:03.458402+0000","last_clean_scrub_stamp":"2024-09-11T08:22:03.458402+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:06.902001+0000","scrub_duration":0.0061927930000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":353,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.640389+0000","last_change":"2024-09-11T08:21:05.777148+0000","last_active":"2024-09-11T08:22:20.640389+0000","last_peered":"2024-09-11T08:22:20.640389+0000","last_clean":"2024-09-11T08:22:20.640389+0000","last_became_active":"2024-09-11T08:15:56.513278+0000","last_became_peered":"2024-09-11T08:15:56.513278+0000","last_unstale":"2024-09-11T08:22:20.640389+0000","last_undegraded":"2024-09-11T08:22:20.640389+0000","last_fullsized":"2024-09-11T08:22:20.640389+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:05.777069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:52.661591+0000","last_clean_scrub_stamp":"2024-09-11T08:21:05.777069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:33.267208+0000","scrub_duration":0.00237024,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"239'13","reported_seq":528,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:23.728238+0000","last_change":"2024-09-11T08:21:52.595127+0000","last_active":"2024-09-11T08:22:23.728238+0000","last_peered":"2024-09-11T08:22:23.728238+0000","last_clean":"2024-09-11T08:22:23.728238+0000","last_became_active":"2024-09-11T08:17:37.558224+0000","last_became_peered":"2024-09-11T08:17:37.558224+0000","last_unstale":"2024-09-11T08:22:23.728238+0000","last_undegraded":"2024-09-11T08:22:23.728238+0000","last_fullsized":"2024-09-11T08:22:23.728238+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'13","last_scrub_stamp":"2024-09-11T08:21:52.595045+0000","last_deep_scrub":"222'12","last_deep_scrub_stamp":"2024-09-11T08:16:52.592856+0000","last_clean_scrub_stamp":"2024-09-11T08:21:52.595045+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:55.458513+0000","scrub_duration":0.00191333,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":331,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.637090+0000","last_change":"2024-09-11T08:20:54.449873+0000","last_active":"2024-09-11T08:22:20.637090+0000","last_peered":"2024-09-11T08:22:20.637090+0000","last_clean":"2024-09-11T08:22:20.637090+0000","last_became_active":"2024-09-11T08:19:29.147682+0000","last_became_peered":"2024-09-11T08:19:29.147682+0000","last_unstale":"2024-09-11T08:22:20.637090+0000","last_undegraded":"2024-09-11T08:22:20.637090+0000","last_fullsized":"2024-09-11T08:22:20.637090+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:54.449786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:53.175994+0000","last_clean_scrub_stamp":"2024-09-11T08:20:54.449786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015389500000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"239'20","reported_seq":199,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.638939+0000","last_change":"2024-09-11T08:21:44.200657+0000","last_active":"2024-09-11T08:22:20.638939+0000","last_peered":"2024-09-11T08:22:20.638939+0000","last_clean":"2024-09-11T08:22:20.638939+0000","last_became_active":"2024-09-11T08:17:37.554357+0000","last_became_peered":"2024-09-11T08:17:37.554357+0000","last_unstale":"2024-09-11T08:22:20.638939+0000","last_undegraded":"2024-09-11T08:22:20.638939+0000","last_fullsized":"2024-09-11T08:22:20.638939+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'20","last_scrub_stamp":"2024-09-11T08:21:44.200602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:21:44.200602+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:01.573512+0000","scrub_duration":0.0020864519999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":149,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.633874+0000","last_change":"2024-09-11T08:21:53.562461+0000","last_active":"2024-09-11T08:22:20.633874+0000","last_peered":"2024-09-11T08:22:20.633874+0000","last_clean":"2024-09-11T08:22:20.633874+0000","last_became_active":"2024-09-11T08:15:56.518515+0000","last_became_peered":"2024-09-11T08:15:56.518515+0000","last_unstale":"2024-09-11T08:22:20.633874+0000","last_undegraded":"2024-09-11T08:22:20.633874+0000","last_fullsized":"2024-09-11T08:22:20.633874+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:53.562379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:21:53.562379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:59.004320+0000","scrub_duration":0.0015591660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"271'10","reported_seq":106,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:23.728840+0000","last_change":"2024-09-11T08:20:16.922523+0000","last_active":"2024-09-11T08:22:23.728840+0000","last_peered":"2024-09-11T08:22:23.728840+0000","last_clean":"2024-09-11T08:22:23.728840+0000","last_became_active":"2024-09-11T08:19:29.165183+0000","last_became_peered":"2024-09-11T08:19:29.165183+0000","last_unstale":"2024-09-11T08:22:23.728840+0000","last_undegraded":"2024-09-11T08:22:23.728840+0000","last_fullsized":"2024-09-11T08:22:23.728840+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'10","last_scrub_stamp":"2024-09-11T08:20:16.922384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:20:16.922384+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0084094979999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":337,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.638013+0000","last_change":"2024-09-11T08:20:23.600147+0000","last_active":"2024-09-11T08:22:20.638013+0000","last_peered":"2024-09-11T08:22:20.638013+0000","last_clean":"2024-09-11T08:22:20.638013+0000","last_became_active":"2024-09-11T08:15:56.513949+0000","last_became_peered":"2024-09-11T08:15:56.513949+0000","last_unstale":"2024-09-11T08:22:20.638013+0000","last_undegraded":"2024-09-11T08:22:20.638013+0000","last_fullsized":"2024-09-11T08:22:20.638013+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:23.600056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:20:23.600056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003973341,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"239'19","reported_seq":349,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.635899+0000","last_change":"2024-09-11T08:20:09.525188+0000","last_active":"2024-09-11T08:22:20.635899+0000","last_peered":"2024-09-11T08:22:20.635899+0000","last_clean":"2024-09-11T08:22:20.635899+0000","last_became_active":"2024-09-11T08:07:32.047981+0000","last_became_peered":"2024-09-11T08:07:32.047981+0000","last_unstale":"2024-09-11T08:22:20.635899+0000","last_undegraded":"2024-09-11T08:22:20.635899+0000","last_fullsized":"2024-09-11T08:22:20.635899+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'19","last_scrub_stamp":"2024-09-11T08:20:09.525095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:20:09.525095+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0063068040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":347,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.638623+0000","last_change":"2024-09-11T08:20:27.515065+0000","last_active":"2024-09-11T08:22:20.638623+0000","last_peered":"2024-09-11T08:22:20.638623+0000","last_clean":"2024-09-11T08:22:20.638623+0000","last_became_active":"2024-09-11T08:15:56.523574+0000","last_became_peered":"2024-09-11T08:15:56.523574+0000","last_unstale":"2024-09-11T08:22:20.638623+0000","last_undegraded":"2024-09-11T08:22:20.638623+0000","last_fullsized":"2024-09-11T08:22:20.638623+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:27.514983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:20:27.514983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.019346374,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":356,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.649995+0000","last_change":"2024-09-11T08:21:43.836231+0000","last_active":"2024-09-11T08:22:20.649995+0000","last_peered":"2024-09-11T08:22:20.649995+0000","last_clean":"2024-09-11T08:22:20.649995+0000","last_became_active":"2024-09-11T08:19:29.141857+0000","last_became_peered":"2024-09-11T08:19:29.141857+0000","last_unstale":"2024-09-11T08:22:20.649995+0000","last_undegraded":"2024-09-11T08:22:20.649995+0000","last_fullsized":"2024-09-11T08:22:20.649995+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:43.836166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:21:43.836166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:00.285779+0000","scrub_duration":0.001475074,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":141,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.637776+0000","last_change":"2024-09-11T08:21:54.636380+0000","last_active":"2024-09-11T08:22:20.637776+0000","last_peered":"2024-09-11T08:22:20.637776+0000","last_clean":"2024-09-11T08:22:20.637776+0000","last_became_active":"2024-09-11T08:19:29.180111+0000","last_became_peered":"2024-09-11T08:19:29.180111+0000","last_unstale":"2024-09-11T08:22:20.637776+0000","last_undegraded":"2024-09-11T08:22:20.637776+0000","last_fullsized":"2024-09-11T08:22:20.637776+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:54.636326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:21:54.636326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:18.539494+0000","scrub_duration":0.001465472,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"239'15","reported_seq":370,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.650281+0000","last_change":"2024-09-11T08:21:44.864734+0000","last_active":"2024-09-11T08:22:20.650281+0000","last_peered":"2024-09-11T08:22:20.650281+0000","last_clean":"2024-09-11T08:22:20.650281+0000","last_became_active":"2024-09-11T08:07:32.049368+0000","last_became_peered":"2024-09-11T08:07:32.049368+0000","last_unstale":"2024-09-11T08:22:20.650281+0000","last_undegraded":"2024-09-11T08:22:20.650281+0000","last_fullsized":"2024-09-11T08:22:20.650281+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'15","last_scrub_stamp":"2024-09-11T08:21:44.864672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:21:44.864672+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:06.795734+0000","scrub_duration":0.0019068259999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":149,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.635004+0000","last_change":"2024-09-11T08:21:43.492563+0000","last_active":"2024-09-11T08:22:20.635004+0000","last_peered":"2024-09-11T08:22:20.635004+0000","last_clean":"2024-09-11T08:22:20.635004+0000","last_became_active":"2024-09-11T08:15:56.521570+0000","last_became_peered":"2024-09-11T08:15:56.521570+0000","last_unstale":"2024-09-11T08:22:20.635004+0000","last_undegraded":"2024-09-11T08:22:20.635004+0000","last_fullsized":"2024-09-11T08:22:20.635004+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:43.492497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:21:43.492497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:59.430693+0000","scrub_duration":0.001773766,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":121,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.637401+0000","last_change":"2024-09-11T08:22:02.568857+0000","last_active":"2024-09-11T08:22:20.637401+0000","last_peered":"2024-09-11T08:22:20.637401+0000","last_clean":"2024-09-11T08:22:20.637401+0000","last_became_active":"2024-09-11T08:17:37.557645+0000","last_became_peered":"2024-09-11T08:17:37.557645+0000","last_unstale":"2024-09-11T08:22:20.637401+0000","last_undegraded":"2024-09-11T08:22:20.637401+0000","last_fullsized":"2024-09-11T08:22:20.637401+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:22:02.568761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:22:02.568761+0000","last_clean_scrub_stamp":"2024-09-11T08:22:02.568761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:23.299593+0000","scrub_duration":0.0057637030000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":340,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.637219+0000","last_change":"2024-09-11T08:21:08.731138+0000","last_active":"2024-09-11T08:22:20.637219+0000","last_peered":"2024-09-11T08:22:20.637219+0000","last_clean":"2024-09-11T08:22:20.637219+0000","last_became_active":"2024-09-11T08:19:29.198224+0000","last_became_peered":"2024-09-11T08:19:29.198224+0000","last_unstale":"2024-09-11T08:22:20.637219+0000","last_undegraded":"2024-09-11T08:22:20.637219+0000","last_fullsized":"2024-09-11T08:22:20.637219+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:08.731053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:09.973943+0000","last_clean_scrub_stamp":"2024-09-11T08:21:08.731053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:26.737814+0000","scrub_duration":0.0019130169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"239'19","reported_seq":417,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.642278+0000","last_change":"2024-09-11T08:21:53.814597+0000","last_active":"2024-09-11T08:22:20.642278+0000","last_peered":"2024-09-11T08:22:20.642278+0000","last_clean":"2024-09-11T08:22:20.642278+0000","last_became_active":"2024-09-11T08:15:56.520058+0000","last_became_peered":"2024-09-11T08:15:56.520058+0000","last_unstale":"2024-09-11T08:22:20.642278+0000","last_undegraded":"2024-09-11T08:22:20.642278+0000","last_fullsized":"2024-09-11T08:22:20.642278+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'19","last_scrub_stamp":"2024-09-11T08:21:53.814502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:21:53.814502+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:55.121172+0000","scrub_duration":0.0020894770000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":66,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.633163+0000","last_change":"2024-09-11T08:21:11.652549+0000","last_active":"2024-09-11T08:22:20.633163+0000","last_peered":"2024-09-11T08:22:20.633163+0000","last_clean":"2024-09-11T08:22:20.633163+0000","last_became_active":"2024-09-11T08:19:29.143511+0000","last_became_peered":"2024-09-11T08:19:29.143511+0000","last_unstale":"2024-09-11T08:22:20.633163+0000","last_undegraded":"2024-09-11T08:22:20.633163+0000","last_fullsized":"2024-09-11T08:22:20.633163+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:11.652456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:21:11.652456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011171949,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":149,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.633766+0000","last_change":"2024-09-11T08:21:07.260652+0000","last_active":"2024-09-11T08:22:20.633766+0000","last_peered":"2024-09-11T08:22:20.633766+0000","last_clean":"2024-09-11T08:22:20.633766+0000","last_became_active":"2024-09-11T08:15:56.523728+0000","last_became_peered":"2024-09-11T08:15:56.523728+0000","last_unstale":"2024-09-11T08:22:20.633766+0000","last_undegraded":"2024-09-11T08:22:20.633766+0000","last_fullsized":"2024-09-11T08:22:20.633766+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:07.260572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:11:34.159053+0000","last_clean_scrub_stamp":"2024-09-11T08:21:07.260572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0061691999999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":321,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.643188+0000","last_change":"2024-09-11T08:21:46.689250+0000","last_active":"2024-09-11T08:22:20.643188+0000","last_peered":"2024-09-11T08:22:20.643188+0000","last_clean":"2024-09-11T08:22:20.643188+0000","last_became_active":"2024-09-11T08:07:36.633368+0000","last_became_peered":"2024-09-11T08:07:36.633368+0000","last_unstale":"2024-09-11T08:22:20.643188+0000","last_undegraded":"2024-09-11T08:22:20.643188+0000","last_fullsized":"2024-09-11T08:22:20.643188+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:46.689190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:46.489220+0000","last_clean_scrub_stamp":"2024-09-11T08:21:46.689190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:48.254640+0000","scrub_duration":0.001726811,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"312'10","reported_seq":339,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.641901+0000","last_change":"2024-09-11T08:21:55.859919+0000","last_active":"2024-09-11T08:22:20.641901+0000","last_peered":"2024-09-11T08:22:20.641901+0000","last_clean":"2024-09-11T08:22:20.641901+0000","last_became_active":"2024-09-11T08:07:32.045728+0000","last_became_peered":"2024-09-11T08:07:32.045728+0000","last_unstale":"2024-09-11T08:22:20.641901+0000","last_undegraded":"2024-09-11T08:22:20.641901+0000","last_fullsized":"2024-09-11T08:22:20.641901+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"312'10","last_scrub_stamp":"2024-09-11T08:21:55.859866+0000","last_deep_scrub":"312'10","last_deep_scrub_stamp":"2024-09-11T08:21:55.859866+0000","last_clean_scrub_stamp":"2024-09-11T08:21:55.859866+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:00.807147+0000","scrub_duration":0.0027238100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":346,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.638752+0000","last_change":"2024-09-11T08:20:12.651491+0000","last_active":"2024-09-11T08:22:20.638752+0000","last_peered":"2024-09-11T08:22:20.638752+0000","last_clean":"2024-09-11T08:22:20.638752+0000","last_became_active":"2024-09-11T08:19:29.161558+0000","last_became_peered":"2024-09-11T08:19:29.161558+0000","last_unstale":"2024-09-11T08:22:20.638752+0000","last_undegraded":"2024-09-11T08:22:20.638752+0000","last_fullsized":"2024-09-11T08:22:20.638752+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:12.651408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:20:12.651408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0071009350000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":379,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.640825+0000","last_change":"2024-09-11T08:21:11.229626+0000","last_active":"2024-09-11T08:22:20.640825+0000","last_peered":"2024-09-11T08:22:20.640825+0000","last_clean":"2024-09-11T08:22:20.640825+0000","last_became_active":"2024-09-11T08:15:56.528607+0000","last_became_peered":"2024-09-11T08:15:56.528607+0000","last_unstale":"2024-09-11T08:22:20.640825+0000","last_undegraded":"2024-09-11T08:22:20.640825+0000","last_fullsized":"2024-09-11T08:22:20.640825+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:11.229540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:21:11.229540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:36.062439+0000","scrub_duration":0.002377456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":139,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.634752+0000","last_change":"2024-09-11T08:20:18.827796+0000","last_active":"2024-09-11T08:22:20.634752+0000","last_peered":"2024-09-11T08:22:20.634752+0000","last_clean":"2024-09-11T08:22:20.634752+0000","last_became_active":"2024-09-11T08:15:56.521488+0000","last_became_peered":"2024-09-11T08:15:56.521488+0000","last_unstale":"2024-09-11T08:22:20.634752+0000","last_undegraded":"2024-09-11T08:22:20.634752+0000","last_fullsized":"2024-09-11T08:22:20.634752+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:18.827709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:20:18.827709+0000","last_clean_scrub_stamp":"2024-09-11T08:20:18.827709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014060774999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":351,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.650252+0000","last_change":"2024-09-11T08:21:03.627814+0000","last_active":"2024-09-11T08:22:20.650252+0000","last_peered":"2024-09-11T08:22:20.650252+0000","last_clean":"2024-09-11T08:22:20.650252+0000","last_became_active":"2024-09-11T08:19:29.153017+0000","last_became_peered":"2024-09-11T08:19:29.153017+0000","last_unstale":"2024-09-11T08:22:20.650252+0000","last_undegraded":"2024-09-11T08:22:20.650252+0000","last_fullsized":"2024-09-11T08:22:20.650252+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:03.627721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:50.585566+0000","last_clean_scrub_stamp":"2024-09-11T08:21:03.627721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:22:31.922693+0000","scrub_duration":0.003783294,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":111,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.640970+0000","last_change":"2024-09-11T08:20:49.725564+0000","last_active":"2024-09-11T08:22:20.640970+0000","last_peered":"2024-09-11T08:22:20.640970+0000","last_clean":"2024-09-11T08:22:20.640970+0000","last_became_active":"2024-09-11T08:17:37.554262+0000","last_became_peered":"2024-09-11T08:17:37.554262+0000","last_unstale":"2024-09-11T08:22:20.640970+0000","last_undegraded":"2024-09-11T08:22:20.640970+0000","last_fullsized":"2024-09-11T08:22:20.640970+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:20:49.725505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:20:49.725505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012049350000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"288'24","reported_seq":424,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.641608+0000","last_change":"2024-09-11T08:20:28.885398+0000","last_active":"2024-09-11T08:22:20.641608+0000","last_peered":"2024-09-11T08:22:20.641608+0000","last_clean":"2024-09-11T08:22:20.641608+0000","last_became_active":"2024-09-11T08:19:29.145221+0000","last_became_peered":"2024-09-11T08:19:29.145221+0000","last_unstale":"2024-09-11T08:22:20.641608+0000","last_undegraded":"2024-09-11T08:22:20.641608+0000","last_fullsized":"2024-09-11T08:22:20.641608+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'24","last_scrub_stamp":"2024-09-11T08:20:28.885295+0000","last_deep_scrub":"288'24","last_deep_scrub_stamp":"2024-09-11T08:20:28.885295+0000","last_clean_scrub_stamp":"2024-09-11T08:20:28.885295+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0080800170000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":177,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-11T08:22:20.641835+0000","last_change":"2024-09-11T08:22:02.333467+0000","last_active":"2024-09-11T08:22:20.641835+0000","last_peered":"2024-09-11T08:22:20.641835+0000","last_clean":"2024-09-11T08:22:20.641835+0000","last_became_active":"2024-09-11T08:15:08.769133+0000","last_became_peered":"2024-09-11T08:15:08.769133+0000","last_unstale":"2024-09-11T08:22:20.641835+0000","last_undegraded":"2024-09-11T08:22:20.641835+0000","last_fullsized":"2024-09-11T08:22:20.641835+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:22:02.333362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:18:39.206016+0000","last_clean_scrub_stamp":"2024-09-11T08:22:02.333362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:23:32.251869+0000","scrub_duration":0.0035016740000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":3860791296,"num_objects":58911,"num_object_clones":0,"num_object_copies":176733,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58911,"num_write_kb":3770304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11616387072,"data_stored":11616387072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58911,"ondisk_log_size":58911,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1125,"num_read_kb":882,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":215,"num_bytes_recovered":4786,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3425,"internal_metadata":0},"log_size":763,"ondisk_log_size":763,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":489,"num_read_kb":484,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":168,"seq":721554505826,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1258188,"kb_used_data":932356,"kb_used_omap":7,"kb_used_meta":325816,"kb_avail":92486964,"statfs":{"total":95995035648,"available":94706651136,"internally_reserved":0,"allocated":954732544,"data_stored":954294537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8027,"internal_metadata":333635749},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:21:30 2024","interfaces":[{"interface":"back","average":{"1min":2.855,"5min":2.262,"15min":1.329},"min":{"1min":0.32800000000000001,"5min":0.223,"15min":0.223},"max":{"1min":32.942999999999998,"5min":67.409999999999997,"15min":67.409999999999997},"last":0.71399999999999997},{"interface":"front","average":{"1min":2.96,"5min":2.1720000000000002,"15min":1.3859999999999999},"min":{"1min":0.28299999999999997,"5min":0.219,"15min":0.219},"max":{"1min":32.665999999999997,"5min":66.959999999999994,"15min":66.959999999999994},"last":1.3919999999999999}]},{"osd":1,"last update":"Wed Sep 11 08:21:30 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":2.3660000000000001,"15min":2.254},"min":{"1min":0.26000000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":4.2880000000000003,"5min":66.971000000000004,"15min":66.971000000000004},"last":0.72799999999999998},{"interface":"front","average":{"1min":1.226,"5min":2.2839999999999998,"15min":2.2240000000000002},"min":{"1min":0.32700000000000001,"5min":0.23699999999999999,"15min":0.23000000000000001},"max":{"1min":6.798,"5min":67.185000000000002,"15min":67.185000000000002},"last":0.504}]},{"osd":2,"last update":"Wed Sep 11 08:21:30 2024","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.02,"15min":1.0109999999999999},"min":{"1min":0.35199999999999998,"5min":0.25700000000000001,"15min":0.22700000000000001},"max":{"1min":4.8920000000000003,"5min":6.444,"15min":6.444},"last":0.62},{"interface":"front","average":{"1min":0.84599999999999997,"5min":1.0640000000000001,"15min":0.83899999999999997},"min":{"1min":0.28599999999999998,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":4.4939999999999998,"5min":6.3460000000000001,"15min":6.3460000000000001},"last":1.5249999999999999}]},{"osd":3,"last update":"Wed Sep 11 08:21:30 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":1.042,"15min":1.153},"min":{"1min":0.28299999999999997,"5min":0.251,"15min":0.251},"max":{"1min":2.532,"5min":5.2149999999999999,"15min":21.460000000000001},"last":0.55100000000000005},{"interface":"front","average":{"1min":1.6020000000000001,"5min":1.0960000000000001,"15min":1.3879999999999999},"min":{"1min":0.34300000000000003,"5min":0.251,"15min":0.251},"max":{"1min":5.7030000000000003,"5min":6.4660000000000002,"15min":21.440000000000001},"last":0.65400000000000003}]},{"osd":4,"last update":"Wed Sep 11 08:21:30 2024","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.85499999999999998,"15min":0.67000000000000004},"min":{"1min":0.28599999999999998,"5min":0.252,"15min":0.252},"max":{"1min":4.3390000000000004,"5min":4.5289999999999999,"15min":4.5289999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.80900000000000005,"15min":0.76700000000000002},"min":{"1min":0.251,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":1.891,"5min":5.2649999999999997,"15min":5.2649999999999997},"last":0.46400000000000002}]},{"osd":5,"last update":"Wed Sep 11 08:21:30 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":1.7010000000000001,"15min":1.129},"min":{"1min":0.28799999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":4.3710000000000004,"5min":65.866,"15min":65.866},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.86699999999999999,"5min":1.7789999999999999,"15min":1.117},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":2.968,"5min":66.019000000000005,"15min":66.019000000000005},"last":0.52900000000000003}]},{"osd":6,"last update":"Wed Sep 11 08:21:30 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":0.77000000000000002,"15min":0.79900000000000004},"min":{"1min":0.311,"5min":0.248,"15min":0.248},"max":{"1min":7.1699999999999999,"5min":7.1699999999999999,"15min":7.1699999999999999},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.92800000000000005,"15min":0.84999999999999998},"min":{"1min":0.36399999999999999,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":4.468,"5min":4.907,"15min":4.907},"last":1.4390000000000001}]}]},{"osd":6,"up_from":45,"seq":193273528509,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1904320,"kb_used_data":1606964,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":91840832,"statfs":{"total":95995035648,"available":94045011968,"internally_reserved":0,"allocated":1645531136,"data_stored":1645109336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:22:22 2024","interfaces":[{"interface":"back","average":{"1min":1.73,"5min":1.7270000000000001,"15min":1.4930000000000001},"min":{"1min":0.28899999999999998,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":24.940999999999999,"5min":25.494,"15min":45.066000000000003},"last":0.89600000000000002},{"interface":"front","average":{"1min":1.946,"5min":1.5669999999999999,"15min":1.4610000000000001},"min":{"1min":0.28599999999999998,"5min":0.23699999999999999,"15min":0.20599999999999999},"max":{"1min":25.216000000000001,"5min":25.48,"15min":43.899000000000001},"last":0.72799999999999998}]},{"osd":1,"last update":"Wed Sep 11 08:22:22 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":1.3799999999999999,"15min":1.206},"min":{"1min":0.29899999999999999,"5min":0.218,"15min":0.214},"max":{"1min":1.488,"5min":20.321000000000002,"15min":44.658000000000001},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.61899999999999999,"5min":1.2509999999999999,"15min":1.26},"min":{"1min":0.29599999999999999,"5min":0.252,"15min":0.22600000000000001},"max":{"1min":3.1920000000000002,"5min":19.652000000000001,"15min":44.694000000000003},"last":0.29699999999999999}]},{"osd":2,"last update":"Wed Sep 11 08:22:22 2024","interfaces":[{"interface":"back","average":{"1min":2.411,"5min":1.054,"15min":0.91000000000000003},"min":{"1min":0.28999999999999998,"5min":0.254,"15min":0.23200000000000001},"max":{"1min":39.652999999999999,"5min":39.652999999999999,"15min":39.652999999999999},"last":0.82499999999999996},{"interface":"front","average":{"1min":2.2480000000000002,"5min":1.1279999999999999,"15min":1.1160000000000001},"min":{"1min":0.316,"5min":0.252,"15min":0.23300000000000001},"max":{"1min":38.390999999999998,"5min":38.390999999999998,"15min":38.390999999999998},"last":0.65900000000000003}]},{"osd":3,"last update":"Wed Sep 11 08:22:22 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.94399999999999995,"15min":1.167},"min":{"1min":0.372,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.524,"5min":5.2969999999999997,"15min":36.061},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.96299999999999997,"15min":1.167},"min":{"1min":0.374,"5min":0.28999999999999998,"15min":0.26800000000000002},"max":{"1min":3.1699999999999999,"5min":5.5720000000000001,"15min":36.093000000000004},"last":0.32500000000000001}]},{"osd":4,"last update":"Wed Sep 11 08:22:22 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.59599999999999997,"15min":0.55700000000000005},"min":{"1min":0.22500000000000001,"5min":0.20499999999999999,"15min":0.20200000000000001},"max":{"1min":2.0840000000000001,"5min":4.492,"15min":4.492},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.495,"5min":0.58299999999999996,"15min":0.57399999999999995},"min":{"1min":0.26100000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.288,"5min":4.1529999999999996,"15min":4.468},"last":0.70599999999999996}]},{"osd":5,"last update":"Wed Sep 11 08:22:22 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.75900000000000001,"15min":0.70299999999999996},"min":{"1min":0.29599999999999999,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.361,"5min":10.49,"15min":10.49},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.76800000000000002,"15min":0.67400000000000004},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.22600000000000001},"max":{"1min":1.6020000000000001,"5min":10.468,"15min":10.468},"last":0.76700000000000002}]},{"osd":7,"last update":"Wed Sep 11 08:21:27 2024","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":0.74399999999999999,"15min":1.016},"min":{"1min":0.317,"5min":0.253,"15min":0.253},"max":{"1min":3.8809999999999998,"5min":3.8809999999999998,"15min":8.4580000000000002},"last":0.39500000000000002},{"interface":"front","average":{"1min":1,"5min":0.63600000000000001,"15min":0.81899999999999995},"min":{"1min":0.28599999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":3.0870000000000002,"5min":3.0870000000000002,"15min":3.593},"last":0.34799999999999998}]}]},{"osd":5,"up_from":38,"seq":163208757442,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1757760,"kb_used_data":1163700,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":91987392,"statfs":{"total":95995035648,"available":94195089408,"internally_reserved":0,"allocated":1191628800,"data_stored":1191206017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:21:58 2024","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":2.0979999999999999,"15min":1.337},"min":{"1min":0.251,"5min":0.214,"15min":0.184},"max":{"1min":5.1200000000000001,"5min":92.924000000000007,"15min":92.924000000000007},"last":0.93500000000000005},{"interface":"front","average":{"1min":1.2490000000000001,"5min":2.0920000000000001,"15min":1.4079999999999999},"min":{"1min":0.221,"5min":0.221,"15min":0.20399999999999999},"max":{"1min":6.468,"5min":93.173000000000002,"15min":93.173000000000002},"last":0.37}]},{"osd":1,"last update":"Wed Sep 11 08:21:58 2024","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":2.351,"15min":1.5880000000000001},"min":{"1min":0.253,"5min":0.22,"15min":0.22},"max":{"1min":6.423,"5min":96.899000000000001,"15min":96.899000000000001},"last":0.34100000000000003},{"interface":"front","average":{"1min":1.236,"5min":1.9730000000000001,"15min":1.385},"min":{"1min":0.35799999999999998,"5min":0.25700000000000001,"15min":0.22700000000000001},"max":{"1min":5.0780000000000003,"5min":92.337000000000003,"15min":92.337000000000003},"last":0.747}]},{"osd":2,"last update":"Wed Sep 11 08:21:58 2024","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":1.0780000000000001,"15min":1.0349999999999999},"min":{"1min":0.28000000000000003,"5min":0.255,"15min":0.255},"max":{"1min":5.085,"5min":15.622,"15min":16.018000000000001},"last":0.64800000000000002},{"interface":"front","average":{"1min":1.9139999999999999,"5min":1.0649999999999999,"15min":0.88800000000000001},"min":{"1min":0.39000000000000001,"5min":0.27300000000000002,"15min":0.22800000000000001},"max":{"1min":16.713999999999999,"5min":16.713999999999999,"15min":16.713999999999999},"last":0.78200000000000003}]},{"osd":3,"last update":"Wed Sep 11 08:21:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":1.9179999999999999,"15min":1.3300000000000001},"min":{"1min":0.35299999999999998,"5min":0.254,"15min":0.25},"max":{"1min":3.7170000000000001,"5min":92.245000000000005,"15min":92.245000000000005},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.99399999999999999,"5min":2.298,"15min":1.51},"min":{"1min":0.39000000000000001,"5min":0.29699999999999999,"15min":0.251},"max":{"1min":5.1429999999999998,"5min":92.274000000000001,"15min":92.274000000000001},"last":0.76800000000000002}]},{"osd":4,"last update":"Wed Sep 11 08:21:58 2024","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":0.874,"15min":0.66000000000000003},"min":{"1min":0.28000000000000003,"5min":0.27100000000000002,"15min":0.21299999999999999},"max":{"1min":6.3879999999999999,"5min":6.3879999999999999,"15min":6.3879999999999999},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.872,"5min":0.74299999999999999,"15min":0.60899999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23100000000000001},"max":{"1min":3.1480000000000001,"5min":4.9720000000000004,"15min":4.9720000000000004},"last":0.80600000000000005}]},{"osd":6,"last update":"Wed Sep 11 08:22:19 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.85299999999999998,"15min":0.628},"min":{"1min":0.26700000000000002,"5min":0.24399999999999999,"15min":0.23400000000000001},"max":{"1min":4.077,"5min":7.5679999999999996,"15min":7.5679999999999996},"last":2.875},{"interface":"front","average":{"1min":0.85899999999999999,"5min":1.0129999999999999,"15min":0.74099999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":4.7610000000000001,"5min":6.4470000000000001,"15min":6.4470000000000001},"last":2.9100000000000001}]},{"osd":7,"last update":"Wed Sep 11 08:21:39 2024","interfaces":[{"interface":"back","average":{"1min":1.913,"5min":1.169,"15min":0.95099999999999996},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":6.5179999999999998,"5min":6.5179999999999998,"15min":6.5179999999999998},"last":0.60099999999999998},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.0720000000000001,"15min":0.88700000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":5.1989999999999998,"5min":7.5339999999999998,"15min":7.5339999999999998},"last":0.96699999999999997}]}]},{"osd":4,"up_from":31,"seq":133143986375,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2271124,"kb_used_data":1965448,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":91474028,"statfs":{"total":95995035648,"available":93669404672,"internally_reserved":0,"allocated":2012618752,"data_stored":2012176803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:21:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.905,"15min":1.53},"min":{"1min":0.224,"5min":0.222,"15min":0.19},"max":{"1min":8.1880000000000006,"5min":60.497999999999998,"15min":60.497999999999998},"last":0.58099999999999996},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.8620000000000001,"15min":1.3740000000000001},"min":{"1min":0.24099999999999999,"5min":0.22600000000000001,"15min":0.221},"max":{"1min":7.5380000000000003,"5min":60.484999999999999,"15min":60.484999999999999},"last":0.36499999999999999}]},{"osd":1,"last update":"Wed Sep 11 08:21:31 2024","interfaces":[{"interface":"back","average":{"1min":1.7989999999999999,"5min":1.381,"15min":1.042},"min":{"1min":0.254,"5min":0.24299999999999999,"15min":0.214},"max":{"1min":15.127000000000001,"5min":20.471,"15min":20.471},"last":1.5249999999999999},{"interface":"front","average":{"1min":1.6910000000000001,"5min":1.2769999999999999,"15min":0.96999999999999997},"min":{"1min":0.247,"5min":0.23100000000000001,"15min":0.22700000000000001},"max":{"1min":14.843,"5min":15.744999999999999,"15min":15.744999999999999},"last":0.77800000000000002}]},{"osd":2,"last update":"Wed Sep 11 08:21:31 2024","interfaces":[{"interface":"back","average":{"1min":2.7879999999999998,"5min":1.119,"15min":0.93799999999999994},"min":{"1min":0.28100000000000003,"5min":0.253,"15min":0.23400000000000001},"max":{"1min":40.993000000000002,"5min":40.993000000000002,"15min":40.993000000000002},"last":0.60499999999999998},{"interface":"front","average":{"1min":2.621,"5min":1.1910000000000001,"15min":0.93500000000000005},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.248},"max":{"1min":41.152000000000001,"5min":41.152000000000001,"15min":41.152000000000001},"last":2.8410000000000002}]},{"osd":3,"last update":"Wed Sep 11 08:21:31 2024","interfaces":[{"interface":"back","average":{"1min":1.6439999999999999,"5min":1.4119999999999999,"15min":1.333},"min":{"1min":0.312,"5min":0.253,"15min":0.253},"max":{"1min":7.8150000000000004,"5min":19.102,"15min":26.884},"last":3.972},{"interface":"front","average":{"1min":1.1479999999999999,"5min":1.052,"15min":1.2290000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.246},"max":{"1min":7.5869999999999997,"5min":7.5869999999999997,"15min":26.783000000000001},"last":0.42199999999999999}]},{"osd":5,"last update":"Wed Sep 11 08:21:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.98299999999999998,"15min":0.65400000000000003},"min":{"1min":0.28000000000000003,"5min":0.26500000000000001,"15min":0.23499999999999999},"max":{"1min":1.4319999999999999,"5min":33.384999999999998,"15min":33.384999999999998},"last":0.624},{"interface":"front","average":{"1min":0.60199999999999998,"5min":1.0700000000000001,"15min":0.68300000000000005},"min":{"1min":0.255,"5min":0.253,"15min":0.24199999999999999},"max":{"1min":1.5209999999999999,"5min":33.856000000000002,"15min":33.856000000000002},"last":0.38700000000000001}]},{"osd":6,"last update":"Wed Sep 11 08:22:22 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.65000000000000002,"15min":0.70299999999999996},"min":{"1min":0.30599999999999999,"5min":0.26800000000000002,"15min":0.23899999999999999},"max":{"1min":1.9319999999999999,"5min":4.6799999999999997,"15min":48.808},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.72499999999999998,"15min":0.76000000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.26200000000000001},"max":{"1min":3.7440000000000002,"5min":4.0499999999999998,"15min":48.746000000000002},"last":0.64400000000000002}]},{"osd":7,"last update":"Wed Sep 11 08:21:40 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.68100000000000005,"15min":0.57399999999999995},"min":{"1min":0.33500000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":2.052,"5min":5.3449999999999998,"15min":5.3449999999999998},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.84299999999999997,"15min":0.73399999999999999},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":1.911,"5min":4.7160000000000002,"15min":4.7160000000000002},"last":0.39300000000000002}]}]},{"osd":3,"up_from":25,"seq":107374182604,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1758428,"kb_used_data":1172692,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":91986724,"statfs":{"total":95995035648,"available":94194405376,"internally_reserved":0,"allocated":1200836608,"data_stored":1200381842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:22:19 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":1.651,"15min":1.5669999999999999},"min":{"1min":0.215,"5min":0.14999999999999999,"15min":0.14999999999999999},"max":{"1min":1.9350000000000001,"5min":58.880000000000003,"15min":60.469999999999999},"last":0.19800000000000001},{"interface":"front","average":{"1min":0.57399999999999995,"5min":1.6870000000000001,"15min":1.665},"min":{"1min":0.222,"5min":0.184,"15min":0.17399999999999999},"max":{"1min":2.992,"5min":59.246000000000002,"15min":60.488999999999997},"last":0.95599999999999996}]},{"osd":1,"last update":"Wed Sep 11 08:22:19 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":1.206,"15min":1.391},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.73099999999999998,"5min":11.148999999999999,"15min":60.578000000000003},"last":0.434},{"interface":"front","average":{"1min":0.58699999999999997,"5min":1.3720000000000001,"15min":1.6899999999999999},"min":{"1min":0.21099999999999999,"5min":0.20000000000000001,"15min":0.19500000000000001},"max":{"1min":2.7679999999999998,"5min":11.853,"15min":63.674999999999997},"last":0.312}]},{"osd":2,"last update":"Wed Sep 11 08:22:19 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":1.1160000000000001,"15min":1.4890000000000001},"min":{"1min":0.26000000000000001,"5min":0.23000000000000001,"15min":0.185},"max":{"1min":2.8090000000000002,"5min":8.4179999999999993,"15min":60.645000000000003},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.52500000000000002,"5min":1.03,"15min":1.373},"min":{"1min":0.25800000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":2.7450000000000001,"5min":5.9500000000000002,"15min":60.723999999999997},"last":0.48999999999999999}]},{"osd":4,"last update":"Wed Sep 11 08:21:26 2024","interfaces":[{"interface":"back","average":{"1min":1.8819999999999999,"5min":1.3320000000000001,"15min":1.335},"min":{"1min":0.307,"5min":0.29499999999999998,"15min":0.24299999999999999},"max":{"1min":11.802,"5min":11.802,"15min":61.290999999999997},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.93200000000000005,"5min":0.998,"15min":1.1779999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.27200000000000002},"max":{"1min":3.9780000000000002,"5min":6.1340000000000003,"15min":60.856000000000002},"last":1.4510000000000001}]},{"osd":5,"last update":"Wed Sep 11 08:22:03 2024","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.8879999999999999,"15min":1.6870000000000001},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.26500000000000001},"max":{"1min":10.885999999999999,"5min":31.460999999999999,"15min":60.914999999999999},"last":1.4890000000000001},{"interface":"front","average":{"1min":0.755,"5min":1.77,"15min":1.7250000000000001},"min":{"1min":0.34200000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":3.5659999999999998,"5min":31.216000000000001,"15min":60.787999999999997},"last":0.46899999999999997}]},{"osd":6,"last update":"Wed Sep 11 08:22:19 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":1.171,"15min":1.5229999999999999},"min":{"1min":0.318,"5min":0.308,"15min":0.308},"max":{"1min":2.8799999999999999,"5min":10.949999999999999,"15min":60.817},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.82399999999999995,"5min":1.4219999999999999,"15min":1.7210000000000001},"min":{"1min":0.36699999999999999,"5min":0.35399999999999998,"15min":0.307},"max":{"1min":2.8180000000000001,"5min":11.750999999999999,"15min":61.264000000000003},"last":0.56399999999999995}]},{"osd":7,"last update":"Wed Sep 11 08:21:37 2024","interfaces":[{"interface":"back","average":{"1min":2.004,"5min":1.6990000000000001,"15min":2.7080000000000002},"min":{"1min":0.35999999999999999,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":11.016999999999999,"5min":31.071000000000002,"15min":60.847000000000001},"last":0.92500000000000004},{"interface":"front","average":{"1min":4.032,"5min":2.0510000000000002,"15min":2.786},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":58.820999999999998,"5min":58.820999999999998,"15min":60.883000000000003},"last":0.51500000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411535,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1712508,"kb_used_data":1402676,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":92032644,"statfs":{"total":95995035648,"available":94241427456,"internally_reserved":0,"allocated":1436340224,"data_stored":1435917353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:21:57 2024","interfaces":[{"interface":"back","average":{"1min":3.0569999999999999,"5min":1.788,"15min":1.639},"min":{"1min":0.221,"5min":0.17799999999999999,"15min":0.17199999999999999},"max":{"1min":45.838000000000001,"5min":45.838000000000001,"15min":45.838000000000001},"last":0.52000000000000002},{"interface":"front","average":{"1min":2.8479999999999999,"5min":2.3100000000000001,"15min":1.8600000000000001},"min":{"1min":0.22500000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":46.508000000000003,"5min":46.508000000000003,"15min":46.508000000000003},"last":0.57499999999999996}]},{"osd":1,"last update":"Wed Sep 11 08:21:57 2024","interfaces":[{"interface":"back","average":{"1min":3.4159999999999999,"5min":1.9610000000000001,"15min":1.4850000000000001},"min":{"1min":0.23200000000000001,"5min":0.20200000000000001,"15min":0.17599999999999999},"max":{"1min":53.347000000000001,"5min":53.347000000000001,"15min":53.347000000000001},"last":0.65100000000000002},{"interface":"front","average":{"1min":3.5099999999999998,"5min":1.7490000000000001,"15min":1.4650000000000001},"min":{"1min":0.246,"5min":0.20799999999999999,"15min":0.20200000000000001},"max":{"1min":53.082999999999998,"5min":53.082999999999998,"15min":53.082999999999998},"last":0.45000000000000001}]},{"osd":3,"last update":"Wed Sep 11 08:22:23 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":1.2829999999999999,"15min":1.3140000000000001},"min":{"1min":0.22800000000000001,"5min":0.221,"15min":0.188},"max":{"1min":2.427,"5min":46.070999999999998,"15min":46.070999999999998},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.82499999999999996,"5min":1.258,"15min":1.2709999999999999},"min":{"1min":0.23899999999999999,"5min":0.21299999999999999,"15min":0.21199999999999999},"max":{"1min":6.3070000000000004,"5min":45.790999999999997,"15min":45.790999999999997},"last":0.36399999999999999}]},{"osd":4,"last update":"Wed Sep 11 08:21:36 2024","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":0.86799999999999999,"15min":0.93200000000000005},"min":{"1min":0.33100000000000002,"5min":0.27900000000000003,"15min":0.25900000000000001},"max":{"1min":5.8680000000000003,"5min":5.8680000000000003,"15min":7.806},"last":4.2439999999999998},{"interface":"front","average":{"1min":1.052,"5min":0.82299999999999995,"15min":0.94899999999999995},"min":{"1min":0.30299999999999999,"5min":0.27500000000000002,"15min":0.252},"max":{"1min":5.7279999999999998,"5min":5.7279999999999998,"15min":12.129},"last":1.353}]},{"osd":5,"last update":"Wed Sep 11 08:22:06 2024","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.7809999999999999,"15min":1.274},"min":{"1min":0.35299999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":5.7530000000000001,"5min":35.963000000000001,"15min":35.963000000000001},"last":1.454},{"interface":"front","average":{"1min":1.494,"5min":1.855,"15min":1.3080000000000001},"min":{"1min":0.29899999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":15.032,"5min":35.104999999999997,"15min":35.104999999999997},"last":0.61499999999999999}]},{"osd":6,"last update":"Wed Sep 11 08:21:25 2024","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":0.89900000000000002,"15min":0.91600000000000004},"min":{"1min":0.311,"5min":0.31,"15min":0.26100000000000001},"max":{"1min":4.4989999999999997,"5min":4.4989999999999997,"15min":7.2249999999999996},"last":3.6000000000000001},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.89700000000000002,"15min":0.996},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32100000000000001},"max":{"1min":5.8010000000000002,"5min":5.8010000000000002,"15min":6.71},"last":1.8620000000000001}]},{"osd":7,"last update":"Wed Sep 11 08:21:36 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":1.431,"15min":1.4079999999999999},"min":{"1min":0.33300000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":2.6880000000000002,"5min":35.274999999999999,"15min":35.274999999999999},"last":0.40000000000000002},{"interface":"front","average":{"1min":1.349,"5min":1.5289999999999999,"15min":1.526},"min":{"1min":0.34599999999999997,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":5.7770000000000001,"5min":35.265000000000001,"15min":35.265000000000001},"last":0.67800000000000005}]}]},{"osd":1,"up_from":13,"seq":55834575061,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1301280,"kb_used_data":711384,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":92443872,"statfs":{"total":95995035648,"available":94662524928,"internally_reserved":0,"allocated":728457216,"data_stored":727998497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:21:28 2024","interfaces":[{"interface":"back","average":{"1min":3.9609999999999999,"5min":2.1120000000000001,"15min":1.516},"min":{"1min":0.185,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":63.932000000000002,"5min":63.932000000000002,"15min":63.932000000000002},"last":2.4009999999999998},{"interface":"front","average":{"1min":4.5099999999999998,"5min":2.2120000000000002,"15min":1.8129999999999999},"min":{"1min":0.17799999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":64.269000000000005,"5min":64.269000000000005,"15min":64.269000000000005},"last":2.1749999999999998}]},{"osd":2,"last update":"Wed Sep 11 08:22:08 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":1.1599999999999999,"15min":1.149},"min":{"1min":0.23000000000000001,"5min":0.189,"15min":0.189},"max":{"1min":1.1659999999999999,"5min":7.798,"15min":8.3819999999999997},"last":1.448},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.98299999999999998,"15min":1.2549999999999999},"min":{"1min":0.224,"5min":0.19,"15min":0.16300000000000001},"max":{"1min":4.9790000000000001,"5min":5.6289999999999996,"15min":10.276},"last":1.3680000000000001}]},{"osd":3,"last update":"Wed Sep 11 08:22:14 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":1.347,"15min":1.784},"min":{"1min":0.20799999999999999,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":3.7170000000000001,"5min":8.0909999999999993,"15min":32.759},"last":0.376},{"interface":"front","average":{"1min":0.53000000000000003,"5min":1.5369999999999999,"15min":1.635},"min":{"1min":0.25700000000000001,"5min":0.25,"15min":0.22600000000000001},"max":{"1min":1.278,"5min":9.0060000000000002,"15min":32.444000000000003},"last":1.548}]},{"osd":4,"last update":"Wed Sep 11 08:21:34 2024","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":2.8700000000000001,"15min":1.706},"min":{"1min":0.308,"5min":0.26200000000000001,"15min":0.252},"max":{"1min":5.7939999999999996,"5min":110.86499999999999,"15min":110.86499999999999},"last":0.36599999999999999},{"interface":"front","average":{"1min":1.5309999999999999,"5min":2.9940000000000002,"15min":1.653},"min":{"1min":0.29299999999999998,"5min":0.28000000000000003,"15min":0.248},"max":{"1min":10.468999999999999,"5min":110.831,"15min":110.831},"last":1.524}]},{"osd":5,"last update":"Wed Sep 11 08:22:08 2024","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":2.625,"15min":1.7130000000000001},"min":{"1min":0.41799999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":5.8300000000000001,"5min":109.208,"15min":109.208},"last":0.46500000000000002},{"interface":"front","average":{"1min":1.272,"5min":2.7410000000000001,"15min":1.821},"min":{"1min":0.36799999999999999,"5min":0.28299999999999997,"15min":0.28100000000000003},"max":{"1min":7.9359999999999999,"5min":111.254,"15min":111.254},"last":0.34899999999999998}]},{"osd":6,"last update":"Wed Sep 11 08:22:14 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":1.155,"15min":1.696},"min":{"1min":0.40100000000000002,"5min":0.30499999999999999,"15min":0.29399999999999998},"max":{"1min":3.806,"5min":7.9809999999999999,"15min":124.31999999999999},"last":0.315},{"interface":"front","average":{"1min":0.71999999999999997,"5min":1.4279999999999999,"15min":1.6779999999999999},"min":{"1min":0.34200000000000003,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.6870000000000001,"5min":8.6099999999999994,"15min":124.958},"last":1.496}]},{"osd":7,"last update":"Wed Sep 11 08:21:34 2024","interfaces":[{"interface":"back","average":{"1min":1.514,"5min":2.528,"15min":1.7210000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":8.5419999999999998,"5min":109.149,"15min":109.149},"last":0.40899999999999997},{"interface":"front","average":{"1min":2.0529999999999999,"5min":2.7519999999999998,"15min":1.9379999999999999},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":10.695,"5min":109.309,"15min":109.309},"last":0.40100000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738585,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2929944,"kb_used_data":2344208,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":90815208,"statfs":{"total":95995035648,"available":92994772992,"internally_reserved":0,"allocated":2400468992,"data_stored":2400019643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 11 08:21:28 2024","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.252,"15min":1.1799999999999999},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17000000000000001},"max":{"1min":4.883,"5min":10.105,"15min":10.105},"last":0.22},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.429,"15min":1.4059999999999999},"min":{"1min":0.23400000000000001,"5min":0.16600000000000001,"15min":0.153},"max":{"1min":3.4119999999999999,"5min":19.852,"15min":19.852},"last":0.20599999999999999}]},{"osd":2,"last update":"Wed Sep 11 08:21:55 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":1,"15min":1.1399999999999999},"min":{"1min":0.21099999999999999,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.77600000000000002,"5min":10.025,"15min":10.351000000000001},"last":0.19},{"interface":"front","average":{"1min":0.48599999999999999,"5min":1.123,"15min":1.236},"min":{"1min":0.245,"5min":0.23699999999999999,"15min":0.186},"max":{"1min":1.526,"5min":10.074999999999999,"15min":10.074999999999999},"last":0.34999999999999998}]},{"osd":3,"last update":"Wed Sep 11 08:22:17 2024","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":1.1539999999999999,"15min":1.5549999999999999},"min":{"1min":0.28899999999999998,"5min":0.251,"15min":0.22900000000000001},"max":{"1min":4.5259999999999998,"5min":8.4149999999999991,"15min":35.491},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.90600000000000003,"15min":1.4670000000000001},"min":{"1min":0.23499999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":1.891,"5min":7.899,"15min":37.563000000000002},"last":0.44500000000000001}]},{"osd":4,"last update":"Wed Sep 11 08:21:40 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.79000000000000004,"15min":1.026},"min":{"1min":0.29599999999999999,"5min":0.252,"15min":0.24099999999999999},"max":{"1min":3.266,"5min":4.5949999999999998,"15min":9.4030000000000005},"last":1.4410000000000001},{"interface":"front","average":{"1min":0.70399999999999996,"5min":1.0449999999999999,"15min":1.0409999999999999},"min":{"1min":0.29299999999999998,"5min":0.27200000000000002,"15min":0.24399999999999999},"max":{"1min":1.5669999999999999,"5min":8.3510000000000009,"15min":8.3510000000000009},"last":1.5289999999999999}]},{"osd":5,"last update":"Wed Sep 11 08:22:00 2024","interfaces":[{"interface":"back","average":{"1min":1.6459999999999999,"5min":1.3839999999999999,"15min":1.3140000000000001},"min":{"1min":0.40300000000000002,"5min":0.28599999999999998,"15min":0.255},"max":{"1min":15.754,"5min":15.754,"15min":24.914999999999999},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.59499999999999997,"5min":1.3720000000000001,"15min":1.399},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27100000000000002},"max":{"1min":1.8180000000000001,"5min":13.885,"15min":25.277000000000001},"last":0.59699999999999998}]},{"osd":6,"last update":"Wed Sep 11 08:22:23 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":0.89000000000000001,"15min":1.4159999999999999},"min":{"1min":0.42499999999999999,"5min":0.32600000000000001,"15min":0.29099999999999998},"max":{"1min":2.9220000000000002,"5min":4.2670000000000003,"15min":78.522000000000006},"last":1.468},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.92500000000000004,"15min":1.3360000000000001},"min":{"1min":0.48599999999999999,"5min":0.30299999999999999,"15min":0.28299999999999997},"max":{"1min":2.528,"5min":4.7539999999999996,"15min":79.808000000000007},"last":0.93100000000000005}]},{"osd":7,"last update":"Wed Sep 11 08:21:40 2024","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":1.0880000000000001,"15min":1.3109999999999999},"min":{"1min":0.39800000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":2.6179999999999999,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.436},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.349,"15min":1.371},"min":{"1min":0.41799999999999998,"5min":0.33600000000000002,"15min":0.32000000000000001},"max":{"1min":8.2780000000000005,"5min":13.914,"15min":13.914},"last":3.101}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3425,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2464808960,"data_stored":2464808960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":740818944,"data_stored":740818944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1435762688,"data_stored":1435762688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1212153856,"data_stored":1212153856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1908932608,"data_stored":1908932608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1217789952,"data_stored":1217789952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1676541952,"data_stored":1676541952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":959578112,"data_stored":959578112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T08:22:33.428 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 6, currently 16 2024-09-11T08:22:33.437 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:33.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:33.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:22:33.539 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:22:33.573 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:33.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:33.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:22:33.675 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:22:33.708 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:33.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:33.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:22:33.810 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:22:33.843 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:33.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:33.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:22:33.937 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:33 smithi150 ceph-mon[29276]: from='client.25426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:22:33.937 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:33 smithi150 ceph-mon[29276]: osdmap e319: 8 total, 8 up, 8 in 2024-09-11T08:22:33.937 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:33 smithi150 ceph-mon[29276]: pgmap v682: 148 pgs: 148 active+clean; 3.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 692 B/s rd, 99 MiB/s wr, 1.58k op/s 2024-09-11T08:22:33.937 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:33 smithi150 ceph-mon[39612]: from='client.25426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:22:33.937 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:33 smithi150 ceph-mon[39612]: osdmap e319: 8 total, 8 up, 8 in 2024-09-11T08:22:33.937 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:33 smithi150 ceph-mon[39612]: pgmap v682: 148 pgs: 148 active+clean; 3.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 692 B/s rd, 99 MiB/s wr, 1.58k op/s 2024-09-11T08:22:33.945 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:22:33.986 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:33.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:33.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:22:34.031 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 33 16 67821 67805 128.392 90.5 0.00717657 0.00778417 2024-09-11T08:22:34.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:33 smithi181 ceph-mon[32701]: from='client.25426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:22:34.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:33 smithi181 ceph-mon[32701]: osdmap e319: 8 total, 8 up, 8 in 2024-09-11T08:22:34.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:33 smithi181 ceph-mon[32701]: pgmap v682: 148 pgs: 148 active+clean; 3.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 692 B/s rd, 99 MiB/s wr, 1.58k op/s 2024-09-11T08:22:34.088 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:34.122 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:34.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:34.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:34.224 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:34.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:34.264 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:34.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:34.365 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:34.404 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:34.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:34.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:34.456 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:22:34.506 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:22:34.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:34.553 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:34.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:22:34.654 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:22:34.698 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:34.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:34.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:22:34.700 INFO:teuthology.orchestra.run.smithi150.stderr:noscrub is unset 2024-09-11T08:22:34.701 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:34 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3232715817' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:22:34.701 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:34 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:22:34.701 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:34 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3232715817' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:22:34.701 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:34 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:22:34.801 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:22:34.838 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:34.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:34.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:22:34.940 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:34.970 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:34.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:34.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:35.031 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 34 16 69249 69233 127.241 89.25 0.0180179 0.00785398 2024-09-11T08:22:35.072 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:22:35.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:34 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3232715817' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:22:35.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:34 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:22:35.110 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:35.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:35.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:22:35.212 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:22:35.251 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:35.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:35.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:22:35.353 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:22:35.397 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:35.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:35.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:22:35.499 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:22:35.543 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:35.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:35.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:22:35.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:22:35 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:22:35] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:22:35.645 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:35.683 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:35.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:35.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:35.785 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:22:35.824 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:35.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:35.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:22:35.926 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:35.960 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:35.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:35.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:35.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:35 smithi150 ceph-mon[29276]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:22:35.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:35 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:22:35.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:35 smithi150 ceph-mon[29276]: osdmap e320: 8 total, 8 up, 8 in 2024-09-11T08:22:35.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:35 smithi150 ceph-mon[29276]: 4.4 scrub starts 2024-09-11T08:22:35.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:35 smithi150 ceph-mon[29276]: 4.4 scrub ok 2024-09-11T08:22:35.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:35 smithi150 ceph-mon[29276]: 6.1b scrub starts 2024-09-11T08:22:35.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:35 smithi150 ceph-mon[29276]: 6.1b scrub ok 2024-09-11T08:22:35.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:35 smithi150 ceph-mon[29276]: pgmap v684: 148 pgs: 148 active+clean; 4.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 80 MiB/s wr, 1.28k op/s 2024-09-11T08:22:35.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:35 smithi150 ceph-mon[29276]: 5.e scrub starts 2024-09-11T08:22:35.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:35 smithi150 ceph-mon[29276]: 5.e scrub ok 2024-09-11T08:22:35.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:35 smithi150 ceph-mon[29276]: osdmap e321: 8 total, 8 up, 8 in 2024-09-11T08:22:35.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:35 smithi150 ceph-mon[39612]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:22:35.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:35 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:22:35.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:35 smithi150 ceph-mon[39612]: osdmap e320: 8 total, 8 up, 8 in 2024-09-11T08:22:35.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:35 smithi150 ceph-mon[39612]: 4.4 scrub starts 2024-09-11T08:22:35.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:35 smithi150 ceph-mon[39612]: 4.4 scrub ok 2024-09-11T08:22:35.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:35 smithi150 ceph-mon[39612]: 6.1b scrub starts 2024-09-11T08:22:35.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:35 smithi150 ceph-mon[39612]: 6.1b scrub ok 2024-09-11T08:22:35.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:35 smithi150 ceph-mon[39612]: pgmap v684: 148 pgs: 148 active+clean; 4.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 80 MiB/s wr, 1.28k op/s 2024-09-11T08:22:35.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:35 smithi150 ceph-mon[39612]: 5.e scrub starts 2024-09-11T08:22:35.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:35 smithi150 ceph-mon[39612]: 5.e scrub ok 2024-09-11T08:22:35.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:35 smithi150 ceph-mon[39612]: osdmap e321: 8 total, 8 up, 8 in 2024-09-11T08:22:36.031 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 35 16 70703 70687 126.202 90.875 0.0133754 0.00791902 2024-09-11T08:22:36.063 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:22:36.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:35 smithi181 ceph-mon[32701]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:22:36.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:35 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:22:36.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:35 smithi181 ceph-mon[32701]: osdmap e320: 8 total, 8 up, 8 in 2024-09-11T08:22:36.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:35 smithi181 ceph-mon[32701]: 4.4 scrub starts 2024-09-11T08:22:36.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:35 smithi181 ceph-mon[32701]: 4.4 scrub ok 2024-09-11T08:22:36.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:35 smithi181 ceph-mon[32701]: 6.1b scrub starts 2024-09-11T08:22:36.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:35 smithi181 ceph-mon[32701]: 6.1b scrub ok 2024-09-11T08:22:36.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:35 smithi181 ceph-mon[32701]: pgmap v684: 148 pgs: 148 active+clean; 4.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 80 MiB/s wr, 1.28k op/s 2024-09-11T08:22:36.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:35 smithi181 ceph-mon[32701]: 5.e scrub starts 2024-09-11T08:22:36.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:35 smithi181 ceph-mon[32701]: 5.e scrub ok 2024-09-11T08:22:36.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:35 smithi181 ceph-mon[32701]: osdmap e321: 8 total, 8 up, 8 in 2024-09-11T08:22:36.099 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:36.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:36.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:22:36.201 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:36.296 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:36.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:36.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:36.398 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:36.437 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:36.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:36.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:36.540 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:36.562 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:36.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:36.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:36.664 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:36.698 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:36.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:36.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:36.800 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:36.834 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:36.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:36.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:36.936 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:36.970 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:36.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:36.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:36.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:36 smithi150 ceph-mon[29276]: 5.17 scrub starts 2024-09-11T08:22:36.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:36 smithi150 ceph-mon[29276]: 5.17 scrub ok 2024-09-11T08:22:36.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:36 smithi150 ceph-mon[39612]: 5.17 scrub starts 2024-09-11T08:22:36.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:36 smithi150 ceph-mon[39612]: 5.17 scrub ok 2024-09-11T08:22:37.031 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 36 16 71954 71938 124.868 78.1875 0.0114726 0.00800393 2024-09-11T08:22:37.072 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:37.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:36 smithi181 ceph-mon[32701]: 5.17 scrub starts 2024-09-11T08:22:37.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:36 smithi181 ceph-mon[32701]: 5.17 scrub ok 2024-09-11T08:22:37.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:37.103 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:37.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:37.157 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-11T08:22:37.157 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [ 2024-09-11T08:22:37.157 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.157 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:72099 10.5 10:ab1def0b:::benchmark_data_smithi150_158067_object72098:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e321)", 2024-09-11T08:22:37.157 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:22:37.147690+0000", 2024-09-11T08:22:37.157 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.0084080619999999991, 2024-09-11T08:22:37.157 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0084428369999999999, 2024-09-11T08:22:37.157 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:22:37.158 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "waiting for sub ops", 2024-09-11T08:22:37.158 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:22:37.158 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:22:37.158 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:22:37.158 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 72099 2024-09-11T08:22:37.158 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:37.158 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:22:37.158 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.158 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:22:37.158 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:37.147690+0000", 2024-09-11T08:22:37.158 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:37.158 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:37.159 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.159 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:22:37.159 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:37.147690+0000", 2024-09-11T08:22:37.159 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:37.159 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:37.159 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.160 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:22:37.160 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:37.147688+0000", 2024-09-11T08:22:37.160 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:22:37.160 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:37.160 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.160 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:22:37.160 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:37.147714+0000", 2024-09-11T08:22:37.160 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.5712e-05 2024-09-11T08:22:37.161 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:37.161 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.161 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:22:37.161 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:37.147727+0000", 2024-09-11T08:22:37.161 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3098999999999999e-05 2024-09-11T08:22:37.161 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:37.161 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.161 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:22:37.161 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:37.147734+0000", 2024-09-11T08:22:37.161 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.6069999999999996e-06 2024-09-11T08:22:37.161 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:37.161 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.161 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:37.162 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:37.147796+0000", 2024-09-11T08:22:37.162 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.2189999999999999e-05 2024-09-11T08:22:37.162 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:37.162 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.162 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:22:37.162 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:37.147923+0000", 2024-09-11T08:22:37.162 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012716800000000001 2024-09-11T08:22:37.162 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:37.162 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.162 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 0,3", 2024-09-11T08:22:37.162 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:37.148211+0000", 2024-09-11T08:22:37.162 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000288051 2024-09-11T08:22:37.162 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:37.162 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:22:37.163 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:37.163 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:37.163 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.163 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:72101 10.5 10.22527465 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e321)", 2024-09-11T08:22:37.163 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:22:37.148115+0000", 2024-09-11T08:22:37.163 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.0079831480000000007, 2024-09-11T08:22:37.163 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0080919200000000007, 2024-09-11T08:22:37.163 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:22:37.163 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "queued for pg", 2024-09-11T08:22:37.163 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:22:37.163 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:22:37.163 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:22:37.163 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 72101 2024-09-11T08:22:37.163 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:37.163 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:22:37.164 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.164 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:22:37.164 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:37.148115+0000", 2024-09-11T08:22:37.164 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:37.164 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:37.164 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.164 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:22:37.164 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:37.148115+0000", 2024-09-11T08:22:37.164 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:37.164 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:37.164 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.164 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:22:37.164 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:37.148113+0000", 2024-09-11T08:22:37.164 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:22:37.164 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:37.165 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.165 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:22:37.165 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:37.148137+0000", 2024-09-11T08:22:37.165 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.3669999999999999e-05 2024-09-11T08:22:37.165 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:37.165 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.165 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:22:37.165 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:37.148153+0000", 2024-09-11T08:22:37.165 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.6368e-05 2024-09-11T08:22:37.165 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:37.165 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.165 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:22:37.165 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:37.148159+0000", 2024-09-11T08:22:37.165 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.733e-06 2024-09-11T08:22:37.165 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:37.165 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:22:37.166 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:37.166 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:37.166 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.166 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:72107 10.3 10:c2194485:::benchmark_data_smithi150_158067_object72106:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e321)", 2024-09-11T08:22:37.166 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:22:37.149678+0000", 2024-09-11T08:22:37.166 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.006419681, 2024-09-11T08:22:37.166 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0065849430000000002, 2024-09-11T08:22:37.166 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:22:37.166 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "waiting for sub ops", 2024-09-11T08:22:37.166 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:22:37.166 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:22:37.166 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:22:37.166 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 72107 2024-09-11T08:22:37.166 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:37.166 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:22:37.167 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.167 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:22:37.167 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:37.149678+0000", 2024-09-11T08:22:37.167 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:37.167 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:37.167 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.167 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:22:37.167 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:37.149678+0000", 2024-09-11T08:22:37.167 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:37.167 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:37.167 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.167 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:22:37.167 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:37.149677+0000", 2024-09-11T08:22:37.167 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:22:37.168 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:37.168 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.168 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:22:37.168 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:37.149741+0000", 2024-09-11T08:22:37.168 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.4097999999999995e-05 2024-09-11T08:22:37.168 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:37.168 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.168 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:22:37.168 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:37.149756+0000", 2024-09-11T08:22:37.168 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4929000000000001e-05 2024-09-11T08:22:37.168 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:37.168 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.168 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:22:37.168 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:37.149762+0000", 2024-09-11T08:22:37.169 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.75e-06 2024-09-11T08:22:37.169 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:37.169 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.169 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:37.169 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:37.154935+0000", 2024-09-11T08:22:37.169 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0051728470000000004 2024-09-11T08:22:37.169 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:37.169 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.169 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:22:37.169 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:37.155063+0000", 2024-09-11T08:22:37.169 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000127952 2024-09-11T08:22:37.169 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:37.170 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.170 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 3,5", 2024-09-11T08:22:37.170 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:37.155195+0000", 2024-09-11T08:22:37.170 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000132834 2024-09-11T08:22:37.170 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:37.170 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:22:37.170 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:37.170 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:37.170 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.170 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.15636.0:72106 10.f e321/308 10:f965ff0e:::benchmark_data_smithi150_158067_object72105:head v 321'4454, mlcod=321'4452)", 2024-09-11T08:22:37.170 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:22:37.150175+0000", 2024-09-11T08:22:37.170 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.005922493, 2024-09-11T08:22:37.170 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0061524290000000001, 2024-09-11T08:22:37.170 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:22:37.170 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "started", 2024-09-11T08:22:37.171 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:22:37.171 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.171 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:22:37.171 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:37.150175+0000", 2024-09-11T08:22:37.171 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:37.171 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:37.171 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.171 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:22:37.171 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:37.150175+0000", 2024-09-11T08:22:37.171 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:37.171 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:37.171 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.171 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:22:37.171 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:37.150174+0000", 2024-09-11T08:22:37.172 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:22:37.172 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:37.172 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.172 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:22:37.172 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:37.150215+0000", 2024-09-11T08:22:37.172 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.091e-05 2024-09-11T08:22:37.172 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:37.172 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.172 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:22:37.172 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:37.150229+0000", 2024-09-11T08:22:37.173 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3373999999999999e-05 2024-09-11T08:22:37.173 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:37.173 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.173 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:22:37.173 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:37.150234+0000", 2024-09-11T08:22:37.173 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.8270000000000003e-06 2024-09-11T08:22:37.173 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:37.173 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.173 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:37.173 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:37.150298+0000", 2024-09-11T08:22:37.173 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.3281999999999995e-05 2024-09-11T08:22:37.173 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:37.173 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.173 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:22:37.173 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:37.150318+0000", 2024-09-11T08:22:37.174 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.0176000000000001e-05 2024-09-11T08:22:37.174 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:37.174 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:22:37.174 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:37.174 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:37.174 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.174 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.15636.0:72108 10.e e321/308 10:72335d80:::benchmark_data_smithi150_158067_object72107:head v 321'4358, mlcod=321'4356)", 2024-09-11T08:22:37.174 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:22:37.153983+0000", 2024-09-11T08:22:37.174 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.0021147679999999999, 2024-09-11T08:22:37.174 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0023987840000000002, 2024-09-11T08:22:37.174 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:22:37.174 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "started", 2024-09-11T08:22:37.174 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:22:37.174 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.175 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:22:37.175 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:37.153983+0000", 2024-09-11T08:22:37.175 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:37.175 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:37.175 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.175 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:22:37.175 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:37.153983+0000", 2024-09-11T08:22:37.175 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:37.175 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:37.175 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.175 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:22:37.175 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:37.153982+0000", 2024-09-11T08:22:37.175 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:22:37.175 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:37.176 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.176 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:22:37.176 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:37.154760+0000", 2024-09-11T08:22:37.176 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000778114 2024-09-11T08:22:37.176 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:37.176 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.176 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:22:37.176 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:37.154787+0000", 2024-09-11T08:22:37.176 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.7002e-05 2024-09-11T08:22:37.176 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:37.176 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.176 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:22:37.176 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:37.154794+0000", 2024-09-11T08:22:37.176 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.7070000000000003e-06 2024-09-11T08:22:37.176 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:37.177 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.177 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:37.177 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:37.155140+0000", 2024-09-11T08:22:37.177 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00034573399999999999 2024-09-11T08:22:37.177 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:37.177 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.177 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:22:37.177 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:37.155159+0000", 2024-09-11T08:22:37.177 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.8961e-05 2024-09-11T08:22:37.177 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:37.177 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:22:37.177 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:37.177 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:37.177 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.178 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:72092 10.3 10:c690f16a:::benchmark_data_smithi150_158067_object72091:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e321)", 2024-09-11T08:22:37.178 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:22:37.140080+0000", 2024-09-11T08:22:37.178 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.016018174, 2024-09-11T08:22:37.178 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.016358257000000001, 2024-09-11T08:22:37.178 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:22:37.178 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "waiting for sub ops", 2024-09-11T08:22:37.178 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:22:37.178 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:22:37.178 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:22:37.178 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 72092 2024-09-11T08:22:37.178 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:37.178 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:22:37.178 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.178 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:22:37.178 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:37.140080+0000", 2024-09-11T08:22:37.178 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:37.179 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:37.179 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.179 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:22:37.179 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:37.140080+0000", 2024-09-11T08:22:37.179 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:37.179 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:37.179 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.179 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:22:37.179 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:37.140078+0000", 2024-09-11T08:22:37.179 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:22:37.179 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:37.179 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.179 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:22:37.179 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:37.140104+0000", 2024-09-11T08:22:37.179 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.6554e-05 2024-09-11T08:22:37.180 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:37.180 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.180 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:22:37.180 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:37.140121+0000", 2024-09-11T08:22:37.180 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.6676000000000001e-05 2024-09-11T08:22:37.180 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:37.180 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.180 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:22:37.180 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:37.140128+0000", 2024-09-11T08:22:37.180 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.5339999999999996e-06 2024-09-11T08:22:37.180 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:37.180 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.180 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:37.180 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:37.142038+0000", 2024-09-11T08:22:37.181 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.001910587 2024-09-11T08:22:37.181 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:37.181 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.181 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:22:37.181 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:37.142529+0000", 2024-09-11T08:22:37.181 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00049037099999999997 2024-09-11T08:22:37.181 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:37.181 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.181 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 3,5", 2024-09-11T08:22:37.181 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:37.143116+0000", 2024-09-11T08:22:37.181 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00058702899999999998 2024-09-11T08:22:37.181 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:37.181 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.181 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:22:37.181 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:37.148910+0000", 2024-09-11T08:22:37.181 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.005794324 2024-09-11T08:22:37.182 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:37.182 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:37.182 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:22:37.182 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:37.149051+0000", 2024-09-11T08:22:37.182 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000141151 2024-09-11T08:22:37.182 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:37.182 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:22:37.182 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:37.182 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:37.182 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-09-11T08:22:37.182 INFO:teuthology.orchestra.run.smithi150.stdout: "num_ops": 6 2024-09-11T08:22:37.182 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-11T08:22:37.205 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:22:37.233 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:37.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:37.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:22:37.335 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:37.370 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:37.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:37.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:37.472 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:37.509 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:37.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:37.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:37.611 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:22:37.647 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:37.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:37.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:22:37.748 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:37.784 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:37.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:37.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:37.886 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:37.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:37.922 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:37.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:37.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:37 smithi150 ceph-mon[29276]: 4.1f scrub starts 2024-09-11T08:22:37.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:37 smithi150 ceph-mon[29276]: 4.1f scrub ok 2024-09-11T08:22:37.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:37 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:22:37.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:37 smithi150 ceph-mon[29276]: 3.0 scrub starts 2024-09-11T08:22:37.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:37 smithi150 ceph-mon[29276]: 3.0 scrub ok 2024-09-11T08:22:37.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:37 smithi150 ceph-mon[29276]: pgmap v686: 148 pgs: 148 active+clean; 4.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 80 MiB/s wr, 1.28k op/s 2024-09-11T08:22:37.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:37 smithi150 ceph-mon[29276]: 5.18 scrub starts 2024-09-11T08:22:37.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:37 smithi150 ceph-mon[29276]: 5.18 scrub ok 2024-09-11T08:22:37.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:37 smithi150 ceph-mon[29276]: 3.2 scrub starts 2024-09-11T08:22:37.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:37 smithi150 ceph-mon[29276]: 3.2 scrub ok 2024-09-11T08:22:37.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:37 smithi150 ceph-mon[39612]: 4.1f scrub starts 2024-09-11T08:22:37.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:37 smithi150 ceph-mon[39612]: 4.1f scrub ok 2024-09-11T08:22:37.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:37 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:22:37.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:37 smithi150 ceph-mon[39612]: 3.0 scrub starts 2024-09-11T08:22:37.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:37 smithi150 ceph-mon[39612]: 3.0 scrub ok 2024-09-11T08:22:37.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:37 smithi150 ceph-mon[39612]: pgmap v686: 148 pgs: 148 active+clean; 4.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 80 MiB/s wr, 1.28k op/s 2024-09-11T08:22:37.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:37 smithi150 ceph-mon[39612]: 5.18 scrub starts 2024-09-11T08:22:37.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:37 smithi150 ceph-mon[39612]: 5.18 scrub ok 2024-09-11T08:22:37.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:37 smithi150 ceph-mon[39612]: 3.2 scrub starts 2024-09-11T08:22:37.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:37 smithi150 ceph-mon[39612]: 3.2 scrub ok 2024-09-11T08:22:38.024 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:22:38.031 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 37 16 73390 73374 123.918 89.75 0.00786956 0.00806466 2024-09-11T08:22:38.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:38.061 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:38.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:22:38.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:37 smithi181 ceph-mon[32701]: 4.1f scrub starts 2024-09-11T08:22:38.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:37 smithi181 ceph-mon[32701]: 4.1f scrub ok 2024-09-11T08:22:38.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:37 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:22:38.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:37 smithi181 ceph-mon[32701]: 3.0 scrub starts 2024-09-11T08:22:38.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:37 smithi181 ceph-mon[32701]: 3.0 scrub ok 2024-09-11T08:22:38.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:37 smithi181 ceph-mon[32701]: pgmap v686: 148 pgs: 148 active+clean; 4.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 80 MiB/s wr, 1.28k op/s 2024-09-11T08:22:38.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:37 smithi181 ceph-mon[32701]: 5.18 scrub starts 2024-09-11T08:22:38.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:37 smithi181 ceph-mon[32701]: 5.18 scrub ok 2024-09-11T08:22:38.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:37 smithi181 ceph-mon[32701]: 3.2 scrub starts 2024-09-11T08:22:38.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:37 smithi181 ceph-mon[32701]: 3.2 scrub ok 2024-09-11T08:22:38.162 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:22:38.203 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:38.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:38.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:22:38.305 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:22:38.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:38.347 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:38.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:22:38.428 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-11T08:22:38.429 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T08:22:38.429 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-09-11T08:22:38.429 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-11T08:22:38.429 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-11T08:22:38.429 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-09-11T08:22:38.447 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:38.513 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:38.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:38.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:38.518 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-11T08:22:38.615 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:38.651 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:38.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:38.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:38.739 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:22:38 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:22:38] "GET /metrics HTTP/1.1" 200 224655 "" "Prometheus/2.43.0" 2024-09-11T08:22:38.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:38 smithi150 ceph-mon[39612]: 6.f scrub starts 2024-09-11T08:22:38.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:38 smithi150 ceph-mon[39612]: 6.f scrub ok 2024-09-11T08:22:38.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:38 smithi150 ceph-mon[39612]: 5.2 scrub starts 2024-09-11T08:22:38.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:38 smithi150 ceph-mon[39612]: 5.2 scrub ok 2024-09-11T08:22:38.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:38 smithi150 ceph-mon[39612]: 5.c scrub starts 2024-09-11T08:22:38.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:38 smithi150 ceph-mon[39612]: 5.c scrub ok 2024-09-11T08:22:38.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:38 smithi150 ceph-mon[29276]: 6.f scrub starts 2024-09-11T08:22:38.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:38 smithi150 ceph-mon[29276]: 6.f scrub ok 2024-09-11T08:22:38.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:38 smithi150 ceph-mon[29276]: 5.2 scrub starts 2024-09-11T08:22:38.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:38 smithi150 ceph-mon[29276]: 5.2 scrub ok 2024-09-11T08:22:38.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:38 smithi150 ceph-mon[29276]: 5.c scrub starts 2024-09-11T08:22:38.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:38 smithi150 ceph-mon[29276]: 5.c scrub ok 2024-09-11T08:22:38.753 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:22:38.793 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:38.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:38.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:22:38.823 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:22:38 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:22:38.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:38 smithi181 ceph-mon[32701]: 6.f scrub starts 2024-09-11T08:22:38.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:38 smithi181 ceph-mon[32701]: 6.f scrub ok 2024-09-11T08:22:38.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:38 smithi181 ceph-mon[32701]: 5.2 scrub starts 2024-09-11T08:22:38.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:38 smithi181 ceph-mon[32701]: 5.2 scrub ok 2024-09-11T08:22:38.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:38 smithi181 ceph-mon[32701]: 5.c scrub starts 2024-09-11T08:22:38.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:38 smithi181 ceph-mon[32701]: 5.c scrub ok 2024-09-11T08:22:38.846 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:22:38.896 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:38.916 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T08:22:38.928 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:38.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:38.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:39.031 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:39.032 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 38 16 74686 74670 122.787 81 0.0107364 0.008139 2024-09-11T08:22:39.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:39.062 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:39.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:39.122 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:22:39.122 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:22:39.154 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:22:39.154 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:22:39.163 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:22:39.177 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:22:39.187 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:22:39.188 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:22:39.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:39.216 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:39.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:22:39.224 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:22:39.224 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:22:39.262 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:22:39.263 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:22:39.296 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:22:39.297 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:22:39.317 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:22:39.326 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:22:39.327 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:22:39.360 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:39.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:39.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:22:39.362 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:22:39.362 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:22:39.462 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:22:39.495 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:39.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:39.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:22:39.596 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:22:39.635 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:39.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:39.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:22:39.736 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:22:39.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:39.789 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:39.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:22:39.890 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:22:39.925 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:39.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:39.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:22:39.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:39 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:22:39.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:39 smithi150 ceph-mon[29276]: 3.d scrub starts 2024-09-11T08:22:39.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:39 smithi150 ceph-mon[29276]: 3.d scrub ok 2024-09-11T08:22:39.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:39 smithi150 ceph-mon[29276]: 6.1 scrub starts 2024-09-11T08:22:39.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:39 smithi150 ceph-mon[29276]: 6.1 scrub ok 2024-09-11T08:22:39.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:39 smithi150 ceph-mon[29276]: pgmap v687: 148 pgs: 148 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.6 KiB/s rd, 144 MiB/s wr, 2.30k op/s 2024-09-11T08:22:39.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:39 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/4068340290' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:22:39.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:39 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:22:39.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:39 smithi150 ceph-mon[39612]: 3.d scrub starts 2024-09-11T08:22:39.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:39 smithi150 ceph-mon[39612]: 3.d scrub ok 2024-09-11T08:22:39.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:39 smithi150 ceph-mon[39612]: 6.1 scrub starts 2024-09-11T08:22:39.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:39 smithi150 ceph-mon[39612]: 6.1 scrub ok 2024-09-11T08:22:39.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:39 smithi150 ceph-mon[39612]: pgmap v687: 148 pgs: 148 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.6 KiB/s rd, 144 MiB/s wr, 2.30k op/s 2024-09-11T08:22:39.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:39 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/4068340290' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:22:40.025 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:39 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:22:40.025 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:39 smithi181 ceph-mon[32701]: 3.d scrub starts 2024-09-11T08:22:40.025 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:39 smithi181 ceph-mon[32701]: 3.d scrub ok 2024-09-11T08:22:40.025 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:39 smithi181 ceph-mon[32701]: 6.1 scrub starts 2024-09-11T08:22:40.025 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:39 smithi181 ceph-mon[32701]: 6.1 scrub ok 2024-09-11T08:22:40.025 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:39 smithi181 ceph-mon[32701]: pgmap v687: 148 pgs: 148 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.6 KiB/s rd, 144 MiB/s wr, 2.30k op/s 2024-09-11T08:22:40.025 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:39 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/4068340290' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:22:40.027 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:40.032 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 39 16 75966 75950 121.69 80 0.00487871 0.00821337 2024-09-11T08:22:40.061 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:40.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:40.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:40.163 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:40.201 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:40.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:40.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:40.303 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:22:40.339 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:40.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:40.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:22:40.441 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:22:40.480 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:40.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:40.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:22:40.582 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:22:40.618 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:40.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:40.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:22:40.719 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:22:40.750 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:40.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:40.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:22:40.759 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-11T08:22:40.759 INFO:teuthology.orchestra.run.smithi181.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-11T08:22:40.759 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-11T08:22:40.852 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:40.884 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:40.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:40.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:40.985 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:22:40.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:40 smithi150 ceph-mon[29276]: 4.1c scrub starts 2024-09-11T08:22:40.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:40 smithi150 ceph-mon[29276]: 4.1c scrub ok 2024-09-11T08:22:40.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:40 smithi150 ceph-mon[29276]: 3.1e scrub starts 2024-09-11T08:22:40.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:40 smithi150 ceph-mon[29276]: 3.1e scrub ok 2024-09-11T08:22:40.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:40 smithi150 ceph-mon[29276]: 4.f scrub starts 2024-09-11T08:22:40.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:40 smithi150 ceph-mon[29276]: 4.f scrub ok 2024-09-11T08:22:40.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:40 smithi150 ceph-mon[29276]: 5.0 scrub starts 2024-09-11T08:22:40.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:40 smithi150 ceph-mon[29276]: 5.0 scrub ok 2024-09-11T08:22:40.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:40 smithi150 ceph-mon[39612]: 4.1c scrub starts 2024-09-11T08:22:40.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:40 smithi150 ceph-mon[39612]: 4.1c scrub ok 2024-09-11T08:22:40.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:40 smithi150 ceph-mon[39612]: 3.1e scrub starts 2024-09-11T08:22:40.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:40 smithi150 ceph-mon[39612]: 3.1e scrub ok 2024-09-11T08:22:40.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:40 smithi150 ceph-mon[39612]: 4.f scrub starts 2024-09-11T08:22:40.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:40 smithi150 ceph-mon[39612]: 4.f scrub ok 2024-09-11T08:22:40.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:40 smithi150 ceph-mon[39612]: 5.0 scrub starts 2024-09-11T08:22:40.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:40 smithi150 ceph-mon[39612]: 5.0 scrub ok 2024-09-11T08:22:41.020 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:41.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:41.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:22:41.032 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:2024-09-11T08:22:41.033173+0000 min lat: 0.00123278 max lat: 0.187209 avg lat: 0.00828527 2024-09-11T08:22:41.032 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-11T08:22:41.032 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 40 16 77234 77218 120.629 79.25 0.00469787 0.00828527 2024-09-11T08:22:41.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:40 smithi181 ceph-mon[32701]: 4.1c scrub starts 2024-09-11T08:22:41.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:40 smithi181 ceph-mon[32701]: 4.1c scrub ok 2024-09-11T08:22:41.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:40 smithi181 ceph-mon[32701]: 3.1e scrub starts 2024-09-11T08:22:41.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:40 smithi181 ceph-mon[32701]: 3.1e scrub ok 2024-09-11T08:22:41.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:40 smithi181 ceph-mon[32701]: 4.f scrub starts 2024-09-11T08:22:41.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:40 smithi181 ceph-mon[32701]: 4.f scrub ok 2024-09-11T08:22:41.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:40 smithi181 ceph-mon[32701]: 5.0 scrub starts 2024-09-11T08:22:41.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:40 smithi181 ceph-mon[32701]: 5.0 scrub ok 2024-09-11T08:22:41.122 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:22:41.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:41.161 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:41.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:22:41.262 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:22:41.305 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:41.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:41.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:22:41.407 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:41.441 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:41.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:41.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:41.544 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:22:41.580 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:41.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:41.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:22:41.682 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:41.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:41.722 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:41.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:41.823 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:41.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:41.862 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:41.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:41.963 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:22:41.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:41 smithi150 ceph-mon[29276]: 5.11 scrub starts 2024-09-11T08:22:41.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:41 smithi150 ceph-mon[29276]: 5.11 scrub ok 2024-09-11T08:22:41.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:41 smithi150 ceph-mon[29276]: 6.12 scrub starts 2024-09-11T08:22:41.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:41 smithi150 ceph-mon[29276]: 6.12 scrub ok 2024-09-11T08:22:41.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:41 smithi150 ceph-mon[29276]: 3.19 scrub starts 2024-09-11T08:22:41.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:41 smithi150 ceph-mon[29276]: 3.19 scrub ok 2024-09-11T08:22:41.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:41 smithi150 ceph-mon[29276]: pgmap v688: 148 pgs: 148 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 84 MiB/s wr, 1.34k op/s 2024-09-11T08:22:41.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:41 smithi150 ceph-mon[29276]: 3.1d scrub starts 2024-09-11T08:22:41.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:41 smithi150 ceph-mon[29276]: 3.1d scrub ok 2024-09-11T08:22:41.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:41 smithi150 ceph-mon[39612]: 5.11 scrub starts 2024-09-11T08:22:41.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:41 smithi150 ceph-mon[39612]: 5.11 scrub ok 2024-09-11T08:22:41.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:41 smithi150 ceph-mon[39612]: 6.12 scrub starts 2024-09-11T08:22:41.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:41 smithi150 ceph-mon[39612]: 6.12 scrub ok 2024-09-11T08:22:41.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:41 smithi150 ceph-mon[39612]: 3.19 scrub starts 2024-09-11T08:22:41.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:41 smithi150 ceph-mon[39612]: 3.19 scrub ok 2024-09-11T08:22:41.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:41 smithi150 ceph-mon[39612]: pgmap v688: 148 pgs: 148 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 84 MiB/s wr, 1.34k op/s 2024-09-11T08:22:41.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:41 smithi150 ceph-mon[39612]: 3.1d scrub starts 2024-09-11T08:22:41.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:41 smithi150 ceph-mon[39612]: 3.1d scrub ok 2024-09-11T08:22:41.999 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:42.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:42.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:22:42.032 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 41 16 78415 78399 119.487 73.8125 0.011194 0.00836403 2024-09-11T08:22:42.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:41 smithi181 ceph-mon[32701]: 5.11 scrub starts 2024-09-11T08:22:42.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:41 smithi181 ceph-mon[32701]: 5.11 scrub ok 2024-09-11T08:22:42.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:41 smithi181 ceph-mon[32701]: 6.12 scrub starts 2024-09-11T08:22:42.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:41 smithi181 ceph-mon[32701]: 6.12 scrub ok 2024-09-11T08:22:42.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:41 smithi181 ceph-mon[32701]: 3.19 scrub starts 2024-09-11T08:22:42.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:41 smithi181 ceph-mon[32701]: 3.19 scrub ok 2024-09-11T08:22:42.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:41 smithi181 ceph-mon[32701]: pgmap v688: 148 pgs: 148 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 84 MiB/s wr, 1.34k op/s 2024-09-11T08:22:42.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:41 smithi181 ceph-mon[32701]: 3.1d scrub starts 2024-09-11T08:22:42.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:41 smithi181 ceph-mon[32701]: 3.1d scrub ok 2024-09-11T08:22:42.102 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:22:42.136 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:42.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:42.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:22:42.238 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:42.273 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:42.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:42.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:42.375 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:22:42.404 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:42.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:42.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:22:42.506 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:22:42.542 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:42.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:42.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:22:42.644 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:22:42.675 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:42.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:42.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:22:42.777 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:42.808 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:42.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:42.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:42.910 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:42.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:42.946 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:42.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:42.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:42 smithi150 ceph-mon[29276]: 3.11 scrub starts 2024-09-11T08:22:42.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:42 smithi150 ceph-mon[29276]: 3.11 scrub ok 2024-09-11T08:22:42.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:42 smithi150 ceph-mon[29276]: 4.8 scrub starts 2024-09-11T08:22:42.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:42 smithi150 ceph-mon[29276]: 4.8 scrub ok 2024-09-11T08:22:42.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:42 smithi150 ceph-mon[39612]: 3.11 scrub starts 2024-09-11T08:22:42.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:42 smithi150 ceph-mon[39612]: 3.11 scrub ok 2024-09-11T08:22:42.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:42 smithi150 ceph-mon[39612]: 4.8 scrub starts 2024-09-11T08:22:42.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:42 smithi150 ceph-mon[39612]: 4.8 scrub ok 2024-09-11T08:22:43.033 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 42 16 79560 79544 118.346 71.5625 0.00943985 0.00844517 2024-09-11T08:22:43.047 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:22:43.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:42 smithi181 ceph-mon[32701]: 3.11 scrub starts 2024-09-11T08:22:43.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:42 smithi181 ceph-mon[32701]: 3.11 scrub ok 2024-09-11T08:22:43.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:42 smithi181 ceph-mon[32701]: 4.8 scrub starts 2024-09-11T08:22:43.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:42 smithi181 ceph-mon[32701]: 4.8 scrub ok 2024-09-11T08:22:43.079 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:43.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:43.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:22:43.181 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:22:43.211 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:43.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:43.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:22:43.313 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:22:43.342 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:43.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:43.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:22:43.444 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:22:43.483 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:43.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:43.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:22:43.585 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:43.606 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:43.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:43.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:43.708 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:22:43.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:43.746 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:43.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:22:43.847 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:43.871 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:43.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:43.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:43.973 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:44.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:44.010 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:44.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:44.032 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 43 16 80678 80662 117.219 69.875 0.0123881 0.00852683 2024-09-11T08:22:44.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:43 smithi181 ceph-mon[32701]: 5.b scrub starts 2024-09-11T08:22:44.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:43 smithi181 ceph-mon[32701]: 5.b scrub ok 2024-09-11T08:22:44.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:43 smithi181 ceph-mon[32701]: 5.19 scrub starts 2024-09-11T08:22:44.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:43 smithi181 ceph-mon[32701]: 5.19 scrub ok 2024-09-11T08:22:44.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:43 smithi181 ceph-mon[32701]: pgmap v689: 148 pgs: 148 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 614 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-09-11T08:22:44.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:43 smithi181 ceph-mon[32701]: 4.e scrub starts 2024-09-11T08:22:44.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:43 smithi181 ceph-mon[32701]: 4.e scrub ok 2024-09-11T08:22:44.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:43 smithi181 ceph-mon[32701]: 4.17 scrub starts 2024-09-11T08:22:44.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:43 smithi181 ceph-mon[32701]: 4.17 scrub ok 2024-09-11T08:22:44.111 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:22:44.147 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:44.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:44.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:22:44.189 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:43 smithi150 ceph-mon[29276]: 5.b scrub starts 2024-09-11T08:22:44.190 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:43 smithi150 ceph-mon[29276]: 5.b scrub ok 2024-09-11T08:22:44.190 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:43 smithi150 ceph-mon[29276]: 5.19 scrub starts 2024-09-11T08:22:44.190 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:43 smithi150 ceph-mon[29276]: 5.19 scrub ok 2024-09-11T08:22:44.190 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:43 smithi150 ceph-mon[29276]: pgmap v689: 148 pgs: 148 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 614 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-09-11T08:22:44.190 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:43 smithi150 ceph-mon[29276]: 4.e scrub starts 2024-09-11T08:22:44.190 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:43 smithi150 ceph-mon[29276]: 4.e scrub ok 2024-09-11T08:22:44.190 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:43 smithi150 ceph-mon[29276]: 4.17 scrub starts 2024-09-11T08:22:44.190 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:43 smithi150 ceph-mon[29276]: 4.17 scrub ok 2024-09-11T08:22:44.190 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:43 smithi150 ceph-mon[39612]: 5.b scrub starts 2024-09-11T08:22:44.191 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:43 smithi150 ceph-mon[39612]: 5.b scrub ok 2024-09-11T08:22:44.191 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:43 smithi150 ceph-mon[39612]: 5.19 scrub starts 2024-09-11T08:22:44.191 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:43 smithi150 ceph-mon[39612]: 5.19 scrub ok 2024-09-11T08:22:44.191 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:43 smithi150 ceph-mon[39612]: pgmap v689: 148 pgs: 148 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 614 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-09-11T08:22:44.191 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:43 smithi150 ceph-mon[39612]: 4.e scrub starts 2024-09-11T08:22:44.191 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:43 smithi150 ceph-mon[39612]: 4.e scrub ok 2024-09-11T08:22:44.191 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:43 smithi150 ceph-mon[39612]: 4.17 scrub starts 2024-09-11T08:22:44.191 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:43 smithi150 ceph-mon[39612]: 4.17 scrub ok 2024-09-11T08:22:44.249 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:22:44.286 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:44.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:44.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:22:44.388 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:44.421 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:44.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:44.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:44.523 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:44.562 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:44.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:44.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:44.664 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:44.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:44.697 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:44.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:44.798 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:22:44.832 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:44.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:44.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:22:44.933 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:44.966 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:44.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:44.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:45.024 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:22:45.035 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 44 15 82012 81997 116.451 83.4375 0.00888612 0.00858283 2024-09-11T08:22:45.068 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:45.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:44 smithi181 ceph-mon[32701]: 6.8 scrub starts 2024-09-11T08:22:45.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:44 smithi181 ceph-mon[32701]: 6.8 scrub ok 2024-09-11T08:22:45.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:44 smithi181 ceph-mon[32701]: 3.1b scrub starts 2024-09-11T08:22:45.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:44 smithi181 ceph-mon[32701]: 3.1b scrub ok 2024-09-11T08:22:45.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:44 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:22:45.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:44 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:22:45.083 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T08:22:45.102 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:45.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:45.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:45.120 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:44 smithi150 ceph-mon[29276]: 6.8 scrub starts 2024-09-11T08:22:45.120 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:44 smithi150 ceph-mon[29276]: 6.8 scrub ok 2024-09-11T08:22:45.120 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:44 smithi150 ceph-mon[29276]: 3.1b scrub starts 2024-09-11T08:22:45.120 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:44 smithi150 ceph-mon[29276]: 3.1b scrub ok 2024-09-11T08:22:45.121 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:44 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:22:45.121 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:44 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:22:45.121 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:44 smithi150 ceph-mon[39612]: 6.8 scrub starts 2024-09-11T08:22:45.121 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:44 smithi150 ceph-mon[39612]: 6.8 scrub ok 2024-09-11T08:22:45.121 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:44 smithi150 ceph-mon[39612]: 3.1b scrub starts 2024-09-11T08:22:45.121 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:44 smithi150 ceph-mon[39612]: 3.1b scrub ok 2024-09-11T08:22:45.121 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:44 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:22:45.121 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:44 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:22:45.204 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:45.241 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:45.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:45.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:45.342 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:22:45.381 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:45.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:45.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:22:45.483 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:22:45.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:45.516 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:45.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:22:45.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:22:45 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:22:45] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:22:45.616 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:45.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:45.667 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:45.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:45.768 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:22:45.772 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-09-11T08:22:45.772 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:22:45.805 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:45.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:45.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:22:45.907 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:22:45.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:45.947 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:45.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:22:45.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:45 smithi150 ceph-mon[29276]: 5.8 scrub starts 2024-09-11T08:22:45.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:45 smithi150 ceph-mon[29276]: 5.8 scrub ok 2024-09-11T08:22:45.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:45 smithi150 ceph-mon[29276]: 6.c scrub starts 2024-09-11T08:22:45.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:45 smithi150 ceph-mon[29276]: 6.c scrub ok 2024-09-11T08:22:45.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:45 smithi150 ceph-mon[29276]: osdmap e322: 8 total, 8 up, 8 in 2024-09-11T08:22:45.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:45 smithi150 ceph-mon[29276]: pgmap v691: 148 pgs: 148 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 88 MiB/s wr, 1.40k op/s 2024-09-11T08:22:45.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:45 smithi150 ceph-mon[29276]: 6.1a scrub starts 2024-09-11T08:22:45.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:45 smithi150 ceph-mon[29276]: 6.1a scrub ok 2024-09-11T08:22:45.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:45 smithi150 ceph-mon[29276]: osdmap e323: 8 total, 8 up, 8 in 2024-09-11T08:22:45.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:45 smithi150 ceph-mon[39612]: 5.8 scrub starts 2024-09-11T08:22:45.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:45 smithi150 ceph-mon[39612]: 5.8 scrub ok 2024-09-11T08:22:45.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:45 smithi150 ceph-mon[39612]: 6.c scrub starts 2024-09-11T08:22:45.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:45 smithi150 ceph-mon[39612]: 6.c scrub ok 2024-09-11T08:22:45.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:45 smithi150 ceph-mon[39612]: osdmap e322: 8 total, 8 up, 8 in 2024-09-11T08:22:45.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:45 smithi150 ceph-mon[39612]: pgmap v691: 148 pgs: 148 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 88 MiB/s wr, 1.40k op/s 2024-09-11T08:22:45.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:45 smithi150 ceph-mon[39612]: 6.1a scrub starts 2024-09-11T08:22:45.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:45 smithi150 ceph-mon[39612]: 6.1a scrub ok 2024-09-11T08:22:45.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:45 smithi150 ceph-mon[39612]: osdmap e323: 8 total, 8 up, 8 in 2024-09-11T08:22:46.033 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 45 16 83184 83168 115.489 73.1875 0.0103462 0.00865433 2024-09-11T08:22:46.048 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:22:46.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:45 smithi181 ceph-mon[32701]: 5.8 scrub starts 2024-09-11T08:22:46.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:45 smithi181 ceph-mon[32701]: 5.8 scrub ok 2024-09-11T08:22:46.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:45 smithi181 ceph-mon[32701]: 6.c scrub starts 2024-09-11T08:22:46.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:45 smithi181 ceph-mon[32701]: 6.c scrub ok 2024-09-11T08:22:46.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:45 smithi181 ceph-mon[32701]: osdmap e322: 8 total, 8 up, 8 in 2024-09-11T08:22:46.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:45 smithi181 ceph-mon[32701]: pgmap v691: 148 pgs: 148 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 88 MiB/s wr, 1.40k op/s 2024-09-11T08:22:46.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:45 smithi181 ceph-mon[32701]: 6.1a scrub starts 2024-09-11T08:22:46.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:45 smithi181 ceph-mon[32701]: 6.1a scrub ok 2024-09-11T08:22:46.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:45 smithi181 ceph-mon[32701]: osdmap e323: 8 total, 8 up, 8 in 2024-09-11T08:22:46.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:46.083 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:46.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:22:46.184 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:46.213 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:46.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:46.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:46.314 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:46.349 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:46.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:46.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:46.451 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:22:46.478 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-11T08:22:46.478 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T08:22:46.478 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-11T08:22:46.479 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 0.000000 2024-09-11T08:22:46.479 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 5 0 2024-09-11T08:22:46.484 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:46.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:46.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:22:46.586 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:46.639 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:46.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:46.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:46.742 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:22:46.762 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:46 smithi150 ceph-mon[29276]: 5.a scrub starts 2024-09-11T08:22:46.762 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:46 smithi150 ceph-mon[29276]: 5.a scrub ok 2024-09-11T08:22:46.762 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:46 smithi150 ceph-mon[29276]: 6.1d scrub starts 2024-09-11T08:22:46.763 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:46 smithi150 ceph-mon[29276]: 6.1d scrub ok 2024-09-11T08:22:46.763 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:46 smithi150 ceph-mon[29276]: 6.d scrub starts 2024-09-11T08:22:46.763 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:46 smithi150 ceph-mon[29276]: 6.d scrub ok 2024-09-11T08:22:46.766 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:46 smithi150 ceph-mon[39612]: 5.a scrub starts 2024-09-11T08:22:46.767 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:46 smithi150 ceph-mon[39612]: 5.a scrub ok 2024-09-11T08:22:46.767 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:46 smithi150 ceph-mon[39612]: 6.1d scrub starts 2024-09-11T08:22:46.767 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:46 smithi150 ceph-mon[39612]: 6.1d scrub ok 2024-09-11T08:22:46.767 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:46 smithi150 ceph-mon[39612]: 6.d scrub starts 2024-09-11T08:22:46.767 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:46 smithi150 ceph-mon[39612]: 6.d scrub ok 2024-09-11T08:22:46.781 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:46.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:46.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:22:46.882 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:22:46.925 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:46.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:46.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:22:47.027 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:47.033 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 46 15 84287 84272 114.479 69 0.0143335 0.00873068 2024-09-11T08:22:47.063 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:47.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:47.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:47.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:46 smithi181 ceph-mon[32701]: 5.a scrub starts 2024-09-11T08:22:47.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:46 smithi181 ceph-mon[32701]: 5.a scrub ok 2024-09-11T08:22:47.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:46 smithi181 ceph-mon[32701]: 6.1d scrub starts 2024-09-11T08:22:47.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:46 smithi181 ceph-mon[32701]: 6.1d scrub ok 2024-09-11T08:22:47.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:46 smithi181 ceph-mon[32701]: 6.d scrub starts 2024-09-11T08:22:47.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:46 smithi181 ceph-mon[32701]: 6.d scrub ok 2024-09-11T08:22:47.165 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:22:47.204 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:47.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:47.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:22:47.299 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-11T08:22:47.305 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:22:47.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:47.384 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:47.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:22:47.485 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:47.516 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:47.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:47.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:47.618 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:22:47.657 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:47.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:47.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:22:47.762 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:22:47.795 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:47.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:47.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:22:47.899 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:22:47.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:47.938 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:47.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:22:47.957 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:47 smithi150 ceph-mon[29276]: 4.0 scrub starts 2024-09-11T08:22:47.957 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:47 smithi150 ceph-mon[29276]: 4.0 scrub ok 2024-09-11T08:22:47.957 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:47 smithi150 ceph-mon[29276]: 4.6 scrub starts 2024-09-11T08:22:47.957 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:47 smithi150 ceph-mon[29276]: 4.6 scrub ok 2024-09-11T08:22:47.957 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:47 smithi150 ceph-mon[29276]: pgmap v693: 148 pgs: 148 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 48 MiB/s wr, 772 op/s 2024-09-11T08:22:47.957 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:47 smithi150 ceph-mon[39612]: 4.0 scrub starts 2024-09-11T08:22:47.958 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:47 smithi150 ceph-mon[39612]: 4.0 scrub ok 2024-09-11T08:22:47.958 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:47 smithi150 ceph-mon[39612]: 4.6 scrub starts 2024-09-11T08:22:47.958 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:47 smithi150 ceph-mon[39612]: 4.6 scrub ok 2024-09-11T08:22:47.958 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:47 smithi150 ceph-mon[39612]: pgmap v693: 148 pgs: 148 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 48 MiB/s wr, 772 op/s 2024-09-11T08:22:48.033 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 47 16 85539 85523 113.706 78.1875 0.00826692 0.00878969 2024-09-11T08:22:48.039 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:22:48.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:47 smithi181 ceph-mon[32701]: 4.0 scrub starts 2024-09-11T08:22:48.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:47 smithi181 ceph-mon[32701]: 4.0 scrub ok 2024-09-11T08:22:48.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:47 smithi181 ceph-mon[32701]: 4.6 scrub starts 2024-09-11T08:22:48.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:47 smithi181 ceph-mon[32701]: 4.6 scrub ok 2024-09-11T08:22:48.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:47 smithi181 ceph-mon[32701]: pgmap v693: 148 pgs: 148 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 48 MiB/s wr, 772 op/s 2024-09-11T08:22:48.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:48.075 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:48.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:22:48.176 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:22:48.214 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:48.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:48.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:22:48.316 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:22:48.349 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:48.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:48.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:22:48.431 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:22:48.451 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:22:48.478 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:48.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:48.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:22:48.580 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:22:48.621 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:48.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:48.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:22:48.634 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:22:48 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:22:48] "GET /metrics HTTP/1.1" 200 224655 "" "Prometheus/2.43.0" 2024-09-11T08:22:48.723 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:22:48.767 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:48.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:48.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:22:48.869 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:48.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:48.908 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:48.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:48.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:22:48 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:22:49.009 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:22:49.034 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 48 16 86789 86773 112.963 78.125 0.0133108 0.00884747 2024-09-11T08:22:49.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:49.056 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:49.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:22:49.157 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:22:49.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:49.196 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:49.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:22:49.258 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:49 smithi150 ceph-mon[29276]: 5.4 scrub starts 2024-09-11T08:22:49.259 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:49 smithi150 ceph-mon[29276]: 5.4 scrub ok 2024-09-11T08:22:49.259 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:49 smithi150 ceph-mon[39612]: 5.4 scrub starts 2024-09-11T08:22:49.259 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:49 smithi150 ceph-mon[39612]: 5.4 scrub ok 2024-09-11T08:22:49.298 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:49.314 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:49 smithi181 ceph-mon[32701]: 5.4 scrub starts 2024-09-11T08:22:49.314 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:49 smithi181 ceph-mon[32701]: 5.4 scrub ok 2024-09-11T08:22:49.337 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:49.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:49.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:49.438 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:22:49.473 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:49.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:49.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:22:49.486 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:22:49.574 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:49.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:49.609 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:49.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:49.710 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:22:49.748 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:49.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:49.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:22:49.850 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:49.881 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:49.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:49.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:49.984 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:50.011 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:50.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:50.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:50.034 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 49 16 87974 87958 112.169 74.0625 0.00822728 0.00891047 2024-09-11T08:22:50.106 INFO:teuthology.orchestra.run.smithi150.stderr:nodeep-scrub is unset 2024-09-11T08:22:50.113 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:22:50.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:50.144 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:50.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:22:50.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:50 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:22:50.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:50 smithi150 ceph-mon[29276]: 3.17 scrub starts 2024-09-11T08:22:50.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:50 smithi150 ceph-mon[29276]: 3.17 scrub ok 2024-09-11T08:22:50.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:50 smithi150 ceph-mon[29276]: pgmap v694: 148 pgs: 148 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-09-11T08:22:50.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:50 smithi150 ceph-mon[29276]: 4.1a scrub starts 2024-09-11T08:22:50.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:50 smithi150 ceph-mon[29276]: 4.1a scrub ok 2024-09-11T08:22:50.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:50 smithi150 ceph-mon[29276]: 5.15 scrub starts 2024-09-11T08:22:50.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:50 smithi150 ceph-mon[29276]: 5.15 scrub ok 2024-09-11T08:22:50.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:50 smithi150 ceph-mon[29276]: 3.e scrub starts 2024-09-11T08:22:50.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:50 smithi150 ceph-mon[29276]: 3.e scrub ok 2024-09-11T08:22:50.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:50 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1771259898' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:22:50.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:50 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:22:50.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:50 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:22:50.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:50 smithi150 ceph-mon[39612]: 3.17 scrub starts 2024-09-11T08:22:50.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:50 smithi150 ceph-mon[39612]: 3.17 scrub ok 2024-09-11T08:22:50.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:50 smithi150 ceph-mon[39612]: pgmap v694: 148 pgs: 148 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-09-11T08:22:50.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:50 smithi150 ceph-mon[39612]: 4.1a scrub starts 2024-09-11T08:22:50.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:50 smithi150 ceph-mon[39612]: 4.1a scrub ok 2024-09-11T08:22:50.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:50 smithi150 ceph-mon[39612]: 5.15 scrub starts 2024-09-11T08:22:50.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:50 smithi150 ceph-mon[39612]: 5.15 scrub ok 2024-09-11T08:22:50.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:50 smithi150 ceph-mon[39612]: 3.e scrub starts 2024-09-11T08:22:50.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:50 smithi150 ceph-mon[39612]: 3.e scrub ok 2024-09-11T08:22:50.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:50 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1771259898' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:22:50.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:50 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:22:50.245 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:22:50.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:50.285 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:50.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:22:50.386 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:22:50.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:50.425 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:50.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:22:50.526 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:50.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:50.562 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:50.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:50.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:50 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:22:50.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:50 smithi181 ceph-mon[32701]: 3.17 scrub starts 2024-09-11T08:22:50.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:50 smithi181 ceph-mon[32701]: 3.17 scrub ok 2024-09-11T08:22:50.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:50 smithi181 ceph-mon[32701]: pgmap v694: 148 pgs: 148 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-09-11T08:22:50.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:50 smithi181 ceph-mon[32701]: 4.1a scrub starts 2024-09-11T08:22:50.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:50 smithi181 ceph-mon[32701]: 4.1a scrub ok 2024-09-11T08:22:50.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:50 smithi181 ceph-mon[32701]: 5.15 scrub starts 2024-09-11T08:22:50.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:50 smithi181 ceph-mon[32701]: 5.15 scrub ok 2024-09-11T08:22:50.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:50 smithi181 ceph-mon[32701]: 3.e scrub starts 2024-09-11T08:22:50.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:50 smithi181 ceph-mon[32701]: 3.e scrub ok 2024-09-11T08:22:50.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:50 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1771259898' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:22:50.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:50 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:22:50.663 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:50.708 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:50.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:50.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:50.810 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:22:50.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:50.853 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:50.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:22:50.954 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:50.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:50.993 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:50.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:51.034 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 50 16 89196 89180 111.453 76.375 0.0103683 0.00896793 2024-09-11T08:22:51.095 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:51.129 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:51.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:51.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:51.231 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:22:51.260 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:51.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:51.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:22:51.362 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:51.392 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:51.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:51.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:51.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:51 smithi150 ceph-mon[29276]: 6.6 scrub starts 2024-09-11T08:22:51.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:51 smithi150 ceph-mon[29276]: 6.6 scrub ok 2024-09-11T08:22:51.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:51 smithi150 ceph-mon[29276]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:22:51.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:51 smithi150 ceph-mon[29276]: Cluster is now healthy 2024-09-11T08:22:51.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:51 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:22:51.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:51 smithi150 ceph-mon[29276]: osdmap e324: 8 total, 8 up, 8 in 2024-09-11T08:22:51.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:51 smithi150 ceph-mon[29276]: 6.2 deep-scrub starts 2024-09-11T08:22:51.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:51 smithi150 ceph-mon[29276]: 6.2 deep-scrub ok 2024-09-11T08:22:51.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:51 smithi150 ceph-mon[29276]: 10.0 scrub starts 2024-09-11T08:22:51.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:51 smithi150 ceph-mon[29276]: osdmap e325: 8 total, 8 up, 8 in 2024-09-11T08:22:51.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:51 smithi150 ceph-mon[39612]: 6.6 scrub starts 2024-09-11T08:22:51.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:51 smithi150 ceph-mon[39612]: 6.6 scrub ok 2024-09-11T08:22:51.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:51 smithi150 ceph-mon[39612]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:22:51.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:51 smithi150 ceph-mon[39612]: Cluster is now healthy 2024-09-11T08:22:51.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:51 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:22:51.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:51 smithi150 ceph-mon[39612]: osdmap e324: 8 total, 8 up, 8 in 2024-09-11T08:22:51.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:51 smithi150 ceph-mon[39612]: 6.2 deep-scrub starts 2024-09-11T08:22:51.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:51 smithi150 ceph-mon[39612]: 6.2 deep-scrub ok 2024-09-11T08:22:51.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:51 smithi150 ceph-mon[39612]: 10.0 scrub starts 2024-09-11T08:22:51.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:51 smithi150 ceph-mon[39612]: osdmap e325: 8 total, 8 up, 8 in 2024-09-11T08:22:51.494 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:22:51.531 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:51.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:51.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:22:51.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:51 smithi181 ceph-mon[32701]: 6.6 scrub starts 2024-09-11T08:22:51.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:51 smithi181 ceph-mon[32701]: 6.6 scrub ok 2024-09-11T08:22:51.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:51 smithi181 ceph-mon[32701]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:22:51.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:51 smithi181 ceph-mon[32701]: Cluster is now healthy 2024-09-11T08:22:51.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:51 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:22:51.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:51 smithi181 ceph-mon[32701]: osdmap e324: 8 total, 8 up, 8 in 2024-09-11T08:22:51.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:51 smithi181 ceph-mon[32701]: 6.2 deep-scrub starts 2024-09-11T08:22:51.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:51 smithi181 ceph-mon[32701]: 6.2 deep-scrub ok 2024-09-11T08:22:51.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:51 smithi181 ceph-mon[32701]: 10.0 scrub starts 2024-09-11T08:22:51.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:51 smithi181 ceph-mon[32701]: osdmap e325: 8 total, 8 up, 8 in 2024-09-11T08:22:51.632 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:51.665 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:51.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:51.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:51.767 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:51.801 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:51.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:51.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:51.903 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:22:51.925 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:51.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:51.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:22:52.026 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:52.034 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 51 16 90185 90169 110.48 61.8125 0.173048 0.00902193 2024-09-11T08:22:52.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:52.065 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:52.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:52.167 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:22:52.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:52.201 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:52.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:22:52.302 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:52.328 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:52.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:52.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:52.429 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:22:52.451 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:52.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:52.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:22:52.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:52 smithi150 ceph-mon[29276]: 6.13 deep-scrub starts 2024-09-11T08:22:52.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:52 smithi150 ceph-mon[29276]: 6.13 deep-scrub ok 2024-09-11T08:22:52.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:52 smithi150 ceph-mon[29276]: pgmap v697: 148 pgs: 148 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 62 MiB/s wr, 1k op/s 2024-09-11T08:22:52.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:52 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:22:52.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:52 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:22:52.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:52 smithi150 ceph-mon[29276]: 6.1c scrub starts 2024-09-11T08:22:52.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:52 smithi150 ceph-mon[29276]: 6.1c scrub ok 2024-09-11T08:22:52.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:52 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:22:52.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:52 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:22:52.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:52 smithi150 ceph-mon[29276]: 10.0 scrub ok 2024-09-11T08:22:52.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:52 smithi150 ceph-mon[39612]: 6.13 deep-scrub starts 2024-09-11T08:22:52.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:52 smithi150 ceph-mon[39612]: 6.13 deep-scrub ok 2024-09-11T08:22:52.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:52 smithi150 ceph-mon[39612]: pgmap v697: 148 pgs: 148 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 62 MiB/s wr, 1k op/s 2024-09-11T08:22:52.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:52 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:22:52.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:52 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:22:52.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:52 smithi150 ceph-mon[39612]: 6.1c scrub starts 2024-09-11T08:22:52.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:52 smithi150 ceph-mon[39612]: 6.1c scrub ok 2024-09-11T08:22:52.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:52 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:22:52.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:52 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:22:52.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:52 smithi150 ceph-mon[39612]: 10.0 scrub ok 2024-09-11T08:22:52.553 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:22:52.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:52 smithi181 ceph-mon[32701]: 6.13 deep-scrub starts 2024-09-11T08:22:52.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:52 smithi181 ceph-mon[32701]: 6.13 deep-scrub ok 2024-09-11T08:22:52.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:52 smithi181 ceph-mon[32701]: pgmap v697: 148 pgs: 148 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 62 MiB/s wr, 1k op/s 2024-09-11T08:22:52.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:52 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:22:52.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:52 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:22:52.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:52 smithi181 ceph-mon[32701]: 6.1c scrub starts 2024-09-11T08:22:52.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:52 smithi181 ceph-mon[32701]: 6.1c scrub ok 2024-09-11T08:22:52.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:52 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:22:52.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:52 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:22:52.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:52 smithi181 ceph-mon[32701]: 10.0 scrub ok 2024-09-11T08:22:52.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:52.586 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:52.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:22:52.687 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:22:52.709 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:52.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:52.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:22:52.811 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:52.843 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:52.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:52.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:52.945 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:22:52.979 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:52.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:52.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:22:53.034 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 52 16 91131 91115 109.492 59.125 0.010194 0.00912853 2024-09-11T08:22:53.081 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:22:53.119 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:53.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:53.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:22:53.221 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:53.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:53 smithi150 ceph-mon[29276]: 4.19 scrub starts 2024-09-11T08:22:53.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:53 smithi150 ceph-mon[29276]: 4.19 scrub ok 2024-09-11T08:22:53.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:53 smithi150 ceph-mon[29276]: osdmap e326: 8 total, 8 up, 8 in 2024-09-11T08:22:53.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:53 smithi150 ceph-mon[39612]: 4.19 scrub starts 2024-09-11T08:22:53.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:53 smithi150 ceph-mon[39612]: 4.19 scrub ok 2024-09-11T08:22:53.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:53 smithi150 ceph-mon[39612]: osdmap e326: 8 total, 8 up, 8 in 2024-09-11T08:22:53.253 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:53.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:53.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:53.328 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:22:53.355 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:22:53.385 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:53.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:53.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:22:53.487 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:22:53.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:53.520 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:53.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:22:53.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:53 smithi181 ceph-mon[32701]: 4.19 scrub starts 2024-09-11T08:22:53.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:53 smithi181 ceph-mon[32701]: 4.19 scrub ok 2024-09-11T08:22:53.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:53 smithi181 ceph-mon[32701]: osdmap e326: 8 total, 8 up, 8 in 2024-09-11T08:22:53.622 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:22:53.654 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:53.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:53.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:22:53.756 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:22:53.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:53.794 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:53.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:22:53.895 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:53.939 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:53.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:53.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:54.034 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 53 16 92366 92350 108.882 77.1875 0.00560279 0.00917909 2024-09-11T08:22:54.041 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:54.113 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:54.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:54.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:54.215 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:54.245 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:54.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:54.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:54.347 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:54.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:54.381 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:54.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:54.389 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:22:54.389 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:22:54.390 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:54 smithi150 ceph-mon[29276]: pgmap v699: 148 pgs: 148 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 92 MiB/s wr, 1.47k op/s 2024-09-11T08:22:54.390 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:54 smithi150 ceph-mon[29276]: 6.1f scrub starts 2024-09-11T08:22:54.391 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:54 smithi150 ceph-mon[29276]: 6.1f scrub ok 2024-09-11T08:22:54.391 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:54 smithi150 ceph-mon[39612]: pgmap v699: 148 pgs: 148 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 92 MiB/s wr, 1.47k op/s 2024-09-11T08:22:54.391 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:54 smithi150 ceph-mon[39612]: 6.1f scrub starts 2024-09-11T08:22:54.391 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:54 smithi150 ceph-mon[39612]: 6.1f scrub ok 2024-09-11T08:22:54.418 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:22:54.419 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:22:54.450 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:22:54.451 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:22:54.482 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:22:54.487 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:22:54.487 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:22:54.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:54.512 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:54.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:22:54.520 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:22:54.520 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:22:54.554 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:22:54.555 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:22:54.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:54 smithi181 ceph-mon[32701]: pgmap v699: 148 pgs: 148 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 92 MiB/s wr, 1.47k op/s 2024-09-11T08:22:54.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:54 smithi181 ceph-mon[32701]: 6.1f scrub starts 2024-09-11T08:22:54.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:54 smithi181 ceph-mon[32701]: 6.1f scrub ok 2024-09-11T08:22:54.589 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:22:54.589 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:22:54.612 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:22:54.620 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:22:54.621 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:22:54.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:54.662 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:54.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:22:54.763 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:22:54.799 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:54.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:54.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:22:54.901 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:22:54.937 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:54.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:54.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:22:55.034 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 54 16 93553 93537 108.24 74.1875 0.00721433 0.00923429 2024-09-11T08:22:55.039 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:22:55.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:55.078 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:55.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:22:55.179 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:22:55.214 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:55.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:55.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:22:55.316 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:55.334 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:55 smithi150 ceph-mon[29276]: 6.19 scrub starts 2024-09-11T08:22:55.334 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:55 smithi150 ceph-mon[29276]: 6.19 scrub ok 2024-09-11T08:22:55.334 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:55 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1230449867' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:22:55.335 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:55 smithi150 ceph-mon[39612]: 6.19 scrub starts 2024-09-11T08:22:55.335 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:55 smithi150 ceph-mon[39612]: 6.19 scrub ok 2024-09-11T08:22:55.335 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:55 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1230449867' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:22:55.342 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T08:22:55.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:55.354 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:55.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:55.455 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:22:55.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:55.531 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:55.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:22:55.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:55 smithi181 ceph-mon[32701]: 6.19 scrub starts 2024-09-11T08:22:55.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:55 smithi181 ceph-mon[32701]: 6.19 scrub ok 2024-09-11T08:22:55.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:55 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1230449867' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:22:55.574 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:22:55 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:22:55] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:22:55.631 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:55.666 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:55.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:55.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:55.767 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:22:55.801 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:55.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:55.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:22:55.905 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:55.940 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:55.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:55.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:56.035 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 55 16 94701 94685 107.576 71.75 0.00937068 0.009291 2024-09-11T08:22:56.041 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:22:56.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:56.075 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:56.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:22:56.176 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:56.207 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:56.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:56.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:56.277 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:56 smithi150 ceph-mon[39612]: 3.b scrub starts 2024-09-11T08:22:56.277 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:56 smithi150 ceph-mon[39612]: 3.b scrub ok 2024-09-11T08:22:56.277 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:56 smithi150 ceph-mon[39612]: pgmap v700: 148 pgs: 148 active+clean; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 58 MiB/s wr, 934 op/s 2024-09-11T08:22:56.277 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:56 smithi150 ceph-mon[39612]: 10.f scrub starts 2024-09-11T08:22:56.277 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:56 smithi150 ceph-mon[39612]: 6.7 scrub starts 2024-09-11T08:22:56.277 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:56 smithi150 ceph-mon[39612]: 6.7 scrub ok 2024-09-11T08:22:56.277 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:56 smithi150 ceph-mon[39612]: osdmap e327: 8 total, 8 up, 8 in 2024-09-11T08:22:56.278 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:56 smithi150 ceph-mon[29276]: 3.b scrub starts 2024-09-11T08:22:56.278 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:56 smithi150 ceph-mon[29276]: 3.b scrub ok 2024-09-11T08:22:56.278 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:56 smithi150 ceph-mon[29276]: pgmap v700: 148 pgs: 148 active+clean; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 58 MiB/s wr, 934 op/s 2024-09-11T08:22:56.278 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:56 smithi150 ceph-mon[29276]: 10.f scrub starts 2024-09-11T08:22:56.278 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:56 smithi150 ceph-mon[29276]: 6.7 scrub starts 2024-09-11T08:22:56.278 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:56 smithi150 ceph-mon[29276]: 6.7 scrub ok 2024-09-11T08:22:56.278 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:56 smithi150 ceph-mon[29276]: osdmap e327: 8 total, 8 up, 8 in 2024-09-11T08:22:56.308 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:56.337 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:56.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:56.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:56.439 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:22:56.478 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:56.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:56.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:22:56.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:56 smithi181 ceph-mon[32701]: 3.b scrub starts 2024-09-11T08:22:56.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:56 smithi181 ceph-mon[32701]: 3.b scrub ok 2024-09-11T08:22:56.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:56 smithi181 ceph-mon[32701]: pgmap v700: 148 pgs: 148 active+clean; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 58 MiB/s wr, 934 op/s 2024-09-11T08:22:56.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:56 smithi181 ceph-mon[32701]: 10.f scrub starts 2024-09-11T08:22:56.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:56 smithi181 ceph-mon[32701]: 6.7 scrub starts 2024-09-11T08:22:56.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:56 smithi181 ceph-mon[32701]: 6.7 scrub ok 2024-09-11T08:22:56.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:56 smithi181 ceph-mon[32701]: osdmap e327: 8 total, 8 up, 8 in 2024-09-11T08:22:56.581 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:22:56.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:56.619 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:56.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:22:56.721 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:22:56.763 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:56.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:56.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:22:56.865 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:22:56.899 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:56.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:56.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:22:57.000 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:57.024 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:57.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:57.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:57.035 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 56 16 95770 95754 106.848 66.8125 0.0102189 0.00934915 2024-09-11T08:22:57.125 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:57.157 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:57.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:57.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:57.258 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:22:57.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:57.293 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:57.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:22:57.394 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:22:57.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:57.430 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:57.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:22:57.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:57 smithi150 ceph-mon[29276]: 4.c scrub starts 2024-09-11T08:22:57.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:57 smithi150 ceph-mon[29276]: 4.c scrub ok 2024-09-11T08:22:57.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:57 smithi150 ceph-mon[29276]: 5.1b scrub starts 2024-09-11T08:22:57.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:57 smithi150 ceph-mon[29276]: 5.1b scrub ok 2024-09-11T08:22:57.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:57 smithi150 ceph-mon[29276]: 10.a scrub starts 2024-09-11T08:22:57.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:57 smithi150 ceph-mon[39612]: 4.c scrub starts 2024-09-11T08:22:57.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:57 smithi150 ceph-mon[39612]: 4.c scrub ok 2024-09-11T08:22:57.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:57 smithi150 ceph-mon[39612]: 5.1b scrub starts 2024-09-11T08:22:57.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:57 smithi150 ceph-mon[39612]: 5.1b scrub ok 2024-09-11T08:22:57.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:57 smithi150 ceph-mon[39612]: 10.a scrub starts 2024-09-11T08:22:57.532 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:57.566 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:57.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:57.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:57.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:57 smithi181 ceph-mon[32701]: 4.c scrub starts 2024-09-11T08:22:57.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:57 smithi181 ceph-mon[32701]: 4.c scrub ok 2024-09-11T08:22:57.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:57 smithi181 ceph-mon[32701]: 5.1b scrub starts 2024-09-11T08:22:57.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:57 smithi181 ceph-mon[32701]: 5.1b scrub ok 2024-09-11T08:22:57.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:57 smithi181 ceph-mon[32701]: 10.a scrub starts 2024-09-11T08:22:57.668 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:57.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:57.692 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:57.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:57.793 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:57.827 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:57.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:57.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:57.929 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:22:57.972 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:57.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:57.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:22:58.035 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 57 16 96607 96591 105.892 52.3125 0.00996182 0.00943896 2024-09-11T08:22:58.041 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T08:22:58.074 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:22:58.106 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:58.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:58.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:22:58.164 INFO:teuthology.orchestra.run.smithi150.stderr:set osd.5 primary-affinity to 0 (802) 2024-09-11T08:22:58.208 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:58.235 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:58.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:58.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:58.312 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-11T08:22:58.312 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 20, 2024-09-11T08:22:58.312 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 600, 2024-09-11T08:22:58.312 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [ 2024-09-11T08:22:58.312 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.312 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.15087.0:54333 8.f e175/159 8:f9667130:::benchmark_data_smithi150_107646_object54332:head v 175'3422, mlcod=175'3420)", 2024-09-11T08:22:58.313 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:14:39.758366+0000", 2024-09-11T08:22:58.313 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 497.456258166, 2024-09-11T08:22:58.313 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.098538254000000006, 2024-09-11T08:22:58.313 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:22:58.313 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:22:58.313 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:22:58.313 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.313 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:22:58.313 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:14:39.758366+0000", 2024-09-11T08:22:58.313 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:58.313 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.313 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.313 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:22:58.313 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:14:39.758366+0000", 2024-09-11T08:22:58.314 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:58.314 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.314 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.314 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:22:58.314 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:14:39.758366+0000", 2024-09-11T08:22:58.314 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:22:58.314 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.314 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.314 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:22:58.314 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:14:39.758411+0000", 2024-09-11T08:22:58.314 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.5618000000000003e-05 2024-09-11T08:22:58.314 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.314 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.315 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:22:58.315 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:14:39.758422+0000", 2024-09-11T08:22:58.315 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0407999999999999e-05 2024-09-11T08:22:58.315 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.315 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.315 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:22:58.315 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:14:39.758428+0000", 2024-09-11T08:22:58.315 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.2809999999999997e-06 2024-09-11T08:22:58.315 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.315 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.315 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:58.315 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:14:39.758614+0000", 2024-09-11T08:22:58.315 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000186069 2024-09-11T08:22:58.316 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.316 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.316 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:22:58.316 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:14:39.758642+0000", 2024-09-11T08:22:58.316 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.7387000000000001e-05 2024-09-11T08:22:58.316 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.316 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.316 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:22:58.316 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:14:39.856870+0000", 2024-09-11T08:22:58.316 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.098228812999999998 2024-09-11T08:22:58.316 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.316 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.316 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:22:58.317 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:14:39.856905+0000", 2024-09-11T08:22:58.317 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.4248999999999997e-05 2024-09-11T08:22:58.317 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:58.317 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:22:58.317 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:58.317 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.317 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.317 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14451.0:551 4.14 4:2c0c0c23:reshard::reshard.0000000001:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-11T08:22:58.317 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:17:37.174916+0000", 2024-09-11T08:22:58.317 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 320.03970839700003, 2024-09-11T08:22:58.317 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.383420284, 2024-09-11T08:22:58.317 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:22:58.317 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:22:58.318 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:22:58.318 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14451", 2024-09-11T08:22:58.318 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/3411048405", 2024-09-11T08:22:58.318 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 551 2024-09-11T08:22:58.318 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.318 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:22:58.318 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.318 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:22:58.318 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:37.174916+0000", 2024-09-11T08:22:58.318 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:58.318 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.318 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.318 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:22:58.318 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:37.174916+0000", 2024-09-11T08:22:58.319 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:58.319 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.319 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.319 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:22:58.319 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:37.174914+0000", 2024-09-11T08:22:58.319 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999976 2024-09-11T08:22:58.319 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.319 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.319 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:22:58.319 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:37.174925+0000", 2024-09-11T08:22:58.319 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1334e-05 2024-09-11T08:22:58.319 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.319 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.319 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:22:58.319 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:37.174944+0000", 2024-09-11T08:22:58.320 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.9320000000000001e-05 2024-09-11T08:22:58.320 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.320 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.320 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:22:58.320 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:37.174952+0000", 2024-09-11T08:22:58.320 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.8460000000000008e-06 2024-09-11T08:22:58.320 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.320 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.320 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:58.320 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:37.175008+0000", 2024-09-11T08:22:58.320 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.5556999999999997e-05 2024-09-11T08:22:58.320 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.320 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.320 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:22:58.321 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:37.175016+0000", 2024-09-11T08:22:58.321 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.5550000000000003e-06 2024-09-11T08:22:58.321 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.321 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.321 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:58.321 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:37.554585+0000", 2024-09-11T08:22:58.321 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.37956889799999999 2024-09-11T08:22:58.321 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.321 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.321 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:22:58.321 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:37.554780+0000", 2024-09-11T08:22:58.321 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00019447900000000001 2024-09-11T08:22:58.321 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.321 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.322 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 3,7", 2024-09-11T08:22:58.322 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:37.555004+0000", 2024-09-11T08:22:58.322 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00022432600000000001 2024-09-11T08:22:58.322 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.322 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.322 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:22:58.322 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:37.558110+0000", 2024-09-11T08:22:58.322 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.003105675 2024-09-11T08:22:58.322 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.322 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.322 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:22:58.322 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:37.558156+0000", 2024-09-11T08:22:58.322 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.6628999999999997e-05 2024-09-11T08:22:58.322 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.323 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.323 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:22:58.323 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:37.558206+0000", 2024-09-11T08:22:58.323 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.9336999999999999e-05 2024-09-11T08:22:58.323 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.323 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.323 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:22:58.323 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:37.558297+0000", 2024-09-11T08:22:58.323 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.1520999999999993e-05 2024-09-11T08:22:58.323 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.323 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.323 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:22:58.323 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:37.558336+0000", 2024-09-11T08:22:58.323 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.9298000000000002e-05 2024-09-11T08:22:58.323 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:58.324 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:22:58.324 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:58.324 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.324 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.324 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.14451.0:560 4.19 4:9ed7f5b7:::obj_delete_at_hint.0000000004:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-11T08:22:58.324 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:17:37.426731+0000", 2024-09-11T08:22:58.324 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 319.78789367399997, 2024-09-11T08:22:58.324 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.130209783, 2024-09-11T08:22:58.324 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:22:58.324 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:22:58.324 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:22:58.324 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.14451", 2024-09-11T08:22:58.324 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/3411048405", 2024-09-11T08:22:58.324 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 560 2024-09-11T08:22:58.325 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.325 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:22:58.325 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.325 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:22:58.325 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:37.426731+0000", 2024-09-11T08:22:58.325 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:58.325 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.325 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.325 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:22:58.325 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:37.426731+0000", 2024-09-11T08:22:58.325 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:58.326 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.326 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.326 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:22:58.326 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:37.426729+0000", 2024-09-11T08:22:58.326 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:22:58.326 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.326 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.326 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:22:58.326 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:37.426739+0000", 2024-09-11T08:22:58.326 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.9310000000000003e-06 2024-09-11T08:22:58.326 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.326 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.327 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:22:58.327 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:37.426756+0000", 2024-09-11T08:22:58.327 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.7054999999999998e-05 2024-09-11T08:22:58.327 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.327 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.327 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:22:58.327 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:37.426764+0000", 2024-09-11T08:22:58.327 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.8909999999999995e-06 2024-09-11T08:22:58.327 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.327 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.327 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:58.327 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:37.426827+0000", 2024-09-11T08:22:58.327 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.2861999999999995e-05 2024-09-11T08:22:58.327 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.328 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.328 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:22:58.328 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:37.426838+0000", 2024-09-11T08:22:58.328 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1273e-05 2024-09-11T08:22:58.328 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.328 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.328 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:58.328 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:37.554804+0000", 2024-09-11T08:22:58.328 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.12796565600000001 2024-09-11T08:22:58.328 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.328 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.328 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:22:58.328 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:37.555485+0000", 2024-09-11T08:22:58.328 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00068077999999999997 2024-09-11T08:22:58.329 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.329 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.329 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 0,3", 2024-09-11T08:22:58.329 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:37.555637+0000", 2024-09-11T08:22:58.329 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00015271699999999999 2024-09-11T08:22:58.329 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.329 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.329 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:22:58.329 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:37.556540+0000", 2024-09-11T08:22:58.329 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00090217800000000001 2024-09-11T08:22:58.329 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.329 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.329 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:22:58.329 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:37.556548+0000", 2024-09-11T08:22:58.330 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.833e-06 2024-09-11T08:22:58.330 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.330 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.330 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:22:58.330 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:37.556857+0000", 2024-09-11T08:22:58.330 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00030850099999999998 2024-09-11T08:22:58.330 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.330 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.330 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:22:58.330 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:37.556905+0000", 2024-09-11T08:22:58.330 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.7790000000000002e-05 2024-09-11T08:22:58.330 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.330 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.330 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:22:58.331 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:37.556941+0000", 2024-09-11T08:22:58.331 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.5975e-05 2024-09-11T08:22:58.331 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:58.331 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:22:58.331 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:58.331 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.331 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.331 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15369.0:96106 9.6 9:6057837d:::benchmark_data_smithi150_133616_object96105:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-11T08:22:58.331 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:19:08.607644+0000", 2024-09-11T08:22:58.331 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 228.60698097299999, 2024-09-11T08:22:58.331 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.177517404, 2024-09-11T08:22:58.331 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:22:58.331 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:22:58.331 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:22:58.332 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15369", 2024-09-11T08:22:58.332 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:22:58.332 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 96106 2024-09-11T08:22:58.332 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.332 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:22:58.332 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.332 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:22:58.332 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.607644+0000", 2024-09-11T08:22:58.332 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:58.332 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.332 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.332 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:22:58.332 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.607644+0000", 2024-09-11T08:22:58.333 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:58.333 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.333 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.333 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:22:58.333 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.607642+0000", 2024-09-11T08:22:58.333 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:22:58.333 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.333 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.333 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:22:58.333 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.607696+0000", 2024-09-11T08:22:58.333 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.4276e-05 2024-09-11T08:22:58.333 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.333 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.333 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:22:58.334 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.607707+0000", 2024-09-11T08:22:58.334 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1105e-05 2024-09-11T08:22:58.334 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.334 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.334 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:22:58.334 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.607713+0000", 2024-09-11T08:22:58.334 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.5949999999999998e-06 2024-09-11T08:22:58.334 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.334 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.334 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:58.334 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.607941+0000", 2024-09-11T08:22:58.334 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000228361 2024-09-11T08:22:58.334 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.334 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.335 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:22:58.335 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.607966+0000", 2024-09-11T08:22:58.335 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.4641e-05 2024-09-11T08:22:58.335 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.335 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.335 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:58.335 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.588318+0000", 2024-09-11T08:22:58.335 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.98035270299999999 2024-09-11T08:22:58.335 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.335 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.335 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:22:58.335 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.588330+0000", 2024-09-11T08:22:58.335 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.164e-05 2024-09-11T08:22:58.335 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.336 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.336 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:58.336 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.761111+0000", 2024-09-11T08:22:58.336 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1727814190000001 2024-09-11T08:22:58.336 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.336 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.336 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:22:58.336 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.765416+0000", 2024-09-11T08:22:58.336 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.004304296 2024-09-11T08:22:58.336 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.336 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.337 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 1,5,6", 2024-09-11T08:22:58.337 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.765590+0000", 2024-09-11T08:22:58.337 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00017429300000000001 2024-09-11T08:22:58.337 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.337 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.337 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:22:58.337 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.784410+0000", 2024-09-11T08:22:58.337 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.018820027 2024-09-11T08:22:58.337 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.337 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.337 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:22:58.337 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.784473+0000", 2024-09-11T08:22:58.337 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.3227999999999998e-05 2024-09-11T08:22:58.338 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.338 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.338 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:22:58.338 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.784534+0000", 2024-09-11T08:22:58.338 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.1198000000000006e-05 2024-09-11T08:22:58.340 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.340 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.340 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:22:58.340 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.784919+0000", 2024-09-11T08:22:58.340 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00038449300000000002 2024-09-11T08:22:58.340 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.341 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.341 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:22:58.341 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.785036+0000", 2024-09-11T08:22:58.341 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00011725 2024-09-11T08:22:58.341 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.341 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.341 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:22:58.341 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.785161+0000", 2024-09-11T08:22:58.341 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012483399999999999 2024-09-11T08:22:58.341 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:58.341 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:22:58.341 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:58.341 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.341 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.341 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15369.0:96112 9.6 9:61fdd20f:::benchmark_data_smithi150_133616_object96111:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-11T08:22:58.342 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:19:08.620746+0000", 2024-09-11T08:22:58.342 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 228.593878553, 2024-09-11T08:22:58.342 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.1646099950000002, 2024-09-11T08:22:58.342 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:22:58.342 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:22:58.342 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:22:58.342 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15369", 2024-09-11T08:22:58.342 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:22:58.342 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 96112 2024-09-11T08:22:58.342 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.342 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:22:58.342 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.342 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:22:58.343 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.620746+0000", 2024-09-11T08:22:58.343 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:58.343 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.343 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.343 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:22:58.343 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.620746+0000", 2024-09-11T08:22:58.343 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:58.343 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.344 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.344 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:22:58.344 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.620744+0000", 2024-09-11T08:22:58.344 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:22:58.344 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.344 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.344 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:22:58.344 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.620796+0000", 2024-09-11T08:22:58.344 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.1513e-05 2024-09-11T08:22:58.344 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.344 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.345 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:22:58.345 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.620812+0000", 2024-09-11T08:22:58.345 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.6291e-05 2024-09-11T08:22:58.345 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.345 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.345 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:22:58.345 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.620820+0000", 2024-09-11T08:22:58.345 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.242e-06 2024-09-11T08:22:58.345 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.345 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.345 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:58.345 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.620881+0000", 2024-09-11T08:22:58.345 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.0772e-05 2024-09-11T08:22:58.345 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.346 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.346 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:22:58.346 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.620902+0000", 2024-09-11T08:22:58.346 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.1483000000000001e-05 2024-09-11T08:22:58.346 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.346 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.346 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:58.346 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.590175+0000", 2024-09-11T08:22:58.346 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.969272519 2024-09-11T08:22:58.346 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.346 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.346 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:22:58.346 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.590185+0000", 2024-09-11T08:22:58.347 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.6190000000000006e-06 2024-09-11T08:22:58.347 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.347 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.347 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:58.347 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.769061+0000", 2024-09-11T08:22:58.347 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1788760979999999 2024-09-11T08:22:58.347 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.347 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.347 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:22:58.347 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.769199+0000", 2024-09-11T08:22:58.347 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00013856099999999999 2024-09-11T08:22:58.347 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.347 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.347 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 1,5,6", 2024-09-11T08:22:58.347 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.769362+0000", 2024-09-11T08:22:58.348 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00016318300000000001 2024-09-11T08:22:58.348 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.348 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.348 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:22:58.348 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.784443+0000", 2024-09-11T08:22:58.348 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.015080464999999999 2024-09-11T08:22:58.348 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.348 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.348 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:22:58.348 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.784503+0000", 2024-09-11T08:22:58.348 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.0556000000000001e-05 2024-09-11T08:22:58.349 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.349 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.349 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:22:58.349 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.784907+0000", 2024-09-11T08:22:58.349 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00040319299999999999 2024-09-11T08:22:58.349 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.349 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.349 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:22:58.349 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.785170+0000", 2024-09-11T08:22:58.350 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00026354599999999997 2024-09-11T08:22:58.350 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.350 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.350 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:22:58.350 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.785255+0000", 2024-09-11T08:22:58.350 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.4587999999999997e-05 2024-09-11T08:22:58.350 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.350 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.350 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:22:58.350 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.785356+0000", 2024-09-11T08:22:58.350 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00010129299999999999 2024-09-11T08:22:58.350 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:58.350 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:22:58.350 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:58.351 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.351 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.351 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.24320.0:1794 5.d 5:b4812045:::notify.4:head [watch ping cookie 140425499249504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e259)", 2024-09-11T08:22:58.351 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:19:08.688284+0000", 2024-09-11T08:22:58.351 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 228.52634052600001, 2024-09-11T08:22:58.351 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.92840292300000005, 2024-09-11T08:22:58.351 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:22:58.351 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "started", 2024-09-11T08:22:58.351 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:22:58.351 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.24320", 2024-09-11T08:22:58.351 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1503848706", 2024-09-11T08:22:58.351 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 1794 2024-09-11T08:22:58.351 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.351 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:22:58.352 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.352 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:22:58.352 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.688284+0000", 2024-09-11T08:22:58.352 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:58.352 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.352 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.352 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:22:58.352 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.688284+0000", 2024-09-11T08:22:58.352 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:58.352 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.352 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.352 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:22:58.352 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.688281+0000", 2024-09-11T08:22:58.352 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999967 2024-09-11T08:22:58.353 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.353 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.353 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:22:58.353 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.688311+0000", 2024-09-11T08:22:58.353 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.9952000000000001e-05 2024-09-11T08:22:58.353 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.353 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.353 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:22:58.353 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.688331+0000", 2024-09-11T08:22:58.353 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.0709000000000001e-05 2024-09-11T08:22:58.353 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.353 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.353 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:22:58.353 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.688338+0000", 2024-09-11T08:22:58.354 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.5200000000000003e-06 2024-09-11T08:22:58.354 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.354 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.354 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:58.354 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.688440+0000", 2024-09-11T08:22:58.354 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000102241 2024-09-11T08:22:58.354 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.354 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.354 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:22:58.354 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.688453+0000", 2024-09-11T08:22:58.354 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2767e-05 2024-09-11T08:22:58.354 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.354 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.354 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:58.355 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.611369+0000", 2024-09-11T08:22:58.355 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.92291567600000002 2024-09-11T08:22:58.355 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.355 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.355 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for degraded object", 2024-09-11T08:22:58.355 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.611669+0000", 2024-09-11T08:22:58.355 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00030050399999999999 2024-09-11T08:22:58.355 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.355 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.355 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:58.355 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.616504+0000", 2024-09-11T08:22:58.355 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0048346969999999998 2024-09-11T08:22:58.355 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.355 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.355 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:22:58.356 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.616590+0000", 2024-09-11T08:22:58.356 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.5826000000000001e-05 2024-09-11T08:22:58.356 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.356 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.356 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:22:58.356 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.616687+0000", 2024-09-11T08:22:58.356 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.7417000000000006e-05 2024-09-11T08:22:58.356 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:58.356 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:22:58.356 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:58.356 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.356 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.356 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.24320.0:1802 5.d 5:b4812045:::notify.4:head [watch reconnect cookie 140425499249504 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-11T08:22:58.357 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:19:08.688747+0000", 2024-09-11T08:22:58.357 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 228.52587740499999, 2024-09-11T08:22:58.357 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.92997097399999995, 2024-09-11T08:22:58.357 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:22:58.357 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "started", 2024-09-11T08:22:58.357 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:22:58.357 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.24320", 2024-09-11T08:22:58.357 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1503848706", 2024-09-11T08:22:58.357 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 1802 2024-09-11T08:22:58.357 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.357 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:22:58.357 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.358 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:22:58.358 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.688747+0000", 2024-09-11T08:22:58.358 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:58.358 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.358 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.358 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:22:58.358 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.688747+0000", 2024-09-11T08:22:58.358 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:58.358 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.358 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.358 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:22:58.358 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.688746+0000", 2024-09-11T08:22:58.358 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:22:58.358 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.358 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.358 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:22:58.359 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.688752+0000", 2024-09-11T08:22:58.359 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.8139999999999997e-06 2024-09-11T08:22:58.359 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.359 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.359 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:22:58.359 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.688766+0000", 2024-09-11T08:22:58.359 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4270000000000001e-05 2024-09-11T08:22:58.359 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.359 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.359 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:22:58.359 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.688772+0000", 2024-09-11T08:22:58.359 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.1120000000000001e-06 2024-09-11T08:22:58.359 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.360 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.360 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:58.360 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.688823+0000", 2024-09-11T08:22:58.360 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.0930999999999997e-05 2024-09-11T08:22:58.360 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.360 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.360 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:22:58.360 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.688829+0000", 2024-09-11T08:22:58.360 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.711e-06 2024-09-11T08:22:58.360 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.360 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.360 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:58.360 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.611693+0000", 2024-09-11T08:22:58.361 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.92286374900000001 2024-09-11T08:22:58.361 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.361 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.361 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for degraded object", 2024-09-11T08:22:58.361 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.611746+0000", 2024-09-11T08:22:58.361 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.2964999999999999e-05 2024-09-11T08:22:58.361 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.361 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.361 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:58.361 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.616724+0000", 2024-09-11T08:22:58.362 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0049778629999999999 2024-09-11T08:22:58.362 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.362 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.362 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:22:58.362 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.616936+0000", 2024-09-11T08:22:58.362 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000212563 2024-09-11T08:22:58.363 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.363 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.363 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:22:58.363 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.618718+0000", 2024-09-11T08:22:58.363 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0017820360000000001 2024-09-11T08:22:58.363 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:58.364 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:22:58.364 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:58.364 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.364 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.364 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.15369.0:206706 9.2 e297/246 9:487dc9b7:::benchmark_data_smithi150_133616_object91100:head v 297'12895, mlcod=297'12893)", 2024-09-11T08:22:58.364 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:21:01.166265+0000", 2024-09-11T08:22:58.365 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 116.048359964, 2024-09-11T08:22:58.365 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.18571310899999999, 2024-09-11T08:22:58.365 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:22:58.365 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:22:58.365 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:22:58.366 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.366 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:22:58.366 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.166265+0000", 2024-09-11T08:22:58.366 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:58.366 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.366 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.367 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:22:58.367 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.166265+0000", 2024-09-11T08:22:58.367 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:58.367 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.367 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.367 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:22:58.368 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.166263+0000", 2024-09-11T08:22:58.368 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:22:58.368 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.368 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.368 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:22:58.369 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.166273+0000", 2024-09-11T08:22:58.369 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.2359999999999993e-06 2024-09-11T08:22:58.369 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.369 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.369 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:22:58.369 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.166287+0000", 2024-09-11T08:22:58.370 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4613999999999999e-05 2024-09-11T08:22:58.370 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.370 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.370 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:22:58.370 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.166294+0000", 2024-09-11T08:22:58.371 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.0759999999999997e-06 2024-09-11T08:22:58.371 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.371 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.371 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:58.371 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.166364+0000", 2024-09-11T08:22:58.371 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.9377999999999999e-05 2024-09-11T08:22:58.372 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.372 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.372 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:22:58.372 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.166383+0000", 2024-09-11T08:22:58.372 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.9188999999999999e-05 2024-09-11T08:22:58.373 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.373 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.373 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:22:58.373 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.351938+0000", 2024-09-11T08:22:58.373 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.18555543099999999 2024-09-11T08:22:58.373 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.374 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.374 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:22:58.374 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.351978+0000", 2024-09-11T08:22:58.374 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.9413999999999999e-05 2024-09-11T08:22:58.374 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:58.374 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:22:58.375 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:58.375 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.375 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.375 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.15369.0:206708 9.2 e297/246 9:483f0111:::benchmark_data_smithi150_133616_object91102:head v 297'12896, mlcod=297'12893)", 2024-09-11T08:22:58.375 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:21:01.169235+0000", 2024-09-11T08:22:58.376 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 116.04538921, 2024-09-11T08:22:58.376 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.19022061200000001, 2024-09-11T08:22:58.376 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:22:58.376 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:22:58.376 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:22:58.376 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.377 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:22:58.377 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.169235+0000", 2024-09-11T08:22:58.377 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:58.377 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.377 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.377 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:22:58.378 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.169235+0000", 2024-09-11T08:22:58.378 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:58.378 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.378 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.378 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:22:58.378 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.169234+0000", 2024-09-11T08:22:58.378 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:22:58.379 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.379 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.379 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:22:58.379 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.169242+0000", 2024-09-11T08:22:58.379 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.8150000000000006e-06 2024-09-11T08:22:58.379 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.380 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.380 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:22:58.380 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.344735+0000", 2024-09-11T08:22:58.380 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.175493225 2024-09-11T08:22:58.380 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.380 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.380 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:22:58.380 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.344749+0000", 2024-09-11T08:22:58.381 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3777999999999999e-05 2024-09-11T08:22:58.381 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.381 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.381 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:58.381 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.346911+0000", 2024-09-11T08:22:58.381 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.002161662 2024-09-11T08:22:58.381 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.381 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.381 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:22:58.381 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.346926+0000", 2024-09-11T08:22:58.381 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.5183e-05 2024-09-11T08:22:58.381 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.382 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.382 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:22:58.382 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.359425+0000", 2024-09-11T08:22:58.382 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.012499331000000001 2024-09-11T08:22:58.382 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.382 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.382 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:22:58.382 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.359456+0000", 2024-09-11T08:22:58.382 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.0539e-05 2024-09-11T08:22:58.382 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:58.382 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:22:58.382 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:58.382 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.382 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.383 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.15369.0:206711 9.6 e297/271 9:67241916:::benchmark_data_smithi150_133616_object91105:head v 297'12985, mlcod=297'12983)", 2024-09-11T08:22:58.383 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:21:01.169993+0000", 2024-09-11T08:22:58.383 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 116.044631251, 2024-09-11T08:22:58.383 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.18313901099999999, 2024-09-11T08:22:58.383 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:22:58.383 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:22:58.383 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:22:58.383 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.383 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:22:58.383 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.169993+0000", 2024-09-11T08:22:58.383 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:58.383 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.383 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.384 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:22:58.384 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.169993+0000", 2024-09-11T08:22:58.384 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:58.384 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.384 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.384 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:22:58.384 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.169992+0000", 2024-09-11T08:22:58.384 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:22:58.384 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.384 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.384 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:22:58.384 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.170002+0000", 2024-09-11T08:22:58.384 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.1090000000000004e-06 2024-09-11T08:22:58.385 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.385 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.385 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:22:58.385 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.344763+0000", 2024-09-11T08:22:58.385 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.17476161900000001 2024-09-11T08:22:58.385 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.385 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.385 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:22:58.385 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.344773+0000", 2024-09-11T08:22:58.385 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0222e-05 2024-09-11T08:22:58.385 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.385 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.385 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:58.385 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.344838+0000", 2024-09-11T08:22:58.386 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.4636999999999998e-05 2024-09-11T08:22:58.386 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.386 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.386 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:22:58.386 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.344864+0000", 2024-09-11T08:22:58.386 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.5996999999999999e-05 2024-09-11T08:22:58.386 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.386 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.386 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:22:58.386 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.353039+0000", 2024-09-11T08:22:58.386 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0081747790000000001 2024-09-11T08:22:58.386 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.386 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.387 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:22:58.387 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.353132+0000", 2024-09-11T08:22:58.387 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.3546000000000002e-05 2024-09-11T08:22:58.387 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:58.387 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:22:58.387 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:58.387 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.387 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.387 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:21381 10.5 10:a78b2459:::benchmark_data_smithi150_158067_object21380:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-09-11T08:22:58.387 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:22:07.763549+0000", 2024-09-11T08:22:58.387 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 49.451075564999996, 2024-09-11T08:22:58.387 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.104191874, 2024-09-11T08:22:58.387 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:22:58.387 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:22:58.388 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:22:58.388 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:22:58.388 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:22:58.388 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 21381 2024-09-11T08:22:58.388 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.388 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:22:58.388 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.388 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:22:58.388 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:07.763549+0000", 2024-09-11T08:22:58.388 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:58.388 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.389 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.389 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:22:58.389 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:07.763549+0000", 2024-09-11T08:22:58.389 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:58.389 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.389 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.389 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:22:58.389 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:07.763547+0000", 2024-09-11T08:22:58.389 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:22:58.389 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.390 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.390 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:22:58.390 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:07.763579+0000", 2024-09-11T08:22:58.390 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.1649999999999997e-05 2024-09-11T08:22:58.390 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.390 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.390 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:22:58.390 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:07.763602+0000", 2024-09-11T08:22:58.390 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.2974000000000001e-05 2024-09-11T08:22:58.390 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.390 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.390 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:22:58.390 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:07.763608+0000", 2024-09-11T08:22:58.390 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.5879999999999997e-06 2024-09-11T08:22:58.391 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.391 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.391 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:58.391 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:07.763756+0000", 2024-09-11T08:22:58.391 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00014806699999999999 2024-09-11T08:22:58.391 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.391 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.391 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:22:58.391 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:07.763950+0000", 2024-09-11T08:22:58.391 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00019390400000000001 2024-09-11T08:22:58.391 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.391 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.391 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 0,3", 2024-09-11T08:22:58.392 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:07.764075+0000", 2024-09-11T08:22:58.392 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012573400000000001 2024-09-11T08:22:58.392 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.392 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.392 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:22:58.392 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:07.768236+0000", 2024-09-11T08:22:58.392 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0041609840000000004 2024-09-11T08:22:58.392 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.392 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.392 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:22:58.392 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:07.768336+0000", 2024-09-11T08:22:58.392 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000100028 2024-09-11T08:22:58.392 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.393 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.393 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:22:58.393 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:07.867627+0000", 2024-09-11T08:22:58.393 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.099291038999999998 2024-09-11T08:22:58.393 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.393 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.393 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:22:58.393 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:07.867699+0000", 2024-09-11T08:22:58.393 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.1828000000000004e-05 2024-09-11T08:22:58.393 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.393 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.393 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:22:58.393 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:07.867741+0000", 2024-09-11T08:22:58.393 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1687000000000003e-05 2024-09-11T08:22:58.394 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:58.394 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:22:58.394 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:58.394 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.394 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.394 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:22392 10.5 10:a3b3476d:::benchmark_data_smithi150_158067_object22391:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-09-11T08:22:58.394 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:22:08.350848+0000", 2024-09-11T08:22:58.394 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 48.863776606000002, 2024-09-11T08:22:58.394 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.118444944, 2024-09-11T08:22:58.394 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:22:58.394 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:22:58.394 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:22:58.394 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:22:58.394 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:22:58.394 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 22392 2024-09-11T08:22:58.395 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.395 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:22:58.395 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.395 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:22:58.395 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:08.350848+0000", 2024-09-11T08:22:58.395 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:58.395 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.395 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.395 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:22:58.395 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:08.350848+0000", 2024-09-11T08:22:58.395 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:58.395 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.395 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.396 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:22:58.396 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:08.350847+0000", 2024-09-11T08:22:58.396 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:22:58.396 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.396 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.396 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:22:58.396 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:08.350871+0000", 2024-09-11T08:22:58.396 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.4170999999999999e-05 2024-09-11T08:22:58.396 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.396 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.396 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:22:58.396 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:08.350880+0000", 2024-09-11T08:22:58.396 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.2139999999999996e-06 2024-09-11T08:22:58.397 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.397 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.397 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:22:58.397 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:08.350884+0000", 2024-09-11T08:22:58.397 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1799999999999998e-06 2024-09-11T08:22:58.397 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.397 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.397 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:58.397 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:08.350928+0000", 2024-09-11T08:22:58.397 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.4530000000000002e-05 2024-09-11T08:22:58.397 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.397 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.397 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:22:58.398 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:08.351013+0000", 2024-09-11T08:22:58.398 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.4653000000000004e-05 2024-09-11T08:22:58.398 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.398 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.398 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 0,3", 2024-09-11T08:22:58.398 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:08.351089+0000", 2024-09-11T08:22:58.398 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.6452000000000006e-05 2024-09-11T08:22:58.398 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.398 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.398 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:22:58.398 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:08.351810+0000", 2024-09-11T08:22:58.399 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00072009799999999996 2024-09-11T08:22:58.399 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.399 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.399 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:22:58.399 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:08.352025+0000", 2024-09-11T08:22:58.399 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00021526999999999999 2024-09-11T08:22:58.399 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.399 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.399 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:22:58.399 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:08.469208+0000", 2024-09-11T08:22:58.399 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.117183391 2024-09-11T08:22:58.399 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.399 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.399 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:22:58.400 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:08.469257+0000", 2024-09-11T08:22:58.400 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.8526000000000003e-05 2024-09-11T08:22:58.400 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.400 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.400 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:22:58.400 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:08.469293+0000", 2024-09-11T08:22:58.400 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.6162999999999999e-05 2024-09-11T08:22:58.400 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:58.400 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:22:58.400 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:58.400 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.400 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.400 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:27647 10.5 10:ad0a4f7d:::benchmark_data_smithi150_158067_object27646:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-09-11T08:22:58.401 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:22:10.774683+0000", 2024-09-11T08:22:58.401 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 46.439941752999999, 2024-09-11T08:22:58.401 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.12618488899999999, 2024-09-11T08:22:58.401 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:22:58.401 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:22:58.401 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:22:58.401 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:22:58.401 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:22:58.401 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 27647 2024-09-11T08:22:58.401 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.401 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:22:58.401 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.401 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:22:58.401 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:10.774683+0000", 2024-09-11T08:22:58.402 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:58.402 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.402 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.402 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:22:58.402 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:10.774683+0000", 2024-09-11T08:22:58.402 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:58.402 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.402 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.402 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:22:58.402 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:10.774681+0000", 2024-09-11T08:22:58.402 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:22:58.402 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.402 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.402 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:22:58.403 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:10.774704+0000", 2024-09-11T08:22:58.403 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.3421000000000002e-05 2024-09-11T08:22:58.403 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.403 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.403 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:22:58.403 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:10.774721+0000", 2024-09-11T08:22:58.403 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.6736e-05 2024-09-11T08:22:58.403 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.403 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.403 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:22:58.403 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:10.774728+0000", 2024-09-11T08:22:58.403 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.5849999999999996e-06 2024-09-11T08:22:58.403 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.403 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.404 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:58.404 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:10.774770+0000", 2024-09-11T08:22:58.404 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1999999999999998e-05 2024-09-11T08:22:58.404 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.404 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.404 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:22:58.404 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:10.774884+0000", 2024-09-11T08:22:58.404 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000114023 2024-09-11T08:22:58.404 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.404 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.404 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 0,3", 2024-09-11T08:22:58.404 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:10.774969+0000", 2024-09-11T08:22:58.405 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.5109000000000001e-05 2024-09-11T08:22:58.405 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.405 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.405 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:22:58.405 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:10.776018+0000", 2024-09-11T08:22:58.405 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0010490600000000001 2024-09-11T08:22:58.405 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.405 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.405 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:22:58.405 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:10.776676+0000", 2024-09-11T08:22:58.405 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00065811000000000005 2024-09-11T08:22:58.406 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.406 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.406 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:22:58.406 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:10.900765+0000", 2024-09-11T08:22:58.406 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.12408931500000001 2024-09-11T08:22:58.406 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.406 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.406 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:22:58.406 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:10.900827+0000", 2024-09-11T08:22:58.406 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.2088999999999995e-05 2024-09-11T08:22:58.407 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.407 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.407 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:22:58.407 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:10.900868+0000", 2024-09-11T08:22:58.407 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.0204000000000003e-05 2024-09-11T08:22:58.407 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:58.407 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:22:58.407 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:58.407 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.407 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.407 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:71288 10.5 10:a09f00c7:::benchmark_data_smithi150_158067_object71287:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e321)", 2024-09-11T08:22:58.407 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:22:36.466555+0000", 2024-09-11T08:22:58.407 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 20.748069835999999, 2024-09-11T08:22:58.407 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.10398816800000001, 2024-09-11T08:22:58.408 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:22:58.408 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:22:58.408 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:22:58.408 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:22:58.408 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:22:58.408 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 71288 2024-09-11T08:22:58.408 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.408 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:22:58.408 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.408 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:22:58.408 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:36.466555+0000", 2024-09-11T08:22:58.408 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:58.408 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.409 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.409 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:22:58.409 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:36.466555+0000", 2024-09-11T08:22:58.409 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:58.409 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.409 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.409 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:22:58.409 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:36.466554+0000", 2024-09-11T08:22:58.409 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:22:58.409 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.409 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.409 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:22:58.409 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:36.466574+0000", 2024-09-11T08:22:58.409 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.0377000000000002e-05 2024-09-11T08:22:58.410 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.410 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.410 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:22:58.410 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:36.466585+0000", 2024-09-11T08:22:58.410 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1334e-05 2024-09-11T08:22:58.410 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.410 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.410 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:22:58.410 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:36.466591+0000", 2024-09-11T08:22:58.410 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.7069999999999997e-06 2024-09-11T08:22:58.411 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.411 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.411 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:58.411 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:36.466634+0000", 2024-09-11T08:22:58.411 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.3359999999999998e-05 2024-09-11T08:22:58.411 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.411 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.411 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:22:58.412 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:36.466715+0000", 2024-09-11T08:22:58.412 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.0584e-05 2024-09-11T08:22:58.412 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.412 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.412 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 0,3", 2024-09-11T08:22:58.412 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:36.466819+0000", 2024-09-11T08:22:58.412 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000103608 2024-09-11T08:22:58.412 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.413 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.413 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:22:58.413 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:36.475154+0000", 2024-09-11T08:22:58.413 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0083350739999999996 2024-09-11T08:22:58.413 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.413 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.413 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:22:58.413 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:36.475165+0000", 2024-09-11T08:22:58.413 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0746e-05 2024-09-11T08:22:58.414 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.414 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.414 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:22:58.414 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:36.570449+0000", 2024-09-11T08:22:58.414 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.095284923999999993 2024-09-11T08:22:58.414 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.414 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.414 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:22:58.414 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:36.570502+0000", 2024-09-11T08:22:58.415 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.2336000000000002e-05 2024-09-11T08:22:58.415 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.415 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.415 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:22:58.415 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:36.570543+0000", 2024-09-11T08:22:58.415 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1124999999999997e-05 2024-09-11T08:22:58.415 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:58.415 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:22:58.416 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:58.416 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.416 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.416 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:71291 10.5 10:a8bd8c44:::benchmark_data_smithi150_158067_object71290:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e321)", 2024-09-11T08:22:58.416 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:22:36.468942+0000", 2024-09-11T08:22:58.416 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 20.745682603999999, 2024-09-11T08:22:58.416 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.101738065, 2024-09-11T08:22:58.416 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:22:58.416 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:22:58.417 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:22:58.417 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:22:58.417 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:22:58.417 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 71291 2024-09-11T08:22:58.417 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.417 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:22:58.417 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.417 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:22:58.418 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:36.468942+0000", 2024-09-11T08:22:58.418 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:58.418 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.418 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.418 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:22:58.418 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:36.468942+0000", 2024-09-11T08:22:58.418 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:58.419 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.419 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.419 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:22:58.419 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:36.468940+0000", 2024-09-11T08:22:58.419 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:22:58.419 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.419 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.419 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:22:58.419 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:36.468964+0000", 2024-09-11T08:22:58.420 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.3799e-05 2024-09-11T08:22:58.420 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.420 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.420 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:22:58.420 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:36.468977+0000", 2024-09-11T08:22:58.420 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2461e-05 2024-09-11T08:22:58.420 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.420 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.421 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:22:58.421 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:36.468984+0000", 2024-09-11T08:22:58.421 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.8839999999999997e-06 2024-09-11T08:22:58.421 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.421 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.421 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:58.421 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:36.470297+0000", 2024-09-11T08:22:58.421 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0013138469999999999 2024-09-11T08:22:58.422 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.422 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.422 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:22:58.422 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:36.470406+0000", 2024-09-11T08:22:58.422 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000108554 2024-09-11T08:22:58.422 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.422 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.422 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 0,3", 2024-09-11T08:22:58.422 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:36.470529+0000", 2024-09-11T08:22:58.423 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012286400000000001 2024-09-11T08:22:58.423 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.423 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.423 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:22:58.423 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:36.475869+0000", 2024-09-11T08:22:58.423 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0053397059999999996 2024-09-11T08:22:58.423 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.423 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.424 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:22:58.424 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:36.476751+0000", 2024-09-11T08:22:58.424 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00088236399999999996 2024-09-11T08:22:58.424 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.424 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.424 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:22:58.424 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:36.570586+0000", 2024-09-11T08:22:58.424 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.093835036999999996 2024-09-11T08:22:58.424 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.425 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.425 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:22:58.425 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:36.570644+0000", 2024-09-11T08:22:58.425 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.8273999999999998e-05 2024-09-11T08:22:58.425 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.425 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.425 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:22:58.425 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:36.570680+0000", 2024-09-11T08:22:58.425 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.5744000000000002e-05 2024-09-11T08:22:58.425 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:58.425 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:22:58.425 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:58.425 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.425 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.426 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:95745 10.3 10:cde25498:::benchmark_data_smithi150_158067_object95744:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e327)", 2024-09-11T08:22:58.426 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:22:56.980175+0000", 2024-09-11T08:22:58.426 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.234449346, 2024-09-11T08:22:58.426 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.090468539000000001, 2024-09-11T08:22:58.426 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:22:58.426 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:22:58.426 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:22:58.426 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:22:58.426 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:22:58.426 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 95745 2024-09-11T08:22:58.427 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.427 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:22:58.427 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.427 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:22:58.427 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:56.980175+0000", 2024-09-11T08:22:58.427 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:58.427 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.427 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.427 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:22:58.427 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:56.980175+0000", 2024-09-11T08:22:58.427 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:58.427 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.427 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.428 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:22:58.428 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:56.980175+0000", 2024-09-11T08:22:58.428 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:22:58.428 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.428 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.428 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:22:58.428 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:56.980226+0000", 2024-09-11T08:22:58.428 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.0881000000000003e-05 2024-09-11T08:22:58.428 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.428 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.428 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:22:58.428 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:56.980233+0000", 2024-09-11T08:22:58.428 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.9310000000000008e-06 2024-09-11T08:22:58.428 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.429 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.429 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:22:58.429 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:56.980238+0000", 2024-09-11T08:22:58.429 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.7809999999999996e-06 2024-09-11T08:22:58.429 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.429 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.429 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:58.429 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:56.986790+0000", 2024-09-11T08:22:58.429 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0065515670000000003 2024-09-11T08:22:58.429 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.429 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.429 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:22:58.429 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:56.986901+0000", 2024-09-11T08:22:58.429 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000111623 2024-09-11T08:22:58.430 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.430 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.430 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 3,5", 2024-09-11T08:22:58.430 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:56.987024+0000", 2024-09-11T08:22:58.430 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012286 2024-09-11T08:22:58.430 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.430 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.430 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:22:58.430 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:56.999790+0000", 2024-09-11T08:22:58.430 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.012765235 2024-09-11T08:22:58.430 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.430 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.430 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:22:58.430 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:56.999818+0000", 2024-09-11T08:22:58.431 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.8507e-05 2024-09-11T08:22:58.431 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.431 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.431 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:22:58.431 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.070555+0000", 2024-09-11T08:22:58.431 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.070736805 2024-09-11T08:22:58.431 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.431 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.431 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:22:58.431 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.070603+0000", 2024-09-11T08:22:58.431 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.8143000000000003e-05 2024-09-11T08:22:58.431 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.431 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.432 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:22:58.432 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.070644+0000", 2024-09-11T08:22:58.432 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.0784000000000001e-05 2024-09-11T08:22:58.432 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:58.432 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:22:58.432 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:58.432 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.432 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.432 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:95747 10.3 10:cb5b0079:::benchmark_data_smithi150_158067_object95746:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e327)", 2024-09-11T08:22:58.432 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:22:56.984137+0000", 2024-09-11T08:22:58.432 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.23048786099999999, 2024-09-11T08:22:58.432 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.086666460000000001, 2024-09-11T08:22:58.432 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:22:58.433 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:22:58.433 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:22:58.433 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:22:58.433 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:22:58.433 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 95747 2024-09-11T08:22:58.433 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.433 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:22:58.433 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.433 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:22:58.433 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:56.984137+0000", 2024-09-11T08:22:58.433 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:58.433 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.433 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.433 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:22:58.434 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:56.984137+0000", 2024-09-11T08:22:58.434 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:58.434 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.434 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.434 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:22:58.434 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:56.984135+0000", 2024-09-11T08:22:58.434 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:22:58.434 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.434 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.434 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:22:58.434 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:56.984190+0000", 2024-09-11T08:22:58.434 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.5058e-05 2024-09-11T08:22:58.434 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.434 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.435 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:22:58.435 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:56.984205+0000", 2024-09-11T08:22:58.435 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4637999999999999e-05 2024-09-11T08:22:58.435 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.435 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.435 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:22:58.435 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:56.984212+0000", 2024-09-11T08:22:58.435 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.9700000000000002e-06 2024-09-11T08:22:58.435 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.435 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.435 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:58.435 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:56.992248+0000", 2024-09-11T08:22:58.435 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0080362439999999997 2024-09-11T08:22:58.435 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.436 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.436 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:22:58.436 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:56.992377+0000", 2024-09-11T08:22:58.436 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012874300000000001 2024-09-11T08:22:58.436 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.436 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.436 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 3,5", 2024-09-11T08:22:58.436 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:56.992661+0000", 2024-09-11T08:22:58.436 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00028449399999999999 2024-09-11T08:22:58.436 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.436 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.436 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:22:58.436 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.066935+0000", 2024-09-11T08:22:58.437 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.074273507000000002 2024-09-11T08:22:58.437 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.437 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.437 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:22:58.437 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.066968+0000", 2024-09-11T08:22:58.437 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.2839999999999997e-05 2024-09-11T08:22:58.437 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.437 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.437 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:22:58.437 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.070676+0000", 2024-09-11T08:22:58.437 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0037085780000000001 2024-09-11T08:22:58.438 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.438 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.438 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:22:58.438 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.070760+0000", 2024-09-11T08:22:58.438 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.3571000000000003e-05 2024-09-11T08:22:58.438 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.438 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.438 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:22:58.438 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.070803+0000", 2024-09-11T08:22:58.438 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.3442999999999997e-05 2024-09-11T08:22:58.438 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:58.438 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:22:58.438 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:58.439 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.439 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.439 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:95749 10.5 10:a2f68dea:::benchmark_data_smithi150_158067_object95748:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e327)", 2024-09-11T08:22:58.439 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:22:56.987946+0000", 2024-09-11T08:22:58.439 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.22667810799999999, 2024-09-11T08:22:58.439 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.075494183000000006, 2024-09-11T08:22:58.439 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:22:58.439 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:22:58.439 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:22:58.439 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:22:58.439 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:22:58.439 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 95749 2024-09-11T08:22:58.439 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.439 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:22:58.439 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.440 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:22:58.440 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:56.987946+0000", 2024-09-11T08:22:58.440 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:58.440 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.440 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.440 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:22:58.440 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:56.987946+0000", 2024-09-11T08:22:58.440 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:58.440 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.440 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.440 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:22:58.440 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:56.987945+0000", 2024-09-11T08:22:58.440 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:22:58.440 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.441 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.441 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:22:58.441 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:56.987969+0000", 2024-09-11T08:22:58.441 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.4409999999999998e-05 2024-09-11T08:22:58.441 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.441 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.441 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:22:58.441 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:56.987981+0000", 2024-09-11T08:22:58.441 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1294000000000001e-05 2024-09-11T08:22:58.441 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.441 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.441 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:22:58.441 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:56.987986+0000", 2024-09-11T08:22:58.441 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.4770000000000002e-06 2024-09-11T08:22:58.442 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.442 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.442 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:58.442 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:56.989061+0000", 2024-09-11T08:22:58.442 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0010752419999999999 2024-09-11T08:22:58.442 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.442 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.442 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:22:58.442 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:56.989175+0000", 2024-09-11T08:22:58.442 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000113627 2024-09-11T08:22:58.442 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.442 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.442 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 0,3", 2024-09-11T08:22:58.442 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:56.989301+0000", 2024-09-11T08:22:58.443 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000125726 2024-09-11T08:22:58.443 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.443 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.443 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:22:58.443 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:56.994266+0000", 2024-09-11T08:22:58.443 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0049655890000000003 2024-09-11T08:22:58.443 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.443 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.443 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:22:58.443 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:56.999985+0000", 2024-09-11T08:22:58.443 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0057184890000000002 2024-09-11T08:22:58.443 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.443 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.444 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:22:58.444 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.000088+0000", 2024-09-11T08:22:58.444 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000103607 2024-09-11T08:22:58.444 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.444 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.444 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:22:58.444 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.000141+0000", 2024-09-11T08:22:58.444 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.2899000000000003e-05 2024-09-11T08:22:58.444 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.444 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.444 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:22:58.444 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.063441+0000", 2024-09-11T08:22:58.444 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.063299322000000005 2024-09-11T08:22:58.444 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:58.445 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:22:58.445 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:58.445 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.445 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.445 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:95759 10.3 10:ca026021:::benchmark_data_smithi150_158067_object95758:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e327)", 2024-09-11T08:22:58.445 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:22:56.994059+0000", 2024-09-11T08:22:58.445 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.22056558800000001, 2024-09-11T08:22:58.445 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.076986313000000001, 2024-09-11T08:22:58.445 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:22:58.446 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:22:58.446 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:22:58.446 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:22:58.446 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:22:58.446 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 95759 2024-09-11T08:22:58.446 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.446 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:22:58.446 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.446 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:22:58.446 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:56.994059+0000", 2024-09-11T08:22:58.446 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:58.447 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.447 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.447 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:22:58.447 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:56.994059+0000", 2024-09-11T08:22:58.447 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:58.447 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.447 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.447 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:22:58.447 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:56.994057+0000", 2024-09-11T08:22:58.447 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:22:58.447 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.447 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.447 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:22:58.448 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:56.994095+0000", 2024-09-11T08:22:58.448 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.786e-05 2024-09-11T08:22:58.448 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.448 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.448 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:22:58.448 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:56.994179+0000", 2024-09-11T08:22:58.448 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.4184999999999994e-05 2024-09-11T08:22:58.448 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.448 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.448 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:22:58.448 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:56.994186+0000", 2024-09-11T08:22:58.448 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.7830000000000001e-06 2024-09-11T08:22:58.448 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.448 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.449 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:58.449 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:56.996193+0000", 2024-09-11T08:22:58.449 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.002006585 2024-09-11T08:22:58.449 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.449 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.449 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:22:58.449 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:56.996321+0000", 2024-09-11T08:22:58.449 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012816300000000001 2024-09-11T08:22:58.449 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.450 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.450 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 3,5", 2024-09-11T08:22:58.450 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:56.996466+0000", 2024-09-11T08:22:58.450 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00014516400000000001 2024-09-11T08:22:58.450 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.450 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.450 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:22:58.450 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.066997+0000", 2024-09-11T08:22:58.450 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.070530490000000001 2024-09-11T08:22:58.450 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.450 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.450 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:22:58.450 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.067026+0000", 2024-09-11T08:22:58.450 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.9788e-05 2024-09-11T08:22:58.451 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.451 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.451 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:22:58.451 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.070809+0000", 2024-09-11T08:22:58.451 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0037830210000000001 2024-09-11T08:22:58.451 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.451 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.451 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:22:58.451 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.070973+0000", 2024-09-11T08:22:58.451 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000163688 2024-09-11T08:22:58.451 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.451 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.451 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:22:58.451 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.071045+0000", 2024-09-11T08:22:58.452 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.2247000000000001e-05 2024-09-11T08:22:58.452 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:58.452 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:22:58.452 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:58.452 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.452 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.452 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:95760 10.3 10:cbd540a4:::benchmark_data_smithi150_158067_object95759:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e327)", 2024-09-11T08:22:58.452 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:22:56.996751+0000", 2024-09-11T08:22:58.452 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.217873079, 2024-09-11T08:22:58.452 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.082767932000000002, 2024-09-11T08:22:58.452 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:22:58.452 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:22:58.452 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:22:58.453 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:22:58.453 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:22:58.453 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 95760 2024-09-11T08:22:58.453 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.453 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:22:58.453 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.453 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:22:58.453 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:56.996751+0000", 2024-09-11T08:22:58.453 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:58.453 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.453 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.453 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:22:58.453 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:56.996751+0000", 2024-09-11T08:22:58.453 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:22:58.454 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.454 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.454 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:22:58.454 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:56.996750+0000", 2024-09-11T08:22:58.454 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:22:58.454 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.454 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.454 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:22:58.454 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:56.996808+0000", 2024-09-11T08:22:58.454 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.7686000000000003e-05 2024-09-11T08:22:58.454 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.454 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.454 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:22:58.454 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:56.996818+0000", 2024-09-11T08:22:58.455 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0035e-05 2024-09-11T08:22:58.455 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.455 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.455 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:22:58.455 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:56.996823+0000", 2024-09-11T08:22:58.455 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.0660000000000003e-06 2024-09-11T08:22:58.455 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.455 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.455 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:22:58.455 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:56.999839+0000", 2024-09-11T08:22:58.455 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0030157869999999998 2024-09-11T08:22:58.455 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.455 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.455 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:22:58.455 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.000115+0000", 2024-09-11T08:22:58.456 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00027604700000000002 2024-09-11T08:22:58.456 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.456 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.456 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 3,5", 2024-09-11T08:22:58.456 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.063739+0000", 2024-09-11T08:22:58.456 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.063624060999999996 2024-09-11T08:22:58.456 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.456 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.456 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:22:58.456 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.070479+0000", 2024-09-11T08:22:58.457 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0067406710000000002 2024-09-11T08:22:58.457 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.457 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.457 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:22:58.457 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.071130+0000", 2024-09-11T08:22:58.457 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00065073599999999996 2024-09-11T08:22:58.457 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.457 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.457 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:22:58.457 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.079195+0000", 2024-09-11T08:22:58.457 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0080645549999999993 2024-09-11T08:22:58.457 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.457 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.458 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:22:58.458 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.079471+0000", 2024-09-11T08:22:58.458 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000276455 2024-09-11T08:22:58.458 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:22:58.458 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:22:58.458 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:22:58.458 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.079519+0000", 2024-09-11T08:22:58.458 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.8220000000000002e-05 2024-09-11T08:22:58.458 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:58.458 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:22:58.458 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:58.458 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:22:58.458 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:22:58.459 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-11T08:22:58.459 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:58.480 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:58.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:58.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:58.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:58 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:22:58.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:58 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:22:58.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:58 smithi150 ceph-mon[29276]: pgmap v702: 148 pgs: 148 active+clean; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 801 B/s rd, 55 MiB/s wr, 878 op/s 2024-09-11T08:22:58.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:58 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/154101828' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]: dispatch 2024-09-11T08:22:58.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:58 smithi150 ceph-mon[29276]: osdmap e328: 8 total, 8 up, 8 in 2024-09-11T08:22:58.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:58 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]: dispatch 2024-09-11T08:22:58.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:58 smithi150 ceph-mon[29276]: 4.13 scrub starts 2024-09-11T08:22:58.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:58 smithi150 ceph-mon[29276]: 4.13 scrub ok 2024-09-11T08:22:58.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:58 smithi150 ceph-mon[29276]: 5.10 scrub starts 2024-09-11T08:22:58.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:58 smithi150 ceph-mon[29276]: 5.10 scrub ok 2024-09-11T08:22:58.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:58 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:22:58.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:58 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:22:58.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:58 smithi150 ceph-mon[39612]: pgmap v702: 148 pgs: 148 active+clean; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 801 B/s rd, 55 MiB/s wr, 878 op/s 2024-09-11T08:22:58.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:58 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/154101828' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]: dispatch 2024-09-11T08:22:58.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:58 smithi150 ceph-mon[39612]: osdmap e328: 8 total, 8 up, 8 in 2024-09-11T08:22:58.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:58 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]: dispatch 2024-09-11T08:22:58.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:58 smithi150 ceph-mon[39612]: 4.13 scrub starts 2024-09-11T08:22:58.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:58 smithi150 ceph-mon[39612]: 4.13 scrub ok 2024-09-11T08:22:58.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:58 smithi150 ceph-mon[39612]: 5.10 scrub starts 2024-09-11T08:22:58.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:58 smithi150 ceph-mon[39612]: 5.10 scrub ok 2024-09-11T08:22:58.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:58 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:22:58.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:58 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:22:58.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:58 smithi181 ceph-mon[32701]: pgmap v702: 148 pgs: 148 active+clean; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 801 B/s rd, 55 MiB/s wr, 878 op/s 2024-09-11T08:22:58.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:58 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/154101828' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]: dispatch 2024-09-11T08:22:58.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:58 smithi181 ceph-mon[32701]: osdmap e328: 8 total, 8 up, 8 in 2024-09-11T08:22:58.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:58 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]: dispatch 2024-09-11T08:22:58.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:58 smithi181 ceph-mon[32701]: 4.13 scrub starts 2024-09-11T08:22:58.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:58 smithi181 ceph-mon[32701]: 4.13 scrub ok 2024-09-11T08:22:58.575 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:58 smithi181 ceph-mon[32701]: 5.10 scrub starts 2024-09-11T08:22:58.575 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:58 smithi181 ceph-mon[32701]: 5.10 scrub ok 2024-09-11T08:22:58.582 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:58.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:58.605 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:58.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:58.706 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:22:58.727 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:58.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:58.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:22:58.829 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:22:58.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:58.852 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:58.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:22:58.863 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:22:58.952 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:22:58 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:22:58] "GET /metrics HTTP/1.1" 200 224661 "" "Prometheus/2.43.0" 2024-09-11T08:22:58.953 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:58.974 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:58.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:58.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:59.035 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 58 16 96822 96806 104.298 13.4375 0.00378189 0.00945508 2024-09-11T08:22:59.073 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:22:58 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:22:59.075 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:22:59.098 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:59.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:59.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:22:59.199 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:59.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:59 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]': finished 2024-09-11T08:22:59.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:59 smithi150 ceph-mon[39612]: osdmap e329: 8 total, 8 up, 8 in 2024-09-11T08:22:59.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:59 smithi150 ceph-mon[39612]: 10.f scrub ok 2024-09-11T08:22:59.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:59 smithi150 ceph-mon[39612]: 10.a scrub ok 2024-09-11T08:22:59.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:22:59 smithi150 ceph-mon[39612]: osdmap e330: 8 total, 8 up, 8 in 2024-09-11T08:22:59.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:59 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]': finished 2024-09-11T08:22:59.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:59 smithi150 ceph-mon[29276]: osdmap e329: 8 total, 8 up, 8 in 2024-09-11T08:22:59.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:59 smithi150 ceph-mon[29276]: 10.f scrub ok 2024-09-11T08:22:59.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:59 smithi150 ceph-mon[29276]: 10.a scrub ok 2024-09-11T08:22:59.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:22:59 smithi150 ceph-mon[29276]: osdmap e330: 8 total, 8 up, 8 in 2024-09-11T08:22:59.242 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:59.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:59.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:59.344 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:22:59.375 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:59.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:59.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:22:59.408 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-11T08:22:59.477 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:22:59.532 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:59.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:59.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:22:59.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:59 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]': finished 2024-09-11T08:22:59.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:59 smithi181 ceph-mon[32701]: osdmap e329: 8 total, 8 up, 8 in 2024-09-11T08:22:59.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:59 smithi181 ceph-mon[32701]: 10.f scrub ok 2024-09-11T08:22:59.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:59 smithi181 ceph-mon[32701]: 10.a scrub ok 2024-09-11T08:22:59.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:22:59 smithi181 ceph-mon[32701]: osdmap e330: 8 total, 8 up, 8 in 2024-09-11T08:22:59.634 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:22:59.669 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:59.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:59.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:22:59.771 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:22:59.811 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:22:59.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:59.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:22:59.913 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:22:59.956 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:22:59.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:22:59.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:00.035 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 59 16 97783 97767 103.548 60.0625 0.00816681 0.00965272 2024-09-11T08:23:00.058 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:23:00.095 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:00.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:00.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:23:00.196 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:23:00.225 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:00.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:00.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:23:00.327 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:00.362 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:00.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:00.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:00.464 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:00.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:00 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:23:00.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:00 smithi150 ceph-mon[29276]: 4.16 scrub starts 2024-09-11T08:23:00.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:00 smithi150 ceph-mon[29276]: 4.16 scrub ok 2024-09-11T08:23:00.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:00 smithi150 ceph-mon[29276]: 4.11 scrub starts 2024-09-11T08:23:00.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:00 smithi150 ceph-mon[29276]: 4.11 scrub ok 2024-09-11T08:23:00.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:00 smithi150 ceph-mon[29276]: pgmap v705: 148 pgs: 11 peering, 2 active+clean+scrubbing, 135 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.7 KiB/s rd, 84 MiB/s wr, 1.35k op/s 2024-09-11T08:23:00.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:00 smithi150 ceph-mon[29276]: Health check failed: Reduced data availability: 7 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-09-11T08:23:00.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:00 smithi150 ceph-mon[29276]: 10.d scrub starts 2024-09-11T08:23:00.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:00 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:23:00.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:00 smithi150 ceph-mon[39612]: 4.16 scrub starts 2024-09-11T08:23:00.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:00 smithi150 ceph-mon[39612]: 4.16 scrub ok 2024-09-11T08:23:00.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:00 smithi150 ceph-mon[39612]: 4.11 scrub starts 2024-09-11T08:23:00.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:00 smithi150 ceph-mon[39612]: 4.11 scrub ok 2024-09-11T08:23:00.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:00 smithi150 ceph-mon[39612]: pgmap v705: 148 pgs: 11 peering, 2 active+clean+scrubbing, 135 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.7 KiB/s rd, 84 MiB/s wr, 1.35k op/s 2024-09-11T08:23:00.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:00 smithi150 ceph-mon[39612]: Health check failed: Reduced data availability: 7 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-09-11T08:23:00.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:00 smithi150 ceph-mon[39612]: 10.d scrub starts 2024-09-11T08:23:00.506 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:00.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:00.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:00.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:00 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:23:00.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:00 smithi181 ceph-mon[32701]: 4.16 scrub starts 2024-09-11T08:23:00.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:00 smithi181 ceph-mon[32701]: 4.16 scrub ok 2024-09-11T08:23:00.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:00 smithi181 ceph-mon[32701]: 4.11 scrub starts 2024-09-11T08:23:00.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:00 smithi181 ceph-mon[32701]: 4.11 scrub ok 2024-09-11T08:23:00.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:00 smithi181 ceph-mon[32701]: pgmap v705: 148 pgs: 11 peering, 2 active+clean+scrubbing, 135 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.7 KiB/s rd, 84 MiB/s wr, 1.35k op/s 2024-09-11T08:23:00.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:00 smithi181 ceph-mon[32701]: Health check failed: Reduced data availability: 7 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-09-11T08:23:00.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:00 smithi181 ceph-mon[32701]: 10.d scrub starts 2024-09-11T08:23:00.608 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:23:00.637 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:00.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:00.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:23:00.739 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:23:00.776 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:00.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:00.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:23:00.878 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:00.921 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:00.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:00.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:01.023 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:01.035 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:2024-09-11T08:23:01.035998+0000 min lat: 0.00123278 max lat: 0.991736 avg lat: 0.00970627 2024-09-11T08:23:01.035 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-11T08:23:01.035 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 60 16 98886 98870 102.971 68.9375 0.00980382 0.00970627 2024-09-11T08:23:01.065 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:01.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:01.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:01.167 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:01.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:01.217 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:01.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:01.294 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:23:01.314 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:23:01.318 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:23:01.375 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:01.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:01.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:23:01.404 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:01 smithi150 ceph-mon[29276]: 5.7 scrub starts 2024-09-11T08:23:01.404 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:01 smithi150 ceph-mon[29276]: 5.7 scrub ok 2024-09-11T08:23:01.404 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:01 smithi150 ceph-mon[29276]: 3.5 scrub starts 2024-09-11T08:23:01.404 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:01 smithi150 ceph-mon[29276]: 3.5 scrub ok 2024-09-11T08:23:01.404 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:01 smithi150 ceph-mon[29276]: 10.9 scrub starts 2024-09-11T08:23:01.404 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:01 smithi150 ceph-mon[29276]: 5.9 scrub starts 2024-09-11T08:23:01.404 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:01 smithi150 ceph-mon[29276]: 5.9 scrub ok 2024-09-11T08:23:01.404 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:01 smithi150 ceph-mon[29276]: osdmap e331: 8 total, 8 up, 8 in 2024-09-11T08:23:01.405 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:01 smithi150 ceph-mon[39612]: 5.7 scrub starts 2024-09-11T08:23:01.405 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:01 smithi150 ceph-mon[39612]: 5.7 scrub ok 2024-09-11T08:23:01.405 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:01 smithi150 ceph-mon[39612]: 3.5 scrub starts 2024-09-11T08:23:01.405 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:01 smithi150 ceph-mon[39612]: 3.5 scrub ok 2024-09-11T08:23:01.405 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:01 smithi150 ceph-mon[39612]: 10.9 scrub starts 2024-09-11T08:23:01.405 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:01 smithi150 ceph-mon[39612]: 5.9 scrub starts 2024-09-11T08:23:01.405 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:01 smithi150 ceph-mon[39612]: 5.9 scrub ok 2024-09-11T08:23:01.405 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:01 smithi150 ceph-mon[39612]: osdmap e331: 8 total, 8 up, 8 in 2024-09-11T08:23:01.477 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:01.510 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:01.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:01.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:01.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:01 smithi181 ceph-mon[32701]: 5.7 scrub starts 2024-09-11T08:23:01.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:01 smithi181 ceph-mon[32701]: 5.7 scrub ok 2024-09-11T08:23:01.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:01 smithi181 ceph-mon[32701]: 3.5 scrub starts 2024-09-11T08:23:01.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:01 smithi181 ceph-mon[32701]: 3.5 scrub ok 2024-09-11T08:23:01.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:01 smithi181 ceph-mon[32701]: 10.9 scrub starts 2024-09-11T08:23:01.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:01 smithi181 ceph-mon[32701]: 5.9 scrub starts 2024-09-11T08:23:01.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:01 smithi181 ceph-mon[32701]: 5.9 scrub ok 2024-09-11T08:23:01.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:01 smithi181 ceph-mon[32701]: osdmap e331: 8 total, 8 up, 8 in 2024-09-11T08:23:01.611 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:23:01.645 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:01.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:01.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:23:01.747 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:01.791 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:01.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:01.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:01.893 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:23:01.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:01.928 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:01.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:23:02.029 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:23:02.037 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 61 16 99996 99980 102.42 69.375 0.00504406 0.00975786 2024-09-11T08:23:02.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:02.060 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:02.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:23:02.162 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:02.202 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:02.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:02.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:02.304 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:02.351 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:02.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:02.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:02.442 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:02 smithi150 ceph-mon[29276]: 6.16 scrub starts 2024-09-11T08:23:02.442 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:02 smithi150 ceph-mon[29276]: 6.16 scrub ok 2024-09-11T08:23:02.442 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:02 smithi150 ceph-mon[29276]: pgmap v708: 148 pgs: 11 peering, 2 active+clean+scrubbing, 135 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.33k op/s 2024-09-11T08:23:02.442 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:02 smithi150 ceph-mon[29276]: 3.14 scrub starts 2024-09-11T08:23:02.442 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:02 smithi150 ceph-mon[29276]: 3.14 scrub ok 2024-09-11T08:23:02.442 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:02 smithi150 ceph-mon[29276]: 4.a scrub starts 2024-09-11T08:23:02.443 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:02 smithi150 ceph-mon[29276]: 4.a scrub ok 2024-09-11T08:23:02.443 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:02 smithi150 ceph-mon[29276]: 10.d scrub ok 2024-09-11T08:23:02.443 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:02 smithi150 ceph-mon[39612]: 6.16 scrub starts 2024-09-11T08:23:02.443 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:02 smithi150 ceph-mon[39612]: 6.16 scrub ok 2024-09-11T08:23:02.443 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:02 smithi150 ceph-mon[39612]: pgmap v708: 148 pgs: 11 peering, 2 active+clean+scrubbing, 135 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.33k op/s 2024-09-11T08:23:02.443 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:02 smithi150 ceph-mon[39612]: 3.14 scrub starts 2024-09-11T08:23:02.443 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:02 smithi150 ceph-mon[39612]: 3.14 scrub ok 2024-09-11T08:23:02.443 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:02 smithi150 ceph-mon[39612]: 4.a scrub starts 2024-09-11T08:23:02.443 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:02 smithi150 ceph-mon[39612]: 4.a scrub ok 2024-09-11T08:23:02.443 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:02 smithi150 ceph-mon[39612]: 10.d scrub ok 2024-09-11T08:23:02.454 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:23:02.495 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:02.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:02.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:23:02.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:02 smithi181 ceph-mon[32701]: 6.16 scrub starts 2024-09-11T08:23:02.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:02 smithi181 ceph-mon[32701]: 6.16 scrub ok 2024-09-11T08:23:02.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:02 smithi181 ceph-mon[32701]: pgmap v708: 148 pgs: 11 peering, 2 active+clean+scrubbing, 135 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.33k op/s 2024-09-11T08:23:02.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:02 smithi181 ceph-mon[32701]: 3.14 scrub starts 2024-09-11T08:23:02.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:02 smithi181 ceph-mon[32701]: 3.14 scrub ok 2024-09-11T08:23:02.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:02 smithi181 ceph-mon[32701]: 4.a scrub starts 2024-09-11T08:23:02.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:02 smithi181 ceph-mon[32701]: 4.a scrub ok 2024-09-11T08:23:02.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:02 smithi181 ceph-mon[32701]: 10.d scrub ok 2024-09-11T08:23:02.597 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:02.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:02.644 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:02.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:02.745 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:02.783 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:02.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:02.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:02.885 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:02.928 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:02.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:02.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:03.030 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:23:03.035 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 62 16 101128 101112 101.909 70.75 0.00741615 0.00980839 2024-09-11T08:23:03.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:03.063 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:03.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:23:03.164 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:03.203 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:03.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:03.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:03.223 INFO:teuthology.orchestra.run.smithi150.stderr:noscrub is set 2024-09-11T08:23:03.305 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:23:03.341 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:03.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:03.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:23:03.405 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:03 smithi150 ceph-mon[29276]: 3.18 scrub starts 2024-09-11T08:23:03.405 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:03 smithi150 ceph-mon[29276]: 3.18 scrub ok 2024-09-11T08:23:03.405 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:03 smithi150 ceph-mon[29276]: 6.4 scrub starts 2024-09-11T08:23:03.405 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:03 smithi150 ceph-mon[29276]: 6.4 scrub ok 2024-09-11T08:23:03.405 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:03 smithi150 ceph-mon[29276]: 3.c scrub starts 2024-09-11T08:23:03.405 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:03 smithi150 ceph-mon[29276]: 3.c scrub ok 2024-09-11T08:23:03.405 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:03 smithi150 ceph-mon[29276]: 3.7 scrub starts 2024-09-11T08:23:03.406 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:03 smithi150 ceph-mon[29276]: 3.7 scrub ok 2024-09-11T08:23:03.406 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:03 smithi150 ceph-mon[29276]: 10.9 scrub ok 2024-09-11T08:23:03.406 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:03 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3827891137' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:23:03.406 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:03 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:23:03.406 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:03 smithi150 ceph-mon[39612]: 3.18 scrub starts 2024-09-11T08:23:03.406 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:03 smithi150 ceph-mon[39612]: 3.18 scrub ok 2024-09-11T08:23:03.406 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:03 smithi150 ceph-mon[39612]: 6.4 scrub starts 2024-09-11T08:23:03.406 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:03 smithi150 ceph-mon[39612]: 6.4 scrub ok 2024-09-11T08:23:03.406 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:03 smithi150 ceph-mon[39612]: 3.c scrub starts 2024-09-11T08:23:03.406 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:03 smithi150 ceph-mon[39612]: 3.c scrub ok 2024-09-11T08:23:03.406 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:03 smithi150 ceph-mon[39612]: 3.7 scrub starts 2024-09-11T08:23:03.407 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:03 smithi150 ceph-mon[39612]: 3.7 scrub ok 2024-09-11T08:23:03.407 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:03 smithi150 ceph-mon[39612]: 10.9 scrub ok 2024-09-11T08:23:03.407 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:03 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3827891137' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:23:03.407 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:03 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:23:03.444 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:03.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:03.481 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:03.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:03.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:03 smithi181 ceph-mon[32701]: 3.18 scrub starts 2024-09-11T08:23:03.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:03 smithi181 ceph-mon[32701]: 3.18 scrub ok 2024-09-11T08:23:03.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:03 smithi181 ceph-mon[32701]: 6.4 scrub starts 2024-09-11T08:23:03.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:03 smithi181 ceph-mon[32701]: 6.4 scrub ok 2024-09-11T08:23:03.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:03 smithi181 ceph-mon[32701]: 3.c scrub starts 2024-09-11T08:23:03.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:03 smithi181 ceph-mon[32701]: 3.c scrub ok 2024-09-11T08:23:03.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:03 smithi181 ceph-mon[32701]: 3.7 scrub starts 2024-09-11T08:23:03.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:03 smithi181 ceph-mon[32701]: 3.7 scrub ok 2024-09-11T08:23:03.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:03 smithi181 ceph-mon[32701]: 10.9 scrub ok 2024-09-11T08:23:03.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:03 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3827891137' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:23:03.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:03 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:23:03.582 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:23:03.618 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:03.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:03.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:23:03.720 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:23:03.756 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:03.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:03.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:23:03.858 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:03.892 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:03.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:03.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:03.994 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:23:04.029 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:04.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:04.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:23:04.035 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 63 16 102327 102311 101.481 74.9375 0.0144434 0.00984999 2024-09-11T08:23:04.131 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:23:04.159 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:04.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:04.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:23:04.261 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:04.300 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:04.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:04.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:04.402 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:23:04.430 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:04.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:04.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:23:04.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:04 smithi150 ceph-mon[29276]: pgmap v709: 148 pgs: 11 peering, 137 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 869 B/s rd, 80 MiB/s wr, 1.28k op/s 2024-09-11T08:23:04.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:04 smithi150 ceph-mon[29276]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:23:04.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:04 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:23:04.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:04 smithi150 ceph-mon[29276]: osdmap e332: 8 total, 8 up, 8 in 2024-09-11T08:23:04.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:04 smithi150 ceph-mon[39612]: pgmap v709: 148 pgs: 11 peering, 137 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 869 B/s rd, 80 MiB/s wr, 1.28k op/s 2024-09-11T08:23:04.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:04 smithi150 ceph-mon[39612]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:23:04.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:04 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:23:04.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:04 smithi150 ceph-mon[39612]: osdmap e332: 8 total, 8 up, 8 in 2024-09-11T08:23:04.533 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:23:04.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:04 smithi181 ceph-mon[32701]: pgmap v709: 148 pgs: 11 peering, 137 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 869 B/s rd, 80 MiB/s wr, 1.28k op/s 2024-09-11T08:23:04.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:04 smithi181 ceph-mon[32701]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:23:04.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:04 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:23:04.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:04 smithi181 ceph-mon[32701]: osdmap e332: 8 total, 8 up, 8 in 2024-09-11T08:23:04.576 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:04.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:04.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:23:04.679 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:23:04.710 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:04.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:04.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:23:04.812 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:23:04.836 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-11T08:23:04.836 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T08:23:04.836 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-09-11T08:23:04.836 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-11T08:23:04.837 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-11T08:23:04.837 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-09-11T08:23:04.854 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:04.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:04.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:23:04.956 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:04.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:05.000 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:05.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:05.036 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 64 15 103488 103473 101.03 72.625 0.00747395 0.00989219 2024-09-11T08:23:05.101 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:23:05.123 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:05.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:05.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:23:05.225 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:05.256 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-11T08:23:05.256 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [ 2024-09-11T08:23:05.256 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:05.256 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.15636.0:103641 10.4 e332/308 10:28e62547:::benchmark_data_smithi150_158067_object103640:head v 332'6504, mlcod=332'6501)", 2024-09-11T08:23:05.256 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:23:05.242574+0000", 2024-09-11T08:23:05.256 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.007010306, 2024-09-11T08:23:05.256 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0070400669999999997, 2024-09-11T08:23:05.257 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:23:05.257 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "started", 2024-09-11T08:23:05.257 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:23:05.257 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:05.257 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:23:05.257 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:23:05.242574+0000", 2024-09-11T08:23:05.257 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:23:05.257 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:05.257 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:05.257 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:23:05.257 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:23:05.242574+0000", 2024-09-11T08:23:05.257 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:23:05.257 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:05.258 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:05.258 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:23:05.258 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:23:05.242573+0000", 2024-09-11T08:23:05.258 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:23:05.258 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:05.258 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:05.258 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:23:05.258 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:23:05.242679+0000", 2024-09-11T08:23:05.258 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000106412 2024-09-11T08:23:05.258 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:05.258 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:05.258 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:23:05.258 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:23:05.242694+0000", 2024-09-11T08:23:05.258 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4875000000000001e-05 2024-09-11T08:23:05.258 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:05.259 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:05.259 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:23:05.259 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:23:05.242700+0000", 2024-09-11T08:23:05.259 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.0020000000000003e-06 2024-09-11T08:23:05.259 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:05.259 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:05.259 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:23:05.259 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:23:05.242834+0000", 2024-09-11T08:23:05.259 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000133926 2024-09-11T08:23:05.259 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:05.259 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:05.259 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:23:05.259 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:23:05.242887+0000", 2024-09-11T08:23:05.259 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.312e-05 2024-09-11T08:23:05.259 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:23:05.260 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:23:05.260 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:23:05.260 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:23:05.260 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-09-11T08:23:05.260 INFO:teuthology.orchestra.run.smithi150.stdout: "num_ops": 1 2024-09-11T08:23:05.260 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-11T08:23:05.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:05.263 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:05.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:05.270 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:23:05 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:23:05] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:23:05.287 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:23:05.363 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:23:05.398 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:05.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:05.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:23:05.436 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:05 smithi150 ceph-mon[29276]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 9 pgs peering) 2024-09-11T08:23:05.436 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:05 smithi150 ceph-mon[39612]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 9 pgs peering) 2024-09-11T08:23:05.501 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:05.524 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:05.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:05.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:05.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:05 smithi181 ceph-mon[32701]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 9 pgs peering) 2024-09-11T08:23:05.626 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:23:05.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:05.659 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:05.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:23:05.760 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:23:05.796 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:05.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:05.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:23:05.898 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:23:05.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:05.944 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:05.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:23:06.036 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 65 16 104466 104450 100.415 61.0625 0.00678737 0.00994935 2024-09-11T08:23:06.045 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:06.066 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:06.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:06.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:06.168 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:23:06.205 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:06.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:06.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:23:06.307 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:06.350 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:06.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:06.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:06.452 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:06.484 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:06.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:06.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:06.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:06 smithi150 ceph-mon[29276]: pgmap v711: 148 pgs: 148 active+clean; 6.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 49 MiB/s wr, 786 op/s 2024-09-11T08:23:06.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:06 smithi150 ceph-mon[29276]: osdmap e333: 8 total, 8 up, 8 in 2024-09-11T08:23:06.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:06 smithi150 ceph-mon[39612]: pgmap v711: 148 pgs: 148 active+clean; 6.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 49 MiB/s wr, 786 op/s 2024-09-11T08:23:06.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:06 smithi150 ceph-mon[39612]: osdmap e333: 8 total, 8 up, 8 in 2024-09-11T08:23:06.554 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:06 smithi181 ceph-mon[32701]: pgmap v711: 148 pgs: 148 active+clean; 6.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 49 MiB/s wr, 786 op/s 2024-09-11T08:23:06.554 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:06 smithi181 ceph-mon[32701]: osdmap e333: 8 total, 8 up, 8 in 2024-09-11T08:23:06.586 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:23:06.617 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:06.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:06.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:23:06.647 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-11T08:23:06.719 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:23:06.750 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:06.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:06.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:23:06.851 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:23:06.893 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:06.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:06.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:23:06.995 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:23:07.023 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:07.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:07.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:23:07.036 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 66 15 105574 105559 99.9439 69.3125 0.00976834 0.0100012 2024-09-11T08:23:07.124 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:07.167 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:07.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:07.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:07.269 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:07.291 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:07.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:07.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:07.393 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:23:07.432 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:07.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:07.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:23:07.534 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:23:07.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:07.573 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:07.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:23:07.674 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:23:07.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:07.711 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:07.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:23:07.812 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:23:07.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:07 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:23:07.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:07 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:23:07.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:07 smithi181 ceph-mon[32701]: pgmap v713: 148 pgs: 148 active+clean; 6.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 801 B/s rd, 46 MiB/s wr, 738 op/s 2024-09-11T08:23:07.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:07 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2389385380' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:23:07.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:07.848 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:07.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:23:07.949 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:23:07.974 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:07.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:07.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:23:08.024 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:07 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:23:08.024 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:07 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:23:08.024 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:07 smithi150 ceph-mon[29276]: pgmap v713: 148 pgs: 148 active+clean; 6.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 801 B/s rd, 46 MiB/s wr, 738 op/s 2024-09-11T08:23:08.024 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:07 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2389385380' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:23:08.025 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:07 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:23:08.025 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:07 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:23:08.025 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:07 smithi150 ceph-mon[39612]: pgmap v713: 148 pgs: 148 active+clean; 6.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 801 B/s rd, 46 MiB/s wr, 738 op/s 2024-09-11T08:23:08.025 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:07 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2389385380' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:23:08.025 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:23:08.025 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:23:08.026 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:23:08.026 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:23:08.026 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:23:08.026 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:23:08.026 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:23:08.026 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:23:08.027 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:23:08.027 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:23:08.027 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:23:08.027 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:23:08.027 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:23:08.027 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:23:08.027 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:23:08.027 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:23:08.036 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 67 16 106495 106479 99.3104 57.5 0.00464864 0.0100534 2024-09-11T08:23:08.076 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:08.097 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:08.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:08.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:08.199 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:23:08.246 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:08.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:08.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:23:08.348 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:23:08.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:08.370 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:08.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:23:08.470 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:08.492 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:08.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:08.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:08.594 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:23:08.633 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:08.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:08.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:23:08.734 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:08.771 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:08.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:08.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:08.872 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:23:08.882 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:23:08 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:23:08.882 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:08 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:23:08.882 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:08 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:23:08.882 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:08 smithi181 ceph-mon[32701]: osdmap e334: 8 total, 8 up, 8 in 2024-09-11T08:23:08.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:08.948 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:08.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:23:08.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:08 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:23:08.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:08 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:23:08.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:08 smithi150 ceph-mon[29276]: osdmap e334: 8 total, 8 up, 8 in 2024-09-11T08:23:08.988 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:23:08 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:23:08] "GET /metrics HTTP/1.1" 200 224681 "" "Prometheus/2.43.0" 2024-09-11T08:23:08.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:08 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:23:08.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:08 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:23:08.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:08 smithi150 ceph-mon[39612]: osdmap e334: 8 total, 8 up, 8 in 2024-09-11T08:23:09.036 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 68 16 107219 107203 98.5154 45.25 0.0149987 0.0101467 2024-09-11T08:23:09.048 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:09.073 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:09.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:09.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:09.158 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-11T08:23:09.158 INFO:teuthology.orchestra.run.smithi181.stdout: "success": "" 2024-09-11T08:23:09.158 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-11T08:23:09.175 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:23:09.197 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:09.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:09.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:23:09.298 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:23:09.320 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:09.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:09.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:23:09.422 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:23:09.443 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:09.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:09.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:23:09.545 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:23:09.578 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:09.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:09.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:23:09.680 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:23:09.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:09.711 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:09.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:23:09.812 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:09.855 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:09.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:09.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:09.957 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:23:09.998 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:09.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:09.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:23:10.036 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 69 16 107986 107970 97.7824 47.9375 0.00758428 0.0102221 2024-09-11T08:23:10.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:09 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:23:10.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:09 smithi181 ceph-mon[32701]: pgmap v715: 148 pgs: 148 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.7 KiB/s rd, 81 MiB/s wr, 1.29k op/s 2024-09-11T08:23:10.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:09 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:23:10.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:09 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:23:10.100 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:23:10.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:10.133 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:10.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:23:10.226 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:09 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:23:10.226 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:09 smithi150 ceph-mon[29276]: pgmap v715: 148 pgs: 148 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.7 KiB/s rd, 81 MiB/s wr, 1.29k op/s 2024-09-11T08:23:10.226 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:09 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:23:10.226 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:09 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:23:10.226 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:09 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:23:10.226 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:09 smithi150 ceph-mon[39612]: pgmap v715: 148 pgs: 148 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.7 KiB/s rd, 81 MiB/s wr, 1.29k op/s 2024-09-11T08:23:10.226 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:09 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:23:10.226 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:09 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:23:10.234 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:23:10.265 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:10.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:10.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:23:10.366 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:10.407 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:10.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:10.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:10.508 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:23:10.543 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:10.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:10.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:23:10.645 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:10.683 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:10.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:10.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:10.784 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:23:10.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:10.818 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:10.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:23:10.920 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:23:10.951 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:10.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:10.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:23:11.005 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:23:11.036 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 70 16 109207 109191 97.4756 76.3125 0.0175725 0.0102543 2024-09-11T08:23:11.054 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:11.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:10 smithi181 ceph-mon[32701]: osdmap e335: 8 total, 8 up, 8 in 2024-09-11T08:23:11.098 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:11.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:11.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:11.148 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:10 smithi150 ceph-mon[29276]: osdmap e335: 8 total, 8 up, 8 in 2024-09-11T08:23:11.148 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:10 smithi150 ceph-mon[39612]: osdmap e335: 8 total, 8 up, 8 in 2024-09-11T08:23:11.200 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:23:11.233 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:11.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:11.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:23:11.335 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:11.376 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:11.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:11.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:11.477 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:23:11.506 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:11.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:11.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:23:11.607 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:23:11.642 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:11.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:11.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:23:11.744 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:11.780 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:11.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:11.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:11.882 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:11.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:11.917 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:11.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:12.018 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:23:12.036 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 71 16 110404 110388 97.1564 74.8125 0.00927863 0.0102882 2024-09-11T08:23:12.055 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:12.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:12.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:23:12.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:11 smithi181 ceph-mon[32701]: pgmap v717: 148 pgs: 148 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 55 MiB/s wr, 875 op/s 2024-09-11T08:23:12.156 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:12.196 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:12.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:12.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:12.206 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:11 smithi150 ceph-mon[29276]: pgmap v717: 148 pgs: 148 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 55 MiB/s wr, 875 op/s 2024-09-11T08:23:12.206 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:11 smithi150 ceph-mon[39612]: pgmap v717: 148 pgs: 148 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 55 MiB/s wr, 875 op/s 2024-09-11T08:23:12.278 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T08:23:12.298 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:23:12.328 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:12.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:12.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:23:12.429 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:23:12.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:12.458 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:12.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:23:12.559 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:23:12.588 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:12.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:12.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:23:12.692 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:23:12.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:12.735 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:12.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:23:12.837 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:12.854 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:12 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:23:12.873 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:12 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:23:12.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:12.892 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:12.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:12.897 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T08:23:12.993 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:13.037 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 72 16 111454 111438 96.7184 65.625 0.0121279 0.0103333 2024-09-11T08:23:13.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:13.057 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:13.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:13.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:12 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:23:13.141 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-09-11T08:23:13.141 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:23:13.158 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:23:13.185 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:13.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:13.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:23:13.287 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:23:13.314 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:13.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:13.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:23:13.416 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:23:13.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:13.464 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:13.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:23:13.565 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:23:13.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:13.599 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:13.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:23:13.700 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:23:13.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:13.735 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:13.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:23:13.836 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:23:13.873 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:13.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:13.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:23:13.975 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:14.007 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:14.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:14.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:14.036 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 73 16 112457 112441 96.2522 62.6875 0.0120475 0.0103847 2024-09-11T08:23:14.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:13 smithi181 ceph-mon[32701]: pgmap v718: 148 pgs: 148 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 692 B/s rd, 65 MiB/s wr, 1.05k op/s 2024-09-11T08:23:14.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:13 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:23:14.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:13 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:23:14.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:13 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:23:14.111 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:14.136 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:14.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:14.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:14.238 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:23:14.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:13 smithi150 ceph-mon[29276]: pgmap v718: 148 pgs: 148 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 692 B/s rd, 65 MiB/s wr, 1.05k op/s 2024-09-11T08:23:14.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:13 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:23:14.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:13 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:23:14.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:13 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:23:14.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:13 smithi150 ceph-mon[39612]: pgmap v718: 148 pgs: 148 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 692 B/s rd, 65 MiB/s wr, 1.05k op/s 2024-09-11T08:23:14.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:13 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:23:14.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:13 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:23:14.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:13 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:23:14.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:14.277 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:14.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:23:14.378 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:23:14.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:14.413 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:14.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:23:14.514 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:14.555 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:14.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:14.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:14.657 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:23:14.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:14.696 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:14.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:23:14.797 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:23:14.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:14.830 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:14.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:23:14.931 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:14.972 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:14.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:14.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:14.980 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-11T08:23:14.980 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T08:23:14.980 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-11T08:23:14.980 INFO:tasks.thrashosds.thrasher:Setting osd 4 primary_affinity to 0.271963 2024-09-11T08:23:14.980 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 4 0.2719625924056829 2024-09-11T08:23:14.996 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:23:15.001 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-11T08:23:15.036 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 74 16 113628 113612 95.9404 73.1875 0.0106018 0.010417 2024-09-11T08:23:15.075 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:23:15.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:15.107 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:15.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:23:15.208 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:15.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:15.275 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:15.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:15.376 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:23:15.408 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:15.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:15.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:23:15.510 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:15.554 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:15.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:15.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:15.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:23:15 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:23:15] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:23:15.655 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:23:15.689 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:15.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:15.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:23:15.791 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:15.821 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:23:15.832 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:15.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:15.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:15.935 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:15.977 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:15.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:15.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:16.036 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 75 16 114709 114693 95.562 67.5625 0.00623775 0.0104602 2024-09-11T08:23:16.079 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:16.111 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:16.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:16.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:16.213 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:16.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:16 smithi150 ceph-mon[29276]: pgmap v719: 148 pgs: 148 active+clean; 6.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.29k op/s 2024-09-11T08:23:16.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:16 smithi150 ceph-mon[39612]: pgmap v719: 148 pgs: 148 active+clean; 6.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.29k op/s 2024-09-11T08:23:16.248 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:16.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:16.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:16.350 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:16.387 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:16.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:16.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:16.489 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:16.526 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:16.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:16.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:16.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:16 smithi181 ceph-mon[32701]: pgmap v719: 148 pgs: 148 active+clean; 6.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.29k op/s 2024-09-11T08:23:16.627 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:23:16.661 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:16.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:16.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:23:16.763 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:23:16.813 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:16.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:16.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:23:16.915 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:23:16.949 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:16.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:16.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:23:17.037 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 76 16 115885 115869 95.2716 73.5 0.0117431 0.0104919 2024-09-11T08:23:17.051 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:17.103 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:17.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:17.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:17.204 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:23:17.237 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:17.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:17.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:23:17.338 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:17.382 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:17.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:17.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:17.493 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:23:17.523 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:17.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:17.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:23:17.624 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:23:17.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:17.665 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:17.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:23:17.766 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:17.810 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:17.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:17.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:17.912 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:17.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:17.954 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:17.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:18.001 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:23:18.037 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 77 16 117034 117018 94.9669 71.8125 0.0158798 0.0105246 2024-09-11T08:23:18.056 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:23:18.106 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:18.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:18.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:23:18.208 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:23:18.238 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:18.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:18 smithi150 ceph-mon[29276]: pgmap v720: 148 pgs: 148 active+clean; 6.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 65 MiB/s wr, 1.04k op/s 2024-09-11T08:23:18.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:18 smithi150 ceph-mon[39612]: pgmap v720: 148 pgs: 148 active+clean; 6.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 65 MiB/s wr, 1.04k op/s 2024-09-11T08:23:18.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:18.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:23:18.340 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:18.375 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:18 smithi181 ceph-mon[32701]: pgmap v720: 148 pgs: 148 active+clean; 6.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 65 MiB/s wr, 1.04k op/s 2024-09-11T08:23:18.375 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:18.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:18.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:18.477 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:23:18.518 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:18.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:18.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:23:18.620 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:18.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:18.662 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:18.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:18.763 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:23:18.795 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:18.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:18.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:23:18.868 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:23:18 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:23:18] "GET /metrics HTTP/1.1" 200 224681 "" "Prometheus/2.43.0" 2024-09-11T08:23:18.897 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:18.940 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:18.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:18.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:19.038 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 78 16 118094 118078 94.5972 66.25 0.00859068 0.0105668 2024-09-11T08:23:19.042 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:23:19.051 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:23:19.052 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:23:19.073 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:23:18 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:23:19.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:19.085 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:19.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:23:19.093 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:23:19.093 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:23:19.123 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:23:19.123 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:23:19.137 INFO:teuthology.orchestra.run.smithi150.stderr:nodeep-scrub is set 2024-09-11T08:23:19.154 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:23:19.155 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:23:19.186 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:19.204 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:23:19.205 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:23:19.225 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:19.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:19.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:19.238 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:23:19.239 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:23:19.277 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:23:19.277 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:23:19.311 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:23:19.311 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:23:19.328 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:19.369 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:19.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:19.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:19.472 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:23:19.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:19 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3808786467' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:23:19.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:19 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/4160659772' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:23:19.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:19 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3808786467' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:23:19.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:19 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/4160659772' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:23:19.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:19.505 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:19.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:23:19.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:19 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3808786467' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:23:19.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:19 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/4160659772' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:23:19.606 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:23:19.646 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:19.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:19.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:23:19.748 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:19.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:19.784 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:19.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:19.885 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:23:19.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:19.926 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:19.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:23:20.026 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:20.038 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 79 16 119158 119142 94.2414 66.5 0.00574281 0.0106069 2024-09-11T08:23:20.056 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:20.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:20.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:20.158 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:20.192 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:20.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:20.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:20.294 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:23:20.328 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:20.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:20.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:23:20.386 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:20 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:23:20.386 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:20 smithi150 ceph-mon[29276]: pgmap v721: 148 pgs: 148 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 67 MiB/s wr, 1.06k op/s 2024-09-11T08:23:20.386 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:20 smithi150 ceph-mon[29276]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:23:20.386 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:20 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3808786467' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:23:20.386 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:20 smithi150 ceph-mon[29276]: osdmap e336: 8 total, 8 up, 8 in 2024-09-11T08:23:20.387 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:20 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:23:20.387 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:20 smithi150 ceph-mon[39612]: pgmap v721: 148 pgs: 148 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 67 MiB/s wr, 1.06k op/s 2024-09-11T08:23:20.387 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:20 smithi150 ceph-mon[39612]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:23:20.387 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:20 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3808786467' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:23:20.387 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:20 smithi150 ceph-mon[39612]: osdmap e336: 8 total, 8 up, 8 in 2024-09-11T08:23:20.430 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:20.462 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:20.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:20.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:20.564 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:23:20.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:20 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:23:20.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:20 smithi181 ceph-mon[32701]: pgmap v721: 148 pgs: 148 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 67 MiB/s wr, 1.06k op/s 2024-09-11T08:23:20.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:20 smithi181 ceph-mon[32701]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:23:20.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:20 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3808786467' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:23:20.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:20 smithi181 ceph-mon[32701]: osdmap e336: 8 total, 8 up, 8 in 2024-09-11T08:23:20.592 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:23:20.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:20.602 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:20.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:23:20.703 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:20.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:20.748 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:20.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:20.849 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:20.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:20.897 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:20.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:20.998 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:23:21.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:21.026 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:21.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:23:21.038 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:2024-09-11T08:23:21.039047+0000 min lat: 0.00123278 max lat: 0.991736 avg lat: 0.0106462 2024-09-11T08:23:21.038 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-11T08:23:21.038 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 80 16 120210 120194 93.8853 65.75 0.015524 0.0106462 2024-09-11T08:23:21.127 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:23:21.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:21.170 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:21.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:23:21.271 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:21.313 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:21.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:21.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:21.415 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:21.454 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:21.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:21.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:21.556 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:23:21.592 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:21.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:21.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:23:21.669 INFO:teuthology.orchestra.run.smithi150.stderr:set osd.4 primary-affinity to 0.271963 (8178232) 2024-09-11T08:23:21.694 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:23:21.720 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:21.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:21.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:23:21.822 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:23:21.846 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:21.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:21.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:23:21.948 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:23:21.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:21.970 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:21.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:23:21.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:21 smithi150 ceph-mon[29276]: pgmap v723: 148 pgs: 148 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 67 MiB/s wr, 1.06k op/s 2024-09-11T08:23:21.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:21 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/4149174307' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.2719625924056829}]: dispatch 2024-09-11T08:23:21.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:21 smithi150 ceph-mon[39612]: pgmap v723: 148 pgs: 148 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 67 MiB/s wr, 1.06k op/s 2024-09-11T08:23:21.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:21 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/4149174307' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.2719625924056829}]: dispatch 2024-09-11T08:23:22.038 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 81 16 121144 121128 93.4468 58.375 0.00387172 0.0106608 2024-09-11T08:23:22.071 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:22.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:21 smithi181 ceph-mon[32701]: pgmap v723: 148 pgs: 148 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 67 MiB/s wr, 1.06k op/s 2024-09-11T08:23:22.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:21 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/4149174307' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.2719625924056829}]: dispatch 2024-09-11T08:23:22.092 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:22.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:22.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:22.194 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:22.216 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:22.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:22.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:22.318 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:22.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:22.340 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:22.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:22.441 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:22.462 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:22.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:22.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:22.564 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:22.586 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:22.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:22.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:22.687 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:22.725 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:22.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:22.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:22.827 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:23:22.861 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:22.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:22.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:23:22.964 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:22.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:22 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/4149174307' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.2719625924056829}]': finished 2024-09-11T08:23:22.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:22 smithi150 ceph-mon[29276]: osdmap e337: 8 total, 8 up, 8 in 2024-09-11T08:23:22.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:22 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:23:22.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:22 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/4149174307' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.2719625924056829}]': finished 2024-09-11T08:23:22.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:22 smithi150 ceph-mon[39612]: osdmap e337: 8 total, 8 up, 8 in 2024-09-11T08:23:22.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:22 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:23:23.000 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:23.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:23.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:23.038 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 82 16 121556 121540 92.6213 25.75 0.0108948 0.010792 2024-09-11T08:23:23.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:22 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/4149174307' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.2719625924056829}]': finished 2024-09-11T08:23:23.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:22 smithi181 ceph-mon[32701]: osdmap e337: 8 total, 8 up, 8 in 2024-09-11T08:23:23.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:22 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:23:23.102 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:23:23.132 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:23.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:23.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:23:23.234 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:23:23.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:23.279 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:23.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:23:23.380 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:23:23.415 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:23.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:23.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:23:23.517 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:23:23.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:23.552 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:23.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:23:23.626 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T08:23:23.653 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:23.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:23.722 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:23.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:23.823 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:23.873 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:23.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:23.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:23.975 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:23.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:23 smithi150 ceph-mon[29276]: osdmap e338: 8 total, 8 up, 8 in 2024-09-11T08:23:23.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:23 smithi150 ceph-mon[29276]: pgmap v726: 148 pgs: 148 active+clean; 7.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 170 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-09-11T08:23:23.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:23 smithi150 ceph-mon[39612]: osdmap e338: 8 total, 8 up, 8 in 2024-09-11T08:23:23.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:23 smithi150 ceph-mon[39612]: pgmap v726: 148 pgs: 148 active+clean; 7.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 170 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-09-11T08:23:24.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:24.028 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:24.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:24.038 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 83 16 122677 122661 92.3494 70.0625 0.0394551 0.0108231 2024-09-11T08:23:24.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:23 smithi181 ceph-mon[32701]: osdmap e338: 8 total, 8 up, 8 in 2024-09-11T08:23:24.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:23 smithi181 ceph-mon[32701]: pgmap v726: 148 pgs: 148 active+clean; 7.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 170 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-09-11T08:23:24.129 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:24.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:24.175 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:24.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:24.276 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:23:24.323 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:24.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:24.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:23:24.329 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-11T08:23:24.330 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 20, 2024-09-11T08:23:24.330 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 600, 2024-09-11T08:23:24.330 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [ 2024-09-11T08:23:24.330 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.330 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15087.0:254113 8.b 8:d32650ea:::benchmark_data_smithi150_107646_object125281:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-11T08:23:24.330 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:17:32.516600+0000", 2024-09-11T08:23:24.330 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 351.80906483400003, 2024-09-11T08:23:24.330 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.13402002099999999, 2024-09-11T08:23:24.330 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:23:24.330 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:23:24.330 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:23:24.330 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15087", 2024-09-11T08:23:24.330 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/3299560913", 2024-09-11T08:23:24.331 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 254113 2024-09-11T08:23:24.331 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.331 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:23:24.331 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.331 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:23:24.331 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:32.516600+0000", 2024-09-11T08:23:24.331 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:23:24.331 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.331 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.331 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:23:24.331 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:32.516600+0000", 2024-09-11T08:23:24.331 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:23:24.331 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.332 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.332 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:23:24.332 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:32.516599+0000", 2024-09-11T08:23:24.332 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:23:24.332 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.332 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.332 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:23:24.332 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:32.516604+0000", 2024-09-11T08:23:24.332 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.057e-06 2024-09-11T08:23:24.332 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.333 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.333 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:23:24.333 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:32.516614+0000", 2024-09-11T08:23:24.333 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.7049999999999994e-06 2024-09-11T08:23:24.333 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.333 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.333 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:23:24.333 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:32.516619+0000", 2024-09-11T08:23:24.333 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.8640000000000004e-06 2024-09-11T08:23:24.333 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.333 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.333 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:23:24.333 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:32.516656+0000", 2024-09-11T08:23:24.334 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.6983000000000001e-05 2024-09-11T08:23:24.334 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.334 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.334 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:23:24.334 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:32.516759+0000", 2024-09-11T08:23:24.334 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00010391399999999999 2024-09-11T08:23:24.334 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.334 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.334 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 1,4", 2024-09-11T08:23:24.334 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:32.516869+0000", 2024-09-11T08:23:24.334 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00010964900000000001 2024-09-11T08:23:24.334 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.334 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.334 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:23:24.335 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:32.518985+0000", 2024-09-11T08:23:24.335 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0021154310000000001 2024-09-11T08:23:24.335 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.335 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.335 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:23:24.335 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:32.520229+0000", 2024-09-11T08:23:24.335 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0012447039999999999 2024-09-11T08:23:24.335 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.335 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.335 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:23:24.335 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:32.650412+0000", 2024-09-11T08:23:24.335 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.13018290800000001 2024-09-11T08:23:24.335 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.335 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.336 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:23:24.336 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:32.650579+0000", 2024-09-11T08:23:24.336 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000166782 2024-09-11T08:23:24.336 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.336 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.336 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:23:24.336 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:17:32.650620+0000", 2024-09-11T08:23:24.336 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1097000000000003e-05 2024-09-11T08:23:24.336 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:23:24.336 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:23:24.336 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:23:24.336 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.336 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.336 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15369.0:28035 9.2 9:48f7c719:::benchmark_data_smithi150_133616_object28034:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e252)", 2024-09-11T08:23:24.336 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:18:22.532044+0000", 2024-09-11T08:23:24.337 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 301.79362050600002, 2024-09-11T08:23:24.337 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.138700556, 2024-09-11T08:23:24.337 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:23:24.337 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:23:24.337 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:23:24.337 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15369", 2024-09-11T08:23:24.337 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:23:24.337 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 28035 2024-09-11T08:23:24.337 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.337 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:23:24.337 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.337 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:23:24.337 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:22.532044+0000", 2024-09-11T08:23:24.338 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:23:24.338 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.338 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.338 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:23:24.338 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:22.532044+0000", 2024-09-11T08:23:24.338 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:23:24.338 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.338 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.338 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:23:24.338 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:22.532043+0000", 2024-09-11T08:23:24.338 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:23:24.338 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.338 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.338 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:23:24.339 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:22.532066+0000", 2024-09-11T08:23:24.339 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.3572999999999999e-05 2024-09-11T08:23:24.339 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.339 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.339 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:23:24.339 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:22.532079+0000", 2024-09-11T08:23:24.339 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2116000000000001e-05 2024-09-11T08:23:24.339 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.339 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.339 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:23:24.339 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:22.532085+0000", 2024-09-11T08:23:24.339 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.144e-06 2024-09-11T08:23:24.339 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.339 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.339 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:23:24.340 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:22.532126+0000", 2024-09-11T08:23:24.340 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1316000000000002e-05 2024-09-11T08:23:24.340 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.340 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.340 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:23:24.340 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:22.532220+0000", 2024-09-11T08:23:24.340 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.4155999999999998e-05 2024-09-11T08:23:24.340 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.340 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.340 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 1,2", 2024-09-11T08:23:24.340 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:22.532394+0000", 2024-09-11T08:23:24.340 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00017396800000000001 2024-09-11T08:23:24.340 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.341 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.341 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:23:24.341 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:22.533473+0000", 2024-09-11T08:23:24.341 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0010786470000000001 2024-09-11T08:23:24.341 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.341 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.342 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:23:24.342 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:22.533652+0000", 2024-09-11T08:23:24.342 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000179242 2024-09-11T08:23:24.342 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.342 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.342 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:23:24.342 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:22.670604+0000", 2024-09-11T08:23:24.343 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.13695171 2024-09-11T08:23:24.343 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.343 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.343 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:23:24.343 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:22.670670+0000", 2024-09-11T08:23:24.343 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.6481000000000003e-05 2024-09-11T08:23:24.343 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.343 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.343 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:23:24.344 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:22.670745+0000", 2024-09-11T08:23:24.344 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.4596000000000002e-05 2024-09-11T08:23:24.344 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:23:24.344 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:23:24.344 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:23:24.344 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.344 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.344 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15369.0:30534 9.2 9:4209b46e:::benchmark_data_smithi150_133616_object30533:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e252)", 2024-09-11T08:23:24.345 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:18:23.989103+0000", 2024-09-11T08:23:24.345 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 300.33656207199999, 2024-09-11T08:23:24.345 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.162855108, 2024-09-11T08:23:24.345 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:23:24.345 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:23:24.345 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:23:24.345 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15369", 2024-09-11T08:23:24.345 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:23:24.345 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 30534 2024-09-11T08:23:24.345 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.345 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:23:24.345 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.346 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:23:24.346 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:23.989103+0000", 2024-09-11T08:23:24.346 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:23:24.346 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.346 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.346 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:23:24.347 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:23.989103+0000", 2024-09-11T08:23:24.347 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:23:24.347 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.347 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.347 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:23:24.348 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:23.989101+0000", 2024-09-11T08:23:24.348 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:23:24.348 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.348 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.348 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:23:24.349 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:23.989126+0000", 2024-09-11T08:23:24.349 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.4474e-05 2024-09-11T08:23:24.349 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.349 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.349 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:23:24.349 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:23.989137+0000", 2024-09-11T08:23:24.350 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1109e-05 2024-09-11T08:23:24.350 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.350 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.350 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:23:24.350 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:23.989143+0000", 2024-09-11T08:23:24.350 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.9440000000000002e-06 2024-09-11T08:23:24.351 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.351 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.351 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:23:24.351 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:23.989193+0000", 2024-09-11T08:23:24.351 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.0766000000000001e-05 2024-09-11T08:23:24.352 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.352 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.352 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:23:24.352 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:23.989402+0000", 2024-09-11T08:23:24.352 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000208839 2024-09-11T08:23:24.352 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.353 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.353 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 1,2", 2024-09-11T08:23:24.353 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:23.989548+0000", 2024-09-11T08:23:24.353 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00014581199999999999 2024-09-11T08:23:24.353 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.353 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.353 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:23:24.353 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:23.991428+0000", 2024-09-11T08:23:24.353 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.001880147 2024-09-11T08:23:24.353 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.353 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.353 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:23:24.354 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:23.991525+0000", 2024-09-11T08:23:24.354 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.6718000000000005e-05 2024-09-11T08:23:24.354 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.354 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.354 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:23:24.354 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:24.151867+0000", 2024-09-11T08:23:24.354 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.160342019 2024-09-11T08:23:24.354 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.354 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.354 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:23:24.354 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:24.151917+0000", 2024-09-11T08:23:24.354 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.9935999999999998e-05 2024-09-11T08:23:24.354 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.354 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.355 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:23:24.355 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:18:24.151958+0000", 2024-09-11T08:23:24.355 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.0936000000000002e-05 2024-09-11T08:23:24.355 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:23:24.355 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:23:24.355 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:23:24.355 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.355 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.355 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15369.0:206706 9.2 9:487dc9b7:::benchmark_data_smithi150_133616_object91100:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-09-11T08:23:24.355 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:21:01.165521+0000", 2024-09-11T08:23:24.355 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 143.160144049, 2024-09-11T08:23:24.355 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.186702917, 2024-09-11T08:23:24.355 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:23:24.355 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:23:24.355 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:23:24.356 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15369", 2024-09-11T08:23:24.356 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:23:24.356 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 206706 2024-09-11T08:23:24.356 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.356 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:23:24.356 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.356 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:23:24.356 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.165521+0000", 2024-09-11T08:23:24.356 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:23:24.356 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.356 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.356 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:23:24.356 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.165521+0000", 2024-09-11T08:23:24.356 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:23:24.357 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.357 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.357 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:23:24.357 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.165519+0000", 2024-09-11T08:23:24.357 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:23:24.357 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.357 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.357 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:23:24.357 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.165527+0000", 2024-09-11T08:23:24.357 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.2740000000000007e-06 2024-09-11T08:23:24.357 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.357 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.357 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:23:24.357 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.165541+0000", 2024-09-11T08:23:24.357 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4248e-05 2024-09-11T08:23:24.358 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.358 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.358 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:23:24.358 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.165547+0000", 2024-09-11T08:23:24.358 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.1099999999999999e-06 2024-09-11T08:23:24.358 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.358 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.358 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:23:24.358 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.165614+0000", 2024-09-11T08:23:24.358 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.6278999999999993e-05 2024-09-11T08:23:24.358 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.358 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.358 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:23:24.358 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.165778+0000", 2024-09-11T08:23:24.359 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00016466199999999999 2024-09-11T08:23:24.359 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.359 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.359 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 1,2", 2024-09-11T08:23:24.359 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.166117+0000", 2024-09-11T08:23:24.359 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00033881900000000002 2024-09-11T08:23:24.359 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.359 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.359 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:23:24.359 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.172090+0000", 2024-09-11T08:23:24.359 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.005972471 2024-09-11T08:23:24.359 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.359 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.360 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:23:24.360 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.172241+0000", 2024-09-11T08:23:24.360 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00015164200000000001 2024-09-11T08:23:24.360 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.360 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.360 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:23:24.360 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.352141+0000", 2024-09-11T08:23:24.360 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.179899278 2024-09-11T08:23:24.360 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.360 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.360 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:23:24.360 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.352177+0000", 2024-09-11T08:23:24.360 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.6270999999999999e-05 2024-09-11T08:23:24.360 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.361 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.361 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:23:24.361 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.352224+0000", 2024-09-11T08:23:24.361 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.6900999999999997e-05 2024-09-11T08:23:24.361 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:23:24.361 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:23:24.361 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:23:24.361 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.361 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.361 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15369.0:206708 9.2 9:483f0111:::benchmark_data_smithi150_133616_object91102:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-09-11T08:23:24.361 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:21:01.166444+0000", 2024-09-11T08:23:24.361 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 143.15922075899999, 2024-09-11T08:23:24.361 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.193311078, 2024-09-11T08:23:24.361 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:23:24.361 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:23:24.362 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:23:24.362 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15369", 2024-09-11T08:23:24.362 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:23:24.362 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 206708 2024-09-11T08:23:24.362 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.362 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:23:24.362 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.362 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:23:24.362 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.166444+0000", 2024-09-11T08:23:24.362 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:23:24.362 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.362 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.362 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:23:24.362 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.166444+0000", 2024-09-11T08:23:24.363 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:23:24.363 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.363 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.363 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:23:24.363 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.166443+0000", 2024-09-11T08:23:24.363 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:23:24.363 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.363 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.363 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:23:24.363 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.166449+0000", 2024-09-11T08:23:24.364 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.4249999999999997e-06 2024-09-11T08:23:24.364 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.364 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.364 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:23:24.364 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.166461+0000", 2024-09-11T08:23:24.364 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2509e-05 2024-09-11T08:23:24.365 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.365 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.365 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:23:24.365 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.166466+0000", 2024-09-11T08:23:24.365 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.4240000000000001e-06 2024-09-11T08:23:24.365 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.365 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.365 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:23:24.365 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.168868+0000", 2024-09-11T08:23:24.365 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0024011079999999999 2024-09-11T08:23:24.365 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.365 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.365 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:23:24.365 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.168996+0000", 2024-09-11T08:23:24.366 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012857100000000001 2024-09-11T08:23:24.366 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.366 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.366 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 1,2", 2024-09-11T08:23:24.366 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.169130+0000", 2024-09-11T08:23:24.366 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000133671 2024-09-11T08:23:24.366 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.366 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.366 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:23:24.366 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.172175+0000", 2024-09-11T08:23:24.366 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0030451660000000002 2024-09-11T08:23:24.366 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.366 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.366 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:23:24.367 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.173103+0000", 2024-09-11T08:23:24.367 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000928206 2024-09-11T08:23:24.367 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.367 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.367 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:23:24.367 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.359647+0000", 2024-09-11T08:23:24.367 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.18654348600000001 2024-09-11T08:23:24.367 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.367 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.367 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:23:24.367 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.359711+0000", 2024-09-11T08:23:24.367 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.4467000000000005e-05 2024-09-11T08:23:24.367 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.367 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.368 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:23:24.368 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.359755+0000", 2024-09-11T08:23:24.368 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.4017000000000002e-05 2024-09-11T08:23:24.368 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:23:24.368 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:23:24.368 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:23:24.368 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.368 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.368 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15369.0:206741 9.2 9:410b4dca:::benchmark_data_smithi150_133616_object91135:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-09-11T08:23:24.368 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:21:01.198723+0000", 2024-09-11T08:23:24.368 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 143.12694158299999, 2024-09-11T08:23:24.369 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.16125584100000001, 2024-09-11T08:23:24.369 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:23:24.369 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:23:24.369 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:23:24.369 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15369", 2024-09-11T08:23:24.369 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:23:24.369 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 206741 2024-09-11T08:23:24.369 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.369 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:23:24.369 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.369 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:23:24.370 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.198723+0000", 2024-09-11T08:23:24.370 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:23:24.370 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.370 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.370 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:23:24.370 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.198723+0000", 2024-09-11T08:23:24.370 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:23:24.370 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.370 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.370 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:23:24.371 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.198722+0000", 2024-09-11T08:23:24.371 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:23:24.371 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.371 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.371 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:23:24.371 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.198728+0000", 2024-09-11T08:23:24.371 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.348e-06 2024-09-11T08:23:24.371 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.371 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.371 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:23:24.371 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.198738+0000", 2024-09-11T08:23:24.371 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.8579999999999995e-06 2024-09-11T08:23:24.372 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.372 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.372 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:23:24.372 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.198743+0000", 2024-09-11T08:23:24.372 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.464e-06 2024-09-11T08:23:24.372 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.372 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.372 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:23:24.372 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.198791+0000", 2024-09-11T08:23:24.372 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.8798000000000003e-05 2024-09-11T08:23:24.372 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.372 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.372 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:23:24.372 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.198870+0000", 2024-09-11T08:23:24.373 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.8734999999999997e-05 2024-09-11T08:23:24.373 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.373 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.373 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 1,2", 2024-09-11T08:23:24.373 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.198942+0000", 2024-09-11T08:23:24.373 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.1892999999999998e-05 2024-09-11T08:23:24.373 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.373 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.373 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:23:24.373 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.203943+0000", 2024-09-11T08:23:24.374 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0050011910000000003 2024-09-11T08:23:24.374 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.374 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.374 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:23:24.374 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.206821+0000", 2024-09-11T08:23:24.374 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0028777199999999998 2024-09-11T08:23:24.374 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.374 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.374 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:23:24.375 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.359803+0000", 2024-09-11T08:23:24.375 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.15298236600000001 2024-09-11T08:23:24.375 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.375 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.375 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:23:24.375 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.359849+0000", 2024-09-11T08:23:24.375 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.5385e-05 2024-09-11T08:23:24.375 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.375 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.375 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:23:24.375 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.359979+0000", 2024-09-11T08:23:24.375 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00013032899999999999 2024-09-11T08:23:24.375 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:23:24.375 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:23:24.375 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:23:24.376 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.376 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.376 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15369.0:206745 9.2 9:4f76d45b:::benchmark_data_smithi150_133616_object91139:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-09-11T08:23:24.376 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:21:01.200741+0000", 2024-09-11T08:23:24.376 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 143.124924037, 2024-09-11T08:23:24.377 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.15938677800000001, 2024-09-11T08:23:24.377 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:23:24.377 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:23:24.377 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:23:24.377 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15369", 2024-09-11T08:23:24.377 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:23:24.378 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 206745 2024-09-11T08:23:24.378 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.378 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:23:24.378 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.378 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:23:24.378 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.200741+0000", 2024-09-11T08:23:24.378 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:23:24.379 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.379 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.379 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:23:24.379 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.200741+0000", 2024-09-11T08:23:24.379 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:23:24.379 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.379 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.379 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:23:24.379 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.200740+0000", 2024-09-11T08:23:24.379 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:23:24.380 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.380 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.380 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:23:24.380 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.200746+0000", 2024-09-11T08:23:24.380 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.3810000000000004e-06 2024-09-11T08:23:24.380 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.380 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.380 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:23:24.380 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.200760+0000", 2024-09-11T08:23:24.380 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4229e-05 2024-09-11T08:23:24.380 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.380 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.380 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:23:24.380 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.200766+0000", 2024-09-11T08:23:24.381 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.1530000000000002e-06 2024-09-11T08:23:24.381 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.381 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.381 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:23:24.381 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.201146+0000", 2024-09-11T08:23:24.381 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00037946300000000002 2024-09-11T08:23:24.381 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.381 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.381 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:23:24.381 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.201251+0000", 2024-09-11T08:23:24.381 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00010490800000000001 2024-09-11T08:23:24.381 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.381 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.381 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 1,2", 2024-09-11T08:23:24.382 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.201358+0000", 2024-09-11T08:23:24.382 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000107753 2024-09-11T08:23:24.382 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.382 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.382 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:23:24.382 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.204067+0000", 2024-09-11T08:23:24.382 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0027084959999999999 2024-09-11T08:23:24.382 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.382 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.382 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:23:24.382 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.210515+0000", 2024-09-11T08:23:24.383 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0064477040000000003 2024-09-11T08:23:24.383 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.383 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.383 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:23:24.383 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.360078+0000", 2024-09-11T08:23:24.383 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.149562949 2024-09-11T08:23:24.383 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.383 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.383 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:23:24.383 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.360109+0000", 2024-09-11T08:23:24.383 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.1624000000000001e-05 2024-09-11T08:23:24.383 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.384 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.384 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:23:24.384 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.360128+0000", 2024-09-11T08:23:24.384 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.8328000000000001e-05 2024-09-11T08:23:24.384 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:23:24.384 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:23:24.384 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:23:24.384 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.384 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.384 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15369.0:206750 9.2 9:4ea3965e:::benchmark_data_smithi150_133616_object91144:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-09-11T08:23:24.384 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:21:01.207473+0000", 2024-09-11T08:23:24.384 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 143.11819202699999, 2024-09-11T08:23:24.384 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.15275596, 2024-09-11T08:23:24.384 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:23:24.385 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:23:24.385 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:23:24.385 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15369", 2024-09-11T08:23:24.385 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:23:24.385 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 206750 2024-09-11T08:23:24.385 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.385 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:23:24.385 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.385 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:23:24.385 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.207473+0000", 2024-09-11T08:23:24.385 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:23:24.385 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.385 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.385 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:23:24.385 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.207473+0000", 2024-09-11T08:23:24.386 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:23:24.386 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.386 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.386 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:23:24.386 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.207471+0000", 2024-09-11T08:23:24.386 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:23:24.386 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.386 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.386 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:23:24.386 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.207479+0000", 2024-09-11T08:23:24.386 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.4939999999999997e-06 2024-09-11T08:23:24.386 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.386 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.386 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:23:24.387 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.207491+0000", 2024-09-11T08:23:24.387 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.291e-05 2024-09-11T08:23:24.387 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.387 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.387 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:23:24.387 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.207497+0000", 2024-09-11T08:23:24.387 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.48e-06 2024-09-11T08:23:24.387 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.387 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.387 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:23:24.387 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.207543+0000", 2024-09-11T08:23:24.387 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.6433000000000001e-05 2024-09-11T08:23:24.387 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.387 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.388 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:23:24.388 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.207695+0000", 2024-09-11T08:23:24.388 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00015181400000000001 2024-09-11T08:23:24.388 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.388 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.388 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 1,2", 2024-09-11T08:23:24.388 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.207832+0000", 2024-09-11T08:23:24.388 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00013699099999999999 2024-09-11T08:23:24.388 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.388 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.388 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:23:24.388 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.210992+0000", 2024-09-11T08:23:24.388 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0031602079999999999 2024-09-11T08:23:24.388 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.388 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.389 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:23:24.389 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.213173+0000", 2024-09-11T08:23:24.389 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0021802890000000002 2024-09-11T08:23:24.389 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.389 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.389 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:23:24.389 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.360164+0000", 2024-09-11T08:23:24.389 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.14699149 2024-09-11T08:23:24.389 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.389 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.389 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:23:24.389 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.360191+0000", 2024-09-11T08:23:24.389 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.7214999999999999e-05 2024-09-11T08:23:24.389 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.390 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.390 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:23:24.390 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.360229+0000", 2024-09-11T08:23:24.390 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.7336000000000003e-05 2024-09-11T08:23:24.390 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:23:24.390 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:23:24.390 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:23:24.390 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.390 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.390 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15369.0:206762 9.2 9:47575c41:::benchmark_data_smithi150_133616_object91156:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-09-11T08:23:24.390 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:21:01.217687+0000", 2024-09-11T08:23:24.390 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 143.10797757899999, 2024-09-11T08:23:24.390 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.14270024100000001, 2024-09-11T08:23:24.390 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:23:24.390 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:23:24.391 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:23:24.391 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15369", 2024-09-11T08:23:24.391 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:23:24.391 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 206762 2024-09-11T08:23:24.391 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.391 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:23:24.391 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.391 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:23:24.391 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.217687+0000", 2024-09-11T08:23:24.391 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:23:24.391 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.391 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.391 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:23:24.391 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.217687+0000", 2024-09-11T08:23:24.392 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:23:24.392 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.392 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.392 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:23:24.392 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.217685+0000", 2024-09-11T08:23:24.392 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:23:24.392 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.392 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.392 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:23:24.392 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.217693+0000", 2024-09-11T08:23:24.392 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.6299999999999998e-06 2024-09-11T08:23:24.392 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.392 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.392 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:23:24.393 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.217707+0000", 2024-09-11T08:23:24.393 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4535000000000001e-05 2024-09-11T08:23:24.393 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.393 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.393 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:23:24.393 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.217714+0000", 2024-09-11T08:23:24.393 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.6939999999999998e-06 2024-09-11T08:23:24.393 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.393 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.393 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:23:24.393 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.217764+0000", 2024-09-11T08:23:24.394 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.9951000000000003e-05 2024-09-11T08:23:24.394 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.394 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.394 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:23:24.394 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.217893+0000", 2024-09-11T08:23:24.394 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012909799999999999 2024-09-11T08:23:24.394 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.394 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.394 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 1,2", 2024-09-11T08:23:24.394 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.218011+0000", 2024-09-11T08:23:24.394 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00011834599999999999 2024-09-11T08:23:24.394 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.394 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.394 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:23:24.395 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.220362+0000", 2024-09-11T08:23:24.395 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0023508579999999999 2024-09-11T08:23:24.395 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.395 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.395 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:23:24.395 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.221363+0000", 2024-09-11T08:23:24.395 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.001000672 2024-09-11T08:23:24.395 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.395 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.395 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:23:24.395 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.360291+0000", 2024-09-11T08:23:24.395 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.13892821499999999 2024-09-11T08:23:24.395 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.395 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.396 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:23:24.396 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.360355+0000", 2024-09-11T08:23:24.396 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.3732999999999994e-05 2024-09-11T08:23:24.396 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.396 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.396 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:23:24.396 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:01.360387+0000", 2024-09-11T08:23:24.396 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.2524000000000002e-05 2024-09-11T08:23:24.396 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:23:24.396 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:23:24.396 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:23:24.396 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.396 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.396 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15369.0:212789 9.2 9:4d7a08f9:::benchmark_data_smithi150_133616_object97183:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-11T08:23:24.397 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:21:06.627483+0000", 2024-09-11T08:23:24.397 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 137.698181852, 2024-09-11T08:23:24.397 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.19118728300000001, 2024-09-11T08:23:24.397 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:23:24.397 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:23:24.397 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:23:24.397 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15369", 2024-09-11T08:23:24.397 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:23:24.397 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 212789 2024-09-11T08:23:24.397 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.397 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:23:24.397 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.397 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:23:24.397 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.627483+0000", 2024-09-11T08:23:24.397 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:23:24.398 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.398 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.398 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:23:24.398 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.627483+0000", 2024-09-11T08:23:24.398 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:23:24.398 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.398 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.398 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:23:24.398 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.627481+0000", 2024-09-11T08:23:24.398 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:23:24.398 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.398 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.398 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:23:24.398 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.627488+0000", 2024-09-11T08:23:24.399 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.9789999999999996e-06 2024-09-11T08:23:24.399 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.399 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.399 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:23:24.399 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.627505+0000", 2024-09-11T08:23:24.399 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.6874000000000001e-05 2024-09-11T08:23:24.399 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.399 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.399 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:23:24.399 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.627511+0000", 2024-09-11T08:23:24.399 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.8119999999999996e-06 2024-09-11T08:23:24.399 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.399 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.399 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:23:24.400 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.627556+0000", 2024-09-11T08:23:24.400 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.5090999999999999e-05 2024-09-11T08:23:24.400 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.400 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.400 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:23:24.400 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.627667+0000", 2024-09-11T08:23:24.400 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000110821 2024-09-11T08:23:24.400 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.400 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.400 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 1,2", 2024-09-11T08:23:24.400 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.627805+0000", 2024-09-11T08:23:24.400 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000138431 2024-09-11T08:23:24.400 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.400 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.400 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:23:24.401 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.631079+0000", 2024-09-11T08:23:24.401 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.003274534 2024-09-11T08:23:24.401 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.401 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.401 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:23:24.401 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.631371+0000", 2024-09-11T08:23:24.401 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00029168500000000002 2024-09-11T08:23:24.401 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.401 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.401 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:23:24.401 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.818586+0000", 2024-09-11T08:23:24.401 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.187214977 2024-09-11T08:23:24.401 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.401 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.402 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:23:24.402 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.818634+0000", 2024-09-11T08:23:24.402 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.7420999999999999e-05 2024-09-11T08:23:24.402 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.402 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.402 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:23:24.402 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.818670+0000", 2024-09-11T08:23:24.402 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.6695e-05 2024-09-11T08:23:24.402 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:23:24.402 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:23:24.402 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:23:24.402 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.403 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.403 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15369.0:212795 9.2 9:4db65d24:::benchmark_data_smithi150_133616_object97189:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-11T08:23:24.403 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:21:06.637343+0000", 2024-09-11T08:23:24.403 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 137.68832168099999, 2024-09-11T08:23:24.403 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.181466457, 2024-09-11T08:23:24.403 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:23:24.403 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:23:24.403 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:23:24.403 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15369", 2024-09-11T08:23:24.403 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:23:24.403 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 212795 2024-09-11T08:23:24.403 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.403 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:23:24.403 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.404 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:23:24.404 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.637343+0000", 2024-09-11T08:23:24.404 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:23:24.404 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.404 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.404 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:23:24.404 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.637343+0000", 2024-09-11T08:23:24.404 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:23:24.404 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.404 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.404 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:23:24.404 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.637342+0000", 2024-09-11T08:23:24.404 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:23:24.404 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.405 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.405 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:23:24.405 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.637347+0000", 2024-09-11T08:23:24.405 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.3310000000000003e-06 2024-09-11T08:23:24.405 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.405 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.405 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:23:24.405 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.637357+0000", 2024-09-11T08:23:24.405 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0028e-05 2024-09-11T08:23:24.405 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.405 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.405 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:23:24.405 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.637363+0000", 2024-09-11T08:23:24.405 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.1919999999999996e-06 2024-09-11T08:23:24.406 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.406 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.406 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:23:24.406 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.637405+0000", 2024-09-11T08:23:24.406 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.2191999999999999e-05 2024-09-11T08:23:24.406 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.406 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.406 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:23:24.406 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.637485+0000", 2024-09-11T08:23:24.406 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.9784999999999996e-05 2024-09-11T08:23:24.406 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.406 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.406 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 1,2", 2024-09-11T08:23:24.406 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.637554+0000", 2024-09-11T08:23:24.406 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.9801999999999993e-05 2024-09-11T08:23:24.407 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.407 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.407 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:23:24.407 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.639535+0000", 2024-09-11T08:23:24.407 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0019807660000000001 2024-09-11T08:23:24.407 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.407 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.407 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:23:24.407 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.639728+0000", 2024-09-11T08:23:24.407 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00019297699999999999 2024-09-11T08:23:24.407 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.408 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.408 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:23:24.408 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.818714+0000", 2024-09-11T08:23:24.408 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.17898599600000001 2024-09-11T08:23:24.408 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.408 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.408 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:23:24.408 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.818751+0000", 2024-09-11T08:23:24.408 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.6847999999999997e-05 2024-09-11T08:23:24.408 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.408 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.408 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:23:24.409 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.818810+0000", 2024-09-11T08:23:24.409 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.8695e-05 2024-09-11T08:23:24.409 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:23:24.409 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:23:24.409 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:23:24.409 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.409 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.410 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.15636.0:27718 10.7 e314/308 10:e40ef5a8:::benchmark_data_smithi150_158067_object27717:head v 314'1741, mlcod=314'1739)", 2024-09-11T08:23:24.410 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:22:10.928758+0000", 2024-09-11T08:23:24.410 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 73.396906893999997, 2024-09-11T08:23:24.410 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.18417624699999999, 2024-09-11T08:23:24.410 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:23:24.410 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:23:24.410 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:23:24.410 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.410 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:23:24.410 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:10.928758+0000", 2024-09-11T08:23:24.410 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:23:24.411 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.411 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.411 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:23:24.411 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:10.928758+0000", 2024-09-11T08:23:24.411 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:23:24.411 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.411 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.411 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:23:24.411 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:10.928757+0000", 2024-09-11T08:23:24.411 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:23:24.411 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.411 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.411 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:23:24.411 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:10.928814+0000", 2024-09-11T08:23:24.412 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.6786000000000001e-05 2024-09-11T08:23:24.412 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.412 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.412 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:23:24.412 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:10.928838+0000", 2024-09-11T08:23:24.412 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.4233999999999999e-05 2024-09-11T08:23:24.412 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.412 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.412 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:23:24.412 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:10.928844+0000", 2024-09-11T08:23:24.412 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.716e-06 2024-09-11T08:23:24.412 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.412 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.412 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:23:24.413 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:10.928926+0000", 2024-09-11T08:23:24.413 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.1709e-05 2024-09-11T08:23:24.413 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.413 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.413 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:23:24.413 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:10.928944+0000", 2024-09-11T08:23:24.413 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.8318999999999999e-05 2024-09-11T08:23:24.413 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.413 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.413 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:23:24.413 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:10.931635+0000", 2024-09-11T08:23:24.413 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0026910419999999998 2024-09-11T08:23:24.413 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.413 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.414 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:23:24.414 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:11.112934+0000", 2024-09-11T08:23:24.414 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.181299301 2024-09-11T08:23:24.414 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:23:24.414 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:23:24.414 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:23:24.414 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.414 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.414 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.15636.0:27720 10.3 e314/308 10:cade902a:::benchmark_data_smithi150_158067_object27719:head v 314'1774, mlcod=314'1772)", 2024-09-11T08:23:24.414 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:22:10.930296+0000", 2024-09-11T08:23:24.414 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 73.395368637999994, 2024-09-11T08:23:24.414 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.18380169399999999, 2024-09-11T08:23:24.414 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:23:24.414 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:23:24.415 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:23:24.415 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.415 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:23:24.415 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:10.930296+0000", 2024-09-11T08:23:24.415 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:23:24.415 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.415 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.415 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:23:24.415 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:10.930296+0000", 2024-09-11T08:23:24.415 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:23:24.415 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.416 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.416 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:23:24.416 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:10.930295+0000", 2024-09-11T08:23:24.416 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:23:24.416 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.416 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.416 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:23:24.416 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:10.930328+0000", 2024-09-11T08:23:24.416 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.2753999999999998e-05 2024-09-11T08:23:24.416 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.416 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.416 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:23:24.417 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:10.930338+0000", 2024-09-11T08:23:24.417 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0356999999999999e-05 2024-09-11T08:23:24.417 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.417 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.417 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:23:24.417 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:10.930344+0000", 2024-09-11T08:23:24.417 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.4410000000000001e-06 2024-09-11T08:23:24.417 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.417 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.417 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:23:24.417 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:10.931123+0000", 2024-09-11T08:23:24.417 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00077935700000000005 2024-09-11T08:23:24.417 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.417 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.418 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:23:24.418 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:10.931142+0000", 2024-09-11T08:23:24.418 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.9117e-05 2024-09-11T08:23:24.418 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.418 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.418 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:23:24.418 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:11.114057+0000", 2024-09-11T08:23:24.418 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.182914944 2024-09-11T08:23:24.418 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.418 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.418 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:23:24.418 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:11.114098+0000", 2024-09-11T08:23:24.418 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.0550000000000003e-05 2024-09-11T08:23:24.418 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:23:24.419 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:23:24.419 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:23:24.419 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.419 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.419 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.15636.0:27725 10.3 e314/308 10:c261e12e:::benchmark_data_smithi150_158067_object27724:head v 314'1775, mlcod=314'1772)", 2024-09-11T08:23:24.419 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:22:10.932199+0000", 2024-09-11T08:23:24.419 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 73.393466111999999, 2024-09-11T08:23:24.419 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.18531141800000001, 2024-09-11T08:23:24.419 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:23:24.419 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:23:24.419 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:23:24.419 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.419 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:23:24.419 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:10.932199+0000", 2024-09-11T08:23:24.420 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:23:24.420 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.420 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.420 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:23:24.420 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:10.932199+0000", 2024-09-11T08:23:24.420 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:23:24.420 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.420 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.420 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:23:24.420 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:10.932198+0000", 2024-09-11T08:23:24.420 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:23:24.420 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.420 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.420 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:23:24.421 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:10.932222+0000", 2024-09-11T08:23:24.421 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.3646e-05 2024-09-11T08:23:24.421 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.421 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.421 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:23:24.421 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:11.112912+0000", 2024-09-11T08:23:24.421 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.18069068999999999 2024-09-11T08:23:24.421 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.421 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.421 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:23:24.421 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:11.112921+0000", 2024-09-11T08:23:24.421 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.4770000000000003e-06 2024-09-11T08:23:24.421 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.421 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.422 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:23:24.422 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:11.112957+0000", 2024-09-11T08:23:24.422 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.6251000000000002e-05 2024-09-11T08:23:24.422 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.422 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.422 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:23:24.422 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:11.112981+0000", 2024-09-11T08:23:24.422 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.4502e-05 2024-09-11T08:23:24.422 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.422 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.422 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:23:24.422 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:11.117424+0000", 2024-09-11T08:23:24.422 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0044426429999999996 2024-09-11T08:23:24.422 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.422 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.423 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:23:24.423 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:11.117510+0000", 2024-09-11T08:23:24.423 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.5996999999999997e-05 2024-09-11T08:23:24.423 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:23:24.423 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:23:24.423 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:23:24.423 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.423 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.423 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.15636.0:27728 10.7 e314/308 10:e54d94c8:::benchmark_data_smithi150_158067_object27727:head v 314'1742, mlcod=314'1739)", 2024-09-11T08:23:24.423 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:22:10.932677+0000", 2024-09-11T08:23:24.423 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 73.392987464000001, 2024-09-11T08:23:24.423 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.18333144500000001, 2024-09-11T08:23:24.423 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:23:24.423 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:23:24.424 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:23:24.424 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.424 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:23:24.424 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:23:24.424 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:10.932677+0000", 2024-09-11T08:23:24.425 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:23:24.425 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.425 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.425 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:23:24.425 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:10.932677+0000", 2024-09-11T08:23:24.425 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:23:24.425 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.425 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.425 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:23:24.425 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:10.932670+0000", 2024-09-11T08:23:24.425 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999928 2024-09-11T08:23:24.425 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.425 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.425 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:23:24.426 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:10.932706+0000", 2024-09-11T08:23:24.426 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.5741000000000002e-05 2024-09-11T08:23:24.426 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.426 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.426 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:23:24.426 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:11.112907+0000", 2024-09-11T08:23:24.426 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.18020065599999999 2024-09-11T08:23:24.426 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.426 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.426 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:23:24.426 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:11.112920+0000", 2024-09-11T08:23:24.426 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3429e-05 2024-09-11T08:23:24.426 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.426 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.427 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:23:24.427 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:11.112963+0000", 2024-09-11T08:23:24.427 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.2861000000000002e-05 2024-09-11T08:23:24.427 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.427 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.427 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:23:24.427 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:11.112983+0000", 2024-09-11T08:23:24.427 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.9660000000000002e-05 2024-09-11T08:23:24.427 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.427 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.427 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:23:24.427 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:11.115955+0000", 2024-09-11T08:23:24.427 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0029720160000000001 2024-09-11T08:23:24.427 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.428 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.428 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:23:24.428 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:11.116009+0000", 2024-09-11T08:23:24.428 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.4221000000000001e-05 2024-09-11T08:23:24.428 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:23:24.428 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:23:24.428 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:23:24.428 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.428 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.428 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.15636.0:27736 10.4 e314/308 10:21da539c:::benchmark_data_smithi150_158067_object27735:head v 314'1762, mlcod=314'1760)", 2024-09-11T08:23:24.428 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:22:10.936659+0000", 2024-09-11T08:23:24.428 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 73.389005417999996, 2024-09-11T08:23:24.428 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.178211059, 2024-09-11T08:23:24.428 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:23:24.429 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:23:24.429 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:23:24.429 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.429 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:23:24.429 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:10.936659+0000", 2024-09-11T08:23:24.429 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:23:24.429 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.429 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.429 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:23:24.429 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:10.936659+0000", 2024-09-11T08:23:24.429 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:23:24.429 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.429 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.430 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:23:24.430 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:10.936658+0000", 2024-09-11T08:23:24.430 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:23:24.430 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.430 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.430 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:23:24.430 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:10.936715+0000", 2024-09-11T08:23:24.430 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.6538e-05 2024-09-11T08:23:24.430 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.430 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.430 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:23:24.430 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:11.112908+0000", 2024-09-11T08:23:24.430 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.176193604 2024-09-11T08:23:24.430 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.431 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.431 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:23:24.431 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:11.112919+0000", 2024-09-11T08:23:24.431 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1158999999999999e-05 2024-09-11T08:23:24.431 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.431 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.431 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:23:24.431 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:11.113120+0000", 2024-09-11T08:23:24.431 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00020045999999999999 2024-09-11T08:23:24.431 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.431 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.431 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:23:24.431 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:11.113140+0000", 2024-09-11T08:23:24.431 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.0440000000000001e-05 2024-09-11T08:23:24.431 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.432 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.432 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:23:24.432 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:11.114795+0000", 2024-09-11T08:23:24.432 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.001654378 2024-09-11T08:23:24.432 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.432 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.432 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:23:24.432 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:11.114870+0000", 2024-09-11T08:23:24.432 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.5710999999999994e-05 2024-09-11T08:23:24.432 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:23:24.432 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:23:24.432 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:23:24.433 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.433 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.433 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.15636.0:96183 10.4 e328/308 10:240e877a:::benchmark_data_smithi150_158067_object96182:head v 328'6018, mlcod=328'6016)", 2024-09-11T08:23:24.433 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:22:57.540252+0000", 2024-09-11T08:23:24.433 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 26.785412829999999, 2024-09-11T08:23:24.433 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.17433538900000001, 2024-09-11T08:23:24.433 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:23:24.433 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:23:24.433 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:23:24.434 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.434 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:23:24.434 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.540252+0000", 2024-09-11T08:23:24.434 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:23:24.434 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.434 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.434 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:23:24.434 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.540252+0000", 2024-09-11T08:23:24.434 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:23:24.435 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.435 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.435 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:23:24.435 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.540251+0000", 2024-09-11T08:23:24.435 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:23:24.435 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.435 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.435 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:23:24.436 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.540275+0000", 2024-09-11T08:23:24.436 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.4379000000000002e-05 2024-09-11T08:23:24.436 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.436 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.436 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:23:24.436 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.540286+0000", 2024-09-11T08:23:24.436 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0369e-05 2024-09-11T08:23:24.436 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.436 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.437 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:23:24.437 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.540291+0000", 2024-09-11T08:23:24.437 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.2619999999999998e-06 2024-09-11T08:23:24.437 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.437 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.437 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:23:24.437 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.540347+0000", 2024-09-11T08:23:24.437 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.6276000000000001e-05 2024-09-11T08:23:24.438 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.438 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.438 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:23:24.438 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.540380+0000", 2024-09-11T08:23:24.438 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.294e-05 2024-09-11T08:23:24.438 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.438 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.438 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:23:24.438 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.714539+0000", 2024-09-11T08:23:24.439 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.17415872700000001 2024-09-11T08:23:24.439 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.439 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.439 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:23:24.439 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.714587+0000", 2024-09-11T08:23:24.439 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.8313999999999999e-05 2024-09-11T08:23:24.439 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:23:24.439 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:23:24.439 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:23:24.440 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.440 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.440 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.15636.0:96188 10.4 e328/308 10:2c3fee1d:::benchmark_data_smithi150_158067_object96187:head v 328'6019, mlcod=328'6016)", 2024-09-11T08:23:24.440 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:22:57.543615+0000", 2024-09-11T08:23:24.440 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 26.782050135999999, 2024-09-11T08:23:24.440 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.173366145, 2024-09-11T08:23:24.440 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:23:24.440 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:23:24.441 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:23:24.441 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.441 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:23:24.441 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.543615+0000", 2024-09-11T08:23:24.441 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:23:24.441 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.441 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.441 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:23:24.441 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.543615+0000", 2024-09-11T08:23:24.442 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:23:24.442 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.442 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.442 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:23:24.442 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.543614+0000", 2024-09-11T08:23:24.442 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:23:24.442 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.442 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.443 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:23:24.443 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.543664+0000", 2024-09-11T08:23:24.443 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.0163000000000001e-05 2024-09-11T08:23:24.443 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.443 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.443 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:23:24.443 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.704581+0000", 2024-09-11T08:23:24.444 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.16091766499999999 2024-09-11T08:23:24.444 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.444 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.444 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:23:24.444 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.704595+0000", 2024-09-11T08:23:24.444 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3852999999999999e-05 2024-09-11T08:23:24.444 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.444 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.444 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:23:24.445 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.707294+0000", 2024-09-11T08:23:24.445 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0026990109999999999 2024-09-11T08:23:24.445 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.445 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.445 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:23:24.445 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.707314+0000", 2024-09-11T08:23:24.445 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.0095999999999999e-05 2024-09-11T08:23:24.445 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.446 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.446 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:23:24.446 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.716942+0000", 2024-09-11T08:23:24.446 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0096280459999999995 2024-09-11T08:23:24.446 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.446 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.446 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:23:24.446 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.716981+0000", 2024-09-11T08:23:24.446 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.8393999999999999e-05 2024-09-11T08:23:24.447 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:23:24.447 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:23:24.447 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:23:24.447 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.447 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.447 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "replica_scrub(pg: 10.f,from:0'0,to:326'5631,epoch:328/308,start:10:fd017469:::benchmark_data_smithi150_158067_object87751:0,end:10:fd14bd55:::benchmark_data_smithi150_158067_object72690:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-11T08:23:24.447 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:22:57.544024+0000", 2024-09-11T08:23:24.447 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 26.781640727999999, 2024-09-11T08:23:24.447 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.16101012100000001, 2024-09-11T08:23:24.447 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:23:24.447 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "started", 2024-09-11T08:23:24.447 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:23:24.447 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.448 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:23:24.448 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.544024+0000", 2024-09-11T08:23:24.448 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:23:24.448 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.448 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.448 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:23:24.448 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.544024+0000", 2024-09-11T08:23:24.448 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:23:24.448 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.448 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.448 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:23:24.448 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.544023+0000", 2024-09-11T08:23:24.448 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:23:24.448 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.449 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.449 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:23:24.449 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.544031+0000", 2024-09-11T08:23:24.449 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.8949999999999998e-06 2024-09-11T08:23:24.449 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.449 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.449 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:23:24.449 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.704587+0000", 2024-09-11T08:23:24.449 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.16055534799999999 2024-09-11T08:23:24.449 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.449 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.449 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:23:24.449 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.704598+0000", 2024-09-11T08:23:24.449 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1759999999999999e-05 2024-09-11T08:23:24.450 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.450 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.450 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:23:24.450 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.705002+0000", 2024-09-11T08:23:24.450 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00040374299999999997 2024-09-11T08:23:24.450 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.450 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.450 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:23:24.450 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.705012+0000", 2024-09-11T08:23:24.450 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0162e-05 2024-09-11T08:23:24.450 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.450 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.450 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:23:24.450 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.705034+0000", 2024-09-11T08:23:24.451 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.2036999999999999e-05 2024-09-11T08:23:24.451 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:23:24.451 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:23:24.451 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:23:24.451 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.451 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.451 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.15636.0:96191 10.3 e328/308 10:c11ce886:::benchmark_data_smithi150_158067_object96190:head v 328'5964, mlcod=328'5962)", 2024-09-11T08:23:24.451 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:22:57.544935+0000", 2024-09-11T08:23:24.451 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 26.780729522000001, 2024-09-11T08:23:24.451 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.16793745900000001, 2024-09-11T08:23:24.451 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:23:24.451 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:23:24.452 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:23:24.452 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.452 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:23:24.452 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.544935+0000", 2024-09-11T08:23:24.452 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:23:24.452 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.452 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.452 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:23:24.452 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.544935+0000", 2024-09-11T08:23:24.452 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:23:24.452 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.452 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.452 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:23:24.453 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.544934+0000", 2024-09-11T08:23:24.453 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:23:24.453 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.453 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.453 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:23:24.453 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.544981+0000", 2024-09-11T08:23:24.453 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.6576000000000003e-05 2024-09-11T08:23:24.453 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.453 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.453 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:23:24.453 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.704583+0000", 2024-09-11T08:23:24.453 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.15960197800000001 2024-09-11T08:23:24.453 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.453 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.454 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:23:24.454 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.704595+0000", 2024-09-11T08:23:24.454 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1793999999999999e-05 2024-09-11T08:23:24.454 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.454 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.454 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:23:24.454 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.704635+0000", 2024-09-11T08:23:24.454 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.9768999999999998e-05 2024-09-11T08:23:24.454 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.454 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.454 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:23:24.454 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.704654+0000", 2024-09-11T08:23:24.454 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.9130000000000001e-05 2024-09-11T08:23:24.454 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.454 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.455 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:23:24.455 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.712785+0000", 2024-09-11T08:23:24.455 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0081307970000000004 2024-09-11T08:23:24.455 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:23:24.455 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:23:24.455 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:23:24.455 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:57.712873+0000", 2024-09-11T08:23:24.455 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.8220000000000006e-05 2024-09-11T08:23:24.455 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:23:24.455 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:23:24.455 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:23:24.455 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:23:24.455 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:23:24.455 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-11T08:23:24.456 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:23:24.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:24.489 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:24.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:23:24.590 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:24.629 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:24.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:24.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:24.731 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:24.733 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T08:23:24.768 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:24.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:24.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:24.870 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:23:24.899 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:24.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:24.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:23:25.001 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:23:25.033 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:25.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:25.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:23:25.039 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 84 16 123703 123687 92.0134 64.125 0.0159518 0.010864 2024-09-11T08:23:25.135 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:23:25.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:25.168 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:25.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:23:25.269 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:25.312 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:25.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:25.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:25.414 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:23:25.448 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:25.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:25.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:23:25.551 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:25.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:23:25 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:23:25] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:23:25.591 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:25.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:25.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:25.693 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:23:25.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:25.734 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:25.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:23:25.835 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:25.878 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:25.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:25.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:25.979 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:26.018 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:26.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:26.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:26.039 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 85 16 124885 124869 91.7999 73.875 0.00829039 0.0108889 2024-09-11T08:23:26.120 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:23:26.127 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-11T08:23:26.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:26.159 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:26.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:23:26.261 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:26.301 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:26.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:26.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:26.377 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:26 smithi150 ceph-mon[29276]: pgmap v727: 148 pgs: 148 active+clean; 7.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 48 MiB/s wr, 763 op/s 2024-09-11T08:23:26.377 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:26 smithi150 ceph-mon[39612]: pgmap v727: 148 pgs: 148 active+clean; 7.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 48 MiB/s wr, 763 op/s 2024-09-11T08:23:26.403 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:23:26.447 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:26.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:26.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:23:26.549 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:26.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:26 smithi181 ceph-mon[32701]: pgmap v727: 148 pgs: 148 active+clean; 7.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 48 MiB/s wr, 763 op/s 2024-09-11T08:23:26.581 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:26.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:26.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:26.645 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:23:26.683 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:23:26.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:26.729 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:26.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:23:26.831 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:23:26.866 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:26.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:26.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:23:26.967 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:27.007 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:27.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:27.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:27.041 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 86 16 125994 125978 91.5384 69.3125 0.016872 0.01092 2024-09-11T08:23:27.109 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:27.164 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:27.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:27.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:27.267 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:27.307 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:27.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:27.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:27.438 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:23:27.477 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:27.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:27.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:23:27.579 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:23:27.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:27.608 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:27.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:23:27.709 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:27.751 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:27.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:27.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:27.847 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:23:27.853 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:27.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:27.910 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:27.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:28.011 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:28.039 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 87 16 127023 127007 91.2254 64.3125 0.0203345 0.0109574 2024-09-11T08:23:28.050 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:28.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:28.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:28.152 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:23:28.191 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:28.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:28.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:23:28.293 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:23:28.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:28.328 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:28.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:23:28.429 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:23:28.430 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:28 smithi181 ceph-mon[32701]: pgmap v728: 148 pgs: 148 active+clean; 7.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 36 MiB/s wr, 579 op/s 2024-09-11T08:23:28.458 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:28 smithi150 ceph-mon[29276]: pgmap v728: 148 pgs: 148 active+clean; 7.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 36 MiB/s wr, 579 op/s 2024-09-11T08:23:28.458 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:28 smithi150 ceph-mon[39612]: pgmap v728: 148 pgs: 148 active+clean; 7.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 36 MiB/s wr, 579 op/s 2024-09-11T08:23:28.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:28.487 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:28.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:23:28.589 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:28.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:28.627 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:28.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:28.681 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:23:28 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:23:28.710 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:23:28 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:23:28] "GET /metrics HTTP/1.1" 200 224650 "" "Prometheus/2.43.0" 2024-09-11T08:23:28.727 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:23:28.759 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:28.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:28.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:23:28.782 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-11T08:23:28.782 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T08:23:28.783 INFO:tasks.thrashosds.thrasher:Killing osd 5, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-11T08:23:28.783 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-11T08:23:28.783 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-c1bddff4-7013-11ef-bcea-c7b262605968@osd.5 2024-09-11T08:23:28.862 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:28.900 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:28.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:28.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:28.993 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:23:28 smithi181 systemd[1]: Stopping Ceph osd.5 for c1bddff4-7013-11ef-bcea-c7b262605968... 2024-09-11T08:23:29.003 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:29.036 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:29.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:29.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:29.039 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 88 16 128133 128117 90.9771 69.375 0.0139797 0.0109872 2024-09-11T08:23:29.138 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:23:29.141 INFO:teuthology.orchestra.run.smithi150.stderr:noscrub is unset 2024-09-11T08:23:29.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:29.183 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:29.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:23:29.275 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:29 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3734552934' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:23:29.283 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:23:29.317 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:29.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:29.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:23:29.420 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:29.459 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:29.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:29.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:29.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:29 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3734552934' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:23:29.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:29 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3734552934' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:23:29.561 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:29.598 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:29.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:29.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:29.700 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:23:29.738 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:29.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:29.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:23:29.762 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-11T08:23:29.762 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-09-11T08:23:29.762 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:29.763 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.15636.0:128231 10.0 e338/308 10:02d913ac:::benchmark_data_smithi150_158067_object128230:head v 338'7895, mlcod=338'7893)", 2024-09-11T08:23:29.763 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:23:29.135745+0000", 2024-09-11T08:23:29.763 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.010884738999999999, 2024-09-11T08:23:29.763 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.010911077999999999, 2024-09-11T08:23:29.763 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:23:29.763 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-11T08:23:29.763 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:23:29.763 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:29.763 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:23:29.763 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:29.135745+0000", 2024-09-11T08:23:29.763 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:23:29.763 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:29.763 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:29.764 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:23:29.764 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:29.135745+0000", 2024-09-11T08:23:29.764 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:23:29.764 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:29.764 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:29.764 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:23:29.764 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:29.135745+0000", 2024-09-11T08:23:29.764 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-09-11T08:23:29.764 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:29.764 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:29.764 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:23:29.764 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:29.135783+0000", 2024-09-11T08:23:29.764 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8766000000000001e-05 2024-09-11T08:23:29.764 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:29.765 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:29.765 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:23:29.765 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:29.135798+0000", 2024-09-11T08:23:29.765 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4973e-05 2024-09-11T08:23:29.765 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:29.765 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:29.765 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:23:29.765 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:29.135804+0000", 2024-09-11T08:23:29.765 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.896e-06 2024-09-11T08:23:29.765 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:29.765 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:29.765 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:23:29.765 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:29.135898+0000", 2024-09-11T08:23:29.766 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.3941000000000001e-05 2024-09-11T08:23:29.766 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:29.766 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:29.766 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:23:29.766 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:29.135917+0000", 2024-09-11T08:23:29.766 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8697000000000001e-05 2024-09-11T08:23:29.766 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:23:29.766 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:23:29.766 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:23:29.766 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:29.766 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:29.766 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.15636.0:128225 10.e e338/337 10:732d77b2:::benchmark_data_smithi150_158067_object128224:head v 338'7897, mlcod=338'7893)", 2024-09-11T08:23:29.766 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:23:29.135816+0000", 2024-09-11T08:23:29.767 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.010813644000000001, 2024-09-11T08:23:29.767 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.010927849, 2024-09-11T08:23:29.767 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:23:29.767 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-11T08:23:29.767 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:23:29.767 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:29.767 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:23:29.767 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:29.135816+0000", 2024-09-11T08:23:29.767 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:23:29.767 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:29.767 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:29.767 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:23:29.767 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:29.135816+0000", 2024-09-11T08:23:29.768 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:23:29.768 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:29.768 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:29.768 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:23:29.768 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:29.135816+0000", 2024-09-11T08:23:29.768 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-09-11T08:23:29.768 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:29.768 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:29.768 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:23:29.768 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:29.135835+0000", 2024-09-11T08:23:29.768 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8972999999999999e-05 2024-09-11T08:23:29.768 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:29.768 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:29.768 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:23:29.769 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:29.135841+0000", 2024-09-11T08:23:29.769 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7790000000000001e-06 2024-09-11T08:23:29.769 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:29.769 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:29.769 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:23:29.769 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:29.135845+0000", 2024-09-11T08:23:29.769 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9700000000000001e-06 2024-09-11T08:23:29.769 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:29.769 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:29.769 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:23:29.769 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:29.139294+0000", 2024-09-11T08:23:29.769 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003449349 2024-09-11T08:23:29.769 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:29.770 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:29.770 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:23:29.770 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:29.139313+0000", 2024-09-11T08:23:29.770 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8561999999999999e-05 2024-09-11T08:23:29.770 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:23:29.770 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:23:29.770 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:23:29.770 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:29.770 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:29.770 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.15636.0:128233 10.4 e338/308 10:2aa6eb6b:::benchmark_data_smithi150_158067_object128232:head v 338'8072, mlcod=338'8070)", 2024-09-11T08:23:29.770 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:23:29.137213+0000", 2024-09-11T08:23:29.770 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.0094165729999999993, 2024-09-11T08:23:29.770 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0096096069999999992, 2024-09-11T08:23:29.770 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:23:29.771 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-11T08:23:29.771 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:23:29.771 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:29.771 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:23:29.771 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:29.137213+0000", 2024-09-11T08:23:29.771 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:23:29.771 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:29.771 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:29.771 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:23:29.771 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:29.137213+0000", 2024-09-11T08:23:29.771 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:23:29.771 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:29.771 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:29.772 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:23:29.772 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:29.137213+0000", 2024-09-11T08:23:29.772 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:23:29.772 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:29.772 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:29.772 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:23:29.772 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:29.137243+0000", 2024-09-11T08:23:29.772 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0239e-05 2024-09-11T08:23:29.772 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:29.772 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:29.772 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:23:29.772 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:29.137253+0000", 2024-09-11T08:23:29.772 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7869999999999996e-06 2024-09-11T08:23:29.772 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:29.773 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:29.773 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:23:29.773 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:29.137259+0000", 2024-09-11T08:23:29.773 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.981e-06 2024-09-11T08:23:29.773 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:29.773 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:29.773 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:23:29.773 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:29.140992+0000", 2024-09-11T08:23:29.773 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0037337099999999999 2024-09-11T08:23:29.773 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:29.773 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:29.773 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:23:29.773 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:29.141014+0000", 2024-09-11T08:23:29.774 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2197000000000001e-05 2024-09-11T08:23:29.774 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:23:29.774 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:23:29.774 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:23:29.774 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:29.774 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:29.774 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.15636.0:128234 10.d e338/308 10:b560b03c:::benchmark_data_smithi150_158067_object128233:head v 338'8073, mlcod=338'8071)", 2024-09-11T08:23:29.774 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:23:29.141531+0000", 2024-09-11T08:23:29.774 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.0050989060000000003, 2024-09-11T08:23:29.774 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0053484839999999997, 2024-09-11T08:23:29.774 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:23:29.774 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-11T08:23:29.774 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:23:29.774 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:29.775 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:23:29.775 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:29.141531+0000", 2024-09-11T08:23:29.775 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:23:29.775 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:29.775 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:29.775 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:23:29.775 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:29.141531+0000", 2024-09-11T08:23:29.775 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:23:29.775 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:29.775 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:29.775 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:23:29.775 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:29.141530+0000", 2024-09-11T08:23:29.775 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:23:29.775 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:29.776 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:29.776 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:23:29.776 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:29.141554+0000", 2024-09-11T08:23:29.776 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4043999999999999e-05 2024-09-11T08:23:29.776 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:29.776 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:29.776 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:23:29.776 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:29.141565+0000", 2024-09-11T08:23:29.776 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0522e-05 2024-09-11T08:23:29.776 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:29.776 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:29.776 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:23:29.777 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:29.141571+0000", 2024-09-11T08:23:29.777 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7220000000000002e-06 2024-09-11T08:23:29.777 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:29.777 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:29.777 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:23:29.777 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:29.141644+0000", 2024-09-11T08:23:29.777 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2193999999999994e-05 2024-09-11T08:23:29.777 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:29.777 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:29.777 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:23:29.777 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:29.141661+0000", 2024-09-11T08:23:29.777 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7291000000000001e-05 2024-09-11T08:23:29.777 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:23:29.778 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:23:29.778 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:23:29.778 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:29.778 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:29.778 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.15636.0:128224 10.e e338/337 10:721cc5e6:::benchmark_data_smithi150_158067_object128223:head v 338'7896, mlcod=338'7893)", 2024-09-11T08:23:29.778 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:23:29.134666+0000", 2024-09-11T08:23:29.778 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.011964115000000001, 2024-09-11T08:23:29.778 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.012274808, 2024-09-11T08:23:29.778 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:23:29.778 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-11T08:23:29.778 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:23:29.778 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:29.778 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:23:29.778 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:29.134666+0000", 2024-09-11T08:23:29.778 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:23:29.779 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:29.779 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:29.779 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:23:29.779 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:29.134666+0000", 2024-09-11T08:23:29.779 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:23:29.779 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:29.779 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:29.779 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:23:29.779 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:29.134665+0000", 2024-09-11T08:23:29.779 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:23:29.779 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:29.779 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:29.779 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:23:29.779 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:29.134857+0000", 2024-09-11T08:23:29.780 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00019214199999999999 2024-09-11T08:23:29.780 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:29.780 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:29.780 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:23:29.780 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:29.134870+0000", 2024-09-11T08:23:29.780 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.313e-05 2024-09-11T08:23:29.780 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:29.780 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:29.780 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:23:29.780 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:29.134876+0000", 2024-09-11T08:23:29.780 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9739999999999999e-06 2024-09-11T08:23:29.780 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:29.780 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:29.780 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:23:29.781 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:29.134952+0000", 2024-09-11T08:23:29.781 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6366999999999996e-05 2024-09-11T08:23:29.781 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:29.781 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:29.781 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:23:29.781 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:29.134973+0000", 2024-09-11T08:23:29.781 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0823e-05 2024-09-11T08:23:29.781 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:23:29.781 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:23:29.781 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:23:29.781 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:23:29.781 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-09-11T08:23:29.781 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 5 2024-09-11T08:23:29.782 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-11T08:23:29.840 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:23:29.877 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:29.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:29.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:23:29.978 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:30.017 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:30.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:30.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:30.039 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 89 16 129186 129170 90.6943 65.8125 0.0131706 0.0110218 2024-09-11T08:23:30.120 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:23:30.156 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:30.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:30.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:23:30.258 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:30.293 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:30.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:30.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:30.376 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:30 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:23:30.376 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:30 smithi150 ceph-mon[29276]: pgmap v729: 148 pgs: 148 active+clean; 7.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-11T08:23:30.376 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:30 smithi150 ceph-mon[29276]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:23:30.377 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:30 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3734552934' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:23:30.377 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:30 smithi150 ceph-mon[29276]: osdmap e339: 8 total, 8 up, 8 in 2024-09-11T08:23:30.377 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:30 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:23:30.377 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:30 smithi150 ceph-mon[39612]: pgmap v729: 148 pgs: 148 active+clean; 7.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-11T08:23:30.377 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:30 smithi150 ceph-mon[39612]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:23:30.377 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:30 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3734552934' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:23:30.377 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:30 smithi150 ceph-mon[39612]: osdmap e339: 8 total, 8 up, 8 in 2024-09-11T08:23:30.395 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:30.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:30.446 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:30.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:30.475 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:30 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:23:30.475 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:30 smithi181 ceph-mon[32701]: pgmap v729: 148 pgs: 148 active+clean; 7.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-11T08:23:30.475 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:30 smithi181 ceph-mon[32701]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:23:30.475 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:30 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3734552934' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:23:30.475 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:30 smithi181 ceph-mon[32701]: osdmap e339: 8 total, 8 up, 8 in 2024-09-11T08:23:30.518 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-11T08:23:30.548 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:23:30.578 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:30.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:30.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:23:30.679 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:23:30.718 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:30.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:30.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:23:30.820 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:30.852 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:30.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:30.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:30.953 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:23:30.957 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:23:30.992 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:30.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:30.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:23:31.039 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Total time run: 90.0094 2024-09-11T08:23:31.039 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Total writes made: 130241 2024-09-11T08:23:31.039 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Write size: 65536 2024-09-11T08:23:31.039 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Object size: 65536 2024-09-11T08:23:31.039 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Bandwidth (MB/sec): 90.4357 2024-09-11T08:23:31.040 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Stddev Bandwidth: 39.3122 2024-09-11T08:23:31.040 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Max bandwidth (MB/sec): 246.875 2024-09-11T08:23:31.040 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Min bandwidth (MB/sec): 13.4375 2024-09-11T08:23:31.040 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Average IOPS: 1446 2024-09-11T08:23:31.040 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Stddev IOPS: 628.995 2024-09-11T08:23:31.040 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Max IOPS: 3950 2024-09-11T08:23:31.040 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Min IOPS: 215 2024-09-11T08:23:31.040 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Average Latency(s): 0.0110537 2024-09-11T08:23:31.040 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Stddev Latency(s): 0.0177877 2024-09-11T08:23:31.040 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Max latency(s): 1.04238 2024-09-11T08:23:31.040 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Min latency(s): 0.00123278 2024-09-11T08:23:31.045 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Cleaning up (deleting benchmark objects) 2024-09-11T08:23:31.073 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:23:30 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5[43199]: 2024-09-11T08:23:30.772+0000 7fbb09fa9640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-11T08:23:31.073 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:23:30 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5[43199]: 2024-09-11T08:23:30.772+0000 7fbb09fa9640 -1 osd.5 339 *** Got signal Terminated *** 2024-09-11T08:23:31.073 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:23:30 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5[43199]: 2024-09-11T08:23:30.772+0000 7fbb09fa9640 -1 osd.5 339 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-11T08:23:31.094 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:23:31.127 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:31.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:31.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:23:31.229 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:31.267 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:31.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:31.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:31.369 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:31.377 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:31 smithi150 ceph-mon[29276]: 3.1f scrub starts 2024-09-11T08:23:31.377 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:31 smithi150 ceph-mon[29276]: 3.1f scrub ok 2024-09-11T08:23:31.377 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:31 smithi150 ceph-mon[29276]: osd.5 marked itself down and dead 2024-09-11T08:23:31.378 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:31 smithi150 ceph-mon[39612]: 3.1f scrub starts 2024-09-11T08:23:31.378 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:31 smithi150 ceph-mon[39612]: 3.1f scrub ok 2024-09-11T08:23:31.378 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:31 smithi150 ceph-mon[39612]: osd.5 marked itself down and dead 2024-09-11T08:23:31.395 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:31 smithi181 ceph-mon[32701]: 3.1f scrub starts 2024-09-11T08:23:31.395 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:31 smithi181 ceph-mon[32701]: 3.1f scrub ok 2024-09-11T08:23:31.395 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:31 smithi181 ceph-mon[32701]: osd.5 marked itself down and dead 2024-09-11T08:23:31.423 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:31.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:31.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:31.525 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:23:31.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:31.571 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:31.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:23:31.672 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:23:31.717 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:31.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:31.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:23:31.721 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:23:31.721 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:23:31.745 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:23:31.746 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:23:31.757 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:23:31.758 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:23:31.772 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:23:31.772 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:23:31.788 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:23:31.788 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:23:31.820 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:23:31.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:31.845 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:31.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:23:31.947 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:31.968 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:31.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:31.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:32.070 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:23:32.093 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:32.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:32.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:23:32.170 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:32 smithi181 ceph-mon[32701]: pgmap v731: 148 pgs: 148 active+clean; 7.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-11T08:23:32.170 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:32 smithi181 ceph-mon[32701]: 3.9 scrub starts 2024-09-11T08:23:32.170 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:32 smithi181 ceph-mon[32701]: 3.9 scrub ok 2024-09-11T08:23:32.170 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:32 smithi181 ceph-mon[32701]: 4.10 scrub starts 2024-09-11T08:23:32.170 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:32 smithi181 ceph-mon[32701]: 4.10 scrub ok 2024-09-11T08:23:32.170 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:32 smithi181 ceph-mon[32701]: 4.1 scrub starts 2024-09-11T08:23:32.170 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:32 smithi181 ceph-mon[32701]: 4.1 scrub ok 2024-09-11T08:23:32.170 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:32 smithi181 ceph-mon[32701]: 6.e scrub starts 2024-09-11T08:23:32.170 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:32 smithi181 ceph-mon[32701]: 6.e scrub ok 2024-09-11T08:23:32.170 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:32 smithi181 ceph-mon[32701]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-11T08:23:32.170 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:32 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2419868464' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:23:32.171 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:32 smithi181 ceph-mon[32701]: osdmap e340: 8 total, 7 up, 8 in 2024-09-11T08:23:32.194 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:23:32.216 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:32.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:32.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:23:32.318 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:23:32.324 INFO:teuthology.orchestra.run.smithi150.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-11T08:23:32.340 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:23:32.340 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:23:32.340 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:32.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:32.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:23:32.355 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:23:32.355 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:23:32.422 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T08:23:32.442 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:23:32.444 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:32 smithi150 ceph-mon[29276]: pgmap v731: 148 pgs: 148 active+clean; 7.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-11T08:23:32.444 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:32 smithi150 ceph-mon[29276]: 3.9 scrub starts 2024-09-11T08:23:32.444 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:32 smithi150 ceph-mon[29276]: 3.9 scrub ok 2024-09-11T08:23:32.445 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:32 smithi150 ceph-mon[29276]: 4.10 scrub starts 2024-09-11T08:23:32.445 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:32 smithi150 ceph-mon[29276]: 4.10 scrub ok 2024-09-11T08:23:32.445 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:32 smithi150 ceph-mon[29276]: 4.1 scrub starts 2024-09-11T08:23:32.445 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:32 smithi150 ceph-mon[29276]: 4.1 scrub ok 2024-09-11T08:23:32.445 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:32 smithi150 ceph-mon[29276]: 6.e scrub starts 2024-09-11T08:23:32.445 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:32 smithi150 ceph-mon[29276]: 6.e scrub ok 2024-09-11T08:23:32.445 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:32 smithi150 ceph-mon[29276]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-11T08:23:32.445 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:32 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2419868464' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:23:32.445 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:32 smithi150 ceph-mon[29276]: osdmap e340: 8 total, 7 up, 8 in 2024-09-11T08:23:32.445 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:32 smithi150 ceph-mon[39612]: pgmap v731: 148 pgs: 148 active+clean; 7.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-11T08:23:32.445 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:32 smithi150 ceph-mon[39612]: 3.9 scrub starts 2024-09-11T08:23:32.445 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:32 smithi150 ceph-mon[39612]: 3.9 scrub ok 2024-09-11T08:23:32.446 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:32 smithi150 ceph-mon[39612]: 4.10 scrub starts 2024-09-11T08:23:32.446 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:32 smithi150 ceph-mon[39612]: 4.10 scrub ok 2024-09-11T08:23:32.446 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:32 smithi150 ceph-mon[39612]: 4.1 scrub starts 2024-09-11T08:23:32.446 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:32 smithi150 ceph-mon[39612]: 4.1 scrub ok 2024-09-11T08:23:32.446 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:32 smithi150 ceph-mon[39612]: 6.e scrub starts 2024-09-11T08:23:32.446 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:32 smithi150 ceph-mon[39612]: 6.e scrub ok 2024-09-11T08:23:32.446 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:32 smithi150 ceph-mon[39612]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-11T08:23:32.446 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:32 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2419868464' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:23:32.446 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:32 smithi150 ceph-mon[39612]: osdmap e340: 8 total, 7 up, 8 in 2024-09-11T08:23:32.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:32.465 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:32.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:23:32.466 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:23:32 smithi181 podman[129210]: 2024-09-11 08:23:32.346919089 +0000 UTC m=+1.734719075 container died 0464f2432e0652d5a6219f31b52cd155080a39d8fd5c72f4c2c11477a082783c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True) 2024-09-11T08:23:32.569 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:23:32.591 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:32.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:32.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:23:32.670 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:23:32.693 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:32.743 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:32.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:32.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:32.845 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:32.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:32.877 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:32.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:32.978 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:32.993 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-11T08:23:32.993 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:23:33.012 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:33.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:33.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:33.115 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:23:33.148 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:33.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:33.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:23:33.250 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:33.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:33.288 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:33.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:33.389 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:33.426 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:33.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:33.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:33.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:33 smithi150 ceph-mon[29276]: 4.d scrub starts 2024-09-11T08:23:33.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:33 smithi150 ceph-mon[29276]: 4.d scrub ok 2024-09-11T08:23:33.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:33 smithi150 ceph-mon[29276]: 6.b scrub starts 2024-09-11T08:23:33.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:33 smithi150 ceph-mon[29276]: 6.b scrub ok 2024-09-11T08:23:33.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:33 smithi150 ceph-mon[29276]: osdmap e341: 8 total, 7 up, 8 in 2024-09-11T08:23:33.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:33 smithi150 ceph-mon[39612]: 4.d scrub starts 2024-09-11T08:23:33.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:33 smithi150 ceph-mon[39612]: 4.d scrub ok 2024-09-11T08:23:33.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:33 smithi150 ceph-mon[39612]: 6.b scrub starts 2024-09-11T08:23:33.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:33 smithi150 ceph-mon[39612]: 6.b scrub ok 2024-09-11T08:23:33.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:33 smithi150 ceph-mon[39612]: osdmap e341: 8 total, 7 up, 8 in 2024-09-11T08:23:33.529 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:33.535 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-11T08:23:33.535 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-11T08:23:33.564 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:33.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:33.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:33.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:33 smithi181 ceph-mon[32701]: 4.d scrub starts 2024-09-11T08:23:33.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:33 smithi181 ceph-mon[32701]: 4.d scrub ok 2024-09-11T08:23:33.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:33 smithi181 ceph-mon[32701]: 6.b scrub starts 2024-09-11T08:23:33.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:33 smithi181 ceph-mon[32701]: 6.b scrub ok 2024-09-11T08:23:33.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:33 smithi181 ceph-mon[32701]: osdmap e341: 8 total, 7 up, 8 in 2024-09-11T08:23:33.666 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:23:33.707 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-11T08:23:33.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:33.708 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:33.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:23:33.809 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:23:33.851 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:33.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:33.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:23:33.953 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:23:33.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:33.995 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:33.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:23:34.043 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:23:33 smithi181 podman[129210]: 2024-09-11 08:23:33.833063848 +0000 UTC m=+3.220863831 container cleanup 0464f2432e0652d5a6219f31b52cd155080a39d8fd5c72f4c2c11477a082783c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-11T08:23:34.044 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:23:33 smithi181 bash[129210]: ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5 2024-09-11T08:23:34.096 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:23:34.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:34.150 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:34.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:23:34.176 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:23:34.251 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:34.284 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:34.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:34.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:34.320 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:23:34 smithi181 podman[129397]: 2024-09-11 08:23:34.043563365 +0000 UTC m=+1.697051832 container remove 0464f2432e0652d5a6219f31b52cd155080a39d8fd5c72f4c2c11477a082783c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-11T08:23:34.320 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:34 smithi181 ceph-mon[32701]: pgmap v734: 148 pgs: 9 active+undersized, 1 stale+active+clean, 6 active+undersized+degraded, 132 active+clean; 7.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1023 B/s rd, 71 MiB/s wr, 1.23k op/s; 16304/385146 objects degraded (4.233%) 2024-09-11T08:23:34.320 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:34 smithi181 ceph-mon[32701]: 3.8 scrub starts 2024-09-11T08:23:34.321 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:34 smithi181 ceph-mon[32701]: 3.8 scrub ok 2024-09-11T08:23:34.321 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:34 smithi181 ceph-mon[32701]: 5.1f scrub starts 2024-09-11T08:23:34.321 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:34 smithi181 ceph-mon[32701]: 5.1f scrub ok 2024-09-11T08:23:34.321 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:34 smithi181 ceph-mon[32701]: 5.1d scrub starts 2024-09-11T08:23:34.321 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:34 smithi181 ceph-mon[32701]: 5.1d scrub ok 2024-09-11T08:23:34.321 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:34 smithi181 ceph-mon[32701]: Health check failed: Degraded data redundancy: 16304/385146 objects degraded (4.233%), 6 pgs degraded (PG_DEGRADED) 2024-09-11T08:23:34.386 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:23:34.423 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:34.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:34.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:23:34.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:34 smithi150 ceph-mon[29276]: pgmap v734: 148 pgs: 9 active+undersized, 1 stale+active+clean, 6 active+undersized+degraded, 132 active+clean; 7.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1023 B/s rd, 71 MiB/s wr, 1.23k op/s; 16304/385146 objects degraded (4.233%) 2024-09-11T08:23:34.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:34 smithi150 ceph-mon[29276]: 3.8 scrub starts 2024-09-11T08:23:34.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:34 smithi150 ceph-mon[29276]: 3.8 scrub ok 2024-09-11T08:23:34.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:34 smithi150 ceph-mon[29276]: 5.1f scrub starts 2024-09-11T08:23:34.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:34 smithi150 ceph-mon[29276]: 5.1f scrub ok 2024-09-11T08:23:34.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:34 smithi150 ceph-mon[29276]: 5.1d scrub starts 2024-09-11T08:23:34.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:34 smithi150 ceph-mon[29276]: 5.1d scrub ok 2024-09-11T08:23:34.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:34 smithi150 ceph-mon[29276]: Health check failed: Degraded data redundancy: 16304/385146 objects degraded (4.233%), 6 pgs degraded (PG_DEGRADED) 2024-09-11T08:23:34.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:34 smithi150 ceph-mon[39612]: pgmap v734: 148 pgs: 9 active+undersized, 1 stale+active+clean, 6 active+undersized+degraded, 132 active+clean; 7.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1023 B/s rd, 71 MiB/s wr, 1.23k op/s; 16304/385146 objects degraded (4.233%) 2024-09-11T08:23:34.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:34 smithi150 ceph-mon[39612]: 3.8 scrub starts 2024-09-11T08:23:34.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:34 smithi150 ceph-mon[39612]: 3.8 scrub ok 2024-09-11T08:23:34.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:34 smithi150 ceph-mon[39612]: 5.1f scrub starts 2024-09-11T08:23:34.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:34 smithi150 ceph-mon[39612]: 5.1f scrub ok 2024-09-11T08:23:34.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:34 smithi150 ceph-mon[39612]: 5.1d scrub starts 2024-09-11T08:23:34.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:34 smithi150 ceph-mon[39612]: 5.1d scrub ok 2024-09-11T08:23:34.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:34 smithi150 ceph-mon[39612]: Health check failed: Degraded data redundancy: 16304/385146 objects degraded (4.233%), 6 pgs degraded (PG_DEGRADED) 2024-09-11T08:23:34.525 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:23:34.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:34.556 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:34.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:23:34.657 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:34.694 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:34.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:34.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:34.796 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:23:34.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:34.841 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:34.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:23:34.942 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:23:34.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:34.982 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:34.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:23:35.083 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:23:35.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:35.114 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:35.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:23:35.215 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:23:35.243 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:35.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:35.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:23:35.345 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:23:35.383 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:35.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:35.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:23:35.485 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:35.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:35 smithi150 ceph-mon[29276]: 5.1a scrub starts 2024-09-11T08:23:35.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:35 smithi150 ceph-mon[29276]: 5.1a scrub ok 2024-09-11T08:23:35.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:35 smithi150 ceph-mon[29276]: 10.b scrub starts 2024-09-11T08:23:35.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:35 smithi150 ceph-mon[39612]: 5.1a scrub starts 2024-09-11T08:23:35.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:35 smithi150 ceph-mon[39612]: 5.1a scrub ok 2024-09-11T08:23:35.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:35 smithi150 ceph-mon[39612]: 10.b scrub starts 2024-09-11T08:23:35.519 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:35.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:35.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:35.536 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T08:23:35.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:23:35 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:23:35] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:23:35.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:35 smithi181 ceph-mon[32701]: 5.1a scrub starts 2024-09-11T08:23:35.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:35 smithi181 ceph-mon[32701]: 5.1a scrub ok 2024-09-11T08:23:35.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:35 smithi181 ceph-mon[32701]: 10.b scrub starts 2024-09-11T08:23:35.621 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:35.658 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:35.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:35.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:35.760 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:35.797 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:35.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:35.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:35.900 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:35.921 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:35.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:35.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:36.022 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:23:36.058 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:36.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:36.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:23:36.160 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:36.190 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:36.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:36.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:36.292 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:23:36.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:36 smithi181 ceph-mon[32701]: 6.15 scrub starts 2024-09-11T08:23:36.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:36 smithi181 ceph-mon[32701]: 6.15 scrub ok 2024-09-11T08:23:36.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:36 smithi181 ceph-mon[32701]: 3.1 scrub starts 2024-09-11T08:23:36.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:36 smithi181 ceph-mon[32701]: 3.1 scrub ok 2024-09-11T08:23:36.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:36 smithi181 ceph-mon[32701]: pgmap v735: 148 pgs: 34 active+undersized, 19 active+undersized+degraded, 95 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1023 B/s rd, 31 MiB/s wr, 742 op/s; 40341/386871 objects degraded (10.428%); 0 B/s, 0 objects/s recovering 2024-09-11T08:23:36.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:36 smithi181 ceph-mon[32701]: 5.1c scrub starts 2024-09-11T08:23:36.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:36 smithi181 ceph-mon[32701]: 5.1c scrub ok 2024-09-11T08:23:36.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:36 smithi181 ceph-mon[32701]: 10.e scrub starts 2024-09-11T08:23:36.337 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:36.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:36.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:23:36.439 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:36.480 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:36.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:36.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:36.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:36 smithi150 ceph-mon[29276]: 6.15 scrub starts 2024-09-11T08:23:36.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:36 smithi150 ceph-mon[29276]: 6.15 scrub ok 2024-09-11T08:23:36.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:36 smithi150 ceph-mon[29276]: 3.1 scrub starts 2024-09-11T08:23:36.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:36 smithi150 ceph-mon[29276]: 3.1 scrub ok 2024-09-11T08:23:36.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:36 smithi150 ceph-mon[29276]: pgmap v735: 148 pgs: 34 active+undersized, 19 active+undersized+degraded, 95 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1023 B/s rd, 31 MiB/s wr, 742 op/s; 40341/386871 objects degraded (10.428%); 0 B/s, 0 objects/s recovering 2024-09-11T08:23:36.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:36 smithi150 ceph-mon[29276]: 5.1c scrub starts 2024-09-11T08:23:36.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:36 smithi150 ceph-mon[29276]: 5.1c scrub ok 2024-09-11T08:23:36.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:36 smithi150 ceph-mon[29276]: 10.e scrub starts 2024-09-11T08:23:36.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:36 smithi150 ceph-mon[39612]: 6.15 scrub starts 2024-09-11T08:23:36.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:36 smithi150 ceph-mon[39612]: 6.15 scrub ok 2024-09-11T08:23:36.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:36 smithi150 ceph-mon[39612]: 3.1 scrub starts 2024-09-11T08:23:36.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:36 smithi150 ceph-mon[39612]: 3.1 scrub ok 2024-09-11T08:23:36.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:36 smithi150 ceph-mon[39612]: pgmap v735: 148 pgs: 34 active+undersized, 19 active+undersized+degraded, 95 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1023 B/s rd, 31 MiB/s wr, 742 op/s; 40341/386871 objects degraded (10.428%); 0 B/s, 0 objects/s recovering 2024-09-11T08:23:36.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:36 smithi150 ceph-mon[39612]: 5.1c scrub starts 2024-09-11T08:23:36.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:36 smithi150 ceph-mon[39612]: 5.1c scrub ok 2024-09-11T08:23:36.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:36 smithi150 ceph-mon[39612]: 10.e scrub starts 2024-09-11T08:23:36.583 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:23:36.624 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:36.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:36.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:23:36.726 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:23:36.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:36.783 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:36.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:23:36.864 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:23:36.883 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:36.928 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:36.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:36.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:37.030 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:23:37.064 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:37.065 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-11T08:23:37.065 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:23:37.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:37.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:23:37.166 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:23:37.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:37.199 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:37.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:23:37.300 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:23:37.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:37.330 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:37.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:23:37.431 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:23:37.465 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:37.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:37.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:23:37.567 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:37.598 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:37.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:37.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:37.699 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:37.734 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:37.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:37.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:37.793 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-11T08:23:37.836 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:23:37.891 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:37.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:37.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:23:37.993 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:38.015 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:37 smithi181 ceph-mon[32701]: 4.14 scrub starts 2024-09-11T08:23:38.015 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:37 smithi181 ceph-mon[32701]: 4.14 scrub ok 2024-09-11T08:23:38.015 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:37 smithi181 ceph-mon[32701]: 5.14 scrub starts 2024-09-11T08:23:38.015 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:37 smithi181 ceph-mon[32701]: 5.14 scrub ok 2024-09-11T08:23:38.015 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:37 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:23:38.015 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:37 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:23:38.015 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:37 smithi181 ceph-mon[32701]: pgmap v736: 148 pgs: 34 active+undersized, 19 active+undersized+degraded, 95 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 775 B/s rd, 23 MiB/s wr, 562 op/s; 40341/386871 objects degraded (10.428%); 0 B/s, 0 objects/s recovering 2024-09-11T08:23:38.015 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:37 smithi181 ceph-mon[32701]: 4.1b scrub starts 2024-09-11T08:23:38.015 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:37 smithi181 ceph-mon[32701]: 4.1b scrub ok 2024-09-11T08:23:38.019 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:38.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:38.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:38.121 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:38.154 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:38.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:38.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:38.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:37 smithi150 ceph-mon[29276]: 4.14 scrub starts 2024-09-11T08:23:38.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:37 smithi150 ceph-mon[29276]: 4.14 scrub ok 2024-09-11T08:23:38.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:37 smithi150 ceph-mon[29276]: 5.14 scrub starts 2024-09-11T08:23:38.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:37 smithi150 ceph-mon[29276]: 5.14 scrub ok 2024-09-11T08:23:38.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:37 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:23:38.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:37 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:23:38.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:37 smithi150 ceph-mon[29276]: pgmap v736: 148 pgs: 34 active+undersized, 19 active+undersized+degraded, 95 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 775 B/s rd, 23 MiB/s wr, 562 op/s; 40341/386871 objects degraded (10.428%); 0 B/s, 0 objects/s recovering 2024-09-11T08:23:38.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:37 smithi150 ceph-mon[29276]: 4.1b scrub starts 2024-09-11T08:23:38.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:37 smithi150 ceph-mon[29276]: 4.1b scrub ok 2024-09-11T08:23:38.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:37 smithi150 ceph-mon[39612]: 4.14 scrub starts 2024-09-11T08:23:38.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:37 smithi150 ceph-mon[39612]: 4.14 scrub ok 2024-09-11T08:23:38.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:37 smithi150 ceph-mon[39612]: 5.14 scrub starts 2024-09-11T08:23:38.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:37 smithi150 ceph-mon[39612]: 5.14 scrub ok 2024-09-11T08:23:38.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:37 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:23:38.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:37 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:23:38.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:37 smithi150 ceph-mon[39612]: pgmap v736: 148 pgs: 34 active+undersized, 19 active+undersized+degraded, 95 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 775 B/s rd, 23 MiB/s wr, 562 op/s; 40341/386871 objects degraded (10.428%); 0 B/s, 0 objects/s recovering 2024-09-11T08:23:38.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:37 smithi150 ceph-mon[39612]: 4.1b scrub starts 2024-09-11T08:23:38.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:37 smithi150 ceph-mon[39612]: 4.1b scrub ok 2024-09-11T08:23:38.256 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:38.285 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:38.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:38.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:38.323 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:23:38 smithi181 podman[129910]: 2024-09-11 08:23:37.929738542 +0000 UTC m=+0.039230175 image pull 4a918172a49766385959227c53e75e2c302e923539324792cb5bee6cad0b2f2b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40 2024-09-11T08:23:38.323 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:23:38 smithi181 podman[129910]: 2024-09-11 08:23:38.065500439 +0000 UTC m=+0.174992061 container create aeb7cc64a52c4d4aeb856ec004006d66019244d1890db0bde53ff2ba9d5ac591 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240731, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-11T08:23:38.387 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:23:38.431 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:38.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:38.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:23:38.498 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:23:38.533 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:23:38.574 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:38.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:38.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:23:38.634 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:23:38 smithi181 podman[129910]: 2024-09-11 08:23:38.335708246 +0000 UTC m=+0.445199874 container init aeb7cc64a52c4d4aeb856ec004006d66019244d1890db0bde53ff2ba9d5ac591 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5-deactivate, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-11T08:23:38.634 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:23:38 smithi181 podman[129910]: 2024-09-11 08:23:38.342517943 +0000 UTC m=+0.452009568 container start aeb7cc64a52c4d4aeb856ec004006d66019244d1890db0bde53ff2ba9d5ac591 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-11T08:23:38.634 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:23:38 smithi181 podman[129910]: 2024-09-11 08:23:38.443679881 +0000 UTC m=+0.553171507 container attach aeb7cc64a52c4d4aeb856ec004006d66019244d1890db0bde53ff2ba9d5ac591 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, ceph=True, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-09-11T08:23:38.676 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:38.705 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:38.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:38.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:38.806 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:38.840 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:38.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:38.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:38.942 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:38.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:38.981 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:38.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:38.988 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:23:38 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:23:38] "GET /metrics HTTP/1.1" 200 224693 "" "Prometheus/2.43.0" 2024-09-11T08:23:38.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:38 smithi150 ceph-mon[29276]: 1.0 scrub starts 2024-09-11T08:23:38.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:38 smithi150 ceph-mon[29276]: 1.0 scrub ok 2024-09-11T08:23:38.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:38 smithi150 ceph-mon[29276]: 10.b scrub ok 2024-09-11T08:23:38.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:38 smithi150 ceph-mon[29276]: 4.b scrub starts 2024-09-11T08:23:38.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:38 smithi150 ceph-mon[29276]: 4.b scrub ok 2024-09-11T08:23:38.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:38 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:23:38.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:38 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:23:38.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:38 smithi150 ceph-mon[39612]: 1.0 scrub starts 2024-09-11T08:23:38.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:38 smithi150 ceph-mon[39612]: 1.0 scrub ok 2024-09-11T08:23:38.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:38 smithi150 ceph-mon[39612]: 10.b scrub ok 2024-09-11T08:23:38.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:38 smithi150 ceph-mon[39612]: 4.b scrub starts 2024-09-11T08:23:38.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:38 smithi150 ceph-mon[39612]: 4.b scrub ok 2024-09-11T08:23:38.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:38 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:23:38.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:38 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:23:39.074 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:23:38 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:23:39.074 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:23:38 smithi181 conmon[129945]: conmon aeb7cc64a52c4d4aeb85 : Failed to open cgroups file: /sys/fs/cgroup/machine.slice/libpod-aeb7cc64a52c4d4aeb856ec004006d66019244d1890db0bde53ff2ba9d5ac591.scope/container/memory.events 2024-09-11T08:23:39.074 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:23:38 smithi181 podman[129910]: 2024-09-11 08:23:38.799020886 +0000 UTC m=+0.908512512 container died aeb7cc64a52c4d4aeb856ec004006d66019244d1890db0bde53ff2ba9d5ac591 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-09-11T08:23:39.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:38 smithi181 ceph-mon[32701]: 1.0 scrub starts 2024-09-11T08:23:39.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:38 smithi181 ceph-mon[32701]: 1.0 scrub ok 2024-09-11T08:23:39.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:38 smithi181 ceph-mon[32701]: 10.b scrub ok 2024-09-11T08:23:39.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:38 smithi181 ceph-mon[32701]: 4.b scrub starts 2024-09-11T08:23:39.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:38 smithi181 ceph-mon[32701]: 4.b scrub ok 2024-09-11T08:23:39.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:38 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:23:39.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:38 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:23:39.082 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:23:39.110 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:39.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:39.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:23:39.212 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:39.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:39.247 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:39.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:39.348 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:23:39.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:39.393 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:39.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:23:39.494 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:23:39.530 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:39.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:39.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:23:39.632 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:39.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:39.668 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:39.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:39.769 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:23:39.785 INFO:teuthology.orchestra.run.smithi150.stderr:nodeep-scrub is unset 2024-09-11T08:23:39.798 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:39.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:39.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:23:39.900 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:23:39.916 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:39 smithi150 ceph-mon[29276]: 10.e scrub ok 2024-09-11T08:23:39.916 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:39 smithi150 ceph-mon[29276]: 4.5 scrub starts 2024-09-11T08:23:39.916 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:39 smithi150 ceph-mon[29276]: 4.5 scrub ok 2024-09-11T08:23:39.916 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:39 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:23:39.916 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:39 smithi150 ceph-mon[29276]: osdmap e342: 8 total, 7 up, 8 in 2024-09-11T08:23:39.916 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:39 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:23:39.917 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:39 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:23:39.917 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:39 smithi150 ceph-mon[29276]: 3.12 scrub starts 2024-09-11T08:23:39.917 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:39 smithi150 ceph-mon[29276]: 3.12 scrub ok 2024-09-11T08:23:39.917 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:39 smithi150 ceph-mon[29276]: pgmap v738: 148 pgs: 1 active+clean+scrubbing, 34 active+undersized, 19 active+undersized+degraded, 94 active+clean; 7.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.5 KiB/s rd, 25 MiB/s wr, 1.38k op/s; 38532/369699 objects degraded (10.423%); 0 B/s, 0 objects/s recovering 2024-09-11T08:23:39.917 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:39 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3537463712' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:23:39.917 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:39 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:23:39.917 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:39 smithi150 ceph-mon[39612]: 10.e scrub ok 2024-09-11T08:23:39.917 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:39 smithi150 ceph-mon[39612]: 4.5 scrub starts 2024-09-11T08:23:39.917 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:39 smithi150 ceph-mon[39612]: 4.5 scrub ok 2024-09-11T08:23:39.917 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:39 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:23:39.917 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:39 smithi150 ceph-mon[39612]: osdmap e342: 8 total, 7 up, 8 in 2024-09-11T08:23:39.918 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:39 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:23:39.918 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:39 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:23:39.918 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:39 smithi150 ceph-mon[39612]: 3.12 scrub starts 2024-09-11T08:23:39.918 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:39 smithi150 ceph-mon[39612]: 3.12 scrub ok 2024-09-11T08:23:39.918 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:39 smithi150 ceph-mon[39612]: pgmap v738: 148 pgs: 1 active+clean+scrubbing, 34 active+undersized, 19 active+undersized+degraded, 94 active+clean; 7.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.5 KiB/s rd, 25 MiB/s wr, 1.38k op/s; 38532/369699 objects degraded (10.423%); 0 B/s, 0 objects/s recovering 2024-09-11T08:23:39.918 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:39 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3537463712' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:23:39.918 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:39 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:23:39.936 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:39.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:39.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:23:40.037 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:40.039 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:39 smithi181 ceph-mon[32701]: 10.e scrub ok 2024-09-11T08:23:40.039 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:39 smithi181 ceph-mon[32701]: 4.5 scrub starts 2024-09-11T08:23:40.039 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:39 smithi181 ceph-mon[32701]: 4.5 scrub ok 2024-09-11T08:23:40.039 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:39 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:23:40.039 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:39 smithi181 ceph-mon[32701]: osdmap e342: 8 total, 7 up, 8 in 2024-09-11T08:23:40.040 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:39 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:23:40.040 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:39 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:23:40.040 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:39 smithi181 ceph-mon[32701]: 3.12 scrub starts 2024-09-11T08:23:40.040 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:39 smithi181 ceph-mon[32701]: 3.12 scrub ok 2024-09-11T08:23:40.040 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:39 smithi181 ceph-mon[32701]: pgmap v738: 148 pgs: 1 active+clean+scrubbing, 34 active+undersized, 19 active+undersized+degraded, 94 active+clean; 7.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.5 KiB/s rd, 25 MiB/s wr, 1.38k op/s; 38532/369699 objects degraded (10.423%); 0 B/s, 0 objects/s recovering 2024-09-11T08:23:40.040 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:39 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3537463712' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:23:40.040 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:39 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:23:40.073 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:40.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:40.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:40.176 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:23:40.209 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:40.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:40.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:23:40.311 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:23:40.341 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:40.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:40.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:23:40.443 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:23:40.474 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:40.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:40.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:23:40.576 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:23:40.609 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:40.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:40.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:23:40.682 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c1bddff4-7013-11ef-bcea-c7b262605968@osd.5.service' 2024-09-11T08:23:40.712 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:40.736 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-11T08:23:40.736 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-11T08:23:40.736 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.5 2024-09-11T08:23:40.737 INFO:tasks.thrashosds.thrasher:remote for osd 5 is ubuntu@smithi181.front.sepia.ceph.com 2024-09-11T08:23:40.737 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -n osd.5 --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --op list-pgs 2024-09-11T08:23:40.743 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:40.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:40.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:40.845 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:40.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:40.883 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:40.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:40.942 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:40 smithi181 ceph-mon[32701]: 2.1 scrub starts 2024-09-11T08:23:40.942 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:40 smithi181 ceph-mon[32701]: 2.1 scrub ok 2024-09-11T08:23:40.942 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:40 smithi181 ceph-mon[32701]: Health check update: Degraded data redundancy: 38532/369699 objects degraded (10.423%), 19 pgs degraded (PG_DEGRADED) 2024-09-11T08:23:40.943 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:40 smithi181 ceph-mon[32701]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:23:40.943 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:40 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:23:40.943 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:40 smithi181 ceph-mon[32701]: osdmap e343: 8 total, 7 up, 8 in 2024-09-11T08:23:40.984 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:23:41.028 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:41.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:41.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:23:41.032 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:40 smithi150 ceph-mon[29276]: 2.1 scrub starts 2024-09-11T08:23:41.033 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:40 smithi150 ceph-mon[29276]: 2.1 scrub ok 2024-09-11T08:23:41.033 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:40 smithi150 ceph-mon[29276]: Health check update: Degraded data redundancy: 38532/369699 objects degraded (10.423%), 19 pgs degraded (PG_DEGRADED) 2024-09-11T08:23:41.033 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:40 smithi150 ceph-mon[29276]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:23:41.033 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:40 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:23:41.033 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:40 smithi150 ceph-mon[29276]: osdmap e343: 8 total, 7 up, 8 in 2024-09-11T08:23:41.033 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:40 smithi150 ceph-mon[39612]: 2.1 scrub starts 2024-09-11T08:23:41.033 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:40 smithi150 ceph-mon[39612]: 2.1 scrub ok 2024-09-11T08:23:41.033 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:40 smithi150 ceph-mon[39612]: Health check update: Degraded data redundancy: 38532/369699 objects degraded (10.423%), 19 pgs degraded (PG_DEGRADED) 2024-09-11T08:23:41.034 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:40 smithi150 ceph-mon[39612]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:23:41.034 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:40 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:23:41.034 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:40 smithi150 ceph-mon[39612]: osdmap e343: 8 total, 7 up, 8 in 2024-09-11T08:23:41.130 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:23:41.173 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:41.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:41.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:23:41.228 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-11T08:23:41.228 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:23:41.233 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/osd.5/config 2024-09-11T08:23:41.277 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:41.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:41.312 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:41.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:41.413 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:41.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:41.456 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:41.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:41.556 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:23:41.593 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:41.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:41.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:23:41.696 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:23:41.723 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:41.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:41.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:23:41.825 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:23:41.852 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:41.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:41.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:23:41.895 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:41 smithi181 ceph-mon[32701]: pgmap v740: 148 pgs: 1 active+clean+scrubbing, 34 active+undersized, 19 active+undersized+degraded, 94 active+clean; 7.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 12 MiB/s wr, 1.03k op/s; 38532/369699 objects degraded (10.423%); 0 B/s, 0 objects/s recovering 2024-09-11T08:23:41.915 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:41 smithi150 ceph-mon[29276]: pgmap v740: 148 pgs: 1 active+clean+scrubbing, 34 active+undersized, 19 active+undersized+degraded, 94 active+clean; 7.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 12 MiB/s wr, 1.03k op/s; 38532/369699 objects degraded (10.423%); 0 B/s, 0 objects/s recovering 2024-09-11T08:23:41.916 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:41 smithi150 ceph-mon[39612]: pgmap v740: 148 pgs: 1 active+clean+scrubbing, 34 active+undersized, 19 active+undersized+degraded, 94 active+clean; 7.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 12 MiB/s wr, 1.03k op/s; 38532/369699 objects degraded (10.423%); 0 B/s, 0 objects/s recovering 2024-09-11T08:23:41.934 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-11T08:23:41.953 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:41.976 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:23:41.977 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:23:41.989 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:41.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:41.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:42.013 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:23:42.013 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:23:42.045 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:23:42.045 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:23:42.077 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:23:42.077 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:23:42.091 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:42.112 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:23:42.112 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:23:42.113 INFO:teuthology.orchestra.run.smithi150.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-11T08:23:42.123 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:42.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:42.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:42.143 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:23:42.143 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:23:42.171 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:23:42.172 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:23:42.226 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:42.258 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:42.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:42.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:42.360 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:23:42.394 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:42.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:42.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:23:42.496 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:42.528 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:42.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:42.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:42.630 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:42.664 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:42.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:42.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:42.766 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:42.801 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:42.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:42.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:42.903 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:42.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:42.939 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:42.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:42.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:42 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/4101111355' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:23:42.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:42 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/4101111355' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:23:43.041 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:23:43.046 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-11T08:23:43.046 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-11T08:23:43.068 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:43.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:43.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:23:43.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:42 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/4101111355' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:23:43.150 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:23:43.171 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:43.205 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:43.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:43.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:43.307 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:23:43.336 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:43.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:43.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:23:43.405 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T08:23:43.439 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:43.503 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:43.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:43.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:43.605 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:43.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:43.649 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:43.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:43.750 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:43.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:43.815 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:43.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:43.857 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:23:43.916 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:23:43.944 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:43.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:43.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:23:44.046 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:44.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:43 smithi181 ceph-mon[32701]: 10.5 scrub starts 2024-09-11T08:23:44.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:43 smithi181 ceph-mon[32701]: 3.13 deep-scrub starts 2024-09-11T08:23:44.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:43 smithi181 ceph-mon[32701]: 3.13 deep-scrub ok 2024-09-11T08:23:44.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:43 smithi181 ceph-mon[32701]: pgmap v741: 148 pgs: 34 active+undersized, 19 active+undersized+degraded, 95 active+clean; 7.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.04k op/s; 37788/361788 objects degraded (10.445%) 2024-09-11T08:23:44.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:43 smithi181 ceph-mon[32701]: 3.1a scrub starts 2024-09-11T08:23:44.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:43 smithi181 ceph-mon[32701]: 3.1a scrub ok 2024-09-11T08:23:44.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:44.075 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:44.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:44.176 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:23:44.204 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:44.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:44.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:23:44.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:43 smithi150 ceph-mon[29276]: 10.5 scrub starts 2024-09-11T08:23:44.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:43 smithi150 ceph-mon[29276]: 3.13 deep-scrub starts 2024-09-11T08:23:44.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:43 smithi150 ceph-mon[29276]: 3.13 deep-scrub ok 2024-09-11T08:23:44.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:43 smithi150 ceph-mon[29276]: pgmap v741: 148 pgs: 34 active+undersized, 19 active+undersized+degraded, 95 active+clean; 7.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.04k op/s; 37788/361788 objects degraded (10.445%) 2024-09-11T08:23:44.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:43 smithi150 ceph-mon[29276]: 3.1a scrub starts 2024-09-11T08:23:44.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:43 smithi150 ceph-mon[29276]: 3.1a scrub ok 2024-09-11T08:23:44.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:43 smithi150 ceph-mon[39612]: 10.5 scrub starts 2024-09-11T08:23:44.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:43 smithi150 ceph-mon[39612]: 3.13 deep-scrub starts 2024-09-11T08:23:44.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:43 smithi150 ceph-mon[39612]: 3.13 deep-scrub ok 2024-09-11T08:23:44.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:43 smithi150 ceph-mon[39612]: pgmap v741: 148 pgs: 34 active+undersized, 19 active+undersized+degraded, 95 active+clean; 7.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.04k op/s; 37788/361788 objects degraded (10.445%) 2024-09-11T08:23:44.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:43 smithi150 ceph-mon[39612]: 3.1a scrub starts 2024-09-11T08:23:44.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:43 smithi150 ceph-mon[39612]: 3.1a scrub ok 2024-09-11T08:23:44.306 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:44.350 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:44.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:44.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:44.452 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:44.493 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:44.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:44.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:44.595 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:44.616 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:44.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:44.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:44.718 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:23:44.745 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:44.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:44.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:23:44.847 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:23:44.886 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:44.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:44.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:23:44.987 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:23:45.021 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:45.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:45.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:23:45.047 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T08:23:45.079 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:44 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:23:45.080 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:44 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:23:45.080 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:44 smithi150 ceph-mon[29276]: 4.12 deep-scrub starts 2024-09-11T08:23:45.080 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:44 smithi150 ceph-mon[29276]: 4.12 deep-scrub ok 2024-09-11T08:23:45.081 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:44 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:23:45.081 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:44 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:23:45.083 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:44 smithi181 ceph-mon[32701]: 4.12 deep-scrub starts 2024-09-11T08:23:45.084 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:44 smithi181 ceph-mon[32701]: 4.12 deep-scrub ok 2024-09-11T08:23:45.084 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:44 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:23:45.084 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:44 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:23:45.084 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:44 smithi150 ceph-mon[39612]: 4.12 deep-scrub starts 2024-09-11T08:23:45.084 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:44 smithi150 ceph-mon[39612]: 4.12 deep-scrub ok 2024-09-11T08:23:45.123 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:45.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:45.171 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:45.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:45.272 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:23:45.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:45.302 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:45.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:23:45.403 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:45.448 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:45.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:45.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:45.504 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:23:45 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:23:45] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:23:45.551 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:23:45.589 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:45.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:45.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:23:45.691 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:45.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:45.734 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:45.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:45.835 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:23:45.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:45.863 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:45.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:23:45.963 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:46.003 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:46.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:46.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:46.084 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-11T08:23:46.084 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-11T08:23:46.084 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-09-11T08:23:46.084 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-09-11T08:23:46.084 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-11T08:23:46.084 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:45 smithi181 ceph-mon[32701]: 4.1d scrub starts 2024-09-11T08:23:46.084 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:45 smithi181 ceph-mon[32701]: 4.1d scrub ok 2024-09-11T08:23:46.084 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:45 smithi181 ceph-mon[32701]: osdmap e344: 8 total, 7 up, 8 in 2024-09-11T08:23:46.084 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:45 smithi181 ceph-mon[32701]: pgmap v743: 148 pgs: 1 active+clean+scrubbing, 34 active+undersized, 19 active+undersized+degraded, 94 active+clean; 7.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 814 B/s rd, 0 B/s wr, 1.15k op/s; 36645/351429 objects degraded (10.427%) 2024-09-11T08:23:46.085 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:45 smithi181 ceph-mon[32701]: Health check update: Degraded data redundancy: 37788/361788 objects degraded (10.445%), 19 pgs degraded (PG_DEGRADED) 2024-09-11T08:23:46.105 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:46.140 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:46.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:46.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:46.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:45 smithi150 ceph-mon[29276]: 4.1d scrub starts 2024-09-11T08:23:46.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:45 smithi150 ceph-mon[29276]: 4.1d scrub ok 2024-09-11T08:23:46.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:45 smithi150 ceph-mon[29276]: osdmap e344: 8 total, 7 up, 8 in 2024-09-11T08:23:46.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:45 smithi150 ceph-mon[29276]: pgmap v743: 148 pgs: 1 active+clean+scrubbing, 34 active+undersized, 19 active+undersized+degraded, 94 active+clean; 7.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 814 B/s rd, 0 B/s wr, 1.15k op/s; 36645/351429 objects degraded (10.427%) 2024-09-11T08:23:46.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:45 smithi150 ceph-mon[29276]: Health check update: Degraded data redundancy: 37788/361788 objects degraded (10.445%), 19 pgs degraded (PG_DEGRADED) 2024-09-11T08:23:46.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:45 smithi150 ceph-mon[39612]: 4.1d scrub starts 2024-09-11T08:23:46.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:45 smithi150 ceph-mon[39612]: 4.1d scrub ok 2024-09-11T08:23:46.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:45 smithi150 ceph-mon[39612]: osdmap e344: 8 total, 7 up, 8 in 2024-09-11T08:23:46.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:45 smithi150 ceph-mon[39612]: pgmap v743: 148 pgs: 1 active+clean+scrubbing, 34 active+undersized, 19 active+undersized+degraded, 94 active+clean; 7.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 814 B/s rd, 0 B/s wr, 1.15k op/s; 36645/351429 objects degraded (10.427%) 2024-09-11T08:23:46.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:45 smithi150 ceph-mon[39612]: Health check update: Degraded data redundancy: 37788/361788 objects degraded (10.445%), 19 pgs degraded (PG_DEGRADED) 2024-09-11T08:23:46.242 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:23:46.275 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:46.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:46.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:23:46.342 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:23:46.377 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:23:46.405 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:46.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:46.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:23:46.507 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:23:46.537 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:46.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:46.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:23:46.638 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:46.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:46.679 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:46.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:46.768 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-11T08:23:46.780 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:23:46.857 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:46.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:46.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:23:46.959 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:46.995 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:46.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:46.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:47.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:46 smithi181 ceph-mon[32701]: 10.5 scrub ok 2024-09-11T08:23:47.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:46 smithi181 ceph-mon[32701]: 6.9 scrub starts 2024-09-11T08:23:47.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:46 smithi181 ceph-mon[32701]: 6.9 scrub ok 2024-09-11T08:23:47.097 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:23:47.135 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:47.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:47.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:23:47.209 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:23:47.237 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:23:47.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:46 smithi150 ceph-mon[29276]: 10.5 scrub ok 2024-09-11T08:23:47.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:46 smithi150 ceph-mon[29276]: 6.9 scrub starts 2024-09-11T08:23:47.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:46 smithi150 ceph-mon[29276]: 6.9 scrub ok 2024-09-11T08:23:47.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:46 smithi150 ceph-mon[39612]: 10.5 scrub ok 2024-09-11T08:23:47.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:46 smithi150 ceph-mon[39612]: 6.9 scrub starts 2024-09-11T08:23:47.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:46 smithi150 ceph-mon[39612]: 6.9 scrub ok 2024-09-11T08:23:47.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:47.284 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:47.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:23:47.385 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:23:47.414 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:47.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:47.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:23:47.516 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:23:47.544 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:47.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:47.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:23:47.646 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:23:47.678 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:47.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:47.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:23:47.780 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:47.827 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:47.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:47.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:47.929 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:47.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:47.965 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:47.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:47.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:47 smithi150 ceph-mon[29276]: 6.14 scrub starts 2024-09-11T08:23:47.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:47 smithi150 ceph-mon[29276]: 6.14 scrub ok 2024-09-11T08:23:47.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:47 smithi150 ceph-mon[29276]: pgmap v744: 148 pgs: 1 active+clean+scrubbing, 34 active+undersized, 19 active+undersized+degraded, 94 active+clean; 7.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 761 op/s; 36645/351429 objects degraded (10.427%) 2024-09-11T08:23:47.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:47 smithi150 ceph-mon[39612]: 6.14 scrub starts 2024-09-11T08:23:47.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:47 smithi150 ceph-mon[39612]: 6.14 scrub ok 2024-09-11T08:23:47.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:47 smithi150 ceph-mon[39612]: pgmap v744: 148 pgs: 1 active+clean+scrubbing, 34 active+undersized, 19 active+undersized+degraded, 94 active+clean; 7.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 761 op/s; 36645/351429 objects degraded (10.427%) 2024-09-11T08:23:48.066 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:48.094 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:47 smithi181 ceph-mon[32701]: 6.14 scrub starts 2024-09-11T08:23:48.094 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:47 smithi181 ceph-mon[32701]: 6.14 scrub ok 2024-09-11T08:23:48.094 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:47 smithi181 ceph-mon[32701]: pgmap v744: 148 pgs: 1 active+clean+scrubbing, 34 active+undersized, 19 active+undersized+degraded, 94 active+clean; 7.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 761 op/s; 36645/351429 objects degraded (10.427%) 2024-09-11T08:23:48.102 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:48.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:48.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:48.204 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:23:48.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:48.240 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:48.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:23:48.342 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:23:48.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:48.385 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:48.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:23:48.485 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:23:48.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:48.539 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:48.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:23:48.641 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:23:48.678 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:48.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:48.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:23:48.729 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:23:48 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:23:48] "GET /metrics HTTP/1.1" 200 224693 "" "Prometheus/2.43.0" 2024-09-11T08:23:48.781 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:23:48.796 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:23:48.797 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:23:48.811 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:48.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:48.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:23:48.823 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:23:48 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:23:48.832 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:23:48.832 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:23:48.862 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:23:48.863 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:23:48.871 INFO:teuthology.orchestra.run.smithi150.stderr:noscrub is set 2024-09-11T08:23:48.894 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:23:48.894 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:23:48.913 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:48.937 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:23:48.938 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:23:48.941 INFO:teuthology.orchestra.run.smithi150.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-11T08:23:48.950 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:48.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:48.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:48.987 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:23:48.987 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:23:48.987 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:48 smithi150 ceph-mon[29276]: 3.15 scrub starts 2024-09-11T08:23:48.987 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:48 smithi150 ceph-mon[29276]: 3.15 scrub ok 2024-09-11T08:23:48.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:48 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2162093932' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:23:48.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:48 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:23:48.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:48 smithi150 ceph-mon[29276]: 10.4 scrub starts 2024-09-11T08:23:48.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:48 smithi150 ceph-mon[29276]: 10.2 scrub starts 2024-09-11T08:23:48.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:48 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2177922392' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:23:48.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:48 smithi150 ceph-mon[39612]: 3.15 scrub starts 2024-09-11T08:23:48.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:48 smithi150 ceph-mon[39612]: 3.15 scrub ok 2024-09-11T08:23:48.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:48 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2162093932' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:23:48.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:48 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:23:48.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:48 smithi150 ceph-mon[39612]: 10.4 scrub starts 2024-09-11T08:23:48.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:48 smithi150 ceph-mon[39612]: 10.2 scrub starts 2024-09-11T08:23:48.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:48 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2177922392' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:23:49.018 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:23:49.018 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:23:49.052 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:49.084 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:49.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:49.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:49.186 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:23:49.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:49.220 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:49.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:23:49.321 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:49.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:48 smithi181 ceph-mon[32701]: 3.15 scrub starts 2024-09-11T08:23:49.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:48 smithi181 ceph-mon[32701]: 3.15 scrub ok 2024-09-11T08:23:49.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:48 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2162093932' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:23:49.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:48 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:23:49.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:48 smithi181 ceph-mon[32701]: 10.4 scrub starts 2024-09-11T08:23:49.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:48 smithi181 ceph-mon[32701]: 10.2 scrub starts 2024-09-11T08:23:49.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:48 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2177922392' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:23:49.347 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:49.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:49.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:49.448 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:23:49.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:49.479 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:49.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:23:49.580 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:23:49.613 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:49.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:49.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:23:49.715 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:49.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:49.750 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:49.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:49.851 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:23:49.889 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:49.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:49.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:23:49.990 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:23:50.026 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:50.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:50.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:23:50.128 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:50.153 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:49 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:23:50.153 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:49 smithi150 ceph-mon[29276]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:23:50.153 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:49 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:23:50.153 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:49 smithi150 ceph-mon[29276]: osdmap e345: 8 total, 7 up, 8 in 2024-09-11T08:23:50.153 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:49 smithi150 ceph-mon[29276]: pgmap v746: 148 pgs: 1 active+clean+scrubbing, 34 active+undersized, 19 active+undersized+degraded, 94 active+clean; 6.8 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.48k op/s; 34832/334299 objects degraded (10.419%) 2024-09-11T08:23:50.153 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:49 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:23:50.153 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:49 smithi150 ceph-mon[39612]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:23:50.153 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:49 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:23:50.153 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:49 smithi150 ceph-mon[39612]: osdmap e345: 8 total, 7 up, 8 in 2024-09-11T08:23:50.154 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:49 smithi150 ceph-mon[39612]: pgmap v746: 148 pgs: 1 active+clean+scrubbing, 34 active+undersized, 19 active+undersized+degraded, 94 active+clean; 6.8 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.48k op/s; 34832/334299 objects degraded (10.419%) 2024-09-11T08:23:50.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:50.164 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:50.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:50.265 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:50.297 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:50.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:50.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:50.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:49 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:23:50.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:49 smithi181 ceph-mon[32701]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:23:50.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:49 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:23:50.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:49 smithi181 ceph-mon[32701]: osdmap e345: 8 total, 7 up, 8 in 2024-09-11T08:23:50.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:49 smithi181 ceph-mon[32701]: pgmap v746: 148 pgs: 1 active+clean+scrubbing, 34 active+undersized, 19 active+undersized+degraded, 94 active+clean; 6.8 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.48k op/s; 34832/334299 objects degraded (10.419%) 2024-09-11T08:23:50.354 INFO:teuthology.orchestra.run.smithi181.stdout:6.7 2024-09-11T08:23:50.354 INFO:teuthology.orchestra.run.smithi181.stdout:6.1f 2024-09-11T08:23:50.354 INFO:teuthology.orchestra.run.smithi181.stdout:6.c 2024-09-11T08:23:50.355 INFO:teuthology.orchestra.run.smithi181.stdout:6.1a 2024-09-11T08:23:50.355 INFO:teuthology.orchestra.run.smithi181.stdout:6.19 2024-09-11T08:23:50.355 INFO:teuthology.orchestra.run.smithi181.stdout:6.11 2024-09-11T08:23:50.355 INFO:teuthology.orchestra.run.smithi181.stdout:6.10 2024-09-11T08:23:50.355 INFO:teuthology.orchestra.run.smithi181.stdout:5.e 2024-09-11T08:23:50.355 INFO:teuthology.orchestra.run.smithi181.stdout:5.d 2024-09-11T08:23:50.355 INFO:teuthology.orchestra.run.smithi181.stdout:5.b 2024-09-11T08:23:50.355 INFO:teuthology.orchestra.run.smithi181.stdout:5.4 2024-09-11T08:23:50.355 INFO:teuthology.orchestra.run.smithi181.stdout:5.2 2024-09-11T08:23:50.355 INFO:teuthology.orchestra.run.smithi181.stdout:5.1b 2024-09-11T08:23:50.355 INFO:teuthology.orchestra.run.smithi181.stdout:3.1d 2024-09-11T08:23:50.355 INFO:teuthology.orchestra.run.smithi181.stdout:5.3 2024-09-11T08:23:50.355 INFO:teuthology.orchestra.run.smithi181.stdout:3.1f 2024-09-11T08:23:50.356 INFO:teuthology.orchestra.run.smithi181.stdout:6.1d 2024-09-11T08:23:50.356 INFO:teuthology.orchestra.run.smithi181.stdout:4.1 2024-09-11T08:23:50.356 INFO:teuthology.orchestra.run.smithi181.stdout:3.17 2024-09-11T08:23:50.356 INFO:teuthology.orchestra.run.smithi181.stdout:3.10 2024-09-11T08:23:50.356 INFO:teuthology.orchestra.run.smithi181.stdout:3.2 2024-09-11T08:23:50.356 INFO:teuthology.orchestra.run.smithi181.stdout:3.c 2024-09-11T08:23:50.356 INFO:teuthology.orchestra.run.smithi181.stdout:6.4 2024-09-11T08:23:50.356 INFO:teuthology.orchestra.run.smithi181.stdout:6.1e 2024-09-11T08:23:50.356 INFO:teuthology.orchestra.run.smithi181.stdout:5.f 2024-09-11T08:23:50.356 INFO:teuthology.orchestra.run.smithi181.stdout:4.8 2024-09-11T08:23:50.356 INFO:teuthology.orchestra.run.smithi181.stdout:10.9 2024-09-11T08:23:50.356 INFO:teuthology.orchestra.run.smithi181.stdout:3.16 2024-09-11T08:23:50.356 INFO:teuthology.orchestra.run.smithi181.stdout:6.d 2024-09-11T08:23:50.357 INFO:teuthology.orchestra.run.smithi181.stdout:10.7 2024-09-11T08:23:50.357 INFO:teuthology.orchestra.run.smithi181.stdout:10.a 2024-09-11T08:23:50.357 INFO:teuthology.orchestra.run.smithi181.stdout:6.17 2024-09-11T08:23:50.357 INFO:teuthology.orchestra.run.smithi181.stdout:10.6 2024-09-11T08:23:50.357 INFO:teuthology.orchestra.run.smithi181.stdout:4.1f 2024-09-11T08:23:50.357 INFO:teuthology.orchestra.run.smithi181.stdout:3.4 2024-09-11T08:23:50.357 INFO:teuthology.orchestra.run.smithi181.stdout:4.2 2024-09-11T08:23:50.357 INFO:teuthology.orchestra.run.smithi181.stdout:6.a 2024-09-11T08:23:50.357 INFO:teuthology.orchestra.run.smithi181.stdout:2.2 2024-09-11T08:23:50.357 INFO:teuthology.orchestra.run.smithi181.stdout:4.e 2024-09-11T08:23:50.357 INFO:teuthology.orchestra.run.smithi181.stdout:5.19 2024-09-11T08:23:50.357 INFO:teuthology.orchestra.run.smithi181.stdout:10.3 2024-09-11T08:23:50.357 INFO:teuthology.orchestra.run.smithi181.stdout:6.1c 2024-09-11T08:23:50.357 INFO:teuthology.orchestra.run.smithi181.stdout:4.7 2024-09-11T08:23:50.358 INFO:teuthology.orchestra.run.smithi181.stdout:5.6 2024-09-11T08:23:50.358 INFO:teuthology.orchestra.run.smithi181.stdout:3.1c 2024-09-11T08:23:50.358 INFO:teuthology.orchestra.run.smithi181.stdout:4.15 2024-09-11T08:23:50.358 INFO:teuthology.orchestra.run.smithi181.stdout:5.16 2024-09-11T08:23:50.358 INFO:teuthology.orchestra.run.smithi181.stdout:3.5 2024-09-11T08:23:50.358 INFO:teuthology.orchestra.run.smithi181.stdout:4.3 2024-09-11T08:23:50.358 INFO:teuthology.orchestra.run.smithi181.stdout:5.12 2024-09-11T08:23:50.358 INFO:teuthology.orchestra.run.smithi181.stdout:5.7 2024-09-11T08:23:50.358 INFO:teuthology.orchestra.run.smithi181.stdout:3.d 2024-09-11T08:23:50.358 INFO:teuthology.orchestra.run.smithi181.stdout:5.15 2024-09-11T08:23:50.399 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:50.428 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:50.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:50.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:50.530 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:50.561 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:50.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:50.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:50.663 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:50.696 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:50.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:50.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:50.798 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:50.833 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:50.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:50.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:50.935 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:23:50.981 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:50.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:50.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:23:51.083 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:23:51.110 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:51.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:51.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:23:51.135 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:50 smithi181 ceph-mon[32701]: Health check update: Degraded data redundancy: 34832/334299 objects degraded (10.419%), 19 pgs degraded (PG_DEGRADED) 2024-09-11T08:23:51.212 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:23:51.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:50 smithi150 ceph-mon[29276]: Health check update: Degraded data redundancy: 34832/334299 objects degraded (10.419%), 19 pgs degraded (PG_DEGRADED) 2024-09-11T08:23:51.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:50 smithi150 ceph-mon[39612]: Health check update: Degraded data redundancy: 34832/334299 objects degraded (10.419%), 19 pgs degraded (PG_DEGRADED) 2024-09-11T08:23:51.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:51.244 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:51.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:23:51.315 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-11T08:23:51.316 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-11T08:23:51.344 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:51.387 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:51.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:51.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:51.489 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:51.522 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:51.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:51.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:51.623 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:51.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:51.659 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:51.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:51.760 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:23:51.782 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-11T08:23:51.782 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:23:51.793 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:51.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:51.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:23:51.896 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:51.927 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:51.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:51.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:51.985 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T08:23:52.029 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:52.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:51 smithi181 ceph-mon[32701]: pgmap v747: 148 pgs: 1 active+clean+scrubbing, 34 active+undersized, 19 active+undersized+degraded, 94 active+clean; 6.8 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 34832/334299 objects degraded (10.419%) 2024-09-11T08:23:52.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:51 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:23:52.076 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:52.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:52.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:52.178 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:52.227 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:52.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:52.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:52.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:51 smithi150 ceph-mon[29276]: pgmap v747: 148 pgs: 1 active+clean+scrubbing, 34 active+undersized, 19 active+undersized+degraded, 94 active+clean; 6.8 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 34832/334299 objects degraded (10.419%) 2024-09-11T08:23:52.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:51 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:23:52.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:51 smithi150 ceph-mon[39612]: pgmap v747: 148 pgs: 1 active+clean+scrubbing, 34 active+undersized, 19 active+undersized+degraded, 94 active+clean; 6.8 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 34832/334299 objects degraded (10.419%) 2024-09-11T08:23:52.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:51 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:23:52.329 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:52.339 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -n osd.5 --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --op export --pgid 6.4 --file /var/log/ceph/exp.6.4.5 2024-09-11T08:23:52.385 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:52.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:52.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:52.424 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:23:52.487 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:23:52.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:52.529 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:52.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:23:52.631 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:52.672 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:52.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:52.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:52.774 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:23:52.801 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:52.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:52.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:23:52.903 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:23:52.956 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:52.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:52.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:23:53.000 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-11T08:23:53.046 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/osd.5/config 2024-09-11T08:23:53.058 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:23:53.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:53.109 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:53.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:23:53.210 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:23:53.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:53.250 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:53.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:23:53.317 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T08:23:53.351 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:23:53.387 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:53.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:53.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:23:53.490 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:53.533 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:53.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:53.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:53.635 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:23:53.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:53.665 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:53.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:23:53.766 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:53.801 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:53.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:53.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:53.902 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:23:53.939 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:53.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:53.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:23:54.041 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:54.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:54.087 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:54.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:54.188 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:23:54.215 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:54.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:54.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:23:54.317 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:54.353 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:54.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:54.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:54.391 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:54 smithi181 ceph-mon[32701]: pgmap v748: 148 pgs: 1 active+clean+scrubbing, 34 active+undersized, 19 active+undersized+degraded, 94 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 621 B/s rd, 0 B/s wr, 1.01k op/s; 34099/326574 objects degraded (10.441%) 2024-09-11T08:23:54.447 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:54 smithi150 ceph-mon[29276]: pgmap v748: 148 pgs: 1 active+clean+scrubbing, 34 active+undersized, 19 active+undersized+degraded, 94 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 621 B/s rd, 0 B/s wr, 1.01k op/s; 34099/326574 objects degraded (10.441%) 2024-09-11T08:23:54.447 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:54 smithi150 ceph-mon[39612]: pgmap v748: 148 pgs: 1 active+clean+scrubbing, 34 active+undersized, 19 active+undersized+degraded, 94 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 621 B/s rd, 0 B/s wr, 1.01k op/s; 34099/326574 objects degraded (10.441%) 2024-09-11T08:23:54.455 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:54.467 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:23:54.488 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:54.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:54.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:54.590 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:23:54.622 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:54.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:54.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:23:54.724 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:23:54.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:54.766 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:54.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:23:54.815 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:23:54.868 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:54.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:54.915 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:54.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:55.016 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:55.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:55.064 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:55.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:55.165 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:23:55.192 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:55.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:55.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:23:55.294 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:55.323 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:23:55 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:23:55] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:23:55.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:55.338 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:55.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:55.417 INFO:teuthology.orchestra.run.smithi181.stderr:Exporting 6.4 info 6.4( empty local-lis/les=270/271 n=0 ec=64/64 lis/c=270/270 les/c/f=271/271/0 sis=270) 2024-09-11T08:23:55.422 INFO:teuthology.orchestra.run.smithi181.stderr:Export successful 2024-09-11T08:23:55.439 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:23:55.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:55.469 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:55.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:23:55.570 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:55.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:55.614 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:55.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:55.715 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:55.752 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:55.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:55.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:55.854 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:23:55.888 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:55.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:55.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:23:55.990 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:23:56.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:56.034 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:56.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:23:56.135 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:56.142 INFO:teuthology.orchestra.run.smithi150.stderr:nodeep-scrub is set 2024-09-11T08:23:56.173 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:56.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:56.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:56.274 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:23:56.303 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:56.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:56.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:23:56.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:56 smithi181 ceph-mon[32701]: pgmap v749: 148 pgs: 34 active+undersized, 19 active+undersized+degraded, 95 active+clean; 6.4 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.16k op/s; 32982/316560 objects degraded (10.419%) 2024-09-11T08:23:56.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:56 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2958101933' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:23:56.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:56 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:23:56.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:56 smithi181 ceph-mon[32701]: Health check update: Degraded data redundancy: 32982/316560 objects degraded (10.419%), 19 pgs degraded (PG_DEGRADED) 2024-09-11T08:23:56.405 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:23:56.436 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:56.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:56.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:23:56.474 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:56 smithi150 ceph-mon[29276]: pgmap v749: 148 pgs: 34 active+undersized, 19 active+undersized+degraded, 95 active+clean; 6.4 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.16k op/s; 32982/316560 objects degraded (10.419%) 2024-09-11T08:23:56.475 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:56 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2958101933' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:23:56.475 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:56 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:23:56.475 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:56 smithi150 ceph-mon[29276]: Health check update: Degraded data redundancy: 32982/316560 objects degraded (10.419%), 19 pgs degraded (PG_DEGRADED) 2024-09-11T08:23:56.475 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:56 smithi150 ceph-mon[39612]: pgmap v749: 148 pgs: 34 active+undersized, 19 active+undersized+degraded, 95 active+clean; 6.4 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.16k op/s; 32982/316560 objects degraded (10.419%) 2024-09-11T08:23:56.475 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:56 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2958101933' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:23:56.475 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:56 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:23:56.475 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:56 smithi150 ceph-mon[39612]: Health check update: Degraded data redundancy: 32982/316560 objects degraded (10.419%), 19 pgs degraded (PG_DEGRADED) 2024-09-11T08:23:56.540 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:23:56.560 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:56.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:56.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:23:56.662 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:56.695 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:56.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:56.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:56.797 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:23:56.828 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:56.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:56.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:23:56.930 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:23:56.953 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:56.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:56.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:23:57.054 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:23:57.082 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:57.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:57.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:23:57.177 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:57 smithi150 ceph-mon[29276]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:23:57.177 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:57 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:23:57.177 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:57 smithi150 ceph-mon[29276]: osdmap e346: 8 total, 7 up, 8 in 2024-09-11T08:23:57.177 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:57 smithi150 ceph-mon[39612]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:23:57.177 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:57 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:23:57.178 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:57 smithi150 ceph-mon[39612]: osdmap e346: 8 total, 7 up, 8 in 2024-09-11T08:23:57.184 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:23:57.211 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:57.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:57.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:23:57.313 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:57.361 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:57.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:57.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:57.463 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:57.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:57.502 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:57.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:57.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:57 smithi181 ceph-mon[32701]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:23:57.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:57 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:23:57.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:57 smithi181 ceph-mon[32701]: osdmap e346: 8 total, 7 up, 8 in 2024-09-11T08:23:57.603 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:23:57.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:57.634 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:57.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:23:57.735 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:57.769 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:57.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:57.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:57.874 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:23:57.907 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:57.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:57.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:23:58.008 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:23:58.034 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:58.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:58.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:23:58.136 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:23:58.136 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:23:58.136 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:23:58.136 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:23:58.136 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:23:58.136 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:23:58.136 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:23:58.137 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:23:58.137 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:23:58.137 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:23:58.137 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:23:58.137 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:23:58.137 INFO:teuthology.orchestra.run.smithi150.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-11T08:23:58.137 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:23:58.137 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:23:58.137 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:23:58.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:58.168 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:58.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:23:58.268 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:58.310 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:58.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:58.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:58.412 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:23:58.445 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:58.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:58.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:23:58.451 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:58 smithi150 ceph-mon[29276]: pgmap v751: 148 pgs: 34 active+undersized, 19 active+undersized+degraded, 95 active+clean; 6.4 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 624 B/s rd, 0 B/s wr, 856 op/s; 32982/316560 objects degraded (10.419%) 2024-09-11T08:23:58.451 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:23:58 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/383143432' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:23:58.451 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:58 smithi150 ceph-mon[39612]: pgmap v751: 148 pgs: 34 active+undersized, 19 active+undersized+degraded, 95 active+clean; 6.4 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 624 B/s rd, 0 B/s wr, 856 op/s; 32982/316560 objects degraded (10.419%) 2024-09-11T08:23:58.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:23:58 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/383143432' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:23:58.546 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:58.568 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:58 smithi181 ceph-mon[32701]: pgmap v751: 148 pgs: 34 active+undersized, 19 active+undersized+degraded, 95 active+clean; 6.4 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 624 B/s rd, 0 B/s wr, 856 op/s; 32982/316560 objects degraded (10.419%) 2024-09-11T08:23:58.568 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:23:58 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/383143432' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:23:58.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:58.589 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:58.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:58.690 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:23:58.726 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:58.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:58.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:23:58.738 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:23:58 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:23:58] "GET /metrics HTTP/1.1" 200 224705 "" "Prometheus/2.43.0" 2024-09-11T08:23:58.823 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:23:58 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:23:58.828 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:23:58.854 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:58.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:58.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:23:58.880 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-11T08:23:58.880 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-11T08:23:58.956 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:23:58.984 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:58.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:58.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:23:59.086 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:23:59.120 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:59.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:59.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:23:59.223 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:23:59.251 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:59.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:59.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:23:59.353 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:23:59.361 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -n osd.5 --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --force --op remove --pgid 6.4 2024-09-11T08:23:59.384 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:23:59.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:59.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:23:59.486 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:59.517 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:59.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:59.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:59.619 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:23:59.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:59.653 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:59.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:23:59.687 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/osd.5/config 2024-09-11T08:23:59.755 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:59.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:59.795 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:59.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:59.896 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:23:59.934 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:23:59.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:23:59.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:23:59.962 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-11T08:23:59.962 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-09-11T08:23:59.962 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:59.962 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.15636.0:163041 10.9 e346/340 10:92c57982:::benchmark_data_smithi150_158067_object32797:head v 346'10261, mlcod=346'10259)", 2024-09-11T08:23:59.963 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:23:59.958124+0000", 2024-09-11T08:23:59.963 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.0030753500000000001, 2024-09-11T08:23:59.963 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0031069280000000001, 2024-09-11T08:23:59.963 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:23:59.963 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-11T08:23:59.963 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:23:59.963 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:59.963 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:23:59.963 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:59.958124+0000", 2024-09-11T08:23:59.963 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:23:59.963 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:59.963 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:59.963 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:23:59.963 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:59.958124+0000", 2024-09-11T08:23:59.964 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:23:59.964 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:59.964 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:59.964 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:23:59.964 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:59.958123+0000", 2024-09-11T08:23:59.964 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:23:59.964 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:59.964 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:59.964 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:23:59.964 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:59.958135+0000", 2024-09-11T08:23:59.964 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2661e-05 2024-09-11T08:23:59.964 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:59.964 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:59.964 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:23:59.965 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:59.958154+0000", 2024-09-11T08:23:59.965 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8464000000000001e-05 2024-09-11T08:23:59.965 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:59.965 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:59.965 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:23:59.965 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:59.958161+0000", 2024-09-11T08:23:59.965 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5089999999999996e-06 2024-09-11T08:23:59.965 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:59.965 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:59.965 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:23:59.965 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:59.958204+0000", 2024-09-11T08:23:59.965 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2632000000000001e-05 2024-09-11T08:23:59.965 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:59.966 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:59.966 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:23:59.966 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:59.958235+0000", 2024-09-11T08:23:59.966 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1427999999999998e-05 2024-09-11T08:23:59.966 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:23:59.966 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:23:59.966 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:23:59.966 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:59.966 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:59.966 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15636.0:163043 10.1 10:8e7730e9:::benchmark_data_smithi150_158067_object32799:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-11T08:23:59.966 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:23:59.959360+0000", 2024-09-11T08:23:59.966 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.001839292, 2024-09-11T08:23:59.966 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001936702, 2024-09-11T08:23:59.967 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:23:59.967 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "waiting for sub ops", 2024-09-11T08:23:59.967 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:23:59.967 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15636", 2024-09-11T08:23:59.967 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:23:59.967 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 163043 2024-09-11T08:23:59.967 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:59.967 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:23:59.967 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:59.967 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:23:59.967 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:59.959360+0000", 2024-09-11T08:23:59.967 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:23:59.967 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:59.967 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:59.968 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:23:59.968 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:59.959360+0000", 2024-09-11T08:23:59.968 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:23:59.968 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:59.968 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:59.968 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:23:59.968 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:59.959358+0000", 2024-09-11T08:23:59.968 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:23:59.968 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:59.968 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:59.968 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:23:59.968 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:59.959365+0000", 2024-09-11T08:23:59.968 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3199999999999996e-06 2024-09-11T08:23:59.969 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:59.969 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:59.969 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:23:59.969 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:59.959380+0000", 2024-09-11T08:23:59.969 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5634e-05 2024-09-11T08:23:59.969 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:59.969 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:59.969 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:23:59.969 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:59.959387+0000", 2024-09-11T08:23:59.969 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3990000000000001e-06 2024-09-11T08:23:59.969 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:59.969 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:59.969 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:23:59.969 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:59.959434+0000", 2024-09-11T08:23:59.970 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7088e-05 2024-09-11T08:23:59.970 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:59.970 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:59.970 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:23:59.970 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:59.959570+0000", 2024-09-11T08:23:59.970 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013657799999999999 2024-09-11T08:23:59.970 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:23:59.970 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:23:59.970 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,4", 2024-09-11T08:23:59.970 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:59.959683+0000", 2024-09-11T08:23:59.971 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011302399999999999 2024-09-11T08:23:59.971 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:23:59.971 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:23:59.971 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:23:59.971 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:23:59.971 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-09-11T08:23:59.971 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 2 2024-09-11T08:23:59.971 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-11T08:24:00.036 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:00.060 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T08:24:00.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:00.078 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:00.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:00.179 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:24:00.190 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:00 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:24:00.190 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:00 smithi181 ceph-mon[32701]: pgmap v752: 148 pgs: 34 active+undersized, 19 active+undersized+degraded, 95 active+clean; 6.1 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.16k op/s; 31121/299394 objects degraded (10.395%) 2024-09-11T08:24:00.219 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:00.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:00.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:24:00.321 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:24:00.347 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:00.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:00.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:24:00.450 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:00.475 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:00.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:00.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:00.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:00 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:24:00.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:00 smithi150 ceph-mon[29276]: pgmap v752: 148 pgs: 34 active+undersized, 19 active+undersized+degraded, 95 active+clean; 6.1 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.16k op/s; 31121/299394 objects degraded (10.395%) 2024-09-11T08:24:00.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:00 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:24:00.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:00 smithi150 ceph-mon[39612]: pgmap v752: 148 pgs: 34 active+undersized, 19 active+undersized+degraded, 95 active+clean; 6.1 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.16k op/s; 31121/299394 objects degraded (10.395%) 2024-09-11T08:24:00.495 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:24:00.577 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:24:00.620 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:00.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:00.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:24:00.722 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:00.755 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:00.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:00.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:00.849 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-11T08:24:00.857 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:00.882 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T08:24:00.895 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:00.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:00.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:00.996 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:24:01.034 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:01.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:01.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:24:01.136 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:24:01.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:01.165 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:01.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:24:01.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:01 smithi150 ceph-mon[29276]: Health check update: Degraded data redundancy: 31121/299394 objects degraded (10.395%), 19 pgs degraded (PG_DEGRADED) 2024-09-11T08:24:01.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:01 smithi150 ceph-mon[39612]: Health check update: Degraded data redundancy: 31121/299394 objects degraded (10.395%), 19 pgs degraded (PG_DEGRADED) 2024-09-11T08:24:01.266 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:01.302 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:01.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:01.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:01.404 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:01.426 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:01 smithi181 ceph-mon[32701]: Health check update: Degraded data redundancy: 31121/299394 objects degraded (10.395%), 19 pgs degraded (PG_DEGRADED) 2024-09-11T08:24:01.430 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:01.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:01.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:01.531 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:01.563 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:01.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:01.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:01.665 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:01.688 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:01.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:01.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:01.789 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:01.810 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:01.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:01.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:01.912 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:24:01.948 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:01.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:01.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:24:02.050 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:24:02.088 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:02.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:02.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:24:02.190 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:24:02.240 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:02.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:02.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:24:02.274 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:02 smithi150 ceph-mon[29276]: pgmap v753: 148 pgs: 34 active+undersized, 19 active+undersized+degraded, 95 active+clean; 6.1 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.16k op/s; 31121/299394 objects degraded (10.395%) 2024-09-11T08:24:02.274 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:02 smithi150 ceph-mon[39612]: pgmap v753: 148 pgs: 34 active+undersized, 19 active+undersized+degraded, 95 active+clean; 6.1 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.16k op/s; 31121/299394 objects degraded (10.395%) 2024-09-11T08:24:02.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:02 smithi181 ceph-mon[32701]: pgmap v753: 148 pgs: 34 active+undersized, 19 active+undersized+degraded, 95 active+clean; 6.1 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.16k op/s; 31121/299394 objects degraded (10.395%) 2024-09-11T08:24:02.342 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:02.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:02.374 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:02.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:02.475 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:02.519 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:02.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:02.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:02.562 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:24:02.579 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:24:02.622 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:02.665 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:02.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:02.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:02.767 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:02.801 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:02.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:02.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:02.903 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:02.936 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:02.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:02.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:03.038 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:03.071 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:03.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:03.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:03.173 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:24:03.197 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:03.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:03.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:24:03.299 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:03.335 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:03.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:03.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:03.437 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:24:03.463 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:03.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:03.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:24:03.551 INFO:teuthology.orchestra.run.smithi181.stdout: marking collection for removal 2024-09-11T08:24:03.551 INFO:teuthology.orchestra.run.smithi181.stdout:setting '_remove' omap key 2024-09-11T08:24:03.551 INFO:teuthology.orchestra.run.smithi181.stdout:finish_remove_pgs 6.4_head removing 6.4 2024-09-11T08:24:03.551 INFO:teuthology.orchestra.run.smithi181.stdout:Remove successful 2024-09-11T08:24:03.565 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:03.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:03.609 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:03.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:03.709 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:24:03.748 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:03.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:03.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:24:03.849 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:03.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:03.882 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:03.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:03.982 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:04.015 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:04.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:04.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:04.053 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:24:03 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=sqlstore.transactions t=2024-09-11T08:24:03.7336058Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-11T08:24:04.117 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:24:04.169 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:04.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:04.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:24:04.271 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:04.306 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:04.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:04.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:04.313 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:04 smithi181 ceph-mon[32701]: pgmap v754: 148 pgs: 34 active+undersized, 19 active+undersized+degraded, 95 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s; 30344/291381 objects degraded (10.414%) 2024-09-11T08:24:04.408 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:04.438 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:04.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:04.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:04.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:04 smithi150 ceph-mon[29276]: pgmap v754: 148 pgs: 34 active+undersized, 19 active+undersized+degraded, 95 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s; 30344/291381 objects degraded (10.414%) 2024-09-11T08:24:04.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:04 smithi150 ceph-mon[39612]: pgmap v754: 148 pgs: 34 active+undersized, 19 active+undersized+degraded, 95 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s; 30344/291381 objects degraded (10.414%) 2024-09-11T08:24:04.540 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:04.577 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:04.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:04.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:04.678 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:04.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:04.717 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:04.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:04.818 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:04.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:04.868 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:04.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:04.969 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:24:05.007 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:05.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:05.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:24:05.109 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:05.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:05.142 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:05.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:05.180 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -n osd.5 --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.6.4.5 2024-09-11T08:24:05.243 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:05.275 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:05.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:05.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:05.376 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:05.420 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:05.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:05.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:05.437 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/osd.5/config 2024-09-11T08:24:05.457 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:24:05 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:24:05] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:24:05.522 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:24:05.563 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:05.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:05.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:24:05.666 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:05.705 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:05.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:05.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:05.807 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:05.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:05.843 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:05.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:05.888 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:24:05.889 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:24:05.920 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:24:05.920 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:24:05.944 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:05.953 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:24:05.953 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:24:05.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:05.986 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:05.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:05.989 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:24:05.989 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:24:06.020 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:24:06.020 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:24:06.021 INFO:teuthology.orchestra.run.smithi150.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-11T08:24:06.050 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:24:06.050 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:24:06.082 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:24:06.082 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:24:06.087 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:06.125 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:06.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:06.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:06.181 INFO:teuthology.orchestra.run.smithi150.stderr:noscrub is unset 2024-09-11T08:24:06.227 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:24:06.261 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:06.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:06.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:24:06.363 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:06.396 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:06.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:06.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:06.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:06 smithi150 ceph-mon[29276]: pgmap v755: 148 pgs: 34 active+undersized, 19 active+undersized+degraded, 95 active+clean; 5.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s; 29265/281505 objects degraded (10.396%) 2024-09-11T08:24:06.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:06 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2671259593' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:24:06.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:06 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:24:06.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:06 smithi150 ceph-mon[29276]: Health check update: Degraded data redundancy: 29265/281505 objects degraded (10.396%), 19 pgs degraded (PG_DEGRADED) 2024-09-11T08:24:06.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:06 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1558114537' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:24:06.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:06 smithi150 ceph-mon[39612]: pgmap v755: 148 pgs: 34 active+undersized, 19 active+undersized+degraded, 95 active+clean; 5.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s; 29265/281505 objects degraded (10.396%) 2024-09-11T08:24:06.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:06 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2671259593' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:24:06.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:06 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:24:06.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:06 smithi150 ceph-mon[39612]: Health check update: Degraded data redundancy: 29265/281505 objects degraded (10.396%), 19 pgs degraded (PG_DEGRADED) 2024-09-11T08:24:06.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:06 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1558114537' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:24:06.498 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:06.532 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:06.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:06.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:06.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:06 smithi181 ceph-mon[32701]: pgmap v755: 148 pgs: 34 active+undersized, 19 active+undersized+degraded, 95 active+clean; 5.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s; 29265/281505 objects degraded (10.396%) 2024-09-11T08:24:06.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:06 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2671259593' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:24:06.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:06 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:24:06.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:06 smithi181 ceph-mon[32701]: Health check update: Degraded data redundancy: 29265/281505 objects degraded (10.396%), 19 pgs degraded (PG_DEGRADED) 2024-09-11T08:24:06.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:06 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1558114537' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:24:06.634 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:06.670 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-11T08:24:06.670 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-11T08:24:06.670 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-09-11T08:24:06.670 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-09-11T08:24:06.670 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-11T08:24:06.673 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:06.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:06.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:06.775 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:24:06.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:06.801 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:06.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:24:06.902 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:06.940 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:06.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:06.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:07.042 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:07.082 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:07.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:07.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:07.185 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:07.217 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:07.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:07.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:07.318 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:07.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:07 smithi181 ceph-mon[32701]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:24:07.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:07 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:24:07.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:07 smithi181 ceph-mon[32701]: osdmap e347: 8 total, 7 up, 8 in 2024-09-11T08:24:07.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:07 smithi181 ceph-mon[32701]: 10.4 scrub starts 2024-09-11T08:24:07.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:07 smithi181 ceph-mon[32701]: 10.2 scrub starts 2024-09-11T08:24:07.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:07 smithi181 ceph-mon[32701]: 3.a scrub starts 2024-09-11T08:24:07.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:07 smithi181 ceph-mon[32701]: 3.a scrub ok 2024-09-11T08:24:07.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:07 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:24:07.347 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:07.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:07.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:07.410 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-11T08:24:07.449 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:24:07.482 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:07.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:07.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:24:07.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:07 smithi150 ceph-mon[29276]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:24:07.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:07 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:24:07.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:07 smithi150 ceph-mon[29276]: osdmap e347: 8 total, 7 up, 8 in 2024-09-11T08:24:07.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:07 smithi150 ceph-mon[29276]: 10.4 scrub starts 2024-09-11T08:24:07.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:07 smithi150 ceph-mon[29276]: 10.2 scrub starts 2024-09-11T08:24:07.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:07 smithi150 ceph-mon[29276]: 3.a scrub starts 2024-09-11T08:24:07.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:07 smithi150 ceph-mon[29276]: 3.a scrub ok 2024-09-11T08:24:07.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:07 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:24:07.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:07 smithi150 ceph-mon[39612]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:24:07.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:07 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:24:07.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:07 smithi150 ceph-mon[39612]: osdmap e347: 8 total, 7 up, 8 in 2024-09-11T08:24:07.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:07 smithi150 ceph-mon[39612]: 10.4 scrub starts 2024-09-11T08:24:07.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:07 smithi150 ceph-mon[39612]: 10.2 scrub starts 2024-09-11T08:24:07.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:07 smithi150 ceph-mon[39612]: 3.a scrub starts 2024-09-11T08:24:07.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:07 smithi150 ceph-mon[39612]: 3.a scrub ok 2024-09-11T08:24:07.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:07 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:24:07.583 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:07.611 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:07.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:07.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:07.713 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:07.754 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:07.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:07.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:07.855 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:07.894 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:07.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:07.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:07.902 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:24:07.996 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:24:08.040 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:08.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:08.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:24:08.142 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:08.187 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:08.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:08.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:08.289 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:08.329 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:08.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:08.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:08.429 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:08 smithi181 ceph-mon[32701]: pgmap v757: 148 pgs: 34 active+undersized, 19 active+undersized+degraded, 95 active+clean; 5.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s; 29265/281505 objects degraded (10.396%) 2024-09-11T08:24:08.430 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:08 smithi181 ceph-mon[32701]: 6.5 scrub starts 2024-09-11T08:24:08.430 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:08 smithi181 ceph-mon[32701]: 6.5 scrub ok 2024-09-11T08:24:08.430 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:08 smithi181 ceph-mon[32701]: 4.9 scrub starts 2024-09-11T08:24:08.430 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:08 smithi181 ceph-mon[32701]: 4.9 scrub ok 2024-09-11T08:24:08.430 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:08 smithi181 ceph-mon[32701]: 5.1e scrub starts 2024-09-11T08:24:08.430 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:08 smithi181 ceph-mon[32701]: 5.1e scrub ok 2024-09-11T08:24:08.432 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:24:08.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:08.465 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:08.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:24:08.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:08 smithi150 ceph-mon[29276]: pgmap v757: 148 pgs: 34 active+undersized, 19 active+undersized+degraded, 95 active+clean; 5.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s; 29265/281505 objects degraded (10.396%) 2024-09-11T08:24:08.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:08 smithi150 ceph-mon[29276]: 6.5 scrub starts 2024-09-11T08:24:08.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:08 smithi150 ceph-mon[29276]: 6.5 scrub ok 2024-09-11T08:24:08.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:08 smithi150 ceph-mon[29276]: 4.9 scrub starts 2024-09-11T08:24:08.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:08 smithi150 ceph-mon[29276]: 4.9 scrub ok 2024-09-11T08:24:08.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:08 smithi150 ceph-mon[29276]: 5.1e scrub starts 2024-09-11T08:24:08.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:08 smithi150 ceph-mon[29276]: 5.1e scrub ok 2024-09-11T08:24:08.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:08 smithi150 ceph-mon[39612]: pgmap v757: 148 pgs: 34 active+undersized, 19 active+undersized+degraded, 95 active+clean; 5.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s; 29265/281505 objects degraded (10.396%) 2024-09-11T08:24:08.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:08 smithi150 ceph-mon[39612]: 6.5 scrub starts 2024-09-11T08:24:08.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:08 smithi150 ceph-mon[39612]: 6.5 scrub ok 2024-09-11T08:24:08.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:08 smithi150 ceph-mon[39612]: 4.9 scrub starts 2024-09-11T08:24:08.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:08 smithi150 ceph-mon[39612]: 4.9 scrub ok 2024-09-11T08:24:08.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:08 smithi150 ceph-mon[39612]: 5.1e scrub starts 2024-09-11T08:24:08.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:08 smithi150 ceph-mon[39612]: 5.1e scrub ok 2024-09-11T08:24:08.554 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-11T08:24:08.554 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-11T08:24:08.565 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:24:08.603 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:08.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:08.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:24:08.706 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:08.736 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:08.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:08.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:08.838 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:24:08.877 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:08.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:08.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:24:08.979 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:08.989 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:24:08 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:24:08] "GET /metrics HTTP/1.1" 200 224676 "" "Prometheus/2.43.0" 2024-09-11T08:24:09.021 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:09.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:09.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:09.073 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:24:08 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:24:09.125 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:09.169 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:09.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:09.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:09.270 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:09.299 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:09.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:09.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:09.401 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:24:09.429 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:09.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:09.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:24:09.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:09 smithi150 ceph-mon[29276]: 5.5 scrub starts 2024-09-11T08:24:09.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:09 smithi150 ceph-mon[29276]: 5.5 scrub ok 2024-09-11T08:24:09.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:09 smithi150 ceph-mon[29276]: 5.c scrub starts 2024-09-11T08:24:09.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:09 smithi150 ceph-mon[29276]: 5.c scrub ok 2024-09-11T08:24:09.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:09 smithi150 ceph-mon[29276]: 5.18 scrub starts 2024-09-11T08:24:09.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:09 smithi150 ceph-mon[29276]: 5.18 scrub ok 2024-09-11T08:24:09.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:09 smithi150 ceph-mon[39612]: 5.5 scrub starts 2024-09-11T08:24:09.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:09 smithi150 ceph-mon[39612]: 5.5 scrub ok 2024-09-11T08:24:09.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:09 smithi150 ceph-mon[39612]: 5.c scrub starts 2024-09-11T08:24:09.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:09 smithi150 ceph-mon[39612]: 5.c scrub ok 2024-09-11T08:24:09.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:09 smithi150 ceph-mon[39612]: 5.18 scrub starts 2024-09-11T08:24:09.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:09 smithi150 ceph-mon[39612]: 5.18 scrub ok 2024-09-11T08:24:09.533 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:09.568 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:09.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:09.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:09.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:09 smithi181 ceph-mon[32701]: 5.5 scrub starts 2024-09-11T08:24:09.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:09 smithi181 ceph-mon[32701]: 5.5 scrub ok 2024-09-11T08:24:09.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:09 smithi181 ceph-mon[32701]: 5.c scrub starts 2024-09-11T08:24:09.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:09 smithi181 ceph-mon[32701]: 5.c scrub ok 2024-09-11T08:24:09.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:09 smithi181 ceph-mon[32701]: 5.18 scrub starts 2024-09-11T08:24:09.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:09 smithi181 ceph-mon[32701]: 5.18 scrub ok 2024-09-11T08:24:09.670 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:09.707 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:09.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:09.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:09.809 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:09.830 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:09.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:09.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:09.931 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:09.966 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:09.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:09.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:10.068 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:10.095 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:10.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:10.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:10.197 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:10.238 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:10.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:10.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:10.340 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:10.373 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:10.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:10.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:10.475 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:10.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:10 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:24:10.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:10 smithi150 ceph-mon[29276]: 5.1 scrub starts 2024-09-11T08:24:10.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:10 smithi150 ceph-mon[29276]: 5.1 scrub ok 2024-09-11T08:24:10.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:10 smithi150 ceph-mon[29276]: 10.2 scrub ok 2024-09-11T08:24:10.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:10 smithi150 ceph-mon[29276]: 10.4 scrub ok 2024-09-11T08:24:10.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:10 smithi150 ceph-mon[29276]: pgmap v758: 148 pgs: 2 active+clean+scrubbing, 34 active+undersized, 19 active+undersized+degraded, 93 active+clean; 5.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.18k op/s; 27465/264144 objects degraded (10.398%) 2024-09-11T08:24:10.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:10 smithi150 ceph-mon[29276]: 5.11 scrub starts 2024-09-11T08:24:10.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:10 smithi150 ceph-mon[29276]: 5.11 scrub ok 2024-09-11T08:24:10.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:10 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:24:10.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:10 smithi150 ceph-mon[39612]: 5.1 scrub starts 2024-09-11T08:24:10.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:10 smithi150 ceph-mon[39612]: 5.1 scrub ok 2024-09-11T08:24:10.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:10 smithi150 ceph-mon[39612]: 10.2 scrub ok 2024-09-11T08:24:10.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:10 smithi150 ceph-mon[39612]: 10.4 scrub ok 2024-09-11T08:24:10.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:10 smithi150 ceph-mon[39612]: pgmap v758: 148 pgs: 2 active+clean+scrubbing, 34 active+undersized, 19 active+undersized+degraded, 93 active+clean; 5.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.18k op/s; 27465/264144 objects degraded (10.398%) 2024-09-11T08:24:10.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:10 smithi150 ceph-mon[39612]: 5.11 scrub starts 2024-09-11T08:24:10.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:10 smithi150 ceph-mon[39612]: 5.11 scrub ok 2024-09-11T08:24:10.510 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:10.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:10 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:24:10.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:10 smithi181 ceph-mon[32701]: 5.1 scrub starts 2024-09-11T08:24:10.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:10 smithi181 ceph-mon[32701]: 5.1 scrub ok 2024-09-11T08:24:10.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:10 smithi181 ceph-mon[32701]: 10.2 scrub ok 2024-09-11T08:24:10.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:10 smithi181 ceph-mon[32701]: 10.4 scrub ok 2024-09-11T08:24:10.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:10 smithi181 ceph-mon[32701]: pgmap v758: 148 pgs: 2 active+clean+scrubbing, 34 active+undersized, 19 active+undersized+degraded, 93 active+clean; 5.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.18k op/s; 27465/264144 objects degraded (10.398%) 2024-09-11T08:24:10.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:10 smithi181 ceph-mon[32701]: 5.11 scrub starts 2024-09-11T08:24:10.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:10 smithi181 ceph-mon[32701]: 5.11 scrub ok 2024-09-11T08:24:10.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:10.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:10.555 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T08:24:10.613 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:24:10.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:10.657 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:10.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:24:10.758 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:10.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:10.803 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:10.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:10.904 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:10.953 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:10.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:10.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:10.999 INFO:teuthology.orchestra.run.smithi181.stdout:get_pg_num_history pg_num_history pg_num_history(e339 pg_nums {1={19=1},2={58=3},3={58=32},4={60=32},5={62=32},6={64=32},9={246=16},10={308=16}} deleted_pools 305,9) 2024-09-11T08:24:10.999 INFO:teuthology.orchestra.run.smithi181.stdout:Importing pgid 6.4 2024-09-11T08:24:10.999 INFO:teuthology.orchestra.run.smithi181.stdout:write_pg epoch 331 info 6.4( empty local-lis/les=270/271 n=0 ec=64/64 lis/c=270/270 les/c/f=271/271/0 sis=270) 2024-09-11T08:24:10.999 INFO:teuthology.orchestra.run.smithi181.stdout:Import successful 2024-09-11T08:24:11.003 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:24:11.054 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:11.097 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:11.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:11.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:11.110 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T08:24:11.199 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:11.232 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:11.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:11.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:11.333 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:11 smithi181 ceph-mon[32701]: 4.1c scrub starts 2024-09-11T08:24:11.333 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:11 smithi181 ceph-mon[32701]: 4.1c scrub ok 2024-09-11T08:24:11.333 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:11 smithi181 ceph-mon[32701]: 10.0 scrub starts 2024-09-11T08:24:11.333 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:11 smithi181 ceph-mon[32701]: Health check update: Degraded data redundancy: 27465/264144 objects degraded (10.398%), 19 pgs degraded (PG_DEGRADED) 2024-09-11T08:24:11.335 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:11.367 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-11T08:24:11.368 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:24:11.378 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:11.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:11.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:11.427 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:11 smithi150 ceph-mon[29276]: 4.1c scrub starts 2024-09-11T08:24:11.427 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:11 smithi150 ceph-mon[29276]: 4.1c scrub ok 2024-09-11T08:24:11.427 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:11 smithi150 ceph-mon[29276]: 10.0 scrub starts 2024-09-11T08:24:11.427 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:11 smithi150 ceph-mon[29276]: Health check update: Degraded data redundancy: 27465/264144 objects degraded (10.398%), 19 pgs degraded (PG_DEGRADED) 2024-09-11T08:24:11.427 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:11 smithi150 ceph-mon[39612]: 4.1c scrub starts 2024-09-11T08:24:11.427 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:11 smithi150 ceph-mon[39612]: 4.1c scrub ok 2024-09-11T08:24:11.427 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:11 smithi150 ceph-mon[39612]: 10.0 scrub starts 2024-09-11T08:24:11.427 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:11 smithi150 ceph-mon[39612]: Health check update: Degraded data redundancy: 27465/264144 objects degraded (10.398%), 19 pgs degraded (PG_DEGRADED) 2024-09-11T08:24:11.480 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:11.527 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:11.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:11.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:11.629 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:11.674 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:11.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:11.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:11.775 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:24:11.808 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:11.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:11.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:24:11.910 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:11.939 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:11.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:11.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:12.041 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:12.071 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:12.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:12.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:12.173 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:12.200 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:12.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:12.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:12.302 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:12.326 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:12.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:12.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:12.428 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:12.466 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:12.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:12.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:12.485 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:12 smithi181 ceph-mon[32701]: pgmap v759: 148 pgs: 2 active+clean+scrubbing, 34 active+undersized, 19 active+undersized+degraded, 93 active+clean; 5.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.18k op/s; 27465/264144 objects degraded (10.398%) 2024-09-11T08:24:12.485 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:12 smithi181 ceph-mon[32701]: 10.8 scrub starts 2024-09-11T08:24:12.486 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:12 smithi181 ceph-mon[32701]: 4.f scrub starts 2024-09-11T08:24:12.486 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:12 smithi181 ceph-mon[32701]: 4.f scrub ok 2024-09-11T08:24:12.486 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:12 smithi181 ceph-mon[32701]: 4.18 scrub starts 2024-09-11T08:24:12.486 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:12 smithi181 ceph-mon[32701]: 4.18 scrub ok 2024-09-11T08:24:12.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:12 smithi150 ceph-mon[29276]: pgmap v759: 148 pgs: 2 active+clean+scrubbing, 34 active+undersized, 19 active+undersized+degraded, 93 active+clean; 5.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.18k op/s; 27465/264144 objects degraded (10.398%) 2024-09-11T08:24:12.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:12 smithi150 ceph-mon[29276]: 10.8 scrub starts 2024-09-11T08:24:12.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:12 smithi150 ceph-mon[29276]: 4.f scrub starts 2024-09-11T08:24:12.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:12 smithi150 ceph-mon[29276]: 4.f scrub ok 2024-09-11T08:24:12.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:12 smithi150 ceph-mon[29276]: 4.18 scrub starts 2024-09-11T08:24:12.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:12 smithi150 ceph-mon[29276]: 4.18 scrub ok 2024-09-11T08:24:12.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:12 smithi150 ceph-mon[39612]: pgmap v759: 148 pgs: 2 active+clean+scrubbing, 34 active+undersized, 19 active+undersized+degraded, 93 active+clean; 5.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.18k op/s; 27465/264144 objects degraded (10.398%) 2024-09-11T08:24:12.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:12 smithi150 ceph-mon[39612]: 10.8 scrub starts 2024-09-11T08:24:12.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:12 smithi150 ceph-mon[39612]: 4.f scrub starts 2024-09-11T08:24:12.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:12 smithi150 ceph-mon[39612]: 4.f scrub ok 2024-09-11T08:24:12.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:12 smithi150 ceph-mon[39612]: 4.18 scrub starts 2024-09-11T08:24:12.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:12 smithi150 ceph-mon[39612]: 4.18 scrub ok 2024-09-11T08:24:12.567 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:24:12.599 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:12.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:12.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:24:12.701 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:12.728 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:12.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:12.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:12.830 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:12.864 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:12.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:12.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:12.940 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-11T08:24:12.966 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:13.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:13.042 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:13.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:13.143 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:13.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:13.198 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:13.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:13.299 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:13.324 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:13.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:13.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:13.369 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f /var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/exp.6.4.5 2024-09-11T08:24:13.426 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:13.449 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:13.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:13.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:13.550 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:24:13.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:13 smithi181 ceph-mon[32701]: 10.0 scrub ok 2024-09-11T08:24:13.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:13 smithi181 ceph-mon[32701]: 3.0 scrub starts 2024-09-11T08:24:13.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:13 smithi181 ceph-mon[32701]: 3.0 scrub ok 2024-09-11T08:24:13.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:13 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:24:13.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:13 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:24:13.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:13 smithi181 ceph-mon[32701]: 10.f scrub starts 2024-09-11T08:24:13.582 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:13.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:13.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:24:13.684 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:13.713 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:13.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:13.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:13.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:13 smithi150 ceph-mon[29276]: 10.0 scrub ok 2024-09-11T08:24:13.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:13 smithi150 ceph-mon[29276]: 3.0 scrub starts 2024-09-11T08:24:13.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:13 smithi150 ceph-mon[29276]: 3.0 scrub ok 2024-09-11T08:24:13.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:13 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:24:13.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:13 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:24:13.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:13 smithi150 ceph-mon[29276]: 10.f scrub starts 2024-09-11T08:24:13.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:13 smithi150 ceph-mon[39612]: 10.0 scrub ok 2024-09-11T08:24:13.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:13 smithi150 ceph-mon[39612]: 3.0 scrub starts 2024-09-11T08:24:13.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:13 smithi150 ceph-mon[39612]: 3.0 scrub ok 2024-09-11T08:24:13.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:13 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:24:13.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:13 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:24:13.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:13 smithi150 ceph-mon[39612]: 10.f scrub starts 2024-09-11T08:24:13.815 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:13.844 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:13.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:13.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:13.908 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:24:13.946 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:13.972 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:13.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:13.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:14.074 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:14.117 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:14.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:14.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:14.219 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:14.246 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:14.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:14.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:14.348 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:14.360 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:14 smithi150 ceph-mon[29276]: 5.8 scrub starts 2024-09-11T08:24:14.360 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:14 smithi150 ceph-mon[29276]: 5.8 scrub ok 2024-09-11T08:24:14.360 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:14 smithi150 ceph-mon[29276]: pgmap v760: 148 pgs: 3 active+clean+scrubbing, 34 active+undersized, 19 active+undersized+degraded, 92 active+clean; 5.2 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.16k op/s; 26728/256701 objects degraded (10.412%) 2024-09-11T08:24:14.360 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:14 smithi150 ceph-mon[29276]: osdmap e348: 8 total, 7 up, 8 in 2024-09-11T08:24:14.361 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:14 smithi150 ceph-mon[29276]: 3.3 scrub starts 2024-09-11T08:24:14.361 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:14 smithi150 ceph-mon[29276]: 3.3 scrub ok 2024-09-11T08:24:14.361 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:14 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:24:14.361 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:14 smithi150 ceph-mon[29276]: 4.4 scrub starts 2024-09-11T08:24:14.361 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:14 smithi150 ceph-mon[29276]: 4.4 scrub ok 2024-09-11T08:24:14.361 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:14 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:24:14.361 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:14 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:24:14.361 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:14 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:24:14.361 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:14 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:24:14.363 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:14 smithi150 ceph-mon[39612]: 5.8 scrub starts 2024-09-11T08:24:14.363 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:14 smithi150 ceph-mon[39612]: 5.8 scrub ok 2024-09-11T08:24:14.363 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:14 smithi150 ceph-mon[39612]: pgmap v760: 148 pgs: 3 active+clean+scrubbing, 34 active+undersized, 19 active+undersized+degraded, 92 active+clean; 5.2 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.16k op/s; 26728/256701 objects degraded (10.412%) 2024-09-11T08:24:14.363 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:14 smithi150 ceph-mon[39612]: osdmap e348: 8 total, 7 up, 8 in 2024-09-11T08:24:14.363 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:14 smithi150 ceph-mon[39612]: 3.3 scrub starts 2024-09-11T08:24:14.363 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:14 smithi150 ceph-mon[39612]: 3.3 scrub ok 2024-09-11T08:24:14.363 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:14 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:24:14.363 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:14 smithi150 ceph-mon[39612]: 4.4 scrub starts 2024-09-11T08:24:14.363 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:14 smithi150 ceph-mon[39612]: 4.4 scrub ok 2024-09-11T08:24:14.363 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:14 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:24:14.363 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:14 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:24:14.364 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:14 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:24:14.364 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:14 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:24:14.397 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:14.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:14.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:14.499 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:14.548 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:14.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:14.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:14.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:14 smithi181 ceph-mon[32701]: 5.8 scrub starts 2024-09-11T08:24:14.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:14 smithi181 ceph-mon[32701]: 5.8 scrub ok 2024-09-11T08:24:14.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:14 smithi181 ceph-mon[32701]: pgmap v760: 148 pgs: 3 active+clean+scrubbing, 34 active+undersized, 19 active+undersized+degraded, 92 active+clean; 5.2 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.16k op/s; 26728/256701 objects degraded (10.412%) 2024-09-11T08:24:14.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:14 smithi181 ceph-mon[32701]: osdmap e348: 8 total, 7 up, 8 in 2024-09-11T08:24:14.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:14 smithi181 ceph-mon[32701]: 3.3 scrub starts 2024-09-11T08:24:14.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:14 smithi181 ceph-mon[32701]: 3.3 scrub ok 2024-09-11T08:24:14.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:14 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:24:14.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:14 smithi181 ceph-mon[32701]: 4.4 scrub starts 2024-09-11T08:24:14.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:14 smithi181 ceph-mon[32701]: 4.4 scrub ok 2024-09-11T08:24:14.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:14 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:24:14.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:14 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:24:14.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:14 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:24:14.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:14 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:24:14.649 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:14.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:14.676 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:14.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:14.755 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:24:14.755 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:24:14.777 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:14.794 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:24:14.794 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:24:14.801 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:14.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:14.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:14.831 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:24:14.832 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:24:14.877 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:24:14.877 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:24:14.904 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:14.911 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:24:14.911 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:24:14.915 INFO:teuthology.orchestra.run.smithi150.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-11T08:24:14.929 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:14.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:14.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:14.948 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:24:14.948 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:24:14.978 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:24:14.978 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:24:15.031 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:15.062 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:15.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:15.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:15.164 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:15.206 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:15.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:15.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:15.308 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:24:15.337 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:15.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:15.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:24:15.439 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:15.479 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:15.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:15.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:15.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:24:15 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:24:15] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:24:15.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:15 smithi181 ceph-mon[32701]: 10.8 scrub ok 2024-09-11T08:24:15.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:15 smithi181 ceph-mon[32701]: 5.a scrub starts 2024-09-11T08:24:15.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:15 smithi181 ceph-mon[32701]: 5.a scrub ok 2024-09-11T08:24:15.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:15 smithi181 ceph-mon[32701]: 5.0 scrub starts 2024-09-11T08:24:15.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:15 smithi181 ceph-mon[32701]: 5.0 scrub ok 2024-09-11T08:24:15.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:15 smithi181 ceph-mon[32701]: 10.f scrub ok 2024-09-11T08:24:15.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:15 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2834251918' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:24:15.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:15 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:24:15.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:15 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:24:15.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:15 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:24:15.581 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:15.607 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:15.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:15.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:15.708 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:24:15.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:15 smithi150 ceph-mon[39612]: 10.8 scrub ok 2024-09-11T08:24:15.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:15 smithi150 ceph-mon[39612]: 5.a scrub starts 2024-09-11T08:24:15.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:15 smithi150 ceph-mon[39612]: 5.a scrub ok 2024-09-11T08:24:15.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:15 smithi150 ceph-mon[39612]: 5.0 scrub starts 2024-09-11T08:24:15.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:15 smithi150 ceph-mon[39612]: 5.0 scrub ok 2024-09-11T08:24:15.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:15 smithi150 ceph-mon[39612]: 10.f scrub ok 2024-09-11T08:24:15.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:15 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2834251918' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:24:15.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:15 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:24:15.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:15 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:24:15.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:15 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:24:15.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:15 smithi150 ceph-mon[29276]: 10.8 scrub ok 2024-09-11T08:24:15.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:15 smithi150 ceph-mon[29276]: 5.a scrub starts 2024-09-11T08:24:15.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:15 smithi150 ceph-mon[29276]: 5.a scrub ok 2024-09-11T08:24:15.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:15 smithi150 ceph-mon[29276]: 5.0 scrub starts 2024-09-11T08:24:15.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:15 smithi150 ceph-mon[29276]: 5.0 scrub ok 2024-09-11T08:24:15.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:15 smithi150 ceph-mon[29276]: 10.f scrub ok 2024-09-11T08:24:15.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:15 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2834251918' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:24:15.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:15 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:24:15.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:15 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:24:15.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:15 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:24:15.754 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:15.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:15.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:24:15.856 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:15.887 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:15.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:15.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:15.947 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-11T08:24:15.947 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-11T08:24:15.989 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:15.990 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:24:16.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:16.042 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:16.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:16.143 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:16.176 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:16.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:16.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:16.278 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:24:16.312 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:16.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:16.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:24:16.414 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:16.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:16.447 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:16.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:16.548 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:16.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:16 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:24:16.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:16 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:24:16.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:16 smithi181 ceph-mon[32701]: pgmap v762: 148 pgs: 1 active+clean+scrubbing, 34 active+undersized, 19 active+undersized+degraded, 94 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.29k op/s; 25667/247098 objects degraded (10.387%) 2024-09-11T08:24:16.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:16 smithi181 ceph-mon[32701]: osdmap e349: 8 total, 7 up, 8 in 2024-09-11T08:24:16.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:16 smithi181 ceph-mon[32701]: 10.d scrub starts 2024-09-11T08:24:16.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:16 smithi181 ceph-mon[32701]: Health check update: Degraded data redundancy: 26728/256701 objects degraded (10.412%), 19 pgs degraded (PG_DEGRADED) 2024-09-11T08:24:16.577 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:16.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:16.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:16.679 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:16.714 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:16.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:16.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:16.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:16 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:24:16.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:16 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:24:16.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:16 smithi150 ceph-mon[29276]: pgmap v762: 148 pgs: 1 active+clean+scrubbing, 34 active+undersized, 19 active+undersized+degraded, 94 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.29k op/s; 25667/247098 objects degraded (10.387%) 2024-09-11T08:24:16.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:16 smithi150 ceph-mon[29276]: osdmap e349: 8 total, 7 up, 8 in 2024-09-11T08:24:16.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:16 smithi150 ceph-mon[29276]: 10.d scrub starts 2024-09-11T08:24:16.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:16 smithi150 ceph-mon[29276]: Health check update: Degraded data redundancy: 26728/256701 objects degraded (10.412%), 19 pgs degraded (PG_DEGRADED) 2024-09-11T08:24:16.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:16 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:24:16.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:16 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:24:16.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:16 smithi150 ceph-mon[39612]: pgmap v762: 148 pgs: 1 active+clean+scrubbing, 34 active+undersized, 19 active+undersized+degraded, 94 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.29k op/s; 25667/247098 objects degraded (10.387%) 2024-09-11T08:24:16.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:16 smithi150 ceph-mon[39612]: osdmap e349: 8 total, 7 up, 8 in 2024-09-11T08:24:16.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:16 smithi150 ceph-mon[39612]: 10.d scrub starts 2024-09-11T08:24:16.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:16 smithi150 ceph-mon[39612]: Health check update: Degraded data redundancy: 26728/256701 objects degraded (10.412%), 19 pgs degraded (PG_DEGRADED) 2024-09-11T08:24:16.816 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:24:16.853 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:16.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:16.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:24:16.955 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:16.988 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:16.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:16.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:17.090 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:17.133 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:17.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:17.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:17.235 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:17.291 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:17.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:17.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:17.393 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:24:17.423 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:17 smithi150 ceph-mon[29276]: 4.19 scrub starts 2024-09-11T08:24:17.423 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:17 smithi150 ceph-mon[29276]: 4.19 scrub ok 2024-09-11T08:24:17.423 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:17 smithi150 ceph-mon[29276]: 6.6 scrub starts 2024-09-11T08:24:17.423 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:17 smithi150 ceph-mon[29276]: 6.6 scrub ok 2024-09-11T08:24:17.424 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:17 smithi150 ceph-mon[29276]: 3.1e scrub starts 2024-09-11T08:24:17.424 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:17 smithi150 ceph-mon[29276]: 3.1e scrub ok 2024-09-11T08:24:17.424 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:17 smithi150 ceph-mon[29276]: 3.f scrub starts 2024-09-11T08:24:17.424 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:17 smithi150 ceph-mon[29276]: 3.f scrub ok 2024-09-11T08:24:17.424 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:17 smithi150 ceph-mon[39612]: 4.19 scrub starts 2024-09-11T08:24:17.424 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:17 smithi150 ceph-mon[39612]: 4.19 scrub ok 2024-09-11T08:24:17.424 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:17 smithi150 ceph-mon[39612]: 6.6 scrub starts 2024-09-11T08:24:17.424 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:17 smithi150 ceph-mon[39612]: 6.6 scrub ok 2024-09-11T08:24:17.424 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:17 smithi150 ceph-mon[39612]: 3.1e scrub starts 2024-09-11T08:24:17.424 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:17 smithi150 ceph-mon[39612]: 3.1e scrub ok 2024-09-11T08:24:17.425 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:17 smithi150 ceph-mon[39612]: 3.f scrub starts 2024-09-11T08:24:17.425 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:17 smithi150 ceph-mon[39612]: 3.f scrub ok 2024-09-11T08:24:17.441 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:17.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:17.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:24:17.543 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:17.572 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:17.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:17.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:17.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:17 smithi181 ceph-mon[32701]: 4.19 scrub starts 2024-09-11T08:24:17.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:17 smithi181 ceph-mon[32701]: 4.19 scrub ok 2024-09-11T08:24:17.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:17 smithi181 ceph-mon[32701]: 6.6 scrub starts 2024-09-11T08:24:17.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:17 smithi181 ceph-mon[32701]: 6.6 scrub ok 2024-09-11T08:24:17.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:17 smithi181 ceph-mon[32701]: 3.1e scrub starts 2024-09-11T08:24:17.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:17 smithi181 ceph-mon[32701]: 3.1e scrub ok 2024-09-11T08:24:17.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:17 smithi181 ceph-mon[32701]: 3.f scrub starts 2024-09-11T08:24:17.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:17 smithi181 ceph-mon[32701]: 3.f scrub ok 2024-09-11T08:24:17.674 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:17.701 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:17.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:17.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:17.803 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:17.845 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:17.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:17.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:17.947 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:17.948 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T08:24:17.992 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:17.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:17.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:18.094 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:24:18.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:18.148 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:18.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:24:18.249 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:24:18.276 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:18.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:18.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:24:18.378 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:18.406 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [5] live_osds: [0, 1, 2, 3, 4, 6, 7] 2024-09-11T08:24:18.406 INFO:tasks.thrashosds.thrasher:Reviving osd 5 2024-09-11T08:24:18.406 INFO:tasks.cephadm.osd.5:Restarting osd.5 (starting--it wasn't running)... 2024-09-11T08:24:18.406 DEBUG:teuthology.orchestra.run.smithi181:osd.5> sudo journalctl -f -n 0 -u ceph-c1bddff4-7013-11ef-bcea-c7b262605968@osd.5.service 2024-09-11T08:24:18.408 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:18.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:18.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:18.410 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl start ceph-c1bddff4-7013-11ef-bcea-c7b262605968@osd.5 2024-09-11T08:24:18.510 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:18.549 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:18.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:18.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:18.569 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:18 smithi150 ceph-mon[29276]: 6.f scrub starts 2024-09-11T08:24:18.569 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:18 smithi150 ceph-mon[29276]: 6.f scrub ok 2024-09-11T08:24:18.569 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:18 smithi150 ceph-mon[29276]: pgmap v764: 148 pgs: 1 active+clean+scrubbing, 34 active+undersized, 19 active+undersized+degraded, 94 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 710 op/s; 25667/247098 objects degraded (10.387%) 2024-09-11T08:24:18.569 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:18 smithi150 ceph-mon[29276]: 10.d scrub ok 2024-09-11T08:24:18.569 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:18 smithi150 ceph-mon[29276]: 3.19 scrub starts 2024-09-11T08:24:18.569 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:18 smithi150 ceph-mon[29276]: 3.19 scrub ok 2024-09-11T08:24:18.569 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:18 smithi150 ceph-mon[29276]: 10.c scrub starts 2024-09-11T08:24:18.570 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:18 smithi150 ceph-mon[39612]: 6.f scrub starts 2024-09-11T08:24:18.570 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:18 smithi150 ceph-mon[39612]: 6.f scrub ok 2024-09-11T08:24:18.570 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:18 smithi150 ceph-mon[39612]: pgmap v764: 148 pgs: 1 active+clean+scrubbing, 34 active+undersized, 19 active+undersized+degraded, 94 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 710 op/s; 25667/247098 objects degraded (10.387%) 2024-09-11T08:24:18.570 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:18 smithi150 ceph-mon[39612]: 10.d scrub ok 2024-09-11T08:24:18.570 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:18 smithi150 ceph-mon[39612]: 3.19 scrub starts 2024-09-11T08:24:18.570 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:18 smithi150 ceph-mon[39612]: 3.19 scrub ok 2024-09-11T08:24:18.570 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:18 smithi150 ceph-mon[39612]: 10.c scrub starts 2024-09-11T08:24:18.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:18 smithi181 ceph-mon[32701]: 6.f scrub starts 2024-09-11T08:24:18.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:18 smithi181 ceph-mon[32701]: 6.f scrub ok 2024-09-11T08:24:18.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:18 smithi181 ceph-mon[32701]: pgmap v764: 148 pgs: 1 active+clean+scrubbing, 34 active+undersized, 19 active+undersized+degraded, 94 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 710 op/s; 25667/247098 objects degraded (10.387%) 2024-09-11T08:24:18.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:18 smithi181 ceph-mon[32701]: 10.d scrub ok 2024-09-11T08:24:18.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:18 smithi181 ceph-mon[32701]: 3.19 scrub starts 2024-09-11T08:24:18.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:18 smithi181 ceph-mon[32701]: 3.19 scrub ok 2024-09-11T08:24:18.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:18 smithi181 ceph-mon[32701]: 10.c scrub starts 2024-09-11T08:24:18.574 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:24:18 smithi181 systemd[1]: Starting Ceph osd.5 for c1bddff4-7013-11ef-bcea-c7b262605968... 2024-09-11T08:24:18.652 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:24:18.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:18.694 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:18.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:24:18.795 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:18.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:18.837 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:18.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:18.938 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:18.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:18.976 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:18.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:18.988 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:24:18 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:24:18] "GET /metrics HTTP/1.1" 200 224676 "" "Prometheus/2.43.0" 2024-09-11T08:24:18.989 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:24:18 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:24:18.990 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:24:18 smithi181 podman[134983]: 2024-09-11 08:24:18.728491233 +0000 UTC m=+0.030888988 image pull 4a918172a49766385959227c53e75e2c302e923539324792cb5bee6cad0b2f2b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40 2024-09-11T08:24:18.990 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:24:18 smithi181 podman[134983]: 2024-09-11 08:24:18.855183561 +0000 UTC m=+0.157581293 container create e14575866f2926266900afa9f87cf11b3722409d86cc47d459919272ace33cf0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5-activate, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-09-11T08:24:19.077 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:19.112 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:19.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:19.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:19.214 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:19.243 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:24:19 smithi181 podman[134983]: 2024-09-11 08:24:19.141602767 +0000 UTC m=+0.444000496 container init e14575866f2926266900afa9f87cf11b3722409d86cc47d459919272ace33cf0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5-activate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.vendor=CentOS) 2024-09-11T08:24:19.243 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:24:19 smithi181 podman[134983]: 2024-09-11 08:24:19.145113232 +0000 UTC m=+0.447510967 container start e14575866f2926266900afa9f87cf11b3722409d86cc47d459919272ace33cf0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5-activate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, io.buildah.version=1.37.2) 2024-09-11T08:24:19.248 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:19.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:19.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:19.288 INFO:teuthology.orchestra.run.smithi150.stderr:nodeep-scrub is unset 2024-09-11T08:24:19.350 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:19.389 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:19.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:19.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:19.492 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:19.533 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:19.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:19.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:19.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:19 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1531254319' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:24:19.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:19 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:24:19.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:19 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:24:19.573 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:24:19 smithi181 podman[134983]: 2024-09-11 08:24:19.24250647 +0000 UTC m=+0.544904217 container attach e14575866f2926266900afa9f87cf11b3722409d86cc47d459919272ace33cf0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5-activate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, ceph=True) 2024-09-11T08:24:19.621 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:19 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1531254319' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:24:19.621 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:19 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:24:19.621 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:19 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:24:19.621 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:19 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1531254319' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:24:19.622 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:19 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:24:19.622 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:19 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:24:19.636 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:24:19.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:19.682 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:19.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:24:19.783 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:19.835 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:19.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:19.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:19.937 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:24:19.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:19.972 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:19.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:24:20.048 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:24:19 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5-activate[135013]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-11T08:24:20.049 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:24:19 smithi181 bash[134983]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-11T08:24:20.049 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:24:19 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5-activate[135013]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-11T08:24:20.049 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:24:19 smithi181 bash[134983]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-11T08:24:20.049 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:24:19 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5-activate[135013]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-11T08:24:20.049 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:24:19 smithi181 bash[134983]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-11T08:24:20.049 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:24:19 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5-activate[135013]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-11T08:24:20.049 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:24:19 smithi181 bash[134983]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-11T08:24:20.049 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:24:19 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5-activate[135013]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-09-11T08:24:20.049 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:24:19 smithi181 bash[134983]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-09-11T08:24:20.049 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:24:19 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5-activate[135013]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-11T08:24:20.049 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:24:19 smithi181 bash[134983]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-11T08:24:20.049 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:24:19 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5-activate[135013]: --> ceph-volume raw activate successful for osd ID: 5 2024-09-11T08:24:20.050 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:24:19 smithi181 bash[134983]: --> ceph-volume raw activate successful for osd ID: 5 2024-09-11T08:24:20.050 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:24:19 smithi181 podman[135189]: 2024-09-11 08:24:19.73207939 +0000 UTC m=+0.037072305 container died e14575866f2926266900afa9f87cf11b3722409d86cc47d459919272ace33cf0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5-activate, org.label-schema.build-date=20240731, ceph=True, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-09-11T08:24:20.073 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:20.107 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:20.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:20.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:20.209 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:20.242 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:20.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:20.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:20.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:20 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:24:20.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:20 smithi181 ceph-mon[32701]: pgmap v765: 148 pgs: 1 active+clean+scrubbing, 34 active+undersized, 19 active+undersized+degraded, 94 active+clean; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.42k op/s; 23873/230058 objects degraded (10.377%) 2024-09-11T08:24:20.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:20 smithi181 ceph-mon[32701]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:24:20.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:20 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1531254319' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:24:20.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:20 smithi181 ceph-mon[32701]: osdmap e350: 8 total, 7 up, 8 in 2024-09-11T08:24:20.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:20 smithi181 ceph-mon[32701]: 2.0 scrub starts 2024-09-11T08:24:20.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:20 smithi181 ceph-mon[32701]: 2.0 scrub ok 2024-09-11T08:24:20.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:20 smithi181 ceph-mon[32701]: 10.c scrub ok 2024-09-11T08:24:20.344 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:20.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:20.387 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:20.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:20.488 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:20.523 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:20.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:20.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:20.579 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:20 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:24:20.579 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:20 smithi150 ceph-mon[29276]: pgmap v765: 148 pgs: 1 active+clean+scrubbing, 34 active+undersized, 19 active+undersized+degraded, 94 active+clean; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.42k op/s; 23873/230058 objects degraded (10.377%) 2024-09-11T08:24:20.579 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:20 smithi150 ceph-mon[29276]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:24:20.580 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:20 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1531254319' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:24:20.580 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:20 smithi150 ceph-mon[29276]: osdmap e350: 8 total, 7 up, 8 in 2024-09-11T08:24:20.580 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:20 smithi150 ceph-mon[29276]: 2.0 scrub starts 2024-09-11T08:24:20.580 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:20 smithi150 ceph-mon[29276]: 2.0 scrub ok 2024-09-11T08:24:20.580 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:20 smithi150 ceph-mon[29276]: 10.c scrub ok 2024-09-11T08:24:20.580 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:20 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:24:20.580 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:20 smithi150 ceph-mon[39612]: pgmap v765: 148 pgs: 1 active+clean+scrubbing, 34 active+undersized, 19 active+undersized+degraded, 94 active+clean; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.42k op/s; 23873/230058 objects degraded (10.377%) 2024-09-11T08:24:20.580 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:20 smithi150 ceph-mon[39612]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:24:20.580 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:20 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1531254319' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:24:20.580 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:20 smithi150 ceph-mon[39612]: osdmap e350: 8 total, 7 up, 8 in 2024-09-11T08:24:20.580 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:20 smithi150 ceph-mon[39612]: 2.0 scrub starts 2024-09-11T08:24:20.581 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:20 smithi150 ceph-mon[39612]: 2.0 scrub ok 2024-09-11T08:24:20.581 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:20 smithi150 ceph-mon[39612]: 10.c scrub ok 2024-09-11T08:24:20.625 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:20.627 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:24:20 smithi181 podman[135189]: 2024-09-11 08:24:20.372737397 +0000 UTC m=+0.677730313 container remove e14575866f2926266900afa9f87cf11b3722409d86cc47d459919272ace33cf0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5-activate, GIT_CLEAN=True, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-11T08:24:20.658 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:20.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:20.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:20.759 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:24:20.790 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:20.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:20.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:24:20.892 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:20.918 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:20.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:20.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:20.952 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:24:20 smithi181 podman[135273]: 2024-09-11 08:24:20.532251476 +0000 UTC m=+0.022816987 image pull 4a918172a49766385959227c53e75e2c302e923539324792cb5bee6cad0b2f2b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40 2024-09-11T08:24:20.952 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:24:20 smithi181 podman[135273]: 2024-09-11 08:24:20.665719639 +0000 UTC m=+0.156285132 container create 5c7014cbea51deea3fcbbd66fb19d6ade19f9a377ba2e605c8790809987ad9fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, CEPH_POINT_RELEASE=) 2024-09-11T08:24:21.014 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:24:21.020 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:24:21.054 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-11T08:24:21.054 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [ 2024-09-11T08:24:21.055 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:21.055 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:187058 10.8 10:140e55aa:::benchmark_data_smithi150_158067_object56814:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e350)", 2024-09-11T08:24:21.055 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:24:21.027056+0000", 2024-09-11T08:24:21.055 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.02213321, 2024-09-11T08:24:21.055 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.022165745000000001, 2024-09-11T08:24:21.055 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:24:21.055 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "waiting for sub ops", 2024-09-11T08:24:21.055 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:24:21.055 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:24:21.055 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:24:21.055 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 187058 2024-09-11T08:24:21.055 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:21.055 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:24:21.055 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:21.056 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:24:21.056 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:21.027056+0000", 2024-09-11T08:24:21.056 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:24:21.056 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:21.056 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:21.056 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:24:21.056 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:21.027056+0000", 2024-09-11T08:24:21.056 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:24:21.056 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:21.056 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:21.056 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:24:21.056 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:21.027055+0000", 2024-09-11T08:24:21.056 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:24:21.056 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:21.057 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:21.057 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:24:21.057 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:21.027061+0000", 2024-09-11T08:24:21.057 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.3010000000000003e-06 2024-09-11T08:24:21.057 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:21.057 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:21.057 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:24:21.057 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:21.027071+0000", 2024-09-11T08:24:21.057 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0327e-05 2024-09-11T08:24:21.057 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:21.057 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:21.057 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:24:21.057 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:21.027077+0000", 2024-09-11T08:24:21.058 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.3079999999999998e-06 2024-09-11T08:24:21.058 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:21.058 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:21.058 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:21.058 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:21.029065+0000", 2024-09-11T08:24:21.058 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.001988517 2024-09-11T08:24:21.058 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:21.058 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:21.058 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:24:21.058 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:21.029180+0000", 2024-09-11T08:24:21.058 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000115097 2024-09-11T08:24:21.058 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:21.058 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:21.058 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 2,7", 2024-09-11T08:24:21.059 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:21.029301+0000", 2024-09-11T08:24:21.059 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000120745 2024-09-11T08:24:21.059 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:21.059 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:21.059 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:24:21.059 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:21.037554+0000", 2024-09-11T08:24:21.059 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0082525299999999992 2024-09-11T08:24:21.059 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:21.059 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:21.059 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:24:21.059 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:21.037571+0000", 2024-09-11T08:24:21.059 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.7711999999999999e-05 2024-09-11T08:24:21.059 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:21.059 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:24:21.059 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:21.060 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:21.060 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:21.060 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:187073 10.8 10:10312464:::benchmark_data_smithi150_158067_object56829:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e350)", 2024-09-11T08:24:21.060 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:24:21.044618+0000", 2024-09-11T08:24:21.060 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.004571922, 2024-09-11T08:24:21.060 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0047195409999999998, 2024-09-11T08:24:21.060 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:24:21.060 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "waiting for sub ops", 2024-09-11T08:24:21.060 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:24:21.060 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:24:21.060 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:24:21.060 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 187073 2024-09-11T08:24:21.060 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:21.060 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:24:21.061 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:21.061 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:24:21.061 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:21.044618+0000", 2024-09-11T08:24:21.061 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:24:21.061 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:21.061 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:21.061 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:24:21.061 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:21.044618+0000", 2024-09-11T08:24:21.061 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:24:21.061 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:21.061 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:21.061 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:24:21.061 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:21.044616+0000", 2024-09-11T08:24:21.061 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:24:21.061 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:21.062 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:21.062 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:24:21.062 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:21.044624+0000", 2024-09-11T08:24:21.062 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.7959999999999996e-06 2024-09-11T08:24:21.062 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:21.062 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:21.062 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:24:21.062 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:21.044638+0000", 2024-09-11T08:24:21.062 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4201e-05 2024-09-11T08:24:21.062 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:21.062 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:21.062 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:24:21.062 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:21.044644+0000", 2024-09-11T08:24:21.063 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.1310000000000003e-06 2024-09-11T08:24:21.063 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:21.063 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:21.063 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:21.063 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:21.044758+0000", 2024-09-11T08:24:21.063 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00011356700000000001 2024-09-11T08:24:21.063 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:21.063 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:21.063 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:24:21.063 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:21.046973+0000", 2024-09-11T08:24:21.063 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.002215243 2024-09-11T08:24:21.063 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:21.063 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:21.063 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 2,7", 2024-09-11T08:24:21.064 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:21.047094+0000", 2024-09-11T08:24:21.064 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000120544 2024-09-11T08:24:21.064 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:21.064 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:24:21.064 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:21.064 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:21.064 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:21.064 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.15636.0:187074 10.c e350/308 10:32243128:::benchmark_data_smithi150_158067_object56830:head v 350'11645, mlcod=350'11643)", 2024-09-11T08:24:21.064 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:24:21.044907+0000", 2024-09-11T08:24:21.064 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.004282887, 2024-09-11T08:24:21.064 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00449849, 2024-09-11T08:24:21.064 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:24:21.064 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "started", 2024-09-11T08:24:21.064 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:24:21.064 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:21.065 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:24:21.065 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:21.044907+0000", 2024-09-11T08:24:21.065 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:24:21.065 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:21.065 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:21.065 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:24:21.065 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:21.044907+0000", 2024-09-11T08:24:21.065 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:24:21.065 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:21.065 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:21.065 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:24:21.065 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:21.044906+0000", 2024-09-11T08:24:21.065 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:24:21.065 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:21.065 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:21.066 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:24:21.066 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:21.044913+0000", 2024-09-11T08:24:21.066 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.2599999999999999e-06 2024-09-11T08:24:21.066 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:21.066 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:21.066 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:24:21.066 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:21.044922+0000", 2024-09-11T08:24:21.066 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.6209999999999993e-06 2024-09-11T08:24:21.066 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:21.066 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:21.066 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:24:21.066 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:21.044927+0000", 2024-09-11T08:24:21.066 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.7670000000000003e-06 2024-09-11T08:24:21.066 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:21.067 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:21.067 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:21.067 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:21.045030+0000", 2024-09-11T08:24:21.067 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00010298600000000001 2024-09-11T08:24:21.067 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:21.067 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:21.067 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:24:21.067 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:21.045047+0000", 2024-09-11T08:24:21.067 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.6971000000000001e-05 2024-09-11T08:24:21.067 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:21.067 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:24:21.067 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:21.067 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:21.067 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:21.068 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop_reply(client.15636.0:187058 10.8 e350/308)", 2024-09-11T08:24:21.068 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:24:21.047704+0000", 2024-09-11T08:24:21.068 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.0014854849999999999, 2024-09-11T08:24:21.068 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.001751018, 2024-09-11T08:24:21.068 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:24:21.068 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "queued for pg", 2024-09-11T08:24:21.068 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:24:21.068 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:21.068 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:24:21.068 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:21.047704+0000", 2024-09-11T08:24:21.068 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:24:21.068 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:21.068 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:21.068 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:24:21.069 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:21.047704+0000", 2024-09-11T08:24:21.069 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:24:21.069 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:21.069 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:21.069 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:24:21.069 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:21.047701+0000", 2024-09-11T08:24:21.069 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999971 2024-09-11T08:24:21.069 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:21.069 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:21.069 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:24:21.069 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:21.047709+0000", 2024-09-11T08:24:21.069 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.2160000000000006e-06 2024-09-11T08:24:21.069 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:21.070 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:21.070 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:24:21.070 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:21.047723+0000", 2024-09-11T08:24:21.070 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3755e-05 2024-09-11T08:24:21.070 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:21.070 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:21.070 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:24:21.070 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:21.047729+0000", 2024-09-11T08:24:21.070 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.9739999999999999e-06 2024-09-11T08:24:21.070 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:21.070 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:24:21.070 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:21.070 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:21.070 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-09-11T08:24:21.071 INFO:teuthology.orchestra.run.smithi150.stdout: "num_ops": 4 2024-09-11T08:24:21.071 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-11T08:24:21.072 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:21.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:21.074 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-11T08:24:21.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:24:21.176 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:21.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:21.216 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:21.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:21.288 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:24:20 smithi181 podman[135273]: 2024-09-11 08:24:20.952234093 +0000 UTC m=+0.442799597 container init 5c7014cbea51deea3fcbbd66fb19d6ade19f9a377ba2e605c8790809987ad9fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-11T08:24:21.288 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:24:20 smithi181 podman[135273]: 2024-09-11 08:24:20.960191522 +0000 UTC m=+0.450757018 container start 5c7014cbea51deea3fcbbd66fb19d6ade19f9a377ba2e605c8790809987ad9fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-11T08:24:21.289 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:24:20 smithi181 ceph-osd[135322]: -- 172.21.15.181:0/3665293135 <== mon.1 v2:172.21.15.181:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5566cb360d00 con 0x5566cb333400 2024-09-11T08:24:21.289 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:24:21 smithi181 bash[135273]: 5c7014cbea51deea3fcbbd66fb19d6ade19f9a377ba2e605c8790809987ad9fb 2024-09-11T08:24:21.289 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:24:21 smithi181 systemd[1]: Started Ceph osd.5 for c1bddff4-7013-11ef-bcea-c7b262605968. 2024-09-11T08:24:21.317 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:21.350 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:21.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:21.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:21.452 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:21.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:21 smithi150 ceph-mon[29276]: Health check update: Degraded data redundancy: 23873/230058 objects degraded (10.377%), 19 pgs degraded (PG_DEGRADED) 2024-09-11T08:24:21.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:21 smithi150 ceph-mon[39612]: Health check update: Degraded data redundancy: 23873/230058 objects degraded (10.377%), 19 pgs degraded (PG_DEGRADED) 2024-09-11T08:24:21.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:21.504 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:21.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:21.516 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:24:21.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:21 smithi181 ceph-mon[32701]: Health check update: Degraded data redundancy: 23873/230058 objects degraded (10.377%), 19 pgs degraded (PG_DEGRADED) 2024-09-11T08:24:21.573 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:24:21 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5[135300]: 2024-09-11T08:24:21.506+0000 7f2955d83440 -1 Falling back to public interface 2024-09-11T08:24:21.605 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:21.636 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:21.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:21.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:21.738 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:24:21.769 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:21.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:21.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:24:21.823 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-11T08:24:21.871 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:21.924 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:21.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:21.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:22.025 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:22.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:22.072 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:22.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:22.173 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:22.224 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:22.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:22.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:22.325 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:22.357 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:22.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:22.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:22.459 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:22.461 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T08:24:22.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:22 smithi150 ceph-mon[29276]: pgmap v767: 148 pgs: 1 active+clean+scrubbing, 34 active+undersized, 19 active+undersized+degraded, 94 active+clean; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 653 B/s rd, 0 B/s wr, 1.04k op/s; 23873/230058 objects degraded (10.377%) 2024-09-11T08:24:22.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:22 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:24:22.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:22 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:24:22.489 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:22.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:22 smithi150 ceph-mon[39612]: pgmap v767: 148 pgs: 1 active+clean+scrubbing, 34 active+undersized, 19 active+undersized+degraded, 94 active+clean; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 653 B/s rd, 0 B/s wr, 1.04k op/s; 23873/230058 objects degraded (10.377%) 2024-09-11T08:24:22.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:22 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:24:22.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:22 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:24:22.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:22.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:22.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:22 smithi181 ceph-mon[32701]: pgmap v767: 148 pgs: 1 active+clean+scrubbing, 34 active+undersized, 19 active+undersized+degraded, 94 active+clean; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 653 B/s rd, 0 B/s wr, 1.04k op/s; 23873/230058 objects degraded (10.377%) 2024-09-11T08:24:22.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:22 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:24:22.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:22 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:24:22.590 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:22.620 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:22.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:22.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:22.722 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:22.747 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:22.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:22.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:22.850 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:22.889 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:22.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:22.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:22.991 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:23.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:23.029 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:23.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:23.130 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:23.165 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:23.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:23.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:23.267 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:23.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:23.296 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:23.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:23.397 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:23.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:23.436 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:23.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:23.537 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:23.569 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:23.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:23.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:23.671 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:24:23.708 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:23.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:23.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:24:23.810 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:24:23.845 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:23.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:23.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:24:23.947 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:23.971 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:23.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:23.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:24.038 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:24:24.073 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:24.107 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:24.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:24.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:24.209 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:24.231 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:24.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:24.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:24.333 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:24.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:24.385 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:24.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:24.486 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:24.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:24 smithi150 ceph-mon[29276]: 4.1e deep-scrub starts 2024-09-11T08:24:24.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:24 smithi150 ceph-mon[29276]: pgmap v768: 148 pgs: 1 active+clean+scrubbing, 34 active+undersized, 19 active+undersized+degraded, 94 active+clean; 4.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.03k op/s; 23131/222378 objects degraded (10.402%) 2024-09-11T08:24:24.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:24 smithi150 ceph-mon[29276]: 4.1e deep-scrub ok 2024-09-11T08:24:24.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:24 smithi150 ceph-mon[29276]: 6.3 scrub starts 2024-09-11T08:24:24.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:24 smithi150 ceph-mon[29276]: 6.3 scrub ok 2024-09-11T08:24:24.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:24 smithi150 ceph-mon[39612]: 4.1e deep-scrub starts 2024-09-11T08:24:24.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:24 smithi150 ceph-mon[39612]: pgmap v768: 148 pgs: 1 active+clean+scrubbing, 34 active+undersized, 19 active+undersized+degraded, 94 active+clean; 4.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.03k op/s; 23131/222378 objects degraded (10.402%) 2024-09-11T08:24:24.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:24 smithi150 ceph-mon[39612]: 4.1e deep-scrub ok 2024-09-11T08:24:24.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:24 smithi150 ceph-mon[39612]: 6.3 scrub starts 2024-09-11T08:24:24.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:24 smithi150 ceph-mon[39612]: 6.3 scrub ok 2024-09-11T08:24:24.521 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:24.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:24.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:24.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:24 smithi181 ceph-mon[32701]: 4.1e deep-scrub starts 2024-09-11T08:24:24.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:24 smithi181 ceph-mon[32701]: pgmap v768: 148 pgs: 1 active+clean+scrubbing, 34 active+undersized, 19 active+undersized+degraded, 94 active+clean; 4.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.03k op/s; 23131/222378 objects degraded (10.402%) 2024-09-11T08:24:24.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:24 smithi181 ceph-mon[32701]: 4.1e deep-scrub ok 2024-09-11T08:24:24.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:24 smithi181 ceph-mon[32701]: 6.3 scrub starts 2024-09-11T08:24:24.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:24 smithi181 ceph-mon[32701]: 6.3 scrub ok 2024-09-11T08:24:24.623 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:24.644 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:24.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:24.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:24.746 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:24.775 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:24.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:24.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:24.876 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:24.907 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T08:24:24.907 INFO:teuthology.orchestra.run.smithi181.stderr:0 2024-09-11T08:24:24.907 INFO:teuthology.orchestra.run.smithi181.stderr:1 2024-09-11T08:24:24.907 INFO:teuthology.orchestra.run.smithi181.stderr:2 2024-09-11T08:24:24.907 INFO:teuthology.orchestra.run.smithi181.stderr:abort 2024-09-11T08:24:24.907 INFO:teuthology.orchestra.run.smithi181.stderr:assert 2024-09-11T08:24:24.907 INFO:teuthology.orchestra.run.smithi181.stderr:bluefs debug_inject_read_zeros 2024-09-11T08:24:24.907 INFO:teuthology.orchestra.run.smithi181.stderr:bluefs files list 2024-09-11T08:24:24.907 INFO:teuthology.orchestra.run.smithi181.stderr:bluefs stats 2024-09-11T08:24:24.907 INFO:teuthology.orchestra.run.smithi181.stderr:bluestore allocator dump block 2024-09-11T08:24:24.907 INFO:teuthology.orchestra.run.smithi181.stderr:bluestore allocator fragmentation block 2024-09-11T08:24:24.908 INFO:teuthology.orchestra.run.smithi181.stderr:admin_socket: invalid command 2024-09-11T08:24:24.910 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:24.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:24.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:24.920 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:24:24.920 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:24:24.965 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:24:24.966 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:24:25.007 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:24:25.008 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:24:25.012 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:25.048 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:24:25.048 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:24:25.058 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:25.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:25.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:25.084 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:24:25.084 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:24:25.085 INFO:teuthology.orchestra.run.smithi150.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-11T08:24:25.125 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:24:25.126 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:24:25.159 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:24:25.159 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:24:25.160 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:25.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:25.206 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:25.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:25.307 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:25.346 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:25.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:25.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:25.448 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:24:25.482 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:25.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:25.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:24:25.519 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:25 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:24:25.520 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:25 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:24:25.520 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:25 smithi181 ceph-mon[32701]: 6.1 scrub starts 2024-09-11T08:24:25.520 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:25 smithi181 ceph-mon[32701]: 6.1 scrub ok 2024-09-11T08:24:25.520 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:25 smithi181 ceph-mon[32701]: 4.c scrub starts 2024-09-11T08:24:25.520 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:25 smithi181 ceph-mon[32701]: 4.c scrub ok 2024-09-11T08:24:25.520 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:25 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/326025033' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:24:25.520 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:25 smithi181 ceph-mon[32701]: 10.1 scrub starts 2024-09-11T08:24:25.520 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:24:25 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:24:25] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:24:25.585 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:25.617 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:25.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:25.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:25.659 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T08:24:25.660 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-5, ['dump_ops_in_flight'] 2024-09-11T08:24:25.718 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:25.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:25 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:24:25.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:25 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:24:25.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:25 smithi150 ceph-mon[29276]: 6.1 scrub starts 2024-09-11T08:24:25.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:25 smithi150 ceph-mon[29276]: 6.1 scrub ok 2024-09-11T08:24:25.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:25 smithi150 ceph-mon[29276]: 4.c scrub starts 2024-09-11T08:24:25.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:25 smithi150 ceph-mon[29276]: 4.c scrub ok 2024-09-11T08:24:25.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:25 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/326025033' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:24:25.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:25 smithi150 ceph-mon[29276]: 10.1 scrub starts 2024-09-11T08:24:25.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:25 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:24:25.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:25 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:24:25.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:25 smithi150 ceph-mon[39612]: 6.1 scrub starts 2024-09-11T08:24:25.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:25 smithi150 ceph-mon[39612]: 6.1 scrub ok 2024-09-11T08:24:25.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:25 smithi150 ceph-mon[39612]: 4.c scrub starts 2024-09-11T08:24:25.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:25 smithi150 ceph-mon[39612]: 4.c scrub ok 2024-09-11T08:24:25.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:25 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/326025033' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:24:25.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:25 smithi150 ceph-mon[39612]: 10.1 scrub starts 2024-09-11T08:24:25.761 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:25.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:25.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:25.823 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:24:25 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5[135300]: 2024-09-11T08:24:25.518+0000 7f2955d83440 -1 osd.5 339 log_to_monitors true 2024-09-11T08:24:25.864 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:25.899 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:25.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:25.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:26.001 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:26.030 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:26.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:26.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:26.132 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:26.168 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:26.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:26.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:26.270 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:26.303 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:26.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:26.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:26.405 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:26.415 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-11T08:24:26.415 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-11T08:24:26.442 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:26.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:26.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:26.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:26 smithi150 ceph-mon[29276]: 6.1b scrub starts 2024-09-11T08:24:26.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:26 smithi150 ceph-mon[29276]: 6.1b scrub ok 2024-09-11T08:24:26.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:26 smithi150 ceph-mon[29276]: pgmap v769: 148 pgs: 34 active+undersized, 19 active+undersized+degraded, 95 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.18k op/s; 22034/212409 objects degraded (10.373%) 2024-09-11T08:24:26.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:26 smithi150 ceph-mon[29276]: osdmap e351: 8 total, 7 up, 8 in 2024-09-11T08:24:26.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:26 smithi150 ceph-mon[29276]: from='osd.5 v2:172.21.15.181:6804/1821561027' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-11T08:24:26.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:26 smithi150 ceph-mon[29276]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-11T08:24:26.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:26 smithi150 ceph-mon[29276]: 4.13 scrub starts 2024-09-11T08:24:26.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:26 smithi150 ceph-mon[29276]: 4.13 scrub ok 2024-09-11T08:24:26.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:26 smithi150 ceph-mon[29276]: Health check update: Degraded data redundancy: 22034/212409 objects degraded (10.373%), 19 pgs degraded (PG_DEGRADED) 2024-09-11T08:24:26.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:26 smithi150 ceph-mon[29276]: 3.11 scrub starts 2024-09-11T08:24:26.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:26 smithi150 ceph-mon[29276]: 3.11 scrub ok 2024-09-11T08:24:26.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:26 smithi150 ceph-mon[39612]: 6.1b scrub starts 2024-09-11T08:24:26.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:26 smithi150 ceph-mon[39612]: 6.1b scrub ok 2024-09-11T08:24:26.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:26 smithi150 ceph-mon[39612]: pgmap v769: 148 pgs: 34 active+undersized, 19 active+undersized+degraded, 95 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.18k op/s; 22034/212409 objects degraded (10.373%) 2024-09-11T08:24:26.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:26 smithi150 ceph-mon[39612]: osdmap e351: 8 total, 7 up, 8 in 2024-09-11T08:24:26.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:26 smithi150 ceph-mon[39612]: from='osd.5 v2:172.21.15.181:6804/1821561027' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-11T08:24:26.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:26 smithi150 ceph-mon[39612]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-11T08:24:26.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:26 smithi150 ceph-mon[39612]: 4.13 scrub starts 2024-09-11T08:24:26.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:26 smithi150 ceph-mon[39612]: 4.13 scrub ok 2024-09-11T08:24:26.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:26 smithi150 ceph-mon[39612]: Health check update: Degraded data redundancy: 22034/212409 objects degraded (10.373%), 19 pgs degraded (PG_DEGRADED) 2024-09-11T08:24:26.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:26 smithi150 ceph-mon[39612]: 3.11 scrub starts 2024-09-11T08:24:26.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:26 smithi150 ceph-mon[39612]: 3.11 scrub ok 2024-09-11T08:24:26.545 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:26.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:26 smithi181 ceph-mon[32701]: 6.1b scrub starts 2024-09-11T08:24:26.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:26 smithi181 ceph-mon[32701]: 6.1b scrub ok 2024-09-11T08:24:26.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:26 smithi181 ceph-mon[32701]: pgmap v769: 148 pgs: 34 active+undersized, 19 active+undersized+degraded, 95 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.18k op/s; 22034/212409 objects degraded (10.373%) 2024-09-11T08:24:26.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:26 smithi181 ceph-mon[32701]: osdmap e351: 8 total, 7 up, 8 in 2024-09-11T08:24:26.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:26 smithi181 ceph-mon[32701]: from='osd.5 v2:172.21.15.181:6804/1821561027' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-11T08:24:26.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:26 smithi181 ceph-mon[32701]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-11T08:24:26.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:26 smithi181 ceph-mon[32701]: 4.13 scrub starts 2024-09-11T08:24:26.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:26 smithi181 ceph-mon[32701]: 4.13 scrub ok 2024-09-11T08:24:26.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:26 smithi181 ceph-mon[32701]: Health check update: Degraded data redundancy: 22034/212409 objects degraded (10.373%), 19 pgs degraded (PG_DEGRADED) 2024-09-11T08:24:26.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:26 smithi181 ceph-mon[32701]: 3.11 scrub starts 2024-09-11T08:24:26.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:26 smithi181 ceph-mon[32701]: 3.11 scrub ok 2024-09-11T08:24:26.574 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:24:26 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5[135300]: 2024-09-11T08:24:26.359+0000 7f294a328640 -1 osd.5 339 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T08:24:26.583 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:26.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:26.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:26.685 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:26.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:26.713 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:26.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:26.814 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:26.850 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:26.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:26.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:26.952 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:26.988 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:26.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:26.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:27.012 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:24:27.092 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:27.135 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:27.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:27.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:27.237 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:24:27.268 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:27.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:27.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:24:27.370 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:27.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:27.400 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:27.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:27.501 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:27.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:27.523 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:27.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:27.587 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:27 smithi150 ceph-mon[29276]: 6.18 scrub starts 2024-09-11T08:24:27.587 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:27 smithi150 ceph-mon[29276]: 6.18 scrub ok 2024-09-11T08:24:27.587 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:27 smithi150 ceph-mon[29276]: 10.1 scrub ok 2024-09-11T08:24:27.587 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:27 smithi150 ceph-mon[29276]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-11T08:24:27.587 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:27 smithi150 ceph-mon[29276]: osdmap e352: 8 total, 7 up, 8 in 2024-09-11T08:24:27.587 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:27 smithi150 ceph-mon[29276]: from='osd.5 v2:172.21.15.181:6804/1821561027' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-09-11T08:24:27.587 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:27 smithi150 ceph-mon[29276]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-09-11T08:24:27.587 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:27 smithi150 ceph-mon[29276]: 6.8 scrub starts 2024-09-11T08:24:27.587 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:27 smithi150 ceph-mon[29276]: 6.8 scrub ok 2024-09-11T08:24:27.588 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:27 smithi150 ceph-mon[39612]: 6.18 scrub starts 2024-09-11T08:24:27.588 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:27 smithi150 ceph-mon[39612]: 6.18 scrub ok 2024-09-11T08:24:27.588 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:27 smithi150 ceph-mon[39612]: 10.1 scrub ok 2024-09-11T08:24:27.588 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:27 smithi150 ceph-mon[39612]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-11T08:24:27.588 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:27 smithi150 ceph-mon[39612]: osdmap e352: 8 total, 7 up, 8 in 2024-09-11T08:24:27.588 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:27 smithi150 ceph-mon[39612]: from='osd.5 v2:172.21.15.181:6804/1821561027' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-09-11T08:24:27.588 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:27 smithi150 ceph-mon[39612]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-09-11T08:24:27.588 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:27 smithi150 ceph-mon[39612]: 6.8 scrub starts 2024-09-11T08:24:27.588 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:27 smithi150 ceph-mon[39612]: 6.8 scrub ok 2024-09-11T08:24:27.624 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:24:27.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:27.650 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:27.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:24:27.751 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:24:27.773 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:27.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:27.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:24:27.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:27 smithi181 ceph-mon[32701]: 6.18 scrub starts 2024-09-11T08:24:27.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:27 smithi181 ceph-mon[32701]: 6.18 scrub ok 2024-09-11T08:24:27.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:27 smithi181 ceph-mon[32701]: 10.1 scrub ok 2024-09-11T08:24:27.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:27 smithi181 ceph-mon[32701]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-11T08:24:27.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:27 smithi181 ceph-mon[32701]: osdmap e352: 8 total, 7 up, 8 in 2024-09-11T08:24:27.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:27 smithi181 ceph-mon[32701]: from='osd.5 v2:172.21.15.181:6804/1821561027' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-09-11T08:24:27.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:27 smithi181 ceph-mon[32701]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-09-11T08:24:27.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:27 smithi181 ceph-mon[32701]: 6.8 scrub starts 2024-09-11T08:24:27.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:27 smithi181 ceph-mon[32701]: 6.8 scrub ok 2024-09-11T08:24:27.875 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:27.897 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:27.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:27.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:27.999 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:28.020 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:28.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:28.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:28.122 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:28.144 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:28.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:28.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:28.236 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-09-11T08:24:28.236 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:24:28.246 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:28.269 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:28.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:28.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:28.370 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:24:28.393 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:28.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:28.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:24:28.416 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T08:24:28.424 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:28 smithi150 ceph-mon[39612]: pgmap v772: 148 pgs: 34 active+undersized, 19 active+undersized+degraded, 95 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 656 B/s rd, 0 B/s wr, 755 op/s; 22034/212409 objects degraded (10.373%) 2024-09-11T08:24:28.425 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:28 smithi150 ceph-mon[39612]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-11T08:24:28.425 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:28 smithi150 ceph-mon[39612]: osd.5 v2:172.21.15.181:6804/1821561027 boot 2024-09-11T08:24:28.425 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:28 smithi150 ceph-mon[39612]: osdmap e353: 8 total, 8 up, 8 in 2024-09-11T08:24:28.425 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:28 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-11T08:24:28.425 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:28 smithi150 ceph-mon[39612]: osdmap e354: 8 total, 8 up, 8 in 2024-09-11T08:24:28.425 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:28 smithi150 ceph-mon[29276]: pgmap v772: 148 pgs: 34 active+undersized, 19 active+undersized+degraded, 95 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 656 B/s rd, 0 B/s wr, 755 op/s; 22034/212409 objects degraded (10.373%) 2024-09-11T08:24:28.425 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:28 smithi150 ceph-mon[29276]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-11T08:24:28.425 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:28 smithi150 ceph-mon[29276]: osd.5 v2:172.21.15.181:6804/1821561027 boot 2024-09-11T08:24:28.425 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:28 smithi150 ceph-mon[29276]: osdmap e353: 8 total, 8 up, 8 in 2024-09-11T08:24:28.426 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:28 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-11T08:24:28.426 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:28 smithi150 ceph-mon[29276]: osdmap e354: 8 total, 8 up, 8 in 2024-09-11T08:24:28.495 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:28.517 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:28.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:28.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:28.618 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:28.640 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:28.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:28.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:28.717 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:28 smithi181 ceph-mon[32701]: pgmap v772: 148 pgs: 34 active+undersized, 19 active+undersized+degraded, 95 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 656 B/s rd, 0 B/s wr, 755 op/s; 22034/212409 objects degraded (10.373%) 2024-09-11T08:24:28.718 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:28 smithi181 ceph-mon[32701]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-11T08:24:28.718 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:28 smithi181 ceph-mon[32701]: osd.5 v2:172.21.15.181:6804/1821561027 boot 2024-09-11T08:24:28.718 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:28 smithi181 ceph-mon[32701]: osdmap e353: 8 total, 8 up, 8 in 2024-09-11T08:24:28.718 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:28 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-11T08:24:28.718 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:28 smithi181 ceph-mon[32701]: osdmap e354: 8 total, 8 up, 8 in 2024-09-11T08:24:28.738 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:24:28 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:24:28] "GET /metrics HTTP/1.1" 200 224719 "" "Prometheus/2.43.0" 2024-09-11T08:24:28.741 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:28.763 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:28.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:28.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:28.865 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:28.888 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:28.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:28.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:28.990 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:29.012 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:29.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:29.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:29.073 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:24:28 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:24:29.114 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:29.135 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:29.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:29.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:29.237 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:29.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:29.259 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:29.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:29.360 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:24:29.388 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:29.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:29.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:24:29.490 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:29.513 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:29.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:29.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:29.615 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:24:29.644 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:29.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:29.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:24:29.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:29 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:24:29.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:29 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:24:29.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:29 smithi181 ceph-mon[32701]: 5.17 scrub starts 2024-09-11T08:24:29.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:29 smithi181 ceph-mon[32701]: 5.17 scrub ok 2024-09-11T08:24:29.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:29 smithi181 ceph-mon[32701]: 5.12 scrub starts 2024-09-11T08:24:29.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:29 smithi181 ceph-mon[32701]: 5.12 scrub ok 2024-09-11T08:24:29.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:29 smithi181 ceph-mon[32701]: 3.16 scrub starts 2024-09-11T08:24:29.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:29 smithi181 ceph-mon[32701]: 3.16 scrub ok 2024-09-11T08:24:29.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:29 smithi181 ceph-mon[32701]: osdmap e355: 8 total, 8 up, 8 in 2024-09-11T08:24:29.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:29 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:24:29.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:29 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:24:29.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:29 smithi150 ceph-mon[29276]: 5.17 scrub starts 2024-09-11T08:24:29.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:29 smithi150 ceph-mon[29276]: 5.17 scrub ok 2024-09-11T08:24:29.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:29 smithi150 ceph-mon[29276]: 5.12 scrub starts 2024-09-11T08:24:29.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:29 smithi150 ceph-mon[29276]: 5.12 scrub ok 2024-09-11T08:24:29.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:29 smithi150 ceph-mon[29276]: 3.16 scrub starts 2024-09-11T08:24:29.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:29 smithi150 ceph-mon[29276]: 3.16 scrub ok 2024-09-11T08:24:29.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:29 smithi150 ceph-mon[29276]: osdmap e355: 8 total, 8 up, 8 in 2024-09-11T08:24:29.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:29 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:24:29.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:29 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:24:29.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:29 smithi150 ceph-mon[39612]: 5.17 scrub starts 2024-09-11T08:24:29.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:29 smithi150 ceph-mon[39612]: 5.17 scrub ok 2024-09-11T08:24:29.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:29 smithi150 ceph-mon[39612]: 5.12 scrub starts 2024-09-11T08:24:29.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:29 smithi150 ceph-mon[39612]: 5.12 scrub ok 2024-09-11T08:24:29.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:29 smithi150 ceph-mon[39612]: 3.16 scrub starts 2024-09-11T08:24:29.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:29 smithi150 ceph-mon[39612]: 3.16 scrub ok 2024-09-11T08:24:29.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:29 smithi150 ceph-mon[39612]: osdmap e355: 8 total, 8 up, 8 in 2024-09-11T08:24:29.745 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:29.785 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:29.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:29.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:29.887 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:29.923 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:29.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:29.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:30.025 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:24:30.068 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:30.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:30.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:24:30.170 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:30.203 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:30.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:30.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:30.304 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:30.337 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-11T08:24:30.354 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:30.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:30.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:30.456 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:30.494 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:30.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:30.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:30.596 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:30.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:30.633 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:30.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:30.661 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-11T08:24:30.699 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:30 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:24:30.699 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:30 smithi181 ceph-mon[32701]: pgmap v775: 148 pgs: 3 remapped+peering, 21 peering, 1 active+undersized, 123 active+clean; 4.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.13k op/s 2024-09-11T08:24:30.699 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:30 smithi181 ceph-mon[32701]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 22034/212409 objects degraded (10.373%), 19 pgs degraded) 2024-09-11T08:24:30.699 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:30 smithi181 ceph-mon[32701]: Cluster is now healthy 2024-09-11T08:24:30.699 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:30 smithi181 ceph-mon[32701]: 2.2 scrub starts 2024-09-11T08:24:30.699 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:30 smithi181 ceph-mon[32701]: 2.2 scrub ok 2024-09-11T08:24:30.699 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:30 smithi181 ceph-mon[32701]: 6.1e scrub starts 2024-09-11T08:24:30.699 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:30 smithi181 ceph-mon[32701]: 6.1e scrub ok 2024-09-11T08:24:30.734 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:30.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:30 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:24:30.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:30 smithi150 ceph-mon[29276]: pgmap v775: 148 pgs: 3 remapped+peering, 21 peering, 1 active+undersized, 123 active+clean; 4.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.13k op/s 2024-09-11T08:24:30.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:30 smithi150 ceph-mon[29276]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 22034/212409 objects degraded (10.373%), 19 pgs degraded) 2024-09-11T08:24:30.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:30 smithi150 ceph-mon[29276]: Cluster is now healthy 2024-09-11T08:24:30.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:30 smithi150 ceph-mon[29276]: 2.2 scrub starts 2024-09-11T08:24:30.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:30 smithi150 ceph-mon[29276]: 2.2 scrub ok 2024-09-11T08:24:30.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:30 smithi150 ceph-mon[29276]: 6.1e scrub starts 2024-09-11T08:24:30.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:30 smithi150 ceph-mon[29276]: 6.1e scrub ok 2024-09-11T08:24:30.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:30 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:24:30.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:30 smithi150 ceph-mon[39612]: pgmap v775: 148 pgs: 3 remapped+peering, 21 peering, 1 active+undersized, 123 active+clean; 4.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.13k op/s 2024-09-11T08:24:30.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:30 smithi150 ceph-mon[39612]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 22034/212409 objects degraded (10.373%), 19 pgs degraded) 2024-09-11T08:24:30.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:30 smithi150 ceph-mon[39612]: Cluster is now healthy 2024-09-11T08:24:30.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:30 smithi150 ceph-mon[39612]: 2.2 scrub starts 2024-09-11T08:24:30.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:30 smithi150 ceph-mon[39612]: 2.2 scrub ok 2024-09-11T08:24:30.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:30 smithi150 ceph-mon[39612]: 6.1e scrub starts 2024-09-11T08:24:30.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:30 smithi150 ceph-mon[39612]: 6.1e scrub ok 2024-09-11T08:24:30.765 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:30.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:30.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:30.866 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:24:30.905 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:30.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:30.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:24:31.007 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:31.041 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:31.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:31.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:31.143 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:31.177 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:31.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:31.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:31.279 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:31.308 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:31.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:31.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:31.396 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:24:31.411 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:31.437 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:31.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:31.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:31.471 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:31 smithi150 ceph-mon[29276]: 6.7 scrub starts 2024-09-11T08:24:31.471 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:31 smithi150 ceph-mon[29276]: 6.7 scrub ok 2024-09-11T08:24:31.471 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:31 smithi150 ceph-mon[39612]: 6.7 scrub starts 2024-09-11T08:24:31.471 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:31 smithi150 ceph-mon[39612]: 6.7 scrub ok 2024-09-11T08:24:31.539 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:31.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:31.576 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:31.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:31.677 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:31.699 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:24:31.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:31.709 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:31.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:31.746 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:24:31.809 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:31.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:31 smithi181 ceph-mon[32701]: 6.7 scrub starts 2024-09-11T08:24:31.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:31 smithi181 ceph-mon[32701]: 6.7 scrub ok 2024-09-11T08:24:31.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:31.838 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:31.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:31.939 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:31.960 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:31.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:31.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:32.062 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:32.083 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:32.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:32.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:32.185 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:32.211 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:32.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:32.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:32.312 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:32.345 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:32.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:32.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:32.447 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:32.488 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:32.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:32.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:32.590 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:32.622 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:32.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:32.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:32.700 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:32 smithi150 ceph-mon[29276]: pgmap v777: 148 pgs: 3 remapped+peering, 21 peering, 1 active+undersized, 123 active+clean; 4.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 956 op/s 2024-09-11T08:24:32.700 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:32 smithi150 ceph-mon[29276]: 3.4 deep-scrub starts 2024-09-11T08:24:32.700 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:32 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/596415305' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:24:32.700 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:32 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:24:32.700 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:32 smithi150 ceph-mon[39612]: pgmap v777: 148 pgs: 3 remapped+peering, 21 peering, 1 active+undersized, 123 active+clean; 4.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 956 op/s 2024-09-11T08:24:32.700 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:32 smithi150 ceph-mon[39612]: 3.4 deep-scrub starts 2024-09-11T08:24:32.700 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:32 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/596415305' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:24:32.700 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:32 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:24:32.702 INFO:teuthology.orchestra.run.smithi150.stderr:noscrub is set 2024-09-11T08:24:32.725 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:32.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:32.775 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:32.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:32.800 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:32 smithi181 ceph-mon[32701]: pgmap v777: 148 pgs: 3 remapped+peering, 21 peering, 1 active+undersized, 123 active+clean; 4.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 956 op/s 2024-09-11T08:24:32.800 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:32 smithi181 ceph-mon[32701]: 3.4 deep-scrub starts 2024-09-11T08:24:32.800 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:32 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/596415305' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:24:32.800 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:32 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:24:32.875 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:32.911 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:32.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:32.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:33.013 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:33.038 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:33.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:33.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:33.140 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:33.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:33.180 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:33.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:33.281 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:33.316 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:33.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:33.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:33.418 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:24:33.458 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:33.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:33.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:24:33.559 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:33.592 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:33.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:33.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:33.627 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:33 smithi150 ceph-mon[29276]: 4.7 scrub starts 2024-09-11T08:24:33.628 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:33 smithi150 ceph-mon[29276]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:24:33.628 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:33 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:24:33.628 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:33 smithi150 ceph-mon[29276]: osdmap e356: 8 total, 8 up, 8 in 2024-09-11T08:24:33.628 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:33 smithi150 ceph-mon[39612]: 4.7 scrub starts 2024-09-11T08:24:33.628 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:33 smithi150 ceph-mon[39612]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:24:33.628 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:33 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:24:33.628 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:33 smithi150 ceph-mon[39612]: osdmap e356: 8 total, 8 up, 8 in 2024-09-11T08:24:33.694 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:33.729 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:33.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:33.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:33.821 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:33 smithi181 ceph-mon[32701]: 4.7 scrub starts 2024-09-11T08:24:33.821 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:33 smithi181 ceph-mon[32701]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:24:33.822 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:33 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:24:33.822 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:33 smithi181 ceph-mon[32701]: osdmap e356: 8 total, 8 up, 8 in 2024-09-11T08:24:33.831 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:33.862 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:33.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:33.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:33.963 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:34.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:34.019 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:34.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:34.120 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:34.148 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:34.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:34.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:34.250 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:24:34.281 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:34.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:34.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:24:34.383 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:34.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:34.410 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:34.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:34.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:34 smithi150 ceph-mon[29276]: pgmap v779: 148 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 remapped+peering, 6 peering, 137 active+clean; 4.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 891 B/s rd, 0 B/s wr, 994 op/s; 4005/195297 objects degraded (2.051%); 0 B/s, 19 objects/s recovering 2024-09-11T08:24:34.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:34 smithi150 ceph-mon[29276]: Health check failed: Degraded data redundancy: 4005/195297 objects degraded (2.051%), 1 pg degraded (PG_DEGRADED) 2024-09-11T08:24:34.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:34 smithi150 ceph-mon[39612]: pgmap v779: 148 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 remapped+peering, 6 peering, 137 active+clean; 4.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 891 B/s rd, 0 B/s wr, 994 op/s; 4005/195297 objects degraded (2.051%); 0 B/s, 19 objects/s recovering 2024-09-11T08:24:34.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:34 smithi150 ceph-mon[39612]: Health check failed: Degraded data redundancy: 4005/195297 objects degraded (2.051%), 1 pg degraded (PG_DEGRADED) 2024-09-11T08:24:34.511 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:34.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:34.554 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:34.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:34.655 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:24:34.699 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:34.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:34.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:24:34.802 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:24:34.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:34 smithi181 ceph-mon[32701]: pgmap v779: 148 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 remapped+peering, 6 peering, 137 active+clean; 4.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 891 B/s rd, 0 B/s wr, 994 op/s; 4005/195297 objects degraded (2.051%); 0 B/s, 19 objects/s recovering 2024-09-11T08:24:34.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:34 smithi181 ceph-mon[32701]: Health check failed: Degraded data redundancy: 4005/195297 objects degraded (2.051%), 1 pg degraded (PG_DEGRADED) 2024-09-11T08:24:34.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:34.841 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:34.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:24:34.942 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:34.979 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:34.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:34.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:35.081 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:35.109 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:35.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:35.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:35.212 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:35.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:35.247 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:35.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:35.348 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:24:35.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:35.397 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:35.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:24:35.480 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:24:35 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:24:35] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:24:35.494 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T08:24:35.498 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:35.553 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:35.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:35.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:35.655 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:35.688 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:35.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:35.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:35.790 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:35.827 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:35.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:35.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:35.929 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:35.957 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:35.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:35.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:36.059 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:36.106 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:36.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:36.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:36.180 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-11T08:24:36.180 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-09-11T08:24:36.180 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.180 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.15636.0:199655 10.6 e356/354 10:63f98c8d:::benchmark_data_smithi150_158067_object69411:head v 356'12381, mlcod=341'8366)", 2024-09-11T08:24:36.180 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:24:36.146468+0000", 2024-09-11T08:24:36.180 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.032233747, 2024-09-11T08:24:36.180 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.032263747000000002, 2024-09-11T08:24:36.181 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:24:36.181 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-11T08:24:36.181 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:24:36.181 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.181 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:24:36.181 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.146468+0000", 2024-09-11T08:24:36.181 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:24:36.181 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.181 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.181 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:24:36.181 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.146468+0000", 2024-09-11T08:24:36.181 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:24:36.181 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.181 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.182 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:24:36.182 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.146466+0000", 2024-09-11T08:24:36.182 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:24:36.182 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.182 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.182 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:24:36.182 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.146474+0000", 2024-09-11T08:24:36.182 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2889999999999998e-06 2024-09-11T08:24:36.182 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.182 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.182 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:24:36.182 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.146486+0000", 2024-09-11T08:24:36.182 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1148000000000001e-05 2024-09-11T08:24:36.182 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.183 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.183 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:24:36.183 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.146491+0000", 2024-09-11T08:24:36.183 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3970000000000002e-06 2024-09-11T08:24:36.183 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.183 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.183 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:24:36.183 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.159041+0000", 2024-09-11T08:24:36.183 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.012549592 2024-09-11T08:24:36.184 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.184 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.184 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:24:36.184 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.159068+0000", 2024-09-11T08:24:36.184 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7696999999999999e-05 2024-09-11T08:24:36.184 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:24:36.184 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:24:36.184 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:24:36.184 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.184 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.184 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.15636.0:199657 10.6 e356/354 10:6c18b8ac:::benchmark_data_smithi150_158067_object69413:head v 356'12382, mlcod=341'8366)", 2024-09-11T08:24:36.184 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:24:36.146496+0000", 2024-09-11T08:24:36.184 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.032204871000000003, 2024-09-11T08:24:36.185 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.032298659, 2024-09-11T08:24:36.185 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:24:36.185 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-11T08:24:36.185 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:24:36.185 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.185 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:24:36.185 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.146496+0000", 2024-09-11T08:24:36.185 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:24:36.185 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.185 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.185 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:24:36.185 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.146496+0000", 2024-09-11T08:24:36.185 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:24:36.185 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.186 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.186 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:24:36.186 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.146496+0000", 2024-09-11T08:24:36.186 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967296 2024-09-11T08:24:36.186 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.186 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.186 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:24:36.186 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.146500+0000", 2024-09-11T08:24:36.186 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7050000000000001e-06 2024-09-11T08:24:36.186 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.186 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.186 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:24:36.186 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.146504+0000", 2024-09-11T08:24:36.186 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0910000000000003e-06 2024-09-11T08:24:36.187 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.187 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.187 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:24:36.187 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.146507+0000", 2024-09-11T08:24:36.187 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9459999999999998e-06 2024-09-11T08:24:36.187 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.187 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.187 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:24:36.187 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.173091+0000", 2024-09-11T08:24:36.188 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.026584222000000001 2024-09-11T08:24:36.188 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.188 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.188 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:24:36.188 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.173114+0000", 2024-09-11T08:24:36.189 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2561999999999998e-05 2024-09-11T08:24:36.189 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:24:36.189 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:24:36.189 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:24:36.189 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.189 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.189 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.15636.0:199668 10.6 e356/354)", 2024-09-11T08:24:36.189 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:24:36.162467+0000", 2024-09-11T08:24:36.190 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.016234416000000002, 2024-09-11T08:24:36.190 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0163792, 2024-09-11T08:24:36.190 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:24:36.190 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "queued for pg", 2024-09-11T08:24:36.190 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:24:36.190 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.190 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:24:36.190 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.162467+0000", 2024-09-11T08:24:36.190 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:24:36.191 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.191 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.191 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:24:36.191 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.162467+0000", 2024-09-11T08:24:36.191 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:24:36.191 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.191 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.191 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:24:36.192 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.162465+0000", 2024-09-11T08:24:36.192 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999981 2024-09-11T08:24:36.192 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.192 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.192 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:24:36.192 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.162475+0000", 2024-09-11T08:24:36.192 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7569999999999999e-06 2024-09-11T08:24:36.192 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.192 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.192 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:24:36.193 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.162488+0000", 2024-09-11T08:24:36.193 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2942000000000001e-05 2024-09-11T08:24:36.193 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.193 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.193 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:24:36.193 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.162495+0000", 2024-09-11T08:24:36.193 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.759e-06 2024-09-11T08:24:36.193 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:24:36.193 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:24:36.193 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:24:36.193 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.193 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.193 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.15636.0:199672 10.a e356/354 10:5796b343:::benchmark_data_smithi150_158067_object69428:head v 356'12611, mlcod=341'8453)", 2024-09-11T08:24:36.193 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:24:36.165122+0000", 2024-09-11T08:24:36.194 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.013579272, 2024-09-11T08:24:36.194 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.013767547999999999, 2024-09-11T08:24:36.194 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:24:36.194 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-11T08:24:36.194 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:24:36.194 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.194 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:24:36.194 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.165122+0000", 2024-09-11T08:24:36.194 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:24:36.194 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.194 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.194 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:24:36.194 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.165122+0000", 2024-09-11T08:24:36.195 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:24:36.195 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.195 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.195 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:24:36.195 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.165121+0000", 2024-09-11T08:24:36.195 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-09-11T08:24:36.195 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.195 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.195 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:24:36.195 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.165129+0000", 2024-09-11T08:24:36.195 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0910000000000004e-06 2024-09-11T08:24:36.195 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.195 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.196 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:24:36.196 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.165153+0000", 2024-09-11T08:24:36.196 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4474999999999999e-05 2024-09-11T08:24:36.196 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.196 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.196 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:24:36.196 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.165160+0000", 2024-09-11T08:24:36.196 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.934e-06 2024-09-11T08:24:36.196 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.196 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.196 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:24:36.196 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.165332+0000", 2024-09-11T08:24:36.196 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017168700000000001 2024-09-11T08:24:36.196 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.197 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.197 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:24:36.197 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.165354+0000", 2024-09-11T08:24:36.197 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2578e-05 2024-09-11T08:24:36.197 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:24:36.197 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:24:36.197 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:24:36.197 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.197 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.198 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.15636.0:199674 10.a e356/354 10:5b614db6:::benchmark_data_smithi150_158067_object69430:head v 356'12612, mlcod=341'8453)", 2024-09-11T08:24:36.198 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:24:36.172532+0000", 2024-09-11T08:24:36.198 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.0061690169999999997, 2024-09-11T08:24:36.198 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0064118639999999998, 2024-09-11T08:24:36.198 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:24:36.198 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-11T08:24:36.198 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:24:36.199 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.199 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:24:36.199 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.172532+0000", 2024-09-11T08:24:36.199 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:24:36.199 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.199 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.199 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:24:36.199 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.172532+0000", 2024-09-11T08:24:36.199 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:24:36.199 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.199 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.199 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:24:36.200 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.172531+0000", 2024-09-11T08:24:36.200 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:24:36.200 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.200 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.200 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:24:36.200 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.172540+0000", 2024-09-11T08:24:36.200 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.9279999999999999e-06 2024-09-11T08:24:36.200 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.200 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.200 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:24:36.200 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.172553+0000", 2024-09-11T08:24:36.200 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2743e-05 2024-09-11T08:24:36.201 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.201 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.201 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:24:36.201 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.172560+0000", 2024-09-11T08:24:36.201 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0820000000000001e-06 2024-09-11T08:24:36.201 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.201 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.201 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:24:36.201 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.176002+0000", 2024-09-11T08:24:36.201 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0034418320000000001 2024-09-11T08:24:36.201 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.201 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.201 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:24:36.202 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.176030+0000", 2024-09-11T08:24:36.202 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7659000000000001e-05 2024-09-11T08:24:36.202 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:24:36.202 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:24:36.202 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:24:36.202 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.202 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.202 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.15636.0:199645 10.6 e356/354 10:67c08bb8:::benchmark_data_smithi150_158067_object69401:head v 356'12378, mlcod=341'8365)", 2024-09-11T08:24:36.202 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:24:36.126581+0000", 2024-09-11T08:24:36.202 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.052120535000000003, 2024-09-11T08:24:36.202 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.052416169999999998, 2024-09-11T08:24:36.202 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:24:36.202 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-11T08:24:36.202 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:24:36.203 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.203 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:24:36.203 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.126581+0000", 2024-09-11T08:24:36.203 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:24:36.203 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.203 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.203 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:24:36.203 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.126581+0000", 2024-09-11T08:24:36.203 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:24:36.203 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.203 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.203 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:24:36.203 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.126580+0000", 2024-09-11T08:24:36.203 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:24:36.204 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.204 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.204 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:24:36.204 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.126590+0000", 2024-09-11T08:24:36.204 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0025e-05 2024-09-11T08:24:36.204 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.204 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.204 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:24:36.204 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.126607+0000", 2024-09-11T08:24:36.204 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7751999999999999e-05 2024-09-11T08:24:36.204 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.204 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.205 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:24:36.205 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.126615+0000", 2024-09-11T08:24:36.205 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1779999999999997e-06 2024-09-11T08:24:36.205 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.205 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.205 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:24:36.205 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.135166+0000", 2024-09-11T08:24:36.205 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0085512880000000006 2024-09-11T08:24:36.205 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.205 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.205 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:24:36.205 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.135215+0000", 2024-09-11T08:24:36.205 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9154999999999999e-05 2024-09-11T08:24:36.206 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:24:36.206 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:24:36.206 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:24:36.206 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.206 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.206 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.15636.0:199651 10.6 e356/354 10:6e50c64f:::benchmark_data_smithi150_158067_object69407:head v 356'12379, mlcod=341'8366)", 2024-09-11T08:24:36.206 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:24:36.137636+0000", 2024-09-11T08:24:36.206 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.041065178000000001, 2024-09-11T08:24:36.206 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.041414060000000003, 2024-09-11T08:24:36.206 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:24:36.206 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-11T08:24:36.206 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:24:36.206 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.206 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:24:36.207 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.137636+0000", 2024-09-11T08:24:36.207 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:24:36.207 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.207 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.207 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:24:36.207 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.137636+0000", 2024-09-11T08:24:36.207 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:24:36.207 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.207 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.207 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:24:36.207 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.137636+0000", 2024-09-11T08:24:36.207 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-09-11T08:24:36.207 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.207 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.208 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:24:36.208 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.137642+0000", 2024-09-11T08:24:36.208 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4389999999999997e-06 2024-09-11T08:24:36.208 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.208 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.208 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:24:36.208 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.137652+0000", 2024-09-11T08:24:36.208 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.8009999999999998e-06 2024-09-11T08:24:36.208 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.208 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.208 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:24:36.208 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.137657+0000", 2024-09-11T08:24:36.209 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1370000000000001e-06 2024-09-11T08:24:36.209 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.209 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.209 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:24:36.209 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.146345+0000", 2024-09-11T08:24:36.209 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0086875609999999999 2024-09-11T08:24:36.209 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.209 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.209 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:24:36.209 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.146366+0000", 2024-09-11T08:24:36.209 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1543e-05 2024-09-11T08:24:36.209 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:24:36.209 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:24:36.209 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:24:36.210 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.210 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.210 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.15636.0:199652 10.6 e356/354 10:6816967f:::benchmark_data_smithi150_158067_object69408:head v 356'12380, mlcod=341'8366)", 2024-09-11T08:24:36.210 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:24:36.140787+0000", 2024-09-11T08:24:36.210 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.037914471999999998, 2024-09-11T08:24:36.210 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.038316535999999998, 2024-09-11T08:24:36.210 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:24:36.210 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-11T08:24:36.210 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:24:36.210 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.210 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:24:36.210 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.140787+0000", 2024-09-11T08:24:36.210 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:24:36.211 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.211 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.211 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:24:36.211 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.140787+0000", 2024-09-11T08:24:36.211 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:24:36.211 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.211 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.211 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:24:36.211 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.140786+0000", 2024-09-11T08:24:36.211 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:24:36.211 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.211 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.211 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:24:36.211 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.140793+0000", 2024-09-11T08:24:36.212 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7970000000000003e-06 2024-09-11T08:24:36.212 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.212 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.212 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:24:36.212 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.140804+0000", 2024-09-11T08:24:36.212 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1659e-05 2024-09-11T08:24:36.212 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.212 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.212 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:24:36.212 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.140810+0000", 2024-09-11T08:24:36.212 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8030000000000002e-06 2024-09-11T08:24:36.212 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.212 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.212 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:24:36.213 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.152692+0000", 2024-09-11T08:24:36.213 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.011881371 2024-09-11T08:24:36.213 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:24:36.213 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:24:36.213 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:24:36.213 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.152720+0000", 2024-09-11T08:24:36.213 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8357999999999999e-05 2024-09-11T08:24:36.213 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:24:36.213 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:24:36.213 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:24:36.213 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:24:36.213 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-09-11T08:24:36.213 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 8 2024-09-11T08:24:36.213 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-11T08:24:36.214 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:24:36.249 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:36.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:36.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:24:36.351 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:36.387 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:36.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:36.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:36.489 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:36.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:36.533 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:36.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:36.634 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:36.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:36.671 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:36.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:36.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:36 smithi150 ceph-mon[29276]: pgmap v780: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 3.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 759 B/s rd, 0 B/s wr, 690 op/s; 7980/190068 objects degraded (4.198%); 0 B/s, 61 objects/s recovering 2024-09-11T08:24:36.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:36 smithi150 ceph-mon[39612]: pgmap v780: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 3.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 759 B/s rd, 0 B/s wr, 690 op/s; 7980/190068 objects degraded (4.198%); 0 B/s, 61 objects/s recovering 2024-09-11T08:24:36.772 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:24:36.806 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:36.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:36.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:24:36.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:36 smithi181 ceph-mon[32701]: pgmap v780: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 3.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 759 B/s rd, 0 B/s wr, 690 op/s; 7980/190068 objects degraded (4.198%); 0 B/s, 61 objects/s recovering 2024-09-11T08:24:36.908 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:24:36.945 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:36.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:36.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:24:37.011 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd reweight 0 1 2024-09-11T08:24:37.047 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:37.098 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:37.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:37.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:37.199 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:37.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:37.244 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:37.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:37.345 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:37.376 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:37.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:37.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:37.478 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:37.510 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:37.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:37.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:37.611 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:37.641 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:37.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:37.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:37.743 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:24:37.786 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:37.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:37.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:24:37.888 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:37.927 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:37.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:37.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:38.029 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:38.072 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:38.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:38.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:38.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:37 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:24:38.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:37 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:24:38.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:37 smithi181 ceph-mon[32701]: pgmap v781: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 3.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 364 op/s; 7980/190068 objects degraded (4.198%); 0 B/s, 51 objects/s recovering 2024-09-11T08:24:38.173 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:38.206 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:38.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:38.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:38.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:37 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:24:38.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:37 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:24:38.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:37 smithi150 ceph-mon[29276]: pgmap v781: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 3.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 364 op/s; 7980/190068 objects degraded (4.198%); 0 B/s, 51 objects/s recovering 2024-09-11T08:24:38.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:37 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:24:38.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:37 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:24:38.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:37 smithi150 ceph-mon[39612]: pgmap v781: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 3.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 364 op/s; 7980/190068 objects degraded (4.198%); 0 B/s, 51 objects/s recovering 2024-09-11T08:24:38.309 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:38.341 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:38.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:38.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:38.444 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:24:38.470 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:38.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:38.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:24:38.572 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:38.616 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:38.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:38.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:38.718 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:38.739 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:24:38 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:24:38] "GET /metrics HTTP/1.1" 200 224457 "" "Prometheus/2.43.0" 2024-09-11T08:24:38.757 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:38.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:38.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:38.858 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:24:38.895 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:38.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:38.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:24:38.997 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:24:39.027 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:39.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:39.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:24:39.040 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:24:39.073 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:24:38 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:24:39.089 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:24:39.089 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:24:39.122 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:24:39.122 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:24:39.128 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:24:39.151 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:24:39.151 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:24:39.163 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:39.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:39.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:24:39.182 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:24:39.183 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:24:39.212 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:24:39.212 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:24:39.252 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:24:39.253 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:24:39.264 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:39.284 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:24:39.287 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:39 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1193273274' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:24:39.287 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:39 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1193273274' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:24:39.287 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:24:39.287 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:24:39.317 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:39.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:39.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:39.330 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:24:39.330 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:24:39.419 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:39.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:39.462 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:39.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:39.563 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:24:39.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:39 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1193273274' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:24:39.595 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:39.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:39.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:24:39.697 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:39.732 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:39.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:39.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:39.796 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-11T08:24:39.797 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 20, 2024-09-11T08:24:39.797 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 600, 2024-09-11T08:24:39.797 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [ 2024-09-11T08:24:39.797 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.797 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15087.0:146930 8.c 8:31c400d5:::benchmark_data_smithi150_107646_object18098:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206)", 2024-09-11T08:24:39.797 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:15:55.523301+0000", 2024-09-11T08:24:39.797 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 524.26827789200001, 2024-09-11T08:24:39.797 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.079087951, 2024-09-11T08:24:39.797 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:24:39.797 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:24:39.797 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:24:39.797 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15087", 2024-09-11T08:24:39.798 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/3299560913", 2024-09-11T08:24:39.798 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 146930 2024-09-11T08:24:39.798 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.798 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:24:39.798 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.798 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:24:39.798 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.523301+0000", 2024-09-11T08:24:39.798 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:24:39.798 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.798 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.798 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:24:39.798 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.523301+0000", 2024-09-11T08:24:39.798 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:24:39.799 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.799 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.799 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:24:39.799 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.523301+0000", 2024-09-11T08:24:39.799 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:24:39.799 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.799 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.799 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:24:39.799 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.523303+0000", 2024-09-11T08:24:39.799 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.069e-06 2024-09-11T08:24:39.799 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.799 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.799 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:24:39.799 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.523308+0000", 2024-09-11T08:24:39.799 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.3409999999999996e-06 2024-09-11T08:24:39.800 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.800 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.800 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:24:39.800 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.523323+0000", 2024-09-11T08:24:39.800 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.5199e-05 2024-09-11T08:24:39.800 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.800 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.800 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.800 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.538637+0000", 2024-09-11T08:24:39.800 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.015313841999999999 2024-09-11T08:24:39.800 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.800 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.800 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:24:39.800 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.538646+0000", 2024-09-11T08:24:39.801 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.6919999999999997e-06 2024-09-11T08:24:39.801 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.801 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.801 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.801 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:56.507259+0000", 2024-09-11T08:24:39.801 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.96861286000000002 2024-09-11T08:24:39.801 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.801 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.801 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:24:39.801 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:56.507270+0000", 2024-09-11T08:24:39.801 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0601000000000001e-05 2024-09-11T08:24:39.802 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.802 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.802 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.802 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.587911+0000", 2024-09-11T08:24:39.802 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.08064096 2024-09-11T08:24:39.802 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.802 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.802 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:24:39.802 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.588002+0000", 2024-09-11T08:24:39.802 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.0873999999999998e-05 2024-09-11T08:24:39.802 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.802 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.803 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 3,5,7", 2024-09-11T08:24:39.803 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.588089+0000", 2024-09-11T08:24:39.803 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.7820999999999998e-05 2024-09-11T08:24:39.803 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.803 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.803 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:24:39.803 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.594545+0000", 2024-09-11T08:24:39.803 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0064555339999999998 2024-09-11T08:24:39.803 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.803 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.803 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:24:39.803 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.602172+0000", 2024-09-11T08:24:39.803 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0076274469999999999 2024-09-11T08:24:39.804 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.804 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.804 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:24:39.804 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.602220+0000", 2024-09-11T08:24:39.804 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.7599999999999998e-05 2024-09-11T08:24:39.804 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.804 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.804 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:24:39.804 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.602252+0000", 2024-09-11T08:24:39.804 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.1680999999999997e-05 2024-09-11T08:24:39.804 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.804 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.804 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:24:39.805 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.602286+0000", 2024-09-11T08:24:39.805 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.4335000000000002e-05 2024-09-11T08:24:39.805 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.805 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.805 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:24:39.805 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.602389+0000", 2024-09-11T08:24:39.805 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00010343799999999999 2024-09-11T08:24:39.805 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:39.805 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:24:39.805 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:39.805 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.805 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.805 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15087.0:146937 8.c 8:3212a04b:::benchmark_data_smithi150_107646_object18105:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-11T08:24:39.806 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:15:55.550521+0000", 2024-09-11T08:24:39.806 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 524.24105800799998, 2024-09-11T08:24:39.806 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.0522560310000002, 2024-09-11T08:24:39.806 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:24:39.806 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:24:39.806 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:24:39.806 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15087", 2024-09-11T08:24:39.806 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/3299560913", 2024-09-11T08:24:39.806 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 146937 2024-09-11T08:24:39.806 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.806 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:24:39.806 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.806 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:24:39.806 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.550521+0000", 2024-09-11T08:24:39.807 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:24:39.807 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.807 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.807 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:24:39.807 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.550521+0000", 2024-09-11T08:24:39.807 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:24:39.807 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.807 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.807 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:24:39.807 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.550521+0000", 2024-09-11T08:24:39.807 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:24:39.807 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.807 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.807 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:24:39.807 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.550524+0000", 2024-09-11T08:24:39.808 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.8100000000000002e-06 2024-09-11T08:24:39.808 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.808 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.808 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:24:39.808 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.550529+0000", 2024-09-11T08:24:39.808 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.0209999999999999e-06 2024-09-11T08:24:39.808 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.808 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.808 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:24:39.808 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.550533+0000", 2024-09-11T08:24:39.808 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.9990000000000002e-06 2024-09-11T08:24:39.808 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.808 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.808 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.809 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.551490+0000", 2024-09-11T08:24:39.809 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00095763200000000004 2024-09-11T08:24:39.809 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.809 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.809 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:24:39.809 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.551498+0000", 2024-09-11T08:24:39.809 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.3919999999999997e-06 2024-09-11T08:24:39.809 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.809 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.809 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.809 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:56.507292+0000", 2024-09-11T08:24:39.809 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.95579458299999998 2024-09-11T08:24:39.809 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.810 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.810 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:24:39.810 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:56.507300+0000", 2024-09-11T08:24:39.810 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.1350000000000003e-06 2024-09-11T08:24:39.810 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.810 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.810 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.810 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.589968+0000", 2024-09-11T08:24:39.810 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.082668492 2024-09-11T08:24:39.810 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.810 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.810 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:24:39.810 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.590039+0000", 2024-09-11T08:24:39.810 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.1237999999999998e-05 2024-09-11T08:24:39.810 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.810 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.811 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 3,5,7", 2024-09-11T08:24:39.811 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.590108+0000", 2024-09-11T08:24:39.811 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.8931999999999997e-05 2024-09-11T08:24:39.811 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.811 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.811 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:24:39.811 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.598232+0000", 2024-09-11T08:24:39.812 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0081239180000000008 2024-09-11T08:24:39.812 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.812 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.812 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:24:39.812 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.602439+0000", 2024-09-11T08:24:39.812 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.004207171 2024-09-11T08:24:39.812 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.812 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.812 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:24:39.812 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.602494+0000", 2024-09-11T08:24:39.812 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.482e-05 2024-09-11T08:24:39.812 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.812 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.813 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:24:39.813 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.602665+0000", 2024-09-11T08:24:39.813 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00017059599999999999 2024-09-11T08:24:39.813 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.813 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.813 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:24:39.813 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.602694+0000", 2024-09-11T08:24:39.813 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.9677999999999999e-05 2024-09-11T08:24:39.813 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.813 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.814 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:24:39.814 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.602777+0000", 2024-09-11T08:24:39.814 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.2998999999999996e-05 2024-09-11T08:24:39.814 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:39.814 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:24:39.814 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:39.814 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.814 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.814 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15087.0:146950 8.c 8:370007d3:::benchmark_data_smithi150_107646_object18118:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-11T08:24:39.814 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:15:55.557762+0000", 2024-09-11T08:24:39.814 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 524.23381780199998, 2024-09-11T08:24:39.814 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.045360783, 2024-09-11T08:24:39.816 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:24:39.816 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:24:39.816 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:24:39.816 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15087", 2024-09-11T08:24:39.816 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/3299560913", 2024-09-11T08:24:39.817 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 146950 2024-09-11T08:24:39.817 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.817 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:24:39.817 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.817 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:24:39.817 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.557762+0000", 2024-09-11T08:24:39.817 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:24:39.817 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.817 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.818 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:24:39.818 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.557762+0000", 2024-09-11T08:24:39.818 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:24:39.818 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.818 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.818 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:24:39.818 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.557760+0000", 2024-09-11T08:24:39.818 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:24:39.818 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.818 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.818 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:24:39.818 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.557766+0000", 2024-09-11T08:24:39.818 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.682e-06 2024-09-11T08:24:39.818 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.819 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.819 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:24:39.819 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.557785+0000", 2024-09-11T08:24:39.819 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.8970999999999998e-05 2024-09-11T08:24:39.819 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.819 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.819 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:24:39.819 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.557791+0000", 2024-09-11T08:24:39.819 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.1380000000000004e-06 2024-09-11T08:24:39.819 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.819 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.819 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.819 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.557905+0000", 2024-09-11T08:24:39.820 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00011333 2024-09-11T08:24:39.820 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.820 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.820 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:24:39.820 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.557915+0000", 2024-09-11T08:24:39.820 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0309000000000001e-05 2024-09-11T08:24:39.820 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.820 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.820 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.820 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:56.507321+0000", 2024-09-11T08:24:39.820 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.94940656499999998 2024-09-11T08:24:39.820 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.820 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.820 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:24:39.821 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:56.507328+0000", 2024-09-11T08:24:39.821 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.7309999999999996e-06 2024-09-11T08:24:39.821 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.821 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.821 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.821 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.591252+0000", 2024-09-11T08:24:39.821 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.083923615 2024-09-11T08:24:39.821 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.821 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.821 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:24:39.821 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.591317+0000", 2024-09-11T08:24:39.821 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.5529000000000002e-05 2024-09-11T08:24:39.821 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.822 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.822 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 3,5,7", 2024-09-11T08:24:39.822 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.591397+0000", 2024-09-11T08:24:39.822 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.9554000000000004e-05 2024-09-11T08:24:39.822 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.822 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.822 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:24:39.822 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.598239+0000", 2024-09-11T08:24:39.822 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0068420030000000001 2024-09-11T08:24:39.822 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.823 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.823 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:24:39.823 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.602543+0000", 2024-09-11T08:24:39.823 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0043036209999999997 2024-09-11T08:24:39.823 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.823 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.823 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:24:39.823 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.602605+0000", 2024-09-11T08:24:39.823 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.2460999999999997e-05 2024-09-11T08:24:39.823 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.823 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.823 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:24:39.823 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.602919+0000", 2024-09-11T08:24:39.823 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000313715 2024-09-11T08:24:39.824 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.824 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.824 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:24:39.824 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.602992+0000", 2024-09-11T08:24:39.824 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.2818000000000007e-05 2024-09-11T08:24:39.824 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.824 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.824 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:24:39.824 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.603122+0000", 2024-09-11T08:24:39.824 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00013082999999999999 2024-09-11T08:24:39.824 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:39.824 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:24:39.824 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:39.824 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.825 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.825 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15087.0:146968 8.c 8:33522256:::benchmark_data_smithi150_107646_object18136:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-11T08:24:39.825 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:15:55.577972+0000", 2024-09-11T08:24:39.825 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 524.21360743499997, 2024-09-11T08:24:39.825 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.025394522, 2024-09-11T08:24:39.825 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:24:39.825 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:24:39.825 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:24:39.825 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15087", 2024-09-11T08:24:39.825 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/3299560913", 2024-09-11T08:24:39.825 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 146968 2024-09-11T08:24:39.825 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.825 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:24:39.825 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.825 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:24:39.826 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.577972+0000", 2024-09-11T08:24:39.826 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:24:39.826 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.826 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.826 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:24:39.826 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.577972+0000", 2024-09-11T08:24:39.826 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:24:39.826 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.826 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.826 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:24:39.826 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.577970+0000", 2024-09-11T08:24:39.826 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:24:39.826 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.827 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.827 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:24:39.827 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.577979+0000", 2024-09-11T08:24:39.827 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.5520000000000005e-06 2024-09-11T08:24:39.827 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.827 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.827 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:24:39.827 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.577996+0000", 2024-09-11T08:24:39.827 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.6960999999999999e-05 2024-09-11T08:24:39.827 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.827 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.828 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:24:39.828 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.578001+0000", 2024-09-11T08:24:39.828 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.6720000000000001e-06 2024-09-11T08:24:39.828 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.828 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.828 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.828 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.578049+0000", 2024-09-11T08:24:39.828 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.7596999999999998e-05 2024-09-11T08:24:39.828 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.828 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.828 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:24:39.829 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.578057+0000", 2024-09-11T08:24:39.829 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.3100000000000001e-06 2024-09-11T08:24:39.829 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.829 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.829 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.829 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:56.507350+0000", 2024-09-11T08:24:39.829 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.92929289100000001 2024-09-11T08:24:39.829 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.829 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.829 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:24:39.829 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:56.507357+0000", 2024-09-11T08:24:39.829 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.6390000000000003e-06 2024-09-11T08:24:39.829 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.829 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.830 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.830 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.592617+0000", 2024-09-11T08:24:39.830 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0852604079999999 2024-09-11T08:24:39.830 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.830 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.830 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:24:39.830 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.592715+0000", 2024-09-11T08:24:39.830 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.8097000000000006e-05 2024-09-11T08:24:39.830 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.830 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.830 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 3,5,7", 2024-09-11T08:24:39.830 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.592796+0000", 2024-09-11T08:24:39.830 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.0505000000000003e-05 2024-09-11T08:24:39.830 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.830 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.831 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:24:39.831 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.601827+0000", 2024-09-11T08:24:39.831 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0090313560000000008 2024-09-11T08:24:39.831 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.831 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.831 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:24:39.831 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.602578+0000", 2024-09-11T08:24:39.831 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00075117300000000005 2024-09-11T08:24:39.831 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.831 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.831 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:24:39.831 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.602640+0000", 2024-09-11T08:24:39.831 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.1645999999999999e-05 2024-09-11T08:24:39.832 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.832 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.832 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:24:39.832 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.603182+0000", 2024-09-11T08:24:39.832 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00054204600000000002 2024-09-11T08:24:39.832 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.832 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.832 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:24:39.832 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.603226+0000", 2024-09-11T08:24:39.832 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.4459000000000002e-05 2024-09-11T08:24:39.832 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.832 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.833 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:24:39.833 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.603366+0000", 2024-09-11T08:24:39.833 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00014007000000000001 2024-09-11T08:24:39.833 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:39.833 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:24:39.833 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:39.833 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.833 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.833 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15087.0:146971 8.c 8:3ef36992:::benchmark_data_smithi150_107646_object18139:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-11T08:24:39.833 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:15:55.585289+0000", 2024-09-11T08:24:39.833 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 524.20629058300005, 2024-09-11T08:24:39.833 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.0288565780000001, 2024-09-11T08:24:39.833 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:24:39.834 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:24:39.834 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:24:39.834 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15087", 2024-09-11T08:24:39.834 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/3299560913", 2024-09-11T08:24:39.834 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 146971 2024-09-11T08:24:39.834 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.834 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:24:39.834 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.834 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:24:39.834 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.585289+0000", 2024-09-11T08:24:39.834 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:24:39.835 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.835 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.835 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:24:39.835 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.585289+0000", 2024-09-11T08:24:39.835 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:24:39.835 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.835 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.835 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:24:39.835 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.585287+0000", 2024-09-11T08:24:39.835 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:24:39.835 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.835 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.835 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:24:39.835 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.585298+0000", 2024-09-11T08:24:39.836 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1035e-05 2024-09-11T08:24:39.836 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.836 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.836 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:24:39.836 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.585321+0000", 2024-09-11T08:24:39.836 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.3589999999999999e-05 2024-09-11T08:24:39.836 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.836 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.836 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:24:39.836 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.585330+0000", 2024-09-11T08:24:39.836 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.6580000000000008e-06 2024-09-11T08:24:39.836 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.836 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.837 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.837 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.585399+0000", 2024-09-11T08:24:39.837 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.9215999999999996e-05 2024-09-11T08:24:39.837 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.837 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.837 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:24:39.837 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.585410+0000", 2024-09-11T08:24:39.837 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0772e-05 2024-09-11T08:24:39.837 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.838 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.838 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.838 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:56.507378+0000", 2024-09-11T08:24:39.838 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.92196784700000001 2024-09-11T08:24:39.838 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.838 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.838 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:24:39.838 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:56.507384+0000", 2024-09-11T08:24:39.838 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.2169999999999999e-06 2024-09-11T08:24:39.838 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.838 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.838 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.838 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.594588+0000", 2024-09-11T08:24:39.838 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.087204286 2024-09-11T08:24:39.839 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.839 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.839 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:24:39.839 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.594691+0000", 2024-09-11T08:24:39.839 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000102296 2024-09-11T08:24:39.839 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.839 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.839 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 3,5,7", 2024-09-11T08:24:39.839 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.594764+0000", 2024-09-11T08:24:39.839 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.3912999999999995e-05 2024-09-11T08:24:39.839 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.839 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.840 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:24:39.840 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.601834+0000", 2024-09-11T08:24:39.840 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0070695640000000004 2024-09-11T08:24:39.840 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.840 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.840 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:24:39.840 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.602836+0000", 2024-09-11T08:24:39.840 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0010021139999999999 2024-09-11T08:24:39.840 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.840 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.840 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:24:39.841 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.603431+0000", 2024-09-11T08:24:39.841 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000594832 2024-09-11T08:24:39.841 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.841 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.841 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:24:39.841 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.613998+0000", 2024-09-11T08:24:39.841 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.010566923000000001 2024-09-11T08:24:39.841 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.841 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.841 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:24:39.841 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.614043+0000", 2024-09-11T08:24:39.841 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.4641999999999997e-05 2024-09-11T08:24:39.841 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.841 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.842 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:24:39.842 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.614145+0000", 2024-09-11T08:24:39.842 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00010281300000000001 2024-09-11T08:24:39.842 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:39.842 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:24:39.842 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:39.842 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.842 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.842 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15087.0:147002 8.c 8:3977710b:::benchmark_data_smithi150_107646_object18170:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-11T08:24:39.842 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:15:55.642339+0000", 2024-09-11T08:24:39.842 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 524.14924053799996, 2024-09-11T08:24:39.842 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.9719673499999999, 2024-09-11T08:24:39.842 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:24:39.842 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:24:39.843 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:39.844 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:24:39.844 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15087", 2024-09-11T08:24:39.844 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/3299560913", 2024-09-11T08:24:39.844 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 147002 2024-09-11T08:24:39.844 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.844 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:24:39.844 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.844 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:24:39.844 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.642339+0000", 2024-09-11T08:24:39.844 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:24:39.845 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.845 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.845 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:24:39.845 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.642339+0000", 2024-09-11T08:24:39.845 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:24:39.845 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.845 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.845 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:24:39.845 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.642338+0000", 2024-09-11T08:24:39.845 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:24:39.845 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.845 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.845 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:24:39.846 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.642344+0000", 2024-09-11T08:24:39.846 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.3160000000000002e-06 2024-09-11T08:24:39.846 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.846 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.846 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:24:39.846 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.642354+0000", 2024-09-11T08:24:39.846 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0587e-05 2024-09-11T08:24:39.846 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.846 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.846 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:24:39.846 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.642360+0000", 2024-09-11T08:24:39.846 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.7660000000000004e-06 2024-09-11T08:24:39.846 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.847 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.847 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.847 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.642413+0000", 2024-09-11T08:24:39.847 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.2397e-05 2024-09-11T08:24:39.847 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.847 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.847 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:24:39.847 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.642422+0000", 2024-09-11T08:24:39.847 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.2320000000000007e-06 2024-09-11T08:24:39.847 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.848 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.848 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.848 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:56.507404+0000", 2024-09-11T08:24:39.848 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.86498259899999996 2024-09-11T08:24:39.848 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.848 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.848 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:24:39.848 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:56.507410+0000", 2024-09-11T08:24:39.848 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.8610000000000003e-06 2024-09-11T08:24:39.848 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.848 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.848 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.849 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.596147+0000", 2024-09-11T08:24:39.849 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0887365870000001 2024-09-11T08:24:39.849 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.849 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.849 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:24:39.849 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.596268+0000", 2024-09-11T08:24:39.849 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012079299999999999 2024-09-11T08:24:39.849 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.849 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.849 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 3,5,7", 2024-09-11T08:24:39.849 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.596390+0000", 2024-09-11T08:24:39.849 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012239799999999999 2024-09-11T08:24:39.849 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.849 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.850 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:24:39.850 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.601961+0000", 2024-09-11T08:24:39.850 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0055706610000000002 2024-09-11T08:24:39.850 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.850 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.850 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:24:39.850 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.602876+0000", 2024-09-11T08:24:39.850 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00091555499999999997 2024-09-11T08:24:39.850 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.850 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.850 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:24:39.851 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.606935+0000", 2024-09-11T08:24:39.851 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0040587760000000001 2024-09-11T08:24:39.851 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.851 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.851 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:24:39.851 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.614180+0000", 2024-09-11T08:24:39.851 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0072450559999999997 2024-09-11T08:24:39.851 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.851 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.851 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:24:39.851 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.614215+0000", 2024-09-11T08:24:39.851 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.5114999999999998e-05 2024-09-11T08:24:39.851 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.851 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.852 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:24:39.852 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.614306+0000", 2024-09-11T08:24:39.852 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.0927999999999994e-05 2024-09-11T08:24:39.852 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:39.852 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:24:39.852 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:39.852 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.852 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.852 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15087.0:147025 8.c 8:3a2660c9:::benchmark_data_smithi150_107646_object18193:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-11T08:24:39.853 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:15:55.675112+0000", 2024-09-11T08:24:39.853 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 524.11646720399995, 2024-09-11T08:24:39.853 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.9393519669999999, 2024-09-11T08:24:39.853 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:24:39.853 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:24:39.853 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:24:39.853 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15087", 2024-09-11T08:24:39.853 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/3299560913", 2024-09-11T08:24:39.853 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 147025 2024-09-11T08:24:39.854 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.854 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:24:39.854 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.854 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:24:39.854 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.675112+0000", 2024-09-11T08:24:39.854 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:24:39.854 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.854 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.854 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:24:39.854 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.675112+0000", 2024-09-11T08:24:39.854 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:24:39.854 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.854 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.855 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:24:39.855 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.675111+0000", 2024-09-11T08:24:39.855 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:24:39.855 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.855 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.855 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:24:39.855 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.675117+0000", 2024-09-11T08:24:39.855 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.1299999999999998e-06 2024-09-11T08:24:39.855 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.855 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.855 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:24:39.855 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.675130+0000", 2024-09-11T08:24:39.855 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3278000000000001e-05 2024-09-11T08:24:39.856 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.856 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.856 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:24:39.856 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.675137+0000", 2024-09-11T08:24:39.856 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.404e-06 2024-09-11T08:24:39.856 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.856 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.856 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.856 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.675191+0000", 2024-09-11T08:24:39.856 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.3724999999999998e-05 2024-09-11T08:24:39.856 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.856 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.856 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:24:39.856 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.675198+0000", 2024-09-11T08:24:39.857 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.024e-06 2024-09-11T08:24:39.857 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.857 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.857 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.857 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:56.507428+0000", 2024-09-11T08:24:39.857 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.83223069599999999 2024-09-11T08:24:39.857 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.857 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.857 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:24:39.857 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:56.507434+0000", 2024-09-11T08:24:39.858 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.1929999999999998e-06 2024-09-11T08:24:39.858 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.858 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.858 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.858 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.598273+0000", 2024-09-11T08:24:39.858 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.090838132 2024-09-11T08:24:39.858 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.858 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.858 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:24:39.859 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.598359+0000", 2024-09-11T08:24:39.859 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.6082999999999996e-05 2024-09-11T08:24:39.859 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.859 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.859 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 3,5,7", 2024-09-11T08:24:39.859 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.598425+0000", 2024-09-11T08:24:39.859 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.5868e-05 2024-09-11T08:24:39.859 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.859 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.860 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:24:39.860 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.601965+0000", 2024-09-11T08:24:39.860 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0035409030000000002 2024-09-11T08:24:39.860 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.860 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.860 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:24:39.860 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.603462+0000", 2024-09-11T08:24:39.860 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0014965029999999999 2024-09-11T08:24:39.860 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.861 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.861 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:24:39.861 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.606974+0000", 2024-09-11T08:24:39.861 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0035119209999999999 2024-09-11T08:24:39.861 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.861 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.861 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:24:39.861 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.614338+0000", 2024-09-11T08:24:39.861 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0073643609999999998 2024-09-11T08:24:39.861 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.861 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.861 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:24:39.861 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.614373+0000", 2024-09-11T08:24:39.861 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.4489000000000001e-05 2024-09-11T08:24:39.862 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.862 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.862 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:24:39.862 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.614464+0000", 2024-09-11T08:24:39.862 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.1457000000000002e-05 2024-09-11T08:24:39.862 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:39.862 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:24:39.862 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:39.862 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.863 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.863 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15087.0:147033 8.c 8:3ab6f78b:::benchmark_data_smithi150_107646_object18201:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-11T08:24:39.863 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:15:55.700638+0000", 2024-09-11T08:24:39.863 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 524.09094125499996, 2024-09-11T08:24:39.863 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.9140247399999999, 2024-09-11T08:24:39.863 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:24:39.863 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:24:39.863 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:24:39.863 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15087", 2024-09-11T08:24:39.863 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/3299560913", 2024-09-11T08:24:39.864 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 147033 2024-09-11T08:24:39.864 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.864 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:24:39.864 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.864 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:24:39.864 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.700638+0000", 2024-09-11T08:24:39.864 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:24:39.864 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.864 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.864 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:24:39.864 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.700638+0000", 2024-09-11T08:24:39.864 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:24:39.865 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.865 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.865 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:24:39.865 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.700637+0000", 2024-09-11T08:24:39.865 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:24:39.865 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.865 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.865 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:24:39.865 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.700643+0000", 2024-09-11T08:24:39.865 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.0909999999999998e-06 2024-09-11T08:24:39.865 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.865 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.866 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:24:39.866 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.700654+0000", 2024-09-11T08:24:39.866 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0669999999999999e-05 2024-09-11T08:24:39.866 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.866 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.866 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:24:39.866 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.700659+0000", 2024-09-11T08:24:39.866 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.6720000000000001e-06 2024-09-11T08:24:39.866 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.866 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.866 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.866 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.700717+0000", 2024-09-11T08:24:39.866 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.7318000000000002e-05 2024-09-11T08:24:39.867 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.867 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.867 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:24:39.867 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:55.700724+0000", 2024-09-11T08:24:39.867 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.1840000000000002e-06 2024-09-11T08:24:39.867 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.867 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.867 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.867 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:56.508060+0000", 2024-09-11T08:24:39.867 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.80733569999999999 2024-09-11T08:24:39.867 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.867 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.867 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:24:39.867 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:56.508090+0000", 2024-09-11T08:24:39.868 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.0278999999999999e-05 2024-09-11T08:24:39.868 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.868 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.868 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.868 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.599837+0000", 2024-09-11T08:24:39.868 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0917466650000001 2024-09-11T08:24:39.868 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.868 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.868 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:24:39.869 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.599953+0000", 2024-09-11T08:24:39.869 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000116809 2024-09-11T08:24:39.869 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.869 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.869 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 3,5,7", 2024-09-11T08:24:39.869 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.600078+0000", 2024-09-11T08:24:39.869 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000124146 2024-09-11T08:24:39.869 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.869 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.869 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:24:39.870 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.602178+0000", 2024-09-11T08:24:39.870 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0021000960000000001 2024-09-11T08:24:39.870 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.870 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.870 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:24:39.870 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.603497+0000", 2024-09-11T08:24:39.870 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.001319104 2024-09-11T08:24:39.870 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.870 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.870 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:24:39.870 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.613960+0000", 2024-09-11T08:24:39.871 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.010463586 2024-09-11T08:24:39.871 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.871 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.871 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:24:39.871 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.614523+0000", 2024-09-11T08:24:39.871 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00056302200000000004 2024-09-11T08:24:39.871 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.871 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.871 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:24:39.871 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.614564+0000", 2024-09-11T08:24:39.871 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.0565999999999998e-05 2024-09-11T08:24:39.871 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.871 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.871 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:24:39.872 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:15:57.614663+0000", 2024-09-11T08:24:39.872 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.8929000000000001e-05 2024-09-11T08:24:39.872 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:39.872 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:24:39.872 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:39.872 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.872 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.872 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15087.0:189219 8.c 8:32ae3dd6:::benchmark_data_smithi150_107646_object60387:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-09-11T08:24:39.872 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:16:34.143887+0000", 2024-09-11T08:24:39.872 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 485.64769221900002, 2024-09-11T08:24:39.872 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.048841806, 2024-09-11T08:24:39.872 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:24:39.872 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-09-11T08:24:39.873 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:24:39.873 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15087", 2024-09-11T08:24:39.873 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/3299560913", 2024-09-11T08:24:39.873 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 189219 2024-09-11T08:24:39.873 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.873 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:24:39.873 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.873 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:24:39.873 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:16:34.143887+0000", 2024-09-11T08:24:39.873 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:24:39.873 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.873 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.873 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:24:39.874 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:16:34.143887+0000", 2024-09-11T08:24:39.874 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:24:39.874 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.874 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.874 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:24:39.874 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:16:34.143886+0000", 2024-09-11T08:24:39.874 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:24:39.874 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.874 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.874 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:24:39.874 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:16:34.143891+0000", 2024-09-11T08:24:39.874 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.6550000000000001e-06 2024-09-11T08:24:39.874 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.874 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.875 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:24:39.875 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:16:34.143908+0000", 2024-09-11T08:24:39.875 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.6980999999999999e-05 2024-09-11T08:24:39.875 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.875 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.875 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:24:39.875 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:16:34.143915+0000", 2024-09-11T08:24:39.875 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.4280000000000001e-06 2024-09-11T08:24:39.875 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.875 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.875 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.875 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:16:34.144807+0000", 2024-09-11T08:24:39.875 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000892021 2024-09-11T08:24:39.875 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.876 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.876 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:24:39.876 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:16:34.144939+0000", 2024-09-11T08:24:39.876 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00013193499999999999 2024-09-11T08:24:39.876 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.876 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.876 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 3,5,7", 2024-09-11T08:24:39.876 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:16:34.145066+0000", 2024-09-11T08:24:39.876 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000127355 2024-09-11T08:24:39.876 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.876 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.876 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:24:39.876 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:16:34.149063+0000", 2024-09-11T08:24:39.877 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.003996515 2024-09-11T08:24:39.877 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.877 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.877 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:24:39.877 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:16:34.149629+0000", 2024-09-11T08:24:39.877 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00056642800000000001 2024-09-11T08:24:39.877 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.877 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.877 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.877 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:16:34.156053+0000", 2024-09-11T08:24:39.877 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.006423748 2024-09-11T08:24:39.877 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.877 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.878 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:24:39.878 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:16:34.156062+0000", 2024-09-11T08:24:39.878 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.7169999999999997e-06 2024-09-11T08:24:39.878 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.878 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.878 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.878 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:16:35.192682+0000", 2024-09-11T08:24:39.878 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0366201349999999 2024-09-11T08:24:39.878 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.878 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.878 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:24:39.878 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:16:35.192729+0000", 2024-09-11T08:24:39.878 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.7289000000000001e-05 2024-09-11T08:24:39.879 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:39.879 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:24:39.879 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:39.879 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.879 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.879 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15087.0:189229 8.c 8:3e7fc498:::benchmark_data_smithi150_107646_object60397:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-09-11T08:24:39.879 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:16:34.149747+0000", 2024-09-11T08:24:39.879 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 485.641831954, 2024-09-11T08:24:39.879 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0694402629999999, 2024-09-11T08:24:39.879 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:24:39.879 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:24:39.879 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:24:39.879 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15087", 2024-09-11T08:24:39.880 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/3299560913", 2024-09-11T08:24:39.880 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 189229 2024-09-11T08:24:39.880 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.880 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:24:39.880 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.880 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:24:39.880 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:16:34.149747+0000", 2024-09-11T08:24:39.880 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:24:39.881 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.881 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.881 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:24:39.881 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:16:34.149747+0000", 2024-09-11T08:24:39.881 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:24:39.881 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.881 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.881 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:24:39.881 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:16:34.149747+0000", 2024-09-11T08:24:39.882 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:24:39.882 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.882 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.882 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:24:39.882 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:16:34.149752+0000", 2024-09-11T08:24:39.882 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.1499999999999998e-06 2024-09-11T08:24:39.882 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.882 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.882 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:24:39.882 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:16:34.149762+0000", 2024-09-11T08:24:39.882 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.9920000000000002e-06 2024-09-11T08:24:39.882 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.882 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.883 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:24:39.883 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:16:34.149768+0000", 2024-09-11T08:24:39.883 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.7219999999999996e-06 2024-09-11T08:24:39.883 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.883 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.883 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.883 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:16:34.149819+0000", 2024-09-11T08:24:39.883 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.1335000000000002e-05 2024-09-11T08:24:39.883 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.883 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.883 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op must wait for map", 2024-09-11T08:24:39.883 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:16:34.149825+0000", 2024-09-11T08:24:39.884 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.0789999999999997e-06 2024-09-11T08:24:39.884 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.884 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.884 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.884 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:16:34.156079+0000", 2024-09-11T08:24:39.884 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0062544949999999997 2024-09-11T08:24:39.884 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.884 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.884 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:24:39.884 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:16:34.156085+0000", 2024-09-11T08:24:39.884 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.4040000000000003e-06 2024-09-11T08:24:39.884 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.884 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.885 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.885 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:16:35.192745+0000", 2024-09-11T08:24:39.885 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0366596939999999 2024-09-11T08:24:39.885 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.885 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.885 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:24:39.885 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:16:35.192825+0000", 2024-09-11T08:24:39.885 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.0663999999999999e-05 2024-09-11T08:24:39.885 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.885 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.885 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 3,7", 2024-09-11T08:24:39.885 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:16:35.192905+0000", 2024-09-11T08:24:39.885 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.9682e-05 2024-09-11T08:24:39.886 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.886 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.886 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:24:39.886 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:16:35.201494+0000", 2024-09-11T08:24:39.886 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0085893380000000002 2024-09-11T08:24:39.886 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.886 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.886 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:24:39.886 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:16:35.219003+0000", 2024-09-11T08:24:39.886 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.017509192 2024-09-11T08:24:39.886 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.887 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.887 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:24:39.887 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:16:35.219101+0000", 2024-09-11T08:24:39.887 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.7909000000000001e-05 2024-09-11T08:24:39.887 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.887 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.887 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:24:39.887 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:16:35.219149+0000", 2024-09-11T08:24:39.887 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.7153000000000001e-05 2024-09-11T08:24:39.887 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.887 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.887 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:24:39.887 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:16:35.219188+0000", 2024-09-11T08:24:39.887 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.9165000000000002e-05 2024-09-11T08:24:39.887 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:39.888 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:24:39.888 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:39.888 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.888 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.888 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15369.0:96081 9.9 9:9ef22e50:::benchmark_data_smithi150_133616_object96080:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e259)", 2024-09-11T08:24:39.888 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:19:08.570968+0000", 2024-09-11T08:24:39.888 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 331.22061138599997, 2024-09-11T08:24:39.888 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.2956648120000001, 2024-09-11T08:24:39.888 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:24:39.888 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-09-11T08:24:39.888 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:24:39.888 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15369", 2024-09-11T08:24:39.888 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:24:39.889 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 96081 2024-09-11T08:24:39.889 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.889 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:24:39.889 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.889 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:24:39.889 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.570968+0000", 2024-09-11T08:24:39.889 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:24:39.889 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.889 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.889 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:24:39.889 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.570968+0000", 2024-09-11T08:24:39.889 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:24:39.889 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.890 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.890 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:24:39.890 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.570967+0000", 2024-09-11T08:24:39.890 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:24:39.890 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.890 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.890 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:24:39.890 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.570992+0000", 2024-09-11T08:24:39.890 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.4756000000000002e-05 2024-09-11T08:24:39.890 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.890 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.890 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:24:39.890 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.571003+0000", 2024-09-11T08:24:39.890 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1029999999999999e-05 2024-09-11T08:24:39.890 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.891 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.891 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:24:39.891 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.571009+0000", 2024-09-11T08:24:39.891 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.8289999999999996e-06 2024-09-11T08:24:39.891 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.891 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.891 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.891 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.575404+0000", 2024-09-11T08:24:39.893 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0043950339999999999 2024-09-11T08:24:39.893 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.893 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.893 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:24:39.893 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.575531+0000", 2024-09-11T08:24:39.893 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000127515 2024-09-11T08:24:39.893 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.893 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.893 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,7", 2024-09-11T08:24:39.894 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.575923+0000", 2024-09-11T08:24:39.894 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00039200599999999997 2024-09-11T08:24:39.894 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.894 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.894 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:24:39.894 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.589837+0000", 2024-09-11T08:24:39.894 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.013914015 2024-09-11T08:24:39.894 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.894 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.894 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.894 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.613186+0000", 2024-09-11T08:24:39.894 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.023348784000000001 2024-09-11T08:24:39.894 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.895 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.895 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:24:39.895 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.613196+0000", 2024-09-11T08:24:39.895 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0465999999999999e-05 2024-09-11T08:24:39.895 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.895 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.895 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.895 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.602630+0000", 2024-09-11T08:24:39.895 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.98943398299999996 2024-09-11T08:24:39.895 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.895 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.895 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:24:39.896 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.602639+0000", 2024-09-11T08:24:39.896 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.6649999999999992e-06 2024-09-11T08:24:39.896 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.896 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.896 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.896 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.864599+0000", 2024-09-11T08:24:39.896 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2619598349999999 2024-09-11T08:24:39.896 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.897 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.897 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:24:39.897 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.866633+0000", 2024-09-11T08:24:39.897 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0020339830000000001 2024-09-11T08:24:39.897 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:39.897 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:24:39.897 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:39.897 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.897 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.897 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15369.0:96088 9.9 9:90c5c778:::benchmark_data_smithi150_133616_object96087:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e259)", 2024-09-11T08:24:39.897 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:19:08.573763+0000", 2024-09-11T08:24:39.897 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 331.21781632900002, 2024-09-11T08:24:39.897 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.291237916, 2024-09-11T08:24:39.898 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:24:39.898 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-09-11T08:24:39.898 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:24:39.898 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15369", 2024-09-11T08:24:39.898 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:24:39.898 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 96088 2024-09-11T08:24:39.898 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.899 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:24:39.899 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.899 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:24:39.899 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.573763+0000", 2024-09-11T08:24:39.899 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:24:39.899 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.899 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.899 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:24:39.900 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.573763+0000", 2024-09-11T08:24:39.900 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:24:39.900 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.900 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.900 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:24:39.900 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.573762+0000", 2024-09-11T08:24:39.900 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:24:39.900 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.900 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.900 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:24:39.900 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.573793+0000", 2024-09-11T08:24:39.901 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.1029999999999999e-05 2024-09-11T08:24:39.901 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.901 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.901 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:24:39.901 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.573808+0000", 2024-09-11T08:24:39.901 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.5073e-05 2024-09-11T08:24:39.901 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.901 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.901 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:24:39.901 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.573814+0000", 2024-09-11T08:24:39.901 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.8359999999999997e-06 2024-09-11T08:24:39.901 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.901 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.901 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.902 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.580054+0000", 2024-09-11T08:24:39.902 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0062404540000000003 2024-09-11T08:24:39.902 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.902 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.902 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:24:39.902 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.580170+0000", 2024-09-11T08:24:39.902 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000115812 2024-09-11T08:24:39.902 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.903 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.903 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,7", 2024-09-11T08:24:39.903 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.580298+0000", 2024-09-11T08:24:39.903 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000128231 2024-09-11T08:24:39.903 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.903 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.903 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:24:39.903 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.589864+0000", 2024-09-11T08:24:39.903 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0095654590000000001 2024-09-11T08:24:39.903 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.903 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.903 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.903 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.613228+0000", 2024-09-11T08:24:39.903 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.023364772999999998 2024-09-11T08:24:39.904 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.904 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.904 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:24:39.904 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.613272+0000", 2024-09-11T08:24:39.904 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.3186999999999998e-05 2024-09-11T08:24:39.904 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.904 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.904 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.904 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.602667+0000", 2024-09-11T08:24:39.904 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.98939526499999997 2024-09-11T08:24:39.904 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.904 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.904 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:24:39.904 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.602673+0000", 2024-09-11T08:24:39.905 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.5220000000000004e-06 2024-09-11T08:24:39.905 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.905 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.905 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.905 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.864929+0000", 2024-09-11T08:24:39.905 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2622552309999999 2024-09-11T08:24:39.905 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.905 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.905 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:24:39.905 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.865001+0000", 2024-09-11T08:24:39.905 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.2358000000000001e-05 2024-09-11T08:24:39.905 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:39.905 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:24:39.906 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:39.906 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.906 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.906 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15369.0:96090 9.9 9:937fd0b9:::benchmark_data_smithi150_133616_object96089:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e259)", 2024-09-11T08:24:39.906 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:19:08.576879+0000", 2024-09-11T08:24:39.906 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 331.21470013700002, 2024-09-11T08:24:39.906 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.2882108680000002, 2024-09-11T08:24:39.906 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:24:39.906 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-09-11T08:24:39.906 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:24:39.906 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15369", 2024-09-11T08:24:39.906 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:24:39.906 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 96090 2024-09-11T08:24:39.907 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.907 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:24:39.907 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.907 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:24:39.907 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.576879+0000", 2024-09-11T08:24:39.907 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:24:39.907 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.907 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.907 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:24:39.907 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.576879+0000", 2024-09-11T08:24:39.907 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:24:39.907 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.908 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.908 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:24:39.908 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.576878+0000", 2024-09-11T08:24:39.908 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:24:39.908 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.908 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.908 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:24:39.908 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.576908+0000", 2024-09-11T08:24:39.908 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.0020999999999999e-05 2024-09-11T08:24:39.908 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.909 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.909 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:24:39.909 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.576923+0000", 2024-09-11T08:24:39.909 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4887e-05 2024-09-11T08:24:39.909 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.909 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.909 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:24:39.909 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.576929+0000", 2024-09-11T08:24:39.909 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.2720000000000003e-06 2024-09-11T08:24:39.909 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.909 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.909 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.909 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.583535+0000", 2024-09-11T08:24:39.910 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0066064549999999998 2024-09-11T08:24:39.910 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.910 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.910 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:24:39.910 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.583659+0000", 2024-09-11T08:24:39.910 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012306700000000001 2024-09-11T08:24:39.910 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.910 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.910 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,7", 2024-09-11T08:24:39.910 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.583809+0000", 2024-09-11T08:24:39.910 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00015080900000000001 2024-09-11T08:24:39.910 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.910 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.910 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:24:39.911 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.592518+0000", 2024-09-11T08:24:39.911 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0087083590000000006 2024-09-11T08:24:39.911 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.911 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.911 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.911 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.613355+0000", 2024-09-11T08:24:39.911 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.020837073000000001 2024-09-11T08:24:39.911 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.911 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.911 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:24:39.911 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.613361+0000", 2024-09-11T08:24:39.911 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.2400000000000004e-06 2024-09-11T08:24:39.912 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.912 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.912 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.912 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.602699+0000", 2024-09-11T08:24:39.912 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.98933749599999998 2024-09-11T08:24:39.912 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.912 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.912 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:24:39.912 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.602705+0000", 2024-09-11T08:24:39.912 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.8260000000000004e-06 2024-09-11T08:24:39.913 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.913 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.913 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.913 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.865029+0000", 2024-09-11T08:24:39.913 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2623238969999999 2024-09-11T08:24:39.913 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.914 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.914 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:24:39.914 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.865090+0000", 2024-09-11T08:24:39.914 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.0872999999999998e-05 2024-09-11T08:24:39.914 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:39.914 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:24:39.914 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:39.914 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.914 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.914 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15369.0:96105 9.9 9:97afa5c7:::benchmark_data_smithi150_133616_object96104:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-11T08:24:39.914 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:19:08.610376+0000", 2024-09-11T08:24:39.914 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 331.18120302699998, 2024-09-11T08:24:39.915 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.3038915310000001, 2024-09-11T08:24:39.915 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:24:39.915 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:24:39.915 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:24:39.915 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15369", 2024-09-11T08:24:39.915 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:24:39.915 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 96105 2024-09-11T08:24:39.915 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.915 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:24:39.915 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.915 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:24:39.915 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.610376+0000", 2024-09-11T08:24:39.915 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:24:39.916 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.916 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.916 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:24:39.916 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.610376+0000", 2024-09-11T08:24:39.916 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:24:39.916 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.916 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.916 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:24:39.916 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.610375+0000", 2024-09-11T08:24:39.917 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:24:39.917 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.917 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.917 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:24:39.917 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.610401+0000", 2024-09-11T08:24:39.917 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.6548000000000001e-05 2024-09-11T08:24:39.917 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.917 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.917 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:24:39.917 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.610410+0000", 2024-09-11T08:24:39.917 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.2640000000000008e-06 2024-09-11T08:24:39.918 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.918 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.918 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:24:39.918 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.610416+0000", 2024-09-11T08:24:39.918 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.2770000000000002e-06 2024-09-11T08:24:39.918 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.918 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.918 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.918 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.614510+0000", 2024-09-11T08:24:39.918 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0040939319999999998 2024-09-11T08:24:39.918 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.919 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.919 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:24:39.919 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.614517+0000", 2024-09-11T08:24:39.919 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.2400000000000001e-06 2024-09-11T08:24:39.919 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.919 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.919 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.919 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.602724+0000", 2024-09-11T08:24:39.919 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.98820673699999995 2024-09-11T08:24:39.919 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.919 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.919 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:24:39.919 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.602731+0000", 2024-09-11T08:24:39.919 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.6880000000000002e-06 2024-09-11T08:24:39.920 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.920 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.920 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.920 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.865109+0000", 2024-09-11T08:24:39.920 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2623779960000001 2024-09-11T08:24:39.920 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.920 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.920 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:24:39.920 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.865225+0000", 2024-09-11T08:24:39.920 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000115911 2024-09-11T08:24:39.920 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.920 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.920 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 3,7", 2024-09-11T08:24:39.920 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.865767+0000", 2024-09-11T08:24:39.921 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00054236499999999995 2024-09-11T08:24:39.921 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.921 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.921 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:24:39.921 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.885412+0000", 2024-09-11T08:24:39.921 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.019645187000000001 2024-09-11T08:24:39.921 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.921 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.921 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:24:39.921 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.890697+0000", 2024-09-11T08:24:39.921 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0052851699999999996 2024-09-11T08:24:39.921 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.921 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.921 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:24:39.922 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.909083+0000", 2024-09-11T08:24:39.922 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.018386070000000001 2024-09-11T08:24:39.922 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.922 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.922 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:24:39.922 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.909169+0000", 2024-09-11T08:24:39.922 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.5953999999999997e-05 2024-09-11T08:24:39.922 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.922 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.922 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:24:39.922 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.914268+0000", 2024-09-11T08:24:39.923 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0050986199999999999 2024-09-11T08:24:39.923 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:39.923 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:24:39.923 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:39.923 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.923 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.923 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15369.0:96107 9.9 9:9e673b22:::benchmark_data_smithi150_133616_object96106:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-11T08:24:39.923 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:19:08.610424+0000", 2024-09-11T08:24:39.923 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 331.181155606, 2024-09-11T08:24:39.923 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.3087517069999999, 2024-09-11T08:24:39.923 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:24:39.924 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:24:39.924 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:24:39.924 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15369", 2024-09-11T08:24:39.924 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:24:39.924 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 96107 2024-09-11T08:24:39.924 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.924 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:24:39.924 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.924 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:24:39.925 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.610424+0000", 2024-09-11T08:24:39.925 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:24:39.925 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.925 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.925 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:24:39.925 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.610424+0000", 2024-09-11T08:24:39.925 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:24:39.925 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.925 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.925 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:24:39.925 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.610423+0000", 2024-09-11T08:24:39.925 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:24:39.925 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.926 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.926 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:24:39.926 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.610445+0000", 2024-09-11T08:24:39.926 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.1322e-05 2024-09-11T08:24:39.926 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.926 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.926 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:24:39.926 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.610450+0000", 2024-09-11T08:24:39.926 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.3789999999999997e-06 2024-09-11T08:24:39.926 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.926 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.926 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:24:39.926 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.610454+0000", 2024-09-11T08:24:39.926 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.0090000000000001e-06 2024-09-11T08:24:39.927 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.927 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.927 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.927 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.614538+0000", 2024-09-11T08:24:39.927 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0040842200000000004 2024-09-11T08:24:39.927 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.927 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.927 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:24:39.927 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.614544+0000", 2024-09-11T08:24:39.927 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.1009999999999997e-06 2024-09-11T08:24:39.927 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.927 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.927 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.927 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.602748+0000", 2024-09-11T08:24:39.928 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.98820334399999998 2024-09-11T08:24:39.928 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.928 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.928 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:24:39.928 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.602754+0000", 2024-09-11T08:24:39.928 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.0159999999999997e-06 2024-09-11T08:24:39.928 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.928 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.928 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.928 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.873544+0000", 2024-09-11T08:24:39.928 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2707898580000001 2024-09-11T08:24:39.928 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.928 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.929 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:24:39.929 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.873670+0000", 2024-09-11T08:24:39.929 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000126766 2024-09-11T08:24:39.929 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.929 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.929 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 3,7", 2024-09-11T08:24:39.929 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.873839+0000", 2024-09-11T08:24:39.929 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000169025 2024-09-11T08:24:39.929 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.929 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.929 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:24:39.929 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.885452+0000", 2024-09-11T08:24:39.929 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.011612447999999999 2024-09-11T08:24:39.930 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.930 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.930 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:24:39.930 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.890704+0000", 2024-09-11T08:24:39.930 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0052519020000000001 2024-09-11T08:24:39.930 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.930 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.930 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:24:39.930 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.914526+0000", 2024-09-11T08:24:39.930 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.023822471000000001 2024-09-11T08:24:39.930 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.930 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.930 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:24:39.930 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.914585+0000", 2024-09-11T08:24:39.931 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.9111999999999999e-05 2024-09-11T08:24:39.931 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.931 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.931 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:24:39.931 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.919175+0000", 2024-09-11T08:24:39.931 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0045901170000000003 2024-09-11T08:24:39.931 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:39.931 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:24:39.931 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:39.931 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.931 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.931 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15369.0:96108 9.b 9:db475b9c:::benchmark_data_smithi150_133616_object96107:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-11T08:24:39.931 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:19:08.610458+0000", 2024-09-11T08:24:39.931 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 331.18112126800003, 2024-09-11T08:24:39.932 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.3092126090000002, 2024-09-11T08:24:39.932 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:24:39.932 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:24:39.932 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:24:39.932 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15369", 2024-09-11T08:24:39.932 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:24:39.932 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 96108 2024-09-11T08:24:39.932 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.932 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:24:39.932 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.932 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:24:39.932 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.610458+0000", 2024-09-11T08:24:39.933 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:24:39.933 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.933 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.933 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:24:39.933 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.610458+0000", 2024-09-11T08:24:39.933 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:24:39.933 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.933 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.933 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:24:39.933 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.610458+0000", 2024-09-11T08:24:39.933 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:24:39.933 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.933 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.933 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:24:39.934 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.610480+0000", 2024-09-11T08:24:39.934 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.1861999999999998e-05 2024-09-11T08:24:39.934 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.934 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.934 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:24:39.934 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.610485+0000", 2024-09-11T08:24:39.934 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.0830000000000003e-06 2024-09-11T08:24:39.934 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.934 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.935 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:24:39.935 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.610488+0000", 2024-09-11T08:24:39.935 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.619e-06 2024-09-11T08:24:39.935 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.935 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.935 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.935 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.616830+0000", 2024-09-11T08:24:39.935 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0063418579999999997 2024-09-11T08:24:39.935 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.936 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.936 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:24:39.936 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.616840+0000", 2024-09-11T08:24:39.936 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.611e-06 2024-09-11T08:24:39.936 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.936 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.936 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.936 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.607837+0000", 2024-09-11T08:24:39.936 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.99099718299999995 2024-09-11T08:24:39.936 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.936 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.936 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:24:39.936 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.607847+0000", 2024-09-11T08:24:39.936 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0501e-05 2024-09-11T08:24:39.937 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.937 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.937 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.937 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.873810+0000", 2024-09-11T08:24:39.937 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.265962279 2024-09-11T08:24:39.937 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.937 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.937 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:24:39.937 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.873928+0000", 2024-09-11T08:24:39.937 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00011868500000000001 2024-09-11T08:24:39.937 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.937 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.937 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 2,6", 2024-09-11T08:24:39.938 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.874457+0000", 2024-09-11T08:24:39.938 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000528851 2024-09-11T08:24:39.938 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.938 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.938 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:24:39.938 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.894530+0000", 2024-09-11T08:24:39.938 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.020072349 2024-09-11T08:24:39.938 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.938 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.938 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:24:39.938 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.898520+0000", 2024-09-11T08:24:39.938 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0039902920000000003 2024-09-11T08:24:39.938 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.938 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.938 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:24:39.939 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.914580+0000", 2024-09-11T08:24:39.939 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.016060540000000002 2024-09-11T08:24:39.939 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.939 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.939 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:24:39.939 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.914628+0000", 2024-09-11T08:24:39.939 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.7704000000000003e-05 2024-09-11T08:24:39.939 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.939 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.939 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:24:39.939 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.919671+0000", 2024-09-11T08:24:39.939 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0050425569999999996 2024-09-11T08:24:39.939 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:39.940 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:24:39.940 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:39.940 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.940 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.940 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15369.0:96111 9.b 9:d376a5e3:::benchmark_data_smithi150_133616_object96110:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-11T08:24:39.940 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:19:08.616227+0000", 2024-09-11T08:24:39.940 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 331.175352492, 2024-09-11T08:24:39.940 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.3142295430000002, 2024-09-11T08:24:39.941 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:24:39.941 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:24:39.941 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:24:39.941 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15369", 2024-09-11T08:24:39.941 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:24:39.941 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 96111 2024-09-11T08:24:39.941 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.941 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:24:39.941 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.941 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:24:39.941 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.616227+0000", 2024-09-11T08:24:39.941 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:24:39.942 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.942 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.942 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:24:39.942 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.616227+0000", 2024-09-11T08:24:39.942 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:24:39.942 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.942 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.942 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:24:39.942 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.616225+0000", 2024-09-11T08:24:39.942 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:24:39.942 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.942 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.942 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:24:39.942 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.616254+0000", 2024-09-11T08:24:39.943 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.8079e-05 2024-09-11T08:24:39.943 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.943 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.943 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:24:39.943 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.616262+0000", 2024-09-11T08:24:39.943 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.8550000000000008e-06 2024-09-11T08:24:39.943 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.943 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.943 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:24:39.943 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.616270+0000", 2024-09-11T08:24:39.943 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.413e-06 2024-09-11T08:24:39.943 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.943 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.943 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.943 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.616864+0000", 2024-09-11T08:24:39.944 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00059373800000000001 2024-09-11T08:24:39.944 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.944 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.944 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:24:39.944 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.616870+0000", 2024-09-11T08:24:39.944 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.2720000000000003e-06 2024-09-11T08:24:39.944 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.944 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.944 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.944 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.607897+0000", 2024-09-11T08:24:39.944 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.99102741699999997 2024-09-11T08:24:39.944 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.944 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.945 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:24:39.945 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.607905+0000", 2024-09-11T08:24:39.945 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.3309999999999998e-06 2024-09-11T08:24:39.945 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.945 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.945 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.945 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.884529+0000", 2024-09-11T08:24:39.945 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2766241059999999 2024-09-11T08:24:39.946 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.946 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.946 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:24:39.946 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.884831+0000", 2024-09-11T08:24:39.946 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00030185699999999999 2024-09-11T08:24:39.946 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.946 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.946 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 2,6", 2024-09-11T08:24:39.946 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.885022+0000", 2024-09-11T08:24:39.946 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000191184 2024-09-11T08:24:39.947 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.947 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.947 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:24:39.947 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.894572+0000", 2024-09-11T08:24:39.947 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0095500270000000009 2024-09-11T08:24:39.947 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.947 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.947 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:24:39.947 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.898529+0000", 2024-09-11T08:24:39.947 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0039576910000000002 2024-09-11T08:24:39.947 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.947 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.947 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:24:39.948 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.925242+0000", 2024-09-11T08:24:39.948 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.026713012000000001 2024-09-11T08:24:39.948 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.948 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.948 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:24:39.948 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.925297+0000", 2024-09-11T08:24:39.948 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.4160999999999999e-05 2024-09-11T08:24:39.948 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.948 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.948 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:24:39.948 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.930456+0000", 2024-09-11T08:24:39.948 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0051598659999999999 2024-09-11T08:24:39.948 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:39.949 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:24:39.949 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:39.949 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.949 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.949 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15369.0:96118 9.8 9:18676dc0:::benchmark_data_smithi150_133616_object96117:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-11T08:24:39.949 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:19:08.629161+0000", 2024-09-11T08:24:39.949 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 331.162418683, 2024-09-11T08:24:39.949 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.2526056940000001, 2024-09-11T08:24:39.949 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:24:39.949 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:24:39.949 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:24:39.949 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15369", 2024-09-11T08:24:39.949 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:24:39.950 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 96118 2024-09-11T08:24:39.950 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.950 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:24:39.950 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.950 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:24:39.950 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.629161+0000", 2024-09-11T08:24:39.950 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:24:39.950 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.950 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.950 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:24:39.950 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.629161+0000", 2024-09-11T08:24:39.950 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:24:39.950 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.951 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.951 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:24:39.951 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.629158+0000", 2024-09-11T08:24:39.951 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999976 2024-09-11T08:24:39.951 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.951 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.951 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:24:39.951 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.629185+0000", 2024-09-11T08:24:39.951 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.6849999999999999e-05 2024-09-11T08:24:39.951 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.951 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.951 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:24:39.951 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.629203+0000", 2024-09-11T08:24:39.951 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.7411e-05 2024-09-11T08:24:39.952 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.952 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.952 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:24:39.952 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.629209+0000", 2024-09-11T08:24:39.952 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.3489999999999997e-06 2024-09-11T08:24:39.952 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.952 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.952 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.952 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.629261+0000", 2024-09-11T08:24:39.952 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.2225000000000002e-05 2024-09-11T08:24:39.952 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.952 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.952 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:24:39.953 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.629268+0000", 2024-09-11T08:24:39.953 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.6590000000000001e-06 2024-09-11T08:24:39.953 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.953 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.953 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.953 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.599755+0000", 2024-09-11T08:24:39.953 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.970487129 2024-09-11T08:24:39.953 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.953 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.953 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:24:39.953 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.599766+0000", 2024-09-11T08:24:39.953 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0803e-05 2024-09-11T08:24:39.953 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.954 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.954 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.954 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.849312+0000", 2024-09-11T08:24:39.954 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.24954637 2024-09-11T08:24:39.954 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.954 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.954 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:24:39.954 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.849466+0000", 2024-09-11T08:24:39.954 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00015406 2024-09-11T08:24:39.954 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.954 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.954 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 3,7", 2024-09-11T08:24:39.954 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.850103+0000", 2024-09-11T08:24:39.954 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00063667200000000002 2024-09-11T08:24:39.955 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.955 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.955 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:24:39.955 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.871862+0000", 2024-09-11T08:24:39.955 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.021759013000000001 2024-09-11T08:24:39.955 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.955 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.955 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:24:39.957 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.871874+0000", 2024-09-11T08:24:39.957 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.225e-05 2024-09-11T08:24:39.957 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.957 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.957 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:24:39.957 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.871918+0000", 2024-09-11T08:24:39.957 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.3695000000000001e-05 2024-09-11T08:24:39.957 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.957 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.957 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:24:39.957 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.871999+0000", 2024-09-11T08:24:39.957 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.1187999999999996e-05 2024-09-11T08:24:39.957 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.958 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.958 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:24:39.958 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.881766+0000", 2024-09-11T08:24:39.958 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0097672809999999992 2024-09-11T08:24:39.958 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:39.958 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:24:39.958 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:39.958 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.958 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.958 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:96740 10.6 10:608cc7b1:::benchmark_data_smithi150_158067_object96739:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e329)", 2024-09-11T08:24:39.958 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:22:58.203679+0000", 2024-09-11T08:24:39.958 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 101.587899967, 2024-09-11T08:24:39.958 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.988728685, 2024-09-11T08:24:39.959 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:24:39.959 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:24:39.959 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:24:39.959 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:24:39.959 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:24:39.959 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 96740 2024-09-11T08:24:39.959 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.959 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:24:39.959 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.959 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:24:39.959 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.203679+0000", 2024-09-11T08:24:39.959 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:24:39.959 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.959 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.959 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:24:39.960 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.203679+0000", 2024-09-11T08:24:39.960 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:24:39.960 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.960 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.960 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:24:39.960 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.203679+0000", 2024-09-11T08:24:39.960 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:24:39.960 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.960 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.960 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:24:39.960 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.203699+0000", 2024-09-11T08:24:39.960 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.9962999999999998e-05 2024-09-11T08:24:39.960 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.961 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.961 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:24:39.961 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.203705+0000", 2024-09-11T08:24:39.961 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.4420000000000003e-06 2024-09-11T08:24:39.961 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.961 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.961 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:24:39.961 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.203710+0000", 2024-09-11T08:24:39.961 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.3290000000000004e-06 2024-09-11T08:24:39.961 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.961 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.961 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.961 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.207276+0000", 2024-09-11T08:24:39.961 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0035664920000000001 2024-09-11T08:24:39.962 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.962 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.962 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:24:39.962 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.207284+0000", 2024-09-11T08:24:39.962 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.422e-06 2024-09-11T08:24:39.962 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.962 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.962 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.962 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.167636+0000", 2024-09-11T08:24:39.962 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.96035166299999997 2024-09-11T08:24:39.962 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.962 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.963 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:24:39.963 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.169442+0000", 2024-09-11T08:24:39.963 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.001805699 2024-09-11T08:24:39.963 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.963 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.963 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 4,5", 2024-09-11T08:24:39.963 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.169678+0000", 2024-09-11T08:24:39.963 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00023583400000000001 2024-09-11T08:24:39.963 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.964 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.964 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:24:39.964 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.176352+0000", 2024-09-11T08:24:39.964 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0066739939999999999 2024-09-11T08:24:39.964 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.964 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.964 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:24:39.964 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.192259+0000", 2024-09-11T08:24:39.964 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.015906973000000001 2024-09-11T08:24:39.964 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.964 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.964 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:24:39.964 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.192276+0000", 2024-09-11T08:24:39.964 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.7173999999999999e-05 2024-09-11T08:24:39.965 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.965 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.965 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:24:39.965 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.192369+0000", 2024-09-11T08:24:39.965 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.2881999999999996e-05 2024-09-11T08:24:39.965 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.965 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.965 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:24:39.965 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.192408+0000", 2024-09-11T08:24:39.965 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.9437999999999997e-05 2024-09-11T08:24:39.965 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:39.965 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:24:39.965 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:39.965 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.966 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.966 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:96752 10.6 10:600083c2:::benchmark_data_smithi150_158067_object96751:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e329)", 2024-09-11T08:24:39.966 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:22:58.219995+0000", 2024-09-11T08:24:39.966 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 101.57158405, 2024-09-11T08:24:39.966 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.97261681899999997, 2024-09-11T08:24:39.966 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:24:39.966 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:24:39.966 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:24:39.966 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:24:39.966 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:24:39.966 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 96752 2024-09-11T08:24:39.966 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.966 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:24:39.966 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.966 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:24:39.967 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.219995+0000", 2024-09-11T08:24:39.967 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:24:39.967 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.967 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.967 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:24:39.967 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.219995+0000", 2024-09-11T08:24:39.967 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:24:39.967 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.968 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.968 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:24:39.968 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.219994+0000", 2024-09-11T08:24:39.968 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:24:39.968 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.968 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.968 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:24:39.968 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.220021+0000", 2024-09-11T08:24:39.968 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.7277e-05 2024-09-11T08:24:39.968 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.968 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.968 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:24:39.968 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.220033+0000", 2024-09-11T08:24:39.968 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2023e-05 2024-09-11T08:24:39.969 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.969 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.969 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:24:39.969 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.220040+0000", 2024-09-11T08:24:39.969 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.6640000000000001e-06 2024-09-11T08:24:39.969 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.969 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.969 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.969 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.225438+0000", 2024-09-11T08:24:39.969 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0053980979999999996 2024-09-11T08:24:39.969 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.969 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.970 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:24:39.970 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.225448+0000", 2024-09-11T08:24:39.970 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.6840000000000008e-06 2024-09-11T08:24:39.970 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.970 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.970 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:39.970 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.172192+0000", 2024-09-11T08:24:39.970 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.94674475499999999 2024-09-11T08:24:39.970 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.970 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.970 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:24:39.970 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.172273+0000", 2024-09-11T08:24:39.970 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.0925000000000002e-05 2024-09-11T08:24:39.971 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.971 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.971 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 4,5", 2024-09-11T08:24:39.971 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.172366+0000", 2024-09-11T08:24:39.971 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.2423999999999995e-05 2024-09-11T08:24:39.971 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.971 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.971 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:24:39.971 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.180239+0000", 2024-09-11T08:24:39.971 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0078728459999999993 2024-09-11T08:24:39.971 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.971 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.971 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:24:39.971 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.192445+0000", 2024-09-11T08:24:39.972 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.01220685 2024-09-11T08:24:39.972 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.972 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.972 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:24:39.972 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.192474+0000", 2024-09-11T08:24:39.972 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.8180000000000001e-05 2024-09-11T08:24:39.972 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.972 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.972 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:24:39.972 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.192573+0000", 2024-09-11T08:24:39.972 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.9289000000000004e-05 2024-09-11T08:24:39.972 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:39.972 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:39.972 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:24:39.973 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.192612+0000", 2024-09-11T08:24:39.973 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.9332000000000002e-05 2024-09-11T08:24:39.973 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:39.973 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:24:39.973 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:39.973 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:39.973 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:24:39.973 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-11T08:24:39.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:39.974 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:39.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:40.075 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:40.109 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:40.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:40.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:40.211 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:24:40.257 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:40.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:40.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:24:40.359 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:40.408 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:40.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:40.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:40.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:40 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:24:40.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:40 smithi150 ceph-mon[29276]: pgmap v782: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 3.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 728 op/s; 7980/177534 objects degraded (4.495%); 0 B/s, 101 objects/s recovering 2024-09-11T08:24:40.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:40 smithi150 ceph-mon[29276]: Health check update: Degraded data redundancy: 7980/177534 objects degraded (4.495%), 2 pgs degraded (PG_DEGRADED) 2024-09-11T08:24:40.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:40 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:24:40.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:40 smithi150 ceph-mon[39612]: pgmap v782: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 3.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 728 op/s; 7980/177534 objects degraded (4.495%); 0 B/s, 101 objects/s recovering 2024-09-11T08:24:40.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:40 smithi150 ceph-mon[39612]: Health check update: Degraded data redundancy: 7980/177534 objects degraded (4.495%), 2 pgs degraded (PG_DEGRADED) 2024-09-11T08:24:40.510 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:24:40.556 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:40.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:40.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:24:40.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:40 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:24:40.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:40 smithi181 ceph-mon[32701]: pgmap v782: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 3.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 728 op/s; 7980/177534 objects degraded (4.495%); 0 B/s, 101 objects/s recovering 2024-09-11T08:24:40.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:40 smithi181 ceph-mon[32701]: Health check update: Degraded data redundancy: 7980/177534 objects degraded (4.495%), 2 pgs degraded (PG_DEGRADED) 2024-09-11T08:24:40.658 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:24:40.696 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:40.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:40.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:24:40.797 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:40.836 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:40.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:40.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:40.938 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:40.979 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:40.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:40.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:41.081 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:41.119 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:41.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:41.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:41.221 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:41.260 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:41.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:41.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:41.362 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:41.401 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:41.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:41.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:41.503 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:41.549 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:41.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:41.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:41.596 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-11T08:24:41.651 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:41.689 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:41.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:41.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:41.791 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:41.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:41.843 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:41.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:41.944 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:24:41.952 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:41 smithi150 ceph-mon[29276]: pgmap v783: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 3.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 709 op/s; 7980/177534 objects degraded (4.495%); 0 B/s, 98 objects/s recovering 2024-09-11T08:24:41.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:41 smithi150 ceph-mon[39612]: pgmap v783: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 3.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 709 op/s; 7980/177534 objects degraded (4.495%); 0 B/s, 98 objects/s recovering 2024-09-11T08:24:41.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:41.984 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:41.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:24:42.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:41 smithi181 ceph-mon[32701]: pgmap v783: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 3.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 709 op/s; 7980/177534 objects degraded (4.495%); 0 B/s, 98 objects/s recovering 2024-09-11T08:24:42.086 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:42.119 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:42.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:42.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:42.221 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:42.258 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:42.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:42.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:42.360 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:24:42.394 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:42.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:42.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:24:42.495 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:42.538 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:42.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:42.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:42.639 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:42.684 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:42.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:42.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:42.786 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:24:42.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:42.830 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:42.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:24:42.930 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:24:42.955 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:42.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:42.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:24:43.057 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:43.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:43.090 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:43.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:43.191 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:43.218 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:43.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:43.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:43.320 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:43.353 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:43.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:43.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:43.455 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:43.477 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:43.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:43.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:43.578 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:43.618 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:43.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:43.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:43.720 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:43.749 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:43.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:43.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:43.851 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:43.888 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:43.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:43.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:43.990 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:24:44.022 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:44.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:44.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:24:44.124 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:44.156 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:44.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:44.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:44.258 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:44.298 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:44.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:44.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:44.400 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:44.422 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:44.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:44.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:44.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:44 smithi150 ceph-mon[29276]: pgmap v784: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 3.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 985 B/s rd, 0 B/s wr, 683 op/s; 7980/177534 objects degraded (4.495%); 0 B/s, 95 objects/s recovering 2024-09-11T08:24:44.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:44 smithi150 ceph-mon[39612]: pgmap v784: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 3.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 985 B/s rd, 0 B/s wr, 683 op/s; 7980/177534 objects degraded (4.495%); 0 B/s, 95 objects/s recovering 2024-09-11T08:24:44.524 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:44.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:44.563 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:44.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:44.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:44 smithi181 ceph-mon[32701]: pgmap v784: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 3.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 985 B/s rd, 0 B/s wr, 683 op/s; 7980/177534 objects degraded (4.495%); 0 B/s, 95 objects/s recovering 2024-09-11T08:24:44.665 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:44.702 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:44.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:44.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:44.804 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:44.839 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:44.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:44.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:44.941 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:44.975 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:44.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:44.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:45.077 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:45.110 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:45.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:45.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:45.212 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:24:45.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:45.250 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:45.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:24:45.351 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:45.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:45.391 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:45.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:45.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:45 smithi150 ceph-mon[29276]: Health check update: Degraded data redundancy: 7980/163854 objects degraded (4.870%), 2 pgs degraded (PG_DEGRADED) 2024-09-11T08:24:45.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:45 smithi150 ceph-mon[39612]: Health check update: Degraded data redundancy: 7980/163854 objects degraded (4.870%), 2 pgs degraded (PG_DEGRADED) 2024-09-11T08:24:45.492 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:24:45.528 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:45.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:45.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:24:45.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:45 smithi181 ceph-mon[32701]: Health check update: Degraded data redundancy: 7980/163854 objects degraded (4.870%), 2 pgs degraded (PG_DEGRADED) 2024-09-11T08:24:45.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:24:45 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:24:45] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:24:45.630 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:45.666 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:45.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:45.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:45.769 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:45.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:45.806 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:45.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:45.908 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:45.938 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:45.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:45.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:46.040 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:24:46.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:46.064 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:46.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:24:46.165 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:46.186 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:46.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:46.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:46.288 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:46.318 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:46.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:46.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:46.420 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:46.464 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:46.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:46.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:46.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:46 smithi150 ceph-mon[29276]: pgmap v785: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 3.3 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 874 op/s; 7980/163854 objects degraded (4.870%); 0 B/s, 123 objects/s recovering 2024-09-11T08:24:46.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:46 smithi150 ceph-mon[39612]: pgmap v785: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 3.3 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 874 op/s; 7980/163854 objects degraded (4.870%); 0 B/s, 123 objects/s recovering 2024-09-11T08:24:46.566 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:46.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:46 smithi181 ceph-mon[32701]: pgmap v785: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 3.3 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 874 op/s; 7980/163854 objects degraded (4.870%); 0 B/s, 123 objects/s recovering 2024-09-11T08:24:46.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:46.600 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:46.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:46.701 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:46.738 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:46.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:46.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:46.840 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:24:46.861 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:46.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:46.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:24:46.963 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:46.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:46.993 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:46.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:47.094 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:24:47.125 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:47.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:47.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:24:47.227 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:47.261 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:47.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:47.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:47.363 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:24:47.396 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:47.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:47.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:24:47.498 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:24:47.519 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:47.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:47.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:24:47.621 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:47.657 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:47.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:47.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:47.759 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:47.798 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:47.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:47.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:47.900 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:47.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:47.929 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:47.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:48.030 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:48.060 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:48.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:48.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:48.162 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:48.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:48.198 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:48.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:48.299 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:24:48.331 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:48.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:48.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:24:48.433 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:48.465 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:48.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:48.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:48.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:48 smithi150 ceph-mon[29276]: pgmap v786: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 3.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 728 op/s; 7980/163854 objects degraded (4.870%); 0 B/s, 98 objects/s recovering 2024-09-11T08:24:48.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:48 smithi150 ceph-mon[39612]: pgmap v786: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 3.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 728 op/s; 7980/163854 objects degraded (4.870%); 0 B/s, 98 objects/s recovering 2024-09-11T08:24:48.567 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:48.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:48 smithi181 ceph-mon[32701]: pgmap v786: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 3.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 728 op/s; 7980/163854 objects degraded (4.870%); 0 B/s, 98 objects/s recovering 2024-09-11T08:24:48.600 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:48.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:48.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:48.702 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:48.744 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:48.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:48.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:48.845 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:48.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:48.869 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:48.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:48.970 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:24:48.988 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:24:48 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:24:48] "GET /metrics HTTP/1.1" 200 224457 "" "Prometheus/2.43.0" 2024-09-11T08:24:49.003 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:49.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:49.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:24:49.074 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:24:48 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:24:49.106 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:49.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:49.141 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:49.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:49.241 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:49.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:49.286 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:49.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:49.387 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:49.416 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:49.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:49.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:49.518 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:24:49.561 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:49.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:49.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:24:49.663 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:49.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:49.695 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:49.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:49.796 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:49.823 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:49.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:49.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:49.924 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:49.960 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:49.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:49.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:50.062 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:50.086 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:50.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:50.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:50.188 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:50.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:50.234 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:50.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:50.335 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:50.381 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:50.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:50.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:50.482 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:50.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:50 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:24:50.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:50 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:24:50.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:50 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:24:50.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:50 smithi150 ceph-mon[29276]: pgmap v787: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 3.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s; 7980/151341 objects degraded (5.273%); 0 B/s, 155 objects/s recovering 2024-09-11T08:24:50.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:50 smithi150 ceph-mon[29276]: osdmap e357: 8 total, 8 up, 8 in 2024-09-11T08:24:50.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:50 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:24:50.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:50 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:24:50.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:50 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:24:50.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:50 smithi150 ceph-mon[39612]: pgmap v787: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 3.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s; 7980/151341 objects degraded (5.273%); 0 B/s, 155 objects/s recovering 2024-09-11T08:24:50.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:50 smithi150 ceph-mon[39612]: osdmap e357: 8 total, 8 up, 8 in 2024-09-11T08:24:50.514 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:50.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:50.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:50.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:50 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:24:50.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:50 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:24:50.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:50 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:24:50.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:50 smithi181 ceph-mon[32701]: pgmap v787: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 3.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s; 7980/151341 objects degraded (5.273%); 0 B/s, 155 objects/s recovering 2024-09-11T08:24:50.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:50 smithi181 ceph-mon[32701]: osdmap e357: 8 total, 8 up, 8 in 2024-09-11T08:24:50.616 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:50.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:50.664 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:50.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:50.765 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:24:50.795 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:50.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:50.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:24:50.897 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:50.928 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:50.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:50.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:51.030 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:51.055 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:51.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:51.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:51.157 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:51.200 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:51.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:51.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:51.302 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:51.338 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:51.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:51.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:51.440 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:24:51.472 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:51.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:51.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:24:51.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:51 smithi150 ceph-mon[29276]: Health check update: Degraded data redundancy: 7980/151341 objects degraded (5.273%), 2 pgs degraded (PG_DEGRADED) 2024-09-11T08:24:51.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:51 smithi150 ceph-mon[39612]: Health check update: Degraded data redundancy: 7980/151341 objects degraded (5.273%), 2 pgs degraded (PG_DEGRADED) 2024-09-11T08:24:51.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:51 smithi181 ceph-mon[32701]: Health check update: Degraded data redundancy: 7980/151341 objects degraded (5.273%), 2 pgs degraded (PG_DEGRADED) 2024-09-11T08:24:51.574 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:51.617 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:51.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:51.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:51.719 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:24:51.763 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:51.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:51.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:24:51.866 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:51.900 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:51.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:51.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:52.002 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:52.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:52.033 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:52.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:52.134 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:52.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:52.170 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:52.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:52.271 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:52.329 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:52.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:52.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:52.431 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:52.472 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:52.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:52.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:52.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:52 smithi150 ceph-mon[29276]: pgmap v789: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 3.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 873 op/s; 7980/151341 objects degraded (5.273%); 0 B/s, 129 objects/s recovering 2024-09-11T08:24:52.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:52 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:24:52.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:52 smithi150 ceph-mon[39612]: pgmap v789: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 3.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 873 op/s; 7980/151341 objects degraded (5.273%); 0 B/s, 129 objects/s recovering 2024-09-11T08:24:52.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:52 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:24:52.507 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:24:52.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:52 smithi181 ceph-mon[32701]: pgmap v789: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 3.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 873 op/s; 7980/151341 objects degraded (5.273%); 0 B/s, 129 objects/s recovering 2024-09-11T08:24:52.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:52 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:24:52.574 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:52.614 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:52.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:52.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:52.717 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:52.757 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:52.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:52.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:52.850 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T08:24:52.860 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:52.942 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:52.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:52.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:53.044 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:24:53.109 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:53.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:53.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:24:53.211 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:24:53.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:53.253 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:53.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:24:53.354 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:53.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:53.400 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:53.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:53.501 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:53.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:53.545 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:53.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:53.646 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:53.688 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:53.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:53.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:53.791 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:24:53.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:53.823 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:53.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:24:53.924 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:53.960 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:53.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:53.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:54.062 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:54.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:54.098 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:54.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:54.199 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:54.240 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:54.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:54.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:54.242 INFO:teuthology.orchestra.run.smithi150.stderr:reweighted osd.0 to 1 (10000) 2024-09-11T08:24:54.247 INFO:teuthology.orchestra.run.smithi150.stderr:nodeep-scrub is set 2024-09-11T08:24:54.264 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:54 smithi150 ceph-mon[29276]: pgmap v790: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 3.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 873 op/s; 7980/151341 objects degraded (5.273%); 0 B/s, 129 objects/s recovering 2024-09-11T08:24:54.264 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:54 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/83301417' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:24:54.264 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:54 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:24:54.264 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:54 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/145504000' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-11T08:24:54.264 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:54 smithi150 ceph-mon[39612]: pgmap v790: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 3.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 873 op/s; 7980/151341 objects degraded (5.273%); 0 B/s, 129 objects/s recovering 2024-09-11T08:24:54.265 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:54 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/83301417' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:24:54.265 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:54 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:24:54.265 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:54 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/145504000' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-11T08:24:54.342 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:54.400 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:54.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:54.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:54.502 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:24:54.545 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:54.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:54.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:24:54.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:54 smithi181 ceph-mon[32701]: pgmap v790: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 3.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 873 op/s; 7980/151341 objects degraded (5.273%); 0 B/s, 129 objects/s recovering 2024-09-11T08:24:54.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:54 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/83301417' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:24:54.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:54 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:24:54.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:54 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/145504000' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-11T08:24:54.647 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:24:54.681 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:54.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:54.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:24:54.783 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:54.823 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:54.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:54.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:54.924 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:54.958 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:54.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:54.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:55.060 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:24:55.098 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:55.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:55.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:24:55.200 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:55.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:55.247 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:55.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:55.348 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:24:55.382 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:55.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:55.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:24:55.484 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:55.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:55 smithi150 ceph-mon[29276]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:24:55.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:55 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:24:55.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:55 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/145504000' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-11T08:24:55.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:55 smithi150 ceph-mon[29276]: osdmap e358: 8 total, 8 up, 8 in 2024-09-11T08:24:55.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:55 smithi150 ceph-mon[39612]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:24:55.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:55 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:24:55.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:55 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/145504000' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-11T08:24:55.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:55 smithi150 ceph-mon[39612]: osdmap e358: 8 total, 8 up, 8 in 2024-09-11T08:24:55.521 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:55.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:55.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:55.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:55 smithi181 ceph-mon[32701]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:24:55.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:55 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:24:55.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:55 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/145504000' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-11T08:24:55.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:55 smithi181 ceph-mon[32701]: osdmap e358: 8 total, 8 up, 8 in 2024-09-11T08:24:55.574 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:24:55 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:24:55] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:24:55.623 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:55.649 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-11T08:24:55.649 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [ 2024-09-11T08:24:55.649 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:55.649 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:216762 10.2 10:42af097d:::benchmark_data_smithi150_158067_object86518:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e358)", 2024-09-11T08:24:55.649 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:24:55.646503+0000", 2024-09-11T08:24:55.649 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.0018678519999999999, 2024-09-11T08:24:55.649 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0018984900000000001, 2024-09-11T08:24:55.650 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:24:55.650 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "waiting for sub ops", 2024-09-11T08:24:55.650 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:24:55.650 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:24:55.650 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:24:55.650 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 216762 2024-09-11T08:24:55.650 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:55.650 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:24:55.650 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:55.650 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:24:55.650 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:55.646503+0000", 2024-09-11T08:24:55.650 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:24:55.650 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:55.650 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:55.651 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:24:55.651 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:55.646503+0000", 2024-09-11T08:24:55.651 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:24:55.651 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:55.651 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:55.651 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:24:55.651 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:55.646501+0000", 2024-09-11T08:24:55.651 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:24:55.651 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:55.651 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:55.651 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:24:55.651 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:55.646509+0000", 2024-09-11T08:24:55.651 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.8229999999999995e-06 2024-09-11T08:24:55.651 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:55.652 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:55.652 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:24:55.652 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:55.646526+0000", 2024-09-11T08:24:55.652 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.7238999999999999e-05 2024-09-11T08:24:55.652 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:55.652 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:55.652 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:24:55.652 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:55.646533+0000", 2024-09-11T08:24:55.652 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.6640000000000001e-06 2024-09-11T08:24:55.652 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:55.652 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:55.652 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:24:55.652 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:55.646635+0000", 2024-09-11T08:24:55.652 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000102265 2024-09-11T08:24:55.653 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:55.653 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:55.653 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:24:55.653 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:55.646748+0000", 2024-09-11T08:24:55.653 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000113263 2024-09-11T08:24:55.653 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:55.653 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:55.653 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 0,4", 2024-09-11T08:24:55.653 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:55.646881+0000", 2024-09-11T08:24:55.653 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00013255799999999999 2024-09-11T08:24:55.653 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:55.653 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:24:55.653 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:55.653 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:55.654 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:55.654 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.15636.0:216757 10.0 e358/308)", 2024-09-11T08:24:55.654 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:24:55.646591+0000", 2024-09-11T08:24:55.654 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.001779751, 2024-09-11T08:24:55.654 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0018809759999999999, 2024-09-11T08:24:55.654 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:24:55.654 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "queued for pg", 2024-09-11T08:24:55.654 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:24:55.654 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:55.654 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:24:55.654 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:55.646591+0000", 2024-09-11T08:24:55.654 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:24:55.654 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:55.654 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:55.655 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:24:55.655 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:55.646591+0000", 2024-09-11T08:24:55.655 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:24:55.655 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:55.655 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:55.655 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:24:55.655 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:55.646590+0000", 2024-09-11T08:24:55.655 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:24:55.655 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:55.655 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:55.655 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:24:55.655 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:55.646599+0000", 2024-09-11T08:24:55.655 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.4919999999999993e-06 2024-09-11T08:24:55.655 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:55.655 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:55.656 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:24:55.656 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:55.646616+0000", 2024-09-11T08:24:55.656 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.7078000000000001e-05 2024-09-11T08:24:55.656 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:24:55.656 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:24:55.656 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:24:55.656 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:55.646622+0000", 2024-09-11T08:24:55.656 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.472e-06 2024-09-11T08:24:55.656 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:55.656 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:24:55.656 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:55.656 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:24:55.656 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-09-11T08:24:55.656 INFO:teuthology.orchestra.run.smithi150.stdout: "num_ops": 2 2024-09-11T08:24:55.657 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-11T08:24:55.658 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:55.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:55.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:55.759 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:55.797 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:55.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:55.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:55.899 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:55.936 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:55.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:55.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:56.038 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:56.071 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:56.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:56.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:56.173 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:24:56.182 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:24:56.221 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd reweight 1 1 2024-09-11T08:24:56.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:56.245 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:56.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:24:56.347 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:56.375 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:56.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:56.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:56.477 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:56.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:56 smithi150 ceph-mon[29276]: pgmap v792: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 2.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.07k op/s; 7980/138120 objects degraded (5.778%); 0 B/s, 162 objects/s recovering 2024-09-11T08:24:56.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:56 smithi150 ceph-mon[29276]: Health check update: Degraded data redundancy: 7980/138120 objects degraded (5.778%), 2 pgs degraded (PG_DEGRADED) 2024-09-11T08:24:56.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:56 smithi150 ceph-mon[39612]: pgmap v792: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 2.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.07k op/s; 7980/138120 objects degraded (5.778%); 0 B/s, 162 objects/s recovering 2024-09-11T08:24:56.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:56 smithi150 ceph-mon[39612]: Health check update: Degraded data redundancy: 7980/138120 objects degraded (5.778%), 2 pgs degraded (PG_DEGRADED) 2024-09-11T08:24:56.514 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:56.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:56.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:56.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:56 smithi181 ceph-mon[32701]: pgmap v792: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 2.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.07k op/s; 7980/138120 objects degraded (5.778%); 0 B/s, 162 objects/s recovering 2024-09-11T08:24:56.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:56 smithi181 ceph-mon[32701]: Health check update: Degraded data redundancy: 7980/138120 objects degraded (5.778%), 2 pgs degraded (PG_DEGRADED) 2024-09-11T08:24:56.617 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:24:56.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:56.659 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:56.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:24:56.760 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:24:56.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:56.798 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:56.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:24:56.898 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:56.931 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:56.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:56.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:57.033 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:57.065 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:57.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:57.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:57.167 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:24:57.221 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:57.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:57.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:24:57.323 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:24:57.368 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:57.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:57.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:24:57.470 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:57.505 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:57.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:57.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:57.607 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:57.642 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:57.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:57.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:57.744 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:24:57.785 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:57.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:57.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:24:57.807 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-11T08:24:57.887 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:57.923 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:57.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:57.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:58.025 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:58.083 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:58.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:58.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:58.185 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:24:58.226 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:58.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:58.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:24:58.328 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:24:58.365 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:58.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:58.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:24:58.467 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:58.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:24:58 smithi150 ceph-mon[29276]: pgmap v793: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 2.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 551 op/s; 7980/138120 objects degraded (5.778%); 0 B/s, 77 objects/s recovering 2024-09-11T08:24:58.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:24:58 smithi150 ceph-mon[39612]: pgmap v793: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 2.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 551 op/s; 7980/138120 objects degraded (5.778%); 0 B/s, 77 objects/s recovering 2024-09-11T08:24:58.504 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:58.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:58.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:58.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:24:58 smithi181 ceph-mon[32701]: pgmap v793: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 2.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 551 op/s; 7980/138120 objects degraded (5.778%); 0 B/s, 77 objects/s recovering 2024-09-11T08:24:58.607 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:24:58.638 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:58.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:58.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:24:58.739 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:24:58.762 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:58.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:58.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:24:58.825 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T08:24:58.858 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:24:58 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:24:58] "GET /metrics HTTP/1.1" 200 224444 "" "Prometheus/2.43.0" 2024-09-11T08:24:58.864 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:24:58.904 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:58.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:58.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:24:59.006 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:59.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:59.051 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:59.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:59.073 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:24:58 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:24:59.151 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:59.186 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:59.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:59.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:59.288 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:59.316 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:59.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:59.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:24:59.418 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:24:59.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:59.464 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:59.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:24:59.565 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:59.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:59.601 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:59.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:59.702 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:24:59.742 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:24:59.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:59.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:24:59.844 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:24:59.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:24:59.867 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:24:59.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:24:59.968 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:24:59.995 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:25:00.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:00.005 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:00.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:25:00.107 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:25:00.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:00.156 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:00.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:25:00.257 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:00.289 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:00.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:00.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:00.391 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:25:00.427 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:00.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:00.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:25:00.448 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:00 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:25:00.448 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:00 smithi150 ceph-mon[29276]: pgmap v794: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 2.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 916 op/s; 7980/124188 objects degraded (6.426%); 0 B/s, 135 objects/s recovering 2024-09-11T08:25:00.448 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:00 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:25:00.448 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:00 smithi150 ceph-mon[39612]: pgmap v794: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 2.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 916 op/s; 7980/124188 objects degraded (6.426%); 0 B/s, 135 objects/s recovering 2024-09-11T08:25:00.529 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:25:00.564 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:00.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:00.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:25:00.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:00 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:25:00.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:00 smithi181 ceph-mon[32701]: pgmap v794: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 2.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 916 op/s; 7980/124188 objects degraded (6.426%); 0 B/s, 135 objects/s recovering 2024-09-11T08:25:00.667 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:25:00.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:00.704 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:00.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:25:00.806 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:25:00.839 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:00.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:00.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:25:00.941 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:25:00.978 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:00.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:00.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:25:01.080 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:25:01.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:01.116 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:01.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:25:01.217 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:01.263 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:01.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:01.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:01.365 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:01.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:01.403 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:01.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:01.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:01 smithi150 ceph-mon[29276]: Health check update: Degraded data redundancy: 7980/124188 objects degraded (6.426%), 2 pgs degraded (PG_DEGRADED) 2024-09-11T08:25:01.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:01 smithi150 ceph-mon[39612]: Health check update: Degraded data redundancy: 7980/124188 objects degraded (6.426%), 2 pgs degraded (PG_DEGRADED) 2024-09-11T08:25:01.504 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:01.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:01.546 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:01.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:01.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:01 smithi181 ceph-mon[32701]: Health check update: Degraded data redundancy: 7980/124188 objects degraded (6.426%), 2 pgs degraded (PG_DEGRADED) 2024-09-11T08:25:01.648 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:25:01.686 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:01.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:01.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:25:01.788 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:25:01.833 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:01.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:01.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:25:01.934 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:01.964 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:01.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:01.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:02.066 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:02.103 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:02.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:02.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:02.206 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:02.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:02.236 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:02.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:02.337 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:02.364 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:02.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:02.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:02.466 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:02.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:02.503 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:02.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:02.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:02 smithi181 ceph-mon[32701]: pgmap v795: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 2.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 905 op/s; 7980/124188 objects degraded (6.426%); 0 B/s, 133 objects/s recovering 2024-09-11T08:25:02.604 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:02.648 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:02.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:02.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:02.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:02 smithi150 ceph-mon[29276]: pgmap v795: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 2.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 905 op/s; 7980/124188 objects degraded (6.426%); 0 B/s, 133 objects/s recovering 2024-09-11T08:25:02.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:02 smithi150 ceph-mon[39612]: pgmap v795: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 2.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 905 op/s; 7980/124188 objects degraded (6.426%); 0 B/s, 133 objects/s recovering 2024-09-11T08:25:02.750 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:25:02.791 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:02.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:02.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:25:02.893 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:25:02.922 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:02.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:02.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:25:03.024 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:03.055 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:03.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:03.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:03.157 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:25:03.187 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:03.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:03.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:25:03.282 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:25:03.289 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:03.322 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:03.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:03.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:03.424 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:03.467 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:03.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:03.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:03.568 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:03.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:03.597 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:03.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:03.698 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:03.728 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:03.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:03.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:03.830 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:25:03.870 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:03.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:03.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:25:03.972 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:04.008 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:04.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:04.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:04.110 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:04.148 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:04.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:04.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:04.251 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:04.287 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:04.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:04.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:04.389 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:04.413 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:04.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:04.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:04.515 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:25:04.537 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:04.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:04.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:25:04.577 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:04 smithi181 ceph-mon[32701]: pgmap v796: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 2.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 905 op/s; 7980/124188 objects degraded (6.426%); 0 B/s, 133 objects/s recovering 2024-09-11T08:25:04.577 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:04 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:25:04.577 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:04 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:25:04.577 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:04 smithi181 ceph-mon[32701]: osdmap e359: 8 total, 8 up, 8 in 2024-09-11T08:25:04.639 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:25:04.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:04.672 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:04.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:25:04.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:04 smithi150 ceph-mon[29276]: pgmap v796: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 2.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 905 op/s; 7980/124188 objects degraded (6.426%); 0 B/s, 133 objects/s recovering 2024-09-11T08:25:04.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:04 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:25:04.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:04 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:25:04.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:04 smithi150 ceph-mon[29276]: osdmap e359: 8 total, 8 up, 8 in 2024-09-11T08:25:04.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:04 smithi150 ceph-mon[39612]: pgmap v796: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 2.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 905 op/s; 7980/124188 objects degraded (6.426%); 0 B/s, 133 objects/s recovering 2024-09-11T08:25:04.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:04 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:25:04.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:04 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:25:04.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:04 smithi150 ceph-mon[39612]: osdmap e359: 8 total, 8 up, 8 in 2024-09-11T08:25:04.772 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:04.810 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:04.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:04.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:04.912 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:25:04.935 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:04.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:04.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:25:05.037 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:05.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:05.074 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:05.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:05.175 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:05.205 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:05.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:05.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:05.307 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:25:05.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:05.360 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:05.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:25:05.461 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:05.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:05.513 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:05.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:05.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:25:05 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:25:05] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:25:05.614 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:25:05.649 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:05.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:05.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:25:05.751 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:25:05.790 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:05.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:05.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:25:05.892 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:05.935 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:05.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:05.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:06.037 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:06.074 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:06.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:06.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:06.176 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:06.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:06.199 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:06.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:06.300 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:25:06.328 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:06.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:06.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:25:06.430 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:06.459 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:06.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:06.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:06.561 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:06.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:06 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:25:06.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:06 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:25:06.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:06 smithi181 ceph-mon[32701]: pgmap v798: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 2.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 912 op/s; 7980/110769 objects degraded (7.204%); 0 B/s, 147 objects/s recovering 2024-09-11T08:25:06.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:06 smithi181 ceph-mon[32701]: osdmap e360: 8 total, 8 up, 8 in 2024-09-11T08:25:06.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:06 smithi181 ceph-mon[32701]: Health check update: Degraded data redundancy: 7980/110769 objects degraded (7.204%), 2 pgs degraded (PG_DEGRADED) 2024-09-11T08:25:06.599 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:06.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:06.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:06.673 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:06 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:25:06.673 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:06 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:25:06.673 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:06 smithi150 ceph-mon[29276]: pgmap v798: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 2.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 912 op/s; 7980/110769 objects degraded (7.204%); 0 B/s, 147 objects/s recovering 2024-09-11T08:25:06.673 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:06 smithi150 ceph-mon[29276]: osdmap e360: 8 total, 8 up, 8 in 2024-09-11T08:25:06.673 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:06 smithi150 ceph-mon[29276]: Health check update: Degraded data redundancy: 7980/110769 objects degraded (7.204%), 2 pgs degraded (PG_DEGRADED) 2024-09-11T08:25:06.673 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:06 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:25:06.673 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:06 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:25:06.673 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:06 smithi150 ceph-mon[39612]: pgmap v798: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 2.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 912 op/s; 7980/110769 objects degraded (7.204%); 0 B/s, 147 objects/s recovering 2024-09-11T08:25:06.674 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:06 smithi150 ceph-mon[39612]: osdmap e360: 8 total, 8 up, 8 in 2024-09-11T08:25:06.674 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:06 smithi150 ceph-mon[39612]: Health check update: Degraded data redundancy: 7980/110769 objects degraded (7.204%), 2 pgs degraded (PG_DEGRADED) 2024-09-11T08:25:06.701 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:06.740 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:06.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:06.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:06.842 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:25:06.884 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:06.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:06.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:25:06.987 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:07.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:07.027 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:07.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:07.128 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:07.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:07.154 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:07.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:07.255 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:07.277 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:07.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:07.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:07.378 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:07.424 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:07.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:07.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:07.526 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:25:07.557 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:07.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:07.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:25:07.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:07 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:25:07.659 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:07.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:07.701 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:07.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:07.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:07 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:25:07.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:07 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:25:07.802 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:07.840 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:07.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:07.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:07.942 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:25:07.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:07.982 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:07.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:25:08.083 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:08.105 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:08.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:08.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:08.118 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:25:08.206 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:25:08.234 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:08.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:08.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:25:08.336 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:08.375 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:08.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:08.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:08.477 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:08.512 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:08.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:08.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:08.557 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:08 smithi150 ceph-mon[29276]: pgmap v800: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 2.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 559 op/s; 7980/110769 objects degraded (7.204%); 0 B/s, 95 objects/s recovering 2024-09-11T08:25:08.558 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:08 smithi150 ceph-mon[39612]: pgmap v800: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 2.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 559 op/s; 7980/110769 objects degraded (7.204%); 0 B/s, 95 objects/s recovering 2024-09-11T08:25:08.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:08 smithi181 ceph-mon[32701]: pgmap v800: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 2.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 559 op/s; 7980/110769 objects degraded (7.204%); 0 B/s, 95 objects/s recovering 2024-09-11T08:25:08.614 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:08.646 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:08.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:08.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:08.748 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:25:08.782 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:08.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:08.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:25:08.884 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:25:08.914 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:08.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:08.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:25:08.988 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:25:08 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:25:08] "GET /metrics HTTP/1.1" 200 224446 "" "Prometheus/2.43.0" 2024-09-11T08:25:09.015 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:09.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:09.054 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:09.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:09.073 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:25:08 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:25:09.155 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:09.191 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:09.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:09.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:09.293 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:09.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:09.331 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:09.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:09.432 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:25:09.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:09.455 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:09.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:25:09.556 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:09.589 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:09.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:09.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:09.691 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:25:09.733 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:09.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:09.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:25:09.835 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:09.856 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:09.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:09.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:09.957 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:25:09.995 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:09.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:09.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:25:10.097 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:25:10.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:10.134 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:10.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:25:10.235 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:25:10.274 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:10.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:10.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:25:10.375 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:25:10.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:10.398 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:10.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:25:10.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:10 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:25:10.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:10 smithi150 ceph-mon[29276]: pgmap v801: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 2.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 964 op/s; 7980/101055 objects degraded (7.897%); 0 B/s, 181 objects/s recovering 2024-09-11T08:25:10.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:10 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:25:10.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:10 smithi150 ceph-mon[39612]: pgmap v801: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 2.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 964 op/s; 7980/101055 objects degraded (7.897%); 0 B/s, 181 objects/s recovering 2024-09-11T08:25:10.499 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:10.539 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:10.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:10.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:10.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:10 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:25:10.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:10 smithi181 ceph-mon[32701]: pgmap v801: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 2.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 964 op/s; 7980/101055 objects degraded (7.897%); 0 B/s, 181 objects/s recovering 2024-09-11T08:25:10.641 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:10.674 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:10.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:10.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:10.776 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:25:10.816 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:10.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:10.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:25:10.918 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:10.953 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:10.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:10.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:11.055 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:11.092 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:11.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:11.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:11.194 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:25:11.233 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:11.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:11.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:25:11.335 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:11.358 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:11.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:11.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:11.460 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:25:11.494 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:11.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:11.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:25:11.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:11 smithi181 ceph-mon[32701]: Health check update: Degraded data redundancy: 7980/101055 objects degraded (7.897%), 2 pgs degraded (PG_DEGRADED) 2024-09-11T08:25:11.596 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:11.633 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:11.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:11.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:11.734 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:25:11.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:11 smithi150 ceph-mon[29276]: Health check update: Degraded data redundancy: 7980/101055 objects degraded (7.897%), 2 pgs degraded (PG_DEGRADED) 2024-09-11T08:25:11.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:11 smithi150 ceph-mon[39612]: Health check update: Degraded data redundancy: 7980/101055 objects degraded (7.897%), 2 pgs degraded (PG_DEGRADED) 2024-09-11T08:25:11.773 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:11.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:11.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:25:11.874 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:25:11.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:11.914 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:11.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:25:12.015 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:25:12.057 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:12.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:12.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:25:12.159 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:12.193 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:12.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:12.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:12.295 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:12.335 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:12.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:12.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:12.437 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:25:12.467 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:12.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:12.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:25:12.569 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:12.572 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:12 smithi181 ceph-mon[32701]: pgmap v802: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 2.0 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.06k op/s; 7980/98760 objects degraded (8.080%); 0 B/s, 216 objects/s recovering 2024-09-11T08:25:12.608 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:12 smithi150 ceph-mon[29276]: pgmap v802: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 2.0 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.06k op/s; 7980/98760 objects degraded (8.080%); 0 B/s, 216 objects/s recovering 2024-09-11T08:25:12.608 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:12 smithi150 ceph-mon[39612]: pgmap v802: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 2.0 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.06k op/s; 7980/98760 objects degraded (8.080%); 0 B/s, 216 objects/s recovering 2024-09-11T08:25:12.609 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:12.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:12.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:12.711 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:25:12.746 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:12.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:12.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:25:12.848 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:25:12.880 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:12.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:12.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:25:12.982 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:25:13.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:13.019 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:13.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:25:13.120 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:25:13.160 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:13.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:13.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:25:13.262 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:13.294 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:13.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:13.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:13.396 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:25:13.428 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:13.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:13.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:25:13.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:13 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1534201281' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:25:13.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:13 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1534201281' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:25:13.530 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:25:13.570 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:13.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:13.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:25:13.672 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:13.699 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:13.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:13.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:13.801 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:13.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:13 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1534201281' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:25:13.825 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:25:13.825 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:25:13.826 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:25:13.826 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:25:13.826 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:25:13.826 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:25:13.826 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:25:13.826 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:25:13.826 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:25:13.826 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:25:13.826 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:25:13.827 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:25:13.827 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:25:13.827 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:25:13.827 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:25:13.827 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:25:13.837 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:13.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:13.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:13.939 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:25:13.967 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:13.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:13.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:25:14.069 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:25:14.118 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:14.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:14.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:25:14.220 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:14.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:14.260 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:14.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:14.362 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:14.396 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:14.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:14.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:14.482 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:14 smithi150 ceph-mon[29276]: pgmap v803: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 2.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 520 B/s rd, 0 B/s wr, 609 op/s; 7980/98760 objects degraded (8.080%); 0 B/s, 143 objects/s recovering 2024-09-11T08:25:14.482 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:14 smithi150 ceph-mon[39612]: pgmap v803: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 2.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 520 B/s rd, 0 B/s wr, 609 op/s; 7980/98760 objects degraded (8.080%); 0 B/s, 143 objects/s recovering 2024-09-11T08:25:14.498 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:14.531 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:14.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:14.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:14.633 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:25:14.666 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:14.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:14.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:25:14.767 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:14.804 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:14.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:14.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:14.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:14 smithi181 ceph-mon[32701]: pgmap v803: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 2.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 520 B/s rd, 0 B/s wr, 609 op/s; 7980/98760 objects degraded (8.080%); 0 B/s, 143 objects/s recovering 2024-09-11T08:25:14.905 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:25:14.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:14.939 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:14.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:25:15.040 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:15.076 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:15.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:15.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:15.178 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:15.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:15.231 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:15.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:15.332 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:15.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:15.369 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:15.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:15.470 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:15.504 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:15.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:15.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:15.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:25:15 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:25:15] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:25:15.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:15 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:25:15.607 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:15.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:15.653 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:15.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:15.666 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:15 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:25:15.667 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:15 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:25:15.754 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:15.785 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:15.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:15.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:15.887 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:25:15.929 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:15.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:15.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:25:16.030 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:16.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:16.075 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:16.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:16.176 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:25:16.208 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:16.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:16.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:25:16.310 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:25:16.347 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:16.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:16.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:25:16.362 INFO:teuthology.orchestra.run.smithi150.stderr:reweighted osd.1 to 1 (10000) 2024-09-11T08:25:16.448 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:25:16.491 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:16.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:16.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:25:16.522 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T08:25:16.593 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:16.657 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:16.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:16.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:16.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:16 smithi150 ceph-mon[29276]: pgmap v804: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 1.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 862 op/s; 7980/84909 objects degraded (9.398%); 0 B/s, 181 objects/s recovering 2024-09-11T08:25:16.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:16 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1085928785' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-11T08:25:16.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:16 smithi150 ceph-mon[29276]: Health check update: Degraded data redundancy: 7980/84909 objects degraded (9.398%), 2 pgs degraded (PG_DEGRADED) 2024-09-11T08:25:16.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:16 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:25:16.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:16 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:25:16.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:16 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:25:16.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:16 smithi150 ceph-mon[39612]: pgmap v804: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 1.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 862 op/s; 7980/84909 objects degraded (9.398%); 0 B/s, 181 objects/s recovering 2024-09-11T08:25:16.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:16 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1085928785' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-11T08:25:16.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:16 smithi150 ceph-mon[39612]: Health check update: Degraded data redundancy: 7980/84909 objects degraded (9.398%), 2 pgs degraded (PG_DEGRADED) 2024-09-11T08:25:16.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:16 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:25:16.742 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:16 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:25:16.742 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:16 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:25:16.759 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:25:16.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:16.799 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:16.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:25:16.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:16 smithi181 ceph-mon[32701]: pgmap v804: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 1.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 862 op/s; 7980/84909 objects degraded (9.398%); 0 B/s, 181 objects/s recovering 2024-09-11T08:25:16.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:16 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1085928785' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-11T08:25:16.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:16 smithi181 ceph-mon[32701]: Health check update: Degraded data redundancy: 7980/84909 objects degraded (9.398%), 2 pgs degraded (PG_DEGRADED) 2024-09-11T08:25:16.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:16 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:25:16.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:16 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:25:16.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:16 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:25:16.899 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:16.938 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:16.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:16.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:17.040 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:17.082 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:17.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:17.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:17.184 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:17.217 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:17.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:17.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:17.319 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:25:17.353 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:17.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:17.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:25:17.455 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:17.490 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:17.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:17.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:17.591 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:25:17.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:17.633 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:17.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:25:17.734 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:17.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:17 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1085928785' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-11T08:25:17.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:17 smithi150 ceph-mon[29276]: osdmap e361: 8 total, 8 up, 8 in 2024-09-11T08:25:17.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:17 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1085928785' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-11T08:25:17.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:17 smithi150 ceph-mon[39612]: osdmap e361: 8 total, 8 up, 8 in 2024-09-11T08:25:17.777 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:17.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:17.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:17.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:17 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1085928785' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-11T08:25:17.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:17 smithi181 ceph-mon[32701]: osdmap e361: 8 total, 8 up, 8 in 2024-09-11T08:25:17.878 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:25:17.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:17.914 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:17.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:25:18.015 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:18.039 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:18.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:18.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:18.141 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:25:18.176 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:18.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:18.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:25:18.278 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:18.309 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:18.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:18.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:18.411 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:18.439 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:18.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:18.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:18.541 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:25:18.580 INFO:teuthology.orchestra.run.smithi150.stderr:noscrub is unset 2024-09-11T08:25:18.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:18.582 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:18.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:25:18.683 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:18.719 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:18.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:18.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:18.724 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:18 smithi181 ceph-mon[32701]: pgmap v806: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 1.7 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 862 op/s; 7980/84909 objects degraded (9.398%); 0 B/s, 181 objects/s recovering 2024-09-11T08:25:18.724 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:18 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2502806976' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:25:18.724 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:18 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:25:18.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:18 smithi150 ceph-mon[29276]: pgmap v806: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 1.7 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 862 op/s; 7980/84909 objects degraded (9.398%); 0 B/s, 181 objects/s recovering 2024-09-11T08:25:18.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:18 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2502806976' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:25:18.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:18 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:25:18.738 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:25:18 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:25:18] "GET /metrics HTTP/1.1" 200 224446 "" "Prometheus/2.43.0" 2024-09-11T08:25:18.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:18 smithi150 ceph-mon[39612]: pgmap v806: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 1.7 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 862 op/s; 7980/84909 objects degraded (9.398%); 0 B/s, 181 objects/s recovering 2024-09-11T08:25:18.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:18 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2502806976' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:25:18.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:18 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:25:18.820 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:25:18.853 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:18.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:18.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:25:18.955 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:18.989 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:18.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:18.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:19.073 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:25:18 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:25:19.090 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:25:19.125 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:19.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:19.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:25:19.227 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:19.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:19.264 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:19.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:19.365 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:19.401 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:19.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:19.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:19.503 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:25:19.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:19.542 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:19.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:25:19.643 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:19.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:19.682 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:19.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:19.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:19 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:25:19.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:19 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:25:19.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:19 smithi150 ceph-mon[29276]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:25:19.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:19 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:25:19.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:19 smithi150 ceph-mon[29276]: osdmap e362: 8 total, 8 up, 8 in 2024-09-11T08:25:19.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:19 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:25:19.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:19 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:25:19.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:19 smithi150 ceph-mon[39612]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:25:19.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:19 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:25:19.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:19 smithi150 ceph-mon[39612]: osdmap e362: 8 total, 8 up, 8 in 2024-09-11T08:25:19.784 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:19.809 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd reweight 2 1 2024-09-11T08:25:19.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:19 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:25:19.825 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:19 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:25:19.825 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:19 smithi181 ceph-mon[32701]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:25:19.825 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:19 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:25:19.825 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:19 smithi181 ceph-mon[32701]: osdmap e362: 8 total, 8 up, 8 in 2024-09-11T08:25:19.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:19.826 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:19.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:19.926 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:19.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:19.966 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:19.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:20.067 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:20.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:20.112 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:20.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:20.213 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:20.245 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:20.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:20.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:20.348 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:20.396 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:20.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:20.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:20.428 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:20 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:25:20.428 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:20 smithi150 ceph-mon[29276]: pgmap v808: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 1.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.05k op/s; 7980/73572 objects degraded (10.847%); 0 B/s, 188 objects/s recovering 2024-09-11T08:25:20.428 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:20 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:25:20.428 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:20 smithi150 ceph-mon[39612]: pgmap v808: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 1.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.05k op/s; 7980/73572 objects degraded (10.847%); 0 B/s, 188 objects/s recovering 2024-09-11T08:25:20.498 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:20.536 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:20.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:20.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:20.638 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:20.718 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:20.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:20.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:20.820 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:20.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:20 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:25:20.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:20 smithi181 ceph-mon[32701]: pgmap v808: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 1.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.05k op/s; 7980/73572 objects degraded (10.847%); 0 B/s, 188 objects/s recovering 2024-09-11T08:25:20.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:20.857 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:20.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:20.886 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:25:20.886 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:25:20.916 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-09-11T08:25:20.916 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:25:20.958 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:25:20.991 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:20.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:20.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:25:21.094 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:21.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:21.125 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:21.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:21.226 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:25:21.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:21.269 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:21.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:25:21.370 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:25:21.408 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:21.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:21.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:25:21.510 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:21.547 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:21.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:21.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:21.649 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:25:21.681 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:21.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:21.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:25:21.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:21 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:25:21.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:21 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:25:21.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:21 smithi150 ceph-mon[29276]: osdmap e363: 8 total, 8 up, 8 in 2024-09-11T08:25:21.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:21 smithi150 ceph-mon[29276]: 3.4 scrub starts 2024-09-11T08:25:21.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:21 smithi150 ceph-mon[29276]: Health check update: Degraded data redundancy: 7980/73572 objects degraded (10.847%), 2 pgs degraded (PG_DEGRADED) 2024-09-11T08:25:21.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:21 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:25:21.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:21 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:25:21.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:21 smithi150 ceph-mon[39612]: osdmap e363: 8 total, 8 up, 8 in 2024-09-11T08:25:21.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:21 smithi150 ceph-mon[39612]: 3.4 scrub starts 2024-09-11T08:25:21.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:21 smithi150 ceph-mon[39612]: Health check update: Degraded data redundancy: 7980/73572 objects degraded (10.847%), 2 pgs degraded (PG_DEGRADED) 2024-09-11T08:25:21.783 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:21.822 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:21.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:21.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:21.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:21 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:25:21.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:21 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:25:21.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:21 smithi181 ceph-mon[32701]: osdmap e363: 8 total, 8 up, 8 in 2024-09-11T08:25:21.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:21 smithi181 ceph-mon[32701]: 3.4 scrub starts 2024-09-11T08:25:21.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:21 smithi181 ceph-mon[32701]: Health check update: Degraded data redundancy: 7980/73572 objects degraded (10.847%), 2 pgs degraded (PG_DEGRADED) 2024-09-11T08:25:21.923 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:25:21.960 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:21.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:21.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:25:22.013 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-11T08:25:22.062 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:25:22.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:22.094 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:22.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:25:22.195 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:22.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:22.225 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:22.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:22.288 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T08:25:22.326 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:22.359 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:22.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:22.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:22.461 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:25:22.482 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:22.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:22.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:25:22.584 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:22.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:22.622 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:22.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:22.723 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:25:22.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:22 smithi150 ceph-mon[29276]: pgmap v810: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 1.4 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 777 op/s; 7980/70929 objects degraded (11.251%); 0 B/s, 154 objects/s recovering 2024-09-11T08:25:22.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:22 smithi150 ceph-mon[29276]: 4.7 scrub starts 2024-09-11T08:25:22.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:22 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:25:22.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:22 smithi150 ceph-mon[39612]: pgmap v810: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 1.4 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 777 op/s; 7980/70929 objects degraded (11.251%); 0 B/s, 154 objects/s recovering 2024-09-11T08:25:22.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:22 smithi150 ceph-mon[39612]: 4.7 scrub starts 2024-09-11T08:25:22.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:22 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:25:22.753 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:22.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:22.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:25:22.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:22 smithi181 ceph-mon[32701]: pgmap v810: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 1.4 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 777 op/s; 7980/70929 objects degraded (11.251%); 0 B/s, 154 objects/s recovering 2024-09-11T08:25:22.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:22 smithi181 ceph-mon[32701]: 4.7 scrub starts 2024-09-11T08:25:22.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:22 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:25:22.855 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:22.883 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:22.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:22.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:22.985 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:25:23.014 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:23.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:23.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:25:23.116 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:23.146 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:23.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:23.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:23.248 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:23.269 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:23.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:23.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:23.371 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:23.399 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:23.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:23.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:23.501 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:23.534 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:23.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:23.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:23.636 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:23.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:23.665 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:23.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:23.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:23 smithi150 ceph-mon[29276]: 4.2 scrub starts 2024-09-11T08:25:23.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:23 smithi150 ceph-mon[39612]: 4.2 scrub starts 2024-09-11T08:25:23.766 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:25:23.805 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:23.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:23.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:25:23.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:23 smithi181 ceph-mon[32701]: 4.2 scrub starts 2024-09-11T08:25:23.907 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:25:23.942 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:23.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:23.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:25:24.044 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:24.077 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:24.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:24.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:24.179 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:25:24.200 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:24.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:24.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:25:24.302 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:24.331 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:24.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:24.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:24.432 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:24.463 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:24.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:24.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:24.565 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:25:24.603 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:24.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:24.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:25:24.705 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:24.726 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:24.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:24.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:24.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:24 smithi150 ceph-mon[29276]: pgmap v811: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 1.4 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 758 B/s rd, 0 B/s wr, 691 op/s; 7980/70929 objects degraded (11.251%); 0 B/s, 137 objects/s recovering 2024-09-11T08:25:24.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:24 smithi150 ceph-mon[39612]: pgmap v811: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 1.4 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 758 B/s rd, 0 B/s wr, 691 op/s; 7980/70929 objects degraded (11.251%); 0 B/s, 137 objects/s recovering 2024-09-11T08:25:24.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:24 smithi181 ceph-mon[32701]: pgmap v811: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 1.4 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 758 B/s rd, 0 B/s wr, 691 op/s; 7980/70929 objects degraded (11.251%); 0 B/s, 137 objects/s recovering 2024-09-11T08:25:24.827 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:24.858 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:24.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:24.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:24.960 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:25:24.993 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:24.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:24.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:25:25.095 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:25.128 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:25.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:25.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:25.230 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:25:25.251 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:25.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:25.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:25:25.352 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:25:25.382 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:25.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:25.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:25:25.420 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:25:25 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:25:25] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:25:25.484 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:25.513 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:25.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:25.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:25.615 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:25.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:25.650 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:25.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:25.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:25 smithi150 ceph-mon[29276]: 3.1c scrub starts 2024-09-11T08:25:25.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:25 smithi150 ceph-mon[39612]: 3.1c scrub starts 2024-09-11T08:25:25.752 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:25:25.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:25.792 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:25.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:25:25.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:25 smithi181 ceph-mon[32701]: 3.1c scrub starts 2024-09-11T08:25:25.892 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:25.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:25.933 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:25.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:26.034 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:25:26.055 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:26.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:26.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:25:26.157 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:25:26.193 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:26.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:26.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:25:26.295 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:26.335 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:26.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:26.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:26.437 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:25:26.473 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:26.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:26.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:25:26.575 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:26.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:26.612 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:26.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:26.713 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:25:26.737 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:26.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:26.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:25:26.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:26 smithi150 ceph-mon[29276]: pgmap v812: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 1.2 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 7980/57291 objects degraded (13.929%); 0 B/s, 238 objects/s recovering 2024-09-11T08:25:26.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:26 smithi150 ceph-mon[29276]: 5.19 scrub starts 2024-09-11T08:25:26.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:26 smithi150 ceph-mon[29276]: Health check update: Degraded data redundancy: 7980/57291 objects degraded (13.929%), 2 pgs degraded (PG_DEGRADED) 2024-09-11T08:25:26.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:26 smithi150 ceph-mon[39612]: pgmap v812: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 1.2 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 7980/57291 objects degraded (13.929%); 0 B/s, 238 objects/s recovering 2024-09-11T08:25:26.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:26 smithi150 ceph-mon[39612]: 5.19 scrub starts 2024-09-11T08:25:26.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:26 smithi150 ceph-mon[39612]: Health check update: Degraded data redundancy: 7980/57291 objects degraded (13.929%), 2 pgs degraded (PG_DEGRADED) 2024-09-11T08:25:26.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:26 smithi181 ceph-mon[32701]: pgmap v812: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 1.2 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 7980/57291 objects degraded (13.929%); 0 B/s, 238 objects/s recovering 2024-09-11T08:25:26.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:26 smithi181 ceph-mon[32701]: 5.19 scrub starts 2024-09-11T08:25:26.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:26 smithi181 ceph-mon[32701]: Health check update: Degraded data redundancy: 7980/57291 objects degraded (13.929%), 2 pgs degraded (PG_DEGRADED) 2024-09-11T08:25:26.839 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:26.870 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:26.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:26.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:26.972 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:25:27.010 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:27.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:27.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:25:27.112 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:27.147 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:27.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:27.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:27.250 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:25:27.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:27.272 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:27.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:25:27.373 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:27.408 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:27.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:27.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:27.509 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:25:27.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:27.544 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:27.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:25:27.645 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:25:27.690 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:27.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:27.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:25:27.792 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:25:27.830 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:27.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:27.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:25:27.932 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:27.956 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:27.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:27.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:28.058 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:28.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:28.093 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:28.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:28.194 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:28.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:28.241 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:28.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:28.342 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:25:28.379 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:28.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:28.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:25:28.481 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:28.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:28.522 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:28.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:28.623 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:28.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:28.656 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:28.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:28.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:28 smithi150 ceph-mon[29276]: pgmap v813: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 1.2 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 587 B/s rd, 0 B/s wr, 754 op/s; 7980/57291 objects degraded (13.929%); 0 B/s, 176 objects/s recovering 2024-09-11T08:25:28.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:28 smithi150 ceph-mon[39612]: pgmap v813: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 1.2 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 587 B/s rd, 0 B/s wr, 754 op/s; 7980/57291 objects degraded (13.929%); 0 B/s, 176 objects/s recovering 2024-09-11T08:25:28.739 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:25:28 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:25:28] "GET /metrics HTTP/1.1" 200 224445 "" "Prometheus/2.43.0" 2024-09-11T08:25:28.757 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:25:28.779 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:28 smithi181 ceph-mon[32701]: pgmap v813: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 1.2 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 587 B/s rd, 0 B/s wr, 754 op/s; 7980/57291 objects degraded (13.929%); 0 B/s, 176 objects/s recovering 2024-09-11T08:25:28.791 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:28.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:28.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:25:28.894 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:28.924 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:28.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:28.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:29.026 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:29.060 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:29.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:29.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:29.073 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:25:28 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:25:29.162 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:25:29.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:29.204 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:29.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:25:29.306 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:25:29.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:29.345 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:29.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:25:29.446 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:25:29.484 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:29.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:29.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:25:29.585 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:29.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:29.620 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:29.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:29.684 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:29 smithi150 ceph-mon[29276]: 6.d scrub starts 2024-09-11T08:25:29.684 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:29 smithi150 ceph-mon[39612]: 6.d scrub starts 2024-09-11T08:25:29.721 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:29.743 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:29.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:29.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:29.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:29 smithi181 ceph-mon[32701]: 6.d scrub starts 2024-09-11T08:25:29.845 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:29.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:29.885 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:29.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:29.986 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:25:30.019 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:30.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:30.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:25:30.120 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:30.155 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:30.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:30.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:30.257 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:25:30.291 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:30.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:30.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:25:30.393 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:30.423 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:30.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:30.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:30.525 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:25:30.561 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:30.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:30.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:25:30.663 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:25:30.689 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:30.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:30.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:25:30.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:30 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:25:30.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:30 smithi150 ceph-mon[29276]: pgmap v814: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 955 MiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 906 op/s; 7980/46386 objects degraded (17.203%); 0 B/s, 227 objects/s recovering 2024-09-11T08:25:30.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:30 smithi150 ceph-mon[29276]: 6.1a scrub starts 2024-09-11T08:25:30.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:30 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:25:30.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:30 smithi150 ceph-mon[39612]: pgmap v814: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 955 MiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 906 op/s; 7980/46386 objects degraded (17.203%); 0 B/s, 227 objects/s recovering 2024-09-11T08:25:30.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:30 smithi150 ceph-mon[39612]: 6.1a scrub starts 2024-09-11T08:25:30.791 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:30.815 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:30.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:30.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:30.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:30 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:25:30.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:30 smithi181 ceph-mon[32701]: pgmap v814: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 955 MiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 906 op/s; 7980/46386 objects degraded (17.203%); 0 B/s, 227 objects/s recovering 2024-09-11T08:25:30.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:30 smithi181 ceph-mon[32701]: 6.1a scrub starts 2024-09-11T08:25:30.917 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:25:30.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:30.942 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:30.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:25:31.043 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:31.067 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:31.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:31.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:31.169 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:31.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:31.192 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:31.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:31.293 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:25:31.314 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:31.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:31.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:25:31.416 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:31.439 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:31.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:31.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:31.541 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:31.568 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:31.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:31.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:31.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:31 smithi181 ceph-mon[32701]: osdmap e364: 8 total, 8 up, 8 in 2024-09-11T08:25:31.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:31 smithi181 ceph-mon[32701]: Health check update: Degraded data redundancy: 7980/46386 objects degraded (17.203%), 2 pgs degraded (PG_DEGRADED) 2024-09-11T08:25:31.670 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:25:31.705 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:31.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:31.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:25:31.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:31 smithi150 ceph-mon[29276]: osdmap e364: 8 total, 8 up, 8 in 2024-09-11T08:25:31.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:31 smithi150 ceph-mon[29276]: Health check update: Degraded data redundancy: 7980/46386 objects degraded (17.203%), 2 pgs degraded (PG_DEGRADED) 2024-09-11T08:25:31.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:31 smithi150 ceph-mon[39612]: osdmap e364: 8 total, 8 up, 8 in 2024-09-11T08:25:31.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:31 smithi150 ceph-mon[39612]: Health check update: Degraded data redundancy: 7980/46386 objects degraded (17.203%), 2 pgs degraded (PG_DEGRADED) 2024-09-11T08:25:31.807 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:31.839 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:31.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:31.841 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:31.942 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:31.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:31.981 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:31.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:32.082 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:25:32.125 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:32.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:32.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:25:32.227 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:32.265 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:32.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:32.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:32.367 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:32.399 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:32.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:32.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:32.501 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:25:32.540 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:32.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:32.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:25:32.642 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:25:32.688 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:32.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:32.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:25:32.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:32 smithi150 ceph-mon[29276]: pgmap v816: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 901 MiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 904 op/s; 7980/43815 objects degraded (18.213%); 0 B/s, 229 objects/s recovering 2024-09-11T08:25:32.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:32 smithi150 ceph-mon[29276]: osdmap e365: 8 total, 8 up, 8 in 2024-09-11T08:25:32.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:32 smithi150 ceph-mon[39612]: pgmap v816: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 901 MiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 904 op/s; 7980/43815 objects degraded (18.213%); 0 B/s, 229 objects/s recovering 2024-09-11T08:25:32.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:32 smithi150 ceph-mon[39612]: osdmap e365: 8 total, 8 up, 8 in 2024-09-11T08:25:32.790 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:25:32.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:32 smithi181 ceph-mon[32701]: pgmap v816: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 901 MiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 904 op/s; 7980/43815 objects degraded (18.213%); 0 B/s, 229 objects/s recovering 2024-09-11T08:25:32.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:32 smithi181 ceph-mon[32701]: osdmap e365: 8 total, 8 up, 8 in 2024-09-11T08:25:32.825 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:32.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:32.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:25:32.927 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:32.963 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:32.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:32.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:33.065 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:33.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:33.102 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:33.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:33.204 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:25:33.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:33.248 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:33.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:25:33.349 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:33.391 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:33.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:33.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:33.493 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:33.527 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:33.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:33.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:33.629 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:33.668 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:33.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:33.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:33.770 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:25:33.807 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:33.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:33.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:25:33.856 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:25:33.889 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:25:33.909 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:25:33.959 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:33.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:33.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:25:34.061 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:34.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:34.097 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:34.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:34.198 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:34.240 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:34.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:34.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:34.342 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:34.375 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:34.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:34.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:34.478 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:34.520 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:34.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:34.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:34.622 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:25:34.653 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:34.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:34.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:25:34.695 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:34 smithi150 ceph-mon[29276]: pgmap v818: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 901 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 561 op/s; 7980/43815 objects degraded (18.213%); 0 B/s, 165 objects/s recovering 2024-09-11T08:25:34.695 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:34 smithi150 ceph-mon[39612]: pgmap v818: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 901 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 561 op/s; 7980/43815 objects degraded (18.213%); 0 B/s, 165 objects/s recovering 2024-09-11T08:25:34.725 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:25:34.725 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:25:34.755 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:25:34.762 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:25:34.763 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:25:34.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:34.792 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:34.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:25:34.792 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:25:34.792 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:25:34.820 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:25:34.820 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:25:34.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:34 smithi181 ceph-mon[32701]: pgmap v818: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 901 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 561 op/s; 7980/43815 objects degraded (18.213%); 0 B/s, 165 objects/s recovering 2024-09-11T08:25:34.858 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:25:34.859 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:25:34.893 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:34.894 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:25:34.894 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:25:34.930 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:25:34.930 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:25:34.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:34.939 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:34.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:34.971 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:25:34.971 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:25:35.040 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:35.078 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:35.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:35.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:35.180 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:35.212 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:35.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:35.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:35.314 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:35.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:35.352 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:35.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:35.453 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:35.461 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:25:35 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:25:35] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:25:35.484 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:35.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:35.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:35.586 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:35.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:35.625 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:35.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:35.726 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:25:35.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:35 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2668940241' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:25:35.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:35 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2668940241' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:25:35.767 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:35.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:35.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:25:35.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:35 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2668940241' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:25:35.870 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:25:35.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:35.898 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:35.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:25:36.000 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:25:36.028 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:36.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:36.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:25:36.130 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:25:36.164 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:36.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:36.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:25:36.266 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:25:36.298 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:36.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:36.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:25:36.400 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:25:36.433 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:36.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:36.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:25:36.535 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:25:36.566 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:36.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:36.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:25:36.667 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:36.707 INFO:teuthology.orchestra.run.smithi150.stderr:reweighted osd.2 to 1 (10000) 2024-09-11T08:25:36.708 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:36.708 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:36 smithi150 ceph-mon[29276]: pgmap v819: 148 pgs: 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 144 active+clean; 650 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.06k op/s; 3975/31776 objects degraded (12.509%); 0 B/s, 349 objects/s recovering 2024-09-11T08:25:36.708 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:36 smithi150 ceph-mon[29276]: 5.15 scrub starts 2024-09-11T08:25:36.708 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:36 smithi150 ceph-mon[29276]: Health check update: Degraded data redundancy: 3975/31776 objects degraded (12.509%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-11T08:25:36.708 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:36 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2935406787' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-11T08:25:36.708 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:36 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-11T08:25:36.709 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:36 smithi150 ceph-mon[39612]: pgmap v819: 148 pgs: 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 144 active+clean; 650 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.06k op/s; 3975/31776 objects degraded (12.509%); 0 B/s, 349 objects/s recovering 2024-09-11T08:25:36.709 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:36 smithi150 ceph-mon[39612]: 5.15 scrub starts 2024-09-11T08:25:36.709 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:36 smithi150 ceph-mon[39612]: Health check update: Degraded data redundancy: 3975/31776 objects degraded (12.509%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-11T08:25:36.709 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:36 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2935406787' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-11T08:25:36.709 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:36 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-11T08:25:36.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:36.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:36.811 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:36.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:36 smithi181 ceph-mon[32701]: pgmap v819: 148 pgs: 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 144 active+clean; 650 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.06k op/s; 3975/31776 objects degraded (12.509%); 0 B/s, 349 objects/s recovering 2024-09-11T08:25:36.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:36 smithi181 ceph-mon[32701]: 5.15 scrub starts 2024-09-11T08:25:36.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:36 smithi181 ceph-mon[32701]: Health check update: Degraded data redundancy: 3975/31776 objects degraded (12.509%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-11T08:25:36.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:36 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2935406787' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-11T08:25:36.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:36 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-11T08:25:36.842 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:36.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:36.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:36.944 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:36.972 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:36.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:36.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:37.074 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:37.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:37.115 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:37.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:37.216 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:37.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:37.254 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:37.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:37.355 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:37.395 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:37.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:37.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:37.497 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:25:37.535 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:37.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:37.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:25:37.637 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:25:37.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:37.675 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:37.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:25:37.776 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:37.813 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:37.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:37.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:37.827 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:37 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-11T08:25:37.827 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:37 smithi150 ceph-mon[29276]: osdmap e366: 8 total, 8 up, 8 in 2024-09-11T08:25:37.827 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:37 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:25:37.827 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:37 smithi150 ceph-mon[29276]: pgmap v821: 148 pgs: 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 144 active+clean; 650 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 767 B/s rd, 0 B/s wr, 602 op/s; 3975/31776 objects degraded (12.509%); 0 B/s, 220 objects/s recovering 2024-09-11T08:25:37.828 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:37 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-11T08:25:37.828 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:37 smithi150 ceph-mon[39612]: osdmap e366: 8 total, 8 up, 8 in 2024-09-11T08:25:37.828 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:37 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:25:37.828 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:37 smithi150 ceph-mon[39612]: pgmap v821: 148 pgs: 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 144 active+clean; 650 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 767 B/s rd, 0 B/s wr, 602 op/s; 3975/31776 objects degraded (12.509%); 0 B/s, 220 objects/s recovering 2024-09-11T08:25:37.915 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:25:37.955 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:37.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:37.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:25:38.057 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:38.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:37 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-11T08:25:38.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:37 smithi181 ceph-mon[32701]: osdmap e366: 8 total, 8 up, 8 in 2024-09-11T08:25:38.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:37 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:25:38.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:37 smithi181 ceph-mon[32701]: pgmap v821: 148 pgs: 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 144 active+clean; 650 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 767 B/s rd, 0 B/s wr, 602 op/s; 3975/31776 objects degraded (12.509%); 0 B/s, 220 objects/s recovering 2024-09-11T08:25:38.092 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:38.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:38.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:38.194 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:25:38.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:38.241 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:38.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:25:38.342 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:25:38.368 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:38.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:38.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:25:38.470 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:38.498 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:38.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:38.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:38.600 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:38.635 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:38.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:38.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:38.737 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:38.738 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:25:38 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:25:38] "GET /metrics HTTP/1.1" 200 224445 "" "Prometheus/2.43.0" 2024-09-11T08:25:38.766 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:38.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:38.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:38.868 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:38.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:38.904 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:38.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:39.005 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:39.038 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:39.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:39.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:39.073 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:25:38 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:25:39.121 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd reweight 3 1 2024-09-11T08:25:39.140 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:39.185 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:39.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:39.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:39.287 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:25:39.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:39.323 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:39.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:25:39.424 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:39.457 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:39.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:39.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:39.559 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:25:39.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:39.596 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:39.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:25:39.697 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:39.733 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:39.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:39.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:39.835 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:39.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:39.877 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:39.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:39.978 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:40.016 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:40.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:40.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:40.118 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:25:40.153 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:40.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:40.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:25:40.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:40 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:25:40.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:40 smithi150 ceph-mon[29276]: pgmap v822: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 144 active+clean; 402 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 999 op/s; 3975/19851 objects degraded (20.024%); 0 B/s, 263 objects/s recovering 2024-09-11T08:25:40.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:40 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:25:40.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:40 smithi150 ceph-mon[39612]: pgmap v822: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 144 active+clean; 402 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 999 op/s; 3975/19851 objects degraded (20.024%); 0 B/s, 263 objects/s recovering 2024-09-11T08:25:40.254 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:40.296 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:40.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:40.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:40.398 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:25:40.420 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-11T08:25:40.420 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 20, 2024-09-11T08:25:40.421 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 600, 2024-09-11T08:25:40.421 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [ 2024-09-11T08:25:40.421 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.421 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15369.0:96087 9.a 9:5eafb6c5:::benchmark_data_smithi150_133616_object96086:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e260)", 2024-09-11T08:25:40.421 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:19:08.596504+0000", 2024-09-11T08:25:40.421 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 391.82179931899998, 2024-09-11T08:25:40.421 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.0948638279999998, 2024-09-11T08:25:40.421 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:25:40.421 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-09-11T08:25:40.421 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:25:40.421 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15369", 2024-09-11T08:25:40.421 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:25:40.422 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 96087 2024-09-11T08:25:40.422 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.422 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:25:40.422 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.422 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:25:40.422 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.596504+0000", 2024-09-11T08:25:40.422 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:25:40.422 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.422 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.422 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:25:40.422 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.596504+0000", 2024-09-11T08:25:40.422 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:25:40.422 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.422 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.423 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:25:40.423 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.596503+0000", 2024-09-11T08:25:40.423 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:25:40.423 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.423 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.423 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:25:40.423 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.596526+0000", 2024-09-11T08:25:40.423 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.3686e-05 2024-09-11T08:25:40.423 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.423 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.423 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:25:40.423 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.596537+0000", 2024-09-11T08:25:40.424 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0324e-05 2024-09-11T08:25:40.424 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.424 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.424 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:25:40.424 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.596542+0000", 2024-09-11T08:25:40.424 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.6729999999999997e-06 2024-09-11T08:25:40.424 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.424 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.424 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:25:40.424 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.601303+0000", 2024-09-11T08:25:40.424 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0047603510000000003 2024-09-11T08:25:40.424 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.425 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.425 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:25:40.425 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.601314+0000", 2024-09-11T08:25:40.425 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1639e-05 2024-09-11T08:25:40.425 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.425 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.425 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:25:40.425 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.591260+0000", 2024-09-11T08:25:40.425 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.98994515299999997 2024-09-11T08:25:40.425 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.426 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.426 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:25:40.426 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.591270+0000", 2024-09-11T08:25:40.426 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0226e-05 2024-09-11T08:25:40.426 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.426 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.426 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:25:40.426 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.689068+0000", 2024-09-11T08:25:40.426 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.097797911 2024-09-11T08:25:40.426 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.426 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.426 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:25:40.427 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.691368+0000", 2024-09-11T08:25:40.427 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0023004119999999999 2024-09-11T08:25:40.427 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:25:40.427 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:25:40.427 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:25:40.427 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.427 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.427 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15369.0:96116 9.a 9:59cffadf:::benchmark_data_smithi150_133616_object96115:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-11T08:25:40.427 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:19:08.628434+0000", 2024-09-11T08:25:40.427 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 391.78986920300002, 2024-09-11T08:25:40.427 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.082338676, 2024-09-11T08:25:40.427 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:25:40.428 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:25:40.428 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:25:40.428 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15369", 2024-09-11T08:25:40.428 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:25:40.428 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 96116 2024-09-11T08:25:40.428 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.428 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:25:40.428 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.428 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:25:40.428 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.628434+0000", 2024-09-11T08:25:40.428 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:25:40.428 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.428 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.429 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:25:40.429 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.628434+0000", 2024-09-11T08:25:40.429 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:25:40.429 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.429 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.429 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:25:40.429 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.628432+0000", 2024-09-11T08:25:40.429 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:25:40.429 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.429 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.429 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:25:40.429 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.628461+0000", 2024-09-11T08:25:40.429 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.8496999999999998e-05 2024-09-11T08:25:40.430 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.430 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.430 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:25:40.430 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.628471+0000", 2024-09-11T08:25:40.430 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0362e-05 2024-09-11T08:25:40.430 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.430 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.430 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:25:40.430 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.628477+0000", 2024-09-11T08:25:40.430 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.1859999999999997e-06 2024-09-11T08:25:40.430 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.430 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.431 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:25:40.431 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.628547+0000", 2024-09-11T08:25:40.431 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.9460000000000002e-05 2024-09-11T08:25:40.431 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.431 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.431 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:25:40.431 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.628558+0000", 2024-09-11T08:25:40.431 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0811e-05 2024-09-11T08:25:40.431 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.431 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.431 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:25:40.431 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.591329+0000", 2024-09-11T08:25:40.432 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.96277097700000003 2024-09-11T08:25:40.432 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.432 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.432 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:25:40.432 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.591336+0000", 2024-09-11T08:25:40.432 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.5129999999999999e-06 2024-09-11T08:25:40.432 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.432 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.432 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:25:40.432 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.691674+0000", 2024-09-11T08:25:40.432 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.100337771 2024-09-11T08:25:40.432 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.432 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.433 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:25:40.433 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.692428+0000", 2024-09-11T08:25:40.433 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00075426699999999998 2024-09-11T08:25:40.433 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.433 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.433 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 3,5,7", 2024-09-11T08:25:40.433 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.692745+0000", 2024-09-11T08:25:40.433 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000317003 2024-09-11T08:25:40.433 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.433 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.433 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:25:40.433 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.702335+0000", 2024-09-11T08:25:40.433 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0095893450000000009 2024-09-11T08:25:40.433 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.433 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.434 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:25:40.434 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.702358+0000", 2024-09-11T08:25:40.434 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.3431e-05 2024-09-11T08:25:40.434 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.434 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.434 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:25:40.434 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.702626+0000", 2024-09-11T08:25:40.434 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00026750599999999998 2024-09-11T08:25:40.434 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.434 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.434 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:25:40.434 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.710136+0000", 2024-09-11T08:25:40.434 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0075106670000000004 2024-09-11T08:25:40.435 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.435 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.435 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:25:40.435 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.710657+0000", 2024-09-11T08:25:40.435 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00052111099999999999 2024-09-11T08:25:40.435 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.435 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.435 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:25:40.435 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.710773+0000", 2024-09-11T08:25:40.435 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000115777 2024-09-11T08:25:40.435 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:25:40.435 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:25:40.435 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:25:40.435 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.436 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.436 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15369.0:96117 9.f 9:fa88c3cd:::benchmark_data_smithi150_133616_object96116:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-11T08:25:40.436 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:19:08.628715+0000", 2024-09-11T08:25:40.437 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 391.78958885399999, 2024-09-11T08:25:40.437 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.2202831299999999, 2024-09-11T08:25:40.437 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:25:40.437 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:25:40.437 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:25:40.437 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15369", 2024-09-11T08:25:40.437 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:25:40.437 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 96117 2024-09-11T08:25:40.437 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.437 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:25:40.437 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.437 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:25:40.437 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.628715+0000", 2024-09-11T08:25:40.438 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:25:40.438 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.438 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.438 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:25:40.438 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.628715+0000", 2024-09-11T08:25:40.438 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:25:40.438 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.438 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.438 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:25:40.438 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.628714+0000", 2024-09-11T08:25:40.438 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:25:40.438 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.438 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.438 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:25:40.439 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.628738+0000", 2024-09-11T08:25:40.439 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.4105999999999999e-05 2024-09-11T08:25:40.439 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.439 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.439 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:25:40.439 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.628752+0000", 2024-09-11T08:25:40.439 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3909e-05 2024-09-11T08:25:40.439 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.440 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.440 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:25:40.440 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.628758+0000", 2024-09-11T08:25:40.440 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.9830000000000001e-06 2024-09-11T08:25:40.440 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.440 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.440 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:25:40.440 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.628814+0000", 2024-09-11T08:25:40.440 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.6297e-05 2024-09-11T08:25:40.440 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.440 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.440 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:25:40.440 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.628820+0000", 2024-09-11T08:25:40.441 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.4200000000000004e-06 2024-09-11T08:25:40.441 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.441 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.441 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:25:40.441 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.601901+0000", 2024-09-11T08:25:40.441 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.97308029200000001 2024-09-11T08:25:40.441 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.441 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.441 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:25:40.441 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.601916+0000", 2024-09-11T08:25:40.441 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4888e-05 2024-09-11T08:25:40.441 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.441 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.442 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:25:40.442 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.787726+0000", 2024-09-11T08:25:40.442 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.185809914 2024-09-11T08:25:40.442 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.442 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.442 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:25:40.442 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.790484+0000", 2024-09-11T08:25:40.442 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0027581490000000001 2024-09-11T08:25:40.444 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.444 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.444 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 2,3,5", 2024-09-11T08:25:40.444 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.790769+0000", 2024-09-11T08:25:40.444 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00028509599999999998 2024-09-11T08:25:40.444 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.444 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.444 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:25:40.444 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.835253+0000", 2024-09-11T08:25:40.444 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.044484502000000002 2024-09-11T08:25:40.445 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.445 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.445 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:25:40.445 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.835360+0000", 2024-09-11T08:25:40.445 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00010631799999999999 2024-09-11T08:25:40.445 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.445 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.445 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:25:40.445 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.835369+0000", 2024-09-11T08:25:40.445 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.55e-06 2024-09-11T08:25:40.445 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.445 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.445 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:25:40.446 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.848835+0000", 2024-09-11T08:25:40.446 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.013465886999999999 2024-09-11T08:25:40.446 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.446 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.446 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:25:40.446 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.848891+0000", 2024-09-11T08:25:40.446 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.5890999999999997e-05 2024-09-11T08:25:40.446 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.446 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.446 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:25:40.446 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.848998+0000", 2024-09-11T08:25:40.446 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000106939 2024-09-11T08:25:40.446 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:25:40.446 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:25:40.446 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:25:40.447 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.447 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.447 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.24320.0:1804 5.15 5:a93a5511:::notify.2:head [watch reconnect cookie 94783556101344 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-11T08:25:40.447 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:19:08.688799+0000", 2024-09-11T08:25:40.447 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 391.729504147, 2024-09-11T08:25:40.447 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.92763364500000001, 2024-09-11T08:25:40.447 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:25:40.447 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "started", 2024-09-11T08:25:40.447 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:25:40.447 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.24320", 2024-09-11T08:25:40.447 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1503848706", 2024-09-11T08:25:40.447 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 1804 2024-09-11T08:25:40.447 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.447 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:25:40.447 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.447 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:25:40.448 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.688799+0000", 2024-09-11T08:25:40.448 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:25:40.448 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.448 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.448 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:25:40.448 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.688799+0000", 2024-09-11T08:25:40.448 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:25:40.448 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.448 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.448 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:25:40.448 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.688797+0000", 2024-09-11T08:25:40.448 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:25:40.448 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.448 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.448 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:25:40.449 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.688807+0000", 2024-09-11T08:25:40.449 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.1889999999999996e-06 2024-09-11T08:25:40.449 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.449 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.449 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:25:40.449 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.688826+0000", 2024-09-11T08:25:40.449 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.9813000000000002e-05 2024-09-11T08:25:40.449 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.449 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.449 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:25:40.449 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.688834+0000", 2024-09-11T08:25:40.449 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.7819999999999994e-06 2024-09-11T08:25:40.449 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.450 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.450 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:25:40.450 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.688883+0000", 2024-09-11T08:25:40.450 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.8806e-05 2024-09-11T08:25:40.450 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.450 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.450 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:25:40.450 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.688890+0000", 2024-09-11T08:25:40.450 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.9280000000000004e-06 2024-09-11T08:25:40.450 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.450 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.450 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:25:40.450 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.612372+0000", 2024-09-11T08:25:40.450 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.92348205699999997 2024-09-11T08:25:40.450 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.451 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.451 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for degraded object", 2024-09-11T08:25:40.451 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.612676+0000", 2024-09-11T08:25:40.451 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00030383 2024-09-11T08:25:40.451 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.451 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.451 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:25:40.451 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.613796+0000", 2024-09-11T08:25:40.451 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0011199339999999999 2024-09-11T08:25:40.451 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.451 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.451 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:25:40.451 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.613872+0000", 2024-09-11T08:25:40.451 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.6050000000000005e-05 2024-09-11T08:25:40.452 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.452 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.452 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:25:40.452 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.616433+0000", 2024-09-11T08:25:40.452 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0025613749999999998 2024-09-11T08:25:40.452 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:25:40.452 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:25:40.452 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:25:40.452 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.452 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.452 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.15369.0:212763 9.3 e298/246 9:ce43ae85:::benchmark_data_smithi150_133616_object97157:head v 298'15904, mlcod=298'15902)", 2024-09-11T08:25:40.452 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:21:06.533414+0000", 2024-09-11T08:25:40.452 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 273.884889816, 2024-09-11T08:25:40.453 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.27374648499999998, 2024-09-11T08:25:40.453 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:25:40.453 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:25:40.453 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:25:40.453 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.453 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:25:40.453 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.533414+0000", 2024-09-11T08:25:40.453 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:25:40.453 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.453 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.453 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:25:40.453 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.533414+0000", 2024-09-11T08:25:40.453 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:25:40.453 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.454 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.454 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:25:40.454 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.533413+0000", 2024-09-11T08:25:40.454 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:25:40.454 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.454 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.454 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:25:40.454 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.533420+0000", 2024-09-11T08:25:40.454 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.2239999999999998e-06 2024-09-11T08:25:40.454 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.454 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.454 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:25:40.454 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.618921+0000", 2024-09-11T08:25:40.455 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.085500823000000004 2024-09-11T08:25:40.455 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.455 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.455 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:25:40.455 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.618927+0000", 2024-09-11T08:25:40.455 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.4979999999999997e-06 2024-09-11T08:25:40.455 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.455 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.455 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:25:40.455 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.618972+0000", 2024-09-11T08:25:40.455 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.5472000000000001e-05 2024-09-11T08:25:40.455 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.456 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.456 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:25:40.456 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.618994+0000", 2024-09-11T08:25:40.456 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.1676e-05 2024-09-11T08:25:40.456 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.456 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.456 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:25:40.456 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.807131+0000", 2024-09-11T08:25:40.456 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.18813698000000001 2024-09-11T08:25:40.456 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.456 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.456 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:25:40.456 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.807160+0000", 2024-09-11T08:25:40.457 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.9496999999999999e-05 2024-09-11T08:25:40.457 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:25:40.457 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:25:40.457 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:25:40.457 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.457 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.457 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.15369.0:212766 9.3 e298/246 9:ca00ffc2:::benchmark_data_smithi150_133616_object97160:head v 298'15905, mlcod=298'15902)", 2024-09-11T08:25:40.457 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:21:06.618944+0000", 2024-09-11T08:25:40.457 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 273.79935959300002, 2024-09-11T08:25:40.457 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.19080667700000001, 2024-09-11T08:25:40.457 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:25:40.457 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:25:40.457 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:25:40.457 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.458 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:25:40.458 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.618944+0000", 2024-09-11T08:25:40.458 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:25:40.458 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.458 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.458 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:25:40.458 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.618944+0000", 2024-09-11T08:25:40.458 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:25:40.458 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.458 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.458 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:25:40.458 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.618943+0000", 2024-09-11T08:25:40.458 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:25:40.459 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.459 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.459 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:25:40.459 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.618951+0000", 2024-09-11T08:25:40.459 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.6199999999999999e-06 2024-09-11T08:25:40.459 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.459 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.459 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:25:40.459 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.618957+0000", 2024-09-11T08:25:40.459 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.6189999999999996e-06 2024-09-11T08:25:40.459 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.459 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.459 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:25:40.460 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.618961+0000", 2024-09-11T08:25:40.460 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.2329999999999998e-06 2024-09-11T08:25:40.460 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.460 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.460 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:25:40.460 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.621622+0000", 2024-09-11T08:25:40.460 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0026617519999999999 2024-09-11T08:25:40.460 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.460 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.460 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:25:40.460 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.621639+0000", 2024-09-11T08:25:40.460 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.6543000000000001e-05 2024-09-11T08:25:40.460 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.461 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.461 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:25:40.461 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.809723+0000", 2024-09-11T08:25:40.461 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.188084476 2024-09-11T08:25:40.461 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.461 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.461 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:25:40.461 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.809751+0000", 2024-09-11T08:25:40.461 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.7189999999999999e-05 2024-09-11T08:25:40.461 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:25:40.461 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:25:40.461 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:25:40.461 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.462 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.462 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.15369.0:212770 9.e e298/246 9:79e6cbd3:::benchmark_data_smithi150_133616_object97164:head v 298'13227, mlcod=298'13225)", 2024-09-11T08:25:40.462 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:21:06.618975+0000", 2024-09-11T08:25:40.462 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 273.79932869700002, 2024-09-11T08:25:40.462 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.18815554900000001, 2024-09-11T08:25:40.462 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:25:40.462 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:25:40.462 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:25:40.462 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.462 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:25:40.462 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.618975+0000", 2024-09-11T08:25:40.462 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:25:40.462 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.463 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.463 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:25:40.463 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.618975+0000", 2024-09-11T08:25:40.463 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:25:40.463 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.463 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.463 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:25:40.463 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.618975+0000", 2024-09-11T08:25:40.463 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:25:40.463 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.463 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.463 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:25:40.463 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.618978+0000", 2024-09-11T08:25:40.463 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.3129999999999999e-06 2024-09-11T08:25:40.464 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.464 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.464 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:25:40.464 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.618983+0000", 2024-09-11T08:25:40.464 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.6e-06 2024-09-11T08:25:40.464 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.464 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.464 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:25:40.464 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.618987+0000", 2024-09-11T08:25:40.464 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.9890000000000003e-06 2024-09-11T08:25:40.464 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.464 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.464 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:25:40.464 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.619438+0000", 2024-09-11T08:25:40.465 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00045184400000000002 2024-09-11T08:25:40.465 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.465 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.465 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:25:40.465 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.619457+0000", 2024-09-11T08:25:40.465 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.8578999999999999e-05 2024-09-11T08:25:40.465 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.465 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.465 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:25:40.465 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.625854+0000", 2024-09-11T08:25:40.465 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0063966759999999996 2024-09-11T08:25:40.465 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.465 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.466 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:25:40.466 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.807130+0000", 2024-09-11T08:25:40.466 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.18127680099999999 2024-09-11T08:25:40.466 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:25:40.466 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:25:40.466 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:25:40.466 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.466 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.466 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.15369.0:212772 9.3 e298/246 9:cfdf609d:::benchmark_data_smithi150_133616_object97166:head v 298'15906, mlcod=298'15902)", 2024-09-11T08:25:40.466 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:21:06.618997+0000", 2024-09-11T08:25:40.466 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 273.79930661899999, 2024-09-11T08:25:40.466 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.190769993, 2024-09-11T08:25:40.466 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:25:40.466 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:25:40.467 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:25:40.467 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.467 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:25:40.467 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.618997+0000", 2024-09-11T08:25:40.467 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:25:40.467 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.467 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.467 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:25:40.467 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.618997+0000", 2024-09-11T08:25:40.467 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:25:40.467 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.467 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.467 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:25:40.468 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.618997+0000", 2024-09-11T08:25:40.468 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967296 2024-09-11T08:25:40.468 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.468 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.468 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:25:40.468 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.619000+0000", 2024-09-11T08:25:40.468 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.3790000000000002e-06 2024-09-11T08:25:40.468 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.468 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.468 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:25:40.468 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.619004+0000", 2024-09-11T08:25:40.468 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.0239999999999999e-06 2024-09-11T08:25:40.468 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.468 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.469 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:25:40.469 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.619007+0000", 2024-09-11T08:25:40.469 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.1e-06 2024-09-11T08:25:40.469 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.469 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.469 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:25:40.469 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.624034+0000", 2024-09-11T08:25:40.469 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.005027129 2024-09-11T08:25:40.469 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.469 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.469 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:25:40.469 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.624052+0000", 2024-09-11T08:25:40.469 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.8068999999999999e-05 2024-09-11T08:25:40.469 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.470 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.470 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:25:40.470 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.809753+0000", 2024-09-11T08:25:40.470 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.18570086899999999 2024-09-11T08:25:40.470 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.470 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.470 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:25:40.470 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.809767+0000", 2024-09-11T08:25:40.470 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3621999999999999e-05 2024-09-11T08:25:40.470 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:25:40.471 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:25:40.471 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:25:40.471 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.471 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.471 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.15369.0:212774 9.6 e298/271 9:6c4ab803:::benchmark_data_smithi150_133616_object97168:head v 298'13385, mlcod=298'13383)", 2024-09-11T08:25:40.471 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:21:06.619006+0000", 2024-09-11T08:25:40.471 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 273.799297276, 2024-09-11T08:25:40.471 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.18817231600000001, 2024-09-11T08:25:40.471 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:25:40.471 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:25:40.471 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:25:40.471 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.471 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:25:40.472 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.619006+0000", 2024-09-11T08:25:40.472 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:25:40.472 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.472 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.472 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:25:40.472 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.619006+0000", 2024-09-11T08:25:40.472 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:25:40.472 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.472 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.472 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:25:40.472 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.619006+0000", 2024-09-11T08:25:40.472 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:25:40.473 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.473 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.473 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:25:40.473 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.619013+0000", 2024-09-11T08:25:40.473 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.8399999999999997e-06 2024-09-11T08:25:40.473 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.473 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.473 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:25:40.473 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.619019+0000", 2024-09-11T08:25:40.473 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.5119999999999997e-06 2024-09-11T08:25:40.473 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.473 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.473 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:25:40.474 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.619023+0000", 2024-09-11T08:25:40.474 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1160000000000001e-06 2024-09-11T08:25:40.474 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.474 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.474 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:25:40.474 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.619452+0000", 2024-09-11T08:25:40.474 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00042851400000000001 2024-09-11T08:25:40.474 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.474 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.474 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:25:40.474 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.619469+0000", 2024-09-11T08:25:40.474 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.7521999999999999e-05 2024-09-11T08:25:40.474 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.474 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.475 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:25:40.475 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.807155+0000", 2024-09-11T08:25:40.475 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.18768555200000001 2024-09-11T08:25:40.475 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.475 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.475 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:25:40.475 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.807179+0000", 2024-09-11T08:25:40.475 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.3921e-05 2024-09-11T08:25:40.475 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:25:40.475 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:25:40.475 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:25:40.475 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.475 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.475 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.15369.0:212776 9.3 e298/246 9:cbebc502:::benchmark_data_smithi150_133616_object97170:head v 298'15907, mlcod=298'15902)", 2024-09-11T08:25:40.475 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:21:06.619011+0000", 2024-09-11T08:25:40.476 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 273.79929257800001, 2024-09-11T08:25:40.476 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.19077562100000001, 2024-09-11T08:25:40.476 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:25:40.476 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:25:40.476 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:25:40.476 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.476 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:25:40.476 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.619011+0000", 2024-09-11T08:25:40.476 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:25:40.476 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.476 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.477 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:25:40.477 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.619011+0000", 2024-09-11T08:25:40.477 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:25:40.477 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.477 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.477 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:25:40.477 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.619011+0000", 2024-09-11T08:25:40.477 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:25:40.477 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.477 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.477 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:25:40.477 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.619019+0000", 2024-09-11T08:25:40.477 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.3860000000000007e-06 2024-09-11T08:25:40.478 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.478 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.478 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:25:40.478 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.619023+0000", 2024-09-11T08:25:40.478 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.2200000000000003e-06 2024-09-11T08:25:40.478 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.478 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.478 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:25:40.478 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.619026+0000", 2024-09-11T08:25:40.478 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.05e-06 2024-09-11T08:25:40.478 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.478 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.478 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:25:40.478 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.626513+0000", 2024-09-11T08:25:40.479 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0074867429999999997 2024-09-11T08:25:40.479 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.479 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.479 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:25:40.479 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.626532+0000", 2024-09-11T08:25:40.479 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.8998999999999999e-05 2024-09-11T08:25:40.479 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.479 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.479 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:25:40.479 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.809769+0000", 2024-09-11T08:25:40.479 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.183236497 2024-09-11T08:25:40.479 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.479 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.479 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:25:40.480 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.809787+0000", 2024-09-11T08:25:40.480 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.7992999999999999e-05 2024-09-11T08:25:40.480 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:25:40.480 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:25:40.480 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:25:40.480 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.480 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.480 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15369.0:212781 9.0 9:0e872012:::benchmark_data_smithi150_133616_object97175:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-11T08:25:40.480 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:21:06.619479+0000", 2024-09-11T08:25:40.480 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 273.79882455799998, 2024-09-11T08:25:40.480 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.18786146400000001, 2024-09-11T08:25:40.480 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:25:40.480 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:25:40.480 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:25:40.480 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15369", 2024-09-11T08:25:40.481 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:25:40.481 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 212781 2024-09-11T08:25:40.481 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.481 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:25:40.481 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.481 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:25:40.481 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.619479+0000", 2024-09-11T08:25:40.482 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:25:40.482 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.482 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.482 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:25:40.482 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.619479+0000", 2024-09-11T08:25:40.482 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:25:40.482 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.482 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.482 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:25:40.482 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.619479+0000", 2024-09-11T08:25:40.482 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:25:40.482 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.482 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.483 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:25:40.483 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.619481+0000", 2024-09-11T08:25:40.483 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.5919999999999999e-06 2024-09-11T08:25:40.483 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.483 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.483 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:25:40.483 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.619487+0000", 2024-09-11T08:25:40.483 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.3299999999999998e-06 2024-09-11T08:25:40.483 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.483 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.483 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:25:40.483 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.619490+0000", 2024-09-11T08:25:40.483 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.8940000000000003e-06 2024-09-11T08:25:40.484 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.484 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.484 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:25:40.484 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.622440+0000", 2024-09-11T08:25:40.484 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0029497939999999999 2024-09-11T08:25:40.484 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.484 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.484 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:25:40.484 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.622558+0000", 2024-09-11T08:25:40.484 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000117433 2024-09-11T08:25:40.484 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.484 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.484 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 2,6", 2024-09-11T08:25:40.484 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.622661+0000", 2024-09-11T08:25:40.485 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00010323 2024-09-11T08:25:40.485 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.485 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.485 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:25:40.485 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.628517+0000", 2024-09-11T08:25:40.485 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0058560330000000001 2024-09-11T08:25:40.485 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.485 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.485 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:25:40.485 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.807157+0000", 2024-09-11T08:25:40.485 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.17864007700000001 2024-09-11T08:25:40.485 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.485 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.485 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:25:40.486 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.807252+0000", 2024-09-11T08:25:40.486 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.4641000000000004e-05 2024-09-11T08:25:40.486 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.486 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.486 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:25:40.486 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.807299+0000", 2024-09-11T08:25:40.486 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.6944999999999999e-05 2024-09-11T08:25:40.486 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.486 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.486 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:25:40.486 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.807341+0000", 2024-09-11T08:25:40.486 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1922999999999998e-05 2024-09-11T08:25:40.487 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:25:40.487 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:25:40.487 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:25:40.487 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.487 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.487 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.15369.0:212782 9.6 e298/271 9:625a7268:::benchmark_data_smithi150_133616_object97176:head v 298'13386, mlcod=298'13383)", 2024-09-11T08:25:40.487 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:21:06.620315+0000", 2024-09-11T08:25:40.487 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 273.79798875199998, 2024-09-11T08:25:40.488 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.18688198, 2024-09-11T08:25:40.488 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:25:40.488 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:25:40.488 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:25:40.488 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.488 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:25:40.488 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.620315+0000", 2024-09-11T08:25:40.488 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:25:40.488 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.488 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.488 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:25:40.488 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.620315+0000", 2024-09-11T08:25:40.488 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:25:40.488 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.489 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.489 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:25:40.489 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.620314+0000", 2024-09-11T08:25:40.489 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:25:40.489 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.489 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.489 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:25:40.489 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.620319+0000", 2024-09-11T08:25:40.489 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.2839999999999997e-06 2024-09-11T08:25:40.489 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.489 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.489 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:25:40.489 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.620328+0000", 2024-09-11T08:25:40.489 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.5359999999999992e-06 2024-09-11T08:25:40.489 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.490 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.490 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:25:40.490 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.620333+0000", 2024-09-11T08:25:40.490 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.079e-06 2024-09-11T08:25:40.490 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.490 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.490 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:25:40.490 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.625927+0000", 2024-09-11T08:25:40.490 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0055937030000000002 2024-09-11T08:25:40.490 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.490 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.490 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:25:40.490 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.625943+0000", 2024-09-11T08:25:40.490 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.6362e-05 2024-09-11T08:25:40.490 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.491 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.491 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:25:40.491 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.807182+0000", 2024-09-11T08:25:40.491 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.18123858400000001 2024-09-11T08:25:40.491 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.491 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.491 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:25:40.491 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.807197+0000", 2024-09-11T08:25:40.491 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.5098e-05 2024-09-11T08:25:40.491 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:25:40.491 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:25:40.491 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:25:40.491 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.491 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.491 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.15369.0:212789 9.2 e298/246 9:4d7a08f9:::benchmark_data_smithi150_133616_object97183:head v 298'13254, mlcod=298'13252)", 2024-09-11T08:25:40.492 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:21:06.629297+0000", 2024-09-11T08:25:40.492 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 273.78900663600001, 2024-09-11T08:25:40.492 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.18916815200000001, 2024-09-11T08:25:40.492 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:25:40.492 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:25:40.492 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:25:40.492 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.492 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:25:40.492 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.629297+0000", 2024-09-11T08:25:40.492 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:25:40.492 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.492 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.492 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:25:40.493 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.629297+0000", 2024-09-11T08:25:40.493 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:25:40.493 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.493 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.493 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:25:40.493 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.629296+0000", 2024-09-11T08:25:40.493 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:25:40.493 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.493 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.493 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:25:40.494 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.629305+0000", 2024-09-11T08:25:40.494 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.5150000000000006e-06 2024-09-11T08:25:40.494 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.494 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.494 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:25:40.494 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.807125+0000", 2024-09-11T08:25:40.494 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.1778209 2024-09-11T08:25:40.494 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.494 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.494 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:25:40.494 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.807134+0000", 2024-09-11T08:25:40.494 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.0259999999999996e-06 2024-09-11T08:25:40.494 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.494 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.494 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:25:40.495 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.807191+0000", 2024-09-11T08:25:40.495 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.6659000000000001e-05 2024-09-11T08:25:40.495 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.495 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.495 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:25:40.495 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.807216+0000", 2024-09-11T08:25:40.495 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.4712999999999999e-05 2024-09-11T08:25:40.495 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.495 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.495 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:25:40.495 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.818435+0000", 2024-09-11T08:25:40.495 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.011218756999999999 2024-09-11T08:25:40.495 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.495 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.495 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:25:40.496 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:21:06.818465+0000", 2024-09-11T08:25:40.496 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.0519999999999999e-05 2024-09-11T08:25:40.496 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:25:40.496 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:25:40.496 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:25:40.496 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.496 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:40.497 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:40.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:25:40.498 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.24320.0:2225 5.15 5:a93a5511:::notify.2:head [watch reconnect cookie 94783556101344 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e329)", 2024-09-11T08:25:40.498 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:22:58.734595+0000", 2024-09-11T08:25:40.498 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 161.683708924, 2024-09-11T08:25:40.498 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.42897647900000002, 2024-09-11T08:25:40.498 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:25:40.498 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "started", 2024-09-11T08:25:40.498 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:25:40.498 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.24320", 2024-09-11T08:25:40.498 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1503848706", 2024-09-11T08:25:40.498 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 2225 2024-09-11T08:25:40.499 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.499 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:25:40.499 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.499 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:25:40.499 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.734595+0000", 2024-09-11T08:25:40.499 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:25:40.499 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.499 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.499 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:25:40.499 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.734595+0000", 2024-09-11T08:25:40.499 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:25:40.499 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.500 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.500 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:25:40.500 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.734594+0000", 2024-09-11T08:25:40.500 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:25:40.500 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.500 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.500 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:25:40.500 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.734599+0000", 2024-09-11T08:25:40.500 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.3669999999999996e-06 2024-09-11T08:25:40.500 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.500 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.500 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:25:40.501 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.734612+0000", 2024-09-11T08:25:40.501 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2509e-05 2024-09-11T08:25:40.501 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.501 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.501 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:25:40.501 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.734618+0000", 2024-09-11T08:25:40.501 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.1040000000000003e-06 2024-09-11T08:25:40.501 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.501 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.501 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:25:40.501 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.734662+0000", 2024-09-11T08:25:40.501 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.4629999999999998e-05 2024-09-11T08:25:40.502 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.502 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.502 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:25:40.502 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.734672+0000", 2024-09-11T08:25:40.502 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.5079999999999994e-06 2024-09-11T08:25:40.502 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.502 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.502 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:25:40.502 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.160757+0000", 2024-09-11T08:25:40.502 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.42608511500000001 2024-09-11T08:25:40.502 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.503 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.503 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:25:40.503 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.160901+0000", 2024-09-11T08:25:40.503 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00014394399999999999 2024-09-11T08:25:40.503 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.503 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.503 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:25:40.503 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.163571+0000", 2024-09-11T08:25:40.503 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0026702879999999998 2024-09-11T08:25:40.503 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:25:40.503 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:25:40.503 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:25:40.504 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.504 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.504 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:165003 10.2 10:4bd4d3a2:::benchmark_data_smithi150_158067_object34759:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-11T08:25:40.504 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:24:01.628825+0000", 2024-09-11T08:25:40.504 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 98.789478177999996, 2024-09-11T08:25:40.504 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.24413458399999999, 2024-09-11T08:25:40.504 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:25:40.504 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:25:40.504 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:25:40.504 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:25:40.504 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:25:40.504 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 165003 2024-09-11T08:25:40.505 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.505 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:25:40.505 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.505 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:25:40.505 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.628825+0000", 2024-09-11T08:25:40.505 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:25:40.505 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.505 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.505 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:25:40.505 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.628825+0000", 2024-09-11T08:25:40.505 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:25:40.505 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.506 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.506 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:25:40.506 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.628823+0000", 2024-09-11T08:25:40.506 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:25:40.506 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.506 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.506 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:25:40.506 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.628833+0000", 2024-09-11T08:25:40.506 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.4709999999999996e-06 2024-09-11T08:25:40.506 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.506 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.506 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:25:40.507 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.628848+0000", 2024-09-11T08:25:40.507 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.5594e-05 2024-09-11T08:25:40.507 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.507 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.507 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:25:40.507 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.628855+0000", 2024-09-11T08:25:40.507 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.1010000000000003e-06 2024-09-11T08:25:40.507 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.507 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.507 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:25:40.507 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.628904+0000", 2024-09-11T08:25:40.507 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.8809e-05 2024-09-11T08:25:40.507 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.507 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.507 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:25:40.508 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.629016+0000", 2024-09-11T08:25:40.508 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00011197 2024-09-11T08:25:40.508 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.508 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.508 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 0,4", 2024-09-11T08:25:40.508 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.629120+0000", 2024-09-11T08:25:40.508 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00010416099999999999 2024-09-11T08:25:40.508 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.508 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.508 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:25:40.508 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.632345+0000", 2024-09-11T08:25:40.508 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0032245920000000001 2024-09-11T08:25:40.508 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.509 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.509 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:25:40.509 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.635420+0000", 2024-09-11T08:25:40.509 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0030745579999999998 2024-09-11T08:25:40.509 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.509 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.509 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:25:40.509 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.872700+0000", 2024-09-11T08:25:40.509 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.237280138 2024-09-11T08:25:40.509 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.509 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.509 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:25:40.510 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.872818+0000", 2024-09-11T08:25:40.510 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000118532 2024-09-11T08:25:40.510 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.510 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.510 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:25:40.510 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.872960+0000", 2024-09-11T08:25:40.510 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000141769 2024-09-11T08:25:40.510 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:25:40.510 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:25:40.510 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:25:40.510 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.510 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.510 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:165008 10.2 10:4b786cc7:::benchmark_data_smithi150_158067_object34764:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-11T08:25:40.510 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:24:01.631868+0000", 2024-09-11T08:25:40.510 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 98.786435229000006, 2024-09-11T08:25:40.511 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.24477638600000001, 2024-09-11T08:25:40.511 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:25:40.511 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:25:40.511 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:25:40.511 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:25:40.511 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:25:40.511 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 165008 2024-09-11T08:25:40.511 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.511 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:25:40.511 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.511 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:25:40.511 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.631868+0000", 2024-09-11T08:25:40.511 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:25:40.511 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.511 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.512 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:25:40.512 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.631868+0000", 2024-09-11T08:25:40.512 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:25:40.512 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.512 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.512 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:25:40.512 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.631867+0000", 2024-09-11T08:25:40.512 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:25:40.512 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.512 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.512 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:25:40.512 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.631874+0000", 2024-09-11T08:25:40.513 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.1280000000000002e-06 2024-09-11T08:25:40.513 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.513 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.513 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:25:40.513 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:25:40.514 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.631887+0000", 2024-09-11T08:25:40.514 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3380999999999999e-05 2024-09-11T08:25:40.514 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.514 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.514 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:25:40.514 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.631894+0000", 2024-09-11T08:25:40.514 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.4640000000000003e-06 2024-09-11T08:25:40.514 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.514 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.514 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:25:40.514 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.632363+0000", 2024-09-11T08:25:40.514 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00046943899999999999 2024-09-11T08:25:40.515 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.515 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.515 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:25:40.515 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.632459+0000", 2024-09-11T08:25:40.515 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.5537000000000004e-05 2024-09-11T08:25:40.515 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.515 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.515 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 0,4", 2024-09-11T08:25:40.515 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.632557+0000", 2024-09-11T08:25:40.515 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.8307000000000006e-05 2024-09-11T08:25:40.515 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.515 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.515 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:25:40.515 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.635405+0000", 2024-09-11T08:25:40.516 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0028483549999999999 2024-09-11T08:25:40.516 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.516 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.516 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:25:40.516 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.637635+0000", 2024-09-11T08:25:40.516 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0022297110000000001 2024-09-11T08:25:40.516 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.516 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.517 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:25:40.517 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.876559+0000", 2024-09-11T08:25:40.517 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.23892380799999999 2024-09-11T08:25:40.517 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.517 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.517 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:25:40.517 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.876608+0000", 2024-09-11T08:25:40.517 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.8980000000000002e-05 2024-09-11T08:25:40.517 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.517 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.517 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:25:40.517 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.876645+0000", 2024-09-11T08:25:40.518 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.6801000000000002e-05 2024-09-11T08:25:40.518 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:25:40.518 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:25:40.518 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:25:40.518 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.518 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.518 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:165016 10.2 10:4c21f478:::benchmark_data_smithi150_158067_object34772:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-11T08:25:40.518 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:24:01.641003+0000", 2024-09-11T08:25:40.518 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 98.777300423, 2024-09-11T08:25:40.518 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.23675271000000001, 2024-09-11T08:25:40.518 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:25:40.518 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:25:40.518 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:25:40.519 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:25:40.519 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:25:40.519 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 165016 2024-09-11T08:25:40.519 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.519 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:25:40.519 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.519 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:25:40.519 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.641003+0000", 2024-09-11T08:25:40.519 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:25:40.519 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.519 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.519 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:25:40.519 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.641003+0000", 2024-09-11T08:25:40.520 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:25:40.520 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.520 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.520 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:25:40.520 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.641001+0000", 2024-09-11T08:25:40.520 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:25:40.520 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.520 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.520 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:25:40.520 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.641010+0000", 2024-09-11T08:25:40.520 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.7930000000000003e-06 2024-09-11T08:25:40.520 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.521 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.521 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:25:40.521 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.641025+0000", 2024-09-11T08:25:40.521 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4543e-05 2024-09-11T08:25:40.521 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.521 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.521 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:25:40.521 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.641031+0000", 2024-09-11T08:25:40.521 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.759e-06 2024-09-11T08:25:40.521 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.521 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.521 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:25:40.521 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.641078+0000", 2024-09-11T08:25:40.522 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.6922999999999998e-05 2024-09-11T08:25:40.522 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.522 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.522 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:25:40.522 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.641203+0000", 2024-09-11T08:25:40.522 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012512499999999999 2024-09-11T08:25:40.522 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.522 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.522 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 0,4", 2024-09-11T08:25:40.522 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.641569+0000", 2024-09-11T08:25:40.522 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00036591199999999998 2024-09-11T08:25:40.522 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.522 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.523 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:25:40.523 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.647616+0000", 2024-09-11T08:25:40.523 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0060464400000000001 2024-09-11T08:25:40.523 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.523 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.523 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:25:40.523 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.647650+0000", 2024-09-11T08:25:40.523 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.4282999999999997e-05 2024-09-11T08:25:40.523 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.523 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.523 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:25:40.523 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.877296+0000", 2024-09-11T08:25:40.523 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.22964607300000001 2024-09-11T08:25:40.523 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.523 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.524 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:25:40.524 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.877693+0000", 2024-09-11T08:25:40.524 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00039742999999999999 2024-09-11T08:25:40.524 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.524 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.524 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:25:40.524 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.877756+0000", 2024-09-11T08:25:40.524 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.2422000000000006e-05 2024-09-11T08:25:40.524 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:25:40.524 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:25:40.524 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:25:40.524 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.524 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.525 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:165018 10.2 10:421759e8:::benchmark_data_smithi150_158067_object34774:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-11T08:25:40.525 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:24:01.641527+0000", 2024-09-11T08:25:40.525 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 98.776776357000003, 2024-09-11T08:25:40.525 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.236359863, 2024-09-11T08:25:40.525 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:25:40.525 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:25:40.525 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:25:40.525 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:25:40.525 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:25:40.525 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 165018 2024-09-11T08:25:40.525 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.525 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:25:40.525 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.525 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:25:40.526 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.641527+0000", 2024-09-11T08:25:40.526 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:25:40.526 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.526 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.526 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:25:40.526 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.641527+0000", 2024-09-11T08:25:40.526 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:25:40.526 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.526 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.526 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:25:40.526 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.641526+0000", 2024-09-11T08:25:40.526 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:25:40.526 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.527 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.527 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:25:40.527 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.641531+0000", 2024-09-11T08:25:40.527 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.8359999999999999e-06 2024-09-11T08:25:40.527 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.527 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.527 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:25:40.527 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.641541+0000", 2024-09-11T08:25:40.527 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.5810000000000003e-06 2024-09-11T08:25:40.527 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.527 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.527 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:25:40.527 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.641575+0000", 2024-09-11T08:25:40.527 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.4115000000000001e-05 2024-09-11T08:25:40.528 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.528 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.528 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:25:40.528 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.644853+0000", 2024-09-11T08:25:40.528 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0032777180000000002 2024-09-11T08:25:40.528 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.528 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.528 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:25:40.528 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.644952+0000", 2024-09-11T08:25:40.528 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.9467000000000002e-05 2024-09-11T08:25:40.528 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.528 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.528 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 0,4", 2024-09-11T08:25:40.528 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.645052+0000", 2024-09-11T08:25:40.529 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.9320000000000005e-05 2024-09-11T08:25:40.529 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.529 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.529 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:25:40.529 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.647643+0000", 2024-09-11T08:25:40.529 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.002591561 2024-09-11T08:25:40.529 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.529 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.529 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:25:40.529 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.647689+0000", 2024-09-11T08:25:40.529 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.6019000000000001e-05 2024-09-11T08:25:40.529 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.529 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.529 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:25:40.530 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.877808+0000", 2024-09-11T08:25:40.530 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.230118814 2024-09-11T08:25:40.530 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.530 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.530 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:25:40.530 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.877850+0000", 2024-09-11T08:25:40.530 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.2109e-05 2024-09-11T08:25:40.530 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.530 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.530 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:25:40.530 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:01.877887+0000", 2024-09-11T08:25:40.530 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.7081e-05 2024-09-11T08:25:40.530 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:25:40.530 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:25:40.531 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:25:40.531 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.531 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.531 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.15636.0:208423 10.0 e356/308 10:09f2f3e7:::benchmark_data_smithi150_158067_object78179:head v 356'12855, mlcod=356'12853)", 2024-09-11T08:25:40.531 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:24:46.028364+0000", 2024-09-11T08:25:40.531 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 54.389939429999998, 2024-09-11T08:25:40.531 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.19364426600000001, 2024-09-11T08:25:40.531 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:25:40.531 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:25:40.531 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:25:40.531 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.531 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:25:40.532 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:46.028364+0000", 2024-09-11T08:25:40.532 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:25:40.532 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.532 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.532 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:25:40.532 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:46.028364+0000", 2024-09-11T08:25:40.532 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:25:40.532 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.532 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.532 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:25:40.532 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:46.028363+0000", 2024-09-11T08:25:40.532 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:25:40.532 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.533 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.533 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:25:40.533 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:46.028374+0000", 2024-09-11T08:25:40.533 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1525000000000001e-05 2024-09-11T08:25:40.533 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.533 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.533 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:25:40.533 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:46.028387+0000", 2024-09-11T08:25:40.533 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2435999999999999e-05 2024-09-11T08:25:40.533 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.533 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.533 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:25:40.533 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:46.028392+0000", 2024-09-11T08:25:40.533 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.6500000000000001e-06 2024-09-11T08:25:40.534 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.534 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.534 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:25:40.534 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:46.028431+0000", 2024-09-11T08:25:40.534 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.8466000000000001e-05 2024-09-11T08:25:40.534 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.534 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.534 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:25:40.534 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:46.028446+0000", 2024-09-11T08:25:40.534 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4800999999999999e-05 2024-09-11T08:25:40.534 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.534 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.535 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:25:40.535 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:46.030967+0000", 2024-09-11T08:25:40.535 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.002521077 2024-09-11T08:25:40.535 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.535 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.535 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:25:40.535 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:46.222008+0000", 2024-09-11T08:25:40.535 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.191041605 2024-09-11T08:25:40.535 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:25:40.535 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:25:40.535 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:25:40.535 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.535 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.535 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.15636.0:208429 10.b e356/308 10:de874e57:::benchmark_data_smithi150_158067_object78185:head v 356'13233, mlcod=356'13231)", 2024-09-11T08:25:40.536 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:24:46.037198+0000", 2024-09-11T08:25:40.536 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 54.381105466999998, 2024-09-11T08:25:40.536 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.18815517700000001, 2024-09-11T08:25:40.536 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:25:40.536 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:25:40.536 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:25:40.536 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.536 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:25:40.536 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:46.037198+0000", 2024-09-11T08:25:40.536 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:25:40.536 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.536 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.537 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:25:40.537 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:46.037198+0000", 2024-09-11T08:25:40.537 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:25:40.537 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.537 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.537 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:25:40.537 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:46.037197+0000", 2024-09-11T08:25:40.537 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:25:40.537 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.537 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.537 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:25:40.537 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:46.037207+0000", 2024-09-11T08:25:40.537 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0215e-05 2024-09-11T08:25:40.538 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.538 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.538 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:25:40.538 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:46.221925+0000", 2024-09-11T08:25:40.538 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.18471836899999999 2024-09-11T08:25:40.538 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.538 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.538 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:25:40.538 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:46.221939+0000", 2024-09-11T08:25:40.538 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3838e-05 2024-09-11T08:25:40.538 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.538 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.538 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:25:40.539 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:46.221985+0000", 2024-09-11T08:25:40.539 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.6032000000000002e-05 2024-09-11T08:25:40.539 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.539 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.539 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:25:40.539 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:46.222046+0000", 2024-09-11T08:25:40.539 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.1047e-05 2024-09-11T08:25:40.539 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.539 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.539 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:25:40.540 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:46.225327+0000", 2024-09-11T08:25:40.540 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0032805809999999999 2024-09-11T08:25:40.540 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:25:40.540 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:25:40.540 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:25:40.540 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:24:46.225353+0000", 2024-09-11T08:25:40.540 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.6449999999999999e-05 2024-09-11T08:25:40.540 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:25:40.540 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:25:40.540 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:25:40.540 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:25:40.540 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:25:40.540 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-11T08:25:40.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:40 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:25:40.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:40 smithi181 ceph-mon[32701]: pgmap v822: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 144 active+clean; 402 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 999 op/s; 3975/19851 objects degraded (20.024%); 0 B/s, 263 objects/s recovering 2024-09-11T08:25:40.598 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:40.635 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:40.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:40.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:40.737 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:25:40.767 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:40.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:40.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:25:40.869 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:25:40.907 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:40.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:40.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:25:41.009 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:41.036 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:41.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:41.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:41.139 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:41.163 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:41.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:41.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:41.265 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:25:41.287 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:41.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:41.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:25:41.389 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:25:41.411 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:41.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:41.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:25:41.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:41 smithi150 ceph-mon[39612]: Health check update: Degraded data redundancy: 3975/19851 objects degraded (20.024%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-11T08:25:41.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:41 smithi150 ceph-mon[39612]: osdmap e367: 8 total, 8 up, 8 in 2024-09-11T08:25:41.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:41 smithi150 ceph-mon[29276]: Health check update: Degraded data redundancy: 3975/19851 objects degraded (20.024%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-11T08:25:41.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:41 smithi150 ceph-mon[29276]: osdmap e367: 8 total, 8 up, 8 in 2024-09-11T08:25:41.513 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:25:41.534 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:41.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:41.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:25:41.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:41 smithi181 ceph-mon[32701]: Health check update: Degraded data redundancy: 3975/19851 objects degraded (20.024%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-11T08:25:41.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:41 smithi181 ceph-mon[32701]: osdmap e367: 8 total, 8 up, 8 in 2024-09-11T08:25:41.636 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:41.658 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:41.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:41.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:41.760 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:41.792 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:41.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:41.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:41.893 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:25:41.927 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:41.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:41.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:25:42.029 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:42.055 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:42.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:42.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:42.128 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T08:25:42.157 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:25:42.179 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:42.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:42.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:25:42.280 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:25:42.302 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:42.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:42.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:25:42.404 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:42.426 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:42.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:42.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:42.459 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:42 smithi150 ceph-mon[29276]: 3.9 scrub starts 2024-09-11T08:25:42.459 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:42 smithi150 ceph-mon[29276]: pgmap v824: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 144 active+clean; 345 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s; 3975/17118 objects degraded (23.221%); 0 B/s, 321 objects/s recovering 2024-09-11T08:25:42.459 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:42 smithi150 ceph-mon[29276]: osdmap e368: 8 total, 8 up, 8 in 2024-09-11T08:25:42.459 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:42 smithi150 ceph-mon[39612]: 3.9 scrub starts 2024-09-11T08:25:42.459 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:42 smithi150 ceph-mon[39612]: pgmap v824: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 144 active+clean; 345 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s; 3975/17118 objects degraded (23.221%); 0 B/s, 321 objects/s recovering 2024-09-11T08:25:42.460 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:42 smithi150 ceph-mon[39612]: osdmap e368: 8 total, 8 up, 8 in 2024-09-11T08:25:42.528 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:25:42.550 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:42.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:42.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:25:42.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:42 smithi181 ceph-mon[32701]: 3.9 scrub starts 2024-09-11T08:25:42.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:42 smithi181 ceph-mon[32701]: pgmap v824: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 144 active+clean; 345 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s; 3975/17118 objects degraded (23.221%); 0 B/s, 321 objects/s recovering 2024-09-11T08:25:42.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:42 smithi181 ceph-mon[32701]: osdmap e368: 8 total, 8 up, 8 in 2024-09-11T08:25:42.652 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:42.674 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:42.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:42.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:42.776 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:42.807 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:42.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:42.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:42.909 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:25:42.935 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:42.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:42.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:25:43.037 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:25:43.058 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:43.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:43.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:25:43.160 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:25:43.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:43.183 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:43.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:25:43.284 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:43.317 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:43.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:43.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:43.419 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:25:43.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:43.462 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:43.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:25:43.474 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:43 smithi150 ceph-mon[39612]: 6.10 scrub starts 2024-09-11T08:25:43.475 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:43 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:25:43.475 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:43 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:25:43.475 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:43 smithi150 ceph-mon[39612]: osdmap e369: 8 total, 8 up, 8 in 2024-09-11T08:25:43.475 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:43 smithi150 ceph-mon[29276]: 6.10 scrub starts 2024-09-11T08:25:43.475 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:43 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:25:43.475 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:43 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:25:43.475 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:43 smithi150 ceph-mon[29276]: osdmap e369: 8 total, 8 up, 8 in 2024-09-11T08:25:43.563 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:25:43.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:43 smithi181 ceph-mon[32701]: 6.10 scrub starts 2024-09-11T08:25:43.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:43 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:25:43.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:43 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:25:43.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:43 smithi181 ceph-mon[32701]: osdmap e369: 8 total, 8 up, 8 in 2024-09-11T08:25:43.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:43.608 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:43.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:25:43.709 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:43.730 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:43.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:43.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:43.832 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:43.869 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:43.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:43.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:43.971 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:25:44.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:44.008 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:44.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:25:44.109 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:44.144 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:44.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:44.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:44.246 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:25:44.287 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:44.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:44.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:25:44.388 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:25:44.426 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:44.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:44.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:25:44.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:44 smithi150 ceph-mon[29276]: 4.15 scrub starts 2024-09-11T08:25:44.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:44 smithi150 ceph-mon[29276]: 5.1 scrub starts 2024-09-11T08:25:44.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:44 smithi150 ceph-mon[29276]: pgmap v827: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 144 active+clean; 345 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 814 op/s; 3975/17118 objects degraded (23.221%); 0 B/s, 184 objects/s recovering 2024-09-11T08:25:44.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:44 smithi150 ceph-mon[39612]: 4.15 scrub starts 2024-09-11T08:25:44.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:44 smithi150 ceph-mon[39612]: 5.1 scrub starts 2024-09-11T08:25:44.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:44 smithi150 ceph-mon[39612]: pgmap v827: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 144 active+clean; 345 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 814 op/s; 3975/17118 objects degraded (23.221%); 0 B/s, 184 objects/s recovering 2024-09-11T08:25:44.528 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:44.534 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-11T08:25:44.557 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:44.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:44.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:44.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:44 smithi181 ceph-mon[32701]: 4.15 scrub starts 2024-09-11T08:25:44.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:44 smithi181 ceph-mon[32701]: 5.1 scrub starts 2024-09-11T08:25:44.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:44 smithi181 ceph-mon[32701]: pgmap v827: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 144 active+clean; 345 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 814 op/s; 3975/17118 objects degraded (23.221%); 0 B/s, 184 objects/s recovering 2024-09-11T08:25:44.659 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:44.702 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:44.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:44.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:44.804 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:25:44.834 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:44.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:44.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:25:44.936 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:44.970 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:44.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:44.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:45.073 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:45.099 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:45.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:45.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:45.191 INFO:teuthology.orchestra.run.smithi150.stderr:nodeep-scrub is unset 2024-09-11T08:25:45.201 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:45.238 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:45.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:45.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:45.340 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:25:45.383 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:45.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:45.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:25:45.485 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:25:45.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:45 smithi150 ceph-mon[29276]: 3.d scrub starts 2024-09-11T08:25:45.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:45 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:25:45.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:45 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:25:45.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:45 smithi150 ceph-mon[29276]: 10.6 scrub starts 2024-09-11T08:25:45.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:45 smithi150 ceph-mon[29276]: 6.4 scrub starts 2024-09-11T08:25:45.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:45 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3351643659' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:25:45.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:45 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:25:45.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:45 smithi150 ceph-mon[39612]: 3.d scrub starts 2024-09-11T08:25:45.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:45 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:25:45.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:45 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:25:45.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:45 smithi150 ceph-mon[39612]: 10.6 scrub starts 2024-09-11T08:25:45.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:45 smithi150 ceph-mon[39612]: 6.4 scrub starts 2024-09-11T08:25:45.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:45 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3351643659' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:25:45.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:45 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:25:45.525 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:45.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:45.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:25:45.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:45 smithi181 ceph-mon[32701]: 3.d scrub starts 2024-09-11T08:25:45.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:45 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:25:45.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:45 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:25:45.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:45 smithi181 ceph-mon[32701]: 10.6 scrub starts 2024-09-11T08:25:45.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:45 smithi181 ceph-mon[32701]: 6.4 scrub starts 2024-09-11T08:25:45.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:45 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3351643659' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:25:45.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:45 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:25:45.574 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:25:45 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:25:45] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:25:45.626 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:25:45.662 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:45.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:45.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:25:45.727 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Removed 130241 objects 2024-09-11T08:25:45.728 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Clean up completed and total clean up time :134.683 2024-09-11T08:25:45.764 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:25:45.786 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:45.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:45.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:25:45.861 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:25:45.888 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:25:45.910 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:45.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:45.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:25:46.012 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:25:46.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:46.035 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:46.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:25:46.137 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:25:46.158 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:46.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:46.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:25:46.260 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:25:46.282 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:46.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:46.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:25:46.383 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:46.405 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:46.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:46.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:46.441 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:46 smithi150 ceph-mon[29276]: pgmap v828: 148 pgs: 2 active+recovering+undersized+remapped, 146 active+clean; 140 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 698 op/s; 0 B/s, 267 objects/s recovering 2024-09-11T08:25:46.441 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:46 smithi150 ceph-mon[29276]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:25:46.441 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:46 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:25:46.441 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:46 smithi150 ceph-mon[29276]: osdmap e370: 8 total, 8 up, 8 in 2024-09-11T08:25:46.442 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:46 smithi150 ceph-mon[29276]: 5.7 scrub starts 2024-09-11T08:25:46.442 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:46 smithi150 ceph-mon[29276]: Health check update: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-09-11T08:25:46.442 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:46 smithi150 ceph-mon[39612]: pgmap v828: 148 pgs: 2 active+recovering+undersized+remapped, 146 active+clean; 140 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 698 op/s; 0 B/s, 267 objects/s recovering 2024-09-11T08:25:46.442 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:46 smithi150 ceph-mon[39612]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:25:46.442 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:46 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:25:46.442 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:46 smithi150 ceph-mon[39612]: osdmap e370: 8 total, 8 up, 8 in 2024-09-11T08:25:46.442 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:46 smithi150 ceph-mon[39612]: 5.7 scrub starts 2024-09-11T08:25:46.442 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:46 smithi150 ceph-mon[39612]: Health check update: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-09-11T08:25:46.507 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:46.529 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:46.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:46.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:46.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:46 smithi181 ceph-mon[32701]: pgmap v828: 148 pgs: 2 active+recovering+undersized+remapped, 146 active+clean; 140 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 698 op/s; 0 B/s, 267 objects/s recovering 2024-09-11T08:25:46.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:46 smithi181 ceph-mon[32701]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:25:46.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:46 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:25:46.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:46 smithi181 ceph-mon[32701]: osdmap e370: 8 total, 8 up, 8 in 2024-09-11T08:25:46.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:46 smithi181 ceph-mon[32701]: 5.7 scrub starts 2024-09-11T08:25:46.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:46 smithi181 ceph-mon[32701]: Health check update: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-09-11T08:25:46.631 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:46.652 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:46.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:46.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:46.727 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-09-11T08:25:46.727 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-09-11T08:25:46.753 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:46.776 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:46.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:46.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:46.877 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:25:46.899 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:46.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:46.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:25:47.001 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:47.023 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:47.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:47.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:47.125 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:25:47.147 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:47.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:47.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:25:47.249 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:47.272 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:47.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:47.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:47.375 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:25:47.397 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:47.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:47.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:25:47.470 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:47 smithi150 ceph-mon[29276]: 4.18 deep-scrub starts 2024-09-11T08:25:47.470 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:47 smithi150 ceph-mon[29276]: 6.11 scrub starts 2024-09-11T08:25:47.470 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:47 smithi150 ceph-mon[39612]: 4.18 deep-scrub starts 2024-09-11T08:25:47.471 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:47 smithi150 ceph-mon[39612]: 6.11 scrub starts 2024-09-11T08:25:47.499 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:25:47.520 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:47.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:47.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:25:47.532 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:25:47.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:47 smithi181 ceph-mon[32701]: 4.18 deep-scrub starts 2024-09-11T08:25:47.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:47 smithi181 ceph-mon[32701]: 6.11 scrub starts 2024-09-11T08:25:47.622 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:25:47.645 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:47.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:47.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:25:47.747 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:25:47.769 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:47.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:47.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:25:47.870 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:47.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:47.896 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:47.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:47.997 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:48.018 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:48.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:48.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:48.120 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:25:48.142 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:48.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:48.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:25:48.244 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:25:48.267 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:48.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:48.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:25:48.369 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:25:48.393 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:48.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:48.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:25:48.474 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:48 smithi181 ceph-mon[32701]: pgmap v830: 148 pgs: 2 active+recovering+undersized+remapped, 146 active+clean; 140 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 546 op/s; 0 B/s, 188 objects/s recovering 2024-09-11T08:25:48.475 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T08:25:48.478 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:48 smithi150 ceph-mon[29276]: pgmap v830: 148 pgs: 2 active+recovering+undersized+remapped, 146 active+clean; 140 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 546 op/s; 0 B/s, 188 objects/s recovering 2024-09-11T08:25:48.478 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:48 smithi150 ceph-mon[39612]: pgmap v830: 148 pgs: 2 active+recovering+undersized+remapped, 146 active+clean; 140 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 546 op/s; 0 B/s, 188 objects/s recovering 2024-09-11T08:25:48.495 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:48.540 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:48.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:48.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:48.642 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:48.664 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:48.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:48.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:48.738 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:25:48 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:25:48] "GET /metrics HTTP/1.1" 200 224445 "" "Prometheus/2.43.0" 2024-09-11T08:25:48.766 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:25:48.788 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:48.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:48.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:25:48.890 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:25:48.913 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:48.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:48.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:25:49.014 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:49.036 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:49.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:49.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:49.073 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:25:48 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:25:49.137 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:49.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:49.160 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:49.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:49.198 INFO:teuthology.orchestra.run.smithi150.stderr:reweighted osd.3 to 1 (10000) 2024-09-11T08:25:49.261 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:25:49.284 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:49.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:49.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:25:49.386 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:25:49.407 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:49.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:49.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:25:49.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:49 smithi150 ceph-mon[29276]: 4.3 scrub starts 2024-09-11T08:25:49.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:49 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2453278628' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-11T08:25:49.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:49 smithi150 ceph-mon[39612]: 4.3 scrub starts 2024-09-11T08:25:49.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:49 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2453278628' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-11T08:25:49.509 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:25:49.530 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:49.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:49.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:25:49.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:49 smithi181 ceph-mon[32701]: 4.3 scrub starts 2024-09-11T08:25:49.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:49 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2453278628' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-11T08:25:49.632 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:25:49.654 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:49.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:49.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:25:49.756 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:49.778 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:49.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:49.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:49.880 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:25:49.901 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:49.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:49.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:25:50.004 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:50.025 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:50.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:50.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:50.127 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:25:50.149 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:50.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:50.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:25:50.251 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:50.272 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:50.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:50.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:50.374 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:50.397 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:50.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:50.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:50.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:50 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:25:50.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:50 smithi150 ceph-mon[39612]: 3.9 scrub starts 2024-09-11T08:25:50.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:50 smithi150 ceph-mon[39612]: pgmap v831: 148 pgs: 2 active+recovering+undersized+remapped, 146 active+clean; 20 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 704 op/s; 0 B/s, 295 objects/s recovering 2024-09-11T08:25:50.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:50 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2453278628' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-11T08:25:50.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:50 smithi150 ceph-mon[39612]: osdmap e371: 8 total, 8 up, 8 in 2024-09-11T08:25:50.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:50 smithi150 ceph-mon[39612]: 6.1d scrub starts 2024-09-11T08:25:50.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:50 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:25:50.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:50 smithi150 ceph-mon[29276]: 3.9 scrub starts 2024-09-11T08:25:50.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:50 smithi150 ceph-mon[29276]: pgmap v831: 148 pgs: 2 active+recovering+undersized+remapped, 146 active+clean; 20 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 704 op/s; 0 B/s, 295 objects/s recovering 2024-09-11T08:25:50.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:50 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2453278628' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-11T08:25:50.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:50 smithi150 ceph-mon[29276]: osdmap e371: 8 total, 8 up, 8 in 2024-09-11T08:25:50.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:50 smithi150 ceph-mon[29276]: 6.1d scrub starts 2024-09-11T08:25:50.499 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:50.522 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:50.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:50.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:50.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:50 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:25:50.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:50 smithi181 ceph-mon[32701]: 3.9 scrub starts 2024-09-11T08:25:50.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:50 smithi181 ceph-mon[32701]: pgmap v831: 148 pgs: 2 active+recovering+undersized+remapped, 146 active+clean; 20 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 704 op/s; 0 B/s, 295 objects/s recovering 2024-09-11T08:25:50.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:50 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2453278628' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-11T08:25:50.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:50 smithi181 ceph-mon[32701]: osdmap e371: 8 total, 8 up, 8 in 2024-09-11T08:25:50.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:50 smithi181 ceph-mon[32701]: 6.1d scrub starts 2024-09-11T08:25:50.624 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:25:50.647 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:50.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:50.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:25:50.749 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:50.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:50.771 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:50.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:50.872 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:50.895 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:50.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:50.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:50.997 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:51.019 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:51.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:51.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:51.121 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:51.143 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:51.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:51.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:51.245 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:51.267 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:51.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:51.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:51.369 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:51.392 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:51.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:51.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:51.494 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:51.518 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:51.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:51.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:51.620 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:51.643 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:51.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:51.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:51.745 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:51.768 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:51.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:51.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:51.869 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:25:51.891 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:51.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:51.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:25:51.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:51 smithi150 ceph-mon[29276]: 4.0 scrub starts 2024-09-11T08:25:51.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:51 smithi150 ceph-mon[29276]: pgmap v833: 148 pgs: 2 active+recovering+undersized+remapped, 146 active+clean; 582 KiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 689 op/s; 0 B/s, 273 objects/s recovering 2024-09-11T08:25:51.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:51 smithi150 ceph-mon[29276]: 3.6 scrub starts 2024-09-11T08:25:51.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:51 smithi150 ceph-mon[29276]: 3.6 scrub ok 2024-09-11T08:25:51.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:51 smithi150 ceph-mon[39612]: 4.0 scrub starts 2024-09-11T08:25:51.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:51 smithi150 ceph-mon[39612]: pgmap v833: 148 pgs: 2 active+recovering+undersized+remapped, 146 active+clean; 582 KiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 689 op/s; 0 B/s, 273 objects/s recovering 2024-09-11T08:25:51.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:51 smithi150 ceph-mon[39612]: 3.6 scrub starts 2024-09-11T08:25:51.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:51 smithi150 ceph-mon[39612]: 3.6 scrub ok 2024-09-11T08:25:51.992 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:25:52.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:52.016 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:52.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:25:52.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:51 smithi181 ceph-mon[32701]: 4.0 scrub starts 2024-09-11T08:25:52.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:51 smithi181 ceph-mon[32701]: pgmap v833: 148 pgs: 2 active+recovering+undersized+remapped, 146 active+clean; 582 KiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 689 op/s; 0 B/s, 273 objects/s recovering 2024-09-11T08:25:52.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:51 smithi181 ceph-mon[32701]: 3.6 scrub starts 2024-09-11T08:25:52.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:51 smithi181 ceph-mon[32701]: 3.6 scrub ok 2024-09-11T08:25:52.117 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:25:52.140 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:52.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:52.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:25:52.242 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:52.264 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:52.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:52.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:52.366 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:52.387 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:52.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:52.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:52.490 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:52.511 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:52.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:52.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:52.613 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:52.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:52.636 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:52.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:52.737 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:52.761 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:52.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:52.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:52.863 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:25:52.888 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:52.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:52.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:25:52.990 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:53.014 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:53.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:53.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:53.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:52 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:25:53.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:52 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:25:53.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:52 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:25:53.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:52 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:25:53.116 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:25:53.141 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:53.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:53.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:25:53.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:52 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:25:53.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:52 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:25:53.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:52 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:25:53.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:52 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:25:53.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:52 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:25:53.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:52 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:25:53.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:52 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:25:53.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:52 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:25:53.243 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:25:53.264 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:53.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:53.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:25:53.367 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:25:53.388 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:53.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:53.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:25:53.490 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:53.515 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:53.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:53.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:53.617 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:25:53.640 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:53.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:53.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:25:53.742 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:53.766 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:53.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:53.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:53.867 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:53.890 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:53.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:53.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:53.991 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:54.016 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:54.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:54.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:54.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:53 smithi181 ceph-mon[32701]: osdmap e372: 8 total, 8 up, 8 in 2024-09-11T08:25:54.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:53 smithi181 ceph-mon[32701]: pgmap v835: 148 pgs: 2 active+recovering+undersized+remapped, 146 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 645 B/s rd, 0 B/s wr, 282 op/s; 0 B/s, 132 objects/s recovering 2024-09-11T08:25:54.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:53 smithi181 ceph-mon[32701]: 5.13 scrub starts 2024-09-11T08:25:54.118 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:54.142 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:54.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:54.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:54.145 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:53 smithi150 ceph-mon[29276]: osdmap e372: 8 total, 8 up, 8 in 2024-09-11T08:25:54.145 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:53 smithi150 ceph-mon[29276]: pgmap v835: 148 pgs: 2 active+recovering+undersized+remapped, 146 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 645 B/s rd, 0 B/s wr, 282 op/s; 0 B/s, 132 objects/s recovering 2024-09-11T08:25:54.145 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:53 smithi150 ceph-mon[29276]: 5.13 scrub starts 2024-09-11T08:25:54.145 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:53 smithi150 ceph-mon[39612]: osdmap e372: 8 total, 8 up, 8 in 2024-09-11T08:25:54.146 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:53 smithi150 ceph-mon[39612]: pgmap v835: 148 pgs: 2 active+recovering+undersized+remapped, 146 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 645 B/s rd, 0 B/s wr, 282 op/s; 0 B/s, 132 objects/s recovering 2024-09-11T08:25:54.146 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:53 smithi150 ceph-mon[39612]: 5.13 scrub starts 2024-09-11T08:25:54.244 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:54.267 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:54.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:54.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:54.369 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:54.394 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:54.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:54.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:54.495 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:25:54.518 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:54.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:54.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:25:54.620 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:54.641 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:54.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:54.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:54.743 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:54.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:54.768 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:54.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:54.869 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:25:54.891 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:54.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:54.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:25:54.993 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:55.016 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:55.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:55.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:55.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:54 smithi181 ceph-mon[32701]: 4.8 scrub starts 2024-09-11T08:25:55.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:54 smithi181 ceph-mon[32701]: 6.0 scrub starts 2024-09-11T08:25:55.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:54 smithi181 ceph-mon[32701]: 6.19 scrub starts 2024-09-11T08:25:55.118 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:25:55.140 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:55.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:55.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:25:55.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:54 smithi150 ceph-mon[29276]: 4.8 scrub starts 2024-09-11T08:25:55.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:54 smithi150 ceph-mon[29276]: 6.0 scrub starts 2024-09-11T08:25:55.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:54 smithi150 ceph-mon[29276]: 6.19 scrub starts 2024-09-11T08:25:55.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:54 smithi150 ceph-mon[39612]: 4.8 scrub starts 2024-09-11T08:25:55.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:54 smithi150 ceph-mon[39612]: 6.0 scrub starts 2024-09-11T08:25:55.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:54 smithi150 ceph-mon[39612]: 6.19 scrub starts 2024-09-11T08:25:55.243 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:55.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:55.265 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:55.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:55.366 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:25:55.389 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:55.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:55.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:25:55.491 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:25:55.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:55.513 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:55.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:25:55.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:25:55 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:25:55] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:25:55.614 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:55.635 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:55.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:55.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:55.737 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:55.759 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:55.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:55.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:55.861 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:25:55.910 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:55.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:55.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:25:55.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:55 smithi150 ceph-mon[29276]: 6.12 deep-scrub starts 2024-09-11T08:25:55.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:55 smithi150 ceph-mon[29276]: pgmap v836: 148 pgs: 2 active+recovering+undersized+remapped, 146 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 280 op/s; 0 B/s, 434 objects/s recovering 2024-09-11T08:25:55.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:55 smithi150 ceph-mon[29276]: osdmap e373: 8 total, 8 up, 8 in 2024-09-11T08:25:55.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:55 smithi150 ceph-mon[39612]: 6.12 deep-scrub starts 2024-09-11T08:25:55.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:55 smithi150 ceph-mon[39612]: pgmap v836: 148 pgs: 2 active+recovering+undersized+remapped, 146 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 280 op/s; 0 B/s, 434 objects/s recovering 2024-09-11T08:25:55.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:55 smithi150 ceph-mon[39612]: osdmap e373: 8 total, 8 up, 8 in 2024-09-11T08:25:56.012 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:56.034 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:56.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:56.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:56.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:55 smithi181 ceph-mon[32701]: 6.12 deep-scrub starts 2024-09-11T08:25:56.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:55 smithi181 ceph-mon[32701]: pgmap v836: 148 pgs: 2 active+recovering+undersized+remapped, 146 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 280 op/s; 0 B/s, 434 objects/s recovering 2024-09-11T08:25:56.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:55 smithi181 ceph-mon[32701]: osdmap e373: 8 total, 8 up, 8 in 2024-09-11T08:25:56.136 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:25:56.159 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:56.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:56.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:25:56.261 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:25:56.281 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:56.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:56.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:25:56.384 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:25:56.405 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:56.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:56.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:25:56.507 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:56.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:56.530 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:56.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:56.631 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:56.653 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:56.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:56.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:56.755 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:56.778 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:56.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:56.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:56.880 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:25:56.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:56.902 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:56.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:25:57.004 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:57.025 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:57.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:57.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:57.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:56 smithi181 ceph-mon[32701]: 3.10 deep-scrub starts 2024-09-11T08:25:57.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:56 smithi181 ceph-mon[32701]: 5.4 scrub starts 2024-09-11T08:25:57.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:56 smithi181 ceph-mon[32701]: 5.1 scrub starts 2024-09-11T08:25:57.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:56 smithi181 ceph-mon[32701]: osdmap e374: 8 total, 8 up, 8 in 2024-09-11T08:25:57.127 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:57.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:57.151 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:57.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:57.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:56 smithi150 ceph-mon[29276]: 3.10 deep-scrub starts 2024-09-11T08:25:57.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:56 smithi150 ceph-mon[29276]: 5.4 scrub starts 2024-09-11T08:25:57.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:56 smithi150 ceph-mon[29276]: 5.1 scrub starts 2024-09-11T08:25:57.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:56 smithi150 ceph-mon[29276]: osdmap e374: 8 total, 8 up, 8 in 2024-09-11T08:25:57.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:56 smithi150 ceph-mon[39612]: 3.10 deep-scrub starts 2024-09-11T08:25:57.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:56 smithi150 ceph-mon[39612]: 5.4 scrub starts 2024-09-11T08:25:57.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:56 smithi150 ceph-mon[39612]: 5.1 scrub starts 2024-09-11T08:25:57.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:56 smithi150 ceph-mon[39612]: osdmap e374: 8 total, 8 up, 8 in 2024-09-11T08:25:57.252 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:57.274 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:57.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:57.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:57.376 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:25:57.399 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:57.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:57.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:25:57.501 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:57.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:57.525 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:57.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:57.626 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:25:57.648 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:57.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:57.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:25:57.750 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:57.773 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:57.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:57.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:57.875 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:25:57.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:57.898 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:57.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:25:57.999 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:58.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:58.023 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:58.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:58.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:57 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:25:58.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:57 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:25:58.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:57 smithi181 ceph-mon[32701]: 6.1c scrub starts 2024-09-11T08:25:58.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:57 smithi181 ceph-mon[32701]: 4.18 scrub starts 2024-09-11T08:25:58.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:57 smithi181 ceph-mon[32701]: 4.18 scrub ok 2024-09-11T08:25:58.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:57 smithi181 ceph-mon[32701]: pgmap v839: 148 pgs: 2 active+recovering+undersized+remapped, 146 active+clean; 582 KiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 0 B/s, 403 objects/s recovering 2024-09-11T08:25:58.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:57 smithi181 ceph-mon[32701]: 5.1d scrub starts 2024-09-11T08:25:58.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:57 smithi181 ceph-mon[32701]: 5.1d scrub ok 2024-09-11T08:25:58.081 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:57 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:25:58.081 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:57 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:25:58.082 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:57 smithi150 ceph-mon[29276]: 6.1c scrub starts 2024-09-11T08:25:58.082 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:57 smithi150 ceph-mon[29276]: 4.18 scrub starts 2024-09-11T08:25:58.082 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:57 smithi150 ceph-mon[29276]: 4.18 scrub ok 2024-09-11T08:25:58.082 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:57 smithi150 ceph-mon[29276]: pgmap v839: 148 pgs: 2 active+recovering+undersized+remapped, 146 active+clean; 582 KiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 0 B/s, 403 objects/s recovering 2024-09-11T08:25:58.082 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:57 smithi150 ceph-mon[29276]: 5.1d scrub starts 2024-09-11T08:25:58.082 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:57 smithi150 ceph-mon[29276]: 5.1d scrub ok 2024-09-11T08:25:58.082 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:57 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:25:58.082 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:57 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:25:58.082 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:57 smithi150 ceph-mon[39612]: 6.1c scrub starts 2024-09-11T08:25:58.082 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:57 smithi150 ceph-mon[39612]: 4.18 scrub starts 2024-09-11T08:25:58.083 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:57 smithi150 ceph-mon[39612]: 4.18 scrub ok 2024-09-11T08:25:58.083 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:57 smithi150 ceph-mon[39612]: pgmap v839: 148 pgs: 2 active+recovering+undersized+remapped, 146 active+clean; 582 KiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 0 B/s, 403 objects/s recovering 2024-09-11T08:25:58.083 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:57 smithi150 ceph-mon[39612]: 5.1d scrub starts 2024-09-11T08:25:58.083 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:57 smithi150 ceph-mon[39612]: 5.1d scrub ok 2024-09-11T08:25:58.108 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd reweight 4 1 2024-09-11T08:25:58.125 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:25:58.146 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:58.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:58.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:25:58.249 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:25:58.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:58.271 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:58.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:25:58.372 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:58.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:58.394 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:58.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:58.495 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:25:58.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:58.518 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:58.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:25:58.619 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:58.640 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:58.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:58.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:58.742 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:25:58.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:58.764 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:58.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:25:58.866 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:58.887 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:58.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:58.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:58.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:58 smithi150 ceph-mon[29276]: 3.e scrub starts 2024-09-11T08:25:58.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:58 smithi150 ceph-mon[29276]: 3.e scrub ok 2024-09-11T08:25:58.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:58 smithi150 ceph-mon[29276]: 3.9 scrub starts 2024-09-11T08:25:58.989 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:25:58 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:25:58] "GET /metrics HTTP/1.1" 200 224407 "" "Prometheus/2.43.0" 2024-09-11T08:25:58.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:58 smithi150 ceph-mon[39612]: 3.e scrub starts 2024-09-11T08:25:58.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:58 smithi150 ceph-mon[39612]: 3.e scrub ok 2024-09-11T08:25:58.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:58 smithi150 ceph-mon[39612]: 3.9 scrub starts 2024-09-11T08:25:58.989 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:25:59.010 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:59.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:59.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:25:59.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:58 smithi181 ceph-mon[32701]: 3.e scrub starts 2024-09-11T08:25:59.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:58 smithi181 ceph-mon[32701]: 3.e scrub ok 2024-09-11T08:25:59.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:58 smithi181 ceph-mon[32701]: 3.9 scrub starts 2024-09-11T08:25:59.073 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:25:58 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:25:59.113 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:25:59.134 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:59.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:59.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:25:59.236 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:59.258 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:59.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:59.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:59.360 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:25:59.381 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:59.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:59.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:25:59.483 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:25:59.505 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:25:59.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:59.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:25:59.607 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:25:59.630 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:59.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:59.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:25:59.733 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:59.755 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:59.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:59.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:59.857 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:25:59.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:25:59.880 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:25:59.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:25:59.981 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:00.003 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:00.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:00.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:00.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:59 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:26:00.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:59 smithi181 ceph-mon[32701]: 5.9 scrub starts 2024-09-11T08:26:00.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:59 smithi181 ceph-mon[32701]: 5.9 scrub ok 2024-09-11T08:26:00.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:59 smithi181 ceph-mon[32701]: pgmap v840: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 582 KiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s; 0 B/s, 850 objects/s recovering 2024-09-11T08:26:00.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:59 smithi181 ceph-mon[32701]: 5.16 scrub starts 2024-09-11T08:26:00.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:25:59 smithi181 ceph-mon[32701]: 5.16 scrub ok 2024-09-11T08:26:00.105 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:00.128 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:00.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:00.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:00.230 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:00.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:59 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:26:00.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:59 smithi150 ceph-mon[29276]: 5.9 scrub starts 2024-09-11T08:26:00.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:59 smithi150 ceph-mon[29276]: 5.9 scrub ok 2024-09-11T08:26:00.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:59 smithi150 ceph-mon[29276]: pgmap v840: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 582 KiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s; 0 B/s, 850 objects/s recovering 2024-09-11T08:26:00.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:59 smithi150 ceph-mon[29276]: 5.16 scrub starts 2024-09-11T08:26:00.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:25:59 smithi150 ceph-mon[29276]: 5.16 scrub ok 2024-09-11T08:26:00.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:59 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:26:00.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:59 smithi150 ceph-mon[39612]: 5.9 scrub starts 2024-09-11T08:26:00.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:59 smithi150 ceph-mon[39612]: 5.9 scrub ok 2024-09-11T08:26:00.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:59 smithi150 ceph-mon[39612]: pgmap v840: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 582 KiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s; 0 B/s, 850 objects/s recovering 2024-09-11T08:26:00.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:59 smithi150 ceph-mon[39612]: 5.16 scrub starts 2024-09-11T08:26:00.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:25:59 smithi150 ceph-mon[39612]: 5.16 scrub ok 2024-09-11T08:26:00.252 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:00.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:00.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:00.354 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:00.377 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:00.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:00.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:00.479 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:00.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:00.504 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:00.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:00.605 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:00.627 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:00.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:00.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:00.729 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:00.751 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:00.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:00.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:00.853 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:26:00.875 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:00.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:00.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:26:00.977 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:00.999 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:00.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:01.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:01.027 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:00 smithi150 ceph-mon[29276]: 6.17 scrub starts 2024-09-11T08:26:01.027 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:00 smithi150 ceph-mon[29276]: 6.17 scrub ok 2024-09-11T08:26:01.027 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:00 smithi150 ceph-mon[29276]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-11T08:26:01.028 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:00 smithi150 ceph-mon[29276]: osdmap e375: 8 total, 8 up, 8 in 2024-09-11T08:26:01.028 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:00 smithi150 ceph-mon[29276]: 10.7 scrub starts 2024-09-11T08:26:01.028 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:00 smithi150 ceph-mon[29276]: 10.7 scrub ok 2024-09-11T08:26:01.028 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:00 smithi150 ceph-mon[29276]: 5.e scrub starts 2024-09-11T08:26:01.028 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:00 smithi150 ceph-mon[29276]: 5.e scrub ok 2024-09-11T08:26:01.028 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:00 smithi150 ceph-mon[39612]: 6.17 scrub starts 2024-09-11T08:26:01.028 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:00 smithi150 ceph-mon[39612]: 6.17 scrub ok 2024-09-11T08:26:01.028 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:00 smithi150 ceph-mon[39612]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-11T08:26:01.028 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:00 smithi150 ceph-mon[39612]: osdmap e375: 8 total, 8 up, 8 in 2024-09-11T08:26:01.028 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:00 smithi150 ceph-mon[39612]: 10.7 scrub starts 2024-09-11T08:26:01.028 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:00 smithi150 ceph-mon[39612]: 10.7 scrub ok 2024-09-11T08:26:01.029 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:00 smithi150 ceph-mon[39612]: 5.e scrub starts 2024-09-11T08:26:01.029 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:00 smithi150 ceph-mon[39612]: 5.e scrub ok 2024-09-11T08:26:01.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:00 smithi181 ceph-mon[32701]: 6.17 scrub starts 2024-09-11T08:26:01.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:00 smithi181 ceph-mon[32701]: 6.17 scrub ok 2024-09-11T08:26:01.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:00 smithi181 ceph-mon[32701]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-11T08:26:01.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:00 smithi181 ceph-mon[32701]: osdmap e375: 8 total, 8 up, 8 in 2024-09-11T08:26:01.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:00 smithi181 ceph-mon[32701]: 10.7 scrub starts 2024-09-11T08:26:01.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:00 smithi181 ceph-mon[32701]: 10.7 scrub ok 2024-09-11T08:26:01.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:00 smithi181 ceph-mon[32701]: 5.e scrub starts 2024-09-11T08:26:01.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:00 smithi181 ceph-mon[32701]: 5.e scrub ok 2024-09-11T08:26:01.101 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:01.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:01.126 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:01.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:01.227 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:01.250 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:01.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:01.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:01.352 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:01.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:01.374 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:01.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:01.475 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:26:01.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:01.502 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:01.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:26:01.603 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:01.626 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:01.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:01.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:01.727 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:01.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:01.750 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:01.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:01.851 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:26:01.875 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:01.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:01.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:26:01.976 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:01.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:01 smithi150 ceph-mon[29276]: 5.f scrub starts 2024-09-11T08:26:01.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:01 smithi150 ceph-mon[29276]: 5.f scrub ok 2024-09-11T08:26:01.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:01 smithi150 ceph-mon[29276]: 6.7 scrub starts 2024-09-11T08:26:01.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:01 smithi150 ceph-mon[29276]: 6.7 scrub ok 2024-09-11T08:26:01.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:01 smithi150 ceph-mon[29276]: osdmap e376: 8 total, 8 up, 8 in 2024-09-11T08:26:01.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:01 smithi150 ceph-mon[29276]: pgmap v843: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 582 KiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 944 B/s rd, 0 op/s; 0 B/s, 552 objects/s recovering 2024-09-11T08:26:01.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:01 smithi150 ceph-mon[39612]: 5.f scrub starts 2024-09-11T08:26:01.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:01 smithi150 ceph-mon[39612]: 5.f scrub ok 2024-09-11T08:26:01.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:01 smithi150 ceph-mon[39612]: 6.7 scrub starts 2024-09-11T08:26:01.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:01 smithi150 ceph-mon[39612]: 6.7 scrub ok 2024-09-11T08:26:01.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:01 smithi150 ceph-mon[39612]: osdmap e376: 8 total, 8 up, 8 in 2024-09-11T08:26:01.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:01 smithi150 ceph-mon[39612]: pgmap v843: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 582 KiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 944 B/s rd, 0 op/s; 0 B/s, 552 objects/s recovering 2024-09-11T08:26:01.999 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:02.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:02.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:02.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:01 smithi181 ceph-mon[32701]: 5.f scrub starts 2024-09-11T08:26:02.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:01 smithi181 ceph-mon[32701]: 5.f scrub ok 2024-09-11T08:26:02.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:01 smithi181 ceph-mon[32701]: 6.7 scrub starts 2024-09-11T08:26:02.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:01 smithi181 ceph-mon[32701]: 6.7 scrub ok 2024-09-11T08:26:02.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:01 smithi181 ceph-mon[32701]: osdmap e376: 8 total, 8 up, 8 in 2024-09-11T08:26:02.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:01 smithi181 ceph-mon[32701]: pgmap v843: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 582 KiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 944 B/s rd, 0 op/s; 0 B/s, 552 objects/s recovering 2024-09-11T08:26:02.101 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:02.123 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:02.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:02.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:02.141 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:26:02.225 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:26:02.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:02.248 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:02.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:26:02.349 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:02.371 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:02.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:02.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:02.473 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:02.494 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:02.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:02.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:02.596 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:02.621 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:02.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:02.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:02.723 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:02.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:02.746 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:02.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:02.848 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:02.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:02.872 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:02.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:02.973 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:02.995 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:02.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:02.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:03.019 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:26:03.020 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:26:03.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:02 smithi181 ceph-mon[32701]: 5.d scrub starts 2024-09-11T08:26:03.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:02 smithi181 ceph-mon[32701]: 5.d scrub ok 2024-09-11T08:26:03.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:02 smithi181 ceph-mon[32701]: 5.1 scrub starts 2024-09-11T08:26:03.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:02 smithi181 ceph-mon[32701]: 5.1 scrub ok 2024-09-11T08:26:03.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:02 smithi181 ceph-mon[32701]: 4.1a scrub starts 2024-09-11T08:26:03.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:02 smithi181 ceph-mon[32701]: 3.14 scrub starts 2024-09-11T08:26:03.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:02 smithi181 ceph-mon[32701]: 4.1a scrub ok 2024-09-11T08:26:03.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:02 smithi181 ceph-mon[32701]: 3.14 scrub ok 2024-09-11T08:26:03.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:02 smithi181 ceph-mon[32701]: 3.5 scrub starts 2024-09-11T08:26:03.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:02 smithi181 ceph-mon[32701]: 3.5 scrub ok 2024-09-11T08:26:03.097 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:03.119 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:03.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:03.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:03.143 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:02 smithi150 ceph-mon[29276]: 5.d scrub starts 2024-09-11T08:26:03.143 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:02 smithi150 ceph-mon[29276]: 5.d scrub ok 2024-09-11T08:26:03.143 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:02 smithi150 ceph-mon[29276]: 5.1 scrub starts 2024-09-11T08:26:03.143 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:02 smithi150 ceph-mon[29276]: 5.1 scrub ok 2024-09-11T08:26:03.143 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:02 smithi150 ceph-mon[29276]: 4.1a scrub starts 2024-09-11T08:26:03.143 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:02 smithi150 ceph-mon[29276]: 3.14 scrub starts 2024-09-11T08:26:03.143 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:02 smithi150 ceph-mon[29276]: 4.1a scrub ok 2024-09-11T08:26:03.143 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:02 smithi150 ceph-mon[29276]: 3.14 scrub ok 2024-09-11T08:26:03.143 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:02 smithi150 ceph-mon[29276]: 3.5 scrub starts 2024-09-11T08:26:03.143 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:02 smithi150 ceph-mon[29276]: 3.5 scrub ok 2024-09-11T08:26:03.144 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:02 smithi150 ceph-mon[39612]: 5.d scrub starts 2024-09-11T08:26:03.144 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:02 smithi150 ceph-mon[39612]: 5.d scrub ok 2024-09-11T08:26:03.144 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:02 smithi150 ceph-mon[39612]: 5.1 scrub starts 2024-09-11T08:26:03.144 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:02 smithi150 ceph-mon[39612]: 5.1 scrub ok 2024-09-11T08:26:03.144 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:02 smithi150 ceph-mon[39612]: 4.1a scrub starts 2024-09-11T08:26:03.144 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:02 smithi150 ceph-mon[39612]: 3.14 scrub starts 2024-09-11T08:26:03.144 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:02 smithi150 ceph-mon[39612]: 4.1a scrub ok 2024-09-11T08:26:03.144 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:02 smithi150 ceph-mon[39612]: 3.14 scrub ok 2024-09-11T08:26:03.144 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:02 smithi150 ceph-mon[39612]: 3.5 scrub starts 2024-09-11T08:26:03.144 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:02 smithi150 ceph-mon[39612]: 3.5 scrub ok 2024-09-11T08:26:03.221 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:26:03.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:03.243 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:03.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:26:03.344 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:26:03.366 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:03.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:03.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:26:03.467 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:03.491 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:03.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:03.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:03.593 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:26:03.615 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:03.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:03.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:26:03.716 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:26:03.737 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:03.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:03.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:26:03.839 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:03.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:03.865 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:03.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:03.966 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:03.990 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:03.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:03.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:04.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:03 smithi181 ceph-mon[32701]: 5.3 deep-scrub starts 2024-09-11T08:26:04.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:03 smithi181 ceph-mon[32701]: 5.3 deep-scrub ok 2024-09-11T08:26:04.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:03 smithi181 ceph-mon[32701]: 5.10 scrub starts 2024-09-11T08:26:04.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:03 smithi181 ceph-mon[32701]: 5.10 scrub ok 2024-09-11T08:26:04.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:03 smithi181 ceph-mon[32701]: pgmap v844: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 797 B/s rd, 0 op/s; 0 B/s, 466 objects/s recovering 2024-09-11T08:26:04.077 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:03 smithi150 ceph-mon[39612]: 5.3 deep-scrub starts 2024-09-11T08:26:04.077 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:03 smithi150 ceph-mon[39612]: 5.3 deep-scrub ok 2024-09-11T08:26:04.077 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:03 smithi150 ceph-mon[39612]: 5.10 scrub starts 2024-09-11T08:26:04.077 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:03 smithi150 ceph-mon[39612]: 5.10 scrub ok 2024-09-11T08:26:04.077 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:03 smithi150 ceph-mon[39612]: pgmap v844: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 797 B/s rd, 0 op/s; 0 B/s, 466 objects/s recovering 2024-09-11T08:26:04.078 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:03 smithi150 ceph-mon[29276]: 5.3 deep-scrub starts 2024-09-11T08:26:04.078 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:03 smithi150 ceph-mon[29276]: 5.3 deep-scrub ok 2024-09-11T08:26:04.078 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:03 smithi150 ceph-mon[29276]: 5.10 scrub starts 2024-09-11T08:26:04.078 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:03 smithi150 ceph-mon[29276]: 5.10 scrub ok 2024-09-11T08:26:04.078 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:03 smithi150 ceph-mon[29276]: pgmap v844: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 797 B/s rd, 0 op/s; 0 B/s, 466 objects/s recovering 2024-09-11T08:26:04.091 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:26:04.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:04.114 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:04.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:26:04.215 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:04.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:04.241 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:04.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:04.342 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:04.366 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:04.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:04.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:04.468 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:26:04.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:04.491 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:04.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:26:04.592 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:04.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:04.613 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:04.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:04.714 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:04.740 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:04.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:04.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:04.842 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:04.866 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:04.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:04.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:04.968 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:04.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:04 smithi150 ceph-mon[39612]: 4.17 deep-scrub starts 2024-09-11T08:26:04.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:04 smithi150 ceph-mon[39612]: 4.17 deep-scrub ok 2024-09-11T08:26:04.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:04 smithi150 ceph-mon[39612]: 5.1b scrub starts 2024-09-11T08:26:04.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:04 smithi150 ceph-mon[39612]: 5.1b scrub ok 2024-09-11T08:26:04.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:04 smithi150 ceph-mon[39612]: 3.1a scrub starts 2024-09-11T08:26:04.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:04 smithi150 ceph-mon[39612]: 3.1a scrub ok 2024-09-11T08:26:04.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:04 smithi150 ceph-mon[29276]: 4.17 deep-scrub starts 2024-09-11T08:26:04.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:04 smithi150 ceph-mon[29276]: 4.17 deep-scrub ok 2024-09-11T08:26:04.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:04 smithi150 ceph-mon[29276]: 5.1b scrub starts 2024-09-11T08:26:04.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:04 smithi150 ceph-mon[29276]: 5.1b scrub ok 2024-09-11T08:26:04.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:04 smithi150 ceph-mon[29276]: 3.1a scrub starts 2024-09-11T08:26:04.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:04 smithi150 ceph-mon[29276]: 3.1a scrub ok 2024-09-11T08:26:04.991 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:04.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:04.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:05.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:04 smithi181 ceph-mon[32701]: 4.17 deep-scrub starts 2024-09-11T08:26:05.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:04 smithi181 ceph-mon[32701]: 4.17 deep-scrub ok 2024-09-11T08:26:05.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:04 smithi181 ceph-mon[32701]: 5.1b scrub starts 2024-09-11T08:26:05.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:04 smithi181 ceph-mon[32701]: 5.1b scrub ok 2024-09-11T08:26:05.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:04 smithi181 ceph-mon[32701]: 3.1a scrub starts 2024-09-11T08:26:05.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:04 smithi181 ceph-mon[32701]: 3.1a scrub ok 2024-09-11T08:26:05.093 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:05.117 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:05.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:05.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:05.219 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:05.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:05.242 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:05.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:05.343 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:05.370 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:05.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:05.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:05.473 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:05.499 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:05.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:05.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:05.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:26:05 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:26:05] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:26:05.602 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:05.627 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:05.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:05.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:05.729 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:05.754 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:05.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:05.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:05.856 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:05.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:05.881 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:05.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:05.886 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:05 smithi150 ceph-mon[29276]: 10.3 scrub starts 2024-09-11T08:26:05.887 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:05 smithi150 ceph-mon[29276]: 10.3 scrub ok 2024-09-11T08:26:05.887 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:05 smithi150 ceph-mon[29276]: pgmap v845: 148 pgs: 148 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 415 objects/s recovering 2024-09-11T08:26:05.887 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:05 smithi150 ceph-mon[39612]: 10.3 scrub starts 2024-09-11T08:26:05.887 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:05 smithi150 ceph-mon[39612]: 10.3 scrub ok 2024-09-11T08:26:05.888 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:05 smithi150 ceph-mon[39612]: pgmap v845: 148 pgs: 148 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 415 objects/s recovering 2024-09-11T08:26:05.982 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:06.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:06.006 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:06.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:06.026 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-09-11T08:26:06.026 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:26:06.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:05 smithi181 ceph-mon[32701]: 10.3 scrub starts 2024-09-11T08:26:06.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:05 smithi181 ceph-mon[32701]: 10.3 scrub ok 2024-09-11T08:26:06.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:05 smithi181 ceph-mon[32701]: pgmap v845: 148 pgs: 148 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 415 objects/s recovering 2024-09-11T08:26:06.107 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:06.131 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:06.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:06.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:06.233 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:06.256 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:06.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:06.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:06.358 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:06.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:06.382 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:06.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:06.483 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:06.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:06.508 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:06.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:06.609 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:06.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:06.634 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:06.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:06.735 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:06.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:06.760 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:06.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:06.861 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:06.884 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:06.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:06.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:06.986 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:07.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:07.009 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:07.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:07.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:06 smithi181 ceph-mon[32701]: 5.6 scrub starts 2024-09-11T08:26:07.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:06 smithi181 ceph-mon[32701]: 5.6 scrub ok 2024-09-11T08:26:07.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:06 smithi181 ceph-mon[32701]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-11T08:26:07.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:06 smithi181 ceph-mon[32701]: Cluster is now healthy 2024-09-11T08:26:07.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:06 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1243526272' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:26:07.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:06 smithi181 ceph-mon[32701]: 3.1b scrub starts 2024-09-11T08:26:07.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:06 smithi181 ceph-mon[32701]: 3.1b scrub ok 2024-09-11T08:26:07.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:06 smithi181 ceph-mon[32701]: 6.e scrub starts 2024-09-11T08:26:07.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:06 smithi181 ceph-mon[32701]: 6.e scrub ok 2024-09-11T08:26:07.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:06 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:26:07.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:06 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:26:07.110 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:07.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:07.134 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:07.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:07.235 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:07.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:06 smithi150 ceph-mon[39612]: 5.6 scrub starts 2024-09-11T08:26:07.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:06 smithi150 ceph-mon[39612]: 5.6 scrub ok 2024-09-11T08:26:07.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:06 smithi150 ceph-mon[39612]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-11T08:26:07.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:06 smithi150 ceph-mon[39612]: Cluster is now healthy 2024-09-11T08:26:07.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:06 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1243526272' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:26:07.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:06 smithi150 ceph-mon[39612]: 3.1b scrub starts 2024-09-11T08:26:07.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:06 smithi150 ceph-mon[39612]: 3.1b scrub ok 2024-09-11T08:26:07.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:06 smithi150 ceph-mon[39612]: 6.e scrub starts 2024-09-11T08:26:07.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:06 smithi150 ceph-mon[39612]: 6.e scrub ok 2024-09-11T08:26:07.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:06 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:26:07.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:06 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:26:07.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:06 smithi150 ceph-mon[29276]: 5.6 scrub starts 2024-09-11T08:26:07.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:06 smithi150 ceph-mon[29276]: 5.6 scrub ok 2024-09-11T08:26:07.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:06 smithi150 ceph-mon[29276]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-11T08:26:07.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:06 smithi150 ceph-mon[29276]: Cluster is now healthy 2024-09-11T08:26:07.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:06 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1243526272' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:26:07.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:06 smithi150 ceph-mon[29276]: 3.1b scrub starts 2024-09-11T08:26:07.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:06 smithi150 ceph-mon[29276]: 3.1b scrub ok 2024-09-11T08:26:07.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:06 smithi150 ceph-mon[29276]: 6.e scrub starts 2024-09-11T08:26:07.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:06 smithi150 ceph-mon[29276]: 6.e scrub ok 2024-09-11T08:26:07.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:06 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:26:07.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:06 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:26:07.257 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:07.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:07.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:07.359 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:07.381 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:07.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:07.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:07.483 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:26:07.505 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:07.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:07.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:26:07.607 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:07.633 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:07.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:07.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:07.736 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:26:07.757 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:07.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:07.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:26:07.813 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:26:07.813 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:26:07.813 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:26:07.813 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:26:07.813 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:26:07.813 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:26:07.813 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:26:07.814 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:26:07.814 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:26:07.814 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:26:07.814 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:26:07.814 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:26:07.814 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:26:07.814 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:26:07.814 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:26:07.814 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:26:07.859 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:07.885 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:07.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:07.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:07.987 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:08.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:08.014 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:08.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:08.058 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:07 smithi150 ceph-mon[29276]: 5.b deep-scrub starts 2024-09-11T08:26:08.059 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:07 smithi150 ceph-mon[29276]: 5.b deep-scrub ok 2024-09-11T08:26:08.059 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:07 smithi150 ceph-mon[29276]: 6.13 scrub starts 2024-09-11T08:26:08.059 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:07 smithi150 ceph-mon[29276]: 6.13 scrub ok 2024-09-11T08:26:08.059 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:07 smithi150 ceph-mon[29276]: 6.a scrub starts 2024-09-11T08:26:08.059 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:07 smithi150 ceph-mon[29276]: 6.a scrub ok 2024-09-11T08:26:08.059 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:07 smithi150 ceph-mon[29276]: 3.9 scrub starts 2024-09-11T08:26:08.059 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:07 smithi150 ceph-mon[29276]: 3.9 scrub ok 2024-09-11T08:26:08.059 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:07 smithi150 ceph-mon[29276]: pgmap v846: 148 pgs: 148 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 0 B/s, 41 objects/s recovering 2024-09-11T08:26:08.059 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:07 smithi150 ceph-mon[29276]: 4.12 scrub starts 2024-09-11T08:26:08.059 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:07 smithi150 ceph-mon[29276]: 4.12 scrub ok 2024-09-11T08:26:08.059 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:07 smithi150 ceph-mon[39612]: 5.b deep-scrub starts 2024-09-11T08:26:08.060 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:07 smithi150 ceph-mon[39612]: 5.b deep-scrub ok 2024-09-11T08:26:08.060 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:07 smithi150 ceph-mon[39612]: 6.13 scrub starts 2024-09-11T08:26:08.060 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:07 smithi150 ceph-mon[39612]: 6.13 scrub ok 2024-09-11T08:26:08.060 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:07 smithi150 ceph-mon[39612]: 6.a scrub starts 2024-09-11T08:26:08.060 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:07 smithi150 ceph-mon[39612]: 6.a scrub ok 2024-09-11T08:26:08.060 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:07 smithi150 ceph-mon[39612]: 3.9 scrub starts 2024-09-11T08:26:08.060 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:07 smithi150 ceph-mon[39612]: 3.9 scrub ok 2024-09-11T08:26:08.060 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:07 smithi150 ceph-mon[39612]: pgmap v846: 148 pgs: 148 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 0 B/s, 41 objects/s recovering 2024-09-11T08:26:08.060 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:07 smithi150 ceph-mon[39612]: 4.12 scrub starts 2024-09-11T08:26:08.060 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:07 smithi150 ceph-mon[39612]: 4.12 scrub ok 2024-09-11T08:26:08.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:07 smithi181 ceph-mon[32701]: 5.b deep-scrub starts 2024-09-11T08:26:08.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:07 smithi181 ceph-mon[32701]: 5.b deep-scrub ok 2024-09-11T08:26:08.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:07 smithi181 ceph-mon[32701]: 6.13 scrub starts 2024-09-11T08:26:08.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:07 smithi181 ceph-mon[32701]: 6.13 scrub ok 2024-09-11T08:26:08.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:07 smithi181 ceph-mon[32701]: 6.a scrub starts 2024-09-11T08:26:08.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:07 smithi181 ceph-mon[32701]: 6.a scrub ok 2024-09-11T08:26:08.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:07 smithi181 ceph-mon[32701]: 3.9 scrub starts 2024-09-11T08:26:08.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:07 smithi181 ceph-mon[32701]: 3.9 scrub ok 2024-09-11T08:26:08.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:07 smithi181 ceph-mon[32701]: pgmap v846: 148 pgs: 148 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 0 B/s, 41 objects/s recovering 2024-09-11T08:26:08.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:07 smithi181 ceph-mon[32701]: 4.12 scrub starts 2024-09-11T08:26:08.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:07 smithi181 ceph-mon[32701]: 4.12 scrub ok 2024-09-11T08:26:08.115 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:08.136 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:08.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:08.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:08.238 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:26:08.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:08.261 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:08.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:26:08.362 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:26:08.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:08.384 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:08.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:26:08.485 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:08.511 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:08.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:08.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:08.613 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:08.635 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:08.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:08.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:08.737 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:26:08.745 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:26:08 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:26:08] "GET /metrics HTTP/1.1" 200 224336 "" "Prometheus/2.43.0" 2024-09-11T08:26:08.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:08.761 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:08.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:26:08.862 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:08.887 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:08.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:08.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:08.989 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:26:09.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:09.013 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:09.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:26:09.073 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:26:08 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:26:09.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:08 smithi181 ceph-mon[32701]: 10.9 scrub starts 2024-09-11T08:26:09.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:08 smithi181 ceph-mon[32701]: 10.9 scrub ok 2024-09-11T08:26:09.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:08 smithi181 ceph-mon[32701]: 3.2 deep-scrub starts 2024-09-11T08:26:09.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:08 smithi181 ceph-mon[32701]: 3.2 deep-scrub ok 2024-09-11T08:26:09.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:08 smithi181 ceph-mon[32701]: 3.7 scrub starts 2024-09-11T08:26:09.075 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:08 smithi181 ceph-mon[32701]: 3.7 scrub ok 2024-09-11T08:26:09.114 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:09.141 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:09.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:09.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:09.150 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:08 smithi150 ceph-mon[29276]: 10.9 scrub starts 2024-09-11T08:26:09.150 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:08 smithi150 ceph-mon[29276]: 10.9 scrub ok 2024-09-11T08:26:09.150 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:08 smithi150 ceph-mon[29276]: 3.2 deep-scrub starts 2024-09-11T08:26:09.150 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:08 smithi150 ceph-mon[29276]: 3.2 deep-scrub ok 2024-09-11T08:26:09.150 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:08 smithi150 ceph-mon[29276]: 3.7 scrub starts 2024-09-11T08:26:09.150 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:08 smithi150 ceph-mon[29276]: 3.7 scrub ok 2024-09-11T08:26:09.150 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:08 smithi150 ceph-mon[39612]: 10.9 scrub starts 2024-09-11T08:26:09.150 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:08 smithi150 ceph-mon[39612]: 10.9 scrub ok 2024-09-11T08:26:09.150 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:08 smithi150 ceph-mon[39612]: 3.2 deep-scrub starts 2024-09-11T08:26:09.151 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:08 smithi150 ceph-mon[39612]: 3.2 deep-scrub ok 2024-09-11T08:26:09.151 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:08 smithi150 ceph-mon[39612]: 3.7 scrub starts 2024-09-11T08:26:09.151 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:08 smithi150 ceph-mon[39612]: 3.7 scrub ok 2024-09-11T08:26:09.243 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:09.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:09.266 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:09.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:09.367 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:26:09.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:09.389 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:09.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:26:09.490 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:09.516 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:09.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:09.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:09.617 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-11T08:26:09.618 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:09.642 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:09.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:09.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:09.744 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:26:09.766 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:09.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:09.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:26:09.868 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:09.894 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:09.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:09.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:09.937 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:09 smithi150 ceph-mon[29276]: 4.d scrub starts 2024-09-11T08:26:09.937 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:09 smithi150 ceph-mon[29276]: 4.d scrub ok 2024-09-11T08:26:09.937 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:09 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:26:09.937 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:09 smithi150 ceph-mon[29276]: 4.6 deep-scrub starts 2024-09-11T08:26:09.937 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:09 smithi150 ceph-mon[29276]: 4.6 deep-scrub ok 2024-09-11T08:26:09.937 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:09 smithi150 ceph-mon[29276]: 6.15 scrub starts 2024-09-11T08:26:09.938 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:09 smithi150 ceph-mon[29276]: 6.15 scrub ok 2024-09-11T08:26:09.938 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:09 smithi150 ceph-mon[29276]: pgmap v847: 148 pgs: 148 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 0 B/s, 35 objects/s recovering 2024-09-11T08:26:09.938 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:09 smithi150 ceph-mon[39612]: 4.d scrub starts 2024-09-11T08:26:09.938 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:09 smithi150 ceph-mon[39612]: 4.d scrub ok 2024-09-11T08:26:09.938 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:09 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:26:09.938 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:09 smithi150 ceph-mon[39612]: 4.6 deep-scrub starts 2024-09-11T08:26:09.938 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:09 smithi150 ceph-mon[39612]: 4.6 deep-scrub ok 2024-09-11T08:26:09.938 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:09 smithi150 ceph-mon[39612]: 6.15 scrub starts 2024-09-11T08:26:09.939 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:09 smithi150 ceph-mon[39612]: 6.15 scrub ok 2024-09-11T08:26:09.939 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:09 smithi150 ceph-mon[39612]: pgmap v847: 148 pgs: 148 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 0 B/s, 35 objects/s recovering 2024-09-11T08:26:09.996 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:26:10.018 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:10.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:10.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:26:10.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:09 smithi181 ceph-mon[32701]: 4.d scrub starts 2024-09-11T08:26:10.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:09 smithi181 ceph-mon[32701]: 4.d scrub ok 2024-09-11T08:26:10.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:09 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:26:10.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:09 smithi181 ceph-mon[32701]: 4.6 deep-scrub starts 2024-09-11T08:26:10.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:09 smithi181 ceph-mon[32701]: 4.6 deep-scrub ok 2024-09-11T08:26:10.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:09 smithi181 ceph-mon[32701]: 6.15 scrub starts 2024-09-11T08:26:10.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:09 smithi181 ceph-mon[32701]: 6.15 scrub ok 2024-09-11T08:26:10.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:09 smithi181 ceph-mon[32701]: pgmap v847: 148 pgs: 148 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 0 B/s, 35 objects/s recovering 2024-09-11T08:26:10.120 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:10.146 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:10.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:10.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:10.247 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:26:10.270 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:10.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:10.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:26:10.304 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:26:10.372 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:26:10.393 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:10.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:10.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:26:10.495 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:10.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:10.517 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:10.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:10.519 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T08:26:10.618 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:26:10.640 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:10.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:10.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:26:10.742 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:10.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:10.770 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:10.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:10.829 INFO:teuthology.orchestra.run.smithi150.stderr:reweighted osd.4 to 1 (10000) 2024-09-11T08:26:10.829 INFO:teuthology.orchestra.run.smithi150.stderr:noscrub is set 2024-09-11T08:26:10.871 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:26:10.893 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:10.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:10.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:26:10.995 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:11.021 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:11.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:11.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:11.126 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:11.152 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:11.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:11.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:11.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:10 smithi150 ceph-mon[29276]: 4.14 scrub starts 2024-09-11T08:26:11.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:10 smithi150 ceph-mon[29276]: 4.14 scrub ok 2024-09-11T08:26:11.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:10 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1947213286' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-11T08:26:11.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:10 smithi150 ceph-mon[29276]: 6.16 scrub starts 2024-09-11T08:26:11.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:10 smithi150 ceph-mon[29276]: 6.16 scrub ok 2024-09-11T08:26:11.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:10 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2288134508' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:26:11.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:10 smithi150 ceph-mon[29276]: 3.0 scrub starts 2024-09-11T08:26:11.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:10 smithi150 ceph-mon[29276]: 3.0 scrub ok 2024-09-11T08:26:11.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:10 smithi150 ceph-mon[39612]: 4.14 scrub starts 2024-09-11T08:26:11.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:10 smithi150 ceph-mon[39612]: 4.14 scrub ok 2024-09-11T08:26:11.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:10 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1947213286' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-11T08:26:11.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:10 smithi150 ceph-mon[39612]: 6.16 scrub starts 2024-09-11T08:26:11.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:10 smithi150 ceph-mon[39612]: 6.16 scrub ok 2024-09-11T08:26:11.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:10 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2288134508' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:26:11.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:10 smithi150 ceph-mon[39612]: 3.0 scrub starts 2024-09-11T08:26:11.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:10 smithi150 ceph-mon[39612]: 3.0 scrub ok 2024-09-11T08:26:11.254 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:11.278 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:11.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:11.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:11.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:10 smithi181 ceph-mon[32701]: 4.14 scrub starts 2024-09-11T08:26:11.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:10 smithi181 ceph-mon[32701]: 4.14 scrub ok 2024-09-11T08:26:11.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:10 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1947213286' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-11T08:26:11.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:10 smithi181 ceph-mon[32701]: 6.16 scrub starts 2024-09-11T08:26:11.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:10 smithi181 ceph-mon[32701]: 6.16 scrub ok 2024-09-11T08:26:11.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:10 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2288134508' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:26:11.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:10 smithi181 ceph-mon[32701]: 3.0 scrub starts 2024-09-11T08:26:11.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:10 smithi181 ceph-mon[32701]: 3.0 scrub ok 2024-09-11T08:26:11.380 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:26:11.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:11.403 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:11.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:26:11.504 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:26:11.527 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:11.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:11.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:26:11.629 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:11.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:11.657 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:11.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:11.758 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:11.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:11.781 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:11.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:11.882 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:11.908 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:11.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:11.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:11.983 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:11 smithi150 ceph-mon[39612]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:26:11.983 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:11 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1947213286' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-11T08:26:11.983 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:11 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2288134508' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:26:11.983 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:11 smithi150 ceph-mon[39612]: osdmap e377: 8 total, 8 up, 8 in 2024-09-11T08:26:11.983 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:11 smithi150 ceph-mon[39612]: pgmap v849: 148 pgs: 148 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 0 B/s, 33 objects/s recovering 2024-09-11T08:26:11.984 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:11 smithi150 ceph-mon[29276]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:26:11.984 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:11 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1947213286' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-11T08:26:11.984 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:11 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2288134508' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:26:11.984 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:11 smithi150 ceph-mon[29276]: osdmap e377: 8 total, 8 up, 8 in 2024-09-11T08:26:11.984 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:11 smithi150 ceph-mon[29276]: pgmap v849: 148 pgs: 148 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 0 B/s, 33 objects/s recovering 2024-09-11T08:26:11.985 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd reweight 6 1 2024-09-11T08:26:12.010 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:12.035 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:12.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:12.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:12.137 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:12.159 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:12.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:12.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:12.261 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:12.286 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:12.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:12.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:11 smithi181 ceph-mon[32701]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:26:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:11 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1947213286' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-11T08:26:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:11 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2288134508' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:26:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:11 smithi181 ceph-mon[32701]: osdmap e377: 8 total, 8 up, 8 in 2024-09-11T08:26:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:11 smithi181 ceph-mon[32701]: pgmap v849: 148 pgs: 148 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 0 B/s, 33 objects/s recovering 2024-09-11T08:26:12.388 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:12.409 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:12.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:12.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:12.511 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:12.537 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:12.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:12.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:12.639 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:26:12.660 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:12.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:12.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:26:12.763 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:12.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:12.785 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:12.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:12.886 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:12.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:12.909 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:12.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:13.010 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:13.036 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:13.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:13.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:13.138 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:13.164 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:13.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:13.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:13.266 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:13.288 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:13.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:13.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:13.390 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:26:13.412 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:13.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:13.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:26:13.514 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:13.535 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:13.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:13.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:13.610 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-09-11T08:26:13.611 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:26:13.637 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:26:13.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:13.659 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:13.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:26:13.760 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:13.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:13.784 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:13.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:13.885 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:13.911 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:13.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:13.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:14.013 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:26:14.035 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:14.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:14.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:26:14.137 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:14.162 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:14.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:14.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:14.186 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'unique_pool_3' removed 2024-09-11T08:26:14.264 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:14.286 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:14.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:14.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:14.358 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-11T08:26:14.387 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:14.409 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:14.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:14.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:14.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:14 smithi150 ceph-mon[29276]: pgmap v850: 148 pgs: 148 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 0 B/s, 33 objects/s recovering 2024-09-11T08:26:14.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:14 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2661279851' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-11T08:26:14.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:14 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-11T08:26:14.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:14 smithi150 ceph-mon[39612]: pgmap v850: 148 pgs: 148 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 0 B/s, 33 objects/s recovering 2024-09-11T08:26:14.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:14 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2661279851' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-11T08:26:14.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:14 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-11T08:26:14.511 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:14.536 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:14.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:14.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:14.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:14 smithi181 ceph-mon[32701]: pgmap v850: 148 pgs: 148 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 0 B/s, 33 objects/s recovering 2024-09-11T08:26:14.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:14 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2661279851' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-11T08:26:14.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:14 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-11T08:26:14.638 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:14.659 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:14.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:14.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:14.761 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:14.783 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:14.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:14.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:14.885 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:14.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:14.909 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:14.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:15.010 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:15.031 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:15.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:15.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:15.133 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:26:15.154 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:15.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:15.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:26:15.177 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:26:15.178 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:26:15.256 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:15.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:15.279 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:15.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:15.380 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:15.402 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:15.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:15.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:15.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:15 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-11T08:26:15.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:15 smithi150 ceph-mon[29276]: osdmap e378: 8 total, 8 up, 8 in 2024-09-11T08:26:15.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:15 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-11T08:26:15.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:15 smithi150 ceph-mon[39612]: osdmap e378: 8 total, 8 up, 8 in 2024-09-11T08:26:15.504 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:15.525 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:15.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:15.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:15.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:15 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-11T08:26:15.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:15 smithi181 ceph-mon[32701]: osdmap e378: 8 total, 8 up, 8 in 2024-09-11T08:26:15.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:26:15 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:26:15] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:26:15.626 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:15.651 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:15.652 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-11T08:26:15.653 INFO:tasks.radosbench:Beginning radosbench... 2024-09-11T08:26:15.653 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-09-11T08:26:15.653 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd pool create unique_pool_4 16 2024-09-11T08:26:15.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:15.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:15.755 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:15.777 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:15.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:15.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:15.878 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:15.900 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:15.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:15.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:16.001 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:16.022 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:16.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:16.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:16.124 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:16.145 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:16.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:16.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:16.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:16 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:26:16.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:16 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:26:16.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:16 smithi150 ceph-mon[29276]: pgmap v852: 132 pgs: 132 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-11T08:26:16.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:16 smithi150 ceph-mon[29276]: osdmap e379: 8 total, 8 up, 8 in 2024-09-11T08:26:16.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:16 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:26:16.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:16 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:26:16.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:16 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:26:16.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:16 smithi150 ceph-mon[39612]: pgmap v852: 132 pgs: 132 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-11T08:26:16.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:16 smithi150 ceph-mon[39612]: osdmap e379: 8 total, 8 up, 8 in 2024-09-11T08:26:16.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:16 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:26:16.247 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:16.269 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:16.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:16.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:16.370 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:16.391 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:16.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:16.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:16.493 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:16.514 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:16.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:16.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:16.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:16 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:26:16.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:16 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:26:16.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:16 smithi181 ceph-mon[32701]: pgmap v852: 132 pgs: 132 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-11T08:26:16.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:16 smithi181 ceph-mon[32701]: osdmap e379: 8 total, 8 up, 8 in 2024-09-11T08:26:16.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:16 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:26:16.616 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:16.636 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:16.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:16.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:16.738 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:26:16.759 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:16.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:16.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:26:16.785 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:26:16.862 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:16.883 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:16.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:16.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:16.985 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:17.006 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:17.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:17.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:17.108 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:17.127 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T08:26:17.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:17.133 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:17.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:17.235 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:17.260 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:17.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:17.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:17.276 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:17 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:26:17.276 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:17 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:26:17.276 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:17 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:26:17.276 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:17 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:26:17.276 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:17 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:26:17.276 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:17 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:26:17.362 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:17.384 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:17.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:17.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:17.486 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:26:17.507 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:17.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:17.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:26:17.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:17 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:26:17.576 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:17 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:26:17.576 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:17 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:26:17.609 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:17.630 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:17.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:17.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:17.732 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:17.754 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:17.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:17.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:17.855 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:17.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:17.877 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:17.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:17.941 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:26:17.941 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:26:17.942 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:26:17.942 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:26:17.942 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:26:17.942 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:26:17.948 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:26:17.948 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:26:17.964 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:26:17.964 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:26:17.978 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:17.981 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:26:17.983 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:26:17.984 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:26:18.001 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:26:18.001 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:26:18.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:18.002 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:18.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:18.016 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:26:18.017 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:26:18.103 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:18.125 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:18.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:18.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:18.227 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:26:18.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:18.250 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:18.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:26:18.351 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:26:18.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:18.373 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:18.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:26:18.474 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:18.497 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:18.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:18.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:18.599 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:26:18.621 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:18.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:18.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:26:18.682 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:18 smithi150 ceph-mon[29276]: pgmap v854: 132 pgs: 132 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 650 B/s rd, 0 op/s 2024-09-11T08:26:18.682 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:18 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2012559113' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:26:18.683 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:26:18 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:26:18] "GET /metrics HTTP/1.1" 200 224336 "" "Prometheus/2.43.0" 2024-09-11T08:26:18.683 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:18 smithi150 ceph-mon[39612]: pgmap v854: 132 pgs: 132 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 650 B/s rd, 0 op/s 2024-09-11T08:26:18.683 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:18 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2012559113' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:26:18.724 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:18.745 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:18.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:18.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:18.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:18 smithi181 ceph-mon[32701]: pgmap v854: 132 pgs: 132 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 650 B/s rd, 0 op/s 2024-09-11T08:26:18.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:18 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2012559113' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:26:18.847 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:18.872 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:18.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:18.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:18.973 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:26:18.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:18.997 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:18.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:26:19.099 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:19.125 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:19.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:19.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:19.226 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:19.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:19.254 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:19.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:19.323 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:26:18 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:26:19.355 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:19.380 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:19.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:19.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:19.482 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:19.506 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:19.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:19.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:19.608 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:19.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:19.634 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:19.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:19.735 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:19.759 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:19.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:19.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:19.861 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:19.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:19.886 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:19.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:19.987 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:20.012 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:20.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:20.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:20.114 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:20.137 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:20.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:20.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:20.240 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:20.263 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:20.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:20.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:20.365 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:20.388 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:20.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:20.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:20.490 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:20.516 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:20.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:20.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:20.618 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:26:20.640 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:20.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:20.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:26:20.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:20 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:26:20.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:20 smithi150 ceph-mon[29276]: pgmap v855: 132 pgs: 132 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-11T08:26:20.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:20 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:26:20.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:20 smithi150 ceph-mon[39612]: pgmap v855: 132 pgs: 132 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-11T08:26:20.742 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:20.767 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:20.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:20.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:20.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:20 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:26:20.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:20 smithi181 ceph-mon[32701]: pgmap v855: 132 pgs: 132 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-11T08:26:20.869 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:20.890 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:20.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:20.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:20.992 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:26:21.014 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:21.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:21.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:26:21.115 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:21.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:21.143 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:21.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:21.244 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:21.269 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:21.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:21.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:21.371 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:21.393 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:21.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:21.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:21.495 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:21.521 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:21.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:21.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:21.623 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:26:21.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:21.646 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:21.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:26:21.721 INFO:teuthology.orchestra.run.smithi150.stderr:reweighted osd.6 to 1 (10000) 2024-09-11T08:26:21.748 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:21.774 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:21.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:21.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:21.876 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:21.901 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:21.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:21.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:21.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:21 smithi150 ceph-mon[29276]: pgmap v856: 132 pgs: 132 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-11T08:26:21.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:21 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2733227158' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-11T08:26:21.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:21 smithi150 ceph-mon[39612]: pgmap v856: 132 pgs: 132 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-11T08:26:21.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:21 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2733227158' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-11T08:26:22.003 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:22.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:22.030 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:22.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:22.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:21 smithi181 ceph-mon[32701]: pgmap v856: 132 pgs: 132 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-11T08:26:22.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:21 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2733227158' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-11T08:26:22.132 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:22.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:22.157 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:22.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:22.258 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:22.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:22.285 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:22.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:22.386 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:26:22.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:22.410 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:22.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:26:22.511 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:22.537 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:22.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:22.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:22.639 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:22.665 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:22.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:22.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:22.702 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T08:26:22.767 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:22.789 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:22.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:22.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:22.891 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:22.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:22.919 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:22.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:22.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:22 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2733227158' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-11T08:26:22.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:22 smithi150 ceph-mon[29276]: osdmap e380: 8 total, 8 up, 8 in 2024-09-11T08:26:22.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:22 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:26:22.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:22 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:26:22.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:22 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2733227158' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-11T08:26:22.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:22 smithi150 ceph-mon[39612]: osdmap e380: 8 total, 8 up, 8 in 2024-09-11T08:26:22.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:22 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:26:22.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:22 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:26:23.019 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:26:23.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:23.043 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:23.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:26:23.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:22 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2733227158' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-11T08:26:23.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:22 smithi181 ceph-mon[32701]: osdmap e380: 8 total, 8 up, 8 in 2024-09-11T08:26:23.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:22 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:26:23.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:22 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:26:23.144 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:26:23.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:23.166 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:23.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:26:23.267 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:23.294 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:23.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:23.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:23.396 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:23.418 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:23.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:23.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:23.520 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:23.545 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:23.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:23.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:23.647 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:23.669 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:23.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:23.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:23.771 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:26:23.793 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:23.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:23.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:26:23.894 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:23.908 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-11T08:26:23.908 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 20, 2024-09-11T08:26:23.909 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 600, 2024-09-11T08:26:23.909 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [ 2024-09-11T08:26:23.909 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.909 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15369.0:96106 9.6 9:6057837d:::benchmark_data_smithi150_133616_object96105:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-11T08:26:23.909 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:19:08.607644+0000", 2024-09-11T08:26:23.909 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 435.29858740899999, 2024-09-11T08:26:23.909 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.177517404, 2024-09-11T08:26:23.909 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:26:23.909 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:26:23.909 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:26:23.909 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15369", 2024-09-11T08:26:23.909 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:26:23.910 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 96106 2024-09-11T08:26:23.910 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.910 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:26:23.910 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.910 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:26:23.910 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.607644+0000", 2024-09-11T08:26:23.910 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:26:23.910 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.910 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.910 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:26:23.910 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.607644+0000", 2024-09-11T08:26:23.910 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:26:23.910 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.911 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.911 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:26:23.911 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.607642+0000", 2024-09-11T08:26:23.911 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:26:23.911 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.911 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.911 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:26:23.911 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.607696+0000", 2024-09-11T08:26:23.911 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.4276e-05 2024-09-11T08:26:23.911 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.911 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.911 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:26:23.911 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.607707+0000", 2024-09-11T08:26:23.911 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1105e-05 2024-09-11T08:26:23.912 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.912 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.912 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:26:23.912 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.607713+0000", 2024-09-11T08:26:23.912 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.5949999999999998e-06 2024-09-11T08:26:23.912 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.912 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.912 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:26:23.912 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.607941+0000", 2024-09-11T08:26:23.912 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000228361 2024-09-11T08:26:23.912 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.912 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.913 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:26:23.913 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.607966+0000", 2024-09-11T08:26:23.913 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.4641e-05 2024-09-11T08:26:23.913 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.913 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.913 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:26:23.913 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.588318+0000", 2024-09-11T08:26:23.913 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.98035270299999999 2024-09-11T08:26:23.913 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.913 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.913 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:26:23.913 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.588330+0000", 2024-09-11T08:26:23.913 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.164e-05 2024-09-11T08:26:23.914 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.914 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.914 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:26:23.914 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.761111+0000", 2024-09-11T08:26:23.914 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1727814190000001 2024-09-11T08:26:23.914 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.914 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.914 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:26:23.914 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.765416+0000", 2024-09-11T08:26:23.914 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.004304296 2024-09-11T08:26:23.914 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.914 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.914 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 1,5,6", 2024-09-11T08:26:23.915 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.765590+0000", 2024-09-11T08:26:23.915 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00017429300000000001 2024-09-11T08:26:23.915 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.915 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.915 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:26:23.915 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.784410+0000", 2024-09-11T08:26:23.915 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.018820027 2024-09-11T08:26:23.915 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.915 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.915 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:26:23.915 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.784473+0000", 2024-09-11T08:26:23.915 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.3227999999999998e-05 2024-09-11T08:26:23.915 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.916 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.916 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:26:23.916 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.784534+0000", 2024-09-11T08:26:23.916 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.1198000000000006e-05 2024-09-11T08:26:23.916 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.916 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.916 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:26:23.916 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.784919+0000", 2024-09-11T08:26:23.916 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00038449300000000002 2024-09-11T08:26:23.916 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.916 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.916 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:26:23.916 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.785036+0000", 2024-09-11T08:26:23.916 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00011725 2024-09-11T08:26:23.917 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.917 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.917 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:26:23.917 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.785161+0000", 2024-09-11T08:26:23.917 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012483399999999999 2024-09-11T08:26:23.917 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:26:23.917 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:26:23.917 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:26:23.917 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.917 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.917 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15369.0:96112 9.6 9:61fdd20f:::benchmark_data_smithi150_133616_object96111:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-11T08:26:23.917 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:19:08.620746+0000", 2024-09-11T08:26:23.917 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 435.285484989, 2024-09-11T08:26:23.917 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.1646099950000002, 2024-09-11T08:26:23.918 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:26:23.918 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:26:23.918 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:26:23.918 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15369", 2024-09-11T08:26:23.918 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:26:23.918 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 96112 2024-09-11T08:26:23.918 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.918 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:26:23.918 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.918 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:26:23.918 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.620746+0000", 2024-09-11T08:26:23.918 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:26:23.918 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.919 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.919 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:26:23.919 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.620746+0000", 2024-09-11T08:26:23.919 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:26:23.919 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.919 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.919 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:26:23.919 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.620744+0000", 2024-09-11T08:26:23.919 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:26:23.919 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.919 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.920 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:26:23.920 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.620796+0000", 2024-09-11T08:26:23.920 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.1513e-05 2024-09-11T08:26:23.920 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.920 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.920 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:26:23.920 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.620812+0000", 2024-09-11T08:26:23.920 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.6291e-05 2024-09-11T08:26:23.920 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.920 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.920 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:26:23.920 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.620820+0000", 2024-09-11T08:26:23.920 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.242e-06 2024-09-11T08:26:23.921 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.921 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.921 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:26:23.921 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.620881+0000", 2024-09-11T08:26:23.921 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.0772e-05 2024-09-11T08:26:23.921 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.921 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.921 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:26:23.921 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.620902+0000", 2024-09-11T08:26:23.921 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.1483000000000001e-05 2024-09-11T08:26:23.921 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.921 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.921 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:26:23.922 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.590175+0000", 2024-09-11T08:26:23.922 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.969272519 2024-09-11T08:26:23.922 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.922 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.922 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:26:23.922 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.590185+0000", 2024-09-11T08:26:23.922 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.6190000000000006e-06 2024-09-11T08:26:23.922 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.922 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.922 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:26:23.922 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.769061+0000", 2024-09-11T08:26:23.922 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1788760979999999 2024-09-11T08:26:23.922 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.923 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.923 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:26:23.923 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.769199+0000", 2024-09-11T08:26:23.923 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00013856099999999999 2024-09-11T08:26:23.923 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.923 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.923 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 1,5,6", 2024-09-11T08:26:23.923 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.769362+0000", 2024-09-11T08:26:23.923 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00016318300000000001 2024-09-11T08:26:23.923 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.923 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.923 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:26:23.923 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.784443+0000", 2024-09-11T08:26:23.924 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.015080464999999999 2024-09-11T08:26:23.924 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.924 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.924 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:26:23.924 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.784503+0000", 2024-09-11T08:26:23.924 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.0556000000000001e-05 2024-09-11T08:26:23.924 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.924 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.924 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:26:23.924 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.784907+0000", 2024-09-11T08:26:23.924 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00040319299999999999 2024-09-11T08:26:23.924 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.924 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.925 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:26:23.925 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.785170+0000", 2024-09-11T08:26:23.925 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00026354599999999997 2024-09-11T08:26:23.925 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.925 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.925 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:26:23.925 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.785255+0000", 2024-09-11T08:26:23.925 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.4587999999999997e-05 2024-09-11T08:26:23.928 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.928 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.928 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:26:23.928 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.785356+0000", 2024-09-11T08:26:23.928 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00010129299999999999 2024-09-11T08:26:23.928 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:26:23.928 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:26:23.928 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:26:23.929 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.929 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.929 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.24320.0:1794 5.d 5:b4812045:::notify.4:head [watch ping cookie 140425499249504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e259)", 2024-09-11T08:26:23.929 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:19:08.688284+0000", 2024-09-11T08:26:23.929 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 435.21794696199998, 2024-09-11T08:26:23.929 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.92840292300000005, 2024-09-11T08:26:23.929 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:26:23.929 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "started", 2024-09-11T08:26:23.929 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:26:23.929 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.24320", 2024-09-11T08:26:23.929 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1503848706", 2024-09-11T08:26:23.929 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 1794 2024-09-11T08:26:23.929 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.930 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:26:23.930 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.930 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:26:23.930 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.688284+0000", 2024-09-11T08:26:23.930 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:26:23.930 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.930 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.930 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:26:23.930 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.688284+0000", 2024-09-11T08:26:23.930 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:26:23.930 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.930 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.930 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:26:23.931 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.688281+0000", 2024-09-11T08:26:23.931 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999967 2024-09-11T08:26:23.931 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.931 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.931 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:26:23.931 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.688311+0000", 2024-09-11T08:26:23.931 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.9952000000000001e-05 2024-09-11T08:26:23.931 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.931 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.931 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:26:23.931 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.688331+0000", 2024-09-11T08:26:23.931 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.0709000000000001e-05 2024-09-11T08:26:23.932 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.932 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.932 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:26:23.932 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.688338+0000", 2024-09-11T08:26:23.932 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.5200000000000003e-06 2024-09-11T08:26:23.932 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.932 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.932 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:26:23.932 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.688440+0000", 2024-09-11T08:26:23.932 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000102241 2024-09-11T08:26:23.932 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.933 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.933 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:26:23.933 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.688453+0000", 2024-09-11T08:26:23.933 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2767e-05 2024-09-11T08:26:23.933 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.933 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.933 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:26:23.933 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.611369+0000", 2024-09-11T08:26:23.933 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.92291567600000002 2024-09-11T08:26:23.933 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.933 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.933 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for degraded object", 2024-09-11T08:26:23.933 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.611669+0000", 2024-09-11T08:26:23.933 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00030050399999999999 2024-09-11T08:26:23.934 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.934 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.934 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:26:23.934 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.616504+0000", 2024-09-11T08:26:23.934 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0048346969999999998 2024-09-11T08:26:23.934 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.934 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.934 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:26:23.934 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.616590+0000", 2024-09-11T08:26:23.934 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.5826000000000001e-05 2024-09-11T08:26:23.934 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.934 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.934 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:26:23.935 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.616687+0000", 2024-09-11T08:26:23.935 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.7417000000000006e-05 2024-09-11T08:26:23.935 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:26:23.935 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:26:23.935 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:26:23.935 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.935 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.935 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.24320.0:1802 5.d 5:b4812045:::notify.4:head [watch reconnect cookie 140425499249504 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-11T08:26:23.935 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:19:08.688747+0000", 2024-09-11T08:26:23.935 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 435.21748384099999, 2024-09-11T08:26:23.935 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.92997097399999995, 2024-09-11T08:26:23.935 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:26:23.935 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "started", 2024-09-11T08:26:23.935 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:26:23.936 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.24320", 2024-09-11T08:26:23.936 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1503848706", 2024-09-11T08:26:23.936 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 1802 2024-09-11T08:26:23.936 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.936 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:26:23.936 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.936 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:26:23.936 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.688747+0000", 2024-09-11T08:26:23.936 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:26:23.936 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.936 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.936 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:26:23.936 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.688747+0000", 2024-09-11T08:26:23.937 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:26:23.937 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.937 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.937 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:26:23.937 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.688746+0000", 2024-09-11T08:26:23.937 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:26:23.937 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.937 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.937 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:26:23.937 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.688752+0000", 2024-09-11T08:26:23.937 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.8139999999999997e-06 2024-09-11T08:26:23.937 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.937 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.937 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:26:23.938 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.688766+0000", 2024-09-11T08:26:23.938 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4270000000000001e-05 2024-09-11T08:26:23.938 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.938 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.938 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:26:23.938 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.688772+0000", 2024-09-11T08:26:23.938 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.1120000000000001e-06 2024-09-11T08:26:23.938 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.938 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.938 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:26:23.938 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.688823+0000", 2024-09-11T08:26:23.938 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.0930999999999997e-05 2024-09-11T08:26:23.939 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.939 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.939 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:26:23.939 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.688829+0000", 2024-09-11T08:26:23.939 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.711e-06 2024-09-11T08:26:23.939 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.939 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.939 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:26:23.939 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.611693+0000", 2024-09-11T08:26:23.939 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.92286374900000001 2024-09-11T08:26:23.939 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.939 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.939 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for degraded object", 2024-09-11T08:26:23.939 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.611746+0000", 2024-09-11T08:26:23.940 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.2964999999999999e-05 2024-09-11T08:26:23.940 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.940 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.940 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:26:23.940 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.616724+0000", 2024-09-11T08:26:23.940 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0049778629999999999 2024-09-11T08:26:23.940 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.940 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.940 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:26:23.940 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.616936+0000", 2024-09-11T08:26:23.940 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000212563 2024-09-11T08:26:23.940 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:23.943 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:23.943 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.943 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:26:23.943 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.618718+0000", 2024-09-11T08:26:23.943 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0017820360000000001 2024-09-11T08:26:23.943 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:26:23.943 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:26:23.944 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:26:23.944 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.944 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.944 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:96722 10.9 10:974c6299:::benchmark_data_smithi150_158067_object96721:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e329)", 2024-09-11T08:26:23.944 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:22:58.186775+0000", 2024-09-11T08:26:23.944 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 205.71945649200001, 2024-09-11T08:26:23.944 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.99107653200000001, 2024-09-11T08:26:23.944 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:26:23.944 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:26:23.944 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:26:23.944 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:26:23.944 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:26:23.944 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 96722 2024-09-11T08:26:23.945 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.945 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:26:23.945 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.945 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:26:23.945 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.186775+0000", 2024-09-11T08:26:23.945 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:26:23.945 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.945 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.945 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:26:23.945 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.186775+0000", 2024-09-11T08:26:23.945 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:26:23.945 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.946 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.946 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:26:23.946 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.186773+0000", 2024-09-11T08:26:23.946 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:26:23.946 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.946 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.946 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:26:23.946 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.186821+0000", 2024-09-11T08:26:23.946 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.8099999999999997e-05 2024-09-11T08:26:23.946 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.946 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.946 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:26:23.946 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.186834+0000", 2024-09-11T08:26:23.947 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3344e-05 2024-09-11T08:26:23.947 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.947 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.947 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:26:23.947 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.186840+0000", 2024-09-11T08:26:23.947 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.7289999999999997e-06 2024-09-11T08:26:23.947 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.947 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.947 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:26:23.947 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.186893+0000", 2024-09-11T08:26:23.947 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.2549000000000001e-05 2024-09-11T08:26:23.947 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.947 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.947 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:26:23.948 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.186904+0000", 2024-09-11T08:26:23.948 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1029000000000001e-05 2024-09-11T08:26:23.948 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.948 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.948 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:26:23.948 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.167018+0000", 2024-09-11T08:26:23.948 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.98011428899999997 2024-09-11T08:26:23.948 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.948 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.948 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:26:23.948 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.168686+0000", 2024-09-11T08:26:23.948 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.001667717 2024-09-11T08:26:23.949 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.949 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.949 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,7", 2024-09-11T08:26:23.949 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.168855+0000", 2024-09-11T08:26:23.949 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00016924299999999999 2024-09-11T08:26:23.949 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.949 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.949 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:26:23.949 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.175952+0000", 2024-09-11T08:26:23.949 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0070972479999999996 2024-09-11T08:26:23.949 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.949 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.950 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:26:23.950 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.177762+0000", 2024-09-11T08:26:23.950 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.001810416 2024-09-11T08:26:23.950 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.950 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.950 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:26:23.950 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.177797+0000", 2024-09-11T08:26:23.950 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.4048999999999999e-05 2024-09-11T08:26:23.950 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.950 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.950 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:26:23.950 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.177827+0000", 2024-09-11T08:26:23.950 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.0725999999999997e-05 2024-09-11T08:26:23.951 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.951 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.951 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:26:23.951 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.177851+0000", 2024-09-11T08:26:23.951 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.3777e-05 2024-09-11T08:26:23.951 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:26:23.951 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:26:23.952 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:26:23.952 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.952 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.952 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:96770 10.9 10:90fb0a7a:::benchmark_data_smithi150_158067_object96769:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e329)", 2024-09-11T08:26:23.952 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:22:58.240293+0000", 2024-09-11T08:26:23.952 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 205.66593775699999, 2024-09-11T08:26:23.952 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.94590357400000002, 2024-09-11T08:26:23.952 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:26:23.952 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:26:23.952 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:26:23.952 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:26:23.952 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:26:23.953 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 96770 2024-09-11T08:26:23.953 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.953 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:26:23.953 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.953 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:26:23.953 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.240293+0000", 2024-09-11T08:26:23.953 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:26:23.953 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.953 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.953 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:26:23.953 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.240293+0000", 2024-09-11T08:26:23.953 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:26:23.953 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.953 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.954 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:26:23.954 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.240291+0000", 2024-09-11T08:26:23.954 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:26:23.954 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.954 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.954 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:26:23.954 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.240315+0000", 2024-09-11T08:26:23.954 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.4329e-05 2024-09-11T08:26:23.954 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.954 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.954 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:26:23.954 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.240326+0000", 2024-09-11T08:26:23.954 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0969e-05 2024-09-11T08:26:23.955 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.955 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.955 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:26:23.955 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.240333+0000", 2024-09-11T08:26:23.955 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.1670000000000004e-06 2024-09-11T08:26:23.955 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.955 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.955 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:26:23.955 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.240389+0000", 2024-09-11T08:26:23.955 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.6026000000000002e-05 2024-09-11T08:26:23.955 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.955 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.955 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:26:23.956 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.240399+0000", 2024-09-11T08:26:23.956 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.9210000000000004e-06 2024-09-11T08:26:23.956 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.956 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.956 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:26:23.956 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.171307+0000", 2024-09-11T08:26:23.956 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.93090849499999995 2024-09-11T08:26:23.956 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.956 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.956 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:26:23.956 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.171435+0000", 2024-09-11T08:26:23.956 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012800100000000001 2024-09-11T08:26:23.956 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.957 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.957 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,7", 2024-09-11T08:26:23.957 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.171575+0000", 2024-09-11T08:26:23.957 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00014013599999999999 2024-09-11T08:26:23.957 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.957 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.957 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:26:23.957 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.177885+0000", 2024-09-11T08:26:23.957 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0063097320000000002 2024-09-11T08:26:23.957 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.957 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.957 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:26:23.957 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.177934+0000", 2024-09-11T08:26:23.958 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.9217e-05 2024-09-11T08:26:23.958 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.958 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.958 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:26:23.958 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.186143+0000", 2024-09-11T08:26:23.958 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0082086190000000003 2024-09-11T08:26:23.958 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.958 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.958 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:26:23.958 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.186173+0000", 2024-09-11T08:26:23.958 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.0252000000000001e-05 2024-09-11T08:26:23.958 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.958 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.958 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:26:23.959 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.186197+0000", 2024-09-11T08:26:23.959 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.3816999999999999e-05 2024-09-11T08:26:23.959 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:26:23.959 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:26:23.959 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:26:23.959 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.959 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.959 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:96776 10.9 10:94bc6bc2:::benchmark_data_smithi150_158067_object96775:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e329)", 2024-09-11T08:26:23.959 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:22:58.245049+0000", 2024-09-11T08:26:23.959 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 205.66118201699999, 2024-09-11T08:26:23.959 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.94422798699999999, 2024-09-11T08:26:23.959 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:26:23.959 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:26:23.960 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:26:23.960 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:26:23.960 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:26:23.960 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 96776 2024-09-11T08:26:23.960 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.960 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:26:23.960 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.960 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:26:23.960 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.245049+0000", 2024-09-11T08:26:23.960 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:26:23.960 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.960 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.960 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:26:23.960 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.245049+0000", 2024-09-11T08:26:23.961 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:26:23.961 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.961 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.961 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:26:23.961 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.245047+0000", 2024-09-11T08:26:23.961 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:26:23.961 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.961 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.961 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:26:23.961 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.245073+0000", 2024-09-11T08:26:23.961 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.5703000000000001e-05 2024-09-11T08:26:23.961 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.962 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.962 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:26:23.962 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.245097+0000", 2024-09-11T08:26:23.962 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.4056000000000001e-05 2024-09-11T08:26:23.962 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.962 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.962 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:26:23.962 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.245104+0000", 2024-09-11T08:26:23.962 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.8519999999999998e-06 2024-09-11T08:26:23.962 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.962 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.962 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:26:23.962 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.245219+0000", 2024-09-11T08:26:23.962 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000115035 2024-09-11T08:26:23.962 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.963 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.963 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:26:23.963 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.245228+0000", 2024-09-11T08:26:23.963 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.4630000000000007e-06 2024-09-11T08:26:23.963 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.963 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.963 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:26:23.963 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.173417+0000", 2024-09-11T08:26:23.963 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.92818864300000004 2024-09-11T08:26:23.963 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.963 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.963 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:26:23.963 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.173529+0000", 2024-09-11T08:26:23.963 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000111631 2024-09-11T08:26:23.964 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.964 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.964 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,7", 2024-09-11T08:26:23.964 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.173666+0000", 2024-09-11T08:26:23.964 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00013709 2024-09-11T08:26:23.964 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.964 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.964 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:26:23.964 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.186114+0000", 2024-09-11T08:26:23.964 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.012448232 2024-09-11T08:26:23.964 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.964 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.964 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:26:23.965 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.186812+0000", 2024-09-11T08:26:23.965 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00069784800000000004 2024-09-11T08:26:23.965 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.965 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.965 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:26:23.965 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.189185+0000", 2024-09-11T08:26:23.965 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.002373502 2024-09-11T08:26:23.965 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.965 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.965 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:26:23.965 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.189239+0000", 2024-09-11T08:26:23.965 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.3560999999999998e-05 2024-09-11T08:26:23.965 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.966 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.966 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:26:23.966 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.189277+0000", 2024-09-11T08:26:23.966 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.8133000000000002e-05 2024-09-11T08:26:23.966 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:26:23.966 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:26:23.966 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:26:23.966 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.966 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.966 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:96783 10.9 10:958af421:::benchmark_data_smithi150_158067_object96782:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e329)", 2024-09-11T08:26:23.966 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:22:58.250957+0000", 2024-09-11T08:26:23.966 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 205.655274093, 2024-09-11T08:26:23.966 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.93903492600000005, 2024-09-11T08:26:23.967 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:26:23.967 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:26:23.967 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:26:23.967 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:26:23.967 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:26:23.967 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 96783 2024-09-11T08:26:23.967 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.967 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:26:23.967 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.967 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:26:23.967 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.250957+0000", 2024-09-11T08:26:23.967 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:26:23.968 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.968 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.968 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:26:23.968 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.250957+0000", 2024-09-11T08:26:23.968 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:26:23.968 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.968 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.968 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:26:23.968 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.250955+0000", 2024-09-11T08:26:23.968 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:26:23.968 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.968 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.969 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:26:23.969 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.250982+0000", 2024-09-11T08:26:23.969 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.6687999999999999e-05 2024-09-11T08:26:23.969 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.969 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.969 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:26:23.969 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.250996+0000", 2024-09-11T08:26:23.969 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4239e-05 2024-09-11T08:26:23.969 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.969 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.969 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:26:23.969 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.251002+0000", 2024-09-11T08:26:23.969 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.7629999999999997e-06 2024-09-11T08:26:23.969 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.970 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.970 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:26:23.970 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.251052+0000", 2024-09-11T08:26:23.970 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.0389999999999997e-05 2024-09-11T08:26:23.970 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.970 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.970 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:26:23.970 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.251062+0000", 2024-09-11T08:26:23.970 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.7780000000000002e-06 2024-09-11T08:26:23.970 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.970 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.970 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:26:23.971 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.175994+0000", 2024-09-11T08:26:23.971 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.92493149600000002 2024-09-11T08:26:23.971 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.971 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.971 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:26:23.971 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.176096+0000", 2024-09-11T08:26:23.971 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00010289300000000001 2024-09-11T08:26:23.971 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.971 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.971 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,7", 2024-09-11T08:26:23.971 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.176181+0000", 2024-09-11T08:26:23.971 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.4065000000000002e-05 2024-09-11T08:26:23.971 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.971 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.972 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:26:23.972 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.186119+0000", 2024-09-11T08:26:23.972 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0099386590000000007 2024-09-11T08:26:23.972 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.972 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.972 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:26:23.972 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.189331+0000", 2024-09-11T08:26:23.972 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0032119990000000001 2024-09-11T08:26:23.972 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.972 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.972 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:26:23.972 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.189914+0000", 2024-09-11T08:26:23.972 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00058295999999999999 2024-09-11T08:26:23.973 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.973 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.973 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:26:23.973 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.189957+0000", 2024-09-11T08:26:23.973 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.3161999999999998e-05 2024-09-11T08:26:23.973 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.973 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.973 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:26:23.973 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.189992+0000", 2024-09-11T08:26:23.973 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.4501000000000001e-05 2024-09-11T08:26:23.973 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:26:23.973 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:26:23.973 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:26:23.974 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.974 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.974 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:96784 10.9 10:9dd50853:::benchmark_data_smithi150_158067_object96783:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e329)", 2024-09-11T08:26:23.974 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:22:58.251848+0000", 2024-09-11T08:26:23.974 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 205.654383443, 2024-09-11T08:26:23.974 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.93834602199999995, 2024-09-11T08:26:23.974 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:26:23.974 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:26:23.974 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:26:23.974 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:26:23.974 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:26:23.974 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 96784 2024-09-11T08:26:23.974 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.974 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:26:23.975 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.975 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:26:23.975 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.251848+0000", 2024-09-11T08:26:23.975 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:26:23.975 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.975 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.975 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:26:23.975 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.251848+0000", 2024-09-11T08:26:23.975 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:26:23.975 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.975 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.975 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:26:23.975 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.251847+0000", 2024-09-11T08:26:23.976 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:26:23.976 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.976 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.976 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:26:23.976 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.251871+0000", 2024-09-11T08:26:23.976 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.4564999999999999e-05 2024-09-11T08:26:23.976 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.976 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.976 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:26:23.976 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.251884+0000", 2024-09-11T08:26:23.976 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2635e-05 2024-09-11T08:26:23.976 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.976 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.976 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:26:23.977 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.251889+0000", 2024-09-11T08:26:23.977 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.6899999999999997e-06 2024-09-11T08:26:23.977 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.977 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.977 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:26:23.977 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.252285+0000", 2024-09-11T08:26:23.977 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00039541899999999999 2024-09-11T08:26:23.977 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.977 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.978 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:26:23.978 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.252294+0000", 2024-09-11T08:26:23.978 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.9390000000000003e-06 2024-09-11T08:26:23.978 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.978 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.978 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:26:23.978 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.177963+0000", 2024-09-11T08:26:23.978 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.92566948999999998 2024-09-11T08:26:23.978 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.978 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.978 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:26:23.978 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.178056+0000", 2024-09-11T08:26:23.978 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.2879000000000003e-05 2024-09-11T08:26:23.978 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.979 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.979 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,7", 2024-09-11T08:26:23.979 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.178173+0000", 2024-09-11T08:26:23.979 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000116852 2024-09-11T08:26:23.979 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.979 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.979 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:26:23.979 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.186763+0000", 2024-09-11T08:26:23.979 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0085899740000000002 2024-09-11T08:26:23.979 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.979 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.979 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:26:23.980 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.189382+0000", 2024-09-11T08:26:23.980 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.002619241 2024-09-11T08:26:23.980 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.980 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.980 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:26:23.980 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.190101+0000", 2024-09-11T08:26:23.980 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00071871100000000002 2024-09-11T08:26:23.980 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.980 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.980 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:26:23.980 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.190152+0000", 2024-09-11T08:26:23.981 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.1384999999999997e-05 2024-09-11T08:26:23.981 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.981 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.981 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:26:23.981 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.190194+0000", 2024-09-11T08:26:23.981 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1291999999999997e-05 2024-09-11T08:26:23.981 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:26:23.981 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:26:23.981 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:26:23.981 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.981 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.981 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:96788 10.9 10:95b153f1:::benchmark_data_smithi150_158067_object96787:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e329)", 2024-09-11T08:26:23.982 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:22:58.253032+0000", 2024-09-11T08:26:23.982 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 205.65319907, 2024-09-11T08:26:23.982 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.93740148099999998, 2024-09-11T08:26:23.982 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:26:23.982 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:26:23.982 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:26:23.982 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:26:23.982 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:26:23.982 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 96788 2024-09-11T08:26:23.982 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.982 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:26:23.982 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.982 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:26:23.982 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.253032+0000", 2024-09-11T08:26:23.983 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:26:23.983 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.983 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.983 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:26:23.983 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.253032+0000", 2024-09-11T08:26:23.983 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:26:23.983 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.983 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.983 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:26:23.983 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.253031+0000", 2024-09-11T08:26:23.983 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:26:23.983 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.983 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.983 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:26:23.984 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.253051+0000", 2024-09-11T08:26:23.984 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.9853000000000001e-05 2024-09-11T08:26:23.984 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.984 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.984 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:26:23.984 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.253059+0000", 2024-09-11T08:26:23.984 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.0399999999999993e-06 2024-09-11T08:26:23.984 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.984 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.985 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:26:23.985 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.253064+0000", 2024-09-11T08:26:23.985 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.1309999999999997e-06 2024-09-11T08:26:23.985 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.985 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.985 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:26:23.985 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.253121+0000", 2024-09-11T08:26:23.985 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.7419e-05 2024-09-11T08:26:23.985 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.986 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.986 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:26:23.986 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.253130+0000", 2024-09-11T08:26:23.986 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.2320000000000001e-06 2024-09-11T08:26:23.986 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.986 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.986 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:26:23.986 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.180611+0000", 2024-09-11T08:26:23.986 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.92748144600000004 2024-09-11T08:26:23.986 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.986 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.986 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:26:23.986 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.180735+0000", 2024-09-11T08:26:23.987 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000124148 2024-09-11T08:26:23.987 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.987 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.987 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,7", 2024-09-11T08:26:23.987 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.180878+0000", 2024-09-11T08:26:23.987 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00014298499999999999 2024-09-11T08:26:23.987 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.987 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.987 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:26:23.987 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.186769+0000", 2024-09-11T08:26:23.987 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0058905939999999999 2024-09-11T08:26:23.987 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.987 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.988 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:26:23.988 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.189514+0000", 2024-09-11T08:26:23.988 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0027449240000000001 2024-09-11T08:26:23.988 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.988 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.988 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:26:23.988 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.190296+0000", 2024-09-11T08:26:23.988 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00078191700000000003 2024-09-11T08:26:23.988 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.988 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.988 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:26:23.988 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.190388+0000", 2024-09-11T08:26:23.989 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.2152999999999997e-05 2024-09-11T08:26:23.989 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.989 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.989 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:26:23.989 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.190433+0000", 2024-09-11T08:26:23.989 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.5637999999999999e-05 2024-09-11T08:26:23.989 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:26:23.989 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:26:23.989 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:26:23.989 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.989 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.989 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:96804 10.9 10:9cf2d17b:::benchmark_data_smithi150_158067_object96803:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e329)", 2024-09-11T08:26:23.989 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:22:58.268425+0000", 2024-09-11T08:26:23.990 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 205.63780624, 2024-09-11T08:26:23.990 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.93122529300000001, 2024-09-11T08:26:23.990 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:26:23.990 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:26:23.990 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:26:23.990 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:26:23.990 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:26:23.990 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 96804 2024-09-11T08:26:23.990 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.990 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:26:23.990 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.990 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:26:23.990 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.268425+0000", 2024-09-11T08:26:23.991 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:26:23.991 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.991 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.991 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:26:23.991 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.268425+0000", 2024-09-11T08:26:23.991 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:26:23.991 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.991 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.991 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:26:23.991 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.268423+0000", 2024-09-11T08:26:23.991 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:26:23.991 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.992 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.992 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:26:23.992 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.268447+0000", 2024-09-11T08:26:23.992 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.4218000000000001e-05 2024-09-11T08:26:23.992 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.992 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.992 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:26:23.992 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.268464+0000", 2024-09-11T08:26:23.992 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.6532999999999999e-05 2024-09-11T08:26:23.992 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.992 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.992 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:26:23.992 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.268470+0000", 2024-09-11T08:26:23.993 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.5390000000000004e-06 2024-09-11T08:26:23.993 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.993 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.993 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:26:23.993 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.268518+0000", 2024-09-11T08:26:23.993 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.7905000000000003e-05 2024-09-11T08:26:23.993 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.993 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.993 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:26:23.993 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.268530+0000", 2024-09-11T08:26:23.993 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1474000000000001e-05 2024-09-11T08:26:23.993 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.993 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.994 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:26:23.994 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.183442+0000", 2024-09-11T08:26:23.994 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.91491235500000001 2024-09-11T08:26:23.994 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.994 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.994 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:26:23.994 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.183576+0000", 2024-09-11T08:26:23.994 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00013377800000000001 2024-09-11T08:26:23.994 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.994 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.994 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,7", 2024-09-11T08:26:23.994 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.183732+0000", 2024-09-11T08:26:23.994 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00015577599999999999 2024-09-11T08:26:23.995 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.995 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.995 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:26:23.995 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.188244+0000", 2024-09-11T08:26:23.995 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0045119039999999997 2024-09-11T08:26:23.995 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.995 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.995 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:26:23.995 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.194947+0000", 2024-09-11T08:26:23.995 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.006702894 2024-09-11T08:26:23.995 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.995 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.995 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:26:23.996 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.199521+0000", 2024-09-11T08:26:23.996 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0045744100000000001 2024-09-11T08:26:23.996 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.996 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.996 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:26:23.996 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.199611+0000", 2024-09-11T08:26:23.996 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.9751000000000002e-05 2024-09-11T08:26:23.996 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.996 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.996 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:26:23.996 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.199650+0000", 2024-09-11T08:26:23.996 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.9385000000000003e-05 2024-09-11T08:26:23.996 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:26:23.997 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:26:23.997 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:26:23.997 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.997 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.997 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:257624 10.9 10:92df583c:::benchmark_data_smithi150_158067_object127380:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-09-11T08:26:23.997 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:25:41.808160+0000", 2024-09-11T08:26:23.997 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 42.098071484000002, 2024-09-11T08:26:23.997 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.94614705899999996, 2024-09-11T08:26:23.997 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:26:23.997 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:26:23.997 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:26:23.997 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:26:23.997 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:26:23.997 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 257624 2024-09-11T08:26:23.997 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.998 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:26:23.998 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.998 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:26:23.998 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.808160+0000", 2024-09-11T08:26:23.998 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:26:23.998 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.998 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.998 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:26:23.998 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.808160+0000", 2024-09-11T08:26:23.998 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:26:23.998 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.998 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.998 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:26:23.998 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.808159+0000", 2024-09-11T08:26:23.999 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:26:23.999 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.999 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.999 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:26:23.999 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.808162+0000", 2024-09-11T08:26:23.999 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.886e-06 2024-09-11T08:26:23.999 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.999 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.999 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:26:23.999 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.808283+0000", 2024-09-11T08:26:23.999 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000121332 2024-09-11T08:26:23.999 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:23.999 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:23.999 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:26:23.999 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.808291+0000", 2024-09-11T08:26:24.000 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.5340000000000002e-06 2024-09-11T08:26:24.000 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.000 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.000 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:26:24.000 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.808344+0000", 2024-09-11T08:26:24.000 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.3319999999999998e-05 2024-09-11T08:26:24.000 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.000 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.000 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:26:24.000 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.808353+0000", 2024-09-11T08:26:24.000 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.585e-06 2024-09-11T08:26:24.000 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.000 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.001 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:26:24.001 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.746194+0000", 2024-09-11T08:26:24.001 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.93784158100000004 2024-09-11T08:26:24.001 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.001 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.001 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:26:24.001 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.746308+0000", 2024-09-11T08:26:24.001 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00011389 2024-09-11T08:26:24.001 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.001 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.001 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,7", 2024-09-11T08:26:24.001 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.746439+0000", 2024-09-11T08:26:24.001 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00013035999999999999 2024-09-11T08:26:24.002 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.002 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.002 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:26:24.002 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.749375+0000", 2024-09-11T08:26:24.002 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0029365369999999999 2024-09-11T08:26:24.002 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.002 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.002 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:26:24.002 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.754199+0000", 2024-09-11T08:26:24.002 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0048243000000000001 2024-09-11T08:26:24.002 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.002 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.003 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:26:24.003 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.754240+0000", 2024-09-11T08:26:24.003 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.0219000000000002e-05 2024-09-11T08:26:24.003 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.003 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.003 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:26:24.003 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.754269+0000", 2024-09-11T08:26:24.003 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.9298e-05 2024-09-11T08:26:24.003 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.003 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.003 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:26:24.003 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.754307+0000", 2024-09-11T08:26:24.003 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.7611999999999998e-05 2024-09-11T08:26:24.004 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:26:24.004 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:26:24.004 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:26:24.004 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.004 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.004 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:257630 10.9 10:921d42f5:::benchmark_data_smithi150_158067_object127386:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-09-11T08:26:24.004 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:25:41.816616+0000", 2024-09-11T08:26:24.004 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 42.089614589, 2024-09-11T08:26:24.004 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.94358571800000002, 2024-09-11T08:26:24.004 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:26:24.004 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:26:24.004 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:26:24.005 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:26:24.005 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:26:24.005 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 257630 2024-09-11T08:26:24.005 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.005 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:26:24.005 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.005 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:26:24.005 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.816616+0000", 2024-09-11T08:26:24.005 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:26:24.005 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.005 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.005 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:26:24.005 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.816616+0000", 2024-09-11T08:26:24.006 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:26:24.006 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.006 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.006 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:26:24.006 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.816615+0000", 2024-09-11T08:26:24.006 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:26:24.006 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.006 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.006 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:26:24.006 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.816631+0000", 2024-09-11T08:26:24.006 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.6232999999999999e-05 2024-09-11T08:26:24.006 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.006 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.006 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:26:24.007 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.816643+0000", 2024-09-11T08:26:24.007 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1464999999999999e-05 2024-09-11T08:26:24.007 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.007 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.007 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:26:24.007 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.816649+0000", 2024-09-11T08:26:24.007 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.0909999999999998e-06 2024-09-11T08:26:24.007 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.007 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.007 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:26:24.007 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.816722+0000", 2024-09-11T08:26:24.007 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.3164999999999995e-05 2024-09-11T08:26:24.007 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.007 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.008 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:26:24.008 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.816731+0000", 2024-09-11T08:26:24.008 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.7690000000000002e-06 2024-09-11T08:26:24.008 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.008 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.008 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:26:24.008 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.749413+0000", 2024-09-11T08:26:24.008 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.93268246099999996 2024-09-11T08:26:24.008 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.008 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.008 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:26:24.008 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.749749+0000", 2024-09-11T08:26:24.008 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00033593300000000001 2024-09-11T08:26:24.009 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.009 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.009 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,7", 2024-09-11T08:26:24.009 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.749871+0000", 2024-09-11T08:26:24.009 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012230100000000001 2024-09-11T08:26:24.009 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.009 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.009 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:26:24.009 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.754347+0000", 2024-09-11T08:26:24.009 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0044754699999999996 2024-09-11T08:26:24.009 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.009 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.009 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:26:24.009 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.754372+0000", 2024-09-11T08:26:24.010 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.5201000000000001e-05 2024-09-11T08:26:24.010 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.010 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.010 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:26:24.010 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.760122+0000", 2024-09-11T08:26:24.010 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0057503980000000003 2024-09-11T08:26:24.010 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.010 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.010 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:26:24.010 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.760164+0000", 2024-09-11T08:26:24.010 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1284e-05 2024-09-11T08:26:24.010 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.010 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.011 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:26:24.011 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.760202+0000", 2024-09-11T08:26:24.011 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.8415999999999999e-05 2024-09-11T08:26:24.011 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:26:24.011 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:26:24.011 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:26:24.011 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.011 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.011 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:257632 10.9 10:91c6bcfb:::benchmark_data_smithi150_158067_object127388:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-09-11T08:26:24.011 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:25:41.816663+0000", 2024-09-11T08:26:24.011 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 42.089568198999999, 2024-09-11T08:26:24.011 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.95093924299999999, 2024-09-11T08:26:24.011 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:26:24.011 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:26:24.012 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:26:24.012 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:26:24.012 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:26:24.012 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 257632 2024-09-11T08:26:24.012 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.012 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:26:24.012 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.012 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:26:24.012 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.816663+0000", 2024-09-11T08:26:24.012 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:26:24.012 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.012 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.012 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:26:24.012 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.816663+0000", 2024-09-11T08:26:24.013 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:26:24.013 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.013 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.013 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:26:24.013 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.816662+0000", 2024-09-11T08:26:24.013 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:26:24.013 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.013 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.013 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:26:24.013 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.816666+0000", 2024-09-11T08:26:24.013 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.2899999999999998e-06 2024-09-11T08:26:24.013 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.013 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.013 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:26:24.014 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.816671+0000", 2024-09-11T08:26:24.014 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.5300000000000004e-06 2024-09-11T08:26:24.014 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.014 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.014 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:26:24.014 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.816676+0000", 2024-09-11T08:26:24.014 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.0860000000000001e-06 2024-09-11T08:26:24.014 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.014 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.014 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:26:24.014 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.816749+0000", 2024-09-11T08:26:24.014 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.2861000000000006e-05 2024-09-11T08:26:24.014 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.014 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.015 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:26:24.015 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.816755+0000", 2024-09-11T08:26:24.015 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.5319999999999997e-06 2024-09-11T08:26:24.015 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.015 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.015 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:26:24.015 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.752363+0000", 2024-09-11T08:26:24.015 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.93560816999999996 2024-09-11T08:26:24.015 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.015 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.015 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:26:24.015 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.752431+0000", 2024-09-11T08:26:24.015 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.8224000000000003e-05 2024-09-11T08:26:24.016 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.016 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.016 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,7", 2024-09-11T08:26:24.016 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.752493+0000", 2024-09-11T08:26:24.016 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.2367999999999996e-05 2024-09-11T08:26:24.016 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.016 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.016 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:26:24.016 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.760021+0000", 2024-09-11T08:26:24.016 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.007527527 2024-09-11T08:26:24.016 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.016 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.017 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:26:24.017 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.760074+0000", 2024-09-11T08:26:24.017 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.3037000000000001e-05 2024-09-11T08:26:24.017 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.017 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.017 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:26:24.017 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.767528+0000", 2024-09-11T08:26:24.017 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0074539940000000002 2024-09-11T08:26:24.017 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.017 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.017 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:26:24.017 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.767559+0000", 2024-09-11T08:26:24.017 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.1288000000000003e-05 2024-09-11T08:26:24.018 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.018 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.018 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:26:24.018 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.767602+0000", 2024-09-11T08:26:24.018 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.2808000000000001e-05 2024-09-11T08:26:24.018 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:26:24.018 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:26:24.018 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:26:24.018 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.018 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.018 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:257647 10.9 10:91ee9aaf:::benchmark_data_smithi150_158067_object127403:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-09-11T08:26:24.018 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:25:41.831664+0000", 2024-09-11T08:26:24.018 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 42.074566652999998, 2024-09-11T08:26:24.018 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.93606025599999998, 2024-09-11T08:26:24.019 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:26:24.019 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:26:24.019 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:26:24.019 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:26:24.019 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:26:24.019 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 257647 2024-09-11T08:26:24.019 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.019 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:26:24.019 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.019 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:26:24.019 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.831664+0000", 2024-09-11T08:26:24.019 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:26:24.019 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.020 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.020 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:26:24.020 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.831664+0000", 2024-09-11T08:26:24.020 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:26:24.020 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.020 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.020 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:26:24.020 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.831663+0000", 2024-09-11T08:26:24.020 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:26:24.020 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.020 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.020 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:26:24.020 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.831669+0000", 2024-09-11T08:26:24.021 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.2709999999999998e-06 2024-09-11T08:26:24.021 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.021 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.021 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:26:24.021 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.831679+0000", 2024-09-11T08:26:24.021 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.9990000000000003e-06 2024-09-11T08:26:24.021 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.021 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.021 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:26:24.021 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.831683+0000", 2024-09-11T08:26:24.021 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.3030000000000001e-06 2024-09-11T08:26:24.021 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.021 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.021 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:26:24.022 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.831724+0000", 2024-09-11T08:26:24.022 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.0228000000000002e-05 2024-09-11T08:26:24.022 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.022 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.022 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:26:24.022 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.831731+0000", 2024-09-11T08:26:24.022 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.939e-06 2024-09-11T08:26:24.022 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.022 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.022 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:26:24.022 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.754402+0000", 2024-09-11T08:26:24.022 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.92267161099999995 2024-09-11T08:26:24.022 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.023 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.023 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:26:24.023 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.754503+0000", 2024-09-11T08:26:24.023 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000100719 2024-09-11T08:26:24.023 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.023 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.023 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,7", 2024-09-11T08:26:24.023 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.754618+0000", 2024-09-11T08:26:24.023 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00011491499999999999 2024-09-11T08:26:24.023 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.023 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.023 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:26:24.023 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.760222+0000", 2024-09-11T08:26:24.024 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0056036810000000001 2024-09-11T08:26:24.024 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.024 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.024 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:26:24.024 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.760242+0000", 2024-09-11T08:26:24.024 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.0888000000000001e-05 2024-09-11T08:26:24.024 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.024 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.024 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:26:24.024 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.767646+0000", 2024-09-11T08:26:24.024 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0074040030000000001 2024-09-11T08:26:24.024 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.024 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.025 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:26:24.025 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.767691+0000", 2024-09-11T08:26:24.025 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.4127e-05 2024-09-11T08:26:24.025 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.025 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.025 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:26:24.025 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.767725+0000", 2024-09-11T08:26:24.025 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.4060999999999998e-05 2024-09-11T08:26:24.025 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:26:24.025 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:26:24.025 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:26:24.025 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.025 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.026 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:257660 10.9 10:90fd90c6:::benchmark_data_smithi150_158067_object127416:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-09-11T08:26:24.026 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:25:41.846903+0000", 2024-09-11T08:26:24.026 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 42.059327983000003, 2024-09-11T08:26:24.026 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.92092837400000005, 2024-09-11T08:26:24.026 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:26:24.026 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:26:24.026 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:26:24.026 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:26:24.026 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:26:24.026 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 257660 2024-09-11T08:26:24.026 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.026 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:26:24.026 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.026 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:26:24.027 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.846903+0000", 2024-09-11T08:26:24.027 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:26:24.027 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.027 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.027 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:26:24.027 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.846903+0000", 2024-09-11T08:26:24.027 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:26:24.027 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.027 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.027 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:26:24.027 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.846902+0000", 2024-09-11T08:26:24.027 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:26:24.027 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.028 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.028 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:26:24.028 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.846909+0000", 2024-09-11T08:26:24.028 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.7600000000000002e-06 2024-09-11T08:26:24.028 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.028 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.028 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:26:24.028 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.846922+0000", 2024-09-11T08:26:24.028 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3118999999999999e-05 2024-09-11T08:26:24.028 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.028 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.028 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:26:24.029 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.846929+0000", 2024-09-11T08:26:24.029 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.5289999999999997e-06 2024-09-11T08:26:24.029 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.029 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.029 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:26:24.029 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.846980+0000", 2024-09-11T08:26:24.029 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.0834000000000002e-05 2024-09-11T08:26:24.029 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.029 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.029 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:26:24.029 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.846989+0000", 2024-09-11T08:26:24.029 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.9819999999999997e-06 2024-09-11T08:26:24.029 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.030 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.030 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:26:24.030 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.757407+0000", 2024-09-11T08:26:24.030 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.91041787399999996 2024-09-11T08:26:24.030 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.030 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.030 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:26:24.030 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.757487+0000", 2024-09-11T08:26:24.030 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.0476999999999995e-05 2024-09-11T08:26:24.030 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.030 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:24.031 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,7", 2024-09-11T08:26:24.031 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.757564+0000", 2024-09-11T08:26:24.032 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.6508999999999995e-05 2024-09-11T08:26:24.032 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.032 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.032 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:26:24.032 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.760247+0000", 2024-09-11T08:26:24.032 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.002683586 2024-09-11T08:26:24.032 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.032 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.032 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:26:24.032 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.767438+0000", 2024-09-11T08:26:24.032 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0071904480000000003 2024-09-11T08:26:24.032 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.032 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.033 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:26:24.033 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.767761+0000", 2024-09-11T08:26:24.033 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0003233 2024-09-11T08:26:24.033 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.033 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.033 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:26:24.033 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.767802+0000", 2024-09-11T08:26:24.033 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1099000000000001e-05 2024-09-11T08:26:24.033 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.033 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.033 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:26:24.033 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.767831+0000", 2024-09-11T08:26:24.034 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.9352999999999998e-05 2024-09-11T08:26:24.034 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:26:24.034 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:26:24.034 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:26:24.034 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.034 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.034 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:257667 10.9 10:919671da:::benchmark_data_smithi150_158067_object127423:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-09-11T08:26:24.034 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:25:41.852473+0000", 2024-09-11T08:26:24.034 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 42.053758211999998, 2024-09-11T08:26:24.034 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.91546256800000003, 2024-09-11T08:26:24.034 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:26:24.034 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:26:24.034 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:26:24.035 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:26:24.035 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:26:24.035 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 257667 2024-09-11T08:26:24.035 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.035 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:26:24.035 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.035 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:26:24.035 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.852473+0000", 2024-09-11T08:26:24.035 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:26:24.035 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.035 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.035 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:26:24.035 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.852473+0000", 2024-09-11T08:26:24.036 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:26:24.036 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.036 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.036 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:26:24.036 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.852471+0000", 2024-09-11T08:26:24.036 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:26:24.036 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.036 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.036 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:26:24.036 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.852479+0000", 2024-09-11T08:26:24.036 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.0239999999999997e-06 2024-09-11T08:26:24.036 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.037 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.037 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:26:24.037 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.852494+0000", 2024-09-11T08:26:24.037 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4826e-05 2024-09-11T08:26:24.037 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.037 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.037 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:26:24.037 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.852501+0000", 2024-09-11T08:26:24.037 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.3640000000000001e-06 2024-09-11T08:26:24.037 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.037 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.037 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:26:24.037 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.852556+0000", 2024-09-11T08:26:24.038 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.5037999999999997e-05 2024-09-11T08:26:24.038 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.038 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.038 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:26:24.038 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.852567+0000", 2024-09-11T08:26:24.038 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.024e-05 2024-09-11T08:26:24.038 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.038 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.038 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:26:24.038 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.760268+0000", 2024-09-11T08:26:24.038 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.90770139999999999 2024-09-11T08:26:24.038 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.038 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.039 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:26:24.039 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.760368+0000", 2024-09-11T08:26:24.039 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.9957999999999994e-05 2024-09-11T08:26:24.039 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.039 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.039 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,7", 2024-09-11T08:26:24.039 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.760476+0000", 2024-09-11T08:26:24.039 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000108194 2024-09-11T08:26:24.039 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.039 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.039 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:26:24.039 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.767480+0000", 2024-09-11T08:26:24.040 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.007003713 2024-09-11T08:26:24.040 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.040 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.040 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:26:24.040 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.767489+0000", 2024-09-11T08:26:24.040 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.2210000000000003e-06 2024-09-11T08:26:24.040 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.040 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.040 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:26:24.040 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.767868+0000", 2024-09-11T08:26:24.040 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00037919899999999999 2024-09-11T08:26:24.040 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.040 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.040 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:26:24.041 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.767904+0000", 2024-09-11T08:26:24.041 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.5939999999999998e-05 2024-09-11T08:26:24.041 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.041 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.041 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:26:24.041 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.767935+0000", 2024-09-11T08:26:24.041 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.1050000000000003e-05 2024-09-11T08:26:24.041 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:26:24.041 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:26:24.041 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:26:24.041 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.041 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.041 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:257682 10.9 10:98bd2a93:::benchmark_data_smithi150_158067_object127438:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-09-11T08:26:24.042 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:25:41.863234+0000", 2024-09-11T08:26:24.042 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 42.042997026000002, 2024-09-11T08:26:24.042 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.91371584900000002, 2024-09-11T08:26:24.042 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:26:24.042 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:26:24.042 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:26:24.042 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:26:24.042 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:26:24.042 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 257682 2024-09-11T08:26:24.042 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.042 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:26:24.042 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.042 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:26:24.042 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.863234+0000", 2024-09-11T08:26:24.043 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:26:24.043 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.043 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.043 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:26:24.043 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.863234+0000", 2024-09-11T08:26:24.043 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:26:24.043 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.043 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.043 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:26:24.043 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.863233+0000", 2024-09-11T08:26:24.043 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:26:24.043 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.043 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.044 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:26:24.044 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.863238+0000", 2024-09-11T08:26:24.044 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.6199999999999998e-06 2024-09-11T08:26:24.044 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.044 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.044 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:26:24.044 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.863250+0000", 2024-09-11T08:26:24.044 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2194e-05 2024-09-11T08:26:24.044 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.044 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.044 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:26:24.044 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.863255+0000", 2024-09-11T08:26:24.044 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.2290000000000002e-06 2024-09-11T08:26:24.045 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.045 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.045 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:26:24.045 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.863308+0000", 2024-09-11T08:26:24.045 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.3270000000000003e-05 2024-09-11T08:26:24.045 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.045 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.045 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:26:24.045 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.863316+0000", 2024-09-11T08:26:24.045 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.8329999999999994e-06 2024-09-11T08:26:24.045 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.045 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.045 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:26:24.045 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.764311+0000", 2024-09-11T08:26:24.046 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.90099498099999997 2024-09-11T08:26:24.046 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.046 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.046 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:26:24.046 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.764396+0000", 2024-09-11T08:26:24.046 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.4468000000000005e-05 2024-09-11T08:26:24.046 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.046 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.046 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,7", 2024-09-11T08:26:24.046 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.764493+0000", 2024-09-11T08:26:24.046 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.7134999999999997e-05 2024-09-11T08:26:24.046 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.047 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.047 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:26:24.047 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.767838+0000", 2024-09-11T08:26:24.047 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.003344914 2024-09-11T08:26:24.047 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.047 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.047 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:26:24.047 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.776429+0000", 2024-09-11T08:26:24.047 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0085910139999999992 2024-09-11T08:26:24.047 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.047 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.047 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:26:24.047 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.776513+0000", 2024-09-11T08:26:24.047 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.4544999999999998e-05 2024-09-11T08:26:24.048 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.048 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.048 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:26:24.048 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.776557+0000", 2024-09-11T08:26:24.048 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.3744e-05 2024-09-11T08:26:24.048 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.048 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.048 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:26:24.048 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.776950+0000", 2024-09-11T08:26:24.048 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000392731 2024-09-11T08:26:24.048 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:26:24.048 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:26:24.048 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:26:24.048 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.049 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.049 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:257707 10.9 10:945d1c69:::benchmark_data_smithi150_158067_object127463:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-09-11T08:26:24.049 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:25:41.889556+0000", 2024-09-11T08:26:24.049 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 42.01667466, 2024-09-11T08:26:24.049 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.90106649900000002, 2024-09-11T08:26:24.049 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:26:24.049 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:26:24.049 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:26:24.049 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:26:24.049 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:26:24.049 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 257707 2024-09-11T08:26:24.049 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.050 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:26:24.050 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.050 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:26:24.050 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.889556+0000", 2024-09-11T08:26:24.050 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:26:24.050 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.050 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.050 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:26:24.050 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.889556+0000", 2024-09-11T08:26:24.050 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:26:24.050 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.050 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.050 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:26:24.051 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.889555+0000", 2024-09-11T08:26:24.051 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:26:24.051 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.051 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.051 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:26:24.051 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.889562+0000", 2024-09-11T08:26:24.051 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.8619999999999997e-06 2024-09-11T08:26:24.051 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.051 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.051 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:26:24.051 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.889575+0000", 2024-09-11T08:26:24.051 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3456e-05 2024-09-11T08:26:24.051 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.052 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.052 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:26:24.052 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.889580+0000", 2024-09-11T08:26:24.052 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.2860000000000001e-06 2024-09-11T08:26:24.052 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.052 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.052 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:26:24.052 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.889629+0000", 2024-09-11T08:26:24.052 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.977e-05 2024-09-11T08:26:24.052 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.052 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.052 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:26:24.052 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.889639+0000", 2024-09-11T08:26:24.053 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.944e-06 2024-09-11T08:26:24.053 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.053 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.053 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:26:24.053 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.767958+0000", 2024-09-11T08:26:24.053 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.87831847799999996 2024-09-11T08:26:24.053 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.053 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.053 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:26:24.053 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.768059+0000", 2024-09-11T08:26:24.053 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00010106 2024-09-11T08:26:24.053 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.053 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.054 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,7", 2024-09-11T08:26:24.054 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.768172+0000", 2024-09-11T08:26:24.054 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000113616 2024-09-11T08:26:24.054 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.054 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.054 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:26:24.054 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.776472+0000", 2024-09-11T08:26:24.054 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0083000000000000001 2024-09-11T08:26:24.054 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.054 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.054 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:26:24.054 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.776481+0000", 2024-09-11T08:26:24.054 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.7280000000000001e-06 2024-09-11T08:26:24.054 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.055 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.055 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:26:24.055 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.790537+0000", 2024-09-11T08:26:24.055 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.014056325999999999 2024-09-11T08:26:24.055 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.055 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.055 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:26:24.055 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.790582+0000", 2024-09-11T08:26:24.055 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.4231999999999999e-05 2024-09-11T08:26:24.055 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.055 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.055 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:26:24.055 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.790623+0000", 2024-09-11T08:26:24.056 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1119999999999999e-05 2024-09-11T08:26:24.056 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:26:24.056 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:26:24.056 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:26:24.056 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.056 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.056 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:257736 10.9 10:981ba157:::benchmark_data_smithi150_158067_object127492:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-09-11T08:26:24.056 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:25:41.920790+0000", 2024-09-11T08:26:24.056 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 41.985441293000001, 2024-09-11T08:26:24.056 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.86997259400000004, 2024-09-11T08:26:24.056 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:26:24.056 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:26:24.056 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:26:24.056 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:26:24.057 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:26:24.057 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 257736 2024-09-11T08:26:24.057 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.057 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:26:24.057 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.057 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:26:24.057 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.920790+0000", 2024-09-11T08:26:24.057 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:26:24.057 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.057 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.057 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:26:24.057 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.920790+0000", 2024-09-11T08:26:24.057 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:26:24.058 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.058 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.058 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:26:24.058 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.920788+0000", 2024-09-11T08:26:24.058 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:26:24.058 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.058 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.058 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:26:24.058 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.920796+0000", 2024-09-11T08:26:24.058 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.9859999999999994e-06 2024-09-11T08:26:24.058 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.058 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.058 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:26:24.058 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.920811+0000", 2024-09-11T08:26:24.059 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4701e-05 2024-09-11T08:26:24.059 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.059 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.059 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:26:24.059 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.920818+0000", 2024-09-11T08:26:24.059 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.4289999999999998e-06 2024-09-11T08:26:24.059 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.059 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.059 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:26:24.059 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.920870+0000", 2024-09-11T08:26:24.059 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.2927000000000003e-05 2024-09-11T08:26:24.059 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.059 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.059 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:26:24.060 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.920882+0000", 2024-09-11T08:26:24.060 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1199999999999999e-05 2024-09-11T08:26:24.060 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.060 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.060 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:26:24.060 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.772565+0000", 2024-09-11T08:26:24.060 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.85168382799999998 2024-09-11T08:26:24.060 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.060 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.060 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:26:24.060 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.772688+0000", 2024-09-11T08:26:24.060 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000122657 2024-09-11T08:26:24.060 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.060 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.061 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,7", 2024-09-11T08:26:24.061 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.772824+0000", 2024-09-11T08:26:24.061 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00013596699999999999 2024-09-11T08:26:24.061 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.061 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.061 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:26:24.061 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.777003+0000", 2024-09-11T08:26:24.061 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0041784910000000003 2024-09-11T08:26:24.061 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.061 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.061 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:26:24.061 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.777008+0000", 2024-09-11T08:26:24.062 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.5840000000000002e-06 2024-09-11T08:26:24.062 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.062 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.062 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:26:24.062 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.790695+0000", 2024-09-11T08:26:24.062 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.013687220999999999 2024-09-11T08:26:24.062 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.062 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.062 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:26:24.062 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.790731+0000", 2024-09-11T08:26:24.062 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.5442999999999999e-05 2024-09-11T08:26:24.062 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:26:24.062 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:26:24.062 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:26:24.063 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.790762+0000", 2024-09-11T08:26:24.063 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.1469e-05 2024-09-11T08:26:24.063 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:26:24.063 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:26:24.063 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:26:24.063 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:26:24.063 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:26:24.063 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-11T08:26:24.063 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:23 smithi150 ceph-mon[29276]: pgmap v858: 132 pgs: 132 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-11T08:26:24.064 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:23 smithi150 ceph-mon[39612]: pgmap v858: 132 pgs: 132 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-11T08:26:24.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:23 smithi181 ceph-mon[32701]: pgmap v858: 132 pgs: 132 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-11T08:26:24.131 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:24.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:24.154 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:24.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:24.255 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:24.281 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:24.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:24.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:24.383 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:24.408 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:24.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:24.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:24.510 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:24.536 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:24.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:24.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:24.638 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:24.662 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:24.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:24.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:24.764 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:24.789 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:24.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:24.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:24.892 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:24.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:24.917 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:24.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:24.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:24 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:26:24.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:24 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:26:24.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:24 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:26:24.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:24 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:26:25.018 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:25.043 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:25.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:25.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:25.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:24 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:26:25.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:24 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:26:25.145 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:25.167 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-11T08:26:25.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:25.170 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:25.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:25.271 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:25.294 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:25.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:25.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:25.396 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:25.418 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:25.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:25.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:25.520 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:25.542 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:25.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:25.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:25.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:26:25 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:26:25] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:26:25.639 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:26:25.644 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:25.662 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd reweight 7 1 2024-09-11T08:26:25.679 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:25.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:25.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:25.685 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:26:25.781 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:25.803 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:25.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:25.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:25.905 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:26:25.927 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:25.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:25.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:26:25.955 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:25 smithi150 ceph-mon[29276]: osdmap e381: 8 total, 8 up, 8 in 2024-09-11T08:26:25.955 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:25 smithi150 ceph-mon[29276]: pgmap v860: 132 pgs: 132 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:26:25.956 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:25 smithi150 ceph-mon[39612]: osdmap e381: 8 total, 8 up, 8 in 2024-09-11T08:26:25.956 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:25 smithi150 ceph-mon[39612]: pgmap v860: 132 pgs: 132 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:26:26.029 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:26.053 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:26.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:26.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:26.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:25 smithi181 ceph-mon[32701]: osdmap e381: 8 total, 8 up, 8 in 2024-09-11T08:26:26.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:25 smithi181 ceph-mon[32701]: pgmap v860: 132 pgs: 132 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:26:26.155 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:26.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:26.181 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:26.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:26.283 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:26.307 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:26.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:26.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:26.409 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:26.432 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:26.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:26.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:26.534 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:26.563 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:26.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:26.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:26.665 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:26.688 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:26.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:26.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:26.753 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'unique_pool_4' created 2024-09-11T08:26:26.790 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:26.814 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:26.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:26.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:26.916 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:26.938 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:26.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:26.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:27.040 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:27.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:27.065 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:27.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:27.072 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:26 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1227729970' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-09-11T08:26:27.166 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:27.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:27.191 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:27.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:27.210 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:26 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1227729970' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-09-11T08:26:27.211 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:26 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1227729970' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-09-11T08:26:27.292 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:27.316 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:27.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:27.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:27.418 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:27.442 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:27.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:27.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:27.544 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:26:27.568 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:27.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:27.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:26:27.670 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:27.695 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:27.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:27.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:27.797 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:27.823 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:27.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:27.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:27.924 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:26:27.947 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:27.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:27.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:26:27.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:27 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1227729970' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-11T08:26:27.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:27 smithi150 ceph-mon[39612]: osdmap e382: 8 total, 8 up, 8 in 2024-09-11T08:26:27.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:27 smithi150 ceph-mon[39612]: pgmap v862: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:26:27.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:27 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1227729970' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-11T08:26:27.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:27 smithi150 ceph-mon[29276]: osdmap e382: 8 total, 8 up, 8 in 2024-09-11T08:26:27.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:27 smithi150 ceph-mon[29276]: pgmap v862: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:26:28.049 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:28.071 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:28.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:28.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:28.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:27 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1227729970' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-11T08:26:28.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:27 smithi181 ceph-mon[32701]: osdmap e382: 8 total, 8 up, 8 in 2024-09-11T08:26:28.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:27 smithi181 ceph-mon[32701]: pgmap v862: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:26:28.172 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:28.198 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:28.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:28.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:28.300 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:28.323 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:28.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:28.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:28.425 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:28.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:28.448 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:28.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:28.550 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:28.572 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:28.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:28.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:28.674 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:28.696 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:28.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:28.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:28.738 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:26:28 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:26:28] "GET /metrics HTTP/1.1" 200 221993 "" "Prometheus/2.43.0" 2024-09-11T08:26:28.798 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:28.821 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:28.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:28.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:28.922 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:28.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:28.947 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:28.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:29.042 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:28 smithi150 ceph-mon[29276]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-11T08:26:29.042 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:28 smithi150 ceph-mon[29276]: osdmap e383: 8 total, 8 up, 8 in 2024-09-11T08:26:29.042 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:28 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:26:29.043 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:28 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:26:29.043 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:28 smithi150 ceph-mon[39612]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-11T08:26:29.043 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:28 smithi150 ceph-mon[39612]: osdmap e383: 8 total, 8 up, 8 in 2024-09-11T08:26:29.043 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:28 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:26:29.044 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:28 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:26:29.048 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:26:29.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:29.072 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:29.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:26:29.073 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:26:28 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:26:29.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:28 smithi181 ceph-mon[32701]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-11T08:26:29.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:28 smithi181 ceph-mon[32701]: osdmap e383: 8 total, 8 up, 8 in 2024-09-11T08:26:29.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:28 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:26:29.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:28 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:26:29.173 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:29.194 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:29.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:29.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:29.296 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:29.318 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:29.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:29.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:29.419 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:29.441 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:29.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:29.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:29.543 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:29.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:29.568 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:29.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:29.617 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-09-11T08:26:29.669 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:29.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:29.691 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:29.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:29.771 INFO:teuthology.orchestra.run.smithi150.stderr:nodeep-scrub is set 2024-09-11T08:26:29.792 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:26:29.816 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:29.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:29.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:26:29.840 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:29 smithi150 ceph-mon[29276]: osdmap e384: 8 total, 8 up, 8 in 2024-09-11T08:26:29.840 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:29 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:26:29.840 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:29 smithi150 ceph-mon[29276]: pgmap v865: 148 pgs: 1 unknown, 147 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-11T08:26:29.840 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:29 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/789668070' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:26:29.840 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:29 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:26:29.840 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:29 smithi150 ceph-mon[39612]: osdmap e384: 8 total, 8 up, 8 in 2024-09-11T08:26:29.840 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:29 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:26:29.841 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:29 smithi150 ceph-mon[39612]: pgmap v865: 148 pgs: 1 unknown, 147 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-11T08:26:29.841 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:29 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/789668070' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:26:29.841 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:29 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:26:29.918 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:29.943 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:29.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:29.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:30.046 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:26:30.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:30.069 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:30.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:26:30.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:29 smithi181 ceph-mon[32701]: osdmap e384: 8 total, 8 up, 8 in 2024-09-11T08:26:30.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:29 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:26:30.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:29 smithi181 ceph-mon[32701]: pgmap v865: 148 pgs: 1 unknown, 147 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-11T08:26:30.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:29 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/789668070' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:26:30.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:29 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:26:30.171 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:30.196 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:30.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:30.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:30.298 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:26:30.320 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:30.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:30.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:26:30.423 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:30.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:30.450 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:30.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:30.551 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:26:30.574 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:30.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:30.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:26:30.675 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:30.701 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:30.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:30.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:30.803 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:30.828 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:30.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:30.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:30.930 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:30.955 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:30.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:30.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:31.057 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:26:31.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:30 smithi181 ceph-mon[32701]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:26:31.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:30 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:26:31.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:30 smithi181 ceph-mon[32701]: osdmap e385: 8 total, 8 up, 8 in 2024-09-11T08:26:31.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:30 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:26:31.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:30 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:26:31.078 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:31.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:31.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:26:31.091 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:30 smithi150 ceph-mon[29276]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:26:31.091 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:30 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:26:31.091 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:30 smithi150 ceph-mon[29276]: osdmap e385: 8 total, 8 up, 8 in 2024-09-11T08:26:31.091 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:30 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:26:31.091 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:30 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:26:31.091 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:30 smithi150 ceph-mon[39612]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:26:31.092 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:30 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:26:31.092 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:30 smithi150 ceph-mon[39612]: osdmap e385: 8 total, 8 up, 8 in 2024-09-11T08:26:31.092 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:30 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:26:31.092 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:30 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:26:31.101 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:26:31.101 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:26:31.107 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:26:31.107 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:26:31.129 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:26:31.129 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:26:31.139 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:26:31.148 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:26:31.148 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:26:31.164 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:26:31.165 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:26:31.180 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:26:31.180 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:26:31.180 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:26:31.196 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:26:31.196 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:26:31.202 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:31.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:31.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:26:31.212 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:26:31.212 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:26:31.228 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:26:31.229 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:26:31.304 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:31.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:31.331 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:31.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:31.432 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:26:31.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:31.456 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:31.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:26:31.558 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:26:31.579 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:31.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:31.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:26:31.681 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:31.708 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:31.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:31.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:31.810 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:31.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:31.837 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:31.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:31.938 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:26:31.944 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:31 smithi150 ceph-mon[29276]: osdmap e386: 8 total, 8 up, 8 in 2024-09-11T08:26:31.944 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:31 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2286605429' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:26:31.944 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:31 smithi150 ceph-mon[29276]: pgmap v868: 148 pgs: 148 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:26:31.944 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:31 smithi150 ceph-mon[39612]: osdmap e386: 8 total, 8 up, 8 in 2024-09-11T08:26:31.944 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:31 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2286605429' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:26:31.945 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:31 smithi150 ceph-mon[39612]: pgmap v868: 148 pgs: 148 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:26:31.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:31.962 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:31.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:26:32.063 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:32.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:31 smithi181 ceph-mon[32701]: osdmap e386: 8 total, 8 up, 8 in 2024-09-11T08:26:32.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:31 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2286605429' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:26:32.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:31 smithi181 ceph-mon[32701]: pgmap v868: 148 pgs: 148 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:26:32.089 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:32.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:32.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:32.191 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:32.216 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:32.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:32.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:32.317 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:32.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:32.341 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:32.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:32.442 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:26:32.464 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:32.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:32.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:26:32.566 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:32.591 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:32.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:32.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:32.693 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:32.719 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:32.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:32.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:32.821 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:32.840 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T08:26:32.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:32.847 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:32.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:32.948 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:32.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:32.973 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:32.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:33.075 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:33.098 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:33.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:33.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:33.200 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:33.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:33.223 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:33.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:33.324 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:33.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:33.350 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:33.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:33.451 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:33.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:33.476 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:33.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:33.577 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:33.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:33.601 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:33.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:33.702 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:33.728 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:33.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:33.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:33.829 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:33.855 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:33.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:33.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:33.957 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:26:33.978 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:33.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:33.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:26:34.080 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:34.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:34.101 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:34.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:34.202 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:26:34.224 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:34.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:34.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:26:34.326 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:34.353 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:34.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:34.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:34.455 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:34.480 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:34.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:34.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:34.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:34 smithi150 ceph-mon[29276]: pgmap v869: 148 pgs: 148 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 953 B/s rd, 0 op/s 2024-09-11T08:26:34.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:34 smithi150 ceph-mon[39612]: pgmap v869: 148 pgs: 148 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 953 B/s rd, 0 op/s 2024-09-11T08:26:34.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:34 smithi181 ceph-mon[32701]: pgmap v869: 148 pgs: 148 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 953 B/s rd, 0 op/s 2024-09-11T08:26:34.581 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:34.607 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:34.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:34.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:34.709 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:34.734 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:34.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:34.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:34.836 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:34.862 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:34.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:34.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:34.964 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:26:34.987 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:34.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:34.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:26:35.089 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:35.115 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:35.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:35.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:35.193 INFO:teuthology.orchestra.run.smithi150.stderr:reweighted osd.7 to 1 (10000) 2024-09-11T08:26:35.217 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:35.243 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:35.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:35.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:35.345 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:35.371 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:35.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:35.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:35.472 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:35.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:35 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1567986110' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-11T08:26:35.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:35 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-11T08:26:35.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:35 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1567986110' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-11T08:26:35.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:35 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-11T08:26:35.494 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:35.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:35.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:35.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:35 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1567986110' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-11T08:26:35.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:35 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-11T08:26:35.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:26:35 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:26:35] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:26:35.596 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:35.619 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:35.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:35.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:35.721 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:35.746 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:35.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:35.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:35.847 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:26:35.852 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T08:26:35.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:35.870 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:35.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:26:35.971 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:35.993 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:35.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:35.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:36.095 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:26:36.117 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:36.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:36.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:26:36.218 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:36.245 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:36.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:36.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:36.346 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:36.374 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:36.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:36.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:36.468 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:36 smithi150 ceph-mon[29276]: pgmap v870: 148 pgs: 148 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 803 B/s rd, 0 op/s 2024-09-11T08:26:36.468 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:36 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-11T08:26:36.468 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:36 smithi150 ceph-mon[29276]: osdmap e387: 8 total, 8 up, 8 in 2024-09-11T08:26:36.468 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:36 smithi150 ceph-mon[39612]: pgmap v870: 148 pgs: 148 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 803 B/s rd, 0 op/s 2024-09-11T08:26:36.469 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:36 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-11T08:26:36.469 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:36 smithi150 ceph-mon[39612]: osdmap e387: 8 total, 8 up, 8 in 2024-09-11T08:26:36.475 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:36.500 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:36.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:36.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:36.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:36 smithi181 ceph-mon[32701]: pgmap v870: 148 pgs: 148 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 803 B/s rd, 0 op/s 2024-09-11T08:26:36.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:36 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-11T08:26:36.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:36 smithi181 ceph-mon[32701]: osdmap e387: 8 total, 8 up, 8 in 2024-09-11T08:26:36.602 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:26:36.624 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:36.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:36.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:26:36.726 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:36.752 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:36.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:36.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:36.854 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:36.879 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:36.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:36.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:36.981 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:37.002 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:37.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:37.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:37.105 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:37.126 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:37.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:37.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:37.228 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:37.253 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:37.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:37.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:37.355 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:26:37.377 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:37.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:37.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:26:37.399 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:26:37.479 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:37.504 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:37.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:37.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:37.606 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:37.631 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:37.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:37.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:37.733 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:37.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:37.762 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:37.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:37.770 INFO:teuthology.orchestra.run.smithi150.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-09-11T08:26:37.863 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:37.889 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:37.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:37.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:37.946 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:37 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:26:37.946 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:37 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:26:37.946 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:37 smithi150 ceph-mon[39612]: pgmap v872: 148 pgs: 148 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 695 B/s rd, 0 op/s 2024-09-11T08:26:37.946 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:37 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/4140274542' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-11T08:26:37.947 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:37 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-11T08:26:37.947 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:37 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:26:37.947 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:37 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:26:37.947 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:37 smithi150 ceph-mon[29276]: pgmap v872: 148 pgs: 148 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 695 B/s rd, 0 op/s 2024-09-11T08:26:37.947 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:37 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/4140274542' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-11T08:26:37.947 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:37 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-11T08:26:37.991 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:38.015 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:38.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:38.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:38.021 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd reweight 5 1 2024-09-11T08:26:38.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:37 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:26:38.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:37 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:26:38.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:37 smithi181 ceph-mon[32701]: pgmap v872: 148 pgs: 148 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 695 B/s rd, 0 op/s 2024-09-11T08:26:38.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:37 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/4140274542' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-11T08:26:38.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:37 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-11T08:26:38.116 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:38.137 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:38.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:38.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:38.239 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:38.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:38.265 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:38.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:38.366 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:38.390 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:38.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:38.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:38.446 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-09-11T08:26:38.446 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:26:38.492 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:38.519 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:38.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:38.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:38.620 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:38.642 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:38.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:38.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:38.744 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:38.766 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:38.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:38.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:38.868 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:38.875 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:38 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-11T08:26:38.875 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:38 smithi150 ceph-mon[29276]: osdmap e388: 8 total, 8 up, 8 in 2024-09-11T08:26:38.876 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:26:38 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:26:38] "GET /metrics HTTP/1.1" 200 224320 "" "Prometheus/2.43.0" 2024-09-11T08:26:38.876 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:38 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-11T08:26:38.876 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:38 smithi150 ceph-mon[39612]: osdmap e388: 8 total, 8 up, 8 in 2024-09-11T08:26:38.894 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:38.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:38.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:38.996 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:26:39.017 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:39.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:39.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:26:39.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:38 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-11T08:26:39.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:38 smithi181 ceph-mon[32701]: osdmap e388: 8 total, 8 up, 8 in 2024-09-11T08:26:39.073 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:26:38 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:26:39.119 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:39.145 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:39.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:39.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:39.246 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:26:39.268 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:39.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:39.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:26:39.370 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:26:39.391 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:39.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:39.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:26:39.493 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:39.519 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:39.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:39.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:39.621 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:39.647 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:39.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:39.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:39.749 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:39.775 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:39.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:39.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:39.794 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:26:39.817 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-11T08:26:39.877 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:39.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:39.901 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:39.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:40.002 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:40.013 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:39 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:26:40.013 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:39 smithi150 ceph-mon[29276]: pgmap v874: 148 pgs: 148 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:26:40.014 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:39 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:26:40.014 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:39 smithi150 ceph-mon[39612]: pgmap v874: 148 pgs: 148 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:26:40.030 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:40.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:40.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:40.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:39 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:26:40.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:39 smithi181 ceph-mon[32701]: pgmap v874: 148 pgs: 148 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:26:40.131 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:40.157 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:40.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:40.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:40.259 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:40.282 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:40.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:40.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:40.384 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:40.395 DEBUG:teuthology.orchestra.run.smithi150:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-09-11T08:26:40.398 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-11T08:26:40.398 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-11T08:26:40.410 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:40.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:40.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:40.512 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:26:40.534 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:40.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:40.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:26:40.610 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:hints = 1 2024-09-11T08:26:40.610 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-11T08:26:40.610 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Object prefix: benchmark_data_smithi150_187410 2024-09-11T08:26:40.610 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-11T08:26:40.611 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 0 0 0 0 0 0 - 0 2024-09-11T08:26:40.635 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:40.661 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:40.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:40.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:40.763 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:40.797 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:40.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:40.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:40.899 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:40.934 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:40.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:40.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:41.036 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:41.063 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:40 smithi150 ceph-mon[39612]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-11T08:26:41.063 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:40 smithi150 ceph-mon[29276]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-11T08:26:41.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:40 smithi181 ceph-mon[32701]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-11T08:26:41.075 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:41.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:41.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:41.177 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:41.216 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:41.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:41.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:41.291 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:26:41.318 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:41.370 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:41.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:41.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:41.472 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:41.506 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:41.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:41.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:41.608 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:41.610 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 1 16 3653 3637 227.3 227.312 0.00255879 0.0043812 2024-09-11T08:26:41.636 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:41.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:41.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:41.738 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:41.773 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:41.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:41.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:41.875 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:41.906 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:41.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:41.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:42.008 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:42.050 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:42.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:42.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:42.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:41 smithi181 ceph-mon[32701]: pgmap v875: 148 pgs: 148 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:26:42.152 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:42.184 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:42.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:42.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:42.213 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:41 smithi150 ceph-mon[39612]: pgmap v875: 148 pgs: 148 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:26:42.214 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:41 smithi150 ceph-mon[29276]: pgmap v875: 148 pgs: 148 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:26:42.286 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:42.320 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:42.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:42.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:42.422 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:42.452 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:42.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:42.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:42.554 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:26:42.593 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:42.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:42.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:26:42.610 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 2 16 7104 7088 221.484 215.688 0.00741541 0.00450342 2024-09-11T08:26:42.695 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:42.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:42.737 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:42.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:42.838 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:42.845 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:26:42.846 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:26:42.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:42.878 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:42.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:42.879 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:26:42.879 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:26:42.911 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:26:42.912 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:26:42.943 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:26:42.944 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:26:42.973 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:26:42.973 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:26:42.979 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:43.004 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:26:43.004 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:26:43.017 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:43.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:43.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:43.033 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:26:43.033 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:26:43.065 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:42 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2698136694' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:26:43.065 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:26:43.066 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:26:43.066 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:42 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2698136694' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:26:43.118 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:43.152 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:43.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:43.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:43.254 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:43.292 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:43.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:43.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:43.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:42 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2698136694' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:26:43.394 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:43.424 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:43.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:43.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:43.526 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:26:43.561 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:43.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:43.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:26:43.611 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 3 15 10154 10139 211.212 190.688 0.0073578 0.00472851 2024-09-11T08:26:43.664 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:43.685 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:43.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:43.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:43.787 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:26:43.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:43.825 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:43.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:26:43.926 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:43.959 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:43.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:43.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:44.060 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:44.093 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:44.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:44.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:44.195 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:44.227 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:44.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:44.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:44.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:43 smithi150 ceph-mon[29276]: pgmap v876: 148 pgs: 148 active+clean; 33 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 4.0 MiB/s wr, 64 op/s 2024-09-11T08:26:44.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:43 smithi150 ceph-mon[39612]: pgmap v876: 148 pgs: 148 active+clean; 33 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 4.0 MiB/s wr, 64 op/s 2024-09-11T08:26:44.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:43 smithi181 ceph-mon[32701]: pgmap v876: 148 pgs: 148 active+clean; 33 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 4.0 MiB/s wr, 64 op/s 2024-09-11T08:26:44.329 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:44.361 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:44.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:44.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:44.463 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:44.496 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:44.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:44.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:44.597 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:44.611 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 4 16 13143 13127 205.093 186.75 0.00630445 0.00486658 2024-09-11T08:26:44.625 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:44.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:44.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:44.727 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:44.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:44.760 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:44.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:44.861 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:44.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:44.894 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:44.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:44.995 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:45.038 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:45.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:45.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:45.139 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:45.169 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:45.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:45.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:45.271 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:45.305 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:45.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:45.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:45.407 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:45.446 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:45.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:45.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:45.548 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:45.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:26:45 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:26:45] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:26:45.582 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:45.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:45.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:45.611 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 5 15 16163 16148 201.833 188.812 0.0064838 0.00494866 2024-09-11T08:26:45.683 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:26:45.700 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:26:45.736 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:45.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:45.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:26:45.838 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:45.882 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:45.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:45.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:45.984 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:46.016 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:46.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:46.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:46.118 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:46.155 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:46.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:46.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:46.234 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:46 smithi150 ceph-mon[29276]: pgmap v877: 148 pgs: 148 active+clean; 595 MiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 60 MiB/s wr, 957 op/s 2024-09-11T08:26:46.235 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:46 smithi150 ceph-mon[39612]: pgmap v877: 148 pgs: 148 active+clean; 595 MiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 60 MiB/s wr, 957 op/s 2024-09-11T08:26:46.257 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:46.299 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:46.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:46.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:46.402 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:46.432 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:46.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:46.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:46.534 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:46.568 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:46.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:46.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:46.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:46 smithi181 ceph-mon[32701]: pgmap v877: 148 pgs: 148 active+clean; 595 MiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 60 MiB/s wr, 957 op/s 2024-09-11T08:26:46.611 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 6 16 18753 18737 195.161 161.812 0.00814068 0.00511766 2024-09-11T08:26:46.670 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:46.703 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:46.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:46.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:46.805 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:26:46.827 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:46.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:46.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:26:46.928 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:46.964 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:46.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:46.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:47.024 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T08:26:47.066 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:47.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:47.111 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:47.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:47.212 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:47.240 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:47.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:47.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:47.342 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:47.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:47.385 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:47.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:47.487 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:47.520 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:47.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:47.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:47.611 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 7 16 20627 20611 184.011 117.125 0.0023835 0.00542768 2024-09-11T08:26:47.621 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:47.653 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:47.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:47.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:47.755 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:26:47.791 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:47.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:47.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:26:47.896 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:26:47.935 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:47.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:47.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:26:48.038 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:48.074 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:48.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:48.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:48.176 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:26:48.216 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:48.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:48.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:26:48.318 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:48.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:48.339 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:48.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:48.440 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:48.474 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:48.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:48.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:48.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:48 smithi150 ceph-mon[29276]: pgmap v878: 148 pgs: 148 active+clean; 595 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 59 MiB/s wr, 951 op/s 2024-09-11T08:26:48.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:48 smithi150 ceph-mon[39612]: pgmap v878: 148 pgs: 148 active+clean; 595 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 59 MiB/s wr, 951 op/s 2024-09-11T08:26:48.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:48 smithi181 ceph-mon[32701]: pgmap v878: 148 pgs: 148 active+clean; 595 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 59 MiB/s wr, 951 op/s 2024-09-11T08:26:48.577 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:48.611 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 8 16 22867 22851 178.508 140 0.00472229 0.00559374 2024-09-11T08:26:48.615 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:48.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:48.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:48.717 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:48.749 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:48.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:48.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:48.851 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:26:48.884 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:48.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:48.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:26:48.986 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:48.988 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:26:48 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:26:48] "GET /metrics HTTP/1.1" 200 224320 "" "Prometheus/2.43.0" 2024-09-11T08:26:49.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:49.008 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:49.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:49.109 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:49.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:49.132 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:49.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:49.194 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:26:48 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:26:49.233 INFO:teuthology.orchestra.run.smithi150.stderr:reweighted osd.5 to 1 (10000) 2024-09-11T08:26:49.233 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:49.277 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:49.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:49.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:49.378 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:49.412 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:49.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:49.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:49.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:49 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3617428583' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-11T08:26:49.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:49 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3617428583' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-11T08:26:49.514 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:49.546 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:49.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:49.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:49.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:49 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3617428583' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-11T08:26:49.611 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 9 16 24685 24669 171.298 113.625 0.00687196 0.00583171 2024-09-11T08:26:49.649 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:26:49.688 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:49.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:49.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:26:49.790 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:49.814 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:49.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:49.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:49.915 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:49.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:49.953 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:49.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:50.054 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:50.088 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:50.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:50.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:50.190 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:26:50.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:50.230 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:50.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:26:50.331 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:50.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:50.369 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:50.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:50.470 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:50.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:50 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:26:50.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:50 smithi150 ceph-mon[29276]: pgmap v879: 148 pgs: 148 active+clean; 1.3 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1.3 KiB/s rd, 119 MiB/s wr, 1.91k op/s 2024-09-11T08:26:50.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:50 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3617428583' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-11T08:26:50.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:50 smithi150 ceph-mon[29276]: osdmap e389: 8 total, 8 up, 8 in 2024-09-11T08:26:50.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:50 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:26:50.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:50 smithi150 ceph-mon[39612]: pgmap v879: 148 pgs: 148 active+clean; 1.3 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1.3 KiB/s rd, 119 MiB/s wr, 1.91k op/s 2024-09-11T08:26:50.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:50 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3617428583' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-11T08:26:50.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:50 smithi150 ceph-mon[39612]: osdmap e389: 8 total, 8 up, 8 in 2024-09-11T08:26:50.502 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:50.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:50.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:50.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:50 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:26:50.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:50 smithi181 ceph-mon[32701]: pgmap v879: 148 pgs: 148 active+clean; 1.3 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1.3 KiB/s rd, 119 MiB/s wr, 1.91k op/s 2024-09-11T08:26:50.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:50 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3617428583' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-11T08:26:50.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:50 smithi181 ceph-mon[32701]: osdmap e389: 8 total, 8 up, 8 in 2024-09-11T08:26:50.604 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:50.611 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 10 15 26963 26948 168.409 142.438 0.00786653 0.0059313 2024-09-11T08:26:50.641 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:50.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:50.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:50.743 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:50.769 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:50.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:50.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:50.871 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:50.904 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:50.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:50.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:51.006 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:51.038 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:51.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:51.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:51.140 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:26:51.179 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:51.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:51.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:26:51.281 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:51.311 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:51.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:51.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:51.413 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:51.452 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:51.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:51.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:51.554 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:51.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:51.585 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:51.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:51.611 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 11 16 28865 28849 163.9 118.812 0.00589905 0.00609561 2024-09-11T08:26:51.686 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:51.714 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:51.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:51.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:51.817 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:26:51.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:51.840 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:51.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:26:51.941 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:26:51.977 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:51.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:51.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:26:51.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:51 smithi150 ceph-mon[29276]: pgmap v881: 148 pgs: 148 active+clean; 1.4 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 140 MiB/s wr, 2.24k op/s 2024-09-11T08:26:51.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:51 smithi150 ceph-mon[39612]: pgmap v881: 148 pgs: 148 active+clean; 1.4 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 140 MiB/s wr, 2.24k op/s 2024-09-11T08:26:52.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:51 smithi181 ceph-mon[32701]: pgmap v881: 148 pgs: 148 active+clean; 1.4 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 140 MiB/s wr, 2.24k op/s 2024-09-11T08:26:52.079 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:52.112 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:52.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:52.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:52.214 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:52.250 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:52.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:52.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:52.352 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:52.396 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:52.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:52.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:52.498 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:52.536 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:52.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:52.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:52.611 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 12 16 30621 30605 159.386 109.75 0.0082535 0.00626844 2024-09-11T08:26:52.638 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:52.673 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:52.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:52.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:52.771 INFO:teuthology.orchestra.run.smithi150.stderr:noscrub is unset 2024-09-11T08:26:52.775 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:52.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:52.818 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:52.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:52.904 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:52 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:26:52.904 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:52 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:26:52.904 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:52 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/611680997' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:26:52.904 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:52 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:26:52.904 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:52 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:26:52.904 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:52 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:26:52.905 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:52 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/611680997' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:26:52.905 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:52 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:26:52.919 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:52.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:52.950 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:52.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:52.951 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-11T08:26:52.952 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T08:26:53.052 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:53.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:52 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:26:53.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:52 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:26:53.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:52 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/611680997' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:26:53.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:52 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:26:53.106 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:53.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:53.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:53.208 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:53.237 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:53.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:53.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:53.339 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:26:53.384 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:53.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:53.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:26:53.486 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:26:53.520 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:53.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:53.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:26:53.612 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 13 16 32724 32708 157.236 131.438 0.00689916 0.0063546 2024-09-11T08:26:53.622 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:26:53.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:53.661 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:53.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:26:53.762 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:53.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:53.811 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:53.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:53.912 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:53.954 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:53.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:53.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:53.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:53 smithi150 ceph-mon[29276]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:26:53.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:53 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:26:53.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:53 smithi150 ceph-mon[29276]: osdmap e390: 8 total, 8 up, 8 in 2024-09-11T08:26:53.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:53 smithi150 ceph-mon[29276]: pgmap v883: 148 pgs: 148 active+clean; 1.4 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 106 MiB/s wr, 1.69k op/s 2024-09-11T08:26:53.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:53 smithi150 ceph-mon[39612]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:26:53.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:53 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:26:53.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:53 smithi150 ceph-mon[39612]: osdmap e390: 8 total, 8 up, 8 in 2024-09-11T08:26:53.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:53 smithi150 ceph-mon[39612]: pgmap v883: 148 pgs: 148 active+clean; 1.4 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 106 MiB/s wr, 1.69k op/s 2024-09-11T08:26:54.052 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:26:54.057 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:54.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:53 smithi181 ceph-mon[32701]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:26:54.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:53 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:26:54.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:53 smithi181 ceph-mon[32701]: osdmap e390: 8 total, 8 up, 8 in 2024-09-11T08:26:54.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:53 smithi181 ceph-mon[32701]: pgmap v883: 148 pgs: 148 active+clean; 1.4 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 106 MiB/s wr, 1.69k op/s 2024-09-11T08:26:54.083 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:54.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:54.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:54.185 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:54.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:54.215 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:54.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:54.316 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:54.348 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:54.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:54.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:54.449 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:26:54.492 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:54.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:54.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:26:54.594 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:26:54.612 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 14 15 34830 34815 155.4 131.688 0.00522026 0.00642986 2024-09-11T08:26:54.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:54.638 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:54.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:26:54.739 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:54.768 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:54.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:54.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:54.870 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:54.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:54.919 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:54.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:55.020 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:55.053 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:55.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:55.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:55.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:54 smithi181 ceph-mon[32701]: 2.1 scrub starts 2024-09-11T08:26:55.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:54 smithi181 ceph-mon[32701]: 2.1 scrub ok 2024-09-11T08:26:55.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:54 smithi181 ceph-mon[32701]: 4.15 scrub starts 2024-09-11T08:26:55.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:54 smithi181 ceph-mon[32701]: 4.15 scrub ok 2024-09-11T08:26:55.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:54 smithi181 ceph-mon[32701]: 6.2 scrub starts 2024-09-11T08:26:55.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:54 smithi181 ceph-mon[32701]: 6.2 scrub ok 2024-09-11T08:26:55.155 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:55.193 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:55.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:55.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:55.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:54 smithi150 ceph-mon[29276]: 2.1 scrub starts 2024-09-11T08:26:55.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:54 smithi150 ceph-mon[29276]: 2.1 scrub ok 2024-09-11T08:26:55.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:54 smithi150 ceph-mon[29276]: 4.15 scrub starts 2024-09-11T08:26:55.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:54 smithi150 ceph-mon[29276]: 4.15 scrub ok 2024-09-11T08:26:55.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:54 smithi150 ceph-mon[29276]: 6.2 scrub starts 2024-09-11T08:26:55.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:54 smithi150 ceph-mon[29276]: 6.2 scrub ok 2024-09-11T08:26:55.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:54 smithi150 ceph-mon[39612]: 2.1 scrub starts 2024-09-11T08:26:55.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:54 smithi150 ceph-mon[39612]: 2.1 scrub ok 2024-09-11T08:26:55.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:54 smithi150 ceph-mon[39612]: 4.15 scrub starts 2024-09-11T08:26:55.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:54 smithi150 ceph-mon[39612]: 4.15 scrub ok 2024-09-11T08:26:55.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:54 smithi150 ceph-mon[39612]: 6.2 scrub starts 2024-09-11T08:26:55.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:54 smithi150 ceph-mon[39612]: 6.2 scrub ok 2024-09-11T08:26:55.296 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:55.330 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:55.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:55.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:55.432 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:55.471 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:55.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:55.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:55.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:26:55 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:26:55] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:26:55.573 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:55.611 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:55.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:55.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:55.613 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 15 16 36679 36663 152.739 115.5 0.00831547 0.00654285 2024-09-11T08:26:55.714 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:55.752 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:55.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:55.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:55.783 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T08:26:55.838 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:55 smithi150 ceph-mon[29276]: 3.10 scrub starts 2024-09-11T08:26:55.838 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:55 smithi150 ceph-mon[29276]: 3.10 scrub ok 2024-09-11T08:26:55.838 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:55 smithi150 ceph-mon[29276]: pgmap v884: 148 pgs: 148 active+clean; 2.0 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 178 MiB/s wr, 2.85k op/s 2024-09-11T08:26:55.838 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:55 smithi150 ceph-mon[29276]: osdmap e391: 8 total, 8 up, 8 in 2024-09-11T08:26:55.838 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:55 smithi150 ceph-mon[39612]: 3.10 scrub starts 2024-09-11T08:26:55.838 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:55 smithi150 ceph-mon[39612]: 3.10 scrub ok 2024-09-11T08:26:55.838 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:55 smithi150 ceph-mon[39612]: pgmap v884: 148 pgs: 148 active+clean; 2.0 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 178 MiB/s wr, 2.85k op/s 2024-09-11T08:26:55.839 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:55 smithi150 ceph-mon[39612]: osdmap e391: 8 total, 8 up, 8 in 2024-09-11T08:26:55.854 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:55.886 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:55.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:55.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:55.988 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:56.016 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:56.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:56.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:56.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:55 smithi181 ceph-mon[32701]: 3.10 scrub starts 2024-09-11T08:26:56.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:55 smithi181 ceph-mon[32701]: 3.10 scrub ok 2024-09-11T08:26:56.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:55 smithi181 ceph-mon[32701]: pgmap v884: 148 pgs: 148 active+clean; 2.0 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 178 MiB/s wr, 2.85k op/s 2024-09-11T08:26:56.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:55 smithi181 ceph-mon[32701]: osdmap e391: 8 total, 8 up, 8 in 2024-09-11T08:26:56.118 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:56.153 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:56.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:56.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:56.255 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:56.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:56.294 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:56.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:56.395 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:56.430 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:56.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:56.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:56.532 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:56.567 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:56.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:56.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:56.613 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 16 16 38371 38355 149.802 105.75 0.00444076 0.00666962 2024-09-11T08:26:56.668 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:26:56.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:56.706 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:56.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:26:56.807 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:56.833 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:56.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:56.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:56.935 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:56.960 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:56.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:56.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:57.061 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:57.072 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:56 smithi181 ceph-mon[32701]: 3.4 scrub starts 2024-09-11T08:26:57.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:56 smithi181 ceph-mon[32701]: 3.4 scrub ok 2024-09-11T08:26:57.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:57.093 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:57.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:57.194 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:26:57.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:57.233 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:57.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:26:57.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:56 smithi150 ceph-mon[29276]: 3.4 scrub starts 2024-09-11T08:26:57.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:56 smithi150 ceph-mon[29276]: 3.4 scrub ok 2024-09-11T08:26:57.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:56 smithi150 ceph-mon[39612]: 3.4 scrub starts 2024-09-11T08:26:57.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:56 smithi150 ceph-mon[39612]: 3.4 scrub ok 2024-09-11T08:26:57.334 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:57.373 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:57.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:57.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:57.475 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:57.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:57.513 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:57.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:57.596 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:26:57.613 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 17 16 40028 40012 147.08 103.562 0.0162198 0.00679161 2024-09-11T08:26:57.614 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:57.640 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:57.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:57.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:57.742 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:57.781 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:57.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:57.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:57.883 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:57.909 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:57.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:57.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:58.011 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:58.036 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:58.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:58.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:58.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:57 smithi181 ceph-mon[32701]: 4.1f scrub starts 2024-09-11T08:26:58.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:57 smithi181 ceph-mon[32701]: 4.1f scrub ok 2024-09-11T08:26:58.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:57 smithi181 ceph-mon[32701]: 4.7 scrub starts 2024-09-11T08:26:58.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:57 smithi181 ceph-mon[32701]: 4.7 scrub ok 2024-09-11T08:26:58.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:57 smithi181 ceph-mon[32701]: pgmap v886: 148 pgs: 148 active+clean; 2.0 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 646 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-09-11T08:26:58.130 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:57 smithi150 ceph-mon[29276]: 4.1f scrub starts 2024-09-11T08:26:58.130 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:57 smithi150 ceph-mon[29276]: 4.1f scrub ok 2024-09-11T08:26:58.130 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:57 smithi150 ceph-mon[29276]: 4.7 scrub starts 2024-09-11T08:26:58.130 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:57 smithi150 ceph-mon[29276]: 4.7 scrub ok 2024-09-11T08:26:58.130 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:57 smithi150 ceph-mon[29276]: pgmap v886: 148 pgs: 148 active+clean; 2.0 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 646 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-09-11T08:26:58.130 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:57 smithi150 ceph-mon[39612]: 4.1f scrub starts 2024-09-11T08:26:58.131 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:57 smithi150 ceph-mon[39612]: 4.1f scrub ok 2024-09-11T08:26:58.131 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:57 smithi150 ceph-mon[39612]: 4.7 scrub starts 2024-09-11T08:26:58.131 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:57 smithi150 ceph-mon[39612]: 4.7 scrub ok 2024-09-11T08:26:58.131 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:57 smithi150 ceph-mon[39612]: pgmap v886: 148 pgs: 148 active+clean; 2.0 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 646 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-09-11T08:26:58.138 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:26:58.180 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:58.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:58.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:26:58.282 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:58.310 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:58.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:58.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:58.312 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:26:58.312 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:26:58.355 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:26:58.355 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:26:58.385 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:26:58.386 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:26:58.412 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:58.423 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:26:58.423 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:26:58.439 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:58.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:58.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:58.452 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:26:58.452 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:26:58.482 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:26:58.483 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:26:58.520 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:26:58.520 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:26:58.542 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:58.552 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:26:58.553 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:26:58.570 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:58.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:58.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:58.613 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 18 16 41637 41621 144.495 100.562 0.0122189 0.00691531 2024-09-11T08:26:58.672 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:58.702 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:58.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:58.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:58.738 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:26:58 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:26:58] "GET /metrics HTTP/1.1" 200 224506 "" "Prometheus/2.43.0" 2024-09-11T08:26:58.804 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:26:58.837 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:58.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:58.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:26:58.939 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:26:58.970 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:58.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:58.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:26:59.072 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:26:59.073 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:26:58 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:26:59.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:58 smithi181 ceph-mon[32701]: 4.1d scrub starts 2024-09-11T08:26:59.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:58 smithi181 ceph-mon[32701]: 4.1d scrub ok 2024-09-11T08:26:59.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:58 smithi181 ceph-mon[32701]: 4.2 scrub starts 2024-09-11T08:26:59.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:58 smithi181 ceph-mon[32701]: 4.2 scrub ok 2024-09-11T08:26:59.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:58 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1853909296' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:26:59.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:58 smithi181 ceph-mon[32701]: 4.11 scrub starts 2024-09-11T08:26:59.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:58 smithi181 ceph-mon[32701]: 4.11 scrub ok 2024-09-11T08:26:59.111 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:59.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:59.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:26:59.213 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:59.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:58 smithi150 ceph-mon[29276]: 4.1d scrub starts 2024-09-11T08:26:59.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:58 smithi150 ceph-mon[29276]: 4.1d scrub ok 2024-09-11T08:26:59.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:58 smithi150 ceph-mon[29276]: 4.2 scrub starts 2024-09-11T08:26:59.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:58 smithi150 ceph-mon[29276]: 4.2 scrub ok 2024-09-11T08:26:59.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:58 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1853909296' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:26:59.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:58 smithi150 ceph-mon[29276]: 4.11 scrub starts 2024-09-11T08:26:59.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:58 smithi150 ceph-mon[29276]: 4.11 scrub ok 2024-09-11T08:26:59.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:58 smithi150 ceph-mon[39612]: 4.1d scrub starts 2024-09-11T08:26:59.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:58 smithi150 ceph-mon[39612]: 4.1d scrub ok 2024-09-11T08:26:59.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:58 smithi150 ceph-mon[39612]: 4.2 scrub starts 2024-09-11T08:26:59.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:58 smithi150 ceph-mon[39612]: 4.2 scrub ok 2024-09-11T08:26:59.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:58 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1853909296' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:26:59.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:58 smithi150 ceph-mon[39612]: 4.11 scrub starts 2024-09-11T08:26:59.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:58 smithi150 ceph-mon[39612]: 4.11 scrub ok 2024-09-11T08:26:59.254 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:59.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:59.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:59.356 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:26:59.400 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:59.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:59.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:26:59.503 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:26:59.540 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:59.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:59.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:26:59.613 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 19 16 43586 43570 143.301 121.812 0.00421328 0.00697354 2024-09-11T08:26:59.642 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:26:59.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:59.679 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:59.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:26:59.780 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:26:59.817 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:26:59.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:59.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:26:59.918 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:26:59.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:59.955 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:26:59.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:00.056 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:00.089 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:00.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:00.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:00.122 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:59 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:27:00.122 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:26:59 smithi150 ceph-mon[29276]: pgmap v887: 148 pgs: 148 active+clean; 2.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 143 MiB/s wr, 2.29k op/s 2024-09-11T08:27:00.122 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:59 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:27:00.122 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:26:59 smithi150 ceph-mon[39612]: pgmap v887: 148 pgs: 148 active+clean; 2.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 143 MiB/s wr, 2.29k op/s 2024-09-11T08:27:00.190 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:00.224 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:00.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:00.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:00.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:59 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:27:00.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:26:59 smithi181 ceph-mon[32701]: pgmap v887: 148 pgs: 148 active+clean; 2.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 143 MiB/s wr, 2.29k op/s 2024-09-11T08:27:00.326 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:27:00.353 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:00.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:00.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:27:00.396 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:27:00.428 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-09-11T08:27:00.428 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:27:00.454 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:00.481 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:00.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:00.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:00.583 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:27:00.613 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:2024-09-11T08:27:00.614318+0000 min lat: 0.00120292 max lat: 0.208687 avg lat: 0.00705119 2024-09-11T08:27:00.613 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-11T08:27:00.614 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 20 16 45372 45356 141.716 111.625 0.00616351 0.00705119 2024-09-11T08:27:00.619 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:00.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:00.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:27:00.721 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:00.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:00.766 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:00.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:00.867 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:00.903 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:00.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:00.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:01.005 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:01.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:01.035 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:01.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:01.136 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:01.141 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:00 smithi150 ceph-mon[29276]: 4.e scrub starts 2024-09-11T08:27:01.141 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:00 smithi150 ceph-mon[29276]: 4.e scrub ok 2024-09-11T08:27:01.142 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:00 smithi150 ceph-mon[29276]: osdmap e392: 8 total, 8 up, 8 in 2024-09-11T08:27:01.142 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:00 smithi150 ceph-mon[39612]: 4.e scrub starts 2024-09-11T08:27:01.142 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:00 smithi150 ceph-mon[39612]: 4.e scrub ok 2024-09-11T08:27:01.142 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:00 smithi150 ceph-mon[39612]: osdmap e392: 8 total, 8 up, 8 in 2024-09-11T08:27:01.167 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:01.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:01.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:01.269 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:01.305 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:01.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:01.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:01.322 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:00 smithi181 ceph-mon[32701]: 4.e scrub starts 2024-09-11T08:27:01.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:00 smithi181 ceph-mon[32701]: 4.e scrub ok 2024-09-11T08:27:01.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:00 smithi181 ceph-mon[32701]: osdmap e392: 8 total, 8 up, 8 in 2024-09-11T08:27:01.342 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-11T08:27:01.407 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:27:01.437 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:01.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:01.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:27:01.539 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:27:01.565 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:01.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:01.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:27:01.617 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 21 16 47161 47145 140.292 111.812 0.0149871 0.00712222 2024-09-11T08:27:01.668 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:01.707 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:01.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:01.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:01.809 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:27:01.843 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:01.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:01.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:27:01.866 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T08:27:01.944 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:01.986 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:01.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:01.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:02.088 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:27:02.115 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:02.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:02.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:27:02.217 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:02.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:01 smithi150 ceph-mon[29276]: 6.c scrub starts 2024-09-11T08:27:02.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:01 smithi150 ceph-mon[29276]: 6.c scrub ok 2024-09-11T08:27:02.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:01 smithi150 ceph-mon[29276]: 6.14 scrub starts 2024-09-11T08:27:02.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:01 smithi150 ceph-mon[29276]: 6.14 scrub ok 2024-09-11T08:27:02.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:01 smithi150 ceph-mon[29276]: 3.1c scrub starts 2024-09-11T08:27:02.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:01 smithi150 ceph-mon[29276]: 3.1c scrub ok 2024-09-11T08:27:02.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:01 smithi150 ceph-mon[29276]: pgmap v889: 148 pgs: 148 active+clean; 2.5 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 143 MiB/s wr, 2.29k op/s 2024-09-11T08:27:02.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:01 smithi150 ceph-mon[39612]: 6.c scrub starts 2024-09-11T08:27:02.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:01 smithi150 ceph-mon[39612]: 6.c scrub ok 2024-09-11T08:27:02.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:01 smithi150 ceph-mon[39612]: 6.14 scrub starts 2024-09-11T08:27:02.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:01 smithi150 ceph-mon[39612]: 6.14 scrub ok 2024-09-11T08:27:02.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:01 smithi150 ceph-mon[39612]: 3.1c scrub starts 2024-09-11T08:27:02.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:01 smithi150 ceph-mon[39612]: 3.1c scrub ok 2024-09-11T08:27:02.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:01 smithi150 ceph-mon[39612]: pgmap v889: 148 pgs: 148 active+clean; 2.5 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 143 MiB/s wr, 2.29k op/s 2024-09-11T08:27:02.248 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:02.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:02.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:02.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:01 smithi181 ceph-mon[32701]: 6.c scrub starts 2024-09-11T08:27:02.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:01 smithi181 ceph-mon[32701]: 6.c scrub ok 2024-09-11T08:27:02.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:01 smithi181 ceph-mon[32701]: 6.14 scrub starts 2024-09-11T08:27:02.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:01 smithi181 ceph-mon[32701]: 6.14 scrub ok 2024-09-11T08:27:02.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:01 smithi181 ceph-mon[32701]: 3.1c scrub starts 2024-09-11T08:27:02.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:01 smithi181 ceph-mon[32701]: 3.1c scrub ok 2024-09-11T08:27:02.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:01 smithi181 ceph-mon[32701]: pgmap v889: 148 pgs: 148 active+clean; 2.5 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 143 MiB/s wr, 2.29k op/s 2024-09-11T08:27:02.350 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:02.390 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:02.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:02.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:02.492 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:02.531 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:02.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:02.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:02.614 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 22 15 48692 48677 138.267 95.75 0.0100844 0.00722809 2024-09-11T08:27:02.633 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:27:02.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:02.667 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:02.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:27:02.769 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:02.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:02.806 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:02.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:02.907 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:02.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:02.934 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:02.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:03.035 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:03.075 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:03.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:03.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:03.177 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:03.206 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:03.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:03.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:03.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:02 smithi150 ceph-mon[29276]: 4.18 scrub starts 2024-09-11T08:27:03.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:02 smithi150 ceph-mon[29276]: 4.18 scrub ok 2024-09-11T08:27:03.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:02 smithi150 ceph-mon[29276]: 3.d scrub starts 2024-09-11T08:27:03.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:02 smithi150 ceph-mon[29276]: 3.d scrub ok 2024-09-11T08:27:03.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:02 smithi150 ceph-mon[39612]: 4.18 scrub starts 2024-09-11T08:27:03.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:02 smithi150 ceph-mon[39612]: 4.18 scrub ok 2024-09-11T08:27:03.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:02 smithi150 ceph-mon[39612]: 3.d scrub starts 2024-09-11T08:27:03.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:02 smithi150 ceph-mon[39612]: 3.d scrub ok 2024-09-11T08:27:03.308 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:27:03.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:02 smithi181 ceph-mon[32701]: 4.18 scrub starts 2024-09-11T08:27:03.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:02 smithi181 ceph-mon[32701]: 4.18 scrub ok 2024-09-11T08:27:03.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:02 smithi181 ceph-mon[32701]: 3.d scrub starts 2024-09-11T08:27:03.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:02 smithi181 ceph-mon[32701]: 3.d scrub ok 2024-09-11T08:27:03.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:03.346 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:03.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:27:03.447 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:03.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:03.469 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:03.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:03.570 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:03.606 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:03.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:03.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:03.613 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 23 15 50176 50161 136.288 92.75 0.00621081 0.00733246 2024-09-11T08:27:03.708 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:03.744 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:03.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:03.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:03.846 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:03.885 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:03.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:03.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:03.987 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:04.017 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:04.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:04.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:04.119 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:27:04.146 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:04.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:04.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:27:04.211 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:27:04.232 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:27:04.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:03 smithi150 ceph-mon[29276]: pgmap v890: 148 pgs: 148 active+clean; 2.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-09-11T08:27:04.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:03 smithi150 ceph-mon[29276]: 3.c scrub starts 2024-09-11T08:27:04.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:03 smithi150 ceph-mon[29276]: 3.c scrub ok 2024-09-11T08:27:04.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:03 smithi150 ceph-mon[39612]: pgmap v890: 148 pgs: 148 active+clean; 2.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-09-11T08:27:04.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:03 smithi150 ceph-mon[39612]: 3.c scrub starts 2024-09-11T08:27:04.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:03 smithi150 ceph-mon[39612]: 3.c scrub ok 2024-09-11T08:27:04.247 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:04.310 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:04.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:04.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:04.322 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:03 smithi181 ceph-mon[32701]: pgmap v890: 148 pgs: 148 active+clean; 2.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-09-11T08:27:04.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:03 smithi181 ceph-mon[32701]: 3.c scrub starts 2024-09-11T08:27:04.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:03 smithi181 ceph-mon[32701]: 3.c scrub ok 2024-09-11T08:27:04.411 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:04.452 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:04.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:04.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:04.554 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:27:04.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:04.589 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:04.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:27:04.614 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 24 16 51807 51791 134.853 101.875 0.00336378 0.00740662 2024-09-11T08:27:04.689 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:27:04.725 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:04.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:04.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:27:04.826 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:04.856 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:04.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:04.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:04.959 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:04.992 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:04.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:04.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:05.094 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:27:05.132 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:05.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:05.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:27:05.142 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:04 smithi181 ceph-mon[32701]: 6.10 scrub starts 2024-09-11T08:27:05.143 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:04 smithi181 ceph-mon[32701]: 6.10 scrub ok 2024-09-11T08:27:05.143 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:04 smithi181 ceph-mon[32701]: 5.2 scrub starts 2024-09-11T08:27:05.143 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:04 smithi181 ceph-mon[32701]: 5.2 scrub ok 2024-09-11T08:27:05.143 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:04 smithi181 ceph-mon[32701]: 5.19 scrub starts 2024-09-11T08:27:05.143 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:04 smithi181 ceph-mon[32701]: 5.19 scrub ok 2024-09-11T08:27:05.173 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:04 smithi150 ceph-mon[29276]: 6.10 scrub starts 2024-09-11T08:27:05.173 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:04 smithi150 ceph-mon[29276]: 6.10 scrub ok 2024-09-11T08:27:05.174 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:04 smithi150 ceph-mon[29276]: 5.2 scrub starts 2024-09-11T08:27:05.174 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:04 smithi150 ceph-mon[29276]: 5.2 scrub ok 2024-09-11T08:27:05.174 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:04 smithi150 ceph-mon[29276]: 5.19 scrub starts 2024-09-11T08:27:05.174 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:04 smithi150 ceph-mon[29276]: 5.19 scrub ok 2024-09-11T08:27:05.174 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:04 smithi150 ceph-mon[39612]: 6.10 scrub starts 2024-09-11T08:27:05.174 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:04 smithi150 ceph-mon[39612]: 6.10 scrub ok 2024-09-11T08:27:05.174 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:04 smithi150 ceph-mon[39612]: 5.2 scrub starts 2024-09-11T08:27:05.174 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:04 smithi150 ceph-mon[39612]: 5.2 scrub ok 2024-09-11T08:27:05.174 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:04 smithi150 ceph-mon[39612]: 5.19 scrub starts 2024-09-11T08:27:05.174 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:04 smithi150 ceph-mon[39612]: 5.19 scrub ok 2024-09-11T08:27:05.234 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:05.269 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:05.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:05.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:05.371 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:27:05.403 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:05.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:05.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:27:05.504 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:05.545 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:05.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:05.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:05.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:27:05 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:27:05] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:27:05.614 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 25 16 53293 53277 133.174 92.875 0.00756417 0.00750373 2024-09-11T08:27:05.648 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:05.684 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:05.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:05.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:05.786 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:27:05.819 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:05.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:05.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:27:05.921 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:27:05.959 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:05.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:05.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:27:06.061 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:06.102 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:06.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:06.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:06.203 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:06.205 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:05 smithi150 ceph-mon[29276]: 6.11 scrub starts 2024-09-11T08:27:06.205 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:05 smithi150 ceph-mon[29276]: 6.11 scrub ok 2024-09-11T08:27:06.205 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:05 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:27:06.205 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:05 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:27:06.205 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:05 smithi150 ceph-mon[29276]: pgmap v891: 148 pgs: 148 active+clean; 3.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 117 MiB/s wr, 1.88k op/s 2024-09-11T08:27:06.205 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:05 smithi150 ceph-mon[29276]: osdmap e393: 8 total, 8 up, 8 in 2024-09-11T08:27:06.205 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:05 smithi150 ceph-mon[39612]: 6.11 scrub starts 2024-09-11T08:27:06.205 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:05 smithi150 ceph-mon[39612]: 6.11 scrub ok 2024-09-11T08:27:06.205 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:05 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:27:06.205 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:05 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:27:06.206 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:05 smithi150 ceph-mon[39612]: pgmap v891: 148 pgs: 148 active+clean; 3.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 117 MiB/s wr, 1.88k op/s 2024-09-11T08:27:06.206 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:05 smithi150 ceph-mon[39612]: osdmap e393: 8 total, 8 up, 8 in 2024-09-11T08:27:06.255 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:06.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:06.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:06.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:05 smithi181 ceph-mon[32701]: 6.11 scrub starts 2024-09-11T08:27:06.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:05 smithi181 ceph-mon[32701]: 6.11 scrub ok 2024-09-11T08:27:06.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:05 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:27:06.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:05 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:27:06.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:05 smithi181 ceph-mon[32701]: pgmap v891: 148 pgs: 148 active+clean; 3.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 117 MiB/s wr, 1.88k op/s 2024-09-11T08:27:06.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:05 smithi181 ceph-mon[32701]: osdmap e393: 8 total, 8 up, 8 in 2024-09-11T08:27:06.359 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:27:06.387 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:06.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:06.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:27:06.489 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:06.526 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:06.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:06.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:06.614 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 26 16 54808 54792 131.694 94.6875 0.0101148 0.00758841 2024-09-11T08:27:06.624 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:27:06.627 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-09-11T08:27:06.628 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:06.674 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:06.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:06.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:06.776 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:06.817 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:06.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:06.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:06.919 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:06.956 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:06.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:06.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:07.058 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:07.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:07.093 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:07.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:07.151 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:06 smithi150 ceph-mon[29276]: 6.d scrub starts 2024-09-11T08:27:07.151 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:06 smithi150 ceph-mon[29276]: 6.d scrub ok 2024-09-11T08:27:07.151 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:06 smithi150 ceph-mon[29276]: 5.8 scrub starts 2024-09-11T08:27:07.151 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:06 smithi150 ceph-mon[29276]: 5.8 scrub ok 2024-09-11T08:27:07.151 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:06 smithi150 ceph-mon[29276]: 3.1d scrub starts 2024-09-11T08:27:07.152 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:06 smithi150 ceph-mon[29276]: 3.1d scrub ok 2024-09-11T08:27:07.152 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:06 smithi150 ceph-mon[29276]: 5.1c scrub starts 2024-09-11T08:27:07.152 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:06 smithi150 ceph-mon[29276]: 5.1c scrub ok 2024-09-11T08:27:07.152 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:06 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:27:07.152 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:06 smithi150 ceph-mon[39612]: 6.d scrub starts 2024-09-11T08:27:07.152 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:06 smithi150 ceph-mon[39612]: 6.d scrub ok 2024-09-11T08:27:07.152 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:06 smithi150 ceph-mon[39612]: 5.8 scrub starts 2024-09-11T08:27:07.152 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:06 smithi150 ceph-mon[39612]: 5.8 scrub ok 2024-09-11T08:27:07.152 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:06 smithi150 ceph-mon[39612]: 3.1d scrub starts 2024-09-11T08:27:07.153 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:06 smithi150 ceph-mon[39612]: 3.1d scrub ok 2024-09-11T08:27:07.153 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:06 smithi150 ceph-mon[39612]: 5.1c scrub starts 2024-09-11T08:27:07.153 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:06 smithi150 ceph-mon[39612]: 5.1c scrub ok 2024-09-11T08:27:07.153 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:06 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:27:07.193 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:07.228 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:07.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:07.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:07.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:06 smithi181 ceph-mon[32701]: 6.d scrub starts 2024-09-11T08:27:07.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:06 smithi181 ceph-mon[32701]: 6.d scrub ok 2024-09-11T08:27:07.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:06 smithi181 ceph-mon[32701]: 5.8 scrub starts 2024-09-11T08:27:07.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:06 smithi181 ceph-mon[32701]: 5.8 scrub ok 2024-09-11T08:27:07.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:06 smithi181 ceph-mon[32701]: 3.1d scrub starts 2024-09-11T08:27:07.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:06 smithi181 ceph-mon[32701]: 3.1d scrub ok 2024-09-11T08:27:07.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:06 smithi181 ceph-mon[32701]: 5.1c scrub starts 2024-09-11T08:27:07.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:06 smithi181 ceph-mon[32701]: 5.1c scrub ok 2024-09-11T08:27:07.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:06 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:27:07.330 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:27:07.362 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:07.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:07.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:27:07.464 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:27:07.496 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:07.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:07.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:27:07.598 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:07.614 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 27 16 56359 56343 130.403 96.9375 0.00901559 0.00766336 2024-09-11T08:27:07.637 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:07.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:07.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:07.739 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:07.770 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:07.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:07.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:07.871 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:07.902 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:07.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:07.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:08.004 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:27:08.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:08.039 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:08.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:27:08.140 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:27:08.172 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:08.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:08.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:27:08.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:07 smithi150 ceph-mon[29276]: from='client.15915 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:27:08.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:07 smithi150 ceph-mon[29276]: 6.1f scrub starts 2024-09-11T08:27:08.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:07 smithi150 ceph-mon[29276]: 6.5 scrub starts 2024-09-11T08:27:08.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:07 smithi150 ceph-mon[29276]: 6.1f scrub ok 2024-09-11T08:27:08.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:07 smithi150 ceph-mon[29276]: 6.5 scrub ok 2024-09-11T08:27:08.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:07 smithi150 ceph-mon[29276]: 6.1a scrub starts 2024-09-11T08:27:08.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:07 smithi150 ceph-mon[29276]: 6.1a scrub ok 2024-09-11T08:27:08.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:07 smithi150 ceph-mon[29276]: pgmap v893: 148 pgs: 148 active+clean; 3.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-09-11T08:27:08.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:07 smithi150 ceph-mon[29276]: 4.1b scrub starts 2024-09-11T08:27:08.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:07 smithi150 ceph-mon[29276]: 4.1b scrub ok 2024-09-11T08:27:08.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:07 smithi150 ceph-mon[39612]: from='client.15915 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:27:08.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:07 smithi150 ceph-mon[39612]: 6.1f scrub starts 2024-09-11T08:27:08.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:07 smithi150 ceph-mon[39612]: 6.5 scrub starts 2024-09-11T08:27:08.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:07 smithi150 ceph-mon[39612]: 6.1f scrub ok 2024-09-11T08:27:08.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:07 smithi150 ceph-mon[39612]: 6.5 scrub ok 2024-09-11T08:27:08.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:07 smithi150 ceph-mon[39612]: 6.1a scrub starts 2024-09-11T08:27:08.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:07 smithi150 ceph-mon[39612]: 6.1a scrub ok 2024-09-11T08:27:08.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:07 smithi150 ceph-mon[39612]: pgmap v893: 148 pgs: 148 active+clean; 3.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-09-11T08:27:08.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:07 smithi150 ceph-mon[39612]: 4.1b scrub starts 2024-09-11T08:27:08.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:07 smithi150 ceph-mon[39612]: 4.1b scrub ok 2024-09-11T08:27:08.274 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:08.307 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:08.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:08.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:08.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:07 smithi181 ceph-mon[32701]: from='client.15915 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:27:08.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:07 smithi181 ceph-mon[32701]: 6.1f scrub starts 2024-09-11T08:27:08.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:07 smithi181 ceph-mon[32701]: 6.5 scrub starts 2024-09-11T08:27:08.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:07 smithi181 ceph-mon[32701]: 6.1f scrub ok 2024-09-11T08:27:08.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:07 smithi181 ceph-mon[32701]: 6.5 scrub ok 2024-09-11T08:27:08.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:07 smithi181 ceph-mon[32701]: 6.1a scrub starts 2024-09-11T08:27:08.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:07 smithi181 ceph-mon[32701]: 6.1a scrub ok 2024-09-11T08:27:08.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:07 smithi181 ceph-mon[32701]: pgmap v893: 148 pgs: 148 active+clean; 3.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-09-11T08:27:08.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:07 smithi181 ceph-mon[32701]: 4.1b scrub starts 2024-09-11T08:27:08.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:07 smithi181 ceph-mon[32701]: 4.1b scrub ok 2024-09-11T08:27:08.409 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:27:08.437 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:08.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:08.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:27:08.539 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:08.574 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:08.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:08.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:08.615 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 28 16 57910 57894 129.208 96.9375 0.00504071 0.00773103 2024-09-11T08:27:08.677 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:08.706 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:08.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:08.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:08.808 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:08.845 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:08.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:08.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:08.896 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:27:08 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:27:08] "GET /metrics HTTP/1.1" 200 224606 "" "Prometheus/2.43.0" 2024-09-11T08:27:08.947 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:08.979 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:08.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:08.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:09.081 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:09.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:09.118 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:09.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:09.219 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:27:09.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:08 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:27:09.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:08 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:27:09.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:08 smithi150 ceph-mon[29276]: 4.a scrub starts 2024-09-11T08:27:09.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:08 smithi150 ceph-mon[29276]: 4.a scrub ok 2024-09-11T08:27:09.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:08 smithi150 ceph-mon[29276]: osdmap e394: 8 total, 8 up, 8 in 2024-09-11T08:27:09.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:08 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:27:09.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:08 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:27:09.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:08 smithi150 ceph-mon[39612]: 4.a scrub starts 2024-09-11T08:27:09.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:08 smithi150 ceph-mon[39612]: 4.a scrub ok 2024-09-11T08:27:09.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:08 smithi150 ceph-mon[39612]: osdmap e394: 8 total, 8 up, 8 in 2024-09-11T08:27:09.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:09.252 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:09.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:27:09.323 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:27:08 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:27:09.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:08 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:27:09.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:08 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:27:09.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:08 smithi181 ceph-mon[32701]: 4.a scrub starts 2024-09-11T08:27:09.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:08 smithi181 ceph-mon[32701]: 4.a scrub ok 2024-09-11T08:27:09.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:08 smithi181 ceph-mon[32701]: osdmap e394: 8 total, 8 up, 8 in 2024-09-11T08:27:09.353 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:09.382 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:09.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:09.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:09.484 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:09.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:09.507 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:09.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:09.609 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:09.615 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 29 16 59146 59130 127.416 77.25 0.0137951 0.0078431 2024-09-11T08:27:09.645 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:09.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:09.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:09.746 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:09.779 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:09.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:09.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:09.881 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:09.923 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:09.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:09.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:10.025 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:10.047 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:10.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:10.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:10.149 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:10.177 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:10.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:10.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:10.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:09 smithi150 ceph-mon[29276]: 4.1c scrub starts 2024-09-11T08:27:10.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:09 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:27:10.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:09 smithi150 ceph-mon[29276]: 4.1c scrub ok 2024-09-11T08:27:10.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:09 smithi150 ceph-mon[29276]: pgmap v895: 148 pgs: 148 active+clean; 3.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 123 MiB/s wr, 1.97k op/s 2024-09-11T08:27:10.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:09 smithi150 ceph-mon[39612]: 4.1c scrub starts 2024-09-11T08:27:10.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:09 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:27:10.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:09 smithi150 ceph-mon[39612]: 4.1c scrub ok 2024-09-11T08:27:10.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:09 smithi150 ceph-mon[39612]: pgmap v895: 148 pgs: 148 active+clean; 3.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 123 MiB/s wr, 1.97k op/s 2024-09-11T08:27:10.278 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:10.313 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:10.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:10.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:10.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:09 smithi181 ceph-mon[32701]: 4.1c scrub starts 2024-09-11T08:27:10.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:09 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:27:10.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:09 smithi181 ceph-mon[32701]: 4.1c scrub ok 2024-09-11T08:27:10.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:09 smithi181 ceph-mon[32701]: pgmap v895: 148 pgs: 148 active+clean; 3.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 123 MiB/s wr, 1.97k op/s 2024-09-11T08:27:10.415 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:10.450 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:10.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:10.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:10.552 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:10.583 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:10.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:10.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:10.615 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 30 16 60274 60258 125.519 70.5 0.00799472 0.00796211 2024-09-11T08:27:10.686 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:10.727 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:10.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:10.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:10.828 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:10.868 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:10.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:10.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:10.895 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":892,"stamp":"2024-09-11T08:27:05.724820+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3275102428,"num_objects":50161,"num_object_clones":0,"num_object_copies":150483,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50161,"num_whiteouts":0,"num_read":1999,"num_read_kb":1858,"num_write":50794,"num_write_kb":3199431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":208,"num_bytes_recovered":1776519,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50934,"ondisk_log_size":50934,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":14805196,"kb_used_data":9367488,"kb_used_omap":22,"kb_used_meta":5437609,"kb_avail":735156020,"statfs":{"total":767960285184,"available":752799764480,"internally_reserved":0,"allocated":9592307712,"data_stored":9588768061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22853,"internal_metadata":5568112315},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1156644864,"num_objects":17649,"num_object_clones":0,"num_object_copies":52947,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17649,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":17649,"num_write_kb":1129536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.591390"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":448,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.733770+0000","last_change":"2024-09-11T08:24:28.359613+0000","last_active":"2024-09-11T08:26:55.733770+0000","last_peered":"2024-09-11T08:26:55.733770+0000","last_clean":"2024-09-11T08:26:55.733770+0000","last_became_active":"2024-09-11T08:24:28.358401+0000","last_became_peered":"2024-09-11T08:24:28.358401+0000","last_unstale":"2024-09-11T08:26:55.733770+0000","last_undegraded":"2024-09-11T08:26:55.733770+0000","last_fullsized":"2024-09-11T08:26:55.733770+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:23:29.976662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:23:29.976662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0090761950000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"312'19","reported_seq":452,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-11T08:27:02.000495+0000","last_change":"2024-09-11T08:27:02.000454+0000","last_active":"2024-09-11T08:27:02.000495+0000","last_peered":"2024-09-11T08:27:02.000495+0000","last_clean":"2024-09-11T08:27:02.000495+0000","last_became_active":"2024-09-11T08:07:32.048048+0000","last_became_peered":"2024-09-11T08:07:32.048048+0000","last_unstale":"2024-09-11T08:27:02.000495+0000","last_undegraded":"2024-09-11T08:27:02.000495+0000","last_fullsized":"2024-09-11T08:27:02.000495+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"312'19","last_scrub_stamp":"2024-09-11T08:27:02.000321+0000","last_deep_scrub":"312'19","last_deep_scrub_stamp":"2024-09-11T08:21:59.122122+0000","last_clean_scrub_stamp":"2024-09-11T08:27:02.000321+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:26.414397+0000","scrub_duration":0.0035901290000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":494,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.725876+0000","last_change":"2024-09-11T08:25:25.398253+0000","last_active":"2024-09-11T08:26:55.725876+0000","last_peered":"2024-09-11T08:26:55.725876+0000","last_clean":"2024-09-11T08:26:55.725876+0000","last_became_active":"2024-09-11T08:24:28.372160+0000","last_became_peered":"2024-09-11T08:24:28.372160+0000","last_unstale":"2024-09-11T08:26:55.725876+0000","last_undegraded":"2024-09-11T08:26:55.725876+0000","last_fullsized":"2024-09-11T08:26:55.725876+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:22:42.839072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:59.531202+0000","last_clean_scrub_stamp":"2024-09-11T08:22:42.839072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050530970000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":352,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.724706+0000","last_change":"2024-09-11T08:25:29.470636+0000","last_active":"2024-09-11T08:26:55.724706+0000","last_peered":"2024-09-11T08:26:55.724706+0000","last_clean":"2024-09-11T08:26:55.724706+0000","last_became_active":"2024-09-11T08:24:28.354834+0000","last_became_peered":"2024-09-11T08:24:28.354834+0000","last_unstale":"2024-09-11T08:26:55.724706+0000","last_undegraded":"2024-09-11T08:26:55.724706+0000","last_fullsized":"2024-09-11T08:26:55.724706+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:22:45.150733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:02.470649+0000","last_clean_scrub_stamp":"2024-09-11T08:22:45.150733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001328805,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":433,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.729941+0000","last_change":"2024-09-11T08:24:25.004241+0000","last_active":"2024-09-11T08:26:55.729941+0000","last_peered":"2024-09-11T08:26:55.729941+0000","last_clean":"2024-09-11T08:26:55.729941+0000","last_became_active":"2024-09-11T08:15:56.521104+0000","last_became_peered":"2024-09-11T08:15:56.521104+0000","last_unstale":"2024-09-11T08:26:55.729941+0000","last_undegraded":"2024-09-11T08:26:55.729941+0000","last_fullsized":"2024-09-11T08:26:55.729941+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:25.004147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:24:25.004147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004254351,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":216,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.727076+0000","last_change":"2024-09-11T08:24:16.568741+0000","last_active":"2024-09-11T08:26:55.727076+0000","last_peered":"2024-09-11T08:26:55.727076+0000","last_clean":"2024-09-11T08:26:55.727076+0000","last_became_active":"2024-09-11T08:17:37.550048+0000","last_became_peered":"2024-09-11T08:17:37.550048+0000","last_unstale":"2024-09-11T08:26:55.727076+0000","last_undegraded":"2024-09-11T08:26:55.727076+0000","last_fullsized":"2024-09-11T08:26:55.727076+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:16.568660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:24:16.568660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.006748937,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"239'34","reported_seq":376,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.728004+0000","last_change":"2024-09-11T08:24:16.013840+0000","last_active":"2024-09-11T08:26:55.728004+0000","last_peered":"2024-09-11T08:26:55.728004+0000","last_clean":"2024-09-11T08:26:55.728004+0000","last_became_active":"2024-09-11T08:17:37.553866+0000","last_became_peered":"2024-09-11T08:17:37.553866+0000","last_unstale":"2024-09-11T08:26:55.728004+0000","last_undegraded":"2024-09-11T08:26:55.728004+0000","last_fullsized":"2024-09-11T08:26:55.728004+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'34","last_scrub_stamp":"2024-09-11T08:24:16.013714+0000","last_deep_scrub":"145'22","last_deep_scrub_stamp":"2024-09-11T08:15:57.478870+0000","last_clean_scrub_stamp":"2024-09-11T08:24:16.013714+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0074771159999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":77,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.726873+0000","last_change":"2024-09-11T08:24:08.583672+0000","last_active":"2024-09-11T08:26:55.726873+0000","last_peered":"2024-09-11T08:26:55.726873+0000","last_clean":"2024-09-11T08:26:55.726873+0000","last_became_active":"2024-09-11T08:23:22.687041+0000","last_became_peered":"2024-09-11T08:23:22.687041+0000","last_unstale":"2024-09-11T08:26:55.726873+0000","last_undegraded":"2024-09-11T08:26:55.726873+0000","last_fullsized":"2024-09-11T08:26:55.726873+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:08.583528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:18:36.998503+0000","last_clean_scrub_stamp":"2024-09-11T08:24:08.583528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002003758,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":108,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.727829+0000","last_change":"2024-09-11T08:24:28.360364+0000","last_active":"2024-09-11T08:26:55.727829+0000","last_peered":"2024-09-11T08:26:55.727829+0000","last_clean":"2024-09-11T08:26:55.727829+0000","last_became_active":"2024-09-11T08:24:28.360021+0000","last_became_peered":"2024-09-11T08:24:28.360021+0000","last_unstale":"2024-09-11T08:26:55.727829+0000","last_undegraded":"2024-09-11T08:26:55.727829+0000","last_fullsized":"2024-09-11T08:26:55.727829+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:22:41.311684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:54.878409+0000","last_clean_scrub_stamp":"2024-09-11T08:22:41.311684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"239'15","reported_seq":327,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.733746+0000","last_change":"2024-09-11T08:26:02.101012+0000","last_active":"2024-09-11T08:26:55.733746+0000","last_peered":"2024-09-11T08:26:55.733746+0000","last_clean":"2024-09-11T08:26:55.733746+0000","last_became_active":"2024-09-11T08:23:22.687182+0000","last_became_peered":"2024-09-11T08:23:22.687182+0000","last_unstale":"2024-09-11T08:26:55.733746+0000","last_undegraded":"2024-09-11T08:26:55.733746+0000","last_fullsized":"2024-09-11T08:26:55.733746+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'15","last_scrub_stamp":"2024-09-11T08:26:02.100940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:26:02.100940+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:30.876179+0000","scrub_duration":0.0028851139999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":333,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.733915+0000","last_change":"2024-09-11T08:26:04.073177+0000","last_active":"2024-09-11T08:26:55.733915+0000","last_peered":"2024-09-11T08:26:55.733915+0000","last_clean":"2024-09-11T08:26:55.733915+0000","last_became_active":"2024-09-11T08:24:28.363985+0000","last_became_peered":"2024-09-11T08:24:28.363985+0000","last_unstale":"2024-09-11T08:26:55.733915+0000","last_undegraded":"2024-09-11T08:26:55.733915+0000","last_fullsized":"2024-09-11T08:26:55.733915+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:04.073122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:20:18.900482+0000","last_clean_scrub_stamp":"2024-09-11T08:26:04.073122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:07.560997+0000","scrub_duration":0.00163221,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":422,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.732862+0000","last_change":"2024-09-11T08:24:26.079306+0000","last_active":"2024-09-11T08:26:55.732862+0000","last_peered":"2024-09-11T08:26:55.732862+0000","last_clean":"2024-09-11T08:26:55.732862+0000","last_became_active":"2024-09-11T08:14:24.792595+0000","last_became_peered":"2024-09-11T08:14:24.792595+0000","last_unstale":"2024-09-11T08:26:55.732862+0000","last_undegraded":"2024-09-11T08:26:55.732862+0000","last_fullsized":"2024-09-11T08:26:55.732862+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:26.079152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:24:26.079152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037087930000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":120,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-11T08:27:00.877313+0000","last_change":"2024-09-11T08:27:00.877188+0000","last_active":"2024-09-11T08:27:00.877313+0000","last_peered":"2024-09-11T08:27:00.877313+0000","last_clean":"2024-09-11T08:27:00.877313+0000","last_became_active":"2024-09-11T08:24:28.359153+0000","last_became_peered":"2024-09-11T08:24:28.359153+0000","last_unstale":"2024-09-11T08:27:00.877313+0000","last_undegraded":"2024-09-11T08:27:00.877313+0000","last_fullsized":"2024-09-11T08:27:00.877313+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:00.877062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:27:00.877062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:06.321049+0000","scrub_duration":0.0048170670000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"239'9","reported_seq":307,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.730334+0000","last_change":"2024-09-11T08:23:37.161973+0000","last_active":"2024-09-11T08:26:55.730334+0000","last_peered":"2024-09-11T08:26:55.730334+0000","last_clean":"2024-09-11T08:26:55.730334+0000","last_became_active":"2024-09-11T08:23:22.681876+0000","last_became_peered":"2024-09-11T08:23:22.681876+0000","last_unstale":"2024-09-11T08:26:55.730334+0000","last_undegraded":"2024-09-11T08:26:55.730334+0000","last_fullsized":"2024-09-11T08:26:55.730334+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'9","last_scrub_stamp":"2024-09-11T08:23:37.161836+0000","last_deep_scrub":"139'7","last_deep_scrub_stamp":"2024-09-11T08:12:24.869318+0000","last_clean_scrub_stamp":"2024-09-11T08:23:37.161836+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0074395970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":234,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.723012+0000","last_change":"2024-09-11T08:23:33.277787+0000","last_active":"2024-09-11T08:26:55.723012+0000","last_peered":"2024-09-11T08:26:55.723012+0000","last_clean":"2024-09-11T08:26:55.723012+0000","last_became_active":"2024-09-11T08:15:56.526954+0000","last_became_peered":"2024-09-11T08:15:56.526954+0000","last_unstale":"2024-09-11T08:26:55.723012+0000","last_undegraded":"2024-09-11T08:26:55.723012+0000","last_fullsized":"2024-09-11T08:26:55.723012+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:23:33.277734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:23:33.277734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.023249810999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":114,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.724092+0000","last_change":"2024-09-11T08:25:54.524776+0000","last_active":"2024-09-11T08:26:55.724092+0000","last_peered":"2024-09-11T08:26:55.724092+0000","last_clean":"2024-09-11T08:26:55.724092+0000","last_became_active":"2024-09-11T08:24:28.372097+0000","last_became_peered":"2024-09-11T08:24:28.372097+0000","last_unstale":"2024-09-11T08:26:55.724092+0000","last_undegraded":"2024-09-11T08:26:55.724092+0000","last_fullsized":"2024-09-11T08:26:55.724092+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:22:54.272360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:20:19.927616+0000","last_clean_scrub_stamp":"2024-09-11T08:22:54.272360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":360,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.726812+0000","last_change":"2024-09-11T08:24:29.597394+0000","last_active":"2024-09-11T08:26:55.726812+0000","last_peered":"2024-09-11T08:26:55.726812+0000","last_clean":"2024-09-11T08:26:55.726812+0000","last_became_active":"2024-09-11T08:24:28.368560+0000","last_became_peered":"2024-09-11T08:24:28.368560+0000","last_unstale":"2024-09-11T08:26:55.726812+0000","last_undegraded":"2024-09-11T08:26:55.726812+0000","last_fullsized":"2024-09-11T08:26:55.726812+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:29.597313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:24:29.597313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024078900000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":441,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.734353+0000","last_change":"2024-09-11T08:26:06.019004+0000","last_active":"2024-09-11T08:26:55.734353+0000","last_peered":"2024-09-11T08:26:55.734353+0000","last_clean":"2024-09-11T08:26:55.734353+0000","last_became_active":"2024-09-11T08:15:56.516941+0000","last_became_peered":"2024-09-11T08:15:56.516941+0000","last_unstale":"2024-09-11T08:26:55.734353+0000","last_undegraded":"2024-09-11T08:26:55.734353+0000","last_fullsized":"2024-09-11T08:26:55.734353+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:06.018947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:26:06.018947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:29.870750+0000","scrub_duration":0.001287098,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"239'34","reported_seq":479,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.730347+0000","last_change":"2024-09-11T08:24:10.015624+0000","last_active":"2024-09-11T08:26:55.730347+0000","last_peered":"2024-09-11T08:26:55.730347+0000","last_clean":"2024-09-11T08:26:55.730347+0000","last_became_active":"2024-09-11T08:07:32.046363+0000","last_became_peered":"2024-09-11T08:07:32.046363+0000","last_unstale":"2024-09-11T08:26:55.730347+0000","last_undegraded":"2024-09-11T08:26:55.730347+0000","last_fullsized":"2024-09-11T08:26:55.730347+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'34","last_scrub_stamp":"2024-09-11T08:24:10.015536+0000","last_deep_scrub":"107'22","last_deep_scrub_stamp":"2024-09-11T08:16:04.159211+0000","last_clean_scrub_stamp":"2024-09-11T08:24:10.015536+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0067714109999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":437,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.725795+0000","last_change":"2024-09-11T08:25:57.566953+0000","last_active":"2024-09-11T08:26:55.725795+0000","last_peered":"2024-09-11T08:26:55.725795+0000","last_clean":"2024-09-11T08:26:55.725795+0000","last_became_active":"2024-09-11T08:07:34.055410+0000","last_became_peered":"2024-09-11T08:07:34.055410+0000","last_unstale":"2024-09-11T08:26:55.725795+0000","last_undegraded":"2024-09-11T08:26:55.725795+0000","last_fullsized":"2024-09-11T08:26:55.725795+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:25:57.566898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:19:28.870211+0000","last_clean_scrub_stamp":"2024-09-11T08:25:57.566898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001280427,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":243,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.726755+0000","last_change":"2024-09-11T08:24:28.359151+0000","last_active":"2024-09-11T08:26:55.726755+0000","last_peered":"2024-09-11T08:26:55.726755+0000","last_clean":"2024-09-11T08:26:55.726755+0000","last_became_active":"2024-09-11T08:24:28.357957+0000","last_became_peered":"2024-09-11T08:24:28.357957+0000","last_unstale":"2024-09-11T08:26:55.726755+0000","last_undegraded":"2024-09-11T08:26:55.726755+0000","last_fullsized":"2024-09-11T08:26:55.726755+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:22:53.184916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:22:53.184916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019406600000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":86,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.725338+0000","last_change":"2024-09-11T08:26:04.502886+0000","last_active":"2024-09-11T08:26:55.725338+0000","last_peered":"2024-09-11T08:26:55.725338+0000","last_clean":"2024-09-11T08:26:55.725338+0000","last_became_active":"2024-09-11T08:23:22.686935+0000","last_became_peered":"2024-09-11T08:23:22.686935+0000","last_unstale":"2024-09-11T08:26:55.725338+0000","last_undegraded":"2024-09-11T08:26:55.725338+0000","last_fullsized":"2024-09-11T08:26:55.725338+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:04.502835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:26:04.502835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:07.064477+0000","scrub_duration":0.001566196,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"239'18","reported_seq":286,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:57.767410+0000","last_change":"2024-09-11T08:26:57.767336+0000","last_active":"2024-09-11T08:26:57.767410+0000","last_peered":"2024-09-11T08:26:57.767410+0000","last_clean":"2024-09-11T08:26:57.767410+0000","last_became_active":"2024-09-11T08:17:37.553402+0000","last_became_peered":"2024-09-11T08:17:37.553402+0000","last_unstale":"2024-09-11T08:26:57.767410+0000","last_undegraded":"2024-09-11T08:26:57.767410+0000","last_fullsized":"2024-09-11T08:26:57.767410+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'18","last_scrub_stamp":"2024-09-11T08:26:57.767252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:26:57.767252+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:25.867282+0000","scrub_duration":0.0051635600000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":77,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.729684+0000","last_change":"2024-09-11T08:23:35.151633+0000","last_active":"2024-09-11T08:26:55.729684+0000","last_peered":"2024-09-11T08:26:55.729684+0000","last_clean":"2024-09-11T08:26:55.729684+0000","last_became_active":"2024-09-11T08:23:22.684583+0000","last_became_peered":"2024-09-11T08:23:22.684583+0000","last_unstale":"2024-09-11T08:26:55.729684+0000","last_undegraded":"2024-09-11T08:26:55.729684+0000","last_fullsized":"2024-09-11T08:26:55.729684+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:23:35.151580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:10:40.110490+0000","last_clean_scrub_stamp":"2024-09-11T08:23:35.151580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013829059999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":409,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.724296+0000","last_change":"2024-09-11T08:25:56.984646+0000","last_active":"2024-09-11T08:26:55.724296+0000","last_peered":"2024-09-11T08:26:55.724296+0000","last_clean":"2024-09-11T08:26:55.724296+0000","last_became_active":"2024-09-11T08:24:28.357198+0000","last_became_peered":"2024-09-11T08:24:28.357198+0000","last_unstale":"2024-09-11T08:26:55.724296+0000","last_undegraded":"2024-09-11T08:26:55.724296+0000","last_fullsized":"2024-09-11T08:26:55.724296+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-11T08:22:51.567081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:22:51.567081+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041879530000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":431,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.726003+0000","last_change":"2024-09-11T08:24:17.569443+0000","last_active":"2024-09-11T08:26:55.726003+0000","last_peered":"2024-09-11T08:26:55.726003+0000","last_clean":"2024-09-11T08:26:55.726003+0000","last_became_active":"2024-09-11T08:07:30.045932+0000","last_became_peered":"2024-09-11T08:07:30.045932+0000","last_unstale":"2024-09-11T08:26:55.726003+0000","last_undegraded":"2024-09-11T08:26:55.726003+0000","last_fullsized":"2024-09-11T08:26:55.726003+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.569369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.569369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028606489999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"350'25","reported_seq":517,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.734439+0000","last_change":"2024-09-11T08:24:23.093331+0000","last_active":"2024-09-11T08:26:55.734439+0000","last_peered":"2024-09-11T08:26:55.734439+0000","last_clean":"2024-09-11T08:26:55.734439+0000","last_became_active":"2024-09-11T08:15:56.515817+0000","last_became_peered":"2024-09-11T08:15:56.515817+0000","last_unstale":"2024-09-11T08:26:55.734439+0000","last_undegraded":"2024-09-11T08:26:55.734439+0000","last_fullsized":"2024-09-11T08:26:55.734439+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'25","last_scrub_stamp":"2024-09-11T08:24:23.093269+0000","last_deep_scrub":"350'25","last_deep_scrub_stamp":"2024-09-11T08:24:23.093269+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.093269+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015481372,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"254'12","reported_seq":719,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:58.788315+0000","last_change":"2024-09-11T08:23:33.540404+0000","last_active":"2024-09-11T08:26:58.788315+0000","last_peered":"2024-09-11T08:26:58.788315+0000","last_clean":"2024-09-11T08:26:58.788315+0000","last_became_active":"2024-09-11T08:15:56.525498+0000","last_became_peered":"2024-09-11T08:15:56.525498+0000","last_unstale":"2024-09-11T08:26:58.788315+0000","last_undegraded":"2024-09-11T08:26:58.788315+0000","last_fullsized":"2024-09-11T08:26:58.788315+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'12","last_scrub_stamp":"2024-09-11T08:23:33.540345+0000","last_deep_scrub":"254'12","last_deep_scrub_stamp":"2024-09-11T08:18:34.253865+0000","last_clean_scrub_stamp":"2024-09-11T08:23:33.540345+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049407269999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.0","version":"392'3061","reported_seq":3079,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-11T08:27:03.759367+0000","last_change":"2024-09-11T08:26:27.764038+0000","last_active":"2024-09-11T08:27:03.759367+0000","last_peered":"2024-09-11T08:27:03.759367+0000","last_clean":"2024-09-11T08:27:03.759367+0000","last_became_active":"2024-09-11T08:26:27.763713+0000","last_became_peered":"2024-09-11T08:26:27.763713+0000","last_unstale":"2024-09-11T08:27:03.759367+0000","last_undegraded":"2024-09-11T08:27:03.759367+0000","last_fullsized":"2024-09-11T08:27:03.759367+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:26:26.740307+0000","objects_scrubbed":0,"log_size":3061,"log_dups_size":0,"ondisk_log_size":3061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:43.830110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200605696,"num_objects":3061,"num_object_clones":0,"num_object_copies":9183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3061,"num_write_kb":195904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"239'27","reported_seq":482,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.728255+0000","last_change":"2024-09-11T08:24:11.562773+0000","last_active":"2024-09-11T08:26:55.728255+0000","last_peered":"2024-09-11T08:26:55.728255+0000","last_clean":"2024-09-11T08:26:55.728255+0000","last_became_active":"2024-09-11T08:07:32.054317+0000","last_became_peered":"2024-09-11T08:07:32.054317+0000","last_unstale":"2024-09-11T08:26:55.728255+0000","last_undegraded":"2024-09-11T08:26:55.728255+0000","last_fullsized":"2024-09-11T08:26:55.728255+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'27","last_scrub_stamp":"2024-09-11T08:24:11.562679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:24:11.562679+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0092178139999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":442,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.732538+0000","last_change":"2024-09-11T08:23:33.180926+0000","last_active":"2024-09-11T08:26:55.732538+0000","last_peered":"2024-09-11T08:26:55.732538+0000","last_clean":"2024-09-11T08:26:55.732538+0000","last_became_active":"2024-09-11T08:15:56.521943+0000","last_became_peered":"2024-09-11T08:15:56.521943+0000","last_unstale":"2024-09-11T08:26:55.732538+0000","last_undegraded":"2024-09-11T08:26:55.732538+0000","last_fullsized":"2024-09-11T08:26:55.732538+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:23:33.180843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:22:00.707990+0000","last_clean_scrub_stamp":"2024-09-11T08:23:33.180843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015073790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"354'13","reported_seq":160,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-11T08:27:00.736968+0000","last_change":"2024-09-11T08:26:00.096276+0000","last_active":"2024-09-11T08:27:00.736968+0000","last_peered":"2024-09-11T08:27:00.736968+0000","last_clean":"2024-09-11T08:27:00.736968+0000","last_became_active":"2024-09-11T08:24:28.368212+0000","last_became_peered":"2024-09-11T08:24:28.368212+0000","last_unstale":"2024-09-11T08:27:00.736968+0000","last_undegraded":"2024-09-11T08:27:00.736968+0000","last_fullsized":"2024-09-11T08:27:00.736968+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'13","last_scrub_stamp":"2024-09-11T08:26:00.096215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:26:00.096215+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:04.199732+0000","scrub_duration":0.0016774890000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":114,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.725717+0000","last_change":"2024-09-11T08:25:28.469235+0000","last_active":"2024-09-11T08:26:55.725717+0000","last_peered":"2024-09-11T08:26:55.725717+0000","last_clean":"2024-09-11T08:26:55.725717+0000","last_became_active":"2024-09-11T08:24:28.353556+0000","last_became_peered":"2024-09-11T08:24:28.353556+0000","last_unstale":"2024-09-11T08:26:55.725717+0000","last_undegraded":"2024-09-11T08:26:55.725717+0000","last_fullsized":"2024-09-11T08:26:55.725717+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:22:46.326167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:06.616332+0000","last_clean_scrub_stamp":"2024-09-11T08:22:46.326167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.f","version":"392'3095","reported_seq":3113,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-11T08:27:03.204823+0000","last_change":"2024-09-11T08:26:27.766747+0000","last_active":"2024-09-11T08:27:03.204823+0000","last_peered":"2024-09-11T08:27:03.204823+0000","last_clean":"2024-09-11T08:27:03.204823+0000","last_became_active":"2024-09-11T08:26:27.766540+0000","last_became_peered":"2024-09-11T08:26:27.766540+0000","last_unstale":"2024-09-11T08:27:03.204823+0000","last_undegraded":"2024-09-11T08:27:03.204823+0000","last_fullsized":"2024-09-11T08:27:03.204823+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:26:26.740307+0000","objects_scrubbed":0,"log_size":3095,"log_dups_size":0,"ondisk_log_size":3095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:39.323593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202833920,"num_objects":3095,"num_object_clones":0,"num_object_copies":9285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3095,"num_write_kb":198080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"239'30","reported_seq":461,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.723183+0000","last_change":"2024-09-11T08:25:50.930305+0000","last_active":"2024-09-11T08:26:55.723183+0000","last_peered":"2024-09-11T08:26:55.723183+0000","last_clean":"2024-09-11T08:26:55.723183+0000","last_became_active":"2024-09-11T08:17:37.555703+0000","last_became_peered":"2024-09-11T08:17:37.555703+0000","last_unstale":"2024-09-11T08:26:55.723183+0000","last_undegraded":"2024-09-11T08:26:55.723183+0000","last_fullsized":"2024-09-11T08:26:55.723183+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'30","last_scrub_stamp":"2024-09-11T08:22:46.618318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:22:46.618318+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052837379999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":402,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.723953+0000","last_change":"2024-09-11T08:26:08.112087+0000","last_active":"2024-09-11T08:26:55.723953+0000","last_peered":"2024-09-11T08:26:55.723953+0000","last_clean":"2024-09-11T08:26:55.723953+0000","last_became_active":"2024-09-11T08:17:37.553579+0000","last_became_peered":"2024-09-11T08:17:37.553579+0000","last_unstale":"2024-09-11T08:26:55.723953+0000","last_undegraded":"2024-09-11T08:26:55.723953+0000","last_fullsized":"2024-09-11T08:26:55.723953+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:08.112023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:26:08.112023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:30.647649+0000","scrub_duration":0.001608578,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":441,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.719852+0000","last_change":"2024-09-11T08:26:01.994013+0000","last_active":"2024-09-11T08:26:55.719852+0000","last_peered":"2024-09-11T08:26:55.719852+0000","last_clean":"2024-09-11T08:26:55.719852+0000","last_became_active":"2024-09-11T08:15:56.522606+0000","last_became_peered":"2024-09-11T08:15:56.522606+0000","last_unstale":"2024-09-11T08:26:55.719852+0000","last_undegraded":"2024-09-11T08:26:55.719852+0000","last_fullsized":"2024-09-11T08:26:55.719852+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:01.993924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:26:01.993924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:24.892591+0000","scrub_duration":0.0012801519999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":77,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.734240+0000","last_change":"2024-09-11T08:26:54.574588+0000","last_active":"2024-09-11T08:26:55.734240+0000","last_peered":"2024-09-11T08:26:55.734240+0000","last_clean":"2024-09-11T08:26:55.734240+0000","last_became_active":"2024-09-11T08:23:22.683474+0000","last_became_peered":"2024-09-11T08:23:22.683474+0000","last_unstale":"2024-09-11T08:26:55.734240+0000","last_undegraded":"2024-09-11T08:26:55.734240+0000","last_fullsized":"2024-09-11T08:26:55.734240+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:54.574506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:22:50.151585+0000","last_clean_scrub_stamp":"2024-09-11T08:26:54.574506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:21.672267+0000","scrub_duration":0.012208658000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"392'3148","reported_seq":3166,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-11T08:27:02.747632+0000","last_change":"2024-09-11T08:26:27.765017+0000","last_active":"2024-09-11T08:27:02.747632+0000","last_peered":"2024-09-11T08:27:02.747632+0000","last_clean":"2024-09-11T08:27:02.747632+0000","last_became_active":"2024-09-11T08:26:27.764671+0000","last_became_peered":"2024-09-11T08:26:27.764671+0000","last_unstale":"2024-09-11T08:27:02.747632+0000","last_undegraded":"2024-09-11T08:27:02.747632+0000","last_fullsized":"2024-09-11T08:27:02.747632+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:26:26.740307+0000","objects_scrubbed":0,"log_size":3148,"log_dups_size":0,"ondisk_log_size":3148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:47.926361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206307328,"num_objects":3148,"num_object_clones":0,"num_object_copies":9444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3148,"num_write_kb":201472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"239'28","reported_seq":351,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.728111+0000","last_change":"2024-09-11T08:24:28.358516+0000","last_active":"2024-09-11T08:26:55.728111+0000","last_peered":"2024-09-11T08:26:55.728111+0000","last_clean":"2024-09-11T08:26:55.728111+0000","last_became_active":"2024-09-11T08:24:28.358175+0000","last_became_peered":"2024-09-11T08:24:28.358175+0000","last_unstale":"2024-09-11T08:26:55.728111+0000","last_undegraded":"2024-09-11T08:26:55.728111+0000","last_fullsized":"2024-09-11T08:26:55.728111+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'28","last_scrub_stamp":"2024-09-11T08:23:31.557041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:23:31.557041+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022343200000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":421,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.734708+0000","last_change":"2024-09-11T08:25:51.199008+0000","last_active":"2024-09-11T08:26:55.734708+0000","last_peered":"2024-09-11T08:26:55.734708+0000","last_clean":"2024-09-11T08:26:55.734708+0000","last_became_active":"2024-09-11T08:07:30.050484+0000","last_became_peered":"2024-09-11T08:07:30.050484+0000","last_unstale":"2024-09-11T08:26:55.734708+0000","last_undegraded":"2024-09-11T08:26:55.734708+0000","last_fullsized":"2024-09-11T08:26:55.734708+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:25:51.198956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:47.960329+0000","last_clean_scrub_stamp":"2024-09-11T08:25:51.198956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001153823,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"82'9","reported_seq":674,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:58.788515+0000","last_change":"2024-09-11T08:24:14.087666+0000","last_active":"2024-09-11T08:26:58.788515+0000","last_peered":"2024-09-11T08:26:58.788515+0000","last_clean":"2024-09-11T08:26:58.788515+0000","last_became_active":"2024-09-11T08:07:34.056143+0000","last_became_peered":"2024-09-11T08:07:34.056143+0000","last_unstale":"2024-09-11T08:26:58.788515+0000","last_undegraded":"2024-09-11T08:26:58.788515+0000","last_fullsized":"2024-09-11T08:26:58.788515+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-11T08:24:14.087574+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-09-11T08:21:00.841426+0000","last_clean_scrub_stamp":"2024-09-11T08:24:14.087574+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043101909999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":396,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.723823+0000","last_change":"2024-09-11T08:24:24.021006+0000","last_active":"2024-09-11T08:26:55.723823+0000","last_peered":"2024-09-11T08:26:55.723823+0000","last_clean":"2024-09-11T08:26:55.723823+0000","last_became_active":"2024-09-11T08:14:24.788968+0000","last_became_peered":"2024-09-11T08:14:24.788968+0000","last_unstale":"2024-09-11T08:26:55.723823+0000","last_undegraded":"2024-09-11T08:26:55.723823+0000","last_fullsized":"2024-09-11T08:26:55.723823+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:24.020925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:24:24.020925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010911288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.d","version":"392'3089","reported_seq":3107,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-11T08:27:03.501850+0000","last_change":"2024-09-11T08:26:27.763488+0000","last_active":"2024-09-11T08:27:03.501850+0000","last_peered":"2024-09-11T08:27:03.501850+0000","last_clean":"2024-09-11T08:27:03.501850+0000","last_became_active":"2024-09-11T08:26:27.763225+0000","last_became_peered":"2024-09-11T08:26:27.763225+0000","last_unstale":"2024-09-11T08:27:03.501850+0000","last_undegraded":"2024-09-11T08:27:03.501850+0000","last_fullsized":"2024-09-11T08:27:03.501850+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:26:26.740307+0000","objects_scrubbed":0,"log_size":3089,"log_dups_size":0,"ondisk_log_size":3089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:56.503503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202440704,"num_objects":3089,"num_object_clones":0,"num_object_copies":9267,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3089,"num_write_kb":197696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"239'20","reported_seq":519,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:57.866510+0000","last_change":"2024-09-11T08:26:57.866443+0000","last_active":"2024-09-11T08:26:57.866510+0000","last_peered":"2024-09-11T08:26:57.866510+0000","last_clean":"2024-09-11T08:26:57.866510+0000","last_became_active":"2024-09-11T08:24:28.355516+0000","last_became_peered":"2024-09-11T08:24:28.355516+0000","last_unstale":"2024-09-11T08:26:57.866510+0000","last_undegraded":"2024-09-11T08:26:57.866510+0000","last_fullsized":"2024-09-11T08:26:57.866510+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'20","last_scrub_stamp":"2024-09-11T08:26:57.866352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:26:57.866352+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:03.651194+0000","scrub_duration":0.0030126530000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":132,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.730843+0000","last_change":"2024-09-11T08:26:02.614677+0000","last_active":"2024-09-11T08:26:55.730843+0000","last_peered":"2024-09-11T08:26:55.730843+0000","last_clean":"2024-09-11T08:26:55.730843+0000","last_became_active":"2024-09-11T08:24:28.352741+0000","last_became_peered":"2024-09-11T08:24:28.352741+0000","last_unstale":"2024-09-11T08:26:55.730843+0000","last_undegraded":"2024-09-11T08:26:55.730843+0000","last_fullsized":"2024-09-11T08:26:55.730843+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:02.614567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:26:02.614567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:10.560045+0000","scrub_duration":0.0018716119999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"376'15","reported_seq":359,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:58.788348+0000","last_change":"2024-09-11T08:26:02.786822+0000","last_active":"2024-09-11T08:26:58.788348+0000","last_peered":"2024-09-11T08:26:58.788348+0000","last_clean":"2024-09-11T08:26:58.788348+0000","last_became_active":"2024-09-11T08:24:28.359787+0000","last_became_peered":"2024-09-11T08:24:28.359787+0000","last_unstale":"2024-09-11T08:26:58.788348+0000","last_undegraded":"2024-09-11T08:26:58.788348+0000","last_fullsized":"2024-09-11T08:26:58.788348+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'15","last_scrub_stamp":"2024-09-11T08:26:02.786767+0000","last_deep_scrub":"376'15","last_deep_scrub_stamp":"2024-09-11T08:26:02.786767+0000","last_clean_scrub_stamp":"2024-09-11T08:26:02.786767+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:12.732955+0000","scrub_duration":0.0023292619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"66'1","reported_seq":399,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.732995+0000","last_change":"2024-09-11T08:25:54.225938+0000","last_active":"2024-09-11T08:26:55.732995+0000","last_peered":"2024-09-11T08:26:55.732995+0000","last_clean":"2024-09-11T08:26:55.732995+0000","last_became_active":"2024-09-11T08:07:36.072593+0000","last_became_peered":"2024-09-11T08:07:36.072593+0000","last_unstale":"2024-09-11T08:26:55.732995+0000","last_undegraded":"2024-09-11T08:26:55.732995+0000","last_fullsized":"2024-09-11T08:26:55.732995+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-11T08:22:04.543159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:22:04.543159+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002903625,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"392'3124","reported_seq":3142,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-11T08:27:03.773513+0000","last_change":"2024-09-11T08:26:27.765203+0000","last_active":"2024-09-11T08:27:03.773513+0000","last_peered":"2024-09-11T08:27:03.773513+0000","last_clean":"2024-09-11T08:27:03.773513+0000","last_became_active":"2024-09-11T08:26:27.764927+0000","last_became_peered":"2024-09-11T08:26:27.764927+0000","last_unstale":"2024-09-11T08:27:03.773513+0000","last_undegraded":"2024-09-11T08:27:03.773513+0000","last_fullsized":"2024-09-11T08:27:03.773513+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:26:26.740307+0000","objects_scrubbed":0,"log_size":3124,"log_dups_size":0,"ondisk_log_size":3124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:29.693074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204734464,"num_objects":3124,"num_object_clones":0,"num_object_copies":9372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3124,"num_write_kb":199936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"239'29","reported_seq":589,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.734051+0000","last_change":"2024-09-11T08:25:48.174622+0000","last_active":"2024-09-11T08:26:55.734051+0000","last_peered":"2024-09-11T08:26:55.734051+0000","last_clean":"2024-09-11T08:26:55.734051+0000","last_became_active":"2024-09-11T08:24:28.364043+0000","last_became_peered":"2024-09-11T08:24:28.364043+0000","last_unstale":"2024-09-11T08:26:55.734051+0000","last_undegraded":"2024-09-11T08:26:55.734051+0000","last_fullsized":"2024-09-11T08:26:55.734051+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'29","last_scrub_stamp":"2024-09-11T08:21:44.561906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:21:44.561906+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024798810000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":477,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.920646+0000","last_change":"2024-09-11T08:26:55.920593+0000","last_active":"2024-09-11T08:26:55.920646+0000","last_peered":"2024-09-11T08:26:55.920646+0000","last_clean":"2024-09-11T08:26:55.920646+0000","last_became_active":"2024-09-11T08:24:28.358796+0000","last_became_peered":"2024-09-11T08:24:28.358796+0000","last_unstale":"2024-09-11T08:26:55.920646+0000","last_undegraded":"2024-09-11T08:26:55.920646+0000","last_fullsized":"2024-09-11T08:26:55.920646+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:55.920512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:26:55.920512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:16.956014+0000","scrub_duration":0.011546753999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":464,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.727299+0000","last_change":"2024-09-11T08:24:28.362467+0000","last_active":"2024-09-11T08:26:55.727299+0000","last_peered":"2024-09-11T08:26:55.727299+0000","last_clean":"2024-09-11T08:26:55.727299+0000","last_became_active":"2024-09-11T08:24:28.360235+0000","last_became_peered":"2024-09-11T08:24:28.360235+0000","last_unstale":"2024-09-11T08:26:55.727299+0000","last_undegraded":"2024-09-11T08:26:55.727299+0000","last_fullsized":"2024-09-11T08:26:55.727299+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:22:37.263677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:22:37.263677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065067149999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":425,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.724164+0000","last_change":"2024-09-11T08:24:24.506494+0000","last_active":"2024-09-11T08:26:55.724164+0000","last_peered":"2024-09-11T08:26:55.724164+0000","last_clean":"2024-09-11T08:26:55.724164+0000","last_became_active":"2024-09-11T08:07:36.631668+0000","last_became_peered":"2024-09-11T08:07:36.631668+0000","last_unstale":"2024-09-11T08:26:55.724164+0000","last_undegraded":"2024-09-11T08:26:55.724164+0000","last_fullsized":"2024-09-11T08:26:55.724164+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:24.506402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:58.517624+0000","last_clean_scrub_stamp":"2024-09-11T08:24:24.506402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0096043550000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.b","version":"392'3111","reported_seq":3129,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-11T08:27:03.514369+0000","last_change":"2024-09-11T08:26:27.764606+0000","last_active":"2024-09-11T08:27:03.514369+0000","last_peered":"2024-09-11T08:27:03.514369+0000","last_clean":"2024-09-11T08:27:03.514369+0000","last_became_active":"2024-09-11T08:26:27.764354+0000","last_became_peered":"2024-09-11T08:26:27.764354+0000","last_unstale":"2024-09-11T08:27:03.514369+0000","last_undegraded":"2024-09-11T08:27:03.514369+0000","last_fullsized":"2024-09-11T08:27:03.514369+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:26:26.740307+0000","objects_scrubbed":0,"log_size":3111,"log_dups_size":0,"ondisk_log_size":3111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:56.383464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203882496,"num_objects":3111,"num_object_clones":0,"num_object_copies":9333,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3111,"num_write_kb":199104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"296'52","reported_seq":231,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:58.790349+0000","last_change":"2024-09-11T08:24:13.567684+0000","last_active":"2024-09-11T08:26:58.790349+0000","last_peered":"2024-09-11T08:26:58.790349+0000","last_clean":"2024-09-11T08:26:58.790349+0000","last_became_active":"2024-09-11T08:20:52.086843+0000","last_became_peered":"2024-09-11T08:20:52.086843+0000","last_unstale":"2024-09-11T08:26:58.790349+0000","last_undegraded":"2024-09-11T08:26:58.790349+0000","last_fullsized":"2024-09-11T08:26:58.790349+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'52","last_scrub_stamp":"2024-09-11T08:24:13.567595+0000","last_deep_scrub":"190'39","last_deep_scrub_stamp":"2024-09-11T08:15:52.847968+0000","last_clean_scrub_stamp":"2024-09-11T08:24:13.567595+0000","objects_scrubbed":10,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028629020000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"59'1","reported_seq":85,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.734170+0000","last_change":"2024-09-11T08:24:13.277960+0000","last_active":"2024-09-11T08:26:55.734170+0000","last_peered":"2024-09-11T08:26:55.734170+0000","last_clean":"2024-09-11T08:26:55.734170+0000","last_became_active":"2024-09-11T08:23:22.691726+0000","last_became_peered":"2024-09-11T08:23:22.691726+0000","last_unstale":"2024-09-11T08:26:55.734170+0000","last_undegraded":"2024-09-11T08:26:55.734170+0000","last_fullsized":"2024-09-11T08:26:55.734170+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-11T08:24:13.275269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:24:13.275269+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.007807863,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":141,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.725060+0000","last_change":"2024-09-11T08:24:29.448464+0000","last_active":"2024-09-11T08:26:55.725060+0000","last_peered":"2024-09-11T08:26:55.725060+0000","last_clean":"2024-09-11T08:26:55.725060+0000","last_became_active":"2024-09-11T08:24:28.362590+0000","last_became_peered":"2024-09-11T08:24:28.362590+0000","last_unstale":"2024-09-11T08:26:55.725060+0000","last_undegraded":"2024-09-11T08:26:55.725060+0000","last_fullsized":"2024-09-11T08:26:55.725060+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-11T08:24:29.448408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:24:29.448408+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0075800399999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":407,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.733716+0000","last_change":"2024-09-11T08:24:08.219265+0000","last_active":"2024-09-11T08:26:55.733716+0000","last_peered":"2024-09-11T08:26:55.733716+0000","last_clean":"2024-09-11T08:26:55.733716+0000","last_became_active":"2024-09-11T08:07:34.057154+0000","last_became_peered":"2024-09-11T08:07:34.057154+0000","last_unstale":"2024-09-11T08:26:55.733716+0000","last_undegraded":"2024-09-11T08:26:55.733716+0000","last_fullsized":"2024-09-11T08:26:55.733716+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:08.219182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:10:37.957096+0000","last_clean_scrub_stamp":"2024-09-11T08:24:08.219182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0084274109999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":443,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.730595+0000","last_change":"2024-09-11T08:24:16.032604+0000","last_active":"2024-09-11T08:26:55.730595+0000","last_peered":"2024-09-11T08:26:55.730595+0000","last_clean":"2024-09-11T08:26:55.730595+0000","last_became_active":"2024-09-11T08:15:56.525099+0000","last_became_peered":"2024-09-11T08:15:56.525099+0000","last_unstale":"2024-09-11T08:26:55.730595+0000","last_undegraded":"2024-09-11T08:26:55.730595+0000","last_fullsized":"2024-09-11T08:26:55.730595+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:16.032515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:24:16.032515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044030470000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"392'3215","reported_seq":3233,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-11T08:27:03.154527+0000","last_change":"2024-09-11T08:26:28.083045+0000","last_active":"2024-09-11T08:27:03.154527+0000","last_peered":"2024-09-11T08:27:03.154527+0000","last_clean":"2024-09-11T08:27:03.154527+0000","last_became_active":"2024-09-11T08:26:28.082722+0000","last_became_peered":"2024-09-11T08:26:28.082722+0000","last_unstale":"2024-09-11T08:27:03.154527+0000","last_undegraded":"2024-09-11T08:27:03.154527+0000","last_fullsized":"2024-09-11T08:27:03.154527+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:26:26.740307+0000","objects_scrubbed":0,"log_size":3215,"log_dups_size":0,"ondisk_log_size":3215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:56.502731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210698240,"num_objects":3215,"num_object_clones":0,"num_object_copies":9645,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3215,"num_write_kb":205760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"239'17","reported_seq":532,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:56.891223+0000","last_change":"2024-09-11T08:26:56.891150+0000","last_active":"2024-09-11T08:26:56.891223+0000","last_peered":"2024-09-11T08:26:56.891223+0000","last_clean":"2024-09-11T08:26:56.891223+0000","last_became_active":"2024-09-11T08:24:28.358121+0000","last_became_peered":"2024-09-11T08:24:28.358121+0000","last_unstale":"2024-09-11T08:26:56.891223+0000","last_undegraded":"2024-09-11T08:26:56.891223+0000","last_fullsized":"2024-09-11T08:26:56.891223+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'17","last_scrub_stamp":"2024-09-11T08:26:56.890998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:26:56.890998+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:16.584252+0000","scrub_duration":0.012396804000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":444,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.727204+0000","last_change":"2024-09-11T08:26:10.658086+0000","last_active":"2024-09-11T08:26:55.727204+0000","last_peered":"2024-09-11T08:26:55.727204+0000","last_clean":"2024-09-11T08:26:55.727204+0000","last_became_active":"2024-09-11T08:07:30.047014+0000","last_became_peered":"2024-09-11T08:07:30.047014+0000","last_unstale":"2024-09-11T08:26:55.727204+0000","last_undegraded":"2024-09-11T08:26:55.727204+0000","last_fullsized":"2024-09-11T08:26:55.727204+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:10.658003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:26:10.658003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:16.143224+0000","scrub_duration":0.001689107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"127'2","reported_seq":461,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.728080+0000","last_change":"2024-09-11T08:26:53.737700+0000","last_active":"2024-09-11T08:26:55.728080+0000","last_peered":"2024-09-11T08:26:55.728080+0000","last_clean":"2024-09-11T08:26:55.728080+0000","last_became_active":"2024-09-11T08:07:30.045009+0000","last_became_peered":"2024-09-11T08:07:30.045009+0000","last_unstale":"2024-09-11T08:26:55.728080+0000","last_undegraded":"2024-09-11T08:26:55.728080+0000","last_fullsized":"2024-09-11T08:26:55.728080+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'2","last_scrub_stamp":"2024-09-11T08:26:53.737601+0000","last_deep_scrub":"127'2","last_deep_scrub_stamp":"2024-09-11T08:15:01.938497+0000","last_clean_scrub_stamp":"2024-09-11T08:26:53.737601+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:56.865318+0000","scrub_duration":0.0039636020000000001,"objects_trimmed":0,"snaptrim_duration":0.001859533,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":487,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.728777+0000","last_change":"2024-09-11T08:26:05.592197+0000","last_active":"2024-09-11T08:26:55.728777+0000","last_peered":"2024-09-11T08:26:55.728777+0000","last_clean":"2024-09-11T08:26:55.728777+0000","last_became_active":"2024-09-11T08:24:28.357141+0000","last_became_peered":"2024-09-11T08:24:28.357141+0000","last_unstale":"2024-09-11T08:26:55.728777+0000","last_undegraded":"2024-09-11T08:26:55.728777+0000","last_fullsized":"2024-09-11T08:26:55.728777+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:05.592132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:22:00.200872+0000","last_clean_scrub_stamp":"2024-09-11T08:26:05.592132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:31.177648+0000","scrub_duration":0.001603754,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":137,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.728710+0000","last_change":"2024-09-11T08:24:07.109892+0000","last_active":"2024-09-11T08:26:55.728710+0000","last_peered":"2024-09-11T08:26:55.728710+0000","last_clean":"2024-09-11T08:26:55.728710+0000","last_became_active":"2024-09-11T08:20:16.108674+0000","last_became_peered":"2024-09-11T08:20:16.108674+0000","last_unstale":"2024-09-11T08:26:55.728710+0000","last_undegraded":"2024-09-11T08:26:55.728710+0000","last_fullsized":"2024-09-11T08:26:55.728710+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:07.109803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:58.796217+0000","last_clean_scrub_stamp":"2024-09-11T08:24:07.109803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.036467430000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.9","version":"392'3149","reported_seq":3167,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-11T08:27:03.497396+0000","last_change":"2024-09-11T08:26:27.761702+0000","last_active":"2024-09-11T08:27:03.497396+0000","last_peered":"2024-09-11T08:27:03.497396+0000","last_clean":"2024-09-11T08:27:03.497396+0000","last_became_active":"2024-09-11T08:26:27.761414+0000","last_became_peered":"2024-09-11T08:26:27.761414+0000","last_unstale":"2024-09-11T08:27:03.497396+0000","last_undegraded":"2024-09-11T08:27:03.497396+0000","last_fullsized":"2024-09-11T08:27:03.497396+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:26:26.740307+0000","objects_scrubbed":0,"log_size":3149,"log_dups_size":0,"ondisk_log_size":3149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:40.911723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206372864,"num_objects":3149,"num_object_clones":0,"num_object_copies":9447,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3149,"num_write_kb":201536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"287'30","reported_seq":461,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.734558+0000","last_change":"2024-09-11T08:26:09.030251+0000","last_active":"2024-09-11T08:26:55.734558+0000","last_peered":"2024-09-11T08:26:55.734558+0000","last_clean":"2024-09-11T08:26:55.734558+0000","last_became_active":"2024-09-11T08:07:32.055407+0000","last_became_peered":"2024-09-11T08:07:32.055407+0000","last_unstale":"2024-09-11T08:26:55.734558+0000","last_undegraded":"2024-09-11T08:26:55.734558+0000","last_fullsized":"2024-09-11T08:26:55.734558+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"287'30","last_scrub_stamp":"2024-09-11T08:26:09.030192+0000","last_deep_scrub":"287'30","last_deep_scrub_stamp":"2024-09-11T08:26:09.030192+0000","last_clean_scrub_stamp":"2024-09-11T08:26:09.030192+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:18.828125+0000","scrub_duration":0.0020634899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"59'1","reported_seq":418,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.733595+0000","last_change":"2024-09-11T08:23:34.954017+0000","last_active":"2024-09-11T08:26:55.733595+0000","last_peered":"2024-09-11T08:26:55.733595+0000","last_clean":"2024-09-11T08:26:55.733595+0000","last_became_active":"2024-09-11T08:07:30.050504+0000","last_became_peered":"2024-09-11T08:07:30.050504+0000","last_unstale":"2024-09-11T08:26:55.733595+0000","last_undegraded":"2024-09-11T08:26:55.733595+0000","last_fullsized":"2024-09-11T08:26:55.733595+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-11T08:23:34.953923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:23:34.953923+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035255019999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"224'6","reported_seq":1372,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-11T08:27:02.143741+0000","last_change":"2024-09-11T08:24:20.011482+0000","last_active":"2024-09-11T08:27:02.143741+0000","last_peered":"2024-09-11T08:27:02.143741+0000","last_clean":"2024-09-11T08:27:02.143741+0000","last_became_active":"2024-09-11T08:14:24.793048+0000","last_became_peered":"2024-09-11T08:14:24.793048+0000","last_unstale":"2024-09-11T08:27:02.143741+0000","last_undegraded":"2024-09-11T08:27:02.143741+0000","last_fullsized":"2024-09-11T08:27:02.143741+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'6","last_scrub_stamp":"2024-09-11T08:24:20.011383+0000","last_deep_scrub":"224'6","last_deep_scrub_stamp":"2024-09-11T08:16:55.778284+0000","last_clean_scrub_stamp":"2024-09-11T08:24:20.011383+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0060728680000000004,"objects_trimmed":0,"snaptrim_duration":0.000318821,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":764,"num_read_kb":759,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":124,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.728071+0000","last_change":"2024-09-11T08:25:45.513126+0000","last_active":"2024-09-11T08:26:55.728071+0000","last_peered":"2024-09-11T08:26:55.728071+0000","last_clean":"2024-09-11T08:26:55.728071+0000","last_became_active":"2024-09-11T08:24:28.353018+0000","last_became_peered":"2024-09-11T08:24:28.353018+0000","last_unstale":"2024-09-11T08:26:55.728071+0000","last_undegraded":"2024-09-11T08:26:55.728071+0000","last_fullsized":"2024-09-11T08:26:55.728071+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:22:59.813214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:22:59.813214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012786004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":468,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.726586+0000","last_change":"2024-09-11T08:25:44.543653+0000","last_active":"2024-09-11T08:26:55.726586+0000","last_peered":"2024-09-11T08:26:55.726586+0000","last_clean":"2024-09-11T08:26:55.726586+0000","last_became_active":"2024-09-11T08:24:28.358723+0000","last_became_peered":"2024-09-11T08:24:28.358723+0000","last_unstale":"2024-09-11T08:26:55.726586+0000","last_undegraded":"2024-09-11T08:26:55.726586+0000","last_fullsized":"2024-09-11T08:26:55.726586+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:23:01.870598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:23:01.870598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041336189999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"392'3126","reported_seq":3144,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-11T08:27:03.184718+0000","last_change":"2024-09-11T08:26:28.083341+0000","last_active":"2024-09-11T08:27:03.184718+0000","last_peered":"2024-09-11T08:27:03.184718+0000","last_clean":"2024-09-11T08:27:03.184718+0000","last_became_active":"2024-09-11T08:26:28.082996+0000","last_became_peered":"2024-09-11T08:26:28.082996+0000","last_unstale":"2024-09-11T08:27:03.184718+0000","last_undegraded":"2024-09-11T08:27:03.184718+0000","last_fullsized":"2024-09-11T08:27:03.184718+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:26:26.740307+0000","objects_scrubbed":0,"log_size":3126,"log_dups_size":0,"ondisk_log_size":3126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:50.235347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204865536,"num_objects":3126,"num_object_clones":0,"num_object_copies":9378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3126,"num_write_kb":200064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"239'33","reported_seq":493,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.727994+0000","last_change":"2024-09-11T08:23:38.646247+0000","last_active":"2024-09-11T08:26:55.727994+0000","last_peered":"2024-09-11T08:26:55.727994+0000","last_clean":"2024-09-11T08:26:55.727994+0000","last_became_active":"2024-09-11T08:14:24.789154+0000","last_became_peered":"2024-09-11T08:14:24.789154+0000","last_unstale":"2024-09-11T08:26:55.727994+0000","last_undegraded":"2024-09-11T08:26:55.727994+0000","last_fullsized":"2024-09-11T08:26:55.727994+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'33","last_scrub_stamp":"2024-09-11T08:23:38.646051+0000","last_deep_scrub":"84'21","last_deep_scrub_stamp":"2024-09-11T08:08:56.705306+0000","last_clean_scrub_stamp":"2024-09-11T08:23:38.646051+0000","objects_scrubbed":7,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034659019999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":126,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.727916+0000","last_change":"2024-09-11T08:26:07.749930+0000","last_active":"2024-09-11T08:26:55.727916+0000","last_peered":"2024-09-11T08:26:55.727916+0000","last_clean":"2024-09-11T08:26:55.727916+0000","last_became_active":"2024-09-11T08:24:28.362417+0000","last_became_peered":"2024-09-11T08:24:28.362417+0000","last_unstale":"2024-09-11T08:26:55.727916+0000","last_undegraded":"2024-09-11T08:26:55.727916+0000","last_fullsized":"2024-09-11T08:26:55.727916+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:07.749828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:07.749828+0000","last_clean_scrub_stamp":"2024-09-11T08:26:07.749828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:35.713503+0000","scrub_duration":0.0015131179999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":240,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.722877+0000","last_change":"2024-09-11T08:23:37.276623+0000","last_active":"2024-09-11T08:26:55.722877+0000","last_peered":"2024-09-11T08:26:55.722877+0000","last_clean":"2024-09-11T08:26:55.722877+0000","last_became_active":"2024-09-11T08:15:56.523430+0000","last_became_peered":"2024-09-11T08:15:56.523430+0000","last_unstale":"2024-09-11T08:26:55.722877+0000","last_undegraded":"2024-09-11T08:26:55.722877+0000","last_fullsized":"2024-09-11T08:26:55.722877+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-11T08:23:37.276533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:05:21.231856+0000","last_clean_scrub_stamp":"2024-09-11T08:23:37.276533+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0089351699999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":264,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.724117+0000","last_change":"2024-09-11T08:25:55.938265+0000","last_active":"2024-09-11T08:26:55.724117+0000","last_peered":"2024-09-11T08:26:55.724117+0000","last_clean":"2024-09-11T08:26:55.724117+0000","last_became_active":"2024-09-11T08:24:28.357166+0000","last_became_peered":"2024-09-11T08:24:28.357166+0000","last_unstale":"2024-09-11T08:26:55.724117+0000","last_undegraded":"2024-09-11T08:26:55.724117+0000","last_fullsized":"2024-09-11T08:26:55.724117+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:22:48.591411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:20:25.775869+0000","last_clean_scrub_stamp":"2024-09-11T08:22:48.591411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0087669500000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":171,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.726399+0000","last_change":"2024-09-11T08:26:00.778375+0000","last_active":"2024-09-11T08:26:55.726399+0000","last_peered":"2024-09-11T08:26:55.726399+0000","last_clean":"2024-09-11T08:26:55.726399+0000","last_became_active":"2024-09-11T08:24:28.356071+0000","last_became_peered":"2024-09-11T08:24:28.356071+0000","last_unstale":"2024-09-11T08:26:55.726399+0000","last_undegraded":"2024-09-11T08:26:55.726399+0000","last_fullsized":"2024-09-11T08:26:55.726399+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:00.778324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:01.777429+0000","last_clean_scrub_stamp":"2024-09-11T08:26:00.778324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:10.704121+0000","scrub_duration":0.001196814,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"392'3020","reported_seq":3038,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-11T08:27:03.191216+0000","last_change":"2024-09-11T08:26:27.764691+0000","last_active":"2024-09-11T08:27:03.191216+0000","last_peered":"2024-09-11T08:27:03.191216+0000","last_clean":"2024-09-11T08:27:03.191216+0000","last_became_active":"2024-09-11T08:26:27.763926+0000","last_became_peered":"2024-09-11T08:26:27.763926+0000","last_unstale":"2024-09-11T08:27:03.191216+0000","last_undegraded":"2024-09-11T08:27:03.191216+0000","last_fullsized":"2024-09-11T08:27:03.191216+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:26:26.740307+0000","objects_scrubbed":0,"log_size":3020,"log_dups_size":0,"ondisk_log_size":3020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:27.656690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197918720,"num_objects":3020,"num_object_clones":0,"num_object_copies":9060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3020,"num_write_kb":193280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"239'24","reported_seq":355,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:59.682929+0000","last_change":"2024-09-11T08:26:59.682857+0000","last_active":"2024-09-11T08:26:59.682929+0000","last_peered":"2024-09-11T08:26:59.682929+0000","last_clean":"2024-09-11T08:26:59.682929+0000","last_became_active":"2024-09-11T08:24:28.368759+0000","last_became_peered":"2024-09-11T08:24:28.368759+0000","last_unstale":"2024-09-11T08:26:59.682929+0000","last_undegraded":"2024-09-11T08:26:59.682929+0000","last_fullsized":"2024-09-11T08:26:59.682929+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'24","last_scrub_stamp":"2024-09-11T08:26:59.682768+0000","last_deep_scrub":"88'16","last_deep_scrub_stamp":"2024-09-11T08:09:39.635984+0000","last_clean_scrub_stamp":"2024-09-11T08:26:59.682768+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:17.305928+0000","scrub_duration":0.0052817990000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":438,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.720147+0000","last_change":"2024-09-11T08:26:06.958652+0000","last_active":"2024-09-11T08:26:55.720147+0000","last_peered":"2024-09-11T08:26:55.720147+0000","last_clean":"2024-09-11T08:26:55.720147+0000","last_became_active":"2024-09-11T08:14:24.789888+0000","last_became_peered":"2024-09-11T08:14:24.789888+0000","last_unstale":"2024-09-11T08:26:55.720147+0000","last_undegraded":"2024-09-11T08:26:55.720147+0000","last_fullsized":"2024-09-11T08:26:55.720147+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:06.958598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:36.417911+0000","last_clean_scrub_stamp":"2024-09-11T08:26:06.958598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:31.023729+0000","scrub_duration":0.001521522,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":118,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.727196+0000","last_change":"2024-09-11T08:26:00.761197+0000","last_active":"2024-09-11T08:26:55.727196+0000","last_peered":"2024-09-11T08:26:55.727196+0000","last_clean":"2024-09-11T08:26:55.727196+0000","last_became_active":"2024-09-11T08:24:28.361261+0000","last_became_peered":"2024-09-11T08:24:28.361261+0000","last_unstale":"2024-09-11T08:26:55.727196+0000","last_undegraded":"2024-09-11T08:26:55.727196+0000","last_fullsized":"2024-09-11T08:26:55.727196+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:00.761139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:01.558340+0000","last_clean_scrub_stamp":"2024-09-11T08:26:00.761139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:06.959545+0000","scrub_duration":0.001250099,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":256,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:27:00.720904+0000","last_change":"2024-09-11T08:27:00.720851+0000","last_active":"2024-09-11T08:27:00.720904+0000","last_peered":"2024-09-11T08:27:00.720904+0000","last_clean":"2024-09-11T08:27:00.720904+0000","last_became_active":"2024-09-11T08:24:28.357842+0000","last_became_peered":"2024-09-11T08:24:28.357842+0000","last_unstale":"2024-09-11T08:27:00.720904+0000","last_undegraded":"2024-09-11T08:27:00.720904+0000","last_fullsized":"2024-09-11T08:27:00.720904+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:00.720766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:27:00.720766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:21.874998+0000","scrub_duration":0.001951798,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"392'3203","reported_seq":3221,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-11T08:27:03.492403+0000","last_change":"2024-09-11T08:26:27.763889+0000","last_active":"2024-09-11T08:27:03.492403+0000","last_peered":"2024-09-11T08:27:03.492403+0000","last_clean":"2024-09-11T08:27:03.492403+0000","last_became_active":"2024-09-11T08:26:27.763602+0000","last_became_peered":"2024-09-11T08:26:27.763602+0000","last_unstale":"2024-09-11T08:27:03.492403+0000","last_undegraded":"2024-09-11T08:27:03.492403+0000","last_fullsized":"2024-09-11T08:27:03.492403+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:26:26.740307+0000","objects_scrubbed":0,"log_size":3203,"log_dups_size":0,"ondisk_log_size":3203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:29.384434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209911808,"num_objects":3203,"num_object_clones":0,"num_object_copies":9609,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3203,"num_write_kb":204992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"239'39","reported_seq":397,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.728959+0000","last_change":"2024-09-11T08:26:08.526760+0000","last_active":"2024-09-11T08:26:55.728959+0000","last_peered":"2024-09-11T08:26:55.728959+0000","last_clean":"2024-09-11T08:26:55.728959+0000","last_became_active":"2024-09-11T08:23:22.681452+0000","last_became_peered":"2024-09-11T08:23:22.681452+0000","last_unstale":"2024-09-11T08:26:55.728959+0000","last_undegraded":"2024-09-11T08:26:55.728959+0000","last_fullsized":"2024-09-11T08:26:55.728959+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'39","last_scrub_stamp":"2024-09-11T08:26:08.526702+0000","last_deep_scrub":"77'25","last_deep_scrub_stamp":"2024-09-11T08:08:32.458210+0000","last_clean_scrub_stamp":"2024-09-11T08:26:08.526702+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:37.269972+0000","scrub_duration":0.0021726229999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":437,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.727495+0000","last_change":"2024-09-11T08:24:06.575259+0000","last_active":"2024-09-11T08:26:55.727495+0000","last_peered":"2024-09-11T08:26:55.727495+0000","last_clean":"2024-09-11T08:26:55.727495+0000","last_became_active":"2024-09-11T08:07:30.042555+0000","last_became_peered":"2024-09-11T08:07:30.042555+0000","last_unstale":"2024-09-11T08:26:55.727495+0000","last_undegraded":"2024-09-11T08:26:55.727495+0000","last_fullsized":"2024-09-11T08:26:55.727495+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:06.575206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:24:06.575206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014429386000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":427,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.726648+0000","last_change":"2024-09-11T08:24:08.512741+0000","last_active":"2024-09-11T08:26:55.726648+0000","last_peered":"2024-09-11T08:26:55.726648+0000","last_clean":"2024-09-11T08:26:55.726648+0000","last_became_active":"2024-09-11T08:07:34.054961+0000","last_became_peered":"2024-09-11T08:07:34.054961+0000","last_unstale":"2024-09-11T08:26:55.726648+0000","last_undegraded":"2024-09-11T08:26:55.726648+0000","last_fullsized":"2024-09-11T08:26:55.726648+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:08.512659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:24:08.512659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010695556,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":445,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.728814+0000","last_change":"2024-09-11T08:24:17.014955+0000","last_active":"2024-09-11T08:26:55.728814+0000","last_peered":"2024-09-11T08:26:55.728814+0000","last_clean":"2024-09-11T08:26:55.728814+0000","last_became_active":"2024-09-11T08:07:36.071194+0000","last_became_peered":"2024-09-11T08:07:36.071194+0000","last_unstale":"2024-09-11T08:26:55.728814+0000","last_undegraded":"2024-09-11T08:26:55.728814+0000","last_fullsized":"2024-09-11T08:26:55.728814+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.014880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:07.930137+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.014880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.01453079,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"392'3088","reported_seq":3106,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-11T08:27:03.174367+0000","last_change":"2024-09-11T08:26:27.764903+0000","last_active":"2024-09-11T08:27:03.174367+0000","last_peered":"2024-09-11T08:27:03.174367+0000","last_clean":"2024-09-11T08:27:03.174367+0000","last_became_active":"2024-09-11T08:26:27.764638+0000","last_became_peered":"2024-09-11T08:26:27.764638+0000","last_unstale":"2024-09-11T08:27:03.174367+0000","last_undegraded":"2024-09-11T08:27:03.174367+0000","last_fullsized":"2024-09-11T08:27:03.174367+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:26:26.740307+0000","objects_scrubbed":0,"log_size":3088,"log_dups_size":0,"ondisk_log_size":3088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:38.288480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202375168,"num_objects":3088,"num_object_clones":0,"num_object_copies":9264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3088,"num_write_kb":197632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"239'25","reported_seq":337,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.726878+0000","last_change":"2024-09-11T08:24:24.576788+0000","last_active":"2024-09-11T08:26:55.726878+0000","last_peered":"2024-09-11T08:26:55.726878+0000","last_clean":"2024-09-11T08:26:55.726878+0000","last_became_active":"2024-09-11T08:23:22.688631+0000","last_became_peered":"2024-09-11T08:23:22.688631+0000","last_unstale":"2024-09-11T08:26:55.726878+0000","last_undegraded":"2024-09-11T08:26:55.726878+0000","last_fullsized":"2024-09-11T08:26:55.726878+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'25","last_scrub_stamp":"2024-09-11T08:24:24.576699+0000","last_deep_scrub":"224'15","last_deep_scrub_stamp":"2024-09-11T08:17:01.126478+0000","last_clean_scrub_stamp":"2024-09-11T08:24:24.576699+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003559857,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":196,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.734292+0000","last_change":"2024-09-11T08:22:54.830543+0000","last_active":"2024-09-11T08:26:55.734292+0000","last_peered":"2024-09-11T08:26:55.734292+0000","last_clean":"2024-09-11T08:26:55.734292+0000","last_became_active":"2024-09-11T08:17:37.557278+0000","last_became_peered":"2024-09-11T08:17:37.557278+0000","last_unstale":"2024-09-11T08:26:55.734292+0000","last_undegraded":"2024-09-11T08:26:55.734292+0000","last_fullsized":"2024-09-11T08:26:55.734292+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:22:54.830465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:45.979847+0000","last_clean_scrub_stamp":"2024-09-11T08:22:54.830465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0062432310000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"354'16","reported_seq":740,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:58.788532+0000","last_change":"2024-09-11T08:26:01.536740+0000","last_active":"2024-09-11T08:26:58.788532+0000","last_peered":"2024-09-11T08:26:58.788532+0000","last_clean":"2024-09-11T08:26:58.788532+0000","last_became_active":"2024-09-11T08:24:28.356749+0000","last_became_peered":"2024-09-11T08:24:28.356749+0000","last_unstale":"2024-09-11T08:26:58.788532+0000","last_undegraded":"2024-09-11T08:26:58.788532+0000","last_fullsized":"2024-09-11T08:26:58.788532+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'16","last_scrub_stamp":"2024-09-11T08:26:01.536677+0000","last_deep_scrub":"171'12","last_deep_scrub_stamp":"2024-09-11T08:14:56.881299+0000","last_clean_scrub_stamp":"2024-09-11T08:26:01.536677+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:06.651254+0000","scrub_duration":0.0018058200000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":86,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.725899+0000","last_change":"2024-09-11T08:26:06.582469+0000","last_active":"2024-09-11T08:26:55.725899+0000","last_peered":"2024-09-11T08:26:55.725899+0000","last_clean":"2024-09-11T08:26:55.725899+0000","last_became_active":"2024-09-11T08:23:22.686992+0000","last_became_peered":"2024-09-11T08:23:22.686992+0000","last_unstale":"2024-09-11T08:26:55.725899+0000","last_undegraded":"2024-09-11T08:26:55.725899+0000","last_fullsized":"2024-09-11T08:26:55.725899+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:06.582413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:14.613532+0000","last_clean_scrub_stamp":"2024-09-11T08:26:06.582413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:15.324916+0000","scrub_duration":0.0014342210000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.4","version":"392'3031","reported_seq":3049,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-11T08:27:03.177719+0000","last_change":"2024-09-11T08:26:27.765910+0000","last_active":"2024-09-11T08:27:03.177719+0000","last_peered":"2024-09-11T08:27:03.177719+0000","last_clean":"2024-09-11T08:27:03.177719+0000","last_became_active":"2024-09-11T08:26:27.765686+0000","last_became_peered":"2024-09-11T08:26:27.765686+0000","last_unstale":"2024-09-11T08:27:03.177719+0000","last_undegraded":"2024-09-11T08:27:03.177719+0000","last_fullsized":"2024-09-11T08:27:03.177719+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:26:26.740307+0000","objects_scrubbed":0,"log_size":3031,"log_dups_size":0,"ondisk_log_size":3031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:39.176596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198639616,"num_objects":3031,"num_object_clones":0,"num_object_copies":9093,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3031,"num_write_kb":193984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"239'15","reported_seq":434,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.733983+0000","last_change":"2024-09-11T08:23:37.991614+0000","last_active":"2024-09-11T08:26:55.733983+0000","last_peered":"2024-09-11T08:26:55.733983+0000","last_clean":"2024-09-11T08:26:55.733983+0000","last_became_active":"2024-09-11T08:07:32.054885+0000","last_became_peered":"2024-09-11T08:07:32.054885+0000","last_unstale":"2024-09-11T08:26:55.733983+0000","last_undegraded":"2024-09-11T08:26:55.733983+0000","last_fullsized":"2024-09-11T08:26:55.733983+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'15","last_scrub_stamp":"2024-09-11T08:23:37.991518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:23:37.991518+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0096807339999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":122,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.730376+0000","last_change":"2024-09-11T08:24:28.360594+0000","last_active":"2024-09-11T08:26:55.730376+0000","last_peered":"2024-09-11T08:26:55.730376+0000","last_clean":"2024-09-11T08:26:55.730376+0000","last_became_active":"2024-09-11T08:24:28.359057+0000","last_became_peered":"2024-09-11T08:24:28.359057+0000","last_unstale":"2024-09-11T08:26:55.730376+0000","last_undegraded":"2024-09-11T08:26:55.730376+0000","last_fullsized":"2024-09-11T08:26:55.730376+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:23:02.004444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:23:02.004444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039846120000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":447,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.729038+0000","last_change":"2024-09-11T08:24:14.017136+0000","last_active":"2024-09-11T08:26:55.729038+0000","last_peered":"2024-09-11T08:26:55.729038+0000","last_clean":"2024-09-11T08:26:55.729038+0000","last_became_active":"2024-09-11T08:07:34.056606+0000","last_became_peered":"2024-09-11T08:07:34.056606+0000","last_unstale":"2024-09-11T08:26:55.729038+0000","last_undegraded":"2024-09-11T08:26:55.729038+0000","last_fullsized":"2024-09-11T08:26:55.729038+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:14.017048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:24:14.017048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013235778,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":432,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.733525+0000","last_change":"2024-09-11T08:23:46.203003+0000","last_active":"2024-09-11T08:26:55.733525+0000","last_peered":"2024-09-11T08:26:55.733525+0000","last_clean":"2024-09-11T08:26:55.733525+0000","last_became_active":"2024-09-11T08:15:56.518884+0000","last_became_peered":"2024-09-11T08:15:56.518884+0000","last_unstale":"2024-09-11T08:26:55.733525+0000","last_undegraded":"2024-09-11T08:26:55.733525+0000","last_fullsized":"2024-09-11T08:26:55.733525+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:23:46.202841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:18:32.278257+0000","last_clean_scrub_stamp":"2024-09-11T08:23:46.202841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014919458999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"392'3107","reported_seq":3125,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-11T08:27:03.215942+0000","last_change":"2024-09-11T08:26:28.083356+0000","last_active":"2024-09-11T08:27:03.215942+0000","last_peered":"2024-09-11T08:27:03.215942+0000","last_clean":"2024-09-11T08:27:03.215942+0000","last_became_active":"2024-09-11T08:26:28.083033+0000","last_became_peered":"2024-09-11T08:26:28.083033+0000","last_unstale":"2024-09-11T08:27:03.215942+0000","last_undegraded":"2024-09-11T08:27:03.215942+0000","last_fullsized":"2024-09-11T08:27:03.215942+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:26:26.740307+0000","objects_scrubbed":0,"log_size":3107,"log_dups_size":0,"ondisk_log_size":3107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:51.535683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203620352,"num_objects":3107,"num_object_clones":0,"num_object_copies":9321,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3107,"num_write_kb":198848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"239'35","reported_seq":561,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.727842+0000","last_change":"2024-09-11T08:23:01.280555+0000","last_active":"2024-09-11T08:26:55.727842+0000","last_peered":"2024-09-11T08:26:55.727842+0000","last_clean":"2024-09-11T08:26:55.727842+0000","last_became_active":"2024-09-11T08:15:56.526961+0000","last_became_peered":"2024-09-11T08:15:56.526961+0000","last_unstale":"2024-09-11T08:26:55.727842+0000","last_undegraded":"2024-09-11T08:26:55.727842+0000","last_fullsized":"2024-09-11T08:26:55.727842+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'35","last_scrub_stamp":"2024-09-11T08:23:01.280334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:23:01.280334+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038257009999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":274,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-11T08:27:02.188946+0000","last_change":"2024-09-11T08:27:02.188913+0000","last_active":"2024-09-11T08:27:02.188946+0000","last_peered":"2024-09-11T08:27:02.188946+0000","last_clean":"2024-09-11T08:27:02.188946+0000","last_became_active":"2024-09-11T08:24:28.353720+0000","last_became_peered":"2024-09-11T08:24:28.353720+0000","last_unstale":"2024-09-11T08:27:02.188946+0000","last_undegraded":"2024-09-11T08:27:02.188946+0000","last_fullsized":"2024-09-11T08:27:02.188946+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:02.188743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:44.385372+0000","last_clean_scrub_stamp":"2024-09-11T08:27:02.188743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:16.282589+0000","scrub_duration":0.002554925,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":477,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.728500+0000","last_change":"2024-09-11T08:26:06.546660+0000","last_active":"2024-09-11T08:26:55.728500+0000","last_peered":"2024-09-11T08:26:55.728500+0000","last_clean":"2024-09-11T08:26:55.728500+0000","last_became_active":"2024-09-11T08:24:28.365792+0000","last_became_peered":"2024-09-11T08:24:28.365792+0000","last_unstale":"2024-09-11T08:26:55.728500+0000","last_undegraded":"2024-09-11T08:26:55.728500+0000","last_fullsized":"2024-09-11T08:26:55.728500+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:06.546573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:06.546573+0000","last_clean_scrub_stamp":"2024-09-11T08:26:06.546573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:21.762385+0000","scrub_duration":0.0020691960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":234,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.722967+0000","last_change":"2024-09-11T08:24:26.983211+0000","last_active":"2024-09-11T08:26:55.722967+0000","last_peered":"2024-09-11T08:26:55.722967+0000","last_clean":"2024-09-11T08:26:55.722967+0000","last_became_active":"2024-09-11T08:15:56.518169+0000","last_became_peered":"2024-09-11T08:15:56.518169+0000","last_unstale":"2024-09-11T08:26:55.722967+0000","last_undegraded":"2024-09-11T08:26:55.722967+0000","last_fullsized":"2024-09-11T08:26:55.722967+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:26.983134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:24:26.983134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0081131989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"392'3233","reported_seq":3251,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-11T08:27:04.188866+0000","last_change":"2024-09-11T08:26:27.764924+0000","last_active":"2024-09-11T08:27:04.188866+0000","last_peered":"2024-09-11T08:27:04.188866+0000","last_clean":"2024-09-11T08:27:04.188866+0000","last_became_active":"2024-09-11T08:26:27.764636+0000","last_became_peered":"2024-09-11T08:26:27.764636+0000","last_unstale":"2024-09-11T08:27:04.188866+0000","last_undegraded":"2024-09-11T08:27:04.188866+0000","last_fullsized":"2024-09-11T08:27:04.188866+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:26:26.740307+0000","objects_scrubbed":0,"log_size":3233,"log_dups_size":0,"ondisk_log_size":3233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:28.732207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211877888,"num_objects":3233,"num_object_clones":0,"num_object_copies":9699,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3233,"num_write_kb":206912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"239'23","reported_seq":338,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.725941+0000","last_change":"2024-09-11T08:24:07.501579+0000","last_active":"2024-09-11T08:26:55.725941+0000","last_peered":"2024-09-11T08:26:55.725941+0000","last_clean":"2024-09-11T08:26:55.725941+0000","last_became_active":"2024-09-11T08:23:22.686387+0000","last_became_peered":"2024-09-11T08:23:22.686387+0000","last_unstale":"2024-09-11T08:26:55.725941+0000","last_undegraded":"2024-09-11T08:26:55.725941+0000","last_fullsized":"2024-09-11T08:26:55.725941+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'23","last_scrub_stamp":"2024-09-11T08:24:07.501504+0000","last_deep_scrub":"145'17","last_deep_scrub_stamp":"2024-09-11T08:13:08.160848+0000","last_clean_scrub_stamp":"2024-09-11T08:24:07.501504+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011071242,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":244,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.723364+0000","last_change":"2024-09-11T08:25:57.957773+0000","last_active":"2024-09-11T08:26:55.723364+0000","last_peered":"2024-09-11T08:26:55.723364+0000","last_clean":"2024-09-11T08:26:55.723364+0000","last_became_active":"2024-09-11T08:15:56.525845+0000","last_became_peered":"2024-09-11T08:15:56.525845+0000","last_unstale":"2024-09-11T08:26:55.723364+0000","last_undegraded":"2024-09-11T08:26:55.723364+0000","last_fullsized":"2024-09-11T08:26:55.723364+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:25:57.957725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:21:06.262572+0000","last_clean_scrub_stamp":"2024-09-11T08:25:57.957725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012472080000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":427,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.727653+0000","last_change":"2024-09-11T08:24:13.019271+0000","last_active":"2024-09-11T08:26:55.727653+0000","last_peered":"2024-09-11T08:26:55.727653+0000","last_clean":"2024-09-11T08:26:55.727653+0000","last_became_active":"2024-09-11T08:07:34.057479+0000","last_became_peered":"2024-09-11T08:07:34.057479+0000","last_unstale":"2024-09-11T08:26:55.727653+0000","last_undegraded":"2024-09-11T08:26:55.727653+0000","last_fullsized":"2024-09-11T08:26:55.727653+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:13.019188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:24:13.019188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.038354963999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":443,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.730150+0000","last_change":"2024-09-11T08:23:32.212887+0000","last_active":"2024-09-11T08:26:55.730150+0000","last_peered":"2024-09-11T08:26:55.730150+0000","last_clean":"2024-09-11T08:26:55.730150+0000","last_became_active":"2024-09-11T08:15:56.522624+0000","last_became_peered":"2024-09-11T08:15:56.522624+0000","last_unstale":"2024-09-11T08:26:55.730150+0000","last_undegraded":"2024-09-11T08:26:55.730150+0000","last_fullsized":"2024-09-11T08:26:55.730150+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:23:32.212832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:23:32.212832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011649189999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"392'3165","reported_seq":3181,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-11T08:27:03.785515+0000","last_change":"2024-09-11T08:26:28.083401+0000","last_active":"2024-09-11T08:27:03.785515+0000","last_peered":"2024-09-11T08:27:03.785515+0000","last_clean":"2024-09-11T08:27:03.785515+0000","last_became_active":"2024-09-11T08:26:28.083190+0000","last_became_peered":"2024-09-11T08:26:28.083190+0000","last_unstale":"2024-09-11T08:27:03.785515+0000","last_undegraded":"2024-09-11T08:27:03.785515+0000","last_fullsized":"2024-09-11T08:27:03.785515+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:26:26.740307+0000","objects_scrubbed":0,"log_size":3165,"log_dups_size":0,"ondisk_log_size":3165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:43.060889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207421440,"num_objects":3165,"num_object_clones":0,"num_object_copies":9495,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3165,"num_write_kb":202560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"239'27","reported_seq":467,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.722781+0000","last_change":"2024-09-11T08:25:53.986001+0000","last_active":"2024-09-11T08:26:55.722781+0000","last_peered":"2024-09-11T08:26:55.722781+0000","last_clean":"2024-09-11T08:26:55.722781+0000","last_became_active":"2024-09-11T08:24:28.353397+0000","last_became_peered":"2024-09-11T08:24:28.353397+0000","last_unstale":"2024-09-11T08:26:55.722781+0000","last_undegraded":"2024-09-11T08:26:55.722781+0000","last_fullsized":"2024-09-11T08:26:55.722781+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'27","last_scrub_stamp":"2024-09-11T08:22:42.281532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:22:42.281532+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010207872,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":240,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.724025+0000","last_change":"2024-09-11T08:24:17.089977+0000","last_active":"2024-09-11T08:26:55.724025+0000","last_peered":"2024-09-11T08:26:55.724025+0000","last_clean":"2024-09-11T08:26:55.724025+0000","last_became_active":"2024-09-11T08:15:56.515328+0000","last_became_peered":"2024-09-11T08:15:56.515328+0000","last_unstale":"2024-09-11T08:26:55.724025+0000","last_undegraded":"2024-09-11T08:26:55.724025+0000","last_fullsized":"2024-09-11T08:26:55.724025+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-11T08:24:17.089895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.089895+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019391429999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"209'11","reported_seq":371,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:58.788272+0000","last_change":"2024-09-11T08:25:59.008089+0000","last_active":"2024-09-11T08:26:58.788272+0000","last_peered":"2024-09-11T08:26:58.788272+0000","last_clean":"2024-09-11T08:26:58.788272+0000","last_became_active":"2024-09-11T08:15:56.524513+0000","last_became_peered":"2024-09-11T08:15:56.524513+0000","last_unstale":"2024-09-11T08:26:58.788272+0000","last_undegraded":"2024-09-11T08:26:58.788272+0000","last_fullsized":"2024-09-11T08:26:58.788272+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"209'11","last_scrub_stamp":"2024-09-11T08:25:59.008036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:25:59.008036+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:02.045232+0000","scrub_duration":0.0014184130000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":122,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.727227+0000","last_change":"2024-09-11T08:26:06.772735+0000","last_active":"2024-09-11T08:26:55.727227+0000","last_peered":"2024-09-11T08:26:55.727227+0000","last_clean":"2024-09-11T08:26:55.727227+0000","last_became_active":"2024-09-11T08:24:28.360658+0000","last_became_peered":"2024-09-11T08:24:28.360658+0000","last_unstale":"2024-09-11T08:26:55.727227+0000","last_undegraded":"2024-09-11T08:26:55.727227+0000","last_fullsized":"2024-09-11T08:26:55.727227+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:06.772675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:03.484755+0000","last_clean_scrub_stamp":"2024-09-11T08:26:06.772675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:18.455364+0000","scrub_duration":0.0013072069999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":481,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.727073+0000","last_change":"2024-09-11T08:26:54.619464+0000","last_active":"2024-09-11T08:26:55.727073+0000","last_peered":"2024-09-11T08:26:55.727073+0000","last_clean":"2024-09-11T08:26:55.727073+0000","last_became_active":"2024-09-11T08:24:28.357401+0000","last_became_peered":"2024-09-11T08:24:28.357401+0000","last_unstale":"2024-09-11T08:26:55.727073+0000","last_undegraded":"2024-09-11T08:26:55.727073+0000","last_fullsized":"2024-09-11T08:26:55.727073+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:54.619370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:26:54.619370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:24.338015+0000","scrub_duration":0.0025718440000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"239'15","reported_seq":484,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.730908+0000","last_change":"2024-09-11T08:26:03.664732+0000","last_active":"2024-09-11T08:26:55.730908+0000","last_peered":"2024-09-11T08:26:55.730908+0000","last_clean":"2024-09-11T08:26:55.730908+0000","last_became_active":"2024-09-11T08:15:56.526325+0000","last_became_peered":"2024-09-11T08:15:56.526325+0000","last_unstale":"2024-09-11T08:26:55.730908+0000","last_undegraded":"2024-09-11T08:26:55.730908+0000","last_fullsized":"2024-09-11T08:26:55.730908+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'15","last_scrub_stamp":"2024-09-11T08:26:03.664676+0000","last_deep_scrub":"239'15","last_deep_scrub_stamp":"2024-09-11T08:26:03.664676+0000","last_clean_scrub_stamp":"2024-09-11T08:26:03.664676+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:17.627809+0000","scrub_duration":0.001893128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":122,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.731086+0000","last_change":"2024-09-11T08:25:59.563428+0000","last_active":"2024-09-11T08:26:55.731086+0000","last_peered":"2024-09-11T08:26:55.731086+0000","last_clean":"2024-09-11T08:26:55.731086+0000","last_became_active":"2024-09-11T08:24:28.358300+0000","last_became_peered":"2024-09-11T08:24:28.358300+0000","last_unstale":"2024-09-11T08:26:55.731086+0000","last_undegraded":"2024-09-11T08:26:55.731086+0000","last_fullsized":"2024-09-11T08:26:55.731086+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:25:59.563368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:53.548967+0000","last_clean_scrub_stamp":"2024-09-11T08:25:59.563368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:00.993605+0000","scrub_duration":0.0013934489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":369,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.723607+0000","last_change":"2024-09-11T08:26:09.119039+0000","last_active":"2024-09-11T08:26:55.723607+0000","last_peered":"2024-09-11T08:26:55.723607+0000","last_clean":"2024-09-11T08:26:55.723607+0000","last_became_active":"2024-09-11T08:14:24.790678+0000","last_became_peered":"2024-09-11T08:14:24.790678+0000","last_unstale":"2024-09-11T08:26:55.723607+0000","last_undegraded":"2024-09-11T08:26:55.723607+0000","last_fullsized":"2024-09-11T08:26:55.723607+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:09.118973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:26:09.118973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:31.753802+0000","scrub_duration":0.0016651439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"239'19","reported_seq":475,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.734638+0000","last_change":"2024-09-11T08:22:58.755615+0000","last_active":"2024-09-11T08:26:55.734638+0000","last_peered":"2024-09-11T08:26:55.734638+0000","last_clean":"2024-09-11T08:26:55.734638+0000","last_became_active":"2024-09-11T08:15:56.514793+0000","last_became_peered":"2024-09-11T08:15:56.514793+0000","last_unstale":"2024-09-11T08:26:55.734638+0000","last_undegraded":"2024-09-11T08:26:55.734638+0000","last_fullsized":"2024-09-11T08:26:55.734638+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'19","last_scrub_stamp":"2024-09-11T08:22:58.755559+0000","last_deep_scrub":"84'13","last_deep_scrub_stamp":"2024-09-11T08:08:57.835634+0000","last_clean_scrub_stamp":"2024-09-11T08:22:58.755559+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001670223,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":392,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.723143+0000","last_change":"2024-09-11T08:24:26.028722+0000","last_active":"2024-09-11T08:26:55.723143+0000","last_peered":"2024-09-11T08:26:55.723143+0000","last_clean":"2024-09-11T08:26:55.723143+0000","last_became_active":"2024-09-11T08:14:24.786021+0000","last_became_peered":"2024-09-11T08:14:24.786021+0000","last_unstale":"2024-09-11T08:26:55.723143+0000","last_undegraded":"2024-09-11T08:26:55.723143+0000","last_fullsized":"2024-09-11T08:26:55.723143+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:26.028642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:30.471620+0000","last_clean_scrub_stamp":"2024-09-11T08:24:26.028642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0073996629999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":445,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.730766+0000","last_change":"2024-09-11T08:24:28.006448+0000","last_active":"2024-09-11T08:26:55.730766+0000","last_peered":"2024-09-11T08:26:55.730766+0000","last_clean":"2024-09-11T08:26:55.730766+0000","last_became_active":"2024-09-11T08:15:56.525717+0000","last_became_peered":"2024-09-11T08:15:56.525717+0000","last_unstale":"2024-09-11T08:26:55.730766+0000","last_undegraded":"2024-09-11T08:26:55.730766+0000","last_fullsized":"2024-09-11T08:26:55.730766+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:28.006392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:04.370734+0000","last_clean_scrub_stamp":"2024-09-11T08:24:28.006392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001372453,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":455,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-11T08:27:00.857707+0000","last_change":"2024-09-11T08:27:00.857665+0000","last_active":"2024-09-11T08:27:00.857707+0000","last_peered":"2024-09-11T08:27:00.857707+0000","last_clean":"2024-09-11T08:27:00.857707+0000","last_became_active":"2024-09-11T08:15:56.525120+0000","last_became_peered":"2024-09-11T08:15:56.525120+0000","last_unstale":"2024-09-11T08:27:00.857707+0000","last_undegraded":"2024-09-11T08:27:00.857707+0000","last_fullsized":"2024-09-11T08:27:00.857707+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:00.857585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:54.184372+0000","last_clean_scrub_stamp":"2024-09-11T08:27:00.857585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:28.826864+0000","scrub_duration":0.009399101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"313'19","reported_seq":171,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.723294+0000","last_change":"2024-09-11T08:26:54.105820+0000","last_active":"2024-09-11T08:26:55.723294+0000","last_peered":"2024-09-11T08:26:55.723294+0000","last_clean":"2024-09-11T08:26:55.723294+0000","last_became_active":"2024-09-11T08:24:28.353043+0000","last_became_peered":"2024-09-11T08:24:28.353043+0000","last_unstale":"2024-09-11T08:26:55.723294+0000","last_undegraded":"2024-09-11T08:26:55.723294+0000","last_fullsized":"2024-09-11T08:26:55.723294+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"313'19","last_scrub_stamp":"2024-09-11T08:26:54.105645+0000","last_deep_scrub":"313'19","last_deep_scrub_stamp":"2024-09-11T08:22:03.458402+0000","last_clean_scrub_stamp":"2024-09-11T08:26:54.105645+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:13.089493+0000","scrub_duration":0.0041528440000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":438,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.733835+0000","last_change":"2024-09-11T08:23:39.025692+0000","last_active":"2024-09-11T08:26:55.733835+0000","last_peered":"2024-09-11T08:26:55.733835+0000","last_clean":"2024-09-11T08:26:55.733835+0000","last_became_active":"2024-09-11T08:15:56.513278+0000","last_became_peered":"2024-09-11T08:15:56.513278+0000","last_unstale":"2024-09-11T08:26:55.733835+0000","last_undegraded":"2024-09-11T08:26:55.733835+0000","last_fullsized":"2024-09-11T08:26:55.733835+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:23:39.025620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:52.661591+0000","last_clean_scrub_stamp":"2024-09-11T08:23:39.025620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046140570000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"239'13","reported_seq":668,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:58.788262+0000","last_change":"2024-09-11T08:23:36.270712+0000","last_active":"2024-09-11T08:26:58.788262+0000","last_peered":"2024-09-11T08:26:58.788262+0000","last_clean":"2024-09-11T08:26:58.788262+0000","last_became_active":"2024-09-11T08:17:37.558224+0000","last_became_peered":"2024-09-11T08:17:37.558224+0000","last_unstale":"2024-09-11T08:26:58.788262+0000","last_undegraded":"2024-09-11T08:26:58.788262+0000","last_fullsized":"2024-09-11T08:26:58.788262+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'13","last_scrub_stamp":"2024-09-11T08:23:36.270623+0000","last_deep_scrub":"222'12","last_deep_scrub_stamp":"2024-09-11T08:16:52.592856+0000","last_clean_scrub_stamp":"2024-09-11T08:23:36.270623+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.006222399,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":360,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.728736+0000","last_change":"2024-09-11T08:25:59.575043+0000","last_active":"2024-09-11T08:26:55.728736+0000","last_peered":"2024-09-11T08:26:55.728736+0000","last_clean":"2024-09-11T08:26:55.728736+0000","last_became_active":"2024-09-11T08:24:28.357799+0000","last_became_peered":"2024-09-11T08:24:28.357799+0000","last_unstale":"2024-09-11T08:26:55.728736+0000","last_undegraded":"2024-09-11T08:26:55.728736+0000","last_fullsized":"2024-09-11T08:26:55.728736+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:25:59.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:53.175994+0000","last_clean_scrub_stamp":"2024-09-11T08:25:59.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:22.362275+0000","scrub_duration":0.001608221,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"239'20","reported_seq":294,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.730000+0000","last_change":"2024-09-11T08:26:09.487437+0000","last_active":"2024-09-11T08:26:55.730000+0000","last_peered":"2024-09-11T08:26:55.730000+0000","last_clean":"2024-09-11T08:26:55.730000+0000","last_became_active":"2024-09-11T08:17:37.554357+0000","last_became_peered":"2024-09-11T08:17:37.554357+0000","last_unstale":"2024-09-11T08:26:55.730000+0000","last_undegraded":"2024-09-11T08:26:55.730000+0000","last_fullsized":"2024-09-11T08:26:55.730000+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'20","last_scrub_stamp":"2024-09-11T08:26:09.487383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:26:09.487383+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:22.234597+0000","scrub_duration":0.002145428,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":234,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.723753+0000","last_change":"2024-09-11T08:23:42.199613+0000","last_active":"2024-09-11T08:26:55.723753+0000","last_peered":"2024-09-11T08:26:55.723753+0000","last_clean":"2024-09-11T08:26:55.723753+0000","last_became_active":"2024-09-11T08:15:56.518515+0000","last_became_peered":"2024-09-11T08:15:56.518515+0000","last_unstale":"2024-09-11T08:26:55.723753+0000","last_undegraded":"2024-09-11T08:26:55.723753+0000","last_fullsized":"2024-09-11T08:26:55.723753+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:23:42.199527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:23:42.199527+0000","last_clean_scrub_stamp":"2024-09-11T08:23:42.199527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054572529999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"354'13","reported_seq":192,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:58.790404+0000","last_change":"2024-09-11T08:25:35.435091+0000","last_active":"2024-09-11T08:26:58.790404+0000","last_peered":"2024-09-11T08:26:58.790404+0000","last_clean":"2024-09-11T08:26:58.790404+0000","last_became_active":"2024-09-11T08:24:28.353115+0000","last_became_peered":"2024-09-11T08:24:28.353115+0000","last_unstale":"2024-09-11T08:26:58.790404+0000","last_undegraded":"2024-09-11T08:26:58.790404+0000","last_fullsized":"2024-09-11T08:26:58.790404+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'10","last_scrub_stamp":"2024-09-11T08:22:49.194037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:22:49.194037+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":432,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.734358+0000","last_change":"2024-09-11T08:26:10.045998+0000","last_active":"2024-09-11T08:26:55.734358+0000","last_peered":"2024-09-11T08:26:55.734358+0000","last_clean":"2024-09-11T08:26:55.734358+0000","last_became_active":"2024-09-11T08:15:56.513949+0000","last_became_peered":"2024-09-11T08:15:56.513949+0000","last_unstale":"2024-09-11T08:26:55.734358+0000","last_undegraded":"2024-09-11T08:26:55.734358+0000","last_fullsized":"2024-09-11T08:26:55.734358+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:10.045943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:26:10.045943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:14.663252+0000","scrub_duration":0.0017620470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"239'19","reported_seq":333,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.727730+0000","last_change":"2024-09-11T08:24:25.584115+0000","last_active":"2024-09-11T08:26:55.727730+0000","last_peered":"2024-09-11T08:26:55.727730+0000","last_clean":"2024-09-11T08:26:55.727730+0000","last_became_active":"2024-09-11T08:23:22.688993+0000","last_became_peered":"2024-09-11T08:23:22.688993+0000","last_unstale":"2024-09-11T08:26:55.727730+0000","last_undegraded":"2024-09-11T08:26:55.727730+0000","last_fullsized":"2024-09-11T08:26:55.727730+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'19","last_scrub_stamp":"2024-09-11T08:24:25.584019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:24:25.584019+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047681069999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":283,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.723369+0000","last_change":"2024-09-11T08:26:02.102192+0000","last_active":"2024-09-11T08:26:55.723369+0000","last_peered":"2024-09-11T08:26:55.723369+0000","last_clean":"2024-09-11T08:26:55.723369+0000","last_became_active":"2024-09-11T08:23:22.682569+0000","last_became_peered":"2024-09-11T08:23:22.682569+0000","last_unstale":"2024-09-11T08:26:55.723369+0000","last_undegraded":"2024-09-11T08:26:55.723369+0000","last_fullsized":"2024-09-11T08:26:55.723369+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:02.102138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:26:02.102138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:19.976358+0000","scrub_duration":0.0012891669999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":457,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.724849+0000","last_change":"2024-09-11T08:24:28.466446+0000","last_active":"2024-09-11T08:26:55.724849+0000","last_peered":"2024-09-11T08:26:55.724849+0000","last_clean":"2024-09-11T08:26:55.724849+0000","last_became_active":"2024-09-11T08:24:28.355004+0000","last_became_peered":"2024-09-11T08:24:28.355004+0000","last_unstale":"2024-09-11T08:26:55.724849+0000","last_undegraded":"2024-09-11T08:26:55.724849+0000","last_fullsized":"2024-09-11T08:26:55.724849+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:28.466393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:24:28.466393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001587551,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":236,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.733295+0000","last_change":"2024-09-11T08:25:46.175087+0000","last_active":"2024-09-11T08:26:55.733295+0000","last_peered":"2024-09-11T08:26:55.733295+0000","last_clean":"2024-09-11T08:26:55.733295+0000","last_became_active":"2024-09-11T08:24:28.372049+0000","last_became_peered":"2024-09-11T08:24:28.372049+0000","last_unstale":"2024-09-11T08:26:55.733295+0000","last_undegraded":"2024-09-11T08:26:55.733295+0000","last_fullsized":"2024-09-11T08:26:55.733295+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:21:54.636326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:21:54.636326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001465472,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"343'19","reported_seq":468,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.725135+0000","last_change":"2024-09-11T08:26:07.610504+0000","last_active":"2024-09-11T08:26:55.725135+0000","last_peered":"2024-09-11T08:26:55.725135+0000","last_clean":"2024-09-11T08:26:55.725135+0000","last_became_active":"2024-09-11T08:07:32.049368+0000","last_became_peered":"2024-09-11T08:07:32.049368+0000","last_unstale":"2024-09-11T08:26:55.725135+0000","last_undegraded":"2024-09-11T08:26:55.725135+0000","last_fullsized":"2024-09-11T08:26:55.725135+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'19","last_scrub_stamp":"2024-09-11T08:26:07.610413+0000","last_deep_scrub":"343'19","last_deep_scrub_stamp":"2024-09-11T08:23:44.640077+0000","last_clean_scrub_stamp":"2024-09-11T08:26:07.610413+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:17.842578+0000","scrub_duration":0.0020145950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":234,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.723364+0000","last_change":"2024-09-11T08:23:47.170953+0000","last_active":"2024-09-11T08:26:55.723364+0000","last_peered":"2024-09-11T08:26:55.723364+0000","last_clean":"2024-09-11T08:26:55.723364+0000","last_became_active":"2024-09-11T08:15:56.521570+0000","last_became_peered":"2024-09-11T08:15:56.521570+0000","last_unstale":"2024-09-11T08:26:55.723364+0000","last_undegraded":"2024-09-11T08:26:55.723364+0000","last_fullsized":"2024-09-11T08:26:55.723364+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:23:47.170848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:23:47.170848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0107554,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":199,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.733718+0000","last_change":"2024-09-11T08:25:53.178915+0000","last_active":"2024-09-11T08:26:55.733718+0000","last_peered":"2024-09-11T08:26:55.733718+0000","last_clean":"2024-09-11T08:26:55.733718+0000","last_became_active":"2024-09-11T08:17:37.557645+0000","last_became_peered":"2024-09-11T08:17:37.557645+0000","last_unstale":"2024-09-11T08:26:55.733718+0000","last_undegraded":"2024-09-11T08:26:55.733718+0000","last_fullsized":"2024-09-11T08:26:55.733718+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:22:02.568761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:22:02.568761+0000","last_clean_scrub_stamp":"2024-09-11T08:22:02.568761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0057637030000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":445,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-11T08:27:03.810772+0000","last_change":"2024-09-11T08:27:03.810727+0000","last_active":"2024-09-11T08:27:03.810772+0000","last_peered":"2024-09-11T08:27:03.810772+0000","last_clean":"2024-09-11T08:27:03.810772+0000","last_became_active":"2024-09-11T08:24:28.363959+0000","last_became_peered":"2024-09-11T08:24:28.363959+0000","last_unstale":"2024-09-11T08:27:03.810772+0000","last_undegraded":"2024-09-11T08:27:03.810772+0000","last_fullsized":"2024-09-11T08:27:03.810772+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:03.810647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:09.973943+0000","last_clean_scrub_stamp":"2024-09-11T08:27:03.810647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:25.890757+0000","scrub_duration":0.0051903160000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"239'19","reported_seq":512,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:58.609593+0000","last_change":"2024-09-11T08:26:58.609528+0000","last_active":"2024-09-11T08:26:58.609593+0000","last_peered":"2024-09-11T08:26:58.609593+0000","last_clean":"2024-09-11T08:26:58.609593+0000","last_became_active":"2024-09-11T08:15:56.520058+0000","last_became_peered":"2024-09-11T08:15:56.520058+0000","last_unstale":"2024-09-11T08:26:58.609593+0000","last_undegraded":"2024-09-11T08:26:58.609593+0000","last_fullsized":"2024-09-11T08:26:58.609593+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'19","last_scrub_stamp":"2024-09-11T08:26:58.609443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:26:58.609443+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:18.718575+0000","scrub_duration":0.017621226,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":337,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.723195+0000","last_change":"2024-09-11T08:24:28.951646+0000","last_active":"2024-09-11T08:26:55.723195+0000","last_peered":"2024-09-11T08:26:55.723195+0000","last_clean":"2024-09-11T08:26:55.723195+0000","last_became_active":"2024-09-11T08:24:28.356974+0000","last_became_peered":"2024-09-11T08:24:28.356974+0000","last_unstale":"2024-09-11T08:26:55.723195+0000","last_undegraded":"2024-09-11T08:26:55.723195+0000","last_fullsized":"2024-09-11T08:26:55.723195+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:28.951592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:24:28.951592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001286382,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":244,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.722960+0000","last_change":"2024-09-11T08:26:03.094530+0000","last_active":"2024-09-11T08:26:55.722960+0000","last_peered":"2024-09-11T08:26:55.722960+0000","last_clean":"2024-09-11T08:26:55.722960+0000","last_became_active":"2024-09-11T08:15:56.523728+0000","last_became_peered":"2024-09-11T08:15:56.523728+0000","last_unstale":"2024-09-11T08:26:55.722960+0000","last_undegraded":"2024-09-11T08:26:55.722960+0000","last_fullsized":"2024-09-11T08:26:55.722960+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:03.094474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:11:34.159053+0000","last_clean_scrub_stamp":"2024-09-11T08:26:03.094474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:16.180366+0000","scrub_duration":0.001235589,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":416,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.730416+0000","last_change":"2024-09-11T08:26:06.683231+0000","last_active":"2024-09-11T08:26:55.730416+0000","last_peered":"2024-09-11T08:26:55.730416+0000","last_clean":"2024-09-11T08:26:55.730416+0000","last_became_active":"2024-09-11T08:07:36.633368+0000","last_became_peered":"2024-09-11T08:07:36.633368+0000","last_unstale":"2024-09-11T08:26:55.730416+0000","last_undegraded":"2024-09-11T08:26:55.730416+0000","last_fullsized":"2024-09-11T08:26:55.730416+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:06.683182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:22:50.992034+0000","last_clean_scrub_stamp":"2024-09-11T08:26:06.683182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:28.391283+0000","scrub_duration":0.001432284,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"312'10","reported_seq":424,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.732260+0000","last_change":"2024-09-11T08:23:31.252251+0000","last_active":"2024-09-11T08:26:55.732260+0000","last_peered":"2024-09-11T08:26:55.732260+0000","last_clean":"2024-09-11T08:26:55.732260+0000","last_became_active":"2024-09-11T08:07:32.045728+0000","last_became_peered":"2024-09-11T08:07:32.045728+0000","last_unstale":"2024-09-11T08:26:55.732260+0000","last_undegraded":"2024-09-11T08:26:55.732260+0000","last_fullsized":"2024-09-11T08:26:55.732260+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"312'10","last_scrub_stamp":"2024-09-11T08:23:31.252174+0000","last_deep_scrub":"312'10","last_deep_scrub_stamp":"2024-09-11T08:21:55.859866+0000","last_clean_scrub_stamp":"2024-09-11T08:23:31.252174+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.021596985999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":448,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.733856+0000","last_change":"2024-09-11T08:24:28.358269+0000","last_active":"2024-09-11T08:26:55.733856+0000","last_peered":"2024-09-11T08:26:55.733856+0000","last_clean":"2024-09-11T08:26:55.733856+0000","last_became_active":"2024-09-11T08:24:28.357978+0000","last_became_peered":"2024-09-11T08:24:28.357978+0000","last_unstale":"2024-09-11T08:26:55.733856+0000","last_undegraded":"2024-09-11T08:26:55.733856+0000","last_fullsized":"2024-09-11T08:26:55.733856+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:22:48.784020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:22:48.784020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0083483829999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":474,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.728033+0000","last_change":"2024-09-11T08:24:09.590462+0000","last_active":"2024-09-11T08:26:55.728033+0000","last_peered":"2024-09-11T08:26:55.728033+0000","last_clean":"2024-09-11T08:26:55.728033+0000","last_became_active":"2024-09-11T08:15:56.528607+0000","last_became_peered":"2024-09-11T08:15:56.528607+0000","last_unstale":"2024-09-11T08:26:55.728033+0000","last_undegraded":"2024-09-11T08:26:55.728033+0000","last_fullsized":"2024-09-11T08:26:55.728033+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:09.590401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:24:09.590401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016035050000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":227,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.723025+0000","last_change":"2024-09-11T08:25:54.968708+0000","last_active":"2024-09-11T08:26:55.723025+0000","last_peered":"2024-09-11T08:26:55.723025+0000","last_clean":"2024-09-11T08:26:55.723025+0000","last_became_active":"2024-09-11T08:15:56.521488+0000","last_became_peered":"2024-09-11T08:15:56.521488+0000","last_unstale":"2024-09-11T08:26:55.723025+0000","last_undegraded":"2024-09-11T08:26:55.723025+0000","last_fullsized":"2024-09-11T08:26:55.723025+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:22:40.715019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:20:18.827709+0000","last_clean_scrub_stamp":"2024-09-11T08:22:40.715019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0097997290000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":455,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.725510+0000","last_change":"2024-09-11T08:25:49.449719+0000","last_active":"2024-09-11T08:26:55.725510+0000","last_peered":"2024-09-11T08:26:55.725510+0000","last_clean":"2024-09-11T08:26:55.725510+0000","last_became_active":"2024-09-11T08:24:28.354141+0000","last_became_peered":"2024-09-11T08:24:28.354141+0000","last_unstale":"2024-09-11T08:26:55.725510+0000","last_undegraded":"2024-09-11T08:26:55.725510+0000","last_fullsized":"2024-09-11T08:26:55.725510+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:22:45.821971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:50.585566+0000","last_clean_scrub_stamp":"2024-09-11T08:22:45.821971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041247130000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":196,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.733204+0000","last_change":"2024-09-11T08:23:01.737938+0000","last_active":"2024-09-11T08:26:55.733204+0000","last_peered":"2024-09-11T08:26:55.733204+0000","last_clean":"2024-09-11T08:26:55.733204+0000","last_became_active":"2024-09-11T08:17:37.554262+0000","last_became_peered":"2024-09-11T08:17:37.554262+0000","last_unstale":"2024-09-11T08:26:55.733204+0000","last_undegraded":"2024-09-11T08:26:55.733204+0000","last_fullsized":"2024-09-11T08:26:55.733204+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:23:01.737888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:23:01.737888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020103790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"288'24","reported_seq":536,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:56.653395+0000","last_change":"2024-09-11T08:26:56.653326+0000","last_active":"2024-09-11T08:26:56.653395+0000","last_peered":"2024-09-11T08:26:56.653395+0000","last_clean":"2024-09-11T08:26:56.653395+0000","last_became_active":"2024-09-11T08:24:28.360937+0000","last_became_peered":"2024-09-11T08:24:28.360937+0000","last_unstale":"2024-09-11T08:26:56.653395+0000","last_undegraded":"2024-09-11T08:26:56.653395+0000","last_fullsized":"2024-09-11T08:26:56.653395+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'24","last_scrub_stamp":"2024-09-11T08:26:56.653232+0000","last_deep_scrub":"288'24","last_deep_scrub_stamp":"2024-09-11T08:20:28.885295+0000","last_clean_scrub_stamp":"2024-09-11T08:26:56.653232+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:04.332641+0000","scrub_duration":0.0098689580000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":262,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-11T08:26:55.726741+0000","last_change":"2024-09-11T08:24:07.560376+0000","last_active":"2024-09-11T08:26:55.726741+0000","last_peered":"2024-09-11T08:26:55.726741+0000","last_clean":"2024-09-11T08:26:55.726741+0000","last_became_active":"2024-09-11T08:15:08.769133+0000","last_became_peered":"2024-09-11T08:15:08.769133+0000","last_unstale":"2024-09-11T08:26:55.726741+0000","last_undegraded":"2024-09-11T08:26:55.726741+0000","last_fullsized":"2024-09-11T08:26:55.726741+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:07.560293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:18:39.206016+0000","last_clean_scrub_stamp":"2024-09-11T08:24:07.560293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0083464200000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":3274506240,"num_objects":49965,"num_object_clones":0,"num_object_copies":149895,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49965,"num_write_kb":3197760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9791209472,"data_stored":9791209472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49965,"ondisk_log_size":49965,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102,"ondisk_log_size":102,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1117,"num_read_kb":874,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":4566,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3425,"internal_metadata":0},"log_size":772,"ondisk_log_size":772,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":764,"num_read_kb":759,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":168,"seq":721554505881,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1759760,"kb_used_data":1145544,"kb_used_omap":7,"kb_used_meta":614200,"kb_avail":91985392,"statfs":{"total":95995035648,"available":94193041408,"internally_reserved":0,"allocated":1173037056,"data_stored":1172594922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8027,"internal_metadata":628940965},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:26:37 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":3.9620000000000002,"15min":2.3740000000000001},"min":{"1min":0.24399999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.79300000000000004,"5min":117.57599999999999,"15min":117.57599999999999},"last":1.0580000000000001},{"interface":"front","average":{"1min":0.52900000000000003,"5min":3.7069999999999999,"15min":2.29},"min":{"1min":0.317,"5min":0.22800000000000001,"15min":0.219},"max":{"1min":1,"5min":117.542,"15min":117.542},"last":0.79000000000000004}]},{"osd":1,"last update":"Wed Sep 11 08:26:37 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":1.1200000000000001,"15min":1.9790000000000001},"min":{"1min":0.27200000000000002,"5min":0.24099999999999999,"15min":0.20899999999999999},"max":{"1min":0.61899999999999999,"5min":13.252000000000001,"15min":66.971000000000004},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.70299999999999996,"15min":1.7769999999999999},"min":{"1min":0.28000000000000003,"5min":0.25800000000000001,"15min":0.23000000000000001},"max":{"1min":0.69599999999999995,"5min":3.5510000000000002,"15min":67.185000000000002},"last":0.39000000000000001}]},{"osd":2,"last update":"Wed Sep 11 08:26:37 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":1.0920000000000001,"15min":1.028},"min":{"1min":0.29299999999999998,"5min":0.23799999999999999,"15min":0.22700000000000001},"max":{"1min":2.1840000000000002,"5min":19.097000000000001,"15min":19.097000000000001},"last":0.374},{"interface":"front","average":{"1min":0.58099999999999996,"5min":1.3879999999999999,"15min":1.0609999999999999},"min":{"1min":0.33400000000000002,"5min":0.255,"15min":0.23699999999999999},"max":{"1min":2.2080000000000002,"5min":21.594000000000001,"15min":21.594000000000001},"last":0.315}]},{"osd":3,"last update":"Wed Sep 11 08:26:37 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.86299999999999999,"15min":1.083},"min":{"1min":0.33300000000000002,"5min":0.26600000000000001,"15min":0.251},"max":{"1min":0.65900000000000003,"5min":6.375,"15min":21.460000000000001},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.86199999999999999,"15min":1.2},"min":{"1min":0.309,"5min":0.27200000000000002,"15min":0.251},"max":{"1min":0.97499999999999998,"5min":13.228999999999999,"15min":21.440000000000001},"last":0.40999999999999998}]},{"osd":4,"last update":"Wed Sep 11 08:26:37 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":1.002,"15min":0.82799999999999996},"min":{"1min":0.30399999999999999,"5min":0.25800000000000001,"15min":0.252},"max":{"1min":1.966,"5min":5.7240000000000002,"15min":5.7240000000000002},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.83199999999999996,"15min":0.79300000000000004},"min":{"1min":0.28899999999999998,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.73899999999999999,"5min":4.2060000000000004,"15min":5.2649999999999997},"last":0.35499999999999998}]},{"osd":5,"last update":"Wed Sep 11 08:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":1.2969999999999999,"15min":1.379},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":5.8360000000000003,"5min":5.8360000000000003,"15min":5.8360000000000003},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.88900000000000001,"15min":0.93500000000000005},"min":{"1min":0.33700000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":2.75,"5min":2.75,"15min":2.75},"last":0.42999999999999999}]},{"osd":6,"last update":"Wed Sep 11 08:26:37 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.90100000000000002,"15min":0.83499999999999996},"min":{"1min":0.36799999999999999,"5min":0.26900000000000002,"15min":0.248},"max":{"1min":0.95099999999999996,"5min":5.3230000000000004,"15min":7.1699999999999999},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.66100000000000003,"5min":1.032,"15min":0.93000000000000005},"min":{"1min":0.42399999999999999,"5min":0.315,"15min":0.25700000000000001},"max":{"1min":1.8600000000000001,"5min":6.0439999999999996,"15min":6.0439999999999996},"last":0.626}]}]},{"osd":6,"up_from":45,"seq":193273528564,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2152512,"kb_used_data":1566776,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":91592640,"statfs":{"total":95995035648,"available":93790863360,"internally_reserved":0,"allocated":1604378624,"data_stored":1603952697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:26:28 2024","interfaces":[{"interface":"back","average":{"1min":17.887,"5min":6.7610000000000001,"15min":3.4820000000000002},"min":{"1min":0.224,"5min":0.224,"15min":0.17999999999999999},"max":{"1min":331.14800000000002,"5min":331.14800000000002,"15min":331.14800000000002},"last":0.60499999999999998},{"interface":"front","average":{"1min":17.945,"5min":6.2949999999999999,"15min":3.2559999999999998},"min":{"1min":0.35099999999999998,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":331.31,"5min":331.31,"15min":331.31},"last":0.47799999999999998}]},{"osd":1,"last update":"Wed Sep 11 08:26:28 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":1.1719999999999999,"15min":1.4199999999999999},"min":{"1min":0.33500000000000002,"5min":0.223,"15min":0.214},"max":{"1min":3.3220000000000001,"5min":10.775,"15min":44.658000000000001},"last":0.627},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.95699999999999996,"15min":1.3919999999999999},"min":{"1min":0.252,"5min":0.22700000000000001,"15min":0.22600000000000001},"max":{"1min":2.3050000000000002,"5min":8.4109999999999996,"15min":44.694000000000003},"last":0.74099999999999999}]},{"osd":2,"last update":"Wed Sep 11 08:26:28 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":1.7270000000000001,"15min":1.165},"min":{"1min":0.32700000000000001,"5min":0.23799999999999999,"15min":0.23200000000000001},"max":{"1min":2.4209999999999998,"5min":39.652999999999999,"15min":39.652999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.63700000000000001,"5min":1.748,"15min":1.383},"min":{"1min":0.379,"5min":0.26100000000000001,"15min":0.23300000000000001},"max":{"1min":2.085,"5min":38.390999999999998,"15min":38.390999999999998},"last":0.52800000000000002}]},{"osd":3,"last update":"Wed Sep 11 08:26:28 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":1.377,"15min":1.3839999999999999},"min":{"1min":0.35399999999999998,"5min":0.253,"15min":0.22700000000000001},"max":{"1min":3.2970000000000002,"5min":10.144,"15min":36.061},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.621,"5min":1.1499999999999999,"15min":1.3129999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28399999999999997},"max":{"1min":2.3580000000000001,"5min":8.4399999999999995,"15min":36.093000000000004},"last":0.72099999999999997}]},{"osd":4,"last update":"Wed Sep 11 08:26:28 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.85599999999999998,"15min":0.69999999999999996},"min":{"1min":0.26800000000000002,"5min":0.20599999999999999,"15min":0.20499999999999999},"max":{"1min":2.1190000000000002,"5min":9.7560000000000002,"15min":9.7560000000000002},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.78700000000000003,"15min":0.68400000000000005},"min":{"1min":0.32000000000000001,"5min":0.26100000000000001,"15min":0.23699999999999999},"max":{"1min":2.4039999999999999,"5min":4.181,"15min":4.468},"last":0.54800000000000004}]},{"osd":5,"last update":"Wed Sep 11 08:26:28 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":1.6200000000000001,"15min":1.766},"min":{"1min":0.375,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":2.1400000000000001,"5min":9.7989999999999995,"15min":9.7989999999999995},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.76100000000000001,"5min":1.429,"15min":1.54},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":3.2040000000000002,"5min":7.2999999999999998,"15min":7.2999999999999998},"last":0.69099999999999995}]},{"osd":7,"last update":"Wed Sep 11 08:26:37 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":1.2270000000000001,"15min":1.0429999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.253},"max":{"1min":3.8780000000000001,"5min":10.167999999999999,"15min":10.167999999999999},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.85999999999999999,"15min":0.80500000000000005},"min":{"1min":0.35499999999999998,"5min":0.28399999999999997,"15min":0.25700000000000001},"max":{"1min":0.74299999999999999,"5min":5.3579999999999997,"15min":5.3579999999999997},"last":1.5169999999999999}]}]},{"osd":5,"up_from":353,"seq":1516123455520,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1114432,"kb_used_data":758644,"kb_used_omap":4,"kb_used_meta":355771,"kb_avail":92630720,"statfs":{"total":95995035648,"available":94853857280,"internally_reserved":0,"allocated":776851456,"data_stored":776428155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":364310022},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:26:31 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":1.6990000000000001,"15min":1.835},"min":{"1min":0.32500000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":7.6870000000000003,"5min":19.379999999999999,"15min":19.379999999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.89600000000000002,"5min":1.673,"15min":1.8029999999999999},"min":{"1min":0.29299999999999998,"5min":0.218,"15min":0.218},"max":{"1min":8.202,"5min":18.238,"15min":18.238},"last":0.24199999999999999}]},{"osd":1,"last update":"Wed Sep 11 08:26:31 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.998,"15min":1.0820000000000001},"min":{"1min":0.30599999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.76400000000000001,"5min":5.3170000000000002,"15min":5.3170000000000002},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.66400000000000003,"15min":0.69299999999999995},"min":{"1min":0.34599999999999997,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.71499999999999997,"5min":2.6280000000000001,"15min":2.6280000000000001},"last":0.42799999999999999}]},{"osd":2,"last update":"Wed Sep 11 08:26:31 2024","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":1.879,"15min":2.036},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":8.8000000000000007,"5min":18.350999999999999,"15min":18.350999999999999},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.93999999999999995,"5min":1.748,"15min":1.8819999999999999},"min":{"1min":0.32200000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":8.1910000000000007,"5min":18.381,"15min":18.381},"last":0.37}]},{"osd":3,"last update":"Wed Sep 11 08:26:31 2024","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":1.113,"15min":1.153},"min":{"1min":0.33600000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":7.2290000000000001,"5min":7.2290000000000001,"15min":7.2290000000000001},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.86399999999999999,"15min":0.85799999999999998},"min":{"1min":0.39200000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":7.4109999999999996,"5min":7.4109999999999996,"15min":7.4109999999999996},"last":0.47699999999999998}]},{"osd":4,"last update":"Wed Sep 11 08:26:31 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":1.0880000000000001,"15min":1.1930000000000001},"min":{"1min":0.26700000000000002,"5min":0.253,"15min":0.253},"max":{"1min":1.0629999999999999,"5min":5.7670000000000003,"15min":5.7670000000000003},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.67200000000000004,"5min":1.181,"15min":1.266},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":4.093,"5min":5.8979999999999997,"15min":5.8979999999999997},"last":0.443}]},{"osd":6,"last update":"Wed Sep 11 08:26:31 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":1.3899999999999999,"15min":1.506},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":4.0490000000000004,"5min":5.9779999999999998,"15min":5.9779999999999998},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.47199999999999998,"5min":1.639,"15min":1.833},"min":{"1min":0.34999999999999998,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.66100000000000003,"5min":5.6680000000000001,"15min":5.6680000000000001},"last":0.41199999999999998}]},{"osd":7,"last update":"Wed Sep 11 08:26:31 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":1.1060000000000001,"15min":1.212},"min":{"1min":0.308,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.65200000000000002,"5min":4.9699999999999998,"15min":4.9699999999999998},"last":0.317},{"interface":"front","average":{"1min":0.54300000000000004,"5min":1.5820000000000001,"15min":1.7549999999999999},"min":{"1min":0.36299999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.749,"5min":5.8529999999999998,"15min":5.8529999999999998},"last":0.46500000000000002}]}]},{"osd":4,"up_from":31,"seq":133143986430,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1350488,"kb_used_data":764748,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":92394664,"statfs":{"total":95995035648,"available":94612135936,"internally_reserved":0,"allocated":783101952,"data_stored":782655876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:26:42 2024","interfaces":[{"interface":"back","average":{"1min":3.6859999999999999,"5min":2.706,"15min":1.8899999999999999},"min":{"1min":0.251,"5min":0.217,"15min":0.19},"max":{"1min":64.329999999999998,"5min":64.329999999999998,"15min":64.329999999999998},"last":0.84499999999999997},{"interface":"front","average":{"1min":3.79,"5min":2.9609999999999999,"15min":1.891},"min":{"1min":0.33300000000000002,"5min":0.24399999999999999,"15min":0.221},"max":{"1min":66.819999999999993,"5min":66.819999999999993,"15min":66.819999999999993},"last":0.47199999999999998}]},{"osd":1,"last update":"Wed Sep 11 08:26:42 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":1.101,"15min":1.1619999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.214},"max":{"1min":0.66600000000000004,"5min":35.156999999999996,"15min":35.156999999999996},"last":3.1560000000000001},{"interface":"front","average":{"1min":0.46899999999999997,"5min":1.456,"15min":1.24},"min":{"1min":0.30299999999999999,"5min":0.23599999999999999,"15min":0.22900000000000001},"max":{"1min":0.68100000000000005,"5min":35.088999999999999,"15min":35.088999999999999},"last":0.64300000000000002}]},{"osd":2,"last update":"Wed Sep 11 08:26:42 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":2.0169999999999999,"15min":1.397},"min":{"1min":0.33400000000000002,"5min":0.25800000000000001,"15min":0.23400000000000001},"max":{"1min":0.72299999999999998,"5min":54.317999999999998,"15min":54.317999999999998},"last":0.91500000000000004},{"interface":"front","average":{"1min":0.48699999999999999,"5min":2.2749999999999999,"15min":1.446},"min":{"1min":0.30299999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.75900000000000001,"5min":54.430999999999997,"15min":54.430999999999997},"last":1.1899999999999999}]},{"osd":3,"last update":"Wed Sep 11 08:26:42 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":1.2190000000000001,"15min":1.3149999999999999},"min":{"1min":0.29099999999999998,"5min":0.27900000000000003,"15min":0.253},"max":{"1min":0.73599999999999999,"5min":30.321000000000002,"15min":30.321000000000002},"last":5.306},{"interface":"front","average":{"1min":0.52700000000000002,"5min":1.0740000000000001,"15min":1.194},"min":{"1min":0.28399999999999997,"5min":0.26500000000000001,"15min":0.246},"max":{"1min":1.1739999999999999,"5min":30.620000000000001,"15min":30.620000000000001},"last":1.4159999999999999}]},{"osd":5,"last update":"Wed Sep 11 08:26:29 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":1.226,"15min":1.3160000000000001},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":4.5869999999999997,"5min":5.1050000000000004,"15min":5.1050000000000004},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.421,"15min":1.4930000000000001},"min":{"1min":0.30099999999999999,"5min":0.255,"15min":0.255},"max":{"1min":5.5750000000000002,"5min":6.1150000000000002,"15min":6.1150000000000002},"last":0.51600000000000001}]},{"osd":6,"last update":"Wed Sep 11 08:26:29 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.84299999999999997,"15min":0.83499999999999996},"min":{"1min":0.28199999999999997,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":5.2640000000000002,"5min":5.2640000000000002,"15min":48.808},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.94899999999999995,"5min":0.90700000000000003,"15min":0.88},"min":{"1min":0.309,"5min":0.251,"15min":0.251},"max":{"1min":5.3159999999999998,"5min":5.3159999999999998,"15min":48.746000000000002},"last":0.88600000000000001}]},{"osd":7,"last update":"Wed Sep 11 08:26:54 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.85699999999999998,"15min":0.70099999999999996},"min":{"1min":0.29799999999999999,"5min":0.249,"15min":0.249},"max":{"1min":1.7070000000000001,"5min":6.093,"15min":6.093},"last":1.3100000000000001},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.67300000000000004,"15min":0.73799999999999999},"min":{"1min":0.32900000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.88500000000000001,"5min":4.8259999999999996,"15min":4.8259999999999996},"last":0.60899999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182659,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2269796,"kb_used_data":1364440,"kb_used_omap":1,"kb_used_meta":905342,"kb_avail":91475356,"statfs":{"total":95995035648,"available":93670764544,"internally_reserved":0,"allocated":1397186560,"data_stored":1396727667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":927070552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:26:30 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":1.5109999999999999,"15min":1.875},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.14999999999999999},"max":{"1min":4.298,"5min":24.616,"15min":60.469999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.76600000000000001,"5min":1.2609999999999999,"15min":1.889},"min":{"1min":0.26300000000000001,"5min":0.20599999999999999,"15min":0.17399999999999999},"max":{"1min":4.375,"5min":23.640000000000001,"15min":60.488999999999997},"last":0.58699999999999997}]},{"osd":1,"last update":"Wed Sep 11 08:26:30 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":1.2130000000000001,"15min":1.619},"min":{"1min":0.20399999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":4.1520000000000001,"5min":33.402000000000001,"15min":60.578000000000003},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.88100000000000001,"15min":1.7390000000000001},"min":{"1min":0.20899999999999999,"5min":0.19700000000000001,"15min":0.19500000000000001},"max":{"1min":4.0979999999999999,"5min":5.8040000000000003,"15min":63.674999999999997},"last":4.306}]},{"osd":2,"last update":"Wed Sep 11 08:26:30 2024","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":1.7789999999999999,"15min":1.877},"min":{"1min":0.23999999999999999,"5min":0.219,"15min":0.185},"max":{"1min":7.665,"5min":67.503,"15min":67.503},"last":4.1390000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":1.778,"15min":1.7549999999999999},"min":{"1min":0.251,"5min":0.24299999999999999,"15min":0.19900000000000001},"max":{"1min":4.4550000000000001,"5min":67.313999999999993,"15min":67.313999999999993},"last":3.194}]},{"osd":4,"last update":"Wed Sep 11 08:26:37 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":1.113,"15min":1.46},"min":{"1min":0.32900000000000001,"5min":0.29199999999999998,"15min":0.24299999999999999},"max":{"1min":0.753,"5min":4.274,"15min":61.290999999999997},"last":4.2149999999999999},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.90500000000000003,"15min":1.2450000000000001},"min":{"1min":0.32500000000000001,"5min":0.29299999999999998,"15min":0.27500000000000002},"max":{"1min":0.73399999999999999,"5min":5.3259999999999996,"15min":60.856000000000002},"last":0.47899999999999998}]},{"osd":5,"last update":"Wed Sep 11 08:26:30 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":1.577,"15min":1.7549999999999999},"min":{"1min":0.36399999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.64300000000000002,"5min":7.3849999999999998,"15min":7.3849999999999998},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":1.109,"15min":1.198},"min":{"1min":0.38600000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.879,"5min":3.508,"15min":3.508},"last":4.1820000000000004}]},{"osd":6,"last update":"Wed Sep 11 08:26:30 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":1.246,"15min":1.631},"min":{"1min":0.32700000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":4.3289999999999997,"5min":6.2850000000000001,"15min":60.817},"last":0.432},{"interface":"front","average":{"1min":0.54600000000000004,"5min":1.1180000000000001,"15min":1.73},"min":{"1min":0.35599999999999998,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.90100000000000002,"5min":4.407,"15min":61.264000000000003},"last":4.2409999999999997}]},{"osd":7,"last update":"Wed Sep 11 08:26:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.90100000000000002,"15min":1.962},"min":{"1min":0.374,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.86399999999999999,"5min":4.0090000000000003,"15min":60.847000000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.98299999999999998,"15min":2.089},"min":{"1min":0.374,"5min":0.34499999999999997,"15min":0.33800000000000002},"max":{"1min":1.272,"5min":4.5220000000000002,"15min":60.883000000000003},"last":0.51800000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411590,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1773120,"kb_used_data":1179064,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":91972032,"statfs":{"total":95995035648,"available":94179360768,"internally_reserved":0,"allocated":1207361536,"data_stored":1206934538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:26:05 2024","interfaces":[{"interface":"back","average":{"1min":5.5800000000000001,"5min":4.5629999999999997,"15min":2.8130000000000002},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17199999999999999},"max":{"1min":81.655000000000001,"5min":93.971000000000004,"15min":93.971000000000004},"last":0.40999999999999998},{"interface":"front","average":{"1min":5.8769999999999998,"5min":4.8310000000000004,"15min":3.1459999999999999},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.17999999999999999},"max":{"1min":81.391000000000005,"5min":93.790999999999997,"15min":93.790999999999997},"last":2.6579999999999999}]},{"osd":1,"last update":"Wed Sep 11 08:26:05 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":1.6819999999999999,"15min":1.657},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.17599999999999999},"max":{"1min":1.347,"5min":53.347000000000001,"15min":53.347000000000001},"last":2.605},{"interface":"front","average":{"1min":0.56999999999999995,"5min":1.764,"15min":1.665},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.1269999999999998,"5min":53.082999999999998,"15min":53.082999999999998},"last":1.4330000000000001}]},{"osd":3,"last update":"Wed Sep 11 08:26:26 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":1.2749999999999999,"15min":1.544},"min":{"1min":0.22,"5min":0.22,"15min":0.188},"max":{"1min":1.1579999999999999,"5min":8.0630000000000006,"15min":46.070999999999998},"last":0.437},{"interface":"front","average":{"1min":0.44400000000000001,"5min":1.3740000000000001,"15min":1.516},"min":{"1min":0.255,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":1.3149999999999999,"5min":7.827,"15min":45.790999999999997},"last":1.3500000000000001}]},{"osd":4,"last update":"Wed Sep 11 08:26:43 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":1.1539999999999999,"15min":1.0680000000000001},"min":{"1min":0.26700000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.871,"5min":5.9470000000000001,"15min":7.2480000000000002},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.51600000000000001,"5min":1.361,"15min":1.1120000000000001},"min":{"1min":0.34200000000000003,"5min":0.28699999999999998,"15min":0.252},"max":{"1min":0.72599999999999998,"5min":7.8970000000000002,"15min":7.8970000000000002},"last":1.393}]},{"osd":5,"last update":"Wed Sep 11 08:26:28 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":1.5049999999999999,"15min":1.6759999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.70299999999999996,"5min":3.4620000000000002,"15min":3.4620000000000002},"last":1.3779999999999999},{"interface":"front","average":{"1min":0.58899999999999997,"5min":1.3,"15min":1.4179999999999999},"min":{"1min":0.311,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.629,"5min":3.9039999999999999,"15min":3.9039999999999999},"last":0.625}]},{"osd":6,"last update":"Wed Sep 11 08:26:34 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":1.341,"15min":1.1499999999999999},"min":{"1min":0.30299999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.452,"5min":8.0350000000000001,"15min":8.0350000000000001},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":1.381,"15min":1.2210000000000001},"min":{"1min":0.375,"5min":0.35099999999999998,"15min":0.32500000000000001},"max":{"1min":0.85699999999999998,"5min":7.8710000000000004,"15min":7.8710000000000004},"last":1.448}]},{"osd":7,"last update":"Wed Sep 11 08:26:43 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":1.087,"15min":1.3069999999999999},"min":{"1min":0.312,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.83799999999999997,"5min":4.7300000000000004,"15min":35.274999999999999},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.58699999999999997,"5min":1.27,"15min":1.4370000000000001},"min":{"1min":0.38800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.82599999999999996,"5min":7.9219999999999997,"15min":35.265000000000001},"last":1.4099999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575116,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2460068,"kb_used_data":1573980,"kb_used_omap":1,"kb_used_meta":886078,"kb_avail":91285084,"statfs":{"total":95995035648,"available":93475926016,"internally_reserved":0,"allocated":1611755520,"data_stored":1611292674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":907344216},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:26:42 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":1.008,"15min":1.4990000000000001},"min":{"1min":0.26200000000000001,"5min":0.16700000000000001,"15min":0.16200000000000001},"max":{"1min":0.99399999999999999,"5min":6.9100000000000001,"15min":63.932000000000002},"last":2.4409999999999998},{"interface":"front","average":{"1min":0.48699999999999999,"5min":1.4550000000000001,"15min":1.8700000000000001},"min":{"1min":0.25,"5min":0.182,"15min":0.16800000000000001},"max":{"1min":1.4910000000000001,"5min":13.023999999999999,"15min":64.269000000000005},"last":0.20100000000000001}]},{"osd":2,"last update":"Wed Sep 11 08:26:19 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":1.2090000000000001,"15min":1.3879999999999999},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.189},"max":{"1min":0.88600000000000001,"5min":20.925999999999998,"15min":20.925999999999998},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.52300000000000002,"5min":1.2290000000000001,"15min":1.452},"min":{"1min":0.23899999999999999,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":2.4089999999999998,"5min":20.812000000000001,"15min":20.812000000000001},"last":0.314}]},{"osd":3,"last update":"Wed Sep 11 08:26:30 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.86499999999999999,"15min":1.643},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19700000000000001},"max":{"1min":0.71699999999999997,"5min":8.0579999999999998,"15min":16.783999999999999},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.96499999999999997,"15min":1.5900000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22600000000000001},"max":{"1min":6.5010000000000003,"5min":6.5010000000000003,"15min":14.973000000000001},"last":0.877}]},{"osd":4,"last update":"Wed Sep 11 08:26:49 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":1.1140000000000001,"15min":1.772},"min":{"1min":0.36199999999999999,"5min":0.254,"15min":0.252},"max":{"1min":1.1899999999999999,"5min":7.4850000000000003,"15min":110.86499999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.622,"5min":1.2969999999999999,"15min":1.8069999999999999},"min":{"1min":0.41799999999999998,"5min":0.28399999999999997,"15min":0.248},"max":{"1min":1.137,"5min":5.7119999999999997,"15min":110.831},"last":0.96199999999999997}]},{"osd":5,"last update":"Wed Sep 11 08:26:30 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":1.4450000000000001,"15min":1.579},"min":{"1min":0.35999999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.8600000000000001,"5min":4.1050000000000004,"15min":4.1050000000000004},"last":0.93799999999999994},{"interface":"front","average":{"1min":0.81100000000000005,"5min":1.4390000000000001,"15min":1.5429999999999999},"min":{"1min":0.38700000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":2.5920000000000001,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":0.44800000000000001}]},{"osd":6,"last update":"Wed Sep 11 08:26:30 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":1.256,"15min":1.8220000000000001},"min":{"1min":0.39900000000000002,"5min":0.315,"15min":0.29399999999999998},"max":{"1min":2.6579999999999999,"5min":8.1059999999999999,"15min":124.31999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.77900000000000003,"5min":1.1759999999999999,"15min":1.8140000000000001},"min":{"1min":0.38300000000000001,"5min":0.32300000000000001,"15min":0.29799999999999999},"max":{"1min":2.415,"5min":4.8170000000000002,"15min":124.958},"last":0.98299999999999998}]},{"osd":7,"last update":"Wed Sep 11 08:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":1.0800000000000001,"15min":1.6859999999999999},"min":{"1min":0.35499999999999998,"5min":0.318,"15min":0.28599999999999998},"max":{"1min":1.2250000000000001,"5min":7.5229999999999997,"15min":109.149},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.77500000000000002,"5min":1.0920000000000001,"15min":1.8089999999999999},"min":{"1min":0.372,"5min":0.32200000000000001,"15min":0.318},"max":{"1min":3.004,"5min":8.141,"15min":109.309},"last":0.38}]}]},{"osd":0,"up_from":8,"seq":34359738640,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1925020,"kb_used_data":1014292,"kb_used_omap":1,"kb_used_meta":910718,"kb_avail":91820132,"statfs":{"total":95995035648,"available":94023815168,"internally_reserved":0,"allocated":1038635008,"data_stored":1038181532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":932575576},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 11 08:26:36 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.89600000000000002,"15min":1.236},"min":{"1min":0.20599999999999999,"5min":0.159,"15min":0.159},"max":{"1min":3.7970000000000002,"5min":8.0370000000000008,"15min":10.105},"last":45.902000000000001},{"interface":"front","average":{"1min":0.64800000000000002,"5min":1.0669999999999999,"15min":1.5369999999999999},"min":{"1min":0.219,"5min":0.17299999999999999,"15min":0.153},"max":{"1min":4.0410000000000004,"5min":9.6799999999999997,"15min":19.852},"last":45.883000000000003}]},{"osd":2,"last update":"Wed Sep 11 08:27:02 2024","interfaces":[{"interface":"back","average":{"1min":2.5840000000000001,"5min":1.8720000000000001,"15min":1.4790000000000001},"min":{"1min":0.22800000000000001,"5min":0.19,"15min":0.19},"max":{"1min":46.417000000000002,"5min":46.417000000000002,"15min":46.417000000000002},"last":46.417000000000002},{"interface":"front","average":{"1min":2.625,"5min":1.8999999999999999,"15min":1.554},"min":{"1min":0.26000000000000001,"5min":0.22600000000000001,"15min":0.186},"max":{"1min":46.438000000000002,"5min":46.438000000000002,"15min":46.438000000000002},"last":46.438000000000002}]},{"osd":3,"last update":"Wed Sep 11 08:26:24 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":2.7200000000000002,"15min":2.1179999999999999},"min":{"1min":0.24399999999999999,"5min":0.24199999999999999,"15min":0.22900000000000001},"max":{"1min":3.8519999999999999,"5min":172.149,"15min":172.149},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.71199999999999997,"5min":2.7589999999999999,"15min":2.1429999999999998},"min":{"1min":0.26400000000000001,"5min":0.20599999999999999,"15min":0.20300000000000001},"max":{"1min":4.2709999999999999,"5min":174.899,"15min":174.899},"last":0.26100000000000001}]},{"osd":4,"last update":"Wed Sep 11 08:26:49 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":1.391,"15min":1.228},"min":{"1min":0.308,"5min":0.308,"15min":0.24099999999999999},"max":{"1min":0.89800000000000002,"5min":9.6560000000000006,"15min":9.6560000000000006},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":1.304,"15min":1.2130000000000001},"min":{"1min":0.315,"5min":0.26400000000000001,"15min":0.24399999999999999},"max":{"1min":0.79100000000000004,"5min":6.6100000000000003,"15min":8.3510000000000009},"last":0.54200000000000004}]},{"osd":5,"last update":"Wed Sep 11 08:26:30 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":1.7889999999999999,"15min":1.9470000000000001},"min":{"1min":0.317,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":3.8940000000000001,"5min":4.8220000000000001,"15min":4.8220000000000001},"last":0.625},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.663,"15min":1.7649999999999999},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":4.5090000000000003,"5min":7.718,"15min":7.718},"last":0.54900000000000004}]},{"osd":6,"last update":"Wed Sep 11 08:26:30 2024","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":1.4099999999999999,"15min":1.593},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.29099999999999998},"max":{"1min":4.2389999999999999,"5min":9.7029999999999994,"15min":78.522000000000006},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.97599999999999998,"5min":1.452,"15min":1.5},"min":{"1min":0.38100000000000001,"5min":0.33100000000000002,"15min":0.28299999999999997},"max":{"1min":4.5830000000000002,"5min":6.8150000000000004,"15min":79.808000000000007},"last":0.50900000000000001}]},{"osd":7,"last update":"Wed Sep 11 08:26:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":1.214,"15min":1.262},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.30399999999999999},"max":{"1min":0.95399999999999996,"5min":6.8520000000000003,"15min":7.5350000000000001},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.60299999999999998,"5min":1.3,"15min":1.353},"min":{"1min":0.40699999999999997,"5min":0.32900000000000001,"15min":0.32000000000000001},"max":{"1min":1.393,"5min":6.875,"15min":13.914},"last":0.64900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3425,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1050542080,"data_stored":1050542080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1651900416,"data_stored":1651900416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1225850880,"data_stored":1225850880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1437925376,"data_stored":1437925376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":804585472,"data_stored":804585472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":783089664,"data_stored":783089664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1623654400,"data_stored":1623654400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1213661184,"data_stored":1213661184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T08:27:10.906 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T08:27:10.970 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:11.031 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:11.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:11.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:11.133 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:11.162 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:11.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:11.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:11.225 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:10 smithi150 ceph-mon[29276]: 5.a scrub starts 2024-09-11T08:27:11.225 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:10 smithi150 ceph-mon[29276]: 5.a scrub ok 2024-09-11T08:27:11.225 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:10 smithi150 ceph-mon[29276]: 5.15 scrub starts 2024-09-11T08:27:11.225 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:10 smithi150 ceph-mon[29276]: 5.15 scrub ok 2024-09-11T08:27:11.225 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:10 smithi150 ceph-mon[29276]: 3.8 scrub starts 2024-09-11T08:27:11.225 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:10 smithi150 ceph-mon[29276]: 3.8 scrub ok 2024-09-11T08:27:11.225 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:10 smithi150 ceph-mon[29276]: osdmap e395: 8 total, 8 up, 8 in 2024-09-11T08:27:11.226 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:10 smithi150 ceph-mon[39612]: 5.a scrub starts 2024-09-11T08:27:11.226 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:10 smithi150 ceph-mon[39612]: 5.a scrub ok 2024-09-11T08:27:11.226 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:10 smithi150 ceph-mon[39612]: 5.15 scrub starts 2024-09-11T08:27:11.226 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:10 smithi150 ceph-mon[39612]: 5.15 scrub ok 2024-09-11T08:27:11.226 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:10 smithi150 ceph-mon[39612]: 3.8 scrub starts 2024-09-11T08:27:11.226 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:10 smithi150 ceph-mon[39612]: 3.8 scrub ok 2024-09-11T08:27:11.226 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:10 smithi150 ceph-mon[39612]: osdmap e395: 8 total, 8 up, 8 in 2024-09-11T08:27:11.265 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:11.296 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:11.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:11.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:11.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:10 smithi181 ceph-mon[32701]: 5.a scrub starts 2024-09-11T08:27:11.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:10 smithi181 ceph-mon[32701]: 5.a scrub ok 2024-09-11T08:27:11.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:10 smithi181 ceph-mon[32701]: 5.15 scrub starts 2024-09-11T08:27:11.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:10 smithi181 ceph-mon[32701]: 5.15 scrub ok 2024-09-11T08:27:11.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:10 smithi181 ceph-mon[32701]: 3.8 scrub starts 2024-09-11T08:27:11.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:10 smithi181 ceph-mon[32701]: 3.8 scrub ok 2024-09-11T08:27:11.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:10 smithi181 ceph-mon[32701]: osdmap e395: 8 total, 8 up, 8 in 2024-09-11T08:27:11.398 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:11.435 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:11.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:11.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:11.537 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:11.577 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:11.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:11.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:11.615 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 31 15 61739 61724 124.425 91.625 0.00361505 0.00803196 2024-09-11T08:27:11.679 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:11.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:11.709 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:11.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:11.810 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:11.857 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:11.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:11.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:11.961 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:12.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:12.006 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:12.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:12.107 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:12.137 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:12.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:12.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:12.175 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:11 smithi150 ceph-mon[29276]: 4.1 scrub starts 2024-09-11T08:27:12.175 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:11 smithi150 ceph-mon[29276]: 4.1 scrub ok 2024-09-11T08:27:12.175 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:11 smithi150 ceph-mon[29276]: 4.19 scrub starts 2024-09-11T08:27:12.176 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:11 smithi150 ceph-mon[29276]: 4.19 scrub ok 2024-09-11T08:27:12.176 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:11 smithi150 ceph-mon[29276]: 6.4 scrub starts 2024-09-11T08:27:12.176 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:11 smithi150 ceph-mon[29276]: 6.4 scrub ok 2024-09-11T08:27:12.176 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:11 smithi150 ceph-mon[29276]: pgmap v897: 148 pgs: 148 active+clean; 3.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 80 MiB/s wr, 1.27k op/s 2024-09-11T08:27:12.176 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:11 smithi150 ceph-mon[39612]: 4.1 scrub starts 2024-09-11T08:27:12.176 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:11 smithi150 ceph-mon[39612]: 4.1 scrub ok 2024-09-11T08:27:12.176 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:11 smithi150 ceph-mon[39612]: 4.19 scrub starts 2024-09-11T08:27:12.176 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:11 smithi150 ceph-mon[39612]: 4.19 scrub ok 2024-09-11T08:27:12.176 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:11 smithi150 ceph-mon[39612]: 6.4 scrub starts 2024-09-11T08:27:12.177 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:11 smithi150 ceph-mon[39612]: 6.4 scrub ok 2024-09-11T08:27:12.177 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:11 smithi150 ceph-mon[39612]: pgmap v897: 148 pgs: 148 active+clean; 3.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 80 MiB/s wr, 1.27k op/s 2024-09-11T08:27:12.239 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:12.269 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:12.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:12.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:11 smithi181 ceph-mon[32701]: 4.1 scrub starts 2024-09-11T08:27:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:11 smithi181 ceph-mon[32701]: 4.1 scrub ok 2024-09-11T08:27:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:11 smithi181 ceph-mon[32701]: 4.19 scrub starts 2024-09-11T08:27:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:11 smithi181 ceph-mon[32701]: 4.19 scrub ok 2024-09-11T08:27:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:11 smithi181 ceph-mon[32701]: 6.4 scrub starts 2024-09-11T08:27:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:11 smithi181 ceph-mon[32701]: 6.4 scrub ok 2024-09-11T08:27:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:11 smithi181 ceph-mon[32701]: pgmap v897: 148 pgs: 148 active+clean; 3.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 80 MiB/s wr, 1.27k op/s 2024-09-11T08:27:12.375 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:27:12.411 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:12.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:12.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:27:12.513 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:12.549 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:12.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:12.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:12.620 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 32 16 63137 63121 123.254 87.3125 0.0150462 0.00810767 2024-09-11T08:27:12.650 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:27:12.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:12.681 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:12.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:27:12.782 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:12.811 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:12.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:12.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:12.913 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:27:12.939 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:12.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:12.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:27:13.041 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:13.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:13.068 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:13.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:13.169 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:13.202 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:13.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:13.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:13.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:12 smithi150 ceph-mon[29276]: 6.f scrub starts 2024-09-11T08:27:13.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:12 smithi150 ceph-mon[29276]: 6.f scrub ok 2024-09-11T08:27:13.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:12 smithi150 ceph-mon[29276]: 6.7 scrub starts 2024-09-11T08:27:13.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:12 smithi150 ceph-mon[29276]: 6.7 scrub ok 2024-09-11T08:27:13.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:12 smithi150 ceph-mon[29276]: 4.10 scrub starts 2024-09-11T08:27:13.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:12 smithi150 ceph-mon[29276]: 4.10 scrub ok 2024-09-11T08:27:13.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:12 smithi150 ceph-mon[39612]: 6.f scrub starts 2024-09-11T08:27:13.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:12 smithi150 ceph-mon[39612]: 6.f scrub ok 2024-09-11T08:27:13.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:12 smithi150 ceph-mon[39612]: 6.7 scrub starts 2024-09-11T08:27:13.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:12 smithi150 ceph-mon[39612]: 6.7 scrub ok 2024-09-11T08:27:13.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:12 smithi150 ceph-mon[39612]: 4.10 scrub starts 2024-09-11T08:27:13.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:12 smithi150 ceph-mon[39612]: 4.10 scrub ok 2024-09-11T08:27:13.304 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:27:13.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:12 smithi181 ceph-mon[32701]: 6.f scrub starts 2024-09-11T08:27:13.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:12 smithi181 ceph-mon[32701]: 6.f scrub ok 2024-09-11T08:27:13.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:12 smithi181 ceph-mon[32701]: 6.7 scrub starts 2024-09-11T08:27:13.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:12 smithi181 ceph-mon[32701]: 6.7 scrub ok 2024-09-11T08:27:13.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:12 smithi181 ceph-mon[32701]: 4.10 scrub starts 2024-09-11T08:27:13.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:12 smithi181 ceph-mon[32701]: 4.10 scrub ok 2024-09-11T08:27:13.343 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:13.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:13.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:27:13.444 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:27:13.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:13.488 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:13.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:27:13.589 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:13.618 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 33 16 64406 64390 121.923 79.3125 0.0119159 0.00819668 2024-09-11T08:27:13.626 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:13.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:13.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:13.729 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:13.764 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:13.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:13.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:13.866 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:27:13.890 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:13.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:13.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:27:13.992 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:14.017 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:14.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:14.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:14.119 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:14.166 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:14.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:14.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:14.212 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:13 smithi150 ceph-mon[39612]: 4.0 scrub starts 2024-09-11T08:27:14.212 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:13 smithi150 ceph-mon[39612]: 4.0 scrub ok 2024-09-11T08:27:14.212 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:13 smithi150 ceph-mon[39612]: 5.d scrub starts 2024-09-11T08:27:14.212 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:13 smithi150 ceph-mon[39612]: 5.d scrub ok 2024-09-11T08:27:14.212 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:13 smithi150 ceph-mon[39612]: pgmap v898: 148 pgs: 148 active+clean; 3.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 690 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-09-11T08:27:14.213 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:13 smithi150 ceph-mon[29276]: 4.0 scrub starts 2024-09-11T08:27:14.213 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:13 smithi150 ceph-mon[29276]: 4.0 scrub ok 2024-09-11T08:27:14.213 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:13 smithi150 ceph-mon[29276]: 5.d scrub starts 2024-09-11T08:27:14.213 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:13 smithi150 ceph-mon[29276]: 5.d scrub ok 2024-09-11T08:27:14.213 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:13 smithi150 ceph-mon[29276]: pgmap v898: 148 pgs: 148 active+clean; 3.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 690 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-09-11T08:27:14.269 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:14.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:14.310 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:14.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:14.322 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:13 smithi181 ceph-mon[32701]: 4.0 scrub starts 2024-09-11T08:27:14.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:13 smithi181 ceph-mon[32701]: 4.0 scrub ok 2024-09-11T08:27:14.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:13 smithi181 ceph-mon[32701]: 5.d scrub starts 2024-09-11T08:27:14.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:13 smithi181 ceph-mon[32701]: 5.d scrub ok 2024-09-11T08:27:14.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:13 smithi181 ceph-mon[32701]: pgmap v898: 148 pgs: 148 active+clean; 3.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 690 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-09-11T08:27:14.398 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:27:14.398 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:27:14.411 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:14.421 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-09-11T08:27:14.421 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:27:14.431 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:27:14.431 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:27:14.452 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:14.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:14.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:14.463 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:27:14.463 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:27:14.495 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:27:14.496 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:27:14.528 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:27:14.528 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:27:14.554 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:27:14.558 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:27:14.558 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:27:14.584 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:14.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:14.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:27:14.588 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:27:14.588 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:27:14.618 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 34 16 65509 65493 120.364 68.9375 0.0115275 0.00830293 2024-09-11T08:27:14.620 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:27:14.621 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:27:14.686 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:14.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:14.725 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:14.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:14.826 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:14.864 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:14.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:14.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:14.960 INFO:teuthology.orchestra.run.smithi150.stderr:nodeep-scrub is unset 2024-09-11T08:27:14.966 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:14.983 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:14 smithi150 ceph-mon[29276]: 5.7 scrub starts 2024-09-11T08:27:14.984 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:14 smithi150 ceph-mon[29276]: 5.7 scrub ok 2024-09-11T08:27:14.984 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:14 smithi150 ceph-mon[29276]: 4.8 scrub starts 2024-09-11T08:27:14.984 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:14 smithi150 ceph-mon[29276]: 4.8 scrub ok 2024-09-11T08:27:14.984 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:14 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3688518279' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:27:14.984 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:14 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/180886096' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:27:14.984 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:14 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:27:14.984 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:14 smithi150 ceph-mon[39612]: 5.7 scrub starts 2024-09-11T08:27:14.984 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:14 smithi150 ceph-mon[39612]: 5.7 scrub ok 2024-09-11T08:27:14.984 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:14 smithi150 ceph-mon[39612]: 4.8 scrub starts 2024-09-11T08:27:14.984 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:14 smithi150 ceph-mon[39612]: 4.8 scrub ok 2024-09-11T08:27:14.985 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:14 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3688518279' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:27:14.985 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:14 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/180886096' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:27:14.985 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:14 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:27:15.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:15.008 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:15.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:15.109 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:15.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:15.150 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:15.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:15.251 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:15.275 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:15.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:15.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:15.280 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-11T08:27:15.323 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:27:15 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:27:15] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:27:15.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:14 smithi181 ceph-mon[32701]: 5.7 scrub starts 2024-09-11T08:27:15.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:14 smithi181 ceph-mon[32701]: 5.7 scrub ok 2024-09-11T08:27:15.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:14 smithi181 ceph-mon[32701]: 4.8 scrub starts 2024-09-11T08:27:15.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:14 smithi181 ceph-mon[32701]: 4.8 scrub ok 2024-09-11T08:27:15.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:14 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3688518279' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:27:15.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:14 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/180886096' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:27:15.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:14 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:27:15.378 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:27:15.415 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:15.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:15.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:27:15.517 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:15.560 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:15.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:15.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:15.618 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 35 16 66848 66832 119.316 83.6875 0.0134605 0.00837695 2024-09-11T08:27:15.662 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:15.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:15.696 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:15.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:15.720 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:27:15.797 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:15.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:15.829 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:15.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:15.930 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:15.958 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:15.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:15.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:16.060 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:16.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:16.101 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:16.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:16.162 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:15 smithi150 ceph-mon[29276]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:27:16.162 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:15 smithi150 ceph-mon[29276]: Cluster is now healthy 2024-09-11T08:27:16.162 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:15 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:27:16.162 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:15 smithi150 ceph-mon[29276]: osdmap e396: 8 total, 8 up, 8 in 2024-09-11T08:27:16.162 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:15 smithi150 ceph-mon[29276]: pgmap v900: 148 pgs: 148 active+clean; 3.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.4 KiB/s rd, 118 MiB/s wr, 1.89k op/s 2024-09-11T08:27:16.163 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:15 smithi150 ceph-mon[39612]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:27:16.163 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:15 smithi150 ceph-mon[39612]: Cluster is now healthy 2024-09-11T08:27:16.163 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:15 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:27:16.163 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:15 smithi150 ceph-mon[39612]: osdmap e396: 8 total, 8 up, 8 in 2024-09-11T08:27:16.163 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:15 smithi150 ceph-mon[39612]: pgmap v900: 148 pgs: 148 active+clean; 3.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.4 KiB/s rd, 118 MiB/s wr, 1.89k op/s 2024-09-11T08:27:16.190 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:15 smithi181 ceph-mon[32701]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:27:16.190 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:15 smithi181 ceph-mon[32701]: Cluster is now healthy 2024-09-11T08:27:16.190 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:15 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:27:16.190 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:15 smithi181 ceph-mon[32701]: osdmap e396: 8 total, 8 up, 8 in 2024-09-11T08:27:16.190 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:15 smithi181 ceph-mon[32701]: pgmap v900: 148 pgs: 148 active+clean; 3.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.4 KiB/s rd, 118 MiB/s wr, 1.89k op/s 2024-09-11T08:27:16.203 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:16.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:16.239 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:16.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:16.340 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:27:16.381 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:16.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:16.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:27:16.486 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:27:16.520 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:16.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:16.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:27:16.618 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 36 16 68167 68151 118.291 82.4375 0.00445003 0.00844757 2024-09-11T08:27:16.622 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:27:16.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:16.661 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:16.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:27:16.762 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:16.798 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:16.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:16.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:16.854 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:27:16.900 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:16.957 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:16.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:16.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:17.059 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:17.099 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:17.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:17.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:17.184 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:16 smithi150 ceph-mon[29276]: 6.b scrub starts 2024-09-11T08:27:17.184 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:16 smithi150 ceph-mon[29276]: 6.b scrub ok 2024-09-11T08:27:17.184 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:16 smithi150 ceph-mon[29276]: 4.5 scrub starts 2024-09-11T08:27:17.184 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:16 smithi150 ceph-mon[29276]: 4.5 scrub ok 2024-09-11T08:27:17.184 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:16 smithi150 ceph-mon[29276]: 6.12 scrub starts 2024-09-11T08:27:17.184 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:16 smithi150 ceph-mon[29276]: 6.12 scrub ok 2024-09-11T08:27:17.184 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:16 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:27:17.184 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:16 smithi150 ceph-mon[29276]: 4.3 deep-scrub starts 2024-09-11T08:27:17.184 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:16 smithi150 ceph-mon[29276]: 4.3 deep-scrub ok 2024-09-11T08:27:17.185 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:16 smithi150 ceph-mon[39612]: 6.b scrub starts 2024-09-11T08:27:17.185 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:16 smithi150 ceph-mon[39612]: 6.b scrub ok 2024-09-11T08:27:17.185 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:16 smithi150 ceph-mon[39612]: 4.5 scrub starts 2024-09-11T08:27:17.185 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:16 smithi150 ceph-mon[39612]: 4.5 scrub ok 2024-09-11T08:27:17.185 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:16 smithi150 ceph-mon[39612]: 6.12 scrub starts 2024-09-11T08:27:17.185 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:16 smithi150 ceph-mon[39612]: 6.12 scrub ok 2024-09-11T08:27:17.185 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:16 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:27:17.185 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:16 smithi150 ceph-mon[39612]: 4.3 deep-scrub starts 2024-09-11T08:27:17.186 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:16 smithi150 ceph-mon[39612]: 4.3 deep-scrub ok 2024-09-11T08:27:17.201 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:17.242 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:17.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:17.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:17.297 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:16 smithi181 ceph-mon[32701]: 6.b scrub starts 2024-09-11T08:27:17.298 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:16 smithi181 ceph-mon[32701]: 6.b scrub ok 2024-09-11T08:27:17.298 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:16 smithi181 ceph-mon[32701]: 4.5 scrub starts 2024-09-11T08:27:17.298 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:16 smithi181 ceph-mon[32701]: 4.5 scrub ok 2024-09-11T08:27:17.298 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:16 smithi181 ceph-mon[32701]: 6.12 scrub starts 2024-09-11T08:27:17.298 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:16 smithi181 ceph-mon[32701]: 6.12 scrub ok 2024-09-11T08:27:17.298 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:16 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:27:17.298 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:16 smithi181 ceph-mon[32701]: 4.3 deep-scrub starts 2024-09-11T08:27:17.298 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:16 smithi181 ceph-mon[32701]: 4.3 deep-scrub ok 2024-09-11T08:27:17.345 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:27:17.372 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:17.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:17.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:27:17.474 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:27:17.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:17.509 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:17.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:27:17.610 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:17.618 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 37 16 69320 69304 117.042 72.0625 0.00685074 0.00853896 2024-09-11T08:27:17.640 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:17.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:17.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:17.742 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:17.779 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:17.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:17.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:17.811 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-11T08:27:17.812 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-09-11T08:27:17.812 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:17.812 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.25799.0:69581 11.7 e396/382 11:e0a35e86:::benchmark_data_smithi150_187410_object69580:head v 396'4370, mlcod=396'4368)", 2024-09-11T08:27:17.812 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:27:17.809575+0000", 2024-09-11T08:27:17.812 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.0015712339999999999, 2024-09-11T08:27:17.812 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001590209, 2024-09-11T08:27:17.812 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:27:17.812 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-11T08:27:17.812 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:27:17.812 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:17.812 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:27:17.812 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:17.809575+0000", 2024-09-11T08:27:17.813 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:17.813 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:17.813 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:17.813 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:27:17.813 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:17.809575+0000", 2024-09-11T08:27:17.813 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:17.813 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:17.813 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:17.813 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:27:17.813 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:17.809574+0000", 2024-09-11T08:27:17.813 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:27:17.813 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:17.813 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:17.813 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:27:17.814 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:17.809630+0000", 2024-09-11T08:27:17.814 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5516999999999997e-05 2024-09-11T08:27:17.814 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:17.814 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:17.814 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:27:17.814 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:17.809642+0000", 2024-09-11T08:27:17.814 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2074e-05 2024-09-11T08:27:17.814 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:17.814 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:17.814 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:27:17.814 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:17.809648+0000", 2024-09-11T08:27:17.814 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.75e-06 2024-09-11T08:27:17.814 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:17.814 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:17.815 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:17.815 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:17.809698+0000", 2024-09-11T08:27:17.815 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0757000000000002e-05 2024-09-11T08:27:17.815 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:17.815 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:17.815 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:27:17.815 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:17.809714+0000", 2024-09-11T08:27:17.815 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5223e-05 2024-09-11T08:27:17.815 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:17.815 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:27:17.815 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:17.815 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:17.815 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-09-11T08:27:17.815 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 1 2024-09-11T08:27:17.816 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-11T08:27:17.880 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:17.923 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:17.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:17.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:18.025 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:27:18.059 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:18.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:18.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:27:18.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:17 smithi181 ceph-mon[32701]: 5.1f scrub starts 2024-09-11T08:27:18.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:17 smithi181 ceph-mon[32701]: 5.1f scrub ok 2024-09-11T08:27:18.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:17 smithi181 ceph-mon[32701]: pgmap v901: 148 pgs: 148 active+clean; 3.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 55 MiB/s wr, 874 op/s 2024-09-11T08:27:18.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:17 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:27:18.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:17 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:27:18.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:17 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:27:18.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:17 smithi181 ceph-mon[32701]: 5.13 scrub starts 2024-09-11T08:27:18.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:17 smithi181 ceph-mon[32701]: 5.13 scrub ok 2024-09-11T08:27:18.161 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:27:18.200 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:18.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:18.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:27:18.207 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T08:27:18.233 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:17 smithi150 ceph-mon[29276]: 5.1f scrub starts 2024-09-11T08:27:18.233 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:17 smithi150 ceph-mon[29276]: 5.1f scrub ok 2024-09-11T08:27:18.233 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:17 smithi150 ceph-mon[29276]: pgmap v901: 148 pgs: 148 active+clean; 3.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 55 MiB/s wr, 874 op/s 2024-09-11T08:27:18.233 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:17 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:27:18.233 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:17 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:27:18.233 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:17 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:27:18.233 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:17 smithi150 ceph-mon[29276]: 5.13 scrub starts 2024-09-11T08:27:18.233 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:17 smithi150 ceph-mon[29276]: 5.13 scrub ok 2024-09-11T08:27:18.236 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:17 smithi150 ceph-mon[39612]: 5.1f scrub starts 2024-09-11T08:27:18.236 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:17 smithi150 ceph-mon[39612]: 5.1f scrub ok 2024-09-11T08:27:18.236 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:17 smithi150 ceph-mon[39612]: pgmap v901: 148 pgs: 148 active+clean; 3.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 55 MiB/s wr, 874 op/s 2024-09-11T08:27:18.236 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:17 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:27:18.236 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:17 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:27:18.236 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:17 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:27:18.236 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:17 smithi150 ceph-mon[39612]: 5.13 scrub starts 2024-09-11T08:27:18.236 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:17 smithi150 ceph-mon[39612]: 5.13 scrub ok 2024-09-11T08:27:18.302 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:18.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:18.365 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:18.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:18.466 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:27:18.502 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:18.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:18.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:27:18.604 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:18.618 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 38 16 70625 70609 116.108 81.5625 0.0145593 0.0086084 2024-09-11T08:27:18.646 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:18.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:18.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:18.708 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-11T08:27:18.748 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:18.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:18.786 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:18.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:18.842 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:27:18 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:27:18] "GET /metrics HTTP/1.1" 200 224606 "" "Prometheus/2.43.0" 2024-09-11T08:27:18.888 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:27:18.890 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T08:27:18.933 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:18.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:18.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:27:19.034 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:19.044 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:27:19.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:19.071 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:19.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:19.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:18 smithi181 ceph-mon[32701]: 6.1d scrub starts 2024-09-11T08:27:19.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:18 smithi181 ceph-mon[32701]: 6.1d scrub ok 2024-09-11T08:27:19.074 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:27:18 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:27:19.172 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:27:19.203 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:27:19.203 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:18 smithi150 ceph-mon[29276]: 6.1d scrub starts 2024-09-11T08:27:19.203 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:18 smithi150 ceph-mon[29276]: 6.1d scrub ok 2024-09-11T08:27:19.204 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:18 smithi150 ceph-mon[39612]: 6.1d scrub starts 2024-09-11T08:27:19.204 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:18 smithi150 ceph-mon[39612]: 6.1d scrub ok 2024-09-11T08:27:19.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:19.234 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:19.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:27:19.321 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:27:19.323 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-09-11T08:27:19.335 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:27:19.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:19.364 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:19.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:27:19.465 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:27:19.499 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:19.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:19.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:27:19.602 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:19.619 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 39 16 71680 71664 114.82 65.9375 0.00950745 0.00870401 2024-09-11T08:27:19.639 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:19.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:19.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:19.741 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:19.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:19.764 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:19.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:19.865 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:27:19.902 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:19.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:19.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:27:20.004 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:27:20.046 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:20.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:20.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:27:20.148 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:27:20.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:20.182 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:20.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:27:20.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:20 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:27:20.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:20 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:27:20.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:20 smithi150 ceph-mon[29276]: 5.1e scrub starts 2024-09-11T08:27:20.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:20 smithi150 ceph-mon[29276]: 5.1e scrub ok 2024-09-11T08:27:20.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:20 smithi150 ceph-mon[29276]: 5.0 scrub starts 2024-09-11T08:27:20.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:20 smithi150 ceph-mon[29276]: 5.0 scrub ok 2024-09-11T08:27:20.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:20 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:27:20.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:20 smithi150 ceph-mon[29276]: osdmap e397: 8 total, 8 up, 8 in 2024-09-11T08:27:20.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:20 smithi150 ceph-mon[29276]: pgmap v903: 148 pgs: 148 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.55k op/s 2024-09-11T08:27:20.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:20 smithi150 ceph-mon[29276]: from='client.25636 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:27:20.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:20 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:27:20.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:20 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:27:20.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:20 smithi150 ceph-mon[39612]: 5.1e scrub starts 2024-09-11T08:27:20.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:20 smithi150 ceph-mon[39612]: 5.1e scrub ok 2024-09-11T08:27:20.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:20 smithi150 ceph-mon[39612]: 5.0 scrub starts 2024-09-11T08:27:20.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:20 smithi150 ceph-mon[39612]: 5.0 scrub ok 2024-09-11T08:27:20.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:20 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:27:20.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:20 smithi150 ceph-mon[39612]: osdmap e397: 8 total, 8 up, 8 in 2024-09-11T08:27:20.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:20 smithi150 ceph-mon[39612]: pgmap v903: 148 pgs: 148 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.55k op/s 2024-09-11T08:27:20.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:20 smithi150 ceph-mon[39612]: from='client.25636 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:27:20.283 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:27:20.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:20.306 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:20.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:27:20.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:20 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:27:20.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:20 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:27:20.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:20 smithi181 ceph-mon[32701]: 5.1e scrub starts 2024-09-11T08:27:20.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:20 smithi181 ceph-mon[32701]: 5.1e scrub ok 2024-09-11T08:27:20.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:20 smithi181 ceph-mon[32701]: 5.0 scrub starts 2024-09-11T08:27:20.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:20 smithi181 ceph-mon[32701]: 5.0 scrub ok 2024-09-11T08:27:20.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:20 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:27:20.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:20 smithi181 ceph-mon[32701]: osdmap e397: 8 total, 8 up, 8 in 2024-09-11T08:27:20.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:20 smithi181 ceph-mon[32701]: pgmap v903: 148 pgs: 148 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.55k op/s 2024-09-11T08:27:20.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:20 smithi181 ceph-mon[32701]: from='client.25636 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:27:20.407 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:27:20.430 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:20.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:20.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:27:20.532 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:20.568 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:20.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:20.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:20.619 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:2024-09-11T08:27:20.620168+0000 min lat: 0.00120292 max lat: 0.282428 avg lat: 0.00881978 2024-09-11T08:27:20.620 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-11T08:27:20.620 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 40 16 72556 72540 113.319 54.75 0.00887236 0.00881978 2024-09-11T08:27:20.670 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:20.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:20.711 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:20.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:20.812 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:27:20.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:20.857 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:20.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:27:20.958 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:20.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:20.989 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:20.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:21.090 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:21.127 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:21.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:21.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:21.175 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:21 smithi181 ceph-mon[32701]: 5.18 scrub starts 2024-09-11T08:27:21.175 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:21 smithi181 ceph-mon[32701]: 5.18 scrub ok 2024-09-11T08:27:21.175 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:21 smithi181 ceph-mon[32701]: 6.1b scrub starts 2024-09-11T08:27:21.175 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:21 smithi181 ceph-mon[32701]: 6.1b scrub ok 2024-09-11T08:27:21.175 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:21 smithi181 ceph-mon[32701]: osdmap e398: 8 total, 8 up, 8 in 2024-09-11T08:27:21.228 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:27:21.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:21.262 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:21.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:27:21.363 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:27:21.407 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:21.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:21.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:27:21.417 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:21 smithi150 ceph-mon[29276]: 5.18 scrub starts 2024-09-11T08:27:21.417 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:21 smithi150 ceph-mon[29276]: 5.18 scrub ok 2024-09-11T08:27:21.417 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:21 smithi150 ceph-mon[29276]: 6.1b scrub starts 2024-09-11T08:27:21.417 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:21 smithi150 ceph-mon[29276]: 6.1b scrub ok 2024-09-11T08:27:21.418 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:21 smithi150 ceph-mon[29276]: osdmap e398: 8 total, 8 up, 8 in 2024-09-11T08:27:21.423 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:21 smithi150 ceph-mon[39612]: 5.18 scrub starts 2024-09-11T08:27:21.423 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:21 smithi150 ceph-mon[39612]: 5.18 scrub ok 2024-09-11T08:27:21.424 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:21 smithi150 ceph-mon[39612]: 6.1b scrub starts 2024-09-11T08:27:21.424 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:21 smithi150 ceph-mon[39612]: 6.1b scrub ok 2024-09-11T08:27:21.424 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:21 smithi150 ceph-mon[39612]: osdmap e398: 8 total, 8 up, 8 in 2024-09-11T08:27:21.457 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-11T08:27:21.457 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-11T08:27:21.458 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-09-11T08:27:21.458 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-09-11T08:27:21.458 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-11T08:27:21.510 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:21.553 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:21.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:21.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:21.619 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 41 16 73752 73736 112.378 74.75 0.0109294 0.00889389 2024-09-11T08:27:21.656 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:21.690 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:21.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:21.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:21.792 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:27:21.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:21.829 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:21.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:27:21.930 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:21.973 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:21.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:21.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:22.075 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:22.108 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:22.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:22.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:22.209 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:22.231 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:22.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:22.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:22.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:22 smithi181 ceph-mon[32701]: pgmap v905: 148 pgs: 148 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 826 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-09-11T08:27:22.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:22 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:27:22.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:22 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:27:22.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:22 smithi181 ceph-mon[32701]: 5.b scrub starts 2024-09-11T08:27:22.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:22 smithi181 ceph-mon[32701]: 5.b scrub ok 2024-09-11T08:27:22.333 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:27:22.354 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:22.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:22.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:27:22.456 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:22.485 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:22.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:22.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:22.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:22 smithi150 ceph-mon[29276]: pgmap v905: 148 pgs: 148 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 826 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-09-11T08:27:22.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:22 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:27:22.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:22 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:27:22.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:22 smithi150 ceph-mon[29276]: 5.b scrub starts 2024-09-11T08:27:22.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:22 smithi150 ceph-mon[29276]: 5.b scrub ok 2024-09-11T08:27:22.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:22 smithi150 ceph-mon[39612]: pgmap v905: 148 pgs: 148 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 826 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-09-11T08:27:22.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:22 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:27:22.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:22 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:27:22.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:22 smithi150 ceph-mon[39612]: 5.b scrub starts 2024-09-11T08:27:22.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:22 smithi150 ceph-mon[39612]: 5.b scrub ok 2024-09-11T08:27:22.587 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:27:22.613 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:22.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:22.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:27:22.620 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 42 16 74636 74620 111.017 55.25 0.017043 0.00900286 2024-09-11T08:27:22.624 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":903,"stamp":"2024-09-11T08:27:19.142802+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4588771548,"num_objects":70206,"num_object_clones":0,"num_object_copies":210618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70206,"num_whiteouts":0,"num_read":2014,"num_read_kb":1873,"num_write":70839,"num_write_kb":4482311,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":208,"num_bytes_recovered":1776519,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70979,"ondisk_log_size":70979,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":18688956,"kb_used_data":13247092,"kb_used_omap":22,"kb_used_meta":5441769,"kb_avail":731272260,"statfs":{"total":767960285184,"available":748822794240,"internally_reserved":0,"allocated":13565022208,"data_stored":13561317414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22853,"internal_metadata":5572372155},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":811991040,"num_objects":12390,"num_object_clones":0,"num_object_copies":37170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12390,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":12390,"num_write_kb":792960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002384"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":451,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-11T08:27:07.898474+0000","last_change":"2024-09-11T08:24:28.359613+0000","last_active":"2024-09-11T08:27:07.898474+0000","last_peered":"2024-09-11T08:27:07.898474+0000","last_clean":"2024-09-11T08:27:07.898474+0000","last_became_active":"2024-09-11T08:24:28.358401+0000","last_became_peered":"2024-09-11T08:24:28.358401+0000","last_unstale":"2024-09-11T08:27:07.898474+0000","last_undegraded":"2024-09-11T08:27:07.898474+0000","last_fullsized":"2024-09-11T08:27:07.898474+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:23:29.976662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:23:29.976662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0090761950000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"312'19","reported_seq":456,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.728879+0000","last_change":"2024-09-11T08:27:02.000454+0000","last_active":"2024-09-11T08:27:10.728879+0000","last_peered":"2024-09-11T08:27:10.728879+0000","last_clean":"2024-09-11T08:27:10.728879+0000","last_became_active":"2024-09-11T08:07:32.048048+0000","last_became_peered":"2024-09-11T08:07:32.048048+0000","last_unstale":"2024-09-11T08:27:10.728879+0000","last_undegraded":"2024-09-11T08:27:10.728879+0000","last_fullsized":"2024-09-11T08:27:10.728879+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"312'19","last_scrub_stamp":"2024-09-11T08:27:02.000321+0000","last_deep_scrub":"312'19","last_deep_scrub_stamp":"2024-09-11T08:21:59.122122+0000","last_clean_scrub_stamp":"2024-09-11T08:27:02.000321+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:26.414397+0000","scrub_duration":0.0035901290000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":508,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.734390+0000","last_change":"2024-09-11T08:27:03.997743+0000","last_active":"2024-09-11T08:27:10.734390+0000","last_peered":"2024-09-11T08:27:10.734390+0000","last_clean":"2024-09-11T08:27:10.734390+0000","last_became_active":"2024-09-11T08:24:28.372160+0000","last_became_peered":"2024-09-11T08:24:28.372160+0000","last_unstale":"2024-09-11T08:27:10.734390+0000","last_undegraded":"2024-09-11T08:27:10.734390+0000","last_fullsized":"2024-09-11T08:27:10.734390+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:03.997657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:59.531202+0000","last_clean_scrub_stamp":"2024-09-11T08:27:03.997657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:26.638065+0000","scrub_duration":0.0035479840000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":366,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.734600+0000","last_change":"2024-09-11T08:27:06.987974+0000","last_active":"2024-09-11T08:27:10.734600+0000","last_peered":"2024-09-11T08:27:10.734600+0000","last_clean":"2024-09-11T08:27:10.734600+0000","last_became_active":"2024-09-11T08:24:28.354834+0000","last_became_peered":"2024-09-11T08:24:28.354834+0000","last_unstale":"2024-09-11T08:27:10.734600+0000","last_undegraded":"2024-09-11T08:27:10.734600+0000","last_fullsized":"2024-09-11T08:27:10.734600+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:06.987888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:02.470649+0000","last_clean_scrub_stamp":"2024-09-11T08:27:06.987888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:25.277118+0000","scrub_duration":0.0030657430000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":437,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.732420+0000","last_change":"2024-09-11T08:24:25.004241+0000","last_active":"2024-09-11T08:27:10.732420+0000","last_peered":"2024-09-11T08:27:10.732420+0000","last_clean":"2024-09-11T08:27:10.732420+0000","last_became_active":"2024-09-11T08:15:56.521104+0000","last_became_peered":"2024-09-11T08:15:56.521104+0000","last_unstale":"2024-09-11T08:27:10.732420+0000","last_undegraded":"2024-09-11T08:27:10.732420+0000","last_fullsized":"2024-09-11T08:27:10.732420+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:25.004147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:24:25.004147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004254351,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":220,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.733169+0000","last_change":"2024-09-11T08:24:16.568741+0000","last_active":"2024-09-11T08:27:10.733169+0000","last_peered":"2024-09-11T08:27:10.733169+0000","last_clean":"2024-09-11T08:27:10.733169+0000","last_became_active":"2024-09-11T08:17:37.550048+0000","last_became_peered":"2024-09-11T08:17:37.550048+0000","last_unstale":"2024-09-11T08:27:10.733169+0000","last_undegraded":"2024-09-11T08:27:10.733169+0000","last_fullsized":"2024-09-11T08:27:10.733169+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:16.568660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:24:16.568660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.006748937,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"239'34","reported_seq":390,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.894329+0000","last_change":"2024-09-11T08:27:10.894267+0000","last_active":"2024-09-11T08:27:10.894329+0000","last_peered":"2024-09-11T08:27:10.894329+0000","last_clean":"2024-09-11T08:27:10.894329+0000","last_became_active":"2024-09-11T08:17:37.553866+0000","last_became_peered":"2024-09-11T08:17:37.553866+0000","last_unstale":"2024-09-11T08:27:10.894329+0000","last_undegraded":"2024-09-11T08:27:10.894329+0000","last_fullsized":"2024-09-11T08:27:10.894329+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'34","last_scrub_stamp":"2024-09-11T08:27:10.894181+0000","last_deep_scrub":"145'22","last_deep_scrub_stamp":"2024-09-11T08:15:57.478870+0000","last_clean_scrub_stamp":"2024-09-11T08:27:10.894181+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:31.391831+0000","scrub_duration":0.014739263000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":81,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.733521+0000","last_change":"2024-09-11T08:24:08.583672+0000","last_active":"2024-09-11T08:27:10.733521+0000","last_peered":"2024-09-11T08:27:10.733521+0000","last_clean":"2024-09-11T08:27:10.733521+0000","last_became_active":"2024-09-11T08:23:22.687041+0000","last_became_peered":"2024-09-11T08:23:22.687041+0000","last_unstale":"2024-09-11T08:27:10.733521+0000","last_undegraded":"2024-09-11T08:27:10.733521+0000","last_fullsized":"2024-09-11T08:27:10.733521+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:08.583528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:18:36.998503+0000","last_clean_scrub_stamp":"2024-09-11T08:24:08.583528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002003758,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":122,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.733710+0000","last_change":"2024-09-11T08:27:05.913172+0000","last_active":"2024-09-11T08:27:10.733710+0000","last_peered":"2024-09-11T08:27:10.733710+0000","last_clean":"2024-09-11T08:27:10.733710+0000","last_became_active":"2024-09-11T08:24:28.360021+0000","last_became_peered":"2024-09-11T08:24:28.360021+0000","last_unstale":"2024-09-11T08:27:10.733710+0000","last_undegraded":"2024-09-11T08:27:10.733710+0000","last_fullsized":"2024-09-11T08:27:10.733710+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:05.913078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:54.878409+0000","last_clean_scrub_stamp":"2024-09-11T08:27:05.913078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:19.980455+0000","scrub_duration":0.001905669,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"239'15","reported_seq":330,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-11T08:27:07.899389+0000","last_change":"2024-09-11T08:26:02.101012+0000","last_active":"2024-09-11T08:27:07.899389+0000","last_peered":"2024-09-11T08:27:07.899389+0000","last_clean":"2024-09-11T08:27:07.899389+0000","last_became_active":"2024-09-11T08:23:22.687182+0000","last_became_peered":"2024-09-11T08:23:22.687182+0000","last_unstale":"2024-09-11T08:27:07.899389+0000","last_undegraded":"2024-09-11T08:27:07.899389+0000","last_fullsized":"2024-09-11T08:27:07.899389+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'15","last_scrub_stamp":"2024-09-11T08:26:02.100940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:26:02.100940+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:30.876179+0000","scrub_duration":0.0028851139999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":336,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-11T08:27:07.898232+0000","last_change":"2024-09-11T08:26:04.073177+0000","last_active":"2024-09-11T08:27:07.898232+0000","last_peered":"2024-09-11T08:27:07.898232+0000","last_clean":"2024-09-11T08:27:07.898232+0000","last_became_active":"2024-09-11T08:24:28.363985+0000","last_became_peered":"2024-09-11T08:24:28.363985+0000","last_unstale":"2024-09-11T08:27:07.898232+0000","last_undegraded":"2024-09-11T08:27:07.898232+0000","last_fullsized":"2024-09-11T08:27:07.898232+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:04.073122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:20:18.900482+0000","last_clean_scrub_stamp":"2024-09-11T08:26:04.073122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00163221,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":425,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-11T08:27:07.897287+0000","last_change":"2024-09-11T08:24:26.079306+0000","last_active":"2024-09-11T08:27:07.897287+0000","last_peered":"2024-09-11T08:27:07.897287+0000","last_clean":"2024-09-11T08:27:07.897287+0000","last_became_active":"2024-09-11T08:14:24.792595+0000","last_became_peered":"2024-09-11T08:14:24.792595+0000","last_unstale":"2024-09-11T08:27:07.897287+0000","last_undegraded":"2024-09-11T08:27:07.897287+0000","last_fullsized":"2024-09-11T08:27:07.897287+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:26.079152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:24:26.079152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037087930000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":124,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.735678+0000","last_change":"2024-09-11T08:27:00.877188+0000","last_active":"2024-09-11T08:27:10.735678+0000","last_peered":"2024-09-11T08:27:10.735678+0000","last_clean":"2024-09-11T08:27:10.735678+0000","last_became_active":"2024-09-11T08:24:28.359153+0000","last_became_peered":"2024-09-11T08:24:28.359153+0000","last_unstale":"2024-09-11T08:27:10.735678+0000","last_undegraded":"2024-09-11T08:27:10.735678+0000","last_fullsized":"2024-09-11T08:27:10.735678+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:00.877062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:27:00.877062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:06.321049+0000","scrub_duration":0.0048170670000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"239'9","reported_seq":321,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.732718+0000","last_change":"2024-09-11T08:27:07.592807+0000","last_active":"2024-09-11T08:27:10.732718+0000","last_peered":"2024-09-11T08:27:10.732718+0000","last_clean":"2024-09-11T08:27:10.732718+0000","last_became_active":"2024-09-11T08:23:22.681876+0000","last_became_peered":"2024-09-11T08:23:22.681876+0000","last_unstale":"2024-09-11T08:27:10.732718+0000","last_undegraded":"2024-09-11T08:27:10.732718+0000","last_fullsized":"2024-09-11T08:27:10.732718+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'9","last_scrub_stamp":"2024-09-11T08:27:07.592703+0000","last_deep_scrub":"139'7","last_deep_scrub_stamp":"2024-09-11T08:12:24.869318+0000","last_clean_scrub_stamp":"2024-09-11T08:27:07.592703+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:21.681522+0000","scrub_duration":0.0060076349999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":238,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.727536+0000","last_change":"2024-09-11T08:23:33.277787+0000","last_active":"2024-09-11T08:27:10.727536+0000","last_peered":"2024-09-11T08:27:10.727536+0000","last_clean":"2024-09-11T08:27:10.727536+0000","last_became_active":"2024-09-11T08:15:56.526954+0000","last_became_peered":"2024-09-11T08:15:56.526954+0000","last_unstale":"2024-09-11T08:27:10.727536+0000","last_undegraded":"2024-09-11T08:27:10.727536+0000","last_fullsized":"2024-09-11T08:27:10.727536+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:23:33.277734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:23:33.277734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.023249810999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":118,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.734215+0000","last_change":"2024-09-11T08:25:54.524776+0000","last_active":"2024-09-11T08:27:10.734215+0000","last_peered":"2024-09-11T08:27:10.734215+0000","last_clean":"2024-09-11T08:27:10.734215+0000","last_became_active":"2024-09-11T08:24:28.372097+0000","last_became_peered":"2024-09-11T08:24:28.372097+0000","last_unstale":"2024-09-11T08:27:10.734215+0000","last_undegraded":"2024-09-11T08:27:10.734215+0000","last_fullsized":"2024-09-11T08:27:10.734215+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:22:54.272360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:20:19.927616+0000","last_clean_scrub_stamp":"2024-09-11T08:22:54.272360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":364,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.732833+0000","last_change":"2024-09-11T08:24:29.597394+0000","last_active":"2024-09-11T08:27:10.732833+0000","last_peered":"2024-09-11T08:27:10.732833+0000","last_clean":"2024-09-11T08:27:10.732833+0000","last_became_active":"2024-09-11T08:24:28.368560+0000","last_became_peered":"2024-09-11T08:24:28.368560+0000","last_unstale":"2024-09-11T08:27:10.732833+0000","last_undegraded":"2024-09-11T08:27:10.732833+0000","last_fullsized":"2024-09-11T08:27:10.732833+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:29.597313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:24:29.597313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024078900000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":444,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-11T08:27:07.898730+0000","last_change":"2024-09-11T08:26:06.019004+0000","last_active":"2024-09-11T08:27:07.898730+0000","last_peered":"2024-09-11T08:27:07.898730+0000","last_clean":"2024-09-11T08:27:07.898730+0000","last_became_active":"2024-09-11T08:15:56.516941+0000","last_became_peered":"2024-09-11T08:15:56.516941+0000","last_unstale":"2024-09-11T08:27:07.898730+0000","last_undegraded":"2024-09-11T08:27:07.898730+0000","last_fullsized":"2024-09-11T08:27:07.898730+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:06.018947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:26:06.018947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:29.870750+0000","scrub_duration":0.001287098,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"239'34","reported_seq":493,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.741434+0000","last_change":"2024-09-11T08:27:08.899940+0000","last_active":"2024-09-11T08:27:10.741434+0000","last_peered":"2024-09-11T08:27:10.741434+0000","last_clean":"2024-09-11T08:27:10.741434+0000","last_became_active":"2024-09-11T08:07:32.046363+0000","last_became_peered":"2024-09-11T08:07:32.046363+0000","last_unstale":"2024-09-11T08:27:10.741434+0000","last_undegraded":"2024-09-11T08:27:10.741434+0000","last_fullsized":"2024-09-11T08:27:10.741434+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'34","last_scrub_stamp":"2024-09-11T08:27:08.899726+0000","last_deep_scrub":"107'22","last_deep_scrub_stamp":"2024-09-11T08:16:04.159211+0000","last_clean_scrub_stamp":"2024-09-11T08:27:08.899726+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:19.926838+0000","scrub_duration":0.050618700000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":441,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.735943+0000","last_change":"2024-09-11T08:25:57.566953+0000","last_active":"2024-09-11T08:27:10.735943+0000","last_peered":"2024-09-11T08:27:10.735943+0000","last_clean":"2024-09-11T08:27:10.735943+0000","last_became_active":"2024-09-11T08:07:34.055410+0000","last_became_peered":"2024-09-11T08:07:34.055410+0000","last_unstale":"2024-09-11T08:27:10.735943+0000","last_undegraded":"2024-09-11T08:27:10.735943+0000","last_fullsized":"2024-09-11T08:27:10.735943+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:25:57.566898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:19:28.870211+0000","last_clean_scrub_stamp":"2024-09-11T08:25:57.566898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001280427,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":257,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.732989+0000","last_change":"2024-09-11T08:27:06.905574+0000","last_active":"2024-09-11T08:27:10.732989+0000","last_peered":"2024-09-11T08:27:10.732989+0000","last_clean":"2024-09-11T08:27:10.732989+0000","last_became_active":"2024-09-11T08:24:28.357957+0000","last_became_peered":"2024-09-11T08:24:28.357957+0000","last_unstale":"2024-09-11T08:27:10.732989+0000","last_undegraded":"2024-09-11T08:27:10.732989+0000","last_fullsized":"2024-09-11T08:27:10.732989+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:06.905489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:27:06.905489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:33.245384+0000","scrub_duration":0.0037508659999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":90,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.735135+0000","last_change":"2024-09-11T08:26:04.502886+0000","last_active":"2024-09-11T08:27:10.735135+0000","last_peered":"2024-09-11T08:27:10.735135+0000","last_clean":"2024-09-11T08:27:10.735135+0000","last_became_active":"2024-09-11T08:23:22.686935+0000","last_became_peered":"2024-09-11T08:23:22.686935+0000","last_unstale":"2024-09-11T08:27:10.735135+0000","last_undegraded":"2024-09-11T08:27:10.735135+0000","last_fullsized":"2024-09-11T08:27:10.735135+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:04.502835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:26:04.502835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001566196,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"239'18","reported_seq":290,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.742043+0000","last_change":"2024-09-11T08:26:57.767336+0000","last_active":"2024-09-11T08:27:10.742043+0000","last_peered":"2024-09-11T08:27:10.742043+0000","last_clean":"2024-09-11T08:27:10.742043+0000","last_became_active":"2024-09-11T08:17:37.553402+0000","last_became_peered":"2024-09-11T08:17:37.553402+0000","last_unstale":"2024-09-11T08:27:10.742043+0000","last_undegraded":"2024-09-11T08:27:10.742043+0000","last_fullsized":"2024-09-11T08:27:10.742043+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'18","last_scrub_stamp":"2024-09-11T08:26:57.767252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:26:57.767252+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:25.867282+0000","scrub_duration":0.0051635600000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":91,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.732556+0000","last_change":"2024-09-11T08:27:06.590237+0000","last_active":"2024-09-11T08:27:10.732556+0000","last_peered":"2024-09-11T08:27:10.732556+0000","last_clean":"2024-09-11T08:27:10.732556+0000","last_became_active":"2024-09-11T08:23:22.684583+0000","last_became_peered":"2024-09-11T08:23:22.684583+0000","last_unstale":"2024-09-11T08:27:10.732556+0000","last_undegraded":"2024-09-11T08:27:10.732556+0000","last_fullsized":"2024-09-11T08:27:10.732556+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:06.590161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:10:40.110490+0000","last_clean_scrub_stamp":"2024-09-11T08:27:06.590161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:11.903001+0000","scrub_duration":0.00158486,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":413,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.728034+0000","last_change":"2024-09-11T08:25:56.984646+0000","last_active":"2024-09-11T08:27:10.728034+0000","last_peered":"2024-09-11T08:27:10.728034+0000","last_clean":"2024-09-11T08:27:10.728034+0000","last_became_active":"2024-09-11T08:24:28.357198+0000","last_became_peered":"2024-09-11T08:24:28.357198+0000","last_unstale":"2024-09-11T08:27:10.728034+0000","last_undegraded":"2024-09-11T08:27:10.728034+0000","last_fullsized":"2024-09-11T08:27:10.728034+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-11T08:22:51.567081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:22:51.567081+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041879530000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":435,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.734517+0000","last_change":"2024-09-11T08:24:17.569443+0000","last_active":"2024-09-11T08:27:10.734517+0000","last_peered":"2024-09-11T08:27:10.734517+0000","last_clean":"2024-09-11T08:27:10.734517+0000","last_became_active":"2024-09-11T08:07:30.045932+0000","last_became_peered":"2024-09-11T08:07:30.045932+0000","last_unstale":"2024-09-11T08:27:10.734517+0000","last_undegraded":"2024-09-11T08:27:10.734517+0000","last_fullsized":"2024-09-11T08:27:10.734517+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.569369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.569369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028606489999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"350'25","reported_seq":520,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-11T08:27:07.898172+0000","last_change":"2024-09-11T08:24:23.093331+0000","last_active":"2024-09-11T08:27:07.898172+0000","last_peered":"2024-09-11T08:27:07.898172+0000","last_clean":"2024-09-11T08:27:07.898172+0000","last_became_active":"2024-09-11T08:15:56.515817+0000","last_became_peered":"2024-09-11T08:15:56.515817+0000","last_unstale":"2024-09-11T08:27:07.898172+0000","last_undegraded":"2024-09-11T08:27:07.898172+0000","last_fullsized":"2024-09-11T08:27:07.898172+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'25","last_scrub_stamp":"2024-09-11T08:24:23.093269+0000","last_deep_scrub":"350'25","last_deep_scrub_stamp":"2024-09-11T08:24:23.093269+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.093269+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015481372,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"254'12","reported_seq":736,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-09-11T08:27:16.738850+0000","last_change":"2024-09-11T08:27:16.738809+0000","last_active":"2024-09-11T08:27:16.738850+0000","last_peered":"2024-09-11T08:27:16.738850+0000","last_clean":"2024-09-11T08:27:16.738850+0000","last_became_active":"2024-09-11T08:15:56.525498+0000","last_became_peered":"2024-09-11T08:15:56.525498+0000","last_unstale":"2024-09-11T08:27:16.738850+0000","last_undegraded":"2024-09-11T08:27:16.738850+0000","last_fullsized":"2024-09-11T08:27:16.738850+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'12","last_scrub_stamp":"2024-09-11T08:27:16.738727+0000","last_deep_scrub":"254'12","last_deep_scrub_stamp":"2024-09-11T08:18:34.253865+0000","last_clean_scrub_stamp":"2024-09-11T08:27:16.738727+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:21.793809+0000","scrub_duration":0.0092781189999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.0","version":"396'4316","reported_seq":4338,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-09-11T08:27:18.792263+0000","last_change":"2024-09-11T08:26:27.764038+0000","last_active":"2024-09-11T08:27:18.792263+0000","last_peered":"2024-09-11T08:27:18.792263+0000","last_clean":"2024-09-11T08:27:18.792263+0000","last_became_active":"2024-09-11T08:26:27.763713+0000","last_became_peered":"2024-09-11T08:26:27.763713+0000","last_unstale":"2024-09-11T08:27:18.792263+0000","last_undegraded":"2024-09-11T08:27:18.792263+0000","last_fullsized":"2024-09-11T08:27:18.792263+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:26:26.740307+0000","objects_scrubbed":0,"log_size":4316,"log_dups_size":0,"ondisk_log_size":4316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:43.830110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282853376,"num_objects":4316,"num_object_clones":0,"num_object_copies":12948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4316,"num_write_kb":276224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"239'27","reported_seq":486,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.737647+0000","last_change":"2024-09-11T08:24:11.562773+0000","last_active":"2024-09-11T08:27:10.737647+0000","last_peered":"2024-09-11T08:27:10.737647+0000","last_clean":"2024-09-11T08:27:10.737647+0000","last_became_active":"2024-09-11T08:07:32.054317+0000","last_became_peered":"2024-09-11T08:07:32.054317+0000","last_unstale":"2024-09-11T08:27:10.737647+0000","last_undegraded":"2024-09-11T08:27:10.737647+0000","last_fullsized":"2024-09-11T08:27:10.737647+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'27","last_scrub_stamp":"2024-09-11T08:24:11.562679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:24:11.562679+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0092178139999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":456,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.733380+0000","last_change":"2024-09-11T08:27:10.670832+0000","last_active":"2024-09-11T08:27:10.733380+0000","last_peered":"2024-09-11T08:27:10.733380+0000","last_clean":"2024-09-11T08:27:10.733380+0000","last_became_active":"2024-09-11T08:15:56.521943+0000","last_became_peered":"2024-09-11T08:15:56.521943+0000","last_unstale":"2024-09-11T08:27:10.733380+0000","last_undegraded":"2024-09-11T08:27:10.733380+0000","last_fullsized":"2024-09-11T08:27:10.733380+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:10.670751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:22:00.707990+0000","last_clean_scrub_stamp":"2024-09-11T08:27:10.670751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:36.828574+0000","scrub_duration":0.0086032320000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"354'13","reported_seq":165,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.740104+0000","last_change":"2024-09-11T08:26:00.096276+0000","last_active":"2024-09-11T08:27:10.740104+0000","last_peered":"2024-09-11T08:27:10.740104+0000","last_clean":"2024-09-11T08:27:10.740104+0000","last_became_active":"2024-09-11T08:24:28.368212+0000","last_became_peered":"2024-09-11T08:24:28.368212+0000","last_unstale":"2024-09-11T08:27:10.740104+0000","last_undegraded":"2024-09-11T08:27:10.740104+0000","last_fullsized":"2024-09-11T08:27:10.740104+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'13","last_scrub_stamp":"2024-09-11T08:26:00.096215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:26:00.096215+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016774890000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":128,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.735872+0000","last_change":"2024-09-11T08:27:05.040598+0000","last_active":"2024-09-11T08:27:10.735872+0000","last_peered":"2024-09-11T08:27:10.735872+0000","last_clean":"2024-09-11T08:27:10.735872+0000","last_became_active":"2024-09-11T08:24:28.353556+0000","last_became_peered":"2024-09-11T08:24:28.353556+0000","last_unstale":"2024-09-11T08:27:10.735872+0000","last_undegraded":"2024-09-11T08:27:10.735872+0000","last_fullsized":"2024-09-11T08:27:10.735872+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:05.040523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:06.616332+0000","last_clean_scrub_stamp":"2024-09-11T08:27:05.040523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:34.516330+0000","scrub_duration":0.0050758849999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.f","version":"396'4393","reported_seq":4414,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-09-11T08:27:18.182420+0000","last_change":"2024-09-11T08:26:27.766747+0000","last_active":"2024-09-11T08:27:18.182420+0000","last_peered":"2024-09-11T08:27:18.182420+0000","last_clean":"2024-09-11T08:27:18.182420+0000","last_became_active":"2024-09-11T08:26:27.766540+0000","last_became_peered":"2024-09-11T08:26:27.766540+0000","last_unstale":"2024-09-11T08:27:18.182420+0000","last_undegraded":"2024-09-11T08:27:18.182420+0000","last_fullsized":"2024-09-11T08:27:18.182420+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:26:26.740307+0000","objects_scrubbed":0,"log_size":4393,"log_dups_size":0,"ondisk_log_size":4393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:39.323593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287899648,"num_objects":4393,"num_object_clones":0,"num_object_copies":13179,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4393,"num_write_kb":281152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"239'30","reported_seq":475,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:12.224968+0000","last_change":"2024-09-11T08:27:12.224895+0000","last_active":"2024-09-11T08:27:12.224968+0000","last_peered":"2024-09-11T08:27:12.224968+0000","last_clean":"2024-09-11T08:27:12.224968+0000","last_became_active":"2024-09-11T08:17:37.555703+0000","last_became_peered":"2024-09-11T08:17:37.555703+0000","last_unstale":"2024-09-11T08:27:12.224968+0000","last_undegraded":"2024-09-11T08:27:12.224968+0000","last_fullsized":"2024-09-11T08:27:12.224968+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'30","last_scrub_stamp":"2024-09-11T08:27:12.224825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:27:12.224825+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:18.909592+0000","scrub_duration":0.0073510700000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":406,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.727914+0000","last_change":"2024-09-11T08:26:08.112087+0000","last_active":"2024-09-11T08:27:10.727914+0000","last_peered":"2024-09-11T08:27:10.727914+0000","last_clean":"2024-09-11T08:27:10.727914+0000","last_became_active":"2024-09-11T08:17:37.553579+0000","last_became_peered":"2024-09-11T08:17:37.553579+0000","last_unstale":"2024-09-11T08:27:10.727914+0000","last_undegraded":"2024-09-11T08:27:10.727914+0000","last_fullsized":"2024-09-11T08:27:10.727914+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:08.112023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:26:08.112023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:30.647649+0000","scrub_duration":0.001608578,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":445,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.728915+0000","last_change":"2024-09-11T08:26:01.994013+0000","last_active":"2024-09-11T08:27:10.728915+0000","last_peered":"2024-09-11T08:27:10.728915+0000","last_clean":"2024-09-11T08:27:10.728915+0000","last_became_active":"2024-09-11T08:15:56.522606+0000","last_became_peered":"2024-09-11T08:15:56.522606+0000","last_unstale":"2024-09-11T08:27:10.728915+0000","last_undegraded":"2024-09-11T08:27:10.728915+0000","last_fullsized":"2024-09-11T08:27:10.728915+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:01.993924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:26:01.993924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:24.892591+0000","scrub_duration":0.0012801519999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":81,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.732069+0000","last_change":"2024-09-11T08:26:54.574588+0000","last_active":"2024-09-11T08:27:10.732069+0000","last_peered":"2024-09-11T08:27:10.732069+0000","last_clean":"2024-09-11T08:27:10.732069+0000","last_became_active":"2024-09-11T08:23:22.683474+0000","last_became_peered":"2024-09-11T08:23:22.683474+0000","last_unstale":"2024-09-11T08:27:10.732069+0000","last_undegraded":"2024-09-11T08:27:10.732069+0000","last_fullsized":"2024-09-11T08:27:10.732069+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:54.574506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:22:50.151585+0000","last_clean_scrub_stamp":"2024-09-11T08:26:54.574506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:21.672267+0000","scrub_duration":0.012208658000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"396'4444","reported_seq":4466,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-09-11T08:27:17.753275+0000","last_change":"2024-09-11T08:26:27.765017+0000","last_active":"2024-09-11T08:27:17.753275+0000","last_peered":"2024-09-11T08:27:17.753275+0000","last_clean":"2024-09-11T08:27:17.753275+0000","last_became_active":"2024-09-11T08:26:27.764671+0000","last_became_peered":"2024-09-11T08:26:27.764671+0000","last_unstale":"2024-09-11T08:27:17.753275+0000","last_undegraded":"2024-09-11T08:27:17.753275+0000","last_fullsized":"2024-09-11T08:27:17.753275+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:26:26.740307+0000","objects_scrubbed":0,"log_size":4444,"log_dups_size":0,"ondisk_log_size":4444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:47.926361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":291241984,"num_objects":4444,"num_object_clones":0,"num_object_copies":13332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4444,"num_write_kb":284416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"239'28","reported_seq":365,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.869021+0000","last_change":"2024-09-11T08:27:10.868771+0000","last_active":"2024-09-11T08:27:10.869021+0000","last_peered":"2024-09-11T08:27:10.869021+0000","last_clean":"2024-09-11T08:27:10.869021+0000","last_became_active":"2024-09-11T08:24:28.358175+0000","last_became_peered":"2024-09-11T08:24:28.358175+0000","last_unstale":"2024-09-11T08:27:10.869021+0000","last_undegraded":"2024-09-11T08:27:10.869021+0000","last_fullsized":"2024-09-11T08:27:10.869021+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'28","last_scrub_stamp":"2024-09-11T08:27:10.868676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:27:10.868676+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:26.816074+0000","scrub_duration":0.0026168770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":424,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-11T08:27:07.898398+0000","last_change":"2024-09-11T08:25:51.199008+0000","last_active":"2024-09-11T08:27:07.898398+0000","last_peered":"2024-09-11T08:27:07.898398+0000","last_clean":"2024-09-11T08:27:07.898398+0000","last_became_active":"2024-09-11T08:07:30.050484+0000","last_became_peered":"2024-09-11T08:07:30.050484+0000","last_unstale":"2024-09-11T08:27:07.898398+0000","last_undegraded":"2024-09-11T08:27:07.898398+0000","last_fullsized":"2024-09-11T08:27:07.898398+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:25:51.198956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:47.960329+0000","last_clean_scrub_stamp":"2024-09-11T08:25:51.198956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001153823,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"82'9","reported_seq":681,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:13.806145+0000","last_change":"2024-09-11T08:24:14.087666+0000","last_active":"2024-09-11T08:27:13.806145+0000","last_peered":"2024-09-11T08:27:13.806145+0000","last_clean":"2024-09-11T08:27:13.806145+0000","last_became_active":"2024-09-11T08:07:34.056143+0000","last_became_peered":"2024-09-11T08:07:34.056143+0000","last_unstale":"2024-09-11T08:27:13.806145+0000","last_undegraded":"2024-09-11T08:27:13.806145+0000","last_fullsized":"2024-09-11T08:27:13.806145+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-11T08:24:14.087574+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-09-11T08:21:00.841426+0000","last_clean_scrub_stamp":"2024-09-11T08:24:14.087574+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043101909999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":400,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.728403+0000","last_change":"2024-09-11T08:24:24.021006+0000","last_active":"2024-09-11T08:27:10.728403+0000","last_peered":"2024-09-11T08:27:10.728403+0000","last_clean":"2024-09-11T08:27:10.728403+0000","last_became_active":"2024-09-11T08:14:24.788968+0000","last_became_peered":"2024-09-11T08:14:24.788968+0000","last_unstale":"2024-09-11T08:27:10.728403+0000","last_undegraded":"2024-09-11T08:27:10.728403+0000","last_fullsized":"2024-09-11T08:27:10.728403+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:24.020925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:24:24.020925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010911288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.d","version":"396'4375","reported_seq":4397,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-09-11T08:27:18.503206+0000","last_change":"2024-09-11T08:26:27.763488+0000","last_active":"2024-09-11T08:27:18.503206+0000","last_peered":"2024-09-11T08:27:18.503206+0000","last_clean":"2024-09-11T08:27:18.503206+0000","last_became_active":"2024-09-11T08:26:27.763225+0000","last_became_peered":"2024-09-11T08:26:27.763225+0000","last_unstale":"2024-09-11T08:27:18.503206+0000","last_undegraded":"2024-09-11T08:27:18.503206+0000","last_fullsized":"2024-09-11T08:27:18.503206+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:26:26.740307+0000","objects_scrubbed":0,"log_size":4375,"log_dups_size":0,"ondisk_log_size":4375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:56.503503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":286720000,"num_objects":4375,"num_object_clones":0,"num_object_copies":13125,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4375,"num_write_kb":280000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"239'20","reported_seq":523,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.734869+0000","last_change":"2024-09-11T08:26:57.866443+0000","last_active":"2024-09-11T08:27:10.734869+0000","last_peered":"2024-09-11T08:27:10.734869+0000","last_clean":"2024-09-11T08:27:10.734869+0000","last_became_active":"2024-09-11T08:24:28.355516+0000","last_became_peered":"2024-09-11T08:24:28.355516+0000","last_unstale":"2024-09-11T08:27:10.734869+0000","last_undegraded":"2024-09-11T08:27:10.734869+0000","last_fullsized":"2024-09-11T08:27:10.734869+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'20","last_scrub_stamp":"2024-09-11T08:26:57.866352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:26:57.866352+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:03.651194+0000","scrub_duration":0.0030126530000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":136,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.734453+0000","last_change":"2024-09-11T08:26:02.614677+0000","last_active":"2024-09-11T08:27:10.734453+0000","last_peered":"2024-09-11T08:27:10.734453+0000","last_clean":"2024-09-11T08:27:10.734453+0000","last_became_active":"2024-09-11T08:24:28.352741+0000","last_became_peered":"2024-09-11T08:24:28.352741+0000","last_unstale":"2024-09-11T08:27:10.734453+0000","last_undegraded":"2024-09-11T08:27:10.734453+0000","last_fullsized":"2024-09-11T08:27:10.734453+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:02.614567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:26:02.614567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018716119999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"376'15","reported_seq":366,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:13.805434+0000","last_change":"2024-09-11T08:26:02.786822+0000","last_active":"2024-09-11T08:27:13.805434+0000","last_peered":"2024-09-11T08:27:13.805434+0000","last_clean":"2024-09-11T08:27:13.805434+0000","last_became_active":"2024-09-11T08:24:28.359787+0000","last_became_peered":"2024-09-11T08:24:28.359787+0000","last_unstale":"2024-09-11T08:27:13.805434+0000","last_undegraded":"2024-09-11T08:27:13.805434+0000","last_fullsized":"2024-09-11T08:27:13.805434+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'15","last_scrub_stamp":"2024-09-11T08:26:02.786767+0000","last_deep_scrub":"376'15","last_deep_scrub_stamp":"2024-09-11T08:26:02.786767+0000","last_clean_scrub_stamp":"2024-09-11T08:26:02.786767+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023292619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"66'1","reported_seq":402,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-11T08:27:07.897426+0000","last_change":"2024-09-11T08:25:54.225938+0000","last_active":"2024-09-11T08:27:07.897426+0000","last_peered":"2024-09-11T08:27:07.897426+0000","last_clean":"2024-09-11T08:27:07.897426+0000","last_became_active":"2024-09-11T08:07:36.072593+0000","last_became_peered":"2024-09-11T08:07:36.072593+0000","last_unstale":"2024-09-11T08:27:07.897426+0000","last_undegraded":"2024-09-11T08:27:07.897426+0000","last_fullsized":"2024-09-11T08:27:07.897426+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-11T08:22:04.543159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:22:04.543159+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002903625,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"396'4403","reported_seq":4425,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-09-11T08:27:18.793293+0000","last_change":"2024-09-11T08:26:27.765203+0000","last_active":"2024-09-11T08:27:18.793293+0000","last_peered":"2024-09-11T08:27:18.793293+0000","last_clean":"2024-09-11T08:27:18.793293+0000","last_became_active":"2024-09-11T08:26:27.764927+0000","last_became_peered":"2024-09-11T08:26:27.764927+0000","last_unstale":"2024-09-11T08:27:18.793293+0000","last_undegraded":"2024-09-11T08:27:18.793293+0000","last_fullsized":"2024-09-11T08:27:18.793293+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:26:26.740307+0000","objects_scrubbed":0,"log_size":4403,"log_dups_size":0,"ondisk_log_size":4403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:29.693074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288555008,"num_objects":4403,"num_object_clones":0,"num_object_copies":13209,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4403,"num_write_kb":281792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"239'29","reported_seq":592,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-11T08:27:07.898394+0000","last_change":"2024-09-11T08:25:48.174622+0000","last_active":"2024-09-11T08:27:07.898394+0000","last_peered":"2024-09-11T08:27:07.898394+0000","last_clean":"2024-09-11T08:27:07.898394+0000","last_became_active":"2024-09-11T08:24:28.364043+0000","last_became_peered":"2024-09-11T08:24:28.364043+0000","last_unstale":"2024-09-11T08:27:07.898394+0000","last_undegraded":"2024-09-11T08:27:07.898394+0000","last_fullsized":"2024-09-11T08:27:07.898394+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'29","last_scrub_stamp":"2024-09-11T08:21:44.561906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:21:44.561906+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024798810000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":481,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.735890+0000","last_change":"2024-09-11T08:26:55.920593+0000","last_active":"2024-09-11T08:27:10.735890+0000","last_peered":"2024-09-11T08:27:10.735890+0000","last_clean":"2024-09-11T08:27:10.735890+0000","last_became_active":"2024-09-11T08:24:28.358796+0000","last_became_peered":"2024-09-11T08:24:28.358796+0000","last_unstale":"2024-09-11T08:27:10.735890+0000","last_undegraded":"2024-09-11T08:27:10.735890+0000","last_fullsized":"2024-09-11T08:27:10.735890+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:55.920512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:26:55.920512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:16.956014+0000","scrub_duration":0.011546753999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":478,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.732401+0000","last_change":"2024-09-11T08:27:03.828113+0000","last_active":"2024-09-11T08:27:10.732401+0000","last_peered":"2024-09-11T08:27:10.732401+0000","last_clean":"2024-09-11T08:27:10.732401+0000","last_became_active":"2024-09-11T08:24:28.360235+0000","last_became_peered":"2024-09-11T08:24:28.360235+0000","last_unstale":"2024-09-11T08:27:10.732401+0000","last_undegraded":"2024-09-11T08:27:10.732401+0000","last_fullsized":"2024-09-11T08:27:10.732401+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:03.828043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:27:03.828043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:33.026945+0000","scrub_duration":0.0023665330000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":429,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.734326+0000","last_change":"2024-09-11T08:24:24.506494+0000","last_active":"2024-09-11T08:27:10.734326+0000","last_peered":"2024-09-11T08:27:10.734326+0000","last_clean":"2024-09-11T08:27:10.734326+0000","last_became_active":"2024-09-11T08:07:36.631668+0000","last_became_peered":"2024-09-11T08:07:36.631668+0000","last_unstale":"2024-09-11T08:27:10.734326+0000","last_undegraded":"2024-09-11T08:27:10.734326+0000","last_fullsized":"2024-09-11T08:27:10.734326+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:24.506402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:58.517624+0000","last_clean_scrub_stamp":"2024-09-11T08:24:24.506402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0096043550000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.b","version":"396'4425","reported_seq":4447,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-09-11T08:27:18.513168+0000","last_change":"2024-09-11T08:26:27.764606+0000","last_active":"2024-09-11T08:27:18.513168+0000","last_peered":"2024-09-11T08:27:18.513168+0000","last_clean":"2024-09-11T08:27:18.513168+0000","last_became_active":"2024-09-11T08:26:27.764354+0000","last_became_peered":"2024-09-11T08:26:27.764354+0000","last_unstale":"2024-09-11T08:27:18.513168+0000","last_undegraded":"2024-09-11T08:27:18.513168+0000","last_fullsized":"2024-09-11T08:27:18.513168+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:26:26.740307+0000","objects_scrubbed":0,"log_size":4425,"log_dups_size":0,"ondisk_log_size":4425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:56.383464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289996800,"num_objects":4425,"num_object_clones":0,"num_object_copies":13275,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4425,"num_write_kb":283200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"296'52","reported_seq":239,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-09-11T08:27:14.955157+0000","last_change":"2024-09-11T08:24:13.567684+0000","last_active":"2024-09-11T08:27:14.955157+0000","last_peered":"2024-09-11T08:27:14.955157+0000","last_clean":"2024-09-11T08:27:14.955157+0000","last_became_active":"2024-09-11T08:20:52.086843+0000","last_became_peered":"2024-09-11T08:20:52.086843+0000","last_unstale":"2024-09-11T08:27:14.955157+0000","last_undegraded":"2024-09-11T08:27:14.955157+0000","last_fullsized":"2024-09-11T08:27:14.955157+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'52","last_scrub_stamp":"2024-09-11T08:24:13.567595+0000","last_deep_scrub":"190'39","last_deep_scrub_stamp":"2024-09-11T08:15:52.847968+0000","last_clean_scrub_stamp":"2024-09-11T08:24:13.567595+0000","objects_scrubbed":10,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028629020000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"59'1","reported_seq":88,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-11T08:27:07.898532+0000","last_change":"2024-09-11T08:24:13.277960+0000","last_active":"2024-09-11T08:27:07.898532+0000","last_peered":"2024-09-11T08:27:07.898532+0000","last_clean":"2024-09-11T08:27:07.898532+0000","last_became_active":"2024-09-11T08:23:22.691726+0000","last_became_peered":"2024-09-11T08:23:22.691726+0000","last_unstale":"2024-09-11T08:27:07.898532+0000","last_undegraded":"2024-09-11T08:27:07.898532+0000","last_fullsized":"2024-09-11T08:27:07.898532+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-11T08:24:13.275269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:24:13.275269+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.007807863,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":145,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.734950+0000","last_change":"2024-09-11T08:24:29.448464+0000","last_active":"2024-09-11T08:27:10.734950+0000","last_peered":"2024-09-11T08:27:10.734950+0000","last_clean":"2024-09-11T08:27:10.734950+0000","last_became_active":"2024-09-11T08:24:28.362590+0000","last_became_peered":"2024-09-11T08:24:28.362590+0000","last_unstale":"2024-09-11T08:27:10.734950+0000","last_undegraded":"2024-09-11T08:27:10.734950+0000","last_fullsized":"2024-09-11T08:27:10.734950+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-11T08:24:29.448408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:24:29.448408+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0075800399999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":410,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-11T08:27:07.898149+0000","last_change":"2024-09-11T08:24:08.219265+0000","last_active":"2024-09-11T08:27:07.898149+0000","last_peered":"2024-09-11T08:27:07.898149+0000","last_clean":"2024-09-11T08:27:07.898149+0000","last_became_active":"2024-09-11T08:07:34.057154+0000","last_became_peered":"2024-09-11T08:07:34.057154+0000","last_unstale":"2024-09-11T08:27:07.898149+0000","last_undegraded":"2024-09-11T08:27:07.898149+0000","last_fullsized":"2024-09-11T08:27:07.898149+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:08.219182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:10:37.957096+0000","last_clean_scrub_stamp":"2024-09-11T08:24:08.219182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0084274109999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":447,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.733163+0000","last_change":"2024-09-11T08:24:16.032604+0000","last_active":"2024-09-11T08:27:10.733163+0000","last_peered":"2024-09-11T08:27:10.733163+0000","last_clean":"2024-09-11T08:27:10.733163+0000","last_became_active":"2024-09-11T08:15:56.525099+0000","last_became_peered":"2024-09-11T08:15:56.525099+0000","last_unstale":"2024-09-11T08:27:10.733163+0000","last_undegraded":"2024-09-11T08:27:10.733163+0000","last_fullsized":"2024-09-11T08:27:10.733163+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:16.032515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:24:16.032515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044030470000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"396'4484","reported_seq":4506,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-09-11T08:27:18.183095+0000","last_change":"2024-09-11T08:26:28.083045+0000","last_active":"2024-09-11T08:27:18.183095+0000","last_peered":"2024-09-11T08:27:18.183095+0000","last_clean":"2024-09-11T08:27:18.183095+0000","last_became_active":"2024-09-11T08:26:28.082722+0000","last_became_peered":"2024-09-11T08:26:28.082722+0000","last_unstale":"2024-09-11T08:27:18.183095+0000","last_undegraded":"2024-09-11T08:27:18.183095+0000","last_fullsized":"2024-09-11T08:27:18.183095+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:26:26.740307+0000","objects_scrubbed":0,"log_size":4484,"log_dups_size":0,"ondisk_log_size":4484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:56.502731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293863424,"num_objects":4484,"num_object_clones":0,"num_object_copies":13452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4484,"num_write_kb":286976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"239'17","reported_seq":536,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.737712+0000","last_change":"2024-09-11T08:26:56.891150+0000","last_active":"2024-09-11T08:27:10.737712+0000","last_peered":"2024-09-11T08:27:10.737712+0000","last_clean":"2024-09-11T08:27:10.737712+0000","last_became_active":"2024-09-11T08:24:28.358121+0000","last_became_peered":"2024-09-11T08:24:28.358121+0000","last_unstale":"2024-09-11T08:27:10.737712+0000","last_undegraded":"2024-09-11T08:27:10.737712+0000","last_fullsized":"2024-09-11T08:27:10.737712+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'17","last_scrub_stamp":"2024-09-11T08:26:56.890998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:26:56.890998+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:16.584252+0000","scrub_duration":0.012396804000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":448,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.734957+0000","last_change":"2024-09-11T08:26:10.658086+0000","last_active":"2024-09-11T08:27:10.734957+0000","last_peered":"2024-09-11T08:27:10.734957+0000","last_clean":"2024-09-11T08:27:10.734957+0000","last_became_active":"2024-09-11T08:07:30.047014+0000","last_became_peered":"2024-09-11T08:07:30.047014+0000","last_unstale":"2024-09-11T08:27:10.734957+0000","last_undegraded":"2024-09-11T08:27:10.734957+0000","last_fullsized":"2024-09-11T08:27:10.734957+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:10.658003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:26:10.658003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:16.143224+0000","scrub_duration":0.001689107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"127'2","reported_seq":465,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.741342+0000","last_change":"2024-09-11T08:26:53.737700+0000","last_active":"2024-09-11T08:27:10.741342+0000","last_peered":"2024-09-11T08:27:10.741342+0000","last_clean":"2024-09-11T08:27:10.741342+0000","last_became_active":"2024-09-11T08:07:30.045009+0000","last_became_peered":"2024-09-11T08:07:30.045009+0000","last_unstale":"2024-09-11T08:27:10.741342+0000","last_undegraded":"2024-09-11T08:27:10.741342+0000","last_fullsized":"2024-09-11T08:27:10.741342+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'2","last_scrub_stamp":"2024-09-11T08:26:53.737601+0000","last_deep_scrub":"127'2","last_deep_scrub_stamp":"2024-09-11T08:15:01.938497+0000","last_clean_scrub_stamp":"2024-09-11T08:26:53.737601+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:56.865318+0000","scrub_duration":0.0039636020000000001,"objects_trimmed":0,"snaptrim_duration":0.001859533,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":491,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.743101+0000","last_change":"2024-09-11T08:26:05.592197+0000","last_active":"2024-09-11T08:27:10.743101+0000","last_peered":"2024-09-11T08:27:10.743101+0000","last_clean":"2024-09-11T08:27:10.743101+0000","last_became_active":"2024-09-11T08:24:28.357141+0000","last_became_peered":"2024-09-11T08:24:28.357141+0000","last_unstale":"2024-09-11T08:27:10.743101+0000","last_undegraded":"2024-09-11T08:27:10.743101+0000","last_fullsized":"2024-09-11T08:27:10.743101+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:05.592132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:22:00.200872+0000","last_clean_scrub_stamp":"2024-09-11T08:26:05.592132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:31.177648+0000","scrub_duration":0.001603754,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":151,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.741966+0000","last_change":"2024-09-11T08:27:06.910328+0000","last_active":"2024-09-11T08:27:10.741966+0000","last_peered":"2024-09-11T08:27:10.741966+0000","last_clean":"2024-09-11T08:27:10.741966+0000","last_became_active":"2024-09-11T08:20:16.108674+0000","last_became_peered":"2024-09-11T08:20:16.108674+0000","last_unstale":"2024-09-11T08:27:10.741966+0000","last_undegraded":"2024-09-11T08:27:10.741966+0000","last_fullsized":"2024-09-11T08:27:10.741966+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:06.910177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:58.796217+0000","last_clean_scrub_stamp":"2024-09-11T08:27:06.910177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:10.778030+0000","scrub_duration":0.0054417340000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.9","version":"396'4377","reported_seq":4399,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-09-11T08:27:18.514935+0000","last_change":"2024-09-11T08:26:27.761702+0000","last_active":"2024-09-11T08:27:18.514935+0000","last_peered":"2024-09-11T08:27:18.514935+0000","last_clean":"2024-09-11T08:27:18.514935+0000","last_became_active":"2024-09-11T08:26:27.761414+0000","last_became_peered":"2024-09-11T08:26:27.761414+0000","last_unstale":"2024-09-11T08:27:18.514935+0000","last_undegraded":"2024-09-11T08:27:18.514935+0000","last_fullsized":"2024-09-11T08:27:18.514935+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:26:26.740307+0000","objects_scrubbed":0,"log_size":4377,"log_dups_size":0,"ondisk_log_size":4377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:40.911723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":286851072,"num_objects":4377,"num_object_clones":0,"num_object_copies":13131,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4377,"num_write_kb":280128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"287'30","reported_seq":464,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-11T08:27:07.898249+0000","last_change":"2024-09-11T08:26:09.030251+0000","last_active":"2024-09-11T08:27:07.898249+0000","last_peered":"2024-09-11T08:27:07.898249+0000","last_clean":"2024-09-11T08:27:07.898249+0000","last_became_active":"2024-09-11T08:07:32.055407+0000","last_became_peered":"2024-09-11T08:07:32.055407+0000","last_unstale":"2024-09-11T08:27:07.898249+0000","last_undegraded":"2024-09-11T08:27:07.898249+0000","last_fullsized":"2024-09-11T08:27:07.898249+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"287'30","last_scrub_stamp":"2024-09-11T08:26:09.030192+0000","last_deep_scrub":"287'30","last_deep_scrub_stamp":"2024-09-11T08:26:09.030192+0000","last_clean_scrub_stamp":"2024-09-11T08:26:09.030192+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:18.828125+0000","scrub_duration":0.0020634899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"59'1","reported_seq":421,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-11T08:27:07.898029+0000","last_change":"2024-09-11T08:23:34.954017+0000","last_active":"2024-09-11T08:27:07.898029+0000","last_peered":"2024-09-11T08:27:07.898029+0000","last_clean":"2024-09-11T08:27:07.898029+0000","last_became_active":"2024-09-11T08:07:30.050504+0000","last_became_peered":"2024-09-11T08:07:30.050504+0000","last_unstale":"2024-09-11T08:27:07.898029+0000","last_undegraded":"2024-09-11T08:27:07.898029+0000","last_fullsized":"2024-09-11T08:27:07.898029+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-11T08:23:34.953923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:23:34.953923+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035255019999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"224'6","reported_seq":1391,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-09-11T08:27:17.165280+0000","last_change":"2024-09-11T08:24:20.011482+0000","last_active":"2024-09-11T08:27:17.165280+0000","last_peered":"2024-09-11T08:27:17.165280+0000","last_clean":"2024-09-11T08:27:17.165280+0000","last_became_active":"2024-09-11T08:14:24.793048+0000","last_became_peered":"2024-09-11T08:14:24.793048+0000","last_unstale":"2024-09-11T08:27:17.165280+0000","last_undegraded":"2024-09-11T08:27:17.165280+0000","last_fullsized":"2024-09-11T08:27:17.165280+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'6","last_scrub_stamp":"2024-09-11T08:24:20.011383+0000","last_deep_scrub":"224'6","last_deep_scrub_stamp":"2024-09-11T08:16:55.778284+0000","last_clean_scrub_stamp":"2024-09-11T08:24:20.011383+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0060728680000000004,"objects_trimmed":0,"snaptrim_duration":0.000318821,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":779,"num_read_kb":774,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":138,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:14.117586+0000","last_change":"2024-09-11T08:27:14.117515+0000","last_active":"2024-09-11T08:27:14.117586+0000","last_peered":"2024-09-11T08:27:14.117586+0000","last_clean":"2024-09-11T08:27:14.117586+0000","last_became_active":"2024-09-11T08:24:28.353018+0000","last_became_peered":"2024-09-11T08:24:28.353018+0000","last_unstale":"2024-09-11T08:27:14.117586+0000","last_undegraded":"2024-09-11T08:27:14.117586+0000","last_fullsized":"2024-09-11T08:27:14.117586+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:14.114373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:27:14.114373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:20.579280+0000","scrub_duration":0.0066634249999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":482,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:11.028871+0000","last_change":"2024-09-11T08:27:11.025202+0000","last_active":"2024-09-11T08:27:11.028871+0000","last_peered":"2024-09-11T08:27:11.028871+0000","last_clean":"2024-09-11T08:27:11.028871+0000","last_became_active":"2024-09-11T08:24:28.358723+0000","last_became_peered":"2024-09-11T08:24:28.358723+0000","last_unstale":"2024-09-11T08:27:11.028871+0000","last_undegraded":"2024-09-11T08:27:11.028871+0000","last_fullsized":"2024-09-11T08:27:11.028871+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:11.025109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:27:11.025109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:23.447076+0000","scrub_duration":0.0049163100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"396'4437","reported_seq":4459,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-09-11T08:27:18.124820+0000","last_change":"2024-09-11T08:26:28.083341+0000","last_active":"2024-09-11T08:27:18.124820+0000","last_peered":"2024-09-11T08:27:18.124820+0000","last_clean":"2024-09-11T08:27:18.124820+0000","last_became_active":"2024-09-11T08:26:28.082996+0000","last_became_peered":"2024-09-11T08:26:28.082996+0000","last_unstale":"2024-09-11T08:27:18.124820+0000","last_undegraded":"2024-09-11T08:27:18.124820+0000","last_fullsized":"2024-09-11T08:27:18.124820+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:26:26.740307+0000","objects_scrubbed":0,"log_size":4437,"log_dups_size":0,"ondisk_log_size":4437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:50.235347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":290783232,"num_objects":4437,"num_object_clones":0,"num_object_copies":13311,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4437,"num_write_kb":283968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"239'33","reported_seq":507,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-09-11T08:27:15.771139+0000","last_change":"2024-09-11T08:27:15.771100+0000","last_active":"2024-09-11T08:27:15.771139+0000","last_peered":"2024-09-11T08:27:15.771139+0000","last_clean":"2024-09-11T08:27:15.771139+0000","last_became_active":"2024-09-11T08:14:24.789154+0000","last_became_peered":"2024-09-11T08:14:24.789154+0000","last_unstale":"2024-09-11T08:27:15.771139+0000","last_undegraded":"2024-09-11T08:27:15.771139+0000","last_fullsized":"2024-09-11T08:27:15.771139+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'33","last_scrub_stamp":"2024-09-11T08:27:15.770971+0000","last_deep_scrub":"84'21","last_deep_scrub_stamp":"2024-09-11T08:08:56.705306+0000","last_clean_scrub_stamp":"2024-09-11T08:27:15.770971+0000","objects_scrubbed":7,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:20.921251+0000","scrub_duration":0.0065168980000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":130,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.732882+0000","last_change":"2024-09-11T08:26:07.749930+0000","last_active":"2024-09-11T08:27:10.732882+0000","last_peered":"2024-09-11T08:27:10.732882+0000","last_clean":"2024-09-11T08:27:10.732882+0000","last_became_active":"2024-09-11T08:24:28.362417+0000","last_became_peered":"2024-09-11T08:24:28.362417+0000","last_unstale":"2024-09-11T08:27:10.732882+0000","last_undegraded":"2024-09-11T08:27:10.732882+0000","last_fullsized":"2024-09-11T08:27:10.732882+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:07.749828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:07.749828+0000","last_clean_scrub_stamp":"2024-09-11T08:26:07.749828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:35.713503+0000","scrub_duration":0.0015131179999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":244,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.727631+0000","last_change":"2024-09-11T08:23:37.276623+0000","last_active":"2024-09-11T08:27:10.727631+0000","last_peered":"2024-09-11T08:27:10.727631+0000","last_clean":"2024-09-11T08:27:10.727631+0000","last_became_active":"2024-09-11T08:15:56.523430+0000","last_became_peered":"2024-09-11T08:15:56.523430+0000","last_unstale":"2024-09-11T08:27:10.727631+0000","last_undegraded":"2024-09-11T08:27:10.727631+0000","last_fullsized":"2024-09-11T08:27:10.727631+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-11T08:23:37.276533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:05:21.231856+0000","last_clean_scrub_stamp":"2024-09-11T08:23:37.276533+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0089351699999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":268,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.727858+0000","last_change":"2024-09-11T08:25:55.938265+0000","last_active":"2024-09-11T08:27:10.727858+0000","last_peered":"2024-09-11T08:27:10.727858+0000","last_clean":"2024-09-11T08:27:10.727858+0000","last_became_active":"2024-09-11T08:24:28.357166+0000","last_became_peered":"2024-09-11T08:24:28.357166+0000","last_unstale":"2024-09-11T08:27:10.727858+0000","last_undegraded":"2024-09-11T08:27:10.727858+0000","last_fullsized":"2024-09-11T08:27:10.727858+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:22:48.591411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:20:25.775869+0000","last_clean_scrub_stamp":"2024-09-11T08:22:48.591411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0087669500000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":185,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:12.366844+0000","last_change":"2024-09-11T08:27:12.366790+0000","last_active":"2024-09-11T08:27:12.366844+0000","last_peered":"2024-09-11T08:27:12.366844+0000","last_clean":"2024-09-11T08:27:12.366844+0000","last_became_active":"2024-09-11T08:24:28.356071+0000","last_became_peered":"2024-09-11T08:24:28.356071+0000","last_unstale":"2024-09-11T08:27:12.366844+0000","last_undegraded":"2024-09-11T08:27:12.366844+0000","last_fullsized":"2024-09-11T08:27:12.366844+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:12.366709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:01.777429+0000","last_clean_scrub_stamp":"2024-09-11T08:27:12.366709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:12.647850+0000","scrub_duration":0.0043386370000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"396'4352","reported_seq":4373,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-09-11T08:27:18.187437+0000","last_change":"2024-09-11T08:26:27.764691+0000","last_active":"2024-09-11T08:27:18.187437+0000","last_peered":"2024-09-11T08:27:18.187437+0000","last_clean":"2024-09-11T08:27:18.187437+0000","last_became_active":"2024-09-11T08:26:27.763926+0000","last_became_peered":"2024-09-11T08:26:27.763926+0000","last_unstale":"2024-09-11T08:27:18.187437+0000","last_undegraded":"2024-09-11T08:27:18.187437+0000","last_fullsized":"2024-09-11T08:27:18.187437+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:26:26.740307+0000","objects_scrubbed":0,"log_size":4352,"log_dups_size":0,"ondisk_log_size":4352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:27.656690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":285212672,"num_objects":4352,"num_object_clones":0,"num_object_copies":13056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4352,"num_write_kb":278528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"239'24","reported_seq":359,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.732916+0000","last_change":"2024-09-11T08:26:59.682857+0000","last_active":"2024-09-11T08:27:10.732916+0000","last_peered":"2024-09-11T08:27:10.732916+0000","last_clean":"2024-09-11T08:27:10.732916+0000","last_became_active":"2024-09-11T08:24:28.368759+0000","last_became_peered":"2024-09-11T08:24:28.368759+0000","last_unstale":"2024-09-11T08:27:10.732916+0000","last_undegraded":"2024-09-11T08:27:10.732916+0000","last_fullsized":"2024-09-11T08:27:10.732916+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'24","last_scrub_stamp":"2024-09-11T08:26:59.682768+0000","last_deep_scrub":"88'16","last_deep_scrub_stamp":"2024-09-11T08:09:39.635984+0000","last_clean_scrub_stamp":"2024-09-11T08:26:59.682768+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:17.305928+0000","scrub_duration":0.0052817990000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":442,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.729123+0000","last_change":"2024-09-11T08:26:06.958652+0000","last_active":"2024-09-11T08:27:10.729123+0000","last_peered":"2024-09-11T08:27:10.729123+0000","last_clean":"2024-09-11T08:27:10.729123+0000","last_became_active":"2024-09-11T08:14:24.789888+0000","last_became_peered":"2024-09-11T08:14:24.789888+0000","last_unstale":"2024-09-11T08:27:10.729123+0000","last_undegraded":"2024-09-11T08:27:10.729123+0000","last_fullsized":"2024-09-11T08:27:10.729123+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:06.958598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:36.417911+0000","last_clean_scrub_stamp":"2024-09-11T08:26:06.958598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:31.023729+0000","scrub_duration":0.001521522,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":122,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.733060+0000","last_change":"2024-09-11T08:26:00.761197+0000","last_active":"2024-09-11T08:27:10.733060+0000","last_peered":"2024-09-11T08:27:10.733060+0000","last_clean":"2024-09-11T08:27:10.733060+0000","last_became_active":"2024-09-11T08:24:28.361261+0000","last_became_peered":"2024-09-11T08:24:28.361261+0000","last_unstale":"2024-09-11T08:27:10.733060+0000","last_undegraded":"2024-09-11T08:27:10.733060+0000","last_fullsized":"2024-09-11T08:27:10.733060+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:00.761139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:01.558340+0000","last_clean_scrub_stamp":"2024-09-11T08:26:00.761139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001250099,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":260,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.732726+0000","last_change":"2024-09-11T08:27:00.720851+0000","last_active":"2024-09-11T08:27:10.732726+0000","last_peered":"2024-09-11T08:27:10.732726+0000","last_clean":"2024-09-11T08:27:10.732726+0000","last_became_active":"2024-09-11T08:24:28.357842+0000","last_became_peered":"2024-09-11T08:24:28.357842+0000","last_unstale":"2024-09-11T08:27:10.732726+0000","last_undegraded":"2024-09-11T08:27:10.732726+0000","last_fullsized":"2024-09-11T08:27:10.732726+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:00.720766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:27:00.720766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:21.874998+0000","scrub_duration":0.001951798,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"396'4512","reported_seq":4534,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-09-11T08:27:18.476991+0000","last_change":"2024-09-11T08:26:27.763889+0000","last_active":"2024-09-11T08:27:18.476991+0000","last_peered":"2024-09-11T08:27:18.476991+0000","last_clean":"2024-09-11T08:27:18.476991+0000","last_became_active":"2024-09-11T08:26:27.763602+0000","last_became_peered":"2024-09-11T08:26:27.763602+0000","last_unstale":"2024-09-11T08:27:18.476991+0000","last_undegraded":"2024-09-11T08:27:18.476991+0000","last_fullsized":"2024-09-11T08:27:18.476991+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:26:26.740307+0000","objects_scrubbed":0,"log_size":4512,"log_dups_size":0,"ondisk_log_size":4512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:29.384434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":295698432,"num_objects":4512,"num_object_clones":0,"num_object_copies":13536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4512,"num_write_kb":288768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"239'39","reported_seq":401,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.742202+0000","last_change":"2024-09-11T08:26:08.526760+0000","last_active":"2024-09-11T08:27:10.742202+0000","last_peered":"2024-09-11T08:27:10.742202+0000","last_clean":"2024-09-11T08:27:10.742202+0000","last_became_active":"2024-09-11T08:23:22.681452+0000","last_became_peered":"2024-09-11T08:23:22.681452+0000","last_unstale":"2024-09-11T08:27:10.742202+0000","last_undegraded":"2024-09-11T08:27:10.742202+0000","last_fullsized":"2024-09-11T08:27:10.742202+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'39","last_scrub_stamp":"2024-09-11T08:26:08.526702+0000","last_deep_scrub":"77'25","last_deep_scrub_stamp":"2024-09-11T08:08:32.458210+0000","last_clean_scrub_stamp":"2024-09-11T08:26:08.526702+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:37.269972+0000","scrub_duration":0.0021726229999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":441,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.732578+0000","last_change":"2024-09-11T08:24:06.575259+0000","last_active":"2024-09-11T08:27:10.732578+0000","last_peered":"2024-09-11T08:27:10.732578+0000","last_clean":"2024-09-11T08:27:10.732578+0000","last_became_active":"2024-09-11T08:07:30.042555+0000","last_became_peered":"2024-09-11T08:07:30.042555+0000","last_unstale":"2024-09-11T08:27:10.732578+0000","last_undegraded":"2024-09-11T08:27:10.732578+0000","last_fullsized":"2024-09-11T08:27:10.732578+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:06.575206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:24:06.575206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014429386000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":431,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.735541+0000","last_change":"2024-09-11T08:24:08.512741+0000","last_active":"2024-09-11T08:27:10.735541+0000","last_peered":"2024-09-11T08:27:10.735541+0000","last_clean":"2024-09-11T08:27:10.735541+0000","last_became_active":"2024-09-11T08:07:34.054961+0000","last_became_peered":"2024-09-11T08:07:34.054961+0000","last_unstale":"2024-09-11T08:27:10.735541+0000","last_undegraded":"2024-09-11T08:27:10.735541+0000","last_fullsized":"2024-09-11T08:27:10.735541+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:08.512659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:24:08.512659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010695556,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":459,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:11.873385+0000","last_change":"2024-09-11T08:27:11.873295+0000","last_active":"2024-09-11T08:27:11.873385+0000","last_peered":"2024-09-11T08:27:11.873385+0000","last_clean":"2024-09-11T08:27:11.873385+0000","last_became_active":"2024-09-11T08:07:36.071194+0000","last_became_peered":"2024-09-11T08:07:36.071194+0000","last_unstale":"2024-09-11T08:27:11.873385+0000","last_undegraded":"2024-09-11T08:27:11.873385+0000","last_fullsized":"2024-09-11T08:27:11.873385+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:11.873150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:07.930137+0000","last_clean_scrub_stamp":"2024-09-11T08:27:11.873150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:35.375212+0000","scrub_duration":0.0065665899999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"396'4336","reported_seq":4358,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-09-11T08:27:18.170328+0000","last_change":"2024-09-11T08:26:27.764903+0000","last_active":"2024-09-11T08:27:18.170328+0000","last_peered":"2024-09-11T08:27:18.170328+0000","last_clean":"2024-09-11T08:27:18.170328+0000","last_became_active":"2024-09-11T08:26:27.764638+0000","last_became_peered":"2024-09-11T08:26:27.764638+0000","last_unstale":"2024-09-11T08:27:18.170328+0000","last_undegraded":"2024-09-11T08:27:18.170328+0000","last_fullsized":"2024-09-11T08:27:18.170328+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:26:26.740307+0000","objects_scrubbed":0,"log_size":4336,"log_dups_size":0,"ondisk_log_size":4336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:38.288480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284164096,"num_objects":4336,"num_object_clones":0,"num_object_copies":13008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4336,"num_write_kb":277504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"239'25","reported_seq":341,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.732591+0000","last_change":"2024-09-11T08:24:24.576788+0000","last_active":"2024-09-11T08:27:10.732591+0000","last_peered":"2024-09-11T08:27:10.732591+0000","last_clean":"2024-09-11T08:27:10.732591+0000","last_became_active":"2024-09-11T08:23:22.688631+0000","last_became_peered":"2024-09-11T08:23:22.688631+0000","last_unstale":"2024-09-11T08:27:10.732591+0000","last_undegraded":"2024-09-11T08:27:10.732591+0000","last_fullsized":"2024-09-11T08:27:10.732591+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'25","last_scrub_stamp":"2024-09-11T08:24:24.576699+0000","last_deep_scrub":"224'15","last_deep_scrub_stamp":"2024-09-11T08:17:01.126478+0000","last_clean_scrub_stamp":"2024-09-11T08:24:24.576699+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003559857,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":199,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-11T08:27:07.898660+0000","last_change":"2024-09-11T08:22:54.830543+0000","last_active":"2024-09-11T08:27:07.898660+0000","last_peered":"2024-09-11T08:27:07.898660+0000","last_clean":"2024-09-11T08:27:07.898660+0000","last_became_active":"2024-09-11T08:17:37.557278+0000","last_became_peered":"2024-09-11T08:17:37.557278+0000","last_unstale":"2024-09-11T08:27:07.898660+0000","last_undegraded":"2024-09-11T08:27:07.898660+0000","last_fullsized":"2024-09-11T08:27:07.898660+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:22:54.830465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:45.979847+0000","last_clean_scrub_stamp":"2024-09-11T08:22:54.830465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0062432310000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"354'16","reported_seq":757,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:13.809503+0000","last_change":"2024-09-11T08:27:12.918897+0000","last_active":"2024-09-11T08:27:13.809503+0000","last_peered":"2024-09-11T08:27:13.809503+0000","last_clean":"2024-09-11T08:27:13.809503+0000","last_became_active":"2024-09-11T08:24:28.356749+0000","last_became_peered":"2024-09-11T08:24:28.356749+0000","last_unstale":"2024-09-11T08:27:13.809503+0000","last_undegraded":"2024-09-11T08:27:13.809503+0000","last_fullsized":"2024-09-11T08:27:13.809503+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'16","last_scrub_stamp":"2024-09-11T08:27:12.918714+0000","last_deep_scrub":"171'12","last_deep_scrub_stamp":"2024-09-11T08:14:56.881299+0000","last_clean_scrub_stamp":"2024-09-11T08:27:12.918714+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:14.209536+0000","scrub_duration":0.0038742020000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":90,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.736061+0000","last_change":"2024-09-11T08:26:06.582469+0000","last_active":"2024-09-11T08:27:10.736061+0000","last_peered":"2024-09-11T08:27:10.736061+0000","last_clean":"2024-09-11T08:27:10.736061+0000","last_became_active":"2024-09-11T08:23:22.686992+0000","last_became_peered":"2024-09-11T08:23:22.686992+0000","last_unstale":"2024-09-11T08:27:10.736061+0000","last_undegraded":"2024-09-11T08:27:10.736061+0000","last_fullsized":"2024-09-11T08:27:10.736061+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:06.582413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:14.613532+0000","last_clean_scrub_stamp":"2024-09-11T08:26:06.582413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:15.324916+0000","scrub_duration":0.0014342210000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.4","version":"396'4252","reported_seq":4274,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-09-11T08:27:18.179577+0000","last_change":"2024-09-11T08:26:27.765910+0000","last_active":"2024-09-11T08:27:18.179577+0000","last_peered":"2024-09-11T08:27:18.179577+0000","last_clean":"2024-09-11T08:27:18.179577+0000","last_became_active":"2024-09-11T08:26:27.765686+0000","last_became_peered":"2024-09-11T08:26:27.765686+0000","last_unstale":"2024-09-11T08:27:18.179577+0000","last_undegraded":"2024-09-11T08:27:18.179577+0000","last_fullsized":"2024-09-11T08:27:18.179577+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:26:26.740307+0000","objects_scrubbed":0,"log_size":4252,"log_dups_size":0,"ondisk_log_size":4252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:39.176596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278659072,"num_objects":4252,"num_object_clones":0,"num_object_copies":12756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4252,"num_write_kb":272128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"239'15","reported_seq":437,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-11T08:27:07.898311+0000","last_change":"2024-09-11T08:23:37.991614+0000","last_active":"2024-09-11T08:27:07.898311+0000","last_peered":"2024-09-11T08:27:07.898311+0000","last_clean":"2024-09-11T08:27:07.898311+0000","last_became_active":"2024-09-11T08:07:32.054885+0000","last_became_peered":"2024-09-11T08:07:32.054885+0000","last_unstale":"2024-09-11T08:27:07.898311+0000","last_undegraded":"2024-09-11T08:27:07.898311+0000","last_fullsized":"2024-09-11T08:27:07.898311+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'15","last_scrub_stamp":"2024-09-11T08:23:37.991518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:23:37.991518+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0096807339999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":136,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.732963+0000","last_change":"2024-09-11T08:27:03.600628+0000","last_active":"2024-09-11T08:27:10.732963+0000","last_peered":"2024-09-11T08:27:10.732963+0000","last_clean":"2024-09-11T08:27:10.732963+0000","last_became_active":"2024-09-11T08:24:28.359057+0000","last_became_peered":"2024-09-11T08:24:28.359057+0000","last_unstale":"2024-09-11T08:27:10.732963+0000","last_undegraded":"2024-09-11T08:27:10.732963+0000","last_fullsized":"2024-09-11T08:27:10.732963+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:03.600550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:27:03.600550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:14.549415+0000","scrub_duration":0.0052685750000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":461,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.740794+0000","last_change":"2024-09-11T08:27:09.839390+0000","last_active":"2024-09-11T08:27:10.740794+0000","last_peered":"2024-09-11T08:27:10.740794+0000","last_clean":"2024-09-11T08:27:10.740794+0000","last_became_active":"2024-09-11T08:07:34.056606+0000","last_became_peered":"2024-09-11T08:07:34.056606+0000","last_unstale":"2024-09-11T08:27:10.740794+0000","last_undegraded":"2024-09-11T08:27:10.740794+0000","last_fullsized":"2024-09-11T08:27:10.740794+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:09.839220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:27:09.839220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:10.408269+0000","scrub_duration":0.0042163340000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":435,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-11T08:27:07.897950+0000","last_change":"2024-09-11T08:23:46.203003+0000","last_active":"2024-09-11T08:27:07.897950+0000","last_peered":"2024-09-11T08:27:07.897950+0000","last_clean":"2024-09-11T08:27:07.897950+0000","last_became_active":"2024-09-11T08:15:56.518884+0000","last_became_peered":"2024-09-11T08:15:56.518884+0000","last_unstale":"2024-09-11T08:27:07.897950+0000","last_undegraded":"2024-09-11T08:27:07.897950+0000","last_fullsized":"2024-09-11T08:27:07.897950+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:23:46.202841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:18:32.278257+0000","last_clean_scrub_stamp":"2024-09-11T08:23:46.202841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014919458999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"396'4360","reported_seq":4382,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-09-11T08:27:18.204885+0000","last_change":"2024-09-11T08:26:28.083356+0000","last_active":"2024-09-11T08:27:18.204885+0000","last_peered":"2024-09-11T08:27:18.204885+0000","last_clean":"2024-09-11T08:27:18.204885+0000","last_became_active":"2024-09-11T08:26:28.083033+0000","last_became_peered":"2024-09-11T08:26:28.083033+0000","last_unstale":"2024-09-11T08:27:18.204885+0000","last_undegraded":"2024-09-11T08:27:18.204885+0000","last_fullsized":"2024-09-11T08:27:18.204885+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:26:26.740307+0000","objects_scrubbed":0,"log_size":4360,"log_dups_size":0,"ondisk_log_size":4360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:51.535683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":285736960,"num_objects":4360,"num_object_clones":0,"num_object_copies":13080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4360,"num_write_kb":279040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"239'35","reported_seq":575,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.732736+0000","last_change":"2024-09-11T08:27:07.886113+0000","last_active":"2024-09-11T08:27:10.732736+0000","last_peered":"2024-09-11T08:27:10.732736+0000","last_clean":"2024-09-11T08:27:10.732736+0000","last_became_active":"2024-09-11T08:15:56.526961+0000","last_became_peered":"2024-09-11T08:15:56.526961+0000","last_unstale":"2024-09-11T08:27:10.732736+0000","last_undegraded":"2024-09-11T08:27:10.732736+0000","last_fullsized":"2024-09-11T08:27:10.732736+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'35","last_scrub_stamp":"2024-09-11T08:27:07.886024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:27:07.886024+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:25.429274+0000","scrub_duration":0.0038597950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":278,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.727692+0000","last_change":"2024-09-11T08:27:02.188913+0000","last_active":"2024-09-11T08:27:10.727692+0000","last_peered":"2024-09-11T08:27:10.727692+0000","last_clean":"2024-09-11T08:27:10.727692+0000","last_became_active":"2024-09-11T08:24:28.353720+0000","last_became_peered":"2024-09-11T08:24:28.353720+0000","last_unstale":"2024-09-11T08:27:10.727692+0000","last_undegraded":"2024-09-11T08:27:10.727692+0000","last_fullsized":"2024-09-11T08:27:10.727692+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:02.188743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:44.385372+0000","last_clean_scrub_stamp":"2024-09-11T08:27:02.188743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:16.282589+0000","scrub_duration":0.002554925,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":481,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.741482+0000","last_change":"2024-09-11T08:26:06.546660+0000","last_active":"2024-09-11T08:27:10.741482+0000","last_peered":"2024-09-11T08:27:10.741482+0000","last_clean":"2024-09-11T08:27:10.741482+0000","last_became_active":"2024-09-11T08:24:28.365792+0000","last_became_peered":"2024-09-11T08:24:28.365792+0000","last_unstale":"2024-09-11T08:27:10.741482+0000","last_undegraded":"2024-09-11T08:27:10.741482+0000","last_fullsized":"2024-09-11T08:27:10.741482+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:06.546573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:06.546573+0000","last_clean_scrub_stamp":"2024-09-11T08:26:06.546573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:21.762385+0000","scrub_duration":0.0020691960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":238,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.727422+0000","last_change":"2024-09-11T08:24:26.983211+0000","last_active":"2024-09-11T08:27:10.727422+0000","last_peered":"2024-09-11T08:27:10.727422+0000","last_clean":"2024-09-11T08:27:10.727422+0000","last_became_active":"2024-09-11T08:15:56.518169+0000","last_became_peered":"2024-09-11T08:15:56.518169+0000","last_unstale":"2024-09-11T08:27:10.727422+0000","last_undegraded":"2024-09-11T08:27:10.727422+0000","last_fullsized":"2024-09-11T08:27:10.727422+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:26.983134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:24:26.983134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0081131989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"395'4097","reported_seq":4118,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:14.202667+0000","last_change":"2024-09-11T08:26:27.764924+0000","last_active":"2024-09-11T08:27:14.202667+0000","last_peered":"2024-09-11T08:27:14.202667+0000","last_clean":"2024-09-11T08:27:14.202667+0000","last_became_active":"2024-09-11T08:26:27.764636+0000","last_became_peered":"2024-09-11T08:26:27.764636+0000","last_unstale":"2024-09-11T08:27:14.202667+0000","last_undegraded":"2024-09-11T08:27:14.202667+0000","last_fullsized":"2024-09-11T08:27:14.202667+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:26:26.740307+0000","objects_scrubbed":0,"log_size":4097,"log_dups_size":0,"ondisk_log_size":4097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:28.732207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268500992,"num_objects":4097,"num_object_clones":0,"num_object_copies":12291,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4097,"num_write_kb":262208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"239'23","reported_seq":342,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.734456+0000","last_change":"2024-09-11T08:24:07.501579+0000","last_active":"2024-09-11T08:27:10.734456+0000","last_peered":"2024-09-11T08:27:10.734456+0000","last_clean":"2024-09-11T08:27:10.734456+0000","last_became_active":"2024-09-11T08:23:22.686387+0000","last_became_peered":"2024-09-11T08:23:22.686387+0000","last_unstale":"2024-09-11T08:27:10.734456+0000","last_undegraded":"2024-09-11T08:27:10.734456+0000","last_fullsized":"2024-09-11T08:27:10.734456+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'23","last_scrub_stamp":"2024-09-11T08:24:07.501504+0000","last_deep_scrub":"145'17","last_deep_scrub_stamp":"2024-09-11T08:13:08.160848+0000","last_clean_scrub_stamp":"2024-09-11T08:24:07.501504+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011071242,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":248,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.728087+0000","last_change":"2024-09-11T08:25:57.957773+0000","last_active":"2024-09-11T08:27:10.728087+0000","last_peered":"2024-09-11T08:27:10.728087+0000","last_clean":"2024-09-11T08:27:10.728087+0000","last_became_active":"2024-09-11T08:15:56.525845+0000","last_became_peered":"2024-09-11T08:15:56.525845+0000","last_unstale":"2024-09-11T08:27:10.728087+0000","last_undegraded":"2024-09-11T08:27:10.728087+0000","last_fullsized":"2024-09-11T08:27:10.728087+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:25:57.957725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:21:06.262572+0000","last_clean_scrub_stamp":"2024-09-11T08:25:57.957725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012472080000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":441,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.740987+0000","last_change":"2024-09-11T08:27:05.881925+0000","last_active":"2024-09-11T08:27:10.740987+0000","last_peered":"2024-09-11T08:27:10.740987+0000","last_clean":"2024-09-11T08:27:10.740987+0000","last_became_active":"2024-09-11T08:07:34.057479+0000","last_became_peered":"2024-09-11T08:07:34.057479+0000","last_unstale":"2024-09-11T08:27:10.740987+0000","last_undegraded":"2024-09-11T08:27:10.740987+0000","last_fullsized":"2024-09-11T08:27:10.740987+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:05.881761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:27:05.881761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:11.743757+0000","scrub_duration":0.004658255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":457,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-09-11T08:27:15.756360+0000","last_change":"2024-09-11T08:27:15.756321+0000","last_active":"2024-09-11T08:27:15.756360+0000","last_peered":"2024-09-11T08:27:15.756360+0000","last_clean":"2024-09-11T08:27:15.756360+0000","last_became_active":"2024-09-11T08:15:56.522624+0000","last_became_peered":"2024-09-11T08:15:56.522624+0000","last_unstale":"2024-09-11T08:27:15.756360+0000","last_undegraded":"2024-09-11T08:27:15.756360+0000","last_fullsized":"2024-09-11T08:27:15.756360+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:15.756244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:27:15.756244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:23.287193+0000","scrub_duration":0.015967552999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"396'4447","reported_seq":4469,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-09-11T08:27:18.755560+0000","last_change":"2024-09-11T08:26:28.083401+0000","last_active":"2024-09-11T08:27:18.755560+0000","last_peered":"2024-09-11T08:27:18.755560+0000","last_clean":"2024-09-11T08:27:18.755560+0000","last_became_active":"2024-09-11T08:26:28.083190+0000","last_became_peered":"2024-09-11T08:26:28.083190+0000","last_unstale":"2024-09-11T08:27:18.755560+0000","last_undegraded":"2024-09-11T08:27:18.755560+0000","last_fullsized":"2024-09-11T08:27:18.755560+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:26:26.740307+0000","objects_scrubbed":0,"log_size":4447,"log_dups_size":0,"ondisk_log_size":4447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:43.060889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":291438592,"num_objects":4447,"num_object_clones":0,"num_object_copies":13341,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4447,"num_write_kb":284608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"239'27","reported_seq":481,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:14.203807+0000","last_change":"2024-09-11T08:27:14.203688+0000","last_active":"2024-09-11T08:27:14.203807+0000","last_peered":"2024-09-11T08:27:14.203807+0000","last_clean":"2024-09-11T08:27:14.203807+0000","last_became_active":"2024-09-11T08:24:28.353397+0000","last_became_peered":"2024-09-11T08:24:28.353397+0000","last_unstale":"2024-09-11T08:27:14.203807+0000","last_undegraded":"2024-09-11T08:27:14.203807+0000","last_fullsized":"2024-09-11T08:27:14.203807+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'27","last_scrub_stamp":"2024-09-11T08:27:14.203596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:27:14.203596+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:17.865945+0000","scrub_duration":0.0026965510000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":244,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.728132+0000","last_change":"2024-09-11T08:24:17.089977+0000","last_active":"2024-09-11T08:27:10.728132+0000","last_peered":"2024-09-11T08:27:10.728132+0000","last_clean":"2024-09-11T08:27:10.728132+0000","last_became_active":"2024-09-11T08:15:56.515328+0000","last_became_peered":"2024-09-11T08:15:56.515328+0000","last_unstale":"2024-09-11T08:27:10.728132+0000","last_undegraded":"2024-09-11T08:27:10.728132+0000","last_fullsized":"2024-09-11T08:27:10.728132+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-11T08:24:17.089895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.089895+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019391429999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"209'11","reported_seq":378,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:13.805704+0000","last_change":"2024-09-11T08:25:59.008089+0000","last_active":"2024-09-11T08:27:13.805704+0000","last_peered":"2024-09-11T08:27:13.805704+0000","last_clean":"2024-09-11T08:27:13.805704+0000","last_became_active":"2024-09-11T08:15:56.524513+0000","last_became_peered":"2024-09-11T08:15:56.524513+0000","last_unstale":"2024-09-11T08:27:13.805704+0000","last_undegraded":"2024-09-11T08:27:13.805704+0000","last_fullsized":"2024-09-11T08:27:13.805704+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"209'11","last_scrub_stamp":"2024-09-11T08:25:59.008036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:25:59.008036+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014184130000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":126,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.732812+0000","last_change":"2024-09-11T08:26:06.772735+0000","last_active":"2024-09-11T08:27:10.732812+0000","last_peered":"2024-09-11T08:27:10.732812+0000","last_clean":"2024-09-11T08:27:10.732812+0000","last_became_active":"2024-09-11T08:24:28.360658+0000","last_became_peered":"2024-09-11T08:24:28.360658+0000","last_unstale":"2024-09-11T08:27:10.732812+0000","last_undegraded":"2024-09-11T08:27:10.732812+0000","last_fullsized":"2024-09-11T08:27:10.732812+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:06.772675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:03.484755+0000","last_clean_scrub_stamp":"2024-09-11T08:26:06.772675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:18.455364+0000","scrub_duration":0.0013072069999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":485,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.733516+0000","last_change":"2024-09-11T08:26:54.619464+0000","last_active":"2024-09-11T08:27:10.733516+0000","last_peered":"2024-09-11T08:27:10.733516+0000","last_clean":"2024-09-11T08:27:10.733516+0000","last_became_active":"2024-09-11T08:24:28.357401+0000","last_became_peered":"2024-09-11T08:24:28.357401+0000","last_unstale":"2024-09-11T08:27:10.733516+0000","last_undegraded":"2024-09-11T08:27:10.733516+0000","last_fullsized":"2024-09-11T08:27:10.733516+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:54.619370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:26:54.619370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:24.338015+0000","scrub_duration":0.0025718440000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"239'15","reported_seq":488,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.733631+0000","last_change":"2024-09-11T08:26:03.664732+0000","last_active":"2024-09-11T08:27:10.733631+0000","last_peered":"2024-09-11T08:27:10.733631+0000","last_clean":"2024-09-11T08:27:10.733631+0000","last_became_active":"2024-09-11T08:15:56.526325+0000","last_became_peered":"2024-09-11T08:15:56.526325+0000","last_unstale":"2024-09-11T08:27:10.733631+0000","last_undegraded":"2024-09-11T08:27:10.733631+0000","last_fullsized":"2024-09-11T08:27:10.733631+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'15","last_scrub_stamp":"2024-09-11T08:26:03.664676+0000","last_deep_scrub":"239'15","last_deep_scrub_stamp":"2024-09-11T08:26:03.664676+0000","last_clean_scrub_stamp":"2024-09-11T08:26:03.664676+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:17.627809+0000","scrub_duration":0.001893128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":126,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.733035+0000","last_change":"2024-09-11T08:25:59.563428+0000","last_active":"2024-09-11T08:27:10.733035+0000","last_peered":"2024-09-11T08:27:10.733035+0000","last_clean":"2024-09-11T08:27:10.733035+0000","last_became_active":"2024-09-11T08:24:28.358300+0000","last_became_peered":"2024-09-11T08:24:28.358300+0000","last_unstale":"2024-09-11T08:27:10.733035+0000","last_undegraded":"2024-09-11T08:27:10.733035+0000","last_fullsized":"2024-09-11T08:27:10.733035+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:25:59.563368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:53.548967+0000","last_clean_scrub_stamp":"2024-09-11T08:25:59.563368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013934489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":373,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.727775+0000","last_change":"2024-09-11T08:26:09.119039+0000","last_active":"2024-09-11T08:27:10.727775+0000","last_peered":"2024-09-11T08:27:10.727775+0000","last_clean":"2024-09-11T08:27:10.727775+0000","last_became_active":"2024-09-11T08:14:24.790678+0000","last_became_peered":"2024-09-11T08:14:24.790678+0000","last_unstale":"2024-09-11T08:27:10.727775+0000","last_undegraded":"2024-09-11T08:27:10.727775+0000","last_fullsized":"2024-09-11T08:27:10.727775+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:09.118973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:26:09.118973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:31.753802+0000","scrub_duration":0.0016651439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"239'19","reported_seq":478,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-11T08:27:07.898331+0000","last_change":"2024-09-11T08:22:58.755615+0000","last_active":"2024-09-11T08:27:07.898331+0000","last_peered":"2024-09-11T08:27:07.898331+0000","last_clean":"2024-09-11T08:27:07.898331+0000","last_became_active":"2024-09-11T08:15:56.514793+0000","last_became_peered":"2024-09-11T08:15:56.514793+0000","last_unstale":"2024-09-11T08:27:07.898331+0000","last_undegraded":"2024-09-11T08:27:07.898331+0000","last_fullsized":"2024-09-11T08:27:07.898331+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'19","last_scrub_stamp":"2024-09-11T08:22:58.755559+0000","last_deep_scrub":"84'13","last_deep_scrub_stamp":"2024-09-11T08:08:57.835634+0000","last_clean_scrub_stamp":"2024-09-11T08:22:58.755559+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001670223,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":396,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.727946+0000","last_change":"2024-09-11T08:24:26.028722+0000","last_active":"2024-09-11T08:27:10.727946+0000","last_peered":"2024-09-11T08:27:10.727946+0000","last_clean":"2024-09-11T08:27:10.727946+0000","last_became_active":"2024-09-11T08:14:24.786021+0000","last_became_peered":"2024-09-11T08:14:24.786021+0000","last_unstale":"2024-09-11T08:27:10.727946+0000","last_undegraded":"2024-09-11T08:27:10.727946+0000","last_fullsized":"2024-09-11T08:27:10.727946+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:26.028642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:30.471620+0000","last_clean_scrub_stamp":"2024-09-11T08:24:26.028642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0073996629999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":449,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.733509+0000","last_change":"2024-09-11T08:24:28.006448+0000","last_active":"2024-09-11T08:27:10.733509+0000","last_peered":"2024-09-11T08:27:10.733509+0000","last_clean":"2024-09-11T08:27:10.733509+0000","last_became_active":"2024-09-11T08:15:56.525717+0000","last_became_peered":"2024-09-11T08:15:56.525717+0000","last_unstale":"2024-09-11T08:27:10.733509+0000","last_undegraded":"2024-09-11T08:27:10.733509+0000","last_fullsized":"2024-09-11T08:27:10.733509+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:28.006392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:04.370734+0000","last_clean_scrub_stamp":"2024-09-11T08:24:28.006392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001372453,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":459,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.741016+0000","last_change":"2024-09-11T08:27:00.857665+0000","last_active":"2024-09-11T08:27:10.741016+0000","last_peered":"2024-09-11T08:27:10.741016+0000","last_clean":"2024-09-11T08:27:10.741016+0000","last_became_active":"2024-09-11T08:15:56.525120+0000","last_became_peered":"2024-09-11T08:15:56.525120+0000","last_unstale":"2024-09-11T08:27:10.741016+0000","last_undegraded":"2024-09-11T08:27:10.741016+0000","last_fullsized":"2024-09-11T08:27:10.741016+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:00.857585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:54.184372+0000","last_clean_scrub_stamp":"2024-09-11T08:27:00.857585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:28.826864+0000","scrub_duration":0.009399101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"313'19","reported_seq":175,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.727549+0000","last_change":"2024-09-11T08:26:54.105820+0000","last_active":"2024-09-11T08:27:10.727549+0000","last_peered":"2024-09-11T08:27:10.727549+0000","last_clean":"2024-09-11T08:27:10.727549+0000","last_became_active":"2024-09-11T08:24:28.353043+0000","last_became_peered":"2024-09-11T08:24:28.353043+0000","last_unstale":"2024-09-11T08:27:10.727549+0000","last_undegraded":"2024-09-11T08:27:10.727549+0000","last_fullsized":"2024-09-11T08:27:10.727549+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"313'19","last_scrub_stamp":"2024-09-11T08:26:54.105645+0000","last_deep_scrub":"313'19","last_deep_scrub_stamp":"2024-09-11T08:22:03.458402+0000","last_clean_scrub_stamp":"2024-09-11T08:26:54.105645+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:13.089493+0000","scrub_duration":0.0041528440000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":441,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-11T08:27:07.899451+0000","last_change":"2024-09-11T08:23:39.025692+0000","last_active":"2024-09-11T08:27:07.899451+0000","last_peered":"2024-09-11T08:27:07.899451+0000","last_clean":"2024-09-11T08:27:07.899451+0000","last_became_active":"2024-09-11T08:15:56.513278+0000","last_became_peered":"2024-09-11T08:15:56.513278+0000","last_unstale":"2024-09-11T08:27:07.899451+0000","last_undegraded":"2024-09-11T08:27:07.899451+0000","last_fullsized":"2024-09-11T08:27:07.899451+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:23:39.025620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:52.661591+0000","last_clean_scrub_stamp":"2024-09-11T08:23:39.025620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046140570000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"239'13","reported_seq":675,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:13.805715+0000","last_change":"2024-09-11T08:23:36.270712+0000","last_active":"2024-09-11T08:27:13.805715+0000","last_peered":"2024-09-11T08:27:13.805715+0000","last_clean":"2024-09-11T08:27:13.805715+0000","last_became_active":"2024-09-11T08:17:37.558224+0000","last_became_peered":"2024-09-11T08:17:37.558224+0000","last_unstale":"2024-09-11T08:27:13.805715+0000","last_undegraded":"2024-09-11T08:27:13.805715+0000","last_fullsized":"2024-09-11T08:27:13.805715+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'13","last_scrub_stamp":"2024-09-11T08:23:36.270623+0000","last_deep_scrub":"222'12","last_deep_scrub_stamp":"2024-09-11T08:16:52.592856+0000","last_clean_scrub_stamp":"2024-09-11T08:23:36.270623+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.006222399,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":364,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.741120+0000","last_change":"2024-09-11T08:25:59.575043+0000","last_active":"2024-09-11T08:27:10.741120+0000","last_peered":"2024-09-11T08:27:10.741120+0000","last_clean":"2024-09-11T08:27:10.741120+0000","last_became_active":"2024-09-11T08:24:28.357799+0000","last_became_peered":"2024-09-11T08:24:28.357799+0000","last_unstale":"2024-09-11T08:27:10.741120+0000","last_undegraded":"2024-09-11T08:27:10.741120+0000","last_fullsized":"2024-09-11T08:27:10.741120+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:25:59.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:53.175994+0000","last_clean_scrub_stamp":"2024-09-11T08:25:59.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:22.362275+0000","scrub_duration":0.001608221,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"239'20","reported_seq":298,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.740942+0000","last_change":"2024-09-11T08:26:09.487437+0000","last_active":"2024-09-11T08:27:10.740942+0000","last_peered":"2024-09-11T08:27:10.740942+0000","last_clean":"2024-09-11T08:27:10.740942+0000","last_became_active":"2024-09-11T08:17:37.554357+0000","last_became_peered":"2024-09-11T08:17:37.554357+0000","last_unstale":"2024-09-11T08:27:10.740942+0000","last_undegraded":"2024-09-11T08:27:10.740942+0000","last_fullsized":"2024-09-11T08:27:10.740942+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'20","last_scrub_stamp":"2024-09-11T08:26:09.487383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:26:09.487383+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:22.234597+0000","scrub_duration":0.002145428,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":238,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.728330+0000","last_change":"2024-09-11T08:23:42.199613+0000","last_active":"2024-09-11T08:27:10.728330+0000","last_peered":"2024-09-11T08:27:10.728330+0000","last_clean":"2024-09-11T08:27:10.728330+0000","last_became_active":"2024-09-11T08:15:56.518515+0000","last_became_peered":"2024-09-11T08:15:56.518515+0000","last_unstale":"2024-09-11T08:27:10.728330+0000","last_undegraded":"2024-09-11T08:27:10.728330+0000","last_fullsized":"2024-09-11T08:27:10.728330+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:23:42.199527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:23:42.199527+0000","last_clean_scrub_stamp":"2024-09-11T08:23:42.199527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054572529999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"354'13","reported_seq":210,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-09-11T08:27:14.954497+0000","last_change":"2024-09-11T08:27:09.973738+0000","last_active":"2024-09-11T08:27:14.954497+0000","last_peered":"2024-09-11T08:27:14.954497+0000","last_clean":"2024-09-11T08:27:14.954497+0000","last_became_active":"2024-09-11T08:24:28.353115+0000","last_became_peered":"2024-09-11T08:24:28.353115+0000","last_unstale":"2024-09-11T08:27:14.954497+0000","last_undegraded":"2024-09-11T08:27:14.954497+0000","last_fullsized":"2024-09-11T08:27:14.954497+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'13","last_scrub_stamp":"2024-09-11T08:27:09.973656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:27:09.973656+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:13.849398+0000","scrub_duration":0.0018473879999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":435,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-11T08:27:07.897878+0000","last_change":"2024-09-11T08:26:10.045998+0000","last_active":"2024-09-11T08:27:07.897878+0000","last_peered":"2024-09-11T08:27:07.897878+0000","last_clean":"2024-09-11T08:27:07.897878+0000","last_became_active":"2024-09-11T08:15:56.513949+0000","last_became_peered":"2024-09-11T08:15:56.513949+0000","last_unstale":"2024-09-11T08:27:07.897878+0000","last_undegraded":"2024-09-11T08:27:07.897878+0000","last_fullsized":"2024-09-11T08:27:07.897878+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:10.045943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:26:10.045943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:14.663252+0000","scrub_duration":0.0017620470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"239'19","reported_seq":337,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.732712+0000","last_change":"2024-09-11T08:24:25.584115+0000","last_active":"2024-09-11T08:27:10.732712+0000","last_peered":"2024-09-11T08:27:10.732712+0000","last_clean":"2024-09-11T08:27:10.732712+0000","last_became_active":"2024-09-11T08:23:22.688993+0000","last_became_peered":"2024-09-11T08:23:22.688993+0000","last_unstale":"2024-09-11T08:27:10.732712+0000","last_undegraded":"2024-09-11T08:27:10.732712+0000","last_fullsized":"2024-09-11T08:27:10.732712+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'19","last_scrub_stamp":"2024-09-11T08:24:25.584019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:24:25.584019+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047681069999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":287,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.727708+0000","last_change":"2024-09-11T08:26:02.102192+0000","last_active":"2024-09-11T08:27:10.727708+0000","last_peered":"2024-09-11T08:27:10.727708+0000","last_clean":"2024-09-11T08:27:10.727708+0000","last_became_active":"2024-09-11T08:23:22.682569+0000","last_became_peered":"2024-09-11T08:23:22.682569+0000","last_unstale":"2024-09-11T08:27:10.727708+0000","last_undegraded":"2024-09-11T08:27:10.727708+0000","last_fullsized":"2024-09-11T08:27:10.727708+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:02.102138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:26:02.102138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:19.976358+0000","scrub_duration":0.0012891669999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":461,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.734704+0000","last_change":"2024-09-11T08:24:28.466446+0000","last_active":"2024-09-11T08:27:10.734704+0000","last_peered":"2024-09-11T08:27:10.734704+0000","last_clean":"2024-09-11T08:27:10.734704+0000","last_became_active":"2024-09-11T08:24:28.355004+0000","last_became_peered":"2024-09-11T08:24:28.355004+0000","last_unstale":"2024-09-11T08:27:10.734704+0000","last_undegraded":"2024-09-11T08:27:10.734704+0000","last_fullsized":"2024-09-11T08:27:10.734704+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:28.466393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:24:28.466393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001587551,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":249,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-11T08:27:07.897562+0000","last_change":"2024-09-11T08:27:04.846909+0000","last_active":"2024-09-11T08:27:07.897562+0000","last_peered":"2024-09-11T08:27:07.897562+0000","last_clean":"2024-09-11T08:27:07.897562+0000","last_became_active":"2024-09-11T08:24:28.372049+0000","last_became_peered":"2024-09-11T08:24:28.372049+0000","last_unstale":"2024-09-11T08:27:07.897562+0000","last_undegraded":"2024-09-11T08:27:07.897562+0000","last_fullsized":"2024-09-11T08:27:07.897562+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:04.846717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:27:04.846717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:20.813065+0000","scrub_duration":0.0041876129999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"343'19","reported_seq":472,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.735023+0000","last_change":"2024-09-11T08:26:07.610504+0000","last_active":"2024-09-11T08:27:10.735023+0000","last_peered":"2024-09-11T08:27:10.735023+0000","last_clean":"2024-09-11T08:27:10.735023+0000","last_became_active":"2024-09-11T08:07:32.049368+0000","last_became_peered":"2024-09-11T08:07:32.049368+0000","last_unstale":"2024-09-11T08:27:10.735023+0000","last_undegraded":"2024-09-11T08:27:10.735023+0000","last_fullsized":"2024-09-11T08:27:10.735023+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'19","last_scrub_stamp":"2024-09-11T08:26:07.610413+0000","last_deep_scrub":"343'19","last_deep_scrub_stamp":"2024-09-11T08:23:44.640077+0000","last_clean_scrub_stamp":"2024-09-11T08:26:07.610413+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:17.842578+0000","scrub_duration":0.0020145950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":238,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.727618+0000","last_change":"2024-09-11T08:23:47.170953+0000","last_active":"2024-09-11T08:27:10.727618+0000","last_peered":"2024-09-11T08:27:10.727618+0000","last_clean":"2024-09-11T08:27:10.727618+0000","last_became_active":"2024-09-11T08:15:56.521570+0000","last_became_peered":"2024-09-11T08:15:56.521570+0000","last_unstale":"2024-09-11T08:27:10.727618+0000","last_undegraded":"2024-09-11T08:27:10.727618+0000","last_fullsized":"2024-09-11T08:27:10.727618+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:23:47.170848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:23:47.170848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0107554,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":202,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-11T08:27:07.898163+0000","last_change":"2024-09-11T08:25:53.178915+0000","last_active":"2024-09-11T08:27:07.898163+0000","last_peered":"2024-09-11T08:27:07.898163+0000","last_clean":"2024-09-11T08:27:07.898163+0000","last_became_active":"2024-09-11T08:17:37.557645+0000","last_became_peered":"2024-09-11T08:17:37.557645+0000","last_unstale":"2024-09-11T08:27:07.898163+0000","last_undegraded":"2024-09-11T08:27:07.898163+0000","last_fullsized":"2024-09-11T08:27:07.898163+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:22:02.568761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:22:02.568761+0000","last_clean_scrub_stamp":"2024-09-11T08:22:02.568761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0057637030000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":448,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-11T08:27:07.897169+0000","last_change":"2024-09-11T08:27:03.810727+0000","last_active":"2024-09-11T08:27:07.897169+0000","last_peered":"2024-09-11T08:27:07.897169+0000","last_clean":"2024-09-11T08:27:07.897169+0000","last_became_active":"2024-09-11T08:24:28.363959+0000","last_became_peered":"2024-09-11T08:24:28.363959+0000","last_unstale":"2024-09-11T08:27:07.897169+0000","last_undegraded":"2024-09-11T08:27:07.897169+0000","last_fullsized":"2024-09-11T08:27:07.897169+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:03.810647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:09.973943+0000","last_clean_scrub_stamp":"2024-09-11T08:27:03.810647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:25.890757+0000","scrub_duration":0.0051903160000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"239'19","reported_seq":516,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.733990+0000","last_change":"2024-09-11T08:26:58.609528+0000","last_active":"2024-09-11T08:27:10.733990+0000","last_peered":"2024-09-11T08:27:10.733990+0000","last_clean":"2024-09-11T08:27:10.733990+0000","last_became_active":"2024-09-11T08:15:56.520058+0000","last_became_peered":"2024-09-11T08:15:56.520058+0000","last_unstale":"2024-09-11T08:27:10.733990+0000","last_undegraded":"2024-09-11T08:27:10.733990+0000","last_fullsized":"2024-09-11T08:27:10.733990+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'19","last_scrub_stamp":"2024-09-11T08:26:58.609443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:26:58.609443+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:18.718575+0000","scrub_duration":0.017621226,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":341,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.727563+0000","last_change":"2024-09-11T08:24:28.951646+0000","last_active":"2024-09-11T08:27:10.727563+0000","last_peered":"2024-09-11T08:27:10.727563+0000","last_clean":"2024-09-11T08:27:10.727563+0000","last_became_active":"2024-09-11T08:24:28.356974+0000","last_became_peered":"2024-09-11T08:24:28.356974+0000","last_unstale":"2024-09-11T08:27:10.727563+0000","last_undegraded":"2024-09-11T08:27:10.727563+0000","last_fullsized":"2024-09-11T08:27:10.727563+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:28.951592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:24:28.951592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001286382,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":248,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.727623+0000","last_change":"2024-09-11T08:26:03.094530+0000","last_active":"2024-09-11T08:27:10.727623+0000","last_peered":"2024-09-11T08:27:10.727623+0000","last_clean":"2024-09-11T08:27:10.727623+0000","last_became_active":"2024-09-11T08:15:56.523728+0000","last_became_peered":"2024-09-11T08:15:56.523728+0000","last_unstale":"2024-09-11T08:27:10.727623+0000","last_undegraded":"2024-09-11T08:27:10.727623+0000","last_fullsized":"2024-09-11T08:27:10.727623+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:03.094474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:11:34.159053+0000","last_clean_scrub_stamp":"2024-09-11T08:26:03.094474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:16.180366+0000","scrub_duration":0.001235589,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":420,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.732797+0000","last_change":"2024-09-11T08:26:06.683231+0000","last_active":"2024-09-11T08:27:10.732797+0000","last_peered":"2024-09-11T08:27:10.732797+0000","last_clean":"2024-09-11T08:27:10.732797+0000","last_became_active":"2024-09-11T08:07:36.633368+0000","last_became_peered":"2024-09-11T08:07:36.633368+0000","last_unstale":"2024-09-11T08:27:10.732797+0000","last_undegraded":"2024-09-11T08:27:10.732797+0000","last_fullsized":"2024-09-11T08:27:10.732797+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:06.683182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:22:50.992034+0000","last_clean_scrub_stamp":"2024-09-11T08:26:06.683182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:28.391283+0000","scrub_duration":0.001432284,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"312'10","reported_seq":438,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:12.683185+0000","last_change":"2024-09-11T08:27:12.683107+0000","last_active":"2024-09-11T08:27:12.683185+0000","last_peered":"2024-09-11T08:27:12.683185+0000","last_clean":"2024-09-11T08:27:12.683185+0000","last_became_active":"2024-09-11T08:07:32.045728+0000","last_became_peered":"2024-09-11T08:07:32.045728+0000","last_unstale":"2024-09-11T08:27:12.683185+0000","last_undegraded":"2024-09-11T08:27:12.683185+0000","last_fullsized":"2024-09-11T08:27:12.683185+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"312'10","last_scrub_stamp":"2024-09-11T08:27:12.683016+0000","last_deep_scrub":"312'10","last_deep_scrub_stamp":"2024-09-11T08:21:55.859866+0000","last_clean_scrub_stamp":"2024-09-11T08:27:12.683016+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:19.488917+0000","scrub_duration":0.017774564,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":451,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-11T08:27:07.898546+0000","last_change":"2024-09-11T08:24:28.358269+0000","last_active":"2024-09-11T08:27:07.898546+0000","last_peered":"2024-09-11T08:27:07.898546+0000","last_clean":"2024-09-11T08:27:07.898546+0000","last_became_active":"2024-09-11T08:24:28.357978+0000","last_became_peered":"2024-09-11T08:24:28.357978+0000","last_unstale":"2024-09-11T08:27:07.898546+0000","last_undegraded":"2024-09-11T08:27:07.898546+0000","last_fullsized":"2024-09-11T08:27:07.898546+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:22:48.784020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:22:48.784020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0083483829999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":478,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.733198+0000","last_change":"2024-09-11T08:24:09.590462+0000","last_active":"2024-09-11T08:27:10.733198+0000","last_peered":"2024-09-11T08:27:10.733198+0000","last_clean":"2024-09-11T08:27:10.733198+0000","last_became_active":"2024-09-11T08:15:56.528607+0000","last_became_peered":"2024-09-11T08:15:56.528607+0000","last_unstale":"2024-09-11T08:27:10.733198+0000","last_undegraded":"2024-09-11T08:27:10.733198+0000","last_fullsized":"2024-09-11T08:27:10.733198+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:09.590401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:24:09.590401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016035050000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":241,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-09-11T08:27:16.194914+0000","last_change":"2024-09-11T08:27:16.194874+0000","last_active":"2024-09-11T08:27:16.194914+0000","last_peered":"2024-09-11T08:27:16.194914+0000","last_clean":"2024-09-11T08:27:16.194914+0000","last_became_active":"2024-09-11T08:15:56.521488+0000","last_became_peered":"2024-09-11T08:15:56.521488+0000","last_unstale":"2024-09-11T08:27:16.194914+0000","last_undegraded":"2024-09-11T08:27:16.194914+0000","last_fullsized":"2024-09-11T08:27:16.194914+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:16.194807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:20:18.827709+0000","last_clean_scrub_stamp":"2024-09-11T08:27:16.194807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:21.088138+0000","scrub_duration":0.0021734469999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":469,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-09-11T08:27:17.179484+0000","last_change":"2024-09-11T08:27:17.179442+0000","last_active":"2024-09-11T08:27:17.179484+0000","last_peered":"2024-09-11T08:27:17.179484+0000","last_clean":"2024-09-11T08:27:17.179484+0000","last_became_active":"2024-09-11T08:24:28.354141+0000","last_became_peered":"2024-09-11T08:24:28.354141+0000","last_unstale":"2024-09-11T08:27:17.179484+0000","last_undegraded":"2024-09-11T08:27:17.179484+0000","last_fullsized":"2024-09-11T08:27:17.179484+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:17.179345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:50.585566+0000","last_clean_scrub_stamp":"2024-09-11T08:27:17.179345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:46.719858+0000","scrub_duration":0.0077697699999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":199,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-11T08:27:07.897615+0000","last_change":"2024-09-11T08:23:01.737938+0000","last_active":"2024-09-11T08:27:07.897615+0000","last_peered":"2024-09-11T08:27:07.897615+0000","last_clean":"2024-09-11T08:27:07.897615+0000","last_became_active":"2024-09-11T08:17:37.554262+0000","last_became_peered":"2024-09-11T08:17:37.554262+0000","last_unstale":"2024-09-11T08:27:07.897615+0000","last_undegraded":"2024-09-11T08:27:07.897615+0000","last_fullsized":"2024-09-11T08:27:07.897615+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:23:01.737888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:23:01.737888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020103790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"288'24","reported_seq":540,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.733133+0000","last_change":"2024-09-11T08:26:56.653326+0000","last_active":"2024-09-11T08:27:10.733133+0000","last_peered":"2024-09-11T08:27:10.733133+0000","last_clean":"2024-09-11T08:27:10.733133+0000","last_became_active":"2024-09-11T08:24:28.360937+0000","last_became_peered":"2024-09-11T08:24:28.360937+0000","last_unstale":"2024-09-11T08:27:10.733133+0000","last_undegraded":"2024-09-11T08:27:10.733133+0000","last_fullsized":"2024-09-11T08:27:10.733133+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'24","last_scrub_stamp":"2024-09-11T08:26:56.653232+0000","last_deep_scrub":"288'24","last_deep_scrub_stamp":"2024-09-11T08:20:28.885295+0000","last_clean_scrub_stamp":"2024-09-11T08:26:56.653232+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:04.332641+0000","scrub_duration":0.0098689580000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":266,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-11T08:27:10.732751+0000","last_change":"2024-09-11T08:24:07.560376+0000","last_active":"2024-09-11T08:27:10.732751+0000","last_peered":"2024-09-11T08:27:10.732751+0000","last_clean":"2024-09-11T08:27:10.732751+0000","last_became_active":"2024-09-11T08:15:08.769133+0000","last_became_peered":"2024-09-11T08:15:08.769133+0000","last_unstale":"2024-09-11T08:27:10.732751+0000","last_undegraded":"2024-09-11T08:27:10.732751+0000","last_fullsized":"2024-09-11T08:27:10.732751+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:07.560293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:18:39.206016+0000","last_clean_scrub_stamp":"2024-09-11T08:24:07.560293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0083464200000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":4588175360,"num_objects":70010,"num_object_clones":0,"num_object_copies":210030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70010,"num_write_kb":4480640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13689028608,"data_stored":13689028608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70010,"ondisk_log_size":70010,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102,"ondisk_log_size":102,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1117,"num_read_kb":874,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":4566,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3425,"internal_metadata":0},"log_size":772,"ondisk_log_size":772,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":779,"num_read_kb":774,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":168,"seq":721554505884,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2285504,"kb_used_data":1671288,"kb_used_omap":7,"kb_used_meta":614200,"kb_avail":91459648,"statfs":{"total":95995035648,"available":93654679552,"internally_reserved":0,"allocated":1711398912,"data_stored":1710935466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8027,"internal_metadata":628940965},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:26:37 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":3.9620000000000002,"15min":2.3740000000000001},"min":{"1min":0.24399999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.79300000000000004,"5min":117.57599999999999,"15min":117.57599999999999},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":3.7069999999999999,"15min":2.29},"min":{"1min":0.317,"5min":0.22800000000000001,"15min":0.219},"max":{"1min":1,"5min":117.542,"15min":117.542},"last":0.26600000000000001}]},{"osd":1,"last update":"Wed Sep 11 08:26:37 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":1.1200000000000001,"15min":1.9790000000000001},"min":{"1min":0.27200000000000002,"5min":0.24099999999999999,"15min":0.20899999999999999},"max":{"1min":0.61899999999999999,"5min":13.252000000000001,"15min":66.971000000000004},"last":1.597},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.70299999999999996,"15min":1.7769999999999999},"min":{"1min":0.28000000000000003,"5min":0.25800000000000001,"15min":0.23000000000000001},"max":{"1min":0.69599999999999995,"5min":3.5510000000000002,"15min":67.185000000000002},"last":0.55800000000000005}]},{"osd":2,"last update":"Wed Sep 11 08:26:37 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":1.0920000000000001,"15min":1.028},"min":{"1min":0.29299999999999998,"5min":0.23799999999999999,"15min":0.22700000000000001},"max":{"1min":2.1840000000000002,"5min":19.097000000000001,"15min":19.097000000000001},"last":1.712},{"interface":"front","average":{"1min":0.58099999999999996,"5min":1.3879999999999999,"15min":1.0609999999999999},"min":{"1min":0.33400000000000002,"5min":0.255,"15min":0.23699999999999999},"max":{"1min":2.2080000000000002,"5min":21.594000000000001,"15min":21.594000000000001},"last":2.5030000000000001}]},{"osd":3,"last update":"Wed Sep 11 08:26:37 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.86299999999999999,"15min":1.083},"min":{"1min":0.33300000000000002,"5min":0.26600000000000001,"15min":0.251},"max":{"1min":0.65900000000000003,"5min":6.375,"15min":21.460000000000001},"last":1.79},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.86199999999999999,"15min":1.2},"min":{"1min":0.309,"5min":0.27200000000000002,"15min":0.251},"max":{"1min":0.97499999999999998,"5min":13.228999999999999,"15min":21.440000000000001},"last":1.74}]},{"osd":4,"last update":"Wed Sep 11 08:26:37 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":1.002,"15min":0.82799999999999996},"min":{"1min":0.30399999999999999,"5min":0.25800000000000001,"15min":0.252},"max":{"1min":1.966,"5min":5.7240000000000002,"15min":5.7240000000000002},"last":0.436},{"interface":"front","average":{"1min":0.496,"5min":0.83199999999999996,"15min":0.79300000000000004},"min":{"1min":0.28899999999999998,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.73899999999999999,"5min":4.2060000000000004,"15min":5.2649999999999997},"last":0.51800000000000002}]},{"osd":5,"last update":"Wed Sep 11 08:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":1.2969999999999999,"15min":1.379},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":5.8360000000000003,"5min":5.8360000000000003,"15min":5.8360000000000003},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.88900000000000001,"15min":0.93500000000000005},"min":{"1min":0.33700000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":2.75,"5min":2.75,"15min":2.75},"last":0.54200000000000004}]},{"osd":6,"last update":"Wed Sep 11 08:26:37 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.90100000000000002,"15min":0.83499999999999996},"min":{"1min":0.36799999999999999,"5min":0.26900000000000002,"15min":0.248},"max":{"1min":0.95099999999999996,"5min":5.3230000000000004,"15min":7.1699999999999999},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.66100000000000003,"5min":1.032,"15min":0.93000000000000005},"min":{"1min":0.42399999999999999,"5min":0.315,"15min":0.25700000000000001},"max":{"1min":1.8600000000000001,"5min":6.0439999999999996,"15min":6.0439999999999996},"last":0.504}]}]},{"osd":6,"up_from":45,"seq":193273528567,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2802804,"kb_used_data":2212904,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":90942348,"statfs":{"total":95995035648,"available":93124964352,"internally_reserved":0,"allocated":2266013696,"data_stored":2265566457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:26:28 2024","interfaces":[{"interface":"back","average":{"1min":17.887,"5min":6.7610000000000001,"15min":3.4820000000000002},"min":{"1min":0.224,"5min":0.224,"15min":0.17999999999999999},"max":{"1min":331.14800000000002,"5min":331.14800000000002,"15min":331.14800000000002},"last":0.23300000000000001},{"interface":"front","average":{"1min":17.945,"5min":6.2949999999999999,"15min":3.2559999999999998},"min":{"1min":0.35099999999999998,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":331.31,"5min":331.31,"15min":331.31},"last":0.55700000000000005}]},{"osd":1,"last update":"Wed Sep 11 08:26:28 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":1.1719999999999999,"15min":1.4199999999999999},"min":{"1min":0.33500000000000002,"5min":0.223,"15min":0.214},"max":{"1min":3.3220000000000001,"5min":10.775,"15min":44.658000000000001},"last":1.9159999999999999},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.95699999999999996,"15min":1.3919999999999999},"min":{"1min":0.252,"5min":0.22700000000000001,"15min":0.22600000000000001},"max":{"1min":2.3050000000000002,"5min":8.4109999999999996,"15min":44.694000000000003},"last":1.52}]},{"osd":2,"last update":"Wed Sep 11 08:26:28 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":1.7270000000000001,"15min":1.165},"min":{"1min":0.32700000000000001,"5min":0.23799999999999999,"15min":0.23200000000000001},"max":{"1min":2.4209999999999998,"5min":39.652999999999999,"15min":39.652999999999999},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.63700000000000001,"5min":1.748,"15min":1.383},"min":{"1min":0.379,"5min":0.26100000000000001,"15min":0.23300000000000001},"max":{"1min":2.085,"5min":38.390999999999998,"15min":38.390999999999998},"last":0.251}]},{"osd":3,"last update":"Wed Sep 11 08:26:28 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":1.377,"15min":1.3839999999999999},"min":{"1min":0.35399999999999998,"5min":0.253,"15min":0.22700000000000001},"max":{"1min":3.2970000000000002,"5min":10.144,"15min":36.061},"last":1.7709999999999999},{"interface":"front","average":{"1min":0.621,"5min":1.1499999999999999,"15min":1.3129999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28399999999999997},"max":{"1min":2.3580000000000001,"5min":8.4399999999999995,"15min":36.093000000000004},"last":0.35399999999999998}]},{"osd":4,"last update":"Wed Sep 11 08:26:28 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.85599999999999998,"15min":0.69999999999999996},"min":{"1min":0.26800000000000002,"5min":0.20599999999999999,"15min":0.20499999999999999},"max":{"1min":2.1190000000000002,"5min":9.7560000000000002,"15min":9.7560000000000002},"last":0.501},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.78700000000000003,"15min":0.68400000000000005},"min":{"1min":0.32000000000000001,"5min":0.26100000000000001,"15min":0.23699999999999999},"max":{"1min":2.4039999999999999,"5min":4.181,"15min":4.468},"last":0.29099999999999998}]},{"osd":5,"last update":"Wed Sep 11 08:26:28 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":1.6200000000000001,"15min":1.766},"min":{"1min":0.375,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":2.1400000000000001,"5min":9.7989999999999995,"15min":9.7989999999999995},"last":3.0640000000000001},{"interface":"front","average":{"1min":0.76100000000000001,"5min":1.429,"15min":1.54},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":3.2040000000000002,"5min":7.2999999999999998,"15min":7.2999999999999998},"last":2.5209999999999999}]},{"osd":7,"last update":"Wed Sep 11 08:26:37 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":1.2270000000000001,"15min":1.0429999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.253},"max":{"1min":3.8780000000000001,"5min":10.167999999999999,"15min":10.167999999999999},"last":1.8160000000000001},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.85999999999999999,"15min":0.80500000000000005},"min":{"1min":0.35499999999999998,"5min":0.28399999999999997,"15min":0.25700000000000001},"max":{"1min":0.74299999999999999,"5min":5.3579999999999997,"15min":5.3579999999999997},"last":3.6160000000000001}]}]},{"osd":5,"up_from":353,"seq":1516123455523,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1458416,"kb_used_data":1098468,"kb_used_omap":4,"kb_used_meta":359931,"kb_avail":92286736,"statfs":{"total":95995035648,"available":94501617664,"internally_reserved":0,"allocated":1124831232,"data_stored":1124386619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":368569862},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:26:31 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":1.6990000000000001,"15min":1.835},"min":{"1min":0.32500000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":7.6870000000000003,"5min":19.379999999999999,"15min":19.379999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.89600000000000002,"5min":1.673,"15min":1.8029999999999999},"min":{"1min":0.29299999999999998,"5min":0.218,"15min":0.218},"max":{"1min":8.202,"5min":18.238,"15min":18.238},"last":0.36799999999999999}]},{"osd":1,"last update":"Wed Sep 11 08:26:31 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.998,"15min":1.0820000000000001},"min":{"1min":0.30599999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.76400000000000001,"5min":5.3170000000000002,"15min":5.3170000000000002},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.66400000000000003,"15min":0.69299999999999995},"min":{"1min":0.34599999999999997,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.71499999999999997,"5min":2.6280000000000001,"15min":2.6280000000000001},"last":0.24199999999999999}]},{"osd":2,"last update":"Wed Sep 11 08:26:31 2024","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":1.879,"15min":2.036},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":8.8000000000000007,"5min":18.350999999999999,"15min":18.350999999999999},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.93999999999999995,"5min":1.748,"15min":1.8819999999999999},"min":{"1min":0.32200000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":8.1910000000000007,"5min":18.381,"15min":18.381},"last":0.70799999999999996}]},{"osd":3,"last update":"Wed Sep 11 08:26:31 2024","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":1.113,"15min":1.153},"min":{"1min":0.33600000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":7.2290000000000001,"5min":7.2290000000000001,"15min":7.2290000000000001},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.86399999999999999,"15min":0.85799999999999998},"min":{"1min":0.39200000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":7.4109999999999996,"5min":7.4109999999999996,"15min":7.4109999999999996},"last":0.44}]},{"osd":4,"last update":"Wed Sep 11 08:26:31 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":1.0880000000000001,"15min":1.1930000000000001},"min":{"1min":0.26700000000000002,"5min":0.253,"15min":0.253},"max":{"1min":1.0629999999999999,"5min":5.7670000000000003,"15min":5.7670000000000003},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.67200000000000004,"5min":1.181,"15min":1.266},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":4.093,"5min":5.8979999999999997,"15min":5.8979999999999997},"last":0.29999999999999999}]},{"osd":6,"last update":"Wed Sep 11 08:26:31 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":1.3899999999999999,"15min":1.506},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":4.0490000000000004,"5min":5.9779999999999998,"15min":5.9779999999999998},"last":0.502},{"interface":"front","average":{"1min":0.47199999999999998,"5min":1.639,"15min":1.833},"min":{"1min":0.34999999999999998,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.66100000000000003,"5min":5.6680000000000001,"15min":5.6680000000000001},"last":0.40500000000000003}]},{"osd":7,"last update":"Wed Sep 11 08:26:31 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":1.1060000000000001,"15min":1.212},"min":{"1min":0.308,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.65200000000000002,"5min":4.9699999999999998,"15min":4.9699999999999998},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":1.5820000000000001,"15min":1.7549999999999999},"min":{"1min":0.36299999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.749,"5min":5.8529999999999998,"15min":5.8529999999999998},"last":0.38300000000000001}]}]},{"osd":4,"up_from":31,"seq":133143986433,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1690948,"kb_used_data":1101052,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":92054204,"statfs":{"total":95995035648,"available":94263504896,"internally_reserved":0,"allocated":1127477248,"data_stored":1127009860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:26:42 2024","interfaces":[{"interface":"back","average":{"1min":3.6859999999999999,"5min":2.706,"15min":1.8899999999999999},"min":{"1min":0.251,"5min":0.217,"15min":0.19},"max":{"1min":64.329999999999998,"5min":64.329999999999998,"15min":64.329999999999998},"last":3.032},{"interface":"front","average":{"1min":3.79,"5min":2.9609999999999999,"15min":1.891},"min":{"1min":0.33300000000000002,"5min":0.24399999999999999,"15min":0.221},"max":{"1min":66.819999999999993,"5min":66.819999999999993,"15min":66.819999999999993},"last":0.49099999999999999}]},{"osd":1,"last update":"Wed Sep 11 08:26:42 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":1.101,"15min":1.1619999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.214},"max":{"1min":0.66600000000000004,"5min":35.156999999999996,"15min":35.156999999999996},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.46899999999999997,"5min":1.456,"15min":1.24},"min":{"1min":0.30299999999999999,"5min":0.23599999999999999,"15min":0.22900000000000001},"max":{"1min":0.68100000000000005,"5min":35.088999999999999,"15min":35.088999999999999},"last":0.73799999999999999}]},{"osd":2,"last update":"Wed Sep 11 08:26:42 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":2.0169999999999999,"15min":1.397},"min":{"1min":0.33400000000000002,"5min":0.25800000000000001,"15min":0.23400000000000001},"max":{"1min":0.72299999999999998,"5min":54.317999999999998,"15min":54.317999999999998},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.48699999999999999,"5min":2.2749999999999999,"15min":1.446},"min":{"1min":0.30299999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.75900000000000001,"5min":54.430999999999997,"15min":54.430999999999997},"last":0.46700000000000003}]},{"osd":3,"last update":"Wed Sep 11 08:26:42 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":1.2190000000000001,"15min":1.3149999999999999},"min":{"1min":0.29099999999999998,"5min":0.27900000000000003,"15min":0.253},"max":{"1min":0.73599999999999999,"5min":30.321000000000002,"15min":30.321000000000002},"last":0.438},{"interface":"front","average":{"1min":0.52700000000000002,"5min":1.0740000000000001,"15min":1.194},"min":{"1min":0.28399999999999997,"5min":0.26500000000000001,"15min":0.246},"max":{"1min":1.1739999999999999,"5min":30.620000000000001,"15min":30.620000000000001},"last":0.29099999999999998}]},{"osd":5,"last update":"Wed Sep 11 08:26:29 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":1.226,"15min":1.3160000000000001},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":4.5869999999999997,"5min":5.1050000000000004,"15min":5.1050000000000004},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.421,"15min":1.4930000000000001},"min":{"1min":0.30099999999999999,"5min":0.255,"15min":0.255},"max":{"1min":5.5750000000000002,"5min":6.1150000000000002,"15min":6.1150000000000002},"last":0.374}]},{"osd":6,"last update":"Wed Sep 11 08:26:29 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.84299999999999997,"15min":0.83499999999999996},"min":{"1min":0.28199999999999997,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":5.2640000000000002,"5min":5.2640000000000002,"15min":48.808},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.94899999999999995,"5min":0.90700000000000003,"15min":0.88},"min":{"1min":0.309,"5min":0.251,"15min":0.251},"max":{"1min":5.3159999999999998,"5min":5.3159999999999998,"15min":48.746000000000002},"last":0.61399999999999999}]},{"osd":7,"last update":"Wed Sep 11 08:26:54 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.85699999999999998,"15min":0.70099999999999996},"min":{"1min":0.29799999999999999,"5min":0.249,"15min":0.249},"max":{"1min":1.7070000000000001,"5min":6.093,"15min":6.093},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.67300000000000004,"15min":0.73799999999999999},"min":{"1min":0.32900000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.88500000000000001,"5min":4.8259999999999996,"15min":4.8259999999999996},"last":0.45700000000000002}]}]},{"osd":3,"up_from":25,"seq":107374182662,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2855632,"kb_used_data":1954440,"kb_used_omap":1,"kb_used_meta":901182,"kb_avail":90889520,"statfs":{"total":95995035648,"available":93070868480,"internally_reserved":0,"allocated":2001346560,"data_stored":2000866355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":922810712},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:26:30 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":1.5109999999999999,"15min":1.875},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.14999999999999999},"max":{"1min":4.298,"5min":24.616,"15min":60.469999999999999},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.76600000000000001,"5min":1.2609999999999999,"15min":1.889},"min":{"1min":0.26300000000000001,"5min":0.20599999999999999,"15min":0.17399999999999999},"max":{"1min":4.375,"5min":23.640000000000001,"15min":60.488999999999997},"last":0.78600000000000003}]},{"osd":1,"last update":"Wed Sep 11 08:26:30 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":1.2130000000000001,"15min":1.619},"min":{"1min":0.20399999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":4.1520000000000001,"5min":33.402000000000001,"15min":60.578000000000003},"last":1.1399999999999999},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.88100000000000001,"15min":1.7390000000000001},"min":{"1min":0.20899999999999999,"5min":0.19700000000000001,"15min":0.19500000000000001},"max":{"1min":4.0979999999999999,"5min":5.8040000000000003,"15min":63.674999999999997},"last":0.54300000000000004}]},{"osd":2,"last update":"Wed Sep 11 08:26:30 2024","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":1.7789999999999999,"15min":1.877},"min":{"1min":0.23999999999999999,"5min":0.219,"15min":0.185},"max":{"1min":7.665,"5min":67.503,"15min":67.503},"last":1.1719999999999999},{"interface":"front","average":{"1min":0.63400000000000001,"5min":1.778,"15min":1.7549999999999999},"min":{"1min":0.251,"5min":0.24299999999999999,"15min":0.19900000000000001},"max":{"1min":4.4550000000000001,"5min":67.313999999999993,"15min":67.313999999999993},"last":0.35899999999999999}]},{"osd":4,"last update":"Wed Sep 11 08:26:37 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":1.113,"15min":1.46},"min":{"1min":0.32900000000000001,"5min":0.29199999999999998,"15min":0.24299999999999999},"max":{"1min":0.753,"5min":4.274,"15min":61.290999999999997},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.90500000000000003,"15min":1.2450000000000001},"min":{"1min":0.32500000000000001,"5min":0.29299999999999998,"15min":0.27500000000000002},"max":{"1min":0.73399999999999999,"5min":5.3259999999999996,"15min":60.856000000000002},"last":0.316}]},{"osd":5,"last update":"Wed Sep 11 08:26:30 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":1.577,"15min":1.7549999999999999},"min":{"1min":0.36399999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.64300000000000002,"5min":7.3849999999999998,"15min":7.3849999999999998},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.57399999999999995,"5min":1.109,"15min":1.198},"min":{"1min":0.38600000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.879,"5min":3.508,"15min":3.508},"last":0.52400000000000002}]},{"osd":6,"last update":"Wed Sep 11 08:26:30 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":1.246,"15min":1.631},"min":{"1min":0.32700000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":4.3289999999999997,"5min":6.2850000000000001,"15min":60.817},"last":0.443},{"interface":"front","average":{"1min":0.54600000000000004,"5min":1.1180000000000001,"15min":1.73},"min":{"1min":0.35599999999999998,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.90100000000000002,"5min":4.407,"15min":61.264000000000003},"last":0.503}]},{"osd":7,"last update":"Wed Sep 11 08:26:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.90100000000000002,"15min":1.962},"min":{"1min":0.374,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.86399999999999999,"5min":4.0090000000000003,"15min":60.847000000000001},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.98299999999999998,"15min":2.089},"min":{"1min":0.374,"5min":0.34499999999999997,"15min":0.33800000000000002},"max":{"1min":1.272,"5min":4.5220000000000002,"15min":60.883000000000003},"last":0.38700000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411593,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2256880,"kb_used_data":1671144,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":91488272,"statfs":{"total":95995035648,"available":93683990528,"internally_reserved":0,"allocated":1711251456,"data_stored":1710803146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:27:05 2024","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":4.1180000000000003,"15min":2.7250000000000001},"min":{"1min":0.216,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":4.7080000000000002,"5min":93.971000000000004,"15min":93.971000000000004},"last":4.6479999999999997},{"interface":"front","average":{"1min":0.88100000000000001,"5min":4.4379999999999997,"15min":3.056},"min":{"1min":0.219,"5min":0.20599999999999999,"15min":0.17999999999999999},"max":{"1min":4.5300000000000002,"5min":93.790999999999997,"15min":93.790999999999997},"last":0.32500000000000001}]},{"osd":1,"last update":"Wed Sep 11 08:27:05 2024","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":1.2010000000000001,"15min":1.607},"min":{"1min":0.20899999999999999,"5min":0.19800000000000001,"15min":0.17599999999999999},"max":{"1min":4.6689999999999996,"5min":7.9560000000000004,"15min":53.347000000000001},"last":1.857},{"interface":"front","average":{"1min":0.86399999999999999,"5min":1.2350000000000001,"15min":1.607},"min":{"1min":0.28999999999999998,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":4.5579999999999998,"5min":7.7880000000000003,"15min":53.082999999999998},"last":0.35799999999999998}]},{"osd":3,"last update":"Wed Sep 11 08:26:26 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":1.2749999999999999,"15min":1.544},"min":{"1min":0.22,"5min":0.22,"15min":0.188},"max":{"1min":1.1579999999999999,"5min":8.0630000000000006,"15min":46.070999999999998},"last":1.831},{"interface":"front","average":{"1min":0.44400000000000001,"5min":1.3740000000000001,"15min":1.516},"min":{"1min":0.255,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":1.3149999999999999,"5min":7.827,"15min":45.790999999999997},"last":2.5640000000000001}]},{"osd":4,"last update":"Wed Sep 11 08:26:43 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":1.1539999999999999,"15min":1.0680000000000001},"min":{"1min":0.26700000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.871,"5min":5.9470000000000001,"15min":7.2480000000000002},"last":1.9710000000000001},{"interface":"front","average":{"1min":0.51600000000000001,"5min":1.361,"15min":1.1120000000000001},"min":{"1min":0.34200000000000003,"5min":0.28699999999999998,"15min":0.252},"max":{"1min":0.72599999999999998,"5min":7.8970000000000002,"15min":7.8970000000000002},"last":0.434}]},{"osd":5,"last update":"Wed Sep 11 08:26:28 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":1.5049999999999999,"15min":1.6759999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.70299999999999996,"5min":3.4620000000000002,"15min":3.4620000000000002},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.58899999999999997,"5min":1.3,"15min":1.4179999999999999},"min":{"1min":0.311,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.629,"5min":3.9039999999999999,"15min":3.9039999999999999},"last":1.9390000000000001}]},{"osd":6,"last update":"Wed Sep 11 08:26:34 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":1.341,"15min":1.1499999999999999},"min":{"1min":0.30299999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.452,"5min":8.0350000000000001,"15min":8.0350000000000001},"last":1.7949999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":1.381,"15min":1.2210000000000001},"min":{"1min":0.375,"5min":0.35099999999999998,"15min":0.32500000000000001},"max":{"1min":0.85699999999999998,"5min":7.8710000000000004,"15min":7.8710000000000004},"last":0.39300000000000002}]},{"osd":7,"last update":"Wed Sep 11 08:26:43 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":1.087,"15min":1.3069999999999999},"min":{"1min":0.312,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.83799999999999997,"5min":4.7300000000000004,"15min":35.274999999999999},"last":1.9139999999999999},{"interface":"front","average":{"1min":0.58699999999999997,"5min":1.27,"15min":1.4370000000000001},"min":{"1min":0.38800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.82599999999999996,"5min":7.9219999999999997,"15min":35.265000000000001},"last":0.45200000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575119,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3133652,"kb_used_data":2243404,"kb_used_omap":1,"kb_used_meta":890238,"kb_avail":90611500,"statfs":{"total":95995035648,"available":92786176000,"internally_reserved":0,"allocated":2297245696,"data_stored":2296761538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":911604056},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:26:42 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":1.008,"15min":1.4990000000000001},"min":{"1min":0.26200000000000001,"5min":0.16700000000000001,"15min":0.16200000000000001},"max":{"1min":0.99399999999999999,"5min":6.9100000000000001,"15min":63.932000000000002},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.48699999999999999,"5min":1.4550000000000001,"15min":1.8700000000000001},"min":{"1min":0.25,"5min":0.182,"15min":0.16800000000000001},"max":{"1min":1.4910000000000001,"5min":13.023999999999999,"15min":64.269000000000005},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Sep 11 08:26:19 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":1.2090000000000001,"15min":1.3879999999999999},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.189},"max":{"1min":0.88600000000000001,"5min":20.925999999999998,"15min":20.925999999999998},"last":1.1319999999999999},{"interface":"front","average":{"1min":0.52300000000000002,"5min":1.2290000000000001,"15min":1.452},"min":{"1min":0.23899999999999999,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":2.4089999999999998,"5min":20.812000000000001,"15min":20.812000000000001},"last":0.96999999999999997}]},{"osd":3,"last update":"Wed Sep 11 08:26:30 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.86499999999999999,"15min":1.643},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19700000000000001},"max":{"1min":0.71699999999999997,"5min":8.0579999999999998,"15min":16.783999999999999},"last":2.125},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.96499999999999997,"15min":1.5900000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22600000000000001},"max":{"1min":6.5010000000000003,"5min":6.5010000000000003,"15min":14.973000000000001},"last":2.0880000000000001}]},{"osd":4,"last update":"Wed Sep 11 08:26:49 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":1.1140000000000001,"15min":1.772},"min":{"1min":0.36199999999999999,"5min":0.254,"15min":0.252},"max":{"1min":1.1899999999999999,"5min":7.4850000000000003,"15min":110.86499999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.622,"5min":1.2969999999999999,"15min":1.8069999999999999},"min":{"1min":0.41799999999999998,"5min":0.28399999999999997,"15min":0.248},"max":{"1min":1.137,"5min":5.7119999999999997,"15min":110.831},"last":0.71199999999999997}]},{"osd":5,"last update":"Wed Sep 11 08:26:30 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":1.4450000000000001,"15min":1.579},"min":{"1min":0.35999999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.8600000000000001,"5min":4.1050000000000004,"15min":4.1050000000000004},"last":0.75},{"interface":"front","average":{"1min":0.81100000000000005,"5min":1.4390000000000001,"15min":1.5429999999999999},"min":{"1min":0.38700000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":2.5920000000000001,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":0.77300000000000002}]},{"osd":6,"last update":"Wed Sep 11 08:26:30 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":1.256,"15min":1.8220000000000001},"min":{"1min":0.39900000000000002,"5min":0.315,"15min":0.29399999999999998},"max":{"1min":2.6579999999999999,"5min":8.1059999999999999,"15min":124.31999999999999},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.77900000000000003,"5min":1.1759999999999999,"15min":1.8140000000000001},"min":{"1min":0.38300000000000001,"5min":0.32300000000000001,"15min":0.29799999999999999},"max":{"1min":2.415,"5min":4.8170000000000002,"15min":124.958},"last":0.79500000000000004}]},{"osd":7,"last update":"Wed Sep 11 08:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":1.0800000000000001,"15min":1.6859999999999999},"min":{"1min":0.35499999999999998,"5min":0.318,"15min":0.28599999999999998},"max":{"1min":1.2250000000000001,"5min":7.5229999999999997,"15min":109.149},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.77500000000000002,"5min":1.0920000000000001,"15min":1.8089999999999999},"min":{"1min":0.372,"5min":0.32200000000000001,"15min":0.318},"max":{"1min":3.004,"5min":8.141,"15min":109.309},"last":0.63900000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738642,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2205120,"kb_used_data":1294392,"kb_used_omap":1,"kb_used_meta":910718,"kb_avail":91540032,"statfs":{"total":95995035648,"available":93736992768,"internally_reserved":0,"allocated":1325457408,"data_stored":1324987973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":932575576},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 11 08:26:36 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.89600000000000002,"15min":1.236},"min":{"1min":0.20599999999999999,"5min":0.159,"15min":0.159},"max":{"1min":3.7970000000000002,"5min":8.0370000000000008,"15min":10.105},"last":98.373999999999995},{"interface":"front","average":{"1min":0.64800000000000002,"5min":1.0669999999999999,"15min":1.5369999999999999},"min":{"1min":0.219,"5min":0.17299999999999999,"15min":0.153},"max":{"1min":4.0410000000000004,"5min":9.6799999999999997,"15min":19.852},"last":98.257999999999996}]},{"osd":2,"last update":"Wed Sep 11 08:27:02 2024","interfaces":[{"interface":"back","average":{"1min":2.5840000000000001,"5min":1.8720000000000001,"15min":1.4790000000000001},"min":{"1min":0.22800000000000001,"5min":0.19,"15min":0.19},"max":{"1min":46.417000000000002,"5min":46.417000000000002,"15min":46.417000000000002},"last":0.316},{"interface":"front","average":{"1min":2.625,"5min":1.8999999999999999,"15min":1.554},"min":{"1min":0.26000000000000001,"5min":0.22600000000000001,"15min":0.186},"max":{"1min":46.438000000000002,"5min":46.438000000000002,"15min":46.438000000000002},"last":0.379}]},{"osd":3,"last update":"Wed Sep 11 08:26:24 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":2.7200000000000002,"15min":2.1179999999999999},"min":{"1min":0.24399999999999999,"5min":0.24199999999999999,"15min":0.22900000000000001},"max":{"1min":3.8519999999999999,"5min":172.149,"15min":172.149},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.71199999999999997,"5min":2.7589999999999999,"15min":2.1429999999999998},"min":{"1min":0.26400000000000001,"5min":0.20599999999999999,"15min":0.20300000000000001},"max":{"1min":4.2709999999999999,"5min":174.899,"15min":174.899},"last":0.36199999999999999}]},{"osd":4,"last update":"Wed Sep 11 08:26:49 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":1.391,"15min":1.228},"min":{"1min":0.308,"5min":0.308,"15min":0.24099999999999999},"max":{"1min":0.89800000000000002,"5min":9.6560000000000006,"15min":9.6560000000000006},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":1.304,"15min":1.2130000000000001},"min":{"1min":0.315,"5min":0.26400000000000001,"15min":0.24399999999999999},"max":{"1min":0.79100000000000004,"5min":6.6100000000000003,"15min":8.3510000000000009},"last":0.48399999999999999}]},{"osd":5,"last update":"Wed Sep 11 08:26:30 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":1.7889999999999999,"15min":1.9470000000000001},"min":{"1min":0.317,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":3.8940000000000001,"5min":4.8220000000000001,"15min":4.8220000000000001},"last":0.43099999999999999},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.663,"15min":1.7649999999999999},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":4.5090000000000003,"5min":7.718,"15min":7.718},"last":0.49299999999999999}]},{"osd":6,"last update":"Wed Sep 11 08:26:30 2024","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":1.4099999999999999,"15min":1.593},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.29099999999999998},"max":{"1min":4.2389999999999999,"5min":9.7029999999999994,"15min":78.522000000000006},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.97599999999999998,"5min":1.452,"15min":1.5},"min":{"1min":0.38100000000000001,"5min":0.33100000000000002,"15min":0.28299999999999997},"max":{"1min":4.5830000000000002,"5min":6.8150000000000004,"15min":79.808000000000007},"last":0.53700000000000003}]},{"osd":7,"last update":"Wed Sep 11 08:26:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":1.214,"15min":1.262},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.30399999999999999},"max":{"1min":0.95399999999999996,"5min":6.8520000000000003,"15min":7.5350000000000001},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.60299999999999998,"5min":1.3,"15min":1.353},"min":{"1min":0.40699999999999997,"5min":0.32900000000000001,"15min":0.32000000000000001},"max":{"1min":1.393,"5min":6.875,"15min":13.914},"last":0.56200000000000006}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3425,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1336803328,"data_stored":1336803328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2321154048,"data_stored":2321154048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1729822720,"data_stored":1729822720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2027945984,"data_stored":2027945984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1143799808,"data_stored":1143799808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1130037248,"data_stored":1130037248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2288779264,"data_stored":2288779264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1710686208,"data_stored":1710686208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T08:27:22.632 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-11T08:27:22.632 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T08:27:22.633 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-09-11T08:27:22.633 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-11T08:27:22.633 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-11T08:27:22.633 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.0 config set filestore_inject_stall 3 2024-09-11T08:27:22.715 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:22.747 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:22.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:22.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:22.849 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:22.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:22.903 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:22.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:23.004 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:27:23.058 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:23.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:23.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:27:23.161 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:27:23.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:23.223 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:23.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:27:23.234 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:27:23.238 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:27:23.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:23 smithi181 ceph-mon[32701]: 6.19 scrub starts 2024-09-11T08:27:23.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:23 smithi181 ceph-mon[32701]: 6.19 scrub ok 2024-09-11T08:27:23.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:23 smithi181 ceph-mon[32701]: 3.a scrub starts 2024-09-11T08:27:23.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:23 smithi181 ceph-mon[32701]: 3.a scrub ok 2024-09-11T08:27:23.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:23 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:27:23.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:23 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:27:23.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:23 smithi181 ceph-mon[32701]: 4.14 scrub starts 2024-09-11T08:27:23.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:23 smithi181 ceph-mon[32701]: 4.14 scrub ok 2024-09-11T08:27:23.324 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:23.353 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:23 smithi150 ceph-mon[29276]: 6.19 scrub starts 2024-09-11T08:27:23.353 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:23 smithi150 ceph-mon[29276]: 6.19 scrub ok 2024-09-11T08:27:23.353 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:23 smithi150 ceph-mon[29276]: 3.a scrub starts 2024-09-11T08:27:23.353 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:23 smithi150 ceph-mon[29276]: 3.a scrub ok 2024-09-11T08:27:23.353 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:23 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:27:23.353 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:23 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:27:23.354 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:23 smithi150 ceph-mon[29276]: 4.14 scrub starts 2024-09-11T08:27:23.354 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:23 smithi150 ceph-mon[29276]: 4.14 scrub ok 2024-09-11T08:27:23.354 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:23 smithi150 ceph-mon[39612]: 6.19 scrub starts 2024-09-11T08:27:23.354 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:23 smithi150 ceph-mon[39612]: 6.19 scrub ok 2024-09-11T08:27:23.354 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:23 smithi150 ceph-mon[39612]: 3.a scrub starts 2024-09-11T08:27:23.355 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:23 smithi150 ceph-mon[39612]: 3.a scrub ok 2024-09-11T08:27:23.355 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:23 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:27:23.355 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:23 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:27:23.355 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:23 smithi150 ceph-mon[39612]: 4.14 scrub starts 2024-09-11T08:27:23.355 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:23 smithi150 ceph-mon[39612]: 4.14 scrub ok 2024-09-11T08:27:23.367 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:23.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:23.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:23.469 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:27:23.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:23.509 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:23.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:27:23.610 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:23.629 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 43 16 75787 75771 110.084 71.9375 0.00865909 0.00907874 2024-09-11T08:27:23.636 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:23.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:23.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:23.738 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:23.771 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:23.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:23.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:23.873 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:23.911 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:23.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:23.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:24.013 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:24.040 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:24.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:24.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:24.141 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:24.193 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:24.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:24.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:24.282 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-11T08:27:24.295 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:24.316 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:24 smithi181 ceph-mon[32701]: 4.9 scrub starts 2024-09-11T08:27:24.316 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:24 smithi181 ceph-mon[32701]: 4.9 scrub ok 2024-09-11T08:27:24.316 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:24 smithi181 ceph-mon[32701]: osdmap e399: 8 total, 8 up, 8 in 2024-09-11T08:27:24.316 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:24 smithi181 ceph-mon[32701]: pgmap v907: 148 pgs: 148 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-09-11T08:27:24.324 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:24 smithi150 ceph-mon[29276]: 4.9 scrub starts 2024-09-11T08:27:24.324 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:24 smithi150 ceph-mon[29276]: 4.9 scrub ok 2024-09-11T08:27:24.324 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:24 smithi150 ceph-mon[29276]: osdmap e399: 8 total, 8 up, 8 in 2024-09-11T08:27:24.324 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:24 smithi150 ceph-mon[29276]: pgmap v907: 148 pgs: 148 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-09-11T08:27:24.324 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:24 smithi150 ceph-mon[39612]: 4.9 scrub starts 2024-09-11T08:27:24.324 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:24 smithi150 ceph-mon[39612]: 4.9 scrub ok 2024-09-11T08:27:24.324 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:24 smithi150 ceph-mon[39612]: osdmap e399: 8 total, 8 up, 8 in 2024-09-11T08:27:24.324 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:24 smithi150 ceph-mon[39612]: pgmap v907: 148 pgs: 148 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-09-11T08:27:24.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:24.336 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:24.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:24.438 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:24.482 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:24.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:24.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:24.584 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:27:24.629 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 44 16 76918 76902 109.188 70.6875 0.00849544 0.00915351 2024-09-11T08:27:24.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:24.641 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:24.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:27:24.665 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:27:24.743 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:27:24.770 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:24.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:24.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:27:24.872 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:24.911 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:24.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:24.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:25.013 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:25.058 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:25.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:25.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:25.089 INFO:teuthology.orchestra.run.smithi150.stderr:noscrub is set 2024-09-11T08:27:25.159 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:27:25.192 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:25.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:25.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:27:25.294 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:25.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:25 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/329700010' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:27:25.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:25 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:27:25.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:25 smithi181 ceph-mon[32701]: 6.0 scrub starts 2024-09-11T08:27:25.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:25 smithi181 ceph-mon[32701]: 6.0 scrub ok 2024-09-11T08:27:25.323 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:27:25 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:27:25] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:27:25.337 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:25.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:25.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:25.439 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:25.475 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:25.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:25.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:25.483 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:25 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/329700010' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:27:25.483 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:25 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:27:25.483 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:25 smithi150 ceph-mon[29276]: 6.0 scrub starts 2024-09-11T08:27:25.483 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:25 smithi150 ceph-mon[29276]: 6.0 scrub ok 2024-09-11T08:27:25.484 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:25 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/329700010' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:27:25.484 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:25 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:27:25.484 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:25 smithi150 ceph-mon[39612]: 6.0 scrub starts 2024-09-11T08:27:25.484 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:25 smithi150 ceph-mon[39612]: 6.0 scrub ok 2024-09-11T08:27:25.577 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:25.608 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:25.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:25.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:25.630 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 45 16 77992 77976 108.254 67.125 0.0238533 0.00923229 2024-09-11T08:27:25.709 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:25.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:25.753 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:25.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:25.855 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:25.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:25.889 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:25.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:25.990 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:26.037 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:26.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:26.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:26.111 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:26 smithi181 ceph-mon[32701]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:27:26.111 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:26 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:27:26.111 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:26 smithi181 ceph-mon[32701]: osdmap e400: 8 total, 8 up, 8 in 2024-09-11T08:27:26.112 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:26 smithi181 ceph-mon[32701]: pgmap v909: 148 pgs: 148 active+clean; 4.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 58 MiB/s wr, 927 op/s 2024-09-11T08:27:26.112 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:26 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:27:26.112 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:26 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:27:26.139 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:26.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:26.187 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:26.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:26.288 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:26.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:26.318 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:26.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:26.420 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:27:26.458 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:26.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:26.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:27:26.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:26 smithi150 ceph-mon[29276]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:27:26.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:26 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:27:26.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:26 smithi150 ceph-mon[29276]: osdmap e400: 8 total, 8 up, 8 in 2024-09-11T08:27:26.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:26 smithi150 ceph-mon[29276]: pgmap v909: 148 pgs: 148 active+clean; 4.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 58 MiB/s wr, 927 op/s 2024-09-11T08:27:26.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:26 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:27:26.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:26 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:27:26.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:26 smithi150 ceph-mon[39612]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:27:26.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:26 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:27:26.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:26 smithi150 ceph-mon[39612]: osdmap e400: 8 total, 8 up, 8 in 2024-09-11T08:27:26.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:26 smithi150 ceph-mon[39612]: pgmap v909: 148 pgs: 148 active+clean; 4.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 58 MiB/s wr, 927 op/s 2024-09-11T08:27:26.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:26 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:27:26.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:26 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:27:26.540 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-11T08:27:26.540 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-09-11T08:27:26.540 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-09-11T08:27:26.540 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-09-11T08:27:26.540 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.540 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.15369.0:22563 9.4 e252/246 9:2f4bd629:::benchmark_data_smithi150_133616_object22562:head v 252'1386, mlcod=252'1381)", 2024-09-11T08:27:26.540 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:18:19.413764+0000", 2024-09-11T08:27:26.540 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 547.12229571700004, 2024-09-11T08:27:26.541 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.153809685, 2024-09-11T08:27:26.541 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:27:26.541 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:27:26.541 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:27:26.541 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.541 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:27:26.541 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.413764+0000", 2024-09-11T08:27:26.541 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:26.541 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.541 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.541 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:27:26.541 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.413764+0000", 2024-09-11T08:27:26.541 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:26.541 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.542 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.542 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:27:26.542 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.413763+0000", 2024-09-11T08:27:26.542 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-09-11T08:27:26.542 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.542 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.542 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:27:26.542 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.413847+0000", 2024-09-11T08:27:26.542 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3230000000000001e-05 2024-09-11T08:27:26.542 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.542 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.542 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:27:26.542 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.413857+0000", 2024-09-11T08:27:26.543 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0164e-05 2024-09-11T08:27:26.543 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.543 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.543 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:27:26.543 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.413862+0000", 2024-09-11T08:27:26.543 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2209999999999996e-06 2024-09-11T08:27:26.543 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.543 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.543 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:26.543 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.413915+0000", 2024-09-11T08:27:26.543 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2558000000000001e-05 2024-09-11T08:27:26.543 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.543 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.543 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:27:26.544 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.413932+0000", 2024-09-11T08:27:26.544 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7907e-05 2024-09-11T08:27:26.544 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.544 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.544 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:27:26.544 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.567542+0000", 2024-09-11T08:27:26.544 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.153609146 2024-09-11T08:27:26.544 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.544 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.544 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:27:26.544 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.567574+0000", 2024-09-11T08:27:26.544 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2060000000000001e-05 2024-09-11T08:27:26.544 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:26.544 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:27:26.545 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:26.545 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.545 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.545 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.15369.0:22571 9.e e252/246 9:7e6ad66b:::benchmark_data_smithi150_133616_object22570:head v 252'1338, mlcod=252'1336)", 2024-09-11T08:27:26.545 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:18:19.414276+0000", 2024-09-11T08:27:26.545 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 547.12178375300005, 2024-09-11T08:27:26.545 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.15584709999999999, 2024-09-11T08:27:26.545 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:27:26.545 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:27:26.545 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:27:26.545 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.545 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:27:26.545 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.414276+0000", 2024-09-11T08:27:26.545 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:26.546 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.546 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.546 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:27:26.546 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.414276+0000", 2024-09-11T08:27:26.546 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:26.546 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.546 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.546 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:27:26.546 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.414275+0000", 2024-09-11T08:27:26.546 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:27:26.546 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.546 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.546 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:27:26.546 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.414321+0000", 2024-09-11T08:27:26.547 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.596e-05 2024-09-11T08:27:26.547 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.547 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.547 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:27:26.547 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.414331+0000", 2024-09-11T08:27:26.547 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7890000000000006e-06 2024-09-11T08:27:26.547 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.547 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.547 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:27:26.547 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.414336+0000", 2024-09-11T08:27:26.547 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3700000000000003e-06 2024-09-11T08:27:26.547 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.547 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.547 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:26.548 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.414388+0000", 2024-09-11T08:27:26.548 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1499000000000003e-05 2024-09-11T08:27:26.548 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.548 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.548 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:27:26.548 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.414406+0000", 2024-09-11T08:27:26.548 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8329999999999999e-05 2024-09-11T08:27:26.548 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.548 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.548 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:27:26.548 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.570091+0000", 2024-09-11T08:27:26.548 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.15568486300000001 2024-09-11T08:27:26.548 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.548 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.549 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:27:26.549 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.570123+0000", 2024-09-11T08:27:26.549 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2048999999999998e-05 2024-09-11T08:27:26.549 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:26.549 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:27:26.549 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:26.549 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.549 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.549 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.15369.0:22576 9.4 e252/246 9:27986325:::benchmark_data_smithi150_133616_object22575:head v 252'1387, mlcod=252'1383)", 2024-09-11T08:27:26.549 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:18:19.417025+0000", 2024-09-11T08:27:26.549 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 547.11903501999996, 2024-09-11T08:27:26.549 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.15302355300000001, 2024-09-11T08:27:26.549 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:27:26.549 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:27:26.550 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:27:26.550 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.550 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:27:26.550 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.417025+0000", 2024-09-11T08:27:26.550 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:26.550 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.550 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.550 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:27:26.550 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.417025+0000", 2024-09-11T08:27:26.550 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:26.550 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.550 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.550 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:27:26.550 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.417023+0000", 2024-09-11T08:27:26.551 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:27:26.551 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.551 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.551 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:27:26.551 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.417065+0000", 2024-09-11T08:27:26.551 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.193e-05 2024-09-11T08:27:26.551 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.551 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.551 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:27:26.551 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.567545+0000", 2024-09-11T08:27:26.551 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.15047953 2024-09-11T08:27:26.551 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.551 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.551 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:27:26.552 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.567554+0000", 2024-09-11T08:27:26.552 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8540000000000003e-06 2024-09-11T08:27:26.552 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.552 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.552 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:26.552 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.567599+0000", 2024-09-11T08:27:26.552 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5454000000000002e-05 2024-09-11T08:27:26.552 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.552 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.552 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:27:26.552 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.567616+0000", 2024-09-11T08:27:26.553 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7173e-05 2024-09-11T08:27:26.553 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.553 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.553 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:27:26.553 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.570013+0000", 2024-09-11T08:27:26.554 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0023963600000000002 2024-09-11T08:27:26.554 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.554 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.554 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:27:26.554 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:19.570048+0000", 2024-09-11T08:27:26.554 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5463999999999998e-05 2024-09-11T08:27:26.554 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:26.554 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:27:26.554 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:26.554 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.554 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.554 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:66554 9.c 9:381c7998:::benchmark_data_smithi150_133616_object66553:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-09-11T08:27:26.554 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:18:44.642501+0000", 2024-09-11T08:27:26.555 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 521.89355911400003, 2024-09-11T08:27:26.555 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.13431788, 2024-09-11T08:27:26.555 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:27:26.555 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:27:26.555 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:27:26.555 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:27:26.555 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:27:26.555 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 66554 2024-09-11T08:27:26.555 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.555 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:27:26.555 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.555 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:27:26.555 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:44.642501+0000", 2024-09-11T08:27:26.555 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:26.556 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.556 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.556 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:27:26.556 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:44.642501+0000", 2024-09-11T08:27:26.556 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:26.556 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.556 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.556 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:27:26.556 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:44.642500+0000", 2024-09-11T08:27:26.556 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:27:26.557 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.557 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.557 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:27:26.557 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:44.642527+0000", 2024-09-11T08:27:26.557 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7770999999999999e-05 2024-09-11T08:27:26.557 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.557 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.557 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:27:26.557 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:44.642534+0000", 2024-09-11T08:27:26.557 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1890000000000004e-06 2024-09-11T08:27:26.558 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.558 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.558 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:27:26.558 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:44.642537+0000", 2024-09-11T08:27:26.558 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2329999999999998e-06 2024-09-11T08:27:26.558 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.558 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.558 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:26.558 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:44.642563+0000", 2024-09-11T08:27:26.558 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6633999999999999e-05 2024-09-11T08:27:26.558 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.558 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.558 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:27:26.558 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:44.642649+0000", 2024-09-11T08:27:26.559 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5389999999999994e-05 2024-09-11T08:27:26.559 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.559 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.559 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3,7", 2024-09-11T08:27:26.559 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:44.642730+0000", 2024-09-11T08:27:26.559 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0916999999999998e-05 2024-09-11T08:27:26.559 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.559 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.559 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:27:26.559 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:44.644391+0000", 2024-09-11T08:27:26.559 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0016608899999999999 2024-09-11T08:27:26.559 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.559 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.559 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:26.560 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:44.645849+0000", 2024-09-11T08:27:26.560 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00145871 2024-09-11T08:27:26.560 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.560 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.560 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:26.560 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:44.776209+0000", 2024-09-11T08:27:26.560 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.13035939499999999 2024-09-11T08:27:26.560 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.560 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.560 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:27:26.560 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:44.776550+0000", 2024-09-11T08:27:26.560 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00034083799999999999 2024-09-11T08:27:26.560 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.561 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.561 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:27:26.561 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:44.776818+0000", 2024-09-11T08:27:26.561 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000268881 2024-09-11T08:27:26.561 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:26.561 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:27:26.561 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:26.561 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.561 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.561 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:67150 9.c 9:346bcf81:::benchmark_data_smithi150_133616_object67149:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-09-11T08:27:26.561 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:18:45.195553+0000", 2024-09-11T08:27:26.561 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 521.34050652099995, 2024-09-11T08:27:26.561 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.12375691699999999, 2024-09-11T08:27:26.561 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:27:26.562 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:27:26.562 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:27:26.562 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:27:26.562 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:27:26.562 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 67150 2024-09-11T08:27:26.562 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.562 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:27:26.562 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.562 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:27:26.562 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:45.195553+0000", 2024-09-11T08:27:26.562 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:26.562 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.562 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.563 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:27:26.563 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:45.195553+0000", 2024-09-11T08:27:26.563 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:26.563 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.563 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.563 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:27:26.563 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:45.195551+0000", 2024-09-11T08:27:26.563 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:27:26.563 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.563 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.563 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:27:26.563 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:45.195599+0000", 2024-09-11T08:27:26.563 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7015999999999998e-05 2024-09-11T08:27:26.563 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.563 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.564 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:27:26.564 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:45.195612+0000", 2024-09-11T08:27:26.564 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3086e-05 2024-09-11T08:27:26.564 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.564 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.564 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:27:26.564 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:45.195618+0000", 2024-09-11T08:27:26.564 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5950000000000004e-06 2024-09-11T08:27:26.564 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.564 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.564 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:26.564 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:45.195662+0000", 2024-09-11T08:27:26.564 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3443999999999999e-05 2024-09-11T08:27:26.564 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.565 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.565 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:27:26.565 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:45.195768+0000", 2024-09-11T08:27:26.565 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010684 2024-09-11T08:27:26.565 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.565 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.565 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3,7", 2024-09-11T08:27:26.565 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:45.195889+0000", 2024-09-11T08:27:26.565 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000120531 2024-09-11T08:27:26.565 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.565 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.565 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:27:26.565 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:45.198369+0000", 2024-09-11T08:27:26.565 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0024802079999999998 2024-09-11T08:27:26.566 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.566 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.566 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:26.566 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:45.200598+0000", 2024-09-11T08:27:26.566 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002228299 2024-09-11T08:27:26.566 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.566 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.566 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:26.566 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:45.319139+0000", 2024-09-11T08:27:26.566 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.11854167 2024-09-11T08:27:26.566 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.566 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.566 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:27:26.566 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:45.319269+0000", 2024-09-11T08:27:26.567 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012987099999999999 2024-09-11T08:27:26.567 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.567 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.567 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:27:26.567 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:45.319310+0000", 2024-09-11T08:27:26.567 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1020999999999999e-05 2024-09-11T08:27:26.567 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:26.567 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:27:26.567 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:26.567 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.567 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.567 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:67155 9.c 9:3332e946:::benchmark_data_smithi150_133616_object67154:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-09-11T08:27:26.567 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:18:45.198607+0000", 2024-09-11T08:27:26.567 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 521.337452601, 2024-09-11T08:27:26.567 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.12993901399999999, 2024-09-11T08:27:26.568 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:27:26.568 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:27:26.568 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:27:26.568 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:27:26.568 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:27:26.568 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 67155 2024-09-11T08:27:26.568 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.568 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:27:26.568 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.568 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:27:26.568 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:45.198607+0000", 2024-09-11T08:27:26.568 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:26.568 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.568 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.569 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:27:26.569 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:45.198607+0000", 2024-09-11T08:27:26.569 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:26.569 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:26.570 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.570 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.570 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:27:26.570 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:45.198605+0000", 2024-09-11T08:27:26.570 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:27:26.570 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.570 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.570 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:27:26.571 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:45.198710+0000", 2024-09-11T08:27:26.571 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010420300000000001 2024-09-11T08:27:26.571 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.571 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.571 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:27:26.571 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:45.198721+0000", 2024-09-11T08:27:26.571 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1416e-05 2024-09-11T08:27:26.571 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.571 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.571 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:27:26.571 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:45.198726+0000", 2024-09-11T08:27:26.571 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4480000000000002e-06 2024-09-11T08:27:26.571 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.571 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.572 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:26.572 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:45.198766+0000", 2024-09-11T08:27:26.572 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9073999999999999e-05 2024-09-11T08:27:26.572 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.572 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.572 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:27:26.572 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:45.198874+0000", 2024-09-11T08:27:26.572 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000108185 2024-09-11T08:27:26.572 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.572 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.572 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3,7", 2024-09-11T08:27:26.572 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:45.199001+0000", 2024-09-11T08:27:26.572 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012727999999999999 2024-09-11T08:27:26.572 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.573 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.573 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:27:26.573 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:45.200699+0000", 2024-09-11T08:27:26.573 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0016983370000000001 2024-09-11T08:27:26.573 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.573 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.573 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:26.573 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:45.202014+0000", 2024-09-11T08:27:26.573 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00131423 2024-09-11T08:27:26.573 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.573 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.573 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:26.573 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:45.328411+0000", 2024-09-11T08:27:26.573 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.12639699600000001 2024-09-11T08:27:26.574 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.574 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.574 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:27:26.574 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:45.328504+0000", 2024-09-11T08:27:26.574 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.2961000000000007e-05 2024-09-11T08:27:26.574 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.574 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.574 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:27:26.574 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:45.328546+0000", 2024-09-11T08:27:26.574 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2525999999999999e-05 2024-09-11T08:27:26.574 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:26.574 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:27:26.574 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:26.574 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.575 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.575 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:69570 9.c 9:34a50db1:::benchmark_data_smithi150_133616_object69569:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e256)", 2024-09-11T08:27:26.575 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:18:47.408564+0000", 2024-09-11T08:27:26.575 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 519.12749589999999, 2024-09-11T08:27:26.575 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.118172306, 2024-09-11T08:27:26.575 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:27:26.575 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:27:26.575 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:27:26.575 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:27:26.575 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:27:26.575 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 69570 2024-09-11T08:27:26.575 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.575 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:27:26.575 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.576 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:27:26.576 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:47.408564+0000", 2024-09-11T08:27:26.576 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:26.576 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.576 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.576 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:27:26.576 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:47.408564+0000", 2024-09-11T08:27:26.576 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:26.576 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.576 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.576 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:27:26.576 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:47.408562+0000", 2024-09-11T08:27:26.577 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999981 2024-09-11T08:27:26.577 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.577 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.577 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:27:26.577 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:47.408603+0000", 2024-09-11T08:27:26.577 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0747999999999997e-05 2024-09-11T08:27:26.577 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.577 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.577 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:27:26.577 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:47.408614+0000", 2024-09-11T08:27:26.577 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1735e-05 2024-09-11T08:27:26.577 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.577 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.578 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:27:26.578 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:47.408620+0000", 2024-09-11T08:27:26.578 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.947e-06 2024-09-11T08:27:26.578 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.578 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.578 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:26.578 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:47.408663+0000", 2024-09-11T08:27:26.578 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2799999999999997e-05 2024-09-11T08:27:26.578 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.578 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.578 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:27:26.578 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:47.408757+0000", 2024-09-11T08:27:26.578 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.3387999999999994e-05 2024-09-11T08:27:26.578 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.579 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.579 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3,7", 2024-09-11T08:27:26.579 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:47.408835+0000", 2024-09-11T08:27:26.579 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8972000000000002e-05 2024-09-11T08:27:26.579 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.579 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.579 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:27:26.579 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:47.414320+0000", 2024-09-11T08:27:26.579 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0054847999999999997 2024-09-11T08:27:26.579 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.579 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.579 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:26.579 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:47.414913+0000", 2024-09-11T08:27:26.579 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00059317699999999996 2024-09-11T08:27:26.580 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.580 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.580 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:26.580 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:47.526648+0000", 2024-09-11T08:27:26.580 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.11173414399999999 2024-09-11T08:27:26.580 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.580 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.580 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:27:26.580 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:47.526697+0000", 2024-09-11T08:27:26.580 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9639999999999999e-05 2024-09-11T08:27:26.580 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.580 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.580 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:27:26.580 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:47.526736+0000", 2024-09-11T08:27:26.581 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8825999999999997e-05 2024-09-11T08:27:26.581 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:26.581 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:27:26.581 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:26.581 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.581 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.581 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:69571 9.c 9:331fb8da:::benchmark_data_smithi150_133616_object69570:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e256)", 2024-09-11T08:27:26.581 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:18:47.408634+0000", 2024-09-11T08:27:26.581 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 519.12742570099999, 2024-09-11T08:27:26.581 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.11830056799999999, 2024-09-11T08:27:26.581 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:27:26.581 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:27:26.581 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:27:26.581 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:27:26.582 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:27:26.582 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 69571 2024-09-11T08:27:26.582 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.582 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:27:26.582 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.582 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:27:26.582 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:47.408634+0000", 2024-09-11T08:27:26.582 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:26.582 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.582 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.582 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:27:26.582 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:47.408634+0000", 2024-09-11T08:27:26.582 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:26.582 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.583 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.583 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:27:26.583 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:47.408633+0000", 2024-09-11T08:27:26.583 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-09-11T08:27:26.583 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.583 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.583 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:27:26.583 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:47.408672+0000", 2024-09-11T08:27:26.583 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.875e-05 2024-09-11T08:27:26.583 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.583 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.583 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:27:26.583 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:47.408679+0000", 2024-09-11T08:27:26.583 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5760000000000002e-06 2024-09-11T08:27:26.583 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.584 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.584 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:27:26.584 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:47.408683+0000", 2024-09-11T08:27:26.584 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2479999999999998e-06 2024-09-11T08:27:26.584 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.584 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.584 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:26.584 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:47.410842+0000", 2024-09-11T08:27:26.584 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002159429 2024-09-11T08:27:26.584 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.584 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.584 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:27:26.584 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:47.410951+0000", 2024-09-11T08:27:26.585 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000108356 2024-09-11T08:27:26.585 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.585 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.585 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3,7", 2024-09-11T08:27:26.585 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:47.411077+0000", 2024-09-11T08:27:26.585 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000125785 2024-09-11T08:27:26.585 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.585 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.585 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:27:26.585 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:47.414324+0000", 2024-09-11T08:27:26.585 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0032479090000000002 2024-09-11T08:27:26.585 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.585 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.586 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:26.586 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:47.416673+0000", 2024-09-11T08:27:26.586 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0023488580000000001 2024-09-11T08:27:26.586 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.586 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.586 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:26.586 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:47.526807+0000", 2024-09-11T08:27:26.586 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.110133332 2024-09-11T08:27:26.586 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.586 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.586 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:27:26.586 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:47.526885+0000", 2024-09-11T08:27:26.586 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8181000000000002e-05 2024-09-11T08:27:26.587 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.587 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.587 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:27:26.587 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:47.526935+0000", 2024-09-11T08:27:26.587 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9693e-05 2024-09-11T08:27:26.587 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:26.587 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:27:26.587 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:26.587 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.588 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.588 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:69572 9.c 9:3f5004e5:::benchmark_data_smithi150_133616_object69571:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e256)", 2024-09-11T08:27:26.588 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:18:47.410457+0000", 2024-09-11T08:27:26.588 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 519.12560246500004, 2024-09-11T08:27:26.588 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.116625719, 2024-09-11T08:27:26.588 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:27:26.588 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:27:26.588 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:27:26.588 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:27:26.588 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:27:26.588 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 69572 2024-09-11T08:27:26.588 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.588 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:27:26.589 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.589 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:27:26.589 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:47.410457+0000", 2024-09-11T08:27:26.589 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:26.589 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.589 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.589 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:27:26.589 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:47.410457+0000", 2024-09-11T08:27:26.589 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:26.589 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.589 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.589 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:27:26.589 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:47.410456+0000", 2024-09-11T08:27:26.589 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:27:26.590 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.590 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.590 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:27:26.590 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:47.410507+0000", 2024-09-11T08:27:26.590 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0671000000000003e-05 2024-09-11T08:27:26.590 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.590 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.590 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:27:26.590 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:47.410520+0000", 2024-09-11T08:27:26.590 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2916e-05 2024-09-11T08:27:26.590 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.590 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.590 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:27:26.590 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:47.410526+0000", 2024-09-11T08:27:26.591 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2589999999999997e-06 2024-09-11T08:27:26.591 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.591 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.591 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:26.591 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:47.413063+0000", 2024-09-11T08:27:26.591 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0025365460000000002 2024-09-11T08:27:26.591 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.591 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.591 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:27:26.591 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:47.413156+0000", 2024-09-11T08:27:26.591 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.3398000000000003e-05 2024-09-11T08:27:26.591 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.591 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.591 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3,7", 2024-09-11T08:27:26.591 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:47.413241+0000", 2024-09-11T08:27:26.592 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5031999999999995e-05 2024-09-11T08:27:26.592 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.592 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.592 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:27:26.592 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:47.414432+0000", 2024-09-11T08:27:26.592 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0011912209999999999 2024-09-11T08:27:26.592 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.592 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.592 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:26.592 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:47.416800+0000", 2024-09-11T08:27:26.592 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002367952 2024-09-11T08:27:26.592 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.592 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.593 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:26.593 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:47.526988+0000", 2024-09-11T08:27:26.593 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.110187891 2024-09-11T08:27:26.593 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.593 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.593 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:27:26.593 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:47.527046+0000", 2024-09-11T08:27:26.593 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7454000000000003e-05 2024-09-11T08:27:26.593 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.593 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.593 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:27:26.593 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:18:47.527083+0000", 2024-09-11T08:27:26.593 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7274999999999998e-05 2024-09-11T08:27:26.594 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:26.594 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:27:26.594 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:26.594 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.594 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.594 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24320.0:1796 5.e 5:7759931f:::notify.3:head [watch ping cookie 94783556098720] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e259)", 2024-09-11T08:27:26.594 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:19:08.688270+0000", 2024-09-11T08:27:26.594 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 497.847789987, 2024-09-11T08:27:26.594 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.91962045800000003, 2024-09-11T08:27:26.594 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:27:26.594 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-11T08:27:26.594 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:27:26.594 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24320", 2024-09-11T08:27:26.594 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/1503848706", 2024-09-11T08:27:26.595 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 1796 2024-09-11T08:27:26.595 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.595 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:27:26.595 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.595 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:27:26.595 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.688270+0000", 2024-09-11T08:27:26.595 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:26.595 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.595 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.595 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:27:26.595 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.688270+0000", 2024-09-11T08:27:26.595 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:26.595 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.595 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.595 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:27:26.596 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.688267+0000", 2024-09-11T08:27:26.596 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999971 2024-09-11T08:27:26.596 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.596 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.596 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:27:26.596 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.688285+0000", 2024-09-11T08:27:26.596 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8587e-05 2024-09-11T08:27:26.596 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.596 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.596 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:27:26.596 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.688321+0000", 2024-09-11T08:27:26.596 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5225000000000002e-05 2024-09-11T08:27:26.596 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.597 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.597 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:27:26.597 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.688343+0000", 2024-09-11T08:27:26.597 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2257000000000001e-05 2024-09-11T08:27:26.597 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.597 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.597 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:26.597 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.688521+0000", 2024-09-11T08:27:26.597 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000178176 2024-09-11T08:27:26.597 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.597 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.597 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:27:26.597 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.688532+0000", 2024-09-11T08:27:26.597 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1107999999999999e-05 2024-09-11T08:27:26.598 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.598 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.598 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:26.598 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:09.601059+0000", 2024-09-11T08:27:26.598 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.91252694999999995 2024-09-11T08:27:26.598 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.598 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.598 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for degraded object", 2024-09-11T08:27:26.598 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:09.601352+0000", 2024-09-11T08:27:26.598 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00029261400000000001 2024-09-11T08:27:26.598 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.598 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.598 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:26.599 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:09.607729+0000", 2024-09-11T08:27:26.599 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0063777249999999999 2024-09-11T08:27:26.599 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.599 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.599 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:27:26.599 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:09.607812+0000", 2024-09-11T08:27:26.599 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3028000000000005e-05 2024-09-11T08:27:26.599 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.599 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.599 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:27:26.599 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:09.607890+0000", 2024-09-11T08:27:26.599 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7684999999999999e-05 2024-09-11T08:27:26.599 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:26.599 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:27:26.600 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:26.600 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.600 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.600 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24320.0:1803 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 94783556098720 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-11T08:27:26.600 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:19:08.688649+0000", 2024-09-11T08:27:26.600 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 497.84741030200001, 2024-09-11T08:27:26.600 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.925214231, 2024-09-11T08:27:26.600 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:27:26.600 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-11T08:27:26.600 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:27:26.600 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24320", 2024-09-11T08:27:26.600 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/1503848706", 2024-09-11T08:27:26.600 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 1803 2024-09-11T08:27:26.601 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.601 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:27:26.601 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.601 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:27:26.601 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.688649+0000", 2024-09-11T08:27:26.601 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:26.601 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.601 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.601 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:27:26.601 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.688649+0000", 2024-09-11T08:27:26.601 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:26.601 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.601 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.602 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:27:26.602 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.688649+0000", 2024-09-11T08:27:26.602 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:27:26.602 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.602 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.602 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:27:26.602 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.688653+0000", 2024-09-11T08:27:26.602 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5290000000000002e-06 2024-09-11T08:27:26.602 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.602 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.602 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:27:26.602 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.688663+0000", 2024-09-11T08:27:26.602 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0149000000000001e-05 2024-09-11T08:27:26.602 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.603 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.603 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:27:26.603 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.688669+0000", 2024-09-11T08:27:26.603 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.711e-06 2024-09-11T08:27:26.603 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.603 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.603 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:26.603 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.688717+0000", 2024-09-11T08:27:26.603 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8384999999999998e-05 2024-09-11T08:27:26.603 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.603 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.603 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:27:26.603 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.688726+0000", 2024-09-11T08:27:26.604 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3000000000000002e-06 2024-09-11T08:27:26.604 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.604 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.604 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:26.604 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:09.602676+0000", 2024-09-11T08:27:26.604 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.91395006300000003 2024-09-11T08:27:26.604 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.604 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.604 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for degraded object", 2024-09-11T08:27:26.604 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:09.602726+0000", 2024-09-11T08:27:26.604 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0015000000000001e-05 2024-09-11T08:27:26.604 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.604 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.604 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:26.605 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:09.607945+0000", 2024-09-11T08:27:26.605 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0052189690000000004 2024-09-11T08:27:26.605 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.605 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.605 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:27:26.605 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:09.608013+0000", 2024-09-11T08:27:26.605 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7896000000000002e-05 2024-09-11T08:27:26.605 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.605 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.605 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:27:26.605 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:09.613864+0000", 2024-09-11T08:27:26.605 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0058509800000000004 2024-09-11T08:27:26.605 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:26.605 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:27:26.606 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:26.606 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.606 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.606 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15636.0:21390 10.1 10:8a05f28d:::benchmark_data_smithi150_158067_object21389:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-09-11T08:27:26.606 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:22:07.766472+0000", 2024-09-11T08:27:26.606 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 318.76958767600001, 2024-09-11T08:27:26.606 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102182746, 2024-09-11T08:27:26.606 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:27:26.606 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:27:26.606 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:27:26.606 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15636", 2024-09-11T08:27:26.607 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:27:26.607 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 21390 2024-09-11T08:27:26.607 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.607 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:27:26.607 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.607 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:27:26.607 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:07.766472+0000", 2024-09-11T08:27:26.607 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:26.607 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.607 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.607 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:27:26.607 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:07.766472+0000", 2024-09-11T08:27:26.607 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:26.607 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.608 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.608 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:27:26.608 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:07.766470+0000", 2024-09-11T08:27:26.608 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999981 2024-09-11T08:27:26.608 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.608 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.608 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:27:26.608 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:07.766509+0000", 2024-09-11T08:27:26.608 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8566000000000003e-05 2024-09-11T08:27:26.608 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.608 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.608 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:27:26.608 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:07.766527+0000", 2024-09-11T08:27:26.608 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7864999999999999e-05 2024-09-11T08:27:26.609 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.609 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.609 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:27:26.609 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:07.766533+0000", 2024-09-11T08:27:26.609 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6749999999999996e-06 2024-09-11T08:27:26.609 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.609 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.609 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:26.609 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:07.766618+0000", 2024-09-11T08:27:26.609 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4293000000000001e-05 2024-09-11T08:27:26.609 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.609 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.609 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:27:26.609 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:07.766734+0000", 2024-09-11T08:27:26.610 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011651599999999999 2024-09-11T08:27:26.610 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.610 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.610 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,7", 2024-09-11T08:27:26.610 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:07.766858+0000", 2024-09-11T08:27:26.610 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012348700000000001 2024-09-11T08:27:26.610 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.610 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.610 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:26.610 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:07.768351+0000", 2024-09-11T08:27:26.610 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001493734 2024-09-11T08:27:26.610 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.610 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.610 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:27:26.611 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:07.769481+0000", 2024-09-11T08:27:26.611 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001129187 2024-09-11T08:27:26.611 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.611 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.611 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:26.611 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:07.868571+0000", 2024-09-11T08:27:26.611 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.099090618000000005 2024-09-11T08:27:26.611 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.611 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.611 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:27:26.611 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:07.868618+0000", 2024-09-11T08:27:26.611 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6535000000000001e-05 2024-09-11T08:27:26.611 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.611 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.612 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:27:26.612 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:07.868655+0000", 2024-09-11T08:27:26.612 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6987999999999999e-05 2024-09-11T08:27:26.612 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:26.612 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:27:26.612 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:26.613 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.613 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.613 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15636.0:22387 10.1 10:8cd0ab0d:::benchmark_data_smithi150_158067_object22386:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-09-11T08:27:26.613 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:22:08.343799+0000", 2024-09-11T08:27:26.613 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 318.192260987, 2024-09-11T08:27:26.614 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.12519976299999999, 2024-09-11T08:27:26.614 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:27:26.614 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:27:26.614 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:27:26.614 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15636", 2024-09-11T08:27:26.614 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:27:26.614 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 22387 2024-09-11T08:27:26.614 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.614 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:27:26.614 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.614 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:27:26.614 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:08.343799+0000", 2024-09-11T08:27:26.614 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:26.615 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.615 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.615 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:27:26.615 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:08.343799+0000", 2024-09-11T08:27:26.615 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:26.615 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.615 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.615 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:27:26.615 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:08.343797+0000", 2024-09-11T08:27:26.615 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:27:26.615 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.615 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.615 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:27:26.615 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:08.343849+0000", 2024-09-11T08:27:26.616 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1765999999999999e-05 2024-09-11T08:27:26.616 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.616 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.616 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:27:26.616 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:08.343862+0000", 2024-09-11T08:27:26.616 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3448000000000001e-05 2024-09-11T08:27:26.616 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.616 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.616 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:27:26.616 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:08.343867+0000", 2024-09-11T08:27:26.616 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3220000000000003e-06 2024-09-11T08:27:26.616 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.616 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.617 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:26.617 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:08.343905+0000", 2024-09-11T08:27:26.617 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8293000000000001e-05 2024-09-11T08:27:26.617 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.617 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.617 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:27:26.617 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:08.343998+0000", 2024-09-11T08:27:26.617 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.2517999999999997e-05 2024-09-11T08:27:26.617 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.617 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.617 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,7", 2024-09-11T08:27:26.617 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:08.344107+0000", 2024-09-11T08:27:26.617 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010926500000000001 2024-09-11T08:27:26.617 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.618 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.618 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:27:26.618 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:08.344926+0000", 2024-09-11T08:27:26.618 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00081900899999999999 2024-09-11T08:27:26.618 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.618 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.618 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:26.618 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:08.345305+0000", 2024-09-11T08:27:26.618 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00037882900000000001 2024-09-11T08:27:26.618 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.618 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.618 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:26.618 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:08.468919+0000", 2024-09-11T08:27:26.618 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.123614497 2024-09-11T08:27:26.619 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.619 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.619 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:27:26.619 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:08.468962+0000", 2024-09-11T08:27:26.619 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3353000000000003e-05 2024-09-11T08:27:26.619 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.619 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.619 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:27:26.619 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:08.468998+0000", 2024-09-11T08:27:26.619 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5966000000000001e-05 2024-09-11T08:27:26.619 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:26.619 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:27:26.619 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:26.620 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.620 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.620 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15636.0:23392 10.e 10:7f41910c:::benchmark_data_smithi150_158067_object23391:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-09-11T08:27:26.620 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:22:08.833528+0000", 2024-09-11T08:27:26.621 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 317.70253216600003, 2024-09-11T08:27:26.621 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10022840399999999, 2024-09-11T08:27:26.621 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:27:26.621 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:27:26.621 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:27:26.621 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15636", 2024-09-11T08:27:26.621 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:27:26.621 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 23392 2024-09-11T08:27:26.621 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.621 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:27:26.621 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.621 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:27:26.621 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:08.833528+0000", 2024-09-11T08:27:26.622 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:26.622 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.622 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.622 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:27:26.622 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:08.833528+0000", 2024-09-11T08:27:26.622 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:26.622 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.622 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.622 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:27:26.622 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:08.833527+0000", 2024-09-11T08:27:26.622 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-09-11T08:27:26.624 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.625 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.625 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:27:26.625 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:08.833603+0000", 2024-09-11T08:27:26.625 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6289000000000001e-05 2024-09-11T08:27:26.625 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.625 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.625 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:27:26.625 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:08.833610+0000", 2024-09-11T08:27:26.625 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1049999999999997e-06 2024-09-11T08:27:26.625 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.625 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.625 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:27:26.625 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:08.833615+0000", 2024-09-11T08:27:26.625 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.34e-06 2024-09-11T08:27:26.626 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.626 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.626 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:26.626 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:08.833651+0000", 2024-09-11T08:27:26.626 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6371000000000001e-05 2024-09-11T08:27:26.626 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.626 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.626 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:27:26.626 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:08.833752+0000", 2024-09-11T08:27:26.626 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010101 2024-09-11T08:27:26.626 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.626 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.626 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2,6", 2024-09-11T08:27:26.626 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:08.833867+0000", 2024-09-11T08:27:26.627 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011496 2024-09-11T08:27:26.627 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.627 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.627 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:27:26.627 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:08.835245+0000", 2024-09-11T08:27:26.627 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0013779809999999999 2024-09-11T08:27:26.627 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.627 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.627 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:26.627 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:08.838029+0000", 2024-09-11T08:27:26.627 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002784197 2024-09-11T08:27:26.627 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.627 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.627 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:26.628 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:08.933675+0000", 2024-09-11T08:27:26.628 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.095646118000000002 2024-09-11T08:27:26.628 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.628 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.628 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:27:26.628 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:08.933717+0000", 2024-09-11T08:27:26.628 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2089000000000003e-05 2024-09-11T08:27:26.628 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.628 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.628 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:27:26.628 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:08.933756+0000", 2024-09-11T08:27:26.628 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8563000000000003e-05 2024-09-11T08:27:26.628 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:26.628 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:27:26.629 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:26.629 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.629 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.629 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15636.0:27637 10.1 10:888abffb:::benchmark_data_smithi150_158067_object27636:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-09-11T08:27:26.629 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:22:10.769541+0000", 2024-09-11T08:27:26.629 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 315.76651889800002, 2024-09-11T08:27:26.629 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.12866815000000001, 2024-09-11T08:27:26.629 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:27:26.629 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:27:26.629 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:27:26.629 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15636", 2024-09-11T08:27:26.629 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:27:26.629 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 27637 2024-09-11T08:27:26.629 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.630 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:27:26.630 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.630 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:27:26.630 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:10.769541+0000", 2024-09-11T08:27:26.630 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:26.630 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.630 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.630 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:27:26.630 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:10.769541+0000", 2024-09-11T08:27:26.630 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:26.630 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.630 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.630 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:27:26.631 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:10.769539+0000", 2024-09-11T08:27:26.631 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:27:26.631 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.631 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.631 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:27:26.631 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:10.769573+0000", 2024-09-11T08:27:26.631 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4088999999999998e-05 2024-09-11T08:27:26.631 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.631 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.631 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:27:26.631 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:10.769584+0000", 2024-09-11T08:27:26.632 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0298e-05 2024-09-11T08:27:26.632 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.632 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.632 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:27:26.632 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:10.769589+0000", 2024-09-11T08:27:26.632 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5990000000000001e-06 2024-09-11T08:27:26.632 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.632 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.632 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:26.632 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:10.769710+0000", 2024-09-11T08:27:26.632 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012084899999999999 2024-09-11T08:27:26.632 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.632 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.633 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:27:26.633 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:10.769823+0000", 2024-09-11T08:27:26.633 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000112927 2024-09-11T08:27:26.633 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.633 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.633 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,7", 2024-09-11T08:27:26.633 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:10.769949+0000", 2024-09-11T08:27:26.633 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000126196 2024-09-11T08:27:26.633 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.633 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.633 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:27:26.633 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:10.772208+0000", 2024-09-11T08:27:26.633 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0022590869999999999 2024-09-11T08:27:26.633 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.634 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.634 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:26.634 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:10.772978+0000", 2024-09-11T08:27:26.634 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000769851 2024-09-11T08:27:26.634 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.634 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.634 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:26.634 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:10.898064+0000", 2024-09-11T08:27:26.634 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.12508587900000001 2024-09-11T08:27:26.634 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.634 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.634 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:27:26.634 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:10.898139+0000", 2024-09-11T08:27:26.634 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5302000000000005e-05 2024-09-11T08:27:26.635 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.635 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.635 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:27:26.635 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:10.898209+0000", 2024-09-11T08:27:26.635 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9457999999999998e-05 2024-09-11T08:27:26.635 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:26.635 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:27:26.635 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:26.635 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.635 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.635 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15636.0:27641 10.1 10:8c9342d6:::benchmark_data_smithi150_158067_object27640:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-09-11T08:27:26.635 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:22:10.770988+0000", 2024-09-11T08:27:26.635 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 315.76507199000002, 2024-09-11T08:27:26.635 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.127358267, 2024-09-11T08:27:26.636 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:27:26.636 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:27:26.636 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:27:26.636 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15636", 2024-09-11T08:27:26.636 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:27:26.636 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 27641 2024-09-11T08:27:26.636 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.636 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:27:26.637 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.637 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:27:26.637 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:10.770988+0000", 2024-09-11T08:27:26.637 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:26.637 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.637 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.637 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:27:26.637 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:10.770988+0000", 2024-09-11T08:27:26.637 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:26.637 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.637 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.637 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:27:26.637 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:10.770987+0000", 2024-09-11T08:27:26.637 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:27:26.638 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.638 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.638 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:27:26.638 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:10.771021+0000", 2024-09-11T08:27:26.638 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4897999999999997e-05 2024-09-11T08:27:26.638 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.638 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.638 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:27:26.638 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:10.771031+0000", 2024-09-11T08:27:26.638 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.1230000000000006e-06 2024-09-11T08:27:26.638 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.638 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.638 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:27:26.638 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:10.771036+0000", 2024-09-11T08:27:26.639 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5840000000000002e-06 2024-09-11T08:27:26.639 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.639 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.639 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:26.639 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:10.771149+0000", 2024-09-11T08:27:26.639 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000112922 2024-09-11T08:27:26.639 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.639 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.639 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:27:26.639 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:10.771269+0000", 2024-09-11T08:27:26.639 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011991 2024-09-11T08:27:26.639 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.639 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.639 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,7", 2024-09-11T08:27:26.640 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:10.771426+0000", 2024-09-11T08:27:26.640 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000157188 2024-09-11T08:27:26.640 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.640 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.640 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:27:26.640 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:10.772436+0000", 2024-09-11T08:27:26.640 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0010093490000000001 2024-09-11T08:27:26.640 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.640 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.640 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:26.640 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:10.773108+0000", 2024-09-11T08:27:26.640 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00067258599999999997 2024-09-11T08:27:26.640 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.641 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.641 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:26.641 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:10.898260+0000", 2024-09-11T08:27:26.641 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.125151398 2024-09-11T08:27:26.641 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.641 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.641 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:27:26.641 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:10.898310+0000", 2024-09-11T08:27:26.641 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0943000000000003e-05 2024-09-11T08:27:26.641 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.641 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.641 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:27:26.641 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:10.898346+0000", 2024-09-11T08:27:26.641 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5475999999999997e-05 2024-09-11T08:27:26.642 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:26.642 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:27:26.642 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:26.642 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.642 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.642 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15636.0:27644 10.1 10:8b4861d2:::benchmark_data_smithi150_158067_object27643:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-09-11T08:27:26.642 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:22:10.773563+0000", 2024-09-11T08:27:26.642 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 315.76249637299998, 2024-09-11T08:27:26.642 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.12674559699999999, 2024-09-11T08:27:26.642 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:27:26.642 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:27:26.642 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:27:26.642 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15636", 2024-09-11T08:27:26.642 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:27:26.643 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 27644 2024-09-11T08:27:26.643 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.643 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:27:26.643 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.643 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:27:26.643 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:10.773563+0000", 2024-09-11T08:27:26.643 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:26.643 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.643 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.643 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:27:26.643 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:10.773563+0000", 2024-09-11T08:27:26.643 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:26.643 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.643 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.644 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:27:26.644 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:10.773562+0000", 2024-09-11T08:27:26.644 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:27:26.644 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.644 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.644 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:27:26.644 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:10.773603+0000", 2024-09-11T08:27:26.644 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0836e-05 2024-09-11T08:27:26.644 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.644 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.644 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:27:26.644 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:10.773613+0000", 2024-09-11T08:27:26.644 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0192e-05 2024-09-11T08:27:26.645 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.645 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.645 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:27:26.645 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:10.773617+0000", 2024-09-11T08:27:26.645 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2590000000000002e-06 2024-09-11T08:27:26.645 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.645 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.645 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:26.645 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:10.773650+0000", 2024-09-11T08:27:26.645 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2818000000000003e-05 2024-09-11T08:27:26.645 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.645 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.645 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:27:26.646 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:10.773735+0000", 2024-09-11T08:27:26.646 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.496e-05 2024-09-11T08:27:26.646 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.646 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.646 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,7", 2024-09-11T08:27:26.646 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:10.773818+0000", 2024-09-11T08:27:26.646 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3307999999999996e-05 2024-09-11T08:27:26.646 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.646 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.646 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:27:26.646 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:10.774884+0000", 2024-09-11T08:27:26.646 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001065629 2024-09-11T08:27:26.646 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.647 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.647 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:26.647 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:10.774999+0000", 2024-09-11T08:27:26.647 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000115358 2024-09-11T08:27:26.647 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.647 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.647 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:26.647 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:10.900233+0000", 2024-09-11T08:27:26.647 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.12523357299999999 2024-09-11T08:27:26.647 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.647 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.647 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:27:26.647 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:10.900276+0000", 2024-09-11T08:27:26.648 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2806000000000003e-05 2024-09-11T08:27:26.648 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.648 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.648 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:27:26.648 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:22:10.900309+0000", 2024-09-11T08:27:26.648 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3246e-05 2024-09-11T08:27:26.648 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:26.648 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:27:26.648 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:26.648 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.648 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.648 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.25799.0:19088 11.e 11:7d4760ee:::benchmark_data_smithi150_187410_object19087:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e388)", 2024-09-11T08:27:26.648 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:26:46.728851+0000", 2024-09-11T08:27:26.648 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 39.807208267999997, 2024-09-11T08:27:26.649 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19838055099999999, 2024-09-11T08:27:26.649 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:27:26.649 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:27:26.649 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:27:26.649 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.25799", 2024-09-11T08:27:26.649 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/1403604046", 2024-09-11T08:27:26.649 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 19088 2024-09-11T08:27:26.649 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.649 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:27:26.649 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.649 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:27:26.649 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:26:46.728851+0000", 2024-09-11T08:27:26.649 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:26.650 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.650 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.650 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:27:26.650 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:26:46.728851+0000", 2024-09-11T08:27:26.650 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:26.650 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.650 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.650 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:27:26.650 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:26:46.728850+0000", 2024-09-11T08:27:26.650 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:27:26.650 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.650 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.650 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:27:26.650 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:26:46.728898+0000", 2024-09-11T08:27:26.651 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7827000000000001e-05 2024-09-11T08:27:26.651 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.651 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.651 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:27:26.651 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:26:46.728912+0000", 2024-09-11T08:27:26.651 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4681999999999999e-05 2024-09-11T08:27:26.651 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.651 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.651 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:27:26.651 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:26:46.728918+0000", 2024-09-11T08:27:26.651 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8259999999999998e-06 2024-09-11T08:27:26.651 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.651 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.651 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:26.652 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:26:46.728963+0000", 2024-09-11T08:27:26.652 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4842000000000002e-05 2024-09-11T08:27:26.652 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.652 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.652 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:27:26.652 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:26:46.729049+0000", 2024-09-11T08:27:26.652 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6475000000000001e-05 2024-09-11T08:27:26.652 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.652 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.652 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,7", 2024-09-11T08:27:26.652 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:26:46.729135+0000", 2024-09-11T08:27:26.652 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6063000000000006e-05 2024-09-11T08:27:26.652 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.652 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.653 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:27:26.653 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:26:46.729971+0000", 2024-09-11T08:27:26.653 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00083547600000000004 2024-09-11T08:27:26.653 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.653 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.653 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:26.653 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:26:46.730110+0000", 2024-09-11T08:27:26.653 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013891700000000001 2024-09-11T08:27:26.653 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.653 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.653 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:26.653 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:26:46.927032+0000", 2024-09-11T08:27:26.653 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19692167199999999 2024-09-11T08:27:26.653 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.654 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.654 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:27:26.654 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:26:46.927184+0000", 2024-09-11T08:27:26.654 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000152464 2024-09-11T08:27:26.654 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.654 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.654 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:27:26.654 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:26:46.927232+0000", 2024-09-11T08:27:26.654 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7938000000000001e-05 2024-09-11T08:27:26.654 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:26.654 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:27:26.654 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:26.654 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.654 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.654 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.25799.0:74347 11.e 11:7992709a:::benchmark_data_smithi150_187410_object74346:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-09-11T08:27:26.655 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:27:22.096549+0000", 2024-09-11T08:27:26.655 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.4395104009999997, 2024-09-11T08:27:26.655 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.33292110000000003, 2024-09-11T08:27:26.655 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:27:26.655 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:27:26.655 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:27:26.655 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.25799", 2024-09-11T08:27:26.655 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/1403604046", 2024-09-11T08:27:26.655 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 74347 2024-09-11T08:27:26.655 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.655 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:27:26.655 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.655 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:27:26.656 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.096549+0000", 2024-09-11T08:27:26.656 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:26.656 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.656 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.656 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:27:26.656 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.096549+0000", 2024-09-11T08:27:26.656 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:26.656 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.656 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.656 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:27:26.656 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.096537+0000", 2024-09-11T08:27:26.656 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999881 2024-09-11T08:27:26.656 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.656 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.657 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:27:26.657 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.096581+0000", 2024-09-11T08:27:26.657 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3813000000000002e-05 2024-09-11T08:27:26.657 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.657 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.657 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:27:26.657 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.096598+0000", 2024-09-11T08:27:26.657 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6629999999999998e-05 2024-09-11T08:27:26.657 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.658 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.658 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:27:26.658 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.096610+0000", 2024-09-11T08:27:26.658 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1956e-05 2024-09-11T08:27:26.658 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.658 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.658 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:26.658 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.096665+0000", 2024-09-11T08:27:26.658 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5268e-05 2024-09-11T08:27:26.658 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.658 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.658 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:27:26.658 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.096808+0000", 2024-09-11T08:27:26.658 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014249900000000001 2024-09-11T08:27:26.659 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.659 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.659 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,7", 2024-09-11T08:27:26.659 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.096927+0000", 2024-09-11T08:27:26.659 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000119585 2024-09-11T08:27:26.659 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.659 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.659 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:26.659 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.426390+0000", 2024-09-11T08:27:26.659 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.329463014 2024-09-11T08:27:26.659 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.659 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.659 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:27:26.660 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.426397+0000", 2024-09-11T08:27:26.660 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.579e-06 2024-09-11T08:27:26.660 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.660 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.660 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:26.660 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.429383+0000", 2024-09-11T08:27:26.660 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002985765 2024-09-11T08:27:26.660 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.660 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.660 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:27:26.660 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.429427+0000", 2024-09-11T08:27:26.660 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4100999999999997e-05 2024-09-11T08:27:26.660 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.661 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.661 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:27:26.661 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.429470+0000", 2024-09-11T08:27:26.661 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.371e-05 2024-09-11T08:27:26.661 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:26.661 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:27:26.661 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:26.661 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.661 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.661 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.25799.0:74339 11.7 e398/382 11:ebfe37ab:::benchmark_data_smithi150_187410_object74338:head v 398'4647, mlcod=398'4645)", 2024-09-11T08:27:26.661 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:27:22.098217+0000", 2024-09-11T08:27:26.661 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.4378430460000002, 2024-09-11T08:27:26.662 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.33765329500000002, 2024-09-11T08:27:26.662 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:27:26.662 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:27:26.662 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:27:26.662 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.662 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:27:26.662 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.098217+0000", 2024-09-11T08:27:26.662 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:26.662 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.662 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.662 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:27:26.662 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.098217+0000", 2024-09-11T08:27:26.662 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:26.662 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.662 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.663 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:27:26.663 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.098216+0000", 2024-09-11T08:27:26.663 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:27:26.663 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.663 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.663 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:27:26.663 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.098257+0000", 2024-09-11T08:27:26.663 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0911999999999997e-05 2024-09-11T08:27:26.663 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.663 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.663 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:27:26.663 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.420483+0000", 2024-09-11T08:27:26.663 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.32222661499999999 2024-09-11T08:27:26.664 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.664 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.664 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:27:26.664 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.420493+0000", 2024-09-11T08:27:26.664 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0243e-05 2024-09-11T08:27:26.664 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.664 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.664 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:26.664 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.420543+0000", 2024-09-11T08:27:26.664 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9296999999999999e-05 2024-09-11T08:27:26.664 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.664 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.664 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:27:26.664 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.420562+0000", 2024-09-11T08:27:26.665 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9570000000000001e-05 2024-09-11T08:27:26.665 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.665 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.665 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:27:26.665 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.435681+0000", 2024-09-11T08:27:26.665 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.015118424 2024-09-11T08:27:26.665 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:26.665 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:26.665 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:27:26.665 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.435870+0000", 2024-09-11T08:27:26.665 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00018918299999999999 2024-09-11T08:27:26.665 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:26.665 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:27:26.666 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:26.666 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:26.666 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:27:26.666 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-11T08:27:26.667 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 46 16 79234 79218 107.583 77.625 0.00823474 0.00929031 2024-09-11T08:27:26.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:26.668 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:26.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:26.769 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:26.813 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:26.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:26.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:26.915 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:27:26.953 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:26.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:26.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:27:27.055 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:27.096 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:27.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:27.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:27.198 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:27.240 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:27.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:27.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:27.342 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:27.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:27.379 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:27.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:27.394 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:27 smithi150 ceph-mon[39612]: osdmap e401: 8 total, 8 up, 8 in 2024-09-11T08:27:27.394 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:27 smithi150 ceph-mon[29276]: osdmap e401: 8 total, 8 up, 8 in 2024-09-11T08:27:27.480 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:27.514 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:27.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:27.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:27.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:27 smithi181 ceph-mon[32701]: osdmap e401: 8 total, 8 up, 8 in 2024-09-11T08:27:27.616 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:27:27.632 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 47 16 80449 80433 106.91 75.9375 0.026215 0.00934864 2024-09-11T08:27:27.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:27.647 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:27.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:27:27.748 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:27.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:27.789 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:27.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:27.842 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-11T08:27:27.890 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:27.928 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:27.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:27.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:28.030 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:28.084 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-11T08:27:28.084 INFO:teuthology.orchestra.run.smithi150.stdout: "success": "" 2024-09-11T08:27:28.084 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-11T08:27:28.104 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:28.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:28.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:28.182 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:28 smithi150 ceph-mon[29276]: pgmap v911: 148 pgs: 148 active+clean; 4.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 54 MiB/s wr, 863 op/s 2024-09-11T08:27:28.182 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:28 smithi150 ceph-mon[39612]: pgmap v911: 148 pgs: 148 active+clean; 4.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 54 MiB/s wr, 863 op/s 2024-09-11T08:27:28.207 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:27:28.218 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:27:28.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:28.249 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:28.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:27:28.350 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:27:28.367 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:27:28.368 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:27:28.380 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:28.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:28.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:27:28.398 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:27:28.398 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:27:28.430 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:27:28.430 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:27:28.466 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:27:28.467 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:27:28.481 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:27:28.499 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:27:28.499 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:27:28.512 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:28.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:28.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:27:28.529 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:27:28.529 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:27:28.561 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:27:28.561 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:27:28.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:28 smithi181 ceph-mon[32701]: pgmap v911: 148 pgs: 148 active+clean; 4.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 54 MiB/s wr, 863 op/s 2024-09-11T08:27:28.593 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:27:28.593 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:27:28.615 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:28.632 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 48 16 81608 81592 106.19 72.4375 0.0203353 0.00941219 2024-09-11T08:27:28.663 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:28.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:28.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:28.738 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:27:28 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:27:28] "GET /metrics HTTP/1.1" 200 224627 "" "Prometheus/2.43.0" 2024-09-11T08:27:28.765 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:28.779 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T08:27:28.799 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:28.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:28.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:28.901 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:28.923 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:28.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:28.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:29.025 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:29.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:29.074 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:29.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:29.111 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:29 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/4217778486' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:27:29.175 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:27:29.209 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:29.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:29.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:27:29.311 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:29.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:29 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/4217778486' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:27:29.323 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:27:28 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:27:29.348 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:29.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:29.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:29.421 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:29 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/4217778486' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:27:29.450 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:29.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:29.479 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:29.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:29.580 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:29.621 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:29.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:29.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:29.633 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 49 15 82768 82753 105.504 72.5625 0.0156715 0.00947363 2024-09-11T08:27:29.723 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:27:29.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:29.775 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:29.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:27:29.876 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:27:29.920 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:29.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:29.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:27:30.022 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:30.052 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:30.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:30.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:30.154 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:30.164 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:30 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:27:30.164 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:30 smithi181 ceph-mon[32701]: pgmap v912: 148 pgs: 148 active+clean; 4.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.6 KiB/s rd, 105 MiB/s wr, 1.69k op/s 2024-09-11T08:27:30.186 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:30.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:30.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:30.287 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:30.304 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-11T08:27:30.304 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:27:30.323 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:30.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:30.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:30.363 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:27:30.425 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:30.468 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:30.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:30.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:30.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:30 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:27:30.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:30 smithi150 ceph-mon[29276]: pgmap v912: 148 pgs: 148 active+clean; 4.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.6 KiB/s rd, 105 MiB/s wr, 1.69k op/s 2024-09-11T08:27:30.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:30 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:27:30.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:30 smithi150 ceph-mon[39612]: pgmap v912: 148 pgs: 148 active+clean; 4.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.6 KiB/s rd, 105 MiB/s wr, 1.69k op/s 2024-09-11T08:27:30.570 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:27:30.598 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:30.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:30.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:27:30.633 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 50 16 83963 83947 104.887 74.625 0.00524022 0.00952893 2024-09-11T08:27:30.699 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:27:30.731 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:30.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:30.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:27:30.833 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:27:30.868 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:30.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:30.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:27:30.970 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:31.020 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:31.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:31.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:31.042 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-11T08:27:31.123 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:27:31.162 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:31.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:31.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:27:31.264 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:31.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:31.297 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:31.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:31.398 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:27:31.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:31.441 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:31.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:27:31.479 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:27:31.543 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:31.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:31.587 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:31.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:31.633 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 51 16 85189 85173 104.333 76.625 0.0136986 0.00957949 2024-09-11T08:27:31.689 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:31.727 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:31.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:31.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:31.829 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:27:31.876 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:31.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:31.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:27:31.883 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:31 smithi181 ceph-mon[32701]: osdmap e402: 8 total, 8 up, 8 in 2024-09-11T08:27:31.883 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:31 smithi181 ceph-mon[32701]: pgmap v914: 148 pgs: 148 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 843 B/s rd, 60 MiB/s wr, 961 op/s 2024-09-11T08:27:31.978 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:27:31.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:31 smithi150 ceph-mon[39612]: osdmap e402: 8 total, 8 up, 8 in 2024-09-11T08:27:31.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:31 smithi150 ceph-mon[39612]: pgmap v914: 148 pgs: 148 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 843 B/s rd, 60 MiB/s wr, 961 op/s 2024-09-11T08:27:31.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:31 smithi150 ceph-mon[29276]: osdmap e402: 8 total, 8 up, 8 in 2024-09-11T08:27:31.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:31 smithi150 ceph-mon[29276]: pgmap v914: 148 pgs: 148 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 843 B/s rd, 60 MiB/s wr, 961 op/s 2024-09-11T08:27:32.019 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:32.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:32.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:27:32.121 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:32.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:32.176 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:32.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:32.265 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T08:27:32.277 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:32.301 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:32.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:32.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:32.403 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:32.454 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:32.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:32.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:32.555 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:32.582 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:32.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:32.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:32.634 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 52 16 86235 86219 103.582 65.375 0.0208383 0.00964821 2024-09-11T08:27:32.683 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:32.728 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:27:32.738 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:32.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:32.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:32.836 INFO:teuthology.orchestra.run.smithi150.stderr:nodeep-scrub is set 2024-09-11T08:27:32.840 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:32.874 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:32.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:32.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:32.976 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:33.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:33.011 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:33.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:33.034 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:32 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/4203953232' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:27:33.034 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:32 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:27:33.034 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:32 smithi181 ceph-mon[32701]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:27:33.112 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:33.147 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:33.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:33.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:33.237 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:32 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/4203953232' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:27:33.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:32 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:27:33.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:32 smithi150 ceph-mon[29276]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:27:33.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:32 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/4203953232' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:27:33.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:32 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:27:33.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:32 smithi150 ceph-mon[39612]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:27:33.249 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:27:33.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:33.272 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:33.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:27:33.373 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:33.395 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:33.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:33.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:33.496 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:33.517 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-11T08:27:33.517 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:27:33.522 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:33.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:33.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:33.624 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:33.634 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 53 16 87268 87252 102.846 64.5625 0.00895318 0.00971882 2024-09-11T08:27:33.659 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:33.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:33.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:33.760 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:33.789 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:33.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:33.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:33.891 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:33.929 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:33.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:33.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:34.031 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:34.071 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:34.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:34.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:34.173 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:27:34.213 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:34.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:34.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:27:34.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:33 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:27:34.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:33 smithi150 ceph-mon[29276]: osdmap e403: 8 total, 8 up, 8 in 2024-09-11T08:27:34.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:33 smithi150 ceph-mon[29276]: pgmap v916: 148 pgs: 148 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 725 B/s rd, 55 MiB/s wr, 875 op/s 2024-09-11T08:27:34.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:33 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:27:34.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:33 smithi150 ceph-mon[39612]: osdmap e403: 8 total, 8 up, 8 in 2024-09-11T08:27:34.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:33 smithi150 ceph-mon[39612]: pgmap v916: 148 pgs: 148 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 725 B/s rd, 55 MiB/s wr, 875 op/s 2024-09-11T08:27:34.285 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:33 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:27:34.285 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:33 smithi181 ceph-mon[32701]: osdmap e403: 8 total, 8 up, 8 in 2024-09-11T08:27:34.285 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:33 smithi181 ceph-mon[32701]: pgmap v916: 148 pgs: 148 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 725 B/s rd, 55 MiB/s wr, 875 op/s 2024-09-11T08:27:34.314 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:27:34.335 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:34.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:34.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:27:34.437 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:27:34.458 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-09-11T08:27:34.459 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T08:27:34.459 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-11T08:27:34.459 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-11T08:27:34.468 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:34.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:34.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:27:34.570 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:34.601 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-11T08:27:34.614 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:34.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:34.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:34.635 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 54 16 88326 88310 102.166 66.125 0.00815325 0.00978255 2024-09-11T08:27:34.716 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:27:34.747 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:34.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:34.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:27:34.849 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:27:34.898 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:34.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:34.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:27:34.966 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:27:35.000 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:35.020 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:35.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:35.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:35.122 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:35.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:35.155 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:35.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:35.256 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:35.300 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:35.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:35.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:35.331 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:27:35 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:27:35] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:27:35.402 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:35.432 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:35.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:35.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:35.534 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:35.569 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:35.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:35.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:35.634 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 55 16 89401 89385 101.53 67.1875 0.0149475 0.00984434 2024-09-11T08:27:35.672 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:27:35.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:35.701 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:35.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:27:35.802 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:35.830 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:35.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:35.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:35.932 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:35.962 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:35.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:35.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:36.064 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:36.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:36.092 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:36.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:36.137 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:35 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:27:36.137 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:35 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:27:36.137 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:35 smithi181 ceph-mon[32701]: osdmap e404: 8 total, 8 up, 8 in 2024-09-11T08:27:36.137 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:35 smithi181 ceph-mon[32701]: pgmap v918: 148 pgs: 148 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-09-11T08:27:36.193 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:36.216 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:36.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:36.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:36.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:35 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:27:36.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:35 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:27:36.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:35 smithi150 ceph-mon[29276]: osdmap e404: 8 total, 8 up, 8 in 2024-09-11T08:27:36.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:35 smithi150 ceph-mon[29276]: pgmap v918: 148 pgs: 148 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-09-11T08:27:36.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:35 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:27:36.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:35 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:27:36.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:35 smithi150 ceph-mon[39612]: osdmap e404: 8 total, 8 up, 8 in 2024-09-11T08:27:36.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:35 smithi150 ceph-mon[39612]: pgmap v918: 148 pgs: 148 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-09-11T08:27:36.318 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:36.357 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:36.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:36.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:36.459 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:36.492 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:36.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:36.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:36.594 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:36.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:36.632 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:36.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:36.634 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 56 16 90440 90424 100.877 64.9375 0.013609 0.00990835 2024-09-11T08:27:36.733 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:27:36.764 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-11T08:27:36.764 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:27:36.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:36.775 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:36.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:27:36.876 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:27:36.910 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:36.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:36.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:27:37.012 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:37.039 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:37.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:37.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:37.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:36 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:27:37.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:36 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:27:37.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:36 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:27:37.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:36 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:27:37.141 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:27:37.174 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:37.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:37.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:27:37.216 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:36 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:27:37.216 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:36 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:27:37.216 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:36 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:27:37.216 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:36 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:27:37.217 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:36 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:27:37.217 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:36 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:27:37.217 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:36 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:27:37.217 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:36 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:27:37.277 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:27:37.319 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:37.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:37.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:27:37.421 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:37.464 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:37.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:37.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:37.472 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-11T08:27:37.566 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:37.617 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:37.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:37.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:37.636 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 57 16 91623 91607 100.401 73.9375 0.00941733 0.0099556 2024-09-11T08:27:37.719 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:37.762 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:37.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:37.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:37.864 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:37.866 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:27:37.906 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:37.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:37.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:38.008 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:27:38.050 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:38.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:38.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:27:38.141 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:37 smithi181 ceph-mon[32701]: osdmap e405: 8 total, 8 up, 8 in 2024-09-11T08:27:38.141 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:37 smithi181 ceph-mon[32701]: pgmap v920: 148 pgs: 148 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 60 MiB/s wr, 953 op/s 2024-09-11T08:27:38.152 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:38.158 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:37 smithi150 ceph-mon[29276]: osdmap e405: 8 total, 8 up, 8 in 2024-09-11T08:27:38.158 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:37 smithi150 ceph-mon[29276]: pgmap v920: 148 pgs: 148 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 60 MiB/s wr, 953 op/s 2024-09-11T08:27:38.161 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:37 smithi150 ceph-mon[39612]: osdmap e405: 8 total, 8 up, 8 in 2024-09-11T08:27:38.161 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:37 smithi150 ceph-mon[39612]: pgmap v920: 148 pgs: 148 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 60 MiB/s wr, 953 op/s 2024-09-11T08:27:38.169 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:27:38.205 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:38.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:38.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:38.307 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:38.343 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:38.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:38.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:38.445 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:38.481 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:38.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:38.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:38.583 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:27:38.625 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:38.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:38.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:27:38.636 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 58 16 92773 92757 99.9091 71.875 0.00952605 0.0100041 2024-09-11T08:27:38.727 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:38.763 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:38.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:38.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:38.865 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:38.887 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:27:38 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:27:38] "GET /metrics HTTP/1.1" 200 224635 "" "Prometheus/2.43.0" 2024-09-11T08:27:38.914 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:38.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:38.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:39.016 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:39.062 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:39.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:39.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:39.065 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T08:27:39.164 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:39.175 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:27:38 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:27:39.197 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:39.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:39.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:39.299 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:39.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:39.373 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:39.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:39.474 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:39.502 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:39.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:39.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:39.604 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:39.636 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 59 16 93856 93840 99.3631 67.6875 0.00388336 0.01006 2024-09-11T08:27:39.640 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:39.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:39.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:39.742 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:39.791 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:39.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:39.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:39.893 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:39.957 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:39.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:39.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:40.028 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-11T08:27:40.028 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:27:40.032 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:27:40.032 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:27:40.058 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:40.068 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:27:40.069 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:27:40.085 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:40.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:40.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:40.102 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:27:40.102 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:27:40.146 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:27:40.146 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:27:40.176 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:27:40.176 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:27:40.187 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:40.206 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:27:40.207 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:27:40.239 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:40.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:40.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:40.240 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:27:40.241 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:27:40.273 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:27:40.274 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:27:40.290 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:40 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:27:40.290 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:40 smithi150 ceph-mon[29276]: pgmap v921: 148 pgs: 148 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 22 KiB/s rd, 104 MiB/s wr, 1.69k op/s 2024-09-11T08:27:40.290 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:40 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3611781843' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:27:40.290 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:40 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:27:40.290 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:40 smithi150 ceph-mon[39612]: pgmap v921: 148 pgs: 148 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 22 KiB/s rd, 104 MiB/s wr, 1.69k op/s 2024-09-11T08:27:40.291 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:40 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3611781843' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:27:40.341 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:40.369 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:40.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:40.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:40.471 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:40.484 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:40 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:27:40.484 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:40 smithi181 ceph-mon[32701]: pgmap v921: 148 pgs: 148 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 22 KiB/s rd, 104 MiB/s wr, 1.69k op/s 2024-09-11T08:27:40.484 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:40 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3611781843' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:27:40.504 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:40.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:40.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:40.606 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:40.636 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:2024-09-11T08:27:40.637386+0000 min lat: 0.00120292 max lat: 0.344516 avg lat: 0.0101136 2024-09-11T08:27:40.637 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-11T08:27:40.637 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 60 16 94922 94906 98.8175 66.625 0.00953263 0.0101136 2024-09-11T08:27:40.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:40.646 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:40.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:40.747 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:40.775 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:40.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:40.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:40.877 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:40.924 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:40.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:40.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:40.934 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-11T08:27:41.025 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:27:41.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:41.061 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:41.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:27:41.102 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:27:41.102 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-11T08:27:41.102 INFO:teuthology.orchestra.run.smithi150.stdout: "epoch": 406, 2024-09-11T08:27:41.103 INFO:teuthology.orchestra.run.smithi150.stdout: "fsid": "c1bddff4-7013-11ef-bcea-c7b262605968", 2024-09-11T08:27:41.103 INFO:teuthology.orchestra.run.smithi150.stdout: "created": "2024-09-11T08:02:15.344618+0000", 2024-09-11T08:27:41.103 INFO:teuthology.orchestra.run.smithi150.stdout: "modified": "2024-09-11T08:27:40.730800+0000", 2024-09-11T08:27:41.103 INFO:teuthology.orchestra.run.smithi150.stdout: "last_up_change": "2024-09-11T08:24:27.333279+0000", 2024-09-11T08:27:41.103 INFO:teuthology.orchestra.run.smithi150.stdout: "last_in_change": "2024-09-11T08:19:28.121267+0000", 2024-09-11T08:27:41.103 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-11T08:27:41.103 INFO:teuthology.orchestra.run.smithi150.stdout: "flags_num": 5806080, 2024-09-11T08:27:41.103 INFO:teuthology.orchestra.run.smithi150.stdout: "flags_set": [ 2024-09-11T08:27:41.103 INFO:teuthology.orchestra.run.smithi150.stdout: "nodeep-scrub", 2024-09-11T08:27:41.103 INFO:teuthology.orchestra.run.smithi150.stdout: "noscrub", 2024-09-11T08:27:41.104 INFO:teuthology.orchestra.run.smithi150.stdout: "pglog_hardlimit", 2024-09-11T08:27:41.104 INFO:teuthology.orchestra.run.smithi150.stdout: "purged_snapdirs", 2024-09-11T08:27:41.104 INFO:teuthology.orchestra.run.smithi150.stdout: "recovery_deletes", 2024-09-11T08:27:41.104 INFO:teuthology.orchestra.run.smithi150.stdout: "sortbitwise" 2024-09-11T08:27:41.104 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-09-11T08:27:41.104 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_version": 20, 2024-09-11T08:27:41.104 INFO:teuthology.orchestra.run.smithi150.stdout: "full_ratio": 0.94999998807907104, 2024-09-11T08:27:41.104 INFO:teuthology.orchestra.run.smithi150.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-11T08:27:41.104 INFO:teuthology.orchestra.run.smithi150.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-11T08:27:41.104 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_snapshot": "", 2024-09-11T08:27:41.104 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_max": 11, 2024-09-11T08:27:41.104 INFO:teuthology.orchestra.run.smithi150.stdout: "max_osd": 8, 2024-09-11T08:27:41.104 INFO:teuthology.orchestra.run.smithi150.stdout: "require_min_compat_client": "luminous", 2024-09-11T08:27:41.104 INFO:teuthology.orchestra.run.smithi150.stdout: "min_compat_client": "luminous", 2024-09-11T08:27:41.105 INFO:teuthology.orchestra.run.smithi150.stdout: "require_osd_release": "quincy", 2024-09-11T08:27:41.105 INFO:teuthology.orchestra.run.smithi150.stdout: "pools": [ 2024-09-11T08:27:41.105 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.105 INFO:teuthology.orchestra.run.smithi150.stdout: "pool": 1, 2024-09-11T08:27:41.105 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_name": ".mgr", 2024-09-11T08:27:41.105 INFO:teuthology.orchestra.run.smithi150.stdout: "create_time": "2024-09-11T08:05:21.044729+0000", 2024-09-11T08:27:41.105 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-09-11T08:27:41.105 INFO:teuthology.orchestra.run.smithi150.stdout: "flags_names": "hashpspool", 2024-09-11T08:27:41.105 INFO:teuthology.orchestra.run.smithi150.stdout: "type": 1, 2024-09-11T08:27:41.105 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 3, 2024-09-11T08:27:41.105 INFO:teuthology.orchestra.run.smithi150.stdout: "min_size": 2, 2024-09-11T08:27:41.105 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_rule": 0, 2024-09-11T08:27:41.105 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_count": 0, 2024-09-11T08:27:41.106 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_target": 0, 2024-09-11T08:27:41.106 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T08:27:41.106 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T08:27:41.106 INFO:teuthology.orchestra.run.smithi150.stdout: "object_hash": 2, 2024-09-11T08:27:41.106 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_autoscale_mode": "off", 2024-09-11T08:27:41.106 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num": 1, 2024-09-11T08:27:41.106 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num": 1, 2024-09-11T08:27:41.106 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num_target": 1, 2024-09-11T08:27:41.106 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_target": 1, 2024-09-11T08:27:41.106 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_pending": 1, 2024-09-11T08:27:41.106 INFO:teuthology.orchestra.run.smithi150.stdout: "last_pg_merge_meta": { 2024-09-11T08:27:41.106 INFO:teuthology.orchestra.run.smithi150.stdout: "source_pgid": "0.0", 2024-09-11T08:27:41.106 INFO:teuthology.orchestra.run.smithi150.stdout: "ready_epoch": 0, 2024-09-11T08:27:41.107 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_started": 0, 2024-09-11T08:27:41.107 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_clean": 0, 2024-09-11T08:27:41.107 INFO:teuthology.orchestra.run.smithi150.stdout: "source_version": "0'0", 2024-09-11T08:27:41.107 INFO:teuthology.orchestra.run.smithi150.stdout: "target_version": "0'0" 2024-09-11T08:27:41.107 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.107 INFO:teuthology.orchestra.run.smithi150.stdout: "last_change": "255", 2024-09-11T08:27:41.107 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend": "0", 2024-09-11T08:27:41.107 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T08:27:41.108 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T08:27:41.108 INFO:teuthology.orchestra.run.smithi150.stdout: "auid": 0, 2024-09-11T08:27:41.108 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_mode": "selfmanaged", 2024-09-11T08:27:41.108 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_seq": 0, 2024-09-11T08:27:41.108 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_epoch": 0, 2024-09-11T08:27:41.108 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_snaps": [], 2024-09-11T08:27:41.108 INFO:teuthology.orchestra.run.smithi150.stdout: "removed_snaps": "[]", 2024-09-11T08:27:41.108 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_bytes": 0, 2024-09-11T08:27:41.108 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_objects": 0, 2024-09-11T08:27:41.108 INFO:teuthology.orchestra.run.smithi150.stdout: "tiers": [], 2024-09-11T08:27:41.108 INFO:teuthology.orchestra.run.smithi150.stdout: "tier_of": -1, 2024-09-11T08:27:41.108 INFO:teuthology.orchestra.run.smithi150.stdout: "read_tier": -1, 2024-09-11T08:27:41.108 INFO:teuthology.orchestra.run.smithi150.stdout: "write_tier": -1, 2024-09-11T08:27:41.108 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_mode": "none", 2024-09-11T08:27:41.109 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_bytes": 0, 2024-09-11T08:27:41.109 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_objects": 0, 2024-09-11T08:27:41.109 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T08:27:41.109 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T08:27:41.109 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T08:27:41.109 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_flush_age": 0, 2024-09-11T08:27:41.109 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_evict_age": 0, 2024-09-11T08:27:41.109 INFO:teuthology.orchestra.run.smithi150.stdout: "erasure_code_profile": "", 2024-09-11T08:27:41.109 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_params": { 2024-09-11T08:27:41.110 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "none" 2024-09-11T08:27:41.110 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.110 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_period": 0, 2024-09-11T08:27:41.110 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_count": 0, 2024-09-11T08:27:41.110 INFO:teuthology.orchestra.run.smithi150.stdout: "use_gmt_hitset": true, 2024-09-11T08:27:41.110 INFO:teuthology.orchestra.run.smithi150.stdout: "min_read_recency_for_promote": 0, 2024-09-11T08:27:41.111 INFO:teuthology.orchestra.run.smithi150.stdout: "min_write_recency_for_promote": 0, 2024-09-11T08:27:41.111 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T08:27:41.111 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_search_last_n": 0, 2024-09-11T08:27:41.112 INFO:teuthology.orchestra.run.smithi150.stdout: "grade_table": [], 2024-09-11T08:27:41.112 INFO:teuthology.orchestra.run.smithi150.stdout: "stripe_width": 0, 2024-09-11T08:27:41.112 INFO:teuthology.orchestra.run.smithi150.stdout: "expected_num_objects": 0, 2024-09-11T08:27:41.112 INFO:teuthology.orchestra.run.smithi150.stdout: "fast_read": false, 2024-09-11T08:27:41.112 INFO:teuthology.orchestra.run.smithi150.stdout: "options": { 2024-09-11T08:27:41.112 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_max": 32, 2024-09-11T08:27:41.112 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_min": 1 2024-09-11T08:27:41.113 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.113 INFO:teuthology.orchestra.run.smithi150.stdout: "application_metadata": { 2024-09-11T08:27:41.113 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": {} 2024-09-11T08:27:41.113 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:27:41.113 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.113 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.113 INFO:teuthology.orchestra.run.smithi150.stdout: "pool": 2, 2024-09-11T08:27:41.113 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_name": "datapool", 2024-09-11T08:27:41.114 INFO:teuthology.orchestra.run.smithi150.stdout: "create_time": "2024-09-11T08:07:28.786572+0000", 2024-09-11T08:27:41.114 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 8193, 2024-09-11T08:27:41.114 INFO:teuthology.orchestra.run.smithi150.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-11T08:27:41.114 INFO:teuthology.orchestra.run.smithi150.stdout: "type": 1, 2024-09-11T08:27:41.114 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 3, 2024-09-11T08:27:41.114 INFO:teuthology.orchestra.run.smithi150.stdout: "min_size": 2, 2024-09-11T08:27:41.115 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_rule": 0, 2024-09-11T08:27:41.115 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_count": 0, 2024-09-11T08:27:41.115 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_target": 0, 2024-09-11T08:27:41.115 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T08:27:41.115 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T08:27:41.116 INFO:teuthology.orchestra.run.smithi150.stdout: "object_hash": 2, 2024-09-11T08:27:41.116 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_autoscale_mode": "off", 2024-09-11T08:27:41.116 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num": 3, 2024-09-11T08:27:41.116 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num": 3, 2024-09-11T08:27:41.116 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num_target": 3, 2024-09-11T08:27:41.116 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_target": 3, 2024-09-11T08:27:41.117 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_pending": 3, 2024-09-11T08:27:41.117 INFO:teuthology.orchestra.run.smithi150.stdout: "last_pg_merge_meta": { 2024-09-11T08:27:41.117 INFO:teuthology.orchestra.run.smithi150.stdout: "source_pgid": "0.0", 2024-09-11T08:27:41.117 INFO:teuthology.orchestra.run.smithi150.stdout: "ready_epoch": 0, 2024-09-11T08:27:41.117 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_started": 0, 2024-09-11T08:27:41.117 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_clean": 0, 2024-09-11T08:27:41.117 INFO:teuthology.orchestra.run.smithi150.stdout: "source_version": "0'0", 2024-09-11T08:27:41.117 INFO:teuthology.orchestra.run.smithi150.stdout: "target_version": "0'0" 2024-09-11T08:27:41.117 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.117 INFO:teuthology.orchestra.run.smithi150.stdout: "last_change": "63", 2024-09-11T08:27:41.117 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend": "0", 2024-09-11T08:27:41.117 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T08:27:41.117 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T08:27:41.118 INFO:teuthology.orchestra.run.smithi150.stdout: "auid": 0, 2024-09-11T08:27:41.118 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_mode": "selfmanaged", 2024-09-11T08:27:41.118 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_seq": 3, 2024-09-11T08:27:41.118 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_epoch": 63, 2024-09-11T08:27:41.118 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_snaps": [], 2024-09-11T08:27:41.118 INFO:teuthology.orchestra.run.smithi150.stdout: "removed_snaps": "[]", 2024-09-11T08:27:41.118 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_bytes": 0, 2024-09-11T08:27:41.118 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_objects": 0, 2024-09-11T08:27:41.118 INFO:teuthology.orchestra.run.smithi150.stdout: "tiers": [], 2024-09-11T08:27:41.118 INFO:teuthology.orchestra.run.smithi150.stdout: "tier_of": -1, 2024-09-11T08:27:41.118 INFO:teuthology.orchestra.run.smithi150.stdout: "read_tier": -1, 2024-09-11T08:27:41.119 INFO:teuthology.orchestra.run.smithi150.stdout: "write_tier": -1, 2024-09-11T08:27:41.119 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_mode": "none", 2024-09-11T08:27:41.119 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_bytes": 0, 2024-09-11T08:27:41.119 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_objects": 0, 2024-09-11T08:27:41.119 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T08:27:41.119 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T08:27:41.119 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T08:27:41.119 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_flush_age": 0, 2024-09-11T08:27:41.119 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_evict_age": 0, 2024-09-11T08:27:41.119 INFO:teuthology.orchestra.run.smithi150.stdout: "erasure_code_profile": "", 2024-09-11T08:27:41.119 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_params": { 2024-09-11T08:27:41.119 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "none" 2024-09-11T08:27:41.120 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.120 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_period": 0, 2024-09-11T08:27:41.120 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_count": 0, 2024-09-11T08:27:41.120 INFO:teuthology.orchestra.run.smithi150.stdout: "use_gmt_hitset": true, 2024-09-11T08:27:41.120 INFO:teuthology.orchestra.run.smithi150.stdout: "min_read_recency_for_promote": 0, 2024-09-11T08:27:41.120 INFO:teuthology.orchestra.run.smithi150.stdout: "min_write_recency_for_promote": 0, 2024-09-11T08:27:41.120 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T08:27:41.120 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_search_last_n": 0, 2024-09-11T08:27:41.120 INFO:teuthology.orchestra.run.smithi150.stdout: "grade_table": [], 2024-09-11T08:27:41.120 INFO:teuthology.orchestra.run.smithi150.stdout: "stripe_width": 0, 2024-09-11T08:27:41.120 INFO:teuthology.orchestra.run.smithi150.stdout: "expected_num_objects": 0, 2024-09-11T08:27:41.120 INFO:teuthology.orchestra.run.smithi150.stdout: "fast_read": false, 2024-09-11T08:27:41.120 INFO:teuthology.orchestra.run.smithi150.stdout: "options": {}, 2024-09-11T08:27:41.120 INFO:teuthology.orchestra.run.smithi150.stdout: "application_metadata": { 2024-09-11T08:27:41.121 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd": {} 2024-09-11T08:27:41.121 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:27:41.121 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.121 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.121 INFO:teuthology.orchestra.run.smithi150.stdout: "pool": 3, 2024-09-11T08:27:41.121 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_name": ".rgw.root", 2024-09-11T08:27:41.121 INFO:teuthology.orchestra.run.smithi150.stdout: "create_time": "2024-09-11T08:07:29.021286+0000", 2024-09-11T08:27:41.121 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-09-11T08:27:41.121 INFO:teuthology.orchestra.run.smithi150.stdout: "flags_names": "hashpspool", 2024-09-11T08:27:41.121 INFO:teuthology.orchestra.run.smithi150.stdout: "type": 1, 2024-09-11T08:27:41.121 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 3, 2024-09-11T08:27:41.121 INFO:teuthology.orchestra.run.smithi150.stdout: "min_size": 2, 2024-09-11T08:27:41.121 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_rule": 0, 2024-09-11T08:27:41.122 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_count": 0, 2024-09-11T08:27:41.122 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_target": 0, 2024-09-11T08:27:41.122 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T08:27:41.122 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T08:27:41.122 INFO:teuthology.orchestra.run.smithi150.stdout: "object_hash": 2, 2024-09-11T08:27:41.122 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_autoscale_mode": "off", 2024-09-11T08:27:41.122 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num": 32, 2024-09-11T08:27:41.122 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num": 32, 2024-09-11T08:27:41.122 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num_target": 32, 2024-09-11T08:27:41.122 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_target": 32, 2024-09-11T08:27:41.122 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_pending": 32, 2024-09-11T08:27:41.122 INFO:teuthology.orchestra.run.smithi150.stdout: "last_pg_merge_meta": { 2024-09-11T08:27:41.123 INFO:teuthology.orchestra.run.smithi150.stdout: "source_pgid": "0.0", 2024-09-11T08:27:41.123 INFO:teuthology.orchestra.run.smithi150.stdout: "ready_epoch": 0, 2024-09-11T08:27:41.123 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_started": 0, 2024-09-11T08:27:41.123 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_clean": 0, 2024-09-11T08:27:41.123 INFO:teuthology.orchestra.run.smithi150.stdout: "source_version": "0'0", 2024-09-11T08:27:41.123 INFO:teuthology.orchestra.run.smithi150.stdout: "target_version": "0'0" 2024-09-11T08:27:41.123 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.123 INFO:teuthology.orchestra.run.smithi150.stdout: "last_change": "60", 2024-09-11T08:27:41.123 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend": "0", 2024-09-11T08:27:41.123 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T08:27:41.123 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T08:27:41.123 INFO:teuthology.orchestra.run.smithi150.stdout: "auid": 0, 2024-09-11T08:27:41.123 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_mode": "selfmanaged", 2024-09-11T08:27:41.123 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_seq": 0, 2024-09-11T08:27:41.124 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_epoch": 0, 2024-09-11T08:27:41.124 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_snaps": [], 2024-09-11T08:27:41.124 INFO:teuthology.orchestra.run.smithi150.stdout: "removed_snaps": "[]", 2024-09-11T08:27:41.124 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_bytes": 0, 2024-09-11T08:27:41.124 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_objects": 0, 2024-09-11T08:27:41.124 INFO:teuthology.orchestra.run.smithi150.stdout: "tiers": [], 2024-09-11T08:27:41.124 INFO:teuthology.orchestra.run.smithi150.stdout: "tier_of": -1, 2024-09-11T08:27:41.124 INFO:teuthology.orchestra.run.smithi150.stdout: "read_tier": -1, 2024-09-11T08:27:41.124 INFO:teuthology.orchestra.run.smithi150.stdout: "write_tier": -1, 2024-09-11T08:27:41.124 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_mode": "none", 2024-09-11T08:27:41.124 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_bytes": 0, 2024-09-11T08:27:41.124 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_objects": 0, 2024-09-11T08:27:41.124 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T08:27:41.124 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T08:27:41.125 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T08:27:41.125 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_flush_age": 0, 2024-09-11T08:27:41.125 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_evict_age": 0, 2024-09-11T08:27:41.125 INFO:teuthology.orchestra.run.smithi150.stdout: "erasure_code_profile": "", 2024-09-11T08:27:41.125 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_params": { 2024-09-11T08:27:41.125 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "none" 2024-09-11T08:27:41.126 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.126 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_period": 0, 2024-09-11T08:27:41.126 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_count": 0, 2024-09-11T08:27:41.126 INFO:teuthology.orchestra.run.smithi150.stdout: "use_gmt_hitset": true, 2024-09-11T08:27:41.126 INFO:teuthology.orchestra.run.smithi150.stdout: "min_read_recency_for_promote": 0, 2024-09-11T08:27:41.126 INFO:teuthology.orchestra.run.smithi150.stdout: "min_write_recency_for_promote": 0, 2024-09-11T08:27:41.126 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T08:27:41.126 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_search_last_n": 0, 2024-09-11T08:27:41.126 INFO:teuthology.orchestra.run.smithi150.stdout: "grade_table": [], 2024-09-11T08:27:41.126 INFO:teuthology.orchestra.run.smithi150.stdout: "stripe_width": 0, 2024-09-11T08:27:41.126 INFO:teuthology.orchestra.run.smithi150.stdout: "expected_num_objects": 0, 2024-09-11T08:27:41.126 INFO:teuthology.orchestra.run.smithi150.stdout: "fast_read": false, 2024-09-11T08:27:41.126 INFO:teuthology.orchestra.run.smithi150.stdout: "options": {}, 2024-09-11T08:27:41.127 INFO:teuthology.orchestra.run.smithi150.stdout: "application_metadata": { 2024-09-11T08:27:41.127 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw": {} 2024-09-11T08:27:41.127 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:27:41.127 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.127 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.127 INFO:teuthology.orchestra.run.smithi150.stdout: "pool": 4, 2024-09-11T08:27:41.127 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_name": "default.rgw.log", 2024-09-11T08:27:41.127 INFO:teuthology.orchestra.run.smithi150.stdout: "create_time": "2024-09-11T08:07:30.334181+0000", 2024-09-11T08:27:41.127 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-09-11T08:27:41.127 INFO:teuthology.orchestra.run.smithi150.stdout: "flags_names": "hashpspool", 2024-09-11T08:27:41.127 INFO:teuthology.orchestra.run.smithi150.stdout: "type": 1, 2024-09-11T08:27:41.127 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 3, 2024-09-11T08:27:41.127 INFO:teuthology.orchestra.run.smithi150.stdout: "min_size": 2, 2024-09-11T08:27:41.127 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_rule": 0, 2024-09-11T08:27:41.127 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_count": 0, 2024-09-11T08:27:41.128 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_target": 0, 2024-09-11T08:27:41.128 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T08:27:41.128 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T08:27:41.128 INFO:teuthology.orchestra.run.smithi150.stdout: "object_hash": 2, 2024-09-11T08:27:41.128 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_autoscale_mode": "off", 2024-09-11T08:27:41.128 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num": 32, 2024-09-11T08:27:41.128 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num": 32, 2024-09-11T08:27:41.128 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num_target": 32, 2024-09-11T08:27:41.128 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_target": 32, 2024-09-11T08:27:41.128 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_pending": 32, 2024-09-11T08:27:41.128 INFO:teuthology.orchestra.run.smithi150.stdout: "last_pg_merge_meta": { 2024-09-11T08:27:41.128 INFO:teuthology.orchestra.run.smithi150.stdout: "source_pgid": "0.0", 2024-09-11T08:27:41.128 INFO:teuthology.orchestra.run.smithi150.stdout: "ready_epoch": 0, 2024-09-11T08:27:41.129 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_started": 0, 2024-09-11T08:27:41.129 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_clean": 0, 2024-09-11T08:27:41.129 INFO:teuthology.orchestra.run.smithi150.stdout: "source_version": "0'0", 2024-09-11T08:27:41.129 INFO:teuthology.orchestra.run.smithi150.stdout: "target_version": "0'0" 2024-09-11T08:27:41.129 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.129 INFO:teuthology.orchestra.run.smithi150.stdout: "last_change": "62", 2024-09-11T08:27:41.129 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend": "0", 2024-09-11T08:27:41.129 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T08:27:41.129 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T08:27:41.129 INFO:teuthology.orchestra.run.smithi150.stdout: "auid": 0, 2024-09-11T08:27:41.130 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_mode": "selfmanaged", 2024-09-11T08:27:41.130 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_seq": 0, 2024-09-11T08:27:41.130 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_epoch": 0, 2024-09-11T08:27:41.130 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_snaps": [], 2024-09-11T08:27:41.130 INFO:teuthology.orchestra.run.smithi150.stdout: "removed_snaps": "[]", 2024-09-11T08:27:41.130 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_bytes": 0, 2024-09-11T08:27:41.130 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_objects": 0, 2024-09-11T08:27:41.130 INFO:teuthology.orchestra.run.smithi150.stdout: "tiers": [], 2024-09-11T08:27:41.130 INFO:teuthology.orchestra.run.smithi150.stdout: "tier_of": -1, 2024-09-11T08:27:41.130 INFO:teuthology.orchestra.run.smithi150.stdout: "read_tier": -1, 2024-09-11T08:27:41.130 INFO:teuthology.orchestra.run.smithi150.stdout: "write_tier": -1, 2024-09-11T08:27:41.130 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_mode": "none", 2024-09-11T08:27:41.131 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_bytes": 0, 2024-09-11T08:27:41.131 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_objects": 0, 2024-09-11T08:27:41.131 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T08:27:41.131 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T08:27:41.131 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T08:27:41.131 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_flush_age": 0, 2024-09-11T08:27:41.131 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_evict_age": 0, 2024-09-11T08:27:41.131 INFO:teuthology.orchestra.run.smithi150.stdout: "erasure_code_profile": "", 2024-09-11T08:27:41.131 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_params": { 2024-09-11T08:27:41.131 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "none" 2024-09-11T08:27:41.131 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.131 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_period": 0, 2024-09-11T08:27:41.132 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_count": 0, 2024-09-11T08:27:41.132 INFO:teuthology.orchestra.run.smithi150.stdout: "use_gmt_hitset": true, 2024-09-11T08:27:41.132 INFO:teuthology.orchestra.run.smithi150.stdout: "min_read_recency_for_promote": 0, 2024-09-11T08:27:41.132 INFO:teuthology.orchestra.run.smithi150.stdout: "min_write_recency_for_promote": 0, 2024-09-11T08:27:41.132 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T08:27:41.132 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_search_last_n": 0, 2024-09-11T08:27:41.132 INFO:teuthology.orchestra.run.smithi150.stdout: "grade_table": [], 2024-09-11T08:27:41.132 INFO:teuthology.orchestra.run.smithi150.stdout: "stripe_width": 0, 2024-09-11T08:27:41.132 INFO:teuthology.orchestra.run.smithi150.stdout: "expected_num_objects": 0, 2024-09-11T08:27:41.133 INFO:teuthology.orchestra.run.smithi150.stdout: "fast_read": false, 2024-09-11T08:27:41.133 INFO:teuthology.orchestra.run.smithi150.stdout: "options": {}, 2024-09-11T08:27:41.133 INFO:teuthology.orchestra.run.smithi150.stdout: "application_metadata": { 2024-09-11T08:27:41.133 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw": {} 2024-09-11T08:27:41.133 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:27:41.133 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.133 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.133 INFO:teuthology.orchestra.run.smithi150.stdout: "pool": 5, 2024-09-11T08:27:41.133 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_name": "default.rgw.control", 2024-09-11T08:27:41.134 INFO:teuthology.orchestra.run.smithi150.stdout: "create_time": "2024-09-11T08:07:32.289626+0000", 2024-09-11T08:27:41.134 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-09-11T08:27:41.134 INFO:teuthology.orchestra.run.smithi150.stdout: "flags_names": "hashpspool", 2024-09-11T08:27:41.134 INFO:teuthology.orchestra.run.smithi150.stdout: "type": 1, 2024-09-11T08:27:41.134 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 3, 2024-09-11T08:27:41.134 INFO:teuthology.orchestra.run.smithi150.stdout: "min_size": 2, 2024-09-11T08:27:41.134 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_rule": 0, 2024-09-11T08:27:41.134 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_count": 0, 2024-09-11T08:27:41.134 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_target": 0, 2024-09-11T08:27:41.134 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T08:27:41.134 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T08:27:41.134 INFO:teuthology.orchestra.run.smithi150.stdout: "object_hash": 2, 2024-09-11T08:27:41.134 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_autoscale_mode": "off", 2024-09-11T08:27:41.135 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num": 32, 2024-09-11T08:27:41.135 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num": 32, 2024-09-11T08:27:41.135 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num_target": 32, 2024-09-11T08:27:41.135 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_target": 32, 2024-09-11T08:27:41.135 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_pending": 32, 2024-09-11T08:27:41.135 INFO:teuthology.orchestra.run.smithi150.stdout: "last_pg_merge_meta": { 2024-09-11T08:27:41.135 INFO:teuthology.orchestra.run.smithi150.stdout: "source_pgid": "0.0", 2024-09-11T08:27:41.135 INFO:teuthology.orchestra.run.smithi150.stdout: "ready_epoch": 0, 2024-09-11T08:27:41.135 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_started": 0, 2024-09-11T08:27:41.135 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_clean": 0, 2024-09-11T08:27:41.135 INFO:teuthology.orchestra.run.smithi150.stdout: "source_version": "0'0", 2024-09-11T08:27:41.135 INFO:teuthology.orchestra.run.smithi150.stdout: "target_version": "0'0" 2024-09-11T08:27:41.135 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.135 INFO:teuthology.orchestra.run.smithi150.stdout: "last_change": "64", 2024-09-11T08:27:41.136 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend": "0", 2024-09-11T08:27:41.136 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T08:27:41.136 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T08:27:41.136 INFO:teuthology.orchestra.run.smithi150.stdout: "auid": 0, 2024-09-11T08:27:41.136 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_mode": "selfmanaged", 2024-09-11T08:27:41.136 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_seq": 0, 2024-09-11T08:27:41.136 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_epoch": 0, 2024-09-11T08:27:41.136 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_snaps": [], 2024-09-11T08:27:41.136 INFO:teuthology.orchestra.run.smithi150.stdout: "removed_snaps": "[]", 2024-09-11T08:27:41.136 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_bytes": 0, 2024-09-11T08:27:41.136 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_objects": 0, 2024-09-11T08:27:41.136 INFO:teuthology.orchestra.run.smithi150.stdout: "tiers": [], 2024-09-11T08:27:41.137 INFO:teuthology.orchestra.run.smithi150.stdout: "tier_of": -1, 2024-09-11T08:27:41.137 INFO:teuthology.orchestra.run.smithi150.stdout: "read_tier": -1, 2024-09-11T08:27:41.137 INFO:teuthology.orchestra.run.smithi150.stdout: "write_tier": -1, 2024-09-11T08:27:41.137 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_mode": "none", 2024-09-11T08:27:41.137 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_bytes": 0, 2024-09-11T08:27:41.137 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_objects": 0, 2024-09-11T08:27:41.137 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T08:27:41.137 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T08:27:41.137 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T08:27:41.137 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_flush_age": 0, 2024-09-11T08:27:41.137 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_evict_age": 0, 2024-09-11T08:27:41.137 INFO:teuthology.orchestra.run.smithi150.stdout: "erasure_code_profile": "", 2024-09-11T08:27:41.137 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_params": { 2024-09-11T08:27:41.138 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "none" 2024-09-11T08:27:41.138 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.138 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_period": 0, 2024-09-11T08:27:41.138 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_count": 0, 2024-09-11T08:27:41.138 INFO:teuthology.orchestra.run.smithi150.stdout: "use_gmt_hitset": true, 2024-09-11T08:27:41.138 INFO:teuthology.orchestra.run.smithi150.stdout: "min_read_recency_for_promote": 0, 2024-09-11T08:27:41.138 INFO:teuthology.orchestra.run.smithi150.stdout: "min_write_recency_for_promote": 0, 2024-09-11T08:27:41.138 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T08:27:41.138 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_search_last_n": 0, 2024-09-11T08:27:41.138 INFO:teuthology.orchestra.run.smithi150.stdout: "grade_table": [], 2024-09-11T08:27:41.138 INFO:teuthology.orchestra.run.smithi150.stdout: "stripe_width": 0, 2024-09-11T08:27:41.138 INFO:teuthology.orchestra.run.smithi150.stdout: "expected_num_objects": 0, 2024-09-11T08:27:41.138 INFO:teuthology.orchestra.run.smithi150.stdout: "fast_read": false, 2024-09-11T08:27:41.139 INFO:teuthology.orchestra.run.smithi150.stdout: "options": {}, 2024-09-11T08:27:41.139 INFO:teuthology.orchestra.run.smithi150.stdout: "application_metadata": { 2024-09-11T08:27:41.139 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw": {} 2024-09-11T08:27:41.139 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:27:41.139 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.139 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.139 INFO:teuthology.orchestra.run.smithi150.stdout: "pool": 6, 2024-09-11T08:27:41.139 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_name": "default.rgw.meta", 2024-09-11T08:27:41.139 INFO:teuthology.orchestra.run.smithi150.stdout: "create_time": "2024-09-11T08:07:34.351861+0000", 2024-09-11T08:27:41.139 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-09-11T08:27:41.139 INFO:teuthology.orchestra.run.smithi150.stdout: "flags_names": "hashpspool", 2024-09-11T08:27:41.139 INFO:teuthology.orchestra.run.smithi150.stdout: "type": 1, 2024-09-11T08:27:41.139 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 3, 2024-09-11T08:27:41.139 INFO:teuthology.orchestra.run.smithi150.stdout: "min_size": 2, 2024-09-11T08:27:41.140 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_rule": 0, 2024-09-11T08:27:41.140 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_count": 0, 2024-09-11T08:27:41.140 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_target": 0, 2024-09-11T08:27:41.140 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T08:27:41.140 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T08:27:41.140 INFO:teuthology.orchestra.run.smithi150.stdout: "object_hash": 2, 2024-09-11T08:27:41.140 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_autoscale_mode": "off", 2024-09-11T08:27:41.140 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num": 32, 2024-09-11T08:27:41.140 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num": 32, 2024-09-11T08:27:41.140 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num_target": 32, 2024-09-11T08:27:41.140 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_target": 32, 2024-09-11T08:27:41.140 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_pending": 32, 2024-09-11T08:27:41.141 INFO:teuthology.orchestra.run.smithi150.stdout: "last_pg_merge_meta": { 2024-09-11T08:27:41.141 INFO:teuthology.orchestra.run.smithi150.stdout: "source_pgid": "0.0", 2024-09-11T08:27:41.141 INFO:teuthology.orchestra.run.smithi150.stdout: "ready_epoch": 0, 2024-09-11T08:27:41.141 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_started": 0, 2024-09-11T08:27:41.141 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_clean": 0, 2024-09-11T08:27:41.141 INFO:teuthology.orchestra.run.smithi150.stdout: "source_version": "0'0", 2024-09-11T08:27:41.141 INFO:teuthology.orchestra.run.smithi150.stdout: "target_version": "0'0" 2024-09-11T08:27:41.141 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.141 INFO:teuthology.orchestra.run.smithi150.stdout: "last_change": "66", 2024-09-11T08:27:41.141 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend": "0", 2024-09-11T08:27:41.141 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T08:27:41.141 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T08:27:41.141 INFO:teuthology.orchestra.run.smithi150.stdout: "auid": 0, 2024-09-11T08:27:41.141 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_mode": "selfmanaged", 2024-09-11T08:27:41.142 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_seq": 0, 2024-09-11T08:27:41.142 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_epoch": 0, 2024-09-11T08:27:41.142 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_snaps": [], 2024-09-11T08:27:41.142 INFO:teuthology.orchestra.run.smithi150.stdout: "removed_snaps": "[]", 2024-09-11T08:27:41.142 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_bytes": 0, 2024-09-11T08:27:41.142 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_objects": 0, 2024-09-11T08:27:41.142 INFO:teuthology.orchestra.run.smithi150.stdout: "tiers": [], 2024-09-11T08:27:41.142 INFO:teuthology.orchestra.run.smithi150.stdout: "tier_of": -1, 2024-09-11T08:27:41.142 INFO:teuthology.orchestra.run.smithi150.stdout: "read_tier": -1, 2024-09-11T08:27:41.142 INFO:teuthology.orchestra.run.smithi150.stdout: "write_tier": -1, 2024-09-11T08:27:41.142 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_mode": "none", 2024-09-11T08:27:41.142 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_bytes": 0, 2024-09-11T08:27:41.143 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_objects": 0, 2024-09-11T08:27:41.143 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T08:27:41.143 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T08:27:41.143 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T08:27:41.143 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_flush_age": 0, 2024-09-11T08:27:41.143 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_evict_age": 0, 2024-09-11T08:27:41.143 INFO:teuthology.orchestra.run.smithi150.stdout: "erasure_code_profile": "", 2024-09-11T08:27:41.143 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_params": { 2024-09-11T08:27:41.143 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "none" 2024-09-11T08:27:41.143 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.143 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_period": 0, 2024-09-11T08:27:41.143 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_count": 0, 2024-09-11T08:27:41.143 INFO:teuthology.orchestra.run.smithi150.stdout: "use_gmt_hitset": true, 2024-09-11T08:27:41.144 INFO:teuthology.orchestra.run.smithi150.stdout: "min_read_recency_for_promote": 0, 2024-09-11T08:27:41.144 INFO:teuthology.orchestra.run.smithi150.stdout: "min_write_recency_for_promote": 0, 2024-09-11T08:27:41.144 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T08:27:41.144 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_search_last_n": 0, 2024-09-11T08:27:41.144 INFO:teuthology.orchestra.run.smithi150.stdout: "grade_table": [], 2024-09-11T08:27:41.144 INFO:teuthology.orchestra.run.smithi150.stdout: "stripe_width": 0, 2024-09-11T08:27:41.144 INFO:teuthology.orchestra.run.smithi150.stdout: "expected_num_objects": 0, 2024-09-11T08:27:41.144 INFO:teuthology.orchestra.run.smithi150.stdout: "fast_read": false, 2024-09-11T08:27:41.144 INFO:teuthology.orchestra.run.smithi150.stdout: "options": { 2024-09-11T08:27:41.144 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_autoscale_bias": 4 2024-09-11T08:27:41.144 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.145 INFO:teuthology.orchestra.run.smithi150.stdout: "application_metadata": { 2024-09-11T08:27:41.145 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw": {} 2024-09-11T08:27:41.145 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:27:41.145 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.145 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.145 INFO:teuthology.orchestra.run.smithi150.stdout: "pool": 11, 2024-09-11T08:27:41.145 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_name": "unique_pool_4", 2024-09-11T08:27:41.145 INFO:teuthology.orchestra.run.smithi150.stdout: "create_time": "2024-09-11T08:26:26.054499+0000", 2024-09-11T08:27:41.145 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-09-11T08:27:41.145 INFO:teuthology.orchestra.run.smithi150.stdout: "flags_names": "hashpspool", 2024-09-11T08:27:41.145 INFO:teuthology.orchestra.run.smithi150.stdout: "type": 1, 2024-09-11T08:27:41.145 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 3, 2024-09-11T08:27:41.145 INFO:teuthology.orchestra.run.smithi150.stdout: "min_size": 2, 2024-09-11T08:27:41.146 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_rule": 0, 2024-09-11T08:27:41.146 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_count": 0, 2024-09-11T08:27:41.146 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_target": 0, 2024-09-11T08:27:41.146 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T08:27:41.146 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T08:27:41.146 INFO:teuthology.orchestra.run.smithi150.stdout: "object_hash": 2, 2024-09-11T08:27:41.146 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_autoscale_mode": "off", 2024-09-11T08:27:41.146 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num": 16, 2024-09-11T08:27:41.146 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num": 16, 2024-09-11T08:27:41.146 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num_target": 16, 2024-09-11T08:27:41.146 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_target": 16, 2024-09-11T08:27:41.147 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_pending": 16, 2024-09-11T08:27:41.147 INFO:teuthology.orchestra.run.smithi150.stdout: "last_pg_merge_meta": { 2024-09-11T08:27:41.147 INFO:teuthology.orchestra.run.smithi150.stdout: "source_pgid": "0.0", 2024-09-11T08:27:41.147 INFO:teuthology.orchestra.run.smithi150.stdout: "ready_epoch": 0, 2024-09-11T08:27:41.147 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_started": 0, 2024-09-11T08:27:41.147 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_clean": 0, 2024-09-11T08:27:41.147 INFO:teuthology.orchestra.run.smithi150.stdout: "source_version": "0'0", 2024-09-11T08:27:41.147 INFO:teuthology.orchestra.run.smithi150.stdout: "target_version": "0'0" 2024-09-11T08:27:41.147 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.147 INFO:teuthology.orchestra.run.smithi150.stdout: "last_change": "388", 2024-09-11T08:27:41.147 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend": "0", 2024-09-11T08:27:41.147 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T08:27:41.147 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T08:27:41.148 INFO:teuthology.orchestra.run.smithi150.stdout: "auid": 0, 2024-09-11T08:27:41.148 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_mode": "selfmanaged", 2024-09-11T08:27:41.148 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_seq": 0, 2024-09-11T08:27:41.148 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_epoch": 0, 2024-09-11T08:27:41.148 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_snaps": [], 2024-09-11T08:27:41.148 INFO:teuthology.orchestra.run.smithi150.stdout: "removed_snaps": "[]", 2024-09-11T08:27:41.148 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_bytes": 0, 2024-09-11T08:27:41.148 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_objects": 0, 2024-09-11T08:27:41.148 INFO:teuthology.orchestra.run.smithi150.stdout: "tiers": [], 2024-09-11T08:27:41.148 INFO:teuthology.orchestra.run.smithi150.stdout: "tier_of": -1, 2024-09-11T08:27:41.148 INFO:teuthology.orchestra.run.smithi150.stdout: "read_tier": -1, 2024-09-11T08:27:41.148 INFO:teuthology.orchestra.run.smithi150.stdout: "write_tier": -1, 2024-09-11T08:27:41.149 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_mode": "none", 2024-09-11T08:27:41.149 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_bytes": 0, 2024-09-11T08:27:41.149 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_objects": 0, 2024-09-11T08:27:41.149 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T08:27:41.149 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T08:27:41.149 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T08:27:41.149 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_flush_age": 0, 2024-09-11T08:27:41.149 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_evict_age": 0, 2024-09-11T08:27:41.149 INFO:teuthology.orchestra.run.smithi150.stdout: "erasure_code_profile": "", 2024-09-11T08:27:41.149 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_params": { 2024-09-11T08:27:41.149 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "none" 2024-09-11T08:27:41.149 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.149 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_period": 0, 2024-09-11T08:27:41.149 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_count": 0, 2024-09-11T08:27:41.150 INFO:teuthology.orchestra.run.smithi150.stdout: "use_gmt_hitset": true, 2024-09-11T08:27:41.150 INFO:teuthology.orchestra.run.smithi150.stdout: "min_read_recency_for_promote": 0, 2024-09-11T08:27:41.150 INFO:teuthology.orchestra.run.smithi150.stdout: "min_write_recency_for_promote": 0, 2024-09-11T08:27:41.150 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T08:27:41.150 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_search_last_n": 0, 2024-09-11T08:27:41.150 INFO:teuthology.orchestra.run.smithi150.stdout: "grade_table": [], 2024-09-11T08:27:41.150 INFO:teuthology.orchestra.run.smithi150.stdout: "stripe_width": 0, 2024-09-11T08:27:41.150 INFO:teuthology.orchestra.run.smithi150.stdout: "expected_num_objects": 0, 2024-09-11T08:27:41.150 INFO:teuthology.orchestra.run.smithi150.stdout: "fast_read": false, 2024-09-11T08:27:41.150 INFO:teuthology.orchestra.run.smithi150.stdout: "options": {}, 2024-09-11T08:27:41.150 INFO:teuthology.orchestra.run.smithi150.stdout: "application_metadata": { 2024-09-11T08:27:41.150 INFO:teuthology.orchestra.run.smithi150.stdout: "rados": {} 2024-09-11T08:27:41.150 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:27:41.150 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:27:41.151 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-09-11T08:27:41.151 INFO:teuthology.orchestra.run.smithi150.stdout: "osds": [ 2024-09-11T08:27:41.151 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.151 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 0, 2024-09-11T08:27:41.151 INFO:teuthology.orchestra.run.smithi150.stdout: "uuid": "37b63db9-ed2a-4701-b63d-03ff9885cc0e", 2024-09-11T08:27:41.151 INFO:teuthology.orchestra.run.smithi150.stdout: "up": 1, 2024-09-11T08:27:41.151 INFO:teuthology.orchestra.run.smithi150.stdout: "in": 1, 2024-09-11T08:27:41.151 INFO:teuthology.orchestra.run.smithi150.stdout: "weight": 1, 2024-09-11T08:27:41.151 INFO:teuthology.orchestra.run.smithi150.stdout: "primary_affinity": 1, 2024-09-11T08:27:41.151 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_begin": 0, 2024-09-11T08:27:41.151 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_end": 0, 2024-09-11T08:27:41.151 INFO:teuthology.orchestra.run.smithi150.stdout: "up_from": 8, 2024-09-11T08:27:41.151 INFO:teuthology.orchestra.run.smithi150.stdout: "up_thru": 382, 2024-09-11T08:27:41.151 INFO:teuthology.orchestra.run.smithi150.stdout: "down_at": 0, 2024-09-11T08:27:41.151 INFO:teuthology.orchestra.run.smithi150.stdout: "lost_at": 0, 2024-09-11T08:27:41.152 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addrs": { 2024-09-11T08:27:41.152 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:27:41.152 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.152 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:27:41.152 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6801", 2024-09-11T08:27:41.152 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 334255279 2024-09-11T08:27:41.152 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:27:41.152 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:27:41.152 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.152 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addrs": { 2024-09-11T08:27:41.152 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:27:41.152 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.152 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:27:41.153 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6802", 2024-09-11T08:27:41.153 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 334255279 2024-09-11T08:27:41.153 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:27:41.153 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:27:41.153 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.153 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addrs": { 2024-09-11T08:27:41.153 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:27:41.153 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.153 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:27:41.153 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6804", 2024-09-11T08:27:41.153 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 334255279 2024-09-11T08:27:41.153 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:27:41.153 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:27:41.153 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.154 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addrs": { 2024-09-11T08:27:41.154 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:27:41.154 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.154 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:27:41.154 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6803", 2024-09-11T08:27:41.154 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 334255279 2024-09-11T08:27:41.154 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:27:41.154 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:27:41.154 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.154 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addr": "172.21.15.150:6801/334255279", 2024-09-11T08:27:41.154 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addr": "172.21.15.150:6802/334255279", 2024-09-11T08:27:41.154 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addr": "172.21.15.150:6804/334255279", 2024-09-11T08:27:41.154 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addr": "172.21.15.150:6803/334255279", 2024-09-11T08:27:41.154 INFO:teuthology.orchestra.run.smithi150.stdout: "state": [ 2024-09-11T08:27:41.154 INFO:teuthology.orchestra.run.smithi150.stdout: "exists", 2024-09-11T08:27:41.155 INFO:teuthology.orchestra.run.smithi150.stdout: "up" 2024-09-11T08:27:41.155 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:27:41.155 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.155 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.155 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 1, 2024-09-11T08:27:41.155 INFO:teuthology.orchestra.run.smithi150.stdout: "uuid": "80378442-02bc-48aa-b8ca-6113a4092079", 2024-09-11T08:27:41.155 INFO:teuthology.orchestra.run.smithi150.stdout: "up": 1, 2024-09-11T08:27:41.155 INFO:teuthology.orchestra.run.smithi150.stdout: "in": 1, 2024-09-11T08:27:41.155 INFO:teuthology.orchestra.run.smithi150.stdout: "weight": 1, 2024-09-11T08:27:41.155 INFO:teuthology.orchestra.run.smithi150.stdout: "primary_affinity": 1, 2024-09-11T08:27:41.155 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_begin": 0, 2024-09-11T08:27:41.155 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_end": 0, 2024-09-11T08:27:41.155 INFO:teuthology.orchestra.run.smithi150.stdout: "up_from": 13, 2024-09-11T08:27:41.155 INFO:teuthology.orchestra.run.smithi150.stdout: "up_thru": 382, 2024-09-11T08:27:41.156 INFO:teuthology.orchestra.run.smithi150.stdout: "down_at": 0, 2024-09-11T08:27:41.156 INFO:teuthology.orchestra.run.smithi150.stdout: "lost_at": 0, 2024-09-11T08:27:41.156 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addrs": { 2024-09-11T08:27:41.156 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:27:41.156 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.156 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:27:41.156 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6805", 2024-09-11T08:27:41.156 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1129510619 2024-09-11T08:27:41.156 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:27:41.156 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:27:41.156 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.156 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addrs": { 2024-09-11T08:27:41.156 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:27:41.156 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.156 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:27:41.157 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6806", 2024-09-11T08:27:41.157 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1129510619 2024-09-11T08:27:41.157 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:27:41.157 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:27:41.157 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.157 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addrs": { 2024-09-11T08:27:41.157 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:27:41.157 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.157 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:27:41.157 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6808", 2024-09-11T08:27:41.157 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1129510619 2024-09-11T08:27:41.157 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:27:41.157 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:27:41.158 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.158 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addrs": { 2024-09-11T08:27:41.158 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:27:41.158 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.158 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:27:41.158 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6807", 2024-09-11T08:27:41.158 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1129510619 2024-09-11T08:27:41.158 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:27:41.158 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:27:41.158 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.158 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addr": "172.21.15.150:6805/1129510619", 2024-09-11T08:27:41.158 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addr": "172.21.15.150:6806/1129510619", 2024-09-11T08:27:41.158 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addr": "172.21.15.150:6808/1129510619", 2024-09-11T08:27:41.158 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addr": "172.21.15.150:6807/1129510619", 2024-09-11T08:27:41.159 INFO:teuthology.orchestra.run.smithi150.stdout: "state": [ 2024-09-11T08:27:41.159 INFO:teuthology.orchestra.run.smithi150.stdout: "exists", 2024-09-11T08:27:41.159 INFO:teuthology.orchestra.run.smithi150.stdout: "up" 2024-09-11T08:27:41.159 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:27:41.159 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.159 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.159 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 2, 2024-09-11T08:27:41.159 INFO:teuthology.orchestra.run.smithi150.stdout: "uuid": "0955bb92-36a0-4e9a-98e1-b3a5af89f6d2", 2024-09-11T08:27:41.159 INFO:teuthology.orchestra.run.smithi150.stdout: "up": 1, 2024-09-11T08:27:41.159 INFO:teuthology.orchestra.run.smithi150.stdout: "in": 1, 2024-09-11T08:27:41.159 INFO:teuthology.orchestra.run.smithi150.stdout: "weight": 1, 2024-09-11T08:27:41.159 INFO:teuthology.orchestra.run.smithi150.stdout: "primary_affinity": 1, 2024-09-11T08:27:41.159 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_begin": 0, 2024-09-11T08:27:41.159 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_end": 0, 2024-09-11T08:27:41.159 INFO:teuthology.orchestra.run.smithi150.stdout: "up_from": 18, 2024-09-11T08:27:41.160 INFO:teuthology.orchestra.run.smithi150.stdout: "up_thru": 382, 2024-09-11T08:27:41.160 INFO:teuthology.orchestra.run.smithi150.stdout: "down_at": 0, 2024-09-11T08:27:41.160 INFO:teuthology.orchestra.run.smithi150.stdout: "lost_at": 0, 2024-09-11T08:27:41.160 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addrs": { 2024-09-11T08:27:41.160 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:27:41.160 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.160 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:27:41.160 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6809", 2024-09-11T08:27:41.160 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 719975422 2024-09-11T08:27:41.160 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:27:41.161 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:27:41.161 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.161 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addrs": { 2024-09-11T08:27:41.161 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:27:41.161 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.161 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:27:41.161 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6810", 2024-09-11T08:27:41.161 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 719975422 2024-09-11T08:27:41.161 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:27:41.161 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:27:41.161 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.161 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addrs": { 2024-09-11T08:27:41.161 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:27:41.161 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.162 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:27:41.162 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6812", 2024-09-11T08:27:41.162 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 719975422 2024-09-11T08:27:41.162 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:27:41.162 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:27:41.162 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.162 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addrs": { 2024-09-11T08:27:41.162 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:27:41.162 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.162 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:27:41.162 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6811", 2024-09-11T08:27:41.162 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 719975422 2024-09-11T08:27:41.162 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:27:41.162 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:27:41.163 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.163 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addr": "172.21.15.150:6809/719975422", 2024-09-11T08:27:41.163 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addr": "172.21.15.150:6810/719975422", 2024-09-11T08:27:41.163 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addr": "172.21.15.150:6812/719975422", 2024-09-11T08:27:41.163 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addr": "172.21.15.150:6811/719975422", 2024-09-11T08:27:41.163 INFO:teuthology.orchestra.run.smithi150.stdout: "state": [ 2024-09-11T08:27:41.163 INFO:teuthology.orchestra.run.smithi150.stdout: "exists", 2024-09-11T08:27:41.163 INFO:teuthology.orchestra.run.smithi150.stdout: "up" 2024-09-11T08:27:41.163 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:27:41.164 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.164 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:41.165 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.165 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 3, 2024-09-11T08:27:41.165 INFO:teuthology.orchestra.run.smithi150.stdout: "uuid": "1d04d4d3-62e8-4460-8a07-77f7e5c62929", 2024-09-11T08:27:41.165 INFO:teuthology.orchestra.run.smithi150.stdout: "up": 1, 2024-09-11T08:27:41.165 INFO:teuthology.orchestra.run.smithi150.stdout: "in": 1, 2024-09-11T08:27:41.165 INFO:teuthology.orchestra.run.smithi150.stdout: "weight": 1, 2024-09-11T08:27:41.166 INFO:teuthology.orchestra.run.smithi150.stdout: "primary_affinity": 0.611114501953125, 2024-09-11T08:27:41.166 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_begin": 0, 2024-09-11T08:27:41.166 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_end": 0, 2024-09-11T08:27:41.166 INFO:teuthology.orchestra.run.smithi150.stdout: "up_from": 25, 2024-09-11T08:27:41.166 INFO:teuthology.orchestra.run.smithi150.stdout: "up_thru": 382, 2024-09-11T08:27:41.166 INFO:teuthology.orchestra.run.smithi150.stdout: "down_at": 0, 2024-09-11T08:27:41.166 INFO:teuthology.orchestra.run.smithi150.stdout: "lost_at": 0, 2024-09-11T08:27:41.166 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addrs": { 2024-09-11T08:27:41.166 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:27:41.166 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.166 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:27:41.166 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6813", 2024-09-11T08:27:41.166 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 3101032500 2024-09-11T08:27:41.166 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:27:41.167 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:27:41.167 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.167 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addrs": { 2024-09-11T08:27:41.167 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:27:41.167 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.167 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:27:41.167 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6814", 2024-09-11T08:27:41.167 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 3101032500 2024-09-11T08:27:41.167 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:27:41.167 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:27:41.167 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.167 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addrs": { 2024-09-11T08:27:41.167 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:27:41.168 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.168 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:27:41.168 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6816", 2024-09-11T08:27:41.168 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 3101032500 2024-09-11T08:27:41.168 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:27:41.168 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:27:41.168 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.168 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addrs": { 2024-09-11T08:27:41.168 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:27:41.168 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.168 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:27:41.168 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6815", 2024-09-11T08:27:41.168 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 3101032500 2024-09-11T08:27:41.168 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:27:41.169 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:27:41.169 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.169 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addr": "172.21.15.150:6813/3101032500", 2024-09-11T08:27:41.169 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addr": "172.21.15.150:6814/3101032500", 2024-09-11T08:27:41.169 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addr": "172.21.15.150:6816/3101032500", 2024-09-11T08:27:41.169 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addr": "172.21.15.150:6815/3101032500", 2024-09-11T08:27:41.169 INFO:teuthology.orchestra.run.smithi150.stdout: "state": [ 2024-09-11T08:27:41.169 INFO:teuthology.orchestra.run.smithi150.stdout: "exists", 2024-09-11T08:27:41.169 INFO:teuthology.orchestra.run.smithi150.stdout: "up" 2024-09-11T08:27:41.169 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:27:41.169 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.169 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.169 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 4, 2024-09-11T08:27:41.169 INFO:teuthology.orchestra.run.smithi150.stdout: "uuid": "c5321a3c-d2f0-4440-8475-2c111d261139", 2024-09-11T08:27:41.170 INFO:teuthology.orchestra.run.smithi150.stdout: "up": 1, 2024-09-11T08:27:41.170 INFO:teuthology.orchestra.run.smithi150.stdout: "in": 1, 2024-09-11T08:27:41.170 INFO:teuthology.orchestra.run.smithi150.stdout: "weight": 1, 2024-09-11T08:27:41.170 INFO:teuthology.orchestra.run.smithi150.stdout: "primary_affinity": 0.2719573974609375, 2024-09-11T08:27:41.170 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_begin": 0, 2024-09-11T08:27:41.170 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_end": 0, 2024-09-11T08:27:41.170 INFO:teuthology.orchestra.run.smithi150.stdout: "up_from": 31, 2024-09-11T08:27:41.170 INFO:teuthology.orchestra.run.smithi150.stdout: "up_thru": 382, 2024-09-11T08:27:41.170 INFO:teuthology.orchestra.run.smithi150.stdout: "down_at": 0, 2024-09-11T08:27:41.170 INFO:teuthology.orchestra.run.smithi150.stdout: "lost_at": 0, 2024-09-11T08:27:41.170 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addrs": { 2024-09-11T08:27:41.170 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:27:41.170 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.170 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:27:41.170 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6800", 2024-09-11T08:27:41.171 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2567452825 2024-09-11T08:27:41.171 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:27:41.171 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:27:41.171 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.171 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addrs": { 2024-09-11T08:27:41.171 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:27:41.171 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.171 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:27:41.171 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6801", 2024-09-11T08:27:41.171 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2567452825 2024-09-11T08:27:41.171 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:27:41.171 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:27:41.171 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.171 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addrs": { 2024-09-11T08:27:41.172 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:27:41.172 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.172 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:27:41.172 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6803", 2024-09-11T08:27:41.172 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2567452825 2024-09-11T08:27:41.172 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:27:41.172 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:27:41.172 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.172 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addrs": { 2024-09-11T08:27:41.172 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:27:41.172 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.172 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:27:41.172 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6802", 2024-09-11T08:27:41.172 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2567452825 2024-09-11T08:27:41.173 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:27:41.173 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:27:41.173 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.173 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addr": "172.21.15.181:6800/2567452825", 2024-09-11T08:27:41.173 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addr": "172.21.15.181:6801/2567452825", 2024-09-11T08:27:41.173 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addr": "172.21.15.181:6803/2567452825", 2024-09-11T08:27:41.173 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addr": "172.21.15.181:6802/2567452825", 2024-09-11T08:27:41.173 INFO:teuthology.orchestra.run.smithi150.stdout: "state": [ 2024-09-11T08:27:41.173 INFO:teuthology.orchestra.run.smithi150.stdout: "exists", 2024-09-11T08:27:41.173 INFO:teuthology.orchestra.run.smithi150.stdout: "up" 2024-09-11T08:27:41.174 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:27:41.174 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.174 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.174 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 5, 2024-09-11T08:27:41.174 INFO:teuthology.orchestra.run.smithi150.stdout: "uuid": "1262b2fe-f253-4a26-bfd4-c9dffb697c77", 2024-09-11T08:27:41.174 INFO:teuthology.orchestra.run.smithi150.stdout: "up": 1, 2024-09-11T08:27:41.174 INFO:teuthology.orchestra.run.smithi150.stdout: "in": 1, 2024-09-11T08:27:41.174 INFO:teuthology.orchestra.run.smithi150.stdout: "weight": 1, 2024-09-11T08:27:41.174 INFO:teuthology.orchestra.run.smithi150.stdout: "primary_affinity": 0, 2024-09-11T08:27:41.174 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_begin": 38, 2024-09-11T08:27:41.174 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_end": 339, 2024-09-11T08:27:41.174 INFO:teuthology.orchestra.run.smithi150.stdout: "up_from": 353, 2024-09-11T08:27:41.174 INFO:teuthology.orchestra.run.smithi150.stdout: "up_thru": 353, 2024-09-11T08:27:41.174 INFO:teuthology.orchestra.run.smithi150.stdout: "down_at": 340, 2024-09-11T08:27:41.175 INFO:teuthology.orchestra.run.smithi150.stdout: "lost_at": 0, 2024-09-11T08:27:41.175 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addrs": { 2024-09-11T08:27:41.175 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:27:41.175 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.175 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:27:41.175 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6804", 2024-09-11T08:27:41.175 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1821561027 2024-09-11T08:27:41.175 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:27:41.175 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:27:41.175 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.175 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addrs": { 2024-09-11T08:27:41.175 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:27:41.175 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.175 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:27:41.176 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6805", 2024-09-11T08:27:41.176 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1821561027 2024-09-11T08:27:41.176 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:27:41.176 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:27:41.176 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.176 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addrs": { 2024-09-11T08:27:41.176 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:27:41.176 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.176 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:27:41.176 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6807", 2024-09-11T08:27:41.176 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1821561027 2024-09-11T08:27:41.176 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:27:41.177 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:27:41.177 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.177 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addrs": { 2024-09-11T08:27:41.177 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:27:41.177 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.177 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:27:41.178 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6806", 2024-09-11T08:27:41.178 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1821561027 2024-09-11T08:27:41.178 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:27:41.178 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:27:41.178 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.178 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addr": "172.21.15.181:6804/1821561027", 2024-09-11T08:27:41.178 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addr": "172.21.15.181:6805/1821561027", 2024-09-11T08:27:41.179 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addr": "172.21.15.181:6807/1821561027", 2024-09-11T08:27:41.179 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addr": "172.21.15.181:6806/1821561027", 2024-09-11T08:27:41.179 INFO:teuthology.orchestra.run.smithi150.stdout: "state": [ 2024-09-11T08:27:41.179 INFO:teuthology.orchestra.run.smithi150.stdout: "exists", 2024-09-11T08:27:41.179 INFO:teuthology.orchestra.run.smithi150.stdout: "up" 2024-09-11T08:27:41.179 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:27:41.180 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.180 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.180 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 6, 2024-09-11T08:27:41.180 INFO:teuthology.orchestra.run.smithi150.stdout: "uuid": "7facca22-b3f1-4c20-833e-e3db798a08c4", 2024-09-11T08:27:41.180 INFO:teuthology.orchestra.run.smithi150.stdout: "up": 1, 2024-09-11T08:27:41.180 INFO:teuthology.orchestra.run.smithi150.stdout: "in": 1, 2024-09-11T08:27:41.181 INFO:teuthology.orchestra.run.smithi150.stdout: "weight": 1, 2024-09-11T08:27:41.181 INFO:teuthology.orchestra.run.smithi150.stdout: "primary_affinity": 1, 2024-09-11T08:27:41.181 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_begin": 0, 2024-09-11T08:27:41.181 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_end": 0, 2024-09-11T08:27:41.181 INFO:teuthology.orchestra.run.smithi150.stdout: "up_from": 45, 2024-09-11T08:27:41.181 INFO:teuthology.orchestra.run.smithi150.stdout: "up_thru": 382, 2024-09-11T08:27:41.182 INFO:teuthology.orchestra.run.smithi150.stdout: "down_at": 0, 2024-09-11T08:27:41.182 INFO:teuthology.orchestra.run.smithi150.stdout: "lost_at": 0, 2024-09-11T08:27:41.182 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addrs": { 2024-09-11T08:27:41.182 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:27:41.182 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.182 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:27:41.183 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6808", 2024-09-11T08:27:41.183 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1789880331 2024-09-11T08:27:41.183 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:27:41.183 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:27:41.183 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.183 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addrs": { 2024-09-11T08:27:41.184 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:27:41.184 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.184 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:27:41.184 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6809", 2024-09-11T08:27:41.184 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1789880331 2024-09-11T08:27:41.184 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:27:41.185 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:27:41.185 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.185 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addrs": { 2024-09-11T08:27:41.185 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:27:41.185 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.186 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:27:41.186 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6811", 2024-09-11T08:27:41.186 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1789880331 2024-09-11T08:27:41.186 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:27:41.186 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:27:41.186 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.187 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addrs": { 2024-09-11T08:27:41.187 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:27:41.187 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.187 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:27:41.187 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6810", 2024-09-11T08:27:41.188 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1789880331 2024-09-11T08:27:41.188 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:27:41.188 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:27:41.188 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.188 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addr": "172.21.15.181:6808/1789880331", 2024-09-11T08:27:41.188 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addr": "172.21.15.181:6809/1789880331", 2024-09-11T08:27:41.189 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addr": "172.21.15.181:6811/1789880331", 2024-09-11T08:27:41.189 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addr": "172.21.15.181:6810/1789880331", 2024-09-11T08:27:41.189 INFO:teuthology.orchestra.run.smithi150.stdout: "state": [ 2024-09-11T08:27:41.189 INFO:teuthology.orchestra.run.smithi150.stdout: "exists", 2024-09-11T08:27:41.189 INFO:teuthology.orchestra.run.smithi150.stdout: "up" 2024-09-11T08:27:41.190 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:27:41.190 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.190 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.190 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 7, 2024-09-11T08:27:41.190 INFO:teuthology.orchestra.run.smithi150.stdout: "uuid": "6ac1b06a-79bd-43ca-b91e-fcb0f4033a34", 2024-09-11T08:27:41.190 INFO:teuthology.orchestra.run.smithi150.stdout: "up": 1, 2024-09-11T08:27:41.191 INFO:teuthology.orchestra.run.smithi150.stdout: "in": 1, 2024-09-11T08:27:41.191 INFO:teuthology.orchestra.run.smithi150.stdout: "weight": 1, 2024-09-11T08:27:41.191 INFO:teuthology.orchestra.run.smithi150.stdout: "primary_affinity": 1, 2024-09-11T08:27:41.191 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_begin": 53, 2024-09-11T08:27:41.192 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_end": 139, 2024-09-11T08:27:41.192 INFO:teuthology.orchestra.run.smithi150.stdout: "up_from": 168, 2024-09-11T08:27:41.192 INFO:teuthology.orchestra.run.smithi150.stdout: "up_thru": 382, 2024-09-11T08:27:41.192 INFO:teuthology.orchestra.run.smithi150.stdout: "down_at": 140, 2024-09-11T08:27:41.192 INFO:teuthology.orchestra.run.smithi150.stdout: "lost_at": 0, 2024-09-11T08:27:41.192 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addrs": { 2024-09-11T08:27:41.193 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:27:41.193 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.193 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:27:41.193 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6812", 2024-09-11T08:27:41.193 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2706455857 2024-09-11T08:27:41.193 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:27:41.194 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:27:41.194 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.194 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addrs": { 2024-09-11T08:27:41.194 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:27:41.194 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.194 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:27:41.195 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6813", 2024-09-11T08:27:41.195 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2706455857 2024-09-11T08:27:41.195 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:27:41.195 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:27:41.195 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.196 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addrs": { 2024-09-11T08:27:41.196 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:27:41.196 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.196 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:27:41.196 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6815", 2024-09-11T08:27:41.196 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2706455857 2024-09-11T08:27:41.197 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:27:41.197 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:27:41.197 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.197 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addrs": { 2024-09-11T08:27:41.197 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:27:41.197 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.198 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:27:41.198 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6814", 2024-09-11T08:27:41.198 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2706455857 2024-09-11T08:27:41.198 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:27:41.198 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:27:41.198 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.199 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addr": "172.21.15.181:6812/2706455857", 2024-09-11T08:27:41.199 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addr": "172.21.15.181:6813/2706455857", 2024-09-11T08:27:41.199 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addr": "172.21.15.181:6815/2706455857", 2024-09-11T08:27:41.199 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addr": "172.21.15.181:6814/2706455857", 2024-09-11T08:27:41.199 INFO:teuthology.orchestra.run.smithi150.stdout: "state": [ 2024-09-11T08:27:41.199 INFO:teuthology.orchestra.run.smithi150.stdout: "exists", 2024-09-11T08:27:41.200 INFO:teuthology.orchestra.run.smithi150.stdout: "up" 2024-09-11T08:27:41.200 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:27:41.200 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:27:41.200 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-09-11T08:27:41.200 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_xinfo": [ 2024-09-11T08:27:41.201 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.201 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 0, 2024-09-11T08:27:41.201 INFO:teuthology.orchestra.run.smithi150.stdout: "down_stamp": "0.000000", 2024-09-11T08:27:41.201 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_probability": 0, 2024-09-11T08:27:41.201 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_interval": 0, 2024-09-11T08:27:41.202 INFO:teuthology.orchestra.run.smithi150.stdout: "features": 4540138320759226367, 2024-09-11T08:27:41.202 INFO:teuthology.orchestra.run.smithi150.stdout: "old_weight": 0, 2024-09-11T08:27:41.202 INFO:teuthology.orchestra.run.smithi150.stdout: "last_purged_snaps_scrub": "2024-09-11T08:27:22.582809+0000", 2024-09-11T08:27:41.202 INFO:teuthology.orchestra.run.smithi150.stdout: "dead_epoch": 0 2024-09-11T08:27:41.203 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.203 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.203 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 1, 2024-09-11T08:27:41.203 INFO:teuthology.orchestra.run.smithi150.stdout: "down_stamp": "0.000000", 2024-09-11T08:27:41.203 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_probability": 0, 2024-09-11T08:27:41.203 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_interval": 0, 2024-09-11T08:27:41.203 INFO:teuthology.orchestra.run.smithi150.stdout: "features": 4540138320759226367, 2024-09-11T08:27:41.204 INFO:teuthology.orchestra.run.smithi150.stdout: "old_weight": 0, 2024-09-11T08:27:41.204 INFO:teuthology.orchestra.run.smithi150.stdout: "last_purged_snaps_scrub": "2024-09-11T08:27:18.665261+0000", 2024-09-11T08:27:41.204 INFO:teuthology.orchestra.run.smithi150.stdout: "dead_epoch": 0 2024-09-11T08:27:41.204 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.204 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.204 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 2, 2024-09-11T08:27:41.204 INFO:teuthology.orchestra.run.smithi150.stdout: "down_stamp": "0.000000", 2024-09-11T08:27:41.205 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_probability": 0, 2024-09-11T08:27:41.205 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_interval": 0, 2024-09-11T08:27:41.205 INFO:teuthology.orchestra.run.smithi150.stdout: "features": 4540138320759226367, 2024-09-11T08:27:41.205 INFO:teuthology.orchestra.run.smithi150.stdout: "old_weight": 0, 2024-09-11T08:27:41.205 INFO:teuthology.orchestra.run.smithi150.stdout: "last_purged_snaps_scrub": "2024-09-11T08:27:07.879336+0000", 2024-09-11T08:27:41.205 INFO:teuthology.orchestra.run.smithi150.stdout: "dead_epoch": 0 2024-09-11T08:27:41.205 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.205 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.205 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 3, 2024-09-11T08:27:41.205 INFO:teuthology.orchestra.run.smithi150.stdout: "down_stamp": "0.000000", 2024-09-11T08:27:41.205 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_probability": 0, 2024-09-11T08:27:41.205 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_interval": 0, 2024-09-11T08:27:41.206 INFO:teuthology.orchestra.run.smithi150.stdout: "features": 4540138320759226367, 2024-09-11T08:27:41.206 INFO:teuthology.orchestra.run.smithi150.stdout: "old_weight": 0, 2024-09-11T08:27:41.206 INFO:teuthology.orchestra.run.smithi150.stdout: "last_purged_snaps_scrub": "2024-09-11T08:26:23.945545+0000", 2024-09-11T08:27:41.206 INFO:teuthology.orchestra.run.smithi150.stdout: "dead_epoch": 0 2024-09-11T08:27:41.206 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.206 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.206 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 4, 2024-09-11T08:27:41.206 INFO:teuthology.orchestra.run.smithi150.stdout: "down_stamp": "0.000000", 2024-09-11T08:27:41.206 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_probability": 0, 2024-09-11T08:27:41.206 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_interval": 0, 2024-09-11T08:27:41.206 INFO:teuthology.orchestra.run.smithi150.stdout: "features": 4540138320759226367, 2024-09-11T08:27:41.206 INFO:teuthology.orchestra.run.smithi150.stdout: "old_weight": 0, 2024-09-11T08:27:41.206 INFO:teuthology.orchestra.run.smithi150.stdout: "last_purged_snaps_scrub": "2024-09-11T08:27:36.658567+0000", 2024-09-11T08:27:41.206 INFO:teuthology.orchestra.run.smithi150.stdout: "dead_epoch": 0 2024-09-11T08:27:41.207 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.207 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.207 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 5, 2024-09-11T08:27:41.207 INFO:teuthology.orchestra.run.smithi150.stdout: "down_stamp": "0.000000", 2024-09-11T08:27:41.207 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_probability": 0, 2024-09-11T08:27:41.207 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_interval": 0, 2024-09-11T08:27:41.207 INFO:teuthology.orchestra.run.smithi150.stdout: "features": 4540138320759226367, 2024-09-11T08:27:41.207 INFO:teuthology.orchestra.run.smithi150.stdout: "old_weight": 0, 2024-09-11T08:27:41.207 INFO:teuthology.orchestra.run.smithi150.stdout: "last_purged_snaps_scrub": "2024-09-11T08:27:25.485440+0000", 2024-09-11T08:27:41.207 INFO:teuthology.orchestra.run.smithi150.stdout: "dead_epoch": 339 2024-09-11T08:27:41.207 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.207 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.207 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 6, 2024-09-11T08:27:41.207 INFO:teuthology.orchestra.run.smithi150.stdout: "down_stamp": "0.000000", 2024-09-11T08:27:41.208 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_probability": 0, 2024-09-11T08:27:41.208 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_interval": 0, 2024-09-11T08:27:41.208 INFO:teuthology.orchestra.run.smithi150.stdout: "features": 4540138320759226367, 2024-09-11T08:27:41.208 INFO:teuthology.orchestra.run.smithi150.stdout: "old_weight": 0, 2024-09-11T08:27:41.208 INFO:teuthology.orchestra.run.smithi150.stdout: "last_purged_snaps_scrub": "2024-09-11T08:27:34.212957+0000", 2024-09-11T08:27:41.208 INFO:teuthology.orchestra.run.smithi150.stdout: "dead_epoch": 0 2024-09-11T08:27:41.208 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.208 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.208 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 7, 2024-09-11T08:27:41.208 INFO:teuthology.orchestra.run.smithi150.stdout: "down_stamp": "0.000000", 2024-09-11T08:27:41.208 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_probability": 0, 2024-09-11T08:27:41.208 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_interval": 0, 2024-09-11T08:27:41.208 INFO:teuthology.orchestra.run.smithi150.stdout: "features": 4540138320759226367, 2024-09-11T08:27:41.209 INFO:teuthology.orchestra.run.smithi150.stdout: "old_weight": 0, 2024-09-11T08:27:41.209 INFO:teuthology.orchestra.run.smithi150.stdout: "last_purged_snaps_scrub": "2024-09-11T08:27:04.972361+0000", 2024-09-11T08:27:41.209 INFO:teuthology.orchestra.run.smithi150.stdout: "dead_epoch": 139 2024-09-11T08:27:41.209 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:27:41.209 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-09-11T08:27:41.209 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_upmap": [ 2024-09-11T08:27:41.209 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.209 INFO:teuthology.orchestra.run.smithi150.stdout: "pgid": "6.5", 2024-09-11T08:27:41.209 INFO:teuthology.orchestra.run.smithi150.stdout: "osds": [ 2024-09-11T08:27:41.209 INFO:teuthology.orchestra.run.smithi150.stdout: 2, 2024-09-11T08:27:41.209 INFO:teuthology.orchestra.run.smithi150.stdout: 4, 2024-09-11T08:27:41.209 INFO:teuthology.orchestra.run.smithi150.stdout: 7 2024-09-11T08:27:41.209 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:27:41.209 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:27:41.210 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-09-11T08:27:41.210 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_upmap_items": [ 2024-09-11T08:27:41.210 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.210 INFO:teuthology.orchestra.run.smithi150.stdout: "pgid": "4.e", 2024-09-11T08:27:41.210 INFO:teuthology.orchestra.run.smithi150.stdout: "mappings": [ 2024-09-11T08:27:41.210 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.210 INFO:teuthology.orchestra.run.smithi150.stdout: "from": 1, 2024-09-11T08:27:41.210 INFO:teuthology.orchestra.run.smithi150.stdout: "to": 5 2024-09-11T08:27:41.210 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:27:41.210 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:27:41.210 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.210 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.210 INFO:teuthology.orchestra.run.smithi150.stdout: "pgid": "4.14", 2024-09-11T08:27:41.210 INFO:teuthology.orchestra.run.smithi150.stdout: "mappings": [ 2024-09-11T08:27:41.211 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.211 INFO:teuthology.orchestra.run.smithi150.stdout: "from": 1, 2024-09-11T08:27:41.211 INFO:teuthology.orchestra.run.smithi150.stdout: "to": 2 2024-09-11T08:27:41.211 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:27:41.211 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:27:41.211 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.211 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.212 INFO:teuthology.orchestra.run.smithi150.stdout: "pgid": "4.1d", 2024-09-11T08:27:41.212 INFO:teuthology.orchestra.run.smithi150.stdout: "mappings": [ 2024-09-11T08:27:41.212 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.212 INFO:teuthology.orchestra.run.smithi150.stdout: "from": 1, 2024-09-11T08:27:41.212 INFO:teuthology.orchestra.run.smithi150.stdout: "to": 2 2024-09-11T08:27:41.212 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:27:41.212 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:27:41.212 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.212 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.212 INFO:teuthology.orchestra.run.smithi150.stdout: "pgid": "5.3", 2024-09-11T08:27:41.212 INFO:teuthology.orchestra.run.smithi150.stdout: "mappings": [ 2024-09-11T08:27:41.212 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.212 INFO:teuthology.orchestra.run.smithi150.stdout: "from": 0, 2024-09-11T08:27:41.213 INFO:teuthology.orchestra.run.smithi150.stdout: "to": 3 2024-09-11T08:27:41.213 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:27:41.213 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:27:41.213 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.213 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.213 INFO:teuthology.orchestra.run.smithi150.stdout: "pgid": "5.1e", 2024-09-11T08:27:41.213 INFO:teuthology.orchestra.run.smithi150.stdout: "mappings": [ 2024-09-11T08:27:41.213 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:27:41.213 INFO:teuthology.orchestra.run.smithi150.stdout: "from": 0, 2024-09-11T08:27:41.213 INFO:teuthology.orchestra.run.smithi150.stdout: "to": 6 2024-09-11T08:27:41.213 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:27:41.213 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:27:41.214 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:27:41.214 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-09-11T08:27:41.214 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_temp": [], 2024-09-11T08:27:41.214 INFO:teuthology.orchestra.run.smithi150.stdout: "primary_temp": [], 2024-09-11T08:27:41.214 INFO:teuthology.orchestra.run.smithi150.stdout: "blocklist": { 2024-09-11T08:27:41.214 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/2620931070": "2024-09-12T08:08:06.649692+0000", 2024-09-11T08:27:41.214 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/1073581409": "2024-09-12T08:02:45.986151+0000", 2024-09-11T08:27:41.214 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/2363649697": "2024-09-12T08:08:06.649692+0000", 2024-09-11T08:27:41.214 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:6800/603451071": "2024-09-12T08:02:45.986151+0000", 2024-09-11T08:27:41.214 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/1614413548": "2024-09-12T08:02:45.986151+0000", 2024-09-11T08:27:41.214 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/3673218166": "2024-09-12T08:03:08.740828+0000", 2024-09-11T08:27:41.214 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:6800/556498589": "2024-09-12T08:03:08.740828+0000", 2024-09-11T08:27:41.214 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/3666733994": "2024-09-12T08:03:08.740828+0000", 2024-09-11T08:27:41.215 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/3271479010": "2024-09-12T08:03:08.740828+0000", 2024-09-11T08:27:41.215 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/3185731090": "2024-09-12T08:08:06.649692+0000", 2024-09-11T08:27:41.215 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/2539412006": "2024-09-12T08:08:06.649692+0000", 2024-09-11T08:27:41.215 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/2740222408": "2024-09-12T08:02:45.986151+0000", 2024-09-11T08:27:41.215 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:6800/127856313": "2024-09-12T08:08:06.649692+0000", 2024-09-11T08:27:41.215 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/2375121603": "2024-09-12T08:08:06.649692+0000" 2024-09-11T08:27:41.215 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.215 INFO:teuthology.orchestra.run.smithi150.stdout: "range_blocklist": {}, 2024-09-11T08:27:41.215 INFO:teuthology.orchestra.run.smithi150.stdout: "erasure_code_profiles": { 2024-09-11T08:27:41.215 INFO:teuthology.orchestra.run.smithi150.stdout: "default": { 2024-09-11T08:27:41.215 INFO:teuthology.orchestra.run.smithi150.stdout: "crush-failure-domain": "osd", 2024-09-11T08:27:41.215 INFO:teuthology.orchestra.run.smithi150.stdout: "k": "2", 2024-09-11T08:27:41.215 INFO:teuthology.orchestra.run.smithi150.stdout: "m": "1", 2024-09-11T08:27:41.215 INFO:teuthology.orchestra.run.smithi150.stdout: "plugin": "jerasure", 2024-09-11T08:27:41.216 INFO:teuthology.orchestra.run.smithi150.stdout: "technique": "reed_sol_van" 2024-09-11T08:27:41.216 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:27:41.216 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:27:41.216 INFO:teuthology.orchestra.run.smithi150.stdout: "removed_snaps_queue": [], 2024-09-11T08:27:41.216 INFO:teuthology.orchestra.run.smithi150.stdout: "new_removed_snaps": [], 2024-09-11T08:27:41.216 INFO:teuthology.orchestra.run.smithi150.stdout: "new_purged_snaps": [], 2024-09-11T08:27:41.216 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_node_flags": {}, 2024-09-11T08:27:41.216 INFO:teuthology.orchestra.run.smithi150.stdout: "device_class_flags": {}, 2024-09-11T08:27:41.216 INFO:teuthology.orchestra.run.smithi150.stdout: "stretch_mode": { 2024-09-11T08:27:41.216 INFO:teuthology.orchestra.run.smithi150.stdout: "stretch_mode_enabled": false, 2024-09-11T08:27:41.216 INFO:teuthology.orchestra.run.smithi150.stdout: "stretch_bucket_count": 0, 2024-09-11T08:27:41.216 INFO:teuthology.orchestra.run.smithi150.stdout: "degraded_stretch_mode": 0, 2024-09-11T08:27:41.216 INFO:teuthology.orchestra.run.smithi150.stdout: "recovering_stretch_mode": 0, 2024-09-11T08:27:41.217 INFO:teuthology.orchestra.run.smithi150.stdout: "stretch_mode_bucket": 0 2024-09-11T08:27:41.217 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:27:41.217 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-11T08:27:41.262 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:41.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:41.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:41.315 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:27:41.364 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:41.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:41.404 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:41.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:41.505 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:41.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:41.541 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:41.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:41.636 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 61 16 96012 95996 98.3144 68.125 0.0098702 0.0101663 2024-09-11T08:27:41.642 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:41.676 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:41.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:41.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:41.778 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:41.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:41.812 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:41.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:41.913 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:41.945 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:41.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:41.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:41.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:41 smithi150 ceph-mon[29276]: osdmap e406: 8 total, 8 up, 8 in 2024-09-11T08:27:41.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:41 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3675455879' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-11T08:27:41.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:41 smithi150 ceph-mon[29276]: pgmap v923: 148 pgs: 148 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 32 KiB/s rd, 54 MiB/s wr, 916 op/s 2024-09-11T08:27:41.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:41 smithi150 ceph-mon[39612]: osdmap e406: 8 total, 8 up, 8 in 2024-09-11T08:27:41.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:41 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3675455879' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-11T08:27:41.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:41 smithi150 ceph-mon[39612]: pgmap v923: 148 pgs: 148 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 32 KiB/s rd, 54 MiB/s wr, 916 op/s 2024-09-11T08:27:42.046 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:42.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:41 smithi181 ceph-mon[32701]: osdmap e406: 8 total, 8 up, 8 in 2024-09-11T08:27:42.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:41 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3675455879' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-11T08:27:42.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:41 smithi181 ceph-mon[32701]: pgmap v923: 148 pgs: 148 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 32 KiB/s rd, 54 MiB/s wr, 916 op/s 2024-09-11T08:27:42.079 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:42.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:42.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:42.181 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:27:42.213 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:42.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:42.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:27:42.315 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:42.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:42.349 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:42.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:42.451 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:42.482 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:42.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:42.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:42.515 INFO:tasks.thrashosds.thrasher:j is {'epoch': 406, 'fsid': 'c1bddff4-7013-11ef-bcea-c7b262605968', 'created': '2024-09-11T08:02:15.344618+0000', 'modified': '2024-09-11T08:27:40.730800+0000', 'last_up_change': '2024-09-11T08:24:27.333279+0000', 'last_in_change': '2024-09-11T08:19:28.121267+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 20, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 11, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-11T08:05:21.044729+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '255', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-11T08:07:28.786572+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 63, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-11T08:07:29.021286+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-11T08:07:30.334181+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-11T08:07:32.289626+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-11T08:07:34.351861+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 11, 'pool_name': 'unique_pool_4', 'create_time': '2024-09-11T08:26:26.054499+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '388', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': '37b63db9-ed2a-4701-b63d-03ff9885cc0e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 382, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6801', 'nonce': 334255279}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6802', 'nonce': 334255279}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6804', 'nonce': 334255279}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6803', 'nonce': 334255279}]}, 'public_addr': '172.21.15.150:6801/334255279', 'cluster_addr': '172.21.15.150:6802/334255279', 'heartbeat_back_addr': '172.21.15.150:6804/334255279', 'heartbeat_front_addr': '172.21.15.150:6803/334255279', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '80378442-02bc-48aa-b8ca-6113a4092079', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 382, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6805', 'nonce': 1129510619}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6806', 'nonce': 1129510619}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6808', 'nonce': 1129510619}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6807', 'nonce': 1129510619}]}, 'public_addr': '172.21.15.150:6805/1129510619', 'cluster_addr': '172.21.15.150:6806/1129510619', 'heartbeat_back_addr': '172.21.15.150:6808/1129510619', 'heartbeat_front_addr': '172.21.15.150:6807/1129510619', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '0955bb92-36a0-4e9a-98e1-b3a5af89f6d2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 382, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6809', 'nonce': 719975422}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6810', 'nonce': 719975422}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6812', 'nonce': 719975422}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6811', 'nonce': 719975422}]}, 'public_addr': '172.21.15.150:6809/719975422', 'cluster_addr': '172.21.15.150:6810/719975422', 'heartbeat_back_addr': '172.21.15.150:6812/719975422', 'heartbeat_front_addr': '172.21.15.150:6811/719975422', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '1d04d4d3-62e8-4460-8a07-77f7e5c62929', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.611114501953125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 382, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6813', 'nonce': 3101032500}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6814', 'nonce': 3101032500}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6816', 'nonce': 3101032500}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6815', 'nonce': 3101032500}]}, 'public_addr': '172.21.15.150:6813/3101032500', 'cluster_addr': '172.21.15.150:6814/3101032500', 'heartbeat_back_addr': '172.21.15.150:6816/3101032500', 'heartbeat_front_addr': '172.21.15.150:6815/3101032500', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'c5321a3c-d2f0-4440-8475-2c111d261139', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.2719573974609375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 382, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6800', 'nonce': 2567452825}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6801', 'nonce': 2567452825}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6803', 'nonce': 2567452825}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6802', 'nonce': 2567452825}]}, 'public_addr': '172.21.15.181:6800/2567452825', 'cluster_addr': '172.21.15.181:6801/2567452825', 'heartbeat_back_addr': '172.21.15.181:6803/2567452825', 'heartbeat_front_addr': '172.21.15.181:6802/2567452825', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '1262b2fe-f253-4a26-bfd4-c9dffb697c77', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 38, 'last_clean_end': 339, 'up_from': 353, 'up_thru': 353, 'down_at': 340, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6804', 'nonce': 1821561027}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6805', 'nonce': 1821561027}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6807', 'nonce': 1821561027}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6806', 'nonce': 1821561027}]}, 'public_addr': '172.21.15.181:6804/1821561027', 'cluster_addr': '172.21.15.181:6805/1821561027', 'heartbeat_back_addr': '172.21.15.181:6807/1821561027', 'heartbeat_front_addr': '172.21.15.181:6806/1821561027', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '7facca22-b3f1-4c20-833e-e3db798a08c4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 382, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6808', 'nonce': 1789880331}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6809', 'nonce': 1789880331}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6811', 'nonce': 1789880331}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6810', 'nonce': 1789880331}]}, 'public_addr': '172.21.15.181:6808/1789880331', 'cluster_addr': '172.21.15.181:6809/1789880331', 'heartbeat_back_addr': '172.21.15.181:6811/1789880331', 'heartbeat_front_addr': '172.21.15.181:6810/1789880331', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '6ac1b06a-79bd-43ca-b91e-fcb0f4033a34', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 53, 'last_clean_end': 139, 'up_from': 168, 'up_thru': 382, 'down_at': 140, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6812', 'nonce': 2706455857}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6813', 'nonce': 2706455857}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6815', 'nonce': 2706455857}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6814', 'nonce': 2706455857}]}, 'public_addr': '172.21.15.181:6812/2706455857', 'cluster_addr': '172.21.15.181:6813/2706455857', 'heartbeat_back_addr': '172.21.15.181:6815/2706455857', 'heartbeat_front_addr': '172.21.15.181:6814/2706455857', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T08:27:22.582809+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T08:27:18.665261+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T08:27:07.879336+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T08:26:23.945545+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T08:27:36.658567+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T08:27:25.485440+0000', 'dead_epoch': 339}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T08:27:34.212957+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T08:27:04.972361+0000', 'dead_epoch': 139}], 'pg_upmap': [{'pgid': '6.5', 'osds': [2, 4, 7]}], 'pg_upmap_items': [{'pgid': '4.e', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.14', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1d', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '5.3', 'mappings': [{'from': 0, 'to': 3}]}, {'pgid': '5.1e', 'mappings': [{'from': 0, 'to': 6}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.150:0/2620931070': '2024-09-12T08:08:06.649692+0000', '172.21.15.150:0/1073581409': '2024-09-12T08:02:45.986151+0000', '172.21.15.150:0/2363649697': '2024-09-12T08:08:06.649692+0000', '172.21.15.150:6800/603451071': '2024-09-12T08:02:45.986151+0000', '172.21.15.150:0/1614413548': '2024-09-12T08:02:45.986151+0000', '172.21.15.150:0/3673218166': '2024-09-12T08:03:08.740828+0000', '172.21.15.150:6800/556498589': '2024-09-12T08:03:08.740828+0000', '172.21.15.150:0/3666733994': '2024-09-12T08:03:08.740828+0000', '172.21.15.150:0/3271479010': '2024-09-12T08:03:08.740828+0000', '172.21.15.150:0/3185731090': '2024-09-12T08:08:06.649692+0000', '172.21.15.150:0/2539412006': '2024-09-12T08:08:06.649692+0000', '172.21.15.150:0/2740222408': '2024-09-12T08:02:45.986151+0000', '172.21.15.150:6800/127856313': '2024-09-12T08:08:06.649692+0000', '172.21.15.150:0/2375121603': '2024-09-12T08:08:06.649692+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-11T08:27:42.515 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T08:27:42.584 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:42.596 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:27:42.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:42.637 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:42.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:42.637 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 62 16 97202 97186 97.9283 74.375 0.0172314 0.0102062 2024-09-11T08:27:42.738 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:27:42.767 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:42.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:42.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:27:42.869 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:42.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:42.913 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:42.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:43.014 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:43.044 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:43.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:43.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:43.146 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:43.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:43.182 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:43.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:43.257 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T08:27:43.283 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:27:43.319 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:43.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:43.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:27:43.462 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:43.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:43.514 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:43.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:43.560 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-11T08:27:43.560 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-11T08:27:43.561 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-09-11T08:27:43.561 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-09-11T08:27:43.561 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-11T08:27:43.615 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:27:43.637 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 63 16 98248 98232 97.4116 65.375 0.0140474 0.01026 2024-09-11T08:27:43.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:43.641 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:43.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:27:43.742 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:43.779 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:43.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:43.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:43.872 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:27:43.881 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:43.938 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:43.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:43.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:44.041 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:44.084 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:44.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:44.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:44.186 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:44.213 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:44.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:44.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:44.287 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:44 smithi150 ceph-mon[29276]: pgmap v924: 148 pgs: 148 active+clean; 5.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 26 KiB/s rd, 45 MiB/s wr, 766 op/s 2024-09-11T08:27:44.287 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:44 smithi150 ceph-mon[39612]: pgmap v924: 148 pgs: 148 active+clean; 5.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 26 KiB/s rd, 45 MiB/s wr, 766 op/s 2024-09-11T08:27:44.306 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-11T08:27:44.315 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:44.386 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:44.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:44.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:44.488 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:44.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:44.535 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:44.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:44.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:44 smithi181 ceph-mon[32701]: pgmap v924: 148 pgs: 148 active+clean; 5.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 26 KiB/s rd, 45 MiB/s wr, 766 op/s 2024-09-11T08:27:44.637 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:44.637 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 64 16 99383 99367 96.998 70.9375 0.00846757 0.0103048 2024-09-11T08:27:44.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:44.681 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:44.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:44.701 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:27:44.781 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:27:44.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:44.816 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:44.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:27:44.917 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:44.961 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:44.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:44.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:45.063 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:27:45.094 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:45.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:45.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:27:45.196 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:27:45.250 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:45.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:45.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:27:45.352 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:27:45.386 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:45.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:45.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:27:45.488 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:45.513 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:45.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:45.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:45.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:27:45 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:27:45] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:27:45.614 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:27:45.637 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 65 16 100532 100516 96.6105 71.8125 0.0110768 0.0103457 2024-09-11T08:27:45.653 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:45.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:45.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:27:45.755 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:45.787 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:45.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:45.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:45.889 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:27:45.925 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:45.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:45.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:27:46.027 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:27:46.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:46.068 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:46.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:27:46.169 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:46.214 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:46.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:46.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:46.315 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:46.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:46.354 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:46.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:46.455 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:27:46.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:46.477 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:46.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:27:46.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:46 smithi150 ceph-mon[29276]: pgmap v925: 148 pgs: 148 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 53 KiB/s rd, 83 MiB/s wr, 1.41k op/s 2024-09-11T08:27:46.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:46 smithi150 ceph-mon[39612]: pgmap v925: 148 pgs: 148 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 53 KiB/s rd, 83 MiB/s wr, 1.41k op/s 2024-09-11T08:27:46.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:46 smithi181 ceph-mon[32701]: pgmap v925: 148 pgs: 148 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 53 KiB/s rd, 83 MiB/s wr, 1.41k op/s 2024-09-11T08:27:46.579 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:27:46.604 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:46.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:46.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:27:46.637 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 66 16 101489 101473 96.0531 59.8125 0.0155179 0.0104066 2024-09-11T08:27:46.707 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:27:46.731 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:46.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:46.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:27:46.833 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:46.857 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:27:46.861 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-09-11T08:27:46.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:46.873 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:46.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:46.974 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:47.013 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:47.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:47.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:47.115 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:47.144 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:47.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:47.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:47.246 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:47.247 INFO:teuthology.orchestra.run.smithi150.stderr:noscrub is unset 2024-09-11T08:27:47.291 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:47.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:47.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:47.393 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:47.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:47.441 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:47.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:47.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:47 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1399037435' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:27:47.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:47 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:27:47.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:47 smithi150 ceph-mon[29276]: from='client.25874 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:27:47.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:47 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1399037435' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:27:47.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:47 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:27:47.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:47 smithi150 ceph-mon[39612]: from='client.25874 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:27:47.542 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:47.570 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:47 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1399037435' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:27:47.570 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:47 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:27:47.570 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:47 smithi181 ceph-mon[32701]: from='client.25874 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:27:47.581 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:47.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:47.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:47.637 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 67 16 102438 102422 95.5048 59.3125 0.017085 0.0104656 2024-09-11T08:27:47.683 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:47.710 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:47.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:47.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:47.812 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:47.851 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-11T08:27:47.852 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-09-11T08:27:47.852 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-09-11T08:27:47.852 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-09-11T08:27:47.852 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.852 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:96100 9.d 9:b6837c16:::benchmark_data_smithi150_133616_object96099:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e259)", 2024-09-11T08:27:47.852 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:19:08.589253+0000", 2024-09-11T08:27:47.852 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 519.25806377599997, 2024-09-11T08:27:47.852 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.241510726, 2024-09-11T08:27:47.852 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:27:47.852 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:27:47.853 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:27:47.853 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:27:47.853 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:27:47.853 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 96100 2024-09-11T08:27:47.853 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.853 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:27:47.853 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.853 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:27:47.853 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.589253+0000", 2024-09-11T08:27:47.853 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:47.853 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.853 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.853 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:27:47.853 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.589253+0000", 2024-09-11T08:27:47.854 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:47.854 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.854 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.854 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:27:47.854 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.589251+0000", 2024-09-11T08:27:47.854 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:27:47.854 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.854 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.854 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:27:47.854 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.589287+0000", 2024-09-11T08:27:47.854 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5831999999999998e-05 2024-09-11T08:27:47.854 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.854 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.854 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:27:47.855 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.589299+0000", 2024-09-11T08:27:47.855 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2143e-05 2024-09-11T08:27:47.855 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.855 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.855 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:27:47.855 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.589306+0000", 2024-09-11T08:27:47.855 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1800000000000001e-06 2024-09-11T08:27:47.855 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.855 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.855 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:47.855 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.615074+0000", 2024-09-11T08:27:47.855 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.025768183 2024-09-11T08:27:47.855 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.856 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.856 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:27:47.856 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.615093+0000", 2024-09-11T08:27:47.856 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9256e-05 2024-09-11T08:27:47.856 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.856 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.856 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:47.856 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:09.593235+0000", 2024-09-11T08:27:47.856 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.97814237000000004 2024-09-11T08:27:47.856 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.856 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.856 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:27:47.856 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:09.593247+0000", 2024-09-11T08:27:47.857 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1423999999999999e-05 2024-09-11T08:27:47.857 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.857 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.857 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:47.857 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:10.734531+0000", 2024-09-11T08:27:47.857 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.141284204 2024-09-11T08:27:47.857 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.857 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.857 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:27:47.857 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:10.734645+0000", 2024-09-11T08:27:47.857 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000114392 2024-09-11T08:27:47.857 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.857 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.857 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,4,5", 2024-09-11T08:27:47.858 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:10.735075+0000", 2024-09-11T08:27:47.858 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00042998299999999999 2024-09-11T08:27:47.858 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.858 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.858 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:27:47.858 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:10.745751+0000", 2024-09-11T08:27:47.858 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.010675326000000001 2024-09-11T08:27:47.858 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.858 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.858 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:47.858 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:10.745850+0000", 2024-09-11T08:27:47.859 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9628000000000002e-05 2024-09-11T08:27:47.859 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.859 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.859 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:47.859 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:10.745917+0000", 2024-09-11T08:27:47.859 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6192999999999995e-05 2024-09-11T08:27:47.859 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.859 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.859 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:47.859 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:10.830643+0000", 2024-09-11T08:27:47.859 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.084726478999999993 2024-09-11T08:27:47.859 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.859 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.859 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:27:47.860 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:10.830687+0000", 2024-09-11T08:27:47.860 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4487999999999998e-05 2024-09-11T08:27:47.860 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.860 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.860 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:27:47.860 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:10.830764+0000", 2024-09-11T08:27:47.860 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6274999999999997e-05 2024-09-11T08:27:47.860 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:47.860 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:27:47.861 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:47.861 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.861 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.861 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:96096 9.7 9:ee1b318e:::benchmark_data_smithi150_133616_object96095:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e260)", 2024-09-11T08:27:47.861 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:19:08.595939+0000", 2024-09-11T08:27:47.861 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 519.25137830300002, 2024-09-11T08:27:47.861 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.187328741, 2024-09-11T08:27:47.861 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:27:47.861 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-11T08:27:47.861 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:27:47.861 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:27:47.862 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:27:47.862 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 96096 2024-09-11T08:27:47.862 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.862 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:27:47.862 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.862 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:27:47.862 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.595939+0000", 2024-09-11T08:27:47.862 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:47.862 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.863 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.863 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:27:47.863 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.595939+0000", 2024-09-11T08:27:47.863 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:47.863 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.863 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.863 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:27:47.863 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.595937+0000", 2024-09-11T08:27:47.863 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999981 2024-09-11T08:27:47.863 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.863 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.863 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:27:47.863 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.595973+0000", 2024-09-11T08:27:47.863 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5794000000000003e-05 2024-09-11T08:27:47.863 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.864 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.864 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:27:47.864 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.596800+0000", 2024-09-11T08:27:47.864 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00082782999999999999 2024-09-11T08:27:47.864 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.864 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.864 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:27:47.864 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.596810+0000", 2024-09-11T08:27:47.864 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7969999999999995e-06 2024-09-11T08:27:47.864 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.865 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.865 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:47.865 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.605998+0000", 2024-09-11T08:27:47.865 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0091875180000000004 2024-09-11T08:27:47.865 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.865 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.865 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:27:47.866 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.606008+0000", 2024-09-11T08:27:47.866 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0223e-05 2024-09-11T08:27:47.866 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.866 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.866 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:47.866 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:09.594589+0000", 2024-09-11T08:27:47.866 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.988581402 2024-09-11T08:27:47.866 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.866 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.866 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:27:47.866 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:09.594601+0000", 2024-09-11T08:27:47.866 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1715999999999999e-05 2024-09-11T08:27:47.866 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.866 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.867 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:47.867 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:10.782301+0000", 2024-09-11T08:27:47.867 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1876996 2024-09-11T08:27:47.867 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.867 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.867 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:27:47.867 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:10.783267+0000", 2024-09-11T08:27:47.867 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00096663700000000001 2024-09-11T08:27:47.867 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:47.867 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:27:47.867 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:47.867 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.867 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.867 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:96109 9.d 9:b94774bb:::benchmark_data_smithi150_133616_object96108:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-11T08:27:47.868 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:19:08.614378+0000", 2024-09-11T08:27:47.868 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 519.23293921699997, 2024-09-11T08:27:47.868 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2165278509999999, 2024-09-11T08:27:47.868 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:27:47.868 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:27:47.868 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:27:47.868 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:27:47.868 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:27:47.868 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 96109 2024-09-11T08:27:47.868 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.868 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:27:47.868 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.868 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:27:47.869 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.614378+0000", 2024-09-11T08:27:47.869 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:47.869 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.869 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.869 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:27:47.869 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.614378+0000", 2024-09-11T08:27:47.869 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:47.869 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.869 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.869 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:27:47.869 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.614376+0000", 2024-09-11T08:27:47.869 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999981 2024-09-11T08:27:47.869 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.869 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.870 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:27:47.870 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.614423+0000", 2024-09-11T08:27:47.870 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6887e-05 2024-09-11T08:27:47.870 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.870 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.870 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:27:47.870 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.614433+0000", 2024-09-11T08:27:47.870 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.049e-05 2024-09-11T08:27:47.870 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.870 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.871 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:27:47.871 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.614440+0000", 2024-09-11T08:27:47.871 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7129999999999999e-06 2024-09-11T08:27:47.871 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.871 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.871 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:47.871 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.615331+0000", 2024-09-11T08:27:47.871 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00089052399999999996 2024-09-11T08:27:47.871 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.871 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.871 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:27:47.871 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:08.615370+0000", 2024-09-11T08:27:47.871 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9857999999999997e-05 2024-09-11T08:27:47.871 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.872 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.872 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:47.872 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:09.593268+0000", 2024-09-11T08:27:47.872 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.97789712600000001 2024-09-11T08:27:47.872 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.872 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.872 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:27:47.872 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:09.593274+0000", 2024-09-11T08:27:47.872 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7000000000000002e-06 2024-09-11T08:27:47.872 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.872 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.872 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:47.872 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:10.740372+0000", 2024-09-11T08:27:47.872 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.147097665 2024-09-11T08:27:47.873 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.873 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.873 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:27:47.873 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:10.740532+0000", 2024-09-11T08:27:47.873 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016001500000000001 2024-09-11T08:27:47.873 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.873 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.873 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,4,5", 2024-09-11T08:27:47.873 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:10.740723+0000", 2024-09-11T08:27:47.873 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00019085900000000001 2024-09-11T08:27:47.873 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.873 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.873 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:27:47.873 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:10.745859+0000", 2024-09-11T08:27:47.874 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0051359630000000003 2024-09-11T08:27:47.874 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.874 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.874 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:47.874 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:10.745892+0000", 2024-09-11T08:27:47.874 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3126000000000001e-05 2024-09-11T08:27:47.874 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.874 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.874 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:47.874 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:10.745963+0000", 2024-09-11T08:27:47.874 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1587000000000004e-05 2024-09-11T08:27:47.874 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.874 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.874 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:47.874 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:10.830805+0000", 2024-09-11T08:27:47.875 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.084841406999999994 2024-09-11T08:27:47.875 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.875 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.875 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:27:47.875 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:10.830843+0000", 2024-09-11T08:27:47.875 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7951999999999998e-05 2024-09-11T08:27:47.875 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.875 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.875 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:27:47.875 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:10.830905+0000", 2024-09-11T08:27:47.875 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2671999999999999e-05 2024-09-11T08:27:47.875 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:47.875 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:27:47.875 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:47.876 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.876 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.876 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:115604 9.d 9:b8615e71:::benchmark_last_metadata:head [writefull 0~24] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e277)", 2024-09-11T08:27:47.876 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:19:42.782701+0000", 2024-09-11T08:27:47.876 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 485.06461592300002, 2024-09-11T08:27:47.876 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0163880489999999, 2024-09-11T08:27:47.876 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:27:47.876 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-11T08:27:47.876 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:27:47.876 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:27:47.876 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:27:47.876 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 115604 2024-09-11T08:27:47.876 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.877 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:27:47.877 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.877 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:27:47.877 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:42.782701+0000", 2024-09-11T08:27:47.877 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:47.877 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.877 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.877 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:27:47.877 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:42.782701+0000", 2024-09-11T08:27:47.877 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:47.877 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.877 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.877 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:27:47.877 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:42.782699+0000", 2024-09-11T08:27:47.877 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999981 2024-09-11T08:27:47.878 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.878 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.878 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:27:47.878 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:42.782711+0000", 2024-09-11T08:27:47.878 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1805e-05 2024-09-11T08:27:47.878 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.878 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.878 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:27:47.878 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:42.782732+0000", 2024-09-11T08:27:47.878 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0863e-05 2024-09-11T08:27:47.878 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.878 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.878 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:27:47.878 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:42.782740+0000", 2024-09-11T08:27:47.879 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.9299999999999992e-06 2024-09-11T08:27:47.879 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.879 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.879 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:47.879 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:42.782800+0000", 2024-09-11T08:27:47.879 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9763999999999999e-05 2024-09-11T08:27:47.879 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.879 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.879 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:27:47.879 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:42.782925+0000", 2024-09-11T08:27:47.879 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012444800000000001 2024-09-11T08:27:47.879 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.879 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.880 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,4,5", 2024-09-11T08:27:47.880 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:42.783031+0000", 2024-09-11T08:27:47.880 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000106807 2024-09-11T08:27:47.880 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.880 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.880 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:47.880 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:42.786504+0000", 2024-09-11T08:27:47.880 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003472709 2024-09-11T08:27:47.880 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.880 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.880 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:47.880 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:42.787835+0000", 2024-09-11T08:27:47.880 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0013306469999999999 2024-09-11T08:27:47.880 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.881 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.881 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:27:47.881 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:42.787875+0000", 2024-09-11T08:27:47.881 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0329e-05 2024-09-11T08:27:47.881 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.881 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.881 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:47.881 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:43.799012+0000", 2024-09-11T08:27:47.882 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0111368759999999 2024-09-11T08:27:47.882 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.882 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.882 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:27:47.882 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:19:43.799089+0000", 2024-09-11T08:27:47.882 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6925e-05 2024-09-11T08:27:47.882 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:47.882 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:27:47.882 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:47.882 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.882 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.882 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:212763 9.3 9:ce43ae85:::benchmark_data_smithi150_133616_object97157:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-11T08:27:47.883 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:21:06.528719+0000", 2024-09-11T08:27:47.883 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 401.31859787899998, 2024-09-11T08:27:47.883 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.27869297399999998, 2024-09-11T08:27:47.883 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:27:47.883 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:27:47.883 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:27:47.883 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:27:47.883 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:27:47.883 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 212763 2024-09-11T08:27:47.883 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.883 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:27:47.883 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.883 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:27:47.883 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.528719+0000", 2024-09-11T08:27:47.884 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:47.884 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.884 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.884 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:27:47.884 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.528719+0000", 2024-09-11T08:27:47.884 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:47.884 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.884 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.884 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:27:47.884 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.528718+0000", 2024-09-11T08:27:47.884 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:27:47.884 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.884 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.885 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:27:47.885 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.528724+0000", 2024-09-11T08:27:47.885 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0059999999999998e-06 2024-09-11T08:27:47.885 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.885 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.885 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:27:47.885 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.528737+0000", 2024-09-11T08:27:47.885 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3701e-05 2024-09-11T08:27:47.885 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.885 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.885 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:27:47.885 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.528743+0000", 2024-09-11T08:27:47.885 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5480000000000001e-06 2024-09-11T08:27:47.885 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.886 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.886 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:47.886 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.529022+0000", 2024-09-11T08:27:47.886 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000279499 2024-09-11T08:27:47.886 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.886 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.886 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:27:47.886 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.529119+0000", 2024-09-11T08:27:47.886 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.6080000000000002e-05 2024-09-11T08:27:47.886 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.886 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.886 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-09-11T08:27:47.886 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.529218+0000", 2024-09-11T08:27:47.886 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9675999999999999e-05 2024-09-11T08:27:47.887 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.887 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.887 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:27:47.887 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.537360+0000", 2024-09-11T08:27:47.887 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0081422119999999994 2024-09-11T08:27:47.887 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.887 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.887 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:47.887 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.616690+0000", 2024-09-11T08:27:47.887 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.079329685999999996 2024-09-11T08:27:47.887 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.887 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.887 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:47.887 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.807330+0000", 2024-09-11T08:27:47.887 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.190640169 2024-09-11T08:27:47.888 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.888 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.888 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:27:47.888 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.807372+0000", 2024-09-11T08:27:47.888 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1925000000000003e-05 2024-09-11T08:27:47.888 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.888 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.888 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:27:47.888 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.807412+0000", 2024-09-11T08:27:47.888 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9736e-05 2024-09-11T08:27:47.888 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:47.888 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:27:47.888 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:47.889 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.889 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.889 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15369.0:212766 9.3 9:ca00ffc2:::benchmark_data_smithi150_133616_object97160:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-11T08:27:47.889 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:21:06.528973+0000", 2024-09-11T08:27:47.889 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 401.31834345800002, 2024-09-11T08:27:47.889 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.28338277899999997, 2024-09-11T08:27:47.889 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:27:47.889 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:27:47.889 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:27:47.889 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15369", 2024-09-11T08:27:47.889 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:27:47.889 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 212766 2024-09-11T08:27:47.889 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.889 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:27:47.890 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.890 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:27:47.890 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.528973+0000", 2024-09-11T08:27:47.890 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:47.890 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.890 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.890 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:27:47.890 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.528973+0000", 2024-09-11T08:27:47.890 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:47.890 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.890 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.890 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:27:47.890 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.528973+0000", 2024-09-11T08:27:47.890 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:27:47.891 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.891 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.891 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:27:47.891 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.528977+0000", 2024-09-11T08:27:47.891 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.262e-06 2024-09-11T08:27:47.891 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.891 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.891 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:27:47.891 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.528983+0000", 2024-09-11T08:27:47.891 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2169999999999999e-06 2024-09-11T08:27:47.891 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.891 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.891 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:27:47.891 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.528988+0000", 2024-09-11T08:27:47.892 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5419999999999999e-06 2024-09-11T08:27:47.892 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.892 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.892 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:47.892 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.532155+0000", 2024-09-11T08:27:47.892 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0031670380000000001 2024-09-11T08:27:47.892 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.892 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.892 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:27:47.892 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.532257+0000", 2024-09-11T08:27:47.892 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000102718 2024-09-11T08:27:47.892 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.892 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.892 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-09-11T08:27:47.893 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.532366+0000", 2024-09-11T08:27:47.893 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000108285 2024-09-11T08:27:47.893 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.893 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.893 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:27:47.893 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.539502+0000", 2024-09-11T08:27:47.893 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0071359320000000002 2024-09-11T08:27:47.893 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.893 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.893 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:47.893 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.616774+0000", 2024-09-11T08:27:47.893 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.077272399000000005 2024-09-11T08:27:47.893 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.893 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.894 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:47.894 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.812218+0000", 2024-09-11T08:27:47.894 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19544420500000001 2024-09-11T08:27:47.894 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.894 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.894 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:27:47.894 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.812309+0000", 2024-09-11T08:27:47.895 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0586000000000003e-05 2024-09-11T08:27:47.895 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.895 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.895 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:27:47.895 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:21:06.812356+0000", 2024-09-11T08:27:47.895 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7338e-05 2024-09-11T08:27:47.896 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:47.896 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:27:47.896 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:47.896 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.896 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.896 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15636.0:120934 10.1 10:8e017b5c:::benchmark_data_smithi150_158067_object120933:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e337)", 2024-09-11T08:27:47.897 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:23:21.679741+0000", 2024-09-11T08:27:47.897 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 266.16757567600001, 2024-09-11T08:27:47.897 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0085350930000001, 2024-09-11T08:27:47.897 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:27:47.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:47.899 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:47.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:47.901 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-11T08:27:47.901 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:27:47.902 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15636", 2024-09-11T08:27:47.902 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:27:47.902 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 120934 2024-09-11T08:27:47.902 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.902 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:27:47.902 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.903 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:27:47.903 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.679741+0000", 2024-09-11T08:27:47.903 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:47.903 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.903 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.903 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:27:47.904 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.679741+0000", 2024-09-11T08:27:47.904 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:47.904 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.904 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.904 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:27:47.905 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.679739+0000", 2024-09-11T08:27:47.905 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999981 2024-09-11T08:27:47.905 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.905 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.905 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:27:47.905 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.679792+0000", 2024-09-11T08:27:47.906 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3069999999999998e-05 2024-09-11T08:27:47.906 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.906 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.906 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:27:47.906 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.679818+0000", 2024-09-11T08:27:47.906 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5236999999999999e-05 2024-09-11T08:27:47.907 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.907 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.907 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:27:47.907 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.679826+0000", 2024-09-11T08:27:47.907 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2500000000000006e-06 2024-09-11T08:27:47.907 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.908 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.908 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:47.908 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.679933+0000", 2024-09-11T08:27:47.908 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000107398 2024-09-11T08:27:47.908 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.909 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.909 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:27:47.909 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.679944+0000", 2024-09-11T08:27:47.909 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0567000000000001e-05 2024-09-11T08:27:47.909 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.909 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.910 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:47.910 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:22.686428+0000", 2024-09-11T08:27:47.910 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.006484267 2024-09-11T08:27:47.910 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.910 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.910 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:27:47.911 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:22.688276+0000", 2024-09-11T08:27:47.911 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001848092 2024-09-11T08:27:47.911 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:47.911 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:27:47.911 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:47.911 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.912 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.912 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15636.0:120935 10.1 10:8d74a371:::benchmark_data_smithi150_158067_object120934:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e337)", 2024-09-11T08:27:47.912 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:23:21.679844+0000", 2024-09-11T08:27:47.912 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 266.16747307499998, 2024-09-11T08:27:47.912 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.008638562, 2024-09-11T08:27:47.912 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:27:47.913 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-11T08:27:47.913 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:27:47.913 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15636", 2024-09-11T08:27:47.913 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:27:47.913 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 120935 2024-09-11T08:27:47.914 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.914 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:27:47.914 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.914 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:27:47.914 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.679844+0000", 2024-09-11T08:27:47.914 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:47.914 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.915 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.915 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:27:47.915 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.679844+0000", 2024-09-11T08:27:47.915 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:47.915 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.915 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.915 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:27:47.916 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.679843+0000", 2024-09-11T08:27:47.916 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-09-11T08:27:47.916 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.916 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.916 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:27:47.917 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.679873+0000", 2024-09-11T08:27:47.917 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0258000000000001e-05 2024-09-11T08:27:47.917 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.917 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.917 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:27:47.917 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.679881+0000", 2024-09-11T08:27:47.917 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8469999999999996e-06 2024-09-11T08:27:47.917 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.917 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.917 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:27:47.918 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.679886+0000", 2024-09-11T08:27:47.918 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8019999999999999e-06 2024-09-11T08:27:47.918 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.918 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.918 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:47.918 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.679970+0000", 2024-09-11T08:27:47.918 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3916000000000001e-05 2024-09-11T08:27:47.918 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.918 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.918 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:27:47.918 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.679977+0000", 2024-09-11T08:27:47.918 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8260000000000004e-06 2024-09-11T08:27:47.918 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.918 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.918 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:47.919 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:22.688322+0000", 2024-09-11T08:27:47.919 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.008345491 2024-09-11T08:27:47.919 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.919 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.919 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:27:47.919 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:22.688482+0000", 2024-09-11T08:27:47.919 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000160023 2024-09-11T08:27:47.919 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:47.919 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:27:47.919 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:47.919 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.919 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.919 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15636.0:120953 10.1 10:89c0995d:::benchmark_data_smithi150_158067_object120952:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-09-11T08:27:47.919 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:23:21.685662+0000", 2024-09-11T08:27:47.919 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 266.16165434300001, 2024-09-11T08:27:47.920 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0197454850000001, 2024-09-11T08:27:47.920 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:27:47.920 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:27:47.920 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:27:47.920 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15636", 2024-09-11T08:27:47.920 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:27:47.920 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 120953 2024-09-11T08:27:47.920 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.920 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:27:47.920 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.920 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:27:47.920 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.685662+0000", 2024-09-11T08:27:47.920 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:47.921 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.921 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.921 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:27:47.921 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.685662+0000", 2024-09-11T08:27:47.921 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:47.921 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.921 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.921 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:27:47.921 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.685661+0000", 2024-09-11T08:27:47.921 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999981 2024-09-11T08:27:47.921 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.921 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.921 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:27:47.922 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.685712+0000", 2024-09-11T08:27:47.922 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1220000000000001e-05 2024-09-11T08:27:47.922 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.922 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.922 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:27:47.922 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.685740+0000", 2024-09-11T08:27:47.922 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7787999999999999e-05 2024-09-11T08:27:47.922 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.922 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.922 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:27:47.922 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.685748+0000", 2024-09-11T08:27:47.922 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8709999999999997e-06 2024-09-11T08:27:47.922 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.923 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.923 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:47.923 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.686890+0000", 2024-09-11T08:27:47.923 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0011420110000000001 2024-09-11T08:27:47.923 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.923 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.923 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:27:47.923 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.686913+0000", 2024-09-11T08:27:47.923 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3629999999999999e-05 2024-09-11T08:27:47.923 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.923 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.923 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:47.923 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:22.688514+0000", 2024-09-11T08:27:47.924 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0016008890000001 2024-09-11T08:27:47.924 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.924 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.924 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:27:47.924 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:22.689007+0000", 2024-09-11T08:27:47.924 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00049329199999999999 2024-09-11T08:27:47.924 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.924 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.924 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,4", 2024-09-11T08:27:47.924 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:22.689134+0000", 2024-09-11T08:27:47.924 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012621400000000001 2024-09-11T08:27:47.924 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.924 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.925 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:27:47.925 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:22.697204+0000", 2024-09-11T08:27:47.925 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0080706409999999999 2024-09-11T08:27:47.925 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.925 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.925 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:47.925 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:22.705272+0000", 2024-09-11T08:27:47.925 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0080673870000000005 2024-09-11T08:27:47.925 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.925 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.926 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:47.926 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:22.705323+0000", 2024-09-11T08:27:47.926 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1335999999999998e-05 2024-09-11T08:27:47.926 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.926 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.926 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:27:47.926 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:22.705372+0000", 2024-09-11T08:27:47.926 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9172999999999998e-05 2024-09-11T08:27:47.926 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.926 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.926 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:27:47.927 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:22.705408+0000", 2024-09-11T08:27:47.927 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5753999999999997e-05 2024-09-11T08:27:47.927 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:47.927 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:27:47.927 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:47.927 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.927 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.927 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15636.0:120986 10.1 10:8aa57949:::benchmark_data_smithi150_158067_object120985:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-09-11T08:27:47.927 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:23:21.721714+0000", 2024-09-11T08:27:47.927 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 266.12560246999999, 2024-09-11T08:27:47.927 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.98402379699999998, 2024-09-11T08:27:47.927 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:27:47.928 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:27:47.928 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:27:47.928 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15636", 2024-09-11T08:27:47.928 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:27:47.928 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 120986 2024-09-11T08:27:47.928 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.928 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:27:47.928 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.928 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:27:47.928 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.721714+0000", 2024-09-11T08:27:47.929 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:47.929 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.929 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.929 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:27:47.929 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.721714+0000", 2024-09-11T08:27:47.929 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:47.929 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.929 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.929 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:27:47.929 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.721712+0000", 2024-09-11T08:27:47.929 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999981 2024-09-11T08:27:47.930 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.930 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.930 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:27:47.930 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.721754+0000", 2024-09-11T08:27:47.930 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2039000000000002e-05 2024-09-11T08:27:47.930 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.930 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.930 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:27:47.930 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.721771+0000", 2024-09-11T08:27:47.930 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6436e-05 2024-09-11T08:27:47.930 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.930 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.931 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:27:47.931 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.721778+0000", 2024-09-11T08:27:47.931 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.776e-06 2024-09-11T08:27:47.931 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.931 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.931 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:47.931 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.721825+0000", 2024-09-11T08:27:47.931 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7110999999999997e-05 2024-09-11T08:27:47.931 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.931 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.931 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:27:47.931 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.721834+0000", 2024-09-11T08:27:47.932 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8189999999999998e-06 2024-09-11T08:27:47.932 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.932 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.932 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:47.932 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:22.691821+0000", 2024-09-11T08:27:47.932 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.96998791299999998 2024-09-11T08:27:47.932 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.932 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.932 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:27:47.932 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:22.691961+0000", 2024-09-11T08:27:47.932 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013966900000000001 2024-09-11T08:27:47.932 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.932 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.933 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,4", 2024-09-11T08:27:47.933 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:22.692108+0000", 2024-09-11T08:27:47.933 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014641799999999999 2024-09-11T08:27:47.933 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.933 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.933 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:27:47.933 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:22.702158+0000", 2024-09-11T08:27:47.933 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.010050722 2024-09-11T08:27:47.933 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.933 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.933 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:47.934 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:22.705539+0000", 2024-09-11T08:27:47.934 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003380615 2024-09-11T08:27:47.934 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.934 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.934 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:47.934 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:22.705644+0000", 2024-09-11T08:27:47.934 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010527999999999999 2024-09-11T08:27:47.934 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.934 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.934 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:27:47.936 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:22.705696+0000", 2024-09-11T08:27:47.936 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1368e-05 2024-09-11T08:27:47.936 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.936 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.936 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:27:47.936 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:22.705738+0000", 2024-09-11T08:27:47.936 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2624999999999999e-05 2024-09-11T08:27:47.936 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:47.936 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:27:47.936 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:47.936 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.936 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.937 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15636.0:120987 10.1 10:8bdeb3cc:::benchmark_data_smithi150_158067_object120986:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-09-11T08:27:47.937 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:23:21.721902+0000", 2024-09-11T08:27:47.937 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 266.12541522700002, 2024-09-11T08:27:47.937 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.98398008699999995, 2024-09-11T08:27:47.937 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:27:47.937 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:27:47.937 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:27:47.938 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15636", 2024-09-11T08:27:47.938 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:27:47.938 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 120987 2024-09-11T08:27:47.938 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.938 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:27:47.938 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.938 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:27:47.938 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.721902+0000", 2024-09-11T08:27:47.938 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:47.938 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.938 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.938 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:27:47.938 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.721902+0000", 2024-09-11T08:27:47.938 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:47.939 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.939 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.939 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:27:47.939 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.721901+0000", 2024-09-11T08:27:47.939 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-09-11T08:27:47.939 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.939 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.939 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:27:47.939 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.722008+0000", 2024-09-11T08:27:47.939 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000107246 2024-09-11T08:27:47.939 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.939 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.939 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:27:47.939 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.722015+0000", 2024-09-11T08:27:47.939 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5100000000000004e-06 2024-09-11T08:27:47.940 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.940 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.940 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:27:47.940 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.722018+0000", 2024-09-11T08:27:47.940 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0189999999999998e-06 2024-09-11T08:27:47.940 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.940 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.940 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:47.940 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.722041+0000", 2024-09-11T08:27:47.940 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3263000000000001e-05 2024-09-11T08:27:47.940 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.940 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.940 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:27:47.941 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.722045+0000", 2024-09-11T08:27:47.941 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6169999999999998e-06 2024-09-11T08:27:47.941 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.941 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.941 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:47.941 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:22.694714+0000", 2024-09-11T08:27:47.941 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.97266932699999997 2024-09-11T08:27:47.941 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.941 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.941 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:27:47.941 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:22.694873+0000", 2024-09-11T08:27:47.941 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015907700000000001 2024-09-11T08:27:47.941 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.941 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.942 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,4", 2024-09-11T08:27:47.942 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:22.695038+0000", 2024-09-11T08:27:47.942 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000165267 2024-09-11T08:27:47.942 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.942 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.942 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:27:47.942 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:22.705281+0000", 2024-09-11T08:27:47.942 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.010242261000000001 2024-09-11T08:27:47.942 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.942 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.942 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:47.942 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:22.705582+0000", 2024-09-11T08:27:47.942 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000301589 2024-09-11T08:27:47.943 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.943 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.943 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:47.943 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:22.705798+0000", 2024-09-11T08:27:47.943 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000216209 2024-09-11T08:27:47.943 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.943 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.943 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:27:47.943 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:22.705844+0000", 2024-09-11T08:27:47.943 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5621999999999998e-05 2024-09-11T08:27:47.943 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.943 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.943 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:27:47.943 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:22.705882+0000", 2024-09-11T08:27:47.944 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7586000000000002e-05 2024-09-11T08:27:47.944 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:47.944 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:27:47.944 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:47.944 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.944 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.944 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15636.0:121102 10.1 10:8f88d702:::benchmark_data_smithi150_158067_object121101:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-09-11T08:27:47.944 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:23:21.860106+0000", 2024-09-11T08:27:47.944 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 265.987210339, 2024-09-11T08:27:47.944 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.84596945999999995, 2024-09-11T08:27:47.944 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:27:47.944 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:27:47.944 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:27:47.945 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15636", 2024-09-11T08:27:47.945 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:27:47.945 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 121102 2024-09-11T08:27:47.945 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.945 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:27:47.945 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.945 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:27:47.945 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.860106+0000", 2024-09-11T08:27:47.945 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:47.945 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.945 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.945 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:27:47.945 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.860106+0000", 2024-09-11T08:27:47.945 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:47.946 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.946 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.946 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:27:47.946 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.860105+0000", 2024-09-11T08:27:47.946 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:27:47.946 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.946 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.946 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:27:47.946 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.860149+0000", 2024-09-11T08:27:47.946 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4416000000000003e-05 2024-09-11T08:27:47.946 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.946 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.946 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:27:47.947 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.860163+0000", 2024-09-11T08:27:47.947 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4044e-05 2024-09-11T08:27:47.947 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.947 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.947 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:27:47.947 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.860171+0000", 2024-09-11T08:27:47.947 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.3719999999999999e-06 2024-09-11T08:27:47.947 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.947 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.947 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:47.947 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.860224+0000", 2024-09-11T08:27:47.947 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3127999999999997e-05 2024-09-11T08:27:47.947 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.948 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.948 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:27:47.948 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.860233+0000", 2024-09-11T08:27:47.948 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.9409999999999996e-06 2024-09-11T08:27:47.948 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.948 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.948 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:47.948 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:22.697249+0000", 2024-09-11T08:27:47.948 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.83701648399999995 2024-09-11T08:27:47.948 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.948 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.948 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:27:47.948 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:22.697419+0000", 2024-09-11T08:27:47.948 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017009300000000001 2024-09-11T08:27:47.949 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.949 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.949 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,4", 2024-09-11T08:27:47.949 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:22.697592+0000", 2024-09-11T08:27:47.949 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0001728 2024-09-11T08:27:47.949 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.949 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.949 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:27:47.949 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:22.705291+0000", 2024-09-11T08:27:47.949 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0076991400000000002 2024-09-11T08:27:47.949 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.949 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.949 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:47.950 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:22.705765+0000", 2024-09-11T08:27:47.950 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00047409200000000001 2024-09-11T08:27:47.950 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.950 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.950 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:47.950 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:22.705999+0000", 2024-09-11T08:27:47.950 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00023347600000000001 2024-09-11T08:27:47.950 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.950 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.950 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:27:47.950 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:22.706041+0000", 2024-09-11T08:27:47.950 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1628999999999997e-05 2024-09-11T08:27:47.950 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.950 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.951 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:27:47.951 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:22.706076+0000", 2024-09-11T08:27:47.951 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5337999999999999e-05 2024-09-11T08:27:47.951 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:47.951 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:27:47.951 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:47.951 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.951 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.951 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15636.0:121121 10.1 10:8be9f1f5:::benchmark_data_smithi150_158067_object121120:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-09-11T08:27:47.951 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:23:21.886722+0000", 2024-09-11T08:27:47.951 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 265.96059432300001, 2024-09-11T08:27:47.951 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.82410526699999997, 2024-09-11T08:27:47.951 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:27:47.951 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:27:47.952 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:27:47.952 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15636", 2024-09-11T08:27:47.952 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:27:47.952 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 121121 2024-09-11T08:27:47.952 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.952 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:27:47.952 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.952 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:27:47.952 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.886722+0000", 2024-09-11T08:27:47.952 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:47.952 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.952 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.952 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:27:47.952 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.886722+0000", 2024-09-11T08:27:47.953 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:47.953 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.953 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.953 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:27:47.953 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.886721+0000", 2024-09-11T08:27:47.953 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999981 2024-09-11T08:27:47.953 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.953 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.953 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:27:47.953 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.886775+0000", 2024-09-11T08:27:47.953 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4054000000000001e-05 2024-09-11T08:27:47.953 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.953 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.953 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:27:47.953 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.886791+0000", 2024-09-11T08:27:47.954 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6025e-05 2024-09-11T08:27:47.954 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.954 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.954 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:27:47.954 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.886797+0000", 2024-09-11T08:27:47.954 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6830000000000002e-06 2024-09-11T08:27:47.954 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.954 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.954 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:47.954 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.886859+0000", 2024-09-11T08:27:47.954 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1717000000000006e-05 2024-09-11T08:27:47.954 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.954 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.955 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:27:47.955 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.886869+0000", 2024-09-11T08:27:47.955 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0368e-05 2024-09-11T08:27:47.955 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.955 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.955 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:47.955 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:22.699579+0000", 2024-09-11T08:27:47.955 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.81270997300000003 2024-09-11T08:27:47.955 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.955 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.955 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:27:47.955 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:22.699686+0000", 2024-09-11T08:27:47.955 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000106162 2024-09-11T08:27:47.955 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.956 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.956 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,4", 2024-09-11T08:27:47.956 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:22.699831+0000", 2024-09-11T08:27:47.956 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014557400000000001 2024-09-11T08:27:47.956 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.956 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.956 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:27:47.956 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:22.705419+0000", 2024-09-11T08:27:47.956 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0055875350000000002 2024-09-11T08:27:47.956 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.956 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.956 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:47.956 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:22.705914+0000", 2024-09-11T08:27:47.957 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00049516399999999998 2024-09-11T08:27:47.957 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.957 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.957 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:47.957 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:22.710677+0000", 2024-09-11T08:27:47.957 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0047626769999999999 2024-09-11T08:27:47.957 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.957 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.957 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:27:47.957 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:22.710767+0000", 2024-09-11T08:27:47.957 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0849999999999999e-05 2024-09-11T08:27:47.957 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.957 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.958 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:27:47.958 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:22.710828+0000", 2024-09-11T08:27:47.958 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0367e-05 2024-09-11T08:27:47.958 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:47.958 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:27:47.958 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:47.958 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.958 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.958 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15636.0:121144 10.1 10:8da2e128:::benchmark_data_smithi150_158067_object121143:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-09-11T08:27:47.958 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:23:21.936825+0000", 2024-09-11T08:27:47.958 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 265.910491519, 2024-09-11T08:27:47.958 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.77817572899999998, 2024-09-11T08:27:47.958 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:27:47.958 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:27:47.959 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:27:47.959 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15636", 2024-09-11T08:27:47.959 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:27:47.959 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 121144 2024-09-11T08:27:47.959 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.959 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:27:47.959 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.959 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:27:47.959 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.936825+0000", 2024-09-11T08:27:47.959 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:47.959 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.959 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.959 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:27:47.959 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.936825+0000", 2024-09-11T08:27:47.960 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:47.960 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.960 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.960 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:27:47.960 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.936824+0000", 2024-09-11T08:27:47.960 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:27:47.960 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.960 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.960 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:27:47.960 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.936873+0000", 2024-09-11T08:27:47.960 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8946000000000002e-05 2024-09-11T08:27:47.960 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.960 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.960 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:27:47.961 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.936888+0000", 2024-09-11T08:27:47.961 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5698000000000001e-05 2024-09-11T08:27:47.961 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.961 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.961 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:27:47.961 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.936894+0000", 2024-09-11T08:27:47.961 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7470000000000002e-06 2024-09-11T08:27:47.961 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.961 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.961 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:47.961 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.936939+0000", 2024-09-11T08:27:47.961 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5210000000000003e-05 2024-09-11T08:27:47.961 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.962 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.962 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-11T08:27:47.962 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:21.936946+0000", 2024-09-11T08:27:47.962 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9199999999999998e-06 2024-09-11T08:27:47.962 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.962 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.962 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:47.962 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:22.702199+0000", 2024-09-11T08:27:47.962 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.76525302500000003 2024-09-11T08:27:47.962 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.962 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.962 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:27:47.962 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:22.702316+0000", 2024-09-11T08:27:47.963 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011684300000000001 2024-09-11T08:27:47.963 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.963 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.963 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,4", 2024-09-11T08:27:47.963 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:22.702494+0000", 2024-09-11T08:27:47.963 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000177646 2024-09-11T08:27:47.963 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.963 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.963 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:27:47.963 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:22.705424+0000", 2024-09-11T08:27:47.963 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0029305030000000001 2024-09-11T08:27:47.963 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.963 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.963 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:47.964 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:22.705950+0000", 2024-09-11T08:27:47.964 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00052532000000000002 2024-09-11T08:27:47.964 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.964 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.964 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:47.964 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:22.714912+0000", 2024-09-11T08:27:47.964 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0089623059999999997 2024-09-11T08:27:47.964 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.964 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.964 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:27:47.964 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:22.714964+0000", 2024-09-11T08:27:47.964 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2206999999999997e-05 2024-09-11T08:27:47.964 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.965 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.965 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:27:47.965 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:23:22.715001+0000", 2024-09-11T08:27:47.965 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6844000000000002e-05 2024-09-11T08:27:47.965 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:47.965 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:27:47.965 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:47.965 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.965 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.965 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15636.0:257990 10.1 10:809b246f:::benchmark_data_smithi150_158067_object127746:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e369)", 2024-09-11T08:27:47.966 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:25:42.894685+0000", 2024-09-11T08:27:47.966 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 124.952631539, 2024-09-11T08:27:47.966 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.34914292000000002, 2024-09-11T08:27:47.966 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:27:47.966 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:27:47.966 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:27:47.966 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15636", 2024-09-11T08:27:47.966 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:27:47.966 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 257990 2024-09-11T08:27:47.966 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.966 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:27:47.966 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.966 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:27:47.966 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.894685+0000", 2024-09-11T08:27:47.967 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:47.967 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.967 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.967 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:27:47.967 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.894685+0000", 2024-09-11T08:27:47.967 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:47.967 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.967 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.967 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:27:47.967 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.894684+0000", 2024-09-11T08:27:47.967 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:27:47.967 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.967 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.967 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:27:47.968 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.894691+0000", 2024-09-11T08:27:47.968 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7410000000000003e-06 2024-09-11T08:27:47.968 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.968 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.968 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:27:47.968 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.894705+0000", 2024-09-11T08:27:47.968 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4069e-05 2024-09-11T08:27:47.968 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.968 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.968 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:27:47.968 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.894711+0000", 2024-09-11T08:27:47.968 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2689999999999996e-06 2024-09-11T08:27:47.968 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.968 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.969 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:47.969 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.897800+0000", 2024-09-11T08:27:47.969 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0030893829999999998 2024-09-11T08:27:47.969 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.969 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.969 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:27:47.969 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.898061+0000", 2024-09-11T08:27:47.969 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00026054499999999999 2024-09-11T08:27:47.969 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.969 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.969 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,4", 2024-09-11T08:27:47.969 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.898660+0000", 2024-09-11T08:27:47.969 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00059860100000000004 2024-09-11T08:27:47.970 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.970 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.970 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:27:47.970 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.903201+0000", 2024-09-11T08:27:47.970 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0045414560000000001 2024-09-11T08:27:47.970 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.970 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.970 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:47.970 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.912417+0000", 2024-09-11T08:27:47.970 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0092157839999999994 2024-09-11T08:27:47.970 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.970 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.970 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:47.970 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:43.243698+0000", 2024-09-11T08:27:47.971 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.33128102599999998 2024-09-11T08:27:47.971 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.971 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.971 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:27:47.971 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:43.243789+0000", 2024-09-11T08:27:47.971 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0871999999999993e-05 2024-09-11T08:27:47.971 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.971 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.971 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:27:47.971 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:43.243828+0000", 2024-09-11T08:27:47.971 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9490000000000003e-05 2024-09-11T08:27:47.971 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:47.971 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:27:47.972 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:47.972 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.972 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.972 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15636.0:257996 10.c 10:315177d1:::benchmark_data_smithi150_158067_object127752:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e369)", 2024-09-11T08:27:47.972 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:25:42.902453+0000", 2024-09-11T08:27:47.972 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 124.944864252, 2024-09-11T08:27:47.972 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.35815583600000001, 2024-09-11T08:27:47.972 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:27:47.972 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:27:47.972 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:27:47.972 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15636", 2024-09-11T08:27:47.972 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:27:47.972 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 257996 2024-09-11T08:27:47.972 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.973 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:27:47.973 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.973 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:27:47.973 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.902453+0000", 2024-09-11T08:27:47.973 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:47.973 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.973 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.973 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:27:47.973 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.902453+0000", 2024-09-11T08:27:47.973 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:47.973 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.973 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.973 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:27:47.974 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.902451+0000", 2024-09-11T08:27:47.974 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:27:47.974 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.974 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.974 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:27:47.974 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.902460+0000", 2024-09-11T08:27:47.974 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.7609999999999996e-06 2024-09-11T08:27:47.974 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.974 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.974 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:27:47.974 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.902469+0000", 2024-09-11T08:27:47.974 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4690000000000003e-06 2024-09-11T08:27:47.974 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.974 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.975 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:27:47.975 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.902475+0000", 2024-09-11T08:27:47.975 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6060000000000002e-06 2024-09-11T08:27:47.975 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.975 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.975 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:47.975 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.902529+0000", 2024-09-11T08:27:47.975 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3665000000000002e-05 2024-09-11T08:27:47.975 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.975 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.975 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:27:47.975 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.902691+0000", 2024-09-11T08:27:47.975 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016279099999999999 2024-09-11T08:27:47.975 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.976 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.976 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,4", 2024-09-11T08:27:47.976 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.902837+0000", 2024-09-11T08:27:47.976 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014509700000000001 2024-09-11T08:27:47.976 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.976 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.976 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:27:47.976 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.906067+0000", 2024-09-11T08:27:47.976 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0032304550000000001 2024-09-11T08:27:47.976 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.976 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.977 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:47.977 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.906494+0000", 2024-09-11T08:27:47.977 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00042726899999999998 2024-09-11T08:27:47.977 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.977 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.977 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:47.977 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:43.260514+0000", 2024-09-11T08:27:47.977 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.35401971599999998 2024-09-11T08:27:47.977 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.977 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.977 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:27:47.977 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:43.260564+0000", 2024-09-11T08:27:47.977 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9642999999999999e-05 2024-09-11T08:27:47.977 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.978 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.978 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:27:47.978 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:43.260608+0000", 2024-09-11T08:27:47.978 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4750999999999999e-05 2024-09-11T08:27:47.978 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:47.978 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:27:47.978 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:47.978 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.978 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.978 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15636.0:257997 10.1 10:8c348cc4:::benchmark_data_smithi150_158067_object127753:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e369)", 2024-09-11T08:27:47.978 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:25:42.903111+0000", 2024-09-11T08:27:47.978 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 124.94420588, 2024-09-11T08:27:47.978 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.347070657, 2024-09-11T08:27:47.978 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:27:47.979 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:27:47.979 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:27:47.979 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15636", 2024-09-11T08:27:47.979 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:27:47.979 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 257997 2024-09-11T08:27:47.979 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.979 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:27:47.979 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.979 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:27:47.979 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.903111+0000", 2024-09-11T08:27:47.979 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:47.979 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.979 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.979 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:27:47.980 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.903111+0000", 2024-09-11T08:27:47.980 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:47.980 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.980 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.980 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:27:47.980 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.903110+0000", 2024-09-11T08:27:47.980 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-09-11T08:27:47.980 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.980 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.980 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:27:47.980 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.903114+0000", 2024-09-11T08:27:47.980 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1359999999999999e-06 2024-09-11T08:27:47.980 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.980 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.981 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:27:47.981 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.903129+0000", 2024-09-11T08:27:47.981 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4107999999999999e-05 2024-09-11T08:27:47.981 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.981 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.981 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:27:47.981 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.903133+0000", 2024-09-11T08:27:47.981 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5499999999999996e-06 2024-09-11T08:27:47.981 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.981 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.981 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:47.981 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.903246+0000", 2024-09-11T08:27:47.981 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000112979 2024-09-11T08:27:47.981 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.981 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.982 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:27:47.982 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.903364+0000", 2024-09-11T08:27:47.982 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000117621 2024-09-11T08:27:47.982 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.982 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.982 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,4", 2024-09-11T08:27:47.982 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.903494+0000", 2024-09-11T08:27:47.982 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013014400000000001 2024-09-11T08:27:47.982 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.982 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.982 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:27:47.982 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.906295+0000", 2024-09-11T08:27:47.982 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0028015750000000002 2024-09-11T08:27:47.982 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.983 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.983 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:47.983 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.913306+0000", 2024-09-11T08:27:47.983 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0070104640000000001 2024-09-11T08:27:47.984 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.984 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.984 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:47.984 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:43.250047+0000", 2024-09-11T08:27:47.984 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.33674112299999998 2024-09-11T08:27:47.984 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.984 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.984 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:27:47.984 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:43.250103+0000", 2024-09-11T08:27:47.984 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6088999999999998e-05 2024-09-11T08:27:47.984 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.984 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.985 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:27:47.985 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:43.250182+0000", 2024-09-11T08:27:47.985 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8486999999999996e-05 2024-09-11T08:27:47.985 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:47.985 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:27:47.985 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:47.985 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.985 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.985 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15636.0:258002 10.c 10:35a08b1b:::benchmark_data_smithi150_158067_object127758:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e369)", 2024-09-11T08:27:47.985 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:25:42.908922+0000", 2024-09-11T08:27:47.985 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 124.938395214, 2024-09-11T08:27:47.985 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.351848261, 2024-09-11T08:27:47.985 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:27:47.985 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:27:47.986 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:27:47.986 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15636", 2024-09-11T08:27:47.986 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:27:47.986 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 258002 2024-09-11T08:27:47.986 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.986 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:27:47.986 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.986 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:27:47.986 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.908922+0000", 2024-09-11T08:27:47.986 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:47.986 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.986 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.986 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:27:47.987 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.908922+0000", 2024-09-11T08:27:47.987 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:47.987 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.987 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.987 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:27:47.987 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.908920+0000", 2024-09-11T08:27:47.987 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:27:47.987 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.987 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.987 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:27:47.987 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.908926+0000", 2024-09-11T08:27:47.987 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2620000000000004e-06 2024-09-11T08:27:47.987 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.987 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.988 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:27:47.988 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.908937+0000", 2024-09-11T08:27:47.988 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0369e-05 2024-09-11T08:27:47.988 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.988 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.988 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:27:47.988 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.908942+0000", 2024-09-11T08:27:47.988 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1789999999999999e-06 2024-09-11T08:27:47.988 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.988 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.988 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:47.988 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.908985+0000", 2024-09-11T08:27:47.988 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2588e-05 2024-09-11T08:27:47.988 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.989 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.989 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:27:47.989 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.909098+0000", 2024-09-11T08:27:47.989 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011301 2024-09-11T08:27:47.989 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.989 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.989 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,4", 2024-09-11T08:27:47.989 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.909209+0000", 2024-09-11T08:27:47.989 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000111829 2024-09-11T08:27:47.989 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.989 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.989 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:27:47.989 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.915122+0000", 2024-09-11T08:27:47.989 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0059129430000000004 2024-09-11T08:27:47.990 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.990 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.990 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:47.990 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.916442+0000", 2024-09-11T08:27:47.990 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0013196709999999999 2024-09-11T08:27:47.990 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.990 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.990 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:47.990 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:43.260650+0000", 2024-09-11T08:27:47.990 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.34420784100000001 2024-09-11T08:27:47.990 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.990 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.990 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:27:47.991 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:43.260721+0000", 2024-09-11T08:27:47.991 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0956000000000003e-05 2024-09-11T08:27:47.991 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.991 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.991 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:27:47.991 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:43.260770+0000", 2024-09-11T08:27:47.991 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9098000000000003e-05 2024-09-11T08:27:47.991 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:47.991 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:27:47.991 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:47.991 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.991 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.991 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15636.0:258008 10.1 10:8f1dcb6e:::benchmark_data_smithi150_158067_object127764:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e369)", 2024-09-11T08:27:47.991 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:25:42.919343+0000", 2024-09-11T08:27:47.992 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 124.927974259, 2024-09-11T08:27:47.992 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.33164628499999999, 2024-09-11T08:27:47.992 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:27:47.992 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:27:47.992 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:27:47.992 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15636", 2024-09-11T08:27:47.992 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:27:47.992 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 258008 2024-09-11T08:27:47.992 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.992 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:27:47.992 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.992 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:27:47.992 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.919343+0000", 2024-09-11T08:27:47.993 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:47.993 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.993 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.993 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:27:47.993 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.919343+0000", 2024-09-11T08:27:47.993 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:47.993 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.993 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.993 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:27:47.993 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.919341+0000", 2024-09-11T08:27:47.993 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:27:47.993 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.993 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.993 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:27:47.994 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.919347+0000", 2024-09-11T08:27:47.994 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0170000000000002e-06 2024-09-11T08:27:47.994 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.994 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.994 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:27:47.994 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.919357+0000", 2024-09-11T08:27:47.994 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0268999999999999e-05 2024-09-11T08:27:47.994 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.994 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.994 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:27:47.994 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.919363+0000", 2024-09-11T08:27:47.994 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2249999999999997e-06 2024-09-11T08:27:47.994 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.994 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.995 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:47.995 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.919432+0000", 2024-09-11T08:27:47.995 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8993999999999997e-05 2024-09-11T08:27:47.995 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.995 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.995 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:27:47.995 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.919584+0000", 2024-09-11T08:27:47.995 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000151817 2024-09-11T08:27:47.995 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.995 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.995 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,4", 2024-09-11T08:27:47.995 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.919732+0000", 2024-09-11T08:27:47.995 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014749699999999999 2024-09-11T08:27:47.995 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.996 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.996 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:27:47.996 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.923072+0000", 2024-09-11T08:27:47.996 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0033404889999999999 2024-09-11T08:27:47.996 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.996 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.996 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:47.996 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:42.923576+0000", 2024-09-11T08:27:47.996 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00050364099999999998 2024-09-11T08:27:47.996 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.996 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.996 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:47.996 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:43.250914+0000", 2024-09-11T08:27:47.997 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.327337763 2024-09-11T08:27:47.997 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.997 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.997 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:27:47.997 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:43.250956+0000", 2024-09-11T08:27:47.997 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2423999999999999e-05 2024-09-11T08:27:47.997 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.997 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.997 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:27:47.997 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:25:43.250989+0000", 2024-09-11T08:27:47.997 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2752000000000001e-05 2024-09-11T08:27:47.997 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:47.997 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:27:47.997 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:47.998 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.998 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.998 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.25799.0:74373 11.5 11:ad86ba69:::benchmark_data_smithi150_187410_object74372:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-09-11T08:27:47.998 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:27:22.119390+0000", 2024-09-11T08:27:47.998 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 25.727926805999999, 2024-09-11T08:27:47.998 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30173128500000002, 2024-09-11T08:27:47.998 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:27:47.998 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:27:47.998 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-11T08:27:47.998 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.25799", 2024-09-11T08:27:47.998 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.150:0/1403604046", 2024-09-11T08:27:47.998 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 74373 2024-09-11T08:27:47.998 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.999 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:27:47.999 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.999 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:27:47.999 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.119390+0000", 2024-09-11T08:27:47.999 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:47.999 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.999 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.999 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:27:47.999 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.119390+0000", 2024-09-11T08:27:47.999 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:27:47.999 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:47.999 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:47.999 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:27:47.999 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.119389+0000", 2024-09-11T08:27:47.999 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:27:48.000 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:48.000 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:48.000 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:27:48.000 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.119438+0000", 2024-09-11T08:27:48.000 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8940999999999998e-05 2024-09-11T08:27:48.000 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:48.000 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:48.000 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:27:48.000 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.119464+0000", 2024-09-11T08:27:48.000 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6152999999999998e-05 2024-09-11T08:27:48.000 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:48.000 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:48.000 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:27:48.001 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.119469+0000", 2024-09-11T08:27:48.001 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.826e-06 2024-09-11T08:27:48.001 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:48.001 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:48.001 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:27:48.001 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.119513+0000", 2024-09-11T08:27:48.001 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3977999999999998e-05 2024-09-11T08:27:48.001 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:48.001 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:48.001 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:27:48.001 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.119633+0000", 2024-09-11T08:27:48.001 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012016 2024-09-11T08:27:48.001 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:48.001 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:48.002 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 5,6", 2024-09-11T08:27:48.002 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.119718+0000", 2024-09-11T08:27:48.002 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4698999999999997e-05 2024-09-11T08:27:48.002 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:48.002 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:48.002 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-11T08:27:48.002 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.121722+0000", 2024-09-11T08:27:48.002 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0020039989999999998 2024-09-11T08:27:48.002 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:48.002 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:48.002 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:48.002 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.121851+0000", 2024-09-11T08:27:48.002 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012974300000000001 2024-09-11T08:27:48.002 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:48.003 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:48.003 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:27:48.003 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.421025+0000", 2024-09-11T08:27:48.003 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.29917375400000001 2024-09-11T08:27:48.003 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:48.003 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:48.003 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:27:48.003 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.421078+0000", 2024-09-11T08:27:48.003 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3015e-05 2024-09-11T08:27:48.003 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:27:48.003 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:27:48.003 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:27:48.003 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.421121+0000", 2024-09-11T08:27:48.003 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3170000000000002e-05 2024-09-11T08:27:48.004 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:48.004 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:27:48.004 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:48.004 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:27:48.004 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:27:48.004 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-11T08:27:48.004 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:27:48.033 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:48.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:48.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:27:48.134 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:48.178 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:48.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:48.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:48.279 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:48.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:48.316 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:48.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:48.417 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:27:48.451 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:48 smithi181 ceph-mon[32701]: pgmap v926: 148 pgs: 148 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 44 KiB/s rd, 68 MiB/s wr, 1.17k op/s 2024-09-11T08:27:48.451 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:48 smithi181 ceph-mon[32701]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:27:48.451 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:48 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:27:48.451 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:48 smithi181 ceph-mon[32701]: osdmap e407: 8 total, 8 up, 8 in 2024-09-11T08:27:48.451 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:48 smithi181 ceph-mon[32701]: 3.1e scrub starts 2024-09-11T08:27:48.451 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:48 smithi181 ceph-mon[32701]: 3.1e scrub ok 2024-09-11T08:27:48.457 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:48.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:48.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:27:48.559 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:48.568 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:48 smithi150 ceph-mon[29276]: pgmap v926: 148 pgs: 148 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 44 KiB/s rd, 68 MiB/s wr, 1.17k op/s 2024-09-11T08:27:48.568 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:48 smithi150 ceph-mon[29276]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:27:48.568 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:48 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:27:48.568 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:48 smithi150 ceph-mon[29276]: osdmap e407: 8 total, 8 up, 8 in 2024-09-11T08:27:48.568 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:48 smithi150 ceph-mon[29276]: 3.1e scrub starts 2024-09-11T08:27:48.569 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:48 smithi150 ceph-mon[29276]: 3.1e scrub ok 2024-09-11T08:27:48.569 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:48 smithi150 ceph-mon[39612]: pgmap v926: 148 pgs: 148 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 44 KiB/s rd, 68 MiB/s wr, 1.17k op/s 2024-09-11T08:27:48.569 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:48 smithi150 ceph-mon[39612]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:27:48.569 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:48 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:27:48.569 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:48 smithi150 ceph-mon[39612]: osdmap e407: 8 total, 8 up, 8 in 2024-09-11T08:27:48.569 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:48 smithi150 ceph-mon[39612]: 3.1e scrub starts 2024-09-11T08:27:48.569 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:48 smithi150 ceph-mon[39612]: 3.1e scrub ok 2024-09-11T08:27:48.580 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:48.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:48.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:48.637 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 68 16 103300 103284 94.8927 53.875 0.0193816 0.0105337 2024-09-11T08:27:48.683 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:48.716 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:48.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:48.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:48.818 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:48.845 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:48.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:48.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:48.947 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:48.987 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:48.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:48.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:48.989 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:27:48 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:27:48] "GET /metrics HTTP/1.1" 200 224635 "" "Prometheus/2.43.0" 2024-09-11T08:27:49.089 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:27:49.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:49.121 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:49.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:27:49.222 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:49.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:49.253 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:49.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:49.264 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:27:48 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:27:49.354 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:49.390 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:49.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:49.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:49.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:49 smithi150 ceph-mon[29276]: 6.6 scrub starts 2024-09-11T08:27:49.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:49 smithi150 ceph-mon[29276]: 6.6 scrub ok 2024-09-11T08:27:49.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:49 smithi150 ceph-mon[29276]: 6.17 scrub starts 2024-09-11T08:27:49.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:49 smithi150 ceph-mon[29276]: 6.17 scrub ok 2024-09-11T08:27:49.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:49 smithi150 ceph-mon[29276]: 5.11 scrub starts 2024-09-11T08:27:49.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:49 smithi150 ceph-mon[29276]: 5.11 scrub ok 2024-09-11T08:27:49.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:49 smithi150 ceph-mon[29276]: 5.c scrub starts 2024-09-11T08:27:49.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:49 smithi150 ceph-mon[29276]: 5.c scrub ok 2024-09-11T08:27:49.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:49 smithi150 ceph-mon[39612]: 6.6 scrub starts 2024-09-11T08:27:49.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:49 smithi150 ceph-mon[39612]: 6.6 scrub ok 2024-09-11T08:27:49.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:49 smithi150 ceph-mon[39612]: 6.17 scrub starts 2024-09-11T08:27:49.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:49 smithi150 ceph-mon[39612]: 6.17 scrub ok 2024-09-11T08:27:49.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:49 smithi150 ceph-mon[39612]: 5.11 scrub starts 2024-09-11T08:27:49.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:49 smithi150 ceph-mon[39612]: 5.11 scrub ok 2024-09-11T08:27:49.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:49 smithi150 ceph-mon[39612]: 5.c scrub starts 2024-09-11T08:27:49.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:49 smithi150 ceph-mon[39612]: 5.c scrub ok 2024-09-11T08:27:49.492 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:49.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:49.515 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:49.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:49.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:49 smithi181 ceph-mon[32701]: 6.6 scrub starts 2024-09-11T08:27:49.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:49 smithi181 ceph-mon[32701]: 6.6 scrub ok 2024-09-11T08:27:49.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:49 smithi181 ceph-mon[32701]: 6.17 scrub starts 2024-09-11T08:27:49.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:49 smithi181 ceph-mon[32701]: 6.17 scrub ok 2024-09-11T08:27:49.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:49 smithi181 ceph-mon[32701]: 5.11 scrub starts 2024-09-11T08:27:49.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:49 smithi181 ceph-mon[32701]: 5.11 scrub ok 2024-09-11T08:27:49.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:49 smithi181 ceph-mon[32701]: 5.c scrub starts 2024-09-11T08:27:49.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:49 smithi181 ceph-mon[32701]: 5.c scrub ok 2024-09-11T08:27:49.616 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:49.637 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:49.637 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 69 16 104321 104305 94.4423 63.8125 0.00364636 0.0105756 2024-09-11T08:27:49.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:49.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:49.739 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:49.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:49.778 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:49.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:49.879 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:49.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:49.913 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:49.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:50.014 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:50.015 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":925,"stamp":"2024-09-11T08:27:45.154286+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6427515100,"num_objects":98263,"num_object_clones":0,"num_object_copies":294789,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98263,"num_whiteouts":0,"num_read":2468,"num_read_kb":2327,"num_write":99182,"num_write_kb":6277959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":208,"num_bytes_recovered":1776519,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87830,"ondisk_log_size":87830,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":24135312,"kb_used_data":18710080,"kb_used_omap":22,"kb_used_meta":5425129,"kb_avail":725825904,"statfs":{"total":767960285184,"available":743245725696,"internally_reserved":0,"allocated":19159121920,"data_stored":19154985533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22853,"internal_metadata":5555332795},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,4,5,9,8],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":717619200,"num_objects":10950,"num_object_clones":0,"num_object_copies":32850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10950,"num_whiteouts":0,"num_read":439,"num_read_kb":439,"num_write":11236,"num_write_kb":700800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.282194"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":462,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.879499+0000","last_change":"2024-09-11T08:24:28.359613+0000","last_active":"2024-09-11T08:27:36.879499+0000","last_peered":"2024-09-11T08:27:36.879499+0000","last_clean":"2024-09-11T08:27:36.879499+0000","last_became_active":"2024-09-11T08:24:28.358401+0000","last_became_peered":"2024-09-11T08:24:28.358401+0000","last_unstale":"2024-09-11T08:27:36.879499+0000","last_undegraded":"2024-09-11T08:27:36.879499+0000","last_fullsized":"2024-09-11T08:27:36.879499+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:23:29.976662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:23:29.976662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0090761950000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"405'25","reported_seq":475,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:38.301523+0000","last_change":"2024-09-11T08:27:02.000454+0000","last_active":"2024-09-11T08:27:38.301523+0000","last_peered":"2024-09-11T08:27:38.301523+0000","last_clean":"2024-09-11T08:27:38.301523+0000","last_became_active":"2024-09-11T08:07:32.048048+0000","last_became_peered":"2024-09-11T08:07:32.048048+0000","last_unstale":"2024-09-11T08:27:38.301523+0000","last_undegraded":"2024-09-11T08:27:38.301523+0000","last_fullsized":"2024-09-11T08:27:38.301523+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"312'19","last_scrub_stamp":"2024-09-11T08:27:02.000321+0000","last_deep_scrub":"312'19","last_deep_scrub_stamp":"2024-09-11T08:21:59.122122+0000","last_clean_scrub_stamp":"2024-09-11T08:27:02.000321+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:26.414397+0000","scrub_duration":0.0035901290000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":518,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.883918+0000","last_change":"2024-09-11T08:27:03.997743+0000","last_active":"2024-09-11T08:27:36.883918+0000","last_peered":"2024-09-11T08:27:36.883918+0000","last_clean":"2024-09-11T08:27:36.883918+0000","last_became_active":"2024-09-11T08:24:28.372160+0000","last_became_peered":"2024-09-11T08:24:28.372160+0000","last_unstale":"2024-09-11T08:27:36.883918+0000","last_undegraded":"2024-09-11T08:27:36.883918+0000","last_fullsized":"2024-09-11T08:27:36.883918+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:03.997657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:59.531202+0000","last_clean_scrub_stamp":"2024-09-11T08:27:03.997657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:26.638065+0000","scrub_duration":0.0035479840000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":376,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.884144+0000","last_change":"2024-09-11T08:27:06.987974+0000","last_active":"2024-09-11T08:27:36.884144+0000","last_peered":"2024-09-11T08:27:36.884144+0000","last_clean":"2024-09-11T08:27:36.884144+0000","last_became_active":"2024-09-11T08:24:28.354834+0000","last_became_peered":"2024-09-11T08:24:28.354834+0000","last_unstale":"2024-09-11T08:27:36.884144+0000","last_undegraded":"2024-09-11T08:27:36.884144+0000","last_fullsized":"2024-09-11T08:27:36.884144+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:06.987888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:02.470649+0000","last_clean_scrub_stamp":"2024-09-11T08:27:06.987888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:25.277118+0000","scrub_duration":0.0030657430000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":457,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.875338+0000","last_change":"2024-09-11T08:27:19.829769+0000","last_active":"2024-09-11T08:27:36.875338+0000","last_peered":"2024-09-11T08:27:36.875338+0000","last_clean":"2024-09-11T08:27:36.875338+0000","last_became_active":"2024-09-11T08:15:56.521104+0000","last_became_peered":"2024-09-11T08:15:56.521104+0000","last_unstale":"2024-09-11T08:27:36.875338+0000","last_undegraded":"2024-09-11T08:27:36.875338+0000","last_fullsized":"2024-09-11T08:27:36.875338+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:19.829691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:27:19.829691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:32.949746+0000","scrub_duration":0.015796428000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":230,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.882597+0000","last_change":"2024-09-11T08:24:16.568741+0000","last_active":"2024-09-11T08:27:36.882597+0000","last_peered":"2024-09-11T08:27:36.882597+0000","last_clean":"2024-09-11T08:27:36.882597+0000","last_became_active":"2024-09-11T08:17:37.550048+0000","last_became_peered":"2024-09-11T08:17:37.550048+0000","last_unstale":"2024-09-11T08:27:36.882597+0000","last_undegraded":"2024-09-11T08:27:36.882597+0000","last_fullsized":"2024-09-11T08:27:36.882597+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:16.568660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:24:16.568660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.006748937,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"405'46","reported_seq":418,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:39.763250+0000","last_change":"2024-09-11T08:27:10.894267+0000","last_active":"2024-09-11T08:27:39.763250+0000","last_peered":"2024-09-11T08:27:39.763250+0000","last_clean":"2024-09-11T08:27:39.763250+0000","last_became_active":"2024-09-11T08:17:37.553866+0000","last_became_peered":"2024-09-11T08:17:37.553866+0000","last_unstale":"2024-09-11T08:27:39.763250+0000","last_undegraded":"2024-09-11T08:27:39.763250+0000","last_fullsized":"2024-09-11T08:27:39.763250+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'34","last_scrub_stamp":"2024-09-11T08:27:10.894181+0000","last_deep_scrub":"145'22","last_deep_scrub_stamp":"2024-09-11T08:15:57.478870+0000","last_clean_scrub_stamp":"2024-09-11T08:27:10.894181+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:31.391831+0000","scrub_duration":0.014739263000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":101,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.882228+0000","last_change":"2024-09-11T08:27:19.693280+0000","last_active":"2024-09-11T08:27:36.882228+0000","last_peered":"2024-09-11T08:27:36.882228+0000","last_clean":"2024-09-11T08:27:36.882228+0000","last_became_active":"2024-09-11T08:23:22.687041+0000","last_became_peered":"2024-09-11T08:23:22.687041+0000","last_unstale":"2024-09-11T08:27:36.882228+0000","last_undegraded":"2024-09-11T08:27:36.882228+0000","last_fullsized":"2024-09-11T08:27:36.882228+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:19.693176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:18:36.998503+0000","last_clean_scrub_stamp":"2024-09-11T08:27:19.693176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:31.818849+0000","scrub_duration":0.0071117860000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":132,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.883125+0000","last_change":"2024-09-11T08:27:05.913172+0000","last_active":"2024-09-11T08:27:36.883125+0000","last_peered":"2024-09-11T08:27:36.883125+0000","last_clean":"2024-09-11T08:27:36.883125+0000","last_became_active":"2024-09-11T08:24:28.360021+0000","last_became_peered":"2024-09-11T08:24:28.360021+0000","last_unstale":"2024-09-11T08:27:36.883125+0000","last_undegraded":"2024-09-11T08:27:36.883125+0000","last_fullsized":"2024-09-11T08:27:36.883125+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:05.913078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:54.878409+0000","last_clean_scrub_stamp":"2024-09-11T08:27:05.913078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:19.980455+0000","scrub_duration":0.001905669,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"405'21","reported_seq":350,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:39.024879+0000","last_change":"2024-09-11T08:26:02.101012+0000","last_active":"2024-09-11T08:27:39.024879+0000","last_peered":"2024-09-11T08:27:39.024879+0000","last_clean":"2024-09-11T08:27:39.024879+0000","last_became_active":"2024-09-11T08:23:22.687182+0000","last_became_peered":"2024-09-11T08:23:22.687182+0000","last_unstale":"2024-09-11T08:27:39.024879+0000","last_undegraded":"2024-09-11T08:27:39.024879+0000","last_fullsized":"2024-09-11T08:27:39.024879+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'15","last_scrub_stamp":"2024-09-11T08:26:02.100940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:26:02.100940+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028851139999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":347,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.879183+0000","last_change":"2024-09-11T08:26:04.073177+0000","last_active":"2024-09-11T08:27:36.879183+0000","last_peered":"2024-09-11T08:27:36.879183+0000","last_clean":"2024-09-11T08:27:36.879183+0000","last_became_active":"2024-09-11T08:24:28.363985+0000","last_became_peered":"2024-09-11T08:24:28.363985+0000","last_unstale":"2024-09-11T08:27:36.879183+0000","last_undegraded":"2024-09-11T08:27:36.879183+0000","last_fullsized":"2024-09-11T08:27:36.879183+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:04.073122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:20:18.900482+0000","last_clean_scrub_stamp":"2024-09-11T08:26:04.073122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00163221,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":436,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.878647+0000","last_change":"2024-09-11T08:24:26.079306+0000","last_active":"2024-09-11T08:27:36.878647+0000","last_peered":"2024-09-11T08:27:36.878647+0000","last_clean":"2024-09-11T08:27:36.878647+0000","last_became_active":"2024-09-11T08:14:24.792595+0000","last_became_peered":"2024-09-11T08:14:24.792595+0000","last_unstale":"2024-09-11T08:27:36.878647+0000","last_undegraded":"2024-09-11T08:27:36.878647+0000","last_fullsized":"2024-09-11T08:27:36.878647+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:26.079152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:24:26.079152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037087930000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":134,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.885043+0000","last_change":"2024-09-11T08:27:00.877188+0000","last_active":"2024-09-11T08:27:36.885043+0000","last_peered":"2024-09-11T08:27:36.885043+0000","last_clean":"2024-09-11T08:27:36.885043+0000","last_became_active":"2024-09-11T08:24:28.359153+0000","last_became_peered":"2024-09-11T08:24:28.359153+0000","last_unstale":"2024-09-11T08:27:36.885043+0000","last_undegraded":"2024-09-11T08:27:36.885043+0000","last_fullsized":"2024-09-11T08:27:36.885043+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:00.877062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:27:00.877062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:06.321049+0000","scrub_duration":0.0048170670000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"405'11","reported_seq":334,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:38.858144+0000","last_change":"2024-09-11T08:27:07.592807+0000","last_active":"2024-09-11T08:27:38.858144+0000","last_peered":"2024-09-11T08:27:38.858144+0000","last_clean":"2024-09-11T08:27:38.858144+0000","last_became_active":"2024-09-11T08:23:22.681876+0000","last_became_peered":"2024-09-11T08:23:22.681876+0000","last_unstale":"2024-09-11T08:27:38.858144+0000","last_undegraded":"2024-09-11T08:27:38.858144+0000","last_fullsized":"2024-09-11T08:27:38.858144+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'9","last_scrub_stamp":"2024-09-11T08:27:07.592703+0000","last_deep_scrub":"139'7","last_deep_scrub_stamp":"2024-09-11T08:12:24.869318+0000","last_clean_scrub_stamp":"2024-09-11T08:27:07.592703+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:21.681522+0000","scrub_duration":0.0060076349999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":248,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.874895+0000","last_change":"2024-09-11T08:23:33.277787+0000","last_active":"2024-09-11T08:27:36.874895+0000","last_peered":"2024-09-11T08:27:36.874895+0000","last_clean":"2024-09-11T08:27:36.874895+0000","last_became_active":"2024-09-11T08:15:56.526954+0000","last_became_peered":"2024-09-11T08:15:56.526954+0000","last_unstale":"2024-09-11T08:27:36.874895+0000","last_undegraded":"2024-09-11T08:27:36.874895+0000","last_fullsized":"2024-09-11T08:27:36.874895+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:23:33.277734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:23:33.277734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.023249810999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":138,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.884768+0000","last_change":"2024-09-11T08:27:21.243352+0000","last_active":"2024-09-11T08:27:36.884768+0000","last_peered":"2024-09-11T08:27:36.884768+0000","last_clean":"2024-09-11T08:27:36.884768+0000","last_became_active":"2024-09-11T08:24:28.372097+0000","last_became_peered":"2024-09-11T08:24:28.372097+0000","last_unstale":"2024-09-11T08:27:36.884768+0000","last_undegraded":"2024-09-11T08:27:36.884768+0000","last_fullsized":"2024-09-11T08:27:36.884768+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:21.243267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:20:19.927616+0000","last_clean_scrub_stamp":"2024-09-11T08:27:21.243267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:43.008877+0000","scrub_duration":0.0078709639999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":374,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.882458+0000","last_change":"2024-09-11T08:24:29.597394+0000","last_active":"2024-09-11T08:27:36.882458+0000","last_peered":"2024-09-11T08:27:36.882458+0000","last_clean":"2024-09-11T08:27:36.882458+0000","last_became_active":"2024-09-11T08:24:28.368560+0000","last_became_peered":"2024-09-11T08:24:28.368560+0000","last_unstale":"2024-09-11T08:27:36.882458+0000","last_undegraded":"2024-09-11T08:27:36.882458+0000","last_fullsized":"2024-09-11T08:27:36.882458+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:29.597313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:24:29.597313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024078900000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":455,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.879847+0000","last_change":"2024-09-11T08:26:06.019004+0000","last_active":"2024-09-11T08:27:36.879847+0000","last_peered":"2024-09-11T08:27:36.879847+0000","last_clean":"2024-09-11T08:27:36.879847+0000","last_became_active":"2024-09-11T08:15:56.516941+0000","last_became_peered":"2024-09-11T08:15:56.516941+0000","last_unstale":"2024-09-11T08:27:36.879847+0000","last_undegraded":"2024-09-11T08:27:36.879847+0000","last_fullsized":"2024-09-11T08:27:36.879847+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:06.018947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:26:06.018947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001287098,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"405'46","reported_seq":521,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:39.251993+0000","last_change":"2024-09-11T08:27:08.899940+0000","last_active":"2024-09-11T08:27:39.251993+0000","last_peered":"2024-09-11T08:27:39.251993+0000","last_clean":"2024-09-11T08:27:39.251993+0000","last_became_active":"2024-09-11T08:07:32.046363+0000","last_became_peered":"2024-09-11T08:07:32.046363+0000","last_unstale":"2024-09-11T08:27:39.251993+0000","last_undegraded":"2024-09-11T08:27:39.251993+0000","last_fullsized":"2024-09-11T08:27:39.251993+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'34","last_scrub_stamp":"2024-09-11T08:27:08.899726+0000","last_deep_scrub":"107'22","last_deep_scrub_stamp":"2024-09-11T08:16:04.159211+0000","last_clean_scrub_stamp":"2024-09-11T08:27:08.899726+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:19.926838+0000","scrub_duration":0.050618700000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":451,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.885449+0000","last_change":"2024-09-11T08:25:57.566953+0000","last_active":"2024-09-11T08:27:36.885449+0000","last_peered":"2024-09-11T08:27:36.885449+0000","last_clean":"2024-09-11T08:27:36.885449+0000","last_became_active":"2024-09-11T08:07:34.055410+0000","last_became_peered":"2024-09-11T08:07:34.055410+0000","last_unstale":"2024-09-11T08:27:36.885449+0000","last_undegraded":"2024-09-11T08:27:36.885449+0000","last_fullsized":"2024-09-11T08:27:36.885449+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:25:57.566898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:19:28.870211+0000","last_clean_scrub_stamp":"2024-09-11T08:25:57.566898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001280427,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":267,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.882680+0000","last_change":"2024-09-11T08:27:06.905574+0000","last_active":"2024-09-11T08:27:36.882680+0000","last_peered":"2024-09-11T08:27:36.882680+0000","last_clean":"2024-09-11T08:27:36.882680+0000","last_became_active":"2024-09-11T08:24:28.357957+0000","last_became_peered":"2024-09-11T08:24:28.357957+0000","last_unstale":"2024-09-11T08:27:36.882680+0000","last_undegraded":"2024-09-11T08:27:36.882680+0000","last_fullsized":"2024-09-11T08:27:36.882680+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:06.905489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:27:06.905489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:33.245384+0000","scrub_duration":0.0037508659999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":100,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.884686+0000","last_change":"2024-09-11T08:26:04.502886+0000","last_active":"2024-09-11T08:27:36.884686+0000","last_peered":"2024-09-11T08:27:36.884686+0000","last_clean":"2024-09-11T08:27:36.884686+0000","last_became_active":"2024-09-11T08:23:22.686935+0000","last_became_peered":"2024-09-11T08:23:22.686935+0000","last_unstale":"2024-09-11T08:27:36.884686+0000","last_undegraded":"2024-09-11T08:27:36.884686+0000","last_fullsized":"2024-09-11T08:27:36.884686+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:04.502835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:26:04.502835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001566196,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"405'24","reported_seq":309,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:39.786191+0000","last_change":"2024-09-11T08:26:57.767336+0000","last_active":"2024-09-11T08:27:39.786191+0000","last_peered":"2024-09-11T08:27:39.786191+0000","last_clean":"2024-09-11T08:27:39.786191+0000","last_became_active":"2024-09-11T08:17:37.553402+0000","last_became_peered":"2024-09-11T08:17:37.553402+0000","last_unstale":"2024-09-11T08:27:39.786191+0000","last_undegraded":"2024-09-11T08:27:39.786191+0000","last_fullsized":"2024-09-11T08:27:39.786191+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'18","last_scrub_stamp":"2024-09-11T08:26:57.767252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:26:57.767252+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:25.867282+0000","scrub_duration":0.0051635600000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":101,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.875480+0000","last_change":"2024-09-11T08:27:06.590237+0000","last_active":"2024-09-11T08:27:36.875480+0000","last_peered":"2024-09-11T08:27:36.875480+0000","last_clean":"2024-09-11T08:27:36.875480+0000","last_became_active":"2024-09-11T08:23:22.684583+0000","last_became_peered":"2024-09-11T08:23:22.684583+0000","last_unstale":"2024-09-11T08:27:36.875480+0000","last_undegraded":"2024-09-11T08:27:36.875480+0000","last_fullsized":"2024-09-11T08:27:36.875480+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:06.590161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:10:40.110490+0000","last_clean_scrub_stamp":"2024-09-11T08:27:06.590161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:11.903001+0000","scrub_duration":0.00158486,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":423,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.882323+0000","last_change":"2024-09-11T08:25:56.984646+0000","last_active":"2024-09-11T08:27:36.882323+0000","last_peered":"2024-09-11T08:27:36.882323+0000","last_clean":"2024-09-11T08:27:36.882323+0000","last_became_active":"2024-09-11T08:24:28.357198+0000","last_became_peered":"2024-09-11T08:24:28.357198+0000","last_unstale":"2024-09-11T08:27:36.882323+0000","last_undegraded":"2024-09-11T08:27:36.882323+0000","last_fullsized":"2024-09-11T08:27:36.882323+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-11T08:22:51.567081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:22:51.567081+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041879530000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":445,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.884062+0000","last_change":"2024-09-11T08:24:17.569443+0000","last_active":"2024-09-11T08:27:36.884062+0000","last_peered":"2024-09-11T08:27:36.884062+0000","last_clean":"2024-09-11T08:27:36.884062+0000","last_became_active":"2024-09-11T08:07:30.045932+0000","last_became_peered":"2024-09-11T08:07:30.045932+0000","last_unstale":"2024-09-11T08:27:36.884062+0000","last_undegraded":"2024-09-11T08:27:36.884062+0000","last_fullsized":"2024-09-11T08:27:36.884062+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.569369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.569369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028606489999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"405'35","reported_seq":546,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:39.659549+0000","last_change":"2024-09-11T08:24:23.093331+0000","last_active":"2024-09-11T08:27:39.659549+0000","last_peered":"2024-09-11T08:27:39.659549+0000","last_clean":"2024-09-11T08:27:39.659549+0000","last_became_active":"2024-09-11T08:15:56.515817+0000","last_became_peered":"2024-09-11T08:15:56.515817+0000","last_unstale":"2024-09-11T08:27:39.659549+0000","last_undegraded":"2024-09-11T08:27:39.659549+0000","last_fullsized":"2024-09-11T08:27:39.659549+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'25","last_scrub_stamp":"2024-09-11T08:24:23.093269+0000","last_deep_scrub":"350'25","last_deep_scrub_stamp":"2024-09-11T08:24:23.093269+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.093269+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015481372,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"254'12","reported_seq":751,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:38.814930+0000","last_change":"2024-09-11T08:27:16.738809+0000","last_active":"2024-09-11T08:27:38.814930+0000","last_peered":"2024-09-11T08:27:38.814930+0000","last_clean":"2024-09-11T08:27:38.814930+0000","last_became_active":"2024-09-11T08:15:56.525498+0000","last_became_peered":"2024-09-11T08:15:56.525498+0000","last_unstale":"2024-09-11T08:27:38.814930+0000","last_undegraded":"2024-09-11T08:27:38.814930+0000","last_fullsized":"2024-09-11T08:27:38.814930+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'12","last_scrub_stamp":"2024-09-11T08:27:16.738727+0000","last_deep_scrub":"254'12","last_deep_scrub_stamp":"2024-09-11T08:18:34.253865+0000","last_clean_scrub_stamp":"2024-09-11T08:27:16.738727+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:21.793809+0000","scrub_duration":0.0092781189999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.0","version":"406'6017","reported_seq":6049,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-11T08:27:43.781826+0000","last_change":"2024-09-11T08:26:27.764038+0000","last_active":"2024-09-11T08:27:43.781826+0000","last_peered":"2024-09-11T08:27:43.781826+0000","last_clean":"2024-09-11T08:27:43.781826+0000","last_became_active":"2024-09-11T08:26:27.763713+0000","last_became_peered":"2024-09-11T08:26:27.763713+0000","last_unstale":"2024-09-11T08:27:43.781826+0000","last_undegraded":"2024-09-11T08:27:43.781826+0000","last_fullsized":"2024-09-11T08:27:43.781826+0000","mapping_epoch":382,"log_start":"388'1100","ondisk_log_start":"388'1100","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:26:26.740307+0000","objects_scrubbed":0,"log_size":4917,"log_dups_size":0,"ondisk_log_size":4917,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:43.830110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394330112,"num_objects":6017,"num_object_clones":0,"num_object_copies":18051,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6017,"num_write_kb":385088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"405'39","reported_seq":514,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:39.852827+0000","last_change":"2024-09-11T08:24:11.562773+0000","last_active":"2024-09-11T08:27:39.852827+0000","last_peered":"2024-09-11T08:27:39.852827+0000","last_clean":"2024-09-11T08:27:39.852827+0000","last_became_active":"2024-09-11T08:07:32.054317+0000","last_became_peered":"2024-09-11T08:07:32.054317+0000","last_unstale":"2024-09-11T08:27:39.852827+0000","last_undegraded":"2024-09-11T08:27:39.852827+0000","last_fullsized":"2024-09-11T08:27:39.852827+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'27","last_scrub_stamp":"2024-09-11T08:24:11.562679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:24:11.562679+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0092178139999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":466,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.878099+0000","last_change":"2024-09-11T08:27:10.670832+0000","last_active":"2024-09-11T08:27:36.878099+0000","last_peered":"2024-09-11T08:27:36.878099+0000","last_clean":"2024-09-11T08:27:36.878099+0000","last_became_active":"2024-09-11T08:15:56.521943+0000","last_became_peered":"2024-09-11T08:15:56.521943+0000","last_unstale":"2024-09-11T08:27:36.878099+0000","last_undegraded":"2024-09-11T08:27:36.878099+0000","last_fullsized":"2024-09-11T08:27:36.878099+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:10.670751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:22:00.707990+0000","last_clean_scrub_stamp":"2024-09-11T08:27:10.670751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:36.828574+0000","scrub_duration":0.0086032320000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"354'13","reported_seq":182,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-11T08:27:40.751709+0000","last_change":"2024-09-11T08:26:00.096276+0000","last_active":"2024-09-11T08:27:40.751709+0000","last_peered":"2024-09-11T08:27:40.751709+0000","last_clean":"2024-09-11T08:27:40.751709+0000","last_became_active":"2024-09-11T08:24:28.368212+0000","last_became_peered":"2024-09-11T08:24:28.368212+0000","last_unstale":"2024-09-11T08:27:40.751709+0000","last_undegraded":"2024-09-11T08:27:40.751709+0000","last_fullsized":"2024-09-11T08:27:40.751709+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'13","last_scrub_stamp":"2024-09-11T08:26:00.096215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:26:00.096215+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016774890000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":138,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.885379+0000","last_change":"2024-09-11T08:27:05.040598+0000","last_active":"2024-09-11T08:27:36.885379+0000","last_peered":"2024-09-11T08:27:36.885379+0000","last_clean":"2024-09-11T08:27:36.885379+0000","last_became_active":"2024-09-11T08:24:28.353556+0000","last_became_peered":"2024-09-11T08:24:28.353556+0000","last_unstale":"2024-09-11T08:27:36.885379+0000","last_undegraded":"2024-09-11T08:27:36.885379+0000","last_fullsized":"2024-09-11T08:27:36.885379+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:05.040523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:06.616332+0000","last_clean_scrub_stamp":"2024-09-11T08:27:05.040523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:34.516330+0000","scrub_duration":0.0050758849999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.f","version":"406'6102","reported_seq":6134,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-11T08:27:43.218276+0000","last_change":"2024-09-11T08:26:27.766747+0000","last_active":"2024-09-11T08:27:43.218276+0000","last_peered":"2024-09-11T08:27:43.218276+0000","last_clean":"2024-09-11T08:27:43.218276+0000","last_became_active":"2024-09-11T08:26:27.766540+0000","last_became_peered":"2024-09-11T08:26:27.766540+0000","last_unstale":"2024-09-11T08:27:43.218276+0000","last_undegraded":"2024-09-11T08:27:43.218276+0000","last_fullsized":"2024-09-11T08:27:43.218276+0000","mapping_epoch":382,"log_start":"388'1000","ondisk_log_start":"388'1000","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:26:26.740307+0000","objects_scrubbed":0,"log_size":5102,"log_dups_size":0,"ondisk_log_size":5102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399900672,"num_objects":6102,"num_object_clones":0,"num_object_copies":18306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6102,"num_write_kb":390528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"405'42","reported_seq":503,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:39.459517+0000","last_change":"2024-09-11T08:27:12.224895+0000","last_active":"2024-09-11T08:27:39.459517+0000","last_peered":"2024-09-11T08:27:39.459517+0000","last_clean":"2024-09-11T08:27:39.459517+0000","last_became_active":"2024-09-11T08:17:37.555703+0000","last_became_peered":"2024-09-11T08:17:37.555703+0000","last_unstale":"2024-09-11T08:27:39.459517+0000","last_undegraded":"2024-09-11T08:27:39.459517+0000","last_fullsized":"2024-09-11T08:27:39.459517+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'30","last_scrub_stamp":"2024-09-11T08:27:12.224825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:27:12.224825+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:18.909592+0000","scrub_duration":0.0073510700000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":416,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.880142+0000","last_change":"2024-09-11T08:26:08.112087+0000","last_active":"2024-09-11T08:27:36.880142+0000","last_peered":"2024-09-11T08:27:36.880142+0000","last_clean":"2024-09-11T08:27:36.880142+0000","last_became_active":"2024-09-11T08:17:37.553579+0000","last_became_peered":"2024-09-11T08:17:37.553579+0000","last_unstale":"2024-09-11T08:27:36.880142+0000","last_undegraded":"2024-09-11T08:27:36.880142+0000","last_fullsized":"2024-09-11T08:27:36.880142+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:08.112023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:26:08.112023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001608578,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":455,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.879496+0000","last_change":"2024-09-11T08:26:01.994013+0000","last_active":"2024-09-11T08:27:36.879496+0000","last_peered":"2024-09-11T08:27:36.879496+0000","last_clean":"2024-09-11T08:27:36.879496+0000","last_became_active":"2024-09-11T08:15:56.522606+0000","last_became_peered":"2024-09-11T08:15:56.522606+0000","last_unstale":"2024-09-11T08:27:36.879496+0000","last_undegraded":"2024-09-11T08:27:36.879496+0000","last_fullsized":"2024-09-11T08:27:36.879496+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:01.993924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:26:01.993924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012801519999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":91,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.875939+0000","last_change":"2024-09-11T08:26:54.574588+0000","last_active":"2024-09-11T08:27:36.875939+0000","last_peered":"2024-09-11T08:27:36.875939+0000","last_clean":"2024-09-11T08:27:36.875939+0000","last_became_active":"2024-09-11T08:23:22.683474+0000","last_became_peered":"2024-09-11T08:23:22.683474+0000","last_unstale":"2024-09-11T08:27:36.875939+0000","last_undegraded":"2024-09-11T08:27:36.875939+0000","last_fullsized":"2024-09-11T08:27:36.875939+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:54.574506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:22:50.151585+0000","last_clean_scrub_stamp":"2024-09-11T08:26:54.574506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:21.672267+0000","scrub_duration":0.012208658000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"406'6162","reported_seq":6194,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-11T08:27:42.701265+0000","last_change":"2024-09-11T08:26:27.765017+0000","last_active":"2024-09-11T08:27:42.701265+0000","last_peered":"2024-09-11T08:27:42.701265+0000","last_clean":"2024-09-11T08:27:42.701265+0000","last_became_active":"2024-09-11T08:26:27.764671+0000","last_became_peered":"2024-09-11T08:26:27.764671+0000","last_unstale":"2024-09-11T08:27:42.701265+0000","last_undegraded":"2024-09-11T08:27:42.701265+0000","last_fullsized":"2024-09-11T08:27:42.701265+0000","mapping_epoch":382,"log_start":"388'1400","ondisk_log_start":"388'1400","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:26:26.740307+0000","objects_scrubbed":0,"log_size":4762,"log_dups_size":0,"ondisk_log_size":4762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:47.926361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403832832,"num_objects":6162,"num_object_clones":0,"num_object_copies":18486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6162,"num_write_kb":394368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"405'42","reported_seq":396,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:39.666968+0000","last_change":"2024-09-11T08:27:10.868771+0000","last_active":"2024-09-11T08:27:39.666968+0000","last_peered":"2024-09-11T08:27:39.666968+0000","last_clean":"2024-09-11T08:27:39.666968+0000","last_became_active":"2024-09-11T08:24:28.358175+0000","last_became_peered":"2024-09-11T08:24:28.358175+0000","last_unstale":"2024-09-11T08:27:39.666968+0000","last_undegraded":"2024-09-11T08:27:39.666968+0000","last_fullsized":"2024-09-11T08:27:39.666968+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'28","last_scrub_stamp":"2024-09-11T08:27:10.868676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:27:10.868676+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:26.816074+0000","scrub_duration":0.0026168770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":435,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.879568+0000","last_change":"2024-09-11T08:25:51.199008+0000","last_active":"2024-09-11T08:27:36.879568+0000","last_peered":"2024-09-11T08:27:36.879568+0000","last_clean":"2024-09-11T08:27:36.879568+0000","last_became_active":"2024-09-11T08:07:30.050484+0000","last_became_peered":"2024-09-11T08:07:30.050484+0000","last_unstale":"2024-09-11T08:27:36.879568+0000","last_undegraded":"2024-09-11T08:27:36.879568+0000","last_fullsized":"2024-09-11T08:27:36.879568+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:25:51.198956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:47.960329+0000","last_clean_scrub_stamp":"2024-09-11T08:25:51.198956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001153823,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"82'9","reported_seq":706,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:38.813086+0000","last_change":"2024-09-11T08:27:18.825321+0000","last_active":"2024-09-11T08:27:38.813086+0000","last_peered":"2024-09-11T08:27:38.813086+0000","last_clean":"2024-09-11T08:27:38.813086+0000","last_became_active":"2024-09-11T08:07:34.056143+0000","last_became_peered":"2024-09-11T08:07:34.056143+0000","last_unstale":"2024-09-11T08:27:38.813086+0000","last_undegraded":"2024-09-11T08:27:38.813086+0000","last_fullsized":"2024-09-11T08:27:38.813086+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-11T08:27:18.825233+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-09-11T08:21:00.841426+0000","last_clean_scrub_stamp":"2024-09-11T08:27:18.825233+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:22.786207+0000","scrub_duration":0.012602754000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":410,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.878322+0000","last_change":"2024-09-11T08:24:24.021006+0000","last_active":"2024-09-11T08:27:36.878322+0000","last_peered":"2024-09-11T08:27:36.878322+0000","last_clean":"2024-09-11T08:27:36.878322+0000","last_became_active":"2024-09-11T08:14:24.788968+0000","last_became_peered":"2024-09-11T08:14:24.788968+0000","last_unstale":"2024-09-11T08:27:36.878322+0000","last_undegraded":"2024-09-11T08:27:36.878322+0000","last_fullsized":"2024-09-11T08:27:36.878322+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:24.020925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:24:24.020925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010911288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.d","version":"406'6087","reported_seq":6119,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-11T08:27:43.533813+0000","last_change":"2024-09-11T08:26:27.763488+0000","last_active":"2024-09-11T08:27:43.533813+0000","last_peered":"2024-09-11T08:27:43.533813+0000","last_clean":"2024-09-11T08:27:43.533813+0000","last_became_active":"2024-09-11T08:26:27.763225+0000","last_became_peered":"2024-09-11T08:26:27.763225+0000","last_unstale":"2024-09-11T08:27:43.533813+0000","last_undegraded":"2024-09-11T08:27:43.533813+0000","last_fullsized":"2024-09-11T08:27:43.533813+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:26:26.740307+0000","objects_scrubbed":0,"log_size":6087,"log_dups_size":0,"ondisk_log_size":6087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:56.503503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398917632,"num_objects":6087,"num_object_clones":0,"num_object_copies":18261,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6087,"num_write_kb":389568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"405'30","reported_seq":548,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:39.552974+0000","last_change":"2024-09-11T08:26:57.866443+0000","last_active":"2024-09-11T08:27:39.552974+0000","last_peered":"2024-09-11T08:27:39.552974+0000","last_clean":"2024-09-11T08:27:39.552974+0000","last_became_active":"2024-09-11T08:24:28.355516+0000","last_became_peered":"2024-09-11T08:24:28.355516+0000","last_unstale":"2024-09-11T08:27:39.552974+0000","last_undegraded":"2024-09-11T08:27:39.552974+0000","last_fullsized":"2024-09-11T08:27:39.552974+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'20","last_scrub_stamp":"2024-09-11T08:26:57.866352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:26:57.866352+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:03.651194+0000","scrub_duration":0.0030126530000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":146,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.876028+0000","last_change":"2024-09-11T08:26:02.614677+0000","last_active":"2024-09-11T08:27:36.876028+0000","last_peered":"2024-09-11T08:27:36.876028+0000","last_clean":"2024-09-11T08:27:36.876028+0000","last_became_active":"2024-09-11T08:24:28.352741+0000","last_became_peered":"2024-09-11T08:24:28.352741+0000","last_unstale":"2024-09-11T08:27:36.876028+0000","last_undegraded":"2024-09-11T08:27:36.876028+0000","last_fullsized":"2024-09-11T08:27:36.876028+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:02.614567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:26:02.614567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018716119999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"376'15","reported_seq":381,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:38.814971+0000","last_change":"2024-09-11T08:26:02.786822+0000","last_active":"2024-09-11T08:27:38.814971+0000","last_peered":"2024-09-11T08:27:38.814971+0000","last_clean":"2024-09-11T08:27:38.814971+0000","last_became_active":"2024-09-11T08:24:28.359787+0000","last_became_peered":"2024-09-11T08:24:28.359787+0000","last_unstale":"2024-09-11T08:27:38.814971+0000","last_undegraded":"2024-09-11T08:27:38.814971+0000","last_fullsized":"2024-09-11T08:27:38.814971+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'15","last_scrub_stamp":"2024-09-11T08:26:02.786767+0000","last_deep_scrub":"376'15","last_deep_scrub_stamp":"2024-09-11T08:26:02.786767+0000","last_clean_scrub_stamp":"2024-09-11T08:26:02.786767+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023292619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"66'1","reported_seq":423,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.878726+0000","last_change":"2024-09-11T08:27:24.605347+0000","last_active":"2024-09-11T08:27:36.878726+0000","last_peered":"2024-09-11T08:27:36.878726+0000","last_clean":"2024-09-11T08:27:36.878726+0000","last_became_active":"2024-09-11T08:07:36.072593+0000","last_became_peered":"2024-09-11T08:07:36.072593+0000","last_unstale":"2024-09-11T08:27:36.878726+0000","last_undegraded":"2024-09-11T08:27:36.878726+0000","last_fullsized":"2024-09-11T08:27:36.878726+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-11T08:27:24.605264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:27:24.605264+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:46.599632+0000","scrub_duration":0.0040252509999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"406'6157","reported_seq":6189,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-11T08:27:43.822532+0000","last_change":"2024-09-11T08:26:27.765203+0000","last_active":"2024-09-11T08:27:43.822532+0000","last_peered":"2024-09-11T08:27:43.822532+0000","last_clean":"2024-09-11T08:27:43.822532+0000","last_became_active":"2024-09-11T08:26:27.764927+0000","last_became_peered":"2024-09-11T08:26:27.764927+0000","last_unstale":"2024-09-11T08:27:43.822532+0000","last_undegraded":"2024-09-11T08:27:43.822532+0000","last_fullsized":"2024-09-11T08:27:43.822532+0000","mapping_epoch":382,"log_start":"388'1300","ondisk_log_start":"388'1300","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:26:26.740307+0000","objects_scrubbed":0,"log_size":4857,"log_dups_size":0,"ondisk_log_size":4857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403505152,"num_objects":6157,"num_object_clones":0,"num_object_copies":18471,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6157,"num_write_kb":394048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"405'47","reported_seq":636,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:39.512362+0000","last_change":"2024-09-11T08:27:16.673939+0000","last_active":"2024-09-11T08:27:39.512362+0000","last_peered":"2024-09-11T08:27:39.512362+0000","last_clean":"2024-09-11T08:27:39.512362+0000","last_became_active":"2024-09-11T08:24:28.364043+0000","last_became_peered":"2024-09-11T08:24:28.364043+0000","last_unstale":"2024-09-11T08:27:39.512362+0000","last_undegraded":"2024-09-11T08:27:39.512362+0000","last_fullsized":"2024-09-11T08:27:39.512362+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"396'37","last_scrub_stamp":"2024-09-11T08:27:16.673846+0000","last_deep_scrub":"396'37","last_deep_scrub_stamp":"2024-09-11T08:27:16.673846+0000","last_clean_scrub_stamp":"2024-09-11T08:27:16.673846+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:26.414974+0000","scrub_duration":0.02185254,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":491,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.885190+0000","last_change":"2024-09-11T08:26:55.920593+0000","last_active":"2024-09-11T08:27:36.885190+0000","last_peered":"2024-09-11T08:27:36.885190+0000","last_clean":"2024-09-11T08:27:36.885190+0000","last_became_active":"2024-09-11T08:24:28.358796+0000","last_became_peered":"2024-09-11T08:24:28.358796+0000","last_unstale":"2024-09-11T08:27:36.885190+0000","last_undegraded":"2024-09-11T08:27:36.885190+0000","last_fullsized":"2024-09-11T08:27:36.885190+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:55.920512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:26:55.920512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:16.956014+0000","scrub_duration":0.011546753999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":488,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.882573+0000","last_change":"2024-09-11T08:27:03.828113+0000","last_active":"2024-09-11T08:27:36.882573+0000","last_peered":"2024-09-11T08:27:36.882573+0000","last_clean":"2024-09-11T08:27:36.882573+0000","last_became_active":"2024-09-11T08:24:28.360235+0000","last_became_peered":"2024-09-11T08:24:28.360235+0000","last_unstale":"2024-09-11T08:27:36.882573+0000","last_undegraded":"2024-09-11T08:27:36.882573+0000","last_fullsized":"2024-09-11T08:27:36.882573+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:03.828043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:27:03.828043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:33.026945+0000","scrub_duration":0.0023665330000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":439,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.883842+0000","last_change":"2024-09-11T08:24:24.506494+0000","last_active":"2024-09-11T08:27:36.883842+0000","last_peered":"2024-09-11T08:27:36.883842+0000","last_clean":"2024-09-11T08:27:36.883842+0000","last_became_active":"2024-09-11T08:07:36.631668+0000","last_became_peered":"2024-09-11T08:07:36.631668+0000","last_unstale":"2024-09-11T08:27:36.883842+0000","last_undegraded":"2024-09-11T08:27:36.883842+0000","last_fullsized":"2024-09-11T08:27:36.883842+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:24.506402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:58.517624+0000","last_clean_scrub_stamp":"2024-09-11T08:24:24.506402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0096043550000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.b","version":"406'6131","reported_seq":6163,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-11T08:27:43.557069+0000","last_change":"2024-09-11T08:26:27.764606+0000","last_active":"2024-09-11T08:27:43.557069+0000","last_peered":"2024-09-11T08:27:43.557069+0000","last_clean":"2024-09-11T08:27:43.557069+0000","last_became_active":"2024-09-11T08:26:27.764354+0000","last_became_peered":"2024-09-11T08:26:27.764354+0000","last_unstale":"2024-09-11T08:27:43.557069+0000","last_undegraded":"2024-09-11T08:27:43.557069+0000","last_fullsized":"2024-09-11T08:27:43.557069+0000","mapping_epoch":382,"log_start":"388'1300","ondisk_log_start":"388'1300","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:26:26.740307+0000","objects_scrubbed":0,"log_size":4831,"log_dups_size":0,"ondisk_log_size":4831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:56.383464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":401801216,"num_objects":6131,"num_object_clones":0,"num_object_copies":18393,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6131,"num_write_kb":392384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"405'64","reported_seq":272,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-11T08:27:40.748058+0000","last_change":"2024-09-11T08:24:13.567684+0000","last_active":"2024-09-11T08:27:40.748058+0000","last_peered":"2024-09-11T08:27:40.748058+0000","last_clean":"2024-09-11T08:27:40.748058+0000","last_became_active":"2024-09-11T08:20:52.086843+0000","last_became_peered":"2024-09-11T08:20:52.086843+0000","last_unstale":"2024-09-11T08:27:40.748058+0000","last_undegraded":"2024-09-11T08:27:40.748058+0000","last_fullsized":"2024-09-11T08:27:40.748058+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'52","last_scrub_stamp":"2024-09-11T08:24:13.567595+0000","last_deep_scrub":"190'39","last_deep_scrub_stamp":"2024-09-11T08:15:52.847968+0000","last_clean_scrub_stamp":"2024-09-11T08:24:13.567595+0000","objects_scrubbed":10,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028629020000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":84,"num_read_kb":69,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"59'1","reported_seq":99,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.879727+0000","last_change":"2024-09-11T08:24:13.277960+0000","last_active":"2024-09-11T08:27:36.879727+0000","last_peered":"2024-09-11T08:27:36.879727+0000","last_clean":"2024-09-11T08:27:36.879727+0000","last_became_active":"2024-09-11T08:23:22.691726+0000","last_became_peered":"2024-09-11T08:23:22.691726+0000","last_unstale":"2024-09-11T08:27:36.879727+0000","last_undegraded":"2024-09-11T08:27:36.879727+0000","last_fullsized":"2024-09-11T08:27:36.879727+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-11T08:24:13.275269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:24:13.275269+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.007807863,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":155,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.884478+0000","last_change":"2024-09-11T08:24:29.448464+0000","last_active":"2024-09-11T08:27:36.884478+0000","last_peered":"2024-09-11T08:27:36.884478+0000","last_clean":"2024-09-11T08:27:36.884478+0000","last_became_active":"2024-09-11T08:24:28.362590+0000","last_became_peered":"2024-09-11T08:24:28.362590+0000","last_unstale":"2024-09-11T08:27:36.884478+0000","last_undegraded":"2024-09-11T08:27:36.884478+0000","last_fullsized":"2024-09-11T08:27:36.884478+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-11T08:24:29.448408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:24:29.448408+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0075800399999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":421,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.879200+0000","last_change":"2024-09-11T08:24:08.219265+0000","last_active":"2024-09-11T08:27:36.879200+0000","last_peered":"2024-09-11T08:27:36.879200+0000","last_clean":"2024-09-11T08:27:36.879200+0000","last_became_active":"2024-09-11T08:07:34.057154+0000","last_became_peered":"2024-09-11T08:07:34.057154+0000","last_unstale":"2024-09-11T08:27:36.879200+0000","last_undegraded":"2024-09-11T08:27:36.879200+0000","last_fullsized":"2024-09-11T08:27:36.879200+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:08.219182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:10:37.957096+0000","last_clean_scrub_stamp":"2024-09-11T08:24:08.219182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0084274109999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":457,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.876214+0000","last_change":"2024-09-11T08:24:16.032604+0000","last_active":"2024-09-11T08:27:36.876214+0000","last_peered":"2024-09-11T08:27:36.876214+0000","last_clean":"2024-09-11T08:27:36.876214+0000","last_became_active":"2024-09-11T08:15:56.525099+0000","last_became_peered":"2024-09-11T08:15:56.525099+0000","last_unstale":"2024-09-11T08:27:36.876214+0000","last_undegraded":"2024-09-11T08:27:36.876214+0000","last_fullsized":"2024-09-11T08:27:36.876214+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:16.032515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:24:16.032515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044030470000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"406'6184","reported_seq":6216,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-11T08:27:43.156473+0000","last_change":"2024-09-11T08:26:28.083045+0000","last_active":"2024-09-11T08:27:43.156473+0000","last_peered":"2024-09-11T08:27:43.156473+0000","last_clean":"2024-09-11T08:27:43.156473+0000","last_became_active":"2024-09-11T08:26:28.082722+0000","last_became_peered":"2024-09-11T08:26:28.082722+0000","last_unstale":"2024-09-11T08:27:43.156473+0000","last_undegraded":"2024-09-11T08:27:43.156473+0000","last_fullsized":"2024-09-11T08:27:43.156473+0000","mapping_epoch":382,"log_start":"388'300","ondisk_log_start":"388'300","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:26:26.740307+0000","objects_scrubbed":0,"log_size":5884,"log_dups_size":0,"ondisk_log_size":5884,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:56.502731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":405274624,"num_objects":6184,"num_object_clones":0,"num_object_copies":18552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6184,"num_write_kb":395776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"405'21","reported_seq":552,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:39.140429+0000","last_change":"2024-09-11T08:26:56.891150+0000","last_active":"2024-09-11T08:27:39.140429+0000","last_peered":"2024-09-11T08:27:39.140429+0000","last_clean":"2024-09-11T08:27:39.140429+0000","last_became_active":"2024-09-11T08:24:28.358121+0000","last_became_peered":"2024-09-11T08:24:28.358121+0000","last_unstale":"2024-09-11T08:27:39.140429+0000","last_undegraded":"2024-09-11T08:27:39.140429+0000","last_fullsized":"2024-09-11T08:27:39.140429+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'17","last_scrub_stamp":"2024-09-11T08:26:56.890998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:26:56.890998+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:16.584252+0000","scrub_duration":0.012396804000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":458,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.881108+0000","last_change":"2024-09-11T08:26:10.658086+0000","last_active":"2024-09-11T08:27:36.881108+0000","last_peered":"2024-09-11T08:27:36.881108+0000","last_clean":"2024-09-11T08:27:36.881108+0000","last_became_active":"2024-09-11T08:07:30.047014+0000","last_became_peered":"2024-09-11T08:07:30.047014+0000","last_unstale":"2024-09-11T08:27:36.881108+0000","last_undegraded":"2024-09-11T08:27:36.881108+0000","last_fullsized":"2024-09-11T08:27:36.881108+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:10.658003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:26:10.658003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001689107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"127'2","reported_seq":475,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.882324+0000","last_change":"2024-09-11T08:26:53.737700+0000","last_active":"2024-09-11T08:27:36.882324+0000","last_peered":"2024-09-11T08:27:36.882324+0000","last_clean":"2024-09-11T08:27:36.882324+0000","last_became_active":"2024-09-11T08:07:30.045009+0000","last_became_peered":"2024-09-11T08:07:30.045009+0000","last_unstale":"2024-09-11T08:27:36.882324+0000","last_undegraded":"2024-09-11T08:27:36.882324+0000","last_fullsized":"2024-09-11T08:27:36.882324+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'2","last_scrub_stamp":"2024-09-11T08:26:53.737601+0000","last_deep_scrub":"127'2","last_deep_scrub_stamp":"2024-09-11T08:15:01.938497+0000","last_clean_scrub_stamp":"2024-09-11T08:26:53.737601+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:56.865318+0000","scrub_duration":0.0039636020000000001,"objects_trimmed":0,"snaptrim_duration":0.001859533,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":501,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.882838+0000","last_change":"2024-09-11T08:26:05.592197+0000","last_active":"2024-09-11T08:27:36.882838+0000","last_peered":"2024-09-11T08:27:36.882838+0000","last_clean":"2024-09-11T08:27:36.882838+0000","last_became_active":"2024-09-11T08:24:28.357141+0000","last_became_peered":"2024-09-11T08:24:28.357141+0000","last_unstale":"2024-09-11T08:27:36.882838+0000","last_undegraded":"2024-09-11T08:27:36.882838+0000","last_fullsized":"2024-09-11T08:27:36.882838+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:05.592132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:22:00.200872+0000","last_clean_scrub_stamp":"2024-09-11T08:26:05.592132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001603754,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":161,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.882479+0000","last_change":"2024-09-11T08:27:06.910328+0000","last_active":"2024-09-11T08:27:36.882479+0000","last_peered":"2024-09-11T08:27:36.882479+0000","last_clean":"2024-09-11T08:27:36.882479+0000","last_became_active":"2024-09-11T08:20:16.108674+0000","last_became_peered":"2024-09-11T08:20:16.108674+0000","last_unstale":"2024-09-11T08:27:36.882479+0000","last_undegraded":"2024-09-11T08:27:36.882479+0000","last_fullsized":"2024-09-11T08:27:36.882479+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:06.910177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:58.796217+0000","last_clean_scrub_stamp":"2024-09-11T08:27:06.910177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:10.778030+0000","scrub_duration":0.0054417340000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.9","version":"406'6070","reported_seq":6102,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-11T08:27:43.574898+0000","last_change":"2024-09-11T08:26:27.761702+0000","last_active":"2024-09-11T08:27:43.574898+0000","last_peered":"2024-09-11T08:27:43.574898+0000","last_clean":"2024-09-11T08:27:43.574898+0000","last_became_active":"2024-09-11T08:26:27.761414+0000","last_became_peered":"2024-09-11T08:26:27.761414+0000","last_unstale":"2024-09-11T08:27:43.574898+0000","last_undegraded":"2024-09-11T08:27:43.574898+0000","last_fullsized":"2024-09-11T08:27:43.574898+0000","mapping_epoch":382,"log_start":"388'1300","ondisk_log_start":"388'1300","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:26:26.740307+0000","objects_scrubbed":0,"log_size":4770,"log_dups_size":0,"ondisk_log_size":4770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":397803520,"num_objects":6070,"num_object_clones":0,"num_object_copies":18210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6070,"num_write_kb":388480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"405'42","reported_seq":493,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:39.231069+0000","last_change":"2024-09-11T08:26:09.030251+0000","last_active":"2024-09-11T08:27:39.231069+0000","last_peered":"2024-09-11T08:27:39.231069+0000","last_clean":"2024-09-11T08:27:39.231069+0000","last_became_active":"2024-09-11T08:07:32.055407+0000","last_became_peered":"2024-09-11T08:07:32.055407+0000","last_unstale":"2024-09-11T08:27:39.231069+0000","last_undegraded":"2024-09-11T08:27:39.231069+0000","last_fullsized":"2024-09-11T08:27:39.231069+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"287'30","last_scrub_stamp":"2024-09-11T08:26:09.030192+0000","last_deep_scrub":"287'30","last_deep_scrub_stamp":"2024-09-11T08:26:09.030192+0000","last_clean_scrub_stamp":"2024-09-11T08:26:09.030192+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020634899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"59'1","reported_seq":432,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.880054+0000","last_change":"2024-09-11T08:23:34.954017+0000","last_active":"2024-09-11T08:27:36.880054+0000","last_peered":"2024-09-11T08:27:36.880054+0000","last_clean":"2024-09-11T08:27:36.880054+0000","last_became_active":"2024-09-11T08:07:30.050504+0000","last_became_peered":"2024-09-11T08:07:30.050504+0000","last_unstale":"2024-09-11T08:27:36.880054+0000","last_undegraded":"2024-09-11T08:27:36.880054+0000","last_fullsized":"2024-09-11T08:27:36.880054+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-11T08:23:34.953923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:23:34.953923+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035255019999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"224'6","reported_seq":1426,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-11T08:27:42.209802+0000","last_change":"2024-09-11T08:24:20.011482+0000","last_active":"2024-09-11T08:27:42.209802+0000","last_peered":"2024-09-11T08:27:42.209802+0000","last_clean":"2024-09-11T08:27:42.209802+0000","last_became_active":"2024-09-11T08:14:24.793048+0000","last_became_peered":"2024-09-11T08:14:24.793048+0000","last_unstale":"2024-09-11T08:27:42.209802+0000","last_undegraded":"2024-09-11T08:27:42.209802+0000","last_fullsized":"2024-09-11T08:27:42.209802+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'6","last_scrub_stamp":"2024-09-11T08:24:20.011383+0000","last_deep_scrub":"224'6","last_deep_scrub_stamp":"2024-09-11T08:16:55.778284+0000","last_clean_scrub_stamp":"2024-09-11T08:24:20.011383+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0060728680000000004,"objects_trimmed":0,"snaptrim_duration":0.000318821,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":804,"num_read_kb":799,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":148,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.887950+0000","last_change":"2024-09-11T08:27:14.117515+0000","last_active":"2024-09-11T08:27:36.887950+0000","last_peered":"2024-09-11T08:27:36.887950+0000","last_clean":"2024-09-11T08:27:36.887950+0000","last_became_active":"2024-09-11T08:24:28.353018+0000","last_became_peered":"2024-09-11T08:24:28.353018+0000","last_unstale":"2024-09-11T08:27:36.887950+0000","last_undegraded":"2024-09-11T08:27:36.887950+0000","last_fullsized":"2024-09-11T08:27:36.887950+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:14.114373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:27:14.114373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:20.579280+0000","scrub_duration":0.0066634249999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":492,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.883939+0000","last_change":"2024-09-11T08:27:11.025202+0000","last_active":"2024-09-11T08:27:36.883939+0000","last_peered":"2024-09-11T08:27:36.883939+0000","last_clean":"2024-09-11T08:27:36.883939+0000","last_became_active":"2024-09-11T08:24:28.358723+0000","last_became_peered":"2024-09-11T08:24:28.358723+0000","last_unstale":"2024-09-11T08:27:36.883939+0000","last_undegraded":"2024-09-11T08:27:36.883939+0000","last_fullsized":"2024-09-11T08:27:36.883939+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:11.025109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:27:11.025109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:23.447076+0000","scrub_duration":0.0049163100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"406'6130","reported_seq":6162,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-11T08:27:43.213306+0000","last_change":"2024-09-11T08:26:28.083341+0000","last_active":"2024-09-11T08:27:43.213306+0000","last_peered":"2024-09-11T08:27:43.213306+0000","last_clean":"2024-09-11T08:27:43.213306+0000","last_became_active":"2024-09-11T08:26:28.082996+0000","last_became_peered":"2024-09-11T08:26:28.082996+0000","last_unstale":"2024-09-11T08:27:43.213306+0000","last_undegraded":"2024-09-11T08:27:43.213306+0000","last_fullsized":"2024-09-11T08:27:43.213306+0000","mapping_epoch":382,"log_start":"388'200","ondisk_log_start":"388'200","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:26:26.740307+0000","objects_scrubbed":0,"log_size":5930,"log_dups_size":0,"ondisk_log_size":5930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:50.235347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":401735680,"num_objects":6130,"num_object_clones":0,"num_object_copies":18390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6130,"num_write_kb":392320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"405'45","reported_seq":535,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:39.618673+0000","last_change":"2024-09-11T08:27:15.771100+0000","last_active":"2024-09-11T08:27:39.618673+0000","last_peered":"2024-09-11T08:27:39.618673+0000","last_clean":"2024-09-11T08:27:39.618673+0000","last_became_active":"2024-09-11T08:14:24.789154+0000","last_became_peered":"2024-09-11T08:14:24.789154+0000","last_unstale":"2024-09-11T08:27:39.618673+0000","last_undegraded":"2024-09-11T08:27:39.618673+0000","last_fullsized":"2024-09-11T08:27:39.618673+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'33","last_scrub_stamp":"2024-09-11T08:27:15.770971+0000","last_deep_scrub":"84'21","last_deep_scrub_stamp":"2024-09-11T08:08:56.705306+0000","last_clean_scrub_stamp":"2024-09-11T08:27:15.770971+0000","objects_scrubbed":7,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:20.921251+0000","scrub_duration":0.0065168980000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":140,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.882973+0000","last_change":"2024-09-11T08:26:07.749930+0000","last_active":"2024-09-11T08:27:36.882973+0000","last_peered":"2024-09-11T08:27:36.882973+0000","last_clean":"2024-09-11T08:27:36.882973+0000","last_became_active":"2024-09-11T08:24:28.362417+0000","last_became_peered":"2024-09-11T08:24:28.362417+0000","last_unstale":"2024-09-11T08:27:36.882973+0000","last_undegraded":"2024-09-11T08:27:36.882973+0000","last_fullsized":"2024-09-11T08:27:36.882973+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:07.749828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:07.749828+0000","last_clean_scrub_stamp":"2024-09-11T08:26:07.749828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015131179999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":254,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.874386+0000","last_change":"2024-09-11T08:23:37.276623+0000","last_active":"2024-09-11T08:27:36.874386+0000","last_peered":"2024-09-11T08:27:36.874386+0000","last_clean":"2024-09-11T08:27:36.874386+0000","last_became_active":"2024-09-11T08:15:56.523430+0000","last_became_peered":"2024-09-11T08:15:56.523430+0000","last_unstale":"2024-09-11T08:27:36.874386+0000","last_undegraded":"2024-09-11T08:27:36.874386+0000","last_fullsized":"2024-09-11T08:27:36.874386+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-11T08:23:37.276533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:05:21.231856+0000","last_clean_scrub_stamp":"2024-09-11T08:23:37.276533+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0089351699999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":278,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.882187+0000","last_change":"2024-09-11T08:25:55.938265+0000","last_active":"2024-09-11T08:27:36.882187+0000","last_peered":"2024-09-11T08:27:36.882187+0000","last_clean":"2024-09-11T08:27:36.882187+0000","last_became_active":"2024-09-11T08:24:28.357166+0000","last_became_peered":"2024-09-11T08:24:28.357166+0000","last_unstale":"2024-09-11T08:27:36.882187+0000","last_undegraded":"2024-09-11T08:27:36.882187+0000","last_fullsized":"2024-09-11T08:27:36.882187+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:22:48.591411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:20:25.775869+0000","last_clean_scrub_stamp":"2024-09-11T08:22:48.591411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0087669500000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":195,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.888133+0000","last_change":"2024-09-11T08:27:12.366790+0000","last_active":"2024-09-11T08:27:36.888133+0000","last_peered":"2024-09-11T08:27:36.888133+0000","last_clean":"2024-09-11T08:27:36.888133+0000","last_became_active":"2024-09-11T08:24:28.356071+0000","last_became_peered":"2024-09-11T08:24:28.356071+0000","last_unstale":"2024-09-11T08:27:36.888133+0000","last_undegraded":"2024-09-11T08:27:36.888133+0000","last_fullsized":"2024-09-11T08:27:36.888133+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:12.366709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:01.777429+0000","last_clean_scrub_stamp":"2024-09-11T08:27:12.366709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:12.647850+0000","scrub_duration":0.0043386370000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"406'6098","reported_seq":6130,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-11T08:27:43.207393+0000","last_change":"2024-09-11T08:26:27.764691+0000","last_active":"2024-09-11T08:27:43.207393+0000","last_peered":"2024-09-11T08:27:43.207393+0000","last_clean":"2024-09-11T08:27:43.207393+0000","last_became_active":"2024-09-11T08:26:27.763926+0000","last_became_peered":"2024-09-11T08:26:27.763926+0000","last_unstale":"2024-09-11T08:27:43.207393+0000","last_undegraded":"2024-09-11T08:27:43.207393+0000","last_fullsized":"2024-09-11T08:27:43.207393+0000","mapping_epoch":382,"log_start":"388'200","ondisk_log_start":"388'200","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:26:26.740307+0000","objects_scrubbed":0,"log_size":5898,"log_dups_size":0,"ondisk_log_size":5898,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399638528,"num_objects":6098,"num_object_clones":0,"num_object_copies":18294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6098,"num_write_kb":390272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"405'32","reported_seq":381,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:39.650954+0000","last_change":"2024-09-11T08:26:59.682857+0000","last_active":"2024-09-11T08:27:39.650954+0000","last_peered":"2024-09-11T08:27:39.650954+0000","last_clean":"2024-09-11T08:27:39.650954+0000","last_became_active":"2024-09-11T08:24:28.368759+0000","last_became_peered":"2024-09-11T08:24:28.368759+0000","last_unstale":"2024-09-11T08:27:39.650954+0000","last_undegraded":"2024-09-11T08:27:39.650954+0000","last_fullsized":"2024-09-11T08:27:39.650954+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'24","last_scrub_stamp":"2024-09-11T08:26:59.682768+0000","last_deep_scrub":"88'16","last_deep_scrub_stamp":"2024-09-11T08:09:39.635984+0000","last_clean_scrub_stamp":"2024-09-11T08:26:59.682768+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:17.305928+0000","scrub_duration":0.0052817990000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":452,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.879794+0000","last_change":"2024-09-11T08:26:06.958652+0000","last_active":"2024-09-11T08:27:36.879794+0000","last_peered":"2024-09-11T08:27:36.879794+0000","last_clean":"2024-09-11T08:27:36.879794+0000","last_became_active":"2024-09-11T08:14:24.789888+0000","last_became_peered":"2024-09-11T08:14:24.789888+0000","last_unstale":"2024-09-11T08:27:36.879794+0000","last_undegraded":"2024-09-11T08:27:36.879794+0000","last_fullsized":"2024-09-11T08:27:36.879794+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:06.958598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:36.417911+0000","last_clean_scrub_stamp":"2024-09-11T08:26:06.958598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001521522,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":132,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.882753+0000","last_change":"2024-09-11T08:26:00.761197+0000","last_active":"2024-09-11T08:27:36.882753+0000","last_peered":"2024-09-11T08:27:36.882753+0000","last_clean":"2024-09-11T08:27:36.882753+0000","last_became_active":"2024-09-11T08:24:28.361261+0000","last_became_peered":"2024-09-11T08:24:28.361261+0000","last_unstale":"2024-09-11T08:27:36.882753+0000","last_undegraded":"2024-09-11T08:27:36.882753+0000","last_fullsized":"2024-09-11T08:27:36.882753+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:00.761139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:01.558340+0000","last_clean_scrub_stamp":"2024-09-11T08:26:00.761139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001250099,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":270,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.886378+0000","last_change":"2024-09-11T08:27:00.720851+0000","last_active":"2024-09-11T08:27:36.886378+0000","last_peered":"2024-09-11T08:27:36.886378+0000","last_clean":"2024-09-11T08:27:36.886378+0000","last_became_active":"2024-09-11T08:24:28.357842+0000","last_became_peered":"2024-09-11T08:24:28.357842+0000","last_unstale":"2024-09-11T08:27:36.886378+0000","last_undegraded":"2024-09-11T08:27:36.886378+0000","last_fullsized":"2024-09-11T08:27:36.886378+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:00.720766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:27:00.720766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:21.874998+0000","scrub_duration":0.001951798,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"406'6303","reported_seq":6335,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-11T08:27:43.552982+0000","last_change":"2024-09-11T08:26:27.763889+0000","last_active":"2024-09-11T08:27:43.552982+0000","last_peered":"2024-09-11T08:27:43.552982+0000","last_clean":"2024-09-11T08:27:43.552982+0000","last_became_active":"2024-09-11T08:26:27.763602+0000","last_became_peered":"2024-09-11T08:26:27.763602+0000","last_unstale":"2024-09-11T08:27:43.552982+0000","last_undegraded":"2024-09-11T08:27:43.552982+0000","last_fullsized":"2024-09-11T08:27:43.552982+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:26:26.740307+0000","objects_scrubbed":0,"log_size":6303,"log_dups_size":0,"ondisk_log_size":6303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":413073408,"num_objects":6303,"num_object_clones":0,"num_object_copies":18909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6303,"num_write_kb":403392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"405'53","reported_seq":432,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:39.644290+0000","last_change":"2024-09-11T08:26:08.526760+0000","last_active":"2024-09-11T08:27:39.644290+0000","last_peered":"2024-09-11T08:27:39.644290+0000","last_clean":"2024-09-11T08:27:39.644290+0000","last_became_active":"2024-09-11T08:23:22.681452+0000","last_became_peered":"2024-09-11T08:23:22.681452+0000","last_unstale":"2024-09-11T08:27:39.644290+0000","last_undegraded":"2024-09-11T08:27:39.644290+0000","last_fullsized":"2024-09-11T08:27:39.644290+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'39","last_scrub_stamp":"2024-09-11T08:26:08.526702+0000","last_deep_scrub":"77'25","last_deep_scrub_stamp":"2024-09-11T08:08:32.458210+0000","last_clean_scrub_stamp":"2024-09-11T08:26:08.526702+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021726229999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":461,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.882734+0000","last_change":"2024-09-11T08:27:21.640335+0000","last_active":"2024-09-11T08:27:36.882734+0000","last_peered":"2024-09-11T08:27:36.882734+0000","last_clean":"2024-09-11T08:27:36.882734+0000","last_became_active":"2024-09-11T08:07:30.042555+0000","last_became_peered":"2024-09-11T08:07:30.042555+0000","last_unstale":"2024-09-11T08:27:36.882734+0000","last_undegraded":"2024-09-11T08:27:36.882734+0000","last_fullsized":"2024-09-11T08:27:36.882734+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:21.640254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:27:21.640254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:34.983975+0000","scrub_duration":0.0016510489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":441,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.884862+0000","last_change":"2024-09-11T08:24:08.512741+0000","last_active":"2024-09-11T08:27:36.884862+0000","last_peered":"2024-09-11T08:27:36.884862+0000","last_clean":"2024-09-11T08:27:36.884862+0000","last_became_active":"2024-09-11T08:07:34.054961+0000","last_became_peered":"2024-09-11T08:07:34.054961+0000","last_unstale":"2024-09-11T08:27:36.884862+0000","last_undegraded":"2024-09-11T08:27:36.884862+0000","last_fullsized":"2024-09-11T08:27:36.884862+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:08.512659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:24:08.512659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010695556,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":469,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.887797+0000","last_change":"2024-09-11T08:27:11.873295+0000","last_active":"2024-09-11T08:27:36.887797+0000","last_peered":"2024-09-11T08:27:36.887797+0000","last_clean":"2024-09-11T08:27:36.887797+0000","last_became_active":"2024-09-11T08:07:36.071194+0000","last_became_peered":"2024-09-11T08:07:36.071194+0000","last_unstale":"2024-09-11T08:27:36.887797+0000","last_undegraded":"2024-09-11T08:27:36.887797+0000","last_fullsized":"2024-09-11T08:27:36.887797+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:11.873150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:07.930137+0000","last_clean_scrub_stamp":"2024-09-11T08:27:11.873150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:35.375212+0000","scrub_duration":0.0065665899999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"406'6126","reported_seq":6158,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-11T08:27:43.180046+0000","last_change":"2024-09-11T08:26:27.764903+0000","last_active":"2024-09-11T08:27:43.180046+0000","last_peered":"2024-09-11T08:27:43.180046+0000","last_clean":"2024-09-11T08:27:43.180046+0000","last_became_active":"2024-09-11T08:26:27.764638+0000","last_became_peered":"2024-09-11T08:26:27.764638+0000","last_unstale":"2024-09-11T08:27:43.180046+0000","last_undegraded":"2024-09-11T08:27:43.180046+0000","last_fullsized":"2024-09-11T08:27:43.180046+0000","mapping_epoch":382,"log_start":"388'200","ondisk_log_start":"388'200","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:26:26.740307+0000","objects_scrubbed":0,"log_size":5926,"log_dups_size":0,"ondisk_log_size":5926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":401473536,"num_objects":6126,"num_object_clones":0,"num_object_copies":18378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6126,"num_write_kb":392064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"405'35","reported_seq":366,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:39.308788+0000","last_change":"2024-09-11T08:24:24.576788+0000","last_active":"2024-09-11T08:27:39.308788+0000","last_peered":"2024-09-11T08:27:39.308788+0000","last_clean":"2024-09-11T08:27:39.308788+0000","last_became_active":"2024-09-11T08:23:22.688631+0000","last_became_peered":"2024-09-11T08:23:22.688631+0000","last_unstale":"2024-09-11T08:27:39.308788+0000","last_undegraded":"2024-09-11T08:27:39.308788+0000","last_fullsized":"2024-09-11T08:27:39.308788+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'25","last_scrub_stamp":"2024-09-11T08:24:24.576699+0000","last_deep_scrub":"224'15","last_deep_scrub_stamp":"2024-09-11T08:17:01.126478+0000","last_clean_scrub_stamp":"2024-09-11T08:24:24.576699+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003559857,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":210,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.879770+0000","last_change":"2024-09-11T08:22:54.830543+0000","last_active":"2024-09-11T08:27:36.879770+0000","last_peered":"2024-09-11T08:27:36.879770+0000","last_clean":"2024-09-11T08:27:36.879770+0000","last_became_active":"2024-09-11T08:17:37.557278+0000","last_became_peered":"2024-09-11T08:17:37.557278+0000","last_unstale":"2024-09-11T08:27:36.879770+0000","last_undegraded":"2024-09-11T08:27:36.879770+0000","last_fullsized":"2024-09-11T08:27:36.879770+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:22:54.830465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:45.979847+0000","last_clean_scrub_stamp":"2024-09-11T08:22:54.830465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0062432310000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"354'16","reported_seq":772,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:38.812277+0000","last_change":"2024-09-11T08:27:12.918897+0000","last_active":"2024-09-11T08:27:38.812277+0000","last_peered":"2024-09-11T08:27:38.812277+0000","last_clean":"2024-09-11T08:27:38.812277+0000","last_became_active":"2024-09-11T08:24:28.356749+0000","last_became_peered":"2024-09-11T08:24:28.356749+0000","last_unstale":"2024-09-11T08:27:38.812277+0000","last_undegraded":"2024-09-11T08:27:38.812277+0000","last_fullsized":"2024-09-11T08:27:38.812277+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'16","last_scrub_stamp":"2024-09-11T08:27:12.918714+0000","last_deep_scrub":"171'12","last_deep_scrub_stamp":"2024-09-11T08:14:56.881299+0000","last_clean_scrub_stamp":"2024-09-11T08:27:12.918714+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:14.209536+0000","scrub_duration":0.0038742020000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":100,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.887882+0000","last_change":"2024-09-11T08:26:06.582469+0000","last_active":"2024-09-11T08:27:36.887882+0000","last_peered":"2024-09-11T08:27:36.887882+0000","last_clean":"2024-09-11T08:27:36.887882+0000","last_became_active":"2024-09-11T08:23:22.686992+0000","last_became_peered":"2024-09-11T08:23:22.686992+0000","last_unstale":"2024-09-11T08:27:36.887882+0000","last_undegraded":"2024-09-11T08:27:36.887882+0000","last_fullsized":"2024-09-11T08:27:36.887882+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:06.582413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:14.613532+0000","last_clean_scrub_stamp":"2024-09-11T08:26:06.582413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014342210000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.4","version":"406'6018","reported_seq":6050,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-11T08:27:43.212936+0000","last_change":"2024-09-11T08:26:27.765910+0000","last_active":"2024-09-11T08:27:43.212936+0000","last_peered":"2024-09-11T08:27:43.212936+0000","last_clean":"2024-09-11T08:27:43.212936+0000","last_became_active":"2024-09-11T08:26:27.765686+0000","last_became_peered":"2024-09-11T08:26:27.765686+0000","last_unstale":"2024-09-11T08:27:43.212936+0000","last_undegraded":"2024-09-11T08:27:43.212936+0000","last_fullsized":"2024-09-11T08:27:43.212936+0000","mapping_epoch":382,"log_start":"388'900","ondisk_log_start":"388'900","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:26:26.740307+0000","objects_scrubbed":0,"log_size":5118,"log_dups_size":0,"ondisk_log_size":5118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394395648,"num_objects":6018,"num_object_clones":0,"num_object_copies":18054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6018,"num_write_kb":385152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"405'21","reported_seq":457,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:39.493002+0000","last_change":"2024-09-11T08:23:37.991614+0000","last_active":"2024-09-11T08:27:39.493002+0000","last_peered":"2024-09-11T08:27:39.493002+0000","last_clean":"2024-09-11T08:27:39.493002+0000","last_became_active":"2024-09-11T08:07:32.054885+0000","last_became_peered":"2024-09-11T08:07:32.054885+0000","last_unstale":"2024-09-11T08:27:39.493002+0000","last_undegraded":"2024-09-11T08:27:39.493002+0000","last_fullsized":"2024-09-11T08:27:39.493002+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'15","last_scrub_stamp":"2024-09-11T08:23:37.991518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:23:37.991518+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0096807339999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":146,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.875853+0000","last_change":"2024-09-11T08:27:03.600628+0000","last_active":"2024-09-11T08:27:36.875853+0000","last_peered":"2024-09-11T08:27:36.875853+0000","last_clean":"2024-09-11T08:27:36.875853+0000","last_became_active":"2024-09-11T08:24:28.359057+0000","last_became_peered":"2024-09-11T08:24:28.359057+0000","last_unstale":"2024-09-11T08:27:36.875853+0000","last_undegraded":"2024-09-11T08:27:36.875853+0000","last_fullsized":"2024-09-11T08:27:36.875853+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:03.600550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:27:03.600550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:14.549415+0000","scrub_duration":0.0052685750000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":471,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.882368+0000","last_change":"2024-09-11T08:27:09.839390+0000","last_active":"2024-09-11T08:27:36.882368+0000","last_peered":"2024-09-11T08:27:36.882368+0000","last_clean":"2024-09-11T08:27:36.882368+0000","last_became_active":"2024-09-11T08:07:34.056606+0000","last_became_peered":"2024-09-11T08:07:34.056606+0000","last_unstale":"2024-09-11T08:27:36.882368+0000","last_undegraded":"2024-09-11T08:27:36.882368+0000","last_fullsized":"2024-09-11T08:27:36.882368+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:09.839220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:27:09.839220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:10.408269+0000","scrub_duration":0.0042163340000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":446,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.879979+0000","last_change":"2024-09-11T08:23:46.203003+0000","last_active":"2024-09-11T08:27:36.879979+0000","last_peered":"2024-09-11T08:27:36.879979+0000","last_clean":"2024-09-11T08:27:36.879979+0000","last_became_active":"2024-09-11T08:15:56.518884+0000","last_became_peered":"2024-09-11T08:15:56.518884+0000","last_unstale":"2024-09-11T08:27:36.879979+0000","last_undegraded":"2024-09-11T08:27:36.879979+0000","last_fullsized":"2024-09-11T08:27:36.879979+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:23:46.202841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:18:32.278257+0000","last_clean_scrub_stamp":"2024-09-11T08:23:46.202841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014919458999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"406'6108","reported_seq":6140,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-11T08:27:43.222208+0000","last_change":"2024-09-11T08:26:28.083356+0000","last_active":"2024-09-11T08:27:43.222208+0000","last_peered":"2024-09-11T08:27:43.222208+0000","last_clean":"2024-09-11T08:27:43.222208+0000","last_became_active":"2024-09-11T08:26:28.083033+0000","last_became_peered":"2024-09-11T08:26:28.083033+0000","last_unstale":"2024-09-11T08:27:43.222208+0000","last_undegraded":"2024-09-11T08:27:43.222208+0000","last_fullsized":"2024-09-11T08:27:43.222208+0000","mapping_epoch":382,"log_start":"388'1000","ondisk_log_start":"388'1000","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:26:26.740307+0000","objects_scrubbed":0,"log_size":5108,"log_dups_size":0,"ondisk_log_size":5108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:27:51.535683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400293888,"num_objects":6108,"num_object_clones":0,"num_object_copies":18324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6108,"num_write_kb":390912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"405'51","reported_seq":609,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:39.840942+0000","last_change":"2024-09-11T08:27:07.886113+0000","last_active":"2024-09-11T08:27:39.840942+0000","last_peered":"2024-09-11T08:27:39.840942+0000","last_clean":"2024-09-11T08:27:39.840942+0000","last_became_active":"2024-09-11T08:15:56.526961+0000","last_became_peered":"2024-09-11T08:15:56.526961+0000","last_unstale":"2024-09-11T08:27:39.840942+0000","last_undegraded":"2024-09-11T08:27:39.840942+0000","last_fullsized":"2024-09-11T08:27:39.840942+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'35","last_scrub_stamp":"2024-09-11T08:27:07.886024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:27:07.886024+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:25.429274+0000","scrub_duration":0.0038597950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":288,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.880286+0000","last_change":"2024-09-11T08:27:02.188913+0000","last_active":"2024-09-11T08:27:36.880286+0000","last_peered":"2024-09-11T08:27:36.880286+0000","last_clean":"2024-09-11T08:27:36.880286+0000","last_became_active":"2024-09-11T08:24:28.353720+0000","last_became_peered":"2024-09-11T08:24:28.353720+0000","last_unstale":"2024-09-11T08:27:36.880286+0000","last_undegraded":"2024-09-11T08:27:36.880286+0000","last_fullsized":"2024-09-11T08:27:36.880286+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:02.188743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:44.385372+0000","last_clean_scrub_stamp":"2024-09-11T08:27:02.188743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:16.282589+0000","scrub_duration":0.002554925,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":501,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.882456+0000","last_change":"2024-09-11T08:27:21.811407+0000","last_active":"2024-09-11T08:27:36.882456+0000","last_peered":"2024-09-11T08:27:36.882456+0000","last_clean":"2024-09-11T08:27:36.882456+0000","last_became_active":"2024-09-11T08:24:28.365792+0000","last_became_peered":"2024-09-11T08:24:28.365792+0000","last_unstale":"2024-09-11T08:27:36.882456+0000","last_undegraded":"2024-09-11T08:27:36.882456+0000","last_fullsized":"2024-09-11T08:27:36.882456+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:21.811323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:06.546573+0000","last_clean_scrub_stamp":"2024-09-11T08:27:21.811323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:27.942264+0000","scrub_duration":0.0043008960000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":248,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.874474+0000","last_change":"2024-09-11T08:24:26.983211+0000","last_active":"2024-09-11T08:27:36.874474+0000","last_peered":"2024-09-11T08:27:36.874474+0000","last_clean":"2024-09-11T08:27:36.874474+0000","last_became_active":"2024-09-11T08:15:56.518169+0000","last_became_peered":"2024-09-11T08:15:56.518169+0000","last_unstale":"2024-09-11T08:27:36.874474+0000","last_undegraded":"2024-09-11T08:27:36.874474+0000","last_fullsized":"2024-09-11T08:27:36.874474+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:26.983134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:24:26.983134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0081131989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"406'6238","reported_seq":6270,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-11T08:27:44.218265+0000","last_change":"2024-09-11T08:26:27.764924+0000","last_active":"2024-09-11T08:27:44.218265+0000","last_peered":"2024-09-11T08:27:44.218265+0000","last_clean":"2024-09-11T08:27:44.218265+0000","last_became_active":"2024-09-11T08:26:27.764636+0000","last_became_peered":"2024-09-11T08:26:27.764636+0000","last_unstale":"2024-09-11T08:27:44.218265+0000","last_undegraded":"2024-09-11T08:27:44.218265+0000","last_fullsized":"2024-09-11T08:27:44.218265+0000","mapping_epoch":382,"log_start":"388'100","ondisk_log_start":"388'100","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:26:26.740307+0000","objects_scrubbed":0,"log_size":6138,"log_dups_size":0,"ondisk_log_size":6138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":408813568,"num_objects":6238,"num_object_clones":0,"num_object_copies":18714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6238,"num_write_kb":399232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"405'29","reported_seq":371,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:38.430450+0000","last_change":"2024-09-11T08:27:22.427676+0000","last_active":"2024-09-11T08:27:38.430450+0000","last_peered":"2024-09-11T08:27:38.430450+0000","last_clean":"2024-09-11T08:27:38.430450+0000","last_became_active":"2024-09-11T08:23:22.686387+0000","last_became_peered":"2024-09-11T08:23:22.686387+0000","last_unstale":"2024-09-11T08:27:38.430450+0000","last_undegraded":"2024-09-11T08:27:38.430450+0000","last_fullsized":"2024-09-11T08:27:38.430450+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'23","last_scrub_stamp":"2024-09-11T08:27:22.427580+0000","last_deep_scrub":"145'17","last_deep_scrub_stamp":"2024-09-11T08:13:08.160848+0000","last_clean_scrub_stamp":"2024-09-11T08:27:22.427580+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:30.114320+0000","scrub_duration":0.17013154699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":258,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.884692+0000","last_change":"2024-09-11T08:25:57.957773+0000","last_active":"2024-09-11T08:27:36.884692+0000","last_peered":"2024-09-11T08:27:36.884692+0000","last_clean":"2024-09-11T08:27:36.884692+0000","last_became_active":"2024-09-11T08:15:56.525845+0000","last_became_peered":"2024-09-11T08:15:56.525845+0000","last_unstale":"2024-09-11T08:27:36.884692+0000","last_undegraded":"2024-09-11T08:27:36.884692+0000","last_fullsized":"2024-09-11T08:27:36.884692+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:25:57.957725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:21:06.262572+0000","last_clean_scrub_stamp":"2024-09-11T08:25:57.957725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012472080000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":451,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.881960+0000","last_change":"2024-09-11T08:27:05.881925+0000","last_active":"2024-09-11T08:27:36.881960+0000","last_peered":"2024-09-11T08:27:36.881960+0000","last_clean":"2024-09-11T08:27:36.881960+0000","last_became_active":"2024-09-11T08:07:34.057479+0000","last_became_peered":"2024-09-11T08:07:34.057479+0000","last_unstale":"2024-09-11T08:27:36.881960+0000","last_undegraded":"2024-09-11T08:27:36.881960+0000","last_fullsized":"2024-09-11T08:27:36.881960+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:05.881761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:27:05.881761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:11.743757+0000","scrub_duration":0.004658255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":467,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.875492+0000","last_change":"2024-09-11T08:27:15.756321+0000","last_active":"2024-09-11T08:27:36.875492+0000","last_peered":"2024-09-11T08:27:36.875492+0000","last_clean":"2024-09-11T08:27:36.875492+0000","last_became_active":"2024-09-11T08:15:56.522624+0000","last_became_peered":"2024-09-11T08:15:56.522624+0000","last_unstale":"2024-09-11T08:27:36.875492+0000","last_undegraded":"2024-09-11T08:27:36.875492+0000","last_fullsized":"2024-09-11T08:27:36.875492+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:15.756244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:27:15.756244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:23.287193+0000","scrub_duration":0.015967552999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"406'6136","reported_seq":6168,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-11T08:27:43.808078+0000","last_change":"2024-09-11T08:26:28.083401+0000","last_active":"2024-09-11T08:27:43.808078+0000","last_peered":"2024-09-11T08:27:43.808078+0000","last_clean":"2024-09-11T08:27:43.808078+0000","last_became_active":"2024-09-11T08:26:28.083190+0000","last_became_peered":"2024-09-11T08:26:28.083190+0000","last_unstale":"2024-09-11T08:27:43.808078+0000","last_undegraded":"2024-09-11T08:27:43.808078+0000","last_fullsized":"2024-09-11T08:27:43.808078+0000","mapping_epoch":382,"log_start":"388'1200","ondisk_log_start":"388'1200","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:26:26.740307+0000","objects_scrubbed":0,"log_size":4936,"log_dups_size":0,"ondisk_log_size":4936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402128896,"num_objects":6136,"num_object_clones":0,"num_object_copies":18408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6136,"num_write_kb":392704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"405'39","reported_seq":509,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:39.302155+0000","last_change":"2024-09-11T08:27:14.203688+0000","last_active":"2024-09-11T08:27:39.302155+0000","last_peered":"2024-09-11T08:27:39.302155+0000","last_clean":"2024-09-11T08:27:39.302155+0000","last_became_active":"2024-09-11T08:24:28.353397+0000","last_became_peered":"2024-09-11T08:24:28.353397+0000","last_unstale":"2024-09-11T08:27:39.302155+0000","last_undegraded":"2024-09-11T08:27:39.302155+0000","last_fullsized":"2024-09-11T08:27:39.302155+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'27","last_scrub_stamp":"2024-09-11T08:27:14.203596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:27:14.203596+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:17.865945+0000","scrub_duration":0.0026965510000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":254,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.880221+0000","last_change":"2024-09-11T08:24:17.089977+0000","last_active":"2024-09-11T08:27:36.880221+0000","last_peered":"2024-09-11T08:27:36.880221+0000","last_clean":"2024-09-11T08:27:36.880221+0000","last_became_active":"2024-09-11T08:15:56.515328+0000","last_became_peered":"2024-09-11T08:15:56.515328+0000","last_unstale":"2024-09-11T08:27:36.880221+0000","last_undegraded":"2024-09-11T08:27:36.880221+0000","last_fullsized":"2024-09-11T08:27:36.880221+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-11T08:24:17.089895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.089895+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019391429999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"209'11","reported_seq":393,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:38.812339+0000","last_change":"2024-09-11T08:25:59.008089+0000","last_active":"2024-09-11T08:27:38.812339+0000","last_peered":"2024-09-11T08:27:38.812339+0000","last_clean":"2024-09-11T08:27:38.812339+0000","last_became_active":"2024-09-11T08:15:56.524513+0000","last_became_peered":"2024-09-11T08:15:56.524513+0000","last_unstale":"2024-09-11T08:27:38.812339+0000","last_undegraded":"2024-09-11T08:27:38.812339+0000","last_fullsized":"2024-09-11T08:27:38.812339+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"209'11","last_scrub_stamp":"2024-09-11T08:25:59.008036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:25:59.008036+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014184130000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":136,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.882511+0000","last_change":"2024-09-11T08:26:06.772735+0000","last_active":"2024-09-11T08:27:36.882511+0000","last_peered":"2024-09-11T08:27:36.882511+0000","last_clean":"2024-09-11T08:27:36.882511+0000","last_became_active":"2024-09-11T08:24:28.360658+0000","last_became_peered":"2024-09-11T08:24:28.360658+0000","last_unstale":"2024-09-11T08:27:36.882511+0000","last_undegraded":"2024-09-11T08:27:36.882511+0000","last_fullsized":"2024-09-11T08:27:36.882511+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:06.772675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:03.484755+0000","last_clean_scrub_stamp":"2024-09-11T08:26:06.772675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013072069999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":495,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.882372+0000","last_change":"2024-09-11T08:26:54.619464+0000","last_active":"2024-09-11T08:27:36.882372+0000","last_peered":"2024-09-11T08:27:36.882372+0000","last_clean":"2024-09-11T08:27:36.882372+0000","last_became_active":"2024-09-11T08:24:28.357401+0000","last_became_peered":"2024-09-11T08:24:28.357401+0000","last_unstale":"2024-09-11T08:27:36.882372+0000","last_undegraded":"2024-09-11T08:27:36.882372+0000","last_fullsized":"2024-09-11T08:27:36.882372+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:54.619370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:26:54.619370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:24.338015+0000","scrub_duration":0.0025718440000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"405'21","reported_seq":507,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:39.406981+0000","last_change":"2024-09-11T08:26:03.664732+0000","last_active":"2024-09-11T08:27:39.406981+0000","last_peered":"2024-09-11T08:27:39.406981+0000","last_clean":"2024-09-11T08:27:39.406981+0000","last_became_active":"2024-09-11T08:15:56.526325+0000","last_became_peered":"2024-09-11T08:15:56.526325+0000","last_unstale":"2024-09-11T08:27:39.406981+0000","last_undegraded":"2024-09-11T08:27:39.406981+0000","last_fullsized":"2024-09-11T08:27:39.406981+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'15","last_scrub_stamp":"2024-09-11T08:26:03.664676+0000","last_deep_scrub":"239'15","last_deep_scrub_stamp":"2024-09-11T08:26:03.664676+0000","last_clean_scrub_stamp":"2024-09-11T08:26:03.664676+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001893128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":136,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.876142+0000","last_change":"2024-09-11T08:25:59.563428+0000","last_active":"2024-09-11T08:27:36.876142+0000","last_peered":"2024-09-11T08:27:36.876142+0000","last_clean":"2024-09-11T08:27:36.876142+0000","last_became_active":"2024-09-11T08:24:28.358300+0000","last_became_peered":"2024-09-11T08:24:28.358300+0000","last_unstale":"2024-09-11T08:27:36.876142+0000","last_undegraded":"2024-09-11T08:27:36.876142+0000","last_fullsized":"2024-09-11T08:27:36.876142+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:25:59.563368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:53.548967+0000","last_clean_scrub_stamp":"2024-09-11T08:25:59.563368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013934489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":383,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.880390+0000","last_change":"2024-09-11T08:26:09.119039+0000","last_active":"2024-09-11T08:27:36.880390+0000","last_peered":"2024-09-11T08:27:36.880390+0000","last_clean":"2024-09-11T08:27:36.880390+0000","last_became_active":"2024-09-11T08:14:24.790678+0000","last_became_peered":"2024-09-11T08:14:24.790678+0000","last_unstale":"2024-09-11T08:27:36.880390+0000","last_undegraded":"2024-09-11T08:27:36.880390+0000","last_fullsized":"2024-09-11T08:27:36.880390+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:09.118973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:26:09.118973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016651439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"405'25","reported_seq":498,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:38.546409+0000","last_change":"2024-09-11T08:22:58.755615+0000","last_active":"2024-09-11T08:27:38.546409+0000","last_peered":"2024-09-11T08:27:38.546409+0000","last_clean":"2024-09-11T08:27:38.546409+0000","last_became_active":"2024-09-11T08:15:56.514793+0000","last_became_peered":"2024-09-11T08:15:56.514793+0000","last_unstale":"2024-09-11T08:27:38.546409+0000","last_undegraded":"2024-09-11T08:27:38.546409+0000","last_fullsized":"2024-09-11T08:27:38.546409+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'19","last_scrub_stamp":"2024-09-11T08:22:58.755559+0000","last_deep_scrub":"84'13","last_deep_scrub_stamp":"2024-09-11T08:08:57.835634+0000","last_clean_scrub_stamp":"2024-09-11T08:22:58.755559+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001670223,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":406,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.874693+0000","last_change":"2024-09-11T08:24:26.028722+0000","last_active":"2024-09-11T08:27:36.874693+0000","last_peered":"2024-09-11T08:27:36.874693+0000","last_clean":"2024-09-11T08:27:36.874693+0000","last_became_active":"2024-09-11T08:14:24.786021+0000","last_became_peered":"2024-09-11T08:14:24.786021+0000","last_unstale":"2024-09-11T08:27:36.874693+0000","last_undegraded":"2024-09-11T08:27:36.874693+0000","last_fullsized":"2024-09-11T08:27:36.874693+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:26.028642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:30.471620+0000","last_clean_scrub_stamp":"2024-09-11T08:24:26.028642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0073996629999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":459,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.876538+0000","last_change":"2024-09-11T08:24:28.006448+0000","last_active":"2024-09-11T08:27:36.876538+0000","last_peered":"2024-09-11T08:27:36.876538+0000","last_clean":"2024-09-11T08:27:36.876538+0000","last_became_active":"2024-09-11T08:15:56.525717+0000","last_became_peered":"2024-09-11T08:15:56.525717+0000","last_unstale":"2024-09-11T08:27:36.876538+0000","last_undegraded":"2024-09-11T08:27:36.876538+0000","last_fullsized":"2024-09-11T08:27:36.876538+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:28.006392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:04.370734+0000","last_clean_scrub_stamp":"2024-09-11T08:24:28.006392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001372453,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":469,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.888633+0000","last_change":"2024-09-11T08:27:00.857665+0000","last_active":"2024-09-11T08:27:36.888633+0000","last_peered":"2024-09-11T08:27:36.888633+0000","last_clean":"2024-09-11T08:27:36.888633+0000","last_became_active":"2024-09-11T08:15:56.525120+0000","last_became_peered":"2024-09-11T08:15:56.525120+0000","last_unstale":"2024-09-11T08:27:36.888633+0000","last_undegraded":"2024-09-11T08:27:36.888633+0000","last_fullsized":"2024-09-11T08:27:36.888633+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:00.857585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:54.184372+0000","last_clean_scrub_stamp":"2024-09-11T08:27:00.857585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:28.826864+0000","scrub_duration":0.009399101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"405'25","reported_seq":194,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:39.542436+0000","last_change":"2024-09-11T08:26:54.105820+0000","last_active":"2024-09-11T08:27:39.542436+0000","last_peered":"2024-09-11T08:27:39.542436+0000","last_clean":"2024-09-11T08:27:39.542436+0000","last_became_active":"2024-09-11T08:24:28.353043+0000","last_became_peered":"2024-09-11T08:24:28.353043+0000","last_unstale":"2024-09-11T08:27:39.542436+0000","last_undegraded":"2024-09-11T08:27:39.542436+0000","last_fullsized":"2024-09-11T08:27:39.542436+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"313'19","last_scrub_stamp":"2024-09-11T08:26:54.105645+0000","last_deep_scrub":"313'19","last_deep_scrub_stamp":"2024-09-11T08:22:03.458402+0000","last_clean_scrub_stamp":"2024-09-11T08:26:54.105645+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:13.089493+0000","scrub_duration":0.0041528440000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":452,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.879412+0000","last_change":"2024-09-11T08:23:39.025692+0000","last_active":"2024-09-11T08:27:36.879412+0000","last_peered":"2024-09-11T08:27:36.879412+0000","last_clean":"2024-09-11T08:27:36.879412+0000","last_became_active":"2024-09-11T08:15:56.513278+0000","last_became_peered":"2024-09-11T08:15:56.513278+0000","last_unstale":"2024-09-11T08:27:36.879412+0000","last_undegraded":"2024-09-11T08:27:36.879412+0000","last_fullsized":"2024-09-11T08:27:36.879412+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:23:39.025620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:52.661591+0000","last_clean_scrub_stamp":"2024-09-11T08:23:39.025620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046140570000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"239'13","reported_seq":690,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:38.812330+0000","last_change":"2024-09-11T08:23:36.270712+0000","last_active":"2024-09-11T08:27:38.812330+0000","last_peered":"2024-09-11T08:27:38.812330+0000","last_clean":"2024-09-11T08:27:38.812330+0000","last_became_active":"2024-09-11T08:17:37.558224+0000","last_became_peered":"2024-09-11T08:17:37.558224+0000","last_unstale":"2024-09-11T08:27:38.812330+0000","last_undegraded":"2024-09-11T08:27:38.812330+0000","last_fullsized":"2024-09-11T08:27:38.812330+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'13","last_scrub_stamp":"2024-09-11T08:23:36.270623+0000","last_deep_scrub":"222'12","last_deep_scrub_stamp":"2024-09-11T08:16:52.592856+0000","last_clean_scrub_stamp":"2024-09-11T08:23:36.270623+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.006222399,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":374,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.887720+0000","last_change":"2024-09-11T08:25:59.575043+0000","last_active":"2024-09-11T08:27:36.887720+0000","last_peered":"2024-09-11T08:27:36.887720+0000","last_clean":"2024-09-11T08:27:36.887720+0000","last_became_active":"2024-09-11T08:24:28.357799+0000","last_became_peered":"2024-09-11T08:24:28.357799+0000","last_unstale":"2024-09-11T08:27:36.887720+0000","last_undegraded":"2024-09-11T08:27:36.887720+0000","last_fullsized":"2024-09-11T08:27:36.887720+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:25:59.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:53.175994+0000","last_clean_scrub_stamp":"2024-09-11T08:25:59.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001608221,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"405'30","reported_seq":333,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:39.525890+0000","last_change":"2024-09-11T08:27:22.846611+0000","last_active":"2024-09-11T08:27:39.525890+0000","last_peered":"2024-09-11T08:27:39.525890+0000","last_clean":"2024-09-11T08:27:39.525890+0000","last_became_active":"2024-09-11T08:17:37.554357+0000","last_became_peered":"2024-09-11T08:17:37.554357+0000","last_unstale":"2024-09-11T08:27:39.525890+0000","last_undegraded":"2024-09-11T08:27:39.525890+0000","last_fullsized":"2024-09-11T08:27:39.525890+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'20","last_scrub_stamp":"2024-09-11T08:27:22.846335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:27:22.846335+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:29.653164+0000","scrub_duration":0.012702801999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":248,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.878253+0000","last_change":"2024-09-11T08:23:42.199613+0000","last_active":"2024-09-11T08:27:36.878253+0000","last_peered":"2024-09-11T08:27:36.878253+0000","last_clean":"2024-09-11T08:27:36.878253+0000","last_became_active":"2024-09-11T08:15:56.518515+0000","last_became_peered":"2024-09-11T08:15:56.518515+0000","last_unstale":"2024-09-11T08:27:36.878253+0000","last_undegraded":"2024-09-11T08:27:36.878253+0000","last_fullsized":"2024-09-11T08:27:36.878253+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:23:42.199527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:23:42.199527+0000","last_clean_scrub_stamp":"2024-09-11T08:23:42.199527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054572529999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"354'13","reported_seq":225,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-11T08:27:40.747811+0000","last_change":"2024-09-11T08:27:09.973738+0000","last_active":"2024-09-11T08:27:40.747811+0000","last_peered":"2024-09-11T08:27:40.747811+0000","last_clean":"2024-09-11T08:27:40.747811+0000","last_became_active":"2024-09-11T08:24:28.353115+0000","last_became_peered":"2024-09-11T08:24:28.353115+0000","last_unstale":"2024-09-11T08:27:40.747811+0000","last_undegraded":"2024-09-11T08:27:40.747811+0000","last_fullsized":"2024-09-11T08:27:40.747811+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'13","last_scrub_stamp":"2024-09-11T08:27:09.973656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:27:09.973656+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:13.849398+0000","scrub_duration":0.0018473879999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":446,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.879384+0000","last_change":"2024-09-11T08:26:10.045998+0000","last_active":"2024-09-11T08:27:36.879384+0000","last_peered":"2024-09-11T08:27:36.879384+0000","last_clean":"2024-09-11T08:27:36.879384+0000","last_became_active":"2024-09-11T08:15:56.513949+0000","last_became_peered":"2024-09-11T08:15:56.513949+0000","last_unstale":"2024-09-11T08:27:36.879384+0000","last_undegraded":"2024-09-11T08:27:36.879384+0000","last_fullsized":"2024-09-11T08:27:36.879384+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:10.045943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:26:10.045943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017620470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"405'27","reported_seq":359,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:39.468228+0000","last_change":"2024-09-11T08:24:25.584115+0000","last_active":"2024-09-11T08:27:39.468228+0000","last_peered":"2024-09-11T08:27:39.468228+0000","last_clean":"2024-09-11T08:27:39.468228+0000","last_became_active":"2024-09-11T08:23:22.688993+0000","last_became_peered":"2024-09-11T08:23:22.688993+0000","last_unstale":"2024-09-11T08:27:39.468228+0000","last_undegraded":"2024-09-11T08:27:39.468228+0000","last_fullsized":"2024-09-11T08:27:39.468228+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'19","last_scrub_stamp":"2024-09-11T08:24:25.584019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:24:25.584019+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047681069999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":297,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.882001+0000","last_change":"2024-09-11T08:26:02.102192+0000","last_active":"2024-09-11T08:27:36.882001+0000","last_peered":"2024-09-11T08:27:36.882001+0000","last_clean":"2024-09-11T08:27:36.882001+0000","last_became_active":"2024-09-11T08:23:22.682569+0000","last_became_peered":"2024-09-11T08:23:22.682569+0000","last_unstale":"2024-09-11T08:27:36.882001+0000","last_undegraded":"2024-09-11T08:27:36.882001+0000","last_fullsized":"2024-09-11T08:27:36.882001+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:02.102138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:26:02.102138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012891669999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":471,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.884264+0000","last_change":"2024-09-11T08:24:28.466446+0000","last_active":"2024-09-11T08:27:36.884264+0000","last_peered":"2024-09-11T08:27:36.884264+0000","last_clean":"2024-09-11T08:27:36.884264+0000","last_became_active":"2024-09-11T08:24:28.355004+0000","last_became_peered":"2024-09-11T08:24:28.355004+0000","last_unstale":"2024-09-11T08:27:36.884264+0000","last_undegraded":"2024-09-11T08:27:36.884264+0000","last_fullsized":"2024-09-11T08:27:36.884264+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:28.466393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:24:28.466393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001587551,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":260,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.878967+0000","last_change":"2024-09-11T08:27:04.846909+0000","last_active":"2024-09-11T08:27:36.878967+0000","last_peered":"2024-09-11T08:27:36.878967+0000","last_clean":"2024-09-11T08:27:36.878967+0000","last_became_active":"2024-09-11T08:24:28.372049+0000","last_became_peered":"2024-09-11T08:24:28.372049+0000","last_unstale":"2024-09-11T08:27:36.878967+0000","last_undegraded":"2024-09-11T08:27:36.878967+0000","last_fullsized":"2024-09-11T08:27:36.878967+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:04.846717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:27:04.846717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:20.813065+0000","scrub_duration":0.0041876129999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"405'25","reported_seq":491,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:38.958188+0000","last_change":"2024-09-11T08:26:07.610504+0000","last_active":"2024-09-11T08:27:38.958188+0000","last_peered":"2024-09-11T08:27:38.958188+0000","last_clean":"2024-09-11T08:27:38.958188+0000","last_became_active":"2024-09-11T08:07:32.049368+0000","last_became_peered":"2024-09-11T08:07:32.049368+0000","last_unstale":"2024-09-11T08:27:38.958188+0000","last_undegraded":"2024-09-11T08:27:38.958188+0000","last_fullsized":"2024-09-11T08:27:38.958188+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'19","last_scrub_stamp":"2024-09-11T08:26:07.610413+0000","last_deep_scrub":"343'19","last_deep_scrub_stamp":"2024-09-11T08:23:44.640077+0000","last_clean_scrub_stamp":"2024-09-11T08:26:07.610413+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020145950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":248,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.880214+0000","last_change":"2024-09-11T08:23:47.170953+0000","last_active":"2024-09-11T08:27:36.880214+0000","last_peered":"2024-09-11T08:27:36.880214+0000","last_clean":"2024-09-11T08:27:36.880214+0000","last_became_active":"2024-09-11T08:15:56.521570+0000","last_became_peered":"2024-09-11T08:15:56.521570+0000","last_unstale":"2024-09-11T08:27:36.880214+0000","last_undegraded":"2024-09-11T08:27:36.880214+0000","last_fullsized":"2024-09-11T08:27:36.880214+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:23:47.170848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:23:47.170848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0107554,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":223,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.879643+0000","last_change":"2024-09-11T08:27:17.688452+0000","last_active":"2024-09-11T08:27:36.879643+0000","last_peered":"2024-09-11T08:27:36.879643+0000","last_clean":"2024-09-11T08:27:36.879643+0000","last_became_active":"2024-09-11T08:17:37.557645+0000","last_became_peered":"2024-09-11T08:17:37.557645+0000","last_unstale":"2024-09-11T08:27:36.879643+0000","last_undegraded":"2024-09-11T08:27:36.879643+0000","last_fullsized":"2024-09-11T08:27:36.879643+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:17.688268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:22:02.568761+0000","last_clean_scrub_stamp":"2024-09-11T08:27:17.688268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:24.641068+0000","scrub_duration":0.0062502779999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":459,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.878588+0000","last_change":"2024-09-11T08:27:03.810727+0000","last_active":"2024-09-11T08:27:36.878588+0000","last_peered":"2024-09-11T08:27:36.878588+0000","last_clean":"2024-09-11T08:27:36.878588+0000","last_became_active":"2024-09-11T08:24:28.363959+0000","last_became_peered":"2024-09-11T08:24:28.363959+0000","last_unstale":"2024-09-11T08:27:36.878588+0000","last_undegraded":"2024-09-11T08:27:36.878588+0000","last_fullsized":"2024-09-11T08:27:36.878588+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:03.810647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:09.973943+0000","last_clean_scrub_stamp":"2024-09-11T08:27:03.810647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:25.890757+0000","scrub_duration":0.0051903160000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"405'27","reported_seq":538,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:39.709829+0000","last_change":"2024-09-11T08:26:58.609528+0000","last_active":"2024-09-11T08:27:39.709829+0000","last_peered":"2024-09-11T08:27:39.709829+0000","last_clean":"2024-09-11T08:27:39.709829+0000","last_became_active":"2024-09-11T08:15:56.520058+0000","last_became_peered":"2024-09-11T08:15:56.520058+0000","last_unstale":"2024-09-11T08:27:39.709829+0000","last_undegraded":"2024-09-11T08:27:39.709829+0000","last_fullsized":"2024-09-11T08:27:39.709829+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'19","last_scrub_stamp":"2024-09-11T08:26:58.609443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:26:58.609443+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:18.718575+0000","scrub_duration":0.017621226,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":351,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.880727+0000","last_change":"2024-09-11T08:24:28.951646+0000","last_active":"2024-09-11T08:27:36.880727+0000","last_peered":"2024-09-11T08:27:36.880727+0000","last_clean":"2024-09-11T08:27:36.880727+0000","last_became_active":"2024-09-11T08:24:28.356974+0000","last_became_peered":"2024-09-11T08:24:28.356974+0000","last_unstale":"2024-09-11T08:27:36.880727+0000","last_undegraded":"2024-09-11T08:27:36.880727+0000","last_fullsized":"2024-09-11T08:27:36.880727+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:28.951592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:24:28.951592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001286382,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":258,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.874540+0000","last_change":"2024-09-11T08:26:03.094530+0000","last_active":"2024-09-11T08:27:36.874540+0000","last_peered":"2024-09-11T08:27:36.874540+0000","last_clean":"2024-09-11T08:27:36.874540+0000","last_became_active":"2024-09-11T08:15:56.523728+0000","last_became_peered":"2024-09-11T08:15:56.523728+0000","last_unstale":"2024-09-11T08:27:36.874540+0000","last_undegraded":"2024-09-11T08:27:36.874540+0000","last_fullsized":"2024-09-11T08:27:36.874540+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:03.094474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:11:34.159053+0000","last_clean_scrub_stamp":"2024-09-11T08:26:03.094474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001235589,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":430,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.875695+0000","last_change":"2024-09-11T08:26:06.683231+0000","last_active":"2024-09-11T08:27:36.875695+0000","last_peered":"2024-09-11T08:27:36.875695+0000","last_clean":"2024-09-11T08:27:36.875695+0000","last_became_active":"2024-09-11T08:07:36.633368+0000","last_became_peered":"2024-09-11T08:07:36.633368+0000","last_unstale":"2024-09-11T08:27:36.875695+0000","last_undegraded":"2024-09-11T08:27:36.875695+0000","last_fullsized":"2024-09-11T08:27:36.875695+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:26:06.683182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:22:50.992034+0000","last_clean_scrub_stamp":"2024-09-11T08:26:06.683182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001432284,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"405'14","reported_seq":454,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:38.456766+0000","last_change":"2024-09-11T08:27:12.683107+0000","last_active":"2024-09-11T08:27:38.456766+0000","last_peered":"2024-09-11T08:27:38.456766+0000","last_clean":"2024-09-11T08:27:38.456766+0000","last_became_active":"2024-09-11T08:07:32.045728+0000","last_became_peered":"2024-09-11T08:07:32.045728+0000","last_unstale":"2024-09-11T08:27:38.456766+0000","last_undegraded":"2024-09-11T08:27:38.456766+0000","last_fullsized":"2024-09-11T08:27:38.456766+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"312'10","last_scrub_stamp":"2024-09-11T08:27:12.683016+0000","last_deep_scrub":"312'10","last_deep_scrub_stamp":"2024-09-11T08:21:55.859866+0000","last_clean_scrub_stamp":"2024-09-11T08:27:12.683016+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:19.488917+0000","scrub_duration":0.017774564,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":462,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.879570+0000","last_change":"2024-09-11T08:24:28.358269+0000","last_active":"2024-09-11T08:27:36.879570+0000","last_peered":"2024-09-11T08:27:36.879570+0000","last_clean":"2024-09-11T08:27:36.879570+0000","last_became_active":"2024-09-11T08:24:28.357978+0000","last_became_peered":"2024-09-11T08:24:28.357978+0000","last_unstale":"2024-09-11T08:27:36.879570+0000","last_undegraded":"2024-09-11T08:27:36.879570+0000","last_fullsized":"2024-09-11T08:27:36.879570+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:22:48.784020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:22:48.784020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0083483829999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":488,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.883374+0000","last_change":"2024-09-11T08:24:09.590462+0000","last_active":"2024-09-11T08:27:36.883374+0000","last_peered":"2024-09-11T08:27:36.883374+0000","last_clean":"2024-09-11T08:27:36.883374+0000","last_became_active":"2024-09-11T08:15:56.528607+0000","last_became_peered":"2024-09-11T08:15:56.528607+0000","last_unstale":"2024-09-11T08:27:36.883374+0000","last_undegraded":"2024-09-11T08:27:36.883374+0000","last_fullsized":"2024-09-11T08:27:36.883374+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:09.590401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:24:09.590401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016035050000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":251,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.874975+0000","last_change":"2024-09-11T08:27:16.194874+0000","last_active":"2024-09-11T08:27:36.874975+0000","last_peered":"2024-09-11T08:27:36.874975+0000","last_clean":"2024-09-11T08:27:36.874975+0000","last_became_active":"2024-09-11T08:15:56.521488+0000","last_became_peered":"2024-09-11T08:15:56.521488+0000","last_unstale":"2024-09-11T08:27:36.874975+0000","last_undegraded":"2024-09-11T08:27:36.874975+0000","last_fullsized":"2024-09-11T08:27:36.874975+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:16.194807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:20:18.827709+0000","last_clean_scrub_stamp":"2024-09-11T08:27:16.194807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:21.088138+0000","scrub_duration":0.0021734469999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":479,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.885195+0000","last_change":"2024-09-11T08:27:17.179442+0000","last_active":"2024-09-11T08:27:36.885195+0000","last_peered":"2024-09-11T08:27:36.885195+0000","last_clean":"2024-09-11T08:27:36.885195+0000","last_became_active":"2024-09-11T08:24:28.354141+0000","last_became_peered":"2024-09-11T08:24:28.354141+0000","last_unstale":"2024-09-11T08:27:36.885195+0000","last_undegraded":"2024-09-11T08:27:36.885195+0000","last_fullsized":"2024-09-11T08:27:36.885195+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:17.179345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:50.585566+0000","last_clean_scrub_stamp":"2024-09-11T08:27:17.179345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:46.719858+0000","scrub_duration":0.0077697699999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":210,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.878842+0000","last_change":"2024-09-11T08:23:01.737938+0000","last_active":"2024-09-11T08:27:36.878842+0000","last_peered":"2024-09-11T08:27:36.878842+0000","last_clean":"2024-09-11T08:27:36.878842+0000","last_became_active":"2024-09-11T08:17:37.554262+0000","last_became_peered":"2024-09-11T08:17:37.554262+0000","last_unstale":"2024-09-11T08:27:36.878842+0000","last_undegraded":"2024-09-11T08:27:36.878842+0000","last_fullsized":"2024-09-11T08:27:36.878842+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:23:01.737888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:23:01.737888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020103790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"405'32","reported_seq":562,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:39.736450+0000","last_change":"2024-09-11T08:26:56.653326+0000","last_active":"2024-09-11T08:27:39.736450+0000","last_peered":"2024-09-11T08:27:39.736450+0000","last_clean":"2024-09-11T08:27:39.736450+0000","last_became_active":"2024-09-11T08:24:28.360937+0000","last_became_peered":"2024-09-11T08:24:28.360937+0000","last_unstale":"2024-09-11T08:27:39.736450+0000","last_undegraded":"2024-09-11T08:27:39.736450+0000","last_fullsized":"2024-09-11T08:27:39.736450+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'24","last_scrub_stamp":"2024-09-11T08:26:56.653232+0000","last_deep_scrub":"288'24","last_deep_scrub_stamp":"2024-09-11T08:20:28.885295+0000","last_clean_scrub_stamp":"2024-09-11T08:26:56.653232+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:04.332641+0000","scrub_duration":0.0098689580000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":286,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-11T08:27:36.882377+0000","last_change":"2024-09-11T08:27:18.733368+0000","last_active":"2024-09-11T08:27:36.882377+0000","last_peered":"2024-09-11T08:27:36.882377+0000","last_clean":"2024-09-11T08:27:36.882377+0000","last_became_active":"2024-09-11T08:15:08.769133+0000","last_became_peered":"2024-09-11T08:15:08.769133+0000","last_unstale":"2024-09-11T08:27:36.882377+0000","last_undegraded":"2024-09-11T08:27:36.882377+0000","last_fullsized":"2024-09-11T08:27:36.882377+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:18.733225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:18:39.206016+0000","last_clean_scrub_stamp":"2024-09-11T08:27:18.733225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:28:44.736552+0000","scrub_duration":0.0057441890000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":6426918912,"num_objects":98067,"num_object_clones":0,"num_object_copies":294201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":98067,"num_write_kb":6276288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19259785216,"data_stored":19259785216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86567,"ondisk_log_size":86567,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102,"ondisk_log_size":102,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1546,"num_read_kb":1303,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":4566,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3425,"internal_metadata":0},"log_size":1066,"ondisk_log_size":1066,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":804,"num_read_kb":799,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":168,"seq":721554505889,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2961404,"kb_used_data":2338864,"kb_used_omap":7,"kb_used_meta":622520,"kb_avail":90783748,"statfs":{"total":95995035648,"available":92962557952,"internally_reserved":0,"allocated":2394996736,"data_stored":2394480010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8027,"internal_metadata":637460645},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:27:41 2024","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":3.754,"15min":2.379},"min":{"1min":0.23899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":2.6989999999999998,"5min":117.57599999999999,"15min":117.57599999999999},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.79800000000000004,"5min":3.726,"15min":2.2770000000000001},"min":{"1min":0.24199999999999999,"5min":0.23400000000000001,"15min":0.219},"max":{"1min":2.5609999999999999,"5min":117.542,"15min":117.542},"last":0.24199999999999999}]},{"osd":1,"last update":"Wed Sep 11 08:27:41 2024","interfaces":[{"interface":"back","average":{"1min":2.4529999999999998,"5min":1.468,"15min":2.0129999999999999},"min":{"1min":0.254,"5min":0.24099999999999999,"15min":0.20899999999999999},"max":{"1min":24.347999999999999,"5min":24.347999999999999,"15min":66.971000000000004},"last":0.92900000000000005},{"interface":"front","average":{"1min":2.536,"5min":1.1020000000000001,"15min":1.8100000000000001},"min":{"1min":0.34399999999999997,"5min":0.25800000000000001,"15min":0.23000000000000001},"max":{"1min":25.382000000000001,"5min":25.382000000000001,"15min":67.185000000000002},"last":0.99299999999999999}]},{"osd":2,"last update":"Wed Sep 11 08:27:41 2024","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.1879999999999999,"15min":1.036},"min":{"1min":0.317,"5min":0.23799999999999999,"15min":0.22700000000000001},"max":{"1min":7.4199999999999999,"5min":19.097000000000001,"15min":19.097000000000001},"last":0.34100000000000003},{"interface":"front","average":{"1min":1.5860000000000001,"5min":1.518,"15min":1.119},"min":{"1min":0.315,"5min":0.27200000000000002,"15min":0.23699999999999999},"max":{"1min":7.7699999999999996,"5min":21.594000000000001,"15min":21.594000000000001},"last":0.89900000000000002}]},{"osd":3,"last update":"Wed Sep 11 08:27:41 2024","interfaces":[{"interface":"back","average":{"1min":2.0720000000000001,"5min":1.1439999999999999,"15min":1.149},"min":{"1min":0.29199999999999998,"5min":0.26600000000000001,"15min":0.251},"max":{"1min":24.315000000000001,"5min":24.315000000000001,"15min":24.315000000000001},"last":0.59699999999999998},{"interface":"front","average":{"1min":2.3460000000000001,"5min":1.194,"15min":1.262},"min":{"1min":0.33400000000000002,"5min":0.27200000000000002,"15min":0.251},"max":{"1min":24.864999999999998,"5min":24.864999999999998,"15min":24.864999999999998},"last":0.371}]},{"osd":4,"last update":"Wed Sep 11 08:27:41 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.96499999999999997,"15min":0.83399999999999996},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.252},"max":{"1min":1.931,"5min":5.7240000000000002,"15min":5.7240000000000002},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.83099999999999996,"15min":0.78300000000000003},"min":{"1min":0.26300000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":2.5670000000000002,"5min":4.2060000000000004,"15min":5.2649999999999997},"last":0.498}]},{"osd":5,"last update":"Wed Sep 11 08:27:37 2024","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":1.1879999999999999,"15min":1.3420000000000001},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":6.3979999999999997,"5min":6.3979999999999997,"15min":6.3979999999999997},"last":0.95499999999999996},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.83999999999999997,"15min":0.91800000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":2.887,"5min":2.887,"15min":2.887},"last":0.56799999999999995}]},{"osd":6,"last update":"Wed Sep 11 08:27:41 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.88600000000000001,"15min":0.83799999999999997},"min":{"1min":0.31900000000000001,"5min":0.26900000000000002,"15min":0.248},"max":{"1min":3.7490000000000001,"5min":5.3230000000000004,"15min":7.1699999999999999},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.83699999999999997,"5min":1.0409999999999999,"15min":0.93400000000000005},"min":{"1min":0.311,"5min":0.311,"15min":0.25700000000000001},"max":{"1min":2.2759999999999998,"5min":6.0439999999999996,"15min":6.0439999999999996},"last":1.0189999999999999}]}]},{"osd":6,"up_from":45,"seq":193273528572,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3697384,"kb_used_data":3103328,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":90047768,"statfs":{"total":95995035648,"available":92208914432,"internally_reserved":0,"allocated":3177807872,"data_stored":3177307353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:27:30 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":6.6020000000000003,"15min":3.2719999999999998},"min":{"1min":0.23300000000000001,"5min":0.224,"15min":0.17999999999999999},"max":{"1min":3.7919999999999998,"5min":331.14800000000002,"15min":331.14800000000002},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.82699999999999996,"5min":6.0709999999999997,"15min":3.133},"min":{"1min":0.35999999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":2.5369999999999999,"5min":331.31,"15min":331.31},"last":0.42199999999999999}]},{"osd":1,"last update":"Wed Sep 11 08:27:30 2024","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.268,"15min":1.3899999999999999},"min":{"1min":0.22900000000000001,"5min":0.223,"15min":0.214},"max":{"1min":3.823,"5min":10.775,"15min":44.658000000000001},"last":4.0730000000000004},{"interface":"front","average":{"1min":1.2490000000000001,"5min":1.083,"15min":1.3740000000000001},"min":{"1min":0.34799999999999998,"5min":0.22700000000000001,"15min":0.22600000000000001},"max":{"1min":3.839,"5min":8.4109999999999996,"15min":44.694000000000003},"last":4.3700000000000001}]},{"osd":2,"last update":"Wed Sep 11 08:27:30 2024","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.452,"15min":1.161},"min":{"1min":0.32000000000000001,"5min":0.23799999999999999,"15min":0.23200000000000001},"max":{"1min":3.528,"5min":21.015000000000001,"15min":39.652999999999999},"last":4.1609999999999996},{"interface":"front","average":{"1min":1.0880000000000001,"5min":1.516,"15min":1.3149999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":6.0860000000000003,"5min":20.98,"15min":38.390999999999998},"last":0.35299999999999998}]},{"osd":3,"last update":"Wed Sep 11 08:27:30 2024","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.51,"15min":1.2350000000000001},"min":{"1min":0.309,"5min":0.253,"15min":0.22700000000000001},"max":{"1min":5.9669999999999996,"5min":10.144,"15min":24.850999999999999},"last":4.8799999999999999},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.288,"15min":1.181},"min":{"1min":0.35399999999999998,"5min":0.28599999999999998,"15min":0.28399999999999997},"max":{"1min":5.9470000000000001,"5min":8.4399999999999995,"15min":24.876000000000001},"last":4.8940000000000001}]},{"osd":4,"last update":"Wed Sep 11 08:27:30 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.88400000000000001,"15min":0.70999999999999996},"min":{"1min":0.26500000000000001,"5min":0.20599999999999999,"15min":0.20499999999999999},"max":{"1min":2.375,"5min":9.7560000000000002,"15min":9.7560000000000002},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.82399999999999995,"15min":0.69499999999999995},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.23699999999999999},"max":{"1min":2.4849999999999999,"5min":4.181,"15min":4.468},"last":0.44900000000000001}]},{"osd":5,"last update":"Wed Sep 11 08:27:30 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":1.4179999999999999,"15min":1.6990000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":3.0640000000000001,"5min":9.7989999999999995,"15min":9.7989999999999995},"last":0.47899999999999998},{"interface":"front","average":{"1min":1.139,"5min":1.337,"15min":1.5089999999999999},"min":{"1min":0.40300000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":3.4620000000000002,"5min":7.2999999999999998,"15min":7.2999999999999998},"last":0.65200000000000002}]},{"osd":7,"last update":"Wed Sep 11 08:27:39 2024","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.3029999999999999,"15min":1.034},"min":{"1min":0.33400000000000002,"5min":0.25900000000000001,"15min":0.253},"max":{"1min":3.8450000000000002,"5min":10.167999999999999,"15min":10.167999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":1.2250000000000001,"5min":0.98299999999999998,"15min":0.82599999999999996},"min":{"1min":0.372,"5min":0.28399999999999997,"15min":0.25700000000000001},"max":{"1min":3.6160000000000001,"5min":5.3579999999999997,"15min":5.3579999999999997},"last":0.57899999999999996}]}]},{"osd":5,"up_from":353,"seq":1516123455528,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1891944,"kb_used_data":1536156,"kb_used_omap":4,"kb_used_meta":355771,"kb_avail":91853208,"statfs":{"total":95995035648,"available":94057684992,"internally_reserved":0,"allocated":1573023744,"data_stored":1572525851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":364310022},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:27:33 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":1.478,"15min":1.762},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":4.0199999999999996,"5min":19.379999999999999,"15min":19.379999999999999},"last":0.26300000000000001},{"interface":"front","average":{"1min":1.0309999999999999,"5min":1.506,"15min":1.7470000000000001},"min":{"1min":0.24199999999999999,"5min":0.218,"15min":0.218},"max":{"1min":5.8979999999999997,"5min":18.238,"15min":18.238},"last":0.33200000000000002}]},{"osd":1,"last update":"Wed Sep 11 08:27:33 2024","interfaces":[{"interface":"back","average":{"1min":3.8929999999999998,"5min":1.5509999999999999,"15min":1.266},"min":{"1min":0.33300000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":39.426000000000002,"5min":39.426000000000002,"15min":39.426000000000002},"last":0.34599999999999997},{"interface":"front","average":{"1min":4.0750000000000002,"5min":1.3380000000000001,"15min":0.91800000000000004},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":39.472000000000001,"5min":39.472000000000001,"15min":39.472000000000001},"last":0.29499999999999998}]},{"osd":2,"last update":"Wed Sep 11 08:27:33 2024","interfaces":[{"interface":"back","average":{"1min":1.448,"5min":1.746,"15min":1.992},"min":{"1min":0.29099999999999998,"5min":0.251,"15min":0.251},"max":{"1min":11.262,"5min":18.350999999999999,"15min":18.350999999999999},"last":1.526},{"interface":"front","average":{"1min":1.492,"5min":1.6559999999999999,"15min":1.8520000000000001},"min":{"1min":0.317,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":11.363,"5min":18.381,"15min":18.381},"last":1.073}]},{"osd":3,"last update":"Wed Sep 11 08:27:33 2024","interfaces":[{"interface":"back","average":{"1min":3.1560000000000001,"5min":1.51,"15min":1.2849999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":38.344000000000001,"5min":38.344000000000001,"15min":38.344000000000001},"last":1.974},{"interface":"front","average":{"1min":2.7650000000000001,"5min":1.246,"15min":0.98499999999999999},"min":{"1min":0.29699999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":38.386000000000003,"5min":38.386000000000003,"15min":38.386000000000003},"last":2.77}]},{"osd":4,"last update":"Wed Sep 11 08:27:33 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.97099999999999997,"15min":1.1539999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":5.8630000000000004,"5min":5.8630000000000004,"15min":5.8630000000000004},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.41899999999999998,"5min":1.0029999999999999,"15min":1.2070000000000001},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.68700000000000006,"5min":5.8979999999999997,"15min":5.8979999999999997},"last":0.36899999999999999}]},{"osd":6,"last update":"Wed Sep 11 08:27:33 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":1.1739999999999999,"15min":1.4339999999999999},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.72799999999999998,"5min":5.9779999999999998,"15min":5.9779999999999998},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.498,"5min":1.3520000000000001,"15min":1.738},"min":{"1min":0.29099999999999998,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.149,"5min":5.6680000000000001,"15min":5.6680000000000001},"last":0.504}]},{"osd":7,"last update":"Wed Sep 11 08:27:33 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":1.0089999999999999,"15min":1.1799999999999999},"min":{"1min":0.29799999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":5.6779999999999999,"5min":5.6779999999999999,"15min":5.6779999999999999},"last":0.93799999999999994},{"interface":"front","average":{"1min":0.50600000000000001,"5min":1.3149999999999999,"15min":1.6659999999999999},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.84599999999999997,"5min":5.8529999999999998,"15min":5.8529999999999998},"last":0.79600000000000004}]}]},{"osd":4,"up_from":31,"seq":133143986438,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2128448,"kb_used_data":1542708,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":91616704,"statfs":{"total":95995035648,"available":93815504896,"internally_reserved":0,"allocated":1579732992,"data_stored":1579212324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:27:42 2024","interfaces":[{"interface":"back","average":{"1min":1.976,"5min":2.8700000000000001,"15min":1.9239999999999999},"min":{"1min":0.25600000000000001,"5min":0.217,"15min":0.19},"max":{"1min":14.891,"5min":64.329999999999998,"15min":64.329999999999998},"last":14.891},{"interface":"front","average":{"1min":0.71099999999999997,"5min":2.7879999999999998,"15min":1.8799999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.221},"max":{"1min":1.645,"5min":66.819999999999993,"15min":66.819999999999993},"last":0.49399999999999999}]},{"osd":1,"last update":"Wed Sep 11 08:27:42 2024","interfaces":[{"interface":"back","average":{"1min":1.778,"5min":1.22,"15min":1.2130000000000001},"min":{"1min":0.38400000000000001,"5min":0.22,"15min":0.214},"max":{"1min":5.1360000000000001,"5min":35.156999999999996,"15min":35.156999999999996},"last":5.1360000000000001},{"interface":"front","average":{"1min":1.546,"5min":1.639,"15min":1.258},"min":{"1min":0.28000000000000003,"5min":0.23599999999999999,"15min":0.23100000000000001},"max":{"1min":5.5030000000000001,"5min":35.088999999999999,"15min":35.088999999999999},"last":0.28000000000000003}]},{"osd":2,"last update":"Wed Sep 11 08:27:42 2024","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":1.921,"15min":1.375},"min":{"1min":0.29299999999999998,"5min":0.25800000000000001,"15min":0.23400000000000001},"max":{"1min":5.3650000000000002,"5min":54.317999999999998,"15min":54.317999999999998},"last":5.3650000000000002},{"interface":"front","average":{"1min":1.478,"5min":2.2400000000000002,"15min":1.4730000000000001},"min":{"1min":0.30299999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":5.3479999999999999,"5min":54.430999999999997,"15min":54.430999999999997},"last":5.3479999999999999}]},{"osd":3,"last update":"Wed Sep 11 08:27:42 2024","interfaces":[{"interface":"back","average":{"1min":1.597,"5min":1.1419999999999999,"15min":1.325},"min":{"1min":0.32000000000000001,"5min":0.27900000000000003,"15min":0.253},"max":{"1min":5.306,"5min":5.9829999999999997,"15min":30.321000000000002},"last":0.54100000000000004},{"interface":"front","average":{"1min":1.371,"5min":0.98099999999999998,"15min":1.1719999999999999},"min":{"1min":0.28599999999999998,"5min":0.26500000000000001,"15min":0.246},"max":{"1min":3.4289999999999998,"5min":4.8680000000000003,"15min":30.620000000000001},"last":0.42899999999999999}]},{"osd":5,"last update":"Wed Sep 11 08:27:33 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":1.0509999999999999,"15min":1.2569999999999999},"min":{"1min":0.28699999999999998,"5min":0.25,"15min":0.25},"max":{"1min":0.996,"5min":5.1050000000000004,"15min":5.1050000000000004},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.55700000000000005,"5min":1.226,"15min":1.4279999999999999},"min":{"1min":0.32700000000000001,"5min":0.255,"15min":0.255},"max":{"1min":1.681,"5min":6.1150000000000002,"15min":6.1150000000000002},"last":0.56200000000000006}]},{"osd":6,"last update":"Wed Sep 11 08:27:33 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.879,"15min":0.83999999999999997},"min":{"1min":0.29499999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":2.6880000000000002,"5min":5.2640000000000002,"15min":48.808},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.747,"5min":0.90900000000000003,"15min":0.89100000000000001},"min":{"1min":0.32800000000000001,"5min":0.251,"15min":0.251},"max":{"1min":2.0859999999999999,"5min":5.3159999999999998,"15min":48.746000000000002},"last":0.44}]},{"osd":7,"last update":"Wed Sep 11 08:26:54 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.85699999999999998,"15min":0.70099999999999996},"min":{"1min":0.29799999999999999,"5min":0.249,"15min":0.249},"max":{"1min":1.7070000000000001,"5min":6.093,"15min":6.093},"last":0.628},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.67300000000000004,"15min":0.73799999999999999},"min":{"1min":0.32900000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.88500000000000001,"5min":4.8259999999999996,"15min":4.8259999999999996},"last":0.47799999999999998}]}]},{"osd":3,"up_from":25,"seq":107374182667,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3625484,"kb_used_data":2732608,"kb_used_omap":1,"kb_used_meta":892862,"kb_avail":90119668,"statfs":{"total":95995035648,"available":92282540032,"internally_reserved":0,"allocated":2798190592,"data_stored":2797657107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":914291032},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:27:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7729999999999999,"5min":1.758,"15min":1.8300000000000001},"min":{"1min":0.29299999999999998,"5min":0.16300000000000001,"15min":0.14999999999999999},"max":{"1min":8.9930000000000003,"5min":24.616,"15min":60.469999999999999},"last":0.28199999999999997},{"interface":"front","average":{"1min":1.27,"5min":1.401,"15min":1.778},"min":{"1min":0.24299999999999999,"5min":0.20599999999999999,"15min":0.17399999999999999},"max":{"1min":8.3279999999999994,"5min":23.640000000000001,"15min":60.488999999999997},"last":4.6529999999999996}]},{"osd":1,"last update":"Wed Sep 11 08:27:33 2024","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.3660000000000001,"15min":1.579},"min":{"1min":0.26500000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":4.8609999999999998,"5min":33.402000000000001,"15min":60.578000000000003},"last":6.1950000000000003},{"interface":"front","average":{"1min":1.5309999999999999,"5min":1.0700000000000001,"15min":1.6619999999999999},"min":{"1min":0.318,"5min":0.19700000000000001,"15min":0.19500000000000001},"max":{"1min":4.306,"5min":5.8040000000000003,"15min":63.674999999999997},"last":1.8480000000000001}]},{"osd":2,"last update":"Wed Sep 11 08:27:33 2024","interfaces":[{"interface":"back","average":{"1min":1.329,"5min":1.9339999999999999,"15min":1.792},"min":{"1min":0.29599999999999999,"5min":0.219,"15min":0.185},"max":{"1min":4.1390000000000002,"5min":67.503,"15min":67.503},"last":0.36799999999999999},{"interface":"front","average":{"1min":1.78,"5min":2.0289999999999999,"15min":1.7090000000000001},"min":{"1min":0.35899999999999999,"5min":0.24299999999999999,"15min":0.19900000000000001},"max":{"1min":9.2940000000000005,"5min":67.313999999999993,"15min":67.313999999999993},"last":0.44400000000000001}]},{"osd":4,"last update":"Wed Sep 11 08:27:41 2024","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.228,"15min":1.4419999999999999},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":4.2149999999999999,"5min":4.274,"15min":61.290999999999997},"last":0.30099999999999999},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.022,"15min":1.234},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":6.46,"5min":6.46,"15min":60.856000000000002},"last":0.40799999999999997}]},{"osd":5,"last update":"Wed Sep 11 08:27:33 2024","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":1.4099999999999999,"15min":1.7},"min":{"1min":0.42099999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":4.7969999999999997,"5min":7.3849999999999998,"15min":7.3849999999999998},"last":4.7240000000000002},{"interface":"front","average":{"1min":1.353,"5min":1.131,"15min":1.2050000000000001},"min":{"1min":0.36699999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":4.1820000000000004,"5min":4.1820000000000004,"15min":4.1820000000000004},"last":0.39500000000000002}]},{"osd":6,"last update":"Wed Sep 11 08:27:33 2024","interfaces":[{"interface":"back","average":{"1min":1.153,"5min":1.345,"15min":1.635},"min":{"1min":0.39700000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":4.8339999999999996,"5min":6.2850000000000001,"15min":60.817},"last":4.6929999999999996},{"interface":"front","average":{"1min":1.403,"5min":1.234,"15min":1.698},"min":{"1min":0.33900000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":4.2409999999999997,"5min":4.407,"15min":61.264000000000003},"last":0.33700000000000002}]},{"osd":7,"last update":"Wed Sep 11 08:26:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.90100000000000002,"15min":1.962},"min":{"1min":0.374,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.86399999999999999,"5min":4.0090000000000003,"15min":60.847000000000001},"last":4.8140000000000001},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.98299999999999998,"15min":2.089},"min":{"1min":0.374,"5min":0.34499999999999997,"15min":0.33800000000000002},"max":{"1min":1.272,"5min":4.5220000000000002,"15min":60.883000000000003},"last":0.46800000000000003}]}]},{"osd":2,"up_from":18,"seq":77309411598,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2938152,"kb_used_data":2348256,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":90807000,"statfs":{"total":95995035648,"available":92986368000,"internally_reserved":0,"allocated":2404614144,"data_stored":2404112554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:27:05 2024","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":4.1180000000000003,"15min":2.7250000000000001},"min":{"1min":0.216,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":4.7080000000000002,"5min":93.971000000000004,"15min":93.971000000000004},"last":3.359},{"interface":"front","average":{"1min":0.88100000000000001,"5min":4.4379999999999997,"15min":3.056},"min":{"1min":0.219,"5min":0.20599999999999999,"15min":0.17999999999999999},"max":{"1min":4.5300000000000002,"5min":93.790999999999997,"15min":93.790999999999997},"last":3.8969999999999998}]},{"osd":1,"last update":"Wed Sep 11 08:27:05 2024","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":1.2010000000000001,"15min":1.607},"min":{"1min":0.20899999999999999,"5min":0.19800000000000001,"15min":0.17599999999999999},"max":{"1min":4.6689999999999996,"5min":7.9560000000000004,"15min":53.347000000000001},"last":3.0139999999999998},{"interface":"front","average":{"1min":0.86399999999999999,"5min":1.2350000000000001,"15min":1.607},"min":{"1min":0.28999999999999998,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":4.5579999999999998,"5min":7.7880000000000003,"15min":53.082999999999998},"last":3.3769999999999998}]},{"osd":3,"last update":"Wed Sep 11 08:27:28 2024","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.3700000000000001,"15min":1.4730000000000001},"min":{"1min":0.32300000000000001,"5min":0.22,"15min":0.21299999999999999},"max":{"1min":3.9470000000000001,"5min":8.0630000000000006,"15min":46.070999999999998},"last":3.0379999999999998},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.4750000000000001,"15min":1.454},"min":{"1min":0.251,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":4.6159999999999997,"5min":7.827,"15min":45.790999999999997},"last":0.35299999999999998}]},{"osd":4,"last update":"Wed Sep 11 08:26:43 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":1.1539999999999999,"15min":1.0680000000000001},"min":{"1min":0.26700000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.871,"5min":5.9470000000000001,"15min":7.2480000000000002},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.51600000000000001,"5min":1.361,"15min":1.1120000000000001},"min":{"1min":0.34200000000000003,"5min":0.28699999999999998,"15min":0.252},"max":{"1min":0.72599999999999998,"5min":7.8970000000000002,"15min":7.8970000000000002},"last":0.42899999999999999}]},{"osd":5,"last update":"Wed Sep 11 08:27:28 2024","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.391,"15min":1.6379999999999999},"min":{"1min":0.32100000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":4.5839999999999996,"5min":4.5839999999999996,"15min":4.5839999999999996},"last":0.40799999999999997},{"interface":"front","average":{"1min":1.0009999999999999,"5min":1.204,"15min":1.3859999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.6779999999999999,"5min":3.9039999999999999,"15min":3.9039999999999999},"last":0.49099999999999999}]},{"osd":6,"last update":"Wed Sep 11 08:27:38 2024","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.349,"15min":1.119},"min":{"1min":0.32300000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":3.992,"5min":8.0350000000000001,"15min":8.0350000000000001},"last":0.29399999999999998},{"interface":"front","average":{"1min":1.464,"5min":1.5049999999999999,"15min":1.177},"min":{"1min":0.38900000000000001,"5min":0.371,"15min":0.32500000000000001},"max":{"1min":4.681,"5min":7.8710000000000004,"15min":7.8710000000000004},"last":0.38400000000000001}]},{"osd":7,"last update":"Wed Sep 11 08:26:43 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":1.087,"15min":1.3069999999999999},"min":{"1min":0.312,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.83799999999999997,"5min":4.7300000000000004,"15min":35.274999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.58699999999999997,"5min":1.27,"15min":1.4370000000000001},"min":{"1min":0.38800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.82599999999999996,"5min":7.9219999999999997,"15min":35.265000000000001},"last":0.46000000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575124,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4014732,"kb_used_data":3128644,"kb_used_omap":1,"kb_used_meta":886078,"kb_avail":89730420,"statfs":{"total":95995035648,"available":91883950080,"internally_reserved":0,"allocated":3203731456,"data_stored":3203194018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":907344216},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,4,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:27:42 2024","interfaces":[{"interface":"back","average":{"1min":2.4969999999999999,"5min":1.3540000000000001,"15min":1.5209999999999999},"min":{"1min":0.25,"5min":0.188,"15min":0.16200000000000001},"max":{"1min":14.045,"5min":14.045,"15min":63.932000000000002},"last":0.33300000000000002},{"interface":"front","average":{"1min":2.0459999999999998,"5min":1.734,"15min":1.8240000000000001},"min":{"1min":0.20100000000000001,"5min":0.182,"15min":0.16800000000000001},"max":{"1min":14.127000000000001,"5min":14.127000000000001,"15min":64.269000000000005},"last":0.27400000000000002}]},{"osd":2,"last update":"Wed Sep 11 08:27:23 2024","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.3799999999999999,"15min":1.3140000000000001},"min":{"1min":0.28599999999999998,"5min":0.19800000000000001,"15min":0.189},"max":{"1min":3.5870000000000002,"5min":20.925999999999998,"15min":20.925999999999998},"last":0.58099999999999996},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.3089999999999999,"15min":1.3},"min":{"1min":0.27000000000000002,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":4.5979999999999999,"5min":20.812000000000001,"15min":20.812000000000001},"last":0.67800000000000005}]},{"osd":3,"last update":"Wed Sep 11 08:27:32 2024","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.0129999999999999,"15min":1.5169999999999999},"min":{"1min":0.31900000000000001,"5min":0.20399999999999999,"15min":0.19700000000000001},"max":{"1min":4.9349999999999996,"5min":8.0579999999999998,"15min":16.783999999999999},"last":0.48899999999999999},{"interface":"front","average":{"1min":1.6839999999999999,"5min":1.196,"15min":1.5249999999999999},"min":{"1min":0.36899999999999999,"5min":0.22800000000000001,"15min":0.22600000000000001},"max":{"1min":4.7850000000000001,"5min":6.5010000000000003,"15min":14.973000000000001},"last":0.39800000000000002}]},{"osd":4,"last update":"Wed Sep 11 08:26:49 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":1.1140000000000001,"15min":1.772},"min":{"1min":0.36199999999999999,"5min":0.254,"15min":0.252},"max":{"1min":1.1899999999999999,"5min":7.4850000000000003,"15min":110.86499999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.622,"5min":1.2969999999999999,"15min":1.8069999999999999},"min":{"1min":0.41799999999999998,"5min":0.28399999999999997,"15min":0.248},"max":{"1min":1.137,"5min":5.7119999999999997,"15min":110.831},"last":0.53900000000000003}]},{"osd":5,"last update":"Wed Sep 11 08:27:32 2024","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.3600000000000001,"15min":1.5509999999999999},"min":{"1min":0.41699999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":3.4700000000000002,"5min":4.1050000000000004,"15min":4.1050000000000004},"last":0.60399999999999998},{"interface":"front","average":{"1min":1.077,"5min":1.335,"15min":1.5089999999999999},"min":{"1min":0.441,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":2.149,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":0.56399999999999995}]},{"osd":6,"last update":"Wed Sep 11 08:27:32 2024","interfaces":[{"interface":"back","average":{"1min":1.5429999999999999,"5min":1.4119999999999999,"15min":1.784},"min":{"1min":0.33200000000000002,"5min":0.315,"15min":0.30499999999999999},"max":{"1min":4.8970000000000002,"5min":8.1059999999999999,"15min":124.31999999999999},"last":0.73299999999999998},{"interface":"front","average":{"1min":1.4750000000000001,"5min":1.327,"15min":1.7709999999999999},"min":{"1min":0.45300000000000001,"5min":0.32300000000000001,"15min":0.29799999999999999},"max":{"1min":3.5030000000000001,"5min":4.8170000000000002,"15min":124.958},"last":3.5310000000000001}]},{"osd":7,"last update":"Wed Sep 11 08:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":1.0800000000000001,"15min":1.6859999999999999},"min":{"1min":0.35499999999999998,"5min":0.318,"15min":0.28599999999999998},"max":{"1min":1.2250000000000001,"5min":7.5229999999999997,"15min":109.149},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.77500000000000002,"5min":1.0920000000000001,"15min":1.8089999999999999},"min":{"1min":0.372,"5min":0.32200000000000001,"15min":0.318},"max":{"1min":3.004,"5min":8.141,"15min":109.309},"last":0.70299999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738648,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2877764,"kb_used_data":1979516,"kb_used_omap":1,"kb_used_meta":898238,"kb_avail":90867388,"statfs":{"total":95995035648,"available":93048205312,"internally_reserved":0,"allocated":2027024384,"data_stored":2026496316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":919796056},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 11 08:27:36 2024","interfaces":[{"interface":"back","average":{"1min":8.7490000000000006,"5min":2.5,"15min":1.6919999999999999},"min":{"1min":0.23899999999999999,"5min":0.159,"15min":0.159},"max":{"1min":98.373999999999995,"5min":98.373999999999995,"15min":98.373999999999995},"last":5.2539999999999996},{"interface":"front","average":{"1min":8.5139999999999993,"5min":2.5859999999999999,"15min":1.9350000000000001},"min":{"1min":0.28599999999999998,"5min":0.17299999999999999,"15min":0.153},"max":{"1min":98.257999999999996,"5min":98.257999999999996,"15min":98.257999999999996},"last":4.5199999999999996}]},{"osd":2,"last update":"Wed Sep 11 08:27:02 2024","interfaces":[{"interface":"back","average":{"1min":2.5840000000000001,"5min":1.8720000000000001,"15min":1.4790000000000001},"min":{"1min":0.22800000000000001,"5min":0.19,"15min":0.19},"max":{"1min":46.417000000000002,"5min":46.417000000000002,"15min":46.417000000000002},"last":0.32800000000000001},{"interface":"front","average":{"1min":2.625,"5min":1.8999999999999999,"15min":1.554},"min":{"1min":0.26000000000000001,"5min":0.22600000000000001,"15min":0.186},"max":{"1min":46.438000000000002,"5min":46.438000000000002,"15min":46.438000000000002},"last":4.492}]},{"osd":3,"last update":"Wed Sep 11 08:27:29 2024","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":2.8159999999999998,"15min":2.04},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":7.633,"5min":172.149,"15min":172.149},"last":4.7489999999999997},{"interface":"front","average":{"1min":1.1299999999999999,"5min":2.8650000000000002,"15min":2.0699999999999998},"min":{"1min":0.26100000000000001,"5min":0.20599999999999999,"15min":0.20300000000000001},"max":{"1min":3.589,"5min":174.899,"15min":174.899},"last":4.9690000000000003}]},{"osd":4,"last update":"Wed Sep 11 08:26:49 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":1.391,"15min":1.228},"min":{"1min":0.308,"5min":0.308,"15min":0.24099999999999999},"max":{"1min":0.89800000000000002,"5min":9.6560000000000006,"15min":9.6560000000000006},"last":4.6890000000000001},{"interface":"front","average":{"1min":0.52600000000000002,"5min":1.304,"15min":1.2130000000000001},"min":{"1min":0.315,"5min":0.26400000000000001,"15min":0.24399999999999999},"max":{"1min":0.79100000000000004,"5min":6.6100000000000003,"15min":8.3510000000000009},"last":4.5330000000000004}]},{"osd":5,"last update":"Wed Sep 11 08:27:34 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.677,"15min":1.909},"min":{"1min":0.34499999999999997,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":7.7030000000000003,"5min":7.7030000000000003,"15min":7.7030000000000003},"last":5.0789999999999997},{"interface":"front","average":{"1min":0.81100000000000005,"5min":1.4610000000000001,"15min":1.698},"min":{"1min":0.311,"5min":0.309,"15min":0.309},"max":{"1min":2.0289999999999999,"5min":7.718,"15min":7.718},"last":0.39200000000000002}]},{"osd":6,"last update":"Wed Sep 11 08:27:34 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.4099999999999999,"15min":1.5369999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":2.903,"5min":9.7029999999999994,"15min":78.522000000000006},"last":4.6050000000000004},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.4450000000000001,"15min":1.486},"min":{"1min":0.40799999999999997,"5min":0.33100000000000002,"15min":0.28299999999999997},"max":{"1min":2.1360000000000001,"5min":6.8150000000000004,"15min":79.808000000000007},"last":0.41999999999999998}]},{"osd":7,"last update":"Wed Sep 11 08:26:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":1.214,"15min":1.262},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.30399999999999999},"max":{"1min":0.95399999999999996,"5min":6.8520000000000003,"15min":7.5350000000000001},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.60299999999999998,"5min":1.3,"15min":1.353},"min":{"1min":0.40699999999999997,"5min":0.32900000000000001,"15min":0.32000000000000001},"max":{"1min":1.393,"5min":6.875,"15min":13.914},"last":0.505}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3425,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2034630656,"data_stored":2034630656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3225550848,"data_stored":3225550848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2417229824,"data_stored":2417229824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2817458176,"data_stored":2817458176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1596588032,"data_stored":1596588032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1577385984,"data_stored":1577385984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3196780544,"data_stored":3196780544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2394161152,"data_stored":2394161152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T08:27:50.023 INFO:tasks.thrashosds.thrasher:Setting 3.4 to [4, 2, 3] 2024-09-11T08:27:50.023 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '3.4', '4', '2', '3'] 2024-09-11T08:27:50.023 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd pg-upmap 3.4 4 2 3 2024-09-11T08:27:50.099 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-11T08:27:50.104 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:50.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:50.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:50.206 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:50.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:50.247 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:50.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:50.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:50 smithi181 ceph-mon[32701]: 5.1 scrub starts 2024-09-11T08:27:50.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:50 smithi181 ceph-mon[32701]: 5.1 scrub ok 2024-09-11T08:27:50.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:50 smithi181 ceph-mon[32701]: 5.17 scrub starts 2024-09-11T08:27:50.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:50 smithi181 ceph-mon[32701]: 5.17 scrub ok 2024-09-11T08:27:50.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:50 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:27:50.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:50 smithi181 ceph-mon[32701]: pgmap v928: 148 pgs: 148 active+clean; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 29 KiB/s rd, 76 MiB/s wr, 1.27k op/s 2024-09-11T08:27:50.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:50 smithi181 ceph-mon[32701]: 5.4 scrub starts 2024-09-11T08:27:50.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:50 smithi181 ceph-mon[32701]: 5.4 scrub ok 2024-09-11T08:27:50.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:50 smithi181 ceph-mon[32701]: 3.19 scrub starts 2024-09-11T08:27:50.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:50 smithi181 ceph-mon[32701]: 3.19 scrub ok 2024-09-11T08:27:50.347 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:27:50.393 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:50.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:50.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:27:50.495 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:50.535 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:50.536 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:27:50.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:50.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:50.636 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:50 smithi150 ceph-mon[29276]: 5.1 scrub starts 2024-09-11T08:27:50.636 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:50 smithi150 ceph-mon[29276]: 5.1 scrub ok 2024-09-11T08:27:50.636 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:50 smithi150 ceph-mon[29276]: 5.17 scrub starts 2024-09-11T08:27:50.637 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:50 smithi150 ceph-mon[29276]: 5.17 scrub ok 2024-09-11T08:27:50.637 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:50 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:27:50.637 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:50 smithi150 ceph-mon[29276]: pgmap v928: 148 pgs: 148 active+clean; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 29 KiB/s rd, 76 MiB/s wr, 1.27k op/s 2024-09-11T08:27:50.637 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:50 smithi150 ceph-mon[29276]: 5.4 scrub starts 2024-09-11T08:27:50.637 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:50 smithi150 ceph-mon[29276]: 5.4 scrub ok 2024-09-11T08:27:50.637 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:50 smithi150 ceph-mon[29276]: 3.19 scrub starts 2024-09-11T08:27:50.637 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:50 smithi150 ceph-mon[29276]: 3.19 scrub ok 2024-09-11T08:27:50.637 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:50 smithi150 ceph-mon[39612]: 5.1 scrub starts 2024-09-11T08:27:50.637 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:50 smithi150 ceph-mon[39612]: 5.1 scrub ok 2024-09-11T08:27:50.637 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:50 smithi150 ceph-mon[39612]: 5.17 scrub starts 2024-09-11T08:27:50.637 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:50 smithi150 ceph-mon[39612]: 5.17 scrub ok 2024-09-11T08:27:50.638 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:50 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:27:50.638 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:50 smithi150 ceph-mon[39612]: pgmap v928: 148 pgs: 148 active+clean; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 29 KiB/s rd, 76 MiB/s wr, 1.27k op/s 2024-09-11T08:27:50.638 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:50 smithi150 ceph-mon[39612]: 5.4 scrub starts 2024-09-11T08:27:50.638 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:50 smithi150 ceph-mon[39612]: 5.4 scrub ok 2024-09-11T08:27:50.638 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:50 smithi150 ceph-mon[39612]: 3.19 scrub starts 2024-09-11T08:27:50.638 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:50 smithi150 ceph-mon[39612]: 3.19 scrub ok 2024-09-11T08:27:50.638 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 70 16 105373 105357 94.0325 65.75 0.0145558 0.0106293 2024-09-11T08:27:50.638 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:50.646 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:27:50.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:50.670 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:50.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:50.771 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:50.801 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:50.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:50.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:50.902 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:27:50.940 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:50.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:50.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:27:51.042 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:51.080 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:51.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:51.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:51.182 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:27:51.213 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:51.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:51.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:27:51.318 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:27:51.351 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:51.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:51.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:27:51.453 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:51.465 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:27:51.483 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:51.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:51.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:51.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:51 smithi181 ceph-mon[32701]: 3.17 scrub starts 2024-09-11T08:27:51.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:51 smithi181 ceph-mon[32701]: 3.17 scrub ok 2024-09-11T08:27:51.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:51 smithi181 ceph-mon[32701]: 5.16 scrub starts 2024-09-11T08:27:51.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:51 smithi181 ceph-mon[32701]: 5.16 scrub ok 2024-09-11T08:27:51.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:51 smithi181 ceph-mon[32701]: osdmap e408: 8 total, 8 up, 8 in 2024-09-11T08:27:51.585 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:51.634 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:51.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:51.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:51.637 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 71 16 106458 106442 93.6632 67.8125 0.0116802 0.0106715 2024-09-11T08:27:51.736 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:27:51.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:51 smithi150 ceph-mon[29276]: 3.17 scrub starts 2024-09-11T08:27:51.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:51 smithi150 ceph-mon[29276]: 3.17 scrub ok 2024-09-11T08:27:51.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:51 smithi150 ceph-mon[29276]: 5.16 scrub starts 2024-09-11T08:27:51.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:51 smithi150 ceph-mon[29276]: 5.16 scrub ok 2024-09-11T08:27:51.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:51 smithi150 ceph-mon[29276]: osdmap e408: 8 total, 8 up, 8 in 2024-09-11T08:27:51.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:51 smithi150 ceph-mon[39612]: 3.17 scrub starts 2024-09-11T08:27:51.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:51 smithi150 ceph-mon[39612]: 3.17 scrub ok 2024-09-11T08:27:51.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:51 smithi150 ceph-mon[39612]: 5.16 scrub starts 2024-09-11T08:27:51.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:51 smithi150 ceph-mon[39612]: 5.16 scrub ok 2024-09-11T08:27:51.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:51 smithi150 ceph-mon[39612]: osdmap e408: 8 total, 8 up, 8 in 2024-09-11T08:27:51.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:51.764 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:51.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:27:51.865 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:27:51.903 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:51.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:51.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:27:52.006 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:52.044 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:52.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:52.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:52.146 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:52.187 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:52.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:52.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:52.289 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:52.313 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:52 smithi150 ceph-mon[29276]: 11.2 scrub starts 2024-09-11T08:27:52.313 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:52 smithi150 ceph-mon[29276]: pgmap v930: 148 pgs: 148 active+clean; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 29 KiB/s rd, 80 MiB/s wr, 1.33k op/s 2024-09-11T08:27:52.314 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:52 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:27:52.314 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:52 smithi150 ceph-mon[39612]: 11.2 scrub starts 2024-09-11T08:27:52.314 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:52 smithi150 ceph-mon[39612]: pgmap v930: 148 pgs: 148 active+clean; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 29 KiB/s rd, 80 MiB/s wr, 1.33k op/s 2024-09-11T08:27:52.314 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:52 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:27:52.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:52.326 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:52.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:52.396 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-11T08:27:52.396 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-11T08:27:52.396 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-09-11T08:27:52.396 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-11T08:27:52.427 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:52.456 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:52.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:52.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:52.558 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:52.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:52 smithi181 ceph-mon[32701]: 11.2 scrub starts 2024-09-11T08:27:52.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:52 smithi181 ceph-mon[32701]: pgmap v930: 148 pgs: 148 active+clean; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 29 KiB/s rd, 80 MiB/s wr, 1.33k op/s 2024-09-11T08:27:52.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:52 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:27:52.582 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:52.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:52.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:52.637 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 72 16 107470 107454 93.2408 63.25 0.00517391 0.01072 2024-09-11T08:27:52.685 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:52.688 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T08:27:52.726 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:52.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:52.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:52.828 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:52.858 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:52.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:52.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:52.939 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:27:52.940 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:27:52.960 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:52.980 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:27:52.981 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:27:53.013 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:27:53.013 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:27:53.022 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:53.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:53.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:53.050 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:27:53.051 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:27:53.080 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:27:53.081 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:27:53.108 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:27:53.108 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:27:53.125 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:53.156 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:27:53.156 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:27:53.165 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:53.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:53.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:53.186 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:27:53.186 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:27:53.268 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:53.296 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:53.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:53.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:53.398 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:27:53.428 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:53.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:53.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:27:53.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:53 smithi150 ceph-mon[29276]: 3.18 scrub starts 2024-09-11T08:27:53.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:53 smithi150 ceph-mon[29276]: 3.18 scrub ok 2024-09-11T08:27:53.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:53 smithi150 ceph-mon[29276]: 6.1c scrub starts 2024-09-11T08:27:53.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:53 smithi150 ceph-mon[29276]: 6.1c scrub ok 2024-09-11T08:27:53.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:53 smithi150 ceph-mon[29276]: 6.1e scrub starts 2024-09-11T08:27:53.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:53 smithi150 ceph-mon[29276]: 6.1e scrub ok 2024-09-11T08:27:53.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:53 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1194006913' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:27:53.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:53 smithi150 ceph-mon[39612]: 3.18 scrub starts 2024-09-11T08:27:53.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:53 smithi150 ceph-mon[39612]: 3.18 scrub ok 2024-09-11T08:27:53.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:53 smithi150 ceph-mon[39612]: 6.1c scrub starts 2024-09-11T08:27:53.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:53 smithi150 ceph-mon[39612]: 6.1c scrub ok 2024-09-11T08:27:53.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:53 smithi150 ceph-mon[39612]: 6.1e scrub starts 2024-09-11T08:27:53.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:53 smithi150 ceph-mon[39612]: 6.1e scrub ok 2024-09-11T08:27:53.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:53 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1194006913' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:27:53.529 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:53.563 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:53.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:53.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:53.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:53 smithi181 ceph-mon[32701]: 3.18 scrub starts 2024-09-11T08:27:53.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:53 smithi181 ceph-mon[32701]: 3.18 scrub ok 2024-09-11T08:27:53.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:53 smithi181 ceph-mon[32701]: 6.1c scrub starts 2024-09-11T08:27:53.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:53 smithi181 ceph-mon[32701]: 6.1c scrub ok 2024-09-11T08:27:53.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:53 smithi181 ceph-mon[32701]: 6.1e scrub starts 2024-09-11T08:27:53.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:53 smithi181 ceph-mon[32701]: 6.1e scrub ok 2024-09-11T08:27:53.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:53 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1194006913' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:27:53.638 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 73 16 108477 108461 92.8258 62.9375 0.0240756 0.0107686 2024-09-11T08:27:53.665 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:53.706 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:53.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:53.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:53.808 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:53.829 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:53.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:53.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:53.931 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:27:53.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:53.974 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:53.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:27:54.010 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-11T08:27:54.076 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:54.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:54.108 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:54.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:54.209 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:27:54.248 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:54.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:54.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:27:54.350 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:27:54.385 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:54.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:54.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:27:54.406 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:27:54.487 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:54.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:54 smithi150 ceph-mon[29276]: 3.5 scrub starts 2024-09-11T08:27:54.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:54 smithi150 ceph-mon[29276]: 3.5 scrub ok 2024-09-11T08:27:54.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:54 smithi150 ceph-mon[29276]: pgmap v931: 148 pgs: 148 active+clean; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 42 MiB/s wr, 678 op/s 2024-09-11T08:27:54.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:54 smithi150 ceph-mon[29276]: 3.9 scrub starts 2024-09-11T08:27:54.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:54 smithi150 ceph-mon[29276]: 3.9 scrub ok 2024-09-11T08:27:54.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:54 smithi150 ceph-mon[29276]: 4.13 scrub starts 2024-09-11T08:27:54.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:54 smithi150 ceph-mon[29276]: 4.13 scrub ok 2024-09-11T08:27:54.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:54 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:27:54.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:54 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:27:54.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:54 smithi150 ceph-mon[39612]: 3.5 scrub starts 2024-09-11T08:27:54.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:54 smithi150 ceph-mon[39612]: 3.5 scrub ok 2024-09-11T08:27:54.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:54 smithi150 ceph-mon[39612]: pgmap v931: 148 pgs: 148 active+clean; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 42 MiB/s wr, 678 op/s 2024-09-11T08:27:54.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:54 smithi150 ceph-mon[39612]: 3.9 scrub starts 2024-09-11T08:27:54.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:54 smithi150 ceph-mon[39612]: 3.9 scrub ok 2024-09-11T08:27:54.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:54 smithi150 ceph-mon[39612]: 4.13 scrub starts 2024-09-11T08:27:54.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:54 smithi150 ceph-mon[39612]: 4.13 scrub ok 2024-09-11T08:27:54.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:54 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:27:54.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:54 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:27:54.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:54.536 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:54.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:54.537 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:54 smithi181 ceph-mon[32701]: 3.5 scrub starts 2024-09-11T08:27:54.538 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:54 smithi181 ceph-mon[32701]: 3.5 scrub ok 2024-09-11T08:27:54.538 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:54 smithi181 ceph-mon[32701]: pgmap v931: 148 pgs: 148 active+clean; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 42 MiB/s wr, 678 op/s 2024-09-11T08:27:54.538 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:54 smithi181 ceph-mon[32701]: 3.9 scrub starts 2024-09-11T08:27:54.538 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:54 smithi181 ceph-mon[32701]: 3.9 scrub ok 2024-09-11T08:27:54.538 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:54 smithi181 ceph-mon[32701]: 4.13 scrub starts 2024-09-11T08:27:54.538 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:54 smithi181 ceph-mon[32701]: 4.13 scrub ok 2024-09-11T08:27:54.539 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:54 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:27:54.539 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:54 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:27:54.637 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:27:54.638 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 74 16 109440 109424 92.3848 60.1875 0.0324195 0.0108193 2024-09-11T08:27:54.680 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:54.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:54.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:27:54.782 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:54.817 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:54.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:54.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:54.919 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:54.956 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:54.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:54.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:55.059 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:55.095 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:55.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:55.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:55.198 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:55.242 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:55.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:55.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:55.322 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:55 smithi150 ceph-mon[39612]: 4.17 scrub starts 2024-09-11T08:27:55.322 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:55 smithi150 ceph-mon[39612]: 4.17 scrub ok 2024-09-11T08:27:55.322 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:55 smithi150 ceph-mon[39612]: 5.6 scrub starts 2024-09-11T08:27:55.322 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:55 smithi150 ceph-mon[39612]: 5.6 scrub ok 2024-09-11T08:27:55.322 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:55 smithi150 ceph-mon[39612]: 5.1a scrub starts 2024-09-11T08:27:55.322 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:55 smithi150 ceph-mon[39612]: 5.1a scrub ok 2024-09-11T08:27:55.322 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:55 smithi150 ceph-mon[39612]: osdmap e409: 8 total, 8 up, 8 in 2024-09-11T08:27:55.322 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:55 smithi150 ceph-mon[39612]: 11.e scrub starts 2024-09-11T08:27:55.322 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:55 smithi150 ceph-mon[39612]: 11.2 scrub ok 2024-09-11T08:27:55.323 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:55 smithi150 ceph-mon[29276]: 4.17 scrub starts 2024-09-11T08:27:55.323 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:55 smithi150 ceph-mon[29276]: 4.17 scrub ok 2024-09-11T08:27:55.323 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:55 smithi150 ceph-mon[29276]: 5.6 scrub starts 2024-09-11T08:27:55.323 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:55 smithi150 ceph-mon[29276]: 5.6 scrub ok 2024-09-11T08:27:55.323 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:55 smithi150 ceph-mon[29276]: 5.1a scrub starts 2024-09-11T08:27:55.323 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:55 smithi150 ceph-mon[29276]: 5.1a scrub ok 2024-09-11T08:27:55.323 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:55 smithi150 ceph-mon[29276]: osdmap e409: 8 total, 8 up, 8 in 2024-09-11T08:27:55.323 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:55 smithi150 ceph-mon[29276]: 11.e scrub starts 2024-09-11T08:27:55.323 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:55 smithi150 ceph-mon[29276]: 11.2 scrub ok 2024-09-11T08:27:55.344 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:55.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:55.380 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:55.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:55.481 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:27:55.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:55.517 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:55.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:27:55.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:55 smithi181 ceph-mon[32701]: 4.17 scrub starts 2024-09-11T08:27:55.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:55 smithi181 ceph-mon[32701]: 4.17 scrub ok 2024-09-11T08:27:55.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:55 smithi181 ceph-mon[32701]: 5.6 scrub starts 2024-09-11T08:27:55.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:55 smithi181 ceph-mon[32701]: 5.6 scrub ok 2024-09-11T08:27:55.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:55 smithi181 ceph-mon[32701]: 5.1a scrub starts 2024-09-11T08:27:55.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:55 smithi181 ceph-mon[32701]: 5.1a scrub ok 2024-09-11T08:27:55.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:55 smithi181 ceph-mon[32701]: osdmap e409: 8 total, 8 up, 8 in 2024-09-11T08:27:55.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:55 smithi181 ceph-mon[32701]: 11.e scrub starts 2024-09-11T08:27:55.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:55 smithi181 ceph-mon[32701]: 11.2 scrub ok 2024-09-11T08:27:55.574 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:27:55 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:27:55] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:27:55.619 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:27:55.638 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 75 16 110545 110529 92.0738 69.0625 0.0105882 0.0108565 2024-09-11T08:27:55.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:55.657 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:55.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:27:55.726 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:27:55.758 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:27:55.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:55.791 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:55.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:27:55.892 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:55.948 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:55.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:55.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:56.050 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:27:56.084 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:56.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:56.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:27:56.186 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:56.235 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:56.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:56.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:56.261 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-11T08:27:56.261 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:27:56.337 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:27:56.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:56.372 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:56.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:27:56.473 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:27:56.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:56 smithi150 ceph-mon[29276]: 4.16 scrub starts 2024-09-11T08:27:56.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:56 smithi150 ceph-mon[29276]: 4.16 scrub ok 2024-09-11T08:27:56.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:56 smithi150 ceph-mon[29276]: 6.13 scrub starts 2024-09-11T08:27:56.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:56 smithi150 ceph-mon[29276]: 6.13 scrub ok 2024-09-11T08:27:56.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:56 smithi150 ceph-mon[29276]: pgmap v933: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.3 KiB/s rd, 80 MiB/s wr, 1.29k op/s 2024-09-11T08:27:56.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:56 smithi150 ceph-mon[29276]: osdmap e410: 8 total, 8 up, 8 in 2024-09-11T08:27:56.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:56 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/786596799' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "3.4", "id": [4, 2, 3]}]: dispatch 2024-09-11T08:27:56.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:56 smithi150 ceph-mon[39612]: 4.16 scrub starts 2024-09-11T08:27:56.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:56 smithi150 ceph-mon[39612]: 4.16 scrub ok 2024-09-11T08:27:56.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:56 smithi150 ceph-mon[39612]: 6.13 scrub starts 2024-09-11T08:27:56.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:56 smithi150 ceph-mon[39612]: 6.13 scrub ok 2024-09-11T08:27:56.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:56 smithi150 ceph-mon[39612]: pgmap v933: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.3 KiB/s rd, 80 MiB/s wr, 1.29k op/s 2024-09-11T08:27:56.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:56 smithi150 ceph-mon[39612]: osdmap e410: 8 total, 8 up, 8 in 2024-09-11T08:27:56.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:56 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/786596799' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "3.4", "id": [4, 2, 3]}]: dispatch 2024-09-11T08:27:56.512 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:56.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:56.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:27:56.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:56 smithi181 ceph-mon[32701]: 4.16 scrub starts 2024-09-11T08:27:56.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:56 smithi181 ceph-mon[32701]: 4.16 scrub ok 2024-09-11T08:27:56.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:56 smithi181 ceph-mon[32701]: 6.13 scrub starts 2024-09-11T08:27:56.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:56 smithi181 ceph-mon[32701]: 6.13 scrub ok 2024-09-11T08:27:56.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:56 smithi181 ceph-mon[32701]: pgmap v933: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.3 KiB/s rd, 80 MiB/s wr, 1.29k op/s 2024-09-11T08:27:56.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:56 smithi181 ceph-mon[32701]: osdmap e410: 8 total, 8 up, 8 in 2024-09-11T08:27:56.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:56 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/786596799' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "3.4", "id": [4, 2, 3]}]: dispatch 2024-09-11T08:27:56.614 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:56.638 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 76 16 111563 111547 91.6995 63.625 0.02133 0.0109009 2024-09-11T08:27:56.644 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T08:27:56.649 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:56.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:56.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:56.751 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:27:56.770 INFO:teuthology.orchestra.run.smithi150.stderr:set 3.4 pg_upmap mapping to [4,2,3] 2024-09-11T08:27:56.782 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:56.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:56.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:27:56.883 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:56.924 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:56.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:56.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:57.005 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-11T08:27:57.025 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:27:57.086 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:57.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:57.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:27:57.187 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:27:57.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:57.229 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:57.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:27:57.330 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:57.370 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:57.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:57.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:57.404 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:27:57.472 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:57.513 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:57.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:57.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:57.615 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:57.638 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 77 16 112631 112615 91.3755 66.75 0.0109035 0.0109394 2024-09-11T08:27:57.644 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:57.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:57.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:57.745 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:57.787 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:57.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:57.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:57.814 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:57 smithi181 ceph-mon[32701]: 4.d scrub starts 2024-09-11T08:27:57.814 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:57 smithi181 ceph-mon[32701]: 4.d scrub ok 2024-09-11T08:27:57.814 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:57 smithi181 ceph-mon[32701]: 6.1 scrub starts 2024-09-11T08:27:57.814 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:57 smithi181 ceph-mon[32701]: 6.1 scrub ok 2024-09-11T08:27:57.814 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:57 smithi181 ceph-mon[32701]: 3.13 scrub starts 2024-09-11T08:27:57.814 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:57 smithi181 ceph-mon[32701]: 3.13 scrub ok 2024-09-11T08:27:57.815 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:57 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/786596799' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "3.4", "id": [4, 2, 3]}]': finished 2024-09-11T08:27:57.815 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:57 smithi181 ceph-mon[32701]: osdmap e411: 8 total, 8 up, 8 in 2024-09-11T08:27:57.815 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:57 smithi181 ceph-mon[32701]: 11.e scrub ok 2024-09-11T08:27:57.815 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:57 smithi181 ceph-mon[32701]: pgmap v936: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 852 B/s rd, 53 MiB/s wr, 848 op/s 2024-09-11T08:27:57.889 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:57.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:57.925 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:57.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:58.026 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:27:58.053 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:57 smithi150 ceph-mon[29276]: 4.d scrub starts 2024-09-11T08:27:58.053 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:57 smithi150 ceph-mon[29276]: 4.d scrub ok 2024-09-11T08:27:58.053 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:57 smithi150 ceph-mon[29276]: 6.1 scrub starts 2024-09-11T08:27:58.053 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:57 smithi150 ceph-mon[29276]: 6.1 scrub ok 2024-09-11T08:27:58.053 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:57 smithi150 ceph-mon[29276]: 3.13 scrub starts 2024-09-11T08:27:58.053 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:57 smithi150 ceph-mon[29276]: 3.13 scrub ok 2024-09-11T08:27:58.053 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:57 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/786596799' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "3.4", "id": [4, 2, 3]}]': finished 2024-09-11T08:27:58.053 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:57 smithi150 ceph-mon[29276]: osdmap e411: 8 total, 8 up, 8 in 2024-09-11T08:27:58.053 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:57 smithi150 ceph-mon[29276]: 11.e scrub ok 2024-09-11T08:27:58.053 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:57 smithi150 ceph-mon[29276]: pgmap v936: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 852 B/s rd, 53 MiB/s wr, 848 op/s 2024-09-11T08:27:58.054 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:57 smithi150 ceph-mon[39612]: 4.d scrub starts 2024-09-11T08:27:58.054 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:57 smithi150 ceph-mon[39612]: 4.d scrub ok 2024-09-11T08:27:58.054 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:57 smithi150 ceph-mon[39612]: 6.1 scrub starts 2024-09-11T08:27:58.054 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:57 smithi150 ceph-mon[39612]: 6.1 scrub ok 2024-09-11T08:27:58.054 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:57 smithi150 ceph-mon[39612]: 3.13 scrub starts 2024-09-11T08:27:58.054 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:57 smithi150 ceph-mon[39612]: 3.13 scrub ok 2024-09-11T08:27:58.054 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:57 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/786596799' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "3.4", "id": [4, 2, 3]}]': finished 2024-09-11T08:27:58.054 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:57 smithi150 ceph-mon[39612]: osdmap e411: 8 total, 8 up, 8 in 2024-09-11T08:27:58.054 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:57 smithi150 ceph-mon[39612]: 11.e scrub ok 2024-09-11T08:27:58.054 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:57 smithi150 ceph-mon[39612]: pgmap v936: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 852 B/s rd, 53 MiB/s wr, 848 op/s 2024-09-11T08:27:58.055 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:58.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:58.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:27:58.157 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:27:58.195 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:58.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:58.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:27:58.297 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:58.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:58.330 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:58.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:58.340 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:27:58.430 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:58.480 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:58.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:58.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:58.581 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:58.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:58.617 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:58.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:58.638 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 78 16 113628 113612 91.0029 62.3125 0.0111728 0.0109835 2024-09-11T08:27:58.673 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:27:58 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:27:58] "GET /metrics HTTP/1.1" 200 224640 "" "Prometheus/2.43.0" 2024-09-11T08:27:58.720 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:27:58.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:58.751 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:58.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:27:58.796 INFO:teuthology.orchestra.run.smithi150.stderr:nodeep-scrub is unset 2024-09-11T08:27:58.852 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:58.889 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:58.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:58.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:58.937 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:58 smithi181 ceph-mon[32701]: osdmap e412: 8 total, 8 up, 8 in 2024-09-11T08:27:58.937 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:58 smithi181 ceph-mon[32701]: 5.14 scrub starts 2024-09-11T08:27:58.937 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:58 smithi181 ceph-mon[32701]: 5.14 scrub ok 2024-09-11T08:27:58.937 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:58 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3898263119' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:27:58.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:58 smithi150 ceph-mon[29276]: osdmap e412: 8 total, 8 up, 8 in 2024-09-11T08:27:58.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:58 smithi150 ceph-mon[29276]: 5.14 scrub starts 2024-09-11T08:27:58.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:58 smithi150 ceph-mon[29276]: 5.14 scrub ok 2024-09-11T08:27:58.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:58 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3898263119' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:27:58.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:58 smithi150 ceph-mon[39612]: osdmap e412: 8 total, 8 up, 8 in 2024-09-11T08:27:58.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:58 smithi150 ceph-mon[39612]: 5.14 scrub starts 2024-09-11T08:27:58.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:58 smithi150 ceph-mon[39612]: 5.14 scrub ok 2024-09-11T08:27:58.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:58 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3898263119' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:27:58.991 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:27:59.036 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:59.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:59.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:27:59.138 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:27:59.168 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:59.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:59.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:27:59.270 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:27:59.307 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:59.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:59.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:27:59.323 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:27:58 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:27:59.408 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:59.447 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:59.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:59.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:59.548 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:27:59.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:59.588 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:59.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:27:59.638 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 79 16 114640 114624 90.6516 63.25 0.011362 0.0110266 2024-09-11T08:27:59.689 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:59.721 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:27:59.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:59.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:27:59.823 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:27:59.877 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:27:59.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:27:59.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:27:59.978 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:27:59.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:59 smithi150 ceph-mon[29276]: 11.9 scrub starts 2024-09-11T08:27:59.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:59 smithi150 ceph-mon[29276]: 11.d scrub starts 2024-09-11T08:27:59.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:59 smithi150 ceph-mon[29276]: 4.f scrub starts 2024-09-11T08:27:59.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:59 smithi150 ceph-mon[29276]: 4.f scrub ok 2024-09-11T08:27:59.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:59 smithi150 ceph-mon[29276]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:27:59.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:59 smithi150 ceph-mon[29276]: Cluster is now healthy 2024-09-11T08:27:59.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:59 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3898263119' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:27:59.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:59 smithi150 ceph-mon[29276]: osdmap e413: 8 total, 8 up, 8 in 2024-09-11T08:27:59.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:59 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:27:59.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:27:59 smithi150 ceph-mon[29276]: pgmap v939: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 6.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-09-11T08:27:59.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:59 smithi150 ceph-mon[39612]: 11.9 scrub starts 2024-09-11T08:27:59.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:59 smithi150 ceph-mon[39612]: 11.d scrub starts 2024-09-11T08:27:59.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:59 smithi150 ceph-mon[39612]: 4.f scrub starts 2024-09-11T08:27:59.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:59 smithi150 ceph-mon[39612]: 4.f scrub ok 2024-09-11T08:27:59.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:59 smithi150 ceph-mon[39612]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:27:59.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:59 smithi150 ceph-mon[39612]: Cluster is now healthy 2024-09-11T08:27:59.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:59 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3898263119' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:27:59.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:59 smithi150 ceph-mon[39612]: osdmap e413: 8 total, 8 up, 8 in 2024-09-11T08:27:59.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:59 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:27:59.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:27:59 smithi150 ceph-mon[39612]: pgmap v939: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 6.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-09-11T08:28:00.006 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:00.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:00.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:28:00.025 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:59 smithi181 ceph-mon[32701]: 11.9 scrub starts 2024-09-11T08:28:00.026 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:59 smithi181 ceph-mon[32701]: 11.d scrub starts 2024-09-11T08:28:00.026 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:59 smithi181 ceph-mon[32701]: 4.f scrub starts 2024-09-11T08:28:00.026 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:59 smithi181 ceph-mon[32701]: 4.f scrub ok 2024-09-11T08:28:00.026 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:59 smithi181 ceph-mon[32701]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:28:00.026 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:59 smithi181 ceph-mon[32701]: Cluster is now healthy 2024-09-11T08:28:00.026 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:59 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3898263119' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:28:00.026 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:59 smithi181 ceph-mon[32701]: osdmap e413: 8 total, 8 up, 8 in 2024-09-11T08:28:00.026 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:59 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:28:00.026 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:27:59 smithi181 ceph-mon[32701]: pgmap v939: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 6.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-09-11T08:28:00.108 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:28:00.129 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:00.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:00.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:28:00.232 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:28:00.260 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:00.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:00.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:28:00.362 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:00.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:00.385 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:00.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:00.486 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:28:00.515 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:00.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:00.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:28:00.617 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:28:00.638 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:2024-09-11T08:28:00.639235+0000 min lat: 0.00120292 max lat: 0.344516 avg lat: 0.011072 2024-09-11T08:28:00.639 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-11T08:28:00.639 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 80 16 115602 115586 90.2701 60.125 0.00692597 0.011072 2024-09-11T08:28:00.646 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:00.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:00.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:28:00.748 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:28:00.780 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:00.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:00.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:28:00.882 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:28:00.938 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:00.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:00.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:28:01.040 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:01.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:00 smithi181 ceph-mon[32701]: 4.c scrub starts 2024-09-11T08:28:01.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:00 smithi181 ceph-mon[32701]: 4.c scrub ok 2024-09-11T08:28:01.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:00 smithi181 ceph-mon[32701]: 3.b deep-scrub starts 2024-09-11T08:28:01.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:00 smithi181 ceph-mon[32701]: 3.b deep-scrub ok 2024-09-11T08:28:01.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:00 smithi181 ceph-mon[32701]: osdmap e414: 8 total, 8 up, 8 in 2024-09-11T08:28:01.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:01.075 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:01.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:01.129 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:00 smithi150 ceph-mon[29276]: 4.c scrub starts 2024-09-11T08:28:01.129 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:00 smithi150 ceph-mon[29276]: 4.c scrub ok 2024-09-11T08:28:01.129 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:00 smithi150 ceph-mon[29276]: 3.b deep-scrub starts 2024-09-11T08:28:01.129 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:00 smithi150 ceph-mon[29276]: 3.b deep-scrub ok 2024-09-11T08:28:01.130 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:00 smithi150 ceph-mon[29276]: osdmap e414: 8 total, 8 up, 8 in 2024-09-11T08:28:01.130 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:00 smithi150 ceph-mon[39612]: 4.c scrub starts 2024-09-11T08:28:01.130 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:00 smithi150 ceph-mon[39612]: 4.c scrub ok 2024-09-11T08:28:01.130 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:00 smithi150 ceph-mon[39612]: 3.b deep-scrub starts 2024-09-11T08:28:01.130 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:00 smithi150 ceph-mon[39612]: 3.b deep-scrub ok 2024-09-11T08:28:01.130 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:00 smithi150 ceph-mon[39612]: osdmap e414: 8 total, 8 up, 8 in 2024-09-11T08:28:01.176 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:01.183 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:28:01.183 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:28:01.209 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:01.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:01.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:01.215 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:28:01.215 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:28:01.246 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:28:01.246 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:28:01.283 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:28:01.283 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:28:01.312 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:01.315 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:28:01.315 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:28:01.348 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:28:01.349 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:28:01.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:01.351 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:01.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:01.378 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:28:01.378 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:28:01.410 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:28:01.411 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:28:01.452 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:01.494 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:01.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:01.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:01.596 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:28:01.630 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:01.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:01.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:28:01.638 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 81 15 116470 116455 89.8262 54.3125 0.0317714 0.0111278 2024-09-11T08:28:01.732 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:28:01.772 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:01.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:01.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:28:01.813 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-11T08:28:01.813 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-09-11T08:28:01.813 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-09-11T08:28:01.813 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-09-11T08:28:01.813 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.813 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.15636.0:196218 10.6 e355/354 10:6517f7ac:::benchmark_data_smithi150_158067_object65974:head v 355'12160, mlcod=341'8187)", 2024-09-11T08:28:01.813 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:24:31.870498+0000", 2024-09-11T08:28:01.813 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 209.938968176, 2024-09-11T08:28:01.813 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.25163597399999998, 2024-09-11T08:28:01.813 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:28:01.814 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:28:01.814 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:28:01.814 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.814 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:28:01.814 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.870498+0000", 2024-09-11T08:28:01.814 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:28:01.814 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.814 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.814 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:28:01.814 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.870498+0000", 2024-09-11T08:28:01.814 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:28:01.814 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.814 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.815 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:28:01.815 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.870496+0000", 2024-09-11T08:28:01.815 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:28:01.815 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.815 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.815 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:28:01.815 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.870507+0000", 2024-09-11T08:28:01.815 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0407999999999999e-05 2024-09-11T08:28:01.815 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.815 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.815 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:28:01.815 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.870521+0000", 2024-09-11T08:28:01.815 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3933e-05 2024-09-11T08:28:01.815 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.815 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.816 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:28:01.816 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.870528+0000", 2024-09-11T08:28:01.816 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.3819999999999998e-06 2024-09-11T08:28:01.816 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.816 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.816 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:28:01.816 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.891667+0000", 2024-09-11T08:28:01.816 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.021139017 2024-09-11T08:28:01.816 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.816 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.816 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:28:01.816 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.891694+0000", 2024-09-11T08:28:01.817 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6893000000000002e-05 2024-09-11T08:28:01.817 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.817 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.817 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:28:01.817 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.913900+0000", 2024-09-11T08:28:01.817 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.022206133999999999 2024-09-11T08:28:01.817 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.817 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.817 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:28:01.817 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:32.122134+0000", 2024-09-11T08:28:01.817 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20823351400000001 2024-09-11T08:28:01.817 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:28:01.818 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:28:01.818 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:28:01.818 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.818 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.818 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.15636.0:196217 10.a e355/354 10:5e111a8d:::benchmark_data_smithi150_158067_object65973:head v 355'12386, mlcod=341'8311)", 2024-09-11T08:28:01.818 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:24:31.876760+0000", 2024-09-11T08:28:01.818 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 209.93270556100001, 2024-09-11T08:28:01.818 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.24763364800000001, 2024-09-11T08:28:01.818 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:28:01.818 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:28:01.818 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:28:01.818 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.818 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:28:01.818 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.876760+0000", 2024-09-11T08:28:01.819 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:28:01.819 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.819 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.819 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:28:01.819 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.876760+0000", 2024-09-11T08:28:01.819 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:28:01.819 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.819 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.819 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:28:01.819 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.876760+0000", 2024-09-11T08:28:01.819 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-09-11T08:28:01.819 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.820 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.820 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:28:01.820 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.876765+0000", 2024-09-11T08:28:01.820 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3510000000000001e-06 2024-09-11T08:28:01.820 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.820 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.820 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:28:01.820 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.876772+0000", 2024-09-11T08:28:01.820 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8580000000000002e-06 2024-09-11T08:28:01.820 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.820 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.820 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:28:01.820 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.876776+0000", 2024-09-11T08:28:01.820 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2309999999999998e-06 2024-09-11T08:28:01.821 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.821 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.821 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:28:01.821 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.877054+0000", 2024-09-11T08:28:01.821 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00027749900000000001 2024-09-11T08:28:01.821 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.821 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.821 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:28:01.821 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.877078+0000", 2024-09-11T08:28:01.821 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4445000000000001e-05 2024-09-11T08:28:01.821 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.821 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.821 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:28:01.821 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:32.124356+0000", 2024-09-11T08:28:01.822 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.24727754599999999 2024-09-11T08:28:01.822 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.822 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.822 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:28:01.822 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:32.124394+0000", 2024-09-11T08:28:01.822 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8068000000000002e-05 2024-09-11T08:28:01.822 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:28:01.822 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:28:01.822 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:28:01.822 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.822 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.823 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.15636.0:196222 10.6 e355/354 10:6d6a1501:::benchmark_data_smithi150_158067_object65978:head v 355'12161, mlcod=341'8187)", 2024-09-11T08:28:01.823 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:24:31.876807+0000", 2024-09-11T08:28:01.823 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 209.93265918, 2024-09-11T08:28:01.823 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.252421334, 2024-09-11T08:28:01.823 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:28:01.823 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:28:01.823 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:28:01.823 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.823 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:28:01.823 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.876807+0000", 2024-09-11T08:28:01.823 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:28:01.823 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.823 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.823 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:28:01.824 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.876807+0000", 2024-09-11T08:28:01.824 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:28:01.824 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.824 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.824 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:28:01.824 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.876806+0000", 2024-09-11T08:28:01.824 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-09-11T08:28:01.824 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.824 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.824 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:28:01.824 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.876810+0000", 2024-09-11T08:28:01.824 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1280000000000001e-06 2024-09-11T08:28:01.824 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.824 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.825 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:28:01.825 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.876817+0000", 2024-09-11T08:28:01.825 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6000000000000003e-06 2024-09-11T08:28:01.825 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.825 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.825 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:28:01.825 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.876821+0000", 2024-09-11T08:28:01.825 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6059999999999999e-06 2024-09-11T08:28:01.825 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.825 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.825 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:28:01.825 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.903777+0000", 2024-09-11T08:28:01.825 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.026956099000000001 2024-09-11T08:28:01.826 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.826 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.826 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:28:01.826 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.903800+0000", 2024-09-11T08:28:01.826 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3419999999999999e-05 2024-09-11T08:28:01.826 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.826 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.826 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:28:01.826 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:32.129192+0000", 2024-09-11T08:28:01.826 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.22539187799999999 2024-09-11T08:28:01.826 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.826 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.826 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:28:01.826 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:32.129228+0000", 2024-09-11T08:28:01.826 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5979999999999998e-05 2024-09-11T08:28:01.826 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:28:01.827 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:28:01.827 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:28:01.827 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.827 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.827 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.15636.0:196219 10.a e355/354 10:5d29fb95:::benchmark_data_smithi150_158067_object65975:head v 355'12387, mlcod=341'8311)", 2024-09-11T08:28:01.827 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:24:31.880885+0000", 2024-09-11T08:28:01.827 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 209.92858118000001, 2024-09-11T08:28:01.827 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.26408515999999999, 2024-09-11T08:28:01.827 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:28:01.827 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:28:01.827 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:28:01.827 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.827 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:28:01.827 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.880885+0000", 2024-09-11T08:28:01.827 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:28:01.828 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.828 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.828 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:28:01.828 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.880885+0000", 2024-09-11T08:28:01.828 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:28:01.828 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.828 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.828 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:28:01.828 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.880883+0000", 2024-09-11T08:28:01.828 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:28:01.828 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.828 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.828 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:28:01.828 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.881249+0000", 2024-09-11T08:28:01.829 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00036611199999999998 2024-09-11T08:28:01.829 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.829 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.829 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:28:01.829 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.881266+0000", 2024-09-11T08:28:01.829 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7090000000000001e-05 2024-09-11T08:28:01.829 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.829 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.829 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:28:01.829 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.881274+0000", 2024-09-11T08:28:01.829 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.3019999999999997e-06 2024-09-11T08:28:01.829 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.829 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.830 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:28:01.830 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.889368+0000", 2024-09-11T08:28:01.830 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0080942770000000004 2024-09-11T08:28:01.830 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.830 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.830 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:28:01.830 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.889392+0000", 2024-09-11T08:28:01.830 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4346e-05 2024-09-11T08:28:01.830 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.830 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.830 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:28:01.830 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:32.144889+0000", 2024-09-11T08:28:01.830 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.25549671800000001 2024-09-11T08:28:01.830 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.830 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.831 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:28:01.831 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:32.144970+0000", 2024-09-11T08:28:01.831 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0729999999999994e-05 2024-09-11T08:28:01.831 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:28:01.831 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:28:01.831 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:28:01.831 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.831 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.831 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.15636.0:196223 10.a e355/354 10:5642a284:::benchmark_data_smithi150_158067_object65979:head v 355'12388, mlcod=341'8311)", 2024-09-11T08:28:01.831 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:24:31.882819+0000", 2024-09-11T08:28:01.831 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 209.926647113, 2024-09-11T08:28:01.831 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.26223830100000001, 2024-09-11T08:28:01.831 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:28:01.832 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:28:01.832 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:28:01.832 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.832 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:28:01.832 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.882819+0000", 2024-09-11T08:28:01.832 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:28:01.832 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.832 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.832 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:28:01.832 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.882819+0000", 2024-09-11T08:28:01.832 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:28:01.832 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.832 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.832 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:28:01.833 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.882817+0000", 2024-09-11T08:28:01.833 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:28:01.833 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.833 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.833 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:28:01.833 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.882826+0000", 2024-09-11T08:28:01.833 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1149999999999994e-06 2024-09-11T08:28:01.833 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.833 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.833 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:28:01.833 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.882837+0000", 2024-09-11T08:28:01.833 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1785e-05 2024-09-11T08:28:01.833 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.833 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.834 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:28:01.834 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.883228+0000", 2024-09-11T08:28:01.834 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00039057099999999998 2024-09-11T08:28:01.834 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.834 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.834 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:28:01.834 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.898778+0000", 2024-09-11T08:28:01.834 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.015550110000000001 2024-09-11T08:28:01.834 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.834 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.834 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:28:01.834 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.898803+0000", 2024-09-11T08:28:01.834 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4796000000000001e-05 2024-09-11T08:28:01.835 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.835 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.835 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:28:01.835 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:32.144976+0000", 2024-09-11T08:28:01.835 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.24617307999999999 2024-09-11T08:28:01.835 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.835 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.835 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:28:01.835 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:32.145057+0000", 2024-09-11T08:28:01.835 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1149000000000005e-05 2024-09-11T08:28:01.835 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:28:01.835 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:28:01.835 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:28:01.835 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.835 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.836 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.15636.0:196232 10.a e355/354 10:5d454f6d:::benchmark_data_smithi150_158067_object65988:head v 355'12389, mlcod=341'8311)", 2024-09-11T08:28:01.836 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:24:31.887751+0000", 2024-09-11T08:28:01.836 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 209.92171472199999, 2024-09-11T08:28:01.836 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.25738095100000002, 2024-09-11T08:28:01.836 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:28:01.836 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:28:01.836 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:28:01.836 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.836 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:28:01.836 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.887751+0000", 2024-09-11T08:28:01.836 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:28:01.836 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.836 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.836 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:28:01.836 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.887751+0000", 2024-09-11T08:28:01.837 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:28:01.837 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.837 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.837 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:28:01.837 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.887748+0000", 2024-09-11T08:28:01.837 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999971 2024-09-11T08:28:01.837 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.837 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.837 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:28:01.837 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.887761+0000", 2024-09-11T08:28:01.837 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.235e-05 2024-09-11T08:28:01.837 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.837 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.838 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:28:01.838 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.887773+0000", 2024-09-11T08:28:01.838 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2396e-05 2024-09-11T08:28:01.838 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.838 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.838 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:28:01.838 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.887780+0000", 2024-09-11T08:28:01.838 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0899999999999999e-06 2024-09-11T08:28:01.838 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.838 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.838 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:28:01.838 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.907119+0000", 2024-09-11T08:28:01.838 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.019338327999999998 2024-09-11T08:28:01.839 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.839 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.839 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:28:01.839 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.907144+0000", 2024-09-11T08:28:01.839 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5341999999999999e-05 2024-09-11T08:28:01.839 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.839 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.839 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:28:01.839 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:32.145059+0000", 2024-09-11T08:28:01.839 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.237915349 2024-09-11T08:28:01.839 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.839 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.840 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:28:01.840 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:32.145132+0000", 2024-09-11T08:28:01.840 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2789999999999999e-05 2024-09-11T08:28:01.840 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:28:01.840 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:28:01.840 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:28:01.840 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.840 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.840 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.15636.0:196235 10.a e355/354 10:5c8ac806:::benchmark_data_smithi150_158067_object65991:head v 355'12390, mlcod=341'8311)", 2024-09-11T08:28:01.840 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:24:31.891664+0000", 2024-09-11T08:28:01.840 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 209.91780144000001, 2024-09-11T08:28:01.840 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.25360851600000001, 2024-09-11T08:28:01.840 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:28:01.841 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:28:01.841 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:28:01.841 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.841 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:28:01.841 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.891664+0000", 2024-09-11T08:28:01.841 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:28:01.841 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.841 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.841 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:28:01.841 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.891664+0000", 2024-09-11T08:28:01.842 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:28:01.842 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.843 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.843 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:28:01.843 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.891663+0000", 2024-09-11T08:28:01.843 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:28:01.843 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.843 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.843 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:28:01.843 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.891674+0000", 2024-09-11T08:28:01.843 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.065e-05 2024-09-11T08:28:01.843 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.843 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.843 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:28:01.843 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.891684+0000", 2024-09-11T08:28:01.844 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.8160000000000005e-06 2024-09-11T08:28:01.844 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.844 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.844 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:28:01.844 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.891690+0000", 2024-09-11T08:28:01.844 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1410000000000002e-06 2024-09-11T08:28:01.844 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.844 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.844 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:28:01.844 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:32.124437+0000", 2024-09-11T08:28:01.844 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.232747228 2024-09-11T08:28:01.844 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.844 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.845 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:28:01.845 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:32.124475+0000", 2024-09-11T08:28:01.845 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7826e-05 2024-09-11T08:28:01.845 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.845 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.845 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:28:01.845 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:32.145192+0000", 2024-09-11T08:28:01.845 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.020717596000000001 2024-09-11T08:28:01.845 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.845 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.845 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:28:01.845 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:32.145273+0000", 2024-09-11T08:28:01.845 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0595999999999999e-05 2024-09-11T08:28:01.845 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:28:01.846 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:28:01.846 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:28:01.846 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.846 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.846 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDPGRecoveryDelete(10.6 e355,354 10:6efc1b9c:::benchmark_data_smithi150_158067_object1794:head,341'8189)", 2024-09-11T08:28:01.846 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:24:31.892040+0000", 2024-09-11T08:28:01.846 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 209.917425506, 2024-09-11T08:28:01.846 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.23010265999999999, 2024-09-11T08:28:01.846 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:28:01.846 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-11T08:28:01.846 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:28:01.846 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.846 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:28:01.847 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.892040+0000", 2024-09-11T08:28:01.847 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:28:01.847 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.847 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.847 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:28:01.847 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.892040+0000", 2024-09-11T08:28:01.847 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:28:01.847 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.847 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.847 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:28:01.847 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.892040+0000", 2024-09-11T08:28:01.847 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:28:01.847 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.848 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.848 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:28:01.848 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.892046+0000", 2024-09-11T08:28:01.848 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9349999999999999e-06 2024-09-11T08:28:01.848 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.848 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.848 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:28:01.848 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.892057+0000", 2024-09-11T08:28:01.848 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1114999999999999e-05 2024-09-11T08:28:01.848 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.848 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.848 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:28:01.848 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.892063+0000", 2024-09-11T08:28:01.848 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.896e-06 2024-09-11T08:28:01.849 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.849 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.849 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:28:01.849 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.913283+0000", 2024-09-11T08:28:01.849 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.021220465000000001 2024-09-11T08:28:01.849 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.849 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.849 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:28:01.849 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.913291+0000", 2024-09-11T08:28:01.849 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6190000000000002e-06 2024-09-11T08:28:01.849 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.850 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.850 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:28:01.850 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:32.122143+0000", 2024-09-11T08:28:01.850 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20885239799999999 2024-09-11T08:28:01.850 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:28:01.850 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:28:01.850 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:28:01.850 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.850 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.850 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.15636.0:196233 10.6 e355/354 10:6b43b83e:::benchmark_data_smithi150_158067_object65989:head v 355'12162, mlcod=341'8187)", 2024-09-11T08:28:01.850 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:24:31.893384+0000", 2024-09-11T08:28:01.851 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 209.91608182799999, 2024-09-11T08:28:01.851 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.26703940199999998, 2024-09-11T08:28:01.851 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:28:01.851 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:28:01.851 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:28:01.851 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.851 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:28:01.851 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.893384+0000", 2024-09-11T08:28:01.851 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:28:01.851 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.851 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.852 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:28:01.852 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.893384+0000", 2024-09-11T08:28:01.852 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:28:01.852 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.852 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.852 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:28:01.852 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.893383+0000", 2024-09-11T08:28:01.852 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:28:01.852 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.852 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.852 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:28:01.853 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.893392+0000", 2024-09-11T08:28:01.853 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.1400000000000006e-06 2024-09-11T08:28:01.853 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.853 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.853 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:28:01.853 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.893404+0000", 2024-09-11T08:28:01.853 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2356e-05 2024-09-11T08:28:01.853 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.853 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.853 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:28:01.853 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.893411+0000", 2024-09-11T08:28:01.853 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9410000000000001e-06 2024-09-11T08:28:01.854 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.854 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.854 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:28:01.854 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:32.122173+0000", 2024-09-11T08:28:01.854 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.228761348 2024-09-11T08:28:01.854 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.854 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.854 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:28:01.854 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:32.122204+0000", 2024-09-11T08:28:01.854 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1089e-05 2024-09-11T08:28:01.854 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.854 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.855 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:28:01.855 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:32.160385+0000", 2024-09-11T08:28:01.855 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.038181447 2024-09-11T08:28:01.855 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.855 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.855 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:28:01.855 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:32.160423+0000", 2024-09-11T08:28:01.855 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8229999999999998e-05 2024-09-11T08:28:01.855 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:28:01.855 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:28:01.855 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:28:01.855 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.856 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.856 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.15636.0:196237 10.6 e355/354 10:61ee5639:::benchmark_data_smithi150_158067_object65993:head v 355'12163, mlcod=341'8187)", 2024-09-11T08:28:01.856 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:24:31.894926+0000", 2024-09-11T08:28:01.856 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 209.914539843, 2024-09-11T08:28:01.856 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.26554819600000001, 2024-09-11T08:28:01.856 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:28:01.856 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:28:01.856 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:28:01.856 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.856 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:28:01.856 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.894926+0000", 2024-09-11T08:28:01.857 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:28:01.857 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.857 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.857 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:28:01.857 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.894926+0000", 2024-09-11T08:28:01.857 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:28:01.857 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.857 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.857 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:28:01.857 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.894925+0000", 2024-09-11T08:28:01.857 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:28:01.857 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.858 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.858 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:28:01.858 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.894935+0000", 2024-09-11T08:28:01.858 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.8020000000000003e-06 2024-09-11T08:28:01.858 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.858 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.858 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:28:01.858 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.894950+0000", 2024-09-11T08:28:01.858 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4949999999999999e-05 2024-09-11T08:28:01.858 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.858 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.858 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:28:01.858 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.894956+0000", 2024-09-11T08:28:01.859 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7299999999999999e-06 2024-09-11T08:28:01.859 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.859 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.859 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:28:01.859 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:32.131491+0000", 2024-09-11T08:28:01.859 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.23653447599999999 2024-09-11T08:28:01.859 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.859 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.859 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:28:01.859 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:32.131536+0000", 2024-09-11T08:28:01.859 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5423999999999997e-05 2024-09-11T08:28:01.859 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.859 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.859 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:28:01.859 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:32.160457+0000", 2024-09-11T08:28:01.860 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.028920646000000001 2024-09-11T08:28:01.860 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.860 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.860 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:28:01.860 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:32.160474+0000", 2024-09-11T08:28:01.860 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7312e-05 2024-09-11T08:28:01.860 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:28:01.860 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:28:01.860 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:28:01.860 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.860 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.860 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.15636.0:196243 10.a e355/354 10:5e4cc00b:::benchmark_data_smithi150_158067_object65999:head v 355'12391, mlcod=341'8311)", 2024-09-11T08:28:01.860 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:24:31.901459+0000", 2024-09-11T08:28:01.860 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 209.908006793, 2024-09-11T08:28:01.861 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.24389323500000001, 2024-09-11T08:28:01.861 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:28:01.861 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:28:01.861 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:28:01.861 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.861 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:28:01.861 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.901459+0000", 2024-09-11T08:28:01.861 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:28:01.861 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.861 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.861 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:28:01.861 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.901459+0000", 2024-09-11T08:28:01.861 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:28:01.861 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.862 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.862 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:28:01.862 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.901458+0000", 2024-09-11T08:28:01.862 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:28:01.862 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.862 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.862 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:28:01.862 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.901468+0000", 2024-09-11T08:28:01.862 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0131e-05 2024-09-11T08:28:01.862 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.862 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.862 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:28:01.862 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.901483+0000", 2024-09-11T08:28:01.863 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4861000000000001e-05 2024-09-11T08:28:01.863 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.863 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.863 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:28:01.863 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.901491+0000", 2024-09-11T08:28:01.863 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.3699999999999997e-06 2024-09-11T08:28:01.863 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.863 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.863 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:28:01.863 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:32.131241+0000", 2024-09-11T08:28:01.863 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.22975014699999999 2024-09-11T08:28:01.863 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.863 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.863 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:28:01.864 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:32.131262+0000", 2024-09-11T08:28:01.864 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0922999999999999e-05 2024-09-11T08:28:01.864 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.864 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.864 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:28:01.864 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:32.145275+0000", 2024-09-11T08:28:01.864 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.014013424 2024-09-11T08:28:01.864 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.864 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.864 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:28:01.864 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:32.145352+0000", 2024-09-11T08:28:01.864 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7333e-05 2024-09-11T08:28:01.864 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:28:01.864 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:28:01.864 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:28:01.865 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.865 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.865 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.15636.0:196244 10.3 e355/354 10:cea07365:::benchmark_data_smithi150_158067_object66000:head v 355'12189, mlcod=341'8099)", 2024-09-11T08:28:01.865 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:24:31.902499+0000", 2024-09-11T08:28:01.865 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 209.906966885, 2024-09-11T08:28:01.865 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.22322567800000001, 2024-09-11T08:28:01.865 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:28:01.865 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:28:01.865 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:28:01.865 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.865 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:28:01.865 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.902499+0000", 2024-09-11T08:28:01.865 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:28:01.865 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.866 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.866 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:28:01.866 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.902499+0000", 2024-09-11T08:28:01.866 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:28:01.866 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.866 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.866 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:28:01.866 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.902498+0000", 2024-09-11T08:28:01.866 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:28:01.866 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.866 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.866 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:28:01.866 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.902507+0000", 2024-09-11T08:28:01.866 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.9390000000000003e-06 2024-09-11T08:28:01.866 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.867 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.867 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:28:01.867 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.902516+0000", 2024-09-11T08:28:01.867 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.789e-06 2024-09-11T08:28:01.867 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.867 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.867 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:28:01.867 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.902522+0000", 2024-09-11T08:28:01.867 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8189999999999997e-06 2024-09-11T08:28:01.867 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.867 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.868 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:28:01.868 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.902564+0000", 2024-09-11T08:28:01.868 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2341e-05 2024-09-11T08:28:01.868 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.868 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.868 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:28:01.868 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.902588+0000", 2024-09-11T08:28:01.868 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4218000000000001e-05 2024-09-11T08:28:01.868 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.868 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.868 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:28:01.868 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:32.125689+0000", 2024-09-11T08:28:01.868 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.22310111399999999 2024-09-11T08:28:01.868 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.869 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.869 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:28:01.869 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:32.125725+0000", 2024-09-11T08:28:01.869 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5503999999999998e-05 2024-09-11T08:28:01.869 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:28:01.869 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:28:01.869 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:28:01.869 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.869 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.869 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.15636.0:196246 10.a e355/354 10:5801044b:::benchmark_data_smithi150_158067_object66002:head v 355'12392, mlcod=341'8311)", 2024-09-11T08:28:01.869 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:24:31.905558+0000", 2024-09-11T08:28:01.870 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 209.90390796400001, 2024-09-11T08:28:01.870 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.23987054099999999, 2024-09-11T08:28:01.870 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:28:01.870 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:28:01.870 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:28:01.870 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.870 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:28:01.870 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.905558+0000", 2024-09-11T08:28:01.870 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:28:01.870 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.870 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.871 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:28:01.871 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.905558+0000", 2024-09-11T08:28:01.871 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:28:01.871 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.871 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.871 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:28:01.871 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.905556+0000", 2024-09-11T08:28:01.871 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:28:01.871 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.871 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.871 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:28:01.871 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.905568+0000", 2024-09-11T08:28:01.871 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1868999999999999e-05 2024-09-11T08:28:01.872 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.872 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.872 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:28:01.872 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.905583+0000", 2024-09-11T08:28:01.872 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4392999999999999e-05 2024-09-11T08:28:01.872 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.872 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.872 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:28:01.872 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.905590+0000", 2024-09-11T08:28:01.872 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7249999999999997e-06 2024-09-11T08:28:01.872 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.872 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.873 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:28:01.873 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:32.138522+0000", 2024-09-11T08:28:01.873 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.23293195899999999 2024-09-11T08:28:01.873 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.873 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.873 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:28:01.873 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:32.138544+0000", 2024-09-11T08:28:01.873 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1739999999999999e-05 2024-09-11T08:28:01.873 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.873 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.873 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:28:01.873 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:32.145356+0000", 2024-09-11T08:28:01.873 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0068124370000000002 2024-09-11T08:28:01.873 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.874 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.874 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:28:01.874 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:32.145428+0000", 2024-09-11T08:28:01.874 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2002999999999995e-05 2024-09-11T08:28:01.874 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:28:01.874 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:28:01.874 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:28:01.874 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.874 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.874 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.15636.0:196247 10.3 e355/354 10:c43df253:::benchmark_data_smithi150_158067_object66003:head v 355'12190, mlcod=341'8099)", 2024-09-11T08:28:01.874 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:24:31.906593+0000", 2024-09-11T08:28:01.874 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 209.902872508, 2024-09-11T08:28:01.874 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.21915532099999999, 2024-09-11T08:28:01.875 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:28:01.875 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:28:01.875 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:28:01.875 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.875 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:28:01.875 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.906593+0000", 2024-09-11T08:28:01.875 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:28:01.875 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.875 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.875 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:28:01.875 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.906593+0000", 2024-09-11T08:28:01.875 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:28:01.875 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.876 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.876 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:28:01.876 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.906592+0000", 2024-09-11T08:28:01.876 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:28:01.876 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.876 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.876 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:28:01.876 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.906603+0000", 2024-09-11T08:28:01.876 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1175000000000001e-05 2024-09-11T08:28:01.876 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.876 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.876 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:28:01.876 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.906630+0000", 2024-09-11T08:28:01.877 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6896000000000001e-05 2024-09-11T08:28:01.877 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.877 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.877 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:28:01.877 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.906639+0000", 2024-09-11T08:28:01.877 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0280000000000006e-06 2024-09-11T08:28:01.877 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.877 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.877 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:28:01.877 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.911714+0000", 2024-09-11T08:28:01.877 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0050747880000000002 2024-09-11T08:28:01.877 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.878 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.878 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:28:01.878 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.911742+0000", 2024-09-11T08:28:01.878 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7682e-05 2024-09-11T08:28:01.878 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.878 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.878 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:28:01.878 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:32.125728+0000", 2024-09-11T08:28:01.878 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.21398619799999999 2024-09-11T08:28:01.878 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.878 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.878 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:28:01.879 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:32.125749+0000", 2024-09-11T08:28:01.879 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0879000000000001e-05 2024-09-11T08:28:01.879 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:28:01.879 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:28:01.879 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:28:01.879 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.879 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.879 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.15636.0:196251 10.a e355/354 10:573ffd81:::benchmark_data_smithi150_158067_object66007:head v 355'12393, mlcod=341'8311)", 2024-09-11T08:28:01.879 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:24:31.913429+0000", 2024-09-11T08:28:01.879 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 209.89603727599999, 2024-09-11T08:28:01.879 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.24156082800000001, 2024-09-11T08:28:01.879 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:28:01.879 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:28:01.879 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:28:01.879 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.880 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:28:01.880 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.913429+0000", 2024-09-11T08:28:01.880 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:28:01.880 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.880 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.880 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:28:01.880 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.913429+0000", 2024-09-11T08:28:01.880 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:28:01.880 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.880 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.880 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:28:01.880 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.913428+0000", 2024-09-11T08:28:01.881 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:28:01.881 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.881 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.881 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:28:01.881 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.913868+0000", 2024-09-11T08:28:01.881 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00044025800000000001 2024-09-11T08:28:01.881 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.881 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.881 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:28:01.881 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.913880+0000", 2024-09-11T08:28:01.881 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2481e-05 2024-09-11T08:28:01.881 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.882 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.882 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:28:01.882 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.913888+0000", 2024-09-11T08:28:01.882 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5769999999999996e-06 2024-09-11T08:28:01.882 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.882 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.882 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:28:01.882 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:32.145503+0000", 2024-09-11T08:28:01.882 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.23161479099999999 2024-09-11T08:28:01.882 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.882 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.882 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:28:01.883 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:32.145537+0000", 2024-09-11T08:28:01.883 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.417e-05 2024-09-11T08:28:01.883 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.883 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.883 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:28:01.883 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:32.154819+0000", 2024-09-11T08:28:01.883 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0092820769999999997 2024-09-11T08:28:01.883 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.883 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.883 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:28:01.883 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:32.154989+0000", 2024-09-11T08:28:01.883 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017054599999999999 2024-09-11T08:28:01.883 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:28:01.883 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:28:01.884 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:28:01.884 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.884 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.884 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.15636.0:196254 10.6 e355/354 10:67a5dd0d:::benchmark_data_smithi150_158067_object66010:head v 355'12164, mlcod=341'8188)", 2024-09-11T08:28:01.884 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:24:31.918206+0000", 2024-09-11T08:28:01.884 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 209.891260015, 2024-09-11T08:28:01.884 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.24228917599999999, 2024-09-11T08:28:01.884 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:28:01.884 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:28:01.884 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:28:01.884 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.885 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:28:01.885 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.918206+0000", 2024-09-11T08:28:01.885 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:28:01.885 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.885 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.885 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:28:01.885 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.918206+0000", 2024-09-11T08:28:01.885 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:28:01.885 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.885 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.885 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:28:01.885 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.918205+0000", 2024-09-11T08:28:01.885 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:28:01.886 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.886 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.886 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:28:01.886 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:31.918213+0000", 2024-09-11T08:28:01.886 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6510000000000001e-06 2024-09-11T08:28:01.886 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.886 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.886 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:28:01.886 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:32.122119+0000", 2024-09-11T08:28:01.886 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20390652100000001 2024-09-11T08:28:01.886 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.886 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.887 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:28:01.887 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:32.122130+0000", 2024-09-11T08:28:01.887 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0899e-05 2024-09-11T08:28:01.887 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.887 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.887 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:28:01.887 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:32.137531+0000", 2024-09-11T08:28:01.887 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.015400707 2024-09-11T08:28:01.887 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.887 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.887 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:28:01.887 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:32.137555+0000", 2024-09-11T08:28:01.887 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3964999999999998e-05 2024-09-11T08:28:01.888 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.888 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.888 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:28:01.888 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:32.160479+0000", 2024-09-11T08:28:01.888 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0229245 2024-09-11T08:28:01.888 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.888 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.888 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:28:01.888 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:28:01.889 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:32.160495+0000", 2024-09-11T08:28:01.889 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5953000000000001e-05 2024-09-11T08:28:01.890 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:28:01.890 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:28:01.890 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:28:01.890 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.890 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.890 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.15636.0:199908 10.7 e356/354 10:eaf86491:::benchmark_data_smithi150_158067_object69664:head v 356'12643, mlcod=339'8276)", 2024-09-11T08:28:01.890 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:24:36.481355+0000", 2024-09-11T08:28:01.890 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 205.328111208, 2024-09-11T08:28:01.890 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.079579400999999994, 2024-09-11T08:28:01.890 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:28:01.890 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:28:01.890 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:28:01.891 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.891 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:28:01.891 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.481355+0000", 2024-09-11T08:28:01.891 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:28:01.891 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.891 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.891 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:28:01.891 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.481355+0000", 2024-09-11T08:28:01.891 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:28:01.891 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.891 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.891 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:28:01.891 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.481354+0000", 2024-09-11T08:28:01.891 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-09-11T08:28:01.892 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.892 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.892 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:28:01.892 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.481358+0000", 2024-09-11T08:28:01.892 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3500000000000001e-06 2024-09-11T08:28:01.892 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.892 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.892 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:28:01.892 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.481363+0000", 2024-09-11T08:28:01.892 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.899e-06 2024-09-11T08:28:01.892 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.892 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.892 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:28:01.892 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.481366+0000", 2024-09-11T08:28:01.893 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2370000000000001e-06 2024-09-11T08:28:01.893 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.893 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.893 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:28:01.893 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.505805+0000", 2024-09-11T08:28:01.893 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.024438794 2024-09-11T08:28:01.893 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.893 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.893 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:28:01.893 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.505828+0000", 2024-09-11T08:28:01.893 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3383e-05 2024-09-11T08:28:01.893 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.893 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.894 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:28:01.894 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.560697+0000", 2024-09-11T08:28:01.894 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.054868900999999998 2024-09-11T08:28:01.894 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.894 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.894 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:28:01.894 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:24:36.560934+0000", 2024-09-11T08:28:01.894 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00023710800000000001 2024-09-11T08:28:01.894 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:28:01.894 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:28:01.894 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:28:01.894 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.894 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.894 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.25799.0:74361 11.a e398/382 11:5fd4b742:::benchmark_data_smithi150_187410_object74360:head v 398'4699, mlcod=398'4695)", 2024-09-11T08:28:01.894 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:27:22.109814+0000", 2024-09-11T08:28:01.895 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 39.699651649000003, 2024-09-11T08:28:01.895 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.314858786, 2024-09-11T08:28:01.895 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:28:01.895 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:28:01.895 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:28:01.895 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.895 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:28:01.895 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.109814+0000", 2024-09-11T08:28:01.895 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:28:01.895 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.895 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.895 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:28:01.895 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.109814+0000", 2024-09-11T08:28:01.895 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:28:01.895 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.896 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.896 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:28:01.896 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.109813+0000", 2024-09-11T08:28:01.896 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:28:01.896 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.896 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.896 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:28:01.896 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.109834+0000", 2024-09-11T08:28:01.896 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.054e-05 2024-09-11T08:28:01.896 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.896 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.896 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:28:01.896 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.109842+0000", 2024-09-11T08:28:01.896 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2570000000000007e-06 2024-09-11T08:28:01.897 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.897 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.897 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:28:01.897 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.109848+0000", 2024-09-11T08:28:01.897 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.981e-06 2024-09-11T08:28:01.897 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.897 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.897 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:28:01.897 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.109877+0000", 2024-09-11T08:28:01.897 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9189000000000001e-05 2024-09-11T08:28:01.897 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.897 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.897 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:28:01.897 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.109895+0000", 2024-09-11T08:28:01.898 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7736000000000001e-05 2024-09-11T08:28:01.898 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.898 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.898 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:28:01.898 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.424601+0000", 2024-09-11T08:28:01.898 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.31470593099999999 2024-09-11T08:28:01.898 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.898 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.898 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:28:01.898 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.424673+0000", 2024-09-11T08:28:01.898 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1895000000000002e-05 2024-09-11T08:28:01.898 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:28:01.898 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:28:01.898 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:28:01.898 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.898 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.899 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.25799.0:74366 11.7 e398/382 11:ea35c452:::benchmark_data_smithi150_187410_object74365:head v 398'4649, mlcod=398'4645)", 2024-09-11T08:28:01.899 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:27:22.113713+0000", 2024-09-11T08:28:01.899 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 39.695752480000003, 2024-09-11T08:28:01.899 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.31713921, 2024-09-11T08:28:01.899 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:28:01.899 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:28:01.899 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:28:01.899 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.899 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:28:01.899 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.113713+0000", 2024-09-11T08:28:01.899 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:28:01.899 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.899 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.899 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:28:01.900 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.113713+0000", 2024-09-11T08:28:01.900 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:28:01.900 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.900 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.900 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:28:01.900 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.113712+0000", 2024-09-11T08:28:01.900 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-09-11T08:28:01.900 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.900 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.900 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:28:01.900 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.113754+0000", 2024-09-11T08:28:01.900 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1755000000000003e-05 2024-09-11T08:28:01.900 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.900 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.901 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:28:01.901 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.417952+0000", 2024-09-11T08:28:01.901 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30419859399999999 2024-09-11T08:28:01.901 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.901 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.901 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:28:01.901 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.417970+0000", 2024-09-11T08:28:01.901 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7170999999999999e-05 2024-09-11T08:28:01.901 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.901 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.901 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:28:01.901 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.418336+0000", 2024-09-11T08:28:01.901 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00036620100000000003 2024-09-11T08:28:01.902 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.902 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.902 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:28:01.902 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.418359+0000", 2024-09-11T08:28:01.902 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2745e-05 2024-09-11T08:28:01.902 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.902 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.902 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:28:01.902 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.430805+0000", 2024-09-11T08:28:01.902 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.012446453999999999 2024-09-11T08:28:01.902 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.902 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.902 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:28:01.902 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.430853+0000", 2024-09-11T08:28:01.902 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7595000000000001e-05 2024-09-11T08:28:01.903 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:28:01.903 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:28:01.903 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:28:01.903 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.903 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.903 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.25799.0:74371 11.8 e398/382 11:185a39dc:::benchmark_data_smithi150_187410_object74370:head v 398'4751, mlcod=398'4749)", 2024-09-11T08:28:01.903 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-11T08:27:22.115493+0000", 2024-09-11T08:28:01.903 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 39.693973301, 2024-09-11T08:28:01.903 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30769450300000001, 2024-09-11T08:28:01.903 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-11T08:28:01.903 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:28:01.903 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-11T08:28:01.903 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.903 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-11T08:28:01.904 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.115493+0000", 2024-09-11T08:28:01.904 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:28:01.904 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.904 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.904 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-11T08:28:01.904 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.115493+0000", 2024-09-11T08:28:01.904 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-11T08:28:01.904 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.904 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.904 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-11T08:28:01.904 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.115492+0000", 2024-09-11T08:28:01.904 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-09-11T08:28:01.904 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.904 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.905 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-11T08:28:01.905 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.115513+0000", 2024-09-11T08:28:01.905 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0735999999999999e-05 2024-09-11T08:28:01.905 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.905 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.905 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-11T08:28:01.905 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.417949+0000", 2024-09-11T08:28:01.905 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30243646099999999 2024-09-11T08:28:01.905 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.905 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.905 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-11T08:28:01.905 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.417969+0000", 2024-09-11T08:28:01.905 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9599e-05 2024-09-11T08:28:01.906 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.906 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.906 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-11T08:28:01.906 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.418331+0000", 2024-09-11T08:28:01.906 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00036193400000000001 2024-09-11T08:28:01.906 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.906 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.906 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-11T08:28:01.906 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.418351+0000", 2024-09-11T08:28:01.906 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0026000000000001e-05 2024-09-11T08:28:01.906 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.906 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.907 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-11T08:28:01.907 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.423076+0000", 2024-09-11T08:28:01.907 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0047255170000000003 2024-09-11T08:28:01.907 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T08:28:01.907 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T08:28:01.907 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-11T08:28:01.907 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-11T08:27:22.423187+0000", 2024-09-11T08:28:01.907 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000110954 2024-09-11T08:28:01.907 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:28:01.907 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:28:01.907 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:28:01.907 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T08:28:01.907 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T08:28:01.907 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-11T08:28:01.941 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:01.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:01.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:28:02.042 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:02.060 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:01 smithi150 ceph-mon[29276]: 4.4 scrub starts 2024-09-11T08:28:02.061 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:01 smithi150 ceph-mon[29276]: 4.4 scrub ok 2024-09-11T08:28:02.061 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:01 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3135973415' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:28:02.061 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:01 smithi150 ceph-mon[29276]: pgmap v941: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 6.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.1 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-09-11T08:28:02.064 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:01 smithi150 ceph-mon[39612]: 4.4 scrub starts 2024-09-11T08:28:02.064 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:01 smithi150 ceph-mon[39612]: 4.4 scrub ok 2024-09-11T08:28:02.064 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:01 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3135973415' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:28:02.065 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:01 smithi150 ceph-mon[39612]: pgmap v941: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 6.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.1 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-09-11T08:28:02.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:01 smithi181 ceph-mon[32701]: 4.4 scrub starts 2024-09-11T08:28:02.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:01 smithi181 ceph-mon[32701]: 4.4 scrub ok 2024-09-11T08:28:02.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:01 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3135973415' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:28:02.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:01 smithi181 ceph-mon[32701]: pgmap v941: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 6.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.1 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-09-11T08:28:02.088 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:02.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:02.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:02.192 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:02.231 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:02.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:02.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:02.333 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:28:02.370 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:02.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:02.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:28:02.472 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:28:02.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:02.511 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:02.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:28:02.527 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T08:28:02.612 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:02.640 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 82 16 117654 117638 89.6309 73.9375 0.00650122 0.0111518 2024-09-11T08:28:02.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:02.663 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:02.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:02.764 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:02.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:02.808 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:02.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:02.840 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-11T08:28:02.864 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-09-11T08:28:02.864 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T08:28:02.864 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-09-11T08:28:02.864 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-11T08:28:02.864 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-11T08:28:02.865 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-09-11T08:28:02.898 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:02 smithi150 ceph-mon[29276]: 2.1 scrub starts 2024-09-11T08:28:02.898 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:02 smithi150 ceph-mon[29276]: 2.1 scrub ok 2024-09-11T08:28:02.898 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:02 smithi150 ceph-mon[29276]: 1.0 scrub starts 2024-09-11T08:28:02.898 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:02 smithi150 ceph-mon[29276]: 1.0 scrub ok 2024-09-11T08:28:02.898 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:02 smithi150 ceph-mon[29276]: 11.d scrub ok 2024-09-11T08:28:02.898 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:02 smithi150 ceph-mon[39612]: 2.1 scrub starts 2024-09-11T08:28:02.898 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:02 smithi150 ceph-mon[39612]: 2.1 scrub ok 2024-09-11T08:28:02.898 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:02 smithi150 ceph-mon[39612]: 1.0 scrub starts 2024-09-11T08:28:02.898 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:02 smithi150 ceph-mon[39612]: 1.0 scrub ok 2024-09-11T08:28:02.899 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:02 smithi150 ceph-mon[39612]: 11.d scrub ok 2024-09-11T08:28:02.909 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:02.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:02.988 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:02.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:03.021 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:28:03.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:02 smithi181 ceph-mon[32701]: 2.1 scrub starts 2024-09-11T08:28:03.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:02 smithi181 ceph-mon[32701]: 2.1 scrub ok 2024-09-11T08:28:03.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:02 smithi181 ceph-mon[32701]: 1.0 scrub starts 2024-09-11T08:28:03.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:02 smithi181 ceph-mon[32701]: 1.0 scrub ok 2024-09-11T08:28:03.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:02 smithi181 ceph-mon[32701]: 11.d scrub ok 2024-09-11T08:28:03.089 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:03.152 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:03.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:03.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:03.218 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:28:03.253 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:03.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:03.300 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:03.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:03.401 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:28:03.437 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:03.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:03.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:28:03.539 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:03.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:03.580 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:03.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:03.640 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 83 16 118622 118606 89.28 60.5 0.0152794 0.0111962 2024-09-11T08:28:03.681 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:28:03.715 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:03.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:03.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:28:03.817 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:03.845 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:03.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:03.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:03.947 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:03.978 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:03.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:03.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:04.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:03 smithi181 ceph-mon[32701]: 11.9 scrub ok 2024-09-11T08:28:04.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:03 smithi181 ceph-mon[32701]: 5.f scrub starts 2024-09-11T08:28:04.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:03 smithi181 ceph-mon[32701]: 5.f scrub ok 2024-09-11T08:28:04.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:03 smithi181 ceph-mon[32701]: 4.b scrub starts 2024-09-11T08:28:04.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:03 smithi181 ceph-mon[32701]: 4.b scrub ok 2024-09-11T08:28:04.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:03 smithi181 ceph-mon[32701]: pgmap v942: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 6.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 853 B/s rd, 57 MiB/s wr, 918 op/s 2024-09-11T08:28:04.080 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:28:04.107 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T08:28:04.108 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:04.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:04.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:28:04.138 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:03 smithi150 ceph-mon[29276]: 11.9 scrub ok 2024-09-11T08:28:04.139 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:03 smithi150 ceph-mon[29276]: 5.f scrub starts 2024-09-11T08:28:04.139 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:03 smithi150 ceph-mon[29276]: 5.f scrub ok 2024-09-11T08:28:04.139 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:03 smithi150 ceph-mon[29276]: 4.b scrub starts 2024-09-11T08:28:04.139 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:03 smithi150 ceph-mon[29276]: 4.b scrub ok 2024-09-11T08:28:04.139 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:03 smithi150 ceph-mon[29276]: pgmap v942: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 6.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 853 B/s rd, 57 MiB/s wr, 918 op/s 2024-09-11T08:28:04.139 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:03 smithi150 ceph-mon[39612]: 11.9 scrub ok 2024-09-11T08:28:04.139 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:03 smithi150 ceph-mon[39612]: 5.f scrub starts 2024-09-11T08:28:04.139 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:03 smithi150 ceph-mon[39612]: 5.f scrub ok 2024-09-11T08:28:04.139 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:03 smithi150 ceph-mon[39612]: 4.b scrub starts 2024-09-11T08:28:04.140 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:03 smithi150 ceph-mon[39612]: 4.b scrub ok 2024-09-11T08:28:04.140 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:03 smithi150 ceph-mon[39612]: pgmap v942: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 6.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 853 B/s rd, 57 MiB/s wr, 918 op/s 2024-09-11T08:28:04.210 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:04.239 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:04.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:04.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:04.341 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:04.399 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:04.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:04.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:04.501 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:28:04.536 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:04.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:04.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:28:04.638 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:28:04.640 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 84 16 119622 119606 88.9612 62.5 0.00798538 0.0112337 2024-09-11T08:28:04.666 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:04.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:04.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:28:04.768 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:04.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:04.811 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:04.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:04.912 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:28:04.940 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:04.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:04.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:28:05.042 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:05.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:04 smithi181 ceph-mon[32701]: 11.b scrub starts 2024-09-11T08:28:05.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:04 smithi181 ceph-mon[32701]: 3.4 deep-scrub starts 2024-09-11T08:28:05.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:04 smithi181 ceph-mon[32701]: 3.4 deep-scrub ok 2024-09-11T08:28:05.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:04 smithi181 ceph-mon[32701]: 3.1f scrub starts 2024-09-11T08:28:05.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:04 smithi181 ceph-mon[32701]: 3.1f scrub ok 2024-09-11T08:28:05.089 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:05.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:05.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:05.173 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:04 smithi150 ceph-mon[39612]: 11.b scrub starts 2024-09-11T08:28:05.173 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:04 smithi150 ceph-mon[39612]: 3.4 deep-scrub starts 2024-09-11T08:28:05.173 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:04 smithi150 ceph-mon[39612]: 3.4 deep-scrub ok 2024-09-11T08:28:05.173 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:04 smithi150 ceph-mon[39612]: 3.1f scrub starts 2024-09-11T08:28:05.173 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:04 smithi150 ceph-mon[39612]: 3.1f scrub ok 2024-09-11T08:28:05.175 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:04 smithi150 ceph-mon[29276]: 11.b scrub starts 2024-09-11T08:28:05.175 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:04 smithi150 ceph-mon[29276]: 3.4 deep-scrub starts 2024-09-11T08:28:05.175 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:04 smithi150 ceph-mon[29276]: 3.4 deep-scrub ok 2024-09-11T08:28:05.175 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:04 smithi150 ceph-mon[29276]: 3.1f scrub starts 2024-09-11T08:28:05.175 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:04 smithi150 ceph-mon[29276]: 3.1f scrub ok 2024-09-11T08:28:05.191 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:28:05.224 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:05.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:05.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:28:05.245 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-11T08:28:05.245 INFO:teuthology.orchestra.run.smithi181.stdout: "success": "" 2024-09-11T08:28:05.245 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-11T08:28:05.280 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:28:05.326 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:05.380 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:05.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:05.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:05.482 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:28:05.511 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:05.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:05.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:28:05.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:28:05 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:28:05] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:28:05.614 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:05.640 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 85 16 120623 120607 88.6506 62.5625 0.0206392 0.0112756 2024-09-11T08:28:05.667 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:05.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:05.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:05.769 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:28:05.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:05.813 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:05.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:28:05.915 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:28:05.950 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:05.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:05.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:28:05.964 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:05 smithi150 ceph-mon[29276]: 3.1 scrub starts 2024-09-11T08:28:05.964 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:05 smithi150 ceph-mon[29276]: 3.1 scrub ok 2024-09-11T08:28:05.964 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:05 smithi150 ceph-mon[29276]: pgmap v943: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.4 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-09-11T08:28:05.964 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:05 smithi150 ceph-mon[39612]: 3.1 scrub starts 2024-09-11T08:28:05.965 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:05 smithi150 ceph-mon[39612]: 3.1 scrub ok 2024-09-11T08:28:05.965 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:05 smithi150 ceph-mon[39612]: pgmap v943: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.4 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-09-11T08:28:06.052 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:06.095 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:06.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:06.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:06.197 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:28:06.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:06.226 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:06.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:28:06.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:05 smithi181 ceph-mon[32701]: 3.1 scrub starts 2024-09-11T08:28:06.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:05 smithi181 ceph-mon[32701]: 3.1 scrub ok 2024-09-11T08:28:06.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:05 smithi181 ceph-mon[32701]: pgmap v943: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.4 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-09-11T08:28:06.326 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:06.353 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:06.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:06.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:06.454 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:28:06.495 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:06.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:06.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:28:06.596 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:06.635 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:06.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:06.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:06.640 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 86 16 121644 121628 88.3618 63.8125 0.0124479 0.0113121 2024-09-11T08:28:06.669 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:28:06.737 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:28:06.771 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:06.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:06.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:28:06.853 INFO:teuthology.orchestra.run.smithi150.stderr:noscrub is set 2024-09-11T08:28:06.873 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:28:06.910 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:06.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:06.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:28:07.012 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:07.044 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:07.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:07.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:07.106 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:06 smithi150 ceph-mon[29276]: 3.15 deep-scrub starts 2024-09-11T08:28:07.106 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:06 smithi150 ceph-mon[29276]: 3.15 deep-scrub ok 2024-09-11T08:28:07.106 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:06 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3768631238' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:28:07.106 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:06 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:28:07.106 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:06 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:28:07.106 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:06 smithi150 ceph-mon[39612]: 3.15 deep-scrub starts 2024-09-11T08:28:07.107 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:06 smithi150 ceph-mon[39612]: 3.15 deep-scrub ok 2024-09-11T08:28:07.107 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:06 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3768631238' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:28:07.107 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:06 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:28:07.107 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:06 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:28:07.145 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:07.188 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:07.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:07.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:07.290 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:07.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:06 smithi181 ceph-mon[32701]: 3.15 deep-scrub starts 2024-09-11T08:28:07.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:06 smithi181 ceph-mon[32701]: 3.15 deep-scrub ok 2024-09-11T08:28:07.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:06 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3768631238' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:28:07.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:06 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:28:07.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:06 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:28:07.324 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:07.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:07.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:07.426 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:07.483 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:07.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:07.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:07.585 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:07.619 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:07.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:07.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:07.642 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 87 16 122732 122716 88.1272 68 0.0107187 0.0113422 2024-09-11T08:28:07.721 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:28:07.751 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:07.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:07.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:28:07.853 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:07.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:07.891 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:07.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:07.992 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:28:08.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:08.030 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:08.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:28:08.131 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:08.134 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:07 smithi150 ceph-mon[29276]: 2.0 scrub starts 2024-09-11T08:28:08.134 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:07 smithi150 ceph-mon[29276]: 2.0 scrub ok 2024-09-11T08:28:08.135 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:07 smithi150 ceph-mon[29276]: 11.b scrub ok 2024-09-11T08:28:08.135 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:07 smithi150 ceph-mon[29276]: 5.3 scrub starts 2024-09-11T08:28:08.135 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:07 smithi150 ceph-mon[29276]: 5.3 scrub ok 2024-09-11T08:28:08.135 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:07 smithi150 ceph-mon[29276]: 3.12 scrub starts 2024-09-11T08:28:08.135 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:07 smithi150 ceph-mon[29276]: 3.12 scrub ok 2024-09-11T08:28:08.135 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:07 smithi150 ceph-mon[29276]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:28:08.135 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:07 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3768631238' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:28:08.135 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:07 smithi150 ceph-mon[29276]: osdmap e415: 8 total, 8 up, 8 in 2024-09-11T08:28:08.135 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:07 smithi150 ceph-mon[29276]: pgmap v945: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 639 B/s rd, 41 MiB/s wr, 663 op/s 2024-09-11T08:28:08.136 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:07 smithi150 ceph-mon[39612]: 2.0 scrub starts 2024-09-11T08:28:08.136 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:07 smithi150 ceph-mon[39612]: 2.0 scrub ok 2024-09-11T08:28:08.136 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:07 smithi150 ceph-mon[39612]: 11.b scrub ok 2024-09-11T08:28:08.136 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:07 smithi150 ceph-mon[39612]: 5.3 scrub starts 2024-09-11T08:28:08.136 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:07 smithi150 ceph-mon[39612]: 5.3 scrub ok 2024-09-11T08:28:08.136 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:07 smithi150 ceph-mon[39612]: 3.12 scrub starts 2024-09-11T08:28:08.136 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:07 smithi150 ceph-mon[39612]: 3.12 scrub ok 2024-09-11T08:28:08.136 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:07 smithi150 ceph-mon[39612]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:28:08.136 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:07 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3768631238' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:28:08.136 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:07 smithi150 ceph-mon[39612]: osdmap e415: 8 total, 8 up, 8 in 2024-09-11T08:28:08.136 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:07 smithi150 ceph-mon[39612]: pgmap v945: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 639 B/s rd, 41 MiB/s wr, 663 op/s 2024-09-11T08:28:08.174 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:08.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:08.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:08.276 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:08.315 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:08.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:08.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:08.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:07 smithi181 ceph-mon[32701]: 2.0 scrub starts 2024-09-11T08:28:08.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:07 smithi181 ceph-mon[32701]: 2.0 scrub ok 2024-09-11T08:28:08.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:07 smithi181 ceph-mon[32701]: 11.b scrub ok 2024-09-11T08:28:08.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:07 smithi181 ceph-mon[32701]: 5.3 scrub starts 2024-09-11T08:28:08.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:07 smithi181 ceph-mon[32701]: 5.3 scrub ok 2024-09-11T08:28:08.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:07 smithi181 ceph-mon[32701]: 3.12 scrub starts 2024-09-11T08:28:08.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:07 smithi181 ceph-mon[32701]: 3.12 scrub ok 2024-09-11T08:28:08.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:07 smithi181 ceph-mon[32701]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:28:08.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:07 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3768631238' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:28:08.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:07 smithi181 ceph-mon[32701]: osdmap e415: 8 total, 8 up, 8 in 2024-09-11T08:28:08.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:07 smithi181 ceph-mon[32701]: pgmap v945: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 639 B/s rd, 41 MiB/s wr, 663 op/s 2024-09-11T08:28:08.416 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:28:08.449 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:08.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:08.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:28:08.551 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:08.590 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:08.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:08.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:08.641 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 88 16 123906 123890 87.9595 73.375 0.0152053 0.0113642 2024-09-11T08:28:08.692 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:28:08.727 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:08.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:08.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:28:08.738 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:28:08 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:28:08] "GET /metrics HTTP/1.1" 200 224703 "" "Prometheus/2.43.0" 2024-09-11T08:28:08.829 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:28:08.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:08.868 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:08.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:28:08.970 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:28:09.008 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:09.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:09.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:28:09.110 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:09.145 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:09.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:09.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:09.228 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:08 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 2]}]: dispatch 2024-09-11T08:28:09.228 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:08 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 2]}]: dispatch 2024-09-11T08:28:09.248 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:09.289 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:09.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:09.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:09.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:08 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 2]}]: dispatch 2024-09-11T08:28:09.324 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:28:08 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:28:09.390 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:28:09.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:09.422 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:09.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:28:09.524 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:28:09.557 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:09.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:09.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:28:09.641 INFO:tasks.radosbench.radosbench.0.smithi150.stdout: 89 16 125054 125038 87.7774 71.75 0.00554026 0.0113873 2024-09-11T08:28:09.659 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:09.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:09.699 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:09.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:09.800 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:09.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:09.835 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:09.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:09.917 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T08:28:09.936 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:28:09.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:09.996 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:09.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:28:10.097 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:10.139 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:10.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:10.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:10.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:09 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 2]}]': finished 2024-09-11T08:28:10.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:09 smithi150 ceph-mon[29276]: osdmap e416: 8 total, 8 up, 8 in 2024-09-11T08:28:10.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:09 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:28:10.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:09 smithi150 ceph-mon[29276]: pgmap v947: 148 pgs: 148 active+clean; 7.5 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.24k op/s 2024-09-11T08:28:10.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:09 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 2]}]': finished 2024-09-11T08:28:10.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:09 smithi150 ceph-mon[39612]: osdmap e416: 8 total, 8 up, 8 in 2024-09-11T08:28:10.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:09 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:28:10.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:09 smithi150 ceph-mon[39612]: pgmap v947: 148 pgs: 148 active+clean; 7.5 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.24k op/s 2024-09-11T08:28:10.240 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:28:10.267 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:10.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:10.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:28:10.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:09 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 2]}]': finished 2024-09-11T08:28:10.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:09 smithi181 ceph-mon[32701]: osdmap e416: 8 total, 8 up, 8 in 2024-09-11T08:28:10.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:09 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:28:10.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:09 smithi181 ceph-mon[32701]: pgmap v947: 148 pgs: 148 active+clean; 7.5 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.24k op/s 2024-09-11T08:28:10.370 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:28:10.405 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:10.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:10.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:28:10.508 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:10.535 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:10.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:10.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:10.637 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:28:10.641 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Total time run: 90.0086 2024-09-11T08:28:10.641 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Total writes made: 126046 2024-09-11T08:28:10.641 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Write size: 65536 2024-09-11T08:28:10.642 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Object size: 65536 2024-09-11T08:28:10.642 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Bandwidth (MB/sec): 87.5236 2024-09-11T08:28:10.642 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Stddev Bandwidth: 36.057 2024-09-11T08:28:10.642 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Max bandwidth (MB/sec): 227.312 2024-09-11T08:28:10.642 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Min bandwidth (MB/sec): 53.875 2024-09-11T08:28:10.642 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Average IOPS: 1400 2024-09-11T08:28:10.642 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Stddev IOPS: 576.912 2024-09-11T08:28:10.642 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Max IOPS: 3637 2024-09-11T08:28:10.642 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Min IOPS: 862 2024-09-11T08:28:10.642 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Average Latency(s): 0.0114215 2024-09-11T08:28:10.642 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Stddev Latency(s): 0.0104595 2024-09-11T08:28:10.642 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Max latency(s): 0.344516 2024-09-11T08:28:10.642 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Min latency(s): 0.00120292 2024-09-11T08:28:10.646 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Cleaning up (deleting benchmark objects) 2024-09-11T08:28:10.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:10.665 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:10.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:28:10.766 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:10.798 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:10.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:10.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:10.899 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:28:10.925 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:10.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:10.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:28:10.942 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-09-11T08:28:10.943 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd reweight 0 1 2024-09-11T08:28:11.027 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:11.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:11.068 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:11.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:11.169 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:28:11.197 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:11.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:11.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:28:11.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:10 smithi150 ceph-mon[29276]: osdmap e417: 8 total, 8 up, 8 in 2024-09-11T08:28:11.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:10 smithi150 ceph-mon[39612]: osdmap e417: 8 total, 8 up, 8 in 2024-09-11T08:28:11.299 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:28:11.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:10 smithi181 ceph-mon[32701]: osdmap e417: 8 total, 8 up, 8 in 2024-09-11T08:28:11.333 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:11.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:11.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:28:11.435 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:11.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:11.471 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:11.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:11.572 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:28:11.611 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:11.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:11.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:28:11.713 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:28:11.742 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:11.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:11.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:28:11.844 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:11.883 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:11.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:11.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:11.985 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:28:12.017 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:12.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:12.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:28:12.119 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:12.161 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:12.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:12.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:12.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:11 smithi150 ceph-mon[29276]: pgmap v949: 148 pgs: 1 peering, 147 active+clean; 7.6 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 852 B/s rd, 55 MiB/s wr, 878 op/s 2024-09-11T08:28:12.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:11 smithi150 ceph-mon[39612]: pgmap v949: 148 pgs: 1 peering, 147 active+clean; 7.6 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 852 B/s rd, 55 MiB/s wr, 878 op/s 2024-09-11T08:28:12.263 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:12.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:12.302 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:12.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:11 smithi181 ceph-mon[32701]: pgmap v949: 148 pgs: 1 peering, 147 active+clean; 7.6 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 852 B/s rd, 55 MiB/s wr, 878 op/s 2024-09-11T08:28:12.403 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:28:12.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:12.435 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:12.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:28:12.443 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-11T08:28:12.443 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [ 2024-09-11T08:28:12.443 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:12.444 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.25799.0:126203 11.6 11:6fe52e4c:::benchmark_data_smithi150_187410_object154:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-09-11T08:28:12.444 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:28:10.791860+0000", 2024-09-11T08:28:12.444 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.0035712600000000001, 2024-09-11T08:28:12.444 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0036003189999999998, 2024-09-11T08:28:12.444 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:28:12.444 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "waiting for sub ops", 2024-09-11T08:28:12.444 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:28:12.444 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.25799", 2024-09-11T08:28:12.444 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1403604046", 2024-09-11T08:28:12.444 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 126203 2024-09-11T08:28:12.444 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:12.445 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:28:12.445 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:12.445 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:28:12.445 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:10.791860+0000", 2024-09-11T08:28:12.445 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:28:12.445 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:12.445 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:12.445 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:28:12.445 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:10.791860+0000", 2024-09-11T08:28:12.445 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:28:12.445 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:12.445 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:12.446 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:28:12.446 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:10.791859+0000", 2024-09-11T08:28:12.446 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:28:12.446 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:12.446 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:12.446 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:28:12.446 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:10.791865+0000", 2024-09-11T08:28:12.446 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.421e-06 2024-09-11T08:28:12.446 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:12.446 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:12.446 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:28:12.446 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:10.791881+0000", 2024-09-11T08:28:12.447 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.5577999999999999e-05 2024-09-11T08:28:12.447 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:12.447 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:12.447 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:28:12.447 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:10.791887+0000", 2024-09-11T08:28:12.447 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.2319999999999998e-06 2024-09-11T08:28:12.447 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:12.447 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:12.447 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:28:12.447 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:10.791931+0000", 2024-09-11T08:28:12.447 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.4511999999999997e-05 2024-09-11T08:28:12.447 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:12.447 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:12.447 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:28:12.448 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:10.792034+0000", 2024-09-11T08:28:12.448 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000102675 2024-09-11T08:28:12.448 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:12.448 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:12.448 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 1,2", 2024-09-11T08:28:12.448 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:10.792138+0000", 2024-09-11T08:28:12.448 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000103609 2024-09-11T08:28:12.448 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:28:12.448 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:28:12.448 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:28:12.448 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:12.448 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:12.448 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.25799.0:126205 11.6 11:611ed9d5:::benchmark_data_smithi150_187410_object156:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-09-11T08:28:12.449 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:28:10.793222+0000", 2024-09-11T08:28:12.449 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.0022097480000000001, 2024-09-11T08:28:12.449 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0023134760000000001, 2024-09-11T08:28:12.449 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:28:12.449 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "waiting for sub ops", 2024-09-11T08:28:12.449 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:28:12.449 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.25799", 2024-09-11T08:28:12.449 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1403604046", 2024-09-11T08:28:12.449 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 126205 2024-09-11T08:28:12.449 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:12.449 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:28:12.449 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:12.449 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:28:12.449 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:10.793222+0000", 2024-09-11T08:28:12.450 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:28:12.450 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:12.450 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:12.450 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:28:12.450 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:10.793222+0000", 2024-09-11T08:28:12.450 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:28:12.450 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:12.450 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:12.450 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:28:12.450 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:10.793221+0000", 2024-09-11T08:28:12.451 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:28:12.451 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:12.451 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:12.451 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:28:12.451 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:10.793226+0000", 2024-09-11T08:28:12.451 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.7699999999999998e-06 2024-09-11T08:28:12.451 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:12.451 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:12.451 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:28:12.452 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:10.793242+0000", 2024-09-11T08:28:12.452 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.5733e-05 2024-09-11T08:28:12.452 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:12.452 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:12.452 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:28:12.452 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:10.793248+0000", 2024-09-11T08:28:12.452 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.0090000000000004e-06 2024-09-11T08:28:12.452 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:12.452 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:12.453 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:28:12.453 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:10.795326+0000", 2024-09-11T08:28:12.453 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0020779570000000001 2024-09-11T08:28:12.453 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:12.453 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:12.453 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:28:12.453 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:10.795428+0000", 2024-09-11T08:28:12.453 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000101986 2024-09-11T08:28:12.453 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:12.453 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:12.453 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 1,2", 2024-09-11T08:28:12.453 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:10.795531+0000", 2024-09-11T08:28:12.453 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000102874 2024-09-11T08:28:12.454 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:28:12.454 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:28:12.454 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:28:12.454 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:12.454 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:12.454 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.25799.0:126206 11.b e417/382 11:d7ed2fae:::benchmark_data_smithi150_187410_object157:head v 417'7909, mlcod=417'7907)", 2024-09-11T08:28:12.454 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:28:10.795304+0000", 2024-09-11T08:28:12.454 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.00012763499999999999, 2024-09-11T08:28:12.454 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00029853099999999999, 2024-09-11T08:28:12.454 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:28:12.454 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "started", 2024-09-11T08:28:12.454 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:28:12.455 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:12.455 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:28:12.455 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:10.795304+0000", 2024-09-11T08:28:12.455 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:28:12.455 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:12.455 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:12.455 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:28:12.455 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:10.795304+0000", 2024-09-11T08:28:12.455 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:28:12.455 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:12.456 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:12.456 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:28:12.456 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:10.795303+0000", 2024-09-11T08:28:12.456 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:28:12.456 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:12.456 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:12.456 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:28:12.456 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:10.795311+0000", 2024-09-11T08:28:12.456 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.0120000000000005e-06 2024-09-11T08:28:12.456 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:12.456 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:12.456 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:28:12.456 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:10.795321+0000", 2024-09-11T08:28:12.456 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0395e-05 2024-09-11T08:28:12.457 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:12.457 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:12.457 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:28:12.457 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:10.795327+0000", 2024-09-11T08:28:12.457 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.5230000000000003e-06 2024-09-11T08:28:12.457 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:12.457 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:12.458 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:28:12.458 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:10.795390+0000", 2024-09-11T08:28:12.458 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.3246999999999999e-05 2024-09-11T08:28:12.458 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:12.458 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:12.458 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:28:12.458 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:10.795409+0000", 2024-09-11T08:28:12.458 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.8468e-05 2024-09-11T08:28:12.458 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:28:12.458 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:28:12.458 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:28:12.458 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:28:12.458 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-09-11T08:28:12.458 INFO:teuthology.orchestra.run.smithi150.stdout: "num_ops": 3 2024-09-11T08:28:12.459 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-11T08:28:12.536 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:12.574 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:12.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:12.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:12.676 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:12.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:12.712 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:12.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:12.813 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:28:12.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:12.850 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:12.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:28:12.951 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:12.985 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:12.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:12.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:13.088 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:13.118 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:13.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:13.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:13.219 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:13.255 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:13.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:13.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:13.360 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:13.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:13.382 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:13.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:13.484 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:13.508 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:13.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:13.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:13.610 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:13.644 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:13.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:13.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:13.746 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:28:13.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:13.772 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:13.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:28:13.873 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:13.900 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:13.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:13.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:14.001 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:28:14.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:14.036 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:14.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:28:14.137 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:14.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:14.176 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:14.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:14.278 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:14.299 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:14.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:14.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:14.400 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:28:14.442 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:14.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:14.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:28:14.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:14 smithi150 ceph-mon[29276]: pgmap v950: 148 pgs: 1 peering, 147 active+clean; 7.6 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 809 B/s rd, 54 MiB/s wr, 890 op/s 2024-09-11T08:28:14.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:14 smithi150 ceph-mon[39612]: pgmap v950: 148 pgs: 1 peering, 147 active+clean; 7.6 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 809 B/s rd, 54 MiB/s wr, 890 op/s 2024-09-11T08:28:14.544 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:14.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:14.566 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:14.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:14.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:14 smithi181 ceph-mon[32701]: pgmap v950: 148 pgs: 1 peering, 147 active+clean; 7.6 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 809 B/s rd, 54 MiB/s wr, 890 op/s 2024-09-11T08:28:14.667 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:14.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:14.712 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:14.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:14.814 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:14.853 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:14.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:14.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:14.954 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:14.976 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:14.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:14.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:15.078 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:28:15.118 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:15.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:15.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:28:15.220 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:15.253 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:15.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:15.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:15.355 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:15.380 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:15.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:15.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:15.483 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:15.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:15.514 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:15.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:15.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:28:15 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:28:15] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:28:15.616 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:28:15.655 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:15.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:15.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:28:15.757 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:28:15.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:15.794 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:15.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:28:15.895 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:28:15.922 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:15.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:15.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:28:16.024 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:16.066 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:16.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:16.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:16.169 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:28:16.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:16.200 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:16.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:28:16.301 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:28:16.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:16.323 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:16.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:28:16.425 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:28:16.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:16.455 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:16.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:28:16.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:16 smithi150 ceph-mon[29276]: pgmap v951: 148 pgs: 148 active+clean; 7.5 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 1.4 KiB/s rd, 59 MiB/s wr, 1.35k op/s 2024-09-11T08:28:16.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:16 smithi150 ceph-mon[39612]: pgmap v951: 148 pgs: 148 active+clean; 7.5 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 1.4 KiB/s rd, 59 MiB/s wr, 1.35k op/s 2024-09-11T08:28:16.556 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:16.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:16 smithi181 ceph-mon[32701]: pgmap v951: 148 pgs: 148 active+clean; 7.5 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 1.4 KiB/s rd, 59 MiB/s wr, 1.35k op/s 2024-09-11T08:28:16.581 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:16.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:16.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:16.683 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:28:16.709 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:16.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:16.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:28:16.811 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:28:16.838 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:16.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:16.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:28:16.940 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:28:16.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:16.975 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:16.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:28:17.076 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:17.109 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:17.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:17.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:17.211 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:17.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:17.249 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:17.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:17.350 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:28:17.382 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:17.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:17.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:28:17.484 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:28:17.500 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-11T08:28:17.510 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:17.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:17.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:28:17.611 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:17.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:17.661 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:17.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:17.762 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:17.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:17.841 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:17.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:17.942 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:28:17.975 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:17.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:17.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:28:18.051 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:28:18.078 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:28:18.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:18.138 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:18.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:28:18.162 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:28:18.163 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:28:18.187 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:28:18.187 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:28:18.202 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:28:18.202 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:28:18.235 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:28:18.235 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:28:18.239 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:18.267 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:28:18.267 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:28:18.281 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:18 smithi150 ceph-mon[39612]: pgmap v952: 148 pgs: 148 active+clean; 7.5 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 741 B/s rd, 20 MiB/s wr, 701 op/s 2024-09-11T08:28:18.282 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:18 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:28:18.282 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:18 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1039074931' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:28:18.283 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:18 smithi150 ceph-mon[29276]: pgmap v952: 148 pgs: 148 active+clean; 7.5 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 741 B/s rd, 20 MiB/s wr, 701 op/s 2024-09-11T08:28:18.283 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:18 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:28:18.283 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:18 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1039074931' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:28:18.284 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:18.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:18.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:18.299 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:28:18.299 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:28:18.344 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:28:18.345 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:28:18.375 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:28:18.376 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:28:18.389 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:28:18.409 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:28:18.410 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:28:18.416 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:18.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:18.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:28:18.518 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:18.558 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:18.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:18.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:18.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:18 smithi181 ceph-mon[32701]: pgmap v952: 148 pgs: 148 active+clean; 7.5 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 741 B/s rd, 20 MiB/s wr, 701 op/s 2024-09-11T08:28:18.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:18 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:28:18.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:18 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1039074931' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:28:18.660 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:28:18.700 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:18.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:18.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:28:18.802 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:28:18.821 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:28:18 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:28:18] "GET /metrics HTTP/1.1" 200 224703 "" "Prometheus/2.43.0" 2024-09-11T08:28:18.832 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:18.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:18.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:28:18.934 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:18.965 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:18.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:18.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:19.067 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:19.073 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:28:18 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:28:19.111 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:19.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:19.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:19.213 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:19.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:19.243 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:19.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:19.344 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:28:19.379 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:19.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:19.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:28:19.481 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:19.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:19 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:28:19.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:19 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:28:19.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:19 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:28:19.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:19 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:28:19.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:19 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:28:19.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:19 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:28:19.517 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:19.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:19.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:19.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:19 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:28:19.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:19 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:28:19.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:19 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:28:19.619 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:28:19.647 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:19.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:19.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:28:19.749 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:19.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:19.783 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:19.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:19.884 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:19.921 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:19.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:19.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:20.023 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:20.061 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:20.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:20.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:20.163 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:20.202 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:20.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:20.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:20.304 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:20.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:20.340 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:20.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:20.441 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:20.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:20.490 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:20.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:20.527 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:20 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:28:20.527 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:20 smithi150 ceph-mon[29276]: pgmap v953: 148 pgs: 148 active+clean; 7.2 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 1.1 KiB/s rd, 16 MiB/s wr, 1.06k op/s 2024-09-11T08:28:20.527 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:20 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:28:20.527 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:20 smithi150 ceph-mon[39612]: pgmap v953: 148 pgs: 148 active+clean; 7.2 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 1.1 KiB/s rd, 16 MiB/s wr, 1.06k op/s 2024-09-11T08:28:20.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:20 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:28:20.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:20 smithi181 ceph-mon[32701]: pgmap v953: 148 pgs: 148 active+clean; 7.2 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 1.1 KiB/s rd, 16 MiB/s wr, 1.06k op/s 2024-09-11T08:28:20.591 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:28:20.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:20.622 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:20.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:28:20.723 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:20.760 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:20.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:20.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:20.862 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:20.902 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:20.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:20.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:21.004 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:21.031 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:21.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:21.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:21.133 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:21.160 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:21.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:21.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:21.262 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:21.300 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:21.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:21.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:21.402 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:21.435 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:21.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:21.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:21.537 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:21.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:21.578 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:21.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:21.679 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:28:21.699 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:21.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:21.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:28:21.801 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:28:21.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:21.829 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:21.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:28:21.930 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:28:21.957 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:21.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:21.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:28:22.058 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:22.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:21 smithi181 ceph-mon[32701]: pgmap v954: 148 pgs: 148 active+clean; 7.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 999 B/s rd, 13 MiB/s wr, 941 op/s 2024-09-11T08:28:22.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:22.097 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:22.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:22.198 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:22.225 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:22.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:22.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:22.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:21 smithi150 ceph-mon[29276]: pgmap v954: 148 pgs: 148 active+clean; 7.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 999 B/s rd, 13 MiB/s wr, 941 op/s 2024-09-11T08:28:22.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:21 smithi150 ceph-mon[39612]: pgmap v954: 148 pgs: 148 active+clean; 7.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 999 B/s rd, 13 MiB/s wr, 941 op/s 2024-09-11T08:28:22.326 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:22.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:22.372 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:22.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:22.452 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T08:28:22.474 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:28:22.501 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:22.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:22.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:28:22.603 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:22.630 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:22.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:22.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:22.732 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:28:22.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:22.761 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:22.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:28:22.862 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:22.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:22.913 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:22.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:23.015 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:28:23.028 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:22 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:28:23.028 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:22 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:28:23.031 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:22 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:28:23.032 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:22 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:28:23.061 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:23.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:23.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:28:23.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:22 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:28:23.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:22 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:28:23.163 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:23.209 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:23.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:23.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:23.310 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:23.339 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:23.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:23.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:23.440 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:23.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:23.476 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:23.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:23.577 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:23.612 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:23.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:23.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:23.718 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:23.749 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:23.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:23.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:23.851 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:28:23.884 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:23.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:23.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:28:23.973 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:23 smithi150 ceph-mon[29276]: pgmap v955: 148 pgs: 148 active+clean; 7.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 938 B/s rd, 12 MiB/s wr, 914 op/s 2024-09-11T08:28:23.974 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:23 smithi150 ceph-mon[39612]: pgmap v955: 148 pgs: 148 active+clean; 7.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 938 B/s rd, 12 MiB/s wr, 914 op/s 2024-09-11T08:28:23.986 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:28:24.017 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:24.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:24.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:28:24.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:23 smithi181 ceph-mon[32701]: pgmap v955: 148 pgs: 148 active+clean; 7.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 938 B/s rd, 12 MiB/s wr, 914 op/s 2024-09-11T08:28:24.118 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:24.152 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-09-11T08:28:24.152 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:28:24.155 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:24.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:24.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:24.257 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:24.292 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:24.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:24.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:24.393 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:24.431 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:24.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:24.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:24.533 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:24.564 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:24.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:24.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:24.666 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:24.692 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:24.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:24.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:24.793 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:24.835 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:24.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:24.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:24.937 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:28:24.940 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:24 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:28:24.940 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:24 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:28:24.940 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:24 smithi150 ceph-mon[29276]: osdmap e418: 8 total, 8 up, 8 in 2024-09-11T08:28:24.941 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:24 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:28:24.941 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:24 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:28:24.941 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:24 smithi150 ceph-mon[39612]: osdmap e418: 8 total, 8 up, 8 in 2024-09-11T08:28:24.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:24.980 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:24.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:28:25.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:24 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:28:25.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:24 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:28:25.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:24 smithi181 ceph-mon[32701]: osdmap e418: 8 total, 8 up, 8 in 2024-09-11T08:28:25.082 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:25.118 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:25.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:25.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:25.220 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:25.249 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:25.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:25.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:25.351 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:28:25.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:25.380 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:25.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:28:25.436 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-11T08:28:25.481 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:25.522 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:25.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:25.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:25.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:28:25 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:28:25] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:28:25.624 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:28:25.650 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:25.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:25.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:28:25.752 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:28:25.799 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:25.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:25.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:28:25.853 INFO:teuthology.orchestra.run.smithi150.stderr:reweighted osd.0 to 1 (10000) 2024-09-11T08:28:25.901 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:25.941 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:25.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:25.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:25.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:25 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3702783914' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-11T08:28:25.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:25 smithi150 ceph-mon[29276]: pgmap v957: 148 pgs: 148 active+clean; 6.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 1.08k op/s 2024-09-11T08:28:25.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:25 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3702783914' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-11T08:28:25.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:25 smithi150 ceph-mon[39612]: pgmap v957: 148 pgs: 148 active+clean; 6.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 1.08k op/s 2024-09-11T08:28:26.043 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:26.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:25 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3702783914' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-11T08:28:26.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:25 smithi181 ceph-mon[32701]: pgmap v957: 148 pgs: 148 active+clean; 6.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 1.08k op/s 2024-09-11T08:28:26.081 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:26.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:26.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:26.183 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:28:26.214 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:26.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:26.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:28:26.316 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:26.354 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:26.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:26.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:26.456 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:28:26.487 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:26.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:26.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:28:26.507 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:28:26.589 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:28:26.628 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:26.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:26.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:28:26.730 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:26.769 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:26.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:26.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:26.871 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:26.898 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:26.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:26.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:26.974 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:26 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3702783914' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-11T08:28:26.974 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:26 smithi150 ceph-mon[29276]: osdmap e419: 8 total, 8 up, 8 in 2024-09-11T08:28:26.974 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:26 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3702783914' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-11T08:28:26.974 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:26 smithi150 ceph-mon[39612]: osdmap e419: 8 total, 8 up, 8 in 2024-09-11T08:28:27.001 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:27.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:27.036 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:27.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:27.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:26 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3702783914' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-11T08:28:27.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:26 smithi181 ceph-mon[32701]: osdmap e419: 8 total, 8 up, 8 in 2024-09-11T08:28:27.136 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:27.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:27.173 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:27.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:27.273 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:27.300 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:27.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:27.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:27.313 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd reweight 1 1 2024-09-11T08:28:27.405 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:28:27.429 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:27.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:27.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:28:27.531 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:28:27.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:27.560 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:27.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:28:27.661 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:28:27.709 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:27.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:27.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:28:27.812 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:27.822 INFO:teuthology.orchestra.run.smithi150.stderr:nodeep-scrub is set 2024-09-11T08:28:27.862 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:27.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:27.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:27.963 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:27.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:27.999 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:27.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:28.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:27 smithi181 ceph-mon[32701]: pgmap v959: 148 pgs: 148 active+clean; 6.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 753 op/s 2024-09-11T08:28:28.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:27 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3054085224' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:28:28.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:27 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:28:28.084 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:27 smithi150 ceph-mon[29276]: pgmap v959: 148 pgs: 148 active+clean; 6.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 753 op/s 2024-09-11T08:28:28.084 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:27 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3054085224' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:28:28.085 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:27 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:28:28.085 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:27 smithi150 ceph-mon[39612]: pgmap v959: 148 pgs: 148 active+clean; 6.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 753 op/s 2024-09-11T08:28:28.085 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:27 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3054085224' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:28:28.085 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:27 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:28:28.099 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:28.135 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:28.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:28.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:28.237 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:28:28.272 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:28.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:28.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:28:28.374 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:28:28.417 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:28.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:28.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:28:28.519 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:28:28.557 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:28.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:28.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:28:28.659 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:28:28.694 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:28.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:28.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:28:28.796 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:28.802 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:28:28 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:28:28] "GET /metrics HTTP/1.1" 200 224656 "" "Prometheus/2.43.0" 2024-09-11T08:28:28.832 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:28.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:28.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:28.892 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:28:28.934 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:28.963 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:28.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:28.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:29.065 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:28:29.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:28 smithi181 ceph-mon[32701]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:28:29.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:28 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:28:29.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:28 smithi181 ceph-mon[32701]: osdmap e420: 8 total, 8 up, 8 in 2024-09-11T08:28:29.073 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:28:28 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:28:29.116 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:29.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:29.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:28:29.122 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:28 smithi150 ceph-mon[29276]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:28:29.122 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:28 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:28:29.122 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:28 smithi150 ceph-mon[29276]: osdmap e420: 8 total, 8 up, 8 in 2024-09-11T08:28:29.122 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:28 smithi150 ceph-mon[39612]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:28:29.122 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:28 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:28:29.122 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:28 smithi150 ceph-mon[39612]: osdmap e420: 8 total, 8 up, 8 in 2024-09-11T08:28:29.218 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:29.251 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:29.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:29.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:29.352 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:29.384 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:29.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:29.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:29.537 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:29.584 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:29.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:29.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:29.686 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:29.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:29.735 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:29.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:29.836 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:28:29.862 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:29.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:29.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:28:29.964 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:28:30.000 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:30.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:30.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:28:30.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:29 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:28:30.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:29 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:28:30.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:29 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:28:30.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:29 smithi181 ceph-mon[32701]: pgmap v961: 148 pgs: 148 active+clean; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.78k op/s 2024-09-11T08:28:30.074 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:29 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:28:30.074 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:29 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:28:30.074 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:29 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:28:30.074 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:29 smithi150 ceph-mon[29276]: pgmap v961: 148 pgs: 148 active+clean; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.78k op/s 2024-09-11T08:28:30.074 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:29 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:28:30.074 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:29 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:28:30.074 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:29 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:28:30.074 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:29 smithi150 ceph-mon[39612]: pgmap v961: 148 pgs: 148 active+clean; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.78k op/s 2024-09-11T08:28:30.102 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:30.137 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:30.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:30.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:30.239 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:28:30.269 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:30.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:30.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:28:30.371 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:30.406 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:30.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:30.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:30.508 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:30.554 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:30.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:30.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:30.656 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:28:30.687 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:30.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:30.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:28:30.752 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:28:30.752 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:28:30.752 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:28:30.753 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:28:30.753 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:28:30.753 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:28:30.753 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:28:30.753 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:28:30.753 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:28:30.753 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:28:30.753 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:28:30.753 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:28:30.753 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:28:30.753 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:28:30.754 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:28:30.754 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:28:30.789 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:30.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:30.828 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:30.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:30.877 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T08:28:30.921 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:30 smithi150 ceph-mon[29276]: osdmap e421: 8 total, 8 up, 8 in 2024-09-11T08:28:30.921 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:30 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3905537839' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:28:30.922 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:30 smithi150 ceph-mon[39612]: osdmap e421: 8 total, 8 up, 8 in 2024-09-11T08:28:30.922 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:30 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3905537839' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:28:30.929 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:30.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:30.968 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:30.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:31.069 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:31.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:30 smithi181 ceph-mon[32701]: osdmap e421: 8 total, 8 up, 8 in 2024-09-11T08:28:31.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:30 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3905537839' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:28:31.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:31.115 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:31.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:31.216 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:28:31.246 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:31.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:31.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:28:31.348 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:28:31.378 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:31.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:31.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:28:31.480 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:31.514 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:31.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:31.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:31.616 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:31.650 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:31.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:31.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:31.752 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:31.788 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:31.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:31.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:31.890 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:28:31.922 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:31.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:31.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:28:32.024 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:28:32.055 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:32.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:32.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:28:32.157 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:28:32.184 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:32.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:32.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:28:32.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:31 smithi150 ceph-mon[29276]: pgmap v963: 148 pgs: 148 active+clean; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 851 B/s rd, 0 B/s wr, 953 op/s 2024-09-11T08:28:32.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:31 smithi150 ceph-mon[39612]: pgmap v963: 148 pgs: 148 active+clean; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 851 B/s rd, 0 B/s wr, 953 op/s 2024-09-11T08:28:32.286 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:28:32.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:31 smithi181 ceph-mon[32701]: pgmap v963: 148 pgs: 148 active+clean; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 851 B/s rd, 0 B/s wr, 953 op/s 2024-09-11T08:28:32.323 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:32.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:32.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:28:32.425 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:32.461 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:32.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:32.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:32.563 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:32.593 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:32.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:32.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:32.695 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:28:32.726 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:32.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:32.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:28:32.828 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:32.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:32.862 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:32.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:32.963 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:32.994 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:32.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:32.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:33.097 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:28:33.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:33.137 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:33.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:28:33.238 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:33.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:32 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:28:33.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:32 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:28:33.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:32 smithi150 ceph-mon[29276]: osdmap e422: 8 total, 8 up, 8 in 2024-09-11T08:28:33.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:32 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:28:33.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:32 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:28:33.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:32 smithi150 ceph-mon[39612]: osdmap e422: 8 total, 8 up, 8 in 2024-09-11T08:28:33.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:33.282 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:33.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:33.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:32 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:28:33.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:32 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:28:33.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:32 smithi181 ceph-mon[32701]: osdmap e422: 8 total, 8 up, 8 in 2024-09-11T08:28:33.382 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:33.427 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:33.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:33.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:33.529 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:33.564 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:33.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:33.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:33.666 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:28:33.687 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:33.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:33.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:28:33.789 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:33.811 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:33.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:33.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:33.860 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:28:33.914 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:33.936 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:33.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:33.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:34.038 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:34.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:34.084 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:34.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:34.186 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:28:34.211 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:34.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:34.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:28:34.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:33 smithi150 ceph-mon[29276]: pgmap v965: 148 pgs: 148 active+clean; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 851 B/s rd, 0 B/s wr, 1.01k op/s 2024-09-11T08:28:34.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:33 smithi150 ceph-mon[39612]: pgmap v965: 148 pgs: 148 active+clean; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 851 B/s rd, 0 B/s wr, 1.01k op/s 2024-09-11T08:28:34.313 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:28:34.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:33 smithi181 ceph-mon[32701]: pgmap v965: 148 pgs: 148 active+clean; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 851 B/s rd, 0 B/s wr, 1.01k op/s 2024-09-11T08:28:34.353 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:34.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:34.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:28:34.455 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:34.489 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:34.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:34.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:34.590 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:28:34.619 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:34.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:34.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:28:34.721 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:34.754 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:34.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:34.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:34.856 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:34.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:34.896 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:34.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:34.997 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:35.038 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:35.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:35.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:35.140 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:35.177 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:35.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:35.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:35.280 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:35.315 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:35.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:35.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:35.417 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:28:35.452 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:35.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:35.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:28:35.554 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:28:35.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:28:35 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:28:35] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:28:35.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:35.587 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:35.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:28:35.688 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:28:35.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:35.717 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:35.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:28:35.818 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:35.855 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:35.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:35.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:35.957 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:28:35.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:35.991 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:35.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:28:36.092 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:36.137 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:36.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:36.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:36.239 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:28:36.265 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:36.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:36.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:28:36.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:36 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:28:36.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:36 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:28:36.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:36 smithi181 ceph-mon[32701]: osdmap e423: 8 total, 8 up, 8 in 2024-09-11T08:28:36.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:36 smithi181 ceph-mon[32701]: pgmap v967: 148 pgs: 148 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 851 B/s rd, 0 B/s wr, 1.06k op/s 2024-09-11T08:28:36.360 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:36 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:28:36.361 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:36 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:28:36.361 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:36 smithi150 ceph-mon[29276]: osdmap e423: 8 total, 8 up, 8 in 2024-09-11T08:28:36.361 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:36 smithi150 ceph-mon[29276]: pgmap v967: 148 pgs: 148 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 851 B/s rd, 0 B/s wr, 1.06k op/s 2024-09-11T08:28:36.361 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:36 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:28:36.361 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:36 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:28:36.361 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:36 smithi150 ceph-mon[39612]: osdmap e423: 8 total, 8 up, 8 in 2024-09-11T08:28:36.361 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:36 smithi150 ceph-mon[39612]: pgmap v967: 148 pgs: 148 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 851 B/s rd, 0 B/s wr, 1.06k op/s 2024-09-11T08:28:36.367 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:28:36.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:36.414 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:36.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:28:36.516 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:36.524 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-11T08:28:36.524 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 20, 2024-09-11T08:28:36.524 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 600, 2024-09-11T08:28:36.524 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [ 2024-09-11T08:28:36.524 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.524 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15369.0:96081 9.9 9:9ef22e50:::benchmark_data_smithi150_133616_object96080:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e259)", 2024-09-11T08:28:36.524 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:19:08.570968+0000", 2024-09-11T08:28:36.524 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 567.95017448299996, 2024-09-11T08:28:36.524 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.2956648120000001, 2024-09-11T08:28:36.524 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:28:36.525 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-09-11T08:28:36.525 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:28:36.525 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15369", 2024-09-11T08:28:36.525 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:28:36.525 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 96081 2024-09-11T08:28:36.525 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.525 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:28:36.525 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.525 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:28:36.525 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.570968+0000", 2024-09-11T08:28:36.525 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:28:36.525 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.526 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.526 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:28:36.526 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.570968+0000", 2024-09-11T08:28:36.526 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:28:36.526 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.526 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.526 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:28:36.526 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.570967+0000", 2024-09-11T08:28:36.526 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:28:36.526 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.526 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.526 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:28:36.526 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.570992+0000", 2024-09-11T08:28:36.527 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.4756000000000002e-05 2024-09-11T08:28:36.527 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.527 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.527 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:28:36.527 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.571003+0000", 2024-09-11T08:28:36.527 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1029999999999999e-05 2024-09-11T08:28:36.527 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.527 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.527 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:28:36.527 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.571009+0000", 2024-09-11T08:28:36.527 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.8289999999999996e-06 2024-09-11T08:28:36.527 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.527 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.528 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:28:36.528 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.575404+0000", 2024-09-11T08:28:36.528 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0043950339999999999 2024-09-11T08:28:36.528 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.528 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.528 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:28:36.528 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.575531+0000", 2024-09-11T08:28:36.528 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000127515 2024-09-11T08:28:36.528 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.528 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.528 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,7", 2024-09-11T08:28:36.528 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.575923+0000", 2024-09-11T08:28:36.528 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00039200599999999997 2024-09-11T08:28:36.529 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.529 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.529 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:28:36.529 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.589837+0000", 2024-09-11T08:28:36.529 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.013914015 2024-09-11T08:28:36.529 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.529 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.529 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:28:36.529 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.613186+0000", 2024-09-11T08:28:36.529 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.023348784000000001 2024-09-11T08:28:36.529 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.529 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.529 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:28:36.530 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.613196+0000", 2024-09-11T08:28:36.530 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0465999999999999e-05 2024-09-11T08:28:36.530 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.530 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.530 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:28:36.530 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.602630+0000", 2024-09-11T08:28:36.530 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.98943398299999996 2024-09-11T08:28:36.530 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.530 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.530 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:28:36.530 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.602639+0000", 2024-09-11T08:28:36.530 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.6649999999999992e-06 2024-09-11T08:28:36.530 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.531 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.531 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:28:36.531 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.864599+0000", 2024-09-11T08:28:36.531 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2619598349999999 2024-09-11T08:28:36.531 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.531 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.531 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:28:36.531 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.866633+0000", 2024-09-11T08:28:36.531 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0020339830000000001 2024-09-11T08:28:36.531 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:28:36.531 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:28:36.531 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:28:36.531 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.531 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.532 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15369.0:96088 9.9 9:90c5c778:::benchmark_data_smithi150_133616_object96087:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e259)", 2024-09-11T08:28:36.532 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:19:08.573763+0000", 2024-09-11T08:28:36.532 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 567.947379426, 2024-09-11T08:28:36.532 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.291237916, 2024-09-11T08:28:36.532 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:28:36.532 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-09-11T08:28:36.532 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:28:36.532 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15369", 2024-09-11T08:28:36.532 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:28:36.532 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 96088 2024-09-11T08:28:36.532 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.532 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:28:36.532 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.533 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:28:36.533 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.573763+0000", 2024-09-11T08:28:36.533 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:28:36.533 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.533 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.533 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:28:36.533 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.573763+0000", 2024-09-11T08:28:36.533 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:28:36.533 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.533 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.533 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:28:36.533 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.573762+0000", 2024-09-11T08:28:36.533 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:28:36.533 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.534 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.534 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:28:36.534 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.573793+0000", 2024-09-11T08:28:36.534 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.1029999999999999e-05 2024-09-11T08:28:36.534 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.534 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.534 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:28:36.534 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.573808+0000", 2024-09-11T08:28:36.534 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.5073e-05 2024-09-11T08:28:36.534 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.534 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.534 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:28:36.534 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.573814+0000", 2024-09-11T08:28:36.534 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.8359999999999997e-06 2024-09-11T08:28:36.535 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.535 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.535 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:28:36.535 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.580054+0000", 2024-09-11T08:28:36.535 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0062404540000000003 2024-09-11T08:28:36.535 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.535 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.535 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:28:36.535 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.580170+0000", 2024-09-11T08:28:36.535 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000115812 2024-09-11T08:28:36.535 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.535 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.535 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,7", 2024-09-11T08:28:36.535 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.580298+0000", 2024-09-11T08:28:36.536 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000128231 2024-09-11T08:28:36.536 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.536 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.536 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:28:36.536 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.589864+0000", 2024-09-11T08:28:36.536 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0095654590000000001 2024-09-11T08:28:36.536 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.536 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.536 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:28:36.536 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.613228+0000", 2024-09-11T08:28:36.536 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.023364772999999998 2024-09-11T08:28:36.536 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.536 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.536 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:28:36.537 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.613272+0000", 2024-09-11T08:28:36.537 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.3186999999999998e-05 2024-09-11T08:28:36.537 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.537 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.537 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:28:36.537 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.602667+0000", 2024-09-11T08:28:36.537 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.98939526499999997 2024-09-11T08:28:36.537 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.537 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.537 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:28:36.537 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.602673+0000", 2024-09-11T08:28:36.537 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.5220000000000004e-06 2024-09-11T08:28:36.537 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.537 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.538 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:28:36.538 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.864929+0000", 2024-09-11T08:28:36.538 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2622552309999999 2024-09-11T08:28:36.538 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.538 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.538 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:28:36.539 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.865001+0000", 2024-09-11T08:28:36.539 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.2358000000000001e-05 2024-09-11T08:28:36.539 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:28:36.539 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:28:36.539 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:28:36.539 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.539 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.539 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15369.0:96090 9.9 9:937fd0b9:::benchmark_data_smithi150_133616_object96089:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e259)", 2024-09-11T08:28:36.539 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:19:08.576879+0000", 2024-09-11T08:28:36.539 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 567.944263234, 2024-09-11T08:28:36.539 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.2882108680000002, 2024-09-11T08:28:36.539 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:28:36.539 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-09-11T08:28:36.540 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:28:36.540 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15369", 2024-09-11T08:28:36.540 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:28:36.540 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 96090 2024-09-11T08:28:36.540 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.540 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:28:36.540 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.540 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:28:36.540 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.576879+0000", 2024-09-11T08:28:36.540 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:28:36.540 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.540 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.540 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:28:36.541 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.576879+0000", 2024-09-11T08:28:36.541 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:28:36.541 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.541 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.541 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:28:36.541 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.576878+0000", 2024-09-11T08:28:36.541 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:28:36.541 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.541 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.541 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:28:36.541 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.576908+0000", 2024-09-11T08:28:36.541 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.0020999999999999e-05 2024-09-11T08:28:36.541 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.541 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.542 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:28:36.542 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.576923+0000", 2024-09-11T08:28:36.542 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4887e-05 2024-09-11T08:28:36.542 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.542 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.542 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:28:36.542 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.576929+0000", 2024-09-11T08:28:36.542 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.2720000000000003e-06 2024-09-11T08:28:36.542 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.542 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.542 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:28:36.542 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.583535+0000", 2024-09-11T08:28:36.542 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0066064549999999998 2024-09-11T08:28:36.542 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.543 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.543 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:28:36.543 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.583659+0000", 2024-09-11T08:28:36.543 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012306700000000001 2024-09-11T08:28:36.543 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.543 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.543 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,7", 2024-09-11T08:28:36.543 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.583809+0000", 2024-09-11T08:28:36.543 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00015080900000000001 2024-09-11T08:28:36.543 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.543 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.543 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:28:36.543 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.592518+0000", 2024-09-11T08:28:36.543 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0087083590000000006 2024-09-11T08:28:36.544 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.544 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.544 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:28:36.544 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.613355+0000", 2024-09-11T08:28:36.544 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.020837073000000001 2024-09-11T08:28:36.544 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.544 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.544 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:28:36.544 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.613361+0000", 2024-09-11T08:28:36.544 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.2400000000000004e-06 2024-09-11T08:28:36.544 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.544 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.544 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:28:36.545 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.602699+0000", 2024-09-11T08:28:36.545 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.98933749599999998 2024-09-11T08:28:36.545 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.545 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.545 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:28:36.545 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.602705+0000", 2024-09-11T08:28:36.545 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.8260000000000004e-06 2024-09-11T08:28:36.545 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.545 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.545 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:28:36.545 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.865029+0000", 2024-09-11T08:28:36.545 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2623238969999999 2024-09-11T08:28:36.545 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.545 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.546 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:28:36.546 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.865090+0000", 2024-09-11T08:28:36.546 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.0872999999999998e-05 2024-09-11T08:28:36.546 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:28:36.546 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:28:36.546 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:28:36.546 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.546 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.546 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15369.0:96105 9.9 9:97afa5c7:::benchmark_data_smithi150_133616_object96104:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-11T08:28:36.546 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:19:08.610376+0000", 2024-09-11T08:28:36.546 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 567.91076612400002, 2024-09-11T08:28:36.546 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.3038915310000001, 2024-09-11T08:28:36.546 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:28:36.547 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:28:36.547 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:28:36.547 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15369", 2024-09-11T08:28:36.547 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:28:36.547 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 96105 2024-09-11T08:28:36.547 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.547 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:28:36.547 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.547 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:28:36.547 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.610376+0000", 2024-09-11T08:28:36.547 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:28:36.547 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.547 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.547 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:28:36.548 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.610376+0000", 2024-09-11T08:28:36.548 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:28:36.548 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.548 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.548 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:28:36.548 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.610375+0000", 2024-09-11T08:28:36.548 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:28:36.548 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.548 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.548 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:28:36.548 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.610401+0000", 2024-09-11T08:28:36.548 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.6548000000000001e-05 2024-09-11T08:28:36.548 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.548 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.549 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:28:36.549 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.610410+0000", 2024-09-11T08:28:36.549 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.2640000000000008e-06 2024-09-11T08:28:36.549 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.549 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.549 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:28:36.549 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.610416+0000", 2024-09-11T08:28:36.549 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.2770000000000002e-06 2024-09-11T08:28:36.549 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.549 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.549 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:28:36.549 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.614510+0000", 2024-09-11T08:28:36.549 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0040939319999999998 2024-09-11T08:28:36.549 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.550 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.550 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:28:36.550 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.614517+0000", 2024-09-11T08:28:36.550 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.2400000000000001e-06 2024-09-11T08:28:36.550 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.550 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.550 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:28:36.550 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.602724+0000", 2024-09-11T08:28:36.550 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.98820673699999995 2024-09-11T08:28:36.550 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.550 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.550 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:28:36.550 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.602731+0000", 2024-09-11T08:28:36.550 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.6880000000000002e-06 2024-09-11T08:28:36.551 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.551 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.551 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:28:36.551 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.865109+0000", 2024-09-11T08:28:36.551 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2623779960000001 2024-09-11T08:28:36.551 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.551 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.551 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:28:36.551 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.865225+0000", 2024-09-11T08:28:36.551 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000115911 2024-09-11T08:28:36.551 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.551 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.551 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 3,7", 2024-09-11T08:28:36.551 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.865767+0000", 2024-09-11T08:28:36.551 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00054236499999999995 2024-09-11T08:28:36.552 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.552 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.552 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:28:36.552 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.885412+0000", 2024-09-11T08:28:36.552 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.019645187000000001 2024-09-11T08:28:36.552 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.552 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.552 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:28:36.552 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.890697+0000", 2024-09-11T08:28:36.552 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0052851699999999996 2024-09-11T08:28:36.552 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.552 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.552 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:28:36.553 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.909083+0000", 2024-09-11T08:28:36.553 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.018386070000000001 2024-09-11T08:28:36.553 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.553 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.553 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:28:36.553 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.909169+0000", 2024-09-11T08:28:36.553 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.5953999999999997e-05 2024-09-11T08:28:36.553 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.553 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.553 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:28:36.553 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.914268+0000", 2024-09-11T08:28:36.553 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0050986199999999999 2024-09-11T08:28:36.553 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:28:36.553 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:28:36.554 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:28:36.554 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.554 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.554 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15369.0:96107 9.9 9:9e673b22:::benchmark_data_smithi150_133616_object96106:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-11T08:28:36.554 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:19:08.610424+0000", 2024-09-11T08:28:36.554 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 567.91071870300004, 2024-09-11T08:28:36.554 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.3087517069999999, 2024-09-11T08:28:36.554 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:28:36.554 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:28:36.554 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:28:36.554 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15369", 2024-09-11T08:28:36.554 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:28:36.554 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 96107 2024-09-11T08:28:36.555 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.555 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:28:36.555 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.555 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:28:36.555 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.610424+0000", 2024-09-11T08:28:36.555 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:28:36.555 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.555 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.555 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:28:36.555 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.610424+0000", 2024-09-11T08:28:36.555 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:28:36.555 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.555 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.555 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:28:36.556 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.610423+0000", 2024-09-11T08:28:36.556 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:28:36.556 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.556 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.556 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:28:36.556 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.610445+0000", 2024-09-11T08:28:36.556 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.1322e-05 2024-09-11T08:28:36.556 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.556 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.556 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:28:36.556 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.610450+0000", 2024-09-11T08:28:36.556 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.3789999999999997e-06 2024-09-11T08:28:36.556 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.556 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.557 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:28:36.557 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.610454+0000", 2024-09-11T08:28:36.557 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.0090000000000001e-06 2024-09-11T08:28:36.557 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.557 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.557 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:28:36.557 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.614538+0000", 2024-09-11T08:28:36.557 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0040842200000000004 2024-09-11T08:28:36.557 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.557 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.557 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:28:36.557 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.614544+0000", 2024-09-11T08:28:36.557 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.1009999999999997e-06 2024-09-11T08:28:36.558 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.558 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.558 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:28:36.558 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.602748+0000", 2024-09-11T08:28:36.558 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.98820334399999998 2024-09-11T08:28:36.558 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.558 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.558 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:28:36.558 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.602754+0000", 2024-09-11T08:28:36.558 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.0159999999999997e-06 2024-09-11T08:28:36.558 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.558 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.558 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:28:36.558 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.873544+0000", 2024-09-11T08:28:36.559 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2707898580000001 2024-09-11T08:28:36.559 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.559 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.559 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:28:36.559 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.873670+0000", 2024-09-11T08:28:36.559 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000126766 2024-09-11T08:28:36.559 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.559 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.559 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 3,7", 2024-09-11T08:28:36.559 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.873839+0000", 2024-09-11T08:28:36.559 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000169025 2024-09-11T08:28:36.559 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.559 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.559 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:28:36.560 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.885452+0000", 2024-09-11T08:28:36.560 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.011612447999999999 2024-09-11T08:28:36.560 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.560 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.560 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:28:36.560 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.890704+0000", 2024-09-11T08:28:36.560 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0052519020000000001 2024-09-11T08:28:36.560 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.560 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.560 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:28:36.560 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.914526+0000", 2024-09-11T08:28:36.560 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.023822471000000001 2024-09-11T08:28:36.560 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.561 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.561 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:28:36.561 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.914585+0000", 2024-09-11T08:28:36.561 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.9111999999999999e-05 2024-09-11T08:28:36.561 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.561 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.561 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:28:36.561 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.919175+0000", 2024-09-11T08:28:36.561 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0045901170000000003 2024-09-11T08:28:36.561 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:28:36.561 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:28:36.561 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:28:36.561 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.561 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.562 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15369.0:96108 9.b 9:db475b9c:::benchmark_data_smithi150_133616_object96107:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-11T08:28:36.562 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:19:08.610458+0000", 2024-09-11T08:28:36.562 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 567.91068436499995, 2024-09-11T08:28:36.562 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.3092126090000002, 2024-09-11T08:28:36.562 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:28:36.562 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:28:36.562 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:28:36.562 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15369", 2024-09-11T08:28:36.562 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:28:36.562 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 96108 2024-09-11T08:28:36.562 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.562 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:28:36.562 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.562 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:28:36.563 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.610458+0000", 2024-09-11T08:28:36.563 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:28:36.563 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.563 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.563 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:28:36.563 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.610458+0000", 2024-09-11T08:28:36.563 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:28:36.563 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.563 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.563 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:28:36.563 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.610458+0000", 2024-09-11T08:28:36.563 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:28:36.563 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.563 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.563 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:28:36.564 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.610480+0000", 2024-09-11T08:28:36.564 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.1861999999999998e-05 2024-09-11T08:28:36.564 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.564 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.564 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:28:36.564 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.610485+0000", 2024-09-11T08:28:36.564 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.0830000000000003e-06 2024-09-11T08:28:36.564 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.564 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.564 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:28:36.564 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.610488+0000", 2024-09-11T08:28:36.564 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.619e-06 2024-09-11T08:28:36.564 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.564 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.565 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:28:36.565 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.616830+0000", 2024-09-11T08:28:36.565 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0063418579999999997 2024-09-11T08:28:36.565 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.565 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.565 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:28:36.565 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.616840+0000", 2024-09-11T08:28:36.565 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.611e-06 2024-09-11T08:28:36.565 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.565 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.565 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:28:36.565 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.607837+0000", 2024-09-11T08:28:36.565 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.99099718299999995 2024-09-11T08:28:36.565 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.566 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.566 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:28:36.566 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.607847+0000", 2024-09-11T08:28:36.566 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0501e-05 2024-09-11T08:28:36.567 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.567 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.567 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:28:36.567 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.873810+0000", 2024-09-11T08:28:36.567 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.265962279 2024-09-11T08:28:36.567 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.567 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.567 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:28:36.567 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.873928+0000", 2024-09-11T08:28:36.567 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00011868500000000001 2024-09-11T08:28:36.567 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.567 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.567 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 2,6", 2024-09-11T08:28:36.568 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.874457+0000", 2024-09-11T08:28:36.568 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000528851 2024-09-11T08:28:36.568 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.568 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.568 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:28:36.568 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.894530+0000", 2024-09-11T08:28:36.568 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.020072349 2024-09-11T08:28:36.568 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.568 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.568 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:28:36.568 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.898520+0000", 2024-09-11T08:28:36.568 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0039902920000000003 2024-09-11T08:28:36.568 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.568 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.569 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:28:36.569 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.914580+0000", 2024-09-11T08:28:36.569 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.016060540000000002 2024-09-11T08:28:36.569 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.569 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.569 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:28:36.569 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.914628+0000", 2024-09-11T08:28:36.569 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.7704000000000003e-05 2024-09-11T08:28:36.569 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.569 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.569 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:28:36.569 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.919671+0000", 2024-09-11T08:28:36.570 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0050425569999999996 2024-09-11T08:28:36.570 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:28:36.570 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:28:36.570 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:28:36.570 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.570 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.570 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15369.0:96111 9.b 9:d376a5e3:::benchmark_data_smithi150_133616_object96110:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-11T08:28:36.570 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:19:08.616227+0000", 2024-09-11T08:28:36.570 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 567.90491558899998, 2024-09-11T08:28:36.570 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.3142295430000002, 2024-09-11T08:28:36.570 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:28:36.570 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:28:36.570 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:28:36.571 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15369", 2024-09-11T08:28:36.571 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:28:36.571 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 96111 2024-09-11T08:28:36.571 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.571 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:28:36.571 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.571 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:28:36.571 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.616227+0000", 2024-09-11T08:28:36.571 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:28:36.571 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.571 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.571 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:28:36.571 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.616227+0000", 2024-09-11T08:28:36.571 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:28:36.572 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.572 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.572 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:28:36.572 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.616225+0000", 2024-09-11T08:28:36.572 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:28:36.572 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.572 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.572 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:28:36.572 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.616254+0000", 2024-09-11T08:28:36.572 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.8079e-05 2024-09-11T08:28:36.572 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.572 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.572 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:28:36.573 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.616262+0000", 2024-09-11T08:28:36.573 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.8550000000000008e-06 2024-09-11T08:28:36.573 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.573 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.573 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:28:36.573 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.616270+0000", 2024-09-11T08:28:36.573 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.413e-06 2024-09-11T08:28:36.573 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.573 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.573 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:28:36.573 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.616864+0000", 2024-09-11T08:28:36.573 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00059373800000000001 2024-09-11T08:28:36.573 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.573 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.574 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:28:36.574 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.616870+0000", 2024-09-11T08:28:36.574 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.2720000000000003e-06 2024-09-11T08:28:36.574 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.574 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.574 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:28:36.574 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.607897+0000", 2024-09-11T08:28:36.574 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.99102741699999997 2024-09-11T08:28:36.574 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.574 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.574 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:28:36.574 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.607905+0000", 2024-09-11T08:28:36.574 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.3309999999999998e-06 2024-09-11T08:28:36.574 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.575 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.575 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:28:36.575 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.884529+0000", 2024-09-11T08:28:36.575 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2766241059999999 2024-09-11T08:28:36.575 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.575 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.575 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:28:36.575 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.884831+0000", 2024-09-11T08:28:36.575 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00030185699999999999 2024-09-11T08:28:36.575 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.575 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.575 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 2,6", 2024-09-11T08:28:36.575 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.885022+0000", 2024-09-11T08:28:36.575 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000191184 2024-09-11T08:28:36.576 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.576 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.576 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:28:36.576 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.894572+0000", 2024-09-11T08:28:36.576 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0095500270000000009 2024-09-11T08:28:36.576 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.576 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.576 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:28:36.576 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.898529+0000", 2024-09-11T08:28:36.576 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0039576910000000002 2024-09-11T08:28:36.576 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.576 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.576 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:28:36.576 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.925242+0000", 2024-09-11T08:28:36.577 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.026713012000000001 2024-09-11T08:28:36.577 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.577 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.577 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:28:36.577 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.925297+0000", 2024-09-11T08:28:36.577 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.4160999999999999e-05 2024-09-11T08:28:36.577 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.577 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.577 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:28:36.577 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.930456+0000", 2024-09-11T08:28:36.577 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0051598659999999999 2024-09-11T08:28:36.577 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:28:36.577 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:28:36.577 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:28:36.578 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.578 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.578 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15369.0:96118 9.8 9:18676dc0:::benchmark_data_smithi150_133616_object96117:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-11T08:28:36.578 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:19:08.629161+0000", 2024-09-11T08:28:36.578 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 567.89198178000004, 2024-09-11T08:28:36.578 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.2526056940000001, 2024-09-11T08:28:36.578 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:28:36.578 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:28:36.578 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:28:36.578 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15369", 2024-09-11T08:28:36.578 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/207988314", 2024-09-11T08:28:36.578 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 96118 2024-09-11T08:28:36.578 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.578 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:28:36.579 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.579 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:28:36.579 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.629161+0000", 2024-09-11T08:28:36.579 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:28:36.579 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.579 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.579 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:28:36.579 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.629161+0000", 2024-09-11T08:28:36.579 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:28:36.579 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.579 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.579 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:28:36.579 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.629158+0000", 2024-09-11T08:28:36.579 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999976 2024-09-11T08:28:36.579 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.580 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.580 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:28:36.580 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.629185+0000", 2024-09-11T08:28:36.580 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.6849999999999999e-05 2024-09-11T08:28:36.580 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.580 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.580 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:28:36.580 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.629203+0000", 2024-09-11T08:28:36.580 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.7411e-05 2024-09-11T08:28:36.580 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.580 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.580 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:28:36.580 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.629209+0000", 2024-09-11T08:28:36.581 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.3489999999999997e-06 2024-09-11T08:28:36.581 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.581 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.581 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:28:36.581 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.629261+0000", 2024-09-11T08:28:36.581 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.2225000000000002e-05 2024-09-11T08:28:36.581 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.581 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.581 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:28:36.581 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:08.629268+0000", 2024-09-11T08:28:36.581 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.6590000000000001e-06 2024-09-11T08:28:36.581 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.581 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.582 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:28:36.582 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.599755+0000", 2024-09-11T08:28:36.582 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.970487129 2024-09-11T08:28:36.582 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.582 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.582 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:28:36.582 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:09.599766+0000", 2024-09-11T08:28:36.582 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0803e-05 2024-09-11T08:28:36.582 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.582 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.582 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:28:36.582 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.849312+0000", 2024-09-11T08:28:36.582 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.24954637 2024-09-11T08:28:36.582 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.583 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.583 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:28:36.583 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.849466+0000", 2024-09-11T08:28:36.583 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00015406 2024-09-11T08:28:36.583 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.583 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.583 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 3,7", 2024-09-11T08:28:36.583 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.850103+0000", 2024-09-11T08:28:36.583 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00063667200000000002 2024-09-11T08:28:36.583 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.583 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.583 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:28:36.583 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.871862+0000", 2024-09-11T08:28:36.584 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.021759013000000001 2024-09-11T08:28:36.584 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.584 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.584 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:28:36.584 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.871874+0000", 2024-09-11T08:28:36.584 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.225e-05 2024-09-11T08:28:36.584 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.584 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.584 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:28:36.584 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.871918+0000", 2024-09-11T08:28:36.584 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.3695000000000001e-05 2024-09-11T08:28:36.584 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.584 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.584 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:28:36.585 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.871999+0000", 2024-09-11T08:28:36.585 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.1187999999999996e-05 2024-09-11T08:28:36.585 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.585 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.585 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:28:36.585 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:19:10.881766+0000", 2024-09-11T08:28:36.585 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0097672809999999992 2024-09-11T08:28:36.585 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:28:36.585 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:28:36.585 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:28:36.585 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.585 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.585 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:257271 10.6 10:636a8c14:::benchmark_data_smithi150_158067_object127027:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e367)", 2024-09-11T08:28:36.586 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:25:40.715790+0000", 2024-09-11T08:28:36.586 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 175.80535214400001, 2024-09-11T08:28:36.586 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0469452480000001, 2024-09-11T08:28:36.586 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:28:36.586 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:28:36.586 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:28:36.586 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:28:36.586 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:28:36.586 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 257271 2024-09-11T08:28:36.586 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.586 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:28:36.586 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.586 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:28:36.586 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:40.715790+0000", 2024-09-11T08:28:36.587 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:28:36.587 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.587 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.587 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:28:36.587 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:40.715790+0000", 2024-09-11T08:28:36.587 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:28:36.587 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.587 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.587 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:28:36.587 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:40.715790+0000", 2024-09-11T08:28:36.587 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:28:36.587 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.587 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.587 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:28:36.588 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:40.715793+0000", 2024-09-11T08:28:36.588 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.0790000000000001e-06 2024-09-11T08:28:36.588 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.588 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.588 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:28:36.588 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:40.715798+0000", 2024-09-11T08:28:36.588 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.2109999999999997e-06 2024-09-11T08:28:36.588 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.588 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.588 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:28:36.588 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:40.715802+0000", 2024-09-11T08:28:36.588 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.484e-06 2024-09-11T08:28:36.588 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.589 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.589 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:28:36.589 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:40.721010+0000", 2024-09-11T08:28:36.589 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0052081669999999997 2024-09-11T08:28:36.589 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.589 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.589 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:28:36.589 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:40.721020+0000", 2024-09-11T08:28:36.589 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0259e-05 2024-09-11T08:28:36.589 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.589 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.589 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:28:36.589 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.738170+0000", 2024-09-11T08:28:36.589 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0171502059999999 2024-09-11T08:28:36.590 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.590 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.590 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:28:36.590 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.738255+0000", 2024-09-11T08:28:36.590 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.4247999999999997e-05 2024-09-11T08:28:36.590 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.590 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.590 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 4,5", 2024-09-11T08:28:36.590 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.738345+0000", 2024-09-11T08:28:36.590 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.0892999999999999e-05 2024-09-11T08:28:36.590 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.590 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.590 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:28:36.590 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.745007+0000", 2024-09-11T08:28:36.591 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0066614960000000003 2024-09-11T08:28:36.591 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.591 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.591 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:28:36.591 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.762625+0000", 2024-09-11T08:28:36.591 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.017618455000000002 2024-09-11T08:28:36.591 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.591 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.591 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:28:36.591 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.762656+0000", 2024-09-11T08:28:36.591 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.0380000000000001e-05 2024-09-11T08:28:36.591 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.591 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.591 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:28:36.592 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.762700+0000", 2024-09-11T08:28:36.592 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.3881999999999998e-05 2024-09-11T08:28:36.592 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.592 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.592 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:28:36.592 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.762736+0000", 2024-09-11T08:28:36.592 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.5932000000000001e-05 2024-09-11T08:28:36.592 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:28:36.592 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:28:36.592 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:28:36.592 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.592 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.592 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:257281 10.6 10:6048b861:::benchmark_data_smithi150_158067_object127037:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e367)", 2024-09-11T08:28:36.592 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:25:40.732311+0000", 2024-09-11T08:28:36.593 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 175.78883118799999, 2024-09-11T08:28:36.593 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.03065271, 2024-09-11T08:28:36.593 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:28:36.593 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:28:36.593 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:28:36.593 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:28:36.593 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:28:36.593 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 257281 2024-09-11T08:28:36.593 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.593 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:28:36.593 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.593 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:28:36.593 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:40.732311+0000", 2024-09-11T08:28:36.593 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:28:36.594 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.594 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.594 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:28:36.594 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:40.732311+0000", 2024-09-11T08:28:36.594 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:28:36.594 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:36.595 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:36.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:36.596 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.596 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:28:36.596 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:40.732311+0000", 2024-09-11T08:28:36.596 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:28:36.597 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.597 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.597 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:28:36.597 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:40.732314+0000", 2024-09-11T08:28:36.597 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.7520000000000001e-06 2024-09-11T08:28:36.597 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.597 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.597 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:28:36.597 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:40.732319+0000", 2024-09-11T08:28:36.597 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.3489999999999999e-06 2024-09-11T08:28:36.597 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.597 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.597 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:28:36.597 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:40.732323+0000", 2024-09-11T08:28:36.598 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.7610000000000001e-06 2024-09-11T08:28:36.598 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.598 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.598 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:28:36.598 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:40.732375+0000", 2024-09-11T08:28:36.598 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.2096999999999999e-05 2024-09-11T08:28:36.598 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.598 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.598 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:28:36.598 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:40.732385+0000", 2024-09-11T08:28:36.598 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0642000000000001e-05 2024-09-11T08:28:36.598 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.598 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.599 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:28:36.599 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.740597+0000", 2024-09-11T08:28:36.599 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0082119759999999 2024-09-11T08:28:36.599 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.599 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.599 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:28:36.599 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.740659+0000", 2024-09-11T08:28:36.599 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.1123000000000004e-05 2024-09-11T08:28:36.599 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.599 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.599 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 4,5", 2024-09-11T08:28:36.599 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.740728+0000", 2024-09-11T08:28:36.600 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.9684999999999994e-05 2024-09-11T08:28:36.600 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.600 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.600 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:28:36.600 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.745011+0000", 2024-09-11T08:28:36.600 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0042830589999999997 2024-09-11T08:28:36.600 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.600 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.600 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:28:36.600 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.762771+0000", 2024-09-11T08:28:36.600 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.017759257000000001 2024-09-11T08:28:36.600 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.600 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.601 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:28:36.601 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.762896+0000", 2024-09-11T08:28:36.601 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012507499999999999 2024-09-11T08:28:36.601 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.601 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.601 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:28:36.601 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.762932+0000", 2024-09-11T08:28:36.601 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.6072999999999998e-05 2024-09-11T08:28:36.601 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.601 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.601 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:28:36.601 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.762964+0000", 2024-09-11T08:28:36.601 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.2354000000000002e-05 2024-09-11T08:28:36.602 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:28:36.602 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:28:36.602 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:28:36.602 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.602 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.602 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.25799.0:129047 11.6 11:69624e9a:::benchmark_data_smithi150_187410_object2998:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-09-11T08:28:36.602 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:28:13.214331+0000", 2024-09-11T08:28:36.602 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 23.306811106000001, 2024-09-11T08:28:36.602 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.32310001100000002, 2024-09-11T08:28:36.602 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:28:36.602 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:28:36.602 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:28:36.602 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.25799", 2024-09-11T08:28:36.603 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1403604046", 2024-09-11T08:28:36.603 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 129047 2024-09-11T08:28:36.603 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.603 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:28:36.603 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.603 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:28:36.603 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.214331+0000", 2024-09-11T08:28:36.603 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:28:36.603 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.603 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.603 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:28:36.603 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.214331+0000", 2024-09-11T08:28:36.603 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:28:36.603 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.604 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.604 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:28:36.604 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.214330+0000", 2024-09-11T08:28:36.604 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:28:36.604 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.604 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.604 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:28:36.604 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.214337+0000", 2024-09-11T08:28:36.604 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.0720000000000002e-06 2024-09-11T08:28:36.604 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.604 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.604 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:28:36.604 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.214349+0000", 2024-09-11T08:28:36.604 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2609000000000001e-05 2024-09-11T08:28:36.605 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.605 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.605 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:28:36.605 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.214356+0000", 2024-09-11T08:28:36.605 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.9589999999999998e-06 2024-09-11T08:28:36.605 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.605 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.605 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:28:36.605 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.218046+0000", 2024-09-11T08:28:36.605 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0036896630000000001 2024-09-11T08:28:36.605 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.605 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.605 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:28:36.605 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.218142+0000", 2024-09-11T08:28:36.605 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.6288999999999999e-05 2024-09-11T08:28:36.606 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.606 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.606 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 1,2", 2024-09-11T08:28:36.606 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.218241+0000", 2024-09-11T08:28:36.606 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.8617999999999997e-05 2024-09-11T08:28:36.606 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.606 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.606 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:28:36.606 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.231028+0000", 2024-09-11T08:28:36.606 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.012787267 2024-09-11T08:28:36.606 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.606 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.606 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:28:36.606 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.231245+0000", 2024-09-11T08:28:36.607 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00021692 2024-09-11T08:28:36.607 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.607 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.607 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:28:36.607 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.238444+0000", 2024-09-11T08:28:36.607 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0071992050000000002 2024-09-11T08:28:36.607 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.607 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.607 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:28:36.607 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.238489+0000", 2024-09-11T08:28:36.607 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.5015999999999997e-05 2024-09-11T08:28:36.607 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.607 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.607 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:28:36.608 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.537431+0000", 2024-09-11T08:28:36.608 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.29894222199999998 2024-09-11T08:28:36.608 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:28:36.608 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:28:36.608 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:28:36.608 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.608 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.608 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.25799.0:129074 11.6 11:6d7be738:::benchmark_data_smithi150_187410_object3025:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-09-11T08:28:36.608 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:28:13.232467+0000", 2024-09-11T08:28:36.608 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 23.288675708, 2024-09-11T08:28:36.608 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.305335252, 2024-09-11T08:28:36.608 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:28:36.608 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:28:36.608 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:28:36.609 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.25799", 2024-09-11T08:28:36.609 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1403604046", 2024-09-11T08:28:36.609 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 129074 2024-09-11T08:28:36.609 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.609 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:28:36.609 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.609 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:28:36.609 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.232467+0000", 2024-09-11T08:28:36.609 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:28:36.609 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.609 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.609 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:28:36.609 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.232467+0000", 2024-09-11T08:28:36.609 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:28:36.610 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.610 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.610 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:28:36.610 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.232465+0000", 2024-09-11T08:28:36.610 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:28:36.610 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.610 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.610 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:28:36.610 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.232472+0000", 2024-09-11T08:28:36.610 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.5289999999999997e-06 2024-09-11T08:28:36.610 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.610 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.610 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:28:36.610 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.232482+0000", 2024-09-11T08:28:36.611 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0471e-05 2024-09-11T08:28:36.611 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.611 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.611 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:28:36.611 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.232488+0000", 2024-09-11T08:28:36.611 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.3990000000000003e-06 2024-09-11T08:28:36.611 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.611 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.611 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:28:36.611 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.233726+0000", 2024-09-11T08:28:36.611 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0012378949999999999 2024-09-11T08:28:36.611 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.611 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.612 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:28:36.612 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.233856+0000", 2024-09-11T08:28:36.612 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012989300000000001 2024-09-11T08:28:36.612 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.612 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.612 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 1,2", 2024-09-11T08:28:36.612 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.233971+0000", 2024-09-11T08:28:36.612 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000115783 2024-09-11T08:28:36.612 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.612 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.612 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:28:36.612 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.537453+0000", 2024-09-11T08:28:36.612 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.30348171000000002 2024-09-11T08:28:36.612 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.613 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.613 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:28:36.613 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.537669+0000", 2024-09-11T08:28:36.613 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00021578799999999999 2024-09-11T08:28:36.613 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.613 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.613 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:28:36.613 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.537717+0000", 2024-09-11T08:28:36.613 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.7939999999999998e-05 2024-09-11T08:28:36.613 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.613 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.613 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:28:36.613 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.537757+0000", 2024-09-11T08:28:36.614 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.0448000000000003e-05 2024-09-11T08:28:36.614 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.614 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.614 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:28:36.614 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.537802+0000", 2024-09-11T08:28:36.614 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.4799000000000003e-05 2024-09-11T08:28:36.614 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:28:36.614 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:28:36.614 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:28:36.614 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.614 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.614 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.25799.0:129079 11.1 e417/382 11:8f25168a:::benchmark_data_smithi150_187410_object3030:head v 417'8004, mlcod=417'8002)", 2024-09-11T08:28:36.614 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:28:13.238480+0000", 2024-09-11T08:28:36.614 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 23.282662029000001, 2024-09-11T08:28:36.615 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.30733429499999998, 2024-09-11T08:28:36.615 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:28:36.615 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:28:36.615 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:28:36.615 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.615 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:28:36.615 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.238480+0000", 2024-09-11T08:28:36.615 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:28:36.615 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.615 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.615 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:28:36.615 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.238480+0000", 2024-09-11T08:28:36.615 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:28:36.616 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.616 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.616 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:28:36.616 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.238480+0000", 2024-09-11T08:28:36.616 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:28:36.616 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.616 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.616 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:28:36.616 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.238487+0000", 2024-09-11T08:28:36.616 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.6120000000000001e-06 2024-09-11T08:28:36.616 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.616 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.616 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:28:36.616 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.238498+0000", 2024-09-11T08:28:36.617 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1085e-05 2024-09-11T08:28:36.617 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.617 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.617 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:28:36.617 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.238504+0000", 2024-09-11T08:28:36.617 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.6550000000000001e-06 2024-09-11T08:28:36.617 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.617 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.617 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:28:36.617 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.538026+0000", 2024-09-11T08:28:36.617 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.29952187200000002 2024-09-11T08:28:36.617 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.617 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.618 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:28:36.618 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.538045+0000", 2024-09-11T08:28:36.618 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.9559e-05 2024-09-11T08:28:36.618 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.618 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.618 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:28:36.618 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.545725+0000", 2024-09-11T08:28:36.618 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0076795209999999999 2024-09-11T08:28:36.618 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.618 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.618 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:28:36.618 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.545815+0000", 2024-09-11T08:28:36.618 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.9877999999999996e-05 2024-09-11T08:28:36.618 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:28:36.619 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:28:36.619 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:28:36.619 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.619 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.619 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.25799.0:129078 11.9 e417/382 11:9063d9ea:::benchmark_data_smithi150_187410_object3029:head v 417'8004, mlcod=417'8002)", 2024-09-11T08:28:36.619 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:28:13.238523+0000", 2024-09-11T08:28:36.619 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 23.282619134000001, 2024-09-11T08:28:36.619 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.30493448200000001, 2024-09-11T08:28:36.619 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:28:36.619 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:28:36.619 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:28:36.619 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.619 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:28:36.619 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.238523+0000", 2024-09-11T08:28:36.620 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:28:36.620 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.620 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.620 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:28:36.620 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.238523+0000", 2024-09-11T08:28:36.620 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:28:36.620 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.620 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.620 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:28:36.620 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.238523+0000", 2024-09-11T08:28:36.620 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:28:36.620 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.620 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.620 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:28:36.621 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.238527+0000", 2024-09-11T08:28:36.621 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.4900000000000002e-06 2024-09-11T08:28:36.621 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.621 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.621 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:28:36.621 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.537277+0000", 2024-09-11T08:28:36.621 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.29874995100000001 2024-09-11T08:28:36.621 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.621 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.621 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:28:36.621 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.537297+0000", 2024-09-11T08:28:36.621 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.9687e-05 2024-09-11T08:28:36.621 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.622 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.622 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:28:36.622 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.537604+0000", 2024-09-11T08:28:36.622 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00030737800000000002 2024-09-11T08:28:36.622 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.622 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.622 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:28:36.622 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.537630+0000", 2024-09-11T08:28:36.622 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.5510000000000001e-05 2024-09-11T08:28:36.622 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.622 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.622 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:28:36.622 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.543424+0000", 2024-09-11T08:28:36.622 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.005794581 2024-09-11T08:28:36.622 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.623 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.623 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:28:36.623 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.543458+0000", 2024-09-11T08:28:36.623 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.3356999999999999e-05 2024-09-11T08:28:36.623 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:28:36.623 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:28:36.623 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:28:36.623 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.623 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.623 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.25799.0:129086 11.2 e417/382 11:41b78a5e:::benchmark_data_smithi150_187410_object3037:head v 417'8213, mlcod=417'8211)", 2024-09-11T08:28:36.623 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:28:13.242644+0000", 2024-09-11T08:28:36.623 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 23.278498707000001, 2024-09-11T08:28:36.623 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.304782424, 2024-09-11T08:28:36.623 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:28:36.624 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:28:36.624 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:28:36.624 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.624 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:28:36.624 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.242644+0000", 2024-09-11T08:28:36.624 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:28:36.624 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.624 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.624 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:28:36.624 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.242644+0000", 2024-09-11T08:28:36.624 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:28:36.624 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.624 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.625 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:28:36.625 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.242642+0000", 2024-09-11T08:28:36.625 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:28:36.625 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.625 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.625 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:28:36.625 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.242652+0000", 2024-09-11T08:28:36.625 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.6460000000000005e-06 2024-09-11T08:28:36.625 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.625 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.625 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:28:36.625 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.537472+0000", 2024-09-11T08:28:36.625 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.29481940800000001 2024-09-11T08:28:36.625 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.626 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.626 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:28:36.626 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.537478+0000", 2024-09-11T08:28:36.626 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.4799999999999998e-06 2024-09-11T08:28:36.626 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.626 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.626 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:28:36.626 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.538064+0000", 2024-09-11T08:28:36.626 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00058608399999999995 2024-09-11T08:28:36.626 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.626 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.626 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:28:36.626 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.538082+0000", 2024-09-11T08:28:36.626 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.8011000000000001e-05 2024-09-11T08:28:36.626 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.627 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.627 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:28:36.627 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.547285+0000", 2024-09-11T08:28:36.627 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0092033600000000007 2024-09-11T08:28:36.627 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.627 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.627 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:28:36.627 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.547426+0000", 2024-09-11T08:28:36.627 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00014073499999999999 2024-09-11T08:28:36.627 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:28:36.627 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:28:36.627 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:28:36.627 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.627 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.628 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.25799.0:151899 11.6 11:6dc21136:::benchmark_data_smithi150_187410_object25850:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e422)", 2024-09-11T08:28:36.628 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:28:33.550285+0000", 2024-09-11T08:28:36.628 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 2.9708577140000001, 2024-09-11T08:28:36.628 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.30208732100000002, 2024-09-11T08:28:36.628 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:28:36.628 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:28:36.628 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:28:36.628 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.25799", 2024-09-11T08:28:36.628 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1403604046", 2024-09-11T08:28:36.628 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 151899 2024-09-11T08:28:36.628 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.628 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:28:36.628 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.629 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:28:36.629 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.550285+0000", 2024-09-11T08:28:36.629 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:28:36.629 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.629 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.629 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:28:36.629 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.550285+0000", 2024-09-11T08:28:36.629 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:28:36.629 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.629 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.629 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:28:36.629 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.550283+0000", 2024-09-11T08:28:36.629 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999976 2024-09-11T08:28:36.629 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.629 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.630 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:28:36.630 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.550290+0000", 2024-09-11T08:28:36.630 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.7840000000000004e-06 2024-09-11T08:28:36.630 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.630 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.630 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:28:36.630 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.550303+0000", 2024-09-11T08:28:36.630 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2744e-05 2024-09-11T08:28:36.630 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.630 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.630 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:28:36.630 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.550309+0000", 2024-09-11T08:28:36.630 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.1770000000000003e-06 2024-09-11T08:28:36.630 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.631 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.631 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:28:36.631 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.550361+0000", 2024-09-11T08:28:36.631 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.206e-05 2024-09-11T08:28:36.631 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.631 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.631 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:28:36.631 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.550468+0000", 2024-09-11T08:28:36.631 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00010655800000000001 2024-09-11T08:28:36.631 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.631 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.631 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 1,2", 2024-09-11T08:28:36.631 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.550571+0000", 2024-09-11T08:28:36.631 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00010272099999999999 2024-09-11T08:28:36.632 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.632 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.632 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:28:36.632 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.554020+0000", 2024-09-11T08:28:36.632 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.003449625 2024-09-11T08:28:36.632 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.632 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.632 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:28:36.632 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.560068+0000", 2024-09-11T08:28:36.632 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0060480760000000003 2024-09-11T08:28:36.632 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.632 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.632 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:28:36.632 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.852228+0000", 2024-09-11T08:28:36.633 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.29215988700000001 2024-09-11T08:28:36.633 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.633 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.633 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:28:36.633 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.852327+0000", 2024-09-11T08:28:36.633 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.8601e-05 2024-09-11T08:28:36.633 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.633 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.633 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:28:36.633 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.852372+0000", 2024-09-11T08:28:36.633 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.5247000000000002e-05 2024-09-11T08:28:36.633 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:28:36.633 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:28:36.633 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:28:36.633 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.634 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.634 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.25799.0:151910 11.1 e422/382 11:850a2b2d:::benchmark_data_smithi150_187410_object25861:head v 422'9432, mlcod=422'9429)", 2024-09-11T08:28:36.634 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:28:33.560337+0000", 2024-09-11T08:28:36.634 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 2.960805857, 2024-09-11T08:28:36.634 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.30559690699999997, 2024-09-11T08:28:36.634 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:28:36.634 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:28:36.634 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:28:36.634 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.634 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:28:36.634 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.560337+0000", 2024-09-11T08:28:36.634 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:28:36.634 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.634 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.635 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:28:36.635 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.560337+0000", 2024-09-11T08:28:36.635 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:28:36.635 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.635 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.635 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:28:36.635 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.560336+0000", 2024-09-11T08:28:36.635 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:28:36.635 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.635 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.635 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:28:36.635 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.560343+0000", 2024-09-11T08:28:36.635 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.2200000000000003e-06 2024-09-11T08:28:36.635 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.636 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.636 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:28:36.636 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.560353+0000", 2024-09-11T08:28:36.636 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0034000000000001e-05 2024-09-11T08:28:36.636 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.636 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.636 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:28:36.636 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.560359+0000", 2024-09-11T08:28:36.636 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.6180000000000003e-06 2024-09-11T08:28:36.636 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.636 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.636 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:28:36.636 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.562094+0000", 2024-09-11T08:28:36.636 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.001735826 2024-09-11T08:28:36.637 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.637 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.637 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:28:36.637 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.562112+0000", 2024-09-11T08:28:36.637 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.7900000000000001e-05 2024-09-11T08:28:36.637 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.637 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.637 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:28:36.637 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.865841+0000", 2024-09-11T08:28:36.637 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.30372890000000002 2024-09-11T08:28:36.637 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.637 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.637 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:28:36.637 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.865934+0000", 2024-09-11T08:28:36.638 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.2257999999999997e-05 2024-09-11T08:28:36.638 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:28:36.638 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:28:36.638 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:28:36.638 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.638 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.638 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.25799.0:151912 11.1 e422/382 11:8b3ccd84:::benchmark_data_smithi150_187410_object25863:head v 422'9433, mlcod=422'9429)", 2024-09-11T08:28:36.639 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:28:33.563238+0000", 2024-09-11T08:28:36.639 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 2.9579041699999999, 2024-09-11T08:28:36.639 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.30621411999999998, 2024-09-11T08:28:36.639 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:28:36.639 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:28:36.639 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:28:36.639 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.639 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:28:36.639 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.563238+0000", 2024-09-11T08:28:36.639 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:28:36.639 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.639 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.639 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:28:36.639 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.563238+0000", 2024-09-11T08:28:36.640 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:28:36.640 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.640 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.640 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:28:36.640 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.563237+0000", 2024-09-11T08:28:36.640 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:28:36.640 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.640 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.640 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:28:36.640 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.563245+0000", 2024-09-11T08:28:36.640 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.4410000000000004e-06 2024-09-11T08:28:36.640 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.640 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.641 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:28:36.641 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.563255+0000", 2024-09-11T08:28:36.641 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0321e-05 2024-09-11T08:28:36.641 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.641 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.641 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:28:36.641 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.563261+0000", 2024-09-11T08:28:36.641 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.7180000000000001e-06 2024-09-11T08:28:36.641 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.641 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.641 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:28:36.641 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.852660+0000", 2024-09-11T08:28:36.641 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.28939911299999999 2024-09-11T08:28:36.641 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.642 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.642 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:28:36.642 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.852679+0000", 2024-09-11T08:28:36.642 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.8732999999999998e-05 2024-09-11T08:28:36.642 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.642 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.642 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:28:36.642 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.869363+0000", 2024-09-11T08:28:36.642 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.016684036999999999 2024-09-11T08:28:36.642 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.642 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.642 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:28:36.642 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.869452+0000", 2024-09-11T08:28:36.642 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.9619999999999999e-05 2024-09-11T08:28:36.643 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:28:36.643 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:28:36.643 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:28:36.643 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.643 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.643 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.25799.0:151908 11.9 e422/382 11:9f54dc84:::benchmark_data_smithi150_187410_object25859:head v 422'9422, mlcod=422'9419)", 2024-09-11T08:28:36.643 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:28:33.565044+0000", 2024-09-11T08:28:36.643 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 2.9560987139999999, 2024-09-11T08:28:36.643 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.30431567500000001, 2024-09-11T08:28:36.643 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:28:36.643 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:28:36.643 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:28:36.643 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.643 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:28:36.644 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.565044+0000", 2024-09-11T08:28:36.644 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:28:36.644 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.644 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.644 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:28:36.644 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.565044+0000", 2024-09-11T08:28:36.644 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:28:36.644 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.644 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.644 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:28:36.644 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.565044+0000", 2024-09-11T08:28:36.644 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:28:36.644 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.644 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.645 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:28:36.645 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.565047+0000", 2024-09-11T08:28:36.645 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.7270000000000001e-06 2024-09-11T08:28:36.645 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.645 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.645 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:28:36.645 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.851867+0000", 2024-09-11T08:28:36.645 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.28682008199999998 2024-09-11T08:28:36.645 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.645 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.645 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:28:36.645 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.851884+0000", 2024-09-11T08:28:36.645 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.6735000000000002e-05 2024-09-11T08:28:36.646 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.646 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.646 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:28:36.646 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.852076+0000", 2024-09-11T08:28:36.646 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00019153799999999999 2024-09-11T08:28:36.646 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.646 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.646 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:28:36.646 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.852096+0000", 2024-09-11T08:28:36.646 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.033e-05 2024-09-11T08:28:36.646 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.646 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.646 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:28:36.646 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.869239+0000", 2024-09-11T08:28:36.647 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.017143366 2024-09-11T08:28:36.647 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.647 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.647 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:28:36.647 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.869359+0000", 2024-09-11T08:28:36.647 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012014399999999999 2024-09-11T08:28:36.647 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:28:36.647 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:28:36.647 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:28:36.647 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.647 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.647 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.25799.0:151915 11.1 e422/382 11:87edea39:::benchmark_data_smithi150_187410_object25866:head v 422'9434, mlcod=422'9429)", 2024-09-11T08:28:36.647 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:28:33.565601+0000", 2024-09-11T08:28:36.647 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 2.9555413399999999, 2024-09-11T08:28:36.647 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.30401631099999998, 2024-09-11T08:28:36.648 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:28:36.648 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:28:36.648 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:28:36.648 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.648 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:28:36.648 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.565601+0000", 2024-09-11T08:28:36.648 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:28:36.648 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.648 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.648 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:28:36.648 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.565601+0000", 2024-09-11T08:28:36.648 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:28:36.648 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.648 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.649 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:28:36.649 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.565600+0000", 2024-09-11T08:28:36.649 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:28:36.649 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.649 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.649 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:28:36.649 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.565608+0000", 2024-09-11T08:28:36.649 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.3379999999999999e-06 2024-09-11T08:28:36.649 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.649 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.649 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:28:36.649 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.853055+0000", 2024-09-11T08:28:36.649 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.287447591 2024-09-11T08:28:36.649 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.650 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.650 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:28:36.650 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.853062+0000", 2024-09-11T08:28:36.650 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.0450000000000003e-06 2024-09-11T08:28:36.650 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.650 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.650 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:28:36.650 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.858627+0000", 2024-09-11T08:28:36.650 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0055646769999999996 2024-09-11T08:28:36.650 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.650 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.650 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:28:36.650 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.858711+0000", 2024-09-11T08:28:36.650 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.3677000000000005e-05 2024-09-11T08:28:36.651 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.651 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.651 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:28:36.651 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.869538+0000", 2024-09-11T08:28:36.651 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.010827427000000001 2024-09-11T08:28:36.651 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:28:36.651 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:28:36.651 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:28:36.651 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.869617+0000", 2024-09-11T08:28:36.651 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.9305e-05 2024-09-11T08:28:36.651 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:28:36.651 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:28:36.651 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:28:36.651 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:28:36.651 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:28:36.652 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-11T08:28:36.697 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:36.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:36.734 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:36.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:36.835 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:36.875 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:36.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:36.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:36.977 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:37.015 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:37.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:37.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:37.117 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:28:37.142 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:37.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:37.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:28:37.230 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T08:28:37.244 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:37.274 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:37 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:28:37.275 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:37 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:28:37.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:37.314 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:37.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:37.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:37 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:28:37.415 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:28:37.473 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:37.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:37.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:28:37.575 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:37.626 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:37.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:37.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:37.728 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:37.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:37.766 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:37.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:37.867 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:37.891 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:37.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:37.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:37.994 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:28:38.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:38.024 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:38.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:28:38.126 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:38.182 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:38.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:38.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:38.284 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:38.321 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:38.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:38.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:38.322 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:38 smithi181 ceph-mon[32701]: pgmap v968: 148 pgs: 148 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 694 B/s rd, 0 B/s wr, 813 op/s 2024-09-11T08:28:38.423 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:28:38.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:38.453 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:38.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:28:38.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:38 smithi150 ceph-mon[29276]: pgmap v968: 148 pgs: 148 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 694 B/s rd, 0 B/s wr, 813 op/s 2024-09-11T08:28:38.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:38 smithi150 ceph-mon[39612]: pgmap v968: 148 pgs: 148 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 694 B/s rd, 0 B/s wr, 813 op/s 2024-09-11T08:28:38.555 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:28:38.590 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:38.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:38.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:28:38.611 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-11T08:28:38.642 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:28:38.678 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:28:38.692 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:28:38.715 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:38.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:38.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:28:38.817 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:28:38.838 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:28:38 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:28:38] "GET /metrics HTTP/1.1" 200 224695 "" "Prometheus/2.43.0" 2024-09-11T08:28:38.875 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:38.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:38.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:28:38.977 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:39.032 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:39.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:39.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:39.041 INFO:teuthology.orchestra.run.smithi150.stderr:reweighted osd.1 to 1 (10000) 2024-09-11T08:28:39.117 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:39 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:28:39.117 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:39 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:28:39.117 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:39 smithi150 ceph-mon[29276]: osdmap e424: 8 total, 8 up, 8 in 2024-09-11T08:28:39.117 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:39 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3633556905' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-11T08:28:39.117 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:39 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:28:39.118 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:39 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:28:39.118 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:39 smithi150 ceph-mon[39612]: osdmap e424: 8 total, 8 up, 8 in 2024-09-11T08:28:39.118 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:39 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3633556905' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-11T08:28:39.134 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:28:39.161 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:39.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:39.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:28:39.262 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:39.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:39.304 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:39.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:39.323 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:28:38 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:28:39.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:39 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:28:39.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:39 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:28:39.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:39 smithi181 ceph-mon[32701]: osdmap e424: 8 total, 8 up, 8 in 2024-09-11T08:28:39.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:39 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3633556905' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-11T08:28:39.405 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:28:39.440 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:39.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:39.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:28:39.542 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:39.572 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:39.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:39.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:39.674 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:28:39.711 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:39.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:39.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:28:39.813 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:28:39.842 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:39.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:39.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:28:39.944 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:39.981 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:39.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:39.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:40.049 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd reweight 2 1 2024-09-11T08:28:40.083 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:28:40.109 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:40.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:40.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:28:40.211 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:40.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:40.238 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:40.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:40.268 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:40 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:28:40.269 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:40 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3633556905' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-11T08:28:40.269 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:40 smithi150 ceph-mon[29276]: osdmap e425: 8 total, 8 up, 8 in 2024-09-11T08:28:40.269 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:40 smithi150 ceph-mon[29276]: pgmap v971: 148 pgs: 148 active+clean; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.81k op/s 2024-09-11T08:28:40.269 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:40 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:28:40.269 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:40 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3633556905' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-11T08:28:40.269 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:40 smithi150 ceph-mon[39612]: osdmap e425: 8 total, 8 up, 8 in 2024-09-11T08:28:40.269 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:40 smithi150 ceph-mon[39612]: pgmap v971: 148 pgs: 148 active+clean; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.81k op/s 2024-09-11T08:28:40.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:40 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:28:40.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:40 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3633556905' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-11T08:28:40.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:40 smithi181 ceph-mon[32701]: osdmap e425: 8 total, 8 up, 8 in 2024-09-11T08:28:40.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:40 smithi181 ceph-mon[32701]: pgmap v971: 148 pgs: 148 active+clean; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.81k op/s 2024-09-11T08:28:40.339 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:40.365 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:40.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:40.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:40.467 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:28:40.495 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:40.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:40.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:28:40.597 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:28:40.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:40.626 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:40.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:28:40.727 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:40.764 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:40.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:40.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:40.866 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:40.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:40.901 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:40.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:41.002 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:41.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:41.039 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:41.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:41.140 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:28:41.172 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:41.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:41.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:28:41.274 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:41.314 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:41.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:41.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:41.416 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:41.448 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:41.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:41.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:41.533 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:28:41.550 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:28:41.616 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:41.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:41.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:28:41.718 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:28:41.760 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:41.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:41.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:28:41.862 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:28:41.899 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:41.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:41.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:28:42.001 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:42.036 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:42.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:42.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:42.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:41 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:28:42.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:41 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:28:42.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:41 smithi181 ceph-mon[32701]: osdmap e426: 8 total, 8 up, 8 in 2024-09-11T08:28:42.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:41 smithi181 ceph-mon[32701]: pgmap v973: 148 pgs: 148 active+clean; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 929 op/s 2024-09-11T08:28:42.137 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:28:42.177 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:42.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:42.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:28:42.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:41 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:28:42.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:41 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:28:42.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:41 smithi150 ceph-mon[29276]: osdmap e426: 8 total, 8 up, 8 in 2024-09-11T08:28:42.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:41 smithi150 ceph-mon[29276]: pgmap v973: 148 pgs: 148 active+clean; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 929 op/s 2024-09-11T08:28:42.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:41 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:28:42.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:41 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:28:42.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:41 smithi150 ceph-mon[39612]: osdmap e426: 8 total, 8 up, 8 in 2024-09-11T08:28:42.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:41 smithi150 ceph-mon[39612]: pgmap v973: 148 pgs: 148 active+clean; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 929 op/s 2024-09-11T08:28:42.279 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:42.309 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:42.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:42.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:42.411 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:28:42.442 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:28:42.444 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:42.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:42.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:28:42.546 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:42.575 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:42.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:42.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:42.677 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:42.714 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:42.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:42.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:42.816 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:28:42.863 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:42.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:42.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:28:42.966 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:28:42.998 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:42.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:42.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:28:43.100 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:28:43.134 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:43.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:43.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:28:43.236 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:28:43.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:43.266 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:43.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:28:43.367 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:43.410 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:43.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:43.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:43.512 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:43.546 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:43.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:43.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:43.648 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:43.678 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:43.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:43.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:43.780 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:43.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:43.824 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:43.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:43.925 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:43.986 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:43.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:43.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:44.088 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:28:44.125 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:44.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:44.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:28:44.227 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:44.258 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:44.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:44.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:44.359 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:44.386 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:44.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:44.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:44.408 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:28:44.408 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:28:44.445 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:28:44.445 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:28:44.446 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:44 smithi150 ceph-mon[39612]: pgmap v974: 148 pgs: 148 active+clean; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 993 op/s 2024-09-11T08:28:44.446 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:44 smithi150 ceph-mon[29276]: pgmap v974: 148 pgs: 148 active+clean; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 993 op/s 2024-09-11T08:28:44.485 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:28:44.485 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:28:44.488 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:28:44.519 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:28:44.520 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:28:44.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:44.523 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:44.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:28:44.551 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:28:44.552 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:28:44.572 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:44 smithi181 ceph-mon[32701]: pgmap v974: 148 pgs: 148 active+clean; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 993 op/s 2024-09-11T08:28:44.584 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:28:44.584 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:28:44.621 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:28:44.621 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:28:44.625 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:44.662 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:28:44.663 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:28:44.670 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:44.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:44.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:44.772 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:44.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:44.819 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:44.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:44.920 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:28:44.944 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:44.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:44.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:28:45.046 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:28:45.073 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:45.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:45.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:28:45.175 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:45.206 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:45.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:45.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:45.268 INFO:teuthology.orchestra.run.smithi150.stderr:noscrub is unset 2024-09-11T08:28:45.308 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:28:45.338 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:45.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:45.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:28:45.440 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:45.467 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:45.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:45.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:45.569 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:28:45.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:28:45 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:28:45] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:28:45.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:45 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3689760440' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:28:45.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:45 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/240351707' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:28:45.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:45.605 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:45.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:28:45.651 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:45 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3689760440' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:28:45.652 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:45 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/240351707' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:28:45.652 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:45 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3689760440' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:28:45.652 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:45 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/240351707' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:28:45.707 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:28:45.741 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:45.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:45.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:28:45.842 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:45.876 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:45.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:45.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:45.978 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:46.015 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:46.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:46.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:46.118 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:46.150 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:46.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:46.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:46.252 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:46.285 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:46.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:46.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:46.386 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:46.408 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:46 smithi150 ceph-mon[29276]: pgmap v975: 148 pgs: 148 active+clean; 5.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.54k op/s 2024-09-11T08:28:46.408 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:46 smithi150 ceph-mon[29276]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:28:46.408 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:46 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/240351707' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:28:46.408 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:46 smithi150 ceph-mon[29276]: osdmap e427: 8 total, 8 up, 8 in 2024-09-11T08:28:46.408 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:46 smithi150 ceph-mon[29276]: 3.f scrub starts 2024-09-11T08:28:46.409 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:46 smithi150 ceph-mon[29276]: 3.f scrub ok 2024-09-11T08:28:46.409 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:46 smithi150 ceph-mon[39612]: pgmap v975: 148 pgs: 148 active+clean; 5.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.54k op/s 2024-09-11T08:28:46.409 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:46 smithi150 ceph-mon[39612]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:28:46.409 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:46 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/240351707' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:28:46.409 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:46 smithi150 ceph-mon[39612]: osdmap e427: 8 total, 8 up, 8 in 2024-09-11T08:28:46.409 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:46 smithi150 ceph-mon[39612]: 3.f scrub starts 2024-09-11T08:28:46.409 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:46 smithi150 ceph-mon[39612]: 3.f scrub ok 2024-09-11T08:28:46.421 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:46.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:46.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:46.523 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:46.552 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:46.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:46.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:46.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:46 smithi181 ceph-mon[32701]: pgmap v975: 148 pgs: 148 active+clean; 5.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.54k op/s 2024-09-11T08:28:46.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:46 smithi181 ceph-mon[32701]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:28:46.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:46 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/240351707' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:28:46.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:46 smithi181 ceph-mon[32701]: osdmap e427: 8 total, 8 up, 8 in 2024-09-11T08:28:46.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:46 smithi181 ceph-mon[32701]: 3.f scrub starts 2024-09-11T08:28:46.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:46 smithi181 ceph-mon[32701]: 3.f scrub ok 2024-09-11T08:28:46.654 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:46.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:46.687 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:46.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:46.788 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:28:46.820 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:46.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:46.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:28:46.922 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:28:46.951 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:46.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:46.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:28:47.053 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:47.080 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:47.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:47.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:47.182 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:28:47.207 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:47.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:47.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:28:47.309 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:28:47.347 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:47.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:47.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:28:47.449 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:47.481 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:47.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:47.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:47.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:47 smithi181 ceph-mon[32701]: 6.3 scrub starts 2024-09-11T08:28:47.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:47 smithi181 ceph-mon[32701]: 6.3 scrub ok 2024-09-11T08:28:47.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:47 smithi181 ceph-mon[32701]: 6.a scrub starts 2024-09-11T08:28:47.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:47 smithi181 ceph-mon[32701]: 6.a scrub ok 2024-09-11T08:28:47.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:47 smithi181 ceph-mon[32701]: 5.a scrub starts 2024-09-11T08:28:47.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:47 smithi181 ceph-mon[32701]: 5.a scrub ok 2024-09-11T08:28:47.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:47 smithi181 ceph-mon[32701]: 6.7 scrub starts 2024-09-11T08:28:47.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:47 smithi181 ceph-mon[32701]: 6.7 scrub ok 2024-09-11T08:28:47.583 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:28:47.610 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:47.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:47.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:28:47.667 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:47 smithi150 ceph-mon[29276]: 6.3 scrub starts 2024-09-11T08:28:47.667 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:47 smithi150 ceph-mon[29276]: 6.3 scrub ok 2024-09-11T08:28:47.668 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:47 smithi150 ceph-mon[29276]: 6.a scrub starts 2024-09-11T08:28:47.668 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:47 smithi150 ceph-mon[29276]: 6.a scrub ok 2024-09-11T08:28:47.668 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:47 smithi150 ceph-mon[29276]: 5.a scrub starts 2024-09-11T08:28:47.668 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:47 smithi150 ceph-mon[29276]: 5.a scrub ok 2024-09-11T08:28:47.668 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:47 smithi150 ceph-mon[29276]: 6.7 scrub starts 2024-09-11T08:28:47.668 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:47 smithi150 ceph-mon[29276]: 6.7 scrub ok 2024-09-11T08:28:47.668 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:47 smithi150 ceph-mon[39612]: 6.3 scrub starts 2024-09-11T08:28:47.668 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:47 smithi150 ceph-mon[39612]: 6.3 scrub ok 2024-09-11T08:28:47.668 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:47 smithi150 ceph-mon[39612]: 6.a scrub starts 2024-09-11T08:28:47.668 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:47 smithi150 ceph-mon[39612]: 6.a scrub ok 2024-09-11T08:28:47.668 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:47 smithi150 ceph-mon[39612]: 5.a scrub starts 2024-09-11T08:28:47.669 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:47 smithi150 ceph-mon[39612]: 5.a scrub ok 2024-09-11T08:28:47.669 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:47 smithi150 ceph-mon[39612]: 6.7 scrub starts 2024-09-11T08:28:47.669 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:47 smithi150 ceph-mon[39612]: 6.7 scrub ok 2024-09-11T08:28:47.694 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T08:28:47.711 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:47.736 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:47.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:47.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:47.822 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-09-11T08:28:47.822 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:28:47.838 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:28:47.893 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:47.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:47.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:28:47.996 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:48.053 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:48.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:48.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:48.155 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:48.207 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:48.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:48.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:48.309 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:48.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:48.352 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:48.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:48.450 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T08:28:48.452 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:28:48.485 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:48.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:48.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:28:48.569 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:48 smithi150 ceph-mon[29276]: 6.9 scrub starts 2024-09-11T08:28:48.569 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:48 smithi150 ceph-mon[29276]: 6.9 scrub ok 2024-09-11T08:28:48.569 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:48 smithi150 ceph-mon[29276]: 4.18 scrub starts 2024-09-11T08:28:48.569 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:48 smithi150 ceph-mon[29276]: 4.18 scrub ok 2024-09-11T08:28:48.569 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:48 smithi150 ceph-mon[29276]: pgmap v977: 148 pgs: 148 active+clean; 5.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 774 op/s 2024-09-11T08:28:48.569 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:48 smithi150 ceph-mon[29276]: 11.1 scrub starts 2024-09-11T08:28:48.569 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:48 smithi150 ceph-mon[29276]: 5.12 scrub starts 2024-09-11T08:28:48.569 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:48 smithi150 ceph-mon[29276]: 5.12 scrub ok 2024-09-11T08:28:48.569 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:48 smithi150 ceph-mon[39612]: 6.9 scrub starts 2024-09-11T08:28:48.570 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:48 smithi150 ceph-mon[39612]: 6.9 scrub ok 2024-09-11T08:28:48.570 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:48 smithi150 ceph-mon[39612]: 4.18 scrub starts 2024-09-11T08:28:48.570 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:48 smithi150 ceph-mon[39612]: 4.18 scrub ok 2024-09-11T08:28:48.570 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:48 smithi150 ceph-mon[39612]: pgmap v977: 148 pgs: 148 active+clean; 5.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 774 op/s 2024-09-11T08:28:48.570 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:48 smithi150 ceph-mon[39612]: 11.1 scrub starts 2024-09-11T08:28:48.570 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:48 smithi150 ceph-mon[39612]: 5.12 scrub starts 2024-09-11T08:28:48.570 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:48 smithi150 ceph-mon[39612]: 5.12 scrub ok 2024-09-11T08:28:48.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:48 smithi181 ceph-mon[32701]: 6.9 scrub starts 2024-09-11T08:28:48.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:48 smithi181 ceph-mon[32701]: 6.9 scrub ok 2024-09-11T08:28:48.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:48 smithi181 ceph-mon[32701]: 4.18 scrub starts 2024-09-11T08:28:48.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:48 smithi181 ceph-mon[32701]: 4.18 scrub ok 2024-09-11T08:28:48.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:48 smithi181 ceph-mon[32701]: pgmap v977: 148 pgs: 148 active+clean; 5.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 774 op/s 2024-09-11T08:28:48.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:48 smithi181 ceph-mon[32701]: 11.1 scrub starts 2024-09-11T08:28:48.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:48 smithi181 ceph-mon[32701]: 5.12 scrub starts 2024-09-11T08:28:48.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:48 smithi181 ceph-mon[32701]: 5.12 scrub ok 2024-09-11T08:28:48.588 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:28:48.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:48.623 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:48.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:28:48.725 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:28:48.754 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:48.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:48.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:28:48.856 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:48.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:48.890 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:48.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:48.988 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:28:48 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:28:48] "GET /metrics HTTP/1.1" 200 224695 "" "Prometheus/2.43.0" 2024-09-11T08:28:48.991 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:28:49.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:49.028 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:49.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:28:49.129 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:49.175 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:49.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:49.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:49.254 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:28:48 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:28:49.277 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:28:49.311 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:49.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:49.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:28:49.413 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:28:49.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:49.448 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:49.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:28:49.549 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:28:49.571 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:49 smithi150 ceph-mon[29276]: 3.2 scrub starts 2024-09-11T08:28:49.571 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:49 smithi150 ceph-mon[29276]: 3.2 scrub ok 2024-09-11T08:28:49.571 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:49 smithi150 ceph-mon[29276]: 5.1c scrub starts 2024-09-11T08:28:49.571 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:49 smithi150 ceph-mon[29276]: 5.1c scrub ok 2024-09-11T08:28:49.571 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:49 smithi150 ceph-mon[39612]: 3.2 scrub starts 2024-09-11T08:28:49.571 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:49 smithi150 ceph-mon[39612]: 3.2 scrub ok 2024-09-11T08:28:49.571 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:49 smithi150 ceph-mon[39612]: 5.1c scrub starts 2024-09-11T08:28:49.572 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:49 smithi150 ceph-mon[39612]: 5.1c scrub ok 2024-09-11T08:28:49.572 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:49 smithi181 ceph-mon[32701]: 3.2 scrub starts 2024-09-11T08:28:49.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:49 smithi181 ceph-mon[32701]: 3.2 scrub ok 2024-09-11T08:28:49.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:49 smithi181 ceph-mon[32701]: 5.1c scrub starts 2024-09-11T08:28:49.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:49 smithi181 ceph-mon[32701]: 5.1c scrub ok 2024-09-11T08:28:49.591 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:49.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:49.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:28:49.693 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:49.724 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:49.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:49.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:49.826 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:49.857 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:49.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:49.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:49.959 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:49.987 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:49.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:49.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:50.088 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:50.121 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:50.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:50.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:50.223 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:50.272 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:50.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:50.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:50.311 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-11T08:28:50.374 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:50.405 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:50.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:50.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:50.432 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:50 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:28:50.432 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:50 smithi150 ceph-mon[29276]: pgmap v978: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-09-11T08:28:50.432 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:50 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:28:50.432 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:50 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:28:50.432 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:50 smithi150 ceph-mon[29276]: 11.1 scrub ok 2024-09-11T08:28:50.433 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:50 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:28:50.433 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:50 smithi150 ceph-mon[39612]: pgmap v978: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-09-11T08:28:50.433 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:50 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:28:50.433 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:50 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:28:50.433 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:50 smithi150 ceph-mon[39612]: 11.1 scrub ok 2024-09-11T08:28:50.506 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:28:50.522 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:28:50.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:50 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:28:50.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:50 smithi181 ceph-mon[32701]: pgmap v978: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-09-11T08:28:50.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:50 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:28:50.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:50 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:28:50.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:50 smithi181 ceph-mon[32701]: 11.1 scrub ok 2024-09-11T08:28:50.605 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:50.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:50.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:28:50.707 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:28:50.736 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:50.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:50.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:28:50.838 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:28:50.872 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:50.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:50.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:28:50.974 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:28:51.004 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:51.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:51.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:28:51.106 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:51.144 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:51.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:51.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:51.246 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:51.273 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:51.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:51.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:51.325 INFO:teuthology.orchestra.run.smithi150.stderr:reweighted osd.2 to 1 (10000) 2024-09-11T08:28:51.375 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:28:51.415 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:51.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:51.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:28:51.517 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:28:51.553 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:51.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:51.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:28:51.634 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:51 smithi150 ceph-mon[39612]: osdmap e428: 8 total, 8 up, 8 in 2024-09-11T08:28:51.634 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:51 smithi150 ceph-mon[39612]: 11.3 scrub starts 2024-09-11T08:28:51.634 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:51 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/4224178076' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-11T08:28:51.634 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:51 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-11T08:28:51.635 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:51 smithi150 ceph-mon[29276]: osdmap e428: 8 total, 8 up, 8 in 2024-09-11T08:28:51.635 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:51 smithi150 ceph-mon[29276]: 11.3 scrub starts 2024-09-11T08:28:51.635 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:51 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/4224178076' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-11T08:28:51.635 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:51 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-11T08:28:51.655 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:28:51.691 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:51.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:51.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:28:51.793 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:28:51.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:51 smithi181 ceph-mon[32701]: osdmap e428: 8 total, 8 up, 8 in 2024-09-11T08:28:51.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:51 smithi181 ceph-mon[32701]: 11.3 scrub starts 2024-09-11T08:28:51.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:51 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/4224178076' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-11T08:28:51.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:51 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-11T08:28:51.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:51.824 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:51.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:28:51.925 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:28:51.954 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:51.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:51.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:28:52.056 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:28:52.098 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:52.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:52.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:28:52.200 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:28:52.229 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:52.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:52.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:28:52.330 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:28:52.331 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:52.354 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:52.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:52.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:52.456 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:52.487 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:52.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:52.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:52.589 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:28:52.631 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:52.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:52.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:28:52.733 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:28:52.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:52 smithi150 ceph-mon[29276]: 5.5 scrub starts 2024-09-11T08:28:52.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:52 smithi150 ceph-mon[29276]: 5.5 scrub ok 2024-09-11T08:28:52.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:52 smithi150 ceph-mon[29276]: pgmap v980: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-09-11T08:28:52.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:52 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-11T08:28:52.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:52 smithi150 ceph-mon[29276]: osdmap e429: 8 total, 8 up, 8 in 2024-09-11T08:28:52.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:52 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:28:52.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:52 smithi150 ceph-mon[29276]: 2.2 scrub starts 2024-09-11T08:28:52.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:52 smithi150 ceph-mon[29276]: 2.2 scrub ok 2024-09-11T08:28:52.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:52 smithi150 ceph-mon[39612]: 5.5 scrub starts 2024-09-11T08:28:52.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:52 smithi150 ceph-mon[39612]: 5.5 scrub ok 2024-09-11T08:28:52.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:52 smithi150 ceph-mon[39612]: pgmap v980: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-09-11T08:28:52.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:52 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-11T08:28:52.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:52 smithi150 ceph-mon[39612]: osdmap e429: 8 total, 8 up, 8 in 2024-09-11T08:28:52.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:52 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:28:52.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:52 smithi150 ceph-mon[39612]: 2.2 scrub starts 2024-09-11T08:28:52.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:52 smithi150 ceph-mon[39612]: 2.2 scrub ok 2024-09-11T08:28:52.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:52.766 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:52.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:28:52.818 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd reweight 3 1 2024-09-11T08:28:52.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:52 smithi181 ceph-mon[32701]: 5.5 scrub starts 2024-09-11T08:28:52.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:52 smithi181 ceph-mon[32701]: 5.5 scrub ok 2024-09-11T08:28:52.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:52 smithi181 ceph-mon[32701]: pgmap v980: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-09-11T08:28:52.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:52 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-11T08:28:52.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:52 smithi181 ceph-mon[32701]: osdmap e429: 8 total, 8 up, 8 in 2024-09-11T08:28:52.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:52 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:28:52.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:52 smithi181 ceph-mon[32701]: 2.2 scrub starts 2024-09-11T08:28:52.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:52 smithi181 ceph-mon[32701]: 2.2 scrub ok 2024-09-11T08:28:52.867 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:52.944 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:52.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:52.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:53.046 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:53.090 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:53.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:53.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:53.192 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:53.234 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:53.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:53.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:53.335 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:28:53.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:53.364 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:53.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:28:53.423 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:28:53.466 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:28:53.497 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:53.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:53.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:28:53.507 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:28:53.508 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:28:53.542 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:28:53.542 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:28:53.577 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:28:53.577 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:28:53.599 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:53.610 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:28:53.610 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:28:53.611 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:53 smithi150 ceph-mon[29276]: 11.3 scrub ok 2024-09-11T08:28:53.611 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:53 smithi150 ceph-mon[29276]: 11.a scrub starts 2024-09-11T08:28:53.611 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:53 smithi150 ceph-mon[39612]: 11.3 scrub ok 2024-09-11T08:28:53.611 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:53 smithi150 ceph-mon[39612]: 11.a scrub starts 2024-09-11T08:28:53.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:53.646 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:53.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:53.650 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:28:53.650 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:28:53.679 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:28:53.679 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:28:53.715 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:28:53.715 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:28:53.742 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:28:53.743 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:28:53.747 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:28:53.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:53.774 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:53.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:28:53.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:53 smithi181 ceph-mon[32701]: 11.3 scrub ok 2024-09-11T08:28:53.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:53 smithi181 ceph-mon[32701]: 11.a scrub starts 2024-09-11T08:28:53.876 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:28:53.907 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:53.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:53.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:28:54.009 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:28:54.050 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:54.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:54.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:28:54.152 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:28:54.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:54.190 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:54.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:28:54.291 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:54.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:54.328 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:54.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:54.429 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:28:54.465 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:54.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:54.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:28:54.567 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:28:54.597 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:54.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:54.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:28:54.699 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:54.720 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:54.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:54.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:54.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:54 smithi150 ceph-mon[39612]: pgmap v982: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 645 B/s rd, 0 B/s wr, 764 op/s 2024-09-11T08:28:54.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:54 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2224084681' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:28:54.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:54 smithi150 ceph-mon[39612]: 3.3 scrub starts 2024-09-11T08:28:54.741 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:54 smithi150 ceph-mon[39612]: 3.3 scrub ok 2024-09-11T08:28:54.742 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:54 smithi150 ceph-mon[29276]: pgmap v982: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 645 B/s rd, 0 B/s wr, 764 op/s 2024-09-11T08:28:54.742 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:54 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2224084681' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:28:54.742 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:54 smithi150 ceph-mon[29276]: 3.3 scrub starts 2024-09-11T08:28:54.742 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:54 smithi150 ceph-mon[29276]: 3.3 scrub ok 2024-09-11T08:28:54.822 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:54.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:54 smithi181 ceph-mon[32701]: pgmap v982: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 645 B/s rd, 0 B/s wr, 764 op/s 2024-09-11T08:28:54.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:54 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2224084681' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:28:54.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:54 smithi181 ceph-mon[32701]: 3.3 scrub starts 2024-09-11T08:28:54.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:54 smithi181 ceph-mon[32701]: 3.3 scrub ok 2024-09-11T08:28:54.858 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:54.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:54.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:54.961 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:28:54.994 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:54.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:54.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:28:55.095 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:55.129 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:55.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:55.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:55.231 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:55.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:55.270 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:55.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:55.371 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:28:55.412 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:55.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:55.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:28:55.514 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:28:55.542 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:55.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:55.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:28:55.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:55 smithi181 ceph-mon[32701]: 11.a scrub ok 2024-09-11T08:28:55.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:55 smithi181 ceph-mon[32701]: 11.8 scrub starts 2024-09-11T08:28:55.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:55 smithi181 ceph-mon[32701]: 3.c scrub starts 2024-09-11T08:28:55.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:55 smithi181 ceph-mon[32701]: 3.c scrub ok 2024-09-11T08:28:55.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:28:55 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:28:55] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:28:55.644 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:28:55.679 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:55.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:55.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:28:55.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:55 smithi150 ceph-mon[29276]: 11.a scrub ok 2024-09-11T08:28:55.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:55 smithi150 ceph-mon[29276]: 11.8 scrub starts 2024-09-11T08:28:55.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:55 smithi150 ceph-mon[29276]: 3.c scrub starts 2024-09-11T08:28:55.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:55 smithi150 ceph-mon[29276]: 3.c scrub ok 2024-09-11T08:28:55.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:55 smithi150 ceph-mon[39612]: 11.a scrub ok 2024-09-11T08:28:55.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:55 smithi150 ceph-mon[39612]: 11.8 scrub starts 2024-09-11T08:28:55.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:55 smithi150 ceph-mon[39612]: 3.c scrub starts 2024-09-11T08:28:55.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:55 smithi150 ceph-mon[39612]: 3.c scrub ok 2024-09-11T08:28:55.781 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:28:55.812 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:55.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:55.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:28:55.914 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:55.952 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:55.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:55.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:56.054 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:56.078 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:56.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:56.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:56.180 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:56.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:56.226 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:56.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:56.327 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:28:56.348 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:56.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:56.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:28:56.450 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:28:56.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:56 smithi150 ceph-mon[29276]: pgmap v983: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-09-11T08:28:56.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:56 smithi150 ceph-mon[29276]: 6.8 scrub starts 2024-09-11T08:28:56.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:56 smithi150 ceph-mon[29276]: 6.8 scrub ok 2024-09-11T08:28:56.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:56 smithi150 ceph-mon[29276]: 5.1d scrub starts 2024-09-11T08:28:56.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:56 smithi150 ceph-mon[29276]: 5.1d scrub ok 2024-09-11T08:28:56.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:56 smithi150 ceph-mon[29276]: 4.11 scrub starts 2024-09-11T08:28:56.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:56 smithi150 ceph-mon[29276]: 4.11 scrub ok 2024-09-11T08:28:56.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:56 smithi150 ceph-mon[29276]: 11.8 scrub ok 2024-09-11T08:28:56.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:56 smithi150 ceph-mon[39612]: pgmap v983: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-09-11T08:28:56.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:56 smithi150 ceph-mon[39612]: 6.8 scrub starts 2024-09-11T08:28:56.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:56 smithi150 ceph-mon[39612]: 6.8 scrub ok 2024-09-11T08:28:56.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:56 smithi150 ceph-mon[39612]: 5.1d scrub starts 2024-09-11T08:28:56.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:56 smithi150 ceph-mon[39612]: 5.1d scrub ok 2024-09-11T08:28:56.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:56 smithi150 ceph-mon[39612]: 4.11 scrub starts 2024-09-11T08:28:56.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:56 smithi150 ceph-mon[39612]: 4.11 scrub ok 2024-09-11T08:28:56.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:56 smithi150 ceph-mon[39612]: 11.8 scrub ok 2024-09-11T08:28:56.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:56.490 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:56.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:28:56.591 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:28:56.624 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:56.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:56.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:28:56.726 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:56.759 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:56.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:56.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:56.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:56 smithi181 ceph-mon[32701]: pgmap v983: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-09-11T08:28:56.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:56 smithi181 ceph-mon[32701]: 6.8 scrub starts 2024-09-11T08:28:56.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:56 smithi181 ceph-mon[32701]: 6.8 scrub ok 2024-09-11T08:28:56.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:56 smithi181 ceph-mon[32701]: 5.1d scrub starts 2024-09-11T08:28:56.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:56 smithi181 ceph-mon[32701]: 5.1d scrub ok 2024-09-11T08:28:56.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:56 smithi181 ceph-mon[32701]: 4.11 scrub starts 2024-09-11T08:28:56.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:56 smithi181 ceph-mon[32701]: 4.11 scrub ok 2024-09-11T08:28:56.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:56 smithi181 ceph-mon[32701]: 11.8 scrub ok 2024-09-11T08:28:56.861 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:28:56.882 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:56.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:56.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:28:56.984 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:57.021 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:57.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:57.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:57.123 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:28:57.154 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:57.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:57.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:28:57.256 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:57.283 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:57.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:57.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:57.386 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:57.419 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:57.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:57.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:57.521 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:28:57.552 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:57.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:57.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:28:57.654 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:28:57.690 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:57.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:57.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:28:57.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:57 smithi150 ceph-mon[29276]: 3.11 scrub starts 2024-09-11T08:28:57.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:57 smithi150 ceph-mon[29276]: 3.11 scrub ok 2024-09-11T08:28:57.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:57 smithi150 ceph-mon[29276]: 4.10 scrub starts 2024-09-11T08:28:57.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:57 smithi150 ceph-mon[29276]: 4.10 scrub ok 2024-09-11T08:28:57.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:57 smithi150 ceph-mon[29276]: 4.1e scrub starts 2024-09-11T08:28:57.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:57 smithi150 ceph-mon[39612]: 3.11 scrub starts 2024-09-11T08:28:57.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:57 smithi150 ceph-mon[39612]: 3.11 scrub ok 2024-09-11T08:28:57.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:57 smithi150 ceph-mon[39612]: 4.10 scrub starts 2024-09-11T08:28:57.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:57 smithi150 ceph-mon[39612]: 4.10 scrub ok 2024-09-11T08:28:57.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:57 smithi150 ceph-mon[39612]: 4.1e scrub starts 2024-09-11T08:28:57.792 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:28:57.821 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:57.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:57.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:28:57.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:57 smithi181 ceph-mon[32701]: 3.11 scrub starts 2024-09-11T08:28:57.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:57 smithi181 ceph-mon[32701]: 3.11 scrub ok 2024-09-11T08:28:57.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:57 smithi181 ceph-mon[32701]: 4.10 scrub starts 2024-09-11T08:28:57.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:57 smithi181 ceph-mon[32701]: 4.10 scrub ok 2024-09-11T08:28:57.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:57 smithi181 ceph-mon[32701]: 4.1e scrub starts 2024-09-11T08:28:57.924 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:28:57.962 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:57.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:57.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:28:58.064 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:28:58.093 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:58.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:58.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:28:58.195 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:28:58.222 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:58.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:58.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:28:58.324 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:58.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:58.366 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:58.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:58.467 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:28:58.501 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:58.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:58.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:28:58.603 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:58.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:58.642 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:58.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:58.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:58 smithi150 ceph-mon[29276]: 4.1e scrub ok 2024-09-11T08:28:58.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:58 smithi150 ceph-mon[29276]: pgmap v984: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 713 op/s 2024-09-11T08:28:58.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:58 smithi150 ceph-mon[29276]: 6.2 scrub starts 2024-09-11T08:28:58.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:58 smithi150 ceph-mon[29276]: 6.2 scrub ok 2024-09-11T08:28:58.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:58 smithi150 ceph-mon[29276]: 6.18 scrub starts 2024-09-11T08:28:58.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:58 smithi150 ceph-mon[29276]: 6.18 scrub ok 2024-09-11T08:28:58.739 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:28:58 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:28:58] "GET /metrics HTTP/1.1" 200 224692 "" "Prometheus/2.43.0" 2024-09-11T08:28:58.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:58 smithi150 ceph-mon[39612]: 4.1e scrub ok 2024-09-11T08:28:58.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:58 smithi150 ceph-mon[39612]: pgmap v984: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 713 op/s 2024-09-11T08:28:58.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:58 smithi150 ceph-mon[39612]: 6.2 scrub starts 2024-09-11T08:28:58.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:58 smithi150 ceph-mon[39612]: 6.2 scrub ok 2024-09-11T08:28:58.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:58 smithi150 ceph-mon[39612]: 6.18 scrub starts 2024-09-11T08:28:58.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:58 smithi150 ceph-mon[39612]: 6.18 scrub ok 2024-09-11T08:28:58.743 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:28:58.775 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:58.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:58.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:28:58.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:58 smithi181 ceph-mon[32701]: 4.1e scrub ok 2024-09-11T08:28:58.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:58 smithi181 ceph-mon[32701]: pgmap v984: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 713 op/s 2024-09-11T08:28:58.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:58 smithi181 ceph-mon[32701]: 6.2 scrub starts 2024-09-11T08:28:58.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:58 smithi181 ceph-mon[32701]: 6.2 scrub ok 2024-09-11T08:28:58.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:58 smithi181 ceph-mon[32701]: 6.18 scrub starts 2024-09-11T08:28:58.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:58 smithi181 ceph-mon[32701]: 6.18 scrub ok 2024-09-11T08:28:58.877 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:28:58.898 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:58.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:58.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:28:59.000 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:28:59.041 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:59.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:59.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:28:59.143 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:59.164 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:59.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:59.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:59.266 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:59.300 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:59.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:59.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:59.323 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:28:59 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:28:59.401 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:28:59.435 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:59.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:59.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:28:59.537 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:28:59.570 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:59.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:59.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:28:59.664 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:59 smithi181 ceph-mon[32701]: 5.1 scrub starts 2024-09-11T08:28:59.664 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:59 smithi181 ceph-mon[32701]: 5.1 scrub ok 2024-09-11T08:28:59.664 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:59 smithi181 ceph-mon[32701]: 4.1b scrub starts 2024-09-11T08:28:59.664 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:28:59 smithi181 ceph-mon[32701]: 4.1b scrub ok 2024-09-11T08:28:59.672 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:28:59.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:59.704 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:28:59.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:28:59.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:59 smithi150 ceph-mon[29276]: 5.1 scrub starts 2024-09-11T08:28:59.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:59 smithi150 ceph-mon[29276]: 5.1 scrub ok 2024-09-11T08:28:59.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:59 smithi150 ceph-mon[29276]: 4.1b scrub starts 2024-09-11T08:28:59.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:28:59 smithi150 ceph-mon[29276]: 4.1b scrub ok 2024-09-11T08:28:59.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:59 smithi150 ceph-mon[39612]: 5.1 scrub starts 2024-09-11T08:28:59.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:59 smithi150 ceph-mon[39612]: 5.1 scrub ok 2024-09-11T08:28:59.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:59 smithi150 ceph-mon[39612]: 4.1b scrub starts 2024-09-11T08:28:59.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:28:59 smithi150 ceph-mon[39612]: 4.1b scrub ok 2024-09-11T08:28:59.805 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:28:59.840 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:59.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:59.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:28:59.942 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:28:59.971 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:28:59.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:28:59.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:00.072 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:00.105 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:00.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:00.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:00.226 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:29:00.255 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:00.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:00.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:29:00.357 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:00.386 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:00.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:00.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:00.488 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:00.530 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:00.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:00.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:00.632 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:29:00.662 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:00.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:00.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:29:00.714 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:00 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:29:00.715 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:00 smithi150 ceph-mon[29276]: pgmap v985: 148 pgs: 148 active+clean; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-09-11T08:29:00.715 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:00 smithi150 ceph-mon[29276]: 5.0 scrub starts 2024-09-11T08:29:00.715 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:00 smithi150 ceph-mon[29276]: 5.0 scrub ok 2024-09-11T08:29:00.715 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:00 smithi150 ceph-mon[29276]: 3.6 scrub starts 2024-09-11T08:29:00.715 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:00 smithi150 ceph-mon[29276]: 3.6 scrub ok 2024-09-11T08:29:00.715 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:00 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:29:00.715 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:00 smithi150 ceph-mon[39612]: pgmap v985: 148 pgs: 148 active+clean; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-09-11T08:29:00.715 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:00 smithi150 ceph-mon[39612]: 5.0 scrub starts 2024-09-11T08:29:00.715 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:00 smithi150 ceph-mon[39612]: 5.0 scrub ok 2024-09-11T08:29:00.715 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:00 smithi150 ceph-mon[39612]: 3.6 scrub starts 2024-09-11T08:29:00.715 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:00 smithi150 ceph-mon[39612]: 3.6 scrub ok 2024-09-11T08:29:00.726 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:29:00.765 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:00.789 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:00.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:00.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:00.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:00 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:29:00.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:00 smithi181 ceph-mon[32701]: pgmap v985: 148 pgs: 148 active+clean; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-09-11T08:29:00.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:00 smithi181 ceph-mon[32701]: 5.0 scrub starts 2024-09-11T08:29:00.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:00 smithi181 ceph-mon[32701]: 5.0 scrub ok 2024-09-11T08:29:00.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:00 smithi181 ceph-mon[32701]: 3.6 scrub starts 2024-09-11T08:29:00.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:00 smithi181 ceph-mon[32701]: 3.6 scrub ok 2024-09-11T08:29:00.892 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:00.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:00.936 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:00.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:01.037 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:01.069 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:01.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:01.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:01.171 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:01.204 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:01.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:01.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:01.306 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:01.338 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:01.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:01.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:01.440 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:29:01.471 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:01.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:01.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:29:01.572 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:01.601 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:01.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:01.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:01.703 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:01.730 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:01.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:01.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:01.832 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:01.861 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:01.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:01.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:01.905 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T08:29:01.953 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:01 smithi150 ceph-mon[29276]: 3.16 scrub starts 2024-09-11T08:29:01.953 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:01 smithi150 ceph-mon[29276]: 3.16 scrub ok 2024-09-11T08:29:01.953 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:01 smithi150 ceph-mon[29276]: 3.9 scrub starts 2024-09-11T08:29:01.954 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:01 smithi150 ceph-mon[29276]: 3.9 scrub ok 2024-09-11T08:29:01.954 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:01 smithi150 ceph-mon[29276]: pgmap v986: 148 pgs: 148 active+clean; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.11k op/s 2024-09-11T08:29:01.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:01 smithi150 ceph-mon[39612]: 3.16 scrub starts 2024-09-11T08:29:01.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:01 smithi150 ceph-mon[39612]: 3.16 scrub ok 2024-09-11T08:29:01.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:01 smithi150 ceph-mon[39612]: 3.9 scrub starts 2024-09-11T08:29:01.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:01 smithi150 ceph-mon[39612]: 3.9 scrub ok 2024-09-11T08:29:01.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:01 smithi150 ceph-mon[39612]: pgmap v986: 148 pgs: 148 active+clean; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.11k op/s 2024-09-11T08:29:01.963 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:01.990 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:01.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:01.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:02.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:01 smithi181 ceph-mon[32701]: 3.16 scrub starts 2024-09-11T08:29:02.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:01 smithi181 ceph-mon[32701]: 3.16 scrub ok 2024-09-11T08:29:02.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:01 smithi181 ceph-mon[32701]: 3.9 scrub starts 2024-09-11T08:29:02.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:01 smithi181 ceph-mon[32701]: 3.9 scrub ok 2024-09-11T08:29:02.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:01 smithi181 ceph-mon[32701]: pgmap v986: 148 pgs: 148 active+clean; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.11k op/s 2024-09-11T08:29:02.091 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:29:02.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:02.134 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:02.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:29:02.235 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:02.286 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:02.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:02.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:02.388 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:02.437 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:02.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:02.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:02.540 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:29:02.561 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:02.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:02.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:29:02.663 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:29:02.691 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:02.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:02.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:29:02.793 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:29:02.820 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:02.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:02.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:29:02.922 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:02.955 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:02.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:02.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:03.057 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:03.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:02 smithi181 ceph-mon[32701]: 6.5 scrub starts 2024-09-11T08:29:03.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:02 smithi181 ceph-mon[32701]: 6.5 scrub ok 2024-09-11T08:29:03.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:02 smithi181 ceph-mon[32701]: 6.b scrub starts 2024-09-11T08:29:03.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:02 smithi181 ceph-mon[32701]: 6.b scrub ok 2024-09-11T08:29:03.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:02 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3441169854' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:29:03.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:02 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:29:03.088 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:03.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:03.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:03.190 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:29:03.227 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:03.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:03.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:29:03.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:02 smithi150 ceph-mon[29276]: 6.5 scrub starts 2024-09-11T08:29:03.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:02 smithi150 ceph-mon[29276]: 6.5 scrub ok 2024-09-11T08:29:03.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:02 smithi150 ceph-mon[29276]: 6.b scrub starts 2024-09-11T08:29:03.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:02 smithi150 ceph-mon[29276]: 6.b scrub ok 2024-09-11T08:29:03.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:02 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3441169854' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:29:03.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:02 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:29:03.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:02 smithi150 ceph-mon[39612]: 6.5 scrub starts 2024-09-11T08:29:03.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:02 smithi150 ceph-mon[39612]: 6.5 scrub ok 2024-09-11T08:29:03.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:02 smithi150 ceph-mon[39612]: 6.b scrub starts 2024-09-11T08:29:03.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:02 smithi150 ceph-mon[39612]: 6.b scrub ok 2024-09-11T08:29:03.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:02 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3441169854' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:29:03.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:02 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:29:03.329 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:03.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:03.363 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:03.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:03.464 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:03.488 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:03.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:03.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:03.590 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:03.611 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:03.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:03.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:03.713 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:03.744 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:03.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:03.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:03.846 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:03.873 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:03.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:03.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:03.975 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:03.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:03 smithi150 ceph-mon[29276]: 3.e scrub starts 2024-09-11T08:29:03.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:03 smithi150 ceph-mon[29276]: 3.e scrub ok 2024-09-11T08:29:03.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:03 smithi150 ceph-mon[29276]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:29:03.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:03 smithi150 ceph-mon[29276]: Cluster is now healthy 2024-09-11T08:29:03.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:03 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:29:03.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:03 smithi150 ceph-mon[29276]: osdmap e430: 8 total, 8 up, 8 in 2024-09-11T08:29:03.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:03 smithi150 ceph-mon[29276]: pgmap v988: 148 pgs: 148 active+clean; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s 2024-09-11T08:29:03.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:03 smithi150 ceph-mon[39612]: 3.e scrub starts 2024-09-11T08:29:03.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:03 smithi150 ceph-mon[39612]: 3.e scrub ok 2024-09-11T08:29:03.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:03 smithi150 ceph-mon[39612]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:29:03.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:03 smithi150 ceph-mon[39612]: Cluster is now healthy 2024-09-11T08:29:03.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:03 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:29:03.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:03 smithi150 ceph-mon[39612]: osdmap e430: 8 total, 8 up, 8 in 2024-09-11T08:29:03.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:03 smithi150 ceph-mon[39612]: pgmap v988: 148 pgs: 148 active+clean; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s 2024-09-11T08:29:03.996 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:03.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:03.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:04.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:03 smithi181 ceph-mon[32701]: 3.e scrub starts 2024-09-11T08:29:04.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:03 smithi181 ceph-mon[32701]: 3.e scrub ok 2024-09-11T08:29:04.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:03 smithi181 ceph-mon[32701]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:29:04.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:03 smithi181 ceph-mon[32701]: Cluster is now healthy 2024-09-11T08:29:04.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:03 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:29:04.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:03 smithi181 ceph-mon[32701]: osdmap e430: 8 total, 8 up, 8 in 2024-09-11T08:29:04.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:03 smithi181 ceph-mon[32701]: pgmap v988: 148 pgs: 148 active+clean; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s 2024-09-11T08:29:04.098 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:29:04.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:04.137 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:04.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:29:04.238 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:04.274 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:04.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:04.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:04.377 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:04.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:04.417 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:04.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:04.428 INFO:teuthology.orchestra.run.smithi150.stderr:nodeep-scrub is unset 2024-09-11T08:29:04.519 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:04.552 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:04.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:04.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:04.654 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:29:04.694 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:04.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:04.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:29:04.791 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:29:04 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a[63452]: logger=cleanup t=2024-09-11T08:29:04.495961853Z level=info msg="Completed cleanup jobs" duration=11.577994ms 2024-09-11T08:29:04.795 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:04.823 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:04.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:04.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:04.925 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:04.964 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:04.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:04.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:05.066 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:05.072 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:04 smithi181 ceph-mon[32701]: 5.8 scrub starts 2024-09-11T08:29:05.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:04 smithi181 ceph-mon[32701]: 5.8 scrub ok 2024-09-11T08:29:05.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:04 smithi181 ceph-mon[32701]: 6.1b scrub starts 2024-09-11T08:29:05.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:04 smithi181 ceph-mon[32701]: 6.1b scrub ok 2024-09-11T08:29:05.102 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:05.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:05.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:05.204 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:29:05.234 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:04 smithi150 ceph-mon[29276]: 5.8 scrub starts 2024-09-11T08:29:05.234 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:04 smithi150 ceph-mon[29276]: 5.8 scrub ok 2024-09-11T08:29:05.234 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:04 smithi150 ceph-mon[29276]: 6.1b scrub starts 2024-09-11T08:29:05.234 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:04 smithi150 ceph-mon[29276]: 6.1b scrub ok 2024-09-11T08:29:05.235 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:04 smithi150 ceph-mon[39612]: 5.8 scrub starts 2024-09-11T08:29:05.235 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:04 smithi150 ceph-mon[39612]: 5.8 scrub ok 2024-09-11T08:29:05.235 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:04 smithi150 ceph-mon[39612]: 6.1b scrub starts 2024-09-11T08:29:05.235 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:04 smithi150 ceph-mon[39612]: 6.1b scrub ok 2024-09-11T08:29:05.236 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:05.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:05.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:29:05.338 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:29:05.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:05.373 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:05.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:29:05.474 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:29:05.502 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:05.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:05.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:29:05.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:29:05 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:29:05] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:29:05.604 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:29:05.642 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:05.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:05.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:29:05.744 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:05.781 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:05.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:05.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:05.829 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:05 smithi150 ceph-mon[29276]: 5.e scrub starts 2024-09-11T08:29:05.829 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:05 smithi150 ceph-mon[29276]: 5.e scrub ok 2024-09-11T08:29:05.829 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:05 smithi150 ceph-mon[29276]: 5.9 scrub starts 2024-09-11T08:29:05.829 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:05 smithi150 ceph-mon[29276]: 5.9 scrub ok 2024-09-11T08:29:05.829 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:05 smithi150 ceph-mon[29276]: pgmap v989: 148 pgs: 148 active+clean; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s 2024-09-11T08:29:05.829 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:05 smithi150 ceph-mon[39612]: 5.e scrub starts 2024-09-11T08:29:05.829 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:05 smithi150 ceph-mon[39612]: 5.e scrub ok 2024-09-11T08:29:05.829 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:05 smithi150 ceph-mon[39612]: 5.9 scrub starts 2024-09-11T08:29:05.829 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:05 smithi150 ceph-mon[39612]: 5.9 scrub ok 2024-09-11T08:29:05.829 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:05 smithi150 ceph-mon[39612]: pgmap v989: 148 pgs: 148 active+clean; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s 2024-09-11T08:29:05.883 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:29:05.931 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:05.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:05.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:29:06.033 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:29:06.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:06.071 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:06.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:29:06.072 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:05 smithi181 ceph-mon[32701]: 5.e scrub starts 2024-09-11T08:29:06.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:05 smithi181 ceph-mon[32701]: 5.e scrub ok 2024-09-11T08:29:06.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:05 smithi181 ceph-mon[32701]: 5.9 scrub starts 2024-09-11T08:29:06.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:05 smithi181 ceph-mon[32701]: 5.9 scrub ok 2024-09-11T08:29:06.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:05 smithi181 ceph-mon[32701]: pgmap v989: 148 pgs: 148 active+clean; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s 2024-09-11T08:29:06.117 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-11T08:29:06.117 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [], 2024-09-11T08:29:06.117 INFO:teuthology.orchestra.run.smithi150.stdout: "complaint_time": 30, 2024-09-11T08:29:06.117 INFO:teuthology.orchestra.run.smithi150.stdout: "num_blocked_ops": 0 2024-09-11T08:29:06.117 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-11T08:29:06.172 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:29:06.198 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:06.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:06.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:29:06.299 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:06.344 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:06.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:06.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:06.447 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:29:06.486 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:06.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:06.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:29:06.588 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:06.620 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:06.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:06.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:06.722 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:06.755 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:06.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:06.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:06.829 INFO:teuthology.orchestra.run.smithi150.stderr:reweighted osd.3 to 1 (10000) 2024-09-11T08:29:06.857 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:06.896 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:06.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:06.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:06.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:06 smithi150 ceph-mon[29276]: 5.1b scrub starts 2024-09-11T08:29:06.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:06 smithi150 ceph-mon[29276]: 5.1b scrub ok 2024-09-11T08:29:06.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:06 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1929942659' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-11T08:29:06.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:06 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:29:06.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:06 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:29:06.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:06 smithi150 ceph-mon[39612]: 5.1b scrub starts 2024-09-11T08:29:06.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:06 smithi150 ceph-mon[39612]: 5.1b scrub ok 2024-09-11T08:29:06.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:06 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1929942659' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-11T08:29:06.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:06 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:29:06.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:06 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:29:06.998 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:29:07.037 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:07.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:07.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:29:07.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:06 smithi181 ceph-mon[32701]: 5.1b scrub starts 2024-09-11T08:29:07.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:06 smithi181 ceph-mon[32701]: 5.1b scrub ok 2024-09-11T08:29:07.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:06 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1929942659' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-11T08:29:07.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:06 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:29:07.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:06 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:29:07.139 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:29:07.163 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:07.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:07.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:29:07.168 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-11T08:29:07.265 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:29:07.294 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:07.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:07.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:29:07.396 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:07.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:07.446 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:07.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:07.547 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:07.586 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:07.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:07.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:07.688 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:07.725 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:07.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:07.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:07.827 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:29:07.860 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:07.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:07.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:29:07.962 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:07.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:07 smithi150 ceph-mon[29276]: 6.16 deep-scrub starts 2024-09-11T08:29:07.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:07 smithi150 ceph-mon[29276]: 6.16 deep-scrub ok 2024-09-11T08:29:07.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:07 smithi150 ceph-mon[29276]: 5.10 scrub starts 2024-09-11T08:29:07.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:07 smithi150 ceph-mon[29276]: 5.10 scrub ok 2024-09-11T08:29:07.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:07 smithi150 ceph-mon[29276]: 5.d scrub starts 2024-09-11T08:29:07.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:07 smithi150 ceph-mon[29276]: 5.d scrub ok 2024-09-11T08:29:07.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:07 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1929942659' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-11T08:29:07.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:07 smithi150 ceph-mon[29276]: osdmap e431: 8 total, 8 up, 8 in 2024-09-11T08:29:07.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:07 smithi150 ceph-mon[29276]: pgmap v991: 148 pgs: 148 active+clean; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 733 op/s 2024-09-11T08:29:07.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:07 smithi150 ceph-mon[39612]: 6.16 deep-scrub starts 2024-09-11T08:29:07.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:07 smithi150 ceph-mon[39612]: 6.16 deep-scrub ok 2024-09-11T08:29:07.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:07 smithi150 ceph-mon[39612]: 5.10 scrub starts 2024-09-11T08:29:07.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:07 smithi150 ceph-mon[39612]: 5.10 scrub ok 2024-09-11T08:29:07.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:07 smithi150 ceph-mon[39612]: 5.d scrub starts 2024-09-11T08:29:07.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:07 smithi150 ceph-mon[39612]: 5.d scrub ok 2024-09-11T08:29:07.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:07 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1929942659' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-11T08:29:07.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:07 smithi150 ceph-mon[39612]: osdmap e431: 8 total, 8 up, 8 in 2024-09-11T08:29:07.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:07 smithi150 ceph-mon[39612]: pgmap v991: 148 pgs: 148 active+clean; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 733 op/s 2024-09-11T08:29:08.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:08.008 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:08.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:08.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:07 smithi181 ceph-mon[32701]: 6.16 deep-scrub starts 2024-09-11T08:29:08.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:07 smithi181 ceph-mon[32701]: 6.16 deep-scrub ok 2024-09-11T08:29:08.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:07 smithi181 ceph-mon[32701]: 5.10 scrub starts 2024-09-11T08:29:08.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:07 smithi181 ceph-mon[32701]: 5.10 scrub ok 2024-09-11T08:29:08.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:07 smithi181 ceph-mon[32701]: 5.d scrub starts 2024-09-11T08:29:08.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:07 smithi181 ceph-mon[32701]: 5.d scrub ok 2024-09-11T08:29:08.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:07 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1929942659' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-11T08:29:08.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:07 smithi181 ceph-mon[32701]: osdmap e431: 8 total, 8 up, 8 in 2024-09-11T08:29:08.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:07 smithi181 ceph-mon[32701]: pgmap v991: 148 pgs: 148 active+clean; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 733 op/s 2024-09-11T08:29:08.109 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:29:08.148 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:08.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:08.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:29:08.250 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:29:08.281 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:08.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:08.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:29:08.382 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:08.396 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:29:08.406 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd reweight 4 1 2024-09-11T08:29:08.409 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:08.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:08.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:08.425 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:29:08.511 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:08.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:08.558 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:08.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:08.607 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:29:08 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:29:08] "GET /metrics HTTP/1.1" 200 224699 "" "Prometheus/2.43.0" 2024-09-11T08:29:08.660 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:08.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:08.689 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:08.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:08.790 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:29:08.813 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:08.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:08.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:29:08.867 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:08 smithi150 ceph-mon[29276]: 4.1f deep-scrub starts 2024-09-11T08:29:08.867 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:08 smithi150 ceph-mon[29276]: 4.1f deep-scrub ok 2024-09-11T08:29:08.867 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:08 smithi150 ceph-mon[29276]: 4.1c scrub starts 2024-09-11T08:29:08.867 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:08 smithi150 ceph-mon[29276]: 4.1c scrub ok 2024-09-11T08:29:08.868 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:08 smithi150 ceph-mon[39612]: 4.1f deep-scrub starts 2024-09-11T08:29:08.868 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:08 smithi150 ceph-mon[39612]: 4.1f deep-scrub ok 2024-09-11T08:29:08.868 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:08 smithi150 ceph-mon[39612]: 4.1c scrub starts 2024-09-11T08:29:08.868 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:08 smithi150 ceph-mon[39612]: 4.1c scrub ok 2024-09-11T08:29:08.915 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:08.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:08.950 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:08.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:09.051 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:09.078 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:09.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:09.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:09.180 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:09.203 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:09.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:09.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:09.305 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:09.323 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:29:09 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:29:09.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:08 smithi181 ceph-mon[32701]: 4.1f deep-scrub starts 2024-09-11T08:29:09.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:08 smithi181 ceph-mon[32701]: 4.1f deep-scrub ok 2024-09-11T08:29:09.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:08 smithi181 ceph-mon[32701]: 4.1c scrub starts 2024-09-11T08:29:09.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:08 smithi181 ceph-mon[32701]: 4.1c scrub ok 2024-09-11T08:29:09.336 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:09.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:09.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:09.438 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:29:09.466 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:09.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:09.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:29:09.568 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:09.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:09.607 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:09.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:09.660 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T08:29:09.708 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:09.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:09.743 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:09.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:09.844 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:09.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:09.908 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:09.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:09.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:09 smithi150 ceph-mon[29276]: 3.1a scrub starts 2024-09-11T08:29:09.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:09 smithi150 ceph-mon[29276]: 3.1a scrub ok 2024-09-11T08:29:09.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:09 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:29:09.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:09 smithi150 ceph-mon[29276]: pgmap v992: 148 pgs: 148 active+clean; 3.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s 2024-09-11T08:29:09.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:09 smithi150 ceph-mon[39612]: 3.1a scrub starts 2024-09-11T08:29:09.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:09 smithi150 ceph-mon[39612]: 3.1a scrub ok 2024-09-11T08:29:09.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:09 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:29:09.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:09 smithi150 ceph-mon[39612]: pgmap v992: 148 pgs: 148 active+clean; 3.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s 2024-09-11T08:29:10.009 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:10.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:10.061 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:10.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:10.163 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:10.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:10.185 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:10.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:10.286 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:10.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:09 smithi181 ceph-mon[32701]: 3.1a scrub starts 2024-09-11T08:29:10.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:09 smithi181 ceph-mon[32701]: 3.1a scrub ok 2024-09-11T08:29:10.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:09 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:29:10.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:09 smithi181 ceph-mon[32701]: pgmap v992: 148 pgs: 148 active+clean; 3.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s 2024-09-11T08:29:10.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:10.326 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:10.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:10.427 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:29:10.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:10.461 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:10.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:29:10.561 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:10.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:10.602 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:10.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:10.703 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:29:10.741 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:10.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:10.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:29:10.760 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:29:10.843 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:10.895 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:10.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:10.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:10.997 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:11.043 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:11.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:11.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:11.144 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:29:11.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:11.181 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:11.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:29:11.231 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:10 smithi150 ceph-mon[29276]: 4.1d scrub starts 2024-09-11T08:29:11.231 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:10 smithi150 ceph-mon[29276]: 4.1d scrub ok 2024-09-11T08:29:11.231 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:10 smithi150 ceph-mon[39612]: 4.1d scrub starts 2024-09-11T08:29:11.231 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:10 smithi150 ceph-mon[39612]: 4.1d scrub ok 2024-09-11T08:29:11.282 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:11.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:11.313 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:11.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:11.322 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:10 smithi181 ceph-mon[32701]: 4.1d scrub starts 2024-09-11T08:29:11.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:10 smithi181 ceph-mon[32701]: 4.1d scrub ok 2024-09-11T08:29:11.414 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:11.444 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:11.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:11.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:11.546 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:11.586 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:11.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:11.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:11.688 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:11.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:11.718 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:11.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:11.819 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:29:11.853 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:11.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:11.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:29:11.955 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:12.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:12.000 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:12.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:12.101 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:29:12.137 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:11 smithi150 ceph-mon[29276]: 4.6 scrub starts 2024-09-11T08:29:12.137 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:11 smithi150 ceph-mon[29276]: 4.6 scrub ok 2024-09-11T08:29:12.137 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:11 smithi150 ceph-mon[29276]: 4.e deep-scrub starts 2024-09-11T08:29:12.137 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:11 smithi150 ceph-mon[29276]: 4.e deep-scrub ok 2024-09-11T08:29:12.137 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:11 smithi150 ceph-mon[29276]: pgmap v993: 148 pgs: 148 active+clean; 3.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-09-11T08:29:12.141 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:11 smithi150 ceph-mon[39612]: 4.6 scrub starts 2024-09-11T08:29:12.141 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:11 smithi150 ceph-mon[39612]: 4.6 scrub ok 2024-09-11T08:29:12.141 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:11 smithi150 ceph-mon[39612]: 4.e deep-scrub starts 2024-09-11T08:29:12.141 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:11 smithi150 ceph-mon[39612]: 4.e deep-scrub ok 2024-09-11T08:29:12.141 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:11 smithi150 ceph-mon[39612]: pgmap v993: 148 pgs: 148 active+clean; 3.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-09-11T08:29:12.163 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:12.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:12.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:29:12.265 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:29:12.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:12.301 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:12.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:29:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:11 smithi181 ceph-mon[32701]: 4.6 scrub starts 2024-09-11T08:29:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:11 smithi181 ceph-mon[32701]: 4.6 scrub ok 2024-09-11T08:29:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:11 smithi181 ceph-mon[32701]: 4.e deep-scrub starts 2024-09-11T08:29:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:11 smithi181 ceph-mon[32701]: 4.e deep-scrub ok 2024-09-11T08:29:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:11 smithi181 ceph-mon[32701]: pgmap v993: 148 pgs: 148 active+clean; 3.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-09-11T08:29:12.401 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:29:12.432 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:12.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:12.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:29:12.533 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:12.577 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:12.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:12.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:12.679 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:12.680 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:29:12.681 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:29:12.708 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:12.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:12.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:12.715 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:29:12.715 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:29:12.753 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:29:12.753 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:29:12.795 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:29:12.795 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:29:12.810 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:12.837 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:29:12.837 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:29:12.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:12.848 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:12.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:12.881 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:29:12.881 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:29:12.912 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:29:12.913 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:29:12.918 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-09-11T08:29:12.918 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:29:12.933 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:12 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:29:12.933 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:12 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:29:12.933 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:12 smithi150 ceph-mon[39612]: 5.b scrub starts 2024-09-11T08:29:12.933 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:12 smithi150 ceph-mon[39612]: 5.b scrub ok 2024-09-11T08:29:12.933 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:12 smithi150 ceph-mon[39612]: osdmap e432: 8 total, 8 up, 8 in 2024-09-11T08:29:12.933 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:12 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/4147989958' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:29:12.934 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:12 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:29:12.934 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:12 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:29:12.934 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:12 smithi150 ceph-mon[29276]: 5.b scrub starts 2024-09-11T08:29:12.934 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:12 smithi150 ceph-mon[29276]: 5.b scrub ok 2024-09-11T08:29:12.934 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:12 smithi150 ceph-mon[29276]: osdmap e432: 8 total, 8 up, 8 in 2024-09-11T08:29:12.934 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:12 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/4147989958' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:29:12.947 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:29:12.948 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:29:12.949 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:12.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:12.987 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:12.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:13.089 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:13.118 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:13.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:13.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:13.219 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:13.259 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:13.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:13.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:13.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:12 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:29:13.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:12 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:29:13.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:12 smithi181 ceph-mon[32701]: 5.b scrub starts 2024-09-11T08:29:13.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:12 smithi181 ceph-mon[32701]: 5.b scrub ok 2024-09-11T08:29:13.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:12 smithi181 ceph-mon[32701]: osdmap e432: 8 total, 8 up, 8 in 2024-09-11T08:29:13.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:12 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/4147989958' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:29:13.362 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:13.395 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:13.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:13.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:13.497 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:13.525 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:13.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:13.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:13.626 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:13.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:13.659 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:13.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:13.761 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:29:13.785 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:13.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:13.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:29:13.887 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:13.928 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:13.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:13.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:14.030 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:14.058 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:14.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:14.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:14.160 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:14.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:14.203 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:14.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:14.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:13 smithi150 ceph-mon[29276]: 11.6 scrub starts 2024-09-11T08:29:14.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:13 smithi150 ceph-mon[29276]: 3.1d scrub starts 2024-09-11T08:29:14.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:13 smithi150 ceph-mon[29276]: 3.1d scrub ok 2024-09-11T08:29:14.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:13 smithi150 ceph-mon[29276]: 3.14 scrub starts 2024-09-11T08:29:14.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:13 smithi150 ceph-mon[29276]: 3.14 scrub ok 2024-09-11T08:29:14.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:13 smithi150 ceph-mon[29276]: 6.14 scrub starts 2024-09-11T08:29:14.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:13 smithi150 ceph-mon[29276]: 6.14 scrub ok 2024-09-11T08:29:14.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:13 smithi150 ceph-mon[29276]: pgmap v995: 148 pgs: 148 active+clean; 3.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 752 op/s 2024-09-11T08:29:14.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:13 smithi150 ceph-mon[39612]: 11.6 scrub starts 2024-09-11T08:29:14.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:13 smithi150 ceph-mon[39612]: 3.1d scrub starts 2024-09-11T08:29:14.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:13 smithi150 ceph-mon[39612]: 3.1d scrub ok 2024-09-11T08:29:14.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:13 smithi150 ceph-mon[39612]: 3.14 scrub starts 2024-09-11T08:29:14.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:13 smithi150 ceph-mon[39612]: 3.14 scrub ok 2024-09-11T08:29:14.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:13 smithi150 ceph-mon[39612]: 6.14 scrub starts 2024-09-11T08:29:14.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:13 smithi150 ceph-mon[39612]: 6.14 scrub ok 2024-09-11T08:29:14.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:13 smithi150 ceph-mon[39612]: pgmap v995: 148 pgs: 148 active+clean; 3.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 752 op/s 2024-09-11T08:29:14.304 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:29:14.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:13 smithi181 ceph-mon[32701]: 11.6 scrub starts 2024-09-11T08:29:14.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:13 smithi181 ceph-mon[32701]: 3.1d scrub starts 2024-09-11T08:29:14.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:13 smithi181 ceph-mon[32701]: 3.1d scrub ok 2024-09-11T08:29:14.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:13 smithi181 ceph-mon[32701]: 3.14 scrub starts 2024-09-11T08:29:14.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:13 smithi181 ceph-mon[32701]: 3.14 scrub ok 2024-09-11T08:29:14.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:13 smithi181 ceph-mon[32701]: 6.14 scrub starts 2024-09-11T08:29:14.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:13 smithi181 ceph-mon[32701]: 6.14 scrub ok 2024-09-11T08:29:14.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:13 smithi181 ceph-mon[32701]: pgmap v995: 148 pgs: 148 active+clean; 3.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 752 op/s 2024-09-11T08:29:14.346 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:14.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:14.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:29:14.447 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:14.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:14.486 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:14.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:14.587 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-11T08:29:14.587 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:14.614 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:14.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:14.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:14.716 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:14.746 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:14.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:14.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:14.848 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:14.876 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:14.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:14.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:14.978 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:29:15.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:15.008 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:15.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:29:15.109 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:15.143 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:14 smithi181 ceph-mon[32701]: 4.5 scrub starts 2024-09-11T08:29:15.143 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:14 smithi181 ceph-mon[32701]: 4.5 scrub ok 2024-09-11T08:29:15.143 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:14 smithi181 ceph-mon[32701]: 11.6 scrub ok 2024-09-11T08:29:15.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:15.164 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:15.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:15.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:14 smithi150 ceph-mon[29276]: 4.5 scrub starts 2024-09-11T08:29:15.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:14 smithi150 ceph-mon[29276]: 4.5 scrub ok 2024-09-11T08:29:15.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:14 smithi150 ceph-mon[29276]: 11.6 scrub ok 2024-09-11T08:29:15.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:14 smithi150 ceph-mon[39612]: 4.5 scrub starts 2024-09-11T08:29:15.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:14 smithi150 ceph-mon[39612]: 4.5 scrub ok 2024-09-11T08:29:15.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:14 smithi150 ceph-mon[39612]: 11.6 scrub ok 2024-09-11T08:29:15.265 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:15.306 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:15.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:15.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:15.408 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:15.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:15.441 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:15.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:15.542 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:15.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:29:15 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:29:15] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:29:15.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:15.575 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:15.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:15.676 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:29:15.711 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:29:15.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:15.712 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:15.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:29:15.734 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:29:15.813 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:29:15.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:15.862 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:15.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:29:15.963 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:15.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:15 smithi150 ceph-mon[29276]: 5.1f scrub starts 2024-09-11T08:29:15.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:15 smithi150 ceph-mon[29276]: 5.1f scrub ok 2024-09-11T08:29:15.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:15 smithi150 ceph-mon[29276]: 3.8 deep-scrub starts 2024-09-11T08:29:15.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:15 smithi150 ceph-mon[29276]: 3.8 deep-scrub ok 2024-09-11T08:29:15.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:15 smithi150 ceph-mon[29276]: 6.e scrub starts 2024-09-11T08:29:15.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:15 smithi150 ceph-mon[29276]: 6.e scrub ok 2024-09-11T08:29:15.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:15 smithi150 ceph-mon[29276]: pgmap v996: 148 pgs: 148 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.34k op/s 2024-09-11T08:29:15.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:15 smithi150 ceph-mon[39612]: 5.1f scrub starts 2024-09-11T08:29:15.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:15 smithi150 ceph-mon[39612]: 5.1f scrub ok 2024-09-11T08:29:15.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:15 smithi150 ceph-mon[39612]: 3.8 deep-scrub starts 2024-09-11T08:29:15.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:15 smithi150 ceph-mon[39612]: 3.8 deep-scrub ok 2024-09-11T08:29:15.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:15 smithi150 ceph-mon[39612]: 6.e scrub starts 2024-09-11T08:29:15.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:15 smithi150 ceph-mon[39612]: 6.e scrub ok 2024-09-11T08:29:15.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:15 smithi150 ceph-mon[39612]: pgmap v996: 148 pgs: 148 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.34k op/s 2024-09-11T08:29:15.992 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:15.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:15.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:16.094 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:16.125 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:16.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:16.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:16.227 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:16.256 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:16.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:16.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:16.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:15 smithi181 ceph-mon[32701]: 5.1f scrub starts 2024-09-11T08:29:16.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:15 smithi181 ceph-mon[32701]: 5.1f scrub ok 2024-09-11T08:29:16.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:15 smithi181 ceph-mon[32701]: 3.8 deep-scrub starts 2024-09-11T08:29:16.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:15 smithi181 ceph-mon[32701]: 3.8 deep-scrub ok 2024-09-11T08:29:16.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:15 smithi181 ceph-mon[32701]: 6.e scrub starts 2024-09-11T08:29:16.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:15 smithi181 ceph-mon[32701]: 6.e scrub ok 2024-09-11T08:29:16.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:15 smithi181 ceph-mon[32701]: pgmap v996: 148 pgs: 148 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.34k op/s 2024-09-11T08:29:16.358 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:16.396 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:16.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:16.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:16.498 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:16.535 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:16.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:16.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:16.637 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:16.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:16.672 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:16.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:16.773 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:16.813 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:16.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:16.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:16.914 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:16.943 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:16.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:16.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:17.045 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:17.059 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T08:29:17.075 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:17.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:17.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:17.177 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:17.190 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:16 smithi150 ceph-mon[29276]: 4.17 deep-scrub starts 2024-09-11T08:29:17.190 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:16 smithi150 ceph-mon[29276]: 4.14 scrub starts 2024-09-11T08:29:17.190 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:16 smithi150 ceph-mon[29276]: 4.17 deep-scrub ok 2024-09-11T08:29:17.190 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:16 smithi150 ceph-mon[29276]: 4.14 scrub ok 2024-09-11T08:29:17.190 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:16 smithi150 ceph-mon[29276]: 3.0 scrub starts 2024-09-11T08:29:17.190 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:16 smithi150 ceph-mon[29276]: 3.0 scrub ok 2024-09-11T08:29:17.191 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:16 smithi150 ceph-mon[39612]: 4.17 deep-scrub starts 2024-09-11T08:29:17.191 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:16 smithi150 ceph-mon[39612]: 4.14 scrub starts 2024-09-11T08:29:17.191 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:16 smithi150 ceph-mon[39612]: 4.17 deep-scrub ok 2024-09-11T08:29:17.191 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:16 smithi150 ceph-mon[39612]: 4.14 scrub ok 2024-09-11T08:29:17.191 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:16 smithi150 ceph-mon[39612]: 3.0 scrub starts 2024-09-11T08:29:17.191 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:16 smithi150 ceph-mon[39612]: 3.0 scrub ok 2024-09-11T08:29:17.220 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:17.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:17.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:17.322 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:29:17.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:16 smithi181 ceph-mon[32701]: 4.17 deep-scrub starts 2024-09-11T08:29:17.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:16 smithi181 ceph-mon[32701]: 4.14 scrub starts 2024-09-11T08:29:17.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:16 smithi181 ceph-mon[32701]: 4.17 deep-scrub ok 2024-09-11T08:29:17.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:16 smithi181 ceph-mon[32701]: 4.14 scrub ok 2024-09-11T08:29:17.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:16 smithi181 ceph-mon[32701]: 3.0 scrub starts 2024-09-11T08:29:17.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:16 smithi181 ceph-mon[32701]: 3.0 scrub ok 2024-09-11T08:29:17.348 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:17.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:17.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:29:17.450 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:17.474 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:17.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:17.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:17.576 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:17.619 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:17.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:17.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:17.721 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:17.742 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:17.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:17.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:17.843 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:17.877 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:17.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:17.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:17.979 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:18.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:18.006 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:18.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:18.107 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:18.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:18.134 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:18.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:18.215 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:17 smithi150 ceph-mon[29276]: 3.1b scrub starts 2024-09-11T08:29:18.216 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:17 smithi150 ceph-mon[29276]: 3.1b scrub ok 2024-09-11T08:29:18.216 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:17 smithi150 ceph-mon[29276]: 6.c deep-scrub starts 2024-09-11T08:29:18.216 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:17 smithi150 ceph-mon[29276]: 6.c deep-scrub ok 2024-09-11T08:29:18.216 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:17 smithi150 ceph-mon[29276]: pgmap v997: 148 pgs: 148 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s 2024-09-11T08:29:18.216 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:17 smithi150 ceph-mon[39612]: 3.1b scrub starts 2024-09-11T08:29:18.216 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:17 smithi150 ceph-mon[39612]: 3.1b scrub ok 2024-09-11T08:29:18.216 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:17 smithi150 ceph-mon[39612]: 6.c deep-scrub starts 2024-09-11T08:29:18.216 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:17 smithi150 ceph-mon[39612]: 6.c deep-scrub ok 2024-09-11T08:29:18.217 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:17 smithi150 ceph-mon[39612]: pgmap v997: 148 pgs: 148 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s 2024-09-11T08:29:18.224 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:17 smithi181 ceph-mon[32701]: 3.1b scrub starts 2024-09-11T08:29:18.224 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:17 smithi181 ceph-mon[32701]: 3.1b scrub ok 2024-09-11T08:29:18.224 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:17 smithi181 ceph-mon[32701]: 6.c deep-scrub starts 2024-09-11T08:29:18.224 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:17 smithi181 ceph-mon[32701]: 6.c deep-scrub ok 2024-09-11T08:29:18.224 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:17 smithi181 ceph-mon[32701]: pgmap v997: 148 pgs: 148 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s 2024-09-11T08:29:18.234 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:18.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:18.273 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:18.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:18.375 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:29:18.412 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:18.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:18.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:29:18.514 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:18.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:18.552 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:18.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:18.653 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:29:18.692 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:18.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:18.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:29:18.794 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:18.841 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:18.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:18.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:18.917 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:29:18 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:29:18] "GET /metrics HTTP/1.1" 200 224699 "" "Prometheus/2.43.0" 2024-09-11T08:29:18.943 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:18.973 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:18.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:18.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:19.075 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:19.102 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:19.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:19.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:19.171 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:18 smithi150 ceph-mon[29276]: 3.10 scrub starts 2024-09-11T08:29:19.171 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:18 smithi150 ceph-mon[29276]: 3.10 scrub ok 2024-09-11T08:29:19.172 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:18 smithi150 ceph-mon[29276]: 4.19 scrub starts 2024-09-11T08:29:19.172 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:18 smithi150 ceph-mon[29276]: 4.19 scrub ok 2024-09-11T08:29:19.172 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:18 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:29:19.172 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:18 smithi150 ceph-mon[39612]: 3.10 scrub starts 2024-09-11T08:29:19.172 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:18 smithi150 ceph-mon[39612]: 3.10 scrub ok 2024-09-11T08:29:19.172 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:18 smithi150 ceph-mon[39612]: 4.19 scrub starts 2024-09-11T08:29:19.172 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:18 smithi150 ceph-mon[39612]: 4.19 scrub ok 2024-09-11T08:29:19.172 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:18 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:29:19.204 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:29:19.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:19.247 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:19.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:29:19.323 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:29:19 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:29:19.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:18 smithi181 ceph-mon[32701]: 3.10 scrub starts 2024-09-11T08:29:19.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:18 smithi181 ceph-mon[32701]: 3.10 scrub ok 2024-09-11T08:29:19.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:18 smithi181 ceph-mon[32701]: 4.19 scrub starts 2024-09-11T08:29:19.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:18 smithi181 ceph-mon[32701]: 4.19 scrub ok 2024-09-11T08:29:19.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:18 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:29:19.349 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:19.380 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:19.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:19.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:19.481 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:29:19.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:19.520 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:19.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:29:19.621 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:29:19.650 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:19.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:19.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:29:19.752 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:29:19.787 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:19.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:19.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:29:19.888 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:29:19.925 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:19.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:19.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:29:19.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:19 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:29:19.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:19 smithi150 ceph-mon[29276]: 4.1a scrub starts 2024-09-11T08:29:19.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:19 smithi150 ceph-mon[29276]: 4.1a scrub ok 2024-09-11T08:29:19.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:19 smithi150 ceph-mon[29276]: pgmap v998: 148 pgs: 148 active+clean; 3.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s 2024-09-11T08:29:19.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:19 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:29:19.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:19 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:29:19.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:19 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:29:19.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:19 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/4022420084' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-11T08:29:19.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:19 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-11T08:29:19.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:19 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:29:19.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:19 smithi150 ceph-mon[39612]: 4.1a scrub starts 2024-09-11T08:29:19.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:19 smithi150 ceph-mon[39612]: 4.1a scrub ok 2024-09-11T08:29:19.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:19 smithi150 ceph-mon[39612]: pgmap v998: 148 pgs: 148 active+clean; 3.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s 2024-09-11T08:29:19.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:19 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:29:19.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:19 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:29:19.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:19 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:29:19.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:19 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/4022420084' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-11T08:29:19.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:19 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-11T08:29:20.026 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:20.061 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:20.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:20.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:20.163 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:20.208 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:20.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:20.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:20.310 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:20.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:19 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:29:20.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:19 smithi181 ceph-mon[32701]: 4.1a scrub starts 2024-09-11T08:29:20.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:19 smithi181 ceph-mon[32701]: 4.1a scrub ok 2024-09-11T08:29:20.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:19 smithi181 ceph-mon[32701]: pgmap v998: 148 pgs: 148 active+clean; 3.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s 2024-09-11T08:29:20.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:19 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:29:20.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:19 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:29:20.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:19 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:29:20.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:19 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/4022420084' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-11T08:29:20.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:19 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-11T08:29:20.346 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:20.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:20.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:20.448 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:29:20.476 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:20.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:20.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:29:20.554 INFO:teuthology.orchestra.run.smithi150.stderr:reweighted osd.4 to 1 (10000) 2024-09-11T08:29:20.578 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:29:20.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:20.651 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:20.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:29:20.752 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:20.780 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:20.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:20.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:20.882 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:20.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:20.925 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:20.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:21.026 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:29:21.058 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:21.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:21.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:29:21.160 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:21.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:21.207 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:21.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:21.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:20 smithi150 ceph-mon[29276]: 4.a scrub starts 2024-09-11T08:29:21.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:20 smithi150 ceph-mon[29276]: 4.a scrub ok 2024-09-11T08:29:21.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:20 smithi150 ceph-mon[29276]: 3.7 scrub starts 2024-09-11T08:29:21.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:20 smithi150 ceph-mon[29276]: 3.7 scrub ok 2024-09-11T08:29:21.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:20 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-11T08:29:21.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:20 smithi150 ceph-mon[29276]: osdmap e433: 8 total, 8 up, 8 in 2024-09-11T08:29:21.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:20 smithi150 ceph-mon[39612]: 4.a scrub starts 2024-09-11T08:29:21.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:20 smithi150 ceph-mon[39612]: 4.a scrub ok 2024-09-11T08:29:21.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:20 smithi150 ceph-mon[39612]: 3.7 scrub starts 2024-09-11T08:29:21.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:20 smithi150 ceph-mon[39612]: 3.7 scrub ok 2024-09-11T08:29:21.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:20 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-11T08:29:21.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:20 smithi150 ceph-mon[39612]: osdmap e433: 8 total, 8 up, 8 in 2024-09-11T08:29:21.309 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:21.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:20 smithi181 ceph-mon[32701]: 4.a scrub starts 2024-09-11T08:29:21.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:20 smithi181 ceph-mon[32701]: 4.a scrub ok 2024-09-11T08:29:21.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:20 smithi181 ceph-mon[32701]: 3.7 scrub starts 2024-09-11T08:29:21.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:20 smithi181 ceph-mon[32701]: 3.7 scrub ok 2024-09-11T08:29:21.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:20 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-11T08:29:21.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:20 smithi181 ceph-mon[32701]: osdmap e433: 8 total, 8 up, 8 in 2024-09-11T08:29:21.338 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:21.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:21.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:21.440 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:21.465 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:21.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:21.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:21.567 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:21.606 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:21.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:21.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:21.707 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:21.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:21.747 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:21.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:21.766 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd reweight 6 1 2024-09-11T08:29:21.849 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:21.898 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:21.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:21.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:21.977 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:21 smithi150 ceph-mon[29276]: 6.15 deep-scrub starts 2024-09-11T08:29:21.977 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:21 smithi150 ceph-mon[29276]: 6.15 deep-scrub ok 2024-09-11T08:29:21.977 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:21 smithi150 ceph-mon[29276]: 4.12 scrub starts 2024-09-11T08:29:21.977 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:21 smithi150 ceph-mon[29276]: 4.12 scrub ok 2024-09-11T08:29:21.977 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:21 smithi150 ceph-mon[29276]: pgmap v1000: 148 pgs: 148 active+clean; 3.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.20k op/s 2024-09-11T08:29:21.977 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:21 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:29:21.977 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:21 smithi150 ceph-mon[39612]: 6.15 deep-scrub starts 2024-09-11T08:29:21.978 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:21 smithi150 ceph-mon[39612]: 6.15 deep-scrub ok 2024-09-11T08:29:21.978 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:21 smithi150 ceph-mon[39612]: 4.12 scrub starts 2024-09-11T08:29:21.978 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:21 smithi150 ceph-mon[39612]: 4.12 scrub ok 2024-09-11T08:29:21.978 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:21 smithi150 ceph-mon[39612]: pgmap v1000: 148 pgs: 148 active+clean; 3.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.20k op/s 2024-09-11T08:29:21.978 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:21 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:29:22.000 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:22.028 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:22.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:22.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:22.129 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:22.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:22.200 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:22.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:22.301 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:22.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:21 smithi181 ceph-mon[32701]: 6.15 deep-scrub starts 2024-09-11T08:29:22.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:21 smithi181 ceph-mon[32701]: 6.15 deep-scrub ok 2024-09-11T08:29:22.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:21 smithi181 ceph-mon[32701]: 4.12 scrub starts 2024-09-11T08:29:22.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:21 smithi181 ceph-mon[32701]: 4.12 scrub ok 2024-09-11T08:29:22.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:21 smithi181 ceph-mon[32701]: pgmap v1000: 148 pgs: 148 active+clean; 3.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.20k op/s 2024-09-11T08:29:22.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:21 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:29:22.328 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:22.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:22.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:22.430 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:22.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:22.458 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:22.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:22.559 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:22.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:22.597 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:22.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:22.658 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-09-11T08:29:22.658 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:29:22.698 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:22.734 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:22.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:22.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:22.836 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:29:22.868 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:22.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:22.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:29:22.889 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:29:22.970 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:23.007 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:23.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:23.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:23.108 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:23.152 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:23.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:23.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:23.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:22 smithi150 ceph-mon[29276]: 4.1 scrub starts 2024-09-11T08:29:23.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:22 smithi150 ceph-mon[29276]: 4.1 scrub ok 2024-09-11T08:29:23.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:22 smithi150 ceph-mon[29276]: 11.4 scrub starts 2024-09-11T08:29:23.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:22 smithi150 ceph-mon[39612]: 4.1 scrub starts 2024-09-11T08:29:23.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:22 smithi150 ceph-mon[39612]: 4.1 scrub ok 2024-09-11T08:29:23.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:22 smithi150 ceph-mon[39612]: 11.4 scrub starts 2024-09-11T08:29:23.254 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:29:23.284 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:23.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:23.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:29:23.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:22 smithi181 ceph-mon[32701]: 4.1 scrub starts 2024-09-11T08:29:23.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:22 smithi181 ceph-mon[32701]: 4.1 scrub ok 2024-09-11T08:29:23.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:22 smithi181 ceph-mon[32701]: 11.4 scrub starts 2024-09-11T08:29:23.386 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:23.414 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:23.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:23.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:23.508 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-11T08:29:23.516 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:23.591 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:23.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:23.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:23.694 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:23.738 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:23.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:23.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:23.840 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:23.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:23.897 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:23.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:23.998 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:24.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:24.039 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:24.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:24.140 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:29:24.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:24.174 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:24.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:29:24.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:23 smithi150 ceph-mon[29276]: 11.4 scrub ok 2024-09-11T08:29:24.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:23 smithi150 ceph-mon[29276]: pgmap v1001: 148 pgs: 148 active+clean; 3.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-09-11T08:29:24.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:23 smithi150 ceph-mon[29276]: 6.11 scrub starts 2024-09-11T08:29:24.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:23 smithi150 ceph-mon[29276]: 6.11 scrub ok 2024-09-11T08:29:24.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:23 smithi150 ceph-mon[39612]: 11.4 scrub ok 2024-09-11T08:29:24.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:23 smithi150 ceph-mon[39612]: pgmap v1001: 148 pgs: 148 active+clean; 3.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-09-11T08:29:24.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:23 smithi150 ceph-mon[39612]: 6.11 scrub starts 2024-09-11T08:29:24.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:23 smithi150 ceph-mon[39612]: 6.11 scrub ok 2024-09-11T08:29:24.275 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:29:24.296 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:24.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:24.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:29:24.322 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:23 smithi181 ceph-mon[32701]: 11.4 scrub ok 2024-09-11T08:29:24.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:23 smithi181 ceph-mon[32701]: pgmap v1001: 148 pgs: 148 active+clean; 3.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-09-11T08:29:24.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:23 smithi181 ceph-mon[32701]: 6.11 scrub starts 2024-09-11T08:29:24.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:23 smithi181 ceph-mon[32701]: 6.11 scrub ok 2024-09-11T08:29:24.398 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:24.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:24.427 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:24.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:24.528 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:24.562 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:24.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:24.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:24.663 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:29:24.700 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:24.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:24.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:29:24.802 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:29:24.835 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:24.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:24.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:29:24.937 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:24.967 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:24.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:24.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:24.984 INFO:teuthology.orchestra.run.smithi150.stderr:noscrub is set 2024-09-11T08:29:25.069 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:25.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:25.108 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:25.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:25.209 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:29:25.234 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:25.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:25.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:29:25.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:24 smithi150 ceph-mon[29276]: 5.18 scrub starts 2024-09-11T08:29:25.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:24 smithi150 ceph-mon[29276]: 5.18 scrub ok 2024-09-11T08:29:25.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:24 smithi150 ceph-mon[29276]: 11.f scrub starts 2024-09-11T08:29:25.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:24 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1639760387' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:29:25.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:24 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:29:25.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:24 smithi150 ceph-mon[39612]: 5.18 scrub starts 2024-09-11T08:29:25.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:24 smithi150 ceph-mon[39612]: 5.18 scrub ok 2024-09-11T08:29:25.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:24 smithi150 ceph-mon[39612]: 11.f scrub starts 2024-09-11T08:29:25.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:24 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1639760387' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:29:25.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:24 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:29:25.323 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:29:25 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:29:25] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:29:25.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:24 smithi181 ceph-mon[32701]: 5.18 scrub starts 2024-09-11T08:29:25.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:24 smithi181 ceph-mon[32701]: 5.18 scrub ok 2024-09-11T08:29:25.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:24 smithi181 ceph-mon[32701]: 11.f scrub starts 2024-09-11T08:29:25.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:24 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1639760387' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:29:25.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:24 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:29:25.336 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:29:25.364 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:25.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:25.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:29:25.466 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:25.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:25.503 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:25.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:25.604 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:29:25.628 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:25.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:25.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:29:25.730 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:25.773 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:25.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:25.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:25.875 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:25.911 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:25.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:25.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:26.012 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:26.044 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:26.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:26.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:26.146 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:26.183 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:26.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:26.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:26.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:26 smithi150 ceph-mon[29276]: 5.2 deep-scrub starts 2024-09-11T08:29:26.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:26 smithi150 ceph-mon[29276]: 5.2 deep-scrub ok 2024-09-11T08:29:26.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:26 smithi150 ceph-mon[29276]: 11.f scrub ok 2024-09-11T08:29:26.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:26 smithi150 ceph-mon[29276]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:29:26.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:26 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:29:26.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:26 smithi150 ceph-mon[29276]: osdmap e434: 8 total, 8 up, 8 in 2024-09-11T08:29:26.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:26 smithi150 ceph-mon[29276]: pgmap v1003: 148 pgs: 148 active+clean; 2.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.37k op/s 2024-09-11T08:29:26.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:26 smithi150 ceph-mon[39612]: 5.2 deep-scrub starts 2024-09-11T08:29:26.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:26 smithi150 ceph-mon[39612]: 5.2 deep-scrub ok 2024-09-11T08:29:26.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:26 smithi150 ceph-mon[39612]: 11.f scrub ok 2024-09-11T08:29:26.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:26 smithi150 ceph-mon[39612]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:29:26.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:26 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:29:26.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:26 smithi150 ceph-mon[39612]: osdmap e434: 8 total, 8 up, 8 in 2024-09-11T08:29:26.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:26 smithi150 ceph-mon[39612]: pgmap v1003: 148 pgs: 148 active+clean; 2.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.37k op/s 2024-09-11T08:29:26.285 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:26.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:26.318 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:26.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:26.322 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:26 smithi181 ceph-mon[32701]: 5.2 deep-scrub starts 2024-09-11T08:29:26.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:26 smithi181 ceph-mon[32701]: 5.2 deep-scrub ok 2024-09-11T08:29:26.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:26 smithi181 ceph-mon[32701]: 11.f scrub ok 2024-09-11T08:29:26.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:26 smithi181 ceph-mon[32701]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:29:26.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:26 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:29:26.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:26 smithi181 ceph-mon[32701]: osdmap e434: 8 total, 8 up, 8 in 2024-09-11T08:29:26.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:26 smithi181 ceph-mon[32701]: pgmap v1003: 148 pgs: 148 active+clean; 2.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.37k op/s 2024-09-11T08:29:26.351 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:29:26.419 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:26.435 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:29:26.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:26.476 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:26.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:26.577 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:29:26.608 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:26.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:26.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:29:26.709 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:26.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:26.758 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:26.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:26.859 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:29:26.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:26.886 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:26.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:29:26.987 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:27.019 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:27.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:27.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:27.121 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:29:27.153 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:27.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:27.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:29:27.255 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:27.291 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:27.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:27.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:27.392 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:27.427 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:27.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:27.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:27.529 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:27.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:27.563 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:27.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:27.664 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:29:27.695 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:27.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:27.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:29:27.796 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:27.834 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:27.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:27.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:27.936 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:27.942 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T08:29:27.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:27.972 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:27.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:28.073 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:28.111 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:28.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:28.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:28.212 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:28.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:28.247 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:28.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:28.348 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:29:28.376 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:28.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:28.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:29:28.478 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:28.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:28.521 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:28.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:28.571 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:28 smithi150 ceph-mon[29276]: pgmap v1004: 148 pgs: 148 active+clean; 2.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 711 op/s 2024-09-11T08:29:28.572 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:28 smithi150 ceph-mon[39612]: pgmap v1004: 148 pgs: 148 active+clean; 2.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 711 op/s 2024-09-11T08:29:28.572 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:28 smithi181 ceph-mon[32701]: pgmap v1004: 148 pgs: 148 active+clean; 2.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 711 op/s 2024-09-11T08:29:28.622 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:28.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:28.662 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:28.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:28.763 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:28.797 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:28.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:28.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:28.899 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:28.937 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:28.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:28.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:28.988 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:29:28 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:29:28] "GET /metrics HTTP/1.1" 200 224726 "" "Prometheus/2.43.0" 2024-09-11T08:29:29.040 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:29.076 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:29.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:29.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:29.178 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:29:29.217 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:29.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:29.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:29:29.319 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:29:29.322 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:29:29 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:29:29.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:29.353 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:29.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:29:29.454 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:29.491 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:29.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:29.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:29.592 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:29.626 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:29.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:29.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:29.727 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:29.764 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:29.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:29.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:29.866 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:29:29.896 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:29.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:29.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:29:29.997 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:30.028 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:30.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:30.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:30.130 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:29:30.159 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:30.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:30.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:29:30.261 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:29:30.294 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:30.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:30.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:29:30.396 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:30.432 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:30.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:30.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:30.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:30 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:29:30.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:30 smithi150 ceph-mon[29276]: pgmap v1005: 148 pgs: 148 active+clean; 2.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-09-11T08:29:30.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:30 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:29:30.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:30 smithi150 ceph-mon[39612]: pgmap v1005: 148 pgs: 148 active+clean; 2.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-09-11T08:29:30.534 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:29:30.570 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:30.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:30.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:29:30.572 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:30 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:29:30.572 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:30 smithi181 ceph-mon[32701]: pgmap v1005: 148 pgs: 148 active+clean; 2.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-09-11T08:29:30.672 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:30.712 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:30.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:30.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:30.815 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:30.851 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:30.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:30.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:30.953 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:30.981 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:30.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:30.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:31.083 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:31.116 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:31.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:31.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:31.218 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:31.248 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:31.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:31.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:31.349 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:31.392 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:31.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:31.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:31.494 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:31.529 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:31.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:31.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:31.632 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:31.665 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:31.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:31.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:31.766 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:29:31.799 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:31.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:31.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:29:31.901 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:31.929 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:31.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:31.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:32.031 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:29:32.070 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:32.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:32.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:29:32.172 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:32.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:32.211 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:32.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:32.312 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:32.343 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:32.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:32.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:32.445 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:32.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:32.484 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:32.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:32.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:32 smithi150 ceph-mon[29276]: pgmap v1006: 148 pgs: 148 active+clean; 2.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 1.09k op/s 2024-09-11T08:29:32.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:32 smithi150 ceph-mon[39612]: pgmap v1006: 148 pgs: 148 active+clean; 2.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 1.09k op/s 2024-09-11T08:29:32.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:32 smithi181 ceph-mon[32701]: pgmap v1006: 148 pgs: 148 active+clean; 2.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 1.09k op/s 2024-09-11T08:29:32.585 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:32.596 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:29:32.597 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:29:32.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:32.627 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:32.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:32.631 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:29:32.631 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:29:32.659 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:29:32.659 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:29:32.694 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:29:32.695 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:29:32.726 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:29:32.727 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:29:32.728 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:29:32.754 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:29:32.754 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:29:32.767 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:32.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:32.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:29:32.785 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:29:32.785 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:29:32.817 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:29:32.818 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:29:32.868 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:29:32.895 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:32.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:32.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:29:32.997 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:33.038 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:33.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:33.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:33.140 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:33.170 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:33.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:33.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:33.272 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:29:33.300 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:33.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:33.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:29:33.401 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:33 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2181600880' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:29:33.401 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:33 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2181600880' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:29:33.402 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:29:33.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:33.426 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:33.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:29:33.527 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:33.558 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:33.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:33.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:33.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:33 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2181600880' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:29:33.660 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:33.694 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:33.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:33.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:33.776 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-11T08:29:33.776 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [], 2024-09-11T08:29:33.776 INFO:teuthology.orchestra.run.smithi150.stdout: "complaint_time": 30, 2024-09-11T08:29:33.776 INFO:teuthology.orchestra.run.smithi150.stdout: "num_blocked_ops": 0 2024-09-11T08:29:33.776 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-11T08:29:33.795 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:33.828 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:33.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:33.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:33.930 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:29:33.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:33.960 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:33.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:29:34.061 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:34.097 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:34.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:34.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:34.199 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:34.232 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:34.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:34.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:34.334 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:29:34.359 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:34.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:34.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:29:34.423 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:34 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:29:34.424 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:34 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:29:34.424 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:34 smithi150 ceph-mon[29276]: pgmap v1007: 148 pgs: 148 active+clean; 2.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 1.11k op/s 2024-09-11T08:29:34.424 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:34 smithi150 ceph-mon[29276]: osdmap e435: 8 total, 8 up, 8 in 2024-09-11T08:29:34.424 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:34 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:29:34.424 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:34 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:29:34.424 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:34 smithi150 ceph-mon[39612]: pgmap v1007: 148 pgs: 148 active+clean; 2.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 1.11k op/s 2024-09-11T08:29:34.425 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:34 smithi150 ceph-mon[39612]: osdmap e435: 8 total, 8 up, 8 in 2024-09-11T08:29:34.461 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:34.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:34.493 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:34.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:34.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:34 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:29:34.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:34 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:29:34.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:34 smithi181 ceph-mon[32701]: pgmap v1007: 148 pgs: 148 active+clean; 2.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 1.11k op/s 2024-09-11T08:29:34.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:34 smithi181 ceph-mon[32701]: osdmap e435: 8 total, 8 up, 8 in 2024-09-11T08:29:34.594 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:34.623 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:34.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:34.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:34.724 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:29:34.754 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:34.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:34.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:29:34.856 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:34.889 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:34.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:34.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:34.991 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:35.033 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:35.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:35.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:35.135 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:35.163 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:35.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:35.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:35.265 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:35.301 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:35.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:35.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:35.313 INFO:teuthology.orchestra.run.smithi150.stderr:reweighted osd.6 to 1 (10000) 2024-09-11T08:29:35.403 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:29:35.438 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:35.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:35.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:29:35.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:35 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/64229310' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-11T08:29:35.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:35 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-11T08:29:35.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:35 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/64229310' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-11T08:29:35.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:35 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-11T08:29:35.540 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:35.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:35 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/64229310' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-11T08:29:35.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:35 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-11T08:29:35.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:29:35 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:29:35] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:29:35.577 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:35.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:35.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:35.638 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-11T08:29:35.679 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:29:35.705 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:35.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:35.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:29:35.807 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:35.836 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:35.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:35.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:35.878 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:29:35.938 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:36.025 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:36.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:36.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:36.127 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:36.153 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:36.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:36.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:36.255 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:29:36.286 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:36.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:36.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:29:36.333 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd reweight 7 1 2024-09-11T08:29:36.388 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:29:36.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:36.450 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:36.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:29:36.551 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:29:36.572 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:36 smithi181 ceph-mon[32701]: pgmap v1009: 148 pgs: 148 active+clean; 2.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.15k op/s 2024-09-11T08:29:36.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:36 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-11T08:29:36.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:36 smithi181 ceph-mon[32701]: osdmap e436: 8 total, 8 up, 8 in 2024-09-11T08:29:36.592 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:36.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:36.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:29:36.644 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:36 smithi150 ceph-mon[29276]: pgmap v1009: 148 pgs: 148 active+clean; 2.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.15k op/s 2024-09-11T08:29:36.645 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:36 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-11T08:29:36.645 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:36 smithi150 ceph-mon[29276]: osdmap e436: 8 total, 8 up, 8 in 2024-09-11T08:29:36.645 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:36 smithi150 ceph-mon[39612]: pgmap v1009: 148 pgs: 148 active+clean; 2.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.15k op/s 2024-09-11T08:29:36.645 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:36 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-11T08:29:36.645 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:36 smithi150 ceph-mon[39612]: osdmap e436: 8 total, 8 up, 8 in 2024-09-11T08:29:36.694 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:36.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:36.742 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:36.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:36.843 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:36.887 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:36.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:36.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:36.989 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:37.028 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:37.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:37.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:37.130 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:37.133 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T08:29:37.153 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:37.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:37.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:37.197 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:29:37.257 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:37.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:37.341 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:37.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:37.442 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:37.493 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:37.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:37.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:37.594 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:37.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:37.627 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:37.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:37.675 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:29:37.728 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:29:37.752 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:37.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:37.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:29:37.854 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:37.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:37.892 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:37.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:37.986 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:37 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:29:37.986 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:37 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:29:37.987 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:37 smithi150 ceph-mon[29276]: pgmap v1011: 148 pgs: 148 active+clean; 2.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 785 op/s 2024-09-11T08:29:37.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:37 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:29:37.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:37 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:29:37.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:37 smithi150 ceph-mon[39612]: pgmap v1011: 148 pgs: 148 active+clean; 2.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 785 op/s 2024-09-11T08:29:37.993 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:38.027 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:38.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:38.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:38.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:37 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:29:38.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:37 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:29:38.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:37 smithi181 ceph-mon[32701]: pgmap v1011: 148 pgs: 148 active+clean; 2.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 785 op/s 2024-09-11T08:29:38.128 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:29:38.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:38.159 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:38.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:29:38.260 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:38.298 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:38.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:38.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:38.400 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:38.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:38.430 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:38.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:38.531 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:38.563 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:38.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:38.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:38.665 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:29:38.693 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:29:38.710 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:38.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:38.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:29:38.812 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:29:38.859 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:38.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:38.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:29:38.960 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:29:38.988 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:29:38 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:29:38] "GET /metrics HTTP/1.1" 200 224727 "" "Prometheus/2.43.0" 2024-09-11T08:29:38.993 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:38.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:38.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:29:39.096 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:39.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:39.132 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:39.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:39.233 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:39.269 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:39.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:39.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:39.323 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:29:39 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:29:39.371 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:39.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:39.401 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:39.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:39.502 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:29:39.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:39.537 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:39.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:29:39.638 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:39.675 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:39.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:39.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:39.777 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:39.807 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:39.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:39.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:39.909 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:39.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:39.942 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:39.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:40.044 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:29:40.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:40.080 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:40.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:29:40.181 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:40.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:40.215 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:40.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:40.316 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:40.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:40.342 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:40.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:40.443 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:40.464 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:40.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:40.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:40.565 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:40.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:40 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:29:40.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:40 smithi181 ceph-mon[32701]: pgmap v1012: 148 pgs: 148 active+clean; 1.7 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.41k op/s 2024-09-11T08:29:40.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:40 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:29:40.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:40 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:29:40.587 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:40.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:40.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:40.688 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:40.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:40.717 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:40.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:40.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:40 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:29:40.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:40 smithi150 ceph-mon[29276]: pgmap v1012: 148 pgs: 148 active+clean; 1.7 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.41k op/s 2024-09-11T08:29:40.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:40 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:29:40.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:40 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:29:40.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:40 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:29:40.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:40 smithi150 ceph-mon[39612]: pgmap v1012: 148 pgs: 148 active+clean; 1.7 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.41k op/s 2024-09-11T08:29:40.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:40 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:29:40.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:40 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:29:40.818 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:40.846 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:40.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:40.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:40.948 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:29:40.977 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:40.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:40.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:29:41.080 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:41.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:41.118 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:41.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:41.220 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:41.251 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:41.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:41.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:41.353 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:41.399 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:41.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:41.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:41.413 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:41 smithi150 ceph-mon[29276]: osdmap e437: 8 total, 8 up, 8 in 2024-09-11T08:29:41.413 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:41 smithi150 ceph-mon[39612]: osdmap e437: 8 total, 8 up, 8 in 2024-09-11T08:29:41.502 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:41.532 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:41.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:41.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:41.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:41 smithi181 ceph-mon[32701]: osdmap e437: 8 total, 8 up, 8 in 2024-09-11T08:29:41.634 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:29:41.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:41.675 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:41.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:29:41.777 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:41.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:41.822 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:41.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:41.923 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:41.961 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:41.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:41.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:42.005 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-11T08:29:42.006 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 20, 2024-09-11T08:29:42.006 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 600, 2024-09-11T08:29:42.006 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [ 2024-09-11T08:29:42.006 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.006 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:96722 10.9 10:974c6299:::benchmark_data_smithi150_158067_object96721:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e329)", 2024-09-11T08:29:42.006 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:22:58.186775+0000", 2024-09-11T08:29:42.006 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 403.81472536400003, 2024-09-11T08:29:42.006 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.99107653200000001, 2024-09-11T08:29:42.006 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:29:42.006 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:29:42.006 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:29:42.006 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:29:42.006 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:29:42.007 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 96722 2024-09-11T08:29:42.007 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.007 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:29:42.007 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.007 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:29:42.007 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.186775+0000", 2024-09-11T08:29:42.007 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:29:42.007 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.007 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.007 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:29:42.007 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.186775+0000", 2024-09-11T08:29:42.007 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:29:42.007 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.007 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.008 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:29:42.008 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.186773+0000", 2024-09-11T08:29:42.008 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:29:42.008 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.008 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.008 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:29:42.008 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.186821+0000", 2024-09-11T08:29:42.008 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.8099999999999997e-05 2024-09-11T08:29:42.008 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.008 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.008 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:29:42.008 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.186834+0000", 2024-09-11T08:29:42.009 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3344e-05 2024-09-11T08:29:42.009 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.009 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.009 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:29:42.009 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.186840+0000", 2024-09-11T08:29:42.009 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.7289999999999997e-06 2024-09-11T08:29:42.009 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.009 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.009 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:29:42.009 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.186893+0000", 2024-09-11T08:29:42.009 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.2549000000000001e-05 2024-09-11T08:29:42.009 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.009 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.010 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:29:42.010 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.186904+0000", 2024-09-11T08:29:42.010 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1029000000000001e-05 2024-09-11T08:29:42.010 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.010 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.010 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:29:42.010 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.167018+0000", 2024-09-11T08:29:42.010 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.98011428899999997 2024-09-11T08:29:42.010 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.010 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.011 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:29:42.011 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.168686+0000", 2024-09-11T08:29:42.011 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.001667717 2024-09-11T08:29:42.011 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.011 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.011 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,7", 2024-09-11T08:29:42.011 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.168855+0000", 2024-09-11T08:29:42.011 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00016924299999999999 2024-09-11T08:29:42.011 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.011 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.011 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:29:42.011 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.175952+0000", 2024-09-11T08:29:42.012 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0070972479999999996 2024-09-11T08:29:42.012 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.012 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.012 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:29:42.012 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.177762+0000", 2024-09-11T08:29:42.012 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.001810416 2024-09-11T08:29:42.012 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.012 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.012 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:29:42.012 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.177797+0000", 2024-09-11T08:29:42.012 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.4048999999999999e-05 2024-09-11T08:29:42.012 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.012 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.013 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:29:42.013 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.177827+0000", 2024-09-11T08:29:42.013 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.0725999999999997e-05 2024-09-11T08:29:42.013 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.013 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.013 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:29:42.013 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.177851+0000", 2024-09-11T08:29:42.013 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.3777e-05 2024-09-11T08:29:42.013 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:29:42.013 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:29:42.013 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:29:42.013 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.013 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.014 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:96770 10.9 10:90fb0a7a:::benchmark_data_smithi150_158067_object96769:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e329)", 2024-09-11T08:29:42.014 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:22:58.240293+0000", 2024-09-11T08:29:42.014 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 403.76120662900001, 2024-09-11T08:29:42.014 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.94590357400000002, 2024-09-11T08:29:42.014 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:29:42.014 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:29:42.014 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:29:42.014 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:29:42.014 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:29:42.014 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 96770 2024-09-11T08:29:42.014 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.015 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:29:42.015 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.015 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:29:42.015 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.240293+0000", 2024-09-11T08:29:42.015 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:29:42.015 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.015 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.015 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:29:42.015 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.240293+0000", 2024-09-11T08:29:42.015 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:29:42.015 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.015 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.015 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:29:42.015 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.240291+0000", 2024-09-11T08:29:42.016 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:29:42.016 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.016 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.016 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:29:42.016 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.240315+0000", 2024-09-11T08:29:42.016 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.4329e-05 2024-09-11T08:29:42.016 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.016 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.016 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:29:42.016 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.240326+0000", 2024-09-11T08:29:42.016 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0969e-05 2024-09-11T08:29:42.016 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.016 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.016 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:29:42.016 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.240333+0000", 2024-09-11T08:29:42.017 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.1670000000000004e-06 2024-09-11T08:29:42.017 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.017 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.017 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:29:42.017 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.240389+0000", 2024-09-11T08:29:42.017 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.6026000000000002e-05 2024-09-11T08:29:42.017 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.017 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.017 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:29:42.017 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.240399+0000", 2024-09-11T08:29:42.017 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.9210000000000004e-06 2024-09-11T08:29:42.017 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.017 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.017 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:29:42.017 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.171307+0000", 2024-09-11T08:29:42.018 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.93090849499999995 2024-09-11T08:29:42.018 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.018 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.018 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:29:42.018 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.171435+0000", 2024-09-11T08:29:42.018 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012800100000000001 2024-09-11T08:29:42.018 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.018 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.018 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,7", 2024-09-11T08:29:42.018 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.171575+0000", 2024-09-11T08:29:42.018 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00014013599999999999 2024-09-11T08:29:42.018 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.018 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.019 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:29:42.019 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.177885+0000", 2024-09-11T08:29:42.019 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0063097320000000002 2024-09-11T08:29:42.019 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.019 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.019 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:29:42.019 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.177934+0000", 2024-09-11T08:29:42.019 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.9217e-05 2024-09-11T08:29:42.019 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.019 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.019 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:29:42.019 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.186143+0000", 2024-09-11T08:29:42.019 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0082086190000000003 2024-09-11T08:29:42.020 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.020 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.020 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:29:42.020 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.186173+0000", 2024-09-11T08:29:42.020 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.0252000000000001e-05 2024-09-11T08:29:42.020 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.020 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.020 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:29:42.020 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.186197+0000", 2024-09-11T08:29:42.020 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.3816999999999999e-05 2024-09-11T08:29:42.020 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:29:42.020 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:29:42.021 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:29:42.021 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.021 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.021 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:96776 10.9 10:94bc6bc2:::benchmark_data_smithi150_158067_object96775:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e329)", 2024-09-11T08:29:42.021 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:22:58.245049+0000", 2024-09-11T08:29:42.021 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 403.75645088900001, 2024-09-11T08:29:42.021 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.94422798699999999, 2024-09-11T08:29:42.021 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:29:42.021 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:29:42.021 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:29:42.021 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:29:42.021 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:29:42.021 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 96776 2024-09-11T08:29:42.022 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.022 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:29:42.022 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.022 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:29:42.022 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.245049+0000", 2024-09-11T08:29:42.022 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:29:42.022 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.022 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.022 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:29:42.022 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.245049+0000", 2024-09-11T08:29:42.022 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:29:42.022 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.022 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.022 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:29:42.023 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.245047+0000", 2024-09-11T08:29:42.023 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:29:42.023 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.023 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.023 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:29:42.023 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.245073+0000", 2024-09-11T08:29:42.023 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.5703000000000001e-05 2024-09-11T08:29:42.023 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.023 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.023 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:29:42.023 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.245097+0000", 2024-09-11T08:29:42.023 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.4056000000000001e-05 2024-09-11T08:29:42.023 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.023 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.023 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:29:42.024 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.245104+0000", 2024-09-11T08:29:42.024 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.8519999999999998e-06 2024-09-11T08:29:42.024 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.024 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.024 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:29:42.024 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.245219+0000", 2024-09-11T08:29:42.024 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000115035 2024-09-11T08:29:42.024 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.024 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.024 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:29:42.024 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.245228+0000", 2024-09-11T08:29:42.024 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.4630000000000007e-06 2024-09-11T08:29:42.024 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.024 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.025 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:29:42.025 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.173417+0000", 2024-09-11T08:29:42.025 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.92818864300000004 2024-09-11T08:29:42.025 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.025 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.025 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:29:42.025 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.173529+0000", 2024-09-11T08:29:42.025 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000111631 2024-09-11T08:29:42.025 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.025 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.025 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,7", 2024-09-11T08:29:42.025 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.173666+0000", 2024-09-11T08:29:42.025 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00013709 2024-09-11T08:29:42.025 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.025 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.026 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:29:42.026 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.186114+0000", 2024-09-11T08:29:42.026 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.012448232 2024-09-11T08:29:42.026 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.026 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.026 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:29:42.026 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.186812+0000", 2024-09-11T08:29:42.026 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00069784800000000004 2024-09-11T08:29:42.026 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.026 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.026 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:29:42.026 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.189185+0000", 2024-09-11T08:29:42.026 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.002373502 2024-09-11T08:29:42.026 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.026 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.026 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:29:42.027 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.189239+0000", 2024-09-11T08:29:42.027 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.3560999999999998e-05 2024-09-11T08:29:42.027 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.027 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.027 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:29:42.027 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.189277+0000", 2024-09-11T08:29:42.027 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.8133000000000002e-05 2024-09-11T08:29:42.027 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:29:42.027 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:29:42.027 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:29:42.027 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.027 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.027 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:96783 10.9 10:958af421:::benchmark_data_smithi150_158067_object96782:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e329)", 2024-09-11T08:29:42.027 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:22:58.250957+0000", 2024-09-11T08:29:42.027 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 403.75054296500002, 2024-09-11T08:29:42.028 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.93903492600000005, 2024-09-11T08:29:42.028 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:29:42.028 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:29:42.028 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:29:42.028 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:29:42.028 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:29:42.028 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 96783 2024-09-11T08:29:42.028 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.028 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:29:42.028 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.028 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:29:42.028 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.250957+0000", 2024-09-11T08:29:42.028 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:29:42.028 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.028 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.029 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:29:42.029 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.250957+0000", 2024-09-11T08:29:42.029 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:29:42.029 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.029 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.029 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:29:42.029 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.250955+0000", 2024-09-11T08:29:42.029 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:29:42.029 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.029 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.029 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:29:42.029 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.250982+0000", 2024-09-11T08:29:42.029 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.6687999999999999e-05 2024-09-11T08:29:42.029 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.029 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.030 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:29:42.030 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.250996+0000", 2024-09-11T08:29:42.030 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4239e-05 2024-09-11T08:29:42.030 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.030 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.030 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:29:42.030 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.251002+0000", 2024-09-11T08:29:42.030 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.7629999999999997e-06 2024-09-11T08:29:42.030 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.030 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.030 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:29:42.030 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.251052+0000", 2024-09-11T08:29:42.030 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.0389999999999997e-05 2024-09-11T08:29:42.030 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.030 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.031 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:29:42.031 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.251062+0000", 2024-09-11T08:29:42.031 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.7780000000000002e-06 2024-09-11T08:29:42.031 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.031 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.031 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:29:42.031 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.175994+0000", 2024-09-11T08:29:42.031 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.92493149600000002 2024-09-11T08:29:42.031 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.031 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.031 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:29:42.031 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.176096+0000", 2024-09-11T08:29:42.031 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00010289300000000001 2024-09-11T08:29:42.031 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.031 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.031 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,7", 2024-09-11T08:29:42.032 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.176181+0000", 2024-09-11T08:29:42.032 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.4065000000000002e-05 2024-09-11T08:29:42.032 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.032 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.032 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:29:42.032 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.186119+0000", 2024-09-11T08:29:42.032 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0099386590000000007 2024-09-11T08:29:42.032 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.032 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.032 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:29:42.032 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.189331+0000", 2024-09-11T08:29:42.032 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0032119990000000001 2024-09-11T08:29:42.032 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.032 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.032 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:29:42.033 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.189914+0000", 2024-09-11T08:29:42.033 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00058295999999999999 2024-09-11T08:29:42.033 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.033 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.033 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:29:42.034 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.189957+0000", 2024-09-11T08:29:42.034 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.3161999999999998e-05 2024-09-11T08:29:42.034 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.034 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.034 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:29:42.034 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.189992+0000", 2024-09-11T08:29:42.034 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.4501000000000001e-05 2024-09-11T08:29:42.034 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:29:42.034 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:29:42.034 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:29:42.034 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.034 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.034 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:96784 10.9 10:9dd50853:::benchmark_data_smithi150_158067_object96783:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e329)", 2024-09-11T08:29:42.034 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:22:58.251848+0000", 2024-09-11T08:29:42.035 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 403.74965231499999, 2024-09-11T08:29:42.035 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.93834602199999995, 2024-09-11T08:29:42.035 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:29:42.035 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:29:42.035 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:29:42.035 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:29:42.035 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:29:42.035 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 96784 2024-09-11T08:29:42.035 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.035 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:29:42.035 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.035 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:29:42.035 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.251848+0000", 2024-09-11T08:29:42.035 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:29:42.035 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.036 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.036 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:29:42.036 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.251848+0000", 2024-09-11T08:29:42.036 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:29:42.036 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.036 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.036 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:29:42.036 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.251847+0000", 2024-09-11T08:29:42.036 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:29:42.036 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.036 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.036 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:29:42.036 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.251871+0000", 2024-09-11T08:29:42.037 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.4564999999999999e-05 2024-09-11T08:29:42.037 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.037 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.037 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:29:42.037 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.251884+0000", 2024-09-11T08:29:42.037 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2635e-05 2024-09-11T08:29:42.037 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.037 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.037 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:29:42.037 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.251889+0000", 2024-09-11T08:29:42.037 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.6899999999999997e-06 2024-09-11T08:29:42.037 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.037 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.037 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:29:42.038 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.252285+0000", 2024-09-11T08:29:42.038 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00039541899999999999 2024-09-11T08:29:42.038 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.038 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.038 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:29:42.038 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.252294+0000", 2024-09-11T08:29:42.038 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.9390000000000003e-06 2024-09-11T08:29:42.038 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.038 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.038 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:29:42.038 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.177963+0000", 2024-09-11T08:29:42.038 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.92566948999999998 2024-09-11T08:29:42.038 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.039 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.039 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:29:42.039 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.178056+0000", 2024-09-11T08:29:42.039 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.2879000000000003e-05 2024-09-11T08:29:42.039 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.039 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.039 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,7", 2024-09-11T08:29:42.039 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.178173+0000", 2024-09-11T08:29:42.039 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000116852 2024-09-11T08:29:42.039 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.039 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.039 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:29:42.039 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.186763+0000", 2024-09-11T08:29:42.040 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0085899740000000002 2024-09-11T08:29:42.040 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.040 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.040 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:29:42.040 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.189382+0000", 2024-09-11T08:29:42.040 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.002619241 2024-09-11T08:29:42.040 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.040 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.040 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:29:42.040 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.190101+0000", 2024-09-11T08:29:42.040 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00071871100000000002 2024-09-11T08:29:42.040 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.040 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.040 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:29:42.041 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.190152+0000", 2024-09-11T08:29:42.041 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.1384999999999997e-05 2024-09-11T08:29:42.041 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.041 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.041 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:29:42.041 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.190194+0000", 2024-09-11T08:29:42.041 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1291999999999997e-05 2024-09-11T08:29:42.041 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:29:42.041 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:29:42.041 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:29:42.041 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.041 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.041 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:96788 10.9 10:95b153f1:::benchmark_data_smithi150_158067_object96787:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e329)", 2024-09-11T08:29:42.042 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:22:58.253032+0000", 2024-09-11T08:29:42.042 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 403.74846794199999, 2024-09-11T08:29:42.042 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.93740148099999998, 2024-09-11T08:29:42.042 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:29:42.042 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:29:42.042 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:29:42.042 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:29:42.042 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:29:42.042 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 96788 2024-09-11T08:29:42.042 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.042 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:29:42.042 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.042 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:29:42.042 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.253032+0000", 2024-09-11T08:29:42.043 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:29:42.043 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.043 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.043 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:29:42.043 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.253032+0000", 2024-09-11T08:29:42.043 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:29:42.043 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.043 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.043 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:29:42.043 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.253031+0000", 2024-09-11T08:29:42.043 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:29:42.043 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.043 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.043 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:29:42.043 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.253051+0000", 2024-09-11T08:29:42.044 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.9853000000000001e-05 2024-09-11T08:29:42.044 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.044 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.044 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:29:42.044 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.253059+0000", 2024-09-11T08:29:42.044 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.0399999999999993e-06 2024-09-11T08:29:42.044 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.044 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.044 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:29:42.044 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.253064+0000", 2024-09-11T08:29:42.044 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.1309999999999997e-06 2024-09-11T08:29:42.044 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.044 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.045 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:29:42.045 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.253121+0000", 2024-09-11T08:29:42.045 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.7419e-05 2024-09-11T08:29:42.045 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.045 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.045 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:29:42.045 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.253130+0000", 2024-09-11T08:29:42.045 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.2320000000000001e-06 2024-09-11T08:29:42.045 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.045 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.045 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:29:42.045 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.180611+0000", 2024-09-11T08:29:42.045 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.92748144600000004 2024-09-11T08:29:42.045 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.045 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.046 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:29:42.046 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.180735+0000", 2024-09-11T08:29:42.046 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000124148 2024-09-11T08:29:42.046 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.046 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.046 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,7", 2024-09-11T08:29:42.046 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.180878+0000", 2024-09-11T08:29:42.046 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00014298499999999999 2024-09-11T08:29:42.046 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.046 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.046 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:29:42.046 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.186769+0000", 2024-09-11T08:29:42.046 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0058905939999999999 2024-09-11T08:29:42.046 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.046 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.047 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:29:42.047 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.189514+0000", 2024-09-11T08:29:42.047 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0027449240000000001 2024-09-11T08:29:42.047 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.047 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.047 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:29:42.047 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.190296+0000", 2024-09-11T08:29:42.047 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00078191700000000003 2024-09-11T08:29:42.047 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.047 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.047 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:29:42.047 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.190388+0000", 2024-09-11T08:29:42.047 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.2152999999999997e-05 2024-09-11T08:29:42.047 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.047 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.048 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:29:42.048 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.190433+0000", 2024-09-11T08:29:42.048 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.5637999999999999e-05 2024-09-11T08:29:42.048 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:29:42.048 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:29:42.048 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:29:42.048 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.048 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.048 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:96804 10.9 10:9cf2d17b:::benchmark_data_smithi150_158067_object96803:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e329)", 2024-09-11T08:29:42.048 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:22:58.268425+0000", 2024-09-11T08:29:42.048 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 403.73307511199999, 2024-09-11T08:29:42.048 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.93122529300000001, 2024-09-11T08:29:42.048 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:29:42.048 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:29:42.048 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:29:42.049 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:29:42.049 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:29:42.049 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 96804 2024-09-11T08:29:42.049 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.049 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:29:42.049 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.049 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:29:42.049 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.268425+0000", 2024-09-11T08:29:42.049 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:29:42.049 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.049 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.049 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:29:42.049 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.268425+0000", 2024-09-11T08:29:42.049 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:29:42.049 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.050 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.050 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:29:42.050 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.268423+0000", 2024-09-11T08:29:42.050 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:29:42.050 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.050 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.050 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:29:42.050 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.268447+0000", 2024-09-11T08:29:42.050 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.4218000000000001e-05 2024-09-11T08:29:42.050 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.050 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.050 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:29:42.050 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.268464+0000", 2024-09-11T08:29:42.050 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.6532999999999999e-05 2024-09-11T08:29:42.050 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.050 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.051 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:29:42.051 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.268470+0000", 2024-09-11T08:29:42.051 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.5390000000000004e-06 2024-09-11T08:29:42.051 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.051 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.051 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:29:42.051 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.268518+0000", 2024-09-11T08:29:42.051 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.7905000000000003e-05 2024-09-11T08:29:42.051 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.051 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.051 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:29:42.051 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:58.268530+0000", 2024-09-11T08:29:42.051 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1474000000000001e-05 2024-09-11T08:29:42.051 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.052 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.052 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:29:42.052 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.183442+0000", 2024-09-11T08:29:42.052 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.91491235500000001 2024-09-11T08:29:42.052 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.052 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.052 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:29:42.052 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.183576+0000", 2024-09-11T08:29:42.052 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00013377800000000001 2024-09-11T08:29:42.052 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.052 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.052 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,7", 2024-09-11T08:29:42.052 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.183732+0000", 2024-09-11T08:29:42.053 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00015577599999999999 2024-09-11T08:29:42.053 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.053 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.053 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:29:42.053 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.188244+0000", 2024-09-11T08:29:42.053 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0045119039999999997 2024-09-11T08:29:42.053 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.053 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.053 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:29:42.053 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.194947+0000", 2024-09-11T08:29:42.053 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.006702894 2024-09-11T08:29:42.053 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.053 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.053 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:29:42.054 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.199521+0000", 2024-09-11T08:29:42.054 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0045744100000000001 2024-09-11T08:29:42.054 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.054 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.054 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:29:42.054 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.199611+0000", 2024-09-11T08:29:42.054 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.9751000000000002e-05 2024-09-11T08:29:42.054 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.054 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.054 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:29:42.054 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:22:59.199650+0000", 2024-09-11T08:29:42.054 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.9385000000000003e-05 2024-09-11T08:29:42.055 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:29:42.055 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:29:42.055 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:29:42.055 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.055 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.055 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:257624 10.9 10:92df583c:::benchmark_data_smithi150_158067_object127380:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-09-11T08:29:42.055 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:25:41.808160+0000", 2024-09-11T08:29:42.055 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 240.19334035599999, 2024-09-11T08:29:42.055 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.94614705899999996, 2024-09-11T08:29:42.055 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:29:42.055 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:29:42.055 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:29:42.055 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:29:42.055 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:29:42.056 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 257624 2024-09-11T08:29:42.056 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.056 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:29:42.056 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.056 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:29:42.056 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.808160+0000", 2024-09-11T08:29:42.056 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:29:42.056 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.056 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.056 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:29:42.056 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.808160+0000", 2024-09-11T08:29:42.056 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:29:42.056 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.056 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.056 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:29:42.057 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.808159+0000", 2024-09-11T08:29:42.057 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:29:42.057 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.057 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.057 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:29:42.057 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.808162+0000", 2024-09-11T08:29:42.057 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.886e-06 2024-09-11T08:29:42.057 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.057 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.057 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:29:42.057 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.808283+0000", 2024-09-11T08:29:42.057 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000121332 2024-09-11T08:29:42.057 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.057 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.057 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:29:42.057 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.808291+0000", 2024-09-11T08:29:42.058 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.5340000000000002e-06 2024-09-11T08:29:42.058 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.058 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.058 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:29:42.058 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.808344+0000", 2024-09-11T08:29:42.058 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.3319999999999998e-05 2024-09-11T08:29:42.058 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.058 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.058 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:29:42.058 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.808353+0000", 2024-09-11T08:29:42.058 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.585e-06 2024-09-11T08:29:42.058 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.058 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.058 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:29:42.058 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.746194+0000", 2024-09-11T08:29:42.059 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.93784158100000004 2024-09-11T08:29:42.059 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.059 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.059 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:29:42.059 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.746308+0000", 2024-09-11T08:29:42.059 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00011389 2024-09-11T08:29:42.059 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.059 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.059 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,7", 2024-09-11T08:29:42.059 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.746439+0000", 2024-09-11T08:29:42.059 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00013035999999999999 2024-09-11T08:29:42.059 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.059 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.060 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:29:42.060 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.749375+0000", 2024-09-11T08:29:42.060 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0029365369999999999 2024-09-11T08:29:42.060 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.060 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.060 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:29:42.060 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.754199+0000", 2024-09-11T08:29:42.060 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0048243000000000001 2024-09-11T08:29:42.060 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.060 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.060 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:29:42.060 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.754240+0000", 2024-09-11T08:29:42.060 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.0219000000000002e-05 2024-09-11T08:29:42.060 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.061 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.061 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:29:42.061 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.754269+0000", 2024-09-11T08:29:42.061 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.9298e-05 2024-09-11T08:29:42.061 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.061 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.061 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:29:42.061 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.754307+0000", 2024-09-11T08:29:42.061 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.7611999999999998e-05 2024-09-11T08:29:42.061 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:29:42.061 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:29:42.061 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:29:42.061 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.062 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.062 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:257630 10.9 10:921d42f5:::benchmark_data_smithi150_158067_object127386:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-09-11T08:29:42.062 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:25:41.816616+0000", 2024-09-11T08:29:42.062 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 240.184883461, 2024-09-11T08:29:42.062 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.94358571800000002, 2024-09-11T08:29:42.062 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:29:42.062 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:29:42.062 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:29:42.062 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:29:42.062 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:29:42.062 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 257630 2024-09-11T08:29:42.062 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.062 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:29:42.062 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.063 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:29:42.063 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.816616+0000", 2024-09-11T08:29:42.063 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:29:42.063 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.063 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.063 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:29:42.063 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.816616+0000", 2024-09-11T08:29:42.063 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:29:42.063 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.063 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.063 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:29:42.063 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.816615+0000", 2024-09-11T08:29:42.063 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:29:42.063 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.063 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.063 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:29:42.064 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.816631+0000", 2024-09-11T08:29:42.064 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.6232999999999999e-05 2024-09-11T08:29:42.064 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.064 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.064 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:29:42.064 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.816643+0000", 2024-09-11T08:29:42.064 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1464999999999999e-05 2024-09-11T08:29:42.064 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.064 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.064 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:29:42.064 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.816649+0000", 2024-09-11T08:29:42.064 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.0909999999999998e-06 2024-09-11T08:29:42.064 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.064 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.065 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:29:42.065 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.816722+0000", 2024-09-11T08:29:42.065 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.3164999999999995e-05 2024-09-11T08:29:42.065 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.065 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.065 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:29:42.065 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.816731+0000", 2024-09-11T08:29:42.065 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.7690000000000002e-06 2024-09-11T08:29:42.065 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.065 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.065 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:29:42.065 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.749413+0000", 2024-09-11T08:29:42.065 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.93268246099999996 2024-09-11T08:29:42.065 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.065 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.065 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:29:42.066 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.749749+0000", 2024-09-11T08:29:42.066 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00033593300000000001 2024-09-11T08:29:42.066 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.066 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.066 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,7", 2024-09-11T08:29:42.066 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.749871+0000", 2024-09-11T08:29:42.066 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012230100000000001 2024-09-11T08:29:42.066 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.066 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.066 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:29:42.066 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.754347+0000", 2024-09-11T08:29:42.066 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0044754699999999996 2024-09-11T08:29:42.066 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.066 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.066 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:29:42.067 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.754372+0000", 2024-09-11T08:29:42.067 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.5201000000000001e-05 2024-09-11T08:29:42.067 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.067 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.067 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:29:42.067 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.760122+0000", 2024-09-11T08:29:42.067 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0057503980000000003 2024-09-11T08:29:42.067 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.067 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.067 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:29:42.067 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.760164+0000", 2024-09-11T08:29:42.067 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1284e-05 2024-09-11T08:29:42.067 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.067 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.067 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:29:42.067 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.760202+0000", 2024-09-11T08:29:42.068 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.8415999999999999e-05 2024-09-11T08:29:42.068 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:29:42.068 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:29:42.068 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:29:42.068 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.068 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.068 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:257632 10.9 10:91c6bcfb:::benchmark_data_smithi150_158067_object127388:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-09-11T08:29:42.068 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:25:41.816663+0000", 2024-09-11T08:29:42.068 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 240.184837071, 2024-09-11T08:29:42.068 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.95093924299999999, 2024-09-11T08:29:42.068 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:29:42.068 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:29:42.068 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:29:42.068 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:29:42.068 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:29:42.069 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 257632 2024-09-11T08:29:42.069 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.069 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:29:42.069 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.069 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:29:42.069 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.816663+0000", 2024-09-11T08:29:42.069 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:29:42.069 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.069 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.069 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:29:42.069 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.816663+0000", 2024-09-11T08:29:42.069 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:29:42.069 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.069 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.069 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:29:42.070 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.816662+0000", 2024-09-11T08:29:42.070 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:29:42.070 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.070 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.070 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:29:42.070 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.816666+0000", 2024-09-11T08:29:42.070 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.2899999999999998e-06 2024-09-11T08:29:42.070 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.070 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.070 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:29:42.070 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.816671+0000", 2024-09-11T08:29:42.070 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.5300000000000004e-06 2024-09-11T08:29:42.070 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.070 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.070 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:29:42.071 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.816676+0000", 2024-09-11T08:29:42.071 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.0860000000000001e-06 2024-09-11T08:29:42.071 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.071 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.071 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:29:42.071 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.816749+0000", 2024-09-11T08:29:42.071 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.2861000000000006e-05 2024-09-11T08:29:42.071 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.072 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.072 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:29:42.072 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.816755+0000", 2024-09-11T08:29:42.072 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.5319999999999997e-06 2024-09-11T08:29:42.072 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.072 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.072 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:29:42.072 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.752363+0000", 2024-09-11T08:29:42.072 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.93560816999999996 2024-09-11T08:29:42.072 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.072 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.072 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:29:42.073 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.752431+0000", 2024-09-11T08:29:42.073 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.8224000000000003e-05 2024-09-11T08:29:42.073 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.073 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.073 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,7", 2024-09-11T08:29:42.073 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.752493+0000", 2024-09-11T08:29:42.073 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.2367999999999996e-05 2024-09-11T08:29:42.073 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.073 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.073 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:29:42.073 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:42.074 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.760021+0000", 2024-09-11T08:29:42.074 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.007527527 2024-09-11T08:29:42.075 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.075 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.075 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:29:42.075 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.760074+0000", 2024-09-11T08:29:42.075 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.3037000000000001e-05 2024-09-11T08:29:42.075 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.075 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.075 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:29:42.075 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.767528+0000", 2024-09-11T08:29:42.076 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0074539940000000002 2024-09-11T08:29:42.076 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.076 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.076 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:29:42.076 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.767559+0000", 2024-09-11T08:29:42.076 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.1288000000000003e-05 2024-09-11T08:29:42.076 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.076 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.076 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:29:42.076 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.767602+0000", 2024-09-11T08:29:42.076 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.2808000000000001e-05 2024-09-11T08:29:42.076 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:29:42.076 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:29:42.076 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:29:42.077 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.077 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.077 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:257647 10.9 10:91ee9aaf:::benchmark_data_smithi150_158067_object127403:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-09-11T08:29:42.077 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:25:41.831664+0000", 2024-09-11T08:29:42.077 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 240.169835525, 2024-09-11T08:29:42.077 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.93606025599999998, 2024-09-11T08:29:42.077 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:29:42.077 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:29:42.077 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:29:42.077 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:29:42.077 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:29:42.077 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 257647 2024-09-11T08:29:42.077 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.078 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:29:42.078 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.078 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:29:42.078 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.831664+0000", 2024-09-11T08:29:42.078 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:29:42.078 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.078 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.078 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:29:42.078 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.831664+0000", 2024-09-11T08:29:42.078 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:29:42.078 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.078 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.078 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:29:42.079 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.831663+0000", 2024-09-11T08:29:42.079 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:29:42.079 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.079 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.079 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:29:42.079 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.831669+0000", 2024-09-11T08:29:42.079 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.2709999999999998e-06 2024-09-11T08:29:42.079 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.079 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.079 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:29:42.079 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.831679+0000", 2024-09-11T08:29:42.079 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.9990000000000003e-06 2024-09-11T08:29:42.079 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.080 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.080 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:29:42.080 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.831683+0000", 2024-09-11T08:29:42.080 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.3030000000000001e-06 2024-09-11T08:29:42.080 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.080 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.080 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:29:42.080 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.831724+0000", 2024-09-11T08:29:42.080 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.0228000000000002e-05 2024-09-11T08:29:42.080 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.080 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.080 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:29:42.080 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.831731+0000", 2024-09-11T08:29:42.081 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.939e-06 2024-09-11T08:29:42.081 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.081 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.081 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:29:42.081 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.754402+0000", 2024-09-11T08:29:42.081 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.92267161099999995 2024-09-11T08:29:42.081 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.081 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.081 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:29:42.081 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.754503+0000", 2024-09-11T08:29:42.081 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000100719 2024-09-11T08:29:42.081 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.081 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.082 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,7", 2024-09-11T08:29:42.082 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.754618+0000", 2024-09-11T08:29:42.082 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00011491499999999999 2024-09-11T08:29:42.082 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.082 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.082 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:29:42.082 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.760222+0000", 2024-09-11T08:29:42.082 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0056036810000000001 2024-09-11T08:29:42.082 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.082 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.082 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:29:42.082 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.760242+0000", 2024-09-11T08:29:42.082 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.0888000000000001e-05 2024-09-11T08:29:42.082 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.083 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.083 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:29:42.083 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.767646+0000", 2024-09-11T08:29:42.083 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0074040030000000001 2024-09-11T08:29:42.083 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.083 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.083 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:29:42.083 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.767691+0000", 2024-09-11T08:29:42.083 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.4127e-05 2024-09-11T08:29:42.083 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.083 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.083 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:29:42.083 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.767725+0000", 2024-09-11T08:29:42.083 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.4060999999999998e-05 2024-09-11T08:29:42.083 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:29:42.083 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:29:42.084 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:29:42.084 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.084 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.084 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:257660 10.9 10:90fd90c6:::benchmark_data_smithi150_158067_object127416:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-09-11T08:29:42.084 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:25:41.846903+0000", 2024-09-11T08:29:42.084 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 240.15459685499999, 2024-09-11T08:29:42.084 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.92092837400000005, 2024-09-11T08:29:42.084 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:29:42.084 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:29:42.084 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:29:42.084 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:29:42.084 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:29:42.084 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 257660 2024-09-11T08:29:42.084 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.084 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:29:42.085 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.085 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:29:42.085 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.846903+0000", 2024-09-11T08:29:42.085 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:29:42.085 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.085 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.085 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:29:42.085 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.846903+0000", 2024-09-11T08:29:42.085 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:29:42.085 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.085 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.085 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:29:42.085 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.846902+0000", 2024-09-11T08:29:42.085 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:29:42.085 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.085 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.086 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:29:42.086 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.846909+0000", 2024-09-11T08:29:42.086 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.7600000000000002e-06 2024-09-11T08:29:42.086 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.086 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.086 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:29:42.086 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.846922+0000", 2024-09-11T08:29:42.086 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3118999999999999e-05 2024-09-11T08:29:42.086 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.086 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.086 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:29:42.086 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.846929+0000", 2024-09-11T08:29:42.086 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.5289999999999997e-06 2024-09-11T08:29:42.086 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.086 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.086 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:29:42.087 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.846980+0000", 2024-09-11T08:29:42.087 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.0834000000000002e-05 2024-09-11T08:29:42.087 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.087 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.087 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:29:42.087 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.846989+0000", 2024-09-11T08:29:42.087 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.9819999999999997e-06 2024-09-11T08:29:42.087 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.087 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.087 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:29:42.087 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.757407+0000", 2024-09-11T08:29:42.087 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.91041787399999996 2024-09-11T08:29:42.087 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.087 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.088 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:29:42.088 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.757487+0000", 2024-09-11T08:29:42.088 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.0476999999999995e-05 2024-09-11T08:29:42.088 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.088 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.088 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,7", 2024-09-11T08:29:42.088 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.757564+0000", 2024-09-11T08:29:42.088 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.6508999999999995e-05 2024-09-11T08:29:42.088 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.088 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.088 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:29:42.088 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.760247+0000", 2024-09-11T08:29:42.088 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.002683586 2024-09-11T08:29:42.089 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.089 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.089 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:29:42.089 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.767438+0000", 2024-09-11T08:29:42.089 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0071904480000000003 2024-09-11T08:29:42.089 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.089 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.089 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:29:42.089 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.767761+0000", 2024-09-11T08:29:42.089 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0003233 2024-09-11T08:29:42.089 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.089 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.089 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:29:42.089 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.767802+0000", 2024-09-11T08:29:42.089 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1099000000000001e-05 2024-09-11T08:29:42.089 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.090 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.090 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:29:42.090 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.767831+0000", 2024-09-11T08:29:42.090 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.9352999999999998e-05 2024-09-11T08:29:42.090 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:29:42.090 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:29:42.090 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:29:42.090 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.090 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.090 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:257667 10.9 10:919671da:::benchmark_data_smithi150_158067_object127423:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-09-11T08:29:42.090 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:25:41.852473+0000", 2024-09-11T08:29:42.090 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 240.14902708400001, 2024-09-11T08:29:42.090 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.91546256800000003, 2024-09-11T08:29:42.090 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:29:42.090 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:29:42.091 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:29:42.091 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:29:42.091 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:29:42.091 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 257667 2024-09-11T08:29:42.091 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.091 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:29:42.091 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.091 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:29:42.091 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.852473+0000", 2024-09-11T08:29:42.091 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:29:42.091 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.091 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.091 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:29:42.092 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.852473+0000", 2024-09-11T08:29:42.092 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:29:42.092 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.092 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.092 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:29:42.092 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.852471+0000", 2024-09-11T08:29:42.092 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:29:42.092 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.092 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.092 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:29:42.092 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.852479+0000", 2024-09-11T08:29:42.092 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.0239999999999997e-06 2024-09-11T08:29:42.092 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.093 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.093 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:29:42.093 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.852494+0000", 2024-09-11T08:29:42.093 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4826e-05 2024-09-11T08:29:42.093 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.093 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.093 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:29:42.093 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.852501+0000", 2024-09-11T08:29:42.093 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.3640000000000001e-06 2024-09-11T08:29:42.093 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.094 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.094 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:29:42.094 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.852556+0000", 2024-09-11T08:29:42.094 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.5037999999999997e-05 2024-09-11T08:29:42.094 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.094 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.094 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:29:42.094 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.852567+0000", 2024-09-11T08:29:42.094 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.024e-05 2024-09-11T08:29:42.094 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.094 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.094 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:29:42.095 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.760268+0000", 2024-09-11T08:29:42.095 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.90770139999999999 2024-09-11T08:29:42.095 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.095 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.095 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:29:42.095 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.760368+0000", 2024-09-11T08:29:42.095 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.9957999999999994e-05 2024-09-11T08:29:42.095 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.095 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.095 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,7", 2024-09-11T08:29:42.095 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.760476+0000", 2024-09-11T08:29:42.095 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000108194 2024-09-11T08:29:42.095 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.095 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.096 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:29:42.096 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.767480+0000", 2024-09-11T08:29:42.096 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.007003713 2024-09-11T08:29:42.096 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.096 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.096 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:29:42.096 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.767489+0000", 2024-09-11T08:29:42.096 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.2210000000000003e-06 2024-09-11T08:29:42.096 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.096 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.096 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:29:42.096 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.767868+0000", 2024-09-11T08:29:42.096 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00037919899999999999 2024-09-11T08:29:42.096 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.097 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.097 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:29:42.097 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.767904+0000", 2024-09-11T08:29:42.097 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.5939999999999998e-05 2024-09-11T08:29:42.097 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.097 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.097 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:29:42.097 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.767935+0000", 2024-09-11T08:29:42.097 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.1050000000000003e-05 2024-09-11T08:29:42.097 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:29:42.098 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:29:42.098 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:29:42.098 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.098 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.098 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:257682 10.9 10:98bd2a93:::benchmark_data_smithi150_158067_object127438:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-09-11T08:29:42.098 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:25:41.863234+0000", 2024-09-11T08:29:42.098 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 240.13826589799999, 2024-09-11T08:29:42.098 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.91371584900000002, 2024-09-11T08:29:42.098 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:29:42.098 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:29:42.098 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:29:42.098 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:29:42.098 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:29:42.098 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 257682 2024-09-11T08:29:42.099 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.099 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:29:42.099 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.099 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:29:42.099 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.863234+0000", 2024-09-11T08:29:42.099 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:29:42.099 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.099 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.099 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:29:42.099 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.863234+0000", 2024-09-11T08:29:42.099 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:29:42.099 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.099 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.100 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:29:42.100 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.863233+0000", 2024-09-11T08:29:42.100 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:29:42.100 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.100 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.100 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:29:42.100 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.863238+0000", 2024-09-11T08:29:42.100 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.6199999999999998e-06 2024-09-11T08:29:42.100 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.100 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.100 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:29:42.100 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.863250+0000", 2024-09-11T08:29:42.100 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2194e-05 2024-09-11T08:29:42.100 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.101 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.101 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:29:42.101 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.863255+0000", 2024-09-11T08:29:42.101 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.2290000000000002e-06 2024-09-11T08:29:42.101 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.101 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.101 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:29:42.101 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.863308+0000", 2024-09-11T08:29:42.101 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.3270000000000003e-05 2024-09-11T08:29:42.101 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.101 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.101 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:29:42.101 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.863316+0000", 2024-09-11T08:29:42.102 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.8329999999999994e-06 2024-09-11T08:29:42.102 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.102 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.102 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:29:42.102 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.764311+0000", 2024-09-11T08:29:42.102 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.90099498099999997 2024-09-11T08:29:42.102 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.102 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.102 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:29:42.102 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.764396+0000", 2024-09-11T08:29:42.102 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.4468000000000005e-05 2024-09-11T08:29:42.102 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.103 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.103 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,7", 2024-09-11T08:29:42.103 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.764493+0000", 2024-09-11T08:29:42.103 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.7134999999999997e-05 2024-09-11T08:29:42.103 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.103 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.103 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:29:42.103 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.767838+0000", 2024-09-11T08:29:42.103 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.003344914 2024-09-11T08:29:42.103 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.103 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.103 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:29:42.103 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.776429+0000", 2024-09-11T08:29:42.103 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0085910139999999992 2024-09-11T08:29:42.104 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.104 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.104 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:29:42.104 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.776513+0000", 2024-09-11T08:29:42.104 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.4544999999999998e-05 2024-09-11T08:29:42.104 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.104 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.104 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:29:42.104 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.776557+0000", 2024-09-11T08:29:42.104 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.3744e-05 2024-09-11T08:29:42.104 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.104 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.104 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:29:42.105 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.776950+0000", 2024-09-11T08:29:42.105 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000392731 2024-09-11T08:29:42.105 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:29:42.105 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:29:42.105 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:29:42.105 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.105 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.105 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:257707 10.9 10:945d1c69:::benchmark_data_smithi150_158067_object127463:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-09-11T08:29:42.105 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:25:41.889556+0000", 2024-09-11T08:29:42.105 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 240.111943532, 2024-09-11T08:29:42.105 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.90106649900000002, 2024-09-11T08:29:42.105 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:29:42.106 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:29:42.106 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:29:42.106 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:29:42.106 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:29:42.106 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 257707 2024-09-11T08:29:42.106 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.106 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:29:42.106 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.106 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:29:42.106 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.889556+0000", 2024-09-11T08:29:42.106 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:29:42.106 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.107 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.107 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:29:42.107 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.889556+0000", 2024-09-11T08:29:42.107 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:29:42.107 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.107 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.107 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:29:42.107 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.889555+0000", 2024-09-11T08:29:42.107 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:29:42.107 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.107 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.107 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:29:42.107 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.889562+0000", 2024-09-11T08:29:42.107 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.8619999999999997e-06 2024-09-11T08:29:42.108 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.108 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.108 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:29:42.108 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.889575+0000", 2024-09-11T08:29:42.108 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3456e-05 2024-09-11T08:29:42.108 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.108 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.108 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:29:42.108 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.889580+0000", 2024-09-11T08:29:42.108 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.2860000000000001e-06 2024-09-11T08:29:42.108 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.108 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.109 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:29:42.109 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.889629+0000", 2024-09-11T08:29:42.109 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.977e-05 2024-09-11T08:29:42.109 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.109 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.109 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:29:42.109 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.889639+0000", 2024-09-11T08:29:42.109 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.944e-06 2024-09-11T08:29:42.109 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.109 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.109 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:29:42.109 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.767958+0000", 2024-09-11T08:29:42.109 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.87831847799999996 2024-09-11T08:29:42.109 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.110 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.110 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:29:42.110 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.768059+0000", 2024-09-11T08:29:42.110 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00010106 2024-09-11T08:29:42.110 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.110 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.110 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,7", 2024-09-11T08:29:42.110 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.768172+0000", 2024-09-11T08:29:42.110 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000113616 2024-09-11T08:29:42.110 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.110 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.110 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:29:42.110 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.776472+0000", 2024-09-11T08:29:42.111 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0083000000000000001 2024-09-11T08:29:42.111 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.111 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.111 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:29:42.111 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.776481+0000", 2024-09-11T08:29:42.111 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.7280000000000001e-06 2024-09-11T08:29:42.111 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.111 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.111 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:29:42.111 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.790537+0000", 2024-09-11T08:29:42.111 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.014056325999999999 2024-09-11T08:29:42.111 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.111 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.112 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:29:42.112 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.790582+0000", 2024-09-11T08:29:42.112 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.4231999999999999e-05 2024-09-11T08:29:42.112 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.112 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.112 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:29:42.112 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.790623+0000", 2024-09-11T08:29:42.112 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1119999999999999e-05 2024-09-11T08:29:42.112 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:29:42.112 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:29:42.112 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:29:42.112 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.112 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.113 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.15636.0:257736 10.9 10:981ba157:::benchmark_data_smithi150_158067_object127492:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-09-11T08:29:42.113 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:25:41.920790+0000", 2024-09-11T08:29:42.113 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 240.080710165, 2024-09-11T08:29:42.113 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.86997259400000004, 2024-09-11T08:29:42.113 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:29:42.113 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:29:42.113 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:29:42.113 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.15636", 2024-09-11T08:29:42.113 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1349228941", 2024-09-11T08:29:42.113 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 257736 2024-09-11T08:29:42.113 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.113 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:29:42.113 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.114 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:29:42.114 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.920790+0000", 2024-09-11T08:29:42.114 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:29:42.114 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.114 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.114 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:29:42.114 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.920790+0000", 2024-09-11T08:29:42.114 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:29:42.114 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.114 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.114 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:29:42.114 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.920788+0000", 2024-09-11T08:29:42.115 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:29:42.115 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.115 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.115 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:29:42.115 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.920796+0000", 2024-09-11T08:29:42.115 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.9859999999999994e-06 2024-09-11T08:29:42.115 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.115 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.115 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:29:42.115 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.920811+0000", 2024-09-11T08:29:42.115 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4701e-05 2024-09-11T08:29:42.115 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.115 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.115 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:29:42.116 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.920818+0000", 2024-09-11T08:29:42.116 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.4289999999999998e-06 2024-09-11T08:29:42.116 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.116 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.116 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:29:42.116 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.920870+0000", 2024-09-11T08:29:42.116 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.2927000000000003e-05 2024-09-11T08:29:42.116 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.116 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.116 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for peered", 2024-09-11T08:29:42.116 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:41.920882+0000", 2024-09-11T08:29:42.117 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1199999999999999e-05 2024-09-11T08:29:42.117 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.117 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.117 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:29:42.117 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.772565+0000", 2024-09-11T08:29:42.117 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.85168382799999998 2024-09-11T08:29:42.117 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.117 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.117 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:29:42.117 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.772688+0000", 2024-09-11T08:29:42.117 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000122657 2024-09-11T08:29:42.117 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.117 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.118 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 5,7", 2024-09-11T08:29:42.118 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.772824+0000", 2024-09-11T08:29:42.119 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00013596699999999999 2024-09-11T08:29:42.119 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.119 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.119 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:29:42.119 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.777003+0000", 2024-09-11T08:29:42.119 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0041784910000000003 2024-09-11T08:29:42.119 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.119 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.119 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:29:42.119 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.777008+0000", 2024-09-11T08:29:42.119 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.5840000000000002e-06 2024-09-11T08:29:42.119 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.119 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.120 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:29:42.120 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.790695+0000", 2024-09-11T08:29:42.120 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.013687220999999999 2024-09-11T08:29:42.120 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.120 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.120 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:29:42.120 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.790731+0000", 2024-09-11T08:29:42.120 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.5442999999999999e-05 2024-09-11T08:29:42.120 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.120 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.120 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:29:42.120 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:25:42.790762+0000", 2024-09-11T08:29:42.120 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.1469e-05 2024-09-11T08:29:42.121 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:29:42.121 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:29:42.121 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:29:42.121 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.121 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.121 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.25799.0:227057 11.2 11:4dd55f2f:::benchmark_data_smithi150_187410_object101008:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e436)", 2024-09-11T08:29:42.121 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:29:40.248421+0000", 2024-09-11T08:29:42.121 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 1.7530792509999999, 2024-09-11T08:29:42.121 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.30159206199999999, 2024-09-11T08:29:42.121 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:29:42.121 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:29:42.121 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:29:42.121 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.25799", 2024-09-11T08:29:42.122 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1403604046", 2024-09-11T08:29:42.122 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 227057 2024-09-11T08:29:42.122 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.122 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:29:42.122 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.122 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:29:42.122 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.248421+0000", 2024-09-11T08:29:42.122 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:29:42.122 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.122 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.122 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:29:42.122 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.248421+0000", 2024-09-11T08:29:42.123 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:29:42.123 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.123 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.123 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:29:42.123 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.248419+0000", 2024-09-11T08:29:42.123 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:29:42.123 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.123 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.123 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:29:42.123 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.248426+0000", 2024-09-11T08:29:42.123 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.8419999999999999e-06 2024-09-11T08:29:42.123 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.123 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.123 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:29:42.124 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.248437+0000", 2024-09-11T08:29:42.124 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1011e-05 2024-09-11T08:29:42.124 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.124 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.124 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:29:42.124 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.248443+0000", 2024-09-11T08:29:42.124 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.7599999999999999e-06 2024-09-11T08:29:42.124 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.124 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.124 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:29:42.124 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.250800+0000", 2024-09-11T08:29:42.124 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0023571410000000001 2024-09-11T08:29:42.124 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.125 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.125 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:29:42.125 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.250957+0000", 2024-09-11T08:29:42.125 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00015653999999999999 2024-09-11T08:29:42.125 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.125 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.125 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 0,3", 2024-09-11T08:29:42.125 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.251085+0000", 2024-09-11T08:29:42.125 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012861 2024-09-11T08:29:42.125 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.125 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.125 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:29:42.125 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.254927+0000", 2024-09-11T08:29:42.125 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0038415469999999998 2024-09-11T08:29:42.125 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.126 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.126 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:29:42.126 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.258969+0000", 2024-09-11T08:29:42.126 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0040422560000000001 2024-09-11T08:29:42.126 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.126 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.126 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:29:42.126 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.549895+0000", 2024-09-11T08:29:42.126 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.290926095 2024-09-11T08:29:42.126 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.126 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.126 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:29:42.126 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.549949+0000", 2024-09-11T08:29:42.126 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.3563000000000002e-05 2024-09-11T08:29:42.127 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.127 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.127 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:29:42.127 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.550013+0000", 2024-09-11T08:29:42.127 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.4095000000000002e-05 2024-09-11T08:29:42.127 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:29:42.127 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:29:42.127 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:29:42.127 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.127 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.127 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.25799.0:227082 11.6 e436/382 11:6f7a5427:::benchmark_data_smithi150_187410_object101033:head v 436'14265, mlcod=436'14263)", 2024-09-11T08:29:42.127 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:29:40.336741+0000", 2024-09-11T08:29:42.127 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 1.664758806, 2024-09-11T08:29:42.127 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.315156194, 2024-09-11T08:29:42.127 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:29:42.127 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:29:42.128 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:29:42.128 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.128 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:29:42.128 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.336741+0000", 2024-09-11T08:29:42.128 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:29:42.128 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.128 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.128 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:29:42.128 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.336741+0000", 2024-09-11T08:29:42.128 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:29:42.128 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.128 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.128 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:29:42.128 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.336740+0000", 2024-09-11T08:29:42.129 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:29:42.129 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.129 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.129 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:29:42.129 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.336747+0000", 2024-09-11T08:29:42.129 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.9099999999999999e-06 2024-09-11T08:29:42.129 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.129 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.129 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:29:42.129 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.336763+0000", 2024-09-11T08:29:42.129 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.6025e-05 2024-09-11T08:29:42.129 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.129 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.129 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:29:42.129 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.336770+0000", 2024-09-11T08:29:42.129 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.6469999999999998e-06 2024-09-11T08:29:42.130 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.130 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.130 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:29:42.130 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.336826+0000", 2024-09-11T08:29:42.130 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.5047999999999999e-05 2024-09-11T08:29:42.130 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.130 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.130 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:29:42.130 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.336847+0000", 2024-09-11T08:29:42.130 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.1925000000000001e-05 2024-09-11T08:29:42.130 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.130 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.130 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:29:42.130 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.651811+0000", 2024-09-11T08:29:42.131 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.31496361 2024-09-11T08:29:42.131 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.131 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.131 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:29:42.131 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.651897+0000", 2024-09-11T08:29:42.131 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.6211000000000005e-05 2024-09-11T08:29:42.131 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:29:42.131 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:29:42.131 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:29:42.131 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.131 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.131 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.25799.0:227099 11.6 e436/382 11:60ef7dfe:::benchmark_data_smithi150_187410_object101050:head v 436'14266, mlcod=436'14263)", 2024-09-11T08:29:42.131 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:29:40.338885+0000", 2024-09-11T08:29:42.131 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 1.6626152350000001, 2024-09-11T08:29:42.131 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.31540736600000002, 2024-09-11T08:29:42.132 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:29:42.132 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:29:42.132 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:29:42.132 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.132 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:29:42.132 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.338885+0000", 2024-09-11T08:29:42.132 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:29:42.132 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.132 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.132 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:29:42.132 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.338885+0000", 2024-09-11T08:29:42.132 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:29:42.132 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.132 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.133 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:29:42.133 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.338884+0000", 2024-09-11T08:29:42.133 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:29:42.133 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.133 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.133 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:29:42.133 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.338892+0000", 2024-09-11T08:29:42.133 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.7229999999999993e-06 2024-09-11T08:29:42.133 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.133 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.133 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:29:42.133 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.338904+0000", 2024-09-11T08:29:42.133 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1742999999999999e-05 2024-09-11T08:29:42.133 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.133 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.134 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:29:42.134 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.338910+0000", 2024-09-11T08:29:42.134 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.8340000000000004e-06 2024-09-11T08:29:42.134 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.134 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.134 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:29:42.134 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.549912+0000", 2024-09-11T08:29:42.134 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.21100190599999999 2024-09-11T08:29:42.134 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.134 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.134 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:29:42.134 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.550212+0000", 2024-09-11T08:29:42.134 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00030025000000000001 2024-09-11T08:29:42.134 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.135 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.135 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:29:42.135 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.651992+0000", 2024-09-11T08:29:42.135 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.10178023 2024-09-11T08:29:42.135 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.135 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.135 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:29:42.135 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.654292+0000", 2024-09-11T08:29:42.135 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0022995899999999998 2024-09-11T08:29:42.135 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:29:42.135 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:29:42.135 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:29:42.135 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.135 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.136 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.25799.0:227108 11.2 11:41d1b75f:::benchmark_data_smithi150_187410_object101059:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e437)", 2024-09-11T08:29:42.136 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:29:40.348261+0000", 2024-09-11T08:29:42.136 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 1.65323887, 2024-09-11T08:29:42.136 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.317956352, 2024-09-11T08:29:42.136 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:29:42.136 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:29:42.136 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:29:42.136 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.25799", 2024-09-11T08:29:42.136 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1403604046", 2024-09-11T08:29:42.136 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 227108 2024-09-11T08:29:42.136 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.136 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:29:42.136 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.136 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:29:42.136 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.348261+0000", 2024-09-11T08:29:42.137 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:29:42.137 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.137 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.137 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:29:42.137 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.348261+0000", 2024-09-11T08:29:42.137 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:29:42.137 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.137 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.137 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:29:42.137 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.348259+0000", 2024-09-11T08:29:42.137 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999976 2024-09-11T08:29:42.137 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.137 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.137 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:29:42.138 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.348268+0000", 2024-09-11T08:29:42.138 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.2450000000000004e-06 2024-09-11T08:29:42.138 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.138 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.138 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:29:42.138 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.549391+0000", 2024-09-11T08:29:42.138 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.201123722 2024-09-11T08:29:42.138 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.138 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.138 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:29:42.138 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.549403+0000", 2024-09-11T08:29:42.138 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1252e-05 2024-09-11T08:29:42.138 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.139 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.139 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:29:42.139 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.550450+0000", 2024-09-11T08:29:42.139 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0010470130000000001 2024-09-11T08:29:42.139 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.139 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.139 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:29:42.139 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.651868+0000", 2024-09-11T08:29:42.139 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.101418469 2024-09-11T08:29:42.139 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.139 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.139 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 0,3", 2024-09-11T08:29:42.139 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.652026+0000", 2024-09-11T08:29:42.140 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00015796000000000001 2024-09-11T08:29:42.140 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.140 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.140 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:29:42.140 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.658548+0000", 2024-09-11T08:29:42.140 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0065223110000000003 2024-09-11T08:29:42.140 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.140 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.140 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:29:42.140 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.666045+0000", 2024-09-11T08:29:42.140 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0074962409999999998 2024-09-11T08:29:42.140 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.140 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.140 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:29:42.141 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.666080+0000", 2024-09-11T08:29:42.141 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.5136999999999999e-05 2024-09-11T08:29:42.141 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.141 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.141 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:29:42.141 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.666176+0000", 2024-09-11T08:29:42.141 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.5779999999999994e-05 2024-09-11T08:29:42.141 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:29:42.141 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:29:42.141 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:29:42.141 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.666217+0000", 2024-09-11T08:29:42.141 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1702999999999997e-05 2024-09-11T08:29:42.141 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:29:42.141 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:29:42.142 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:29:42.142 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:29:42.142 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:29:42.142 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-11T08:29:42.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:42.155 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:42.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:42.256 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:29:42.302 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:42.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:42.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:29:42.403 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:42.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:42.442 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:42.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:42.540 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:42 smithi150 ceph-mon[29276]: pgmap v1014: 148 pgs: 148 active+clean; 1.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 645 B/s rd, 0 B/s wr, 1.28k op/s 2024-09-11T08:29:42.540 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:42 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:29:42.540 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:42 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:29:42.541 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:42 smithi150 ceph-mon[39612]: pgmap v1014: 148 pgs: 148 active+clean; 1.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 645 B/s rd, 0 B/s wr, 1.28k op/s 2024-09-11T08:29:42.541 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:42 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:29:42.541 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:42 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:29:42.543 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:42.572 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:42 smithi181 ceph-mon[32701]: pgmap v1014: 148 pgs: 148 active+clean; 1.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 645 B/s rd, 0 B/s wr, 1.28k op/s 2024-09-11T08:29:42.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:42 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:29:42.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:42 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:29:42.600 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:42.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:42.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:42.702 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:42.730 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:42.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:42.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:42.831 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:29:42.859 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:42.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:42.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:29:42.961 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:29:42.999 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:42.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:43.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:29:43.101 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:43.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:43.137 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:43.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:43.238 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:43.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:43.273 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:43.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:43.321 INFO:teuthology.orchestra.run.smithi150.stderr:nodeep-scrub is set 2024-09-11T08:29:43.374 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:29:43.404 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:43.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:43.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:29:43.506 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:43.537 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:43.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:43.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:43.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:43 smithi181 ceph-mon[32701]: osdmap e438: 8 total, 8 up, 8 in 2024-09-11T08:29:43.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:43 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1466269496' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:29:43.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:43 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:29:43.606 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:43 smithi150 ceph-mon[29276]: osdmap e438: 8 total, 8 up, 8 in 2024-09-11T08:29:43.606 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:43 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1466269496' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:29:43.606 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:43 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:29:43.607 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:43 smithi150 ceph-mon[39612]: osdmap e438: 8 total, 8 up, 8 in 2024-09-11T08:29:43.607 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:43 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1466269496' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:29:43.607 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:43 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:29:43.639 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:29:43.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:43.668 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:43.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:29:43.770 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:29:43.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:43.810 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:43.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:29:43.911 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:43.947 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:43.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:43.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:44.049 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:29:44.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:44.082 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:44.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:29:44.184 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:44.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:44.226 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:44.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:44.327 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:44.357 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:44.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:44.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:44.459 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:44.496 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:44.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:44.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:44.598 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:44.642 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:44.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:44.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:44.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:44 smithi150 ceph-mon[39612]: pgmap v1016: 148 pgs: 148 active+clean; 1.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 893 op/s 2024-09-11T08:29:44.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:44 smithi150 ceph-mon[39612]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:29:44.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:44 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:29:44.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:44 smithi150 ceph-mon[39612]: osdmap e439: 8 total, 8 up, 8 in 2024-09-11T08:29:44.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:44 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:29:44.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:44 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:29:44.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:44 smithi150 ceph-mon[29276]: pgmap v1016: 148 pgs: 148 active+clean; 1.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 893 op/s 2024-09-11T08:29:44.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:44 smithi150 ceph-mon[29276]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:29:44.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:44 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:29:44.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:44 smithi150 ceph-mon[29276]: osdmap e439: 8 total, 8 up, 8 in 2024-09-11T08:29:44.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:44 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:29:44.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:44 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:29:44.744 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:44.770 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:44.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:44.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:44.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:44 smithi181 ceph-mon[32701]: pgmap v1016: 148 pgs: 148 active+clean; 1.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 893 op/s 2024-09-11T08:29:44.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:44 smithi181 ceph-mon[32701]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:29:44.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:44 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T08:29:44.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:44 smithi181 ceph-mon[32701]: osdmap e439: 8 total, 8 up, 8 in 2024-09-11T08:29:44.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:44 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:29:44.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:44 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:29:44.873 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:44.914 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:44.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:44.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:45.016 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:29:45.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:45.052 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:45.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:29:45.153 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:45.189 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:45.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:45.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:45.291 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:29:45.325 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:45.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:45.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:29:45.427 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:29:45.461 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:45.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:45.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:29:45.562 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:45.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:45 smithi181 ceph-mon[32701]: osdmap e440: 8 total, 8 up, 8 in 2024-09-11T08:29:45.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:29:45 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:29:45] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:29:45.605 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:45.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:45.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:45.706 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:45.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:45 smithi150 ceph-mon[29276]: osdmap e440: 8 total, 8 up, 8 in 2024-09-11T08:29:45.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:45 smithi150 ceph-mon[39612]: osdmap e440: 8 total, 8 up, 8 in 2024-09-11T08:29:45.746 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:45.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:45.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:45.848 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:45.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:45.880 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:45.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:45.981 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:46.006 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:46.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:46.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:46.108 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:46.145 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:46.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:46.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:46.247 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:46.281 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:46.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:46.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:46.383 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:46.413 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:46.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:46.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:46.515 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:46.548 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:46.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:46.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:46.650 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:46.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:46.691 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:46.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:46.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:46 smithi150 ceph-mon[29276]: pgmap v1019: 148 pgs: 148 active+clean; 1.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.08k op/s 2024-09-11T08:29:46.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:46 smithi150 ceph-mon[39612]: pgmap v1019: 148 pgs: 148 active+clean; 1.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.08k op/s 2024-09-11T08:29:46.793 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:46.822 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:46 smithi181 ceph-mon[32701]: pgmap v1019: 148 pgs: 148 active+clean; 1.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.08k op/s 2024-09-11T08:29:46.832 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:46.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:46.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:46.886 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T08:29:46.897 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-11T08:29:46.934 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:29:47.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:47.009 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:47.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:29:47.110 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:47.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:47.214 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:47.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:47.315 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:47.339 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:47.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:47.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:47.441 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:29:47.472 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:47.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:47.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:29:47.574 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:29:47.603 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:47.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:47.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:29:47.705 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:29:47.727 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:47.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:47.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:29:47.829 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:29:47.851 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:47.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:47.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:29:47.952 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:47.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:47.989 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:47.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:48.038 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:29:48.038 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:29:48.038 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:29:48.038 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:29:48.039 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:29:48.039 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:29:48.039 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:29:48.039 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:29:48.039 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:29:48.039 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:29:48.061 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:29:48.061 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:29:48.090 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:48.092 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:29:48.092 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:29:48.115 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:48.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:48.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:48.131 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:29:48.131 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:29:48.145 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:29:48.152 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:29:48.217 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:48.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:48.257 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:48.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:48.358 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:48.384 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:48.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:48.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:48.485 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:48.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:48.514 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:48.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:48.615 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:48.625 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:48 smithi181 ceph-mon[32701]: pgmap v1020: 148 pgs: 148 active+clean; 1.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 886 op/s 2024-09-11T08:29:48.625 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:48 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1919016701' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:29:48.626 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:48 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:29:48.626 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:48 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:29:48.645 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:48.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:48.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:48.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:48 smithi150 ceph-mon[29276]: pgmap v1020: 148 pgs: 148 active+clean; 1.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 886 op/s 2024-09-11T08:29:48.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:48 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1919016701' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:29:48.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:48 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:29:48.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:48 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:29:48.739 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:29:48 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:29:48] "GET /metrics HTTP/1.1" 200 224727 "" "Prometheus/2.43.0" 2024-09-11T08:29:48.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:48 smithi150 ceph-mon[39612]: pgmap v1020: 148 pgs: 148 active+clean; 1.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 886 op/s 2024-09-11T08:29:48.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:48 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1919016701' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:29:48.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:48 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:29:48.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:48 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:29:48.747 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:48.776 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:48.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:48.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:48.877 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:29:48.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:48.910 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:48.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:29:49.010 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:49.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:49.043 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:49.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:49.144 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:49.180 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:49.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:49.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:49.282 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:49.323 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:29:49 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:29:49.323 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:49.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:49.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:49.425 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:49.447 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:49 smithi150 ceph-mon[29276]: osdmap e441: 8 total, 8 up, 8 in 2024-09-11T08:29:49.447 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:49 smithi150 ceph-mon[39612]: osdmap e441: 8 total, 8 up, 8 in 2024-09-11T08:29:49.462 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:49.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:49.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:49.564 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:49.591 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:49.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:49.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:49.693 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:49.725 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:49.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:49.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:49.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:49 smithi181 ceph-mon[32701]: osdmap e441: 8 total, 8 up, 8 in 2024-09-11T08:29:49.827 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:29:49.861 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:49.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:49.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:29:49.963 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:29:49.989 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:49.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:49.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:29:50.092 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:29:50.122 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:50.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:50.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:29:50.224 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:29:50.254 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:50.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:50.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:29:50.356 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:50.393 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:50.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:50.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:50.407 INFO:teuthology.orchestra.run.smithi150.stderr:reweighted osd.7 to 1 (10000) 2024-09-11T08:29:50.495 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:50.524 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:50.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:50.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:50.626 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:50.656 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:50.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:50.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:50.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:50 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:29:50.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:50 smithi150 ceph-mon[29276]: pgmap v1022: 148 pgs: 148 active+clean; 1.0 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.53k op/s 2024-09-11T08:29:50.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:50 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2695265830' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-11T08:29:50.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:50 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:29:50.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:50 smithi150 ceph-mon[39612]: pgmap v1022: 148 pgs: 148 active+clean; 1.0 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.53k op/s 2024-09-11T08:29:50.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:50 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2695265830' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-11T08:29:50.757 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:29:50.791 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:50.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:50.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:29:50.822 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:50 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:29:50.822 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:50 smithi181 ceph-mon[32701]: pgmap v1022: 148 pgs: 148 active+clean; 1.0 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.53k op/s 2024-09-11T08:29:50.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:50 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2695265830' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-11T08:29:50.892 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:50.928 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:50.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:50.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:50.977 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T08:29:51.031 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:29:51.058 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:51.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:51.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:29:51.160 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:51.189 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:51.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:51.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:51.291 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:29:51.320 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:51.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:51.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:29:51.422 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:29:51.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:51.450 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:51.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:29:51.551 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:51.589 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:51.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:51.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:51.691 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:29:51.720 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:51.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:51.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:29:51.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:51 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2695265830' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-11T08:29:51.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:51 smithi150 ceph-mon[29276]: osdmap e442: 8 total, 8 up, 8 in 2024-09-11T08:29:51.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:51 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2695265830' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-11T08:29:51.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:51 smithi150 ceph-mon[39612]: osdmap e442: 8 total, 8 up, 8 in 2024-09-11T08:29:51.822 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:29:51.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:51 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2695265830' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-11T08:29:51.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:51 smithi181 ceph-mon[32701]: osdmap e442: 8 total, 8 up, 8 in 2024-09-11T08:29:51.850 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:51.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:51.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:29:51.952 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:29:51.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:51.988 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:51.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:29:52.089 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:52.118 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:52.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:52.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:52.220 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:52.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:52.255 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:52.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:52.356 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:52.378 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:52.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:52.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:52.480 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:29:52.518 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:52.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:52.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:29:52.620 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:52.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:52.655 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:52.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:52.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:52 smithi150 ceph-mon[29276]: pgmap v1024: 148 pgs: 148 active+clean; 1017 MiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 741 B/s rd, 0 B/s wr, 853 op/s 2024-09-11T08:29:52.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:52 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:29:52.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:52 smithi150 ceph-mon[39612]: pgmap v1024: 148 pgs: 148 active+clean; 1017 MiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 741 B/s rd, 0 B/s wr, 853 op/s 2024-09-11T08:29:52.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:52 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:29:52.757 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:29:52.786 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:52.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:52.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:29:52.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:52 smithi181 ceph-mon[32701]: pgmap v1024: 148 pgs: 148 active+clean; 1017 MiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 741 B/s rd, 0 B/s wr, 853 op/s 2024-09-11T08:29:52.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:52 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:29:52.889 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:52.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:52.925 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:52.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:53.027 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:53.050 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:53.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:53.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:53.151 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:29:53.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:53.186 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:53.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:29:53.287 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:53.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:53.327 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:53.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:53.428 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:53.455 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:53.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:53.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:53.556 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:29:53.590 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:53.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:53.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:29:53.692 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:53.733 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:53.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:53.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:53.835 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:53.873 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:53.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:53.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:53.975 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:54.001 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:54.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:54.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:54.103 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:54.146 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:54.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:54.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:54.248 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:54.291 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:54.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:54.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:54.393 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:54.424 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:54.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:54.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:54.526 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:54.553 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:54.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:54.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:54.655 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:54.682 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:54.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:54.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:54.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:54 smithi150 ceph-mon[29276]: pgmap v1025: 148 pgs: 148 active+clean; 931 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 866 op/s 2024-09-11T08:29:54.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:54 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:29:54.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:54 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:29:54.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:54 smithi150 ceph-mon[39612]: pgmap v1025: 148 pgs: 148 active+clean; 931 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 866 op/s 2024-09-11T08:29:54.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:54 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:29:54.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:54 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:29:54.783 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:54.813 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:54.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:54.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:54.822 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:54 smithi181 ceph-mon[32701]: pgmap v1025: 148 pgs: 148 active+clean; 931 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 866 op/s 2024-09-11T08:29:54.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:54 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:29:54.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:54 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:29:54.915 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:54.966 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:54.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:54.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:54.997 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd reweight 5 1 2024-09-11T08:29:55.067 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:55.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:55.100 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:55.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:55.201 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:55.231 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:55.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:55.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:55.332 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:55.389 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:55.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:55.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:55.446 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:29:55 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:29:55] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:29:55.491 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:55.539 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:55.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:55.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:55.641 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:55.660 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:55 smithi150 ceph-mon[29276]: osdmap e443: 8 total, 8 up, 8 in 2024-09-11T08:29:55.661 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:55 smithi150 ceph-mon[39612]: osdmap e443: 8 total, 8 up, 8 in 2024-09-11T08:29:55.667 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:55.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:55.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:55.769 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:55.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:55.817 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:55.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:55.817 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-09-11T08:29:55.817 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:29:55.822 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:55 smithi181 ceph-mon[32701]: osdmap e443: 8 total, 8 up, 8 in 2024-09-11T08:29:55.918 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:55.957 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:55.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:55.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:56.059 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:56.096 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:56.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:56.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:56.198 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:29:56.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:56.231 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:56.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:29:56.332 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:56.365 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:56.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:56.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:56.451 INFO:teuthology.orchestra.run.smithi150.stderr:noscrub is unset 2024-09-11T08:29:56.467 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:56.494 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:56.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:56.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:56.595 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:56.629 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:56.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:56.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:56.731 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:29:56.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:56 smithi150 ceph-mon[39612]: pgmap v1027: 148 pgs: 148 active+clean; 671 MiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 747 B/s rd, 0 B/s wr, 1.21k op/s 2024-09-11T08:29:56.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:56 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/860972855' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:29:56.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:56 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:29:56.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:56 smithi150 ceph-mon[29276]: pgmap v1027: 148 pgs: 148 active+clean; 671 MiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 747 B/s rd, 0 B/s wr, 1.21k op/s 2024-09-11T08:29:56.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:56 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/860972855' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:29:56.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:56 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:29:56.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:56.770 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:56.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:29:56.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:56 smithi181 ceph-mon[32701]: pgmap v1027: 148 pgs: 148 active+clean; 671 MiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 747 B/s rd, 0 B/s wr, 1.21k op/s 2024-09-11T08:29:56.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:56 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/860972855' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:29:56.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:56 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:29:56.870 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:56.913 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:56.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:56.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:57.015 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:57.051 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:57.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:57.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:57.153 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:29:57.181 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:57.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:57.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:29:57.283 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:29:57.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:57.323 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:57.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:29:57.424 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:57.456 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:57.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:57.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:57.519 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-11T08:29:57.531 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:29:57.532 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:29:57.558 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:57.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:57.614 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:57.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:57.715 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:57.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:57 smithi150 ceph-mon[29276]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:29:57.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:57 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:29:57.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:57 smithi150 ceph-mon[29276]: osdmap e444: 8 total, 8 up, 8 in 2024-09-11T08:29:57.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:57 smithi150 ceph-mon[29276]: 6.1f scrub starts 2024-09-11T08:29:57.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:57 smithi150 ceph-mon[29276]: 6.1f scrub ok 2024-09-11T08:29:57.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:57 smithi150 ceph-mon[29276]: 6.f scrub starts 2024-09-11T08:29:57.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:57 smithi150 ceph-mon[29276]: 6.f scrub ok 2024-09-11T08:29:57.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:57 smithi150 ceph-mon[29276]: 6.13 scrub starts 2024-09-11T08:29:57.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:57 smithi150 ceph-mon[29276]: 6.13 scrub ok 2024-09-11T08:29:57.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:57 smithi150 ceph-mon[39612]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:29:57.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:57 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:29:57.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:57 smithi150 ceph-mon[39612]: osdmap e444: 8 total, 8 up, 8 in 2024-09-11T08:29:57.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:57 smithi150 ceph-mon[39612]: 6.1f scrub starts 2024-09-11T08:29:57.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:57 smithi150 ceph-mon[39612]: 6.1f scrub ok 2024-09-11T08:29:57.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:57 smithi150 ceph-mon[39612]: 6.f scrub starts 2024-09-11T08:29:57.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:57 smithi150 ceph-mon[39612]: 6.f scrub ok 2024-09-11T08:29:57.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:57 smithi150 ceph-mon[39612]: 6.13 scrub starts 2024-09-11T08:29:57.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:57 smithi150 ceph-mon[39612]: 6.13 scrub ok 2024-09-11T08:29:57.741 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:57.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:57.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:57.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:57 smithi181 ceph-mon[32701]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:29:57.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:57 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:29:57.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:57 smithi181 ceph-mon[32701]: osdmap e444: 8 total, 8 up, 8 in 2024-09-11T08:29:57.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:57 smithi181 ceph-mon[32701]: 6.1f scrub starts 2024-09-11T08:29:57.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:57 smithi181 ceph-mon[32701]: 6.1f scrub ok 2024-09-11T08:29:57.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:57 smithi181 ceph-mon[32701]: 6.f scrub starts 2024-09-11T08:29:57.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:57 smithi181 ceph-mon[32701]: 6.f scrub ok 2024-09-11T08:29:57.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:57 smithi181 ceph-mon[32701]: 6.13 scrub starts 2024-09-11T08:29:57.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:57 smithi181 ceph-mon[32701]: 6.13 scrub ok 2024-09-11T08:29:57.843 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:57.884 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:57.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:57.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:57.986 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:29:58.018 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:58.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:58.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:29:58.120 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:58.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:58.175 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:58.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:58.276 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:58.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:58.310 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:58.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:58.411 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:29:58.438 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:58.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:58.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:29:58.540 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:58.563 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:58 smithi150 ceph-mon[29276]: pgmap v1029: 148 pgs: 148 active+clean; 671 MiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 749 B/s rd, 0 B/s wr, 810 op/s 2024-09-11T08:29:58.563 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:58 smithi150 ceph-mon[29276]: 11.e scrub starts 2024-09-11T08:29:58.563 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:58 smithi150 ceph-mon[29276]: 11.e scrub ok 2024-09-11T08:29:58.563 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:58 smithi150 ceph-mon[29276]: 3.a scrub starts 2024-09-11T08:29:58.564 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:58 smithi150 ceph-mon[29276]: 3.a scrub ok 2024-09-11T08:29:58.564 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:58 smithi150 ceph-mon[29276]: 3.4 scrub starts 2024-09-11T08:29:58.564 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:58 smithi150 ceph-mon[29276]: 3.4 scrub ok 2024-09-11T08:29:58.564 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:58 smithi150 ceph-mon[29276]: 5.17 scrub starts 2024-09-11T08:29:58.564 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:58 smithi150 ceph-mon[29276]: 5.17 scrub ok 2024-09-11T08:29:58.564 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:58 smithi150 ceph-mon[39612]: pgmap v1029: 148 pgs: 148 active+clean; 671 MiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 749 B/s rd, 0 B/s wr, 810 op/s 2024-09-11T08:29:58.564 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:58 smithi150 ceph-mon[39612]: 11.e scrub starts 2024-09-11T08:29:58.564 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:58 smithi150 ceph-mon[39612]: 11.e scrub ok 2024-09-11T08:29:58.565 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:58 smithi150 ceph-mon[39612]: 3.a scrub starts 2024-09-11T08:29:58.565 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:58 smithi150 ceph-mon[39612]: 3.a scrub ok 2024-09-11T08:29:58.565 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:58 smithi150 ceph-mon[39612]: 3.4 scrub starts 2024-09-11T08:29:58.565 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:58 smithi150 ceph-mon[39612]: 3.4 scrub ok 2024-09-11T08:29:58.565 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:58 smithi150 ceph-mon[39612]: 5.17 scrub starts 2024-09-11T08:29:58.565 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:58 smithi150 ceph-mon[39612]: 5.17 scrub ok 2024-09-11T08:29:58.579 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:58.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:58.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:58.681 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:29:58.721 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:58.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:58.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:29:58.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:58 smithi181 ceph-mon[32701]: pgmap v1029: 148 pgs: 148 active+clean; 671 MiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 749 B/s rd, 0 B/s wr, 810 op/s 2024-09-11T08:29:58.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:58 smithi181 ceph-mon[32701]: 11.e scrub starts 2024-09-11T08:29:58.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:58 smithi181 ceph-mon[32701]: 11.e scrub ok 2024-09-11T08:29:58.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:58 smithi181 ceph-mon[32701]: 3.a scrub starts 2024-09-11T08:29:58.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:58 smithi181 ceph-mon[32701]: 3.a scrub ok 2024-09-11T08:29:58.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:58 smithi181 ceph-mon[32701]: 3.4 scrub starts 2024-09-11T08:29:58.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:58 smithi181 ceph-mon[32701]: 3.4 scrub ok 2024-09-11T08:29:58.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:58 smithi181 ceph-mon[32701]: 5.17 scrub starts 2024-09-11T08:29:58.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:58 smithi181 ceph-mon[32701]: 5.17 scrub ok 2024-09-11T08:29:58.824 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:29:58.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:58.856 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:58.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:29:58.898 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:29:58.957 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:58.988 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:29:58 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:29:58] "GET /metrics HTTP/1.1" 200 224729 "" "Prometheus/2.43.0" 2024-09-11T08:29:58.991 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:58.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:58.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:59.093 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:59.138 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:59.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:59.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:29:59.240 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:29:59.277 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:59.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:59.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:29:59.323 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:29:59 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:29:59.378 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:29:59.414 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:59.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:59.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:29:59.516 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:59.540 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:59.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:59.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:59.642 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:29:59.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:59.677 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:59.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:29:59.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:59 smithi150 ceph-mon[29276]: 5.13 scrub starts 2024-09-11T08:29:59.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:59 smithi150 ceph-mon[29276]: 5.13 scrub ok 2024-09-11T08:29:59.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:59 smithi150 ceph-mon[29276]: 5.1e scrub starts 2024-09-11T08:29:59.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:59 smithi150 ceph-mon[29276]: 5.1e scrub ok 2024-09-11T08:29:59.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:59 smithi150 ceph-mon[29276]: 6.17 scrub starts 2024-09-11T08:29:59.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:59 smithi150 ceph-mon[29276]: 6.17 scrub ok 2024-09-11T08:29:59.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:59 smithi150 ceph-mon[29276]: 11.5 scrub starts 2024-09-11T08:29:59.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:29:59 smithi150 ceph-mon[29276]: 11.5 scrub ok 2024-09-11T08:29:59.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:59 smithi150 ceph-mon[39612]: 5.13 scrub starts 2024-09-11T08:29:59.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:59 smithi150 ceph-mon[39612]: 5.13 scrub ok 2024-09-11T08:29:59.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:59 smithi150 ceph-mon[39612]: 5.1e scrub starts 2024-09-11T08:29:59.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:59 smithi150 ceph-mon[39612]: 5.1e scrub ok 2024-09-11T08:29:59.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:59 smithi150 ceph-mon[39612]: 6.17 scrub starts 2024-09-11T08:29:59.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:59 smithi150 ceph-mon[39612]: 6.17 scrub ok 2024-09-11T08:29:59.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:59 smithi150 ceph-mon[39612]: 11.5 scrub starts 2024-09-11T08:29:59.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:29:59 smithi150 ceph-mon[39612]: 11.5 scrub ok 2024-09-11T08:29:59.778 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:29:59.805 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:29:59.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:59.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:29:59.822 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:59 smithi181 ceph-mon[32701]: 5.13 scrub starts 2024-09-11T08:29:59.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:59 smithi181 ceph-mon[32701]: 5.13 scrub ok 2024-09-11T08:29:59.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:59 smithi181 ceph-mon[32701]: 5.1e scrub starts 2024-09-11T08:29:59.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:59 smithi181 ceph-mon[32701]: 5.1e scrub ok 2024-09-11T08:29:59.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:59 smithi181 ceph-mon[32701]: 6.17 scrub starts 2024-09-11T08:29:59.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:59 smithi181 ceph-mon[32701]: 6.17 scrub ok 2024-09-11T08:29:59.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:59 smithi181 ceph-mon[32701]: 11.5 scrub starts 2024-09-11T08:29:59.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:29:59 smithi181 ceph-mon[32701]: 11.5 scrub ok 2024-09-11T08:29:59.907 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:29:59.943 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:29:59.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:29:59.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:30:00.045 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:30:00.076 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T08:30:00.077 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:00.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:00.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:30:00.179 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:30:00.221 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:00.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:00.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:30:00.322 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:30:00.348 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:00.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:00.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:30:00.449 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:30:00.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:00.481 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:00.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:30:00.582 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:30:00.599 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:00 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:30:00.599 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:00 smithi150 ceph-mon[39612]: pgmap v1030: 148 pgs: 148 active+clean; 347 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.34k op/s 2024-09-11T08:30:00.599 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:00 smithi150 ceph-mon[39612]: 4.18 scrub starts 2024-09-11T08:30:00.599 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:00 smithi150 ceph-mon[39612]: 4.18 scrub ok 2024-09-11T08:30:00.599 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:00 smithi150 ceph-mon[39612]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-09-11T08:30:00.599 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:00 smithi150 ceph-mon[39612]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-09-11T08:30:00.600 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:00 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:30:00.600 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:00 smithi150 ceph-mon[29276]: pgmap v1030: 148 pgs: 148 active+clean; 347 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.34k op/s 2024-09-11T08:30:00.600 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:00 smithi150 ceph-mon[29276]: 4.18 scrub starts 2024-09-11T08:30:00.600 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:00 smithi150 ceph-mon[29276]: 4.18 scrub ok 2024-09-11T08:30:00.600 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:00 smithi150 ceph-mon[29276]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-09-11T08:30:00.600 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:00 smithi150 ceph-mon[29276]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-09-11T08:30:00.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:00.630 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:00.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:30:00.731 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:30:00.772 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:00.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:00.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:30:00.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:00 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:30:00.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:00 smithi181 ceph-mon[32701]: pgmap v1030: 148 pgs: 148 active+clean; 347 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.34k op/s 2024-09-11T08:30:00.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:00 smithi181 ceph-mon[32701]: 4.18 scrub starts 2024-09-11T08:30:00.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:00 smithi181 ceph-mon[32701]: 4.18 scrub ok 2024-09-11T08:30:00.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:00 smithi181 ceph-mon[32701]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-09-11T08:30:00.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:00 smithi181 ceph-mon[32701]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-09-11T08:30:00.874 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:30:00.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:00.905 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:00.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:30:01.006 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:30:01.038 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:01.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:01.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:30:01.140 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:30:01.172 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:01.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:01.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:30:01.274 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:30:01.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:01.313 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:01.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:30:01.414 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:30:01.453 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:01.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:01.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:30:01.555 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:30:01.582 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:01.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:01.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:30:01.684 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:30:01.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:01.712 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:01.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:30:01.813 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:30:01.843 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:01.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:01.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:30:01.945 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:30:01.975 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:01.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:01.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:30:02.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:01 smithi181 ceph-mon[32701]: 3.1e scrub starts 2024-09-11T08:30:02.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:01 smithi181 ceph-mon[32701]: 3.1e scrub ok 2024-09-11T08:30:02.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:01 smithi181 ceph-mon[32701]: 5.6 scrub starts 2024-09-11T08:30:02.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:01 smithi181 ceph-mon[32701]: 5.6 scrub ok 2024-09-11T08:30:02.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:01 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:30:02.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:01 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:30:02.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:01 smithi181 ceph-mon[32701]: pgmap v1031: 148 pgs: 148 active+clean; 326 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.21k op/s 2024-09-11T08:30:02.077 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:30:02.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:02.112 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:02.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:30:02.213 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:30:02.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:01 smithi150 ceph-mon[29276]: 3.1e scrub starts 2024-09-11T08:30:02.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:01 smithi150 ceph-mon[29276]: 3.1e scrub ok 2024-09-11T08:30:02.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:01 smithi150 ceph-mon[29276]: 5.6 scrub starts 2024-09-11T08:30:02.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:01 smithi150 ceph-mon[29276]: 5.6 scrub ok 2024-09-11T08:30:02.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:01 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:30:02.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:01 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:30:02.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:01 smithi150 ceph-mon[29276]: pgmap v1031: 148 pgs: 148 active+clean; 326 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.21k op/s 2024-09-11T08:30:02.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:01 smithi150 ceph-mon[39612]: 3.1e scrub starts 2024-09-11T08:30:02.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:01 smithi150 ceph-mon[39612]: 3.1e scrub ok 2024-09-11T08:30:02.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:01 smithi150 ceph-mon[39612]: 5.6 scrub starts 2024-09-11T08:30:02.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:01 smithi150 ceph-mon[39612]: 5.6 scrub ok 2024-09-11T08:30:02.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:01 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:30:02.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:01 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:30:02.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:01 smithi150 ceph-mon[39612]: pgmap v1031: 148 pgs: 148 active+clean; 326 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.21k op/s 2024-09-11T08:30:02.242 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:02.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:02.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:30:02.343 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:30:02.378 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:02.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:02.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:30:02.480 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:30:02.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:02.507 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:02.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:30:02.608 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:30:02.645 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:02.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:02.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:30:02.747 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:30:02.777 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:02.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:02.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:30:02.879 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:30:02.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:02.914 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:02.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:30:03.015 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:30:03.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:03.045 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:03.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:30:03.061 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Removed 126046 objects 2024-09-11T08:30:03.062 INFO:tasks.radosbench.radosbench.0.smithi150.stdout:Clean up completed and total clean up time :112.416 2024-09-11T08:30:03.072 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:02 smithi181 ceph-mon[32701]: 4.c scrub starts 2024-09-11T08:30:03.072 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:02 smithi181 ceph-mon[32701]: 4.c scrub ok 2024-09-11T08:30:03.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:02 smithi181 ceph-mon[32701]: osdmap e445: 8 total, 8 up, 8 in 2024-09-11T08:30:03.141 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:02 smithi150 ceph-mon[29276]: 4.c scrub starts 2024-09-11T08:30:03.142 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:02 smithi150 ceph-mon[29276]: 4.c scrub ok 2024-09-11T08:30:03.142 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:02 smithi150 ceph-mon[29276]: osdmap e445: 8 total, 8 up, 8 in 2024-09-11T08:30:03.142 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:02 smithi150 ceph-mon[39612]: 4.c scrub starts 2024-09-11T08:30:03.142 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:02 smithi150 ceph-mon[39612]: 4.c scrub ok 2024-09-11T08:30:03.142 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:02 smithi150 ceph-mon[39612]: osdmap e445: 8 total, 8 up, 8 in 2024-09-11T08:30:03.147 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:30:03.168 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:03.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:03.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:30:03.270 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:30:03.291 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:03.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:03.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:30:03.393 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:30:03.414 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:03.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:03.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:30:03.515 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:30:03.538 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:03.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:03.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:30:03.640 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:30:03.662 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:03.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:03.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:30:03.764 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:30:03.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:03.786 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:03.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:30:03.887 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:30:03.909 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:03.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:03.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:30:03.932 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:03 smithi150 ceph-mon[29276]: 6.10 scrub starts 2024-09-11T08:30:03.932 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:03 smithi150 ceph-mon[29276]: 6.10 scrub ok 2024-09-11T08:30:03.932 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:03 smithi150 ceph-mon[29276]: 4.d scrub starts 2024-09-11T08:30:03.932 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:03 smithi150 ceph-mon[29276]: 4.d scrub ok 2024-09-11T08:30:03.932 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:03 smithi150 ceph-mon[29276]: pgmap v1033: 148 pgs: 148 active+clean; 235 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 873 op/s 2024-09-11T08:30:03.933 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:03 smithi150 ceph-mon[39612]: 6.10 scrub starts 2024-09-11T08:30:03.933 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:03 smithi150 ceph-mon[39612]: 6.10 scrub ok 2024-09-11T08:30:03.933 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:03 smithi150 ceph-mon[39612]: 4.d scrub starts 2024-09-11T08:30:03.933 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:03 smithi150 ceph-mon[39612]: 4.d scrub ok 2024-09-11T08:30:03.933 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:03 smithi150 ceph-mon[39612]: pgmap v1033: 148 pgs: 148 active+clean; 235 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 873 op/s 2024-09-11T08:30:03.942 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-09-11T08:30:03.942 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:30:04.010 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:30:04.033 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:04.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:04.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:30:04.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:03 smithi181 ceph-mon[32701]: 6.10 scrub starts 2024-09-11T08:30:04.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:03 smithi181 ceph-mon[32701]: 6.10 scrub ok 2024-09-11T08:30:04.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:03 smithi181 ceph-mon[32701]: 4.d scrub starts 2024-09-11T08:30:04.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:03 smithi181 ceph-mon[32701]: 4.d scrub ok 2024-09-11T08:30:04.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:03 smithi181 ceph-mon[32701]: pgmap v1033: 148 pgs: 148 active+clean; 235 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 873 op/s 2024-09-11T08:30:04.134 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:30:04.156 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:04.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:04.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:30:04.158 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:30:04.158 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:30:04.158 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:30:04.158 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:30:04.159 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:30:04.159 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:30:04.175 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:30:04.176 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:30:04.191 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:30:04.192 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:30:04.208 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:30:04.208 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:30:04.223 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:30:04.223 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:30:04.238 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:30:04.239 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:30:04.259 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:30:04.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:04.281 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:04.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:30:04.382 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:30:04.405 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:04.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:04.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:30:04.507 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:30:04.529 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:04.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:04.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:30:04.631 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:30:04.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:04.653 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:04.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:30:04.754 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:30:04.775 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:04.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:04.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:30:04.877 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:30:04.898 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:04.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:04.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:30:04.917 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:04 smithi150 ceph-mon[29276]: 4.3 scrub starts 2024-09-11T08:30:04.917 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:04 smithi150 ceph-mon[29276]: 4.3 scrub ok 2024-09-11T08:30:04.917 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:04 smithi150 ceph-mon[29276]: 5.11 scrub starts 2024-09-11T08:30:04.917 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:04 smithi150 ceph-mon[29276]: 5.11 scrub ok 2024-09-11T08:30:04.918 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:04 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1023794385' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:30:04.918 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:04 smithi150 ceph-mon[29276]: 11.c scrub starts 2024-09-11T08:30:04.918 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:04 smithi150 ceph-mon[29276]: 11.c scrub ok 2024-09-11T08:30:04.918 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:04 smithi150 ceph-mon[39612]: 4.3 scrub starts 2024-09-11T08:30:04.918 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:04 smithi150 ceph-mon[39612]: 4.3 scrub ok 2024-09-11T08:30:04.918 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:04 smithi150 ceph-mon[39612]: 5.11 scrub starts 2024-09-11T08:30:04.918 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:04 smithi150 ceph-mon[39612]: 5.11 scrub ok 2024-09-11T08:30:04.918 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:04 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1023794385' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:30:04.918 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:04 smithi150 ceph-mon[39612]: 11.c scrub starts 2024-09-11T08:30:04.918 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:04 smithi150 ceph-mon[39612]: 11.c scrub ok 2024-09-11T08:30:05.000 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:30:05.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:05.022 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:05.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:30:05.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:04 smithi181 ceph-mon[32701]: 4.3 scrub starts 2024-09-11T08:30:05.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:04 smithi181 ceph-mon[32701]: 4.3 scrub ok 2024-09-11T08:30:05.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:04 smithi181 ceph-mon[32701]: 5.11 scrub starts 2024-09-11T08:30:05.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:04 smithi181 ceph-mon[32701]: 5.11 scrub ok 2024-09-11T08:30:05.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:04 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1023794385' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:30:05.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:04 smithi181 ceph-mon[32701]: 11.c scrub starts 2024-09-11T08:30:05.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:04 smithi181 ceph-mon[32701]: 11.c scrub ok 2024-09-11T08:30:05.123 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:30:05.146 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:05.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:05.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:30:05.247 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:30:05.270 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:05.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:05.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:30:05.285 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-11T08:30:05.372 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:30:05.394 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:05.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:05.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:30:05.496 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:30:05.518 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:05.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:05.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:30:05.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:30:05 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:30:05] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:30:05.619 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:30:05.641 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:05.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:05.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:30:05.743 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:30:05.766 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:05.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:05.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:30:05.868 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:30:05.889 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:05.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:05.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:30:05.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:05 smithi150 ceph-mon[29276]: 11.2 scrub starts 2024-09-11T08:30:05.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:05 smithi150 ceph-mon[29276]: 11.2 scrub ok 2024-09-11T08:30:05.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:05 smithi150 ceph-mon[29276]: pgmap v1034: 148 pgs: 148 active+clean; 5.3 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-09-11T08:30:05.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:05 smithi150 ceph-mon[39612]: 11.2 scrub starts 2024-09-11T08:30:05.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:05 smithi150 ceph-mon[39612]: 11.2 scrub ok 2024-09-11T08:30:05.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:05 smithi150 ceph-mon[39612]: pgmap v1034: 148 pgs: 148 active+clean; 5.3 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-09-11T08:30:05.991 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:30:06.015 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:06.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:06.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:30:06.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:05 smithi181 ceph-mon[32701]: 11.2 scrub starts 2024-09-11T08:30:06.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:05 smithi181 ceph-mon[32701]: 11.2 scrub ok 2024-09-11T08:30:06.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:05 smithi181 ceph-mon[32701]: pgmap v1034: 148 pgs: 148 active+clean; 5.3 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-09-11T08:30:06.080 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:30:06.117 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:30:06.139 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:06.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:06.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:30:06.240 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:30:06.266 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:06.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:06.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:30:06.368 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:30:06.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:06.391 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:06.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:30:06.492 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:30:06.518 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:06.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:06.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:30:06.620 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:30:06.642 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:06.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:06.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:30:06.744 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:30:06.767 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:06.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:06.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:30:06.816 INFO:teuthology.orchestra.run.smithi150.stderr:reweighted osd.5 to 1 (10000) 2024-09-11T08:30:06.869 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:30:06.891 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:06.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:06.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:30:06.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:06 smithi150 ceph-mon[39612]: 6.1e scrub starts 2024-09-11T08:30:06.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:06 smithi150 ceph-mon[39612]: 6.1e scrub ok 2024-09-11T08:30:06.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:06 smithi150 ceph-mon[39612]: 4.15 scrub starts 2024-09-11T08:30:06.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:06 smithi150 ceph-mon[39612]: 4.15 scrub ok 2024-09-11T08:30:06.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:06 smithi150 ceph-mon[39612]: 11.7 scrub starts 2024-09-11T08:30:06.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:06 smithi150 ceph-mon[39612]: 11.7 scrub ok 2024-09-11T08:30:06.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:06 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2214239215' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-11T08:30:06.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:06 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:30:06.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:06 smithi150 ceph-mon[29276]: 6.1e scrub starts 2024-09-11T08:30:06.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:06 smithi150 ceph-mon[29276]: 6.1e scrub ok 2024-09-11T08:30:06.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:06 smithi150 ceph-mon[29276]: 4.15 scrub starts 2024-09-11T08:30:06.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:06 smithi150 ceph-mon[29276]: 4.15 scrub ok 2024-09-11T08:30:06.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:06 smithi150 ceph-mon[29276]: 11.7 scrub starts 2024-09-11T08:30:06.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:06 smithi150 ceph-mon[29276]: 11.7 scrub ok 2024-09-11T08:30:06.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:06 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2214239215' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-11T08:30:06.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:06 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:30:06.993 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:30:07.019 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:07.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:07.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:30:07.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:06 smithi181 ceph-mon[32701]: 6.1e scrub starts 2024-09-11T08:30:07.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:06 smithi181 ceph-mon[32701]: 6.1e scrub ok 2024-09-11T08:30:07.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:06 smithi181 ceph-mon[32701]: 4.15 scrub starts 2024-09-11T08:30:07.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:06 smithi181 ceph-mon[32701]: 4.15 scrub ok 2024-09-11T08:30:07.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:06 smithi181 ceph-mon[32701]: 11.7 scrub starts 2024-09-11T08:30:07.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:06 smithi181 ceph-mon[32701]: 11.7 scrub ok 2024-09-11T08:30:07.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:06 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2214239215' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-11T08:30:07.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:06 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:30:07.121 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:30:07.143 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:07.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:07.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:30:07.245 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:30:07.271 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:07.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:07.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:30:07.373 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:30:07.395 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:07.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:07.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:30:07.496 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-11T08:30:07.497 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T08:30:07.497 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:30:07.500 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:30:07.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:07.520 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:07.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:30:07.621 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:30:07.646 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:07.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:07.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:30:07.748 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:30:07.754 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T08:30:07.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:07.772 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:07.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:30:07.841 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-09-11T08:30:07.841 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-09-11T08:30:07.874 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:30:07.899 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:07.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:07.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:30:08.001 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:30:08.023 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:08.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:08.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:30:08.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:07 smithi181 ceph-mon[32701]: 5.f scrub starts 2024-09-11T08:30:08.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:07 smithi181 ceph-mon[32701]: 5.f scrub ok 2024-09-11T08:30:08.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:07 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2214239215' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-11T08:30:08.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:07 smithi181 ceph-mon[32701]: osdmap e446: 8 total, 8 up, 8 in 2024-09-11T08:30:08.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:07 smithi181 ceph-mon[32701]: 11.0 scrub starts 2024-09-11T08:30:08.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:07 smithi181 ceph-mon[32701]: 11.0 scrub ok 2024-09-11T08:30:08.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:07 smithi181 ceph-mon[32701]: pgmap v1036: 148 pgs: 148 active+clean; 5.3 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 683 op/s 2024-09-11T08:30:08.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:07 smithi181 ceph-mon[32701]: 3.9 scrub starts 2024-09-11T08:30:08.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:07 smithi181 ceph-mon[32701]: 3.9 scrub ok 2024-09-11T08:30:08.125 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:30:08.147 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:08.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:08.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:30:08.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:07 smithi150 ceph-mon[29276]: 5.f scrub starts 2024-09-11T08:30:08.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:07 smithi150 ceph-mon[29276]: 5.f scrub ok 2024-09-11T08:30:08.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:07 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2214239215' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-11T08:30:08.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:07 smithi150 ceph-mon[29276]: osdmap e446: 8 total, 8 up, 8 in 2024-09-11T08:30:08.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:07 smithi150 ceph-mon[29276]: 11.0 scrub starts 2024-09-11T08:30:08.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:07 smithi150 ceph-mon[29276]: 11.0 scrub ok 2024-09-11T08:30:08.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:07 smithi150 ceph-mon[29276]: pgmap v1036: 148 pgs: 148 active+clean; 5.3 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 683 op/s 2024-09-11T08:30:08.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:07 smithi150 ceph-mon[29276]: 3.9 scrub starts 2024-09-11T08:30:08.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:07 smithi150 ceph-mon[29276]: 3.9 scrub ok 2024-09-11T08:30:08.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:07 smithi150 ceph-mon[39612]: 5.f scrub starts 2024-09-11T08:30:08.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:07 smithi150 ceph-mon[39612]: 5.f scrub ok 2024-09-11T08:30:08.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:07 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2214239215' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-11T08:30:08.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:07 smithi150 ceph-mon[39612]: osdmap e446: 8 total, 8 up, 8 in 2024-09-11T08:30:08.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:07 smithi150 ceph-mon[39612]: 11.0 scrub starts 2024-09-11T08:30:08.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:07 smithi150 ceph-mon[39612]: 11.0 scrub ok 2024-09-11T08:30:08.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:07 smithi150 ceph-mon[39612]: pgmap v1036: 148 pgs: 148 active+clean; 5.3 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 683 op/s 2024-09-11T08:30:08.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:07 smithi150 ceph-mon[39612]: 3.9 scrub starts 2024-09-11T08:30:08.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:07 smithi150 ceph-mon[39612]: 3.9 scrub ok 2024-09-11T08:30:08.249 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:30:08.275 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:08.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:08.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:30:08.377 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:30:08.402 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:08.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:08.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:30:08.504 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:30:08.528 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:08.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:08.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:30:08.630 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:30:08.653 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:08.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:08.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:30:08.755 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:30:08.778 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:08.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:08.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:30:08.820 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:30:08 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:30:08] "GET /metrics HTTP/1.1" 200 224672 "" "Prometheus/2.43.0" 2024-09-11T08:30:08.879 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:30:08.903 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:08.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:08.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:30:09.005 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:30:09.029 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:09.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:09.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:30:09.071 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:08 smithi181 ceph-mon[32701]: 6.0 scrub starts 2024-09-11T08:30:09.071 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:08 smithi181 ceph-mon[32701]: 6.0 scrub ok 2024-09-11T08:30:09.072 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:08 smithi181 ceph-mon[32701]: 11.d scrub starts 2024-09-11T08:30:09.072 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:08 smithi181 ceph-mon[32701]: 11.d scrub ok 2024-09-11T08:30:09.131 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:30:09.139 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:08 smithi150 ceph-mon[29276]: 6.0 scrub starts 2024-09-11T08:30:09.139 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:08 smithi150 ceph-mon[29276]: 6.0 scrub ok 2024-09-11T08:30:09.139 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:08 smithi150 ceph-mon[29276]: 11.d scrub starts 2024-09-11T08:30:09.140 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:08 smithi150 ceph-mon[29276]: 11.d scrub ok 2024-09-11T08:30:09.140 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:08 smithi150 ceph-mon[39612]: 6.0 scrub starts 2024-09-11T08:30:09.140 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:08 smithi150 ceph-mon[39612]: 6.0 scrub ok 2024-09-11T08:30:09.140 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:08 smithi150 ceph-mon[39612]: 11.d scrub starts 2024-09-11T08:30:09.140 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:08 smithi150 ceph-mon[39612]: 11.d scrub ok 2024-09-11T08:30:09.156 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:09.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:09.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:30:09.257 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:30:09.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:09.280 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:09.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:30:09.323 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:30:09 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:30:09.382 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:30:09.403 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:09.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:09.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:30:09.505 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:30:09.531 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:09.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:09.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:30:09.633 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:30:09.655 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:09.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:09.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:30:09.760 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:30:09.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:09.787 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:09.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:30:09.888 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:30:09.910 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:09.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:09.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:30:10.011 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:30:10.037 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:10.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:10.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:30:10.064 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:30:10.073 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-11T08:30:10.073 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 20, 2024-09-11T08:30:10.073 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 600, 2024-09-11T08:30:10.074 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [ 2024-09-11T08:30:10.074 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.074 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.25799.0:129078 11.9 11:9063d9ea:::benchmark_data_smithi150_187410_object3029:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-09-11T08:30:10.074 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:28:13.237961+0000", 2024-09-11T08:30:10.074 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 116.83239632900001, 2024-09-11T08:30:10.074 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.30655492699999998, 2024-09-11T08:30:10.075 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:30:10.075 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:30:10.075 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:30:10.075 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.25799", 2024-09-11T08:30:10.075 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1403604046", 2024-09-11T08:30:10.075 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 129078 2024-09-11T08:30:10.076 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.076 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:30:10.076 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.076 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:30:10.076 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.237961+0000", 2024-09-11T08:30:10.077 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:30:10.077 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.077 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.077 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:30:10.077 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.237961+0000", 2024-09-11T08:30:10.078 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:30:10.078 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.078 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.078 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:30:10.078 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.237960+0000", 2024-09-11T08:30:10.079 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:30:10.079 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.079 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.079 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:30:10.079 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.237965+0000", 2024-09-11T08:30:10.079 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.0050000000000004e-06 2024-09-11T08:30:10.079 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.079 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.079 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:30:10.079 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.237972+0000", 2024-09-11T08:30:10.079 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.5100000000000001e-06 2024-09-11T08:30:10.079 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.079 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.080 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:30:10.080 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.237976+0000", 2024-09-11T08:30:10.080 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.6990000000000001e-06 2024-09-11T08:30:10.080 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.080 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.080 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:30:10.080 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.238013+0000", 2024-09-11T08:30:10.080 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.6704e-05 2024-09-11T08:30:10.080 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.080 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.080 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:30:10.080 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.238177+0000", 2024-09-11T08:30:10.081 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00016445300000000001 2024-09-11T08:30:10.081 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.081 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.081 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 0,6", 2024-09-11T08:30:10.081 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.238327+0000", 2024-09-11T08:30:10.081 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00014976200000000001 2024-09-11T08:30:10.081 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.081 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.081 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:30:10.081 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.241048+0000", 2024-09-11T08:30:10.081 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0027211380000000001 2024-09-11T08:30:10.081 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.081 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.082 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:30:10.082 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.247102+0000", 2024-09-11T08:30:10.082 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0060539399999999998 2024-09-11T08:30:10.082 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.082 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.082 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:30:10.082 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.544338+0000", 2024-09-11T08:30:10.082 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.29723569100000002 2024-09-11T08:30:10.082 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.082 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.082 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:30:10.082 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.544472+0000", 2024-09-11T08:30:10.082 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00013406 2024-09-11T08:30:10.083 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.083 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.083 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:30:10.083 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.544516+0000", 2024-09-11T08:30:10.083 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.3967000000000001e-05 2024-09-11T08:30:10.083 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:10.083 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:10.083 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:10.083 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.083 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.083 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.25799.0:129087 11.9 11:9bd564df:::benchmark_data_smithi150_187410_object3038:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-09-11T08:30:10.083 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:28:13.243546+0000", 2024-09-11T08:30:10.083 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 116.82681070300001, 2024-09-11T08:30:10.083 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.30518488900000001, 2024-09-11T08:30:10.084 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:30:10.084 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:30:10.084 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:30:10.084 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.25799", 2024-09-11T08:30:10.084 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1403604046", 2024-09-11T08:30:10.084 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 129087 2024-09-11T08:30:10.084 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.084 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:30:10.084 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.084 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:30:10.084 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.243546+0000", 2024-09-11T08:30:10.084 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:30:10.084 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.084 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.085 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:30:10.085 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.243546+0000", 2024-09-11T08:30:10.085 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:30:10.085 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.085 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.085 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:30:10.085 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.243544+0000", 2024-09-11T08:30:10.085 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:30:10.086 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.086 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.086 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:30:10.086 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.243551+0000", 2024-09-11T08:30:10.086 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.9879999999999998e-06 2024-09-11T08:30:10.086 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.086 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.086 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:30:10.086 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.243566+0000", 2024-09-11T08:30:10.086 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4915e-05 2024-09-11T08:30:10.086 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.086 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.087 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:30:10.087 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.243572+0000", 2024-09-11T08:30:10.087 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.8159999999999999e-06 2024-09-11T08:30:10.087 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.087 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.087 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:30:10.087 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.243755+0000", 2024-09-11T08:30:10.087 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000182757 2024-09-11T08:30:10.087 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.087 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.087 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:30:10.087 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.243992+0000", 2024-09-11T08:30:10.087 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000237567 2024-09-11T08:30:10.088 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.088 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.088 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 0,6", 2024-09-11T08:30:10.088 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.244106+0000", 2024-09-11T08:30:10.088 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00011314700000000001 2024-09-11T08:30:10.088 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.088 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.088 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:30:10.088 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.247126+0000", 2024-09-11T08:30:10.088 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0030202720000000001 2024-09-11T08:30:10.088 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.088 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.088 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:30:10.089 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.252860+0000", 2024-09-11T08:30:10.089 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0057337739999999996 2024-09-11T08:30:10.089 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.089 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.089 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:30:10.089 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.548601+0000", 2024-09-11T08:30:10.089 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.29574137299999997 2024-09-11T08:30:10.089 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.089 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.089 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:30:10.089 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.548690+0000", 2024-09-11T08:30:10.089 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.9485000000000001e-05 2024-09-11T08:30:10.090 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.090 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.090 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:30:10.090 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.548731+0000", 2024-09-11T08:30:10.090 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.0874999999999997e-05 2024-09-11T08:30:10.090 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:10.090 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:10.090 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:10.090 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.090 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.090 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.25799.0:129092 11.9 11:9e056507:::benchmark_data_smithi150_187410_object3043:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-09-11T08:30:10.090 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:28:13.246932+0000", 2024-09-11T08:30:10.091 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 116.82342480699999, 2024-09-11T08:30:10.091 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.30972266399999998, 2024-09-11T08:30:10.091 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:30:10.091 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:30:10.091 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:30:10.091 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.25799", 2024-09-11T08:30:10.091 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1403604046", 2024-09-11T08:30:10.091 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 129092 2024-09-11T08:30:10.091 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.091 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:30:10.091 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.092 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:30:10.092 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.246932+0000", 2024-09-11T08:30:10.093 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:30:10.093 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.093 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.093 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:30:10.093 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.246932+0000", 2024-09-11T08:30:10.093 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:30:10.093 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.093 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.093 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:30:10.093 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.246930+0000", 2024-09-11T08:30:10.093 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:30:10.094 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.094 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.094 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:30:10.094 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.246937+0000", 2024-09-11T08:30:10.094 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.4670000000000001e-06 2024-09-11T08:30:10.094 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.094 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.094 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:30:10.094 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.246950+0000", 2024-09-11T08:30:10.094 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3002e-05 2024-09-11T08:30:10.094 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.094 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.094 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:30:10.094 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.246956+0000", 2024-09-11T08:30:10.095 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.9460000000000003e-06 2024-09-11T08:30:10.095 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.095 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.095 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:30:10.095 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.247145+0000", 2024-09-11T08:30:10.095 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000189197 2024-09-11T08:30:10.095 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.095 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.095 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:30:10.095 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.247252+0000", 2024-09-11T08:30:10.095 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000106545 2024-09-11T08:30:10.095 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.095 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.095 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 0,6", 2024-09-11T08:30:10.095 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.247323+0000", 2024-09-11T08:30:10.096 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.1896000000000004e-05 2024-09-11T08:30:10.096 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.096 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.096 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:30:10.096 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.252748+0000", 2024-09-11T08:30:10.096 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0054249779999999996 2024-09-11T08:30:10.096 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.096 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.096 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:30:10.096 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.252969+0000", 2024-09-11T08:30:10.096 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00022018999999999999 2024-09-11T08:30:10.096 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.096 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.096 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:30:10.097 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.556505+0000", 2024-09-11T08:30:10.097 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.303536111 2024-09-11T08:30:10.097 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.097 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.097 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:30:10.097 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.556615+0000", 2024-09-11T08:30:10.097 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000110245 2024-09-11T08:30:10.097 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.097 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.097 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:30:10.097 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.556655+0000", 2024-09-11T08:30:10.097 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.9984000000000002e-05 2024-09-11T08:30:10.097 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:10.098 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:10.098 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:10.098 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.098 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.098 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.25799.0:129096 11.9 11:916ced12:::benchmark_data_smithi150_187410_object3047:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-09-11T08:30:10.098 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:28:13.248102+0000", 2024-09-11T08:30:10.098 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 116.822255394, 2024-09-11T08:30:10.098 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.30873741700000001, 2024-09-11T08:30:10.098 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:30:10.098 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:30:10.098 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:30:10.098 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.25799", 2024-09-11T08:30:10.098 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1403604046", 2024-09-11T08:30:10.098 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 129096 2024-09-11T08:30:10.099 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.099 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:30:10.099 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.099 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:30:10.099 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.248102+0000", 2024-09-11T08:30:10.099 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:30:10.099 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.099 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.099 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:30:10.099 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.248102+0000", 2024-09-11T08:30:10.099 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:30:10.099 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.099 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.099 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:30:10.100 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.248101+0000", 2024-09-11T08:30:10.100 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:30:10.100 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.100 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.100 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:30:10.100 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.248106+0000", 2024-09-11T08:30:10.100 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.3460000000000001e-06 2024-09-11T08:30:10.100 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.100 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.100 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:30:10.100 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.248119+0000", 2024-09-11T08:30:10.100 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2804e-05 2024-09-11T08:30:10.100 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.101 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.101 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:30:10.101 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.248125+0000", 2024-09-11T08:30:10.101 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.0549999999999996e-06 2024-09-11T08:30:10.101 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.101 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.101 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:30:10.101 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.249714+0000", 2024-09-11T08:30:10.101 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.001589481 2024-09-11T08:30:10.101 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.101 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.101 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:30:10.101 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.249862+0000", 2024-09-11T08:30:10.102 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000147226 2024-09-11T08:30:10.102 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.102 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.102 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 0,6", 2024-09-11T08:30:10.102 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.249965+0000", 2024-09-11T08:30:10.102 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000103257 2024-09-11T08:30:10.102 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.102 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.102 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:30:10.102 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.252775+0000", 2024-09-11T08:30:10.102 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.002810206 2024-09-11T08:30:10.102 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.102 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.102 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:30:10.103 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.252991+0000", 2024-09-11T08:30:10.103 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00021607700000000001 2024-09-11T08:30:10.103 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.103 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.103 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:30:10.103 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.556685+0000", 2024-09-11T08:30:10.103 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.303694146 2024-09-11T08:30:10.103 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.103 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.103 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:30:10.103 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.556796+0000", 2024-09-11T08:30:10.103 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000110213 2024-09-11T08:30:10.103 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.103 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.103 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:30:10.103 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.556839+0000", 2024-09-11T08:30:10.104 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.3631000000000003e-05 2024-09-11T08:30:10.104 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:10.104 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:10.104 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:10.104 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.104 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.104 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.25799.0:129100 11.9 11:961a1cb9:::benchmark_data_smithi150_187410_object3051:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-09-11T08:30:10.104 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:28:13.255232+0000", 2024-09-11T08:30:10.104 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 116.815124822, 2024-09-11T08:30:10.104 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.30173964399999997, 2024-09-11T08:30:10.104 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:30:10.104 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:30:10.104 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:30:10.104 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.25799", 2024-09-11T08:30:10.105 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1403604046", 2024-09-11T08:30:10.105 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 129100 2024-09-11T08:30:10.105 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.105 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:30:10.105 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.105 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:30:10.105 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.255232+0000", 2024-09-11T08:30:10.105 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:30:10.105 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.105 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.105 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:30:10.105 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.255232+0000", 2024-09-11T08:30:10.105 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:30:10.105 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.106 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.106 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:30:10.106 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.255231+0000", 2024-09-11T08:30:10.106 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:30:10.106 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.106 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.106 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:30:10.106 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.255239+0000", 2024-09-11T08:30:10.106 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.9030000000000004e-06 2024-09-11T08:30:10.106 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.106 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.107 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:30:10.107 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.255251+0000", 2024-09-11T08:30:10.107 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2794e-05 2024-09-11T08:30:10.107 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.107 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.107 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:30:10.107 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.255258+0000", 2024-09-11T08:30:10.107 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.8530000000000003e-06 2024-09-11T08:30:10.107 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.108 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.108 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:30:10.108 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.255299+0000", 2024-09-11T08:30:10.108 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1140000000000003e-05 2024-09-11T08:30:10.108 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.108 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.108 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:30:10.108 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.255448+0000", 2024-09-11T08:30:10.109 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000148598 2024-09-11T08:30:10.109 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.109 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.109 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 0,6", 2024-09-11T08:30:10.109 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.255569+0000", 2024-09-11T08:30:10.109 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012060100000000001 2024-09-11T08:30:10.109 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.110 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.110 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:30:10.110 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.257620+0000", 2024-09-11T08:30:10.110 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0020513340000000001 2024-09-11T08:30:10.110 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.110 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.110 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:30:10.110 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.257661+0000", 2024-09-11T08:30:10.110 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1477999999999998e-05 2024-09-11T08:30:10.111 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.111 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.111 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:30:10.111 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.556869+0000", 2024-09-11T08:30:10.111 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.29920722100000002 2024-09-11T08:30:10.111 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.111 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.112 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:30:10.112 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.556938+0000", 2024-09-11T08:30:10.112 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.8869999999999996e-05 2024-09-11T08:30:10.112 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.112 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.112 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:30:10.112 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:13.556972+0000", 2024-09-11T08:30:10.112 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.4390000000000001e-05 2024-09-11T08:30:10.112 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:10.113 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:10.113 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:10.113 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.113 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.113 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.25799.0:151907 11.9 11:98d94c34:::benchmark_data_smithi150_187410_object25858:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e422)", 2024-09-11T08:30:10.113 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:28:33.559338+0000", 2024-09-11T08:30:10.113 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 96.511019431999998, 2024-09-11T08:30:10.113 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.301040535, 2024-09-11T08:30:10.114 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:30:10.114 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:30:10.114 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:30:10.114 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.25799", 2024-09-11T08:30:10.114 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1403604046", 2024-09-11T08:30:10.114 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 151907 2024-09-11T08:30:10.114 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.114 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:30:10.115 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.115 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:30:10.115 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.559338+0000", 2024-09-11T08:30:10.115 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:30:10.115 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.115 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.115 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:30:10.115 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.559338+0000", 2024-09-11T08:30:10.116 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:30:10.116 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.116 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.116 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:30:10.116 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.559336+0000", 2024-09-11T08:30:10.116 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999981 2024-09-11T08:30:10.116 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.116 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.116 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:30:10.117 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.559345+0000", 2024-09-11T08:30:10.117 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.0829999999999993e-06 2024-09-11T08:30:10.117 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.117 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.117 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:30:10.117 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.559363+0000", 2024-09-11T08:30:10.117 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.8206000000000002e-05 2024-09-11T08:30:10.117 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.118 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.118 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:30:10.118 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.559370+0000", 2024-09-11T08:30:10.118 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.0910000000000004e-06 2024-09-11T08:30:10.118 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.118 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.118 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:30:10.118 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.559549+0000", 2024-09-11T08:30:10.118 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000179135 2024-09-11T08:30:10.119 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.119 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.119 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:30:10.119 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.559770+0000", 2024-09-11T08:30:10.119 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000220375 2024-09-11T08:30:10.119 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.119 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.119 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 0,6", 2024-09-11T08:30:10.120 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.559880+0000", 2024-09-11T08:30:10.120 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000110312 2024-09-11T08:30:10.120 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.120 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.120 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:30:10.120 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.568494+0000", 2024-09-11T08:30:10.120 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0086134269999999999 2024-09-11T08:30:10.120 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.120 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.120 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:30:10.120 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.574277+0000", 2024-09-11T08:30:10.120 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0057834840000000002 2024-09-11T08:30:10.121 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.121 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.121 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:30:10.121 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.860270+0000", 2024-09-11T08:30:10.121 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.28599298699999998 2024-09-11T08:30:10.121 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.121 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.121 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:30:10.121 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.860329+0000", 2024-09-11T08:30:10.121 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.8615e-05 2024-09-11T08:30:10.121 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.121 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.121 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:30:10.122 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.860378+0000", 2024-09-11T08:30:10.122 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.9602999999999999e-05 2024-09-11T08:30:10.122 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:10.122 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:10.122 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:10.122 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.122 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.122 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.25799.0:151908 11.9 11:9f54dc84:::benchmark_data_smithi150_187410_object25859:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e422)", 2024-09-11T08:30:10.122 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:28:33.559383+0000", 2024-09-11T08:30:10.122 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 96.510974121999993, 2024-09-11T08:30:10.122 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.31059516199999998, 2024-09-11T08:30:10.122 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:30:10.122 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:30:10.122 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:30:10.122 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.25799", 2024-09-11T08:30:10.123 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1403604046", 2024-09-11T08:30:10.123 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 151908 2024-09-11T08:30:10.123 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.123 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:30:10.123 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.123 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:30:10.123 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.559383+0000", 2024-09-11T08:30:10.123 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:30:10.123 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.123 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.123 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:30:10.123 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.559383+0000", 2024-09-11T08:30:10.123 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:30:10.123 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.124 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.124 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:30:10.124 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.559383+0000", 2024-09-11T08:30:10.124 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:30:10.124 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.124 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.124 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:30:10.124 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.559385+0000", 2024-09-11T08:30:10.124 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.9119999999999998e-06 2024-09-11T08:30:10.124 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.124 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.125 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:30:10.125 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.559391+0000", 2024-09-11T08:30:10.125 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.6239999999999999e-06 2024-09-11T08:30:10.125 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.125 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.125 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:30:10.125 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.559395+0000", 2024-09-11T08:30:10.125 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.0810000000000004e-06 2024-09-11T08:30:10.125 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.125 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.125 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:30:10.125 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.562666+0000", 2024-09-11T08:30:10.125 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0032708580000000002 2024-09-11T08:30:10.126 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.126 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.126 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:30:10.126 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.562765+0000", 2024-09-11T08:30:10.126 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.8958999999999999e-05 2024-09-11T08:30:10.126 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:09 smithi150 ceph-mon[29276]: 2.1 scrub starts 2024-09-11T08:30:10.126 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:09 smithi150 ceph-mon[29276]: 2.1 scrub ok 2024-09-11T08:30:10.127 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:09 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:30:10.127 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:09 smithi150 ceph-mon[29276]: pgmap v1037: 148 pgs: 148 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 651 op/s 2024-09-11T08:30:10.127 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:09 smithi150 ceph-mon[39612]: 2.1 scrub starts 2024-09-11T08:30:10.127 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:09 smithi150 ceph-mon[39612]: 2.1 scrub ok 2024-09-11T08:30:10.127 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:09 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:30:10.127 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:09 smithi150 ceph-mon[39612]: pgmap v1037: 148 pgs: 148 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 651 op/s 2024-09-11T08:30:10.128 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.128 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.128 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 0,6", 2024-09-11T08:30:10.128 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.562870+0000", 2024-09-11T08:30:10.128 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000105386 2024-09-11T08:30:10.128 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.128 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.128 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:30:10.128 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.571137+0000", 2024-09-11T08:30:10.129 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0082670560000000001 2024-09-11T08:30:10.129 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.129 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.129 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:30:10.129 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.574311+0000", 2024-09-11T08:30:10.129 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.003173684 2024-09-11T08:30:10.129 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.129 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.129 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:30:10.129 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.869681+0000", 2024-09-11T08:30:10.129 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.29536957899999999 2024-09-11T08:30:10.129 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.129 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.130 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:30:10.130 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.869843+0000", 2024-09-11T08:30:10.130 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00016209899999999999 2024-09-11T08:30:10.130 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.130 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.130 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:30:10.130 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.869978+0000", 2024-09-11T08:30:10.130 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000135427 2024-09-11T08:30:10.130 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:10.130 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:10.130 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:10.130 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.130 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.130 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.25799.0:151909 11.9 11:9509d1d7:::benchmark_data_smithi150_187410_object25860:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e422)", 2024-09-11T08:30:10.130 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:28:33.559407+0000", 2024-09-11T08:30:10.130 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 96.510949717000003, 2024-09-11T08:30:10.131 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.31125763200000001, 2024-09-11T08:30:10.131 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:30:10.131 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:30:10.131 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:30:10.131 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.25799", 2024-09-11T08:30:10.131 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1403604046", 2024-09-11T08:30:10.131 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 151909 2024-09-11T08:30:10.131 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.131 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:30:10.131 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.131 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:30:10.131 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.559407+0000", 2024-09-11T08:30:10.131 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:30:10.131 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.132 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.132 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:30:10.132 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.559407+0000", 2024-09-11T08:30:10.132 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:30:10.132 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.132 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.132 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:30:10.132 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.559407+0000", 2024-09-11T08:30:10.132 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:30:10.132 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.132 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.132 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:30:10.132 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.559410+0000", 2024-09-11T08:30:10.132 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.9179999999999998e-06 2024-09-11T08:30:10.132 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.133 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.133 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:30:10.133 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.559416+0000", 2024-09-11T08:30:10.133 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.7960000000000001e-06 2024-09-11T08:30:10.133 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.133 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.133 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:30:10.133 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.559420+0000", 2024-09-11T08:30:10.133 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.8850000000000001e-06 2024-09-11T08:30:10.133 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.133 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.133 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:30:10.133 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.571159+0000", 2024-09-11T08:30:10.133 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.011739171 2024-09-11T08:30:10.134 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.134 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.134 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:30:10.134 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.571260+0000", 2024-09-11T08:30:10.134 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000101411 2024-09-11T08:30:10.134 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.134 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.134 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 0,6", 2024-09-11T08:30:10.134 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.571372+0000", 2024-09-11T08:30:10.134 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000112312 2024-09-11T08:30:10.134 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.134 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.134 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:30:10.135 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.574342+0000", 2024-09-11T08:30:10.135 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0029693969999999999 2024-09-11T08:30:10.135 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.135 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.135 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:30:10.135 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.574627+0000", 2024-09-11T08:30:10.135 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00028479999999999998 2024-09-11T08:30:10.135 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.135 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.135 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:30:10.135 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.870418+0000", 2024-09-11T08:30:10.135 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.29579151799999998 2024-09-11T08:30:10.135 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.135 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.135 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:30:10.136 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.870536+0000", 2024-09-11T08:30:10.136 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000117893 2024-09-11T08:30:10.136 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.136 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.136 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:30:10.136 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.870665+0000", 2024-09-11T08:30:10.136 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012897700000000001 2024-09-11T08:30:10.136 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:10.136 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:10.136 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:10.136 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.136 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.136 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.25799.0:151914 11.b 11:de436040:::benchmark_data_smithi150_187410_object25865:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e422)", 2024-09-11T08:30:10.136 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:28:33.561846+0000", 2024-09-11T08:30:10.136 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 96.508511049000006, 2024-09-11T08:30:10.137 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.29904313999999999, 2024-09-11T08:30:10.137 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:30:10.137 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:30:10.137 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:30:10.137 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.25799", 2024-09-11T08:30:10.137 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1403604046", 2024-09-11T08:30:10.137 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 151914 2024-09-11T08:30:10.140 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.140 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:30:10.140 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.140 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:30:10.140 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.561846+0000", 2024-09-11T08:30:10.140 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:30:10.140 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.140 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.140 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:30:10.140 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.561846+0000", 2024-09-11T08:30:10.140 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:30:10.141 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.141 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.141 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:30:10.141 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.561845+0000", 2024-09-11T08:30:10.141 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:30:10.141 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.141 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.142 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:30:10.142 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.561851+0000", 2024-09-11T08:30:10.142 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.7989999999999999e-06 2024-09-11T08:30:10.142 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.142 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.143 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:30:10.143 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.561863+0000", 2024-09-11T08:30:10.143 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1717e-05 2024-09-11T08:30:10.143 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.143 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.143 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:30:10.143 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.561868+0000", 2024-09-11T08:30:10.143 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.5380000000000002e-06 2024-09-11T08:30:10.143 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.143 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.143 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:30:10.143 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.561912+0000", 2024-09-11T08:30:10.144 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.4215000000000003e-05 2024-09-11T08:30:10.144 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.144 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.144 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:30:10.144 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.562016+0000", 2024-09-11T08:30:10.144 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000103667 2024-09-11T08:30:10.145 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.145 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.145 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 0,1", 2024-09-11T08:30:10.145 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.562122+0000", 2024-09-11T08:30:10.145 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00010560699999999999 2024-09-11T08:30:10.145 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.145 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.145 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:30:10.145 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.566597+0000", 2024-09-11T08:30:10.145 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.004475283 2024-09-11T08:30:10.145 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.145 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.146 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:30:10.146 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.570652+0000", 2024-09-11T08:30:10.146 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0040545950000000002 2024-09-11T08:30:10.146 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.146 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.146 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:30:10.146 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.860814+0000", 2024-09-11T08:30:10.146 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.29016205499999997 2024-09-11T08:30:10.146 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.146 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.146 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:30:10.146 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.860854+0000", 2024-09-11T08:30:10.146 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.0241000000000003e-05 2024-09-11T08:30:10.147 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.147 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.147 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:30:10.147 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:28:33.860889+0000", 2024-09-11T08:30:10.147 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.5429000000000002e-05 2024-09-11T08:30:10.147 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:10.147 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:10.147 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:10.147 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.147 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.147 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.25799.0:227063 11.9 11:9adeaf2e:::benchmark_data_smithi150_187410_object101014:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e436)", 2024-09-11T08:30:10.147 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:29:40.252549+0000", 2024-09-11T08:30:10.147 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 29.817807979000001, 2024-09-11T08:30:10.147 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.39895291700000002, 2024-09-11T08:30:10.147 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:30:10.148 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:30:10.148 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:30:10.148 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.25799", 2024-09-11T08:30:10.148 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1403604046", 2024-09-11T08:30:10.148 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 227063 2024-09-11T08:30:10.148 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.148 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:30:10.148 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.148 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:30:10.148 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.252549+0000", 2024-09-11T08:30:10.148 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:30:10.148 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.148 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.148 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:30:10.148 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.252549+0000", 2024-09-11T08:30:10.149 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:30:10.149 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.149 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.149 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:30:10.149 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.252548+0000", 2024-09-11T08:30:10.149 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:30:10.149 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.149 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.149 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:30:10.149 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.252554+0000", 2024-09-11T08:30:10.149 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.5790000000000003e-06 2024-09-11T08:30:10.149 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.149 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.149 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:30:10.150 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.252566+0000", 2024-09-11T08:30:10.150 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2241e-05 2024-09-11T08:30:10.150 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.150 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.150 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:30:10.150 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.252572+0000", 2024-09-11T08:30:10.150 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.9970000000000004e-06 2024-09-11T08:30:10.150 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.150 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.150 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:30:10.150 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.252735+0000", 2024-09-11T08:30:10.150 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00016273899999999999 2024-09-11T08:30:10.150 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.150 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.150 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:30:10.151 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.252826+0000", 2024-09-11T08:30:10.151 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.1669000000000006e-05 2024-09-11T08:30:10.151 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.151 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.151 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 0,6", 2024-09-11T08:30:10.151 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.252944+0000", 2024-09-11T08:30:10.151 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00011768 2024-09-11T08:30:10.151 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.151 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.151 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:30:10.151 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.256132+0000", 2024-09-11T08:30:10.151 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0031881420000000001 2024-09-11T08:30:10.151 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.151 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.152 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:30:10.152 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.256310+0000", 2024-09-11T08:30:10.152 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000177543 2024-09-11T08:30:10.152 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.152 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.152 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:30:10.152 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.651416+0000", 2024-09-11T08:30:10.152 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.39510658599999998 2024-09-11T08:30:10.152 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.152 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.152 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:30:10.152 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.651466+0000", 2024-09-11T08:30:10.152 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.024e-05 2024-09-11T08:30:10.153 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.153 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.153 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:30:10.153 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.651502+0000", 2024-09-11T08:30:10.153 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.5710000000000002e-05 2024-09-11T08:30:10.153 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:10.153 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:10.153 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:10.153 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.153 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.153 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.25799.0:227064 11.b 11:de29a234:::benchmark_data_smithi150_187410_object101015:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e436)", 2024-09-11T08:30:10.153 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:29:40.252659+0000", 2024-09-11T08:30:10.153 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 29.817697771999999, 2024-09-11T08:30:10.153 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.39877969899999999, 2024-09-11T08:30:10.153 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:30:10.154 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:30:10.154 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:30:10.154 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.25799", 2024-09-11T08:30:10.154 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1403604046", 2024-09-11T08:30:10.154 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 227064 2024-09-11T08:30:10.154 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.154 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:30:10.154 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.154 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:30:10.154 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.252659+0000", 2024-09-11T08:30:10.154 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:30:10.154 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.154 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.154 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:30:10.154 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.252659+0000", 2024-09-11T08:30:10.154 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:30:10.155 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.155 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.155 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:30:10.155 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.252659+0000", 2024-09-11T08:30:10.155 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:30:10.155 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.155 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.155 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:30:10.155 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.252663+0000", 2024-09-11T08:30:10.155 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1150000000000004e-06 2024-09-11T08:30:10.155 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.155 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.155 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:30:10.155 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.252671+0000", 2024-09-11T08:30:10.155 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.7179999999999996e-06 2024-09-11T08:30:10.156 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.156 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.156 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:30:10.156 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.252675+0000", 2024-09-11T08:30:10.156 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.8760000000000004e-06 2024-09-11T08:30:10.156 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.156 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.156 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:30:10.156 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.252723+0000", 2024-09-11T08:30:10.156 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.7188000000000003e-05 2024-09-11T08:30:10.156 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.156 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.156 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:30:10.156 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.252983+0000", 2024-09-11T08:30:10.156 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00026030500000000001 2024-09-11T08:30:10.157 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.157 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.157 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 0,1", 2024-09-11T08:30:10.157 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.253083+0000", 2024-09-11T08:30:10.157 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000100294 2024-09-11T08:30:10.157 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.157 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.157 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:30:10.157 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.256326+0000", 2024-09-11T08:30:10.157 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0032424440000000001 2024-09-11T08:30:10.157 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.157 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.157 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:30:10.157 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.258827+0000", 2024-09-11T08:30:10.157 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0025011360000000002 2024-09-11T08:30:10.158 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.158 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.158 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:30:10.158 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.651353+0000", 2024-09-11T08:30:10.158 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.39252587300000003 2024-09-11T08:30:10.158 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.158 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.158 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:30:10.158 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.651400+0000", 2024-09-11T08:30:10.158 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.7790999999999997e-05 2024-09-11T08:30:10.158 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.159 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.159 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:30:10.159 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.651439+0000", 2024-09-11T08:30:10.159 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.8621999999999997e-05 2024-09-11T08:30:10.159 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:10.159 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:10.159 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:10.159 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.159 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.159 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.25799.0:227075 11.9 11:933e6058:::benchmark_data_smithi150_187410_object101026:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e436)", 2024-09-11T08:30:10.159 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:29:40.257889+0000", 2024-09-11T08:30:10.159 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 29.812468259999999, 2024-09-11T08:30:10.159 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.40006691100000003, 2024-09-11T08:30:10.159 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:30:10.159 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:30:10.160 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:30:10.160 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.25799", 2024-09-11T08:30:10.160 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1403604046", 2024-09-11T08:30:10.160 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 227075 2024-09-11T08:30:10.160 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.160 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:30:10.160 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.160 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:30:10.160 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.257889+0000", 2024-09-11T08:30:10.160 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:30:10.160 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.160 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.160 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:30:10.160 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.257889+0000", 2024-09-11T08:30:10.160 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:30:10.161 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.161 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.161 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:30:10.161 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.257887+0000", 2024-09-11T08:30:10.161 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:30:10.161 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.161 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.161 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:30:10.161 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.257892+0000", 2024-09-11T08:30:10.161 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.7299999999999996e-06 2024-09-11T08:30:10.161 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.161 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.162 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:30:10.162 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.257901+0000", 2024-09-11T08:30:10.162 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.1379999999999996e-06 2024-09-11T08:30:10.162 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.162 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.162 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:30:10.162 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.257907+0000", 2024-09-11T08:30:10.162 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.8109999999999999e-06 2024-09-11T08:30:10.162 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.162 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.163 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:30:10.163 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.258030+0000", 2024-09-11T08:30:10.163 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012337700000000001 2024-09-11T08:30:10.163 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.163 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.163 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:30:10.163 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.258147+0000", 2024-09-11T08:30:10.163 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00011657 2024-09-11T08:30:10.163 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.163 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.163 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 0,6", 2024-09-11T08:30:10.163 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.258263+0000", 2024-09-11T08:30:10.163 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00011637999999999999 2024-09-11T08:30:10.163 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.164 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.164 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:30:10.164 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.262372+0000", 2024-09-11T08:30:10.164 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0041091169999999998 2024-09-11T08:30:10.164 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.164 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.164 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:30:10.164 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.263089+0000", 2024-09-11T08:30:10.164 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00071676300000000004 2024-09-11T08:30:10.164 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.164 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.164 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:30:10.165 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.657883+0000", 2024-09-11T08:30:10.165 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.394794276 2024-09-11T08:30:10.165 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.165 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.165 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:30:10.165 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.657923+0000", 2024-09-11T08:30:10.165 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.9697999999999998e-05 2024-09-11T08:30:10.165 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.165 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.165 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:30:10.165 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.657956+0000", 2024-09-11T08:30:10.165 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.2719000000000003e-05 2024-09-11T08:30:10.165 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:10.165 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:10.166 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:10.166 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.166 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.166 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.25799.0:227080 11.b 11:d2241989:::benchmark_data_smithi150_187410_object101031:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e436)", 2024-09-11T08:30:10.166 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:29:40.261238+0000", 2024-09-11T08:30:10.166 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 29.809118768000001, 2024-09-11T08:30:10.166 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.39237186400000001, 2024-09-11T08:30:10.166 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:30:10.166 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:30:10.166 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:30:10.166 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.25799", 2024-09-11T08:30:10.166 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1403604046", 2024-09-11T08:30:10.166 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 227080 2024-09-11T08:30:10.166 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.166 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:30:10.166 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.167 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:30:10.167 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.261238+0000", 2024-09-11T08:30:10.167 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:30:10.167 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.167 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.167 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:30:10.167 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.261238+0000", 2024-09-11T08:30:10.167 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:30:10.167 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.167 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.167 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:30:10.167 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.261237+0000", 2024-09-11T08:30:10.167 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:30:10.167 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.167 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.168 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:30:10.168 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.261244+0000", 2024-09-11T08:30:10.168 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.6100000000000002e-06 2024-09-11T08:30:10.168 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.168 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.168 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:30:10.168 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.261256+0000", 2024-09-11T08:30:10.168 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2418999999999999e-05 2024-09-11T08:30:10.168 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.168 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.168 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:30:10.168 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.261262+0000", 2024-09-11T08:30:10.168 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.3679999999999998e-06 2024-09-11T08:30:10.169 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.169 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.169 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:30:10.169 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.261306+0000", 2024-09-11T08:30:10.169 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.3902000000000001e-05 2024-09-11T08:30:10.169 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.169 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.169 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:30:10.169 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.261411+0000", 2024-09-11T08:30:10.169 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000104499 2024-09-11T08:30:10.170 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.170 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.170 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 0,1", 2024-09-11T08:30:10.170 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.261517+0000", 2024-09-11T08:30:10.170 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00010582 2024-09-11T08:30:10.170 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.170 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.170 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:30:10.170 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.264792+0000", 2024-09-11T08:30:10.170 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.003275198 2024-09-11T08:30:10.170 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.170 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.170 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:30:10.170 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.266413+0000", 2024-09-11T08:30:10.171 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.001621068 2024-09-11T08:30:10.171 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.171 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.171 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:30:10.171 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.651482+0000", 2024-09-11T08:30:10.171 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.38506908200000001 2024-09-11T08:30:10.171 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.171 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.171 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:30:10.171 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.653572+0000", 2024-09-11T08:30:10.171 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0020905540000000001 2024-09-11T08:30:10.171 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.171 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.171 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:30:10.172 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.653610+0000", 2024-09-11T08:30:10.172 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.7764e-05 2024-09-11T08:30:10.172 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:10.172 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:10.172 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:10.172 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.172 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.172 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.25799.0:227100 11.8 e436/382 11:170b63e4:::benchmark_data_smithi150_187410_object101051:head v 436'14305, mlcod=436'14303)", 2024-09-11T08:30:10.172 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:29:40.283941+0000", 2024-09-11T08:30:10.172 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 29.786416372000001, 2024-09-11T08:30:10.172 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.37348681700000003, 2024-09-11T08:30:10.172 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:30:10.173 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:30:10.173 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:30:10.173 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.173 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:30:10.173 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.283941+0000", 2024-09-11T08:30:10.173 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:30:10.173 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.173 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.173 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:30:10.173 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.283941+0000", 2024-09-11T08:30:10.173 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:30:10.173 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.173 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.173 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:30:10.174 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.283940+0000", 2024-09-11T08:30:10.174 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:30:10.174 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.174 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.174 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:30:10.174 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.283946+0000", 2024-09-11T08:30:10.174 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.4829999999999996e-06 2024-09-11T08:30:10.174 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.174 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.174 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:30:10.174 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.283958+0000", 2024-09-11T08:30:10.174 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1445e-05 2024-09-11T08:30:10.174 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.174 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.175 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:30:10.175 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.283963+0000", 2024-09-11T08:30:10.175 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.4620000000000004e-06 2024-09-11T08:30:10.175 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.175 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.175 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:30:10.175 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.283995+0000", 2024-09-11T08:30:10.175 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.1170999999999997e-05 2024-09-11T08:30:10.175 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.175 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.175 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:30:10.175 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.284007+0000", 2024-09-11T08:30:10.175 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2328e-05 2024-09-11T08:30:10.176 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.176 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.176 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:30:10.176 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.651529+0000", 2024-09-11T08:30:10.176 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.36752190499999998 2024-09-11T08:30:10.176 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.176 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.176 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:30:10.176 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.657428+0000", 2024-09-11T08:30:10.176 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0058988310000000002 2024-09-11T08:30:10.176 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:10.177 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:10.177 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:10.177 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.177 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.177 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.25799.0:227101 11.8 e436/382 11:11563ca3:::benchmark_data_smithi150_187410_object101052:head v 436'14306, mlcod=436'14303)", 2024-09-11T08:30:10.177 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:29:40.286964+0000", 2024-09-11T08:30:10.177 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 29.783393264000001, 2024-09-11T08:30:10.177 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.38618186700000001, 2024-09-11T08:30:10.177 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:30:10.178 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:30:10.178 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:30:10.178 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.178 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:30:10.178 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.286964+0000", 2024-09-11T08:30:10.178 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:30:10.178 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.178 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.178 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:30:10.179 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.286964+0000", 2024-09-11T08:30:10.179 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:30:10.179 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.179 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.179 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:30:10.180 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.286963+0000", 2024-09-11T08:30:10.180 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:30:10.180 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.180 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.180 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:30:10.180 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.286971+0000", 2024-09-11T08:30:10.180 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.8189999999999992e-06 2024-09-11T08:30:10.180 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.180 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.180 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:30:10.180 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.286981+0000", 2024-09-11T08:30:10.181 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0179e-05 2024-09-11T08:30:10.181 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.181 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.181 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:30:10.181 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.286987+0000", 2024-09-11T08:30:10.181 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.8499999999999999e-06 2024-09-11T08:30:10.181 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.181 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.181 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:30:10.181 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.287032+0000", 2024-09-11T08:30:10.181 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.5633000000000002e-05 2024-09-11T08:30:10.181 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.181 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.182 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:30:10.182 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.287046+0000", 2024-09-11T08:30:10.182 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3134e-05 2024-09-11T08:30:10.182 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.182 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.182 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:30:10.182 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.673107+0000", 2024-09-11T08:30:10.182 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.38606139499999997 2024-09-11T08:30:10.182 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.182 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.182 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:30:10.182 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.673146+0000", 2024-09-11T08:30:10.182 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.8824e-05 2024-09-11T08:30:10.182 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:10.183 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:10.183 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:10.183 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.183 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.183 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop_reply(client.25799.0:227064 11.b e436/382 ondisk, result = 0)", 2024-09-11T08:30:10.183 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:29:40.340345+0000", 2024-09-11T08:30:10.183 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 29.730012552000002, 2024-09-11T08:30:10.183 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.31110248099999999, 2024-09-11T08:30:10.183 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:30:10.183 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "started", 2024-09-11T08:30:10.183 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:30:10.183 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.183 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:30:10.183 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.340345+0000", 2024-09-11T08:30:10.184 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:30:10.184 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.184 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.184 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:30:10.184 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.340345+0000", 2024-09-11T08:30:10.184 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:30:10.184 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.184 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.184 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:30:10.184 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.340343+0000", 2024-09-11T08:30:10.184 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:30:10.184 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.184 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.184 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:30:10.185 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.340351+0000", 2024-09-11T08:30:10.185 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.1200000000000002e-06 2024-09-11T08:30:10.185 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.185 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.185 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:30:10.185 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.650709+0000", 2024-09-11T08:30:10.185 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.31035785100000002 2024-09-11T08:30:10.185 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.185 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.185 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:30:10.186 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.650716+0000", 2024-09-11T08:30:10.186 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.5810000000000002e-06 2024-09-11T08:30:10.186 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.186 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.186 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:30:10.186 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.651336+0000", 2024-09-11T08:30:10.186 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00062078800000000005 2024-09-11T08:30:10.186 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.186 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.187 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:30:10.187 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.651346+0000", 2024-09-11T08:30:10.187 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.7249999999999992e-06 2024-09-11T08:30:10.187 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.187 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.187 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:30:10.187 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.651447+0000", 2024-09-11T08:30:10.187 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000100868 2024-09-11T08:30:10.187 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:10.187 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:10.187 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:10.187 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.187 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.187 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.25799.0:227106 11.0 e437/382 11:0d4d3d36:::benchmark_data_smithi150_187410_object101057:head v 437'13993, mlcod=436'13991)", 2024-09-11T08:30:10.187 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:29:40.344479+0000", 2024-09-11T08:30:10.188 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 29.725878185999999, 2024-09-11T08:30:10.188 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.32869229599999999, 2024-09-11T08:30:10.188 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:30:10.188 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:30:10.188 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:30:10.188 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.188 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:30:10.188 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.344479+0000", 2024-09-11T08:30:10.188 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:30:10.188 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.188 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.188 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:30:10.188 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.344479+0000", 2024-09-11T08:30:10.189 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:30:10.189 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.189 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.189 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:30:10.189 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.344479+0000", 2024-09-11T08:30:10.189 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2024-09-11T08:30:10.189 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.189 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.189 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:30:10.189 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.344485+0000", 2024-09-11T08:30:10.190 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.4609999999999996e-06 2024-09-11T08:30:10.190 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.190 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.190 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:30:10.190 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.651254+0000", 2024-09-11T08:30:10.190 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.306768349 2024-09-11T08:30:10.190 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.190 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.190 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:30:10.190 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.651260+0000", 2024-09-11T08:30:10.190 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.5119999999999997e-06 2024-09-11T08:30:10.190 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.190 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.191 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:30:10.191 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:30:10.192 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:30:10.192 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.655136+0000", 2024-09-11T08:30:10.192 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.003876165 2024-09-11T08:30:10.192 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.192 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.192 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:30:10.192 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.655154+0000", 2024-09-11T08:30:10.192 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.7884999999999999e-05 2024-09-11T08:30:10.193 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.193 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.193 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:30:10.193 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.673152+0000", 2024-09-11T08:30:10.193 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.017997570000000001 2024-09-11T08:30:10.193 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.193 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.193 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:30:10.193 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:40.673171+0000", 2024-09-11T08:30:10.193 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.9621999999999999e-05 2024-09-11T08:30:10.193 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:10.193 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:10.193 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:10.193 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.194 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.194 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.25799.0:235056 11.b 11:db33a4b6:::benchmark_data_smithi150_187410_object109007:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e440)", 2024-09-11T08:30:10.194 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:29:47.561879+0000", 2024-09-11T08:30:10.194 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 22.508478378, 2024-09-11T08:30:10.194 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.34244497600000001, 2024-09-11T08:30:10.194 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:30:10.194 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:30:10.194 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:30:10.194 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.25799", 2024-09-11T08:30:10.194 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1403604046", 2024-09-11T08:30:10.194 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 235056 2024-09-11T08:30:10.194 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.194 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:30:10.194 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.195 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:30:10.195 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:47.561879+0000", 2024-09-11T08:30:10.195 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:30:10.195 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.195 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.195 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:30:10.195 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:47.561879+0000", 2024-09-11T08:30:10.195 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:30:10.195 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.195 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.195 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:30:10.195 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:47.561878+0000", 2024-09-11T08:30:10.195 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:30:10.195 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.196 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.196 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:30:10.196 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:47.561884+0000", 2024-09-11T08:30:10.196 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.9120000000000003e-06 2024-09-11T08:30:10.196 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.196 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.196 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:30:10.196 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:47.561894+0000", 2024-09-11T08:30:10.196 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.061e-05 2024-09-11T08:30:10.196 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.196 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.196 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:30:10.196 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:47.561901+0000", 2024-09-11T08:30:10.197 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.3960000000000003e-06 2024-09-11T08:30:10.197 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.197 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.197 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:30:10.197 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:47.569402+0000", 2024-09-11T08:30:10.197 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.007501621 2024-09-11T08:30:10.197 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.197 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.197 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:30:10.197 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:47.569519+0000", 2024-09-11T08:30:10.197 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00011636799999999999 2024-09-11T08:30:10.197 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.197 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.197 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 0,1", 2024-09-11T08:30:10.198 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:47.569834+0000", 2024-09-11T08:30:10.198 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00031521300000000001 2024-09-11T08:30:10.198 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.198 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.198 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:30:10.198 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:47.575276+0000", 2024-09-11T08:30:10.198 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0054423199999999996 2024-09-11T08:30:10.198 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.198 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.198 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:30:10.198 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:47.578119+0000", 2024-09-11T08:30:10.198 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0028431620000000002 2024-09-11T08:30:10.198 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.198 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.199 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:30:10.199 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:47.904183+0000", 2024-09-11T08:30:10.199 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.32606311199999999 2024-09-11T08:30:10.199 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.199 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.199 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:30:10.199 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:47.904248+0000", 2024-09-11T08:30:10.199 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.5464999999999997e-05 2024-09-11T08:30:10.199 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.199 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.199 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:30:10.199 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:47.904324+0000", 2024-09-11T08:30:10.199 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.5758000000000002e-05 2024-09-11T08:30:10.200 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:10.200 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:10.200 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:10.200 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.200 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.200 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.25799.0:235060 11.b 11:d204c740:::benchmark_data_smithi150_187410_object109011:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e440)", 2024-09-11T08:30:10.200 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:29:47.565817+0000", 2024-09-11T08:30:10.200 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 22.504540474999999, 2024-09-11T08:30:10.200 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.33870478399999998, 2024-09-11T08:30:10.200 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:30:10.200 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:30:10.200 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:30:10.200 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.25799", 2024-09-11T08:30:10.200 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1403604046", 2024-09-11T08:30:10.201 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 235060 2024-09-11T08:30:10.201 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.201 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:30:10.201 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.201 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:30:10.201 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:47.565817+0000", 2024-09-11T08:30:10.201 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:30:10.201 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.201 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.201 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:30:10.201 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:47.565817+0000", 2024-09-11T08:30:10.201 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:30:10.201 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.201 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.201 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:30:10.202 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:47.565816+0000", 2024-09-11T08:30:10.202 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.999999 2024-09-11T08:30:10.202 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.202 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.202 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:30:10.202 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:47.565822+0000", 2024-09-11T08:30:10.202 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.1659999999999999e-06 2024-09-11T08:30:10.202 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.202 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.202 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:30:10.202 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:47.565831+0000", 2024-09-11T08:30:10.202 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.6840000000000008e-06 2024-09-11T08:30:10.202 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.202 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.203 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:30:10.203 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:47.565837+0000", 2024-09-11T08:30:10.203 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.4020000000000001e-06 2024-09-11T08:30:10.203 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.203 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.203 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:30:10.203 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:47.572382+0000", 2024-09-11T08:30:10.203 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0065453869999999997 2024-09-11T08:30:10.203 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.203 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.203 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:30:10.203 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:47.572498+0000", 2024-09-11T08:30:10.203 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000115457 2024-09-11T08:30:10.204 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.204 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.204 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 0,1", 2024-09-11T08:30:10.204 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:47.572614+0000", 2024-09-11T08:30:10.204 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00011618 2024-09-11T08:30:10.204 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.204 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.204 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:30:10.204 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:47.575283+0000", 2024-09-11T08:30:10.204 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0026689639999999998 2024-09-11T08:30:10.204 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.204 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.204 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:30:10.204 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:47.580356+0000", 2024-09-11T08:30:10.204 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0050735789999999999 2024-09-11T08:30:10.205 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.205 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.205 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:30:10.205 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:47.904409+0000", 2024-09-11T08:30:10.205 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.32405298300000002 2024-09-11T08:30:10.205 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.205 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.205 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:30:10.205 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:47.904486+0000", 2024-09-11T08:30:10.205 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.6780999999999996e-05 2024-09-11T08:30:10.205 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.205 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.205 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:30:10.206 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:47.904521+0000", 2024-09-11T08:30:10.206 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.5253999999999998e-05 2024-09-11T08:30:10.206 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:10.206 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:10.206 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:10.206 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.206 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.206 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.25799.0:235061 11.b 11:d72d7cb2:::benchmark_data_smithi150_187410_object109012:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e440)", 2024-09-11T08:30:10.206 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-09-11T08:29:47.568908+0000", 2024-09-11T08:30:10.206 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 22.501449090000001, 2024-09-11T08:30:10.206 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.34247831899999998, 2024-09-11T08:30:10.206 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-09-11T08:30:10.206 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T08:30:10.207 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-09-11T08:30:10.207 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.25799", 2024-09-11T08:30:10.207 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.150:0/1403604046", 2024-09-11T08:30:10.207 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 235061 2024-09-11T08:30:10.207 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.207 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-09-11T08:30:10.207 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.207 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-09-11T08:30:10.207 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:47.568908+0000", 2024-09-11T08:30:10.207 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:30:10.207 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.207 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.207 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-09-11T08:30:10.207 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:47.568908+0000", 2024-09-11T08:30:10.207 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-09-11T08:30:10.208 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.208 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.208 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-09-11T08:30:10.208 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:47.568906+0000", 2024-09-11T08:30:10.208 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999986 2024-09-11T08:30:10.208 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.208 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.208 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-09-11T08:30:10.208 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:47.568914+0000", 2024-09-11T08:30:10.208 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.9920000000000007e-06 2024-09-11T08:30:10.208 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.208 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.208 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-09-11T08:30:10.208 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:47.568931+0000", 2024-09-11T08:30:10.209 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.6167e-05 2024-09-11T08:30:10.209 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.209 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.209 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-09-11T08:30:10.209 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:47.568938+0000", 2024-09-11T08:30:10.209 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.3329999999999999e-06 2024-09-11T08:30:10.209 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.209 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.209 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-09-11T08:30:10.209 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:47.575301+0000", 2024-09-11T08:30:10.209 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0063635139999999998 2024-09-11T08:30:10.209 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.209 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.209 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-09-11T08:30:10.210 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:47.575417+0000", 2024-09-11T08:30:10.210 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00011547 2024-09-11T08:30:10.210 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.210 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.210 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for subops from 0,1", 2024-09-11T08:30:10.210 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:47.575523+0000", 2024-09-11T08:30:10.210 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00010558200000000001 2024-09-11T08:30:10.210 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.210 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.210 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2024-09-11T08:30:10.210 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:47.579192+0000", 2024-09-11T08:30:10.210 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.003668962 2024-09-11T08:30:10.210 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.210 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.211 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:30:10.211 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:47.580487+0000", 2024-09-11T08:30:10.211 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.001295563 2024-09-11T08:30:10.211 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.211 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.211 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2024-09-11T08:30:10.211 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:47.911308+0000", 2024-09-11T08:30:10.211 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.33082060699999999 2024-09-11T08:30:10.211 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.211 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.211 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-09-11T08:30:10.211 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:47.911349+0000", 2024-09-11T08:30:10.211 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1038000000000002e-05 2024-09-11T08:30:10.211 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:10.212 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:10.212 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-09-11T08:30:10.212 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-09-11T08:29:47.911386+0000", 2024-09-11T08:30:10.212 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.7660000000000002e-05 2024-09-11T08:30:10.212 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:10.212 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:10.212 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:10.212 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:10.212 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:10.212 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-11T08:30:10.212 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:30:10.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:10.235 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:10.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:30:10.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:09 smithi181 ceph-mon[32701]: 2.1 scrub starts 2024-09-11T08:30:10.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:09 smithi181 ceph-mon[32701]: 2.1 scrub ok 2024-09-11T08:30:10.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:09 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:30:10.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:09 smithi181 ceph-mon[32701]: pgmap v1037: 148 pgs: 148 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 651 op/s 2024-09-11T08:30:10.337 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:30:10.358 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:10.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:10.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:30:10.460 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:30:10.481 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:10.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:10.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:30:10.583 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:30:10.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:10.611 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:10.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:30:10.712 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:30:10.738 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:10.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:10.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:30:10.837 INFO:teuthology.orchestra.run.smithi150.stderr:nodeep-scrub is unset 2024-09-11T08:30:10.839 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:30:10.864 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:10.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:10.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:30:10.966 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:30:10.990 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:10.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:10.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:30:11.092 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:30:11.115 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:11.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:11.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:30:11.217 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:30:11.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:10 smithi150 ceph-mon[29276]: 4.16 scrub starts 2024-09-11T08:30:11.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:10 smithi150 ceph-mon[29276]: 4.16 scrub ok 2024-09-11T08:30:11.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:10 smithi150 ceph-mon[29276]: 5.3 scrub starts 2024-09-11T08:30:11.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:10 smithi150 ceph-mon[29276]: 5.3 scrub ok 2024-09-11T08:30:11.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:10 smithi150 ceph-mon[29276]: 5.a scrub starts 2024-09-11T08:30:11.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:10 smithi150 ceph-mon[29276]: 5.a scrub ok 2024-09-11T08:30:11.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:10 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1029783935' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:30:11.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:10 smithi150 ceph-mon[29276]: 4.2 scrub starts 2024-09-11T08:30:11.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:10 smithi150 ceph-mon[29276]: 4.2 scrub ok 2024-09-11T08:30:11.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:10 smithi150 ceph-mon[39612]: 4.16 scrub starts 2024-09-11T08:30:11.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:10 smithi150 ceph-mon[39612]: 4.16 scrub ok 2024-09-11T08:30:11.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:10 smithi150 ceph-mon[39612]: 5.3 scrub starts 2024-09-11T08:30:11.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:10 smithi150 ceph-mon[39612]: 5.3 scrub ok 2024-09-11T08:30:11.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:10 smithi150 ceph-mon[39612]: 5.a scrub starts 2024-09-11T08:30:11.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:10 smithi150 ceph-mon[39612]: 5.a scrub ok 2024-09-11T08:30:11.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:10 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1029783935' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:30:11.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:10 smithi150 ceph-mon[39612]: 4.2 scrub starts 2024-09-11T08:30:11.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:10 smithi150 ceph-mon[39612]: 4.2 scrub ok 2024-09-11T08:30:11.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:11.241 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:11.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:30:11.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:10 smithi181 ceph-mon[32701]: 4.16 scrub starts 2024-09-11T08:30:11.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:10 smithi181 ceph-mon[32701]: 4.16 scrub ok 2024-09-11T08:30:11.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:10 smithi181 ceph-mon[32701]: 5.3 scrub starts 2024-09-11T08:30:11.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:10 smithi181 ceph-mon[32701]: 5.3 scrub ok 2024-09-11T08:30:11.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:10 smithi181 ceph-mon[32701]: 5.a scrub starts 2024-09-11T08:30:11.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:10 smithi181 ceph-mon[32701]: 5.a scrub ok 2024-09-11T08:30:11.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:10 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1029783935' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:30:11.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:10 smithi181 ceph-mon[32701]: 4.2 scrub starts 2024-09-11T08:30:11.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:10 smithi181 ceph-mon[32701]: 4.2 scrub ok 2024-09-11T08:30:11.342 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:30:11.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:11.369 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:11.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:30:11.470 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:30:11.493 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:11.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:11.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:30:11.595 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:30:11.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:11.617 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:11.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:30:11.718 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:30:11.745 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:11.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:11.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:30:11.846 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:30:11.872 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:11.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:11.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:30:11.909 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:11 smithi150 ceph-mon[29276]: 3.12 scrub starts 2024-09-11T08:30:11.909 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:11 smithi150 ceph-mon[29276]: 3.12 scrub ok 2024-09-11T08:30:11.910 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:11 smithi150 ceph-mon[29276]: 3.d scrub starts 2024-09-11T08:30:11.910 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:11 smithi150 ceph-mon[29276]: 3.d scrub ok 2024-09-11T08:30:11.910 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:11 smithi150 ceph-mon[29276]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:30:11.910 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:11 smithi150 ceph-mon[29276]: Cluster is now healthy 2024-09-11T08:30:11.910 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:11 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1029783935' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:30:11.910 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:11 smithi150 ceph-mon[29276]: osdmap e447: 8 total, 8 up, 8 in 2024-09-11T08:30:11.910 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:11 smithi150 ceph-mon[29276]: pgmap v1039: 148 pgs: 148 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 468 op/s 2024-09-11T08:30:11.910 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:11 smithi150 ceph-mon[39612]: 3.12 scrub starts 2024-09-11T08:30:11.910 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:11 smithi150 ceph-mon[39612]: 3.12 scrub ok 2024-09-11T08:30:11.910 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:11 smithi150 ceph-mon[39612]: 3.d scrub starts 2024-09-11T08:30:11.910 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:11 smithi150 ceph-mon[39612]: 3.d scrub ok 2024-09-11T08:30:11.910 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:11 smithi150 ceph-mon[39612]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:30:11.911 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:11 smithi150 ceph-mon[39612]: Cluster is now healthy 2024-09-11T08:30:11.911 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:11 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1029783935' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:30:11.911 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:11 smithi150 ceph-mon[39612]: osdmap e447: 8 total, 8 up, 8 in 2024-09-11T08:30:11.911 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:11 smithi150 ceph-mon[39612]: pgmap v1039: 148 pgs: 148 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 468 op/s 2024-09-11T08:30:11.974 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:30:11.996 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:11.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:11.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:30:12.098 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:30:12.124 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:12.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:12.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:30:12.226 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:30:12.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:12.249 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:12.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:30:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:11 smithi181 ceph-mon[32701]: 3.12 scrub starts 2024-09-11T08:30:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:11 smithi181 ceph-mon[32701]: 3.12 scrub ok 2024-09-11T08:30:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:11 smithi181 ceph-mon[32701]: 3.d scrub starts 2024-09-11T08:30:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:11 smithi181 ceph-mon[32701]: 3.d scrub ok 2024-09-11T08:30:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:11 smithi181 ceph-mon[32701]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T08:30:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:11 smithi181 ceph-mon[32701]: Cluster is now healthy 2024-09-11T08:30:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:11 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1029783935' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:30:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:11 smithi181 ceph-mon[32701]: osdmap e447: 8 total, 8 up, 8 in 2024-09-11T08:30:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:11 smithi181 ceph-mon[32701]: pgmap v1039: 148 pgs: 148 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 468 op/s 2024-09-11T08:30:12.350 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:30:12.377 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:12.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:12.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:30:12.478 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:30:12.503 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:12.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:12.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:30:12.550 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-11T08:30:12.605 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:30:12.628 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:12.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:12.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:30:12.730 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:30:12.751 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:12.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:12.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:30:12.807 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:30:12.843 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:12 smithi150 ceph-mon[39612]: 3.17 scrub starts 2024-09-11T08:30:12.844 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:12 smithi150 ceph-mon[39612]: 3.17 scrub ok 2024-09-11T08:30:12.844 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:12 smithi150 ceph-mon[39612]: 4.13 scrub starts 2024-09-11T08:30:12.844 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:12 smithi150 ceph-mon[39612]: 4.13 scrub ok 2024-09-11T08:30:12.844 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:12 smithi150 ceph-mon[39612]: 4.8 scrub starts 2024-09-11T08:30:12.844 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:12 smithi150 ceph-mon[39612]: 4.8 scrub ok 2024-09-11T08:30:12.853 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:30:12.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:12.876 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:12.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:30:12.977 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:30:12.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:12.998 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:12.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:30:13.099 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:30:13.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:13.122 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:13.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:30:13.155 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:12 smithi150 ceph-mon[29276]: 3.17 scrub starts 2024-09-11T08:30:13.155 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:12 smithi150 ceph-mon[29276]: 3.17 scrub ok 2024-09-11T08:30:13.155 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:12 smithi150 ceph-mon[29276]: 4.13 scrub starts 2024-09-11T08:30:13.155 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:12 smithi150 ceph-mon[29276]: 4.13 scrub ok 2024-09-11T08:30:13.155 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:12 smithi150 ceph-mon[29276]: 4.8 scrub starts 2024-09-11T08:30:13.156 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:12 smithi150 ceph-mon[29276]: 4.8 scrub ok 2024-09-11T08:30:13.178 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:12 smithi181 ceph-mon[32701]: 3.17 scrub starts 2024-09-11T08:30:13.178 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:12 smithi181 ceph-mon[32701]: 3.17 scrub ok 2024-09-11T08:30:13.178 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:12 smithi181 ceph-mon[32701]: 4.13 scrub starts 2024-09-11T08:30:13.178 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:12 smithi181 ceph-mon[32701]: 4.13 scrub ok 2024-09-11T08:30:13.178 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:12 smithi181 ceph-mon[32701]: 4.8 scrub starts 2024-09-11T08:30:13.179 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:12 smithi181 ceph-mon[32701]: 4.8 scrub ok 2024-09-11T08:30:13.223 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:30:13.245 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:13.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:13.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:30:13.347 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:30:13.368 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:13.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:13.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:30:13.470 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:30:13.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:13.493 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:13.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:30:13.594 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:30:13.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:13.618 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:13.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:30:13.719 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:30:13.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:13.743 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:13.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:30:13.845 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:30:13.869 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:13.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:13.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:30:13.971 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:30:13.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:13.994 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:13.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:30:14.095 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:30:14.117 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:14.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:14.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:30:14.179 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:13 smithi150 ceph-mon[29276]: 5.16 scrub starts 2024-09-11T08:30:14.179 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:13 smithi150 ceph-mon[29276]: 5.16 scrub ok 2024-09-11T08:30:14.179 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:13 smithi150 ceph-mon[29276]: 3.1c scrub starts 2024-09-11T08:30:14.179 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:13 smithi150 ceph-mon[29276]: 3.1c scrub ok 2024-09-11T08:30:14.179 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:13 smithi150 ceph-mon[29276]: pgmap v1040: 148 pgs: 148 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-11T08:30:14.180 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:13 smithi150 ceph-mon[39612]: 5.16 scrub starts 2024-09-11T08:30:14.180 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:13 smithi150 ceph-mon[39612]: 5.16 scrub ok 2024-09-11T08:30:14.180 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:13 smithi150 ceph-mon[39612]: 3.1c scrub starts 2024-09-11T08:30:14.180 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:13 smithi150 ceph-mon[39612]: 3.1c scrub ok 2024-09-11T08:30:14.180 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:13 smithi150 ceph-mon[39612]: pgmap v1040: 148 pgs: 148 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-11T08:30:14.191 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:13 smithi181 ceph-mon[32701]: 5.16 scrub starts 2024-09-11T08:30:14.191 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:13 smithi181 ceph-mon[32701]: 5.16 scrub ok 2024-09-11T08:30:14.191 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:13 smithi181 ceph-mon[32701]: 3.1c scrub starts 2024-09-11T08:30:14.191 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:13 smithi181 ceph-mon[32701]: 3.1c scrub ok 2024-09-11T08:30:14.191 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:13 smithi181 ceph-mon[32701]: pgmap v1040: 148 pgs: 148 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-11T08:30:14.220 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:30:14.242 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:14.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:14.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:30:14.344 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:30:14.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:14.365 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:14.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:30:14.466 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:30:14.490 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:14.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:14.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:30:14.586 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-11T08:30:14.586 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-11T08:30:14.586 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-09-11T08:30:14.586 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-11T08:30:14.592 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:30:14.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:14.616 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:14.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:30:14.717 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:30:14.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:14.740 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:14.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:30:14.841 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:30:14.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:14.864 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:14.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:30:14.948 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:14 smithi150 ceph-mon[29276]: 4.0 scrub starts 2024-09-11T08:30:14.948 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:14 smithi150 ceph-mon[29276]: 4.0 scrub ok 2024-09-11T08:30:14.948 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:14 smithi150 ceph-mon[39612]: 4.0 scrub starts 2024-09-11T08:30:14.948 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:14 smithi150 ceph-mon[39612]: 4.0 scrub ok 2024-09-11T08:30:14.965 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:30:14.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:14.987 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:14.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:30:15.088 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:30:15.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:15.111 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:15.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:30:15.144 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:14 smithi181 ceph-mon[32701]: 4.0 scrub starts 2024-09-11T08:30:15.144 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:14 smithi181 ceph-mon[32701]: 4.0 scrub ok 2024-09-11T08:30:15.212 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:30:15.214 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-11T08:30:15.234 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:15.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:15.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:30:15.335 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:30:15.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:15.357 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:15.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:30:15.459 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:30:15.467 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:30:15.481 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:15.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:15.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:30:15.493 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:30:15 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:30:15] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:30:15.583 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:30:15.605 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:15.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:15.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:30:15.707 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:30:15.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:15.729 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:15.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:30:15.830 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:30:15.853 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:15.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:15.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:30:15.955 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:30:15.961 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:15 smithi150 ceph-mon[39612]: 6.a scrub starts 2024-09-11T08:30:15.961 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:15 smithi150 ceph-mon[39612]: 6.a scrub ok 2024-09-11T08:30:15.961 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:15 smithi150 ceph-mon[39612]: 6.5 scrub starts 2024-09-11T08:30:15.961 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:15 smithi150 ceph-mon[39612]: 6.5 scrub ok 2024-09-11T08:30:15.961 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:15 smithi150 ceph-mon[39612]: 5.15 scrub starts 2024-09-11T08:30:15.961 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:15 smithi150 ceph-mon[39612]: 5.15 scrub ok 2024-09-11T08:30:15.961 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:15 smithi150 ceph-mon[39612]: pgmap v1041: 148 pgs: 148 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-11T08:30:15.961 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:15 smithi150 ceph-mon[29276]: 6.a scrub starts 2024-09-11T08:30:15.962 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:15 smithi150 ceph-mon[29276]: 6.a scrub ok 2024-09-11T08:30:15.962 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:15 smithi150 ceph-mon[29276]: 6.5 scrub starts 2024-09-11T08:30:15.962 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:15 smithi150 ceph-mon[29276]: 6.5 scrub ok 2024-09-11T08:30:15.962 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:15 smithi150 ceph-mon[29276]: 5.15 scrub starts 2024-09-11T08:30:15.962 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:15 smithi150 ceph-mon[29276]: 5.15 scrub ok 2024-09-11T08:30:15.962 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:15 smithi150 ceph-mon[29276]: pgmap v1041: 148 pgs: 148 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-11T08:30:15.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:15.979 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:15.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:30:16.003 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:30:16.006 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-09-11T08:30:16.081 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:30:16.102 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:15 smithi181 ceph-mon[32701]: 6.a scrub starts 2024-09-11T08:30:16.102 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:15 smithi181 ceph-mon[32701]: 6.a scrub ok 2024-09-11T08:30:16.103 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:15 smithi181 ceph-mon[32701]: 6.5 scrub starts 2024-09-11T08:30:16.103 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:15 smithi181 ceph-mon[32701]: 6.5 scrub ok 2024-09-11T08:30:16.103 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:15 smithi181 ceph-mon[32701]: 5.15 scrub starts 2024-09-11T08:30:16.103 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:15 smithi181 ceph-mon[32701]: 5.15 scrub ok 2024-09-11T08:30:16.103 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:15 smithi181 ceph-mon[32701]: pgmap v1041: 148 pgs: 148 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-11T08:30:16.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:16.104 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:16.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:30:16.205 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:30:16.226 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:16.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:16.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:30:16.328 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7 2024-09-11T08:30:16.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:16.353 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:16.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.7' 2024-09-11T08:30:16.455 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:30:16.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:16.477 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:16.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:30:16.578 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:30:16.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:16.600 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:16.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:30:16.658 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T08:30:16.701 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:30:16.723 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:16.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:16.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:30:16.816 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":1041,"stamp":"2024-09-11T08:30:15.239645+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596188,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":2618,"num_read_kb":2477,"num_write":253209,"num_write_kb":8068616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":208,"num_bytes_recovered":1776519,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88958,"ondisk_log_size":88958,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6656040,"kb_used_data":7428,"kb_used_omap":22,"kb_used_meta":6647657,"kb_avail":743305176,"statfs":{"total":767960285184,"available":761144500224,"internally_reserved":0,"allocated":7606272,"data_stored":4717564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22853,"internal_metadata":6807201467},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4915200,"num_objects":-75,"num_object_clones":0,"num_object_copies":-225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-75,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":75,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.423973"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":513,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.307529+0000","last_change":"2024-09-11T08:28:03.807764+0000","last_active":"2024-09-11T08:30:07.307529+0000","last_peered":"2024-09-11T08:30:07.307529+0000","last_clean":"2024-09-11T08:30:07.307529+0000","last_became_active":"2024-09-11T08:24:28.358401+0000","last_became_peered":"2024-09-11T08:24:28.358401+0000","last_unstale":"2024-09-11T08:30:07.307529+0000","last_undegraded":"2024-09-11T08:30:07.307529+0000","last_fullsized":"2024-09-11T08:30:07.307529+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:28:03.807673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:28:03.807673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036189030000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"405'25","reported_seq":536,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:06.818317+0000","last_change":"2024-09-11T08:29:59.282881+0000","last_active":"2024-09-11T08:30:06.818317+0000","last_peered":"2024-09-11T08:30:06.818317+0000","last_clean":"2024-09-11T08:30:06.818317+0000","last_became_active":"2024-09-11T08:07:32.048048+0000","last_became_peered":"2024-09-11T08:07:32.048048+0000","last_unstale":"2024-09-11T08:30:06.818317+0000","last_undegraded":"2024-09-11T08:30:06.818317+0000","last_fullsized":"2024-09-11T08:30:06.818317+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'25","last_scrub_stamp":"2024-09-11T08:29:59.282798+0000","last_deep_scrub":"312'19","last_deep_scrub_stamp":"2024-09-11T08:21:59.122122+0000","last_clean_scrub_stamp":"2024-09-11T08:29:59.282798+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:09.025796+0000","scrub_duration":0.012064749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":559,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.080810+0000","last_change":"2024-09-11T08:27:03.997743+0000","last_active":"2024-09-11T08:30:07.080810+0000","last_peered":"2024-09-11T08:30:07.080810+0000","last_clean":"2024-09-11T08:30:07.080810+0000","last_became_active":"2024-09-11T08:24:28.372160+0000","last_became_peered":"2024-09-11T08:24:28.372160+0000","last_unstale":"2024-09-11T08:30:07.080810+0000","last_undegraded":"2024-09-11T08:30:07.080810+0000","last_fullsized":"2024-09-11T08:30:07.080810+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:03.997657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:59.531202+0000","last_clean_scrub_stamp":"2024-09-11T08:27:03.997657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035479840000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":417,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.080737+0000","last_change":"2024-09-11T08:27:06.987974+0000","last_active":"2024-09-11T08:30:07.080737+0000","last_peered":"2024-09-11T08:30:07.080737+0000","last_clean":"2024-09-11T08:30:07.080737+0000","last_became_active":"2024-09-11T08:24:28.354834+0000","last_became_peered":"2024-09-11T08:24:28.354834+0000","last_unstale":"2024-09-11T08:30:07.080737+0000","last_undegraded":"2024-09-11T08:30:07.080737+0000","last_fullsized":"2024-09-11T08:30:07.080737+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:06.987888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:02.470649+0000","last_clean_scrub_stamp":"2024-09-11T08:27:06.987888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030657430000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":508,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:06.822287+0000","last_change":"2024-09-11T08:29:03.775193+0000","last_active":"2024-09-11T08:30:06.822287+0000","last_peered":"2024-09-11T08:30:06.822287+0000","last_clean":"2024-09-11T08:30:06.822287+0000","last_became_active":"2024-09-11T08:15:56.521104+0000","last_became_peered":"2024-09-11T08:15:56.521104+0000","last_unstale":"2024-09-11T08:30:06.822287+0000","last_undegraded":"2024-09-11T08:30:06.822287+0000","last_fullsized":"2024-09-11T08:30:06.822287+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:03.775098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:29:03.775098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:30.682191+0000","scrub_duration":0.0091193990000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":291,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.475727+0000","last_change":"2024-09-11T08:30:00.543330+0000","last_active":"2024-09-11T08:30:07.475727+0000","last_peered":"2024-09-11T08:30:07.475727+0000","last_clean":"2024-09-11T08:30:07.475727+0000","last_became_active":"2024-09-11T08:17:37.550048+0000","last_became_peered":"2024-09-11T08:17:37.550048+0000","last_unstale":"2024-09-11T08:30:07.475727+0000","last_undegraded":"2024-09-11T08:30:07.475727+0000","last_fullsized":"2024-09-11T08:30:07.475727+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:00.543254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:30:00.543254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:27.529390+0000","scrub_duration":0.0048128770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"405'46","reported_seq":469,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.258724+0000","last_change":"2024-09-11T08:29:17.630424+0000","last_active":"2024-09-11T08:30:07.258724+0000","last_peered":"2024-09-11T08:30:07.258724+0000","last_clean":"2024-09-11T08:30:07.258724+0000","last_became_active":"2024-09-11T08:17:37.553866+0000","last_became_peered":"2024-09-11T08:17:37.553866+0000","last_unstale":"2024-09-11T08:30:07.258724+0000","last_undegraded":"2024-09-11T08:30:07.258724+0000","last_fullsized":"2024-09-11T08:30:07.258724+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'46","last_scrub_stamp":"2024-09-11T08:29:17.630303+0000","last_deep_scrub":"145'22","last_deep_scrub_stamp":"2024-09-11T08:15:57.478870+0000","last_clean_scrub_stamp":"2024-09-11T08:29:17.630303+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:23.815610+0000","scrub_duration":0.014805099,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":152,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.475190+0000","last_change":"2024-09-11T08:29:23.277986+0000","last_active":"2024-09-11T08:30:07.475190+0000","last_peered":"2024-09-11T08:30:07.475190+0000","last_clean":"2024-09-11T08:30:07.475190+0000","last_became_active":"2024-09-11T08:23:22.687041+0000","last_became_peered":"2024-09-11T08:23:22.687041+0000","last_unstale":"2024-09-11T08:30:07.475190+0000","last_undegraded":"2024-09-11T08:30:07.475190+0000","last_fullsized":"2024-09-11T08:30:07.475190+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:23.277854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:18:36.998503+0000","last_clean_scrub_stamp":"2024-09-11T08:29:23.277854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:37.611048+0000","scrub_duration":0.017784378,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":183,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.476389+0000","last_change":"2024-09-11T08:29:12.315549+0000","last_active":"2024-09-11T08:30:07.476389+0000","last_peered":"2024-09-11T08:30:07.476389+0000","last_clean":"2024-09-11T08:30:07.476389+0000","last_became_active":"2024-09-11T08:24:28.360021+0000","last_became_peered":"2024-09-11T08:24:28.360021+0000","last_unstale":"2024-09-11T08:30:07.476389+0000","last_undegraded":"2024-09-11T08:30:07.476389+0000","last_fullsized":"2024-09-11T08:30:07.476389+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:12.315453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:54.878409+0000","last_clean_scrub_stamp":"2024-09-11T08:29:12.315453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:15.538317+0000","scrub_duration":0.00128605,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"405'21","reported_seq":401,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.307084+0000","last_change":"2024-09-11T08:29:19.176724+0000","last_active":"2024-09-11T08:30:07.307084+0000","last_peered":"2024-09-11T08:30:07.307084+0000","last_clean":"2024-09-11T08:30:07.307084+0000","last_became_active":"2024-09-11T08:23:22.687182+0000","last_became_peered":"2024-09-11T08:23:22.687182+0000","last_unstale":"2024-09-11T08:30:07.307084+0000","last_undegraded":"2024-09-11T08:30:07.307084+0000","last_fullsized":"2024-09-11T08:30:07.307084+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'21","last_scrub_stamp":"2024-09-11T08:29:19.176584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:29:19.176584+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:45.130745+0000","scrub_duration":0.022662364000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":398,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.307250+0000","last_change":"2024-09-11T08:29:04.968206+0000","last_active":"2024-09-11T08:30:07.307250+0000","last_peered":"2024-09-11T08:30:07.307250+0000","last_clean":"2024-09-11T08:30:07.307250+0000","last_became_active":"2024-09-11T08:24:28.363985+0000","last_became_peered":"2024-09-11T08:24:28.363985+0000","last_unstale":"2024-09-11T08:30:07.307250+0000","last_undegraded":"2024-09-11T08:30:07.307250+0000","last_fullsized":"2024-09-11T08:30:07.307250+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:04.967870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:20:18.900482+0000","last_clean_scrub_stamp":"2024-09-11T08:29:04.967870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0081125309999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":487,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.306789+0000","last_change":"2024-09-11T08:28:57.875994+0000","last_active":"2024-09-11T08:30:07.306789+0000","last_peered":"2024-09-11T08:30:07.306789+0000","last_clean":"2024-09-11T08:30:07.306789+0000","last_became_active":"2024-09-11T08:14:24.792595+0000","last_became_peered":"2024-09-11T08:14:24.792595+0000","last_unstale":"2024-09-11T08:30:07.306789+0000","last_undegraded":"2024-09-11T08:30:07.306789+0000","last_fullsized":"2024-09-11T08:30:07.306789+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:28:57.875851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:28:57.875851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:14.910779+0000","scrub_duration":0.014796494,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":185,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:13.004174+0000","last_change":"2024-09-11T08:30:13.004150+0000","last_active":"2024-09-11T08:30:13.004174+0000","last_peered":"2024-09-11T08:30:13.004174+0000","last_clean":"2024-09-11T08:30:13.004174+0000","last_became_active":"2024-09-11T08:24:28.359153+0000","last_became_peered":"2024-09-11T08:24:28.359153+0000","last_unstale":"2024-09-11T08:30:13.004174+0000","last_undegraded":"2024-09-11T08:30:13.004174+0000","last_fullsized":"2024-09-11T08:30:13.004174+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:13.004098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:30:13.004098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:24.968043+0000","scrub_duration":0.0014473380000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"405'11","reported_seq":385,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:06.822532+0000","last_change":"2024-09-11T08:28:58.811469+0000","last_active":"2024-09-11T08:30:06.822532+0000","last_peered":"2024-09-11T08:30:06.822532+0000","last_clean":"2024-09-11T08:30:06.822532+0000","last_became_active":"2024-09-11T08:23:22.681876+0000","last_became_peered":"2024-09-11T08:23:22.681876+0000","last_unstale":"2024-09-11T08:30:06.822532+0000","last_undegraded":"2024-09-11T08:30:06.822532+0000","last_fullsized":"2024-09-11T08:30:06.822532+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'11","last_scrub_stamp":"2024-09-11T08:28:58.811380+0000","last_deep_scrub":"139'7","last_deep_scrub_stamp":"2024-09-11T08:12:24.869318+0000","last_clean_scrub_stamp":"2024-09-11T08:28:58.811380+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.007538215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":299,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:06.817626+0000","last_change":"2024-09-11T08:27:54.287836+0000","last_active":"2024-09-11T08:30:06.817626+0000","last_peered":"2024-09-11T08:30:06.817626+0000","last_clean":"2024-09-11T08:30:06.817626+0000","last_became_active":"2024-09-11T08:15:56.526954+0000","last_became_peered":"2024-09-11T08:15:56.526954+0000","last_unstale":"2024-09-11T08:30:06.817626+0000","last_undegraded":"2024-09-11T08:30:06.817626+0000","last_fullsized":"2024-09-11T08:30:06.817626+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:54.287773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:27:54.287773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015716346999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":179,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.080693+0000","last_change":"2024-09-11T08:27:21.243352+0000","last_active":"2024-09-11T08:30:07.080693+0000","last_peered":"2024-09-11T08:30:07.080693+0000","last_clean":"2024-09-11T08:30:07.080693+0000","last_became_active":"2024-09-11T08:24:28.372097+0000","last_became_peered":"2024-09-11T08:24:28.372097+0000","last_unstale":"2024-09-11T08:30:07.080693+0000","last_undegraded":"2024-09-11T08:30:07.080693+0000","last_fullsized":"2024-09-11T08:30:07.080693+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:21.243267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:20:19.927616+0000","last_clean_scrub_stamp":"2024-09-11T08:27:21.243267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0078709639999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":435,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.475482+0000","last_change":"2024-09-11T08:30:05.450807+0000","last_active":"2024-09-11T08:30:07.475482+0000","last_peered":"2024-09-11T08:30:07.475482+0000","last_clean":"2024-09-11T08:30:07.475482+0000","last_became_active":"2024-09-11T08:24:28.368560+0000","last_became_peered":"2024-09-11T08:24:28.368560+0000","last_unstale":"2024-09-11T08:30:07.475482+0000","last_undegraded":"2024-09-11T08:30:07.475482+0000","last_fullsized":"2024-09-11T08:30:07.475482+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:05.450746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:30:05.450746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:15.048635+0000","scrub_duration":0.0012973489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":506,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.307547+0000","last_change":"2024-09-11T08:29:16.034440+0000","last_active":"2024-09-11T08:30:07.307547+0000","last_peered":"2024-09-11T08:30:07.307547+0000","last_clean":"2024-09-11T08:30:07.307547+0000","last_became_active":"2024-09-11T08:15:56.516941+0000","last_became_peered":"2024-09-11T08:15:56.516941+0000","last_unstale":"2024-09-11T08:30:07.307547+0000","last_undegraded":"2024-09-11T08:30:07.307547+0000","last_fullsized":"2024-09-11T08:30:07.307547+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:16.034353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:29:16.034353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:22.408537+0000","scrub_duration":0.0042549959999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"405'46","reported_seq":572,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.259177+0000","last_change":"2024-09-11T08:29:07.522912+0000","last_active":"2024-09-11T08:30:07.259177+0000","last_peered":"2024-09-11T08:30:07.259177+0000","last_clean":"2024-09-11T08:30:07.259177+0000","last_became_active":"2024-09-11T08:07:32.046363+0000","last_became_peered":"2024-09-11T08:07:32.046363+0000","last_unstale":"2024-09-11T08:30:07.259177+0000","last_undegraded":"2024-09-11T08:30:07.259177+0000","last_fullsized":"2024-09-11T08:30:07.259177+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'46","last_scrub_stamp":"2024-09-11T08:29:07.522566+0000","last_deep_scrub":"107'22","last_deep_scrub_stamp":"2024-09-11T08:16:04.159211+0000","last_clean_scrub_stamp":"2024-09-11T08:29:07.522566+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:25.044660+0000","scrub_duration":0.024496106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":502,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.081206+0000","last_change":"2024-09-11T08:28:55.792756+0000","last_active":"2024-09-11T08:30:07.081206+0000","last_peered":"2024-09-11T08:30:07.081206+0000","last_clean":"2024-09-11T08:30:07.081206+0000","last_became_active":"2024-09-11T08:07:34.055410+0000","last_became_peered":"2024-09-11T08:07:34.055410+0000","last_unstale":"2024-09-11T08:30:07.081206+0000","last_undegraded":"2024-09-11T08:30:07.081206+0000","last_fullsized":"2024-09-11T08:30:07.081206+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:28:55.792659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:19:28.870211+0000","last_clean_scrub_stamp":"2024-09-11T08:28:55.792659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:24.688414+0000","scrub_duration":0.015527124999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":318,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.475779+0000","last_change":"2024-09-11T08:29:56.574094+0000","last_active":"2024-09-11T08:30:07.475779+0000","last_peered":"2024-09-11T08:30:07.475779+0000","last_clean":"2024-09-11T08:30:07.475779+0000","last_became_active":"2024-09-11T08:24:28.357957+0000","last_became_peered":"2024-09-11T08:24:28.357957+0000","last_unstale":"2024-09-11T08:30:07.475779+0000","last_undegraded":"2024-09-11T08:30:07.475779+0000","last_fullsized":"2024-09-11T08:30:07.475779+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:56.574045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:29:56.574045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:13.992398+0000","scrub_duration":0.0013195139999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":151,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.081132+0000","last_change":"2024-09-11T08:29:08.797434+0000","last_active":"2024-09-11T08:30:07.081132+0000","last_peered":"2024-09-11T08:30:07.081132+0000","last_clean":"2024-09-11T08:30:07.081132+0000","last_became_active":"2024-09-11T08:23:22.686935+0000","last_became_peered":"2024-09-11T08:23:22.686935+0000","last_unstale":"2024-09-11T08:30:07.081132+0000","last_undegraded":"2024-09-11T08:30:07.081132+0000","last_fullsized":"2024-09-11T08:30:07.081132+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:08.797351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:29:08.797351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:35.864334+0000","scrub_duration":0.0047226200000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"405'24","reported_seq":360,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.258992+0000","last_change":"2024-09-11T08:29:09.521760+0000","last_active":"2024-09-11T08:30:07.258992+0000","last_peered":"2024-09-11T08:30:07.258992+0000","last_clean":"2024-09-11T08:30:07.258992+0000","last_became_active":"2024-09-11T08:17:37.553402+0000","last_became_peered":"2024-09-11T08:17:37.553402+0000","last_unstale":"2024-09-11T08:30:07.258992+0000","last_undegraded":"2024-09-11T08:30:07.258992+0000","last_fullsized":"2024-09-11T08:30:07.258992+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'24","last_scrub_stamp":"2024-09-11T08:29:09.520708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:29:09.520708+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:37.502124+0000","scrub_duration":0.0081557079999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":152,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:06.822858+0000","last_change":"2024-09-11T08:28:48.787378+0000","last_active":"2024-09-11T08:30:06.822858+0000","last_peered":"2024-09-11T08:30:06.822858+0000","last_clean":"2024-09-11T08:30:06.822858+0000","last_became_active":"2024-09-11T08:23:22.684583+0000","last_became_peered":"2024-09-11T08:23:22.684583+0000","last_unstale":"2024-09-11T08:30:06.822858+0000","last_undegraded":"2024-09-11T08:30:06.822858+0000","last_fullsized":"2024-09-11T08:30:06.822858+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:28:48.787291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:10:40.110490+0000","last_clean_scrub_stamp":"2024-09-11T08:28:48.787291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002632129,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":474,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:06.818515+0000","last_change":"2024-09-11T08:27:52.321730+0000","last_active":"2024-09-11T08:30:06.818515+0000","last_peered":"2024-09-11T08:30:06.818515+0000","last_clean":"2024-09-11T08:30:06.818515+0000","last_became_active":"2024-09-11T08:24:28.357198+0000","last_became_peered":"2024-09-11T08:24:28.357198+0000","last_unstale":"2024-09-11T08:30:06.818515+0000","last_undegraded":"2024-09-11T08:30:06.818515+0000","last_fullsized":"2024-09-11T08:30:06.818515+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-11T08:27:52.321646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:27:52.321646+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012244692999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":496,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.080886+0000","last_change":"2024-09-11T08:27:50.150362+0000","last_active":"2024-09-11T08:30:07.080886+0000","last_peered":"2024-09-11T08:30:07.080886+0000","last_clean":"2024-09-11T08:30:07.080886+0000","last_became_active":"2024-09-11T08:07:30.045932+0000","last_became_peered":"2024-09-11T08:07:30.045932+0000","last_unstale":"2024-09-11T08:30:07.080886+0000","last_undegraded":"2024-09-11T08:30:07.080886+0000","last_fullsized":"2024-09-11T08:30:07.080886+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:50.150256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:27:50.150256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026580110000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"405'35","reported_seq":597,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.307566+0000","last_change":"2024-09-11T08:28:56.912293+0000","last_active":"2024-09-11T08:30:07.307566+0000","last_peered":"2024-09-11T08:30:07.307566+0000","last_clean":"2024-09-11T08:30:07.307566+0000","last_became_active":"2024-09-11T08:15:56.515817+0000","last_became_peered":"2024-09-11T08:15:56.515817+0000","last_unstale":"2024-09-11T08:30:07.307566+0000","last_undegraded":"2024-09-11T08:30:07.307566+0000","last_fullsized":"2024-09-11T08:30:07.307566+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'35","last_scrub_stamp":"2024-09-11T08:28:56.911970+0000","last_deep_scrub":"350'25","last_deep_scrub_stamp":"2024-09-11T08:24:23.093269+0000","last_clean_scrub_stamp":"2024-09-11T08:28:56.911970+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:21.445351+0000","scrub_duration":0.066494655,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"254'12","reported_seq":832,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:08.845536+0000","last_change":"2024-09-11T08:29:14.338360+0000","last_active":"2024-09-11T08:30:08.845536+0000","last_peered":"2024-09-11T08:30:08.845536+0000","last_clean":"2024-09-11T08:30:08.845536+0000","last_became_active":"2024-09-11T08:15:56.525498+0000","last_became_peered":"2024-09-11T08:15:56.525498+0000","last_unstale":"2024-09-11T08:30:08.845536+0000","last_undegraded":"2024-09-11T08:30:08.845536+0000","last_fullsized":"2024-09-11T08:30:08.845536+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'12","last_scrub_stamp":"2024-09-11T08:29:14.338276+0000","last_deep_scrub":"254'12","last_deep_scrub_stamp":"2024-09-11T08:18:34.253865+0000","last_clean_scrub_stamp":"2024-09-11T08:29:14.338276+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:42.163322+0000","scrub_duration":0.0021322020000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.0","version":"445'15594","reported_seq":15677,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.088810+0000","last_change":"2024-09-11T08:30:07.088736+0000","last_active":"2024-09-11T08:30:07.088810+0000","last_peered":"2024-09-11T08:30:07.088810+0000","last_clean":"2024-09-11T08:30:07.088810+0000","last_became_active":"2024-09-11T08:26:27.763713+0000","last_became_peered":"2024-09-11T08:26:27.763713+0000","last_unstale":"2024-09-11T08:30:07.088810+0000","last_undegraded":"2024-09-11T08:30:07.088810+0000","last_fullsized":"2024-09-11T08:30:07.088810+0000","mapping_epoch":382,"log_start":"429'10600","ondisk_log_start":"429'10600","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"445'15594","last_scrub_stamp":"2024-09-11T08:30:07.088642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:30:07.088642+0000","objects_scrubbed":0,"log_size":4994,"log_dups_size":0,"ondisk_log_size":4994,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:16.657157+0000","scrub_duration":0.0091242019999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15594,"num_write_kb":499008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"405'39","reported_seq":565,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.081407+0000","last_change":"2024-09-11T08:27:58.226977+0000","last_active":"2024-09-11T08:30:07.081407+0000","last_peered":"2024-09-11T08:30:07.081407+0000","last_clean":"2024-09-11T08:30:07.081407+0000","last_became_active":"2024-09-11T08:07:32.054317+0000","last_became_peered":"2024-09-11T08:07:32.054317+0000","last_unstale":"2024-09-11T08:30:07.081407+0000","last_undegraded":"2024-09-11T08:30:07.081407+0000","last_fullsized":"2024-09-11T08:30:07.081407+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'39","last_scrub_stamp":"2024-09-11T08:27:58.226812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:27:58.226812+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015475453,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":517,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:06.822363+0000","last_change":"2024-09-11T08:29:14.660571+0000","last_active":"2024-09-11T08:30:06.822363+0000","last_peered":"2024-09-11T08:30:06.822363+0000","last_clean":"2024-09-11T08:30:06.822363+0000","last_became_active":"2024-09-11T08:15:56.521943+0000","last_became_peered":"2024-09-11T08:15:56.521943+0000","last_unstale":"2024-09-11T08:30:06.822363+0000","last_undegraded":"2024-09-11T08:30:06.822363+0000","last_fullsized":"2024-09-11T08:30:06.822363+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:14.660494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:29:14.660494+0000","last_clean_scrub_stamp":"2024-09-11T08:29:14.660494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:27.445624+0000","scrub_duration":0.0075203229999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"354'13","reported_seq":263,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.270740+0000","last_change":"2024-09-11T08:29:03.976374+0000","last_active":"2024-09-11T08:30:11.270740+0000","last_peered":"2024-09-11T08:30:11.270740+0000","last_clean":"2024-09-11T08:30:11.270740+0000","last_became_active":"2024-09-11T08:24:28.368212+0000","last_became_peered":"2024-09-11T08:24:28.368212+0000","last_unstale":"2024-09-11T08:30:11.270740+0000","last_undegraded":"2024-09-11T08:30:11.270740+0000","last_fullsized":"2024-09-11T08:30:11.270740+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'13","last_scrub_stamp":"2024-09-11T08:29:03.976322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:29:03.976322+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013262809999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":179,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.081221+0000","last_change":"2024-09-11T08:27:05.040598+0000","last_active":"2024-09-11T08:30:07.081221+0000","last_peered":"2024-09-11T08:30:07.081221+0000","last_clean":"2024-09-11T08:30:07.081221+0000","last_became_active":"2024-09-11T08:24:28.353556+0000","last_became_peered":"2024-09-11T08:24:28.353556+0000","last_unstale":"2024-09-11T08:30:07.081221+0000","last_undegraded":"2024-09-11T08:30:07.081221+0000","last_fullsized":"2024-09-11T08:30:07.081221+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:05.040523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:06.616332+0000","last_clean_scrub_stamp":"2024-09-11T08:27:05.040523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050758849999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.f","version":"445'15830","reported_seq":16025,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:06.818212+0000","last_change":"2024-09-11T08:29:24.806665+0000","last_active":"2024-09-11T08:30:06.818212+0000","last_peered":"2024-09-11T08:30:06.818212+0000","last_clean":"2024-09-11T08:30:06.818212+0000","last_became_active":"2024-09-11T08:26:27.766540+0000","last_became_peered":"2024-09-11T08:26:27.766540+0000","last_unstale":"2024-09-11T08:30:06.818212+0000","last_undegraded":"2024-09-11T08:30:06.818212+0000","last_fullsized":"2024-09-11T08:30:06.818212+0000","mapping_epoch":382,"log_start":"428'10700","ondisk_log_start":"428'10700","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'13079","last_scrub_stamp":"2024-09-11T08:29:24.806585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:29:24.806585+0000","objects_scrubbed":2797,"log_size":5130,"log_dups_size":0,"ondisk_log_size":5130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:44.878875+0000","scrub_duration":1.351947155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15830,"num_write_kb":506560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"405'42","reported_seq":544,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:06.817619+0000","last_change":"2024-09-11T08:27:12.224895+0000","last_active":"2024-09-11T08:30:06.817619+0000","last_peered":"2024-09-11T08:30:06.817619+0000","last_clean":"2024-09-11T08:30:06.817619+0000","last_became_active":"2024-09-11T08:17:37.555703+0000","last_became_peered":"2024-09-11T08:17:37.555703+0000","last_unstale":"2024-09-11T08:30:06.817619+0000","last_undegraded":"2024-09-11T08:30:06.817619+0000","last_fullsized":"2024-09-11T08:30:06.817619+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'30","last_scrub_stamp":"2024-09-11T08:27:12.224825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:27:12.224825+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0073510700000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":467,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:06.818421+0000","last_change":"2024-09-11T08:29:19.403998+0000","last_active":"2024-09-11T08:30:06.818421+0000","last_peered":"2024-09-11T08:30:06.818421+0000","last_clean":"2024-09-11T08:30:06.818421+0000","last_became_active":"2024-09-11T08:17:37.553579+0000","last_became_peered":"2024-09-11T08:17:37.553579+0000","last_unstale":"2024-09-11T08:30:06.818421+0000","last_undegraded":"2024-09-11T08:30:06.818421+0000","last_fullsized":"2024-09-11T08:30:06.818421+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:19.403914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:29:19.403914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:40.219628+0000","scrub_duration":0.0063698699999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":516,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:06.818451+0000","last_change":"2024-09-11T08:28:58.174623+0000","last_active":"2024-09-11T08:30:06.818451+0000","last_peered":"2024-09-11T08:30:06.818451+0000","last_clean":"2024-09-11T08:30:06.818451+0000","last_became_active":"2024-09-11T08:15:56.522606+0000","last_became_peered":"2024-09-11T08:15:56.522606+0000","last_unstale":"2024-09-11T08:30:06.818451+0000","last_undegraded":"2024-09-11T08:30:06.818451+0000","last_fullsized":"2024-09-11T08:30:06.818451+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:28:58.174534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:28:58.174534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:25.648809+0000","scrub_duration":0.0051811230000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":142,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:06.822341+0000","last_change":"2024-09-11T08:28:57.804622+0000","last_active":"2024-09-11T08:30:06.822341+0000","last_peered":"2024-09-11T08:30:06.822341+0000","last_clean":"2024-09-11T08:30:06.822341+0000","last_became_active":"2024-09-11T08:23:22.683474+0000","last_became_peered":"2024-09-11T08:23:22.683474+0000","last_unstale":"2024-09-11T08:30:06.822341+0000","last_undegraded":"2024-09-11T08:30:06.822341+0000","last_fullsized":"2024-09-11T08:30:06.822341+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:28:57.804216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:22:50.151585+0000","last_clean_scrub_stamp":"2024-09-11T08:28:57.804216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015908835999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"445'15860","reported_seq":16248,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:06.819191+0000","last_change":"2024-09-11T08:29:57.428138+0000","last_active":"2024-09-11T08:30:06.819191+0000","last_peered":"2024-09-11T08:30:06.819191+0000","last_clean":"2024-09-11T08:30:06.819191+0000","last_became_active":"2024-09-11T08:26:27.764671+0000","last_became_peered":"2024-09-11T08:26:27.764671+0000","last_unstale":"2024-09-11T08:30:06.819191+0000","last_undegraded":"2024-09-11T08:30:06.819191+0000","last_fullsized":"2024-09-11T08:30:06.819191+0000","mapping_epoch":382,"log_start":"429'11039","ondisk_log_start":"429'11039","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"444'15457","last_scrub_stamp":"2024-09-11T08:29:57.428077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:29:57.428077+0000","objects_scrubbed":410,"log_size":4821,"log_dups_size":0,"ondisk_log_size":4821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:57.685571+0000","scrub_duration":0.16589906700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15860,"num_write_kb":507520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"405'42","reported_seq":447,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.476246+0000","last_change":"2024-09-11T08:29:21.318169+0000","last_active":"2024-09-11T08:30:07.476246+0000","last_peered":"2024-09-11T08:30:07.476246+0000","last_clean":"2024-09-11T08:30:07.476246+0000","last_became_active":"2024-09-11T08:24:28.358175+0000","last_became_peered":"2024-09-11T08:24:28.358175+0000","last_unstale":"2024-09-11T08:30:07.476246+0000","last_undegraded":"2024-09-11T08:30:07.476246+0000","last_fullsized":"2024-09-11T08:30:07.476246+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'42","last_scrub_stamp":"2024-09-11T08:29:21.317951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:29:21.317951+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:41.618306+0000","scrub_duration":0.0023805409999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":496,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.307669+0000","last_change":"2024-09-11T08:28:59.885192+0000","last_active":"2024-09-11T08:30:07.307669+0000","last_peered":"2024-09-11T08:30:07.307669+0000","last_clean":"2024-09-11T08:30:07.307669+0000","last_became_active":"2024-09-11T08:28:09.953224+0000","last_became_peered":"2024-09-11T08:28:09.953224+0000","last_unstale":"2024-09-11T08:30:07.307669+0000","last_undegraded":"2024-09-11T08:30:07.307669+0000","last_fullsized":"2024-09-11T08:30:07.307669+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:28:59.884493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:47.960329+0000","last_clean_scrub_stamp":"2024-09-11T08:28:59.884493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:21.852463+0000","scrub_duration":0.012139997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"82'9","reported_seq":787,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:08.845501+0000","last_change":"2024-09-11T08:28:59.818823+0000","last_active":"2024-09-11T08:30:08.845501+0000","last_peered":"2024-09-11T08:30:08.845501+0000","last_clean":"2024-09-11T08:30:08.845501+0000","last_became_active":"2024-09-11T08:07:34.056143+0000","last_became_peered":"2024-09-11T08:07:34.056143+0000","last_unstale":"2024-09-11T08:30:08.845501+0000","last_undegraded":"2024-09-11T08:30:08.845501+0000","last_fullsized":"2024-09-11T08:30:08.845501+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-11T08:28:59.818645+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-09-11T08:21:00.841426+0000","last_clean_scrub_stamp":"2024-09-11T08:28:59.818645+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011589633,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":461,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:06.818804+0000","last_change":"2024-09-11T08:28:46.329629+0000","last_active":"2024-09-11T08:30:06.818804+0000","last_peered":"2024-09-11T08:30:06.818804+0000","last_clean":"2024-09-11T08:30:06.818804+0000","last_became_active":"2024-09-11T08:14:24.788968+0000","last_became_peered":"2024-09-11T08:14:24.788968+0000","last_unstale":"2024-09-11T08:30:06.818804+0000","last_undegraded":"2024-09-11T08:30:06.818804+0000","last_fullsized":"2024-09-11T08:30:06.818804+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:28:46.329538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:28:46.329538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017611370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.d","version":"445'15700","reported_seq":16078,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.563409+0000","last_change":"2024-09-11T08:30:07.563355+0000","last_active":"2024-09-11T08:30:07.563409+0000","last_peered":"2024-09-11T08:30:07.563409+0000","last_clean":"2024-09-11T08:30:07.563409+0000","last_became_active":"2024-09-11T08:26:27.763225+0000","last_became_peered":"2024-09-11T08:26:27.763225+0000","last_unstale":"2024-09-11T08:30:07.563409+0000","last_undegraded":"2024-09-11T08:30:07.563409+0000","last_fullsized":"2024-09-11T08:30:07.563409+0000","mapping_epoch":382,"log_start":"420'9082","ondisk_log_start":"420'9082","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"445'15700","last_scrub_stamp":"2024-09-11T08:30:07.563287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:30:07.563287+0000","objects_scrubbed":0,"log_size":6618,"log_dups_size":0,"ondisk_log_size":6618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:08.164297+0000","scrub_duration":0.0085828410000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15700,"num_write_kb":502337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"405'30","reported_seq":599,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:10.086452+0000","last_change":"2024-09-11T08:30:10.086374+0000","last_active":"2024-09-11T08:30:10.086452+0000","last_peered":"2024-09-11T08:30:10.086452+0000","last_clean":"2024-09-11T08:30:10.086452+0000","last_became_active":"2024-09-11T08:24:28.355516+0000","last_became_peered":"2024-09-11T08:24:28.355516+0000","last_unstale":"2024-09-11T08:30:10.086452+0000","last_undegraded":"2024-09-11T08:30:10.086452+0000","last_fullsized":"2024-09-11T08:30:10.086452+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'30","last_scrub_stamp":"2024-09-11T08:30:10.086268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:30:10.086268+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:38.169586+0000","scrub_duration":0.0028819319999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":197,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:06.822764+0000","last_change":"2024-09-11T08:27:52.923361+0000","last_active":"2024-09-11T08:30:06.822764+0000","last_peered":"2024-09-11T08:30:06.822764+0000","last_clean":"2024-09-11T08:30:06.822764+0000","last_became_active":"2024-09-11T08:24:28.352741+0000","last_became_peered":"2024-09-11T08:24:28.352741+0000","last_unstale":"2024-09-11T08:30:06.822764+0000","last_undegraded":"2024-09-11T08:30:06.822764+0000","last_fullsized":"2024-09-11T08:30:06.822764+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:52.923262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:27:52.923262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034940769999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"376'15","reported_seq":472,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:09.325529+0000","last_change":"2024-09-11T08:30:09.325453+0000","last_active":"2024-09-11T08:30:09.325529+0000","last_peered":"2024-09-11T08:30:09.325529+0000","last_clean":"2024-09-11T08:30:09.325529+0000","last_became_active":"2024-09-11T08:24:28.359787+0000","last_became_peered":"2024-09-11T08:24:28.359787+0000","last_unstale":"2024-09-11T08:30:09.325529+0000","last_undegraded":"2024-09-11T08:30:09.325529+0000","last_fullsized":"2024-09-11T08:30:09.325529+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'15","last_scrub_stamp":"2024-09-11T08:30:09.325345+0000","last_deep_scrub":"376'15","last_deep_scrub_stamp":"2024-09-11T08:26:02.786767+0000","last_clean_scrub_stamp":"2024-09-11T08:30:09.325345+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:23.599574+0000","scrub_duration":0.001768979,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"66'1","reported_seq":474,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.307759+0000","last_change":"2024-09-11T08:30:07.307684+0000","last_active":"2024-09-11T08:30:07.307759+0000","last_peered":"2024-09-11T08:30:07.307759+0000","last_clean":"2024-09-11T08:30:07.307759+0000","last_became_active":"2024-09-11T08:07:36.072593+0000","last_became_peered":"2024-09-11T08:07:36.072593+0000","last_unstale":"2024-09-11T08:30:07.307759+0000","last_undegraded":"2024-09-11T08:30:07.307759+0000","last_fullsized":"2024-09-11T08:30:07.307759+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-11T08:30:07.307628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:30:07.307628+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:24.772623+0000","scrub_duration":0.0021248930000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"445'15798","reported_seq":15881,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.081437+0000","last_change":"2024-09-11T08:30:04.147215+0000","last_active":"2024-09-11T08:30:07.081437+0000","last_peered":"2024-09-11T08:30:07.081437+0000","last_clean":"2024-09-11T08:30:07.081437+0000","last_became_active":"2024-09-11T08:26:27.764927+0000","last_became_peered":"2024-09-11T08:26:27.764927+0000","last_unstale":"2024-09-11T08:30:07.081437+0000","last_undegraded":"2024-09-11T08:30:07.081437+0000","last_fullsized":"2024-09-11T08:30:07.081437+0000","mapping_epoch":382,"log_start":"429'10800","ondisk_log_start":"429'10800","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"445'15798","last_scrub_stamp":"2024-09-11T08:30:04.147143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:30:04.147143+0000","objects_scrubbed":0,"log_size":4998,"log_dups_size":0,"ondisk_log_size":4998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:21.745442+0000","scrub_duration":0.0083628839999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15798,"num_write_kb":505536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"405'47","reported_seq":687,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.307331+0000","last_change":"2024-09-11T08:30:03.281050+0000","last_active":"2024-09-11T08:30:07.307331+0000","last_peered":"2024-09-11T08:30:07.307331+0000","last_clean":"2024-09-11T08:30:07.307331+0000","last_became_active":"2024-09-11T08:24:28.364043+0000","last_became_peered":"2024-09-11T08:24:28.364043+0000","last_unstale":"2024-09-11T08:30:07.307331+0000","last_undegraded":"2024-09-11T08:30:07.307331+0000","last_fullsized":"2024-09-11T08:30:07.307331+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'47","last_scrub_stamp":"2024-09-11T08:30:03.280942+0000","last_deep_scrub":"396'37","last_deep_scrub_stamp":"2024-09-11T08:27:16.673846+0000","last_clean_scrub_stamp":"2024-09-11T08:30:03.280942+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:25.945261+0000","scrub_duration":0.0024639179999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":483,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.259243+0000","last_change":"2024-09-11T08:29:57.672185+0000","last_active":"2024-09-11T08:30:07.259243+0000","last_peered":"2024-09-11T08:30:07.259243+0000","last_clean":"2024-09-11T08:30:07.259243+0000","last_became_active":"2024-09-11T08:27:57.795859+0000","last_became_peered":"2024-09-11T08:27:57.795859+0000","last_unstale":"2024-09-11T08:30:07.259243+0000","last_undegraded":"2024-09-11T08:30:07.259243+0000","last_fullsized":"2024-09-11T08:30:07.259243+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:57.672105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:28:03.117012+0000","last_clean_scrub_stamp":"2024-09-11T08:29:57.672105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:00.681043+0000","scrub_duration":0.011006079,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":539,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.475553+0000","last_change":"2024-09-11T08:29:24.232937+0000","last_active":"2024-09-11T08:30:07.475553+0000","last_peered":"2024-09-11T08:30:07.475553+0000","last_clean":"2024-09-11T08:30:07.475553+0000","last_became_active":"2024-09-11T08:24:28.360235+0000","last_became_peered":"2024-09-11T08:24:28.360235+0000","last_unstale":"2024-09-11T08:30:07.475553+0000","last_undegraded":"2024-09-11T08:30:07.475553+0000","last_fullsized":"2024-09-11T08:30:07.475553+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:24.232816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:29:24.232816+0000","last_clean_scrub_stamp":"2024-09-11T08:29:24.232816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:42.768241+0000","scrub_duration":0.0060421820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":490,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.080700+0000","last_change":"2024-09-11T08:27:56.203308+0000","last_active":"2024-09-11T08:30:07.080700+0000","last_peered":"2024-09-11T08:30:07.080700+0000","last_clean":"2024-09-11T08:30:07.080700+0000","last_became_active":"2024-09-11T08:07:36.631668+0000","last_became_peered":"2024-09-11T08:07:36.631668+0000","last_unstale":"2024-09-11T08:30:07.080700+0000","last_undegraded":"2024-09-11T08:30:07.080700+0000","last_fullsized":"2024-09-11T08:30:07.080700+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:56.203218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:58.517624+0000","last_clean_scrub_stamp":"2024-09-11T08:27:56.203218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010650626,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.b","version":"445'15778","reported_seq":16161,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:06.822442+0000","last_change":"2024-09-11T08:28:06.559643+0000","last_active":"2024-09-11T08:30:06.822442+0000","last_peered":"2024-09-11T08:30:06.822442+0000","last_clean":"2024-09-11T08:30:06.822442+0000","last_became_active":"2024-09-11T08:26:27.764354+0000","last_became_peered":"2024-09-11T08:26:27.764354+0000","last_unstale":"2024-09-11T08:30:06.822442+0000","last_undegraded":"2024-09-11T08:30:06.822442+0000","last_fullsized":"2024-09-11T08:30:06.822442+0000","mapping_epoch":382,"log_start":"429'11051","ondisk_log_start":"429'11051","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'7590","last_scrub_stamp":"2024-09-11T08:28:06.558470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:28:06.558470+0000","objects_scrubbed":7490,"log_size":4727,"log_dups_size":0,"ondisk_log_size":4727,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"queued for scrub","scrub_duration":3.628423288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15778,"num_write_kb":504896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"405'64","reported_seq":353,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:10.835916+0000","last_change":"2024-09-11T08:28:00.205458+0000","last_active":"2024-09-11T08:30:10.835916+0000","last_peered":"2024-09-11T08:30:10.835916+0000","last_clean":"2024-09-11T08:30:10.835916+0000","last_became_active":"2024-09-11T08:20:52.086843+0000","last_became_peered":"2024-09-11T08:20:52.086843+0000","last_unstale":"2024-09-11T08:30:10.835916+0000","last_undegraded":"2024-09-11T08:30:10.835916+0000","last_fullsized":"2024-09-11T08:30:10.835916+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'64","last_scrub_stamp":"2024-09-11T08:28:00.205375+0000","last_deep_scrub":"190'39","last_deep_scrub_stamp":"2024-09-11T08:15:52.847968+0000","last_clean_scrub_stamp":"2024-09-11T08:28:00.205375+0000","objects_scrubbed":10,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0087966820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":84,"num_read_kb":69,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"59'1","reported_seq":150,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.307416+0000","last_change":"2024-09-11T08:28:53.832466+0000","last_active":"2024-09-11T08:30:07.307416+0000","last_peered":"2024-09-11T08:30:07.307416+0000","last_clean":"2024-09-11T08:30:07.307416+0000","last_became_active":"2024-09-11T08:23:22.691726+0000","last_became_peered":"2024-09-11T08:23:22.691726+0000","last_unstale":"2024-09-11T08:30:07.307416+0000","last_undegraded":"2024-09-11T08:30:07.307416+0000","last_fullsized":"2024-09-11T08:30:07.307416+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-11T08:28:53.832355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:28:53.832355+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0063587790000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":206,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.080990+0000","last_change":"2024-09-11T08:28:51.863387+0000","last_active":"2024-09-11T08:30:07.080990+0000","last_peered":"2024-09-11T08:30:07.080990+0000","last_clean":"2024-09-11T08:30:07.080990+0000","last_became_active":"2024-09-11T08:24:28.362590+0000","last_became_peered":"2024-09-11T08:24:28.362590+0000","last_unstale":"2024-09-11T08:30:07.080990+0000","last_undegraded":"2024-09-11T08:30:07.080990+0000","last_fullsized":"2024-09-11T08:30:07.080990+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-11T08:28:51.863309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:28:51.863309+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:13.298195+0000","scrub_duration":0.00679483,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":472,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.307355+0000","last_change":"2024-09-11T08:28:50.908738+0000","last_active":"2024-09-11T08:30:07.307355+0000","last_peered":"2024-09-11T08:30:07.307355+0000","last_clean":"2024-09-11T08:30:07.307355+0000","last_became_active":"2024-09-11T08:07:34.057154+0000","last_became_peered":"2024-09-11T08:07:34.057154+0000","last_unstale":"2024-09-11T08:30:07.307355+0000","last_undegraded":"2024-09-11T08:30:07.307355+0000","last_fullsized":"2024-09-11T08:30:07.307355+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:28:50.908650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:10:37.957096+0000","last_clean_scrub_stamp":"2024-09-11T08:28:50.908650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:18.089391+0000","scrub_duration":0.014372642,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":508,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:06.822675+0000","last_change":"2024-09-11T08:27:47.874765+0000","last_active":"2024-09-11T08:30:06.822675+0000","last_peered":"2024-09-11T08:30:06.822675+0000","last_clean":"2024-09-11T08:30:06.822675+0000","last_became_active":"2024-09-11T08:15:56.525099+0000","last_became_peered":"2024-09-11T08:15:56.525099+0000","last_unstale":"2024-09-11T08:30:06.822675+0000","last_undegraded":"2024-09-11T08:30:06.822675+0000","last_fullsized":"2024-09-11T08:30:06.822675+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:47.874710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:27:47.874710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049520060000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"445'15898","reported_seq":16170,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.475985+0000","last_change":"2024-09-11T08:28:56.017775+0000","last_active":"2024-09-11T08:30:07.475985+0000","last_peered":"2024-09-11T08:30:07.475985+0000","last_clean":"2024-09-11T08:30:07.475985+0000","last_became_active":"2024-09-11T08:26:28.082722+0000","last_became_peered":"2024-09-11T08:26:28.082722+0000","last_unstale":"2024-09-11T08:30:07.475985+0000","last_undegraded":"2024-09-11T08:30:07.475985+0000","last_fullsized":"2024-09-11T08:30:07.475985+0000","mapping_epoch":382,"log_start":"425'10000","ondisk_log_start":"425'10000","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"429'11228","last_scrub_stamp":"2024-09-11T08:28:56.017718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:28:56.017718+0000","objects_scrubbed":4714,"log_size":5898,"log_dups_size":0,"ondisk_log_size":5898,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1.584546453,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15898,"num_write_kb":508736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"405'21","reported_seq":593,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.081434+0000","last_change":"2024-09-11T08:26:56.891150+0000","last_active":"2024-09-11T08:30:07.081434+0000","last_peered":"2024-09-11T08:30:07.081434+0000","last_clean":"2024-09-11T08:30:07.081434+0000","last_became_active":"2024-09-11T08:24:28.358121+0000","last_became_peered":"2024-09-11T08:24:28.358121+0000","last_unstale":"2024-09-11T08:30:07.081434+0000","last_undegraded":"2024-09-11T08:30:07.081434+0000","last_fullsized":"2024-09-11T08:30:07.081434+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'17","last_scrub_stamp":"2024-09-11T08:26:56.890998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:26:56.890998+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012396804000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":509,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.080647+0000","last_change":"2024-09-11T08:29:15.868218+0000","last_active":"2024-09-11T08:30:07.080647+0000","last_peered":"2024-09-11T08:30:07.080647+0000","last_clean":"2024-09-11T08:30:07.080647+0000","last_became_active":"2024-09-11T08:07:30.047014+0000","last_became_peered":"2024-09-11T08:07:30.047014+0000","last_unstale":"2024-09-11T08:30:07.080647+0000","last_undegraded":"2024-09-11T08:30:07.080647+0000","last_fullsized":"2024-09-11T08:30:07.080647+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:15.867962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:29:15.867962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:39.510619+0000","scrub_duration":0.0075403279999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"127'2","reported_seq":536,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:08.554405+0000","last_change":"2024-09-11T08:30:08.554358+0000","last_active":"2024-09-11T08:30:08.554405+0000","last_peered":"2024-09-11T08:30:08.554405+0000","last_clean":"2024-09-11T08:30:08.554405+0000","last_became_active":"2024-09-11T08:07:30.045009+0000","last_became_peered":"2024-09-11T08:07:30.045009+0000","last_unstale":"2024-09-11T08:30:08.554405+0000","last_undegraded":"2024-09-11T08:30:08.554405+0000","last_fullsized":"2024-09-11T08:30:08.554405+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'2","last_scrub_stamp":"2024-09-11T08:30:08.554296+0000","last_deep_scrub":"127'2","last_deep_scrub_stamp":"2024-09-11T08:15:01.938497+0000","last_clean_scrub_stamp":"2024-09-11T08:30:08.554296+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:21.485645+0000","scrub_duration":0.0016669009999999999,"objects_trimmed":0,"snaptrim_duration":0.001859533,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":562,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.259261+0000","last_change":"2024-09-11T08:30:00.586632+0000","last_active":"2024-09-11T08:30:07.259261+0000","last_peered":"2024-09-11T08:30:07.259261+0000","last_clean":"2024-09-11T08:30:07.259261+0000","last_became_active":"2024-09-11T08:24:28.357141+0000","last_became_peered":"2024-09-11T08:24:28.357141+0000","last_unstale":"2024-09-11T08:30:07.259261+0000","last_undegraded":"2024-09-11T08:30:07.259261+0000","last_fullsized":"2024-09-11T08:30:07.259261+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:00.586547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:22:00.200872+0000","last_clean_scrub_stamp":"2024-09-11T08:30:00.586547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:09.658987+0000","scrub_duration":0.0097564220000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":212,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.258979+0000","last_change":"2024-09-11T08:29:01.399827+0000","last_active":"2024-09-11T08:30:07.258979+0000","last_peered":"2024-09-11T08:30:07.258979+0000","last_clean":"2024-09-11T08:30:07.258979+0000","last_became_active":"2024-09-11T08:20:16.108674+0000","last_became_peered":"2024-09-11T08:20:16.108674+0000","last_unstale":"2024-09-11T08:30:07.258979+0000","last_undegraded":"2024-09-11T08:30:07.258979+0000","last_fullsized":"2024-09-11T08:30:07.258979+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:01.399604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:58.796217+0000","last_clean_scrub_stamp":"2024-09-11T08:29:01.399604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.006953725,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.9","version":"445'15604","reported_seq":15973,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:06.822414+0000","last_change":"2024-09-11T08:28:02.140930+0000","last_active":"2024-09-11T08:30:06.822414+0000","last_peered":"2024-09-11T08:30:06.822414+0000","last_clean":"2024-09-11T08:30:06.822414+0000","last_became_active":"2024-09-11T08:26:27.761414+0000","last_became_peered":"2024-09-11T08:26:27.761414+0000","last_unstale":"2024-09-11T08:30:06.822414+0000","last_undegraded":"2024-09-11T08:30:06.822414+0000","last_fullsized":"2024-09-11T08:30:06.822414+0000","mapping_epoch":382,"log_start":"429'10900","ondisk_log_start":"429'10900","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'7252","last_scrub_stamp":"2024-09-11T08:28:02.140845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:28:02.140845+0000","objects_scrubbed":7120,"log_size":4704,"log_dups_size":0,"ondisk_log_size":4704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"queued for scrub","scrub_duration":4.2275436720000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15604,"num_write_kb":499328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"405'42","reported_seq":544,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.307600+0000","last_change":"2024-09-11T08:29:10.054827+0000","last_active":"2024-09-11T08:30:07.307600+0000","last_peered":"2024-09-11T08:30:07.307600+0000","last_clean":"2024-09-11T08:30:07.307600+0000","last_became_active":"2024-09-11T08:07:32.055407+0000","last_became_peered":"2024-09-11T08:07:32.055407+0000","last_unstale":"2024-09-11T08:30:07.307600+0000","last_undegraded":"2024-09-11T08:30:07.307600+0000","last_fullsized":"2024-09-11T08:30:07.307600+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'42","last_scrub_stamp":"2024-09-11T08:29:10.054727+0000","last_deep_scrub":"287'30","last_deep_scrub_stamp":"2024-09-11T08:26:09.030192+0000","last_clean_scrub_stamp":"2024-09-11T08:29:10.054727+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:34.247277+0000","scrub_duration":0.042708410000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"59'1","reported_seq":483,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.307113+0000","last_change":"2024-09-11T08:28:04.788558+0000","last_active":"2024-09-11T08:30:07.307113+0000","last_peered":"2024-09-11T08:30:07.307113+0000","last_clean":"2024-09-11T08:30:07.307113+0000","last_became_active":"2024-09-11T08:07:30.050504+0000","last_became_peered":"2024-09-11T08:07:30.050504+0000","last_unstale":"2024-09-11T08:30:07.307113+0000","last_undegraded":"2024-09-11T08:30:07.307113+0000","last_fullsized":"2024-09-11T08:30:07.307113+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-11T08:28:04.787985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:28:04.787985+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0077345010000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"224'6","reported_seq":1626,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.596067+0000","last_change":"2024-09-11T08:28:06.324532+0000","last_active":"2024-09-11T08:30:11.596067+0000","last_peered":"2024-09-11T08:30:11.596067+0000","last_clean":"2024-09-11T08:30:11.596067+0000","last_became_active":"2024-09-11T08:14:24.793048+0000","last_became_peered":"2024-09-11T08:14:24.793048+0000","last_unstale":"2024-09-11T08:30:11.596067+0000","last_undegraded":"2024-09-11T08:30:11.596067+0000","last_fullsized":"2024-09-11T08:30:11.596067+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'6","last_scrub_stamp":"2024-09-11T08:28:06.324370+0000","last_deep_scrub":"224'6","last_deep_scrub_stamp":"2024-09-11T08:16:55.778284+0000","last_clean_scrub_stamp":"2024-09-11T08:28:06.324370+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036467399999999999,"objects_trimmed":0,"snaptrim_duration":0.000318821,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":953,"num_read_kb":948,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":189,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.081279+0000","last_change":"2024-09-11T08:27:14.117515+0000","last_active":"2024-09-11T08:30:07.081279+0000","last_peered":"2024-09-11T08:30:07.081279+0000","last_clean":"2024-09-11T08:30:07.081279+0000","last_became_active":"2024-09-11T08:24:28.353018+0000","last_became_peered":"2024-09-11T08:24:28.353018+0000","last_unstale":"2024-09-11T08:30:07.081279+0000","last_undegraded":"2024-09-11T08:30:07.081279+0000","last_fullsized":"2024-09-11T08:30:07.081279+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:14.114373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:27:14.114373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0066634249999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":533,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.081178+0000","last_change":"2024-09-11T08:27:11.025202+0000","last_active":"2024-09-11T08:30:07.081178+0000","last_peered":"2024-09-11T08:30:07.081178+0000","last_clean":"2024-09-11T08:30:07.081178+0000","last_became_active":"2024-09-11T08:24:28.358723+0000","last_became_peered":"2024-09-11T08:24:28.358723+0000","last_unstale":"2024-09-11T08:30:07.081178+0000","last_undegraded":"2024-09-11T08:30:07.081178+0000","last_fullsized":"2024-09-11T08:30:07.081178+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:11.025109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:27:11.025109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049163100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"445'15644","reported_seq":15918,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.475946+0000","last_change":"2024-09-11T08:28:54.217746+0000","last_active":"2024-09-11T08:30:07.475946+0000","last_peered":"2024-09-11T08:30:07.475946+0000","last_clean":"2024-09-11T08:30:07.475946+0000","last_became_active":"2024-09-11T08:26:28.082996+0000","last_became_peered":"2024-09-11T08:26:28.082996+0000","last_unstale":"2024-09-11T08:30:07.475946+0000","last_undegraded":"2024-09-11T08:30:07.475946+0000","last_fullsized":"2024-09-11T08:30:07.475946+0000","mapping_epoch":382,"log_start":"423'9700","ondisk_log_start":"423'9700","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"429'10907","last_scrub_stamp":"2024-09-11T08:28:54.217620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:28:54.217620+0000","objects_scrubbed":4793,"log_size":5944,"log_dups_size":0,"ondisk_log_size":5944,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:17.688420+0000","scrub_duration":1.776915075,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15644,"num_write_kb":500608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"405'45","reported_seq":586,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.476486+0000","last_change":"2024-09-11T08:29:13.340969+0000","last_active":"2024-09-11T08:30:07.476486+0000","last_peered":"2024-09-11T08:30:07.476486+0000","last_clean":"2024-09-11T08:30:07.476486+0000","last_became_active":"2024-09-11T08:14:24.789154+0000","last_became_peered":"2024-09-11T08:14:24.789154+0000","last_unstale":"2024-09-11T08:30:07.476486+0000","last_undegraded":"2024-09-11T08:30:07.476486+0000","last_fullsized":"2024-09-11T08:30:07.476486+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'45","last_scrub_stamp":"2024-09-11T08:29:13.340918+0000","last_deep_scrub":"84'21","last_deep_scrub_stamp":"2024-09-11T08:08:56.705306+0000","last_clean_scrub_stamp":"2024-09-11T08:29:13.340918+0000","objects_scrubbed":7,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:34.815204+0000","scrub_duration":0.001693937,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":191,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.476163+0000","last_change":"2024-09-11T08:28:48.470279+0000","last_active":"2024-09-11T08:30:07.476163+0000","last_peered":"2024-09-11T08:30:07.476163+0000","last_clean":"2024-09-11T08:30:07.476163+0000","last_became_active":"2024-09-11T08:24:28.362417+0000","last_became_peered":"2024-09-11T08:24:28.362417+0000","last_unstale":"2024-09-11T08:30:07.476163+0000","last_undegraded":"2024-09-11T08:30:07.476163+0000","last_fullsized":"2024-09-11T08:30:07.476163+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:28:48.470005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:07.749828+0000","last_clean_scrub_stamp":"2024-09-11T08:28:48.470005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011407896000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":305,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:06.817288+0000","last_change":"2024-09-11T08:28:01.294768+0000","last_active":"2024-09-11T08:30:06.817288+0000","last_peered":"2024-09-11T08:30:06.817288+0000","last_clean":"2024-09-11T08:30:06.817288+0000","last_became_active":"2024-09-11T08:15:56.523430+0000","last_became_peered":"2024-09-11T08:15:56.523430+0000","last_unstale":"2024-09-11T08:30:06.817288+0000","last_undegraded":"2024-09-11T08:30:06.817288+0000","last_fullsized":"2024-09-11T08:30:06.817288+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-11T08:28:01.294597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:05:21.231856+0000","last_clean_scrub_stamp":"2024-09-11T08:28:01.294597+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027233790000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":329,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:06.818008+0000","last_change":"2024-09-11T08:27:49.296146+0000","last_active":"2024-09-11T08:30:06.818008+0000","last_peered":"2024-09-11T08:30:06.818008+0000","last_clean":"2024-09-11T08:30:06.818008+0000","last_became_active":"2024-09-11T08:24:28.357166+0000","last_became_peered":"2024-09-11T08:24:28.357166+0000","last_unstale":"2024-09-11T08:30:06.818008+0000","last_undegraded":"2024-09-11T08:30:06.818008+0000","last_fullsized":"2024-09-11T08:30:06.818008+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:49.296063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:20:25.775869+0000","last_clean_scrub_stamp":"2024-09-11T08:27:49.296063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037448009999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":246,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.476430+0000","last_change":"2024-09-11T08:28:46.490433+0000","last_active":"2024-09-11T08:30:07.476430+0000","last_peered":"2024-09-11T08:30:07.476430+0000","last_clean":"2024-09-11T08:30:07.476430+0000","last_became_active":"2024-09-11T08:24:28.356071+0000","last_became_peered":"2024-09-11T08:24:28.356071+0000","last_unstale":"2024-09-11T08:30:07.476430+0000","last_undegraded":"2024-09-11T08:30:07.476430+0000","last_fullsized":"2024-09-11T08:30:07.476430+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:28:46.490355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:01.777429+0000","last_clean_scrub_stamp":"2024-09-11T08:28:46.490355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:13.864975+0000","scrub_duration":0.0016873459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"445'15644","reported_seq":15935,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.475656+0000","last_change":"2024-09-11T08:28:49.870650+0000","last_active":"2024-09-11T08:30:07.475656+0000","last_peered":"2024-09-11T08:30:07.475656+0000","last_clean":"2024-09-11T08:30:07.475656+0000","last_became_active":"2024-09-11T08:26:27.763926+0000","last_became_peered":"2024-09-11T08:26:27.763926+0000","last_unstale":"2024-09-11T08:30:07.475656+0000","last_undegraded":"2024-09-11T08:30:07.475656+0000","last_fullsized":"2024-09-11T08:30:07.475656+0000","mapping_epoch":382,"log_start":"423'9700","ondisk_log_start":"423'9700","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"427'10531","last_scrub_stamp":"2024-09-11T08:28:49.870574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:28:49.870574+0000","objects_scrubbed":5191,"log_size":5944,"log_dups_size":0,"ondisk_log_size":5944,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2.458437322,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15644,"num_write_kb":500608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"405'32","reported_seq":432,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.475586+0000","last_change":"2024-09-11T08:29:10.333244+0000","last_active":"2024-09-11T08:30:07.475586+0000","last_peered":"2024-09-11T08:30:07.475586+0000","last_clean":"2024-09-11T08:30:07.475586+0000","last_became_active":"2024-09-11T08:24:28.368759+0000","last_became_peered":"2024-09-11T08:24:28.368759+0000","last_unstale":"2024-09-11T08:30:07.475586+0000","last_undegraded":"2024-09-11T08:30:07.475586+0000","last_fullsized":"2024-09-11T08:30:07.475586+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'32","last_scrub_stamp":"2024-09-11T08:29:10.333163+0000","last_deep_scrub":"405'32","last_deep_scrub_stamp":"2024-09-11T08:29:10.333163+0000","last_clean_scrub_stamp":"2024-09-11T08:29:10.333163+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:32.092567+0000","scrub_duration":0.0057426860000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":523,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.261050+0000","last_change":"2024-09-11T08:30:07.260981+0000","last_active":"2024-09-11T08:30:07.261050+0000","last_peered":"2024-09-11T08:30:07.261050+0000","last_clean":"2024-09-11T08:30:07.261050+0000","last_became_active":"2024-09-11T08:14:24.789888+0000","last_became_peered":"2024-09-11T08:14:24.789888+0000","last_unstale":"2024-09-11T08:30:07.261050+0000","last_undegraded":"2024-09-11T08:30:07.261050+0000","last_fullsized":"2024-09-11T08:30:07.261050+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:07.260906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:36.417911+0000","last_clean_scrub_stamp":"2024-09-11T08:30:07.260906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:31.737712+0000","scrub_duration":0.0037043279999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":193,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.475910+0000","last_change":"2024-09-11T08:30:06.443515+0000","last_active":"2024-09-11T08:30:07.475910+0000","last_peered":"2024-09-11T08:30:07.475910+0000","last_clean":"2024-09-11T08:30:07.475910+0000","last_became_active":"2024-09-11T08:24:28.361261+0000","last_became_peered":"2024-09-11T08:24:28.361261+0000","last_unstale":"2024-09-11T08:30:07.475910+0000","last_undegraded":"2024-09-11T08:30:07.475910+0000","last_fullsized":"2024-09-11T08:30:07.475910+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:06.443433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:01.558340+0000","last_clean_scrub_stamp":"2024-09-11T08:30:06.443433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:23.309677+0000","scrub_duration":0.001552765,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":321,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.475947+0000","last_change":"2024-09-11T08:29:16.340861+0000","last_active":"2024-09-11T08:30:07.475947+0000","last_peered":"2024-09-11T08:30:07.475947+0000","last_clean":"2024-09-11T08:30:07.475947+0000","last_became_active":"2024-09-11T08:24:28.357842+0000","last_became_peered":"2024-09-11T08:24:28.357842+0000","last_unstale":"2024-09-11T08:30:07.475947+0000","last_undegraded":"2024-09-11T08:30:07.475947+0000","last_fullsized":"2024-09-11T08:30:07.475947+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:16.340778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:29:16.340778+0000","last_clean_scrub_stamp":"2024-09-11T08:29:16.340778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:43.003239+0000","scrub_duration":0.0019631459999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"445'16002","reported_seq":16370,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.258858+0000","last_change":"2024-09-11T08:30:04.556077+0000","last_active":"2024-09-11T08:30:07.258858+0000","last_peered":"2024-09-11T08:30:07.258858+0000","last_clean":"2024-09-11T08:30:07.258858+0000","last_became_active":"2024-09-11T08:26:27.763602+0000","last_became_peered":"2024-09-11T08:26:27.763602+0000","last_unstale":"2024-09-11T08:30:07.258858+0000","last_undegraded":"2024-09-11T08:30:07.258858+0000","last_fullsized":"2024-09-11T08:30:07.258858+0000","mapping_epoch":382,"log_start":"421'9427","ondisk_log_start":"421'9427","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"445'16002","last_scrub_stamp":"2024-09-11T08:30:04.556008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:30:04.556008+0000","objects_scrubbed":0,"log_size":6575,"log_dups_size":0,"ondisk_log_size":6575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:12.217879+0000","scrub_duration":0.0084202689999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16002,"num_write_kb":512064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"405'53","reported_seq":493,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.259099+0000","last_change":"2024-09-11T08:30:02.596837+0000","last_active":"2024-09-11T08:30:07.259099+0000","last_peered":"2024-09-11T08:30:07.259099+0000","last_clean":"2024-09-11T08:30:07.259099+0000","last_became_active":"2024-09-11T08:23:22.681452+0000","last_became_peered":"2024-09-11T08:23:22.681452+0000","last_unstale":"2024-09-11T08:30:07.259099+0000","last_undegraded":"2024-09-11T08:30:07.259099+0000","last_fullsized":"2024-09-11T08:30:07.259099+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'53","last_scrub_stamp":"2024-09-11T08:30:02.596714+0000","last_deep_scrub":"77'25","last_deep_scrub_stamp":"2024-09-11T08:08:32.458210+0000","last_clean_scrub_stamp":"2024-09-11T08:30:02.596714+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:30.319012+0000","scrub_duration":0.0038267459999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":512,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.475848+0000","last_change":"2024-09-11T08:29:57.622728+0000","last_active":"2024-09-11T08:30:07.475848+0000","last_peered":"2024-09-11T08:30:07.475848+0000","last_clean":"2024-09-11T08:30:07.475848+0000","last_became_active":"2024-09-11T08:07:30.042555+0000","last_became_peered":"2024-09-11T08:07:30.042555+0000","last_unstale":"2024-09-11T08:30:07.475848+0000","last_undegraded":"2024-09-11T08:30:07.475848+0000","last_fullsized":"2024-09-11T08:30:07.475848+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:57.622675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:29:57.622675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:21.379775+0000","scrub_duration":0.0036282020000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":492,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.081330+0000","last_change":"2024-09-11T08:27:49.130976+0000","last_active":"2024-09-11T08:30:07.081330+0000","last_peered":"2024-09-11T08:30:07.081330+0000","last_clean":"2024-09-11T08:30:07.081330+0000","last_became_active":"2024-09-11T08:07:34.054961+0000","last_became_peered":"2024-09-11T08:07:34.054961+0000","last_unstale":"2024-09-11T08:30:07.081330+0000","last_undegraded":"2024-09-11T08:30:07.081330+0000","last_fullsized":"2024-09-11T08:30:07.081330+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:49.130900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:27:49.130900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004734121,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":520,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.259226+0000","last_change":"2024-09-11T08:29:56.700851+0000","last_active":"2024-09-11T08:30:07.259226+0000","last_peered":"2024-09-11T08:30:07.259226+0000","last_clean":"2024-09-11T08:30:07.259226+0000","last_became_active":"2024-09-11T08:07:36.071194+0000","last_became_peered":"2024-09-11T08:07:36.071194+0000","last_unstale":"2024-09-11T08:30:07.259226+0000","last_undegraded":"2024-09-11T08:30:07.259226+0000","last_fullsized":"2024-09-11T08:30:07.259226+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:56.700632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:07.930137+0000","last_clean_scrub_stamp":"2024-09-11T08:29:56.700632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:21.407208+0000","scrub_duration":0.0045560790000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"445'15906","reported_seq":16192,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.476259+0000","last_change":"2024-09-11T08:28:52.231284+0000","last_active":"2024-09-11T08:30:07.476259+0000","last_peered":"2024-09-11T08:30:07.476259+0000","last_clean":"2024-09-11T08:30:07.476259+0000","last_became_active":"2024-09-11T08:26:27.764638+0000","last_became_peered":"2024-09-11T08:26:27.764638+0000","last_unstale":"2024-09-11T08:30:07.476259+0000","last_undegraded":"2024-09-11T08:30:07.476259+0000","last_fullsized":"2024-09-11T08:30:07.476259+0000","mapping_epoch":382,"log_start":"425'10000","ondisk_log_start":"425'10000","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"429'10884","last_scrub_stamp":"2024-09-11T08:28:52.228161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:28:52.228161+0000","objects_scrubbed":5083,"log_size":5906,"log_dups_size":0,"ondisk_log_size":5906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:14.473528+0000","scrub_duration":1.8024673180000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15906,"num_write_kb":508992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"405'35","reported_seq":427,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.475547+0000","last_change":"2024-09-11T08:30:01.519867+0000","last_active":"2024-09-11T08:30:07.475547+0000","last_peered":"2024-09-11T08:30:07.475547+0000","last_clean":"2024-09-11T08:30:07.475547+0000","last_became_active":"2024-09-11T08:23:22.688631+0000","last_became_peered":"2024-09-11T08:23:22.688631+0000","last_unstale":"2024-09-11T08:30:07.475547+0000","last_undegraded":"2024-09-11T08:30:07.475547+0000","last_fullsized":"2024-09-11T08:30:07.475547+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'35","last_scrub_stamp":"2024-09-11T08:30:01.519776+0000","last_deep_scrub":"224'15","last_deep_scrub_stamp":"2024-09-11T08:17:01.126478+0000","last_clean_scrub_stamp":"2024-09-11T08:30:01.519776+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:15.125283+0000","scrub_duration":0.0073496450000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":261,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.307495+0000","last_change":"2024-09-11T08:27:59.866904+0000","last_active":"2024-09-11T08:30:07.307495+0000","last_peered":"2024-09-11T08:30:07.307495+0000","last_clean":"2024-09-11T08:30:07.307495+0000","last_became_active":"2024-09-11T08:17:37.557278+0000","last_became_peered":"2024-09-11T08:17:37.557278+0000","last_unstale":"2024-09-11T08:30:07.307495+0000","last_undegraded":"2024-09-11T08:30:07.307495+0000","last_fullsized":"2024-09-11T08:30:07.307495+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:59.866811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:27:59.866811+0000","last_clean_scrub_stamp":"2024-09-11T08:27:59.866811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010435936,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"354'16","reported_seq":853,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:08.845540+0000","last_change":"2024-09-11T08:29:06.507763+0000","last_active":"2024-09-11T08:30:08.845540+0000","last_peered":"2024-09-11T08:30:08.845540+0000","last_clean":"2024-09-11T08:30:08.845540+0000","last_became_active":"2024-09-11T08:24:28.356749+0000","last_became_peered":"2024-09-11T08:24:28.356749+0000","last_unstale":"2024-09-11T08:30:08.845540+0000","last_undegraded":"2024-09-11T08:30:08.845540+0000","last_fullsized":"2024-09-11T08:30:08.845540+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'16","last_scrub_stamp":"2024-09-11T08:29:06.507663+0000","last_deep_scrub":"171'12","last_deep_scrub_stamp":"2024-09-11T08:14:56.881299+0000","last_clean_scrub_stamp":"2024-09-11T08:29:06.507663+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:33.390159+0000","scrub_duration":0.015460094000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":151,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.081459+0000","last_change":"2024-09-11T08:29:14.863576+0000","last_active":"2024-09-11T08:30:07.081459+0000","last_peered":"2024-09-11T08:30:07.081459+0000","last_clean":"2024-09-11T08:30:07.081459+0000","last_became_active":"2024-09-11T08:23:22.686992+0000","last_became_peered":"2024-09-11T08:23:22.686992+0000","last_unstale":"2024-09-11T08:30:07.081459+0000","last_undegraded":"2024-09-11T08:30:07.081459+0000","last_fullsized":"2024-09-11T08:30:07.081459+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:14.863324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:14.613532+0000","last_clean_scrub_stamp":"2024-09-11T08:29:14.863324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:30.823910+0000","scrub_duration":0.0069445239999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.4","version":"445'15530","reported_seq":15728,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:06.817732+0000","last_change":"2024-09-11T08:29:22.445666+0000","last_active":"2024-09-11T08:30:06.817732+0000","last_peered":"2024-09-11T08:30:06.817732+0000","last_clean":"2024-09-11T08:30:06.817732+0000","last_became_active":"2024-09-11T08:26:27.765686+0000","last_became_peered":"2024-09-11T08:26:27.765686+0000","last_unstale":"2024-09-11T08:30:06.817732+0000","last_undegraded":"2024-09-11T08:30:06.817732+0000","last_fullsized":"2024-09-11T08:30:06.817732+0000","mapping_epoch":382,"log_start":"427'10400","ondisk_log_start":"427'10400","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'12685","last_scrub_stamp":"2024-09-11T08:29:22.445566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:29:22.445566+0000","objects_scrubbed":2877,"log_size":5130,"log_dups_size":0,"ondisk_log_size":5130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:37.122745+0000","scrub_duration":0.99378896999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15530,"num_write_kb":496960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"405'21","reported_seq":508,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.307299+0000","last_change":"2024-09-11T08:28:02.865865+0000","last_active":"2024-09-11T08:30:07.307299+0000","last_peered":"2024-09-11T08:30:07.307299+0000","last_clean":"2024-09-11T08:30:07.307299+0000","last_became_active":"2024-09-11T08:07:32.054885+0000","last_became_peered":"2024-09-11T08:07:32.054885+0000","last_unstale":"2024-09-11T08:30:07.307299+0000","last_undegraded":"2024-09-11T08:30:07.307299+0000","last_fullsized":"2024-09-11T08:30:07.307299+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'21","last_scrub_stamp":"2024-09-11T08:28:02.865748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:28:02.865748+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.016060689,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":197,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:06.823023+0000","last_change":"2024-09-11T08:28:54.875865+0000","last_active":"2024-09-11T08:30:06.823023+0000","last_peered":"2024-09-11T08:30:06.823023+0000","last_clean":"2024-09-11T08:30:06.823023+0000","last_became_active":"2024-09-11T08:24:28.359057+0000","last_became_peered":"2024-09-11T08:24:28.359057+0000","last_unstale":"2024-09-11T08:30:06.823023+0000","last_undegraded":"2024-09-11T08:30:06.823023+0000","last_fullsized":"2024-09-11T08:30:06.823023+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:28:54.875776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:28:54.875776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022324210000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":532,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:09.510412+0000","last_change":"2024-09-11T08:30:09.510347+0000","last_active":"2024-09-11T08:30:09.510412+0000","last_peered":"2024-09-11T08:30:09.510412+0000","last_clean":"2024-09-11T08:30:09.510412+0000","last_became_active":"2024-09-11T08:07:34.056606+0000","last_became_peered":"2024-09-11T08:07:34.056606+0000","last_unstale":"2024-09-11T08:30:09.510412+0000","last_undegraded":"2024-09-11T08:30:09.510412+0000","last_fullsized":"2024-09-11T08:30:09.510412+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:09.510248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:30:09.510248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:16.060314+0000","scrub_duration":0.0016592110000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":497,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.307064+0000","last_change":"2024-09-11T08:28:46.893924+0000","last_active":"2024-09-11T08:30:07.307064+0000","last_peered":"2024-09-11T08:30:07.307064+0000","last_clean":"2024-09-11T08:30:07.307064+0000","last_became_active":"2024-09-11T08:15:56.518884+0000","last_became_peered":"2024-09-11T08:15:56.518884+0000","last_unstale":"2024-09-11T08:30:07.307064+0000","last_undegraded":"2024-09-11T08:30:07.307064+0000","last_fullsized":"2024-09-11T08:30:07.307064+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:28:46.893851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:18:32.278257+0000","last_clean_scrub_stamp":"2024-09-11T08:28:46.893851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0085054269999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"445'15818","reported_seq":15913,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:06.817935+0000","last_change":"2024-09-11T08:29:58.794399+0000","last_active":"2024-09-11T08:30:06.817935+0000","last_peered":"2024-09-11T08:30:06.817935+0000","last_clean":"2024-09-11T08:30:06.817935+0000","last_became_active":"2024-09-11T08:26:28.083033+0000","last_became_peered":"2024-09-11T08:26:28.083033+0000","last_unstale":"2024-09-11T08:30:06.817935+0000","last_undegraded":"2024-09-11T08:30:06.817935+0000","last_fullsized":"2024-09-11T08:30:06.817935+0000","mapping_epoch":382,"log_start":"428'10700","ondisk_log_start":"428'10700","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"444'15517","last_scrub_stamp":"2024-09-11T08:29:58.794323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:29:58.794323+0000","objects_scrubbed":309,"log_size":5118,"log_dups_size":0,"ondisk_log_size":5118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:18.863921+0000","scrub_duration":0.110288894,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15818,"num_write_kb":506176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"405'51","reported_seq":660,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.476043+0000","last_change":"2024-09-11T08:29:19.302174+0000","last_active":"2024-09-11T08:30:07.476043+0000","last_peered":"2024-09-11T08:30:07.476043+0000","last_clean":"2024-09-11T08:30:07.476043+0000","last_became_active":"2024-09-11T08:15:56.526961+0000","last_became_peered":"2024-09-11T08:15:56.526961+0000","last_unstale":"2024-09-11T08:30:07.476043+0000","last_undegraded":"2024-09-11T08:30:07.476043+0000","last_fullsized":"2024-09-11T08:30:07.476043+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'51","last_scrub_stamp":"2024-09-11T08:29:19.302094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:29:19.302094+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:24.531066+0000","scrub_duration":0.0045766959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":339,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:10.617911+0000","last_change":"2024-09-11T08:30:10.617869+0000","last_active":"2024-09-11T08:30:10.617911+0000","last_peered":"2024-09-11T08:30:10.617911+0000","last_clean":"2024-09-11T08:30:10.617911+0000","last_became_active":"2024-09-11T08:24:28.353720+0000","last_became_peered":"2024-09-11T08:24:28.353720+0000","last_unstale":"2024-09-11T08:30:10.617911+0000","last_undegraded":"2024-09-11T08:30:10.617911+0000","last_fullsized":"2024-09-11T08:30:10.617911+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:10.617803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:44.385372+0000","last_clean_scrub_stamp":"2024-09-11T08:30:10.617803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:30.349657+0000","scrub_duration":0.0015332709999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":552,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.258803+0000","last_change":"2024-09-11T08:29:11.612083+0000","last_active":"2024-09-11T08:30:07.258803+0000","last_peered":"2024-09-11T08:30:07.258803+0000","last_clean":"2024-09-11T08:30:07.258803+0000","last_became_active":"2024-09-11T08:24:28.365792+0000","last_became_peered":"2024-09-11T08:24:28.365792+0000","last_unstale":"2024-09-11T08:30:07.258803+0000","last_undegraded":"2024-09-11T08:30:07.258803+0000","last_fullsized":"2024-09-11T08:30:07.258803+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:11.611140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:06.546573+0000","last_clean_scrub_stamp":"2024-09-11T08:29:11.611140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:30.173424+0000","scrub_duration":0.006733915,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":299,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:06.817501+0000","last_change":"2024-09-11T08:28:55.493273+0000","last_active":"2024-09-11T08:30:06.817501+0000","last_peered":"2024-09-11T08:30:06.817501+0000","last_clean":"2024-09-11T08:30:06.817501+0000","last_became_active":"2024-09-11T08:15:56.518169+0000","last_became_peered":"2024-09-11T08:15:56.518169+0000","last_unstale":"2024-09-11T08:30:06.817501+0000","last_undegraded":"2024-09-11T08:30:06.817501+0000","last_fullsized":"2024-09-11T08:30:06.817501+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:28:55.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:28:55.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0069241010000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"445'15782","reported_seq":16004,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.307479+0000","last_change":"2024-09-11T08:29:13.835644+0000","last_active":"2024-09-11T08:30:07.307479+0000","last_peered":"2024-09-11T08:30:07.307479+0000","last_clean":"2024-09-11T08:30:07.307479+0000","last_became_active":"2024-09-11T08:26:27.764636+0000","last_became_peered":"2024-09-11T08:26:27.764636+0000","last_unstale":"2024-09-11T08:30:07.307479+0000","last_undegraded":"2024-09-11T08:30:07.307479+0000","last_fullsized":"2024-09-11T08:30:07.307479+0000","mapping_epoch":382,"log_start":"422'9600","ondisk_log_start":"422'9600","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"432'12360","last_scrub_stamp":"2024-09-11T08:29:13.835554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:29:13.835554+0000","objects_scrubbed":3496,"log_size":6182,"log_dups_size":0,"ondisk_log_size":6182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:20.929839+0000","scrub_duration":1.8879387159999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15782,"num_write_kb":505024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"405'29","reported_seq":412,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.080813+0000","last_change":"2024-09-11T08:27:22.427676+0000","last_active":"2024-09-11T08:30:07.080813+0000","last_peered":"2024-09-11T08:30:07.080813+0000","last_clean":"2024-09-11T08:30:07.080813+0000","last_became_active":"2024-09-11T08:23:22.686387+0000","last_became_peered":"2024-09-11T08:23:22.686387+0000","last_unstale":"2024-09-11T08:30:07.080813+0000","last_undegraded":"2024-09-11T08:30:07.080813+0000","last_fullsized":"2024-09-11T08:30:07.080813+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'23","last_scrub_stamp":"2024-09-11T08:27:22.427580+0000","last_deep_scrub":"145'17","last_deep_scrub_stamp":"2024-09-11T08:13:08.160848+0000","last_clean_scrub_stamp":"2024-09-11T08:27:22.427580+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.17013154699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":309,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:06.818217+0000","last_change":"2024-09-11T08:29:02.433750+0000","last_active":"2024-09-11T08:30:06.818217+0000","last_peered":"2024-09-11T08:30:06.818217+0000","last_clean":"2024-09-11T08:30:06.818217+0000","last_became_active":"2024-09-11T08:15:56.525845+0000","last_became_peered":"2024-09-11T08:15:56.525845+0000","last_unstale":"2024-09-11T08:30:06.818217+0000","last_undegraded":"2024-09-11T08:30:06.818217+0000","last_fullsized":"2024-09-11T08:30:06.818217+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:02.433672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:21:06.262572+0000","last_clean_scrub_stamp":"2024-09-11T08:29:02.433672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:12.363719+0000","scrub_duration":0.0082380349999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":502,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.258676+0000","last_change":"2024-09-11T08:29:03.432065+0000","last_active":"2024-09-11T08:30:07.258676+0000","last_peered":"2024-09-11T08:30:07.258676+0000","last_clean":"2024-09-11T08:30:07.258676+0000","last_became_active":"2024-09-11T08:07:34.057479+0000","last_became_peered":"2024-09-11T08:07:34.057479+0000","last_unstale":"2024-09-11T08:30:07.258676+0000","last_undegraded":"2024-09-11T08:30:07.258676+0000","last_fullsized":"2024-09-11T08:30:07.258676+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:03.431977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:29:03.431977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:17.279703+0000","scrub_duration":0.011606712999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":518,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:06.822456+0000","last_change":"2024-09-11T08:29:01.755692+0000","last_active":"2024-09-11T08:30:06.822456+0000","last_peered":"2024-09-11T08:30:06.822456+0000","last_clean":"2024-09-11T08:30:06.822456+0000","last_became_active":"2024-09-11T08:15:56.522624+0000","last_became_peered":"2024-09-11T08:15:56.522624+0000","last_unstale":"2024-09-11T08:30:06.822456+0000","last_undegraded":"2024-09-11T08:30:06.822456+0000","last_fullsized":"2024-09-11T08:30:06.822456+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:01.755554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:29:01.755554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:13.741386+0000","scrub_duration":0.0037540439999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"445'15706","reported_seq":15789,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.081059+0000","last_change":"2024-09-11T08:30:06.099731+0000","last_active":"2024-09-11T08:30:07.081059+0000","last_peered":"2024-09-11T08:30:07.081059+0000","last_clean":"2024-09-11T08:30:07.081059+0000","last_became_active":"2024-09-11T08:26:28.083190+0000","last_became_peered":"2024-09-11T08:26:28.083190+0000","last_unstale":"2024-09-11T08:30:07.081059+0000","last_undegraded":"2024-09-11T08:30:07.081059+0000","last_fullsized":"2024-09-11T08:30:07.081059+0000","mapping_epoch":382,"log_start":"428'10700","ondisk_log_start":"428'10700","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"445'15706","last_scrub_stamp":"2024-09-11T08:30:06.099622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:26.740307+0000","last_clean_scrub_stamp":"2024-09-11T08:30:06.099622+0000","objects_scrubbed":0,"log_size":5006,"log_dups_size":0,"ondisk_log_size":5006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:35.603772+0000","scrub_duration":0.009009352,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15706,"num_write_kb":502592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"405'39","reported_seq":560,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.596402+0000","last_change":"2024-09-11T08:30:11.596345+0000","last_active":"2024-09-11T08:30:11.596402+0000","last_peered":"2024-09-11T08:30:11.596402+0000","last_clean":"2024-09-11T08:30:11.596402+0000","last_became_active":"2024-09-11T08:24:28.353397+0000","last_became_peered":"2024-09-11T08:24:28.353397+0000","last_unstale":"2024-09-11T08:30:11.596402+0000","last_undegraded":"2024-09-11T08:30:11.596402+0000","last_fullsized":"2024-09-11T08:30:11.596402+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'39","last_scrub_stamp":"2024-09-11T08:30:11.596278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:30:11.596278+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:28.358869+0000","scrub_duration":0.0031679740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":305,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:06.818382+0000","last_change":"2024-09-11T08:28:45.319406+0000","last_active":"2024-09-11T08:30:06.818382+0000","last_peered":"2024-09-11T08:30:06.818382+0000","last_clean":"2024-09-11T08:30:06.818382+0000","last_became_active":"2024-09-11T08:15:56.515328+0000","last_became_peered":"2024-09-11T08:15:56.515328+0000","last_unstale":"2024-09-11T08:30:06.818382+0000","last_undegraded":"2024-09-11T08:30:06.818382+0000","last_fullsized":"2024-09-11T08:30:06.818382+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-11T08:28:45.319230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:28:45.319230+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0062126519999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"209'11","reported_seq":474,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:08.845482+0000","last_change":"2024-09-11T08:29:04.443432+0000","last_active":"2024-09-11T08:30:08.845482+0000","last_peered":"2024-09-11T08:30:08.845482+0000","last_clean":"2024-09-11T08:30:08.845482+0000","last_became_active":"2024-09-11T08:15:56.524513+0000","last_became_peered":"2024-09-11T08:15:56.524513+0000","last_unstale":"2024-09-11T08:30:08.845482+0000","last_undegraded":"2024-09-11T08:30:08.845482+0000","last_fullsized":"2024-09-11T08:30:08.845482+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"209'11","last_scrub_stamp":"2024-09-11T08:29:04.443335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:29:04.443335+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:21.877030+0000","scrub_duration":0.0041068340000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":187,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.475181+0000","last_change":"2024-09-11T08:28:46.374746+0000","last_active":"2024-09-11T08:30:07.475181+0000","last_peered":"2024-09-11T08:30:07.475181+0000","last_clean":"2024-09-11T08:30:07.475181+0000","last_became_active":"2024-09-11T08:24:28.360658+0000","last_became_peered":"2024-09-11T08:24:28.360658+0000","last_unstale":"2024-09-11T08:30:07.475181+0000","last_undegraded":"2024-09-11T08:30:07.475181+0000","last_fullsized":"2024-09-11T08:30:07.475181+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:28:46.374674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:03.484755+0000","last_clean_scrub_stamp":"2024-09-11T08:28:46.374674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045403539999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":546,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.475356+0000","last_change":"2024-09-11T08:29:17.333213+0000","last_active":"2024-09-11T08:30:07.475356+0000","last_peered":"2024-09-11T08:30:07.475356+0000","last_clean":"2024-09-11T08:30:07.475356+0000","last_became_active":"2024-09-11T08:24:28.357401+0000","last_became_peered":"2024-09-11T08:24:28.357401+0000","last_unstale":"2024-09-11T08:30:07.475356+0000","last_undegraded":"2024-09-11T08:30:07.475356+0000","last_fullsized":"2024-09-11T08:30:07.475356+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:17.333163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:29:17.333163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:32.873535+0000","scrub_duration":0.0015736540000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"405'21","reported_seq":568,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:06.822979+0000","last_change":"2024-09-11T08:29:15.625135+0000","last_active":"2024-09-11T08:30:06.822979+0000","last_peered":"2024-09-11T08:30:06.822979+0000","last_clean":"2024-09-11T08:30:06.822979+0000","last_became_active":"2024-09-11T08:15:56.526325+0000","last_became_peered":"2024-09-11T08:15:56.526325+0000","last_unstale":"2024-09-11T08:30:06.822979+0000","last_undegraded":"2024-09-11T08:30:06.822979+0000","last_fullsized":"2024-09-11T08:30:06.822979+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'21","last_scrub_stamp":"2024-09-11T08:29:15.624982+0000","last_deep_scrub":"405'21","last_deep_scrub_stamp":"2024-09-11T08:29:15.624982+0000","last_clean_scrub_stamp":"2024-09-11T08:29:15.624982+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:33.209933+0000","scrub_duration":0.010445571000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":197,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:12.778533+0000","last_change":"2024-09-11T08:30:12.778506+0000","last_active":"2024-09-11T08:30:12.778533+0000","last_peered":"2024-09-11T08:30:12.778533+0000","last_clean":"2024-09-11T08:30:12.778533+0000","last_became_active":"2024-09-11T08:24:28.358300+0000","last_became_peered":"2024-09-11T08:24:28.358300+0000","last_unstale":"2024-09-11T08:30:12.778533+0000","last_undegraded":"2024-09-11T08:30:12.778533+0000","last_fullsized":"2024-09-11T08:30:12.778533+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:12.778451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:53.548967+0000","last_clean_scrub_stamp":"2024-09-11T08:30:12.778451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:30.240863+0000","scrub_duration":0.001203455,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":434,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:06.818548+0000","last_change":"2024-09-11T08:29:20.429043+0000","last_active":"2024-09-11T08:30:06.818548+0000","last_peered":"2024-09-11T08:30:06.818548+0000","last_clean":"2024-09-11T08:30:06.818548+0000","last_became_active":"2024-09-11T08:14:24.790678+0000","last_became_peered":"2024-09-11T08:14:24.790678+0000","last_unstale":"2024-09-11T08:30:06.818548+0000","last_undegraded":"2024-09-11T08:30:06.818548+0000","last_fullsized":"2024-09-11T08:30:06.818548+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:20.428966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:29:20.428966+0000","last_clean_scrub_stamp":"2024-09-11T08:29:20.428966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:46.351908+0000","scrub_duration":0.005613697,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"405'25","reported_seq":559,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:09.312588+0000","last_change":"2024-09-11T08:30:09.312537+0000","last_active":"2024-09-11T08:30:09.312588+0000","last_peered":"2024-09-11T08:30:09.312588+0000","last_clean":"2024-09-11T08:30:09.312588+0000","last_became_active":"2024-09-11T08:15:56.514793+0000","last_became_peered":"2024-09-11T08:15:56.514793+0000","last_unstale":"2024-09-11T08:30:09.312588+0000","last_undegraded":"2024-09-11T08:30:09.312588+0000","last_fullsized":"2024-09-11T08:30:09.312588+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'25","last_scrub_stamp":"2024-09-11T08:30:09.312479+0000","last_deep_scrub":"84'13","last_deep_scrub_stamp":"2024-09-11T08:08:57.835634+0000","last_clean_scrub_stamp":"2024-09-11T08:30:09.312479+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:23.223570+0000","scrub_duration":0.0022105110000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":457,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:06.817542+0000","last_change":"2024-09-11T08:28:56.500137+0000","last_active":"2024-09-11T08:30:06.817542+0000","last_peered":"2024-09-11T08:30:06.817542+0000","last_clean":"2024-09-11T08:30:06.817542+0000","last_became_active":"2024-09-11T08:14:24.786021+0000","last_became_peered":"2024-09-11T08:14:24.786021+0000","last_unstale":"2024-09-11T08:30:06.817542+0000","last_undegraded":"2024-09-11T08:30:06.817542+0000","last_fullsized":"2024-09-11T08:30:06.817542+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:28:56.500057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:30.471620+0000","last_clean_scrub_stamp":"2024-09-11T08:28:56.500057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:23.579349+0000","scrub_duration":0.0062781299999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":520,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:06.822958+0000","last_change":"2024-09-11T08:29:57.839609+0000","last_active":"2024-09-11T08:30:06.822958+0000","last_peered":"2024-09-11T08:30:06.822958+0000","last_clean":"2024-09-11T08:30:06.822958+0000","last_became_active":"2024-09-11T08:15:56.525717+0000","last_became_peered":"2024-09-11T08:15:56.525717+0000","last_unstale":"2024-09-11T08:30:06.822958+0000","last_undegraded":"2024-09-11T08:30:06.822958+0000","last_fullsized":"2024-09-11T08:30:06.822958+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:57.839509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:04.370734+0000","last_clean_scrub_stamp":"2024-09-11T08:29:57.839509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:23.592176+0000","scrub_duration":0.0070850310000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":520,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.258959+0000","last_change":"2024-09-11T08:29:12.626802+0000","last_active":"2024-09-11T08:30:07.258959+0000","last_peered":"2024-09-11T08:30:07.258959+0000","last_clean":"2024-09-11T08:30:07.258959+0000","last_became_active":"2024-09-11T08:15:56.525120+0000","last_became_peered":"2024-09-11T08:15:56.525120+0000","last_unstale":"2024-09-11T08:30:07.258959+0000","last_undegraded":"2024-09-11T08:30:07.258959+0000","last_fullsized":"2024-09-11T08:30:07.258959+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:12.626731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:54.184372+0000","last_clean_scrub_stamp":"2024-09-11T08:29:12.626731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:34.316820+0000","scrub_duration":0.0092815109999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"405'25","reported_seq":245,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:06.817833+0000","last_change":"2024-09-11T08:30:05.609094+0000","last_active":"2024-09-11T08:30:06.817833+0000","last_peered":"2024-09-11T08:30:06.817833+0000","last_clean":"2024-09-11T08:30:06.817833+0000","last_became_active":"2024-09-11T08:24:28.353043+0000","last_became_peered":"2024-09-11T08:24:28.353043+0000","last_unstale":"2024-09-11T08:30:06.817833+0000","last_undegraded":"2024-09-11T08:30:06.817833+0000","last_fullsized":"2024-09-11T08:30:06.817833+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'25","last_scrub_stamp":"2024-09-11T08:30:05.609036+0000","last_deep_scrub":"313'19","last_deep_scrub_stamp":"2024-09-11T08:22:03.458402+0000","last_clean_scrub_stamp":"2024-09-11T08:30:05.609036+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:25.904857+0000","scrub_duration":0.0019394670000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":513,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:10.269737+0000","last_change":"2024-09-11T08:30:10.269699+0000","last_active":"2024-09-11T08:30:10.269737+0000","last_peered":"2024-09-11T08:30:10.269737+0000","last_clean":"2024-09-11T08:30:10.269737+0000","last_became_active":"2024-09-11T08:15:56.513278+0000","last_became_peered":"2024-09-11T08:15:56.513278+0000","last_unstale":"2024-09-11T08:30:10.269737+0000","last_undegraded":"2024-09-11T08:30:10.269737+0000","last_fullsized":"2024-09-11T08:30:10.269737+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:10.269641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:52.661591+0000","last_clean_scrub_stamp":"2024-09-11T08:30:10.269641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:31.914817+0000","scrub_duration":0.001664865,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"239'13","reported_seq":771,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:08.845531+0000","last_change":"2024-09-11T08:27:58.219794+0000","last_active":"2024-09-11T08:30:08.845531+0000","last_peered":"2024-09-11T08:30:08.845531+0000","last_clean":"2024-09-11T08:30:08.845531+0000","last_became_active":"2024-09-11T08:17:37.558224+0000","last_became_peered":"2024-09-11T08:17:37.558224+0000","last_unstale":"2024-09-11T08:30:08.845531+0000","last_undegraded":"2024-09-11T08:30:08.845531+0000","last_fullsized":"2024-09-11T08:30:08.845531+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'13","last_scrub_stamp":"2024-09-11T08:27:58.219700+0000","last_deep_scrub":"222'12","last_deep_scrub_stamp":"2024-09-11T08:16:52.592856+0000","last_clean_scrub_stamp":"2024-09-11T08:27:58.219700+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.01448953,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":435,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.259187+0000","last_change":"2024-09-11T08:29:58.653531+0000","last_active":"2024-09-11T08:30:07.259187+0000","last_peered":"2024-09-11T08:30:07.259187+0000","last_clean":"2024-09-11T08:30:07.259187+0000","last_became_active":"2024-09-11T08:24:28.357799+0000","last_became_peered":"2024-09-11T08:24:28.357799+0000","last_unstale":"2024-09-11T08:30:07.259187+0000","last_undegraded":"2024-09-11T08:30:07.259187+0000","last_fullsized":"2024-09-11T08:30:07.259187+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:58.653393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:53.175994+0000","last_clean_scrub_stamp":"2024-09-11T08:29:58.653393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:17.708980+0000","scrub_duration":0.0087291079999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"405'30","reported_seq":384,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.258895+0000","last_change":"2024-09-11T08:29:15.645126+0000","last_active":"2024-09-11T08:30:07.258895+0000","last_peered":"2024-09-11T08:30:07.258895+0000","last_clean":"2024-09-11T08:30:07.258895+0000","last_became_active":"2024-09-11T08:17:37.554357+0000","last_became_peered":"2024-09-11T08:17:37.554357+0000","last_unstale":"2024-09-11T08:30:07.258895+0000","last_undegraded":"2024-09-11T08:30:07.258895+0000","last_fullsized":"2024-09-11T08:30:07.258895+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'30","last_scrub_stamp":"2024-09-11T08:29:15.645039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:29:15.645039+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:21.270884+0000","scrub_duration":0.021832190000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":299,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:06.818723+0000","last_change":"2024-09-11T08:27:56.258234+0000","last_active":"2024-09-11T08:30:06.818723+0000","last_peered":"2024-09-11T08:30:06.818723+0000","last_clean":"2024-09-11T08:30:06.818723+0000","last_became_active":"2024-09-11T08:15:56.518515+0000","last_became_peered":"2024-09-11T08:15:56.518515+0000","last_unstale":"2024-09-11T08:30:06.818723+0000","last_undegraded":"2024-09-11T08:30:06.818723+0000","last_fullsized":"2024-09-11T08:30:06.818723+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:56.258011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:23:42.199527+0000","last_clean_scrub_stamp":"2024-09-11T08:27:56.258011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022527070000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"354'13","reported_seq":296,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:10.835685+0000","last_change":"2024-09-11T08:27:09.973738+0000","last_active":"2024-09-11T08:30:10.835685+0000","last_peered":"2024-09-11T08:30:10.835685+0000","last_clean":"2024-09-11T08:30:10.835685+0000","last_became_active":"2024-09-11T08:24:28.353115+0000","last_became_peered":"2024-09-11T08:24:28.353115+0000","last_unstale":"2024-09-11T08:30:10.835685+0000","last_undegraded":"2024-09-11T08:30:10.835685+0000","last_fullsized":"2024-09-11T08:30:10.835685+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'13","last_scrub_stamp":"2024-09-11T08:27:09.973656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:27:09.973656+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018473879999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":497,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.307231+0000","last_change":"2024-09-11T08:29:05.959570+0000","last_active":"2024-09-11T08:30:07.307231+0000","last_peered":"2024-09-11T08:30:07.307231+0000","last_clean":"2024-09-11T08:30:07.307231+0000","last_became_active":"2024-09-11T08:15:56.513949+0000","last_became_peered":"2024-09-11T08:15:56.513949+0000","last_unstale":"2024-09-11T08:30:07.307231+0000","last_undegraded":"2024-09-11T08:30:07.307231+0000","last_fullsized":"2024-09-11T08:30:07.307231+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:05.957912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:29:05.957912+0000","last_clean_scrub_stamp":"2024-09-11T08:29:05.957912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:27.228578+0000","scrub_duration":0.0062029260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"405'27","reported_seq":420,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.305458+0000","last_change":"2024-09-11T08:30:11.305412+0000","last_active":"2024-09-11T08:30:11.305458+0000","last_peered":"2024-09-11T08:30:11.305458+0000","last_clean":"2024-09-11T08:30:11.305458+0000","last_became_active":"2024-09-11T08:23:22.688993+0000","last_became_peered":"2024-09-11T08:23:22.688993+0000","last_unstale":"2024-09-11T08:30:11.305458+0000","last_undegraded":"2024-09-11T08:30:11.305458+0000","last_fullsized":"2024-09-11T08:30:11.305458+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'27","last_scrub_stamp":"2024-09-11T08:30:11.305355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:30:11.305355+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:19.774283+0000","scrub_duration":0.0026390200000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":348,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:06.817888+0000","last_change":"2024-09-11T08:29:12.448974+0000","last_active":"2024-09-11T08:30:06.817888+0000","last_peered":"2024-09-11T08:30:06.817888+0000","last_clean":"2024-09-11T08:30:06.817888+0000","last_became_active":"2024-09-11T08:23:22.682569+0000","last_became_peered":"2024-09-11T08:23:22.682569+0000","last_unstale":"2024-09-11T08:30:06.817888+0000","last_undegraded":"2024-09-11T08:30:06.817888+0000","last_fullsized":"2024-09-11T08:30:06.817888+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:12.448898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:29:12.448898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:21.712733+0000","scrub_duration":0.0016737830000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":522,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.080827+0000","last_change":"2024-09-11T08:28:47.755273+0000","last_active":"2024-09-11T08:30:07.080827+0000","last_peered":"2024-09-11T08:30:07.080827+0000","last_clean":"2024-09-11T08:30:07.080827+0000","last_became_active":"2024-09-11T08:24:28.355004+0000","last_became_peered":"2024-09-11T08:24:28.355004+0000","last_unstale":"2024-09-11T08:30:07.080827+0000","last_undegraded":"2024-09-11T08:30:07.080827+0000","last_fullsized":"2024-09-11T08:30:07.080827+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:28:47.755203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:28:47.755203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001837315,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":311,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.306886+0000","last_change":"2024-09-11T08:29:23.241188+0000","last_active":"2024-09-11T08:30:07.306886+0000","last_peered":"2024-09-11T08:30:07.306886+0000","last_clean":"2024-09-11T08:30:07.306886+0000","last_became_active":"2024-09-11T08:24:28.372049+0000","last_became_peered":"2024-09-11T08:24:28.372049+0000","last_unstale":"2024-09-11T08:30:07.306886+0000","last_undegraded":"2024-09-11T08:30:07.306886+0000","last_fullsized":"2024-09-11T08:30:07.306886+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:23.241016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:29:23.241016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:40.844841+0000","scrub_duration":0.0063927849999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"405'25","reported_seq":542,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.081060+0000","last_change":"2024-09-11T08:29:20.881222+0000","last_active":"2024-09-11T08:30:07.081060+0000","last_peered":"2024-09-11T08:30:07.081060+0000","last_clean":"2024-09-11T08:30:07.081060+0000","last_became_active":"2024-09-11T08:07:32.049368+0000","last_became_peered":"2024-09-11T08:07:32.049368+0000","last_unstale":"2024-09-11T08:30:07.081060+0000","last_undegraded":"2024-09-11T08:30:07.081060+0000","last_fullsized":"2024-09-11T08:30:07.081060+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'25","last_scrub_stamp":"2024-09-11T08:29:20.881147+0000","last_deep_scrub":"343'19","last_deep_scrub_stamp":"2024-09-11T08:23:44.640077+0000","last_clean_scrub_stamp":"2024-09-11T08:29:20.881147+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:42.708414+0000","scrub_duration":0.0058682450000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":299,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:06.817875+0000","last_change":"2024-09-11T08:28:05.289814+0000","last_active":"2024-09-11T08:30:06.817875+0000","last_peered":"2024-09-11T08:30:06.817875+0000","last_clean":"2024-09-11T08:30:06.817875+0000","last_became_active":"2024-09-11T08:15:56.521570+0000","last_became_peered":"2024-09-11T08:15:56.521570+0000","last_unstale":"2024-09-11T08:30:06.817875+0000","last_undegraded":"2024-09-11T08:30:06.817875+0000","last_fullsized":"2024-09-11T08:30:06.817875+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:28:05.289763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:28:05.289763+0000","last_clean_scrub_stamp":"2024-09-11T08:28:05.289763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037268660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":274,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.307152+0000","last_change":"2024-09-11T08:29:58.282988+0000","last_active":"2024-09-11T08:30:07.307152+0000","last_peered":"2024-09-11T08:30:07.307152+0000","last_clean":"2024-09-11T08:30:07.307152+0000","last_became_active":"2024-09-11T08:17:37.557645+0000","last_became_peered":"2024-09-11T08:17:37.557645+0000","last_unstale":"2024-09-11T08:30:07.307152+0000","last_undegraded":"2024-09-11T08:30:07.307152+0000","last_fullsized":"2024-09-11T08:30:07.307152+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:58.282759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:22:02.568761+0000","last_clean_scrub_stamp":"2024-09-11T08:29:58.282759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:27.885348+0000","scrub_duration":0.0099211319999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":510,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.306709+0000","last_change":"2024-09-11T08:30:02.281204+0000","last_active":"2024-09-11T08:30:07.306709+0000","last_peered":"2024-09-11T08:30:07.306709+0000","last_clean":"2024-09-11T08:30:07.306709+0000","last_became_active":"2024-09-11T08:24:28.363959+0000","last_became_peered":"2024-09-11T08:24:28.363959+0000","last_unstale":"2024-09-11T08:30:07.306709+0000","last_undegraded":"2024-09-11T08:30:07.306709+0000","last_fullsized":"2024-09-11T08:30:07.306709+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:02.281112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:09.973943+0000","last_clean_scrub_stamp":"2024-09-11T08:30:02.281112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:13.828016+0000","scrub_duration":0.018012968000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"405'27","reported_seq":589,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:06.822212+0000","last_change":"2024-09-11T08:28:55.841566+0000","last_active":"2024-09-11T08:30:06.822212+0000","last_peered":"2024-09-11T08:30:06.822212+0000","last_clean":"2024-09-11T08:30:06.822212+0000","last_became_active":"2024-09-11T08:15:56.520058+0000","last_became_peered":"2024-09-11T08:15:56.520058+0000","last_unstale":"2024-09-11T08:30:06.822212+0000","last_undegraded":"2024-09-11T08:30:06.822212+0000","last_fullsized":"2024-09-11T08:30:06.822212+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'27","last_scrub_stamp":"2024-09-11T08:28:55.841457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:28:55.841457+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:15.929978+0000","scrub_duration":0.012026871999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":402,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:06.817874+0000","last_change":"2024-09-11T08:29:00.479990+0000","last_active":"2024-09-11T08:30:06.817874+0000","last_peered":"2024-09-11T08:30:06.817874+0000","last_clean":"2024-09-11T08:30:06.817874+0000","last_became_active":"2024-09-11T08:24:28.356974+0000","last_became_peered":"2024-09-11T08:24:28.356974+0000","last_unstale":"2024-09-11T08:30:06.817874+0000","last_undegraded":"2024-09-11T08:30:06.817874+0000","last_fullsized":"2024-09-11T08:30:06.817874+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:00.479835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:29:00.479835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:11.964330+0000","scrub_duration":0.002146532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":309,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:06.817438+0000","last_change":"2024-09-11T08:29:06.470596+0000","last_active":"2024-09-11T08:30:06.817438+0000","last_peered":"2024-09-11T08:30:06.817438+0000","last_clean":"2024-09-11T08:30:06.817438+0000","last_became_active":"2024-09-11T08:15:56.523728+0000","last_became_peered":"2024-09-11T08:15:56.523728+0000","last_unstale":"2024-09-11T08:30:06.817438+0000","last_undegraded":"2024-09-11T08:30:06.817438+0000","last_fullsized":"2024-09-11T08:30:06.817438+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:06.470537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:11:34.159053+0000","last_clean_scrub_stamp":"2024-09-11T08:29:06.470537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:12.678975+0000","scrub_duration":0.0014249079999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":491,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:06.822624+0000","last_change":"2024-09-11T08:29:56.882574+0000","last_active":"2024-09-11T08:30:06.822624+0000","last_peered":"2024-09-11T08:30:06.822624+0000","last_clean":"2024-09-11T08:30:06.822624+0000","last_became_active":"2024-09-11T08:07:36.633368+0000","last_became_peered":"2024-09-11T08:07:36.633368+0000","last_unstale":"2024-09-11T08:30:06.822624+0000","last_undegraded":"2024-09-11T08:30:06.822624+0000","last_fullsized":"2024-09-11T08:30:06.822624+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:56.882497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:22:50.992034+0000","last_clean_scrub_stamp":"2024-09-11T08:29:56.882497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:13.644291+0000","scrub_duration":0.0057421800000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"405'14","reported_seq":505,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:06.822103+0000","last_change":"2024-09-11T08:28:56.806080+0000","last_active":"2024-09-11T08:30:06.822103+0000","last_peered":"2024-09-11T08:30:06.822103+0000","last_clean":"2024-09-11T08:30:06.822103+0000","last_became_active":"2024-09-11T08:07:32.045728+0000","last_became_peered":"2024-09-11T08:07:32.045728+0000","last_unstale":"2024-09-11T08:30:06.822103+0000","last_undegraded":"2024-09-11T08:30:06.822103+0000","last_fullsized":"2024-09-11T08:30:06.822103+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'14","last_scrub_stamp":"2024-09-11T08:28:56.805624+0000","last_deep_scrub":"312'10","last_deep_scrub_stamp":"2024-09-11T08:21:55.859866+0000","last_clean_scrub_stamp":"2024-09-11T08:28:56.805624+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:26.046082+0000","scrub_duration":0.019371645,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":523,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.270602+0000","last_change":"2024-09-11T08:30:11.270503+0000","last_active":"2024-09-11T08:30:11.270602+0000","last_peered":"2024-09-11T08:30:11.270602+0000","last_clean":"2024-09-11T08:30:11.270602+0000","last_became_active":"2024-09-11T08:24:28.357978+0000","last_became_peered":"2024-09-11T08:24:28.357978+0000","last_unstale":"2024-09-11T08:30:11.270602+0000","last_undegraded":"2024-09-11T08:30:11.270602+0000","last_fullsized":"2024-09-11T08:30:11.270602+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:11.270399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:30:11.270399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:30.279105+0000","scrub_duration":0.001777708,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":549,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.476285+0000","last_change":"2024-09-11T08:30:03.423705+0000","last_active":"2024-09-11T08:30:07.476285+0000","last_peered":"2024-09-11T08:30:07.476285+0000","last_clean":"2024-09-11T08:30:07.476285+0000","last_became_active":"2024-09-11T08:15:56.528607+0000","last_became_peered":"2024-09-11T08:15:56.528607+0000","last_unstale":"2024-09-11T08:30:07.476285+0000","last_undegraded":"2024-09-11T08:30:07.476285+0000","last_fullsized":"2024-09-11T08:30:07.476285+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:03.423649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:30:03.423649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:13.967310+0000","scrub_duration":0.001131574,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":292,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:06.817738+0000","last_change":"2024-09-11T08:27:16.194874+0000","last_active":"2024-09-11T08:30:06.817738+0000","last_peered":"2024-09-11T08:30:06.817738+0000","last_clean":"2024-09-11T08:30:06.817738+0000","last_became_active":"2024-09-11T08:15:56.521488+0000","last_became_peered":"2024-09-11T08:15:56.521488+0000","last_unstale":"2024-09-11T08:30:06.817738+0000","last_undegraded":"2024-09-11T08:30:06.817738+0000","last_fullsized":"2024-09-11T08:30:06.817738+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:16.194807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:20:18.827709+0000","last_clean_scrub_stamp":"2024-09-11T08:27:16.194807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021734469999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":520,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.081013+0000","last_change":"2024-09-11T08:27:17.179442+0000","last_active":"2024-09-11T08:30:07.081013+0000","last_peered":"2024-09-11T08:30:07.081013+0000","last_clean":"2024-09-11T08:30:07.081013+0000","last_became_active":"2024-09-11T08:24:28.354141+0000","last_became_peered":"2024-09-11T08:24:28.354141+0000","last_unstale":"2024-09-11T08:30:07.081013+0000","last_undegraded":"2024-09-11T08:30:07.081013+0000","last_fullsized":"2024-09-11T08:30:07.081013+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:17.179345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:50.585566+0000","last_clean_scrub_stamp":"2024-09-11T08:27:17.179345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0077697699999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":261,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.307280+0000","last_change":"2024-09-11T08:27:51.916658+0000","last_active":"2024-09-11T08:30:07.307280+0000","last_peered":"2024-09-11T08:30:07.307280+0000","last_clean":"2024-09-11T08:30:07.307280+0000","last_became_active":"2024-09-11T08:17:37.554262+0000","last_became_peered":"2024-09-11T08:17:37.554262+0000","last_unstale":"2024-09-11T08:30:07.307280+0000","last_undegraded":"2024-09-11T08:30:07.307280+0000","last_fullsized":"2024-09-11T08:30:07.307280+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:51.916565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:27:51.916565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012562804,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"405'32","reported_seq":613,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.476109+0000","last_change":"2024-09-11T08:29:07.366953+0000","last_active":"2024-09-11T08:30:07.476109+0000","last_peered":"2024-09-11T08:30:07.476109+0000","last_clean":"2024-09-11T08:30:07.476109+0000","last_became_active":"2024-09-11T08:24:28.360937+0000","last_became_peered":"2024-09-11T08:24:28.360937+0000","last_unstale":"2024-09-11T08:30:07.476109+0000","last_undegraded":"2024-09-11T08:30:07.476109+0000","last_fullsized":"2024-09-11T08:30:07.476109+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'32","last_scrub_stamp":"2024-09-11T08:29:07.366863+0000","last_deep_scrub":"405'32","last_deep_scrub_stamp":"2024-09-11T08:29:07.366863+0000","last_clean_scrub_stamp":"2024-09-11T08:29:07.366863+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:20.629764+0000","scrub_duration":0.0072573189999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":337,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-11T08:30:07.476214+0000","last_change":"2024-09-11T08:29:58.627283+0000","last_active":"2024-09-11T08:30:07.476214+0000","last_peered":"2024-09-11T08:30:07.476214+0000","last_clean":"2024-09-11T08:30:07.476214+0000","last_became_active":"2024-09-11T08:15:08.769133+0000","last_became_peered":"2024-09-11T08:15:08.769133+0000","last_unstale":"2024-09-11T08:30:07.476214+0000","last_undegraded":"2024-09-11T08:30:07.476214+0000","last_fullsized":"2024-09-11T08:30:07.476214+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:58.627201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:18:39.206016+0000","last_clean_scrub_stamp":"2024-09-11T08:29:58.627201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:59.524575+0000","scrub_duration":0.007062512,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":252094,"num_write_kb":8066945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87695,"ondisk_log_size":87695,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102,"ondisk_log_size":102,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1546,"num_read_kb":1303,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":4566,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3425,"internal_metadata":0},"log_size":1066,"ondisk_log_size":1066,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":953,"num_read_kb":948,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":168,"seq":721554505919,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":920780,"kb_used_data":1288,"kb_used_omap":7,"kb_used_meta":919352,"kb_avail":92824372,"statfs":{"total":95995035648,"available":95052156928,"internally_reserved":0,"allocated":1318912,"data_stored":958597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8027,"internal_metadata":941416613},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:29:49 2024","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":1.1060000000000001,"15min":2.419},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":2.2839999999999998,"5min":19.797999999999998,"15min":117.57599999999999},"last":0.498},{"interface":"front","average":{"1min":1.3009999999999999,"5min":1.1379999999999999,"15min":2.2999999999999998},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":4.3070000000000004,"5min":19.899000000000001,"15min":117.542},"last":0.60099999999999998}]},{"osd":1,"last update":"Wed Sep 11 08:29:49 2024","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.2569999999999999,"15min":1.8799999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.20899999999999999},"max":{"1min":4.7220000000000004,"5min":24.347999999999999,"15min":66.971000000000004},"last":0.47199999999999998},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.2150000000000001,"15min":1.702},"min":{"1min":0.246,"5min":0.23400000000000001,"15min":0.23000000000000001},"max":{"1min":3.8069999999999999,"5min":25.382000000000001,"15min":67.185000000000002},"last":0.56299999999999994}]},{"osd":2,"last update":"Wed Sep 11 08:29:49 2024","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":0.97099999999999997,"15min":1.04},"min":{"1min":0.25600000000000001,"5min":0.245,"15min":0.22700000000000001},"max":{"1min":9.3439999999999994,"5min":9.3439999999999994,"15min":19.097000000000001},"last":0.63},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.329,"15min":1.202},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":3.3199999999999998,"5min":7.7699999999999996,"15min":21.594000000000001},"last":0.41399999999999998}]},{"osd":3,"last update":"Wed Sep 11 08:29:49 2024","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.347,"15min":1.23},"min":{"1min":0.28399999999999997,"5min":0.27200000000000002,"15min":0.251},"max":{"1min":3.8300000000000001,"5min":24.315000000000001,"15min":24.315000000000001},"last":0.67000000000000004},{"interface":"front","average":{"1min":1.0880000000000001,"5min":1.147,"15min":1.206},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.251},"max":{"1min":3.516,"5min":24.864999999999998,"15min":24.864999999999998},"last":0.53500000000000003}]},{"osd":4,"last update":"Wed Sep 11 08:29:49 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.73999999999999999,"15min":0.83599999999999997},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":1.3500000000000001,"5min":5.7240000000000002,"15min":5.7240000000000002},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.66000000000000003,"15min":0.755},"min":{"1min":0.27900000000000003,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":2.367,"5min":4.2060000000000004,"15min":5.2649999999999997},"last":0.61699999999999999}]},{"osd":5,"last update":"Wed Sep 11 08:29:44 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.84899999999999998,"15min":1.2290000000000001},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":3.4470000000000001,"5min":6.3979999999999997,"15min":6.3979999999999997},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.68899999999999995,"15min":0.86799999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":2.5960000000000001,"5min":2.887,"15min":2.887},"last":0.64800000000000002}]},{"osd":6,"last update":"Wed Sep 11 08:29:49 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":1.3340000000000001,"15min":1.0009999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.248},"max":{"1min":2.2010000000000001,"5min":42.749000000000002,"15min":42.749000000000002},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.57599999999999996,"5min":1.4139999999999999,"15min":1.093},"min":{"1min":0.32300000000000001,"5min":0.311,"15min":0.25700000000000001},"max":{"1min":1.5900000000000001,"5min":41.914000000000001,"15min":41.914000000000001},"last":0.51800000000000002}]}]},{"osd":6,"up_from":45,"seq":193273528602,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":896892,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":895486,"kb_avail":92848260,"statfs":{"total":95995035648,"available":95076618240,"internally_reserved":0,"allocated":1302528,"data_stored":958420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":916978008},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:29:34 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":4.7800000000000002,"15min":3.1280000000000001},"min":{"1min":0.28999999999999998,"5min":0.21199999999999999,"15min":0.17999999999999999},"max":{"1min":3.649,"5min":331.14800000000002,"15min":331.14800000000002},"last":0.436},{"interface":"front","average":{"1min":1.1639999999999999,"5min":4.4930000000000003,"15min":2.98},"min":{"1min":0.30199999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":3.0939999999999999,"5min":331.31,"15min":331.31},"last":0.58399999999999996}]},{"osd":1,"last update":"Wed Sep 11 08:29:34 2024","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.5880000000000001,"15min":1.3859999999999999},"min":{"1min":0.35799999999999998,"5min":0.22900000000000001,"15min":0.214},"max":{"1min":5.0350000000000001,"5min":29.423999999999999,"15min":29.423999999999999},"last":0.49199999999999999},{"interface":"front","average":{"1min":1.5509999999999999,"5min":1.607,"15min":1.345},"min":{"1min":0.23499999999999999,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":4.4640000000000004,"5min":29.109000000000002,"15min":29.109000000000002},"last":0.443}]},{"osd":2,"last update":"Wed Sep 11 08:29:34 2024","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.6439999999999999,"15min":1.3129999999999999},"min":{"1min":0.25,"5min":0.25,"15min":0.23200000000000001},"max":{"1min":3.218,"5min":28.167999999999999,"15min":39.652999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.587,"15min":1.415},"min":{"1min":0.35199999999999998,"5min":0.251,"15min":0.251},"max":{"1min":4.0099999999999998,"5min":31.648,"15min":38.390999999999998},"last":0.56399999999999995}]},{"osd":3,"last update":"Wed Sep 11 08:29:34 2024","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":1.5660000000000001,"15min":1.25},"min":{"1min":0.30599999999999999,"5min":0.253,"15min":0.22700000000000001},"max":{"1min":1.774,"5min":12.414,"15min":12.414},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.95499999999999996,"5min":1.522,"15min":1.24},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":2.133,"5min":28.004999999999999,"15min":28.004999999999999},"last":0.52200000000000002}]},{"osd":4,"last update":"Wed Sep 11 08:29:34 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.90000000000000002,"15min":0.77500000000000002},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.20499999999999999},"max":{"1min":3.028,"5min":9.7560000000000002,"15min":9.7560000000000002},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.79900000000000004,"15min":0.74399999999999999},"min":{"1min":0.307,"5min":0.29099999999999998,"15min":0.23699999999999999},"max":{"1min":3.5659999999999998,"5min":3.5659999999999998,"15min":4.468},"last":0.32400000000000001}]},{"osd":5,"last update":"Wed Sep 11 08:29:34 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.97699999999999998,"15min":1.552},"min":{"1min":0.29499999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":3.1739999999999999,"5min":9.7989999999999995,"15min":9.7989999999999995},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.99199999999999999,"15min":1.3939999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":2.1019999999999999,"5min":7.2999999999999998,"15min":7.2999999999999998},"last":0.309}]},{"osd":7,"last update":"Wed Sep 11 08:29:46 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.2090000000000001,"15min":1.014},"min":{"1min":0.34599999999999997,"5min":0.25900000000000001,"15min":0.253},"max":{"1min":3.3889999999999998,"5min":7.5529999999999999,"15min":10.167999999999999},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.96099999999999997,"15min":0.82199999999999995},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.25700000000000001},"max":{"1min":2.887,"5min":3.6160000000000001,"15min":5.3579999999999997},"last":0.34100000000000003}]}]},{"osd":5,"up_from":353,"seq":1516123455558,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":360764,"kb_used_data":696,"kb_used_omap":4,"kb_used_meta":359931,"kb_avail":93384388,"statfs":{"total":95995035648,"available":95625613312,"internally_reserved":0,"allocated":712704,"data_stored":367613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":368569862},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:29:39 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.99299999999999999,"15min":1.6000000000000001},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":2.9710000000000001,"5min":19.379999999999999,"15min":19.379999999999999},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.96799999999999997,"5min":1.135,"15min":1.623},"min":{"1min":0.23599999999999999,"5min":0.218,"15min":0.218},"max":{"1min":2.6739999999999999,"5min":18.238,"15min":18.238},"last":0.34100000000000003}]},{"osd":1,"last update":"Wed Sep 11 08:29:39 2024","interfaces":[{"interface":"back","average":{"1min":4.8869999999999996,"5min":2.7749999999999999,"15min":1.6739999999999999},"min":{"1min":0.24199999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":85.728999999999999,"5min":85.728999999999999,"15min":85.728999999999999},"last":0.48299999999999998},{"interface":"front","average":{"1min":5.0979999999999999,"5min":2.8319999999999999,"15min":1.4159999999999999},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":85.668000000000006,"5min":85.668000000000006,"15min":85.668000000000006},"last":0.42099999999999999}]},{"osd":2,"last update":"Wed Sep 11 08:29:39 2024","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":1.4099999999999999,"15min":1.8799999999999999},"min":{"1min":0.26000000000000001,"5min":0.251,"15min":0.251},"max":{"1min":3.7599999999999998,"5min":18.350999999999999,"15min":18.350999999999999},"last":0.373},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.3200000000000001,"15min":1.74},"min":{"1min":0.27400000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":9.6489999999999991,"5min":18.381,"15min":18.381},"last":0.35899999999999999}]},{"osd":3,"last update":"Wed Sep 11 08:29:39 2024","interfaces":[{"interface":"back","average":{"1min":4.4080000000000004,"5min":2.3580000000000001,"15min":1.5680000000000001},"min":{"1min":0.27500000000000002,"5min":0.251,"15min":0.251},"max":{"1min":73.697000000000003,"5min":73.697000000000003,"15min":73.697000000000003},"last":0.53100000000000003},{"interface":"front","average":{"1min":4.2370000000000001,"5min":2.1309999999999998,"15min":1.28},"min":{"1min":0.35399999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":73.727999999999994,"5min":73.727999999999994,"15min":73.727999999999994},"last":0.51600000000000001}]},{"osd":4,"last update":"Wed Sep 11 08:29:39 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.75900000000000001,"15min":1.083},"min":{"1min":0.27000000000000002,"5min":0.249,"15min":0.249},"max":{"1min":2.1230000000000002,"5min":5.8630000000000004,"15min":5.8630000000000004},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.72999999999999998,"15min":1.1160000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":1.8700000000000001,"5min":5.8979999999999997,"15min":5.8979999999999997},"last":0.44600000000000001}]},{"osd":6,"last update":"Wed Sep 11 08:29:39 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.83799999999999997,"15min":1.3220000000000001},"min":{"1min":0.317,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":1.921,"5min":5.9779999999999998,"15min":5.9779999999999998},"last":0.5},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.84099999999999997,"15min":1.5669999999999999},"min":{"1min":0.32300000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.9990000000000001,"5min":5.6680000000000001,"15min":5.6680000000000001},"last":0.55000000000000004}]},{"osd":7,"last update":"Wed Sep 11 08:29:39 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.92800000000000005,"15min":1.153},"min":{"1min":0.29299999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":3.2320000000000002,"5min":7.8719999999999999,"15min":7.8719999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.94999999999999996,"15min":1.544},"min":{"1min":0.29899999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":3.2759999999999998,"5min":6.8739999999999997,"15min":6.8739999999999997},"last":0.56699999999999995}]}]},{"osd":4,"up_from":31,"seq":133143986468,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":894928,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":894078,"kb_avail":92850224,"statfs":{"total":95995035648,"available":95078629376,"internally_reserved":0,"allocated":733184,"data_stored":368927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":915536216},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:29:45 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":2.6219999999999999,"15min":1.948},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":2.3159999999999998,"5min":64.329999999999998,"15min":64.329999999999998},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.68200000000000005,"5min":2.4470000000000001,"15min":1.8759999999999999},"min":{"1min":0.24199999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.3900000000000001,"5min":66.819999999999993,"15min":66.819999999999993},"last":0.52800000000000002}]},{"osd":1,"last update":"Wed Sep 11 08:29:45 2024","interfaces":[{"interface":"back","average":{"1min":7.2530000000000001,"5min":2.2719999999999998,"15min":1.6619999999999999},"min":{"1min":0.27600000000000002,"5min":0.22,"15min":0.214},"max":{"1min":91.852000000000004,"5min":91.852000000000004,"15min":91.852000000000004},"last":0.61199999999999999},{"interface":"front","average":{"1min":6.7080000000000002,"5min":2.4209999999999998,"15min":1.7470000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":91.697000000000003,"5min":91.697000000000003,"15min":91.697000000000003},"last":0.65900000000000003}]},{"osd":2,"last update":"Wed Sep 11 08:29:45 2024","interfaces":[{"interface":"back","average":{"1min":1.819,"5min":2.1499999999999999,"15min":1.4910000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":13.710000000000001,"5min":54.317999999999998,"15min":54.317999999999998},"last":0.59799999999999998},{"interface":"front","average":{"1min":2.2469999999999999,"5min":2.2839999999999998,"15min":1.611},"min":{"1min":0.30299999999999999,"5min":0.246,"15min":0.23400000000000001},"max":{"1min":12.711,"5min":54.430999999999997,"15min":54.430999999999997},"last":0.64600000000000002}]},{"osd":3,"last update":"Wed Sep 11 08:29:45 2024","interfaces":[{"interface":"back","average":{"1min":1.8819999999999999,"5min":1.355,"15min":1.381},"min":{"1min":0.27700000000000002,"5min":0.247,"15min":0.247},"max":{"1min":12.811999999999999,"5min":12.811999999999999,"15min":30.321000000000002},"last":0.621},{"interface":"front","average":{"1min":2.1230000000000002,"5min":1.3360000000000001,"15min":1.2170000000000001},"min":{"1min":0.26700000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":12.978,"5min":12.978,"15min":30.620000000000001},"last":0.58499999999999996}]},{"osd":5,"last update":"Wed Sep 11 08:29:34 2024","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":0.80000000000000004,"15min":1.1739999999999999},"min":{"1min":0.25700000000000001,"5min":0.246,"15min":0.246},"max":{"1min":4.3410000000000002,"5min":5.1050000000000004,"15min":5.1050000000000004},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.84899999999999998,"15min":1.302},"min":{"1min":0.23799999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":1.3200000000000001,"5min":6.1150000000000002,"15min":6.1150000000000002},"last":0.433}]},{"osd":6,"last update":"Wed Sep 11 08:29:34 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.90400000000000003,"15min":0.89800000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.23200000000000001},"max":{"1min":3.036,"5min":6.4039999999999999,"15min":48.808},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.96199999999999997,"15min":0.95399999999999996},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.251},"max":{"1min":2.415,"5min":5.3159999999999998,"15min":48.746000000000002},"last":0.55200000000000005}]},{"osd":7,"last update":"Wed Sep 11 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.78500000000000003,"15min":0.73199999999999998},"min":{"1min":0.27500000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.96599999999999997,"5min":6.093,"15min":6.093},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.498,"5min":0.63100000000000001,"15min":0.70999999999999996},"min":{"1min":0.30099999999999999,"5min":0.29899999999999999,"15min":0.26800000000000002},"max":{"1min":1.1950000000000001,"5min":4.8259999999999996,"15min":4.8259999999999996},"last":0.49199999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182697,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":902044,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":901182,"kb_avail":92843108,"statfs":{"total":95995035648,"available":95071342592,"internally_reserved":0,"allocated":745472,"data_stored":368398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":922810712},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:29:38 2024","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":1.702,"15min":1.8500000000000001},"min":{"1min":0.25900000000000001,"5min":0.16300000000000001,"15min":0.14999999999999999},"max":{"1min":4.9139999999999997,"5min":24.616,"15min":60.469999999999999},"last":0.28599999999999998},{"interface":"front","average":{"1min":1.7450000000000001,"5min":1.7,"15min":1.732},"min":{"1min":0.24099999999999999,"5min":0.20599999999999999,"15min":0.17399999999999999},"max":{"1min":6.7089999999999996,"5min":23.640000000000001,"15min":60.488999999999997},"last":0.251}]},{"osd":1,"last update":"Wed Sep 11 08:29:38 2024","interfaces":[{"interface":"back","average":{"1min":2.2050000000000001,"5min":1.4550000000000001,"15min":1.6459999999999999},"min":{"1min":0.28999999999999998,"5min":0.20399999999999999,"15min":0.19600000000000001},"max":{"1min":7.5739999999999998,"5min":9.5619999999999994,"15min":60.578000000000003},"last":0.35999999999999999},{"interface":"front","average":{"1min":1.5680000000000001,"5min":1.361,"15min":1.6859999999999999},"min":{"1min":0.246,"5min":0.20899999999999999,"15min":0.19500000000000001},"max":{"1min":7.1399999999999997,"5min":7.1399999999999997,"15min":63.674999999999997},"last":0.27200000000000002}]},{"osd":2,"last update":"Wed Sep 11 08:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":2.0430000000000001,"15min":1.675},"min":{"1min":0.28000000000000003,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":2.1899999999999999,"5min":67.503,"15min":67.503},"last":0.32300000000000001},{"interface":"front","average":{"1min":1.5980000000000001,"5min":2.3140000000000001,"15min":1.768},"min":{"1min":0.26500000000000001,"5min":0.251,"15min":0.19900000000000001},"max":{"1min":7.1020000000000003,"5min":67.313999999999993,"15min":67.313999999999993},"last":0.34300000000000003}]},{"osd":4,"last update":"Wed Sep 11 08:29:47 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":1.196,"15min":1.4039999999999999},"min":{"1min":0.29199999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":2.2650000000000001,"5min":6.8680000000000003,"15min":61.290999999999997},"last":0.505},{"interface":"front","average":{"1min":1.423,"5min":1.2470000000000001,"15min":1.28},"min":{"1min":0.33500000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":4.8179999999999996,"5min":6.46,"15min":60.856000000000002},"last":0.65700000000000003}]},{"osd":5,"last update":"Wed Sep 11 08:29:38 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.306,"15min":1.665},"min":{"1min":0.34899999999999998,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":6.6189999999999998,"5min":7.3849999999999998,"15min":7.3849999999999998},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.97799999999999998,"5min":1.1759999999999999,"15min":1.22},"min":{"1min":0.39000000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":3.5049999999999999,"5min":6.835,"15min":6.835},"last":0.70199999999999996}]},{"osd":6,"last update":"Wed Sep 11 08:29:38 2024","interfaces":[{"interface":"back","average":{"1min":1.6759999999999999,"5min":1.4910000000000001,"15min":1.649},"min":{"1min":0.39300000000000002,"5min":0.32700000000000001,"15min":0.29699999999999999},"max":{"1min":6.6520000000000001,"5min":6.6520000000000001,"15min":60.817},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.94699999999999995,"5min":1.2050000000000001,"15min":1.669},"min":{"1min":0.314,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":2.2909999999999999,"5min":6.8970000000000002,"15min":61.264000000000003},"last":0.53300000000000003}]},{"osd":7,"last update":"Wed Sep 11 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":1.7010000000000001,"5min":1.321,"15min":1.677},"min":{"1min":0.441,"5min":0.33500000000000002,"15min":0.29899999999999999},"max":{"1min":6.6840000000000002,"5min":6.6840000000000002,"15min":60.847000000000001},"last":0.71299999999999997},{"interface":"front","average":{"1min":1.641,"5min":1.411,"15min":1.8240000000000001},"min":{"1min":0.36699999999999999,"5min":0.35899999999999999,"15min":0.33800000000000002},"max":{"1min":6.899,"5min":6.899,"15min":60.883000000000003},"last":0.495}]}]},{"osd":2,"up_from":18,"seq":77309411628,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":889980,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":889150,"kb_avail":92855172,"statfs":{"total":95995035648,"available":95083696128,"internally_reserved":0,"allocated":712704,"data_stored":367525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":910489944},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:30:10 2024","interfaces":[{"interface":"back","average":{"1min":7.7149999999999999,"5min":3.585,"15min":3.1840000000000002},"min":{"1min":0.27500000000000002,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":102.18000000000001,"5min":102.18000000000001,"15min":102.18000000000001},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.91900000000000004,"5min":2.0739999999999998,"15min":2.9350000000000001},"min":{"1min":0.29099999999999998,"5min":0.20599999999999999,"15min":0.17999999999999999},"max":{"1min":4.2510000000000003,"5min":81.391000000000005,"15min":93.790999999999997},"last":0.36499999999999999}]},{"osd":1,"last update":"Wed Sep 11 08:30:10 2024","interfaces":[{"interface":"back","average":{"1min":7.907,"5min":2.7069999999999999,"15min":2.0819999999999999},"min":{"1min":0.20399999999999999,"5min":0.19800000000000001,"15min":0.182},"max":{"1min":102.134,"5min":102.134,"15min":102.134},"last":0.38700000000000001},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.206,"15min":1.577},"min":{"1min":0.30499999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":4.1799999999999997,"5min":6.7220000000000004,"15min":53.082999999999998},"last":0.72899999999999998}]},{"osd":3,"last update":"Wed Sep 11 08:29:34 2024","interfaces":[{"interface":"back","average":{"1min":1.97,"5min":2.169,"15min":1.7350000000000001},"min":{"1min":0.41299999999999998,"5min":0.22,"15min":0.21299999999999999},"max":{"1min":6.2549999999999999,"5min":54.289999999999999,"15min":54.289999999999999},"last":0.32300000000000001},{"interface":"front","average":{"1min":1.4039999999999999,"5min":2.0699999999999998,"15min":1.6679999999999999},"min":{"1min":0.34499999999999997,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":6.6470000000000002,"5min":53.927,"15min":53.927},"last":0.41999999999999998}]},{"osd":4,"last update":"Wed Sep 11 08:29:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.169,"15min":1.103},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.24099999999999999},"max":{"1min":6.6920000000000002,"5min":6.6920000000000002,"15min":6.6920000000000002},"last":0.54600000000000004},{"interface":"front","average":{"1min":1.0900000000000001,"5min":1.0429999999999999,"15min":1.107},"min":{"1min":0.39700000000000002,"5min":0.311,"15min":0.27500000000000002},"max":{"1min":4.3070000000000004,"5min":4.6369999999999996,"15min":7.8970000000000002},"last":0.61899999999999999}]},{"osd":5,"last update":"Wed Sep 11 08:29:34 2024","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":1.0529999999999999,"15min":1.5249999999999999},"min":{"1min":0.36599999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":4.2270000000000003,"5min":4.5839999999999996,"15min":4.5839999999999996},"last":0.68200000000000005},{"interface":"front","average":{"1min":1.452,"5min":1.1619999999999999,"15min":1.3720000000000001},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":6.6589999999999998,"5min":6.6589999999999998,"15min":6.6589999999999998},"last":0.47999999999999998}]},{"osd":6,"last update":"Wed Sep 11 08:29:44 2024","interfaces":[{"interface":"back","average":{"1min":1.8149999999999999,"5min":1.3380000000000001,"15min":1.2150000000000001},"min":{"1min":0.371,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":6.3440000000000003,"5min":6.3440000000000003,"15min":8.0350000000000001},"last":0.59799999999999998},{"interface":"front","average":{"1min":1.085,"5min":1.206,"15min":1.1850000000000001},"min":{"1min":0.39400000000000002,"5min":0.375,"15min":0.32500000000000001},"max":{"1min":4.4390000000000001,"5min":5.9619999999999997,"15min":7.8710000000000004},"last":0.65500000000000003}]},{"osd":7,"last update":"Wed Sep 11 08:29:48 2024","interfaces":[{"interface":"back","average":{"1min":1.5629999999999999,"5min":1.3819999999999999,"15min":1.3640000000000001},"min":{"1min":0.41699999999999998,"5min":0.312,"15min":0.27500000000000002},"max":{"1min":7.6950000000000003,"5min":11.279,"15min":35.274999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.3049999999999999,"15min":1.4330000000000001},"min":{"1min":0.44,"5min":0.33600000000000002,"15min":0.32800000000000001},"max":{"1min":4.4660000000000002,"5min":11.305,"15min":35.265000000000001},"last":0.70499999999999996}]}]},{"osd":1,"up_from":13,"seq":55834575154,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":890980,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":890238,"kb_avail":92854172,"statfs":{"total":95995035648,"available":95082672128,"internally_reserved":0,"allocated":749568,"data_stored":368541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":911604056},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:29:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":1.337,"15min":1.5189999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":5.0110000000000001,"5min":14.045,"15min":63.932000000000002},"last":0.20300000000000001},{"interface":"front","average":{"1min":1.5900000000000001,"5min":1.5940000000000001,"15min":1.869},"min":{"1min":0.222,"5min":0.189,"15min":0.16800000000000001},"max":{"1min":5.1120000000000001,"5min":14.127000000000001,"15min":64.269000000000005},"last":0.26300000000000001}]},{"osd":2,"last update":"Wed Sep 11 08:29:24 2024","interfaces":[{"interface":"back","average":{"1min":2.2229999999999999,"5min":1.5700000000000001,"15min":1.3959999999999999},"min":{"1min":0.44800000000000001,"5min":0.19800000000000001,"15min":0.189},"max":{"1min":5.1959999999999997,"5min":13.928000000000001,"15min":20.925999999999998},"last":0.23699999999999999},{"interface":"front","average":{"1min":2.0899999999999999,"5min":1.677,"15min":1.3759999999999999},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19},"max":{"1min":5.3700000000000001,"5min":16.099,"15min":20.812000000000001},"last":0.28299999999999997}]},{"osd":3,"last update":"Wed Sep 11 08:29:36 2024","interfaces":[{"interface":"back","average":{"1min":2.5670000000000002,"5min":1.663,"15min":1.5629999999999999},"min":{"1min":0.27100000000000002,"5min":0.20399999999999999,"15min":0.19700000000000001},"max":{"1min":7.9390000000000001,"5min":16.193999999999999,"15min":16.783999999999999},"last":0.29799999999999999},{"interface":"front","average":{"1min":1.966,"5min":1.591,"15min":1.488},"min":{"1min":0.30499999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":4.415,"5min":16.033000000000001,"15min":16.033000000000001},"last":0.30299999999999999}]},{"osd":4,"last update":"Wed Sep 11 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":2.222,"5min":1.5940000000000001,"15min":1.9399999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.254},"max":{"1min":8.2680000000000007,"5min":16.178000000000001,"15min":110.86499999999999},"last":0.40899999999999997},{"interface":"front","average":{"1min":1.528,"5min":1.6930000000000001,"15min":1.982},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28000000000000003},"max":{"1min":5.1980000000000004,"5min":19.009,"15min":110.831},"last":0.379}]},{"osd":5,"last update":"Wed Sep 11 08:29:36 2024","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.4059999999999999,"15min":1.5660000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":4.2359999999999998,"5min":16.012,"15min":16.012},"last":0.40000000000000002},{"interface":"front","average":{"1min":1.827,"5min":1.508,"15min":1.5660000000000001},"min":{"1min":0.315,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":7.1420000000000003,"5min":16.206,"15min":16.206},"last":0.36799999999999999}]},{"osd":6,"last update":"Wed Sep 11 08:29:36 2024","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.6479999999999999,"15min":1.8660000000000001},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":3.786,"5min":16.143000000000001,"15min":124.31999999999999},"last":0.41799999999999998},{"interface":"front","average":{"1min":1.329,"5min":1.611,"15min":1.831},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.29799999999999999},"max":{"1min":3.6150000000000002,"5min":15.984,"15min":124.958},"last":0.35899999999999999}]},{"osd":7,"last update":"Wed Sep 11 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":2.3540000000000001,"5min":1.706,"15min":1.865},"min":{"1min":0.32500000000000001,"5min":0.318,"15min":0.28599999999999998},"max":{"1min":8.2940000000000005,"5min":16.228999999999999,"15min":109.149},"last":0.44500000000000001},{"interface":"front","average":{"1min":1.387,"5min":1.4419999999999999,"15min":1.869},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.318},"max":{"1min":3.7599999999999998,"5min":16.170999999999999,"15min":109.309},"last":0.38700000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738678,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":899672,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":898238,"kb_avail":92845480,"statfs":{"total":95995035648,"available":95073771520,"internally_reserved":0,"allocated":1331200,"data_stored":959543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":919796056},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 11 08:29:39 2024","interfaces":[{"interface":"back","average":{"1min":2.5299999999999998,"5min":3.2999999999999998,"15min":1.903},"min":{"1min":0.40799999999999997,"5min":0.159,"15min":0.159},"max":{"1min":10.962999999999999,"5min":98.373999999999995,"15min":98.373999999999995},"last":0.374},{"interface":"front","average":{"1min":3.5289999999999999,"5min":3.609,"15min":2.246},"min":{"1min":0.505,"5min":0.17299999999999999,"15min":0.153},"max":{"1min":15.037000000000001,"5min":98.257999999999996,"15min":98.257999999999996},"last":0.42299999999999999}]},{"osd":2,"last update":"Wed Sep 11 08:30:08 2024","interfaces":[{"interface":"back","average":{"1min":1.4259999999999999,"5min":2.0569999999999999,"15min":1.5920000000000001},"min":{"1min":0.29999999999999999,"5min":0.19,"15min":0.19},"max":{"1min":5.2089999999999996,"5min":46.417000000000002,"15min":46.417000000000002},"last":0.45600000000000002},{"interface":"front","average":{"1min":2.5510000000000002,"5min":2.1240000000000001,"15min":1.6359999999999999},"min":{"1min":0.23000000000000001,"5min":0.22,"15min":0.22},"max":{"1min":13.331,"5min":46.438000000000002,"15min":46.438000000000002},"last":0.51500000000000001}]},{"osd":3,"last update":"Wed Sep 11 08:29:30 2024","interfaces":[{"interface":"back","average":{"1min":2.1459999999999999,"5min":1.5569999999999999,"15min":1.95},"min":{"1min":0.30499999999999999,"5min":0.222,"15min":0.222},"max":{"1min":9.5180000000000007,"5min":9.5180000000000007,"15min":172.149},"last":0.44},{"interface":"front","average":{"1min":1.754,"5min":1.647,"15min":1.9239999999999999},"min":{"1min":0.28499999999999998,"5min":0.20599999999999999,"15min":0.20300000000000001},"max":{"1min":8.5109999999999992,"5min":9.6170000000000009,"15min":174.899},"last":0.47399999999999998}]},{"osd":4,"last update":"Wed Sep 11 08:29:55 2024","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":1.2849999999999999,"15min":1.2210000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.252},"max":{"1min":2.6110000000000002,"5min":5.1609999999999996,"15min":9.6560000000000006},"last":0.54900000000000004},{"interface":"front","average":{"1min":1.0069999999999999,"5min":1.179,"15min":1.175},"min":{"1min":0.33400000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":2.6749999999999998,"5min":7.6630000000000003,"15min":8.3510000000000009},"last":0.66600000000000004}]},{"osd":5,"last update":"Wed Sep 11 08:29:35 2024","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.341,"15min":1.7969999999999999},"min":{"1min":0.28499999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":2.7000000000000002,"5min":7.7030000000000003,"15min":7.7030000000000003},"last":0.64000000000000001},{"interface":"front","average":{"1min":1.226,"5min":1.266,"15min":1.633},"min":{"1min":0.36399999999999999,"5min":0.309,"15min":0.309},"max":{"1min":4.2119999999999997,"5min":7.718,"15min":7.718},"last":0.622}]},{"osd":6,"last update":"Wed Sep 11 08:29:35 2024","interfaces":[{"interface":"back","average":{"1min":1.4650000000000001,"5min":1.518,"15min":1.2909999999999999},"min":{"1min":0.317,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":4.0810000000000004,"5min":9.7029999999999994,"15min":9.7029999999999994},"last":0.59099999999999997},{"interface":"front","average":{"1min":1.242,"5min":1.3200000000000001,"15min":1.2609999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.28299999999999997},"max":{"1min":4.1559999999999997,"5min":5.3419999999999996,"15min":7.4820000000000002},"last":0.65500000000000003}]},{"osd":7,"last update":"Wed Sep 11 08:29:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1299999999999999,"5min":1.3220000000000001,"15min":1.268},"min":{"1min":0.34499999999999997,"5min":0.34399999999999997,"15min":0.30399999999999999},"max":{"1min":3.1459999999999999,"5min":5.8049999999999997,"15min":7.5350000000000001},"last":0.68200000000000005},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.296,"15min":1.3540000000000001},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.32000000000000001},"max":{"1min":5.6829999999999998,"5min":5.6829999999999998,"15min":13.914},"last":0.70199999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3425,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T08:30:16.824 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T08:30:16.825 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5 2024-09-11T08:30:16.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:16.847 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:16.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.5' 2024-09-11T08:30:16.863 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'unique_pool_4' removed 2024-09-11T08:30:16.948 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0 2024-09-11T08:30:16.970 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:16 smithi150 ceph-mon[29276]: 11.1 scrub starts 2024-09-11T08:30:16.970 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:16 smithi150 ceph-mon[29276]: 11.1 scrub ok 2024-09-11T08:30:16.970 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:16 smithi150 ceph-mon[29276]: from='client.16119 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:30:16.970 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:16 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1064966995' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-11T08:30:16.970 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:16 smithi150 ceph-mon[39612]: 11.1 scrub starts 2024-09-11T08:30:16.970 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:16 smithi150 ceph-mon[39612]: 11.1 scrub ok 2024-09-11T08:30:16.970 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:16 smithi150 ceph-mon[39612]: from='client.16119 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:30:16.970 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:16 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1064966995' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-11T08:30:17.002 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:17.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:17.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.0' 2024-09-11T08:30:17.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:16 smithi181 ceph-mon[32701]: 11.1 scrub starts 2024-09-11T08:30:17.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:16 smithi181 ceph-mon[32701]: 11.1 scrub ok 2024-09-11T08:30:17.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:16 smithi181 ceph-mon[32701]: from='client.16119 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:30:17.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:16 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1064966995' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-11T08:30:17.104 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:30:17.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:17.150 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:17.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:30:17.251 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:30:17.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:17.273 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:17.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:30:17.309 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-11T08:30:17.309 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-11T08:30:17.309 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-09-11T08:30:17.309 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-09-11T08:30:17.309 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-11T08:30:17.375 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2 2024-09-11T08:30:17.386 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0: {} 2024-09-11T08:30:17.386 INFO:teuthology.orchestra.run.smithi150.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T08:30:17.396 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:17.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:17.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.2' 2024-09-11T08:30:17.405 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1: {} 2024-09-11T08:30:17.405 INFO:teuthology.orchestra.run.smithi150.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T08:30:17.429 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2: {} 2024-09-11T08:30:17.429 INFO:teuthology.orchestra.run.smithi150.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T08:30:17.444 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3: {} 2024-09-11T08:30:17.444 INFO:teuthology.orchestra.run.smithi150.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T08:30:17.460 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4: {} 2024-09-11T08:30:17.460 INFO:teuthology.orchestra.run.smithi150.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T08:30:17.475 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5: {} 2024-09-11T08:30:17.475 INFO:teuthology.orchestra.run.smithi150.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T08:30:17.490 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6: {} 2024-09-11T08:30:17.490 INFO:teuthology.orchestra.run.smithi150.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T08:30:17.498 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4 2024-09-11T08:30:17.505 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7: {} 2024-09-11T08:30:17.505 INFO:teuthology.orchestra.run.smithi150.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T08:30:17.520 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:17.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:17.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.4' 2024-09-11T08:30:17.622 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3 2024-09-11T08:30:17.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:17.644 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:17.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.3' 2024-09-11T08:30:17.745 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6 2024-09-11T08:30:17.767 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-09-11T08:30:17.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:17.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.6' 2024-09-11T08:30:17.869 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:30:17.890 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:17.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:17.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:30:17.908 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:17 smithi181 ceph-mon[32701]: 3.2 scrub starts 2024-09-11T08:30:17.909 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:17 smithi181 ceph-mon[32701]: 3.2 scrub ok 2024-09-11T08:30:17.909 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:17 smithi181 ceph-mon[32701]: 6.12 scrub starts 2024-09-11T08:30:17.909 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:17 smithi181 ceph-mon[32701]: 6.12 scrub ok 2024-09-11T08:30:17.909 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:17 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1064966995' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-11T08:30:17.909 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:17 smithi181 ceph-mon[32701]: osdmap e448: 8 total, 8 up, 8 in 2024-09-11T08:30:17.909 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:17 smithi181 ceph-mon[32701]: 4.7 scrub starts 2024-09-11T08:30:17.909 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:17 smithi181 ceph-mon[32701]: 4.7 scrub ok 2024-09-11T08:30:17.909 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:17 smithi181 ceph-mon[32701]: pgmap v1043: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-11T08:30:17.909 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:17 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/4177480142' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:30:17.933 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-11T08:30:17.992 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1 2024-09-11T08:30:18.014 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: docker: command not found 2024-09-11T08:30:18.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:30:18.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi150 with status 1: 'sudo docker kill -s 1 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd.1' 2024-09-11T08:30:18.057 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-09-11T08:30:18.067 INFO:tasks.thrashosds:joining thrashosds 2024-09-11T08:30:18.175 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:30:18.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:17 smithi150 ceph-mon[39612]: 3.2 scrub starts 2024-09-11T08:30:18.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:17 smithi150 ceph-mon[39612]: 3.2 scrub ok 2024-09-11T08:30:18.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:17 smithi150 ceph-mon[39612]: 6.12 scrub starts 2024-09-11T08:30:18.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:17 smithi150 ceph-mon[39612]: 6.12 scrub ok 2024-09-11T08:30:18.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:17 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1064966995' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-11T08:30:18.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:17 smithi150 ceph-mon[39612]: osdmap e448: 8 total, 8 up, 8 in 2024-09-11T08:30:18.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:17 smithi150 ceph-mon[39612]: 4.7 scrub starts 2024-09-11T08:30:18.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:17 smithi150 ceph-mon[39612]: 4.7 scrub ok 2024-09-11T08:30:18.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:17 smithi150 ceph-mon[39612]: pgmap v1043: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-11T08:30:18.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:17 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/4177480142' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:30:18.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:17 smithi150 ceph-mon[29276]: 3.2 scrub starts 2024-09-11T08:30:18.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:17 smithi150 ceph-mon[29276]: 3.2 scrub ok 2024-09-11T08:30:18.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:17 smithi150 ceph-mon[29276]: 6.12 scrub starts 2024-09-11T08:30:18.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:17 smithi150 ceph-mon[29276]: 6.12 scrub ok 2024-09-11T08:30:18.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:17 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1064966995' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-11T08:30:18.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:17 smithi150 ceph-mon[29276]: osdmap e448: 8 total, 8 up, 8 in 2024-09-11T08:30:18.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:17 smithi150 ceph-mon[29276]: 4.7 scrub starts 2024-09-11T08:30:18.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:17 smithi150 ceph-mon[29276]: 4.7 scrub ok 2024-09-11T08:30:18.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:17 smithi150 ceph-mon[29276]: pgmap v1043: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-11T08:30:18.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:17 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/4177480142' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-11T08:30:18.737 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:30:18 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:30:18] "GET /metrics HTTP/1.1" 200 224672 "" "Prometheus/2.43.0" 2024-09-11T08:30:18.830 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:30:18.830 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:30:18.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:18 smithi150 ceph-mon[29276]: 3.b scrub starts 2024-09-11T08:30:18.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:18 smithi150 ceph-mon[29276]: 3.b scrub ok 2024-09-11T08:30:18.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:18 smithi150 ceph-mon[29276]: 6.6 scrub starts 2024-09-11T08:30:18.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:18 smithi150 ceph-mon[29276]: 6.6 scrub ok 2024-09-11T08:30:18.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:18 smithi150 ceph-mon[29276]: 6.7 scrub starts 2024-09-11T08:30:18.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:18 smithi150 ceph-mon[29276]: 6.7 scrub ok 2024-09-11T08:30:18.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:18 smithi150 ceph-mon[39612]: 3.b scrub starts 2024-09-11T08:30:18.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:18 smithi150 ceph-mon[39612]: 3.b scrub ok 2024-09-11T08:30:18.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:18 smithi150 ceph-mon[39612]: 6.6 scrub starts 2024-09-11T08:30:18.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:18 smithi150 ceph-mon[39612]: 6.6 scrub ok 2024-09-11T08:30:18.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:18 smithi150 ceph-mon[39612]: 6.7 scrub starts 2024-09-11T08:30:18.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:18 smithi150 ceph-mon[39612]: 6.7 scrub ok 2024-09-11T08:30:19.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:18 smithi181 ceph-mon[32701]: 3.b scrub starts 2024-09-11T08:30:19.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:18 smithi181 ceph-mon[32701]: 3.b scrub ok 2024-09-11T08:30:19.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:18 smithi181 ceph-mon[32701]: 6.6 scrub starts 2024-09-11T08:30:19.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:18 smithi181 ceph-mon[32701]: 6.6 scrub ok 2024-09-11T08:30:19.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:18 smithi181 ceph-mon[32701]: 6.7 scrub starts 2024-09-11T08:30:19.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:18 smithi181 ceph-mon[32701]: 6.7 scrub ok 2024-09-11T08:30:19.421 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:30:19 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:30:19.794 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-11T08:30:19.794 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:30:20.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:19 smithi181 ceph-mon[32701]: 3.1 deep-scrub starts 2024-09-11T08:30:20.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:19 smithi181 ceph-mon[32701]: 3.1 deep-scrub ok 2024-09-11T08:30:20.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:19 smithi181 ceph-mon[32701]: 5.8 scrub starts 2024-09-11T08:30:20.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:19 smithi181 ceph-mon[32701]: 5.8 scrub ok 2024-09-11T08:30:20.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:19 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:30:20.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:19 smithi181 ceph-mon[32701]: pgmap v1044: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-11T08:30:20.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:19 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:30:20.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:19 smithi150 ceph-mon[29276]: 3.1 deep-scrub starts 2024-09-11T08:30:20.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:19 smithi150 ceph-mon[29276]: 3.1 deep-scrub ok 2024-09-11T08:30:20.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:19 smithi150 ceph-mon[29276]: 5.8 scrub starts 2024-09-11T08:30:20.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:19 smithi150 ceph-mon[29276]: 5.8 scrub ok 2024-09-11T08:30:20.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:19 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:30:20.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:19 smithi150 ceph-mon[29276]: pgmap v1044: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-11T08:30:20.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:19 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:30:20.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:19 smithi150 ceph-mon[39612]: 3.1 deep-scrub starts 2024-09-11T08:30:20.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:19 smithi150 ceph-mon[39612]: 3.1 deep-scrub ok 2024-09-11T08:30:20.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:19 smithi150 ceph-mon[39612]: 5.8 scrub starts 2024-09-11T08:30:20.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:19 smithi150 ceph-mon[39612]: 5.8 scrub ok 2024-09-11T08:30:20.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:19 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:30:20.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:19 smithi150 ceph-mon[39612]: pgmap v1044: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-11T08:30:20.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:19 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:30:20.441 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-11T08:30:20.685 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:30:21.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:20 smithi181 ceph-mon[32701]: 5.1a scrub starts 2024-09-11T08:30:21.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:20 smithi181 ceph-mon[32701]: 5.1a scrub ok 2024-09-11T08:30:21.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:20 smithi181 ceph-mon[32701]: 3.5 scrub starts 2024-09-11T08:30:21.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:20 smithi181 ceph-mon[32701]: 3.5 scrub ok 2024-09-11T08:30:21.229 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:20 smithi150 ceph-mon[29276]: 5.1a scrub starts 2024-09-11T08:30:21.229 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:20 smithi150 ceph-mon[29276]: 5.1a scrub ok 2024-09-11T08:30:21.229 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:20 smithi150 ceph-mon[29276]: 3.5 scrub starts 2024-09-11T08:30:21.229 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:20 smithi150 ceph-mon[29276]: 3.5 scrub ok 2024-09-11T08:30:21.229 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:20 smithi150 ceph-mon[39612]: 5.1a scrub starts 2024-09-11T08:30:21.230 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:20 smithi150 ceph-mon[39612]: 5.1a scrub ok 2024-09-11T08:30:21.230 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:20 smithi150 ceph-mon[39612]: 3.5 scrub starts 2024-09-11T08:30:21.230 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:20 smithi150 ceph-mon[39612]: 3.5 scrub ok 2024-09-11T08:30:21.985 INFO:teuthology.orchestra.run.smithi150.stderr:noscrub is set 2024-09-11T08:30:22.050 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:21 smithi150 ceph-mon[29276]: 3.18 scrub starts 2024-09-11T08:30:22.050 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:21 smithi150 ceph-mon[29276]: 3.18 scrub ok 2024-09-11T08:30:22.050 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:21 smithi150 ceph-mon[29276]: 5.1c scrub starts 2024-09-11T08:30:22.050 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:21 smithi150 ceph-mon[29276]: 5.1c scrub ok 2024-09-11T08:30:22.050 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:21 smithi150 ceph-mon[29276]: 5.7 scrub starts 2024-09-11T08:30:22.051 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:21 smithi150 ceph-mon[29276]: 5.7 scrub ok 2024-09-11T08:30:22.051 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:21 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:30:22.051 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:21 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:30:22.051 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:21 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:30:22.051 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:21 smithi150 ceph-mon[29276]: pgmap v1045: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-11T08:30:22.051 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:21 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/4054068072' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:30:22.051 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:21 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:30:22.051 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:21 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:30:22.051 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:21 smithi150 ceph-mon[39612]: 3.18 scrub starts 2024-09-11T08:30:22.051 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:21 smithi150 ceph-mon[39612]: 3.18 scrub ok 2024-09-11T08:30:22.051 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:21 smithi150 ceph-mon[39612]: 5.1c scrub starts 2024-09-11T08:30:22.052 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:21 smithi150 ceph-mon[39612]: 5.1c scrub ok 2024-09-11T08:30:22.052 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:21 smithi150 ceph-mon[39612]: 5.7 scrub starts 2024-09-11T08:30:22.052 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:21 smithi150 ceph-mon[39612]: 5.7 scrub ok 2024-09-11T08:30:22.052 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:21 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:30:22.052 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:21 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:30:22.052 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:21 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:30:22.052 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:21 smithi150 ceph-mon[39612]: pgmap v1045: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-11T08:30:22.052 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:21 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/4054068072' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:30:22.052 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:21 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:30:22.052 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:21 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:30:22.143 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:21 smithi181 ceph-mon[32701]: 3.18 scrub starts 2024-09-11T08:30:22.143 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:21 smithi181 ceph-mon[32701]: 3.18 scrub ok 2024-09-11T08:30:22.143 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:21 smithi181 ceph-mon[32701]: 5.1c scrub starts 2024-09-11T08:30:22.143 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:21 smithi181 ceph-mon[32701]: 5.1c scrub ok 2024-09-11T08:30:22.143 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:21 smithi181 ceph-mon[32701]: 5.7 scrub starts 2024-09-11T08:30:22.144 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:21 smithi181 ceph-mon[32701]: 5.7 scrub ok 2024-09-11T08:30:22.144 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:21 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:30:22.144 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:21 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:30:22.144 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:21 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:30:22.144 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:21 smithi181 ceph-mon[32701]: pgmap v1045: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-11T08:30:22.144 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:21 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/4054068072' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-11T08:30:22.144 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:21 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:30:22.144 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:21 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:30:22.369 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-11T08:30:22.370 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:30:22.664 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:30:22.667 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-09-11T08:30:22.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:22 smithi150 ceph-mon[39612]: 3.1f scrub starts 2024-09-11T08:30:22.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:22 smithi150 ceph-mon[39612]: 3.1f scrub ok 2024-09-11T08:30:22.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:22 smithi150 ceph-mon[39612]: 6.4 scrub starts 2024-09-11T08:30:22.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:22 smithi150 ceph-mon[39612]: 6.4 scrub ok 2024-09-11T08:30:22.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:22 smithi150 ceph-mon[39612]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:30:22.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:22 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/4054068072' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:30:22.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:22 smithi150 ceph-mon[39612]: osdmap e449: 8 total, 8 up, 8 in 2024-09-11T08:30:22.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:22 smithi150 ceph-mon[29276]: 3.1f scrub starts 2024-09-11T08:30:22.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:22 smithi150 ceph-mon[29276]: 3.1f scrub ok 2024-09-11T08:30:22.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:22 smithi150 ceph-mon[29276]: 6.4 scrub starts 2024-09-11T08:30:22.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:22 smithi150 ceph-mon[29276]: 6.4 scrub ok 2024-09-11T08:30:22.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:22 smithi150 ceph-mon[29276]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:30:22.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:22 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/4054068072' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:30:22.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:22 smithi150 ceph-mon[29276]: osdmap e449: 8 total, 8 up, 8 in 2024-09-11T08:30:22.989 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:22 smithi181 ceph-mon[32701]: 3.1f scrub starts 2024-09-11T08:30:22.989 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:22 smithi181 ceph-mon[32701]: 3.1f scrub ok 2024-09-11T08:30:22.990 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:22 smithi181 ceph-mon[32701]: 6.4 scrub starts 2024-09-11T08:30:22.990 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:22 smithi181 ceph-mon[32701]: 6.4 scrub ok 2024-09-11T08:30:22.990 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:22 smithi181 ceph-mon[32701]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T08:30:22.990 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:22 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/4054068072' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T08:30:22.990 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:22 smithi181 ceph-mon[32701]: osdmap e449: 8 total, 8 up, 8 in 2024-09-11T08:30:23.016 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-11T08:30:23.261 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:30:23.948 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":1046,"stamp":"2024-09-11T08:30:21.981956+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596188,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":2622,"num_read_kb":2481,"num_write":1115,"num_write_kb":1671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":208,"num_bytes_recovered":1776519,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1264,"ondisk_log_size":1264,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":415,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6655328,"kb_used_data":7556,"kb_used_omap":22,"kb_used_meta":6647657,"kb_avail":743305888,"statfs":{"total":767960285184,"available":761145229312,"internally_reserved":0,"allocated":7737344,"data_stored":4773604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22853,"internal_metadata":6807201467},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":72,"apply_latency_ms":72,"commit_latency_ns":72000000,"apply_latency_ns":72000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":-252094,"num_write_kb":-8066945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.742790"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":514,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.270985+0000","last_change":"2024-09-11T08:28:03.807764+0000","last_active":"2024-09-11T08:30:11.270985+0000","last_peered":"2024-09-11T08:30:11.270985+0000","last_clean":"2024-09-11T08:30:11.270985+0000","last_became_active":"2024-09-11T08:24:28.358401+0000","last_became_peered":"2024-09-11T08:24:28.358401+0000","last_unstale":"2024-09-11T08:30:11.270985+0000","last_undegraded":"2024-09-11T08:30:11.270985+0000","last_fullsized":"2024-09-11T08:30:11.270985+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:28:03.807673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:28:03.807673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036189030000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"405'25","reported_seq":537,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:10.835787+0000","last_change":"2024-09-11T08:29:59.282881+0000","last_active":"2024-09-11T08:30:10.835787+0000","last_peered":"2024-09-11T08:30:10.835787+0000","last_clean":"2024-09-11T08:30:10.835787+0000","last_became_active":"2024-09-11T08:07:32.048048+0000","last_became_peered":"2024-09-11T08:07:32.048048+0000","last_unstale":"2024-09-11T08:30:10.835787+0000","last_undegraded":"2024-09-11T08:30:10.835787+0000","last_fullsized":"2024-09-11T08:30:10.835787+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'25","last_scrub_stamp":"2024-09-11T08:29:59.282798+0000","last_deep_scrub":"312'19","last_deep_scrub_stamp":"2024-09-11T08:21:59.122122+0000","last_clean_scrub_stamp":"2024-09-11T08:29:59.282798+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:09.025796+0000","scrub_duration":0.012064749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":560,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:10.835191+0000","last_change":"2024-09-11T08:27:03.997743+0000","last_active":"2024-09-11T08:30:10.835191+0000","last_peered":"2024-09-11T08:30:10.835191+0000","last_clean":"2024-09-11T08:30:10.835191+0000","last_became_active":"2024-09-11T08:24:28.372160+0000","last_became_peered":"2024-09-11T08:24:28.372160+0000","last_unstale":"2024-09-11T08:30:10.835191+0000","last_undegraded":"2024-09-11T08:30:10.835191+0000","last_fullsized":"2024-09-11T08:30:10.835191+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:03.997657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:59.531202+0000","last_clean_scrub_stamp":"2024-09-11T08:27:03.997657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035479840000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":418,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:10.835016+0000","last_change":"2024-09-11T08:27:06.987974+0000","last_active":"2024-09-11T08:30:10.835016+0000","last_peered":"2024-09-11T08:30:10.835016+0000","last_clean":"2024-09-11T08:30:10.835016+0000","last_became_active":"2024-09-11T08:24:28.354834+0000","last_became_peered":"2024-09-11T08:24:28.354834+0000","last_unstale":"2024-09-11T08:30:10.835016+0000","last_undegraded":"2024-09-11T08:30:10.835016+0000","last_fullsized":"2024-09-11T08:30:10.835016+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:06.987888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:02.470649+0000","last_clean_scrub_stamp":"2024-09-11T08:27:06.987888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030657430000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":509,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:10.837002+0000","last_change":"2024-09-11T08:29:03.775193+0000","last_active":"2024-09-11T08:30:10.837002+0000","last_peered":"2024-09-11T08:30:10.837002+0000","last_clean":"2024-09-11T08:30:10.837002+0000","last_became_active":"2024-09-11T08:15:56.521104+0000","last_became_peered":"2024-09-11T08:15:56.521104+0000","last_unstale":"2024-09-11T08:30:10.837002+0000","last_undegraded":"2024-09-11T08:30:10.837002+0000","last_fullsized":"2024-09-11T08:30:10.837002+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:03.775098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:29:03.775098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:30.682191+0000","scrub_duration":0.0091193990000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":292,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.304929+0000","last_change":"2024-09-11T08:30:00.543330+0000","last_active":"2024-09-11T08:30:11.304929+0000","last_peered":"2024-09-11T08:30:11.304929+0000","last_clean":"2024-09-11T08:30:11.304929+0000","last_became_active":"2024-09-11T08:17:37.550048+0000","last_became_peered":"2024-09-11T08:17:37.550048+0000","last_unstale":"2024-09-11T08:30:11.304929+0000","last_undegraded":"2024-09-11T08:30:11.304929+0000","last_fullsized":"2024-09-11T08:30:11.304929+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:00.543254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:30:00.543254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:27.529390+0000","scrub_duration":0.0048128770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"405'46","reported_seq":470,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.488274+0000","last_change":"2024-09-11T08:29:17.630424+0000","last_active":"2024-09-11T08:30:11.488274+0000","last_peered":"2024-09-11T08:30:11.488274+0000","last_clean":"2024-09-11T08:30:11.488274+0000","last_became_active":"2024-09-11T08:17:37.553866+0000","last_became_peered":"2024-09-11T08:17:37.553866+0000","last_unstale":"2024-09-11T08:30:11.488274+0000","last_undegraded":"2024-09-11T08:30:11.488274+0000","last_fullsized":"2024-09-11T08:30:11.488274+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'46","last_scrub_stamp":"2024-09-11T08:29:17.630303+0000","last_deep_scrub":"145'22","last_deep_scrub_stamp":"2024-09-11T08:15:57.478870+0000","last_clean_scrub_stamp":"2024-09-11T08:29:17.630303+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:23.815610+0000","scrub_duration":0.014805099,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":153,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.304211+0000","last_change":"2024-09-11T08:29:23.277986+0000","last_active":"2024-09-11T08:30:11.304211+0000","last_peered":"2024-09-11T08:30:11.304211+0000","last_clean":"2024-09-11T08:30:11.304211+0000","last_became_active":"2024-09-11T08:23:22.687041+0000","last_became_peered":"2024-09-11T08:23:22.687041+0000","last_unstale":"2024-09-11T08:30:11.304211+0000","last_undegraded":"2024-09-11T08:30:11.304211+0000","last_fullsized":"2024-09-11T08:30:11.304211+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:23.277854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:18:36.998503+0000","last_clean_scrub_stamp":"2024-09-11T08:29:23.277854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:37.611048+0000","scrub_duration":0.017784378,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":184,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.304753+0000","last_change":"2024-09-11T08:29:12.315549+0000","last_active":"2024-09-11T08:30:11.304753+0000","last_peered":"2024-09-11T08:30:11.304753+0000","last_clean":"2024-09-11T08:30:11.304753+0000","last_became_active":"2024-09-11T08:24:28.360021+0000","last_became_peered":"2024-09-11T08:24:28.360021+0000","last_unstale":"2024-09-11T08:30:11.304753+0000","last_undegraded":"2024-09-11T08:30:11.304753+0000","last_fullsized":"2024-09-11T08:30:11.304753+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:12.315453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:54.878409+0000","last_clean_scrub_stamp":"2024-09-11T08:29:12.315453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00128605,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"405'21","reported_seq":402,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.271161+0000","last_change":"2024-09-11T08:29:19.176724+0000","last_active":"2024-09-11T08:30:11.271161+0000","last_peered":"2024-09-11T08:30:11.271161+0000","last_clean":"2024-09-11T08:30:11.271161+0000","last_became_active":"2024-09-11T08:23:22.687182+0000","last_became_peered":"2024-09-11T08:23:22.687182+0000","last_unstale":"2024-09-11T08:30:11.271161+0000","last_undegraded":"2024-09-11T08:30:11.271161+0000","last_fullsized":"2024-09-11T08:30:11.271161+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'21","last_scrub_stamp":"2024-09-11T08:29:19.176584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:29:19.176584+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:45.130745+0000","scrub_duration":0.022662364000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":399,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.270768+0000","last_change":"2024-09-11T08:29:04.968206+0000","last_active":"2024-09-11T08:30:11.270768+0000","last_peered":"2024-09-11T08:30:11.270768+0000","last_clean":"2024-09-11T08:30:11.270768+0000","last_became_active":"2024-09-11T08:24:28.363985+0000","last_became_peered":"2024-09-11T08:24:28.363985+0000","last_unstale":"2024-09-11T08:30:11.270768+0000","last_undegraded":"2024-09-11T08:30:11.270768+0000","last_fullsized":"2024-09-11T08:30:11.270768+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:04.967870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:20:18.900482+0000","last_clean_scrub_stamp":"2024-09-11T08:29:04.967870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0081125309999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":488,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.270475+0000","last_change":"2024-09-11T08:28:57.875994+0000","last_active":"2024-09-11T08:30:11.270475+0000","last_peered":"2024-09-11T08:30:11.270475+0000","last_clean":"2024-09-11T08:30:11.270475+0000","last_became_active":"2024-09-11T08:14:24.792595+0000","last_became_peered":"2024-09-11T08:14:24.792595+0000","last_unstale":"2024-09-11T08:30:11.270475+0000","last_undegraded":"2024-09-11T08:30:11.270475+0000","last_fullsized":"2024-09-11T08:30:11.270475+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:28:57.875851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:28:57.875851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014796494,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":186,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:13.004184+0000","last_change":"2024-09-11T08:30:13.004150+0000","last_active":"2024-09-11T08:30:13.004184+0000","last_peered":"2024-09-11T08:30:13.004184+0000","last_clean":"2024-09-11T08:30:13.004184+0000","last_became_active":"2024-09-11T08:24:28.359153+0000","last_became_peered":"2024-09-11T08:24:28.359153+0000","last_unstale":"2024-09-11T08:30:13.004184+0000","last_undegraded":"2024-09-11T08:30:13.004184+0000","last_fullsized":"2024-09-11T08:30:13.004184+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:13.004098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:30:13.004098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:24.968043+0000","scrub_duration":0.0014473380000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"405'11","reported_seq":386,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:10.837331+0000","last_change":"2024-09-11T08:28:58.811469+0000","last_active":"2024-09-11T08:30:10.837331+0000","last_peered":"2024-09-11T08:30:10.837331+0000","last_clean":"2024-09-11T08:30:10.837331+0000","last_became_active":"2024-09-11T08:23:22.681876+0000","last_became_peered":"2024-09-11T08:23:22.681876+0000","last_unstale":"2024-09-11T08:30:10.837331+0000","last_undegraded":"2024-09-11T08:30:10.837331+0000","last_fullsized":"2024-09-11T08:30:10.837331+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'11","last_scrub_stamp":"2024-09-11T08:28:58.811380+0000","last_deep_scrub":"139'7","last_deep_scrub_stamp":"2024-09-11T08:12:24.869318+0000","last_clean_scrub_stamp":"2024-09-11T08:28:58.811380+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.007538215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":300,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.595292+0000","last_change":"2024-09-11T08:27:54.287836+0000","last_active":"2024-09-11T08:30:11.595292+0000","last_peered":"2024-09-11T08:30:11.595292+0000","last_clean":"2024-09-11T08:30:11.595292+0000","last_became_active":"2024-09-11T08:15:56.526954+0000","last_became_peered":"2024-09-11T08:15:56.526954+0000","last_unstale":"2024-09-11T08:30:11.595292+0000","last_undegraded":"2024-09-11T08:30:11.595292+0000","last_fullsized":"2024-09-11T08:30:11.595292+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:54.287773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:27:54.287773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015716346999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":180,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:10.835012+0000","last_change":"2024-09-11T08:27:21.243352+0000","last_active":"2024-09-11T08:30:10.835012+0000","last_peered":"2024-09-11T08:30:10.835012+0000","last_clean":"2024-09-11T08:30:10.835012+0000","last_became_active":"2024-09-11T08:24:28.372097+0000","last_became_peered":"2024-09-11T08:24:28.372097+0000","last_unstale":"2024-09-11T08:30:10.835012+0000","last_undegraded":"2024-09-11T08:30:10.835012+0000","last_fullsized":"2024-09-11T08:30:10.835012+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:21.243267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:20:19.927616+0000","last_clean_scrub_stamp":"2024-09-11T08:27:21.243267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0078709639999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":436,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.304842+0000","last_change":"2024-09-11T08:30:05.450807+0000","last_active":"2024-09-11T08:30:11.304842+0000","last_peered":"2024-09-11T08:30:11.304842+0000","last_clean":"2024-09-11T08:30:11.304842+0000","last_became_active":"2024-09-11T08:24:28.368560+0000","last_became_peered":"2024-09-11T08:24:28.368560+0000","last_unstale":"2024-09-11T08:30:11.304842+0000","last_undegraded":"2024-09-11T08:30:11.304842+0000","last_fullsized":"2024-09-11T08:30:11.304842+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:05.450746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:30:05.450746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:15.048635+0000","scrub_duration":0.0012973489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":507,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.271191+0000","last_change":"2024-09-11T08:29:16.034440+0000","last_active":"2024-09-11T08:30:11.271191+0000","last_peered":"2024-09-11T08:30:11.271191+0000","last_clean":"2024-09-11T08:30:11.271191+0000","last_became_active":"2024-09-11T08:15:56.516941+0000","last_became_peered":"2024-09-11T08:15:56.516941+0000","last_unstale":"2024-09-11T08:30:11.271191+0000","last_undegraded":"2024-09-11T08:30:11.271191+0000","last_fullsized":"2024-09-11T08:30:11.271191+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:16.034353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:29:16.034353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:22.408537+0000","scrub_duration":0.0042549959999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"405'46","reported_seq":573,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.488707+0000","last_change":"2024-09-11T08:29:07.522912+0000","last_active":"2024-09-11T08:30:11.488707+0000","last_peered":"2024-09-11T08:30:11.488707+0000","last_clean":"2024-09-11T08:30:11.488707+0000","last_became_active":"2024-09-11T08:07:32.046363+0000","last_became_peered":"2024-09-11T08:07:32.046363+0000","last_unstale":"2024-09-11T08:30:11.488707+0000","last_undegraded":"2024-09-11T08:30:11.488707+0000","last_fullsized":"2024-09-11T08:30:11.488707+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'46","last_scrub_stamp":"2024-09-11T08:29:07.522566+0000","last_deep_scrub":"107'22","last_deep_scrub_stamp":"2024-09-11T08:16:04.159211+0000","last_clean_scrub_stamp":"2024-09-11T08:29:07.522566+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:25.044660+0000","scrub_duration":0.024496106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":503,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:10.835853+0000","last_change":"2024-09-11T08:28:55.792756+0000","last_active":"2024-09-11T08:30:10.835853+0000","last_peered":"2024-09-11T08:30:10.835853+0000","last_clean":"2024-09-11T08:30:10.835853+0000","last_became_active":"2024-09-11T08:07:34.055410+0000","last_became_peered":"2024-09-11T08:07:34.055410+0000","last_unstale":"2024-09-11T08:30:10.835853+0000","last_undegraded":"2024-09-11T08:30:10.835853+0000","last_fullsized":"2024-09-11T08:30:10.835853+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:28:55.792659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:19:28.870211+0000","last_clean_scrub_stamp":"2024-09-11T08:28:55.792659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:24.688414+0000","scrub_duration":0.015527124999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":319,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.304984+0000","last_change":"2024-09-11T08:29:56.574094+0000","last_active":"2024-09-11T08:30:11.304984+0000","last_peered":"2024-09-11T08:30:11.304984+0000","last_clean":"2024-09-11T08:30:11.304984+0000","last_became_active":"2024-09-11T08:24:28.357957+0000","last_became_peered":"2024-09-11T08:24:28.357957+0000","last_unstale":"2024-09-11T08:30:11.304984+0000","last_undegraded":"2024-09-11T08:30:11.304984+0000","last_fullsized":"2024-09-11T08:30:11.304984+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:56.574045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:29:56.574045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:13.992398+0000","scrub_duration":0.0013195139999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":152,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:10.835645+0000","last_change":"2024-09-11T08:29:08.797434+0000","last_active":"2024-09-11T08:30:10.835645+0000","last_peered":"2024-09-11T08:30:10.835645+0000","last_clean":"2024-09-11T08:30:10.835645+0000","last_became_active":"2024-09-11T08:23:22.686935+0000","last_became_peered":"2024-09-11T08:23:22.686935+0000","last_unstale":"2024-09-11T08:30:10.835645+0000","last_undegraded":"2024-09-11T08:30:10.835645+0000","last_fullsized":"2024-09-11T08:30:10.835645+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:08.797351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:29:08.797351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:35.864334+0000","scrub_duration":0.0047226200000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"405'24","reported_seq":361,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.488584+0000","last_change":"2024-09-11T08:29:09.521760+0000","last_active":"2024-09-11T08:30:11.488584+0000","last_peered":"2024-09-11T08:30:11.488584+0000","last_clean":"2024-09-11T08:30:11.488584+0000","last_became_active":"2024-09-11T08:17:37.553402+0000","last_became_peered":"2024-09-11T08:17:37.553402+0000","last_unstale":"2024-09-11T08:30:11.488584+0000","last_undegraded":"2024-09-11T08:30:11.488584+0000","last_fullsized":"2024-09-11T08:30:11.488584+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'24","last_scrub_stamp":"2024-09-11T08:29:09.520708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:29:09.520708+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:37.502124+0000","scrub_duration":0.0081557079999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":153,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:10.836493+0000","last_change":"2024-09-11T08:28:48.787378+0000","last_active":"2024-09-11T08:30:10.836493+0000","last_peered":"2024-09-11T08:30:10.836493+0000","last_clean":"2024-09-11T08:30:10.836493+0000","last_became_active":"2024-09-11T08:23:22.684583+0000","last_became_peered":"2024-09-11T08:23:22.684583+0000","last_unstale":"2024-09-11T08:30:10.836493+0000","last_undegraded":"2024-09-11T08:30:10.836493+0000","last_fullsized":"2024-09-11T08:30:10.836493+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:28:48.787291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:10:40.110490+0000","last_clean_scrub_stamp":"2024-09-11T08:28:48.787291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002632129,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":475,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.595801+0000","last_change":"2024-09-11T08:27:52.321730+0000","last_active":"2024-09-11T08:30:11.595801+0000","last_peered":"2024-09-11T08:30:11.595801+0000","last_clean":"2024-09-11T08:30:11.595801+0000","last_became_active":"2024-09-11T08:24:28.357198+0000","last_became_peered":"2024-09-11T08:24:28.357198+0000","last_unstale":"2024-09-11T08:30:11.595801+0000","last_undegraded":"2024-09-11T08:30:11.595801+0000","last_fullsized":"2024-09-11T08:30:11.595801+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-11T08:27:52.321646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:27:52.321646+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012244692999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":497,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:10.835237+0000","last_change":"2024-09-11T08:27:50.150362+0000","last_active":"2024-09-11T08:30:10.835237+0000","last_peered":"2024-09-11T08:30:10.835237+0000","last_clean":"2024-09-11T08:30:10.835237+0000","last_became_active":"2024-09-11T08:07:30.045932+0000","last_became_peered":"2024-09-11T08:07:30.045932+0000","last_unstale":"2024-09-11T08:30:10.835237+0000","last_undegraded":"2024-09-11T08:30:10.835237+0000","last_fullsized":"2024-09-11T08:30:10.835237+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:50.150256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:27:50.150256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026580110000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"405'35","reported_seq":598,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.270985+0000","last_change":"2024-09-11T08:28:56.912293+0000","last_active":"2024-09-11T08:30:11.270985+0000","last_peered":"2024-09-11T08:30:11.270985+0000","last_clean":"2024-09-11T08:30:11.270985+0000","last_became_active":"2024-09-11T08:15:56.515817+0000","last_became_peered":"2024-09-11T08:15:56.515817+0000","last_unstale":"2024-09-11T08:30:11.270985+0000","last_undegraded":"2024-09-11T08:30:11.270985+0000","last_fullsized":"2024-09-11T08:30:11.270985+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'35","last_scrub_stamp":"2024-09-11T08:28:56.911970+0000","last_deep_scrub":"350'25","last_deep_scrub_stamp":"2024-09-11T08:24:23.093269+0000","last_clean_scrub_stamp":"2024-09-11T08:28:56.911970+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:21.445351+0000","scrub_duration":0.066494655,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"254'12","reported_seq":834,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:13.846413+0000","last_change":"2024-09-11T08:29:14.338360+0000","last_active":"2024-09-11T08:30:13.846413+0000","last_peered":"2024-09-11T08:30:13.846413+0000","last_clean":"2024-09-11T08:30:13.846413+0000","last_became_active":"2024-09-11T08:15:56.525498+0000","last_became_peered":"2024-09-11T08:15:56.525498+0000","last_unstale":"2024-09-11T08:30:13.846413+0000","last_undegraded":"2024-09-11T08:30:13.846413+0000","last_fullsized":"2024-09-11T08:30:13.846413+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'12","last_scrub_stamp":"2024-09-11T08:29:14.338276+0000","last_deep_scrub":"254'12","last_deep_scrub_stamp":"2024-09-11T08:18:34.253865+0000","last_clean_scrub_stamp":"2024-09-11T08:29:14.338276+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:42.163322+0000","scrub_duration":0.0021322020000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"405'39","reported_seq":566,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:10.836115+0000","last_change":"2024-09-11T08:27:58.226977+0000","last_active":"2024-09-11T08:30:10.836115+0000","last_peered":"2024-09-11T08:30:10.836115+0000","last_clean":"2024-09-11T08:30:10.836115+0000","last_became_active":"2024-09-11T08:07:32.054317+0000","last_became_peered":"2024-09-11T08:07:32.054317+0000","last_unstale":"2024-09-11T08:30:10.836115+0000","last_undegraded":"2024-09-11T08:30:10.836115+0000","last_fullsized":"2024-09-11T08:30:10.836115+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'39","last_scrub_stamp":"2024-09-11T08:27:58.226812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:27:58.226812+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015475453,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":518,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:10.835891+0000","last_change":"2024-09-11T08:29:14.660571+0000","last_active":"2024-09-11T08:30:10.835891+0000","last_peered":"2024-09-11T08:30:10.835891+0000","last_clean":"2024-09-11T08:30:10.835891+0000","last_became_active":"2024-09-11T08:15:56.521943+0000","last_became_peered":"2024-09-11T08:15:56.521943+0000","last_unstale":"2024-09-11T08:30:10.835891+0000","last_undegraded":"2024-09-11T08:30:10.835891+0000","last_fullsized":"2024-09-11T08:30:10.835891+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:14.660494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:29:14.660494+0000","last_clean_scrub_stamp":"2024-09-11T08:29:14.660494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:27.445624+0000","scrub_duration":0.0075203229999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"354'13","reported_seq":265,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.913348+0000","last_change":"2024-09-11T08:29:03.976374+0000","last_active":"2024-09-11T08:30:16.913348+0000","last_peered":"2024-09-11T08:30:16.913348+0000","last_clean":"2024-09-11T08:30:16.913348+0000","last_became_active":"2024-09-11T08:24:28.368212+0000","last_became_peered":"2024-09-11T08:24:28.368212+0000","last_unstale":"2024-09-11T08:30:16.913348+0000","last_undegraded":"2024-09-11T08:30:16.913348+0000","last_fullsized":"2024-09-11T08:30:16.913348+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'13","last_scrub_stamp":"2024-09-11T08:29:03.976322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:29:03.976322+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013262809999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":180,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:10.835764+0000","last_change":"2024-09-11T08:27:05.040598+0000","last_active":"2024-09-11T08:30:10.835764+0000","last_peered":"2024-09-11T08:30:10.835764+0000","last_clean":"2024-09-11T08:30:10.835764+0000","last_became_active":"2024-09-11T08:24:28.353556+0000","last_became_peered":"2024-09-11T08:24:28.353556+0000","last_unstale":"2024-09-11T08:30:10.835764+0000","last_undegraded":"2024-09-11T08:30:10.835764+0000","last_fullsized":"2024-09-11T08:30:10.835764+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:05.040523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:06.616332+0000","last_clean_scrub_stamp":"2024-09-11T08:27:05.040523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050758849999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"405'42","reported_seq":555,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:13.678107+0000","last_change":"2024-09-11T08:30:13.678049+0000","last_active":"2024-09-11T08:30:13.678107+0000","last_peered":"2024-09-11T08:30:13.678107+0000","last_clean":"2024-09-11T08:30:13.678107+0000","last_became_active":"2024-09-11T08:17:37.555703+0000","last_became_peered":"2024-09-11T08:17:37.555703+0000","last_unstale":"2024-09-11T08:30:13.678107+0000","last_undegraded":"2024-09-11T08:30:13.678107+0000","last_fullsized":"2024-09-11T08:30:13.678107+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'42","last_scrub_stamp":"2024-09-11T08:30:13.677992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:30:13.677992+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:42.259390+0000","scrub_duration":0.0024195800000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":468,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.596102+0000","last_change":"2024-09-11T08:29:19.403998+0000","last_active":"2024-09-11T08:30:11.596102+0000","last_peered":"2024-09-11T08:30:11.596102+0000","last_clean":"2024-09-11T08:30:11.596102+0000","last_became_active":"2024-09-11T08:17:37.553579+0000","last_became_peered":"2024-09-11T08:17:37.553579+0000","last_unstale":"2024-09-11T08:30:11.596102+0000","last_undegraded":"2024-09-11T08:30:11.596102+0000","last_fullsized":"2024-09-11T08:30:11.596102+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:19.403914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:29:19.403914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:40.219628+0000","scrub_duration":0.0063698699999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":517,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:10.835825+0000","last_change":"2024-09-11T08:28:58.174623+0000","last_active":"2024-09-11T08:30:10.835825+0000","last_peered":"2024-09-11T08:30:10.835825+0000","last_clean":"2024-09-11T08:30:10.835825+0000","last_became_active":"2024-09-11T08:15:56.522606+0000","last_became_peered":"2024-09-11T08:15:56.522606+0000","last_unstale":"2024-09-11T08:30:10.835825+0000","last_undegraded":"2024-09-11T08:30:10.835825+0000","last_fullsized":"2024-09-11T08:30:10.835825+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:28:58.174534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:28:58.174534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:25.648809+0000","scrub_duration":0.0051811230000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":143,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:10.837354+0000","last_change":"2024-09-11T08:28:57.804622+0000","last_active":"2024-09-11T08:30:10.837354+0000","last_peered":"2024-09-11T08:30:10.837354+0000","last_clean":"2024-09-11T08:30:10.837354+0000","last_became_active":"2024-09-11T08:23:22.683474+0000","last_became_peered":"2024-09-11T08:23:22.683474+0000","last_unstale":"2024-09-11T08:30:10.837354+0000","last_undegraded":"2024-09-11T08:30:10.837354+0000","last_fullsized":"2024-09-11T08:30:10.837354+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:28:57.804216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:22:50.151585+0000","last_clean_scrub_stamp":"2024-09-11T08:28:57.804216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015908835999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"405'42","reported_seq":448,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.304684+0000","last_change":"2024-09-11T08:29:21.318169+0000","last_active":"2024-09-11T08:30:11.304684+0000","last_peered":"2024-09-11T08:30:11.304684+0000","last_clean":"2024-09-11T08:30:11.304684+0000","last_became_active":"2024-09-11T08:24:28.358175+0000","last_became_peered":"2024-09-11T08:24:28.358175+0000","last_unstale":"2024-09-11T08:30:11.304684+0000","last_undegraded":"2024-09-11T08:30:11.304684+0000","last_fullsized":"2024-09-11T08:30:11.304684+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'42","last_scrub_stamp":"2024-09-11T08:29:21.317951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:29:21.317951+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:41.618306+0000","scrub_duration":0.0023805409999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":497,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.271229+0000","last_change":"2024-09-11T08:28:59.885192+0000","last_active":"2024-09-11T08:30:11.271229+0000","last_peered":"2024-09-11T08:30:11.271229+0000","last_clean":"2024-09-11T08:30:11.271229+0000","last_became_active":"2024-09-11T08:28:09.953224+0000","last_became_peered":"2024-09-11T08:28:09.953224+0000","last_unstale":"2024-09-11T08:30:11.271229+0000","last_undegraded":"2024-09-11T08:30:11.271229+0000","last_fullsized":"2024-09-11T08:30:11.271229+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:28:59.884493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:47.960329+0000","last_clean_scrub_stamp":"2024-09-11T08:28:59.884493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:21.852463+0000","scrub_duration":0.012139997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"82'9","reported_seq":789,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:13.846452+0000","last_change":"2024-09-11T08:28:59.818823+0000","last_active":"2024-09-11T08:30:13.846452+0000","last_peered":"2024-09-11T08:30:13.846452+0000","last_clean":"2024-09-11T08:30:13.846452+0000","last_became_active":"2024-09-11T08:07:34.056143+0000","last_became_peered":"2024-09-11T08:07:34.056143+0000","last_unstale":"2024-09-11T08:30:13.846452+0000","last_undegraded":"2024-09-11T08:30:13.846452+0000","last_fullsized":"2024-09-11T08:30:13.846452+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-11T08:28:59.818645+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-09-11T08:21:00.841426+0000","last_clean_scrub_stamp":"2024-09-11T08:28:59.818645+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011589633,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":462,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.595488+0000","last_change":"2024-09-11T08:28:46.329629+0000","last_active":"2024-09-11T08:30:11.595488+0000","last_peered":"2024-09-11T08:30:11.595488+0000","last_clean":"2024-09-11T08:30:11.595488+0000","last_became_active":"2024-09-11T08:14:24.788968+0000","last_became_peered":"2024-09-11T08:14:24.788968+0000","last_unstale":"2024-09-11T08:30:11.595488+0000","last_undegraded":"2024-09-11T08:30:11.595488+0000","last_fullsized":"2024-09-11T08:30:11.595488+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:28:46.329538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:28:46.329538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017611370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"405'30","reported_seq":600,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:10.835416+0000","last_change":"2024-09-11T08:30:10.086374+0000","last_active":"2024-09-11T08:30:10.835416+0000","last_peered":"2024-09-11T08:30:10.835416+0000","last_clean":"2024-09-11T08:30:10.835416+0000","last_became_active":"2024-09-11T08:24:28.355516+0000","last_became_peered":"2024-09-11T08:24:28.355516+0000","last_unstale":"2024-09-11T08:30:10.835416+0000","last_undegraded":"2024-09-11T08:30:10.835416+0000","last_fullsized":"2024-09-11T08:30:10.835416+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'30","last_scrub_stamp":"2024-09-11T08:30:10.086268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:30:10.086268+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:38.169586+0000","scrub_duration":0.0028819319999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":198,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:10.836757+0000","last_change":"2024-09-11T08:27:52.923361+0000","last_active":"2024-09-11T08:30:10.836757+0000","last_peered":"2024-09-11T08:30:10.836757+0000","last_clean":"2024-09-11T08:30:10.836757+0000","last_became_active":"2024-09-11T08:24:28.352741+0000","last_became_peered":"2024-09-11T08:24:28.352741+0000","last_unstale":"2024-09-11T08:30:10.836757+0000","last_undegraded":"2024-09-11T08:30:10.836757+0000","last_fullsized":"2024-09-11T08:30:10.836757+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:52.923262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:27:52.923262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034940769999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"376'15","reported_seq":474,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:13.846359+0000","last_change":"2024-09-11T08:30:09.325453+0000","last_active":"2024-09-11T08:30:13.846359+0000","last_peered":"2024-09-11T08:30:13.846359+0000","last_clean":"2024-09-11T08:30:13.846359+0000","last_became_active":"2024-09-11T08:24:28.359787+0000","last_became_peered":"2024-09-11T08:24:28.359787+0000","last_unstale":"2024-09-11T08:30:13.846359+0000","last_undegraded":"2024-09-11T08:30:13.846359+0000","last_fullsized":"2024-09-11T08:30:13.846359+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'15","last_scrub_stamp":"2024-09-11T08:30:09.325345+0000","last_deep_scrub":"376'15","last_deep_scrub_stamp":"2024-09-11T08:26:02.786767+0000","last_clean_scrub_stamp":"2024-09-11T08:30:09.325345+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:23.599574+0000","scrub_duration":0.001768979,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"66'1","reported_seq":475,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.270592+0000","last_change":"2024-09-11T08:30:07.307684+0000","last_active":"2024-09-11T08:30:11.270592+0000","last_peered":"2024-09-11T08:30:11.270592+0000","last_clean":"2024-09-11T08:30:11.270592+0000","last_became_active":"2024-09-11T08:07:36.072593+0000","last_became_peered":"2024-09-11T08:07:36.072593+0000","last_unstale":"2024-09-11T08:30:11.270592+0000","last_undegraded":"2024-09-11T08:30:11.270592+0000","last_fullsized":"2024-09-11T08:30:11.270592+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-11T08:30:07.307628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:30:07.307628+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:24.772623+0000","scrub_duration":0.0021248930000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"405'47","reported_seq":688,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.271036+0000","last_change":"2024-09-11T08:30:03.281050+0000","last_active":"2024-09-11T08:30:11.271036+0000","last_peered":"2024-09-11T08:30:11.271036+0000","last_clean":"2024-09-11T08:30:11.271036+0000","last_became_active":"2024-09-11T08:24:28.364043+0000","last_became_peered":"2024-09-11T08:24:28.364043+0000","last_unstale":"2024-09-11T08:30:11.271036+0000","last_undegraded":"2024-09-11T08:30:11.271036+0000","last_fullsized":"2024-09-11T08:30:11.271036+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'47","last_scrub_stamp":"2024-09-11T08:30:03.280942+0000","last_deep_scrub":"396'37","last_deep_scrub_stamp":"2024-09-11T08:27:16.673846+0000","last_clean_scrub_stamp":"2024-09-11T08:30:03.280942+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:25.945261+0000","scrub_duration":0.0024639179999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":484,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.488849+0000","last_change":"2024-09-11T08:29:57.672185+0000","last_active":"2024-09-11T08:30:11.488849+0000","last_peered":"2024-09-11T08:30:11.488849+0000","last_clean":"2024-09-11T08:30:11.488849+0000","last_became_active":"2024-09-11T08:27:57.795859+0000","last_became_peered":"2024-09-11T08:27:57.795859+0000","last_unstale":"2024-09-11T08:30:11.488849+0000","last_undegraded":"2024-09-11T08:30:11.488849+0000","last_fullsized":"2024-09-11T08:30:11.488849+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:57.672105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:28:03.117012+0000","last_clean_scrub_stamp":"2024-09-11T08:29:57.672105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:00.681043+0000","scrub_duration":0.011006079,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":540,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.304203+0000","last_change":"2024-09-11T08:29:24.232937+0000","last_active":"2024-09-11T08:30:11.304203+0000","last_peered":"2024-09-11T08:30:11.304203+0000","last_clean":"2024-09-11T08:30:11.304203+0000","last_became_active":"2024-09-11T08:24:28.360235+0000","last_became_peered":"2024-09-11T08:24:28.360235+0000","last_unstale":"2024-09-11T08:30:11.304203+0000","last_undegraded":"2024-09-11T08:30:11.304203+0000","last_fullsized":"2024-09-11T08:30:11.304203+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:24.232816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:29:24.232816+0000","last_clean_scrub_stamp":"2024-09-11T08:29:24.232816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:42.768241+0000","scrub_duration":0.0060421820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":491,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:10.835084+0000","last_change":"2024-09-11T08:27:56.203308+0000","last_active":"2024-09-11T08:30:10.835084+0000","last_peered":"2024-09-11T08:30:10.835084+0000","last_clean":"2024-09-11T08:30:10.835084+0000","last_became_active":"2024-09-11T08:07:36.631668+0000","last_became_peered":"2024-09-11T08:07:36.631668+0000","last_unstale":"2024-09-11T08:30:10.835084+0000","last_undegraded":"2024-09-11T08:30:10.835084+0000","last_fullsized":"2024-09-11T08:30:10.835084+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:56.203218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:58.517624+0000","last_clean_scrub_stamp":"2024-09-11T08:27:56.203218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010650626,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"405'64","reported_seq":355,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.902008+0000","last_change":"2024-09-11T08:28:00.205458+0000","last_active":"2024-09-11T08:30:16.902008+0000","last_peered":"2024-09-11T08:30:16.902008+0000","last_clean":"2024-09-11T08:30:16.902008+0000","last_became_active":"2024-09-11T08:20:52.086843+0000","last_became_peered":"2024-09-11T08:20:52.086843+0000","last_unstale":"2024-09-11T08:30:16.902008+0000","last_undegraded":"2024-09-11T08:30:16.902008+0000","last_fullsized":"2024-09-11T08:30:16.902008+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'64","last_scrub_stamp":"2024-09-11T08:28:00.205375+0000","last_deep_scrub":"190'39","last_deep_scrub_stamp":"2024-09-11T08:15:52.847968+0000","last_clean_scrub_stamp":"2024-09-11T08:28:00.205375+0000","objects_scrubbed":10,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0087966820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":84,"num_read_kb":69,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"59'1","reported_seq":151,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.271077+0000","last_change":"2024-09-11T08:28:53.832466+0000","last_active":"2024-09-11T08:30:11.271077+0000","last_peered":"2024-09-11T08:30:11.271077+0000","last_clean":"2024-09-11T08:30:11.271077+0000","last_became_active":"2024-09-11T08:23:22.691726+0000","last_became_peered":"2024-09-11T08:23:22.691726+0000","last_unstale":"2024-09-11T08:30:11.271077+0000","last_undegraded":"2024-09-11T08:30:11.271077+0000","last_fullsized":"2024-09-11T08:30:11.271077+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-11T08:28:53.832355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:28:53.832355+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0063587790000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":207,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:10.835421+0000","last_change":"2024-09-11T08:28:51.863387+0000","last_active":"2024-09-11T08:30:10.835421+0000","last_peered":"2024-09-11T08:30:10.835421+0000","last_clean":"2024-09-11T08:30:10.835421+0000","last_became_active":"2024-09-11T08:24:28.362590+0000","last_became_peered":"2024-09-11T08:24:28.362590+0000","last_unstale":"2024-09-11T08:30:10.835421+0000","last_undegraded":"2024-09-11T08:30:10.835421+0000","last_fullsized":"2024-09-11T08:30:10.835421+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-11T08:28:51.863309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:28:51.863309+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00679483,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":473,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.270839+0000","last_change":"2024-09-11T08:28:50.908738+0000","last_active":"2024-09-11T08:30:11.270839+0000","last_peered":"2024-09-11T08:30:11.270839+0000","last_clean":"2024-09-11T08:30:11.270839+0000","last_became_active":"2024-09-11T08:07:34.057154+0000","last_became_peered":"2024-09-11T08:07:34.057154+0000","last_unstale":"2024-09-11T08:30:11.270839+0000","last_undegraded":"2024-09-11T08:30:11.270839+0000","last_fullsized":"2024-09-11T08:30:11.270839+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:28:50.908650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:10:37.957096+0000","last_clean_scrub_stamp":"2024-09-11T08:28:50.908650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:18.089391+0000","scrub_duration":0.014372642,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":519,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:17.821074+0000","last_change":"2024-09-11T08:30:17.821049+0000","last_active":"2024-09-11T08:30:17.821074+0000","last_peered":"2024-09-11T08:30:17.821074+0000","last_clean":"2024-09-11T08:30:17.821074+0000","last_became_active":"2024-09-11T08:15:56.525099+0000","last_became_peered":"2024-09-11T08:15:56.525099+0000","last_unstale":"2024-09-11T08:30:17.821074+0000","last_undegraded":"2024-09-11T08:30:17.821074+0000","last_fullsized":"2024-09-11T08:30:17.821074+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:17.820997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:30:17.820997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:24.444375+0000","scrub_duration":0.0016474370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"405'21","reported_seq":604,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.906332+0000","last_change":"2024-09-11T08:30:16.906268+0000","last_active":"2024-09-11T08:30:16.906332+0000","last_peered":"2024-09-11T08:30:16.906332+0000","last_clean":"2024-09-11T08:30:16.906332+0000","last_became_active":"2024-09-11T08:24:28.358121+0000","last_became_peered":"2024-09-11T08:24:28.358121+0000","last_unstale":"2024-09-11T08:30:16.906332+0000","last_undegraded":"2024-09-11T08:30:16.906332+0000","last_fullsized":"2024-09-11T08:30:16.906332+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'21","last_scrub_stamp":"2024-09-11T08:30:16.906178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:30:16.906178+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:35.667760+0000","scrub_duration":0.0058613850000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":510,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:10.834899+0000","last_change":"2024-09-11T08:29:15.868218+0000","last_active":"2024-09-11T08:30:10.834899+0000","last_peered":"2024-09-11T08:30:10.834899+0000","last_clean":"2024-09-11T08:30:10.834899+0000","last_became_active":"2024-09-11T08:07:30.047014+0000","last_became_peered":"2024-09-11T08:07:30.047014+0000","last_unstale":"2024-09-11T08:30:10.834899+0000","last_undegraded":"2024-09-11T08:30:10.834899+0000","last_fullsized":"2024-09-11T08:30:10.834899+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:15.867962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:29:15.867962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:39.510619+0000","scrub_duration":0.0075403279999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"127'2","reported_seq":537,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.488328+0000","last_change":"2024-09-11T08:30:08.554358+0000","last_active":"2024-09-11T08:30:11.488328+0000","last_peered":"2024-09-11T08:30:11.488328+0000","last_clean":"2024-09-11T08:30:11.488328+0000","last_became_active":"2024-09-11T08:07:30.045009+0000","last_became_peered":"2024-09-11T08:07:30.045009+0000","last_unstale":"2024-09-11T08:30:11.488328+0000","last_undegraded":"2024-09-11T08:30:11.488328+0000","last_fullsized":"2024-09-11T08:30:11.488328+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'2","last_scrub_stamp":"2024-09-11T08:30:08.554296+0000","last_deep_scrub":"127'2","last_deep_scrub_stamp":"2024-09-11T08:15:01.938497+0000","last_clean_scrub_stamp":"2024-09-11T08:30:08.554296+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:21.485645+0000","scrub_duration":0.0016669009999999999,"objects_trimmed":0,"snaptrim_duration":0.001859533,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":563,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.488914+0000","last_change":"2024-09-11T08:30:00.586632+0000","last_active":"2024-09-11T08:30:11.488914+0000","last_peered":"2024-09-11T08:30:11.488914+0000","last_clean":"2024-09-11T08:30:11.488914+0000","last_became_active":"2024-09-11T08:24:28.357141+0000","last_became_peered":"2024-09-11T08:24:28.357141+0000","last_unstale":"2024-09-11T08:30:11.488914+0000","last_undegraded":"2024-09-11T08:30:11.488914+0000","last_fullsized":"2024-09-11T08:30:11.488914+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:00.586547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:22:00.200872+0000","last_clean_scrub_stamp":"2024-09-11T08:30:00.586547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:09.658987+0000","scrub_duration":0.0097564220000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":223,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:14.511554+0000","last_change":"2024-09-11T08:30:14.511521+0000","last_active":"2024-09-11T08:30:14.511554+0000","last_peered":"2024-09-11T08:30:14.511554+0000","last_clean":"2024-09-11T08:30:14.511554+0000","last_became_active":"2024-09-11T08:20:16.108674+0000","last_became_peered":"2024-09-11T08:20:16.108674+0000","last_unstale":"2024-09-11T08:30:14.511554+0000","last_undegraded":"2024-09-11T08:30:14.511554+0000","last_fullsized":"2024-09-11T08:30:14.511554+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:14.511468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:58.796217+0000","last_clean_scrub_stamp":"2024-09-11T08:30:14.511468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:18.157477+0000","scrub_duration":0.001290072,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"405'42","reported_seq":545,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.271063+0000","last_change":"2024-09-11T08:29:10.054827+0000","last_active":"2024-09-11T08:30:11.271063+0000","last_peered":"2024-09-11T08:30:11.271063+0000","last_clean":"2024-09-11T08:30:11.271063+0000","last_became_active":"2024-09-11T08:07:32.055407+0000","last_became_peered":"2024-09-11T08:07:32.055407+0000","last_unstale":"2024-09-11T08:30:11.271063+0000","last_undegraded":"2024-09-11T08:30:11.271063+0000","last_fullsized":"2024-09-11T08:30:11.271063+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'42","last_scrub_stamp":"2024-09-11T08:29:10.054727+0000","last_deep_scrub":"287'30","last_deep_scrub_stamp":"2024-09-11T08:26:09.030192+0000","last_clean_scrub_stamp":"2024-09-11T08:29:10.054727+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:34.247277+0000","scrub_duration":0.042708410000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"448'2","reported_seq":495,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:18.171429+0000","last_change":"2024-09-11T08:30:18.171381+0000","last_active":"2024-09-11T08:30:18.171429+0000","last_peered":"2024-09-11T08:30:18.171429+0000","last_clean":"2024-09-11T08:30:18.171429+0000","last_became_active":"2024-09-11T08:07:30.050504+0000","last_became_peered":"2024-09-11T08:07:30.050504+0000","last_unstale":"2024-09-11T08:30:18.171429+0000","last_undegraded":"2024-09-11T08:30:18.171429+0000","last_fullsized":"2024-09-11T08:30:18.171429+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'2","last_scrub_stamp":"2024-09-11T08:30:18.171322+0000","last_deep_scrub":"448'2","last_deep_scrub_stamp":"2024-09-11T08:30:18.171322+0000","last_clean_scrub_stamp":"2024-09-11T08:30:18.171322+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:41.608010+0000","scrub_duration":0.0023495109999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"224'6","reported_seq":1632,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.861268+0000","last_change":"2024-09-11T08:28:06.324532+0000","last_active":"2024-09-11T08:30:16.861268+0000","last_peered":"2024-09-11T08:30:16.861268+0000","last_clean":"2024-09-11T08:30:16.861268+0000","last_became_active":"2024-09-11T08:14:24.793048+0000","last_became_peered":"2024-09-11T08:14:24.793048+0000","last_unstale":"2024-09-11T08:30:16.861268+0000","last_undegraded":"2024-09-11T08:30:16.861268+0000","last_fullsized":"2024-09-11T08:30:16.861268+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'6","last_scrub_stamp":"2024-09-11T08:28:06.324370+0000","last_deep_scrub":"224'6","last_deep_scrub_stamp":"2024-09-11T08:16:55.778284+0000","last_clean_scrub_stamp":"2024-09-11T08:28:06.324370+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036467399999999999,"objects_trimmed":0,"snaptrim_duration":0.000318821,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":958,"num_read_kb":953,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":190,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:10.835937+0000","last_change":"2024-09-11T08:27:14.117515+0000","last_active":"2024-09-11T08:30:10.835937+0000","last_peered":"2024-09-11T08:30:10.835937+0000","last_clean":"2024-09-11T08:30:10.835937+0000","last_became_active":"2024-09-11T08:24:28.353018+0000","last_became_peered":"2024-09-11T08:24:28.353018+0000","last_unstale":"2024-09-11T08:30:10.835937+0000","last_undegraded":"2024-09-11T08:30:10.835937+0000","last_fullsized":"2024-09-11T08:30:10.835937+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:14.114373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:27:14.114373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0066634249999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":534,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:10.836003+0000","last_change":"2024-09-11T08:27:11.025202+0000","last_active":"2024-09-11T08:30:10.836003+0000","last_peered":"2024-09-11T08:30:10.836003+0000","last_clean":"2024-09-11T08:30:10.836003+0000","last_became_active":"2024-09-11T08:24:28.358723+0000","last_became_peered":"2024-09-11T08:24:28.358723+0000","last_unstale":"2024-09-11T08:30:10.836003+0000","last_undegraded":"2024-09-11T08:30:10.836003+0000","last_fullsized":"2024-09-11T08:30:10.836003+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:11.025109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:27:11.025109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049163100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"405'45","reported_seq":587,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.304931+0000","last_change":"2024-09-11T08:29:13.340969+0000","last_active":"2024-09-11T08:30:11.304931+0000","last_peered":"2024-09-11T08:30:11.304931+0000","last_clean":"2024-09-11T08:30:11.304931+0000","last_became_active":"2024-09-11T08:14:24.789154+0000","last_became_peered":"2024-09-11T08:14:24.789154+0000","last_unstale":"2024-09-11T08:30:11.304931+0000","last_undegraded":"2024-09-11T08:30:11.304931+0000","last_fullsized":"2024-09-11T08:30:11.304931+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'45","last_scrub_stamp":"2024-09-11T08:29:13.340918+0000","last_deep_scrub":"84'21","last_deep_scrub_stamp":"2024-09-11T08:08:56.705306+0000","last_clean_scrub_stamp":"2024-09-11T08:29:13.340918+0000","objects_scrubbed":7,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:34.815204+0000","scrub_duration":0.001693937,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":202,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.289866+0000","last_change":"2024-09-11T08:30:16.289806+0000","last_active":"2024-09-11T08:30:16.289866+0000","last_peered":"2024-09-11T08:30:16.289866+0000","last_clean":"2024-09-11T08:30:16.289866+0000","last_became_active":"2024-09-11T08:24:28.362417+0000","last_became_peered":"2024-09-11T08:24:28.362417+0000","last_unstale":"2024-09-11T08:30:16.289866+0000","last_undegraded":"2024-09-11T08:30:16.289866+0000","last_fullsized":"2024-09-11T08:30:16.289866+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:16.289709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:07.749828+0000","last_clean_scrub_stamp":"2024-09-11T08:30:16.289709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:43.152596+0000","scrub_duration":0.0013265880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":306,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.595831+0000","last_change":"2024-09-11T08:28:01.294768+0000","last_active":"2024-09-11T08:30:11.595831+0000","last_peered":"2024-09-11T08:30:11.595831+0000","last_clean":"2024-09-11T08:30:11.595831+0000","last_became_active":"2024-09-11T08:15:56.523430+0000","last_became_peered":"2024-09-11T08:15:56.523430+0000","last_unstale":"2024-09-11T08:30:11.595831+0000","last_undegraded":"2024-09-11T08:30:11.595831+0000","last_fullsized":"2024-09-11T08:30:11.595831+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-11T08:28:01.294597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:05:21.231856+0000","last_clean_scrub_stamp":"2024-09-11T08:28:01.294597+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027233790000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":330,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.595688+0000","last_change":"2024-09-11T08:27:49.296146+0000","last_active":"2024-09-11T08:30:11.595688+0000","last_peered":"2024-09-11T08:30:11.595688+0000","last_clean":"2024-09-11T08:30:11.595688+0000","last_became_active":"2024-09-11T08:24:28.357166+0000","last_became_peered":"2024-09-11T08:24:28.357166+0000","last_unstale":"2024-09-11T08:30:11.595688+0000","last_undegraded":"2024-09-11T08:30:11.595688+0000","last_fullsized":"2024-09-11T08:30:11.595688+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:49.296063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:20:25.775869+0000","last_clean_scrub_stamp":"2024-09-11T08:27:49.296063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037448009999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":257,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:18.358814+0000","last_change":"2024-09-11T08:30:18.358789+0000","last_active":"2024-09-11T08:30:18.358814+0000","last_peered":"2024-09-11T08:30:18.358814+0000","last_clean":"2024-09-11T08:30:18.358814+0000","last_became_active":"2024-09-11T08:24:28.356071+0000","last_became_peered":"2024-09-11T08:24:28.356071+0000","last_unstale":"2024-09-11T08:30:18.358814+0000","last_undegraded":"2024-09-11T08:30:18.358814+0000","last_fullsized":"2024-09-11T08:30:18.358814+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:18.358735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:01.777429+0000","last_clean_scrub_stamp":"2024-09-11T08:30:18.358735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:32.931299+0000","scrub_duration":0.001168838,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"405'32","reported_seq":433,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.304867+0000","last_change":"2024-09-11T08:29:10.333244+0000","last_active":"2024-09-11T08:30:11.304867+0000","last_peered":"2024-09-11T08:30:11.304867+0000","last_clean":"2024-09-11T08:30:11.304867+0000","last_became_active":"2024-09-11T08:24:28.368759+0000","last_became_peered":"2024-09-11T08:24:28.368759+0000","last_unstale":"2024-09-11T08:30:11.304867+0000","last_undegraded":"2024-09-11T08:30:11.304867+0000","last_fullsized":"2024-09-11T08:30:11.304867+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'32","last_scrub_stamp":"2024-09-11T08:29:10.333163+0000","last_deep_scrub":"405'32","last_deep_scrub_stamp":"2024-09-11T08:29:10.333163+0000","last_clean_scrub_stamp":"2024-09-11T08:29:10.333163+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:32.092567+0000","scrub_duration":0.0057426860000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":524,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:10.836048+0000","last_change":"2024-09-11T08:30:07.260981+0000","last_active":"2024-09-11T08:30:10.836048+0000","last_peered":"2024-09-11T08:30:10.836048+0000","last_clean":"2024-09-11T08:30:10.836048+0000","last_became_active":"2024-09-11T08:14:24.789888+0000","last_became_peered":"2024-09-11T08:14:24.789888+0000","last_unstale":"2024-09-11T08:30:10.836048+0000","last_undegraded":"2024-09-11T08:30:10.836048+0000","last_fullsized":"2024-09-11T08:30:10.836048+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:07.260906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:36.417911+0000","last_clean_scrub_stamp":"2024-09-11T08:30:07.260906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:31.737712+0000","scrub_duration":0.0037043279999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":194,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.304989+0000","last_change":"2024-09-11T08:30:06.443515+0000","last_active":"2024-09-11T08:30:11.304989+0000","last_peered":"2024-09-11T08:30:11.304989+0000","last_clean":"2024-09-11T08:30:11.304989+0000","last_became_active":"2024-09-11T08:24:28.361261+0000","last_became_peered":"2024-09-11T08:24:28.361261+0000","last_unstale":"2024-09-11T08:30:11.304989+0000","last_undegraded":"2024-09-11T08:30:11.304989+0000","last_fullsized":"2024-09-11T08:30:11.304989+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:06.443433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:01.558340+0000","last_clean_scrub_stamp":"2024-09-11T08:30:06.443433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:23.309677+0000","scrub_duration":0.001552765,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":322,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.304711+0000","last_change":"2024-09-11T08:29:16.340861+0000","last_active":"2024-09-11T08:30:11.304711+0000","last_peered":"2024-09-11T08:30:11.304711+0000","last_clean":"2024-09-11T08:30:11.304711+0000","last_became_active":"2024-09-11T08:24:28.357842+0000","last_became_peered":"2024-09-11T08:24:28.357842+0000","last_unstale":"2024-09-11T08:30:11.304711+0000","last_undegraded":"2024-09-11T08:30:11.304711+0000","last_fullsized":"2024-09-11T08:30:11.304711+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:16.340778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:29:16.340778+0000","last_clean_scrub_stamp":"2024-09-11T08:29:16.340778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:43.003239+0000","scrub_duration":0.0019631459999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"405'53","reported_seq":494,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.488071+0000","last_change":"2024-09-11T08:30:02.596837+0000","last_active":"2024-09-11T08:30:11.488071+0000","last_peered":"2024-09-11T08:30:11.488071+0000","last_clean":"2024-09-11T08:30:11.488071+0000","last_became_active":"2024-09-11T08:23:22.681452+0000","last_became_peered":"2024-09-11T08:23:22.681452+0000","last_unstale":"2024-09-11T08:30:11.488071+0000","last_undegraded":"2024-09-11T08:30:11.488071+0000","last_fullsized":"2024-09-11T08:30:11.488071+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'53","last_scrub_stamp":"2024-09-11T08:30:02.596714+0000","last_deep_scrub":"77'25","last_deep_scrub_stamp":"2024-09-11T08:08:32.458210+0000","last_clean_scrub_stamp":"2024-09-11T08:30:02.596714+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:30.319012+0000","scrub_duration":0.0038267459999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":513,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.304436+0000","last_change":"2024-09-11T08:29:57.622728+0000","last_active":"2024-09-11T08:30:11.304436+0000","last_peered":"2024-09-11T08:30:11.304436+0000","last_clean":"2024-09-11T08:30:11.304436+0000","last_became_active":"2024-09-11T08:07:30.042555+0000","last_became_peered":"2024-09-11T08:07:30.042555+0000","last_unstale":"2024-09-11T08:30:11.304436+0000","last_undegraded":"2024-09-11T08:30:11.304436+0000","last_fullsized":"2024-09-11T08:30:11.304436+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:57.622675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:29:57.622675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:21.379775+0000","scrub_duration":0.0036282020000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":493,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:10.836075+0000","last_change":"2024-09-11T08:27:49.130976+0000","last_active":"2024-09-11T08:30:10.836075+0000","last_peered":"2024-09-11T08:30:10.836075+0000","last_clean":"2024-09-11T08:30:10.836075+0000","last_became_active":"2024-09-11T08:07:34.054961+0000","last_became_peered":"2024-09-11T08:07:34.054961+0000","last_unstale":"2024-09-11T08:30:10.836075+0000","last_undegraded":"2024-09-11T08:30:10.836075+0000","last_fullsized":"2024-09-11T08:30:10.836075+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:49.130900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:27:49.130900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004734121,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":521,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.488769+0000","last_change":"2024-09-11T08:29:56.700851+0000","last_active":"2024-09-11T08:30:11.488769+0000","last_peered":"2024-09-11T08:30:11.488769+0000","last_clean":"2024-09-11T08:30:11.488769+0000","last_became_active":"2024-09-11T08:07:36.071194+0000","last_became_peered":"2024-09-11T08:07:36.071194+0000","last_unstale":"2024-09-11T08:30:11.488769+0000","last_undegraded":"2024-09-11T08:30:11.488769+0000","last_fullsized":"2024-09-11T08:30:11.488769+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:56.700632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:07.930137+0000","last_clean_scrub_stamp":"2024-09-11T08:29:56.700632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:21.407208+0000","scrub_duration":0.0045560790000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"405'35","reported_seq":428,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.304481+0000","last_change":"2024-09-11T08:30:01.519867+0000","last_active":"2024-09-11T08:30:11.304481+0000","last_peered":"2024-09-11T08:30:11.304481+0000","last_clean":"2024-09-11T08:30:11.304481+0000","last_became_active":"2024-09-11T08:23:22.688631+0000","last_became_peered":"2024-09-11T08:23:22.688631+0000","last_unstale":"2024-09-11T08:30:11.304481+0000","last_undegraded":"2024-09-11T08:30:11.304481+0000","last_fullsized":"2024-09-11T08:30:11.304481+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'35","last_scrub_stamp":"2024-09-11T08:30:01.519776+0000","last_deep_scrub":"224'15","last_deep_scrub_stamp":"2024-09-11T08:17:01.126478+0000","last_clean_scrub_stamp":"2024-09-11T08:30:01.519776+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:15.125283+0000","scrub_duration":0.0073496450000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":272,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:17.198489+0000","last_change":"2024-09-11T08:30:17.198458+0000","last_active":"2024-09-11T08:30:17.198489+0000","last_peered":"2024-09-11T08:30:17.198489+0000","last_clean":"2024-09-11T08:30:17.198489+0000","last_became_active":"2024-09-11T08:17:37.557278+0000","last_became_peered":"2024-09-11T08:17:37.557278+0000","last_unstale":"2024-09-11T08:30:17.198489+0000","last_undegraded":"2024-09-11T08:30:17.198489+0000","last_fullsized":"2024-09-11T08:30:17.198489+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:17.198403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:27:59.866811+0000","last_clean_scrub_stamp":"2024-09-11T08:30:17.198403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:41.884498+0000","scrub_duration":0.0012428210000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"354'16","reported_seq":855,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:13.846583+0000","last_change":"2024-09-11T08:29:06.507763+0000","last_active":"2024-09-11T08:30:13.846583+0000","last_peered":"2024-09-11T08:30:13.846583+0000","last_clean":"2024-09-11T08:30:13.846583+0000","last_became_active":"2024-09-11T08:24:28.356749+0000","last_became_peered":"2024-09-11T08:24:28.356749+0000","last_unstale":"2024-09-11T08:30:13.846583+0000","last_undegraded":"2024-09-11T08:30:13.846583+0000","last_fullsized":"2024-09-11T08:30:13.846583+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'16","last_scrub_stamp":"2024-09-11T08:29:06.507663+0000","last_deep_scrub":"171'12","last_deep_scrub_stamp":"2024-09-11T08:14:56.881299+0000","last_clean_scrub_stamp":"2024-09-11T08:29:06.507663+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:33.390159+0000","scrub_duration":0.015460094000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":152,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:10.835809+0000","last_change":"2024-09-11T08:29:14.863576+0000","last_active":"2024-09-11T08:30:10.835809+0000","last_peered":"2024-09-11T08:30:10.835809+0000","last_clean":"2024-09-11T08:30:10.835809+0000","last_became_active":"2024-09-11T08:23:22.686992+0000","last_became_peered":"2024-09-11T08:23:22.686992+0000","last_unstale":"2024-09-11T08:30:10.835809+0000","last_undegraded":"2024-09-11T08:30:10.835809+0000","last_fullsized":"2024-09-11T08:30:10.835809+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:14.863324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:14.613532+0000","last_clean_scrub_stamp":"2024-09-11T08:29:14.863324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:30.823910+0000","scrub_duration":0.0069445239999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"405'21","reported_seq":509,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.270917+0000","last_change":"2024-09-11T08:28:02.865865+0000","last_active":"2024-09-11T08:30:11.270917+0000","last_peered":"2024-09-11T08:30:11.270917+0000","last_clean":"2024-09-11T08:30:11.270917+0000","last_became_active":"2024-09-11T08:07:32.054885+0000","last_became_peered":"2024-09-11T08:07:32.054885+0000","last_unstale":"2024-09-11T08:30:11.270917+0000","last_undegraded":"2024-09-11T08:30:11.270917+0000","last_fullsized":"2024-09-11T08:30:11.270917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'21","last_scrub_stamp":"2024-09-11T08:28:02.865748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:28:02.865748+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.016060689,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":198,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:10.837325+0000","last_change":"2024-09-11T08:28:54.875865+0000","last_active":"2024-09-11T08:30:10.837325+0000","last_peered":"2024-09-11T08:30:10.837325+0000","last_clean":"2024-09-11T08:30:10.837325+0000","last_became_active":"2024-09-11T08:24:28.359057+0000","last_became_peered":"2024-09-11T08:24:28.359057+0000","last_unstale":"2024-09-11T08:30:10.837325+0000","last_undegraded":"2024-09-11T08:30:10.837325+0000","last_fullsized":"2024-09-11T08:30:10.837325+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:28:54.875776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:28:54.875776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022324210000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":533,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.488370+0000","last_change":"2024-09-11T08:30:09.510347+0000","last_active":"2024-09-11T08:30:11.488370+0000","last_peered":"2024-09-11T08:30:11.488370+0000","last_clean":"2024-09-11T08:30:11.488370+0000","last_became_active":"2024-09-11T08:07:34.056606+0000","last_became_peered":"2024-09-11T08:07:34.056606+0000","last_unstale":"2024-09-11T08:30:11.488370+0000","last_undegraded":"2024-09-11T08:30:11.488370+0000","last_fullsized":"2024-09-11T08:30:11.488370+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:09.510248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:30:09.510248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:16.060314+0000","scrub_duration":0.0016592110000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":498,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.271367+0000","last_change":"2024-09-11T08:28:46.893924+0000","last_active":"2024-09-11T08:30:11.271367+0000","last_peered":"2024-09-11T08:30:11.271367+0000","last_clean":"2024-09-11T08:30:11.271367+0000","last_became_active":"2024-09-11T08:15:56.518884+0000","last_became_peered":"2024-09-11T08:15:56.518884+0000","last_unstale":"2024-09-11T08:30:11.271367+0000","last_undegraded":"2024-09-11T08:30:11.271367+0000","last_fullsized":"2024-09-11T08:30:11.271367+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:28:46.893851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:18:32.278257+0000","last_clean_scrub_stamp":"2024-09-11T08:28:46.893851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0085054269999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"405'51","reported_seq":661,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.304613+0000","last_change":"2024-09-11T08:29:19.302174+0000","last_active":"2024-09-11T08:30:11.304613+0000","last_peered":"2024-09-11T08:30:11.304613+0000","last_clean":"2024-09-11T08:30:11.304613+0000","last_became_active":"2024-09-11T08:15:56.526961+0000","last_became_peered":"2024-09-11T08:15:56.526961+0000","last_unstale":"2024-09-11T08:30:11.304613+0000","last_undegraded":"2024-09-11T08:30:11.304613+0000","last_fullsized":"2024-09-11T08:30:11.304613+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'51","last_scrub_stamp":"2024-09-11T08:29:19.302094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:29:19.302094+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:24.531066+0000","scrub_duration":0.0045766959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":340,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.595815+0000","last_change":"2024-09-11T08:30:10.617869+0000","last_active":"2024-09-11T08:30:11.595815+0000","last_peered":"2024-09-11T08:30:11.595815+0000","last_clean":"2024-09-11T08:30:11.595815+0000","last_became_active":"2024-09-11T08:24:28.353720+0000","last_became_peered":"2024-09-11T08:24:28.353720+0000","last_unstale":"2024-09-11T08:30:11.595815+0000","last_undegraded":"2024-09-11T08:30:11.595815+0000","last_fullsized":"2024-09-11T08:30:11.595815+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:10.617803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:44.385372+0000","last_clean_scrub_stamp":"2024-09-11T08:30:10.617803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:30.349657+0000","scrub_duration":0.0015332709999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":553,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.488409+0000","last_change":"2024-09-11T08:29:11.612083+0000","last_active":"2024-09-11T08:30:11.488409+0000","last_peered":"2024-09-11T08:30:11.488409+0000","last_clean":"2024-09-11T08:30:11.488409+0000","last_became_active":"2024-09-11T08:24:28.365792+0000","last_became_peered":"2024-09-11T08:24:28.365792+0000","last_unstale":"2024-09-11T08:30:11.488409+0000","last_undegraded":"2024-09-11T08:30:11.488409+0000","last_fullsized":"2024-09-11T08:30:11.488409+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:11.611140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:06.546573+0000","last_clean_scrub_stamp":"2024-09-11T08:29:11.611140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:30.173424+0000","scrub_duration":0.006733915,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":300,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.595825+0000","last_change":"2024-09-11T08:28:55.493273+0000","last_active":"2024-09-11T08:30:11.595825+0000","last_peered":"2024-09-11T08:30:11.595825+0000","last_clean":"2024-09-11T08:30:11.595825+0000","last_became_active":"2024-09-11T08:15:56.518169+0000","last_became_peered":"2024-09-11T08:15:56.518169+0000","last_unstale":"2024-09-11T08:30:11.595825+0000","last_undegraded":"2024-09-11T08:30:11.595825+0000","last_fullsized":"2024-09-11T08:30:11.595825+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:28:55.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:28:55.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0069241010000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"405'29","reported_seq":413,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:10.835171+0000","last_change":"2024-09-11T08:27:22.427676+0000","last_active":"2024-09-11T08:30:10.835171+0000","last_peered":"2024-09-11T08:30:10.835171+0000","last_clean":"2024-09-11T08:30:10.835171+0000","last_became_active":"2024-09-11T08:23:22.686387+0000","last_became_peered":"2024-09-11T08:23:22.686387+0000","last_unstale":"2024-09-11T08:30:10.835171+0000","last_undegraded":"2024-09-11T08:30:10.835171+0000","last_fullsized":"2024-09-11T08:30:10.835171+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'23","last_scrub_stamp":"2024-09-11T08:27:22.427580+0000","last_deep_scrub":"145'17","last_deep_scrub_stamp":"2024-09-11T08:13:08.160848+0000","last_clean_scrub_stamp":"2024-09-11T08:27:22.427580+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.17013154699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":310,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.596072+0000","last_change":"2024-09-11T08:29:02.433750+0000","last_active":"2024-09-11T08:30:11.596072+0000","last_peered":"2024-09-11T08:30:11.596072+0000","last_clean":"2024-09-11T08:30:11.596072+0000","last_became_active":"2024-09-11T08:15:56.525845+0000","last_became_peered":"2024-09-11T08:15:56.525845+0000","last_unstale":"2024-09-11T08:30:11.596072+0000","last_undegraded":"2024-09-11T08:30:11.596072+0000","last_fullsized":"2024-09-11T08:30:11.596072+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:02.433672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:21:06.262572+0000","last_clean_scrub_stamp":"2024-09-11T08:29:02.433672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0082380349999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":513,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:18.591728+0000","last_change":"2024-09-11T08:30:18.591703+0000","last_active":"2024-09-11T08:30:18.591728+0000","last_peered":"2024-09-11T08:30:18.591728+0000","last_clean":"2024-09-11T08:30:18.591728+0000","last_became_active":"2024-09-11T08:07:34.057479+0000","last_became_peered":"2024-09-11T08:07:34.057479+0000","last_unstale":"2024-09-11T08:30:18.591728+0000","last_undegraded":"2024-09-11T08:30:18.591728+0000","last_fullsized":"2024-09-11T08:30:18.591728+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:18.591649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:30:18.591649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:42.108591+0000","scrub_duration":0.0013128949999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":519,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:10.837183+0000","last_change":"2024-09-11T08:29:01.755692+0000","last_active":"2024-09-11T08:30:10.837183+0000","last_peered":"2024-09-11T08:30:10.837183+0000","last_clean":"2024-09-11T08:30:10.837183+0000","last_became_active":"2024-09-11T08:15:56.522624+0000","last_became_peered":"2024-09-11T08:15:56.522624+0000","last_unstale":"2024-09-11T08:30:10.837183+0000","last_undegraded":"2024-09-11T08:30:10.837183+0000","last_fullsized":"2024-09-11T08:30:10.837183+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:01.755554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:29:01.755554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037540439999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"405'39","reported_seq":561,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.596432+0000","last_change":"2024-09-11T08:30:11.596345+0000","last_active":"2024-09-11T08:30:11.596432+0000","last_peered":"2024-09-11T08:30:11.596432+0000","last_clean":"2024-09-11T08:30:11.596432+0000","last_became_active":"2024-09-11T08:24:28.353397+0000","last_became_peered":"2024-09-11T08:24:28.353397+0000","last_unstale":"2024-09-11T08:30:11.596432+0000","last_undegraded":"2024-09-11T08:30:11.596432+0000","last_fullsized":"2024-09-11T08:30:11.596432+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'39","last_scrub_stamp":"2024-09-11T08:30:11.596278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:30:11.596278+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:28.358869+0000","scrub_duration":0.0031679740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":306,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.596207+0000","last_change":"2024-09-11T08:28:45.319406+0000","last_active":"2024-09-11T08:30:11.596207+0000","last_peered":"2024-09-11T08:30:11.596207+0000","last_clean":"2024-09-11T08:30:11.596207+0000","last_became_active":"2024-09-11T08:15:56.515328+0000","last_became_peered":"2024-09-11T08:15:56.515328+0000","last_unstale":"2024-09-11T08:30:11.596207+0000","last_undegraded":"2024-09-11T08:30:11.596207+0000","last_fullsized":"2024-09-11T08:30:11.596207+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-11T08:28:45.319230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:28:45.319230+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0062126519999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"209'11","reported_seq":476,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:13.846363+0000","last_change":"2024-09-11T08:29:04.443432+0000","last_active":"2024-09-11T08:30:13.846363+0000","last_peered":"2024-09-11T08:30:13.846363+0000","last_clean":"2024-09-11T08:30:13.846363+0000","last_became_active":"2024-09-11T08:15:56.524513+0000","last_became_peered":"2024-09-11T08:15:56.524513+0000","last_unstale":"2024-09-11T08:30:13.846363+0000","last_undegraded":"2024-09-11T08:30:13.846363+0000","last_fullsized":"2024-09-11T08:30:13.846363+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"209'11","last_scrub_stamp":"2024-09-11T08:29:04.443335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:29:04.443335+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:21.877030+0000","scrub_duration":0.0041068340000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":198,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:14.310748+0000","last_change":"2024-09-11T08:30:14.310713+0000","last_active":"2024-09-11T08:30:14.310748+0000","last_peered":"2024-09-11T08:30:14.310748+0000","last_clean":"2024-09-11T08:30:14.310748+0000","last_became_active":"2024-09-11T08:24:28.360658+0000","last_became_peered":"2024-09-11T08:24:28.360658+0000","last_unstale":"2024-09-11T08:30:14.310748+0000","last_undegraded":"2024-09-11T08:30:14.310748+0000","last_fullsized":"2024-09-11T08:30:14.310748+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:14.310661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:03.484755+0000","last_clean_scrub_stamp":"2024-09-11T08:30:14.310661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:19.401197+0000","scrub_duration":0.0012596440000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":547,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.304351+0000","last_change":"2024-09-11T08:29:17.333213+0000","last_active":"2024-09-11T08:30:11.304351+0000","last_peered":"2024-09-11T08:30:11.304351+0000","last_clean":"2024-09-11T08:30:11.304351+0000","last_became_active":"2024-09-11T08:24:28.357401+0000","last_became_peered":"2024-09-11T08:24:28.357401+0000","last_unstale":"2024-09-11T08:30:11.304351+0000","last_undegraded":"2024-09-11T08:30:11.304351+0000","last_fullsized":"2024-09-11T08:30:11.304351+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:17.333163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:29:17.333163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:32.873535+0000","scrub_duration":0.0015736540000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"405'21","reported_seq":569,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:10.837311+0000","last_change":"2024-09-11T08:29:15.625135+0000","last_active":"2024-09-11T08:30:10.837311+0000","last_peered":"2024-09-11T08:30:10.837311+0000","last_clean":"2024-09-11T08:30:10.837311+0000","last_became_active":"2024-09-11T08:15:56.526325+0000","last_became_peered":"2024-09-11T08:15:56.526325+0000","last_unstale":"2024-09-11T08:30:10.837311+0000","last_undegraded":"2024-09-11T08:30:10.837311+0000","last_fullsized":"2024-09-11T08:30:10.837311+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'21","last_scrub_stamp":"2024-09-11T08:29:15.624982+0000","last_deep_scrub":"405'21","last_deep_scrub_stamp":"2024-09-11T08:29:15.624982+0000","last_clean_scrub_stamp":"2024-09-11T08:29:15.624982+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:33.209933+0000","scrub_duration":0.010445571000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":198,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:12.778547+0000","last_change":"2024-09-11T08:30:12.778506+0000","last_active":"2024-09-11T08:30:12.778547+0000","last_peered":"2024-09-11T08:30:12.778547+0000","last_clean":"2024-09-11T08:30:12.778547+0000","last_became_active":"2024-09-11T08:24:28.358300+0000","last_became_peered":"2024-09-11T08:24:28.358300+0000","last_unstale":"2024-09-11T08:30:12.778547+0000","last_undegraded":"2024-09-11T08:30:12.778547+0000","last_fullsized":"2024-09-11T08:30:12.778547+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:12.778451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:53.548967+0000","last_clean_scrub_stamp":"2024-09-11T08:30:12.778451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:30.240863+0000","scrub_duration":0.001203455,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":435,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.595967+0000","last_change":"2024-09-11T08:29:20.429043+0000","last_active":"2024-09-11T08:30:11.595967+0000","last_peered":"2024-09-11T08:30:11.595967+0000","last_clean":"2024-09-11T08:30:11.595967+0000","last_became_active":"2024-09-11T08:14:24.790678+0000","last_became_peered":"2024-09-11T08:14:24.790678+0000","last_unstale":"2024-09-11T08:30:11.595967+0000","last_undegraded":"2024-09-11T08:30:11.595967+0000","last_fullsized":"2024-09-11T08:30:11.595967+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:20.428966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:29:20.428966+0000","last_clean_scrub_stamp":"2024-09-11T08:29:20.428966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:46.351908+0000","scrub_duration":0.005613697,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"405'25","reported_seq":560,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.271155+0000","last_change":"2024-09-11T08:30:09.312537+0000","last_active":"2024-09-11T08:30:11.271155+0000","last_peered":"2024-09-11T08:30:11.271155+0000","last_clean":"2024-09-11T08:30:11.271155+0000","last_became_active":"2024-09-11T08:15:56.514793+0000","last_became_peered":"2024-09-11T08:15:56.514793+0000","last_unstale":"2024-09-11T08:30:11.271155+0000","last_undegraded":"2024-09-11T08:30:11.271155+0000","last_fullsized":"2024-09-11T08:30:11.271155+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'25","last_scrub_stamp":"2024-09-11T08:30:09.312479+0000","last_deep_scrub":"84'13","last_deep_scrub_stamp":"2024-09-11T08:08:57.835634+0000","last_clean_scrub_stamp":"2024-09-11T08:30:09.312479+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:23.223570+0000","scrub_duration":0.0022105110000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":458,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.595386+0000","last_change":"2024-09-11T08:28:56.500137+0000","last_active":"2024-09-11T08:30:11.595386+0000","last_peered":"2024-09-11T08:30:11.595386+0000","last_clean":"2024-09-11T08:30:11.595386+0000","last_became_active":"2024-09-11T08:14:24.786021+0000","last_became_peered":"2024-09-11T08:14:24.786021+0000","last_unstale":"2024-09-11T08:30:11.595386+0000","last_undegraded":"2024-09-11T08:30:11.595386+0000","last_fullsized":"2024-09-11T08:30:11.595386+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:28:56.500057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:30.471620+0000","last_clean_scrub_stamp":"2024-09-11T08:28:56.500057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:23.579349+0000","scrub_duration":0.0062781299999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":521,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:10.837165+0000","last_change":"2024-09-11T08:29:57.839609+0000","last_active":"2024-09-11T08:30:10.837165+0000","last_peered":"2024-09-11T08:30:10.837165+0000","last_clean":"2024-09-11T08:30:10.837165+0000","last_became_active":"2024-09-11T08:15:56.525717+0000","last_became_peered":"2024-09-11T08:15:56.525717+0000","last_unstale":"2024-09-11T08:30:10.837165+0000","last_undegraded":"2024-09-11T08:30:10.837165+0000","last_fullsized":"2024-09-11T08:30:10.837165+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:57.839509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:04.370734+0000","last_clean_scrub_stamp":"2024-09-11T08:29:57.839509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:23.592176+0000","scrub_duration":0.0070850310000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":521,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.488510+0000","last_change":"2024-09-11T08:29:12.626802+0000","last_active":"2024-09-11T08:30:11.488510+0000","last_peered":"2024-09-11T08:30:11.488510+0000","last_clean":"2024-09-11T08:30:11.488510+0000","last_became_active":"2024-09-11T08:15:56.525120+0000","last_became_peered":"2024-09-11T08:15:56.525120+0000","last_unstale":"2024-09-11T08:30:11.488510+0000","last_undegraded":"2024-09-11T08:30:11.488510+0000","last_fullsized":"2024-09-11T08:30:11.488510+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:12.626731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:54.184372+0000","last_clean_scrub_stamp":"2024-09-11T08:29:12.626731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:34.316820+0000","scrub_duration":0.0092815109999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"405'25","reported_seq":246,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.595605+0000","last_change":"2024-09-11T08:30:05.609094+0000","last_active":"2024-09-11T08:30:11.595605+0000","last_peered":"2024-09-11T08:30:11.595605+0000","last_clean":"2024-09-11T08:30:11.595605+0000","last_became_active":"2024-09-11T08:24:28.353043+0000","last_became_peered":"2024-09-11T08:24:28.353043+0000","last_unstale":"2024-09-11T08:30:11.595605+0000","last_undegraded":"2024-09-11T08:30:11.595605+0000","last_fullsized":"2024-09-11T08:30:11.595605+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'25","last_scrub_stamp":"2024-09-11T08:30:05.609036+0000","last_deep_scrub":"313'19","last_deep_scrub_stamp":"2024-09-11T08:22:03.458402+0000","last_clean_scrub_stamp":"2024-09-11T08:30:05.609036+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:25.904857+0000","scrub_duration":0.0019394670000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":514,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.271234+0000","last_change":"2024-09-11T08:30:10.269699+0000","last_active":"2024-09-11T08:30:11.271234+0000","last_peered":"2024-09-11T08:30:11.271234+0000","last_clean":"2024-09-11T08:30:11.271234+0000","last_became_active":"2024-09-11T08:15:56.513278+0000","last_became_peered":"2024-09-11T08:15:56.513278+0000","last_unstale":"2024-09-11T08:30:11.271234+0000","last_undegraded":"2024-09-11T08:30:11.271234+0000","last_fullsized":"2024-09-11T08:30:11.271234+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:10.269641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:52.661591+0000","last_clean_scrub_stamp":"2024-09-11T08:30:10.269641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:31.914817+0000","scrub_duration":0.001664865,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"239'13","reported_seq":773,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:13.846358+0000","last_change":"2024-09-11T08:27:58.219794+0000","last_active":"2024-09-11T08:30:13.846358+0000","last_peered":"2024-09-11T08:30:13.846358+0000","last_clean":"2024-09-11T08:30:13.846358+0000","last_became_active":"2024-09-11T08:17:37.558224+0000","last_became_peered":"2024-09-11T08:17:37.558224+0000","last_unstale":"2024-09-11T08:30:13.846358+0000","last_undegraded":"2024-09-11T08:30:13.846358+0000","last_fullsized":"2024-09-11T08:30:13.846358+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'13","last_scrub_stamp":"2024-09-11T08:27:58.219700+0000","last_deep_scrub":"222'12","last_deep_scrub_stamp":"2024-09-11T08:16:52.592856+0000","last_clean_scrub_stamp":"2024-09-11T08:27:58.219700+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.01448953,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":436,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.488724+0000","last_change":"2024-09-11T08:29:58.653531+0000","last_active":"2024-09-11T08:30:11.488724+0000","last_peered":"2024-09-11T08:30:11.488724+0000","last_clean":"2024-09-11T08:30:11.488724+0000","last_became_active":"2024-09-11T08:24:28.357799+0000","last_became_peered":"2024-09-11T08:24:28.357799+0000","last_unstale":"2024-09-11T08:30:11.488724+0000","last_undegraded":"2024-09-11T08:30:11.488724+0000","last_fullsized":"2024-09-11T08:30:11.488724+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:58.653393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:53.175994+0000","last_clean_scrub_stamp":"2024-09-11T08:29:58.653393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:17.708980+0000","scrub_duration":0.0087291079999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"405'30","reported_seq":385,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.488484+0000","last_change":"2024-09-11T08:29:15.645126+0000","last_active":"2024-09-11T08:30:11.488484+0000","last_peered":"2024-09-11T08:30:11.488484+0000","last_clean":"2024-09-11T08:30:11.488484+0000","last_became_active":"2024-09-11T08:17:37.554357+0000","last_became_peered":"2024-09-11T08:17:37.554357+0000","last_unstale":"2024-09-11T08:30:11.488484+0000","last_undegraded":"2024-09-11T08:30:11.488484+0000","last_fullsized":"2024-09-11T08:30:11.488484+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'30","last_scrub_stamp":"2024-09-11T08:29:15.645039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:29:15.645039+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:21.270884+0000","scrub_duration":0.021832190000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":300,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.595464+0000","last_change":"2024-09-11T08:27:56.258234+0000","last_active":"2024-09-11T08:30:11.595464+0000","last_peered":"2024-09-11T08:30:11.595464+0000","last_clean":"2024-09-11T08:30:11.595464+0000","last_became_active":"2024-09-11T08:15:56.518515+0000","last_became_peered":"2024-09-11T08:15:56.518515+0000","last_unstale":"2024-09-11T08:30:11.595464+0000","last_undegraded":"2024-09-11T08:30:11.595464+0000","last_fullsized":"2024-09-11T08:30:11.595464+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:56.258011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:23:42.199527+0000","last_clean_scrub_stamp":"2024-09-11T08:27:56.258011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022527070000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"354'13","reported_seq":308,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.902266+0000","last_change":"2024-09-11T08:30:14.950155+0000","last_active":"2024-09-11T08:30:16.902266+0000","last_peered":"2024-09-11T08:30:16.902266+0000","last_clean":"2024-09-11T08:30:16.902266+0000","last_became_active":"2024-09-11T08:24:28.353115+0000","last_became_peered":"2024-09-11T08:24:28.353115+0000","last_unstale":"2024-09-11T08:30:16.902266+0000","last_undegraded":"2024-09-11T08:30:16.902266+0000","last_fullsized":"2024-09-11T08:30:16.902266+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'13","last_scrub_stamp":"2024-09-11T08:30:14.950098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:30:14.950098+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:32.016065+0000","scrub_duration":0.0018318049999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":498,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.270654+0000","last_change":"2024-09-11T08:29:05.959570+0000","last_active":"2024-09-11T08:30:11.270654+0000","last_peered":"2024-09-11T08:30:11.270654+0000","last_clean":"2024-09-11T08:30:11.270654+0000","last_became_active":"2024-09-11T08:15:56.513949+0000","last_became_peered":"2024-09-11T08:15:56.513949+0000","last_unstale":"2024-09-11T08:30:11.270654+0000","last_undegraded":"2024-09-11T08:30:11.270654+0000","last_fullsized":"2024-09-11T08:30:11.270654+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:05.957912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:29:05.957912+0000","last_clean_scrub_stamp":"2024-09-11T08:29:05.957912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:27.228578+0000","scrub_duration":0.0062029260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"405'27","reported_seq":421,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.305480+0000","last_change":"2024-09-11T08:30:11.305412+0000","last_active":"2024-09-11T08:30:11.305480+0000","last_peered":"2024-09-11T08:30:11.305480+0000","last_clean":"2024-09-11T08:30:11.305480+0000","last_became_active":"2024-09-11T08:23:22.688993+0000","last_became_peered":"2024-09-11T08:23:22.688993+0000","last_unstale":"2024-09-11T08:30:11.305480+0000","last_undegraded":"2024-09-11T08:30:11.305480+0000","last_fullsized":"2024-09-11T08:30:11.305480+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'27","last_scrub_stamp":"2024-09-11T08:30:11.305355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:30:11.305355+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:19.774283+0000","scrub_duration":0.0026390200000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":349,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.595578+0000","last_change":"2024-09-11T08:29:12.448974+0000","last_active":"2024-09-11T08:30:11.595578+0000","last_peered":"2024-09-11T08:30:11.595578+0000","last_clean":"2024-09-11T08:30:11.595578+0000","last_became_active":"2024-09-11T08:23:22.682569+0000","last_became_peered":"2024-09-11T08:23:22.682569+0000","last_unstale":"2024-09-11T08:30:11.595578+0000","last_undegraded":"2024-09-11T08:30:11.595578+0000","last_fullsized":"2024-09-11T08:30:11.595578+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:12.448898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:29:12.448898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:21.712733+0000","scrub_duration":0.0016737830000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":523,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:10.835327+0000","last_change":"2024-09-11T08:28:47.755273+0000","last_active":"2024-09-11T08:30:10.835327+0000","last_peered":"2024-09-11T08:30:10.835327+0000","last_clean":"2024-09-11T08:30:10.835327+0000","last_became_active":"2024-09-11T08:24:28.355004+0000","last_became_peered":"2024-09-11T08:24:28.355004+0000","last_unstale":"2024-09-11T08:30:10.835327+0000","last_undegraded":"2024-09-11T08:30:10.835327+0000","last_fullsized":"2024-09-11T08:30:10.835327+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:28:47.755203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:28:47.755203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001837315,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":312,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.270533+0000","last_change":"2024-09-11T08:29:23.241188+0000","last_active":"2024-09-11T08:30:11.270533+0000","last_peered":"2024-09-11T08:30:11.270533+0000","last_clean":"2024-09-11T08:30:11.270533+0000","last_became_active":"2024-09-11T08:24:28.372049+0000","last_became_peered":"2024-09-11T08:24:28.372049+0000","last_unstale":"2024-09-11T08:30:11.270533+0000","last_undegraded":"2024-09-11T08:30:11.270533+0000","last_fullsized":"2024-09-11T08:30:11.270533+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:23.241016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:29:23.241016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:40.844841+0000","scrub_duration":0.0063927849999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"405'25","reported_seq":543,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:10.835508+0000","last_change":"2024-09-11T08:29:20.881222+0000","last_active":"2024-09-11T08:30:10.835508+0000","last_peered":"2024-09-11T08:30:10.835508+0000","last_clean":"2024-09-11T08:30:10.835508+0000","last_became_active":"2024-09-11T08:07:32.049368+0000","last_became_peered":"2024-09-11T08:07:32.049368+0000","last_unstale":"2024-09-11T08:30:10.835508+0000","last_undegraded":"2024-09-11T08:30:10.835508+0000","last_fullsized":"2024-09-11T08:30:10.835508+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'25","last_scrub_stamp":"2024-09-11T08:29:20.881147+0000","last_deep_scrub":"343'19","last_deep_scrub_stamp":"2024-09-11T08:23:44.640077+0000","last_clean_scrub_stamp":"2024-09-11T08:29:20.881147+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:42.708414+0000","scrub_duration":0.0058682450000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":300,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.595628+0000","last_change":"2024-09-11T08:28:05.289814+0000","last_active":"2024-09-11T08:30:11.595628+0000","last_peered":"2024-09-11T08:30:11.595628+0000","last_clean":"2024-09-11T08:30:11.595628+0000","last_became_active":"2024-09-11T08:15:56.521570+0000","last_became_peered":"2024-09-11T08:15:56.521570+0000","last_unstale":"2024-09-11T08:30:11.595628+0000","last_undegraded":"2024-09-11T08:30:11.595628+0000","last_fullsized":"2024-09-11T08:30:11.595628+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:28:05.289763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:28:05.289763+0000","last_clean_scrub_stamp":"2024-09-11T08:28:05.289763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037268660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":275,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.270688+0000","last_change":"2024-09-11T08:29:58.282988+0000","last_active":"2024-09-11T08:30:11.270688+0000","last_peered":"2024-09-11T08:30:11.270688+0000","last_clean":"2024-09-11T08:30:11.270688+0000","last_became_active":"2024-09-11T08:17:37.557645+0000","last_became_peered":"2024-09-11T08:17:37.557645+0000","last_unstale":"2024-09-11T08:30:11.270688+0000","last_undegraded":"2024-09-11T08:30:11.270688+0000","last_fullsized":"2024-09-11T08:30:11.270688+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:58.282759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:22:02.568761+0000","last_clean_scrub_stamp":"2024-09-11T08:29:58.282759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:27.885348+0000","scrub_duration":0.0099211319999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":511,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.270412+0000","last_change":"2024-09-11T08:30:02.281204+0000","last_active":"2024-09-11T08:30:11.270412+0000","last_peered":"2024-09-11T08:30:11.270412+0000","last_clean":"2024-09-11T08:30:11.270412+0000","last_became_active":"2024-09-11T08:24:28.363959+0000","last_became_peered":"2024-09-11T08:24:28.363959+0000","last_unstale":"2024-09-11T08:30:11.270412+0000","last_undegraded":"2024-09-11T08:30:11.270412+0000","last_fullsized":"2024-09-11T08:30:11.270412+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:02.281112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:09.973943+0000","last_clean_scrub_stamp":"2024-09-11T08:30:02.281112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:13.828016+0000","scrub_duration":0.018012968000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"405'27","reported_seq":590,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:10.836323+0000","last_change":"2024-09-11T08:28:55.841566+0000","last_active":"2024-09-11T08:30:10.836323+0000","last_peered":"2024-09-11T08:30:10.836323+0000","last_clean":"2024-09-11T08:30:10.836323+0000","last_became_active":"2024-09-11T08:15:56.520058+0000","last_became_peered":"2024-09-11T08:15:56.520058+0000","last_unstale":"2024-09-11T08:30:10.836323+0000","last_undegraded":"2024-09-11T08:30:10.836323+0000","last_fullsized":"2024-09-11T08:30:10.836323+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'27","last_scrub_stamp":"2024-09-11T08:28:55.841457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:28:55.841457+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012026871999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":403,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.595722+0000","last_change":"2024-09-11T08:29:00.479990+0000","last_active":"2024-09-11T08:30:11.595722+0000","last_peered":"2024-09-11T08:30:11.595722+0000","last_clean":"2024-09-11T08:30:11.595722+0000","last_became_active":"2024-09-11T08:24:28.356974+0000","last_became_peered":"2024-09-11T08:24:28.356974+0000","last_unstale":"2024-09-11T08:30:11.595722+0000","last_undegraded":"2024-09-11T08:30:11.595722+0000","last_fullsized":"2024-09-11T08:30:11.595722+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:00.479835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:29:00.479835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002146532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":310,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.595961+0000","last_change":"2024-09-11T08:29:06.470596+0000","last_active":"2024-09-11T08:30:11.595961+0000","last_peered":"2024-09-11T08:30:11.595961+0000","last_clean":"2024-09-11T08:30:11.595961+0000","last_became_active":"2024-09-11T08:15:56.523728+0000","last_became_peered":"2024-09-11T08:15:56.523728+0000","last_unstale":"2024-09-11T08:30:11.595961+0000","last_undegraded":"2024-09-11T08:30:11.595961+0000","last_fullsized":"2024-09-11T08:30:11.595961+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:06.470537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:11:34.159053+0000","last_clean_scrub_stamp":"2024-09-11T08:29:06.470537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014249079999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":492,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:10.837331+0000","last_change":"2024-09-11T08:29:56.882574+0000","last_active":"2024-09-11T08:30:10.837331+0000","last_peered":"2024-09-11T08:30:10.837331+0000","last_clean":"2024-09-11T08:30:10.837331+0000","last_became_active":"2024-09-11T08:07:36.633368+0000","last_became_peered":"2024-09-11T08:07:36.633368+0000","last_unstale":"2024-09-11T08:30:10.837331+0000","last_undegraded":"2024-09-11T08:30:10.837331+0000","last_fullsized":"2024-09-11T08:30:10.837331+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:56.882497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:22:50.992034+0000","last_clean_scrub_stamp":"2024-09-11T08:29:56.882497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:13.644291+0000","scrub_duration":0.0057421800000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"405'14","reported_seq":506,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:10.835557+0000","last_change":"2024-09-11T08:28:56.806080+0000","last_active":"2024-09-11T08:30:10.835557+0000","last_peered":"2024-09-11T08:30:10.835557+0000","last_clean":"2024-09-11T08:30:10.835557+0000","last_became_active":"2024-09-11T08:07:32.045728+0000","last_became_peered":"2024-09-11T08:07:32.045728+0000","last_unstale":"2024-09-11T08:30:10.835557+0000","last_undegraded":"2024-09-11T08:30:10.835557+0000","last_fullsized":"2024-09-11T08:30:10.835557+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'14","last_scrub_stamp":"2024-09-11T08:28:56.805624+0000","last_deep_scrub":"312'10","last_deep_scrub_stamp":"2024-09-11T08:21:55.859866+0000","last_clean_scrub_stamp":"2024-09-11T08:28:56.805624+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:26.046082+0000","scrub_duration":0.019371645,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":524,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.271058+0000","last_change":"2024-09-11T08:30:11.270503+0000","last_active":"2024-09-11T08:30:11.271058+0000","last_peered":"2024-09-11T08:30:11.271058+0000","last_clean":"2024-09-11T08:30:11.271058+0000","last_became_active":"2024-09-11T08:24:28.357978+0000","last_became_peered":"2024-09-11T08:24:28.357978+0000","last_unstale":"2024-09-11T08:30:11.271058+0000","last_undegraded":"2024-09-11T08:30:11.271058+0000","last_fullsized":"2024-09-11T08:30:11.271058+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:11.270399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:30:11.270399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:30.279105+0000","scrub_duration":0.001777708,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":550,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.304509+0000","last_change":"2024-09-11T08:30:03.423705+0000","last_active":"2024-09-11T08:30:11.304509+0000","last_peered":"2024-09-11T08:30:11.304509+0000","last_clean":"2024-09-11T08:30:11.304509+0000","last_became_active":"2024-09-11T08:15:56.528607+0000","last_became_peered":"2024-09-11T08:15:56.528607+0000","last_unstale":"2024-09-11T08:30:11.304509+0000","last_undegraded":"2024-09-11T08:30:11.304509+0000","last_fullsized":"2024-09-11T08:30:11.304509+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:03.423649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:30:03.423649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:13.967310+0000","scrub_duration":0.001131574,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":303,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.681484+0000","last_change":"2024-09-11T08:30:16.681411+0000","last_active":"2024-09-11T08:30:16.681484+0000","last_peered":"2024-09-11T08:30:16.681484+0000","last_clean":"2024-09-11T08:30:16.681484+0000","last_became_active":"2024-09-11T08:15:56.521488+0000","last_became_peered":"2024-09-11T08:15:56.521488+0000","last_unstale":"2024-09-11T08:30:16.681484+0000","last_undegraded":"2024-09-11T08:30:16.681484+0000","last_fullsized":"2024-09-11T08:30:16.681484+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:16.681355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:20:18.827709+0000","last_clean_scrub_stamp":"2024-09-11T08:30:16.681355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:46.492811+0000","scrub_duration":0.0012763220000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":521,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:10.835546+0000","last_change":"2024-09-11T08:27:17.179442+0000","last_active":"2024-09-11T08:30:10.835546+0000","last_peered":"2024-09-11T08:30:10.835546+0000","last_clean":"2024-09-11T08:30:10.835546+0000","last_became_active":"2024-09-11T08:24:28.354141+0000","last_became_peered":"2024-09-11T08:24:28.354141+0000","last_unstale":"2024-09-11T08:30:10.835546+0000","last_undegraded":"2024-09-11T08:30:10.835546+0000","last_fullsized":"2024-09-11T08:30:10.835546+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:17.179345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:50.585566+0000","last_clean_scrub_stamp":"2024-09-11T08:27:17.179345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0077697699999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":262,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.270739+0000","last_change":"2024-09-11T08:27:51.916658+0000","last_active":"2024-09-11T08:30:11.270739+0000","last_peered":"2024-09-11T08:30:11.270739+0000","last_clean":"2024-09-11T08:30:11.270739+0000","last_became_active":"2024-09-11T08:17:37.554262+0000","last_became_peered":"2024-09-11T08:17:37.554262+0000","last_unstale":"2024-09-11T08:30:11.270739+0000","last_undegraded":"2024-09-11T08:30:11.270739+0000","last_fullsized":"2024-09-11T08:30:11.270739+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:51.916565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:27:51.916565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012562804,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"405'32","reported_seq":614,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.305088+0000","last_change":"2024-09-11T08:29:07.366953+0000","last_active":"2024-09-11T08:30:11.305088+0000","last_peered":"2024-09-11T08:30:11.305088+0000","last_clean":"2024-09-11T08:30:11.305088+0000","last_became_active":"2024-09-11T08:24:28.360937+0000","last_became_peered":"2024-09-11T08:24:28.360937+0000","last_unstale":"2024-09-11T08:30:11.305088+0000","last_undegraded":"2024-09-11T08:30:11.305088+0000","last_fullsized":"2024-09-11T08:30:11.305088+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'32","last_scrub_stamp":"2024-09-11T08:29:07.366863+0000","last_deep_scrub":"405'32","last_deep_scrub_stamp":"2024-09-11T08:29:07.366863+0000","last_clean_scrub_stamp":"2024-09-11T08:29:07.366863+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:20.629764+0000","scrub_duration":0.0072573189999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":338,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-11T08:30:11.304734+0000","last_change":"2024-09-11T08:29:58.627283+0000","last_active":"2024-09-11T08:30:11.304734+0000","last_peered":"2024-09-11T08:30:11.304734+0000","last_clean":"2024-09-11T08:30:11.304734+0000","last_became_active":"2024-09-11T08:15:08.769133+0000","last_became_peered":"2024-09-11T08:15:08.769133+0000","last_unstale":"2024-09-11T08:30:11.304734+0000","last_undegraded":"2024-09-11T08:30:11.304734+0000","last_fullsized":"2024-09-11T08:30:11.304734+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:58.627201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:18:39.206016+0000","last_clean_scrub_stamp":"2024-09-11T08:29:58.627201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:59.524575+0000","scrub_duration":0.007062512,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102,"ondisk_log_size":102,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1546,"num_read_kb":1303,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":4566,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3425,"internal_metadata":0},"log_size":1066,"ondisk_log_size":1066,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":958,"num_read_kb":953,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":168,"seq":721554505920,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":920676,"kb_used_data":1304,"kb_used_omap":7,"kb_used_meta":919352,"kb_avail":92824476,"statfs":{"total":95995035648,"available":95052263424,"internally_reserved":0,"allocated":1335296,"data_stored":965602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8027,"internal_metadata":941416613},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:29:49 2024","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":1.1060000000000001,"15min":2.419},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":2.2839999999999998,"5min":19.797999999999998,"15min":117.57599999999999},"last":0.22900000000000001},{"interface":"front","average":{"1min":1.3009999999999999,"5min":1.1379999999999999,"15min":2.2999999999999998},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":4.3070000000000004,"5min":19.899000000000001,"15min":117.542},"last":0.38300000000000001}]},{"osd":1,"last update":"Wed Sep 11 08:29:49 2024","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.2569999999999999,"15min":1.8799999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.20899999999999999},"max":{"1min":4.7220000000000004,"5min":24.347999999999999,"15min":66.971000000000004},"last":0.32000000000000001},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.2150000000000001,"15min":1.702},"min":{"1min":0.246,"5min":0.23400000000000001,"15min":0.23000000000000001},"max":{"1min":3.8069999999999999,"5min":25.382000000000001,"15min":67.185000000000002},"last":0.30299999999999999}]},{"osd":2,"last update":"Wed Sep 11 08:29:49 2024","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":0.97099999999999997,"15min":1.04},"min":{"1min":0.25600000000000001,"5min":0.245,"15min":0.22700000000000001},"max":{"1min":9.3439999999999994,"5min":9.3439999999999994,"15min":19.097000000000001},"last":0.248},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.329,"15min":1.202},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":3.3199999999999998,"5min":7.7699999999999996,"15min":21.594000000000001},"last":0.39500000000000002}]},{"osd":3,"last update":"Wed Sep 11 08:29:49 2024","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.347,"15min":1.23},"min":{"1min":0.28399999999999997,"5min":0.27200000000000002,"15min":0.251},"max":{"1min":3.8300000000000001,"5min":24.315000000000001,"15min":24.315000000000001},"last":0.41899999999999998},{"interface":"front","average":{"1min":1.0880000000000001,"5min":1.147,"15min":1.206},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.251},"max":{"1min":3.516,"5min":24.864999999999998,"15min":24.864999999999998},"last":0.33600000000000002}]},{"osd":4,"last update":"Wed Sep 11 08:29:49 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.73999999999999999,"15min":0.83599999999999997},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":1.3500000000000001,"5min":5.7240000000000002,"15min":5.7240000000000002},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.66000000000000003,"15min":0.755},"min":{"1min":0.27900000000000003,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":2.367,"5min":4.2060000000000004,"15min":5.2649999999999997},"last":0.34699999999999998}]},{"osd":5,"last update":"Wed Sep 11 08:29:44 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.84899999999999998,"15min":1.2290000000000001},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":3.4470000000000001,"5min":6.3979999999999997,"15min":6.3979999999999997},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.68899999999999995,"15min":0.86799999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":2.5960000000000001,"5min":2.887,"15min":2.887},"last":0.371}]},{"osd":6,"last update":"Wed Sep 11 08:29:49 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":1.3340000000000001,"15min":1.0009999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.248},"max":{"1min":2.2010000000000001,"5min":42.749000000000002,"15min":42.749000000000002},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.57599999999999996,"5min":1.4139999999999999,"15min":1.093},"min":{"1min":0.32300000000000001,"5min":0.311,"15min":0.25700000000000001},"max":{"1min":1.5900000000000001,"5min":41.914000000000001,"15min":41.914000000000001},"last":0.36099999999999999}]}]},{"osd":6,"up_from":45,"seq":193273528603,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":896788,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":895486,"kb_avail":92848364,"statfs":{"total":95995035648,"available":95076724736,"internally_reserved":0,"allocated":1318912,"data_stored":965425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":916978008},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:29:34 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":4.7800000000000002,"15min":3.1280000000000001},"min":{"1min":0.28999999999999998,"5min":0.21199999999999999,"15min":0.17999999999999999},"max":{"1min":3.649,"5min":331.14800000000002,"15min":331.14800000000002},"last":1.042},{"interface":"front","average":{"1min":1.1639999999999999,"5min":4.4930000000000003,"15min":2.98},"min":{"1min":0.30199999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":3.0939999999999999,"5min":331.31,"15min":331.31},"last":1.478}]},{"osd":1,"last update":"Wed Sep 11 08:29:34 2024","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.5880000000000001,"15min":1.3859999999999999},"min":{"1min":0.35799999999999998,"5min":0.22900000000000001,"15min":0.214},"max":{"1min":5.0350000000000001,"5min":29.423999999999999,"15min":29.423999999999999},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.5509999999999999,"5min":1.607,"15min":1.345},"min":{"1min":0.23499999999999999,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":4.4640000000000004,"5min":29.109000000000002,"15min":29.109000000000002},"last":0.93999999999999995}]},{"osd":2,"last update":"Wed Sep 11 08:29:34 2024","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.6439999999999999,"15min":1.3129999999999999},"min":{"1min":0.25,"5min":0.25,"15min":0.23200000000000001},"max":{"1min":3.218,"5min":28.167999999999999,"15min":39.652999999999999},"last":0.96499999999999997},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.587,"15min":1.415},"min":{"1min":0.35199999999999998,"5min":0.251,"15min":0.251},"max":{"1min":4.0099999999999998,"5min":31.648,"15min":38.390999999999998},"last":1.526}]},{"osd":3,"last update":"Wed Sep 11 08:29:34 2024","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":1.5660000000000001,"15min":1.25},"min":{"1min":0.30599999999999999,"5min":0.253,"15min":0.22700000000000001},"max":{"1min":1.774,"5min":12.414,"15min":12.414},"last":1.0640000000000001},{"interface":"front","average":{"1min":0.95499999999999996,"5min":1.522,"15min":1.24},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":2.133,"5min":28.004999999999999,"15min":28.004999999999999},"last":1.01}]},{"osd":4,"last update":"Wed Sep 11 08:29:34 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.90000000000000002,"15min":0.77500000000000002},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.20499999999999999},"max":{"1min":3.028,"5min":9.7560000000000002,"15min":9.7560000000000002},"last":1.504},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.79900000000000004,"15min":0.74399999999999999},"min":{"1min":0.307,"5min":0.29099999999999998,"15min":0.23699999999999999},"max":{"1min":3.5659999999999998,"5min":3.5659999999999998,"15min":4.468},"last":1.5449999999999999}]},{"osd":5,"last update":"Wed Sep 11 08:29:34 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.97699999999999998,"15min":1.552},"min":{"1min":0.29499999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":3.1739999999999999,"5min":9.7989999999999995,"15min":9.7989999999999995},"last":1.5629999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.99199999999999999,"15min":1.3939999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":2.1019999999999999,"5min":7.2999999999999998,"15min":7.2999999999999998},"last":0.98899999999999999}]},{"osd":7,"last update":"Wed Sep 11 08:29:46 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.2090000000000001,"15min":1.014},"min":{"1min":0.34599999999999997,"5min":0.25900000000000001,"15min":0.253},"max":{"1min":3.3889999999999998,"5min":7.5529999999999999,"15min":10.167999999999999},"last":1.1040000000000001},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.96099999999999997,"15min":0.82199999999999995},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.25700000000000001},"max":{"1min":2.887,"5min":3.6160000000000001,"15min":5.3579999999999997},"last":0.90900000000000003}]}]},{"osd":5,"up_from":353,"seq":1516123455560,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":360656,"kb_used_data":712,"kb_used_omap":4,"kb_used_meta":359931,"kb_avail":93384496,"statfs":{"total":95995035648,"available":95625723904,"internally_reserved":0,"allocated":729088,"data_stored":374618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":368569862},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:29:39 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.99299999999999999,"15min":1.6000000000000001},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":2.9710000000000001,"5min":19.379999999999999,"15min":19.379999999999999},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.96799999999999997,"5min":1.135,"15min":1.623},"min":{"1min":0.23599999999999999,"5min":0.218,"15min":0.218},"max":{"1min":2.6739999999999999,"5min":18.238,"15min":18.238},"last":0.40100000000000002}]},{"osd":1,"last update":"Wed Sep 11 08:29:39 2024","interfaces":[{"interface":"back","average":{"1min":4.8869999999999996,"5min":2.7749999999999999,"15min":1.6739999999999999},"min":{"1min":0.24199999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":85.728999999999999,"5min":85.728999999999999,"15min":85.728999999999999},"last":0.432},{"interface":"front","average":{"1min":5.0979999999999999,"5min":2.8319999999999999,"15min":1.4159999999999999},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":85.668000000000006,"5min":85.668000000000006,"15min":85.668000000000006},"last":0.32600000000000001}]},{"osd":2,"last update":"Wed Sep 11 08:29:39 2024","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":1.4099999999999999,"15min":1.8799999999999999},"min":{"1min":0.26000000000000001,"5min":0.251,"15min":0.251},"max":{"1min":3.7599999999999998,"5min":18.350999999999999,"15min":18.350999999999999},"last":0.38400000000000001},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.3200000000000001,"15min":1.74},"min":{"1min":0.27400000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":9.6489999999999991,"5min":18.381,"15min":18.381},"last":0.36799999999999999}]},{"osd":3,"last update":"Wed Sep 11 08:29:39 2024","interfaces":[{"interface":"back","average":{"1min":4.4080000000000004,"5min":2.3580000000000001,"15min":1.5680000000000001},"min":{"1min":0.27500000000000002,"5min":0.251,"15min":0.251},"max":{"1min":73.697000000000003,"5min":73.697000000000003,"15min":73.697000000000003},"last":0.36099999999999999},{"interface":"front","average":{"1min":4.2370000000000001,"5min":2.1309999999999998,"15min":1.28},"min":{"1min":0.35399999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":73.727999999999994,"5min":73.727999999999994,"15min":73.727999999999994},"last":0.40999999999999998}]},{"osd":4,"last update":"Wed Sep 11 08:29:39 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.75900000000000001,"15min":1.083},"min":{"1min":0.27000000000000002,"5min":0.249,"15min":0.249},"max":{"1min":2.1230000000000002,"5min":5.8630000000000004,"15min":5.8630000000000004},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.72999999999999998,"15min":1.1160000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":1.8700000000000001,"5min":5.8979999999999997,"15min":5.8979999999999997},"last":0.30099999999999999}]},{"osd":6,"last update":"Wed Sep 11 08:29:39 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.83799999999999997,"15min":1.3220000000000001},"min":{"1min":0.317,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":1.921,"5min":5.9779999999999998,"15min":5.9779999999999998},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.84099999999999997,"15min":1.5669999999999999},"min":{"1min":0.32300000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.9990000000000001,"5min":5.6680000000000001,"15min":5.6680000000000001},"last":0.27300000000000002}]},{"osd":7,"last update":"Wed Sep 11 08:29:39 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.92800000000000005,"15min":1.153},"min":{"1min":0.29299999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":3.2320000000000002,"5min":7.8719999999999999,"15min":7.8719999999999999},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.94999999999999996,"15min":1.544},"min":{"1min":0.29899999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":3.2759999999999998,"5min":6.8739999999999997,"15min":6.8739999999999997},"last":0.33300000000000002}]}]},{"osd":4,"up_from":31,"seq":133143986469,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":894824,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":894078,"kb_avail":92850328,"statfs":{"total":95995035648,"available":95078735872,"internally_reserved":0,"allocated":749568,"data_stored":375932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":915536216},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:29:45 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":2.6219999999999999,"15min":1.948},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":2.3159999999999998,"5min":64.329999999999998,"15min":64.329999999999998},"last":0.496},{"interface":"front","average":{"1min":0.68200000000000005,"5min":2.4470000000000001,"15min":1.8759999999999999},"min":{"1min":0.24199999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.3900000000000001,"5min":66.819999999999993,"15min":66.819999999999993},"last":0.54500000000000004}]},{"osd":1,"last update":"Wed Sep 11 08:29:45 2024","interfaces":[{"interface":"back","average":{"1min":7.2530000000000001,"5min":2.2719999999999998,"15min":1.6619999999999999},"min":{"1min":0.27600000000000002,"5min":0.22,"15min":0.214},"max":{"1min":91.852000000000004,"5min":91.852000000000004,"15min":91.852000000000004},"last":0.63500000000000001},{"interface":"front","average":{"1min":6.7080000000000002,"5min":2.4209999999999998,"15min":1.7470000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":91.697000000000003,"5min":91.697000000000003,"15min":91.697000000000003},"last":0.57899999999999996}]},{"osd":2,"last update":"Wed Sep 11 08:29:45 2024","interfaces":[{"interface":"back","average":{"1min":1.819,"5min":2.1499999999999999,"15min":1.4910000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":13.710000000000001,"5min":54.317999999999998,"15min":54.317999999999998},"last":0.51200000000000001},{"interface":"front","average":{"1min":2.2469999999999999,"5min":2.2839999999999998,"15min":1.611},"min":{"1min":0.30299999999999999,"5min":0.246,"15min":0.23400000000000001},"max":{"1min":12.711,"5min":54.430999999999997,"15min":54.430999999999997},"last":0.56200000000000006}]},{"osd":3,"last update":"Wed Sep 11 08:29:45 2024","interfaces":[{"interface":"back","average":{"1min":1.8819999999999999,"5min":1.355,"15min":1.381},"min":{"1min":0.27700000000000002,"5min":0.247,"15min":0.247},"max":{"1min":12.811999999999999,"5min":12.811999999999999,"15min":30.321000000000002},"last":0.626},{"interface":"front","average":{"1min":2.1230000000000002,"5min":1.3360000000000001,"15min":1.2170000000000001},"min":{"1min":0.26700000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":12.978,"5min":12.978,"15min":30.620000000000001},"last":0.504}]},{"osd":5,"last update":"Wed Sep 11 08:29:34 2024","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":0.80000000000000004,"15min":1.1739999999999999},"min":{"1min":0.25700000000000001,"5min":0.246,"15min":0.246},"max":{"1min":4.3410000000000002,"5min":5.1050000000000004,"15min":5.1050000000000004},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.84899999999999998,"15min":1.302},"min":{"1min":0.23799999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":1.3200000000000001,"5min":6.1150000000000002,"15min":6.1150000000000002},"last":0.40899999999999997}]},{"osd":6,"last update":"Wed Sep 11 08:29:34 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.90400000000000003,"15min":0.89800000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.23200000000000001},"max":{"1min":3.036,"5min":6.4039999999999999,"15min":48.808},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.96199999999999997,"15min":0.95399999999999996},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.251},"max":{"1min":2.415,"5min":5.3159999999999998,"15min":48.746000000000002},"last":0.39000000000000001}]},{"osd":7,"last update":"Wed Sep 11 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.78500000000000003,"15min":0.73199999999999998},"min":{"1min":0.27500000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.96599999999999997,"5min":6.093,"15min":6.093},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.498,"5min":0.63100000000000001,"15min":0.70999999999999996},"min":{"1min":0.30099999999999999,"5min":0.29899999999999999,"15min":0.26800000000000002},"max":{"1min":1.1950000000000001,"5min":4.8259999999999996,"15min":4.8259999999999996},"last":0.35499999999999998}]}]},{"osd":3,"up_from":25,"seq":107374182698,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":901940,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":901182,"kb_avail":92843212,"statfs":{"total":95995035648,"available":95071449088,"internally_reserved":0,"allocated":761856,"data_stored":375403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":922810712},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:29:38 2024","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":1.702,"15min":1.8500000000000001},"min":{"1min":0.25900000000000001,"5min":0.16300000000000001,"15min":0.14999999999999999},"max":{"1min":4.9139999999999997,"5min":24.616,"15min":60.469999999999999},"last":0.48999999999999999},{"interface":"front","average":{"1min":1.7450000000000001,"5min":1.7,"15min":1.732},"min":{"1min":0.24099999999999999,"5min":0.20599999999999999,"15min":0.17399999999999999},"max":{"1min":6.7089999999999996,"5min":23.640000000000001,"15min":60.488999999999997},"last":0.41699999999999998}]},{"osd":1,"last update":"Wed Sep 11 08:29:38 2024","interfaces":[{"interface":"back","average":{"1min":2.2050000000000001,"5min":1.4550000000000001,"15min":1.6459999999999999},"min":{"1min":0.28999999999999998,"5min":0.20399999999999999,"15min":0.19600000000000001},"max":{"1min":7.5739999999999998,"5min":9.5619999999999994,"15min":60.578000000000003},"last":0.47399999999999998},{"interface":"front","average":{"1min":1.5680000000000001,"5min":1.361,"15min":1.6859999999999999},"min":{"1min":0.246,"5min":0.20899999999999999,"15min":0.19500000000000001},"max":{"1min":7.1399999999999997,"5min":7.1399999999999997,"15min":63.674999999999997},"last":0.50800000000000001}]},{"osd":2,"last update":"Wed Sep 11 08:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":2.0430000000000001,"15min":1.675},"min":{"1min":0.28000000000000003,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":2.1899999999999999,"5min":67.503,"15min":67.503},"last":0.54500000000000004},{"interface":"front","average":{"1min":1.5980000000000001,"5min":2.3140000000000001,"15min":1.768},"min":{"1min":0.26500000000000001,"5min":0.251,"15min":0.19900000000000001},"max":{"1min":7.1020000000000003,"5min":67.313999999999993,"15min":67.313999999999993},"last":0.437}]},{"osd":4,"last update":"Wed Sep 11 08:29:47 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":1.196,"15min":1.4039999999999999},"min":{"1min":0.29199999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":2.2650000000000001,"5min":6.8680000000000003,"15min":61.290999999999997},"last":0.59699999999999998},{"interface":"front","average":{"1min":1.423,"5min":1.2470000000000001,"15min":1.28},"min":{"1min":0.33500000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":4.8179999999999996,"5min":6.46,"15min":60.856000000000002},"last":0.65400000000000003}]},{"osd":5,"last update":"Wed Sep 11 08:29:38 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.306,"15min":1.665},"min":{"1min":0.34899999999999998,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":6.6189999999999998,"5min":7.3849999999999998,"15min":7.3849999999999998},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.97799999999999998,"5min":1.1759999999999999,"15min":1.22},"min":{"1min":0.39000000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":3.5049999999999999,"5min":6.835,"15min":6.835},"last":0.74299999999999999}]},{"osd":6,"last update":"Wed Sep 11 08:29:38 2024","interfaces":[{"interface":"back","average":{"1min":1.6759999999999999,"5min":1.4910000000000001,"15min":1.649},"min":{"1min":0.39300000000000002,"5min":0.32700000000000001,"15min":0.29699999999999999},"max":{"1min":6.6520000000000001,"5min":6.6520000000000001,"15min":60.817},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.94699999999999995,"5min":1.2050000000000001,"15min":1.669},"min":{"1min":0.314,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":2.2909999999999999,"5min":6.8970000000000002,"15min":61.264000000000003},"last":0.71099999999999997}]},{"osd":7,"last update":"Wed Sep 11 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":1.7010000000000001,"5min":1.321,"15min":1.677},"min":{"1min":0.441,"5min":0.33500000000000002,"15min":0.29899999999999999},"max":{"1min":6.6840000000000002,"5min":6.6840000000000002,"15min":60.847000000000001},"last":0.76900000000000002},{"interface":"front","average":{"1min":1.641,"5min":1.411,"15min":1.8240000000000001},"min":{"1min":0.36699999999999999,"5min":0.35899999999999999,"15min":0.33800000000000002},"max":{"1min":6.899,"5min":6.899,"15min":60.883000000000003},"last":0.70199999999999996}]}]},{"osd":2,"up_from":18,"seq":77309411629,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":889876,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":889150,"kb_avail":92855276,"statfs":{"total":95995035648,"available":95083802624,"internally_reserved":0,"allocated":729088,"data_stored":374530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":910489944},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:30:10 2024","interfaces":[{"interface":"back","average":{"1min":7.7149999999999999,"5min":3.585,"15min":3.1840000000000002},"min":{"1min":0.27500000000000002,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":102.18000000000001,"5min":102.18000000000001,"15min":102.18000000000001},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.91900000000000004,"5min":2.0739999999999998,"15min":2.9350000000000001},"min":{"1min":0.29099999999999998,"5min":0.20599999999999999,"15min":0.17999999999999999},"max":{"1min":4.2510000000000003,"5min":81.391000000000005,"15min":93.790999999999997},"last":0.36499999999999999}]},{"osd":1,"last update":"Wed Sep 11 08:30:10 2024","interfaces":[{"interface":"back","average":{"1min":7.907,"5min":2.7069999999999999,"15min":2.0819999999999999},"min":{"1min":0.20399999999999999,"5min":0.19800000000000001,"15min":0.182},"max":{"1min":102.134,"5min":102.134,"15min":102.134},"last":0.29399999999999998},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.206,"15min":1.577},"min":{"1min":0.30499999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":4.1799999999999997,"5min":6.7220000000000004,"15min":53.082999999999998},"last":0.49299999999999999}]},{"osd":3,"last update":"Wed Sep 11 08:29:34 2024","interfaces":[{"interface":"back","average":{"1min":1.97,"5min":2.169,"15min":1.7350000000000001},"min":{"1min":0.41299999999999998,"5min":0.22,"15min":0.21299999999999999},"max":{"1min":6.2549999999999999,"5min":54.289999999999999,"15min":54.289999999999999},"last":0.317},{"interface":"front","average":{"1min":1.4039999999999999,"5min":2.0699999999999998,"15min":1.6679999999999999},"min":{"1min":0.34499999999999997,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":6.6470000000000002,"5min":53.927,"15min":53.927},"last":0.46999999999999997}]},{"osd":4,"last update":"Wed Sep 11 08:29:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.169,"15min":1.103},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.24099999999999999},"max":{"1min":6.6920000000000002,"5min":6.6920000000000002,"15min":6.6920000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":1.0900000000000001,"5min":1.0429999999999999,"15min":1.107},"min":{"1min":0.39700000000000002,"5min":0.311,"15min":0.27500000000000002},"max":{"1min":4.3070000000000004,"5min":4.6369999999999996,"15min":7.8970000000000002},"last":0.51900000000000002}]},{"osd":5,"last update":"Wed Sep 11 08:29:34 2024","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":1.0529999999999999,"15min":1.5249999999999999},"min":{"1min":0.36599999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":4.2270000000000003,"5min":4.5839999999999996,"15min":4.5839999999999996},"last":0.55700000000000005},{"interface":"front","average":{"1min":1.452,"5min":1.1619999999999999,"15min":1.3720000000000001},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":6.6589999999999998,"5min":6.6589999999999998,"15min":6.6589999999999998},"last":0.42899999999999999}]},{"osd":6,"last update":"Wed Sep 11 08:29:44 2024","interfaces":[{"interface":"back","average":{"1min":1.8149999999999999,"5min":1.3380000000000001,"15min":1.2150000000000001},"min":{"1min":0.371,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":6.3440000000000003,"5min":6.3440000000000003,"15min":8.0350000000000001},"last":0.41299999999999998},{"interface":"front","average":{"1min":1.085,"5min":1.206,"15min":1.1850000000000001},"min":{"1min":0.39400000000000002,"5min":0.375,"15min":0.32500000000000001},"max":{"1min":4.4390000000000001,"5min":5.9619999999999997,"15min":7.8710000000000004},"last":0.58899999999999997}]},{"osd":7,"last update":"Wed Sep 11 08:29:48 2024","interfaces":[{"interface":"back","average":{"1min":1.5629999999999999,"5min":1.3819999999999999,"15min":1.3640000000000001},"min":{"1min":0.41699999999999998,"5min":0.312,"15min":0.27500000000000002},"max":{"1min":7.6950000000000003,"5min":11.279,"15min":35.274999999999999},"last":0.45800000000000002},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.3049999999999999,"15min":1.4330000000000001},"min":{"1min":0.44,"5min":0.33600000000000002,"15min":0.32800000000000001},"max":{"1min":4.4660000000000002,"5min":11.305,"15min":35.265000000000001},"last":0.57399999999999995}]}]},{"osd":1,"up_from":13,"seq":55834575155,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":891000,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":890238,"kb_avail":92854152,"statfs":{"total":95995035648,"available":95082651648,"internally_reserved":0,"allocated":765952,"data_stored":375546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":911604056},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:29:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":1.337,"15min":1.5189999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":5.0110000000000001,"5min":14.045,"15min":63.932000000000002},"last":0.44700000000000001},{"interface":"front","average":{"1min":1.5900000000000001,"5min":1.5940000000000001,"15min":1.869},"min":{"1min":0.222,"5min":0.189,"15min":0.16800000000000001},"max":{"1min":5.1120000000000001,"5min":14.127000000000001,"15min":64.269000000000005},"last":0.27500000000000002}]},{"osd":2,"last update":"Wed Sep 11 08:29:24 2024","interfaces":[{"interface":"back","average":{"1min":2.2229999999999999,"5min":1.5700000000000001,"15min":1.3959999999999999},"min":{"1min":0.44800000000000001,"5min":0.19800000000000001,"15min":0.189},"max":{"1min":5.1959999999999997,"5min":13.928000000000001,"15min":20.925999999999998},"last":0.377},{"interface":"front","average":{"1min":2.0899999999999999,"5min":1.677,"15min":1.3759999999999999},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19},"max":{"1min":5.3700000000000001,"5min":16.099,"15min":20.812000000000001},"last":0.35299999999999998}]},{"osd":3,"last update":"Wed Sep 11 08:29:36 2024","interfaces":[{"interface":"back","average":{"1min":2.5670000000000002,"5min":1.663,"15min":1.5629999999999999},"min":{"1min":0.27100000000000002,"5min":0.20399999999999999,"15min":0.19700000000000001},"max":{"1min":7.9390000000000001,"5min":16.193999999999999,"15min":16.783999999999999},"last":0.314},{"interface":"front","average":{"1min":1.966,"5min":1.591,"15min":1.488},"min":{"1min":0.30499999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":4.415,"5min":16.033000000000001,"15min":16.033000000000001},"last":0.48799999999999999}]},{"osd":4,"last update":"Wed Sep 11 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":2.222,"5min":1.5940000000000001,"15min":1.9399999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.254},"max":{"1min":8.2680000000000007,"5min":16.178000000000001,"15min":110.86499999999999},"last":0.437},{"interface":"front","average":{"1min":1.528,"5min":1.6930000000000001,"15min":1.982},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28000000000000003},"max":{"1min":5.1980000000000004,"5min":19.009,"15min":110.831},"last":0.52000000000000002}]},{"osd":5,"last update":"Wed Sep 11 08:29:36 2024","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.4059999999999999,"15min":1.5660000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":4.2359999999999998,"5min":16.012,"15min":16.012},"last":0.54900000000000004},{"interface":"front","average":{"1min":1.827,"5min":1.508,"15min":1.5660000000000001},"min":{"1min":0.315,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":7.1420000000000003,"5min":16.206,"15min":16.206},"last":0.40699999999999997}]},{"osd":6,"last update":"Wed Sep 11 08:29:36 2024","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.6479999999999999,"15min":1.8660000000000001},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":3.786,"5min":16.143000000000001,"15min":124.31999999999999},"last":0.39700000000000002},{"interface":"front","average":{"1min":1.329,"5min":1.611,"15min":1.831},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.29799999999999999},"max":{"1min":3.6150000000000002,"5min":15.984,"15min":124.958},"last":0.57899999999999996}]},{"osd":7,"last update":"Wed Sep 11 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":2.3540000000000001,"5min":1.706,"15min":1.865},"min":{"1min":0.32500000000000001,"5min":0.318,"15min":0.28599999999999998},"max":{"1min":8.2940000000000005,"5min":16.228999999999999,"15min":109.149},"last":0.46400000000000002},{"interface":"front","average":{"1min":1.387,"5min":1.4419999999999999,"15min":1.869},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.318},"max":{"1min":3.7599999999999998,"5min":16.170999999999999,"15min":109.309},"last":0.41799999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738679,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":899568,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":898238,"kb_avail":92845584,"statfs":{"total":95995035648,"available":95073878016,"internally_reserved":0,"allocated":1347584,"data_stored":966548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":919796056},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 11 08:29:39 2024","interfaces":[{"interface":"back","average":{"1min":2.5299999999999998,"5min":3.2999999999999998,"15min":1.903},"min":{"1min":0.40799999999999997,"5min":0.159,"15min":0.159},"max":{"1min":10.962999999999999,"5min":98.373999999999995,"15min":98.373999999999995},"last":0.32700000000000001},{"interface":"front","average":{"1min":3.5289999999999999,"5min":3.609,"15min":2.246},"min":{"1min":0.505,"5min":0.17299999999999999,"15min":0.153},"max":{"1min":15.037000000000001,"5min":98.257999999999996,"15min":98.257999999999996},"last":0.23799999999999999}]},{"osd":2,"last update":"Wed Sep 11 08:30:08 2024","interfaces":[{"interface":"back","average":{"1min":1.4259999999999999,"5min":2.0569999999999999,"15min":1.5920000000000001},"min":{"1min":0.29999999999999999,"5min":0.19,"15min":0.19},"max":{"1min":5.2089999999999996,"5min":46.417000000000002,"15min":46.417000000000002},"last":0.28000000000000003},{"interface":"front","average":{"1min":2.5510000000000002,"5min":2.1240000000000001,"15min":1.6359999999999999},"min":{"1min":0.23000000000000001,"5min":0.22,"15min":0.22},"max":{"1min":13.331,"5min":46.438000000000002,"15min":46.438000000000002},"last":0.254}]},{"osd":3,"last update":"Wed Sep 11 08:29:30 2024","interfaces":[{"interface":"back","average":{"1min":2.1459999999999999,"5min":1.5569999999999999,"15min":1.95},"min":{"1min":0.30499999999999999,"5min":0.222,"15min":0.222},"max":{"1min":9.5180000000000007,"5min":9.5180000000000007,"15min":172.149},"last":0.29699999999999999},{"interface":"front","average":{"1min":1.754,"5min":1.647,"15min":1.9239999999999999},"min":{"1min":0.28499999999999998,"5min":0.20599999999999999,"15min":0.20300000000000001},"max":{"1min":8.5109999999999992,"5min":9.6170000000000009,"15min":174.899},"last":0.35199999999999998}]},{"osd":4,"last update":"Wed Sep 11 08:29:55 2024","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":1.2849999999999999,"15min":1.2210000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.252},"max":{"1min":2.6110000000000002,"5min":5.1609999999999996,"15min":9.6560000000000006},"last":0.47299999999999998},{"interface":"front","average":{"1min":1.0069999999999999,"5min":1.179,"15min":1.175},"min":{"1min":0.33400000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":2.6749999999999998,"5min":7.6630000000000003,"15min":8.3510000000000009},"last":0.42099999999999999}]},{"osd":5,"last update":"Wed Sep 11 08:29:35 2024","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.341,"15min":1.7969999999999999},"min":{"1min":0.28499999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":2.7000000000000002,"5min":7.7030000000000003,"15min":7.7030000000000003},"last":0.442},{"interface":"front","average":{"1min":1.226,"5min":1.266,"15min":1.633},"min":{"1min":0.36399999999999999,"5min":0.309,"15min":0.309},"max":{"1min":4.2119999999999997,"5min":7.718,"15min":7.718},"last":0.47999999999999998}]},{"osd":6,"last update":"Wed Sep 11 08:29:35 2024","interfaces":[{"interface":"back","average":{"1min":1.4650000000000001,"5min":1.518,"15min":1.2909999999999999},"min":{"1min":0.317,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":4.0810000000000004,"5min":9.7029999999999994,"15min":9.7029999999999994},"last":0.45700000000000002},{"interface":"front","average":{"1min":1.242,"5min":1.3200000000000001,"15min":1.2609999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.28299999999999997},"max":{"1min":4.1559999999999997,"5min":5.3419999999999996,"15min":7.4820000000000002},"last":0.41199999999999998}]},{"osd":7,"last update":"Wed Sep 11 08:29:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1299999999999999,"5min":1.3220000000000001,"15min":1.268},"min":{"1min":0.34499999999999997,"5min":0.34399999999999997,"15min":0.30399999999999999},"max":{"1min":3.1459999999999999,"5min":5.8049999999999997,"15min":7.5350000000000001},"last":0.42799999999999999},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.296,"15min":1.3540000000000001},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.32000000000000001},"max":{"1min":5.6829999999999998,"5min":5.6829999999999998,"15min":13.914},"last":0.51200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3425,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T08:30:23.955 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-11T08:30:23.955 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T08:30:23.956 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-11T08:30:23.956 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-11T08:30:24.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:23 smithi181 ceph-mon[32701]: from='client.16143 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:30:24.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:23 smithi181 ceph-mon[32701]: pgmap v1047: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-11T08:30:24.207 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:30:24.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:23 smithi150 ceph-mon[29276]: from='client.16143 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:30:24.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:23 smithi150 ceph-mon[29276]: pgmap v1047: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-11T08:30:24.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:23 smithi150 ceph-mon[39612]: from='client.16143 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:30:24.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:23 smithi150 ceph-mon[39612]: pgmap v1047: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-11T08:30:24.851 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-11T08:30:24.851 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:30:25.267 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T08:30:25.400 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:30:25 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:30:25] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:30:25.478 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-11T08:30:25.723 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:30:25.967 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:30:26.059 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:30:26.059 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-11T08:30:26.059 INFO:teuthology.orchestra.run.smithi150.stdout: "epoch": 449, 2024-09-11T08:30:26.060 INFO:teuthology.orchestra.run.smithi150.stdout: "fsid": "c1bddff4-7013-11ef-bcea-c7b262605968", 2024-09-11T08:30:26.060 INFO:teuthology.orchestra.run.smithi150.stdout: "created": "2024-09-11T08:02:15.344618+0000", 2024-09-11T08:30:26.060 INFO:teuthology.orchestra.run.smithi150.stdout: "modified": "2024-09-11T08:30:21.980453+0000", 2024-09-11T08:30:26.060 INFO:teuthology.orchestra.run.smithi150.stdout: "last_up_change": "2024-09-11T08:24:27.333279+0000", 2024-09-11T08:30:26.060 INFO:teuthology.orchestra.run.smithi150.stdout: "last_in_change": "2024-09-11T08:19:28.121267+0000", 2024-09-11T08:30:26.060 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-11T08:30:26.060 INFO:teuthology.orchestra.run.smithi150.stdout: "flags_num": 5801984, 2024-09-11T08:30:26.060 INFO:teuthology.orchestra.run.smithi150.stdout: "flags_set": [ 2024-09-11T08:30:26.060 INFO:teuthology.orchestra.run.smithi150.stdout: "noscrub", 2024-09-11T08:30:26.060 INFO:teuthology.orchestra.run.smithi150.stdout: "pglog_hardlimit", 2024-09-11T08:30:26.060 INFO:teuthology.orchestra.run.smithi150.stdout: "purged_snapdirs", 2024-09-11T08:30:26.060 INFO:teuthology.orchestra.run.smithi150.stdout: "recovery_deletes", 2024-09-11T08:30:26.060 INFO:teuthology.orchestra.run.smithi150.stdout: "sortbitwise" 2024-09-11T08:30:26.061 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-09-11T08:30:26.061 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_version": 20, 2024-09-11T08:30:26.061 INFO:teuthology.orchestra.run.smithi150.stdout: "full_ratio": 0.94999998807907104, 2024-09-11T08:30:26.061 INFO:teuthology.orchestra.run.smithi150.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-11T08:30:26.061 INFO:teuthology.orchestra.run.smithi150.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-11T08:30:26.061 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_snapshot": "", 2024-09-11T08:30:26.061 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_max": 11, 2024-09-11T08:30:26.061 INFO:teuthology.orchestra.run.smithi150.stdout: "max_osd": 8, 2024-09-11T08:30:26.061 INFO:teuthology.orchestra.run.smithi150.stdout: "require_min_compat_client": "luminous", 2024-09-11T08:30:26.061 INFO:teuthology.orchestra.run.smithi150.stdout: "min_compat_client": "luminous", 2024-09-11T08:30:26.061 INFO:teuthology.orchestra.run.smithi150.stdout: "require_osd_release": "quincy", 2024-09-11T08:30:26.061 INFO:teuthology.orchestra.run.smithi150.stdout: "pools": [ 2024-09-11T08:30:26.061 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.062 INFO:teuthology.orchestra.run.smithi150.stdout: "pool": 1, 2024-09-11T08:30:26.062 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_name": ".mgr", 2024-09-11T08:30:26.062 INFO:teuthology.orchestra.run.smithi150.stdout: "create_time": "2024-09-11T08:05:21.044729+0000", 2024-09-11T08:30:26.062 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-09-11T08:30:26.062 INFO:teuthology.orchestra.run.smithi150.stdout: "flags_names": "hashpspool", 2024-09-11T08:30:26.062 INFO:teuthology.orchestra.run.smithi150.stdout: "type": 1, 2024-09-11T08:30:26.062 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 3, 2024-09-11T08:30:26.062 INFO:teuthology.orchestra.run.smithi150.stdout: "min_size": 2, 2024-09-11T08:30:26.062 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_rule": 0, 2024-09-11T08:30:26.062 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_count": 0, 2024-09-11T08:30:26.062 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_target": 0, 2024-09-11T08:30:26.062 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T08:30:26.062 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T08:30:26.062 INFO:teuthology.orchestra.run.smithi150.stdout: "object_hash": 2, 2024-09-11T08:30:26.063 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_autoscale_mode": "off", 2024-09-11T08:30:26.063 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num": 1, 2024-09-11T08:30:26.063 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num": 1, 2024-09-11T08:30:26.063 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num_target": 1, 2024-09-11T08:30:26.063 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_target": 1, 2024-09-11T08:30:26.063 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_pending": 1, 2024-09-11T08:30:26.063 INFO:teuthology.orchestra.run.smithi150.stdout: "last_pg_merge_meta": { 2024-09-11T08:30:26.063 INFO:teuthology.orchestra.run.smithi150.stdout: "source_pgid": "0.0", 2024-09-11T08:30:26.063 INFO:teuthology.orchestra.run.smithi150.stdout: "ready_epoch": 0, 2024-09-11T08:30:26.063 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_started": 0, 2024-09-11T08:30:26.063 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_clean": 0, 2024-09-11T08:30:26.063 INFO:teuthology.orchestra.run.smithi150.stdout: "source_version": "0'0", 2024-09-11T08:30:26.063 INFO:teuthology.orchestra.run.smithi150.stdout: "target_version": "0'0" 2024-09-11T08:30:26.063 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.064 INFO:teuthology.orchestra.run.smithi150.stdout: "last_change": "255", 2024-09-11T08:30:26.064 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend": "0", 2024-09-11T08:30:26.064 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T08:30:26.064 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T08:30:26.064 INFO:teuthology.orchestra.run.smithi150.stdout: "auid": 0, 2024-09-11T08:30:26.064 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_mode": "selfmanaged", 2024-09-11T08:30:26.064 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_seq": 0, 2024-09-11T08:30:26.064 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_epoch": 0, 2024-09-11T08:30:26.064 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_snaps": [], 2024-09-11T08:30:26.064 INFO:teuthology.orchestra.run.smithi150.stdout: "removed_snaps": "[]", 2024-09-11T08:30:26.064 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_bytes": 0, 2024-09-11T08:30:26.064 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_objects": 0, 2024-09-11T08:30:26.064 INFO:teuthology.orchestra.run.smithi150.stdout: "tiers": [], 2024-09-11T08:30:26.065 INFO:teuthology.orchestra.run.smithi150.stdout: "tier_of": -1, 2024-09-11T08:30:26.065 INFO:teuthology.orchestra.run.smithi150.stdout: "read_tier": -1, 2024-09-11T08:30:26.065 INFO:teuthology.orchestra.run.smithi150.stdout: "write_tier": -1, 2024-09-11T08:30:26.065 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_mode": "none", 2024-09-11T08:30:26.065 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_bytes": 0, 2024-09-11T08:30:26.065 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_objects": 0, 2024-09-11T08:30:26.065 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T08:30:26.065 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T08:30:26.065 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T08:30:26.065 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_flush_age": 0, 2024-09-11T08:30:26.065 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_evict_age": 0, 2024-09-11T08:30:26.065 INFO:teuthology.orchestra.run.smithi150.stdout: "erasure_code_profile": "", 2024-09-11T08:30:26.065 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_params": { 2024-09-11T08:30:26.065 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "none" 2024-09-11T08:30:26.066 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.066 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_period": 0, 2024-09-11T08:30:26.066 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_count": 0, 2024-09-11T08:30:26.066 INFO:teuthology.orchestra.run.smithi150.stdout: "use_gmt_hitset": true, 2024-09-11T08:30:26.066 INFO:teuthology.orchestra.run.smithi150.stdout: "min_read_recency_for_promote": 0, 2024-09-11T08:30:26.066 INFO:teuthology.orchestra.run.smithi150.stdout: "min_write_recency_for_promote": 0, 2024-09-11T08:30:26.066 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T08:30:26.066 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_search_last_n": 0, 2024-09-11T08:30:26.066 INFO:teuthology.orchestra.run.smithi150.stdout: "grade_table": [], 2024-09-11T08:30:26.066 INFO:teuthology.orchestra.run.smithi150.stdout: "stripe_width": 0, 2024-09-11T08:30:26.066 INFO:teuthology.orchestra.run.smithi150.stdout: "expected_num_objects": 0, 2024-09-11T08:30:26.066 INFO:teuthology.orchestra.run.smithi150.stdout: "fast_read": false, 2024-09-11T08:30:26.066 INFO:teuthology.orchestra.run.smithi150.stdout: "options": { 2024-09-11T08:30:26.066 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_max": 32, 2024-09-11T08:30:26.067 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_min": 1 2024-09-11T08:30:26.067 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.067 INFO:teuthology.orchestra.run.smithi150.stdout: "application_metadata": { 2024-09-11T08:30:26.067 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": {} 2024-09-11T08:30:26.067 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:26.067 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.067 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.067 INFO:teuthology.orchestra.run.smithi150.stdout: "pool": 2, 2024-09-11T08:30:26.067 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_name": "datapool", 2024-09-11T08:30:26.067 INFO:teuthology.orchestra.run.smithi150.stdout: "create_time": "2024-09-11T08:07:28.786572+0000", 2024-09-11T08:30:26.067 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 8193, 2024-09-11T08:30:26.067 INFO:teuthology.orchestra.run.smithi150.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-11T08:30:26.067 INFO:teuthology.orchestra.run.smithi150.stdout: "type": 1, 2024-09-11T08:30:26.067 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 3, 2024-09-11T08:30:26.068 INFO:teuthology.orchestra.run.smithi150.stdout: "min_size": 2, 2024-09-11T08:30:26.068 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_rule": 0, 2024-09-11T08:30:26.068 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_count": 0, 2024-09-11T08:30:26.068 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_target": 0, 2024-09-11T08:30:26.068 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T08:30:26.068 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T08:30:26.068 INFO:teuthology.orchestra.run.smithi150.stdout: "object_hash": 2, 2024-09-11T08:30:26.068 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_autoscale_mode": "off", 2024-09-11T08:30:26.068 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num": 3, 2024-09-11T08:30:26.068 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num": 3, 2024-09-11T08:30:26.068 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num_target": 3, 2024-09-11T08:30:26.068 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_target": 3, 2024-09-11T08:30:26.068 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_pending": 3, 2024-09-11T08:30:26.068 INFO:teuthology.orchestra.run.smithi150.stdout: "last_pg_merge_meta": { 2024-09-11T08:30:26.068 INFO:teuthology.orchestra.run.smithi150.stdout: "source_pgid": "0.0", 2024-09-11T08:30:26.069 INFO:teuthology.orchestra.run.smithi150.stdout: "ready_epoch": 0, 2024-09-11T08:30:26.069 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_started": 0, 2024-09-11T08:30:26.069 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_clean": 0, 2024-09-11T08:30:26.069 INFO:teuthology.orchestra.run.smithi150.stdout: "source_version": "0'0", 2024-09-11T08:30:26.069 INFO:teuthology.orchestra.run.smithi150.stdout: "target_version": "0'0" 2024-09-11T08:30:26.069 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.069 INFO:teuthology.orchestra.run.smithi150.stdout: "last_change": "63", 2024-09-11T08:30:26.069 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend": "0", 2024-09-11T08:30:26.069 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T08:30:26.069 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T08:30:26.069 INFO:teuthology.orchestra.run.smithi150.stdout: "auid": 0, 2024-09-11T08:30:26.069 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_mode": "selfmanaged", 2024-09-11T08:30:26.069 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_seq": 3, 2024-09-11T08:30:26.069 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_epoch": 63, 2024-09-11T08:30:26.070 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_snaps": [], 2024-09-11T08:30:26.070 INFO:teuthology.orchestra.run.smithi150.stdout: "removed_snaps": "[]", 2024-09-11T08:30:26.070 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_bytes": 0, 2024-09-11T08:30:26.070 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_objects": 0, 2024-09-11T08:30:26.070 INFO:teuthology.orchestra.run.smithi150.stdout: "tiers": [], 2024-09-11T08:30:26.070 INFO:teuthology.orchestra.run.smithi150.stdout: "tier_of": -1, 2024-09-11T08:30:26.070 INFO:teuthology.orchestra.run.smithi150.stdout: "read_tier": -1, 2024-09-11T08:30:26.070 INFO:teuthology.orchestra.run.smithi150.stdout: "write_tier": -1, 2024-09-11T08:30:26.070 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_mode": "none", 2024-09-11T08:30:26.070 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_bytes": 0, 2024-09-11T08:30:26.070 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_objects": 0, 2024-09-11T08:30:26.070 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T08:30:26.070 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T08:30:26.071 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T08:30:26.071 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_flush_age": 0, 2024-09-11T08:30:26.071 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_evict_age": 0, 2024-09-11T08:30:26.071 INFO:teuthology.orchestra.run.smithi150.stdout: "erasure_code_profile": "", 2024-09-11T08:30:26.071 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_params": { 2024-09-11T08:30:26.071 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "none" 2024-09-11T08:30:26.071 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.071 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_period": 0, 2024-09-11T08:30:26.071 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_count": 0, 2024-09-11T08:30:26.071 INFO:teuthology.orchestra.run.smithi150.stdout: "use_gmt_hitset": true, 2024-09-11T08:30:26.071 INFO:teuthology.orchestra.run.smithi150.stdout: "min_read_recency_for_promote": 0, 2024-09-11T08:30:26.071 INFO:teuthology.orchestra.run.smithi150.stdout: "min_write_recency_for_promote": 0, 2024-09-11T08:30:26.071 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T08:30:26.071 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_search_last_n": 0, 2024-09-11T08:30:26.071 INFO:teuthology.orchestra.run.smithi150.stdout: "grade_table": [], 2024-09-11T08:30:26.072 INFO:teuthology.orchestra.run.smithi150.stdout: "stripe_width": 0, 2024-09-11T08:30:26.072 INFO:teuthology.orchestra.run.smithi150.stdout: "expected_num_objects": 0, 2024-09-11T08:30:26.072 INFO:teuthology.orchestra.run.smithi150.stdout: "fast_read": false, 2024-09-11T08:30:26.072 INFO:teuthology.orchestra.run.smithi150.stdout: "options": {}, 2024-09-11T08:30:26.072 INFO:teuthology.orchestra.run.smithi150.stdout: "application_metadata": { 2024-09-11T08:30:26.072 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd": {} 2024-09-11T08:30:26.072 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:26.072 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.072 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.072 INFO:teuthology.orchestra.run.smithi150.stdout: "pool": 3, 2024-09-11T08:30:26.072 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_name": ".rgw.root", 2024-09-11T08:30:26.072 INFO:teuthology.orchestra.run.smithi150.stdout: "create_time": "2024-09-11T08:07:29.021286+0000", 2024-09-11T08:30:26.072 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-09-11T08:30:26.073 INFO:teuthology.orchestra.run.smithi150.stdout: "flags_names": "hashpspool", 2024-09-11T08:30:26.073 INFO:teuthology.orchestra.run.smithi150.stdout: "type": 1, 2024-09-11T08:30:26.073 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 3, 2024-09-11T08:30:26.073 INFO:teuthology.orchestra.run.smithi150.stdout: "min_size": 2, 2024-09-11T08:30:26.073 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_rule": 0, 2024-09-11T08:30:26.073 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_count": 0, 2024-09-11T08:30:26.073 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_target": 0, 2024-09-11T08:30:26.073 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T08:30:26.073 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T08:30:26.073 INFO:teuthology.orchestra.run.smithi150.stdout: "object_hash": 2, 2024-09-11T08:30:26.073 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_autoscale_mode": "off", 2024-09-11T08:30:26.073 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num": 32, 2024-09-11T08:30:26.074 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num": 32, 2024-09-11T08:30:26.074 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num_target": 32, 2024-09-11T08:30:26.074 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_target": 32, 2024-09-11T08:30:26.074 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_pending": 32, 2024-09-11T08:30:26.074 INFO:teuthology.orchestra.run.smithi150.stdout: "last_pg_merge_meta": { 2024-09-11T08:30:26.074 INFO:teuthology.orchestra.run.smithi150.stdout: "source_pgid": "0.0", 2024-09-11T08:30:26.074 INFO:teuthology.orchestra.run.smithi150.stdout: "ready_epoch": 0, 2024-09-11T08:30:26.074 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_started": 0, 2024-09-11T08:30:26.074 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_clean": 0, 2024-09-11T08:30:26.074 INFO:teuthology.orchestra.run.smithi150.stdout: "source_version": "0'0", 2024-09-11T08:30:26.074 INFO:teuthology.orchestra.run.smithi150.stdout: "target_version": "0'0" 2024-09-11T08:30:26.074 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.074 INFO:teuthology.orchestra.run.smithi150.stdout: "last_change": "60", 2024-09-11T08:30:26.074 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend": "0", 2024-09-11T08:30:26.075 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T08:30:26.075 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T08:30:26.075 INFO:teuthology.orchestra.run.smithi150.stdout: "auid": 0, 2024-09-11T08:30:26.075 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_mode": "selfmanaged", 2024-09-11T08:30:26.075 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_seq": 0, 2024-09-11T08:30:26.075 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_epoch": 0, 2024-09-11T08:30:26.075 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_snaps": [], 2024-09-11T08:30:26.076 INFO:teuthology.orchestra.run.smithi150.stdout: "removed_snaps": "[]", 2024-09-11T08:30:26.076 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_bytes": 0, 2024-09-11T08:30:26.076 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_objects": 0, 2024-09-11T08:30:26.076 INFO:teuthology.orchestra.run.smithi150.stdout: "tiers": [], 2024-09-11T08:30:26.076 INFO:teuthology.orchestra.run.smithi150.stdout: "tier_of": -1, 2024-09-11T08:30:26.076 INFO:teuthology.orchestra.run.smithi150.stdout: "read_tier": -1, 2024-09-11T08:30:26.076 INFO:teuthology.orchestra.run.smithi150.stdout: "write_tier": -1, 2024-09-11T08:30:26.076 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_mode": "none", 2024-09-11T08:30:26.076 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_bytes": 0, 2024-09-11T08:30:26.076 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_objects": 0, 2024-09-11T08:30:26.076 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T08:30:26.076 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T08:30:26.076 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T08:30:26.076 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_flush_age": 0, 2024-09-11T08:30:26.077 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_evict_age": 0, 2024-09-11T08:30:26.077 INFO:teuthology.orchestra.run.smithi150.stdout: "erasure_code_profile": "", 2024-09-11T08:30:26.077 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_params": { 2024-09-11T08:30:26.077 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "none" 2024-09-11T08:30:26.077 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.078 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_period": 0, 2024-09-11T08:30:26.078 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_count": 0, 2024-09-11T08:30:26.078 INFO:teuthology.orchestra.run.smithi150.stdout: "use_gmt_hitset": true, 2024-09-11T08:30:26.078 INFO:teuthology.orchestra.run.smithi150.stdout: "min_read_recency_for_promote": 0, 2024-09-11T08:30:26.078 INFO:teuthology.orchestra.run.smithi150.stdout: "min_write_recency_for_promote": 0, 2024-09-11T08:30:26.078 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T08:30:26.078 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_search_last_n": 0, 2024-09-11T08:30:26.078 INFO:teuthology.orchestra.run.smithi150.stdout: "grade_table": [], 2024-09-11T08:30:26.078 INFO:teuthology.orchestra.run.smithi150.stdout: "stripe_width": 0, 2024-09-11T08:30:26.079 INFO:teuthology.orchestra.run.smithi150.stdout: "expected_num_objects": 0, 2024-09-11T08:30:26.079 INFO:teuthology.orchestra.run.smithi150.stdout: "fast_read": false, 2024-09-11T08:30:26.079 INFO:teuthology.orchestra.run.smithi150.stdout: "options": {}, 2024-09-11T08:30:26.079 INFO:teuthology.orchestra.run.smithi150.stdout: "application_metadata": { 2024-09-11T08:30:26.079 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw": {} 2024-09-11T08:30:26.079 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:26.079 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.079 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.079 INFO:teuthology.orchestra.run.smithi150.stdout: "pool": 4, 2024-09-11T08:30:26.079 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_name": "default.rgw.log", 2024-09-11T08:30:26.079 INFO:teuthology.orchestra.run.smithi150.stdout: "create_time": "2024-09-11T08:07:30.334181+0000", 2024-09-11T08:30:26.079 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-09-11T08:30:26.079 INFO:teuthology.orchestra.run.smithi150.stdout: "flags_names": "hashpspool", 2024-09-11T08:30:26.080 INFO:teuthology.orchestra.run.smithi150.stdout: "type": 1, 2024-09-11T08:30:26.080 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 3, 2024-09-11T08:30:26.080 INFO:teuthology.orchestra.run.smithi150.stdout: "min_size": 2, 2024-09-11T08:30:26.080 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_rule": 0, 2024-09-11T08:30:26.080 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_count": 0, 2024-09-11T08:30:26.080 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_target": 0, 2024-09-11T08:30:26.080 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T08:30:26.080 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T08:30:26.080 INFO:teuthology.orchestra.run.smithi150.stdout: "object_hash": 2, 2024-09-11T08:30:26.080 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_autoscale_mode": "off", 2024-09-11T08:30:26.080 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num": 32, 2024-09-11T08:30:26.080 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num": 32, 2024-09-11T08:30:26.080 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num_target": 32, 2024-09-11T08:30:26.080 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_target": 32, 2024-09-11T08:30:26.081 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_pending": 32, 2024-09-11T08:30:26.081 INFO:teuthology.orchestra.run.smithi150.stdout: "last_pg_merge_meta": { 2024-09-11T08:30:26.081 INFO:teuthology.orchestra.run.smithi150.stdout: "source_pgid": "0.0", 2024-09-11T08:30:26.081 INFO:teuthology.orchestra.run.smithi150.stdout: "ready_epoch": 0, 2024-09-11T08:30:26.081 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_started": 0, 2024-09-11T08:30:26.081 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_clean": 0, 2024-09-11T08:30:26.081 INFO:teuthology.orchestra.run.smithi150.stdout: "source_version": "0'0", 2024-09-11T08:30:26.081 INFO:teuthology.orchestra.run.smithi150.stdout: "target_version": "0'0" 2024-09-11T08:30:26.081 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.081 INFO:teuthology.orchestra.run.smithi150.stdout: "last_change": "62", 2024-09-11T08:30:26.081 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend": "0", 2024-09-11T08:30:26.081 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T08:30:26.082 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T08:30:26.082 INFO:teuthology.orchestra.run.smithi150.stdout: "auid": 0, 2024-09-11T08:30:26.082 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_mode": "selfmanaged", 2024-09-11T08:30:26.082 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_seq": 0, 2024-09-11T08:30:26.082 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_epoch": 0, 2024-09-11T08:30:26.082 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_snaps": [], 2024-09-11T08:30:26.082 INFO:teuthology.orchestra.run.smithi150.stdout: "removed_snaps": "[]", 2024-09-11T08:30:26.082 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_bytes": 0, 2024-09-11T08:30:26.082 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_objects": 0, 2024-09-11T08:30:26.082 INFO:teuthology.orchestra.run.smithi150.stdout: "tiers": [], 2024-09-11T08:30:26.082 INFO:teuthology.orchestra.run.smithi150.stdout: "tier_of": -1, 2024-09-11T08:30:26.082 INFO:teuthology.orchestra.run.smithi150.stdout: "read_tier": -1, 2024-09-11T08:30:26.082 INFO:teuthology.orchestra.run.smithi150.stdout: "write_tier": -1, 2024-09-11T08:30:26.083 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_mode": "none", 2024-09-11T08:30:26.083 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_bytes": 0, 2024-09-11T08:30:26.083 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_objects": 0, 2024-09-11T08:30:26.083 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T08:30:26.083 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T08:30:26.083 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T08:30:26.083 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_flush_age": 0, 2024-09-11T08:30:26.083 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_evict_age": 0, 2024-09-11T08:30:26.083 INFO:teuthology.orchestra.run.smithi150.stdout: "erasure_code_profile": "", 2024-09-11T08:30:26.083 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_params": { 2024-09-11T08:30:26.083 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "none" 2024-09-11T08:30:26.083 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.083 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_period": 0, 2024-09-11T08:30:26.083 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_count": 0, 2024-09-11T08:30:26.084 INFO:teuthology.orchestra.run.smithi150.stdout: "use_gmt_hitset": true, 2024-09-11T08:30:26.084 INFO:teuthology.orchestra.run.smithi150.stdout: "min_read_recency_for_promote": 0, 2024-09-11T08:30:26.084 INFO:teuthology.orchestra.run.smithi150.stdout: "min_write_recency_for_promote": 0, 2024-09-11T08:30:26.084 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T08:30:26.084 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_search_last_n": 0, 2024-09-11T08:30:26.084 INFO:teuthology.orchestra.run.smithi150.stdout: "grade_table": [], 2024-09-11T08:30:26.084 INFO:teuthology.orchestra.run.smithi150.stdout: "stripe_width": 0, 2024-09-11T08:30:26.084 INFO:teuthology.orchestra.run.smithi150.stdout: "expected_num_objects": 0, 2024-09-11T08:30:26.084 INFO:teuthology.orchestra.run.smithi150.stdout: "fast_read": false, 2024-09-11T08:30:26.084 INFO:teuthology.orchestra.run.smithi150.stdout: "options": {}, 2024-09-11T08:30:26.084 INFO:teuthology.orchestra.run.smithi150.stdout: "application_metadata": { 2024-09-11T08:30:26.084 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw": {} 2024-09-11T08:30:26.084 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:26.085 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.085 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.085 INFO:teuthology.orchestra.run.smithi150.stdout: "pool": 5, 2024-09-11T08:30:26.085 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_name": "default.rgw.control", 2024-09-11T08:30:26.085 INFO:teuthology.orchestra.run.smithi150.stdout: "create_time": "2024-09-11T08:07:32.289626+0000", 2024-09-11T08:30:26.085 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-09-11T08:30:26.085 INFO:teuthology.orchestra.run.smithi150.stdout: "flags_names": "hashpspool", 2024-09-11T08:30:26.085 INFO:teuthology.orchestra.run.smithi150.stdout: "type": 1, 2024-09-11T08:30:26.085 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 3, 2024-09-11T08:30:26.085 INFO:teuthology.orchestra.run.smithi150.stdout: "min_size": 2, 2024-09-11T08:30:26.085 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_rule": 0, 2024-09-11T08:30:26.085 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_count": 0, 2024-09-11T08:30:26.085 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_target": 0, 2024-09-11T08:30:26.086 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T08:30:26.086 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T08:30:26.086 INFO:teuthology.orchestra.run.smithi150.stdout: "object_hash": 2, 2024-09-11T08:30:26.086 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_autoscale_mode": "off", 2024-09-11T08:30:26.086 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num": 32, 2024-09-11T08:30:26.086 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num": 32, 2024-09-11T08:30:26.086 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num_target": 32, 2024-09-11T08:30:26.086 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_target": 32, 2024-09-11T08:30:26.086 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_pending": 32, 2024-09-11T08:30:26.086 INFO:teuthology.orchestra.run.smithi150.stdout: "last_pg_merge_meta": { 2024-09-11T08:30:26.086 INFO:teuthology.orchestra.run.smithi150.stdout: "source_pgid": "0.0", 2024-09-11T08:30:26.086 INFO:teuthology.orchestra.run.smithi150.stdout: "ready_epoch": 0, 2024-09-11T08:30:26.086 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_started": 0, 2024-09-11T08:30:26.087 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_clean": 0, 2024-09-11T08:30:26.087 INFO:teuthology.orchestra.run.smithi150.stdout: "source_version": "0'0", 2024-09-11T08:30:26.087 INFO:teuthology.orchestra.run.smithi150.stdout: "target_version": "0'0" 2024-09-11T08:30:26.087 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.087 INFO:teuthology.orchestra.run.smithi150.stdout: "last_change": "64", 2024-09-11T08:30:26.087 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend": "0", 2024-09-11T08:30:26.087 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T08:30:26.087 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T08:30:26.087 INFO:teuthology.orchestra.run.smithi150.stdout: "auid": 0, 2024-09-11T08:30:26.087 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_mode": "selfmanaged", 2024-09-11T08:30:26.087 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_seq": 0, 2024-09-11T08:30:26.087 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_epoch": 0, 2024-09-11T08:30:26.087 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_snaps": [], 2024-09-11T08:30:26.087 INFO:teuthology.orchestra.run.smithi150.stdout: "removed_snaps": "[]", 2024-09-11T08:30:26.088 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_bytes": 0, 2024-09-11T08:30:26.088 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_objects": 0, 2024-09-11T08:30:26.088 INFO:teuthology.orchestra.run.smithi150.stdout: "tiers": [], 2024-09-11T08:30:26.088 INFO:teuthology.orchestra.run.smithi150.stdout: "tier_of": -1, 2024-09-11T08:30:26.088 INFO:teuthology.orchestra.run.smithi150.stdout: "read_tier": -1, 2024-09-11T08:30:26.088 INFO:teuthology.orchestra.run.smithi150.stdout: "write_tier": -1, 2024-09-11T08:30:26.088 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_mode": "none", 2024-09-11T08:30:26.088 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_bytes": 0, 2024-09-11T08:30:26.088 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_objects": 0, 2024-09-11T08:30:26.088 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T08:30:26.088 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T08:30:26.088 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T08:30:26.088 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_flush_age": 0, 2024-09-11T08:30:26.088 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_evict_age": 0, 2024-09-11T08:30:26.089 INFO:teuthology.orchestra.run.smithi150.stdout: "erasure_code_profile": "", 2024-09-11T08:30:26.089 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_params": { 2024-09-11T08:30:26.089 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "none" 2024-09-11T08:30:26.089 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.089 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_period": 0, 2024-09-11T08:30:26.089 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_count": 0, 2024-09-11T08:30:26.089 INFO:teuthology.orchestra.run.smithi150.stdout: "use_gmt_hitset": true, 2024-09-11T08:30:26.089 INFO:teuthology.orchestra.run.smithi150.stdout: "min_read_recency_for_promote": 0, 2024-09-11T08:30:26.089 INFO:teuthology.orchestra.run.smithi150.stdout: "min_write_recency_for_promote": 0, 2024-09-11T08:30:26.089 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T08:30:26.089 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_search_last_n": 0, 2024-09-11T08:30:26.089 INFO:teuthology.orchestra.run.smithi150.stdout: "grade_table": [], 2024-09-11T08:30:26.089 INFO:teuthology.orchestra.run.smithi150.stdout: "stripe_width": 0, 2024-09-11T08:30:26.089 INFO:teuthology.orchestra.run.smithi150.stdout: "expected_num_objects": 0, 2024-09-11T08:30:26.090 INFO:teuthology.orchestra.run.smithi150.stdout: "fast_read": false, 2024-09-11T08:30:26.090 INFO:teuthology.orchestra.run.smithi150.stdout: "options": {}, 2024-09-11T08:30:26.090 INFO:teuthology.orchestra.run.smithi150.stdout: "application_metadata": { 2024-09-11T08:30:26.090 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw": {} 2024-09-11T08:30:26.090 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:26.090 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.090 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.090 INFO:teuthology.orchestra.run.smithi150.stdout: "pool": 6, 2024-09-11T08:30:26.090 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_name": "default.rgw.meta", 2024-09-11T08:30:26.090 INFO:teuthology.orchestra.run.smithi150.stdout: "create_time": "2024-09-11T08:07:34.351861+0000", 2024-09-11T08:30:26.090 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-09-11T08:30:26.090 INFO:teuthology.orchestra.run.smithi150.stdout: "flags_names": "hashpspool", 2024-09-11T08:30:26.090 INFO:teuthology.orchestra.run.smithi150.stdout: "type": 1, 2024-09-11T08:30:26.091 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 3, 2024-09-11T08:30:26.091 INFO:teuthology.orchestra.run.smithi150.stdout: "min_size": 2, 2024-09-11T08:30:26.091 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_rule": 0, 2024-09-11T08:30:26.091 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_count": 0, 2024-09-11T08:30:26.091 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_target": 0, 2024-09-11T08:30:26.091 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T08:30:26.091 INFO:teuthology.orchestra.run.smithi150.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T08:30:26.091 INFO:teuthology.orchestra.run.smithi150.stdout: "object_hash": 2, 2024-09-11T08:30:26.091 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_autoscale_mode": "off", 2024-09-11T08:30:26.091 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num": 32, 2024-09-11T08:30:26.091 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num": 32, 2024-09-11T08:30:26.091 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_placement_num_target": 32, 2024-09-11T08:30:26.091 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_target": 32, 2024-09-11T08:30:26.091 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num_pending": 32, 2024-09-11T08:30:26.092 INFO:teuthology.orchestra.run.smithi150.stdout: "last_pg_merge_meta": { 2024-09-11T08:30:26.092 INFO:teuthology.orchestra.run.smithi150.stdout: "source_pgid": "0.0", 2024-09-11T08:30:26.092 INFO:teuthology.orchestra.run.smithi150.stdout: "ready_epoch": 0, 2024-09-11T08:30:26.092 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_started": 0, 2024-09-11T08:30:26.092 INFO:teuthology.orchestra.run.smithi150.stdout: "last_epoch_clean": 0, 2024-09-11T08:30:26.092 INFO:teuthology.orchestra.run.smithi150.stdout: "source_version": "0'0", 2024-09-11T08:30:26.092 INFO:teuthology.orchestra.run.smithi150.stdout: "target_version": "0'0" 2024-09-11T08:30:26.092 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.092 INFO:teuthology.orchestra.run.smithi150.stdout: "last_change": "66", 2024-09-11T08:30:26.092 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend": "0", 2024-09-11T08:30:26.092 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T08:30:26.092 INFO:teuthology.orchestra.run.smithi150.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T08:30:26.092 INFO:teuthology.orchestra.run.smithi150.stdout: "auid": 0, 2024-09-11T08:30:26.093 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_mode": "selfmanaged", 2024-09-11T08:30:26.093 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_seq": 0, 2024-09-11T08:30:26.093 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_epoch": 0, 2024-09-11T08:30:26.093 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_snaps": [], 2024-09-11T08:30:26.093 INFO:teuthology.orchestra.run.smithi150.stdout: "removed_snaps": "[]", 2024-09-11T08:30:26.093 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_bytes": 0, 2024-09-11T08:30:26.093 INFO:teuthology.orchestra.run.smithi150.stdout: "quota_max_objects": 0, 2024-09-11T08:30:26.093 INFO:teuthology.orchestra.run.smithi150.stdout: "tiers": [], 2024-09-11T08:30:26.093 INFO:teuthology.orchestra.run.smithi150.stdout: "tier_of": -1, 2024-09-11T08:30:26.093 INFO:teuthology.orchestra.run.smithi150.stdout: "read_tier": -1, 2024-09-11T08:30:26.093 INFO:teuthology.orchestra.run.smithi150.stdout: "write_tier": -1, 2024-09-11T08:30:26.093 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_mode": "none", 2024-09-11T08:30:26.093 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_bytes": 0, 2024-09-11T08:30:26.093 INFO:teuthology.orchestra.run.smithi150.stdout: "target_max_objects": 0, 2024-09-11T08:30:26.094 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T08:30:26.094 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T08:30:26.094 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T08:30:26.094 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_flush_age": 0, 2024-09-11T08:30:26.094 INFO:teuthology.orchestra.run.smithi150.stdout: "cache_min_evict_age": 0, 2024-09-11T08:30:26.094 INFO:teuthology.orchestra.run.smithi150.stdout: "erasure_code_profile": "", 2024-09-11T08:30:26.094 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_params": { 2024-09-11T08:30:26.094 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "none" 2024-09-11T08:30:26.094 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.094 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_period": 0, 2024-09-11T08:30:26.094 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_count": 0, 2024-09-11T08:30:26.094 INFO:teuthology.orchestra.run.smithi150.stdout: "use_gmt_hitset": true, 2024-09-11T08:30:26.094 INFO:teuthology.orchestra.run.smithi150.stdout: "min_read_recency_for_promote": 0, 2024-09-11T08:30:26.094 INFO:teuthology.orchestra.run.smithi150.stdout: "min_write_recency_for_promote": 0, 2024-09-11T08:30:26.095 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T08:30:26.095 INFO:teuthology.orchestra.run.smithi150.stdout: "hit_set_search_last_n": 0, 2024-09-11T08:30:26.095 INFO:teuthology.orchestra.run.smithi150.stdout: "grade_table": [], 2024-09-11T08:30:26.095 INFO:teuthology.orchestra.run.smithi150.stdout: "stripe_width": 0, 2024-09-11T08:30:26.095 INFO:teuthology.orchestra.run.smithi150.stdout: "expected_num_objects": 0, 2024-09-11T08:30:26.095 INFO:teuthology.orchestra.run.smithi150.stdout: "fast_read": false, 2024-09-11T08:30:26.095 INFO:teuthology.orchestra.run.smithi150.stdout: "options": { 2024-09-11T08:30:26.095 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_autoscale_bias": 4 2024-09-11T08:30:26.095 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.095 INFO:teuthology.orchestra.run.smithi150.stdout: "application_metadata": { 2024-09-11T08:30:26.095 INFO:teuthology.orchestra.run.smithi150.stdout: "rgw": {} 2024-09-11T08:30:26.095 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:26.095 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:26.096 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-09-11T08:30:26.096 INFO:teuthology.orchestra.run.smithi150.stdout: "osds": [ 2024-09-11T08:30:26.096 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.096 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 0, 2024-09-11T08:30:26.096 INFO:teuthology.orchestra.run.smithi150.stdout: "uuid": "37b63db9-ed2a-4701-b63d-03ff9885cc0e", 2024-09-11T08:30:26.096 INFO:teuthology.orchestra.run.smithi150.stdout: "up": 1, 2024-09-11T08:30:26.096 INFO:teuthology.orchestra.run.smithi150.stdout: "in": 1, 2024-09-11T08:30:26.096 INFO:teuthology.orchestra.run.smithi150.stdout: "weight": 1, 2024-09-11T08:30:26.096 INFO:teuthology.orchestra.run.smithi150.stdout: "primary_affinity": 1, 2024-09-11T08:30:26.096 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_begin": 0, 2024-09-11T08:30:26.096 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_end": 0, 2024-09-11T08:30:26.096 INFO:teuthology.orchestra.run.smithi150.stdout: "up_from": 8, 2024-09-11T08:30:26.097 INFO:teuthology.orchestra.run.smithi150.stdout: "up_thru": 416, 2024-09-11T08:30:26.097 INFO:teuthology.orchestra.run.smithi150.stdout: "down_at": 0, 2024-09-11T08:30:26.097 INFO:teuthology.orchestra.run.smithi150.stdout: "lost_at": 0, 2024-09-11T08:30:26.097 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addrs": { 2024-09-11T08:30:26.097 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:30:26.097 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.097 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:30:26.097 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6801", 2024-09-11T08:30:26.097 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 334255279 2024-09-11T08:30:26.097 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:26.097 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:26.097 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.097 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addrs": { 2024-09-11T08:30:26.097 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:30:26.098 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.098 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:30:26.098 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6802", 2024-09-11T08:30:26.098 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 334255279 2024-09-11T08:30:26.098 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:26.098 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:26.098 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.098 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addrs": { 2024-09-11T08:30:26.098 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:30:26.098 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.098 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:30:26.098 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6804", 2024-09-11T08:30:26.098 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 334255279 2024-09-11T08:30:26.098 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:26.099 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:26.099 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.099 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addrs": { 2024-09-11T08:30:26.099 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:30:26.099 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.099 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:30:26.099 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6803", 2024-09-11T08:30:26.099 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 334255279 2024-09-11T08:30:26.099 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:26.099 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:26.099 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.099 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addr": "172.21.15.150:6801/334255279", 2024-09-11T08:30:26.099 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addr": "172.21.15.150:6802/334255279", 2024-09-11T08:30:26.100 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addr": "172.21.15.150:6804/334255279", 2024-09-11T08:30:26.100 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addr": "172.21.15.150:6803/334255279", 2024-09-11T08:30:26.100 INFO:teuthology.orchestra.run.smithi150.stdout: "state": [ 2024-09-11T08:30:26.100 INFO:teuthology.orchestra.run.smithi150.stdout: "exists", 2024-09-11T08:30:26.100 INFO:teuthology.orchestra.run.smithi150.stdout: "up" 2024-09-11T08:30:26.100 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:26.100 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.100 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.100 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 1, 2024-09-11T08:30:26.100 INFO:teuthology.orchestra.run.smithi150.stdout: "uuid": "80378442-02bc-48aa-b8ca-6113a4092079", 2024-09-11T08:30:26.100 INFO:teuthology.orchestra.run.smithi150.stdout: "up": 1, 2024-09-11T08:30:26.100 INFO:teuthology.orchestra.run.smithi150.stdout: "in": 1, 2024-09-11T08:30:26.100 INFO:teuthology.orchestra.run.smithi150.stdout: "weight": 1, 2024-09-11T08:30:26.101 INFO:teuthology.orchestra.run.smithi150.stdout: "primary_affinity": 1, 2024-09-11T08:30:26.101 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_begin": 0, 2024-09-11T08:30:26.101 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_end": 0, 2024-09-11T08:30:26.101 INFO:teuthology.orchestra.run.smithi150.stdout: "up_from": 13, 2024-09-11T08:30:26.101 INFO:teuthology.orchestra.run.smithi150.stdout: "up_thru": 382, 2024-09-11T08:30:26.101 INFO:teuthology.orchestra.run.smithi150.stdout: "down_at": 0, 2024-09-11T08:30:26.101 INFO:teuthology.orchestra.run.smithi150.stdout: "lost_at": 0, 2024-09-11T08:30:26.101 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addrs": { 2024-09-11T08:30:26.101 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:30:26.101 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.101 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:30:26.101 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6805", 2024-09-11T08:30:26.101 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1129510619 2024-09-11T08:30:26.101 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:26.102 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:26.102 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.102 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addrs": { 2024-09-11T08:30:26.102 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:30:26.102 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.102 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:30:26.102 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6806", 2024-09-11T08:30:26.102 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1129510619 2024-09-11T08:30:26.102 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:26.102 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:26.102 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.102 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addrs": { 2024-09-11T08:30:26.102 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:30:26.103 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.103 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:30:26.103 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6808", 2024-09-11T08:30:26.103 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1129510619 2024-09-11T08:30:26.103 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:26.103 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:26.103 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.103 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addrs": { 2024-09-11T08:30:26.103 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:30:26.103 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.103 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:30:26.103 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6807", 2024-09-11T08:30:26.103 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1129510619 2024-09-11T08:30:26.104 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:26.104 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:26.104 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.104 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addr": "172.21.15.150:6805/1129510619", 2024-09-11T08:30:26.104 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addr": "172.21.15.150:6806/1129510619", 2024-09-11T08:30:26.104 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addr": "172.21.15.150:6808/1129510619", 2024-09-11T08:30:26.104 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addr": "172.21.15.150:6807/1129510619", 2024-09-11T08:30:26.104 INFO:teuthology.orchestra.run.smithi150.stdout: "state": [ 2024-09-11T08:30:26.104 INFO:teuthology.orchestra.run.smithi150.stdout: "exists", 2024-09-11T08:30:26.104 INFO:teuthology.orchestra.run.smithi150.stdout: "up" 2024-09-11T08:30:26.104 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:26.104 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.104 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.104 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 2, 2024-09-11T08:30:26.105 INFO:teuthology.orchestra.run.smithi150.stdout: "uuid": "0955bb92-36a0-4e9a-98e1-b3a5af89f6d2", 2024-09-11T08:30:26.105 INFO:teuthology.orchestra.run.smithi150.stdout: "up": 1, 2024-09-11T08:30:26.105 INFO:teuthology.orchestra.run.smithi150.stdout: "in": 1, 2024-09-11T08:30:26.105 INFO:teuthology.orchestra.run.smithi150.stdout: "weight": 1, 2024-09-11T08:30:26.105 INFO:teuthology.orchestra.run.smithi150.stdout: "primary_affinity": 1, 2024-09-11T08:30:26.105 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_begin": 0, 2024-09-11T08:30:26.105 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_end": 0, 2024-09-11T08:30:26.105 INFO:teuthology.orchestra.run.smithi150.stdout: "up_from": 18, 2024-09-11T08:30:26.105 INFO:teuthology.orchestra.run.smithi150.stdout: "up_thru": 411, 2024-09-11T08:30:26.105 INFO:teuthology.orchestra.run.smithi150.stdout: "down_at": 0, 2024-09-11T08:30:26.105 INFO:teuthology.orchestra.run.smithi150.stdout: "lost_at": 0, 2024-09-11T08:30:26.105 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addrs": { 2024-09-11T08:30:26.105 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:30:26.106 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.106 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:30:26.106 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6809", 2024-09-11T08:30:26.106 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 719975422 2024-09-11T08:30:26.106 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:26.106 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:26.106 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.106 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addrs": { 2024-09-11T08:30:26.106 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:30:26.106 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.106 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:30:26.106 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6810", 2024-09-11T08:30:26.106 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 719975422 2024-09-11T08:30:26.106 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:26.107 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:26.107 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.107 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addrs": { 2024-09-11T08:30:26.107 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:30:26.107 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.107 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:30:26.107 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6812", 2024-09-11T08:30:26.107 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 719975422 2024-09-11T08:30:26.107 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:26.107 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:26.107 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.107 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addrs": { 2024-09-11T08:30:26.107 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:30:26.107 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.108 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:30:26.108 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6811", 2024-09-11T08:30:26.108 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 719975422 2024-09-11T08:30:26.108 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:26.108 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:26.108 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.108 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addr": "172.21.15.150:6809/719975422", 2024-09-11T08:30:26.108 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addr": "172.21.15.150:6810/719975422", 2024-09-11T08:30:26.108 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addr": "172.21.15.150:6812/719975422", 2024-09-11T08:30:26.108 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addr": "172.21.15.150:6811/719975422", 2024-09-11T08:30:26.108 INFO:teuthology.orchestra.run.smithi150.stdout: "state": [ 2024-09-11T08:30:26.108 INFO:teuthology.orchestra.run.smithi150.stdout: "exists", 2024-09-11T08:30:26.108 INFO:teuthology.orchestra.run.smithi150.stdout: "up" 2024-09-11T08:30:26.109 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:26.109 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.109 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.109 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 3, 2024-09-11T08:30:26.109 INFO:teuthology.orchestra.run.smithi150.stdout: "uuid": "1d04d4d3-62e8-4460-8a07-77f7e5c62929", 2024-09-11T08:30:26.109 INFO:teuthology.orchestra.run.smithi150.stdout: "up": 1, 2024-09-11T08:30:26.109 INFO:teuthology.orchestra.run.smithi150.stdout: "in": 1, 2024-09-11T08:30:26.109 INFO:teuthology.orchestra.run.smithi150.stdout: "weight": 1, 2024-09-11T08:30:26.109 INFO:teuthology.orchestra.run.smithi150.stdout: "primary_affinity": 0.611114501953125, 2024-09-11T08:30:26.109 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_begin": 0, 2024-09-11T08:30:26.109 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_end": 0, 2024-09-11T08:30:26.109 INFO:teuthology.orchestra.run.smithi150.stdout: "up_from": 25, 2024-09-11T08:30:26.109 INFO:teuthology.orchestra.run.smithi150.stdout: "up_thru": 382, 2024-09-11T08:30:26.109 INFO:teuthology.orchestra.run.smithi150.stdout: "down_at": 0, 2024-09-11T08:30:26.110 INFO:teuthology.orchestra.run.smithi150.stdout: "lost_at": 0, 2024-09-11T08:30:26.110 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addrs": { 2024-09-11T08:30:26.110 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:30:26.110 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.110 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:30:26.110 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6813", 2024-09-11T08:30:26.110 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 3101032500 2024-09-11T08:30:26.110 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:26.110 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:26.110 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.110 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addrs": { 2024-09-11T08:30:26.110 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:30:26.110 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.110 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:30:26.111 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6814", 2024-09-11T08:30:26.111 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 3101032500 2024-09-11T08:30:26.111 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:26.111 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:26.111 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.111 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addrs": { 2024-09-11T08:30:26.111 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:30:26.111 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.111 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:30:26.111 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6816", 2024-09-11T08:30:26.111 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 3101032500 2024-09-11T08:30:26.111 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:26.111 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:26.111 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.112 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addrs": { 2024-09-11T08:30:26.112 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:30:26.112 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.112 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:30:26.112 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6815", 2024-09-11T08:30:26.112 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 3101032500 2024-09-11T08:30:26.112 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:26.112 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:26.112 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.113 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addr": "172.21.15.150:6813/3101032500", 2024-09-11T08:30:26.113 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addr": "172.21.15.150:6814/3101032500", 2024-09-11T08:30:26.113 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addr": "172.21.15.150:6816/3101032500", 2024-09-11T08:30:26.113 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addr": "172.21.15.150:6815/3101032500", 2024-09-11T08:30:26.113 INFO:teuthology.orchestra.run.smithi150.stdout: "state": [ 2024-09-11T08:30:26.113 INFO:teuthology.orchestra.run.smithi150.stdout: "exists", 2024-09-11T08:30:26.113 INFO:teuthology.orchestra.run.smithi150.stdout: "up" 2024-09-11T08:30:26.114 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:26.114 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.114 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.115 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 4, 2024-09-11T08:30:26.115 INFO:teuthology.orchestra.run.smithi150.stdout: "uuid": "c5321a3c-d2f0-4440-8475-2c111d261139", 2024-09-11T08:30:26.115 INFO:teuthology.orchestra.run.smithi150.stdout: "up": 1, 2024-09-11T08:30:26.115 INFO:teuthology.orchestra.run.smithi150.stdout: "in": 1, 2024-09-11T08:30:26.115 INFO:teuthology.orchestra.run.smithi150.stdout: "weight": 1, 2024-09-11T08:30:26.115 INFO:teuthology.orchestra.run.smithi150.stdout: "primary_affinity": 0.2719573974609375, 2024-09-11T08:30:26.115 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_begin": 0, 2024-09-11T08:30:26.115 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_end": 0, 2024-09-11T08:30:26.115 INFO:teuthology.orchestra.run.smithi150.stdout: "up_from": 31, 2024-09-11T08:30:26.116 INFO:teuthology.orchestra.run.smithi150.stdout: "up_thru": 382, 2024-09-11T08:30:26.116 INFO:teuthology.orchestra.run.smithi150.stdout: "down_at": 0, 2024-09-11T08:30:26.116 INFO:teuthology.orchestra.run.smithi150.stdout: "lost_at": 0, 2024-09-11T08:30:26.116 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addrs": { 2024-09-11T08:30:26.116 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:30:26.116 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.116 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:30:26.116 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6800", 2024-09-11T08:30:26.116 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2567452825 2024-09-11T08:30:26.116 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:26.116 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:26.116 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.116 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addrs": { 2024-09-11T08:30:26.117 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:30:26.117 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.117 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:30:26.117 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6801", 2024-09-11T08:30:26.117 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2567452825 2024-09-11T08:30:26.117 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:26.117 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:26.117 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.117 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addrs": { 2024-09-11T08:30:26.117 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:30:26.117 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.117 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:30:26.117 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6803", 2024-09-11T08:30:26.117 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2567452825 2024-09-11T08:30:26.118 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:26.118 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:26.118 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.118 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addrs": { 2024-09-11T08:30:26.118 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:30:26.118 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.118 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:30:26.118 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6802", 2024-09-11T08:30:26.118 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2567452825 2024-09-11T08:30:26.118 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:26.118 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:26.118 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.118 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addr": "172.21.15.181:6800/2567452825", 2024-09-11T08:30:26.119 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addr": "172.21.15.181:6801/2567452825", 2024-09-11T08:30:26.119 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addr": "172.21.15.181:6803/2567452825", 2024-09-11T08:30:26.119 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addr": "172.21.15.181:6802/2567452825", 2024-09-11T08:30:26.119 INFO:teuthology.orchestra.run.smithi150.stdout: "state": [ 2024-09-11T08:30:26.119 INFO:teuthology.orchestra.run.smithi150.stdout: "exists", 2024-09-11T08:30:26.119 INFO:teuthology.orchestra.run.smithi150.stdout: "up" 2024-09-11T08:30:26.119 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:26.119 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.119 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.119 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 5, 2024-09-11T08:30:26.119 INFO:teuthology.orchestra.run.smithi150.stdout: "uuid": "1262b2fe-f253-4a26-bfd4-c9dffb697c77", 2024-09-11T08:30:26.119 INFO:teuthology.orchestra.run.smithi150.stdout: "up": 1, 2024-09-11T08:30:26.119 INFO:teuthology.orchestra.run.smithi150.stdout: "in": 1, 2024-09-11T08:30:26.119 INFO:teuthology.orchestra.run.smithi150.stdout: "weight": 1, 2024-09-11T08:30:26.120 INFO:teuthology.orchestra.run.smithi150.stdout: "primary_affinity": 0, 2024-09-11T08:30:26.120 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_begin": 38, 2024-09-11T08:30:26.120 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_end": 339, 2024-09-11T08:30:26.120 INFO:teuthology.orchestra.run.smithi150.stdout: "up_from": 353, 2024-09-11T08:30:26.120 INFO:teuthology.orchestra.run.smithi150.stdout: "up_thru": 353, 2024-09-11T08:30:26.120 INFO:teuthology.orchestra.run.smithi150.stdout: "down_at": 340, 2024-09-11T08:30:26.120 INFO:teuthology.orchestra.run.smithi150.stdout: "lost_at": 0, 2024-09-11T08:30:26.120 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addrs": { 2024-09-11T08:30:26.120 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:30:26.120 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.120 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:30:26.120 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6804", 2024-09-11T08:30:26.120 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1821561027 2024-09-11T08:30:26.120 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:26.121 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:26.121 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.121 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addrs": { 2024-09-11T08:30:26.121 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:30:26.121 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.121 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:30:26.121 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6805", 2024-09-11T08:30:26.121 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1821561027 2024-09-11T08:30:26.121 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:26.121 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:26.122 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.122 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addrs": { 2024-09-11T08:30:26.122 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:30:26.122 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.122 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:30:26.122 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6807", 2024-09-11T08:30:26.122 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1821561027 2024-09-11T08:30:26.122 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:26.122 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:26.122 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.122 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addrs": { 2024-09-11T08:30:26.122 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:30:26.122 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.123 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:30:26.123 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6806", 2024-09-11T08:30:26.123 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1821561027 2024-09-11T08:30:26.123 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:26.123 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:26.123 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.123 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addr": "172.21.15.181:6804/1821561027", 2024-09-11T08:30:26.123 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addr": "172.21.15.181:6805/1821561027", 2024-09-11T08:30:26.123 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addr": "172.21.15.181:6807/1821561027", 2024-09-11T08:30:26.123 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addr": "172.21.15.181:6806/1821561027", 2024-09-11T08:30:26.123 INFO:teuthology.orchestra.run.smithi150.stdout: "state": [ 2024-09-11T08:30:26.123 INFO:teuthology.orchestra.run.smithi150.stdout: "exists", 2024-09-11T08:30:26.123 INFO:teuthology.orchestra.run.smithi150.stdout: "up" 2024-09-11T08:30:26.124 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:26.124 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.124 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.124 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 6, 2024-09-11T08:30:26.124 INFO:teuthology.orchestra.run.smithi150.stdout: "uuid": "7facca22-b3f1-4c20-833e-e3db798a08c4", 2024-09-11T08:30:26.124 INFO:teuthology.orchestra.run.smithi150.stdout: "up": 1, 2024-09-11T08:30:26.124 INFO:teuthology.orchestra.run.smithi150.stdout: "in": 1, 2024-09-11T08:30:26.124 INFO:teuthology.orchestra.run.smithi150.stdout: "weight": 1, 2024-09-11T08:30:26.124 INFO:teuthology.orchestra.run.smithi150.stdout: "primary_affinity": 1, 2024-09-11T08:30:26.124 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_begin": 0, 2024-09-11T08:30:26.124 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_end": 0, 2024-09-11T08:30:26.124 INFO:teuthology.orchestra.run.smithi150.stdout: "up_from": 45, 2024-09-11T08:30:26.124 INFO:teuthology.orchestra.run.smithi150.stdout: "up_thru": 382, 2024-09-11T08:30:26.125 INFO:teuthology.orchestra.run.smithi150.stdout: "down_at": 0, 2024-09-11T08:30:26.125 INFO:teuthology.orchestra.run.smithi150.stdout: "lost_at": 0, 2024-09-11T08:30:26.125 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addrs": { 2024-09-11T08:30:26.125 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:30:26.125 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.125 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:30:26.125 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6808", 2024-09-11T08:30:26.125 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1789880331 2024-09-11T08:30:26.125 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:26.125 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:26.125 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.125 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addrs": { 2024-09-11T08:30:26.125 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:30:26.125 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.126 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:30:26.126 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6809", 2024-09-11T08:30:26.126 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1789880331 2024-09-11T08:30:26.126 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:26.126 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:26.126 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.126 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addrs": { 2024-09-11T08:30:26.126 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:30:26.126 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.126 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:30:26.126 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6811", 2024-09-11T08:30:26.126 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1789880331 2024-09-11T08:30:26.126 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:26.127 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:26.127 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.127 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addrs": { 2024-09-11T08:30:26.127 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:30:26.127 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.127 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:30:26.127 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6810", 2024-09-11T08:30:26.127 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1789880331 2024-09-11T08:30:26.127 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:26.127 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:26.127 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.127 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addr": "172.21.15.181:6808/1789880331", 2024-09-11T08:30:26.127 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addr": "172.21.15.181:6809/1789880331", 2024-09-11T08:30:26.127 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addr": "172.21.15.181:6811/1789880331", 2024-09-11T08:30:26.128 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addr": "172.21.15.181:6810/1789880331", 2024-09-11T08:30:26.128 INFO:teuthology.orchestra.run.smithi150.stdout: "state": [ 2024-09-11T08:30:26.128 INFO:teuthology.orchestra.run.smithi150.stdout: "exists", 2024-09-11T08:30:26.128 INFO:teuthology.orchestra.run.smithi150.stdout: "up" 2024-09-11T08:30:26.128 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:26.128 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.128 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.128 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 7, 2024-09-11T08:30:26.128 INFO:teuthology.orchestra.run.smithi150.stdout: "uuid": "6ac1b06a-79bd-43ca-b91e-fcb0f4033a34", 2024-09-11T08:30:26.128 INFO:teuthology.orchestra.run.smithi150.stdout: "up": 1, 2024-09-11T08:30:26.128 INFO:teuthology.orchestra.run.smithi150.stdout: "in": 1, 2024-09-11T08:30:26.128 INFO:teuthology.orchestra.run.smithi150.stdout: "weight": 1, 2024-09-11T08:30:26.128 INFO:teuthology.orchestra.run.smithi150.stdout: "primary_affinity": 1, 2024-09-11T08:30:26.129 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_begin": 53, 2024-09-11T08:30:26.129 INFO:teuthology.orchestra.run.smithi150.stdout: "last_clean_end": 139, 2024-09-11T08:30:26.129 INFO:teuthology.orchestra.run.smithi150.stdout: "up_from": 168, 2024-09-11T08:30:26.129 INFO:teuthology.orchestra.run.smithi150.stdout: "up_thru": 382, 2024-09-11T08:30:26.129 INFO:teuthology.orchestra.run.smithi150.stdout: "down_at": 140, 2024-09-11T08:30:26.129 INFO:teuthology.orchestra.run.smithi150.stdout: "lost_at": 0, 2024-09-11T08:30:26.129 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addrs": { 2024-09-11T08:30:26.129 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:30:26.129 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.129 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:30:26.129 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6812", 2024-09-11T08:30:26.129 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2706455857 2024-09-11T08:30:26.129 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:26.130 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:26.130 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.130 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addrs": { 2024-09-11T08:30:26.130 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:30:26.130 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.130 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:30:26.130 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6813", 2024-09-11T08:30:26.130 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2706455857 2024-09-11T08:30:26.130 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:26.130 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:26.130 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.130 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addrs": { 2024-09-11T08:30:26.130 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:30:26.130 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.131 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:30:26.131 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6815", 2024-09-11T08:30:26.131 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2706455857 2024-09-11T08:30:26.131 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:26.131 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:26.131 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.131 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addrs": { 2024-09-11T08:30:26.131 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-09-11T08:30:26.131 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.131 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-09-11T08:30:26.131 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.181:6814", 2024-09-11T08:30:26.131 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2706455857 2024-09-11T08:30:26.131 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:26.132 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:26.132 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.132 INFO:teuthology.orchestra.run.smithi150.stdout: "public_addr": "172.21.15.181:6812/2706455857", 2024-09-11T08:30:26.132 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_addr": "172.21.15.181:6813/2706455857", 2024-09-11T08:30:26.132 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_back_addr": "172.21.15.181:6815/2706455857", 2024-09-11T08:30:26.132 INFO:teuthology.orchestra.run.smithi150.stdout: "heartbeat_front_addr": "172.21.15.181:6814/2706455857", 2024-09-11T08:30:26.132 INFO:teuthology.orchestra.run.smithi150.stdout: "state": [ 2024-09-11T08:30:26.132 INFO:teuthology.orchestra.run.smithi150.stdout: "exists", 2024-09-11T08:30:26.132 INFO:teuthology.orchestra.run.smithi150.stdout: "up" 2024-09-11T08:30:26.132 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:26.132 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:26.132 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-09-11T08:30:26.132 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_xinfo": [ 2024-09-11T08:30:26.133 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.133 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 0, 2024-09-11T08:30:26.133 INFO:teuthology.orchestra.run.smithi150.stdout: "down_stamp": "0.000000", 2024-09-11T08:30:26.133 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_probability": 0, 2024-09-11T08:30:26.133 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_interval": 0, 2024-09-11T08:30:26.133 INFO:teuthology.orchestra.run.smithi150.stdout: "features": 4540138320759226367, 2024-09-11T08:30:26.133 INFO:teuthology.orchestra.run.smithi150.stdout: "old_weight": 0, 2024-09-11T08:30:26.133 INFO:teuthology.orchestra.run.smithi150.stdout: "last_purged_snaps_scrub": "2024-09-11T08:29:53.660652+0000", 2024-09-11T08:30:26.133 INFO:teuthology.orchestra.run.smithi150.stdout: "dead_epoch": 0 2024-09-11T08:30:26.133 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.133 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.133 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 1, 2024-09-11T08:30:26.133 INFO:teuthology.orchestra.run.smithi150.stdout: "down_stamp": "0.000000", 2024-09-11T08:30:26.133 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_probability": 0, 2024-09-11T08:30:26.134 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_interval": 0, 2024-09-11T08:30:26.134 INFO:teuthology.orchestra.run.smithi150.stdout: "features": 4540138320759226367, 2024-09-11T08:30:26.134 INFO:teuthology.orchestra.run.smithi150.stdout: "old_weight": 0, 2024-09-11T08:30:26.134 INFO:teuthology.orchestra.run.smithi150.stdout: "last_purged_snaps_scrub": "2024-09-11T08:29:32.963256+0000", 2024-09-11T08:30:26.134 INFO:teuthology.orchestra.run.smithi150.stdout: "dead_epoch": 0 2024-09-11T08:30:26.134 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.134 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.134 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 2, 2024-09-11T08:30:26.134 INFO:teuthology.orchestra.run.smithi150.stdout: "down_stamp": "0.000000", 2024-09-11T08:30:26.134 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_probability": 0, 2024-09-11T08:30:26.134 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_interval": 0, 2024-09-11T08:30:26.134 INFO:teuthology.orchestra.run.smithi150.stdout: "features": 4540138320759226367, 2024-09-11T08:30:26.134 INFO:teuthology.orchestra.run.smithi150.stdout: "old_weight": 0, 2024-09-11T08:30:26.135 INFO:teuthology.orchestra.run.smithi150.stdout: "last_purged_snaps_scrub": "2024-09-11T08:29:44.083464+0000", 2024-09-11T08:30:26.135 INFO:teuthology.orchestra.run.smithi150.stdout: "dead_epoch": 0 2024-09-11T08:30:26.135 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.135 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.135 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 3, 2024-09-11T08:30:26.135 INFO:teuthology.orchestra.run.smithi150.stdout: "down_stamp": "0.000000", 2024-09-11T08:30:26.135 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_probability": 0, 2024-09-11T08:30:26.135 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_interval": 0, 2024-09-11T08:30:26.135 INFO:teuthology.orchestra.run.smithi150.stdout: "features": 4540138320759226367, 2024-09-11T08:30:26.135 INFO:teuthology.orchestra.run.smithi150.stdout: "old_weight": 0, 2024-09-11T08:30:26.135 INFO:teuthology.orchestra.run.smithi150.stdout: "last_purged_snaps_scrub": "2024-09-11T08:29:11.549937+0000", 2024-09-11T08:30:26.135 INFO:teuthology.orchestra.run.smithi150.stdout: "dead_epoch": 0 2024-09-11T08:30:26.135 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.136 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.136 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 4, 2024-09-11T08:30:26.136 INFO:teuthology.orchestra.run.smithi150.stdout: "down_stamp": "0.000000", 2024-09-11T08:30:26.136 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_probability": 0, 2024-09-11T08:30:26.136 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_interval": 0, 2024-09-11T08:30:26.136 INFO:teuthology.orchestra.run.smithi150.stdout: "features": 4540138320759226367, 2024-09-11T08:30:26.136 INFO:teuthology.orchestra.run.smithi150.stdout: "old_weight": 0, 2024-09-11T08:30:26.136 INFO:teuthology.orchestra.run.smithi150.stdout: "last_purged_snaps_scrub": "2024-09-11T08:29:41.619870+0000", 2024-09-11T08:30:26.136 INFO:teuthology.orchestra.run.smithi150.stdout: "dead_epoch": 0 2024-09-11T08:30:26.136 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.136 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.136 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 5, 2024-09-11T08:30:26.136 INFO:teuthology.orchestra.run.smithi150.stdout: "down_stamp": "0.000000", 2024-09-11T08:30:26.137 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_probability": 0, 2024-09-11T08:30:26.137 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_interval": 0, 2024-09-11T08:30:26.137 INFO:teuthology.orchestra.run.smithi150.stdout: "features": 4540138320759226367, 2024-09-11T08:30:26.137 INFO:teuthology.orchestra.run.smithi150.stdout: "old_weight": 0, 2024-09-11T08:30:26.137 INFO:teuthology.orchestra.run.smithi150.stdout: "last_purged_snaps_scrub": "2024-09-11T08:30:00.865291+0000", 2024-09-11T08:30:26.137 INFO:teuthology.orchestra.run.smithi150.stdout: "dead_epoch": 339 2024-09-11T08:30:26.137 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.137 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.137 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 6, 2024-09-11T08:30:26.137 INFO:teuthology.orchestra.run.smithi150.stdout: "down_stamp": "0.000000", 2024-09-11T08:30:26.137 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_probability": 0, 2024-09-11T08:30:26.137 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_interval": 0, 2024-09-11T08:30:26.137 INFO:teuthology.orchestra.run.smithi150.stdout: "features": 4540138320759226367, 2024-09-11T08:30:26.138 INFO:teuthology.orchestra.run.smithi150.stdout: "old_weight": 0, 2024-09-11T08:30:26.138 INFO:teuthology.orchestra.run.smithi150.stdout: "last_purged_snaps_scrub": "2024-09-11T08:29:48.049009+0000", 2024-09-11T08:30:26.138 INFO:teuthology.orchestra.run.smithi150.stdout: "dead_epoch": 0 2024-09-11T08:30:26.138 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.138 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.138 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": 7, 2024-09-11T08:30:26.138 INFO:teuthology.orchestra.run.smithi150.stdout: "down_stamp": "0.000000", 2024-09-11T08:30:26.138 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_probability": 0, 2024-09-11T08:30:26.138 INFO:teuthology.orchestra.run.smithi150.stdout: "laggy_interval": 0, 2024-09-11T08:30:26.138 INFO:teuthology.orchestra.run.smithi150.stdout: "features": 4540138320759226367, 2024-09-11T08:30:26.138 INFO:teuthology.orchestra.run.smithi150.stdout: "old_weight": 0, 2024-09-11T08:30:26.138 INFO:teuthology.orchestra.run.smithi150.stdout: "last_purged_snaps_scrub": "2024-09-11T08:29:39.447672+0000", 2024-09-11T08:30:26.138 INFO:teuthology.orchestra.run.smithi150.stdout: "dead_epoch": 139 2024-09-11T08:30:26.138 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:26.139 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-09-11T08:30:26.139 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_upmap": [ 2024-09-11T08:30:26.139 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.139 INFO:teuthology.orchestra.run.smithi150.stdout: "pgid": "3.4", 2024-09-11T08:30:26.139 INFO:teuthology.orchestra.run.smithi150.stdout: "osds": [ 2024-09-11T08:30:26.139 INFO:teuthology.orchestra.run.smithi150.stdout: 4, 2024-09-11T08:30:26.139 INFO:teuthology.orchestra.run.smithi150.stdout: 2, 2024-09-11T08:30:26.139 INFO:teuthology.orchestra.run.smithi150.stdout: 3 2024-09-11T08:30:26.139 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:26.139 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.139 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.139 INFO:teuthology.orchestra.run.smithi150.stdout: "pgid": "6.5", 2024-09-11T08:30:26.139 INFO:teuthology.orchestra.run.smithi150.stdout: "osds": [ 2024-09-11T08:30:26.140 INFO:teuthology.orchestra.run.smithi150.stdout: 2, 2024-09-11T08:30:26.140 INFO:teuthology.orchestra.run.smithi150.stdout: 4, 2024-09-11T08:30:26.140 INFO:teuthology.orchestra.run.smithi150.stdout: 7 2024-09-11T08:30:26.140 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:26.140 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:26.140 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-09-11T08:30:26.140 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_upmap_items": [ 2024-09-11T08:30:26.140 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.140 INFO:teuthology.orchestra.run.smithi150.stdout: "pgid": "3.6", 2024-09-11T08:30:26.140 INFO:teuthology.orchestra.run.smithi150.stdout: "mappings": [ 2024-09-11T08:30:26.140 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.140 INFO:teuthology.orchestra.run.smithi150.stdout: "from": 4, 2024-09-11T08:30:26.140 INFO:teuthology.orchestra.run.smithi150.stdout: "to": 2 2024-09-11T08:30:26.141 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:26.141 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:26.141 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.141 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.141 INFO:teuthology.orchestra.run.smithi150.stdout: "pgid": "4.e", 2024-09-11T08:30:26.141 INFO:teuthology.orchestra.run.smithi150.stdout: "mappings": [ 2024-09-11T08:30:26.141 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.141 INFO:teuthology.orchestra.run.smithi150.stdout: "from": 1, 2024-09-11T08:30:26.141 INFO:teuthology.orchestra.run.smithi150.stdout: "to": 5 2024-09-11T08:30:26.141 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:26.141 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:26.141 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.141 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.142 INFO:teuthology.orchestra.run.smithi150.stdout: "pgid": "4.14", 2024-09-11T08:30:26.142 INFO:teuthology.orchestra.run.smithi150.stdout: "mappings": [ 2024-09-11T08:30:26.142 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.142 INFO:teuthology.orchestra.run.smithi150.stdout: "from": 1, 2024-09-11T08:30:26.142 INFO:teuthology.orchestra.run.smithi150.stdout: "to": 2 2024-09-11T08:30:26.142 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:26.142 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:26.142 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.142 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.142 INFO:teuthology.orchestra.run.smithi150.stdout: "pgid": "4.1d", 2024-09-11T08:30:26.142 INFO:teuthology.orchestra.run.smithi150.stdout: "mappings": [ 2024-09-11T08:30:26.142 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.142 INFO:teuthology.orchestra.run.smithi150.stdout: "from": 1, 2024-09-11T08:30:26.142 INFO:teuthology.orchestra.run.smithi150.stdout: "to": 2 2024-09-11T08:30:26.143 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:26.143 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:26.143 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.143 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.143 INFO:teuthology.orchestra.run.smithi150.stdout: "pgid": "5.3", 2024-09-11T08:30:26.143 INFO:teuthology.orchestra.run.smithi150.stdout: "mappings": [ 2024-09-11T08:30:26.143 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.143 INFO:teuthology.orchestra.run.smithi150.stdout: "from": 0, 2024-09-11T08:30:26.143 INFO:teuthology.orchestra.run.smithi150.stdout: "to": 3 2024-09-11T08:30:26.143 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:26.143 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:26.143 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.143 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.144 INFO:teuthology.orchestra.run.smithi150.stdout: "pgid": "5.1e", 2024-09-11T08:30:26.144 INFO:teuthology.orchestra.run.smithi150.stdout: "mappings": [ 2024-09-11T08:30:26.144 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-11T08:30:26.144 INFO:teuthology.orchestra.run.smithi150.stdout: "from": 0, 2024-09-11T08:30:26.144 INFO:teuthology.orchestra.run.smithi150.stdout: "to": 6 2024-09-11T08:30:26.144 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:26.144 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-11T08:30:26.144 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:26.144 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-09-11T08:30:26.144 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_temp": [], 2024-09-11T08:30:26.144 INFO:teuthology.orchestra.run.smithi150.stdout: "primary_temp": [], 2024-09-11T08:30:26.144 INFO:teuthology.orchestra.run.smithi150.stdout: "blocklist": { 2024-09-11T08:30:26.144 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/2620931070": "2024-09-12T08:08:06.649692+0000", 2024-09-11T08:30:26.144 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/1073581409": "2024-09-12T08:02:45.986151+0000", 2024-09-11T08:30:26.145 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/2363649697": "2024-09-12T08:08:06.649692+0000", 2024-09-11T08:30:26.145 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:6800/603451071": "2024-09-12T08:02:45.986151+0000", 2024-09-11T08:30:26.145 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/1614413548": "2024-09-12T08:02:45.986151+0000", 2024-09-11T08:30:26.145 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/3673218166": "2024-09-12T08:03:08.740828+0000", 2024-09-11T08:30:26.145 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:6800/556498589": "2024-09-12T08:03:08.740828+0000", 2024-09-11T08:30:26.145 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/3666733994": "2024-09-12T08:03:08.740828+0000", 2024-09-11T08:30:26.145 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/3271479010": "2024-09-12T08:03:08.740828+0000", 2024-09-11T08:30:26.145 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/3185731090": "2024-09-12T08:08:06.649692+0000", 2024-09-11T08:30:26.145 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/2539412006": "2024-09-12T08:08:06.649692+0000", 2024-09-11T08:30:26.145 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/2740222408": "2024-09-12T08:02:45.986151+0000", 2024-09-11T08:30:26.145 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:6800/127856313": "2024-09-12T08:08:06.649692+0000", 2024-09-11T08:30:26.145 INFO:teuthology.orchestra.run.smithi150.stdout: "172.21.15.150:0/2375121603": "2024-09-12T08:08:06.649692+0000" 2024-09-11T08:30:26.145 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.145 INFO:teuthology.orchestra.run.smithi150.stdout: "range_blocklist": {}, 2024-09-11T08:30:26.146 INFO:teuthology.orchestra.run.smithi150.stdout: "erasure_code_profiles": { 2024-09-11T08:30:26.146 INFO:teuthology.orchestra.run.smithi150.stdout: "default": { 2024-09-11T08:30:26.146 INFO:teuthology.orchestra.run.smithi150.stdout: "crush-failure-domain": "osd", 2024-09-11T08:30:26.146 INFO:teuthology.orchestra.run.smithi150.stdout: "k": "2", 2024-09-11T08:30:26.146 INFO:teuthology.orchestra.run.smithi150.stdout: "m": "1", 2024-09-11T08:30:26.146 INFO:teuthology.orchestra.run.smithi150.stdout: "plugin": "jerasure", 2024-09-11T08:30:26.146 INFO:teuthology.orchestra.run.smithi150.stdout: "technique": "reed_sol_van" 2024-09-11T08:30:26.146 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:26.146 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-11T08:30:26.146 INFO:teuthology.orchestra.run.smithi150.stdout: "removed_snaps_queue": [], 2024-09-11T08:30:26.146 INFO:teuthology.orchestra.run.smithi150.stdout: "new_removed_snaps": [], 2024-09-11T08:30:26.146 INFO:teuthology.orchestra.run.smithi150.stdout: "new_purged_snaps": [], 2024-09-11T08:30:26.146 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_node_flags": {}, 2024-09-11T08:30:26.146 INFO:teuthology.orchestra.run.smithi150.stdout: "device_class_flags": {}, 2024-09-11T08:30:26.147 INFO:teuthology.orchestra.run.smithi150.stdout: "stretch_mode": { 2024-09-11T08:30:26.147 INFO:teuthology.orchestra.run.smithi150.stdout: "stretch_mode_enabled": false, 2024-09-11T08:30:26.147 INFO:teuthology.orchestra.run.smithi150.stdout: "stretch_bucket_count": 0, 2024-09-11T08:30:26.147 INFO:teuthology.orchestra.run.smithi150.stdout: "degraded_stretch_mode": 0, 2024-09-11T08:30:26.147 INFO:teuthology.orchestra.run.smithi150.stdout: "recovering_stretch_mode": 0, 2024-09-11T08:30:26.147 INFO:teuthology.orchestra.run.smithi150.stdout: "stretch_mode_bucket": 0 2024-09-11T08:30:26.147 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-11T08:30:26.147 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-11T08:30:26.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:26 smithi150 ceph-mon[39612]: pgmap v1048: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:30:26.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:26 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1206360996' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-11T08:30:26.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:26 smithi150 ceph-mon[29276]: pgmap v1048: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:30:26.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:26 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1206360996' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-11T08:30:26.501 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:26 smithi181 ceph-mon[32701]: pgmap v1048: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:30:26.502 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:26 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1206360996' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-11T08:30:27.231 INFO:tasks.thrashosds.thrasher:j is {'epoch': 449, 'fsid': 'c1bddff4-7013-11ef-bcea-c7b262605968', 'created': '2024-09-11T08:02:15.344618+0000', 'modified': '2024-09-11T08:30:21.980453+0000', 'last_up_change': '2024-09-11T08:24:27.333279+0000', 'last_in_change': '2024-09-11T08:19:28.121267+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 20, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 11, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-11T08:05:21.044729+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '255', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-11T08:07:28.786572+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 63, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-11T08:07:29.021286+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-11T08:07:30.334181+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-11T08:07:32.289626+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-11T08:07:34.351861+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}], 'osds': [{'osd': 0, 'uuid': '37b63db9-ed2a-4701-b63d-03ff9885cc0e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 416, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6801', 'nonce': 334255279}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6802', 'nonce': 334255279}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6804', 'nonce': 334255279}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6803', 'nonce': 334255279}]}, 'public_addr': '172.21.15.150:6801/334255279', 'cluster_addr': '172.21.15.150:6802/334255279', 'heartbeat_back_addr': '172.21.15.150:6804/334255279', 'heartbeat_front_addr': '172.21.15.150:6803/334255279', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '80378442-02bc-48aa-b8ca-6113a4092079', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 382, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6805', 'nonce': 1129510619}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6806', 'nonce': 1129510619}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6808', 'nonce': 1129510619}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6807', 'nonce': 1129510619}]}, 'public_addr': '172.21.15.150:6805/1129510619', 'cluster_addr': '172.21.15.150:6806/1129510619', 'heartbeat_back_addr': '172.21.15.150:6808/1129510619', 'heartbeat_front_addr': '172.21.15.150:6807/1129510619', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '0955bb92-36a0-4e9a-98e1-b3a5af89f6d2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 411, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6809', 'nonce': 719975422}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6810', 'nonce': 719975422}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6812', 'nonce': 719975422}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6811', 'nonce': 719975422}]}, 'public_addr': '172.21.15.150:6809/719975422', 'cluster_addr': '172.21.15.150:6810/719975422', 'heartbeat_back_addr': '172.21.15.150:6812/719975422', 'heartbeat_front_addr': '172.21.15.150:6811/719975422', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '1d04d4d3-62e8-4460-8a07-77f7e5c62929', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.611114501953125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 382, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6813', 'nonce': 3101032500}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6814', 'nonce': 3101032500}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6816', 'nonce': 3101032500}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6815', 'nonce': 3101032500}]}, 'public_addr': '172.21.15.150:6813/3101032500', 'cluster_addr': '172.21.15.150:6814/3101032500', 'heartbeat_back_addr': '172.21.15.150:6816/3101032500', 'heartbeat_front_addr': '172.21.15.150:6815/3101032500', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'c5321a3c-d2f0-4440-8475-2c111d261139', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.2719573974609375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 382, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6800', 'nonce': 2567452825}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6801', 'nonce': 2567452825}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6803', 'nonce': 2567452825}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6802', 'nonce': 2567452825}]}, 'public_addr': '172.21.15.181:6800/2567452825', 'cluster_addr': '172.21.15.181:6801/2567452825', 'heartbeat_back_addr': '172.21.15.181:6803/2567452825', 'heartbeat_front_addr': '172.21.15.181:6802/2567452825', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '1262b2fe-f253-4a26-bfd4-c9dffb697c77', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 38, 'last_clean_end': 339, 'up_from': 353, 'up_thru': 353, 'down_at': 340, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6804', 'nonce': 1821561027}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6805', 'nonce': 1821561027}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6807', 'nonce': 1821561027}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6806', 'nonce': 1821561027}]}, 'public_addr': '172.21.15.181:6804/1821561027', 'cluster_addr': '172.21.15.181:6805/1821561027', 'heartbeat_back_addr': '172.21.15.181:6807/1821561027', 'heartbeat_front_addr': '172.21.15.181:6806/1821561027', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '7facca22-b3f1-4c20-833e-e3db798a08c4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 382, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6808', 'nonce': 1789880331}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6809', 'nonce': 1789880331}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6811', 'nonce': 1789880331}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6810', 'nonce': 1789880331}]}, 'public_addr': '172.21.15.181:6808/1789880331', 'cluster_addr': '172.21.15.181:6809/1789880331', 'heartbeat_back_addr': '172.21.15.181:6811/1789880331', 'heartbeat_front_addr': '172.21.15.181:6810/1789880331', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '6ac1b06a-79bd-43ca-b91e-fcb0f4033a34', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 53, 'last_clean_end': 139, 'up_from': 168, 'up_thru': 382, 'down_at': 140, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6812', 'nonce': 2706455857}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6813', 'nonce': 2706455857}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6815', 'nonce': 2706455857}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6814', 'nonce': 2706455857}]}, 'public_addr': '172.21.15.181:6812/2706455857', 'cluster_addr': '172.21.15.181:6813/2706455857', 'heartbeat_back_addr': '172.21.15.181:6815/2706455857', 'heartbeat_front_addr': '172.21.15.181:6814/2706455857', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T08:29:53.660652+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T08:29:32.963256+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T08:29:44.083464+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T08:29:11.549937+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T08:29:41.619870+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T08:30:00.865291+0000', 'dead_epoch': 339}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T08:29:48.049009+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T08:29:39.447672+0000', 'dead_epoch': 139}], 'pg_upmap': [{'pgid': '3.4', 'osds': [4, 2, 3]}, {'pgid': '6.5', 'osds': [2, 4, 7]}], 'pg_upmap_items': [{'pgid': '3.6', 'mappings': [{'from': 4, 'to': 2}]}, {'pgid': '4.e', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.14', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1d', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '5.3', 'mappings': [{'from': 0, 'to': 3}]}, {'pgid': '5.1e', 'mappings': [{'from': 0, 'to': 6}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.150:0/2620931070': '2024-09-12T08:08:06.649692+0000', '172.21.15.150:0/1073581409': '2024-09-12T08:02:45.986151+0000', '172.21.15.150:0/2363649697': '2024-09-12T08:08:06.649692+0000', '172.21.15.150:6800/603451071': '2024-09-12T08:02:45.986151+0000', '172.21.15.150:0/1614413548': '2024-09-12T08:02:45.986151+0000', '172.21.15.150:0/3673218166': '2024-09-12T08:03:08.740828+0000', '172.21.15.150:6800/556498589': '2024-09-12T08:03:08.740828+0000', '172.21.15.150:0/3666733994': '2024-09-12T08:03:08.740828+0000', '172.21.15.150:0/3271479010': '2024-09-12T08:03:08.740828+0000', '172.21.15.150:0/3185731090': '2024-09-12T08:08:06.649692+0000', '172.21.15.150:0/2539412006': '2024-09-12T08:08:06.649692+0000', '172.21.15.150:0/2740222408': '2024-09-12T08:02:45.986151+0000', '172.21.15.150:6800/127856313': '2024-09-12T08:08:06.649692+0000', '172.21.15.150:0/2375121603': '2024-09-12T08:08:06.649692+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-11T08:30:27.232 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T08:30:27.355 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-11T08:30:27.355 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:30:27.984 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-11T08:30:28.230 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:30:28.467 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:28 smithi150 ceph-mon[29276]: pgmap v1049: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-11T08:30:28.468 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:28 smithi150 ceph-mon[39612]: pgmap v1049: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-11T08:30:28.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:28 smithi181 ceph-mon[32701]: pgmap v1049: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-11T08:30:28.672 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:30:28.738 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:30:28 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:30:28] "GET /metrics HTTP/1.1" 200 222267 "" "Prometheus/2.43.0" 2024-09-11T08:30:29.300 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:30:29 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:30:29.307 INFO:teuthology.orchestra.run.smithi150.stderr:noscrub is unset 2024-09-11T08:30:29.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:29 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/359734062' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:30:29.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:29 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/359734062' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:30:29.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:29 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/359734062' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-11T08:30:29.948 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T08:30:30.307 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-11T08:30:30.307 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:30:30.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:30 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:30:30.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:30 smithi150 ceph-mon[29276]: pgmap v1050: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-11T08:30:30.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:30 smithi150 ceph-mon[29276]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-09-11T08:30:30.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:30 smithi150 ceph-mon[29276]: Cluster is now healthy 2024-09-11T08:30:30.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:30 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/359734062' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:30:30.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:30 smithi150 ceph-mon[29276]: osdmap e450: 8 total, 8 up, 8 in 2024-09-11T08:30:30.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:30 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:30:30.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:30 smithi150 ceph-mon[39612]: pgmap v1050: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-11T08:30:30.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:30 smithi150 ceph-mon[39612]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-09-11T08:30:30.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:30 smithi150 ceph-mon[39612]: Cluster is now healthy 2024-09-11T08:30:30.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:30 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/359734062' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:30:30.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:30 smithi150 ceph-mon[39612]: osdmap e450: 8 total, 8 up, 8 in 2024-09-11T08:30:30.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:30 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:30:30.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:30 smithi181 ceph-mon[32701]: pgmap v1050: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-11T08:30:30.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:30 smithi181 ceph-mon[32701]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-09-11T08:30:30.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:30 smithi181 ceph-mon[32701]: Cluster is now healthy 2024-09-11T08:30:30.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:30 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/359734062' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T08:30:30.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:30 smithi181 ceph-mon[32701]: osdmap e450: 8 total, 8 up, 8 in 2024-09-11T08:30:30.925 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-11T08:30:31.096 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:30:31.172 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:30:31.231 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:30:31.233 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-09-11T08:30:31.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:31 smithi150 ceph-mon[29276]: 6.1a scrub starts 2024-09-11T08:30:31.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:31 smithi150 ceph-mon[29276]: 6.1a scrub ok 2024-09-11T08:30:31.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:31 smithi150 ceph-mon[29276]: 3.1d scrub starts 2024-09-11T08:30:31.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:31 smithi150 ceph-mon[29276]: 3.1d scrub ok 2024-09-11T08:30:31.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:31 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:30:31.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:31 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:30:31.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:31 smithi150 ceph-mon[39612]: 6.1a scrub starts 2024-09-11T08:30:31.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:31 smithi150 ceph-mon[39612]: 6.1a scrub ok 2024-09-11T08:30:31.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:31 smithi150 ceph-mon[39612]: 3.1d scrub starts 2024-09-11T08:30:31.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:31 smithi150 ceph-mon[39612]: 3.1d scrub ok 2024-09-11T08:30:31.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:31 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:30:31.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:31 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:30:31.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:31 smithi181 ceph-mon[32701]: 6.1a scrub starts 2024-09-11T08:30:31.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:31 smithi181 ceph-mon[32701]: 6.1a scrub ok 2024-09-11T08:30:31.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:31 smithi181 ceph-mon[32701]: 3.1d scrub starts 2024-09-11T08:30:31.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:31 smithi181 ceph-mon[32701]: 3.1d scrub ok 2024-09-11T08:30:31.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:31 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:30:31.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:31 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:30:31.890 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":1051,"stamp":"2024-09-11T08:30:29.306442+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596188,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":2632,"num_read_kb":2491,"num_write":1115,"num_write_kb":1671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":208,"num_bytes_recovered":1776519,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1264,"ondisk_log_size":1264,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6655428,"kb_used_data":7652,"kb_used_omap":22,"kb_used_meta":6647657,"kb_avail":743305788,"statfs":{"total":767960285184,"available":761145126912,"internally_reserved":0,"allocated":7835648,"data_stored":4826660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22853,"internal_metadata":6807201467},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.065161"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":525,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:21.199214+0000","last_change":"2024-09-11T08:30:21.199174+0000","last_active":"2024-09-11T08:30:21.199214+0000","last_peered":"2024-09-11T08:30:21.199214+0000","last_clean":"2024-09-11T08:30:21.199214+0000","last_became_active":"2024-09-11T08:24:28.358401+0000","last_became_peered":"2024-09-11T08:24:28.358401+0000","last_unstale":"2024-09-11T08:30:21.199214+0000","last_undegraded":"2024-09-11T08:30:21.199214+0000","last_fullsized":"2024-09-11T08:30:21.199214+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:21.199113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:30:21.199113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:35.680694+0000","scrub_duration":0.001416793,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"405'25","reported_seq":538,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:17.048888+0000","last_change":"2024-09-11T08:29:59.282881+0000","last_active":"2024-09-11T08:30:17.048888+0000","last_peered":"2024-09-11T08:30:17.048888+0000","last_clean":"2024-09-11T08:30:17.048888+0000","last_became_active":"2024-09-11T08:07:32.048048+0000","last_became_peered":"2024-09-11T08:07:32.048048+0000","last_unstale":"2024-09-11T08:30:17.048888+0000","last_undegraded":"2024-09-11T08:30:17.048888+0000","last_fullsized":"2024-09-11T08:30:17.048888+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'25","last_scrub_stamp":"2024-09-11T08:29:59.282798+0000","last_deep_scrub":"312'19","last_deep_scrub_stamp":"2024-09-11T08:21:59.122122+0000","last_clean_scrub_stamp":"2024-09-11T08:29:59.282798+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:09.025796+0000","scrub_duration":0.012064749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":561,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.902096+0000","last_change":"2024-09-11T08:27:03.997743+0000","last_active":"2024-09-11T08:30:16.902096+0000","last_peered":"2024-09-11T08:30:16.902096+0000","last_clean":"2024-09-11T08:30:16.902096+0000","last_became_active":"2024-09-11T08:24:28.372160+0000","last_became_peered":"2024-09-11T08:24:28.372160+0000","last_unstale":"2024-09-11T08:30:16.902096+0000","last_undegraded":"2024-09-11T08:30:16.902096+0000","last_fullsized":"2024-09-11T08:30:16.902096+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:03.997657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:59.531202+0000","last_clean_scrub_stamp":"2024-09-11T08:27:03.997657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035479840000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":419,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.902206+0000","last_change":"2024-09-11T08:27:06.987974+0000","last_active":"2024-09-11T08:30:16.902206+0000","last_peered":"2024-09-11T08:30:16.902206+0000","last_clean":"2024-09-11T08:30:16.902206+0000","last_became_active":"2024-09-11T08:24:28.354834+0000","last_became_peered":"2024-09-11T08:24:28.354834+0000","last_unstale":"2024-09-11T08:30:16.902206+0000","last_undegraded":"2024-09-11T08:30:16.902206+0000","last_fullsized":"2024-09-11T08:30:16.902206+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:06.987888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:02.470649+0000","last_clean_scrub_stamp":"2024-09-11T08:27:06.987888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030657430000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":510,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.861228+0000","last_change":"2024-09-11T08:29:03.775193+0000","last_active":"2024-09-11T08:30:16.861228+0000","last_peered":"2024-09-11T08:30:16.861228+0000","last_clean":"2024-09-11T08:30:16.861228+0000","last_became_active":"2024-09-11T08:15:56.521104+0000","last_became_peered":"2024-09-11T08:15:56.521104+0000","last_unstale":"2024-09-11T08:30:16.861228+0000","last_undegraded":"2024-09-11T08:30:16.861228+0000","last_fullsized":"2024-09-11T08:30:16.861228+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:03.775098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:29:03.775098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:30.682191+0000","scrub_duration":0.0091193990000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":293,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:17.049926+0000","last_change":"2024-09-11T08:30:00.543330+0000","last_active":"2024-09-11T08:30:17.049926+0000","last_peered":"2024-09-11T08:30:17.049926+0000","last_clean":"2024-09-11T08:30:17.049926+0000","last_became_active":"2024-09-11T08:17:37.550048+0000","last_became_peered":"2024-09-11T08:17:37.550048+0000","last_unstale":"2024-09-11T08:30:17.049926+0000","last_undegraded":"2024-09-11T08:30:17.049926+0000","last_fullsized":"2024-09-11T08:30:17.049926+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:00.543254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:30:00.543254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:27.529390+0000","scrub_duration":0.0048128770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"405'46","reported_seq":471,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:17.048279+0000","last_change":"2024-09-11T08:29:17.630424+0000","last_active":"2024-09-11T08:30:17.048279+0000","last_peered":"2024-09-11T08:30:17.048279+0000","last_clean":"2024-09-11T08:30:17.048279+0000","last_became_active":"2024-09-11T08:17:37.553866+0000","last_became_peered":"2024-09-11T08:17:37.553866+0000","last_unstale":"2024-09-11T08:30:17.048279+0000","last_undegraded":"2024-09-11T08:30:17.048279+0000","last_fullsized":"2024-09-11T08:30:17.048279+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'46","last_scrub_stamp":"2024-09-11T08:29:17.630303+0000","last_deep_scrub":"145'22","last_deep_scrub_stamp":"2024-09-11T08:15:57.478870+0000","last_clean_scrub_stamp":"2024-09-11T08:29:17.630303+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:23.815610+0000","scrub_duration":0.014805099,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":154,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:17.056163+0000","last_change":"2024-09-11T08:29:23.277986+0000","last_active":"2024-09-11T08:30:17.056163+0000","last_peered":"2024-09-11T08:30:17.056163+0000","last_clean":"2024-09-11T08:30:17.056163+0000","last_became_active":"2024-09-11T08:23:22.687041+0000","last_became_peered":"2024-09-11T08:23:22.687041+0000","last_unstale":"2024-09-11T08:30:17.056163+0000","last_undegraded":"2024-09-11T08:30:17.056163+0000","last_fullsized":"2024-09-11T08:30:17.056163+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:23.277854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:18:36.998503+0000","last_clean_scrub_stamp":"2024-09-11T08:29:23.277854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:37.611048+0000","scrub_duration":0.017784378,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":185,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:17.052345+0000","last_change":"2024-09-11T08:29:12.315549+0000","last_active":"2024-09-11T08:30:17.052345+0000","last_peered":"2024-09-11T08:30:17.052345+0000","last_clean":"2024-09-11T08:30:17.052345+0000","last_became_active":"2024-09-11T08:24:28.360021+0000","last_became_peered":"2024-09-11T08:24:28.360021+0000","last_unstale":"2024-09-11T08:30:17.052345+0000","last_undegraded":"2024-09-11T08:30:17.052345+0000","last_fullsized":"2024-09-11T08:30:17.052345+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:12.315453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:54.878409+0000","last_clean_scrub_stamp":"2024-09-11T08:29:12.315453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00128605,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"405'21","reported_seq":403,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.909433+0000","last_change":"2024-09-11T08:29:19.176724+0000","last_active":"2024-09-11T08:30:16.909433+0000","last_peered":"2024-09-11T08:30:16.909433+0000","last_clean":"2024-09-11T08:30:16.909433+0000","last_became_active":"2024-09-11T08:23:22.687182+0000","last_became_peered":"2024-09-11T08:23:22.687182+0000","last_unstale":"2024-09-11T08:30:16.909433+0000","last_undegraded":"2024-09-11T08:30:16.909433+0000","last_fullsized":"2024-09-11T08:30:16.909433+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'21","last_scrub_stamp":"2024-09-11T08:29:19.176584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:29:19.176584+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:45.130745+0000","scrub_duration":0.022662364000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":400,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.910711+0000","last_change":"2024-09-11T08:29:04.968206+0000","last_active":"2024-09-11T08:30:16.910711+0000","last_peered":"2024-09-11T08:30:16.910711+0000","last_clean":"2024-09-11T08:30:16.910711+0000","last_became_active":"2024-09-11T08:24:28.363985+0000","last_became_peered":"2024-09-11T08:24:28.363985+0000","last_unstale":"2024-09-11T08:30:16.910711+0000","last_undegraded":"2024-09-11T08:30:16.910711+0000","last_fullsized":"2024-09-11T08:30:16.910711+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:04.967870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:20:18.900482+0000","last_clean_scrub_stamp":"2024-09-11T08:29:04.967870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0081125309999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":489,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.908167+0000","last_change":"2024-09-11T08:28:57.875994+0000","last_active":"2024-09-11T08:30:16.908167+0000","last_peered":"2024-09-11T08:30:16.908167+0000","last_clean":"2024-09-11T08:30:16.908167+0000","last_became_active":"2024-09-11T08:14:24.792595+0000","last_became_peered":"2024-09-11T08:14:24.792595+0000","last_unstale":"2024-09-11T08:30:16.908167+0000","last_undegraded":"2024-09-11T08:30:16.908167+0000","last_fullsized":"2024-09-11T08:30:16.908167+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:28:57.875851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:28:57.875851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014796494,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":187,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.902669+0000","last_change":"2024-09-11T08:30:13.004150+0000","last_active":"2024-09-11T08:30:16.902669+0000","last_peered":"2024-09-11T08:30:16.902669+0000","last_clean":"2024-09-11T08:30:16.902669+0000","last_became_active":"2024-09-11T08:24:28.359153+0000","last_became_peered":"2024-09-11T08:24:28.359153+0000","last_unstale":"2024-09-11T08:30:16.902669+0000","last_undegraded":"2024-09-11T08:30:16.902669+0000","last_fullsized":"2024-09-11T08:30:16.902669+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:13.004098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:30:13.004098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:24.968043+0000","scrub_duration":0.0014473380000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"405'11","reported_seq":387,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.864263+0000","last_change":"2024-09-11T08:28:58.811469+0000","last_active":"2024-09-11T08:30:16.864263+0000","last_peered":"2024-09-11T08:30:16.864263+0000","last_clean":"2024-09-11T08:30:16.864263+0000","last_became_active":"2024-09-11T08:23:22.681876+0000","last_became_peered":"2024-09-11T08:23:22.681876+0000","last_unstale":"2024-09-11T08:30:16.864263+0000","last_undegraded":"2024-09-11T08:30:16.864263+0000","last_fullsized":"2024-09-11T08:30:16.864263+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'11","last_scrub_stamp":"2024-09-11T08:28:58.811380+0000","last_deep_scrub":"139'7","last_deep_scrub_stamp":"2024-09-11T08:12:24.869318+0000","last_clean_scrub_stamp":"2024-09-11T08:28:58.811380+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.007538215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":311,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:19.739093+0000","last_change":"2024-09-11T08:30:19.739061+0000","last_active":"2024-09-11T08:30:19.739093+0000","last_peered":"2024-09-11T08:30:19.739093+0000","last_clean":"2024-09-11T08:30:19.739093+0000","last_became_active":"2024-09-11T08:15:56.526954+0000","last_became_peered":"2024-09-11T08:15:56.526954+0000","last_unstale":"2024-09-11T08:30:19.739093+0000","last_undegraded":"2024-09-11T08:30:19.739093+0000","last_fullsized":"2024-09-11T08:30:19.739093+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:19.739007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:30:19.739007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:27.244365+0000","scrub_duration":0.0012135570000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":181,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.901965+0000","last_change":"2024-09-11T08:27:21.243352+0000","last_active":"2024-09-11T08:30:16.901965+0000","last_peered":"2024-09-11T08:30:16.901965+0000","last_clean":"2024-09-11T08:30:16.901965+0000","last_became_active":"2024-09-11T08:24:28.372097+0000","last_became_peered":"2024-09-11T08:24:28.372097+0000","last_unstale":"2024-09-11T08:30:16.901965+0000","last_undegraded":"2024-09-11T08:30:16.901965+0000","last_fullsized":"2024-09-11T08:30:16.901965+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:21.243267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:20:19.927616+0000","last_clean_scrub_stamp":"2024-09-11T08:27:21.243267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0078709639999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":437,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:17.048745+0000","last_change":"2024-09-11T08:30:05.450807+0000","last_active":"2024-09-11T08:30:17.048745+0000","last_peered":"2024-09-11T08:30:17.048745+0000","last_clean":"2024-09-11T08:30:17.048745+0000","last_became_active":"2024-09-11T08:24:28.368560+0000","last_became_peered":"2024-09-11T08:24:28.368560+0000","last_unstale":"2024-09-11T08:30:17.048745+0000","last_undegraded":"2024-09-11T08:30:17.048745+0000","last_fullsized":"2024-09-11T08:30:17.048745+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:05.450746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:30:05.450746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:15.048635+0000","scrub_duration":0.0012973489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":508,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.911962+0000","last_change":"2024-09-11T08:29:16.034440+0000","last_active":"2024-09-11T08:30:16.911962+0000","last_peered":"2024-09-11T08:30:16.911962+0000","last_clean":"2024-09-11T08:30:16.911962+0000","last_became_active":"2024-09-11T08:15:56.516941+0000","last_became_peered":"2024-09-11T08:15:56.516941+0000","last_unstale":"2024-09-11T08:30:16.911962+0000","last_undegraded":"2024-09-11T08:30:16.911962+0000","last_fullsized":"2024-09-11T08:30:16.911962+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:16.034353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:29:16.034353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:22.408537+0000","scrub_duration":0.0042549959999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"405'46","reported_seq":574,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:17.048957+0000","last_change":"2024-09-11T08:29:07.522912+0000","last_active":"2024-09-11T08:30:17.048957+0000","last_peered":"2024-09-11T08:30:17.048957+0000","last_clean":"2024-09-11T08:30:17.048957+0000","last_became_active":"2024-09-11T08:07:32.046363+0000","last_became_peered":"2024-09-11T08:07:32.046363+0000","last_unstale":"2024-09-11T08:30:17.048957+0000","last_undegraded":"2024-09-11T08:30:17.048957+0000","last_fullsized":"2024-09-11T08:30:17.048957+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'46","last_scrub_stamp":"2024-09-11T08:29:07.522566+0000","last_deep_scrub":"107'22","last_deep_scrub_stamp":"2024-09-11T08:16:04.159211+0000","last_clean_scrub_stamp":"2024-09-11T08:29:07.522566+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:25.044660+0000","scrub_duration":0.024496106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":504,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.902341+0000","last_change":"2024-09-11T08:28:55.792756+0000","last_active":"2024-09-11T08:30:16.902341+0000","last_peered":"2024-09-11T08:30:16.902341+0000","last_clean":"2024-09-11T08:30:16.902341+0000","last_became_active":"2024-09-11T08:07:34.055410+0000","last_became_peered":"2024-09-11T08:07:34.055410+0000","last_unstale":"2024-09-11T08:30:16.902341+0000","last_undegraded":"2024-09-11T08:30:16.902341+0000","last_fullsized":"2024-09-11T08:30:16.902341+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:28:55.792659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:19:28.870211+0000","last_clean_scrub_stamp":"2024-09-11T08:28:55.792659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:24.688414+0000","scrub_duration":0.015527124999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":320,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:17.050010+0000","last_change":"2024-09-11T08:29:56.574094+0000","last_active":"2024-09-11T08:30:17.050010+0000","last_peered":"2024-09-11T08:30:17.050010+0000","last_clean":"2024-09-11T08:30:17.050010+0000","last_became_active":"2024-09-11T08:24:28.357957+0000","last_became_peered":"2024-09-11T08:24:28.357957+0000","last_unstale":"2024-09-11T08:30:17.050010+0000","last_undegraded":"2024-09-11T08:30:17.050010+0000","last_fullsized":"2024-09-11T08:30:17.050010+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:56.574045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:29:56.574045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:13.992398+0000","scrub_duration":0.0013195139999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":153,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.902559+0000","last_change":"2024-09-11T08:29:08.797434+0000","last_active":"2024-09-11T08:30:16.902559+0000","last_peered":"2024-09-11T08:30:16.902559+0000","last_clean":"2024-09-11T08:30:16.902559+0000","last_became_active":"2024-09-11T08:23:22.686935+0000","last_became_peered":"2024-09-11T08:23:22.686935+0000","last_unstale":"2024-09-11T08:30:16.902559+0000","last_undegraded":"2024-09-11T08:30:16.902559+0000","last_fullsized":"2024-09-11T08:30:16.902559+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:08.797351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:29:08.797351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:35.864334+0000","scrub_duration":0.0047226200000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"405'24","reported_seq":362,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:17.048592+0000","last_change":"2024-09-11T08:29:09.521760+0000","last_active":"2024-09-11T08:30:17.048592+0000","last_peered":"2024-09-11T08:30:17.048592+0000","last_clean":"2024-09-11T08:30:17.048592+0000","last_became_active":"2024-09-11T08:17:37.553402+0000","last_became_peered":"2024-09-11T08:17:37.553402+0000","last_unstale":"2024-09-11T08:30:17.048592+0000","last_undegraded":"2024-09-11T08:30:17.048592+0000","last_fullsized":"2024-09-11T08:30:17.048592+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'24","last_scrub_stamp":"2024-09-11T08:29:09.520708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:29:09.520708+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:37.502124+0000","scrub_duration":0.0081557079999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":164,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:20.817654+0000","last_change":"2024-09-11T08:30:20.817617+0000","last_active":"2024-09-11T08:30:20.817654+0000","last_peered":"2024-09-11T08:30:20.817654+0000","last_clean":"2024-09-11T08:30:20.817654+0000","last_became_active":"2024-09-11T08:23:22.684583+0000","last_became_peered":"2024-09-11T08:23:22.684583+0000","last_unstale":"2024-09-11T08:30:20.817654+0000","last_undegraded":"2024-09-11T08:30:20.817654+0000","last_fullsized":"2024-09-11T08:30:20.817654+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:20.817558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:10:40.110490+0000","last_clean_scrub_stamp":"2024-09-11T08:30:20.817558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:39.821285+0000","scrub_duration":0.001195268,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":476,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.864175+0000","last_change":"2024-09-11T08:27:52.321730+0000","last_active":"2024-09-11T08:30:16.864175+0000","last_peered":"2024-09-11T08:30:16.864175+0000","last_clean":"2024-09-11T08:30:16.864175+0000","last_became_active":"2024-09-11T08:24:28.357198+0000","last_became_peered":"2024-09-11T08:24:28.357198+0000","last_unstale":"2024-09-11T08:30:16.864175+0000","last_undegraded":"2024-09-11T08:30:16.864175+0000","last_fullsized":"2024-09-11T08:30:16.864175+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-11T08:27:52.321646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:27:52.321646+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012244692999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":498,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.902172+0000","last_change":"2024-09-11T08:27:50.150362+0000","last_active":"2024-09-11T08:30:16.902172+0000","last_peered":"2024-09-11T08:30:16.902172+0000","last_clean":"2024-09-11T08:30:16.902172+0000","last_became_active":"2024-09-11T08:07:30.045932+0000","last_became_peered":"2024-09-11T08:07:30.045932+0000","last_unstale":"2024-09-11T08:30:16.902172+0000","last_undegraded":"2024-09-11T08:30:16.902172+0000","last_fullsized":"2024-09-11T08:30:16.902172+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:50.150256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:27:50.150256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026580110000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"405'35","reported_seq":599,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.914886+0000","last_change":"2024-09-11T08:28:56.912293+0000","last_active":"2024-09-11T08:30:16.914886+0000","last_peered":"2024-09-11T08:30:16.914886+0000","last_clean":"2024-09-11T08:30:16.914886+0000","last_became_active":"2024-09-11T08:15:56.515817+0000","last_became_peered":"2024-09-11T08:15:56.515817+0000","last_unstale":"2024-09-11T08:30:16.914886+0000","last_undegraded":"2024-09-11T08:30:16.914886+0000","last_fullsized":"2024-09-11T08:30:16.914886+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'35","last_scrub_stamp":"2024-09-11T08:28:56.911970+0000","last_deep_scrub":"350'25","last_deep_scrub_stamp":"2024-09-11T08:24:23.093269+0000","last_clean_scrub_stamp":"2024-09-11T08:28:56.911970+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.066494655,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"254'12","reported_seq":837,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-09-11T08:30:22.150032+0000","last_change":"2024-09-11T08:29:14.338360+0000","last_active":"2024-09-11T08:30:22.150032+0000","last_peered":"2024-09-11T08:30:22.150032+0000","last_clean":"2024-09-11T08:30:22.150032+0000","last_became_active":"2024-09-11T08:15:56.525498+0000","last_became_peered":"2024-09-11T08:15:56.525498+0000","last_unstale":"2024-09-11T08:30:22.150032+0000","last_undegraded":"2024-09-11T08:30:22.150032+0000","last_fullsized":"2024-09-11T08:30:22.150032+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'12","last_scrub_stamp":"2024-09-11T08:29:14.338276+0000","last_deep_scrub":"254'12","last_deep_scrub_stamp":"2024-09-11T08:18:34.253865+0000","last_clean_scrub_stamp":"2024-09-11T08:29:14.338276+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:42.163322+0000","scrub_duration":0.0021322020000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"405'39","reported_seq":567,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.904387+0000","last_change":"2024-09-11T08:27:58.226977+0000","last_active":"2024-09-11T08:30:16.904387+0000","last_peered":"2024-09-11T08:30:16.904387+0000","last_clean":"2024-09-11T08:30:16.904387+0000","last_became_active":"2024-09-11T08:07:32.054317+0000","last_became_peered":"2024-09-11T08:07:32.054317+0000","last_unstale":"2024-09-11T08:30:16.904387+0000","last_undegraded":"2024-09-11T08:30:16.904387+0000","last_fullsized":"2024-09-11T08:30:16.904387+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'39","last_scrub_stamp":"2024-09-11T08:27:58.226812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:27:58.226812+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015475453,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":519,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.861842+0000","last_change":"2024-09-11T08:29:14.660571+0000","last_active":"2024-09-11T08:30:16.861842+0000","last_peered":"2024-09-11T08:30:16.861842+0000","last_clean":"2024-09-11T08:30:16.861842+0000","last_became_active":"2024-09-11T08:15:56.521943+0000","last_became_peered":"2024-09-11T08:15:56.521943+0000","last_unstale":"2024-09-11T08:30:16.861842+0000","last_undegraded":"2024-09-11T08:30:16.861842+0000","last_fullsized":"2024-09-11T08:30:16.861842+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:14.660494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:29:14.660494+0000","last_clean_scrub_stamp":"2024-09-11T08:29:14.660494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:27.445624+0000","scrub_duration":0.0075203229999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"354'13","reported_seq":267,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-09-11T08:30:22.150418+0000","last_change":"2024-09-11T08:29:03.976374+0000","last_active":"2024-09-11T08:30:22.150418+0000","last_peered":"2024-09-11T08:30:22.150418+0000","last_clean":"2024-09-11T08:30:22.150418+0000","last_became_active":"2024-09-11T08:24:28.368212+0000","last_became_peered":"2024-09-11T08:24:28.368212+0000","last_unstale":"2024-09-11T08:30:22.150418+0000","last_undegraded":"2024-09-11T08:30:22.150418+0000","last_fullsized":"2024-09-11T08:30:22.150418+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'13","last_scrub_stamp":"2024-09-11T08:29:03.976322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:29:03.976322+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013262809999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":181,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.902307+0000","last_change":"2024-09-11T08:27:05.040598+0000","last_active":"2024-09-11T08:30:16.902307+0000","last_peered":"2024-09-11T08:30:16.902307+0000","last_clean":"2024-09-11T08:30:16.902307+0000","last_became_active":"2024-09-11T08:24:28.353556+0000","last_became_peered":"2024-09-11T08:24:28.353556+0000","last_unstale":"2024-09-11T08:30:16.902307+0000","last_undegraded":"2024-09-11T08:30:16.902307+0000","last_fullsized":"2024-09-11T08:30:16.902307+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:05.040523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:13:06.616332+0000","last_clean_scrub_stamp":"2024-09-11T08:27:05.040523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050758849999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"405'42","reported_seq":556,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.861351+0000","last_change":"2024-09-11T08:30:13.678049+0000","last_active":"2024-09-11T08:30:16.861351+0000","last_peered":"2024-09-11T08:30:16.861351+0000","last_clean":"2024-09-11T08:30:16.861351+0000","last_became_active":"2024-09-11T08:17:37.555703+0000","last_became_peered":"2024-09-11T08:17:37.555703+0000","last_unstale":"2024-09-11T08:30:16.861351+0000","last_undegraded":"2024-09-11T08:30:16.861351+0000","last_fullsized":"2024-09-11T08:30:16.861351+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'42","last_scrub_stamp":"2024-09-11T08:30:13.677992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:30:13.677992+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:42.259390+0000","scrub_duration":0.0024195800000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":469,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.863748+0000","last_change":"2024-09-11T08:29:19.403998+0000","last_active":"2024-09-11T08:30:16.863748+0000","last_peered":"2024-09-11T08:30:16.863748+0000","last_clean":"2024-09-11T08:30:16.863748+0000","last_became_active":"2024-09-11T08:17:37.553579+0000","last_became_peered":"2024-09-11T08:17:37.553579+0000","last_unstale":"2024-09-11T08:30:16.863748+0000","last_undegraded":"2024-09-11T08:30:16.863748+0000","last_fullsized":"2024-09-11T08:30:16.863748+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:19.403914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:29:19.403914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:40.219628+0000","scrub_duration":0.0063698699999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":518,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:17.049025+0000","last_change":"2024-09-11T08:28:58.174623+0000","last_active":"2024-09-11T08:30:17.049025+0000","last_peered":"2024-09-11T08:30:17.049025+0000","last_clean":"2024-09-11T08:30:17.049025+0000","last_became_active":"2024-09-11T08:15:56.522606+0000","last_became_peered":"2024-09-11T08:15:56.522606+0000","last_unstale":"2024-09-11T08:30:17.049025+0000","last_undegraded":"2024-09-11T08:30:17.049025+0000","last_fullsized":"2024-09-11T08:30:17.049025+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:28:58.174534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:28:58.174534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:25.648809+0000","scrub_duration":0.0051811230000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":144,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.864126+0000","last_change":"2024-09-11T08:28:57.804622+0000","last_active":"2024-09-11T08:30:16.864126+0000","last_peered":"2024-09-11T08:30:16.864126+0000","last_clean":"2024-09-11T08:30:16.864126+0000","last_became_active":"2024-09-11T08:23:22.683474+0000","last_became_peered":"2024-09-11T08:23:22.683474+0000","last_unstale":"2024-09-11T08:30:16.864126+0000","last_undegraded":"2024-09-11T08:30:16.864126+0000","last_fullsized":"2024-09-11T08:30:16.864126+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:28:57.804216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:22:50.151585+0000","last_clean_scrub_stamp":"2024-09-11T08:28:57.804216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015908835999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"405'42","reported_seq":449,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:17.056133+0000","last_change":"2024-09-11T08:29:21.318169+0000","last_active":"2024-09-11T08:30:17.056133+0000","last_peered":"2024-09-11T08:30:17.056133+0000","last_clean":"2024-09-11T08:30:17.056133+0000","last_became_active":"2024-09-11T08:24:28.358175+0000","last_became_peered":"2024-09-11T08:24:28.358175+0000","last_unstale":"2024-09-11T08:30:17.056133+0000","last_undegraded":"2024-09-11T08:30:17.056133+0000","last_fullsized":"2024-09-11T08:30:17.056133+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'42","last_scrub_stamp":"2024-09-11T08:29:21.317951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:29:21.317951+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:41.618306+0000","scrub_duration":0.0023805409999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":498,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.915137+0000","last_change":"2024-09-11T08:28:59.885192+0000","last_active":"2024-09-11T08:30:16.915137+0000","last_peered":"2024-09-11T08:30:16.915137+0000","last_clean":"2024-09-11T08:30:16.915137+0000","last_became_active":"2024-09-11T08:28:09.953224+0000","last_became_peered":"2024-09-11T08:28:09.953224+0000","last_unstale":"2024-09-11T08:30:16.915137+0000","last_undegraded":"2024-09-11T08:30:16.915137+0000","last_fullsized":"2024-09-11T08:30:16.915137+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:28:59.884493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:47.960329+0000","last_clean_scrub_stamp":"2024-09-11T08:28:59.884493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012139997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"82'9","reported_seq":792,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-09-11T08:30:21.983180+0000","last_change":"2024-09-11T08:28:59.818823+0000","last_active":"2024-09-11T08:30:21.983180+0000","last_peered":"2024-09-11T08:30:21.983180+0000","last_clean":"2024-09-11T08:30:21.983180+0000","last_became_active":"2024-09-11T08:07:34.056143+0000","last_became_peered":"2024-09-11T08:07:34.056143+0000","last_unstale":"2024-09-11T08:30:21.983180+0000","last_undegraded":"2024-09-11T08:30:21.983180+0000","last_fullsized":"2024-09-11T08:30:21.983180+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-11T08:28:59.818645+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-09-11T08:21:00.841426+0000","last_clean_scrub_stamp":"2024-09-11T08:28:59.818645+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011589633,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":463,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.863139+0000","last_change":"2024-09-11T08:28:46.329629+0000","last_active":"2024-09-11T08:30:16.863139+0000","last_peered":"2024-09-11T08:30:16.863139+0000","last_clean":"2024-09-11T08:30:16.863139+0000","last_became_active":"2024-09-11T08:14:24.788968+0000","last_became_peered":"2024-09-11T08:14:24.788968+0000","last_unstale":"2024-09-11T08:30:16.863139+0000","last_undegraded":"2024-09-11T08:30:16.863139+0000","last_fullsized":"2024-09-11T08:30:16.863139+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:28:46.329538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:28:46.329538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017611370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"405'30","reported_seq":601,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.902426+0000","last_change":"2024-09-11T08:30:10.086374+0000","last_active":"2024-09-11T08:30:16.902426+0000","last_peered":"2024-09-11T08:30:16.902426+0000","last_clean":"2024-09-11T08:30:16.902426+0000","last_became_active":"2024-09-11T08:24:28.355516+0000","last_became_peered":"2024-09-11T08:24:28.355516+0000","last_unstale":"2024-09-11T08:30:16.902426+0000","last_undegraded":"2024-09-11T08:30:16.902426+0000","last_fullsized":"2024-09-11T08:30:16.902426+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'30","last_scrub_stamp":"2024-09-11T08:30:10.086268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:30:10.086268+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:38.169586+0000","scrub_duration":0.0028819319999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":209,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:19.820001+0000","last_change":"2024-09-11T08:30:19.819949+0000","last_active":"2024-09-11T08:30:19.820001+0000","last_peered":"2024-09-11T08:30:19.820001+0000","last_clean":"2024-09-11T08:30:19.820001+0000","last_became_active":"2024-09-11T08:24:28.352741+0000","last_became_peered":"2024-09-11T08:24:28.352741+0000","last_unstale":"2024-09-11T08:30:19.820001+0000","last_undegraded":"2024-09-11T08:30:19.820001+0000","last_fullsized":"2024-09-11T08:30:19.820001+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:19.819890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:30:19.819890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:31.176436+0000","scrub_duration":0.0013660910000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"376'15","reported_seq":477,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-09-11T08:30:22.149867+0000","last_change":"2024-09-11T08:30:09.325453+0000","last_active":"2024-09-11T08:30:22.149867+0000","last_peered":"2024-09-11T08:30:22.149867+0000","last_clean":"2024-09-11T08:30:22.149867+0000","last_became_active":"2024-09-11T08:24:28.359787+0000","last_became_peered":"2024-09-11T08:24:28.359787+0000","last_unstale":"2024-09-11T08:30:22.149867+0000","last_undegraded":"2024-09-11T08:30:22.149867+0000","last_fullsized":"2024-09-11T08:30:22.149867+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'15","last_scrub_stamp":"2024-09-11T08:30:09.325345+0000","last_deep_scrub":"376'15","last_deep_scrub_stamp":"2024-09-11T08:26:02.786767+0000","last_clean_scrub_stamp":"2024-09-11T08:30:09.325345+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:23.599574+0000","scrub_duration":0.001768979,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"66'1","reported_seq":476,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.908287+0000","last_change":"2024-09-11T08:30:07.307684+0000","last_active":"2024-09-11T08:30:16.908287+0000","last_peered":"2024-09-11T08:30:16.908287+0000","last_clean":"2024-09-11T08:30:16.908287+0000","last_became_active":"2024-09-11T08:07:36.072593+0000","last_became_peered":"2024-09-11T08:07:36.072593+0000","last_unstale":"2024-09-11T08:30:16.908287+0000","last_undegraded":"2024-09-11T08:30:16.908287+0000","last_fullsized":"2024-09-11T08:30:16.908287+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-11T08:30:07.307628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:30:07.307628+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:24.772623+0000","scrub_duration":0.0021248930000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"405'47","reported_seq":689,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.911090+0000","last_change":"2024-09-11T08:30:03.281050+0000","last_active":"2024-09-11T08:30:16.911090+0000","last_peered":"2024-09-11T08:30:16.911090+0000","last_clean":"2024-09-11T08:30:16.911090+0000","last_became_active":"2024-09-11T08:24:28.364043+0000","last_became_peered":"2024-09-11T08:24:28.364043+0000","last_unstale":"2024-09-11T08:30:16.911090+0000","last_undegraded":"2024-09-11T08:30:16.911090+0000","last_fullsized":"2024-09-11T08:30:16.911090+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'47","last_scrub_stamp":"2024-09-11T08:30:03.280942+0000","last_deep_scrub":"396'37","last_deep_scrub_stamp":"2024-09-11T08:27:16.673846+0000","last_clean_scrub_stamp":"2024-09-11T08:30:03.280942+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:25.945261+0000","scrub_duration":0.0024639179999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":485,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:17.049051+0000","last_change":"2024-09-11T08:29:57.672185+0000","last_active":"2024-09-11T08:30:17.049051+0000","last_peered":"2024-09-11T08:30:17.049051+0000","last_clean":"2024-09-11T08:30:17.049051+0000","last_became_active":"2024-09-11T08:27:57.795859+0000","last_became_peered":"2024-09-11T08:27:57.795859+0000","last_unstale":"2024-09-11T08:30:17.049051+0000","last_undegraded":"2024-09-11T08:30:17.049051+0000","last_fullsized":"2024-09-11T08:30:17.049051+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:57.672105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:28:03.117012+0000","last_clean_scrub_stamp":"2024-09-11T08:29:57.672105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:00.681043+0000","scrub_duration":0.011006079,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":541,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:17.048749+0000","last_change":"2024-09-11T08:29:24.232937+0000","last_active":"2024-09-11T08:30:17.048749+0000","last_peered":"2024-09-11T08:30:17.048749+0000","last_clean":"2024-09-11T08:30:17.048749+0000","last_became_active":"2024-09-11T08:24:28.360235+0000","last_became_peered":"2024-09-11T08:24:28.360235+0000","last_unstale":"2024-09-11T08:30:17.048749+0000","last_undegraded":"2024-09-11T08:30:17.048749+0000","last_fullsized":"2024-09-11T08:30:17.048749+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:24.232816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:29:24.232816+0000","last_clean_scrub_stamp":"2024-09-11T08:29:24.232816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:42.768241+0000","scrub_duration":0.0060421820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":492,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.902048+0000","last_change":"2024-09-11T08:27:56.203308+0000","last_active":"2024-09-11T08:30:16.902048+0000","last_peered":"2024-09-11T08:30:16.902048+0000","last_clean":"2024-09-11T08:30:16.902048+0000","last_became_active":"2024-09-11T08:07:36.631668+0000","last_became_peered":"2024-09-11T08:07:36.631668+0000","last_unstale":"2024-09-11T08:30:16.902048+0000","last_undegraded":"2024-09-11T08:30:16.902048+0000","last_fullsized":"2024-09-11T08:30:16.902048+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:56.203218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:58.517624+0000","last_clean_scrub_stamp":"2024-09-11T08:27:56.203218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010650626,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"405'64","reported_seq":358,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-09-11T08:30:23.848353+0000","last_change":"2024-09-11T08:28:00.205458+0000","last_active":"2024-09-11T08:30:23.848353+0000","last_peered":"2024-09-11T08:30:23.848353+0000","last_clean":"2024-09-11T08:30:23.848353+0000","last_became_active":"2024-09-11T08:20:52.086843+0000","last_became_peered":"2024-09-11T08:20:52.086843+0000","last_unstale":"2024-09-11T08:30:23.848353+0000","last_undegraded":"2024-09-11T08:30:23.848353+0000","last_fullsized":"2024-09-11T08:30:23.848353+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'64","last_scrub_stamp":"2024-09-11T08:28:00.205375+0000","last_deep_scrub":"190'39","last_deep_scrub_stamp":"2024-09-11T08:15:52.847968+0000","last_clean_scrub_stamp":"2024-09-11T08:28:00.205375+0000","objects_scrubbed":10,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0087966820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":84,"num_read_kb":69,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"59'1","reported_seq":152,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.911796+0000","last_change":"2024-09-11T08:28:53.832466+0000","last_active":"2024-09-11T08:30:16.911796+0000","last_peered":"2024-09-11T08:30:16.911796+0000","last_clean":"2024-09-11T08:30:16.911796+0000","last_became_active":"2024-09-11T08:23:22.691726+0000","last_became_peered":"2024-09-11T08:23:22.691726+0000","last_unstale":"2024-09-11T08:30:16.911796+0000","last_undegraded":"2024-09-11T08:30:16.911796+0000","last_fullsized":"2024-09-11T08:30:16.911796+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-11T08:28:53.832355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:28:53.832355+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0063587790000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":208,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.902428+0000","last_change":"2024-09-11T08:28:51.863387+0000","last_active":"2024-09-11T08:30:16.902428+0000","last_peered":"2024-09-11T08:30:16.902428+0000","last_clean":"2024-09-11T08:30:16.902428+0000","last_became_active":"2024-09-11T08:24:28.362590+0000","last_became_peered":"2024-09-11T08:24:28.362590+0000","last_unstale":"2024-09-11T08:30:16.902428+0000","last_undegraded":"2024-09-11T08:30:16.902428+0000","last_fullsized":"2024-09-11T08:30:16.902428+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-11T08:28:51.863309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:28:51.863309+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00679483,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":474,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.909970+0000","last_change":"2024-09-11T08:28:50.908738+0000","last_active":"2024-09-11T08:30:16.909970+0000","last_peered":"2024-09-11T08:30:16.909970+0000","last_clean":"2024-09-11T08:30:16.909970+0000","last_became_active":"2024-09-11T08:07:34.057154+0000","last_became_peered":"2024-09-11T08:07:34.057154+0000","last_unstale":"2024-09-11T08:30:16.909970+0000","last_undegraded":"2024-09-11T08:30:16.909970+0000","last_fullsized":"2024-09-11T08:30:16.909970+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:28:50.908650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:10:37.957096+0000","last_clean_scrub_stamp":"2024-09-11T08:28:50.908650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014372642,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":520,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:17.821084+0000","last_change":"2024-09-11T08:30:17.821049+0000","last_active":"2024-09-11T08:30:17.821084+0000","last_peered":"2024-09-11T08:30:17.821084+0000","last_clean":"2024-09-11T08:30:17.821084+0000","last_became_active":"2024-09-11T08:15:56.525099+0000","last_became_peered":"2024-09-11T08:15:56.525099+0000","last_unstale":"2024-09-11T08:30:17.821084+0000","last_undegraded":"2024-09-11T08:30:17.821084+0000","last_fullsized":"2024-09-11T08:30:17.821084+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:17.820997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:30:17.820997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:24.444375+0000","scrub_duration":0.0016474370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"405'21","reported_seq":605,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.906359+0000","last_change":"2024-09-11T08:30:16.906268+0000","last_active":"2024-09-11T08:30:16.906359+0000","last_peered":"2024-09-11T08:30:16.906359+0000","last_clean":"2024-09-11T08:30:16.906359+0000","last_became_active":"2024-09-11T08:24:28.358121+0000","last_became_peered":"2024-09-11T08:24:28.358121+0000","last_unstale":"2024-09-11T08:30:16.906359+0000","last_undegraded":"2024-09-11T08:30:16.906359+0000","last_fullsized":"2024-09-11T08:30:16.906359+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'21","last_scrub_stamp":"2024-09-11T08:30:16.906178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:30:16.906178+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:35.667760+0000","scrub_duration":0.0058613850000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":511,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.901767+0000","last_change":"2024-09-11T08:29:15.868218+0000","last_active":"2024-09-11T08:30:16.901767+0000","last_peered":"2024-09-11T08:30:16.901767+0000","last_clean":"2024-09-11T08:30:16.901767+0000","last_became_active":"2024-09-11T08:07:30.047014+0000","last_became_peered":"2024-09-11T08:07:30.047014+0000","last_unstale":"2024-09-11T08:30:16.901767+0000","last_undegraded":"2024-09-11T08:30:16.901767+0000","last_fullsized":"2024-09-11T08:30:16.901767+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:15.867962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:29:15.867962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:39.510619+0000","scrub_duration":0.0075403279999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"127'2","reported_seq":538,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:17.048347+0000","last_change":"2024-09-11T08:30:08.554358+0000","last_active":"2024-09-11T08:30:17.048347+0000","last_peered":"2024-09-11T08:30:17.048347+0000","last_clean":"2024-09-11T08:30:17.048347+0000","last_became_active":"2024-09-11T08:07:30.045009+0000","last_became_peered":"2024-09-11T08:07:30.045009+0000","last_unstale":"2024-09-11T08:30:17.048347+0000","last_undegraded":"2024-09-11T08:30:17.048347+0000","last_fullsized":"2024-09-11T08:30:17.048347+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'2","last_scrub_stamp":"2024-09-11T08:30:08.554296+0000","last_deep_scrub":"127'2","last_deep_scrub_stamp":"2024-09-11T08:15:01.938497+0000","last_clean_scrub_stamp":"2024-09-11T08:30:08.554296+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:21.485645+0000","scrub_duration":0.0016669009999999999,"objects_trimmed":0,"snaptrim_duration":0.001859533,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":564,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:17.048885+0000","last_change":"2024-09-11T08:30:00.586632+0000","last_active":"2024-09-11T08:30:17.048885+0000","last_peered":"2024-09-11T08:30:17.048885+0000","last_clean":"2024-09-11T08:30:17.048885+0000","last_became_active":"2024-09-11T08:24:28.357141+0000","last_became_peered":"2024-09-11T08:24:28.357141+0000","last_unstale":"2024-09-11T08:30:17.048885+0000","last_undegraded":"2024-09-11T08:30:17.048885+0000","last_fullsized":"2024-09-11T08:30:17.048885+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:00.586547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:22:00.200872+0000","last_clean_scrub_stamp":"2024-09-11T08:30:00.586547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:09.658987+0000","scrub_duration":0.0097564220000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":224,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:17.048563+0000","last_change":"2024-09-11T08:30:14.511521+0000","last_active":"2024-09-11T08:30:17.048563+0000","last_peered":"2024-09-11T08:30:17.048563+0000","last_clean":"2024-09-11T08:30:17.048563+0000","last_became_active":"2024-09-11T08:20:16.108674+0000","last_became_peered":"2024-09-11T08:20:16.108674+0000","last_unstale":"2024-09-11T08:30:17.048563+0000","last_undegraded":"2024-09-11T08:30:17.048563+0000","last_fullsized":"2024-09-11T08:30:17.048563+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:14.511468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:58.796217+0000","last_clean_scrub_stamp":"2024-09-11T08:30:14.511468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:18.157477+0000","scrub_duration":0.001290072,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"405'42","reported_seq":546,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.914976+0000","last_change":"2024-09-11T08:29:10.054827+0000","last_active":"2024-09-11T08:30:16.914976+0000","last_peered":"2024-09-11T08:30:16.914976+0000","last_clean":"2024-09-11T08:30:16.914976+0000","last_became_active":"2024-09-11T08:07:32.055407+0000","last_became_peered":"2024-09-11T08:07:32.055407+0000","last_unstale":"2024-09-11T08:30:16.914976+0000","last_undegraded":"2024-09-11T08:30:16.914976+0000","last_fullsized":"2024-09-11T08:30:16.914976+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'42","last_scrub_stamp":"2024-09-11T08:29:10.054727+0000","last_deep_scrub":"287'30","last_deep_scrub_stamp":"2024-09-11T08:26:09.030192+0000","last_clean_scrub_stamp":"2024-09-11T08:29:10.054727+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:34.247277+0000","scrub_duration":0.042708410000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"448'2","reported_seq":496,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:18.171446+0000","last_change":"2024-09-11T08:30:18.171381+0000","last_active":"2024-09-11T08:30:18.171446+0000","last_peered":"2024-09-11T08:30:18.171446+0000","last_clean":"2024-09-11T08:30:18.171446+0000","last_became_active":"2024-09-11T08:07:30.050504+0000","last_became_peered":"2024-09-11T08:07:30.050504+0000","last_unstale":"2024-09-11T08:30:18.171446+0000","last_undegraded":"2024-09-11T08:30:18.171446+0000","last_fullsized":"2024-09-11T08:30:18.171446+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'2","last_scrub_stamp":"2024-09-11T08:30:18.171322+0000","last_deep_scrub":"448'2","last_deep_scrub_stamp":"2024-09-11T08:30:18.171322+0000","last_clean_scrub_stamp":"2024-09-11T08:30:18.171322+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:41.608010+0000","scrub_duration":0.0023495109999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"224'6","reported_seq":1643,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-09-11T08:30:26.445704+0000","last_change":"2024-09-11T08:28:06.324532+0000","last_active":"2024-09-11T08:30:26.445704+0000","last_peered":"2024-09-11T08:30:26.445704+0000","last_clean":"2024-09-11T08:30:26.445704+0000","last_became_active":"2024-09-11T08:14:24.793048+0000","last_became_peered":"2024-09-11T08:14:24.793048+0000","last_unstale":"2024-09-11T08:30:26.445704+0000","last_undegraded":"2024-09-11T08:30:26.445704+0000","last_fullsized":"2024-09-11T08:30:26.445704+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'6","last_scrub_stamp":"2024-09-11T08:28:06.324370+0000","last_deep_scrub":"224'6","last_deep_scrub_stamp":"2024-09-11T08:16:55.778284+0000","last_clean_scrub_stamp":"2024-09-11T08:28:06.324370+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036467399999999999,"objects_trimmed":0,"snaptrim_duration":0.000318821,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":968,"num_read_kb":963,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":201,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:20.838533+0000","last_change":"2024-09-11T08:30:20.838498+0000","last_active":"2024-09-11T08:30:20.838533+0000","last_peered":"2024-09-11T08:30:20.838533+0000","last_clean":"2024-09-11T08:30:20.838533+0000","last_became_active":"2024-09-11T08:24:28.353018+0000","last_became_peered":"2024-09-11T08:24:28.353018+0000","last_unstale":"2024-09-11T08:30:20.838533+0000","last_undegraded":"2024-09-11T08:30:20.838533+0000","last_fullsized":"2024-09-11T08:30:20.838533+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:20.838437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:30:20.838437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:23.467312+0000","scrub_duration":0.001681231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":545,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:21.789063+0000","last_change":"2024-09-11T08:30:21.789030+0000","last_active":"2024-09-11T08:30:21.789063+0000","last_peered":"2024-09-11T08:30:21.789063+0000","last_clean":"2024-09-11T08:30:21.789063+0000","last_became_active":"2024-09-11T08:24:28.358723+0000","last_became_peered":"2024-09-11T08:24:28.358723+0000","last_unstale":"2024-09-11T08:30:21.789063+0000","last_undegraded":"2024-09-11T08:30:21.789063+0000","last_fullsized":"2024-09-11T08:30:21.789063+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:21.788964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:30:21.788964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:29.292249+0000","scrub_duration":0.0016136849999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"405'45","reported_seq":588,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:17.051398+0000","last_change":"2024-09-11T08:29:13.340969+0000","last_active":"2024-09-11T08:30:17.051398+0000","last_peered":"2024-09-11T08:30:17.051398+0000","last_clean":"2024-09-11T08:30:17.051398+0000","last_became_active":"2024-09-11T08:14:24.789154+0000","last_became_peered":"2024-09-11T08:14:24.789154+0000","last_unstale":"2024-09-11T08:30:17.051398+0000","last_undegraded":"2024-09-11T08:30:17.051398+0000","last_fullsized":"2024-09-11T08:30:17.051398+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'45","last_scrub_stamp":"2024-09-11T08:29:13.340918+0000","last_deep_scrub":"84'21","last_deep_scrub_stamp":"2024-09-11T08:08:56.705306+0000","last_clean_scrub_stamp":"2024-09-11T08:29:13.340918+0000","objects_scrubbed":7,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:34.815204+0000","scrub_duration":0.001693937,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":203,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:17.050771+0000","last_change":"2024-09-11T08:30:16.289806+0000","last_active":"2024-09-11T08:30:17.050771+0000","last_peered":"2024-09-11T08:30:17.050771+0000","last_clean":"2024-09-11T08:30:17.050771+0000","last_became_active":"2024-09-11T08:24:28.362417+0000","last_became_peered":"2024-09-11T08:24:28.362417+0000","last_unstale":"2024-09-11T08:30:17.050771+0000","last_undegraded":"2024-09-11T08:30:17.050771+0000","last_fullsized":"2024-09-11T08:30:17.050771+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:16.289709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:07.749828+0000","last_clean_scrub_stamp":"2024-09-11T08:30:16.289709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:43.152596+0000","scrub_duration":0.0013265880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":307,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.861013+0000","last_change":"2024-09-11T08:28:01.294768+0000","last_active":"2024-09-11T08:30:16.861013+0000","last_peered":"2024-09-11T08:30:16.861013+0000","last_clean":"2024-09-11T08:30:16.861013+0000","last_became_active":"2024-09-11T08:15:56.523430+0000","last_became_peered":"2024-09-11T08:15:56.523430+0000","last_unstale":"2024-09-11T08:30:16.861013+0000","last_undegraded":"2024-09-11T08:30:16.861013+0000","last_fullsized":"2024-09-11T08:30:16.861013+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-11T08:28:01.294597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:05:21.231856+0000","last_clean_scrub_stamp":"2024-09-11T08:28:01.294597+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027233790000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":331,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.863920+0000","last_change":"2024-09-11T08:27:49.296146+0000","last_active":"2024-09-11T08:30:16.863920+0000","last_peered":"2024-09-11T08:30:16.863920+0000","last_clean":"2024-09-11T08:30:16.863920+0000","last_became_active":"2024-09-11T08:24:28.357166+0000","last_became_peered":"2024-09-11T08:24:28.357166+0000","last_unstale":"2024-09-11T08:30:16.863920+0000","last_undegraded":"2024-09-11T08:30:16.863920+0000","last_fullsized":"2024-09-11T08:30:16.863920+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:49.296063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:20:25.775869+0000","last_clean_scrub_stamp":"2024-09-11T08:27:49.296063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037448009999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":258,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:18.358825+0000","last_change":"2024-09-11T08:30:18.358789+0000","last_active":"2024-09-11T08:30:18.358825+0000","last_peered":"2024-09-11T08:30:18.358825+0000","last_clean":"2024-09-11T08:30:18.358825+0000","last_became_active":"2024-09-11T08:24:28.356071+0000","last_became_peered":"2024-09-11T08:24:28.356071+0000","last_unstale":"2024-09-11T08:30:18.358825+0000","last_undegraded":"2024-09-11T08:30:18.358825+0000","last_fullsized":"2024-09-11T08:30:18.358825+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:18.358735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:01.777429+0000","last_clean_scrub_stamp":"2024-09-11T08:30:18.358735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:32.931299+0000","scrub_duration":0.001168838,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"405'32","reported_seq":434,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:17.049852+0000","last_change":"2024-09-11T08:29:10.333244+0000","last_active":"2024-09-11T08:30:17.049852+0000","last_peered":"2024-09-11T08:30:17.049852+0000","last_clean":"2024-09-11T08:30:17.049852+0000","last_became_active":"2024-09-11T08:24:28.368759+0000","last_became_peered":"2024-09-11T08:24:28.368759+0000","last_unstale":"2024-09-11T08:30:17.049852+0000","last_undegraded":"2024-09-11T08:30:17.049852+0000","last_fullsized":"2024-09-11T08:30:17.049852+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'32","last_scrub_stamp":"2024-09-11T08:29:10.333163+0000","last_deep_scrub":"405'32","last_deep_scrub_stamp":"2024-09-11T08:29:10.333163+0000","last_clean_scrub_stamp":"2024-09-11T08:29:10.333163+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:32.092567+0000","scrub_duration":0.0057426860000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":525,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:17.048741+0000","last_change":"2024-09-11T08:30:07.260981+0000","last_active":"2024-09-11T08:30:17.048741+0000","last_peered":"2024-09-11T08:30:17.048741+0000","last_clean":"2024-09-11T08:30:17.048741+0000","last_became_active":"2024-09-11T08:14:24.789888+0000","last_became_peered":"2024-09-11T08:14:24.789888+0000","last_unstale":"2024-09-11T08:30:17.048741+0000","last_undegraded":"2024-09-11T08:30:17.048741+0000","last_fullsized":"2024-09-11T08:30:17.048741+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:07.260906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:36.417911+0000","last_clean_scrub_stamp":"2024-09-11T08:30:07.260906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:31.737712+0000","scrub_duration":0.0037043279999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":195,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:17.050078+0000","last_change":"2024-09-11T08:30:06.443515+0000","last_active":"2024-09-11T08:30:17.050078+0000","last_peered":"2024-09-11T08:30:17.050078+0000","last_clean":"2024-09-11T08:30:17.050078+0000","last_became_active":"2024-09-11T08:24:28.361261+0000","last_became_peered":"2024-09-11T08:24:28.361261+0000","last_unstale":"2024-09-11T08:30:17.050078+0000","last_undegraded":"2024-09-11T08:30:17.050078+0000","last_fullsized":"2024-09-11T08:30:17.050078+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:06.443433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:01.558340+0000","last_clean_scrub_stamp":"2024-09-11T08:30:06.443433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:23.309677+0000","scrub_duration":0.001552765,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":323,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:17.049606+0000","last_change":"2024-09-11T08:29:16.340861+0000","last_active":"2024-09-11T08:30:17.049606+0000","last_peered":"2024-09-11T08:30:17.049606+0000","last_clean":"2024-09-11T08:30:17.049606+0000","last_became_active":"2024-09-11T08:24:28.357842+0000","last_became_peered":"2024-09-11T08:24:28.357842+0000","last_unstale":"2024-09-11T08:30:17.049606+0000","last_undegraded":"2024-09-11T08:30:17.049606+0000","last_fullsized":"2024-09-11T08:30:17.049606+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:16.340778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:29:16.340778+0000","last_clean_scrub_stamp":"2024-09-11T08:29:16.340778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:43.003239+0000","scrub_duration":0.0019631459999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"405'53","reported_seq":495,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:17.048814+0000","last_change":"2024-09-11T08:30:02.596837+0000","last_active":"2024-09-11T08:30:17.048814+0000","last_peered":"2024-09-11T08:30:17.048814+0000","last_clean":"2024-09-11T08:30:17.048814+0000","last_became_active":"2024-09-11T08:23:22.681452+0000","last_became_peered":"2024-09-11T08:23:22.681452+0000","last_unstale":"2024-09-11T08:30:17.048814+0000","last_undegraded":"2024-09-11T08:30:17.048814+0000","last_fullsized":"2024-09-11T08:30:17.048814+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'53","last_scrub_stamp":"2024-09-11T08:30:02.596714+0000","last_deep_scrub":"77'25","last_deep_scrub_stamp":"2024-09-11T08:08:32.458210+0000","last_clean_scrub_stamp":"2024-09-11T08:30:02.596714+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:30.319012+0000","scrub_duration":0.0038267459999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":514,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:17.048931+0000","last_change":"2024-09-11T08:29:57.622728+0000","last_active":"2024-09-11T08:30:17.048931+0000","last_peered":"2024-09-11T08:30:17.048931+0000","last_clean":"2024-09-11T08:30:17.048931+0000","last_became_active":"2024-09-11T08:07:30.042555+0000","last_became_peered":"2024-09-11T08:07:30.042555+0000","last_unstale":"2024-09-11T08:30:17.048931+0000","last_undegraded":"2024-09-11T08:30:17.048931+0000","last_fullsized":"2024-09-11T08:30:17.048931+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:57.622675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:29:57.622675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:21.379775+0000","scrub_duration":0.0036282020000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":494,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.902172+0000","last_change":"2024-09-11T08:27:49.130976+0000","last_active":"2024-09-11T08:30:16.902172+0000","last_peered":"2024-09-11T08:30:16.902172+0000","last_clean":"2024-09-11T08:30:16.902172+0000","last_became_active":"2024-09-11T08:07:34.054961+0000","last_became_peered":"2024-09-11T08:07:34.054961+0000","last_unstale":"2024-09-11T08:30:16.902172+0000","last_undegraded":"2024-09-11T08:30:16.902172+0000","last_fullsized":"2024-09-11T08:30:16.902172+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:49.130900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:27:49.130900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004734121,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":522,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:17.049541+0000","last_change":"2024-09-11T08:29:56.700851+0000","last_active":"2024-09-11T08:30:17.049541+0000","last_peered":"2024-09-11T08:30:17.049541+0000","last_clean":"2024-09-11T08:30:17.049541+0000","last_became_active":"2024-09-11T08:07:36.071194+0000","last_became_peered":"2024-09-11T08:07:36.071194+0000","last_unstale":"2024-09-11T08:30:17.049541+0000","last_undegraded":"2024-09-11T08:30:17.049541+0000","last_fullsized":"2024-09-11T08:30:17.049541+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:56.700632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:07.930137+0000","last_clean_scrub_stamp":"2024-09-11T08:29:56.700632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:21.407208+0000","scrub_duration":0.0045560790000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"405'35","reported_seq":429,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:17.048652+0000","last_change":"2024-09-11T08:30:01.519867+0000","last_active":"2024-09-11T08:30:17.048652+0000","last_peered":"2024-09-11T08:30:17.048652+0000","last_clean":"2024-09-11T08:30:17.048652+0000","last_became_active":"2024-09-11T08:23:22.688631+0000","last_became_peered":"2024-09-11T08:23:22.688631+0000","last_unstale":"2024-09-11T08:30:17.048652+0000","last_undegraded":"2024-09-11T08:30:17.048652+0000","last_fullsized":"2024-09-11T08:30:17.048652+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'35","last_scrub_stamp":"2024-09-11T08:30:01.519776+0000","last_deep_scrub":"224'15","last_deep_scrub_stamp":"2024-09-11T08:17:01.126478+0000","last_clean_scrub_stamp":"2024-09-11T08:30:01.519776+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:15.125283+0000","scrub_duration":0.0073496450000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":273,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:17.198508+0000","last_change":"2024-09-11T08:30:17.198458+0000","last_active":"2024-09-11T08:30:17.198508+0000","last_peered":"2024-09-11T08:30:17.198508+0000","last_clean":"2024-09-11T08:30:17.198508+0000","last_became_active":"2024-09-11T08:17:37.557278+0000","last_became_peered":"2024-09-11T08:17:37.557278+0000","last_unstale":"2024-09-11T08:30:17.198508+0000","last_undegraded":"2024-09-11T08:30:17.198508+0000","last_fullsized":"2024-09-11T08:30:17.198508+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:17.198403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:27:59.866811+0000","last_clean_scrub_stamp":"2024-09-11T08:30:17.198403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:41.884498+0000","scrub_duration":0.0012428210000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"354'16","reported_seq":858,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-09-11T08:30:22.149831+0000","last_change":"2024-09-11T08:29:06.507763+0000","last_active":"2024-09-11T08:30:22.149831+0000","last_peered":"2024-09-11T08:30:22.149831+0000","last_clean":"2024-09-11T08:30:22.149831+0000","last_became_active":"2024-09-11T08:24:28.356749+0000","last_became_peered":"2024-09-11T08:24:28.356749+0000","last_unstale":"2024-09-11T08:30:22.149831+0000","last_undegraded":"2024-09-11T08:30:22.149831+0000","last_fullsized":"2024-09-11T08:30:22.149831+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'16","last_scrub_stamp":"2024-09-11T08:29:06.507663+0000","last_deep_scrub":"171'12","last_deep_scrub_stamp":"2024-09-11T08:14:56.881299+0000","last_clean_scrub_stamp":"2024-09-11T08:29:06.507663+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:33.390159+0000","scrub_duration":0.015460094000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":153,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.903243+0000","last_change":"2024-09-11T08:29:14.863576+0000","last_active":"2024-09-11T08:30:16.903243+0000","last_peered":"2024-09-11T08:30:16.903243+0000","last_clean":"2024-09-11T08:30:16.903243+0000","last_became_active":"2024-09-11T08:23:22.686992+0000","last_became_peered":"2024-09-11T08:23:22.686992+0000","last_unstale":"2024-09-11T08:30:16.903243+0000","last_undegraded":"2024-09-11T08:30:16.903243+0000","last_fullsized":"2024-09-11T08:30:16.903243+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:14.863324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:14.613532+0000","last_clean_scrub_stamp":"2024-09-11T08:29:14.863324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:30.823910+0000","scrub_duration":0.0069445239999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"405'21","reported_seq":510,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.911000+0000","last_change":"2024-09-11T08:28:02.865865+0000","last_active":"2024-09-11T08:30:16.911000+0000","last_peered":"2024-09-11T08:30:16.911000+0000","last_clean":"2024-09-11T08:30:16.911000+0000","last_became_active":"2024-09-11T08:07:32.054885+0000","last_became_peered":"2024-09-11T08:07:32.054885+0000","last_unstale":"2024-09-11T08:30:16.911000+0000","last_undegraded":"2024-09-11T08:30:16.911000+0000","last_fullsized":"2024-09-11T08:30:16.911000+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'21","last_scrub_stamp":"2024-09-11T08:28:02.865748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:28:02.865748+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.016060689,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":199,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.862661+0000","last_change":"2024-09-11T08:28:54.875865+0000","last_active":"2024-09-11T08:30:16.862661+0000","last_peered":"2024-09-11T08:30:16.862661+0000","last_clean":"2024-09-11T08:30:16.862661+0000","last_became_active":"2024-09-11T08:24:28.359057+0000","last_became_peered":"2024-09-11T08:24:28.359057+0000","last_unstale":"2024-09-11T08:30:16.862661+0000","last_undegraded":"2024-09-11T08:30:16.862661+0000","last_fullsized":"2024-09-11T08:30:16.862661+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:28:54.875776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:28:54.875776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022324210000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":534,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:17.048346+0000","last_change":"2024-09-11T08:30:09.510347+0000","last_active":"2024-09-11T08:30:17.048346+0000","last_peered":"2024-09-11T08:30:17.048346+0000","last_clean":"2024-09-11T08:30:17.048346+0000","last_became_active":"2024-09-11T08:07:34.056606+0000","last_became_peered":"2024-09-11T08:07:34.056606+0000","last_unstale":"2024-09-11T08:30:17.048346+0000","last_undegraded":"2024-09-11T08:30:17.048346+0000","last_fullsized":"2024-09-11T08:30:17.048346+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:09.510248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:30:09.510248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:16.060314+0000","scrub_duration":0.0016592110000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":499,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.925445+0000","last_change":"2024-09-11T08:28:46.893924+0000","last_active":"2024-09-11T08:30:16.925445+0000","last_peered":"2024-09-11T08:30:16.925445+0000","last_clean":"2024-09-11T08:30:16.925445+0000","last_became_active":"2024-09-11T08:15:56.518884+0000","last_became_peered":"2024-09-11T08:15:56.518884+0000","last_unstale":"2024-09-11T08:30:16.925445+0000","last_undegraded":"2024-09-11T08:30:16.925445+0000","last_fullsized":"2024-09-11T08:30:16.925445+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:28:46.893851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:18:32.278257+0000","last_clean_scrub_stamp":"2024-09-11T08:28:46.893851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0085054269999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"405'51","reported_seq":662,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:17.050703+0000","last_change":"2024-09-11T08:29:19.302174+0000","last_active":"2024-09-11T08:30:17.050703+0000","last_peered":"2024-09-11T08:30:17.050703+0000","last_clean":"2024-09-11T08:30:17.050703+0000","last_became_active":"2024-09-11T08:15:56.526961+0000","last_became_peered":"2024-09-11T08:15:56.526961+0000","last_unstale":"2024-09-11T08:30:17.050703+0000","last_undegraded":"2024-09-11T08:30:17.050703+0000","last_fullsized":"2024-09-11T08:30:17.050703+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'51","last_scrub_stamp":"2024-09-11T08:29:19.302094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:29:19.302094+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:24.531066+0000","scrub_duration":0.0045766959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":341,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.870589+0000","last_change":"2024-09-11T08:30:10.617869+0000","last_active":"2024-09-11T08:30:16.870589+0000","last_peered":"2024-09-11T08:30:16.870589+0000","last_clean":"2024-09-11T08:30:16.870589+0000","last_became_active":"2024-09-11T08:24:28.353720+0000","last_became_peered":"2024-09-11T08:24:28.353720+0000","last_unstale":"2024-09-11T08:30:16.870589+0000","last_undegraded":"2024-09-11T08:30:16.870589+0000","last_fullsized":"2024-09-11T08:30:16.870589+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:10.617803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:44.385372+0000","last_clean_scrub_stamp":"2024-09-11T08:30:10.617803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:30.349657+0000","scrub_duration":0.0015332709999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":554,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:17.048405+0000","last_change":"2024-09-11T08:29:11.612083+0000","last_active":"2024-09-11T08:30:17.048405+0000","last_peered":"2024-09-11T08:30:17.048405+0000","last_clean":"2024-09-11T08:30:17.048405+0000","last_became_active":"2024-09-11T08:24:28.365792+0000","last_became_peered":"2024-09-11T08:24:28.365792+0000","last_unstale":"2024-09-11T08:30:17.048405+0000","last_undegraded":"2024-09-11T08:30:17.048405+0000","last_fullsized":"2024-09-11T08:30:17.048405+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:11.611140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:06.546573+0000","last_clean_scrub_stamp":"2024-09-11T08:29:11.611140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:30.173424+0000","scrub_duration":0.006733915,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":301,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.861094+0000","last_change":"2024-09-11T08:28:55.493273+0000","last_active":"2024-09-11T08:30:16.861094+0000","last_peered":"2024-09-11T08:30:16.861094+0000","last_clean":"2024-09-11T08:30:16.861094+0000","last_became_active":"2024-09-11T08:15:56.518169+0000","last_became_peered":"2024-09-11T08:15:56.518169+0000","last_unstale":"2024-09-11T08:30:16.861094+0000","last_undegraded":"2024-09-11T08:30:16.861094+0000","last_fullsized":"2024-09-11T08:30:16.861094+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:28:55.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:28:55.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0069241010000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"405'29","reported_seq":414,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.902122+0000","last_change":"2024-09-11T08:27:22.427676+0000","last_active":"2024-09-11T08:30:16.902122+0000","last_peered":"2024-09-11T08:30:16.902122+0000","last_clean":"2024-09-11T08:30:16.902122+0000","last_became_active":"2024-09-11T08:23:22.686387+0000","last_became_peered":"2024-09-11T08:23:22.686387+0000","last_unstale":"2024-09-11T08:30:16.902122+0000","last_undegraded":"2024-09-11T08:30:16.902122+0000","last_fullsized":"2024-09-11T08:30:16.902122+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'23","last_scrub_stamp":"2024-09-11T08:27:22.427580+0000","last_deep_scrub":"145'17","last_deep_scrub_stamp":"2024-09-11T08:13:08.160848+0000","last_clean_scrub_stamp":"2024-09-11T08:27:22.427580+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.17013154699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":311,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.872088+0000","last_change":"2024-09-11T08:29:02.433750+0000","last_active":"2024-09-11T08:30:16.872088+0000","last_peered":"2024-09-11T08:30:16.872088+0000","last_clean":"2024-09-11T08:30:16.872088+0000","last_became_active":"2024-09-11T08:15:56.525845+0000","last_became_peered":"2024-09-11T08:15:56.525845+0000","last_unstale":"2024-09-11T08:30:16.872088+0000","last_undegraded":"2024-09-11T08:30:16.872088+0000","last_fullsized":"2024-09-11T08:30:16.872088+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:02.433672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:21:06.262572+0000","last_clean_scrub_stamp":"2024-09-11T08:29:02.433672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0082380349999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":514,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:18.591738+0000","last_change":"2024-09-11T08:30:18.591703+0000","last_active":"2024-09-11T08:30:18.591738+0000","last_peered":"2024-09-11T08:30:18.591738+0000","last_clean":"2024-09-11T08:30:18.591738+0000","last_became_active":"2024-09-11T08:07:34.057479+0000","last_became_peered":"2024-09-11T08:07:34.057479+0000","last_unstale":"2024-09-11T08:30:18.591738+0000","last_undegraded":"2024-09-11T08:30:18.591738+0000","last_fullsized":"2024-09-11T08:30:18.591738+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:18.591649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:30:18.591649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:42.108591+0000","scrub_duration":0.0013128949999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":520,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.864116+0000","last_change":"2024-09-11T08:29:01.755692+0000","last_active":"2024-09-11T08:30:16.864116+0000","last_peered":"2024-09-11T08:30:16.864116+0000","last_clean":"2024-09-11T08:30:16.864116+0000","last_became_active":"2024-09-11T08:15:56.522624+0000","last_became_peered":"2024-09-11T08:15:56.522624+0000","last_unstale":"2024-09-11T08:30:16.864116+0000","last_undegraded":"2024-09-11T08:30:16.864116+0000","last_fullsized":"2024-09-11T08:30:16.864116+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:01.755554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:29:01.755554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037540439999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"405'39","reported_seq":562,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.860870+0000","last_change":"2024-09-11T08:30:11.596345+0000","last_active":"2024-09-11T08:30:16.860870+0000","last_peered":"2024-09-11T08:30:16.860870+0000","last_clean":"2024-09-11T08:30:16.860870+0000","last_became_active":"2024-09-11T08:24:28.353397+0000","last_became_peered":"2024-09-11T08:24:28.353397+0000","last_unstale":"2024-09-11T08:30:16.860870+0000","last_undegraded":"2024-09-11T08:30:16.860870+0000","last_fullsized":"2024-09-11T08:30:16.860870+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'39","last_scrub_stamp":"2024-09-11T08:30:11.596278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:30:11.596278+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:28.358869+0000","scrub_duration":0.0031679740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":307,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.863823+0000","last_change":"2024-09-11T08:28:45.319406+0000","last_active":"2024-09-11T08:30:16.863823+0000","last_peered":"2024-09-11T08:30:16.863823+0000","last_clean":"2024-09-11T08:30:16.863823+0000","last_became_active":"2024-09-11T08:15:56.515328+0000","last_became_peered":"2024-09-11T08:15:56.515328+0000","last_unstale":"2024-09-11T08:30:16.863823+0000","last_undegraded":"2024-09-11T08:30:16.863823+0000","last_fullsized":"2024-09-11T08:30:16.863823+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-11T08:28:45.319230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:28:45.319230+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0062126519999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"209'11","reported_seq":479,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-09-11T08:30:21.983538+0000","last_change":"2024-09-11T08:29:04.443432+0000","last_active":"2024-09-11T08:30:21.983538+0000","last_peered":"2024-09-11T08:30:21.983538+0000","last_clean":"2024-09-11T08:30:21.983538+0000","last_became_active":"2024-09-11T08:15:56.524513+0000","last_became_peered":"2024-09-11T08:15:56.524513+0000","last_unstale":"2024-09-11T08:30:21.983538+0000","last_undegraded":"2024-09-11T08:30:21.983538+0000","last_fullsized":"2024-09-11T08:30:21.983538+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"209'11","last_scrub_stamp":"2024-09-11T08:29:04.443335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:29:04.443335+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041068340000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":199,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:17.048668+0000","last_change":"2024-09-11T08:30:14.310713+0000","last_active":"2024-09-11T08:30:17.048668+0000","last_peered":"2024-09-11T08:30:17.048668+0000","last_clean":"2024-09-11T08:30:17.048668+0000","last_became_active":"2024-09-11T08:24:28.360658+0000","last_became_peered":"2024-09-11T08:24:28.360658+0000","last_unstale":"2024-09-11T08:30:17.048668+0000","last_undegraded":"2024-09-11T08:30:17.048668+0000","last_fullsized":"2024-09-11T08:30:17.048668+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:14.310661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:03.484755+0000","last_clean_scrub_stamp":"2024-09-11T08:30:14.310661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:19.401197+0000","scrub_duration":0.0012596440000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":548,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:17.056277+0000","last_change":"2024-09-11T08:29:17.333213+0000","last_active":"2024-09-11T08:30:17.056277+0000","last_peered":"2024-09-11T08:30:17.056277+0000","last_clean":"2024-09-11T08:30:17.056277+0000","last_became_active":"2024-09-11T08:24:28.357401+0000","last_became_peered":"2024-09-11T08:24:28.357401+0000","last_unstale":"2024-09-11T08:30:17.056277+0000","last_undegraded":"2024-09-11T08:30:17.056277+0000","last_fullsized":"2024-09-11T08:30:17.056277+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:17.333163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:29:17.333163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:32.873535+0000","scrub_duration":0.0015736540000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"405'21","reported_seq":570,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.862048+0000","last_change":"2024-09-11T08:29:15.625135+0000","last_active":"2024-09-11T08:30:16.862048+0000","last_peered":"2024-09-11T08:30:16.862048+0000","last_clean":"2024-09-11T08:30:16.862048+0000","last_became_active":"2024-09-11T08:15:56.526325+0000","last_became_peered":"2024-09-11T08:15:56.526325+0000","last_unstale":"2024-09-11T08:30:16.862048+0000","last_undegraded":"2024-09-11T08:30:16.862048+0000","last_fullsized":"2024-09-11T08:30:16.862048+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'21","last_scrub_stamp":"2024-09-11T08:29:15.624982+0000","last_deep_scrub":"405'21","last_deep_scrub_stamp":"2024-09-11T08:29:15.624982+0000","last_clean_scrub_stamp":"2024-09-11T08:29:15.624982+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:33.209933+0000","scrub_duration":0.010445571000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":199,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.862750+0000","last_change":"2024-09-11T08:30:12.778506+0000","last_active":"2024-09-11T08:30:16.862750+0000","last_peered":"2024-09-11T08:30:16.862750+0000","last_clean":"2024-09-11T08:30:16.862750+0000","last_became_active":"2024-09-11T08:24:28.358300+0000","last_became_peered":"2024-09-11T08:24:28.358300+0000","last_unstale":"2024-09-11T08:30:16.862750+0000","last_undegraded":"2024-09-11T08:30:16.862750+0000","last_fullsized":"2024-09-11T08:30:16.862750+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:12.778451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:53.548967+0000","last_clean_scrub_stamp":"2024-09-11T08:30:12.778451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:30.240863+0000","scrub_duration":0.001203455,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":436,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.870739+0000","last_change":"2024-09-11T08:29:20.429043+0000","last_active":"2024-09-11T08:30:16.870739+0000","last_peered":"2024-09-11T08:30:16.870739+0000","last_clean":"2024-09-11T08:30:16.870739+0000","last_became_active":"2024-09-11T08:14:24.790678+0000","last_became_peered":"2024-09-11T08:14:24.790678+0000","last_unstale":"2024-09-11T08:30:16.870739+0000","last_undegraded":"2024-09-11T08:30:16.870739+0000","last_fullsized":"2024-09-11T08:30:16.870739+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:20.428966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:29:20.428966+0000","last_clean_scrub_stamp":"2024-09-11T08:29:20.428966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:46.351908+0000","scrub_duration":0.005613697,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"405'25","reported_seq":561,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.915061+0000","last_change":"2024-09-11T08:30:09.312537+0000","last_active":"2024-09-11T08:30:16.915061+0000","last_peered":"2024-09-11T08:30:16.915061+0000","last_clean":"2024-09-11T08:30:16.915061+0000","last_became_active":"2024-09-11T08:15:56.514793+0000","last_became_peered":"2024-09-11T08:15:56.514793+0000","last_unstale":"2024-09-11T08:30:16.915061+0000","last_undegraded":"2024-09-11T08:30:16.915061+0000","last_fullsized":"2024-09-11T08:30:16.915061+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'25","last_scrub_stamp":"2024-09-11T08:30:09.312479+0000","last_deep_scrub":"84'13","last_deep_scrub_stamp":"2024-09-11T08:08:57.835634+0000","last_clean_scrub_stamp":"2024-09-11T08:30:09.312479+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:23.223570+0000","scrub_duration":0.0022105110000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":459,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.861315+0000","last_change":"2024-09-11T08:28:56.500137+0000","last_active":"2024-09-11T08:30:16.861315+0000","last_peered":"2024-09-11T08:30:16.861315+0000","last_clean":"2024-09-11T08:30:16.861315+0000","last_became_active":"2024-09-11T08:14:24.786021+0000","last_became_peered":"2024-09-11T08:14:24.786021+0000","last_unstale":"2024-09-11T08:30:16.861315+0000","last_undegraded":"2024-09-11T08:30:16.861315+0000","last_fullsized":"2024-09-11T08:30:16.861315+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:28:56.500057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:30.471620+0000","last_clean_scrub_stamp":"2024-09-11T08:28:56.500057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:23.579349+0000","scrub_duration":0.0062781299999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":522,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.861907+0000","last_change":"2024-09-11T08:29:57.839609+0000","last_active":"2024-09-11T08:30:16.861907+0000","last_peered":"2024-09-11T08:30:16.861907+0000","last_clean":"2024-09-11T08:30:16.861907+0000","last_became_active":"2024-09-11T08:15:56.525717+0000","last_became_peered":"2024-09-11T08:15:56.525717+0000","last_unstale":"2024-09-11T08:30:16.861907+0000","last_undegraded":"2024-09-11T08:30:16.861907+0000","last_fullsized":"2024-09-11T08:30:16.861907+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:57.839509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:04.370734+0000","last_clean_scrub_stamp":"2024-09-11T08:29:57.839509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:23.592176+0000","scrub_duration":0.0070850310000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":522,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:17.048481+0000","last_change":"2024-09-11T08:29:12.626802+0000","last_active":"2024-09-11T08:30:17.048481+0000","last_peered":"2024-09-11T08:30:17.048481+0000","last_clean":"2024-09-11T08:30:17.048481+0000","last_became_active":"2024-09-11T08:15:56.525120+0000","last_became_peered":"2024-09-11T08:15:56.525120+0000","last_unstale":"2024-09-11T08:30:17.048481+0000","last_undegraded":"2024-09-11T08:30:17.048481+0000","last_fullsized":"2024-09-11T08:30:17.048481+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:12.626731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:54.184372+0000","last_clean_scrub_stamp":"2024-09-11T08:29:12.626731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:34.316820+0000","scrub_duration":0.0092815109999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"405'25","reported_seq":247,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.863904+0000","last_change":"2024-09-11T08:30:05.609094+0000","last_active":"2024-09-11T08:30:16.863904+0000","last_peered":"2024-09-11T08:30:16.863904+0000","last_clean":"2024-09-11T08:30:16.863904+0000","last_became_active":"2024-09-11T08:24:28.353043+0000","last_became_peered":"2024-09-11T08:24:28.353043+0000","last_unstale":"2024-09-11T08:30:16.863904+0000","last_undegraded":"2024-09-11T08:30:16.863904+0000","last_fullsized":"2024-09-11T08:30:16.863904+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'25","last_scrub_stamp":"2024-09-11T08:30:05.609036+0000","last_deep_scrub":"313'19","last_deep_scrub_stamp":"2024-09-11T08:22:03.458402+0000","last_clean_scrub_stamp":"2024-09-11T08:30:05.609036+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:25.904857+0000","scrub_duration":0.0019394670000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":515,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.909511+0000","last_change":"2024-09-11T08:30:10.269699+0000","last_active":"2024-09-11T08:30:16.909511+0000","last_peered":"2024-09-11T08:30:16.909511+0000","last_clean":"2024-09-11T08:30:16.909511+0000","last_became_active":"2024-09-11T08:15:56.513278+0000","last_became_peered":"2024-09-11T08:15:56.513278+0000","last_unstale":"2024-09-11T08:30:16.909511+0000","last_undegraded":"2024-09-11T08:30:16.909511+0000","last_fullsized":"2024-09-11T08:30:16.909511+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:10.269641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:52.661591+0000","last_clean_scrub_stamp":"2024-09-11T08:30:10.269641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:31.914817+0000","scrub_duration":0.001664865,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"239'13","reported_seq":776,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-09-11T08:30:21.984115+0000","last_change":"2024-09-11T08:27:58.219794+0000","last_active":"2024-09-11T08:30:21.984115+0000","last_peered":"2024-09-11T08:30:21.984115+0000","last_clean":"2024-09-11T08:30:21.984115+0000","last_became_active":"2024-09-11T08:17:37.558224+0000","last_became_peered":"2024-09-11T08:17:37.558224+0000","last_unstale":"2024-09-11T08:30:21.984115+0000","last_undegraded":"2024-09-11T08:30:21.984115+0000","last_fullsized":"2024-09-11T08:30:21.984115+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'13","last_scrub_stamp":"2024-09-11T08:27:58.219700+0000","last_deep_scrub":"222'12","last_deep_scrub_stamp":"2024-09-11T08:16:52.592856+0000","last_clean_scrub_stamp":"2024-09-11T08:27:58.219700+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.01448953,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":437,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:17.049430+0000","last_change":"2024-09-11T08:29:58.653531+0000","last_active":"2024-09-11T08:30:17.049430+0000","last_peered":"2024-09-11T08:30:17.049430+0000","last_clean":"2024-09-11T08:30:17.049430+0000","last_became_active":"2024-09-11T08:24:28.357799+0000","last_became_peered":"2024-09-11T08:24:28.357799+0000","last_unstale":"2024-09-11T08:30:17.049430+0000","last_undegraded":"2024-09-11T08:30:17.049430+0000","last_fullsized":"2024-09-11T08:30:17.049430+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:58.653393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:53.175994+0000","last_clean_scrub_stamp":"2024-09-11T08:29:58.653393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:17.708980+0000","scrub_duration":0.0087291079999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"405'30","reported_seq":386,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:17.048397+0000","last_change":"2024-09-11T08:29:15.645126+0000","last_active":"2024-09-11T08:30:17.048397+0000","last_peered":"2024-09-11T08:30:17.048397+0000","last_clean":"2024-09-11T08:30:17.048397+0000","last_became_active":"2024-09-11T08:17:37.554357+0000","last_became_peered":"2024-09-11T08:17:37.554357+0000","last_unstale":"2024-09-11T08:30:17.048397+0000","last_undegraded":"2024-09-11T08:30:17.048397+0000","last_fullsized":"2024-09-11T08:30:17.048397+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'30","last_scrub_stamp":"2024-09-11T08:29:15.645039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:29:15.645039+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.021832190000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":301,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.863080+0000","last_change":"2024-09-11T08:27:56.258234+0000","last_active":"2024-09-11T08:30:16.863080+0000","last_peered":"2024-09-11T08:30:16.863080+0000","last_clean":"2024-09-11T08:30:16.863080+0000","last_became_active":"2024-09-11T08:15:56.518515+0000","last_became_peered":"2024-09-11T08:15:56.518515+0000","last_unstale":"2024-09-11T08:30:16.863080+0000","last_undegraded":"2024-09-11T08:30:16.863080+0000","last_fullsized":"2024-09-11T08:30:16.863080+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:56.258011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:23:42.199527+0000","last_clean_scrub_stamp":"2024-09-11T08:27:56.258011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022527070000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"354'13","reported_seq":311,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-09-11T08:30:23.848408+0000","last_change":"2024-09-11T08:30:14.950155+0000","last_active":"2024-09-11T08:30:23.848408+0000","last_peered":"2024-09-11T08:30:23.848408+0000","last_clean":"2024-09-11T08:30:23.848408+0000","last_became_active":"2024-09-11T08:24:28.353115+0000","last_became_peered":"2024-09-11T08:24:28.353115+0000","last_unstale":"2024-09-11T08:30:23.848408+0000","last_undegraded":"2024-09-11T08:30:23.848408+0000","last_fullsized":"2024-09-11T08:30:23.848408+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'13","last_scrub_stamp":"2024-09-11T08:30:14.950098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:30:14.950098+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:32.016065+0000","scrub_duration":0.0018318049999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":499,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.913261+0000","last_change":"2024-09-11T08:29:05.959570+0000","last_active":"2024-09-11T08:30:16.913261+0000","last_peered":"2024-09-11T08:30:16.913261+0000","last_clean":"2024-09-11T08:30:16.913261+0000","last_became_active":"2024-09-11T08:15:56.513949+0000","last_became_peered":"2024-09-11T08:15:56.513949+0000","last_unstale":"2024-09-11T08:30:16.913261+0000","last_undegraded":"2024-09-11T08:30:16.913261+0000","last_fullsized":"2024-09-11T08:30:16.913261+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:05.957912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:29:05.957912+0000","last_clean_scrub_stamp":"2024-09-11T08:29:05.957912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:27.228578+0000","scrub_duration":0.0062029260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"405'27","reported_seq":422,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:17.053635+0000","last_change":"2024-09-11T08:30:11.305412+0000","last_active":"2024-09-11T08:30:17.053635+0000","last_peered":"2024-09-11T08:30:17.053635+0000","last_clean":"2024-09-11T08:30:17.053635+0000","last_became_active":"2024-09-11T08:23:22.688993+0000","last_became_peered":"2024-09-11T08:23:22.688993+0000","last_unstale":"2024-09-11T08:30:17.053635+0000","last_undegraded":"2024-09-11T08:30:17.053635+0000","last_fullsized":"2024-09-11T08:30:17.053635+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'27","last_scrub_stamp":"2024-09-11T08:30:11.305355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:30:11.305355+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:19.774283+0000","scrub_duration":0.0026390200000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":350,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.863789+0000","last_change":"2024-09-11T08:29:12.448974+0000","last_active":"2024-09-11T08:30:16.863789+0000","last_peered":"2024-09-11T08:30:16.863789+0000","last_clean":"2024-09-11T08:30:16.863789+0000","last_became_active":"2024-09-11T08:23:22.682569+0000","last_became_peered":"2024-09-11T08:23:22.682569+0000","last_unstale":"2024-09-11T08:30:16.863789+0000","last_undegraded":"2024-09-11T08:30:16.863789+0000","last_fullsized":"2024-09-11T08:30:16.863789+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:12.448898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:29:12.448898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016737830000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":524,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.902337+0000","last_change":"2024-09-11T08:28:47.755273+0000","last_active":"2024-09-11T08:30:16.902337+0000","last_peered":"2024-09-11T08:30:16.902337+0000","last_clean":"2024-09-11T08:30:16.902337+0000","last_became_active":"2024-09-11T08:24:28.355004+0000","last_became_peered":"2024-09-11T08:24:28.355004+0000","last_unstale":"2024-09-11T08:30:16.902337+0000","last_undegraded":"2024-09-11T08:30:16.902337+0000","last_fullsized":"2024-09-11T08:30:16.902337+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:28:47.755203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:28:47.755203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001837315,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":313,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.915241+0000","last_change":"2024-09-11T08:29:23.241188+0000","last_active":"2024-09-11T08:30:16.915241+0000","last_peered":"2024-09-11T08:30:16.915241+0000","last_clean":"2024-09-11T08:30:16.915241+0000","last_became_active":"2024-09-11T08:24:28.372049+0000","last_became_peered":"2024-09-11T08:24:28.372049+0000","last_unstale":"2024-09-11T08:30:16.915241+0000","last_undegraded":"2024-09-11T08:30:16.915241+0000","last_fullsized":"2024-09-11T08:30:16.915241+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:23.241016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:29:23.241016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:40.844841+0000","scrub_duration":0.0063927849999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"405'25","reported_seq":544,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.902513+0000","last_change":"2024-09-11T08:29:20.881222+0000","last_active":"2024-09-11T08:30:16.902513+0000","last_peered":"2024-09-11T08:30:16.902513+0000","last_clean":"2024-09-11T08:30:16.902513+0000","last_became_active":"2024-09-11T08:07:32.049368+0000","last_became_peered":"2024-09-11T08:07:32.049368+0000","last_unstale":"2024-09-11T08:30:16.902513+0000","last_undegraded":"2024-09-11T08:30:16.902513+0000","last_fullsized":"2024-09-11T08:30:16.902513+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'25","last_scrub_stamp":"2024-09-11T08:29:20.881147+0000","last_deep_scrub":"343'19","last_deep_scrub_stamp":"2024-09-11T08:23:44.640077+0000","last_clean_scrub_stamp":"2024-09-11T08:29:20.881147+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:42.708414+0000","scrub_duration":0.0058682450000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":301,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.863968+0000","last_change":"2024-09-11T08:28:05.289814+0000","last_active":"2024-09-11T08:30:16.863968+0000","last_peered":"2024-09-11T08:30:16.863968+0000","last_clean":"2024-09-11T08:30:16.863968+0000","last_became_active":"2024-09-11T08:15:56.521570+0000","last_became_peered":"2024-09-11T08:15:56.521570+0000","last_unstale":"2024-09-11T08:30:16.863968+0000","last_undegraded":"2024-09-11T08:30:16.863968+0000","last_fullsized":"2024-09-11T08:30:16.863968+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:28:05.289763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:28:05.289763+0000","last_clean_scrub_stamp":"2024-09-11T08:28:05.289763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037268660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":276,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.910639+0000","last_change":"2024-09-11T08:29:58.282988+0000","last_active":"2024-09-11T08:30:16.910639+0000","last_peered":"2024-09-11T08:30:16.910639+0000","last_clean":"2024-09-11T08:30:16.910639+0000","last_became_active":"2024-09-11T08:17:37.557645+0000","last_became_peered":"2024-09-11T08:17:37.557645+0000","last_unstale":"2024-09-11T08:30:16.910639+0000","last_undegraded":"2024-09-11T08:30:16.910639+0000","last_fullsized":"2024-09-11T08:30:16.910639+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:58.282759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:22:02.568761+0000","last_clean_scrub_stamp":"2024-09-11T08:29:58.282759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:27.885348+0000","scrub_duration":0.0099211319999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":512,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.908091+0000","last_change":"2024-09-11T08:30:02.281204+0000","last_active":"2024-09-11T08:30:16.908091+0000","last_peered":"2024-09-11T08:30:16.908091+0000","last_clean":"2024-09-11T08:30:16.908091+0000","last_became_active":"2024-09-11T08:24:28.363959+0000","last_became_peered":"2024-09-11T08:24:28.363959+0000","last_unstale":"2024-09-11T08:30:16.908091+0000","last_undegraded":"2024-09-11T08:30:16.908091+0000","last_fullsized":"2024-09-11T08:30:16.908091+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:02.281112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:09.973943+0000","last_clean_scrub_stamp":"2024-09-11T08:30:02.281112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:13.828016+0000","scrub_duration":0.018012968000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"405'27","reported_seq":591,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.866347+0000","last_change":"2024-09-11T08:28:55.841566+0000","last_active":"2024-09-11T08:30:16.866347+0000","last_peered":"2024-09-11T08:30:16.866347+0000","last_clean":"2024-09-11T08:30:16.866347+0000","last_became_active":"2024-09-11T08:15:56.520058+0000","last_became_peered":"2024-09-11T08:15:56.520058+0000","last_unstale":"2024-09-11T08:30:16.866347+0000","last_undegraded":"2024-09-11T08:30:16.866347+0000","last_fullsized":"2024-09-11T08:30:16.866347+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'27","last_scrub_stamp":"2024-09-11T08:28:55.841457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:28:55.841457+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012026871999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":404,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.871290+0000","last_change":"2024-09-11T08:29:00.479990+0000","last_active":"2024-09-11T08:30:16.871290+0000","last_peered":"2024-09-11T08:30:16.871290+0000","last_clean":"2024-09-11T08:30:16.871290+0000","last_became_active":"2024-09-11T08:24:28.356974+0000","last_became_peered":"2024-09-11T08:24:28.356974+0000","last_unstale":"2024-09-11T08:30:16.871290+0000","last_undegraded":"2024-09-11T08:30:16.871290+0000","last_fullsized":"2024-09-11T08:30:16.871290+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:00.479835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:29:00.479835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002146532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":311,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.861185+0000","last_change":"2024-09-11T08:29:06.470596+0000","last_active":"2024-09-11T08:30:16.861185+0000","last_peered":"2024-09-11T08:30:16.861185+0000","last_clean":"2024-09-11T08:30:16.861185+0000","last_became_active":"2024-09-11T08:15:56.523728+0000","last_became_peered":"2024-09-11T08:15:56.523728+0000","last_unstale":"2024-09-11T08:30:16.861185+0000","last_undegraded":"2024-09-11T08:30:16.861185+0000","last_fullsized":"2024-09-11T08:30:16.861185+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:06.470537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:11:34.159053+0000","last_clean_scrub_stamp":"2024-09-11T08:29:06.470537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014249079999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":493,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.864339+0000","last_change":"2024-09-11T08:29:56.882574+0000","last_active":"2024-09-11T08:30:16.864339+0000","last_peered":"2024-09-11T08:30:16.864339+0000","last_clean":"2024-09-11T08:30:16.864339+0000","last_became_active":"2024-09-11T08:07:36.633368+0000","last_became_peered":"2024-09-11T08:07:36.633368+0000","last_unstale":"2024-09-11T08:30:16.864339+0000","last_undegraded":"2024-09-11T08:30:16.864339+0000","last_fullsized":"2024-09-11T08:30:16.864339+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:56.882497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:22:50.992034+0000","last_clean_scrub_stamp":"2024-09-11T08:29:56.882497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:13.644291+0000","scrub_duration":0.0057421800000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"405'14","reported_seq":507,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.861086+0000","last_change":"2024-09-11T08:28:56.806080+0000","last_active":"2024-09-11T08:30:16.861086+0000","last_peered":"2024-09-11T08:30:16.861086+0000","last_clean":"2024-09-11T08:30:16.861086+0000","last_became_active":"2024-09-11T08:07:32.045728+0000","last_became_peered":"2024-09-11T08:07:32.045728+0000","last_unstale":"2024-09-11T08:30:16.861086+0000","last_undegraded":"2024-09-11T08:30:16.861086+0000","last_fullsized":"2024-09-11T08:30:16.861086+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'14","last_scrub_stamp":"2024-09-11T08:28:56.805624+0000","last_deep_scrub":"312'10","last_deep_scrub_stamp":"2024-09-11T08:21:55.859866+0000","last_clean_scrub_stamp":"2024-09-11T08:28:56.805624+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:26.046082+0000","scrub_duration":0.019371645,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":525,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.910369+0000","last_change":"2024-09-11T08:30:11.270503+0000","last_active":"2024-09-11T08:30:16.910369+0000","last_peered":"2024-09-11T08:30:16.910369+0000","last_clean":"2024-09-11T08:30:16.910369+0000","last_became_active":"2024-09-11T08:24:28.357978+0000","last_became_peered":"2024-09-11T08:24:28.357978+0000","last_unstale":"2024-09-11T08:30:16.910369+0000","last_undegraded":"2024-09-11T08:30:16.910369+0000","last_fullsized":"2024-09-11T08:30:16.910369+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:11.270399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:30:11.270399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:30.279105+0000","scrub_duration":0.001777708,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":551,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:17.056010+0000","last_change":"2024-09-11T08:30:03.423705+0000","last_active":"2024-09-11T08:30:17.056010+0000","last_peered":"2024-09-11T08:30:17.056010+0000","last_clean":"2024-09-11T08:30:17.056010+0000","last_became_active":"2024-09-11T08:15:56.528607+0000","last_became_peered":"2024-09-11T08:15:56.528607+0000","last_unstale":"2024-09-11T08:30:17.056010+0000","last_undegraded":"2024-09-11T08:30:17.056010+0000","last_fullsized":"2024-09-11T08:30:17.056010+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:03.423649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:30:03.423649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:13.967310+0000","scrub_duration":0.001131574,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":304,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.862146+0000","last_change":"2024-09-11T08:30:16.681411+0000","last_active":"2024-09-11T08:30:16.862146+0000","last_peered":"2024-09-11T08:30:16.862146+0000","last_clean":"2024-09-11T08:30:16.862146+0000","last_became_active":"2024-09-11T08:15:56.521488+0000","last_became_peered":"2024-09-11T08:15:56.521488+0000","last_unstale":"2024-09-11T08:30:16.862146+0000","last_undegraded":"2024-09-11T08:30:16.862146+0000","last_fullsized":"2024-09-11T08:30:16.862146+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:16.681355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:20:18.827709+0000","last_clean_scrub_stamp":"2024-09-11T08:30:16.681355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:46.492811+0000","scrub_duration":0.0012763220000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":522,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:16.902158+0000","last_change":"2024-09-11T08:27:17.179442+0000","last_active":"2024-09-11T08:30:16.902158+0000","last_peered":"2024-09-11T08:30:16.902158+0000","last_clean":"2024-09-11T08:30:16.902158+0000","last_became_active":"2024-09-11T08:24:28.354141+0000","last_became_peered":"2024-09-11T08:24:28.354141+0000","last_unstale":"2024-09-11T08:30:16.902158+0000","last_undegraded":"2024-09-11T08:30:16.902158+0000","last_fullsized":"2024-09-11T08:30:16.902158+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:27:17.179345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:50.585566+0000","last_clean_scrub_stamp":"2024-09-11T08:27:17.179345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0077697699999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":273,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:20.205125+0000","last_change":"2024-09-11T08:30:20.205091+0000","last_active":"2024-09-11T08:30:20.205125+0000","last_peered":"2024-09-11T08:30:20.205125+0000","last_clean":"2024-09-11T08:30:20.205125+0000","last_became_active":"2024-09-11T08:17:37.554262+0000","last_became_peered":"2024-09-11T08:17:37.554262+0000","last_unstale":"2024-09-11T08:30:20.205125+0000","last_undegraded":"2024-09-11T08:30:20.205125+0000","last_fullsized":"2024-09-11T08:30:20.205125+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:20.205039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:30:20.205039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:31:30.314306+0000","scrub_duration":0.001147829,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"405'32","reported_seq":615,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:17.050232+0000","last_change":"2024-09-11T08:29:07.366953+0000","last_active":"2024-09-11T08:30:17.050232+0000","last_peered":"2024-09-11T08:30:17.050232+0000","last_clean":"2024-09-11T08:30:17.050232+0000","last_became_active":"2024-09-11T08:24:28.360937+0000","last_became_peered":"2024-09-11T08:24:28.360937+0000","last_unstale":"2024-09-11T08:30:17.050232+0000","last_undegraded":"2024-09-11T08:30:17.050232+0000","last_fullsized":"2024-09-11T08:30:17.050232+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'32","last_scrub_stamp":"2024-09-11T08:29:07.366863+0000","last_deep_scrub":"405'32","last_deep_scrub_stamp":"2024-09-11T08:29:07.366863+0000","last_clean_scrub_stamp":"2024-09-11T08:29:07.366863+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0072573189999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":339,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-11T08:30:17.048659+0000","last_change":"2024-09-11T08:29:58.627283+0000","last_active":"2024-09-11T08:30:17.048659+0000","last_peered":"2024-09-11T08:30:17.048659+0000","last_clean":"2024-09-11T08:30:17.048659+0000","last_became_active":"2024-09-11T08:15:08.769133+0000","last_became_peered":"2024-09-11T08:15:08.769133+0000","last_unstale":"2024-09-11T08:30:17.048659+0000","last_undegraded":"2024-09-11T08:30:17.048659+0000","last_fullsized":"2024-09-11T08:30:17.048659+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:29:58.627201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:18:39.206016+0000","last_clean_scrub_stamp":"2024-09-11T08:29:58.627201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:30:59.524575+0000","scrub_duration":0.007062512,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102,"ondisk_log_size":102,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1546,"num_read_kb":1303,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":4566,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3425,"internal_metadata":0},"log_size":1066,"ondisk_log_size":1066,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":968,"num_read_kb":963,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":168,"seq":721554505922,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":920688,"kb_used_data":1316,"kb_used_omap":7,"kb_used_meta":919352,"kb_avail":92824464,"statfs":{"total":95995035648,"available":95052251136,"internally_reserved":0,"allocated":1347584,"data_stored":972234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8027,"internal_metadata":941416613},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:29:49 2024","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":1.1060000000000001,"15min":2.419},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":2.2839999999999998,"5min":19.797999999999998,"15min":117.57599999999999},"last":0.36499999999999999},{"interface":"front","average":{"1min":1.3009999999999999,"5min":1.1379999999999999,"15min":2.2999999999999998},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":4.3070000000000004,"5min":19.899000000000001,"15min":117.542},"last":0.35599999999999998}]},{"osd":1,"last update":"Wed Sep 11 08:29:49 2024","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.2569999999999999,"15min":1.8799999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.20899999999999999},"max":{"1min":4.7220000000000004,"5min":24.347999999999999,"15min":66.971000000000004},"last":0.38},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.2150000000000001,"15min":1.702},"min":{"1min":0.246,"5min":0.23400000000000001,"15min":0.23000000000000001},"max":{"1min":3.8069999999999999,"5min":25.382000000000001,"15min":67.185000000000002},"last":0.44600000000000001}]},{"osd":2,"last update":"Wed Sep 11 08:29:49 2024","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":0.97099999999999997,"15min":1.04},"min":{"1min":0.25600000000000001,"5min":0.245,"15min":0.22700000000000001},"max":{"1min":9.3439999999999994,"5min":9.3439999999999994,"15min":19.097000000000001},"last":0.40899999999999997},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.329,"15min":1.202},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":3.3199999999999998,"5min":7.7699999999999996,"15min":21.594000000000001},"last":0.34300000000000003}]},{"osd":3,"last update":"Wed Sep 11 08:29:49 2024","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.347,"15min":1.23},"min":{"1min":0.28399999999999997,"5min":0.27200000000000002,"15min":0.251},"max":{"1min":3.8300000000000001,"5min":24.315000000000001,"15min":24.315000000000001},"last":0.41899999999999998},{"interface":"front","average":{"1min":1.0880000000000001,"5min":1.147,"15min":1.206},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.251},"max":{"1min":3.516,"5min":24.864999999999998,"15min":24.864999999999998},"last":0.32600000000000001}]},{"osd":4,"last update":"Wed Sep 11 08:29:49 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.73999999999999999,"15min":0.83599999999999997},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":1.3500000000000001,"5min":5.7240000000000002,"15min":5.7240000000000002},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.66000000000000003,"15min":0.755},"min":{"1min":0.27900000000000003,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":2.367,"5min":4.2060000000000004,"15min":5.2649999999999997},"last":0.311}]},{"osd":5,"last update":"Wed Sep 11 08:29:44 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.84899999999999998,"15min":1.2290000000000001},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":3.4470000000000001,"5min":6.3979999999999997,"15min":6.3979999999999997},"last":0.439},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.68899999999999995,"15min":0.86799999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":2.5960000000000001,"5min":2.887,"15min":2.887},"last":0.46500000000000002}]},{"osd":6,"last update":"Wed Sep 11 08:29:49 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":1.3340000000000001,"15min":1.0009999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.248},"max":{"1min":2.2010000000000001,"5min":42.749000000000002,"15min":42.749000000000002},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":1.4139999999999999,"15min":1.093},"min":{"1min":0.32300000000000001,"5min":0.311,"15min":0.25700000000000001},"max":{"1min":1.5900000000000001,"5min":41.914000000000001,"15min":41.914000000000001},"last":0.52700000000000002}]}]},{"osd":6,"up_from":45,"seq":193273528605,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":896800,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":895486,"kb_avail":92848352,"statfs":{"total":95995035648,"available":95076712448,"internally_reserved":0,"allocated":1331200,"data_stored":972057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":916978008},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:29:34 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":4.7800000000000002,"15min":3.1280000000000001},"min":{"1min":0.28999999999999998,"5min":0.21199999999999999,"15min":0.17999999999999999},"max":{"1min":3.649,"5min":331.14800000000002,"15min":331.14800000000002},"last":0.63700000000000001},{"interface":"front","average":{"1min":1.1639999999999999,"5min":4.4930000000000003,"15min":2.98},"min":{"1min":0.30199999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":3.0939999999999999,"5min":331.31,"15min":331.31},"last":0.47299999999999998}]},{"osd":1,"last update":"Wed Sep 11 08:29:34 2024","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.5880000000000001,"15min":1.3859999999999999},"min":{"1min":0.35799999999999998,"5min":0.22900000000000001,"15min":0.214},"max":{"1min":5.0350000000000001,"5min":29.423999999999999,"15min":29.423999999999999},"last":0.60599999999999998},{"interface":"front","average":{"1min":1.5509999999999999,"5min":1.607,"15min":1.345},"min":{"1min":0.23499999999999999,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":4.4640000000000004,"5min":29.109000000000002,"15min":29.109000000000002},"last":0.58299999999999996}]},{"osd":2,"last update":"Wed Sep 11 08:29:34 2024","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.6439999999999999,"15min":1.3129999999999999},"min":{"1min":0.25,"5min":0.25,"15min":0.23200000000000001},"max":{"1min":3.218,"5min":28.167999999999999,"15min":39.652999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.587,"15min":1.415},"min":{"1min":0.35199999999999998,"5min":0.251,"15min":0.251},"max":{"1min":4.0099999999999998,"5min":31.648,"15min":38.390999999999998},"last":0.50900000000000001}]},{"osd":3,"last update":"Wed Sep 11 08:29:34 2024","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":1.5660000000000001,"15min":1.25},"min":{"1min":0.30599999999999999,"5min":0.253,"15min":0.22700000000000001},"max":{"1min":1.774,"5min":12.414,"15min":12.414},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.95499999999999996,"5min":1.522,"15min":1.24},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":2.133,"5min":28.004999999999999,"15min":28.004999999999999},"last":0.64700000000000002}]},{"osd":4,"last update":"Wed Sep 11 08:29:34 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.90000000000000002,"15min":0.77500000000000002},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.20499999999999999},"max":{"1min":3.028,"5min":9.7560000000000002,"15min":9.7560000000000002},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.79900000000000004,"15min":0.74399999999999999},"min":{"1min":0.307,"5min":0.29099999999999998,"15min":0.23699999999999999},"max":{"1min":3.5659999999999998,"5min":3.5659999999999998,"15min":4.468},"last":0.54800000000000004}]},{"osd":5,"last update":"Wed Sep 11 08:29:34 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.97699999999999998,"15min":1.552},"min":{"1min":0.29499999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":3.1739999999999999,"5min":9.7989999999999995,"15min":9.7989999999999995},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.99199999999999999,"15min":1.3939999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":2.1019999999999999,"5min":7.2999999999999998,"15min":7.2999999999999998},"last":0.626}]},{"osd":7,"last update":"Wed Sep 11 08:29:46 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.2090000000000001,"15min":1.014},"min":{"1min":0.34599999999999997,"5min":0.25900000000000001,"15min":0.253},"max":{"1min":3.3889999999999998,"5min":7.5529999999999999,"15min":10.167999999999999},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.96099999999999997,"15min":0.82199999999999995},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.25700000000000001},"max":{"1min":2.887,"5min":3.6160000000000001,"15min":5.3579999999999997},"last":0.67100000000000004}]}]},{"osd":5,"up_from":353,"seq":1516123455561,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":360672,"kb_used_data":724,"kb_used_omap":4,"kb_used_meta":359931,"kb_avail":93384480,"statfs":{"total":95995035648,"available":95625707520,"internally_reserved":0,"allocated":741376,"data_stored":381250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":368569862},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:29:39 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.99299999999999999,"15min":1.6000000000000001},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":2.9710000000000001,"5min":19.379999999999999,"15min":19.379999999999999},"last":0.442},{"interface":"front","average":{"1min":0.96799999999999997,"5min":1.135,"15min":1.623},"min":{"1min":0.23599999999999999,"5min":0.218,"15min":0.218},"max":{"1min":2.6739999999999999,"5min":18.238,"15min":18.238},"last":0.35699999999999998}]},{"osd":1,"last update":"Wed Sep 11 08:29:39 2024","interfaces":[{"interface":"back","average":{"1min":4.8869999999999996,"5min":2.7749999999999999,"15min":1.6739999999999999},"min":{"1min":0.24199999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":85.728999999999999,"5min":85.728999999999999,"15min":85.728999999999999},"last":0.39100000000000001},{"interface":"front","average":{"1min":5.0979999999999999,"5min":2.8319999999999999,"15min":1.4159999999999999},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":85.668000000000006,"5min":85.668000000000006,"15min":85.668000000000006},"last":0.45200000000000001}]},{"osd":2,"last update":"Wed Sep 11 08:29:39 2024","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":1.4099999999999999,"15min":1.8799999999999999},"min":{"1min":0.26000000000000001,"5min":0.251,"15min":0.251},"max":{"1min":3.7599999999999998,"5min":18.350999999999999,"15min":18.350999999999999},"last":0.46200000000000002},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.3200000000000001,"15min":1.74},"min":{"1min":0.27400000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":9.6489999999999991,"5min":18.381,"15min":18.381},"last":0.34200000000000003}]},{"osd":3,"last update":"Wed Sep 11 08:29:39 2024","interfaces":[{"interface":"back","average":{"1min":4.4080000000000004,"5min":2.3580000000000001,"15min":1.5680000000000001},"min":{"1min":0.27500000000000002,"5min":0.251,"15min":0.251},"max":{"1min":73.697000000000003,"5min":73.697000000000003,"15min":73.697000000000003},"last":0.49099999999999999},{"interface":"front","average":{"1min":4.2370000000000001,"5min":2.1309999999999998,"15min":1.28},"min":{"1min":0.35399999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":73.727999999999994,"5min":73.727999999999994,"15min":73.727999999999994},"last":0.47299999999999998}]},{"osd":4,"last update":"Wed Sep 11 08:29:39 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.75900000000000001,"15min":1.083},"min":{"1min":0.27000000000000002,"5min":0.249,"15min":0.249},"max":{"1min":2.1230000000000002,"5min":5.8630000000000004,"15min":5.8630000000000004},"last":0.432},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.72999999999999998,"15min":1.1160000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":1.8700000000000001,"5min":5.8979999999999997,"15min":5.8979999999999997},"last":0.372}]},{"osd":6,"last update":"Wed Sep 11 08:29:39 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.83799999999999997,"15min":1.3220000000000001},"min":{"1min":0.317,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":1.921,"5min":5.9779999999999998,"15min":5.9779999999999998},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.84099999999999997,"15min":1.5669999999999999},"min":{"1min":0.32300000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.9990000000000001,"5min":5.6680000000000001,"15min":5.6680000000000001},"last":0.41199999999999998}]},{"osd":7,"last update":"Wed Sep 11 08:29:39 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.92800000000000005,"15min":1.153},"min":{"1min":0.29299999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":3.2320000000000002,"5min":7.8719999999999999,"15min":7.8719999999999999},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.94999999999999996,"15min":1.544},"min":{"1min":0.29899999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":3.2759999999999998,"5min":6.8739999999999997,"15min":6.8739999999999997},"last":0.51600000000000001}]}]},{"osd":4,"up_from":31,"seq":133143986471,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":894836,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":894078,"kb_avail":92850316,"statfs":{"total":95995035648,"available":95078723584,"internally_reserved":0,"allocated":761856,"data_stored":382564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":915536216},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:29:45 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":2.6219999999999999,"15min":1.948},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":2.3159999999999998,"5min":64.329999999999998,"15min":64.329999999999998},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.68200000000000005,"5min":2.4470000000000001,"15min":1.8759999999999999},"min":{"1min":0.24199999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.3900000000000001,"5min":66.819999999999993,"15min":66.819999999999993},"last":0.52000000000000002}]},{"osd":1,"last update":"Wed Sep 11 08:29:45 2024","interfaces":[{"interface":"back","average":{"1min":7.2530000000000001,"5min":2.2719999999999998,"15min":1.6619999999999999},"min":{"1min":0.27600000000000002,"5min":0.22,"15min":0.214},"max":{"1min":91.852000000000004,"5min":91.852000000000004,"15min":91.852000000000004},"last":0.60999999999999999},{"interface":"front","average":{"1min":6.7080000000000002,"5min":2.4209999999999998,"15min":1.7470000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":91.697000000000003,"5min":91.697000000000003,"15min":91.697000000000003},"last":0.54500000000000004}]},{"osd":2,"last update":"Wed Sep 11 08:29:45 2024","interfaces":[{"interface":"back","average":{"1min":1.819,"5min":2.1499999999999999,"15min":1.4910000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":13.710000000000001,"5min":54.317999999999998,"15min":54.317999999999998},"last":0.50800000000000001},{"interface":"front","average":{"1min":2.2469999999999999,"5min":2.2839999999999998,"15min":1.611},"min":{"1min":0.30299999999999999,"5min":0.246,"15min":0.23400000000000001},"max":{"1min":12.711,"5min":54.430999999999997,"15min":54.430999999999997},"last":0.58499999999999996}]},{"osd":3,"last update":"Wed Sep 11 08:29:45 2024","interfaces":[{"interface":"back","average":{"1min":1.8819999999999999,"5min":1.355,"15min":1.381},"min":{"1min":0.27700000000000002,"5min":0.247,"15min":0.247},"max":{"1min":12.811999999999999,"5min":12.811999999999999,"15min":30.321000000000002},"last":0.55300000000000005},{"interface":"front","average":{"1min":2.1230000000000002,"5min":1.3360000000000001,"15min":1.2170000000000001},"min":{"1min":0.26700000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":12.978,"5min":12.978,"15min":30.620000000000001},"last":0.63400000000000001}]},{"osd":5,"last update":"Wed Sep 11 08:29:34 2024","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":0.80000000000000004,"15min":1.1739999999999999},"min":{"1min":0.25700000000000001,"5min":0.246,"15min":0.246},"max":{"1min":4.3410000000000002,"5min":5.1050000000000004,"15min":5.1050000000000004},"last":0.436},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.84899999999999998,"15min":1.302},"min":{"1min":0.23799999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":1.3200000000000001,"5min":6.1150000000000002,"15min":6.1150000000000002},"last":0.40100000000000002}]},{"osd":6,"last update":"Wed Sep 11 08:29:34 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.90400000000000003,"15min":0.89800000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.23200000000000001},"max":{"1min":3.036,"5min":6.4039999999999999,"15min":48.808},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.96199999999999997,"15min":0.95399999999999996},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.251},"max":{"1min":2.415,"5min":5.3159999999999998,"15min":48.746000000000002},"last":0.47999999999999998}]},{"osd":7,"last update":"Wed Sep 11 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.78500000000000003,"15min":0.73199999999999998},"min":{"1min":0.27500000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.96599999999999997,"5min":6.093,"15min":6.093},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.63100000000000001,"15min":0.70999999999999996},"min":{"1min":0.30099999999999999,"5min":0.29899999999999999,"15min":0.26800000000000002},"max":{"1min":1.1950000000000001,"5min":4.8259999999999996,"15min":4.8259999999999996},"last":0.46899999999999997}]}]},{"osd":3,"up_from":25,"seq":107374182700,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":901952,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":901182,"kb_avail":92843200,"statfs":{"total":95995035648,"available":95071436800,"internally_reserved":0,"allocated":774144,"data_stored":382035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":922810712},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:29:38 2024","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":1.702,"15min":1.8500000000000001},"min":{"1min":0.25900000000000001,"5min":0.16300000000000001,"15min":0.14999999999999999},"max":{"1min":4.9139999999999997,"5min":24.616,"15min":60.469999999999999},"last":0.33500000000000002},{"interface":"front","average":{"1min":1.7450000000000001,"5min":1.7,"15min":1.732},"min":{"1min":0.24099999999999999,"5min":0.20599999999999999,"15min":0.17399999999999999},"max":{"1min":6.7089999999999996,"5min":23.640000000000001,"15min":60.488999999999997},"last":0.46000000000000002}]},{"osd":1,"last update":"Wed Sep 11 08:29:38 2024","interfaces":[{"interface":"back","average":{"1min":2.2050000000000001,"5min":1.4550000000000001,"15min":1.6459999999999999},"min":{"1min":0.28999999999999998,"5min":0.20399999999999999,"15min":0.19600000000000001},"max":{"1min":7.5739999999999998,"5min":9.5619999999999994,"15min":60.578000000000003},"last":0.371},{"interface":"front","average":{"1min":1.5680000000000001,"5min":1.361,"15min":1.6859999999999999},"min":{"1min":0.246,"5min":0.20899999999999999,"15min":0.19500000000000001},"max":{"1min":7.1399999999999997,"5min":7.1399999999999997,"15min":63.674999999999997},"last":0.31900000000000001}]},{"osd":2,"last update":"Wed Sep 11 08:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":2.0430000000000001,"15min":1.675},"min":{"1min":0.28000000000000003,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":2.1899999999999999,"5min":67.503,"15min":67.503},"last":0.35699999999999998},{"interface":"front","average":{"1min":1.5980000000000001,"5min":2.3140000000000001,"15min":1.768},"min":{"1min":0.26500000000000001,"5min":0.251,"15min":0.19900000000000001},"max":{"1min":7.1020000000000003,"5min":67.313999999999993,"15min":67.313999999999993},"last":0.39600000000000002}]},{"osd":4,"last update":"Wed Sep 11 08:29:47 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":1.196,"15min":1.4039999999999999},"min":{"1min":0.29199999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":2.2650000000000001,"5min":6.8680000000000003,"15min":61.290999999999997},"last":0.47999999999999998},{"interface":"front","average":{"1min":1.423,"5min":1.2470000000000001,"15min":1.28},"min":{"1min":0.33500000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":4.8179999999999996,"5min":6.46,"15min":60.856000000000002},"last":0.441}]},{"osd":5,"last update":"Wed Sep 11 08:29:38 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.306,"15min":1.665},"min":{"1min":0.34899999999999998,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":6.6189999999999998,"5min":7.3849999999999998,"15min":7.3849999999999998},"last":0.495},{"interface":"front","average":{"1min":0.97799999999999998,"5min":1.1759999999999999,"15min":1.22},"min":{"1min":0.39000000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":3.5049999999999999,"5min":6.835,"15min":6.835},"last":0.45200000000000001}]},{"osd":6,"last update":"Wed Sep 11 08:29:38 2024","interfaces":[{"interface":"back","average":{"1min":1.6759999999999999,"5min":1.4910000000000001,"15min":1.649},"min":{"1min":0.39300000000000002,"5min":0.32700000000000001,"15min":0.29699999999999999},"max":{"1min":6.6520000000000001,"5min":6.6520000000000001,"15min":60.817},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.94699999999999995,"5min":1.2050000000000001,"15min":1.669},"min":{"1min":0.314,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":2.2909999999999999,"5min":6.8970000000000002,"15min":61.264000000000003},"last":0.41099999999999998}]},{"osd":7,"last update":"Wed Sep 11 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":1.7010000000000001,"5min":1.321,"15min":1.677},"min":{"1min":0.441,"5min":0.33500000000000002,"15min":0.29899999999999999},"max":{"1min":6.6840000000000002,"5min":6.6840000000000002,"15min":60.847000000000001},"last":0.432},{"interface":"front","average":{"1min":1.641,"5min":1.411,"15min":1.8240000000000001},"min":{"1min":0.36699999999999999,"5min":0.35899999999999999,"15min":0.33800000000000002},"max":{"1min":6.899,"5min":6.899,"15min":60.883000000000003},"last":0.47099999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411631,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":889888,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":889150,"kb_avail":92855264,"statfs":{"total":95995035648,"available":95083790336,"internally_reserved":0,"allocated":741376,"data_stored":381162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":910489944},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:30:10 2024","interfaces":[{"interface":"back","average":{"1min":7.7149999999999999,"5min":3.585,"15min":3.1840000000000002},"min":{"1min":0.27500000000000002,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":102.18000000000001,"5min":102.18000000000001,"15min":102.18000000000001},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.91900000000000004,"5min":2.0739999999999998,"15min":2.9350000000000001},"min":{"1min":0.29099999999999998,"5min":0.20599999999999999,"15min":0.17999999999999999},"max":{"1min":4.2510000000000003,"5min":81.391000000000005,"15min":93.790999999999997},"last":0.33500000000000002}]},{"osd":1,"last update":"Wed Sep 11 08:30:10 2024","interfaces":[{"interface":"back","average":{"1min":7.907,"5min":2.7069999999999999,"15min":2.0819999999999999},"min":{"1min":0.20399999999999999,"5min":0.19800000000000001,"15min":0.182},"max":{"1min":102.134,"5min":102.134,"15min":102.134},"last":0.373},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.206,"15min":1.577},"min":{"1min":0.30499999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":4.1799999999999997,"5min":6.7220000000000004,"15min":53.082999999999998},"last":0.43099999999999999}]},{"osd":3,"last update":"Wed Sep 11 08:29:34 2024","interfaces":[{"interface":"back","average":{"1min":1.97,"5min":2.169,"15min":1.7350000000000001},"min":{"1min":0.41299999999999998,"5min":0.22,"15min":0.21299999999999999},"max":{"1min":6.2549999999999999,"5min":54.289999999999999,"15min":54.289999999999999},"last":0.40600000000000003},{"interface":"front","average":{"1min":1.4039999999999999,"5min":2.0699999999999998,"15min":1.6679999999999999},"min":{"1min":0.34499999999999997,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":6.6470000000000002,"5min":53.927,"15min":53.927},"last":0.38800000000000001}]},{"osd":4,"last update":"Wed Sep 11 08:29:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.169,"15min":1.103},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.24099999999999999},"max":{"1min":6.6920000000000002,"5min":6.6920000000000002,"15min":6.6920000000000002},"last":0.70399999999999996},{"interface":"front","average":{"1min":1.0900000000000001,"5min":1.0429999999999999,"15min":1.107},"min":{"1min":0.39700000000000002,"5min":0.311,"15min":0.27500000000000002},"max":{"1min":4.3070000000000004,"5min":4.6369999999999996,"15min":7.8970000000000002},"last":0.65300000000000002}]},{"osd":5,"last update":"Wed Sep 11 08:29:34 2024","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":1.0529999999999999,"15min":1.5249999999999999},"min":{"1min":0.36599999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":4.2270000000000003,"5min":4.5839999999999996,"15min":4.5839999999999996},"last":0.498},{"interface":"front","average":{"1min":1.452,"5min":1.1619999999999999,"15min":1.3720000000000001},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":6.6589999999999998,"5min":6.6589999999999998,"15min":6.6589999999999998},"last":0.50700000000000001}]},{"osd":6,"last update":"Wed Sep 11 08:29:44 2024","interfaces":[{"interface":"back","average":{"1min":1.8149999999999999,"5min":1.3380000000000001,"15min":1.2150000000000001},"min":{"1min":0.371,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":6.3440000000000003,"5min":6.3440000000000003,"15min":8.0350000000000001},"last":0.64100000000000001},{"interface":"front","average":{"1min":1.085,"5min":1.206,"15min":1.1850000000000001},"min":{"1min":0.39400000000000002,"5min":0.375,"15min":0.32500000000000001},"max":{"1min":4.4390000000000001,"5min":5.9619999999999997,"15min":7.8710000000000004},"last":0.47199999999999998}]},{"osd":7,"last update":"Wed Sep 11 08:29:48 2024","interfaces":[{"interface":"back","average":{"1min":1.5629999999999999,"5min":1.3819999999999999,"15min":1.3640000000000001},"min":{"1min":0.41699999999999998,"5min":0.312,"15min":0.27500000000000002},"max":{"1min":7.6950000000000003,"5min":11.279,"15min":35.274999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.3049999999999999,"15min":1.4330000000000001},"min":{"1min":0.44,"5min":0.33600000000000002,"15min":0.32800000000000001},"max":{"1min":4.4660000000000002,"5min":11.305,"15min":35.265000000000001},"last":0.52600000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575157,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":891012,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":890238,"kb_avail":92854140,"statfs":{"total":95995035648,"available":95082639360,"internally_reserved":0,"allocated":778240,"data_stored":382178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":911604056},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:29:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":1.337,"15min":1.5189999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":5.0110000000000001,"5min":14.045,"15min":63.932000000000002},"last":0.35199999999999998},{"interface":"front","average":{"1min":1.5900000000000001,"5min":1.5940000000000001,"15min":1.869},"min":{"1min":0.222,"5min":0.189,"15min":0.16800000000000001},"max":{"1min":5.1120000000000001,"5min":14.127000000000001,"15min":64.269000000000005},"last":0.316}]},{"osd":2,"last update":"Wed Sep 11 08:30:27 2024","interfaces":[{"interface":"back","average":{"1min":2.2029999999999998,"5min":1.7290000000000001,"15min":1.502},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.189},"max":{"1min":13.818,"5min":13.928000000000001,"15min":20.925999999999998},"last":0.30299999999999999},{"interface":"front","average":{"1min":3.3199999999999998,"5min":1.9450000000000001,"15min":1.5409999999999999},"min":{"1min":0.20799999999999999,"5min":0.19900000000000001,"15min":0.19},"max":{"1min":13.484,"5min":16.099,"15min":20.812000000000001},"last":0.32700000000000001}]},{"osd":3,"last update":"Wed Sep 11 08:29:36 2024","interfaces":[{"interface":"back","average":{"1min":2.5670000000000002,"5min":1.663,"15min":1.5629999999999999},"min":{"1min":0.27100000000000002,"5min":0.20399999999999999,"15min":0.19700000000000001},"max":{"1min":7.9390000000000001,"5min":16.193999999999999,"15min":16.783999999999999},"last":0.34300000000000003},{"interface":"front","average":{"1min":1.966,"5min":1.591,"15min":1.488},"min":{"1min":0.30499999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":4.415,"5min":16.033000000000001,"15min":16.033000000000001},"last":0.373}]},{"osd":4,"last update":"Wed Sep 11 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":2.222,"5min":1.5940000000000001,"15min":1.9399999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.254},"max":{"1min":8.2680000000000007,"5min":16.178000000000001,"15min":110.86499999999999},"last":0.39300000000000002},{"interface":"front","average":{"1min":1.528,"5min":1.6930000000000001,"15min":1.982},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28000000000000003},"max":{"1min":5.1980000000000004,"5min":19.009,"15min":110.831},"last":0.41799999999999998}]},{"osd":5,"last update":"Wed Sep 11 08:29:36 2024","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.4059999999999999,"15min":1.5660000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":4.2359999999999998,"5min":16.012,"15min":16.012},"last":0.436},{"interface":"front","average":{"1min":1.827,"5min":1.508,"15min":1.5660000000000001},"min":{"1min":0.315,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":7.1420000000000003,"5min":16.206,"15min":16.206},"last":0.44900000000000001}]},{"osd":6,"last update":"Wed Sep 11 08:29:36 2024","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.6479999999999999,"15min":1.8660000000000001},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":3.786,"5min":16.143000000000001,"15min":124.31999999999999},"last":0.54700000000000004},{"interface":"front","average":{"1min":1.329,"5min":1.611,"15min":1.831},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.29799999999999999},"max":{"1min":3.6150000000000002,"5min":15.984,"15min":124.958},"last":0.49099999999999999}]},{"osd":7,"last update":"Wed Sep 11 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":2.3540000000000001,"5min":1.706,"15min":1.865},"min":{"1min":0.32500000000000001,"5min":0.318,"15min":0.28599999999999998},"max":{"1min":8.2940000000000005,"5min":16.228999999999999,"15min":109.149},"last":0.42799999999999999},{"interface":"front","average":{"1min":1.387,"5min":1.4419999999999999,"15min":1.869},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.318},"max":{"1min":3.7599999999999998,"5min":16.170999999999999,"15min":109.309},"last":0.52800000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738680,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":899580,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":898238,"kb_avail":92845572,"statfs":{"total":95995035648,"available":95073865728,"internally_reserved":0,"allocated":1359872,"data_stored":973180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":919796056},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 11 08:29:39 2024","interfaces":[{"interface":"back","average":{"1min":2.5299999999999998,"5min":3.2999999999999998,"15min":1.903},"min":{"1min":0.40799999999999997,"5min":0.159,"15min":0.159},"max":{"1min":10.962999999999999,"5min":98.373999999999995,"15min":98.373999999999995},"last":0.29999999999999999},{"interface":"front","average":{"1min":3.5289999999999999,"5min":3.609,"15min":2.246},"min":{"1min":0.505,"5min":0.17299999999999999,"15min":0.153},"max":{"1min":15.037000000000001,"5min":98.257999999999996,"15min":98.257999999999996},"last":0.34999999999999998}]},{"osd":2,"last update":"Wed Sep 11 08:30:08 2024","interfaces":[{"interface":"back","average":{"1min":1.4259999999999999,"5min":2.0569999999999999,"15min":1.5920000000000001},"min":{"1min":0.29999999999999999,"5min":0.19,"15min":0.19},"max":{"1min":5.2089999999999996,"5min":46.417000000000002,"15min":46.417000000000002},"last":0.27300000000000002},{"interface":"front","average":{"1min":2.5510000000000002,"5min":2.1240000000000001,"15min":1.6359999999999999},"min":{"1min":0.23000000000000001,"5min":0.22,"15min":0.22},"max":{"1min":13.331,"5min":46.438000000000002,"15min":46.438000000000002},"last":0.33700000000000002}]},{"osd":3,"last update":"Wed Sep 11 08:29:30 2024","interfaces":[{"interface":"back","average":{"1min":2.1459999999999999,"5min":1.5569999999999999,"15min":1.95},"min":{"1min":0.30499999999999999,"5min":0.222,"15min":0.222},"max":{"1min":9.5180000000000007,"5min":9.5180000000000007,"15min":172.149},"last":0.36899999999999999},{"interface":"front","average":{"1min":1.754,"5min":1.647,"15min":1.9239999999999999},"min":{"1min":0.28499999999999998,"5min":0.20599999999999999,"15min":0.20300000000000001},"max":{"1min":8.5109999999999992,"5min":9.6170000000000009,"15min":174.899},"last":0.32400000000000001}]},{"osd":4,"last update":"Wed Sep 11 08:29:55 2024","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":1.2849999999999999,"15min":1.2210000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.252},"max":{"1min":2.6110000000000002,"5min":5.1609999999999996,"15min":9.6560000000000006},"last":0.622},{"interface":"front","average":{"1min":1.0069999999999999,"5min":1.179,"15min":1.175},"min":{"1min":0.33400000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":2.6749999999999998,"5min":7.6630000000000003,"15min":8.3510000000000009},"last":0.56000000000000005}]},{"osd":5,"last update":"Wed Sep 11 08:29:35 2024","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.341,"15min":1.7969999999999999},"min":{"1min":0.28499999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":2.7000000000000002,"5min":7.7030000000000003,"15min":7.7030000000000003},"last":0.53100000000000003},{"interface":"front","average":{"1min":1.226,"5min":1.266,"15min":1.633},"min":{"1min":0.36399999999999999,"5min":0.309,"15min":0.309},"max":{"1min":4.2119999999999997,"5min":7.718,"15min":7.718},"last":0.64000000000000001}]},{"osd":6,"last update":"Wed Sep 11 08:29:35 2024","interfaces":[{"interface":"back","average":{"1min":1.4650000000000001,"5min":1.518,"15min":1.2909999999999999},"min":{"1min":0.317,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":4.0810000000000004,"5min":9.7029999999999994,"15min":9.7029999999999994},"last":0.65100000000000002},{"interface":"front","average":{"1min":1.242,"5min":1.3200000000000001,"15min":1.2609999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.28299999999999997},"max":{"1min":4.1559999999999997,"5min":5.3419999999999996,"15min":7.4820000000000002},"last":0.57099999999999995}]},{"osd":7,"last update":"Wed Sep 11 08:29:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1299999999999999,"5min":1.3220000000000001,"15min":1.268},"min":{"1min":0.34499999999999997,"5min":0.34399999999999997,"15min":0.30399999999999999},"max":{"1min":3.1459999999999999,"5min":5.8049999999999997,"15min":7.5350000000000001},"last":0.59799999999999998},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.296,"15min":1.3540000000000001},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.32000000000000001},"max":{"1min":5.6829999999999998,"5min":5.6829999999999998,"15min":13.914},"last":0.66100000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3425,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T08:30:31.896 INFO:tasks.thrashosds.thrasher:Setting 5.1d to [0, 5, 7] 2024-09-11T08:30:31.897 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '5.1d', '0', '5', '7'] 2024-09-11T08:30:31.897 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd pg-upmap 5.1d 0 5 7 2024-09-11T08:30:32.444 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:32 smithi150 ceph-mon[29276]: 5.19 scrub starts 2024-09-11T08:30:32.444 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:32 smithi150 ceph-mon[29276]: 5.19 scrub ok 2024-09-11T08:30:32.444 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:32 smithi150 ceph-mon[29276]: from='client.16161 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:30:32.444 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:32 smithi150 ceph-mon[29276]: pgmap v1052: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-11T08:30:32.445 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:32 smithi150 ceph-mon[29276]: 4.1f scrub starts 2024-09-11T08:30:32.445 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:32 smithi150 ceph-mon[29276]: 4.1f scrub ok 2024-09-11T08:30:32.445 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:32 smithi150 ceph-mon[29276]: osdmap e451: 8 total, 8 up, 8 in 2024-09-11T08:30:32.445 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:32 smithi150 ceph-mon[29276]: 4.b scrub starts 2024-09-11T08:30:32.445 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:32 smithi150 ceph-mon[29276]: 4.b scrub ok 2024-09-11T08:30:32.445 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:32 smithi150 ceph-mon[29276]: 4.14 scrub starts 2024-09-11T08:30:32.445 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:32 smithi150 ceph-mon[29276]: 4.14 scrub ok 2024-09-11T08:30:32.445 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:32 smithi150 ceph-mon[39612]: 5.19 scrub starts 2024-09-11T08:30:32.445 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:32 smithi150 ceph-mon[39612]: 5.19 scrub ok 2024-09-11T08:30:32.445 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:32 smithi150 ceph-mon[39612]: from='client.16161 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:30:32.445 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:32 smithi150 ceph-mon[39612]: pgmap v1052: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-11T08:30:32.446 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:32 smithi150 ceph-mon[39612]: 4.1f scrub starts 2024-09-11T08:30:32.446 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:32 smithi150 ceph-mon[39612]: 4.1f scrub ok 2024-09-11T08:30:32.446 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:32 smithi150 ceph-mon[39612]: osdmap e451: 8 total, 8 up, 8 in 2024-09-11T08:30:32.446 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:32 smithi150 ceph-mon[39612]: 4.b scrub starts 2024-09-11T08:30:32.446 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:32 smithi150 ceph-mon[39612]: 4.b scrub ok 2024-09-11T08:30:32.446 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:32 smithi150 ceph-mon[39612]: 4.14 scrub starts 2024-09-11T08:30:32.446 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:32 smithi150 ceph-mon[39612]: 4.14 scrub ok 2024-09-11T08:30:32.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:32 smithi181 ceph-mon[32701]: 5.19 scrub starts 2024-09-11T08:30:32.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:32 smithi181 ceph-mon[32701]: 5.19 scrub ok 2024-09-11T08:30:32.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:32 smithi181 ceph-mon[32701]: from='client.16161 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:30:32.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:32 smithi181 ceph-mon[32701]: pgmap v1052: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-11T08:30:32.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:32 smithi181 ceph-mon[32701]: 4.1f scrub starts 2024-09-11T08:30:32.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:32 smithi181 ceph-mon[32701]: 4.1f scrub ok 2024-09-11T08:30:32.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:32 smithi181 ceph-mon[32701]: osdmap e451: 8 total, 8 up, 8 in 2024-09-11T08:30:32.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:32 smithi181 ceph-mon[32701]: 4.b scrub starts 2024-09-11T08:30:32.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:32 smithi181 ceph-mon[32701]: 4.b scrub ok 2024-09-11T08:30:32.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:32 smithi181 ceph-mon[32701]: 4.14 scrub starts 2024-09-11T08:30:32.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:32 smithi181 ceph-mon[32701]: 4.14 scrub ok 2024-09-11T08:30:32.828 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:30:32.869 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-11T08:30:32.869 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:30:33.392 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:33 smithi150 ceph-mon[39612]: 4.9 scrub starts 2024-09-11T08:30:33.392 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:33 smithi150 ceph-mon[39612]: 4.9 scrub ok 2024-09-11T08:30:33.392 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:33 smithi150 ceph-mon[39612]: 4.a deep-scrub starts 2024-09-11T08:30:33.392 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:33 smithi150 ceph-mon[39612]: 4.a deep-scrub ok 2024-09-11T08:30:33.392 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:33 smithi150 ceph-mon[39612]: 4.19 scrub starts 2024-09-11T08:30:33.392 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:33 smithi150 ceph-mon[39612]: 4.19 scrub ok 2024-09-11T08:30:33.392 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:33 smithi150 ceph-mon[29276]: 4.9 scrub starts 2024-09-11T08:30:33.392 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:33 smithi150 ceph-mon[29276]: 4.9 scrub ok 2024-09-11T08:30:33.392 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:33 smithi150 ceph-mon[29276]: 4.a deep-scrub starts 2024-09-11T08:30:33.393 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:33 smithi150 ceph-mon[29276]: 4.a deep-scrub ok 2024-09-11T08:30:33.393 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:33 smithi150 ceph-mon[29276]: 4.19 scrub starts 2024-09-11T08:30:33.393 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:33 smithi150 ceph-mon[29276]: 4.19 scrub ok 2024-09-11T08:30:33.482 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:33 smithi181 ceph-mon[32701]: 4.9 scrub starts 2024-09-11T08:30:33.482 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:33 smithi181 ceph-mon[32701]: 4.9 scrub ok 2024-09-11T08:30:33.482 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:33 smithi181 ceph-mon[32701]: 4.a deep-scrub starts 2024-09-11T08:30:33.482 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:33 smithi181 ceph-mon[32701]: 4.a deep-scrub ok 2024-09-11T08:30:33.482 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:33 smithi181 ceph-mon[32701]: 4.19 scrub starts 2024-09-11T08:30:33.482 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:33 smithi181 ceph-mon[32701]: 4.19 scrub ok 2024-09-11T08:30:33.507 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-11T08:30:33.752 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:30:34.320 INFO:teuthology.orchestra.run.smithi150.stderr:nodeep-scrub is unset 2024-09-11T08:30:34.556 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:34 smithi150 ceph-mon[39612]: pgmap v1054: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:30:34.556 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:34 smithi150 ceph-mon[39612]: 4.e scrub starts 2024-09-11T08:30:34.556 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:34 smithi150 ceph-mon[39612]: 4.e scrub ok 2024-09-11T08:30:34.556 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:34 smithi150 ceph-mon[39612]: 6.9 scrub starts 2024-09-11T08:30:34.556 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:34 smithi150 ceph-mon[39612]: 6.9 scrub ok 2024-09-11T08:30:34.557 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:34 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/123561174' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:30:34.557 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:34 smithi150 ceph-mon[29276]: pgmap v1054: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:30:34.557 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:34 smithi150 ceph-mon[29276]: 4.e scrub starts 2024-09-11T08:30:34.557 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:34 smithi150 ceph-mon[29276]: 4.e scrub ok 2024-09-11T08:30:34.557 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:34 smithi150 ceph-mon[29276]: 6.9 scrub starts 2024-09-11T08:30:34.557 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:34 smithi150 ceph-mon[29276]: 6.9 scrub ok 2024-09-11T08:30:34.557 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:34 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/123561174' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:30:34.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:34 smithi181 ceph-mon[32701]: pgmap v1054: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:30:34.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:34 smithi181 ceph-mon[32701]: 4.e scrub starts 2024-09-11T08:30:34.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:34 smithi181 ceph-mon[32701]: 4.e scrub ok 2024-09-11T08:30:34.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:34 smithi181 ceph-mon[32701]: 6.9 scrub starts 2024-09-11T08:30:34.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:34 smithi181 ceph-mon[32701]: 6.9 scrub ok 2024-09-11T08:30:34.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:34 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/123561174' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-11T08:30:35.412 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:30:35 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:30:35] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:30:35.412 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:35 smithi181 ceph-mon[32701]: 6.d scrub starts 2024-09-11T08:30:35.412 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:35 smithi181 ceph-mon[32701]: 6.d scrub ok 2024-09-11T08:30:35.412 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:35 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/123561174' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:30:35.412 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:35 smithi181 ceph-mon[32701]: osdmap e452: 8 total, 8 up, 8 in 2024-09-11T08:30:35.412 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:35 smithi181 ceph-mon[32701]: 3.10 scrub starts 2024-09-11T08:30:35.412 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:35 smithi181 ceph-mon[32701]: 3.10 scrub ok 2024-09-11T08:30:35.468 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-11T08:30:35.468 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:30:35.622 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:35 smithi150 ceph-mon[39612]: 6.d scrub starts 2024-09-11T08:30:35.622 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:35 smithi150 ceph-mon[39612]: 6.d scrub ok 2024-09-11T08:30:35.622 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:35 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/123561174' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:30:35.622 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:35 smithi150 ceph-mon[39612]: osdmap e452: 8 total, 8 up, 8 in 2024-09-11T08:30:35.623 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:35 smithi150 ceph-mon[39612]: 3.10 scrub starts 2024-09-11T08:30:35.623 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:35 smithi150 ceph-mon[39612]: 3.10 scrub ok 2024-09-11T08:30:35.623 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:35 smithi150 ceph-mon[29276]: 6.d scrub starts 2024-09-11T08:30:35.623 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:35 smithi150 ceph-mon[29276]: 6.d scrub ok 2024-09-11T08:30:35.623 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:35 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/123561174' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T08:30:35.623 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:35 smithi150 ceph-mon[29276]: osdmap e452: 8 total, 8 up, 8 in 2024-09-11T08:30:35.623 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:35 smithi150 ceph-mon[29276]: 3.10 scrub starts 2024-09-11T08:30:35.623 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:35 smithi150 ceph-mon[29276]: 3.10 scrub ok 2024-09-11T08:30:36.123 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-11T08:30:36.332 INFO:teuthology.orchestra.run.smithi150.stderr:set 5.1d pg_upmap mapping to [0,5,7] 2024-09-11T08:30:36.371 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:30:36.641 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:36 smithi181 ceph-mon[32701]: pgmap v1056: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:36.641 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:36 smithi181 ceph-mon[32701]: 3.3 scrub starts 2024-09-11T08:30:36.641 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:36 smithi181 ceph-mon[32701]: 3.3 scrub ok 2024-09-11T08:30:36.641 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:36 smithi181 ceph-mon[32701]: 4.1c scrub starts 2024-09-11T08:30:36.641 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:36 smithi181 ceph-mon[32701]: 4.1c scrub ok 2024-09-11T08:30:36.641 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:36 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2111680572' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.1d", "id": [0, 5, 7]}]: dispatch 2024-09-11T08:30:36.641 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:36 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.1d", "id": [0, 5, 7]}]: dispatch 2024-09-11T08:30:36.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:36 smithi150 ceph-mon[29276]: pgmap v1056: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:36.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:36 smithi150 ceph-mon[29276]: 3.3 scrub starts 2024-09-11T08:30:36.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:36 smithi150 ceph-mon[29276]: 3.3 scrub ok 2024-09-11T08:30:36.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:36 smithi150 ceph-mon[29276]: 4.1c scrub starts 2024-09-11T08:30:36.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:36 smithi150 ceph-mon[29276]: 4.1c scrub ok 2024-09-11T08:30:36.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:36 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2111680572' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.1d", "id": [0, 5, 7]}]: dispatch 2024-09-11T08:30:36.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:36 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.1d", "id": [0, 5, 7]}]: dispatch 2024-09-11T08:30:36.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:36 smithi150 ceph-mon[39612]: pgmap v1056: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:36.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:36 smithi150 ceph-mon[39612]: 3.3 scrub starts 2024-09-11T08:30:36.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:36 smithi150 ceph-mon[39612]: 3.3 scrub ok 2024-09-11T08:30:36.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:36 smithi150 ceph-mon[39612]: 4.1c scrub starts 2024-09-11T08:30:36.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:36 smithi150 ceph-mon[39612]: 4.1c scrub ok 2024-09-11T08:30:36.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:36 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2111680572' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.1d", "id": [0, 5, 7]}]: dispatch 2024-09-11T08:30:36.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:36 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.1d", "id": [0, 5, 7]}]: dispatch 2024-09-11T08:30:37.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:37 smithi150 ceph-mon[29276]: 6.19 scrub starts 2024-09-11T08:30:37.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:37 smithi150 ceph-mon[29276]: 6.19 scrub ok 2024-09-11T08:30:37.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:37 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.1d", "id": [0, 5, 7]}]': finished 2024-09-11T08:30:37.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:37 smithi150 ceph-mon[29276]: osdmap e453: 8 total, 8 up, 8 in 2024-09-11T08:30:37.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:37 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:30:37.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:37 smithi150 ceph-mon[29276]: 6.2 deep-scrub starts 2024-09-11T08:30:37.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:37 smithi150 ceph-mon[29276]: 6.2 deep-scrub ok 2024-09-11T08:30:37.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:37 smithi150 ceph-mon[29276]: osdmap e454: 8 total, 8 up, 8 in 2024-09-11T08:30:37.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:37 smithi150 ceph-mon[39612]: 6.19 scrub starts 2024-09-11T08:30:37.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:37 smithi150 ceph-mon[39612]: 6.19 scrub ok 2024-09-11T08:30:37.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:37 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.1d", "id": [0, 5, 7]}]': finished 2024-09-11T08:30:37.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:37 smithi150 ceph-mon[39612]: osdmap e453: 8 total, 8 up, 8 in 2024-09-11T08:30:37.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:37 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:30:37.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:37 smithi150 ceph-mon[39612]: 6.2 deep-scrub starts 2024-09-11T08:30:37.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:37 smithi150 ceph-mon[39612]: 6.2 deep-scrub ok 2024-09-11T08:30:37.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:37 smithi150 ceph-mon[39612]: osdmap e454: 8 total, 8 up, 8 in 2024-09-11T08:30:37.555 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:37 smithi181 ceph-mon[32701]: 6.19 scrub starts 2024-09-11T08:30:37.555 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:37 smithi181 ceph-mon[32701]: 6.19 scrub ok 2024-09-11T08:30:37.555 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:37 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.1d", "id": [0, 5, 7]}]': finished 2024-09-11T08:30:37.555 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:37 smithi181 ceph-mon[32701]: osdmap e453: 8 total, 8 up, 8 in 2024-09-11T08:30:37.556 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:37 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:30:37.556 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:37 smithi181 ceph-mon[32701]: 6.2 deep-scrub starts 2024-09-11T08:30:37.556 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:37 smithi181 ceph-mon[32701]: 6.2 deep-scrub ok 2024-09-11T08:30:37.556 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:37 smithi181 ceph-mon[32701]: osdmap e454: 8 total, 8 up, 8 in 2024-09-11T08:30:38.058 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-11T08:30:38.058 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:30:38.654 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:38 smithi181 ceph-mon[32701]: 6.1d scrub starts 2024-09-11T08:30:38.654 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:38 smithi181 ceph-mon[32701]: 6.1d scrub ok 2024-09-11T08:30:38.654 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:38 smithi181 ceph-mon[32701]: pgmap v1058: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:38.654 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:38 smithi181 ceph-mon[32701]: 5.1 scrub starts 2024-09-11T08:30:38.654 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:38 smithi181 ceph-mon[32701]: 5.1 scrub ok 2024-09-11T08:30:38.654 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:38 smithi181 ceph-mon[32701]: 5.4 scrub starts 2024-09-11T08:30:38.654 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:38 smithi181 ceph-mon[32701]: 5.4 scrub ok 2024-09-11T08:30:38.679 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-11T08:30:38.738 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:30:38 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:30:38] "GET /metrics HTTP/1.1" 200 222241 "" "Prometheus/2.43.0" 2024-09-11T08:30:38.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:38 smithi150 ceph-mon[29276]: 6.1d scrub starts 2024-09-11T08:30:38.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:38 smithi150 ceph-mon[29276]: 6.1d scrub ok 2024-09-11T08:30:38.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:38 smithi150 ceph-mon[29276]: pgmap v1058: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:38.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:38 smithi150 ceph-mon[29276]: 5.1 scrub starts 2024-09-11T08:30:38.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:38 smithi150 ceph-mon[29276]: 5.1 scrub ok 2024-09-11T08:30:38.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:38 smithi150 ceph-mon[29276]: 5.4 scrub starts 2024-09-11T08:30:38.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:38 smithi150 ceph-mon[29276]: 5.4 scrub ok 2024-09-11T08:30:38.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:38 smithi150 ceph-mon[39612]: 6.1d scrub starts 2024-09-11T08:30:38.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:38 smithi150 ceph-mon[39612]: 6.1d scrub ok 2024-09-11T08:30:38.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:38 smithi150 ceph-mon[39612]: pgmap v1058: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:38.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:38 smithi150 ceph-mon[39612]: 5.1 scrub starts 2024-09-11T08:30:38.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:38 smithi150 ceph-mon[39612]: 5.1 scrub ok 2024-09-11T08:30:38.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:38 smithi150 ceph-mon[39612]: 5.4 scrub starts 2024-09-11T08:30:38.740 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:38 smithi150 ceph-mon[39612]: 5.4 scrub ok 2024-09-11T08:30:38.922 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:30:39.323 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:30:39 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:30:39.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:39 smithi150 ceph-mon[29276]: 5.b deep-scrub starts 2024-09-11T08:30:39.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:39 smithi150 ceph-mon[29276]: 5.b deep-scrub ok 2024-09-11T08:30:39.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:39 smithi150 ceph-mon[29276]: 4.1b scrub starts 2024-09-11T08:30:39.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:39 smithi150 ceph-mon[29276]: 4.1b scrub ok 2024-09-11T08:30:39.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:39 smithi150 ceph-mon[39612]: 5.b deep-scrub starts 2024-09-11T08:30:39.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:39 smithi150 ceph-mon[39612]: 5.b deep-scrub ok 2024-09-11T08:30:39.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:39 smithi150 ceph-mon[39612]: 4.1b scrub starts 2024-09-11T08:30:39.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:39 smithi150 ceph-mon[39612]: 4.1b scrub ok 2024-09-11T08:30:39.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:39 smithi181 ceph-mon[32701]: 5.b deep-scrub starts 2024-09-11T08:30:39.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:39 smithi181 ceph-mon[32701]: 5.b deep-scrub ok 2024-09-11T08:30:39.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:39 smithi181 ceph-mon[32701]: 4.1b scrub starts 2024-09-11T08:30:39.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:39 smithi181 ceph-mon[32701]: 4.1b scrub ok 2024-09-11T08:30:40.484 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:40 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:30:40.485 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:40 smithi181 ceph-mon[32701]: pgmap v1060: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:40.707 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-11T08:30:40.707 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:30:40.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:40 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:30:40.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:40 smithi150 ceph-mon[29276]: pgmap v1060: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:40.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:40 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:30:40.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:40 smithi150 ceph-mon[39612]: pgmap v1060: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:41.419 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-11T08:30:41.663 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:30:42.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:41 smithi181 ceph-mon[32701]: 4.5 scrub starts 2024-09-11T08:30:42.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:41 smithi181 ceph-mon[32701]: 4.5 scrub ok 2024-09-11T08:30:42.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:41 smithi181 ceph-mon[32701]: pgmap v1061: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 738 B/s rd, 0 op/s 2024-09-11T08:30:42.190 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-11T08:30:42.220 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:41 smithi150 ceph-mon[29276]: 4.5 scrub starts 2024-09-11T08:30:42.220 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:41 smithi150 ceph-mon[29276]: 4.5 scrub ok 2024-09-11T08:30:42.220 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:41 smithi150 ceph-mon[29276]: pgmap v1061: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 738 B/s rd, 0 op/s 2024-09-11T08:30:42.221 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:41 smithi150 ceph-mon[39612]: 4.5 scrub starts 2024-09-11T08:30:42.221 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:41 smithi150 ceph-mon[39612]: 4.5 scrub ok 2024-09-11T08:30:42.221 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:41 smithi150 ceph-mon[39612]: pgmap v1061: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 738 B/s rd, 0 op/s 2024-09-11T08:30:42.451 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:30:42.831 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:42 smithi150 ceph-mon[29276]: 6.1c scrub starts 2024-09-11T08:30:42.831 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:42 smithi150 ceph-mon[29276]: 6.1c scrub ok 2024-09-11T08:30:42.831 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:42 smithi150 ceph-mon[29276]: 5.0 scrub starts 2024-09-11T08:30:42.831 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:42 smithi150 ceph-mon[29276]: 5.0 scrub ok 2024-09-11T08:30:42.831 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:42 smithi150 ceph-mon[39612]: 6.1c scrub starts 2024-09-11T08:30:42.831 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:42 smithi150 ceph-mon[39612]: 6.1c scrub ok 2024-09-11T08:30:42.831 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:42 smithi150 ceph-mon[39612]: 5.0 scrub starts 2024-09-11T08:30:42.831 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:42 smithi150 ceph-mon[39612]: 5.0 scrub ok 2024-09-11T08:30:43.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:42 smithi181 ceph-mon[32701]: 6.1c scrub starts 2024-09-11T08:30:43.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:42 smithi181 ceph-mon[32701]: 6.1c scrub ok 2024-09-11T08:30:43.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:42 smithi181 ceph-mon[32701]: 5.0 scrub starts 2024-09-11T08:30:43.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:42 smithi181 ceph-mon[32701]: 5.0 scrub ok 2024-09-11T08:30:43.365 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-11T08:30:43.365 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T08:30:43.961 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:43 smithi150 ceph-mon[29276]: 5.d scrub starts 2024-09-11T08:30:43.961 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:43 smithi150 ceph-mon[29276]: 5.d scrub ok 2024-09-11T08:30:43.961 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:43 smithi150 ceph-mon[29276]: 6.1 scrub starts 2024-09-11T08:30:43.961 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:43 smithi150 ceph-mon[29276]: 6.1 scrub ok 2024-09-11T08:30:43.961 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:43 smithi150 ceph-mon[29276]: 3.c scrub starts 2024-09-11T08:30:43.961 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:43 smithi150 ceph-mon[29276]: 3.c scrub ok 2024-09-11T08:30:43.961 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:43 smithi150 ceph-mon[29276]: pgmap v1062: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:30:43.961 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:43 smithi150 ceph-mon[39612]: 5.d scrub starts 2024-09-11T08:30:43.961 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:43 smithi150 ceph-mon[39612]: 5.d scrub ok 2024-09-11T08:30:43.962 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:43 smithi150 ceph-mon[39612]: 6.1 scrub starts 2024-09-11T08:30:43.962 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:43 smithi150 ceph-mon[39612]: 6.1 scrub ok 2024-09-11T08:30:43.962 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:43 smithi150 ceph-mon[39612]: 3.c scrub starts 2024-09-11T08:30:43.962 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:43 smithi150 ceph-mon[39612]: 3.c scrub ok 2024-09-11T08:30:43.962 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:43 smithi150 ceph-mon[39612]: pgmap v1062: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:30:44.058 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:43 smithi181 ceph-mon[32701]: 5.d scrub starts 2024-09-11T08:30:44.058 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:43 smithi181 ceph-mon[32701]: 5.d scrub ok 2024-09-11T08:30:44.059 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:43 smithi181 ceph-mon[32701]: 6.1 scrub starts 2024-09-11T08:30:44.059 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:43 smithi181 ceph-mon[32701]: 6.1 scrub ok 2024-09-11T08:30:44.059 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:43 smithi181 ceph-mon[32701]: 3.c scrub starts 2024-09-11T08:30:44.059 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:43 smithi181 ceph-mon[32701]: 3.c scrub ok 2024-09-11T08:30:44.059 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:43 smithi181 ceph-mon[32701]: pgmap v1062: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:30:44.221 INFO:teuthology.orchestra.run.smithi150.stdout:mon.a: {} 2024-09-11T08:30:44.221 INFO:teuthology.orchestra.run.smithi150.stdout:mon.b: {} 2024-09-11T08:30:44.221 INFO:teuthology.orchestra.run.smithi150.stdout:mon.c: {} 2024-09-11T08:30:44.222 INFO:teuthology.orchestra.run.smithi150.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-11T08:30:44.222 INFO:teuthology.orchestra.run.smithi150.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-11T08:30:44.222 INFO:teuthology.orchestra.run.smithi150.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-11T08:30:44.881 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:44 smithi150 ceph-mon[29276]: 5.c scrub starts 2024-09-11T08:30:44.882 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:44 smithi150 ceph-mon[29276]: 5.c scrub ok 2024-09-11T08:30:44.882 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:44 smithi150 ceph-mon[29276]: 3.13 scrub starts 2024-09-11T08:30:44.882 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:44 smithi150 ceph-mon[29276]: 3.13 scrub ok 2024-09-11T08:30:44.882 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:44 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2864813872' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-11T08:30:44.882 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:44 smithi150 ceph-mon[39612]: 5.c scrub starts 2024-09-11T08:30:44.882 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:44 smithi150 ceph-mon[39612]: 5.c scrub ok 2024-09-11T08:30:44.882 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:44 smithi150 ceph-mon[39612]: 3.13 scrub starts 2024-09-11T08:30:44.883 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:44 smithi150 ceph-mon[39612]: 3.13 scrub ok 2024-09-11T08:30:44.883 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:44 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2864813872' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-11T08:30:44.883 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd reweight 0 1 2024-09-11T08:30:45.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:44 smithi181 ceph-mon[32701]: 5.c scrub starts 2024-09-11T08:30:45.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:44 smithi181 ceph-mon[32701]: 5.c scrub ok 2024-09-11T08:30:45.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:44 smithi181 ceph-mon[32701]: 3.13 scrub starts 2024-09-11T08:30:45.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:44 smithi181 ceph-mon[32701]: 3.13 scrub ok 2024-09-11T08:30:45.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:44 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2864813872' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-11T08:30:45.172 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:30:45.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:30:45 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:30:45] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:30:45.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:45 smithi150 ceph-mon[29276]: 5.18 scrub starts 2024-09-11T08:30:45.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:45 smithi150 ceph-mon[29276]: 5.18 scrub ok 2024-09-11T08:30:45.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:45 smithi150 ceph-mon[29276]: 4.f scrub starts 2024-09-11T08:30:45.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:45 smithi150 ceph-mon[29276]: 4.f scrub ok 2024-09-11T08:30:45.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:45 smithi150 ceph-mon[29276]: pgmap v1063: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-11T08:30:45.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:45 smithi150 ceph-mon[39612]: 5.18 scrub starts 2024-09-11T08:30:45.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:45 smithi150 ceph-mon[39612]: 5.18 scrub ok 2024-09-11T08:30:45.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:45 smithi150 ceph-mon[39612]: 4.f scrub starts 2024-09-11T08:30:45.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:45 smithi150 ceph-mon[39612]: 4.f scrub ok 2024-09-11T08:30:45.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:45 smithi150 ceph-mon[39612]: pgmap v1063: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-11T08:30:46.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:45 smithi181 ceph-mon[32701]: 5.18 scrub starts 2024-09-11T08:30:46.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:45 smithi181 ceph-mon[32701]: 5.18 scrub ok 2024-09-11T08:30:46.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:45 smithi181 ceph-mon[32701]: 4.f scrub starts 2024-09-11T08:30:46.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:45 smithi181 ceph-mon[32701]: 4.f scrub ok 2024-09-11T08:30:46.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:45 smithi181 ceph-mon[32701]: pgmap v1063: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-11T08:30:46.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:46 smithi150 ceph-mon[39612]: 5.1d scrub starts 2024-09-11T08:30:46.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:46 smithi150 ceph-mon[39612]: 5.1d scrub ok 2024-09-11T08:30:46.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:46 smithi150 ceph-mon[39612]: 3.19 scrub starts 2024-09-11T08:30:46.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:46 smithi150 ceph-mon[39612]: 3.19 scrub ok 2024-09-11T08:30:46.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:46 smithi150 ceph-mon[39612]: 1.0 scrub starts 2024-09-11T08:30:46.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:46 smithi150 ceph-mon[39612]: 1.0 scrub ok 2024-09-11T08:30:46.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:46 smithi150 ceph-mon[29276]: 5.1d scrub starts 2024-09-11T08:30:46.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:46 smithi150 ceph-mon[29276]: 5.1d scrub ok 2024-09-11T08:30:46.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:46 smithi150 ceph-mon[29276]: 3.19 scrub starts 2024-09-11T08:30:46.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:46 smithi150 ceph-mon[29276]: 3.19 scrub ok 2024-09-11T08:30:46.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:46 smithi150 ceph-mon[29276]: 1.0 scrub starts 2024-09-11T08:30:46.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:46 smithi150 ceph-mon[29276]: 1.0 scrub ok 2024-09-11T08:30:47.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:46 smithi181 ceph-mon[32701]: 5.1d scrub starts 2024-09-11T08:30:47.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:46 smithi181 ceph-mon[32701]: 5.1d scrub ok 2024-09-11T08:30:47.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:46 smithi181 ceph-mon[32701]: 3.19 scrub starts 2024-09-11T08:30:47.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:46 smithi181 ceph-mon[32701]: 3.19 scrub ok 2024-09-11T08:30:47.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:46 smithi181 ceph-mon[32701]: 1.0 scrub starts 2024-09-11T08:30:47.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:46 smithi181 ceph-mon[32701]: 1.0 scrub ok 2024-09-11T08:30:47.817 INFO:teuthology.orchestra.run.smithi150.stderr:reweighted osd.0 to 1 (10000) 2024-09-11T08:30:48.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:47 smithi181 ceph-mon[32701]: 6.b scrub starts 2024-09-11T08:30:48.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:47 smithi181 ceph-mon[32701]: 6.b scrub ok 2024-09-11T08:30:48.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:47 smithi181 ceph-mon[32701]: 4.1 scrub starts 2024-09-11T08:30:48.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:47 smithi181 ceph-mon[32701]: 4.1 scrub ok 2024-09-11T08:30:48.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:47 smithi181 ceph-mon[32701]: 2.0 deep-scrub starts 2024-09-11T08:30:48.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:47 smithi181 ceph-mon[32701]: 2.0 deep-scrub ok 2024-09-11T08:30:48.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:47 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3447407729' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-11T08:30:48.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:47 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-11T08:30:48.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:47 smithi181 ceph-mon[32701]: pgmap v1064: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-11T08:30:48.183 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:47 smithi150 ceph-mon[29276]: 6.b scrub starts 2024-09-11T08:30:48.183 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:47 smithi150 ceph-mon[29276]: 6.b scrub ok 2024-09-11T08:30:48.184 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:47 smithi150 ceph-mon[29276]: 4.1 scrub starts 2024-09-11T08:30:48.184 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:47 smithi150 ceph-mon[29276]: 4.1 scrub ok 2024-09-11T08:30:48.184 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:47 smithi150 ceph-mon[29276]: 2.0 deep-scrub starts 2024-09-11T08:30:48.184 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:47 smithi150 ceph-mon[29276]: 2.0 deep-scrub ok 2024-09-11T08:30:48.184 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:47 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3447407729' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-11T08:30:48.184 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:47 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-11T08:30:48.184 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:47 smithi150 ceph-mon[29276]: pgmap v1064: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-11T08:30:48.184 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:47 smithi150 ceph-mon[39612]: 6.b scrub starts 2024-09-11T08:30:48.184 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:47 smithi150 ceph-mon[39612]: 6.b scrub ok 2024-09-11T08:30:48.184 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:47 smithi150 ceph-mon[39612]: 4.1 scrub starts 2024-09-11T08:30:48.184 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:47 smithi150 ceph-mon[39612]: 4.1 scrub ok 2024-09-11T08:30:48.185 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:47 smithi150 ceph-mon[39612]: 2.0 deep-scrub starts 2024-09-11T08:30:48.185 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:47 smithi150 ceph-mon[39612]: 2.0 deep-scrub ok 2024-09-11T08:30:48.185 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:47 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3447407729' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-11T08:30:48.185 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:47 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-11T08:30:48.185 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:47 smithi150 ceph-mon[39612]: pgmap v1064: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-11T08:30:48.707 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 0 1 2024-09-11T08:30:48.819 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:30:48 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:30:48] "GET /metrics HTTP/1.1" 200 222241 "" "Prometheus/2.43.0" 2024-09-11T08:30:48.964 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:30:49.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:48 smithi181 ceph-mon[32701]: 4.11 scrub starts 2024-09-11T08:30:49.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:48 smithi181 ceph-mon[32701]: 4.11 scrub ok 2024-09-11T08:30:49.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:48 smithi181 ceph-mon[32701]: 5.1f scrub starts 2024-09-11T08:30:49.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:48 smithi181 ceph-mon[32701]: 5.1f scrub ok 2024-09-11T08:30:49.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:48 smithi181 ceph-mon[32701]: 5.14 scrub starts 2024-09-11T08:30:49.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:48 smithi181 ceph-mon[32701]: 5.14 scrub ok 2024-09-11T08:30:49.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:48 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-11T08:30:49.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:48 smithi181 ceph-mon[32701]: osdmap e455: 8 total, 8 up, 8 in 2024-09-11T08:30:49.085 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:48 smithi150 ceph-mon[29276]: 4.11 scrub starts 2024-09-11T08:30:49.085 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:48 smithi150 ceph-mon[29276]: 4.11 scrub ok 2024-09-11T08:30:49.085 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:48 smithi150 ceph-mon[29276]: 5.1f scrub starts 2024-09-11T08:30:49.085 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:48 smithi150 ceph-mon[29276]: 5.1f scrub ok 2024-09-11T08:30:49.085 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:48 smithi150 ceph-mon[29276]: 5.14 scrub starts 2024-09-11T08:30:49.086 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:48 smithi150 ceph-mon[29276]: 5.14 scrub ok 2024-09-11T08:30:49.086 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:48 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-11T08:30:49.086 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:48 smithi150 ceph-mon[29276]: osdmap e455: 8 total, 8 up, 8 in 2024-09-11T08:30:49.086 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:48 smithi150 ceph-mon[39612]: 4.11 scrub starts 2024-09-11T08:30:49.086 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:48 smithi150 ceph-mon[39612]: 4.11 scrub ok 2024-09-11T08:30:49.086 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:48 smithi150 ceph-mon[39612]: 5.1f scrub starts 2024-09-11T08:30:49.086 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:48 smithi150 ceph-mon[39612]: 5.1f scrub ok 2024-09-11T08:30:49.086 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:48 smithi150 ceph-mon[39612]: 5.14 scrub starts 2024-09-11T08:30:49.086 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:48 smithi150 ceph-mon[39612]: 5.14 scrub ok 2024-09-11T08:30:49.086 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:48 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-11T08:30:49.086 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:48 smithi150 ceph-mon[39612]: osdmap e455: 8 total, 8 up, 8 in 2024-09-11T08:30:49.452 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:30:49 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:30:50.109 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:49 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:30:50.109 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:49 smithi150 ceph-mon[29276]: pgmap v1066: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-11T08:30:50.109 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:49 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:30:50.109 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:49 smithi150 ceph-mon[39612]: pgmap v1066: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-11T08:30:50.207 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:49 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:30:50.207 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:49 smithi181 ceph-mon[32701]: pgmap v1066: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-11T08:30:50.828 INFO:teuthology.orchestra.run.smithi150.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-11T08:30:51.183 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:50 smithi150 ceph-mon[29276]: 5.1b scrub starts 2024-09-11T08:30:51.183 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:50 smithi150 ceph-mon[29276]: 5.1b scrub ok 2024-09-11T08:30:51.183 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:50 smithi150 ceph-mon[29276]: 3.15 deep-scrub starts 2024-09-11T08:30:51.183 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:50 smithi150 ceph-mon[29276]: 3.15 deep-scrub ok 2024-09-11T08:30:51.183 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:50 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3818331101' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-09-11T08:30:51.183 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:50 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-09-11T08:30:51.184 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:50 smithi150 ceph-mon[39612]: 5.1b scrub starts 2024-09-11T08:30:51.184 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:50 smithi150 ceph-mon[39612]: 5.1b scrub ok 2024-09-11T08:30:51.184 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:50 smithi150 ceph-mon[39612]: 3.15 deep-scrub starts 2024-09-11T08:30:51.184 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:50 smithi150 ceph-mon[39612]: 3.15 deep-scrub ok 2024-09-11T08:30:51.184 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:50 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3818331101' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-09-11T08:30:51.184 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:50 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-09-11T08:30:51.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:50 smithi181 ceph-mon[32701]: 5.1b scrub starts 2024-09-11T08:30:51.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:50 smithi181 ceph-mon[32701]: 5.1b scrub ok 2024-09-11T08:30:51.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:50 smithi181 ceph-mon[32701]: 3.15 deep-scrub starts 2024-09-11T08:30:51.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:50 smithi181 ceph-mon[32701]: 3.15 deep-scrub ok 2024-09-11T08:30:51.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:50 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3818331101' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-09-11T08:30:51.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:50 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-09-11T08:30:51.492 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd reweight 1 1 2024-09-11T08:30:51.739 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:30:52.087 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:51 smithi150 ceph-mon[29276]: 4.10 scrub starts 2024-09-11T08:30:52.087 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:51 smithi150 ceph-mon[29276]: 4.10 scrub ok 2024-09-11T08:30:52.087 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:51 smithi150 ceph-mon[29276]: 5.2 deep-scrub starts 2024-09-11T08:30:52.087 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:51 smithi150 ceph-mon[29276]: 5.2 deep-scrub ok 2024-09-11T08:30:52.087 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:51 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-11T08:30:52.087 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:51 smithi150 ceph-mon[29276]: osdmap e456: 8 total, 8 up, 8 in 2024-09-11T08:30:52.087 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:51 smithi150 ceph-mon[29276]: pgmap v1068: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-11T08:30:52.087 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:51 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:30:52.088 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:51 smithi150 ceph-mon[39612]: 4.10 scrub starts 2024-09-11T08:30:52.088 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:51 smithi150 ceph-mon[39612]: 4.10 scrub ok 2024-09-11T08:30:52.088 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:51 smithi150 ceph-mon[39612]: 5.2 deep-scrub starts 2024-09-11T08:30:52.088 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:51 smithi150 ceph-mon[39612]: 5.2 deep-scrub ok 2024-09-11T08:30:52.088 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:51 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-11T08:30:52.088 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:51 smithi150 ceph-mon[39612]: osdmap e456: 8 total, 8 up, 8 in 2024-09-11T08:30:52.088 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:51 smithi150 ceph-mon[39612]: pgmap v1068: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-11T08:30:52.088 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:51 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:30:52.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:51 smithi181 ceph-mon[32701]: 4.10 scrub starts 2024-09-11T08:30:52.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:51 smithi181 ceph-mon[32701]: 4.10 scrub ok 2024-09-11T08:30:52.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:51 smithi181 ceph-mon[32701]: 5.2 deep-scrub starts 2024-09-11T08:30:52.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:51 smithi181 ceph-mon[32701]: 5.2 deep-scrub ok 2024-09-11T08:30:52.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:51 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-11T08:30:52.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:51 smithi181 ceph-mon[32701]: osdmap e456: 8 total, 8 up, 8 in 2024-09-11T08:30:52.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:51 smithi181 ceph-mon[32701]: pgmap v1068: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-11T08:30:52.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:51 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:30:53.035 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:52 smithi150 ceph-mon[29276]: 5.e scrub starts 2024-09-11T08:30:53.035 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:52 smithi150 ceph-mon[29276]: 5.e scrub ok 2024-09-11T08:30:53.035 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:52 smithi150 ceph-mon[29276]: 6.14 deep-scrub starts 2024-09-11T08:30:53.035 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:52 smithi150 ceph-mon[29276]: 6.14 deep-scrub ok 2024-09-11T08:30:53.035 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:52 smithi150 ceph-mon[29276]: 4.4 scrub starts 2024-09-11T08:30:53.035 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:52 smithi150 ceph-mon[29276]: 4.4 scrub ok 2024-09-11T08:30:53.035 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:52 smithi150 ceph-mon[39612]: 5.e scrub starts 2024-09-11T08:30:53.035 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:52 smithi150 ceph-mon[39612]: 5.e scrub ok 2024-09-11T08:30:53.035 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:52 smithi150 ceph-mon[39612]: 6.14 deep-scrub starts 2024-09-11T08:30:53.036 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:52 smithi150 ceph-mon[39612]: 6.14 deep-scrub ok 2024-09-11T08:30:53.036 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:52 smithi150 ceph-mon[39612]: 4.4 scrub starts 2024-09-11T08:30:53.036 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:52 smithi150 ceph-mon[39612]: 4.4 scrub ok 2024-09-11T08:30:53.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:52 smithi181 ceph-mon[32701]: 5.e scrub starts 2024-09-11T08:30:53.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:52 smithi181 ceph-mon[32701]: 5.e scrub ok 2024-09-11T08:30:53.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:52 smithi181 ceph-mon[32701]: 6.14 deep-scrub starts 2024-09-11T08:30:53.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:52 smithi181 ceph-mon[32701]: 6.14 deep-scrub ok 2024-09-11T08:30:53.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:52 smithi181 ceph-mon[32701]: 4.4 scrub starts 2024-09-11T08:30:53.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:52 smithi181 ceph-mon[32701]: 4.4 scrub ok 2024-09-11T08:30:53.843 INFO:teuthology.orchestra.run.smithi150.stderr:reweighted osd.1 to 1 (10000) 2024-09-11T08:30:54.190 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:53 smithi150 ceph-mon[29276]: 4.1d scrub starts 2024-09-11T08:30:54.190 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:53 smithi150 ceph-mon[29276]: 4.1d scrub ok 2024-09-11T08:30:54.190 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:53 smithi150 ceph-mon[29276]: pgmap v1069: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:30:54.190 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:53 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1017652539' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-11T08:30:54.190 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:53 smithi150 ceph-mon[39612]: 4.1d scrub starts 2024-09-11T08:30:54.190 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:53 smithi150 ceph-mon[39612]: 4.1d scrub ok 2024-09-11T08:30:54.190 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:53 smithi150 ceph-mon[39612]: pgmap v1069: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:30:54.190 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:53 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1017652539' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-11T08:30:54.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:53 smithi181 ceph-mon[32701]: 4.1d scrub starts 2024-09-11T08:30:54.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:53 smithi181 ceph-mon[32701]: 4.1d scrub ok 2024-09-11T08:30:54.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:53 smithi181 ceph-mon[32701]: pgmap v1069: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:30:54.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:53 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1017652539' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-11T08:30:54.534 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 1 1 2024-09-11T08:30:54.783 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:30:55.126 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:54 smithi150 ceph-mon[29276]: 6.18 scrub starts 2024-09-11T08:30:55.126 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:54 smithi150 ceph-mon[29276]: 6.18 scrub ok 2024-09-11T08:30:55.126 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:54 smithi150 ceph-mon[29276]: 3.f scrub starts 2024-09-11T08:30:55.127 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:54 smithi150 ceph-mon[29276]: 3.f scrub ok 2024-09-11T08:30:55.127 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:54 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1017652539' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-11T08:30:55.127 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:54 smithi150 ceph-mon[29276]: osdmap e457: 8 total, 8 up, 8 in 2024-09-11T08:30:55.127 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:54 smithi150 ceph-mon[39612]: 6.18 scrub starts 2024-09-11T08:30:55.127 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:54 smithi150 ceph-mon[39612]: 6.18 scrub ok 2024-09-11T08:30:55.127 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:54 smithi150 ceph-mon[39612]: 3.f scrub starts 2024-09-11T08:30:55.127 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:54 smithi150 ceph-mon[39612]: 3.f scrub ok 2024-09-11T08:30:55.127 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:54 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1017652539' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-11T08:30:55.127 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:54 smithi150 ceph-mon[39612]: osdmap e457: 8 total, 8 up, 8 in 2024-09-11T08:30:55.143 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:54 smithi181 ceph-mon[32701]: 6.18 scrub starts 2024-09-11T08:30:55.143 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:54 smithi181 ceph-mon[32701]: 6.18 scrub ok 2024-09-11T08:30:55.143 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:54 smithi181 ceph-mon[32701]: 3.f scrub starts 2024-09-11T08:30:55.143 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:54 smithi181 ceph-mon[32701]: 3.f scrub ok 2024-09-11T08:30:55.143 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:54 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1017652539' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-11T08:30:55.143 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:54 smithi181 ceph-mon[32701]: osdmap e457: 8 total, 8 up, 8 in 2024-09-11T08:30:55.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:30:55 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:30:55] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:30:56.208 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:55 smithi150 ceph-mon[29276]: pgmap v1071: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 688 B/s rd, 0 op/s 2024-09-11T08:30:56.209 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:55 smithi150 ceph-mon[39612]: pgmap v1071: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 688 B/s rd, 0 op/s 2024-09-11T08:30:56.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:55 smithi181 ceph-mon[32701]: pgmap v1071: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 688 B/s rd, 0 op/s 2024-09-11T08:30:56.864 INFO:teuthology.orchestra.run.smithi150.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-11T08:30:57.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:56 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/484926396' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-09-11T08:30:57.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:56 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-09-11T08:30:57.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:56 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/484926396' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-09-11T08:30:57.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:56 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-09-11T08:30:57.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:56 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/484926396' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-09-11T08:30:57.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:56 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-09-11T08:30:57.736 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd reweight 2 1 2024-09-11T08:30:57.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:57 smithi150 ceph-mon[29276]: 6.c scrub starts 2024-09-11T08:30:57.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:57 smithi150 ceph-mon[29276]: 6.c scrub ok 2024-09-11T08:30:57.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:57 smithi150 ceph-mon[29276]: 5.5 deep-scrub starts 2024-09-11T08:30:57.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:57 smithi150 ceph-mon[29276]: 5.5 deep-scrub ok 2024-09-11T08:30:57.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:57 smithi150 ceph-mon[29276]: 5.12 scrub starts 2024-09-11T08:30:57.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:57 smithi150 ceph-mon[29276]: 5.12 scrub ok 2024-09-11T08:30:57.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:57 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-11T08:30:57.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:57 smithi150 ceph-mon[29276]: osdmap e458: 8 total, 8 up, 8 in 2024-09-11T08:30:57.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:57 smithi150 ceph-mon[29276]: pgmap v1073: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 796 B/s rd, 0 op/s 2024-09-11T08:30:57.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:57 smithi150 ceph-mon[39612]: 6.c scrub starts 2024-09-11T08:30:57.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:57 smithi150 ceph-mon[39612]: 6.c scrub ok 2024-09-11T08:30:57.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:57 smithi150 ceph-mon[39612]: 5.5 deep-scrub starts 2024-09-11T08:30:57.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:57 smithi150 ceph-mon[39612]: 5.5 deep-scrub ok 2024-09-11T08:30:57.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:57 smithi150 ceph-mon[39612]: 5.12 scrub starts 2024-09-11T08:30:57.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:57 smithi150 ceph-mon[39612]: 5.12 scrub ok 2024-09-11T08:30:57.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:57 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-11T08:30:57.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:57 smithi150 ceph-mon[39612]: osdmap e458: 8 total, 8 up, 8 in 2024-09-11T08:30:57.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:57 smithi150 ceph-mon[39612]: pgmap v1073: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 796 B/s rd, 0 op/s 2024-09-11T08:30:57.994 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:30:58.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:57 smithi181 ceph-mon[32701]: 6.c scrub starts 2024-09-11T08:30:58.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:57 smithi181 ceph-mon[32701]: 6.c scrub ok 2024-09-11T08:30:58.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:57 smithi181 ceph-mon[32701]: 5.5 deep-scrub starts 2024-09-11T08:30:58.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:57 smithi181 ceph-mon[32701]: 5.5 deep-scrub ok 2024-09-11T08:30:58.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:57 smithi181 ceph-mon[32701]: 5.12 scrub starts 2024-09-11T08:30:58.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:57 smithi181 ceph-mon[32701]: 5.12 scrub ok 2024-09-11T08:30:58.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:57 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-11T08:30:58.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:57 smithi181 ceph-mon[32701]: osdmap e458: 8 total, 8 up, 8 in 2024-09-11T08:30:58.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:57 smithi181 ceph-mon[32701]: pgmap v1073: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 796 B/s rd, 0 op/s 2024-09-11T08:30:58.738 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:30:58 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:30:58] "GET /metrics HTTP/1.1" 200 222246 "" "Prometheus/2.43.0" 2024-09-11T08:30:59.127 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:58 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:30:59.127 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:58 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:30:59.145 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:58 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:30:59.146 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:58 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:30:59.146 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:58 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:30:59.146 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:58 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:30:59.573 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:30:59 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:30:59.882 INFO:teuthology.orchestra.run.smithi150.stderr:reweighted osd.2 to 1 (10000) 2024-09-11T08:31:00.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:59 smithi181 ceph-mon[32701]: osdmap e459: 8 total, 8 up, 8 in 2024-09-11T08:31:00.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:59 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:31:00.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:59 smithi181 ceph-mon[32701]: pgmap v1075: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:00.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:30:59 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1074902991' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-11T08:31:00.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:59 smithi150 ceph-mon[29276]: osdmap e459: 8 total, 8 up, 8 in 2024-09-11T08:31:00.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:59 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:31:00.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:59 smithi150 ceph-mon[29276]: pgmap v1075: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:00.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:30:59 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1074902991' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-11T08:31:00.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:59 smithi150 ceph-mon[39612]: osdmap e459: 8 total, 8 up, 8 in 2024-09-11T08:31:00.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:59 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:31:00.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:59 smithi150 ceph-mon[39612]: pgmap v1075: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:00.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:30:59 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1074902991' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-11T08:31:00.606 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 2 1 2024-09-11T08:31:00.858 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:31:01.130 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:00 smithi150 ceph-mon[29276]: 3.8 scrub starts 2024-09-11T08:31:01.130 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:00 smithi150 ceph-mon[29276]: 3.8 scrub ok 2024-09-11T08:31:01.130 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:00 smithi150 ceph-mon[29276]: 2.2 deep-scrub starts 2024-09-11T08:31:01.130 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:00 smithi150 ceph-mon[29276]: 2.2 deep-scrub ok 2024-09-11T08:31:01.131 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:00 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1074902991' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-11T08:31:01.131 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:00 smithi150 ceph-mon[29276]: osdmap e460: 8 total, 8 up, 8 in 2024-09-11T08:31:01.131 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:00 smithi150 ceph-mon[39612]: 3.8 scrub starts 2024-09-11T08:31:01.131 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:00 smithi150 ceph-mon[39612]: 3.8 scrub ok 2024-09-11T08:31:01.131 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:00 smithi150 ceph-mon[39612]: 2.2 deep-scrub starts 2024-09-11T08:31:01.131 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:00 smithi150 ceph-mon[39612]: 2.2 deep-scrub ok 2024-09-11T08:31:01.131 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:00 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1074902991' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-11T08:31:01.131 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:00 smithi150 ceph-mon[39612]: osdmap e460: 8 total, 8 up, 8 in 2024-09-11T08:31:01.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:00 smithi181 ceph-mon[32701]: 3.8 scrub starts 2024-09-11T08:31:01.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:00 smithi181 ceph-mon[32701]: 3.8 scrub ok 2024-09-11T08:31:01.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:00 smithi181 ceph-mon[32701]: 2.2 deep-scrub starts 2024-09-11T08:31:01.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:00 smithi181 ceph-mon[32701]: 2.2 deep-scrub ok 2024-09-11T08:31:01.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:00 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1074902991' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-11T08:31:01.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:00 smithi181 ceph-mon[32701]: osdmap e460: 8 total, 8 up, 8 in 2024-09-11T08:31:02.145 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:01 smithi150 ceph-mon[39612]: 6.1b deep-scrub starts 2024-09-11T08:31:02.145 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:01 smithi150 ceph-mon[39612]: 6.1b deep-scrub ok 2024-09-11T08:31:02.146 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:01 smithi150 ceph-mon[39612]: 5.1e scrub starts 2024-09-11T08:31:02.146 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:01 smithi150 ceph-mon[39612]: 5.1e scrub ok 2024-09-11T08:31:02.146 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:01 smithi150 ceph-mon[39612]: pgmap v1077: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:02.146 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:01 smithi150 ceph-mon[29276]: 6.1b deep-scrub starts 2024-09-11T08:31:02.146 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:01 smithi150 ceph-mon[29276]: 6.1b deep-scrub ok 2024-09-11T08:31:02.146 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:01 smithi150 ceph-mon[29276]: 5.1e scrub starts 2024-09-11T08:31:02.146 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:01 smithi150 ceph-mon[29276]: 5.1e scrub ok 2024-09-11T08:31:02.146 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:01 smithi150 ceph-mon[29276]: pgmap v1077: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:02.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:01 smithi181 ceph-mon[32701]: 6.1b deep-scrub starts 2024-09-11T08:31:02.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:01 smithi181 ceph-mon[32701]: 6.1b deep-scrub ok 2024-09-11T08:31:02.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:01 smithi181 ceph-mon[32701]: 5.1e scrub starts 2024-09-11T08:31:02.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:01 smithi181 ceph-mon[32701]: 5.1e scrub ok 2024-09-11T08:31:02.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:01 smithi181 ceph-mon[32701]: pgmap v1077: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:02.891 INFO:teuthology.orchestra.run.smithi150.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-11T08:31:03.194 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:02 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2375662137' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-09-11T08:31:03.194 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:02 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2375662137' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-09-11T08:31:03.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:02 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2375662137' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-09-11T08:31:03.612 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd reweight 3 1 2024-09-11T08:31:03.862 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:31:04.190 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:03 smithi150 ceph-mon[39612]: 4.17 scrub starts 2024-09-11T08:31:04.190 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:03 smithi150 ceph-mon[39612]: 4.17 scrub ok 2024-09-11T08:31:04.190 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:03 smithi150 ceph-mon[39612]: 4.1e scrub starts 2024-09-11T08:31:04.190 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:03 smithi150 ceph-mon[39612]: 4.1e scrub ok 2024-09-11T08:31:04.190 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:03 smithi150 ceph-mon[39612]: 3.4 scrub starts 2024-09-11T08:31:04.191 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:03 smithi150 ceph-mon[39612]: 3.4 scrub ok 2024-09-11T08:31:04.191 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:03 smithi150 ceph-mon[39612]: 6.3 scrub starts 2024-09-11T08:31:04.191 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:03 smithi150 ceph-mon[39612]: 6.3 scrub ok 2024-09-11T08:31:04.191 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:03 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2375662137' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-11T08:31:04.191 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:03 smithi150 ceph-mon[39612]: osdmap e461: 8 total, 8 up, 8 in 2024-09-11T08:31:04.191 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:03 smithi150 ceph-mon[39612]: pgmap v1079: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-11T08:31:04.191 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:03 smithi150 ceph-mon[29276]: 4.17 scrub starts 2024-09-11T08:31:04.191 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:03 smithi150 ceph-mon[29276]: 4.17 scrub ok 2024-09-11T08:31:04.191 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:03 smithi150 ceph-mon[29276]: 4.1e scrub starts 2024-09-11T08:31:04.191 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:03 smithi150 ceph-mon[29276]: 4.1e scrub ok 2024-09-11T08:31:04.191 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:03 smithi150 ceph-mon[29276]: 3.4 scrub starts 2024-09-11T08:31:04.192 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:03 smithi150 ceph-mon[29276]: 3.4 scrub ok 2024-09-11T08:31:04.192 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:03 smithi150 ceph-mon[29276]: 6.3 scrub starts 2024-09-11T08:31:04.192 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:03 smithi150 ceph-mon[29276]: 6.3 scrub ok 2024-09-11T08:31:04.192 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:03 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2375662137' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-11T08:31:04.192 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:03 smithi150 ceph-mon[29276]: osdmap e461: 8 total, 8 up, 8 in 2024-09-11T08:31:04.192 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:03 smithi150 ceph-mon[29276]: pgmap v1079: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-11T08:31:04.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:03 smithi181 ceph-mon[32701]: 4.17 scrub starts 2024-09-11T08:31:04.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:03 smithi181 ceph-mon[32701]: 4.17 scrub ok 2024-09-11T08:31:04.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:03 smithi181 ceph-mon[32701]: 4.1e scrub starts 2024-09-11T08:31:04.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:03 smithi181 ceph-mon[32701]: 4.1e scrub ok 2024-09-11T08:31:04.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:03 smithi181 ceph-mon[32701]: 3.4 scrub starts 2024-09-11T08:31:04.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:03 smithi181 ceph-mon[32701]: 3.4 scrub ok 2024-09-11T08:31:04.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:03 smithi181 ceph-mon[32701]: 6.3 scrub starts 2024-09-11T08:31:04.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:03 smithi181 ceph-mon[32701]: 6.3 scrub ok 2024-09-11T08:31:04.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:03 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2375662137' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-11T08:31:04.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:03 smithi181 ceph-mon[32701]: osdmap e461: 8 total, 8 up, 8 in 2024-09-11T08:31:04.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:03 smithi181 ceph-mon[32701]: pgmap v1079: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-11T08:31:05.113 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:04 smithi150 ceph-mon[29276]: 6.e deep-scrub starts 2024-09-11T08:31:05.114 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:04 smithi150 ceph-mon[29276]: 6.e deep-scrub ok 2024-09-11T08:31:05.114 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:04 smithi150 ceph-mon[39612]: 6.e deep-scrub starts 2024-09-11T08:31:05.114 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:04 smithi150 ceph-mon[39612]: 6.e deep-scrub ok 2024-09-11T08:31:05.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:04 smithi181 ceph-mon[32701]: 6.e deep-scrub starts 2024-09-11T08:31:05.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:04 smithi181 ceph-mon[32701]: 6.e deep-scrub ok 2024-09-11T08:31:05.323 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:31:05 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:31:05] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:31:05.903 INFO:teuthology.orchestra.run.smithi150.stderr:reweighted osd.3 to 1 (10000) 2024-09-11T08:31:06.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:05 smithi150 ceph-mon[29276]: 3.1a scrub starts 2024-09-11T08:31:06.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:05 smithi150 ceph-mon[29276]: 3.1a scrub ok 2024-09-11T08:31:06.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:05 smithi150 ceph-mon[29276]: 6.8 scrub starts 2024-09-11T08:31:06.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:05 smithi150 ceph-mon[29276]: 6.8 scrub ok 2024-09-11T08:31:06.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:05 smithi150 ceph-mon[29276]: pgmap v1080: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 802 B/s rd, 0 op/s 2024-09-11T08:31:06.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:05 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1565823460' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-11T08:31:06.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:05 smithi150 ceph-mon[39612]: 3.1a scrub starts 2024-09-11T08:31:06.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:05 smithi150 ceph-mon[39612]: 3.1a scrub ok 2024-09-11T08:31:06.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:05 smithi150 ceph-mon[39612]: 6.8 scrub starts 2024-09-11T08:31:06.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:05 smithi150 ceph-mon[39612]: 6.8 scrub ok 2024-09-11T08:31:06.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:05 smithi150 ceph-mon[39612]: pgmap v1080: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 802 B/s rd, 0 op/s 2024-09-11T08:31:06.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:05 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1565823460' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-11T08:31:06.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:05 smithi181 ceph-mon[32701]: 3.1a scrub starts 2024-09-11T08:31:06.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:05 smithi181 ceph-mon[32701]: 3.1a scrub ok 2024-09-11T08:31:06.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:05 smithi181 ceph-mon[32701]: 6.8 scrub starts 2024-09-11T08:31:06.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:05 smithi181 ceph-mon[32701]: 6.8 scrub ok 2024-09-11T08:31:06.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:05 smithi181 ceph-mon[32701]: pgmap v1080: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 802 B/s rd, 0 op/s 2024-09-11T08:31:06.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:05 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1565823460' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-11T08:31:06.586 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 3 1 2024-09-11T08:31:06.845 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:31:07.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:06 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:31:07.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:06 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:31:07.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:06 smithi150 ceph-mon[39612]: 3.16 scrub starts 2024-09-11T08:31:07.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:06 smithi150 ceph-mon[39612]: 3.16 scrub ok 2024-09-11T08:31:07.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:06 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1565823460' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-11T08:31:07.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:06 smithi150 ceph-mon[39612]: osdmap e462: 8 total, 8 up, 8 in 2024-09-11T08:31:07.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:06 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:31:07.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:06 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:31:07.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:06 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:31:07.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:06 smithi150 ceph-mon[29276]: 3.16 scrub starts 2024-09-11T08:31:07.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:06 smithi150 ceph-mon[29276]: 3.16 scrub ok 2024-09-11T08:31:07.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:06 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1565823460' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-11T08:31:07.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:06 smithi150 ceph-mon[29276]: osdmap e462: 8 total, 8 up, 8 in 2024-09-11T08:31:07.240 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:06 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:31:07.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:06 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:31:07.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:06 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:31:07.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:06 smithi181 ceph-mon[32701]: 3.16 scrub starts 2024-09-11T08:31:07.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:06 smithi181 ceph-mon[32701]: 3.16 scrub ok 2024-09-11T08:31:07.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:06 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1565823460' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-11T08:31:07.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:06 smithi181 ceph-mon[32701]: osdmap e462: 8 total, 8 up, 8 in 2024-09-11T08:31:07.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:06 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:31:07.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:07 smithi150 ceph-mon[29276]: 3.6 scrub starts 2024-09-11T08:31:07.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:07 smithi150 ceph-mon[29276]: 3.6 scrub ok 2024-09-11T08:31:07.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:07 smithi150 ceph-mon[29276]: 3.e scrub starts 2024-09-11T08:31:07.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:07 smithi150 ceph-mon[29276]: 3.e scrub ok 2024-09-11T08:31:07.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:07 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:31:07.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:07 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:31:07.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:07 smithi150 ceph-mon[29276]: pgmap v1082: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 694 B/s rd, 0 op/s 2024-09-11T08:31:07.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:07 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:31:07.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:07 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:31:07.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:07 smithi150 ceph-mon[39612]: 3.6 scrub starts 2024-09-11T08:31:07.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:07 smithi150 ceph-mon[39612]: 3.6 scrub ok 2024-09-11T08:31:07.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:07 smithi150 ceph-mon[39612]: 3.e scrub starts 2024-09-11T08:31:07.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:07 smithi150 ceph-mon[39612]: 3.e scrub ok 2024-09-11T08:31:07.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:07 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:31:07.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:07 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:31:07.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:07 smithi150 ceph-mon[39612]: pgmap v1082: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 694 B/s rd, 0 op/s 2024-09-11T08:31:07.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:07 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:31:07.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:07 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:31:08.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:07 smithi181 ceph-mon[32701]: 3.6 scrub starts 2024-09-11T08:31:08.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:07 smithi181 ceph-mon[32701]: 3.6 scrub ok 2024-09-11T08:31:08.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:07 smithi181 ceph-mon[32701]: 3.e scrub starts 2024-09-11T08:31:08.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:07 smithi181 ceph-mon[32701]: 3.e scrub ok 2024-09-11T08:31:08.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:07 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:31:08.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:07 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:31:08.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:07 smithi181 ceph-mon[32701]: pgmap v1082: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 694 B/s rd, 0 op/s 2024-09-11T08:31:08.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:07 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:31:08.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:07 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:31:08.913 INFO:teuthology.orchestra.run.smithi150.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-11T08:31:08.913 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:31:08 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:31:08] "GET /metrics HTTP/1.1" 200 222261 "" "Prometheus/2.43.0" 2024-09-11T08:31:09.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:08 smithi150 ceph-mon[29276]: osdmap e463: 8 total, 8 up, 8 in 2024-09-11T08:31:09.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:08 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3213860107' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-09-11T08:31:09.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:08 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-09-11T08:31:09.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:08 smithi150 ceph-mon[39612]: osdmap e463: 8 total, 8 up, 8 in 2024-09-11T08:31:09.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:08 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3213860107' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-09-11T08:31:09.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:08 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-09-11T08:31:09.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:08 smithi181 ceph-mon[32701]: osdmap e463: 8 total, 8 up, 8 in 2024-09-11T08:31:09.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:08 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3213860107' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-09-11T08:31:09.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:08 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-09-11T08:31:09.323 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:31:09 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:31:09.609 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd reweight 4 1 2024-09-11T08:31:09.858 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:31:10.192 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:09 smithi150 ceph-mon[29276]: 3.1b scrub starts 2024-09-11T08:31:10.192 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:09 smithi150 ceph-mon[29276]: 3.1b scrub ok 2024-09-11T08:31:10.192 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:09 smithi150 ceph-mon[29276]: 5.10 scrub starts 2024-09-11T08:31:10.192 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:09 smithi150 ceph-mon[29276]: 5.10 scrub ok 2024-09-11T08:31:10.192 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:09 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-11T08:31:10.192 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:09 smithi150 ceph-mon[29276]: osdmap e464: 8 total, 8 up, 8 in 2024-09-11T08:31:10.192 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:09 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:31:10.192 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:09 smithi150 ceph-mon[29276]: pgmap v1085: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:10.193 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:09 smithi150 ceph-mon[39612]: 3.1b scrub starts 2024-09-11T08:31:10.193 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:09 smithi150 ceph-mon[39612]: 3.1b scrub ok 2024-09-11T08:31:10.193 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:09 smithi150 ceph-mon[39612]: 5.10 scrub starts 2024-09-11T08:31:10.193 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:09 smithi150 ceph-mon[39612]: 5.10 scrub ok 2024-09-11T08:31:10.193 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:09 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-11T08:31:10.193 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:09 smithi150 ceph-mon[39612]: osdmap e464: 8 total, 8 up, 8 in 2024-09-11T08:31:10.193 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:09 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:31:10.193 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:09 smithi150 ceph-mon[39612]: pgmap v1085: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:10.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:09 smithi181 ceph-mon[32701]: 3.1b scrub starts 2024-09-11T08:31:10.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:09 smithi181 ceph-mon[32701]: 3.1b scrub ok 2024-09-11T08:31:10.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:09 smithi181 ceph-mon[32701]: 5.10 scrub starts 2024-09-11T08:31:10.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:09 smithi181 ceph-mon[32701]: 5.10 scrub ok 2024-09-11T08:31:10.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:09 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-11T08:31:10.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:09 smithi181 ceph-mon[32701]: osdmap e464: 8 total, 8 up, 8 in 2024-09-11T08:31:10.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:09 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:31:10.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:09 smithi181 ceph-mon[32701]: pgmap v1085: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:11.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:10 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:31:11.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:10 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:31:11.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:10 smithi150 ceph-mon[29276]: osdmap e465: 8 total, 8 up, 8 in 2024-09-11T08:31:11.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:10 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:31:11.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:10 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:31:11.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:10 smithi150 ceph-mon[39612]: osdmap e465: 8 total, 8 up, 8 in 2024-09-11T08:31:11.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:10 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:31:11.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:10 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:31:11.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:10 smithi181 ceph-mon[32701]: osdmap e465: 8 total, 8 up, 8 in 2024-09-11T08:31:11.935 INFO:teuthology.orchestra.run.smithi150.stderr:reweighted osd.4 to 1 (10000) 2024-09-11T08:31:12.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:11 smithi150 ceph-mon[29276]: 6.16 scrub starts 2024-09-11T08:31:12.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:11 smithi150 ceph-mon[29276]: 6.16 scrub ok 2024-09-11T08:31:12.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:11 smithi150 ceph-mon[29276]: 3.0 scrub starts 2024-09-11T08:31:12.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:11 smithi150 ceph-mon[29276]: 3.0 scrub ok 2024-09-11T08:31:12.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:11 smithi150 ceph-mon[29276]: pgmap v1087: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 956 B/s rd, 0 op/s 2024-09-11T08:31:12.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:11 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1397861662' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-11T08:31:12.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:11 smithi150 ceph-mon[39612]: 6.16 scrub starts 2024-09-11T08:31:12.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:11 smithi150 ceph-mon[39612]: 6.16 scrub ok 2024-09-11T08:31:12.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:11 smithi150 ceph-mon[39612]: 3.0 scrub starts 2024-09-11T08:31:12.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:11 smithi150 ceph-mon[39612]: 3.0 scrub ok 2024-09-11T08:31:12.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:11 smithi150 ceph-mon[39612]: pgmap v1087: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 956 B/s rd, 0 op/s 2024-09-11T08:31:12.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:11 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1397861662' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-11T08:31:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:11 smithi181 ceph-mon[32701]: 6.16 scrub starts 2024-09-11T08:31:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:11 smithi181 ceph-mon[32701]: 6.16 scrub ok 2024-09-11T08:31:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:11 smithi181 ceph-mon[32701]: 3.0 scrub starts 2024-09-11T08:31:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:11 smithi181 ceph-mon[32701]: 3.0 scrub ok 2024-09-11T08:31:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:11 smithi181 ceph-mon[32701]: pgmap v1087: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 956 B/s rd, 0 op/s 2024-09-11T08:31:12.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:11 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1397861662' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-11T08:31:12.717 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 4 1 2024-09-11T08:31:12.971 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:31:13.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:12 smithi150 ceph-mon[39612]: 4.12 scrub starts 2024-09-11T08:31:13.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:12 smithi150 ceph-mon[39612]: 4.12 scrub ok 2024-09-11T08:31:13.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:12 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1397861662' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-11T08:31:13.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:12 smithi150 ceph-mon[39612]: osdmap e466: 8 total, 8 up, 8 in 2024-09-11T08:31:13.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:12 smithi150 ceph-mon[39612]: 4.18 scrub starts 2024-09-11T08:31:13.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:12 smithi150 ceph-mon[39612]: 4.18 scrub ok 2024-09-11T08:31:13.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:12 smithi150 ceph-mon[29276]: 4.12 scrub starts 2024-09-11T08:31:13.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:12 smithi150 ceph-mon[29276]: 4.12 scrub ok 2024-09-11T08:31:13.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:12 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1397861662' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-11T08:31:13.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:12 smithi150 ceph-mon[29276]: osdmap e466: 8 total, 8 up, 8 in 2024-09-11T08:31:13.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:12 smithi150 ceph-mon[29276]: 4.18 scrub starts 2024-09-11T08:31:13.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:12 smithi150 ceph-mon[29276]: 4.18 scrub ok 2024-09-11T08:31:13.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:12 smithi181 ceph-mon[32701]: 4.12 scrub starts 2024-09-11T08:31:13.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:12 smithi181 ceph-mon[32701]: 4.12 scrub ok 2024-09-11T08:31:13.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:12 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1397861662' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-11T08:31:13.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:12 smithi181 ceph-mon[32701]: osdmap e466: 8 total, 8 up, 8 in 2024-09-11T08:31:13.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:12 smithi181 ceph-mon[32701]: 4.18 scrub starts 2024-09-11T08:31:13.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:12 smithi181 ceph-mon[32701]: 4.18 scrub ok 2024-09-11T08:31:14.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:13 smithi150 ceph-mon[39612]: 3.14 scrub starts 2024-09-11T08:31:14.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:13 smithi150 ceph-mon[39612]: 3.14 scrub ok 2024-09-11T08:31:14.241 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:13 smithi150 ceph-mon[39612]: pgmap v1089: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 957 B/s rd, 0 op/s 2024-09-11T08:31:14.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:13 smithi150 ceph-mon[29276]: 3.14 scrub starts 2024-09-11T08:31:14.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:13 smithi150 ceph-mon[29276]: 3.14 scrub ok 2024-09-11T08:31:14.241 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:13 smithi150 ceph-mon[29276]: pgmap v1089: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 957 B/s rd, 0 op/s 2024-09-11T08:31:14.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:13 smithi181 ceph-mon[32701]: 3.14 scrub starts 2024-09-11T08:31:14.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:13 smithi181 ceph-mon[32701]: 3.14 scrub ok 2024-09-11T08:31:14.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:13 smithi181 ceph-mon[32701]: pgmap v1089: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 957 B/s rd, 0 op/s 2024-09-11T08:31:15.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:14 smithi150 ceph-mon[29276]: 5.9 scrub starts 2024-09-11T08:31:15.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:14 smithi150 ceph-mon[29276]: 5.9 scrub ok 2024-09-11T08:31:15.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:14 smithi150 ceph-mon[39612]: 5.9 scrub starts 2024-09-11T08:31:15.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:14 smithi150 ceph-mon[39612]: 5.9 scrub ok 2024-09-11T08:31:15.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:14 smithi181 ceph-mon[32701]: 5.9 scrub starts 2024-09-11T08:31:15.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:14 smithi181 ceph-mon[32701]: 5.9 scrub ok 2024-09-11T08:31:15.323 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:31:15 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:31:15] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:31:15.961 INFO:teuthology.orchestra.run.smithi150.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-11T08:31:16.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:15 smithi150 ceph-mon[29276]: 3.7 scrub starts 2024-09-11T08:31:16.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:15 smithi150 ceph-mon[29276]: 3.7 scrub ok 2024-09-11T08:31:16.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:15 smithi150 ceph-mon[29276]: 5.11 scrub starts 2024-09-11T08:31:16.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:15 smithi150 ceph-mon[29276]: 5.11 scrub ok 2024-09-11T08:31:16.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:15 smithi150 ceph-mon[29276]: 4.6 scrub starts 2024-09-11T08:31:16.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:15 smithi150 ceph-mon[29276]: 4.6 scrub ok 2024-09-11T08:31:16.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:15 smithi150 ceph-mon[29276]: 5.6 deep-scrub starts 2024-09-11T08:31:16.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:15 smithi150 ceph-mon[29276]: 5.6 deep-scrub ok 2024-09-11T08:31:16.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:15 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2305398218' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-09-11T08:31:16.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:15 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-09-11T08:31:16.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:15 smithi150 ceph-mon[29276]: pgmap v1090: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail 2024-09-11T08:31:16.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:15 smithi150 ceph-mon[39612]: 3.7 scrub starts 2024-09-11T08:31:16.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:15 smithi150 ceph-mon[39612]: 3.7 scrub ok 2024-09-11T08:31:16.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:15 smithi150 ceph-mon[39612]: 5.11 scrub starts 2024-09-11T08:31:16.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:15 smithi150 ceph-mon[39612]: 5.11 scrub ok 2024-09-11T08:31:16.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:15 smithi150 ceph-mon[39612]: 4.6 scrub starts 2024-09-11T08:31:16.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:15 smithi150 ceph-mon[39612]: 4.6 scrub ok 2024-09-11T08:31:16.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:15 smithi150 ceph-mon[39612]: 5.6 deep-scrub starts 2024-09-11T08:31:16.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:15 smithi150 ceph-mon[39612]: 5.6 deep-scrub ok 2024-09-11T08:31:16.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:15 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2305398218' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-09-11T08:31:16.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:15 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-09-11T08:31:16.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:15 smithi150 ceph-mon[39612]: pgmap v1090: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail 2024-09-11T08:31:16.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:15 smithi181 ceph-mon[32701]: 3.7 scrub starts 2024-09-11T08:31:16.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:15 smithi181 ceph-mon[32701]: 3.7 scrub ok 2024-09-11T08:31:16.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:15 smithi181 ceph-mon[32701]: 5.11 scrub starts 2024-09-11T08:31:16.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:15 smithi181 ceph-mon[32701]: 5.11 scrub ok 2024-09-11T08:31:16.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:15 smithi181 ceph-mon[32701]: 4.6 scrub starts 2024-09-11T08:31:16.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:15 smithi181 ceph-mon[32701]: 4.6 scrub ok 2024-09-11T08:31:16.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:15 smithi181 ceph-mon[32701]: 5.6 deep-scrub starts 2024-09-11T08:31:16.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:15 smithi181 ceph-mon[32701]: 5.6 deep-scrub ok 2024-09-11T08:31:16.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:15 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2305398218' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-09-11T08:31:16.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:15 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-09-11T08:31:16.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:15 smithi181 ceph-mon[32701]: pgmap v1090: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail 2024-09-11T08:31:16.786 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd reweight 6 1 2024-09-11T08:31:17.046 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:31:17.167 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:16 smithi150 ceph-mon[29276]: 3.11 scrub starts 2024-09-11T08:31:17.168 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:16 smithi150 ceph-mon[29276]: 3.11 scrub ok 2024-09-11T08:31:17.168 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:16 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-11T08:31:17.168 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:16 smithi150 ceph-mon[29276]: osdmap e467: 8 total, 8 up, 8 in 2024-09-11T08:31:17.168 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:16 smithi150 ceph-mon[29276]: osdmap e468: 8 total, 8 up, 8 in 2024-09-11T08:31:17.168 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:16 smithi150 ceph-mon[39612]: 3.11 scrub starts 2024-09-11T08:31:17.168 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:16 smithi150 ceph-mon[39612]: 3.11 scrub ok 2024-09-11T08:31:17.168 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:16 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-11T08:31:17.168 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:16 smithi150 ceph-mon[39612]: osdmap e467: 8 total, 8 up, 8 in 2024-09-11T08:31:17.168 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:16 smithi150 ceph-mon[39612]: osdmap e468: 8 total, 8 up, 8 in 2024-09-11T08:31:17.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:16 smithi181 ceph-mon[32701]: 3.11 scrub starts 2024-09-11T08:31:17.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:16 smithi181 ceph-mon[32701]: 3.11 scrub ok 2024-09-11T08:31:17.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:16 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-11T08:31:17.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:16 smithi181 ceph-mon[32701]: osdmap e467: 8 total, 8 up, 8 in 2024-09-11T08:31:17.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:16 smithi181 ceph-mon[32701]: osdmap e468: 8 total, 8 up, 8 in 2024-09-11T08:31:18.224 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:17 smithi181 ceph-mon[32701]: 5.a scrub starts 2024-09-11T08:31:18.224 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:17 smithi181 ceph-mon[32701]: 5.a scrub ok 2024-09-11T08:31:18.224 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:17 smithi181 ceph-mon[32701]: pgmap v1093: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail 2024-09-11T08:31:18.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:17 smithi150 ceph-mon[39612]: 5.a scrub starts 2024-09-11T08:31:18.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:17 smithi150 ceph-mon[39612]: 5.a scrub ok 2024-09-11T08:31:18.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:17 smithi150 ceph-mon[39612]: pgmap v1093: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail 2024-09-11T08:31:18.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:17 smithi150 ceph-mon[29276]: 5.a scrub starts 2024-09-11T08:31:18.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:17 smithi150 ceph-mon[29276]: 5.a scrub ok 2024-09-11T08:31:18.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:17 smithi150 ceph-mon[29276]: pgmap v1093: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail 2024-09-11T08:31:18.859 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:31:18 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:31:18] "GET /metrics HTTP/1.1" 200 222261 "" "Prometheus/2.43.0" 2024-09-11T08:31:19.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:18 smithi150 ceph-mon[39612]: 6.11 scrub starts 2024-09-11T08:31:19.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:18 smithi150 ceph-mon[39612]: 6.11 scrub ok 2024-09-11T08:31:19.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:18 smithi150 ceph-mon[39612]: 4.1a deep-scrub starts 2024-09-11T08:31:19.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:18 smithi150 ceph-mon[39612]: 4.1a deep-scrub ok 2024-09-11T08:31:19.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:18 smithi150 ceph-mon[29276]: 6.11 scrub starts 2024-09-11T08:31:19.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:18 smithi150 ceph-mon[29276]: 6.11 scrub ok 2024-09-11T08:31:19.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:18 smithi150 ceph-mon[29276]: 4.1a deep-scrub starts 2024-09-11T08:31:19.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:18 smithi150 ceph-mon[29276]: 4.1a deep-scrub ok 2024-09-11T08:31:19.323 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:31:19 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:31:19.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:18 smithi181 ceph-mon[32701]: 6.11 scrub starts 2024-09-11T08:31:19.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:18 smithi181 ceph-mon[32701]: 6.11 scrub ok 2024-09-11T08:31:19.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:18 smithi181 ceph-mon[32701]: 4.1a deep-scrub starts 2024-09-11T08:31:19.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:18 smithi181 ceph-mon[32701]: 4.1a deep-scrub ok 2024-09-11T08:31:19.978 INFO:teuthology.orchestra.run.smithi150.stderr:reweighted osd.6 to 1 (10000) 2024-09-11T08:31:20.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:19 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1659592519' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-11T08:31:20.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:19 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:31:20.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:19 smithi150 ceph-mon[29276]: pgmap v1094: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 839 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-11T08:31:20.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:19 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1659592519' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-11T08:31:20.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:19 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:31:20.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:19 smithi150 ceph-mon[39612]: pgmap v1094: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 839 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-11T08:31:20.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:19 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1659592519' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-11T08:31:20.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:19 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:31:20.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:19 smithi181 ceph-mon[32701]: pgmap v1094: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 839 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-11T08:31:20.725 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 6 1 2024-09-11T08:31:20.987 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:31:21.237 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:20 smithi150 ceph-mon[29276]: 6.5 scrub starts 2024-09-11T08:31:21.237 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:20 smithi150 ceph-mon[29276]: 6.5 scrub ok 2024-09-11T08:31:21.237 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:20 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1659592519' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-11T08:31:21.237 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:20 smithi150 ceph-mon[29276]: osdmap e469: 8 total, 8 up, 8 in 2024-09-11T08:31:21.237 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:20 smithi150 ceph-mon[39612]: 6.5 scrub starts 2024-09-11T08:31:21.237 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:20 smithi150 ceph-mon[39612]: 6.5 scrub ok 2024-09-11T08:31:21.237 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:20 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1659592519' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-11T08:31:21.237 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:20 smithi150 ceph-mon[39612]: osdmap e469: 8 total, 8 up, 8 in 2024-09-11T08:31:21.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:20 smithi181 ceph-mon[32701]: 6.5 scrub starts 2024-09-11T08:31:21.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:20 smithi181 ceph-mon[32701]: 6.5 scrub ok 2024-09-11T08:31:21.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:20 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1659592519' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-11T08:31:21.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:20 smithi181 ceph-mon[32701]: osdmap e469: 8 total, 8 up, 8 in 2024-09-11T08:31:22.250 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:21 smithi150 ceph-mon[29276]: 6.15 scrub starts 2024-09-11T08:31:22.250 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:21 smithi150 ceph-mon[29276]: 6.15 scrub ok 2024-09-11T08:31:22.250 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:21 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:31:22.250 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:21 smithi150 ceph-mon[29276]: pgmap v1096: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-09-11T08:31:22.251 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:21 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:31:22.251 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:21 smithi150 ceph-mon[39612]: 6.15 scrub starts 2024-09-11T08:31:22.251 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:21 smithi150 ceph-mon[39612]: 6.15 scrub ok 2024-09-11T08:31:22.251 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:21 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:31:22.251 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:21 smithi150 ceph-mon[39612]: pgmap v1096: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-09-11T08:31:22.251 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:21 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:31:22.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:21 smithi181 ceph-mon[32701]: 6.15 scrub starts 2024-09-11T08:31:22.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:21 smithi181 ceph-mon[32701]: 6.15 scrub ok 2024-09-11T08:31:22.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:21 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:31:22.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:21 smithi181 ceph-mon[32701]: pgmap v1096: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-09-11T08:31:22.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:21 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:31:22.993 INFO:teuthology.orchestra.run.smithi150.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-09-11T08:31:23.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:22 smithi181 ceph-mon[32701]: 6.a scrub starts 2024-09-11T08:31:23.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:22 smithi181 ceph-mon[32701]: 6.a scrub ok 2024-09-11T08:31:23.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:22 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3691977402' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-09-11T08:31:23.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:22 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-09-11T08:31:23.381 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:22 smithi150 ceph-mon[29276]: 6.a scrub starts 2024-09-11T08:31:23.381 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:22 smithi150 ceph-mon[29276]: 6.a scrub ok 2024-09-11T08:31:23.381 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:22 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3691977402' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-09-11T08:31:23.381 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:22 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-09-11T08:31:23.381 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:22 smithi150 ceph-mon[39612]: 6.a scrub starts 2024-09-11T08:31:23.381 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:22 smithi150 ceph-mon[39612]: 6.a scrub ok 2024-09-11T08:31:23.382 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:22 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3691977402' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-09-11T08:31:23.382 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:22 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-09-11T08:31:23.698 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd reweight 7 1 2024-09-11T08:31:24.214 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:31:24.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:23 smithi150 ceph-mon[29276]: 3.a scrub starts 2024-09-11T08:31:24.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:23 smithi150 ceph-mon[29276]: 3.a scrub ok 2024-09-11T08:31:24.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:23 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-11T08:31:24.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:23 smithi150 ceph-mon[29276]: osdmap e470: 8 total, 8 up, 8 in 2024-09-11T08:31:24.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:23 smithi150 ceph-mon[29276]: 3.1e scrub starts 2024-09-11T08:31:24.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:23 smithi150 ceph-mon[29276]: 3.1e scrub ok 2024-09-11T08:31:24.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:23 smithi150 ceph-mon[29276]: pgmap v1098: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.9 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-09-11T08:31:24.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:23 smithi150 ceph-mon[39612]: 3.a scrub starts 2024-09-11T08:31:24.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:23 smithi150 ceph-mon[39612]: 3.a scrub ok 2024-09-11T08:31:24.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:23 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-11T08:31:24.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:23 smithi150 ceph-mon[39612]: osdmap e470: 8 total, 8 up, 8 in 2024-09-11T08:31:24.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:23 smithi150 ceph-mon[39612]: 3.1e scrub starts 2024-09-11T08:31:24.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:23 smithi150 ceph-mon[39612]: 3.1e scrub ok 2024-09-11T08:31:24.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:23 smithi150 ceph-mon[39612]: pgmap v1098: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.9 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-09-11T08:31:24.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:23 smithi181 ceph-mon[32701]: 3.a scrub starts 2024-09-11T08:31:24.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:23 smithi181 ceph-mon[32701]: 3.a scrub ok 2024-09-11T08:31:24.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:23 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-11T08:31:24.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:23 smithi181 ceph-mon[32701]: osdmap e470: 8 total, 8 up, 8 in 2024-09-11T08:31:24.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:23 smithi181 ceph-mon[32701]: 3.1e scrub starts 2024-09-11T08:31:24.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:23 smithi181 ceph-mon[32701]: 3.1e scrub ok 2024-09-11T08:31:24.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:23 smithi181 ceph-mon[32701]: pgmap v1098: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.9 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-09-11T08:31:25.228 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:31:25 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:31:25] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:31:25.228 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:25 smithi181 ceph-mon[32701]: 6.f scrub starts 2024-09-11T08:31:25.228 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:25 smithi181 ceph-mon[32701]: 6.f scrub ok 2024-09-11T08:31:25.229 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:25 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:31:25.229 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:25 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:31:25.229 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:25 smithi181 ceph-mon[32701]: osdmap e471: 8 total, 8 up, 8 in 2024-09-11T08:31:25.229 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:25 smithi181 ceph-mon[32701]: 5.3 scrub starts 2024-09-11T08:31:25.229 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:25 smithi181 ceph-mon[32701]: 5.3 scrub ok 2024-09-11T08:31:25.229 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:25 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:31:25.229 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:25 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:31:25.229 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:25 smithi181 ceph-mon[32701]: 6.10 scrub starts 2024-09-11T08:31:25.229 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:25 smithi181 ceph-mon[32701]: 6.10 scrub ok 2024-09-11T08:31:25.380 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:25 smithi150 ceph-mon[39612]: 6.f scrub starts 2024-09-11T08:31:25.380 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:25 smithi150 ceph-mon[39612]: 6.f scrub ok 2024-09-11T08:31:25.380 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:25 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:31:25.380 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:25 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:31:25.380 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:25 smithi150 ceph-mon[39612]: osdmap e471: 8 total, 8 up, 8 in 2024-09-11T08:31:25.381 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:25 smithi150 ceph-mon[39612]: 5.3 scrub starts 2024-09-11T08:31:25.381 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:25 smithi150 ceph-mon[39612]: 5.3 scrub ok 2024-09-11T08:31:25.381 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:25 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:31:25.381 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:25 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:31:25.381 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:25 smithi150 ceph-mon[39612]: 6.10 scrub starts 2024-09-11T08:31:25.381 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:25 smithi150 ceph-mon[39612]: 6.10 scrub ok 2024-09-11T08:31:25.381 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:25 smithi150 ceph-mon[29276]: 6.f scrub starts 2024-09-11T08:31:25.381 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:25 smithi150 ceph-mon[29276]: 6.f scrub ok 2024-09-11T08:31:25.381 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:25 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:31:25.381 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:25 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:31:25.382 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:25 smithi150 ceph-mon[29276]: osdmap e471: 8 total, 8 up, 8 in 2024-09-11T08:31:25.382 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:25 smithi150 ceph-mon[29276]: 5.3 scrub starts 2024-09-11T08:31:25.382 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:25 smithi150 ceph-mon[29276]: 5.3 scrub ok 2024-09-11T08:31:25.382 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:25 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:31:25.382 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:25 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:31:25.382 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:25 smithi150 ceph-mon[29276]: 6.10 scrub starts 2024-09-11T08:31:25.382 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:25 smithi150 ceph-mon[29276]: 6.10 scrub ok 2024-09-11T08:31:26.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:26 smithi150 ceph-mon[29276]: osdmap e472: 8 total, 8 up, 8 in 2024-09-11T08:31:26.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:26 smithi150 ceph-mon[29276]: 6.13 scrub starts 2024-09-11T08:31:26.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:26 smithi150 ceph-mon[29276]: 6.13 scrub ok 2024-09-11T08:31:26.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:26 smithi150 ceph-mon[29276]: pgmap v1101: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 969 B/s rd, 0 op/s 2024-09-11T08:31:26.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:26 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:31:26.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:26 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:31:26.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:26 smithi150 ceph-mon[39612]: osdmap e472: 8 total, 8 up, 8 in 2024-09-11T08:31:26.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:26 smithi150 ceph-mon[39612]: 6.13 scrub starts 2024-09-11T08:31:26.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:26 smithi150 ceph-mon[39612]: 6.13 scrub ok 2024-09-11T08:31:26.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:26 smithi150 ceph-mon[39612]: pgmap v1101: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 969 B/s rd, 0 op/s 2024-09-11T08:31:26.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:26 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:31:26.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:26 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:31:26.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:26 smithi181 ceph-mon[32701]: osdmap e472: 8 total, 8 up, 8 in 2024-09-11T08:31:26.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:26 smithi181 ceph-mon[32701]: 6.13 scrub starts 2024-09-11T08:31:26.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:26 smithi181 ceph-mon[32701]: 6.13 scrub ok 2024-09-11T08:31:26.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:26 smithi181 ceph-mon[32701]: pgmap v1101: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 969 B/s rd, 0 op/s 2024-09-11T08:31:26.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:26 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:31:26.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:26 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:31:27.012 INFO:teuthology.orchestra.run.smithi150.stderr:reweighted osd.7 to 1 (10000) 2024-09-11T08:31:27.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:27 smithi181 ceph-mon[32701]: 2.1 scrub starts 2024-09-11T08:31:27.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:27 smithi181 ceph-mon[32701]: 2.1 scrub ok 2024-09-11T08:31:27.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:27 smithi181 ceph-mon[32701]: 6.1f scrub starts 2024-09-11T08:31:27.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:27 smithi181 ceph-mon[32701]: 6.1f scrub ok 2024-09-11T08:31:27.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:27 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3779643111' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-11T08:31:27.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:27 smithi181 ceph-mon[32701]: 4.16 scrub starts 2024-09-11T08:31:27.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:27 smithi181 ceph-mon[32701]: 4.16 scrub ok 2024-09-11T08:31:27.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:27 smithi181 ceph-mon[32701]: 4.c scrub starts 2024-09-11T08:31:27.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:27 smithi181 ceph-mon[32701]: 4.c scrub ok 2024-09-11T08:31:27.328 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:27 smithi150 ceph-mon[29276]: 2.1 scrub starts 2024-09-11T08:31:27.328 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:27 smithi150 ceph-mon[29276]: 2.1 scrub ok 2024-09-11T08:31:27.328 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:27 smithi150 ceph-mon[29276]: 6.1f scrub starts 2024-09-11T08:31:27.328 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:27 smithi150 ceph-mon[29276]: 6.1f scrub ok 2024-09-11T08:31:27.329 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:27 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3779643111' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-11T08:31:27.329 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:27 smithi150 ceph-mon[29276]: 4.16 scrub starts 2024-09-11T08:31:27.329 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:27 smithi150 ceph-mon[29276]: 4.16 scrub ok 2024-09-11T08:31:27.329 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:27 smithi150 ceph-mon[29276]: 4.c scrub starts 2024-09-11T08:31:27.329 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:27 smithi150 ceph-mon[29276]: 4.c scrub ok 2024-09-11T08:31:27.329 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:27 smithi150 ceph-mon[39612]: 2.1 scrub starts 2024-09-11T08:31:27.329 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:27 smithi150 ceph-mon[39612]: 2.1 scrub ok 2024-09-11T08:31:27.329 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:27 smithi150 ceph-mon[39612]: 6.1f scrub starts 2024-09-11T08:31:27.329 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:27 smithi150 ceph-mon[39612]: 6.1f scrub ok 2024-09-11T08:31:27.329 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:27 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3779643111' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-11T08:31:27.329 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:27 smithi150 ceph-mon[39612]: 4.16 scrub starts 2024-09-11T08:31:27.330 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:27 smithi150 ceph-mon[39612]: 4.16 scrub ok 2024-09-11T08:31:27.330 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:27 smithi150 ceph-mon[39612]: 4.c scrub starts 2024-09-11T08:31:27.330 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:27 smithi150 ceph-mon[39612]: 4.c scrub ok 2024-09-11T08:31:28.201 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:28 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3779643111' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-11T08:31:28.201 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:28 smithi150 ceph-mon[29276]: osdmap e473: 8 total, 8 up, 8 in 2024-09-11T08:31:28.201 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:28 smithi150 ceph-mon[29276]: pgmap v1103: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail 2024-09-11T08:31:28.201 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:28 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3779643111' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-11T08:31:28.201 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:28 smithi150 ceph-mon[39612]: osdmap e473: 8 total, 8 up, 8 in 2024-09-11T08:31:28.201 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:28 smithi150 ceph-mon[39612]: pgmap v1103: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail 2024-09-11T08:31:28.230 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 7 1 2024-09-11T08:31:28.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:28 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3779643111' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-11T08:31:28.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:28 smithi181 ceph-mon[32701]: osdmap e473: 8 total, 8 up, 8 in 2024-09-11T08:31:28.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:28 smithi181 ceph-mon[32701]: pgmap v1103: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail 2024-09-11T08:31:28.816 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:31:28.938 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:31:28 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:31:28] "GET /metrics HTTP/1.1" 200 222278 "" "Prometheus/2.43.0" 2024-09-11T08:31:29.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:29 smithi150 ceph-mon[29276]: 4.15 scrub starts 2024-09-11T08:31:29.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:29 smithi150 ceph-mon[29276]: 4.15 scrub ok 2024-09-11T08:31:29.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:29 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:31:29.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:29 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:31:29.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:29 smithi150 ceph-mon[39612]: 4.15 scrub starts 2024-09-11T08:31:29.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:29 smithi150 ceph-mon[39612]: 4.15 scrub ok 2024-09-11T08:31:29.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:29 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:31:29.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:29 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:31:29.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:29 smithi181 ceph-mon[32701]: 4.15 scrub starts 2024-09-11T08:31:29.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:29 smithi181 ceph-mon[32701]: 4.15 scrub ok 2024-09-11T08:31:29.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:29 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:31:29.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:29 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:31:29.323 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:31:29 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:31:30.284 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:30 smithi150 ceph-mon[29276]: 5.1a scrub starts 2024-09-11T08:31:30.285 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:30 smithi150 ceph-mon[29276]: 5.1a scrub ok 2024-09-11T08:31:30.285 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:30 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:31:30.285 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:30 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:31:30.285 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:30 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:31:30.285 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:30 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:31:30.285 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:30 smithi150 ceph-mon[29276]: pgmap v1104: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:30.285 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:30 smithi150 ceph-mon[39612]: 5.1a scrub starts 2024-09-11T08:31:30.285 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:30 smithi150 ceph-mon[39612]: 5.1a scrub ok 2024-09-11T08:31:30.286 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:30 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:31:30.286 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:30 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:31:30.286 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:30 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:31:30.286 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:30 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:31:30.286 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:30 smithi150 ceph-mon[39612]: pgmap v1104: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:30.322 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:30 smithi181 ceph-mon[32701]: 5.1a scrub starts 2024-09-11T08:31:30.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:30 smithi181 ceph-mon[32701]: 5.1a scrub ok 2024-09-11T08:31:30.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:30 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:31:30.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:30 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:31:30.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:30 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:31:30.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:30 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:31:30.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:30 smithi181 ceph-mon[32701]: pgmap v1104: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:31.230 INFO:teuthology.orchestra.run.smithi150.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-09-11T08:31:31.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:31 smithi150 ceph-mon[39612]: 6.0 scrub starts 2024-09-11T08:31:31.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:31 smithi150 ceph-mon[39612]: 6.0 scrub ok 2024-09-11T08:31:31.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:31 smithi150 ceph-mon[39612]: 6.17 scrub starts 2024-09-11T08:31:31.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:31 smithi150 ceph-mon[39612]: 6.17 scrub ok 2024-09-11T08:31:31.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:31 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1057689231' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-09-11T08:31:31.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:31 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-09-11T08:31:31.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:31 smithi150 ceph-mon[29276]: 6.0 scrub starts 2024-09-11T08:31:31.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:31 smithi150 ceph-mon[29276]: 6.0 scrub ok 2024-09-11T08:31:31.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:31 smithi150 ceph-mon[29276]: 6.17 scrub starts 2024-09-11T08:31:31.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:31 smithi150 ceph-mon[29276]: 6.17 scrub ok 2024-09-11T08:31:31.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:31 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1057689231' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-09-11T08:31:31.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:31 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-09-11T08:31:31.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:31 smithi181 ceph-mon[32701]: 6.0 scrub starts 2024-09-11T08:31:31.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:31 smithi181 ceph-mon[32701]: 6.0 scrub ok 2024-09-11T08:31:31.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:31 smithi181 ceph-mon[32701]: 6.17 scrub starts 2024-09-11T08:31:31.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:31 smithi181 ceph-mon[32701]: 6.17 scrub ok 2024-09-11T08:31:31.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:31 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1057689231' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-09-11T08:31:31.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:31 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-09-11T08:31:31.851 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd reweight 5 1 2024-09-11T08:31:32.107 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:31:32.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:32 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-11T08:31:32.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:32 smithi150 ceph-mon[39612]: osdmap e474: 8 total, 8 up, 8 in 2024-09-11T08:31:32.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:32 smithi150 ceph-mon[39612]: pgmap v1106: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-09-11T08:31:32.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:32 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-11T08:31:32.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:32 smithi150 ceph-mon[29276]: osdmap e474: 8 total, 8 up, 8 in 2024-09-11T08:31:32.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:32 smithi150 ceph-mon[29276]: pgmap v1106: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-09-11T08:31:32.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:32 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-11T08:31:32.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:32 smithi181 ceph-mon[32701]: osdmap e474: 8 total, 8 up, 8 in 2024-09-11T08:31:32.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:32 smithi181 ceph-mon[32701]: pgmap v1106: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-09-11T08:31:33.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:33 smithi150 ceph-mon[29276]: 4.8 scrub starts 2024-09-11T08:31:33.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:33 smithi150 ceph-mon[29276]: 4.8 scrub ok 2024-09-11T08:31:33.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:33 smithi150 ceph-mon[29276]: 4.3 scrub starts 2024-09-11T08:31:33.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:33 smithi150 ceph-mon[29276]: 4.3 scrub ok 2024-09-11T08:31:33.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:33 smithi150 ceph-mon[39612]: 4.8 scrub starts 2024-09-11T08:31:33.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:33 smithi150 ceph-mon[39612]: 4.8 scrub ok 2024-09-11T08:31:33.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:33 smithi150 ceph-mon[39612]: 4.3 scrub starts 2024-09-11T08:31:33.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:33 smithi150 ceph-mon[39612]: 4.3 scrub ok 2024-09-11T08:31:33.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:33 smithi181 ceph-mon[32701]: 4.8 scrub starts 2024-09-11T08:31:33.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:33 smithi181 ceph-mon[32701]: 4.8 scrub ok 2024-09-11T08:31:33.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:33 smithi181 ceph-mon[32701]: 4.3 scrub starts 2024-09-11T08:31:33.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:33 smithi181 ceph-mon[32701]: 4.3 scrub ok 2024-09-11T08:31:34.245 INFO:teuthology.orchestra.run.smithi150.stderr:reweighted osd.5 to 1 (10000) 2024-09-11T08:31:34.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:34 smithi181 ceph-mon[32701]: pgmap v1107: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:31:34.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:34 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3619759763' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-11T08:31:34.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:34 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-11T08:31:34.601 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:34 smithi150 ceph-mon[29276]: pgmap v1107: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:31:34.601 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:34 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3619759763' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-11T08:31:34.601 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:34 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-11T08:31:34.601 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:34 smithi150 ceph-mon[39612]: pgmap v1107: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:31:34.601 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:34 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3619759763' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-11T08:31:34.601 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:34 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-11T08:31:34.909 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 5 1 2024-09-11T08:31:35.154 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:31:35.249 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:35 smithi150 ceph-mon[29276]: 5.7 deep-scrub starts 2024-09-11T08:31:35.250 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:35 smithi150 ceph-mon[29276]: 5.7 deep-scrub ok 2024-09-11T08:31:35.250 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:35 smithi150 ceph-mon[29276]: 3.d scrub starts 2024-09-11T08:31:35.250 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:35 smithi150 ceph-mon[29276]: 3.d scrub ok 2024-09-11T08:31:35.250 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:35 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-11T08:31:35.250 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:35 smithi150 ceph-mon[29276]: osdmap e475: 8 total, 8 up, 8 in 2024-09-11T08:31:35.250 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:35 smithi150 ceph-mon[29276]: 4.1f deep-scrub starts 2024-09-11T08:31:35.250 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:35 smithi150 ceph-mon[29276]: 4.1f deep-scrub ok 2024-09-11T08:31:35.250 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:35 smithi150 ceph-mon[29276]: 3.17 scrub starts 2024-09-11T08:31:35.250 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:35 smithi150 ceph-mon[29276]: 3.17 scrub ok 2024-09-11T08:31:35.522 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:35 smithi150 ceph-mon[39612]: 5.7 deep-scrub starts 2024-09-11T08:31:35.522 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:35 smithi150 ceph-mon[39612]: 5.7 deep-scrub ok 2024-09-11T08:31:35.522 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:35 smithi150 ceph-mon[39612]: 3.d scrub starts 2024-09-11T08:31:35.522 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:35 smithi150 ceph-mon[39612]: 3.d scrub ok 2024-09-11T08:31:35.522 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:35 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-11T08:31:35.522 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:35 smithi150 ceph-mon[39612]: osdmap e475: 8 total, 8 up, 8 in 2024-09-11T08:31:35.522 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:35 smithi150 ceph-mon[39612]: 4.1f deep-scrub starts 2024-09-11T08:31:35.523 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:35 smithi150 ceph-mon[39612]: 4.1f deep-scrub ok 2024-09-11T08:31:35.523 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:35 smithi150 ceph-mon[39612]: 3.17 scrub starts 2024-09-11T08:31:35.523 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:35 smithi150 ceph-mon[39612]: 3.17 scrub ok 2024-09-11T08:31:35.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:31:35 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:31:35] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:31:35.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:35 smithi181 ceph-mon[32701]: 5.7 deep-scrub starts 2024-09-11T08:31:35.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:35 smithi181 ceph-mon[32701]: 5.7 deep-scrub ok 2024-09-11T08:31:35.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:35 smithi181 ceph-mon[32701]: 3.d scrub starts 2024-09-11T08:31:35.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:35 smithi181 ceph-mon[32701]: 3.d scrub ok 2024-09-11T08:31:35.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:35 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-11T08:31:35.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:35 smithi181 ceph-mon[32701]: osdmap e475: 8 total, 8 up, 8 in 2024-09-11T08:31:35.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:35 smithi181 ceph-mon[32701]: 4.1f deep-scrub starts 2024-09-11T08:31:35.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:35 smithi181 ceph-mon[32701]: 4.1f deep-scrub ok 2024-09-11T08:31:35.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:35 smithi181 ceph-mon[32701]: 3.17 scrub starts 2024-09-11T08:31:35.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:35 smithi181 ceph-mon[32701]: 3.17 scrub ok 2024-09-11T08:31:36.545 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:36 smithi150 ceph-mon[39612]: 6.4 scrub starts 2024-09-11T08:31:36.546 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:36 smithi150 ceph-mon[39612]: 6.4 scrub ok 2024-09-11T08:31:36.546 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:36 smithi150 ceph-mon[39612]: pgmap v1109: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:31:36.546 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:36 smithi150 ceph-mon[39612]: 3.12 scrub starts 2024-09-11T08:31:36.546 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:36 smithi150 ceph-mon[39612]: 3.12 scrub ok 2024-09-11T08:31:36.546 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:36 smithi150 ceph-mon[39612]: 6.1e scrub starts 2024-09-11T08:31:36.546 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:36 smithi150 ceph-mon[39612]: 6.1e scrub ok 2024-09-11T08:31:36.547 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:36 smithi150 ceph-mon[29276]: 6.4 scrub starts 2024-09-11T08:31:36.547 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:36 smithi150 ceph-mon[29276]: 6.4 scrub ok 2024-09-11T08:31:36.547 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:36 smithi150 ceph-mon[29276]: pgmap v1109: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:31:36.547 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:36 smithi150 ceph-mon[29276]: 3.12 scrub starts 2024-09-11T08:31:36.547 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:36 smithi150 ceph-mon[29276]: 3.12 scrub ok 2024-09-11T08:31:36.547 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:36 smithi150 ceph-mon[29276]: 6.1e scrub starts 2024-09-11T08:31:36.547 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:36 smithi150 ceph-mon[29276]: 6.1e scrub ok 2024-09-11T08:31:36.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:36 smithi181 ceph-mon[32701]: 6.4 scrub starts 2024-09-11T08:31:36.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:36 smithi181 ceph-mon[32701]: 6.4 scrub ok 2024-09-11T08:31:36.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:36 smithi181 ceph-mon[32701]: pgmap v1109: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:31:36.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:36 smithi181 ceph-mon[32701]: 3.12 scrub starts 2024-09-11T08:31:36.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:36 smithi181 ceph-mon[32701]: 3.12 scrub ok 2024-09-11T08:31:36.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:36 smithi181 ceph-mon[32701]: 6.1e scrub starts 2024-09-11T08:31:36.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:36 smithi181 ceph-mon[32701]: 6.1e scrub ok 2024-09-11T08:31:37.259 INFO:teuthology.orchestra.run.smithi150.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-11T08:31:37.559 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:37 smithi150 ceph-mon[29276]: 3.1f scrub starts 2024-09-11T08:31:37.559 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:37 smithi150 ceph-mon[29276]: 3.1f scrub ok 2024-09-11T08:31:37.559 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:37 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:31:37.559 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:37 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/324151476' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-09-11T08:31:37.560 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:37 smithi150 ceph-mon[39612]: 3.1f scrub starts 2024-09-11T08:31:37.560 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:37 smithi150 ceph-mon[39612]: 3.1f scrub ok 2024-09-11T08:31:37.560 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:37 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:31:37.560 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:37 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/324151476' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-09-11T08:31:37.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:37 smithi181 ceph-mon[32701]: 3.1f scrub starts 2024-09-11T08:31:37.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:37 smithi181 ceph-mon[32701]: 3.1f scrub ok 2024-09-11T08:31:37.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:37 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:31:37.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:37 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/324151476' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-09-11T08:31:37.913 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-11T08:31:37.913 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-11T08:31:37.913 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-11T08:31:37.913 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-11T08:31:37.913 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-11T08:31:37.913 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd dump --format=json 2024-09-11T08:31:38.166 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:31:38.556 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:38 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/324151476' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-11T08:31:38.556 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:38 smithi150 ceph-mon[29276]: osdmap e476: 8 total, 8 up, 8 in 2024-09-11T08:31:38.556 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:38 smithi150 ceph-mon[29276]: pgmap v1111: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 848 B/s rd, 0 op/s 2024-09-11T08:31:38.556 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:38 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/324151476' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-11T08:31:38.556 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:38 smithi150 ceph-mon[39612]: osdmap e476: 8 total, 8 up, 8 in 2024-09-11T08:31:38.556 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:38 smithi150 ceph-mon[39612]: pgmap v1111: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 848 B/s rd, 0 op/s 2024-09-11T08:31:38.572 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:38 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/324151476' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-11T08:31:38.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:38 smithi181 ceph-mon[32701]: osdmap e476: 8 total, 8 up, 8 in 2024-09-11T08:31:38.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:38 smithi181 ceph-mon[32701]: pgmap v1111: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 848 B/s rd, 0 op/s 2024-09-11T08:31:38.988 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:31:38 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:31:38] "GET /metrics HTTP/1.1" 200 222280 "" "Prometheus/2.43.0" 2024-09-11T08:31:39.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:39 smithi150 ceph-mon[29276]: 5.19 scrub starts 2024-09-11T08:31:39.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:39 smithi150 ceph-mon[29276]: 5.19 scrub ok 2024-09-11T08:31:39.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:39 smithi150 ceph-mon[29276]: 5.13 scrub starts 2024-09-11T08:31:39.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:39 smithi150 ceph-mon[29276]: 5.13 scrub ok 2024-09-11T08:31:39.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:39 smithi150 ceph-mon[29276]: osdmap e477: 8 total, 8 up, 8 in 2024-09-11T08:31:39.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:39 smithi150 ceph-mon[29276]: 5.1c scrub starts 2024-09-11T08:31:39.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:39 smithi150 ceph-mon[29276]: 5.1c scrub ok 2024-09-11T08:31:39.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:39 smithi150 ceph-mon[29276]: 5.17 scrub starts 2024-09-11T08:31:39.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:39 smithi150 ceph-mon[29276]: 5.17 scrub ok 2024-09-11T08:31:39.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:39 smithi150 ceph-mon[39612]: 5.19 scrub starts 2024-09-11T08:31:39.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:39 smithi150 ceph-mon[39612]: 5.19 scrub ok 2024-09-11T08:31:39.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:39 smithi150 ceph-mon[39612]: 5.13 scrub starts 2024-09-11T08:31:39.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:39 smithi150 ceph-mon[39612]: 5.13 scrub ok 2024-09-11T08:31:39.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:39 smithi150 ceph-mon[39612]: osdmap e477: 8 total, 8 up, 8 in 2024-09-11T08:31:39.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:39 smithi150 ceph-mon[39612]: 5.1c scrub starts 2024-09-11T08:31:39.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:39 smithi150 ceph-mon[39612]: 5.1c scrub ok 2024-09-11T08:31:39.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:39 smithi150 ceph-mon[39612]: 5.17 scrub starts 2024-09-11T08:31:39.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:39 smithi150 ceph-mon[39612]: 5.17 scrub ok 2024-09-11T08:31:39.573 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:31:39 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:31:39.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:39 smithi181 ceph-mon[32701]: 5.19 scrub starts 2024-09-11T08:31:39.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:39 smithi181 ceph-mon[32701]: 5.19 scrub ok 2024-09-11T08:31:39.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:39 smithi181 ceph-mon[32701]: 5.13 scrub starts 2024-09-11T08:31:39.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:39 smithi181 ceph-mon[32701]: 5.13 scrub ok 2024-09-11T08:31:39.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:39 smithi181 ceph-mon[32701]: osdmap e477: 8 total, 8 up, 8 in 2024-09-11T08:31:39.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:39 smithi181 ceph-mon[32701]: 5.1c scrub starts 2024-09-11T08:31:39.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:39 smithi181 ceph-mon[32701]: 5.1c scrub ok 2024-09-11T08:31:39.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:39 smithi181 ceph-mon[32701]: 5.17 scrub starts 2024-09-11T08:31:39.574 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:39 smithi181 ceph-mon[32701]: 5.17 scrub ok 2024-09-11T08:31:40.354 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:31:40.354 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":477,"fsid":"c1bddff4-7013-11ef-bcea-c7b262605968","created":"2024-09-11T08:02:15.344618+0000","modified":"2024-09-11T08:31:38.260817+0000","last_up_change":"2024-09-11T08:24:27.333279+0000","last_in_change":"2024-09-11T08:19:28.121267+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T08:05:21.044729+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"255","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"datapool","create_time":"2024-09-11T08:07:28.786572+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":63,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-09-11T08:07:29.021286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-11T08:07:30.334181+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-11T08:07:32.289626+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-11T08:07:34.351861+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"37b63db9-ed2a-4701-b63d-03ff9885cc0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":453,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6801","nonce":334255279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":334255279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":334255279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6803","nonce":334255279}]},"public_addr":"172.21.15.150:6801/334255279","cluster_addr":"172.21.15.150:6802/334255279","heartbeat_back_addr":"172.21.15.150:6804/334255279","heartbeat_front_addr":"172.21.15.150:6803/334255279","state":["exists","up"]},{"osd":1,"uuid":"80378442-02bc-48aa-b8ca-6113a4092079","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":382,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6805","nonce":1129510619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1129510619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1129510619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6807","nonce":1129510619}]},"public_addr":"172.21.15.150:6805/1129510619","cluster_addr":"172.21.15.150:6806/1129510619","heartbeat_back_addr":"172.21.15.150:6808/1129510619","heartbeat_front_addr":"172.21.15.150:6807/1129510619","state":["exists","up"]},{"osd":2,"uuid":"0955bb92-36a0-4e9a-98e1-b3a5af89f6d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":411,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6809","nonce":719975422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":719975422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":719975422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6811","nonce":719975422}]},"public_addr":"172.21.15.150:6809/719975422","cluster_addr":"172.21.15.150:6810/719975422","heartbeat_back_addr":"172.21.15.150:6812/719975422","heartbeat_front_addr":"172.21.15.150:6811/719975422","state":["exists","up"]},{"osd":3,"uuid":"1d04d4d3-62e8-4460-8a07-77f7e5c62929","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":464,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6813","nonce":3101032500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3101032500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3101032500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6815","nonce":3101032500}]},"public_addr":"172.21.15.150:6813/3101032500","cluster_addr":"172.21.15.150:6814/3101032500","heartbeat_back_addr":"172.21.15.150:6816/3101032500","heartbeat_front_addr":"172.21.15.150:6815/3101032500","state":["exists","up"]},{"osd":4,"uuid":"c5321a3c-d2f0-4440-8475-2c111d261139","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":467,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":2567452825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6801","nonce":2567452825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6803","nonce":2567452825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2567452825}]},"public_addr":"172.21.15.181:6800/2567452825","cluster_addr":"172.21.15.181:6801/2567452825","heartbeat_back_addr":"172.21.15.181:6803/2567452825","heartbeat_front_addr":"172.21.15.181:6802/2567452825","state":["exists","up"]},{"osd":5,"uuid":"1262b2fe-f253-4a26-bfd4-c9dffb697c77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":38,"last_clean_end":339,"up_from":353,"up_thru":476,"down_at":340,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":1821561027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6805","nonce":1821561027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6807","nonce":1821561027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":1821561027}]},"public_addr":"172.21.15.181:6804/1821561027","cluster_addr":"172.21.15.181:6805/1821561027","heartbeat_back_addr":"172.21.15.181:6807/1821561027","heartbeat_front_addr":"172.21.15.181:6806/1821561027","state":["exists","up"]},{"osd":6,"uuid":"7facca22-b3f1-4c20-833e-e3db798a08c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":382,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1789880331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6809","nonce":1789880331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6811","nonce":1789880331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1789880331}]},"public_addr":"172.21.15.181:6808/1789880331","cluster_addr":"172.21.15.181:6809/1789880331","heartbeat_back_addr":"172.21.15.181:6811/1789880331","heartbeat_front_addr":"172.21.15.181:6810/1789880331","state":["exists","up"]},{"osd":7,"uuid":"6ac1b06a-79bd-43ca-b91e-fcb0f4033a34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":53,"last_clean_end":139,"up_from":168,"up_thru":382,"down_at":140,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":2706455857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6813","nonce":2706455857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6815","nonce":2706455857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":2706455857}]},"public_addr":"172.21.15.181:6812/2706455857","cluster_addr":"172.21.15.181:6813/2706455857","heartbeat_back_addr":"172.21.15.181:6815/2706455857","heartbeat_front_addr":"172.21.15.181:6814/2706455857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:31:24.262221+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:30:57.943284+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:31:09.844892+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:30:30.627503+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:31:07.675314+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:31:23.884340+0000","dead_epoch":339},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:31:06.955361+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:31:05.429002+0000","dead_epoch":139}],"pg_upmap":[{"pgid":"3.4","osds":[4,2,3]},{"pgid":"5.1d","osds":[0,5,7]},{"pgid":"6.5","osds":[2,4,7]}],"pg_upmap_items":[{"pgid":"3.6","mappings":[{"from":4,"to":2}]},{"pgid":"4.e","mappings":[{"from":1,"to":5}]},{"pgid":"4.14","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":2}]},{"pgid":"5.3","mappings":[{"from":0,"to":3}]},{"pgid":"5.1e","mappings":[{"from":0,"to":6}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.150:0/2620931070":"2024-09-12T08:08:06.649692+0000","172.21.15.150:0/1073581409":"2024-09-12T08:02:45.986151+0000","172.21.15.150:0/2363649697":"2024-09-12T08:08:06.649692+0000","172.21.15.150:6800/603451071":"2024-09-12T08:02:45.986151+0000","172.21.15.150:0/1614413548":"2024-09-12T08:02:45.986151+0000","172.21.15.150:0/3673218166":"2024-09-12T08:03:08.740828+0000","172.21.15.150:6800/556498589":"2024-09-12T08:03:08.740828+0000","172.21.15.150:0/3666733994":"2024-09-12T08:03:08.740828+0000","172.21.15.150:0/3271479010":"2024-09-12T08:03:08.740828+0000","172.21.15.150:0/3185731090":"2024-09-12T08:08:06.649692+0000","172.21.15.150:0/2539412006":"2024-09-12T08:08:06.649692+0000","172.21.15.150:0/2740222408":"2024-09-12T08:02:45.986151+0000","172.21.15.150:6800/127856313":"2024-09-12T08:08:06.649692+0000","172.21.15.150:0/2375121603":"2024-09-12T08:08:06.649692+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T08:31:40.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:40 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:31:40.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:40 smithi181 ceph-mon[32701]: pgmap v1113: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:40.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:40 smithi181 ceph-mon[32701]: 4.13 scrub starts 2024-09-11T08:31:40.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:40 smithi181 ceph-mon[32701]: 4.13 scrub ok 2024-09-11T08:31:40.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:40 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:31:40.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:40 smithi150 ceph-mon[29276]: pgmap v1113: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:40.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:40 smithi150 ceph-mon[29276]: 4.13 scrub starts 2024-09-11T08:31:40.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:40 smithi150 ceph-mon[29276]: 4.13 scrub ok 2024-09-11T08:31:40.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:40 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:31:40.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:40 smithi150 ceph-mon[39612]: pgmap v1113: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:40.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:40 smithi150 ceph-mon[39612]: 4.13 scrub starts 2024-09-11T08:31:40.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:40 smithi150 ceph-mon[39612]: 4.13 scrub ok 2024-09-11T08:31:41.103 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-11T08:31:41.103 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd dump --format=json 2024-09-11T08:31:41.350 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:31:41.470 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:41 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2901957804' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-11T08:31:41.471 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:41 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2901957804' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-11T08:31:41.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:41 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2901957804' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-11T08:31:42.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:42 smithi150 ceph-mon[29276]: 6.6 scrub starts 2024-09-11T08:31:42.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:42 smithi150 ceph-mon[29276]: 6.6 scrub ok 2024-09-11T08:31:42.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:42 smithi150 ceph-mon[29276]: 5.f scrub starts 2024-09-11T08:31:42.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:42 smithi150 ceph-mon[29276]: 5.f scrub ok 2024-09-11T08:31:42.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:42 smithi150 ceph-mon[29276]: pgmap v1114: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 729 B/s rd, 0 op/s 2024-09-11T08:31:42.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:42 smithi150 ceph-mon[39612]: 6.6 scrub starts 2024-09-11T08:31:42.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:42 smithi150 ceph-mon[39612]: 6.6 scrub ok 2024-09-11T08:31:42.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:42 smithi150 ceph-mon[39612]: 5.f scrub starts 2024-09-11T08:31:42.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:42 smithi150 ceph-mon[39612]: 5.f scrub ok 2024-09-11T08:31:42.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:42 smithi150 ceph-mon[39612]: pgmap v1114: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 729 B/s rd, 0 op/s 2024-09-11T08:31:42.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:42 smithi181 ceph-mon[32701]: 6.6 scrub starts 2024-09-11T08:31:42.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:42 smithi181 ceph-mon[32701]: 6.6 scrub ok 2024-09-11T08:31:42.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:42 smithi181 ceph-mon[32701]: 5.f scrub starts 2024-09-11T08:31:42.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:42 smithi181 ceph-mon[32701]: 5.f scrub ok 2024-09-11T08:31:42.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:42 smithi181 ceph-mon[32701]: pgmap v1114: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 729 B/s rd, 0 op/s 2024-09-11T08:31:43.357 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:31:43.357 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":477,"fsid":"c1bddff4-7013-11ef-bcea-c7b262605968","created":"2024-09-11T08:02:15.344618+0000","modified":"2024-09-11T08:31:38.260817+0000","last_up_change":"2024-09-11T08:24:27.333279+0000","last_in_change":"2024-09-11T08:19:28.121267+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T08:05:21.044729+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"255","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"datapool","create_time":"2024-09-11T08:07:28.786572+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":63,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-09-11T08:07:29.021286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-11T08:07:30.334181+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-11T08:07:32.289626+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-11T08:07:34.351861+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"37b63db9-ed2a-4701-b63d-03ff9885cc0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":453,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6801","nonce":334255279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":334255279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":334255279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6803","nonce":334255279}]},"public_addr":"172.21.15.150:6801/334255279","cluster_addr":"172.21.15.150:6802/334255279","heartbeat_back_addr":"172.21.15.150:6804/334255279","heartbeat_front_addr":"172.21.15.150:6803/334255279","state":["exists","up"]},{"osd":1,"uuid":"80378442-02bc-48aa-b8ca-6113a4092079","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":382,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6805","nonce":1129510619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1129510619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1129510619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6807","nonce":1129510619}]},"public_addr":"172.21.15.150:6805/1129510619","cluster_addr":"172.21.15.150:6806/1129510619","heartbeat_back_addr":"172.21.15.150:6808/1129510619","heartbeat_front_addr":"172.21.15.150:6807/1129510619","state":["exists","up"]},{"osd":2,"uuid":"0955bb92-36a0-4e9a-98e1-b3a5af89f6d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":411,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6809","nonce":719975422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":719975422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":719975422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6811","nonce":719975422}]},"public_addr":"172.21.15.150:6809/719975422","cluster_addr":"172.21.15.150:6810/719975422","heartbeat_back_addr":"172.21.15.150:6812/719975422","heartbeat_front_addr":"172.21.15.150:6811/719975422","state":["exists","up"]},{"osd":3,"uuid":"1d04d4d3-62e8-4460-8a07-77f7e5c62929","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":464,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6813","nonce":3101032500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3101032500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3101032500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6815","nonce":3101032500}]},"public_addr":"172.21.15.150:6813/3101032500","cluster_addr":"172.21.15.150:6814/3101032500","heartbeat_back_addr":"172.21.15.150:6816/3101032500","heartbeat_front_addr":"172.21.15.150:6815/3101032500","state":["exists","up"]},{"osd":4,"uuid":"c5321a3c-d2f0-4440-8475-2c111d261139","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":467,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":2567452825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6801","nonce":2567452825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6803","nonce":2567452825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2567452825}]},"public_addr":"172.21.15.181:6800/2567452825","cluster_addr":"172.21.15.181:6801/2567452825","heartbeat_back_addr":"172.21.15.181:6803/2567452825","heartbeat_front_addr":"172.21.15.181:6802/2567452825","state":["exists","up"]},{"osd":5,"uuid":"1262b2fe-f253-4a26-bfd4-c9dffb697c77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":38,"last_clean_end":339,"up_from":353,"up_thru":476,"down_at":340,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":1821561027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6805","nonce":1821561027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6807","nonce":1821561027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":1821561027}]},"public_addr":"172.21.15.181:6804/1821561027","cluster_addr":"172.21.15.181:6805/1821561027","heartbeat_back_addr":"172.21.15.181:6807/1821561027","heartbeat_front_addr":"172.21.15.181:6806/1821561027","state":["exists","up"]},{"osd":6,"uuid":"7facca22-b3f1-4c20-833e-e3db798a08c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":382,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1789880331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6809","nonce":1789880331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6811","nonce":1789880331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1789880331}]},"public_addr":"172.21.15.181:6808/1789880331","cluster_addr":"172.21.15.181:6809/1789880331","heartbeat_back_addr":"172.21.15.181:6811/1789880331","heartbeat_front_addr":"172.21.15.181:6810/1789880331","state":["exists","up"]},{"osd":7,"uuid":"6ac1b06a-79bd-43ca-b91e-fcb0f4033a34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":53,"last_clean_end":139,"up_from":168,"up_thru":382,"down_at":140,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":2706455857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6813","nonce":2706455857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6815","nonce":2706455857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":2706455857}]},"public_addr":"172.21.15.181:6812/2706455857","cluster_addr":"172.21.15.181:6813/2706455857","heartbeat_back_addr":"172.21.15.181:6815/2706455857","heartbeat_front_addr":"172.21.15.181:6814/2706455857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:31:24.262221+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:30:57.943284+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:31:09.844892+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:30:30.627503+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:31:07.675314+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:31:23.884340+0000","dead_epoch":339},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:31:06.955361+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:31:05.429002+0000","dead_epoch":139}],"pg_upmap":[{"pgid":"3.4","osds":[4,2,3]},{"pgid":"5.1d","osds":[0,5,7]},{"pgid":"6.5","osds":[2,4,7]}],"pg_upmap_items":[{"pgid":"3.6","mappings":[{"from":4,"to":2}]},{"pgid":"4.e","mappings":[{"from":1,"to":5}]},{"pgid":"4.14","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":2}]},{"pgid":"5.3","mappings":[{"from":0,"to":3}]},{"pgid":"5.1e","mappings":[{"from":0,"to":6}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.150:0/2620931070":"2024-09-12T08:08:06.649692+0000","172.21.15.150:0/1073581409":"2024-09-12T08:02:45.986151+0000","172.21.15.150:0/2363649697":"2024-09-12T08:08:06.649692+0000","172.21.15.150:6800/603451071":"2024-09-12T08:02:45.986151+0000","172.21.15.150:0/1614413548":"2024-09-12T08:02:45.986151+0000","172.21.15.150:0/3673218166":"2024-09-12T08:03:08.740828+0000","172.21.15.150:6800/556498589":"2024-09-12T08:03:08.740828+0000","172.21.15.150:0/3666733994":"2024-09-12T08:03:08.740828+0000","172.21.15.150:0/3271479010":"2024-09-12T08:03:08.740828+0000","172.21.15.150:0/3185731090":"2024-09-12T08:08:06.649692+0000","172.21.15.150:0/2539412006":"2024-09-12T08:08:06.649692+0000","172.21.15.150:0/2740222408":"2024-09-12T08:02:45.986151+0000","172.21.15.150:6800/127856313":"2024-09-12T08:08:06.649692+0000","172.21.15.150:0/2375121603":"2024-09-12T08:08:06.649692+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T08:31:43.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:43 smithi181 ceph-mon[32701]: 4.0 scrub starts 2024-09-11T08:31:43.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:43 smithi181 ceph-mon[32701]: 4.0 scrub ok 2024-09-11T08:31:43.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:43 smithi181 ceph-mon[32701]: 5.16 scrub starts 2024-09-11T08:31:43.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:43 smithi181 ceph-mon[32701]: 5.16 scrub ok 2024-09-11T08:31:43.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:43 smithi150 ceph-mon[29276]: 4.0 scrub starts 2024-09-11T08:31:43.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:43 smithi150 ceph-mon[29276]: 4.0 scrub ok 2024-09-11T08:31:43.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:43 smithi150 ceph-mon[29276]: 5.16 scrub starts 2024-09-11T08:31:43.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:43 smithi150 ceph-mon[29276]: 5.16 scrub ok 2024-09-11T08:31:43.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:43 smithi150 ceph-mon[39612]: 4.0 scrub starts 2024-09-11T08:31:43.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:43 smithi150 ceph-mon[39612]: 4.0 scrub ok 2024-09-11T08:31:43.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:43 smithi150 ceph-mon[39612]: 5.16 scrub starts 2024-09-11T08:31:43.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:43 smithi150 ceph-mon[39612]: 5.16 scrub ok 2024-09-11T08:31:44.044 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-11T08:31:44.045 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-11T08:31:44.045 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-11T08:31:44.046 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-11T08:31:44.046 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-11T08:31:44.047 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-11T08:31:44.047 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-11T08:31:44.048 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-11T08:31:44.510 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:31:44.559 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:31:44.559 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:31:44.563 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:31:44.564 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:31:44.564 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:31:44.569 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:31:44.569 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:31:44.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:44 smithi181 ceph-mon[32701]: 3.1c scrub starts 2024-09-11T08:31:44.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:44 smithi181 ceph-mon[32701]: 3.1c scrub ok 2024-09-11T08:31:44.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:44 smithi181 ceph-mon[32701]: pgmap v1115: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-11T08:31:44.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:44 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1563833861' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-11T08:31:44.647 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:44 smithi150 ceph-mon[29276]: 3.1c scrub starts 2024-09-11T08:31:44.647 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:44 smithi150 ceph-mon[29276]: 3.1c scrub ok 2024-09-11T08:31:44.647 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:44 smithi150 ceph-mon[29276]: pgmap v1115: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-11T08:31:44.647 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:44 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1563833861' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-11T08:31:44.647 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:44 smithi150 ceph-mon[39612]: 3.1c scrub starts 2024-09-11T08:31:44.647 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:44 smithi150 ceph-mon[39612]: 3.1c scrub ok 2024-09-11T08:31:44.647 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:44 smithi150 ceph-mon[39612]: pgmap v1115: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-11T08:31:44.648 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:44 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1563833861' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-11T08:31:45.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:45 smithi150 ceph-mon[29276]: 3.b scrub starts 2024-09-11T08:31:45.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:45 smithi150 ceph-mon[29276]: 3.b scrub ok 2024-09-11T08:31:45.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:45 smithi150 ceph-mon[39612]: 3.b scrub starts 2024-09-11T08:31:45.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:45 smithi150 ceph-mon[39612]: 3.b scrub ok 2024-09-11T08:31:45.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:45 smithi181 ceph-mon[32701]: 3.b scrub starts 2024-09-11T08:31:45.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:45 smithi181 ceph-mon[32701]: 3.b scrub ok 2024-09-11T08:31:45.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:31:45 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:31:45] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:31:46.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:46 smithi150 ceph-mon[29276]: 3.1 scrub starts 2024-09-11T08:31:46.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:46 smithi150 ceph-mon[29276]: 3.1 scrub ok 2024-09-11T08:31:46.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:46 smithi150 ceph-mon[29276]: pgmap v1116: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-11T08:31:46.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:46 smithi150 ceph-mon[29276]: 3.9 scrub starts 2024-09-11T08:31:46.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:46 smithi150 ceph-mon[29276]: 3.9 scrub ok 2024-09-11T08:31:46.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:46 smithi150 ceph-mon[39612]: 3.1 scrub starts 2024-09-11T08:31:46.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:46 smithi150 ceph-mon[39612]: 3.1 scrub ok 2024-09-11T08:31:46.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:46 smithi150 ceph-mon[39612]: pgmap v1116: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-11T08:31:46.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:46 smithi150 ceph-mon[39612]: 3.9 scrub starts 2024-09-11T08:31:46.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:46 smithi150 ceph-mon[39612]: 3.9 scrub ok 2024-09-11T08:31:46.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:46 smithi181 ceph-mon[32701]: 3.1 scrub starts 2024-09-11T08:31:46.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:46 smithi181 ceph-mon[32701]: 3.1 scrub ok 2024-09-11T08:31:46.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:46 smithi181 ceph-mon[32701]: pgmap v1116: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-11T08:31:46.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:46 smithi181 ceph-mon[32701]: 3.9 scrub starts 2024-09-11T08:31:46.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:46 smithi181 ceph-mon[32701]: 3.9 scrub ok 2024-09-11T08:31:47.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:47 smithi150 ceph-mon[29276]: 4.7 scrub starts 2024-09-11T08:31:47.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:47 smithi150 ceph-mon[29276]: 4.7 scrub ok 2024-09-11T08:31:47.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:47 smithi150 ceph-mon[29276]: 5.8 scrub starts 2024-09-11T08:31:47.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:47 smithi150 ceph-mon[29276]: 5.8 scrub ok 2024-09-11T08:31:47.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:47 smithi150 ceph-mon[29276]: 6.12 scrub starts 2024-09-11T08:31:47.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:47 smithi150 ceph-mon[29276]: 6.12 scrub ok 2024-09-11T08:31:47.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:47 smithi150 ceph-mon[39612]: 4.7 scrub starts 2024-09-11T08:31:47.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:47 smithi150 ceph-mon[39612]: 4.7 scrub ok 2024-09-11T08:31:47.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:47 smithi150 ceph-mon[39612]: 5.8 scrub starts 2024-09-11T08:31:47.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:47 smithi150 ceph-mon[39612]: 5.8 scrub ok 2024-09-11T08:31:47.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:47 smithi150 ceph-mon[39612]: 6.12 scrub starts 2024-09-11T08:31:47.490 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:47 smithi150 ceph-mon[39612]: 6.12 scrub ok 2024-09-11T08:31:47.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:47 smithi181 ceph-mon[32701]: 4.7 scrub starts 2024-09-11T08:31:47.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:47 smithi181 ceph-mon[32701]: 4.7 scrub ok 2024-09-11T08:31:47.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:47 smithi181 ceph-mon[32701]: 5.8 scrub starts 2024-09-11T08:31:47.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:47 smithi181 ceph-mon[32701]: 5.8 scrub ok 2024-09-11T08:31:47.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:47 smithi181 ceph-mon[32701]: 6.12 scrub starts 2024-09-11T08:31:47.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:47 smithi181 ceph-mon[32701]: 6.12 scrub ok 2024-09-11T08:31:48.568 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:48 smithi150 ceph-mon[29276]: 4.2 scrub starts 2024-09-11T08:31:48.568 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:48 smithi150 ceph-mon[29276]: 4.2 scrub ok 2024-09-11T08:31:48.568 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:48 smithi150 ceph-mon[29276]: 3.5 scrub starts 2024-09-11T08:31:48.568 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:48 smithi150 ceph-mon[29276]: 3.5 scrub ok 2024-09-11T08:31:48.568 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:48 smithi150 ceph-mon[29276]: pgmap v1117: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-09-11T08:31:48.569 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:48 smithi150 ceph-mon[29276]: 4.d scrub starts 2024-09-11T08:31:48.569 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:48 smithi150 ceph-mon[29276]: 4.d scrub ok 2024-09-11T08:31:48.569 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:48 smithi150 ceph-mon[39612]: 4.2 scrub starts 2024-09-11T08:31:48.569 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:48 smithi150 ceph-mon[39612]: 4.2 scrub ok 2024-09-11T08:31:48.569 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:48 smithi150 ceph-mon[39612]: 3.5 scrub starts 2024-09-11T08:31:48.569 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:48 smithi150 ceph-mon[39612]: 3.5 scrub ok 2024-09-11T08:31:48.569 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:48 smithi150 ceph-mon[39612]: pgmap v1117: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-09-11T08:31:48.569 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:48 smithi150 ceph-mon[39612]: 4.d scrub starts 2024-09-11T08:31:48.569 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:48 smithi150 ceph-mon[39612]: 4.d scrub ok 2024-09-11T08:31:48.572 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:48 smithi181 ceph-mon[32701]: 4.2 scrub starts 2024-09-11T08:31:48.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:48 smithi181 ceph-mon[32701]: 4.2 scrub ok 2024-09-11T08:31:48.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:48 smithi181 ceph-mon[32701]: 3.5 scrub starts 2024-09-11T08:31:48.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:48 smithi181 ceph-mon[32701]: 3.5 scrub ok 2024-09-11T08:31:48.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:48 smithi181 ceph-mon[32701]: pgmap v1117: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-09-11T08:31:48.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:48 smithi181 ceph-mon[32701]: 4.d scrub starts 2024-09-11T08:31:48.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:48 smithi181 ceph-mon[32701]: 4.d scrub ok 2024-09-11T08:31:48.835 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:31:48 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:31:48] "GET /metrics HTTP/1.1" 200 222280 "" "Prometheus/2.43.0" 2024-09-11T08:31:49.441 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:49 smithi150 ceph-mon[39612]: 6.7 deep-scrub starts 2024-09-11T08:31:49.442 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:49 smithi150 ceph-mon[39612]: 6.7 deep-scrub ok 2024-09-11T08:31:49.442 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:49 smithi150 ceph-mon[39612]: 4.b scrub starts 2024-09-11T08:31:49.442 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:49 smithi150 ceph-mon[39612]: 4.b scrub ok 2024-09-11T08:31:49.442 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:49 smithi150 ceph-mon[39612]: 4.e scrub starts 2024-09-11T08:31:49.442 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:49 smithi150 ceph-mon[39612]: 4.e scrub ok 2024-09-11T08:31:49.442 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:49 smithi150 ceph-mon[39612]: 1.0 scrub starts 2024-09-11T08:31:49.442 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:49 smithi150 ceph-mon[39612]: 1.0 scrub ok 2024-09-11T08:31:49.442 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:49 smithi150 ceph-mon[39612]: 3.18 deep-scrub starts 2024-09-11T08:31:49.442 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:49 smithi150 ceph-mon[39612]: 3.18 deep-scrub ok 2024-09-11T08:31:49.442 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:49 smithi150 ceph-mon[29276]: 6.7 deep-scrub starts 2024-09-11T08:31:49.442 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:49 smithi150 ceph-mon[29276]: 6.7 deep-scrub ok 2024-09-11T08:31:49.443 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:49 smithi150 ceph-mon[29276]: 4.b scrub starts 2024-09-11T08:31:49.443 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:49 smithi150 ceph-mon[29276]: 4.b scrub ok 2024-09-11T08:31:49.443 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:49 smithi150 ceph-mon[29276]: 4.e scrub starts 2024-09-11T08:31:49.443 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:49 smithi150 ceph-mon[29276]: 4.e scrub ok 2024-09-11T08:31:49.443 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:49 smithi150 ceph-mon[29276]: 1.0 scrub starts 2024-09-11T08:31:49.443 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:49 smithi150 ceph-mon[29276]: 1.0 scrub ok 2024-09-11T08:31:49.443 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:49 smithi150 ceph-mon[29276]: 3.18 deep-scrub starts 2024-09-11T08:31:49.443 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:49 smithi150 ceph-mon[29276]: 3.18 deep-scrub ok 2024-09-11T08:31:49.486 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:31:49 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:31:49.486 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:49 smithi181 ceph-mon[32701]: 6.7 deep-scrub starts 2024-09-11T08:31:49.486 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:49 smithi181 ceph-mon[32701]: 6.7 deep-scrub ok 2024-09-11T08:31:49.486 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:49 smithi181 ceph-mon[32701]: 4.b scrub starts 2024-09-11T08:31:49.486 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:49 smithi181 ceph-mon[32701]: 4.b scrub ok 2024-09-11T08:31:49.486 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:49 smithi181 ceph-mon[32701]: 4.e scrub starts 2024-09-11T08:31:49.486 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:49 smithi181 ceph-mon[32701]: 4.e scrub ok 2024-09-11T08:31:49.486 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:49 smithi181 ceph-mon[32701]: 1.0 scrub starts 2024-09-11T08:31:49.487 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:49 smithi181 ceph-mon[32701]: 1.0 scrub ok 2024-09-11T08:31:49.487 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:49 smithi181 ceph-mon[32701]: 3.18 deep-scrub starts 2024-09-11T08:31:49.487 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:49 smithi181 ceph-mon[32701]: 3.18 deep-scrub ok 2024-09-11T08:31:49.872 INFO:teuthology.orchestra.run.smithi150.stdout:193273528622 2024-09-11T08:31:49.873 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-11T08:31:50.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:50 smithi150 ceph-mon[29276]: 5.15 scrub starts 2024-09-11T08:31:50.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:50 smithi150 ceph-mon[29276]: 5.15 scrub ok 2024-09-11T08:31:50.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:50 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:31:50.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:50 smithi150 ceph-mon[29276]: pgmap v1118: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 744 B/s rd, 0 op/s 2024-09-11T08:31:50.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:50 smithi150 ceph-mon[29276]: 3.2 scrub starts 2024-09-11T08:31:50.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:50 smithi150 ceph-mon[29276]: 3.2 scrub ok 2024-09-11T08:31:50.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:50 smithi150 ceph-mon[39612]: 5.15 scrub starts 2024-09-11T08:31:50.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:50 smithi150 ceph-mon[39612]: 5.15 scrub ok 2024-09-11T08:31:50.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:50 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:31:50.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:50 smithi150 ceph-mon[39612]: pgmap v1118: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 744 B/s rd, 0 op/s 2024-09-11T08:31:50.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:50 smithi150 ceph-mon[39612]: 3.2 scrub starts 2024-09-11T08:31:50.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:50 smithi150 ceph-mon[39612]: 3.2 scrub ok 2024-09-11T08:31:50.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:50 smithi181 ceph-mon[32701]: 5.15 scrub starts 2024-09-11T08:31:50.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:50 smithi181 ceph-mon[32701]: 5.15 scrub ok 2024-09-11T08:31:50.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:50 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:31:50.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:50 smithi181 ceph-mon[32701]: pgmap v1118: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 744 B/s rd, 0 op/s 2024-09-11T08:31:50.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:50 smithi181 ceph-mon[32701]: 3.2 scrub starts 2024-09-11T08:31:50.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:50 smithi181 ceph-mon[32701]: 3.2 scrub ok 2024-09-11T08:31:51.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:51 smithi181 ceph-mon[32701]: 3.1d scrub starts 2024-09-11T08:31:51.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:51 smithi181 ceph-mon[32701]: 3.1d scrub ok 2024-09-11T08:31:51.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:51 smithi181 ceph-mon[32701]: 4.5 scrub starts 2024-09-11T08:31:51.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:51 smithi181 ceph-mon[32701]: 4.5 scrub ok 2024-09-11T08:31:51.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:51 smithi181 ceph-mon[32701]: 3.3 scrub starts 2024-09-11T08:31:51.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:51 smithi181 ceph-mon[32701]: 3.3 scrub ok 2024-09-11T08:31:51.613 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:51 smithi150 ceph-mon[29276]: 3.1d scrub starts 2024-09-11T08:31:51.613 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:51 smithi150 ceph-mon[29276]: 3.1d scrub ok 2024-09-11T08:31:51.613 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:51 smithi150 ceph-mon[29276]: 4.5 scrub starts 2024-09-11T08:31:51.613 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:51 smithi150 ceph-mon[29276]: 4.5 scrub ok 2024-09-11T08:31:51.613 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:51 smithi150 ceph-mon[29276]: 3.3 scrub starts 2024-09-11T08:31:51.614 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:51 smithi150 ceph-mon[29276]: 3.3 scrub ok 2024-09-11T08:31:51.614 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:51 smithi150 ceph-mon[39612]: 3.1d scrub starts 2024-09-11T08:31:51.614 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:51 smithi150 ceph-mon[39612]: 3.1d scrub ok 2024-09-11T08:31:51.614 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:51 smithi150 ceph-mon[39612]: 4.5 scrub starts 2024-09-11T08:31:51.614 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:51 smithi150 ceph-mon[39612]: 4.5 scrub ok 2024-09-11T08:31:51.614 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:51 smithi150 ceph-mon[39612]: 3.3 scrub starts 2024-09-11T08:31:51.614 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:51 smithi150 ceph-mon[39612]: 3.3 scrub ok 2024-09-11T08:31:51.639 INFO:teuthology.orchestra.run.smithi150.stdout:77309411648 2024-09-11T08:31:51.640 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-11T08:31:52.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:52 smithi181 ceph-mon[32701]: pgmap v1119: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-11T08:31:52.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:52 smithi181 ceph-mon[32701]: 4.a scrub starts 2024-09-11T08:31:52.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:52 smithi181 ceph-mon[32701]: 4.a scrub ok 2024-09-11T08:31:52.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:52 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:31:52.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:52 smithi150 ceph-mon[29276]: pgmap v1119: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-11T08:31:52.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:52 smithi150 ceph-mon[29276]: 4.a scrub starts 2024-09-11T08:31:52.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:52 smithi150 ceph-mon[29276]: 4.a scrub ok 2024-09-11T08:31:52.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:52 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:31:52.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:52 smithi150 ceph-mon[39612]: pgmap v1119: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-11T08:31:52.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:52 smithi150 ceph-mon[39612]: 4.a scrub starts 2024-09-11T08:31:52.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:52 smithi150 ceph-mon[39612]: 4.a scrub ok 2024-09-11T08:31:52.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:52 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:31:54.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:54 smithi181 ceph-mon[32701]: pgmap v1120: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-11T08:31:54.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:54 smithi181 ceph-mon[32701]: 6.b scrub starts 2024-09-11T08:31:54.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:54 smithi181 ceph-mon[32701]: 6.b scrub ok 2024-09-11T08:31:54.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:54 smithi150 ceph-mon[29276]: pgmap v1120: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-11T08:31:54.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:54 smithi150 ceph-mon[29276]: 6.b scrub starts 2024-09-11T08:31:54.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:54 smithi150 ceph-mon[29276]: 6.b scrub ok 2024-09-11T08:31:54.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:54 smithi150 ceph-mon[39612]: pgmap v1120: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-11T08:31:54.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:54 smithi150 ceph-mon[39612]: 6.b scrub starts 2024-09-11T08:31:54.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:54 smithi150 ceph-mon[39612]: 6.b scrub ok 2024-09-11T08:31:55.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:55 smithi181 ceph-mon[32701]: 5.1 scrub starts 2024-09-11T08:31:55.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:55 smithi181 ceph-mon[32701]: 5.1 scrub ok 2024-09-11T08:31:55.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:31:55 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:31:55] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:31:55.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:55 smithi150 ceph-mon[29276]: 5.1 scrub starts 2024-09-11T08:31:55.738 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:55 smithi150 ceph-mon[29276]: 5.1 scrub ok 2024-09-11T08:31:55.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:55 smithi150 ceph-mon[39612]: 5.1 scrub starts 2024-09-11T08:31:55.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:55 smithi150 ceph-mon[39612]: 5.1 scrub ok 2024-09-11T08:31:56.439 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:56 smithi150 ceph-mon[39612]: pgmap v1121: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:56.439 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:56 smithi150 ceph-mon[39612]: 2.0 scrub starts 2024-09-11T08:31:56.439 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:56 smithi150 ceph-mon[39612]: 2.0 scrub ok 2024-09-11T08:31:56.439 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:56 smithi150 ceph-mon[29276]: pgmap v1121: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:56.439 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:56 smithi150 ceph-mon[29276]: 2.0 scrub starts 2024-09-11T08:31:56.439 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:56 smithi150 ceph-mon[29276]: 2.0 scrub ok 2024-09-11T08:31:56.764 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:31:56.764 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:31:56.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:56 smithi181 ceph-mon[32701]: pgmap v1121: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:56.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:56 smithi181 ceph-mon[32701]: 2.0 scrub starts 2024-09-11T08:31:56.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:56 smithi181 ceph-mon[32701]: 2.0 scrub ok 2024-09-11T08:31:57.617 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:57 smithi150 ceph-mon[29276]: 3.10 scrub starts 2024-09-11T08:31:57.617 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:57 smithi150 ceph-mon[29276]: 3.10 scrub ok 2024-09-11T08:31:57.617 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:57 smithi150 ceph-mon[29276]: 4.14 deep-scrub starts 2024-09-11T08:31:57.617 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:57 smithi150 ceph-mon[29276]: 4.14 deep-scrub ok 2024-09-11T08:31:57.618 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:57 smithi150 ceph-mon[39612]: 3.10 scrub starts 2024-09-11T08:31:57.618 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:57 smithi150 ceph-mon[39612]: 3.10 scrub ok 2024-09-11T08:31:57.618 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:57 smithi150 ceph-mon[39612]: 4.14 deep-scrub starts 2024-09-11T08:31:57.618 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:57 smithi150 ceph-mon[39612]: 4.14 deep-scrub ok 2024-09-11T08:31:57.822 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:57 smithi181 ceph-mon[32701]: 3.10 scrub starts 2024-09-11T08:31:57.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:57 smithi181 ceph-mon[32701]: 3.10 scrub ok 2024-09-11T08:31:57.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:57 smithi181 ceph-mon[32701]: 4.14 deep-scrub starts 2024-09-11T08:31:57.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:57 smithi181 ceph-mon[32701]: 4.14 deep-scrub ok 2024-09-11T08:31:58.549 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:58 smithi150 ceph-mon[29276]: pgmap v1122: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:31:58.550 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:58 smithi150 ceph-mon[39612]: pgmap v1122: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:31:58.822 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:58 smithi181 ceph-mon[32701]: pgmap v1122: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:31:58.988 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:31:58 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:31:58] "GET /metrics HTTP/1.1" 200 222287 "" "Prometheus/2.43.0" 2024-09-11T08:31:59.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:59 smithi150 ceph-mon[29276]: 6.9 scrub starts 2024-09-11T08:31:59.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:59 smithi150 ceph-mon[29276]: 6.9 scrub ok 2024-09-11T08:31:59.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:59 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:31:59.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:59 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:31:59.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:59 smithi150 ceph-mon[29276]: 4.9 scrub starts 2024-09-11T08:31:59.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:31:59 smithi150 ceph-mon[29276]: 4.9 scrub ok 2024-09-11T08:31:59.488 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:59 smithi150 ceph-mon[39612]: 6.9 scrub starts 2024-09-11T08:31:59.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:59 smithi150 ceph-mon[39612]: 6.9 scrub ok 2024-09-11T08:31:59.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:59 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:31:59.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:59 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:31:59.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:59 smithi150 ceph-mon[39612]: 4.9 scrub starts 2024-09-11T08:31:59.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:31:59 smithi150 ceph-mon[39612]: 4.9 scrub ok 2024-09-11T08:31:59.573 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:31:59 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:31:59.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:59 smithi181 ceph-mon[32701]: 6.9 scrub starts 2024-09-11T08:31:59.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:59 smithi181 ceph-mon[32701]: 6.9 scrub ok 2024-09-11T08:31:59.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:59 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:31:59.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:59 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:31:59.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:59 smithi181 ceph-mon[32701]: 4.9 scrub starts 2024-09-11T08:31:59.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:31:59 smithi181 ceph-mon[32701]: 4.9 scrub ok 2024-09-11T08:32:00.097 INFO:teuthology.orchestra.run.smithi150.stdout:55834575175 2024-09-11T08:32:00.098 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-11T08:32:00.572 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:00 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:32:00.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:00 smithi181 ceph-mon[32701]: pgmap v1123: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:32:00.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:00 smithi181 ceph-mon[32701]: osdmap e478: 8 total, 8 up, 8 in 2024-09-11T08:32:00.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:00 smithi181 ceph-mon[32701]: 4.1b scrub starts 2024-09-11T08:32:00.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:00 smithi181 ceph-mon[32701]: 4.1b scrub ok 2024-09-11T08:32:00.623 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:00 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:32:00.623 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:00 smithi150 ceph-mon[29276]: pgmap v1123: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:32:00.623 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:00 smithi150 ceph-mon[29276]: osdmap e478: 8 total, 8 up, 8 in 2024-09-11T08:32:00.623 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:00 smithi150 ceph-mon[29276]: 4.1b scrub starts 2024-09-11T08:32:00.623 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:00 smithi150 ceph-mon[29276]: 4.1b scrub ok 2024-09-11T08:32:00.624 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:00 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:32:00.624 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:00 smithi150 ceph-mon[39612]: pgmap v1123: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:32:00.624 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:00 smithi150 ceph-mon[39612]: osdmap e478: 8 total, 8 up, 8 in 2024-09-11T08:32:00.624 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:00 smithi150 ceph-mon[39612]: 4.1b scrub starts 2024-09-11T08:32:00.624 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:00 smithi150 ceph-mon[39612]: 4.1b scrub ok 2024-09-11T08:32:00.647 INFO:teuthology.orchestra.run.smithi150.stdout:107374182718 2024-09-11T08:32:00.647 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-11T08:32:01.061 INFO:teuthology.orchestra.run.smithi150.stdout:721554505940 2024-09-11T08:32:01.062 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-11T08:32:01.405 INFO:teuthology.orchestra.run.smithi150.stdout:34359738699 2024-09-11T08:32:01.406 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-11T08:32:01.599 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:01 smithi150 ceph-mon[29276]: 5.18 scrub starts 2024-09-11T08:32:01.600 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:01 smithi150 ceph-mon[29276]: 5.18 scrub ok 2024-09-11T08:32:01.600 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:01 smithi150 ceph-mon[29276]: 3.15 scrub starts 2024-09-11T08:32:01.600 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:01 smithi150 ceph-mon[29276]: 3.15 scrub ok 2024-09-11T08:32:01.600 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:01 smithi150 ceph-mon[39612]: 5.18 scrub starts 2024-09-11T08:32:01.600 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:01 smithi150 ceph-mon[39612]: 5.18 scrub ok 2024-09-11T08:32:01.600 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:01 smithi150 ceph-mon[39612]: 3.15 scrub starts 2024-09-11T08:32:01.600 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:01 smithi150 ceph-mon[39612]: 3.15 scrub ok 2024-09-11T08:32:01.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:01 smithi181 ceph-mon[32701]: 5.18 scrub starts 2024-09-11T08:32:01.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:01 smithi181 ceph-mon[32701]: 5.18 scrub ok 2024-09-11T08:32:01.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:01 smithi181 ceph-mon[32701]: 3.15 scrub starts 2024-09-11T08:32:01.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:01 smithi181 ceph-mon[32701]: 3.15 scrub ok 2024-09-11T08:32:02.598 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:02 smithi150 ceph-mon[39612]: 5.1d deep-scrub starts 2024-09-11T08:32:02.598 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:02 smithi150 ceph-mon[39612]: 5.1d deep-scrub ok 2024-09-11T08:32:02.598 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:02 smithi150 ceph-mon[39612]: pgmap v1125: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-11T08:32:02.598 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:02 smithi150 ceph-mon[29276]: 5.1d deep-scrub starts 2024-09-11T08:32:02.598 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:02 smithi150 ceph-mon[29276]: 5.1d deep-scrub ok 2024-09-11T08:32:02.599 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:02 smithi150 ceph-mon[29276]: pgmap v1125: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-11T08:32:02.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:02 smithi181 ceph-mon[32701]: 5.1d deep-scrub starts 2024-09-11T08:32:02.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:02 smithi181 ceph-mon[32701]: 5.1d deep-scrub ok 2024-09-11T08:32:02.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:02 smithi181 ceph-mon[32701]: pgmap v1125: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-11T08:32:03.369 INFO:teuthology.orchestra.run.smithi150.stdout:1516123455580 2024-09-11T08:32:03.369 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-11T08:32:03.605 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:03 smithi150 ceph-mon[29276]: 4.11 scrub starts 2024-09-11T08:32:03.605 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:03 smithi150 ceph-mon[29276]: 4.11 scrub ok 2024-09-11T08:32:03.605 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:03 smithi150 ceph-mon[29276]: 5.e scrub starts 2024-09-11T08:32:03.606 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:03 smithi150 ceph-mon[29276]: 5.e scrub ok 2024-09-11T08:32:03.606 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:03 smithi150 ceph-mon[39612]: 4.11 scrub starts 2024-09-11T08:32:03.606 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:03 smithi150 ceph-mon[39612]: 4.11 scrub ok 2024-09-11T08:32:03.606 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:03 smithi150 ceph-mon[39612]: 5.e scrub starts 2024-09-11T08:32:03.606 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:03 smithi150 ceph-mon[39612]: 5.e scrub ok 2024-09-11T08:32:03.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:03 smithi181 ceph-mon[32701]: 4.11 scrub starts 2024-09-11T08:32:03.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:03 smithi181 ceph-mon[32701]: 4.11 scrub ok 2024-09-11T08:32:03.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:03 smithi181 ceph-mon[32701]: 5.e scrub starts 2024-09-11T08:32:03.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:03 smithi181 ceph-mon[32701]: 5.e scrub ok 2024-09-11T08:32:03.993 INFO:teuthology.orchestra.run.smithi150.stdout:77309411650 2024-09-11T08:32:04.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:04 smithi150 ceph-mon[29276]: 4.19 scrub starts 2024-09-11T08:32:04.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:04 smithi150 ceph-mon[29276]: 4.19 scrub ok 2024-09-11T08:32:04.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:04 smithi150 ceph-mon[29276]: pgmap v1126: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-11T08:32:04.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:04 smithi150 ceph-mon[29276]: 5.4 deep-scrub starts 2024-09-11T08:32:04.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:04 smithi150 ceph-mon[29276]: 5.4 deep-scrub ok 2024-09-11T08:32:04.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:04 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2803204415' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-11T08:32:04.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:04 smithi150 ceph-mon[39612]: 4.19 scrub starts 2024-09-11T08:32:04.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:04 smithi150 ceph-mon[39612]: 4.19 scrub ok 2024-09-11T08:32:04.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:04 smithi150 ceph-mon[39612]: pgmap v1126: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-11T08:32:04.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:04 smithi150 ceph-mon[39612]: 5.4 deep-scrub starts 2024-09-11T08:32:04.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:04 smithi150 ceph-mon[39612]: 5.4 deep-scrub ok 2024-09-11T08:32:04.489 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:04 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2803204415' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-11T08:32:04.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:04 smithi181 ceph-mon[32701]: 4.19 scrub starts 2024-09-11T08:32:04.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:04 smithi181 ceph-mon[32701]: 4.19 scrub ok 2024-09-11T08:32:04.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:04 smithi181 ceph-mon[32701]: pgmap v1126: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-11T08:32:04.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:04 smithi181 ceph-mon[32701]: 5.4 deep-scrub starts 2024-09-11T08:32:04.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:04 smithi181 ceph-mon[32701]: 5.4 deep-scrub ok 2024-09-11T08:32:04.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:04 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2803204415' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-11T08:32:04.890 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411648 got 77309411650 for osd.2 2024-09-11T08:32:04.891 DEBUG:teuthology.parallel:result is None 2024-09-11T08:32:05.458 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:05 smithi150 ceph-mon[29276]: 6.1d scrub starts 2024-09-11T08:32:05.459 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:05 smithi150 ceph-mon[29276]: 6.1d scrub ok 2024-09-11T08:32:05.459 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:05 smithi150 ceph-mon[29276]: 6.1a scrub starts 2024-09-11T08:32:05.459 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:05 smithi150 ceph-mon[29276]: 6.1a scrub ok 2024-09-11T08:32:05.459 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:05 smithi150 ceph-mon[39612]: 6.1d scrub starts 2024-09-11T08:32:05.459 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:05 smithi150 ceph-mon[39612]: 6.1d scrub ok 2024-09-11T08:32:05.459 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:05 smithi150 ceph-mon[39612]: 6.1a scrub starts 2024-09-11T08:32:05.459 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:05 smithi150 ceph-mon[39612]: 6.1a scrub ok 2024-09-11T08:32:05.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:32:05 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:32:05] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:32:05.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:05 smithi181 ceph-mon[32701]: 6.1d scrub starts 2024-09-11T08:32:05.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:05 smithi181 ceph-mon[32701]: 6.1d scrub ok 2024-09-11T08:32:05.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:05 smithi181 ceph-mon[32701]: 6.1a scrub starts 2024-09-11T08:32:05.573 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:05 smithi181 ceph-mon[32701]: 6.1a scrub ok 2024-09-11T08:32:06.148 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:32:06.149 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:32:06.310 INFO:teuthology.orchestra.run.smithi150.stdout:193273528625 2024-09-11T08:32:06.675 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:06 smithi150 ceph-mon[29276]: 4.f deep-scrub starts 2024-09-11T08:32:06.675 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:06 smithi150 ceph-mon[29276]: 4.f deep-scrub ok 2024-09-11T08:32:06.675 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:06 smithi150 ceph-mon[29276]: 5.b scrub starts 2024-09-11T08:32:06.676 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:06 smithi150 ceph-mon[29276]: 5.b scrub ok 2024-09-11T08:32:06.676 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:06 smithi150 ceph-mon[29276]: pgmap v1127: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-11T08:32:06.676 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:06 smithi150 ceph-mon[29276]: 6.2 deep-scrub starts 2024-09-11T08:32:06.676 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:06 smithi150 ceph-mon[29276]: 6.2 deep-scrub ok 2024-09-11T08:32:06.676 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:06 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3452981018' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-11T08:32:06.676 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:06 smithi150 ceph-mon[39612]: 4.f deep-scrub starts 2024-09-11T08:32:06.676 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:06 smithi150 ceph-mon[39612]: 4.f deep-scrub ok 2024-09-11T08:32:06.676 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:06 smithi150 ceph-mon[39612]: 5.b scrub starts 2024-09-11T08:32:06.676 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:06 smithi150 ceph-mon[39612]: 5.b scrub ok 2024-09-11T08:32:06.676 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:06 smithi150 ceph-mon[39612]: pgmap v1127: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-11T08:32:06.676 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:06 smithi150 ceph-mon[39612]: 6.2 deep-scrub starts 2024-09-11T08:32:06.677 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:06 smithi150 ceph-mon[39612]: 6.2 deep-scrub ok 2024-09-11T08:32:06.677 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:06 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3452981018' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-11T08:32:06.699 INFO:teuthology.orchestra.run.smithi150.stdout:133143986491 2024-09-11T08:32:06.699 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-11T08:32:06.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:06 smithi181 ceph-mon[32701]: 4.f deep-scrub starts 2024-09-11T08:32:06.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:06 smithi181 ceph-mon[32701]: 4.f deep-scrub ok 2024-09-11T08:32:06.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:06 smithi181 ceph-mon[32701]: 5.b scrub starts 2024-09-11T08:32:06.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:06 smithi181 ceph-mon[32701]: 5.b scrub ok 2024-09-11T08:32:06.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:06 smithi181 ceph-mon[32701]: pgmap v1127: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-11T08:32:06.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:06 smithi181 ceph-mon[32701]: 6.2 deep-scrub starts 2024-09-11T08:32:06.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:06 smithi181 ceph-mon[32701]: 6.2 deep-scrub ok 2024-09-11T08:32:06.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:06 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3452981018' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-11T08:32:07.639 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:07 smithi150 ceph-mon[29276]: 4.4 scrub starts 2024-09-11T08:32:07.639 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:07 smithi150 ceph-mon[29276]: 4.4 scrub ok 2024-09-11T08:32:07.640 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:07 smithi150 ceph-mon[29276]: 5.14 scrub starts 2024-09-11T08:32:07.640 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:07 smithi150 ceph-mon[29276]: 5.14 scrub ok 2024-09-11T08:32:07.640 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:07 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:32:07.640 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:07 smithi150 ceph-mon[39612]: 4.4 scrub starts 2024-09-11T08:32:07.640 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:07 smithi150 ceph-mon[39612]: 4.4 scrub ok 2024-09-11T08:32:07.640 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:07 smithi150 ceph-mon[39612]: 5.14 scrub starts 2024-09-11T08:32:07.640 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:07 smithi150 ceph-mon[39612]: 5.14 scrub ok 2024-09-11T08:32:07.640 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:07 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:32:07.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:07 smithi181 ceph-mon[32701]: 4.4 scrub starts 2024-09-11T08:32:07.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:07 smithi181 ceph-mon[32701]: 4.4 scrub ok 2024-09-11T08:32:07.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:07 smithi181 ceph-mon[32701]: 5.14 scrub starts 2024-09-11T08:32:07.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:07 smithi181 ceph-mon[32701]: 5.14 scrub ok 2024-09-11T08:32:07.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:07 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:32:08.738 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:32:08 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:32:08] "GET /metrics HTTP/1.1" 200 222303 "" "Prometheus/2.43.0" 2024-09-11T08:32:08.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:08 smithi150 ceph-mon[39612]: 3.19 scrub starts 2024-09-11T08:32:08.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:08 smithi150 ceph-mon[39612]: 3.19 scrub ok 2024-09-11T08:32:08.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:08 smithi150 ceph-mon[39612]: 3.c deep-scrub starts 2024-09-11T08:32:08.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:08 smithi150 ceph-mon[39612]: 3.c deep-scrub ok 2024-09-11T08:32:08.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:08 smithi150 ceph-mon[39612]: 5.1e scrub starts 2024-09-11T08:32:08.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:08 smithi150 ceph-mon[39612]: 5.1e scrub ok 2024-09-11T08:32:08.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:08 smithi150 ceph-mon[39612]: pgmap v1128: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 614 B/s rd, 0 op/s 2024-09-11T08:32:08.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:08 smithi150 ceph-mon[39612]: 4.1c scrub starts 2024-09-11T08:32:08.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:08 smithi150 ceph-mon[39612]: 4.1c scrub ok 2024-09-11T08:32:08.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:08 smithi150 ceph-mon[39612]: 3.4 scrub starts 2024-09-11T08:32:08.739 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:08 smithi150 ceph-mon[39612]: 3.4 scrub ok 2024-09-11T08:32:08.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:08 smithi150 ceph-mon[29276]: 3.19 scrub starts 2024-09-11T08:32:08.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:08 smithi150 ceph-mon[29276]: 3.19 scrub ok 2024-09-11T08:32:08.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:08 smithi150 ceph-mon[29276]: 3.c deep-scrub starts 2024-09-11T08:32:08.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:08 smithi150 ceph-mon[29276]: 3.c deep-scrub ok 2024-09-11T08:32:08.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:08 smithi150 ceph-mon[29276]: 5.1e scrub starts 2024-09-11T08:32:08.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:08 smithi150 ceph-mon[29276]: 5.1e scrub ok 2024-09-11T08:32:08.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:08 smithi150 ceph-mon[29276]: pgmap v1128: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 614 B/s rd, 0 op/s 2024-09-11T08:32:08.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:08 smithi150 ceph-mon[29276]: 4.1c scrub starts 2024-09-11T08:32:08.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:08 smithi150 ceph-mon[29276]: 4.1c scrub ok 2024-09-11T08:32:08.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:08 smithi150 ceph-mon[29276]: 3.4 scrub starts 2024-09-11T08:32:08.740 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:08 smithi150 ceph-mon[29276]: 3.4 scrub ok 2024-09-11T08:32:08.767 INFO:tasks.cephadm.ceph_manager.ceph:need seq 193273528622 got 193273528625 for osd.6 2024-09-11T08:32:08.767 DEBUG:teuthology.parallel:result is None 2024-09-11T08:32:08.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:08 smithi181 ceph-mon[32701]: 3.19 scrub starts 2024-09-11T08:32:08.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:08 smithi181 ceph-mon[32701]: 3.19 scrub ok 2024-09-11T08:32:08.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:08 smithi181 ceph-mon[32701]: 3.c deep-scrub starts 2024-09-11T08:32:08.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:08 smithi181 ceph-mon[32701]: 3.c deep-scrub ok 2024-09-11T08:32:08.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:08 smithi181 ceph-mon[32701]: 5.1e scrub starts 2024-09-11T08:32:08.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:08 smithi181 ceph-mon[32701]: 5.1e scrub ok 2024-09-11T08:32:08.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:08 smithi181 ceph-mon[32701]: pgmap v1128: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 614 B/s rd, 0 op/s 2024-09-11T08:32:08.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:08 smithi181 ceph-mon[32701]: 4.1c scrub starts 2024-09-11T08:32:08.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:08 smithi181 ceph-mon[32701]: 4.1c scrub ok 2024-09-11T08:32:08.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:08 smithi181 ceph-mon[32701]: 3.4 scrub starts 2024-09-11T08:32:08.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:08 smithi181 ceph-mon[32701]: 3.4 scrub ok 2024-09-11T08:32:09.574 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:32:09 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:32:09.722 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:09 smithi181 ceph-mon[32701]: 6.19 scrub starts 2024-09-11T08:32:09.722 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:09 smithi181 ceph-mon[32701]: 6.19 scrub ok 2024-09-11T08:32:09.722 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:09 smithi181 ceph-mon[32701]: 5.0 scrub starts 2024-09-11T08:32:09.722 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:09 smithi181 ceph-mon[32701]: 5.0 scrub ok 2024-09-11T08:32:09.722 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:09 smithi181 ceph-mon[32701]: 4.1 scrub starts 2024-09-11T08:32:09.722 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:09 smithi181 ceph-mon[32701]: 4.1 scrub ok 2024-09-11T08:32:09.722 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:09 smithi150 ceph-mon[29276]: 6.19 scrub starts 2024-09-11T08:32:09.723 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:09 smithi150 ceph-mon[29276]: 6.19 scrub ok 2024-09-11T08:32:09.723 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:09 smithi150 ceph-mon[29276]: 5.0 scrub starts 2024-09-11T08:32:09.723 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:09 smithi150 ceph-mon[29276]: 5.0 scrub ok 2024-09-11T08:32:09.723 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:09 smithi150 ceph-mon[29276]: 4.1 scrub starts 2024-09-11T08:32:09.723 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:09 smithi150 ceph-mon[29276]: 4.1 scrub ok 2024-09-11T08:32:09.723 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:09 smithi150 ceph-mon[39612]: 6.19 scrub starts 2024-09-11T08:32:09.723 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:09 smithi150 ceph-mon[39612]: 6.19 scrub ok 2024-09-11T08:32:09.723 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:09 smithi150 ceph-mon[39612]: 5.0 scrub starts 2024-09-11T08:32:09.723 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:09 smithi150 ceph-mon[39612]: 5.0 scrub ok 2024-09-11T08:32:09.723 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:09 smithi150 ceph-mon[39612]: 4.1 scrub starts 2024-09-11T08:32:09.724 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:09 smithi150 ceph-mon[39612]: 4.1 scrub ok 2024-09-11T08:32:09.724 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:32:09.724 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:32:09.724 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:32:10.471 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:10 smithi150 ceph-mon[29276]: 6.d deep-scrub starts 2024-09-11T08:32:10.471 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:10 smithi150 ceph-mon[29276]: 6.d deep-scrub ok 2024-09-11T08:32:10.471 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:10 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:32:10.472 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:10 smithi150 ceph-mon[29276]: pgmap v1129: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-11T08:32:10.472 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:10 smithi150 ceph-mon[29276]: 6.14 scrub starts 2024-09-11T08:32:10.472 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:10 smithi150 ceph-mon[29276]: 6.14 scrub ok 2024-09-11T08:32:10.472 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:10 smithi150 ceph-mon[39612]: 6.d deep-scrub starts 2024-09-11T08:32:10.472 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:10 smithi150 ceph-mon[39612]: 6.d deep-scrub ok 2024-09-11T08:32:10.472 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:10 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:32:10.472 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:10 smithi150 ceph-mon[39612]: pgmap v1129: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-11T08:32:10.472 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:10 smithi150 ceph-mon[39612]: 6.14 scrub starts 2024-09-11T08:32:10.472 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:10 smithi150 ceph-mon[39612]: 6.14 scrub ok 2024-09-11T08:32:10.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:10 smithi181 ceph-mon[32701]: 6.d deep-scrub starts 2024-09-11T08:32:10.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:10 smithi181 ceph-mon[32701]: 6.d deep-scrub ok 2024-09-11T08:32:10.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:10 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:32:10.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:10 smithi181 ceph-mon[32701]: pgmap v1129: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-11T08:32:10.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:10 smithi181 ceph-mon[32701]: 6.14 scrub starts 2024-09-11T08:32:10.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:10 smithi181 ceph-mon[32701]: 6.14 scrub ok 2024-09-11T08:32:11.217 INFO:teuthology.orchestra.run.smithi150.stdout:721554505943 2024-09-11T08:32:11.954 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:11 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/1541577127' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-11T08:32:11.955 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:11 smithi150 ceph-mon[29276]: pgmap v1130: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 858 B/s rd, 0 op/s 2024-09-11T08:32:11.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:11 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/1541577127' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-11T08:32:11.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:11 smithi150 ceph-mon[39612]: pgmap v1130: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 858 B/s rd, 0 op/s 2024-09-11T08:32:12.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:11 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/1541577127' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-11T08:32:12.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:11 smithi181 ceph-mon[32701]: pgmap v1130: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 858 B/s rd, 0 op/s 2024-09-11T08:32:12.694 INFO:tasks.cephadm.ceph_manager.ceph:need seq 721554505940 got 721554505943 for osd.7 2024-09-11T08:32:12.694 DEBUG:teuthology.parallel:result is None 2024-09-11T08:32:12.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:12 smithi150 ceph-mon[29276]: 5.1b scrub starts 2024-09-11T08:32:12.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:12 smithi150 ceph-mon[29276]: 5.1b scrub ok 2024-09-11T08:32:12.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:12 smithi150 ceph-mon[29276]: 3.f deep-scrub starts 2024-09-11T08:32:12.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:12 smithi150 ceph-mon[29276]: 3.f deep-scrub ok 2024-09-11T08:32:12.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:12 smithi150 ceph-mon[39612]: 5.1b scrub starts 2024-09-11T08:32:12.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:12 smithi150 ceph-mon[39612]: 5.1b scrub ok 2024-09-11T08:32:12.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:12 smithi150 ceph-mon[39612]: 3.f deep-scrub starts 2024-09-11T08:32:12.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:12 smithi150 ceph-mon[39612]: 3.f deep-scrub ok 2024-09-11T08:32:13.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:12 smithi181 ceph-mon[32701]: 5.1b scrub starts 2024-09-11T08:32:13.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:12 smithi181 ceph-mon[32701]: 5.1b scrub ok 2024-09-11T08:32:13.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:12 smithi181 ceph-mon[32701]: 3.f deep-scrub starts 2024-09-11T08:32:13.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:12 smithi181 ceph-mon[32701]: 3.f deep-scrub ok 2024-09-11T08:32:14.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:13 smithi181 ceph-mon[32701]: 6.1 deep-scrub starts 2024-09-11T08:32:14.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:13 smithi181 ceph-mon[32701]: 6.1 deep-scrub ok 2024-09-11T08:32:14.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:13 smithi181 ceph-mon[32701]: 2.2 deep-scrub starts 2024-09-11T08:32:14.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:13 smithi181 ceph-mon[32701]: 2.2 deep-scrub ok 2024-09-11T08:32:14.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:13 smithi181 ceph-mon[32701]: 6.1c deep-scrub starts 2024-09-11T08:32:14.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:13 smithi181 ceph-mon[32701]: 6.1c deep-scrub ok 2024-09-11T08:32:14.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:13 smithi181 ceph-mon[32701]: pgmap v1131: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:32:14.093 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:13 smithi150 ceph-mon[39612]: 6.1 deep-scrub starts 2024-09-11T08:32:14.093 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:13 smithi150 ceph-mon[39612]: 6.1 deep-scrub ok 2024-09-11T08:32:14.093 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:13 smithi150 ceph-mon[39612]: 2.2 deep-scrub starts 2024-09-11T08:32:14.093 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:13 smithi150 ceph-mon[39612]: 2.2 deep-scrub ok 2024-09-11T08:32:14.093 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:13 smithi150 ceph-mon[39612]: 6.1c deep-scrub starts 2024-09-11T08:32:14.093 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:13 smithi150 ceph-mon[39612]: 6.1c deep-scrub ok 2024-09-11T08:32:14.093 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:13 smithi150 ceph-mon[39612]: pgmap v1131: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:32:14.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:13 smithi150 ceph-mon[29276]: 6.1 deep-scrub starts 2024-09-11T08:32:14.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:13 smithi150 ceph-mon[29276]: 6.1 deep-scrub ok 2024-09-11T08:32:14.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:13 smithi150 ceph-mon[29276]: 2.2 deep-scrub starts 2024-09-11T08:32:14.094 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:13 smithi150 ceph-mon[29276]: 2.2 deep-scrub ok 2024-09-11T08:32:14.094 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:13 smithi150 ceph-mon[29276]: 6.1c deep-scrub starts 2024-09-11T08:32:14.094 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:13 smithi150 ceph-mon[29276]: 6.1c deep-scrub ok 2024-09-11T08:32:14.094 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:13 smithi150 ceph-mon[29276]: pgmap v1131: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:32:14.249 INFO:teuthology.orchestra.run.smithi150.stdout:34359738702 2024-09-11T08:32:14.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:14 smithi150 ceph-mon[29276]: 3.13 scrub starts 2024-09-11T08:32:14.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:14 smithi150 ceph-mon[29276]: 3.13 scrub ok 2024-09-11T08:32:14.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:14 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2063775234' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-11T08:32:14.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:14 smithi150 ceph-mon[39612]: 3.13 scrub starts 2024-09-11T08:32:14.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:14 smithi150 ceph-mon[39612]: 3.13 scrub ok 2024-09-11T08:32:14.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:14 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2063775234' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-11T08:32:15.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:14 smithi181 ceph-mon[32701]: 3.13 scrub starts 2024-09-11T08:32:15.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:14 smithi181 ceph-mon[32701]: 3.13 scrub ok 2024-09-11T08:32:15.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:14 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2063775234' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-11T08:32:15.548 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:32:15.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:32:15 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:32:15] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:32:15.619 INFO:teuthology.orchestra.run.smithi150.stdout:55834575178 2024-09-11T08:32:16.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:15 smithi150 ceph-mon[29276]: 5.d scrub starts 2024-09-11T08:32:16.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:15 smithi150 ceph-mon[29276]: 5.d scrub ok 2024-09-11T08:32:16.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:15 smithi150 ceph-mon[29276]: 5.c scrub starts 2024-09-11T08:32:16.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:15 smithi150 ceph-mon[29276]: 5.c scrub ok 2024-09-11T08:32:16.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:15 smithi150 ceph-mon[29276]: 5.10 deep-scrub starts 2024-09-11T08:32:16.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:15 smithi150 ceph-mon[29276]: 5.10 deep-scrub ok 2024-09-11T08:32:16.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:15 smithi150 ceph-mon[29276]: pgmap v1132: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:32:16.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:15 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2690172001' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-11T08:32:16.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:15 smithi150 ceph-mon[39612]: 5.d scrub starts 2024-09-11T08:32:16.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:15 smithi150 ceph-mon[39612]: 5.d scrub ok 2024-09-11T08:32:16.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:15 smithi150 ceph-mon[39612]: 5.c scrub starts 2024-09-11T08:32:16.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:15 smithi150 ceph-mon[39612]: 5.c scrub ok 2024-09-11T08:32:16.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:15 smithi150 ceph-mon[39612]: 5.10 deep-scrub starts 2024-09-11T08:32:16.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:15 smithi150 ceph-mon[39612]: 5.10 deep-scrub ok 2024-09-11T08:32:16.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:15 smithi150 ceph-mon[39612]: pgmap v1132: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:32:16.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:15 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2690172001' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-11T08:32:16.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:15 smithi181 ceph-mon[32701]: 5.d scrub starts 2024-09-11T08:32:16.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:15 smithi181 ceph-mon[32701]: 5.d scrub ok 2024-09-11T08:32:16.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:15 smithi181 ceph-mon[32701]: 5.c scrub starts 2024-09-11T08:32:16.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:15 smithi181 ceph-mon[32701]: 5.c scrub ok 2024-09-11T08:32:16.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:15 smithi181 ceph-mon[32701]: 5.10 deep-scrub starts 2024-09-11T08:32:16.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:15 smithi181 ceph-mon[32701]: 5.10 deep-scrub ok 2024-09-11T08:32:16.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:15 smithi181 ceph-mon[32701]: pgmap v1132: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:32:16.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:15 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2690172001' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-11T08:32:16.452 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738699 got 34359738702 for osd.0 2024-09-11T08:32:16.452 DEBUG:teuthology.parallel:result is None 2024-09-11T08:32:16.795 INFO:teuthology.orchestra.run.smithi150.stdout:107374182722 2024-09-11T08:32:16.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:16 smithi150 ceph-mon[29276]: 5.12 scrub starts 2024-09-11T08:32:16.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:16 smithi150 ceph-mon[29276]: 5.12 scrub ok 2024-09-11T08:32:16.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:16 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2595435698' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-11T08:32:16.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:16 smithi150 ceph-mon[39612]: 5.12 scrub starts 2024-09-11T08:32:16.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:16 smithi150 ceph-mon[39612]: 5.12 scrub ok 2024-09-11T08:32:16.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:16 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2595435698' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-11T08:32:17.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:16 smithi181 ceph-mon[32701]: 5.12 scrub starts 2024-09-11T08:32:17.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:16 smithi181 ceph-mon[32701]: 5.12 scrub ok 2024-09-11T08:32:17.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:16 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2595435698' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-11T08:32:17.865 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834575175 got 55834575178 for osd.1 2024-09-11T08:32:17.866 DEBUG:teuthology.parallel:result is None 2024-09-11T08:32:18.224 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:17 smithi181 ceph-mon[32701]: 5.5 scrub starts 2024-09-11T08:32:18.224 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:17 smithi181 ceph-mon[32701]: 5.5 scrub ok 2024-09-11T08:32:18.224 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:17 smithi181 ceph-mon[32701]: 4.10 scrub starts 2024-09-11T08:32:18.225 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:17 smithi181 ceph-mon[32701]: 4.10 scrub ok 2024-09-11T08:32:18.225 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:17 smithi181 ceph-mon[32701]: 6.3 scrub starts 2024-09-11T08:32:18.225 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:17 smithi181 ceph-mon[32701]: 6.3 scrub ok 2024-09-11T08:32:18.225 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:17 smithi181 ceph-mon[32701]: pgmap v1133: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:32:18.225 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:17 smithi181 ceph-mon[32701]: 3.1a scrub starts 2024-09-11T08:32:18.225 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:17 smithi181 ceph-mon[32701]: 3.1a scrub ok 2024-09-11T08:32:18.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:17 smithi150 ceph-mon[29276]: 5.5 scrub starts 2024-09-11T08:32:18.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:17 smithi150 ceph-mon[29276]: 5.5 scrub ok 2024-09-11T08:32:18.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:17 smithi150 ceph-mon[29276]: 4.10 scrub starts 2024-09-11T08:32:18.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:17 smithi150 ceph-mon[29276]: 4.10 scrub ok 2024-09-11T08:32:18.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:17 smithi150 ceph-mon[29276]: 6.3 scrub starts 2024-09-11T08:32:18.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:17 smithi150 ceph-mon[29276]: 6.3 scrub ok 2024-09-11T08:32:18.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:17 smithi150 ceph-mon[29276]: pgmap v1133: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:32:18.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:17 smithi150 ceph-mon[29276]: 3.1a scrub starts 2024-09-11T08:32:18.239 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:17 smithi150 ceph-mon[29276]: 3.1a scrub ok 2024-09-11T08:32:18.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:17 smithi150 ceph-mon[39612]: 5.5 scrub starts 2024-09-11T08:32:18.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:17 smithi150 ceph-mon[39612]: 5.5 scrub ok 2024-09-11T08:32:18.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:17 smithi150 ceph-mon[39612]: 4.10 scrub starts 2024-09-11T08:32:18.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:17 smithi150 ceph-mon[39612]: 4.10 scrub ok 2024-09-11T08:32:18.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:17 smithi150 ceph-mon[39612]: 6.3 scrub starts 2024-09-11T08:32:18.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:17 smithi150 ceph-mon[39612]: 6.3 scrub ok 2024-09-11T08:32:18.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:17 smithi150 ceph-mon[39612]: pgmap v1133: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:32:18.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:17 smithi150 ceph-mon[39612]: 3.1a scrub starts 2024-09-11T08:32:18.240 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:17 smithi150 ceph-mon[39612]: 3.1a scrub ok 2024-09-11T08:32:18.522 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182718 got 107374182722 for osd.3 2024-09-11T08:32:18.523 DEBUG:teuthology.parallel:result is None 2024-09-11T08:32:18.737 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:32:18 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:32:18] "GET /metrics HTTP/1.1" 200 222303 "" "Prometheus/2.43.0" 2024-09-11T08:32:18.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:18 smithi150 ceph-mon[39612]: 4.12 scrub starts 2024-09-11T08:32:18.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:18 smithi150 ceph-mon[39612]: 4.12 scrub ok 2024-09-11T08:32:18.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:18 smithi150 ceph-mon[29276]: 4.12 scrub starts 2024-09-11T08:32:18.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:18 smithi150 ceph-mon[29276]: 4.12 scrub ok 2024-09-11T08:32:19.134 INFO:teuthology.orchestra.run.smithi150.stdout:1516123455584 2024-09-11T08:32:19.208 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:18 smithi181 ceph-mon[32701]: 4.12 scrub starts 2024-09-11T08:32:19.208 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:18 smithi181 ceph-mon[32701]: 4.12 scrub ok 2024-09-11T08:32:19.573 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:32:19 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:32:19.958 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:19 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/921477262' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-11T08:32:19.959 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:19 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:32:19.959 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:19 smithi150 ceph-mon[29276]: pgmap v1134: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:32:19.959 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:19 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/921477262' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-11T08:32:19.959 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:19 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:32:19.959 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:19 smithi150 ceph-mon[39612]: pgmap v1134: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:32:19.985 INFO:tasks.cephadm.ceph_manager.ceph:need seq 1516123455580 got 1516123455584 for osd.5 2024-09-11T08:32:19.985 DEBUG:teuthology.parallel:result is None 2024-09-11T08:32:20.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:19 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/921477262' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-11T08:32:20.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:19 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:32:20.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:19 smithi181 ceph-mon[32701]: pgmap v1134: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:32:20.630 INFO:teuthology.orchestra.run.smithi150.stdout:133143986494 2024-09-11T08:32:21.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:20 smithi150 ceph-mon[29276]: 3.1b scrub starts 2024-09-11T08:32:21.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:20 smithi150 ceph-mon[29276]: 3.1b scrub ok 2024-09-11T08:32:21.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:20 smithi150 ceph-mon[29276]: 3.16 scrub starts 2024-09-11T08:32:21.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:20 smithi150 ceph-mon[29276]: 3.16 scrub ok 2024-09-11T08:32:21.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:20 smithi150 ceph-mon[29276]: 5.2 scrub starts 2024-09-11T08:32:21.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:20 smithi150 ceph-mon[29276]: 5.2 scrub ok 2024-09-11T08:32:21.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:20 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/2954769980' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-11T08:32:21.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:20 smithi150 ceph-mon[39612]: 3.1b scrub starts 2024-09-11T08:32:21.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:20 smithi150 ceph-mon[39612]: 3.1b scrub ok 2024-09-11T08:32:21.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:20 smithi150 ceph-mon[39612]: 3.16 scrub starts 2024-09-11T08:32:21.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:20 smithi150 ceph-mon[39612]: 3.16 scrub ok 2024-09-11T08:32:21.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:20 smithi150 ceph-mon[39612]: 5.2 scrub starts 2024-09-11T08:32:21.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:20 smithi150 ceph-mon[39612]: 5.2 scrub ok 2024-09-11T08:32:21.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:20 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/2954769980' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-11T08:32:21.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:20 smithi181 ceph-mon[32701]: 3.1b scrub starts 2024-09-11T08:32:21.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:20 smithi181 ceph-mon[32701]: 3.1b scrub ok 2024-09-11T08:32:21.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:20 smithi181 ceph-mon[32701]: 3.16 scrub starts 2024-09-11T08:32:21.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:20 smithi181 ceph-mon[32701]: 3.16 scrub ok 2024-09-11T08:32:21.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:20 smithi181 ceph-mon[32701]: 5.2 scrub starts 2024-09-11T08:32:21.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:20 smithi181 ceph-mon[32701]: 5.2 scrub ok 2024-09-11T08:32:21.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:20 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/2954769980' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-11T08:32:21.452 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986491 got 133143986494 for osd.4 2024-09-11T08:32:21.452 DEBUG:teuthology.parallel:result is None 2024-09-11T08:32:21.452 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-11T08:32:21.453 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T08:32:21.701 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:32:22.079 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:21 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:32:22.079 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:21 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:32:22.079 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:21 smithi150 ceph-mon[29276]: pgmap v1135: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:32:22.079 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:21 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:32:22.080 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:21 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:32:22.080 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:21 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:32:22.080 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:21 smithi150 ceph-mon[39612]: pgmap v1135: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:32:22.080 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:21 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:32:22.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:21 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:32:22.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:21 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:32:22.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:21 smithi181 ceph-mon[32701]: pgmap v1135: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:32:22.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:21 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:32:23.094 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:22 smithi150 ceph-mon[29276]: osdmap e479: 8 total, 8 up, 8 in 2024-09-11T08:32:23.095 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:22 smithi150 ceph-mon[29276]: 3.14 scrub starts 2024-09-11T08:32:23.095 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:22 smithi150 ceph-mon[29276]: 3.14 scrub ok 2024-09-11T08:32:23.095 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:22 smithi150 ceph-mon[39612]: osdmap e479: 8 total, 8 up, 8 in 2024-09-11T08:32:23.095 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:22 smithi150 ceph-mon[39612]: 3.14 scrub starts 2024-09-11T08:32:23.095 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:22 smithi150 ceph-mon[39612]: 3.14 scrub ok 2024-09-11T08:32:23.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:22 smithi181 ceph-mon[32701]: osdmap e479: 8 total, 8 up, 8 in 2024-09-11T08:32:23.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:22 smithi181 ceph-mon[32701]: 3.14 scrub starts 2024-09-11T08:32:23.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:22 smithi181 ceph-mon[32701]: 3.14 scrub ok 2024-09-11T08:32:23.515 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:32:23.518 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-09-11T08:32:24.191 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:23 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:32:24.192 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:23 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:32:24.192 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:23 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:32:24.192 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:23 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:32:24.192 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:23 smithi150 ceph-mon[29276]: 3.e scrub starts 2024-09-11T08:32:24.192 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:23 smithi150 ceph-mon[29276]: 3.e scrub ok 2024-09-11T08:32:24.192 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:23 smithi150 ceph-mon[29276]: pgmap v1137: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-11T08:32:24.192 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:23 smithi150 ceph-mon[29276]: from='client.26285 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:32:24.192 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:23 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:32:24.193 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:23 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:32:24.193 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:23 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:32:24.193 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:23 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:32:24.193 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:23 smithi150 ceph-mon[39612]: 3.e scrub starts 2024-09-11T08:32:24.193 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:23 smithi150 ceph-mon[39612]: 3.e scrub ok 2024-09-11T08:32:24.193 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:23 smithi150 ceph-mon[39612]: pgmap v1137: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-11T08:32:24.193 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:23 smithi150 ceph-mon[39612]: from='client.26285 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:32:24.215 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":1137,"stamp":"2024-09-11T08:32:23.272435+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596188,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":2741,"num_read_kb":2600,"num_write":1115,"num_write_kb":1671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":1776519,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1297,"ondisk_log_size":1297,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6664324,"kb_used_data":8252,"kb_used_omap":22,"kb_used_meta":6655977,"kb_avail":743296892,"statfs":{"total":767960285184,"available":761136017408,"internally_reserved":0,"allocated":8450048,"data_stored":5108450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22853,"internal_metadata":6815721147},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002521"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":564,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:38.270778+0000","last_change":"2024-09-11T08:31:36.361555+0000","last_active":"2024-09-11T08:31:38.270778+0000","last_peered":"2024-09-11T08:31:38.270778+0000","last_clean":"2024-09-11T08:31:38.270778+0000","last_became_active":"2024-09-11T08:24:28.358401+0000","last_became_peered":"2024-09-11T08:24:28.358401+0000","last_unstale":"2024-09-11T08:31:38.270778+0000","last_undegraded":"2024-09-11T08:31:38.270778+0000","last_fullsized":"2024-09-11T08:31:38.270778+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:36.361487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:31:36.361487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:52.481796+0000","scrub_duration":0.0020598159999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"405'25","reported_seq":578,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.487694+0000","last_change":"2024-09-11T08:31:12.599946+0000","last_active":"2024-09-11T08:31:59.487694+0000","last_peered":"2024-09-11T08:31:59.487694+0000","last_clean":"2024-09-11T08:31:59.487694+0000","last_became_active":"2024-09-11T08:07:32.048048+0000","last_became_peered":"2024-09-11T08:07:32.048048+0000","last_unstale":"2024-09-11T08:31:59.487694+0000","last_undegraded":"2024-09-11T08:31:59.487694+0000","last_fullsized":"2024-09-11T08:31:59.487694+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'25","last_scrub_stamp":"2024-09-11T08:31:12.599879+0000","last_deep_scrub":"312'19","last_deep_scrub_stamp":"2024-09-11T08:21:59.122122+0000","last_clean_scrub_stamp":"2024-09-11T08:31:12.599879+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:23.475718+0000","scrub_duration":0.0035721339999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":610,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:38.271294+0000","last_change":"2024-09-11T08:31:37.671883+0000","last_active":"2024-09-11T08:31:38.271294+0000","last_peered":"2024-09-11T08:31:38.271294+0000","last_clean":"2024-09-11T08:31:38.271294+0000","last_became_active":"2024-09-11T08:24:28.372160+0000","last_became_peered":"2024-09-11T08:24:28.372160+0000","last_unstale":"2024-09-11T08:31:38.271294+0000","last_undegraded":"2024-09-11T08:31:38.271294+0000","last_fullsized":"2024-09-11T08:31:38.271294+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:37.671827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:59.531202+0000","last_clean_scrub_stamp":"2024-09-11T08:31:37.671827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:07.023040+0000","scrub_duration":0.0018397439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":396,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:04.497516+0000","last_change":"2024-09-11T08:32:04.497472+0000","last_active":"2024-09-11T08:32:04.497516+0000","last_peered":"2024-09-11T08:32:04.497516+0000","last_clean":"2024-09-11T08:32:04.497516+0000","last_became_active":"2024-09-11T08:31:16.964358+0000","last_became_peered":"2024-09-11T08:31:16.964358+0000","last_unstale":"2024-09-11T08:32:04.497516+0000","last_undegraded":"2024-09-11T08:32:04.497516+0000","last_fullsized":"2024-09-11T08:32:04.497516+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:04.497396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:02.470649+0000","last_clean_scrub_stamp":"2024-09-11T08:32:04.497396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:33.896616+0000","scrub_duration":0.0015759000000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":549,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:38.263464+0000","last_change":"2024-09-11T08:31:00.004778+0000","last_active":"2024-09-11T08:31:38.263464+0000","last_peered":"2024-09-11T08:31:38.263464+0000","last_clean":"2024-09-11T08:31:38.263464+0000","last_became_active":"2024-09-11T08:15:56.521104+0000","last_became_peered":"2024-09-11T08:15:56.521104+0000","last_unstale":"2024-09-11T08:31:38.263464+0000","last_undegraded":"2024-09-11T08:31:38.263464+0000","last_fullsized":"2024-09-11T08:31:38.263464+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:00.004724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:31:00.004724+0000","last_clean_scrub_stamp":"2024-09-11T08:31:00.004724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:28.957317+0000","scrub_duration":0.0035427290000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":252,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:38.264209+0000","last_change":"2024-09-11T08:31:23.119557+0000","last_active":"2024-09-11T08:31:38.264209+0000","last_peered":"2024-09-11T08:31:38.264209+0000","last_clean":"2024-09-11T08:31:38.264209+0000","last_became_active":"2024-09-11T08:31:09.935019+0000","last_became_peered":"2024-09-11T08:31:09.935019+0000","last_unstale":"2024-09-11T08:31:38.264209+0000","last_undegraded":"2024-09-11T08:31:38.264209+0000","last_fullsized":"2024-09-11T08:31:38.264209+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:23.119475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:31:23.119475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:25.064663+0000","scrub_duration":0.0035696360000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"405'46","reported_seq":451,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:03.000115+0000","last_change":"2024-09-11T08:32:03.000087+0000","last_active":"2024-09-11T08:32:03.000115+0000","last_peered":"2024-09-11T08:32:03.000115+0000","last_clean":"2024-09-11T08:32:03.000115+0000","last_became_active":"2024-09-11T08:31:09.932624+0000","last_became_peered":"2024-09-11T08:31:09.932624+0000","last_unstale":"2024-09-11T08:32:03.000115+0000","last_undegraded":"2024-09-11T08:32:03.000115+0000","last_fullsized":"2024-09-11T08:32:03.000115+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'46","last_scrub_stamp":"2024-09-11T08:32:03.000027+0000","last_deep_scrub":"145'22","last_deep_scrub_stamp":"2024-09-11T08:15:57.478870+0000","last_clean_scrub_stamp":"2024-09-11T08:32:03.000027+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:05.275638+0000","scrub_duration":0.0018405229999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":365,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:00.451296+0000","last_change":"2024-09-11T08:32:00.451257+0000","last_active":"2024-09-11T08:32:00.451296+0000","last_peered":"2024-09-11T08:32:00.451296+0000","last_clean":"2024-09-11T08:32:00.451296+0000","last_became_active":"2024-09-11T08:31:16.962706+0000","last_became_peered":"2024-09-11T08:31:16.962706+0000","last_unstale":"2024-09-11T08:32:00.451296+0000","last_undegraded":"2024-09-11T08:32:00.451296+0000","last_fullsized":"2024-09-11T08:32:00.451296+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:00.451200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:18:36.998503+0000","last_clean_scrub_stamp":"2024-09-11T08:32:00.451200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:23.397534+0000","scrub_duration":0.0013099940000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":110,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:49.882347+0000","last_change":"2024-09-11T08:31:49.882290+0000","last_active":"2024-09-11T08:31:49.882347+0000","last_peered":"2024-09-11T08:31:49.882347+0000","last_clean":"2024-09-11T08:31:49.882347+0000","last_became_active":"2024-09-11T08:31:38.278622+0000","last_became_peered":"2024-09-11T08:31:38.278622+0000","last_unstale":"2024-09-11T08:31:49.882347+0000","last_undegraded":"2024-09-11T08:31:49.882347+0000","last_fullsized":"2024-09-11T08:31:49.882347+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:49.882227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:54.878409+0000","last_clean_scrub_stamp":"2024-09-11T08:31:49.882227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:50.427634+0000","scrub_duration":0.0012820970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"468'25","reported_seq":385,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.488181+0000","last_change":"2024-09-11T08:31:18.549005+0000","last_active":"2024-09-11T08:31:59.488181+0000","last_peered":"2024-09-11T08:31:59.488181+0000","last_clean":"2024-09-11T08:31:59.488181+0000","last_became_active":"2024-09-11T08:31:16.962868+0000","last_became_peered":"2024-09-11T08:31:16.962868+0000","last_unstale":"2024-09-11T08:31:59.488181+0000","last_undegraded":"2024-09-11T08:31:59.488181+0000","last_fullsized":"2024-09-11T08:31:59.488181+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"468'25","last_scrub_stamp":"2024-09-11T08:31:18.548955+0000","last_deep_scrub":"468'25","last_deep_scrub_stamp":"2024-09-11T08:31:18.548955+0000","last_clean_scrub_stamp":"2024-09-11T08:31:18.548955+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:46.783158+0000","scrub_duration":0.0031821060000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":300,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:11.855812+0000","last_change":"2024-09-11T08:32:11.855778+0000","last_active":"2024-09-11T08:32:11.855812+0000","last_peered":"2024-09-11T08:32:11.855812+0000","last_clean":"2024-09-11T08:32:11.855812+0000","last_became_active":"2024-09-11T08:31:38.274688+0000","last_became_peered":"2024-09-11T08:31:38.274688+0000","last_unstale":"2024-09-11T08:32:11.855812+0000","last_undegraded":"2024-09-11T08:32:11.855812+0000","last_fullsized":"2024-09-11T08:32:11.855812+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:11.855710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:20:18.900482+0000","last_clean_scrub_stamp":"2024-09-11T08:32:11.855710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:17.685158+0000","scrub_duration":0.0016534309999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":528,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:38.270886+0000","last_change":"2024-09-11T08:30:53.446937+0000","last_active":"2024-09-11T08:31:38.270886+0000","last_peered":"2024-09-11T08:31:38.270886+0000","last_clean":"2024-09-11T08:31:38.270886+0000","last_became_active":"2024-09-11T08:14:24.792595+0000","last_became_peered":"2024-09-11T08:14:24.792595+0000","last_unstale":"2024-09-11T08:31:38.270886+0000","last_undegraded":"2024-09-11T08:31:38.270886+0000","last_fullsized":"2024-09-11T08:31:38.270886+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:53.446882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:30:53.446882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:13.740710+0000","scrub_duration":0.0016646390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":100,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:43.002807+0000","last_change":"2024-09-11T08:31:43.002757+0000","last_active":"2024-09-11T08:31:43.002807+0000","last_peered":"2024-09-11T08:31:43.002807+0000","last_clean":"2024-09-11T08:31:43.002807+0000","last_became_active":"2024-09-11T08:31:38.274258+0000","last_became_peered":"2024-09-11T08:31:38.274258+0000","last_unstale":"2024-09-11T08:31:43.002807+0000","last_undegraded":"2024-09-11T08:31:43.002807+0000","last_fullsized":"2024-09-11T08:31:43.002807+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:43.002690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:31:43.002690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:46.571260+0000","scrub_duration":0.001676507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"405'11","reported_seq":365,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.488847+0000","last_change":"2024-09-11T08:31:59.488753+0000","last_active":"2024-09-11T08:31:59.488847+0000","last_peered":"2024-09-11T08:31:59.488847+0000","last_clean":"2024-09-11T08:31:59.488847+0000","last_became_active":"2024-09-11T08:31:16.962705+0000","last_became_peered":"2024-09-11T08:31:16.962705+0000","last_unstale":"2024-09-11T08:31:59.488847+0000","last_undegraded":"2024-09-11T08:31:59.488847+0000","last_fullsized":"2024-09-11T08:31:59.488847+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'11","last_scrub_stamp":"2024-09-11T08:31:59.488651+0000","last_deep_scrub":"139'7","last_deep_scrub_stamp":"2024-09-11T08:12:24.869318+0000","last_clean_scrub_stamp":"2024-09-11T08:31:59.488651+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:08.361074+0000","scrub_duration":0.0025657850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":351,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.340734+0000","last_change":"2024-09-11T08:31:28.667899+0000","last_active":"2024-09-11T08:31:59.340734+0000","last_peered":"2024-09-11T08:31:59.340734+0000","last_clean":"2024-09-11T08:31:59.340734+0000","last_became_active":"2024-09-11T08:15:56.526954+0000","last_became_peered":"2024-09-11T08:15:56.526954+0000","last_unstale":"2024-09-11T08:31:59.340734+0000","last_undegraded":"2024-09-11T08:31:59.340734+0000","last_fullsized":"2024-09-11T08:31:59.340734+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:28.667828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:31:28.667828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:49.011994+0000","scrub_duration":0.001462996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":110,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:07.876453+0000","last_change":"2024-09-11T08:32:07.876414+0000","last_active":"2024-09-11T08:32:07.876453+0000","last_peered":"2024-09-11T08:32:07.876453+0000","last_clean":"2024-09-11T08:32:07.876453+0000","last_became_active":"2024-09-11T08:31:38.275166+0000","last_became_peered":"2024-09-11T08:31:38.275166+0000","last_unstale":"2024-09-11T08:32:07.876453+0000","last_undegraded":"2024-09-11T08:32:07.876453+0000","last_fullsized":"2024-09-11T08:32:07.876453+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:07.876335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:20:19.927616+0000","last_clean_scrub_stamp":"2024-09-11T08:32:07.876335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:10.106603+0000","scrub_duration":0.0017588199999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":386,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.488456+0000","last_change":"2024-09-11T08:31:35.505163+0000","last_active":"2024-09-11T08:31:59.488456+0000","last_peered":"2024-09-11T08:31:59.488456+0000","last_clean":"2024-09-11T08:31:59.488456+0000","last_became_active":"2024-09-11T08:31:16.964004+0000","last_became_peered":"2024-09-11T08:31:16.964004+0000","last_unstale":"2024-09-11T08:31:59.488456+0000","last_undegraded":"2024-09-11T08:31:59.488456+0000","last_fullsized":"2024-09-11T08:31:59.488456+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:35.505100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:31:35.505100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:05.402855+0000","scrub_duration":0.0015443919999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":557,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:19.169067+0000","last_change":"2024-09-11T08:32:19.169024+0000","last_active":"2024-09-11T08:32:19.169067+0000","last_peered":"2024-09-11T08:32:19.169067+0000","last_clean":"2024-09-11T08:32:19.169067+0000","last_became_active":"2024-09-11T08:15:56.516941+0000","last_became_peered":"2024-09-11T08:15:56.516941+0000","last_unstale":"2024-09-11T08:32:19.169067+0000","last_undegraded":"2024-09-11T08:32:19.169067+0000","last_fullsized":"2024-09-11T08:32:19.169067+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:19.168965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:32:19.168965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:22.316530+0000","scrub_duration":0.0014021610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"405'46","reported_seq":623,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:07.274109+0000","last_change":"2024-09-11T08:32:07.274072+0000","last_active":"2024-09-11T08:32:07.274109+0000","last_peered":"2024-09-11T08:32:07.274109+0000","last_clean":"2024-09-11T08:32:07.274109+0000","last_became_active":"2024-09-11T08:07:32.046363+0000","last_became_peered":"2024-09-11T08:07:32.046363+0000","last_unstale":"2024-09-11T08:32:07.274109+0000","last_undegraded":"2024-09-11T08:32:07.274109+0000","last_fullsized":"2024-09-11T08:32:07.274109+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'46","last_scrub_stamp":"2024-09-11T08:32:07.274016+0000","last_deep_scrub":"107'22","last_deep_scrub_stamp":"2024-09-11T08:16:04.159211+0000","last_clean_scrub_stamp":"2024-09-11T08:32:07.274016+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:26.254312+0000","scrub_duration":0.001758459,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":57,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:01.223926+0000","last_change":"2024-09-11T08:32:01.223891+0000","last_active":"2024-09-11T08:32:01.223926+0000","last_peered":"2024-09-11T08:32:01.223926+0000","last_clean":"2024-09-11T08:32:01.223926+0000","last_became_active":"2024-09-11T08:30:37.522527+0000","last_became_peered":"2024-09-11T08:30:37.522527+0000","last_unstale":"2024-09-11T08:32:01.223926+0000","last_undegraded":"2024-09-11T08:32:01.223926+0000","last_fullsized":"2024-09-11T08:32:01.223926+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:01.223798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:32:01.223798+0000","last_clean_scrub_stamp":"2024-09-11T08:32:01.223798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:10.427791+0000","scrub_duration":0.0016186849999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":249,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:38.264052+0000","last_change":"2024-09-11T08:31:26.107834+0000","last_active":"2024-09-11T08:31:38.264052+0000","last_peered":"2024-09-11T08:31:38.264052+0000","last_clean":"2024-09-11T08:31:38.264052+0000","last_became_active":"2024-09-11T08:31:09.936646+0000","last_became_peered":"2024-09-11T08:31:09.936646+0000","last_unstale":"2024-09-11T08:31:38.264052+0000","last_undegraded":"2024-09-11T08:31:38.264052+0000","last_fullsized":"2024-09-11T08:31:38.264052+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:26.107772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:31:26.107772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:51.630945+0000","scrub_duration":0.0013940230000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":379,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:17.329915+0000","last_change":"2024-09-11T08:32:17.329872+0000","last_active":"2024-09-11T08:32:17.329915+0000","last_peered":"2024-09-11T08:32:17.329915+0000","last_clean":"2024-09-11T08:32:17.329915+0000","last_became_active":"2024-09-11T08:31:16.963363+0000","last_became_peered":"2024-09-11T08:31:16.963363+0000","last_unstale":"2024-09-11T08:32:17.329915+0000","last_undegraded":"2024-09-11T08:32:17.329915+0000","last_fullsized":"2024-09-11T08:32:17.329915+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:17.329804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:32:17.329804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:43.300883+0000","scrub_duration":0.001829656,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"405'24","reported_seq":332,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:38.263869+0000","last_change":"2024-09-11T08:31:09.939854+0000","last_active":"2024-09-11T08:31:38.263869+0000","last_peered":"2024-09-11T08:31:38.263869+0000","last_clean":"2024-09-11T08:31:38.263869+0000","last_became_active":"2024-09-11T08:31:09.939499+0000","last_became_peered":"2024-09-11T08:31:09.939499+0000","last_unstale":"2024-09-11T08:31:38.263869+0000","last_undegraded":"2024-09-11T08:31:38.263869+0000","last_fullsized":"2024-09-11T08:31:38.263869+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'24","last_scrub_stamp":"2024-09-11T08:30:52.542477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:30:52.542477+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:22.534296+0000","scrub_duration":0.0051635600000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":375,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.488073+0000","last_change":"2024-09-11T08:31:38.456553+0000","last_active":"2024-09-11T08:31:59.488073+0000","last_peered":"2024-09-11T08:31:59.488073+0000","last_clean":"2024-09-11T08:31:59.488073+0000","last_became_active":"2024-09-11T08:31:16.964385+0000","last_became_peered":"2024-09-11T08:31:16.964385+0000","last_unstale":"2024-09-11T08:31:59.488073+0000","last_undegraded":"2024-09-11T08:31:59.488073+0000","last_fullsized":"2024-09-11T08:31:59.488073+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:38.456489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:10:40.110490+0000","last_clean_scrub_stamp":"2024-09-11T08:31:38.456489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:48.645086+0000","scrub_duration":0.0015098780000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"478'2","reported_seq":527,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:12.893410+0000","last_change":"2024-09-11T08:32:12.893361+0000","last_active":"2024-09-11T08:32:12.893410+0000","last_peered":"2024-09-11T08:32:12.893410+0000","last_clean":"2024-09-11T08:32:12.893410+0000","last_became_active":"2024-09-11T08:24:28.357198+0000","last_became_peered":"2024-09-11T08:24:28.357198+0000","last_unstale":"2024-09-11T08:32:12.893410+0000","last_undegraded":"2024-09-11T08:32:12.893410+0000","last_fullsized":"2024-09-11T08:32:12.893410+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'2","last_scrub_stamp":"2024-09-11T08:32:12.893299+0000","last_deep_scrub":"478'2","last_deep_scrub_stamp":"2024-09-11T08:32:12.893299+0000","last_clean_scrub_stamp":"2024-09-11T08:32:12.893299+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:25.697133+0000","scrub_duration":0.0029555279999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":547,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:06.620074+0000","last_change":"2024-09-11T08:32:06.620049+0000","last_active":"2024-09-11T08:32:06.620074+0000","last_peered":"2024-09-11T08:32:06.620074+0000","last_clean":"2024-09-11T08:32:06.620074+0000","last_became_active":"2024-09-11T08:07:30.045932+0000","last_became_peered":"2024-09-11T08:07:30.045932+0000","last_unstale":"2024-09-11T08:32:06.620074+0000","last_undegraded":"2024-09-11T08:32:06.620074+0000","last_fullsized":"2024-09-11T08:32:06.620074+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:06.619993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:32:06.619993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:29.478781+0000","scrub_duration":0.001538059,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"405'35","reported_seq":638,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:38.270509+0000","last_change":"2024-09-11T08:31:02.336843+0000","last_active":"2024-09-11T08:31:38.270509+0000","last_peered":"2024-09-11T08:31:38.270509+0000","last_clean":"2024-09-11T08:31:38.270509+0000","last_became_active":"2024-09-11T08:15:56.515817+0000","last_became_peered":"2024-09-11T08:15:56.515817+0000","last_unstale":"2024-09-11T08:31:38.270509+0000","last_undegraded":"2024-09-11T08:31:38.270509+0000","last_fullsized":"2024-09-11T08:31:38.270509+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'35","last_scrub_stamp":"2024-09-11T08:31:02.336770+0000","last_deep_scrub":"350'25","last_deep_scrub_stamp":"2024-09-11T08:24:23.093269+0000","last_clean_scrub_stamp":"2024-09-11T08:31:02.336770+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:11.942901+0000","scrub_duration":0.0020150519999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"254'12","reported_seq":898,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:08.867550+0000","last_change":"2024-09-11T08:30:47.391441+0000","last_active":"2024-09-11T08:32:08.867550+0000","last_peered":"2024-09-11T08:32:08.867550+0000","last_clean":"2024-09-11T08:32:08.867550+0000","last_became_active":"2024-09-11T08:15:56.525498+0000","last_became_peered":"2024-09-11T08:15:56.525498+0000","last_unstale":"2024-09-11T08:32:08.867550+0000","last_undegraded":"2024-09-11T08:32:08.867550+0000","last_fullsized":"2024-09-11T08:32:08.867550+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'12","last_scrub_stamp":"2024-09-11T08:30:47.391368+0000","last_deep_scrub":"254'12","last_deep_scrub_stamp":"2024-09-11T08:18:34.253865+0000","last_clean_scrub_stamp":"2024-09-11T08:30:47.391368+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00171394,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"478'46","reported_seq":623,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:04.621289+0000","last_change":"2024-09-11T08:32:04.621261+0000","last_active":"2024-09-11T08:32:04.621289+0000","last_peered":"2024-09-11T08:32:04.621289+0000","last_clean":"2024-09-11T08:32:04.621289+0000","last_became_active":"2024-09-11T08:07:32.054317+0000","last_became_peered":"2024-09-11T08:07:32.054317+0000","last_unstale":"2024-09-11T08:32:04.621289+0000","last_undegraded":"2024-09-11T08:32:04.621289+0000","last_fullsized":"2024-09-11T08:32:04.621289+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'46","last_scrub_stamp":"2024-09-11T08:32:04.621207+0000","last_deep_scrub":"478'46","last_deep_scrub_stamp":"2024-09-11T08:32:04.621207+0000","last_clean_scrub_stamp":"2024-09-11T08:32:04.621207+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:12.236782+0000","scrub_duration":0.0043013419999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":558,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:38.263418+0000","last_change":"2024-09-11T08:30:59.053520+0000","last_active":"2024-09-11T08:31:38.263418+0000","last_peered":"2024-09-11T08:31:38.263418+0000","last_clean":"2024-09-11T08:31:38.263418+0000","last_became_active":"2024-09-11T08:15:56.521943+0000","last_became_peered":"2024-09-11T08:15:56.521943+0000","last_unstale":"2024-09-11T08:31:38.263418+0000","last_undegraded":"2024-09-11T08:31:38.263418+0000","last_fullsized":"2024-09-11T08:31:38.263418+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:59.053459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:29:14.660494+0000","last_clean_scrub_stamp":"2024-09-11T08:30:59.053459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:22.745259+0000","scrub_duration":0.0027802009999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"468'14","reported_seq":614,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:18.869529+0000","last_change":"2024-09-11T08:32:02.484434+0000","last_active":"2024-09-11T08:32:18.869529+0000","last_peered":"2024-09-11T08:32:18.869529+0000","last_clean":"2024-09-11T08:32:18.869529+0000","last_became_active":"2024-09-11T08:31:16.965418+0000","last_became_peered":"2024-09-11T08:31:16.965418+0000","last_unstale":"2024-09-11T08:32:18.869529+0000","last_undegraded":"2024-09-11T08:32:18.869529+0000","last_fullsized":"2024-09-11T08:32:18.869529+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"468'14","last_scrub_stamp":"2024-09-11T08:32:02.484361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:32:02.484361+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:05.400545+0000","scrub_duration":0.0017041580000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":110,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:08.889392+0000","last_change":"2024-09-11T08:32:08.889322+0000","last_active":"2024-09-11T08:32:08.889392+0000","last_peered":"2024-09-11T08:32:08.889392+0000","last_clean":"2024-09-11T08:32:08.889392+0000","last_became_active":"2024-09-11T08:31:38.276065+0000","last_became_peered":"2024-09-11T08:31:38.276065+0000","last_unstale":"2024-09-11T08:32:08.889392+0000","last_undegraded":"2024-09-11T08:32:08.889392+0000","last_fullsized":"2024-09-11T08:32:08.889392+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:08.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:32:08.889241+0000","last_clean_scrub_stamp":"2024-09-11T08:32:08.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:24.868148+0000","scrub_duration":0.0018205300000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"405'42","reported_seq":536,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:41.997852+0000","last_change":"2024-09-11T08:31:41.997788+0000","last_active":"2024-09-11T08:31:41.997852+0000","last_peered":"2024-09-11T08:31:41.997852+0000","last_clean":"2024-09-11T08:31:41.997852+0000","last_became_active":"2024-09-11T08:31:09.937968+0000","last_became_peered":"2024-09-11T08:31:09.937968+0000","last_unstale":"2024-09-11T08:31:41.997852+0000","last_undegraded":"2024-09-11T08:31:41.997852+0000","last_fullsized":"2024-09-11T08:31:41.997852+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'42","last_scrub_stamp":"2024-09-11T08:31:41.997727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:31:41.997727+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:55.969437+0000","scrub_duration":0.0020628669999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":282,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:38.264341+0000","last_change":"2024-09-11T08:31:14.058970+0000","last_active":"2024-09-11T08:31:38.264341+0000","last_peered":"2024-09-11T08:31:38.264341+0000","last_clean":"2024-09-11T08:31:38.264341+0000","last_became_active":"2024-09-11T08:31:09.937276+0000","last_became_peered":"2024-09-11T08:31:09.937276+0000","last_unstale":"2024-09-11T08:31:38.264341+0000","last_undegraded":"2024-09-11T08:31:38.264341+0000","last_fullsized":"2024-09-11T08:31:38.264341+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:14.058909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:31:14.058909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:40.225603+0000","scrub_duration":0.0017324790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":568,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.487739+0000","last_change":"2024-09-11T08:31:54.535643+0000","last_active":"2024-09-11T08:31:59.487739+0000","last_peered":"2024-09-11T08:31:59.487739+0000","last_clean":"2024-09-11T08:31:59.487739+0000","last_became_active":"2024-09-11T08:15:56.522606+0000","last_became_peered":"2024-09-11T08:15:56.522606+0000","last_unstale":"2024-09-11T08:31:59.487739+0000","last_undegraded":"2024-09-11T08:31:59.487739+0000","last_fullsized":"2024-09-11T08:31:59.487739+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:54.535575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:31:54.535575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:22.276910+0000","scrub_duration":0.0015943730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":376,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:05.460780+0000","last_change":"2024-09-11T08:32:05.460705+0000","last_active":"2024-09-11T08:32:05.460780+0000","last_peered":"2024-09-11T08:32:05.460780+0000","last_clean":"2024-09-11T08:32:05.460780+0000","last_became_active":"2024-09-11T08:31:16.961341+0000","last_became_peered":"2024-09-11T08:31:16.961341+0000","last_unstale":"2024-09-11T08:32:05.460780+0000","last_undegraded":"2024-09-11T08:32:05.460780+0000","last_fullsized":"2024-09-11T08:32:05.460780+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:05.460598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:32:05.460598+0000","last_clean_scrub_stamp":"2024-09-11T08:32:05.460598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:19.903437+0000","scrub_duration":0.0017495709999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"405'42","reported_seq":427,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:08.480795+0000","last_change":"2024-09-11T08:32:08.480744+0000","last_active":"2024-09-11T08:32:08.480795+0000","last_peered":"2024-09-11T08:32:08.480795+0000","last_clean":"2024-09-11T08:32:08.480795+0000","last_became_active":"2024-09-11T08:31:16.964261+0000","last_became_peered":"2024-09-11T08:31:16.964261+0000","last_unstale":"2024-09-11T08:32:08.480795+0000","last_undegraded":"2024-09-11T08:32:08.480795+0000","last_fullsized":"2024-09-11T08:32:08.480795+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'42","last_scrub_stamp":"2024-09-11T08:32:08.480671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:32:08.480671+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:15.968883+0000","scrub_duration":0.0020752930000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":54,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":537,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:38.270755+0000","last_change":"2024-09-11T08:31:06.409776+0000","last_active":"2024-09-11T08:31:38.270755+0000","last_peered":"2024-09-11T08:31:38.270755+0000","last_clean":"2024-09-11T08:31:38.270755+0000","last_became_active":"2024-09-11T08:28:09.953224+0000","last_became_peered":"2024-09-11T08:28:09.953224+0000","last_unstale":"2024-09-11T08:31:38.270755+0000","last_undegraded":"2024-09-11T08:31:38.270755+0000","last_fullsized":"2024-09-11T08:31:38.270755+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:06.409723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:47.960329+0000","last_clean_scrub_stamp":"2024-09-11T08:31:06.409723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:14.371756+0000","scrub_duration":0.0018024880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"82'9","reported_seq":863,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:08.867516+0000","last_change":"2024-09-11T08:32:08.085332+0000","last_active":"2024-09-11T08:32:08.867516+0000","last_peered":"2024-09-11T08:32:08.867516+0000","last_clean":"2024-09-11T08:32:08.867516+0000","last_became_active":"2024-09-11T08:07:34.056143+0000","last_became_peered":"2024-09-11T08:07:34.056143+0000","last_unstale":"2024-09-11T08:32:08.867516+0000","last_undegraded":"2024-09-11T08:32:08.867516+0000","last_fullsized":"2024-09-11T08:32:08.867516+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-11T08:32:08.085273+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-09-11T08:21:00.841426+0000","last_clean_scrub_stamp":"2024-09-11T08:32:08.085273+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:16.498338+0000","scrub_duration":0.001828208,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":513,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:16.913105+0000","last_change":"2024-09-11T08:32:16.913065+0000","last_active":"2024-09-11T08:32:16.913105+0000","last_peered":"2024-09-11T08:32:16.913105+0000","last_clean":"2024-09-11T08:32:16.913105+0000","last_became_active":"2024-09-11T08:14:24.788968+0000","last_became_peered":"2024-09-11T08:14:24.788968+0000","last_unstale":"2024-09-11T08:32:16.913105+0000","last_undegraded":"2024-09-11T08:32:16.913105+0000","last_fullsized":"2024-09-11T08:32:16.913105+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:16.913000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:32:16.913000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:32.069029+0000","scrub_duration":0.0014423400000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"405'30","reported_seq":640,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:46.632936+0000","last_change":"2024-09-11T08:31:46.632892+0000","last_active":"2024-09-11T08:31:46.632936+0000","last_peered":"2024-09-11T08:31:46.632936+0000","last_clean":"2024-09-11T08:31:46.632936+0000","last_became_active":"2024-09-11T08:24:28.355516+0000","last_became_peered":"2024-09-11T08:24:28.355516+0000","last_unstale":"2024-09-11T08:31:46.632936+0000","last_undegraded":"2024-09-11T08:31:46.632936+0000","last_fullsized":"2024-09-11T08:31:46.632936+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'30","last_scrub_stamp":"2024-09-11T08:31:46.632833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:31:46.632833+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:59.791078+0000","scrub_duration":0.0021540579999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":100,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:46.943326+0000","last_change":"2024-09-11T08:31:46.943284+0000","last_active":"2024-09-11T08:31:46.943326+0000","last_peered":"2024-09-11T08:31:46.943326+0000","last_clean":"2024-09-11T08:31:46.943326+0000","last_became_active":"2024-09-11T08:31:38.393030+0000","last_became_peered":"2024-09-11T08:31:38.393030+0000","last_unstale":"2024-09-11T08:31:46.943326+0000","last_undegraded":"2024-09-11T08:31:46.943326+0000","last_fullsized":"2024-09-11T08:31:46.943326+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:46.943222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:31:46.943222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:58.845481+0000","scrub_duration":0.0015361439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"466'16","reported_seq":373,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:08.867499+0000","last_change":"2024-09-11T08:31:24.103808+0000","last_active":"2024-09-11T08:32:08.867499+0000","last_peered":"2024-09-11T08:32:08.867499+0000","last_clean":"2024-09-11T08:32:08.867499+0000","last_became_active":"2024-09-11T08:31:09.939449+0000","last_became_peered":"2024-09-11T08:31:09.939449+0000","last_unstale":"2024-09-11T08:32:08.867499+0000","last_undegraded":"2024-09-11T08:32:08.867499+0000","last_fullsized":"2024-09-11T08:32:08.867499+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"466'16","last_scrub_stamp":"2024-09-11T08:31:24.103737+0000","last_deep_scrub":"376'15","last_deep_scrub_stamp":"2024-09-11T08:26:02.786767+0000","last_clean_scrub_stamp":"2024-09-11T08:31:24.103737+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:45.002015+0000","scrub_duration":0.0038168189999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"66'1","reported_seq":515,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:38.271015+0000","last_change":"2024-09-11T08:31:30.394478+0000","last_active":"2024-09-11T08:31:38.271015+0000","last_peered":"2024-09-11T08:31:38.271015+0000","last_clean":"2024-09-11T08:31:38.271015+0000","last_became_active":"2024-09-11T08:07:36.072593+0000","last_became_peered":"2024-09-11T08:07:36.072593+0000","last_unstale":"2024-09-11T08:31:38.271015+0000","last_undegraded":"2024-09-11T08:31:38.271015+0000","last_fullsized":"2024-09-11T08:31:38.271015+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-11T08:31:30.394422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:31:30.394422+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:38.328646+0000","scrub_duration":0.0013632799999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"405'47","reported_seq":728,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:38.271150+0000","last_change":"2024-09-11T08:31:32.364401+0000","last_active":"2024-09-11T08:31:38.271150+0000","last_peered":"2024-09-11T08:31:38.271150+0000","last_clean":"2024-09-11T08:31:38.271150+0000","last_became_active":"2024-09-11T08:24:28.364043+0000","last_became_peered":"2024-09-11T08:24:28.364043+0000","last_unstale":"2024-09-11T08:31:38.271150+0000","last_undegraded":"2024-09-11T08:31:38.271150+0000","last_fullsized":"2024-09-11T08:31:38.271150+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'47","last_scrub_stamp":"2024-09-11T08:31:32.364338+0000","last_deep_scrub":"396'37","last_deep_scrub_stamp":"2024-09-11T08:27:16.673846+0000","last_clean_scrub_stamp":"2024-09-11T08:31:32.364338+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:45.342025+0000","scrub_duration":0.002656524,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":453,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:07.466426+0000","last_change":"2024-09-11T08:32:07.466385+0000","last_active":"2024-09-11T08:32:07.466426+0000","last_peered":"2024-09-11T08:32:07.466426+0000","last_clean":"2024-09-11T08:32:07.466426+0000","last_became_active":"2024-09-11T08:31:16.962135+0000","last_became_peered":"2024-09-11T08:31:16.962135+0000","last_unstale":"2024-09-11T08:32:07.466426+0000","last_undegraded":"2024-09-11T08:32:07.466426+0000","last_fullsized":"2024-09-11T08:32:07.466426+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:07.466317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:28:03.117012+0000","last_clean_scrub_stamp":"2024-09-11T08:32:07.466317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:24.745309+0000","scrub_duration":0.0016487509999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":580,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:38.270125+0000","last_change":"2024-09-11T08:30:50.313423+0000","last_active":"2024-09-11T08:31:38.270125+0000","last_peered":"2024-09-11T08:31:38.270125+0000","last_clean":"2024-09-11T08:31:38.270125+0000","last_became_active":"2024-09-11T08:24:28.360235+0000","last_became_peered":"2024-09-11T08:24:28.360235+0000","last_unstale":"2024-09-11T08:31:38.270125+0000","last_undegraded":"2024-09-11T08:31:38.270125+0000","last_fullsized":"2024-09-11T08:31:38.270125+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:50.313341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:30:50.313341+0000","last_clean_scrub_stamp":"2024-09-11T08:30:50.313341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:12.901942+0000","scrub_duration":0.0011976599999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":541,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:12.760347+0000","last_change":"2024-09-11T08:32:12.760320+0000","last_active":"2024-09-11T08:32:12.760347+0000","last_peered":"2024-09-11T08:32:12.760347+0000","last_clean":"2024-09-11T08:32:12.760347+0000","last_became_active":"2024-09-11T08:07:36.631668+0000","last_became_peered":"2024-09-11T08:07:36.631668+0000","last_unstale":"2024-09-11T08:32:12.760347+0000","last_undegraded":"2024-09-11T08:32:12.760347+0000","last_fullsized":"2024-09-11T08:32:12.760347+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:12.760261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:32:12.760261+0000","last_clean_scrub_stamp":"2024-09-11T08:32:12.760261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:30.736546+0000","scrub_duration":0.0015255449999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"405'64","reported_seq":429,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:13.868299+0000","last_change":"2024-09-11T08:32:05.645080+0000","last_active":"2024-09-11T08:32:13.868299+0000","last_peered":"2024-09-11T08:32:13.868299+0000","last_clean":"2024-09-11T08:32:13.868299+0000","last_became_active":"2024-09-11T08:20:52.086843+0000","last_became_peered":"2024-09-11T08:20:52.086843+0000","last_unstale":"2024-09-11T08:32:13.868299+0000","last_undegraded":"2024-09-11T08:32:13.868299+0000","last_fullsized":"2024-09-11T08:32:13.868299+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'64","last_scrub_stamp":"2024-09-11T08:32:05.645017+0000","last_deep_scrub":"190'39","last_deep_scrub_stamp":"2024-09-11T08:15:52.847968+0000","last_clean_scrub_stamp":"2024-09-11T08:32:05.645017+0000","objects_scrubbed":10,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:16.632000+0000","scrub_duration":0.002140023,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":84,"num_read_kb":69,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"59'1","reported_seq":388,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.488378+0000","last_change":"2024-09-11T08:31:50.613691+0000","last_active":"2024-09-11T08:31:59.488378+0000","last_peered":"2024-09-11T08:31:59.488378+0000","last_clean":"2024-09-11T08:31:59.488378+0000","last_became_active":"2024-09-11T08:31:16.962180+0000","last_became_peered":"2024-09-11T08:31:16.962180+0000","last_unstale":"2024-09-11T08:31:59.488378+0000","last_undegraded":"2024-09-11T08:31:59.488378+0000","last_fullsized":"2024-09-11T08:31:59.488378+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-11T08:31:50.613635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:31:50.613635+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:02.149456+0000","scrub_duration":0.0016357800000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"459'3","reported_seq":246,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:12.850771+0000","last_change":"2024-09-11T08:32:12.850729+0000","last_active":"2024-09-11T08:32:12.850771+0000","last_peered":"2024-09-11T08:32:12.850771+0000","last_clean":"2024-09-11T08:32:12.850771+0000","last_became_active":"2024-09-11T08:31:38.272317+0000","last_became_peered":"2024-09-11T08:31:38.272317+0000","last_unstale":"2024-09-11T08:32:12.850771+0000","last_undegraded":"2024-09-11T08:32:12.850771+0000","last_fullsized":"2024-09-11T08:32:12.850771+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'3","last_scrub_stamp":"2024-09-11T08:32:12.850643+0000","last_deep_scrub":"459'3","last_deep_scrub_stamp":"2024-09-11T08:32:12.850643+0000","last_clean_scrub_stamp":"2024-09-11T08:32:12.850643+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:30.170764+0000","scrub_duration":0.0021779849999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":523,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:16.164037+0000","last_change":"2024-09-11T08:32:16.163999+0000","last_active":"2024-09-11T08:32:16.164037+0000","last_peered":"2024-09-11T08:32:16.164037+0000","last_clean":"2024-09-11T08:32:16.164037+0000","last_became_active":"2024-09-11T08:07:34.057154+0000","last_became_peered":"2024-09-11T08:07:34.057154+0000","last_unstale":"2024-09-11T08:32:16.164037+0000","last_undegraded":"2024-09-11T08:32:16.164037+0000","last_fullsized":"2024-09-11T08:32:16.164037+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:16.163910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:30:56.391357+0000","last_clean_scrub_stamp":"2024-09-11T08:32:16.163910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:31.559038+0000","scrub_duration":0.001449105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":559,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:40.988294+0000","last_change":"2024-09-11T08:31:40.988258+0000","last_active":"2024-09-11T08:31:40.988294+0000","last_peered":"2024-09-11T08:31:40.988294+0000","last_clean":"2024-09-11T08:31:40.988294+0000","last_became_active":"2024-09-11T08:15:56.525099+0000","last_became_peered":"2024-09-11T08:15:56.525099+0000","last_unstale":"2024-09-11T08:31:40.988294+0000","last_undegraded":"2024-09-11T08:31:40.988294+0000","last_fullsized":"2024-09-11T08:31:40.988294+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:40.988189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:31:40.988189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:43.141955+0000","scrub_duration":0.0018492420000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"405'21","reported_seq":644,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:45.642568+0000","last_change":"2024-09-11T08:31:45.642517+0000","last_active":"2024-09-11T08:31:45.642568+0000","last_peered":"2024-09-11T08:31:45.642568+0000","last_clean":"2024-09-11T08:31:45.642568+0000","last_became_active":"2024-09-11T08:24:28.358121+0000","last_became_peered":"2024-09-11T08:24:28.358121+0000","last_unstale":"2024-09-11T08:31:45.642568+0000","last_undegraded":"2024-09-11T08:31:45.642568+0000","last_fullsized":"2024-09-11T08:31:45.642568+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'21","last_scrub_stamp":"2024-09-11T08:31:45.642458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:31:45.642458+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:50.324424+0000","scrub_duration":0.0019628229999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":550,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:38.269866+0000","last_change":"2024-09-11T08:31:10.561292+0000","last_active":"2024-09-11T08:31:38.269866+0000","last_peered":"2024-09-11T08:31:38.269866+0000","last_clean":"2024-09-11T08:31:38.269866+0000","last_became_active":"2024-09-11T08:07:30.047014+0000","last_became_peered":"2024-09-11T08:07:30.047014+0000","last_unstale":"2024-09-11T08:31:38.269866+0000","last_undegraded":"2024-09-11T08:31:38.269866+0000","last_fullsized":"2024-09-11T08:31:38.269866+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:10.561206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:31:10.561206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:39.438725+0000","scrub_duration":0.0015848870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"127'2","reported_seq":577,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:38.392629+0000","last_change":"2024-09-11T08:31:25.588502+0000","last_active":"2024-09-11T08:31:38.392629+0000","last_peered":"2024-09-11T08:31:38.392629+0000","last_clean":"2024-09-11T08:31:38.392629+0000","last_became_active":"2024-09-11T08:07:30.045009+0000","last_became_peered":"2024-09-11T08:07:30.045009+0000","last_unstale":"2024-09-11T08:31:38.392629+0000","last_undegraded":"2024-09-11T08:31:38.392629+0000","last_fullsized":"2024-09-11T08:31:38.392629+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'2","last_scrub_stamp":"2024-09-11T08:31:25.588404+0000","last_deep_scrub":"127'2","last_deep_scrub_stamp":"2024-09-11T08:15:01.938497+0000","last_clean_scrub_stamp":"2024-09-11T08:31:25.588404+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:54.376950+0000","scrub_duration":0.0018978669999999999,"objects_trimmed":0,"snaptrim_duration":0.001859533,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":603,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:38.392762+0000","last_change":"2024-09-11T08:31:14.589099+0000","last_active":"2024-09-11T08:31:38.392762+0000","last_peered":"2024-09-11T08:31:38.392762+0000","last_clean":"2024-09-11T08:31:38.392762+0000","last_became_active":"2024-09-11T08:24:28.357141+0000","last_became_peered":"2024-09-11T08:24:28.357141+0000","last_unstale":"2024-09-11T08:31:38.392762+0000","last_undegraded":"2024-09-11T08:31:38.392762+0000","last_fullsized":"2024-09-11T08:31:38.392762+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:14.589031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:31:14.589031+0000","last_clean_scrub_stamp":"2024-09-11T08:31:14.589031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:41.206920+0000","scrub_duration":0.013661458,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":263,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:38.392452+0000","last_change":"2024-09-11T08:31:19.639639+0000","last_active":"2024-09-11T08:31:38.392452+0000","last_peered":"2024-09-11T08:31:38.392452+0000","last_clean":"2024-09-11T08:31:38.392452+0000","last_became_active":"2024-09-11T08:20:16.108674+0000","last_became_peered":"2024-09-11T08:20:16.108674+0000","last_unstale":"2024-09-11T08:31:38.392452+0000","last_undegraded":"2024-09-11T08:31:38.392452+0000","last_fullsized":"2024-09-11T08:31:38.392452+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:19.639583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:58.796217+0000","last_clean_scrub_stamp":"2024-09-11T08:31:19.639583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:39.569369+0000","scrub_duration":0.0018697239999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"405'42","reported_seq":585,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:38.270591+0000","last_change":"2024-09-11T08:31:14.379856+0000","last_active":"2024-09-11T08:31:38.270591+0000","last_peered":"2024-09-11T08:31:38.270591+0000","last_clean":"2024-09-11T08:31:38.270591+0000","last_became_active":"2024-09-11T08:07:32.055407+0000","last_became_peered":"2024-09-11T08:07:32.055407+0000","last_unstale":"2024-09-11T08:31:38.270591+0000","last_undegraded":"2024-09-11T08:31:38.270591+0000","last_fullsized":"2024-09-11T08:31:38.270591+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'42","last_scrub_stamp":"2024-09-11T08:31:14.379794+0000","last_deep_scrub":"287'30","last_deep_scrub_stamp":"2024-09-11T08:26:09.030192+0000","last_clean_scrub_stamp":"2024-09-11T08:31:14.379794+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:16.412625+0000","scrub_duration":0.0017853260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"448'2","reported_seq":535,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:45.248686+0000","last_change":"2024-09-11T08:31:45.248631+0000","last_active":"2024-09-11T08:31:45.248686+0000","last_peered":"2024-09-11T08:31:45.248686+0000","last_clean":"2024-09-11T08:31:45.248686+0000","last_became_active":"2024-09-11T08:07:30.050504+0000","last_became_peered":"2024-09-11T08:07:30.050504+0000","last_unstale":"2024-09-11T08:31:45.248686+0000","last_undegraded":"2024-09-11T08:31:45.248686+0000","last_fullsized":"2024-09-11T08:31:45.248686+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'2","last_scrub_stamp":"2024-09-11T08:31:45.248571+0000","last_deep_scrub":"448'2","last_deep_scrub_stamp":"2024-09-11T08:30:18.171322+0000","last_clean_scrub_stamp":"2024-09-11T08:31:45.248571+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:09.565845+0000","scrub_duration":0.0021675890000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"224'6","reported_seq":1808,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-11T08:32:21.851429+0000","last_change":"2024-09-11T08:31:55.952630+0000","last_active":"2024-09-11T08:32:21.851429+0000","last_peered":"2024-09-11T08:32:21.851429+0000","last_clean":"2024-09-11T08:32:21.851429+0000","last_became_active":"2024-09-11T08:14:24.793048+0000","last_became_peered":"2024-09-11T08:14:24.793048+0000","last_unstale":"2024-09-11T08:32:21.851429+0000","last_undegraded":"2024-09-11T08:32:21.851429+0000","last_fullsized":"2024-09-11T08:32:21.851429+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'6","last_scrub_stamp":"2024-09-11T08:31:55.952557+0000","last_deep_scrub":"224'6","last_deep_scrub_stamp":"2024-09-11T08:30:46.710817+0000","last_clean_scrub_stamp":"2024-09-11T08:31:55.952557+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:24.069545+0000","scrub_duration":0.001767121,"objects_trimmed":0,"snaptrim_duration":0.000318821,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1083,"num_read_kb":1078,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":90,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:38.277327+0000","last_change":"2024-09-11T08:31:38.276729+0000","last_active":"2024-09-11T08:31:38.277327+0000","last_peered":"2024-09-11T08:31:38.277327+0000","last_clean":"2024-09-11T08:31:38.277327+0000","last_became_active":"2024-09-11T08:31:38.276422+0000","last_became_peered":"2024-09-11T08:31:38.276422+0000","last_unstale":"2024-09-11T08:31:38.277327+0000","last_undegraded":"2024-09-11T08:31:38.277327+0000","last_fullsized":"2024-09-11T08:31:38.277327+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:33.607524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:31:33.607524+0000","last_clean_scrub_stamp":"2024-09-11T08:31:33.607524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:58.401819+0000","scrub_duration":0.012786004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":584,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:38.272039+0000","last_change":"2024-09-11T08:31:34.640255+0000","last_active":"2024-09-11T08:31:38.272039+0000","last_peered":"2024-09-11T08:31:38.272039+0000","last_clean":"2024-09-11T08:31:38.272039+0000","last_became_active":"2024-09-11T08:24:28.358723+0000","last_became_peered":"2024-09-11T08:24:28.358723+0000","last_unstale":"2024-09-11T08:31:38.272039+0000","last_undegraded":"2024-09-11T08:31:38.272039+0000","last_fullsized":"2024-09-11T08:31:38.272039+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:34.640198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:31:34.640198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:40.799950+0000","scrub_duration":0.0015852520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"405'45","reported_seq":637,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:50.332046+0000","last_change":"2024-09-11T08:31:50.332005+0000","last_active":"2024-09-11T08:31:50.332046+0000","last_peered":"2024-09-11T08:31:50.332046+0000","last_clean":"2024-09-11T08:31:50.332046+0000","last_became_active":"2024-09-11T08:14:24.789154+0000","last_became_peered":"2024-09-11T08:14:24.789154+0000","last_unstale":"2024-09-11T08:31:50.332046+0000","last_undegraded":"2024-09-11T08:31:50.332046+0000","last_fullsized":"2024-09-11T08:31:50.332046+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'45","last_scrub_stamp":"2024-09-11T08:31:50.331948+0000","last_deep_scrub":"84'21","last_deep_scrub_stamp":"2024-09-11T08:08:56.705306+0000","last_clean_scrub_stamp":"2024-09-11T08:31:50.331948+0000","objects_scrubbed":7,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:59.645930+0000","scrub_duration":0.001894913,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":252,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:49.882297+0000","last_change":"2024-09-11T08:31:49.882240+0000","last_active":"2024-09-11T08:31:49.882297+0000","last_peered":"2024-09-11T08:31:49.882297+0000","last_clean":"2024-09-11T08:31:49.882297+0000","last_became_active":"2024-09-11T08:31:09.939314+0000","last_became_peered":"2024-09-11T08:31:09.939314+0000","last_unstale":"2024-09-11T08:31:49.882297+0000","last_undegraded":"2024-09-11T08:31:49.882297+0000","last_fullsized":"2024-09-11T08:31:49.882297+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:49.882137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:07.749828+0000","last_clean_scrub_stamp":"2024-09-11T08:31:49.882137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:50.457770+0000","scrub_duration":0.0015816000000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":357,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.340609+0000","last_change":"2024-09-11T08:31:48.864351+0000","last_active":"2024-09-11T08:31:59.340609+0000","last_peered":"2024-09-11T08:31:59.340609+0000","last_clean":"2024-09-11T08:31:59.340609+0000","last_became_active":"2024-09-11T08:15:56.523430+0000","last_became_peered":"2024-09-11T08:15:56.523430+0000","last_unstale":"2024-09-11T08:31:59.340609+0000","last_undegraded":"2024-09-11T08:31:59.340609+0000","last_fullsized":"2024-09-11T08:31:59.340609+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-11T08:31:48.864284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:05:21.231856+0000","last_clean_scrub_stamp":"2024-09-11T08:31:48.864284+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:57.253529+0000","scrub_duration":0.0019983760000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":381,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:03.934196+0000","last_change":"2024-09-11T08:32:03.934150+0000","last_active":"2024-09-11T08:32:03.934196+0000","last_peered":"2024-09-11T08:32:03.934196+0000","last_clean":"2024-09-11T08:32:03.934196+0000","last_became_active":"2024-09-11T08:24:28.357166+0000","last_became_peered":"2024-09-11T08:24:28.357166+0000","last_unstale":"2024-09-11T08:32:03.934196+0000","last_undegraded":"2024-09-11T08:32:03.934196+0000","last_fullsized":"2024-09-11T08:32:03.934196+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:03.934083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:32:03.934083+0000","last_clean_scrub_stamp":"2024-09-11T08:32:03.934083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:26.397468+0000","scrub_duration":0.0015356499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":295,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:47.935392+0000","last_change":"2024-09-11T08:31:47.935320+0000","last_active":"2024-09-11T08:31:47.935392+0000","last_peered":"2024-09-11T08:31:47.935392+0000","last_clean":"2024-09-11T08:31:47.935392+0000","last_became_active":"2024-09-11T08:31:38.274751+0000","last_became_peered":"2024-09-11T08:31:38.274751+0000","last_unstale":"2024-09-11T08:31:47.935392+0000","last_undegraded":"2024-09-11T08:31:47.935392+0000","last_fullsized":"2024-09-11T08:31:47.935392+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:47.935251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:31:47.935251+0000","last_clean_scrub_stamp":"2024-09-11T08:31:47.935251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:53.434588+0000","scrub_duration":0.001526169,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"405'32","reported_seq":412,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.488565+0000","last_change":"2024-09-11T08:31:48.522005+0000","last_active":"2024-09-11T08:31:59.488565+0000","last_peered":"2024-09-11T08:31:59.488565+0000","last_clean":"2024-09-11T08:31:59.488565+0000","last_became_active":"2024-09-11T08:31:16.964177+0000","last_became_peered":"2024-09-11T08:31:16.964177+0000","last_unstale":"2024-09-11T08:31:59.488565+0000","last_undegraded":"2024-09-11T08:31:59.488565+0000","last_fullsized":"2024-09-11T08:31:59.488565+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'32","last_scrub_stamp":"2024-09-11T08:31:48.521935+0000","last_deep_scrub":"405'32","last_deep_scrub_stamp":"2024-09-11T08:29:10.333163+0000","last_clean_scrub_stamp":"2024-09-11T08:31:48.521935+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:58.358474+0000","scrub_duration":0.0019994190000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":565,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.488114+0000","last_change":"2024-09-11T08:31:45.512656+0000","last_active":"2024-09-11T08:31:59.488114+0000","last_peered":"2024-09-11T08:31:59.488114+0000","last_clean":"2024-09-11T08:31:59.488114+0000","last_became_active":"2024-09-11T08:14:24.789888+0000","last_became_peered":"2024-09-11T08:14:24.789888+0000","last_unstale":"2024-09-11T08:31:59.488114+0000","last_undegraded":"2024-09-11T08:31:59.488114+0000","last_fullsized":"2024-09-11T08:31:59.488114+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:45.512589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:36.417911+0000","last_clean_scrub_stamp":"2024-09-11T08:31:45.512589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:08.979681+0000","scrub_duration":0.0016082480000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":100,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:41.054368+0000","last_change":"2024-09-11T08:31:41.054319+0000","last_active":"2024-09-11T08:31:41.054368+0000","last_peered":"2024-09-11T08:31:41.054368+0000","last_clean":"2024-09-11T08:31:41.054368+0000","last_became_active":"2024-09-11T08:31:38.278516+0000","last_became_peered":"2024-09-11T08:31:38.278516+0000","last_unstale":"2024-09-11T08:31:41.054368+0000","last_undegraded":"2024-09-11T08:31:41.054368+0000","last_fullsized":"2024-09-11T08:31:41.054368+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:41.054258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:01.558340+0000","last_clean_scrub_stamp":"2024-09-11T08:31:41.054258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:44.440320+0000","scrub_duration":0.001448427,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":236,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:38.263755+0000","last_change":"2024-09-11T08:31:09.938640+0000","last_active":"2024-09-11T08:31:38.263755+0000","last_peered":"2024-09-11T08:31:38.263755+0000","last_clean":"2024-09-11T08:31:38.263755+0000","last_became_active":"2024-09-11T08:31:09.938387+0000","last_became_peered":"2024-09-11T08:31:09.938387+0000","last_unstale":"2024-09-11T08:31:38.263755+0000","last_undegraded":"2024-09-11T08:31:38.263755+0000","last_fullsized":"2024-09-11T08:31:38.263755+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:56.263148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:29:16.340778+0000","last_clean_scrub_stamp":"2024-09-11T08:30:56.263148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:14.025083+0000","scrub_duration":0.0055940149999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"405'53","reported_seq":463,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.488535+0000","last_change":"2024-09-11T08:31:47.560601+0000","last_active":"2024-09-11T08:31:59.488535+0000","last_peered":"2024-09-11T08:31:59.488535+0000","last_clean":"2024-09-11T08:31:59.488535+0000","last_became_active":"2024-09-11T08:31:16.962778+0000","last_became_peered":"2024-09-11T08:31:16.962778+0000","last_unstale":"2024-09-11T08:31:59.488535+0000","last_undegraded":"2024-09-11T08:31:59.488535+0000","last_fullsized":"2024-09-11T08:31:59.488535+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'53","last_scrub_stamp":"2024-09-11T08:31:47.560530+0000","last_deep_scrub":"77'25","last_deep_scrub_stamp":"2024-09-11T08:08:32.458210+0000","last_clean_scrub_stamp":"2024-09-11T08:31:47.560530+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:05.414991+0000","scrub_duration":0.0024864760000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":59,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":553,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:38.270418+0000","last_change":"2024-09-11T08:31:22.290865+0000","last_active":"2024-09-11T08:31:38.270418+0000","last_peered":"2024-09-11T08:31:38.270418+0000","last_clean":"2024-09-11T08:31:38.270418+0000","last_became_active":"2024-09-11T08:07:30.042555+0000","last_became_peered":"2024-09-11T08:07:30.042555+0000","last_unstale":"2024-09-11T08:31:38.270418+0000","last_undegraded":"2024-09-11T08:31:38.270418+0000","last_fullsized":"2024-09-11T08:31:38.270418+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:22.290779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:31:22.290779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:45.306345+0000","scrub_duration":0.0013764719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":543,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:14.820310+0000","last_change":"2024-09-11T08:32:14.820270+0000","last_active":"2024-09-11T08:32:14.820310+0000","last_peered":"2024-09-11T08:32:14.820310+0000","last_clean":"2024-09-11T08:32:14.820310+0000","last_became_active":"2024-09-11T08:07:34.054961+0000","last_became_peered":"2024-09-11T08:07:34.054961+0000","last_unstale":"2024-09-11T08:32:14.820310+0000","last_undegraded":"2024-09-11T08:32:14.820310+0000","last_fullsized":"2024-09-11T08:32:14.820310+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:14.820205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:32:14.820205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:29.998148+0000","scrub_duration":0.0016007370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":561,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:38.392627+0000","last_change":"2024-09-11T08:31:23.626128+0000","last_active":"2024-09-11T08:31:38.392627+0000","last_peered":"2024-09-11T08:31:38.392627+0000","last_clean":"2024-09-11T08:31:38.392627+0000","last_became_active":"2024-09-11T08:07:36.071194+0000","last_became_peered":"2024-09-11T08:07:36.071194+0000","last_unstale":"2024-09-11T08:31:38.392627+0000","last_undegraded":"2024-09-11T08:31:38.392627+0000","last_fullsized":"2024-09-11T08:31:38.392627+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:23.626078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:07.930137+0000","last_clean_scrub_stamp":"2024-09-11T08:31:23.626078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:40.412275+0000","scrub_duration":0.001315185,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"405'35","reported_seq":397,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.488385+0000","last_change":"2024-09-11T08:31:26.528006+0000","last_active":"2024-09-11T08:31:59.488385+0000","last_peered":"2024-09-11T08:31:59.488385+0000","last_clean":"2024-09-11T08:31:59.488385+0000","last_became_active":"2024-09-11T08:31:16.965159+0000","last_became_peered":"2024-09-11T08:31:16.965159+0000","last_unstale":"2024-09-11T08:31:59.488385+0000","last_undegraded":"2024-09-11T08:31:59.488385+0000","last_fullsized":"2024-09-11T08:31:59.488385+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'35","last_scrub_stamp":"2024-09-11T08:31:26.527947+0000","last_deep_scrub":"224'15","last_deep_scrub_stamp":"2024-09-11T08:17:01.126478+0000","last_clean_scrub_stamp":"2024-09-11T08:31:26.527947+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:54.768876+0000","scrub_duration":0.0018737820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":262,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:44.943891+0000","last_change":"2024-09-11T08:31:44.943848+0000","last_active":"2024-09-11T08:31:44.943891+0000","last_peered":"2024-09-11T08:31:44.943891+0000","last_clean":"2024-09-11T08:31:44.943891+0000","last_became_active":"2024-09-11T08:31:09.935630+0000","last_became_peered":"2024-09-11T08:31:09.935630+0000","last_unstale":"2024-09-11T08:31:44.943891+0000","last_undegraded":"2024-09-11T08:31:44.943891+0000","last_fullsized":"2024-09-11T08:31:44.943891+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:44.943776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:27:59.866811+0000","last_clean_scrub_stamp":"2024-09-11T08:31:44.943776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:56.179793+0000","scrub_duration":0.0018035600000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"354'16","reported_seq":929,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:14.258209+0000","last_change":"2024-09-11T08:32:14.258182+0000","last_active":"2024-09-11T08:32:14.258209+0000","last_peered":"2024-09-11T08:32:14.258209+0000","last_clean":"2024-09-11T08:32:14.258209+0000","last_became_active":"2024-09-11T08:24:28.356749+0000","last_became_peered":"2024-09-11T08:24:28.356749+0000","last_unstale":"2024-09-11T08:32:14.258209+0000","last_undegraded":"2024-09-11T08:32:14.258209+0000","last_fullsized":"2024-09-11T08:32:14.258209+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'16","last_scrub_stamp":"2024-09-11T08:32:14.258123+0000","last_deep_scrub":"171'12","last_deep_scrub_stamp":"2024-09-11T08:14:56.881299+0000","last_clean_scrub_stamp":"2024-09-11T08:32:14.258123+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:15.941380+0000","scrub_duration":0.0021232379999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":353,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.488436+0000","last_change":"2024-09-11T08:31:16.963036+0000","last_active":"2024-09-11T08:31:59.488436+0000","last_peered":"2024-09-11T08:31:59.488436+0000","last_clean":"2024-09-11T08:31:59.488436+0000","last_became_active":"2024-09-11T08:31:16.962725+0000","last_became_peered":"2024-09-11T08:31:16.962725+0000","last_unstale":"2024-09-11T08:31:59.488436+0000","last_undegraded":"2024-09-11T08:31:59.488436+0000","last_fullsized":"2024-09-11T08:31:59.488436+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:03.429821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:31:03.429821+0000","last_clean_scrub_stamp":"2024-09-11T08:31:03.429821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:24.832924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"405'21","reported_seq":559,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:48.272946+0000","last_change":"2024-09-11T08:31:48.272868+0000","last_active":"2024-09-11T08:31:48.272946+0000","last_peered":"2024-09-11T08:31:48.272946+0000","last_clean":"2024-09-11T08:31:48.272946+0000","last_became_active":"2024-09-11T08:07:32.054885+0000","last_became_peered":"2024-09-11T08:07:32.054885+0000","last_unstale":"2024-09-11T08:31:48.272946+0000","last_undegraded":"2024-09-11T08:31:48.272946+0000","last_fullsized":"2024-09-11T08:31:48.272946+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'21","last_scrub_stamp":"2024-09-11T08:31:48.272742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:31:48.272742+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:59.397071+0000","scrub_duration":0.0025347939999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":100,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:06.889073+0000","last_change":"2024-09-11T08:32:06.889040+0000","last_active":"2024-09-11T08:32:06.889073+0000","last_peered":"2024-09-11T08:32:06.889073+0000","last_clean":"2024-09-11T08:32:06.889073+0000","last_became_active":"2024-09-11T08:31:38.276887+0000","last_became_peered":"2024-09-11T08:31:38.276887+0000","last_unstale":"2024-09-11T08:32:06.889073+0000","last_undegraded":"2024-09-11T08:32:06.889073+0000","last_fullsized":"2024-09-11T08:32:06.889073+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:06.888968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:32:06.888968+0000","last_clean_scrub_stamp":"2024-09-11T08:32:06.888968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:23.537405+0000","scrub_duration":0.001438246,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":573,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:38.392399+0000","last_change":"2024-09-11T08:31:16.558479+0000","last_active":"2024-09-11T08:31:38.392399+0000","last_peered":"2024-09-11T08:31:38.392399+0000","last_clean":"2024-09-11T08:31:38.392399+0000","last_became_active":"2024-09-11T08:07:34.056606+0000","last_became_peered":"2024-09-11T08:07:34.056606+0000","last_unstale":"2024-09-11T08:31:38.392399+0000","last_undegraded":"2024-09-11T08:31:38.392399+0000","last_fullsized":"2024-09-11T08:31:38.392399+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:16.558414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:31:16.558414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:41.693182+0000","scrub_duration":0.0016748749999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":548,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:58.294453+0000","last_change":"2024-09-11T08:31:58.294412+0000","last_active":"2024-09-11T08:31:58.294453+0000","last_peered":"2024-09-11T08:31:58.294453+0000","last_clean":"2024-09-11T08:31:58.294453+0000","last_became_active":"2024-09-11T08:15:56.518884+0000","last_became_peered":"2024-09-11T08:15:56.518884+0000","last_unstale":"2024-09-11T08:31:58.294453+0000","last_undegraded":"2024-09-11T08:31:58.294453+0000","last_fullsized":"2024-09-11T08:31:58.294453+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:58.294355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:18:32.278257+0000","last_clean_scrub_stamp":"2024-09-11T08:31:58.294355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:10.080664+0000","scrub_duration":0.0016822079999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"451'60","reported_seq":720,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:51.304517+0000","last_change":"2024-09-11T08:31:51.304466+0000","last_active":"2024-09-11T08:31:51.304517+0000","last_peered":"2024-09-11T08:31:51.304517+0000","last_clean":"2024-09-11T08:31:51.304517+0000","last_became_active":"2024-09-11T08:15:56.526961+0000","last_became_peered":"2024-09-11T08:15:56.526961+0000","last_unstale":"2024-09-11T08:31:51.304517+0000","last_undegraded":"2024-09-11T08:31:51.304517+0000","last_fullsized":"2024-09-11T08:31:51.304517+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"451'60","last_scrub_stamp":"2024-09-11T08:31:51.304392+0000","last_deep_scrub":"451'60","last_deep_scrub_stamp":"2024-09-11T08:30:32.275144+0000","last_clean_scrub_stamp":"2024-09-11T08:31:51.304392+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:06.903100+0000","scrub_duration":0.002187558,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":381,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.341124+0000","last_change":"2024-09-11T08:31:33.702517+0000","last_active":"2024-09-11T08:31:59.341124+0000","last_peered":"2024-09-11T08:31:59.341124+0000","last_clean":"2024-09-11T08:31:59.341124+0000","last_became_active":"2024-09-11T08:24:28.353720+0000","last_became_peered":"2024-09-11T08:24:28.353720+0000","last_unstale":"2024-09-11T08:31:59.341124+0000","last_undegraded":"2024-09-11T08:31:59.341124+0000","last_fullsized":"2024-09-11T08:31:59.341124+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:33.702440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:44.385372+0000","last_clean_scrub_stamp":"2024-09-11T08:31:33.702440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:46.595277+0000","scrub_duration":0.001500501,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":603,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:05.259520+0000","last_change":"2024-09-11T08:32:05.259494+0000","last_active":"2024-09-11T08:32:05.259520+0000","last_peered":"2024-09-11T08:32:05.259520+0000","last_clean":"2024-09-11T08:32:05.259520+0000","last_became_active":"2024-09-11T08:24:28.365792+0000","last_became_peered":"2024-09-11T08:24:28.365792+0000","last_unstale":"2024-09-11T08:32:05.259520+0000","last_undegraded":"2024-09-11T08:32:05.259520+0000","last_fullsized":"2024-09-11T08:32:05.259520+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:05.259438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:30:38.507031+0000","last_clean_scrub_stamp":"2024-09-11T08:32:05.259438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:19.690276+0000","scrub_duration":0.001688043,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":341,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.342147+0000","last_change":"2024-09-11T08:31:04.709026+0000","last_active":"2024-09-11T08:31:59.342147+0000","last_peered":"2024-09-11T08:31:59.342147+0000","last_clean":"2024-09-11T08:31:59.342147+0000","last_became_active":"2024-09-11T08:15:56.518169+0000","last_became_peered":"2024-09-11T08:15:56.518169+0000","last_unstale":"2024-09-11T08:31:59.342147+0000","last_undegraded":"2024-09-11T08:31:59.342147+0000","last_fullsized":"2024-09-11T08:31:59.342147+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:04.708945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:31:04.708945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:26.492136+0000","scrub_duration":0.001885889,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"405'29","reported_seq":402,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.487847+0000","last_change":"2024-09-11T08:31:58.508552+0000","last_active":"2024-09-11T08:31:59.487847+0000","last_peered":"2024-09-11T08:31:59.487847+0000","last_clean":"2024-09-11T08:31:59.487847+0000","last_became_active":"2024-09-11T08:31:16.964141+0000","last_became_peered":"2024-09-11T08:31:16.964141+0000","last_unstale":"2024-09-11T08:31:59.487847+0000","last_undegraded":"2024-09-11T08:31:59.487847+0000","last_fullsized":"2024-09-11T08:31:59.487847+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'29","last_scrub_stamp":"2024-09-11T08:31:58.508482+0000","last_deep_scrub":"145'17","last_deep_scrub_stamp":"2024-09-11T08:13:08.160848+0000","last_clean_scrub_stamp":"2024-09-11T08:31:58.508482+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:12.085510+0000","scrub_duration":0.0020195640000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":351,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.341460+0000","last_change":"2024-09-11T08:31:06.638726+0000","last_active":"2024-09-11T08:31:59.341460+0000","last_peered":"2024-09-11T08:31:59.341460+0000","last_clean":"2024-09-11T08:31:59.341460+0000","last_became_active":"2024-09-11T08:15:56.525845+0000","last_became_peered":"2024-09-11T08:15:56.525845+0000","last_unstale":"2024-09-11T08:31:59.341460+0000","last_undegraded":"2024-09-11T08:31:59.341460+0000","last_fullsized":"2024-09-11T08:31:59.341460+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:06.638658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:21:06.262572+0000","last_clean_scrub_stamp":"2024-09-11T08:31:06.638658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001497099,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":553,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:46.484769+0000","last_change":"2024-09-11T08:31:46.484735+0000","last_active":"2024-09-11T08:31:46.484769+0000","last_peered":"2024-09-11T08:31:46.484769+0000","last_clean":"2024-09-11T08:31:46.484769+0000","last_became_active":"2024-09-11T08:07:34.057479+0000","last_became_peered":"2024-09-11T08:07:34.057479+0000","last_unstale":"2024-09-11T08:31:46.484769+0000","last_undegraded":"2024-09-11T08:31:46.484769+0000","last_fullsized":"2024-09-11T08:31:46.484769+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:46.484680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:31:46.484680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:14.470245+0000","scrub_duration":0.001263501,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":569,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:53.940518+0000","last_change":"2024-09-11T08:31:53.940475+0000","last_active":"2024-09-11T08:31:53.940518+0000","last_peered":"2024-09-11T08:31:53.940518+0000","last_clean":"2024-09-11T08:31:53.940518+0000","last_became_active":"2024-09-11T08:15:56.522624+0000","last_became_peered":"2024-09-11T08:15:56.522624+0000","last_unstale":"2024-09-11T08:31:53.940518+0000","last_undegraded":"2024-09-11T08:31:53.940518+0000","last_fullsized":"2024-09-11T08:31:53.940518+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:53.940413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:31:53.940413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:59.428036+0000","scrub_duration":0.001889745,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"405'39","reported_seq":520,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:38.275956+0000","last_change":"2024-09-11T08:31:38.274144+0000","last_active":"2024-09-11T08:31:38.275956+0000","last_peered":"2024-09-11T08:31:38.275956+0000","last_clean":"2024-09-11T08:31:38.275956+0000","last_became_active":"2024-09-11T08:31:38.273273+0000","last_became_peered":"2024-09-11T08:31:38.273273+0000","last_unstale":"2024-09-11T08:31:38.275956+0000","last_undegraded":"2024-09-11T08:31:38.275956+0000","last_fullsized":"2024-09-11T08:31:38.275956+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'39","last_scrub_stamp":"2024-09-11T08:31:31.708951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:31:31.708951+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:59.554272+0000","scrub_duration":0.0026965510000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"478'4","reported_seq":359,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:11.862274+0000","last_change":"2024-09-11T08:32:11.862228+0000","last_active":"2024-09-11T08:32:11.862274+0000","last_peered":"2024-09-11T08:32:11.862274+0000","last_clean":"2024-09-11T08:32:11.862274+0000","last_became_active":"2024-09-11T08:15:56.515328+0000","last_became_peered":"2024-09-11T08:15:56.515328+0000","last_unstale":"2024-09-11T08:32:11.862274+0000","last_undegraded":"2024-09-11T08:32:11.862274+0000","last_fullsized":"2024-09-11T08:32:11.862274+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'4","last_scrub_stamp":"2024-09-11T08:32:11.862167+0000","last_deep_scrub":"478'4","last_deep_scrub_stamp":"2024-09-11T08:32:11.862167+0000","last_clean_scrub_stamp":"2024-09-11T08:32:11.862167+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:20.063599+0000","scrub_duration":0.0030199530000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"209'11","reported_seq":542,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:18.869570+0000","last_change":"2024-09-11T08:31:13.678884+0000","last_active":"2024-09-11T08:32:18.869570+0000","last_peered":"2024-09-11T08:32:18.869570+0000","last_clean":"2024-09-11T08:32:18.869570+0000","last_became_active":"2024-09-11T08:15:56.524513+0000","last_became_peered":"2024-09-11T08:15:56.524513+0000","last_unstale":"2024-09-11T08:32:18.869570+0000","last_undegraded":"2024-09-11T08:32:18.869570+0000","last_fullsized":"2024-09-11T08:32:18.869570+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"209'11","last_scrub_stamp":"2024-09-11T08:31:13.678807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:31:13.678807+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:39.976164+0000","scrub_duration":0.001875481,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":90,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:38.275911+0000","last_change":"2024-09-11T08:31:38.273925+0000","last_active":"2024-09-11T08:31:38.275911+0000","last_peered":"2024-09-11T08:31:38.275911+0000","last_clean":"2024-09-11T08:31:38.275911+0000","last_became_active":"2024-09-11T08:31:38.273642+0000","last_became_peered":"2024-09-11T08:31:38.273642+0000","last_unstale":"2024-09-11T08:31:38.275911+0000","last_undegraded":"2024-09-11T08:31:38.275911+0000","last_fullsized":"2024-09-11T08:31:38.275911+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:21.258836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:03.484755+0000","last_clean_scrub_stamp":"2024-09-11T08:31:21.258836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:45.633432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":597,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:56.331149+0000","last_change":"2024-09-11T08:31:56.331092+0000","last_active":"2024-09-11T08:31:56.331149+0000","last_peered":"2024-09-11T08:31:56.331149+0000","last_clean":"2024-09-11T08:31:56.331149+0000","last_became_active":"2024-09-11T08:24:28.357401+0000","last_became_peered":"2024-09-11T08:24:28.357401+0000","last_unstale":"2024-09-11T08:31:56.331149+0000","last_undegraded":"2024-09-11T08:31:56.331149+0000","last_fullsized":"2024-09-11T08:31:56.331149+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:56.331024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:31:56.331024+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:01.009568+0000","scrub_duration":0.0014664470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"405'21","reported_seq":609,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:38.264325+0000","last_change":"2024-09-11T08:31:02.037660+0000","last_active":"2024-09-11T08:31:38.264325+0000","last_peered":"2024-09-11T08:31:38.264325+0000","last_clean":"2024-09-11T08:31:38.264325+0000","last_became_active":"2024-09-11T08:15:56.526325+0000","last_became_peered":"2024-09-11T08:15:56.526325+0000","last_unstale":"2024-09-11T08:31:38.264325+0000","last_undegraded":"2024-09-11T08:31:38.264325+0000","last_fullsized":"2024-09-11T08:31:38.264325+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'21","last_scrub_stamp":"2024-09-11T08:31:02.037597+0000","last_deep_scrub":"405'21","last_deep_scrub_stamp":"2024-09-11T08:29:15.624982+0000","last_clean_scrub_stamp":"2024-09-11T08:31:02.037597+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:20.938739+0000","scrub_duration":0.0022002020000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":100,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:42.012587+0000","last_change":"2024-09-11T08:31:42.012533+0000","last_active":"2024-09-11T08:31:42.012587+0000","last_peered":"2024-09-11T08:31:42.012587+0000","last_clean":"2024-09-11T08:31:42.012587+0000","last_became_active":"2024-09-11T08:31:38.275006+0000","last_became_peered":"2024-09-11T08:31:38.275006+0000","last_unstale":"2024-09-11T08:31:42.012587+0000","last_undegraded":"2024-09-11T08:31:42.012587+0000","last_fullsized":"2024-09-11T08:31:42.012587+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:42.012472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:53.548967+0000","last_clean_scrub_stamp":"2024-09-11T08:31:42.012472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:52.583946+0000","scrub_duration":0.001917462,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":476,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.341263+0000","last_change":"2024-09-11T08:31:20.679585+0000","last_active":"2024-09-11T08:31:59.341263+0000","last_peered":"2024-09-11T08:31:59.341263+0000","last_clean":"2024-09-11T08:31:59.341263+0000","last_became_active":"2024-09-11T08:14:24.790678+0000","last_became_peered":"2024-09-11T08:14:24.790678+0000","last_unstale":"2024-09-11T08:31:59.341263+0000","last_undegraded":"2024-09-11T08:31:59.341263+0000","last_fullsized":"2024-09-11T08:31:59.341263+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:20.679515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:29:20.428966+0000","last_clean_scrub_stamp":"2024-09-11T08:31:20.679515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:43.318353+0000","scrub_duration":0.0015400959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"405'25","reported_seq":600,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:38.270678+0000","last_change":"2024-09-11T08:31:26.439539+0000","last_active":"2024-09-11T08:31:38.270678+0000","last_peered":"2024-09-11T08:31:38.270678+0000","last_clean":"2024-09-11T08:31:38.270678+0000","last_became_active":"2024-09-11T08:15:56.514793+0000","last_became_peered":"2024-09-11T08:15:56.514793+0000","last_unstale":"2024-09-11T08:31:38.270678+0000","last_undegraded":"2024-09-11T08:31:38.270678+0000","last_fullsized":"2024-09-11T08:31:38.270678+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'25","last_scrub_stamp":"2024-09-11T08:31:26.439480+0000","last_deep_scrub":"84'13","last_deep_scrub_stamp":"2024-09-11T08:08:57.835634+0000","last_clean_scrub_stamp":"2024-09-11T08:31:26.439480+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:28.941228+0000","scrub_duration":0.001742469,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":499,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.341344+0000","last_change":"2024-09-11T08:31:15.661657+0000","last_active":"2024-09-11T08:31:59.341344+0000","last_peered":"2024-09-11T08:31:59.341344+0000","last_clean":"2024-09-11T08:31:59.341344+0000","last_became_active":"2024-09-11T08:14:24.786021+0000","last_became_peered":"2024-09-11T08:14:24.786021+0000","last_unstale":"2024-09-11T08:31:59.341344+0000","last_undegraded":"2024-09-11T08:31:59.341344+0000","last_fullsized":"2024-09-11T08:31:59.341344+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:15.661591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:30.471620+0000","last_clean_scrub_stamp":"2024-09-11T08:31:15.661591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:24.759947+0000","scrub_duration":0.0016265870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":561,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:38.946798+0000","last_change":"2024-09-11T08:31:38.946734+0000","last_active":"2024-09-11T08:31:38.946798+0000","last_peered":"2024-09-11T08:31:38.946798+0000","last_clean":"2024-09-11T08:31:38.946798+0000","last_became_active":"2024-09-11T08:15:56.525717+0000","last_became_peered":"2024-09-11T08:15:56.525717+0000","last_unstale":"2024-09-11T08:31:38.946798+0000","last_undegraded":"2024-09-11T08:31:38.946798+0000","last_fullsized":"2024-09-11T08:31:38.946798+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:38.946675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:04.370734+0000","last_clean_scrub_stamp":"2024-09-11T08:31:38.946675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:42.702831+0000","scrub_duration":0.0017947779999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":571,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:10.159949+0000","last_change":"2024-09-11T08:32:10.159924+0000","last_active":"2024-09-11T08:32:10.159949+0000","last_peered":"2024-09-11T08:32:10.159949+0000","last_clean":"2024-09-11T08:32:10.159949+0000","last_became_active":"2024-09-11T08:15:56.525120+0000","last_became_peered":"2024-09-11T08:15:56.525120+0000","last_unstale":"2024-09-11T08:32:10.159949+0000","last_undegraded":"2024-09-11T08:32:10.159949+0000","last_fullsized":"2024-09-11T08:32:10.159949+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:10.159870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:30:51.492107+0000","last_clean_scrub_stamp":"2024-09-11T08:32:10.159870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:31.581341+0000","scrub_duration":0.0015020299999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"405'25","reported_seq":205,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:38.276546+0000","last_change":"2024-09-11T08:31:38.275782+0000","last_active":"2024-09-11T08:31:38.276546+0000","last_peered":"2024-09-11T08:31:38.276546+0000","last_clean":"2024-09-11T08:31:38.276546+0000","last_became_active":"2024-09-11T08:31:38.275398+0000","last_became_peered":"2024-09-11T08:31:38.275398+0000","last_unstale":"2024-09-11T08:31:38.276546+0000","last_undegraded":"2024-09-11T08:31:38.276546+0000","last_fullsized":"2024-09-11T08:31:38.276546+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'25","last_scrub_stamp":"2024-09-11T08:31:27.625910+0000","last_deep_scrub":"313'19","last_deep_scrub_stamp":"2024-09-11T08:22:03.458402+0000","last_clean_scrub_stamp":"2024-09-11T08:31:27.625910+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:31.716398+0000","scrub_duration":0.0041528440000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":554,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:38.270591+0000","last_change":"2024-09-11T08:31:35.353159+0000","last_active":"2024-09-11T08:31:38.270591+0000","last_peered":"2024-09-11T08:31:38.270591+0000","last_clean":"2024-09-11T08:31:38.270591+0000","last_became_active":"2024-09-11T08:15:56.513278+0000","last_became_peered":"2024-09-11T08:15:56.513278+0000","last_unstale":"2024-09-11T08:31:38.270591+0000","last_undegraded":"2024-09-11T08:31:38.270591+0000","last_fullsized":"2024-09-11T08:31:38.270591+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:35.353103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:52.661591+0000","last_clean_scrub_stamp":"2024-09-11T08:31:35.353103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:56.859504+0000","scrub_duration":0.0016773529999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"466'14","reported_seq":437,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:08.867558+0000","last_change":"2024-09-11T08:32:06.015311+0000","last_active":"2024-09-11T08:32:08.867558+0000","last_peered":"2024-09-11T08:32:08.867558+0000","last_clean":"2024-09-11T08:32:08.867558+0000","last_became_active":"2024-09-11T08:31:09.937487+0000","last_became_peered":"2024-09-11T08:31:09.937487+0000","last_unstale":"2024-09-11T08:32:08.867558+0000","last_undegraded":"2024-09-11T08:32:08.867558+0000","last_fullsized":"2024-09-11T08:32:08.867558+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"466'14","last_scrub_stamp":"2024-09-11T08:32:06.015249+0000","last_deep_scrub":"222'12","last_deep_scrub_stamp":"2024-09-11T08:16:52.592856+0000","last_clean_scrub_stamp":"2024-09-11T08:32:06.015249+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:32.595210+0000","scrub_duration":0.0016785470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":383,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.488679+0000","last_change":"2024-09-11T08:31:30.513125+0000","last_active":"2024-09-11T08:31:59.488679+0000","last_peered":"2024-09-11T08:31:59.488679+0000","last_clean":"2024-09-11T08:31:59.488679+0000","last_became_active":"2024-09-11T08:31:16.964077+0000","last_became_peered":"2024-09-11T08:31:16.964077+0000","last_unstale":"2024-09-11T08:31:59.488679+0000","last_undegraded":"2024-09-11T08:31:59.488679+0000","last_fullsized":"2024-09-11T08:31:59.488679+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:30.513054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:53.175994+0000","last_clean_scrub_stamp":"2024-09-11T08:31:30.513054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:58.968082+0000","scrub_duration":0.0014064959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"477'35","reported_seq":371,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:56.969858+0000","last_change":"2024-09-11T08:31:56.969779+0000","last_active":"2024-09-11T08:31:56.969858+0000","last_peered":"2024-09-11T08:31:56.969858+0000","last_clean":"2024-09-11T08:31:56.969858+0000","last_became_active":"2024-09-11T08:31:09.937377+0000","last_became_peered":"2024-09-11T08:31:09.937377+0000","last_unstale":"2024-09-11T08:31:56.969858+0000","last_undegraded":"2024-09-11T08:31:56.969858+0000","last_fullsized":"2024-09-11T08:31:56.969858+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"477'35","last_scrub_stamp":"2024-09-11T08:31:56.969691+0000","last_deep_scrub":"477'35","last_deep_scrub_stamp":"2024-09-11T08:31:56.969691+0000","last_clean_scrub_stamp":"2024-09-11T08:31:56.969691+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:59.322094+0000","scrub_duration":0.003427692,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":351,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:13.881748+0000","last_change":"2024-09-11T08:32:13.881675+0000","last_active":"2024-09-11T08:32:13.881748+0000","last_peered":"2024-09-11T08:32:13.881748+0000","last_clean":"2024-09-11T08:32:13.881748+0000","last_became_active":"2024-09-11T08:15:56.518515+0000","last_became_peered":"2024-09-11T08:15:56.518515+0000","last_unstale":"2024-09-11T08:32:13.881748+0000","last_undegraded":"2024-09-11T08:32:13.881748+0000","last_fullsized":"2024-09-11T08:32:13.881748+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:13.881157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:23:42.199527+0000","last_clean_scrub_stamp":"2024-09-11T08:32:13.881157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:15.725971+0000","scrub_duration":0.001541398,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"477'14","reported_seq":165,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:13.868526+0000","last_change":"2024-09-11T08:31:48.912806+0000","last_active":"2024-09-11T08:32:13.868526+0000","last_peered":"2024-09-11T08:32:13.868526+0000","last_clean":"2024-09-11T08:32:13.868526+0000","last_became_active":"2024-09-11T08:31:38.273682+0000","last_became_peered":"2024-09-11T08:31:38.273682+0000","last_unstale":"2024-09-11T08:32:13.868526+0000","last_undegraded":"2024-09-11T08:32:13.868526+0000","last_fullsized":"2024-09-11T08:32:13.868526+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"477'14","last_scrub_stamp":"2024-09-11T08:31:48.912748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:31:48.912748+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:02.991937+0000","scrub_duration":0.001689704,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":538,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:38.270365+0000","last_change":"2024-09-11T08:31:10.355932+0000","last_active":"2024-09-11T08:31:38.270365+0000","last_peered":"2024-09-11T08:31:38.270365+0000","last_clean":"2024-09-11T08:31:38.270365+0000","last_became_active":"2024-09-11T08:15:56.513949+0000","last_became_peered":"2024-09-11T08:15:56.513949+0000","last_unstale":"2024-09-11T08:31:38.270365+0000","last_undegraded":"2024-09-11T08:31:38.270365+0000","last_fullsized":"2024-09-11T08:31:38.270365+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:10.355845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:29:05.957912+0000","last_clean_scrub_stamp":"2024-09-11T08:31:10.355845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:31.364564+0000","scrub_duration":0.001761047,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"405'27","reported_seq":390,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.488069+0000","last_change":"2024-09-11T08:31:39.447509+0000","last_active":"2024-09-11T08:31:59.488069+0000","last_peered":"2024-09-11T08:31:59.488069+0000","last_clean":"2024-09-11T08:31:59.488069+0000","last_became_active":"2024-09-11T08:31:16.963200+0000","last_became_peered":"2024-09-11T08:31:16.963200+0000","last_unstale":"2024-09-11T08:31:59.488069+0000","last_undegraded":"2024-09-11T08:31:59.488069+0000","last_fullsized":"2024-09-11T08:31:59.488069+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'27","last_scrub_stamp":"2024-09-11T08:31:39.447428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:31:39.447428+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:41.759058+0000","scrub_duration":0.002205252,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":409,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-11T08:32:22.301348+0000","last_change":"2024-09-11T08:32:22.301313+0000","last_active":"2024-09-11T08:32:22.301348+0000","last_peered":"2024-09-11T08:32:22.301348+0000","last_clean":"2024-09-11T08:32:22.301348+0000","last_became_active":"2024-09-11T08:31:16.963281+0000","last_became_peered":"2024-09-11T08:31:16.963281+0000","last_unstale":"2024-09-11T08:32:22.301348+0000","last_undegraded":"2024-09-11T08:32:22.301348+0000","last_fullsized":"2024-09-11T08:32:22.301348+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:22.301245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:32:22.301245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:44.324684+0000","scrub_duration":0.0015087729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":573,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:15.780703+0000","last_change":"2024-09-11T08:32:15.780680+0000","last_active":"2024-09-11T08:32:15.780703+0000","last_peered":"2024-09-11T08:32:15.780703+0000","last_clean":"2024-09-11T08:32:15.780703+0000","last_became_active":"2024-09-11T08:24:28.355004+0000","last_became_peered":"2024-09-11T08:24:28.355004+0000","last_unstale":"2024-09-11T08:32:15.780703+0000","last_undegraded":"2024-09-11T08:32:15.780703+0000","last_fullsized":"2024-09-11T08:32:15.780703+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:15.780625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:32:15.780625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:38.736839+0000","scrub_duration":0.0015874940000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":246,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:38.263278+0000","last_change":"2024-09-11T08:31:16.988591+0000","last_active":"2024-09-11T08:31:38.263278+0000","last_peered":"2024-09-11T08:31:38.263278+0000","last_clean":"2024-09-11T08:31:38.263278+0000","last_became_active":"2024-09-11T08:31:09.939169+0000","last_became_peered":"2024-09-11T08:31:09.939169+0000","last_unstale":"2024-09-11T08:31:38.263278+0000","last_undegraded":"2024-09-11T08:31:38.263278+0000","last_fullsized":"2024-09-11T08:31:38.263278+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:16.988505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:31:16.988505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:35.235815+0000","scrub_duration":0.0018742850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"405'25","reported_seq":593,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:17.847754+0000","last_change":"2024-09-11T08:32:17.847720+0000","last_active":"2024-09-11T08:32:17.847754+0000","last_peered":"2024-09-11T08:32:17.847754+0000","last_clean":"2024-09-11T08:32:17.847754+0000","last_became_active":"2024-09-11T08:07:32.049368+0000","last_became_peered":"2024-09-11T08:07:32.049368+0000","last_unstale":"2024-09-11T08:32:17.847754+0000","last_undegraded":"2024-09-11T08:32:17.847754+0000","last_fullsized":"2024-09-11T08:32:17.847754+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'25","last_scrub_stamp":"2024-09-11T08:32:17.847645+0000","last_deep_scrub":"343'19","last_deep_scrub_stamp":"2024-09-11T08:23:44.640077+0000","last_clean_scrub_stamp":"2024-09-11T08:32:17.847645+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:33.522593+0000","scrub_duration":0.001956157,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":351,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:01.011763+0000","last_change":"2024-09-11T08:32:01.011725+0000","last_active":"2024-09-11T08:32:01.011763+0000","last_peered":"2024-09-11T08:32:01.011763+0000","last_clean":"2024-09-11T08:32:01.011763+0000","last_became_active":"2024-09-11T08:15:56.521570+0000","last_became_peered":"2024-09-11T08:15:56.521570+0000","last_unstale":"2024-09-11T08:32:01.011763+0000","last_undegraded":"2024-09-11T08:32:01.011763+0000","last_fullsized":"2024-09-11T08:32:01.011763+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:01.011667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:30:49.822760+0000","last_clean_scrub_stamp":"2024-09-11T08:32:01.011667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:09.110409+0000","scrub_duration":0.001390126,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":248,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:38.263553+0000","last_change":"2024-09-11T08:31:37.988996+0000","last_active":"2024-09-11T08:31:38.263553+0000","last_peered":"2024-09-11T08:31:38.263553+0000","last_clean":"2024-09-11T08:31:38.263553+0000","last_became_active":"2024-09-11T08:31:10.390449+0000","last_became_peered":"2024-09-11T08:31:10.390449+0000","last_unstale":"2024-09-11T08:31:38.263553+0000","last_undegraded":"2024-09-11T08:31:38.263553+0000","last_fullsized":"2024-09-11T08:31:38.263553+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:37.988934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:22:02.568761+0000","last_clean_scrub_stamp":"2024-09-11T08:31:37.988934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:01.150196+0000","scrub_duration":0.001198899,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":551,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:38.270751+0000","last_change":"2024-09-11T08:31:24.437447+0000","last_active":"2024-09-11T08:31:38.270751+0000","last_peered":"2024-09-11T08:31:38.270751+0000","last_clean":"2024-09-11T08:31:38.270751+0000","last_became_active":"2024-09-11T08:24:28.363959+0000","last_became_peered":"2024-09-11T08:24:28.363959+0000","last_unstale":"2024-09-11T08:31:38.270751+0000","last_undegraded":"2024-09-11T08:31:38.270751+0000","last_fullsized":"2024-09-11T08:31:38.270751+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:24.437358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:09.973943+0000","last_clean_scrub_stamp":"2024-09-11T08:31:24.437358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:39.841834+0000","scrub_duration":0.002335855,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"405'27","reported_seq":640,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:02.030706+0000","last_change":"2024-09-11T08:32:02.030662+0000","last_active":"2024-09-11T08:32:02.030706+0000","last_peered":"2024-09-11T08:32:02.030706+0000","last_clean":"2024-09-11T08:32:02.030706+0000","last_became_active":"2024-09-11T08:15:56.520058+0000","last_became_peered":"2024-09-11T08:15:56.520058+0000","last_unstale":"2024-09-11T08:32:02.030706+0000","last_undegraded":"2024-09-11T08:32:02.030706+0000","last_fullsized":"2024-09-11T08:32:02.030706+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'27","last_scrub_stamp":"2024-09-11T08:32:02.030601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:32:02.030601+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:29.366441+0000","scrub_duration":0.0021067550000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":304,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:19.897173+0000","last_change":"2024-09-11T08:32:19.897141+0000","last_active":"2024-09-11T08:32:19.897173+0000","last_peered":"2024-09-11T08:32:19.897173+0000","last_clean":"2024-09-11T08:32:19.897173+0000","last_became_active":"2024-09-11T08:31:38.276807+0000","last_became_peered":"2024-09-11T08:31:38.276807+0000","last_unstale":"2024-09-11T08:32:19.897173+0000","last_undegraded":"2024-09-11T08:32:19.897173+0000","last_fullsized":"2024-09-11T08:32:19.897173+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:19.897071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:32:19.897071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:32.724831+0000","scrub_duration":0.0014472969999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":361,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:14.880207+0000","last_change":"2024-09-11T08:32:14.880135+0000","last_active":"2024-09-11T08:32:14.880207+0000","last_peered":"2024-09-11T08:32:14.880207+0000","last_clean":"2024-09-11T08:32:14.880207+0000","last_became_active":"2024-09-11T08:15:56.523728+0000","last_became_peered":"2024-09-11T08:15:56.523728+0000","last_unstale":"2024-09-11T08:32:14.880207+0000","last_undegraded":"2024-09-11T08:32:14.880207+0000","last_fullsized":"2024-09-11T08:32:14.880207+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:14.880063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:32:14.880063+0000","last_clean_scrub_stamp":"2024-09-11T08:32:14.880063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:15.965568+0000","scrub_duration":0.001635262,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":532,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:38.263651+0000","last_change":"2024-09-11T08:31:25.137259+0000","last_active":"2024-09-11T08:31:38.263651+0000","last_peered":"2024-09-11T08:31:38.263651+0000","last_clean":"2024-09-11T08:31:38.263651+0000","last_became_active":"2024-09-11T08:07:36.633368+0000","last_became_peered":"2024-09-11T08:07:36.633368+0000","last_unstale":"2024-09-11T08:31:38.263651+0000","last_undegraded":"2024-09-11T08:31:38.263651+0000","last_fullsized":"2024-09-11T08:31:38.263651+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:25.137209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:22:50.992034+0000","last_clean_scrub_stamp":"2024-09-11T08:31:25.137209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:31.673186+0000","scrub_duration":0.0026728099999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"405'14","reported_seq":556,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:16.185373+0000","last_change":"2024-09-11T08:32:16.185348+0000","last_active":"2024-09-11T08:32:16.185373+0000","last_peered":"2024-09-11T08:32:16.185373+0000","last_clean":"2024-09-11T08:32:16.185373+0000","last_became_active":"2024-09-11T08:07:32.045728+0000","last_became_peered":"2024-09-11T08:07:32.045728+0000","last_unstale":"2024-09-11T08:32:16.185373+0000","last_undegraded":"2024-09-11T08:32:16.185373+0000","last_fullsized":"2024-09-11T08:32:16.185373+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'14","last_scrub_stamp":"2024-09-11T08:32:16.185293+0000","last_deep_scrub":"312'10","last_deep_scrub_stamp":"2024-09-11T08:21:55.859866+0000","last_clean_scrub_stamp":"2024-09-11T08:32:16.185293+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:24.334734+0000","scrub_duration":0.0020369009999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":564,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:38.270856+0000","last_change":"2024-09-11T08:31:34.332876+0000","last_active":"2024-09-11T08:31:38.270856+0000","last_peered":"2024-09-11T08:31:38.270856+0000","last_clean":"2024-09-11T08:31:38.270856+0000","last_became_active":"2024-09-11T08:24:28.357978+0000","last_became_peered":"2024-09-11T08:24:28.357978+0000","last_unstale":"2024-09-11T08:31:38.270856+0000","last_undegraded":"2024-09-11T08:31:38.270856+0000","last_fullsized":"2024-09-11T08:31:38.270856+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:34.332817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:31:34.332817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:40.628819+0000","scrub_duration":0.001888837,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":590,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:38.269777+0000","last_change":"2024-09-11T08:31:14.329296+0000","last_active":"2024-09-11T08:31:38.269777+0000","last_peered":"2024-09-11T08:31:38.269777+0000","last_clean":"2024-09-11T08:31:38.269777+0000","last_became_active":"2024-09-11T08:15:56.528607+0000","last_became_peered":"2024-09-11T08:15:56.528607+0000","last_unstale":"2024-09-11T08:31:38.269777+0000","last_undegraded":"2024-09-11T08:31:38.269777+0000","last_fullsized":"2024-09-11T08:31:38.269777+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:14.329233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:31:14.329233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:39.409202+0000","scrub_duration":0.001373134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":344,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.341207+0000","last_change":"2024-09-11T08:31:46.846926+0000","last_active":"2024-09-11T08:31:59.341207+0000","last_peered":"2024-09-11T08:31:59.341207+0000","last_clean":"2024-09-11T08:31:59.341207+0000","last_became_active":"2024-09-11T08:15:56.521488+0000","last_became_peered":"2024-09-11T08:15:56.521488+0000","last_unstale":"2024-09-11T08:31:59.341207+0000","last_undegraded":"2024-09-11T08:31:59.341207+0000","last_fullsized":"2024-09-11T08:31:59.341207+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:46.846860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:20:18.827709+0000","last_clean_scrub_stamp":"2024-09-11T08:31:46.846860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:03.351737+0000","scrub_duration":0.001543598,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":571,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:03.613827+0000","last_change":"2024-09-11T08:32:03.613777+0000","last_active":"2024-09-11T08:32:03.613827+0000","last_peered":"2024-09-11T08:32:03.613827+0000","last_clean":"2024-09-11T08:32:03.613827+0000","last_became_active":"2024-09-11T08:24:28.354141+0000","last_became_peered":"2024-09-11T08:24:28.354141+0000","last_unstale":"2024-09-11T08:32:03.613827+0000","last_undegraded":"2024-09-11T08:32:03.613827+0000","last_fullsized":"2024-09-11T08:32:03.613827+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:03.613679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:50.585566+0000","last_clean_scrub_stamp":"2024-09-11T08:32:03.613679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:27.208307+0000","scrub_duration":0.0019471779999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":262,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:48.930378+0000","last_change":"2024-09-11T08:31:48.930337+0000","last_active":"2024-09-11T08:31:48.930378+0000","last_peered":"2024-09-11T08:31:48.930378+0000","last_clean":"2024-09-11T08:31:48.930378+0000","last_became_active":"2024-09-11T08:31:10.390629+0000","last_became_peered":"2024-09-11T08:31:10.390629+0000","last_unstale":"2024-09-11T08:31:48.930378+0000","last_undegraded":"2024-09-11T08:31:48.930378+0000","last_fullsized":"2024-09-11T08:31:48.930378+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:48.930281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:31:48.930281+0000","last_clean_scrub_stamp":"2024-09-11T08:31:48.930281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:58.685685+0000","scrub_duration":0.00115854,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"405'32","reported_seq":664,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:38.270471+0000","last_change":"2024-09-11T08:31:34.302924+0000","last_active":"2024-09-11T08:31:38.270471+0000","last_peered":"2024-09-11T08:31:38.270471+0000","last_clean":"2024-09-11T08:31:38.270471+0000","last_became_active":"2024-09-11T08:24:28.360937+0000","last_became_peered":"2024-09-11T08:24:28.360937+0000","last_unstale":"2024-09-11T08:31:38.270471+0000","last_undegraded":"2024-09-11T08:31:38.270471+0000","last_fullsized":"2024-09-11T08:31:38.270471+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'32","last_scrub_stamp":"2024-09-11T08:31:34.302851+0000","last_deep_scrub":"405'32","last_deep_scrub_stamp":"2024-09-11T08:31:34.302851+0000","last_clean_scrub_stamp":"2024-09-11T08:31:34.302851+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:43.589103+0000","scrub_duration":0.0026836949999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":388,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:07.248642+0000","last_change":"2024-09-11T08:32:07.248608+0000","last_active":"2024-09-11T08:32:07.248642+0000","last_peered":"2024-09-11T08:32:07.248642+0000","last_clean":"2024-09-11T08:32:07.248642+0000","last_became_active":"2024-09-11T08:15:08.769133+0000","last_became_peered":"2024-09-11T08:15:08.769133+0000","last_unstale":"2024-09-11T08:32:07.248642+0000","last_undegraded":"2024-09-11T08:32:07.248642+0000","last_fullsized":"2024-09-11T08:32:07.248642+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:07.248540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:18:39.206016+0000","last_clean_scrub_stamp":"2024-09-11T08:32:07.248540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:08.720288+0000","scrub_duration":0.0015499400000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106,"ondisk_log_size":106,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1532,"num_read_kb":1289,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":4566,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3425,"internal_metadata":0},"log_size":1091,"ondisk_log_size":1091,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1083,"num_read_kb":1078,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":168,"seq":721554505946,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":920772,"kb_used_data":1400,"kb_used_omap":7,"kb_used_meta":919352,"kb_avail":92824380,"statfs":{"total":95995035648,"available":95052165120,"internally_reserved":0,"allocated":1433600,"data_stored":1012494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8027,"internal_metadata":941416613},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.85899999999999999,"15min":2.3610000000000002},"min":{"1min":0.33600000000000002,"5min":0.186,"15min":0.186},"max":{"1min":0.82199999999999995,"5min":5.6740000000000004,"15min":117.57599999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.82599999999999996,"15min":2.2349999999999999},"min":{"1min":0.35999999999999999,"5min":0.215,"15min":0.215},"max":{"1min":0.68400000000000005,"5min":4.3070000000000004,"15min":117.542},"last":0.61299999999999999}]},{"osd":1,"last update":"Wed Sep 11 08:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":1.075,"15min":1.52},"min":{"1min":0.33700000000000002,"5min":0.23000000000000001,"15min":0.20899999999999999},"max":{"1min":0.69299999999999995,"5min":24.347999999999999,"15min":66.971000000000004},"last":0.5},{"interface":"front","average":{"1min":0.57399999999999995,"5min":1.2450000000000001,"15min":1.4099999999999999},"min":{"1min":0.46600000000000003,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.76200000000000001,"5min":25.382000000000001,"15min":67.185000000000002},"last":0.59799999999999998}]},{"osd":2,"last update":"Wed Sep 11 08:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.91000000000000003,"15min":1.0069999999999999},"min":{"1min":0.36099999999999999,"5min":0.245,"15min":0.23799999999999999},"max":{"1min":0.68100000000000005,"5min":9.3439999999999994,"15min":19.097000000000001},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.0800000000000001,"15min":1.177},"min":{"1min":0.40899999999999997,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.72899999999999998,"5min":7.7699999999999996,"15min":21.594000000000001},"last":0.65400000000000003}]},{"osd":3,"last update":"Wed Sep 11 08:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":1.319,"15min":1.075},"min":{"1min":0.40200000000000002,"5min":0.27200000000000002,"15min":0.251},"max":{"1min":0.70899999999999996,"5min":24.315000000000001,"15min":24.315000000000001},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.58999999999999997,"5min":1.109,"15min":1.022},"min":{"1min":0.40799999999999997,"5min":0.26600000000000001,"15min":0.251},"max":{"1min":0.71599999999999997,"5min":24.864999999999998,"15min":24.864999999999998},"last":0.63900000000000001}]},{"osd":4,"last update":"Wed Sep 11 08:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.52300000000000002,"15min":0.79300000000000004},"min":{"1min":0.377,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.69699999999999995,"5min":3.5550000000000002,"15min":5.7240000000000002},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.53600000000000003,"15min":0.72599999999999998},"min":{"1min":0.34999999999999998,"5min":0.26200000000000001,"15min":0.23000000000000001},"max":{"1min":0.73099999999999998,"5min":2.5670000000000002,"15min":5.2649999999999997},"last":0.40500000000000003}]},{"osd":5,"last update":"Wed Sep 11 08:31:46 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.58799999999999997,"15min":1.101},"min":{"1min":0.32800000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.77000000000000002,"5min":6.3979999999999997,"15min":6.3979999999999997},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.60499999999999998,"15min":0.81699999999999995},"min":{"1min":0.313,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.78300000000000003,"5min":2.887,"15min":2.887},"last":0.48199999999999998}]},{"osd":6,"last update":"Wed Sep 11 08:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":1.177,"15min":0.94899999999999995},"min":{"1min":0.39600000000000002,"5min":0.26500000000000001,"15min":0.248},"max":{"1min":0.749,"5min":42.749000000000002,"15min":42.749000000000002},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":1.1719999999999999,"15min":1.044},"min":{"1min":0.42699999999999999,"5min":0.311,"15min":0.25700000000000001},"max":{"1min":0.73799999999999999,"5min":41.914000000000001,"15min":41.914000000000001},"last":0.42799999999999999}]}]},{"osd":6,"up_from":45,"seq":193273528628,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":901028,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":899646,"kb_avail":92844124,"statfs":{"total":95995035648,"available":95072382976,"internally_reserved":0,"allocated":1404928,"data_stored":1005602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":921237848},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:31:36 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.83599999999999997,"15min":3.0489999999999999},"min":{"1min":0.37,"5min":0.20399999999999999,"15min":0.17999999999999999},"max":{"1min":0.68400000000000005,"5min":3.7919999999999998,"15min":331.14800000000002},"last":0.46700000000000003},{"interface":"front","average":{"1min":1.044,"5min":1.028,"15min":2.948},"min":{"1min":0.38800000000000001,"5min":0.30199999999999999,"15min":0.20499999999999999},"max":{"1min":9.6519999999999992,"5min":9.6519999999999992,"15min":331.31},"last":0.44800000000000001}]},{"osd":1,"last update":"Wed Sep 11 08:31:36 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":1.3600000000000001,"15min":1.306},"min":{"1min":0.35199999999999998,"5min":0.218,"15min":0.218},"max":{"1min":0.72099999999999997,"5min":29.423999999999999,"15min":29.423999999999999},"last":0.40699999999999997},{"interface":"front","average":{"1min":1.016,"5min":1.5820000000000001,"15min":1.3100000000000001},"min":{"1min":0.38600000000000001,"5min":0.23499999999999999,"15min":0.22700000000000001},"max":{"1min":9.6679999999999993,"5min":29.109000000000002,"15min":29.109000000000002},"last":0.60999999999999999}]},{"osd":2,"last update":"Wed Sep 11 08:31:36 2024","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":1.407,"15min":1.2949999999999999},"min":{"1min":0.372,"5min":0.25,"15min":0.23200000000000001},"max":{"1min":9.7110000000000003,"5min":28.167999999999999,"15min":39.652999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.54100000000000004,"5min":1.2969999999999999,"15min":1.3320000000000001},"min":{"1min":0.40000000000000002,"5min":0.251,"15min":0.251},"max":{"1min":0.72299999999999998,"5min":31.648,"15min":38.390999999999998},"last":0.53700000000000003}]},{"osd":3,"last update":"Wed Sep 11 08:31:36 2024","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.345,"15min":1.2490000000000001},"min":{"1min":0.38900000000000001,"5min":0.30599999999999999,"15min":0.22700000000000001},"max":{"1min":9.6850000000000005,"5min":12.414,"15min":12.414},"last":0.52600000000000002},{"interface":"front","average":{"1min":1.071,"5min":1.5189999999999999,"15min":1.246},"min":{"1min":0.46800000000000003,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":9.7400000000000002,"5min":28.004999999999999,"15min":28.004999999999999},"last":0.58899999999999997}]},{"osd":4,"last update":"Wed Sep 11 08:31:36 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.65100000000000002,"15min":0.72399999999999998},"min":{"1min":0.28399999999999997,"5min":0.23400000000000001,"15min":0.20499999999999999},"max":{"1min":0.80600000000000005,"5min":3.8679999999999999,"15min":9.7560000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.67400000000000004,"15min":0.69299999999999995},"min":{"1min":0.35499999999999998,"5min":0.27600000000000002,"15min":0.23699999999999999},"max":{"1min":0.72099999999999997,"5min":3.5659999999999998,"15min":4.181},"last":0.503}]},{"osd":5,"last update":"Wed Sep 11 08:31:36 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.68600000000000005,"15min":1.3819999999999999},"min":{"1min":0.33500000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.78500000000000003,"5min":3.4129999999999998,"15min":9.7989999999999995},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.498,"5min":0.74399999999999999,"15min":1.256},"min":{"1min":0.317,"5min":0.246,"15min":0.246},"max":{"1min":0.61899999999999999,"5min":3.4620000000000002,"15min":7.2999999999999998},"last":0.42899999999999999}]},{"osd":7,"last update":"Wed Sep 11 08:31:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.80700000000000005,"15min":0.92600000000000005},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.253},"max":{"1min":0.73299999999999998,"5min":3.8450000000000002,"15min":10.167999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.82499999999999996,"15min":0.77400000000000002},"min":{"1min":0.33600000000000002,"5min":0.28799999999999998,"15min":0.25700000000000001},"max":{"1min":0.74299999999999999,"5min":3.6160000000000001,"15min":5.3579999999999997},"last":0.49099999999999999}]}]},{"osd":5,"up_from":353,"seq":1516123455585,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":364900,"kb_used_data":796,"kb_used_omap":4,"kb_used_meta":364091,"kb_avail":93380252,"statfs":{"total":95995035648,"available":95621378048,"internally_reserved":0,"allocated":815104,"data_stored":414795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":372829702},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.66800000000000004,"15min":1.4239999999999999},"min":{"1min":0.32700000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.66500000000000004,"5min":4.0199999999999996,"15min":19.379999999999999},"last":0.746},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.79700000000000004,"15min":1.446},"min":{"1min":0.312,"5min":0.22600000000000001,"15min":0.218},"max":{"1min":0.69099999999999995,"5min":5.8979999999999997,"15min":18.238},"last":0.44900000000000001}]},{"osd":1,"last update":"Wed Sep 11 08:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":2.6789999999999998,"15min":1.6000000000000001},"min":{"1min":0.314,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.70799999999999996,"5min":85.728999999999999,"15min":85.728999999999999},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.52900000000000003,"5min":2.8330000000000002,"15min":1.4019999999999999},"min":{"1min":0.35299999999999998,"5min":0.221,"15min":0.221},"max":{"1min":0.69999999999999996,"5min":85.668000000000006,"15min":85.668000000000006},"last":0.497}]},{"osd":2,"last update":"Wed Sep 11 08:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":1.111,"15min":1.702},"min":{"1min":0.379,"5min":0.25800000000000001,"15min":0.251},"max":{"1min":7.3609999999999998,"5min":11.262,"15min":18.350999999999999},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.98599999999999999,"15min":1.5609999999999999},"min":{"1min":0.39500000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.77800000000000002,"5min":11.363,"15min":18.381},"last":0.64200000000000002}]},{"osd":3,"last update":"Wed Sep 11 08:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":2.2120000000000002,"15min":1.4990000000000001},"min":{"1min":0.42799999999999999,"5min":0.251,"15min":0.251},"max":{"1min":0.72699999999999998,"5min":73.697000000000003,"15min":73.697000000000003},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.54800000000000004,"5min":2.0499999999999998,"15min":1.256},"min":{"1min":0.36299999999999999,"5min":0.29699999999999999,"15min":0.28699999999999998},"max":{"1min":0.73699999999999999,"5min":73.727999999999994,"15min":73.727999999999994},"last":0.52800000000000002}]},{"osd":4,"last update":"Wed Sep 11 08:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.58699999999999997,"15min":0.97299999999999998},"min":{"1min":0.26100000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.66100000000000003,"5min":5.8630000000000004,"15min":5.8630000000000004},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.50700000000000001,"15min":0.999},"min":{"1min":0.28999999999999998,"5min":0.252,"15min":0.252},"max":{"1min":0.72299999999999998,"5min":3.1349999999999998,"15min":5.8979999999999997},"last":0.66800000000000004}]},{"osd":6,"last update":"Wed Sep 11 08:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.58399999999999996,"15min":1.179},"min":{"1min":0.34799999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.76600000000000001,"5min":3.1040000000000001,"15min":5.9779999999999998},"last":0.63},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.55000000000000004,"15min":1.373},"min":{"1min":0.311,"5min":0.26000000000000001,"15min":0.23699999999999999},"max":{"1min":0.81200000000000006,"5min":3.7200000000000002,"15min":5.6680000000000001},"last":0.77900000000000003}]},{"osd":7,"last update":"Wed Sep 11 08:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.77700000000000002,"15min":1.05},"min":{"1min":0.32600000000000001,"5min":0.28999999999999998,"15min":0.28499999999999998},"max":{"1min":0.78600000000000003,"5min":7.8719999999999999,"15min":7.8719999999999999},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.69499999999999995,"15min":1.373},"min":{"1min":0.45900000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.76900000000000002,"5min":6.8739999999999997,"15min":6.8739999999999997},"last":0.69599999999999995}]}]},{"osd":4,"up_from":31,"seq":133143986495,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":894920,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":894078,"kb_avail":92850232,"statfs":{"total":95995035648,"available":95078637568,"internally_reserved":0,"allocated":847872,"data_stored":422824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":915536216},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:31:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":1,"15min":1.8700000000000001},"min":{"1min":0.32800000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.879,"5min":14.891,"15min":64.329999999999998},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.76600000000000001,"15min":1.863},"min":{"1min":0.40200000000000002,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.76900000000000002,"5min":7.2800000000000002,"15min":66.819999999999993},"last":0.45000000000000001}]},{"osd":1,"last update":"Wed Sep 11 08:31:49 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":2.2610000000000001,"15min":1.581},"min":{"1min":0.29299999999999998,"5min":0.249,"15min":0.22},"max":{"1min":0.79500000000000004,"5min":91.852000000000004,"15min":91.852000000000004},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.60999999999999999,"5min":2.2919999999999998,"15min":1.675},"min":{"1min":0.433,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.84299999999999997,"5min":91.697000000000003,"15min":91.697000000000003},"last":0.31}]},{"osd":2,"last update":"Wed Sep 11 08:31:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":1.0680000000000001,"15min":1.401},"min":{"1min":0.40400000000000003,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.80600000000000005,"5min":13.710000000000001,"15min":54.317999999999998},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.59699999999999998,"5min":1.2589999999999999,"15min":1.575},"min":{"1min":0.378,"5min":0.246,"15min":0.23400000000000001},"max":{"1min":0.81999999999999995,"5min":12.711,"15min":54.430999999999997},"last":0.31900000000000001}]},{"osd":3,"last update":"Wed Sep 11 08:31:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":1.129,"15min":1.2529999999999999},"min":{"1min":0.311,"5min":0.247,"15min":0.247},"max":{"1min":0.79200000000000004,"5min":12.811999999999999,"15min":30.321000000000002},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.278,"15min":1.135},"min":{"1min":0.371,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.73399999999999999,"5min":12.978,"15min":30.620000000000001},"last":0.46899999999999997}]},{"osd":5,"last update":"Wed Sep 11 08:31:41 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.56899999999999995,"15min":1.052},"min":{"1min":0.29799999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.76200000000000001,"5min":4.3410000000000002,"15min":5.1050000000000004},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.53000000000000003,"15min":1.1599999999999999},"min":{"1min":0.28799999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.68999999999999995,"5min":3.367,"15min":6.1150000000000002},"last":0.36399999999999999}]},{"osd":6,"last update":"Wed Sep 11 08:31:41 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.67000000000000004,"15min":0.72399999999999998},"min":{"1min":0.35099999999999998,"5min":0.247,"15min":0.23200000000000001},"max":{"1min":0.62,"5min":6.4039999999999999,"15min":6.4039999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.73299999999999998,"15min":0.78000000000000003},"min":{"1min":0.32700000000000001,"5min":0.26000000000000001,"15min":0.251},"max":{"1min":0.79100000000000004,"5min":4.2140000000000004,"15min":5.3159999999999998},"last":0.438}]},{"osd":7,"last update":"Wed Sep 11 08:32:07 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.57799999999999996,"15min":0.70499999999999996},"min":{"1min":0.33500000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.72399999999999998,"5min":3.339,"15min":6.093},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.51900000000000002,"15min":0.67800000000000005},"min":{"1min":0.311,"5min":0.29899999999999999,"15min":0.26800000000000002},"max":{"1min":0.74399999999999999,"5min":2.6030000000000002,"15min":4.8259999999999996},"last":0.64700000000000002}]}]},{"osd":3,"up_from":25,"seq":107374182723,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":902020,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":901182,"kb_avail":92843132,"statfs":{"total":95995035648,"available":95071367168,"internally_reserved":0,"allocated":847872,"data_stored":415580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":922810712},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:31:41 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":1.2589999999999999,"15min":1.5069999999999999},"min":{"1min":0.23899999999999999,"5min":0.216,"15min":0.14999999999999999},"max":{"1min":1.153,"5min":8.9930000000000003,"15min":58.880000000000003},"last":0.308},{"interface":"front","average":{"1min":0.35199999999999998,"5min":1.3029999999999999,"15min":1.4530000000000001},"min":{"1min":0.20499999999999999,"5min":0.20100000000000001,"15min":0.184},"max":{"1min":0.57099999999999995,"5min":9.5210000000000008,"15min":59.246000000000002},"last":0.29499999999999998}]},{"osd":1,"last update":"Wed Sep 11 08:31:41 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":1.9239999999999999,"15min":1.504},"min":{"1min":0.28000000000000003,"5min":0.26500000000000001,"15min":0.19600000000000001},"max":{"1min":1.214,"5min":54.859000000000002,"15min":54.859000000000002},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.47099999999999997,"5min":1.887,"15min":1.5649999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.19700000000000001},"max":{"1min":2.2589999999999999,"5min":55.773000000000003,"15min":55.773000000000003},"last":0.27900000000000003}]},{"osd":2,"last update":"Wed Sep 11 08:31:41 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":1.268,"15min":1.429},"min":{"1min":0.28100000000000003,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.3080000000000001,"5min":11.459,"15min":67.503},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.42699999999999999,"5min":1.534,"15min":1.4750000000000001},"min":{"1min":0.25,"5min":0.23799999999999999,"15min":0.19900000000000001},"max":{"1min":1.1850000000000001,"5min":11.516,"15min":67.313999999999993},"last":0.36799999999999999}]},{"osd":4,"last update":"Wed Sep 11 08:31:53 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.98599999999999999,"15min":1.1439999999999999},"min":{"1min":0.502,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":1.383,"5min":6.8680000000000003,"15min":11.802},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.64900000000000002,"5min":1.139,"15min":1.014},"min":{"1min":0.42299999999999999,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":2.1190000000000002,"5min":6.46,"15min":6.46},"last":0.46700000000000003}]},{"osd":5,"last update":"Wed Sep 11 08:31:41 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":1.113,"15min":1.512},"min":{"1min":0.39600000000000002,"5min":0.34899999999999998,"15min":0.32700000000000001},"max":{"1min":1.121,"5min":6.6189999999999998,"15min":7.3849999999999998},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.64900000000000002,"5min":1.0609999999999999,"15min":1.137},"min":{"1min":0.46200000000000002,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.421,"5min":6.835,"15min":6.835},"last":0.53900000000000003}]},{"osd":6,"last update":"Wed Sep 11 08:31:41 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":1.236,"15min":1.2310000000000001},"min":{"1min":0.40200000000000002,"5min":0.39100000000000001,"15min":0.29699999999999999},"max":{"1min":2.3180000000000001,"5min":6.6520000000000001,"15min":10.949999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.60699999999999998,"5min":1.0680000000000001,"15min":1.2470000000000001},"min":{"1min":0.39000000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":1.2509999999999999,"5min":6.8970000000000002,"15min":11.837},"last":0.56699999999999995}]},{"osd":7,"last update":"Wed Sep 11 08:31:58 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":1.266,"15min":1.2889999999999999},"min":{"1min":0.42099999999999999,"5min":0.35499999999999998,"15min":0.29899999999999999},"max":{"1min":1.1679999999999999,"5min":6.6840000000000002,"15min":31.071000000000002},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.70299999999999996,"5min":1.3149999999999999,"15min":1.45},"min":{"1min":0.47199999999999998,"5min":0.35899999999999999,"15min":0.33800000000000002},"max":{"1min":1.784,"5min":6.899,"15min":58.820999999999998},"last":0.502}]}]},{"osd":2,"up_from":18,"seq":77309411654,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":889956,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":889150,"kb_avail":92855196,"statfs":{"total":95995035648,"available":95083720704,"internally_reserved":0,"allocated":815104,"data_stored":414707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":910489944},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:32:11 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":2.4369999999999998,"15min":2.7810000000000001},"min":{"1min":0.23000000000000001,"5min":0.20100000000000001,"15min":0.17399999999999999},"max":{"1min":0.55500000000000005,"5min":102.18000000000001,"15min":102.18000000000001},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.85899999999999999,"15min":2.5350000000000001},"min":{"1min":0.247,"5min":0.23200000000000001,"15min":0.17999999999999999},"max":{"1min":0.50700000000000001,"5min":5.9349999999999996,"15min":93.790999999999997},"last":0.218}]},{"osd":1,"last update":"Wed Sep 11 08:32:11 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":2.5489999999999999,"15min":1.9039999999999999},"min":{"1min":0.26400000000000001,"5min":0.20399999999999999,"15min":0.19800000000000001},"max":{"1min":0.68899999999999995,"5min":102.134,"15min":102.134},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.36599999999999999,"5min":1.073,"15min":1.3520000000000001},"min":{"1min":0.28399999999999997,"5min":0.251,"15min":0.20399999999999999},"max":{"1min":0.57999999999999996,"5min":6.7220000000000004,"15min":53.082999999999998},"last":0.27500000000000002}]},{"osd":3,"last update":"Wed Sep 11 08:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":3.0990000000000002,"15min":1.9199999999999999},"min":{"1min":0.29399999999999998,"5min":0.23699999999999999,"15min":0.22},"max":{"1min":0.54900000000000004,"5min":102.206,"15min":102.206},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.38500000000000001,"5min":2.8839999999999999,"15min":1.8520000000000001},"min":{"1min":0.311,"5min":0.251,"15min":0.21199999999999999},"max":{"1min":0.57399999999999995,"5min":104.768,"15min":104.768},"last":0.30499999999999999}]},{"osd":4,"last update":"Wed Sep 11 08:31:51 2024","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.1930000000000001,"15min":1.071},"min":{"1min":0.41799999999999998,"5min":0.25,"15min":0.24099999999999999},"max":{"1min":14.539,"5min":14.539,"15min":14.539},"last":0.48499999999999999},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.0149999999999999,"15min":1.0660000000000001},"min":{"1min":0.39900000000000002,"5min":0.36199999999999999,"15min":0.27500000000000002},"max":{"1min":14.554,"5min":14.554,"15min":14.554},"last":0.64900000000000002}]},{"osd":5,"last update":"Wed Sep 11 08:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.86499999999999999,"15min":1.377},"min":{"1min":0.44800000000000001,"5min":0.32100000000000001,"15min":0.27700000000000002},"max":{"1min":0.80400000000000005,"5min":4.5839999999999996,"15min":4.5839999999999996},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.99299999999999999,"15min":1.2569999999999999},"min":{"1min":0.39600000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.746,"5min":6.6589999999999998,"15min":6.6589999999999998},"last":0.59199999999999997}]},{"osd":6,"last update":"Wed Sep 11 08:31:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":1.2849999999999999,"15min":1.175},"min":{"1min":0.45100000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.73499999999999999,"5min":6.3440000000000003,"15min":8.0350000000000001},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.97699999999999998,"15min":1.085},"min":{"1min":0.42999999999999999,"5min":0.377,"15min":0.32500000000000001},"max":{"1min":0.68899999999999995,"5min":5.9619999999999997,"15min":7.8710000000000004},"last":0.56200000000000006}]},{"osd":7,"last update":"Wed Sep 11 08:31:51 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":1.25,"15min":1.256},"min":{"1min":0.46000000000000002,"5min":0.315,"15min":0.27500000000000002},"max":{"1min":0.79700000000000004,"5min":11.279,"15min":35.274999999999999},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.60499999999999998,"5min":1.1659999999999999,"15min":1.3220000000000001},"min":{"1min":0.434,"5min":0.34999999999999998,"15min":0.32800000000000001},"max":{"1min":0.84199999999999997,"5min":11.305,"15min":35.265000000000001},"last":0.624}]}]},{"osd":1,"up_from":13,"seq":55834575180,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":891080,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":890238,"kb_avail":92854072,"statfs":{"total":95995035648,"available":95082569728,"internally_reserved":0,"allocated":851968,"data_stored":415723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":911604056},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:31:51 2024","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":1.1579999999999999,"15min":1.4259999999999999},"min":{"1min":0.218,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":0.52300000000000002,"5min":14.045,"15min":63.932000000000002},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.34200000000000003,"5min":1.327,"15min":1.665},"min":{"1min":0.254,"5min":0.182,"15min":0.16800000000000001},"max":{"1min":0.54100000000000004,"5min":14.127000000000001,"15min":64.269000000000005},"last":0.27200000000000002}]},{"osd":2,"last update":"Wed Sep 11 08:31:27 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":1.718,"15min":1.4339999999999999},"min":{"1min":0.27200000000000002,"5min":0.19700000000000001,"15min":0.189},"max":{"1min":0.56599999999999995,"5min":13.928000000000001,"15min":20.925999999999998},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.34499999999999997,"5min":1.9099999999999999,"15min":1.462},"min":{"1min":0.24199999999999999,"5min":0.19900000000000001,"15min":0.19},"max":{"1min":0.49199999999999999,"5min":16.099,"15min":20.812000000000001},"last":0.36899999999999999}]},{"osd":3,"last update":"Wed Sep 11 08:31:42 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":1.6830000000000001,"15min":1.4219999999999999},"min":{"1min":0.215,"5min":0.21199999999999999,"15min":0.19700000000000001},"max":{"1min":0.504,"5min":16.193999999999999,"15min":16.193999999999999},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.40699999999999997,"5min":1.5549999999999999,"15min":1.4390000000000001},"min":{"1min":0.26900000000000002,"5min":0.23499999999999999,"15min":0.22800000000000001},"max":{"1min":0.627,"5min":16.033000000000001,"15min":16.033000000000001},"last":0.40699999999999997}]},{"osd":4,"last update":"Wed Sep 11 08:31:57 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":1.387,"15min":1.79},"min":{"1min":0.38500000000000001,"5min":0.28100000000000003,"15min":0.254},"max":{"1min":0.76100000000000001,"5min":16.178000000000001,"15min":110.86499999999999},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":1.4490000000000001,"15min":1.913},"min":{"1min":0.42699999999999999,"5min":0.28799999999999998,"15min":0.28000000000000003},"max":{"1min":0.72399999999999998,"5min":19.009,"15min":110.831},"last":0.67900000000000005}]},{"osd":5,"last update":"Wed Sep 11 08:31:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":1.302,"15min":1.4650000000000001},"min":{"1min":0.442,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.80400000000000005,"5min":16.012,"15min":16.012},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.59799999999999998,"5min":1.423,"15min":1.486},"min":{"1min":0.42399999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.93400000000000005,"5min":16.206,"15min":16.206},"last":0.66300000000000003}]},{"osd":6,"last update":"Wed Sep 11 08:31:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":1.3999999999999999,"15min":1.369},"min":{"1min":0.41299999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.78100000000000003,"5min":16.143000000000001,"15min":16.143000000000001},"last":0.69599999999999995},{"interface":"front","average":{"1min":1.1080000000000001,"5min":1.4990000000000001,"15min":1.421},"min":{"1min":0.46500000000000002,"5min":0.34799999999999998,"15min":0.29799999999999999},"max":{"1min":10.552,"5min":15.984,"15min":15.984},"last":0.624}]},{"osd":7,"last update":"Wed Sep 11 08:31:57 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":1.5449999999999999,"15min":1.718},"min":{"1min":0.40400000000000003,"5min":0.32500000000000001,"15min":0.28599999999999998},"max":{"1min":0.82599999999999996,"5min":16.228999999999999,"15min":109.149},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.59599999999999997,"5min":1.321,"15min":1.722},"min":{"1min":0.47399999999999998,"5min":0.32800000000000001,"15min":0.318},"max":{"1min":0.80100000000000005,"5min":16.170999999999999,"15min":109.309},"last":0.60199999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738704,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":899648,"kb_used_data":1400,"kb_used_omap":1,"kb_used_meta":898238,"kb_avail":92845504,"statfs":{"total":95995035648,"available":95073796096,"internally_reserved":0,"allocated":1433600,"data_stored":1006725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":919796056},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 11 08:31:41 2024","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":3.1419999999999999,"15min":1.7629999999999999},"min":{"1min":0.215,"5min":0.20399999999999999,"15min":0.159},"max":{"1min":0.69099999999999995,"5min":98.373999999999995,"15min":98.373999999999995},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.34599999999999997,"5min":3.484,"15min":1.9930000000000001},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.16600000000000001},"max":{"1min":0.72299999999999998,"5min":98.257999999999996,"15min":98.257999999999996},"last":0.39100000000000001}]},{"osd":2,"last update":"Wed Sep 11 08:32:12 2024","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":1.2769999999999999,"15min":1.383},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.44600000000000001,"5min":7.5270000000000001,"15min":46.417000000000002},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.32500000000000001,"5min":1.3759999999999999,"15min":1.466},"min":{"1min":0.221,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.40999999999999998,"5min":13.331,"15min":46.438000000000002},"last":0.378}]},{"osd":3,"last update":"Wed Sep 11 08:31:33 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":1.4199999999999999,"15min":1.839},"min":{"1min":0.24399999999999999,"5min":0.222,"15min":0.222},"max":{"1min":0.47599999999999998,"5min":9.5180000000000007,"15min":172.149},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.34599999999999997,"5min":1.413,"15min":1.742},"min":{"1min":0.23300000000000001,"5min":0.22,"15min":0.20300000000000001},"max":{"1min":0.60899999999999999,"5min":9.0129999999999999,"15min":174.899},"last":0.35699999999999998}]},{"osd":4,"last update":"Wed Sep 11 08:31:57 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":1.0549999999999999,"15min":1.079},"min":{"1min":0.41899999999999998,"5min":0.26600000000000001,"15min":0.252},"max":{"1min":0.67800000000000005,"5min":5.1609999999999996,"15min":9.6560000000000006},"last":0.56599999999999995},{"interface":"front","average":{"1min":1.3069999999999999,"5min":1.113,"15min":1.1539999999999999},"min":{"1min":0.32500000000000001,"5min":0.29299999999999998,"15min":0.26400000000000001},"max":{"1min":13.666,"5min":13.666,"15min":13.666},"last":0.64500000000000002}]},{"osd":5,"last update":"Wed Sep 11 08:31:41 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":1.024,"15min":1.613},"min":{"1min":0.39100000000000001,"5min":0.28499999999999998,"15min":0.27400000000000002},"max":{"1min":0.66000000000000003,"5min":7.7030000000000003,"15min":7.7030000000000003},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.97199999999999998,"15min":1.484},"min":{"1min":0.376,"5min":0.311,"15min":0.309},"max":{"1min":0.755,"5min":5.3730000000000002,"15min":7.718},"last":0.55400000000000005}]},{"osd":6,"last update":"Wed Sep 11 08:31:41 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":1.1240000000000001,"15min":1.1359999999999999},"min":{"1min":0.39600000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.64200000000000002,"5min":5.1230000000000002,"15min":9.7029999999999994},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.99199999999999999,"15min":1.169},"min":{"1min":0.40899999999999997,"5min":0.315,"15min":0.28299999999999997},"max":{"1min":0.70199999999999996,"5min":5.3419999999999996,"15min":7.4820000000000002},"last":0.53300000000000003}]},{"osd":7,"last update":"Wed Sep 11 08:31:57 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":1.089,"15min":1.1299999999999999},"min":{"1min":0.36899999999999999,"5min":0.34499999999999997,"15min":0.30399999999999999},"max":{"1min":0.73199999999999998,"5min":5.8049999999999997,"15min":7.5350000000000001},"last":0.505},{"interface":"front","average":{"1min":0.58199999999999996,"5min":1.0549999999999999,"15min":1.2350000000000001},"min":{"1min":0.378,"5min":0.378,"15min":0.32900000000000001},"max":{"1min":0.73999999999999999,"5min":5.6829999999999998,"15min":13.914},"last":0.57499999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3425,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T08:32:24.227 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T08:32:24.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:23 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:32:24.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:23 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:32:24.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:23 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:32:24.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:23 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:32:24.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:23 smithi181 ceph-mon[32701]: 3.e scrub starts 2024-09-11T08:32:24.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:23 smithi181 ceph-mon[32701]: 3.e scrub ok 2024-09-11T08:32:24.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:23 smithi181 ceph-mon[32701]: pgmap v1137: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-11T08:32:24.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:23 smithi181 ceph-mon[32701]: from='client.26285 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:32:24.473 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:32:25.142 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:24 smithi181 ceph-mon[32701]: osdmap e480: 8 total, 8 up, 8 in 2024-09-11T08:32:25.143 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:24 smithi181 ceph-mon[32701]: 4.18 scrub starts 2024-09-11T08:32:25.143 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:24 smithi181 ceph-mon[32701]: 4.18 scrub ok 2024-09-11T08:32:25.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:24 smithi150 ceph-mon[29276]: osdmap e480: 8 total, 8 up, 8 in 2024-09-11T08:32:25.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:24 smithi150 ceph-mon[29276]: 4.18 scrub starts 2024-09-11T08:32:25.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:24 smithi150 ceph-mon[29276]: 4.18 scrub ok 2024-09-11T08:32:25.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:24 smithi150 ceph-mon[39612]: osdmap e480: 8 total, 8 up, 8 in 2024-09-11T08:32:25.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:24 smithi150 ceph-mon[39612]: 4.18 scrub starts 2024-09-11T08:32:25.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:24 smithi150 ceph-mon[39612]: 4.18 scrub ok 2024-09-11T08:32:25.573 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:32:25 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:32:25] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:32:25.953 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:25 smithi150 ceph-mon[29276]: 4.1e scrub starts 2024-09-11T08:32:25.954 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:25 smithi150 ceph-mon[29276]: 4.1e scrub ok 2024-09-11T08:32:25.954 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:25 smithi150 ceph-mon[29276]: 3.11 scrub starts 2024-09-11T08:32:25.954 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:25 smithi150 ceph-mon[29276]: 3.11 scrub ok 2024-09-11T08:32:25.954 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:25 smithi150 ceph-mon[29276]: pgmap v1139: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:32:25.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:25 smithi150 ceph-mon[39612]: 4.1e scrub starts 2024-09-11T08:32:25.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:25 smithi150 ceph-mon[39612]: 4.1e scrub ok 2024-09-11T08:32:25.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:25 smithi150 ceph-mon[39612]: 3.11 scrub starts 2024-09-11T08:32:25.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:25 smithi150 ceph-mon[39612]: 3.11 scrub ok 2024-09-11T08:32:25.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:25 smithi150 ceph-mon[39612]: pgmap v1139: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:32:26.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:25 smithi181 ceph-mon[32701]: 4.1e scrub starts 2024-09-11T08:32:26.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:25 smithi181 ceph-mon[32701]: 4.1e scrub ok 2024-09-11T08:32:26.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:25 smithi181 ceph-mon[32701]: 3.11 scrub starts 2024-09-11T08:32:26.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:25 smithi181 ceph-mon[32701]: 3.11 scrub ok 2024-09-11T08:32:26.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:25 smithi181 ceph-mon[32701]: pgmap v1139: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:32:26.349 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-11T08:32:26.351 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-09-11T08:32:26.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:26 smithi150 ceph-mon[29276]: 6.c scrub starts 2024-09-11T08:32:26.988 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:26 smithi150 ceph-mon[29276]: 6.c scrub ok 2024-09-11T08:32:26.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:26 smithi150 ceph-mon[29276]: 6.18 scrub starts 2024-09-11T08:32:26.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:26 smithi150 ceph-mon[29276]: 6.18 scrub ok 2024-09-11T08:32:26.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:26 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:32:26.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:26 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:32:26.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:26 smithi150 ceph-mon[29276]: osdmap e481: 8 total, 8 up, 8 in 2024-09-11T08:32:26.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:26 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:32:26.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:26 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:32:26.989 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:26 smithi150 ceph-mon[29276]: from='client.16401 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:32:26.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:26 smithi150 ceph-mon[39612]: 6.c scrub starts 2024-09-11T08:32:26.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:26 smithi150 ceph-mon[39612]: 6.c scrub ok 2024-09-11T08:32:26.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:26 smithi150 ceph-mon[39612]: 6.18 scrub starts 2024-09-11T08:32:26.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:26 smithi150 ceph-mon[39612]: 6.18 scrub ok 2024-09-11T08:32:26.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:26 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:32:26.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:26 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:32:26.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:26 smithi150 ceph-mon[39612]: osdmap e481: 8 total, 8 up, 8 in 2024-09-11T08:32:26.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:26 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:32:26.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:26 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:32:26.990 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:26 smithi150 ceph-mon[39612]: from='client.16401 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:32:27.084 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":1140,"stamp":"2024-09-11T08:32:25.864353+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596188,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":2741,"num_read_kb":2600,"num_write":1115,"num_write_kb":1671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":1776519,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1297,"ondisk_log_size":1297,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6664416,"kb_used_data":8324,"kb_used_omap":22,"kb_used_meta":6655977,"kb_avail":743296800,"statfs":{"total":767960285184,"available":761135923200,"internally_reserved":0,"allocated":8523776,"data_stored":5148790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22853,"internal_metadata":6815721147},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.592900"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":566,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-11T08:32:22.037080+0000","last_change":"2024-09-11T08:31:36.361555+0000","last_active":"2024-09-11T08:32:22.037080+0000","last_peered":"2024-09-11T08:32:22.037080+0000","last_clean":"2024-09-11T08:32:22.037080+0000","last_became_active":"2024-09-11T08:24:28.358401+0000","last_became_peered":"2024-09-11T08:24:28.358401+0000","last_unstale":"2024-09-11T08:32:22.037080+0000","last_undegraded":"2024-09-11T08:32:22.037080+0000","last_fullsized":"2024-09-11T08:32:22.037080+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:36.361487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:31:36.361487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:52.481796+0000","scrub_duration":0.0020598159999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"405'25","reported_seq":578,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.487694+0000","last_change":"2024-09-11T08:31:12.599946+0000","last_active":"2024-09-11T08:31:59.487694+0000","last_peered":"2024-09-11T08:31:59.487694+0000","last_clean":"2024-09-11T08:31:59.487694+0000","last_became_active":"2024-09-11T08:07:32.048048+0000","last_became_peered":"2024-09-11T08:07:32.048048+0000","last_unstale":"2024-09-11T08:31:59.487694+0000","last_undegraded":"2024-09-11T08:31:59.487694+0000","last_fullsized":"2024-09-11T08:31:59.487694+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'25","last_scrub_stamp":"2024-09-11T08:31:12.599879+0000","last_deep_scrub":"312'19","last_deep_scrub_stamp":"2024-09-11T08:21:59.122122+0000","last_clean_scrub_stamp":"2024-09-11T08:31:12.599879+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:23.475718+0000","scrub_duration":0.0035721339999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":612,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-11T08:32:22.801559+0000","last_change":"2024-09-11T08:31:37.671883+0000","last_active":"2024-09-11T08:32:22.801559+0000","last_peered":"2024-09-11T08:32:22.801559+0000","last_clean":"2024-09-11T08:32:22.801559+0000","last_became_active":"2024-09-11T08:24:28.372160+0000","last_became_peered":"2024-09-11T08:24:28.372160+0000","last_unstale":"2024-09-11T08:32:22.801559+0000","last_undegraded":"2024-09-11T08:32:22.801559+0000","last_fullsized":"2024-09-11T08:32:22.801559+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:37.671827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:59.531202+0000","last_clean_scrub_stamp":"2024-09-11T08:31:37.671827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:07.023040+0000","scrub_duration":0.0018397439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":396,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:04.497516+0000","last_change":"2024-09-11T08:32:04.497472+0000","last_active":"2024-09-11T08:32:04.497516+0000","last_peered":"2024-09-11T08:32:04.497516+0000","last_clean":"2024-09-11T08:32:04.497516+0000","last_became_active":"2024-09-11T08:31:16.964358+0000","last_became_peered":"2024-09-11T08:31:16.964358+0000","last_unstale":"2024-09-11T08:32:04.497516+0000","last_undegraded":"2024-09-11T08:32:04.497516+0000","last_fullsized":"2024-09-11T08:32:04.497516+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:04.497396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:02.470649+0000","last_clean_scrub_stamp":"2024-09-11T08:32:04.497396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:33.896616+0000","scrub_duration":0.0015759000000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":550,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.340544+0000","last_change":"2024-09-11T08:31:00.004778+0000","last_active":"2024-09-11T08:31:59.340544+0000","last_peered":"2024-09-11T08:31:59.340544+0000","last_clean":"2024-09-11T08:31:59.340544+0000","last_became_active":"2024-09-11T08:15:56.521104+0000","last_became_peered":"2024-09-11T08:15:56.521104+0000","last_unstale":"2024-09-11T08:31:59.340544+0000","last_undegraded":"2024-09-11T08:31:59.340544+0000","last_fullsized":"2024-09-11T08:31:59.340544+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:00.004724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:31:00.004724+0000","last_clean_scrub_stamp":"2024-09-11T08:31:00.004724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:28.957317+0000","scrub_duration":0.0035427290000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":253,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.341247+0000","last_change":"2024-09-11T08:31:23.119557+0000","last_active":"2024-09-11T08:31:59.341247+0000","last_peered":"2024-09-11T08:31:59.341247+0000","last_clean":"2024-09-11T08:31:59.341247+0000","last_became_active":"2024-09-11T08:31:09.935019+0000","last_became_peered":"2024-09-11T08:31:09.935019+0000","last_unstale":"2024-09-11T08:31:59.341247+0000","last_undegraded":"2024-09-11T08:31:59.341247+0000","last_fullsized":"2024-09-11T08:31:59.341247+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:23.119475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:31:23.119475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:25.064663+0000","scrub_duration":0.0035696360000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"405'46","reported_seq":452,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:03.000131+0000","last_change":"2024-09-11T08:32:03.000087+0000","last_active":"2024-09-11T08:32:03.000131+0000","last_peered":"2024-09-11T08:32:03.000131+0000","last_clean":"2024-09-11T08:32:03.000131+0000","last_became_active":"2024-09-11T08:31:09.932624+0000","last_became_peered":"2024-09-11T08:31:09.932624+0000","last_unstale":"2024-09-11T08:32:03.000131+0000","last_undegraded":"2024-09-11T08:32:03.000131+0000","last_fullsized":"2024-09-11T08:32:03.000131+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'46","last_scrub_stamp":"2024-09-11T08:32:03.000027+0000","last_deep_scrub":"145'22","last_deep_scrub_stamp":"2024-09-11T08:15:57.478870+0000","last_clean_scrub_stamp":"2024-09-11T08:32:03.000027+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:05.275638+0000","scrub_duration":0.0018405229999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":365,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:00.451296+0000","last_change":"2024-09-11T08:32:00.451257+0000","last_active":"2024-09-11T08:32:00.451296+0000","last_peered":"2024-09-11T08:32:00.451296+0000","last_clean":"2024-09-11T08:32:00.451296+0000","last_became_active":"2024-09-11T08:31:16.962706+0000","last_became_peered":"2024-09-11T08:31:16.962706+0000","last_unstale":"2024-09-11T08:32:00.451296+0000","last_undegraded":"2024-09-11T08:32:00.451296+0000","last_fullsized":"2024-09-11T08:32:00.451296+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:00.451200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:18:36.998503+0000","last_clean_scrub_stamp":"2024-09-11T08:32:00.451200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:23.397534+0000","scrub_duration":0.0013099940000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":110,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:49.882347+0000","last_change":"2024-09-11T08:31:49.882290+0000","last_active":"2024-09-11T08:31:49.882347+0000","last_peered":"2024-09-11T08:31:49.882347+0000","last_clean":"2024-09-11T08:31:49.882347+0000","last_became_active":"2024-09-11T08:31:38.278622+0000","last_became_peered":"2024-09-11T08:31:38.278622+0000","last_unstale":"2024-09-11T08:31:49.882347+0000","last_undegraded":"2024-09-11T08:31:49.882347+0000","last_fullsized":"2024-09-11T08:31:49.882347+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:49.882227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:54.878409+0000","last_clean_scrub_stamp":"2024-09-11T08:31:49.882227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:50.427634+0000","scrub_duration":0.0012820970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"468'25","reported_seq":385,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.488181+0000","last_change":"2024-09-11T08:31:18.549005+0000","last_active":"2024-09-11T08:31:59.488181+0000","last_peered":"2024-09-11T08:31:59.488181+0000","last_clean":"2024-09-11T08:31:59.488181+0000","last_became_active":"2024-09-11T08:31:16.962868+0000","last_became_peered":"2024-09-11T08:31:16.962868+0000","last_unstale":"2024-09-11T08:31:59.488181+0000","last_undegraded":"2024-09-11T08:31:59.488181+0000","last_fullsized":"2024-09-11T08:31:59.488181+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"468'25","last_scrub_stamp":"2024-09-11T08:31:18.548955+0000","last_deep_scrub":"468'25","last_deep_scrub_stamp":"2024-09-11T08:31:18.548955+0000","last_clean_scrub_stamp":"2024-09-11T08:31:18.548955+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:46.783158+0000","scrub_duration":0.0031821060000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":300,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:11.855812+0000","last_change":"2024-09-11T08:32:11.855778+0000","last_active":"2024-09-11T08:32:11.855812+0000","last_peered":"2024-09-11T08:32:11.855812+0000","last_clean":"2024-09-11T08:32:11.855812+0000","last_became_active":"2024-09-11T08:31:38.274688+0000","last_became_peered":"2024-09-11T08:31:38.274688+0000","last_unstale":"2024-09-11T08:32:11.855812+0000","last_undegraded":"2024-09-11T08:32:11.855812+0000","last_fullsized":"2024-09-11T08:32:11.855812+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:11.855710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:20:18.900482+0000","last_clean_scrub_stamp":"2024-09-11T08:32:11.855710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:17.685158+0000","scrub_duration":0.0016534309999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":530,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-11T08:32:22.036447+0000","last_change":"2024-09-11T08:30:53.446937+0000","last_active":"2024-09-11T08:32:22.036447+0000","last_peered":"2024-09-11T08:32:22.036447+0000","last_clean":"2024-09-11T08:32:22.036447+0000","last_became_active":"2024-09-11T08:14:24.792595+0000","last_became_peered":"2024-09-11T08:14:24.792595+0000","last_unstale":"2024-09-11T08:32:22.036447+0000","last_undegraded":"2024-09-11T08:32:22.036447+0000","last_fullsized":"2024-09-11T08:32:22.036447+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:53.446882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:30:53.446882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016646390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":100,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:43.002807+0000","last_change":"2024-09-11T08:31:43.002757+0000","last_active":"2024-09-11T08:31:43.002807+0000","last_peered":"2024-09-11T08:31:43.002807+0000","last_clean":"2024-09-11T08:31:43.002807+0000","last_became_active":"2024-09-11T08:31:38.274258+0000","last_became_peered":"2024-09-11T08:31:38.274258+0000","last_unstale":"2024-09-11T08:31:43.002807+0000","last_undegraded":"2024-09-11T08:31:43.002807+0000","last_fullsized":"2024-09-11T08:31:43.002807+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:43.002690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:31:43.002690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:46.571260+0000","scrub_duration":0.001676507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"405'11","reported_seq":365,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.488847+0000","last_change":"2024-09-11T08:31:59.488753+0000","last_active":"2024-09-11T08:31:59.488847+0000","last_peered":"2024-09-11T08:31:59.488847+0000","last_clean":"2024-09-11T08:31:59.488847+0000","last_became_active":"2024-09-11T08:31:16.962705+0000","last_became_peered":"2024-09-11T08:31:16.962705+0000","last_unstale":"2024-09-11T08:31:59.488847+0000","last_undegraded":"2024-09-11T08:31:59.488847+0000","last_fullsized":"2024-09-11T08:31:59.488847+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'11","last_scrub_stamp":"2024-09-11T08:31:59.488651+0000","last_deep_scrub":"139'7","last_deep_scrub_stamp":"2024-09-11T08:12:24.869318+0000","last_clean_scrub_stamp":"2024-09-11T08:31:59.488651+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:08.361074+0000","scrub_duration":0.0025657850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":351,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.340734+0000","last_change":"2024-09-11T08:31:28.667899+0000","last_active":"2024-09-11T08:31:59.340734+0000","last_peered":"2024-09-11T08:31:59.340734+0000","last_clean":"2024-09-11T08:31:59.340734+0000","last_became_active":"2024-09-11T08:15:56.526954+0000","last_became_peered":"2024-09-11T08:15:56.526954+0000","last_unstale":"2024-09-11T08:31:59.340734+0000","last_undegraded":"2024-09-11T08:31:59.340734+0000","last_fullsized":"2024-09-11T08:31:59.340734+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:28.667828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:31:28.667828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:49.011994+0000","scrub_duration":0.001462996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":110,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:07.876453+0000","last_change":"2024-09-11T08:32:07.876414+0000","last_active":"2024-09-11T08:32:07.876453+0000","last_peered":"2024-09-11T08:32:07.876453+0000","last_clean":"2024-09-11T08:32:07.876453+0000","last_became_active":"2024-09-11T08:31:38.275166+0000","last_became_peered":"2024-09-11T08:31:38.275166+0000","last_unstale":"2024-09-11T08:32:07.876453+0000","last_undegraded":"2024-09-11T08:32:07.876453+0000","last_fullsized":"2024-09-11T08:32:07.876453+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:07.876335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:20:19.927616+0000","last_clean_scrub_stamp":"2024-09-11T08:32:07.876335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:10.106603+0000","scrub_duration":0.0017588199999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":386,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.488456+0000","last_change":"2024-09-11T08:31:35.505163+0000","last_active":"2024-09-11T08:31:59.488456+0000","last_peered":"2024-09-11T08:31:59.488456+0000","last_clean":"2024-09-11T08:31:59.488456+0000","last_became_active":"2024-09-11T08:31:16.964004+0000","last_became_peered":"2024-09-11T08:31:16.964004+0000","last_unstale":"2024-09-11T08:31:59.488456+0000","last_undegraded":"2024-09-11T08:31:59.488456+0000","last_fullsized":"2024-09-11T08:31:59.488456+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:35.505100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:31:35.505100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:05.402855+0000","scrub_duration":0.0015443919999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":559,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-11T08:32:22.036976+0000","last_change":"2024-09-11T08:32:19.169024+0000","last_active":"2024-09-11T08:32:22.036976+0000","last_peered":"2024-09-11T08:32:22.036976+0000","last_clean":"2024-09-11T08:32:22.036976+0000","last_became_active":"2024-09-11T08:15:56.516941+0000","last_became_peered":"2024-09-11T08:15:56.516941+0000","last_unstale":"2024-09-11T08:32:22.036976+0000","last_undegraded":"2024-09-11T08:32:22.036976+0000","last_fullsized":"2024-09-11T08:32:22.036976+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:19.168965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:32:19.168965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:22.316530+0000","scrub_duration":0.0014021610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"405'46","reported_seq":624,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:07.274133+0000","last_change":"2024-09-11T08:32:07.274072+0000","last_active":"2024-09-11T08:32:07.274133+0000","last_peered":"2024-09-11T08:32:07.274133+0000","last_clean":"2024-09-11T08:32:07.274133+0000","last_became_active":"2024-09-11T08:07:32.046363+0000","last_became_peered":"2024-09-11T08:07:32.046363+0000","last_unstale":"2024-09-11T08:32:07.274133+0000","last_undegraded":"2024-09-11T08:32:07.274133+0000","last_fullsized":"2024-09-11T08:32:07.274133+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'46","last_scrub_stamp":"2024-09-11T08:32:07.274016+0000","last_deep_scrub":"107'22","last_deep_scrub_stamp":"2024-09-11T08:16:04.159211+0000","last_clean_scrub_stamp":"2024-09-11T08:32:07.274016+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:26.254312+0000","scrub_duration":0.001758459,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":59,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-11T08:32:22.036883+0000","last_change":"2024-09-11T08:32:01.223891+0000","last_active":"2024-09-11T08:32:22.036883+0000","last_peered":"2024-09-11T08:32:22.036883+0000","last_clean":"2024-09-11T08:32:22.036883+0000","last_became_active":"2024-09-11T08:30:37.522527+0000","last_became_peered":"2024-09-11T08:30:37.522527+0000","last_unstale":"2024-09-11T08:32:22.036883+0000","last_undegraded":"2024-09-11T08:32:22.036883+0000","last_fullsized":"2024-09-11T08:32:22.036883+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:01.223798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:32:01.223798+0000","last_clean_scrub_stamp":"2024-09-11T08:32:01.223798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:10.427791+0000","scrub_duration":0.0016186849999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":250,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.341072+0000","last_change":"2024-09-11T08:31:26.107834+0000","last_active":"2024-09-11T08:31:59.341072+0000","last_peered":"2024-09-11T08:31:59.341072+0000","last_clean":"2024-09-11T08:31:59.341072+0000","last_became_active":"2024-09-11T08:31:09.936646+0000","last_became_peered":"2024-09-11T08:31:09.936646+0000","last_unstale":"2024-09-11T08:31:59.341072+0000","last_undegraded":"2024-09-11T08:31:59.341072+0000","last_fullsized":"2024-09-11T08:31:59.341072+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:26.107772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:31:26.107772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:51.630945+0000","scrub_duration":0.0013940230000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":379,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:17.329915+0000","last_change":"2024-09-11T08:32:17.329872+0000","last_active":"2024-09-11T08:32:17.329915+0000","last_peered":"2024-09-11T08:32:17.329915+0000","last_clean":"2024-09-11T08:32:17.329915+0000","last_became_active":"2024-09-11T08:31:16.963363+0000","last_became_peered":"2024-09-11T08:31:16.963363+0000","last_unstale":"2024-09-11T08:32:17.329915+0000","last_undegraded":"2024-09-11T08:32:17.329915+0000","last_fullsized":"2024-09-11T08:32:17.329915+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:17.329804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:32:17.329804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:43.300883+0000","scrub_duration":0.001829656,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"405'24","reported_seq":333,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.340888+0000","last_change":"2024-09-11T08:31:09.939854+0000","last_active":"2024-09-11T08:31:59.340888+0000","last_peered":"2024-09-11T08:31:59.340888+0000","last_clean":"2024-09-11T08:31:59.340888+0000","last_became_active":"2024-09-11T08:31:09.939499+0000","last_became_peered":"2024-09-11T08:31:09.939499+0000","last_unstale":"2024-09-11T08:31:59.340888+0000","last_undegraded":"2024-09-11T08:31:59.340888+0000","last_fullsized":"2024-09-11T08:31:59.340888+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'24","last_scrub_stamp":"2024-09-11T08:30:52.542477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:30:52.542477+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:22.534296+0000","scrub_duration":0.0051635600000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":375,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.488073+0000","last_change":"2024-09-11T08:31:38.456553+0000","last_active":"2024-09-11T08:31:59.488073+0000","last_peered":"2024-09-11T08:31:59.488073+0000","last_clean":"2024-09-11T08:31:59.488073+0000","last_became_active":"2024-09-11T08:31:16.964385+0000","last_became_peered":"2024-09-11T08:31:16.964385+0000","last_unstale":"2024-09-11T08:31:59.488073+0000","last_undegraded":"2024-09-11T08:31:59.488073+0000","last_fullsized":"2024-09-11T08:31:59.488073+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:38.456489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:10:40.110490+0000","last_clean_scrub_stamp":"2024-09-11T08:31:38.456489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:48.645086+0000","scrub_duration":0.0015098780000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"478'2","reported_seq":527,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:12.893410+0000","last_change":"2024-09-11T08:32:12.893361+0000","last_active":"2024-09-11T08:32:12.893410+0000","last_peered":"2024-09-11T08:32:12.893410+0000","last_clean":"2024-09-11T08:32:12.893410+0000","last_became_active":"2024-09-11T08:24:28.357198+0000","last_became_peered":"2024-09-11T08:24:28.357198+0000","last_unstale":"2024-09-11T08:32:12.893410+0000","last_undegraded":"2024-09-11T08:32:12.893410+0000","last_fullsized":"2024-09-11T08:32:12.893410+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'2","last_scrub_stamp":"2024-09-11T08:32:12.893299+0000","last_deep_scrub":"478'2","last_deep_scrub_stamp":"2024-09-11T08:32:12.893299+0000","last_clean_scrub_stamp":"2024-09-11T08:32:12.893299+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:25.697133+0000","scrub_duration":0.0029555279999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":549,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-11T08:32:22.801817+0000","last_change":"2024-09-11T08:32:06.620049+0000","last_active":"2024-09-11T08:32:22.801817+0000","last_peered":"2024-09-11T08:32:22.801817+0000","last_clean":"2024-09-11T08:32:22.801817+0000","last_became_active":"2024-09-11T08:07:30.045932+0000","last_became_peered":"2024-09-11T08:07:30.045932+0000","last_unstale":"2024-09-11T08:32:22.801817+0000","last_undegraded":"2024-09-11T08:32:22.801817+0000","last_fullsized":"2024-09-11T08:32:22.801817+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:06.619993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:32:06.619993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:29.478781+0000","scrub_duration":0.001538059,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"405'35","reported_seq":650,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-11T08:32:24.134565+0000","last_change":"2024-09-11T08:32:24.134539+0000","last_active":"2024-09-11T08:32:24.134565+0000","last_peered":"2024-09-11T08:32:24.134565+0000","last_clean":"2024-09-11T08:32:24.134565+0000","last_became_active":"2024-09-11T08:15:56.515817+0000","last_became_peered":"2024-09-11T08:15:56.515817+0000","last_unstale":"2024-09-11T08:32:24.134565+0000","last_undegraded":"2024-09-11T08:32:24.134565+0000","last_fullsized":"2024-09-11T08:32:24.134565+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'35","last_scrub_stamp":"2024-09-11T08:32:24.134478+0000","last_deep_scrub":"350'25","last_deep_scrub_stamp":"2024-09-11T08:24:23.093269+0000","last_clean_scrub_stamp":"2024-09-11T08:32:24.134478+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:42.588817+0000","scrub_duration":0.0022256899999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"254'12","reported_seq":900,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:18.869510+0000","last_change":"2024-09-11T08:30:47.391441+0000","last_active":"2024-09-11T08:32:18.869510+0000","last_peered":"2024-09-11T08:32:18.869510+0000","last_clean":"2024-09-11T08:32:18.869510+0000","last_became_active":"2024-09-11T08:15:56.525498+0000","last_became_peered":"2024-09-11T08:15:56.525498+0000","last_unstale":"2024-09-11T08:32:18.869510+0000","last_undegraded":"2024-09-11T08:32:18.869510+0000","last_fullsized":"2024-09-11T08:32:18.869510+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'12","last_scrub_stamp":"2024-09-11T08:30:47.391368+0000","last_deep_scrub":"254'12","last_deep_scrub_stamp":"2024-09-11T08:18:34.253865+0000","last_clean_scrub_stamp":"2024-09-11T08:30:47.391368+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00171394,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"478'46","reported_seq":625,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-11T08:32:22.802304+0000","last_change":"2024-09-11T08:32:04.621261+0000","last_active":"2024-09-11T08:32:22.802304+0000","last_peered":"2024-09-11T08:32:22.802304+0000","last_clean":"2024-09-11T08:32:22.802304+0000","last_became_active":"2024-09-11T08:07:32.054317+0000","last_became_peered":"2024-09-11T08:07:32.054317+0000","last_unstale":"2024-09-11T08:32:22.802304+0000","last_undegraded":"2024-09-11T08:32:22.802304+0000","last_fullsized":"2024-09-11T08:32:22.802304+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'46","last_scrub_stamp":"2024-09-11T08:32:04.621207+0000","last_deep_scrub":"478'46","last_deep_scrub_stamp":"2024-09-11T08:32:04.621207+0000","last_clean_scrub_stamp":"2024-09-11T08:32:04.621207+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:12.236782+0000","scrub_duration":0.0043013419999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":559,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.340620+0000","last_change":"2024-09-11T08:30:59.053520+0000","last_active":"2024-09-11T08:31:59.340620+0000","last_peered":"2024-09-11T08:31:59.340620+0000","last_clean":"2024-09-11T08:31:59.340620+0000","last_became_active":"2024-09-11T08:15:56.521943+0000","last_became_peered":"2024-09-11T08:15:56.521943+0000","last_unstale":"2024-09-11T08:31:59.340620+0000","last_undegraded":"2024-09-11T08:31:59.340620+0000","last_fullsized":"2024-09-11T08:31:59.340620+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:59.053459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:29:14.660494+0000","last_clean_scrub_stamp":"2024-09-11T08:30:59.053459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:22.745259+0000","scrub_duration":0.0027802009999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"468'14","reported_seq":614,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:18.869529+0000","last_change":"2024-09-11T08:32:02.484434+0000","last_active":"2024-09-11T08:32:18.869529+0000","last_peered":"2024-09-11T08:32:18.869529+0000","last_clean":"2024-09-11T08:32:18.869529+0000","last_became_active":"2024-09-11T08:31:16.965418+0000","last_became_peered":"2024-09-11T08:31:16.965418+0000","last_unstale":"2024-09-11T08:32:18.869529+0000","last_undegraded":"2024-09-11T08:32:18.869529+0000","last_fullsized":"2024-09-11T08:32:18.869529+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"468'14","last_scrub_stamp":"2024-09-11T08:32:02.484361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:32:02.484361+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:05.400545+0000","scrub_duration":0.0017041580000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":110,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:08.889392+0000","last_change":"2024-09-11T08:32:08.889322+0000","last_active":"2024-09-11T08:32:08.889392+0000","last_peered":"2024-09-11T08:32:08.889392+0000","last_clean":"2024-09-11T08:32:08.889392+0000","last_became_active":"2024-09-11T08:31:38.276065+0000","last_became_peered":"2024-09-11T08:31:38.276065+0000","last_unstale":"2024-09-11T08:32:08.889392+0000","last_undegraded":"2024-09-11T08:32:08.889392+0000","last_fullsized":"2024-09-11T08:32:08.889392+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:08.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:32:08.889241+0000","last_clean_scrub_stamp":"2024-09-11T08:32:08.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:24.868148+0000","scrub_duration":0.0018205300000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"405'42","reported_seq":537,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.340671+0000","last_change":"2024-09-11T08:31:41.997788+0000","last_active":"2024-09-11T08:31:59.340671+0000","last_peered":"2024-09-11T08:31:59.340671+0000","last_clean":"2024-09-11T08:31:59.340671+0000","last_became_active":"2024-09-11T08:31:09.937968+0000","last_became_peered":"2024-09-11T08:31:09.937968+0000","last_unstale":"2024-09-11T08:31:59.340671+0000","last_undegraded":"2024-09-11T08:31:59.340671+0000","last_fullsized":"2024-09-11T08:31:59.340671+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'42","last_scrub_stamp":"2024-09-11T08:31:41.997727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:31:41.997727+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:55.969437+0000","scrub_duration":0.0020628669999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":283,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.341348+0000","last_change":"2024-09-11T08:31:14.058970+0000","last_active":"2024-09-11T08:31:59.341348+0000","last_peered":"2024-09-11T08:31:59.341348+0000","last_clean":"2024-09-11T08:31:59.341348+0000","last_became_active":"2024-09-11T08:31:09.937276+0000","last_became_peered":"2024-09-11T08:31:09.937276+0000","last_unstale":"2024-09-11T08:31:59.341348+0000","last_undegraded":"2024-09-11T08:31:59.341348+0000","last_fullsized":"2024-09-11T08:31:59.341348+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:14.058909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:31:14.058909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:40.225603+0000","scrub_duration":0.0017324790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":568,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.487739+0000","last_change":"2024-09-11T08:31:54.535643+0000","last_active":"2024-09-11T08:31:59.487739+0000","last_peered":"2024-09-11T08:31:59.487739+0000","last_clean":"2024-09-11T08:31:59.487739+0000","last_became_active":"2024-09-11T08:15:56.522606+0000","last_became_peered":"2024-09-11T08:15:56.522606+0000","last_unstale":"2024-09-11T08:31:59.487739+0000","last_undegraded":"2024-09-11T08:31:59.487739+0000","last_fullsized":"2024-09-11T08:31:59.487739+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:54.535575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:31:54.535575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:22.276910+0000","scrub_duration":0.0015943730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":376,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:05.460780+0000","last_change":"2024-09-11T08:32:05.460705+0000","last_active":"2024-09-11T08:32:05.460780+0000","last_peered":"2024-09-11T08:32:05.460780+0000","last_clean":"2024-09-11T08:32:05.460780+0000","last_became_active":"2024-09-11T08:31:16.961341+0000","last_became_peered":"2024-09-11T08:31:16.961341+0000","last_unstale":"2024-09-11T08:32:05.460780+0000","last_undegraded":"2024-09-11T08:32:05.460780+0000","last_fullsized":"2024-09-11T08:32:05.460780+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:05.460598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:32:05.460598+0000","last_clean_scrub_stamp":"2024-09-11T08:32:05.460598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:19.903437+0000","scrub_duration":0.0017495709999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"405'42","reported_seq":427,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:08.480795+0000","last_change":"2024-09-11T08:32:08.480744+0000","last_active":"2024-09-11T08:32:08.480795+0000","last_peered":"2024-09-11T08:32:08.480795+0000","last_clean":"2024-09-11T08:32:08.480795+0000","last_became_active":"2024-09-11T08:31:16.964261+0000","last_became_peered":"2024-09-11T08:31:16.964261+0000","last_unstale":"2024-09-11T08:32:08.480795+0000","last_undegraded":"2024-09-11T08:32:08.480795+0000","last_fullsized":"2024-09-11T08:32:08.480795+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'42","last_scrub_stamp":"2024-09-11T08:32:08.480671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:32:08.480671+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:15.968883+0000","scrub_duration":0.0020752930000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":54,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":539,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-11T08:32:22.037091+0000","last_change":"2024-09-11T08:31:06.409776+0000","last_active":"2024-09-11T08:32:22.037091+0000","last_peered":"2024-09-11T08:32:22.037091+0000","last_clean":"2024-09-11T08:32:22.037091+0000","last_became_active":"2024-09-11T08:28:09.953224+0000","last_became_peered":"2024-09-11T08:28:09.953224+0000","last_unstale":"2024-09-11T08:32:22.037091+0000","last_undegraded":"2024-09-11T08:32:22.037091+0000","last_fullsized":"2024-09-11T08:32:22.037091+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:06.409723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:47.960329+0000","last_clean_scrub_stamp":"2024-09-11T08:31:06.409723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018024880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"82'9","reported_seq":865,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:18.869207+0000","last_change":"2024-09-11T08:32:08.085332+0000","last_active":"2024-09-11T08:32:18.869207+0000","last_peered":"2024-09-11T08:32:18.869207+0000","last_clean":"2024-09-11T08:32:18.869207+0000","last_became_active":"2024-09-11T08:07:34.056143+0000","last_became_peered":"2024-09-11T08:07:34.056143+0000","last_unstale":"2024-09-11T08:32:18.869207+0000","last_undegraded":"2024-09-11T08:32:18.869207+0000","last_fullsized":"2024-09-11T08:32:18.869207+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-11T08:32:08.085273+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-09-11T08:21:00.841426+0000","last_clean_scrub_stamp":"2024-09-11T08:32:08.085273+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:16.498338+0000","scrub_duration":0.001828208,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":513,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:16.913105+0000","last_change":"2024-09-11T08:32:16.913065+0000","last_active":"2024-09-11T08:32:16.913105+0000","last_peered":"2024-09-11T08:32:16.913105+0000","last_clean":"2024-09-11T08:32:16.913105+0000","last_became_active":"2024-09-11T08:14:24.788968+0000","last_became_peered":"2024-09-11T08:14:24.788968+0000","last_unstale":"2024-09-11T08:32:16.913105+0000","last_undegraded":"2024-09-11T08:32:16.913105+0000","last_fullsized":"2024-09-11T08:32:16.913105+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:16.913000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:32:16.913000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:32.069029+0000","scrub_duration":0.0014423400000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"405'30","reported_seq":642,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-11T08:32:22.801855+0000","last_change":"2024-09-11T08:31:46.632892+0000","last_active":"2024-09-11T08:32:22.801855+0000","last_peered":"2024-09-11T08:32:22.801855+0000","last_clean":"2024-09-11T08:32:22.801855+0000","last_became_active":"2024-09-11T08:24:28.355516+0000","last_became_peered":"2024-09-11T08:24:28.355516+0000","last_unstale":"2024-09-11T08:32:22.801855+0000","last_undegraded":"2024-09-11T08:32:22.801855+0000","last_fullsized":"2024-09-11T08:32:22.801855+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'30","last_scrub_stamp":"2024-09-11T08:31:46.632833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:31:46.632833+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:59.791078+0000","scrub_duration":0.0021540579999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":100,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:46.943326+0000","last_change":"2024-09-11T08:31:46.943284+0000","last_active":"2024-09-11T08:31:46.943326+0000","last_peered":"2024-09-11T08:31:46.943326+0000","last_clean":"2024-09-11T08:31:46.943326+0000","last_became_active":"2024-09-11T08:31:38.393030+0000","last_became_peered":"2024-09-11T08:31:38.393030+0000","last_unstale":"2024-09-11T08:31:46.943326+0000","last_undegraded":"2024-09-11T08:31:46.943326+0000","last_fullsized":"2024-09-11T08:31:46.943326+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:46.943222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:31:46.943222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:58.845481+0000","scrub_duration":0.0015361439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"466'16","reported_seq":375,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:18.869209+0000","last_change":"2024-09-11T08:31:24.103808+0000","last_active":"2024-09-11T08:32:18.869209+0000","last_peered":"2024-09-11T08:32:18.869209+0000","last_clean":"2024-09-11T08:32:18.869209+0000","last_became_active":"2024-09-11T08:31:09.939449+0000","last_became_peered":"2024-09-11T08:31:09.939449+0000","last_unstale":"2024-09-11T08:32:18.869209+0000","last_undegraded":"2024-09-11T08:32:18.869209+0000","last_fullsized":"2024-09-11T08:32:18.869209+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"466'16","last_scrub_stamp":"2024-09-11T08:31:24.103737+0000","last_deep_scrub":"376'15","last_deep_scrub_stamp":"2024-09-11T08:26:02.786767+0000","last_clean_scrub_stamp":"2024-09-11T08:31:24.103737+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:45.002015+0000","scrub_duration":0.0038168189999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"66'1","reported_seq":517,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-11T08:32:22.036482+0000","last_change":"2024-09-11T08:31:30.394478+0000","last_active":"2024-09-11T08:32:22.036482+0000","last_peered":"2024-09-11T08:32:22.036482+0000","last_clean":"2024-09-11T08:32:22.036482+0000","last_became_active":"2024-09-11T08:07:36.072593+0000","last_became_peered":"2024-09-11T08:07:36.072593+0000","last_unstale":"2024-09-11T08:32:22.036482+0000","last_undegraded":"2024-09-11T08:32:22.036482+0000","last_fullsized":"2024-09-11T08:32:22.036482+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-11T08:31:30.394422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:31:30.394422+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:38.328646+0000","scrub_duration":0.0013632799999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"405'47","reported_seq":730,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-11T08:32:22.036842+0000","last_change":"2024-09-11T08:31:32.364401+0000","last_active":"2024-09-11T08:32:22.036842+0000","last_peered":"2024-09-11T08:32:22.036842+0000","last_clean":"2024-09-11T08:32:22.036842+0000","last_became_active":"2024-09-11T08:24:28.364043+0000","last_became_peered":"2024-09-11T08:24:28.364043+0000","last_unstale":"2024-09-11T08:32:22.036842+0000","last_undegraded":"2024-09-11T08:32:22.036842+0000","last_fullsized":"2024-09-11T08:32:22.036842+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'47","last_scrub_stamp":"2024-09-11T08:31:32.364338+0000","last_deep_scrub":"396'37","last_deep_scrub_stamp":"2024-09-11T08:27:16.673846+0000","last_clean_scrub_stamp":"2024-09-11T08:31:32.364338+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:45.342025+0000","scrub_duration":0.002656524,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":453,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:07.466426+0000","last_change":"2024-09-11T08:32:07.466385+0000","last_active":"2024-09-11T08:32:07.466426+0000","last_peered":"2024-09-11T08:32:07.466426+0000","last_clean":"2024-09-11T08:32:07.466426+0000","last_became_active":"2024-09-11T08:31:16.962135+0000","last_became_peered":"2024-09-11T08:31:16.962135+0000","last_unstale":"2024-09-11T08:32:07.466426+0000","last_undegraded":"2024-09-11T08:32:07.466426+0000","last_fullsized":"2024-09-11T08:32:07.466426+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:07.466317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:28:03.117012+0000","last_clean_scrub_stamp":"2024-09-11T08:32:07.466317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:24.745309+0000","scrub_duration":0.0016487509999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":591,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:20.078811+0000","last_change":"2024-09-11T08:32:20.078755+0000","last_active":"2024-09-11T08:32:20.078811+0000","last_peered":"2024-09-11T08:32:20.078811+0000","last_clean":"2024-09-11T08:32:20.078811+0000","last_became_active":"2024-09-11T08:24:28.360235+0000","last_became_peered":"2024-09-11T08:24:28.360235+0000","last_unstale":"2024-09-11T08:32:20.078811+0000","last_undegraded":"2024-09-11T08:32:20.078811+0000","last_fullsized":"2024-09-11T08:32:20.078811+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:20.078690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:30:50.313341+0000","last_clean_scrub_stamp":"2024-09-11T08:32:20.078690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:29.483921+0000","scrub_duration":0.0014708779999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":543,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-11T08:32:22.801514+0000","last_change":"2024-09-11T08:32:12.760320+0000","last_active":"2024-09-11T08:32:22.801514+0000","last_peered":"2024-09-11T08:32:22.801514+0000","last_clean":"2024-09-11T08:32:22.801514+0000","last_became_active":"2024-09-11T08:07:36.631668+0000","last_became_peered":"2024-09-11T08:07:36.631668+0000","last_unstale":"2024-09-11T08:32:22.801514+0000","last_undegraded":"2024-09-11T08:32:22.801514+0000","last_fullsized":"2024-09-11T08:32:22.801514+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:12.760261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:32:12.760261+0000","last_clean_scrub_stamp":"2024-09-11T08:32:12.760261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:30.736546+0000","scrub_duration":0.0015255449999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"405'64","reported_seq":432,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-11T08:32:23.870262+0000","last_change":"2024-09-11T08:32:05.645080+0000","last_active":"2024-09-11T08:32:23.870262+0000","last_peered":"2024-09-11T08:32:23.870262+0000","last_clean":"2024-09-11T08:32:23.870262+0000","last_became_active":"2024-09-11T08:20:52.086843+0000","last_became_peered":"2024-09-11T08:20:52.086843+0000","last_unstale":"2024-09-11T08:32:23.870262+0000","last_undegraded":"2024-09-11T08:32:23.870262+0000","last_fullsized":"2024-09-11T08:32:23.870262+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'64","last_scrub_stamp":"2024-09-11T08:32:05.645017+0000","last_deep_scrub":"190'39","last_deep_scrub_stamp":"2024-09-11T08:15:52.847968+0000","last_clean_scrub_stamp":"2024-09-11T08:32:05.645017+0000","objects_scrubbed":10,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:16.632000+0000","scrub_duration":0.002140023,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":84,"num_read_kb":69,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"59'1","reported_seq":388,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.488378+0000","last_change":"2024-09-11T08:31:50.613691+0000","last_active":"2024-09-11T08:31:59.488378+0000","last_peered":"2024-09-11T08:31:59.488378+0000","last_clean":"2024-09-11T08:31:59.488378+0000","last_became_active":"2024-09-11T08:31:16.962180+0000","last_became_peered":"2024-09-11T08:31:16.962180+0000","last_unstale":"2024-09-11T08:31:59.488378+0000","last_undegraded":"2024-09-11T08:31:59.488378+0000","last_fullsized":"2024-09-11T08:31:59.488378+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-11T08:31:50.613635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:31:50.613635+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:02.149456+0000","scrub_duration":0.0016357800000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"459'3","reported_seq":246,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:12.850771+0000","last_change":"2024-09-11T08:32:12.850729+0000","last_active":"2024-09-11T08:32:12.850771+0000","last_peered":"2024-09-11T08:32:12.850771+0000","last_clean":"2024-09-11T08:32:12.850771+0000","last_became_active":"2024-09-11T08:31:38.272317+0000","last_became_peered":"2024-09-11T08:31:38.272317+0000","last_unstale":"2024-09-11T08:32:12.850771+0000","last_undegraded":"2024-09-11T08:32:12.850771+0000","last_fullsized":"2024-09-11T08:32:12.850771+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'3","last_scrub_stamp":"2024-09-11T08:32:12.850643+0000","last_deep_scrub":"459'3","last_deep_scrub_stamp":"2024-09-11T08:32:12.850643+0000","last_clean_scrub_stamp":"2024-09-11T08:32:12.850643+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:30.170764+0000","scrub_duration":0.0021779849999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":525,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-11T08:32:22.036933+0000","last_change":"2024-09-11T08:32:16.163999+0000","last_active":"2024-09-11T08:32:22.036933+0000","last_peered":"2024-09-11T08:32:22.036933+0000","last_clean":"2024-09-11T08:32:22.036933+0000","last_became_active":"2024-09-11T08:07:34.057154+0000","last_became_peered":"2024-09-11T08:07:34.057154+0000","last_unstale":"2024-09-11T08:32:22.036933+0000","last_undegraded":"2024-09-11T08:32:22.036933+0000","last_fullsized":"2024-09-11T08:32:22.036933+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:16.163910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:30:56.391357+0000","last_clean_scrub_stamp":"2024-09-11T08:32:16.163910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:31.559038+0000","scrub_duration":0.001449105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":560,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.341072+0000","last_change":"2024-09-11T08:31:40.988258+0000","last_active":"2024-09-11T08:31:59.341072+0000","last_peered":"2024-09-11T08:31:59.341072+0000","last_clean":"2024-09-11T08:31:59.341072+0000","last_became_active":"2024-09-11T08:15:56.525099+0000","last_became_peered":"2024-09-11T08:15:56.525099+0000","last_unstale":"2024-09-11T08:31:59.341072+0000","last_undegraded":"2024-09-11T08:31:59.341072+0000","last_fullsized":"2024-09-11T08:31:59.341072+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:40.988189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:31:40.988189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:43.141955+0000","scrub_duration":0.0018492420000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"405'21","reported_seq":646,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-11T08:32:22.802344+0000","last_change":"2024-09-11T08:31:45.642517+0000","last_active":"2024-09-11T08:32:22.802344+0000","last_peered":"2024-09-11T08:32:22.802344+0000","last_clean":"2024-09-11T08:32:22.802344+0000","last_became_active":"2024-09-11T08:24:28.358121+0000","last_became_peered":"2024-09-11T08:24:28.358121+0000","last_unstale":"2024-09-11T08:32:22.802344+0000","last_undegraded":"2024-09-11T08:32:22.802344+0000","last_fullsized":"2024-09-11T08:32:22.802344+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'21","last_scrub_stamp":"2024-09-11T08:31:45.642458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:31:45.642458+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:50.324424+0000","scrub_duration":0.0019628229999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":552,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-11T08:32:22.801697+0000","last_change":"2024-09-11T08:31:10.561292+0000","last_active":"2024-09-11T08:32:22.801697+0000","last_peered":"2024-09-11T08:32:22.801697+0000","last_clean":"2024-09-11T08:32:22.801697+0000","last_became_active":"2024-09-11T08:07:30.047014+0000","last_became_peered":"2024-09-11T08:07:30.047014+0000","last_unstale":"2024-09-11T08:32:22.801697+0000","last_undegraded":"2024-09-11T08:32:22.801697+0000","last_fullsized":"2024-09-11T08:32:22.801697+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:10.561206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:31:10.561206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:39.438725+0000","scrub_duration":0.0015848870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"127'2","reported_seq":578,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.561535+0000","last_change":"2024-09-11T08:31:25.588502+0000","last_active":"2024-09-11T08:31:59.561535+0000","last_peered":"2024-09-11T08:31:59.561535+0000","last_clean":"2024-09-11T08:31:59.561535+0000","last_became_active":"2024-09-11T08:07:30.045009+0000","last_became_peered":"2024-09-11T08:07:30.045009+0000","last_unstale":"2024-09-11T08:31:59.561535+0000","last_undegraded":"2024-09-11T08:31:59.561535+0000","last_fullsized":"2024-09-11T08:31:59.561535+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'2","last_scrub_stamp":"2024-09-11T08:31:25.588404+0000","last_deep_scrub":"127'2","last_deep_scrub_stamp":"2024-09-11T08:15:01.938497+0000","last_clean_scrub_stamp":"2024-09-11T08:31:25.588404+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:54.376950+0000","scrub_duration":0.0018978669999999999,"objects_trimmed":0,"snaptrim_duration":0.001859533,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":604,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.562207+0000","last_change":"2024-09-11T08:31:14.589099+0000","last_active":"2024-09-11T08:31:59.562207+0000","last_peered":"2024-09-11T08:31:59.562207+0000","last_clean":"2024-09-11T08:31:59.562207+0000","last_became_active":"2024-09-11T08:24:28.357141+0000","last_became_peered":"2024-09-11T08:24:28.357141+0000","last_unstale":"2024-09-11T08:31:59.562207+0000","last_undegraded":"2024-09-11T08:31:59.562207+0000","last_fullsized":"2024-09-11T08:31:59.562207+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:14.589031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:31:14.589031+0000","last_clean_scrub_stamp":"2024-09-11T08:31:14.589031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:41.206920+0000","scrub_duration":0.013661458,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":264,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.561820+0000","last_change":"2024-09-11T08:31:19.639639+0000","last_active":"2024-09-11T08:31:59.561820+0000","last_peered":"2024-09-11T08:31:59.561820+0000","last_clean":"2024-09-11T08:31:59.561820+0000","last_became_active":"2024-09-11T08:20:16.108674+0000","last_became_peered":"2024-09-11T08:20:16.108674+0000","last_unstale":"2024-09-11T08:31:59.561820+0000","last_undegraded":"2024-09-11T08:31:59.561820+0000","last_fullsized":"2024-09-11T08:31:59.561820+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:19.639583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:58.796217+0000","last_clean_scrub_stamp":"2024-09-11T08:31:19.639583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:39.569369+0000","scrub_duration":0.0018697239999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"405'42","reported_seq":587,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-11T08:32:22.037005+0000","last_change":"2024-09-11T08:31:14.379856+0000","last_active":"2024-09-11T08:32:22.037005+0000","last_peered":"2024-09-11T08:32:22.037005+0000","last_clean":"2024-09-11T08:32:22.037005+0000","last_became_active":"2024-09-11T08:07:32.055407+0000","last_became_peered":"2024-09-11T08:07:32.055407+0000","last_unstale":"2024-09-11T08:32:22.037005+0000","last_undegraded":"2024-09-11T08:32:22.037005+0000","last_fullsized":"2024-09-11T08:32:22.037005+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'42","last_scrub_stamp":"2024-09-11T08:31:14.379794+0000","last_deep_scrub":"287'30","last_deep_scrub_stamp":"2024-09-11T08:26:09.030192+0000","last_clean_scrub_stamp":"2024-09-11T08:31:14.379794+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017853260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"448'2","reported_seq":537,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-11T08:32:22.036651+0000","last_change":"2024-09-11T08:31:45.248631+0000","last_active":"2024-09-11T08:32:22.036651+0000","last_peered":"2024-09-11T08:32:22.036651+0000","last_clean":"2024-09-11T08:32:22.036651+0000","last_became_active":"2024-09-11T08:07:30.050504+0000","last_became_peered":"2024-09-11T08:07:30.050504+0000","last_unstale":"2024-09-11T08:32:22.036651+0000","last_undegraded":"2024-09-11T08:32:22.036651+0000","last_fullsized":"2024-09-11T08:32:22.036651+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'2","last_scrub_stamp":"2024-09-11T08:31:45.248571+0000","last_deep_scrub":"448'2","last_deep_scrub_stamp":"2024-09-11T08:30:18.171322+0000","last_clean_scrub_stamp":"2024-09-11T08:31:45.248571+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:09.565845+0000","scrub_duration":0.0021675890000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"224'6","reported_seq":1808,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-11T08:32:21.851429+0000","last_change":"2024-09-11T08:31:55.952630+0000","last_active":"2024-09-11T08:32:21.851429+0000","last_peered":"2024-09-11T08:32:21.851429+0000","last_clean":"2024-09-11T08:32:21.851429+0000","last_became_active":"2024-09-11T08:14:24.793048+0000","last_became_peered":"2024-09-11T08:14:24.793048+0000","last_unstale":"2024-09-11T08:32:21.851429+0000","last_undegraded":"2024-09-11T08:32:21.851429+0000","last_fullsized":"2024-09-11T08:32:21.851429+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'6","last_scrub_stamp":"2024-09-11T08:31:55.952557+0000","last_deep_scrub":"224'6","last_deep_scrub_stamp":"2024-09-11T08:30:46.710817+0000","last_clean_scrub_stamp":"2024-09-11T08:31:55.952557+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:24.069545+0000","scrub_duration":0.001767121,"objects_trimmed":0,"snaptrim_duration":0.000318821,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1083,"num_read_kb":1078,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":90,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:38.277327+0000","last_change":"2024-09-11T08:31:38.276729+0000","last_active":"2024-09-11T08:31:38.277327+0000","last_peered":"2024-09-11T08:31:38.277327+0000","last_clean":"2024-09-11T08:31:38.277327+0000","last_became_active":"2024-09-11T08:31:38.276422+0000","last_became_peered":"2024-09-11T08:31:38.276422+0000","last_unstale":"2024-09-11T08:31:38.277327+0000","last_undegraded":"2024-09-11T08:31:38.277327+0000","last_fullsized":"2024-09-11T08:31:38.277327+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:33.607524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:31:33.607524+0000","last_clean_scrub_stamp":"2024-09-11T08:31:33.607524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:58.401819+0000","scrub_duration":0.012786004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":586,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-11T08:32:22.802012+0000","last_change":"2024-09-11T08:31:34.640255+0000","last_active":"2024-09-11T08:32:22.802012+0000","last_peered":"2024-09-11T08:32:22.802012+0000","last_clean":"2024-09-11T08:32:22.802012+0000","last_became_active":"2024-09-11T08:24:28.358723+0000","last_became_peered":"2024-09-11T08:24:28.358723+0000","last_unstale":"2024-09-11T08:32:22.802012+0000","last_undegraded":"2024-09-11T08:32:22.802012+0000","last_fullsized":"2024-09-11T08:32:22.802012+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:34.640198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:31:34.640198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:40.799950+0000","scrub_duration":0.0015852520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"405'45","reported_seq":638,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.562669+0000","last_change":"2024-09-11T08:31:50.332005+0000","last_active":"2024-09-11T08:31:59.562669+0000","last_peered":"2024-09-11T08:31:59.562669+0000","last_clean":"2024-09-11T08:31:59.562669+0000","last_became_active":"2024-09-11T08:14:24.789154+0000","last_became_peered":"2024-09-11T08:14:24.789154+0000","last_unstale":"2024-09-11T08:31:59.562669+0000","last_undegraded":"2024-09-11T08:31:59.562669+0000","last_fullsized":"2024-09-11T08:31:59.562669+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'45","last_scrub_stamp":"2024-09-11T08:31:50.331948+0000","last_deep_scrub":"84'21","last_deep_scrub_stamp":"2024-09-11T08:08:56.705306+0000","last_clean_scrub_stamp":"2024-09-11T08:31:50.331948+0000","objects_scrubbed":7,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:59.645930+0000","scrub_duration":0.001894913,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":253,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.340661+0000","last_change":"2024-09-11T08:31:49.882240+0000","last_active":"2024-09-11T08:31:59.340661+0000","last_peered":"2024-09-11T08:31:59.340661+0000","last_clean":"2024-09-11T08:31:59.340661+0000","last_became_active":"2024-09-11T08:31:09.939314+0000","last_became_peered":"2024-09-11T08:31:09.939314+0000","last_unstale":"2024-09-11T08:31:59.340661+0000","last_undegraded":"2024-09-11T08:31:59.340661+0000","last_fullsized":"2024-09-11T08:31:59.340661+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:49.882137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:26:07.749828+0000","last_clean_scrub_stamp":"2024-09-11T08:31:49.882137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:50.457770+0000","scrub_duration":0.0015816000000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":357,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.340609+0000","last_change":"2024-09-11T08:31:48.864351+0000","last_active":"2024-09-11T08:31:59.340609+0000","last_peered":"2024-09-11T08:31:59.340609+0000","last_clean":"2024-09-11T08:31:59.340609+0000","last_became_active":"2024-09-11T08:15:56.523430+0000","last_became_peered":"2024-09-11T08:15:56.523430+0000","last_unstale":"2024-09-11T08:31:59.340609+0000","last_undegraded":"2024-09-11T08:31:59.340609+0000","last_fullsized":"2024-09-11T08:31:59.340609+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-11T08:31:48.864284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:05:21.231856+0000","last_clean_scrub_stamp":"2024-09-11T08:31:48.864284+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:57.253529+0000","scrub_duration":0.0019983760000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":381,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:03.934196+0000","last_change":"2024-09-11T08:32:03.934150+0000","last_active":"2024-09-11T08:32:03.934196+0000","last_peered":"2024-09-11T08:32:03.934196+0000","last_clean":"2024-09-11T08:32:03.934196+0000","last_became_active":"2024-09-11T08:24:28.357166+0000","last_became_peered":"2024-09-11T08:24:28.357166+0000","last_unstale":"2024-09-11T08:32:03.934196+0000","last_undegraded":"2024-09-11T08:32:03.934196+0000","last_fullsized":"2024-09-11T08:32:03.934196+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:03.934083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:32:03.934083+0000","last_clean_scrub_stamp":"2024-09-11T08:32:03.934083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:26.397468+0000","scrub_duration":0.0015356499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":295,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:47.935392+0000","last_change":"2024-09-11T08:31:47.935320+0000","last_active":"2024-09-11T08:31:47.935392+0000","last_peered":"2024-09-11T08:31:47.935392+0000","last_clean":"2024-09-11T08:31:47.935392+0000","last_became_active":"2024-09-11T08:31:38.274751+0000","last_became_peered":"2024-09-11T08:31:38.274751+0000","last_unstale":"2024-09-11T08:31:47.935392+0000","last_undegraded":"2024-09-11T08:31:47.935392+0000","last_fullsized":"2024-09-11T08:31:47.935392+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:47.935251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:31:47.935251+0000","last_clean_scrub_stamp":"2024-09-11T08:31:47.935251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:53.434588+0000","scrub_duration":0.001526169,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"405'32","reported_seq":412,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.488565+0000","last_change":"2024-09-11T08:31:48.522005+0000","last_active":"2024-09-11T08:31:59.488565+0000","last_peered":"2024-09-11T08:31:59.488565+0000","last_clean":"2024-09-11T08:31:59.488565+0000","last_became_active":"2024-09-11T08:31:16.964177+0000","last_became_peered":"2024-09-11T08:31:16.964177+0000","last_unstale":"2024-09-11T08:31:59.488565+0000","last_undegraded":"2024-09-11T08:31:59.488565+0000","last_fullsized":"2024-09-11T08:31:59.488565+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'32","last_scrub_stamp":"2024-09-11T08:31:48.521935+0000","last_deep_scrub":"405'32","last_deep_scrub_stamp":"2024-09-11T08:29:10.333163+0000","last_clean_scrub_stamp":"2024-09-11T08:31:48.521935+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:58.358474+0000","scrub_duration":0.0019994190000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":565,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.488114+0000","last_change":"2024-09-11T08:31:45.512656+0000","last_active":"2024-09-11T08:31:59.488114+0000","last_peered":"2024-09-11T08:31:59.488114+0000","last_clean":"2024-09-11T08:31:59.488114+0000","last_became_active":"2024-09-11T08:14:24.789888+0000","last_became_peered":"2024-09-11T08:14:24.789888+0000","last_unstale":"2024-09-11T08:31:59.488114+0000","last_undegraded":"2024-09-11T08:31:59.488114+0000","last_fullsized":"2024-09-11T08:31:59.488114+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:45.512589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:36.417911+0000","last_clean_scrub_stamp":"2024-09-11T08:31:45.512589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:08.979681+0000","scrub_duration":0.0016082480000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":100,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:41.054368+0000","last_change":"2024-09-11T08:31:41.054319+0000","last_active":"2024-09-11T08:31:41.054368+0000","last_peered":"2024-09-11T08:31:41.054368+0000","last_clean":"2024-09-11T08:31:41.054368+0000","last_became_active":"2024-09-11T08:31:38.278516+0000","last_became_peered":"2024-09-11T08:31:38.278516+0000","last_unstale":"2024-09-11T08:31:41.054368+0000","last_undegraded":"2024-09-11T08:31:41.054368+0000","last_fullsized":"2024-09-11T08:31:41.054368+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:41.054258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:01.558340+0000","last_clean_scrub_stamp":"2024-09-11T08:31:41.054258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:44.440320+0000","scrub_duration":0.001448427,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":237,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.340885+0000","last_change":"2024-09-11T08:31:09.938640+0000","last_active":"2024-09-11T08:31:59.340885+0000","last_peered":"2024-09-11T08:31:59.340885+0000","last_clean":"2024-09-11T08:31:59.340885+0000","last_became_active":"2024-09-11T08:31:09.938387+0000","last_became_peered":"2024-09-11T08:31:09.938387+0000","last_unstale":"2024-09-11T08:31:59.340885+0000","last_undegraded":"2024-09-11T08:31:59.340885+0000","last_fullsized":"2024-09-11T08:31:59.340885+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:30:56.263148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:29:16.340778+0000","last_clean_scrub_stamp":"2024-09-11T08:30:56.263148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0055940149999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"405'53","reported_seq":463,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.488535+0000","last_change":"2024-09-11T08:31:47.560601+0000","last_active":"2024-09-11T08:31:59.488535+0000","last_peered":"2024-09-11T08:31:59.488535+0000","last_clean":"2024-09-11T08:31:59.488535+0000","last_became_active":"2024-09-11T08:31:16.962778+0000","last_became_peered":"2024-09-11T08:31:16.962778+0000","last_unstale":"2024-09-11T08:31:59.488535+0000","last_undegraded":"2024-09-11T08:31:59.488535+0000","last_fullsized":"2024-09-11T08:31:59.488535+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'53","last_scrub_stamp":"2024-09-11T08:31:47.560530+0000","last_deep_scrub":"77'25","last_deep_scrub_stamp":"2024-09-11T08:08:32.458210+0000","last_clean_scrub_stamp":"2024-09-11T08:31:47.560530+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:05.414991+0000","scrub_duration":0.0024864760000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":59,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":554,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.562394+0000","last_change":"2024-09-11T08:31:22.290865+0000","last_active":"2024-09-11T08:31:59.562394+0000","last_peered":"2024-09-11T08:31:59.562394+0000","last_clean":"2024-09-11T08:31:59.562394+0000","last_became_active":"2024-09-11T08:07:30.042555+0000","last_became_peered":"2024-09-11T08:07:30.042555+0000","last_unstale":"2024-09-11T08:31:59.562394+0000","last_undegraded":"2024-09-11T08:31:59.562394+0000","last_fullsized":"2024-09-11T08:31:59.562394+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:22.290779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:31:22.290779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:45.306345+0000","scrub_duration":0.0013764719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":545,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-11T08:32:22.801355+0000","last_change":"2024-09-11T08:32:14.820270+0000","last_active":"2024-09-11T08:32:22.801355+0000","last_peered":"2024-09-11T08:32:22.801355+0000","last_clean":"2024-09-11T08:32:22.801355+0000","last_became_active":"2024-09-11T08:07:34.054961+0000","last_became_peered":"2024-09-11T08:07:34.054961+0000","last_unstale":"2024-09-11T08:32:22.801355+0000","last_undegraded":"2024-09-11T08:32:22.801355+0000","last_fullsized":"2024-09-11T08:32:22.801355+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:14.820205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:32:14.820205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:29.998148+0000","scrub_duration":0.0016007370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":562,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.562310+0000","last_change":"2024-09-11T08:31:23.626128+0000","last_active":"2024-09-11T08:31:59.562310+0000","last_peered":"2024-09-11T08:31:59.562310+0000","last_clean":"2024-09-11T08:31:59.562310+0000","last_became_active":"2024-09-11T08:07:36.071194+0000","last_became_peered":"2024-09-11T08:07:36.071194+0000","last_unstale":"2024-09-11T08:31:59.562310+0000","last_undegraded":"2024-09-11T08:31:59.562310+0000","last_fullsized":"2024-09-11T08:31:59.562310+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:23.626078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:14:07.930137+0000","last_clean_scrub_stamp":"2024-09-11T08:31:23.626078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:40.412275+0000","scrub_duration":0.001315185,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"405'35","reported_seq":397,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.488385+0000","last_change":"2024-09-11T08:31:26.528006+0000","last_active":"2024-09-11T08:31:59.488385+0000","last_peered":"2024-09-11T08:31:59.488385+0000","last_clean":"2024-09-11T08:31:59.488385+0000","last_became_active":"2024-09-11T08:31:16.965159+0000","last_became_peered":"2024-09-11T08:31:16.965159+0000","last_unstale":"2024-09-11T08:31:59.488385+0000","last_undegraded":"2024-09-11T08:31:59.488385+0000","last_fullsized":"2024-09-11T08:31:59.488385+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'35","last_scrub_stamp":"2024-09-11T08:31:26.527947+0000","last_deep_scrub":"224'15","last_deep_scrub_stamp":"2024-09-11T08:17:01.126478+0000","last_clean_scrub_stamp":"2024-09-11T08:31:26.527947+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:54.768876+0000","scrub_duration":0.0018737820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":263,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.340804+0000","last_change":"2024-09-11T08:31:44.943848+0000","last_active":"2024-09-11T08:31:59.340804+0000","last_peered":"2024-09-11T08:31:59.340804+0000","last_clean":"2024-09-11T08:31:59.340804+0000","last_became_active":"2024-09-11T08:31:09.935630+0000","last_became_peered":"2024-09-11T08:31:09.935630+0000","last_unstale":"2024-09-11T08:31:59.340804+0000","last_undegraded":"2024-09-11T08:31:59.340804+0000","last_fullsized":"2024-09-11T08:31:59.340804+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:44.943776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:27:59.866811+0000","last_clean_scrub_stamp":"2024-09-11T08:31:44.943776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:56.179793+0000","scrub_duration":0.0018035600000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"354'16","reported_seq":931,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:18.869175+0000","last_change":"2024-09-11T08:32:14.258182+0000","last_active":"2024-09-11T08:32:18.869175+0000","last_peered":"2024-09-11T08:32:18.869175+0000","last_clean":"2024-09-11T08:32:18.869175+0000","last_became_active":"2024-09-11T08:24:28.356749+0000","last_became_peered":"2024-09-11T08:24:28.356749+0000","last_unstale":"2024-09-11T08:32:18.869175+0000","last_undegraded":"2024-09-11T08:32:18.869175+0000","last_fullsized":"2024-09-11T08:32:18.869175+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'16","last_scrub_stamp":"2024-09-11T08:32:14.258123+0000","last_deep_scrub":"171'12","last_deep_scrub_stamp":"2024-09-11T08:14:56.881299+0000","last_clean_scrub_stamp":"2024-09-11T08:32:14.258123+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:15.941380+0000","scrub_duration":0.0021232379999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":353,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.488436+0000","last_change":"2024-09-11T08:31:16.963036+0000","last_active":"2024-09-11T08:31:59.488436+0000","last_peered":"2024-09-11T08:31:59.488436+0000","last_clean":"2024-09-11T08:31:59.488436+0000","last_became_active":"2024-09-11T08:31:16.962725+0000","last_became_peered":"2024-09-11T08:31:16.962725+0000","last_unstale":"2024-09-11T08:31:59.488436+0000","last_undegraded":"2024-09-11T08:31:59.488436+0000","last_fullsized":"2024-09-11T08:31:59.488436+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:03.429821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:31:03.429821+0000","last_clean_scrub_stamp":"2024-09-11T08:31:03.429821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:24.832924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"405'21","reported_seq":561,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-11T08:32:22.036811+0000","last_change":"2024-09-11T08:31:48.272868+0000","last_active":"2024-09-11T08:32:22.036811+0000","last_peered":"2024-09-11T08:32:22.036811+0000","last_clean":"2024-09-11T08:32:22.036811+0000","last_became_active":"2024-09-11T08:07:32.054885+0000","last_became_peered":"2024-09-11T08:07:32.054885+0000","last_unstale":"2024-09-11T08:32:22.036811+0000","last_undegraded":"2024-09-11T08:32:22.036811+0000","last_fullsized":"2024-09-11T08:32:22.036811+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'21","last_scrub_stamp":"2024-09-11T08:31:48.272742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:31:48.272742+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:59.397071+0000","scrub_duration":0.0025347939999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":100,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:06.889073+0000","last_change":"2024-09-11T08:32:06.889040+0000","last_active":"2024-09-11T08:32:06.889073+0000","last_peered":"2024-09-11T08:32:06.889073+0000","last_clean":"2024-09-11T08:32:06.889073+0000","last_became_active":"2024-09-11T08:31:38.276887+0000","last_became_peered":"2024-09-11T08:31:38.276887+0000","last_unstale":"2024-09-11T08:32:06.889073+0000","last_undegraded":"2024-09-11T08:32:06.889073+0000","last_fullsized":"2024-09-11T08:32:06.889073+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:06.888968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:32:06.888968+0000","last_clean_scrub_stamp":"2024-09-11T08:32:06.888968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:23.537405+0000","scrub_duration":0.001438246,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":574,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.561683+0000","last_change":"2024-09-11T08:31:16.558479+0000","last_active":"2024-09-11T08:31:59.561683+0000","last_peered":"2024-09-11T08:31:59.561683+0000","last_clean":"2024-09-11T08:31:59.561683+0000","last_became_active":"2024-09-11T08:07:34.056606+0000","last_became_peered":"2024-09-11T08:07:34.056606+0000","last_unstale":"2024-09-11T08:31:59.561683+0000","last_undegraded":"2024-09-11T08:31:59.561683+0000","last_fullsized":"2024-09-11T08:31:59.561683+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:16.558414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:31:16.558414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:41.693182+0000","scrub_duration":0.0016748749999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":550,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-11T08:32:22.036556+0000","last_change":"2024-09-11T08:31:58.294412+0000","last_active":"2024-09-11T08:32:22.036556+0000","last_peered":"2024-09-11T08:32:22.036556+0000","last_clean":"2024-09-11T08:32:22.036556+0000","last_became_active":"2024-09-11T08:15:56.518884+0000","last_became_peered":"2024-09-11T08:15:56.518884+0000","last_unstale":"2024-09-11T08:32:22.036556+0000","last_undegraded":"2024-09-11T08:32:22.036556+0000","last_fullsized":"2024-09-11T08:32:22.036556+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:58.294355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:18:32.278257+0000","last_clean_scrub_stamp":"2024-09-11T08:31:58.294355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:10.080664+0000","scrub_duration":0.0016822079999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"451'60","reported_seq":721,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.562515+0000","last_change":"2024-09-11T08:31:51.304466+0000","last_active":"2024-09-11T08:31:59.562515+0000","last_peered":"2024-09-11T08:31:59.562515+0000","last_clean":"2024-09-11T08:31:59.562515+0000","last_became_active":"2024-09-11T08:15:56.526961+0000","last_became_peered":"2024-09-11T08:15:56.526961+0000","last_unstale":"2024-09-11T08:31:59.562515+0000","last_undegraded":"2024-09-11T08:31:59.562515+0000","last_fullsized":"2024-09-11T08:31:59.562515+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"451'60","last_scrub_stamp":"2024-09-11T08:31:51.304392+0000","last_deep_scrub":"451'60","last_deep_scrub_stamp":"2024-09-11T08:30:32.275144+0000","last_clean_scrub_stamp":"2024-09-11T08:31:51.304392+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:06.903100+0000","scrub_duration":0.002187558,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":381,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.341124+0000","last_change":"2024-09-11T08:31:33.702517+0000","last_active":"2024-09-11T08:31:59.341124+0000","last_peered":"2024-09-11T08:31:59.341124+0000","last_clean":"2024-09-11T08:31:59.341124+0000","last_became_active":"2024-09-11T08:24:28.353720+0000","last_became_peered":"2024-09-11T08:24:28.353720+0000","last_unstale":"2024-09-11T08:31:59.341124+0000","last_undegraded":"2024-09-11T08:31:59.341124+0000","last_fullsized":"2024-09-11T08:31:59.341124+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:33.702440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:44.385372+0000","last_clean_scrub_stamp":"2024-09-11T08:31:33.702440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:46.595277+0000","scrub_duration":0.001500501,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":604,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:05.259530+0000","last_change":"2024-09-11T08:32:05.259494+0000","last_active":"2024-09-11T08:32:05.259530+0000","last_peered":"2024-09-11T08:32:05.259530+0000","last_clean":"2024-09-11T08:32:05.259530+0000","last_became_active":"2024-09-11T08:24:28.365792+0000","last_became_peered":"2024-09-11T08:24:28.365792+0000","last_unstale":"2024-09-11T08:32:05.259530+0000","last_undegraded":"2024-09-11T08:32:05.259530+0000","last_fullsized":"2024-09-11T08:32:05.259530+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:05.259438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:30:38.507031+0000","last_clean_scrub_stamp":"2024-09-11T08:32:05.259438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:19.690276+0000","scrub_duration":0.001688043,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":341,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.342147+0000","last_change":"2024-09-11T08:31:04.709026+0000","last_active":"2024-09-11T08:31:59.342147+0000","last_peered":"2024-09-11T08:31:59.342147+0000","last_clean":"2024-09-11T08:31:59.342147+0000","last_became_active":"2024-09-11T08:15:56.518169+0000","last_became_peered":"2024-09-11T08:15:56.518169+0000","last_unstale":"2024-09-11T08:31:59.342147+0000","last_undegraded":"2024-09-11T08:31:59.342147+0000","last_fullsized":"2024-09-11T08:31:59.342147+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:04.708945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:31:04.708945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:26.492136+0000","scrub_duration":0.001885889,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"405'29","reported_seq":402,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.487847+0000","last_change":"2024-09-11T08:31:58.508552+0000","last_active":"2024-09-11T08:31:59.487847+0000","last_peered":"2024-09-11T08:31:59.487847+0000","last_clean":"2024-09-11T08:31:59.487847+0000","last_became_active":"2024-09-11T08:31:16.964141+0000","last_became_peered":"2024-09-11T08:31:16.964141+0000","last_unstale":"2024-09-11T08:31:59.487847+0000","last_undegraded":"2024-09-11T08:31:59.487847+0000","last_fullsized":"2024-09-11T08:31:59.487847+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'29","last_scrub_stamp":"2024-09-11T08:31:58.508482+0000","last_deep_scrub":"145'17","last_deep_scrub_stamp":"2024-09-11T08:13:08.160848+0000","last_clean_scrub_stamp":"2024-09-11T08:31:58.508482+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:12.085510+0000","scrub_duration":0.0020195640000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":351,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.341460+0000","last_change":"2024-09-11T08:31:06.638726+0000","last_active":"2024-09-11T08:31:59.341460+0000","last_peered":"2024-09-11T08:31:59.341460+0000","last_clean":"2024-09-11T08:31:59.341460+0000","last_became_active":"2024-09-11T08:15:56.525845+0000","last_became_peered":"2024-09-11T08:15:56.525845+0000","last_unstale":"2024-09-11T08:31:59.341460+0000","last_undegraded":"2024-09-11T08:31:59.341460+0000","last_fullsized":"2024-09-11T08:31:59.341460+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:06.638658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:21:06.262572+0000","last_clean_scrub_stamp":"2024-09-11T08:31:06.638658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001497099,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":554,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.561393+0000","last_change":"2024-09-11T08:31:46.484735+0000","last_active":"2024-09-11T08:31:59.561393+0000","last_peered":"2024-09-11T08:31:59.561393+0000","last_clean":"2024-09-11T08:31:59.561393+0000","last_became_active":"2024-09-11T08:07:34.057479+0000","last_became_peered":"2024-09-11T08:07:34.057479+0000","last_unstale":"2024-09-11T08:31:59.561393+0000","last_undegraded":"2024-09-11T08:31:59.561393+0000","last_fullsized":"2024-09-11T08:31:59.561393+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:46.484680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:31:46.484680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:14.470245+0000","scrub_duration":0.001263501,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":570,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.340622+0000","last_change":"2024-09-11T08:31:53.940475+0000","last_active":"2024-09-11T08:31:59.340622+0000","last_peered":"2024-09-11T08:31:59.340622+0000","last_clean":"2024-09-11T08:31:59.340622+0000","last_became_active":"2024-09-11T08:15:56.522624+0000","last_became_peered":"2024-09-11T08:15:56.522624+0000","last_unstale":"2024-09-11T08:31:59.340622+0000","last_undegraded":"2024-09-11T08:31:59.340622+0000","last_fullsized":"2024-09-11T08:31:59.340622+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:53.940413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:31:53.940413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:59.428036+0000","scrub_duration":0.001889745,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"405'39","reported_seq":520,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:38.275956+0000","last_change":"2024-09-11T08:31:38.274144+0000","last_active":"2024-09-11T08:31:38.275956+0000","last_peered":"2024-09-11T08:31:38.275956+0000","last_clean":"2024-09-11T08:31:38.275956+0000","last_became_active":"2024-09-11T08:31:38.273273+0000","last_became_peered":"2024-09-11T08:31:38.273273+0000","last_unstale":"2024-09-11T08:31:38.275956+0000","last_undegraded":"2024-09-11T08:31:38.275956+0000","last_fullsized":"2024-09-11T08:31:38.275956+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'39","last_scrub_stamp":"2024-09-11T08:31:31.708951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:31:31.708951+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:59.554272+0000","scrub_duration":0.0026965510000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"478'4","reported_seq":359,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:11.862274+0000","last_change":"2024-09-11T08:32:11.862228+0000","last_active":"2024-09-11T08:32:11.862274+0000","last_peered":"2024-09-11T08:32:11.862274+0000","last_clean":"2024-09-11T08:32:11.862274+0000","last_became_active":"2024-09-11T08:15:56.515328+0000","last_became_peered":"2024-09-11T08:15:56.515328+0000","last_unstale":"2024-09-11T08:32:11.862274+0000","last_undegraded":"2024-09-11T08:32:11.862274+0000","last_fullsized":"2024-09-11T08:32:11.862274+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'4","last_scrub_stamp":"2024-09-11T08:32:11.862167+0000","last_deep_scrub":"478'4","last_deep_scrub_stamp":"2024-09-11T08:32:11.862167+0000","last_clean_scrub_stamp":"2024-09-11T08:32:11.862167+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:20.063599+0000","scrub_duration":0.0030199530000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"209'11","reported_seq":542,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:18.869570+0000","last_change":"2024-09-11T08:31:13.678884+0000","last_active":"2024-09-11T08:32:18.869570+0000","last_peered":"2024-09-11T08:32:18.869570+0000","last_clean":"2024-09-11T08:32:18.869570+0000","last_became_active":"2024-09-11T08:15:56.524513+0000","last_became_peered":"2024-09-11T08:15:56.524513+0000","last_unstale":"2024-09-11T08:32:18.869570+0000","last_undegraded":"2024-09-11T08:32:18.869570+0000","last_fullsized":"2024-09-11T08:32:18.869570+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"209'11","last_scrub_stamp":"2024-09-11T08:31:13.678807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:31:13.678807+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:39.976164+0000","scrub_duration":0.001875481,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":90,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:38.275911+0000","last_change":"2024-09-11T08:31:38.273925+0000","last_active":"2024-09-11T08:31:38.275911+0000","last_peered":"2024-09-11T08:31:38.275911+0000","last_clean":"2024-09-11T08:31:38.275911+0000","last_became_active":"2024-09-11T08:31:38.273642+0000","last_became_peered":"2024-09-11T08:31:38.273642+0000","last_unstale":"2024-09-11T08:31:38.275911+0000","last_undegraded":"2024-09-11T08:31:38.275911+0000","last_fullsized":"2024-09-11T08:31:38.275911+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:21.258836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:15:03.484755+0000","last_clean_scrub_stamp":"2024-09-11T08:31:21.258836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:45.633432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":598,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.562266+0000","last_change":"2024-09-11T08:31:56.331092+0000","last_active":"2024-09-11T08:31:59.562266+0000","last_peered":"2024-09-11T08:31:59.562266+0000","last_clean":"2024-09-11T08:31:59.562266+0000","last_became_active":"2024-09-11T08:24:28.357401+0000","last_became_peered":"2024-09-11T08:24:28.357401+0000","last_unstale":"2024-09-11T08:31:59.562266+0000","last_undegraded":"2024-09-11T08:31:59.562266+0000","last_fullsized":"2024-09-11T08:31:59.562266+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:56.331024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:31:56.331024+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:01.009568+0000","scrub_duration":0.0014664470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"405'21","reported_seq":610,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.341295+0000","last_change":"2024-09-11T08:31:02.037660+0000","last_active":"2024-09-11T08:31:59.341295+0000","last_peered":"2024-09-11T08:31:59.341295+0000","last_clean":"2024-09-11T08:31:59.341295+0000","last_became_active":"2024-09-11T08:15:56.526325+0000","last_became_peered":"2024-09-11T08:15:56.526325+0000","last_unstale":"2024-09-11T08:31:59.341295+0000","last_undegraded":"2024-09-11T08:31:59.341295+0000","last_fullsized":"2024-09-11T08:31:59.341295+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'21","last_scrub_stamp":"2024-09-11T08:31:02.037597+0000","last_deep_scrub":"405'21","last_deep_scrub_stamp":"2024-09-11T08:29:15.624982+0000","last_clean_scrub_stamp":"2024-09-11T08:31:02.037597+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022002020000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":100,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:42.012587+0000","last_change":"2024-09-11T08:31:42.012533+0000","last_active":"2024-09-11T08:31:42.012587+0000","last_peered":"2024-09-11T08:31:42.012587+0000","last_clean":"2024-09-11T08:31:42.012587+0000","last_became_active":"2024-09-11T08:31:38.275006+0000","last_became_peered":"2024-09-11T08:31:38.275006+0000","last_unstale":"2024-09-11T08:31:42.012587+0000","last_undegraded":"2024-09-11T08:31:42.012587+0000","last_fullsized":"2024-09-11T08:31:42.012587+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:42.012472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:53.548967+0000","last_clean_scrub_stamp":"2024-09-11T08:31:42.012472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:52.583946+0000","scrub_duration":0.001917462,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":476,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.341263+0000","last_change":"2024-09-11T08:31:20.679585+0000","last_active":"2024-09-11T08:31:59.341263+0000","last_peered":"2024-09-11T08:31:59.341263+0000","last_clean":"2024-09-11T08:31:59.341263+0000","last_became_active":"2024-09-11T08:14:24.790678+0000","last_became_peered":"2024-09-11T08:14:24.790678+0000","last_unstale":"2024-09-11T08:31:59.341263+0000","last_undegraded":"2024-09-11T08:31:59.341263+0000","last_fullsized":"2024-09-11T08:31:59.341263+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:20.679515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:29:20.428966+0000","last_clean_scrub_stamp":"2024-09-11T08:31:20.679515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:43.318353+0000","scrub_duration":0.0015400959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"405'25","reported_seq":602,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-11T08:32:22.037063+0000","last_change":"2024-09-11T08:31:26.439539+0000","last_active":"2024-09-11T08:32:22.037063+0000","last_peered":"2024-09-11T08:32:22.037063+0000","last_clean":"2024-09-11T08:32:22.037063+0000","last_became_active":"2024-09-11T08:15:56.514793+0000","last_became_peered":"2024-09-11T08:15:56.514793+0000","last_unstale":"2024-09-11T08:32:22.037063+0000","last_undegraded":"2024-09-11T08:32:22.037063+0000","last_fullsized":"2024-09-11T08:32:22.037063+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'25","last_scrub_stamp":"2024-09-11T08:31:26.439480+0000","last_deep_scrub":"84'13","last_deep_scrub_stamp":"2024-09-11T08:08:57.835634+0000","last_clean_scrub_stamp":"2024-09-11T08:31:26.439480+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:28.941228+0000","scrub_duration":0.001742469,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":499,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.341344+0000","last_change":"2024-09-11T08:31:15.661657+0000","last_active":"2024-09-11T08:31:59.341344+0000","last_peered":"2024-09-11T08:31:59.341344+0000","last_clean":"2024-09-11T08:31:59.341344+0000","last_became_active":"2024-09-11T08:14:24.786021+0000","last_became_peered":"2024-09-11T08:14:24.786021+0000","last_unstale":"2024-09-11T08:31:59.341344+0000","last_undegraded":"2024-09-11T08:31:59.341344+0000","last_fullsized":"2024-09-11T08:31:59.341344+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:15.661591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:08:30.471620+0000","last_clean_scrub_stamp":"2024-09-11T08:31:15.661591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:24.759947+0000","scrub_duration":0.0016265870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":562,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.341195+0000","last_change":"2024-09-11T08:31:38.946734+0000","last_active":"2024-09-11T08:31:59.341195+0000","last_peered":"2024-09-11T08:31:59.341195+0000","last_clean":"2024-09-11T08:31:59.341195+0000","last_became_active":"2024-09-11T08:15:56.525717+0000","last_became_peered":"2024-09-11T08:15:56.525717+0000","last_unstale":"2024-09-11T08:31:59.341195+0000","last_undegraded":"2024-09-11T08:31:59.341195+0000","last_fullsized":"2024-09-11T08:31:59.341195+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:38.946675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:04.370734+0000","last_clean_scrub_stamp":"2024-09-11T08:31:38.946675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:42.702831+0000","scrub_duration":0.0017947779999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":572,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:10.159959+0000","last_change":"2024-09-11T08:32:10.159924+0000","last_active":"2024-09-11T08:32:10.159959+0000","last_peered":"2024-09-11T08:32:10.159959+0000","last_clean":"2024-09-11T08:32:10.159959+0000","last_became_active":"2024-09-11T08:15:56.525120+0000","last_became_peered":"2024-09-11T08:15:56.525120+0000","last_unstale":"2024-09-11T08:32:10.159959+0000","last_undegraded":"2024-09-11T08:32:10.159959+0000","last_fullsized":"2024-09-11T08:32:10.159959+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:10.159870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:30:51.492107+0000","last_clean_scrub_stamp":"2024-09-11T08:32:10.159870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:31.581341+0000","scrub_duration":0.0015020299999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"405'25","reported_seq":205,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-11T08:31:38.276546+0000","last_change":"2024-09-11T08:31:38.275782+0000","last_active":"2024-09-11T08:31:38.276546+0000","last_peered":"2024-09-11T08:31:38.276546+0000","last_clean":"2024-09-11T08:31:38.276546+0000","last_became_active":"2024-09-11T08:31:38.275398+0000","last_became_peered":"2024-09-11T08:31:38.275398+0000","last_unstale":"2024-09-11T08:31:38.276546+0000","last_undegraded":"2024-09-11T08:31:38.276546+0000","last_fullsized":"2024-09-11T08:31:38.276546+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'25","last_scrub_stamp":"2024-09-11T08:31:27.625910+0000","last_deep_scrub":"313'19","last_deep_scrub_stamp":"2024-09-11T08:22:03.458402+0000","last_clean_scrub_stamp":"2024-09-11T08:31:27.625910+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:31.716398+0000","scrub_duration":0.0041528440000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":556,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-11T08:32:22.036761+0000","last_change":"2024-09-11T08:31:35.353159+0000","last_active":"2024-09-11T08:32:22.036761+0000","last_peered":"2024-09-11T08:32:22.036761+0000","last_clean":"2024-09-11T08:32:22.036761+0000","last_became_active":"2024-09-11T08:15:56.513278+0000","last_became_peered":"2024-09-11T08:15:56.513278+0000","last_unstale":"2024-09-11T08:32:22.036761+0000","last_undegraded":"2024-09-11T08:32:22.036761+0000","last_fullsized":"2024-09-11T08:32:22.036761+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:35.353103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:52.661591+0000","last_clean_scrub_stamp":"2024-09-11T08:31:35.353103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:56.859504+0000","scrub_duration":0.0016773529999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"466'14","reported_seq":439,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:18.869316+0000","last_change":"2024-09-11T08:32:06.015311+0000","last_active":"2024-09-11T08:32:18.869316+0000","last_peered":"2024-09-11T08:32:18.869316+0000","last_clean":"2024-09-11T08:32:18.869316+0000","last_became_active":"2024-09-11T08:31:09.937487+0000","last_became_peered":"2024-09-11T08:31:09.937487+0000","last_unstale":"2024-09-11T08:32:18.869316+0000","last_undegraded":"2024-09-11T08:32:18.869316+0000","last_fullsized":"2024-09-11T08:32:18.869316+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"466'14","last_scrub_stamp":"2024-09-11T08:32:06.015249+0000","last_deep_scrub":"222'12","last_deep_scrub_stamp":"2024-09-11T08:16:52.592856+0000","last_clean_scrub_stamp":"2024-09-11T08:32:06.015249+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:32.595210+0000","scrub_duration":0.0016785470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":383,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.488679+0000","last_change":"2024-09-11T08:31:30.513125+0000","last_active":"2024-09-11T08:31:59.488679+0000","last_peered":"2024-09-11T08:31:59.488679+0000","last_clean":"2024-09-11T08:31:59.488679+0000","last_became_active":"2024-09-11T08:31:16.964077+0000","last_became_peered":"2024-09-11T08:31:16.964077+0000","last_unstale":"2024-09-11T08:31:59.488679+0000","last_undegraded":"2024-09-11T08:31:59.488679+0000","last_fullsized":"2024-09-11T08:31:59.488679+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:30.513054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:17:53.175994+0000","last_clean_scrub_stamp":"2024-09-11T08:31:30.513054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:58.968082+0000","scrub_duration":0.0014064959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"477'35","reported_seq":372,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.341180+0000","last_change":"2024-09-11T08:31:56.969779+0000","last_active":"2024-09-11T08:31:59.341180+0000","last_peered":"2024-09-11T08:31:59.341180+0000","last_clean":"2024-09-11T08:31:59.341180+0000","last_became_active":"2024-09-11T08:31:09.937377+0000","last_became_peered":"2024-09-11T08:31:09.937377+0000","last_unstale":"2024-09-11T08:31:59.341180+0000","last_undegraded":"2024-09-11T08:31:59.341180+0000","last_fullsized":"2024-09-11T08:31:59.341180+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"477'35","last_scrub_stamp":"2024-09-11T08:31:56.969691+0000","last_deep_scrub":"477'35","last_deep_scrub_stamp":"2024-09-11T08:31:56.969691+0000","last_clean_scrub_stamp":"2024-09-11T08:31:56.969691+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:59.322094+0000","scrub_duration":0.003427692,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":351,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:13.881748+0000","last_change":"2024-09-11T08:32:13.881675+0000","last_active":"2024-09-11T08:32:13.881748+0000","last_peered":"2024-09-11T08:32:13.881748+0000","last_clean":"2024-09-11T08:32:13.881748+0000","last_became_active":"2024-09-11T08:15:56.518515+0000","last_became_peered":"2024-09-11T08:15:56.518515+0000","last_unstale":"2024-09-11T08:32:13.881748+0000","last_undegraded":"2024-09-11T08:32:13.881748+0000","last_fullsized":"2024-09-11T08:32:13.881748+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:13.881157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:23:42.199527+0000","last_clean_scrub_stamp":"2024-09-11T08:32:13.881157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:15.725971+0000","scrub_duration":0.001541398,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"477'14","reported_seq":165,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:13.868526+0000","last_change":"2024-09-11T08:31:48.912806+0000","last_active":"2024-09-11T08:32:13.868526+0000","last_peered":"2024-09-11T08:32:13.868526+0000","last_clean":"2024-09-11T08:32:13.868526+0000","last_became_active":"2024-09-11T08:31:38.273682+0000","last_became_peered":"2024-09-11T08:31:38.273682+0000","last_unstale":"2024-09-11T08:32:13.868526+0000","last_undegraded":"2024-09-11T08:32:13.868526+0000","last_fullsized":"2024-09-11T08:32:13.868526+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"477'14","last_scrub_stamp":"2024-09-11T08:31:48.912748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:31:48.912748+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:02.991937+0000","scrub_duration":0.001689704,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":540,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-11T08:32:22.036921+0000","last_change":"2024-09-11T08:31:10.355932+0000","last_active":"2024-09-11T08:32:22.036921+0000","last_peered":"2024-09-11T08:32:22.036921+0000","last_clean":"2024-09-11T08:32:22.036921+0000","last_became_active":"2024-09-11T08:15:56.513949+0000","last_became_peered":"2024-09-11T08:15:56.513949+0000","last_unstale":"2024-09-11T08:32:22.036921+0000","last_undegraded":"2024-09-11T08:32:22.036921+0000","last_fullsized":"2024-09-11T08:32:22.036921+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:10.355845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:29:05.957912+0000","last_clean_scrub_stamp":"2024-09-11T08:31:10.355845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:31.364564+0000","scrub_duration":0.001761047,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"405'27","reported_seq":390,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.488069+0000","last_change":"2024-09-11T08:31:39.447509+0000","last_active":"2024-09-11T08:31:59.488069+0000","last_peered":"2024-09-11T08:31:59.488069+0000","last_clean":"2024-09-11T08:31:59.488069+0000","last_became_active":"2024-09-11T08:31:16.963200+0000","last_became_peered":"2024-09-11T08:31:16.963200+0000","last_unstale":"2024-09-11T08:31:59.488069+0000","last_undegraded":"2024-09-11T08:31:59.488069+0000","last_fullsized":"2024-09-11T08:31:59.488069+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'27","last_scrub_stamp":"2024-09-11T08:31:39.447428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:31:39.447428+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:41.759058+0000","scrub_duration":0.002205252,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":409,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-11T08:32:22.301348+0000","last_change":"2024-09-11T08:32:22.301313+0000","last_active":"2024-09-11T08:32:22.301348+0000","last_peered":"2024-09-11T08:32:22.301348+0000","last_clean":"2024-09-11T08:32:22.301348+0000","last_became_active":"2024-09-11T08:31:16.963281+0000","last_became_peered":"2024-09-11T08:31:16.963281+0000","last_unstale":"2024-09-11T08:32:22.301348+0000","last_undegraded":"2024-09-11T08:32:22.301348+0000","last_fullsized":"2024-09-11T08:32:22.301348+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:22.301245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:32:22.301245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:44.324684+0000","scrub_duration":0.0015087729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":575,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-11T08:32:22.801605+0000","last_change":"2024-09-11T08:32:15.780680+0000","last_active":"2024-09-11T08:32:22.801605+0000","last_peered":"2024-09-11T08:32:22.801605+0000","last_clean":"2024-09-11T08:32:22.801605+0000","last_became_active":"2024-09-11T08:24:28.355004+0000","last_became_peered":"2024-09-11T08:24:28.355004+0000","last_unstale":"2024-09-11T08:32:22.801605+0000","last_undegraded":"2024-09-11T08:32:22.801605+0000","last_fullsized":"2024-09-11T08:32:22.801605+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:15.780625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:32:15.780625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:38.736839+0000","scrub_duration":0.0015874940000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":247,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.340412+0000","last_change":"2024-09-11T08:31:16.988591+0000","last_active":"2024-09-11T08:31:59.340412+0000","last_peered":"2024-09-11T08:31:59.340412+0000","last_clean":"2024-09-11T08:31:59.340412+0000","last_became_active":"2024-09-11T08:31:09.939169+0000","last_became_peered":"2024-09-11T08:31:09.939169+0000","last_unstale":"2024-09-11T08:31:59.340412+0000","last_undegraded":"2024-09-11T08:31:59.340412+0000","last_fullsized":"2024-09-11T08:31:59.340412+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:16.988505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:35.048712+0000","last_clean_scrub_stamp":"2024-09-11T08:31:16.988505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:35.235815+0000","scrub_duration":0.0018742850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"405'25","reported_seq":595,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-11T08:32:22.801946+0000","last_change":"2024-09-11T08:32:17.847720+0000","last_active":"2024-09-11T08:32:22.801946+0000","last_peered":"2024-09-11T08:32:22.801946+0000","last_clean":"2024-09-11T08:32:22.801946+0000","last_became_active":"2024-09-11T08:07:32.049368+0000","last_became_peered":"2024-09-11T08:07:32.049368+0000","last_unstale":"2024-09-11T08:32:22.801946+0000","last_undegraded":"2024-09-11T08:32:22.801946+0000","last_fullsized":"2024-09-11T08:32:22.801946+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'25","last_scrub_stamp":"2024-09-11T08:32:17.847645+0000","last_deep_scrub":"343'19","last_deep_scrub_stamp":"2024-09-11T08:23:44.640077+0000","last_clean_scrub_stamp":"2024-09-11T08:32:17.847645+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:33.522593+0000","scrub_duration":0.001956157,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":351,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:01.011763+0000","last_change":"2024-09-11T08:32:01.011725+0000","last_active":"2024-09-11T08:32:01.011763+0000","last_peered":"2024-09-11T08:32:01.011763+0000","last_clean":"2024-09-11T08:32:01.011763+0000","last_became_active":"2024-09-11T08:15:56.521570+0000","last_became_peered":"2024-09-11T08:15:56.521570+0000","last_unstale":"2024-09-11T08:32:01.011763+0000","last_undegraded":"2024-09-11T08:32:01.011763+0000","last_fullsized":"2024-09-11T08:32:01.011763+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:01.011667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:30:49.822760+0000","last_clean_scrub_stamp":"2024-09-11T08:32:01.011667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:09.110409+0000","scrub_duration":0.001390126,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":249,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.340692+0000","last_change":"2024-09-11T08:31:37.988996+0000","last_active":"2024-09-11T08:31:59.340692+0000","last_peered":"2024-09-11T08:31:59.340692+0000","last_clean":"2024-09-11T08:31:59.340692+0000","last_became_active":"2024-09-11T08:31:10.390449+0000","last_became_peered":"2024-09-11T08:31:10.390449+0000","last_unstale":"2024-09-11T08:31:59.340692+0000","last_undegraded":"2024-09-11T08:31:59.340692+0000","last_fullsized":"2024-09-11T08:31:59.340692+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:37.988934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:22:02.568761+0000","last_clean_scrub_stamp":"2024-09-11T08:31:37.988934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:01.150196+0000","scrub_duration":0.001198899,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":553,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-11T08:32:22.036379+0000","last_change":"2024-09-11T08:31:24.437447+0000","last_active":"2024-09-11T08:32:22.036379+0000","last_peered":"2024-09-11T08:32:22.036379+0000","last_clean":"2024-09-11T08:32:22.036379+0000","last_became_active":"2024-09-11T08:24:28.363959+0000","last_became_peered":"2024-09-11T08:24:28.363959+0000","last_unstale":"2024-09-11T08:32:22.036379+0000","last_undegraded":"2024-09-11T08:32:22.036379+0000","last_fullsized":"2024-09-11T08:32:22.036379+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:24.437358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:09:09.973943+0000","last_clean_scrub_stamp":"2024-09-11T08:31:24.437358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:39.841834+0000","scrub_duration":0.002335855,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"405'27","reported_seq":641,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:02.030729+0000","last_change":"2024-09-11T08:32:02.030662+0000","last_active":"2024-09-11T08:32:02.030729+0000","last_peered":"2024-09-11T08:32:02.030729+0000","last_clean":"2024-09-11T08:32:02.030729+0000","last_became_active":"2024-09-11T08:15:56.520058+0000","last_became_peered":"2024-09-11T08:15:56.520058+0000","last_unstale":"2024-09-11T08:32:02.030729+0000","last_undegraded":"2024-09-11T08:32:02.030729+0000","last_fullsized":"2024-09-11T08:32:02.030729+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'27","last_scrub_stamp":"2024-09-11T08:32:02.030601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:31.028627+0000","last_clean_scrub_stamp":"2024-09-11T08:32:02.030601+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:29.366441+0000","scrub_duration":0.0021067550000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":304,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:19.897173+0000","last_change":"2024-09-11T08:32:19.897141+0000","last_active":"2024-09-11T08:32:19.897173+0000","last_peered":"2024-09-11T08:32:19.897173+0000","last_clean":"2024-09-11T08:32:19.897173+0000","last_became_active":"2024-09-11T08:31:38.276807+0000","last_became_peered":"2024-09-11T08:31:38.276807+0000","last_unstale":"2024-09-11T08:32:19.897173+0000","last_undegraded":"2024-09-11T08:32:19.897173+0000","last_fullsized":"2024-09-11T08:32:19.897173+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:19.897071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:32:19.897071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:32.724831+0000","scrub_duration":0.0014472969999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":361,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:14.880207+0000","last_change":"2024-09-11T08:32:14.880135+0000","last_active":"2024-09-11T08:32:14.880207+0000","last_peered":"2024-09-11T08:32:14.880207+0000","last_clean":"2024-09-11T08:32:14.880207+0000","last_became_active":"2024-09-11T08:15:56.523728+0000","last_became_peered":"2024-09-11T08:15:56.523728+0000","last_unstale":"2024-09-11T08:32:14.880207+0000","last_undegraded":"2024-09-11T08:32:14.880207+0000","last_fullsized":"2024-09-11T08:32:14.880207+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:14.880063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:32:14.880063+0000","last_clean_scrub_stamp":"2024-09-11T08:32:14.880063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:15.965568+0000","scrub_duration":0.001635262,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":533,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.340755+0000","last_change":"2024-09-11T08:31:25.137259+0000","last_active":"2024-09-11T08:31:59.340755+0000","last_peered":"2024-09-11T08:31:59.340755+0000","last_clean":"2024-09-11T08:31:59.340755+0000","last_became_active":"2024-09-11T08:07:36.633368+0000","last_became_peered":"2024-09-11T08:07:36.633368+0000","last_unstale":"2024-09-11T08:31:59.340755+0000","last_undegraded":"2024-09-11T08:31:59.340755+0000","last_fullsized":"2024-09-11T08:31:59.340755+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:25.137209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:22:50.992034+0000","last_clean_scrub_stamp":"2024-09-11T08:31:25.137209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:31.673186+0000","scrub_duration":0.0026728099999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"405'14","reported_seq":557,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:16.185394+0000","last_change":"2024-09-11T08:32:16.185348+0000","last_active":"2024-09-11T08:32:16.185394+0000","last_peered":"2024-09-11T08:32:16.185394+0000","last_clean":"2024-09-11T08:32:16.185394+0000","last_became_active":"2024-09-11T08:07:32.045728+0000","last_became_peered":"2024-09-11T08:07:32.045728+0000","last_unstale":"2024-09-11T08:32:16.185394+0000","last_undegraded":"2024-09-11T08:32:16.185394+0000","last_fullsized":"2024-09-11T08:32:16.185394+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'14","last_scrub_stamp":"2024-09-11T08:32:16.185293+0000","last_deep_scrub":"312'10","last_deep_scrub_stamp":"2024-09-11T08:21:55.859866+0000","last_clean_scrub_stamp":"2024-09-11T08:32:16.185293+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:24.334734+0000","scrub_duration":0.0020369009999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":566,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-11T08:32:22.037087+0000","last_change":"2024-09-11T08:31:34.332876+0000","last_active":"2024-09-11T08:32:22.037087+0000","last_peered":"2024-09-11T08:32:22.037087+0000","last_clean":"2024-09-11T08:32:22.037087+0000","last_became_active":"2024-09-11T08:24:28.357978+0000","last_became_peered":"2024-09-11T08:24:28.357978+0000","last_unstale":"2024-09-11T08:32:22.037087+0000","last_undegraded":"2024-09-11T08:32:22.037087+0000","last_fullsized":"2024-09-11T08:32:22.037087+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:34.332817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:29.021655+0000","last_clean_scrub_stamp":"2024-09-11T08:31:34.332817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:40.628819+0000","scrub_duration":0.001888837,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":591,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.562179+0000","last_change":"2024-09-11T08:31:14.329296+0000","last_active":"2024-09-11T08:31:59.562179+0000","last_peered":"2024-09-11T08:31:59.562179+0000","last_clean":"2024-09-11T08:31:59.562179+0000","last_became_active":"2024-09-11T08:15:56.528607+0000","last_became_peered":"2024-09-11T08:15:56.528607+0000","last_unstale":"2024-09-11T08:31:59.562179+0000","last_undegraded":"2024-09-11T08:31:59.562179+0000","last_fullsized":"2024-09-11T08:31:59.562179+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:14.329233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:07:33.032025+0000","last_clean_scrub_stamp":"2024-09-11T08:31:14.329233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:39.409202+0000","scrub_duration":0.001373134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":344,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.341207+0000","last_change":"2024-09-11T08:31:46.846926+0000","last_active":"2024-09-11T08:31:59.341207+0000","last_peered":"2024-09-11T08:31:59.341207+0000","last_clean":"2024-09-11T08:31:59.341207+0000","last_became_active":"2024-09-11T08:15:56.521488+0000","last_became_peered":"2024-09-11T08:15:56.521488+0000","last_unstale":"2024-09-11T08:31:59.341207+0000","last_undegraded":"2024-09-11T08:31:59.341207+0000","last_fullsized":"2024-09-11T08:31:59.341207+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:46.846860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:20:18.827709+0000","last_clean_scrub_stamp":"2024-09-11T08:31:46.846860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:03.351737+0000","scrub_duration":0.001543598,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":573,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-11T08:32:22.802094+0000","last_change":"2024-09-11T08:32:03.613777+0000","last_active":"2024-09-11T08:32:22.802094+0000","last_peered":"2024-09-11T08:32:22.802094+0000","last_clean":"2024-09-11T08:32:22.802094+0000","last_became_active":"2024-09-11T08:24:28.354141+0000","last_became_peered":"2024-09-11T08:24:28.354141+0000","last_unstale":"2024-09-11T08:32:22.802094+0000","last_undegraded":"2024-09-11T08:32:22.802094+0000","last_fullsized":"2024-09-11T08:32:22.802094+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:03.613679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:16:50.585566+0000","last_clean_scrub_stamp":"2024-09-11T08:32:03.613679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:27.208307+0000","scrub_duration":0.0019471779999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":263,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.340545+0000","last_change":"2024-09-11T08:31:48.930337+0000","last_active":"2024-09-11T08:31:59.340545+0000","last_peered":"2024-09-11T08:31:59.340545+0000","last_clean":"2024-09-11T08:31:59.340545+0000","last_became_active":"2024-09-11T08:31:10.390629+0000","last_became_peered":"2024-09-11T08:31:10.390629+0000","last_unstale":"2024-09-11T08:31:59.340545+0000","last_undegraded":"2024-09-11T08:31:59.340545+0000","last_fullsized":"2024-09-11T08:31:59.340545+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:31:48.930281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:31:48.930281+0000","last_clean_scrub_stamp":"2024-09-11T08:31:48.930281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:58.685685+0000","scrub_duration":0.00115854,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"405'32","reported_seq":665,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:31:59.562839+0000","last_change":"2024-09-11T08:31:34.302924+0000","last_active":"2024-09-11T08:31:59.562839+0000","last_peered":"2024-09-11T08:31:59.562839+0000","last_clean":"2024-09-11T08:31:59.562839+0000","last_became_active":"2024-09-11T08:24:28.360937+0000","last_became_peered":"2024-09-11T08:24:28.360937+0000","last_unstale":"2024-09-11T08:31:59.562839+0000","last_undegraded":"2024-09-11T08:31:59.562839+0000","last_fullsized":"2024-09-11T08:31:59.562839+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'32","last_scrub_stamp":"2024-09-11T08:31:34.302851+0000","last_deep_scrub":"405'32","last_deep_scrub_stamp":"2024-09-11T08:31:34.302851+0000","last_clean_scrub_stamp":"2024-09-11T08:31:34.302851+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:32:43.589103+0000","scrub_duration":0.0026836949999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":389,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-11T08:32:07.248654+0000","last_change":"2024-09-11T08:32:07.248608+0000","last_active":"2024-09-11T08:32:07.248654+0000","last_peered":"2024-09-11T08:32:07.248654+0000","last_clean":"2024-09-11T08:32:07.248654+0000","last_became_active":"2024-09-11T08:15:08.769133+0000","last_became_peered":"2024-09-11T08:15:08.769133+0000","last_unstale":"2024-09-11T08:32:07.248654+0000","last_undegraded":"2024-09-11T08:32:07.248654+0000","last_fullsized":"2024-09-11T08:32:07.248654+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:32:07.248540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:18:39.206016+0000","last_clean_scrub_stamp":"2024-09-11T08:32:07.248540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:33:08.720288+0000","scrub_duration":0.0015499400000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106,"ondisk_log_size":106,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1532,"num_read_kb":1289,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":4566,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3425,"internal_metadata":0},"log_size":1091,"ondisk_log_size":1091,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1083,"num_read_kb":1078,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":168,"seq":721554505946,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":920772,"kb_used_data":1400,"kb_used_omap":7,"kb_used_meta":919352,"kb_avail":92824380,"statfs":{"total":95995035648,"available":95052165120,"internally_reserved":0,"allocated":1433600,"data_stored":1012494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8027,"internal_metadata":941416613},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.85899999999999999,"15min":2.3610000000000002},"min":{"1min":0.33600000000000002,"5min":0.186,"15min":0.186},"max":{"1min":0.82199999999999995,"5min":5.6740000000000004,"15min":117.57599999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.82599999999999996,"15min":2.2349999999999999},"min":{"1min":0.35999999999999999,"5min":0.215,"15min":0.215},"max":{"1min":0.68400000000000005,"5min":4.3070000000000004,"15min":117.542},"last":0.61299999999999999}]},{"osd":1,"last update":"Wed Sep 11 08:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":1.075,"15min":1.52},"min":{"1min":0.33700000000000002,"5min":0.23000000000000001,"15min":0.20899999999999999},"max":{"1min":0.69299999999999995,"5min":24.347999999999999,"15min":66.971000000000004},"last":0.5},{"interface":"front","average":{"1min":0.57399999999999995,"5min":1.2450000000000001,"15min":1.4099999999999999},"min":{"1min":0.46600000000000003,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.76200000000000001,"5min":25.382000000000001,"15min":67.185000000000002},"last":0.59799999999999998}]},{"osd":2,"last update":"Wed Sep 11 08:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.91000000000000003,"15min":1.0069999999999999},"min":{"1min":0.36099999999999999,"5min":0.245,"15min":0.23799999999999999},"max":{"1min":0.68100000000000005,"5min":9.3439999999999994,"15min":19.097000000000001},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.0800000000000001,"15min":1.177},"min":{"1min":0.40899999999999997,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.72899999999999998,"5min":7.7699999999999996,"15min":21.594000000000001},"last":0.65400000000000003}]},{"osd":3,"last update":"Wed Sep 11 08:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":1.319,"15min":1.075},"min":{"1min":0.40200000000000002,"5min":0.27200000000000002,"15min":0.251},"max":{"1min":0.70899999999999996,"5min":24.315000000000001,"15min":24.315000000000001},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.58999999999999997,"5min":1.109,"15min":1.022},"min":{"1min":0.40799999999999997,"5min":0.26600000000000001,"15min":0.251},"max":{"1min":0.71599999999999997,"5min":24.864999999999998,"15min":24.864999999999998},"last":0.63900000000000001}]},{"osd":4,"last update":"Wed Sep 11 08:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.52300000000000002,"15min":0.79300000000000004},"min":{"1min":0.377,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.69699999999999995,"5min":3.5550000000000002,"15min":5.7240000000000002},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.53600000000000003,"15min":0.72599999999999998},"min":{"1min":0.34999999999999998,"5min":0.26200000000000001,"15min":0.23000000000000001},"max":{"1min":0.73099999999999998,"5min":2.5670000000000002,"15min":5.2649999999999997},"last":0.40500000000000003}]},{"osd":5,"last update":"Wed Sep 11 08:31:46 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.58799999999999997,"15min":1.101},"min":{"1min":0.32800000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.77000000000000002,"5min":6.3979999999999997,"15min":6.3979999999999997},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.60499999999999998,"15min":0.81699999999999995},"min":{"1min":0.313,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.78300000000000003,"5min":2.887,"15min":2.887},"last":0.48199999999999998}]},{"osd":6,"last update":"Wed Sep 11 08:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":1.177,"15min":0.94899999999999995},"min":{"1min":0.39600000000000002,"5min":0.26500000000000001,"15min":0.248},"max":{"1min":0.749,"5min":42.749000000000002,"15min":42.749000000000002},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":1.1719999999999999,"15min":1.044},"min":{"1min":0.42699999999999999,"5min":0.311,"15min":0.25700000000000001},"max":{"1min":0.73799999999999999,"5min":41.914000000000001,"15min":41.914000000000001},"last":0.42799999999999999}]}]},{"osd":6,"up_from":45,"seq":193273528629,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":901040,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":899646,"kb_avail":92844112,"statfs":{"total":95995035648,"available":95072370688,"internally_reserved":0,"allocated":1417216,"data_stored":1012317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":921237848},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:31:36 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.83599999999999997,"15min":3.0489999999999999},"min":{"1min":0.37,"5min":0.20399999999999999,"15min":0.17999999999999999},"max":{"1min":0.68400000000000005,"5min":3.7919999999999998,"15min":331.14800000000002},"last":0.79000000000000004},{"interface":"front","average":{"1min":1.044,"5min":1.028,"15min":2.948},"min":{"1min":0.38800000000000001,"5min":0.30199999999999999,"15min":0.20499999999999999},"max":{"1min":9.6519999999999992,"5min":9.6519999999999992,"15min":331.31},"last":0.74399999999999999}]},{"osd":1,"last update":"Wed Sep 11 08:31:36 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":1.3600000000000001,"15min":1.306},"min":{"1min":0.35199999999999998,"5min":0.218,"15min":0.218},"max":{"1min":0.72099999999999997,"5min":29.423999999999999,"15min":29.423999999999999},"last":0.71499999999999997},{"interface":"front","average":{"1min":1.016,"5min":1.5820000000000001,"15min":1.3100000000000001},"min":{"1min":0.38600000000000001,"5min":0.23499999999999999,"15min":0.22700000000000001},"max":{"1min":9.6679999999999993,"5min":29.109000000000002,"15min":29.109000000000002},"last":0.76700000000000002}]},{"osd":2,"last update":"Wed Sep 11 08:31:36 2024","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":1.407,"15min":1.2949999999999999},"min":{"1min":0.372,"5min":0.25,"15min":0.23200000000000001},"max":{"1min":9.7110000000000003,"5min":28.167999999999999,"15min":39.652999999999999},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.54100000000000004,"5min":1.2969999999999999,"15min":1.3320000000000001},"min":{"1min":0.40000000000000002,"5min":0.251,"15min":0.251},"max":{"1min":0.72299999999999998,"5min":31.648,"15min":38.390999999999998},"last":0.70199999999999996}]},{"osd":3,"last update":"Wed Sep 11 08:31:36 2024","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.345,"15min":1.2490000000000001},"min":{"1min":0.38900000000000001,"5min":0.30599999999999999,"15min":0.22700000000000001},"max":{"1min":9.6850000000000005,"5min":12.414,"15min":12.414},"last":0.68899999999999995},{"interface":"front","average":{"1min":1.071,"5min":1.5189999999999999,"15min":1.246},"min":{"1min":0.46800000000000003,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":9.7400000000000002,"5min":28.004999999999999,"15min":28.004999999999999},"last":0.82599999999999996}]},{"osd":4,"last update":"Wed Sep 11 08:31:36 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.65100000000000002,"15min":0.72399999999999998},"min":{"1min":0.28399999999999997,"5min":0.23400000000000001,"15min":0.20499999999999999},"max":{"1min":0.80600000000000005,"5min":3.8679999999999999,"15min":9.7560000000000002},"last":0.501},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.67400000000000004,"15min":0.69299999999999995},"min":{"1min":0.35499999999999998,"5min":0.27600000000000002,"15min":0.23699999999999999},"max":{"1min":0.72099999999999997,"5min":3.5659999999999998,"15min":4.181},"last":0.53400000000000003}]},{"osd":5,"last update":"Wed Sep 11 08:31:36 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.68600000000000005,"15min":1.3819999999999999},"min":{"1min":0.33500000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.78500000000000003,"5min":3.4129999999999998,"15min":9.7989999999999995},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.498,"5min":0.74399999999999999,"15min":1.256},"min":{"1min":0.317,"5min":0.246,"15min":0.246},"max":{"1min":0.61899999999999999,"5min":3.4620000000000002,"15min":7.2999999999999998},"last":0.67100000000000004}]},{"osd":7,"last update":"Wed Sep 11 08:31:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.80700000000000005,"15min":0.92600000000000005},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.253},"max":{"1min":0.73299999999999998,"5min":3.8450000000000002,"15min":10.167999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.82499999999999996,"15min":0.77400000000000002},"min":{"1min":0.33600000000000002,"5min":0.28799999999999998,"15min":0.25700000000000001},"max":{"1min":0.74299999999999999,"5min":3.6160000000000001,"15min":5.3579999999999997},"last":0.57199999999999995}]}]},{"osd":5,"up_from":353,"seq":1516123455585,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":364900,"kb_used_data":796,"kb_used_omap":4,"kb_used_meta":364091,"kb_avail":93380252,"statfs":{"total":95995035648,"available":95621378048,"internally_reserved":0,"allocated":815104,"data_stored":414795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":372829702},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.66800000000000004,"15min":1.4239999999999999},"min":{"1min":0.32700000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.66500000000000004,"5min":4.0199999999999996,"15min":19.379999999999999},"last":0.746},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.79700000000000004,"15min":1.446},"min":{"1min":0.312,"5min":0.22600000000000001,"15min":0.218},"max":{"1min":0.69099999999999995,"5min":5.8979999999999997,"15min":18.238},"last":0.44900000000000001}]},{"osd":1,"last update":"Wed Sep 11 08:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":2.6789999999999998,"15min":1.6000000000000001},"min":{"1min":0.314,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.70799999999999996,"5min":85.728999999999999,"15min":85.728999999999999},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.52900000000000003,"5min":2.8330000000000002,"15min":1.4019999999999999},"min":{"1min":0.35299999999999998,"5min":0.221,"15min":0.221},"max":{"1min":0.69999999999999996,"5min":85.668000000000006,"15min":85.668000000000006},"last":0.497}]},{"osd":2,"last update":"Wed Sep 11 08:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":1.111,"15min":1.702},"min":{"1min":0.379,"5min":0.25800000000000001,"15min":0.251},"max":{"1min":7.3609999999999998,"5min":11.262,"15min":18.350999999999999},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.98599999999999999,"15min":1.5609999999999999},"min":{"1min":0.39500000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.77800000000000002,"5min":11.363,"15min":18.381},"last":0.64200000000000002}]},{"osd":3,"last update":"Wed Sep 11 08:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":2.2120000000000002,"15min":1.4990000000000001},"min":{"1min":0.42799999999999999,"5min":0.251,"15min":0.251},"max":{"1min":0.72699999999999998,"5min":73.697000000000003,"15min":73.697000000000003},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.54800000000000004,"5min":2.0499999999999998,"15min":1.256},"min":{"1min":0.36299999999999999,"5min":0.29699999999999999,"15min":0.28699999999999998},"max":{"1min":0.73699999999999999,"5min":73.727999999999994,"15min":73.727999999999994},"last":0.52800000000000002}]},{"osd":4,"last update":"Wed Sep 11 08:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.58699999999999997,"15min":0.97299999999999998},"min":{"1min":0.26100000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.66100000000000003,"5min":5.8630000000000004,"15min":5.8630000000000004},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.50700000000000001,"15min":0.999},"min":{"1min":0.28999999999999998,"5min":0.252,"15min":0.252},"max":{"1min":0.72299999999999998,"5min":3.1349999999999998,"15min":5.8979999999999997},"last":0.66800000000000004}]},{"osd":6,"last update":"Wed Sep 11 08:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.58399999999999996,"15min":1.179},"min":{"1min":0.34799999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.76600000000000001,"5min":3.1040000000000001,"15min":5.9779999999999998},"last":0.63},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.55000000000000004,"15min":1.373},"min":{"1min":0.311,"5min":0.26000000000000001,"15min":0.23699999999999999},"max":{"1min":0.81200000000000006,"5min":3.7200000000000002,"15min":5.6680000000000001},"last":0.77900000000000003}]},{"osd":7,"last update":"Wed Sep 11 08:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.77700000000000002,"15min":1.05},"min":{"1min":0.32600000000000001,"5min":0.28999999999999998,"15min":0.28499999999999998},"max":{"1min":0.78600000000000003,"5min":7.8719999999999999,"15min":7.8719999999999999},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.69499999999999995,"15min":1.373},"min":{"1min":0.45900000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.76900000000000002,"5min":6.8739999999999997,"15min":6.8739999999999997},"last":0.69599999999999995}]}]},{"osd":4,"up_from":31,"seq":133143986495,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":894920,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":894078,"kb_avail":92850232,"statfs":{"total":95995035648,"available":95078637568,"internally_reserved":0,"allocated":847872,"data_stored":422824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":915536216},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:31:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":1,"15min":1.8700000000000001},"min":{"1min":0.32800000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.879,"5min":14.891,"15min":64.329999999999998},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.76600000000000001,"15min":1.863},"min":{"1min":0.40200000000000002,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.76900000000000002,"5min":7.2800000000000002,"15min":66.819999999999993},"last":0.45000000000000001}]},{"osd":1,"last update":"Wed Sep 11 08:31:49 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":2.2610000000000001,"15min":1.581},"min":{"1min":0.29299999999999998,"5min":0.249,"15min":0.22},"max":{"1min":0.79500000000000004,"5min":91.852000000000004,"15min":91.852000000000004},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.60999999999999999,"5min":2.2919999999999998,"15min":1.675},"min":{"1min":0.433,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.84299999999999997,"5min":91.697000000000003,"15min":91.697000000000003},"last":0.31}]},{"osd":2,"last update":"Wed Sep 11 08:31:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":1.0680000000000001,"15min":1.401},"min":{"1min":0.40400000000000003,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.80600000000000005,"5min":13.710000000000001,"15min":54.317999999999998},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.59699999999999998,"5min":1.2589999999999999,"15min":1.575},"min":{"1min":0.378,"5min":0.246,"15min":0.23400000000000001},"max":{"1min":0.81999999999999995,"5min":12.711,"15min":54.430999999999997},"last":0.31900000000000001}]},{"osd":3,"last update":"Wed Sep 11 08:31:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":1.129,"15min":1.2529999999999999},"min":{"1min":0.311,"5min":0.247,"15min":0.247},"max":{"1min":0.79200000000000004,"5min":12.811999999999999,"15min":30.321000000000002},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.278,"15min":1.135},"min":{"1min":0.371,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.73399999999999999,"5min":12.978,"15min":30.620000000000001},"last":0.46899999999999997}]},{"osd":5,"last update":"Wed Sep 11 08:31:41 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.56899999999999995,"15min":1.052},"min":{"1min":0.29799999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.76200000000000001,"5min":4.3410000000000002,"15min":5.1050000000000004},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.53000000000000003,"15min":1.1599999999999999},"min":{"1min":0.28799999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.68999999999999995,"5min":3.367,"15min":6.1150000000000002},"last":0.36399999999999999}]},{"osd":6,"last update":"Wed Sep 11 08:31:41 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.67000000000000004,"15min":0.72399999999999998},"min":{"1min":0.35099999999999998,"5min":0.247,"15min":0.23200000000000001},"max":{"1min":0.62,"5min":6.4039999999999999,"15min":6.4039999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.73299999999999998,"15min":0.78000000000000003},"min":{"1min":0.32700000000000001,"5min":0.26000000000000001,"15min":0.251},"max":{"1min":0.79100000000000004,"5min":4.2140000000000004,"15min":5.3159999999999998},"last":0.438}]},{"osd":7,"last update":"Wed Sep 11 08:32:07 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.57799999999999996,"15min":0.70499999999999996},"min":{"1min":0.33500000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.72399999999999998,"5min":3.339,"15min":6.093},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.51900000000000002,"15min":0.67800000000000005},"min":{"1min":0.311,"5min":0.29899999999999999,"15min":0.26800000000000002},"max":{"1min":0.74399999999999999,"5min":2.6030000000000002,"15min":4.8259999999999996},"last":0.64700000000000002}]}]},{"osd":3,"up_from":25,"seq":107374182724,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":902036,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":901182,"kb_avail":92843116,"statfs":{"total":95995035648,"available":95071350784,"internally_reserved":0,"allocated":860160,"data_stored":422295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":922810712},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:31:41 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":1.2589999999999999,"15min":1.5069999999999999},"min":{"1min":0.23899999999999999,"5min":0.216,"15min":0.14999999999999999},"max":{"1min":1.153,"5min":8.9930000000000003,"15min":58.880000000000003},"last":0.314},{"interface":"front","average":{"1min":0.35199999999999998,"5min":1.3029999999999999,"15min":1.4530000000000001},"min":{"1min":0.20499999999999999,"5min":0.20100000000000001,"15min":0.184},"max":{"1min":0.57099999999999995,"5min":9.5210000000000008,"15min":59.246000000000002},"last":0.27900000000000003}]},{"osd":1,"last update":"Wed Sep 11 08:31:41 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":1.9239999999999999,"15min":1.504},"min":{"1min":0.28000000000000003,"5min":0.26500000000000001,"15min":0.19600000000000001},"max":{"1min":1.214,"5min":54.859000000000002,"15min":54.859000000000002},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.47099999999999997,"5min":1.887,"15min":1.5649999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.19700000000000001},"max":{"1min":2.2589999999999999,"5min":55.773000000000003,"15min":55.773000000000003},"last":0.36799999999999999}]},{"osd":2,"last update":"Wed Sep 11 08:31:41 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":1.268,"15min":1.429},"min":{"1min":0.28100000000000003,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.3080000000000001,"5min":11.459,"15min":67.503},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.42699999999999999,"5min":1.534,"15min":1.4750000000000001},"min":{"1min":0.25,"5min":0.23799999999999999,"15min":0.19900000000000001},"max":{"1min":1.1850000000000001,"5min":11.516,"15min":67.313999999999993},"last":0.51600000000000001}]},{"osd":4,"last update":"Wed Sep 11 08:31:53 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.98599999999999999,"15min":1.1439999999999999},"min":{"1min":0.502,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":1.383,"5min":6.8680000000000003,"15min":11.802},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.64900000000000002,"5min":1.139,"15min":1.014},"min":{"1min":0.42299999999999999,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":2.1190000000000002,"5min":6.46,"15min":6.46},"last":0.45400000000000001}]},{"osd":5,"last update":"Wed Sep 11 08:31:41 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":1.113,"15min":1.512},"min":{"1min":0.39600000000000002,"5min":0.34899999999999998,"15min":0.32700000000000001},"max":{"1min":1.121,"5min":6.6189999999999998,"15min":7.3849999999999998},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.64900000000000002,"5min":1.0609999999999999,"15min":1.137},"min":{"1min":0.46200000000000002,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.421,"5min":6.835,"15min":6.835},"last":0.54500000000000004}]},{"osd":6,"last update":"Wed Sep 11 08:31:41 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":1.236,"15min":1.2310000000000001},"min":{"1min":0.40200000000000002,"5min":0.39100000000000001,"15min":0.29699999999999999},"max":{"1min":2.3180000000000001,"5min":6.6520000000000001,"15min":10.949999999999999},"last":0.432},{"interface":"front","average":{"1min":0.60699999999999998,"5min":1.0680000000000001,"15min":1.2470000000000001},"min":{"1min":0.39000000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":1.2509999999999999,"5min":6.8970000000000002,"15min":11.837},"last":0.504}]},{"osd":7,"last update":"Wed Sep 11 08:31:58 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":1.266,"15min":1.2889999999999999},"min":{"1min":0.42099999999999999,"5min":0.35499999999999998,"15min":0.29899999999999999},"max":{"1min":1.1679999999999999,"5min":6.6840000000000002,"15min":31.071000000000002},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.70299999999999996,"5min":1.3149999999999999,"15min":1.45},"min":{"1min":0.47199999999999998,"5min":0.35899999999999999,"15min":0.33800000000000002},"max":{"1min":1.784,"5min":6.899,"15min":58.820999999999998},"last":0.57199999999999995}]}]},{"osd":2,"up_from":18,"seq":77309411655,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":889972,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":889150,"kb_avail":92855180,"statfs":{"total":95995035648,"available":95083704320,"internally_reserved":0,"allocated":827392,"data_stored":421422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":910489944},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:32:11 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":2.4369999999999998,"15min":2.7810000000000001},"min":{"1min":0.23000000000000001,"5min":0.20100000000000001,"15min":0.17399999999999999},"max":{"1min":0.55500000000000005,"5min":102.18000000000001,"15min":102.18000000000001},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.85899999999999999,"15min":2.5350000000000001},"min":{"1min":0.247,"5min":0.23200000000000001,"15min":0.17999999999999999},"max":{"1min":0.50700000000000001,"5min":5.9349999999999996,"15min":93.790999999999997},"last":0.28499999999999998}]},{"osd":1,"last update":"Wed Sep 11 08:32:11 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":2.5489999999999999,"15min":1.9039999999999999},"min":{"1min":0.26400000000000001,"5min":0.20399999999999999,"15min":0.19800000000000001},"max":{"1min":0.68899999999999995,"5min":102.134,"15min":102.134},"last":0.307},{"interface":"front","average":{"1min":0.36599999999999999,"5min":1.073,"15min":1.3520000000000001},"min":{"1min":0.28399999999999997,"5min":0.251,"15min":0.20399999999999999},"max":{"1min":0.57999999999999996,"5min":6.7220000000000004,"15min":53.082999999999998},"last":0.36899999999999999}]},{"osd":3,"last update":"Wed Sep 11 08:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":3.0990000000000002,"15min":1.9199999999999999},"min":{"1min":0.29399999999999998,"5min":0.23699999999999999,"15min":0.22},"max":{"1min":0.54900000000000004,"5min":102.206,"15min":102.206},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.38500000000000001,"5min":2.8839999999999999,"15min":1.8520000000000001},"min":{"1min":0.311,"5min":0.251,"15min":0.21199999999999999},"max":{"1min":0.57399999999999995,"5min":104.768,"15min":104.768},"last":0.33900000000000002}]},{"osd":4,"last update":"Wed Sep 11 08:31:51 2024","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.1930000000000001,"15min":1.071},"min":{"1min":0.41799999999999998,"5min":0.25,"15min":0.24099999999999999},"max":{"1min":14.539,"5min":14.539,"15min":14.539},"last":0.53500000000000003},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.0149999999999999,"15min":1.0660000000000001},"min":{"1min":0.39900000000000002,"5min":0.36199999999999999,"15min":0.27500000000000002},"max":{"1min":14.554,"5min":14.554,"15min":14.554},"last":0.58499999999999996}]},{"osd":5,"last update":"Wed Sep 11 08:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.86499999999999999,"15min":1.377},"min":{"1min":0.44800000000000001,"5min":0.32100000000000001,"15min":0.27700000000000002},"max":{"1min":0.80400000000000005,"5min":4.5839999999999996,"15min":4.5839999999999996},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.99299999999999999,"15min":1.2569999999999999},"min":{"1min":0.39600000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.746,"5min":6.6589999999999998,"15min":6.6589999999999998},"last":0.503}]},{"osd":6,"last update":"Wed Sep 11 08:31:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":1.2849999999999999,"15min":1.175},"min":{"1min":0.45100000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.73499999999999999,"5min":6.3440000000000003,"15min":8.0350000000000001},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.97699999999999998,"15min":1.085},"min":{"1min":0.42999999999999999,"5min":0.377,"15min":0.32500000000000001},"max":{"1min":0.68899999999999995,"5min":5.9619999999999997,"15min":7.8710000000000004},"last":0.51400000000000001}]},{"osd":7,"last update":"Wed Sep 11 08:31:51 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":1.25,"15min":1.256},"min":{"1min":0.46000000000000002,"5min":0.315,"15min":0.27500000000000002},"max":{"1min":0.79700000000000004,"5min":11.279,"15min":35.274999999999999},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.60499999999999998,"5min":1.1659999999999999,"15min":1.3220000000000001},"min":{"1min":0.434,"5min":0.34999999999999998,"15min":0.32800000000000001},"max":{"1min":0.84199999999999997,"5min":11.305,"15min":35.265000000000001},"last":0.60799999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575181,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":891096,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":890238,"kb_avail":92854056,"statfs":{"total":95995035648,"available":95082553344,"internally_reserved":0,"allocated":864256,"data_stored":422438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":911604056},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:31:51 2024","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":1.1579999999999999,"15min":1.4259999999999999},"min":{"1min":0.218,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":0.52300000000000002,"5min":14.045,"15min":63.932000000000002},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.34200000000000003,"5min":1.327,"15min":1.665},"min":{"1min":0.254,"5min":0.182,"15min":0.16800000000000001},"max":{"1min":0.54100000000000004,"5min":14.127000000000001,"15min":64.269000000000005},"last":0.25}]},{"osd":2,"last update":"Wed Sep 11 08:31:27 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":1.718,"15min":1.4339999999999999},"min":{"1min":0.27200000000000002,"5min":0.19700000000000001,"15min":0.189},"max":{"1min":0.56599999999999995,"5min":13.928000000000001,"15min":20.925999999999998},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.34499999999999997,"5min":1.9099999999999999,"15min":1.462},"min":{"1min":0.24199999999999999,"5min":0.19900000000000001,"15min":0.19},"max":{"1min":0.49199999999999999,"5min":16.099,"15min":20.812000000000001},"last":0.41399999999999998}]},{"osd":3,"last update":"Wed Sep 11 08:31:42 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":1.6830000000000001,"15min":1.4219999999999999},"min":{"1min":0.215,"5min":0.21199999999999999,"15min":0.19700000000000001},"max":{"1min":0.504,"5min":16.193999999999999,"15min":16.193999999999999},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.40699999999999997,"5min":1.5549999999999999,"15min":1.4390000000000001},"min":{"1min":0.26900000000000002,"5min":0.23499999999999999,"15min":0.22800000000000001},"max":{"1min":0.627,"5min":16.033000000000001,"15min":16.033000000000001},"last":0.435}]},{"osd":4,"last update":"Wed Sep 11 08:31:57 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":1.387,"15min":1.79},"min":{"1min":0.38500000000000001,"5min":0.28100000000000003,"15min":0.254},"max":{"1min":0.76100000000000001,"5min":16.178000000000001,"15min":110.86499999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.56299999999999994,"5min":1.4490000000000001,"15min":1.913},"min":{"1min":0.42699999999999999,"5min":0.28799999999999998,"15min":0.28000000000000003},"max":{"1min":0.72399999999999998,"5min":19.009,"15min":110.831},"last":0.56000000000000005}]},{"osd":5,"last update":"Wed Sep 11 08:31:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":1.302,"15min":1.4650000000000001},"min":{"1min":0.442,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.80400000000000005,"5min":16.012,"15min":16.012},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.59799999999999998,"5min":1.423,"15min":1.486},"min":{"1min":0.42399999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.93400000000000005,"5min":16.206,"15min":16.206},"last":0.52400000000000002}]},{"osd":6,"last update":"Wed Sep 11 08:31:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":1.3999999999999999,"15min":1.369},"min":{"1min":0.41299999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.78100000000000003,"5min":16.143000000000001,"15min":16.143000000000001},"last":0.61199999999999999},{"interface":"front","average":{"1min":1.1080000000000001,"5min":1.4990000000000001,"15min":1.421},"min":{"1min":0.46500000000000002,"5min":0.34799999999999998,"15min":0.29799999999999999},"max":{"1min":10.552,"5min":15.984,"15min":15.984},"last":0.62}]},{"osd":7,"last update":"Wed Sep 11 08:31:57 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":1.5449999999999999,"15min":1.718},"min":{"1min":0.40400000000000003,"5min":0.32500000000000001,"15min":0.28599999999999998},"max":{"1min":0.82599999999999996,"5min":16.228999999999999,"15min":109.149},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":1.321,"15min":1.722},"min":{"1min":0.47399999999999998,"5min":0.32800000000000001,"15min":0.318},"max":{"1min":0.80100000000000005,"5min":16.170999999999999,"15min":109.309},"last":0.35399999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738705,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":899680,"kb_used_data":1424,"kb_used_omap":1,"kb_used_meta":898238,"kb_avail":92845472,"statfs":{"total":95995035648,"available":95073763328,"internally_reserved":0,"allocated":1458176,"data_stored":1020205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":919796056},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 11 08:31:41 2024","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":3.1419999999999999,"15min":1.7629999999999999},"min":{"1min":0.215,"5min":0.20399999999999999,"15min":0.159},"max":{"1min":0.69099999999999995,"5min":98.373999999999995,"15min":98.373999999999995},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.34599999999999997,"5min":3.484,"15min":1.9930000000000001},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.16600000000000001},"max":{"1min":0.72299999999999998,"5min":98.257999999999996,"15min":98.257999999999996},"last":0.47399999999999998}]},{"osd":2,"last update":"Wed Sep 11 08:32:12 2024","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":1.2769999999999999,"15min":1.383},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.44600000000000001,"5min":7.5270000000000001,"15min":46.417000000000002},"last":0.434},{"interface":"front","average":{"1min":0.32500000000000001,"5min":1.3759999999999999,"15min":1.466},"min":{"1min":0.221,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.40999999999999998,"5min":13.331,"15min":46.438000000000002},"last":0.45200000000000001}]},{"osd":3,"last update":"Wed Sep 11 08:31:33 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":1.4199999999999999,"15min":1.839},"min":{"1min":0.24399999999999999,"5min":0.222,"15min":0.222},"max":{"1min":0.47599999999999998,"5min":9.5180000000000007,"15min":172.149},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.34599999999999997,"5min":1.413,"15min":1.742},"min":{"1min":0.23300000000000001,"5min":0.22,"15min":0.20300000000000001},"max":{"1min":0.60899999999999999,"5min":9.0129999999999999,"15min":174.899},"last":0.51700000000000002}]},{"osd":4,"last update":"Wed Sep 11 08:31:57 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":1.0549999999999999,"15min":1.079},"min":{"1min":0.41899999999999998,"5min":0.26600000000000001,"15min":0.252},"max":{"1min":0.67800000000000005,"5min":5.1609999999999996,"15min":9.6560000000000006},"last":0.85199999999999998},{"interface":"front","average":{"1min":1.3069999999999999,"5min":1.113,"15min":1.1539999999999999},"min":{"1min":0.32500000000000001,"5min":0.29299999999999998,"15min":0.26400000000000001},"max":{"1min":13.666,"5min":13.666,"15min":13.666},"last":0.68799999999999994}]},{"osd":5,"last update":"Wed Sep 11 08:31:41 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":1.024,"15min":1.613},"min":{"1min":0.39100000000000001,"5min":0.28499999999999998,"15min":0.27400000000000002},"max":{"1min":0.66000000000000003,"5min":7.7030000000000003,"15min":7.7030000000000003},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.97199999999999998,"15min":1.484},"min":{"1min":0.376,"5min":0.311,"15min":0.309},"max":{"1min":0.755,"5min":5.3730000000000002,"15min":7.718},"last":0.76300000000000001}]},{"osd":6,"last update":"Wed Sep 11 08:31:41 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":1.1240000000000001,"15min":1.1359999999999999},"min":{"1min":0.39600000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.64200000000000002,"5min":5.1230000000000002,"15min":9.7029999999999994},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.99199999999999999,"15min":1.169},"min":{"1min":0.40899999999999997,"5min":0.315,"15min":0.28299999999999997},"max":{"1min":0.70199999999999996,"5min":5.3419999999999996,"15min":7.4820000000000002},"last":0.86499999999999999}]},{"osd":7,"last update":"Wed Sep 11 08:31:57 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":1.089,"15min":1.1299999999999999},"min":{"1min":0.36899999999999999,"5min":0.34499999999999997,"15min":0.30399999999999999},"max":{"1min":0.73199999999999998,"5min":5.8049999999999997,"15min":7.5350000000000001},"last":0.91800000000000004},{"interface":"front","average":{"1min":0.58199999999999996,"5min":1.0549999999999999,"15min":1.2350000000000001},"min":{"1min":0.378,"5min":0.378,"15min":0.32900000000000001},"max":{"1min":0.73999999999999999,"5min":5.6829999999999998,"15min":13.914},"last":0.88700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3425,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T08:32:27.092 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-11T08:32:27.092 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid c1bddff4-7013-11ef-bcea-c7b262605968 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-09-11T08:32:27.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:26 smithi181 ceph-mon[32701]: 6.c scrub starts 2024-09-11T08:32:27.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:26 smithi181 ceph-mon[32701]: 6.c scrub ok 2024-09-11T08:32:27.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:26 smithi181 ceph-mon[32701]: 6.18 scrub starts 2024-09-11T08:32:27.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:26 smithi181 ceph-mon[32701]: 6.18 scrub ok 2024-09-11T08:32:27.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:26 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:32:27.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:26 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:32:27.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:26 smithi181 ceph-mon[32701]: osdmap e481: 8 total, 8 up, 8 in 2024-09-11T08:32:27.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:26 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:32:27.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:26 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:32:27.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:26 smithi181 ceph-mon[32701]: from='client.16401 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:32:27.340 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:32:28.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:27 smithi150 ceph-mon[29276]: osdmap e482: 8 total, 8 up, 8 in 2024-09-11T08:32:28.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:27 smithi150 ceph-mon[29276]: pgmap v1142: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 943 B/s rd, 0 op/s 2024-09-11T08:32:28.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:27 smithi150 ceph-mon[29276]: 6.e deep-scrub starts 2024-09-11T08:32:28.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:27 smithi150 ceph-mon[29276]: 6.e deep-scrub ok 2024-09-11T08:32:28.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:27 smithi150 ceph-mon[39612]: osdmap e482: 8 total, 8 up, 8 in 2024-09-11T08:32:28.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:27 smithi150 ceph-mon[39612]: pgmap v1142: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 943 B/s rd, 0 op/s 2024-09-11T08:32:28.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:27 smithi150 ceph-mon[39612]: 6.e deep-scrub starts 2024-09-11T08:32:28.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:27 smithi150 ceph-mon[39612]: 6.e deep-scrub ok 2024-09-11T08:32:28.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:27 smithi181 ceph-mon[32701]: osdmap e482: 8 total, 8 up, 8 in 2024-09-11T08:32:28.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:27 smithi181 ceph-mon[32701]: pgmap v1142: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 943 B/s rd, 0 op/s 2024-09-11T08:32:28.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:27 smithi181 ceph-mon[32701]: 6.e deep-scrub starts 2024-09-11T08:32:28.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:27 smithi181 ceph-mon[32701]: 6.e deep-scrub ok 2024-09-11T08:32:28.786 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:32:28 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:32:28] "GET /metrics HTTP/1.1" 200 222288 "" "Prometheus/2.43.0" 2024-09-11T08:32:29.078 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:28 smithi150 ceph-mon[29276]: purged_snaps scrub starts 2024-09-11T08:32:29.078 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:28 smithi150 ceph-mon[29276]: purged_snaps scrub ok 2024-09-11T08:32:29.078 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:28 smithi150 ceph-mon[29276]: 6.8 scrub starts 2024-09-11T08:32:29.078 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:28 smithi150 ceph-mon[29276]: 6.8 scrub ok 2024-09-11T08:32:29.078 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:28 smithi150 ceph-mon[39612]: purged_snaps scrub starts 2024-09-11T08:32:29.079 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:28 smithi150 ceph-mon[39612]: purged_snaps scrub ok 2024-09-11T08:32:29.079 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:28 smithi150 ceph-mon[39612]: 6.8 scrub starts 2024-09-11T08:32:29.079 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:28 smithi150 ceph-mon[39612]: 6.8 scrub ok 2024-09-11T08:32:29.214 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:28 smithi181 ceph-mon[32701]: purged_snaps scrub starts 2024-09-11T08:32:29.214 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:28 smithi181 ceph-mon[32701]: purged_snaps scrub ok 2024-09-11T08:32:29.214 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:28 smithi181 ceph-mon[32701]: 6.8 scrub starts 2024-09-11T08:32:29.214 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:28 smithi181 ceph-mon[32701]: 6.8 scrub ok 2024-09-11T08:32:29.573 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:32:29 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug there is no tcmu-runner data available 2024-09-11T08:32:29.648 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:32:29 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: [11/Sep/2024:08:32:29] ENGINE Bus STOPPING 2024-09-11T08:32:29.648 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:32:29 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: [11/Sep/2024:08:32:29] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-11T08:32:29.648 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:32:29 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: [11/Sep/2024:08:32:29] ENGINE Bus STOPPED 2024-09-11T08:32:29.648 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:32:29 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: [11/Sep/2024:08:32:29] ENGINE Bus STARTING 2024-09-11T08:32:29.988 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:32:29 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: [11/Sep/2024:08:32:29] ENGINE Serving on http://:::9283 2024-09-11T08:32:29.988 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:32:29 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: [11/Sep/2024:08:32:29] ENGINE Bus STARTED 2024-09-11T08:32:29.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:29 smithi150 ceph-mon[39612]: 5.1f scrub starts 2024-09-11T08:32:29.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:29 smithi150 ceph-mon[39612]: 5.1f scrub ok 2024-09-11T08:32:29.988 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:29 smithi150 ceph-mon[39612]: osdmap e483: 8 total, 8 up, 8 in 2024-09-11T08:32:29.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:29 smithi150 ceph-mon[39612]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:32:29.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:29 smithi150 ceph-mon[39612]: pgmap v1144: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 944 B/s rd, 0 op/s 2024-09-11T08:32:29.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:29 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:32:29.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:29 smithi150 ceph-mon[39612]: from='client.? 172.21.15.150:0/3446506239' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-09-11T08:32:29.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:29 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-09-11T08:32:29.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:29 smithi150 ceph-mon[39612]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-11T08:32:29.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:29 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:32:29.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:29 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:32:29.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:29 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:32:29.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:29 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:32:29.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:29 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:32:29.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:29 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:32:29.989 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:29 smithi150 ceph-mon[39612]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:32:29.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:29 smithi150 ceph-mon[29276]: 5.1f scrub starts 2024-09-11T08:32:29.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:29 smithi150 ceph-mon[29276]: 5.1f scrub ok 2024-09-11T08:32:29.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:29 smithi150 ceph-mon[29276]: osdmap e483: 8 total, 8 up, 8 in 2024-09-11T08:32:29.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:29 smithi150 ceph-mon[29276]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:32:29.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:29 smithi150 ceph-mon[29276]: pgmap v1144: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 944 B/s rd, 0 op/s 2024-09-11T08:32:29.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:29 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:32:29.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:29 smithi150 ceph-mon[29276]: from='client.? 172.21.15.150:0/3446506239' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-09-11T08:32:29.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:29 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-09-11T08:32:29.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:29 smithi150 ceph-mon[29276]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-11T08:32:29.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:29 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:32:29.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:29 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:32:29.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:29 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:32:29.991 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:29 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:32:29.991 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:29 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:32:29.991 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:29 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:32:29.991 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:29 smithi150 ceph-mon[29276]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:32:30.133 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-11T08:32:30.143 INFO:tasks.cephadm:Teardown begin 2024-09-11T08:32:30.144 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-11T08:32:30.170 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-11T08:32:30.194 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:29 smithi181 ceph-mon[32701]: 5.1f scrub starts 2024-09-11T08:32:30.194 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:29 smithi181 ceph-mon[32701]: 5.1f scrub ok 2024-09-11T08:32:30.194 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:29 smithi181 ceph-mon[32701]: osdmap e483: 8 total, 8 up, 8 in 2024-09-11T08:32:30.194 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:29 smithi181 ceph-mon[32701]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:32:30.194 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:29 smithi181 ceph-mon[32701]: pgmap v1144: 132 pgs: 132 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 944 B/s rd, 0 op/s 2024-09-11T08:32:30.195 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:29 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:32:30.195 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:29 smithi181 ceph-mon[32701]: from='client.? 172.21.15.150:0/3446506239' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-09-11T08:32:30.195 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:29 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-09-11T08:32:30.195 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:29 smithi181 ceph-mon[32701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-11T08:32:30.195 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:29 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:32:30.195 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:29 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:32:30.195 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:29 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:32:30.195 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:29 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:32:30.195 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:29 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:32:30.195 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:29 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:32:30.195 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:29 smithi181 ceph-mon[32701]: from='mgr.14538 172.21.15.150:0/2120198156' entity='mgr.y' 2024-09-11T08:32:30.198 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-11T08:32:30.199 DEBUG:teuthology.orchestra.run.smithi150:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-11T08:32:30.226 DEBUG:teuthology.orchestra.run.smithi181:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-11T08:32:30.254 INFO:tasks.cephadm:Stopping all daemons... 2024-09-11T08:32:30.254 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-09-11T08:32:30.254 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-c1bddff4-7013-11ef-bcea-c7b262605968@mon.a 2024-09-11T08:32:30.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:30 smithi150 systemd[1]: Stopping Ceph mon.a for c1bddff4-7013-11ef-bcea-c7b262605968... 2024-09-11T08:32:30.963 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:30 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mon-a[29272]: 2024-09-11T08:32:30.578+0000 7f1b1d36c640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-11T08:32:30.963 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:30 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mon-a[29272]: 2024-09-11T08:32:30.578+0000 7f1b1d36c640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-11T08:32:30.963 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:30 smithi150 podman[218012]: 2024-09-11 08:32:30.671522644 +0000 UTC m=+0.220413339 container died d7334d5149102c624f81121c01d70bd96d37a60f0157b33fa338f4caf9ffa9de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mon-a, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=) 2024-09-11T08:32:30.963 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:32:30 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: [11/Sep/2024:08:32:30] ENGINE Bus STOPPING 2024-09-11T08:32:30.963 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:32:30 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: [11/Sep/2024:08:32:30] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-11T08:32:30.963 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:32:30 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: [11/Sep/2024:08:32:30] ENGINE Bus STOPPED 2024-09-11T08:32:30.963 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:32:30 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: [11/Sep/2024:08:32:30] ENGINE Bus STARTING 2024-09-11T08:32:30.963 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:32:30 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: [11/Sep/2024:08:32:30] ENGINE Serving on http://:::9283 2024-09-11T08:32:30.964 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:32:30 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: [11/Sep/2024:08:32:30] ENGINE Bus STARTED 2024-09-11T08:32:31.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:31 smithi150 podman[218012]: 2024-09-11 08:32:31.15457718 +0000 UTC m=+0.703467875 container cleanup d7334d5149102c624f81121c01d70bd96d37a60f0157b33fa338f4caf9ffa9de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mon-a, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-11T08:32:31.238 INFO:journalctl@ceph.mon.a.smithi150.stdout:Sep 11 08:32:31 smithi150 bash[218012]: ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mon-a 2024-09-11T08:32:31.555 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c1bddff4-7013-11ef-bcea-c7b262605968@mon.a.service' 2024-09-11T08:32:31.604 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-11T08:32:31.605 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-09-11T08:32:31.605 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-09-11T08:32:31.605 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-c1bddff4-7013-11ef-bcea-c7b262605968@mon.c 2024-09-11T08:32:31.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:31 smithi150 systemd[1]: Stopping Ceph mon.c for c1bddff4-7013-11ef-bcea-c7b262605968... 2024-09-11T08:32:32.238 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:32:31 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: [11/Sep/2024:08:32:31] ENGINE Bus STOPPING 2024-09-11T08:32:32.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:31 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mon-c[39608]: 2024-09-11T08:32:31.891+0000 7f55b414d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-11T08:32:32.238 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:31 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mon-c[39608]: 2024-09-11T08:32:31.891+0000 7f55b414d640 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-09-11T08:32:32.239 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:31 smithi150 podman[218143]: 2024-09-11 08:32:31.9948461 +0000 UTC m=+0.201045150 container died a2ca8ac978f75d735e866b17a98f9ac64aa00dbac8746a97bb511813766705c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mon-c, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.schema-version=1.0) 2024-09-11T08:32:32.559 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:32:32 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: [11/Sep/2024:08:32:32] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-11T08:32:32.559 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:32:32 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: [11/Sep/2024:08:32:32] ENGINE Bus STOPPED 2024-09-11T08:32:32.559 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:32:32 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: [11/Sep/2024:08:32:32] ENGINE Bus STARTING 2024-09-11T08:32:32.559 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:32:32 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: [11/Sep/2024:08:32:32] ENGINE Serving on http://:::9283 2024-09-11T08:32:32.559 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:32:32 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: [11/Sep/2024:08:32:32] ENGINE Bus STARTED 2024-09-11T08:32:32.854 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:32 smithi150 podman[218143]: 2024-09-11 08:32:32.559802359 +0000 UTC m=+0.766001401 container cleanup a2ca8ac978f75d735e866b17a98f9ac64aa00dbac8746a97bb511813766705c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mon-c, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-11T08:32:32.854 INFO:journalctl@ceph.mon.c.smithi150.stdout:Sep 11 08:32:32 smithi150 bash[218143]: ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mon-c 2024-09-11T08:32:32.996 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c1bddff4-7013-11ef-bcea-c7b262605968@mon.c.service' 2024-09-11T08:32:33.031 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-11T08:32:33.031 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-09-11T08:32:33.031 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-09-11T08:32:33.031 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-c1bddff4-7013-11ef-bcea-c7b262605968@mon.b 2024-09-11T08:32:33.322 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:33 smithi181 systemd[1]: Stopping Ceph mon.b for c1bddff4-7013-11ef-bcea-c7b262605968... 2024-09-11T08:32:33.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:33 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mon-b[32684]: 2024-09-11T08:32:33.451+0000 7f94e4ca3640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-11T08:32:33.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:33 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mon-b[32684]: 2024-09-11T08:32:33.451+0000 7f94e4ca3640 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-09-11T08:32:33.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:33 smithi181 podman[162983]: 2024-09-11 08:32:33.614503741 +0000 UTC m=+0.286477315 container died 0b11e8513d3800890de360ff760a72ab4a9683fd1b614e7a71c468ff60d510b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mon-b, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-11T08:32:34.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:34 smithi181 podman[162983]: 2024-09-11 08:32:34.109857422 +0000 UTC m=+0.781830995 container cleanup 0b11e8513d3800890de360ff760a72ab4a9683fd1b614e7a71c468ff60d510b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mon-b, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828) 2024-09-11T08:32:34.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Sep 11 08:32:34 smithi181 bash[162983]: ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mon-b 2024-09-11T08:32:34.530 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c1bddff4-7013-11ef-bcea-c7b262605968@mon.b.service' 2024-09-11T08:32:34.568 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-11T08:32:34.568 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-09-11T08:32:34.569 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-09-11T08:32:34.569 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-c1bddff4-7013-11ef-bcea-c7b262605968@mgr.y 2024-09-11T08:32:34.882 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:32:34 smithi150 systemd[1]: Stopping Ceph mgr.y for c1bddff4-7013-11ef-bcea-c7b262605968... 2024-09-11T08:32:35.210 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:32:34 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:32:34.882+0000 7fb0da175640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-11T08:32:35.210 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:32:34 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y[29503]: 2024-09-11T08:32:34.882+0000 7fb0da175640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-09-11T08:32:35.210 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:32:34 smithi150 podman[218275]: 2024-09-11 08:32:34.91346471 +0000 UTC m=+0.150814018 container died 995a40ad9a2ce775e2bdf0bf07fb082a87e9433284309f21045f29866f625a66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-11T08:32:35.322 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:32:35 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x[34560]: ::ffff:172.21.15.181 - - [11/Sep/2024:08:32:35] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:32:35.488 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:32:35 smithi150 podman[218275]: 2024-09-11 08:32:35.343046114 +0000 UTC m=+0.580395422 container cleanup 995a40ad9a2ce775e2bdf0bf07fb082a87e9433284309f21045f29866f625a66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-11T08:32:35.488 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 11 08:32:35 smithi150 bash[218275]: ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-y 2024-09-11T08:32:35.824 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c1bddff4-7013-11ef-bcea-c7b262605968@mgr.y.service' 2024-09-11T08:32:35.861 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-11T08:32:35.861 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-09-11T08:32:35.861 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-09-11T08:32:35.862 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-c1bddff4-7013-11ef-bcea-c7b262605968@mgr.x 2024-09-11T08:32:36.219 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:32:35 smithi181 systemd[1]: Stopping Ceph mgr.x for c1bddff4-7013-11ef-bcea-c7b262605968... 2024-09-11T08:32:36.544 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:32:36 smithi181 podman[163107]: 2024-09-11 08:32:36.21951886 +0000 UTC m=+0.157088883 container died 6ab82f1ddf8cf4de7d3f22d687e7ebdf4c23a1865b4e1db04a231702afad67ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-11T08:32:36.823 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:32:36 smithi181 podman[163107]: 2024-09-11 08:32:36.74897035 +0000 UTC m=+0.686540363 container cleanup 6ab82f1ddf8cf4de7d3f22d687e7ebdf4c23a1865b4e1db04a231702afad67ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731) 2024-09-11T08:32:36.823 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:32:36 smithi181 bash[163107]: ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x 2024-09-11T08:32:37.225 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c1bddff4-7013-11ef-bcea-c7b262605968@mgr.x.service' 2024-09-11T08:32:37.247 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:32:36 smithi181 podman[163122]: 2024-09-11 08:32:36.97570083 +0000 UTC m=+0.742902757 container remove 6ab82f1ddf8cf4de7d3f22d687e7ebdf4c23a1865b4e1db04a231702afad67ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-mgr-x, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-11T08:32:37.247 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:32:36 smithi181 systemd[1]: ceph-c1bddff4-7013-11ef-bcea-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-09-11T08:32:37.247 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:32:37 smithi181 systemd[1]: ceph-c1bddff4-7013-11ef-bcea-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-09-11T08:32:37.247 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:32:37 smithi181 systemd[1]: Stopped Ceph mgr.x for c1bddff4-7013-11ef-bcea-c7b262605968. 2024-09-11T08:32:37.247 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Sep 11 08:32:37 smithi181 systemd[1]: ceph-c1bddff4-7013-11ef-bcea-c7b262605968@mgr.x.service: Consumed 11.512s CPU time. 2024-09-11T08:32:37.260 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-11T08:32:37.261 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-09-11T08:32:37.261 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-11T08:32:37.261 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-c1bddff4-7013-11ef-bcea-c7b262605968@osd.0 2024-09-11T08:32:37.556 INFO:journalctl@ceph.osd.0.smithi150.stdout:Sep 11 08:32:37 smithi150 systemd[1]: Stopping Ceph osd.0 for c1bddff4-7013-11ef-bcea-c7b262605968... 2024-09-11T08:32:37.988 INFO:journalctl@ceph.osd.0.smithi150.stdout:Sep 11 08:32:37 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-0[43516]: 2024-09-11T08:32:37.555+0000 7fdc65dc9640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-11T08:32:37.988 INFO:journalctl@ceph.osd.0.smithi150.stdout:Sep 11 08:32:37 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-0[43516]: 2024-09-11T08:32:37.555+0000 7fdc65dc9640 -1 osd.0 483 *** Got signal Terminated *** 2024-09-11T08:32:37.988 INFO:journalctl@ceph.osd.0.smithi150.stdout:Sep 11 08:32:37 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-0[43516]: 2024-09-11T08:32:37.555+0000 7fdc65dc9640 -1 osd.0 483 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-11T08:32:43.517 INFO:journalctl@ceph.osd.0.smithi150.stdout:Sep 11 08:32:43 smithi150 podman[218399]: 2024-09-11 08:32:43.248649964 +0000 UTC m=+5.788911887 container died 2279161c1c493db607081f7bc65fd6b5274687b6f73c2921a1cd64a5221fbd15 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-0, CEPH_POINT_RELEASE=, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-11T08:32:43.890 INFO:journalctl@ceph.osd.0.smithi150.stdout:Sep 11 08:32:43 smithi150 podman[218399]: 2024-09-11 08:32:43.689634451 +0000 UTC m=+6.229896366 container cleanup 2279161c1c493db607081f7bc65fd6b5274687b6f73c2921a1cd64a5221fbd15 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-09-11T08:32:43.891 INFO:journalctl@ceph.osd.0.smithi150.stdout:Sep 11 08:32:43 smithi150 bash[218399]: ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-0 2024-09-11T08:32:44.238 INFO:journalctl@ceph.osd.0.smithi150.stdout:Sep 11 08:32:43 smithi150 podman[218414]: 2024-09-11 08:32:43.891206366 +0000 UTC m=+0.638700726 container remove 2279161c1c493db607081f7bc65fd6b5274687b6f73c2921a1cd64a5221fbd15 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-0, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-11T08:32:44.569 INFO:journalctl@ceph.osd.0.smithi150.stdout:Sep 11 08:32:44 smithi150 podman[218505]: 2024-09-11 08:32:44.20160066 +0000 UTC m=+0.016580896 image pull 4a918172a49766385959227c53e75e2c302e923539324792cb5bee6cad0b2f2b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40 2024-09-11T08:32:44.569 INFO:journalctl@ceph.osd.0.smithi150.stdout:Sep 11 08:32:44 smithi150 podman[218505]: 2024-09-11 08:32:44.32560195 +0000 UTC m=+0.140582185 container create 7fb1f1dcbd9c0671d33b82b3c6c95f89ac62be16a8d1f2d8e34e700c0481ea28 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-0-deactivate, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True) 2024-09-11T08:32:44.569 INFO:journalctl@ceph.osd.0.smithi150.stdout:Sep 11 08:32:44 smithi150 podman[218505]: 2024-09-11 08:32:44.569601893 +0000 UTC m=+0.384582137 container init 7fb1f1dcbd9c0671d33b82b3c6c95f89ac62be16a8d1f2d8e34e700c0481ea28 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-11T08:32:44.988 INFO:journalctl@ceph.osd.0.smithi150.stdout:Sep 11 08:32:44 smithi150 podman[218505]: 2024-09-11 08:32:44.573142677 +0000 UTC m=+0.388122913 container start 7fb1f1dcbd9c0671d33b82b3c6c95f89ac62be16a8d1f2d8e34e700c0481ea28 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-0-deactivate, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux ) 2024-09-11T08:32:44.988 INFO:journalctl@ceph.osd.0.smithi150.stdout:Sep 11 08:32:44 smithi150 podman[218505]: 2024-09-11 08:32:44.669736947 +0000 UTC m=+0.484717185 container attach 7fb1f1dcbd9c0671d33b82b3c6c95f89ac62be16a8d1f2d8e34e700c0481ea28 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, io.buildah.version=1.37.2) 2024-09-11T08:32:44.988 INFO:journalctl@ceph.osd.0.smithi150.stdout:Sep 11 08:32:44 smithi150 podman[218505]: 2024-09-11 08:32:44.815537416 +0000 UTC m=+0.630517654 container died 7fb1f1dcbd9c0671d33b82b3c6c95f89ac62be16a8d1f2d8e34e700c0481ea28 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True) 2024-09-11T08:32:45.554 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c1bddff4-7013-11ef-bcea-c7b262605968@osd.0.service' 2024-09-11T08:32:45.598 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-11T08:32:45.598 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-11T08:32:45.598 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-11T08:32:45.599 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-c1bddff4-7013-11ef-bcea-c7b262605968@osd.1 2024-09-11T08:32:45.956 INFO:journalctl@ceph.osd.1.smithi150.stdout:Sep 11 08:32:45 smithi150 systemd[1]: Stopping Ceph osd.1 for c1bddff4-7013-11ef-bcea-c7b262605968... 2024-09-11T08:32:46.238 INFO:journalctl@ceph.osd.1.smithi150.stdout:Sep 11 08:32:45 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-1[49526]: 2024-09-11T08:32:45.956+0000 7f3b89aff640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-11T08:32:46.238 INFO:journalctl@ceph.osd.1.smithi150.stdout:Sep 11 08:32:45 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-1[49526]: 2024-09-11T08:32:45.956+0000 7f3b89aff640 -1 osd.1 483 *** Got signal Terminated *** 2024-09-11T08:32:46.238 INFO:journalctl@ceph.osd.1.smithi150.stdout:Sep 11 08:32:45 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-1[49526]: 2024-09-11T08:32:45.956+0000 7f3b89aff640 -1 osd.1 483 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-11T08:32:51.792 INFO:journalctl@ceph.osd.1.smithi150.stdout:Sep 11 08:32:51 smithi150 podman[218609]: 2024-09-11 08:32:51.507830732 +0000 UTC m=+5.680003822 container died 1140fff6ebf463fab18190a46818da6b0f88f48b07971bd92307b26352b13d5a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-1, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828) 2024-09-11T08:32:52.238 INFO:journalctl@ceph.osd.1.smithi150.stdout:Sep 11 08:32:51 smithi150 podman[218609]: 2024-09-11 08:32:51.959632225 +0000 UTC m=+6.131805314 container cleanup 1140fff6ebf463fab18190a46818da6b0f88f48b07971bd92307b26352b13d5a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-1, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-11T08:32:52.238 INFO:journalctl@ceph.osd.1.smithi150.stdout:Sep 11 08:32:51 smithi150 bash[218609]: ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-1 2024-09-11T08:32:52.619 INFO:journalctl@ceph.osd.1.smithi150.stdout:Sep 11 08:32:52 smithi150 podman[218624]: 2024-09-11 08:32:52.294471237 +0000 UTC m=+0.780468887 container remove 1140fff6ebf463fab18190a46818da6b0f88f48b07971bd92307b26352b13d5a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-11T08:32:52.914 INFO:journalctl@ceph.osd.1.smithi150.stdout:Sep 11 08:32:52 smithi150 podman[218715]: 2024-09-11 08:32:52.522942246 +0000 UTC m=+0.015859524 image pull 4a918172a49766385959227c53e75e2c302e923539324792cb5bee6cad0b2f2b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40 2024-09-11T08:32:52.914 INFO:journalctl@ceph.osd.1.smithi150.stdout:Sep 11 08:32:52 smithi150 podman[218715]: 2024-09-11 08:32:52.637256158 +0000 UTC m=+0.130173419 container create 17ac57085cfc41d18c0ad0452cf6e556e96a3b718448dca72e949a0e5e53a6bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, ceph=True, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-11T08:32:52.914 INFO:journalctl@ceph.osd.1.smithi150.stdout:Sep 11 08:32:52 smithi150 podman[218715]: 2024-09-11 08:32:52.914737533 +0000 UTC m=+0.407654796 container init 17ac57085cfc41d18c0ad0452cf6e556e96a3b718448dca72e949a0e5e53a6bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-1-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-09-11T08:32:53.238 INFO:journalctl@ceph.osd.1.smithi150.stdout:Sep 11 08:32:52 smithi150 podman[218715]: 2024-09-11 08:32:52.918177947 +0000 UTC m=+0.411095212 container start 17ac57085cfc41d18c0ad0452cf6e556e96a3b718448dca72e949a0e5e53a6bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-11T08:32:53.238 INFO:journalctl@ceph.osd.1.smithi150.stdout:Sep 11 08:32:53 smithi150 podman[218715]: 2024-09-11 08:32:53.006605036 +0000 UTC m=+0.499522299 container attach 17ac57085cfc41d18c0ad0452cf6e556e96a3b718448dca72e949a0e5e53a6bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-1-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.37.2) 2024-09-11T08:32:53.239 INFO:journalctl@ceph.osd.1.smithi150.stdout:Sep 11 08:32:53 smithi150 podman[218715]: 2024-09-11 08:32:53.093984301 +0000 UTC m=+0.586901580 container died 17ac57085cfc41d18c0ad0452cf6e556e96a3b718448dca72e949a0e5e53a6bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828) 2024-09-11T08:32:53.794 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c1bddff4-7013-11ef-bcea-c7b262605968@osd.1.service' 2024-09-11T08:32:53.833 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-11T08:32:53.833 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-11T08:32:53.833 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-11T08:32:53.833 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-c1bddff4-7013-11ef-bcea-c7b262605968@osd.2 2024-09-11T08:32:54.146 INFO:journalctl@ceph.osd.2.smithi150.stdout:Sep 11 08:32:53 smithi150 systemd[1]: Stopping Ceph osd.2 for c1bddff4-7013-11ef-bcea-c7b262605968... 2024-09-11T08:32:54.488 INFO:journalctl@ceph.osd.2.smithi150.stdout:Sep 11 08:32:54 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-2[55516]: 2024-09-11T08:32:54.146+0000 7f8300819640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-11T08:32:54.488 INFO:journalctl@ceph.osd.2.smithi150.stdout:Sep 11 08:32:54 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-2[55516]: 2024-09-11T08:32:54.146+0000 7f8300819640 -1 osd.2 483 *** Got signal Terminated *** 2024-09-11T08:32:54.489 INFO:journalctl@ceph.osd.2.smithi150.stdout:Sep 11 08:32:54 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-2[55516]: 2024-09-11T08:32:54.146+0000 7f8300819640 -1 osd.2 483 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-11T08:32:59.988 INFO:journalctl@ceph.osd.2.smithi150.stdout:Sep 11 08:32:59 smithi150 podman[218820]: 2024-09-11 08:32:59.717758567 +0000 UTC m=+5.666046905 container died 9c2f5f67da5ad1566e2bc2779aa90bdfa7d62a4f4d9ff9edb68d8442a01957bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, io.buildah.version=1.37.2) 2024-09-11T08:33:00.457 INFO:journalctl@ceph.osd.2.smithi150.stdout:Sep 11 08:33:00 smithi150 podman[218820]: 2024-09-11 08:33:00.223351886 +0000 UTC m=+6.171640224 container cleanup 9c2f5f67da5ad1566e2bc2779aa90bdfa7d62a4f4d9ff9edb68d8442a01957bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-11T08:33:00.458 INFO:journalctl@ceph.osd.2.smithi150.stdout:Sep 11 08:33:00 smithi150 bash[218820]: ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-2 2024-09-11T08:33:00.738 INFO:journalctl@ceph.osd.2.smithi150.stdout:Sep 11 08:33:00 smithi150 podman[218837]: 2024-09-11 08:33:00.45808105 +0000 UTC m=+0.735581374 container remove 9c2f5f67da5ad1566e2bc2779aa90bdfa7d62a4f4d9ff9edb68d8442a01957bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-2, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-09-11T08:33:01.060 INFO:journalctl@ceph.osd.2.smithi150.stdout:Sep 11 08:33:00 smithi150 podman[218926]: 2024-09-11 08:33:00.667453787 +0000 UTC m=+0.015697916 image pull 4a918172a49766385959227c53e75e2c302e923539324792cb5bee6cad0b2f2b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40 2024-09-11T08:33:01.061 INFO:journalctl@ceph.osd.2.smithi150.stdout:Sep 11 08:33:00 smithi150 podman[218926]: 2024-09-11 08:33:00.775594732 +0000 UTC m=+0.123838844 container create a7039ac2862e58280d95864336d23552d7e5d7434cccfb099c1f4b9ecc227837 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-11T08:33:01.061 INFO:journalctl@ceph.osd.2.smithi150.stdout:Sep 11 08:33:00 smithi150 podman[218926]: 2024-09-11 08:33:00.977847597 +0000 UTC m=+0.326091720 container init a7039ac2862e58280d95864336d23552d7e5d7434cccfb099c1f4b9ecc227837 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-11T08:33:01.061 INFO:journalctl@ceph.osd.2.smithi150.stdout:Sep 11 08:33:00 smithi150 podman[218926]: 2024-09-11 08:33:00.981301095 +0000 UTC m=+0.329545209 container start a7039ac2862e58280d95864336d23552d7e5d7434cccfb099c1f4b9ecc227837 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-2-deactivate, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-11T08:33:01.455 INFO:journalctl@ceph.osd.2.smithi150.stdout:Sep 11 08:33:01 smithi150 podman[218926]: 2024-09-11 08:33:01.061099479 +0000 UTC m=+0.409343595 container attach a7039ac2862e58280d95864336d23552d7e5d7434cccfb099c1f4b9ecc227837 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-09-11T08:33:01.456 INFO:journalctl@ceph.osd.2.smithi150.stdout:Sep 11 08:33:01 smithi150 podman[218926]: 2024-09-11 08:33:01.163046212 +0000 UTC m=+0.511290328 container died a7039ac2862e58280d95864336d23552d7e5d7434cccfb099c1f4b9ecc227837 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-2-deactivate, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-09-11T08:33:01.828 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c1bddff4-7013-11ef-bcea-c7b262605968@osd.2.service' 2024-09-11T08:33:01.867 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-11T08:33:01.867 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-11T08:33:01.867 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-11T08:33:01.867 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-c1bddff4-7013-11ef-bcea-c7b262605968@osd.3 2024-09-11T08:33:02.200 INFO:journalctl@ceph.osd.3.smithi150.stdout:Sep 11 08:33:01 smithi150 systemd[1]: Stopping Ceph osd.3 for c1bddff4-7013-11ef-bcea-c7b262605968... 2024-09-11T08:33:02.488 INFO:journalctl@ceph.osd.3.smithi150.stdout:Sep 11 08:33:02 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-3[61128]: 2024-09-11T08:33:02.200+0000 7f7c29ded640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-11T08:33:02.488 INFO:journalctl@ceph.osd.3.smithi150.stdout:Sep 11 08:33:02 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-3[61128]: 2024-09-11T08:33:02.200+0000 7f7c29ded640 -1 osd.3 483 *** Got signal Terminated *** 2024-09-11T08:33:02.488 INFO:journalctl@ceph.osd.3.smithi150.stdout:Sep 11 08:33:02 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-3[61128]: 2024-09-11T08:33:02.200+0000 7f7c29ded640 -1 osd.3 483 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-11T08:33:04.573 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:04 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:04.091+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:41.728731+0000 front 2024-09-11T08:32:41.728774+0000 (oldest deadline 2024-09-11T08:33:03.428644+0000) 2024-09-11T08:33:05.324 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:05 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:05.242+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:40.703121+0000 front 2024-09-11T08:32:40.703110+0000 (oldest deadline 2024-09-11T08:33:04.803032+0000) 2024-09-11T08:33:05.324 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:05 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:05.072+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:41.728731+0000 front 2024-09-11T08:32:41.728774+0000 (oldest deadline 2024-09-11T08:33:03.428644+0000) 2024-09-11T08:33:05.589 INFO:journalctl@ceph.osd.4.smithi181.stdout:Sep 11 08:33:05 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-4[37497]: 2024-09-11T08:33:05.323+0000 7f104189f640 -1 osd.4 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:40.435809+0000 front 2024-09-11T08:32:40.435988+0000 (oldest deadline 2024-09-11T08:33:05.135689+0000) 2024-09-11T08:33:06.268 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:06 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:06.083+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:41.728731+0000 front 2024-09-11T08:32:41.728774+0000 (oldest deadline 2024-09-11T08:33:03.428644+0000) 2024-09-11T08:33:06.573 INFO:journalctl@ceph.osd.4.smithi181.stdout:Sep 11 08:33:06 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-4[37497]: 2024-09-11T08:33:06.292+0000 7f104189f640 -1 osd.4 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:40.435809+0000 front 2024-09-11T08:32:40.435988+0000 (oldest deadline 2024-09-11T08:33:05.135689+0000) 2024-09-11T08:33:06.573 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:06 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:06.267+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:40.703121+0000 front 2024-09-11T08:32:40.703110+0000 (oldest deadline 2024-09-11T08:33:04.803032+0000) 2024-09-11T08:33:07.323 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:07 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:07.259+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:40.703121+0000 front 2024-09-11T08:32:40.703110+0000 (oldest deadline 2024-09-11T08:33:04.803032+0000) 2024-09-11T08:33:07.323 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:07 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:07.108+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:41.728731+0000 front 2024-09-11T08:32:41.728774+0000 (oldest deadline 2024-09-11T08:33:03.428644+0000) 2024-09-11T08:33:07.324 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:33:07 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5[135300]: 2024-09-11T08:33:07.072+0000 7f294bb2b640 -1 osd.5 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:41.094295+0000 front 2024-09-11T08:32:41.094335+0000 (oldest deadline 2024-09-11T08:33:06.394181+0000) 2024-09-11T08:33:07.822 INFO:journalctl@ceph.osd.4.smithi181.stdout:Sep 11 08:33:07 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-4[37497]: 2024-09-11T08:33:07.323+0000 7f104189f640 -1 osd.4 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:40.435809+0000 front 2024-09-11T08:32:40.435988+0000 (oldest deadline 2024-09-11T08:33:05.135689+0000) 2024-09-11T08:33:07.881 INFO:journalctl@ceph.osd.3.smithi150.stdout:Sep 11 08:33:07 smithi150 podman[219030]: 2024-09-11 08:33:07.615233285 +0000 UTC m=+5.524325277 container died a4d64d62738e3e965e978abada2bf057f09ec67ae53943566e9a73977f307f00 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-3, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, io.buildah.version=1.37.2, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-11T08:33:08.238 INFO:journalctl@ceph.osd.3.smithi150.stdout:Sep 11 08:33:08 smithi150 podman[219030]: 2024-09-11 08:33:08.050338184 +0000 UTC m=+5.959430170 container cleanup a4d64d62738e3e965e978abada2bf057f09ec67ae53943566e9a73977f307f00 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-3, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2) 2024-09-11T08:33:08.238 INFO:journalctl@ceph.osd.3.smithi150.stdout:Sep 11 08:33:08 smithi150 bash[219030]: ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-3 2024-09-11T08:33:08.323 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:08 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:08.238+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:40.703121+0000 front 2024-09-11T08:32:40.703110+0000 (oldest deadline 2024-09-11T08:33:04.803032+0000) 2024-09-11T08:33:08.323 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:08 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:08.156+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:41.728731+0000 front 2024-09-11T08:32:41.728774+0000 (oldest deadline 2024-09-11T08:33:03.428644+0000) 2024-09-11T08:33:08.323 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:33:08 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5[135300]: 2024-09-11T08:33:08.029+0000 7f294bb2b640 -1 osd.5 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:41.094295+0000 front 2024-09-11T08:32:41.094335+0000 (oldest deadline 2024-09-11T08:33:06.394181+0000) 2024-09-11T08:33:08.738 INFO:journalctl@ceph.osd.3.smithi150.stdout:Sep 11 08:33:08 smithi150 podman[219048]: 2024-09-11 08:33:08.352129524 +0000 UTC m=+0.736780403 container remove a4d64d62738e3e965e978abada2bf057f09ec67ae53943566e9a73977f307f00 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-3, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-09-11T08:33:08.822 INFO:journalctl@ceph.osd.4.smithi181.stdout:Sep 11 08:33:08 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-4[37497]: 2024-09-11T08:33:08.326+0000 7f104189f640 -1 osd.4 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:40.435809+0000 front 2024-09-11T08:32:40.435988+0000 (oldest deadline 2024-09-11T08:33:05.135689+0000) 2024-09-11T08:33:09.054 INFO:journalctl@ceph.osd.3.smithi150.stdout:Sep 11 08:33:08 smithi150 podman[219135]: 2024-09-11 08:33:08.705820611 +0000 UTC m=+0.016109620 image pull 4a918172a49766385959227c53e75e2c302e923539324792cb5bee6cad0b2f2b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40 2024-09-11T08:33:09.055 INFO:journalctl@ceph.osd.3.smithi150.stdout:Sep 11 08:33:08 smithi150 podman[219135]: 2024-09-11 08:33:08.82793743 +0000 UTC m=+0.138226474 container create e90c73d3a92f5421b64fc2243d2c7f599816610eef1616134dfe18afb1c7544c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-3-deactivate, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-11T08:33:09.055 INFO:journalctl@ceph.osd.3.smithi150.stdout:Sep 11 08:33:09 smithi150 podman[219135]: 2024-09-11 08:33:09.055352191 +0000 UTC m=+0.365641190 container init e90c73d3a92f5421b64fc2243d2c7f599816610eef1616134dfe18afb1c7544c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-11T08:33:09.322 INFO:journalctl@ceph.osd.4.smithi181.stdout:Sep 11 08:33:09 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-4[37497]: 2024-09-11T08:33:09.285+0000 7f104189f640 -1 osd.4 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:40.435809+0000 front 2024-09-11T08:32:40.435988+0000 (oldest deadline 2024-09-11T08:33:05.135689+0000) 2024-09-11T08:33:09.323 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:09 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:09.258+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:40.703121+0000 front 2024-09-11T08:32:40.703110+0000 (oldest deadline 2024-09-11T08:33:04.803032+0000) 2024-09-11T08:33:09.323 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:33:09 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5[135300]: 2024-09-11T08:33:09.068+0000 7f294bb2b640 -1 osd.5 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:41.094295+0000 front 2024-09-11T08:32:41.094335+0000 (oldest deadline 2024-09-11T08:33:06.394181+0000) 2024-09-11T08:33:09.323 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:09 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:09.109+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:41.728731+0000 front 2024-09-11T08:32:41.728774+0000 (oldest deadline 2024-09-11T08:33:03.428644+0000) 2024-09-11T08:33:09.488 INFO:journalctl@ceph.osd.3.smithi150.stdout:Sep 11 08:33:09 smithi150 podman[219135]: 2024-09-11 08:33:09.05866413 +0000 UTC m=+0.368953133 container start e90c73d3a92f5421b64fc2243d2c7f599816610eef1616134dfe18afb1c7544c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-11T08:33:09.488 INFO:journalctl@ceph.osd.3.smithi150.stdout:Sep 11 08:33:09 smithi150 podman[219135]: 2024-09-11 08:33:09.155324696 +0000 UTC m=+0.465613703 container attach e90c73d3a92f5421b64fc2243d2c7f599816610eef1616134dfe18afb1c7544c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-3-deactivate, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-09-11T08:33:09.489 INFO:journalctl@ceph.osd.3.smithi150.stdout:Sep 11 08:33:09 smithi150 podman[219135]: 2024-09-11 08:33:09.241957487 +0000 UTC m=+0.552246489 container died e90c73d3a92f5421b64fc2243d2c7f599816610eef1616134dfe18afb1c7544c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240731, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-11T08:33:09.819 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c1bddff4-7013-11ef-bcea-c7b262605968@osd.3.service' 2024-09-11T08:33:09.859 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-11T08:33:09.860 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-11T08:33:09.860 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-11T08:33:09.860 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-c1bddff4-7013-11ef-bcea-c7b262605968@osd.4 2024-09-11T08:33:10.159 INFO:journalctl@ceph.osd.4.smithi181.stdout:Sep 11 08:33:09 smithi181 systemd[1]: Stopping Ceph osd.4 for c1bddff4-7013-11ef-bcea-c7b262605968... 2024-09-11T08:33:10.160 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:10 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:10.062+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:41.728731+0000 front 2024-09-11T08:32:41.728774+0000 (oldest deadline 2024-09-11T08:33:03.428644+0000) 2024-09-11T08:33:10.160 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:33:10 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5[135300]: 2024-09-11T08:33:10.035+0000 7f294bb2b640 -1 osd.5 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:41.094295+0000 front 2024-09-11T08:32:41.094335+0000 (oldest deadline 2024-09-11T08:33:06.394181+0000) 2024-09-11T08:33:10.572 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:10 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:10.210+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:40.703121+0000 front 2024-09-11T08:32:40.703110+0000 (oldest deadline 2024-09-11T08:33:04.803032+0000) 2024-09-11T08:33:10.573 INFO:journalctl@ceph.osd.4.smithi181.stdout:Sep 11 08:33:10 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-4[37497]: 2024-09-11T08:33:10.158+0000 7f1043a84640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-11T08:33:10.573 INFO:journalctl@ceph.osd.4.smithi181.stdout:Sep 11 08:33:10 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-4[37497]: 2024-09-11T08:33:10.158+0000 7f1043a84640 -1 osd.4 483 *** Got signal Terminated *** 2024-09-11T08:33:10.573 INFO:journalctl@ceph.osd.4.smithi181.stdout:Sep 11 08:33:10 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-4[37497]: 2024-09-11T08:33:10.158+0000 7f1043a84640 -1 osd.4 483 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-11T08:33:10.573 INFO:journalctl@ceph.osd.4.smithi181.stdout:Sep 11 08:33:10 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-4[37497]: 2024-09-11T08:33:10.266+0000 7f104189f640 -1 osd.4 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:40.435809+0000 front 2024-09-11T08:32:40.435988+0000 (oldest deadline 2024-09-11T08:33:05.135689+0000) 2024-09-11T08:33:11.310 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:11 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:11.194+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:40.703121+0000 front 2024-09-11T08:32:40.703110+0000 (oldest deadline 2024-09-11T08:33:04.803032+0000) 2024-09-11T08:33:11.311 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:11 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:11.047+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:41.728731+0000 front 2024-09-11T08:32:41.728774+0000 (oldest deadline 2024-09-11T08:33:03.428644+0000) 2024-09-11T08:33:11.311 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:33:11 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5[135300]: 2024-09-11T08:33:11.038+0000 7f294bb2b640 -1 osd.5 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:41.094295+0000 front 2024-09-11T08:32:41.094335+0000 (oldest deadline 2024-09-11T08:33:06.394181+0000) 2024-09-11T08:33:11.573 INFO:journalctl@ceph.osd.4.smithi181.stdout:Sep 11 08:33:11 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-4[37497]: 2024-09-11T08:33:11.310+0000 7f104189f640 -1 osd.4 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:40.435809+0000 front 2024-09-11T08:32:40.435988+0000 (oldest deadline 2024-09-11T08:33:05.135689+0000) 2024-09-11T08:33:12.323 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:12 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:12.196+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:40.703121+0000 front 2024-09-11T08:32:40.703110+0000 (oldest deadline 2024-09-11T08:33:04.803032+0000) 2024-09-11T08:33:12.323 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:12 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:12.081+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:41.728731+0000 front 2024-09-11T08:32:41.728774+0000 (oldest deadline 2024-09-11T08:33:03.428644+0000) 2024-09-11T08:33:12.323 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:33:11 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5[135300]: 2024-09-11T08:33:11.989+0000 7f294bb2b640 -1 osd.5 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:41.094295+0000 front 2024-09-11T08:32:41.094335+0000 (oldest deadline 2024-09-11T08:33:06.394181+0000) 2024-09-11T08:33:12.823 INFO:journalctl@ceph.osd.4.smithi181.stdout:Sep 11 08:33:12 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-4[37497]: 2024-09-11T08:33:12.344+0000 7f104189f640 -1 osd.4 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:40.435809+0000 front 2024-09-11T08:32:40.435988+0000 (oldest deadline 2024-09-11T08:33:05.135689+0000) 2024-09-11T08:33:13.323 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:13 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:13.177+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:40.703121+0000 front 2024-09-11T08:32:40.703110+0000 (oldest deadline 2024-09-11T08:33:04.803032+0000) 2024-09-11T08:33:13.324 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:13 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:13.044+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:41.728731+0000 front 2024-09-11T08:32:41.728774+0000 (oldest deadline 2024-09-11T08:33:03.428644+0000) 2024-09-11T08:33:13.324 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:13 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:13.044+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:47.529624+0000 front 2024-09-11T08:32:47.529646+0000 (oldest deadline 2024-09-11T08:33:12.829588+0000) 2024-09-11T08:33:13.324 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:33:13 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5[135300]: 2024-09-11T08:33:13.011+0000 7f294bb2b640 -1 osd.5 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:41.094295+0000 front 2024-09-11T08:32:41.094335+0000 (oldest deadline 2024-09-11T08:33:06.394181+0000) 2024-09-11T08:33:13.324 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:33:13 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5[135300]: 2024-09-11T08:33:13.011+0000 7f294bb2b640 -1 osd.5 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:46.394583+0000 front 2024-09-11T08:32:46.394622+0000 (oldest deadline 2024-09-11T08:33:12.294806+0000) 2024-09-11T08:33:13.822 INFO:journalctl@ceph.osd.4.smithi181.stdout:Sep 11 08:33:13 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-4[37497]: 2024-09-11T08:33:13.325+0000 7f104189f640 -1 osd.4 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:40.435809+0000 front 2024-09-11T08:32:40.435988+0000 (oldest deadline 2024-09-11T08:33:05.135689+0000) 2024-09-11T08:33:14.318 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:14 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:14.162+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:40.703121+0000 front 2024-09-11T08:32:40.703110+0000 (oldest deadline 2024-09-11T08:33:04.803032+0000) 2024-09-11T08:33:14.318 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:14 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:14.064+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:41.728731+0000 front 2024-09-11T08:32:41.728774+0000 (oldest deadline 2024-09-11T08:33:03.428644+0000) 2024-09-11T08:33:14.318 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:14 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:14.064+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:47.529624+0000 front 2024-09-11T08:32:47.529646+0000 (oldest deadline 2024-09-11T08:33:12.829588+0000) 2024-09-11T08:33:14.318 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:33:13 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5[135300]: 2024-09-11T08:33:13.985+0000 7f294bb2b640 -1 osd.5 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:41.094295+0000 front 2024-09-11T08:32:41.094335+0000 (oldest deadline 2024-09-11T08:33:06.394181+0000) 2024-09-11T08:33:14.319 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:33:13 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5[135300]: 2024-09-11T08:33:13.985+0000 7f294bb2b640 -1 osd.5 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:46.394583+0000 front 2024-09-11T08:32:46.394622+0000 (oldest deadline 2024-09-11T08:33:12.294806+0000) 2024-09-11T08:33:14.572 INFO:journalctl@ceph.osd.4.smithi181.stdout:Sep 11 08:33:14 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-4[37497]: 2024-09-11T08:33:14.316+0000 7f104189f640 -1 osd.4 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:40.435809+0000 front 2024-09-11T08:32:40.435988+0000 (oldest deadline 2024-09-11T08:33:05.135689+0000) 2024-09-11T08:33:14.573 INFO:journalctl@ceph.osd.4.smithi181.stdout:Sep 11 08:33:14 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-4[37497]: 2024-09-11T08:33:14.316+0000 7f104189f640 -1 osd.4 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:49.236708+0000 front 2024-09-11T08:32:49.236728+0000 (oldest deadline 2024-09-11T08:33:13.936752+0000) 2024-09-11T08:33:15.323 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:15 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:15.137+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:40.703121+0000 front 2024-09-11T08:32:40.703110+0000 (oldest deadline 2024-09-11T08:33:04.803032+0000) 2024-09-11T08:33:15.323 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:15 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:15.040+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:41.728731+0000 front 2024-09-11T08:32:41.728774+0000 (oldest deadline 2024-09-11T08:33:03.428644+0000) 2024-09-11T08:33:15.323 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:15 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:15.040+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:47.529624+0000 front 2024-09-11T08:32:47.529646+0000 (oldest deadline 2024-09-11T08:33:12.829588+0000) 2024-09-11T08:33:15.323 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:33:14 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5[135300]: 2024-09-11T08:33:14.982+0000 7f294bb2b640 -1 osd.5 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:41.094295+0000 front 2024-09-11T08:32:41.094335+0000 (oldest deadline 2024-09-11T08:33:06.394181+0000) 2024-09-11T08:33:15.323 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:33:14 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5[135300]: 2024-09-11T08:33:14.982+0000 7f294bb2b640 -1 osd.5 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:46.394583+0000 front 2024-09-11T08:32:46.394622+0000 (oldest deadline 2024-09-11T08:33:12.294806+0000) 2024-09-11T08:33:15.993 INFO:journalctl@ceph.osd.4.smithi181.stdout:Sep 11 08:33:15 smithi181 podman[163244]: 2024-09-11 08:33:15.731776922 +0000 UTC m=+5.674293537 container died 2d561f0a93a6cc675198c1620a4288a09e86e37d13f61346016897606715baa8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-4, ceph=True, org.label-schema.build-date=20240731, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-11T08:33:15.993 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:33:15 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5[135300]: 2024-09-11T08:33:15.978+0000 7f294bb2b640 -1 osd.5 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:41.094295+0000 front 2024-09-11T08:32:41.094335+0000 (oldest deadline 2024-09-11T08:33:06.394181+0000) 2024-09-11T08:33:15.994 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:33:15 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5[135300]: 2024-09-11T08:33:15.978+0000 7f294bb2b640 -1 osd.5 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:46.394583+0000 front 2024-09-11T08:32:46.394622+0000 (oldest deadline 2024-09-11T08:33:12.294806+0000) 2024-09-11T08:33:16.316 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:16 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:16.111+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:40.703121+0000 front 2024-09-11T08:32:40.703110+0000 (oldest deadline 2024-09-11T08:33:04.803032+0000) 2024-09-11T08:33:16.317 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:16 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:16.111+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:50.103998+0000 front 2024-09-11T08:32:50.104071+0000 (oldest deadline 2024-09-11T08:33:15.304432+0000) 2024-09-11T08:33:16.317 INFO:journalctl@ceph.osd.4.smithi181.stdout:Sep 11 08:33:16 smithi181 podman[163244]: 2024-09-11 08:33:16.201179179 +0000 UTC m=+6.143695792 container cleanup 2d561f0a93a6cc675198c1620a4288a09e86e37d13f61346016897606715baa8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-4, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-11T08:33:16.317 INFO:journalctl@ceph.osd.4.smithi181.stdout:Sep 11 08:33:16 smithi181 bash[163244]: ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-4 2024-09-11T08:33:16.317 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:15 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:15.993+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:41.728731+0000 front 2024-09-11T08:32:41.728774+0000 (oldest deadline 2024-09-11T08:33:03.428644+0000) 2024-09-11T08:33:16.317 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:15 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:15.993+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:47.529624+0000 front 2024-09-11T08:32:47.529646+0000 (oldest deadline 2024-09-11T08:33:12.829588+0000) 2024-09-11T08:33:16.573 INFO:journalctl@ceph.osd.4.smithi181.stdout:Sep 11 08:33:16 smithi181 podman[163259]: 2024-09-11 08:33:16.503021925 +0000 UTC m=+0.768870149 container remove 2d561f0a93a6cc675198c1620a4288a09e86e37d13f61346016897606715baa8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-4, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-11T08:33:17.123 INFO:journalctl@ceph.osd.4.smithi181.stdout:Sep 11 08:33:16 smithi181 podman[163351]: 2024-09-11 08:33:16.727254001 +0000 UTC m=+0.015932723 image pull 4a918172a49766385959227c53e75e2c302e923539324792cb5bee6cad0b2f2b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40 2024-09-11T08:33:17.123 INFO:journalctl@ceph.osd.4.smithi181.stdout:Sep 11 08:33:16 smithi181 podman[163351]: 2024-09-11 08:33:16.887611548 +0000 UTC m=+0.176290268 container create dcb0cd329296866e12e44b56deb23e354a8b7b5b3cd396822e380d98895bd44f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-11T08:33:17.123 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:16 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:16.945+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:41.728731+0000 front 2024-09-11T08:32:41.728774+0000 (oldest deadline 2024-09-11T08:33:03.428644+0000) 2024-09-11T08:33:17.123 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:16 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:16.945+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:47.529624+0000 front 2024-09-11T08:32:47.529646+0000 (oldest deadline 2024-09-11T08:33:12.829588+0000) 2024-09-11T08:33:17.124 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:33:16 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5[135300]: 2024-09-11T08:33:16.986+0000 7f294bb2b640 -1 osd.5 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:41.094295+0000 front 2024-09-11T08:32:41.094335+0000 (oldest deadline 2024-09-11T08:33:06.394181+0000) 2024-09-11T08:33:17.124 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:33:16 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5[135300]: 2024-09-11T08:33:16.986+0000 7f294bb2b640 -1 osd.5 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:46.394583+0000 front 2024-09-11T08:32:46.394622+0000 (oldest deadline 2024-09-11T08:33:12.294806+0000) 2024-09-11T08:33:17.386 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:17.143+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:40.703121+0000 front 2024-09-11T08:32:40.703110+0000 (oldest deadline 2024-09-11T08:33:04.803032+0000) 2024-09-11T08:33:17.386 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:17.143+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:50.103998+0000 front 2024-09-11T08:32:50.104071+0000 (oldest deadline 2024-09-11T08:33:15.304432+0000) 2024-09-11T08:33:17.386 INFO:journalctl@ceph.osd.4.smithi181.stdout:Sep 11 08:33:17 smithi181 podman[163351]: 2024-09-11 08:33:17.12364352 +0000 UTC m=+0.412322242 container init dcb0cd329296866e12e44b56deb23e354a8b7b5b3cd396822e380d98895bd44f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-09-11T08:33:17.386 INFO:journalctl@ceph.osd.4.smithi181.stdout:Sep 11 08:33:17 smithi181 podman[163351]: 2024-09-11 08:33:17.127341332 +0000 UTC m=+0.416020052 container start dcb0cd329296866e12e44b56deb23e354a8b7b5b3cd396822e380d98895bd44f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0) 2024-09-11T08:33:17.386 INFO:journalctl@ceph.osd.4.smithi181.stdout:Sep 11 08:33:17 smithi181 podman[163351]: 2024-09-11 08:33:17.26540494 +0000 UTC m=+0.554083662 container attach dcb0cd329296866e12e44b56deb23e354a8b7b5b3cd396822e380d98895bd44f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828) 2024-09-11T08:33:17.823 INFO:journalctl@ceph.osd.4.smithi181.stdout:Sep 11 08:33:17 smithi181 podman[163373]: 2024-09-11 08:33:17.386484774 +0000 UTC m=+0.017987618 container died dcb0cd329296866e12e44b56deb23e354a8b7b5b3cd396822e380d98895bd44f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-11T08:33:17.972 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c1bddff4-7013-11ef-bcea-c7b262605968@osd.4.service' 2024-09-11T08:33:18.012 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-11T08:33:18.012 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-11T08:33:18.012 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-11T08:33:18.012 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-c1bddff4-7013-11ef-bcea-c7b262605968@osd.5 2024-09-11T08:33:18.185 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:17.979+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:41.728731+0000 front 2024-09-11T08:32:41.728774+0000 (oldest deadline 2024-09-11T08:33:03.428644+0000) 2024-09-11T08:33:18.185 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:17.979+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:47.529624+0000 front 2024-09-11T08:32:47.529646+0000 (oldest deadline 2024-09-11T08:33:12.829588+0000) 2024-09-11T08:33:18.185 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:33:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5[135300]: 2024-09-11T08:33:17.954+0000 7f294bb2b640 -1 osd.5 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:41.094295+0000 front 2024-09-11T08:32:41.094335+0000 (oldest deadline 2024-09-11T08:33:06.394181+0000) 2024-09-11T08:33:18.186 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:33:17 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5[135300]: 2024-09-11T08:33:17.954+0000 7f294bb2b640 -1 osd.5 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:46.394583+0000 front 2024-09-11T08:32:46.394622+0000 (oldest deadline 2024-09-11T08:33:12.294806+0000) 2024-09-11T08:33:18.186 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:33:18 smithi181 systemd[1]: Stopping Ceph osd.5 for c1bddff4-7013-11ef-bcea-c7b262605968... 2024-09-11T08:33:18.573 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:18 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:18.183+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:40.703121+0000 front 2024-09-11T08:32:40.703110+0000 (oldest deadline 2024-09-11T08:33:04.803032+0000) 2024-09-11T08:33:18.573 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:18 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:18.183+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:50.103998+0000 front 2024-09-11T08:32:50.104071+0000 (oldest deadline 2024-09-11T08:33:15.304432+0000) 2024-09-11T08:33:18.573 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:33:18 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5[135300]: 2024-09-11T08:33:18.346+0000 7f2952519640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-11T08:33:18.573 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:33:18 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5[135300]: 2024-09-11T08:33:18.346+0000 7f2952519640 -1 osd.5 483 *** Got signal Terminated *** 2024-09-11T08:33:18.573 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:33:18 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5[135300]: 2024-09-11T08:33:18.346+0000 7f2952519640 -1 osd.5 483 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-11T08:33:19.191 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:33:18 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5[135300]: 2024-09-11T08:33:18.952+0000 7f294bb2b640 -1 osd.5 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:41.094295+0000 front 2024-09-11T08:32:41.094335+0000 (oldest deadline 2024-09-11T08:33:06.394181+0000) 2024-09-11T08:33:19.191 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:33:18 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5[135300]: 2024-09-11T08:33:18.952+0000 7f294bb2b640 -1 osd.5 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:46.394583+0000 front 2024-09-11T08:32:46.394622+0000 (oldest deadline 2024-09-11T08:33:12.294806+0000) 2024-09-11T08:33:19.191 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:18 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:18.936+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:41.728731+0000 front 2024-09-11T08:32:41.728774+0000 (oldest deadline 2024-09-11T08:33:03.428644+0000) 2024-09-11T08:33:19.191 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:18 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:18.936+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:47.529624+0000 front 2024-09-11T08:32:47.529646+0000 (oldest deadline 2024-09-11T08:33:12.829588+0000) 2024-09-11T08:33:19.573 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:19 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:19.190+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:40.703121+0000 front 2024-09-11T08:32:40.703110+0000 (oldest deadline 2024-09-11T08:33:04.803032+0000) 2024-09-11T08:33:19.573 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:19 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:19.190+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:50.103998+0000 front 2024-09-11T08:32:50.104071+0000 (oldest deadline 2024-09-11T08:33:15.304432+0000) 2024-09-11T08:33:20.179 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:33:19 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5[135300]: 2024-09-11T08:33:19.937+0000 7f294bb2b640 -1 osd.5 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:41.094295+0000 front 2024-09-11T08:32:41.094335+0000 (oldest deadline 2024-09-11T08:33:06.394181+0000) 2024-09-11T08:33:20.179 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:33:19 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5[135300]: 2024-09-11T08:33:19.937+0000 7f294bb2b640 -1 osd.5 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:46.394583+0000 front 2024-09-11T08:32:46.394622+0000 (oldest deadline 2024-09-11T08:33:12.294806+0000) 2024-09-11T08:33:20.179 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:19 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:19.915+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:41.728731+0000 front 2024-09-11T08:32:41.728774+0000 (oldest deadline 2024-09-11T08:33:03.428644+0000) 2024-09-11T08:33:20.179 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:19 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:19.915+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:47.529624+0000 front 2024-09-11T08:32:47.529646+0000 (oldest deadline 2024-09-11T08:33:12.829588+0000) 2024-09-11T08:33:20.179 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:19 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:19.915+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6811 osd.2 since back 2024-09-11T08:32:56.830900+0000 front 2024-09-11T08:32:56.830956+0000 (oldest deadline 2024-09-11T08:33:19.730818+0000) 2024-09-11T08:33:20.573 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:20 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:20.178+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:40.703121+0000 front 2024-09-11T08:32:40.703110+0000 (oldest deadline 2024-09-11T08:33:04.803032+0000) 2024-09-11T08:33:20.573 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:20 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:20.178+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:50.103998+0000 front 2024-09-11T08:32:50.104071+0000 (oldest deadline 2024-09-11T08:33:15.304432+0000) 2024-09-11T08:33:20.573 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:20 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:20.178+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6811 osd.2 since back 2024-09-11T08:32:55.305088+0000 front 2024-09-11T08:32:55.305019+0000 (oldest deadline 2024-09-11T08:33:20.004902+0000) 2024-09-11T08:33:21.162 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:33:20 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5[135300]: 2024-09-11T08:33:20.930+0000 7f294bb2b640 -1 osd.5 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:41.094295+0000 front 2024-09-11T08:32:41.094335+0000 (oldest deadline 2024-09-11T08:33:06.394181+0000) 2024-09-11T08:33:21.163 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:33:20 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5[135300]: 2024-09-11T08:33:20.930+0000 7f294bb2b640 -1 osd.5 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:46.394583+0000 front 2024-09-11T08:32:46.394622+0000 (oldest deadline 2024-09-11T08:33:12.294806+0000) 2024-09-11T08:33:21.163 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:20 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:20.891+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:41.728731+0000 front 2024-09-11T08:32:41.728774+0000 (oldest deadline 2024-09-11T08:33:03.428644+0000) 2024-09-11T08:33:21.163 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:20 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:20.891+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:47.529624+0000 front 2024-09-11T08:32:47.529646+0000 (oldest deadline 2024-09-11T08:33:12.829588+0000) 2024-09-11T08:33:21.163 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:20 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:20.891+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6811 osd.2 since back 2024-09-11T08:32:56.830900+0000 front 2024-09-11T08:32:56.830956+0000 (oldest deadline 2024-09-11T08:33:19.730818+0000) 2024-09-11T08:33:21.572 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:21 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:21.162+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:40.703121+0000 front 2024-09-11T08:32:40.703110+0000 (oldest deadline 2024-09-11T08:33:04.803032+0000) 2024-09-11T08:33:21.573 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:21 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:21.162+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:50.103998+0000 front 2024-09-11T08:32:50.104071+0000 (oldest deadline 2024-09-11T08:33:15.304432+0000) 2024-09-11T08:33:21.573 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:21 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:21.162+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6811 osd.2 since back 2024-09-11T08:32:55.305088+0000 front 2024-09-11T08:32:55.305019+0000 (oldest deadline 2024-09-11T08:33:20.004902+0000) 2024-09-11T08:33:22.195 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:33:21 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5[135300]: 2024-09-11T08:33:21.962+0000 7f294bb2b640 -1 osd.5 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:41.094295+0000 front 2024-09-11T08:32:41.094335+0000 (oldest deadline 2024-09-11T08:33:06.394181+0000) 2024-09-11T08:33:22.195 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:33:21 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5[135300]: 2024-09-11T08:33:21.962+0000 7f294bb2b640 -1 osd.5 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:46.394583+0000 front 2024-09-11T08:32:46.394622+0000 (oldest deadline 2024-09-11T08:33:12.294806+0000) 2024-09-11T08:33:22.195 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:21 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:21.884+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:41.728731+0000 front 2024-09-11T08:32:41.728774+0000 (oldest deadline 2024-09-11T08:33:03.428644+0000) 2024-09-11T08:33:22.196 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:21 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:21.884+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:47.529624+0000 front 2024-09-11T08:32:47.529646+0000 (oldest deadline 2024-09-11T08:33:12.829588+0000) 2024-09-11T08:33:22.196 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:21 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:21.884+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6811 osd.2 since back 2024-09-11T08:32:56.830900+0000 front 2024-09-11T08:32:56.830956+0000 (oldest deadline 2024-09-11T08:33:19.730818+0000) 2024-09-11T08:33:22.573 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:22 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:22.195+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:40.703121+0000 front 2024-09-11T08:32:40.703110+0000 (oldest deadline 2024-09-11T08:33:04.803032+0000) 2024-09-11T08:33:22.573 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:22 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:22.195+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:50.103998+0000 front 2024-09-11T08:32:50.104071+0000 (oldest deadline 2024-09-11T08:33:15.304432+0000) 2024-09-11T08:33:22.573 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:22 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:22.195+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6811 osd.2 since back 2024-09-11T08:32:55.305088+0000 front 2024-09-11T08:32:55.305019+0000 (oldest deadline 2024-09-11T08:33:20.004902+0000) 2024-09-11T08:33:23.189 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:22 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:22.921+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:41.728731+0000 front 2024-09-11T08:32:41.728774+0000 (oldest deadline 2024-09-11T08:33:03.428644+0000) 2024-09-11T08:33:23.190 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:22 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:22.921+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:47.529624+0000 front 2024-09-11T08:32:47.529646+0000 (oldest deadline 2024-09-11T08:33:12.829588+0000) 2024-09-11T08:33:23.190 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:22 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:22.921+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6811 osd.2 since back 2024-09-11T08:32:56.830900+0000 front 2024-09-11T08:32:56.830956+0000 (oldest deadline 2024-09-11T08:33:19.730818+0000) 2024-09-11T08:33:23.190 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:33:22 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5[135300]: 2024-09-11T08:33:22.953+0000 7f294bb2b640 -1 osd.5 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:41.094295+0000 front 2024-09-11T08:32:41.094335+0000 (oldest deadline 2024-09-11T08:33:06.394181+0000) 2024-09-11T08:33:23.190 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:33:22 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5[135300]: 2024-09-11T08:33:22.953+0000 7f294bb2b640 -1 osd.5 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:46.394583+0000 front 2024-09-11T08:32:46.394622+0000 (oldest deadline 2024-09-11T08:33:12.294806+0000) 2024-09-11T08:33:23.190 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:33:22 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5[135300]: 2024-09-11T08:33:22.953+0000 7f294bb2b640 -1 osd.5 483 heartbeat_check: no reply from 172.21.15.150:6811 osd.2 since back 2024-09-11T08:32:56.995845+0000 front 2024-09-11T08:32:56.995779+0000 (oldest deadline 2024-09-11T08:33:22.796084+0000) 2024-09-11T08:33:23.573 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:23 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:23.188+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:40.703121+0000 front 2024-09-11T08:32:40.703110+0000 (oldest deadline 2024-09-11T08:33:04.803032+0000) 2024-09-11T08:33:23.573 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:23 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:23.188+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:50.103998+0000 front 2024-09-11T08:32:50.104071+0000 (oldest deadline 2024-09-11T08:33:15.304432+0000) 2024-09-11T08:33:23.573 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:23 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:23.188+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6811 osd.2 since back 2024-09-11T08:32:55.305088+0000 front 2024-09-11T08:32:55.305019+0000 (oldest deadline 2024-09-11T08:33:20.004902+0000) 2024-09-11T08:33:24.023 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:33:23 smithi181 podman[163463]: 2024-09-11 08:33:23.745747518 +0000 UTC m=+5.507224925 container died 5c7014cbea51deea3fcbbd66fb19d6ade19f9a377ba2e605c8790809987ad9fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-11T08:33:24.023 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:23 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:23.928+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:41.728731+0000 front 2024-09-11T08:32:41.728774+0000 (oldest deadline 2024-09-11T08:33:03.428644+0000) 2024-09-11T08:33:24.023 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:23 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:23.928+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:47.529624+0000 front 2024-09-11T08:32:47.529646+0000 (oldest deadline 2024-09-11T08:33:12.829588+0000) 2024-09-11T08:33:24.023 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:23 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:23.928+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6811 osd.2 since back 2024-09-11T08:32:56.830900+0000 front 2024-09-11T08:32:56.830956+0000 (oldest deadline 2024-09-11T08:33:19.730818+0000) 2024-09-11T08:33:24.323 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:24 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:24.153+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:40.703121+0000 front 2024-09-11T08:32:40.703110+0000 (oldest deadline 2024-09-11T08:33:04.803032+0000) 2024-09-11T08:33:24.323 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:24 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:24.153+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:50.103998+0000 front 2024-09-11T08:32:50.104071+0000 (oldest deadline 2024-09-11T08:33:15.304432+0000) 2024-09-11T08:33:24.323 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:24 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:24.153+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6811 osd.2 since back 2024-09-11T08:32:55.305088+0000 front 2024-09-11T08:32:55.305019+0000 (oldest deadline 2024-09-11T08:33:20.004902+0000) 2024-09-11T08:33:24.323 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:33:24 smithi181 podman[163463]: 2024-09-11 08:33:24.12252096 +0000 UTC m=+5.883998357 container cleanup 5c7014cbea51deea3fcbbd66fb19d6ade19f9a377ba2e605c8790809987ad9fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS) 2024-09-11T08:33:24.323 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:33:24 smithi181 bash[163463]: ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5 2024-09-11T08:33:24.736 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:33:24 smithi181 podman[163480]: 2024-09-11 08:33:24.424607753 +0000 UTC m=+0.675219966 container remove 5c7014cbea51deea3fcbbd66fb19d6ade19f9a377ba2e605c8790809987ad9fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-09-11T08:33:24.989 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:33:24 smithi181 podman[163572]: 2024-09-11 08:33:24.639556602 +0000 UTC m=+0.015469980 image pull 4a918172a49766385959227c53e75e2c302e923539324792cb5bee6cad0b2f2b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40 2024-09-11T08:33:24.989 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:33:24 smithi181 podman[163572]: 2024-09-11 08:33:24.750320811 +0000 UTC m=+0.126234176 container create 95ff38641b3222290fa6dac8f84eaf13d2b4578cc8415f381bc6289334241c18 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-09-11T08:33:24.989 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:33:24 smithi181 podman[163572]: 2024-09-11 08:33:24.985935965 +0000 UTC m=+0.361849347 container init 95ff38641b3222290fa6dac8f84eaf13d2b4578cc8415f381bc6289334241c18 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-09-11T08:33:24.989 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:24 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:24.906+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:41.728731+0000 front 2024-09-11T08:32:41.728774+0000 (oldest deadline 2024-09-11T08:33:03.428644+0000) 2024-09-11T08:33:24.989 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:24 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:24.906+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:47.529624+0000 front 2024-09-11T08:32:47.529646+0000 (oldest deadline 2024-09-11T08:33:12.829588+0000) 2024-09-11T08:33:24.990 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:24 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:24.906+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6811 osd.2 since back 2024-09-11T08:32:56.830900+0000 front 2024-09-11T08:32:56.830956+0000 (oldest deadline 2024-09-11T08:33:19.730818+0000) 2024-09-11T08:33:25.323 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:25 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:25.201+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:40.703121+0000 front 2024-09-11T08:32:40.703110+0000 (oldest deadline 2024-09-11T08:33:04.803032+0000) 2024-09-11T08:33:25.323 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:25 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:25.201+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:50.103998+0000 front 2024-09-11T08:32:50.104071+0000 (oldest deadline 2024-09-11T08:33:15.304432+0000) 2024-09-11T08:33:25.323 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:25 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:25.201+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6811 osd.2 since back 2024-09-11T08:32:55.305088+0000 front 2024-09-11T08:32:55.305019+0000 (oldest deadline 2024-09-11T08:33:20.004902+0000) 2024-09-11T08:33:25.323 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:33:24 smithi181 podman[163572]: 2024-09-11 08:33:24.989149332 +0000 UTC m=+0.365062698 container start 95ff38641b3222290fa6dac8f84eaf13d2b4578cc8415f381bc6289334241c18 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828) 2024-09-11T08:33:25.324 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:33:25 smithi181 podman[163572]: 2024-09-11 08:33:25.094631518 +0000 UTC m=+0.470544900 container attach 95ff38641b3222290fa6dac8f84eaf13d2b4578cc8415f381bc6289334241c18 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-09-11T08:33:25.324 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 11 08:33:25 smithi181 podman[163572]: 2024-09-11 08:33:25.166882579 +0000 UTC m=+0.542795950 container died 95ff38641b3222290fa6dac8f84eaf13d2b4578cc8415f381bc6289334241c18 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-5-deactivate, ceph=True, org.label-schema.build-date=20240731, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-11T08:33:25.816 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c1bddff4-7013-11ef-bcea-c7b262605968@osd.5.service' 2024-09-11T08:33:25.858 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-11T08:33:25.858 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-11T08:33:25.859 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-11T08:33:25.859 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-c1bddff4-7013-11ef-bcea-c7b262605968@osd.6 2024-09-11T08:33:26.072 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:25 smithi181 systemd[1]: Stopping Ceph osd.6 for c1bddff4-7013-11ef-bcea-c7b262605968... 2024-09-11T08:33:26.073 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:25 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:25.863+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:41.728731+0000 front 2024-09-11T08:32:41.728774+0000 (oldest deadline 2024-09-11T08:33:03.428644+0000) 2024-09-11T08:33:26.073 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:25 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:25.863+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:47.529624+0000 front 2024-09-11T08:32:47.529646+0000 (oldest deadline 2024-09-11T08:33:12.829588+0000) 2024-09-11T08:33:26.073 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:25 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:25.863+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6811 osd.2 since back 2024-09-11T08:32:56.830900+0000 front 2024-09-11T08:32:56.830956+0000 (oldest deadline 2024-09-11T08:33:19.730818+0000) 2024-09-11T08:33:26.573 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:26 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:26.185+0000 7fa51388b640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-11T08:33:26.573 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:26 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:26.185+0000 7fa51388b640 -1 osd.6 483 *** Got signal Terminated *** 2024-09-11T08:33:26.573 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:26 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:26.185+0000 7fa51388b640 -1 osd.6 483 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-11T08:33:26.573 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:26 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:26.224+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:40.703121+0000 front 2024-09-11T08:32:40.703110+0000 (oldest deadline 2024-09-11T08:33:04.803032+0000) 2024-09-11T08:33:26.573 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:26 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:26.224+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:50.103998+0000 front 2024-09-11T08:32:50.104071+0000 (oldest deadline 2024-09-11T08:33:15.304432+0000) 2024-09-11T08:33:26.573 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:26 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:26.224+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6811 osd.2 since back 2024-09-11T08:32:55.305088+0000 front 2024-09-11T08:32:55.305019+0000 (oldest deadline 2024-09-11T08:33:20.004902+0000) 2024-09-11T08:33:27.205 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:26 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:26.863+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:41.728731+0000 front 2024-09-11T08:32:41.728774+0000 (oldest deadline 2024-09-11T08:33:03.428644+0000) 2024-09-11T08:33:27.205 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:26 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:26.863+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:47.529624+0000 front 2024-09-11T08:32:47.529646+0000 (oldest deadline 2024-09-11T08:33:12.829588+0000) 2024-09-11T08:33:27.205 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:26 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:26.863+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6811 osd.2 since back 2024-09-11T08:32:56.830900+0000 front 2024-09-11T08:32:56.830956+0000 (oldest deadline 2024-09-11T08:33:19.730818+0000) 2024-09-11T08:33:27.573 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:27 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:27.205+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:40.703121+0000 front 2024-09-11T08:32:40.703110+0000 (oldest deadline 2024-09-11T08:33:04.803032+0000) 2024-09-11T08:33:27.573 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:27 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:27.205+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:50.103998+0000 front 2024-09-11T08:32:50.104071+0000 (oldest deadline 2024-09-11T08:33:15.304432+0000) 2024-09-11T08:33:27.573 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:27 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:27.205+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6811 osd.2 since back 2024-09-11T08:32:55.305088+0000 front 2024-09-11T08:32:55.305019+0000 (oldest deadline 2024-09-11T08:33:20.004902+0000) 2024-09-11T08:33:28.219 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:27 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:27.896+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:41.728731+0000 front 2024-09-11T08:32:41.728774+0000 (oldest deadline 2024-09-11T08:33:03.428644+0000) 2024-09-11T08:33:28.219 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:27 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:27.896+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:47.529624+0000 front 2024-09-11T08:32:47.529646+0000 (oldest deadline 2024-09-11T08:33:12.829588+0000) 2024-09-11T08:33:28.219 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:27 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:27.896+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6811 osd.2 since back 2024-09-11T08:32:56.830900+0000 front 2024-09-11T08:32:56.830956+0000 (oldest deadline 2024-09-11T08:33:19.730818+0000) 2024-09-11T08:33:28.573 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:28 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:28.218+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:40.703121+0000 front 2024-09-11T08:32:40.703110+0000 (oldest deadline 2024-09-11T08:33:04.803032+0000) 2024-09-11T08:33:28.573 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:28 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:28.218+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:50.103998+0000 front 2024-09-11T08:32:50.104071+0000 (oldest deadline 2024-09-11T08:33:15.304432+0000) 2024-09-11T08:33:28.573 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:28 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:28.218+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6811 osd.2 since back 2024-09-11T08:32:55.305088+0000 front 2024-09-11T08:32:55.305019+0000 (oldest deadline 2024-09-11T08:33:20.004902+0000) 2024-09-11T08:33:29.073 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:28 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:28.922+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:41.728731+0000 front 2024-09-11T08:32:41.728774+0000 (oldest deadline 2024-09-11T08:33:03.428644+0000) 2024-09-11T08:33:29.073 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:28 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:28.922+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:47.529624+0000 front 2024-09-11T08:32:47.529646+0000 (oldest deadline 2024-09-11T08:33:12.829588+0000) 2024-09-11T08:33:29.073 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:28 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:28.922+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6811 osd.2 since back 2024-09-11T08:32:56.830900+0000 front 2024-09-11T08:32:56.830956+0000 (oldest deadline 2024-09-11T08:33:19.730818+0000) 2024-09-11T08:33:29.073 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:28 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:28.922+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6815 osd.3 since back 2024-09-11T08:33:04.633228+0000 front 2024-09-11T08:33:04.633283+0000 (oldest deadline 2024-09-11T08:33:28.733235+0000) 2024-09-11T08:33:29.573 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:29 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:29.185+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:40.703121+0000 front 2024-09-11T08:32:40.703110+0000 (oldest deadline 2024-09-11T08:33:04.803032+0000) 2024-09-11T08:33:29.573 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:29 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:29.185+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:50.103998+0000 front 2024-09-11T08:32:50.104071+0000 (oldest deadline 2024-09-11T08:33:15.304432+0000) 2024-09-11T08:33:29.573 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:29 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:29.185+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6811 osd.2 since back 2024-09-11T08:32:55.305088+0000 front 2024-09-11T08:32:55.305019+0000 (oldest deadline 2024-09-11T08:33:20.004902+0000) 2024-09-11T08:33:30.220 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:29 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:29.960+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:41.728731+0000 front 2024-09-11T08:32:41.728774+0000 (oldest deadline 2024-09-11T08:33:03.428644+0000) 2024-09-11T08:33:30.220 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:29 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:29.960+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:47.529624+0000 front 2024-09-11T08:32:47.529646+0000 (oldest deadline 2024-09-11T08:33:12.829588+0000) 2024-09-11T08:33:30.220 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:29 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:29.960+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6811 osd.2 since back 2024-09-11T08:32:56.830900+0000 front 2024-09-11T08:32:56.830956+0000 (oldest deadline 2024-09-11T08:33:19.730818+0000) 2024-09-11T08:33:30.220 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:29 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:29.960+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6815 osd.3 since back 2024-09-11T08:33:04.633228+0000 front 2024-09-11T08:33:04.633283+0000 (oldest deadline 2024-09-11T08:33:28.733235+0000) 2024-09-11T08:33:30.573 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:30 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:30.220+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:40.703121+0000 front 2024-09-11T08:32:40.703110+0000 (oldest deadline 2024-09-11T08:33:04.803032+0000) 2024-09-11T08:33:30.573 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:30 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:30.220+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:50.103998+0000 front 2024-09-11T08:32:50.104071+0000 (oldest deadline 2024-09-11T08:33:15.304432+0000) 2024-09-11T08:33:30.573 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:30 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:30.220+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6811 osd.2 since back 2024-09-11T08:32:55.305088+0000 front 2024-09-11T08:32:55.305019+0000 (oldest deadline 2024-09-11T08:33:20.004902+0000) 2024-09-11T08:33:31.323 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:31 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:31.184+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:40.703121+0000 front 2024-09-11T08:32:40.703110+0000 (oldest deadline 2024-09-11T08:33:04.803032+0000) 2024-09-11T08:33:31.323 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:31 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:31.184+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:50.103998+0000 front 2024-09-11T08:32:50.104071+0000 (oldest deadline 2024-09-11T08:33:15.304432+0000) 2024-09-11T08:33:31.323 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:31 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:31.184+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6811 osd.2 since back 2024-09-11T08:32:55.305088+0000 front 2024-09-11T08:32:55.305019+0000 (oldest deadline 2024-09-11T08:33:20.004902+0000) 2024-09-11T08:33:31.323 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:31 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6[49131]: 2024-09-11T08:33:31.184+0000 7fa510ea5640 -1 osd.6 483 heartbeat_check: no reply from 172.21.15.150:6815 osd.3 since back 2024-09-11T08:33:05.806271+0000 front 2024-09-11T08:33:05.806281+0000 (oldest deadline 2024-09-11T08:33:30.506173+0000) 2024-09-11T08:33:31.323 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:31 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:30.999+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:41.728731+0000 front 2024-09-11T08:32:41.728774+0000 (oldest deadline 2024-09-11T08:33:03.428644+0000) 2024-09-11T08:33:31.324 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:31 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:30.999+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:47.529624+0000 front 2024-09-11T08:32:47.529646+0000 (oldest deadline 2024-09-11T08:33:12.829588+0000) 2024-09-11T08:33:31.324 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:31 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:30.999+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6811 osd.2 since back 2024-09-11T08:32:56.830900+0000 front 2024-09-11T08:32:56.830956+0000 (oldest deadline 2024-09-11T08:33:19.730818+0000) 2024-09-11T08:33:31.324 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:31 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:30.999+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6815 osd.3 since back 2024-09-11T08:33:04.633228+0000 front 2024-09-11T08:33:04.633283+0000 (oldest deadline 2024-09-11T08:33:28.733235+0000) 2024-09-11T08:33:32.073 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:31 smithi181 podman[163673]: 2024-09-11 08:33:31.797075569 +0000 UTC m=+5.712758088 container died ba4cbe8cc4d5e753a3248b575268a078e017ba2c5afc24a78d81ddf4157cebd9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-11T08:33:32.073 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:31 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:31.998+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:41.728731+0000 front 2024-09-11T08:32:41.728774+0000 (oldest deadline 2024-09-11T08:33:03.428644+0000) 2024-09-11T08:33:32.073 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:31 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:31.998+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:47.529624+0000 front 2024-09-11T08:32:47.529646+0000 (oldest deadline 2024-09-11T08:33:12.829588+0000) 2024-09-11T08:33:32.073 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:31 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:31.998+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6811 osd.2 since back 2024-09-11T08:32:56.830900+0000 front 2024-09-11T08:32:56.830956+0000 (oldest deadline 2024-09-11T08:33:19.730818+0000) 2024-09-11T08:33:32.073 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:31 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:31.998+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6815 osd.3 since back 2024-09-11T08:33:04.633228+0000 front 2024-09-11T08:33:04.633283+0000 (oldest deadline 2024-09-11T08:33:28.733235+0000) 2024-09-11T08:33:32.452 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:32 smithi181 podman[163673]: 2024-09-11 08:33:32.217852639 +0000 UTC m=+6.133535169 container cleanup ba4cbe8cc4d5e753a3248b575268a078e017ba2c5afc24a78d81ddf4157cebd9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6, io.buildah.version=1.37.2, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.build-date=20240731) 2024-09-11T08:33:32.452 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:32 smithi181 bash[163673]: ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6 2024-09-11T08:33:32.744 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:32 smithi181 podman[163691]: 2024-09-11 08:33:32.45249376 +0000 UTC m=+0.638003653 container remove ba4cbe8cc4d5e753a3248b575268a078e017ba2c5afc24a78d81ddf4157cebd9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828) 2024-09-11T08:33:33.005 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:32 smithi181 podman[163781]: 2024-09-11 08:33:32.744969297 +0000 UTC m=+0.099265642 container create b7399747453006a2c81055598e91662acc0b2945b3b4f28c81ddcb0625ab325f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-11T08:33:33.005 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:32 smithi181 podman[163781]: 2024-09-11 08:33:32.66124984 +0000 UTC m=+0.015546195 image pull 4a918172a49766385959227c53e75e2c302e923539324792cb5bee6cad0b2f2b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40 2024-09-11T08:33:33.005 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:33 smithi181 podman[163781]: 2024-09-11 08:33:33.00604024 +0000 UTC m=+0.360336586 container init b7399747453006a2c81055598e91662acc0b2945b3b4f28c81ddcb0625ab325f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-11T08:33:33.323 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:33 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:33.026+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:41.728731+0000 front 2024-09-11T08:32:41.728774+0000 (oldest deadline 2024-09-11T08:33:03.428644+0000) 2024-09-11T08:33:33.324 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:33 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:33.026+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:47.529624+0000 front 2024-09-11T08:32:47.529646+0000 (oldest deadline 2024-09-11T08:33:12.829588+0000) 2024-09-11T08:33:33.324 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:33 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:33.026+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6811 osd.2 since back 2024-09-11T08:32:56.830900+0000 front 2024-09-11T08:32:56.830956+0000 (oldest deadline 2024-09-11T08:33:19.730818+0000) 2024-09-11T08:33:33.324 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:33 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:33.026+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6815 osd.3 since back 2024-09-11T08:33:04.633228+0000 front 2024-09-11T08:33:04.633283+0000 (oldest deadline 2024-09-11T08:33:28.733235+0000) 2024-09-11T08:33:33.324 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:33 smithi181 podman[163781]: 2024-09-11 08:33:33.00936917 +0000 UTC m=+0.363665517 container start b7399747453006a2c81055598e91662acc0b2945b3b4f28c81ddcb0625ab325f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-09-11T08:33:33.324 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:33 smithi181 podman[163781]: 2024-09-11 08:33:33.147675129 +0000 UTC m=+0.501971492 container attach b7399747453006a2c81055598e91662acc0b2945b3b4f28c81ddcb0625ab325f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True) 2024-09-11T08:33:33.324 INFO:journalctl@ceph.osd.6.smithi181.stdout:Sep 11 08:33:33 smithi181 podman[163781]: 2024-09-11 08:33:33.179981046 +0000 UTC m=+0.534277402 container died b7399747453006a2c81055598e91662acc0b2945b3b4f28c81ddcb0625ab325f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-6-deactivate, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-11T08:33:33.751 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c1bddff4-7013-11ef-bcea-c7b262605968@osd.6.service' 2024-09-11T08:33:33.791 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-11T08:33:33.791 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-11T08:33:33.792 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-11T08:33:33.792 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-c1bddff4-7013-11ef-bcea-c7b262605968@osd.7 2024-09-11T08:33:34.003 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:33 smithi181 systemd[1]: Stopping Ceph osd.7 for c1bddff4-7013-11ef-bcea-c7b262605968... 2024-09-11T08:33:34.322 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:34 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:34.003+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:41.728731+0000 front 2024-09-11T08:32:41.728774+0000 (oldest deadline 2024-09-11T08:33:03.428644+0000) 2024-09-11T08:33:34.323 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:34 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:34.003+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:47.529624+0000 front 2024-09-11T08:32:47.529646+0000 (oldest deadline 2024-09-11T08:33:12.829588+0000) 2024-09-11T08:33:34.323 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:34 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:34.003+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6811 osd.2 since back 2024-09-11T08:32:56.830900+0000 front 2024-09-11T08:32:56.830956+0000 (oldest deadline 2024-09-11T08:33:19.730818+0000) 2024-09-11T08:33:34.323 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:34 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:34.003+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6815 osd.3 since back 2024-09-11T08:33:04.633228+0000 front 2024-09-11T08:33:04.633283+0000 (oldest deadline 2024-09-11T08:33:28.733235+0000) 2024-09-11T08:33:34.323 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:34 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:34.121+0000 7f3160ae7640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-11T08:33:34.323 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:34 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:34.121+0000 7f3160ae7640 -1 osd.7 483 *** Got signal Terminated *** 2024-09-11T08:33:34.323 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:34 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:34.121+0000 7f3160ae7640 -1 osd.7 483 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-11T08:33:35.279 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:35 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:35.050+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:41.728731+0000 front 2024-09-11T08:32:41.728774+0000 (oldest deadline 2024-09-11T08:33:03.428644+0000) 2024-09-11T08:33:35.280 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:35 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:35.050+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:47.529624+0000 front 2024-09-11T08:32:47.529646+0000 (oldest deadline 2024-09-11T08:33:12.829588+0000) 2024-09-11T08:33:35.280 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:35 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:35.050+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6811 osd.2 since back 2024-09-11T08:32:56.830900+0000 front 2024-09-11T08:32:56.830956+0000 (oldest deadline 2024-09-11T08:33:19.730818+0000) 2024-09-11T08:33:35.280 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:35 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:35.050+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6815 osd.3 since back 2024-09-11T08:33:04.633228+0000 front 2024-09-11T08:33:04.633283+0000 (oldest deadline 2024-09-11T08:33:28.733235+0000) 2024-09-11T08:33:36.323 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:36 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:36.067+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:41.728731+0000 front 2024-09-11T08:32:41.728774+0000 (oldest deadline 2024-09-11T08:33:03.428644+0000) 2024-09-11T08:33:36.323 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:36 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:36.067+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:47.529624+0000 front 2024-09-11T08:32:47.529646+0000 (oldest deadline 2024-09-11T08:33:12.829588+0000) 2024-09-11T08:33:36.323 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:36 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:36.067+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6811 osd.2 since back 2024-09-11T08:32:56.830900+0000 front 2024-09-11T08:32:56.830956+0000 (oldest deadline 2024-09-11T08:33:19.730818+0000) 2024-09-11T08:33:36.323 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:36 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:36.067+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6815 osd.3 since back 2024-09-11T08:33:04.633228+0000 front 2024-09-11T08:33:04.633283+0000 (oldest deadline 2024-09-11T08:33:28.733235+0000) 2024-09-11T08:33:37.323 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:37 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:37.031+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:41.728731+0000 front 2024-09-11T08:32:41.728774+0000 (oldest deadline 2024-09-11T08:33:03.428644+0000) 2024-09-11T08:33:37.323 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:37 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:37.031+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:47.529624+0000 front 2024-09-11T08:32:47.529646+0000 (oldest deadline 2024-09-11T08:33:12.829588+0000) 2024-09-11T08:33:37.323 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:37 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:37.031+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6811 osd.2 since back 2024-09-11T08:32:56.830900+0000 front 2024-09-11T08:32:56.830956+0000 (oldest deadline 2024-09-11T08:33:19.730818+0000) 2024-09-11T08:33:37.323 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:37 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:37.031+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6815 osd.3 since back 2024-09-11T08:33:04.633228+0000 front 2024-09-11T08:33:04.633283+0000 (oldest deadline 2024-09-11T08:33:28.733235+0000) 2024-09-11T08:33:38.323 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:38 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:38.020+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:41.728731+0000 front 2024-09-11T08:32:41.728774+0000 (oldest deadline 2024-09-11T08:33:03.428644+0000) 2024-09-11T08:33:38.323 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:38 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:38.020+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:47.529624+0000 front 2024-09-11T08:32:47.529646+0000 (oldest deadline 2024-09-11T08:33:12.829588+0000) 2024-09-11T08:33:38.323 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:38 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:38.020+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6811 osd.2 since back 2024-09-11T08:32:56.830900+0000 front 2024-09-11T08:32:56.830956+0000 (oldest deadline 2024-09-11T08:33:19.730818+0000) 2024-09-11T08:33:38.323 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:38 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:38.020+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6815 osd.3 since back 2024-09-11T08:33:04.633228+0000 front 2024-09-11T08:33:04.633283+0000 (oldest deadline 2024-09-11T08:33:28.733235+0000) 2024-09-11T08:33:39.323 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:39 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:39.010+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6803 osd.0 since back 2024-09-11T08:32:41.728731+0000 front 2024-09-11T08:32:41.728774+0000 (oldest deadline 2024-09-11T08:33:03.428644+0000) 2024-09-11T08:33:39.323 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:39 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:39.010+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6807 osd.1 since back 2024-09-11T08:32:47.529624+0000 front 2024-09-11T08:32:47.529646+0000 (oldest deadline 2024-09-11T08:33:12.829588+0000) 2024-09-11T08:33:39.323 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:39 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:39.010+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6811 osd.2 since back 2024-09-11T08:32:56.830900+0000 front 2024-09-11T08:32:56.830956+0000 (oldest deadline 2024-09-11T08:33:19.730818+0000) 2024-09-11T08:33:39.323 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:39 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7[86721]: 2024-09-11T08:33:39.010+0000 7f315a0f9640 -1 osd.7 483 heartbeat_check: no reply from 172.21.15.150:6815 osd.3 since back 2024-09-11T08:33:04.633228+0000 front 2024-09-11T08:33:04.633283+0000 (oldest deadline 2024-09-11T08:33:28.733235+0000) 2024-09-11T08:33:39.896 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:39 smithi181 podman[163882]: 2024-09-11 08:33:39.607866711 +0000 UTC m=+5.594337540 container died f35c562f0f3767f3d65faacca7617f9cfd9ad2e30386448a1e84e8964f337643 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-11T08:33:40.271 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:39 smithi181 podman[163882]: 2024-09-11 08:33:39.995336077 +0000 UTC m=+5.981806906 container cleanup f35c562f0f3767f3d65faacca7617f9cfd9ad2e30386448a1e84e8964f337643 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-11T08:33:40.272 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:39 smithi181 bash[163882]: ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7 2024-09-11T08:33:40.572 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:40 smithi181 podman[163910]: 2024-09-11 08:33:40.27189214 +0000 UTC m=+0.657984826 container remove f35c562f0f3767f3d65faacca7617f9cfd9ad2e30386448a1e84e8964f337643 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828) 2024-09-11T08:33:40.874 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:40 smithi181 podman[163998]: 2024-09-11 08:33:40.481004344 +0000 UTC m=+0.015449310 image pull 4a918172a49766385959227c53e75e2c302e923539324792cb5bee6cad0b2f2b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40 2024-09-11T08:33:40.875 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:40 smithi181 podman[163998]: 2024-09-11 08:33:40.589497858 +0000 UTC m=+0.123942824 container create 098766b1c9b622fb90c9b7acbde83f3e8177abf171f4bd8ff923f3fba0987916 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-11T08:33:40.875 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:40 smithi181 podman[163998]: 2024-09-11 08:33:40.800236036 +0000 UTC m=+0.334681002 container init 098766b1c9b622fb90c9b7acbde83f3e8177abf171f4bd8ff923f3fba0987916 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-11T08:33:40.875 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:40 smithi181 podman[163998]: 2024-09-11 08:33:40.803468841 +0000 UTC m=+0.337913810 container start 098766b1c9b622fb90c9b7acbde83f3e8177abf171f4bd8ff923f3fba0987916 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-11T08:33:40.875 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:40 smithi181 podman[163998]: 2024-09-11 08:33:40.8751368 +0000 UTC m=+0.409581780 container attach 098766b1c9b622fb90c9b7acbde83f3e8177abf171f4bd8ff923f3fba0987916 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, ceph=True) 2024-09-11T08:33:41.261 INFO:journalctl@ceph.osd.7.smithi181.stdout:Sep 11 08:33:40 smithi181 podman[163998]: 2024-09-11 08:33:40.974114401 +0000 UTC m=+0.508559376 container died 098766b1c9b622fb90c9b7acbde83f3e8177abf171f4bd8ff923f3fba0987916 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-osd-7-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-11T08:33:41.568 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c1bddff4-7013-11ef-bcea-c7b262605968@osd.7.service' 2024-09-11T08:33:41.608 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-11T08:33:41.608 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-11T08:33:41.608 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-09-11T08:33:41.608 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-c1bddff4-7013-11ef-bcea-c7b262605968@rgw.foo.a 2024-09-11T08:33:41.917 INFO:journalctl@ceph.rgw.foo.a.smithi150.stdout:Sep 11 08:33:41 smithi150 systemd[1]: Stopping Ceph rgw.foo.a for c1bddff4-7013-11ef-bcea-c7b262605968... 2024-09-11T08:33:42.201 INFO:journalctl@ceph.rgw.foo.a.smithi150.stdout:Sep 11 08:33:41 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-rgw-foo-a[65759]: 2024-09-11T08:33:41.916+0000 7fb981b21640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-11T08:33:42.202 INFO:journalctl@ceph.rgw.foo.a.smithi150.stdout:Sep 11 08:33:41 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-rgw-foo-a[65759]: 2024-09-11T08:33:41.916+0000 7fb9860297c0 -1 shutting down 2024-09-11T08:33:51.918 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:33:51 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-alertmanager-a[75052]: ts=2024-09-11T08:33:51.625Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi150.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.150:8443: connect: connection refused" 2024-09-11T08:33:51.918 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:33:51 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-alertmanager-a[75052]: ts=2024-09-11T08:33:51.626Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi181.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.181:8443: connect: connection refused" 2024-09-11T08:33:52.184 INFO:journalctl@ceph.rgw.foo.a.smithi150.stdout:Sep 11 08:33:51 smithi150 bash[219239]: time="2024-09-11T08:33:51Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-c1bddff4-7013-11ef-bcea-c7b262605968-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-09-11T08:33:52.184 INFO:journalctl@ceph.rgw.foo.a.smithi150.stdout:Sep 11 08:33:51 smithi150 podman[219239]: 2024-09-11 08:33:51.933256184 +0000 UTC m=+10.123731678 container died cee87a4b3901c6799e0a4c06b06fe4118900b377c6babeb96466d1b890c3d417 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-rgw-foo-a, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2) 2024-09-11T08:33:52.450 INFO:journalctl@ceph.rgw.foo.a.smithi150.stdout:Sep 11 08:33:52 smithi150 podman[219239]: 2024-09-11 08:33:52.432472829 +0000 UTC m=+10.622948318 container cleanup cee87a4b3901c6799e0a4c06b06fe4118900b377c6babeb96466d1b890c3d417 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-rgw-foo-a, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828) 2024-09-11T08:33:52.450 INFO:journalctl@ceph.rgw.foo.a.smithi150.stdout:Sep 11 08:33:52 smithi150 bash[219239]: ceph-c1bddff4-7013-11ef-bcea-c7b262605968-rgw-foo-a 2024-09-11T08:33:52.823 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c1bddff4-7013-11ef-bcea-c7b262605968@rgw.foo.a.service' 2024-09-11T08:33:52.860 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-11T08:33:52.860 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-09-11T08:33:52.860 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-09-11T08:33:52.860 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-c1bddff4-7013-11ef-bcea-c7b262605968@prometheus.a 2024-09-11T08:33:53.147 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:33:52 smithi181 systemd[1]: Stopping Ceph prometheus.a for c1bddff4-7013-11ef-bcea-c7b262605968... 2024-09-11T08:33:53.447 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:33:53 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[64841]: ts=2024-09-11T08:33:53.147Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-11T08:33:53.448 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:33:53 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[64841]: ts=2024-09-11T08:33:53.147Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-11T08:33:53.448 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:33:53 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[64841]: ts=2024-09-11T08:33:53.147Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-11T08:33:53.448 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:33:53 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[64841]: ts=2024-09-11T08:33:53.147Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-11T08:33:53.448 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:33:53 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[64841]: ts=2024-09-11T08:33:53.147Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-11T08:33:53.448 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:33:53 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[64841]: ts=2024-09-11T08:33:53.147Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-11T08:33:53.448 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:33:53 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[64841]: ts=2024-09-11T08:33:53.147Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-11T08:33:53.448 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:33:53 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[64841]: ts=2024-09-11T08:33:53.147Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-11T08:33:53.448 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:33:53 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[64841]: ts=2024-09-11T08:33:53.152Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-11T08:33:53.448 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:33:53 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[64841]: ts=2024-09-11T08:33:53.153Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-11T08:33:53.448 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:33:53 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[64841]: ts=2024-09-11T08:33:53.153Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-11T08:33:53.448 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:33:53 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a[64841]: ts=2024-09-11T08:33:53.153Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-11T08:33:53.448 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:33:53 smithi181 podman[164099]: 2024-09-11 08:33:53.157734427 +0000 UTC m=+0.108886217 container died 42045a4ed6d004ff9ac97c8f5f13818b06698ca3a11119dc4a9a1247e57b734b (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-11T08:33:53.814 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:33:53 smithi181 podman[164099]: 2024-09-11 08:33:53.555508745 +0000 UTC m=+0.506660534 container cleanup 42045a4ed6d004ff9ac97c8f5f13818b06698ca3a11119dc4a9a1247e57b734b (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-11T08:33:53.814 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Sep 11 08:33:53 smithi181 bash[164099]: ceph-c1bddff4-7013-11ef-bcea-c7b262605968-prometheus-a 2024-09-11T08:33:53.936 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c1bddff4-7013-11ef-bcea-c7b262605968@prometheus.a.service' 2024-09-11T08:33:53.971 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-11T08:33:53.971 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-09-11T08:33:53.971 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c1bddff4-7013-11ef-bcea-c7b262605968 --force --keep-logs 2024-09-11T08:33:55.738 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:33:55 smithi150 systemd[1]: Stopping Ceph alertmanager.a for c1bddff4-7013-11ef-bcea-c7b262605968... 2024-09-11T08:33:56.154 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:33:55 smithi150 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-alertmanager-a[75052]: ts=2024-09-11T08:33:55.879Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-11T08:33:56.155 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:33:55 smithi150 podman[219696]: 2024-09-11 08:33:55.920568075 +0000 UTC m=+0.150950498 container died 908683d933675cb66b516cec0428b4954ea70ed995d24014a63e0fab412ad320 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-11T08:33:56.488 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:33:56 smithi150 podman[219696]: 2024-09-11 08:33:56.346146262 +0000 UTC m=+0.576528736 container cleanup 908683d933675cb66b516cec0428b4954ea70ed995d24014a63e0fab412ad320 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-11T08:33:56.488 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:33:56 smithi150 bash[219696]: ceph-c1bddff4-7013-11ef-bcea-c7b262605968-alertmanager-a 2024-09-11T08:33:56.895 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:33:56 smithi150 podman[219707]: 2024-09-11 08:33:56.539239577 +0000 UTC m=+0.613401721 container remove 908683d933675cb66b516cec0428b4954ea70ed995d24014a63e0fab412ad320 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-11T08:33:56.895 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:33:56 smithi150 podman[219707]: 2024-09-11 08:33:56.623194886 +0000 UTC m=+0.697356984 volume remove e4279716c2bf91d2fae6197b032a9659fd7be03dbc35fc9d6c845bd64c2d4ee3 2024-09-11T08:33:57.238 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:33:56 smithi150 systemd[1]: ceph-c1bddff4-7013-11ef-bcea-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-11T08:33:57.238 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:33:56 smithi150 systemd[1]: Stopped Ceph alertmanager.a for c1bddff4-7013-11ef-bcea-c7b262605968. 2024-09-11T08:33:57.238 INFO:journalctl@ceph.alertmanager.a.smithi150.stdout:Sep 11 08:33:56 smithi150 systemd[1]: ceph-c1bddff4-7013-11ef-bcea-c7b262605968@alertmanager.a.service: Consumed 1.775s CPU time. 2024-09-11T08:33:57.238 INFO:journalctl@ceph.node-exporter.a.smithi150.stdout:Sep 11 08:33:57 smithi150 systemd[1]: Stopping Ceph node-exporter.a for c1bddff4-7013-11ef-bcea-c7b262605968... 2024-09-11T08:33:57.740 INFO:journalctl@ceph.node-exporter.a.smithi150.stdout:Sep 11 08:33:57 smithi150 podman[219817]: 2024-09-11 08:33:57.354763169 +0000 UTC m=+0.128972585 container died 41711bc00ac63b4762e48eda27bee7c87ba34dca19bb6289cd2c9fe806a535fe (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-11T08:33:58.084 INFO:journalctl@ceph.node-exporter.a.smithi150.stdout:Sep 11 08:33:57 smithi150 podman[219817]: 2024-09-11 08:33:57.850927785 +0000 UTC m=+0.625137190 container cleanup 41711bc00ac63b4762e48eda27bee7c87ba34dca19bb6289cd2c9fe806a535fe (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-11T08:33:58.085 INFO:journalctl@ceph.node-exporter.a.smithi150.stdout:Sep 11 08:33:57 smithi150 bash[219817]: ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-a 2024-09-11T08:33:58.348 INFO:journalctl@ceph.node-exporter.a.smithi150.stdout:Sep 11 08:33:58 smithi150 podman[219831]: 2024-09-11 08:33:58.085214564 +0000 UTC m=+0.734153817 container remove 41711bc00ac63b4762e48eda27bee7c87ba34dca19bb6289cd2c9fe806a535fe (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-11T08:33:58.348 INFO:journalctl@ceph.node-exporter.a.smithi150.stdout:Sep 11 08:33:58 smithi150 systemd[1]: ceph-c1bddff4-7013-11ef-bcea-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-09-11T08:33:58.348 INFO:journalctl@ceph.node-exporter.a.smithi150.stdout:Sep 11 08:33:58 smithi150 systemd[1]: ceph-c1bddff4-7013-11ef-bcea-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-09-11T08:33:58.348 INFO:journalctl@ceph.node-exporter.a.smithi150.stdout:Sep 11 08:33:58 smithi150 systemd[1]: Stopped Ceph node-exporter.a for c1bddff4-7013-11ef-bcea-c7b262605968. 2024-09-11T08:33:58.349 INFO:journalctl@ceph.node-exporter.a.smithi150.stdout:Sep 11 08:33:58 smithi150 systemd[1]: ceph-c1bddff4-7013-11ef-bcea-c7b262605968@node-exporter.a.service: Consumed 4.875s CPU time. 2024-09-11T08:33:58.845 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c1bddff4-7013-11ef-bcea-c7b262605968 --force --keep-logs 2024-09-11T08:34:00.573 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:34:00 smithi181 systemd[1]: Stopping Ceph iscsi.iscsi.a for c1bddff4-7013-11ef-bcea-c7b262605968... 2024-09-11T08:34:00.573 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:34:00 smithi181 ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a[59983]: debug Shutdown received 2024-09-11T08:34:10.823 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:34:10 smithi181 bash[164474]: time="2024-09-11T08:34:10Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-09-11T08:34:10.823 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:34:10 smithi181 podman[164474]: 2024-09-11 08:34:10.514813195 +0000 UTC m=+10.127638722 container died ef5f265239d83a21f073a7de0ee3d874eac3c3b9f2eb471ed20ae09b8d09f5da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-11T08:34:11.262 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:34:11 smithi181 podman[164474]: 2024-09-11 08:34:11.04454122 +0000 UTC m=+10.657366746 container cleanup ef5f265239d83a21f073a7de0ee3d874eac3c3b9f2eb471ed20ae09b8d09f5da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-11T08:34:11.262 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:34:11 smithi181 bash[164474]: ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a 2024-09-11T08:34:11.572 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:34:11 smithi181 podman[164489]: 2024-09-11 08:34:11.262509791 +0000 UTC m=+0.743914744 container remove ef5f265239d83a21f073a7de0ee3d874eac3c3b9f2eb471ed20ae09b8d09f5da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2) 2024-09-11T08:34:11.573 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:34:11 smithi181 systemd[1]: ceph-c1bddff4-7013-11ef-bcea-c7b262605968@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-09-11T08:34:11.859 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:34:11 smithi181 podman[164545]: 2024-09-11 08:34:11.608870176 +0000 UTC m=+0.115922417 container died 0a3be48bc1f301a21eb69d05d3bce0fd55ff6afbefc35e9e20270ec83d779b32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a-tcmu, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-11T08:34:12.179 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:34:11 smithi181 podman[164545]: 2024-09-11 08:34:11.965155493 +0000 UTC m=+0.472207780 container cleanup 0a3be48bc1f301a21eb69d05d3bce0fd55ff6afbefc35e9e20270ec83d779b32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a-tcmu, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240731) 2024-09-11T08:34:12.179 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:34:11 smithi181 bash[164545]: ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a-tcmu 2024-09-11T08:34:12.179 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:34:11 smithi181 systemd[1]: ceph-c1bddff4-7013-11ef-bcea-c7b262605968@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-09-11T08:34:12.179 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:34:11 smithi181 systemd[1]: ceph-c1bddff4-7013-11ef-bcea-c7b262605968@iscsi.iscsi.a.service: Unit process 59874 (conmon) remains running after unit stopped. 2024-09-11T08:34:12.179 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:34:11 smithi181 systemd[1]: ceph-c1bddff4-7013-11ef-bcea-c7b262605968@iscsi.iscsi.a.service: Unit process 164559 (podman) remains running after unit stopped. 2024-09-11T08:34:12.179 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:34:11 smithi181 systemd[1]: Stopped Ceph iscsi.iscsi.a for c1bddff4-7013-11ef-bcea-c7b262605968. 2024-09-11T08:34:12.180 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:34:11 smithi181 systemd[1]: ceph-c1bddff4-7013-11ef-bcea-c7b262605968@iscsi.iscsi.a.service: Consumed 3.209s CPU time. 2024-09-11T08:34:12.572 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Sep 11 08:34:12 smithi181 podman[164559]: 2024-09-11 08:34:12.297986516 +0000 UTC m=+0.692275504 container remove 0a3be48bc1f301a21eb69d05d3bce0fd55ff6afbefc35e9e20270ec83d779b32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-iscsi-iscsi-a-tcmu, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-11T08:34:12.573 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:34:12 smithi181 systemd[1]: Stopping Ceph grafana.a for c1bddff4-7013-11ef-bcea-c7b262605968... 2024-09-11T08:34:12.911 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:34:12 smithi181 podman[164677]: 2024-09-11 08:34:12.588877903 +0000 UTC m=+0.103813098 container died d20c42749742dd0da7ff419a232b487ec4285591c8d96fc3fd943b9351f7ba41 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a, vcs-type=git, vendor=Red Hat, Inc., version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=ubi8-container, summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-04-29T11:48:11.055897, distribution-scope=public, release=239.1651231664, io.openshift.tags=base rhel8, name=ubi8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, maintainer=Paul Cuzner , com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, io.buildah.version=1.32.0, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, architecture=x86_64) 2024-09-11T08:34:12.911 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:34:12 smithi181 podman[164677]: 2024-09-11 08:34:12.911422177 +0000 UTC m=+0.426357372 container cleanup d20c42749742dd0da7ff419a232b487ec4285591c8d96fc3fd943b9351f7ba41 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a, summary=Grafana Container configured for Ceph mgr/dashboard integration, maintainer=Paul Cuzner , vendor=Red Hat, Inc., version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64, vcs-type=git, description=Ceph Grafana Container, name=ubi8, build-date=2022-04-29T11:48:11.055897, io.openshift.tags=base rhel8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.component=ubi8-container, distribution-scope=public, release=239.1651231664, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.32.0, io.openshift.expose-services=) 2024-09-11T08:34:13.321 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:34:12 smithi181 bash[164677]: ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a 2024-09-11T08:34:13.321 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:34:13 smithi181 podman[164690]: 2024-09-11 08:34:13.129382419 +0000 UTC m=+0.542450612 container remove d20c42749742dd0da7ff419a232b487ec4285591c8d96fc3fd943b9351f7ba41 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-grafana-a, vcs-type=git, summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-04-29T11:48:11.055897, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.k8s.display-name=Red Hat Universal Base Image 8, description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8.5, vendor=Red Hat, Inc., release=239.1651231664, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, io.openshift.tags=base rhel8, io.buildah.version=1.32.0, architecture=x86_64, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, name=ubi8, com.redhat.component=ubi8-container, distribution-scope=public) 2024-09-11T08:34:13.321 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:34:13 smithi181 systemd[1]: ceph-c1bddff4-7013-11ef-bcea-c7b262605968@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-09-11T08:34:13.775 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:34:13 smithi181 systemd[1]: Stopping Ceph node-exporter.b for c1bddff4-7013-11ef-bcea-c7b262605968... 2024-09-11T08:34:13.776 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:34:13 smithi181 systemd[1]: ceph-c1bddff4-7013-11ef-bcea-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-09-11T08:34:13.776 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:34:13 smithi181 systemd[1]: Stopped Ceph grafana.a for c1bddff4-7013-11ef-bcea-c7b262605968. 2024-09-11T08:34:13.776 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Sep 11 08:34:13 smithi181 systemd[1]: ceph-c1bddff4-7013-11ef-bcea-c7b262605968@grafana.a.service: Consumed 13.674s CPU time. 2024-09-11T08:34:14.073 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:34:13 smithi181 podman[164800]: 2024-09-11 08:34:13.776072536 +0000 UTC m=+0.118428858 container died b9166bc40a5326f874f68362d7558e38183100b6e2db11c98759f0b27273aa7f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-11T08:34:14.399 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:34:14 smithi181 podman[164800]: 2024-09-11 08:34:14.148947282 +0000 UTC m=+0.491303592 container cleanup b9166bc40a5326f874f68362d7558e38183100b6e2db11c98759f0b27273aa7f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-11T08:34:14.399 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:34:14 smithi181 bash[164800]: ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b 2024-09-11T08:34:14.657 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:34:14 smithi181 podman[164813]: 2024-09-11 08:34:14.399955339 +0000 UTC m=+0.627836410 container remove b9166bc40a5326f874f68362d7558e38183100b6e2db11c98759f0b27273aa7f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c1bddff4-7013-11ef-bcea-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-11T08:34:14.657 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:34:14 smithi181 systemd[1]: ceph-c1bddff4-7013-11ef-bcea-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-09-11T08:34:14.658 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:34:14 smithi181 systemd[1]: ceph-c1bddff4-7013-11ef-bcea-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-09-11T08:34:14.658 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:34:14 smithi181 systemd[1]: Stopped Ceph node-exporter.b for c1bddff4-7013-11ef-bcea-c7b262605968. 2024-09-11T08:34:14.658 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Sep 11 08:34:14 smithi181 systemd[1]: ceph-c1bddff4-7013-11ef-bcea-c7b262605968@node-exporter.b.service: Consumed 4.503s CPU time. 2024-09-11T08:34:15.205 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-11T08:34:15.239 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-11T08:34:15.267 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-11T08:34:15.269 DEBUG:teuthology.misc:Transferring archived files from smithi150:/var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/crash to /home/teuthworker/archive/gabrioux-2024-09-11_06:53:43-orch:cephadm-wip-guits-quincy-2024-09-10-2008-quincy-distro-default-smithi/7901248/remote/smithi150/crash 2024-09-11T08:34:15.269 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/crash -- . 2024-09-11T08:34:15.323 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/crash: Cannot open: No such file or directory 2024-09-11T08:34:15.323 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2024-09-11T08:34:15.325 DEBUG:teuthology.misc:Transferring archived files from smithi181:/var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/crash to /home/teuthworker/archive/gabrioux-2024-09-11_06:53:43-orch:cephadm-wip-guits-quincy-2024-09-10-2008-quincy-distro-default-smithi/7901248/remote/smithi181/crash 2024-09-11T08:34:15.326 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar c -f - -C /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/crash -- . 2024-09-11T08:34:15.360 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /var/lib/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/crash: Cannot open: No such file or directory 2024-09-11T08:34:15.360 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now 2024-09-11T08:34:15.361 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-11T08:34:15.362 DEBUG:teuthology.orchestra.run.smithi150:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-09-11T08:34:15.393 INFO:teuthology.orchestra.run.smithi150.stderr:grep: /var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/ceph.log: No such file or directory 2024-09-11T08:34:15.395 INFO:tasks.cephadm:Compressing logs... 2024-09-11T08:34:15.395 DEBUG:teuthology.orchestra.run.smithi150:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-11T08:34:15.438 DEBUG:teuthology.orchestra.run.smithi181:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-11T08:34:15.477 INFO:teuthology.orchestra.run.smithi150.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-11T08:34:15.477 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/ceph-osd.0.log 2024-09-11T08:34:15.477 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/ceph-osd.1.log 2024-09-11T08:34:15.480 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/ceph-volume.log 2024-09-11T08:34:15.480 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/ceph-mon.c.log 2024-09-11T08:34:15.480 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/ceph-mon.a.log 2024-09-11T08:34:15.480 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/ceph-osd.2.log 2024-09-11T08:34:15.480 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/ceph-mgr.y.log 2024-09-11T08:34:15.480 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/ceph-client.rgw.foo.a.log 2024-09-11T08:34:15.480 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/ceph-osd.3.log 2024-09-11T08:34:15.480 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/ceph-client.rgw.foo.a.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-11T08:34:15.480 INFO:teuthology.orchestra.run.smithi150.stderr: 58.1% -- replaced with /var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/ceph-client.rgw.foo.a.log.gz 2024-09-11T08:34:15.505 INFO:teuthology.orchestra.run.smithi181.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-11T08:34:15.505 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/tcmu-runner.log 2024-09-11T08:34:15.505 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/objectstore_tool.2.log 2024-09-11T08:34:15.505 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/tcmu-runner.log: gzip -5 --verbose -- /var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/ceph-osd.7.log 2024-09-11T08:34:15.505 INFO:teuthology.orchestra.run.smithi181.stderr: 62.7% -- replaced with /var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/tcmu-runner.log.gz 2024-09-11T08:34:15.505 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/objectstore_tool.2.log: gzip -5 --verbose -- /var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/ceph-volume.log 2024-09-11T08:34:15.505 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/ceph-osd.4.log 2024-09-11T08:34:15.505 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/ceph-osd.6.log 2024-09-11T08:34:15.505 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/ceph-osd.5.log 2024-09-11T08:34:15.505 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/ceph-mgr.x.log 2024-09-11T08:34:15.505 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/ceph-mon.b.log 2024-09-11T08:34:15.505 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-11T08:34:15.581 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/ceph-mon.b.log: /var/log/ceph/cephadm.log: 92.2% -- replaced with /var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/ceph-mgr.x.log.gz 2024-09-11T08:34:15.641 INFO:teuthology.orchestra.run.smithi181.stderr: 94.9% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-11T08:34:15.764 INFO:teuthology.orchestra.run.smithi181.stderr: 91.2% -- replaced with /var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/ceph-volume.log.gz 2024-09-11T08:34:16.384 INFO:teuthology.orchestra.run.smithi181.stderr: 94.9% -- replaced with /var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/objectstore_tool.2.log.gz 2024-09-11T08:34:16.733 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/ceph-osd.3.log: /var/log/ceph/cephadm.log: 91.2% -- replaced with /var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/ceph-volume.log.gz 2024-09-11T08:34:16.771 INFO:teuthology.orchestra.run.smithi150.stderr: 95.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-11T08:34:18.308 INFO:teuthology.orchestra.run.smithi181.stderr: 91.1% -- replaced with /var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/ceph-mon.b.log.gz 2024-09-11T08:34:18.751 INFO:teuthology.orchestra.run.smithi150.stderr: 89.7% -- replaced with /var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/ceph-mgr.y.log.gz 2024-09-11T08:34:22.532 INFO:teuthology.orchestra.run.smithi150.stderr: 91.2% -- replaced with /var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/ceph-mon.c.log.gz 2024-09-11T08:34:25.289 INFO:teuthology.orchestra.run.smithi150.stderr: 90.4% -- replaced with /var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/ceph-mon.a.log.gz 2024-09-11T08:36:16.625 INFO:teuthology.orchestra.run.smithi181.stderr: 93.9% -- replaced with /var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/ceph-osd.4.log.gz 2024-09-11T08:38:06.705 INFO:teuthology.orchestra.run.smithi181.stderr: 93.3% -- replaced with /var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/ceph-osd.7.log.gz 2024-09-11T08:39:35.440 INFO:teuthology.orchestra.run.smithi181.stderr: 93.3% -- replaced with /var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/ceph-osd.5.log.gz 2024-09-11T08:40:30.923 INFO:teuthology.orchestra.run.smithi181.stderr: 92.7% -- replaced with /var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/ceph-osd.6.log.gz 2024-09-11T08:40:30.925 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-11T08:40:30.925 INFO:teuthology.orchestra.run.smithi181.stderr:real 6m15.473s 2024-09-11T08:40:30.925 INFO:teuthology.orchestra.run.smithi181.stderr:user 4m45.130s 2024-09-11T08:40:30.925 INFO:teuthology.orchestra.run.smithi181.stderr:sys 0m12.569s 2024-09-11T08:42:57.501 INFO:teuthology.orchestra.run.smithi150.stderr: 93.5% -- replaced with /var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/ceph-osd.1.log.gz 2024-09-11T08:44:01.910 INFO:teuthology.orchestra.run.smithi150.stderr: 93.6% -- replaced with /var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/ceph-osd.3.log.gz 2024-09-11T08:45:48.568 INFO:teuthology.orchestra.run.smithi150.stderr: 92.5% -- replaced with /var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/ceph-osd.2.log.gz 2024-09-11T08:46:18.931 INFO:teuthology.orchestra.run.smithi150.stderr: 92.7% -- replaced with /var/log/ceph/c1bddff4-7013-11ef-bcea-c7b262605968/ceph-osd.0.log.gz 2024-09-11T08:46:18.933 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-09-11T08:46:18.934 INFO:teuthology.orchestra.run.smithi150.stderr:real 12m3.482s 2024-09-11T08:46:18.934 INFO:teuthology.orchestra.run.smithi150.stderr:user 5m29.489s 2024-09-11T08:46:18.934 INFO:teuthology.orchestra.run.smithi150.stderr:sys 0m15.962s 2024-09-11T08:46:18.934 INFO:tasks.cephadm:Archiving logs... 2024-09-11T08:46:18.934 DEBUG:teuthology.misc:Transferring archived files from smithi150:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-09-11_06:53:43-orch:cephadm-wip-guits-quincy-2024-09-10-2008-quincy-distro-default-smithi/7901248/remote/smithi150/log 2024-09-11T08:46:18.936 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-11T08:47:10.937 DEBUG:teuthology.misc:Transferring archived files from smithi181:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-09-11_06:53:43-orch:cephadm-wip-guits-quincy-2024-09-10-2008-quincy-distro-default-smithi/7901248/remote/smithi181/log 2024-09-11T08:47:10.938 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-11T08:47:56.091 INFO:tasks.cephadm:Removing cluster... 2024-09-11T08:47:56.092 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c1bddff4-7013-11ef-bcea-c7b262605968 --force 2024-09-11T08:47:56.975 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c1bddff4-7013-11ef-bcea-c7b262605968 --force 2024-09-11T08:47:57.824 INFO:tasks.cephadm:Removing cephadm ... 2024-09-11T08:47:57.824 DEBUG:teuthology.orchestra.run.smithi150:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-11T08:47:57.844 DEBUG:teuthology.orchestra.run.smithi181:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-11T08:47:57.859 INFO:tasks.cephadm:Teardown complete 2024-09-11T08:47:57.860 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-11T08:47:57.869 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-11T08:47:57.869 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-11T08:47:57.886 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-11T08:47:57.991 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-11T08:47:58.000 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-11T08:47:58.000 DEBUG:teuthology.orchestra.run.smithi150:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T08:47:58.004 DEBUG:teuthology.orchestra.run.smithi181:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T08:47:58.017 INFO:teuthology.orchestra.run.smithi150.stderr:bash: line 1: ntpq: command not found 2024-09-11T08:47:58.018 INFO:teuthology.orchestra.run.smithi181.stderr:bash: line 1: ntpq: command not found 2024-09-11T08:47:58.034 INFO:teuthology.orchestra.run.smithi150.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-11T08:47:58.035 INFO:teuthology.orchestra.run.smithi150.stdout:=============================================================================== 2024-09-11T08:47:58.035 INFO:teuthology.orchestra.run.smithi150.stdout:^* hv01.front.sepia.ceph.com 3 6 377 6 -2062us[-2049us] +/- 57ms 2024-09-11T08:47:58.035 INFO:teuthology.orchestra.run.smithi150.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 1 +1751us[+1751us] +/- 79ms 2024-09-11T08:47:58.035 INFO:teuthology.orchestra.run.smithi150.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 64 +1024us[+1032us] +/- 65ms 2024-09-11T08:47:58.035 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T08:47:58.037 INFO:teuthology.orchestra.run.smithi181.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-11T08:47:58.037 INFO:teuthology.orchestra.run.smithi181.stdout:=============================================================================== 2024-09-11T08:47:58.037 INFO:teuthology.orchestra.run.smithi181.stdout:^* hv01.front.sepia.ceph.com 3 6 377 6 -2067us[-2020us] +/- 57ms 2024-09-11T08:47:58.038 INFO:teuthology.orchestra.run.smithi181.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 4 +1750us[+1750us] +/- 79ms 2024-09-11T08:47:58.038 INFO:teuthology.orchestra.run.smithi181.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 7 +1044us[+1091us] +/- 66ms 2024-09-11T08:47:58.038 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T08:47:58.038 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-11T08:47:58.047 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-11T08:47:58.048 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-11T08:47:58.056 DEBUG:teuthology.orchestra.run.smithi150:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-11T08:47:58.192 DEBUG:teuthology.orchestra.run.smithi181:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-11T08:47:58.255 DEBUG:teuthology.orchestra.run.smithi150:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-11T08:47:58.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:47:58.284 DEBUG:teuthology.orchestra.run.smithi181:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-11T08:47:58.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:47:58.310 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-11T08:47:58.319 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-11T08:47:58.328 INFO:teuthology.task.internal:Duration was 3276.967099 seconds 2024-09-11T08:47:58.328 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-11T08:47:58.339 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-11T08:47:58.339 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-11T08:47:58.342 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-11T08:47:58.405 INFO:teuthology.orchestra.run.smithi150.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-11T08:47:58.417 INFO:teuthology.orchestra.run.smithi181.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-11T08:47:58.630 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-11T08:47:58.630 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi150.front.sepia.ceph.com 2024-09-11T08:47:58.630 DEBUG:teuthology.orchestra.run.smithi150:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-11T08:47:58.651 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi181.front.sepia.ceph.com 2024-09-11T08:47:58.652 DEBUG:teuthology.orchestra.run.smithi181:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-11T08:47:58.679 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-11T08:47:58.679 DEBUG:teuthology.orchestra.run.smithi150:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-11T08:47:58.694 DEBUG:teuthology.orchestra.run.smithi181:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-11T08:47:58.749 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-11T08:47:58.750 DEBUG:teuthology.orchestra.run.smithi150:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-11T08:47:58.765 DEBUG:teuthology.orchestra.run.smithi181:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-11T08:48:00.395 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-11T08:48:00.405 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-11T08:48:00.406 DEBUG:teuthology.orchestra.run.smithi150:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-11T08:48:00.431 DEBUG:teuthology.orchestra.run.smithi181:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-11T08:48:00.468 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-11T08:48:00.478 DEBUG:teuthology.orchestra.run.smithi150:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-11T08:48:00.481 DEBUG:teuthology.orchestra.run.smithi181:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-11T08:48:00.514 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern = core 2024-09-11T08:48:00.551 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern = core 2024-09-11T08:48:00.573 DEBUG:teuthology.orchestra.run.smithi150:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-11T08:48:00.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:48:00.588 DEBUG:teuthology.orchestra.run.smithi181:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-11T08:48:00.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:48:00.628 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-11T08:48:00.639 INFO:teuthology.task.internal:Transferring archived files... 2024-09-11T08:48:00.640 DEBUG:teuthology.misc:Transferring archived files from smithi150:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-09-11_06:53:43-orch:cephadm-wip-guits-quincy-2024-09-10-2008-quincy-distro-default-smithi/7901248/remote/smithi150 2024-09-11T08:48:00.640 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-11T08:48:00.712 DEBUG:teuthology.misc:Transferring archived files from smithi181:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-09-11_06:53:43-orch:cephadm-wip-guits-quincy-2024-09-10-2008-quincy-distro-default-smithi/7901248/remote/smithi181 2024-09-11T08:48:00.713 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-11T08:48:00.756 INFO:teuthology.task.internal:Removing archive directory... 2024-09-11T08:48:00.756 DEBUG:teuthology.orchestra.run.smithi150:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-11T08:48:00.758 DEBUG:teuthology.orchestra.run.smithi181:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-11T08:48:00.806 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-11T08:48:00.817 INFO:teuthology.task.internal:Not uploading archives. 2024-09-11T08:48:00.817 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-11T08:48:00.827 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-11T08:48:00.827 DEBUG:teuthology.orchestra.run.smithi150:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-11T08:48:00.830 DEBUG:teuthology.orchestra.run.smithi181:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-11T08:48:00.842 INFO:teuthology.orchestra.run.smithi150.stdout: 265745 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 11 08:48 /home/ubuntu/cephtest 2024-09-11T08:48:00.862 INFO:teuthology.orchestra.run.smithi181.stdout: 265637 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 11 08:48 /home/ubuntu/cephtest 2024-09-11T08:48:00.863 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-11T08:48:00.873 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-11T08:48:00.898 INFO:teuthology.run:Summary data: description: orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} duration: 3276.967099428177 flavor: default owner: scheduled_gabrioux@teuthology success: true 2024-09-11T08:48:00.898 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-11T08:48:00.963 INFO:teuthology.run:pass